Fundamentals: APPENDIX A: SPARKS

 Return to Table of Contents

APPENDIX A: SPARKS

This section is meant for people who do most of their programming in FORTRAN. FORTRAN has the distinction of being essentially the earliest higher level programming language, developed about 1957 by a group at IBM. Since then it and its derivatives have become established as the primary language for scientific and engineering computation. But, with our greater understanding of the process of creating programs has come a realization of the deficiencies of FORTRAN. Creating a program is properly thought of as taking a real world problem and translating it into a computer solution. Concepts in the real world such as a geneology tree or a queue of airplanes must be translated into computer concepts. A language is good if it enables one to describe these abstractions of the real world in a natural way. Perhaps because of its very early development, FORTRAN lacks many such features. In this appendix we explore the idea of writing a preprocessor for FORTRAN which inexpensively adds some of these missing features.

A preprocessor is a program which translates statements written in a language X into FORTRAN. In our case X is called SPARKS. Such a program is normally called a compiler so why give it the special name preprocessor? A preprocessor is distinguished from a compiler in the following way: the source and target language have many statements in common.

Such a translator has many advantages. Most importantly it preserves a close connection with FORTRAN. Despite FORTRAN's many negative attributes, it has several practical pluses: 1) it is almost always available and compilers are often good, 2) there is a language standard which allows a degree of portability not obtainable with other languages, 3) there are extensive subroutine libraries, and 4) there is a large labor force familiar with it. These reasons give FORTRAN a strong hold in the industrial marketplace. A structured FORTRAN translator preserves these virtues while it augments the language with improved syntactical constructs and other useful features.

Another consideration is that at many installations a nicely structured language is unavailable. In this event a translator provides a simple means for supplementing an existing FORTRAN capability. The translator to be described here can be obtained by writing to the address given at the end of this appendix.

In order to see the difference between FORTRAN and SPARKS consider writing a program which searches for X in the sorted array of integers A (N), N [image] 100. The output is the integer J which is either zero if X is not found or A(J) = X, 1 [image] J [image] N. The method used here is the well known binary search algorithm. The FORTRAN version looks something like this:

SUBROUTINE BINS (A,N,X,J)

IMPLICIT INTEGER (A - Z)

DIMENSION A(100)

BOT= 1

TOP= N

J = 0

100  IF (BOT. GT. TOP) RETURN

MID = (BOT + TOP)/2

IF (X. GE. A (MID)) GO TO 101

TOP = MID - 1

GO TO 100

101   IF (X. EQ. A (MID)) GO TO 102

BOT = MID + 1

GO TO 100

102   J = MID

RETURN

END

This may not be the "best" way to write this program, but it is a reasonable attempt. Now we write this algorithm in SPARKS.

SUBROUTINE BINS (A,N,X,J)

IMPLICIT INTEGER (A - Z)

DIMENSION A(100)

BOT = 1; TOP = N; J = 0

WHILE BOT. LE. TOP DO

MID = (BOT + TOP)/2

CASE

: X. LT. A(MID): TOP = MID - 1

: X. GT. A(MID): BOT = MID + 1

:ELSE: J = MID; RETURN

ENDCASE

REPEAT

RETURN

END

The difference between these two algorithms may not be dramatic, but it is significant. The WHILE and CASE statements allow the algorithm to be described in a more natural way. The program can be read from top to bottom without your eyes constantly jumping up and down the page. When such improvements are consistently adopted in a large software project, the resulting code is bound to be easier to comprehend.

We begin by defining precisely the SPARKS language. A distinction is made between FORTRAN statements and SPARKS statements. The latter are recognized by certain keywords and/or delimiters. All other statements are regarded as FORTRAN and are passed directly to the FORTRAN compiler without alteration. Thus, SPARKS is compatible with FORTRAN and a FORTRAN program is a SPARKS program. SPARKS statements cause the translator to produce ANSI FORTRAN statements which accomplish the equivalent computation. Hence, the local compiler ultimately defines the semantics of all SPARKS statements.

The reserved words and special symbols are:

BY       CASE    CYCLE   DO    ELSE   ENDCASE

ENDIF    EOJ     EXIT    FOR   IF     LOOP

REPEAT   UNTIL   WHILE   TO    THEN    :

;         //

Reserved words must always be surrounded by blanks. Reserved means they cannot be used by the programmer as variables.

We now define the SPARKS statements by giving their FORTRAN equivalents. In the following any reference to the term "statements" is meant to include both SPARKS and FORTRAN statements. There are six basic SPARKS statements, two which improve the testing of cases and four which improve the description of looping.

IF cond THEN           IF(.NOT. (cond)) GO TO 100

      S1                   S1

  ELSE                     GO TO 101

      S2           100   S2

ENDIF              101   CONTINUE

S1 and S2 are arbitrary size groups of statements. Cond must be a legal FORTRAN conditional. The ELSE clause is optional but the ENDIF is required and it always terminates the innermost IF.

S1,S2, ...,Sn+1 are arbitrary size groups of statements. Cond1, cond2, ..., condn are legal FORTRAN conditionals. The symbol ELSE surrounded by colons designates that Sn+l will be automatically executed if all previous conditions are false. This part of the case statement is optional.

The four looping statements are:

WHILE cond DO        100   IF(.NOT. (cond)) GO TO 101

   S                            S

REPEAT                     GO TO 100

                     101   CONTINUE

S is an arbitrary group of statements and cond a legal FORTRAN conditional.

LOOP                 100   CONTINUE

   S                       S

UNTIL cond REPEAT            IF(.NOT. (cond)) GO TO 100

S and cond are the same as for the while statement immediately preceding.

LOOP            100        CONTINUE

   S                       S

REPEAT                     GO TO 100

                101        CONTINUE

S is an arbitrary size group of statements.

FOR vble = expl TO exp2 BY exp3 DO

   S

REPEAT

This has a translation of:

     vble = expl

     GO TO 100

102  vble = vble + exp3

100  IF ((vble - (exp2))*(exp3).GT. 0) GO TO 101

       S

       GO TO 102

101  CONTINUE

The three expressions exp1, exp2, exp3 are allowed to be arbitrary FORTRAN arithmetic expressions of any type. Similarly vble may be of any type. However, the comparison test is made against integer zero. Since exp2 and exp3 are re-evaluated each time through the loop, care must be taken in its use.

EXIT is a SPARKS statement which causes a transfer of control to the first statement outside of the innermost LOOP-REPEAT statement which contains it. One example of its use is:

LOOP              100      CONTINUE

 S1                            S1

 IF cond THEN EXIT         IF(.NOT. (cond)) GO TO 102

 ENDIF                       GO TO 101

 S2               102      CONTINUE

REPEAT                     S2

                           GO TO 100

                  101      CONTINUE

A generalization of this statement allows EXIT to be used within any of the four SPARKS looping statements: WHILE, LOOP, LOOP-UNTIL and FOR. When executed, EXIT branches to the statement immediately following the innermost looping statement which contains it.

The statement CYCLE is also used within any SPARKS looping statement. Its execution causes a branch to the end of the innermost loop which contains it. A test may be made and if passed the next iteration is taken. An example of the use of EXIT and CYCLE follow.

LOOP              100      CONTINUE

   S1                           S1

 CASE                      IF(.NOT. (cond1) GO TO 103

  : cond1 : EXIT           GO TO 102

  : cond2 : CYCLE          IF(.NOT. (cond2)) GO TO 104

ENDCASE           103      GO TO 101

S2                             CONTINUE

REPEAT            104        S2

                           GO TO 100

                  101      CONTINUE

                  102

EOJ or end of job must appear at the end of the entire SPARKS program. As a statement, it must appear somewhere in columns 7 through 72 and surrounded by blanks.

ENDIF is used to terminate the IF and ENDCASE to terminate the CASE statement. REPEAT terminates the looping statements WHILE, LOOP and FOR.

Labels follow the FORTRAN convention of being numeric and in columns one to five.

The use of doubleslash is as a delimiter for comments. Thus one can write

//This is a comment//

and all characters within the double slashes will be ignored. Comments are restricted to one line and FORTRAN comments are allowed.

The semi-colon can be used to include more than one statement on a single line For example, beginning in column one the statement

99999 A = B + C; C = D + E; X = A

would be legal in SPARKS. To include a semicolon in a hollerith field it should be followed by a second semicolon. This will be deleted in the resulting FORTRAN.

We are now ready to describe the operation of the translator. Two design approaches are feasible. The first is a table-driven method which scans a program and recognizes keywords. This approach is essentially the way a compiler works in that it requires a scanner, a symbol table (though limited), very limited parsing and the generation of object (FORTRAN) code. A second approach is to write a general macro preprocessor and then to define each SPARKS statement as a new macro. Such a processor is usually small and allows the user to easily define new constructs. However, these processors tend to be slower than the approach of direct translation. Moreover, it is hard to build in the appropriate error detection and recovery facilities which are sorely needed if SPARKS is to be used seriously. Therefore, we have chosen the first approach. Figure A.1 contains a flow description of the translator.

Figure A.1: Overview of SPARKS Translator

The main processing loop consists of determining the next statement and branching within a large CASE. This does whatever translation into FORTRAN is necessary. When EOJ is found the loop is broken and the program is concluded.

The SPARKS translator was first written in SPARKS. The original version was hand translated into FORTRAN to produce our first running system. Since that time it has been used by a variety of people and classes. Thus it is running far better than the original version. Nevertheless, the translator has not been proved correct and so it must be used with caution.

Extensions

Below is a list of possible extensions for SPARKS. Some are relatively easy to implement, while others require a great deal of effort.

E.1 Special cases of the CASE statement

      CASE SGN  : exp:             CASE: integer variable:

        : .EQ.0 : S1                   : 1 : S1

        : .LT.0 : S2        and       : 2 : S2

        : .GT.0 : S3

      ENDCASE                        : n : Sn

                                   ENDCASE

The first gets translated into the FORTRAN arithmetic IF statement. The second form is translated into a FORTRAN computed go to.

E.2 A simple form of the FOR statement would look like

      LOOP exp TIMES

        S

      REPEAT

where exp is an expression which evaluates to a non-negative integer. The statements meaning can be described by the SPARKS for statement:

   FOR ITEMP = 1 TO exp DO

      S

   REPEAT

An internal integer variable ITEMP must be created.

E.3 If F appears in column one then all subsequent cards are assumed to be pure FORTRAN. They are passed directly to the output until an F is encountered in column one.

E.4 Add the capability of profiling a program by determining the number of executions of each loop during a single execution and the value of conditional expressions.

HINT: For each subroutine declare a set of variables which can be inserted after encountering a WHILE, LOOP, REPEAT, FOR, THEN or ELSE statement. At the end of each subroutine a write statement prints the values of these counters.

E.5 Add the multiple replacement statement so that

A = B = C = D + E

is translated into

C = D + E; B = C; A = B

E.6 Add the vector replacement statement so that

(A,B,C) = (X + Y, 10,2*E)

produces A = X + Y: B = 10 ; C = 2*E

E.7 Add an array "fill" statement so that

NAME(*) [image] exp1,exp2,exp3

gets translated into

NAME(1) = exp1; NAME(2) = exp2; NAME(3) = exp3

E.8 Introduce appropriate syntax and reasonable conventions so that SPARKs programs can be recursive.

HINT: Mutually recursive programs are gathered together in a module, MODULE (X(A,B,C)(100)) whose name is X, whose parameters are A,B,C and whose stack size should be 100.

E.9 Add a character string capability to SPARKS.

E.10 Add an internal procedure capability to aid the programmer in doing top-down program refinement.

E.11 Attach sequence numbers to the resulting FORTRAN output which relates each statement back to the original SPARKS statement which generated it. This is particularly helpful for debugging.

E.12 Along with the indented SPARKS source print a number which represents the level of nesting of each statement.

E.13 Generalize the EXIT statement so that upon its execution it can be assigned a value, e.g.,

      LOOP

        S1

        IF condl THEN EXIT : exp1: ENDIF

        S2

        IF cond2 THEN EXIT : exp2: ENDIF

        S3

      REPEAT

will assign either expl or exp2 as the value of the variable EXIT.

E.14 Supply a simplified read and write statement. For example, allow for hollerith strings to be included within quotes and translated to the nH x1 ... xn format.

All further questions about the definition of SPARKS should be addressed to:

Chairman, SPARKS Users Group

Computer Science, Powell Hall

University of Southern California

Los Angeles, California 9007

To receive a complete ANSI FORTRAN version of SPARKS send $20.00 (for postage and handling) to Dr. Ellis Horowitz at the above address.

Go to Appendix B Back to Table of Contents


Fundamentals: APPENDIX B: ETHICAL CODE IN INFORMATION PROCESSING

 Return to Table of Contents

APPENDIX B: ETHICAL CODE IN INFORMATION PROCESSING

ACM CODE OF PROFESSIONAL CONDUCT

PREAMBLE

Recognition of professional status by the public depends not only on skill and dedication but also on adherence to a recognized code of Professional Conduct. The following Code sets forth the general principles (Canons), professional ideals (Ethical Considerations), and mandatory rules (Disciplinary Rules) applicable to each ACM Member.

The verbs "shall" (imperative) and "should" (encouragement) are used purposefully in the Code. The Canons and Ethical Considerations are not, however, binding rules. Each Disciplinary Rule is binding on each individual Member of ACM. Failure to observe the Disciplinary Rules subjects the Member to admonition, suspension, or expulsion from the Association as provided by the Constitution and Bylaws. The term "member(s)" is used in the Code. The Disciplinary Rules of the Code apply, however, only to the classes of membership specified in Article 3, Section 4, of the Constitution of the ACM.

CANON 1

An ACM member shall act at all times with integrity.

Ethical Considerations

EC1.1. An ACM member shall properly qualify himself when expressing an opinion outside his areas of competence. A member is encouraged to express his opinion on subjects within his areas of competence.

EC1.2. An ACM member shall preface an partisan statements about information processing by indicating clearly on whose behalf they are made.

EC1.3. An ACM member shall act faithfully on behalf of his employers or clients.

Disciplinary Rules

DR1.1.1. An ACM member shall not intentionally misrepresent his qualifications or credentials to present or prospective employers or clients.

DR1.1.2. An ACM member shall not make deliberately false or deceptive statements as to the present or expected state of affairs in any aspect of the capability, delivery, or use of information processing systems.

DR1.2.1. An ACM member shall not intentionally conceal or misrepresent on whose behalf any partisan statements are made.

DR1.3.1. An ACM member acting or employed as a consultant shall, prior to accepting information from a prospective client, inform the client of all factors of which the member is aware which may affect the proper performance of the task.

DR1.3.2. An ACM member shall disclose any interest of which he is aware which does or may conflict with his duty to a present or prospective employer or client.

DR1.3.3. An ACM member shall not use any confidential information from any employer or client, past or present, without prior permission.

CANON 2

An ACM member should strive to increase his competence and the competence and prestige of the profession.

Ethical Considerations

EC2.1. An ACM member is encouraged to extend public knowledge, understanding, and appreciation of information processing, and to oppose any false or deceptive statements relating to information processing of which he is aware.

EC2.2 An ACM member shall not use his professional credentials to misrepresent his competence.

EC2.3. An ACM member shall undertake only those professional assignments and commitments for which he is qualified.

EC2.4. An ACM member shall strive to design and develop systems that adequately perform the intended functions and that satisfy his employer's or client's operational needs.

EC2.5. An ACM member should maintain and increase his competence through a program of continuing education encompassing the techniques, technical standards, and practices in his fields of professional activity.

EC2.6. An ACM member should provide opportunity and encouragement for professional development and advancement of both professionals and those aspiring to become professionals.

Disciplinary Rules

DR2.2.1. An ACM member shall not use his professional credentials to misrepresent his competence.

DR2.3.1. An ACM member shall not undertake professional assignments without adequate preparation in the circumstances.

DR2.3.2. An ACM member shall not undertake professional assignments for which he knows or should know he is not competent or cannot become adequately competent without acquiring the assistance of a professional who is competent to perform the assignment.

DR2.4.1. An ACM member shall not represent that a product of his work will perform its function adequately and will meet the receiver's operational needs when he knows or should know that the product is deficient.

CANON 3

An ACM member shall accept responsibility for his work.

Ethical Considerations

EC3.1. An ACM member shall accept only those assignments for which there is reasonable expectancy of meeting requirements or specifications, and shall perform his assignments in a professional manner.

Disciplinary Rules

DR3.1.1. An ACM member shall not neglect any professional assignment which has been accepted.

DR3.1.2. An ACM member shall keep his employer or client properly informed on the progress of his assignments.

DR3.1.3. An ACM member shall not attempt to exonerate himself from, or to limit, his liability to his clients for his personal malpractice.

DR3.1.4. An ACM member shall indicate to his employer or client the consequences to be expected if his professional judgement is overruled.

CANON 4

An ACM member shall act with professional responsibility.

Ethical Considerations

EC4.1 An ACM member shall not use his membership in ACM improperly for professional advantage or to misrepresent the authority of his statements.

EC4.2. An ACM member shall conduct professional activities on a high plane.

EC4.3. An ACM member is encouraged to uphold and improve the professional standards of the Association through participation in their formulation, establishment, and enforcement.

Disciplinary Rules

DR4.1.1. An ACM member shall not speak on behalf of the Association or any of its subgroups without proper authority.

DR4.1.2. An ACM member shall not knowingly misrepresent the policies and views of the Association or any of its subgroups.

DR4.1.3. An ACM member shall preface partisan statements about information processing by indicating clearly on whose behalf they are made.

DR4.2.1. An ACM member shall not maliciously injure the professional reputation of any other person.

DR4.2.2. An ACM member shall not use the services of or his membership in the Association to gain unfair advantage.

DR4.2.3. An ACM member shall take care that credit for work is given to whom credit is properly due.

CANON 5

An ACM member should use his special knowledge and skills for the advancement of human welfare.

Ethical Considerations

EC5.1. An ACM member should consider the health, privacy, and general welfare of the public in the performance of his work.

EC5.2. An ACM member, whenever dealing with data concerning individuals, shall always consider the principle of the individual's privacy and seek the following:

--To minimize the data collected.

--To limit authorized access to the data.

--To provide proper security for the data.

--To determine the required retention period of the data.

--To ensure proper disposal of the data.

Disciplinary rules

DR5.2.1. An ACM member shall express his professional opinion to his employers or clients regarding any adverse consequences to the public which might result from work proposed to him.

Go to Appendix C Back to Table of Contents


Fundamentals: APPENDIX C: ALGORITHM INDEX BY CHAPTER

 Return to Table of Contents

APPENDIX C: ALGORITHM INDEX BY CHAPTER

CHAPTER 1--INTRODUCTION

Exchange sorting--SORT 1.3

Binary search--BINSRCH 1.3

FIBONACCI 1.4

Filling a magic square--MAGIC 1.4

CHAPTER 2--ARRAYS

Polynomial addition--PADD 2.2

Fibonacci polynomials--MAIN 2.2

Sparse matrix transpose--TRANSPOSE 2.3

Sparse matrix transpose--FAST-TRANSPOSE 2.3

Sparse matrix multiplication--MMULT 2.3

CHAPTER 3--STACKS AND QUEUES

Sequential stacks--ADD, DELETE 3.1

Sequential queues--ADDQ, DELETEQ 3.1

Circular sequential queues--ADDQ, DELETEQ 3.1

Path through a maze--PATH 3.2

Evaluation of a postfix expression--EVAL 3.3

Translating infix to postfix--POSTFIX 3.3

M-stacks sequential--ADD, DELETE 3.4

CHAPTER 4--LINKED LISTS

Create a list with 2 nodes--CREATE 2 4.1

Insert a node in a list--INSERT 4.1

Delete a node from a list--DELETE 4.1

Linked stacks--ADDS, DELETES 4.2

Linked queues--ADDQ, DELETEQ 4.2

Finding a new node--GETNODE 4.3

Initializing available space--INIT 4.3

Returning a node to available space--RET 4.3

Linked polynomial addition--PADD 4.4

Erasing a linked list--ERASE 4.4

Erasing a circular list--CERASE 4.4

Circular linked polynomial addition--CPADD 4.4

Inverting a linked list--INVERT 4.5

Concatenating two lists--CONCATENATE 4.5

Circular list insertion--INSERT-FRONT 4.5

Length of a circular list--LENGTH 4.5

Finding equivalence classes--EQUIVALENCE 4.6

Reading in a sparse matrix--MREAD 4.7

Erasing a linked sparse matrix--MERASE 4.7

Doubly linked list--DDELETE, DINSERT 4.8

First fit allocation--FF, ALLOCATE 4.8

Returning freed blocks--FREE

General lists--COPY, EQUAL, DEPTH 4.9

Erasing a list with reference counts--ERASE 4.9

Garbage collection--DRIVER, MARK 1, MARK 2 4.10

Compacting storage--COMPACT 4.10

Strings--SINSERT 4.11

Finding a pattern--FIND, NFIND, PMATCH, FAIL 4.11

Initializing available space in FORTRAN-INIT 4.12

Initializing available space in PASCAL-INIT 4.12

FORTRAN node routines--COEP, SCOEF, EXP, SEXP, LINK, SLINK 4.12

CHAPTER 5--TREES

Inorder traversal--INORDER,INORDER1,2,3 5.4

Preorder traversal--PREORDER 5.4

Postorder traversal--POSTORDER 5.4

Copy a binary tree--COPY 5.5

Equivalence of binary trees--EQUAL 5.5

Evaluating propositional expressions--POSTORDER-EVAL 5.5

The inorder successor--INSUC 5.6

Traversing a threaded binary tree--TINORDER 5.6

Insertion in a threaded tree--INSERT-RIGHT 5.6

Disjoint set union--U, F, UNION, FIND 5.8

Decision trees--EIGHTCOINS 5.8

Game playing--VE,AB(alpha-beta) 5.8

CHAPTER 6--GRAPHS

Depth first search--DFS 6.2

Breadth first search--BFS 6.2

Connected components--COMP 6.2

Minimum spanning tree--KRUSKAL 6.2

Shortest path, single source--SHORTEST-PATH 6.3

Shortest paths, ALL__COSTS 6.3

TOPOLOGICAL-ORDER 6.4

The first m shortest paths--M-SHORTEST 6.5

CHAPTER 7--INTERNAL SORTING

Sequential search--SEQSRCH 7.1

Binary search--BINSRCH 7.1

Fibonacci search--FIBSRCH 7.1

Determining equality of two lists--VERIFY1, VERIFY2 7.1

Insertion sorting--INSERT, INSORT 7.2

Quicksort--QSORT 7.3

Two-way merge sort--MERGE, MPASS, MSORT, RMSORT 7.5

Heapsort--ADJUST, HSORT 7.6

Radix sorting--LRSORT 7.7

Rearranging sorted records--LIST1, LIST2, TABLE 7.8

CHAPTER 8--EXTERNAL SORTING

K-way merging 8.2

K-way sorting--BUFFERING 8.2

Run generation--RUNS 8.2

Tape k-way merge--Ml, M2, M3 8.3

CHAPTER 9--SYMBOL TABLES

Binary search trees--SEARCH 9.1

Minimum external path length--HUFFMAN 9.1

Optimal binary search tree--OBST 9.1

Binary search tree insertion--BST, AVL-INSERT 9.2

Hashing--LINSRCH, CHNSRCH 9.3

CHAPTER 10--FILES

Searching an m-way tree--MSEARCH 10.2

Inserting into a B-tree--INSERTB 10.2

Deletion from a B-tree--DELETEB 10.2

Tree searching--TRIE 10.2

Back to Table of Contents


Fundamentals: CHAPTER 1: INTRODUCTION

 Return to Table of Contents

CHAPTER 1: INTRODUCTION

1.1 OVERVIEW

The field of computer science is so new that one feels obliged to furnish a definition before proceeding with this book. One often quoted definition views computer science as the study of algorithms. This study encompasses four distinct areas:

(i) machines for executing algorithms--this area includes everything from the smallest pocket calculator to the largest general purpose digital computer. The goal is to study various forms of machine fabrication and organization so that algorithms can be effectively carried out.

(ii) languages for describing algorithms--these languages can be placed on a continuum. At one end are the languages which are closest to the physical machine and at the other end are languages designed for sophisticated problem solving. One often distinguishes between two phases of this area: language design and translation. The first calls for methods for specifying the syntax and semantics of a language. The second requires a means for translation into a more basic set of commands.

(iii) foundations of algorithms--here people ask and try to answer such questions as: is a particular task accomplishable by a computing device; or what is the minimum number of operations necessary for any algorithm which performs a certain function? Abstract models of computers are devised so that these properties can be studied.

(iv) analysis of algorithms--whenever an algorithm can be specified it makes sense to wonder about its behavior. This was realized as far back as 1830 by Charles Babbage, the father of computers. An algorithm's behavior pattern or performance profile is measured in terms of the computing time and space that are consumed while the algorithm is processing. Questions such as the worst and average time and how often they occur are typical.

We see that in this definition of computer science, "algorithm" is a fundamental notion. Thus it deserves a precise definition. The dictionary's definition "any mechanical or recursive computational procedure" is not entirely satisfying since these terms are not basic enough.

Definition: An algorithm is a finite set of instructions which, if followed, accomplish a particular task. In addition every algorithm must satisfy the following criteria:

(i) input: there are zero or more quantities which are externally supplied;

(ii) output: at least one quantity is produced;

(iii) definiteness: each instruction must be clear and unambiguous;

(iv) finiteness: if we trace out the instructions of an algorithm, then for all cases the algorithm will terminate after a finite number of steps;

(v) effectiveness: every instruction must be sufficiently basic that it can in principle be carried out by a person using only pencil and paper. It is not enough that each operation be definite as in (iii), but it must also be feasible.

In formal computer science, one distinguishes between an algorithm, and a program. A program does not necessarily satisfy condition (iv). One important example of such a program for a computer is its operating system which never terminates (except for system crashes) but continues in a wait loop until more jobs are entered. In this book we will deal strictly with programs that always terminate. Hence, we will use these terms interchangeably.

An algorithm can be described in many ways. A natural language such as English can be used but we must be very careful that the resulting instructions are definite (condition iii). An improvement over English is to couple its use with a graphical form of notation such as flowcharts. This form places each processing step in a "box" and uses arrows to indicate the next step. Different shaped boxes stand for different kinds of operations. All this can be seen in figure 1.1 where a flowchart is given for obtaining a Coca-Cola from a vending machine. The point is that algorithms can be devised for many common activities.

Have you studied the flowchart? Then you probably have realized that it isn't an algorithm at all! Which properties does it lack?

Returning to our earlier definition of computer science, we find it extremely unsatisfying as it gives us no insight as to why the computer is revolutionizing our society nor why it has made us re-examine certain basic assumptions about our own role in the universe. While this may be an unrealistic demand on a definition even from a technical point of view it is unsatisfying. The definition places great emphasis on the concept of algorithm, but never mentions the word "data". If a computer is merely a means to an end, then the means may be an algorithm but the end is the transformation of data. That is why we often hear a computer referred to as a data processing machine. Raw data is input and algorithms are used to transform it into refined data. So, instead of saying that computer science is the study of algorithms, alternatively, we might say that computer science is the study of data:

(i) machines that hold data;

(ii) languages for describing data manipulation;

(iii) foundations which describe what kinds of refined data can be produced from raw data;

(iv) structures for representing data.

Figure 1.1: Flowchart for obtaining a Coca-Cola

There is an intimate connection between the structuring of data, and the synthesis of algorithms. In fact, a data structure and an algorithm should be thought of as a unit, neither one making sense without the other. For instance, suppose we have a list of n pairs of names and phone numbers (a1,b1)(a2,b2), ..., (an,bn), and we want to write a program which when given any name, prints that person's phone number. This task is called searching. Just how we would write such an algorithm critically depends upon how the names and phone numbers are stored or structured. One algorithm might just forge ahead and examine names, a1,a2,a3, ... etc., until the correct name was found. This might be fine in Oshkosh, but in Los Angeles, with hundreds of thousands of names, it would not be practical. If, however, we knew that the data was structured so that the names were in alphabetical order, then we could do much better. We could make up a second list which told us for each letter in the alphabet, where the first name with that letter appeared. For a name beginning with, say, S, we would avoid having to look at names beginning with other letters. So because of this new structure, a very different algorithm is possible. Other ideas for algorithms become possible when we realize that we can organize the data as we wish. We will discuss many more searching strategies in Chapters 7 and 9.

Therefore, computer science can be defined as the study of data, its representation and transformation by a digital computer. The goal of this book is to explore many different kinds of data objects. For each object, we consider the class of operations to be performed and then the way to represent this object so that these operations may be efficiently carried out. This implies a mastery of two techniques: the ability to devise alternative forms of data representation, and the ability to analyze the algorithm which operates on that structure . The pedagogical style we have chosen is to consider problems which have arisen often in computer applications. For each problem we will specify the data object or objects and what is to be accomplished. After we have decided upon a representation of the objects, we will give a complete algorithm and analyze its computing time. After reading through several of these examples you should be confident enough to try one on your own.

There are several terms we need to define carefully before we proceed. These include data structure, data object, data type and data representation. These four terms have no standard meaning in computer science circles, and they are often used interchangeably.

A data type is a term which refers to the kinds of data that variables may "hold" in a programming language. In FORTRAN the data types are INTEGER, REAL, LOGICAL, COMPLEX, and DOUBLE PRECISION. In PL/I there is the data type CHARACTER. The fundamental data type of SNOBOL is the character string and in LISP it is the list (or S-expression). With every programming language there is a set of built-in data types. This means that the language allows variables to name data of that type and provides a set of operations which meaningfully manipulates these variables. Some data types are easy to provide because they are already built into the computer's machine language instruction set. Integer and real arithmetic are examples of this. Other data types require considerably more effort to implement. In some languages, there are features which allow one to construct combinations of the built-in types. In COBOL and PL/I this feature is called a STRUCTURE while in PASCAL it is called a RECORD. However, it is not necessary to have such a mechanism. All of the data structures we will see here can be reasonably built within a conventional programming language.

Data object is a term referring to a set of elements, say D. For example the data object integers refers to D = {0, [image] 1, [image] 2, ...}. The data object alphabetic character strings of length less than thirty one implies D = {",'A','B', ...,'Z','AA', ...}. Thus, D may be finite or infinite and if D is very large we may need to devise special ways of representing its elements in our computer.

The notion of a data structure as distinguished from a data object is that we want to describe not only the set of objects, but the way they are related. Saying this another way, we want to describe the set of operations which may legally be applied to elements of the data object. This implies that we must specify the set of operations and show how they work. For integers we would have the arithmetic operations +, -, *, / and perhaps many others such as mod, ceil, floor, greater than, less than, etc. The data object integers plus a description of how +, -, *, /, etc. behave constitutes a data structure definition.

To be more precise lets examine a modest example. Suppose we want to define the data structure natural number (abbreviated natno) where natno = {0,1,2,3, ...} with the three operations being a test for zero addition and equality. The following notation can be used:

structure NATNO

1       declare ZERO( )    natno

2               ISZERO(natno)    boolean

3               SUCC(natno)    natno

4               ADD(natno, natno)    natno

5               EQ(natno, natno)    boolean

6       for all x, y    natno let

7               ISZERO(ZERO) ::= true; ISZERO(SUCC(x)) ::= false

8               ADD(ZERO, y) :: = y, ADD(SUCC(x), y) :: =

SUCC(ADD(x, y))

9               EQ(x, ZERO) :: = if ISZERO(x) then true else false

10               EQ(ZERO, SUCC(y)) :: = false

EQ(SUCC(x), SUCC(y)) :: = EQ(x, y)

11       end

end NATNO

In the declare statement five functions are defined by giving their names, inputs and outputs. ZERO is a constant function which means it takes no input arguments and its result is the natural number zero, written as ZERO. ISZERO is a boolean function whose result is either true or false. SUCC stands for successor. Using ZERO and SUCC we can define all of the natural numbers as: ZERO, l = SUCC(ZERO), 2 = SUCC(SUCC(ZERO)), 3 = SUCC(SUCC(SUCC(ZERO))), ... etc. The rules on line 8 tell us exactly how the addition operation works. For example if we wanted to add two and three we would get the following sequence of expressions:

ADD(SUCC(SUCC(ZERO)),SUCC(SUCC(SUCC(ZERO))))

which, by line 8 equals

SUCC(ADD(SUCC(ZERO),SUCC(SUCC(SUCC(ZERO)))))

which, by line 8 equals

SUCC(SUCC(ADD(ZERO,SUCC(SUCC(SUCC(ZERO))))))

which by line 8 equals

SUCC(SUCC(SUCC(SUCC(SUCC(ZERO)))))

Of course, this is not the way to implement addition. In practice we use bit strings which is a data structure that is usually provided on our computers. But however the ADD operation is implemented, it must obey these rules. Hopefully, this motivates the following definition.

Definition: A data structure is a set of domains [image], a designated domain [image], a set of functions [image] and a set of axioms [image]. The triple [image] denotes the data structure d and it will usually be abbreviated by writing d.

In the previous example

The set of axioms describes the semantics of the operations. The form in which we choose to write the axioms is important. Our goal here is to write the axioms in a representation independent way. Then, we discuss ways of implementing the functions using a conventional programming language.

An implementation of a data structure d is a mapping from d to a set of other data structures e. This mapping specifies how every object of d is to be represented by the objects of e. Secondly, it requires that every function of d must be written using the functions of the implementing data structures e. Thus we say that integers are represented by bit strings, boolean is represented by zero and one, an array is represented by a set of consecutive words in memory.

In current parlance the triple [image] is referred to as an abstract data type. It is called abstract precisely because the axioms do not imply a form of representation. Another way of viewing the implementation of a data structure is that it is the process of refining an abstract data type until all of the operations are expressible in terms of directly executable functions. But at the first stage a data structure should be designed so that we know what it does, but not necessarily how it will do it. This division of tasks, called specification and implementation, is useful because it helps to control the complexity of the entire process.

1.2 SPARKS

The choice of an algorithm description language must be carefully made because it plays such an important role throughout the book. We might begin by considering using some existing language; some names which come immediately to mind are ALGOL, ALGOL-W, APL, COBOL, FORTRAN, LISP, PASCAL, PL/I, SNOBOL.

Though some of these are more preferable than others, the choice of a specific language leaves us with many difficulties. First of all, we wish to be able to write our algorithms without dwelling on the idiosyncracies of a given language. Secondly, some languages have already provided the mechanisms we wish to discuss. Thus we would have to make pretense to build up a capability which already exists. Finally, each language has its followers and its detractors. We would rather not have any individual rule us out simply because he did not know or, more particularly, disliked to use the language X.

Furthermore it is not really necessary to write programs in a language for which a compiler exists. Instead we choose to use a language which is tailored to describing the algorithms we want to write. Using it we will not have to define many aspects of a language that we will never use here. Most importantly, the language we use will be close enough to many of the languages mentioned before so that a hand translation will be relatively easy to accomplish. Moreover, one can easily program a translator using some existing, but more primitive higher level language as the output (see Appendix A). We call our language SPARKS. Figure 1.2 shows how a SPARKS program could be executed on any machine.

Figure 1.2: Translation of SPARKS

Many language designers choose a name which is an acronym. But SPARKS was not devised in that way; it just appeared one day as Athena sprang from the head of Zeus. Nevertheless, computerniks still try to attach a meaning. Several cute ideas have been suggested, such as

Structured Programming: A Reasonably Komplete Set

or

Smart Programmers Are Required To Know SPARKS.

SPARKS contains facilities to manipulate numbers, boolean values and characters. The way to assign values is by the assignment statement

variable [image] expression.

In addition to the assignment statement, SPARKS includes statements for conditional testing, iteration, input-output, etc. Several such statements can be combined on a single line if they are separated by a semi-colon. Expressions can be either arithmetic, boolean or of character type. In the boolean case there can be only one of two values,

true or false.

In order to produce these values, the logical operators

and, or, not

are provided, plus the relational operators

A conditional statement has the form

if cond then S1     if cond then S1

or

else S2

where cond is a boolean expression and S1, S2 are arbitrary groups of SPARKS statements. If S1 or S2 contains more than one statement, these will be enclosed in square brackets. Brackets must be used to show how each else corresponds to one if. The meaning of this statement is given by the flow charts:

We will assume that conditional expressions are evaluated in "short circuit" mode; given the boolean expression (cond1 or cond2), if condl is true then cond2 is not evaluated; or, given (condl and cond2), if cond1 is false then cond2 is not evaluated.

To accomplish iteration, several statements are available. One of them is

while cond do

S

end

where cond is as before, S is as S1 before and the meaning is given by

It is well known that all "proper" programs can be written using only the assignment, conditional and while statements. This result was obtained by Bohm and Jacopini. Though this is very interesting from a theoretical viewpoint, we should not take it to mean that this is the way to program. On the contrary, the more expressive our languages are, the more we can accomplish easily. So we will provide other statements such as a second iteration statement, the repeat-until,

repeat

S

until cond

which has the meaning

In contrast to the while statement, the repeat-until guarantees that the statements of S will be executed at least once. Another iteration statement is

loop

S

forever

which has the meaning

As it stands, this describes an infinite loop! However, we assume that this statement is used in conjunction with some test within S which will cause an exit. One way of exiting such a loop is by using a

go to label

statement which transfers control to "label." Label may be anywhere in the procedure. A more restricted form of the go to is the command

exit

which will cause a transfer of control to the first statement after the innermost loop which contains it. This looping statement may be a while, repeat, for or a loop-forever. exit can be used either conditionally or unconditionally, for instance

loop

S1

if cond then exit

S2

forever

which will execute as

The last statement for iteration is called the for-loop, which has the form

for vble    start to finish by increment do

S

end

Vble is a variable, while start, finish and increment are arithmetic expressions. A variable or a constant is a simple form of an expression. The clause "by increment" is optional and taken as +1 if it does not occur. We can write the meaning of this statement in SPARKS as

vble    start

fin    finish

incr    increment

while (vble - fin) * incr   0 do

S

vble  vble + incr

end

Another statement within SPARKS is the case, which allows one to distinguish easily between several alternatives without using multiple if-then-else statements. It has the form

where the Si, 1 [image] i [image] n + 1 are groups of SPARKS statements. The semantics is easily described by the following flowchart:

The else clause is optional.

A complete SPARKS procedure has the form

procedure NAME (parameter list)

S

end

A procedure can be used as a function by using the statement

return (expr)

where the value of expr is delivered as the value of the procedure. The expr may be omitted in which case a return is made to the calling procedure. The execution of an end at the end of procedure implies a return. A procedure may be invoked by using a call statement

call NAME (parameter list)

Procedures may call themselves, direct recursion, or there may be a sequence resulting in indirect recursion. Though recursion often carries with it a severe penalty at execution time, it remains all elegant way to describe many computing processes. This penalty will not deter us from using recursion. Many such programs are easily translatable so that the recursion is removed and efficiency achieved.

A complete SPARKS program is a collection of one or more procedures, the first one taken as the main program. All procedures are treated as external, which means that the only means for communication between them is via parameters. This may be somewhat restrictive in practice, but for the purpose of exposition it helps to list all variables explicitly, as either local or parameter. The association of actual to formal parameters will be handled using the call by reference rule. This means that at run time the address of each parameter is passed to the called procedure. Parameters which are constants or values of expressions are stored into internally generated words whose addresses are then passed to the procedure.

For input/output we assume the existence of two functions

read (argument list), print (argument list)

Arguments may be variables or quoted strings. We avoid the problem of defining a "format" statement as we will need only the simplest form of input and output.

The command stop halts execution of the currently executing procedure. Comments may appear anywhere on a line enclosed by double slashes, e.g.

//this is a comment//

Finally, we note that multi-dimensional arrays are available with arbitrary integer lower and upper bounds. An n-dimensional array A with lower and upper bounds li, ui, 1 [image] i [image] n may be declared by using the syntax declare A(l1:u1, ...,ln:un). We have avoided introducing the record or structure concept. These are often useful features and when available they should be used. However, we will persist in building up a structure from the more elementary array concept. Finally, we emphasize that all of our variables are assumed to be of type INTEGER unless stated otherwise.

Since most of the SPARKS programs will be read many more times than they will be executed, we have tried to make the code readable. This is a goal which should be aimed at by everyone who writes programs. The SPARKS language is rich enough so that one can create a good looking program by applying some simple rules of style.

(i) Every procedure should carefully specify its input and output variables.

(ii) The meaning of variables should be defined.

(iii) The flow of the program should generally be forward except for normal looping or unavoidable instances.

(iv) Indentation rules should be established and followed so that computational units of program text can more easily be identified.

(v) Documentation should be short, but meaningful. Avoid sentences like ''i is increased by one."

(vi) Use subroutines where appropriate.

See the book The Elements of Programming Style by Kernighan and Plauger for more examples of good rules of programming.

1.3 HOW TO CREATE PROGRAMS

Now that you have moved beyond the first course in computer science, you should be capable of developing your programs using something better than the seat-of-the-pants method. This method uses the philosophy: write something down and then try to get it working. Surprisingly, this method is in wide use today, with the result that an average programmer on an average job turns out only between five to ten lines of correct code per day. We hope your productivity will be greater. But to improve requires that you apply some discipline to the process of creating programs. To understand this process better, we consider it as broken up into five phases: requirements, design, analysis, coding, and verification.

(i) Requirements. Make sure you understand the information you are given (the input) and what results you are to produce (the output). Try to write down a rigorous description of the input and output which covers all cases.

You are now ready to proceed to the design phase. Designing an algorithm is a task which can be done independently of the programming language you eventually plan to use. In fact, this is desirable because it means you can postpone questions concerning how to represent your data and what a particular statement looks like and concentrate on the order of processing.

(ii) Design. You may have several data objects (such as a maze, a polynomial, or a list of names). For each object there will be some basic operations to perform on it (such as print the maze, add two polynomials, or find a name in the list). Assume that these operations already exist in the form of procedures and write an algorithm which solves the problem according to the requirements. Use a notation which is natural to the way you wish to describe the order of processing.

(iii) Analysis. Can you think of another algorithm? If so, write it down. Next, try to compare these two methods. It may already be possible to tell if one will be more desirable than the other. If you can't distinguish between the two, choose one to work on for now and we will return to the second version later.

(iv) Refinement and coding. You must now choose representations for your data objects (a maze as a two dimensional array of zeros and ones, a polynomial as a one dimensional array of degree and coefficients, a list of names possibly as an array) and write algorithms for each of the operations on these objects. The order in which you do this may be crucial, because once you choose a representation, the resulting algorithms may be inefficient. Modern pedagogy suggests that all processing which is independent of the data representation be written out first. By postponing the choice of how the data is stored we can try to isolate what operations depend upon the choice of data representation. You should consider alternatives, note them down and review them later. Finally you produce a complete version of your first program.

It is often at this point that one realizes that a much better program could have been built. Perhaps you should have chosen the second design alternative or perhaps you have spoken to a friend who has done it better. This happens to industrial programmers as well. If you have been careful about keeping track of your previous work it may not be too difficult to make changes. One of the criteria of a good design is that it can absorb changes relatively easily. It is usually hard to decide whether to sacrifice this first attempt and begin again or just continue to get the first version working. Different situations call for different decisions, but we suggest you eliminate the idea of working on both at the same time. If you do decide to scrap your work and begin again, you can take comfort in the fact that it will probably be easier the second time. In fact you may save as much debugging time later on by doing a new version now. This is a phenomenon which has been observed in practice.

The graph in figure 1.3 shows the time it took for the same group to build 3 FORTRAN compilers (A, B and C). For each compiler there is the time they estimated it would take them and the time it actually took. For each subsequent compiler their estimates became closer to the truth, but in every case they underestimated. Unwarrented optimism is a familiar disease in computing. But prior experience is definitely helpful and the time to build the third compiler was less than one fifth that for the first one.

Figure 1.3: History of three FORTRAN compilers

(v) Verification. Verification consists of three distinct aspects: program proving, testing and debugging. Each of these is an art in itself. Before executing your program you should attempt to prove it is correct. Proofs about programs are really no different from any other kinds of proofs, only the subject matter is different. If a correct proof can be obtained, then one is assured that for all possible combinations of inputs, the program and its specification agree. Testing is the art of creating sample data upon which to run your program. If the program fails to respond correctly then debugging is needed to determine what went wrong and how to correct it. One proof tells us more than any finite amount of testing, but proofs can be hard to obtain. Many times during the proving process errors are discovered in the code. The proof can't be completed until these are changed. This is another use of program proving, namely as a methodology for discovering errors. Finally there may be tools available at your computing center to aid in the testing process. One such tool instruments your source code and then tells you for every data set: (i) the number of times a statement was executed, (ii) the number of times a branch was taken, (iii) the smallest and largest values of all variables. As a minimal requirement, the test data you construct should force every statement to execute and every condition to assume the value true and false at least once.

One thing you have forgotten to do is to document. But why bother to document until the program is entirely finished and correct ? Because for each procedure you made some assumptions about its input and output. If you have written more than a few procedures, then you have already begun to forget what those assumptions were. If you note them down with the code, the problem of getting the procedures to work together will be easier to solve. The larger the software, the more crucial is the need for documentation.

The previous discussion applies to the construction of a single procedure as well as to the writing of a large software system. Let us concentrate for a while on the question of developing a single procedure which solves a specific task. This shifts our emphasis away from the management and integration of the various procedures to the disciplined formulation of a single, reasonably small and well-defined task. The design process consists essentially of taking a proposed solution and successively refining it until an executable program is achieved. The initial solution may be expressed in English or some form of mathematical notation. At this level the formulation is said to be abstract because it contains no details regarding how the objects will be represented and manipulated in a computer. If possible the designer attempts to partition the solution into logical subtasks. Each subtask is similarly decomposed until all tasks are expressed within a programming language. This method of design is called the top-down approach. Inversely, the designer might choose to solve different parts of the problem directly in his programming language and then combine these pieces into a complete program. This is referred to as the bottom-up approach. Experience suggests that the top-down approach should be followed when creating a program. However, in practice it is not necessary to unswervingly follow the method. A look ahead to problems which may arise later is often useful.

Underlying all of these strategies is the assumption that a language exists for adequately describing the processing of data at several abstract levels. For this purpose we use the language SPARKS coupled with carefully chosen English narrative. Such an algorithm might be called pseudo-SPARKS. Let us examine two examples of top-down program development.

Suppose we devise a program for sorting a set of n [image] 1 distinct integers. One of the simplest solutions is given by the following

"from those integers which remain unsorted, find the smallest and place it next in the sorted list"

This statement is sufficient to construct a sorting program. However, several issues are not fully specified such as where and how the integers are initially stored and where the result is to be placed. One solution is to store the values in an array in such a way that the i-th integer is stored in the i-th array position, A(i) 1 [image] i [image] n. We are now ready to give a second refinement of the solution:

for i    1 to n do

examine A(i) to A(n) and suppose the

smallest integer is at A(j); then

interchange A(i) and A(j).

end

Note how we have begun to use SPARKS pseudo-code. There now remain two clearly defined subtasks: (i) to find the minimum integer and (ii) to interchange it with A(i). This latter problem can be solved by the code

t [image] A(i); A(i) [image] A(j); A(j) [image] t

The first subtask can be solved by assuming the minimum is A (i), checking A(i) with A(i + 1), A(i + 2), ... and whenever a smaller element is found, regarding it as the new minimum. Eventually A(n) is compared to the current minimum and we are done. Putting all these observations together we get

procedure SORT(A,n)

1   for i    1 to n do

2       j    i

3       for k    j + 1 to n do

4           if A(k) < A(j) then j    k

5       end

6       t    A(i); A(i)    A(j); A(j)    t

7   end

end SORT

The obvious question to ask at this point is: "does this program work correctly?"

Theorem: Procedure SORT (A,n) correctly sorts a set of n [image] 1 distinct integers, the result remains in A (1:n) such that A (1) < A (2) < ... < A(n).

Proof: We first note that for any i, say i = q, following the execution of lines 2 thru 6, it is the case that A(q) [image] A(r), q < r [image] n. Also, observe that when i becomes greater than q, A(1 .. q) is unchanged. Hence, following the last execution of these lines, (i.e., i = n), we have A(1) [image] A(2) [image] ... A(n).

We observe at this point that the upper limit of the for-loop in line 1 can be changed to n - 1 without damaging the correctness of the algorithm.

From the standpoint of readability we can ask if this program is good. Is there a more concise way of describing this algorithm which will still be as easy to comprehend? Substituting while statements for the for loops doesn't significantly change anything. Also, extra initialization and increment statements would be required. We might consider a FORTRAN version using the ANSI language standard

IF (N. LE. 1) GO TO 100

NM1 = N - 1

DO 101 I = 1, NM1

J = I

JP1 = J + 1

DO 102 K = JP1, N

IF (A(K).LT.A(J)) J = K

102    CONTINUE

T = A(I)

A(I) = A(J)

A(J) = T

101 CONTINUE

100 CONTINUE

FORTRAN forces us to clutter up our algorithms with extra statements. The test for N = 1 is necessary because FORTRAN DO-LOOPS always insist on executing once. Variables NM1 and JP1 are needed because of the restrictions on lower and upper limits of DO-LOOPS.

Let us develop another program. We assume that we have n [image] 1 distinct integers which are already sorted and stored in the array A(1:n). Our task is to determine if the integer x is present and if so to return j such that x = A(j); otherwise return j = 0. By making use of the fact that the set is sorted we conceive of the following efficient method:

"let A(mid) be the middle element. There are three possibilities. Either x < A(mid) in which case x can only occur as A(1) to A(mid - 1); or x > A(mid) in which case x can only occur as A(mid + l) to A(n): or x = A(mid) in which case set j to mid and return. Continue in this way by keeping two pointers, lower and upper, to indicate the range of elements not yet tested."

At this point you might try the method out on some sample numbers. This method is referred to as binary search. Note how at each stage the number of elements in the remaining set is decreased by about one half. We can now attempt a version using SPARKS pseudo code.

procedure BINSRCH(A,n,x,j)

initialize lower and upper

while there are more elements to check do

let A(mid) be the middle element

case

: x > A(mid): set lower to mid + 1

: x < A(mid): set upper to mid - 1

: else: found

end

end

not found

end BINSRCH

The above is not the only way we might write this program. For instance we could replace the while loop by a repeat-until statement with the same English condition. In fact there are at least six different binary search programs that can be produced which are all correct. There are many more that we might produce which would be incorrect. Part of the freedom comes from the initialization step. Whichever version we choose, we must be sure we understand the relationships between the variables. Below is one complete version.

procedure BINSRCH (A,n,x,j)

 1    lower    1; upper    n

 2    while lower    upper do

 3      mid      (lower + upper) / 2

 4      case

 5      : x > A(mid): lower    mid + 1

 6      : x < A(mid): upper    mid - 1

 7      : else: j    mid; return

 8      end

 9    end

10    j    0

end

To prove this program correct we make assertions about the relationship between variables before and after the while loop of steps 2-9. As we enter this loop and as long as x is not found the following holds:

lower [image] upper and A (lower) [image] x [image] A (upper) and SORTED (A, n)

Now, if control passes out of the while loop past line 9 then we know the condition of line 2 is false

lower > upper.

This, combined with the above assertion implies that x is not present.

Unfortunately a complete proof takes us beyond our scope but for those who wish to pursue program proving they should consult our references at the end of this chapter. An analysis of the computing time for BINSRCH is carried out in section 7.1.

Recursion

We have tried to emphasize the need to structure a program to make it easier to achieve the goals of readability and correctness. Actually one of the most useful syntactical features for accomplishing this is the procedure. Given a set of instructions which perform a logical operation, perhaps a very complex and long operation, they can be grouped together as a procedure. The procedure name and its parameters are viewed as a new instruction which can be used in other programs. Given the input-output specifications of a procedure, we don't even have to know how the task is accomplished, only that it is available. This view of the procedure implies that it is invoked, executed and returns control to the appropriate place in the calling procedure. What this fails to stress is the fact that procedures may call themselves (direct recursion) before they are done or they may call other procedures which again invoke the calling procedure (indirect recursion). These recursive mechanisms are extremely powerful, but even more importantly, many times they can express an otherwise complex process very clearly. For these reasons we introduce recursion here.

Most students of computer science view recursion as a somewhat mystical technique which only is useful for some very special class of problems (such as computing factorials or Ackermann's function). This is unfortunate because any program that can be written using assignment, the if-then-else statement and the while statement can also be written using assignment, if-then-else and recursion. Of course, this does not say that the resulting program will necessarily be easier to understand. However, there are many instances when this will be the case. When is recursion an appropriate mechanism for algorithm exposition? One instance is when the problem itself is recursively defined. Factorial fits this category, also binomial coefficients where

can be recursively computed by the formula

Another example is reversing a character string, S = 'x1 ... xn' where SUBSTRING (S,i,j) is a function which returns the string xi ... xj for appropriately defined i and j and S [image] [image] T stands for concatenation of two strings (as in PL/I). Then the operation REVERSE is easily described recursively as

procedure REVERSE(S)

n    LENGTH(S)

if n = 1 then return (S)

else return (REVERSE(SUBSTRING(S,2,n))

     SUBSTRING(S,1,1))

end REVERSE

If this looks too simple let us develop a more complex recursive procedure. Given a set of n [image] 1 elements the problem is to print all possible permutations of this set. For example if the set is {a,b,c}, then the set of permutations is {(a, b,c), (a,c,b), (b,a,c), (b,c,a), (c,a,b), (c,b,a)}. It is easy to see that given n elements there are n ! different permutations. A simple algorithm can be achieved by looking at the case of four elements (a,b,c,d). The answer is obtained by printing

(i) a followed by all permutations of (b,c,d)

(ii) b followed by all permutations of (a,c,d)

(iii) c followed by all permutations of (b,a,d)

(iv) d followed by all permutations of (b,c,a)

The expression "followed by all permutations" is the clue to recursion. It implies that we can solve the problem for a set with n elements if we had an algorithm which worked on n - 1 elements. These considerations lead to the following procedure which is invoked by call PERM(A,1,n). A is a character string e.g. A ='abcd', and INTERCHANGE (A,k,i) exchanges the k-th character of A with the i-th character of A.

procedure PERM(A,k,n)

if k = n then [print (A); return]

B    A

for i    k to n do

call INTERCHANGE(A,k,i)

call PERM(A,k + 1,n)

A    B

end

end PERM

Try this algorithm out on sets of length one, two, and three to insure that you understand how it works. Then try to do one or more of the exercises at the end of this chapter which ask for recursive procedures.

Another time when recursion is useful is when the data structure that the algorithm is to operate on is recursively defined. We will see several important examples of such structures, especially lists in section 4.9 and binary trees in section 5.4. Another instance when recursion is invaluable is when we want to describe a backtracking procedure. But for now we will content ourselves with examining some simple, iterative programs and show how to eliminate the iteration statements and replace them by recursion. This may sound strange, but the objective is not to show that the result is simpler to understand nor more efficient to execute. The main purpose is to make one more familiar with the execution of a recursive procedure.

Suppose we start with the sorting algorithm presented in this section. To rewrite it recursively the first thing we do is to remove the for loops and express the algorithm using assignment, if-then-else and the go-to statement.

procedure SORT(A,n)

i    1

Ll: if i    n - 1         // for i    1 to n - 1 do//

then [j    i; k    j + 1

L2: if k    n      //for k    j + 1 to n do//

then [if A(k) < A(j)

then j    k

k    k + 1; go to L2]

t    A(i); A(i)    A(j); A(j)    t

i    i + 1; go to L1]

end SORT

Now every place where we have a label we introduce a procedure whose parameters are the variables which are already assigned a value at that point. Every place where a ''go to label'' appears, we replace that statement by a call of the procedure associated with that label. This gives us the following set of three procedures.

procedure SORT(A,n)

call SORTL1(A,n,1)

end SORT

procedure SORTLl(A,n,i)

if i    n - 1

then [j    i; call MAXL2(A,n,j,i + 1)

t    A(i); A(i)    A(j); A(j)    t

call SORTL1(A,n,i + 1)]

end SORTL1

procedure MAXL2(A,n,j,k)

if k    n

then [if A(k) < A(j) then j    k

call MAXL2(A,n,j,k + 1)]

end MAXL2

We can simplify these procedures somewhat by ignoring SORT(A,n) entirely and begin the sorting operation by call SORTL1(A,n,1). Notice how SORTL1 is directly recursive while it also uses procedure MAXL2. Procedure MAXL2 is also directly reculsive. These two procedures use eleven lines while the original iterative version was expressed in nine lines; not much of a difference. Notice how in MAXL2 the fourth parameter k is being changed. The effect of increasing k by one and restarting the procedure has essentially the same effect as the for loop.

Now let us trace the action of these procedures as they sort a set of five integers

When a procedure is invoked an implicit branch to its beginning is made. Thus a recursive call of a program can be made to simulate a go to statement. The parameter mechanism of the procedure is a form of assignment. Thus placing the argument k + 1 as the fourth parameter of MAXL2 is equivalent to the statement k   [image]  k + 1.

In section 4.9 we will see the first example of a recursive data structure, the list. Also in that section are several recursive procedures, followed in some cases by their iterative equivalents. Rules are also given there for eliminating recursion.

1.4 HOW TO ANALYZE PROGRAMS

One goal of this book is to develop skills for making evaluative judgements about programs. There are many criteria upon which we can judge a program, for instance:

(i)     Does it do what we want it to do?

(ii)     Does it work correctly according to the original specifications of the task?

(iii)     Is there documentation which describes how to use it and how it works?

(iv)     Are subroutines created in such a way that they perform logical sub-functions?

(v)     Is the code readable?

The above criteria are all vitally important when it comes to writing software, most especially for large systems. Though we will not be discussing how to reach these goals, we will try to achieve them throughout this book with the programs we write. Hopefully this more subtle approach will gradually infect your own program writing habits so that you will automatically strive to achieve these goals.

There are other criteria for judging programs which have a more direct relationship to performance. These have to do with computing time and storage requirements of the algorithms. Performance evaluation can be loosely divided into 2 major phases: (a) a priori estimates and (b) a posteriori testing. Both of these are equally important.

First consider a priori estimation. Suppose that somewhere in one of your programs is the statement

x   [image]  x + 1.

We would like to determine two numbers for this statement. The first is the amount of time a single execution will take; the second is the number of times it is executed. The product of these numbers will be the total time taken by this statement. The second statistic is called the frequency count, and this may vary from data set to data set. One of the hardest tasks in estimating frequency counts is to choose adequate samples of data. It is impossible to determine exactly how much time it takes to execute any command unless we have the following information:

(i)     the machine we are executing on:

(ii)     its machine language instruction set;

(iii)     the time required by each machine instruction;

(iv)     the translation a compiler will make from the source to the machine language.

It is possible to determine these figures by choosing a real machine and an existing compiler. Another approach would be to define a hypothetical machine (with imaginary execution times), but make the times reasonably close to those of existing hardware so that resulting figures would be representative. Neither of these alternatives seems attractive. In both cases the exact times we would determine would not apply to many machines or to any machine. Also, there would be the problem of the compiler, which could vary from machine to machine. Moreover, it is often difficult to get reliable timing figures because of clock limitations and a multi-programming or time sharing environment. Finally, the difficulty of learning another machine language outweighs the advantage of finding "exact" fictitious times. All these considerations lead us to limit our goals for an a priori analysis. Instead, we will concentrate on developing only the frequency count for all statements. The anomalies of machine configuration and language will be lumped together when we do our experimental studies. Parallelism will not be considered.

Consider the three examples of Figure 1.4 below.

   .                                       for i    1 to n do

   .            for i    1 to n do

   .                                         for j    1 to n do

x    x + l          x    x + 1

   .                                             x    x + 1

   .           end

   .                                         end

                                           end

(a)                    (b)                          (c)

Figure 1.4: Three simple programs for frequency counting.

In program (a) we assume that the statement x   [image]  x + 1 is not contained within any loop either explicit or implicit. Then its frequency count is one. In program (b) the same statement will be executed n times and in program (c) n2 times (assuming n   [image]  1). Now 1, n, and n2 are said to be different and increasing orders of magnitude just like 1, 10, 100 would be if we let n = 10. In our analysis of execution we will be concerned chiefly with determining the order of magnitude of an algorithm. This means determining those statements which may have the greatest frequency count.

To determine the order of magnitude, formulas such as

often occur. In the program segment of figure 1.4(c) the statement x   [image]  x + 1 is executed

Simple forms for the above three formulas are well known, namely,

In general

To clarify some of these ideas, let us look at a simple program for computing the n-th Fibonacci number. The Fibonacci sequence starts as

0, 1, 1, 2, 3, 5, 8, 13, 21, 34, 55, ...

Each new term is obtained by taking the sum of the two previous terms. If we call the first term of the sequence F0 then F0 = 0, F1 = 1 and in general

Fn = Fn-1 + Fn-2, n  [image]  2.

The program on the following page takes any non-negative integer n and prints the value Fn.

1     procedure FIBONACCI

2       read (n)

3-4     if n < 0 then [print ('error'); stop]

5-6     if n = 0 then [print ('0'); stop]

7-8     if n = 1 then [print ('1'); stop]

9       fnm2    0; fnm 1    1

10       for i    2 to n do

11          fn    fnm1 + fnm2

12          fnm2    fnm1

13          fnm1    fn

14       end

15       print (fn)

16     end FIBONACCI

The first problem in beginning an analysis is to determine some reasonable values of n. A complete set would include four cases: n < 0, n = 0, n = 1 and n > 1. Below is a table which summarizes the frequency counts for the first three cases.

Step   n < 0  n = 0  n = 1

--------------------------

  1      1      1      1

  2      1      1      1

  3      1      1      1

  4      1      0      0

  5      0      1      1

  6      0      1      0

  7      0      0      1

  8      0      0      1

 9-15    0      0      0

These three cases are not very interesting. None of them exercises the program very much. Notice, though, how each if statement has two parts: the if condition and the then clause. These may have different execution counts. The most interesting case for analysis comes when n > 1. At this point the for loop will actually be entered. Steps 1, 2, 3, 5, 7 and 9 will be executed once, but steps 4, 6 and 8 not at all. Both commands in step 9 are executed once. Now, for n   [image]  2 how often is step 10 executed: not n - 1 but n times. Though 2 to n is only n - 1 executions, remember that there will be a last return to step 10 where i is incremented to n + 1, the test i > n made and the branch taken to step 15. Thus, steps 11, 12, 13 and 14 will be executed n - 1 times but step 10 will be done n times. We can summarize all of this with a table.

Step  Frequency  Step  Frequency

--------------------------------

 1        1        9       2

 2        1       10       n

 3        1       11      n-1

 4        0       12      n-1

 5        1       13      n-1

 6        0       14      n-1

 7        1       15       1

 8        0       16       1

Figure 1.5: Execution Count for Computing Fn

Each statement is counted once, so step 9 has 2 statements and is executed once for a total of 2. Clearly, the actual time taken by each statement will vary. The for statement is really a combination of several statements, but we will count it as one. The total count then is 5n + 5. We will often write this as O(n), ignoring the two constants 5. This notation means that the order of magnitude is proportional to n.

The notation f(n) = O(g(n)) (read as f of n equals big-oh of g of n) has a precise mathematical definition.

Definition: f(n) = O(g(n)) iff there exist two constants c and no such that |f(n)|   [image]  c|g(n)| for all n   [image]  no.

f(n) will normally represent the computing time of some algorithm. When we say that the computing time of an algorithm is O(g(n)) we mean that its execution takes no more than a constant times g(n). n is a parameter which characterizes the inputs and/or outputs. For example n might be the number of inputs or the number of outputs or their sum or the magnitude of one of them. For the Fibonacci program n represents the magnitude of the input and the time for this program is written as T(FIBONACCI) = O(n).

We write O(1) to mean a computing time which is a constant. O(n) is called linear, O(n2) is called quadratic, O(n3) is called cubic, and O(2n) is called exponential. If an algorithm takes time O(log n) it is faster, for sufficiently large n, than if it had taken O(n). Similarly, O(n log n) is better than O(n2) but not as good as O(n). These seven computing times, O(1), O(log n), O(n), O(n log n), O(n2), O(n3), and O(2n) are the ones we will see most often throughout the book.

If we have two algorithms which perform the same task, and the first has a computing time which is O(n) and the second O(n2), then we will usually take the first as superior. The reason for this is that as n increases the time for the second algorithm will get far worse than the time for the first. For example, if the constant for algorithms one and two are 10 and 1/2 respectively, then we get the following table of computing times:

 n  10n    n2/2

-----------------

 1   10     1/2

 5   50   12-1/2

10  100     50

15  150  112-1/2

20  200    200

25  250  312-1/2

30  300    450

For n   [image]  20, algorithm two had a smaller computing time but once past that point algorithm one became better. This shows why we choose the algorithm with the smaller order of magnitude, but we emphasize that this is not the whole story. For small data sets, the respective constants must be carefully determined. In practice these constants depend on many factors, such as the language and the machine one is using. Thus, we will usually postpone the establishment of the constant until after the program has been written. Then a performance profile can be gathered using real time calculation.

Figures 1.6 and 1.7 show how the computing times (counts) grow with a constant equal to one. Notice how the times O(n) and O(n log n) grow much more slowly than the others. For large data sets, algorithms with a complexity greater than O(n log n) are often impractical. An algorithm which is exponential will work only for very small inputs. For exponential algorithms, even if we improve the constant, say by 1/2 or 1/3, we will not improve the amount of data we can handle by very much.

Given an algorithm, we analyze the frequency count of each statement and total the sum. This may give a polynomial

P(n) = cknk + ck-1 nk-1 + ... + c1n + co

where the ci are constants, ck   [image]  0 and n is a parameter. Using big-oh notation, P(n) = O(nk). On the other hand, if any step is executed 2n times or more the expression

c2n + P(n) = O(2n).

Another valid performance measure of an algorithm is the space it requires. Often one can trade space for time, getting a faster algorithm but using more space. We will see cases of this in subsequent chapters.

Figure 1.6: Rate of Growth of Common Computing Time Functions

log2n  n   nlog2n   n2     n3     2n

--------------------------------------------------------

  0    1    0      1     1      2

  1    2    2      4     8      4

  2    4    8      16    64     16

  3    8    24     64    512    256

  4    16   64     256   4096   65536

  5    32   160    1024  32768  2, 147, 483, 648

Figure 1.7: Values for Computing Functions

We end this chapter with a problem from recreational mathematics which uses many of the SPARKS features that have been discussed. A magic square is an n x n matrix of the integers 1 to n2 such that the sum of every row, column and diagonal is the same. For example, if n = 5 we have

15   8   1  24  17

16  14   7   5  23

22  20  13   6   4

 3  21  19  12  10

 9   2  25  18  11

where the common sum is 65. When n is odd H. Coxeter has given a simple rule for generating a magic square:

"Start with 1 in the middle of the top row; then go up and left assigning numbers in increasing order to empty squares; if you fall off the square imagine the same square as tiling the plane and continue; if a square is occupied, move down instead and continue."

The magic square above was formed using this rule. We now write a SPARKS program for creating an n X n magic square for n odd.

procedure MAGIC(square, n)

//for n odd create a magic square which is declared as an array//

//square (0: n - 1, 0: n - 1)//

//(i,j) is a square position. 2    key    n2 is integer valued.//

if n is even then [print ('input error'); stop]

SQUARE    0

square (0,(n - 1)/2)    1;         //store 1 in middle of first row//

key    2; i    0; j    (n - 1)/2        //i,j are current position//

while key    n2 do

(k,l)    ((i - 1) mod n, (j - 1) mod n)         //look up and left//

if square (k,l)    0

then i    (i + 1) mod n        //square occupied, move down//

else (i,j)    (k,l)    //square (k,l) needs to be assigned//

square (i,j)    key      //assign it a value//

key    key + 1

end

print (n, square)     //output result//

end MAGIC

MAGIC is a complete SPARKS procedure. The statement (i,j)   [image]  (k,l) is a shorthand way of writing i   [image]  k; j   [image]  l. It emphasizes that the variables are thought of as pairs and are changed as a unit. The reserved word mod computes the nonnegative remainder and is a built in function. The magic square is represented using a two dimensional array having n rows and n column. For this application it is convenient to number the rows (and columns) from zero to n - 1 rather than from one to n. Thus, when the program ''falls off the square'' the mod operator sets i and/or j back to zero or n - 1.

The while loop is governed by the variable key which is an integer variable initialized to 2 and increased by one each time through the loop. Thus each statement within the while loop will be executed no more than n2 - 1 times and hence the computing time for MAGIC is O(n2). Since there are n2 positions in which the algorithm must place a number, we see that O(n2) is the best bound an algorithm could have.

REFERENCES

For a discussion of algorithms and how to analyze them see

The Art of Computer Programming: Fundamental Algorithms, by D. E. Knuth, vol. 1, chapter 1, 2-nd edition, Addison-Wesley, 1973.

For a discussion of good programming techniques see

Structured Programming by O. J. Dahl, E. W. Dijkstra, and C. A. R. Hoare, Academic Press, 1972.

The Elements of Programming Style by B. W. Kernighan and P. J. Plauger, McGraw-Hill, 1974.

ACM Computing Surveys, Special Issue: Programming, vol. 6, no. 4, December, 1974.

For a discussion of tools and procedures for developing very large software systems see

Practical Strategies for Developing Large Software Systems, by E. Horowitz, Addison-Wesley, May, 1975.

For a discussion of the more abstract formulation of data structures see

"Toward an understanding of data structures" by J. Earley, CACM, vol. 14, no. 10, October, 1971, pp. 617-627.

"Another look at data," by G. Mealy, Proc. AFIPS Fall Joint Computer Conference, vol. 31, 1967, pp. 525-534.

For a further discussion of program proving see

"Assigning meaning to programs," by R. W. Floyd, Proc. of a Symposium in Applied Mathematics, vol. 19, J. T. Schwartz, ed., American Mathematical Society, Providence, 1967, pp. 19-32.

"An interactive program verification system," by D. I. Good, R. L. London, W. W. Bledsoe, IEEE Transactions on Software Engineering, SE-1, vol. 1, March, 1975, pp. 59-67.

EXERCISES

1.     Look up the word algorithm or its older form algorism in the dictionary.

2.     Consider the two statements: (i) Is n = 2 the largest value of n for which there exists positive integers x, y and z such that xn + yn = zn has a solution; (ii) Store 5 divided by zero into X and go to statement 10. Both do not satisfy one of the five criteria of an algorithm. Which criteria do they violate?

3.     Describe the flowchart in figure 1.1 by using a combination of SPARKS and English. Can you do this without using the go to? Now make it into an algorithm.

4.     Discuss how you would actually represent the list of name and telephone number pairs in a real machine. How would you handle people with the same last name.

5.     Write FORTRAN equivalents of the while, repeat-until, loop-forever and for statements of SPARKS.

6.     Can you think of a clever meaning for S.P.A.R.K.S.? Concentrate on the letter K first.

7.     Determine the frequency counts for all statements in the following two SPARKS program segments:

1 for i    1 to n        1 i    1

2   for j    l to i      2 while i    n do

3     for k    1 to j    3   x   x + 1

4        x    x + 1      4   i    i +1

5     end                5 end

6   end

7 end

     (a)                   (b)

8.     Horner's Rule is a means for evaluating a polynomial A(x) = anxn + an-1xn-1 + ... +a1 x + a0 at a point x0 using a minimum number of multiplications. The rule is:

A(x) = (... ((an.x0 + an-1)x0 + ... + a1)x0 + a0

Write a SPARKS program to evaluate a polynomial using Horner's Rule. Determine how many times each statement is executed.

9.     Given n boolean variables x1,..., xn we wish to print all possible combinations of truth values they can assume. For instance, if n = 2, there are four possibilities: true, true; true, false; false, true; false, false. Write a SPARKS program to accomplish this and do a frequency count.

10.     Compare the two functions n2 and 2n/4 for various values of n. Determine when the second becomes larger than the first.

11.     Write a SPARKS program which prints out the integer values of x, y, z in nondecreasing order. What is the computing time of your method?

12.     Write a SPARKS procedure which searches an array A (1: n) for the element x. If x occurs, then set j to its position in the array else set j to zero. Try writing this without using the go to statement.

13.     One useful facility we might add to SPARKS is the ability to manipulate character strings. If x, y are variables of type character, then we might like to implement the procedures:

(i)     z   [image]  CONCAT(x,y) which concatenates a copy of string y to the end of a copy of string x and assigns the resulting string to z. Strings x and y remain unchanged.

(ii)     z   [image]  SUBSTR(x,i,j) which copies to z the i-th to the j-th character in string x with appropriate definitions for j = 0, i > j, etc. String x is unchanged.

(iii)     z   [image]  INDEX(x,y) which searches string x for the first occurrence of string y and sets z to its starting position in x or else zero.

Implement these procedures using the array facility.

14.     Write a SPARKS procedure which is given an argument STRING, whose value is a character string of length n. Copy STRING into the variable FILE so that every sequence of blanks is reduced to a single blank. The last character of STRING is nonblank.

15.     Design a program that counts the number of occurrences of each character in the string STRING of length n. Represent your answer in the array ANS(1:k,1:2) where ANS(i,l) is the i-th character and ANS(i,2) is the number of times it occurs in STRING.

16.     Trace the action of the procedure below on the elements 2, 4, 6, 8, 10, 12, 14, 16, 18, 20 searching for l, 3, 13 and 21.

i    1; j    n

repeat k    (i + j)/2

if A(k)    x then i    k + 1

else j    k - 1

until i > j

What is the computing time for this segment in terms of n?

17.     Prove by induction:

18.     List as many rules of style in programming that you can think of that you would be willing to follow yourself.

19.     Using the notation introduced at the end of section 1.1, define the structure Boolean with operations AND, OR, NOT, IMP and EQV (equivalent) using only the if-then-else statement. e.g. NOT (X) :: = if X then false else true.

20.     Give a version of a binary search procedure which initializes lower to zero and upper to n + l.

21.     Take any version of binary search, express it using assignment, if-then-else and go to and then give an equivalent recursive program.

22.     Analyze the computing time of procedure SORT as given in section 1.3.

23.     Write a recursive procedure for computing the binomial coefficient   [image]  as defined in section 1.3 where   [image]. Analyze the time and space requirements of your algorithm.

24.     Ackermann's function A(m,n) is defined as follows:

This function is studied because it grows very fast for small values of m and n. Write a recursive procedure for computing this function. Then write a nonrecursive algorithm for computing Ackermann's function.

25.     (Tower of Hanoi) There are three towers and sixty four disks of different diameters placed on the first tower. The disks are in order of decreasing diameter as one scans up the tower. Monks were reputedly supposed to move the disks from tower 1 to tower 3 obeying the rules: (i) only one disk can be moved at any time; (ii) no disk can be placed on top of a disk with smaller diameter. Write a recursive procedure which prints the sequence of moves which accomplish this task.

26.     Write an equivalent recursive version of procedure MAGIC as given in section 1.4.

27.     The pigeon hole principle states that if a function f has n distinct inputs but less than n distinct outputs then there exists two inputs a, b such that a   [image]  b and f(a) = f(b). Give an algorithm which finds the values a, b for which the range values are equal.

28.     Given n, a positive integer determine if n is the sum of all of its divisors; i.e. if n is the sum of all t such that 1   [image]  t < n and t divides n.

29.     Consider the function F(x) defined by

F(x)   [image]  if even(x) then x/2 else F(F(3x + 1))

Prove that F(x) terminates for all integers x. (Hint: consider integers of the form (2i + 1) 2k - 1 and use induction.)

30.     If S is a set of n elements the powerset of S is the set of all possible subsets of S. For example if S = (a,b,c,) then POWERSET(S) = {( ), (a), (b), (c), (a,b), (a,c), (b,c), (a,b,c)}. Write a recursive procedure to compute powerset (S).

Go to Chapter 2     Back to Table of Contents


Fundamentals: CHAPTER 2: ARRAYS

 Return to Table of Contents

CHAPTER 2: ARRAYS

2.1 AXIOMATIZATION

It is appropriate that we begin our study of data structures with the array. The array is often the only means for structuring data which is provided in a programming language. Therefore it deserves a significant amount of attention. If one asks a group of programmers to define an array, the most often quoted saying is: a consecutive set of memory locations. This is unfortunate because it clearly reveals a common point of confusion, namely the distinction between a data structure and its representation. It is true that arrays are almost always implemented by using consecutive memory, but not always. Intuitively, an array is a set of pairs, index and value. For each index which is defined, there is a value associated with that index. In mathematical terms we call this a correspondence or a mapping. However, as computer scientists we want to provide a more functional definition by giving the operations which are permitted on this data structure. For arrays this means we are concerned with only two operations which retrieve and store values. Using our notation this object can be defined as:

structure ARRAY(value, index)

declare CREATE( )    array

RETRIEVE(array,index)    value

STORE(array,index,value)    array;

for all A    array, i,j    index, x    value let

RETRIEVE(CREATE,i) :: = error

RETRIEVE(STORE(A,i,x),j) :: =

if EQUAL(i,j) then x else RETRIEVE(A,j)

end

end ARRAY

The function CREATE produces a new, empty array. RETRIEVE takes as input an array and an index, and either returns the appropriate value or an error. STORE is used to enter new index-value pairs. The second axiom is read as "to retrieve the j-th item where x has already been stored at index i in A is equivalent to checking if i and j are equal and if so, x, or search for the j-th value in the remaining array, A." This axiom was originally given by J. McCarthy. Notice how the axioms are independent of any representation scheme. Also, i and j need not necessarily be integers, but we assume only that an EQUAL function can be devised.

If we restrict the index values to be integers, then assuming a conventional random access memory we can implement STORE and RETRIEVE so that they operate in a constant amount of time. If we interpret the indices to be n-dimensional, (i1,i2, ...,in), then the previous axioms apply immediately and define n-dimensional arrays. In section 2.4 we will examine how to implement RETRIEVE and STORE for multi-dimensional arrays using consecutive memory locations.

2.2 ORDERED LISTS

One of the simplest and most commonly found data object is the ordered or linear list. Examples are the days of the week

(MONDAY, TUESDAY, WEDNESDAY, THURSDAY, FRIDAY,

SATURDAY, SUNDAY)

or the values in a card deck

(2, 3, 4, 5, 6, 7, 8, 9, 10, Jack, Queen, King, Ace)

or the floors of a building

(basement, lobby, mezzanine, first, second, third)

or the years the United States fought in World War II

(1941, 1942, 1943, 1944, 1945).

If we consider an ordered list more abstractly, we say that it is either empty or it can be written as

(a1,a2,a3, ...,an)

where the ai are atoms from some set S.

There are a variety of operations that are performed on these lists. These operations include:

(i) find the length of the list, n;

(ii) read the list from left-to-right (or right-to-left);

(iii) retrieve the i-th element, [image];

(iv) store a new value into the i-th position, [image];

(v) insert a new element at position [image] causing elements numbered i,i + 1, ...,n to become numbered i + 1,i + 2, ...,n + 1;

(vi) delete the element at position [image] causing elements numbered i + 1, ...,n to become numbered i,i + 1, ...,n - 1.

See exercise 24 for a set of axioms which uses these operations to abstractly define an ordered list. It is not always necessary to be able to perform all of these operations; many times a subset will suffice. In the study of data structures we are interested in ways of representing ordered lists so that these operations can be carried out efficiently.

Perhaps the most common way to represent an ordered list is by an array where we associate the list element ai with the array index i. This we will refer to as a sequential mapping, because using the conventional array representation we are storing ai and ai + 1 into consecutive locations i and i + 1 of the array. This gives us the ability to retrieve or modify the values of random elements in the list in a constant amount of time, essentially because a computer memory has random access to any word. We can access the list element values in either direction by changing the subscript values in a controlled way. It is only operations (v) and (vi) which require real effort. Insertion and deletion using sequential allocation forces us to move some of the remaining elements so the sequential mapping is preserved in its proper form. It is precisely this overhead which leads us to consider nonsequential mappings of ordered lists into arrays in Chapter 4.

Let us jump right into a problem requiring ordered lists which we will solve by using one dimensional arrays. This problem has become the classical example for motivating the use of list processing techniques which we will see in later chapters. Therefore, it makes sense to look at the problem and see why arrays offer only a partially adequate solution. The problem calls for building a set of subroutines which allow for the manipulation of symbolic polynomials. By "symbolic," we mean the list of coefficients and exponents which accompany a polynomial, e.g. two such polynomials are

A(x) = 3x2 + 2x + 4 and B(x) = x4 + 10x3 + 3x2 + 1

For a start, the capabilities we would include are the four basic arithmetic operations: addition, subtraction, multiplication, and division. We will also need input and output routines and some suitable format for preparing polynomials as input. The first step is to consider how to define polynomials as a computer structure. For a mathematician a polynomial is a sum of terms where each term has the form axe; x is the variable, a is the coefficient and e is the exponent. However this is not an appropriate definition for our purposes. When defining a data object one must decide what functions will be available, what their input is, what their output is and exactly what it is that they do. A complete specification of the data structure polynomial is now given.

structure POLYNOMIAL

declare ZERO( )    poly; ISZERO(poly)    Boolean

COEF(poly,exp)    coef;

ATTACH(poly,coef,exp)    poly

REM(poly,exp)    poly

SMULT(poly,coef,exp)    poly

ADD(poly,poly)    poly; MULT(poly,poly)    poly;

for all P,Q,    poly c,d,    coef e,f    exp let

REM(ZERO,f) :: = ZERO

REM(ATTACH(P,c,e),f) :: =

if e = f then REM(P,f) else ATTACH(REM(P,f),c,e)

ISZERO(ZERO) :: = true

ISZERO(ATTACH(P,c,e)):: =

if COEF(P,e) = - c then ISZERO(REM(P,e)) else false

COEF(ZERO,e) :: = 0

COEF(ATTACH(P,c,e),f) :: =

if e = f then c + COEF(P,f) else COEF(P,f)

SMULT(ZERO,d,f) :: = ZERO

SMULT(ATTACH(P,c,e),d,f) :: =

ATTACH(SMULT(P,d,f),c  d,e + f)

ADD(P,ZERO):: = P

ADD(P,ATTACH(Q,d,f)) :: = ATTACH(ADD(P,Q),d,f)

MULT(P,ZERO) :: = ZERO

MULT(P,ATTACH(Q,d,f)) :: =

ADD(MULT(P,Q),SMULT(P,d,f))

end

end POLYNOMIAL

In this specification every polynomial is either ZERO or constructed by applying ATTACH to a polynomial. For example the polynomial P = 10x - 12x3 - 10x + 0x2 is represented by the string

ATTACH(ATTACH(ATTACH(ATTACH(ZERO, 10,1),-12,3),

- 10,1),0,2).

Notice the absense of any assumptions about the order of exponents, about nonzero coefficients, etc. These assumptions are decisions of representation. Suppose we wish to remove from P those terms having exponent one. Then we would write REM(P, 1) and by the axioms the above string would be transformed into

ATTACH(REM(ATTACH(ATTACH(ATTACH(ZERO,10,1), - 12,3),

- 10,1),1),0,2)

which is transformed into

ATTACH(REM(ATTACH(ATTACH(ZERO,10,1), - 12,3),1),0,2)

which becomes

ATTACH(ATTACH(REM(ATTACH(ZERO,10,1),1) - 12,3),0,2)

which becomes

ATTACH(ATTACH(REM(ZERO,1), - 12,3),0,2)

which becomes finally

ATTACH(ATTACH(ZERO, - 12,3),0,2)

or - 12x3 + 0x2.

These axioms are valuable in that they describe the meaning of each operation concisely and without implying an implementation. Note how trivial the addition and multiplication operations have become.

Now we can make some representation decisions. Exponents should be unique and in decreasing order is a very reasonable first decision. This considerably simplifies the operations ISZERO, COEF and REM while ADD, SMULT and MULT remain unchanged. Now assuming a new function EXP (poly) [image] exp which returns the leading exponent of poly, we can write a version of ADD which is expressed more like program, but is still representation independent.

//C = A + B where A,B are the input polynomials//

C    ZERO

while not ISZERO(A) and not ISZERO(B) do

case

:  EXP(A) < EXP(B):

C    ATTACH(C,COEF(B,EXP(B)),EXP(B))

B    REM(B,EXP(B))

:EXP(A) = EXP(B):

C    ATTACH(C,COEF(A,EXP(A))+COEF(B,EXP(B)),EXP(A))

A    REM(A,EXP(A)); B    REM(B,EXP(B))

:EXP(A) > EXP(B):

C    ATTACH(C,COEF(A,EXP(A)),EXP(A))

A    REM(A,EXP(A))

end

end

insert any remaining terms in A or B into C

The basic loop of this algorithm consists of merging the terms of the two polynomials, depending upon the result of comparing the exponents. The case statement determines how the exponents are related and performs the proper action. Since the tests within the case statement require two terms, if one polynomial gets exhausted we must exit and the remaining terms of the other can be copied directly into the result. With these insights, suppose we now consider the representation question more carefully.

A general polynomial A(x) can be written as

anxn + an-1xn-1 + ... + a1x + a0

where an [image] 0 and we say that the degree of A is n. Then we can represent A(x) as an ordered list of coefficients using a one dimensional array of length n + 2,

A = (n,an ,an-1, ...,a1,a0).

The first element is the degree of A followed by the n + 1 coefficients in order of decreasing exponent. This representation leads to very simple algorithms for addition and multiplication. We have avoided the need to explicitly store the exponent of each term and instead we can deduce its value by knowing our position in the list and the degree.

But are there any disadvantages to this representation? Hopefully you have already guessed the worst one, which is the large amount of wasted storage for certain polynomials. Consider x1000 + 1, for instance. It will require a vector of length 1002, while 999 of those values will be zero. Therefore, we are led to consider an alternative scheme.

Suppose we take the polynomial A(x) above and keep only its nonzero coefficients. Then we will really have the polynomial

bm-1 xem-1 + bm-2xem-2 + ... + boxeo

(1)

where each bi is a nonzero coefficient of A and the exponents ei are decreasing [image]. If all of A's coefficients are nonzero, then m = n + 1, ei = i, and bi = ai for [image]. Alternatively, only an may be nonzero, in which case m = 1, bo = an, and e0 = n. In general, the polynomial in (1) could be represented by the ordered list of length 2m + 1,

(m,em-1,bm-1,em-2,bm-2, ...,e0,b0).

The first entry is the number of nonzero terms. Then for each term there are two entries representing an exponent-coefficient pair.

Is this method any better than the first scheme? Well, it certainly solves our problem with x1000 + 1, which now would be represented as (2,1000,1,0,1). Basic algorithms will need to be more complex because we must check each exponent before we handle its coefficient, but this is not too serious. As for storage, this scheme could be worse than the former. For example, x4 + 10x3 + 3x2 + 1 would have the two forms

(4,1,10,3,0,1) or (4,4,1,3,10,2,3,0,1).

In the worst case, scheme 2 requires less than twice as much storage as scheme 1 (when the degree = n and all n + 1 coefficients are nonzero). But scheme 1 could be much more wasteful, as in the case of x1000 + 1, where it needs more than 200 times as many locations. Therefore, we will prefer representation scheme 2 and use it.

Let us now write a procedure in SPARKS for adding two polynomials represented as in scheme 2.

procedure PADD(A,B,C)

//A(1:2m + 1), B(1:2n + 1), C(1:2(m + n) + 1)//

1    m    A(1); n    B(l)

2    p    q    r    2

3    while p    2m and q    2n do

4      case       //compare exponents//

:A(p) = B(q): C(r + 1)    A(p + 1) + B(q + 1)

//add coefficients//

if C(r + 1)    0

then [C(r)    A(p); r    r + 2]

//store exponent//

p    p + 2; q    q + 2    //advance to next

terms//

:A(p) < B(q): C(r + 1)    B(q + 1); C(r)    B(q)

//store new term//

q    q + 2; r    r + 2    //advance to next

term//

:A(p) > B(q): C(r + 1)    A(p + 1); C(r)    A(p)

//store new term//

p    p + 2; r    r + 2   //advance to next

term//

end

end

5    while p    2m do  //copy remaining terms of A//

C(r)    A(p); C(r + 1)    A(p + 1)

p    p + 2 ; r    r + 2

end

6    while q    2n do  //copy remaining terms of B//

C(r)    B(q); C(r + 1)    B(q + 1)

q    q + 2; r    r + 2

end

7    C(l)    r/2 - 1         //number of terms in the sum//

end PADD

As this is one of our first complex algorithms written in SPARKS, suppose we point out some features. The procedure has parameters which are polynomial (or array) names, and hence they are capitalized. Three pointers (p,q,r) are used to designate a term in A, B, or C.

Comments appear to the right delimited by double slashes. The basic iteration step is governed by a while loop. Blocks of statements are grouped together using square brackets. Notice how closely the actual program matches with the original design. The code is indented to reinforce readability and to reveal more clearly the scope of reserved words. This is a practice you should adopt in your own coding. Statement two is a shorthand way of writing

r [image] 2; q [image] r; p [image] q

Let us now analyze the computing time of this algorithm. It is natural to carry out this analysis in terms of m and n, the number of nonzero terms in A and B respectively. The assignments of lines 1 and 2 are made only once and hence contribute O(1) to the overall computing time. If either n = 0 or m = 0, the while loop of line 3 is not executed.

In case neither m nor n equals zero, the while loop of line 3 is entered. Each iteration of this while loop requires O(1) time. At each iteration, either the value of p or q or both increases by 2. Since the iteration terminates when either p or q exceeds 2m or 2n respectively, the number of iterations is bounded by m + n - 1. This worst case is achieved, for instance, when A(x) = [image] ni=0x2i and B(x) = [image] ni=0x2i+1 Since none of the exponents are the same in A and B, A(p) [image] B(q). Consequently, on each iteration the value of only one of p or q increases by 2. So, the worst case computing time for this while loop is O(n + m). The total computing time for the while loops of lines 5 and 6 is bounded by O(n + m), as the first cannot be iterated more than m times and the second more than n . Taking the sum of all of these steps, we obtain O(n + m) as the asymptotic computing time of this algorithm.

This example shows the array as a useful representational form for ordered lists. Returning to the abstract object--the ordered list--for a moment, suppose we generalize our problem and say that it is now required to represent a variable number of lists where the size of each may vary. In particular we now have the m lists

(a11,a12, ...,a1n 1), (a2l,a22, ...,a2n 2), ...,(am1,am2, ...,amnm)

where ni, the size of the i-th list, is an integer greater than or equal to zero.

A two dimensional array could be a poor way to represent these lists because we would have to declare it as A(m,max{n1, ...,nm}), which might be very wasteful of space. Instead we might store them in a one dimensional array and include a front(i) and rear(i) pointer for the beginning and end of each list. This only requires 2m + n1 + n2 + ... + nm locations rather than m times max{n1, . . .,nm} locations. But the one dimensional array presents problems when we try to insert an item in list i and there is no more room unless we move the elements of list i + 1 and perhaps list i + 2, ...,list m to the right.

To make this problem more concrete, let us return to the ordered list of polynomials represented using the second scheme. Suppose in addition to PADD, we have also written procedures which subtract, multiply, and divide two polynomials: PSUB, PMUL, and PDIV. We are making these four procedures available to any user who wants to manipulate polynomials. This hypothetical user may have many polynomials he wants to compute and he may not know their sizes.

He would include these subroutines along with a main procedure he writes himself. In this main program he needs to declare arrays for all of his polynomials (which is reasonable) and to declare the maximum size that every polynomial might achieve (which is harder and less reasonable). If he declares the arrays too large, much of that space will be wasted. Consider the main routine our mythical user might write if he wanted to compute the Fibonacci polynomials. These are defined by the recurrence relation

where F0(x) = 1 and F1 (x) = x. For example

F2(x) = x [image] Fl (x) + F0(x) = x2 + 1.

Suppose the programmer decides to use a two dimensional array to store the Fibonacci polynomials, the exponents and coefficients of Fi(x) being stored in the i-th row. For example F(2,*) = (2,2,1,0,1) implies F(2,1) = 2, F(2,2) = 2, F(2,3) = 1, F(2,4) = 0, F(2,5) = 1 and is the polynomial x2 + 1. Then the following program is produced.

procedure MAIN

declare F(0:100,203),TEMP(203)

read (n)

if n > 100 then [print ('n too large') stop]

F(0,*)    (1,0,1)        //set Fo = 1x0//

F(1,*)    (1,1,1)        //set F1 = 1x1//

for i    2 to n do

call PMUL(F(1,1),F(i - 1,1),TEMP(1))    //TEMP=x

Fi-(x)//

call PADD(TEMP(1),F(i - 2,1),F(i,1))    //Fi = TEMP +

Fi-2//

//TEMP is no longer needed//

end

for i    0 to n do

call PPRINT(F(i,1))     //polynomial print routine//

end

end MAIN

The author of this procedure has declared 101 * 203 = 20,503 locations to hold the Fibonacci polynomials, which is about twice as much as is actually needed. A much greater saving could be achieved if Fi(x) were printed as soon as it was computed in the first loop. Then by storing all polynomials in a single array, 1000 locations would be more than adequate. However, storing several polynomials in a single array alters the way one creates new polynomials and complicates matters if they are to be destroyed and the space reused.

This example reveals other limitations of the array as a means for data representation. The array is usually a homogeneous collection of data which will not allow us to intermix data of different types. Exponents and coefficients are really different sorts of numbers, exponents usually being small, non-negative integers whereas coefficients may be positive or negative, integer or rational, possibly double, triple or multiple precision integers or even other polynomials. Different types of data cannot be accommodated within the usual array concept. Ideally, we would like a representation which would:

(i) require the programmer only to name his polynomial variables and declare one maximum size for the entire working space;

(ii) provide a system which would automatically maintain all polynomials until the number of terms exceeds the work space;

(iii) allow the programmer to use different representations for different parts of the data object.

Let's pursue the idea of storing all polynomials in a single array called POLY. Let the polynomials be

A(x) = 2x + 3, B(x) = x2 + 5x + 3, C(x) = 3x10 + 9x4

POLY: 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 ... max

Then the name of a polynomial is never the array POLY, but a simple variable whose value is a pointer into the place in POLY where it begins. For instance, in the above case we might have A = 1, B = 6, and C = 13. Also we need a pointer to tell us where the next free location is, as above where free = 18.

If we made a call to our addition routine, say PADD (A,B,D), then it would take the polynomials starting at POLY(A) and POLY(B) and store the result starting at POLY(free). If the result has k terms, then D [image] free and free [image] free + 2k + 1. Now we have localized all storage to one array. As we create polynomials, free is continually incremented until it tries to exceed max. When this happens must we quit? We must unless there are some polynomials which are no longer needed. There may be several such polynomials whose space can be reused. We could write a subroutine which would compact the remaining polynomials, leaving a large, consecutive free space at one end. But this may require much data movement. Even worse, if we move a polynomial we must change its pointer. This demands a sophisticated compacting routine coupled with a disciplined use of names for polynomials. In Chapter 4 we will see an elegant solution to these problems.

2.3 SPARSE MATRICES

A matrix is a mathematical object which arises in many physical problems. As computer scientists, we are interested in studying ways to represent matrices so that the operations to be performed on them can be carried out efficiently. A general matrix consists of m rows and n columns of numbers as in figure 2.1.

Figure 2.1: Example of 2 matrices

The first matrix has five rows and three columns, the second six rows and six columns. In general, we write m X n (read m by n) to designate a matrix with m rows and n columns. Such a matrix has mn elements. When m is equal to n, we call the matrix square.

It is very natural to store a matrix in a two dimensional array, say A(1:m, 1:n). Then we can work with any element by writing A(i,j); and this element can be found very quickly, as we will see in the next section. Now if we look at the second matrix of figure 2.1, we see that it has many zero entries. Such a matrix is called sparse. There is no precise definition of when a matrix is sparse and when it is not, but it is a concept which we can all recognize intuitively. Above, only 8 out of 36 possible elements are nonzero and that is sparse! A sparse matrix requires us to consider an alternate form of representation. This comes about because in practice many of the matrices we want to deal with are large, e.g., 1000 X 1000, but at the same time they are sparse: say only 1000 out of one million possible elements are nonzero. On most computers today it would be impossible to store a full 1000 X 1000 matrix in the memory at once. Therefore, we ask for an alternative representation for sparse matrices. The alternative representation will explicitly store only the nonzero elements.

Each element of a matrix is uniquely characterized by its row and column position, say i,j. We might then store a matrix as a list of 3-tuples of the form

(i,j,value).

Also it might be helpful to organize this list of 3-tuples in some way, perhaps placing them so that the row numbers are increasing. We can go one step farther and require that all the 3-tuples of any row be stored so that the columns are increasing. Thus, we might store the second matrix of figure 2.1 in the array A(0:t,1:3) where t = 8 is the number of nonzero terms.

      1,  2,   3

     ------------

A(0,  6,  6,    8

 (1,  1,  1,   15

 (2,  1,  4,   22

 (3,  1,  6,  -15

 (4,  2,  2,   11

 (5,  2,  3,    3

 (6,  3,  4,   -6

 (7,  5,  1,   91

 (8,  6,  3,   28

Figure 2.2: Sparse matrix stored as triples

The elements A(0,1) and A(0,2) contain the number of rows and columns of the matrix. A(0,3) contains the number of nonzero terms.

Now what are some of the operations we might want to perform on these matrices? One operation is to compute the transpose matrix. This is where we move the elements so that the element in the i,j position gets put in the j,i position. Another way of saying this is that we are interchanging rows and columns. The elements on the diagonal will remain unchanged, since i = j.

The transpose of the example matrix looks like

      1,  2,    3

     --------------

B(0,  6,  6,    8

 (1,  1,  1,   15

 (2,  1,  5,   91

 (3,  2,  2,   11

 (4,  3,  2,    3

 (5,  3,  6,   28

 (6,  4,  1,   22

 (7,  4,  3,   -6

 (8,  6,  1,  -15

Since A is organized by row, our first idea for a transpose algorithm might be

for each row i do

take element (i,j,val) and

store it in (j,i,val) of the transpose

end

The difficulty is in not knowing where to put the element (j,i,val) until all other elements which precede it have been processed. In our example of figure 2.2, for instance, we have item

(1,1,15) which becomes (1,1,15)

(1,4,22) which becomes (4,1,22)

(1,6, - 15) which becomes (6,1, - 15).

If we just place them consecutively, then we will need to insert many new triples, forcing us to move elements down very often. We can avoid this data movement by finding the elements in the order we want them, which would be as

for all elements in column j do

place element (i,j,val) in position (j,i,val)

end

This says find all elements in column 1 and store them into row 1, find all elements in column 2 and store them in row 2, etc. Since the rows are originally in order, this means that we will locate elements in the correct column order as well. Let us write out the algorithm in full.

procedure TRANSPOSE (A,B)

// A is a matrix represented in sparse form//

// B is set to be its transpose//

1   (m,n,t)    (A(0,l),A(0,2),A(0,3))

2   (B(0,1),B(0,2),B(0,3))    (n,m,t)

3   if t    0 then return     //check for zero matrix//

4   q    1      //q is position of next term in B//

5   for col    1 to n do     //transpose by columns//

6     for p    1 to t do     //for all nonzero terms do//

7       if A(p,2) = col      //correct column//

8            then [(B(q,1),B(q,2),B(q,3))     //insert next term

of B//

9                  (A(p,2),A(p,1),A(p,3))

10                  q    q + 1]

11     end

12   end

13 end TRANSPOSE

The above algorithm makes use of (lines 1, 2, 8 and 9) the vector replacement statement of SPARKS. The statement

(a,b,c) [image] (d,e,f)

is just a shorthand way of saying

a [image] d; b [image] e; c [image] f.

It is not too difficult to see that the algorithm is correct. The variable q always gives us the position in B where the next term in the transpose is to be inserted. The terms in B are generated by rows. Since the rows of B are the columns of A, row i of B is obtained by collecting all the nonzero terms in column i of A. This is precisely what is being done in lines 5-12. On the first iteration of the for loop of lines 5-12 all terms from column 1 of A are collected, then all terms from column 2 and so on until eventually, all terms from column n are collected.

How about the computing time of this algorithm! For each iteration of the loop of lines 5-12, the if clause of line 7 is tested t times. Since the number of iterations of the loop of lines 5-12 is n, the total time for line 7 becomes nt. The assignment in lines 8-10 takes place exactly t times as there are only t nonzero terms in the sparse matrix being generated. Lines 1-4 take a constant amount of time. The total time for the algorithm is therefore O(nt). In addition to the space needed for A and B, the algorithm requires only a fixed amount of additional space, i.e. space for the variables m, n, t, q, col and p.

We now have a matrix transpose algorithm which we believe is correct and which has a computing time of O(nt). This computing time is a little disturbing since we know that in case the matrices had been represented as two dimensional arrays, we could have obtained the transpose of a n X m matrix in time O(nm). The algorithm for this takes the form:

for j    1 to n do

for i    1 to m do

B(j,i)    A(i,j)

end

end

The O(nt) time for algorithm TRANSPOSE becomes O(n2 m) when t is of the order of nm. This is worse than the O(nm) time using arrays. Perhaps, in an effort to conserve space, we have traded away too much time. Actually, we can do much better by using some more storage. We can in fact transpose a matrix represented as a sequence of triples in time O(n + t). This algorithm, FAST__TRANSPOSE, proceeds by first determining the number of elements in each column of A. This gives us the number of elements in each row of B. From this information, the starting point in B of each of its rows is easily obtained. We can now move the elements of A one by one into their correct position in B.

procedure FAST--TRANSPOSE(A,B)

//A is an array representing a sparse m X n matrix with t nonzero

terms. The transpose is stored in B using only O(t + n)

operations//

declare S(1:n),T(1:n);      //local arrays used as pointers//

1      (m,n,t)    (A(0,1),A(0,2),A(0,3))

2      (B(0,1),B(0,2),B(0,3))    (n,m,t)     //store dimensions of

transpose//

3      if t    0 then return  //zero matrix//

4      for i    1 to n do S(i)    0 end

5      for i    1 to t do   //S(k) is the number of//

6        S(A(i,2))    S(A(i,2)) + 1      //elements in row k of B//

7      end

8      T(1)    1

9      for i    2 to n do                //T(i) is the starting//

10         T(i)    T(i - 1) + S(i - 1)    //position of row i in

B//

11      end

12      for i    1 to t do      //move all t elements of A to B//

13        j    A(i,2)         //j is the row in B//

14        (B(T(j),1),B(T(j),2),B(T(j),3))           //store in triple//

15          (A(i,2),A(i,1),A(i,3))

16        T(j)    T(j) + 1      //increase row j to next spot//

17      end

18  end FAST--TRANSPOSE

The correctness of algorithm FAST--TRANSPOSE follows from the preceding discussion and the observation that the starting point of row i, i > 1 of B is T(i - 1) + S(i - 1) where S(i - 1) is the number of elements in row i - 1 of B and T(i - 1) is the starting point of row i - 1. The computation of S and T is carried out in lines 4-11. In lines 12-17 the elements of A are examined one by one starting from the first and successively moving to the t-th element. T(j) is maintained so that it is always the position in B where the next element in row j is to be inserted.

There are four loops in FAST--TRANSPOSE which are executed n, t, n - 1, and t times respectively. Each iteration of the loops takes only a constant amount of time, so the order of magnitude is O(n + t). The computing time of O(n + t) becomes O(nm) when t is of the order of nm. This is the same as when two dimensional arrays were in use. However, the constant factor associated with FAST__TRANSPOSE is bigger than that for the array algorithm. When t is sufficiently small compared to its maximum of nm, FAST__TRANSPOSE will be faster. Hence in this representation, we save both space and time! This was not true of TRANSPOSE since t will almost always be greater than max{n,m} and O(nt) will therefore always be at least O(nm). The constant factor associated with TRANSPOSE is also bigger than the one in the array algorithm. Finally, one should note that FAST__TRANSPOSE requires more space than does TRANSPOSE. The space required by FAST__TRANSPOSE can be reduced by utilizing the same space to represent the two arrays S and T.

If we try the algorithm on the sparse matrix of figure 2.2, then after execution of the third for loop the values of S and T are

     (1)  (2)  (3)  (4)  (5)  (6)

S =   2    1    2    2    0    1

T =   1    3    4    6    8    8

S(i) is the number of entries in row i of the transpose. T(i) points to the position in the transpose where the next element of row i is to be stored.

Suppose now you are working for a machine manufacturer who is using a computer to do inventory control. Associated with each machine that the company produces, say MACH(1) to MACH(m), there is a list of parts that comprise each machine. This information could be represented in a two dimensional table

          PART(1)  PART(2)  PART(3)  ...   PART(n)

         ----------------------------------------

MACH(1)  |  0,       5,       2,     ...,    0

MACH(2)  |  0,       0,       0,     ...,    3

MACH(3)  |  1,       1,       0,     ...,    8

    .    |  .        .        .              .

    .    |  .        .        .              .

    .    |  .        .        .              .

MACH(m)  |  6,       0,       0,     ...,    7

         |

         |        array MACHPT(m,n)

The table will be sparse and all entries will be non-negative integers. MACHPT(i,j) is the number of units of PART(j) in MACH(i). Each part is itself composed of smaller parts called microparts. This data will also be encoded in a table whose rows are PART(1) to PART(n) and whose columns are MICPT(1) to MICPT(p). We want to determine the number of microparts that are necessary to make up each machine.

Observe that the number of MICPT(j) making up MACH(i) is

MACHPT(i,1) * MICPT(1,j) + MACHPT(i,2) * MICPT(2,j)

+ ... + MACHPT(i,n) * MICPT(n,j)

where the arrays are named MACHPT(m,n) and MICPT(n,p). This sum is more conveniently written as

If we compute these sums for each machine and each micropart then we will have a total of mp values which we might store in a third table MACHSUM(m,p). Regarding these tables as matrices this application leads to the general definition of matrix product:

Given A and B where A is m [image] n and B is n [image] p, the product matrix C has dimension m [image] p. Its i,j element is defined as

for [image] and [image]. The product of two sparse matrices may no longer be sparse, for instance,

Consider an algorithm which computes the product of two sparse matrices represented as an ordered list instead of an array. To compute the elements of C row-wise so we can store them in their proper place without moving previously computed elements, we must do the following: fix a row of A and find all elements in column j of B for j = 1,2,...,p. Normally, to find all the elements in column j of B we would have to scan all of B. To avoid this, we can first compute the transpose of B which will put all column elements consecutively. Once the elements in row i of A and column j of B have been located, we just do a merge operation similar to the polynomial addition of section 2.2. An alternative approach is explored in the exercises.

Before we write a matrix multiplication procedure, it will be useful to define a sub-procedure:

procedure STORESUM (C,q,row,col,sum)

//if sum is nonzero then along with its row and column position

it is stored into the q-th entry of the matrix//

if sum    0 then [(C(q,1),C(q,2),C(q,3)

(row,col,sum)

q    q + 1; sum    0]

end STORESUM

The algorithm MMULT which multiplies the matrices A and B to obtain the product matrix C uses the strategy outlined above. It makes use of variables i,j,q,r,col and row__begin. The variable r is the row of A that is currently being multiplied with the columns of B. row__begin is the position in A of the first element of row r. col is the column of B that is currently being multiplied with row r of A. q is the position in C for the next element generated. i and j are used to successively examine elements of row r and column col of A and B respectively. In addition to all this, line 6 of the algorithm introduces a dummy term into each of A and [image]. This enables us to handle end conditions (i.e., computations involving the last row of A or last column of B) in an elegant way.

We leave the correctness proof of this algorithm as an exercise. Let us examine its complexity. In addition to the space needed for A, B, C and some simple variables, space is also needed for the transpose matrix [image]. Algorithm FAST__TRANSPOSE also needs some additional space. The exercises explore a strategy for MMULT which does not explicitly compute [image] and the only additional space needed is the same as that required by FAST__TRANSPOSE. Turning our attention to the computing time of MMULT, we see that lines 1-6 require only O(p + t2) time. The while loop of lines 7-34 is executed at most m times (once for each row of A). In each iteration of the while loop of lines 9-29 either the value of i or j or of both increases by 1 or i and col are reset. The maximum total increment in j over the whole loop is t2. If dr is the number of terms in row r of A then the value of i can increase at most dr times before i moves to the next row of A.

When this happens, i is reset to row__begin in line 13. At the same time col is advanced to the next column. Hence, this resetting can take place at most p times (there are only p columns in B). The total maximum increments in i is therefore pdr. The maximum number of iterations of the while loop of lines 9-29 is therefore p + pdr + t2. The time for this loop while multiplying with row r of A is O(pdr + t2). Lines 30-33 take only O(dr) time. Hence, the time for the outer while loop, lines 7-34, for the iteration with row r of A is O(pdr + t2). The overall time for this loop is then O( [image] r(pdr + t2)) = O(pt1 + mt2).

Once again, we may compare this computing time with the time to multiply matrices when arrays are used. The classical multiplication algorithm is:

for i    1 to m do

for j    1 to p do

sum    0

for k    1 to n do

sum    sum + A(i,k) * B(k,j)

end

C(i,j)    sum

end

end

The time for this is O(mnp). Since t1 [image] nm and t2 [image] np, the time for MMULT is at most O(mnp). However, its constant factor is greater than that for matrix multiplication using arrays. In the worst case when t1 = nm or t2 = np, MMULT will be slower by a constant factor. However, when t1 and t2 are sufficiently smaller than their maximum values i.e., A and B are sparse, MMULT will outperform the above multiplication algorithm for arrays.

The above analysis for MMULT is nontrivial. It introduces some new concepts in algorithm analysis and you should make sure you understand the analysis.

As in the case of polynomials, this representation for sparse matrices permits one to perform operations such as addition, transpose and multiplication efficiently. There are, however, other considerations which make this representation undesirable in certain applications. Since the number of terms in a sparse matrix is variable, we would like to represent all our sparse matrices in one array rather than using a separate array for each matrix. This would enable us to make efficient utilization of space. However, when this is done, we run into difficulties in allocating space from this array to any individual matrix. These difficulties also arise with the polynomial representation of the previous section and will become apparent when we study a similar representation for multiple stacks and queues (section 3.4).

2.4 REPRESENTATION OF ARRAYS

Even though multidimensional arrays are provided as a standard data object in most high level languages, it is interesting to see how they are represented in memory. Recall that memory may be regarded as one dimensional with words numbered from 1 to m. So, we are concerned with representing n dimensional arrays in a one dimensional memory. While many representations might seem plausible, we must select one in which the location in memory of an arbitrary array element, say A(i1,i2, ...,in), can be determined efficiently. This is necessary since programs using arrays may, in general, use array elements in a random order. In addition to being able to retrieve array elements easily, it is also necessary to be able to determine the amount of memory space to be reserved for a particular array. Assuming that each array element requires only one word of memory, the number of words needed is the number of elements in the array. If an array is declared A(l1 :u1,l2:u2, ...,ln:un), then it is easy to see that the number of elements is

One of the common ways to represent an array is in row major order. If we have the declaration

A(4:5, 2:4, 1:2, 3:4)

then we have a total of 2 [image] 3 [image] 2 [image] 2 = 24 elements. Then using row major order these elements will be stored as

A(4,2,1,3), A(4,2,1,4), A(4,2,2,3), A(4,2,2,4)

and continuing

A(4,3,1,3), A(4,3,1,4), A(4,3,2,3), A(4,3,2,4)

for 3 more sets of four until we get

A(5,4,1,3), A(5,4,1,4), A(5,4,2,3), A(5,4,2,4).

We see that the subscript at the right moves the fastest. In fact, if we view the subscripts as numbers, we see that they are, in some sense, increasing:

4213,4214, ...,5423,5424.

Another synonym for row major order is lexicographic order.

From the compiler's point of view, the problem is how to translate from the name A(i1,i2, ...,in) to the correct location in memory. Suppose A(4,2,1,3) is stored at location 100. Then A(4,2,1,4) will be at 101 and A(5,4,2,4) at location 123. These two addresses are easy to guess. In general, we can derive a formula for the address of any element. This formula makes use of only the starting address of the array plus the declared dimensions.

To simplify the discussion we shall assume that the lower bounds on each dimension li are 1. The general case when li can be any integer is discussed in the exercises. Before obtaining a formula for the case of an n-dimensional array, let us look at the row major representation of 1, 2 and 3 dimensional arrays. To begin with, if A is declared A(1:u1), then assuming one word per element, it may be represented in sequential memory as in figure 2.3. If [image] is the address of A(1), then the address of an arbitrary element A(i) is just [image] + (i - 1).

array element:  A(l), A(2), A(3), ..., A(i),      ..., A(u1)

address:          ,    + 1,    + 2,  ...,    + i - 1, ...,    + u1 - 1

                total number of elements = u1

Figure 2.3: Sequential representation of A(1:u1)

The two dimensional array A(1:u1,1:u2) may be interpreted as u1 rows: row 1,row 2, ...,row u1, each row consisting of u2 elements. In a row major representation, these rows would be represented in memory as in figure 2.4.

Figure 2.4: Sequential representation of A(u1,u2)

Again, if [image] is the address of A(1 ,1), then the address of A(i,1) is [image] + (i - 1)u2, as there are i - 1 rows each of size u2 preceding the first element in the i-th row. Knowing the address of A(i,1), we can say that the address of A(i,j) is then simply [image] + (i - 1)u2 + (j - 1).

Figure 2.5 shows the representation of the 3 dimensional array A(1:u1,1:u2,1:u3). This array is interpreted as u1 2 dimensional arrays of dimension u2 x u3 . To locate A(i,j,k), we first obtain [image] + (i - 1) u2 u3 as the address for A(i,1,1) since there are i- 1 2 dimensional arrays of size u2 x u3 preceding this element. From this and the formula for addressing a 2 dimensional array, we obtain [image] + (i - 1) u2 u3 + (j - 1) u3 + (k - 1) as the address of A(i,j,k).

Generalizing on the preceding discussion, the addressing formula for any element A(i1,i2, ...,in) in an n-dimensional array declared as A(u1,u2, ...,un) may be easily obtained. If [image] is the address for A(1,1, ...,1) then [image] + (i1, - 1)u2u3 ... un is the address for A(i1,l, ...,1). The address for A(i1,i2,1, ...,1) is then [image] + (i1, - 1)u2u3 ... un + (i2 - 1)u3u4 ... un.

Repeating in this way the address for A(i1,i2, ...,in) is

(a) 3-dimensional array A(u1,u2,u3) regarded as u1 2-dimensional arrays.

(b) Sequential row major representation of a 3-dimensional array. Each 2-dimensional array is represented as in Figure 2.4.

Figure 2.5: Sequential representation of A(u1,u2,u3)

Note that aj may be computed from [image] using only one multiplication as aj = uj+1 aj+1. Thus, a compiler will initially take the declared bounds u1, ...,un and use them to compute the constants a1, ...,an-1 using n - 2 multiplications. The address of A(i1, ...,in) can then be found using the formula, requiring n - 1 more multiplications and n additions.

An alternative scheme for array representation, column major order, is considered in exercise 21.

To review, in this chapter we have used arrays to represent ordered lists of polynomials and sparse matrices. In all cases we have been able to move the values around, accessing arbitrary elements in a fixed amount of time, and this has given us efficient algorithms. However several problems have been raised. By using a sequential mapping which associates ai of (a1, ...,an) with the i-th element of the array, we are forced to move data around whenever an insert or delete operation is used. Secondly, once we adopt one ordering of the data we sacrifice the ability to have a second ordering simultaneously.

EXERCISES

1. Write a SPARKS procedure which multiplies two polynomials represented using scheme 2 in Section 2.2. What is the computing time of your procedure?

2. Write a SPARKS procedure which evaluates a polynomial at a value x0 using scheme 2 as above. Try to minimize the number of operations.

3. If A = (a1, ...,an) and B = (b1, ...,bm) are ordered lists, then A < B if ai = bi for [image] and aj < bj or if ai = bi for [image] and n < m. Write a procedure which returns -1,0, + 1 depending upon whether A < B, A = B or A > B. Assume you can compare atoms ai and bj.

4. Assume that n lists, n > 1, are being represented sequentially in the one dimensional array SPACE (1:m). Let FRONT(i) be one less than the position of the first element in the ith list and let REAR(i) point to the last element of the ith list, 1 [image] i [image] n. Further assume that REAR(i) [image] FRONT(i + 1), 1 [image] i [image] n with FRONT(n + 1) = m. The functions to be performed on these lists are insertion and deletion.

a) Obtain suitable initial and boundary conditions for FRONT(i) and REAR(i)

b) Write a procedure INSERT(i,j,item) to insert item after the (j - 1)st element in list i. This procedure should fail to make an insertion only if there are already m elements in SPACE.

5. Using the assumptions of (4) above write a procedure DELETE(i,j,item) which sets item to the j-th element of the i-th list and removes it. The i-th list should be maintained as sequentially stored.

6. How much space is actually needed to hold the Fibonacci polynomials F0,F1, ..,F100?

7. In procedure MAIN why is the second dimension of F = 203?

8. The polynomials A(x) = x2n + x2n-2 + ... + x2 + x0 and B(x) = x2n+1 + x2n-l + ... + x3 + x cause PADD to work very hard. For these polynomials determine the exact number of times each statement will be executed.

9. Analyze carefully the computing time and storage requirements of algorithm FAST--TRANSPOSE. What can you say about the existence of an even faster algorithm?

10. Using the idea in FAST--TRANSPOSE of m row pointers, rewrite algorithm MMUL to multiply two sparse matrices A and B represented as in section 2.3 without transposing B. What is the computing time of your algorithm?

11. When all the elements either above or below the main diagonal of a square matrix are zero, then the matrix is said to be triangular. Figure 2.6 shows a lower and upper triangular matrix.

Figure 2.6

In a lower triangular matrix, A, with n rows, the maximum number of nonzero terms in row i is i. Hence, the total number of nonzero terms is [image] ni=1 i = n(n + 1)/2. For large n it would be worthwhile to save the space taken by the zero entries in the upper triangle. Obtain an addressing formula for elements aij in the lower triangle if this lower triangle is stored by rows in an array B(1:n(n + 1)/2) with A(1,1) being stored in B(1).

What is the relationship between i and j for elements in the zero part of A?

12. Let A and B be two lower triangular matrices, each with n rows. The total number of elements in the lower triangles is n(n + 1). Devise a scheme to represent both the triangles in an array C(1:n,1:n + 1). [Hint: represent the triangle of A as the lower triangle of C and the transpose of B as the upper triangle of C.] Write algorithms to determine the values of A(i,j), B(i,j) 1 [image] i, j [image] n from the array C.

13. Another kind of sparse matrix that arises often in numerical analysis is the tridiagonal matrix. In this square matrix, all elements other than those on the major diagonal and on the diagonals immediately above and below this one are zero

Figure 2.7: Tridiagonal matrix A

If the elements in the band formed by these three diagonals are represented rowwise in an array, B, with A (1,1) being stored at B(1), obtain an algorithm to determine the value of A(i,j), 1 [image] i, j [image] n from the array B.

14. Define a square band matrix An,a to be a n x n matrix in which all the nonzero terms lie in a band centered around the main diagonal. The band includes a - 1 diagonals below and above the main diagonal and also the main diagonal.

a) How many elements are there in the band of An,a?

b) What is the relationship between i and j for elements aij in the band of An,a?

c) Assume that the band of An,a is stored sequentially in an array B by diagonals starting with the lowermost diagonal. Thus A4,3 above would have the following representation:

B(1) B(2) B(3) B(4) B(5) B(6) B(7) B(8) B(9) B(10) B(11) B(12) B(13) B(14)

 9    7    8    3    6    6    0    2    8    7     4     9     8     4

a31  a42   a21  a32   a43   a11  a22   a33  a44   a12    a23   a34    a13    a24

Obtain an addressing formula for the location of an element aij in the lower band of An,a.

e.g. LOC(a31) = 1, LOC(a42) = 2 in the example above.

15. A generalized band matrix An,a,b is a n x n matrix A in which all the nonzero terms lie in a band made up of a - 1 diagonals below the main diagonal, the main diagonal and b - 1 diagonals above the main diagonal (see the figure on the next page)

a) How many elements are there in the band of An,a,b?

b) What is the relationship between i and j for elements aij in the band of An,a,b?

c) Obtain a sequential representation of the band of An,a,b in the one dimensional array B. For this representation write an algorithm VALUE (n,a,b,i,j,B) which determines the value of element aij in the matrix An,a,b. The band of An,a,b is represented in the array B.

16. How much time does it take to locate an arbitrary element A(i,j) in the representation of section 2.3 and to change its value?

17. A variation of the scheme discussed in section 2.3 for sparse matrix representation involves representing only the non zero terms in a one dimensional array VA in the order described. In addition, a strip of n x m bits, BA (n,m), is also kept. BA(i,j) = 0 if A(i,j) = 0 and BA(i,j) = 1 if A(i,j) [image] 0. The figure below illustrates the representation for the sparse matrix of figure 2.1.

(i) On a computer with w bits per word, how much storage is needed to represent a sparse matrix Anxm with t nonzero terms?

(ii) Write an algorithm to add two sparse matrices A and C represented as above to obtain D = A + C. How much time does your algorithm take ?

(iii) Discuss the merits of this representation versus the representation of section 2.3. Consider space and time requirements for such operations as random access, add, multiply, and transpose. Note that the random access time can be improved somewhat by keeping another array RA(i) such that RA(i) = number of nonzero terms in rows 1 through i - 1.

18. A complex-valued matrix X is represented by a pair of matrices (A,B) where A and B contain real values. Write a program which computes the product of two complex valued matrices (A,B) and (C,D), where

(A,B) * (C,D) = (A + iB) * (C + iD) = (AC - BD) + i(AD + BC)

Determine the number of additions and multiplications if the matrices are all n [image] n.

19. How many values can be held by an array with dimensions A(0:n), B(-1:n,1:m), C(-n:0,2)?

20. Obtain an addressing formula for the element A(i1,i2, ...,in) in an array declared as A(l1:ul,l2:u2, ...,ln:un). Assume a row major representation of the array with one word per element and [image] the address of A(l1,l2, ...,ln).

21. Do exercise 20 assuming a column major representation. In this representation, a 2 dimensional array is stored sequentially by column rather than by rows.

22. An m X n matrix is said to have a saddle point if some entry A(i,j) is the smallest value in row i and the largest value in column j. Write a SPARKS program which determines the location of a saddle point if one exists. What is the computing time of your method?

23. Given an array A(1:n) produce the array Z(1:n) such that Z(1) = A(n), Z(2) = A(n - 1), ..., Z(n - 1) = A(2), Z(n) = A(1). Use a minimal amount of storage.

24. One possible set of axioms for an ordered list comes from the six operations of section 2.2.

structure ORDERED__LlST(atoms)

declare MTLST( )    list

LEN(list)    integer

RET(list,integer)    atom

STO(list,integer,atom)    list

INS(list,integer,atom)    list

DEL(list,integer)    list;

for all L    list, i.j    integer a,b    atom let

LEN(MTLST) :: = 0; LEN(STO(L,i,a)) :: = 1 + LEN(L)

RET(MTLST,j) :: = error

RET(STO(L,i,a),j) :: =

if i = j then a else RET(L,j)

INS(MTLST,j,b) :: = STO(MTLST,j,b)

INS(STO(L,i,a),j,b) :: =

if i    j then STO(INS(L,j,b), i + 1,a)

else STO(INS(L,j,b),i,a)

DEL(MTLST,j) :: = MTLST

DEL(STO(L,i,a),j) :: =

if i = j then DEL(L,j)

else if i > j then STO(DEL(L,j),i - 1,a)

else STO(DEL(L,j),i,a)

end

end ORDERED__LIST

Use these axioms to describe the list A = (a, b, c, d, e) and show what happens when DEL (A,2) is executed.

25. There are a number of problems, known collectively as "random walk" problems which have been of long standing interest to the mathematical community. All but the most simple of these are extremely difficult to solve and for the most part they remain largely unsolved. One such problem may be stated as follows:

A (drunken) cockroach is placed on a given square in the middle of a tile floor in a rectangular room of size n x m tiles. The bug wanders (possibly in search of an aspirin) randomly from tile to tile throughout the room. Assuming that he may move from his present tile to any of the eight tiles surrounding him (unless he is against a wall) with equal probability, how long will it take him to touch every tile on the floor at least once?

Hard as this problem may be to solve by pure probability theory techniques, the answer is quite easy to solve using the computer. The technique for doing so is called "simulation" and is of wide-scale use in industry to predict traffic-flow, inventory control and so forth. The problem may be simulated using the following method:

An array K [image] UNT dimensioned N X M is used to represent the number of times our cockroach has reached each tile on the floor. All the cells of this array are initialized to zero. The position of the bug on the floor is represented by the coordinates (IBUG,JBUG) and is initialized by a data card. The 8 possible moves of the bug are represented by the tiles located at (IBUG + IM [image] VE(K), JBUG + JM [image] VE(K)) where 1 [image] K [image] 8 and:

IM  VE(1) = -1            JM  VE(1) = 1

IM  VE(2) = 0             JM  VE(2) = 1

IM  VE(3) = 1             JM  VE(3) = 1

IM  VE(4) = 1             JM  VE(4) = 0

IM  VE(5) = 1             JM  VE(5) = -1

IM  VE(6) = 0             JM  VE(6) = -1

IM  VE(7) = -1            JM  VE(7) = -1

IM  VE(8) = -1            JM  VE(8) = 0

A random walk to one of the 8 given squares is simulated by generating a random value for K lying between 1 and 8. Of course the bug cannot move outside the room, so that coordinates which lead up a wall must be ignored and a new random combination formed. Each time a square is entered, the count for that square is incremented so that a non-zero entry shows the number of times the bug has landed on that square so far. When every square has been entered at least once, the experiment is complete.

Write a program to perform the specified simulation experiment. Your program MUST:

1) Handle values of N and M

2 < N [image] 40

2 [image] M [image] 20

2) Perform the experiment for

a) N = 15, M = 15 starting point: (20,10)

b) N = 39, M = 19 starting point: (1,1)

3) Have an iteration limit, that is, a maximum number of squares the bug may enter during the experiment. This assures that your program does not get "hung" in an "infinite" loop. A maximum of 50,000 is appropriate for this lab.

4) For each experiment print: a) the total number of legal moves which the cockroach makes; b) the final K [image] UNT array. This will show the "density" of the walk, that is the number of times each tile on the floor was touched during the experiment.

(Have an aspirin) This exercise was contributed by Olson.

26. Chess provides the setting for many fascinating diversions which are quite independent of the game itself. Many of these are based on the strange "L-shaped" move of the knight. A classical example is the problem of the knight's tour, which has captured the attention of mathematicians and puzzle enthusiasts since the beginning of the eighteenth century. Briefly stated, the problem is to move the knight, beginning from any given square on the chessboard, in such a manner that it travels successively to all 64 squares, touching each square once and only once. It is convenient to represent a solution by placing the numbers 1,2, ...,64 in the squares of the chessboard indicating the order in which the squares are reached. Note that it is not required that the knight be able to reach the initial position by one more move; if this is possible the knight's tour is called re-entrant.

One of the more ingenious methods for solving the problem of the knight's tour is that given by J. C. Warnsdorff in 1823. His rule is that the knight must always be moved to one of the squares from which there are the fewest exits to squares not already traversed.

The goal of this exercise is to write a computer program to implement Warnsdorff's rule. The ensuing discussion will be much easier to follow, however, if the student will first try to construct a particular solution to the problem by hand before reading any further.

The most important decisions to be made in solving a problem of this type are those concerning how the data is to be represented in the computer. Perhaps the most natural way to represent the chessboard is by an 8 x 8 array B [image] ARD as shown in the figure below. The eight possible moves of a knight on square (5,3) are also shown in the figure.

B  ARD

   1  2  3  4  5  6  7  8

   ----------------------

1

2

3     8     1

4  7           2

5        K

6  6           3

7     5     4

8

In general a knight at (I,J) may move to one of the squares (I - 2,J + 1), (I -1,J + 2), (I + 1,J + 2), (I + 2,J + 1), (I + 2,J - 1), (I + 1,J - 2), (I -1,J - 2), (I - 2,J - 1). Notice, however that if (I,J) is located near one of the edges of the board, some of these possibilities could move the knight off the board, and of course this is not permitted. The eight possible knight moves may conveniently be represented by two arrays KTM [image] V1 and KTM [image] V2 as shown below.

KTM  V1  KTM  V2

  -2        1

  -1        2

   1        2

   2        1

   2       -1

   1       -2

  -1       -2

  -2       -1

Then a knight at (I,J) may move to (I + KTM [image] V1(K), J + KTM [image] V2(K)), where K is some value between 1 and 8, provided that the new square lies on the chessboard.

Below is a description of an algorithm for solving the knight's tour problem using Warnsdorff's rule. The data representation discussed in the previous section is assumed.

a. [Initialize chessboard] For 1 [image] I,J [image] 8 set B [image] ARD(I,J) = 0.

b. [Set starting position] Read and print I,J and then set B [image] ARD(I,J) to 1.

c. [Loop] For 2 [image] M [image] 64 do steps d through g.

d. [Form set of possible next squares] Test each of the eight squares one knight's move away from (I,J) and form a list of the possibilities for the next square (NEXTI(L), NEXTJ(L)). Let NP [image] S be the number of possibilities. (That is, after performing this step we will have NEXTI(L) = I + KTM [image] V1(K) and NEXTJ(L) = J + KTM [image] V2(K), for certain values of K between 1 and 8. Some of the squares (I + KTM [image] V1(K), J + KTM [image] V2(K)) may be impossible for the next move either because they lie off the chessboard or because they have been previously occupied by the knight--i.e., they contain a nonzero number. In every case we will have 0 [image] NP [image] S [image] 8.)

e. [Test special cases] If NP [image] S = 0 the knight's tour has come to a premature end; report failure and then go to step h. If NP [image] S = 1 there is only one possibility for the next move; set MIN = 1 and go right to step g.

f. [Find next square with minimum number of exits] For 1 [image] L [image] NP [image] S set EXITS(L) to the number of exits from square (NEXTI(L),NEXTJ(L)). That is, for each of the values of L examine each of the next squares (NEXTI(L) + KTM [image] V1(K), NEXTJ(L) + KTM [image] V2(K)) to see if it is an exit from NEXTI(L), NEXTJ(L)), and count the number of such exits in EXITS(L). (Recall that a square is an exit if it lies on the chessboard and has not been previously occupied by the knight.) Finally, set MIN to the location of the minimum value of EXITS. (There may be more than one occurrences of the minimum value of EXITS. If this happens, it is convenient to let MIN denote the first such occurrence, although it is important to realize that by so doing we are not actually guaranteed of finding a solution. Nevertheless, the chances of finding a complete knight's tour in this way are remarkably good, and that is sufficient for the purposes of this exercise.)

g. [Move knight] Set I = NEXTI(MIN), J = NEXTJ(MIN) and B [image] ARD(I,J) = M. (Thus, (I,J) denotes the new position of the knight, and B [image] ARD(I,J) records the move in proper sequence.)

h. [Print] Print out B [image] ARD showing the solution to the knight's tour, and then terminate the algorithm.

The problem is to write a program which corresponds to this algorithm. This exercise was contributed by Legenhausen and Rebman.

Go to Chapter 3 Back to Table of Contents


Fundamentals: CHAPTER 3: STACKS AND QUEUES

 Return to Table of Contents

CHAPTER 3: STACKS AND QUEUES

3.1 FUNDAMENTALS

Two of the more common data objects found in computer algorithms are stacks and queues. They arise so often that we will discuss them separately before moving on to more complex objects. Both these data objects are special cases of the more general data object, an ordered list which we considered in the previous chapter. Recall that A = (a1, a2, ...,an), is an ordered list of [image] elements. The ai are referred to as atoms which are taken from some set. The null or empty list has n = 0 elements.

A stack is an ordered list in which all insertions and deletions are made at one end, called the top. A queue is an ordered list in which all insertions take place at one end, the rear, while all deletions take place at the other end, the front. Given a stack S = (a1, ...an) then we say that a1 is the bottommost element and element ai is on top of element ai - 1, 1 < i [image] n. When viewed as a queue with an as the rear element one says that ai+1 is behind ai, 1 [image] i< n.

Figure 3.1

The restrictions on a stack imply that if the elements A,B,C,D,E are added to the stack, in that order, then the first element to be removed/deleted must be E. Equivalently we say that the last"element to be inserted into the stack will be the first to be removed. For this reason stacks are sometimes referred to as Last In First Out (LIFO) lists. The restrictions on a queue require that the first element which is inserted into the queue will be the first one to be removed. Thus A is the first letter to be removed, and queues are known as First In First Out (FIFO) lists. Note that the data object queue as defined here need not necessarily correspond to the mathematical concept of queue in which the insert/delete ru les may be different.

One natural example of stacks which arises in computer programming is the processing of subroutine calls and their returns. Suppose we have a main procedure and three subroutines as below:

|  proc MAIN |  |  proc A1 |  |  proc A2 |  |  proc A3 |

|       ___  |  |     ___  |  |     ___  |  |     ___  |

|       ___  |  |     ___  |  |     ___  |  |     ___  |

|       ___  |  |     ___  |  |     ___  |  |     ___  |

|  call A1   |  |  call A2 |  |  call A3 |  |     ___  |

| r:         |  | s:       |  | t:       |  |          |

|       ___  |  |     ___  |  |     ___  |  |     ___  |

|       ___  |  |     ___  |  |     ___  |  |     ___  |

| end        |  | end      |  | end      |  | end      |

Figure 3.2. Sequence of subroutine calls

The MAIN program calls subroutine A1. On completion of A1 execution of MAIN will resume at location r. The address r is passed to Al which saves it in some location for later processing. Al then invokes A2 which in turn calls A3. In each case the calling procedure passes the return address to the called procedure. If we examine the memory while A3 is computing there will be an implicit stack which looks like

(q,r,s,t).

The first entry, q, is the address in the operating system where MAIN returns control. This list operates as a stack since the returns will be made in the reverse order of the calls. Thus t is removed before s, s before r and r before q. Equivalently, this means that A3 must finish processing before A2, A2 before A1, and A1 before MAIN. This list of return addresses need not be maintained in consecutive locations. For each subroutine there is usually a single location associated with the machine code which is used to retain the return address. This can be severely limiting in the case of recursive calls and re-entrant routines, since every time we call a subroutine the new return address wipes out the old one. For example, if we inserted a call to A1 within subroutine A3 expecting the return to be at location u, then at execution time the stack would become (q,u,s,t) and the return address r would be lost. When recursion is allowed, it is no longer adequate to reserve one location for the return address of each subroutine. Since returns are made in the reverse order of calls, an elegant and natural solution to this subroutine return problem is afforded through the explicit use of a stack of return addresses. Whenever a return is made, it is to the top address in the stack. Implementing recursion using a stack is discussed in Section 4.10.

Associated with the object stack there are several operations that are necessary:

CREATE (S) which creates S as an empty stack;

ADD (i,S) which inserts the element i onto the stack S and returns the new stack;

DELETE (S) which removes the top element of stack S and returns the new stack;

TOP (S) which returns the top element of stack S;

ISEMTS (S) which returns true if S is empty else false;

These five functions constitute a working definition of a stack. However we choose to represent a stack, it must be possible to build these operations. But before we do this let us describe formally the structure STACK.

structure STACK (item)

1   declare CREATE ( )    stack

2           ADD (item, stack)    stack

3           DELETE (stack)    stack

4           TOP (stack)    item

5           ISEMTS (stack)    boolean;

6   for all S    stack, i    item let

7       ISEMTS (CREATE)          :: = true

8       ISEMTS (ADD (i,S))       :: = false

9       DELETE (CREATE)          :: = error

10      DELETE (ADD (i,S))       :: = S

11      TOP(CREATE)              :: = error

12      TOP(ADD(i,S))            :: = i

13   end

end STACK

The five functions with their domains and ranges are declared in lines 1 through 5. Lines 6 through 13 are the set of axioms which describe how the functions are related. Lines 10 and 12 are the essential ones which define the last-in-first-out behavior. The above definitions describe an infinite stack for no upper bound on the number of elements is specified. This will be dealt with when we represent this structure in a computer.

The simplest way to represent a stack is by using a one-dimensional array, say STACK(1:n), where n is the maximum number of allowable entries. The first or bottom element in the stack will be stored at STACK(1), the second at STACK(2) and the i-th at STACK(i). Associated with the array will be a variable, top, which points to the top element in the stack. With this decision made the following implementations result:

CREATE ( ) :: = declare STACK(1:n); top    0

ISEMTS(STACK) :: = if top = 0 then true

else false

TOP(STACK) :: = if top = 0 then error

else STACK(top)

The implementations of these three operations using an array are so short that we needn't make them separate procedures but can just use them directly whenever we need to. The ADD and DELETE operations are only a bit more complex.

procedure ADD (item, STACK, n, top)

//insert item into the STACK of maximum size n; top is the number

of elements curently in STACK//

if top    n then call STACK_FULL

top    top + 1

STACK (top)    item

end ADD

procedure DELETE (item, STACK, top)

//removes the top element of STACK and stores it in item

unless STACK is empty//

if top    0 then call STACK_EMPTY

item    STACK (top)

top    top - 1

end DELETE

These two procedures are so simple that they perhaps need no more explanation. Procedure DELETE actually combines the functions TOP and DELETE. STACK_FULL and STACK_EMPTY are procedures which we leave unspecified since they will depend upon the particular application. Often a stack full condition will signal that more storage needs to be allocated and the program re-run. Stack empty is often a meaningful condition. In Section 3.3 we will see a very important computer application of stacks where stack empty signals the end of processing.

The correctness of the stack implementation above may be established by showing that in this implementation, the stack axioms of lines 7-12 of the stack structure definition are true. Let us show this for the first three rules. The remainder of the axioms can be shown to hold similarly .

(i)     line 7: ISEMTS(CREATE):: = true

Since CREATE results in top being initialized to zero, it follows from the implementation of ISEMTS that ISEMTS(CREATE):: = true.

(ii)     line 8: ISEMTS(ADD(i,S)):: = false

The value of top is changed only in procedures CREATE, ADD and DELETE. CREATE initializes top to zero while ADD increments it by 1 so long as top is less than n (this is necessary because we can implement only a finite stack). DELETE decreases top by 1 but never allows its value to become less than zero. Hence, ADD(i,S) either results in an error condition (STACK_FULL), or leaves the value of top > 0. This then implies that ISEMTS(ADD(i,S)):: = false.

(iii)     line 9: DELETE(CREATE):: = error

This follows from the observation that CREATE sets top = 0 and the procedure DELETE signals the error condition STACK_EMPTY when top = 0.

Queues, like stacks, also arise quite naturally in the computer solution of many problems. Perhaps the most common occurrence of a queue in computer applications is for the scheduling of jobs. In batch processing the jobs are ''queued-up'' as they are read-in and executed, one after another in the order they were received. This ignores the possible existence of priorities, in which case there will be one queue for each priority.

As mentioned earlier, when we talk of queues we talk about two distinct ends: the front and the rear. Additions to the queue take place at the rear. Deletions are made from the front. So, if a job is submitted for execution, it joins at the rear of the job queue. The job at the front of the queue is the next one to be executed. A minimal set of useful operations on a queue includes the following:

CREATEQ(Q) which creates Q as an empty queue;

ADDQ(i,Q) which adds the element i to the rear of a queue and returns the new queue;

DELETEQ(Q) which removes the front element from the queue Q and returns the resulting queue;

FRONT(Q) which returns the front element of Q;

ISEMTQ(Q) which returns true if Q is empty else false.

A complete specification of this data structure is

structure QUEUE (item)

1    declare CREATEQ( )    queue

2            ADDQ(item,queue)    queue

3            DELETEQ(queue)    queue

4            FRONT(queue)    item

5            ISEMTQ(queue)    boolean;

6    for all Q    queue, i    item let

7        ISEMTQ(CREATEQ)   :: = true

8        ISEMTQ(ADDQ(i,Q)) :: = false

9        DELETEQ(CREATEQ)  :: = error

10        DELETEQ(ADDQ(i,Q)):: =

11         if ISEMTQ(Q) then CREATEQ

12            else ADDQ(i,DELETEQ(Q))

13        FRONT(CREATEQ)    :: = error

14        FRONT(ADDQ(i,Q))  :: =

15         if ISEMTQ(Q) then i else FRONT(Q)

16    end

17  end QUEUE

The axiom of lines 10-12 shows that deletions are made from the front of the queue.

The representation of a finite queue in sequential locations is somewhat more difficult than a stack. In addition to a one dimensional array Q(1:n), we need two variables, front and rear. The conventions we shall adopt for these two variables are that front is always 1 less than the actual front of the queue and rear always points to the last element in the queue. Thus, front = rear if and only if there are no elements in the queue. The initial condition then is front = rear = 0. With these conventions, let us try an example by inserting and deleting jobs, Ji,from a job queue

             Q(1)   (2)   (3)   (4)    (5)  (6)  (7)  ... Remarks

front  rear

  0     0          queue       empty                      Initial

  0     1    Jl                                           Job 1 joins Q

  0     2    J1     J2                                    Job 2 joins Q

  0     3    J1     J2    J3                              Job 3 joins Q

  1     3           J2    J3                              Job 1 leaves Q

  1     4           J2    J3    J4                        Job 4 joins Q

  2     4                 J3    J4                        Job 2 leaves Q

With this scheme, the following implementation of the CREATEQ, ISEMTQ, and FRONT operations results for a queue with capacity n:

CREATEQ(Q) :: = declare Q(l:n); front    rear    0

ISEMTQ(Q)  :: = if front = rear then true

else false

FRONT(Q)   :: = if ISEMTQ(Q) then error

else Q(front + 1)

The following algorithms for ADDQ and DELETEQ result:

procedure ADDQ(item, Q, n, rear)

//insert item into the queue represented in Q(l:n)//

 if rear = n then call QUEUE_FULL

 rear    rear + 1

 Q(rear)   item

end ADDQ

procedure DELETEQ(item, Q, front, rear)

//delete an element from a queue//

 if front = rear then call QUEUE_EMPTY

 front    front + 1

 item    Q(front)

end DELETEQ

The correctness of this implementation may be established in a manner akin to that used for stacks. With this set up, notice that unless the front regularly catches up with the rear and both pointers are reset to zero, then the QUEUE_FULL signal does not necessarily imply that there are n elements in the queue. That is, the queue will gradually move to the right. One obvious thing to do when QUEUE_FULL is signaled is to move the entire queue to the left so that the first element is again at Q(1) and front = 0. This is time consuming, especially when there are many elements in the queue at the time of the QUEUE_FULL signal.

Let us look at an example which shows what could happen, in the worst case, if each time the queue becomes full we choose to move the entire queue left so that it starts at Q(1). To begin, assume there are n elements J1, ...,Jn in the queue and we next receive alternate requests to delete and add elements. Each time a new element is added, the entire queue of n - 1 elements is moved left.

Figure 3.3

A more efficient queue representation is obtained by regarding the array Q(1:n) as circular. It now becomes more convenient to declare the array as Q(0:n - 1). When rear = n - 1, the next element is entered at Q(0) in case that spot is free. Using the same conventions as before, front will always point one position counterclockwise from the first element in the queue. Again, front = rear if and only if the queue is empty. Initially we have front = rear = 1. Figure 3.4 illustrates some of the possible configurations for a circular queue containing the four elements J1-J4 with n > 4. The assumption of circularity changes the ADD and DELETE algorithms slightly. In order to add an element, it will be necessary to move rear one position clockwise, i.e.,

if rear = n - 1 then rear    0

else rear    rear + 1.

Figure 3.4: Circular queue of n elements and four jobs J1, J2, J3, J4

Using the modulo operator which computes remainders, this is just rear   [image]  (rear + 1)mod n. Similarly, it will be necessary to move front one position clockwise each time a deletion is made. Again, using the modulo operation, this can be accomplished by front   [image]  (front + l)mod n. An examination of the algorithms indicates that addition and deletion can now be carried out in a fixed amount of time or O(1).

procedure ADDQ(item, Q, n, front, rear)

//insert item in the circular queue stored in Q(0:n - 1);

 rear points to the last item and front is one position

 counterclockwise from the first item in Q//

rear    (rear + l)mod n      //advance rear clockwise//

if front = rear then call QUEUE-FULL

Q(rear)    item           //insert new item//

end ADDQ

procedure DELETEQ(item, Q, n, front, rear)

//removes the front element of the queue Q(0:n - 1)//

if front = rear then call QUEUE-EMPTY

front    (front + 1)mod n      //advance front clockwise//

item    Q(front)        //set item to front of queue//

end DELETEQ

One surprising point in the two algorithms is that the test for queue full in ADDQ and the test for queue empty in DELETEQ are the same. In the case of ADDQ, however, when front = rear there is actually one space free, i.e. Q(rear), since the first element in the queue is not at Q(front) but is one position clockwise from this point. However, if we insert an item here, then we will not be able to distinguish between the cases full and empty, since this insertion would leave front = rear. To avoid this, we signal queue-full thus permitting a maximum, of n - 1 rather than n elements to be in the queue at any time. One way to use all n positions would be to use another variable, tag, to distinguish between the two situations, i.e. tag = 0 if and only if the queue is empty. This would however slow down the two algorithms. Since the ADDQ and DELETEQ algorithms will be used many times in any problem involving queues, the loss of one queue position will be more than made up for by the reduction in computing time.

The procedures QUEUE_FULL and QUEUE_EMPTY have been used without explanation, but they are similar to STACK_FULL and STACK_EMPTY. Their function will depend on the particular application.

3.2 A MAZING PROBLEM

The rat-in-a-maze experiment is a classical one from experimental psychology. A rat (or mouse) is placed through the door of a large box without a top. Walls are set up so that movements in most directions are obstructed. The rat is carefully observed by several scientists as it makes its way through the maze until it eventually reaches the other exit. There is only one way out, but at the end is a nice hunk of cheese. The idea is to run the experiment repeatedly until the rat will zip through the maze without taking a single false path. The trials yield his learning curve.

We can write a computer program for getting through a maze and it will probably not be any smarter than the rat on its first try through. It may take many false paths before finding the right one. But the computer can remember the correct path far better than the rat. On its second try it should be able to go right to the end with no false paths taken, so there is no sense re-running the program. Why don't you sit down and try to write this program yourself before you read on and look at our solution. Keep track of how many times you have to go back and correct something. This may give you an idea of your own learning curve as we re-run the experiment throughout the book.

Let us represent the maze by a two dimensional array, MAZE(1:m, 1:n), where a value of 1 implies a blocked path, while a 0 means one can walk right on through. We assume that the rat starts at MAZE(1,1) and the exit is at MAZE(m,n).

Figure 3.5

With the maze represented as a two dimensional array, the location of the rat in the maze can at any time be described by the row, i, and column, j of its position. Now let us consider the possible moves the rat can make at some point (i,j) in the maze. Figure 3.6 shows the possible moves from any point (i,j). The position (i,j) is marked by an X. If all the surrounding squares have a 0 then the rat can choose any of these eight squares as its next position. We call these eight directions by the names of the points on a compass north, northeast, east, southeast, south, southwest, west, and northwest, or N, NE, E, SE, S, SW, W, NW.

Figure 3.6

We must be careful here because not every position has eight neighbors. If (i,j) is on a border where either i = 1 or m, or j = 1 or n, then less than eight and possibly only three neighbors exist. To avoid checking for these border conditions we can surround the maze by a border of ones. The array will therefore be declared as MAZE(0:m + 1,0:n + 1).

Another device which will simplify the problem is to predefine the possible directions to move in a table, MOVE(1:8.1:2), which has the values

MOVE   1   2

      --  --

(1)   -1   0  north

(2)   -1   1  northeast

(3)    0   1  east

(4)    1   1  southeast

(5)    1   0  south

(6)    1  -1  southwest

(7)    0  -1  west

(8)   -1  -1  northwest

By equating the compass names with the numbers 1,2, ...,8 we make it easy to move in any direction. If we are at position (i,j) in the maze and we want to find the position (g,h) which is southwest of i,j, then we set

g   [image]  i + MOVE(6,1); h  [image]  j + MOVE(6,2)

For example, if we are at position (3,4), then position (3 + 1 = 4, 4 +(-1) = 3) is southwest.

As we move through the maze we may have the chance to go in several directions. Not knowing which one to choose, we pick one but save our current position and the direction of the last move in a list. This way if we have taken a false path we can return and try another direction. With each new location we will examine the possibilities, starting from the north and looking clockwise. Finally, in order to prevent us from going down the same path twice we use another array MARK(0:m + 1,0:n + 1) which is initially zero. MARK(i,j) is set to 1 once we arrive at that position. We assume MAZE(m,n) = 0 as otherwise there is no path to the exit. We are now ready to write a first pass at an algorithm.

set list to the maze entrance coordinates and direction north;

while list is not empty do

(i,j, mov)    coordinates and direction from front of list

while there are more moves do

(g,h)    coordinates of next move

if (g,h) = (m,n) then success

if MAZE (g,h) = 0        //the move is legal//

and MARK (g,h) = 0       //we haven't been here before//

then [MARK (g,h)    1

add (i,j, mov) to front of list

(i,j,mov)    (g,h, null)]

end

end

print no path has been found

This is not a SPARKS program and yet it describes the essential processing without too much detail. The use of indentation for delineating important blocks of code plus the use of SPARKS key words make the looping and conditional tests transparent.

What remains to be pinned down? Using the three arrays MAZE, MARK and MOVE we need only specify how to represent the list of new triples. Since the algorithm calls for removing first the most recently entered triple, this list should be a stack. We can use the sequential representation we saw before. All we need to know now is a reasonable bound on the size of this stack. Since each position in the maze is visited at most once, at most mn elements can be placed into the stack. Thus mn locations is a safe but somewhat conservative bound. In the following maze

the only path has at most   [image] m/2  [image]  (n + 1) positions. Thus mn is not too crude a bound. We are now ready to give a precise maze algorithm.

procedure PATH (MAZE, MARK, m, n, MOVE, STACK)

//A binary matrix MAZE (0:m + 1, 0:n + 1) holds the maze.

MARK (0:m + 1, 0:n + 1) is zero in spot (i,j) if MAZE (i,j) has not

yet been reached. MOVE (8,2) is a table used to change coordinates

(i,j) to one of 8 possible directions. STACK (mn,3) holds the

current path// MARK (1,1)    1

(STACK(1,1),STACK(1,2),STACK(1,3))    (1,1,2);top    1

while top   0 do

(i,j,mov)    (STACK(top,1),STACK(top,2), STACK(top,3) + 1)

top    top - 1

while mov    8 do

g    i + MOVE (mov,1); h    j + MOVE(mov,2)

if g = m and h = n

then [for p    1 to top do     //goal//

print (STACK(p,1),STACK(p,2)

end

print(i,j); print(m,n);return]

if MAZE(g,h) = 0 and MARK(g,h) = 0

then[MARK(g,h)    1

top    top + 1

(STACK(top,1),STACK(top,2),STACK(top,3))

(i,j,mov)     //save (i,j) as part of current path//

mov    0; i    g; j    h]

mov    mov + 1           //point to next direction//

end

end

print ('no path has been found')

end PATH

Now, what can we say about the computing time for this algorithm? It is interesting that even though the problem is easy to grasp, it is difficult to make any but the most trivial statement about the computing time. The reason for this is because the number of iterations of the main while loop is entirely dependent upon the given maze. What we can say is that each new position (i,j) that is visited gets marked, so paths are never taken twice. There are at most eight iterations of the inner while loop for each marked position. Each iteration of the inner while loop takes a fixed amount of time, O(1), and if the number of zeros in MAZE is z then at most z positions can get marked. Since z is bounded above by mn, the computing time is bounded by   [image]. (In actual experiments, however, the rat may be inspired by the watching psychologist and the invigorating odor from the cheese at the exit. It might reach its goal by examining far fewer paths than those examined by algorithm PATH. This may happen despite the fact that the rat has no pencil and only a very limited mental stack. It is difficult to incorporate the effect of the cheese odor and the cheering of the psychologists into a computer algorithm.) The array MARK can be eliminated altogether and MAZE(i,j) changed to 1 instead of setting MARK(i,j) to 1, but this will destroy the original maze.

3.3 EVALUATION OF EXPRESSIONS

When pioneering computer scientists conceived the idea of higher level programming languages, they were faced with many technical hurdles. One of the biggest was the question of how to generate machine language instructions which would properly evaluate any arithmetic expression. A complex assignment statement such as

X   [image]  A/B ** C + D* E - A * C

(3.1)

might have several meanings; and even if it were uniquely defined, say by a full use of parentheses, it still seemed a formidable task to generate a correct and reasonable instruction sequence. Fortunately the solution we have today is both elegant and simple. Moreover, it is so simple that this aspect of compiler writing is really one of the more minor issues.

An expression is made up of operands, operators and delimiters. The expression above has five operands: A,B,C,D, and E. Though these are all one letter variables, operands can be any legal variable name or constant in our programming language. In any expression the values that variables take must be consistent with the operations performed on them. These operations are described by the operators. In most programming languages there are several kinds of operators which correspond to the different kinds of data a variable can hold. First, there are the basic arithmetic operators: plus, minus, times, divide, and exponentiation (+,-,*,/,**). Other arithmetic operators include unary plus, unary minus and mod, ceil, and floor. The latter three may sometimes be library subroutines rather than predefined operators. A second class are the relational operators:   . These are usually defined to work for arithmetic operands, but they can just as easily work for character string data. ('CAT' is less than 'DOG' since it precedes 'DOG' in alphabetical order.) The result of an expression which contains relational operators is one of the two constants: true or false. Such all expression is called Boolean, named after the mathematician George Boole, the father of symbolic logic.

The first problem with understanding the meaning of an expression is to decide in what order the operations are carried out. This means that every language must uniquely define such an order. For instance, if A = 4, B = C = 2, D = E = 3, then in eq. 3.1 we might want X to be assigned the value

4/(2 ** 2) + (3 * 3) - (4 * 2)

= (4/4) + 9 - 8

= 2.

However, the true intention of the programmer might have been to assign X the value

(4/2) ** (2 + 3) * (3 - 4) * 2

= (4/2) ** 5* -1 * 2

= (2**5)* -2

= 32* -2

= -64.

Of course, he could specify the latter order of evaluation by using parentheses:

X   [image]  ((((A/ B) ** (C + D)) * (E - A)) * C).

To fix the order of evaluation, we assign to each operator a priority. Then within any pair of parentheses we understand that operators with the highest priority will be evaluated first. A set of sample priorities from PL/I is given in Figure 3.7.

Figure 3.7. Priority of arithmetic, Boolean and relational operators

Notice that all of the relational operators have the same priority. Similarly, exponentiation, unary minus, unary plus and Boolean negation all have top priority. When we have an expression where two adjacent operators have the same priority, we need a rule to tell us which one to perform first. For example, do we want the value of - A ** B to be understood as (-A) ** B or -(A ** B)? Convince yourself that there will be a difference by trying A = -1 and B = 2. From algebra we normally consider A ** B ** C as A ** (B ** C) and so we rule that operators in priority 6 are evaluated right-to-left. However, for expressions such as A * B/ C we generally execute left-to-right or (A * B)/C. So we rule that for all other priorities, evaluation of operators of the same priority will proceed left to right. Remember that by using parentheses we can override these rules, and such expressions are always evaluated with the innermost parenthesized expression first.

Now that we have specified priorities and rules for breaking ties we know how X   [image]  A/B ** C + D * E - A * C will be evaluated, namely as

X   [image]  ((A/(B** C)) + (D* E)) - (A * C).

How can a compiler accept such an expression and produce correct code? The answer is given by reworking the expression into a form we call postfix notation. If e is an expression with operators and operands, the conventional way of writing e is called infix, because the operators come in-between the operands. (Unary operators precede their operand.) The postfix form of an expression calls for each operator to appear after its operands. For example,

infix: A * B/C has postfix: AB * C/.

If we study the postfix form of A * B/C we see that the multiplication comes immediately after its two operands A and B. Now imagine that A * B is computed and stored in T. Then we have the division operator, /, coming immediately after its two arguments T and C.

Let us look at our previous example

infix: A/B ** C + D * E - A * C

postfix: ABC ** /DE * + AC * -

and trace out the meaning of the postfix.

Every time we compute a value let us store it in the temporary location Ti, i   [image]  1. Reading left to right, the first operation is exponentiation:

Operation       Postfix

---------       -------

T1    B **C     AT1/DE * + AC * -

T2    A/T1      T2DE * + AC * -

T3    D * E     T2T3 + AC * -

T4    T2 + T3   T4AC * -

T5    A * C     T4T5 -

T6    T4, - T5  T6

So T6 will contain the result. Notice that if we had parenthesized the expression, this would change the postfix only if the order of normal evaluation were altered. Thus, A / (B ** C) + (D * E) - A * C will have the same postfix form as the previous expression without parentheses. But (A / B) ** (C + D) * (E - A ) * C will have the postfix form AB / CD + ** EA - * C *.

Before attempting an algorithm to translate expressions from infix to postfix notation, let us make some observations regarding the virtues of postfix notation that enable easy evaluation of expressions. To begin with, the need for parentheses is elimlnated. Secondly, the priority of the operators is no longer relevant. The expression may be evaluated by making a left to right scan, stacking operands, and evaluating operators using as operands the correct number from the stack and finally placing the result onto the stack. This evaluation process is much simpler than attempting a direct evaluation from infix notation.

procedure EVAL (E)

//evaluate the postfix expression E. It is assumed that the

last character in E is an '  '. A procedure NEXT-TOKEN is

used to extract from E the next token. A token is either a

operand, operator, or '  '. A one dimensional array STACK(l:n) is

used as a stack//

top    0 // initialize STACK//

loop

x    NEXT-TOKEN (E)

case

: x = '  ' : return//answer is at top of stack//

: x is an operand: call ADD(x,STACK,n,top)

:else: remove the correct number of operands

for operator x from STACK, perform

the operation and store the result, if

any, onto the stack

end

forever

end EVAL

To see how to devise an algorithm for translating from infix to postfix, note that the order of the operands in both forms is the same. In fact, it is simple to describe an algorithm for producing postfix from infix:

1)     fully parenthesize the expression;

2)     move all operators so that they replace their corresponding right parentheses;

3)     delete all parentheses.

For example, A/B ** C + D * E - A * C when fully parenthesized yields

The arrows point from an operator to its corresponding right parenthesis. Performing steps 2 and 3 gives

ABC ** / DE * + AC * -.

The problem with this as an algorithm is that it requires two passes: the first one reads the expression and parenthesizes it while the second actually moves the operators. As we have already observed, the order of the operands is the same in infix and postfix. So as we scan an expression for the first time, we can form the postfix by immediately passing any operands to the output. Then it is just a matter of handling the operators. The solution is to store them in a stack until just the right moment and then to unstack and pass them to the output.

For example, since we want A + B * C to yield ABC * + our algorithm should perform the following sequence of stacking (these stacks will grow to the right):

Next Token  Stack  Output

----------  -----  ------

   none     empty   none

    A       empty    A

    +         +      A

    B         +      AB

At this point the algorithm must determine if * gets placed on top of the stack or if the + gets taken off. Since * has greater priority we should stack * producing

    *         + *    AB

    C         + *    ABC

Now the input expression is exhausted, so we output all remaining operators in the stack to get

ABC * +

For another example, A * (B + C) * D has the postfix form ABC + * D *, and so the algorithm should behave as

Next Token  Stack  Output

----------  -----  ------

none        empty  none

A           empty  A

*           *      A

(           *(     A

B           *(     AB

+           *(+    AB

C           *(+    ABC

At this point we want to unstack down to the corresponding left parenthesis, and then delete the left and right parentheses; this gives us:

)           *      ABC +

*           *      ABC + *

D           *      ABC + * D

done        empty  ABC + * D *

These examples should motivate the following hierarchy scheme for binary arithmetic operators and delimiters. The general case involving all the operators of figure 3.7 is left as an exercise.

Symbol      In-Stack Priority  In-Coming Priority

------      -----------------  ------------------

)           -                  -

**          3                  4

*,/         2                  2

binary +,-  1                  1

(           0                  4

Figure 3.8 Priorities of Operators for Producing Postfix

The rule will be that operators are taken out of the stack as long as their in-stack priority, isp, is greater than or equal to the in-coming priority, icp of the new operator. ISP(X) and ICP(X) are functions which reflect the table of figure 3.8.

procedure POSTFIX (E)

//convert the infix expression E to postfix. Assume the last

character of E is a '  ', which will also be the last character of

the postfix. Procedure NEXT-TOKEN returns either the next

operator, operand or delimiter--whichever comes next.

STACK (1:n) is used as a stack and the character '-  ' with

ISP('-  ') = -1 is used at the bottom of the stack. ISP and ICP

are functions.//

STACK(1)    '-  '; top    1        //initialize stack//

loop

x    NEXT-TOKEN(E)

case

:x = '  ': while top > 1 do //empty the stack//

print(STACK(top));top    top - 1

end

print ('  ')

return

:x is an operand: print (x)

:x = ')': while STACK(top)    '(' do// unstack until '('//

print (STACK(top));top    top - 1

end

top    top - 1       //delete')'//

:else while ISP(STACK(top))    ICP(x) do

print (STACK(top)); top    top - 1

end

call ADD(x,STACK,n,top)          //insert x in STACK//

end

forever

end POSTFIX

As for the computing time, the algorithm makes only one pass across the input. If the expression has n symbols, then the number of operations is proportional to some constant times n. The stack cannot get any deeper than the number of operators plus 1, but it may achieve that bound as it does for A + B * C ** D.

3.4 MULTIPLE STACKS AND QUEUES

Up to now we have been concerned only with the representation of a single stack or a single queue in the memory of a computer. For these two cases we have seen efficient sequential data representations. What happens when a data representation is needed for several stacks and queues? Let us once again limit ourselves, to sequential mappings of these data objects into an array V(1:m). If we have only 2 stacks to represent. then the solution is simple. We can use V(1) for the bottom most element in stack 1 and V(m) for the corresponding element in stack 2. Stack 1 can grow towards V(m) and stack 2 towards V(1). It is therefore possible to utilize efficiently all the available space. Can we do the same when more than 2 stacks are to be represented? The answer is no, because a one dimensional array has only two fixed points V(1) and V(m) and each stack requires a fixed point for its bottommost element. When more than two stacks, say n, are to be represented sequentially, we can initially divide out the available memory V(1:m) into n segments and allocate one of these segments to each of the n stacks. This initial division of V(1:m) into segments may be done in proportion to expected sizes of the various stacks if the sizes are known. In the absence of such information, V(1:m) may be divided into equal segments. For each stack i we shall use B(i) to represent a position one less than the position in V for the bottommost element of that stack. T(i), 1   [image]  i   [image]  n will point to the topmost element of stack i. We shall use the boundary condition B(i) = T(i) iff the i'th stack is empty. If we grow the i'th stack in lower memory indexes than the i + 1'st, then with roughly equal initial segments we have

B (i) = T (i) =   [image] m/n  [image]  (i - 1), 1   [image]  i   [image]  n

(3.2)

as the initial values of B(i) and T(i), (see figure 3.9). Stack i, 1   [image]  i   [image]  n can grow from B(i) + 1 up to B(i + 1) before it catches up with the i + 1'st stack. It is convenient both for the discussion and the algorithms to define B(n + 1) = m. Using this scheme the add and delete algorithms become:

procedure ADD(i,X)

//add element X to the i'th stack, 1    i    n//

if T(i) = B(i + 1) then call STACK-FULL (i)

T(i)    T(i) + 1

V(T(i))    X        //add X to the i'th stack//

end ADD

procedure DELETE(i,X)

//delete topmost element of stack i//

if T(i) = B(i) then call STACK-EMPTY(i)

X    V(T(i))

T(i)   T(i) - 1

end DELETE

The algorithms to add and delete appear to be a simple as in the case of only 1 or 2 stacks. This really is not the case since the STACK_FULL condition in algorithm ADD does not imply that all m locations of V are in use. In fact, there may be a lot of unused space between stacks j and j + 1 for 1   [image]  j   [image]  n and j   [image]  i (figure 3.10). The procedure STACK_FULL (i) should therefore determine whether there is any free space in V and shift stacks around so as to make some of this free space available to the i'th stack.

Several strategies are possible for the design of algorithm STACK_FULL. We shall discuss one strategy in the text and look at some others in the exercises. The primary objective of algorithm STACK_FULL is to permit the adding of elements to stacks so long as there is some free space in V. One way to guarantee this is to design STACK_FULL along the following lines:

a)     determine the least j, i < j   [image]  n such that there is free space between stacks j and j + 1, i.e., T(j) < B(j + 1). If there is such a j, then move stacks i + 1, i + 2, ...,j one position to the right (treating V(1) as leftmost and V(m) as rightmost), thereby creating a space between stacks i and i + 1.

b)     if there is no j as in a), then look to the left of stack i. Find the largest j such that 1   [image]  j < i and there is space between stacks j and j + 1, i.e., T(j) < B(j + 1). If there is such a j, then move stacks j + 1, j + 2, ...,i one space left creating a free space between stacks i and i + 1.

c)     if there is no j satisfying either the conditions of a) or b), then all m spaces of V are utilized and there is no free space.

Figure 3.9 Initial configuration for n stacks in V(1:m). All stacks are empty and memory is divided into roughly equal segments.

Figure 3.10 Configuration when stack i meets with stack i + 1 but there is still free space elsewhere in V.

The writing of algorithm STACK_FULL using the above strategy is left as an exercise. It should be clear that the worst case performance of this representation for the n stacks together with the above strategy for STACK_FULL would be rather poor. In fact, in the worst case O(m) time may be needed for each insertion (see exercises). In the next chapter we shall see that if we do not limit ourselves to sequential mappings of data objects into arrays, then we can obtain a data representation for m stacks that has a much better worst case performance than the representation described here. Sequential representations for n queues and other generalizations are discussed in the exercises.

EXERCISES

1.     Consider a railroad switching network as below

Railroad cars numbered 1,2,3 ...,n are at the right. Each car is brought into the stack and removed at any time. For instance, if n = 3, we could move 1 in, move 2 in, move 3 in and then take the cars out producing the new order 3,2,1. For n = 3 and 4 what are the possible permutations of the cars that can be obtained? Are any permutations not possible?

2.     Using a Boolean variable to distinguish between a circular queue being empty or full, write insert and delete procedures.

3.     Complete the correctness proof for the stack implementation of section 3.1.

4.     Use the queue axioms to prove that the circular queue representation of section 3.1 is correct.

5.     A double ended queue (deque) is a linear list in which additions and deletions may be made at either end. Obtain a data representation mapping a deque into a one dimensional array. Write algorithms to add and delete elements from either end of the deque.

6.     [Mystery function] Let f be an operation whose argument and result is a queue and which is defined by the axioms:

f(CREATEQ)   :: = CREATEQ

f(ADDQ(i,q)) :: = if ISEMTQ(q) then ADDQ(i,q)

else ADDQ(FRONT(q), f(DELETEQ(ADDQ(i,q))))

what does f do?

7.     A linear list is being maintained circularly in an array C(0: n - 1) with F and R set up as for circular queues.

a)     Obtain a formula in terms of F, R and n for the number of elements in the list.

b)     Write an algorithm to delete the k'th element in the list.

c)     Write an algorithm to insert an element Y immediately after the k'th element .

What is the time complexity of your algorithms for b) and c)?

8.     Let L = (a1,a2, ...,an) be a linear list represented in the array V(1:n) using the mapping: the i'th element of L is stored in V(i). Write an algorithm to make an inplace reversal of the order of elements in V. I.e., the algorithm should transform V such that V(i) contains the n - i + 1'st element of L. The only additional space available to your algorithm is that for simple variables. The input to the algorithm is V and n. How much time does your algorithm take to accomplish the reversal?

9.     a) Find a path through the maze of figure 3.5.

b) Trace out the action of procedure PATH on the maze of figure 3.5. Compare this to your own attempt in a).

10.     What is the maximum path length from start to finish in any maze of dimensions n X m?

11.     Write the postfix form of the following expressions:

a)     A ** B ** C

b)     -A + B - C + D

c)     A ** -B + C

d)     (A + B) * D + E/(F + A * D) + C

12.     Obtain isp and icp priorities for all the operators of figure 3.7 together with the delimiters '(',and')'. These priorities should be such that algorithm POSTFIX correctly generates the postfix form for all expressions made up of operands and these operators and delimiters.

13.     Use the isp and icp priorities obtained in exercise 12 to answer the following:

a)     In algorithm POSTFIX what is the maximum number of elements that can be on the stack at any time if the input expression E has n operators and delimiters?

b)     What is the answer to a) if E contains no operators of priority 6, has n operators and the depth of nesting of parentheses is at most 6?

14.     Another expression form that is easy to evaluate and is parenthesis free is known as prefix. In this way of writing expressions, the operators precede their operands. For example:

          infix                  prefix

-----------------------------------------------

        A * B/C                  /* ABC

A/B ** C + D * E - A * C  - +/A ** BC * DE * AC

A * (B + C)/D - G            -/* A + BCDG

Notice that the order of operands is not changed in going from infix to prefix .

a)     What is the prefix form of the expressions in exercise 11.

b)     Write an algorithm to evaluate a prefix expression, E (Hint: Scan E right to left and assume that the leftmost token of E is '  [image] '.)

c)     Write an algorithm to transform an infix expression E into its prefix equivalent. Assume that the input expression E begins with a '  [image] ' and that the prefix expression should begin with a '  [image] '.

What is the time complexity of your algorithms for b) and c)? How much space is needed by each of these algorithms?

15.     Write an algorithm to transform from prefix to postfix. Carefully state any assumptions you make regarding the input. How much time and space does your algorithm take?

16.     Do exercise 15 but this time for a transformation from postfix to prefix.

17.     Write an algorithm to generate fully parenthesized infix expressions from their postfix form. What is the complexity (time and space) of your algorithm?

18.     Do exercise 17 starting from prefix form.

19.     Two stacks are to be represented in an array V(1:m) as described in section 3.4. Write algorithms ADD(i.X) and DELETE(i) to add X and delete an element from stack i, 1   [image]  i   [image]  2. Your algorithms should be able to add elements to the stacks so long as there are fewer than m elements in both stacks together.

20.     Obtain a data representation mapping a stack S and a queue Q into a single array V(1:n). Write algorithms to add and delete elements from these two data objects. What can you say about the suitability of your data representation?

21.     Write a SPARKS algorithm implementing the strategy for STACK_FULL(i) outlined in section 3.4.

22.     For the ADD and DELETE algorithms of section 3.4 and the STACK_FULL(i) algorithm of exercise 21 produce a sequence of adds and deletes that will require O(m) time for each add. Use n = 2 and start from a configuration representing a full utilization of V(1:m).

23.     It has been empirically observed that most programs that get close to using all available space eventually run out of space. In the light of this observation, it seems futile to move stacks around providing space for other stacks to grow in if there is only a limited amount of space that is free. Rewrite the algorithm of exercise 21 so that the algorithm terminates if there are fewer than C free spaces. C is an empirically determined constant and is provided to the algorithm.

24.     Another strategy for the STACK_FULL(i) condition of section 3.4 is to redistribute all the free space in proportion to the rate of growth of individual stacks since the last call to STACK_FULL. This would require the use of another array LT(1:n) where LT(j) is the value of T(j) at the last call to STACK_FULL. Then the amount by which each stack has grown since the last call is T(j) - LT(j). The figure for stack i is actually T(i) - LT(i) + 1, since we are now attempting to add another element to i.

Write algorithm STACK_FULL (i) to redistribute all the stacks so that the free space between stacks j and j + 1 is in proportion to the growth of stack j since the last call to STACK_FULL. STACK_FULL (i) should assign at least 1 free location to stack i.

25.     Design a data representation sequentially mapping n queues into all array V(1:m). Represent each queue as a circular queue within V. Write algorithms ADDQ, DELETEQ and QUEUE-FULL for this representation.

26.     Design a data representation, sequentially mapping n data objects into an array V(1:m). n1 of these data objects are stacks and the remaining n2 = n - n1 are queues. Write algorithms to add and delete elements from these objects. Use the same SPACE_FULL algorithm for both types of data objects. This algorithm should provide space for the i-th data object if there is some space not currently being used. Note that a circular queue with space for r elements can hold only r - 1.

27.     [Landweber]

People have spent so much time playing card games of solitaire that the gambling casinos are now capitalizing on this human weakness. A form of solitaire is described below. Your assignment is to write a computer program to play the game thus freeing hours of time for people to return to more useful endeavors.

To begin the game, 28 cards are dealt into 7 piles. The leftmost pile has 1 card, the next two cards, and so forth up to 7 cards in the rightmost pile. Only the uppermost card of each of the 7 piles is turned face up. The cards are dealt left to right, one card to each pile, dealing to one less pile each time, and turning the first card in each round face up.

On the top-most face up card of each pile you may build in descending sequences red on black or black on red. For example, on the 9 of spades you may place either the 8 of diamonds or the 8 of hearts. All face up cards on a pile are moved as a unit and may be placed on another pile according to the bottommost face up card. For example, the 7 of clubs on the 8 of hearts may be moved as a unit onto the 9 of clubs or the 9 of spades.

Whenever a face down card is uncovered, it is turned face up. If one pile is removed completely, a face-up King may be moved from a pile (together with all cards above it) or the top of the waste pile (see below)) into the vacated space. There are four output piles, one for each suit, and the object of the game is to get as many cards as possible into the output piles. Each time an Ace appears at the top of a pile or the top of the stack it is moved into the appropriate output pile. Cards are added to the output piles in sequence, the suit for each pile being determined by the Ace on the bottom.

From the rest of the deck, called the stock, cards are turned up one by one and placed face up on a waste pile. You may always play cards off the top of the waste pile, but only one at a time. Begin by moving a card from the stock to the top of the waste pile. If there is ever more than one possible play to be made, the following order must be observed:

i)     Move a card from the top of a playing pile or from the top of the waste pile to an output pile. If the waste pile becomes empty, move a card from the stock to the waste pile.

ii)     Move a card from the top of the waste pile to the leftmost playing pile to which it can be moved. If the waste pile becomes empty move a card from the stock to the waste pile.

iii)     Find the leftmost playing pile which can be moved and place it on top of the leftmost playing pile to which it can be moved.

iv)     Try i), ii) and iii) in sequence, restarting with i) whenever a move is made.

v)     If no move is made via (i)-(iv) move a card from the stock to the waste pile and retry (i).

Only the topmost card of the playing piles or the waste pile may be played to an output pile. Once played on an output pile, a card may not be withdrawn to help elsewhere. The game is over when either

i)     all the cards have been played to the output or

ii)     the stock pile has been exhausted and no more cards can be moved

When played for money, the player pays the house $52 at the beginning and wins $5 for every card played to the output piles.

Write your program so that it will play several games, and determine your net winnings. Use a random number generator to shuffle the deck.

Output a complete record of two games in easily understood form. Include as output the number of games played and the net winning (+ or -).

Go to Chapter 4     Back to Table of Contents


Fundamentals: CHAPTER 4: LINKED LISTS

 Return to Table of Contents

CHAPTER 4: LINKED LISTS

4.1 SINGLY LINKED LISTS

In the previous chapters, we studied the representation of simple data structures using an array and a sequential mapping. These representations had the property that successive nodes of the data object were stored a fixed distance apart. Thus, (i) if the element aij of a table was stored at location Lij, then ai,j+1 was at the location Lij + c for some constant c; (ii) if the ith node in a queue was at location Li, then the i + 1- st node was at location Li + c mod n for the circular representation; (iii) if the topmost node of a stack was at location LT, then the node beneath it was at location LT - c, etc. These sequential storage schemes proved adequate given the functions one wished to perform (access to an arbitrary node in a table, insertion or deletion of nodes within a stack or queue).

However when a sequential mapping is used for ordered lists, operations such as insertion and deletion of arbitrary elements become expensive. For example, consider the following list of all of the three letter English words ending in AT:

(BAT, CAT, EAT, FAT, HAT, JAT, LAT, MAT, OAT, PAT, RAT, SAT, TAT, VAT, WAT)

To make this list complete we naturally want to add the word GAT, which means gun or revolver. If we are using an array to keep this list, then the insertion of GAT will require us to move elements already in the list either one location higher or lower. We must either move HAT, JAT, LAT, ..., WAT or else move BAT, CAT, EAT and FAT. If we have to do many such insertions into the middle, then neither alternative is attractive because of the amount of data movement. On the other hand, suppose we decide to remove the word LAT which refers to the Latvian monetary unit. Then again, we have to move many elements so as to maintain the sequential representation of the list.

When our problem called for several ordered lists of varying sizes, sequential representation again proved to be inadequate. By storing each list in a different array of maximum size, storage may be wasted. By maintaining the lists in a single array a potentially large amount of data movement is needed. This was explicitly observed when we represented several stacks, queues, polynomials and matrices. All these data objects are examples of ordered lists. Polynomials are ordered by exponent while matrices are ordered by rows and columns. In this chapter we shall present an alternative representation for ordered lists which will reduce the time needed for arbitrary insertion and deletion.

An elegant solution to this problem of data movement in sequential representations is achieved by using linked representations. Unlike a sequential representation where successive items of a list are located a fixed distance apart, in a linked representation these items may be placed anywhere in memory. Another way of saying this is that in a sequential representation the order of elements is the same as in the ordered list, while in a linked representation these two sequences need not be the same. To access elements in the list in the correct order, with each element we store the address or location of the next element in that list. Thus, associated with each data item in a linked representation is a pointer to the next item. This pointer is often referred to as a link. In general, a node is a collection of data, DATA1, ...,DATAn and links LINK1, ...,LINKm. Each item in a node is called a field. A field contains either a data item or a link.

Figure 4.1 shows how some of the nodes of the list we considered before may be represented in memory by using pointers. The elements of the list are stored in a one dimensional array called DATA. But the elements no longer occur in sequential order, BAT before CAT before EAT, etc. Instead we relax this restriction and allow them to appear anywhere in the array and in any order. In order to remind us of the real order, a second array, LINK, is added. The values in this array are pointers to elements in the DATA array. Since the list starts at DATA(8) = BAT, let us set a variable F = 8. LINK(8) has the value 3, which means it points to DATA(3) which contains CAT. The third element of the list is pointed at by LINK(3) which is EAT. By continuing in this way we can list all the words in the proper order.

Figure 4.1 Non-Sequential List Representation

We recognize that we have come to the end when LINK has a value of zero.

Some of the values of DATA and LINK are undefined such as DATA(2), LINK(2), DATA(5), LINK(5), etc. We shall ignore this for the moment.

It is customary to draw linked lists as an ordered sequence of nodes with links being represented by arrows as in figure 4.2. Notice that we do not explicitly put in the values of the pointers but simply draw arrows to indicate they are there. This is so that we reinforce in our own mind the facts that (i) the nodes do not actually reside in sequential locations, and that (ii) the locations of nodes may change on different runs. Therefore, when we write a program which works with lists, we almost never look for a specific address except when we test for zero.

Figure 4.2 Usual Way to Draw a Linked List

Let us now see why it is easier to make arbitrary insertions and deletions using a linked list rather than a sequential list. To insert the data item GAT between FAT and HAT the following steps are adequate:

(i) get a node which is currently unused; let its address be X;

(ii) set the DATA field of this node to GAT;

(iii) set the LINK field of X to point to the node after FAT which contains HAT;

(iv) set the LINK field of the node containing FAT to X.

Figure 4.3a shows how the arrays DATA and LINK will be changed after we insert GAT. Figure 4.3b shows how we can draw the insertion using our arrow notation. The new arrows are dashed. The important thing to notice is that when we insert GAT we do not have to move any other elements which are already in the list. We have overcome the need to move data at the expense of the storage needed for the second field, LINK. But we will see that this is not too severe a penalty.

Figure 4.3a Insert GAT Into DATA(5)

Figure 4.3b Insert Node GAT Into List

Now suppose we want to delete GAT from the list. All we need to do is find the element which immediately precedes GAT, which is FAT, and set LINK(9) to the position of HAT which is 1. Again, there is no need to move the data around. Even though the LINK field of GAT still contains a pointer to HAT, GAT is no longer in the list (see figure 4.4).

Figure 4.4 Delete GAT from List

From our brief discussion of linked lists we see that the following capabilities are needed to make linked representations possible:

(i) A means for dividing memory into nodes each having at least one link field;

(ii) A mechanism to determine which nodes are in use and which are free;

(iii) A mechanism to transfer nodes from the reserved pool to the free pool and vice-versa.

Though DATA and LINK look like conventional one dimensional arrays, it is not necessary to implement linked lists using them. For the time being let us assume that all free nodes are kept in a "black box" called the storage pool and that there exist subalgorithms:

(i) GETNODE(X) which provides in X a pointer to a free node but if no node is free, it prints an error message and stops;

(ii) RET(X) which returns node X to the storage pool.

In section 4.3 we shall see how to implement these primitives and also how the storage pool is maintained.

Example 4.1: Assume that each node has two fields DATA and LINK. The following algorithm creates a linked list with two nodes whose DATA fields are set to be the values 'MAT' and 'PAT' respectively. T is a pointer to the first node in this list.

procedure CREATE2(T)

call GETNODE(I)       //get an available node//

T    I; DATA(I)    'MAT'     //store information into the node//

call GETNODE(I)       //get a second available node//

LINK(T)    I      // attach first node to the second//

LINK(I)    0: DATA(I)    'PAT'

end CREATE2

The resulting list structure is

Example 4.2: Let T be a pointer to a linked list as in Example 4.1. T= 0 if the list has no nodes. Let X be a pointer to some arbitrary node in the list T. The following algorithm inserts a node with DATA field 'OAT' following the node pointed at by X.

procedure INSERT(T,X)

call GETNODE(I)

DATA(I)    'OAT'

if T = 0 then [T    I; LINK(I)    0]  //insert into empty list//

else [LINK(I)    LINK(X)      //insert after X//

LINK(X)    I]

end INSERT

The resulting list structure for the two cases T = 0 and T [image] 0 is

Example 4.3: Let X be a pointer to some node in a linked list T as in example 4.2. Let Y be the node preceding X. Y = 0 if X is the first node in T (i.e., if X = T). The following algorithm deletes node X from T.

procedure DELETE(X, Y, T)

if Y= 0 then T    LINK(T)     //remove the first node//

else LINK(Y)    LINK(X)      //remove an interior

node//

call RET(X)     //return node to storage pool//

end DELETE

4.2 LINKED STACKS AND QUEUES

We have already seen how to represent stacks and queues sequentially. Such a representation proved efficient if we had only one stack or one queue. However, when several stacks and queues co-exist, there was no efficient way to represent them sequentially. In this section we present a good solution to this problem using linked lists. Figure 4.5 shows a linked stack and a linked queue.

Figure 4.5

Notice that the direction of links for both the stack and queue are such as to facilitate easy insertion and deletion of nodes. In the case of figure 4.5(a), one can easily add a node at the top or delete one from the top. In figure 4.5(b), one can easily add a node at the rear and both addition and deletion can be performed at the front, though for a queue we normally would not wish to add nodes at the front. If we wish to represent n stacks and m queues simultaneously, then the following set of algorithms and initial conditions will serve our purpose:

T(i) = Top of ith stack    1    i    n

F(i) = Front of ith queue  1    i    m

R(i) = Rear of ith queue   1    i    m

Initial conditions:

T(i) = 0                   1    i    n

F(i) = 0                   1    i    m

Boundary conditions:

T(i) = 0     iff stack i empty

F(i) = 0     iff queue i empty

procedure ADDS(i, Y)

//add element Y onto stack i//

call GETNODE(X)

DATA(X)    Y      //store data value Y into new node//

LINK(X)    T(i)     //attach new node to top of i-th stack//

T(i)    X     //reset stack pointer//

end ADDS

procedure DELETES(i, Y)

//delete top node from stack i and set Y to be the DATA field of

this node//

if T(i) = 0 then call STACK__EMPTY

X    T(i)         //set X to top node of stack i//

Y    DATA(X)             //Y gets new data//

T(i)    LINK(X)            //remove node from top of stack i//

call RET(X)          //return node to storage pool//

end DELETES

procedure ADDQ(i,Y)

//add Y to the ith queue//

call GETNODE(X)

DATA(X)    Y: LINK(X)    0

if F(i) = 0 then [F(i)    R(i)    X]     //the queue was empty//

else [LlNK(R(i))    X;R(i)    X]  //the queue was

not empty//

end ADDQ

procedure DELETEQ(i, Y)

//delete the first node in the ith queue, set Y to its DATA field//

if F(i) = 0 then call QUEUE__EMPTY

else [X    F(i); F(i)    LINK(X)

//set X to front node//

Y    DATA(X); call RET(X)]    //remove data

and return node//

end DELETEQ

The solution presented above to the n-stack, m-queue problem is seen to be both computationaliy and conceptually simple. There is no need to shift stacks or queues around to make space. Computation can proceed so long as there are free nodes. Though additional space is needed for the link fields, the cost is no more than a factor of 2. Sometimes the DATA field does not use the whole word and it is possible to pack the LINK and DATA fields into the same word. In such a case the storage requirements for sequential and linked representations would be the same. For the use of linked lists to make sense, the overhead incurred by the storage for the links must be overriden by: (i) the virtue of being able to represent complex lists all within the same array, and (ii) the computing time for manipulating the lists is less than for sequential representation.

Now all that remains is to explain how we might implement the GETNODE and RET procedures.

4.3 THE STORAGE POOL

The storage pool contains all nodes that are not currently being used. So far we have assumed the existence of a RET and a GETNODE procedure which return and remove nodes to and from the pool. In this section we will talk about the implementation of these procedures.

The first problem to be solved when using linked allocation is exactly how a node is to be constructed The number ,and size of the data fields will depend upon the kind of problem one has. The number of pointers will depend upon the structural properties of the data and the operations to be performed. The amount of space to be allocated for each field depends partly on the problem and partly on the addressing characteristics of the machine. The packing and retrieving of information in a single consecutive slice of memory is discussed in section 4.12. For now we will assume that for each field there is a function which can be used to either retrieve the data from that field or store data into the field of a given node.

The next major consideration is whether or not any nodes will ever be returned. In general, we assume we want to construct an arbitrary number of items each of arbitrary size. In that case, whenever some structure is no longer needed, we will "erase" it, returning whatever nodes we can to the available pool. However, some problems are not so general. Instead the problem may call for reading in some data, examining the data and printing some results without ever changing the initial information. In this case a linked structure may be desirable so as to prevent the wasting of space. Since there will never be any returning of nodes there is no need for a RET procedure and we might just as well allocate storage in consecutive order. Thus, if the storage pool has n nodes with fields DATA and LINK, then GETNODE could be implemented as follows:

procedure GETNODE(I)

//I is set as a pointer to the next available node//

if AV > n then call NO__MORE__NODES

I    AV

AV    AV + 1

end GETNODE

The variable AV must initially be set to one and we will assume it is a global variable. In section 4.7 we will see a problem where this type of a routine for GETNODE is used.

Now let us handle the more general case. The main idea is to initially link together all of the available nodes in a single list we call AV. This list will be singly linked where we choose any one of the possible link fields as the field through which the available nodes are linked. This must be done at the beginning of the program, using a procedure such as:

procedure INIT(n)

//initialize the storage pool, through the LlNK field, to contain nodes

with addresses 1,2,3, ...,n and set AV to point to the first node

in this list//

for i    1 to n - 1 do

LINK(i)    i + 1

end

LlNK(n)    0

AV    1

end INIT

This procedure gives us the following list:

Figure 4.6 Initial Available Space List

Once INIT has been executed, the program can begin to use nodes. Every time a new node is needed, a call to the GETNODE procedure is made. GETNODE examines the list AV and returns the first node on the list. This is accomplished by the following:

procedure GETNODE(X)

//X is set to point to a free node if there is one on AV//

if AV = 0 then call NO__MORE__NODES

X    AV

AV    LINK(AV)

end GETNODE

Because AV must be used by several procedures we will assume it is a global variable. Whenever the programmer knows he can return a node he uses procedure RET which will insert the new node at the front of list AV. This makes RET efficient and implies that the list AV is used as a stack since the last node inserted into AV is the first node removed (LIFO).

procedure RET(X)

//X points to a node which is to be returned to the available space

list//

LINK(X)    AV

AV    X

end RET

If we look at the available space pool sometime in the middle of processing, adjacent nodes may no longer have consecutive addresses. Moreover, it is impossible to predict what the order of addresses will be. Suppose we have a variable ptr which is a pointer to a node which is part of a list called SAMPLE.

What are the permissable operations that can be performed on a variable which is a pointer? One legal operation is to test for zero, assuming that is the representation of the empty list. (if ptr = 0 then ... is a correct use of ptr). An illegal operation would be to ask if ptr = 1 or to add one to ptr (ptr   [image]  ptr + 1). These are illegal because we have no way of knowing what data was stored in what node. Therefore, we do not know what is stored either at node one or at node ptr + 1. In short, the only legal questions we can ask about a pointer variable is:

1)     Is ptr = 0 (or is ptr   [image]  0)?

2)     Is ptr equal to the value of another variable of type pointer, e.g., is ptr = SAMPLE?

The only legal operations we can perform on pointer variables is:

1)     Set ptr to zero;

2)     Set ptr to point to a node.

Any other form of arithmetic on pointers is incorrect. Thus, to move down the list SAMPLE and print its values we cannot write:

ptr    SAMPLE

while ptr    5 do

print (DATA (ptr))

ptr    ptr + 1

end

This may be confusing because when we begin a program the first values that are read in are stored sequentially. This is because the nodes we take off of the available space list are in the beginning at consecutive positions 1,2,3,4, ...,max. However, as soon as nodes are returned to the free list, subsequent items may no longer reside at consecutive addresses. A good program returns unused nodes to available space as soon as they are no longer needed. This free list is maintained as a stack and hence the most recently returned node will be the first to be newly allocated. (In some special cases it may make sense to add numbers to pointer variables, e.g., when ptr + i is the location of a field of a node starting at ptr or when nodes are being allocated sequentially, see sections 4.6 and 4.12).

4.4 POLYNOMIAL ADDITION

Let us tackle a reasonable size problem using linked lists. This problem, the manipulation of symbolic polynomials, has become a classical example of the use of list processing. As in chapter 2, we wish to be able to represent any number of different polynomials as long as their combined size does not exceed our block of memory. In general, we want to represent the polynomial

A(x) = amxem + ... + a1xe1

where the ai are non-zero coefficients with exponents ei such that em > em-1 > ... > e2 > e1 >= 0. Each term will be represented by a node. A node will be of fixed size having 3 fields which represent the coefficient and exponent of a term plus a pointer to the next term

           _____________________

          |      |      |       |

          |COEF  | EXP  | LINK  |

          |______|______|_______|

For instance, the polynomial A= 3x14 + 2x8 + 1 would be stored as

while B = 8x14 - 3x10 + 10x6 would look like

In order to add two polynomials together we examine their terms starting at the nodes pointed to by A and B. Two pointers p and q are used to move along the terms of A and B. If the exponents of two terms are equal, then the coefficients are added and a new term created for the result. If the exponent of the current term in A is less than the exponent of the current term of B, then a duplicate of the term of B is created and attached to C. The pointer q is advanced to the next term. Similar action is taken on A if EXP(p) > EXP(q). Figure 4.7 illustrates this addition process on the polynomials A and B above.

Figure 4.7 Generating the First Three Terms of C = A + B

Each time a new node is generated its COEF and EXP fields are set and it is appended to the end of the list C. In order to avoid having to search for the last node in C each time a new node is added, we keep a pointer d which points to the current last node in C. The complete addition algorithm is specified by the procedure PADD. PADD makes use of a subroutine ATTACH which creates a new node and appends it to the end of C. To make things work out neatly, C is initially given a single node with no values which is deleted at the end of the algorithm. Though this is somewhat inelegant, it avoids more computation. As long as its purpose is clearly documented, such a tactic is permissible.

procedure ATTACH(C,E,d)

//create a new term with COEF = C and EXP = E and attach it

to the node pointed at by d//

call GETNODE(I)

EXP(I)    E

COEF(I)    C

LINK(d)    I      //attach this node to the end of this list//

d    I     //move pointer d to the new last node//

end ATTACH

This is our first really complete example of the use of list processing, so it should be carefully studied. The basic algorithm is straightforward, using a merging process which streams along the two polynomials either copying terms or adding them to the result. Thus, the main while loop of lines 3-15 has 3 cases depending upon whether the next pair of exponents are =, <, or >. Notice that there are 5 places where a new term is created, justifying our use of the subroutine ATTACH.

Finally, some comments about the computing time of this algorithm. In order to carry out a computing time analysis it is first necessary to determine which operations contribute to the cost. For this algorithm there are several cost measures:

(i)     coefficient additions;

(ii)     coefficient comparisons;

(iii)     additions/deletions to available space;

procedure PADD(A,B,C)

//polynomials A and B represented as singly linked lists are

summed to form the new list named C//

1   p    A; q    B     //p,q pointers to next term of A, B//

2   call GETNODE(C); d    C       //initial node for C, returned

later//

3   while p    0 and q    0 do   //while there are more terms in

A and B//

4     case

5       : EXP(p) = EXP(q):        //equal exponents//

6          x    COEF(p) + COEF(q)

7          if x    0 then call ATTACH(x, EXP(p),d)

8          p    LINK(p); q    LINK(q)    //advance to next

terms//

9       : EXP(p) < EXP(q):

10          call ATTACH(COEF(q),EXP(q),d)

11          q    LINK(q)     //advance to next term//

12       : else: call ATTACH(COEF(p),EXP(p),d)

13               p    LINK(p)     //advance to next term of A//

14    end

15  end

16  while p    0 do    //copy remaining terms of A//

17    call ATTACH(COEF(p),EXP(p),d)

18    p    LINK(p)

19  end

20  while q    0 do    //copy remaining terms of B//

21    call ATTACH(COEF(q),EXP(q),d)

22    q    LINK(q)

23  end

24  LINK(d)    0; t    C; C    LINK(C)    //delete extra initial

node//

25  call RET(t)

26 end PADD

(iv)     creation of new nodes for C.

Let us assume that each of these four operations, if done once, takes a single unit of time. The total time taken by algorithm PADD is then determined by the number of times these operations are performed. This number clearly depends on how many terms are present in the polynomials A and B. Assume that A and B have m and n terms respectively.

A(x) = amxem + ... + a1xe1, B(x) = bnxfn + ... + b1xf1

where

ai,bi   [image]  0 and em > ... > e1   [image]  0, fn > ... > f1   [image]  0.

Then clearly the number of coefficient additions can vary as

0   [image]  coefficient additions   [image]  min {m, n}.

The lower bound is achieved when none of the exponents are equal, while the upper bound is achieved when the exponents of one polynomial are a subset of the exponents of the other.

As for exponent comparisons, one comparison is made on each iteration of the while loop of lines 3-15. On each iteration either p or q or both move to the next term in their respective polynomials. Since the total number of terms is m + n, the number of iterations and hence the number of exponent comparisons is bounded by m + n. One can easily construct a case when m + n - 1 comparisons will be necessary: e.g. m = n and

em > fn > em-1 > fn-1 > ... > f2 > en-m+2 > ... > e1 > f1.

The maximum number of terms in C is m + n, and so no more than m + n new nodes are created (this excludes the additional node which is attached to the front of C and later returned). In summary then, the maximum number of executions of any of the statements in PADD is bounded above by m + n. Therefore, the computing time is O(m + n). This means that if the algorithm is implemented and run on a computer, the time taken will be c1m + c2n + c3 where c1,c2,c3 are constants. Since any algorithm that adds two polynomials must look at each nonzero term at least once, every algorithm must have a time requirement of c'1m + c'2n + c'3. Hence, algorithm PADD is optimal to within a constant factor.

The use of linked lists is well suited to polynomial operations. We can easily imagine writing a collection of procedures for input, output addition, subtraction and multiplication of polynomials using linked lists as the means of representation. A hypothetical user wishing to read in polynomials A(x), B(x) and C(x) and then compute D(x) = A(x) * B(x) + C(x) would write in his main program:

call READ(A)

call READ(B)

call READ(C)

T    PMUL(A, B)

D    PADD(T, C)

call PRINT(D)

Now our user may wish to continue computing more polynomials. At this point it would be useful to reclaim the nodes which are being used to represent T(x). This polynomial was created only as a partial result towards the answer D(x). By returning the nodes of T(x), they may be used to hold other polynomials.

procedure ERASE(T)

//return all the nodes of T to the available space list avoiding repeated

calls to procedure RET//

if T = 0 then return

p    T

while LINK (p)    0 do      //find the end of T//

p    LINK (p)

end

LlNK (p)   AV        // p points to the last node of T//

AV    T      //available list now includes T//

end ERASE

Study this algorithm carefully. It cleverly avoids using the RET procedure to return the nodes of T one node at a time, but makes use of the fact that the nodes of T are already linked. The time required to erase T(x) is still proportional to the number of nodes in T. This erasing of entire polynomials can be carried out even more efficiently by modifying the list structure so that the LINK field of the last node points back to the first node as in figure 4.8. A list in which the last node points back to the first will be termed a circular list. A chain is a singly linked list in which the last node has a zero link field.

Figure 4.8 Circular List Representation of A = 3x14 + 2x8 + 1

Circular lists may be erased in a fixed amount of time independent of the number of nodes in the list. The algorithm below does this.

procedure CERASE(T)

//return the circular list T to the available pool//

if T = 0 then return;

X    LINK (T)

LINK(T)    AV

AV    X

end CERASE

Figure 4.9 is a schematic showing the link changes involved in erasing a circular list.

Figure 4.9 Dashes Indicate Changes Involved in Erasing a Circular List

A direct changeover to the structure of figure 4.8 however, causes some problems during addition, etc., as the zero polynomial has to be handled as a special case. To avoid such special cases one may introduce a head node into each polynomial; i.e., each polynomial, zero or non-zero, will contain one additional node. The EXP and COEF fields of this node will not be relevant. Thus the zero polynomial will have the representation:

while A = 3x14 + 2x8 + 1 will have the representation

For this circular list with head node representation the test for T = 0 may be removed from CERASE. The only changes to be made to algorithm PADD are:

(i)     at line 1 define p, q by p   [image]  LINK (A); q   [image]  LINK(B)

(ii)     at line 3: while p   [image] A and q   [image]  B do

(iii)     at line 16: while p   [image]  A do

(iv)     at line 20: while q   [image]  B do

(v)     at line 24: replace this line by LINK(d)   [image]  C

(vi)     delete line 25

Thus the algorithm stays essentially the same. Zero polynomials are now handled in the same way as nonzero polynomials.

A further simplification in the addition algorithm is possible if the EXP field of the head node is set to -1. Now when all nodes of A have been examined p = A and EXP(p) = -1. Since -1   [image]  EXP(q) the remaining terms of B can be copied by further executions of the case statement. The same is true if all nodes of B are examined before those of A This implies that there is no need for additional code to copy the remaining terms as in PADD. The final algorithm takes the following simple form.

procedure CPADD (A, B, C)

//polynomials A and B are represented as circular lists with head

nodes so that EXP(A) = EXP(B) = -1. C is returned as their sum,

represented as a circular list//

p    LINK (A); q    LINK(B)

call GETNODE(C); EXP(C)    -1         //set up head node//

d    C                                //last node in C//

loop

case

: EXP(p) = EXP(q): if EXP(p)=-1then [LINK(d)  C; return]

x    COEF(p) + COEF(q)

if x    0 then call ATTACH(x,EXP(p),d

p    LINK(p); q    LINK(q)

: EXP(p) < EXP(q): call ATTACH(COEF(q), EXP(q),d)

q    LINK(q)

: else : call ATTACH(COEF(p), EXP(p),d)

p    LINK(p)

end

forever

end CPADD

Let us review what we have done so far. We have introduced the notion of a singly linked list. Each element on the list is a node of fixed size containing 2 or more fields one of which is a link field. To represent many lists all within the same block of storage we created a special list called the available space list or AV. This list contains all nodes which are cullently not in use. Since all insertions and deletions for AV are made at the front, what we really have is a linked list being used as a stack.

There is nothing sacred about the use of either singly linked lists or about the use of nodes with 2 fields. Our polynomial example used three fields: COEF, EXP and LINK. Also, it was convenient to use circular linked lists for the purpose of fast erasing. As we continue, we will see more problems which call for variations in node structure and representation because of the operations we want to perform.

4.5 MORE ON LINKED LISTS

It is often necessary and desirable to build a variety of routines for manipulating singly linked lists. Some that we have already seen are: 1) INIT which originally links together the AV list; 2) GETNODE and 3) RET which get and return nodes to AV. Another useful operation is one which inverts a chain. This routine is especially interesting because it can be done "in place" if we make use of 3 pointers.

procedure INVERT(X)

//a chain pointed at by X is inverted so that if X = (a1, ...,am)

then after execution X = (am, ...,a1)//

p    X;q    0

while p    0 do

r    q;q    p      //r follows q; q follows p//

p    LINK(p)        //p moves to next node//

LINK(q)    r        //link q to previous node//

end

X    q

end INVERT

The reader should try this algorithm out on at least 3 examples: the empty list, and lists of length 1 and 2 to convince himself that he understands the mechanism. For a list of m   [image]  1 nodes, the while loop is executed m times and so the computing time is linear or O(m).

Another useful subroutine is one which concatenates two chains X and Y.

procedure CONCATENATE(X, Y, Z)

//X = (a1, ...,am), Y = (b1, ...,bn), m,n    0, produces a new chain

Z = (a1, ...,am,b1 , ...,bn)//

Z    X

if X = 0 then [Z    Y; return]

if Y = 0 then return

p    X

while LINK(p)    0 do      //find last node of X//

p    LINK(p)

end

LINK(p)    Y       //link last node of X to Y//

end CONCATENATE

This algorithm is also linear in the length of the first list. From an aesthetic point of view it is nicer to write this procedure using the case statement in SPARKS. This would look like:

procedure CONCATENATE(X, Y, Z)

case

: X = 0 :Z    Y

: Y = 0 : Z    X

: else : p    X; Z    X

while LINK(p)    0 do

p    LINK (p)

end

LINK(p)    Y

end

end CONCATENATE

Now let us take another look at circular lists like the one below:

A = (x1,x2,x3). Suppose we want to insert a new node at the front of this list. We have to change the LINK field of the node containing x3. This requires that we move down the entire length of A until we find the last node. It is more convenient if the name of a circular list points to the last node rather than the first, for example:

Now we can write procedures which insert a node at the front or at the rear of a circular list and take a fixed amount of time.

procedure INSERT__FRONT(A, X)

//insert the node pointed at by X to the front of the circular list

A, where A points to the last node//

if A = 0 then [A    X

LINK (X)    A]

else [LINK(X)    LINK (A)

LINK(A)   X]

end INSERT--FRONT

To insert X at the rear, one only needs to add the additional statement A   [image]  X to the else clause of INSERT_-_FRONT.

As a last example of a simple procedure for circular lists, we write a function which determines the length of such a list.

procedure LENGTH(A)

//find the length of the circular list A//

i    0

if A    0 then [ptr    A

repeat

i    i + 1; ptr    LINK(ptr)

until ptr = A ]

return (i)

end LENGTH

4.6 EQUIVALENCE RELATIONS

Let us put together some of these ideas on linked and sequential representations to solve a problem which arises in the translation of computer languages, the processing of equivalence relations. In FORTRAN one is allowed to share the same storage among several program variables through the use of the EQUIVALENCE statement. For example, the following pair of Fortran statements

DIMENSION A(3), B(2,2), C(6)

EQUIVALENCE (A(2), B(1,2), C(4)), (A(1),D), (D,E,F), (G,H)

would result in the following storage assignment for these variables:

As a result of the equivalencing of A (2), B(1,2) and C(4), these were assigned the same storage word 4. This in turn equivalenced A(1), B(2,1) and C(3), B(1,1) and C(2), and A(3), B(2,2) and C(5). Because of the previous equivalence group, the equivalence pair (A(1), D) also resulted in D sharing the same space as B(2,1) and C(3). Hence, an equivalence of the form (D, C(5)) would conflict with previous equivalences, since C(5) and C(3) cannot be assigned the same storage. Even though the sum of individual storage requirements for these variables is 18, the memory map shows that only 7 words are actually required because of the overlapping specified by the equivalence groups in the EQUIVALENCE statement. The functions to be performed during the processing of equivalence statements, then, are:

(i)     determine whether there are any conflicts;

(ii)     determine the total amount of storage required;

(iii)     determine the relative address of all variables (i.e., the address of A(1), B(1,1), C(1), D, E, F, G and H in the above example).

In the text we shall solve a simplified version of this problem. The extension to the general Fortran equivalencing problem is fairly straight-forward and appears as an exercise. We shall restrict ourselves to the case in which only simple variables are being equivalenced and no arrays are allowed. For ease in processing, we shall assume that all equivalence groups are pairs of numbers (i,j), where if EQUIVALENCE(A,F) appears then i,j are the integers representing A and F. These can be thought of as the addresses of A and F in a symhol table. Furthermore, it is assumed that if there are n variables, then they ale represented by the numbers 1 to n.

The FORTRAN statement EQUIVALENCE specifies a relationship among addresses of variables. This relation has several properties which it shares with other relations such as the conventional mathematical equals. Suppose we denote an arbitrary relation by the symbol   [image]  and suppose that:

(i)     For any variable x, x   [image]  x, e.g. x is to be assigned the same location as itself. Thus   [image]  is reflexive.

(ii)     For any two variables x and y, if x   [image]  y then y   [image]  x, e.g. assigning y the same location as x is the same as assigning x the same location as y. Thus, the relation   [image]  is symmetric.

(iii)     For any three variables x, y and z, if x   [image]  y and y   [image]  z then x   [image]  z, e.g. if x and y are to be assigned the same location and y and z are also to be assigned the same location, then so also are x and z. The relation   [image]  is transitive.

Definition: A relation,   [image], over a set S, is said to be an equivalence relation over S iff it is symmetric, reflexive and transitive over S.

Examples of equivalence relations are numerous. For example, the "equal to" (=) relationship is an equivalence relation since: (i) x = x, (ii) x = y implies y = x, and (iii) x = y and y = z implies x = z. One effect of an equivalence relation is to partition the set S into equivalence classes such that two members x and y of S are in the same equivalence class iff x   [image]  y. For example, if we have 12 variables numbered 1 through 12 and the following equivalences were defined via the EQUIVALENCE statement:

1    5, 4    2, 7    11, 9    10, 8    5, 7    9, 4    6, 3    12 and 12    1

then, as a result of the reflexivity, symmetry and transitivity of the relation =, we get the following partitioning of the 12 variables into 3 equivalence classes:

{1, 3, 5, 8, 12}; {2, 4, 6}; {7, 9, 10, 11}.

So, only three words of storage are needed for the 12 variables. In order to solve the FORTRAN equivalence problem over simple variables, all one has to do is determine the equivallence classes. The number of equivalence classes ii the number of words of storage to be allocated and the members of the same equivalence class are allocated the same word of storage.

The algorithm to determine equivalence classes works in essentially two phases. In the first phase the equivalence pairs (i,j) are read in and stored somewhere. In phase two we begin at one and find all pairs of the form (1,j). The values 1 and j are in the same class. By transitivity, all pairs of the form (j,k) imply k is in the same class. We continue in this way until the entire equivalence class containing one has been found, marked and printed. Then we continue on.

The first design for this algorithm might go this way:

procedure EQUIVALENCE (m,n)

initialize

for k    1 to m do

read the next pair (i,j)

process this pair

end

initialize for output

repeat

output a new equivalence class

until done

end EQUIVALENCE

The inputs m and n represent the number of related pairs and the number of objects respectively. Now we need to determine which data structure should be used to hold these pairs. To determine this we examine the operations that are required. The pair (i,j) is essentially two random integers in the range 1 to n. Easy random access would dictate an array, say PAIRS (1: n, 1: m). The i-th row would contain the elements j which are paired directly to i in the input. However, this would potentially be very wasteful of space since very few of the array elements would be used. It might also require considerable time to insert a new pair, (i,k), into row i since we would have to scan the row for the next free location or use more storage.

These considerations lead us to consider a linked list to represent each row. Each node on the list requires only a DATA and LINK field. However, we still need random access to the i-th row so a one dimensional array, SEQ(1:n) can be used as the headnodes of the n lists. Looking at the second phase of the algorithm we need a mechanism which tells us whether or not object i has already been printed. An array of bits, BIT(1:n) can be used for this. Now we have the next refinement of the algorithm.

procedure EQUIVALENCE (m,n)

declare SEQ(1:n), DATA(1:2m), LINK(1:2m), BIT(1:n)

initialize BIT, SEQ to zero

for k   1 to m do

read the next pair (i,j)

put j on the SEQ(i) list

put i on the SEQ(j) list

end

index    1

repeat

if BIT(index) = 0

then [BIT(index)    1

output this new equivalence class]

index    index + 1

until index > n

end

Let us simulate the algorithm as we have it so far, on the previous data set. After the for loop is completed the lists will look like this.

For each relation i   [image]  j, two nodes are used. SEQ(i) points to a list of nodes which contains every number which is directly equivalenced to i by an input relation.

In phase two we can scan the SEQ array and start with the first i, 1   [image]  i   [image]  n such that BIT(i) = 0. Each element in the list SEQ(i) is printed. In order to process the remaining lists which, by transitivity, belong in the same class as i a stack of their nodes is created. This is accomplished by changing the LINK fields so they point in the reverse direction. The complete algorithm is now given.

procedure EQUIVALENCE (m,n)

//Input: m, the number of equivalence pairs

n, the number of variables

Output: variables 1, ...,n printed in equivalence classes//

declare SEQ(1:n), BIT(1:n)

DATA(1: 2m), LINK(1: 2m);

for i    1 to n do SEQ(i)   BIT(i)    0 end

av   1

for k    1 to m do      //phase 1: process all input//

read next equivalence pair (i, j)

DATA(av)    j; LlNK(av)    SEQ(i)     //add j to list i//

SEQ(i)    av; av    av + 1

DATA(av)   i; LINK(av)    SEQ(j)      //add i to list j//

SEQ(j)    av; av    av + 1

end

index    1

repeat     //phase 2: output all classes//

if BIT (index) = 0

then [print ('A new class', index)

BIT(index)    1        //mark class as output//

ptr    SEQ(index); top    0        //initialize stack//

loop      //find the entire class//

while ptr    0 do       //process a list//

j    DATA (ptr)

if BIT(j) = 0

then [print (j); BIT(j)    1

t    LINK (ptr); LINK (ptr)    top

top    ptr; ptr    t]

else ptr  LINK (ptr)

end

if top = 0 then exit    //stack empty//

ptr    SEQ(DATA(top))

top    LINK (top)

forever]

index    index + 1

until index > n

end EQUIVALENCE

Analysis of Algorithm EQUIVALENCE

The initialization of SEQ and BIT takes O(n) time. The processing of each input pair in phase 1 takes a constant amount of time. Hence, the total time for this phase is O(m). In phase 2 each node is put onto the linked stack at most once. Since there are only 2m nodes and the repeat loop is executed n times, the time for this phase is O(m + n). Hence, the overall computing time is O(m + n). Any algorithm which processes equivalence relations must look at all the m equivalence pairs and also at all the n variables at least once. Thus, there can be no algorithm with a computing time less than O(m + n). This means that the algorithm EQUIVALENCE is optimal to within a constant factor. Unfortunately,the space required by the algorithm is also O(m + n). In chapter 5 we shall see an alternative solution to this problem which requires only O(n) space.

4.7 SPARSE MATRICES

In Chapter 2, we saw that when matrices were sparse (i.e. many of the entries were zero), then much space and computing time could be saved if only the nonzero terms were retained explicitly. In the case where these nonzero terms did not form any "nice" pattern such as a triangle or a band, we devised a sequential scheme in which each nonzero term was represented by a node with three fields: row, column and value. These nodes were sequentially organized. However, as matrix operations such as addition, subtraction and multiplication are performed, the number of nonzero terms in matrices will vary, matrices representing partial computations (as in the case of polynomials) will be created and will have to be destroyed later on to make space for further matrices. Thus, sequential schemes for representing sparse matrices suffer from the same inadequacies as similar schemes for polynomials. In this section we shall study a very general linked list scheme for sparse matrix representation. As we have already seen, linked schemes facilitate efficient representation of varying size structures and here, too, our scheme will overcome the aforementioned shortcomings of the sequential representation studied in Chapter 2.

In the data representation we shall use, each column of a sparse matrix will be represented by a circularly linked list with a head node. In addition, each row will also be a circularly linked list with a head node. Each node in the structure other than a head node will represent a nonzero term in the matrix A and will be made up of five fields:

ROW, COL, DOWN, RIGHT and VALUE. The DOWN field will be used to link to the next nonzero element in the same column, while the RIGHT field will be used to link to the next nonzero element in the same ROW. Thus, if aij   [image]  0, then there will be a node with VALUE field aij, ROW field i and COL field j. This node will be linked into the circular linked list for row i and also into the circular linked list for column j. It will, therefore, be a member of two lists at the same time .

In order to avoid having nodes of two different sizes in the system, we shall assume head nodes to be configured exactly as nodes being used to represent the nonzero terms of the sparse matrix. The ROW and COL fields of head nodes will be set to zero (i.e. we assume that the rows and columns of our matrices have indices >0). Figure 4.12 shows the structure obtained for the 6 x 7 sparse matrix, A, of figure 4.11.

Figure 4.10 Node Structure for Sparse Matrix Representation

Figure 4.11 6 x 7 Sparse Matrix A

For each nonzero term of A, we have one five field node which is in exactly one column list and one row list. The head nodes are marked Hl-H7. As can be seen from the figure, the VALUE field of the head nodes for each column list is used to link to the next head node while the DOWN field links to the first nonzero term in that column (or to itself in case there is no nonzero term in that column). This leaves the RIGHT field unutilized. The head nodes for the row lists have the same ROW and COL values as the head nodes for the column lists. The only other field utilized by the row head nodes is the RIGHT field which is not used in the column head nodes. Hence it is possible to use the same node as the head node for row i as for column i. It is for this reason that the row and column head nodes have the same labels. The head nodes themselves, linked through the VALUE field, form a circularly linked list with a head node pointed to by A. This head node for the list of row and column head nodes contains the dimensions of the matrix. Thus, ROW(A) is the number of rows in the matrix A while COL(A) is the number of columns. As in the case of polynomials, all references to this matrix are made through the variable A. If we wish to represent an n x m sparse matrix with r nonzero terms, then the number of nodes needed is r + max {n,m} + 1. While each node may require 2 to 3 words of memory (see section 4.12), the total storage needed will be less than nm for sufficiently small r.

Figure 4.12 Linked Representation of the Sparse Matrix A

Having arrived at this representation for sparse matrices, let us see how to manipulate it to perform efficiently some of the common operations on matrices. We shall present algorithms to read in a sparse matrix and set up its linked list representation and to erase a sparse matrix (i.e. to return all the nodes to the available space list). The algorithms will make use of the utility algorithm GETNODE(X) to get nodes from the available space list.

To begin with let us look at how to go about writing algorithm MREAD(A) to read in and create the sparse matrix A. We shall assume that the input consists of n, the number of rows of A, m the number of columns of A, and r the number of nonzero terms followed by r triples of the form (i,j,aij). These triples consist of the row, column and value of the nonzero terms of A. It is also assumed that the triples are ordered by rows and that within each row, the triples are ordered by columns. For example, the input for the 6 X 7 sparse matrix of figure 4.11, which has 7 nonzero terms, would take the form: 6,7,7; 1,3,11;1,6,13;2,1,12;2,7,14;3,2,-4;3,6,-8;6,2,-9. We shall not concern ourselves here with the actual format of this input on the input media (cards, disk, etc.) but shall just assume we have some mechanism to get the next triple (see the exercises for one possible input format). The algorithm MREAD will also make use of an auxiliary array HDNODE, which will be assumed to be at least as large as the largest dimensioned matrix to be input. HDNODE(i) will be a pointer to the head node for column i, and hence also for row i. This will permit us to efficiently access columns at random while setting up the input matrix. Algorithm MREAD proceeds by first setting up all the head nodes and then setting up each row list, simultaneously building the column lists. The VALUE field of headnode i is initially used to keep track of the last node in column i. Eventually, in line 27, the headnodes are linked together through this field.

Analysis of Algorithm MREAD

Since GETNODE works in a constant amount of time, all the head nodes may be set up in O(max {n,m}) time, where n is the number of rows and m the number of columns in the matrix being input. Each nonzero term can be set up in a constant amount of time because of the use of the variable LAST and a random access scheme for the bottommost node in each column list. Hence, the for loop of lines 9-20 can be carried out in O(r) time. The rest of the algorithm takes O(max {n,m}) time. The total time is therefore O(max {n,m} + r) = O(n + m + r). Note that this is asymptotically better than the input time of O(nm) for an n x m matrix using a two-dimensional array, but slightly worse than the sequential sparse method of section 2.3.

Before closing this section, let us take a look at an algorithm to return all nodes of a sparse matrix to the available space list.

procedure MERASE(A)

//return all nodes of A to available space list. Assume that the available

space list is a singly linked list linked through the field RIGHT

with AV pointing to the first node in this list.//

RIGHT(A)    AV; AV    A; NEXT    VALUE(A)

while NEXT    A do    //erase circular lists by rows//

T    RIGHT(NEXT)

RIGHT(NEXT)    AV

AV    T

NEXT    VALUE(NEXT)

end

end MERASE

Analysis of Algorithm MERASE

Since each node is in exactly one row list, it is sufficient to just return all the row lists of the matrix A. Each row list is circularly linked through the field RIGHT. Thus, nodes need not be returned one by one as a circular list can be erased in a constant amount of time. The computing time for the algorithm is readily seen to be 0(n + m). Note that even if the available space list had been linked through the field DOWN, then erasing could still have been carried out in 0(n + m) time. The subject of manipulating these matrix structures is studied further in the exercises. The representation studied here is rather general. For most applications this generality is not needed. A simpler representation resulting in simpler algorithms is discussed in the exercises.

procedure MREAD(A)

//read in a matrix A and set up its internal representation as

discussed previously. The triples representing nonzero terms

are assumed ordered by rows and within rows by columns.

An auxiliary array HDNODE is used.//

1   read (n,m,r)    //m, n are assumed positive. r is the number

of nonzero elements//

2   p    max {m,n}

3   for i    1 to p do    //get p headnodes for rows and

columns//

4     call GETNODE(X); HDNODE(i)    X

5     ROW(X)    COL(X)    0

6     RIGHT(X)    VALUE(X)    X     //these fields point to

themselves//

7   end

8   current_row    1; LAST    HDNODE(1)

9   for i    1 to r do

10    read (rrow, ccol, val)    //get next triple//

11    if rrow > current__row     //current row is done; close it and

begin another//

12       then [RIGHT(LAST)    HDNODE(current__row)

13            current__row    rrow; LAST    HDNODE(rrow)]

//LAST points to rightmost node//

14    call GETNODE(X)

15    ROW(X)    rrow; COL(X)    ccol; VALUE(X)    val

//store triple into new node//

16    RIGHT(LAST)    X; LAST    X     //link into row list//

17    DOWN(VALUE(HDNODE(ccol)))    X;

//link into column list//

18    VALUE(HDNODE(ccol))    X

19

20    end

21    //close last row//

if r    0 then RIGHT(LAST)    HDNODE(current__row)

22    for i    1 to m do     //close all column lists//

23      DOWN(VALUE(HDNODE(i))) HDNODE(i)

24    end

25    //set up list of headnodes linked through VALUE field//

26    call GETNODE(A): ROW(A)    n; COL(A)    m

//set up headnode of matrix//

27    for i    1 to p - 1 do VALUE(HDNODE(i))    HDNODE(i + 1)

end

28    if p = 0 then VALUE(A)    A

29             else [VALUE(HDNODE(p))    A

VALUE(A)    HDNODE(1)]

30 end MREAD

4.8 DOUBLY LINKED LISTS AND DYNAMIC STORAGE MANAGEMENT

So far we have been working chiefly with singly linked linear lists. For some problems these would be too restrictive. One difficulty with these lists is that if we are pointing to a specific node, say P, then we can easily move only in the direction of the links. The only way to find the node which precedes P is to start back at the beginning of the list. The same problem arises when one wishes to delete an arbitrary node from a singly linked list. As can be seen from example 4.3, in order to easily delete an arbitrary node one must know the preceding node. If we have a problem where moving in either direction is often necessary, then it is useful to have doubly linked lists. Each node now has two link fields, one linking in the forward direction and one in the backward direction.

A node in a doubly linked list has at least 3 fields, say DATA, LLINK (left link) and RLINK (right link). A doubly linked list may or may not be circular. A sample doubly linked circular list with 3 nodes is given in figure 4.13. Besides these three nodes a special node has been

Figure 4.13 Doubly Linked Circular List with Head Node

added called a head node. As was true in the earlier sections, head nodes are again convenient for the algorithms. The DATA field of the head node will not usually contain information. Now suppose that P points to any node in a doubly linked list. Then it is the case that

P = RLINK (LLINK(P)) = LLINK (RLINK(P)).

This formula reflects the essential virtue of this structure, namely, that one can go back and forth with equal ease. An empty list is not really empty since it will always have its head node and it will look like

Now to work with these lists we must be able to insert and delete nodes. Algorithm DDLETE deletes node X from list L.

procedure DDLETE(X, L)

if X = L then call NO__MORE__NODES

//L is a list with at least one node//

RLINK(LLINK(X))    RLINK(X)

LLINK(RLINK(X))    LLINK(X)

call RET(X)

end DDLETE

X now points to a node which is no longer part of list L. Let us see how the method works on a doubly linked list with only a single node.

Even though the RLINK and LLINK fields of node X still point to the head node, this node has effectively been removed as there is no way to access X through L.

Insertion is only slightly more complex.

procedure DINSERT (P, X)

//insert node P to the right of node X//

LLINK(P)    X         //set LLINK and RLINK fields of node P//

RLINK(P)    RLINK(X)

LLINK(RLINK(X))    P

RLINK(X)    P

end DINSERT

In the next section we will see an important problem from operating systems which is nicely solved by the use of doubly linked lists.

Dynamic Storage Management

In a multiprocessing computer environment, several programs reside in memory at the same time. Different programs have different memory requirements. Thus, one program may require 60K of memory, another 100K, and yet another program may require 300K. Whenever the operating system needs to request memory, it must be able to allocate a block of contiguous storage of the right size. When the execution of a program is complete, it releases or frees the memory block allocated to it and this freed block may now be allocated to another program. In a dynamic environment the request sizes that will be made are not known ahead of time. Moreover, blocks of memory will, in general, be freed in some order different from that in which they were allocated. At the start of the computer system no jobs are in memory and so the whole memory, say of size M words, is available for allocation to programs. Now, jobs are submitted to the computer and requests are made for variable size blocks of memory. Assume we start off with 100,000 words of memory and five programs P1, P2, P3, P4 and P5 make requests of size 10,000, 15,000, 6,000, 8,000 and 20,000 respectively. Figure 4.14 indicates the status of memory after storage for P5 has been allocated. The unshaded area indicates the memory that is currently not in use. Assume that programs P4 and P2 complete execution, freeing the memory used by them. Figure 4.15 show the status of memory after the blocks for P2 and P4 are freed. We now have three blocks of contiguous memory that are in use and another three that are free. In order to make further allocations, it is necessary to keep track of those blocks that are not in use. This problem is similar to the one encountered in the previous sections where we had to maintain a list of all free nodes. The difference between the situation then and the one we have now is that the free space consists of variable size blocks or nodes and that a request for a block of memory may now require allocation of only a portion of a node rather than the whole node. One of the functions of an operating system is to maintain a list of all blocks of storage currently not in use and then to allocate storage from this unused pool as required. One can once again adopt the chain structure used earlier to maintain the available space list. Now, in addition to linking all the free blocks together, it is necessary to retain information regarding the size of each block in this list of free nodes. Thus, each node on the free list has two fields in its first word, i.e., SIZE and LINK. Figure 4.16 shows the free list corresponding to figure 4.15. The use of a head node simplifies later algorithms.

Figure 4.14 Memory After Allocation to P1-P5

Figure 4.15 Status of Memory After Completion of P2 and P4

If we now receive a request for a block of memory of size N, then it is necessary to search down the list of free blocks finding the first block of size   [image]  N and allocating N words out of this block. Such an allocation strategy is called first fit. The algorithm below makes storage allocations using the first fit strategy. An alternate strategy, best fit, calls for finding a free block whose size is as close to N as possible, but not less than N. This strategy is examined in the exercises.

Figure 4.16 Free List with Head Node Corresponding to figure 4.15

procedure FF(n, p)

//AV points to the available space list which is searched for a node

of size at least n. p is set to the address of a block of size n

that may be allocated. If there is no block of that size then p = 0.

It is assumed that the free list has a head node with SIZE field = 0//

p    LINK(AV); q    AV

while p    0 do

if SIZE(p)    n then [SIZE(p)    SIZE(p) - n

if SIZE(p) = 0 then LINK(q)    LINK(p)

else p    p + SIZE(p)

return]

q    p; p    LINK (p)

end

//no block is large enough//

end FF

This algorithm is simple enough to understand. In case only a portion of a free block is to be allocated, the allocation is made from the bottom of the block. This avoids changing any links in the free list unless an entire block is allocated. There are, however, two major problems with FF. First, experiments have shown that after some processing time many small nodes are left in the available space list, these nodes being smaller than any requests that would be made. Thus, a request for 9900 words allocated from a block of size 10,000 would leave behind a block of size 100, which may be smaller than any requests that will be made to the system. Retaining these small nodes on the available space list tends to slow down the allocation process as the time needed to make an allocation is proportional to the number of nodes on the available space list. To get around this, we choose some suitable constant   [image]  such that if the allocation of a portion of a node leaves behind a node of size <  [image] , then the entire node is allocated. I.e., we allocate more storage than requested in this case. The second problem arises from the fact that the search for a large enough node always begins at the front of the list. As a result of this, all the small nodes tend to collect at the front so that it is necessary to examine several nodes before an allocation for larger blocks can be made. In order to distribute small nodes evenly along the list, one can begin searching for a new node from a different point in the list each time a request is made. To implement this. the available space list is maintained as a circular list with a head node of size zero. AV now points to the last node from which an allocation was made. We shall see what the new allocation algorithm looks like after we discuss what has to be done to free a block of storage.

The second operation is the freeing of blocks or returning nodes to AV. Not only must we return the node but we also want to recognize if its neighbors are also free so that they can be coalesced into a single block. Looking back at figure 4.15, we see that if P3 is the next program to terminate, then rather than just adding this node onto the free list to get the free list of figure 4.17, it would be better to combine the adjacent free blocks corresponding to P2 and P4, obtaining the free list of figure 4.18. This combining of adjacent free blocks to get bigger free blocks is necessary. The block allocation algorithm splits big blocks while making allocations. As a result, available block sizes get smaller and smaller. Unless recombination takes place at some point, we will no longer be able to meet large requests for memory.

Figure 4.17 Available Space List When Adjacent Free Blocks Are Not Coalesced.

Figure 4.18 Available Space List When Adjacent Free Blocks Are Coalesced.

With the structure we have for the available space list, it is not easy to determine whether blocks adjacent to the block (n, p) (n = size of block and p = starting location) being returned are free. The only way to do this, at present, is to examine all the nodes in AV to determine whether:

(i)     the left adjacent block is free, i.e., the block ending at p - 1;

(ii)     the right adjacent block is free, i.e., the block beginning at p + n.

In order to determine (i) and (ii) above without searching the available space list, we adopt the node structure of figure 4.19 for allocated and free nodes:

Figure 4.19

The first and last words of each block are reserved for allocation information. The first word of each free block has four fields: LLINK, RLINK, TAG and SIZE. Only the TAG and SIZE field are important for a block in use. The last word in each free block has two fields: TAG and UPLINK. Only the TAG field is important for a block in use. Now by just examining the tag at p - 1 and p + n one can determine whether the adjacent blocks are free. The UPLINK field of a free block points to the start of the block. The available space list will now be a doubly linked circular list, linked through the fields LLINK and RLINK. It will have a head node with SIZE = 0. A doubly linked list is needed, as the return block algorithm will delete nodes at random from AV. The need for UPLINK will become clear when we study the freeing algorithm. Since the first and last nodes of each block have TAG fields, this system of allocation and freeing is called the Boundary Tag method. It should be noted that the TAG fields in allocated and free blocks occupy the same bit position in the first and last words respectively. This is not obvious from figure 4.19 where the LLINK field precedes the TAG field in a free node. The labeling of fields in this figure has been done so as to obtain clean diagrams for the available space list. The algorithms we shall obtain for the boundary tag method will assume that memory is numbered 1 to m and that TAG(0) = TAG(m + 1) = 1. This last requirement will enable us to free the block beginning at 1 and the one ending at m without having to test for these blocks as special cases. Such a test would otherwise have been necessary as the first of these blocks has no left adjacent block while the second has no right adjacent block. While the TAG information is all that is needed in an allocated block, it is customary to also retain the size in the block. Hence, figure 4.19 also includes a SIZE field in an allocated block.

Before presenting the allocate and free algorithms let us study the initial condition of the system when all of memory is free. Assuming memory begins at location 1 and ends at m, the AV list initially looks like:

While these algorithms may appear complex, they are a direct consequence of the doubly linked list structure of the available space list and also of the node structure in use. Notice that the use of a head node eliminates the test for an empty list in both algorithms and hence simplifies them. The use of circular linking makes it easy to start the search for a large enough node at any point in the available space list. The UPLINK field in a free block is needed only when returning a block whose left adjacent block is free (see lines 18 and 24 of algorithm FREE). The readability of algorithm FREE has been greatly enhanced by the use of the case statement. In lines 20 and 27 AV is changed so that it always points to the start of a free block rather than into

procedure ALLOCATE (n,p)

//Use next fit to allocate a block of memory of size at least

n, n > 0. The available space list is maintained as described

above and it is assumed that no blocks of size <    are to

be retained. p is set to be the address of the first word in

the block allocated. AV points to a node on the available list.//

1     p    RLINK(AV)          //begin search at p//

2    repeat

3      if SIZE (p)    n then      //block is big enough//

4         [diff    SlZE(p) - n

5        if diff <    then   //allocate whole block//

6          [RLINK (LLINK(p))    RLINK(p)   //delete node

from AV//

7           LLINK(RLlNK(p))    LLINK(p)

8           TAG(p)    TAG(p + SlZE(p) - 1)    1  //set

tags//

9           AV    LLINK(p)        //set starting point of next

search//

10           return]

11                    else      //allocate lower n words//

12          [SIZE(p)    diff

13           UPLINK (p + diff - 1)    p

14           TAG(p + diff - 1)    0      //set upper portion as

unused//

15           AV    p       //position for next search//

16           p    p + diff      //set p to point to start of allocated

block//

17           SlZE(p)    n

18           TAG(p)    TAG(p + n - 1)    1

//set tags for allocated block//

19           return]]

20        p    RLlNK(p)      //examine next node on list//

21   until p = RLINK(AV)

22   //no block large enough//

23   p    0;

24 end ALLOCATE

procedure FREE(p)

//return a block beginning at p and of size SlZE(p)//

1    n    SlZE(p)

2    case

3      :TAG(p - 1) = 1 and TAG(p + n) = 1:

//both adjacent blocks in use//

4        TAG(p)    TAG(p + n - 1)    0      //set up a free

block//

5        UPLINK (p + n - 1)    p

6        LLINK(p)    AV; RLINK (p)    RLINK (AV)

//insert at right of AV//

7        LLINK (RLlNK(p))    p; RLINK(AV)    p

8      :TAG(p + n) = 1 and TAG(p - 1) = 0:   //only left block

free//

9        q    UPLINK (p - 1 )   //start of left block//

10        SIZE(q)    SlZE(q) + n

11        UPLINK(p + n - 1)    q; TAG(p + n - 1)    0

12      :TAG(p + n) = 0 and TAG(p - 1) = 1:

//only right adjacent block free//

13        RLINK(LLINK(p + n))    p           //replace block

beginning//

14        LLINK(RLINK(p + n))    p           //at p + n by one//

15        LLINK(p)    LLINK(p + n)           //beginning at p//

16        RLINK(p)    RLINK(p + n)

17        SlZE(p)    n + SlZE(p + n)

18        UPLINK(p + SIZE(p) - 1)    p

19        TAG(p)    0

20        AV    p

21      :else: //both adjacent blocks free//

22             //delete right free block from AV list//

  RLINK (LLINK(p + n))    RLINK (p + n)

23              LLINK(RLINK(p + n))LLINK(p + n)

24              q  UPLINK(p - 1)   //start of left free

block//

25              SIZE(q)    SIZE(q) + n + SIZE(p + n)

26              UPLINK(q + SIZE(q) - 1)    q

27              AV    LLINK(p + n)

28   end

29 end FREE

the middle of a free block. One may readily verify that the algorithms work for special cases such as when the available space list contains only the head node.

The best way to understand the algorithms is to simulate an example. Let us start with a memory of size 5000 from which the following allocations are made: r1 = 300, r2 = 600, r3 = 900, r4 = 700, r5 = 1500 and r6 = 1000. At this point the memory configuration is as in figure 4.20. This figure also depicts the different blocks of storage and the available space list. Note that when a portion of a free block is allocated, the allocation is made from the bottom of the block so as to avoid unnecessary link changes in the AV list. First block r1 is freed. Since TAG(5001) = TAG(4700) = 1, no coalescing takes place and the block is inserted into the AV list (figure 4.21a). Next, block r4 is returned. Since both its left adjacent block (r5) and its right adjacent block (r3) are in use at this time (TAG(2500) = TAG(3201) =1), this block is just inserted into the free list to get the configuration of figure 4.21b. Block r3 is next returned. Its left adjacent block is free, TAG(3200) = 0; but its right adjacent block is not, TAG(4101) = 1. So, this block is just attached to the end of its adjacent free block without changing any link fields (figure 4.21c). Block r5 next becomes free. TAG(1000) = 1 and TAG(2501) = 0 and so this block is coalesced with its right adjacent block which is free and inserted into the spot previously occupied by this adjacent free block (figure 4.21(d)). r2 is freed next. Both its upper and lower adjacent blocks are free. The upper block is deleted from the free space list and combined with r2. This bigger block is now just appended to the end of the free block made up of r3, r4 and r5 (figure 4.21(e)).

Figure 4.20

Figure 4.21 Freeing of Blocks in Boundary Tag System

Figure 4.21 Freeing of Blocks in Boundary Tag System (contd.)

Figure 4.21 Freeing of Blocks in Boundary Tag System (contd.)

Figure 4.21 Freeing of Blocks in Boundary Tag System (contd.)

As for the computational complexity of the two algorithms, one may readily verify that the time required to free a block of storage is independent of the number of free blocks in AV. Freeing a block takes a constant amount of time. In order to accomplish this we had to pay a price in terms of storage. The first and last words of each block in use are reserved for TAG information. Though additional space is needed to maintain AV as a doubly linked list, this is of no consequence as all the storage in AV is free in any case. The allocation of a block of storage still requires a search of the AV list. In the worst case all free blocks may be examined.

An alternative scheme for storage allocation, the Buddy System, is investigated in the exercises.

4.9 GENERALIZED LISTS

In Chapter 3, a linear list was defined to be a finite sequence of n   [image]  0 elements,   [image] 1,...,  [image] n, which we write as A = (  [image] 1, ...,  [image] n). The elements of a linear list are restricted to be atoms and thus the only structural property a linear list has is the one of position, i.e.   i precedes   [image] i+1, 1   [image]  i < n. It is sometimes useful to relax this restriction on the elements of a list, permitting them to have a structure of their own. This leads to the notion of a generalized list in which the elements   [image] i, 1   [image]  i   [image]  n may be either atoms or lists.

Definition. A generalized list, A, is a finite sequence of n > 0 elements,   [image] 1, ...,   [image] nwhere the   [image] i are either atoms or lists. The elements   [image] i, 1   [image]  i   [image]  n which are not atoms are said to be the sublists of A.

The list A itself is written as A = (  [image] 1, ...,   [image] n). A is the name of the list (  [image] 1, ...,   [image] n) and n its length. By convention, all list names will be represented by capital letters. Lower case letters will be used to represent atoms. If n   [image]  1, then   [image] 1 is the head of A while (  [image] 2, ...,   [image] 2) is the tail of A.

The above definition is our first example of a recursive definition so one should study it carefully. The definition is recursive because within our description of what a list is, we use the notion of a list. This may appear to be circular, but it is not. It is a compact way of describing a potentially large and varied structure. We will see more such definitions later on. Some examples of generalized lists are:

(i)   D =  ( )        the null or empty list, its length is zero.

(ii)  A = (a, (b,c))  a list of length two; its first element is

                       the atom 'a' and its second element is the

                       linear list (b,c).

(iii) B = (A,A,( ))   a list of length three whose first two

                       elements are the lists A, the third element

                       the null list.

(iv)  C = (a, C)      a recursive list of length two. C corresponds

                       to the infinite list C = (a,(a,(a, ...).

Example one is the empty list and is easily seen to agree with the definition. For list A, we have

head (A) = 'a', tail (A) = ((b,c)).

The tail (A) also has a head and tail which are (b,c) and ( ) respectively. Looking at list B we see that

head (B)) = A, tail (B)) = (A, ( ))

Continuing we have

head (tail(B)) = A, tail (tail(B)) = (( ))

both of which are lists.

Two important consequences of our definition for a list are: (i) lists may be shared by other lists as in example iii, where list A makes up two of the sublists of list B; and (ii) lists may be recursive as in example iv. The implications of these two consequences for the data structures needed to represent lists will become evident as we go along.

First, let us restrict ourselves to the situation where the lists being represented are neither shared nor recursive. To see where this notion of a list may be useful, consider how to represent polynomials in several variables. Suppose we need to devise a data representation for them and consider one typical example, the polynomial P(x,y,z) =

x10 y3 z2 + 2x8 y3 z2 + 3x8 y2 z2 + x4 y4 z + 6x3 y4 z + 2yz

One can easily think of a sequential representation for P, say using

Figure 4.22 Representation of P(x,y,z) using three fields per node

nodes with four fields: COEF, EXPX, EXPY, and EXPZ. But this would mean that polynomials in a different number of variables would need a different number of fields, adding another conceptual inelegance to other difficulties we have already seen with sequential representation of polynomials. If we used linear lists, we might conceive of a node of the form

These nodes would have to vary in size depending on the number of variables, causing difficulties in storage management. The idea of using a general list structure with fixed size nodes arises naturally if we consider re-writing P(x,y,z) as

((x10 + 2x8)y3 + 3x8y2)z2 + ((x4 + 6x3)y4 + 2y)z

Every polynomial can be written in this fashion, factoring out a main variable z, followed by a second variable y, etc. Looking carefully now at P(x,y,z) we see that there are two terms in the variable z, Cz2 + Dz, where C and D are polynomials themselves but in the variables x and y. Looking closer at C(x,y), we see that it is of the form Ey3 + Fy2, where E and F are polynomials in x. Continuing in this way we see that every polynomial consists of a variable plus coefficient exponent pairs. Each coefficient is itself a polynomial (in one less variable) if we regard a single numerical coefficient as a polynomial in zero variables.

We can represent P(x,y,z) as a list in the following way using nodes with three fields each, COEF, EXP, LINK, as in section 4.4. Note that each level has a head node indicating the variable that has been factored out. This variable is stored in the COEF field.

The list structure of figure 4.22 uses only fixed size nodes. There is, however, one difficulty which needs to be resolved: how do we distinguish between a coefficient and a pointer to another list? Ultimately, both of these values will be represented as numbers so we cannot readily distinguish between them. The solution is to add another field to each node. This field called TAG, will be zero if the COEF field truly contains a numerical coefficient and a one otherwise. Using the structure of figure 4.22 the polynomial P= 3x2y would be represented as

Notice that the TAG field is set to one in the nodes containing x and y. This is because the character codes for variable names would not really be stored in the COEF field. The names might be too large. Instead a pointer to another list is kept which represents the name in a symbol table. This setting of TAG = 1 is also convenient because it allows us to distinguish between a variable name and a constant.

It is a little surprising that every generalized list can be represented using the node structure:

           ________________________

          |          |      |      |

          |TAG = 0/1 | DATA | LINK |

          |__________|______|______|

The reader should convince himself that this node structure is adequate for the representation of any list A. The LINK field may be used as a pointer to the tail of the list while the DATA field can hold an atom in case head (A) is an atom or be a pointer to the list representation of head (A) in case it is a list. Using this node structure, the example lists i-iv have the representation shown in figure 4.23.

Figure 4.23 Representation of Lists i-iv

Recursive Algorithms for Lists

Now that we have seen a particular example where generalized lists are useful, let us return to their definition again. Whenever a data object is defined recursively, it is often easy to describe algorithms which work on these objects recursively. If our programming language does not allow recursion, that should not matter because we can always translate a recursive program into a nonrecursive version. To see how recursion is useful, let us write an algorithm which produces an exact copy of a nonrecursive list L in which no sublists are shared. We will assume that each node has three fields, TAG, DATA and LINK, and also that there exists the procedure GETNODE(X) which assigns to X the address of a new node.

procedure COPY (L)

//L points to a nonrecursive list with no common sublists. COPY

returns a pointer to a new list which is a duplicate of L//

ptr    0

if L    0 then

[if TAG(L) = 0 then q    DATA(L)     //save an atom//

else q    COPY(DATA(L))      //recursion//

r    COPY(LINK(L))        //copy tail//

call GETNODE(ptr)        //get a node//

DATA(ptr)    q; LINK(ptr)    r        //combine head and tail//

TAG(ptr)    TAG(L)]

return (ptr)

end COPY

The above procedure reflects exactly the definition of a list. We immediately see that COPY works correctly for an empty list. A simple proof using induction will verify the correctness of the entire procedure. Once we have established that the program is correct we may wish to remove the recursion for efficiency. This can be done using some straightforward rules.

(i)     At the beginning of the procedure, code is inserted which declares a stack and initializes it to be empty. In the most general case, the stack will be used to hold the values of parameters, local variables, function value, and return address for each recursive call.

(ii)     The label L1 is attached to the first executable statement.

Now, each recursive call is replaced by a set of instructions which do the following:

(iii)     Store the values of all parameters and local variables in the stack. The pointer to the top of the stack can be treated as global.

(iv)     Create the ith new label, Li, and store i in the stack. The value i of this label will be used to compute the return address. This label is placed in the program as described in rule (vii).

(v)     Evaluate the arguments of this call (they may be expressions) and assign these values to the appropriate formal parameters.

(vi)     Insert an unconditional branch to the beginning of the procedure.

(vii)     Attach the label created in (iv) to the statement immediately following the unconditional branch. If this procedure is a function, attach the label to a statement which retrieves the function value from the top of the stack. Then make use of this value in whatever way the recursive program describes.

These steps are sufficient to remove all recursive calls in a procedure. We must now alter any return statements in the following way. In place of each return do:

(viii)     If the stack is empty then execute a normal return.

(ix)     therwise take the current values of all output parameters (explicitly or implicitly understood to be of type out or inout) and assign these values to the corresponding variables which are in the top of the stack.

(x)     Now insert code which removes the index of the return address from the stack if one has been placed there. Assign this address to some unused variable.

(xi)     Remove from the stack the values of all local variables and parameters and assign them to their corresponding variables.

(xii)     If this is a function, insert instructions to evaluate the expression immediately following return and store the result in the top of the stack.

(xiii)     Use the index of the label of the return address to execute a branch to that label.

By following these rules carefully one can take any recursive program and produce a program which works in exactly the same way, yet which uses only iteration to control the flow of the program. On many compilers this resultant program will be much more efficient than its recursive version. On other compilers the times may be fairly close. Once the transformation to iterative form has been accomplished, one can often simplify the program even further thereby producing even more gains in efficiency. These rules have been used to produce the iterative version of COPY which appears on the next page.

It is hard to believe that the nonrecursive version is any more intelligible than the recursive one. But it does show explicitly how to implement such an algorithm in, say, FORTRAN. The non-recursive version does have some virtues, namely, it is more efficient. The overhead of parameter passing on most compilers is heavy. Moreover, there are optimizations that can be made on the latter version, but not on the former. Thus,

procedure COPY(L)

//nonrecursive version//

i    0  //initialize stack index//

1: ptr    0

if L    0 then

[if TAG(L) = 0

then q    DATA(L)

else [STACK(i + 1)    q    //stack local variables//

STACK(i + 2)    r

STACK(i + 3)    ptr

STACK(i + 4)    L    //stack parameter//

STACK(i + 5)    2    //stack return address//

i    i + 5;

L    DATA(L); go to 1      //set parameter and

begin//

2:                 q    STACK(i)     //remove function value//

i    i - 1]

STACK(i + 1)    q; STACK(i + 2)    r  //stack

variables and//

STACK(i + 3)    ptr; STACK(i + 4)    L

//parameter for second//

STACK(i + 5)    3; i    i +5     //recursive call//

L    LINK(L); go to 1

3:          r    STACK(i); i    i    1   //remove function value//

call GETNODE(ptr)

DATA(ptr)    q; LlNK(ptr)    r

TAG(ptr)    TAG(L)]

if i    0 then [addr    STACK(i); L    STACK(i - 1)

//execute a return//

t    STACK(i - 2); r    STACK(i - 3)

q    STACK(i - 4);

STACK(i - 4)    ptr; ptr    t   //store function

value//

i    i - 4; go to addr]    //branch to 2 or 3//

return (ptr)

end COPY

both of these forms have their place. We will often use the recursive version for descriptive purposes.

Now let us consider the computing time of this algorithm. The null

Figure 4.24 Linked Representation for A

list takes a constant amount of time. For the list

A = ((a,b),((c,d),e))

which has the representation of figure 4.24 L takes on the values given in figure 4.25.

Levels of              Continuing        Continuing

recursion  Value of L    Levels      L    Levels      L

    1          q           2         r      3         u

    2          s           3         u      4         v

    3          t           4         w      5         o

    4          o           5         x      4         v

    3          t           6         o      3         u

    2          s           5         x      2         r

    1          q           4         w      3         o

                                            2         r

                                            1         q

Figure 4.25 Values of Parameter in Execution of COPY(A)

The sequence of values should be read down the columns. q, r, s, t, u, v, w, x are the addresses of the eight nodes of the list. From this particular example one should be able to see that nodes with TAG = 0 will be visited twice, while nodes with TAG= 1 will be visited three times. Thus, if a list has a total of m nodes, no more than 3m executions of any statement will occur. Hence, the algorithm is O(m) or linear which is the best we could hope to achieve. Another factor of interest is the maximum depth of recursion or, equivalently, how many locations one will need for the stack. Again, by carefully following the algorithm on the previous example one sees that the maximum depth is a combination of the lengths and depths of all sublists. However, a simple upper bound to use is m, the total number of nodes. Though this bound will be extremely large in many cases, it is achievable, for instance, if

L= (((((a))))).

Another procedure which is often useful is one which determines whether two lists are identical. This means they must have the same structure and the same data in corresponding fields. Again, using the recursive definition of a list we can write a short recursive program which accomplishes this task.

procedure EQUAL(S,T)

//S and T are nonrecursive lists, each node having three fields: TAG,

DATA and LINK. The procedure returns the value true if the

lists are identical else false//

ans    false

case

:S = 0 and T = 0: ans    true

:S    0 and T    0:

if TAG(S) = TAG(T)

then [if TAG(S) = 0

then ans    DATA (S) = DATA(T)

else ans    EQUAL(DATA(S),DATA(T))

if ans then

ans    EQUAL(LINK(S),LINK(T))]

end

return (ans)

end EQUAL

Procedure EQUAL is a function which returns either the value true or false. Its computing time is clearly no more than linear when no sublists are shared since it looks at each node of S and T no more than three times each. For unequal lists the procedure terminates as soon as it discovers that the lists are not identical.

Another handy operation on nonrecursive lists is the function which computes the depth of a list. The depth of the empty list is defined to be zero and in general

Procedure DEPTH is a very close transformation of the definition which is itself recursive.

By now you have seen several programs of this type and you should be feeling more comfortable both reading and writing recursive algorithms. To convince yourself that you understand the way these work, try exercises 37, 38, and 39.

procedure DEPTH(S)

//S is a nonrecursive list having nodes with fields TAG, DATA an

LINK. The procedure returns the depth of the list//

max    0

if S = 0 then return (max)      //null list has zero depth//

ptr    S

while ptr    0 do

if TAG(ptr) = 0 then ans    0

else ans    DEPTH(DATA(ptr))    //recursion//

if ans > max    then max  ans      //find a new maximum//

ptr  LINK(ptr)

end

return (max + 1)

end DEPTH

Reference Counts, Shared and Recursive Lists

In this section we shall consider some of the problems that arise when lists are allowed to be shared by other lists and when recursive lists are permitted. Sharing of sublists can in some situations result in great savings in storage used, as identical sublists occupy the same space. In order to facilitate ease in specifying shared sublists, we extend the definition of a list to allow for naming of sublists. A sublist appearing within a list definition may be named through the use of a list name preceding it. For example, in the list A = (a,(b,c)), the sublist (b,c) could be assigned the name Z by writing A = (a,Z(b,c)). In fact, to be consistent we would then write A(a,Z(b,c)) which would define the list A as above.

Lists that are shared by other lists, such as list A of figure 4.23, create problems when one wishes to add or delete a node at the front. If the first node of A is deleted, it is necessary to change the pointers from the list B to point to the second node. In case a new node is added then pointers from B have to be changed to point to the new first node. However, one normally does not know all the points from which a particular list is being referenced. (Even if we did have this information, addition and deletion of nodes could require a large amount of time.) This problem is easily solved through the use of head nodes. In case one expects to perform many add/deletes from the front of lists, then the use of a head node with each list or named sublist will eliminate the need to retain a list of all pointers to any specific list. If each list is to have a head node, then lists i-iv are represented as in figure 4.26. The TAG field of head nodes is not important and is assumed to be zero. Even in situations where one does not wish to dynamically add or delete nodes from lists. ai in the case of multivariate polynomials, head nodoes prove usefull in determining when the nodes of a particular structure may be returned to the storage pool. For example, let T and U be program variables pointing to the two polynomials 3x4 + 5x3 + 7x)y3 and (3x4 + 5x3 + 7x)y6 + (6x)y of figure 4.27. If PERASE(X) is all algorithm to erase the polynomial X, then a call to PERASE(T) should not return the nodes corresponding to the coefficient 3x4 + 5x3 + 7x since this sublist is also part of U.

Thus, whenever lists are being shared by other lists, we need a mechanism to help determine whether or not the list nodes may be physically returned to the available space list. This mechanism is generally provided through the use of a reference count maintained in the head node of each list. Since the DATA field of the head nodes is free, the reference count is maintained in this field. This reference count of a list is the number of pointers (either program variables or pointers from other lists) to that list. If the lists i-iv of figure 4.26 are accessible via the program variables X, Y, Z and W, then the reference counts for the lists are:

(i)     REF(X) = 1      accessible only via X

(ii)     REF(Y) = 3      pointed to by Y and two points from Z

(iii)     REF(Z) = 1      accessible only via Z

(iv)     REF(W) = 2      two pointers to list C

Now a call to LERASE(T) (list erase) should result only in a

Figure 4.26 Structure with Head Nodes for Lists i-iv

Figure 4.27  T = (3x4 + 5x3 + 7x)y3    U = (3x4 + 5x3 + 7x)y6 + 6xy

decrementing by 1 of the reference counter of T. Only if the reference count becomes zero are the nodes of T to be physically returned to the available space list. The same is to be done to the sublists of T.

Assuming the structure TAG, REF, DATA, LINK, an algorithm to erase a list X could proceed by examining the top level nodes of a list whose reference count has become zero. Any sublists encountered are erased and finally, the top level nodes are linked into the available space list. Since the DATA field of head nodes will usually be unused, the REF and DATA fields would be one and the same.

procedure LERASE(X)

//recursive algorithm to erase a list assuming a REF field in each

head node which has the number of pointers to this list and a

TAG field such that TAG(X) = 0 if DATA(X) is actually an atom

and TAG(X) = 1 if DATA(X) is a link to a sublist. The storage

pool is assumed linked through the field LINK with AV pointing

to the first node in the pool//

REF(X)    REF(X) - 1             //decrement reference count//

if REF(X)    0 then return

Y    X // Y will traverse top level of X//

while LINK(Y)    0 do

Y    LINK(Y)

if TAG(Y) = 1 then call LERASE (DATA (Y))// recursion//

end

LINK(Y)    AV         //attach top level nodes to avail list//

AV    X

end LERASE

A call to LERASE(Y) will now only have the effect of decreasing the reference count of Y to 2. Such a call followed by a call to LERASE(Z) will result in:

(i)     reference count of Z becomes zero;

(ii)     next node is processed and REF(Y) reduces to 1;

(iii)     REF(Y) becomes zero and the five nodes of list A(a,(b,c)) are returned to the available space list;

(iv)     the top level nodes of Z are linked into the available space list

The use of head nodes with reference counts solves the problem of determining when nodes are to be physically freed in the case of shared sublists. However, for recursive lists, the reference count never becomes zero. LERASE(W) just results in REF(W) becoming one. The reference count does not become zero even though this list is no longer accessible either through program variables or through other structures. The same is true in the case of indirect recursion (figure 4.28). After calls to LERASE(R) and LERASE(S), REF(R) = 1 and REF(S)= 2 but the structure consisting of R and S is no longer being used and so it should have been returned to available space.

Figure 4.28 Indirect Recursion of Lists A and B Pointed to by Program Variables R and S.

Unfortunately, there is no simple way to supplement the list structure of figure 4.28 so as to be able to determine when recursive lists may be physically erased. It is no longer possible to return all free nodes to the available space list when they become free. So when recursive lists are being used, it is possible to run out of available space even though not all nodes are in use. When this happens, it is possible to collect unused nodes (i.e., garbage nodes) through a process known as garbage collection. This will be described in the next section.

4.10 GARBAGE COLLECTION AND COMPACTION

As remarked at the close of the last section, garbage collection is the process of collecting all unused nodes and returning them to available space. This process is carried out in essentially two phases. In the first phase, known as the marking phase, all nodes in use are marked. In the second phase all unmarked nodes are returned to the available space list. This second phase is trivial when all nodes are of a fixed size. In this case, the second phase requires only the examination of each node to see whether or not it has been marked. If there are a total of n nodes, then the second phase of garbage collection can be carried out in O(n) steps. In this situation it is only the first or marking phase that is of any interest in designing an algorithm. When variable size nodes are in use, it is desirable to compact memory so that all free nodes form a contiguous block of memory. In this case the second phase is referred to as memory compaction. Compaction of disk space to reduce average retrieval time is desirable even for fixed size nodes. In this section we shall study two marking algorithms and one compaction algorithm.

Marking

In order to be able to carry out the marking, we need a mark bit in each node. It will be assumed that this mark bit can be changed at any time by the marking algorithm. Marking algorithms mark all directly accessible nodes (i.e., nodes accessible through program variables referred to as pointer variables) and also all indirectly accessible nodes (i.e., nodes accessible through link fields of nodes in accessible lists). It is assumed that a certain set of variables has been specified as pointer variables and that these variables at all times are either zero (i.e., point to nothing) or are valid pointers to lists. It is also assumed that the link fields of nodes always contain valid link information.

Knowing which variables are pointer variables, it is easy to mark all directly accessible nodes. The indirectly accessible nodes are marked by systematically examining all nodes reachable from these directly accessible nodes. Before examining the marking algorithms let us review the node structure in use. Each node regardless of its usage will have a one bit mark field, MARK, as well as a one bit tag field, TAG. The tag bit of a node will be zero if it contains atomic information. The tag bit is one otherwise. A node with a tag of one has two link fields DLINK and RLINK. Atomic information can be stored only in a node with tag 0. Such nodes are called atomic nodes. All other nodes are list nodes. This node structure is slightly different from the one used in the previous section where a node with tag 0 contained atomic information as well as a RLINK. It is usually the case that the DLINK field is too small for the atomic information and an entire node is required. With this new node structure, the list (a,(b)) is represented as:

Both of the marking algorithms we shall discuss will require that all nodes be initially unmarked (i.e. MARK(i) = 0 for all nodes i) . In addition they will require MARK(0) = 1 and TAG(0) = 0. This will enable us to handle end conditions (such as end of list or empty list) easily. Instead of writing the code for this in both algorithms we shall instead write a driver algorithm to do this. Having initialized all the mark bits as well as TAG(0), this driver will then repeatedly call a marking algorithm to mark all nodes accessible from each of the pointer variables being used. The driver algorithm is fairly simple and we shall just state it without further explanation. In line 7 the algorithm invokes MARK1. In case the second marking algorithm is to be used this can be changed to call MARK2. Both marking algorithms are written so as to work on collections of lists (not necessarily generalized lists as defined here).

procedure DRIVER

//driver for marking algorithm. n is the number of nodes in the

system//

1    for i    1 to n do  //unmark all nodes//

2      MARK (i)    0

3    end

4    MARK (0)    1; TAG (0)    0        //boundary conditions//

5    for each pointer variable X with MARK (X) = 0 do

6      MARK (X)    1

7      if TAG(X) = 1 then call MARK1(X)         //X is a list node//

8    end

9  end DRIVER

The first marking algorithm MARK1(X) will start from the list node X and mark all nodes that can be reached from X via a sequence of RLINK's and DLINK's; examining all such paths will result in the examination of all reachable nodes. While examining any node of type list we will have a choice as to whether to move to the DLINK or to the RLINK. MARK1 will move to the DLINK but will at the same time place the RLINK on a stack in case the RLINK is a list node not yet marked. The use of this stack will enable us to return at a later point to the RLINK and examine all paths from there. This strategy is similar to the one used in the previous section for LERASE.

Analysis of MARK1

In line 5 of MARK1 we check to see if Q= RLINK(P) can lead to other unmarked accessible nodes. If so, Q is stacked. The examination

procedure MARK1 (X)

//X is a list node. Mark all nodes accessible from X. It is assumed

that MARK(0) = 1 and TAG(0) = 0. ADD and DELETE

perform the standard stack operations//

1    P    X; initialize stack

2    loop        //follow all paths from P, P is a marked list node//

3      loop      //move downwards stacking RLINK's if needed//

4         Q    RLINK (P)

5         if TAG(Q) = 1 and MARK(Q) = 0 then call ADD(Q)

//stack Q//

6         MARK(Q)    1       //Q may be atomic//

7         P    DLINK (P)

//any unmarked nodes accessible from P?//

8         if MARK(P) = 1 or TAG(P) = 0 then exit

9         MARK(P)    1

10       forever

11       MARK(P)    1            //P may be an unmarked atomic node//

12       if stack empty then return      //all accessible nodes marked//

13       call DELETE(P)            //unstack P//

14    forever

15  end MARK1

of nodes continues with the node at DLINK(P). When we have moved downwards as far as is possible, line 8, we exit from the loop of lines 3-10. At this point we try out one of the alternative moves from the stack, line 13. One may readily verify that MARK1 does indeed mark all previously unmarked nodes which are accessible from X.

In analyzing the computing time of this algorithm we observe that on each iteration (except for the last) of the loop of lines 3-10, at least one previously unmarked node gets marked (line 9). Thus, if the outer loop, lines 2-14, is iterated r times and the total number of iterations of the inner loop, lines 3-10, is p then at least q = p - r previously unmarked nodes get marked by the algorithm. Let m be the number of new nodes marked. Then m   [image]  q = p - r. Also, the number of iterations of the loop of lines 2-14 is one plus the number of nodes that get stacked. The only nodes that can be stacked are those previously unmarked (line 5). Once a node is stacked it gets marked (line 6). Hence r   [image]  m + 1. From this and the knowledge that m   [image]  p - r, we conclude that p   [image]  2m + 1. The computing time of the algorithm is O(p + r). Substituting for p and r we obtain O(m) as the computing time. The time is linear in the number of new nodes marked! Since any algorithm to mark nodes must spend at least one unit of time on each new node marked, it follows that there is no algorithm with a time less than O(m). Hence MARK1 is optimal to within a constant factor (recall that 2m = O (m) and 10m = O(m)).

Having observed that MARK1 is optimal to within a constant factor you may be tempted to sit back in your arm chair and relish a moment of smugness. There is, unfortunately, a serious flaw with MARK1. This flaw is sufficiently serious as to make the algorithm of little use in many garbage collection applications. Garbage collectors are invoked only when we have run out of space. This means that at the time MARK1 is to operate, we do not have an unlimited amount of space available in which to maintain the stack. In some applications each node might have a free field which can be used to maintain a linked stack. In fact, if variable size nodes are in use and storage compaction is to be carried out then such a field will be available (see the compaction algorithm COMPACT). When fixed size nodes are in use, compaction can be efficiently carried out without this additional field and so we will not be able to maintain a linked stack (see exercises for another special case permitting the growth of a linked stack). Realizing this deficiency in MARK1, let us proceed to another marking algorithm MARK2. MARK2 will not require any additional space in which to maintain a stack. Its computing is also O(m) but the constant factor here is larger than that for MARK1.

Unlike MARK1(X) which does not alter any of the links in the list X, the algorithm MARK2(X) will modify some of these links. However, by the time it finishes its task the list structure will be restored to its original form. Starting from a list node X, MARK2 traces all possible paths made up of DLINK's and RLINK's. Whenever a choice is to be made the DLINK direction is explored first. Instead of maintaining a stack of alternative choices (as was done by MARK1) we now maintain the path taken from X to the node P that is currently being examined. This path is maintained by changing some of the links along the path from X to P.

Consider the example list of figure 4.29(a). Initially, all nodes except node A are unmarked and only node E is atomic. From node A we can either move down to node B or right to node I. MARK2 will always move down when faced with such an alternative. We shall use P to point to the node currently being examined and T to point to the node preceding P in the path from X to P. The path T to X will be maintained as a chain comprised of the nodes on this T- X path. If we advance from node P to node Q then either Q = RLINK(P) or Q = DLINK(P) and Q will become the node currently being examined . The node preceding Q on the X - Q path is P and so the path list must be updated to represent the path from P to X. This is simply done by adding node P to the T - X path already constructed. Nodes will be linked onto this path either through their DLINK or RLINK field. Only list nodes will be placed onto this path chain. When node P is being added to the path chain, P is linked to T via its DLINK field if Q = DLINK(P). When Q = RLINK(P), P is linked to T via its RLINK field. In order to be able to determine whether a node on the T- X path list is linked through its DLINK or RLINK field we make use of the tag field. Notice that since the T- X path list will contain only list nodes, the tag on all these nodes will be one. When the DLINK field is used for linking, this tag will be changed to zero. Thus, for nodes on the T - X path we have:

Figure 4.29 Example List for MARK2

A  B  C  D  F  G  F  D  C  B  H  B  A  I  J  I  A

(e) Path taken by P

Figure 4.29 Example List for MARK2 (contd.)

The tag will be reset to 1 when the node gets off the T - X path list.

Figure 4.29(b) shows the T - X path list when node P is being examined. Nodes A, B and C have a tag of zero indicating that linking on these nodes is via the DLINK field. This also implies that in the original list structure, B = DLINK(A), C = DLINK(B) and D = P = DLINK(C). Thus, the link information destroyed while creating the T- X path list is present in the path list. Nodes B, C, and D have already been marked by the algorithm. In exploring P we first attempt to move down to Q = DLINK(P) = E. E is an atomic node so it gets marked and we then attempt to move right from P. Now, Q = RLINK(P) = F. This is an unmarked list node. So, we add P to the path list and proceed to explore Q. Since P is linked to Q by its RLINK field, the linking of P onto the T - X path is made throught its RLINK field. Figure 4.29(c) shows the list structure at the time node G is being examined. Node G is a deadend. We cannot move further either down or right. At this time we move backwards on the X-T path resetting links and tags until we reach a node whose RLINK has not yet been examined. The marking continues from this node. Because nodes are removed from the T - X path list in the reverse order in which they were added to it, this list behaves as a stack. The remaining details of MARK2 are spelled out in the formal SPARKS algorithm. The same driver as for MARK1 is assumed.

procedure MARK 2(X)

//same function as MARK1//

//it is assumed that MARK(0) = 1 and TAG(0) = 0//

1   P    X; T    0    //initialize T - X path list//

2   repeat

3     Q    DLINK(P)       //go down list//

4     case

5       :MARK(Q) = 0 and TAG(Q) = 1:       //Q is an unexamined

list node//

6         MARK(Q)    1; TAG(P)    0

7         DLINK(P)    T; T    P      //add Pto T - X path list//

8         P    Q      //move down to explore Q//

9       :else:     //Q is an atom or already examined//

10         MARK(Q)    1

11         L1: Q    RLINK(P)    //move right of P//

12           case

13       :MARK(Q) = 0 and TAG(Q) = 1:     //explore Q

further//

14         MARK(Q)    1

15         RLINK (P)    T; T    P

16         P    Q

17       :else: MARK(Q)    1    //Q is not to be explored;

back up on path list//

18         while T    0 do   //while path list not empty//

19           Q    T

20           if TAG(Q) = 0    //link to P through DLINK//

21           then [T < DLINK(Q); DLINK (Q)    P

22                TAG(Q)    1; P    Q; go to L1]

//P is node to right of Q//

23           T    RLINK(Q); RLINK(Q)    P

24           P    Q

25         end

26       end

27    end

28   until T= 0

29 end MARK 2

Analysis of MARK2

The correctness of MARK2 may be proved by first showing that the T - X path list is always maintained correctly and that except when T = 0 it is the case that the node T was previously linked to P. Once this has been shown, it is a simple matter to show that the backup procedure of lines 17-25 restructures the list X correctly. That all paths from X are examined follows from the observation that at every list node when a DLINK path is used, that node gets added to the path list (lines 6-8) and that backing up along the T - X path stops when a node whose RLINK hasn't yet been examined is encountered (lines 20-22). The details of the proof are left as an exercise. One can also show that at the time the algorithm terminates P = X and so we need not use two distinct variables P and X.

Figure 4.29(e) shows the path taken by P on the list of 4.29(a). It should be clear that a list node previously unmarked gets visited at most three times. Except for node X, each time a node already marked is reached at least one previously unmarked node is also examined (i.e. the one that led to this marked node). Hence the computing time of MARK2 is O(m) where m is the number of newly marked nodes. The constant factor associated with m is, however, larger than that for MARK1 but MARK2 does not require the stack space needed by MARK1. A faster marking algorithm can be obtained by judiciously combining the strategies of MARK1 and MARK2 (see the exercises).

When the node structure of section 4.9 is in use, an additional one bit field in each node is needed to implement the strategy of MARK2. This field is used to distinguish between the case when a DLINK is used to link into the path list and when a RLINK is used. The already existing tag field cannot be used as some of the nodes on the T- X path list will originally have a tag of 0 while others will have a tag of 1 and so it will not be possible to correctly reset tag values when nodes are removed from the T - X list.

Storage Compaction

When all requests for storage are of a fixed size, it is enough to just link all unmarked (i.e., free) nodes together into an available space list. However, when storage requests may be for blocks of varying sizes, it is desirable to compact storage so that all the free space forms one contiguous block. Consider the memory configuration of figure 4.30. Nodes in use have a MARK bit = 1 while free nodes have their MARK bit = 0. The nodes are labeled 1 through 8, with ni, 1   [image]  i   [image]  8 being the size of the ith node.

Figure 4.30 Memory Configuration After Marking. Free Nodes Have MARK bit = O

The free nodes could be linked together to obtain the available space list of figure 4.31. While the total amount of memory available is n1 + n3 + n5 + n8, a request for this much memory cannot be met since the memory is fragmented into 4 nonadjacent nodes. Further, with more and more use of these nodes, the size of free nodes will get smaller and smaller.

Figure 4.31 Available Space List Corresponding to Figure 4.30

Ultimately, it will be impossible to meet requests for all but the smallest of nodes. In order to overcome this, it is necessary to reallocate the storage of the nodes in use so that the used part of memory (and hence also the free portion) forms a contiguous block at one end as in figure 4.32. This reallocation of storage resulting in a partitioning of memory into two contiguous blocks (one used, the other free) is referred to as storage compaction. Since there will, in general, be links from one node to another, storage compaction must update these links to point to the relocated address of the respective node. If node ni starts at location li before compaction and at l'i after compaction, then all link references to li must also be changed to l'i in order not to disrupt the linked list structures existing in the system. Figure 4.33(a) shows a possible link configuration at the time the garbage collection process is invoked. Links are shown only for those nodes that were marked during the marking phase. It is assumed that there are only two links per node. Figure 4.33(b) shows the configuration following compaction. Note that the list structure is unchanged even though the actual addresses represented by the links have been changed. With storage compaction we may identify three tasks: (i) determine new addresses for nodes in use; (ii) update all links in nodes in use; and (iii) relocate nodes to new addresses. Our storage compaction algorithm, COMPACT, is fairly straightforward, implementing each of these three tasks in a separate scan of memory. The algorithm assumes that each node, free or in use, has a SIZE field giving the length of the node and an additional field, NEW_ADDR, which may be used to store the relocated address of the node. Further, it is assumed that each node has two link fields LINK1 and LINK2. The extension of the algorithm to the most general situation in which nodes have a variable number of links is simple and requires only a modification of phase II.

Figure 4.32 Memory Configuration After Reallocating Storage to Nodes in Use

Figure 4.33

In analyzing this algorithm we see that if the number of nodes in memory is n, then phases I and II each require n iterations of their respective

procedure COMPACT(MEMORY, MARK,SlZE, M, NEW__ADDR)

//compact storage following the marking phase of garbage collection.

Nodes in use have MARK bit=1. SlZE(i)= number of words

in that node. NEW__ADDR is a free field in each node. Memory

is addressed 1 to M and is an array MEMORY.//

//phase I: Scan memory from left assigning new addresses to nodes

in use. AV = next available word//

AV    1;i    1          //variable i will scan all nodes left to right//

while i    M do

if MARK(i) = 1 then [//relocate node//

NEW__ADDR(i)    AV

AV    AV + SIZE(i)]

i    i + SIZE(i)       //next node//

end

//phase II: update all links. Assume the existence of a fictitious

node with address zero and NEW__ADDR(0) = 0//

i    1

while i    M do

if MARK(i) = 1 then [//update all links to reflect new

addresses//

LINK1(i)    NEW_ADDR(LINK1(i))

LlNK2(i)    NEW_ADDR(LINK2(i))]

i    i + SIZE(i)

end

//phase III: relocate nodes//

i    1

while i    M do

if MARK (i) = 1 then [//relocate to NEW_ADDR(i)//

k    NEW_ADDR(i); l    k

for j   i to i + SIZE(i) - 1 do

MEMORY(k)    MEMORY(j)

k    k+ 1

end

i    i + SIZE(l)]

else i    i + SIZE(i)

end

end COMPACT

while loops. Since each iteration of these loops takes a fixed amount of time, the time for these two phases is O(n). Phase III, however, will in general be more expensive. Though the while loop of this phase is also executed only n times, the time per iteration depends on the size of the node being relocated. If s is the amount of memory in use, then the time for this phase is O(n + s). The overall computing time is, therefore, O(n + s). The value of AV at the end of phase I marks the beginning of the free space. At the termination of the algorithm the space MEMORY(AV) to MEMORY(M) is free space. Finally, the physical relocation of nodes in phase III can be carried out using a long shift in case your computer has this facility.

In conclusion, we remark that both marking and storage compaction are slow processes. The time for the former is O (number of nodes) while the time for the latter is O (number of nodes +   [image]  size of nodes relocated)). In the case of generalized lists, garbage collection is necessitated by the absence of any other efficient means to free storage when needed. Garbage collection has found use in some programming languages where it is desirable to free the user from the task of returning storage. In both situations, a disciplined use of pointer variables and link fields is required. Clever coding tricks involving illegal use of link fields could result in chaos during marking and compaction. While compaction has been presented here primarily for use with generalized list systems using nodes of variable size, compaction can also be used in other environments such as the dynamic storage allocation environment of section 4.8. Even though coalescing of adjacent free blocks takes place in algorithm FREE of section 4.8, it is still possible to have several small nonadjacent blocks of memory free. The total size of these blocks may be large enough to meet a request and it may then be desirable to compact storage. The compaction algorithm in this case is simpler than the one described here. Since all addresses used within a block will be relative to the starting address rather than the absolute address, no updating of links within a block is required. Phases I and III can, therefore, be combined into one phase and phase II eliminated altogether. Since compaction is very slow one would like to minimize the number of times it is carried out. With the introduction of compaction, several alternative schemes for dynamic storage management become viable. The exercises explore some of these alternatives.

4.11 STRINGS--A CASE STUDY

Suppose we have two character strings S = 'x1 ... xm' and T = 'y1 ... yn'. The characters xi,yj come from a set usually referred to as the character set of the programming language. The value n is the length of the character string T, an integer which is greater than or equal to zero. If n = 0 then T is called the empty or null string. In this section we will discuss several alternate ways of implementing strings using the techniques of this chapter.

We begin by defining the data structure STRING using the axiomatic notation. For a set of operations we choose to model this structure after the string operations of PL/I. These include:

(i)     NULL produces an instance of the null string;

(ii)     ISNULL returns true if the string is null else false;

(iii)     IN takes a string and a character and inserts it at the end of the string;

(iv)     LEN returns the length of a string;

(v)     CONCAT places a second string at the end of the first string;

(vi)     SUBSTR returns any length of consecutive characters;

(vii)     INDEX determines if one string is contained within another.

We now formally describe the string data structure.

structure STRING

declare NULL( )    string; ISNULL (string)    boolean

IN (string, char)    string; LEN (string)    integer

CONCAT (string, string)    string

SUBSTR (string, integer, integer)    string

LINK_DEST 3184string, string)    integer;

for all S,T    string, i,j    integer, c,d    char let

ISNULL (NULL) :: = true; ISNULL (IN(S,c)) :: = false

LEN (NULL) :: = 0; LEN (IN(S,c)) :: = 1 + LEN(S)

CONCAT (S,NULL):: = S

CONCAT (S,IN(T,c)) :: = IN (CONCAT (S,T),c)

SUBSTR (NULL,i,j):: = NULL

SUBSTR (IN(S,c),i,j) :: =

if j = 0 or i + j - 1 > LEN(IN(S,c)) then NULL

else if i + j - 1 = LEN(IN(S,c))

then IN(SUBSTR (S,i,j - 1 ),c)

else SUBSTR(S,i,j)

INDEX (S,NULL) :: = LEN (S) + 1

INDEX (NULL,IN(T,d)) :: = 0

INDEX (IN(S,c),IN(T,d)) :: =

if c = d and INDEX(S,T) = LEN(S) - LEN(T) + 1

then INDEX(S,T) else INDEX(S,IN(T,d))

end

end STRING

As an example of how these axioms work, let S = `abcd'. This will be represented as

IN(IN(IN(IN(NULL,a),b),c),d).

Now suppose we follow the axioms as they apply to SUBSTR(S,2,1). By the SUBSTR axioms we get

SUBSTR(S,2,1) = SUBSTR(IN(IN(IN(NULL,a),b),c),2,1)

= SUBSTR(IN(IN(NULL,a),b),2,1)

= IN(SUBSTR(IN(NULL,a),2,0),b)

= IN(NULL,b)

= 'b'

Suppose we try another example, SUBSTR(S,3,2)

= IN(SUBSTR(IN(IN(IN(NULL,a),b),c),3,1),d)

= IN(IN(SUBSTR(IN(IN(NULL,a),b),3,0),c),d)

= IN(IN(NULL,c),d)

= 'cd'

For the readers amusement try to simulate the steps taken for INDEX(S,T) where T= 'bc' = IN(IN(NULL,b),c).

4.11.1 DATA REPRESENTATIONS FOR STRINGS

In deciding on a data representation for a given data object one must take into consideration the cost of performing different operations using that representation. In addition, a hidden cost resulting from the necessary storage management operations must also be taken into account. For strings, all three types of representation: sequential, linked list with fixed size nodes, and linked list with variable size nodes, are possible candidates. Let us look at the first two of these schemes and evaluate them with respect to storage management as well as efficiency of operation. We shall assume that the available memory is an array C of size n and that each element of C is large enough to hold exactly one character.

On most computers this will mean that each element of C uses only a fraction of a word and hence several such elements will be packed into one word. On a 60 bit per word machine requiring 6 bits per character, each word will be able to hold 10 characters. Thus, C(1) through C(10) will occupy one word, C(11) thru C(20) another and so on.

Sequential. In this representation successive characters of a string will be placed in consecutive character positions in the vector C. The string S='x1, ...,xn' could then be represented as in figure 4.34 with S a pointer to the first character. In order to facilitate easy length determination, the length of string S could be kept in another variable, SL. Thus, we would have SL = n. SUBSTRING SUBSTR (S,j,k - j + I) could be done now by copying over the characters Xj,...,Xk from locations C(S + j - I) through C(S + k - 1) into a free space. The length of the string created would be k - j + 1 and the time required O(k - j + I) plus the time needed to locate a free space big enough to hold the string. CONCAT (S, T) could similarly be carried out; the length of the resulting string would be SL + TL. For storage management two possibilities exist. The boundary tag scheme of section 4.8 could be used in conjunction with the storage compaction strategies of section 4.10. The storage overhead would be enormous for small strings. Alternatively, we could use garbage collection and compaction whenever more free space was needed . This would eliminate the need to return free spaces to an available space list and hence simplify the storage allocation process (see exercises).

Figure 4.34 Sequential Representation of S = 'x1 ... xn'

While a sequential representation of strings might be adequate for the functions discussed above, such a representation is not adequate when insertions and deletions into and from the middle of a string are carried out. An insertion of 'y1, ..., yn' after the i-th character of S will, in general require copying over the characters x1, ...,xi followed by y1, ...,ym and then xi+1 ,...,xn into a new free area (see figure 4.35). The time required for this is O(n + m). Deletion of a substring may be carried out by either replacing the deleted characters by a special symbol   [image]  or by compacting the space originally occupied by this substring (figure 4.36). The former entails storage waste while the latter in the worst case takes time proportional to LENGTH(S). The replacement of a substring of S by another string T is efficient only if the length of the substring being replaced is equal to LENGTH(T). If this is not the case, then some form of string movement will be required.

Figure 4.35 Insertion Into a Sequential String

Figure 4.36 Deletion of a Substring

Linked List--Fixed Size Nodes. An alternative to sequential string representation is a linked list representation. Available memory is divided into nodes of a fixed size. Each node has two fields: DATA and LINK. The size of a node is the number of characters that can be stored in the DATA field. Figure 4.37 shows the division of memory into nodes of size 4 with a link field that is two characters long. On a computer with 6 bits/character this would permit link values in the range [0,212- 1]. In the purest form of a linked list representation of strings, each node would be of size one. Normally, this would represent extreme wastage of space.

Figure 4.37 Partitioning Available Memory Into Nodes of Size 4.

With a link field of size two characters, this would mean that only 1/3 of available memory would be available to store string information while the remaining 2/3 will be used only for link information. With a node size of 8, 80% of available memory could be used for string information. When using nodes of size > 1, it is possible that a string may need a fractional number of nodes. With a node size of 4, a string of length 13 will need only 3-1/4 nodes. Since fractional nodes cannot be allocated, 4 full nodes may be used with the last three characters of the last node set to   [image]  (figure 4.38(a) ). An in place insertion might require one node to be split into two as in figure 4.38(b). Deletion of a substring can be carried out by replacing all characters in this substring by   [image]  and freeing nodes in which the DATA field consists only of   [image] 's. In place replacement can be carried similarly. Storage management is very similar to that of section 4.3. Storage compaction may be carried out when there are no free nodes. Strings containing many occurrences of   [image]  could be compacted freeing several nodes. String representation with variable sized nodes is similar.

Figure 4.38

When the node size is l things work out very smoothly. Insertion, deletion and concatenation are particularly easy. The length may be determined easily by retaining a head node with this information. Let us look more closely at the operations of insertion and concatenation when the node size is one and no head node is maintained. First, let us write a procedure which takes two character strings and inserts the second after the ith character of the first.

procedure SINSERT(S,T,i)

//insert string T after the i-th character of S destroying the original//

//strings S,T and creating a new string S//

1    case    //degenerate cases//

2      :i < 0 or i > LENGTH(S): print ('string length error'); stop

3      : T = 0: return

4      : S = 0: S    T; return

5    end

//at this point LENGTH(S) > 0, LENGTH(T) > 0, 0    i

LENGTH(S)//

6    ptr    S; j    1

7    while j < i do   //find i-th character of S//

8     ptr    LINK (ptr); j    j + 1  //ptr points to i-th node//

9    end

10    if i = 0 then [save    S; ptr    T; S    T]  //save i + 1 character//

11          else [save    LINK (ptr)

12                LINK (ptr)    T]     //attach list T to list S//

13    while LINK (ptr)    0 do    //find end of T//

14      ptr    LINK (ptr)

15    end

16    LINK (ptr)    save       //point end of T to//

//i + 1-st character of S//

17  end SINSERT

Examine how the algorithm works on the strings below.

T is to be inserted after the fifth character of S to give the result 'THIS NOW IS.' The fifth character in S and the last character of T are blanks. After the if statement is executed in SINSERT the following holds.

The list S has been split into two parts named S and save. The node pointed at by ptr has been attached to T. This variable will now move across the list T until it points to the last node. The LINK field will be set to point to the same node that save is pointing to, creating a single string.

Figure 4.39 Circular Representation of S and T

The computing time of SINSERT is proportional to i + LENGTH(T). We can produce a more efficient algorithm by altering the data structure only slightly. If we use singly linked circular lists, then S and T will be represented as in Figure 4.39 above. The new version of SINSERT is obtained by replacing lines 6-16 by:

ptr S; j    0

while j < i do

ptr    LINK (ptr); j    j + 1   //find i-th character of S//

end

save    LINK (ptr)     //save i + 1-st character of S//

else LINK (ptr)    LINK (T)

LINK (T)    save       //attach end of T to S//

if ptr = S and i    0 then S    T

By using circular lists we avoided the need to find the end of list T. The computing time for this version is O(i) and is independent of the length of T.

4.11.2 PATTERN MATCHING IN STRINGS

Now let us develop an algorithm for a more sophisticated application of strings. Given two strings S and PAT we regard the value of PAT as a pattern to be searched for in S. If it occurs, then we want to know the node in S where PAT begins. The following procedure can be used.

This algorithm is a straightforward consequence of the data representation. Unfortunately, it is not very efficient. Suppose

S = 'aaa ... a'; PAT = 'aaa ... ab'

where LENGTH(S) = m, LENGTH(PAT)= n and m is much larger than n. Then the first n -1 letters of PAT will match with the a's in string S but the n-th letter of PAT will not. The pointer p will be moved to the second occurrence of 'a' in S and the n - 1 a's of PAT will match with S again. Proceeding in this way we see there will be m - n + 1 times that S and PAT have n - 1 a's in common. Therefore,

procedure FIND (S,PAT,i)

//find in string S the first occurrence of the string PAT and return

i as a pointer to the node in S where PAT begins. Otherwise return

i as zero//

i    0

if PAT = 0 or S = 0 then return

p    S

repeat

save    p; q    PAT      //save the starting position//

while p    0 and q    0 and DATA(p) = DATA(q) do

p    LINK (p); q    LINK (q)       //characters match,

move to next pair//

end

if q = 0 then [i    save; return]   //a match is found//

p    LINK (save)          //start at next character in S//

until p = 0     //loop until no more elements in S//

end FIND

algorithm FIND will require at least (m - n + l) (n - 1) = O(mn) operations. This makes the cost of FIND proportional to the product of the lengths of the two lists or quadratic rather than linear.

There are several improvements that can be made. One is to avoid the situation where LENGTH(PAT) is greater than the remaining length of S but the algorithm is still searching for a match. Another improvement would be to check that the first and last characters of PAT are matched in S before checking the remaining characters. Of course, these improve- will speed up processing on the average. Procedure NFIND incorporates these improvements.

If we apply NFIND to the strings S = 'aa ... a' and PAT = 'a ... ab', then the computing time for these inputs is O(m) where m = LENGTH(S) which is far better than FIND which required O(mn). However, the worst case computing time for NFIND is still O(mn).

NFIND is a reasonably complex program employing linked lists and it should be understood before one reads on. The use of pointers like p, q, j, r is very typical for programs using this data representation.

It would be far better if we could devise an algorithm which works in time O(LENGTH(S) + LENGTH(PAT)), which is linear for this problem since we must certainly look at all of PAT and potentially all of S.

procedure NFIND (S,PAT,i)

//string S is searched for PAT and i is set to the first node in S

where PAT occurs else zero. Initially the last and first characters

of PAT are checked for in S//

i    0

if PAT = 0 or S = 0 then return

p    q    PAT;t    0

while LINK (q)    0 do         //q points to last node of PAT//

q    LINK(q); t    t + 1

end       //t + 1 is the length of PAT//

j    r    save    S

for k    1 to t while j    0 do      //find t + 1-st node of S//

j    LINK (j)

end

while j    0 do      //while S has more chars to inspect//

p    PAT; r    save

if DATA(q) = DATA(j)       //check last characters//

then [while DATA(p) = DATA(r) and p    q do

p    LINK(p); r    LINK(r)    //check pattern//

end

if p = q then [i    save; return]]  //success//

save    LINK (save); j    LINK(j)

end

end NFIND

Figure 4.40 Action of NFIND on S and PAT

Another desirable feature of any pattern finding algorithm is to avoid rescanning the string S. If S is so large that it cannot conveniently be stored in memory, then rescanning adds complications to the buffering operations. Such an algorithm has recently been developed by Knuth, Morris and Pratt. Using their example suppose

PAT = 'a b c a b c a c a b'

Let S = s1 s2... sm and assume that we are currently determining whether or not there is a match beginning at si. If si   [image]  a then clearly, we may proceed by comparing si + 1 and a. Similarly if si = a and si+1   [image]  b then we may proceed by comparing si+1 and a. If si si+1 = ab and si+2   [image]  c then we have the situation:

S = ' - a b ? ? ? . . . . ?'

PAT =   ' a b c a b c a c a b'

The ? implies we do not know what the character in S is. The first ? in S represents si + 2 and si + 2   [image]  c. At this point we know that we may continue the search for a match by comparing the first character in PAT with si+2. There is no need to compare this character of PAT with si + l as we already know that si + 1 is the same as the second character of PAT, b and so si + 1   [image]  a. Let us try this again assuming a match of the first four characters in PAT followed by a non-match i.e. si+4   [image]  b. We now have the situation:

S = ' - a b c a ? ? . . . ?'

PAT =   ' a b c a b c a c a b'

We observe that the search for a match can proceed by comparing si + 4 and the second character in PAT, b. This is the first place a partial match can occur by sliding the pattern PAT towards the right. Thus, by knowing the characters in the pattern and the position in the pattern where a mismatch occurs with a character in S we can determine where in the pattern to continue the search for a match without moving backwards in S. To formalize this, we define a failure function for a pattern. Definition: If P = p1p2 ... pn is a pattern, then its failure function, f, is defined as:

For the example pattern above, PAT = abcabcacab, we have

j  1 2 3 4 5 6 7 8 9 10

PAT  a b c a b c a c a b

f  0 0 0 1 2 3 4 0 1 2

From the definition of the failure function we arrive at the following rule for pattern matching: If a partial match is found such that si-j+l ... si-1 = p1 p2 ... pj-l and si   [image]  pj then matching may be resumed by comparing si and Pf(j-1)+1 If j   [image]  1. If j = 1, then we may continue by comparing si+1 and p1.

In order to use the above rule when the pattern is represented as a linked list as per our earlier discussion, we include in every node representing the pattern, an additional field called NEXT. If LOC(j) is the address of the node representing pj, 1   [image]  j   [image]  n, then we define

Figure 4.41 shows the pattern PAT with the NEXT field included.

With this definition for NEXT, the pattern matching rule translates to the following algorithm:

procedure PMATCH (S,PAT)

//determine if PAT is a substring of S//

1      i    S       //i will move through S//

2      j    PAT         //j will move through PAT//

3      while i    0 and j    0 do

4            if DATA (i) = DATA (j)

5               then [//match//

6                       i    LINK (i)

7                       j    LINK (j)]

8               else [if j = PAT then i    LINK(i)

9                                 else j    NEXT(j]

10      end

11      if j = 0 then print ('a match')

12           else print ('no match')

13  end PMATCH

The correctness of PMATCH follows from the definitions of the failure function and of NEXT. To determine the computing time, we observe that the then clauses of lines 5-7 and 8-9 can be executed for a total of at most m = LENGTH(S) times as in each iteration i moves right on S but i never moves left in the algorithm. As a result j can move right on PAT at most m times (line 7). Since each execution of the else clause in line 9 moves j left on PAT, it follows that this clause can be executed at most m times as otherwise j must fall off the left end of PAT. As a result, the maximum number of iterations of the while loop of lines 3-10 is m and the computing time of PMATCH is O(m). The performance of the algorithm may be improved by starting with a better failure function (see exercise 57).

Figure 4.41 Augmented Pattern PAT with NEXT Field

The preceding discussion shows that NFIND can be improved to an O(m) algorithm provided we are either given the failure function or NEXT. We now look into the problem of determining f. Once f is known, it is easy to get NEXT. From exercise 56 we know that the failure function for any pattern p1 p2 ... pn is given by:

(note that f1 (j) = f(j) and fm(j) = f (fm-1 (j)))

This directly yields the following algorithm to compute f.

procedure FAIL (PAT,f)

//compute the failure function for PAT = p1p2 ... pn//

1       f(1)    0

2       for j    2 to n do  //compute f(j)//

3           i    f(j - 1)

4           while pj    pi + 1 and i > 0 do

5             i    f(i)

6           end

7           if pj = pi + 1 then f(j)    i + 1

8             else f(j)    0

9      end

10  end FAIL

In analyzing the computing time of this algorithm we note that in each iteration of the while loop of lines 4-6 the value of i decreases (by the definition of f). The variable i is reset at the beginning of each iteration of the for loop. However, it is either reset to 0 (when j = 2 or when the previous iteration went through line 8) or it is reset to a value 1 greater than its terminal value on the previous iteration (i.e. when the previous iteration went through line 7). Since only n executions of line 3 are made, the value of i therefore has a total increment of at most n. Hence it cannot be decremented more than n times. Consequently the while loop of lines 4-6 is iterated at most n times over the whole algorithm and the computing time of FAIL is O(n).

Even if we are not given the failure function for a pattern, pattern matching can be carried out in time O(n + m). This is an improvement over NFIND.

4.12 IMPLEMENTING NODE STRUCTURES

Throughout this chapter we have dealt with nodes without really seeing how they may be set up in any high level programming language. Since almost all such languages provide array variables, the easiest way to establish n nodes is to define one dimensional arrays, each of size n, for each of the fields in a node. The node structure to represent univariate polynomials (section 4.4) consisted of three fields: EXP, COEF and LINK. Assuming integer coefficients, 200 polynomial nodes can be set up in FORTRAN through the statement:

INTEGER EXP(200), COEF(200), LINK(200)

The nodes then have indices 1 through 200. The values of various fields in any node i, 1   [image]  i   [image]  200 can be determined or changed through the use of standard assignment statements. With this structure for nodes, the following FORTRAN subroutine would perform the function of algorithm INIT of section 4.3.

    SUBROUTINE INIT(N)

C   LINK N NODES TO FORM THE INITIAL AVAILABLE

C   SPACE LIST AND SET AV TO POINT TO FIRST NODE.

C   ARRAYS EXP, COEF AND LINK AND VARIABLE AV

C   ARE IN BLANK COMMON.

    INTEGER EXP(200),COEF(200),LINK(200),AV

    COMMON EXP,COEF,LINK,AV

    M = N - 1

    DO 10 I = 1, M

10  LINK(I) = I + 1

    LINK(N) = 0

    AV= 1

    RETURN

    END

Using global one dimensional arrays for each field, the function INIT can be realized in PASCAL by the procedure:

PROCEDURE INIT(N: INTEGER);

{This procedure initializes the available space list. LINK is a global

array of type integer. AV is a global variable of type integer.}

VAR I: INTEGER; {Local variable I}

BEGIN FOR I: = 1 TO N - 1 DO LINK(I): = I + 1;

LINK(N): = 0;

AV: = 1

END;

Usually one does not need a full memory word for each field. So space can be saved by packing several fields into one word. If we have a 60 bit word, as in a CDC computer, choosing field sizes of 15 bits for the EXP field, 15 bits for the LINK field and 30 bits for the COEF field would permit the use of one word per node. This node structure would be adequate to handle LINK and EXP values in the range [0,32767] and coefficient values in excess of the range [-5 x 108, + 5 x 108]. On a 32 bit per word machine roughly the same ranges could be obtained through the use of two words per node. One word of each node could be used for the COEF field and half of the second word would be used for each of the remaining two fields. In the case of 60 bit words, this packing of fields would reduce storage requirements by 2/3, while in the 32 bit case the reduction would be by a factor of 1/3 (this, of course, assumes that the field sizes discussed above are enough to handle the problem being solved). If we are going to pack fields into words, then we must have a way to extract and set field values in any node. In a language like FORTRAN this can be done through the use of the SHIFT function together with masking operations using the logical operations of AND, OR and NOT. The SHIFT function has two arguments, X and N, where X is the word whose bits are to be shifted and N is the amount of the shift. If N > 0, then the result of the shift is X shifted circularly left by N bits. The value of X is left unchanged. If N < 0, then the result is X shifted right with sign extension by N bits. X is left unchanged. Figure 4.42 illustrates the result of a left circular and right shift on the bits of a 5 bit word. Figure 4.43 lists FORTRAN subroutines to extract and set the three fields of a 1 word node. Figure 4.44 does this for the two fields packed into 1 word for the case of a 32 bit word.

               30 bits       15 bits      15 bits

          |---------------------------------------|

          |     COEF     |    EXP     |  LINK     |

          |---------------------------------------|

        node structure using one 60 bit word per node

  INTEGER FUNCTION COEF(J)               SUBROUTINE SCOEF(J,I)

C EXTRACT COEF FIELD OF J BY           C SET THE COEF FIELD OF J TO I BY

C USING A RIGHT SHIFT BY 30 BITS.      C FIRST ZEROING OUT OLD

  COEF = SHIFT(J,-30)                  C CONTENTS AND THEN USE AN OR

  RETURN                               C TO PUT IN NEW VALUE. ASSUME I IS

  END                                  C IN RIGHT RANGE.

                                         J = (J.AND.7777777777B).OR.

                                           SHIFT(I.AND.7777777777B,30)

                                         RETURN

                                         END

  INTEGER FUNCTION EXP(J)                SUBROUTINE SEXP(J,I )

C EXTRACT EXP FIELD BY SHIFTING        C SET EXP FIELD OF J TO I

C RIGHT AND ZEROING OUT EXTRA

  BITS                                   J = (J.AND.77777777770000077777B)

  EXP = SHIFT(J, - 15).AND.77777B          .OR.SHIFT(I,15)

  RETURN                                 RETURN

  END                                    END

  FUNCTION LINK(J)                       SUBROUTINE SLINK(J,I)

C EXTRACT LINK FIELD OF J              C SET LINK FIELD OF J TO I

  LINK = J.AND.77777B                    J = (J.AND.77777777777777700000B)

  RETURN                                   .OR. I

  END                                    RETURN

  END

Figure 4.43 Setting and Extracting Fields in a 60 Bit Word (EXP, LINK nonnegative integers)

                       16 bits       16 bits

                  |---------------------------|

                  |     EXP     |    LINK     |

                  |---------------------------|

  INTEGER FUNCTION EXP(J)                SUBROUTINE SEXP(J,I)

C EXTRACT EXP FIELD OF J               C SET EXP FIELD OF J TO I

  EXP = SHIFT(J, - 16).AND.177777B       J = (J.AND.177777B).OR.SHIFT(I,16)

  RETURN  RETURN

  END  END

(NOTE:  the AND operation is needed

        to mask out sign bit extension)

  INTEGER FUNCTION LINK(J)               SUBROUTINE SLINK(J,I)

C EXTRACT LINK FIELD OF J              C SET LINK FIELD OF J TO I

  LINK = J.AND. 177777B                  J = (J.AND.37777600000B).OR.I

  RETURN  RETURN

  END  END

These subroutines and functions assume that negative numbers are stored using '1s complement arithmetic. A "B' following a constant means that the constant is an octal constant.

Figure 4.44 Setting and Extracting Fields in a 32 Bit Word

VAR LINK:  ARRAY [1 . . 200] OF 0 . . 32767;

     EXP:  ARRAY [1 . . 200] OF 0 . . 32767;

    COEF:  ARRAY [1 . . 200] OF - 1073741823 . . 1073741823;

Figure 4.45 PASCAL Array Declaration to Get Field Sizes

Figure 4.42 Left and Right Shifts

In PASCAL, the task of packing fields into words is simplified or eliminated as the language definition itself permits specifying a range of values for each variable. Thus, if the range of variable I is defined to be [0,215 - 1], then only 15 bits are allocated to I. The PASCAL declaration, figure 4.45 for arrays EXP, COEF and LINK would result in the same storage requirements for 200 nodes as the packing and unpacking routines of figure 4.44. Alternatively, one could use record variables as in figure 4.46.

TYPE POLY = PACKED RECORD COEF: -1073741823 . . 1073741823

                          EXP: 0 . . 32767

                          LINK: 0 . . 32767

            END;

VAR NODE: ARRAY [1 . . 200] OF POLY

usage ... NODE [subscript]. COEF; NODE [subscript]. EXP; NODE [subscript]. LINK

Figure 4.46 Declaring NODE(200) to be a Record Variable With Fields EXP, LINK and COEF of the Same Size as in Figure 4.45

REFERENCES

A general reference for the topics discussed here is The Art of Computer Programming: Fundamental Algorithms by D. Knuth, volume I 2nd edition, Addison-Wesley, Reading, 1973.

For a discussion of early list processing systems see "Symmetric list processor" by J. Weizenbaum, CACM, vol. 6, no. 9, Sept. 1963, pp. 524-544.

"A comparison of list processing computer languages," by Bobrow and B. Raphael, CACM, vol. 7, no. 4, April 1964, pp. 231-240.

"An introduction to IPL-V" by A. Newell and F. M. Tonge, CACM, vol. 3, no. 4, April 1960, pp. 205-211.

"Recursive functions of symbolic expressions and their computation by machine: I," by J. McCarthy, CACM, vol. 3, no. 4, April 1960, pp. 184-195.

For a survey of symbol manipulation systems see Proceedings of the Second Symposium on Symbolic and Algebraic Manipulation, ed. S. R. Petrick, March, 1971, available from ACM.

For further articles on dynamic storage allocation see "An estimate of the store size necessary for dynamic storage allocation" by J. Robson, JACM, vol. 18, no. 3, July 1971, pp. 416-423.

"Multiword list items" by W. T. Comfort, CACM, vol. 7, no. 6, June 1964, pp. 357-362.

"A weighted buddy method for dynamic storage allocation" by K. Shen and J. Peterson, CACM, vol. 17, no. 10, October 1974, pp. 558-562.

"Statistical properties of the buddy system," by P. Purdom and S. Stigler JACM, vol. 14, no. 4, October 1970, pp. 683-697.

"A class of dynamic memory allocation algorithms," by D. Hirschberg, CACM, vol. 16, no. 10, October 1973, pp. 615-618.

The reference count technique for lists was first given in "A method for overlapping and erasure of lists," by G. Collins, CACM, vol. 3, no. 12, December 1960, pp. 655-657.

The algorithm MARK2 is adapted from: "An efficient machine-independent procedure for garbage collection in various list structures," by H. Schorr and W. Waite, CACM, vol. 10, no. 8, Aug. 1967, p. 501-506.

More list copying and marking algorithms may be found in "Copying list structures using bounded workspace," by G. Lindstrom, CACM, vol. 17, no. 4, April 1974, p. 198-202, "A nonrecursive list moving algorithm," by E. Reingold, CACM, vol. 16, no. 5, May 1973, p. 305-307, "Bounded workspace garbage collection in an address-order preserving list processing environment," by D. Fisher, Information Processing Letters, vol. 3, no. 1, July 1974, p. 29-32.

For string processing systems see the early works "COMIT" by V. Yngve, CACM. vol. 6, no. 3, March 1963, pp. 83-84. SNOBOL, A String Manipulation Language by D. Farber, R. Griswold and I. Polonsky, JACM, vol. 11, no. 1, 1964, pp. 21-30. "String processing techniques" by S. Madnick, CACM, vol. 10, no. 7, July 1967, pp. 420-427. "Fast Pattern Matching in Strings" by D. Knuth, J. Morris and V. Pratt Stanford Technical Report #74-440, August 1974.

EXERCISES

For exercises 1-6 assume that each node has two fields: DATA and LINK. Also assume the existence of subalgorithms GETNODE(X)and RET(X)to get and return nodes from/to the storage pool.

1.     Write an algorithm LENGTH(P) to count the number of nodes in a singly linked list P, where P points to the first node in the list. The last node has link field 0.

2.     Let P be a pointer to the first node in a singly linked list and X an arbitrary node in this list. Write an algorithm to delete this node from the list. If X = P, then P should be reset to point to the new first node in the list.

3.     Let X = (x1, x2, ..., xn) and Y = (y1, y2, ..., ym) be two linked lists. Write an algorithm to merge the two lists together to obtain the linked list Z = (x1, y1, x2, y2, ..., xm, ym, xm+1, ... xn) if m   [image]  n and Z = (x1, y1, x2, y2, ..., xn, yn, yn+1, ..., ym) if m > n. No additional nodes may be used.

4.     Do exercise 1 for the case of circularly linked lists.

5.     Do exercise 2 for the case of circularly linked lists.

6.     Do exercise 3 for the case of circularly linked lists.

7.     Devise a representation for a list where insertions and deletions can be made at either end. Such a structure is called a deque. Write a procedure for inserting and deleting at either end.

8.     Consider the hypothetical data object X2. X2 is a linear list with the restriction that while additions to the list may be made at either end, deletions can be made from one end only.

Design a linked list representation for X2. Write addition and deletion algorithms for X2. Specify initial and boundary conditions for your representation.

9.     Give an algorithm for a singly linked circular list which reverses the direction of the links.

10.     Let P be a pointer to a circularly linked list. Show how this list may be used as a queue. I.e., write algorithms to add and delete elements. Specify the value for P when the queue is empty.

11.     It is possible to traverse a singly linked list in both directions (i.e., left to right and a restricted right to left traversal) by reversing the links during the left to right traversal. A possible configuration for a list P under this scheme would be:

P points to the node currently being examined and L to the node on its left. Note that all nodes to the left of P have their links reversed.

i)     Write an algorithm to move P, n nodes to the right from a given position (L,P).

ii)     Write an algorithm to move P, n nodes left from any given position (L,P).

12.     Consider the operation XOR (exclusive OR, also written as   [image]) defined as below (for i,j binary):

This differs from the usual OR of logic in that

The definition can be extended to the case where i and j are binary strings (i.e., take the XOR of corresponding bits of i and j). So, for example, if i = 10110 and j = 01100, then i XOR j = i   [image]  j = 11010.

Note that a   [image]  (a   [image]  b) = (a   [image]  a) = b

and that (a   [image]  b)   [image]  = a   [image]  (b   [image]  b) = a

This gives us a space saving device for storing the right and left links of a doubly linked list. The nodes will now have only two fields: INFO and LINK. If L1 is to the left of node X and R1 to its right, then LINK(X) = L1   [image]  R1. For the leftmost node L1 = 0 and for the rightmost node R1 = 0.

Let (L,R) be a doubly linked list so represented. L points to the left most node and R to the right most node in the list.

i)     Write an algorithm to traverse the doubly linked list (L,R) from left to right listing out the contents of the INFO field of each node.

ii)     Write an algorithm to traverse the list right to left listing out the contents of the INFO field of each node.

13.     Write an algorithm PREAD(X) to read in n pairs of coefficients and exponents, (ci,ei) 1   [image]  i   [image]  n of a univariate polynomial, X, and to convert the polynomial into the circular linked list structure of section 4.4. Assume e1 > ei+1, 1   [image]  i < n, and that ci   [image]  0, 1   [image]  i   [image]  n. Your algorithm should leave X pointing to the head node. Show that this operation can be performed in time O(n).

14.     Let A and B be pointers to the head nodes of two polynomials represented as in exercise 13. Write an algorithm to compute the product polynomial C = A * B. Your algorithm should leave A and B unaltered and create C as a new list. Show that if n and m are the number of terms in A and B respectively, then this multiplication can be carried out in time O(nm2) or O(mn2). If A, B are dense show that the multiplication takes O(mn).

15.     Let A be a pointer to the head node of a univariate polynomial as in section 4.4. Write an algorithm, PEVAL(A,x) to evaluate the polynomial A at the point x, where x is some real number.

16.     Extend the equivalence algorithm of section 4.6 to handle the case when the variables being equivalenced may be subscripted. The input now consists of 4-tuples (i, ioff, j, joff) where i and j as before represent the variables. Ioff and joff give the positions in i and j being equivalenced relative to the start of i or j. For example, if i represents the array A dimensioned A(-6:10) and j, the array B dimensioned B(1:20), the equivalence EQUIVALENCE(A(5),B(6)) will be represented by 4-tuple (i,12, j,6). Your algorithm will begin by reading in these 4-tuples and setting them up in lists as in section 4.6. Each node on a list will have three fields: IOFF, JVAR, JOFF. Thus, for the equivalence 4-tuple above, the node   [image]  will be put into the list for i and the node   [image]  onto the list for j. Now process these lists outputting equivalence classes. With each class, output the relative position of each member of the class and check for conflicting equivalences .

In exercises 17-21 the sparse matrices are represented as in section 4.7

17.     Let A and B be two sparse matrices represeted as in section 4.7. Write an algorithm. MADD(A,B,C) to create the matrix C = A + B. Your algorithm should leave the matrices A and B unchanged and set up C as a new matrix in accordance with this data representation. Show that if A and B are n x m matrices with rA and rB nonzero terms, then this addition can be carried out in O(n + m + rA + rB) time.

18.     Let A and B be two sparse matrices. Write an algorithm MMUL(A,B,C) to set up the structure for C = A * B. Show that if A is a n x m matrix with rA nonzero terms and if B is a m x p matrix with rB nonzero terms, then C can be computed in time O(prA + nrB). Can you think of a way to compute C in O(min{prA,nrB})?

19.     Write an algorithm to write out the terms of a sparse matrix A as triples (i,j,aij). The terms are to be output by rows and within rows by columns. Show that this operation can be performed in time O(n + rA) if there are rA nonzero terms in A and A is a n x m matrix.

20.     Write an algorithm MTRP(A,B) to compute the matrix B = AT, the transpose of the sparse matrix A. What is the computing time of your algorithm?

21.     Design an algorithm to copy a sparse matrix. What is the computing time of your method?

22.     A simpler and more efficient representation for sparse matrices can be obtained when one is restricted to the operations of addition, subtraction and multiplication. In this representation nodes have the same fields as in the representation of section 4.7. Each nonzero term is represented by a node. These nodes are linked together to form two circular lists. The first list, the rowlist, is made up by linking nodes by rows and within rows by columns. This is done via the RIGHT field. The second list, the column list, is made up by linking nodes via the DOWN field. In this list, nodes are linked by columns and within columns by rows. These two lists share a common head node. In addition, a node is added to contain the dimensions of the matrix. The matrix A of figure 4.11 has the representation shown on page 206.

Using the same assumptions as for algorithm MREAD of section 4.7 write an algorithm to read in a matrix A and set up its internal representation as above. How much time does your algorithm take? How much additional space is needed?

23.     For the representation of exercise 22 write algorithms to

(i)     erase a matrix

(ii)     add two matrices

(iii)     multiply two matrices

(iv)     print out a matrix

For each of the above obtain computing times. How do these times compare with the corresponding times for the representation of section 4.7?

* solid links represent row list

   dashed links represent column list

Representation of matrix A of figure 4.11 using the scheme of exercise 22.

24.     Compare the sparse representations of exercise 22 and section 4.7 with respect to some other operations. For example how much time is needed to output the entries in an arbitrary row or column?

25.     (a)     Write an algorithm BF(n,p) similar to algorithm FF of section 4.8 to allocate a block of size n using a best fit strategy. Each block in the chain of available blocks has a SIZE field giving the number of words in that block. The chain has a head node, AV (see Figure 4.16). The best fit strategy examines each block in this chain. The allocation is made from the smallest block of size   [image]  n. P is set to the starting address of the space allocated.

(b)     Which of the algorithms BF and FF take less time?

(c)     Give an example of a sequence of requests for memory and memory freeing that can be met by BF but not by FF.

(d)     Do (c) for a sequence that can be met by FF but not by BF.

26.     Which of the two algorithms ALLOCATE and FREE of section 4.8 require the condition TAG(0) = TAG(m + 1) = 1 in order to work right? Why?

27.     The boundary tag method for dynamic storage management maintained the available space list as a doubly linked list. Is the XOR scheme for representing doubly linked lists (see exercise 12) suitable for this application? Why?

28.     Design a storage management scheme for the case when all requests for memory are of the same size, say k. Is it necessary to coalesce adjacent blocks that are free? Write algorithms to free and allocate storage in this scheme.

29.     Consider the dynamic storage management problem in which requests for memory are of varying sizes as in section 4.8. Assume that blocks of storage are freed according to the LAFF discipline (Last Allocated First Freed).

i)     Design a structure to represent the free space.

ii)     Write an algorithm to allocate a block of storage of size n.

iii)     Write an algorithm to free a block of storage of size n beginning at p.

30.     In the case of static storage allocation all the requests are known in advance. If there are n requests r1,r2, ...,rn and   [image] ri   [image]  M where M is the total amount of memory available, then all requests can be met. So, assume   [image] ri> M.

  i)     Which of these n requests should be satisfied if we wish to maximize the number of satisfied requests?

 ii)     Under the maximization criteria of (i), how small can the ratio

      storage allocated

      ----------------- get?

             M

iii)     Would this be a good criteria to use if jobs are charged a flat rate, say $3 per job, independent of the size of the request?

iv)     The pricing policy of (iii) is unrealistic when there can be much variation in request size. A more realistic policy is to charge say x cents per unit of request. Is the criteria of (i) a good one for this pricing policy? What would be a good maximization criteria for storage allocation now?

Write an algorithm to determine which requests are to be satisfied now. How much time does your algorithm take as a function of n, the number of requests? [If your algorithm takes a polynomial amount of time, and works correctly, take it to your instructor immediately. You have made a major discovery.]

31.     [Buddy System] The text examined the boundary tag method for dynamic storage management. The next 6 exercises will examine an alternative approach in which only blocks of size a power of 2 will be allocated. Thus if a request for a block of size n is made, then a block of size 2  [image] log2n  [image]  is allocated. As a result of this, all free blocks are also of size a power of 2. If the total memory size is 2m addressed from 0 to 2m - 1, then the possible sizes for free blocks are 2k, 0   [image]  k   [image]  m. Free blocks of the same size will be maintained in the same available space list. Thus, this system will have m + 1 available space lists. Each list is a doubly linked circular list and has a head node AVAIL(i), 0   [image]  i   [image]  m. Every free node has the following structure:

Initially all of memory is free and consists of one block beginning at 0 and of size 2m. Write an algorithm to initialize all the available space lists.

32.     [Buddy System Allocation] Using the available space list structure of exercise 31 write an algorithm to meet a request of size n if possible. Note that a request of size n is to be met by allocating a block of size 2k k = [log2n]. To do this examine the available space lists AVAIL(i), k   [image]  i   [image]  m finding the smallest i for which AVAIL(i) is not empty. Remove one block from this list. Let P be the starting address of this block. If i > k, then the block is too big and is broken into two blocks of size 2i-1 beginning at P and P + 2i-1 respectively. The block beginning at P + 2i-1 is inserted into the corresponding available space list. If i - 1 > k, then the block is to be further split and so on. Finally, a block of size 2k beginning at P is allocated. A block in use has the form:

i)     Write an algorithm using the strategy outlined above to allocate a block of storage to meet a request for n units of memory.

ii)     For a memory of size 2m = 16 draw the binary tree representing the splitting of blocks taking place in satisfying 16 consecutive requests for memory of size 1. (Note that the use of the TAG in the allocated block does not really create a problem in allocations of size 1 since memory would be allocated in units where 1 unit may be a few thousand words.) Label each node in this tree with its starting address and present KVAL, i.e. power of 2 representing its size.

33.     [Locating Buddies] Two nodes in the tree of exercise 32 are said to be buddies if they are sibling nodes. Prove that two nodes starting at x and y respectively are buddies iff:

 i)     the KVALS for x and y are the same: and

ii)     x = y   [image]  2k where   [image]  is the exclusive OR (XOR) operation defined in exercise 12. The   [image]  is taken pair wise bit wise on the binary representation of y and 2k.

34.     [Freeing and Coalescing Blocks] When a block with KVAL k becomes free it is to be returned to the available space list. Free blocks are combined into bigger free blocks iff they are buddies. This combining follows the reverse process adopted during allocation. If a block beginning at P and of size k becomes free, it is to be combined with its buddy P   [image]  2k if the buddy is free. The new free block beginning at L = min {P,P   [image]  2k}and of size k + 1 is to be combined with its buddy L   [image]  2k+1 if free and so on. Write an algorithm to free a block beginning at P and having KVAL k combining buddies that are free.

35.   i)     Does the freeing algorithm of exercise 34 always combine adjacent free blocks? If not, give a      sequence of allocations and freeings of storage showing this to be the case.

                                          storage requested

 ii)     How small can the ratio------------------------be for the Buddy System?

                                                  storage allocated

     Storage requested =   [image] ni where ni is actual amount requested. Give an example approaching this ratio.

iii)     How much time does the allocation algorithm take in the worst case to make an allocation if the total memory size is 2m?

iv)     How much time does the freeing algorithm take in the worst case to free a block of storage?

36.     [Buddy system when memory size is not a power of 2]

 i)     How are the available space lists to be initialized if the total storage available is not a power of 2?

ii)     What changes are to be made to the block freeing algorithm to take care of this case? Do any changes have to be made to the allocation algorithm?

37.     Write a nonrecursive version of algorithm LERASE(X) of section 4.9.

38.     Write a nonrecursive version of algorithm EQUALS(S,T) of section 4.9.

39.     Write a nonrecursive version of algorithm DEPTH(S) of section 4.9.

40.     Write a procedure which takes an arbitrary nonrecursive list L with no shared sublists and inverts it and all of its sublists. For example, if L = (a, (b,c)), then inverse (L) = ((c,b),a).

41.     Devise a procedure that produces the list representation of an arbitrary list given its linear form as a string of atoms, commas, blanks and parentheses. For example, for the input L = (a,(b,c)) your procedure should produce the structure:

42.     One way to represent generalized lists is through the use of two field nodes and a symbol table which contains all atoms and list names together with pointers to these lists. Let the two fields of each node be named ALINK and BLINK. Then BLINK either points to the next node on the same level, if there is one, or is a zero. The ALINK field either points to a node at a lower level or, in the case of an atom or list name, to the appropriate entry in the symbol table. For example, the list B(A,(D,E),( ),B) would have the representation:

(The list names D and E were already in the table at the time the list B was input. A was not in the table and so assumed to be an atom.)

The symbol table retains a type bit for each entry. Type = 1 if the entry is a list name and type = 0 for atoms. The NIL atom may either be in the table or ALINKS can be set to 0 to represent the NIL atom. Write an algorithm to read in a list in parenthesis notation and to set up its linked representation as above with X set to point to the first node in the list. Note that no head nodes are in use. The following subalgorithms may be used by LREAD:

 i)     GET(A,P) ... searches the symbol table for the name A. P is set to 0 if A is not found in the table, otherwise, P is set to the position of A in the table.

ii)     PUT(A,T,P) ... enters A into the table. P is the position at which A was entered. If A is already in the table, then the type and address fields of the old entry are changed. T = 0 to enter an atom or T > 0 to enter a list with address T. (Note: this permits recursive definition of lists using indirect recursion).

iii)     NEXT TOKEN ... gets next token in input list. (A token may be a list name, atom,'(',')'or ','. A '  [image] 'is returned if there are no more tokens.)

iv)     GETNODE(X) ... gets a node for use.

You may assume that the input list is syntactically correct. In case a sublist is labeled as in the list C(D,E(F,G)) the structure should be set up as in the case C(D,(F,G)) and E should be entered into the symbol table as a list with the appropriate storing address.

43.     Rewrite algorithm MARK1 of section 4.10 using the conventions of section 4.9 for the tag field.

44.     Rewrite algorithm MARK1 of section 4.10 for the case when each list and sublist has a head node. Assume that the DLINK field of each head node is free and so may be used to maintain a linked stack without using any additional space. Show that the computing time is still O(m).

45.     When the DLINK field of a node is used to retain atomic information as in section 4.9, implementing the marking strategy of MARK2 requires an additional bit in each node. In this exercise we shall explore a marking strategy which does not require this additional bit. Its worst case computing time will however be O(mn) where m is the number of nodes marked and n the total number of nodes in the system. Write a marking algorithm using the node structure and conventions of section 4.9. Each node has the fields: MARK, TAG, DLINK and RLINK. Your marking algorithm will use variable P to point to the node currently being examined and NEXT to point to the next node to be examined. If L is the address of the as yet unexplored list node with least address and P the address of the node currently being examined then the value of NEXT will be min {L,P + 1 }. Show that the computing time of your algorithm is O(mn).

46.     Prove that MARK2(X) marks all unmarked nodes accessible from X.

47.     Write a composite marking algorithm using MARK1, MARK2 and a fixed amount M of stack space. Stack nodes as in MARK1 until the stack is full. When the stack becomes full, revert to the strategy of MARK2. On completion of MARK2, pick up a node from the stack and explore it using the composite algorithm. In case the stack never overflows, the composite algorithm will be as fast as MARK1. When M = 0, the algorithm essentially becomes MARK2. The computing time will in general be somewhere in between that of MARK1 and MARK2.

48.     Write a storage compaction algorithm to be used following the marking phase of garbage collection. Assume that all nodes are of a fixed size and can be addressed NODE(i), 1   [image]  i   [image]  m. Show that this can be done in two phases, where in the first phase a left to right scan for free nodes and a right to left scan for nodes in use is carried out. During this phase, used nodes from the right end of memory are moved to free positions at the left end. The relocated address of such nodes is noted in one of the fields of the old address. At the end of this phase all nodes in use occupy a contiguous chunk of memory at the left end. In the second phase links to relocated nodes are updated.

49.     Write a compaction algorithm to be used in conjunction with the boundary tag method for storage management. Show that this can be done in one left to right scan of memory blocks. Assume that memory blocks are independent and do not reference each other. Further assume that all memory references within a block are made relative to the start of the block. Assume the start of each in-use block is in a table external to the space being allocated.

50.     Design a dynamic storage management system in which blocks are to be returned to the available space list only during compaction. At all other times, a TAG is set to indicate the block has become free. Assume that initially all of memory is free and available as one block. Let memory be addressed 1 through M. For your design, write the following algorithms:

(i)     ALLOCATE(n,p) ... allocates a block of size n; p is set to its starting address. Assume that size n includes any space needed for control fields in your design.

(ii)     FREE(n,p) ... free a block of size n beginning at p.

(iii)     COMPACT ... compact memory and reinitialize the available space list. You may assume that all address references within a block are relative to the start of the block and so no link fields within blocks need be changed. Also, there are no interblock references.

51.     Write an algorithm to make an in-place replacement of a substring of X by the string Y. Assume that strings are represented using fixed size nodes and that each node in addition to a link field has space for 4 characters. Use   [image]  to fill up any unused space in a node.

52.     Using the definition of STRING given in section 4.11, simulate the axioms as they would apply to (i) CONCAT(S,T) (ii) SUBSTR(S,2,3), (iii) INDEX(S,T) where S = 'abcde' and T= 'cde'.

53.     If X = (x1, ...,xm) and Y = (y1, ...,yn) are strings where xi and yi are letters of the alphabet, then X is less than Y if xi = yi for 1   [image]  i   [image]  j and xj < yj or if xi = yi for 1   [image]  i   [image]  m and m < n. Write an algorithm which takes two strings X,Y and returns either -1, 0, +1 if X < Y, X = Y or X > Y respectively.

54.     Let X and Y be strings represented as singly linked lists. Write a procedure which finds the first character of X which does not occur in the string Y.

55.     Show that the computing time for procedure NFIND is still O(mn). Find a string and a pattern for which this is true.

56.     (a)     Compute the failure function for the following patterns:

(i) a a a a b

(ii) a b a b a a

(iii) a b a a b a a b b

(b)     For each of the above patterns obtain the linked list representations including the field NEXT as in figure 4.41.

(c)     let p1 p2 ... pn be a pattern of length n. Let f be its failure function. Define f1(j) = f(j) and fm(j) = f(fm-1(j)), 1   [image]  j   [image]  n and m > 1. Show using the definition of f that:

57.     The definition of the failure function may be strengthened to

(a)     Obtain the new failure function for the pattern PAT of the text.

(b)     Show that if this definition for f is used in the definition of NEXT then algorithm PMATCH still works correctly.

(c)     Modify algorithm FAIL to compute f under this definition. Show that the computing time is still O(n).

(d)     Are there any patterns for which the observed computing time of PMATCH is more with the new definition of f than with the old one? Are there any for which it is less? Give examples.

58.     [Programming Project]

Design a linked allocation system to represent and manipulate univariate polynomials with integer coefficients (use circular linked lists with head nodes). Each term of the polynomial will be represented as a node. Thus, a node in this system will have three fields as below:

           ______________________

          |             |        |

          |  Exponent   |   Link |

          |_____________|________|

          |    Coefficient       |

          |______________________|

For purposes of this project you may assume that each field requires one word and that a total of 200 nodes are available.

The available space list is maintained as a chain with AVAIL pointing to the first node. To begin with, write computer subprograms or procedures to perform the following basic list handling tasks:

i)     INIT(N) ... initialize the storage pool. I.e., set it up as a singly linked list linked through the field LINK with AVAIL pointing to the first node in this list.

ii)     GETNODE(X) ... provides the node X from the storage pool for use. It is easiest to provide the first node in AVAIL if there is a free node.

iii)     RET(X) ... return node X to the storage pool.

iv)     LENGTH(X) ... determines the number of nodes in the list X where X may be either a circular list or a chain.

The external (i.e., for input and output ) representation of a univariate polynomial will be assumed to be a sequence of integers of the form: ne1 c1 e2 c2 e3 c3 ... en cn, where the ei represents the exponents and the ci the coefficients. n gives the number of terms in the polynomial. The exponents are in decreasing order, i.e., e1 > e2 > ... > en.

Write and test (using the routines i-iv above) the following routines:

v)     PREAD (X) ... read in an input polynomial and convert it to a circular list representation using a head node. X is set to point to the head node of this polynomial.

vi)     PWRITE(X) ... convert the polynomial X from its linked list representation to external representation and print it out.

     Note: Both PREAD and PWRITE may need a buffer to hold the input while setting up the list or to assemble the output. Assuming that at most 10 terms may appear on an input or output line, we can use the two arrays E(10) and C(10) for this.

vii)     PADD(X, Y, Z) ... Z = X + Y

viii)     PSUB(X, Y, Z) ... Z = X - Y

ix)     PMUL(X, Y, Z) ... Z = X * Y

x)     PEVAL(X, A, V) ... A is a real constant and the polynomial X is evaluated at the point A. V is set to this value.

     Note: Routines vi-x should leave the input polynomials unaltered after completion of their respective tasks.

xi)     PERASE(X) ... return the circular list X to the storage pool.

     Use the routine LENGTH to check that all unused nodes are returned to the storage pool.

     E.g., LO = LENGTH(AVAIL)

     CALL PMUL(X, Y, Z)

     LN = LENGTH(Z) + LENGTH(AVAIL)

     should result in LN= LO.

59.     [Programming Project] In this project, we shall implement a complete linked list system to perform arithmetic on sparse matrices using the representation of section 4.7. First, design a convenient node structure assuming VALUE is an integer for the computer on which the program is to be run. Then decide how many bits will be used for each field of the node. In case the programming language you intend to use is FORTRAN, then write function subprograms to extract various fields that may be packed into a word. You will also need subroutine subprograms to set certain fields. If your language permits use of structure variables, then these routines would not be needed. To begin with, write the basic list processing routines:

a)     INIT(N) ... initialize the available space list to consist of N nodes. Depending on your design each node may be 2 or 3 words long.

b)     GETNODE(X) ... provide node X for use.

c)     RET(X) ... return node X to the available space list.

     Test these routines to see if they work. Now write and test these routines for matrix operations:

d)     MREAD(A) ... read matrix A and set up according to the representation of section 4.7. The input has the following format:

         line 1:       n m r  n = # or rows

                              m = # or columns

                              r = # of nonzero terms

         line 2

                         triples of (row, columns, value) on each line

         line r + 1

     These triples are in increasing order by rows. Within rows, the triples are in increasing order of columns. The data is to be read in one card at a time and converted to internal representation. The variable A is set to point to the head node of the circular list of head nodes (as in the text).

e)     MWRITE(A) ... print out the terms of A. To do this, you will have to design a suitable output format. In any case, the output should be ordered by rows and within rows by columns.

f)     MERASE(A) . . . return all nodes of the sparse matrix A to the available space list.

g)     MADD(A,B,C) ... create the sparse matrix C = A + B. A and B are to be left unaltered.

h)     MSUB(A,B,C) ... C = A - B

i)     MMUL(A,B,C) ... create the sparse matrix C = A * B. A and B are to be left unaltered.

j)     MTRP(A,B) ... create the sparse matrix B = AT. A is to be left unaltered.

60.      [Programming Project] Do the project of exercise 59 using the matrix representation of exercise 22.

61.      (Landweber)

This problem is to simulate an airport landing and takeoff pattern. The airport has 3 runways, runway l, runway 2 and runway 3. There are 4 landing holding patterns, two for each of the first two runways. Arriving planes will enter one of the holding pattern queues, where the queues are to be as close in size as possible. When a plane enters a holding queue, it is assigned an integer ID number and an integer giving the number of time units the plane can remain in the queue before it must land (because of low fuel level). There is also a queue for takeoffs for each of the three runways. Planes arriving in a takeoff queue are also assigned an integer ID. The takeoff queues should be kept the same size.

At each time 0-3 planes may arrive at the landing queues and 0-3 planes may arrive at the takeoff queues. Each runway can handle one takeoff or landing at each time slot. Runway 3 is to be used for takeoffs except when a plane is low on fuel. At each time unit, planes in either landing queue whose air time has reached zero must be given priority over other landings and takeoffs. If only one plane is in this category, runway 3 is to be used. If more than one, then the other runways are also used (at each time at most 3 planes can be serviced in this way).

Use successive even (odd) integers for ID's of planes arriving at takeoff (landing) queues. At each time unit assume that arriving planes are entered into queues before takeoffs or landings occur. Try to design your algorithm so that neither landing nor takeoff queues grow excessively. However, arriving planes must be placed at the ends of queues. Queues cannot be reordered.

The output should clearly indicate what occurs at each time unit. Periodically print (a) the contents of each queue; (b) the average takeoff waiting time; (c) the average landing waiting time; (d) the average flying time remaining on landing; and (e) the number of planes landing with no fuel reserve. (b)-(c) are for planes that have taken off or landed respectively. The output should be self explanatory and easy to understand (and uncluttered).

The input can be on cards (terminal, file) or it can be generated by a random number generator. For each time unit the input is of the form:

Go to Chapter 5     Back to Table of Contents


Fundamentals: CHAPTER 5: TREES

 Return to Table of Contents

CHAPTER 5: TREES

5.1 BASIC TERMINOLOGY

In this chapter we shall study a very important data object, trees. Intuitively, a tree structure means that the data is organized so that items of information are related by branches. One very common place where such a structure arises is in the investigation of genealogies. There are two types of genealogical charts which are used to present such data: the pedigree and the lineal chart. Figure 5.1 gives an example of each.

The pedigree chart shows someone's ancestors, in this case those of Dusty, whose two parents are Honey Bear and Brandy. Brandy's parents are Nuggett and Coyote, who are Dusty's grandparents on her father's side. The chart continues one more generation farther back to the great-grandparents. By the nature of things, we know that the pedigree chart is normally two-way branching, though this does not allow for inbreeding. When that occurs we no longer have a tree structure unless we insist that each occurrence of breeding is separately listed. Inbreeding may occur frequently when describing family histories of flowers or animals.

The lineal chart of figure 5.1(b), though it has nothing to do with people, is still a genealogy. It describes, in somewhat abbreviated form, the ancestry of the modern European languages. Thus, this is a chart of descendants rather than ancestors and each item can produce several others. Latin, for instance, is the forebear of Spanish, French, Italian and Rumanian. Proto Indo-European is a prehistoric language presumed to have existed in the fifth millenium B.C. This tree does not have the regular structure of the pedigree chart, but it is a tree structure nevertheless .

With these two examples as motivation let us define formally what we mean by a tree.

Figure 5.1 Two Types of Geneological Charts

Definition: A tree is a finite set of one or more nodes such that: (i) there is a specially designated node called the root; (ii) the remaining nodes are partitioned into n [image] 0 disjoint sets T1, ...,Tn where each of these sets is a tree. T1, ...,Tn are called the subtrees of the root.

Again we have an instance of a recursive definition (compare this with the definition of a generalized list in section 4.9). If we return to Figure 5.1 we see that the roots of the trees are Dusty and Proto Indo-European. Tree (a) has two subtrees whose roots are Honey Bear and Brandy while tree (b) has 3 subtrees with roots Italic, Hellenic, and Germanic. The condition that T1, ...,Tn be disjoint sets prohibits subtrees from ever connecting together (no cross breeding). It follows that every item in a tree is the root of some subtree of the whole. For instance, West Germanic is the root of a subtree of Germanic which itself has three subtrees with roots: Low German, High German and Yiddish. Yiddish is a root of a tree with no subtrees.

There are many terms which are often used when referring to trees. A node stands for the item of information plus the branches to other items. Consider the tree in figure 5.2. This tree has 13 nodes, each item of data being a single letter for convenience. The root is A and we will normally draw trees with the root at the top. The number of subtrees of a node is called its degree. The degree of A is 3, of C is 1 and of F is zero. Nodes that have degree zero are called leaf or terminal nodes. {K,L,F,G,M,I,J} is the set of leaf nodes. Alternatively, the other nodes are referred to as nonterminals. The roots of the subtrees of a node, X, are the children of X. X is the parent of its children. Thus, the children of D are H, I, J; the parent of D is A. Children of the same parent are said to be siblings. H, I and J are siblings. We can extend this terminology if we need to so that we can ask for the grandparent of M which is D, etc. The degree of a tree is the maximum degree of the nodes in the tree. The tree of figure 5.2 has degree 3. The ancestors of a node are all the nodes along the path from the root to that node. The ancestors of M are A, D and H.

Figure 5.2 A Sample Tree

The level of a node is defined by initially letting the root be at level one. If a node is at level l, then its children are at level l + 1. Figure 5.2 shows the levels of all nodes in that tree. The height or depth of a tree is defined to be the maximum level of any node in the tree.

A forest is a set of n [image] 0 disjoint trees. The notion of a forest is very close to that of a tree because if we remove the root of a tree we get a forest. For example, in figure 5.2 if we remove A we get a forest with three trees.

There are other ways to draw a tree. One useful way is as a list. The tree of figure 5.2 could be written as the list

(A(B(E(K,L),F),C(G),D(H(M),I,J)))

The information in the root node comes first followed by a list of the subtrees of that node.

Now, how do we represent a tree in memory? If we wish to use linked lists, then a node must have a varying number of fields depending upon the number of branches.

            _____________________________________

           |      |        |      |      |       |

           |DATA  |LINK 1  |LINK 2|          |LINK n |

           |______|________|______|______|_______|

However it is often simpler to write algorithms for a data representation where the node size is fixed. Using data and pointer fields we can represent a tree using the fixed node size list structure we devised in Chapter 4. The list representation for the tree of Figure 5.2 is on page 222. We can now make use of many of the general procedures that we originally wrote for handling lists. Thus, the data object tree is a special instance of the data object list and we can specialize the list representation scheme to them. In a later section we will see that another data object which can be used to represent a tree is the data object binary tree.

List representation for the tree of figure 5.2

5.2 BINARY TREES

A binary tree is an important type of tree structure which occurs very often. It is characterized by the fact that any node can have at most two branches, i.e., there is no node with degree greater than two. For binary trees we distinguish between the subtree on the left and on the right, whereas for trees the order of the subtrees was irrelevant. Also a binary tree may have zero nodes. Thus a binary tree is really a different object than a tree.

Definition: A binary tree is a finite set of nodes which is either empty or consists of a root and two disjoint binary trees called the left subtree and the right subtree.

Using the notation introduced in chapter one we can define the data structure binary tree as follows:

structure BTREE

declare CREATE( )    btree

ISMTBT(btree)    boolean

MAKEBT(btree,item,btree)    btree

LCHILD(btree)    btree

DATA(btree)    item

RCHILD(btree)    btree

for all p,r    btree, d    item let

ISMTBT(CREATE) :: = true

ISMTBT(MAKEBT(p,d,r)) :: = false

LCHILD(MAKEBT(p,d,r)):: = p; LCHILD(CREATE):: = error

DATA(MAKEBT(p,d,r)) :: = d; DATA(CREATE) :: = error

RCHILD(MAKEBT(p,d,r)) :: = r; RCHILD(CREATE) :: = error

end

end BTREE

This set of axioms defines only a minimal set of operations on binary trees. Other operations can usually be built in terms of these. See exercise 35 for an example.

The distinctions between a binary tree and a tree should be analyzed. First of all there is no tree having zero nodes, but there is an empty binary tree. The two binary trees below

are different. The first one has an empty right subtree while the second has an empty left subtree. If the above are regarded as trees, then they are the same despite the fact that they are drawn slightly differently.

Figure 5.3 Two Sample Binary Trees

Figure 5.3 shows two sample binary trees. These two trees are special kinds of binary trees. The first is a skewed tree, skewed to the left and there is a corresponding one which skews to the right. Tree 5.3b is called a complete binary tree. This kind of binary tree will be defined formally later on. Notice that all terminal nodes are on adjacent levels. The terms that we introduced for trees such as: degree, level, height, leaf, parent, and child all apply to binary trees in the natural way. Before examining data representations for binary trees, let us first make some relevant observations regarding such trees. First, what is the maximum number of nodes in a binary tree of depth k?

Lemma 5.1

(i) The maximum number of nodes on level i of a binary tree is 2i-1, i [image] 1 and

(ii) The maximum number of nodes in a binary tree of depth k is 2k - 1, k [image] 1.

Proof:

(i) The proof is by induction on i.

Induction Base: The root is the only node on level i= 1. Hence the maximum number of nodes on level i = 1 is 2 [image] = 2i - 1.

Induction Hypothesis: For all j, 1 [image] j < i, the maximum number of nodes on level j is 2j - 1.

Induction Step: The maximum number of nodes on level i - 1 is 2i - 2, by the induction hypothesis. Since each node in a binary tree has maximum degree 2, the maximum number of nodes on level i is 2 times the maximum number on level i - 1 or 2i-1.

(ii) The maximum number of nodes in a binary tree of depth k is [image] (maximum number of nodes on level i)

Next, let us examine the relationship between the number of terminal nodes and the number of nodes of degree 2 in a binary tree.

Lemma 5.2: For any nonempty binary tree, T, if no is the number of ter minal nodes and n2 the number of nodes of degree 2, then n0 = n2 + 1.

Proof: Let n1 be the number of nodes of degree 1 and n the total number of nodes. Since all nodes in T are of degree [image] 2 we have:

n = n0 + n1 + n2

(5.1)

If we count the number of branches in a binary tree, we see that every node except for the root has a branch leading into it. If B is the number of branches, then n = B + 1. All branches emanate either from a node of degree one or from a node of degree 2. Thus, B = n1 + 2n2. Hence, we obtain

n = 1 + n1 + 2n2

(5.2)

Subtracting (5.2) from (5.1) and rearranging terms we get

n0= n2 + 1

In Figure 5.3(a) n0 = 1 and n2 = 0 while in Figure 5.3(b) n0 = 5 and n2 = 4.

As we continue our discussion of binary trees, we shall derive some other interesting properties.

5.3 BINARY TREE REPRESENTATIONS

A full binary tree of depth k is a binary tree of depth k having 2k - 1 nodes. By Lemma 5.1, this is the maximum number of nodes such a binary tree can have. Figure 5.4 shows a full binary tree of depth 4. A very elegant sequential representation for such binary trees results rom sequentially numbering the nodes, starting with nodes on level 1, then those on level 2 and so on. Nodes on any level are numbered from left to right (see figure 5.4). This numbering scheme gives us the definition of a complete binary tree. A binary tree with n nodes and of depth k is complete iff its nodes correspond to the nodes which re numbered one to n in the full binary tree of depth k. The nodes may now be stored in a one dimensional array, TREE, with the node umbered i being stored in TREE(i). Lemma 5.3 enables us to easily determine the locations of the parent, left child and right child of any node i in the binary tree.

Figure 5.4 Full Binary Tree of Depth 4 with Sequential Node Numbers

Lemma 5.3: If a complete binary tree with n nodes (i.e., depth= [log2n] + 1) is represented sequentially as above then for any node with index i, 1 [image] i [image] n we have:

(i) PARENT(i) is at [i/2] if i [image] 1. When i = 1, i is the root and has no parent.

(ii) LCHILD(i) is at 2i if 2i [image] n. If 2i > n, then i has no left child.

(iii) RCHILD(i) is at 2i + 1 if 2i + 1 [image] n. If 2i + 1 > n, then i has no right child.

Proof: We prove (ii). (iii) is an immediate consequence of (ii) and the numberin g of nodes on the same level from left to right. (i) follows from (ii) and (iii). We prove (ii) by induction on i. For i= 1, clearly the left child is at 2 unless 2 > n in which case 1 has no left child. Now assume that for all j, 1 [image] j [image] i, LCHILD(j) is at 2j. Then, the two nodes immediately preceding LCHILD (i + 1) in the representation are the right child of i and the left child of i. The left child of i is at 2i. Hence, the left child of i + 1 is at 2i + 2 = 2(i + 1) unless 2(i + 1) > n in which case i + 1 has no left child.

This representation can clearly be used for all binary trees though in most cases there will be a lot of unutilized space. For complete binary trees the representation is ideal as no space is wasted. For the skewed tree of figure 5.3(a), however, less than half the array is utilized. In the worst case a skewed tree of depth k will require 2k - 1 spaces. Of these only k will be occupied.

      TREE      TREE

      -----     -----

(1)  |  A  |   |  A  |

     |-----|   |-----|

(2)  |  B  |   |  B  |

     |-----|   |-----|

(3)  |  -  |   |  C  |

     |-----|   |-----|

(4)  |  C  |   |  D  |

     |-----|   |-----|

(5)  |  -  |   |  E  |

     |-----|   |-----|

(6)  |  -  |   |  F  |

     |-----|   |-----|

(7)  |  -  |   |  G  |

     |-----|   |-----|

(8)  |  D  |   |  H  |

     |-----|   |-----|

(9)  |  -  |   |  I  |

     |-----|    -----

 .   |  .  |

     |-----|

 .   |  .  |

     |-----|

 .   |  .  |

     |-----|

(16) |  E  |

      -----

Figure 5.5 Array Representation of the Binary Trees of Figure 5.3

While the above representation appears to be good for complete binary trees it is wasteful for many other binary trees. In addition, the representation suffers from the general inadequacies of sequential representations. Insertion or deletion of nodes from the middle of a tree requires the movement of potentially many nodes to reflect the change in level number of these nodes. These problems can be easily overcome through the use of a linked representation. Each node will have three fields LCHILD, DATA and RCHILD as below:

While this node structure will make it difficult to determine the parent of a node, we shall see that for most applications, it is adequate. In case it is necessary to be able to determine the parent of random nodes. then a fourth field PARENT may be included. The representation of the binary trees of figure 5.3 using this node structure is given in figure 5.6.

Figure 5.6 Linked Representation for the Binary Trees of Figure 5.3.

5.4 BINARY TREE TRAVERSAL

There are many operations that we often want to perform on trees. One notion that arises frequently is the idea of traversing a tree or visiting each node in the tree exactly once. A full traversal produces a linear order for the information in a tree. This linear order may be familiar and useful. When traversing a binary tree we want to treat each node and its subtrees in the same fashion. If we let L, D, R stand for moving left, printing the data, and moving right when at a node then there are six possible combinations of traversal: LDR, LRD, DLR, DRL, RDL, and RLD. If we adopt the convention that we traverse left before right then only three traversals remain: LDR, LRD and DLR. To these we assign the names inorder, postorder and preorder because there is a natural correspondence between these traversals and producing the infix, postfix and prefix forms of an expression. Consider the binary tree of figure 5.7. This tree contains an arithmetic expression with binary operators: add(+), multiply(*), divide(/), exponentiation(**) and variables A, B, C, D, and E. We will not worry for now how this binary tree was formed, but assume that it is available. We will define three types of traversals and show the results for this tree.

Figure 5.7 Binary Tree with Arithmetic Expression

Inorder Traversal: informally this calls for moving down the tree towards the left until you can go no farther. Then you "visit" the node, move one node to the right and continue again. If you cannot move to the right, go back one more node. A precise way of describing this traversal is to write it as a recursive procedure.

procedure INORDER(T)

//T is a binary tree where each node has three fields L-

CHILD,DATA,RCHILD//

if T    0 then [call INORDER(LCHILD(T))

print(DATA(T))

call(INORDER(RCHILD(T))]

end INORDER

Recursion is an elegant device for describing this traversal. Let us trace how INORDER works on the tree of figure 5.7.

Call of   value

INORDER  in root    Action

-------  -------  ----------

MAIN        +

 1          *

 2          /

 3          A

 4          0     print('A')

 4          0     print('/')

 3         **

 4          B

 5          0     print('B')

 5          0     print('**')

 4          C

 5          0     print('C')

 5          0     print('*')

 2          D

 3          0     print('D')

 3          0     print('+')

 1          E

 2          0     print('E')

 2          0

The elements get printed in the order

A/B ** C * D + E

which is the infix form of the expression.

A second form of traversal is preorder:

procedure PREORDER (T)

//T is a binary tree where each node has three fields L-

CHILD,DATA,RCHILD//

if T    0 then [print (DATA(T))

call PREORDER(LCHILD(T))

call PREORDER(RCHILD(T))]]

end PREORDER

In words we would say "visit a node, traverse left and continue again. When you cannot continue, move right and begin again or move back until you can move right and resume." The nodes of figure 5.7 would be printed in preorder as

+* / A ** B C D E

which we recognize as the prefix form of the expression.

At this point it should be easy to guess the next traversal method which is called postorder:

procedure POSTORDER (T)

//T is a binary tree where each node has three fields L-

CHILD,DATA,RCHILD//

if T    0 then [call POSTORDER(LCHILD(T))

call POSTORDER(RCHILD(T))

print (DATA(T))]

end POSTORDER

The output produced by POSTORDER is

A B C ** / D * E +

which is the postfix form of our expression.

Though we have written these three algorithms using recursion, it is very easy to produce an equivalent nonrecursive procedure. Let us take inorder as an example. To simulate the recursion we need a stack which will hold pairs of values (pointer, returnad) where pointer points to a node in the tree and returnad to the place where the algorithm should resume after an end is encountered. We replace every recursive call by a mechanism which places the new pair (pointer, returnad) onto the stack and goes to the beginning; and where there is a return or end we insert code which deletes the top pair from the stack if possible and either ends or branches to returnad (see section 4.10 for the exact details).

Though this procedure seems highly unstructured its virtue is that it is semiautomatically produced from the recursive version using a fixed set of rules. Our faith in the correctness of this program can be justified

procedure INORDER1(T)

//a nonrecursive version using a stack of size n//

i    0      //initialize the stack//

L1: if T    0 then [i    i + 2; if i > n then call STACK_FULL

STACK (i - 1)    T; STACK(i)    'L2'

T    LCHILD(T); go to L1;    //traverse left

subtree//

L2: print (DATA(T))

i    i + 2; if i > n then call STACK--FULL

STACK (i - 1)    T; STACK(i)    'L3'

T    RCHILD(T); go to L1]   //traverse right

subtree//

L3: if i    0 then [//stack not empty, simulate a return//

T    STACK (i - 1); X    STACK (i)

i    i - 2; go to X]

end INORDER 1

if we first prove the correctness of the original version and then prove that the transformation rules result in a correct and equivalent program. Also we can simplify this program after we make some observations about its behavior. For every pair (T1,L3) in the stack when we come to label L3 this pair will be removed. All such consecutive pairs will be removed until either i gets set to zero or we reach a pair (T2,L2). Therefore, the presence of L3 pairs is useful in no way and we can delete that part of the algorithm. This means we can eliminate the two lines of code following print (DATA(T)). We next observe that this leaves us with only one return address, L2, so we need not place that on the stack either. Our new version now looks like:

procedure INORDER 2(T)

//a simplified, nonrecursive version using a stack of size n//

i    0       //initialize stack//

L1: if T    0 then [i    i + 1; if i > n then call STACK__FULL

STACK (i)    T; T    LCHILD(T); go to L1

L2: print (DATA(T))

T    RCHILD(T); go to L1]

if i    0 then [//stack not empty//

T    STACK (i): i    i - 1; go to L2]

end INORDER2

This program is considerably simpler than the previous version, but it may still offend some people because of the seemingly undisciplined use of go to's. A SPARKS version without this statement would be:

procedure INORDER3(T)

//a nonrecursive, no go to version using a stack of size n//

i    0 //initialize stack//

loop

while T    0 do          //move down LCHILD fields//

i    i + 1; if i > n then call STACK--FULL

STACK (i)    T; T    LCHILD(T)

end

if i = 0 then return

T    STACK (i); i    i - 1

print(DATA(T)); T    RCHILD(T)

forever

end INORDER3

In going from the recursive version to the iterative version INORDER3 one undesirable side effect has crept in. While in the recursive version T was left pointing to the root of the tree, this is not the case when INORDER3 terminates. This may readily be corrected by introducing another variable P which is used in place of T during the traversal.

What are the computing time and storage requirements of INORDER3 ? Let n be the number of nodes in T. If we consider the action of the above algorithm, we note that every node of the tree is placed on the stack once. Thus, the statements STACK(i) [image] T and T [image] STACK(i) are executed n times. Moreover, T will equal zero once for every zero link in the tree which is exactly

2n0 + n1 = n0 + n1 + n2 + 1 = n + 1.

So every step will be executed no more than some small constant times n or O(n). With some further modifications we can lower the constant (see exercises). The space required for the stack is equal to the depth of T. This is at most n.

Before we leave the topic of tree traversal, we shall consider one final question. Is it possible to traverse binary trees without the use of extra space for a stack? One simple solution is to add a PARENT field to each node. Then we can trace our way back up to any root and down again. Another solution which requires two bits per node is given in section 5.6. If the allocation of this extra space is too costly then we can use the method of algorithm MARK2 of section 4.10. No extra storage is required since during processing the LCHILD and RCHILD fields are used to maintain the paths back to the root. The stack of addresses is stored in the leaf nodes. The exercises examine this algorithm more closely.

5.5 MORE ON BINARY TREES

Using the definition of a binary tree and the recursive version of the traversals, we can easily write other routines for working with binary trees. For instance, if we want to produce an exact copy of a given binary tree we can modify the postorder traversal algorithm only slightly to get:

procedure COPY(T)

//for a binary tree T, COPY returns a pointer to an exact copy of

T; new nodes are gotten using the usual mechanism//

Q    0

if T    0 then [R    COPY(LCHILD(T))      //copy left subtree//

S    COPY(RCHILD(T))      //copy right subtree//

call GETNODE(Q)

LCHILD(Q)    R; RCHILD(Q)    S

//store in fields of Q//

DATA(Q)    DATA(T)]

return(Q)      //copy is a function//

end COPY

Another problem that is especially easy to solve using recursion is determining the equivalence of two binary trees. Binary trees are equivalent if they have the same topology and the information in corresponding nodes is identical. By the same topology we mean that every branch in one tree corresponds to a branch in the second in the same order. Algorithm EQUAL traverses the binary trees in preorder, though any order could be used.

We have seen that binary trees arise naturally with genealogical information and further that there is a natural relationship between the tree traversals and various forms of expressions. There are many other

procedure EQUAL(S,T)

//This procedure has value false if the binary trees S and T are not

equivalent. Otherwise, its value is true//

ans    false

case

: S = 0 and T = 0: ans    true

: S    0 and T    0:

if DATA(S) = DATA(T)

then [ans    EQUAL(LCHILD(S),LCHILD(T))

if ans then ans    EQUAL(RCHILD(S),RCHILD(T))]

end

return (ans)

end EQUAL

instances when binary trees are important, and we will look briefly at two of them now. The first problem has to do with processing a list of alphabetic data, say a list of variable names such as

X1, I, J, Z, FST, X2, K.

We will grow a binary tree as we process these names in such a way that for each new name we do the following: compare it to the root and if the new name alphabetically precedes the name at the root then move left or else move right; continue making comparisons until we fall off an end of the tree; then create a new node and attach it to the tree in that position. The sequence of binary trees obtained for the above data is given in figure 5.8. Given the tree in figure 5.8(g) consider the order of the identifiers if they were printed out using an inorder traversal

FST, I, J, K, X1, X2, Z

So by growing the tree in this way we turn inorder into a sorting method. In Chapter 9 we shall prove that this method works in general.

As a second example of the usefulness of binary trees, consider the set of formulas one can construct by taking variables x1,x2,x3, ... and the operators [image] (and), [image] (or) and [image] (not). These variables can only hold one of two possible values, true or false. The set of expressions which can be formed using these variables and operators is defined by the rules: (i) a variable is an expression, (ii) if x,y are expressions then x [image] y, x [image] y, [image] x are expressions. Parentheses can be used to alter the normal order of evaluation which is not before and before or. This comprises the formulas in the propositional calculus (other operations such as implication can be expressed using [image], [image], [image]). The expression

Figure 5.8 Tree of Identifiers

x1 [image] (x2 [image] [image] x3)

is a formula (read "x1 or x2 and not X3"). If xl and X3 are false and x2 is true, then the value of this expression is

false    (true       false)

= false    true

= true

The satisfiability problem for formulas of the propositional calculus asks if there is an assignment of values to the variables which causes the value of the expression to be true. This problem is of great historical interest in computer science. It was originally used by Newell, Shaw and Simon in the late 1950's to show the viability of heuristic programming (the Logic Theorist).

Again, let us assume that our formula is already in a binary tree, say

(x1 [image] [image] x2) [image] ( [image] x1 [image] x3) [image] [image] x3

in the tree

Figure 5.9 Propositional Formula in a Binary Tree

The inorder of this tree is x1 [image] [image] x2 [image] [image] x1 [image] x3 [image] [image] x3, the infix form of the expression. The most obvious algorithm to determine satisfiability is to let (x1,x2,x3) take on all possible combinations of truth and falsity and to check the formula for each combination. For n variables there are 2n possible combinations of true = t and false = f, e.g. for n = 3 (t,t,t), (t,t,f), (t,f, t), (t,f,f), (f,t,t), (f,t,f), (f,f,t), (f,f,f). The algorithm will take at least O(g2n) or exponential time where g is the time to substitute values for x1,x2,x3 and evaluate the expression.

To evaluate an expression one method would traverse the tree in postorder, evaluating subtrees until the entire expression is reduced to a single value. This corresponds to the postfix evaluation of an arithmetic expression that we saw in section 3.3. Viewing this from the perspective of the tree representation, for every node we reach, the values of its arguments (or children) have already been computed. So when we reach the v node on level two, the values of x1 [image] [image] x2 and [image] x1 [image] x3 will already be available to us and we can apply the rule for or. Notice that a node containing [image] has only a single right branch since not is a unary operator.

For the purposes of this algorithm we assume each node has four fields:

            _____________________________

           |       |      |     |        |

           |LCHILD | DATA | VAL | RCHILD |

           |_______|______|_____|________|

where LCHILD, DATA, RCHILD are as before and VAL is large enough to hold the symbols true or false. Also we assume that DATA(T) instead of containing the variable 'xi', is a pointer to a table DEF which has one entry for each variable. Then DEF(DATA(T)) will contain the current value of the variable. With these preparations and assuming an expression with n variables pointed at by T we can now write a first pass at our algorithm for satisfiability:

for all 2n possible combinations do

generate the next combination;

store it in DEF(1) to DEF(n);

call POSTORDER and evaluate T;

if VAL(T) = true then [print DEF(1) to DEF(n)

stop]

end

print ('no satisfiable combination')

Now let us concentrate on this modified version of postorder. Changing the original recursive version seems the simplest thing to do.

procedure POSTORDER_EVAL(T)

//a binary tree T containing a propositional formula is evaluated with

the result stored in the VAL field of each node//

if T    0 then[ call POSTORDER_EVAL(LCHILD(T))

call POSTORDER_EVAL(RCHILD(T))

case

:DATA(T) = '   : VAL(T)    not VAL(RCHILD(T))

:DATA(T) = '   : VAL(T)    VAL(LCHILD(T) or

VAL(RCHILD(T))

:DATA(T) = '   : VAL(T)    VAL(LCHILD(T) and

VAL(RCHILD(T))

:else:      VAL(T)    DEF(DATA(T))

end]

end POSTORDER_EVAL

5.6 THREADED BINARY TREES

If we look carefully at the linked representation of any binary tree, we notice that there are more null links than actual pointers. As we saw before, there are n + 1 null links and 2n total links. A clever way to make use of these null links has been devised by A. J. Perlis and C. Thornton. Their idea is to replace the null links by pointers, called threads, to other nodes in the tree. If the RCHILD(P) is normally equal to zero, we will replace it by a pointer to the node which would be printed after P when traversing the tree in inorder. A null LCHILD link at node P is replaced by a pointer to the node which immediately precedes node P in inorder. Figure 5.10 shows the binary tree of figure 5.3(b) with its new threads drawn in as dotted lines.

The tree T has 9 nodes and 10 null links which have been replaced by threads. If we traverse T in inorder the nodes will be visited in the order H D I B E A F C G. For example node E has a predecessor thread which points to B and a successor thread which points to A.

In the memory representation we must be able to distinguish between threads and normal pointers. This is done by adding two extra one bit fields LBIT and RBIT.

LBIT(P) =1      if LCHILD(P) is a normal pointer

LBIT(P) = 0     if LCHILD(P) is a thread

RBIT(P) = 1     if RCHILD(P) is a normal pointer

RBIT(P) = 0     if RCHILD(P) is a thread

Figure 5.10 Threaded Tree Corresponding to Figure 5.3(b)

In figure 5.10 we see that two threads have been left dangling in LCHILD(H) and RCHILD(G). In order that we leave no loose threads we will assume a head node for all threaded binary trees. Then the complete memory representation for the tree of figure 5.10 is shown in figure 5.11. The tree T is the left subtree of the head node. We assume that an empty binary tree is represented by its head node as

This assumption will permit easy algorithm design. Now that we have made use of the old null links we will see that the algorithm for inorder traversal is simplified. First, we observe that for any node X in a binary tree, if RBIT(X) = 0, then the inorder successor of X is RCHILD(X) by definition of threads. If RBIT(X) = 1, then the inorder successor of X is obtained by following a path of left child links from the right child of X until a node with LBIT = 0 is reached. The algorithm INSUC finds the inorder successor of any node X in a threaded binary tree.

Figure 5.11 Memory Representation of Threaded Tree

procedure INSUC(X)

//find the inorder succcesor of X in a threaded binary tree//

S    RCHILD(X)         //if RBIT(X) = 0 we are done//

if RBIT(X) = 1 then [while LBIT(S) = 1 do      //follow left//

S   LCHILD(S)         //until a thread//

end]

return (S)

end INSUC

The interesting thing to note about procedure INSUC is that it is now possible to find the inorder successor of an arbitrary node in a threaded binary tree without using any information regarding inorder predecessors and also without using an additional stack. If we wish to list in inorder all the nodes in a threaded binary tree, then we can make repeated calls to the procedure INSUC. Since vhe tree is the left subtree of the head node and because of the choice of RBIT= 1 for the head node, the inorder sequence of nodes for tree T is obtained by the procedure TINORDER.

procedure TINORDER (T)

//traverse the threaded binary tree, T, in inorder//

HEAD    T

loop

T    INSUC(T)

if T = HEAD then return

print(DATA(T))

forever

end TINORDER

The computing time is still O(n) for a binary tree with n nodes. The constant here will be somewhat smaller than for procedure INORDER3.

We have seen how to use the threads of a threaded binary tree for inorder traversal. These threads also simplify the algorithms for preorder and postorder traversal. Before closing this section let us see how to make insertions into a threaded tree. This will give us a procedure for growing threaded trees. We shall study only the case of inserting a node T as the right child of a node S. The case of insertion of a left child is given as an exercise. If S has an empty right subtree, then the insertion is simple and diagrammed in figure 5.12(a). If the right subtree of S is non-empty, then this right subtree is made the right subtree of T after insertion. When this is done, T becomes the inorder predecessor of a node which has a LBIT = 0 and consequently there is a thread which has to be updated to point to T. The node containing this thread was previously the inorder successor of S. Figure 5.12(b) illustrates the insertion for this case. In both cases S is the inorder predecessor of T. The details are spelled out in algorithm INSERT_RIGHT.

Figure 5.12 Insertion of T as a Right Child of S in a Threaded Binary Tree

procedure INSERT_RIGHT (S,T)

//insert node T as the right child of S in a threaded binary tree//

RCHILD(T)    RCHILD(S); RBIT(T)    RBIT(S)

LCHILD(T)    S; LBIT(T)    0       //LCHILD(T) is a thread//

RCHILD(S)    T; RBIT(S)    1       //attach node T to S//

if RBIT (T) = 1 then [W    INSUC(T)//S had a right child//

LCHILD(W)    T]

end INSERT_RIGHT

5.7 BINARY TREE REPRESENTATION OF TREES

We have seen several representations for and uses of binary trees. In this section we will see that every tree can be represented as a binary tree. This is important because the methods for representing a tree as suggested in section 5.1 had some undesirable features. One form of representation used variable size nodes. While the handling of nodes of variable size is not impossible, in section 4.8 we saw that it was considerably more difficult than the handling of fixed size nodes (section 4.3). An alternative would be to use fixed size nodes each node having k child fields if k is the maximum degree of any node. As Lemma 5.4 shows, this would be very wasteful in space.

Lemma 5.4: If T is a k-ary tree (i.e., a tree of degree k) with n nodes, each having a fixed size as in figure 5.13, then n(k- 1) + 1 of the nk link fields are zero, n [image] 1.

Proof: Since each nonzero link points to a node and exactly one link points to each node other than the root, the number of nonzero links in an n node tree is exactly n - 1. The total number of link fields in a k-ary tree with n nodes is nk. Hence, the number of null links is nk - (n - 1) = n(k - 1) + 1. 

Lemma 5.4 implies that for a 3-ary tree more than 2/3 of the link fields are zero! The proportion of zero links approaches 1 as the degree of the tree increases. The importance of using binary trees to represent trees is that for binary trees only about 1/2 of the link fields are zero.

Figure 5.13 Possible Node Structure for a k-ary Tree

In arriving at the binary tree representation of a tree we shall implicitly make use of the fact that the order of the children of a node is not important. Suppose we have the tree of figure 5.14.

Figure 5.14 A Sample Tree

Then, we observe that the reason we needed nodes with many link fields is that the prior representation was based on the parent-child relationship and a node can have any number of children. To obtain a binary tree representation, we need a relationship, between the nodes, that can be characterized by at most two quantities. One such relationship is the leftmost-child-next-right-sibling relationship. Every node has at most one leftmost child and at most one next right sibling. In the tree of figure 5.14, the leftmost child of B is E and the next right sibling of B is C. Strictly speaking, since the order of children in a tree is not important, any of the children of a node could be its leftmost child and any of its siblings could be its next right sibling. For the sake of definiteness, we choose the nodes based upon how the tree is drawn. The binary tree corresponding to the tree of figure 5.14 is thus obtained by connecting together all siblings of a node and deleting all links from a node to its children except for the link to its leftmost child. The node structure corresponds to that of

        ___________________

       |       DATA        |

       |-------------------|

       |  CHILD  | SIBLING |

       |_________|_________|

Using the transformation described above, we obtain the following representation for the tree of figure 5.14.

This does not look like a binary tree, but if we tilt it roughly 45 [image] clockwise we get

Figure 5.15 Associated Binary Tree for Tree of Figure 5.14

Let us try this transformation on some simple trees just to make sure we've got it.

One thing to notice is that the RCHILD of the root node of every resulting binary tree will be empty. This is because the root of the tree we are transforming has no siblings. On the other hand, if we have a forest then these can all be transformed into a single binary tree by first obtaining the binary tree representation of each of the trees in the forest and then linking all the binary trees together through the SIBLING field of the root nodes. For instance, the forest with three trees

yields the binary tree

We can define this transformation in a formal way as follows:

If T1, ...,Tn is a forest of trees, then the binary tree corresponding to this forest, denoted by B(T1, ..., Tn):

(i) is empty if n = 0;

(ii) has root equal to root (T1); has left subtree equal to B(T11,T12, ...,T1m) where T11, ..., T1m are the subtrees of root(T1); and has right subtree B(T2, ..., Tn).

Preorder and inorder traversals of the corresponding binary tree T of a forest F have a natural correspondence with traversals on F. Preorder traversal of T is equivalent to visiting the nodes of F in tree preorder which is defined by:

(i) if F is empty then return;

(ii) visit the root of the first tree of F;

(iii) traverse the subtrees of the first tree in tree preorder;

(iv) traverse the remaining trees of F in tree preorder.

Inorder traversal of T is equivalent to visiting the nodes of F in tree inorder as defined by:

(i) if F is empty then return;

(ii) traverse the subtrees of the first tree in tree inorder;

(iii) visit the root of the first tree;

(iv) traverse the remaining trees in tree inorder.

The above definitions for forest traversal will be referred to as preorder and inorder. The proofs that preorder and inorder on the corresponding binary tree are the same as preorder and inorder on the forest are left as exercises. There is no natural analog for postorder traversal of the corresponding binary tree of a forest. Nevertheless, we can define the postorder traversal of a forest as:

(i) if F is empty then return;

(ii) traverse the subtrees of the first tree of F in tree postorder;

(iii) traverse the remaining trees of F in tree postorder;

(iv) visit the root of the first tree of F.

This traversal is used later on in section 5.8.3 for describing the minimax procedure.

5.8 APPLICATIONS OF TREES

5.8.1 Set Representation

In this section we study the use of trees in the representation of sets. We shall assume that the elements of the sets are the numbers 1, 2, 3, ..., n. These numbers might, in practice, be indices into a symbol table where the actual names of the elements are stored. We shall assume that the sets being represented are pairwise disjoint; i.e., if Si and Sj, i [image] j, are two sets then there is no element which is in both Si and Sj. For example, if we have 10 elements numbered 1 through 10, they may be partitioned into three disjoint sets S1 = {1, 7, 8, 9}; S2= {2, 5, 10} and S3 = {3, 4, 6}. The operations we wish to perform on these sets are:

(i) Disjoint set union ... if Si and Sj are two disjoint sets, then their union Si [image] Sj = {all elements x such that x is in Si or Sj}. Thus, S1 [image] S2 = {1, 7, 8, 9, 2, 5, 10}. Since we have assumed that all sets are disjoint, following the union of Si and Sj we can assume that the sets Si and Sj no longer exist independently, i.e., they are replaced by Si [image] Sj in the collection of sets.

(ii) Find(i) ... find the set containing element i. Thus, 4 is in set S3 and 9 is in set S1.

The sets will be represented by trees. One possible representation for the sets S1, S2 and S3 is:

Note that the nodes are linked on the parent relationship, i.e. each node other than the root is linked to its parent. The advantage of this will become apparent when we present the UNION and FIND algorithms. First, to take the union of S1 and S2 we simply make one of the trees a subtree of the other. S1 [image] S2 could then have one of the following representations

In order to find the union of two sets, all that has to be done is to set the parent field of one of the roots to the other root. This can be accomplished easily if, with each set name, we keep a pointer to the root of the tree representing that set. If, in addition, each root has a pointer to the set name, then to determine which set an element is currently in, we follow parent links to the root of its tree and use the pointer to the set name. The data representation for S1, S2 and S3 may then take the form:

In presenting the UNION and FIND algorithms we shall ignore the actual set names and just identify sets by the roots of the trees representing them. This will simplify the discussion. The transition to set names is easy. If we determine that element i is in a tree with root j and j has a pointer to entry k in the set name table, then the set name is just NAME(k). If we wish to union sets Si and Sj, then we wish to union the trees with roots POINTER(Si) and POINTER(Sj). As we shall see, in many applications the set name is just the element at the root. The operation of FIND(i) now becomes: determine the root of the tree containing element i. UNION(i,j) requires two trees with roots i and j to be joined. We shall assume that the nodes in the trees are numbered 1 through n so that the node index corresponds to the element index. Thus, element 6 is represented by the node with index 6. Consequently, each node needs only one field: the PARENT field to link to its parent. Root nodes have a PARENT field of zero. Based on the above discussion, our first attempt at arriving at UNION, FIND algorithms would result in the algorithms U and F below.

procedure U(i,j)

//replace the disjoint sets with roots i and j, i    j with their union//

PARENT(i)    j

end U

procedure F(i)

//find the root j of the tree containing element i//

j    i

while PARENT(j) > 0 do           //PARENT (j) = 0 if this node is a

root//

j    PARENT(j)

end

return(j)

end F

While these two algorithms are very easy to state, their performance characteristics are not very good. For instance, if we start off with p elements each in a set of its own, i.e., Si = {i}, 1   [image]  i   [image]  p, then the initial configuration consists of a forest with p nodes and PARENT(i) = 0, 1   [image]  i   [image]  p. Now let us process the following sequence of UNION-FIND operations.

U(1,2), F(1), U(2,3), F(1), U(3,4)

F(1), U(4,5), ...,F(1), U(n - 1,n)

This sequence results in the degenerate tree:

Since the time taken for a union is constant, all the n - 1 unions can be processed in time O(n). However, each FIND requires following a chain of PARENT links from one to the root. The time required to process a FIND for an element at level i of a tree is O(i). Hence, the total time needed to process the n - 2 finds is   [image]. It is easy to see that this example represents the worst case behavior of the UNION-FIND algorithms. We can do much better if care is taken to avoid the creation of degenerate trees. In order to accomplish this we shall make use of a Weighting Rule for UNION (i,j). If the number of nodes in tree i is less than the number in tree j, then make j the parent of i, otherwise make i the parent of j. Using this rule on the sequence of set unions given before we obtain the trees on page 252. Remember that the arguments of UNION must both be roots. The time required to process all the n finds is only O(n) since in this case the maximum level of any node is 2. This, however, is not the worst case. In Lemma 5.5 we show that using the weighting rule, the maximum level for any node is   [image]  log n   [image]  + 1. First, let us see how easy it is to implement the weighting rule. We need to know how many nodes there are in any tree. To do this easily, we maintain a count field in the root of every tree. If i is a root node, then COUNT (i) = number of nodes in that tree. The count can be maintained in the PARENT field as a negative number. This is equivalent to using a one bit field to distinguish a count from a pointer. No confusion is created as for all other nodes the PARENT is positive.

Trees obtained using the weighting rule

procedure UNION(i,j)

//union sets with roots i and j, i    j, using the weighting rule. PARENT

(i) = - COUNT(i) and PARENT(j) = - COUNT(j)//

x    PARENT (i) + PARENT(j)

if PARENT(i) > PARENT(j)

then [PARENT (i)    j   //i has fewer nodes//

PARENT (j)    x]

else [PARENT (j)    i   //j has fewer nodes//

PARENT (i)    x]

end UNION

The time required to perform a union has increased somewhat but is still bounded by a constant, i.e. it is O(1). The FIND algorithm remains unchanged. The maximum time to perform a find is determined by Lemma 5.5.

Lemma 5.5: Let T be a tree with n nodes created as a result of algorithm UNION. No node in T has level greater   [image] log2 n  [image]  +  1.

Proof: The lemma is clearly true for n = 1. Assume it is true for all trees with i nodes, i   [image]  n - 1. We shall show that it is also true for i = n. Let T be a tree with n nodes created by the UNION algorithm. Consider the last union operation performed, UNION(k,j). Let m be the number of nodes in tree j and n - m the number in k. Without loss of generality we may assume 1   [image]  m   [image]  n/2. Then the maximum level of any node in T is either the same as that in k or is one more than that in j. If the former is the case, then the maximum level in T is   [image]    [image] log2 (n - m)  [image]  + 1   [image]    [image] log2 n  [image]  + 1. If the latter is the case then the maximum level in T is   [image]    [image] log2 m  [image]  + 2   [image]    [image] log2 n/2  [image]  + 2.      log2 n  [image]  + 1.

Example 5.1 shows that the bound of Lemma 5.5 is achievable for some sequence of unions.

Example 5.1: Consider the behavior of algorithm UNION on the following sequence of unions starting from the initial configuration

PARENT(i) = -COUNT(i) = -1, 1   [image]  i   [image]  n = 23

UNION(1,2),  UNION(3,4),  UNION(5,6),  UNION(7,8),

UNION(1,3),  UNION(5,7),  UNION(1,5).

The following trees are obtained:

As is evident, the maximum level in any tree is   [image] log2 m  [image]  + 1 if the tree has m nodes.

As a result of Lemma 5.5, the maximum time to process a find is at most O(log n) if there are n elements in a tree. If an intermixed sequence of n - 1 UNION and m FIND operations is to be processed, then the worst case time becomes O(n + m log n). Surprisingly, further improvement is possible. This time the modification will be made in the FIND algorithm using the Collapsing Rule: If j is a node on the path from i to its root and PARENT(j)   [image]  root(i) then set PARENT(j)   [image]  root(i). The new algorithm then becomes:

This modification roughly doubles the time for an individual find. However, it reduces the worst case time over a sequence of finds.

Example 5.2: : Consider the tree created by algorithm UNION on the sequence of unions of example 5.1. Now process the following 8 finds:

FIND(8), FIND(8), ... FIND(8)

procedure FIND(i)

//find the root of the tree containing element i. Use the collapsing

rule to collapse all nodes from i to the root j//

j    i

while PARENT(j) > 0 do            //find root//

j    PARENT(j)

end

k    i

while k    j do    //collapse nodes from i to root j//

t    PARENT(k)

PARENT(k)    j

k    t

end

return(j)

end FIND

Using the old version F of algorithm FIND, FIND(8) requires going up 3 parent link fields for a total of 24 moves to process all 8 finds. In algorithm FIND, the first FIND(8) requires going up 3 links and then resetting 3 links. Each of the remaining 7 finds requires going up only 1 link field. The total cost is now only 13 moves.

The worst case behavior of the UNION-FIND algorithms while processing a sequence of unions and finds is stated in Lemma 5.6. Before stating this lemma, let us introduce a very slowly growing function   [image]  (m,n) which is related to a functional inverse of Ackermann's function A(p,q). We have the following definition for   [image]  (m,n):

  (m,n) = min{z   [image]  1   [image]  A(z,4  [image] m/n  [image] ) > log2n}

The definition of Ackermann's function used here is:

The function A(p,q) is a very rapidly growing function. One may prove the following three facts:

If we assume m   [image]  0 then (b) and (c) together with the definition of   [image]  (m,n) imply that   [image]  (m,n)   [image]  3 for log2n < A(3,4). But from (a), A(3,4) is a very large number indeed! In Lemma 5.6 n will be the number of UNlONs performed. For all practical purposes we may assume log2n < A(3,4) and hence   [image]  (m,n)   [image]  3.

Lemma 5.6: [Tarjan] Let T(m,n) be the maximum time required to process any intermixed sequence of m   [image]  n FINDs and n - 1 UNIONs. Then k1m  [image]  (m,n)   [image]  T(m,n)   [image]  k2m  [image]  (m,n) for some positive constants k1 and k2.

Even though the function   [image]  (m,n) is a very slowly growing function, the complexity of UNION-FIND is not linear in m, the number of FINDs. As far as the space requirements are concerned, the space needed is one node for each element.

Let us look at an application of algorithms UNION and FIND to processing the equivalence pairs of section 4.6. The equivalence classes to be generated may be regarded as sets. These sets are disjoint as no variable can be in two equivalence classes. To begin with all n variables are in an equivalence class of their own; thus PARENT(i) = -1, 1   [image]  i   [image]  n. If an equivalence pair, i   [image]  j, is to be processed, we must first determine the sets containing i and j. If these are different, then the two sets are to be replaced by their union. If the two sets are the same, then nothing is to be done as the relation i   [image]  j is redundant; i and j are already in the same equivalence class. To process each equivalence pair we need to perform at most two finds and one union. Thus, if we have n variables and m   [image]  n equivalence pairs, the total processing time is at most O(m  [image]  (2m,m)). While for very large n this is slightly worse than the algorithm of section 4.6, it has the advantage of needing less space and also of being "on line."

In Chapter 6 we shall see another application of the UNION-FIND algorithms.

Example 5.3: We shall use the UNION-FIND algorithms to process the set of equivalence pairs of scction 4.6. Initially, there are 12 trees, one for each variable. PARENT(i) = -1, 1   [image]  i   [image]  12.

Each tree represents an equivalence class. It is possible to determine if two elements are currently in the same equivalence class at each stage of the processing by simply making two finds.

5.8.2 Decision Trees

Another very useful application of trees is in decision making. Consider the well-known eight coins problem. Given coins a,b,c,d,e,f,g,h, we are told that one is a counterfeit and has a different weight than the others. We want to determine which coin it is, making use of an equal arm balance. We want to do so using a minimum number of comparisons and at the same time determine whether the false coin is heavier or lighter than the rest. The tree below represents a set of decisions by which we can get the answer to our problem. This is why it is called a decision tree. The use of capital H or L means that the counterfeit coin is heavier or lighter. Let us trace through one possible sequence. If a + b + c < d + e + f, then we know that the false coin is present among the six and is neither g nor h. If on our next measurement we find that a + d < b + e, then by interchanging d and b we have no change in the inequality. This tells us two things: (i) that c or f is not the culprit, and (ii) that b or d is also not the culprit. If a + d was equal to b + e, then c or f would be the counterfeit coin. Knowing at this point that either a or e is the counterfeit, we compare a with a good coin, say b. If a = b, then e is heavy, otherwise a must be light.

Figure 5.16 Eight Coins Decision Tree

By looking at this tree we see that all possibilities are covered, since there are 8 coins which can be heavy or light and there are 16 terminal nodes. Every path requires exactly 3 comparisons. Though viewing this problem as a decison tree is very useful it does not immediately give us an algorithm. To solve the 8 coins problem with a program, we must write a series of tests which mirror the structure of the tree. If we try to do this in SPARKS using the if-then-else statement, we see that the program looks like a dish of spaghetti. It is impossible to discern the flow of the program. Actually this type of processing is much more clearly expressed using the case statement.

We will make use of a procedure to do the last comparison.

procedure COMP(x,y,z)

//x is compared against the standard coin z//

if x > z then print (x 'heavy')

else print (y 'light')

end COMP

The procedure EIGHTCOINS appears on the next page. The program is now transparent and clearly mirrors the decision tree of figure 5.16.

5.8.3 Game Trees

Another interesting application of trees is in the playing of games such as tic-tac-toe, chess, nim, kalah, checkers, go, etc. As an example, let us consider the game of nim. This game is played by two players A and B. The game itself is described by a board which initially contains a pile of n toothpicks. The players A and B make moves alternately

procedure EIGHTCOINS

//eight weights are input; the different one is discovered using only

3 comparisons//

read (a, b, c, d, e, f, g, h)

case

: a + b + c = d + e + f : if g > h then call COMP(g,h,a)

else call COMP(h,g,a)

: a + b + c > d + e + f : case

: a + d = b + e: call COMP(c,f,a)

: a + d > b + e: call COMP(a,e,b)

: a + d < b + e: call COMP(b,d,a)

end

: a + b + c < d + e + f : case

: a + d = b + e: call COMP(f,c,a)

: a + d > b + e: call COMP(d,b,a)

: a + d < b + e: call COMP(e,a,b)

end

end

end EIGHTCOINS

with A making the first move. A legal move consists of removing either 1, 2 or 3 of the toothpicks from the pile. However, a player cannot remove more toothpicks than there are on the pile. The player who removes the last toothpick loses the game and the other player wins. The board configuration at any time is completely specified by the number of toothpicks remaining in the pile. At any time the game status is determined by the board configuration together with the player whose turn it is to make the next move. A terminal board cofiguration is one which represents either a win, lose or draw situation. All other configurations are nonterminal. In nim there is only one terminal configuration: there are no toothpicks in the pile. This configuration is a win for player A if B made the last move, otherwise it is a win for B. The game of nim cannot end in a draw.

A sequence C1, ...,Cm of board configurations is said to be valid if:

(i)     C1 is the starting configuration of the game;

(ii)     Ci, 0 < i < m, are nonterminal configurations;

(iii)     Ci+1 is obtained from Ci by a legal move made by player A if i is odd and by player B if i is even. It is assumed that there are only finitely many legal moves.

A valid sequence C1, ...,Cm of board configurations with Cm a terminal configuration is an instance of the game. The length of the sequence C1,C2, ...,Cm is m. A finite game is one in which there are no valid sequences of infinite length. All possible instances of a finite game may be represented by a game tree. The tree of figure 5.17 is the game free for nim with n = 6. Each node of the tree represents a board configuration. The root node represents the starting configuration C1. Transitions from one level to the next are made via a move of A or B. Transitions from an odd level represent moves made by A. All other transitions are the result of moves made by B. Square nodes have been used in figure 5.17 to represent board configurations when it was A's turn to move. Circular nodes have been used for other configurations. The edges from level 1 nodes to level 2 nodes and from level 2 nodes to level 3 nodes have been labeled with the move made by A and B respectively (for example, an edge labeled 1 means 1 toothpick is to be removed). It is easy to figure out the labels for the remaining edges of the tree. Terminal configurations are represented by leaf nodes. Leaf nodes have been labeled by the name of the player who wins when that configuration is reached. By the nature of the game of nim, player A can win only at leaf nodes on odd levels while B can win only at leaf nodes on even levels. The degree of any node in a game tree is at most equal to the number of distinct legal moves. In nim there are at most 3 legal moves from any configuration. By definition, the number of legal moves from any configuration is finite. The depth of a game tree is the length of a longest instance of the game. The depth of the nim tree of figure 5.17 is 7. Hence, from start to finish this game involves at most 6 moves. It is not difficult to see how similar game trees may be constructed for other finite games such as chess, tic-tac-toe, kalah, etc. (Strictly speaking, chess is not a finite game as it is possible to repeat board configurations in the game. We can view chess as a finite game by disallowing this possibility. We could, for instance, define the repetition of a board configuration as resulting in a draw.)

Now that we have seen what a game tree is, the next question is "of what use are they?" Game trees are useful in determining the next move a player should make. Starting at the initial configuration represented by the root of figure 5.17 player A is faced with the choice of making any one of three possible moves. Which one should he make? Assuming that player A wants to win the game, he should make the move that maximizes his chances of winning. For the simple tree of figure 5.17 this move is not too difficult to determine. We can use an evaluation function E(X) which assigns a numeric value to the board configuration X. This function is a measure of the value or worth of configuration X to player A. So, E(X) is high for a configuration from which A has a good chance of winning and low for a configuration from which A has a good chance of losing. E(X) has its maximum value for configurations that are either winning terminal configurations for A or configurations from which A is guaranteed to win regardless of B's countermoves. E(X) has its minimum value for configurations from which B is guaranteed to win.

Figure 5.17 Complete Game Tree for Nim with n = 6

For a game such as nim with n = 6, whose game tree has very few nodes, it is sufficient to define E(X) only for terminal configurations. We could define E(X) as:

Using this evaluation function we wish to determine which of the configurations b, c, d player A should move the game into. Clearly, the choice is the one whose value is max {V(b), V(c), V(d)} where V(x) is the value of configuration x. For leaf nodes x, V(x) is taken to be E(x). For all other nodes x let d  [image] 1 be the degree of x and

let c1,c2, ...,cd be the configurations represented by the children of x. Then V(x) is defined by:

(5.3)

The justification for (5.3) is fairly simple. If x is a square node, then it is at an odd level and it will be A's turn to move from here if the game ever reaches this node. Since A wants to win he will move to that child node with maximum value. In case x is a circular node it must be on an even level and if the game ever reaches this node, then it will be B's turn to move. Since B is out to win the game for himself, he will (barring mistakes) make a move that will minimize A's chances of winning. In this case the next configuration will be   [image]  {V(ci)}. Equation (5.3) defines the minimax procedure to determine the value of a configuration x. This is illustrated on the hypothetical game of figure 5.18. P11 represents an arbitrary board configuration from which A has to make a move. The values of the leaf nodes are obtained by evaluating the function E(x). The value of P11 is obtained by starting at the nodes on level 4 and computing their values using eq. (5.3). Since level 4 is a level with circular nodes all unknown values on this level may be obtained by taking the minimum of the children values. Next, values on levels 3, 2 and 1 may be computed in that order. The resulting value for P11 is 3. This means that starting from P11 the best A can hope to do is reach a configuration of value 3. Even though some nodes have value greater than 3, these nodes will not be reached, as B's countermoves will prevent the game from reaching any such configuration (assuming B's countermoves are optimal for B with respect to A's evaluation function). For example, if A made a move to P21, hoping to win the game at P31, A would indeed be surprised by B's countermove to P32 resulting in a loss to A. Given A's evaluation function and the game tree of figure 5.18, the best move for A to make is to configuration P22. Having made this move, the game may still not reach configuration P52 as B would, in general, be using a different evaluation function, which might give different values to various board configurations. In any case, the minimax procedure can be used to determine the best move a player can make given his evaluation function. Using the minimax procedure on the game tree for nim (figure 5.17) we see that the value of the root node is V(a) = 1. Since E(X) for this game was defined to be 1 iff A was guaranteed to win, this means that if A makes the optimal move from node a then no matter what B's countermoves A will win. The optimal move is to node b. One may readily verify that from b A can win the game independent of B's countermove!

Figure 5.18 Portion of Game Tree for a Hypothetical Game. The value of terminal nodes is obtained from the evaluation function E(x) for player A.

For games such as nim with n = 6, the game trees are sufficiently small that it is possible to generate the whole tree. Thus, it is a relatively simple matter to determine whether or not the game has a winning strategy. Moreover, for such games it is possible to make a decision on the next move by looking ahead all the way to terminal configurations. Games of this type are not very interesting since assuming no errors are made by either player, the outcome of the game is predetermined and both players should use similar evaluation functions, i.e., EA(X) = 1 for X a winning configuration and EA(X) = - 1 for X a losing configuration for A; EB(X) = - EA(X).

Of greater interest are games such as chess where the game tree is too large to be generated in its entirety. It is estimated that the game tree for chess has >10100 nodes. Even using a computer which is capable of generating 1011 nodes a second, the complete generation of the game tree for chess would require more than 1080 years. In games with large game trees the decision as to which move to make next can be made only by looking at the game tree for the next few levels. The evaluation function E(X) is used to get the values of the leaf nodes of the subtree generated and then eq. (5.3) can be used to get the values of the remaining nodes and hence to determine the next move. In a game such as chess it may be possible to generate only the next few levels (say 6) of the tree. In such situations both the quality of the resulting game and its outcome will depend upon the quality of the evaluating functions being used by the two players as well as of the algorithm being used to determine V(X) by minimax for the current game configuration. The efficiency of this algorithm will limit the number of nodes of the search tree that can be generated and so will have an effect on the quality of the game.

Let us assume that player A is a computer and attempt to write an algorithm that A can use to compute V(X). It is clear that the procedure to compute V(X) can also be used to determine the next move that A should make. A fairly simple recursive procedure to evaluate V(X) using minimax can be obtained if we recast the definition of minimax into the following form:

(5.4)

where e(X) = E(X) if X is a position from which A is to move and e(X) = - E(X) otherwise.

Starting at a configuration X from which A is to move, one can easily prove that eq. (5.4) computes V'(X) = V(X) as given by eq. (5.3). In fact, values for all nodes on levels from which A is to move are the same as given by eq. (5.3) while values on other levels are the negative of those given by eq. (5.3).

The recursive procedure to evaluate V'(X) based on eq. (5.4) is then VE(X,l). This algorithm evaluates V'(X) by generating only l levels of the game tree beginning with X as root. One may readily verify that this algorithm traverses the desired subtree of the game tree in postorder.

procedure VE(X,l)

//compute V'(X) by looking at most l moves ahead. e(X) is the

evaluation function for player A. For convenience, it is assumed

that starting from any board configuration X the legal moves of

the game permit a transition only to the configurations C1,C2, ...,Cd

if X is not a terminal configuration.//

if X is terminal or l = 0 then return e(X)

ans    - VE(C1,l - 1          //traverse the first subtree//

for i    2 to d do        //traverse the remaining subtrees//

ans    max {ans, - VE(Ci, l - 1)}

end

return (ans)

end VE

An Initial call to algorithm VE with X = P11 and l = 4 for the hypothetical game of figure 5.18 would result in the generation of the complete game tree. The values of various configurations would be determined in the order: P31, P32, P21, P5l, P52, P53, P41, P54, P55, P56, P42, P33, ...,P37,P24, P11. It is possible to introduce, with relative ease, some heuristics into algorithm VE that will in general result in the generation of only a portion of the possible configurations while still computing V'(X) accurately.

Consider the game tree of figure 5.18. After V(P41) has been computed, it is known that V(P33) is at least V(P41)= 3. Next, when V(P55) is determined to be 2, then we know that V(P42) is at most 2. Since P33 is a max position, V(P42) cannot affect V(P33). Regardless of the values of the remaining children of P42, the value of P33 is not determined by V(P42) as V(P42) cannot be more than V(P41). This observation may be stated more formally as the following rule: The alpha value of a max position is defined to be the minimum possible value for that position. If the value of a min position is determined to be less than or equal to the alpha value of its parent, then we may stop generation of the remaining children of this min position. Termination of node generation under this rule is known as alpha cutoff. Once V(P41) in figure 5.18 is determined, the alpha value of P33 becomes 3. V(P55)   [image]  alpha value of P33 implies that P56 need not be generated.

A corresponding rule may be defined for min positions. The beta value of a min position is the maximum possible value for that position. If the value of a max position is determined to be greater than or equal to the beta value of its parent node, then we may stop generation of the remaining children of this max position. Termination of node generation under this rule is called beta cutoff. In figure 5.18, once V(P35) is determined, the beta value of P23 is known to be at most -  [image]. Generation of P57, P58, P59 gives V(P43) = 0. Thus, V(P43) is greater than or equal to the beta value of P23 and we may terminate the generation of the remaining children of P36. The two rules stated above may be combined together to get what is known as alpha-beta pruning. When alpha-beta pruning is used on figure 5.18, the subtree with root P36 is not generated at all! This is because when the value of P23 is being determined the alpha value of P11 is 3. V(P35) is less than the alpha value of P11 and so an alpha cutoff takes place. It should be emphasized that the alpha or beta value of a node is a dynamic quantity. Its value at any time during the game tree generation depends upon which nodes have so far been generated and evaluated.

In actually introducing alpha-beta pruning into algorithm VE it is necessary to restate this rule in terms of the values defined by eq. (5.4). Under eq. (5.4) all positions are max positions since the values of the min positions of eq. (5.3) have been multiplied by -1. The alpha-beta pruning rule now reduces to the following rule: let the B value of a position be the minimum value that that position can have.

For any position X, let B be the B - value of its parent and let D = -B. Then, if the value of X is determined the greater than or equal to D, we may terminate generation of the remaining children of X. Incorporating this rule into algorithm VE is fairly straightforward and results in algorithm VEB. This algorithm has the additional parameter D which is the negative of the B value of the parent of X.

procedure VEB(X,l,D)

//determine V'(X) as in eq. (5.4) using the B-rule and looking

only l moves ahead. Remaining assumptions and notation are

the same as for algorithm VE.//

if X is terminal or l = 0 then return e(x)

ans    -  .          //current lower bound on V'(x)//

for i    l to d do

ans    max {ans, - VEB(Ci,l-1,-ans)}

if ans    D then return (ans)   //use B-rule//

end

return (ans)

end VEB

If Y is a position from which A is to move, then the initial call VEB(Y,l,  [image] ) correctly computes V' (Y) with an l move look ahead. Further pruning of the game tree may be achieved by realizing that the B - value of a node X places a lower bound on the value grandchildren of X must have in order to affect X's value. Consider the subtree of figure 5.19(a). If V' (GC(X))   [image]  B then V' (C(X))   [image]  -B. Following the evaluation of C(X), the B-value of X is max{B,-V' (C(X))} = B as V'(C(X))   [image]  -B. Hence unless V' (GC(X)) > B, it cannot affect V' (X) and so B is a lower bound on the value GC(X) should have. Incorporating this lowerbound into algorithm VEB yields algorithm AB. The additional parameter LB is a lowerbound on the value X should have.

procedure AB(X,l,LB,D)

//same as algorithm VEB. LB is a lowerbound on V' (X)//

if X is terminal or l = 0 then return e(X)

ans    LB           //current lowerbound on V' (X)//

for i    1 to d do

ans    max{ans,-AB(Ci,l-1,-D,-ans)}

if ans    D then return (ans)

end

return (ans)

end AB.

One may easily verify that the initial call AB(Y,l, -  [image],  [image] ) gives the same result as the call VE(Y,l).

Figure 5.19(b) shows a hypothetical game tree in which the use of algorithm AB results in greater pruning than achieved by algorithm VEB. Let us first trace the action of VEB on the tree of figure 5.19(b). We assume the initial call to be VEB(P1,l,  [image]) where l is the depth of the tree. After examining the left subtree of P1, the B value of P1 is set to 10 and nodes P3, P4, P5 and P6 are generated. Following this, V' (P6) is determined to be 9 and then the B-value of P5 becomes -9. Using this, we continue to evaluate the node P7. In the case of AB however, since the B-value of P1 is 10, the lowerbound for P4 is 10 and so the effective B-value of P4 becomes 10. As a result the node P7 is not generated since no matter what its value V' (P5)   [image]  -9 and this will not enable V' (P4) to reach its lower bound.

Figure 5.19 Game trees showing lower bounding

5.9 COUNTING BINARY TREES

As a conclusion to our chapter on trees, we determine the number of distinct binary trees having n nodes. We know that if n = 0 or n = 1 there is one such tree. If n = 2, then there are two distinct binary trees

and if n = 3, there are five

How many distinct binary trees are there with n nodes?

Before solving this problem let us look at some other counting problems that are equivalent to this one.

In section 5.4 we introduced the notion of preorder, inorder and postorder traversals. Suppose we are given the preorder sequence

A B C D E F G H I

and the inorder sequence

B C A E D G H F I

of the same binary tree. Does such a pair of sequences uniquely define a binary tree? Asked another way, can the above pair of sequences come from more than one binary tree. We can construct the binary tree which has these sequences by noticing that the first letter in preorder, A, must be the root and by the definition of inorder all nodes preceding A must occur in the left subtree and the remaining nodes occur in the right subtree.

This gives us

as our first approximation to the correct tree. Moving right in the preorder sequence we find B as the next root and from the inorder we see B has an empty left subtree and C is in its right subtree. This gives

as the next approximation. Continuing in this way we arrive at the binary tree

By formalizing this argument, see the exercises, we can verify that every binary tree has a unique pair of preorder-inorder scquences.

Let the nodes of an n node binary tree be numbered 1 to n. The inorder permutation defined by such a binary tree is the order in which its nodes are visited during an inorder traversal of the tree. A preorder permutation is similarly defined.

As an example, consider the binary tree above with the following numbering of nodes:

Its preorder permutation is 1,2, ...,9 and its inorder permutation is 2,3,1,5,4,7,8,6,9.

If the nodes of a binary tree are numbered such that its preorder permutation is 1,2, ...,n, then from our earlier discussion it follows that distinct binary trees define distinct inorder permutations. The number of distinct binary trees is thus equal to the number of distinct inorder permutations obtainable from binary trees having the preorder permutation 1,2, ...,n.

Using this concept of an inorder permutation, it is possible to show that the number of distinct permutations obtainable by passing the numbers 1 to n through a stack and deleting in all possible ways is equal to the number of distinct binary trees with n nodes (see the exercises). If we start with the numbers 1,2,3 then the possible permutations obtainable by a stack are

1,2,3; 1,3,2; 2,1,3; 3,2,1; 3, 2, 1;

It is not possible to obtain 3,1,2. Each of these five permutations corresponds to one of the five distinct binary trees with 3 nodes

Another problem which surprisingly has connection with the previous two is the following: we have a product of n matrices

M1 * M2 * M3 * ... * Mn

that we wish to compute. We can perform these operations in any order because multiplication of matrices is associative. We ask the question: how many different ways can we perform these multiplications? For example, if n = 3, there are two possibilities

(M1 * M2) * M3 and M1 * (M2 * M3)

and if n = 4, there are five ways

((M1 * M2) * M3) * M4, (M1 * (M2 * M3)) * M4,

M1 * ((M2 * M3) * M4)

(M1 * (M2 * (M3 * M4))), ((M1 * M2) * (M3 * M4))

Let bn be the number of different ways to compute the product of n matrices. Then b2 = 1, b3 = 2, b4 = 5. Let Mij, i   [image]  j, be the product Mi * Mi+1 * ... * Mj. The product we wish to compute is M1n.M1n may be computed by computing any one of the products M1i * Mi+1,n, 1   [image]  i < n. The number of ways to obtain M1i and Mi+1,n is bi and bn-i respectively. Therefore, letting b1 = 1 we have:

If we can determine an expression for bn only in terms of n, then we have a solution to our problem. Now instead let bn be the number of distinct binary trees with n nodes. Again an expression for bn in terms of n is what we want. Then we see that bn is the sum of all possible binary trees formed in the following way, a root and two subtrees with bi and bn-i-1 nodes,

for 0   [image]  i   [image]  n - 1. This says that

(5.5)

This formula and the previous one are essentially the same.

So, the number of binary trees with n nodes, the number of permutations of 1 to n obtainable with a stack, and the number of ways to multiply n + 1 factors are all equal to the same number!

To obtain this number we must solve the recurrence of eq. (5.5). To begin we let

(5.6)

which is the generating function for the number of binary trees. Next, observe that by the recurrence relation we get the identity

x B2(x) = B(x) - 1

Using the formula to solve quadratics and the fact (eq. (5.5)) that B(0) = b0 = 1 we get:

It is not clear at this point that we have made any progress but by using the binomial theorem to expand (1 - 4x)1/2 we get

(5.7)

Comparing eqs. (5.6) and (5.7)we see that bn which is the coefficient of xn in B(x) is:

Some simplification yields the more compact form

which is approximately

bn= O(4n/n3/2)

REFERENCES

For other representations of trees see

The Art of Computer Programming: Fundamental Algorithms, by D. Knuth, second edition, Addison-Wesley, Reading, 1973.

For the use of trees in generating optimal compiled code see

"The generation of optimal code for arithmetic expressions'' by R. Sethi and J. Ullman, JACM, vol. 17, no. 4, October 1970, pp. 715-728.

"The generation of optimal code for a stack machine'' by J. L. Bruno and T. Lassagne, JACM, vol. 22, no. 3, July 1975, pp. 382-396.

Algorithm INORDER4 of the exercises is adapted from

"An improved algorithm for traversing binary trees without auxiliary stack," by J. Robson, Information Processing Letters, vol. 2, no. 1, March 1973, p. 12-14.

Further tree traversal algorithms may be found in:

''Scanning list structures without stacks and tag bits,'' by G. Lindstrom, Information Processing Letters, vol. 2, no. 2, June 1973, p. 47-51.

"Simple algorithms for traversing a tree without an auxiliary stack,'' by B. Dwyer, Information Processing Letters, vol. 2, no. 5, Dec. 1973, p. 143-145.

The use of threads in conneetion with binary trees is given in

"Symbol manipulation by threaded lists,'' by A. Perlis and C. Thornton, CACM, vol. 3, no. 4, April 1960, pp. 195-204.

For a further analysis of the set representation problem see

The Design and Analysis of Computer A1gorithms, A. Aho, J. Hopcroft and J. Ullman, Addison-Wesley, Reading, 1974.

The computing time analysis of the UNION-FIND algorithms may be found in:

"Efficiency of a good but not linear set union algorithm" by R. Tarjan, JACM, vol. 22, no. 2, April 1975, pp. 215-225.

Our discussion of alpha-beta cutoffs is from

"An analysis of alpha beta cutoffs" by D. Knuth, Stanford Technical Report 74-441, Stanford University, 1975.

For more on game playing see

Problem Solving Methods in Artificial Intelligence by N. Nilsson, McGraw-Hill, New York, 1971.

Artificial Intelligence: The Heuristic Programming Approach by J. Slagle, McGraw-Hill, New York, 1971.

EXERCISES

1.     For the binary tree below list the terminal nodes, the nonterminal nodes and the level of each node.

2.     Draw the internal memory representation of the above binary tree using (a) sequential, (b) linked, and (c) threaded linked representations.

3.     Write a procedure which reads in a tree represented as a list as in section 5.1 and creates its internal representation using nodes with 3 fields, TAG, DATA, LINK.

4.     Write a procedure which reverses the above process and takes a pointer to a tree and prints out its list representation.

5.     Write a nonrecursive version of procedure PREORDER.

6.     Write a nonrecursive version of procedure POSTORDER without using go to's.

7.     Rework INORDER3 so it is as fast as possible. (Hint: minimize the stacking and the testing within the loop.)

8.     Write a nonrecursive version of procedure POSTORDER using only a fixed amount of additional space. (See exercise 36 for details)

9.     Do exercise 8 for the case of PREORDER.

10.     Given a tree of names constructed as described in section 5.5 prove that an inorder traversal will always print the names in alphabetical order.

Exercises 11-13 assume a linked representation for a binary tree.

11.     Write an algorithm to list the DATA fields of the nodes of a binary tree T by level. Within levels nodes are to be listed left to right.

12.     Give an algorithm to count the number of leaf nodes in a binary tree T. What is its computing time?

13.     Write an algorithm SWAPTREE(T) which takes a binary tree and swaps the left and right children of every node. For example, if T is the binary tree

14.     Devise an external representation for formulas in the propositional calculus. Write a procedure which reads such a formula and creates a binary tree representation of it. How efficient is your procedure?

15.     Procedure POSTORDER-EVAL must be able to distinguish between the symbols   [image]  ,  [image],   [image]  and a pointer in the DATA field of a node. How should this be done?

16.     What is the computing time for POSTORDER-EVAL? First determine the logical parameters.

17.     Write an algorithm which inserts a new node T as the left child of node S in a threaded binary tree. The left pointer of S becomes the left pointer of T.

18.     Write a procedure which traverses a threaded binary tree in postorder. What is the time and space requirements of your method?

19.     Define the inverse transformation of the one which creates the associated binary tree from a forest. Are these transformations unique?

20.     Prove that preorder traversal on trees and preorder traversal on the associated binary tree gives the same result.

21.     Prove that inorder traversal for trees and inorder traversal on the associated binary tree give the same result.

22.     Using the result of example 5. 3, draw the trees after processing the instruction UNION(12,10).

23.     Consider the hypothetical game tree:

(a)     Using the minimax technique (eq. (5.3)) obtain the value of the root node .

(b)     What move should player A make?

(c)     List the nodes of this game tree in the order in which their value is computed by algorithm VE.

(d)     Using eq. (5.4) compute V' (X) for every node X in the tree.

(e)     Which nodes of this tree are not evaluated during the computation of the value of the root node using algorithm AB with X = root, l=   [image], LB = -  [image]  and D =   [image]?

24.     Show that V'(X) computed by eq. (5.4) is the same as V(X) computed by eq. (5.3) for all nodes on levels from which A is to move. For all other nodes show that V(X) computed by eq. (5.3) is the negative of V'(X) computed by eq. (5.4).

25.      Show that algorithm AB when initially called with LB = -  [image]  and D =  [image]  yields the same results as VE does for the same X and l.

26.     Prove that every binary tree is uniquely defined by its preorder and inorder sequences .

27.     Do the inorder and postorder sequences of a binary tree uniquely define the binary tree? Prove your answer.

28.     Answer exercise 27 for preorder and postorder.

29.     Write an algorithm to construct the binary tree with a given preorder and inorder sequence.

30.     Do exercise 29 for inorder and postorder.

31.     Prove that the number of distinct permutations of 1,2, ...,n obtainable by a stack is equal to the number of distinct binary trees with n nodes. (Hint: Use the concept of an inorder permutation of a tree with preorder permutation 1,2, ...,n).

32.     Using Stirling's formula derive the more accurate value of the number of binary trees with n nodes,

33.     Consider threading a binary tree using preorder threads rather than inorder threads as in the text. Is it possible to traverse a binary tree in preorder without a stack using these threads?

34.     Write an algorithm for traversing an inorder threaded binalry tree in preorder.

35.     The operation PREORD(btree)   [image]  queue returns a queue whose elements are the data items of btree in preorder. Using the operation APPENDQ(queue, queue)   [image]  queue which concatenates two queues, PREORD can be axiomatized by

PREORD(CREATE) :: = MTQ

PREORD(MAKBT(p,d,r)) :: =

APPENDQ(APPENDQ(ADDQ(MTQ,d),(PREORD(p)),PREORD(r))

Devise similar axioms for INORDER AND POSTORDER.

36.     The algorithm on page 281 performs an inorder traversal without using threads, a stack or a PARENT field. Verify that the algorithm is correct by running it on a variety of binary trees which cause every statement to execute at least once. Before attempting to study this algorithm be sure you understand MARK2 of section 4.10.

37.     Extend the equivalence algorithm discussed in section 5.8.1 so it handles the equivalencing of arrays (see section 4.6). Analyze the computing time of your solution.

38.     [Wilczynski] Following the conventions of LISP assume nodes with two fields   [image]. If A = ((a(bc))) then HEAD(A) = (a(bc)), TAIL(A) = NIL, HEAD(HEAD(A)) = a, TAIL(HEAD(A)) = ((bc)). CONS(A,B) gets a new node T, stores A in its HEAD, B in its TAIL and returns T. B must always be a list. If L = a, M = (bc) then CONS(L,M) = (abc), CONS(M,M) = ((bc)bc). Three other useful functions are: ATOM(X) which is true if X is an atom else false, NULL(X) which is true if X is NIL else false, EQUAL(X,Y) which is true if X and Y are the same atoms or equivalent lists else false.

a)     Give a sequence of HEAD, TAIL operations for extracting a from the lists: ((cat)), ((a)), (mart), (((cb))a).

b)     Write recursive procedures for: COPY, REVERSE, APPEND.

c)     Implement this "LISP" subsystem. Store atoms in an array, write procedures MAKELIST and LISTPRINT for input and output of lists.

line procedure INORDER4(T)

//inorder traversal of binary tree T using a fixed amount of

additional storage//

1      if T = 0 then return     //empty binary tree//

2      top    last_right    0; p    q    T     //initialize//

3      loop

4        loop       //move down as far as possible//

5          case

6            :LCHILD(p) = 0 and RCHILD(p) = 0:

//can't move down//

7                          print (DATA(p)); exit

8            :LCHILD(p) = 0:      //move to RCHILD(p)//

9                          print(DATA(p))       //visit p//

10                          r    RCHILD(p); RCHILD(p)    q;

q    p;p    r

11            :else:   //move to LCHILD(p)//

12                    r    LCHILD(p); LCHILD(p)    q; q    p;

p    r

13          end

14        forever

//p is a leaf node, move upwards to a node whose right

subtree hasn't yet been examined//

15        av    p          //leaf node to be used in stack//

16        loop        //move up from p//

17          case

18          :p = T: return       //can't move up from root//

19          :LCHILD(q) = 0:          //q is linked via RCHILD//

20             r    RCHILD(q); RCHILD(q)    p; p    q; q    r

21          :RCHILD(q) = 0:     //q is linked via LCHILD//

22             r    LCHILD(q); LCHILD(q)    p; p    q; q    r;

print(DATA(p))

23          :else:    //check if p is RCHILD of q//

24             if q = last_right then   [//p is RCHILD of q//

25                r    top; last--right    LCHILD(r)   //update

        last--right//

26                top    RCHILD(r);        //unstack//

27                LCHILD(r)    RCHILD(r)    0         //reset leaf

       node links//

28                r    RCHILD(q); RCHILD(q)    p; p    q; q    r]

29                                else [//p is LCHILD of q//

30                print (DATA(q))     //visit q//

31                LCHILD(av)    last_right; RCHILD(av)    top;

top    av

32                last_right    q

33                r    LCHILD(q); LCHILD(q)    p

//restore link to p//

34                r1    RCHILD(q); RCHILD(q)    r; p    r1; exit

//move right//]

35              end

36            forever

37        forever

38      end INORDER4

Go to Chapter 6     Back to Table of Contents


Fundamentals: CHAPTER 6: GRAPHS

 Return to Table of Contents

CHAPTER 6: GRAPHS

6.1 TERMINOLOGY AND REPRESENTATIONS

6.1.1 Introduction

The first recorded evidence of the use of graphs dates back to 1736 when Euler used them to solve the now classical Koenigsberg bridge problem. In the town of Koenigsberg (in Eastern Prussia) the river Pregal flows around the island Kneiphof and then divides into two. There are, therefore, four land areas bordering this river (figure 6.1). These land areas are interconnected by means of seven bridges a-g. The land areas themselves are labeled A-D. The Koenigsberg bridge problem is to determine whether starting at some land area it is possible to walk across all the bridges exactly once returning to the starting land area. One possible walk would be to start from land area B; walk across bridge a to island A; take bridge e to area D; bridge g to C; bridge d to A; bridge b to B and bridge f to D. This walk does not go across all bridges exactly once, nor does it return to the starting land area B. Euler answered the Koenigsberg bridge problem in the negative: The people of Koenigsberg will not be able to walk across each bridge exactly once and return to the starting point. He solved the problem by representing the land areas as vertices and the bridges as edges in a graph (actually a multigraph) as in figure 6.1(b). His solution is elegant and applies to all graphs. Defining the degree of a vertex to be the number of edges incident to it, Euler showed that there is a walk starting at any vertex, going through each edge exactly once and terminating at the start vertex iff the degree of each, vertex is even. A walk which does this is called Eulerian. There is no Eulerian walk for the Koenigsberg bridge problem as all four vertices are of odd degree.

Since this first application of graphs, they have been used in a wide variety of applications. Some of these applications are: analysis of electrical circuits, finding shortest routes, analysis of project planning, identification of chemical compounds, statistical mechanics, genetics, cybernetics, linguistics, social sciences, etc. Indeed, it might well be said that of all mathematical structures, graphs are the most widely used.

Figure 6.1 Section of the river Pregal in Koenigsberg and Euler's graph.

6.1.2 Definitions and Terminology

A graph, G, consists of two sets V and E. V is a finite non-empty set of vertices. E is a set of pairs of vertices, these pairs are called edges. V(G) and E(G) will represent the sets of vertices and edges of graph G. We will also write G = (V,E) to represent a graph. In an undirected graph the pair of vertices representing any edge is unordered . Thus, the pairs (v1, v2) and (v2, v1) represent the same edge. In a directed graph each edge is represented by a directed pair (v1, v2). v1 is the tail and v2 the head of the edge. Therefore <v2, v1> and <v1, v2> represent two different edges. Figure 6.2 shows three graphs G1, G2 and G3.

Figure 6.2 Three sample graphs.

The graphs G1 and G2 are undirected. G3 is a directed graph.

V (G1) = {1,2,3,4}; E(G1) = {(1,2),(1,3),(1,4),(2,3),(2,4),(3,4)}

V (G2) = {1,2,3,4,5,6,7}; E(G2) = {(1,2),(1,3),(2,4),(2,5),(3,6),(3,7)}

V (G3) = {1,2,3}; E(G3) = {<1,2>, <2,1>, <2,3>}.

Note that the edges of a directed graph are drawn with an arrow from the tail to the head. The graph G2 is also a tree while the graphs G1 and G3 are not. Trees can be defined as a special case of graphs, but we need more terminology for that. If (v1,v2) or <v1,v2> is an edge in E(G), then we require v1 [image] v2. In addition, since E(G) is a set, a graph may not have multiple occurrences of the same edge. When this restriction is removed from a graph, the resulting data object is referred to as a multigraph. The data object of figure 6.3 is a multigraph which is not a graph.

The number of distinct unordered pairs (vi,vj) with vi [image] vj in a graph with n vertices is n(n - 1)/2. This is the maximum number of edges in any n vertex undirected graph. An n vertex undirected graph with exactly n(n - 1)/2 edges is said to be complete. G1 is the complete graph on 4 vertices while G2 and G3 are not complete graphs. In the case of a directed graph on n vertices the maximum number of edges is n(n - 1).

If (v1,v2) is an edge in E(G), then we shall say the vertices v1 and v2 are adjacent and that the edge (v1,v2) is incident on vertices v1 and v2. The vertices adjacent to vertex 2 in G2 are 4, 5 and 1. The edges incident on vertex 3 in G2 are (1,3), (3,6) and (3,7). If <v1,v2> is a directed edge, then vertex v1 will be said to be adjacent to v2 while v2 is adjacent from v1. The edge <v1,v2> is incident to v1 and v2. In G3 the edges incident to vertex 2 are <1,2>, <2,1> and <2,3>.

Figure 6.3 Example of a multigraph that is not a graph.

A subgraph of G is a graph G' such that V(G') [image] V(G) and E(G') [image] E(G). Figure 6.4 shows some of the subgraphs of G1 and G3.

A path from vertex vp to vertex vq in graph G is a sequence of vertices vp,vi1,vi2, ...,vin,vq such that (vp,vi1),(vi1,vi2), ...,(vin,vq) are edges in E(G). If G' is directed then the path consists of <vp,vi1>,<vi,vi2>, ...,<vin,vq>, edges in E(G'). The length of a path is the number of edges on it. A simple path is a path in which all vertices except possibly the first and last are distinct. A path such as (1,2) (2,4) (4,3) we write as 1,2,4,3. Paths 1,2,4,3 and 1,2,4,2 are both of length 3 in G1. The first is a simple path while the second is not. 1,2,3 is a simple directed path in G3. 1,2,3,2 is not a path in G3 as the edge <3,2> is not in E(G3). A cycle is a simple path in which the first and last vertices are the same. 1,2,3,1 is a cycle in G1. 1,2,1 is a cycle in G3. For the case of directed graphs we normally add on the prefix "directed" to the terms cycle and path. In an undirected graph, G, two vertices v1 and v2 are said to be connected if there is a path in G from v1 to v2 (since G is undirected, this means there must also be a path from v2 to v1). An undirected graph is said to be connected if for every pair of distinct vertices vi, vi in V(G) there is a path from vi to vj in G. Graphs G1 and G2 are connected while G4 of figure 6.5 is not. A connected component or simply a component of an undirected graph is a maximal connected subgraph. G4 has two components H1 and H2 (see figure 6.5). A tree is a connected acyclic (i.e., has no cycles) graph . A directed graph G is said to be strongly connected if for every pair of distinct vertices vi, vj in V(G) there is a directed path from vi to vj and also from vj to vi. The graph G3 is not strongly connected as there is no path from v3 to v2. A strongly connected component is a maximal subgraph that is strongly connected. G3 has two strongly connected components.

(a) Some of the subgraphs of G1

(b) Some of the subgraphs of G3

Figure 6.4 (a) Subgraphs of G1 and (b) Subgraphs of G3

Figure 6.5 A graph with two connected components.

Figure 6.6 Strongly connected components of G3.

The degree of a vertex is the number of edges incident to that vertex. The degree of vertex 1 in G1 is 3. In case G is a directed graph, we define the in-degree of a vertex v to be the number of edges for which v is the head. The out-degree is defined to be the number of edges for which v is the tail. Vertex 2 of G3 has in-degree 1, out-degree 2 and degree 3. If di is the degree of vertex i in a graph G with n vertices and e edges, then it is easy to see that e = (1/2) [image] .

In the remainder of this chapter we shall refer to a directed graph as a digraph. An undirected graph will sometimes be referred to simply as a graph.

6.1.3 Graph Representations

While several representations for graphs are possible, we shall study only the three most commonly used: adjacency matrices, adjacency lists and adjacency multilists. Once again, the choice of a particular representation will depend upon the application one has in mind and the functions one expects to perform on the graph.

Adjacency Matrix

Let G = (V,E) be a graph with n vertices, n [image] 1. The adjacency matrix of G is a 2-dimensional n [image] n array, say A, with the property that A(i,j) = 1 iff the edge (vi,vj) (<vi,vj> for a directed graph) is in E(G). A(i,j) = 0 if there is no such edge in G. The adjacency matrices for the graphs G1, G3 and G4 are shown in figure 6.7. The adjacency matrix for an undirected graph is symmetric as the edge (vi,vj) is in E(G) iff the edge (vj,vi) is also in E(G). The adjacency matrix for a directed graph need not be symmetric (as is the case for G3). The space needed to represent a graph using its adjacency matrix is n2 bits. About half this space can be saved in the case of undirected graphs by storing only the upper or lower triangle of the matrix.

From the adjacency matrix, one may readily determine if there is an edge connecting any two vertices i and j. For an undirected graph the degree of any vertex i is its row sum [image] A(i,j). For a directed graph the row sum is the out-degree while the column sum is the in-degree. Suppose we want to answer a nontrivial question about graphs such as: How many edges are there in G or is G connected. Using adjacency matrices all algorithms will require at least O(n2) time as n2 - n entries of the matrix (diagonal entries are zero) have to be examined. When graphs are sparse, i.e., most of the terms in the adjacency matrix are zero, one would expect that the former questions would be answerable in significantly less time, say O(e + n) where e is the number of edges in G and e << n2/2. Such a speed up can be made possible through the use of linked lists in which only the edges that are in G are represented. This leads to the next representation for graphs.

Figure 6.7 Adjacency matrices for (i) G1, (ii) G3 and (iii) G4.

Adjacency Lists

In this representation the n rows of the adjacency matrix are represented as n linked lists. There is one list for each vertex in G. The nodes in list i represent the vertices that are adjacent from vertex i. Each node has at least two fields: VERTEX and LINK. The VERTEX fields contain the indices of the vertices adjacent to vertex i. The adjacency lists for G1, G3 and G4 are shown in figure 6.8. Each list has a headnode. The headnodes are sequential providing easy random access to the adjacency list for any particular vertex. In the case of an undirected graph with n vertices and e edges, this representation requires n head nodes and 2e list nodes. Each list node has 2 fields. In terms of the number of bits of storage needed, this count should be multiplied by log n for the head nodes and log n + log e for the list nodes as it takes O(log m) bits to represent a number of value m. Often one can sequentially pack the nodes on the adjacency lists and eliminate the link fields.

(i) Adjacency list for G1

(ii) Adjacency lists for G3

(iii) Adjacency list for G4

Figure 6.8 Adjacency Lists

The degree of any vertex in an undirected graph may be determined by just counting the number of nodes in its adjacency list. The total number of edges in G may, therefore, be determined in time O(n + e). In the case of a digraph the number of list nodes is only e. The out-degree of any vertex may be determined by counting the number of nodes on its adjacency list. The total number of edges in G can, therefore, be determined in O(n + e). Determining the in-degree of a vertex is a little more complex. In case there is a need to repeatedly access all vertices adjacent to another vertex then it may be worth the effort to keep another set of lists in addition to the adjacency lists. This set of lists, called inverse adjacency lists, will contain one list for each vertex. Each list will contain a node for each vertex adjacent to the vertex it represents (figure 6.9). Alternatively, one could adopt a

Figure 6.9 Inverse adjacency lists for G3.

simplified version of the list structure used for sparse matrix representation in section 4.6. Each node would now have four fields and would represent one edge. The node structure would be

Figure 6.10 shows the resulting structure for the graph G3 . The headnodes are stored sequentially.

The nodes in the adjacency lists of figure 6.8 were ordered by the indices of the vertices they represented. It is not necessary that lists be ordered in this way and, in general, the vertices may appear in any order. Thus, the adjacency lists of figure 6.11 would be just as valid a representation of G1.

Adjacency Multilists

In the adjacency list representation of an undirected graph each edge (vi,vj) is represented by two entries, one on the list for vi and the

Figure 6.10 Orthogonal List Representation for G3.

Figure 6.11 Alternate Form Adjacency List for G1.

other on the list for vj. As we shall see, in some situations it is necessary to be able to determine the second entry for a particular edge and mark that edge as already having been examined. This can be accomplished easily if the adjacency lists are actually maintained as multilists (i.e., lists in which nodes may be shared among several lists). For each edge there will be exactly one node, but this node will be in two lists, i.e., the adjacency lists for each of the two nodes it is incident to. The node structure now becomes where

M is a one bit mark field that may be used to indicate whether or not the edge has been examined. The storage requirements are the same as for normal adjacency lists except for the addition of the mark bit M. Figure 6.12 shows the adjacency multilists for G1. We shall study multilists in greater detail in Chapter 10.

Sometimes the edges of a graph have weights assigned to them. These weights may represent the distance from one vertex to another or the

cost of going from one vertex to an adjacent vertex. In this case the adjacency matrix entries A(i,j) would keep this information, too. In the case of adjacency lists and multilists this weight information may be kept in the list nodes by including an additional field. A graph with weighted edges is called a network.

The lists are:  vertex 1:  N1    N2    N3

                vertex 2:  N1    N4    N5

                vertex 3:  N2    N4    N6

                vertex 4:  N3    N5    N6

Figure 6.12 Adjacency Multilists for G1.

6.2 TRAVERSALS, CONNECTED COMPONENTS AND SPANNING TREES

Given the root node of a binary tree, one of the most common things one wishes to do is to traverse the tree and visit the nodes in some order. In the chapter on trees, we defined three ways (preorder, inorder, and postorder) for doing this. An analogous problem arises in the case of graphs. Given an undirected graph G = (V,E) and a vertex v in V(G) we are interested in visiting all vertices in G that are reachable from v (i.e., all vertices connected to v). We shall look at two ways of doing this: Depth First Search and Breadth First Search.

Depth First Search

Depth first search of an undirected graph proceeds as follows. The start vertex v is visited. Next an unvisited vertex w adjacent to v is selected and a depth first search from w initiated. When a vertex u is reached such that all its adjacent vertices have been visited, we back up to the last vertex visited which has an unvisited vertex w adjacent to it and initiate a depth first seaerch from w. The search terminates when no unvisited vertex can be reached from any of the visited one. This procedure is best described recursively as in

procedure DFS(v)

//Given an undirected graph G = (V,E) with n vertices and an array

VlSlTED(n) initially set to zero, this algorithm visits all vertices

reachable from v. G and VISITED are global.//

VISITED (v)    1

for each vertex w adjacent to v do

if VISlTED(w) = 0 then call DFS(w)

end

end DFS

In case G is represented by its adjacency lists then the vertices w adjacent to v can be determined by following a chain of links. Since the algorithm DFS would examine each node in the adjacency lists at most once and there are 2e list nodes, the time to complete the search is O(e). If G is represented by its adjacency matrix, then the time to determine all vertices adjacent to v is O(n). Since at most n vertices are visited, the total time is O(n2).

The graph G of figure 6.13(a) is represented by its adjacency lists as in figure 6.13(b). If a depth first search is initiated from vertex v1, then the vertices of G are visited in the order: v1, v2, v4, v8, v5, v6, v3, v7. One may easily verify that DFS (v1) visits all vertices connected to v1. So, all the vertices visited, together with all edges in G incident to these vertices form a connected component of G.

Breadth First Search

Starting at vertex v and marking it as visited, breadth first search differs from depth first search in that all unvisited vertices adjacent to v are visited next. Then unvisited vertices adjacent to these vertices are visited and so on. A breadth first search beginning at vertex v1 of the graph in figure 6.13(a) would first visit v1 and then v2 and v3. Next vertices v4, v5, v6 and v7 will be visited and finally v8. Algorithm BFS gives the details.

procedure BFS(v)

//A breadth first search of G is carried out beginning at vertex v.

All vertices visited are marked as VISlTED(i)= 1. The graph G

and array VISITED are global and VISITED is initialized to zero.//

VISITED (v)    1

initialize Q to be empty          //Q is a queue//

loop

for all vertices w adjacent to v do

if VISITED(w) = 0           //add w to queue//

then [call ADDQ(w,Q); VISITED(w)  1]

//mark w as VISITED//

end

if Q is empty then return

call DELETEQ(v,Q)

forever

end BFS

Figure 6.13 Graph G and Its Adjacency Lists.

Each vertex visited gets into the queue exactly once, so the loop forever is iverated at most n times. If an adjacency matrix is used, then the for loop takes O(n) time for each vertex visited. The total time is, therefore, O(n2). In case adjacency lists are used the for loop has a total cost of d1 + ... + dn = O(e) where di = degree (vi). Again, all vertices visited, together with all edges incident to them form a connected component of G.

We now look at two simple applications of graph traversal: (i) finding the components of a graph, and (ii) finding a spanning tree of a connected graph.

Connected Components

If G is an undirected graph, then one can determine whether or not it is connected by simply making a call to either DFS or BFS and then determining if there is any unvisited vertex. The time to do this is O(n2) if adjacency matrices are used and O(e) if adjacency lists are used. A more interesting problem is to determine all the connected components of a graph. These may be obtained by making repeated calls to either DFS(v) or BFS(v), with v a vertex not yet visited. This leads to algorithm COMP which determines all the connected components of G. The algorithm uses DFS. BFS may be used instead if desired.The computing time is not affected.

procedure COMP(G,n)

//determine the connected components of G. G has n    1 vertices.

VISITED is now a local array.//

for i    1 to n do

VISITED(i)    0      //initialize all vertices as unvisited//

end

for i   1 to n do

if VISITED(i) = 0 then [call DFS(i);        //find a component//

output all newly visited vertices together

with all edges incident to them]

end

end COMP

If G is represented by its adjacency lists, then the total time taken by DFS is O(e). The output can be completed in time O(e) if DFS keeps a list of all newly visited vertices. Since the for loops take O(n) time, the total time to generate all the connected components is O(n + e).

By the definition of a connected component, there is a path between every pair of vertices in the component and there is no path in G from vertex v to w if v and w are in two different components. Hence, if A is the adjacency matrix of an undirected graph (i.e., A is symmetric) then its transitive closure A+ may be determined in O(n2) time by first determining the connected components. A+(i,j) = 1 iff there is a path from vertex i to j. For every pair of distinct vertices in the same component A + (i,j) = 1. On the diagonal A + (i,i) = 1 iff the component containing i has at least 2 vertices. We shall take a closer look at transitive closure in section 6.3.

Spanning Trees and Minimum Cost Spanning Trees

When the graph G is connected, a depth first or breadth first search starting at any vertex, visits all the vertices in G. In this case the edges of G are partitioned into two sets T (for tree edges) and B (for back edges), where T is the set of edges used or traversed during the search and B the set of remaining edges. The set T may be determined by inserting the statement T [image] T [image] {(v,w)} in the then clauses of DFS and BFS. The edges in T form a tree which includes all the vertices of G. Any tree consisting solely of edges in G and including all vertices in G is called a spanning tree. Figure 6.14 shows a graph and some of its spanning trees. When either DFS or BFS are used the edges of T form a spanning tree. The spanning tree resulting from a call to DFS is known as a depth first spanning tree. When BFS is used, the resulting spanning tree is called a breadth first spanning tree.

Figure 6.14 A Complete Graph and Three of Its Spanning Trees.

Figure 6.15 shows the spanning trees resulting from a depth first and breadth first search starting at vertex v1 in the graph of figure 6.13. If any of the edges (v,w) in B (the set of back edges) is introduced into the spanning tree T, then a cycle is formed. This cycle consists of the edge (v,w) and all the edges on the path from w to v in T. If the edge (8,7) is introduced into the DFS spanning tree of figure 6.15(a), then the resulting cycle is 8,7,3,6,8.

Spanning trees find application in obtaining an independent set of circuit equations for an electrical network. First, a spanning tree for the network is obtained. Then the edges in B (i.e., edges not in the spanning tree) are introduced one at a time. The introduction of each such edge results in a cycle. Kirchoff's second law is used on this cycle to obtain a circuit equation. The cycles obtained in this way are independent (i.e., none of these cycles can be obtained by taking a linear combination of the remaining cycles) as each contains an edge from B which is not contained in any other cycle. Hence, the circuit equations so obtained are also independent. In fact, it may be shown that the cycles obtained by introducing the edges of B one at a time into the resulting spanning tree form a cycle basis and so all other cycles in the graph can be constructed by taking a linear combination of the cycles in the basis (see Harary in the references for further details).

(a) DFS (1) Spanning Tree

(b) BFS (1) Spanning Tree

Figure 6.15 DFS and BFS Spanning Trees for Graph of Figure 6.13.

It is not difficult to imagine other applications for spanning trees. One that is of interest arises from the property that a spanning tree is a minimal subgraph G' of G such that V(G') = V(G) and G' is connected (by a minimal subgraph, we mean one with the fewest number of edges). Any connected graph with n vertices must have at least n - 1 edges and all connected graphs with n - 1 edges are trees. If the nodes of G represent cities and the edges represent possible communication links connecting 2 cities, then the minimum number of links needed to connect the n cities is n - 1. The spanning trees of G will represent all feasible choices. In any practical situation, however, the edges will have weights assigned to them. These weights might represent the cost of construction, the length of the link, etc. Given such a weighted graph one would then wish to select for construction a set of communication links that would connect all the cities and have minimum total cost or be of minimum total length. In either case the links selected will have to form a tree (assuming all weights are positive). In case this is not so, then the selection of links contains a cycle. Removal of any one of the links on this cycle will result in a link selection of less cost connecting all cities. We are, therefore, interested in finding a spanning tree of G with minimum cost. The cost of a spanning tree is the sum of the costs of the edges in that tree.

One approach to determining a minimum cost spanning tree of a graph has been given by Kruskal. In this approach a minimum cost spanning tree, T, is built edge by edge. Edges are considered for inclusion in T in nondecreasing order of their costs. An edge is included in T if it does not form a cycle with the edges already in T. Since G is connected and has n > 0 vertices, exactly n - 1 edges will be selected for inclusion in T. As an example, consider the graph of figure 6.16(a). The edges of this graph are considered for inclusion in the minimum cost spanning tree in the order (2,3), (2,4), (4,3), (2,6), (4,6), (1,2), (4,5), (1,5) and (5,6). This corresponds to the cost sequence 5, 6, 10, 11, 14, 16, 18, 19 and 33. The first two edges (2,3) and (2,4) are included in T. The next edge to be considered is (4,3). This edge, however, connects two vertices already connected in T and so it is rejected. The edge (2,6) is selected while (4,6) is rejected as the vertices 4 and 6 are already connected in T and the inclusion of (4,6) would result in a cycle. Finally, edges (1,2) and (4,5) are included. At this point, T has n - 1 edges and is a tree spanning n vertices. The spanning tree obtained (figure 6.16(b)) has cost 56. It is somewhat surprising that this straightforward approach should always result in a minimum spanning tree. We shall soon prove that this is indeed the case. First, let us look into the details of the algorithm. For clarity, the Kruskal algorithm is written out more formally in figure 6.18. Initially, E is the set of all edges in G. The only functions we wish to perform on this set are: (i) determining an edge with minimum cost (line 3), and (ii) deleting that edge (line 4). Both these functions can be performed efficiently if the edges in E are maintained as a sorted sequential list. In Chapter 7 we shall see how to sort these edges into nondecreasing order in time O(e log e), where e is the number of edges in E. Actually, it is not essential to sort all the edges so long as the next edge for line 3 can be determined easily. It will be seen that the heap of Heapsort (section 7.6) is ideal for this and permits the next edge to be determined and deleted in O(log e) time. The construction of the heap itself takes O(e) time.

Figure 6.16 Graph and A Spanning Tree of Minimum Cost.

Figure 6.17 Stages in Kruskal's Algorithm Leading to a Minimum Cost Spanning Tree.

1  T

2  while T contains less than n - 1 edges and E not empty do

3    choose an edge (v,w) from E of lowest cost;

4    delete (v,w) from E;

5    if (v,w) does not create a cycle in T

6        then add (v,w) to T

7        else discard (v,w)

8  end

9  if T contains fewer than n - 1 edges then print ('no spanning tree')

Figure 6.18 Early Form of Minimum Spanning Tree Algorithm-Kruskal.

In order to be able to perform steps 5 and 6 efficiently, the vertices in G should be grouped together in such a way that one may easily determine if the vertices v and w are already connected by the earlier selection of edges. In case they are, then the edge (v,w) is to be discarded. If they are not, then (v,w) is to be added to T. One possible grouping is to place all vertices in the same connected component of T into a set (all connected components of T will also be trees). Then, two vertices v,w are connected in T iff they are in the same set. For example, when the edge (4,3) is to be considered, the sets would be {1}, {2,3,4}, {5}, {6}. Vertices 4 and 3 are already in the same set and so the edge (4,3) is rejected. The next edge to be considered is (2,6). Since vertices 2 and 6 are in different sets, the edge is accepted. This edge connects the two components {2,3,4} and {6} together and so these two sets should be unioned to obtain the set representing the new component. Using the set representation of section 5.8 and the FIND and UNION algorithms of that section we can obtain an efficient implementation of lines 5 and 6. The computing time is, therefore, determined by the time for lines 3 and 4 which in the worst case is O(e log e). We leave the writing of the resulting algorithm as an exercise. Theorem 6.1 proves that the algorithm resulting from figure 6.18 does yield a minimum spanning tree of G. First, we shall obtain a result that will be useful in the proof of this theorem.

Definition: A spanning forest of a graph G = (V,E) is a collection of vertex disjoint trees Ti = (Vi, Ei), 1 [image] i [image] k such that [image] and Ei [image] E(G), 1 [image] i [image] k.

Lemma 6.1: Let Ti = (Vi,Ei), 1 [image] i [image] k, k > 1, be a spanning forest for the connected undirected graph G= (V,E). Let w be a weighting function for E(G) and let e = (u, v) be an edge of minimum weight such that if u [image] Vi then v [image] Vi. Let i = 1 and [image]. There is a spanning tree for G which includes E' [image] {e} and has minimum weight among all spanning trees for G that include E'.

Proof: If the lemma is false, then there must be a spanning tree T= (V,E") for G such that E" includes E' but not e and T has a weight less than the weight of the minimum spanning tree for G including E' [image] {e}. Since T is a spanning tree, it has a path from u to v. Consequently, the addition of e to E" creates a unique cycle (exercise 22). Since u [image] V1 and v [image] V1, it follows that there is another edge e' = (u',v') on this cycle such that u' [image] V1 and v' [image] V1 (v' may be v). By assumption, w(e) [image] w(e'). Deletion of the edge e' from E" [image] {e} breaks this cycle and leaves behind a spanning tree T' that include E' [image] {e}. But, since w(e) [image] w(e'), it follows that the weight of T' is no more than the weight of T. This contradicts the assumption on T. Hence, there is no such T and the lemma is proved.

Theorem 6.1: The algorithm described in figure 6.18 generates a minimum spanning tree.

Proof: The proof follows from Lemma 6.1 and the fact that the algorithm begins with a spanning forest with no edges and then examines the edges of G in nondecreasing order of weight.

6.3 SHORTEST PATHS AND TRANSITIVE CLOSURE

Graphs may be used to represent the highway structure of a state or country with vertices representing cities and edges representing sections of highway. The edges may then be assigned weights which might be either the distance between the two cities connected by the edge or the average time to drive along that section of highway. A motorist wishing to drive from city A to city B would be interested in answers to the following questions:

(i) Is there a path from A to B?

(ii) If there is more than one path from A to B, which is the shortest path?

The problems defined by (i) and (ii) above are special cases of the path problems we shall be studying in this section. The length of a path is now defined to be the sum of the weights of the edges on that path rather than the number of edges. The starting vertex of the path will be referred to as the source and the last vertex the destination. The graphs will be digraphs to allow for one way streets. Unless otherwise stated, we shall assume that all weights are positive.

Single Source All Destinations

In this problem we are given a directed graph G = (V,E), a weighting function w(e) for the edges of G and a source vertex vo. The problem is to determine the shortest paths from vo to all the remaining vertices of G. It is assumed that all the weights are positive. As an example, consider the directed graph of figure 6.19(a). The numbers on the edges are the weights. If vo is the source vertex, then the shortest path from vo to v1 is vo v2 v3 v1. The length of this path is 10 + 15 + 20 = 45. Even though there are three edges on this path, it is shorter than the path vo v1 which is of length 50. There is no path from vo to v5. Figure 6.19(b) lists the shortest paths from vo to v1, v2, v3 and v4. The paths have been listed in nondecreasing order of path length. If we attempt to devise an algorithm which generates the shortest paths in this order, then we can make several observations. Let S denote the set of vertices (including vo) to which the shortest paths have already been found. For w not in S, let DIST(w) be the length of the shortest path starting from vo going through only those vertices which are in S and ending at w. We observe that:

Figure 6.19 Graph and Shortest Paths from v0 to All Destinations.

(i) If the next shortest path is to vertex u, then the path begins at vo, ends at u and goes through only those vertices which are in S. To prove this we must show that all of the intermediate vertices on the shortest path to u must be in S. Assume there is a vertex w on this path that is not in S. Then, the vo to u path also contains a path from vo to w which is of length less than the vo to u path. By assumption the shortest paths are being generated in nondecreasing order of path length, and so the shorter path vo to w must already have been generated. Hence, there can be no intermediate vertex which is not in S.

(ii) The destination of the next path generated must be that vertex u which has the minimum distance, DIST(u), among all vertices not in S. This follows from the definition of DIST and observation (i). In case there are several vertices not in S with the same DIST, then any of these may be selected.

(iii) Having selected a vertex u as in (ii) and generated the shortest vo to u path, vertex u becomes a member of S. At this point the length of the shortest paths starting at vo, going through vertices only in S and ending at a vertex w not in S may decrease. I.e., the value of DIST(w) may change. If it does change, then it must be due to a shorter path starting at vo going to u and then to w. The intermediate vertices on the vo to u path and the u to w path must all be in S. Further, the vo to u path must be the shortest such path, otherwise DIST(w) is not defined properly. Also, the u to w path can be chosen so as to not contain any intermediate vertices. Therefore, we may conclude that if DIST(w) is to change (i.e., decrease), then it is because of a path from vo to u to w where the path from vo to u is the shortest such path and the path from u to w is the edge <u,w>. The length of this path is DIST(u) + length (<u,w>).

The algorithm SHORTEST_PATH as first given by Dijkstra, makes use of these observations to determine the cost of the shortest paths from vo to all other vertices in G. The actual generation of the paths is a minor extension of the algorithm and is left as an exercise. It is assumed that the n vertices of G are numbered 1 through n. The set S is maintained as a bit array with S(i) = 0 if vertex i is not in S and S(i) = 1 if it is. It is assumed that the graph itself is represented by its cost adjacency matrix with COST(i,j) being the weight of the edge <i,j>. COST(i,j) will be set to some large number, + [image] , in case the edge <i,j> is not in E(G). For i= j, COST(i,j) may be set to any non-negative number without affecting the outcome of the algorithm.

Analysis of Algorithm SHORTEST PATH

From our earlier discussion, it is easy to see that the algorithm works. The time taken by the algorithm on a graph with n vertices is O(n2). To see this note that the for loop of line 1 takes O(n) time. The while loop is executed n - 2 times. Each execution of this loop requires O(n) time at line 6 to select the next vertex and again at lines 8-10 to update DIST. So the total time for the while loop is O(n2). In case a list T of vertices currently not in S is maintained, then the number of nodes on this list would at any time be n - num. This would speed up lines 6 and 8-10, but the asymptotic time would remain O(n2). This and other variations of the algorithm are explored in the exercises.

procedure SHORTEST-PATH (v,COST,DIST,n)

//DIST(j), 1    j    n is set to the length of the shortest path

from vertex v to vertex j in a digraph G with n vertices. DIST(v)

is set to zero. G is represented by its cost adjacency matrix,

COST(n,n)//

declare S (1: n)

1     for i    1 to n do     //initialize set S to empty//

2        S(i)    0; DIST(i)    COST(v,i)

3     end

4     S(v)    1; DIST(v)    0; num    2      //put vertex v in set

S//

5     while num < n do     //determine n - 1 paths from vertex v//

6       choose u: DIST(u) = min {DIST(w)}

S(w)=0

7       S(u)    1; num    num + 1    //put vertex u in set S//

8       for all w with S(w) = 0 do  //update distances//

9          DIST(w)    min {DIST(w),DIST(u) + COST(u,w)}

10       end

11     end

12 end SHORTEST-PATH

Any shortest path algorithm must examine each edge in the graph at least once since any of the edges could be in a shortest path. Hence, the minimum possible time for such an algorithm would be O(e). Since cost adjacency matrices were used to represent the graph, it takes O(n2) time just to determine which edges are in G and so any shortest path algorithm using this representation must take O(n2). For this representation then, algorithm SHORTEST PATH is optimal to within a constant factor. Even if a change to adjacency lists is made, only the overall time for the for loop of lines 8-10 can be brought down to O(e) (since the DIST can change only for those vertices adjacent from u). The total time for line 6 remains O(n2).

Example 6.1: Consider the 8 vertex digraph of figure 6.20(a) with cost adjacency matrix as in 6.20(b). The values of DIST and the vertices selected at each iteration of the while loop of line 5 for finding all the shortest paths from Boston are shown in table 6.21. Note that the algorithm terminates when only seven of the eight vertices are in S. By the definition of DIST, the distance of the last vertex, in this case

Figure 6.20(a)

Figure 6.20(b) Cost Adjacency Matrix for Figure 6.20(a). All Entries not Shown are + .

                 Vertex          LA    SF     D      C    B   NY    M    NO

Iteration  S     Selected  DIST  (1)   (2)   (3)    (4)  (5)  (6)  (7)  (8)

Initial               --          +      +      +      1500  0  250   +      +

1        5            6           +      +      +      1250  0  250  1150  1650

2        5,6          7           +      +      +      1250  0  250  1150  1650

3        5,6,7        4           +      +      2450  1250  0  250  1150  1650

4        5,6,7,4      8           3350  +      2450  1250  0  250  1150  1650

5        5,6,7,4,8    3           3350  3250  2450  1250  0  250  1150  1650

6        5,6,7,4,8,3  2           3350  3250  2450  1250  0  250  1150  1650

         5,6,7,4,8,3,2

Table 6.21 Action of SHORTEST_PATH

Los Angeles is correct as the shortest path from Boston to Los Angeles can go through only the remaining six vertices.

All Pairs Shortest Paths

The all pairs shortest path problem calls for finding the shortest paths between all pairs of vertices vi,vj, i [image] j. One possible solution to this is to apply the algorithm SHORTEST__ PATH n times, once with each vertex in V(G) as the source. The total time taken would be O(n3). For the all pairs problem, we can obtain a conceptually simpler algorithm which will work even when some edges in G have negative weights so long as G has no cycles with negative length. The computing time of this algorithm will still be O(n3) though the constant factor will be smaller.

The graph G is represented by its cost adjacency matrix with COST(i.i)= 0 and COST(i,j) = + [image] in case edge <i,j>, i [image] j is not in G. Define Ak(i,j) to be the cost of the shortest path from i to j going through no intermediate vertex of index greater than k. Then, An(i,j) will be the cost of the shortest i to j path in G since G contains no vertex with index greater than n. Ao(i,j) is just COST(i,j) since the only i to j paths allowed can have no intermediate vertices on them. The basic idea in the all pairs algorithm is to successively generate the matrices A0, A1, A2, ...,An. If we have already generated Ak-1, then we may generate Ak by realizing that for any pair of vertices i,j either (i) the shortest path from i to j going through no vertex with index greater than k does not go through the vertex with index k and so its cost is Ak-1(i,j); or (ii) the shortest such path does go through vertex k. Such a path consists of a path from i to k and another one from k to j. These paths must be the shortest paths from i to k and from k to j going through no vertex with index greater than k - 1, and so their costs are Ak-1(i,k) and Ak-1(k,j). Note that this is true only if G has no cycle with negative length containing vertex k. If this is not true, then the shortest i to j path going through no vertices of index greater than k may make several cycles from k to k and thus have a length substantially less than Ak-1 (i,k) + Ak-1 (k,j) (see example 6.2). Thus, we obtain the following formulas for Ak (i,j):

Ak(i,j) = min {Ak-1(i,j), Ak-1(i,k) + Ak-1 (k,j)} k [image] 1

and

Ao(i,j) = COST(i,j).

Example 6.2: Figure 6.22 shows a digraph together with its matrix A0. For this graph A2(1,3) [image] min {A1(1,3), A1(1,2) + A1(2,3)} = 2. Instead we see that A2(1,3) = - [image] as the length of the path

1,2,1,2,1,2, ......,1,2,3

can be made arbitrarily small. This is so because of the presence of the cycle 1 2 1 which has a length of - 1.

Figure 6.22 Graph with Negative Cycle

The algorithm ALL__COSTS computes An(i,j). The computation is done in place so the superscript on A is dropped. The reason this computation can be carried out in place is that Ak(i,k) = Ak-1 (i,k) and Ak(k,j) = Ak-1 (k,j) and so the in place computation does not alter the outcome.

procedure ALL_COSTS(COST,A,n)

//COST(n,n) is the cost adjacency matrix of a graph with n

vertices; A(i,j) is the cost of the shortest path between vertices

vi,vj. COST(i,i) = 0, 1    i    n//

1  for i    1 to n do

2    for j    1 to n do

3        A (i,j)    COST (i,j)   //copy COST into A//

4    end

5  end

6  for k    1 to n do    //for a path with highest vertex index k//

7    for i    1 to n do    //for all possible pairs of vertices//

8      for j    1 to n do

9          A (i,j)    min {A (i,j),A(i,k) + A(k,j)}

10      end

11    end

12  end

13 end ALL_COSTS

This algorithm is especially easy to analyze because the looping is independent of the data in the matrix A.

The total time for procedure ALL_COSTS is O(n3). An exercise examines the extensions needed to actually obtain the (i,j) paths with these lengths. Some speed up can be obtained by noticing that the innermost for loop need be executed only when A (i,k) and A (k,j) are not equal to [image].

Example 6.3: : Using the graph of figure 6.23(a) we obtain the cost matrix of figure 6.23(b). The initial A matrix, Ao plus its value after 3 iterations A1, A2 ,A3 is given in figure 6.24.

Figure 6.23 Directed Graph and Its Cost Matrix.

A0..1  2  3      A1  1  2  3

----------------------------

1   0  4  11       1   0   4  11

2   6  0   2       2   6   0   2

3   3      0        3   3   7   0

A2  1  2  3      A3  1  2  3

----------------------------

1  0  4  6       1   0  4  6

2  6  0  2       2   5  0  2

3  3  7  0       3   3  7  0

Figure 6.24 Matrices Ak Produced by ALL_COSTS for the Digraph of Figure 6.23.

Transitive Closure

A problem related to the all pairs shortest path problem is that of determining for every pair of vertices i,j in G the existence of a path from i to j. Two cases are of interest, one when all path lengths (i.e., the number of edges on the path) are required to be positive and the other when path lengths are to be nonnegative. If A is the adjacency matrix of G, then the matrix A+ having the property A+ (i,j) = 1 if there is a path of length > 0 from i to j and 0 otherwise is called the transitive closure matrix of G. The matrix A* with the property A* (i,j) = 1 if there is a path of length   [image] 0 from i to j and 0 otherwise is the reflexive transitive closure matrix of G.

Figure 6.25 Graph G and Its Adjacency Matrix A, A+ and A*

Figure 6.25 shows A+ and A* for a digraph. Clearly, the only difference between A* and A+ is in the terms on the diagonal. A+(i,i) = 1 iff there a cycle of length > 1 containing vertex i while A*(i,i) is always one as there is a path of length 0 from i to i. If we use algorithm ALL_COSTS with COST(i,j) = 1 if <i,j> is an edge in G and COST(i,j) = +   [image]  if <i,j> is not in G, then we can easily obtain A+ from the final matrix A by letting A+ (i,j) = 1 iff A (i,j) < +   [image]. A* can be obtained from A+ by setting all diagonal elements equal 1. The total time is O(n3). Some simplification is achieved by slightly modifying the algorithm. In this modification the computation of line 9 of ALL_COSTS becomes A(i,j)   [image]  A (i,j) or (A(i,k) and A(k,)) and COST(i,j) is just the adjacency matrix of G. With this modification, A need only be a bit matrix and then the final matrix A will be A+.

6.4 ACTIVITY NETWORKS, TOPOLOGICAL SORT AND CRITICAL PATHS

Topological Sort

All but the simplest of projects can be subdivided into several subprojects called activities. The successful completion of these activities will result in the completion of the entire project. A student working towards a degree in Computer Science will have to complete several courses successfully. The project in this case is to complete the major, and the activities are the individual courses that have to be taken. Figure 6.26 lists the courses needed for a computer science major at a hypothetical university. Some of these courses may be taken independently of others while other courses have prerequisites and can be taken only if all their prerequisites have already been taken. The data structures course cannot be started until certain programming and math courses have been completed. Thus, prerequisites define precedence relations between the courses. The relationships defined may be more clearly represented using a directed graph in which the vertices represent courses and the directed edges represent prerequisites. This graph has an edge <i,j> iff course i is a prerequisite for course j.

Definition: A directed graph G in which the vertices represent tasks or activities and the edges represent precedence relations between tasks is an activity on vertex network or AOV-network.

Definition: Vertex i in an AOV network G is a predecessor of vertex j iff there is a directed path from vertex i to vertex j. i is an immediate predecessor of j iff <i,j> is an edge in G. If i is a predecessor of j, then j is a successor of i. If i is an immediate predecessor of j, then j is an immediate successor of i.

Figure 6.26(b) is the AOV-network corresponding to the courses of figure 6.26(a). C3, C4 and C10 are the immediate predecessors of C7. C2, C3 and C4 are the immediate successors of C1. C12 is a succesor of C4 but not an immediate successor. The precedence relation defined by the set of edges on the set of vertices is readily seen to be transitive. (Recall that a relation   [image]  is transitive iff it is the case that for all triples i,j,k, i   [image]  j and j   [image]  k   [image]  i k.) In order for an AOV-network to represent a feasible project, the precedence relation should also be irreflexive.

Course Number      Course Name               Prerequisites

     Cl   Introduction to Programming        None

     C2   umerical Analysis                  Cl, C14

     C3   ata Structures                     Cl, C14

     C4   ssembly Language                   Cl, C13

     C5   utomata Theory                     C15

     C6   rtificial Intelligence             C3

     C7   omputer Graphics                   C3, C4, C10

     C8   achine Arithmetic                  C4

     C9   nalysis of Algorithms              C3

     C10  Higher Level Languages             C3, C4

     Cll  Compiler Writing                   C10

     C12  Operating Systems                  Cl l

     C13  Analytic Geometry and Calculus I   None

     C14  Analytic Geometry and Calculus II  C13

     C15  Linear Algebra                     C14

(a) Courses Needed for a Computer Science Degree at Some Hypothetical University

(b) AOV-Network Representing Courses as Vertices and Prerequisites as Edges

Figure 6.26 An Activity on Vertex Network

Definition: A relation   [image]  is irreflexive on a set S if for no element x in S it is the case that x  [image] x. A precedence relation which is both transitive and irreflexive is a partial order.

If the precedence relation is not irreflexive, then there is an activity which is a predecessor of itself and so must be completed before it can be started. This is clearly impossible. When there are no inconsistencies of this type, the project is feasible. Given an AOV network one of our concerns would be to determine whether or not the precedence relation defined by its edges is irreflexive. This is identical to determining whether or not the network contains any directed cycles. A directed graph with no directed cycles is an acyclic graph. Our algorithm to test an AOV-network for feasibility will also generate a linear ordering, vi1, vi2, ...,vin, of the vertices (activities). This linear ordering will have the property that if i is a predecessor of j in the network then i precedes j in the linear ordering. A linear ordering with this property is called a topological order. For the network of figure 6.26(b) two of the possible topological orders are: Cl, C13, C4, C8, C14, C15, C5, C2, C3, C10, C7, C11, C12, C6, C9 and C13, C14, C15, C5, Cl, C4, C8, C2, C3, C10, C7, C6, C9, Cll, C12. If a student were taking just one course per term, then he would have to take them in topological order. If the AOV-network represented the different tasks involved in assembling an automobile, then these tasks would be carried out in topological order on an assembly line. The algorithm to sort the tasks into topological order is straightforward and proceeds by listing out a vertex in the network that has no predecessor. Then, this vertex together with all edges leading out from it are deleted from the network. These two steps are repeated until either all vertices have been listed or all remaining vertices in the network have predecessors and so none can be performed. In this case there is a cycle in the network and the project is infeasible. Figure 6.27 is a crude form of the algorithm.

input the AOV-network. Let n be the number of vertices.

1  for i    1 to n do    //output the vertices//

2    if every vertex has a predecessor

then [the network has a cycle and is infeasible. stop]

3    pick a vertex v which has no predecessors

4    output v

5    delete v and all edges leading out of v from the network

6  end

Figure 6.27 Design of a Topological Sorting Algorithm

Trying this out on the network of figure 6.28 we see that the first vertex to be picked in line 2 is v1, as it is the only one with no predecessors. v1 and the edges <v1,v2>, <v1,v3> and <v1,v4> are deleted. In the resulting network (figure 6.28(b)), v2, v3 and v4 have no predecessor.

Figure 6.28 Action of Algorithm of Figure 6.27 on an AOV Network

Any of these can be the next vertex in the topological order. Assume that v4 is picked. Deletion of v 4 and the edges (v4, v6) and(v4,v5) results in the network of figure 6.28(c). Either v2 or v3 may next be picked. Figure 6.28 shows the progress of the algorithm on the network. In order to obtain a complete algorithm that can be easily translated into a computer program, it is necessary to specify the data representation for the AOV-network. The choice of a data representation, as always, depends on the functions one wishes to perform. In this problem, the functions are: (i) decide whether a vertex has any predecessors (line 2), and (ii) delete a vertex together with all its incident edges. (i) is efficiently done if for each vertex a count of the number of its immediate predecessors is kept. (ii) is easily implemented if the network is represented by its adjacency lists. Then the deletion of all edges leading out of a vertex v can be carried out by decreasing the predecessor count of all vertices on its adjacency list. Whenever the count of a vertex drops to zero, that vertex can be placed onto a list of vertices with a zero count. Then the selection in line 3 just requires removal of a vertex from this list. Filling in these details into the algorithm of figure 6.27, we obtain the SPARKS program TOPOLOGICAL__ORDER.

The algorithm assumes that the network is represented by adjacency lists. The headnodes of these lists contain two fields: COUNT and LINK. The COUNT field contains the in-degree of that vertex and LINK is a pointer to the first node on the adjacency list. Each list node has 2 fields: VERTEX and LINK. COUNT fields can be easily set up at the time of input. When edge <i,j> is input, the count of vertex j is incremented by 1. The list of vertices with zero count is maintained as a stack. A queue could have been used but a stack is slightly simpler. The stack is linked through the COUNT field of the headnodes since this field is of no use after the COUNT has become zero. Figure 6.29(a) shows the input to the algorithm in the case of the network of figure 6.28(a).

Figure 6.29 Input for Algorithm TOPOLOGICAL__ORDER

procedure TOPOLOGICAL__ORDER (COUNT, VERTEX, LINK, n)

//the n vertices of an AOV-network are listed in topological order.

The network is represented as a set of adjacency lists with

COUNT (i) = the in-degree of vertex i//

1   top    0     //initialize stack//

2   for i    1 to n do      //create a linked stack of vertices with

no predecessors//

3     if COUNT(i) = 0 then [COUNT(i)    top; top    i]

4   end

5   for i    1 to n do   //print the vertices in topological order//

6     if top = 0 then [print ('network has a cycle'); stop]

7     j    top; top    COUNT (top); print (j)  //unstack a vertex//

8     ptr    LINK (j)

9     while ptr    0 do

//decrease the count of successor vertices of j//

10      k    VERTEX(ptr)          //k is a successor of j//

11      COUNT(k)    COUNT(k) - 1              //decrease count//

12      if COUNT(k) = 0   //add vertex k to stack//

13         then [COUNT(k)    top; top    k]

14      ptr    LINK(ptr)

15    end

16  end

17 end TOPOLOGICAL_ORDER

As a result of a judicious choice of data structures the algorithm is very efficient. For a network with n vertices and e edges, the loop of lines 2-4 takes O(n) time; Lines 6-8 take O(n) time over the entire algorithm; the while loop takes time O(di) for each vertex i, where di is the out-degree of vertex i. Since this loop is encountered once for each vertex output, the total time for this part of the algorithm is   [image]. Hence, the asymptotic computing time of the algorithm is O(e + n). It is linear in the size of the problem!

Critical Paths

An activity network closely related to the AOV-network is the activity on edge or AOE network. The tasks to be performed on a project are represented by directed edges. Vertices in the network represent events. Events signal the completion of certain activities. Activities represented by edges leaving a vertex cannot be started until the event at that vertex has occurred. An event occurs only when all activities entering it have been completed. Figure 6.30(a) is an AOE network for a hypothetical project with 11 tasks or activities a1, ...,a11. There are 9 events v1, v2, ..., v9. The events vl and v9 may be interpreted as ''start project'' and ''finish project'' respectively. Figure 6.30(b) gives interpretations for some of the 9 events. The number associated with each activity is the time needed to perform that activity. Thus, activity a1 requires 6 days while a11 requires 4 days. Usually, these times are only estimates. Activities a1, a2 and a 3 may be carried out concurrently after the start of the project. a4, a5 and a6 cannot be started until events v2, v3 and v4, respectively, occur. a7 and a8 can be carried out concurrently after the occurrence of event v5 (i.e., after a4 and a5 have been completed). In case additional ordering constraints are to be put on the activities, dummy activities whose time is zero may be introduced. Thus, if we desire that activities a7 and a8 not start until both events v5 and v6 have occurred, a dummy activity a12 represented by an edge <v6, v5> may be introduced. Activity networks of the AOE type have proved very useful in the performance evaluation of several types of projects. This evaluation includes determining such facts about the project as: what is the least amount of time in which the project may be completed (assuming there are no cycles in the network); which activities should be speeded up in order to reduce completion time; etc. Several sophisticated techniques such as PERT (performance evaluation and review technique), CPM (critical path method), RAMPS (resource allocation and multi-project scheduling) have been developed to evaluate network models of projects. CPM was originally developed in connection with maintenance and construction projects. Figure 6.31 shows a network used by the Perinia Corporation of Boston in 1961 to model the construction of a floor in a multistory building. PERT was originally designed for use in the development of the Polaris Missile system.

Figure 6.30

(a) AOE Network. Activity Graph of a Hypothetical Project.

event           interpretation

-----           --------------

 v1    start of project

 v2    completion of activity a1

 v5    completion of activities a4 and a5

 v8    completion of activities a8 and a9

 v9    completion of project

(b) Interpretation for Some of the Events in the Activity Graph of Figure 6.30(a).

Since the activities in an AOE network can be carried out in parallel the minimum time to complete the project is the length of the longest path from the start vertex to the finish vertex (the length of a path is the sum of the times of activities on this path). A path of longest length is a critical path. The path v1, v2, v5, v7, v9 is a critical path in the network of figure 6.30(a). The length of this critical path is 18. A network may have more than one critical path (the path v1, v2, v5, v8, v9 is also critical). The earliest time an event vi can occur is the length of the longest path from the start vertex v1 to the vertex vi. The earliest time event v5 can occur is 7. The earlist time an event can occur determines the earliest start time for all activities represented by edges leaving that vertex. Denote this time by e(i) for activity ai. For example e(7) = e(8) = 7. For every activity ai we may also define the latest time, l(i), an activity may start without increasing the project duration (i.e., length of the longest path from start to finish). In figure 6.30(a) we have e(6) = 5 and l(6) = 8, e(8) = 7 and l(8) = 7. All activities for which e(i) = l(i) are called critical activities. The difference l(i) - e(i) is a measure of the criticality of an activity. It gives the time by which an activity may be delayed or slowed without increasing the total time needed to finish the project. If activity a6 is slowed down to take 2 extra days, this will not affect the project finish time. Clearly, all activities on a critical path are critical and speeding noncritical activities will not reduce the project duration.

The purpose of critical path analysis is to identify critical activities so that resources may be concentrated on these activities in an attempt to reduce project finish time. Speeding a critical activity will not result in a reduced project length unless that activity is on all critical paths. In figure 6.30(a) the activity a11 is critical but speeding it up so that it takes only three days instead of four does not reduce the finish time to seventeen days. This is so because there is another critical path v1,v2,v5,v7,v9 that does not contain this activity. The activities a1 and a4 are on all critical paths. Speeding a1 by two days reduces the critical path length to sixteen days. Critical path methods have proved very valuable in evaluating project performance and identifying bottlenecks. In fact, it is estimated that without such analysis the Polaris missile project would have taken seven instead of the five years it actually took.

Figure 6.31 AOE network for the construction of a typical floor in a multistory building [Redrawn from Engineering News-Record (McGraw-Hill Book Company, Inc., January 26, 1961).]

Critical path analysis can also be carried out with AOV-networks. The length of a path would now be the sum of the activity times of the vertices on that path. For each activity or vertex, we could analogously define the quantities e(i) and l(i). Since the activity times are only estimates, it is necessary to re-evaluate the project during several stages of its completion as more accurate estimates of activity times become available. These changes in activity times could make previously non-critical activities critical and vice versa. Before ending our discussion on activity networks, let us design an algorithm to evaluate e(i) and l(i) for all activities in an AOE-network. Once these quantities are known, then the critical activities may be easily identified. Deleting all noncritical activities from the AOE network, all critical paths may be found by just generating all paths from the start to finish vertex (all such paths will include only critical activities and so must be critical, and since no noncritical activity can be on a critical path, the network with noncritical activities removed contains all critical paths present in the original network).

In obtaining the e(i) and l(i) for the activities of an AOE network, it is easier to first obtain the earliest event occurrence time, ee(j), and latest event occurrence time, le(j), for all events, j, in the network. Then if activity ai is represented by edge <k,l>, we can compute e(i) and l(i) from the formulas:

e(i) = ee(k)

and

l(i) = le(l) - duration of activity ai

(6.1)

The times ee(j) and le(j) are computed in two stages: a forward stage and a backward stage. During the forward stage we start with ee(1) = 0 and compute the remaining early start times, using the formula

(6.2)

where P(j) is the set of all vertices adjacent to vertex j. In case this computation is carried out in topological order, the early start times of all predecessors of j would have been computed prior to the computation of ee(j). The algorithm to do this is obtained easily from algorithm TOPOLOGICAL_ORDER by inserting the step

ee(k)   [image]  max {ee(k), ee(j) + DUR (ptr)}

between lines 11 and 12. It is assumed that the array ee is initialized to zero and that DUR is another field in the adjacency list nodes which contains the activity duration. This modification results in the evaluation of equation (6.2) in parallel with the generation of a topological order. ee(j) is updated each time the ee(i) of one of its predecessors is known (i.e., when i is ready for output). The step print (j) of line 7 may be omitted. To illustrate the working of the modified TOPOLOGICAL__ORDER algorithm let us try it out on the network of figure 6.30(a). The adjacency lists for the network are shown in figure 6.32(a). The order of nodes on these lists determines the order in which vertices will be considered by the algorithm. At the outset the early start time for all vertices is 0, and the start vertex is the only one in the stack. When the adjacency list for this vertex is processed, the early start time of all vertices adjacent from v1 is updated. Since vertices 2, 3 and 4 are now in the stack, all their predecessors have been processed and equation (6.2) evaluated for these three vertices. ee(6) is the next one determined. When vertex v6 is being processed, ee(8) is updated to 11. This, however, is not the true value for ee(8) since equation (6.2) has not been evaluated over all predecessors of v8 (v5 has not yet been considered). This does not matter since v8 cannot get stacked until all its predecessors have been processed. ee(5) is next updated to 5 and finally to 7. At this point ee(5) has been determined as all the predecessors of v5 have been examined. The values of ee(7) and ee(8) are next obtained. ee(9) is ultimately determined to be 18, the length of a critical path. One may readily verify that when a vertex is put into the stack its early time has been correctly computed. The insertion of the new statement does not change the asymptotic computing time; it remains O(e + n).

(a) Adjacency Lists for Figure 6.30(a)

     ee   (1)  (2)  (3)  (4)  (5)  (6)  (7)  (8)  (9)  stack

initial    0    0    0    0    0    0    0    0    0     1

output v1  0    6    4    5    0    0    0    0    0     4

                                                         3

                                                         2

output v4  0    6    4    5    0    7    0    0    0     6

                                                         3

                                                         2

output v6  0    6    4    5    0    7    0   11    0     3

                                                         2

output v3  0    6    4    5    5    7    0   11    0     2

output v2  0    6    4    5    7    7    0   11    0     5

output v5  0    6    4    5    7    7   16   14    0     8

                                                         7

output v8  0    6    4    5    7    7   16   14   16     7

output v7  0    6    4    5    7    7   16   14   18     9

output v9

(b) Computation of ee

Figure 6.32 Action of Modified Topological Order

In the backward stage the values of le(i) are computed using a procedure analogous to that used in the forward stage. We start with le(n) = ee(n) and use the equation

(6.3)

where S(j) is the set of vertices adjacent from vertex j. The initial values for le(i) may be set to ee(n). Basically, equation (6.3) says that if <j,i> is an activity and the latest start time for event i is le(i), then event j must occur no later than le(i) - duration of <j,i>. Before le(j) can be computed for some event j, the latest event time for all successor events (i.e., events adjacent from j) must be computed. These times can be obtained in a manner identical to the computation of the early times by using inverse adjacency lists and inserting the step le(k)   [image]  min {le(k), le(j) - DUR(ptr)} at the same place as before in algorithm TOPOLOGICAL_ORDER. The COUNT field of a headnode will initially be the out-degree of the vertex. Figure 6.33 describes the process on the network of figure 6.30(a). In case the forward stage has already been carried out and a topological ordering of the vertices obtained, then the values of le(i) can be computed directly, using equation (6.3), by performing the computations in the reverse topological order. The topological order generated in figure 6.32(b) is v1, v4, v6, v3, v2, v5, v8, v7, v9,. We may compute the values of le(i) in the order 9,7,8,5,2,3,6,4,1 as all successors of an event precede that event in this order. In practice, one would usually compute both ee and le. The procedure would then be to compute ee first using algorithm TOPOLOGICAL_ORDER modified as discussed for the forward stage and to then compute le directly from equation (6.3) in reverse topological order.

Using the values of ee (figure 6.32) and of le (figure 6.33) and equation 6.1) we may compute the early and late times e(i) and l(i) and the degree of criticality of each task. Figure 6.34 gives the values. The critical activities are a1,a4,a7,a8,a10 and a11. Deleting all noncritical activities from the network we get the directed graph of figure 6.35. All paths from v1 to v9 in this graph are critical paths and there are no critical paths in the orginal network that are not paths in the graph of figure 6.35 .

As a final remark on activity networks we note that the algorithm TOPOLOGICAL-ORDER detects only directed cycles in the network. There may be other flaws, such as vertices not reachable from the start vertex (figure 6.36). When a critical path analysis is carried out on such networks, there will be several vertices with ee(i) = 0. Since all activity times are assumed > 0 only the start vertex can have ee(i) = 0. Hence, critical path analysis can also be used to detect this kind of fault in project planning.

(a) Inverted Adjacency Lists for AOE-Network of Figure 6.30(a)

     le   (1)  (2)  (3)  (4)  (5)  (6)  (7)  (8)  (9)  stack

initial    18   18   18   18   18   18   18   18   18    9

output v9  18   18   18   18   18   18   16   14   18    8

                                                         7

output v8  18   18   18   18    7   10   16   14   18    6

                                                         7

output v6  18   18   18   18    7   10   16   14   18    4

                                                         7

output v4   3   18   18    8    7   10   16   14    18   7

output v7   3   18   18    8    7   10   16   14    18   5

output v5   3    6    6    8    7   10   16   14   18    3

                                                         2

output v3   2    6    6    8    7   10   16   14   18    2

output v2   0    6    6    8    7   10   16   14   18    1

(b) Computation of TOPOLOGICAL-ORDER Modified to Compute Latest Event Times.

le(9) = ee(9) = 18

le(7) = min {le(9) - 2} = 16

le(8) = min {le(9) - 4} = 14

le(5) = min {le(7) - 9, le(8)- 7} = 7

le(2) = min {le(5) - 1} = 6

le(3) = min {le(5) - 1} = 6

le(6) = min {le(8) - 4} = 10

le(4) = min {le(6) - 2} = 8

le(l) = min {le(2) - 6, le(3) - 4, le(4) - 5} = 0

c) Computation of le Directly from Equation (6.3) Using a Reverse Topological Order.

Figure 6.33

activity  e   l  l - e

  a1      0   0    0

  a2      0   2    2

  a3      0   3    3

  a4      6   6    0

  a5      4   6    2

  a6      5   8    3

  a7      7   7    0

  a8      7   7    0

  a9      7  10    3

  a10    16  16    0

  a11    14  14     0

Figure 6.34 Early, Late and Criticality Values

Figure 6.35 Graph Obtained After Deleting All Noncritical Activities.

Figure 6.36 AOE-Network with Some Non-Reachable Activities.

6.5 ENUMERATING ALL PATHS

In section 6.3 we looked at the problem of finding a shortest path between two vertices. In this section we will be concerned with listing all possible simple paths between two vertices in order of nondecreasing path length. Such a listing could be of use, for example, in a situation where we are interested in obtaining the shortest path that satisfies some complex set of constraints. One possible solution to such a problem would be generate in nondecreasing order of path length all paths between the two vertices. Each path generated could be tested against the other constraints and the first path satisfying all these constraints would be the path we were looking for.

Let G = (V,E) be a digraph with n vertices. Let v1 be the source vertex and vn the destination vertex. We shall assume that every edge has a positive cost. Let p1 = r(0), r(1), ..., r(k) be a shortest path from v1 to vn. I.e., p1 starts at vr(0) = v1, goes to vr(1) and then to vr(2), ...,vn = vr(k). If P is the set of all simple v1 to vn paths in G, then it is easy to see that every path in P - {p1} differs from p1 in exactly one of the following k ways:

(1):     It contains the edges (r(1), r(2)), ..., (r(k - 1), r(k))but not (r(0), r(1))

(2):     It contains the edges (r(2), r(3)), ..., (r(k - 1), r(k)) but not (r(1), r(2))

     :

     :

(k):     It does not contain the edge (r(k - 1), r(k)).

More compactly, for every path p in P - {p1} there is exactly one j, 1   [image]  j   [image]  k such that p contains the edges

(r(j), r(j + 1)), ...,(r(k - 1), r(k)) but not (r(j - 1), r(j)).

The set of paths P - {p1} may be partitioned into k disjoint sets P(1), ..., P(k) with set P(j) containing all paths in P - {P1} satisfying condition j above, 1   [image]  j   [image]  k.

Let p(j) be a shortest path in P(j) and let q be the shortest path from v1 to vr(j) in the digraph obtained by deleting from G the edges (r(j - 1),r(j)),(r(j),r(j + 1)), ...,(r(k - 1),r(k)). Then one readily obtains p(j) = q, r(j), r(j + 1), ...,r(k) = n. Let p(l) have the minimum length among p(1), ...,p(k). Then, p(l) also has the least length amongst all paths in P - {p1} and hence must be a second shortest path. The set P(l) - {p(l)} may now be partitioned into disjoint subsets using a criterion identical to that used to partition P - {p1}. If p(l) has k' edges, then this partitioning results in k' disjoint subsets. We next determine the shortest paths in each of these k' subsets. Consider the set Q which is the union of these k' shortest paths and the paths p(1), ...,p(l - 1), ...,p(k). The path with minimum length in Q is a third shortest path p3. The corresponding set may be further partitioned. In this way we may successively generate the v1 to vn paths in nondecreasing order of path length.

At this point, an example would be instructive. Figure 6.38 shows the generation of the v1 to v6 paths of the graph of figure 6.37.

A very informal version of the algorithm appears as the procedure M_SHORTEST.

procedure M_SHORTEST (M)

//Given a digraph G with positive edge weights this procedure outputs

the M shortest paths from v1 to vn. Q contains tuples of the

form (p,C) where p is a shortest path in G satisfying constraints

C. These constraints either require certain edges to be included

or excluded from the path//

1    Q    {(shortest v1 to vn path,   )}

2    for i    1 to M do     //generate M shortest paths//

3      let (p,C) be the tuple in Q such that path p is of minimal length.

//p is the i'th shortest path//

4      print path p; delete path p from Q

5      determine the shortest paths in G under the constraints C and

the additional constraints imposed by the partitioning described

in the text

6      add these shortest paths together with their constraints to Q

7   end

8 end M_SHORTEST

Since the number of v1 to vn paths in a digraph with n vertices ranges from 0 to (n - 1)!, a worst case analysis of the computing time of M_SHORTEST is not very meaningful. Instead, we shall determine the time taken to generate the first m shortest paths. Line 5 of the for loop may require determining n - 1 shortest paths in a n vertex graph. Using a modified version of algorithm SHORTEST_PATH this would take O(n3) for each iteration of the for loop. The total contribution of lkne 5 is therefore O(mn3). In the next chapter, when we study heap sort, we shall see that it is possible to maintain Q as a heap with the result that the total contribution of lines 3 and 6 is less than O(mn3). The total time to generate the m shortest paths is, therefore, O(mn3). The space needed is determined by the number of tuples in Q. Since at most n - 1 shortest paths are generated at each iteration of line 5, at most O(mn) tuples get onto Q. Each path has at most O(n) edges and the additional constraints take less space than this. Hence, the total space requirements are O(mn2)

Figure 6.37 A Graph

Shortest

Path     Cost Included Edges  Excluded Edges     New Path

------   ---- --------------  --------------     --------

v1v3v5v6      8   none            none

              none            (v5v6)              v1v3v4v6 = 9

              (v5v6)           (v3v5)              v1v2v5v6 = 12

              (v3v5)(v5v6)      (v1v3)              v1v2v3v5v6 = 14

v1v3v4v6      9   none            (v5v6)

              none            (v4v6)(v5v6)

              (v4v6)           (v3v4)(v5v6)         v1v2v4v6 = 13

              (v3v4)(v4v6)      (v1v3)(v5v6)         v1v2v3v4v6 = 15

v1v2v5v6     12  (v5v6)            (v3v5)

              (v5v6)            (v2v5)(v3v5)        v1v3v4v5v6 = 16

              (v2v5)(v5v6)       (v1v2)(v3v5)

v1v2v4v6     13  (v4v6)            (v3v4)(v5v6)

              (v4v6)            (v2v4)(v3v4)(v5v6)

              (v2v4)(v4v6) "     (v1v2)(v3v4)(v5v6)

v1v2v3v5v6  14   (v3v5)(v5v6)       (v1v3)

              (v3v5)(v5v6)       (v2v3)(v1v3)

              (v2v3)(v3v5)(v5v6)  (v1v2)(v5v6)

v1v2v3v4v6  15   (v3v4)(v4v6)       (v1v3)(v5v6)

              (v3v4)(v4v6)       (v2v3)(v1v3)(v5v6)

              (v2v3)(v3v5)(v5v6)  (v1v2)(v1v3)(v5v6)

v1v3v4v5v6  16  (v5v6)             (v2v5)(v3v5)

              (v5v6)             (v4v5)(v2v5)(v3v5)

              (v4v5)(v5v6)        (v3v4)(v2v5)(v3v5)   v1v2v4v5v6 = 20

              (v3v4)(v4v5)(v5v6)   (v1v3)0(v2v5)(v3v5)  v1v2v3v4v5v6 = 22

Figure 6.38 Action of M_SHORTEST

REFERENCES

Eulers original paper on the Koenigsberg bridge problem makes interesting reading. This paper has been reprinted in:

"Leonhard Euler and the Koenigsberg Bridges," Scientific American, vol. 189, no. 1, July 1953, pp. 66-70.

Good general references for this chapter are:

Graph Theory by F. Harary, Addison-Wesley, Reading, Massachusetts, 1972.

The theory of graphs and its applications by C. Berge, John Wiley, 1962.

Further algorithms on graphs may be found in:

The design and analysis of computer algorithms by A. Aho, J. Hopcroft and J. Ullman, Addison-Wesley, Reading, Massachusetts, 1974.

Graph theory with applications to engineering and computer science by N. Deo, Prentice-Hall, Englewood Cliffs, New Jersey, 1974.

Combinatorial Optimization by E. Lawler, Holt, Reinhart and Winston, 1976.

"Depth-first search and linear graph algorithms" by R. Tarjan, SIAM Journal on Computing, vol. 1, no. 2, 1972, pp. 146-159.

Flows in Networks by L. Ford and D. Fulkerson, Princeton University Press, 1962.

Integer Programming and Network Flows by T. C. Hu, Addison-Wesley, Reading, Massachusetts, 1970.

For more on activity networks and critical path analysis see:

Project Management with CPM and PERT by Moder and C. Phillips, Van Nostran Reinhold Co., 1970.

EXERCISES

1.     Does the multigraph below have an Eulerian walk? If so, find one.

2.     For the digraph at the top of page 329 obtain:

i) the in degree and out degree of each vertex;

ii) its adjacency matrix;

iii) its adjacency list representation;

iv) its adjacency multilist representation;

v) its strongly connected components

3.     Devise a suitable representation for graphs so they can be stored on punched cards. Write an algorithm which reads in such a graph and creates its adjacency matrix. Write another algorithm which creates the adjacency lists from those input cards.

4.     Draw the complete undirected graphs on one, two, three, four and five vertices. Prove that the number of edges in a n vertex complete graph is n(n - 1)/2.

5.     Is the directed graph below strongly connected? List all the simple paths.

6.     Show how the graph above would look if represented by its adjacency matrix, adjacency lists, adjacency multilist.

7.     For an undirected graph G with n vertices and e edges show that   [image]  where di = degree of vertex i.

8.     (a)     Let G be a connected undirected graph on n vertices. Show that G must have at least n - 1 edges      and that all connected undirected graphs with n - 1 edges are trees.

(b)     What is the minimum number of edges in a strongly connected digraph on n vertices? What shape      do such digraphs have?

9.     For an undirected graph G with n vertices prove that the following are equivalent:

(a)     G is a tree;

(b)     G is connected, but if any edge is removed the resulting graph is not connected;

(c)     For any distinct vertices u   [image]  V(G) and v   [image]  V(G) there is exactly one simple path from u to v;

(d)     G contains no cycles and has n - 1 edges;

(e)     G is connected and has n - 1 edges.

10.     A bipartite graph G = (V,E) is an undirected graph whose vertices can be partitioned into two disjoint sets V1 and V2 = V - V1 with the properties (i) no two vertices in V1 are adjacent in G and (ii) no two vertices in V2 are adjacent in G. The graph G4 of figure 6.5 is bipartite. A possible partitioning of V is: V1 = {1,4.5,7} and V2 = {2,3,6,8}. Write an algorithm to determine whether a graph G is bipartite. In case G is bipartite your algorithm should obtain a partitioning of the vertices into two disjoint sets V1 and V2 satisfying properties (i) and (ii) above. Show that if G is represented by its adjacency lists, then this algorithm can be made to work in time O(n + e) where n =   [image] V  [image]  and e =   [image] E  [image] .

11.     Show that every tree is a bipartite graph.

12.     Prove that a graph G is bipartite iff it contains no cycles of odd length.

13.     The following algorithm was obtained by Stephen Bernard to find an Eulerian circuit in an undirected graph in case there was such a circuit.

procedure EULER (v)

1   path    {  }

2   for all vertices w adjacent to v and edge (v,w) not yet used do

3      mark edge (v,w) as used

4      path    {(v,w)}    EULER(w)    path

5   end

6   return (path)

7 end EULER

a)     Show that if G is represented by its adjacency multilists and path by a linked list, then algorithm EULER works in time O(n + e).

b)     Prove by induction on the number of edges in G that the above algorithm does obtain an Euler circuit for all graphs G having such a circuit. The initial call to Euler can be made with any vertex v.

c)     At termination, what has to be done to determine whether or not G has an Euler circuit?

14.     Apply depth first and breadth first search to the complete graph on four vertices. List the vertices in the order they would be visited.

15.     Show how to modify algorithm DFS as it is used in COMP to produce a list of all newly visited vertices.

16.     Prove that when algorithm DFS is applied to a connected graph the edges of T form a tree.

17.     Prove that when algorithm BFS is applied to a connected graph the edges of T form a tree.

18.     Show that A+ = A* x A where matrix multiplication of the two matrices is defined as   [image]. V is the logical or operation and ^ is the logical and operation.

19.     Obtain the matrices A+ and A* for the digraph of exercise 5.

20.     Another way to represent a graph is by its incidence matrix, INC. There is one row for each vertex and one column for each edge. Then INC (i,j) = 1 if edge j is incident to vertex i. The incidence matrix for the graph of figure 6.13(a) is:

The edges of figure 6.13(a) have been numbered from left to right, top to bottom. Rewrite algorithm DFS so it works on a graph represented by its incidence matrix.

21.     If ADJ is the adjacency matrix of a graph G = (V,E) and INC is the incidence matrix, under what conditions will ADJ = INC x INCT - I where INCT is the transpose of matrix INC. Matrix multiplication is defined as in exercise 18. I is the identity matrix.

22.     Show that if T is a spanning tree for the undirected graph G, then the addition of an edge e, e   [image]  E(T) and e   [image]  E(G), to T creates a unique cycle .

23.     By considering the complete graph with n vertices, show that the number of spanning trees is at least 2n-1 - 1

24.     The radius of a tree is the maximum distance from the root to a leaf. Given a connected, undirected graph write an algorithm for finding a spanning tree of minimum radius. (Hint: use breadth first search.) Prove that your algorithm is correct.

25.     The diameter of a tree is the maximum distance between any two vertices. Given a connected, undirected graph write all algorithm for finding a spanning tree of minimum diameter. Prove the correctness of your algorithm.

26.     Write out Kruskal's minimum spanning tree algorithm (figure 6.18) as a complete SPARKS program. You may use as subroutines the algorithms UNION and FIND of Chapter 5. Use algorithm SORT to sort the edges into nondecreasing order by weight.

27.     Using the idea of algorithm SHORTEST_PATH, give an algorithm for finding a minimum spanning tree whose worst case time is O(n2).

28.     Use algorithm SHORTEST_PATH to obtain in nondecreasing order the lengths of the shortest paths from vertex 1 to all remaining vertices in the digraph:

29.     Rewrite algorithm SHORTEST_PATH under the following assumptions:

(i)     G is represented by its adjacency lists. The head nodes are HEAD(1), ...HEAD(n) and each list node has three fields: VERTEX, COST, and LINK. COST is the length of the corresponding edge and n the number of vertices in G.

(ii)     Instead of representing S, the set of vertices to which the shortest paths have already been found, the set T = V(G) - S is represented using a linked list.

What can you say about the computing time of your new algorithm relative to that of SHORTEST_PATH?

30.     Modify algorithm SHORTEST_PATH so that it obtains the shortest paths in addition to the lengths of these paths. What is the computing time of your algorithm?

31.     Using the directed graph below, explain why SHORTEST_PATH will not work properly. What is the shortest path between vertices v1, and v7?

32.     Modify algorithm ALL_COSTS so that it obtains a shortest path for all pairs of vertices i,j. What is the computing time of your new algorithm?

33.     By considering the complete graph with n vertices show that the maximum number of paths between two vertices is (n - 1)!

34.     Use algorithm ALL_COSTS to obtain the lengths of the shortest paths between all pairs of vertices in the graph of exercise 31. Does ALL_COSTS give the right answers? Why?

35.     Does the following set of precedence relations (<) define a partial order on the elements 1 thru 5? Why?

1 < 2; < 2 <4; 2 < 3, 3 < 4; 3 < 5: 5;< 1

36.     a)     For the AOE network below obtain the early, e( ), and late l( ), start times for each activity. Use the forward-backward approach.

b)     What is the earliest time the project can finish?

c)     Which activities are critical?

d)     Is there any single activity whose speed up would result in a reduction of the project length?

37.     Define a critical AOE-network to be an AOE-network in which all activities are critical. Let G be the undirected graph obtained by removing the directions and weights from the edges of the network.

a)     Show that the project length can be decreased by speeding exactly one activity if there is an edge in G which lies on every path from the start vertex to the finish vertex. Such an edge is called a bridge. Deletion of a bridge from a connected graph disconnects the graph into two connected components.

b)     Write an O(n + e) algorithm using adjacency lists to determine whether the connected graph G has a bridge. In case G has a bridge, your algorithm should output one such bridge.

38.     Write a set of computer programs for manipulating graphs. Such a collection should allow input and input of arbitrary graphs. determining connected components and spanning trees. The capability of attaching weights to the edges should also be provided.

39.     Make sure you can define the following terms.

adjacent(v)       connected(vp, vq)

adjacent-to(v)    connected(G)

adjacent-from(v)  connected component

degree(v)         strongly connected

in-degree(v)      tree

out-degree(v)     network

path              spanning tree

simple path

cycle

subgraph

Go to Chapter 7     Back to Table of Contents


Fundamentals: CHAPTER 7: INTERNAL SORTING

 Return to Table of Contents

CHAPTER 7: INTERNAL SORTING

7.1 SEARCHING

A file is a collection of records, each record having one or more fields The fields used to distinguish among the records are known as keys. Since the same file may be used for several different applications, the key fields for record identification will depend on the particular application. For instance, we may regard a telephone directory as a file, each record having three fields: name, address, and phone number. The key is usually the person's name. However, one may wish to locate the record corresponding to a given number, in which case the phone number field would be the key. In yet another application one may desire the phone number at a particular address, so this field too could be the key. Once we have a collection of records there are at least two ways in which to store them: sequentially or non-sequentially. For the time being let us assume we have a sequential file F and we wish to retrieve a record with a certain key value K. If F has n records with Ki the key value for record Ri, then one may carry out the retrieval by examining the key values Kn,Kn -1, ...,K1 in that order, until the correct record is located. Such a search is known as sequential search since the records are examined sequentially.

procedure SEQSRCH(F,n,i,K)

//Search a file F with key values K1, ...,Kn for a record Ri such

that Ki = K. If there is no such record, i is set to 0//

K0    K; i    n

while Ki    K do

i    i - 1

end

end SEQSRCH

Note that the introduction of the dummy record R0 with key K0 = K simplifies the search by eliminating the need for an end of file test (i < 1) in the while loop. While this might appear to be a minor improvement, it actually reduces the running time by 50% for large n (see table 7.1). If no record in the file has key value K, then i = 0, and the above algorithm requires n + 1 comparisons. The number of key comparisons made in case of a successful search, depends on the position of the key in the file. If all keys are distinct and key Ki is being searched for, then n -i + 1 key comparisons are made. The average number of comparisons for a successful search is, therefore, [image] 1 [image] i [image] n (n - i + 1) / n = (n + 1)/2. For large n this many comparisons is very inefficient. However, we all know that it is possible to do much better when looking up phone numbers. What enables us to make an efficient search? The fact that the entries in the file (i.e., the telephone directory) are in lexicographic order (on the name key) is what enables one to look up a number while examining only a very few entries in the file. So, if the file is ordered one should be able to search for specific records quickly.

One of the better known methods for searching an ordered sequential file is called binary search. In this method, the search begins by examining the record in the middle of the file rather than the one at one of the ends as in sequential search. Let us assume that the file being searched is ordered by nondecreasing values of the key (i.e., in alphabetical order for strings) Then, based on the results of the comparison with the middle key, Km, one can draw one of the following conclusions:

(i) if K < Km then if the record being searched for is in the file, it must be in the lower numbered half of the file;

(ii) if K = Km then the middle record is the one being searched for;

(iii) if K > Km then if the record being searched for is in the file, it must be in the higher numbered half of the file.

Consequently, after each comparison either the search terminates successfully or the size of the file remaining to be searched is about one half of the original size (note that in the case of sequential search, after each comparison the size of the file remaining to be searched decreases by only 1). So after k key comparisons the file remaining to be examined is of size at most [image] n/2k [image] (n is the number of records). Hence, in the worst case, this method requires O (log n) key comparisons to search a file. Algorithm BINSRCH implements the scheme just outlined.

procedure BINSRCH(F,n,i,K)

//Search an ordered sequential file F with records R1, ...,Rn and

the keys K1    K2              Kn for a record Ri such that Ki = K;

i = 0 if there is no such record else Ki = K. Throughout the

algorithm, l is the smallest index such that Kl may be K and u

the largest index such that Ku may be K//

l    1; u    n

while l    u do

m      (l+ u)/2        //compute index of middle record//

case

:K > Km: l    m + 1       //look in upper half//

:K = Km: i    m; return

:K < Km: u    m - 1       //look in lower half//

end

end

i    0    //no record with key K//

end BINSRCH

In the binary search method described above, it is always the key in the middle of the subfile currently being examined that is used for comparison. This splitting process can be described by drawing a binary decision tree in which the value of a node is the index of the key being tested. Suppose there are 31 records, then the first key tested is K16 since [image] (1 + 31) / 2 [image] = 16. If K is less than K16, then K8 is tested next [image] (1 + 15) / 2 [image] = 8; or if K is greater than K16, then K24 is tested. The binary tree describing this process is

A path from the root to any node in the tree represents a sequence of comparisons made by BINSRCH to either find K or determine that it is not present. From the depth of this tree one can easily see that the algorithm makes no more than O(log2n)comparisons.

It is possible to consider other criteria than equal splitting for dividing the remainig file. An alternate method is Fibonacci search, which splits the subfile according to the Fibonacci sequence,

0, 1, 1, 2, 3, 5, 8, 13, 21, 34, ...

which is defined as F0 = 0, F1 = 1 and

Fi = Fi-1 + Fi-2, i [image] 2.

An advantage of Fibonacci search is that it involves only addition and subtraction rather than the division in BINSRCH. So its average performance is better than that of binary search on computers for which division takes sufficiently more time than addition/subtraction.

Suppose we begin by assuming that the number of records is one less than some Fibonacci number, n = Fk - 1. Then, the first comparison of key K is made with KFk-1 with the following outcomes:

(i) K < KFk-1 in which case the subfile from 1 to Fk-1 - 1 is searched and this file has one less than a Fibonacci number of records;

(ii) K = KFk-1 in which case the search terminates successfully;

(iii) K > KFk-1 in which case the subfile from Fk-1 + 1 to Fk - 1 is searched and the size of this file is Fk - 1 - (Fk-1 + 1) + 1 = Fk - Fk-1 = Fk-2 - 1.

Again it is helpful to think of this process as a binary decision tree; the resulting tree for n = 33 is given on page 340.

This tree is an example of a Fibonacci tree. Such a tree has n = Fk -1 nodes, and its left and right subtrees are also Fibonacci trees with Fk-1 - 1 and Fk-2 - 1 nodes respectively. The values in the nodes show how the file will be broken up during the searching process. Note how the values of the children differ from the parent by the same amount. Moreover, this difference is a Fibonacci number. If we look at a grandparent, parent and two children where the parent is a left child, then if the difference from grandparent to present is Fj, the next difference is Fj-1. If instead the parent is a right child then the next difference is Fj-2.

The following algorithm implements this Fibonacci splitting idea.

procedure FlBSRCH(G,n,i,K)

//search a sequential file G with keys ordered in nondecreasing order,

for a record Ri with key Ki = K. Assume that Fk + m = n + 1,

m    0 and Fk + 1 > n + 1. n is the number of records in G. Fk

and Fk+1 are consecutive Fibonacci numbers. If K is not present,

i is set to zero.//

i    Fk-1; p    Fk-2; q    Fk-3

if K > Ki then [//set i so that size of the right subfile is Fk-2//

i    i + m]

while i    0 do

case

:K < Ki: if q = 0 then i    0

else [i    i; - q; t    p; p    q; q    t -q]

:K = Ki: return

:K > Ki: if p = 1 then i    0

else [i    i + q; p    p - q;q    q - p]

end

end

end FIBSRCH

Getting back to our example of the telephone directory, we notice that neither of the two ordered search methods suggested above corresponds to the one actually employed by humans in searching the directory. If we are looking for a name beginning with W, we start the search towards the end of the directory rather than at the middle. A search method based on this interpolation search would then begin by comparing key Ki with [image] are the values of the smallest and largest keys in the file). The behavior of such an algorithm will clearly depend on the distribution of the keys in the file.

The four search procedures sequential, binary, Fibonacci and interpolation search were programmed in Fortran and their performance evaluated on a Cyber 74 computer. The results of this evaluation are presented in table 7.1. Since the input for the last three algorithms is a sorted file, algorithm SEQSRCH was modified to take advantage of this. This was achieved by changing the conditional in the while statement from Ki [image] K to Ki > K and introducing an additional test after the end of the while to determine whether Ki = K. The inferior sequential method referred to in the table differs from the SEQSRCH just described in that the line Ko [image] K is not included and a test for i < 1 is made in the while loop. As the table indicates, inferior sequential search takes almost twice as much time as normal sequential search. For average behavior, binary search is better than sequential search for n > 15, while for worst case behavior binary search is better than a sequential search for n > 4.

Fibonacci search with n = 33

As can be seen, Fibonacci search always performed worse than binary search. The performance of interpolation search is highly dependent on key value distribution. Its performance is best on uniform distributions. Its average behavior on uniform distributions was better than that for binary search for n [image] 500.

We have seen that as far as the searching problem is concerned, something is to be gained by maintaining the file in an ordered manner if the file is to be searched repeatedly. Let us now look at another example where the use of ordered files greatly reduces the computational effort. The problem we are now concerned with is that of comparing two files of records containing data which is essentially the same data but has been obtained from two different sources. We are concerned with verifying that the two files contain the same data. Such a problem could arise, for instance, in the case of the U.S. Internal Revenue Service which might receive millions of forms from various employers stating how much they paid their employees and then another set of forms from individual employees stating how much they received. So we have two files of records, and we wish to verify that there is no discrepancy between the information on the files. Since the forms arrive at the IRS in essentially a random order, we may assume a random arrangement of the records in the files. The key here would probably be the social security numbers. Let the records in the two files, [image] and [image] be ( [image]) and ( [image]). The corresponding keys are [image] and [image]. Let us make the following assumptions about the required verification: (i) if corresponding to a key [image] in the employer file there is no record in the employee file a message is to be sent to the employee; (ii) if the reverse is true, then a message is to be sent to the employer; and (iii) if there is a discrepancy between two records with the same key a message to that effect is to be output.

If one proceeded to carry out the verification directly, one would probably end up with an algorithm similar to VERIFY1.

(a) Worst case times

(b) Average times obtained by searching for each key once and averaging

Table 7.1 Worst case and average times for different search methods. All times are in milliseconds. (Table prepared by Randall Istre)

One may readily verify that the worst case asymptotic computing time of the above algorithm is O(mn). On the other hand if we first ordered the two files and then made the comparison it would be possible to carry out the verification task in time O(tsort(n) + tsort(m) + n + m) where tsort(n) is the time needed to sort a file of n records. As we shall see, it is possible to sort n records in O(n log n) time, so the computing time becomes O(max {n log n, m log m}). The algorithm VERIFY2 achieves this.

We have seen two important uses of sorting: (i) as an aid in searching, and (ii) as a means for matching entries in files. Sorting also finds application in the solution of many other more complex problems, e.g. from operations research and job scheduling. In fact, it is estimated that over 25% of all computing time is spent on sorting with some installations spending more than 50% of their computing time sorting files. Consequently, the problem of sorting has great relevance in the study of computing. Unfortunately, no one method is the "best" for all initial orderings of the file being sorted. We shall therefore study several methods, indicating when one is superior to the others.

First let us formally state the problem we are about to consider.

We are given a file of records (R1,R2, ...,Rn). Each record, Ri, has key value Ki. In addition we assume an ordering relation (<) on the key so that for any two key values x and y either x = y or x < y or y < x. The ordering relation (<) is assumed to be transitive, i.e., for any three values x, y and z, x < y and y < z implies x < z. The sorting problem then is that of finding a permutation, [image], such that K [image] (1) [image] K [image] (i+1), 1 [image] i [image] n - 1. The desired ordering is then (R [image] (1),R [image] (2), ...,R [image] (n)).

Note that in the case when the file has several key values that are identical, the permutation, [image], defined above is not unique. We shall distinguish one permutation, [image] s, from all the others that also order the file. Let [image] s be the permutation with the following properties:

(i) K [image] s(i) [image] K [image] s(i+1),1 [image] i [image] n - 1.

(ii) If i < j and Ki = Kj in the input file, then Ri precedes Rj in the sorted file.

A sorting method generating the permutation [image] s will be said to be stable.

To begin with we characterize sorting methods into two broad categories: (i) internal methods, i.e., methods to be used when the file to be sorted is small enough so that the entire sort can be carried out in main memory; and (ii) external methods, i.e., methods to be used on larger files. In this chapter we shall study the following internal sorting methods:

a) Insertion sort

b) Quick sort

c) Merge sort

d) Heap sort

e) Radix sort

External sorting methods will be studied in Chapter 8.

7.2 INSERTION SORT

The basic step in this method is to insert a record R into a sequence of ordered records, R1,R2, ...,Ri, (K1 [image] K2, ..., [image] Ki) in such a way that the resulting sequence of size i + 1 is also ordered. The algorithm below accomplishes this insertion. It assumes the existence of an artificial record Ro with key Ko = - [image] (i.e., all keys are [image] Ko).

procedure INSERT (R,i)

//Insert record R with key K into the ordered sequence Ro, ...,Ri

in such a way that the resulting sequence is also ordered on key

K. We assume that Ro is a record (maybe a dummy) such that

K    Ko//

j    i

while K < Kj do

//move Rj one space up as R is to be inserted left of Rj//

Rj+1    Rj; j    j - 1

end

Rj+1    R

end INSERT

Again, note that the use of Ro enables us to simplify the while loop, avoiding a test for end of file, i.e., j < 1.

Insertion sort is carried out by beginning with the ordered sequence Ro,R1, and then successively inserting the records R2,R3, ... Rn into the sequence. since each insertion leaves the resultant sequence ordered, the file with n records can be ordered making n - 1 insertions. The details are given in algorithm INSORT on the next page.

Analysis of INSERTION SORT

In the worst case algorithm INSERT(R,i) makes i + 1 comparisons before making the insertion. Hence the computing time for the insertion is O(i). INSORT invokes procedure INSERT for i = 1,2, ...,n - 1

procedure INSORT(R,n)

//sort the records R1, ...,Rn in nondecreasing value of the key K.

Assume n > 1//

Ko    -         //Create a dummy record Ro such that Ko < Ki,

1    i    n//

for j    2 to n do

T    Rj

call INSERT(T, j - 1)     //insert records R2 to Rn//

end

end INSORT

resulting in an overall worst case time of [image].

One may also obtain an estimate of the computing time of this method based upon the relative disorder in the input file. We shall say that the record Ri is left out of order (LOO) iff [image]. Clearly, the insertion step has to be carried out only for those records that are LOO. If k is the number of records LOO, then the computing time is O((k + 1)n). The worst case time is still O(n2). One can also show that O(n2) is the average time.

Example 7.1: Assume n = 5 and the input sequence is (5,4,3,2,1) [note the records have only one field which also happens to be the key]. Then, after each insertion we have the following:

-  , 5, 4, 3, 2, 1                   [initial sequence]

-  , 4, 5, 3, 2, 1                          i = 2

-  , 3, 4, 5, 2, 1                          i = 3

-  , 2, 3, 4, 5, 1                          i = 4

-  , 1, 2, 3, 4, 5                          i = 5

Note that this is an example of the worst case behavior.

Example 7.2: n = 5 and the input sequence is (2, 3, 4, 5, 1). After each execution of INSERT we have:

-  , 2, 3, 4, 5, 1                            [initial]

-  , 2, 3, 4, 5, l                              i = 2

-  , 2, 3, 4, 5, 1                              i = 3

-  , 2. 3, 4, 5, 1                              i = 4

-  , 1, 2, 3, 4, 5                              i = 5

In this example only R5 is LOO and the time for each i = 2,3 and 4 is O(1) while for i = 5 it is O(n).

It should be fairly obvious that this method is stable. The fact that the computing time is O(kn) makes this method very desirable in sorting sequences where only a very few records are LOO (i.e., k << n). The simplicity of this scheme makes it about the fastest sorting method for n [image] 20 - 25 elements, depending upon the implementation and machine properties. For variations on this method see exercises 3 and 4.

7.3 QUICKSORT

We now turn our attention to a sorting scheme with a very good average behavior. The quicksort scheme developed by C. A. R. Hoare has the best average behavior among all the sorting methods we shall be studying. In Insertion Sort the key Ki currently controlling the insertion is placed into the right spot with respect to the sorted subfile (R1, ...,Ri - 1). Quicksort differs from insertion sort in that the key Ki controlling the process is placed at the right spot with respect to the whole file. Thus, if key Ki is placed in position s(i), then Kj [image] Ks(i) for j < s(i) and Kj [image] Ks(i) for j > s(i). Hence after this positioning has been made, the original file is partitioned into two subfiles one consisting of records R1, ...,Rs(i)-1 and the other of records Rs(i)+ 1, ...,Rn. Since in the sorted sequence all records in the first subfile may appear to the left of s(i) and all in the second subfile to the right of s(i), these two subfiles may be sorted independently. The method is best stated recursively as below and where INTERCHANGE (x,y) performs t [image] x; x [image] y; y [image] t.

procedure QSORT(m,n)

//sort records Rm, ...,Rn into nondecreasing order on key K. Key

Km is arbitrarily chosen as the control key. Pointers i and j are

used to partition the subfile so that at any time Kl    K, l < i

and Kl    K, l > j. It is assumed that Km    Kn+1//

if m < n

then [i    m; j    n + 1; K    Km

loop

repeat i    i + 1 until Ki    K;

repeat j    j - 1 until Kj    K;

if i < j

then call INTERCHANGE (R(i),R(j))

else exit

forever

call INTERCHANGE (R(m),R(j))

call QSORT (m,j - 1)

call QSORT (j + 1, n)]

end QSORT

Example 7.3: The input file has 10 records with keys (26, 5, 37, 1, 61, 11, 59, 15, 48, 19). The table below gives the status of the file at each call of QSORT. Square brackets are used to demarcate subfiles yet to be sorted.

 R1  R2  R3   R4   R5   R6  R7   R8   R9  Rl0    m  n

[26  5   37    1  61   11   59  15   48   19]   1  10

[11  5   19    1  15]  26  [59  61   48   37]   1   5

[ 1  5]  11  [19  15]  26  [59  61   48   37]   1   2

  1  5   11  [19  15]  26  [59  61   48   37]   4   5

  1  5   11   15  19   26  [59  61   48   37]   7  10

  1  5   11   15  19   26  [48  37]  59  [61]   7   8

  1  5   11   15  19   26   37  48   59  [61]  10  10

  1  5   11   l5  l9   26   37  48   59   6l

Analysis of Quicksort

The worst case behavior of this algorithm is examined in exercise 5 and shown to be O(n2). However, if we are lucky then each time a record is correctly positioned, the subfile to its left will be of the same size as that to its right. This would leave us with the sorting of two subfiles each of size roughly n/2. The time required to position a record in a file of size n is O(n). If T(n) is the time taken to sort a file of n records, then when the file splits roughly into two equal parts each time a record is positioned correctly we have

T(n)    cn + 2T(n/2) , for some constant c

   cn + 2(cn/2 + 2T(n/4))

   2cn + 4T(n/4)

:

   cn log2n + nT(1) = O(n log2 n)

In our presentation of QSORT, the record whose position was being fixed with respect to the subfile currently being sorted was always chosen to be the first record in that subfile. Exercise 6 examines a better choice for this control record. Lemma 7.1 shows that the average computing time for Quicksort is O(n log2 n). Moreover, experimental results show that as far as average computing time is concerned, it is the best of the internal sorting methods we shall be studying.

Unlike Insertion Sort where the only additional space needed was for one record,Quicksort needs stack space to implement the recursion. In case the files split evenly as in the above analysis, the maximum recursion depth would be log n requiring a stack space of O(log n). The worst case occurs when the file is split into a left subfile of size n - 1 and a right subfile of size 0 at each level of recursion. In this case, the depth of recursion becomes n requiring stack space of O(n). The worst case stack space can be reduced by a factor of 4 by realizing that right subfiles of size less than 2 need not be stacked. An asymptotic reduction in stack space can be achieved by sorting smaller subfiles first. In this case the additional stack space is at most O(log n).

Lemma 7.1: Let Tavg(n) be the expected time for QSORT to sort a file with n records. Then there exists a constant k such that Tavg(n) [image] k n 1oge n for n [image] 2.

Proof: In the call to QSORT (1,n), K1 gets placed at position j. This leaves us with the problem of sorting two subfiles of size j - 1 and n - j. The expected time for this is Tavg(j - 1) + Tavg(n - j). The remainder of the algorithm clearly takes at most cn time for some constant c. Since j may take on any of the values 1 to n with equal probability we have:

(7.1)

We may assume Tavg(0) [image] b and Tavg(1) [image] b for some constant b. We shall now show Tavg(n) [image] kn 1oge n for n [image] 2 and k = 2(b + c). The proof is by induction on n.

Induction Base: For n = 2 we have from eq. (7.1):

Tavg(2) [image] 2c + 2b [image] kn loge2

Induction Hypothesis: Assume Tavg(n) [image] kn 1ogen for 1 [image] n < m

Induction Step: From eq. (7.1) and the induction hypothesis we have:

(7.2)

Since j loge j is an increasing function of j eq. (7.2) yields:

7.4 HOW FAST CAN WE SORT?

Both of the sorting methods we have seen have a worst case behavior of O(n2) It is natural at this point to ask the question: "What is the best computing time for sorting that we can hope for?" The theorem we shall prove shows that if we restrict our question to algorithms for which the only operations permitted on keys are comparisons and interchanges then O(n log2 n) is the best possible time.

The method we use is to consider a tree which describes the sorting process by having a vertex represent a key comparison and the branches indicate the result. Such a tree is called a decision tree. A path through a decision tree represents a possible sequence of computations that an algorithm could produce.

As an example of such a tree, let us look at the tree obtained for Insertion Sort working on a file with three records in it. The input sequence is R1, R2 and R3 so the root of the tree is labeled (1,2,3). Depending on the outcome of the comparison between K1 and K2, this sequence may or may not change. If K2 < K1, then the sequence becomes (2,1,3) otherwise it stays 1,2,3. The full tree resulting from these comparisons is shown below. The leaf nodes are numbered I-VI and are the only points at which the algorithm may terminate. Hence only six permutations of the input sequence are obtainable from this algorithm. Since all six of these are different and 3! = 6, it follows that this algorithm has enough leaves to constitute a valid sorting algorithm for three records. The maximum depth of this tree is 3. The table below gives six different orderings of key values 7, 9, 10 which show that all six permutations are possble. The tree is not a full binary tree of depth 3 and so it has fewer than 23 = 8 leaves. The possible output permutations are:

                   SAMPLE INPUT KEY VALUES WHICH

LEAF  PERMUTATION      GIVE THE PERMUTATION

  I      1 2 3                (7,9,10)

 II      1 3 2                (7,10,9)

III      3 1 2                (9,10,7)

 IV      2 1 3                (9,7,10)

  V      2 3 1                (10,7,9)

 VI      3 2 1                (10,9,7)

The decision tree is

Theorem 7.1: Any decision tree that sorts n distinct elements has a height of at least log2(n!)+ 1.

Proof: When sorting n elements there are n! different possible results. Thus, any decision tree must have n! leaves. But a decision tree is also a binary tree which can have at most 2k-1 leaves if its height is k. Therefore, the height must be at least log2n! + 1.

Corollary: Any algorithm which sorts by comparisons only must have a worst case computing time of O(n log2 n).

Proof: We must show that for every decision tree with n! leaves there is a path of length c nlog2 n, c a constant. By the theorem, there is a path of length log2 n!. Now

             n! = n(n - 1)(n - 2) ... (3)(2)(1)

                   (n/2)n/2,

so log2 n! (n/2) log2 (n/2) = O(n log2 n).

7.5 2-WAY MERGE SORT

Before looking at the merge sort algorithm to sort n records let us see how one may merge two files (Xl, ...,Xm) and (Xm+1, ...,Xn) that are already sorted to get a third file (Zl, ...,Zn) that is also sorted. Since this merging scheme is very simple, we directly present the algorithm.

procedure MERGE(X,l,m,n,Z)

//(Xl, ...,Xm) and (Xm+1, ...,Xn) are two sorted files with keys

xl    ...    xm and Xm+1    ...    xn. They are merged to obtain the

sorted file (Zl, ..., Zn) such that zl    ...    zn//

i    k    l; j    m + 1     //i, j and k are position in the three files//

while i    m and j    n do

if xi    xj then [Zk    Xi; i    i + 1]

else [Zk    Xj; j    j + 1]

k    k + 1

end

if i > m then (Zk, ...,Zn)    (Xj, ...,Xn)

else  (Zk, ...,Zn)    (Xi ...,Xm)

end MERGE

Analysis of Algorithm MERGE

At each iteration of the while loop k increases by 1. The total increment in k is n - l + 1. Hence the while loop is iterated at most n - l + 1 times. The if statement moves at most one record per iteration. The total time is therefore O(n - l + 1). If records are of length M then this time is really O(M(n - l + 1)). When M is greater than 1 we could use linked lists (X1, ...,Xm) and (Xm+1, ...,Xn) and obtain a new sorted linked list containing these n - l + 1 records. Now, we won't need the additional space for n - l + 1 records as needed above for Z. Instead only space for n - l + 1 links is needed. The merge time becomes independent of M and is O(n - l + 1). Note that n - l + 1 is the number of records being merged.

Two-way merge sort begins by interpreting the input as n sorted files each of length 1. These are merged pairwise to obtain n/2 files of size 2 (if n is odd, then one file is of size 1). These n/2 files are then merged pairwise and so on until we are left with only one file. The example below illustrates the process.

Example 7.4.1: The input file is (26, 5, 77, 1, 61, 11, 59, 15, 15, 48, 19). The tree below illustrates the subfiles being merged at each pass:

As is apparent from the example above, merge sort consists of several passes over the records being sorted. In the first pass files of size 1 are merged. In the second, the size of the files being merged is 2. On the ith pass the files being merged are of size 2i-1.Consequently, a total of [image] log2 n [image] passes are made over the data. Since two files can be merged in linear time (algorithm MERGE), each pass of merge sort takes O(n) time. As there are [image] log2 n [image] passes, the total computing time is O(n log n).

In formally writing the algorithm for 2-way merge, it is convenient to first present an algorithm to perform one merge pass of the merge sort.

procedure MPASS(X,Y,n,l)

//This algorithm performs one pass of merge sort. It merges adjacent

pairs of subfiles of length l from file X to file Y. n is the number

of records in X//

i    1

while i    n - 2l + 1 do

call MERGE (X, i, i + l - 1, i + 2l - 1, Y)

i    i + 2l

end

//merge remaining file of length <2l//

if i + l - 1 < n then call MERGE (X, i, i + l - 1, n, Y)

else (Yi, ...Yn)    (Xi, ...,Xn)

end MPASS

The merge sort algorithm then takes the form:

procedure MSORT(X,n)

//Sort the file X = (X1, ...,Xn) into nondecreasing order of the keys

x1, ...,xn//

declare X(n), Y(n)         //Y is an auxilliary array//

//l is the size of subfiles currently being merged//

l  1

while l < n do

call MPASS(X,Y,n,l)

l  2*l

call MPASS(Y,X,n,l)          //interchange role of X and Y//

l  2*l

end

end MSORT

It is easy to verify that the above algorithm results in a stable sorting procedure. Exercise 10 discusses a variation of the two-way merge sort discussed above. In this variation the prevailing order within the input file is taken into account to obtain initially sorted subfiles of length [image] 1.

Recursive Formulation of Merge Sort

Merge sort may also be arrived at recursively. In the recursive formulation we divide the file to be sorted into two roughly equal parts called the left and the right subfiles. These subfiles are sorted using the algorithm recursively and then the two subfiles are merged together to obtain the sorted file. First, let us see how this would work on our earlier example.

Example 7.4.2: The input file (26, 5, 77, 1, 61, 11, 59, 15, 49, 19) is to be sorted using the recursive formulation of 2-way merge sort. If the subfile from l to u is currently to be sorted then its two subfiles are indexed from l to [image] (l + u)/2 [image] and from [image] (l + u)/2 [image] + 1 to u. The subfile partitioning that takes place is described by the following binary tree. Note that the subfiles being merged are different from those being merged in algorithm MSORT.

From the preceding example, we may draw the following conclusion. If algorithm MERGE is used to merge sorted subfiles from one array into another, then it is necessary to copy subfiles. For example to merge [5, 26] and [77] we would have to copy [77] into the same array as [5, 26]. To avoid this unnecessary copying of subfiles using sequential allocation, we look to a linked list representation for subfiles. This method of representation will permit the recursive version of merge sort to work efficiently.

Each record is assumed to have two fields LINK and KEY. LINK (i) and KEY(i) are the link and key value fields in record i, 1 [image] i [image] n. We assume that initially LINK(i) = 0, 1 [image] i [image] n. Thus each record is initially in a chain containing only itself. Let Q and R be pointers to two chains of records. The records on each chain are assumed linked in nondecreasing order of the key field. Let RMERGE(Q,R,P) be an algorithm to merge the two chains Q and R to obtain P which is also linked in nondecreasing order of key values. Then the recursive version of merge sort is given by algorithm RMSORT. To sort the records X1, ...,Xn this algorithm is invoked as call RMSORT(X,1,n,P). P is returned as the start of a chain ordered as described earlier. In case the file is to be physically rearranged into this order then one of the schemes discussed in section 7.8 may be used.

procedure RMSORT(X,l,u,P)

//The file X = (Xl, ...,Xu) is to be sorted on the field KEY. LINK

is a link field in each record and is initially set to 0. The sorted

file is a chain beginning at P//

if l    u then P    l

else [mid      (l + u)/2

call RMSORT(X,l,mid,Q)

call RMSORT(X,mid + 1,u,R)

call RMERGE(Q,R,P)]

end RMSORT

The algorithm RMERGE below uses a dummy record with index d. It is assumed that d is provided externally and that d is not one of the valid indexes of records i.e. d is not one of the numbers 1 through n.

procedure RMERGE(X,Y,Z)

//The linked files X and Y are merged to obtain Z. KEY(i) denotes

the key field and LINK(i) the link field of record i. In X, Y and

Z the records are linked in order of nondecreasing KEY values.

A dummy record with index d is made use of. d is not a valid

index in X or Y//

i    X; j    Y; z    d

while i    0 and j    0 do

if KEY(i)    KEY(j) then [LINK(z)    i

z    i; i    LINK (i)]

else [LINK(z)    j

z    j; j    LINK(j)]

end

//move remainder//

if i = 0 then LINK(z)    j

else LINK(z)    i

Z    LINK(d)

end RMERGE

One may readily verify that this linked version of 2-way merge sort results in a stable sorting procedure and that the computing time is O(n log n).

7.6 HEAP SORT

While the Merge Sort scheme discussed in the previous section has a computing time of O(n log n) both in the worst case and as average behavior, it requires additional storage proportional to the number of records in the file being sorted. The sorting method we are about to study will require only a fixed amount of additional storage and at the same time will have as its worst case and average computing time O(n log n). In this method we shall interpret the file to be sorted R = (R1, ...,Rn) as a binary tree. (Recall that in the sequential representation of a binary tree discussed in Chapter 5 the parent of the node at location i is at [image] i/2 [image] , the left child at 2i and the right child at 2i + 1. If 2i or 2i + 1 is greater than n (the number of nodes), then the corresponding children do not exist.) Thus, initially the file is interpreted as being structured as below:

Heap sort may be regarded as a two stage method. First the tree representing the file is converted into a heap. A heap is defined to be a complete binary tree with the property that the value of each node is at least as large as the value of its children nodes (if they exist) (i.e., K [image] j/2 [image] [image] Kj for 1 [image] [image] j/2 [image] < j [image] n). This implies that the root of the heap has the largest key in the tree. In the second stage the output sequence is generated in decreasing order by successively outputting the root and restructuring the remaining tree into a heap.

Essential to any algorithm for Heap Sort is a subalgorithm that takes a binary tree T whose left and right subtrees satisfy the heap property but whose root may not and adjusts T so that the entire binary tree satisfies the heap property. Algorithm ADJUST does this.

procedure ADJUST (i,n)

//Adjust the binary tree with root i to satisfy the heap property.

The left and right subtrees of i, i.e., with roots 2i and 2i+ 1, already

satisfy the heap property. The nodes of the trees contain records,

R, with keys K. No node has index greater than n//

R    Ri; K    Ki; j    2i

while j    n do

if j < n and Kj < Kj+1 then j    j + 1   //find max of left

and right child//

//compare max. child with K. If K is max. then done//

if K    Kj then exit

R  j/2      Rj; j    2j      //move Rj up the tree//

end

R  j/2      R

end ADJUST

Analysis of Algorithm Adjust

If the depth of the tree with root i is k, then the while loop is executed at most k times. Hence the computing time of the algorithm is O(k).

The heap sort algorithm may now be stated.

procedure HSORT (R,n)

//The file R = (R1, ...,Rn) is sorted into nondecreasing order of

the key K//

for i      n/2   to 1 by -1 do     //convert R into a heap//

call ADJUST (i,n)

end

for i    n - 1 to 1 by -1 do    //sort R//

T    Ri+1; Ri+1    R1; R1    T       //interchange R1 and Ri+1//

call ADJUST (1,i)      //recreate heap//

end

end HSORT

Example 7.5: The input file is (26, 5, 77, 1, 61, 11, 59, 15, 48, 19). Interpreting this as a binary tree we have the following transformations:

The figures on pages 360-361 illustrate the heap after restructuring and the sorted part of the file.

Analysis of Algorithm HSORT

Suppose 2k-1 [image] n < 2k so that the tree has k levels and the number of nodes on level i is 2i-1. In the first for loop ADJUST is called once for each node that has a child. Hence the time required for this loop is the sum, over each level, of the number of nodes on a level times the maximum distance the node can move. This is no more than

In the next for loop n - 1 applications of ADJUST are made with maximum depth k = [image] log2 (n + 1) [image] . Hence the computing time for this loop is O(n log n). Consequently, the total computing time is O(n log n). Note that apart from pointer variables, the only additional space needed is space for one record to carry out the exchange in the second for loop. Note also that instead of making the exchange, one could perform the sequence R [image] Ri+1, Ri+1 [image] R1 and then proceed to adjust the tree.

7.7 SORTING ON SEVERAL KEYS

Let us now look at the problem of sorting records on several keys, K1,K2, ..., Kr (K1 is the most significant key and Kr the least). A file of records R1, ...,Rn will be said to be sorted with respect to the keys K1,K2,...,Kr iff for every pair of records [image]. The r-tuple (x1, ...,xr) is less than or equal to the r-tuple (y1, ...,yr) iff either xi = yi, 1 [image] i [image] j and xj+1 < yj+1 for some j [image] r or xi = yi, 1 [image] i [image] r.

For example, the problem of sorting a deck of cards may be regarded as a sort on two keys, the suit and face values, with the following ordering relations:

and face values: 2 < 3 < 4 ... < 10 < J < Q < K < A.

There appear to be two popular ways to accomplish the sort. The first is to sort on the most significant key K1 obtaining several "piles" of records each having the same value for K1. Then each of these piles is independently sorted on the key K2 into "subpiles" such that all the records in the same subpile have the same values for K1 and K2. The subpiles are then sorted on K3, etc., and the piles put together. In the example above this would mean first sorting the 52 cards into four piles one for each of the suit values. Then sort each pile on the face value. Now place the piles on top of each other to obtain the ordering: 2 [image] , ...,A [image] , ..., ...,2 [image] , ...,A [image].

A sort proceeding in this fashion will be referred to as a most significant digit first (MSD) sort. The second way, quite naturally, is to sort on the least significant digit first (LSD). This would mean sorting the cards first into 13 piles corresponding to their face values (key K2). Then, place the 3's on top of the 2's, ..., the kings on top of the queens, the aces on top of the kings; turn the deck upside down and sort on the suit (K1) using some stable sorting method obtaining four piles each of which is ordered on K2; combine the piles to obtain the required ordering on the cards.

Comparing the two procedures outlined above (MSD and LSD) one notices that LSD is simpler as the piles and subpiles obtained do not have to be sorted independently (provided the sorting scheme used for sorting on key Ki, 1 [image] i < r < is stable). This in turn implies less overhead.

LSD and MSD only specify the order in which the different keys are to be sorted on and not the sorting method to be used within each key. The technique generally used to sort cards is a MSD sort in which the sorting on suit is done by a bin sort (i.e., four "bins" are set up, one for each suit value and the cards are placed into their corresponding "bins"). Next, the cards in each bin are sorted using an algorithm similar to Insertion Sort. However, there is another way to do this. First use a bin sort on the face value. To do this we need thirteen bins one for each distinct face value. Then collect all the cards together as described above and perform bin sort on the suits using four bins. Note that a bin solt requires only O(n) time if the spread in key values is O(n).

LSD or MSD sorting can also be used to sort records on only one logical key by interpreting this key as being composed of several keys. For example, if the keys are numeric, then each decimal digit may be regarded as a key. So if all the keys are in the range 0 [image] K [image] 999, then we can use either the LSD or MSD sorts for three keys (K1,K2,K3), where K1 is the digit in the hundredths place, K2 the digit in the tens place, and K3 that in the units place. Since all the keys lie in the range 0 [image] Ki [image] 9, the sort within the keys can be carried out using a bin sort with ten bins. This, in fact, is essentially the process used to sort records punched on cards using a card sorter. In this case using the LSD process would be more convenient as it eliminates maintaining several independent subpiles. If the key is interpreted as above, the resulting sort is called a radix 10 sort. If the key decomposition is carried out using the binary representation of the keys, then one obtains a radix 2 sort. In general, one could choose any radix r obtaining a radix r sort. The number of bins required is r.

Let us look in greater detail at the implementation of an LSD radix r sort. We assume that the records R1, ...,Rn have keys that are d-tuples (x1,x2, ...,xd) and 0 [image] xi < r. Thus, we shall need r bins. The records are assumed to have a LINK field. The records in each bin will be linked together into a linear linked list with F(i), 0 [image] i < r, a pointer to the first record in bin i and E(i) a pointer to the last record in bin i. These lists will essentially be operated as queues. Algorithm LRSORT formally presents the LSD radix r method.

Analysis of LRSORT

The algorithm makes d passes over the data, each pass taking O(n + r) time. Hence the total computing time is O(d(n + r)). In the sorting of numeric data, the value of d will depend on the choice of the radix r and also on the largest key. Different choices of r will yield different computing times (see Table 7.2).

procedure LRSORT(R,n,d)

//records R = (R1, ...,Rn) are sorted on the keys K1, ...,Kd. The

range of each key is 0    Ki < r. Sorting within a key is done using

bin sort. All records are assumed to be initially linked together

such that LINK(i) points to Ri+1, 1    i    n and LINK(n) = 0.//

declare E(0:r - 1), F(0:r - 1)    //queue pointers//

p    1       //pointer to start of list of records//

for i    d to 1 by - 1 do      //sort on key Ki//

for j    0 to r - 1 do      //initialize bins to be empty queues//

F(j)    0

end

while p    0 do       //put records into queues//

k       //k is the i-th key of p-th record//

if F(k) = 0 then F(k)    p         //attach record p into bin k//

else LINK(E(k))    p

E(k)    p

p    LINK(p)         //get next record//

end

j    0; while F(j) = 0 do j    j + 1 end  //find first nonempty

queue//

p    F(j); t    E(j)

for k    j + 1 to r - 1 do        //concatenate remaining queues//

if F(k)    0 then [LINK(t)    F(k); t   E(k)]

end

LINK(t)    0

end

end LRSORT

Example 7.6: We shall illustrate the operation of algorithm LRSORT while sorting a file of 10 numbers in the range [0,999]. Each decimal digit in the key will be regarded as a subkey. So, the value of d is 3 and that of r is 10. The input file is linked and has the form given on page 365 labeled R1, ...,R10. The figures on pages 365-367 illustrates the r = 10 case and the list after the queues have been collected from the 10 bins at the end of each phase. By using essentially the method above but by varying the radix, one can obtain (see exercises 13 and 14) linear time algorithms to sort n record files when the keys are in the range 0 [image] Ki < nk for some constant k.

7.8 PRACTICAL CONSIDERATIONS FOR INTERNAL SORTING

Apart from radix sort, all the sorting methods we have looked at require excessive data movement; i.e., as the result of a comparison, records may be physically moved. This tends to slow down the sorting process when records are large. In sorting files in which the records are large it is necessary to modify the sorting methods so as to minimize data movement. Methods such as Insertion Sort and Merge Sort can be easily modified to work with a linked file rather than a sequential file. In this case each record will require an additional link field. Instead of physically moving the record, its link field will be changed to reflect the change in position of that record in the file (see exercises 4 and 8). At the end of the sorting process, the records are linked together in the required order. In many applications (e.g., when we just want to sort files and then output them record by record on some external media in the sorted order) this is sufficient. However, in some applications it is necessary to physically rearrange the records in place so that they are in the required order. Even in such cases considerable savings can be achieved by first performing a linked list sort and then physically rearranging the records according to the order specified in the list. This rearranging can be accomplished in linear time using some additional space.

If the file, F, has been sorted so that at the end of the sort P is a pointer to the first record in a linked list of records then each record in this list will have a key which is greater than or equal to the key of the previous record (if there is a previous record), see figure 7.1. To physically rearrange these records into the order specified by the list, we begin by interchanging records R1 and RP. Now, the record in the position R1 has the smallest key. If P [image] 1 then there is some record in the list with link field = 1. If we could change this link field to indicate the new position of the record previously at position 1 then we would be left with records R2, ...,Rn linked together in nondecreasing order. Repeating the above process will, after n - 1 iterations, result in the desired rearrangement. The snag, however, is that in a singly linked list we do not know the predecessor of a node. To overcome this difficulty, our first rearrangement algorithm LIST1, begins by converting the singly linked list P into a doubly linked list and then proceeds to move records into their correct places.

Figure 7.1 List Sort

procedure LIST1(R,n,P)

//P is a pointer to a list of n sorted records linked together by the

field LINK. A second link field, LINKB, is assumed to be present

in each record. The records are rearranged so that the resulting

records R1, ..., Rn are consecutive and sorted//

u    0; s    P

while s    0 do        //convert P into a doubly linked list using LINKB//

LINKB(s)    u              //u follows s//

u    s; s    LINK(s)

end

for i    1 to n - 1 do      //move RP to position i while//

if P    i       //maintaining the list//

then [if LINK (i)    0 then LINKB(LINK (i))    P

LINK(LINKB(i))    P; A    RP

RP    Ri; Ri    A]

P    LINK(i)            //examine the next record//

end

end LIST1

Example 7.7: After a list sort on the input file (35, 18, 12, 42, 26, 14) has been made the file is linked as below (only three fields of each record are shown):

        R1  R2  R3  R4  R5  R6

Key     35  18  12  42  26  14

LINK    4    5   6   0  1    2  P = 3

LINK B

Following the links starting at RP we obtain the logical sequence of records R3, R6, R2, R5, R1, and R4 corresponding to the key sequence 12, 14, 18, 26, 35, and 42. Filling in the backward links, we have

R1  R2  R3  R4  R5  R6

35  18  12  42  26  14

 4   5   6   0   1   2  P = 3

 5   6   0   1   2   3

The configuration at the end of each execution of the for loop is:

12  18  35  42  26  14

 6   5   4   0   3   2  P = 6

 0   6   5   3   2   3

          i = 1

The logical sequence of the remaining list (LS) is: R6, R2, R5, R3, R4. The remaining execution yields

              12    14  35  42  26  18

               6     6   4   0   3   5  P = 6

               0     3   5   3   6   6

LS:  R6.  R5,  R3,  R4    i = 2

          12   14  18  42  26  35

           6    6   5   0   6   4  P = 5

           0    3   6   6   6   5

LS:  R5,  R6,  R4   i = 3

          12  14  18  26  42  35

           6   6   5   6   0   5  P = 6

           0   3   6   6   6   5

LS:  R6,  R5,      i = 4

         12  14  18  26  35  42

          6   6   5   6   6   0  P = 6

          0   3   6   6   5   6

LS: R6,           i = 5

Analysis of Algorithm LIST1

If there are n records in the file then the time required to convert the chain P into a doubly linked list is O(n). The for loop is iterated n-1 times. In each iteration at most two records are interchanged. This requires 3 records to move. If each record is m words long, then the cost per intgrchange is 3m. The total time is therefore O(nm). The worst case of 3(n - 1) record moves is achievable. For example consider the input key sequence R1, R2, ... Rn with R2 < R3 < ... < Rn and R1 > Rn. For n = 4 and keys 4, 1, 2, 3 the file after each iteration has the following form: i = 1: 1,4,2,3; i = /2: 1,2,4,3; i = 3: 1,2,3,4. A total of 9 record moves is made.

Several modifications to algorithm LIST1 are possible. One that is of interest was given by M. D. MacLaren. This results in a rearrangement algorithm in which no additional link fields are necessary. In this algorithm, after the record RP is exchanged with Ri the link field of the new Ri is set to P to indicate that the original record was moved. This, together with the observation that P must always be   [image]  i, permits a correct reordering of the records. The computing time remains O(nm).

procedure LIST2 (R,n,P)

//Same function as LIST1 except that a second link field LINKB

is not reguired//

for i    1 to n - 1 do

//find correct record to place into i'th position. The index of this

record must be    i as records in positions 1,2, ...,i -1 are already

correctly positioned//

while P < i do

P    LINK (P)

end

Q    LINK (P)          //RQ is next record with largest key//

if P    i then [//interchange Ri and Rp moving Rp to its correct

spot as Rp has i'th smallest key. Also set link from

old position of Ri to new one//   T   Ri Ri    Rp; Rp   T

LINK (i)    P]

P    Q

end

end LIST2

Example 7.8: The data is the same as in Example 7.7. After the list sort we have:

      R1  R2  R3  R4  R5  R6

Key   35  18  12  42  26  14

LINK   4   5   6   0   1   2  P = 3

After each iteration of the for loop, we have:

12  18  35  42  26  14

 3   5   4   0   3   2  P = 6

         i = 1

12  14  35  42  26  18

 3   6   4   0   1   5  P = 2

         i = 2

Now P < 3 and so it is advanced to LINK(P) = 6.

12  14  18  42  26  35

 3   6   6   0   1   4 P = 5

         i = 3

12  14  18  26  42  35

 3   6   6   5   0   4  P = 1

         i = 4

Again P < 5 and following links from Rp we find R6 to be the record with fifth smallest key.

12  14  18  26  35  42

 3   6   6   5   6   0  P = 4

         i = 5

Analysis of Algorithm LIST2

The sequence of record moves for LIST2 is identical to that for LIST1. Hence, in the worst case 3(n - 1) record moves for a total cost of O(n m) are made. No node is examined more than once in the while loop. So the total time for the [while loop is O(n). While the asymptotic computing time for both LIST1 and LIST2 is the same and the same number of record moves is made in either case, we would expect LIST2 to be slightly faster than LIST1 because each time two records are interchanged LIST1 does more work than LIST2 does. LIST1 is inferior to LIST2 on both space and time considerations.

The list sort technique discussed above does not appear to be well suited for use with sort methods such as Quick Sort and Heap Sort. The sequential representation of the heap is essential to Heap Sort. In such cases as well as in the methods suited to List Sort, one can maintain an auxiliary table with one entry per record. The entries in this table serve as an indirect reference to the records. Let this table be T(l), T(2), ...,T(n). At the start of the sort T(i) = i, 1   [image]  i   [image]  n. If the sorting algorithm requires an interchange of Ri and Rj, then only the table entries need be interchanged, i.e., T(i) and T(j). At the end of the sort, the record with the smallest key is RT(1) and that with the largest RT(n). In general, following a table sort RT(i) is the record with the i'th smallest key. The required permutation on the records is therefore RT(1), RT(2), ...,RT(n) (see Figure 7.2). This table is adequate even in situations such as binary search, where a sequentially ordered file is needed. In other situations, it may be necessary to physically rearrange the records according to the permutation specified by T. The algorithm to rearrange records corresponding to the permutation T(1), T(2), ...,T(n) is a rather interesting application of a theorem from mathematics: viz, every permutation is made up of disjoint cycles. The cycle for any element i is made up of i, T(i), T2(i), ...,Tk(i), (where Tj(i) = T(Tj-1 (i)) and To(i) = i) such that Tk(i) = i. Thus, the permutation T of figure 7.2 has two cycles, the first involving R1 and R5 and the second involving R4, R3 and R2. Algorithm TABLE utilizes this cyclic decomposition of a permutation. First, the cycle containing R1 is followed and all records moved to their correct positions. The cycle containing R2 is the next one examined unless this cycle has already been examined. The cycles for R3, R4, ...,Rn-1 are followed in that order, achieving a reordering of all the records. While processing a trivial cycle for Ri (i.e. T(i) = i), no rearrangement involving record Ri is required since the condition T(i) = i means that the record with the i'th smallest key is Ri. In processing a nontrivial cycle for record Ri (i.e. T(i)   [image]  i), Ri is moved to a temporary position P, then the record at T(i) is moved to (i); next the record at T(T(i)) is moved to T(i); and so on until the end of the cycle Tk(i) is reached and the record at P is moved to Tk-1 (i).

Figure 7.2 Table Sort

procedure TABLE(R,n,T)

//The records R1, ...,Rn are rearranged to correspond to the sequence

RT(1), ...,RT(n), n    1//

for i    1 to n - 1 do

if T(i)    i then      //there is a nontrivial cycle starting at i//

[P    Ri; j    i      //move Ri to a temporary spot P and follow//

repeat       //cycle i, T(i), T(T(i)), ... until the correct spot//

k    T(j)

Rj    Rk

T(j)    j

j    k

until T(j) = i

Rj    P        //j is position for record P//

T(j)    j]

end

end TABLE

Example 7.9: : Following a table sort on the file F we have the following values for T (only the key values for the 8 records of F are shown):

   R1  R2  R3  R4  R5  R6  R7  R8

F  35  14  12  42  26  50  31  18

T   3   2   8   5   7   1   4  6

There are two nontrivial cycles in the permutation specified by T. The first is R1, R3, R8, R6 and R1. The second is R4, R5, R7, R4. During the first iteration (i = 1) of the for loop of algorithm TABLE, the cycle R1, RT(1), RT2(1), RT3(1), R1 is followed. Record R1 is moved to a temporary spot P. RT(1) (i.e. R3) is moved to the position R1; RT2(1) (i.e. R8) is moved to R3; R6 to R8 and finally P to R6. Thus, at the end of the first iteration we have:

F  12  14  18  42  26  35  31  50

T   1   2   3   5   6   7   4   8

For i = 2,3, T(i) = i, indicating that these records are already in their correct positions. When i = 4, the next nontrivial cycle is discovered and the records on this cycle R4, R5, R7, R4 are moved to their correct positions. Following this we have:

F  12  14  18  26  31  35  42  50

T   1   2   3   5   7   6   4   8

For the remaining values of i (i = 5, 6 and 7), T(i) = i, and no more nontrivial cycles are found.

Analysis of Algorithm TABLE

If each record uses m words of storage then the additional space needed is m words for P plus a few more for variables such as i, j and k. To obtain an estimate of the computing time we observe that the for loop is executed n-1 times. If for some value of i, T(i)   [image]  i then there is a nontrivial cycle including k > 1 distinct records Ri, RT(i), ...,RTk-1(i).   Rearranging these records requires k + 1 record moves. Following this, the records involved in this cycle are not moved again at any time in the algorithm since T(j) = j for all such records Rj. Hence no record can be in two different nontrivial cycles. Let kl be the number of records on a nontrivial cycle starting at Rl when i = l in the algorithm. Let kl = 0 for a trivial cycle. Then, the total number of record moves is   [image]. Since the records on nontrivial cycles must be different,   [image] kl   [image]  n. The total record moves is thus maximum when   [image] kl = n and there are   [image] n/2  [image]  cycles. When n is even, each cycle contains 2 records. Otherwise one contains three and the others two. In either case the number of record moves is   [image] 3n/2  [image]. One record move costs O(m) time. The total computing time is therefore O(nm).

In comparing the algorithms LIST2 and TABLE for rearranging records we see that in the worst case LIST2 makes 3(n - 1) record moves while TABLE makes only   [image] 3n/2  [image]  record moves. For larger values of m it would therefore be worthwhile to make one pass over the sorted list of records creating a table T corresponding to a table sort. This would take O(n) time. Then algorithm TABLE could be used to rearrange the records in the order specified by T.

                        n      10    20    50   100    250   500    1000

Quicksort [with median of 3]

  (File: [N,1,2,3, ...,

  N - 2, N - 1])              .499  1.26  4.05  12.9   68.7  257.   1018.

Quicksort [without median

  of 3] (File: [1,2,3, ...,

  N - 1, N])                  .580  1.92  9.92  38.2  226.   856.   3472.

Insertion Sort

  [with K(0) = -  ]

  (File: [N,N - 1, ...,2,1])  .384  1.38  8.20  31.7  203.   788.    --

Insertion Sort

  [without K(0) = -  ]

  (File: [N,N - 1, ...,2,1])  .382  1.48  9.00  35.6  214.   861.    --

Heap Sort                     .583  1.52  4.96  11.9   36.2   80.5   177.

Merge Sort                    .726  1.66  4.76  11.3   35.3   73.8   151

(a) Worst case times m milliseconds

       n       10       20       50      100      250       500       1000

Radix Sort

 (L.S.D.)

 (RD           1.82     3.05     5.68     9.04     20.1      32.5      58.0

  100,000;

  optimal

  D & R)     R = 18;  R = 18;  R = 47;  R = 47;  R = 317;  R = 317;  R = 317;

              D = 4    D = 4    D = 3    D = 3    D = 2     D = 2     D = 2

Radix Sort

 (L.S.D.)

     R = 10,

     D = 5)   1.95     3.23     6.97     13.2     32.1      66.4      129.

Quicksort

[with

median

  of 3]        .448    1.06     3.17      7.00    20.0      43.1       94.9

Quicksort

 [without

 median

  of 3]        .372     .918    2.89      6.45    20.0      43.6       94.0

Insertion

 Sort          .232     .813    4.28     16.6     97.1     385.         --

Insertion

 Sort

  (without

  K(0) = -  )   .243      .885   4.72     18.5    111.      437.         --

Heap Sort      .512     1.37    4.52     10.9     33.2      76.1     166.

Merge Sort     .642     1.56    4.55     10.5     29.6      68.2     144.

(b) Average times in milliseconds

Table 7.2 Computing times for sorting methods. (Table prepared by Randall Istre)

Of the several sorting methods we have studied there is no one method that is best. Some methods are good for small n, others for large n. Insertion Sort is good when the file is already partially ordered. Because of the low overhead of the method it is also the best sorting method for "small" n. Merge Sort has the best worst case behavior but requires more storage than Heap Sort (though also an O(n log n) method it has slightly more overhead than Merge Sort). Quick Sort has the best average behavior but its worst case behavior is O(n2). The behavior of Radix Sort depends on the size of the keys and the choice of r.

These sorting methods have been programmed in FORTRAN and experiments conducted to determine the behavior of these methods. The results are summarized in Table 7.2. Table 7.2(a) gives the worst case sorting times for the various methods while table 7.2(b) gives the average times. Since the worst case and average times for radix sort are almost the same, only the average times have been reported. Table 7.2(b) contains two rows for Radix Sort. The first gives the times when an optimal radix r is used for the sort. The second row gives the times when the radix is fixed at 10. Both tables contain two rows for Insertion Sort. Comparing the two rows gives us an indication of the time saved by using a dummy key, K(0), in the algorithm as opposed to explicitly checking for the left most record (i.e. R(1)). In a separate study it was determined that for average times, Quicksort is better than Insertion Sort only when n   [image]  23 and that for worst case times Merge Sort is better than Insertion Sort only when n   [image]  25. The exact cut off points will vary with different implementations. In practice, therefore, it would be worthwhile to couple Insertion Sort with other methods so that subfiles of size less than about 20 are sorted using Insertion Sort.

REFERENCES

A comprehensive discussion of sorting and searching may be found in:

The Art of Computer Programming: Sorting and Searching, by D.  Knuth, vol. 3, Addison-Wesley, Reading, Massachusetts, 1973.

Two other useful references on sorting are:

Sorting and Sort Systems by H. Lorin, Addison-Wesley, Reading, Massachusetts, 1975.

Internal sorting methods illustrated with PL/1 Programs by R. Rich, Prentice-Hall, Englewood Cliffs, 1972.

For an in depth study of quicksort and stable merge sort see:

"Quicksort" by R. Sedgewick, STAN-CS-75-492, May 1975, Computer Science Department, Stanford University.

"Stable Sorting and Merging With Optimal Space and Time Bounds" by L. Pardo, STAN-CS-74-470, December 1974, Computer Science Department, Stanford University.

EXERCISES

1.     Work through algorithms BINSRCH and FIBSRCH on an ordered file with keys (1,2,3,4,5,6,7,8,9,10,11,12,13,14,15,16) and determine the number of key comparisons made while searching for the keys 2, 10 and 15. For FIBSRCH we need the three Fibonnaci numbers F5 = 5, F6 = 8, F7 = 13.

2.     [Count sort] About the simplest known sorting method arises from the observation that the position of a record in a sorted file depends on the number of records with smaller keys. Associated with each record there is a COUNT field used to determine the number of records which must precede this one in the sorted file. Write an algorithm to determine the COUNT of each record in an unordered file. Show that if the file has n records then all the COUNTs can be determined by making at most n(n - 1)/2 key comparisons.

3.     The insertion of algorithm INSERT was carried out by (a) searching for the spot at which the insertion is to be made and (b) making the insertion. If as a result of the search it was decided that the insertion had to be made between Ri and Ri+1, then records Ri+1, ...,Rn were moved one space to locations Ri+2, ...,Rn+1. This was carried out in parallel with the search of (a). (a) can be sped up using the idea of BINSRCH or FIBSRCH. Write an INSERT algorithm incorporating one of these two searches.

4.     Phase (b) (see exercise 3) can be sped up by maintaining the sorted file as a linked list. In this case the insertion can be made without any accompanying movement of the other records. However, now (a) must be carried out sequentially as before. Such an insertion scheme is known as list insertion. Write an algorithm for list insertion. Note that the insertion algorithms of exercises 3 and 4 can be used for a sort without making any changes in INSORT.

5.     a)     Show that algorithm QSORT takes O(n2) time when the input file is already in sorted order.

b)      Why is Km   [image]  Kn+1 required in QSORT?

6.     (a)     The quicksort algorithm QSORT presented in section 7.3 always fixes the position of the first record in the subfile currently being sorted. A better choice for this record is to choose the record with key value which is the median of the keys of the first, middle and last record in the subfile. Thus, using this median of three rule we correctly fix the position of the record Ri with Ki = median {Km, K(m+n)/2, Kn} i.e. Ki is the second largest key e.g. median {10,5,7} = 7 = median {10,7,7}. Write a nonrecursive version of QSORT incorporating this median of three rule to determine the record whose position is to be fixed. Also, adopt the suggestion of section 7.8 and use Insertion Sort to sort subfiles of size less than 21. Show that this algorithm takes O(n log n) time on an already sorted file.

(b)     Show that if smaller subfiles are sorted first then the recursion in algorithm QSORT can be simulated by a stack of depth O(log n).

7.     Quicksort is an unstable sorting method. Give an example of an input file in which the order of records with equal keys is not preserved.

8.     a)     Write a nonrecursive merge sort algorithm using linked lists to represent sorte subfiles. Show that if n records each of size m are being sorted then the time required is only O (n log n) as no records are physically moved.

b)     Use the rules of section 4.9 to automatically remove the recursion from the recursive version of merge sort.

c)     Take the two algorithms written above and run them on random data for n = 100, 200, ...,1000 and compare their running times.

9.     (i)     Prove that algorithm MSORT is stable.

(ii)     Heap sort is unstable. Give an example of an input file in which the order of records with equal keys is nto preserved.

10.     In the 2-way merge sort scheme discussed in section 7.5 the sort was started with n sorted files each of size 1. Another approach would be to first make one pass over the data determining sequences of records that are in order and then using these as the initially sorted files. In this case, a left to right pass over the data of example 7.4 would result in the following partitioning of the data file into sorted subfiles. This would be followed by pairwise merging of the files until only one file remains.

Rewrite the 2-way merge sort algorithm to take into account the existing order in the records. How much time does this algorithm take on an initially sorted file? Note that the original algorithm took O(n log n) on such an input file. What is the worst case computing time of the new algorithm? How much additional space is needed? Use linked lists.

11.     Does algorithm LRSORT result in a stable sort when used to sort numbers as in Example 7.6?

12.     Write a sort algorithm to sort records R1, ...,Rn lexically on keys (K1, ...,Kr) for the case when the range of each key is much lager than n. In this case the bin sort scheme used in LRSORT to sort within each key becomes inefficienct (why?). What scheme would you use to sort within a key if we desired an algorithm with

a)     good worst case behavior

b)     good average behavior

c)     n is small, say < 15.

13.     If we have n records with integer keys in the range [0, n2), then they may be sorted in O(n log n) time using heap or merge sort. Radix sort on a single key, i.e., d = 1 and r = n2 takes O(n2) time. Show how to interpret the keys as 2 subkeys so that radix sort will take only O(n) time to sort n records. (Hint: each key, Ki, may be written as   [image]  with   [image]  and   [image]  integers in the range [0, n).)

14.     Generalize the method of the previous exercise to the case of integer keys in the range [0, np) obtaining O(pn) sorting method.

15.     Write the status of the following file F at the end of each phase of the following algorithms;

a)     INSORT

b)     QSORT

c)     MSORT

d)     HSORT

e)     LSORT - radix 10

F = (12, 2, 16, 30, 8, 28, 4, 10, 20, 6, 18)

16.     Write a table sort version of quicksort. Now during the sort, records are not physically moved. Instead, T(i) is the index of the record that would have been in position i if records were physically moved around as in algorithm QSORT. To begin with T(i) = i, 1   [image]  i   [image]  n. At the end of the sort T(i) is the index of the record that should be in the i'th position in the sorted file. So now algorithm TABLE of section 7.8 may be sued to rearrange the records into the sorted order specified by T. Note that this reduces the amount of data movement taking place when compared to QSORT for the case of large records.

17.     Write an algorithm similar to algorithm TABLE to rearrange the records of a file if with each record we have a COUNT of the number of records preceding it in the sorted file (see Exercise 2).

18.     Under what conditions would a MSD Radix sort be more efficient than an LSD Radix sort?

19.     Assume you are given a list of five-letter English words and are faced with the problem of lisitng out these words in sequences such that the words in each sequence are anagrams, i.e., if x and y are in the same sequence, then word x is a permutation of word y. You are required to list out the fewest such sequences. With this restriction show that no word can appear in more than one sequence. How would you go about solving this problem?

20.     Assume you are working in the census department of a small town where the number of records, about 3,000, is small enough to fit into the internal memory of a computer. All the people currently living in this town were born in the United States. There is one record for each person in this town. Each record contains

i)     the state in which the person was born;

ii)     county of birth;

iii)     name of person.

How would you produce a list of all persons living in this town? The list is to be ordered by state. Within each state the persons are to be listed by their counties. The counties being arranged in aphabetical order. Within each county the names are also listed in alphabetical order. Justify any assumptions you may make.

Go to Chapter 8     Back to Table of Contents


Fundamentals: CHAPTER 8: EXTERNAL SORTING

 Return to Table of Contents

CHAPTER 8: EXTERNAL SORTING

In this chapter we consider techniques to sort large files. The files are assumed to be so large that the whole file cannot be contained in the internal memory of a computer, making an internal sort impossible. Before discussing methods available for external sorting it is necessary first to study the characteristics of the external storage devices which can be used to accomplish the sort. External storage devices may broadly be categorized as either sequential access (e.g., tapes) or direct access (e.g., drums and disks). Section 8.1 presents a brief study of the properties of these devices. In sections 8.2 and 8.3 we study sorting methods which make the best use of these external devices.

8.1 STORAGE DEVICES

8.1.1 Magnetic Tapes

Magnetic tape devices for computer input/output are similar in principle to audio tape recorders. The data is recorded on magnetic tape approximately 1/2\" wide. The tape is wound around a spool. A new reel of tape is normally 2400 ft. long (with use, the length of tape in a reel tends to decrease because of frequent cutting off of lengths of the tape). Tracks run across the length of the tape, with a tape having typically 7 to 9 tracks across its width. Depending on the direction of magnetization, a spot on the track can represent either a 0 or a 1 (i.e., a bit of information). At any point along the length of the tape, the combination of bits on the tracks represents a character (e.g., A-Z, 0-9, +, :, ;, etc.). The number of bits that can be written per inch of track is referred to as the tape density. Examples of standard track densities are 800 and 1600 bpi (bits per inch). Since there are enough tracks across the width of the tape to represent a character, this density also gives the number of characters per inch of tape. Figure 8.1 illustrates this. With the conventions of the figure, the code for the first character on the tape is 10010111 while that for the third character is 00011100. If the tape is written using a density of 800 bpi then the length marked x in the figure is 3/800 inches.

Figure 8.1 Segment of a Magnetic Tape

Reading from a magnetic tape or writing onto one is done from a tape drive, as shown in figure 8.2. A tape drive consists of two spindles. On one of the spindles is mounted the source reel and on the other the take up reel. During forward reading or forward writing, the tape is pulled from the source reel across the read/write heads and onto the take up reel. Some tape drives also permit backward reading and writing of tapes; i.e., reading and writing can take place when tape is being moved from the take up to the source reel.

If characters are packed onto a tape at a density of 800 bpi, then a 2400 ft. tape would hold a little over 23 x 106 characters. A density of 1600 bpi would double this figure. However, it is necessary to block data on a tape since each read/write instruction transmits a whole block of information into/from memory. Since normally we would neither have enough space in memory for one full tape load nor would we wish to read the whole tape at once, the information on a tape will be grouped into several blocks. These blocks may be of a variable or fixed size. In between blocks of data is an interblock gap normally about 3/4 inches long. The interblock gap is long enough to permit the tape to accelerate from rest to the correct read/write speed before the beginning of the next block reaches the read/write heads. Figure 8.3 shows a segment of tape with blocked data.

Figure 8.2 A Tape Drive

Figure 8.3 Blocked Data on a Tape

In order to read a block from a tape one specifies the length of the block and also the address, A, in memory where the block is to be transmitted. The block of data is packed into the words A, A + 1, A + 2, .... Similarly, in order to write a block of data onto tape one specifies the starting address in memory and the number of consecutive words to be written. These input and output areas in memory will be referred to as buffers. Usually the block size will correspond to the size of the input/output buffers set up in memory. We would like these blocks to be as large as possible for the following reasons:

(i) Between any pair of blocks there is an interblock gap of 3/4\". With a track density of 800 bpi, this space is long enough to write 600 characters. Using a block length of 1 character/block on a 2400 ft. tape would result in roughly 38,336 blocks or a total of 38,336 characters on the entire tape. Tape utilization is 1/601 < 0.17%. With 600 characters per block, half the tape would be made up of interblock gaps. In this case, the tape would have only about 11.5 x 106 characters of information on it, representing a 50% utilization of tape. Thus, the longer the blocks the more characters we can write onto the tape.

(ii) If the tape starts from rest when the input/output command is issued, then the time required to write a block of n characters onto the tape is ta + ntw where ta is the delay time and tw the time to transmit one character from memory to tape. The delay time is the time needed to cross the interblock gap. If the tape starts from rest then ta includes the time to accelerate to the correct tape speed. In this case ta is larger than when the tape is already moving at the correct speed when a read/write command is issued. Assuming a tape speed of 150 inches per second during read/write and 800 bpi the time to read or write a character is 8.3 x 10-6sec. The transmission rate is therefore 12 x 104 characters/second. The delay time ta may typically be about 10 milliseconds. If the entire tape consisted of just one long block, then it could be read in [image] thus effecting an average transmission rate of almost 12 x 104 charac/sec. If, on the other hand, each block were one character long, then the tape would have at most 38,336 characters or blocks. This would be the worst case and the read time would be about 6 min 24 sec or an average of 100 charac/sec. Note that if the read of the next block is initiated soon enough after the read of the present block, then the delay time would be reduced to 5 milliseconds, corresponding to the time needed to get across the interblock gap of 3/4\" at a tape speed of 150 inches per second. In this case the time to read 38,336 one character blocks would be 3 min 12 sec, corresponding to an average of about 200 charac/sec.

While large blocks are desirable from the standpoint of efficient tape usage as well as reduced input/output time, the amount of internal memory available for use as input/output buffers puts a limitation on block size.

Computer tape is the foremost example of a sequential access device. If the read head is positioned at the front of the tape and one wishes to read the information in a block 2000 ft. down the tape then it is necessary to forward space the tape the correct number of blocks. If now we wish to read the first block, the tape would have to be rewound 2000 ft. to the front before the first block could be read. Typical rewind times over 2400 ft. of tape could be around one minute.

Unless otherwise stated we will make the following assumptions about our tape drives:

(i) Tapes can be written and read in the forward direction only.

(ii) The input/output channel of the computer is such as to permit the following three tasks to be carried out in parallel: writing onto one tape, reading from another and CPU operation.

(iii) If blocks 1, ...,i have been written on a tape, then the tape can be moved backwards block by block using a backspace command or moved to the first block via a rewind command. Overwriting block i- 1 with another block of the same size destroys the leading portion of block i. While this latter assumption is not true of all tape drives, it is characteristic of most of them.

8.1.2 Disk Storage

As an example of direct access external storage, we consider disks. As in the case of tape storage, we have here two distinct components: (1) the disk module (or simply disk or disk pack) on which information is stored (this corresponds to a reel of tape in the case of tape storage) and (2) the disk drive (corresponding to the tape drive) which performs the function of reading and writing information onto disks. Like tapes, disks can be removed from or mounted onto a disk drive. A disk pack consists of several platters that are similar to phonograph records. The number of platters per pack varies and typically is about 6. Figure 8.4 shows a disk pack with 6 platters. Each platter has two surfaces on which information can be recorded. The outer surfaces of the top and bottom platters are not used. This gives the disk of figure 8.4 a total of 10 surfaces on which information may be recorded. A disk drive consists of a spindle on which a disk may be mounted and a set of read/write heads. There is one read/write head for each surface. During a read/write the heads are held stationary over the position of the platter where the read/write is to be performed, while the disk itself rotates at high speeds (speeds of 2000-3000 rpm are fairly common). Thus, this device will read/write in concentric circles on each surface. The area that can be read from or written onto by a single stationary head is referred to as a track. Tracks are thus concentric circles, and each time the disk completes a revolution an entire track passes a read/write head. There may be from 100 to 1000 tracks on each surface of a platter. The collection of tracks simultaneously under a read/write head on the surfaces of all the platters is called a cylinder. Tracks are divided into sectors. A sector is the smallest addressable segment of a track. Information is recorded along the tracks of a surface in blocks. In order to use a disk, one must specify the track or cylinder number, the sector number which is the start of the block and also the surface. The read/write head assembly is first positioned to the right cylinder. Before read/write can commence, one has to wait for the right sector to come beneath the read/write head. Once this has happened, data transmission can take place. Hence, there are three factors contributing to input/output time for disks:

(i) Seek time: time taken to position the read/write heads to the correct cylinder. This will depend on the number of cylinders across which the heads have to move.

(ii) Latency time: time until the right sector of the track is under the read/write head.

(iii) Transmission time: time to transmit the block of data to/from the disk.

Figure 8.4 A Disk Drive with Disk Pack Mounted (Schematic).

Maximum seek times on a disk are around 1/10 sec. A typical revolution speed for disks is 2400 rpm. Hence the latency time is at most 1/40 sec (the time for one revolution of the disk). Transmission rates are typically between 105 characters/second and 5 x 105 characters/second. The number of characters that can be written onto a disk depends on the number of surfaces and tracks per surface. This figure ranges from about 107 characters for small disks to about 5 x 108 characters for a large disk.

8.2 SORTING WITH DISKS

The most popular method for sorting on external storage devices is merge sort. This method consists of essentially two distinct phases. First, segments of the input file are sorted using a good internal sort method. These sorted segments, known as runs, are written out onto external storage as they are generated. Second, the runs generated in phase one are merged together following the merge tree pattern of Example 7.4, until only one run is left. Because the merge algorithm of section 7.5 requires only the leading records of the two runs being merged to be present in memory at one time, it is possible to merge large runs together. It is more difficult to adapt the other methods considered in Chapter 7 to external sorting. Let us look at an example to illustrate the basic external merge sort process and analyze the various contributions to the overall computing time. A file containing 4500 records, A1, . . . ,A4500, is to be sorted using a computer with an internal memory capable of sorting at most 750 records. The input file is maintained on disk and has a block length of 250 records. We have available another disk which may be used as a scratch pad. The input disk is not to be written on. One way to accomplish the sort using the general procedure outlined above is to:

(i) Internally sort three blocks at a time (i.e., 750 records) to obtain six runs R1-R6. A method such as heapsort or quicksort could be used. These six runs are written out onto the scratch disk (figure 8.5).

Figure 8.5 Blocked Runs Obtained After Internal Sorting

(ii) Set aside three blocks of internal memory, each capable of holding 250 records. Two of these blocks will be used as input buffers and the third as an output buffer. Merge runs R1 and R2. This is carried out by first reading one block of each of these runs into input buffers. Blocks of runs are merged from the input buffers into the output buffer. When the output buffer gets full, it is written out onto disk. If an input buffer gets empty, it is refilled with another block from the same run. After runs R1 and R2 have been merged, R3 and R4 and finally R5 and R6 are merged. The result of this pass is 3 runs, each containing 1500 sorted records of 6 blocks. Two of these runs are now merged using the input/output buffers set up as above to obtain a run of size 3000. Finally, this run is merged with the remaining run of size 1500 to obtain the desired sorted file (figure 8.6).

Let us now analyze the method described above to see how much time is required to sort these 4500 records. The analysis will use the following notation:

ts = maximum seek time

tl = maximum latency time

trw = time to read or write one block of 250 records

tIO = ts + tl + trw

tIS = time to internally sort 750 records

n tm = time to merge n records from input buffers to the output buffer

We shall assume that each time a block is read from or written onto the disk, the maximum seek and latency times are experienced. While this is not true in general, it will simplify the analysis. The computing time for the various operations are:

Figure 8.6 Merging the 6 runs

                       Operation                                 Time

                       ----------                                ----

1) read 18 blocks of input, 18tIO, internally sort, 6tIS

   write 18 blocks, 18tIO                                        36 tIO + 6 tIS

2) merge runs 1-6 in pairs                                  36 tIO + 4500 tm

3) merge 2 runs of 1500 records each, 12 blocks             24 tIO + 3000 tm

4) merge one run of 3000 records with one run of 1500

   records                                                  36 tIO + 4500 tm

                                                    --------------------------

                                      Total Time    132 tIO + 12000 tm + 6 tIS

Note that the contribution of seek time can be reduced by writing blocks on the same cylinder or on adjacent cylinders. A close look at the final computing time indicates that it depends chiefly on the number of passes made over the data. In addition to the initial input pass made over the data for the internal sort, the merging of the runs requires 2-2/3 passes over the data (one pass to merge 6 runs of length 750 records, two thirds of a pass to merge two runs of length 1500 and one pass to merge one run of length 3000 and one of length 1500). Since one full pass covers 18 blocks, the input and output time is 2 X (2-2/3 + 1) X 18 tIO = 132 tIO. The leading factor of 2 appears because each record that is read is also written out again. The merge time is 2-2/3 X 4500 tm = 12,000 tm. Because of this close relationship between the overall computing time and the number of passes made over the data, future analysis will be concerned mainly with counting the number of passes being made. Another point to note regarding the above sort is that no attempt was made to use the computer's ability to carry out input/output and CPU operation in parallel and thus overlap some of the time. In the ideal situation we would overlap almost all the input/output time with CPU processing so that the real time would be approximately 132 tIO [image] 12000 tm + 6 tIS.

If we had two disks we could write on one while reading from the other and merging buffer loads already in memory all at the same time. In this case a proper choice of buffer lengths and buffer handling schemes would result in a time of almost 66 tIO. This parallelism is an important consideration when the sorting is being carried out in a non-multi-programming environment. In this situation unless input/output and CPU processing is going on in parallel, the CPU is idle during input/output. In a multi-programming environment, however, the need for the sorting program to carry out input/output and CPU processing in parallel may not be so critical since the CPU can be busy working on another program (if there are other programs in the system at that time), while the sort program waits for the completion of its input/output. Indeed, in many multi-programming environments it may not even be possible to achieve parallel input, output and internal computing because of the structure of the operating system.

The remainder of this section will concern itself with: (1) reduction of the number of passes being made over the data and (2) efficient utilization of program buffers so that input, output and CPU processing is overlapped as much as possible. We shall assume that runs have already been created from the input file using some internal sort scheme. Later, we investigate a method for obtaining runs that are on the average about 2 times as long as those obtainable by the methods discussed in Chapter 7.

8.2.1 k-Way Merging

The 2-way merge algorithm of Section 7.5 is almost identical to the merge procedure just described (figure 8.6). In general, if we started with m runs, then the merge tree corresponding to figure 8.6 would have [image] log2m [image] + 1 levels for a total of [image] log2m [image] passes over the data file. The number of passes over the data can be reduced by using a higher order merge, i.e., k-way merge for k [image] 2. In this case we would simultaneously merge k runs together. Figure 8.7 illustrates a 4-way merge on 16 runs. The number of passes over the data is now 2, versus 4 passes in the case of a 2-way merge. In general, a k-way merge on m runs requires at most [image] logkm [image] passes over the data. Thus, the input/output time may be reduced by using a higher order merge. The use of a higher order merge, however, has some other effects on the sort. To begin with, k-runs of size S1, S2, S3, ...,Sk can no longer be merged internally in [image] time. In a k-way merge, as in a 2-way merge, the next record to be output is the one with the smallest key. The smallest has now to be found from k possibilities and it could be the leading record in any of the k-runs. The most direct way to merge k-runs would be to make k - 1 comparisons to determine the next record to output. The computing time for this would be [image]. Since logkm passes are being made, the total number of key comparisons being made is n(k - 1) logkm = n(k - 1) log2m/log2k where n is the number of records in the file. Hence, (k - 1)/log2k is the factor by which the number of key comparisons increases. As k increases, the reduction in input/output time will be overweighed by the resulting increase in CPU time needed to perform the k-way merge. For large k (say, k [image] 6) we can achieve a significant reduction in the number of comparisons needed to find the next smallest element by using the idea of a selection tree. A selection tree is a binary tree where each node represents the smaller of its two children. Thus, the root node represents the smallest node in the tree. Figure 8.9 illustrates a selection tree for an 8-way merge of 8-runs.

Figure 8.7 A 4-way Merge on 16 Runs

Figure 8.8 A k-Way Merge

The construction of this selection tree may be compared to the playing of a tournament in which the winner is the record with the smaller key. Then, each nonleaf node in the tree represents the winner of a tournament and the root node represents the overall winner or the smallest key. A leaf node here represents the first record in the corresponding run. Since the records being sorted are generally large, each node will contain only a pointer to the record it represents. Thus, the root node contains a pointer to the first record in run 4. The selection tree may be represented using the sequential allocation scheme for binary trees discussed in section 5.3. The number above each node in figure 8.9 represents the address of the node in this sequential representation. The record pointed to by the root has the smallest key and so may be output. Now, the next record from run 4 enters the selection tree. It has a key value of 15. To restructure the tree, the tournament has to be replayed only along the path from node 11 to the root. Thus, the winner from nodes 10 and 11 is again node 11 (15 < 20). The winner from nodes 4 and 5 is node 4 (9 < 15). The winner from 2 and 3 is node 3 (8 < 9). The new tree is shown in figure 8.10. The tournament is played between sibling nodes and the result put in the parent node. Lemma 5.3 may be used to compute the address of sibling and parent nodes efficiently. After each comparison the next takes place one higher level in the tree. The number of levels in the tree is [image] log2k [image] + 1. So, the time to restructure the tree is O(log2k). The tree has to be restructured each time a record is merged into the output file. Hence, the time required to merge all n records is O(n log2k). The time required to set up the selection tree the first time is O(k). Hence, the total time needed per level of the merge tree of figure 8.8 is O(n log2k). Since the number of levels in this tree is O(logkm), the asymptotic internal processing time becomes O(n log2k logkm) = O(n log2 m). The internal processing time is independent of k.

Figure 8.9 Selection tree for 8-way merge showing the first three keys in each of the 8 runs.

Note, however, that the internal processing time will be increased slightly because of the increased overhead associated with maintaining the tree. This overhead may be reduced somewhat if each node represents the loser of the tournament rather than the winner. After the record with smallest key is output, the selection tree of figure 8.9 is to be restructured. Since the record with the smallest key value is in run 4, this restructuring involves inserting the next record from this run into the tree. The next record has key value 15. Tournaments are played between sibling nodes along the path from node 11 to the root. Since these sibling nodes represent the losers of tournaments played earlier, we could simplify the restructuring process by placing in each nonleaf node a pointer to the record that loses the tournament rather than to the winner of the tournament. A tournament tree in which each nonleaf node retains a pointer to the loser is called a tree of losers. Figure 8.11 shows the tree of losers corresponding to the selection tree of figure 8.9. For convenience, each node contains the key value of a record rather than a pointer to the record represented. The leaf nodes represent the first record in each run. An additional node, node 0, has been added to represent the overall winner of the tournament. Following the output of the overall winner, the tree is restructured by playing tournaments along the path from node 11 to node 1. The records with which these tournaments are to be played are readily available from the parent nodes. We shall see more of loser trees when we study run generation in section 8.2.3.

Figure 8.10 Selection tree of Figure 8.9 after one record has been output and tree restructured. Nodes that were changed are marked by .

Figure 8.11 Tree of Losers Corresponding to Figure 8.9

In going to a higher order merge, we save on the amount of input/output being carried out. There is no significant loss in internal processing speed. Even though the internal processing time is relatively insensitive to the order of the merge, the decrease in input/output time is not as much as indicated by the reduction to logk m passes. This is so because the number of input buffers needed to carry out a k-way merge increases with k. Though k + 1 buffers are sufficient, we shall see in section 8.2.2 that the use of 2k + 2 buffers is more desirable. Since the internal memory available is fixed and independent of k, the buffer size must be reduced as k increases. This in turn implies a reduction in the block size on disk. With the reduced block size each pass over the data results in a greater number of blocks being written or read. This represents a potential increase in input/output time from the increased contribution of seek and latency times involved in reading a block of data. Hence, beyond a certain k value the input/output time would actually increase despite the decrease in the number of passes being made. The optimal value for k clearly depends on disk parameters and the amount of internal memory available for buffers (exercise 3).

8.2.2 Buffer Handling for Parallel Operation

If k runs are being merged together by a k-way merge, then we clearly need at least k input buffers and one output buffer to carry out the merge. This, however, is not enough if input, output and internal merging are to be carried out in parallel. For instance, while the output buffer is being written out, internal merging has to be halted since there is no place to collect the merged records. This can be easily overcome through the use of two output buffers. While one is being written out, records are merged into the second. If buffer sizes are chosen correctly, then the time to output one buffer would be the same as the CPU time needed to fill the second buffer. With only k input buffers, internal merging will have to be held up whenever one of these input buffers becomes empty and another block from the corresponding run is being read in. This input delay can also be avoided if we have 2k input buffers. These 2k input buffers have to be cleverly used in order to avoid reaching a situation in which processing has to be held up because of lack of input records from any one run. Simply assigning two buffers per run does not solve the problem. To see this, consider the following example.

Example 8.1: Assume that a two way merge is being carried out using four input buffers, IN(i), 1 [image] i [image] 4, and two output buffers, OU(1) and OU(2). Each buffer is capable of holding two records. The first few records of run 1 have key value 1, 3, 5, 7, 8, 9. The first few records of run 2 have key value 2, 4, 6, 15, 20, 25. Buffers IN(1) and IN(3) are assigned to run 1. The remaining two input buffers are assigned to run 2. We start the merging by reading in one buffer load from each of the two runs. At this time the buffers have the configuration of figure 8.12(a). Now runs 1 and 2 are merged using records from IN (1) and IN(2). In parallel with this the next buffer load from run 1 is input. If we assume that buffer lengths have been chosen such that the times to input, output and generate an output buffer are all the same then when OU(1) is full we have the situation of figure 8.12(b). Next, we simultaneously output OU(1), input into IN(4) from run 2 and merge into OU(2). When OU(2) is full we are in the situation of figure 8.12(c). Continuing in this way we reach the configuration of figure 8.12(e). We now begin to output OU(2), input from run 1 into IN(3) and merge into OU(1). During the merge, all records from run 1 get exhausted before OU(1) gets full. The generation of merged output must now be delayed until the inputting of another buffer load from run 1 is completed!

Example 8.1 makes it clear that if 2k input buffers are to suffice then we cannot assign two buffers per run. Instead, the buffers must be floating in the sense that an individual buffer may be assigned to any run depending upon need. In the buffer assignment strategy we shall describe, for each run there will at any time be, at least one input buffer containing records from that run. The remaining buffers will be filled on a priority basis. I.e., the run for which the k-way merging algorithm will run out of records first is the one from which the next buffer will be filled. One may easily predict which run's records will be exhausted first by simply comparing the keys of the last record read from each of the k runs. The smallest such key determines this run. We shall assume that in the case of equal keys the merge process first merges the record from the run with least index. This means that if the key of the last record read from run i is equal to the key of the last record read from run j, and i < j, then the records read from i will be exhausted before those from j. So, it is possible that at any one time we might have more than two bufferloads from a given run and only one partially full buffer from another run. All bufferloads from the same run are queued together. Before formally presenting the algorithm for buffer utilization, we make the following assumptions about the parallel processing capabilities of the computer system available:

(i) We have two disk drives and the input/output channel is such that it is possible simultaneously to read from one disk and write onto the other.

(ii) While data transmission is taking place between an input/output device and a block of memory, the CPU cannot make references to that same block of memory. Thus, it is not possible to start filling the front of an output buffer while it is being written out. If this were possible, then by coordinating the transmission and merging rate only one output buffer would be needed. By the time the first record for the new output block was determined, the first record of the previous output block would have been written out.

(iii) To simplify the discussion we assume that input and output buffers are to be the same size.

Figure 8.12 Example showing that two fixed buffers per run are not enough for continued parallel operation

Keeping these assumptions in mind, we first formally state the algorithm obtained using the strategy outlined earlier and then illustrate its working through an example. Our algorithm merges k-runs, [image] , using a k-way merge. 2k input buffers and 2 output buffers are used. Each buffer is a contiguous block of memory. Input buffers are queued in k queues, one queue for each run. It is assumed that each input/output buffer is long enough to hold one block of records. Empty buffers are stacked with AV pointing to the top buffer in this stack. The stack is a linked list. The following variables are made use of:

   IN (i) ... input buffers, 1   i    2k

  OUT (i) ... output buffers, 0    i    1

FRONT (i) ... pointer to first buffer in queue for run i, 1    i   k

  END (i) ... end of queue for i-th run, 1    i    k

 LINK (i) ... link field for i-th input buffer

              in a queue or for buffer in stack 1    i   2k

 LAST (i) ... value of key of last record read

              from run i, 1    i    k

       OU ... buffer currently used for output.

The algorithm also assumes that the end of each run has a sentinel record with a very large key, say + [image]. If block lengths and hence buffer lengths are chosen such that the time to merge one output buffer load equals the time to read a block then almost all input, output and computation will be carried out in parallel. It is also assumed that in the case of equal keys the k-way merge algorithm first outputs the record from the run with smallest index.

    procedure BUFFERING

1   for i <-- 1 to k do     //input a block from each run//

2     input first block of run i into IN(i)

3   end

4   while input not complete do end     //wait//

5   for i <-- 1 to k do     //initialize queues and free buffers//

6     FRONT(i) <-- END(i) <-- i

7     LAST(i) <-- last key in buffer IN(i)

8     LINK(k + i) <-- k + i + 1     //stack free buffer//

9   end

10  LINK(2k)  <-- 0; AV <-- k + 1; OU <-- 0

    //first queue exhausted is the one whose last key read is smallest//

11  find j such that LAST(j) =  min  {LAST(i)}

                               l  i  k

12  l <-- AV; AV <-- LINK(AV)      //get next free buffer//

13  if LAST(j)    +    then [begin to read next block for run j into

                               buffer IN(l)]

14  repeat      //KWAYMERGE merges records from the k buffers

                  FRONT(i) into output buffer OU until it is full.

                  If an input buffer becomes empty before OU is filled, the

                  next buffer in the queue for this run is used and the empty

                  buffer is stacked or last key = +   //

15    call KWAYMERGE

16    while input/output not complete do     //wait loop//

17    end

      if LAST(j)    +       then

18    [LINK(END(j)) <-- l; END(j) <-- l; LAST(j) <-- last key read

                                              //queue new block//

19    find j such that LAST(j) =  min {LAST(i)}

                                 l  i  k

20    l <-- AV; AV <-- LINK(AV)]     //get next free buffer//

21    last-key-merged <-- last key in OUT(OU)

22    if LAST(j)    +    then [begin to write OUT(OU) and read next block of

                             run j into IN(l)]

23                     else [begin to write OUT(OU)]

24    OU <-- 1 - OU

25    until last-key-merged = +

26    while output incomplete do     //wait loop//

27    end

28  end BUFFERING

Notes: 1) For large k, determination of the queue that will exhaust first can be made in log2k comparisons by setting up a selection tree for LAST(i), 1 [image] i [image] k, rather than making k - 1 comparisons each time a buffer load is to be read in. The change in computing time will not be significant, since this queue selection represents only a very small fraction of the total time taken by the algorithm.

2) For large k the algorithm KWAYMERGE uses a selection tree as discussed in section 8.2.1.

3) All input/output except for the initial k blocks that are read and the last block output is done concurrently with computing. Since after k runs have been merged we would probably begin to merge another set of k runs, the input for the next set can commence during the final merge stages of the present set of runs. I.e., when LAST(j) = + [image] we begin reading one by one the first blocks from each of the next set of k runs to be merged. In this case, over the entire sorting of a file, the only time that is not overlapped with the internal merging time is the time for the first k blocks of input and that for the last block of output.

4) The algorithm assumes that all blocks are of the same length. This may require inserting a few dummy records into the last block of each run following the sentinal record + [image].

Example 8.2: To illustrate the working of the above algorithm, let us trace through it while it performs a three-way merge on the following three runs:

Each run consists of four blocks of two records each; the last key in the fourth block of each of these three runs is + [image]. We have six input buffers IN(i), 1 [image] i [image] 6, and 2 output buffers OUT(0) and OUT(1). The diagram on page 404 shows the status of the input buffer queues, the run from which the next block is being read and the output buffer being ouput at the beginning of each iteration of the repeat-until of the buffering algorithm (lines 14-25).

image 404_a_a.gif not available.

image 404_a_b.gif not available.

From line 5 it is evident that during the k-way merge the test for "output buffer full?" should be carried out before the test "input buffer empty?" as the next input buffer for that run may not have been read in yet and so there would be no next buffer in that queue. In lines 3 and 4 all 6 input buffers are in use and the stack of free buffers is empty.

We end our discussion of buffer handling by proving that the algorithm BUFFERING works. This is stated formally in Theorem 8.1.

Theorem 8.1: The following is true for algorithm BUFFERING:

(i) There is always a buffer available in which to begin reading the next block; and

(ii) during the k-way merge the next block in the queue has been read in by the time it is needed.

Proof: (i) Each time we get to line 20 of the algorithm there are at most k + 1 buffer loads in memory, one of these being in an output buffer. For each queue there can be at most one buffer that is partially full. If no buffer is available for the next read, then the remaining k buffers must be full. This means that all the k partially full buffers are empty (as otherwise there will be more than k + 1 buffer loads in memory). From the way the merge is set up, only one buffer can be both unavailable and empty. This may happen only if the output buffer gets full exactly when one input buffer becomes empty. But k > 1 contradicts this. So, there is always at least one buffer available when line 20 is being executed.

(ii) Assume this is false. Let run Ri be the one whose queue becomes empty during the KWAYMERGE. We may assume that the last key merged was not the sentinel key + [image] since otherwise KWAYMERGE would terminate the search rather then get another buffer for Ri. This means that there are more blocks of records for run Ri on the input file and LAST(i) [image] + [image]. Consequently, up to this time whenever a block was output another was simultaneously read in (see line 22). Input/output therefore proceeded at the same rate and the number of available blocks of data is always k. An additional block is being read in but it does not get queued until line 18. Since the queue for Ri has become empty first, the selection rule for the next run to read from ensures that there is at most one block of records for each of the remaining k - 1 runs. Furthermore, the output buffer cannot be full at this time as this condition is tested for before the input buffer empty condition. Thus there are fewer than k blocks of data in memory. This contradicts our earlier assertion that there must be exactly k such blocks of data.

8.2.3 Run Generation

Using conventional internal sorting methods such as those of Chapter 7 it is possible to generate runs that are only as large as the number of records that can be held in internal memory at one time. Using a tree of losers it is possible to do better than this. In fact, the algorithm we shall present will on the average generate runs that are twice as long as obtainable by conventional methods. This algorithm was devised by Walters, Painter and Zalk. In addition to being capable of generating longer runs, this algorithm will allow for parallel input, output and internal processing. For almost all the internal sort methods discussed in Chapter 7, this parallelism is not possible. Heap sort is an exception to this. In describing the run generation algorithm, we shall not dwell too much upon the input/output buffering needed. It will be assumed that input/output buffers have been appropriately set up for maximum overlapping of input, output and internal processing. Wherever in the run generation algorithm there is an input/output instruction, it will be assumed that the operation takes place through the input/output buffers. We shall assume that there is enough space to construct a tree of losers for k records, R (i), 0 [image] i < k. This will require a loser tree with k nodes numbered 0 to k - 1. Each node, i, in this tree will have one field L (i). L(i), 1 [image] i < k represents the loser of the tournament played at node i. Node 0 represents the overall winner of the tournament. This node will not be explicitly present in the algorithm. Each of the k record positions R (i), has a run number field RN (i), 0 [image] i < k associated with it. This field will enable us to determine whether or not R (i) can be output as part of the run currently being generated. Whenever the tournament winner is output, a new record (if there is one) is input and the tournament replayed as discussed in section 8.2.1. Algorithm RUNS is simply an implementation of the loser tree strategy discussed earlier. The variables used in this algorithm have the following significance:

  R(i), 0    i < k ... the k records in the tournament tree

KEY(i), 0    i < k ... key value of record R(i)

  L(i), 0 < i < k ... loser of the tournament played at node i

RN(i), 0 <= i < k ... the run number to which R(i) belongs

               RC ... run number of current run

                Q ... overall tournament winner

               RQ ... run number for R(Q)

             RMAX ... number of runs that will be generated

        LAST__KEY ... key value of last record output

The loop of lines 5-25 repeatedly plays the tournament outputting records. The only interesting thing about this algorithm is the way in which the tree of losers is initialized. This is done in lines 1-4 by setting up a fictitious run numbered 0. Thus, we have RN(i) = 0 for each of the k records R(i). Since all but one of the records must be a loser exactly once, the initialization of L(i) [image] i sets up a loser tree with R(0) the winner. With this initialization the loop of lines 5-26 correctly sets up the loser tree for run 1. The test of line 10 suppresses the output of these k fictitious records making up run 0. The variable LAST__KEY is made use of in line 13 to determine whether or not the new record input, R(Q), can be output as part of the current run. If KEY(Q) < LAST__KEY then R(Q) cannot be output as part of the current run RC as a record with larger key value has already been output in this run. When the tree is being readjusted (lines 18-24), a record with lower run number wins over one with a higher run number. When run numbers are equal, the record with lower key value wins. This ensures that records come out of the tree in nondecreasing order of their run numbers. Within the same run, records come out of the tree in nondecreasing order of their key values . RMAX is used to terminate the algorithm. In line 11, when we run out of input, a record with run number RMAX + 1 is introduced. When this record is ready for output, the algorithm terminates in line 8. One may readily verify that when the input file is already sorted, only one run is generated. On the average, the run size is almost 2k. The time required to generate all the runs for an n run file is O(n log k) as it takes O(log k) time to adjust the loser tree each time a record is output. The algorithm may be speeded slightly by explicitly initializing the loser tree using the first k records of the input file rather than k fictitious records as in lines 1-4. In this case the conditional of line 10 may be removed as there will no longer be a need to suppress output of certain records.

8.3 SORTING WITH TAPES

Sorting on tapes is carried out using the same basic steps as sorting on disks. Sections of the input file are internally sorted into runs which are written out onto tape. These runs are repeatedly merged together

procedure RUNS

//generate runs using a tree of losers//

1     for i    1 to k - 1 do      //set up fictitious run 0 to initialize

tree//

2        RN(i)    0; L(i)    i; KEY(i)    0

3     end

4     Q    RQ    RC    RMAX    RN(0)    0; LAST__KEY

5     loop      //output runs//

6       if RQ    RC then [//end of run//

7                          if RC    0 then output end of run marker

8                          if RQ > RMAX then stop

9                                       else RC    RQ]

//output record R(Q) if not fictitious//

10       if RQ    0 then [output R(Q); LAST__KEY    KEY (Q)]

//input new record into tree//

11       if no more input then [RQ    RMAX + 1; RN (Q)    RQ]

12                        else [input a new record into R (Q)

13                              if KEY (Q) < LAST__KEY

14                                 then [//new record belongs to next

run//

15                                         RQ    RQ + 1

16                                         RN (Q)    RQ

RMAX    RQ]

17                                 else [RN (Q)    RC]]

//adjust losers//

18       T      (k + Q)/2              //T is parent of Q//

19       while T    0 do

20        if RN (L(T)) < RQ or (RN(L(T)) = RQ and KEY(L(T)) <

KEY(Q))

21         then [TEMP    Q; Q    L(T);    L(T)    TEMP

//T is the winner//

22               RQ    RN(Q)]

23        T      T/2        //move up tree//

24      end

25   forever

26 end RUNS

until there is only one run. The difference between disk and tape sorting lies essentially in the manner in which runs are maintained on the external storage media. In the case of disks, the seek and latency times are relatively insensitive to the specific location on the disk of the next block to be read with respect to the current position of the read/write heads. (The maximum seek and latency times were about 1/10 sec and 1/40 sec. respectively.) This is not true for tapes. Tapes are sequential access. If the read/write head is currently positioned at the front of the tape and the next block to be read is near the end of the tape (say ~ 2400 feet away), then the seek time, i.e., the time to advance the correct block to the read/write head, is almost 3 min 2 sec. (assuming a tape speed of 150 inches/second). This very high maximum seek time for tapes makes it essential that blocks on tape be arranged in such an order that they would be read in sequentially during the k-way merge of runs. Hence, while in the previous section we did not concern ourselves with the relative position of the blocks on disk, in this section, the distribution of blocks and runs will be our primary concern. We shall study two distribution schemes. Before doing this, let us look at an example to see the different factors involved in a tape sort.

Example 8.3: A file containing 4,500 records, R1, ..., R4500 is to be sorted using a computer with an internal memory large enough to hold only 800 records. The available external storage consists of 4 tapes, T1, T2, T3 and T4. Each block on the input tape contains 250 records. The input tape is not to be destroyed during the sort. Initially, this tape is on one of the 4 available tape drives and so has to be dismounted and replaced by a work tape after all input records have been read. Let us look at the various steps involved in sorting this file. In order to simplify the discussion, we shall assume that the initial run generation is carried out using a conventional internal sort such as quicksort. In this case 3 blocks of input can be read in at a time, sorted and output as a run. We shall also assume that no parallelism in input, output and CPU processing is possible. In this case we shall use only one output buffer and two input buffers. This requires only 750 record spaces.

trw = time to read or write on block of 250 records onto tape starting at present position of read/write head

trew = time to rewind tape over a length correponding to 1 block

ntm = time to merge n records from input buffers to output buffer using a 2-way merge

= delay cause by having to wait for T4 to be mounted in case we are ready to use T4 in step 4 before the completion of this tape mount.

The above computing time analysis assumes that no operations are carried out in parallel. The analysis could be carried out further as in the case of disks to show the dependence of the sort time on the number of passes made over the data.

8.3.1 Balanced Merge Sorts

Example 8.3 performed a 2-way merge of the runs. As in the case of disk sorting, the computing time here too depends essentially on the number of passes being made over the data. Use of a higher order merge results in a decrease in the number of passes being made without significantly changing the internal merge time. Hence, we would like to use as high an order merge as possible. In the case of disks the order of merge was limited essentially by the amount of internal memory available for use as input/output buffers. A k-way merge required the use of 2k + 2 buffers. While this is still true of tapes, another probably more severe restriction on the merge order k is the number of tapes available. In order to avoid excessive tape seek times, it is necessary that runs being merged together be on different tapes. Thus, a k-way merge requires at least k-tapes for use as input tapes during the merge.

In addition, another tape is needed for the output generated during this merge. Hence, at least k + 1 tapes must be available for a k-way tape merge (recall that in the case of a disk, one disk was enough for a k-way merge though two were needed to overlap input and output). Using k + 1 tapes to perform a k-way merge requires an additional pass over the output tape to redistribute the runs onto k-tapes for the next level of merges (see the merge tree of figure 8.8). This redistribution pass can be avoided through the use of 2k tapes. During the k-way merge, k of these tapes are used as input tapes and the remaining k as output tapes. At the next merge level, the role of input-output tapes is interchanged as in example 8.3, where in step 4, T1 and T2 are used as input tapes and T3 and T4 as output tapes while in step 6, T3 and T4 are the input tapes and T1 and T2 the output tapes (though there is no output for T2). These two approaches to a k-way merge are now examined. Algoritms M1 and M2 perform a k-way merge with the k + 1 tapes strategy while algorithm M3 performs a k-way merge using the 2k tapes strategy.

procedure M1

//Sort a file of records from a given input tape using a k-way

merge given tapes T1, ...,Tk+1 are available for the sort.//

1   Create runs from the input file distributing them evenly over

tapes T1, ...,Tk

2   Rewind T1, ...,Tk and also the input tape

3   if there is only 1 run then return      //sorted file on T1//

4   replace input tape by Tk+1

5   loop       //repeatedly merge onto Tk+1 and redistribute back

onto T1, ...,Tk//

6     merge runs from T1, ...,Tk onto Tk+1

7     rewind T1, ....,Tk+1

8     if number of runs on Tk+1 = 1 then return //output on

Tk+1//

9     evenly distribute from Tk+1 onto T1, ...,Tk

10     rewind T1, ...,Tk+1

11   forever

12  end M1

Analysis of Algorithm M1

To simplify the analysis we assume that the number of runs generated, m, is a power of k. Line 1 involves one pass over the entire file. One pass includes both reading and writing. In lines 5-11 the number of passes is logk m merge passes and logkm - 1 redistribution passes. So, the total number of passes being made over the file is 2logkm. If the time to rewind the entire input tape is trew, then the non-overlapping rewind time is roughly 2logkm trew.

A somewhat more clever way to tackle the problem is to rotate the output tape, i.e., tapes are used as output tapes in the cyclic order, k + 1,1,2, ...,k. When this is done, the redistribution from the output tape makes less than a full pass over the file. Algorithm M2 formally describes the process. For simplicity the analysis of M2 assumes that m is a power of k.

Analysis of Algorithm M2

The only difference between algorithms M2 and M1 is the redistributing

procedure M2

//same function as M1//

1   Create runs from the input tape, distributing them evenly over

tapes T1, ...,Tk.

2   rewind T1, ...,Tk and also the input tape

3   if there is only 1 run then return     //sorted file is on T1//

4   replace input tape by Tk+1

5   i    k + 1       //i is index of output tape//

6   loop

7     merge runs from the k tapes Tj, 1    j    k + 1 and j    i onto

Ti

8     rewind tapes T1, ...,Tk+1

9     if number of runs on Ti = 1 then return        //output on Ti//

10     evenly distribute (k - 1)/k of the runs on Ti onto tapes Ti, 1

j    k +1 and j    i and j    i mod(k + 1) + 1

11     rewind tapes Tj, 1    j    k + 1 and j    i

12     i    i mod(k + 1) + 1

13   forever

14 end M2

time. Once again the redistributing is done logkm - 1 times. m is the number of runs generated in line 1. But, now each redistributing pass reads and writes only (k - 1)/k of the file. Hence, the effective number of passes made over the data is (2 - 1/k)logkm + 1/k. For two-way merging on three tapes this means that algorithm M2 will make 3/2 logkm + 1/2 passes while M1 will make 2 logkm. If trew is the rewind time then the non-overlappable rewind time for M2 is at most (1 + 1/k) (logkm) trew + (1 - 1/k) trew as line 11 rewinds only 1/k of the file. Instead of distributing runs as in line 10 we could write the first m/k runs onto one tape, begin rewind, write the next m/k runs onto the second tape, begin rewind, etc. In this case we can begin filling input buffers for the next merge level (line 7) while some of the tapes are still rewinding. This is so because the first few tapes written on would have completed rewinding before the distribution phase is complete (for k > 2).

In case a k-way merge is made using 2k tapes, no redistribution is needed and so the number of passes being made over the file is only logkm + 1. This means that if we have 2k + 1 tapes available, then a 2k-way merge will make (2 - 1/(2k))log2km + 1/(2k) passes while a k-way merge utilizing only 2k tapes will make logkm + 1 passes. Table 8.13 compares the number of passes being made in the two methods for some values of k.

k  2k-way              k-way

1  3/2 log2 m + 1/2         --

2  7/8 log2 m + 1/6    log2 m + 1

3  1.124 log3 m + 1/6  log3 m + 1

4  1.25 log4 m + 1/8   log4 m + 1

Table 8.13 Number of passes using a 2k-way merge versus a k-way merge on 2k + 1 tapes

As is evident from the table, for k > 2 a k-way merge using only 2k tapes is better than a 2k-way merge using 2k + 1 tapes.

Algorithm M3 performs a k-way merge sort using 2k tapes.

procedure M3

//Sort a file of records from a given input tape using a k-way

merge on 2k tapes T1, ...,T2k//

1   Create runs from the input file distributing them evenly over tapes

T1, ...,Tk

2   rewind T1, ...,Tk; rewind the input tape; replace the input tape by

tape T2k; i    0

3   while total number of runs on Tik+1, ...,Tik+k > 1 do

4     j    1 - i

5     perform a k-way merge from Tik+1, ...,Tik+k evenly distributing

output runs onto Tjk+1, ...,Tjk+k.

6     rewind T1, ...,T2k

7     i    j  //switch input an output tapes//

8   end

//sorted file is on Tik+1//

9  end M3

Analysis of M3

To simplify the analysis assume that m is a power of k. In addition to the initial run creation pass, the algorithm makes logkm merge passes. Let trew be the time to rewind the entire input file. The time for line 2 is trew and if m is a power of k then the rewind of line 6 takes trew/k for each but the last iteration of the while loop. The last rewind takes time trew. The total rewind time is therefore bounded by (2 + (logkm - 1)/k)trew. Some of this may be overlapped using the strategy described in the analysis of algorithm M2 (exercise 4).

One should note that M1, M2 and M3 use the buffering algorithm of section 8.2.2 during the k-way merge. This merge itself, for large k, would be carried out using a selection tree as discussed earlier. In both cases the proper choice of buffer lengths and the merge order (restricted by the number of tape drives available) would result in an almost complete overlap of internal processing time with input/output time. At the end of each level of merge, processing will have to wait until the tapes rewind. Once again, this wait can be minimized if the run distribution strategy of exercise 4 is used.

8.3.2 Polyphase Merge

Balanced k-way merging is characterized by a balanced or even distribution of the runs to be merged onto k input tapes. One consequence of this is that 2k tapes are needed to avoid wasteful passes over the data during which runs are just redistributed onto tapes in preparation for the next merge pass. It is possible to avoid altogether these wasteful redistribution passes, when using fewer than 2k tapes and a k-way merge, by distributing the "right number" of runs onto different tapes. We shall see one way to do this for a k-way merge utilizing k + 1 tapes. To begin, let us see how m = 21 runs may be merged using a 2-way merge with 3 tapes T1, T2 and T3. Lengths of runs obtained during the merge will be represented in terms of the length of initial runs created by the internal sort. Thus, the internal sort creates 21 runs of length 1 (the length of initial runs is the unit of measure). The runs on a tape will be denoted by sn where s is the run size and n the number of runs of this size. If there are six runs of length 2 on a tape we shall denote this by 26. The sort is carried out in seven phases. In the first phase the input file is sorted to obtain 21 runs. Thirteen of these are written onto T1 and eight onto T2. In phase 2, 8 runs from T2 are merged with 8 runs from T1 to get 8 runs of size 2 onto T3. In the next phase the 5 runs of length 1 from T1 are merged with 5 runs of length 2 from T3 to obtain 5 runs of length 3 on T2. Table 8.14 shows the 7 phases involved in the sort.

                         Fraction of

                        Total Records

Phase   T1    T2    T3       Read

  1     113   18    --       1         initial distribution

  2     15    --    28       16/21     merge to T3

  3     --    35    23       15/21     merge to T2

  4     53    32    --       15/21     merge to T1

  5     51    --    82       16/21     merge to T3

  6     --    131   81       13/21     merge to T2

  7     211   --    --       1         merge to T1

Table 8.14 Polyphase Merge on 3 Tapes

Counting the number of passes made during each phase we get 1 + 16/21 + 15/21 + 15/21 + 16/21 + 13/21 + 1 = 5-4/7 as the total number of passes needed to merge 21 runs. If algorithm M2 of section 8.3.1 had been used with k = 2, then 3/2 [image] log221 [image] + 1/2 = 8 passes would have been made. Algorithm M3 using 4 tapes would have made [image] log221 [image] = 5 passes. What makes this process work? Examination of Table 8.14 shows that the trick is to distribute the runs initially in such a way that in all phases but the last only 1 tape becomes empty. In this case we can proceed to the next phase without redistribution of runs as we have 2 non-empty input tapes and one empty tape for output. To determine what the correct initial distribution is we work backwards from the last phase. Assume there are n phases. Then in phase n we wish to have exactly one run on T1 and no runs on T2 and T3. Hence, the sort will be complete at phase n. In order to achieve this, this run must be obtained by merging a run from T2 with a run from T3, and these must be the only runs on T2 and T3. Thus, in phase n - 1 we should have one run on each of T2 and T3. The run on T2 is obtained by merging a run from T1 with one from T3. Hence, in phase n - 2 we should have one run on T1 and 2 on T3.

Table 8.15 shows the distribution of runs needed in each phase so that merging can be carried out without any redistribution passes. Thus, if we had 987 runs, then a distribution of 377 runs onto T1 and 610 onto T3 at phase 1 would result in a 15 phase merge. At the end of the fifteenth phase the sorted file would be on T1. No redistribution passes would have been made. The number of runs needed for a n phase merge is readily seen to be Fn + Fn-1 where Fi is the i-th Fibonacci number (recall that F7 = 13 and F6 = 8 and that F15 = 610 and F14 = 377). For this reason this method of distributing runs is also known as Fibonacci merge. It can be shown that for three tapes this distribution of runs and resultant merge pattern requires only 1.04 log2 m + 0.99 passes over the data. This compares very favorably with the log2 m passes needed by algorithm M3 on 4 tapes using a balanced 2-way merge. The method can clearly be generalized to k-way merging on k + 1 tapes using generalized Fibonacci numbers. Table 8.16 gives the run distribution for 3-way merging on four tapes. In this case it can be shown that the number of passes over the data is about 0.703 log2 m + 0.96.

Phase    T1   T2   T3

  n       1    0    0

n - 1     0    1    1

n - 2     1    0    2

n - 3     3    2    0

n - 4     0    5    3

n - 5     5    0    8

n - 6    13    8    0

n - 7     0   21   13

n - 8    21    0   34

n - 9    55   34    0

n - 10    0   89   55

n - 11   89    0  144

n - 12  233  144    0

n - 13    0  377  233

n - 14  377    0  610

Table 8.15 Run Distribution for 3-Tape Polyphase Merge

Phase     T1    T2    T3    T4

  n        1     0     0     0

n - 1      0     1     1     1

n - 2      1     0     2     2

n - 3      3     2     0     4

n - 4      7     6     4     0

n - 5      0    13    11     7

n - 6     13     0    24    20

n - 7     37    24     0    44

n - 8     81    68    44     0

n - 9      0   149   125    81

n - 10   149     0   274   230

n - 11   423   274     0   504

n - 12   927   778   504     0

n - 13     0  1705  1431   927

n - 14  1705     0  3136  2632

Table 8.16 Polyphase Merge Pattern for 3-Way 4-Tape Merging

Example 8.4: The initial internal sort of a file of records creates 57 runs. 4 tapes are available to carry out the merging of these runs. The table below shows the status of the tapes using 3-way polyphase merge.

                                 Fraction of

                                Total Records

Phase    T1    T2    T3     T4      Read

  1      113   --    124     120     1          initial distribution

  2      --    313   111     17      39/57      merge onto T2

  3      57    36    14      --      35/57      merge onto T1

  4      53    32    --      94      36/57      merge onto T4

  5      51    --    172     92      34/57      merge onto T3

  6      --    311   171     91      31/57      merge onto T2

  7      571   --    --      --      1          merge onto T1

The total number of passes over the data is 1 + 39/57 + 35/57 + 36/57 + 34/57 + 31/57 + 1 = 5-4/57 compared to [log257] = 6 passes for 2-way balanced merging on 4 tapes.

Remarks on Polyphase Merging

Our discussion of polyphase merging has ignored altogether the rewind time involved. Before one can begin the merge for the next phase it is necessary to rewind the output tape. During this rewind the computer is essentially idle. It is possible to modify the polyphase merge scheme discussed here so that essentially all rewind time is overlapped with internal processing and read/write on other tapes. This modification requires at least 5 tapes and can be found in Knuth Vol. 3. Further, polyphase merging requires that the initial number of runs be a perfect Fibonacci number (or generalized Fibonacci number). In case this is not so, one can substitute dummy runs to obtain the required number of runs.

Several other ingenious merge schemes have been devised for tape sorts. Knuth, Vol. 3, contains an exhaustive study of these schemes.

8.3.3 Sorting with Fewer Than 3 Tapes

Both the balanced merge scheme of section 8.3.1 and the polyphase scheme of section 8.3.2 required at least 3 tapes to carry out the sort. These methods are readily seen to require only O(n log n) time where n is the number of records. We state without proof the following results for sorting on 1 and 2 tapes.

Theorem 8.2: Any one tape algorithm that sorts n records must take time [image] O(n2).

Theorem 8.3: n records can be sorted on 2 tapes in O(n log n) time if it is possible to perform an inplace rewrite of a record without destroying adjacent records. I.e. if record R2 in the sequence R1 R2 R3 can be rewritten by an equal size recod R'2 to obtain R1R'2R3.

REFERENCES

See the readings for chapter 7.

The algorithm for theorem 8.3 may be found in:

"A linear time two tape merge" by R. Floyd and A. Smith, Information Processing Letters, vol. 2, no. 5, December 1973, pp. 123-126.

EXERCISES

1. Write an algorithm to construct a tree of losers for records [image] with key values [image]. Let the tree nodes be [image] with [image] a pointer to the loser of a tournament and T0 a pointer to the overall winner. Show that this construction can be carried out in time O(k).

2. Write an algorithm, using a tree of losers, to carry out a k-way merge of k runs [image]. Show that if there are n records in the k runs together, then the computing time is O(n log2k).

3. a) n records are to be sorted on a computer with a memory capacity of S records (S << n). Assume that the entire S record capacity may be used for input/output buffers. The input is on disk and consists of m runs. Assume that each time a disk access in made the seek time is ts and the latency time is tl. The transmission time is tt per record transmitted. What is the total input time for phase II of external sorting if a k way merge is used with internal memory partitioned into I/O buffers so as to permit overlap of input, output and CPU processings as in algorithm BUFFERING?

b) Let the CPU time needed to merge all the runs together be tCPU (we may assume it is independent of k and hence constant). Let ts = 80 ms, tl = 20 ms, n = 200,000, m = 64, tt = 10-3 sec/record, S = 2000. Obtain a rough plot of the total input time, tinput, versus k. Will there always be a value of k for which tCPU [image] tinput?

4. a) Modify algorithm M3 using the run distribution strategy described in the analysis of algorithm M2.

b) et trw be the time to read/write a block and trew the time to rewind over one block length. If the initial run creation pass generates m runs for m a power of k, what is the time for a k-way merge using your algorithm. Compare this with the corresponding time for algorithm M2.

5. Obtain a table corresponding to Table 8.16 for the case of a 5-way polyphase merge on 6 tapes. Use this to obtain the correct initial distribution for 497 runs so that the sorted file will be on T1. How many passes are made over the data in achieving the sort? How many passes would have been made by a 5-way balanced merge sort on 6 tapes (algorithm M2)? How many passes would have been made by a 3-way balanced merge sort on 6 tapes (algorithm M3)?

6. In this exercise we shall investigate another run distribution strategy for sorting on tapes. Let us look at a 3-way merge on 4 tapes of 157 runs. These runs are initially distributed according to: 70 runs on T1, 56 on T2 and 31 on T3. The merging takes place according to the table below.

i.e., each phase consists of a 3-way merge followed by a two-way merge and in each phase almost all the initial runs are processed.

a) Show that the total number of passes made to sort 157 runs is 6-62/157.

b) Using an initial distribution from Table 8.16 show that Fibonacci merge on 4 tapes makes 6-59/193 passes when the number of runs is 193.

The distribution required for the process discussed above corresponds to the cascade numbers which are obtained as below for a k-way merge: Each phase (except the last) in a k-way cascade merge consists of a k-way merge followed by a k - 1 way merge followed by a k - 2 way merge . . . a 2-way merge. The last phase consists of only a k-way merge. The table below gives the cascade numbers corresponding to a k-way merge. Each row represents a starting configuration for a merge phase. If at the start of a phase we have the distribution n1, n2, ..., nk where ni > ni+ 1, 1 [image] i < k, then at the start of the previous phase we need [image] runs on the k input tapes respectively.

Initial Distribution for a k-way Cascade Merge

It can be shown that for a 4-way merge Cascade merge results in more passes over the data than a 4-way Fibonacci merge.

7. a) Generate the first 10 rows of the initial distribution table for a 5-way Cascade merge using 6 tapes (see exercise 6).

b) serve that 671 runs corresponds to a perfect 5-way Cascade distribution. How many passes are made in sorting the data using a 5-way Cascade merge?

c) How many passes are made by a 5-way Fibonacci merge starting with 497 runs and the distribution 120, 116, 108, 92, 61?

The number of passes is almost the same even though Cascade merge started with 35% more runs! In general, for [image] 6 tapes Cascade merge makes fewer passes over the data than does Fibonacci merge.

8. List the runs output by algorithm RUNS using the following input file and k = 4.

100,50, 18,60,2,70,30, 16,20, 19,99,55,20

Go to Chapter 9 Back to Table of Contents


Fundamentals: CHAPTER 9: SYMBOL TABLES

 Return to Table of Contents

CHAPTER 9: SYMBOL TABLES

The notion of a symbol table arises frequently in computer science. When building loaders, assemblers, compilers, or any keyword driven translator a symbol table is used. In these contexts a symbol table is a set of name-value pairs. Associated with each name in the table is an attribute, a collection of attributes, or some directions about what further processing is needed. The operations that one generally wants to perform on symbol tables are (i) ask if a particular name is already present, (ii) retrieve the attributes of that name, (iii) insert a new name and its value, and (iv) delete a name and its value. In some applications, one wishes to perform only the first three of these operations. However, in its fullest generality we wish to study the representation of a structure which allows these four operations to be performed efficiently.

Is a symbol table necessarily a set of names or can the same name appear more than once? If we are translating a language with block structure (such as ALGOL or PL/I) then the variable X may be declared at several levels. Each new declaration of .X implies that a new variable (with the same name) must be used. Thus we may also need a mechanism to determine which of several occurrences of the same name has been the most recently introduced.

These considerations lead us to a specificaton of the structure symbol table. A set of axioms are given on the next page.

Imagine the following set of declarations for a language with block structure:

begin

integer i,j;

.

.

.

begin

real x,i;

.

.

The representation of the symbol table for these declarations would look like S =

INSERT(INSERT(INSERT(INSERT(CREATE,i,integer),

j,integer),x,real),i,real)

Notice the identifer i which is declared twice with different attributes. Now suppose we apply FIND (S,i). By the axioms EQUAL(i,i) is tested and has the value true. Thus the value real is returned as a result. If the function DELETE(S,i) is applied then the result is the symbol table

INSERT(INSERT(INSERT(CREATE,i,integer),j,integer),x,real)

If we wanted to change the specification of DELETE so that it removes all occurrences of i then one axiom needs to be redefined, namely

DELETE(INSERT(S,a,r),b) :: =

if EQUAL(a,b) then DELETE(S,b)

else INSERT(DELETE(S,b),a,r)

structure SYMBOL__TABLE

declare CREATE( )    symtb

INSERT(symtb,name, value)    symtb

DELETE(symtb,name)    symtb

FlND(symtb,name)    value

HAS(symtb,name)    Boolean

ISMTST(symtb)    Boolean;

for all S    symtb, a,b    name, r    value let

ISMTST(CREATE) :: = true

ISMTST(INSERT(S,a,r)) :: = false

HAS(CREATE,a) :: = false

HAS(INSERT(S,a,r),b) :: =

if EQUAL(a,b) then true else HAS(S,b)

DELETE(CREATE,a) :: = CREATE

DELETE(INSERT(S,a,r),b) :: =

if EQUAL(a,b) then S

else INSERT(DELETE(S,b),a,r)

FIND(CREATE,a) :: = error

FIND(INSERT(S,a,r),b) :: = if EQUAL(a,b) then r

else FIND(S,b)

end

end SYMBOL__TABLE

The remaining sections of this chapter are organized around different ways to implement symbol tables. Different strategies are preferable given different assumptions. The first case considered is where the identifiers are known in advance and no deletions or insertions are allowed. Symbol tables with this property are called static. One solution is to sort the names and store them sequentially. Using either the binary search or Fibonacci search method of section 7.1 allows us to find any name in O(log2n) operations. If each name is to be searched for with equal probability then this solution, using an essentially balanced binary search tree, is optimal. When different identifiers are searched for with differing probabilities and these probabilities are known in advance this solution may not be the best. An elegant solution to this problem is given in section 9.1.

In section 9.2 we consider the use of binary trees when the identifiers are not known in advance. Sequential allocation is no longer desirable because of the need to insert new elements. The solution which is examined is AVL or height balanced trees. Finally in section 9.3 we examine the most practical of the techniques for dynamic search and insertion, hashing.

9.1 STATIC TREE TABLES

Definition: A binary search tree T is a binary tree; either it is empty or each node in the tree contains an identifier and:

(i) all identifiers in the left subtree of T are less (numerically or alphabetically) than the identifier in the root node T;

(ii) all identifiers in the right subtree of T are greater than the identifier in the root node T;

(iii) the left and right subtrees of T are also binary search trees.

For a given set of identifiers several binary search trees are possible. Figure 9.1 shows two possible binary search trees for a subset of the reserved words of SPARKS.

To determine whether an identifier X is present in a binary search tree, X is compared with the root. If X is less than the identifier in the root, then the search continues in the left subtree; if X equals the identifier in the root, the search terminates successfully; otherwise the search continues in the right subtree. This is formalized in algorithm SEARCH.

Figure 9.1 Two possible binary search trees

procedure SEARCH(T,X,i)

//search the binary search tree T for X. Each node has fields

LCHILD, IDENT, RCHILD. Return i = 0 if X is not in T.

Otherwise, return i such that IDENT(i) = X.//

1   i    T

2   while i    0 do

3     case

4     :X < IDENT(i): i    LCHILD(i)  //search left subtree//

5     :X = IDENT(i): return

6     :X > IDENT(i): i    RCHILD(i)  //search right subtree//

7     end

8   end

9 end SEARCH

In our study of binary search in chapter 7, we saw that every sorted file corresponded to a binary search tree. For instance, a binary search on the file (do, if, stop) corresponds to using algorithm SEARCH on the binary search tree of figure 9.2. While this tree is a full binary tree, it need not be optimal over all binary search trees for this file when the identifiers are searched for with different probabilities. In order to determine an optimal binary search tree for a given static file, we must first decide on a cost measure for search trees. In searching for an identifier at level k using algorithm SEARCH, k iterations of the while loop of lines 2-8 are made. Since this loop determines the cost of the search, it is reasonable to use the level number of a node as its cost.

Figure 9.2 Binary search tree corresponding to a binary search on the file (do,if,stop).

Consider the two search trees of figure 9.1 as possibly representing the symbol table of the SPARKS compiler. As names are encountered a match is searched for in the tree. The second binary tree requires at most three comparisons to decide whether there is a match. The first binary tree may require four comparisons, since any identifier which alphabetically comes after if but precedes repeat will test four nodes. Thus, as far as worst case search time is concerned, this makes the second binary tree more desirable than the first. To search for an identifier in the first tree takes one comparison for the if, two for each of for and while, three for repeat and four for loop. Assuming each is searched for with equal probability, the average number of comparisons for a successful search is 2.4. For the second binary search tree this amount is 2.2. Thus, the second tree has a better average behavior, too.

In evaluating binary search trees, it is useful to add a special "square" node at every place there is a null link. Doing this to the trees of figure 9.1 yields the trees of figure 9.3. Remember that every binary tree with n nodes has n + 1 null links and hence it will have n + 1 square nodes. We shall call these nodes external nodes because they are not part of the original tree. The remaining nodes will be called internal nodes. Each time a binary search tree is examined for an identifier which is not in the tree, the search terminates at an external node. Since all such searches represent unsuccessful searches, external nodes will also be referred to as failure nodes. A binary tree with external nodes added is an extended binary tree. Figure 9.3 shows the extended binary trees corresponding to the search trees of figure 9.1. We define the external path length of a binary tree to be the sum over all external nodes of the lengths of the paths from the root to those nodes. Analogously, the internal path length is defined to be the sum over all internal nodes of the lengths of the paths from the root to those nodes. For the tree of figure 9.3(a) we obtain its internal path length, I, to be:

I = 0 + 1 + 1 + 2 + 3 = 7

Its external path length, E, is:

E = 2 + 2 + 4 + 4 + 3 + 2 = 17.

Figure 9.3 Extended binary trees corresponding to search trees of figure 9.1.

Exercise 1 shows that the internal and external path lengths of a binary tree with n internal nodes are related by the formula E = I + 2n. Hence, binary trees with the maximum E also have maximum I. Over all binary trees with n internal nodes what are the maximum and minimum possible values for I? The worst case, clearly, is when the tree is skewed (i.e., when the tree has a depth of n). In this case,

To obtain trees with minimal I, we must have as many internal nodes as close to the root as possible. We can have at most 2 nodes at distance 1, 4 at distance 2, and in general the smallest value for I is

0 + 2   [image]  1 + 4   [image]  2 + 8   [image]  3 + ....

This can be more compactly written as

One tree with minimal internal path length is the complete binary tree defined in section 5.2.

Before attempting to use these ideas of internal and external path length is to obtain optimal binary search trees, let us look at a related but simpler problem. We are given a set of n + 1 positive weights q1, ...,qn + 1. Exactly one of these weights is to be associated with each of the n + 1 external nodes in a binary tree with n internal nodes. The weighted external path length of such a binary tree is defined to be   [image] 1  [image] i  [image] n + 1 qi ki where ki is the distance from the root node to the external node with weight qi. The problem is to determine a binary tree with minimal weighted external path length. Note that here no information is contained within internal nodes.

For example, suppose n = 3 and we are given the four weights: q1 = 15, q2 = 2, q3 = 4 and q4 = 5. Two possible trees would be:

Their respective weighted external path lengths are:

2   [image]  3 + 4   [image]  3 + 5   [image]  2 + 15   [image]  1 = 43

and

2   [image]  2 + 4   [image]  2 + 5   [image]  2 + 15   [image]  2 = 52.

Binary trees with minimal weighted external path length find application in several areas. One application is to determine an optimal merge pattern for n + 1 runs using a 2-way merge. If we have four runs R1-R4 with qi being the number of records in run Ri, 1   [image]  i   [image]  4, then the skewed binary tree above defines the following merge pattern: merge R2 and R3; merge the result of this with R4 and finally merge with R1. Since two runs with n and m records each can be merged in time O(n + m) (cf., section 7.5), the merge time following the pattern of the above skewed tree is proportional to (q2 + q3) + {(q2 + q3) + q4} + {q1 + q2 + q3 + q4}. This is just the weighted external path length of the tree. In general, if the external node for run Ri is at a distance ki from the root of the merge tree, then the cost of the merge will be proportional to   [image] qiki which is the weighted external path length.

Another application of binary trees with minimal external path length is to obtain an optimal set of codes for messages M1, ...,Mn + 1. Each code is a binary string which will be used for transmission of the corresponding message. At the receiving end the code will be decoded using a decode tree. A decode tree is a binary tree in which external nodes represent messages. The binary bits in the code word for a message determine the branching needed at each level of the decode tree to reach the correct external node. For example, if we interpret a zero as a left branch and a one as a right branch, then the decode tree corresponds to codes 000, 001, 01 and 1 for messages M1, M2, M3 and M4 respectively. These codes are called Huffman codes. The cost of decoding a code word is proportional to the number of bits in the code. This number is equal to the distance of the corresponding external node from the root node. If qi is the relative frequency with which message Mi will be transmitted, then the expected decode time is   [image] 1  [image] i  [image] n + 1 qidi where di is the distance of the external node for message Mi from the root node. The expected decode time is minimized by choosing code words resulting in a decode tree with minimal weighted external path length!

A very nice solution to the problem of finding a binary tree with minimum weighted external path length has been given by D. Huffman. We simply state his algorithm and leave the correctness proof as an exercise. The algorithm HUFFMAN makes use of a list L of extended binary trees. Each node in a tree has three fields: WEIGHT, LCHILD and RCHILD. Initially, all trees in L have only one node. For each tree this node is an external node, and its weight is one of the provided qi's. During the course of the algorithm, for any tree in L with root node T and depth greater than 1, WEIGHT(T) is the sum of weights of all external nodes in T. Algorithm HUFFMAN uses the subalgorithms LEAST and INSERT. LEAST determines a tree in L with minimum WEIGHT and removes it from L. INSERT adds a new tree to the list L.

procedure HUFFMAN(L,n)

//L is a list of n single node binary trees as described above//

for i    1 to n - 1 do            //loop n - 1 times//

call GETNODE(T)                    //create a new binary tree//

LCHILD(T)    LEAST(L)              //by combining the trees//

RCH1LD(T)    LEAST(L)              //with the two smallest weights//

WEIGHT(T)    WEIGHT(LCHILD(T))

+ WEIGHT(RCHILD(T))

call INSERT(L,T)

end

end HUFFMAN

We illustrate the way this algorithm works by an example. Suppose we have the weights q1 = 2, q2 = 3, q3 = 5, q4 = 7, q5 = 9 and q6 = 13. Then the sequence of trees we would get is: (the number in a circular node represents the sum of the weights of external nodes in that subtree).

The weighted external path length of this tree is

2   [image]  4 + 3   [image]  4 + 5   [image]  3 + 13   [image]  2 + 7   [image]  2 + 9   [image]  2 = 93

In comparison, the best complete binary tree has weighted path length 95.

Analysis of Algorithm HUFFMAN

The main loop is executed n - 1 times. If L is maintained as a heap (section 7.6) then each call to LEAST and INSERT requires only O(log n) time. Hence, the asymptotic computing time for the algorithm is O(n log n).

Let us now return to our original problem of representing a symbol table as a binary search tree. If the binary search tree contains the identifiers a1,a2, ...,an with a1 < a2 < ... < an and the probability of searching for each ai is pi, then the total cost of any binary search tree is   [image] 1  [image] i  [image] n pi. level (ai) when only successful searches are made. Since unsuccessful searches, i.e., searches for identifiers not in the table, will also be made, we should include the cost of these searches in our cost measure, too. Unsuccessful searches terminate with i = 0 in algorithm SEARCH. Every node with a null subtree defines a point at which such a termination can take place. Let us replace every null subtree by a failure node. The identifiers not in the binary search tree may be partitioned into n + 1 classes Ei, 0   [image] i   [image]  n. Eo contains all identifiers X such that X< a1. Ei contains all identifiers X such that ai < X < ai+1, 1   [image] i < n and En contains all identifiers X, X > an. It is easy to see that for all identifiers in a particular class Ei, the search terminates at the same failure node and it terminates at different failure nodes for identifiers in different classes. The failure nodes may be numbered 0 to n with i being the failure node for class Ei, 0   [image]  i   [image]  n. If qi is the probability that the identifier being searched for is in Ei, then the cost of the failure nodes is   [image] 0  [image] i  [image] n qi; (level (failure node i) - 1). The total cost of a binary search tree is therefore:

(9.1)

An optimal binary search tree for the identifier set a1, ...,an is one which minimizes eq. (9.1) over all possible binary search trees for this identifier set. Note that since all searches must terminate either successfully or unsuccessfully we have   [image] 1  [image] i  [image] n Pi +   [image] 0  [image] i  [image] n qi = 1.

Example 9.1: The possible binary search trees for the identifier set (a1,a2,a3) = (do, if, stop) are:

With equal probabilities pi = qj = 1/7 for all i and j we have:

cost (tree a) = 15/7; cost (tree b) = 13/7

cost (tree c) = 15/7; cost (tree d) = 15/7

cost (tree e) = 15/7.

As expected, tree b is optimal. With p1 = .5, p2 = .1, p3 = .05, q0 = .15, q1 = .1, q2 = .05 and q3 = .05 we have

cost (tree a) = 2.65; cost (tree b) = 1.9

cost (tree c) = 1.5;  cost (tree d) = 2.05

cost (tree e) = 1.6

Tree c is optimal with this assignment of p's and q's.

From among all the possible binary search trees how does one determine the optimal tree? One possibility would be to proceed as in example 9.1 and explicitly generate all possible binary search trees. The cost of each such tree could be computed and an optimal tree determined. The cost of each of the binary search trees can be determined in time O(n) for an n node tree. If N(n) is the total number of distinct binary search trees with n identifiers, then the complexity of the algorithm becomes O(n N(n)). From section 5.9 we know that N(n) grows too rapidly with increasing n to make this brute force algorithm of any practical significance. Instead we can find a fairly efficient algorithm by making some observations regarding the properties of optimal binary search trees.

Let a1 < a2 < ... < an be the n identifiers to be represented in a binary search tree. Let us denote by Tij an optimal binary search tree for ai+1, ...,aj, i < j. We shall adopt the convention that Tii is an empty tree for 0   [image]  i   [image]  n and that Tij is not defined for i > j. We shall denote by cij the cost of the search tree Tij. By definition cii will be 0. rij will denote the root of Tij and wij = qi +   [image] jk=i+1 (qk + pk) will denote the weight of Tij. By definition we will have rii = 0 and wii = qi, 0   [image]  i   [image]  n. An optimal binary search tree for a1, ...,an is therefore Ton, its cost is con, its weight won and its root ron.

If Tij is an optimal binary search tree for ai+1, ...,aj and rij = k, i < k   [image]  j, then Tij has two subtrees L and R. L is the left subtree and contains the identifiers ai+1, ...,ak-1 and R is the right subtree and contains the identifiers ak+1, ...,aj (figure 9.4). The cost cij of Tij is

ci,j = pk + cost (L) + cost (R) + wi,k-1+ wk,j

weight (L) = weight (Ti,k-1) = wi,k-1

weight (R) = weight (Tk,j) = wk,j

(9.2)

Figure 9.4 An optimal binary search tree Tij

From eq. (9.2) it is clear that if cij is to be minimal, then cost (L) = ci,k-1 and cost (R) = ck,j as otherwise we could replace either L or R by a subtree of lower cost thus getting a binary search tree for ai+1, ...,aj with lower cost than cij. This would violate the assumption that Tij was optimal. Hence, eq. (9.2) becomes:

ci,j = pk + wi,k-1 + wk,j + ci,k-1 + ck,j

= wi,j + ci,k-1 + ck,j

(9.3)

Since Tij is optimal it follows from eq. (9.3) that rij = k is such that

wij + ci,k-1 + ck,j = min {wij + ci,l-1 + cl,j}

i<l  j

or

ci,k-1 + ck,j = min {ci,1-1 + cl,j}

i<l  j

(9.4)

Equation (9.4) gives us a means of obtaining Ton and con starting from the knowledge that Tii =   [image]  and cii = 0.

Example 9.2: Let n = 4 and (a1, a2, a3, a4) = (do, if, read, while). Let (p1, p2, p3, p4) = (3,3,1,1) and (q0, q1, q2, q3, q4) = (2,3,1,1,1). The p's and q's have been multiplied by 16 for convenience. Initially, we have wi,i = qi, cii = 0 and rii = 0, 0   [image]  i   [image]  4. Using eqs. (9.3) and (9.4)we get:

w01 = p1 + w00 + w11 = p1 + q1 + w00 = 8

c01 = w01 + min {c00 + c11} = 8

r01 = 1

w12 = p2 + w11 + w22 = p2 + q2 + w11 = 7

c12 = w12 + min {c11 + c22} = 7

r12 = 2

w23 = p3 + w22 + w33 = p3 + q3 + w22 = 3

c23 = w23 + min {c22 + c33} = 3

r23 = 3

w34 = p4 + w33 + w44 = p4 + q4 + w33 = 3

c34 = w34 + min {c33 + c44} = 3

r34= 4

Knowing wi,i + 1 and ci,i + 1, 0   [image]  i   [image]  4 we can again use equations (9.3) and (9.4)to compute wi,i + 2, ci,i + 2, ri,i + 2, 0   [image]  i   [image]  3. This process may be repeated until w04, c04 and r04 are obtained. The table of figure 9.5 shows the results of this computation. From the table we see that c 04 = 32 is the minimal cost of a binary search tree for a1 to a4. The root of tree T04 is a2. Hence, the left subtree is T01 and the right subtree T24. T01 has root a1 and subtrees T00 and T11. T24 has root a3; its left subtree is therefore T22 and right subtree T34. Thus, with the data in the table it is possible to reconstruct T04. Figure 9.6 shows T04.

Figure 9.5 Computation of c04, w04 and r04. The computation is carried out row wise from row 0 to row 4.

Figure 9.6 Optimal search tree for example 9.2

The above example illustrates how equation (9.4) may be used to determine the c's and r's and also how to reconstruct Ton knowing the r's. Let us examine the complexity of this procedure to evaluate the c's and r's. The evaluation procedure described in the above example requires us to compute cij for (j - i) = 1,2, ...,n in that order. When j - i = m there are n - m + 1 cij's to compute. The computation of each of these cij's requires us to find the minimum of m quantities (see equation (9.4)). Hence, each such cij can be computed in time O(m). The total time for all cij's with j - i = m is therefore O(nm - m2). The total time to evaluate all the cij's and rij's is therefore

  1  [image] m  [image] n (nm - m2) = O(n3).

Actually we can do better than this using a result due to D. E. Knuth which states that the optimal l in equation (9.4) may be found by limiting the search to the range ri,j-1   [image]  l   [image]  ri+1,j. In this case the computing time becomes O(n2) (exercise 4). Algorithm OBST uses this result to obtain in O(n2) time the values of wij, rij and cij, 0   [image] i   [image]  j   [image]  n. The actual tree Ton may be constructed from the values of rij in O(n) time. The algorithm for this is left as an exercise.

procedure OBST(pi,qi,n)

//Given n distinct identifiers a1 < a2 < ... < an and probabilities pi,

1    i    n and qi, 0    i    n this algorithm computes the cost cij

of optimal binary search trees Tij for identifiers ai+1, ...,aj.

It also computes rij, the root of Tij. wij is the weight of Tij//

for i    0 to n - 1 do

(wii,rii,cii)    (qi,0,0)              //initialize//

(wi,i+1,ri,i+1,ci,i+1)    (qi + qi+1 + pi+1, i+1, qi+ qi+1 + pi+1)

//optimal trees with one node//

end

(wnn,rnn,cnn)    (qn,0,0)

for m    2 to n do     //find optimal trees with m nodes//

for i    0 to n - m do

j    i + m

wij    wi,j-1 + pj + qj

k    a value of l in the range ri,j-1    l    ri+1,j

that minimizes {ci,l-1 +cl,j}     //solve (9.4) using

Knuth's result//

cij    wij + ci,k-1 + ck,j        //eq. (9.3)//

rij    k

end

end

end OBST

9.2 DYNAMIC TREE TABLES

Dynamic tables may also be maintained as binary search trees. An identifier X may be inserted into a binary search tree T by using the search algorithm of the previous section to determine the failure node corresponding to X. This gives the position in T where the insertion is to be made. Figure 9.7 shows the binary search tree obtained by entering the months JANUARY to DECEMBER in that order into an initially empty binary search tree. Algorithm BST is a more formal rendition of the insertion process just described.

The maximum number of comparisons needed to search for any identifier in the tree of figure 9.7 is six for NOVEMBER. The average number of comparisons is (1 for JANUARY + 2 each for FEBRUARY and MARCH + 3 each for APRIL, JUNE and MAY + ... + 6 for NOVEMBER) / 12 = 42/ 12 = 3.5. If the months are entered in the order JULY, FEBRUARY, MAY, AUGUST, DECEMBER, MARCH, OCTOBER, APRIL, JANUARY, JUNE, SEPTEMBER and NOVEMBER then the tree of figure 9.8 is obtained. This tree is well balanced and does not have any paths to a node with a null link that are much longer than others. This is not true of the tree of figure 9.7 which has six nodes on the path from the root to NOVEMBER and only two nodes, JANUARY and FEBRUARY, on another path to a null link. Moreover, during the construction of the tree of figure 9.8 all intermediate trees obtained are also well balanced. The maximum number of identifier comparisons needed to find any identifier is now 4 and the average is 37/12   [image]  3.1. If instead the months are entered in lexicographic order, the tree degenerates to a chain as in figure 9.9. The maximum search time is now 12 identifier comparisons and the average is 6.5. Thus, in the worst case, binary search trees correspond to sequential searching in an ordered file. When the identifiers are entered in a random order, the tree tends to be balanced as in figure 9.8. If all permutations are equiprobable then it can be shown the average search and insertion time is O(log n) for an n node binary search tree.

procedure BST (X,T,j)

//search the binary search tree T for the node j such that IDENT(j)

= X. If X is not already in the table then it is entered at the

appropriate point. Each node has LCHILD, IDENT and RCHILD

fields//

p    0; j    T       //p will trail j through the tree//

while j    0 do

case

:X < IDENT(j): p    j; j    LCHILD(j)      //search left

sub-tree//

:X = IDENT(j): return

:X > IDENT(j): p    j; j    RCHILD(j)     //search right

sub-tree//

end

end

//X is not in the tree and can be entered as a child of p//

call GETNODE(j); IDENT(j)    X; LCHILD(j)    0;

RCHILD(j)    0

case

:T = 0: T    j   //insert into empty tree//

:X < IDENT(p): LCHILD(p)    j

:else: RCHILD(p)    j

end

end BST

Figure 9.7 Binary search tree obtained for the months of the year.

Figure 9.8 A balanced tree for the months of the year.

From our earlier study of binary trees, we know that both the average and maximum search time will be minimized if the binary search tree is maintained as a complete binary tree at all times. However, since we are dealing with a dynamic situation, identifiers are being searched for while the table is being built and so it is difficult to achieve this ideal without making the time required to add new entries very high. This is so because in some cases it would be necessary to restructure the whole tree to accommodate the new entry and at the same time have a complete binary search tree. It is, however, possible to keep the trees balanced so as to ensure both an average and worst case retrieval time of O(log n) for a tree with n nodes. We shall study one method of growing balanced binary trees. These balanced trees will have satisfactory search and insertion time properties.

Figure 9.9 Degenerate binary search tree

Height Balanced Binary Trees

Adelson-Velskii and Landis in 1962 introduced a binary tree structure that is balanced with respect to the heights of subtrees. As a result of the balanced nature of this type of tree, dynamic retrievals can be performed in O(log n) time if the tree has n nodes in it. At the same time a new identifier can be entered or deleted from such a tree in time O(log n). The resulting tree remains height balanced. The tree structure introduced by them is given the name AVL-tree. As with binary trees it is natural to define AVL trees recursively.

Definition: An empty tree is height balanced. If T is a nonempty binary tree with TL and TR as its left and right subtrees, then T is height balanced iff (i) TL and TR are height balanced and (ii) |hL - hR|   [image]  1 where hL and hR are the heights of TL and TR respectively.

The definition of a height balanced binary tree requires that every subtree also be height balanced. The binary tree of figure 9.7 is not height balanced since the height of the left subtree of the tree with root 'APRIL' is 0 while that of the right subtree is 2. The tree of figure 9.8 is height balanced while that of figure 9.9 is not. To illustrate the processes involved in maintaining a height balanced binary search tree, let us try to construct such a tree for the months of the year. This time let us assume that the insertions are made in the order MARCH, MAY, NOVEMBER, AUGUST, APRIL, JANUARY, DECEMBER, JULY, FEBRUARY, JUNE, OCTOBER and SEPTEMBER. Figure 9.10 shows the tree as it grows and the restructuring involved in keeping the tree balanced. The numbers within each node represent the difference in heights between the left and right subtrees of that node. This number is referred to as the balance factor of the node.

Definition: The balance factor, BF(T), of a node T in a binary tree is defined to be hL - hR where hL and hR are the heights of the left and right subtrees of T. For any node T in an AVL tree BF(T) = - 1, 0 or 1.

Figure 9.10 Balanced trees obtained for the months of the year.

Figure 9.10 (continued)

Figure 9.10 (continued)

Figure 9.10 (continued)

Figure 9.10 (continued)

Figure 9.10 (continued)

Inserting MARCH and MAY results in the binary search trees (i) and (ii) of figure 9.10. When NOVEMBER is inserted into the tree the height of the right subtree of MARCH becomes 2, while that of the left subtree is 0. The tree has become unbalanced. In order to rebalance the tree, a rotation is performed. MARCH is made the left child of MAY and MAY becomes the root. The introduction of AUGUST leaves the tree balanced. However, the next insertion, APRIL, causes the tree to become unbalanced again. To rebalance the tree, another rotation is performed. This time, it is a clockwise rotation. MARCH is made the right child of AUGUST and AUGUST becomes the root of the subtree (figure 9.10(v)). Note that both the previous rotationswere carried out with respect to the closest parent of the new node having a balance factor of   [image] 2. The insertion of JANUARY results in an unbalanced tree. This time, however, the rotation involved is somewhat more complex than in the earlier situations. The common point, however, is that it is still carried out with respect to the nearest parent of JANUARY with balance factor   [image] 2. MARCH becomes the new root. AUGUST together with its left subtree becomes the left subtree of MARCH. The left subtree of MARCH becomes the right subtree of AUGUST. MAY and its right subtree, which have identifiers greater than MARCH, become the right subtree of MARCH. If MARCH had a non-empty right subtree, this could have become the left subtree of MAY since all identifiers would have been less than MAY. Inserting DECEMBER and JULY necessitates no rebalancing. When FEBRUARYis inserted the tree again becomes unbalanced. The rebalancing process is very similar to that used when JANUARY was inserted. The nearest parent with balance factor   [image] 2 is AUGUST. DECEMBER becomes the new root of that subtree. AUGUST with its left subtree becomes the left subtree. JANUARY and its right subtree becomes the right subtree of DECEMBER, while FEBRUARY becomes the left subtree of JANUARY. If DECEMBER had had a left subtree, it would have become the right subtree of AUGUST. The insertion of JUNE requires the same rebalancing as in figure 9.10 (vi). The rebalancing following the insertion of OCTOBER is identical to that following the insertion of NOVEMBER. Inserting SEPTEMBER leaves the tree balanced.

In the preceding example we saw that the addition of a node to a balanced binary search tree could unbalance it. The rebalancing was carried out using essentially four different kinds of rotations LL, RR, LR and RL (figure 9.10 (v), (iii), (vi) and (ix) respectively). LL and RR are symmetric as are LR and RL. These rotations are characterized by the nearest ancestor, A, of the inserted node, Y, whose balance factor becomes   [image] 2. The following characterization of rotation types is obtained:

LL: new node Y is inserted in the left subtree of the left subtree of A

LR: Y is inserted in the right subtree of the left subtree of A

RR: Y is inserted in the right subtree of the right subtree of A

RL: Y is inserted in the left subtree of the right subtree of A

Figures 9.11 and 9.12 show these rotations in terms of abstract binary trees. The root node in each of the trees of the figures represents the nearest ancestor whose balance factor has become   [image] 2 as a result of the insertion. A moment's reflection will show that if a height balanced binary tree becomes unbalanced as a result of an insertion then these are the only four cases possible for rebalancing (if a moment's reflection doesn't convince you, then try exercise 6). In both the example of figure 9.10 and the rotations of figures 9.11 and 9.12, notice that the height of the subtree involved in the rotation is the same after rebalancing as it was before the insertion. This means that once the rebalancing has been carried out on the subtree in question, it is not necessary to examine the remaining tree. The only nodes whose balance factors can change are those in the subtree that is rotated.

In order to be able to carry out the rebalancing of figures 9.11 and 9.12 it is necessary to locate the node A around which the rotation is to be performed. As remarked earlier, this is the nearest ancestor of the newly inserted node whose balance factor becomes   [image] 2. In order for a node's balance factor to become   [image] 2, its balance factor must have been   [image] 1 before the insertion. Furthermore, the nearest ancestor whose balance factor becomes   [image] 2 is also the nearest ancestor with balance factor   [image] 1 before the insertion. Therefore, before the insertion, the balance factors of all nodes on the path from A to the new insertion point must have been 0. With this information, the node A is readily determined to be the nearest ancestor of the new node having a balance factor   [image] 1 before insertion. To complete the rotations the address of F, the parent of A, is also needed. The changes in the balance factors of the relevant nodes is shown in figures 9.11 and 9.12. Knowing F and A, all these changes can easily be carried out. What happens when the insertion of a node does not result in an unbalanced tree (figure 9.10 (i), (ii), (iv), (vii), (viii) and (xii))? While no restructuring of the tree is needed, the balance factors of several nodes change. Let A be the nearest ancestor of the new node with balance factor   [image] 1 before insertion. If as a result of the insertion the tree did not get unbalanced even though some path length increased by 1, it must be that the new balance factor of A is 0. In case there is no ancestor A with balance factor   [image] 1 (as in figure 9.10 (i), (ii), (iv), (vii) and (xii)), let A be the root. The balance factors of nodes from A to the parent of the new node will change to   [image] (see figure 9.10 (viii), A = JANUARY). Note that in both cases the procedure for determining A is the same as when rebalancing is needed. The remaining details of the insertion-rebalancing process are spelled out in algorithm AVL-INSERT.

Figure 9.11 Rebalancing rotations of type LL and RR

Figure 9.12 Rebalancing rotations of type LR and RL

Figure 9.12 (continued)

In order to really understand the insertion algorithm, the reader should try it out on the example of figure 9.10. Once you are convinced that it does keep the tree balanced, then the next question is how much time does it take to make an insertion? An analysis of the algorithm reveals that if h is the height of the tree before insertion, then the time to insert a new identifier is O(h). This is the same as for unbalanced binary search trees, though the overhead is significantly greater now. In the case of binary search trees, however, if there were n nodes in the tree, then h could in the worst case be n (figure 9.9) and the worst case insertion time was O(n). In the case of AVL trees, however, h can be at most O(log n) and so the worst case insertion time is O(log n). To see this, let Nh be the minimum number of nodes in a height balanced tree of height h. In the worst case, the height of one of the subtrees will be h - 1 and of the other h - 2. Both these subtrees are also height balanced. Hence, Nh = Nh-1 + Nh-2 + 1 and N0 = 0, N1 = 1 and N2 = 2. Note the similarity between this recursive definition for Nh and that for the Fibonacci numbers Fn = Fn-1 + Fn-2, F0 = 0 and F1 = 1. In fact, we can show (exercise 16) that Nh = Fh+2 - 1 for h   [image]  0. From Fibonacci number theory it is known that   [image]  where   [image] . Hence,   [image]. This means that if there are n nodes in the tree, then its height, h, is at most log (  [image]. The worst case insertion time for a height balanced tree with n nodes is, therefore, O(log n).

procedure AVL-INSERT(X,Y,T,)

//the identifier X is inserted into the AVL tree with root T. Each

node is assumed to have an identifier field IDENT, left and right

child fields LCHILD and RCHILD and a two bit balance factor

field BF. BF(P) = height of LCHILD(P) - height of RCHILD(P).

Y is set such that IDENT(Y) = X.//

//special case: empty tree T = 0.//

if T = 0 then [call GETNODE(Y); IDENT(Y)    X; T    Y;

BF(T)    0;

LCHILD(T)    0; RCHILD(T)    0; return]

//Phase 1: Locate insertion point for X. A keeps track of most recent

node with balance factor   1 and F is the parent of A. Q follows

P through the tree.//

F    0; A    T; P    T; Q    0

while P    0 do     //search T for insertion point for X//

if BF(P)    0 then [A    P; F    Q]

case

:X < IDENT(P): Q    P; P    LCHILD(P)     //take left

branch//

:X > IDENT(P): Q    P; P    RCHILD(P)     //take right

branch//

:else: Y    P; return     //X is in T//

end

end

//Phase 2: Insert and rebalance. X is not in T and may be inserted

as the appropriate child of Q//

call GETNODE(Y); IDENT(Y)    X; LCHILD(Y)    0;

RCHILD(Y)    0; BF(Y)    0

if X < IDENT(Q) then LCHILD(Q)    Y     //insert as left child//

else RCHILD(Q)    Y     //insert as right child//

//adjust balance factors of nodes on path from A to Q. Note that

by the definition of A, all nodes on this path must have balance

factors of 0 and so will change to   1. d = +1 implies X is inserted

in left subtree of A. d = -1 implies X is inserted in right subtree

of A//

if X > IDENT(A) then [P    RCHILD(A); B    P; d    -1]

else [P    LCHILD(A); B    P; d    +1]

while P    Y do

if X > IDENT(P)

then [BF(P)    -1; P    RCHILD(P)]     //heigth of right

increases by 1//

else [BF(P)    +1; P    LCHILD(P)]     //height of left

increases by 1//

end

//Is tree unbalanced?//

if BF(A) = 0 then [BF(A)    d; return     //tree still balanced//

if BF(A) + d = 0 then [BF(A)    0; return     //tree is balanced//

//tree unbalanced, determine rotation type//

if d = +1 then     //left imbalance//

case

:BF(B) = +1;     //rotation type LL//

LCHILD(A)    RCHILD(B); RCHILD(B)    A;

BF(A)    BF(B)    0

:else:     //type LR//

C    RCHILD(B)

RCHILD(B)    LCHILD(C)

LCHILD(A)    RCHILD(C)

LCHILD(C)    B

RCHILD(C)    A

case

:BF(C) = +1: BF(A)    -1; BF(B)    0     //LR(b)//

:BF(C) = -1: BF(B)    +1; BF(A)    0     //LR(c)//

:else:BF(B)    0; BF(A)    0;    //LR(a)//

end

BF(C)    0; B    C     //B is a new root//

end

else [// right imbalance; this is symmetric to left imbalance//

//and is left as an exercise//]

//subtree with root B has been rebalanced and is the new subtree//

//of F. The original subtree of F had root A

case

:F = 0 :T    B

:A = LCHILD(F) : LCHILD(F)    B

:A = RCHILD(F) : RCHILD(F)    B

end

end AVL__INSERT

Exercises 9-13 show that it is possible to find and delete a node with identifier X and to find and delete the kth node from a height balanced tree in O(log n) time. Results of an empirical study of deletion in height balanced trees may be found in the paper by Karlton, et.al. (see the references). Their study indicates that a random insertion requires no rebalancing, a rebalancing rotation of type LL or RR and a rebalancing rotation of type LR and RL with probabilities .5349, .2327 and .2324 respectively. Table 9.12 compares the worst case times of certain operations on sorted sequential lists, sorted linked lists and AVL-trees. Other suitable balanced tree schemes are studied in Chapter 10.

    Operation        Sequential List     Linked List       AVL-Tree

---------------------------------------------------------------------

Search for X           O(log n)         O(n)                 O(log n)

Search for kth item    O(1)             O(k)                 O(log n)

Delete X               O(n)             O(1)                 O(log n)

                                      (doubly linked list

                                      and position of X

                                      known)

Delete kth item        O(n - k)         O(k)                 O(log n)

Insert X               O(n)             O(1)                 O(log n)

                                      (if position for

                                      insertion is known)

Output in order        O(n)             O(n)                 O(n)

Table 9.12 Comparison of various structures

9.3 HASH TABLES

In tree tables, the search for an identifier key is carried out via a sequence of comparisons. Hashing differs from this in that the address or location of an identifier, X, is obtained by computing some arithmetic function, f, of X. f(X) gives the address of X in the table. This address will be referred to as the hash or home address of X. The memory available to maintain the symbol table is assumed to be sequential. This memory is referred to as the hash table, HT. The hash table is partitioned into b buckets, HT(0), ...,HT(b - 1). Each bucket is capable of holding s records. Thus, a bucket is said to consist of s slots, each slot being large enough to hold 1 record. Usually s = 1 and each bucket can hold exactly 1 record. A hashing function, f(X), is used to perform an identifier transformation on X. f(X) maps the set of possible identifiers onto the integers 0 through b - 1. If the identifiers were drawn from the set of all legal Fortran variable names then there would be T =   [image] 0  [image] i  [image] 5 26 x 36i >1.6 x 109 distinct possible values for X. Any reasonable program, however, would use far less than all of these identifiers. The ratio n / T is the identifier density, while   [image]  = n /(sb) is the loading density or loading factor. Since the number of identifiers, n, in use is usually several orders of magnitude less than the total number of possible identifiers, T, the number of buckets b, in the hash table is also much less then T. Therefore, the hash function f must map several different identifiers into the same bucket. Two identifiers I1, I2 are said to be synonyms with respect to f if f(I1) = f(I2). Distinct synonyms are entered into the same bucket so long as all the s slots in that bucket have not been used. An overflow is said to occur when a new identifier I is mapped or hashed by f into a full bucket. A collision occurs when two nonidentical identifiers are hashed into the same bucket. When the bucket size s is 1, collisions and overflows occur simultaneously.

As an example, let us consider the hash table HT with b = 26 buckets, each bucket having exactly two slots, i.e., s = 2. Assume that there are n = 10 distinct identifiers in the program and that each identifier begins with a letter. The loading factor,   [image], for this table is 10/52 = 0.19. The hash function f must map each of the possible identifiers into one of the numbers 1-26. If the internal binary representation for the letters A-Z corresponds to the numbers 1-26 respectively, then the function f defined by: f(X) = the first character of X; will hash all identifiers X into the hash table. The identifiers GA, D, A, G, L, A2, A1, A3, A4 and E will be hashed into buckets 7, 4, 1, 7, 12, 1, 1, 1, 1 and 5 respectively by this function. The identifiers A, A1, A2, A3 and A4 are synonyms. So also are G and GA. Figure 9.13 shows the identifiers GA, D, A, G, and A2 entered into the hash table. Note that GA and G are in the same bucket and each bucket has two slots. Similarly, the synonyms A and A2 are in the same bucket. The next identifier, A1, hashes into the bucket HT(1). This bucket is full and a search of the bucket indicates that A1 is not in the bucket. An overflow has now occurred. Where in the table should A1 be entered so that it may be retrieved when needed? We will look into overflow handling strategies in section 9.3.2. In the case where no overflows occur, the time required to enter or search for identifiers using hashing depends only on the time required to compute the hash function f and the time to search one bucket. Since the bucket size s is usually small (for internal tables s is usually 1) the search for an identifier within a bucket is carried out using sequential search. The time, then, is independent of n the number of identifiers in use. For tree tables, this time was, on the average, log n. The hash function in the above example is not very well suited for the use we have in mind because of the very large number of collisions and resulting overflows that occur. This is so because it is not unusual to find programs in which many of the variables begin with the same letter. Ideally, we would like to choose a function f which is both easy to compute and results in very few collisions. Since the ratio b/T is usually very small, it is impossible to avoid collisions altogether.

Figure 9.13 Has table with 26 buckets and two slots per bucket

In summary, hashing schemes perform an identifier transformation through the use of a hash function f. It is desirable to choose a function f which is easily computed and also minimizes the number of collisions. Since the size of the identifier space, T, is usually several orders of magnitude larger than the number of buckets b, and s is small, overflows necessarily occur. Hence a mechanism to handle overflows is also needed.

9.3.1 Hashing Functions

A hashing function, f, transforms an identifier X into a bucket address in the hash table. As mentioned earlier the desired properties of such a function are that it be easily computable and that it minimize the number of collisions. A function such as the one discussed earlier is not a very good choice for a hash function for symbol tables even though it is fairly easy to compute. The reason for this is that it depends only on the first character in the identifier. Since many programs use several identifiers with the same first letter, we expect several collisions to occur. In general, then, we would like the function to depend upon all the characters in the identifiers. In addition, we would like the hash function to be such that it does not result in a biased use of the hash table for random inputs. I.e., if X is an identifier chosen at random from the identifier space, then we want the probability that f(X) = i to be 1/b for all buckets i. Then a random X has an equal chance of hashing into any of the b buckets. A hash function satisfying this property will be termed a uniform hash function.

Several kinds of uniform hash functions are in use. We shall describe four of these.

(i) Mid-Square

One hash function that has found much use in symbol table applications is the 'middle of square' function. This function,   [image] m, is computed by squaring the identifier and then using an appropriate number of bits from the middle of the square to obtain the bucket address; the identifier is assumed to fit into one computer word. Since the middle bits of the square will usually depend upon all of the characters in the identifier, it is expected that different identifiers would result in different hash addresses with high probability even when some of the characters are the same. Figure 9.14 shows the bit configurations resulting from squaring some sample identifiers. The number of bits to be used to obtain the bucket address depends on the table size. If r bits are used, the range of values is 2r, so the size of hash tables is chosen to be a power of 2 when this kind of scheme is used.

IDENTIFIER             INTERNAL REPRESENTATION

X                   X                           X2

--------------------------------------------------

A                   1                            1

A1                134                        20420

A2                135                        20711

A3                136                        21204

A4                137                        21501

A9                144                        23420

B                   2                            4

C                   3                           11

G                   7                           61

DMAX          4150130               21526443617100

DMAX1       415013034             5264473522151420

AMAX          1150130                1345423617100

AMAX1       115013034             3454246522151420

Figure 9.14 Internal representations of X and X2 in octal notation. X is input right justified, zero filled, six bits or 2 octal digits per character.

(ii) Division

Another simple choice for a hash function is obtained by using the modulo (mod) operator. The identifier X is divided by some number M and the remainder is used as the hash address for X.

fD(X) = X mod M

This gives bucket addresses in the range 0 - (M - 1) and so the hash table is at least of size b = M. The choice of M is critical. If M is a power of 2, then fD(X) depends only on the least significant bits of X. For instance, if each character is represented by six bits and identifiers are stored right justified in a 60-bit word with leading bits filled with zeros (figure 9.15) then with M = 2i, i   [image]  6 the identifiers A1, B1, C1, X41, DNTXY1, etc. all have the same bucket address. With M = 2i, i   [image]  12 the identifiers AXY, BXY, WTXY, etc. have the same bucket address. Since programmers have a tendency to use many variables with the same suffix, the choice of M as a power of 2 would result in many collisions. This choice of M would have even more disastrous results if the identifier X is stored left justified zero filled. Then, all 1 character identifiers would map to the same bucket, 0, for M = 2i, i   [image]  54; all 2 character identifiers would map to the bucket 0 for M = 2i, i   [image]  48 etc. As a result of this observation, we see that when the division function fD is used as a hash function, the table size should not be a power of 2 while when the "middle of square" function fm is used the table size is a power of 2. If M is divisible by 2 then odd keys are mapped to odd buckets (as the remainder is odd) and even keys are mapped to even buckets. The use of the hash table is thus biased.

Figure 9.15 Identifier Al right and left justified and zero filled. (6 bits per character)

Let us try some other values for M and see what kind of identifiers get mapped to the same bucket. The goal being that we wish to avoid a choice of M that will lead to many collisions. This kind of an analysis is possible as we have some idea as to the relationships between different variable names programmers tend to use. For instance, the knowledge that a program tends to have variables with the same suffix led us to reject M = 2i. For similar reasons even values of M prove undesirable. Let X = x1x2 and Y = x2x1 be two identifiers each consisting of the characters x1 and x2. If the internal binary representation of x1 has value C(x1) and that for x2 has value C(x2) then if each character is represented by 6 bits, the numeric value of X is 26C(x1) + C(x2) while that for Y is 26C(x2) + C(x1). If p is a prime number dividing M then (fD(X) - fD(Y)) mod p = (26C(x1)mod p + C(x2)mod p - 26C(X2)mod p - C(x1)mod p) mod p. If p = 3, then

(fD(X) - fD (Y))mod p

= (64 mod 3 C(x1)mod 3 + C(x2)mod 3

- 64 mod 3 C(x2)mod 3 - C(x1)mod 3) mod 3

= C(x1)mod 3 + C(x2)mod 3 - C(x2)mod 3 - C(x1)mod 3

= 0 mod 3.

I.e., permutations of the same set of characters are hashed at a distance a factor of 3 apart. Programs in which many variables are permutations of each other would again result in a biased use of the table and hence result in many collisions. This happens because 64 mod 3 = 1. The same behavior can be expected when 7 divides M as 64 mod 7 = 1. These difficulties can be avoided by choosing M a prime number. Then, the only factors of M are M and 1. Knuth has shown that when M divides rk   [image]  a where k and a are small numbers and r is the radix of the character set (in the above example r = 64), then X mod M tends to be a simple superposition of the characters in X. Thus, a good choice for M would be: M a prime number such that M does not divide rk   [image]  a for small k and a. In section 9.3.2 we shall see other reasons for choosing M a prime number. In practice it has been observed that it is sufficient to choose M such that it has no prime divisors less than 20.

(iii) Folding

In this method the identifier X is partitioned into several parts, all but the last being of the same length. These parts are then added together to obtain the hash address for X. There are two ways of carrying out this addition. In the first, all but the last part are shifted so that the least significant bit of each part lines up with the corresponding bit of the last part (figure 9.16(a)). The different parts are now added together to get f(X). This method is known as shift folding. The other method of adding the parts is folding at the boundaries. In this method, the identifier is folded at the part boundaries and digits falling into the same position are added together (figure 9.16(b)) to obtain f(X).

Figure 9.16 Two methods of folding

(iv) Digit Analysis

This method is particularly useful in the case of a static file where all the identifiers in the table are known in advance. Each identifier X is interpreted as a number using some radix r. The same radix is used for all the identifiers in the table. Using this radix, the digits of each identifier are examined. Digits having the most skewed distributions are deleted. Enough digits are deleted so that the number of digits left is small enough to give an address in the range of the hash table. The criterion used to find the digits to be used as addresses, based on the measure of uniformity in the distribution of values in each digit, is to keep those digits having no abnormally high peaks or valleys and those having small standard deviation. The same digits are used for all identifiers.

Experimental results presented in section 9.3.2 suggest the use of the division method with a divisor M that has no prime factors less than 20 for general purpose applications.

9.3.2 Overflow Handling

In order to be able to detect collisions and overflows, it is necessary to initialize the hash table, HT, to represent the situation when all slots are empty. Assuming that no record has identifier zero, then all slots may be initialized to zero.* When a new identifier gets hashed into a full bucket, it is necessary to find another bucket for this identifier.The simplest solution would probably be to find the closest unfilled bucket. Let us illustrate this on a 26-bucket table with one slot per bucket. Assume the identifiers to be GA, D, A, G, L, A2, A1, A3, A4, Z, ZA, E. For simplicity we choose the hash function f(X) = first character of X. Initially, all the entries in the table are zero. f(GA) = 7, this bucket is empty, so GA and any other information making up the record are entered into HT(7). D and A get entered into the buckets HT(4) and HT(1) respectively. The next identifier G has f(G) = 7. This slot is already used by GA. The next vacant slot is HT(8) and so G is entered there. L enters HT(12). A2 collides with A at HT(1), the bucket overflows and A2 is entered at the next vacant slot HT(2). A1, A3 and A4 are entered at HT(3), HT(5) and HT(6) respectively. Z is entered at HT(26), ZA at HT(9), (the hash table is used circularly), and E collides with A3 at HT(5) and is eventually entered at HT(10). Figure 9.17 shows the resulting table. This method of resolving overflows is known as linear probing or linear open addressing.

*A clever way to avoid initializing the hash table has been discovered by T. Gonzalez (see exercise 22).

Figure 9.17 Hash table with linear probing. 26 buckets, 1 slot per bucket

In order to search the table for an identifier, X, it is necessary to first compute f(X) and then examine keys at positions HT(f(X)), HT(f(X) + 1), ...,HT(f(X) + j) such that HT(f(X) + j) either equals X (X is in the table) or 0 (X is not in the table) or we eventually return to HT(f(X)) (the table is full).

procedure LINSRCH (X,HT,b,j)

//search the hash table HT(0: b - 1) (each bucket has exactly 1

slot) using linear probing. If HT (j = 0 then the j-th bucket is empty

and X can be entered into the table. Otherwise HT (j) = X which

is already in the table. f is the hash function//

i    f(X); j    i         // compute hash address for X//

while HT(j)    X and HT(j)    0 do

j    (j + 1) mod b         //treat the table as circular//

if j = i then call TABLE_FULL                //no empty slots//

end

end LINSRCH

Our earlier example shows that when linear probing is used to resolve overflows, identifiers tend to cluster together, and moreover, adjacent clusters tend to coalesce, thus increasing the search time. To locate the identifier, ZA, in the table of figure 9.17, it is necessary to examine HT(26), HT(1), ...,HT(9), a total of ten comparisons. This is far worse than the worst case behavior for tree tables. If each of the identifiers in the table of figure 9.12 was retrieved exactly once, then the number of buckets examined would be 1 for A, 2 for A2, 3 for A1, 1 for D, 5 for A3, 6 for A4, 1 for GA, 2 for G, 10 for ZA, 6 for E, 1 for L and 1 for Z for a total of 39 buckets examined. The average number examined is 3.25 buckets per identifier. An analysis of the method shows that the expected average number of identifier comparisons, P, to look up an identifier is approximately (2 -   [image]) / (2 - 2  [image]) where   [image]  is the loading density. This is the average over all possible sets of identifiers yielding the given loading density and using a uniform function f. In the above example   [image]  = 12/26 = .47 and P = 1.5. Even though the average number of probes is small, the worst case can be quite large.

One of the problems with linear open addressing is that it tends to create clusters of identifiers. Moreover, these clusters tend to merge as more identifiers are entered, leading to bigger clusters. Some improvement in the growth of clusters and hence in the average number of probes needed for searching can be obtained by quadratic probing. Linear probing was characterized by searching the buckets (f(X) + i) mod b; 0   [image]  i   [image]  b - 1 where b is the number of buckets in the table. In quadratic probing, a quadratic function of i is used as the increment. In particular, the search is carried out by examining buckets f(X), (f(X) + i2) mod b and (f(X) - i2) mod b for 1   [image]  i   [image]  (b - 1)/2. When b is a prime number of the form 4j + 3, for j an integer, the quadratic search described above examines every bucket in the table. The proof that when b is of the form 4j + 3, quadratic probing examines all the buckets 0 to b - 1 relies on some results from number theory. We shall not go into the proof here. The interested reader should see Radke [1970] for a proof. Table 9.18 lists some primes of the form 4j + 3. Another possibility is to use a series of hash functions f1,f2, ...,fm. This method is known as rehashing. Buckets fi(X), 1   [image]  i   [image]  m are examined in that order. An alternate method for handling bucket overflow, random probing, is discussed in exercise 19.

  Prime  j  Prime    j

-------------------------

    3    0    43    10

    7    1    59    14

   11    2   127    31

   19    4   251    62

   23    5   503   125

   31    7  1019   254

Table 9.18 Some primes of the form 4j + 3

One of the reasons linear probing and its variations perform poorly is that searching for an identifier involves comparison of identifiers with different hash values. In the hash table of figure 9.17, for instance, searching for the identifier ZA involved comparisons with the buckets HT(1) to HT(8), even though none of the identifiers in these buckets had a collision with HT(26) and so could not possibly be ZA. Many of the comparisons being made could be saved if we maintained lists of identifiers, one list per bucket, each list containing all the synonyms for that bucket. If this were done, a search would then involve computing the hash address f(X) and examining only those identifiers in the list for f(X). Since the sizes of these lists is not known in advance, the best way to maintain them is as linked chains. Additional space for a link is required in each slot. Each chain will have a head node. The head node, however, will usually be much smaller than the other nodes since it has to retain only a link. Since the lists are to be accessed at random, the head nodes should be sequential. We assume they are numbered 1 to M if the hash function f has range 1 to M.

Figure 9.19. Hash chains corresponding to Figure 9.17.

Using chaining to resolve collisions and the hash function used to obtain figure 9.17, the hash chains of figure 9.19 are obtained. When a new identifier, X, is being inserted into a chain, the insertion can be made at either end. This is so because the address of the last node in the chain is known as a result of the search that determined X was not in the list for f(X). In the example of figure 9.19 new identifiers were inserted at the front of the chains. The number of probes needed to search for any of the identifiers is now 1 for each of A4, D, E, G, L, and ZA; 2 for each of A3, GA and Z; 3 for A1: 4 for A2 and 5 for A for a total of 24. The average is now 2 which is considerably less than for linear probing. Additional storage, however, is needed for links.

procedure CHNSRCH (X,HT,b,j)

//search the hash table HT (0:b - 1) for X. Either HT(i) = 0 or it

is a pointer to the list of identifiers X: f(X) = i. List nodes have

field IDENT and LINK. Either j points to the node already containing

X or j = 0//

j    HT (f(X))     //compute head node address//

//search the chain starting at j//

while j    0 and IDENT (j)    X do

j    LINK (j)

end

end CHNSRCH

The expected number of identifier comparison can be shown to be   [image]  where   [image]  is the loading density n / b (b = number of head nodes). For   [image]  = 0.5 this figure is 1.25 and for   [image]  = 1 it is about 1.5. This scheme has the additional advantage that only the b head nodes must be sequential and reserved at the beginning. Each head node, however, will be at most 1/2 to 1 word long. The other nodes will be much bigger and need be allocated only as needed. This could represent an overall reduction in space required for certain loading densities despite the links. If each record in the table is five words long, n = 100 and   [image]  = 0.5, then the hash table will be of size 200 x 5 = 1000 words. Only 500 of these are used as   [image]  = 0.5. On the other hand, if chaining is used with one full word per link, then 200 words are needed for the head nodes (b = 200). Each head node is one word long. One hundred nodes of six words each are needed for the records. The total space needed is thus 800 words, or 20% less than when no chaining was being used. Of course, when   [image]  is close to 1, chaining uses more space than linear probing. However, when   [image]  is close to 1, the average number of probes using linear probing or its variations becomes quite large and the additional space used for chaining can be justified by the reduction in the expected number of probes needed for retrieval. If one wishes to delete an entry from the table, then this can be done by just removing that node from its chain. The problem of deleting entries while using open addressing to resolve collisions is tackled in exercise 17.

Figure 9.20 Average number of bucket accesses per identifier retrieved (condensed from Lum, Yuen and Dodd, "Key-to-Address Transform Techniques: A Fundamental Performance Study on Large Existing Formatted Files," CACM, April 1971, Vol. 14, No. 4, pp. 228-239.)

The results of this section tend to imply that the performance of a hash table depends only on the method used to handle overflows and is independent of the hash function so long as a uniform hash function is being used. While this is true when the identifiers are selected at random from the identifier space, it is not true in practice. In practice, there is a tendency to make a biased use of identifiers. Many identifiers in use have a common suffix or prefix or are simple permutations of other identifiers. Hence, in practice we would expect different hash functions to result in different hash table performance. The table of figure 9.20 presents the results of an empirical study conducted by Lum, Yuen and Dodd. The values in each column give the average number of bucket accesses made in searching eight different tables with 33,575; 24,050; 4909; 3072; 2241; 930; 762 and 500 identifiers each. As expected, chaining outperforms linear open addressing as a method for overflow handling. In looking over the figures for the various hash functions, we see that division is generally superior to the other types of hash functions. For a general application, it is therefore recommended that the division method be used. The divisor should be a prime number, though it is sufficient to choose a divisor that has no prime factors less than 20. The table also gives the theoretical expected number of bucket accesses based on random keys.

9.3.3 THEORETICAL EVALUATION OF OVERFLOW TECHNIQUES

The experimental evaluation of hashing techniques indicates a very good performance over conventional techniques such as balanced trees. The worst case performance for hashing can, however, be very bad. In the worst case an insertion or a search in a hash table with n identifiers may take O(n) time. In this section we present a probabilistic analysis for the expected performance of the chaining method and state without proof the results of similar analyses for the other overflow handling methods. First, we formalize what we mean by expected performance.

Let HT(0: b - 1) be a hash table with b buckets, each bucket having one slot. Let f be a uniform hash function with range [0, b - 1]. If n identifiers X1,X2,...,Xn are entered into the hash table then there are bn distinct hash sequences f(X1), f(X2), ...,f(Xn). Assume that each of these is equally likely to occur. Let Sn denote the expected number of identifier comparisons needed to locate a randomly chosen Xi, 1   [image]  i   [image]  n. Then, Sn is the average number of comparisons needed to find the j'th key Xj; averaged over 1   [image]  j   [image]  n with each j equally likely and averaged over all bn hash sequences assuming each of these to also be equally likely. Let Un be the expected number of identifier comparisons when a search is made for an identifier not in the hash table. This hash table contains n identifiers. The quantity Un may be defined in a manner analogous to that used for Sn.

Theorem 9.1 Let   [image]  = n/b be the loading density of a hash table using a uniform hashing function f. Then:

(i)     for linear open addressing

(ii)     for rehashing, random probing and quadratic probing

(iii)     for chaining

Un

Sn   [image]  1 +   [image] /2

Proof Exact derivations of Un and Sn are fairly involved and can be found in Knuth's book: The Art of Computer Programming: Sorting and Searching. Here, we present a derivation of the approximate formulas for chaining. First, we must make clear our count for Un and Sn. In case the identifier X being searched for has   [image] (X)= i and chain i has k nodes on it (not including the head node) then k comparisons are needed if X is not on the chain. If X is j nodes away from the head node, 1   [image]  j   [image]  k then j comparisons are needed.

When the n identifiers distribute uniformly over the b possible chains, the expected number in each chain is n/b=  [image]. Since, Un = expected number of identifiers on a chain, we get Un =   [image].

When the i'th identifier, Xi, is being entered into the table, the expected number of identifiers on any chain is (i - 1)/b. Hence, the expected number of comparisons needed to search for Xi after all n identifiers have been entered is 1 + (i - 1)/b (this assumes that new entries will be made at the end of the chain). We therefore get:

REFERENCES

The O(n2) optimum binary search tree algorithm is from:

"Optimum Binary Search Trees" by D. Knuth, Acta Informatica, vol. 1, Fasc 1, 1971, pp. 14-25.

For a discussion of heuristics that obtain in O(nlogn) time nearly optimal binary search trees see:

"Nearly Optimal Binary Search Trees," by K. Melhorn, Acta Informatica, vol. 5, pp. 287-295, 1975.

"Binary Search Trees and File Organization," by J. Nievergelt, ACM Computing Surveys, vol. 6, no. 3, Sept. 1974, pp. 195-207.

For more on Huffman codes see: An Optimum Encoding with Minimum Longest Code and Total Number of Digits, by E. Schwartz, Information and Control, vol. 7, 1964, pp. 37-44.

Additional algorithms to manipulate AVL trees may be found in:

"Linear lists and priority queues as balanced binary trees" by C. Crane, STAN-CS-72-259, Computer Science Department, Stanford University, February 1972.

The art of computer programming: sorting and searching by D.Knuth, Addiso-Wesley, Reading, Massachusetts. 1973 (section 6.2.3).

Results of an empirical study on height balanced trees appear in:

"Performance of Height-Balanced Trees," by P. L. Karlton, S. H. Fuller, R. E. Scroggs and E. B. Koehler, CACM, vol. 19, no. 1, Jan. 1976, pp. 23-28.

Several interesting and enlightening works on hash tables exist. Some of these are:

"Scatter storage techniques" by R. Morris, CACM, vol. 11, no. 1, January 1968, pp. 38-44.

"Key to Address Transform Techniques: A Fundamental Performance Study on Large Existing Formatted Files" by V. Lum, P. Yuen and M. Dodd, CACM, vol. 14, no. 4, April 1971, pp. 228-239.

"The quadratic quotient method: a hash code eliminating secondary clustering" by J. Bell, CACM, vol. 13, no. 2, February 1970, pp. 107-109.

"Full table quadratic searching for scatter storage" by A. Day, CACM, vol. 13, no. 8, August 1970, pp. 481-482.

"Identifier search mechanisms: a survey and generalized model" by D. Severence, ACM Computing Surveys, vol. 6, no. 3, September 1974, pp. 175-194.

"A practitioners guide to addressing algorithms: a collection of reference tables and rules-of-thumb" by D. Severence and R. Duhne, Technical Report No. 240, Department of Operations Research, Cornell University, November 1974.

"Hash table methods" by W. Mauer and T. Lewis, ACM Computing Surveys, vol. 7, no. 1, March 1975, pp. 5-20.

"The quadratic hash method when the table size is not prime" by V. Batagelj, CACM, vol. 18, no. 4, April 1975, pp. 216-217.

The art of computer programming: sorting and searching by D. Knuth, Addison-Wesley, Reading, Massachusetts, 1973.

"Reducing the retrieval time of scatter storage techniques" by R. Brent, CACM, vol. 16, no. 2, February 1973, pp. 105-109.

"General performance analysis of key-to-address transformation methods using an abstract file concept" by V. Lum, CACM, vol. 16, no. 10, October 1973, pp. 603-612.

"The use of quadratic residue research," by C. E. Radke, CACM, vol. 13, no. 2, Feb. 1970, pp. 103-105.

A method to avoid hash table initialization may be found in:

"Algorithms for sets and related problems," by T. Gonzalez, University of Oklahoma, Nov. 1975.

EXERCISES

1.     (a) Prove by induction that if T is a binary tree with n internal nodes, I its internal path length and E its external path length, then E = I + 2n, n   [image]  0.

(b) Using the result of (a) show that the average number of comparisons s in a successful search is related to the average number of comparisons, u, in an unsuccessful search by the formula

s = (1 + 1/n)u - 1, n   [image]  1.

2.     (a) Show that algorithm HUFFMAN correctly generates a binary tree of minimal weighted external path length.

(b) When n runs are to be merged together using an m-way merge, Huffman's method generalizes to the following rule: "First add (1 - n) mod (m - 1) runs of length zero to the set of runs. Then repeatedly merge together the m shortest remaining runs until only one run is left." Show that this rule yields an optimal merge pattern for m-way merging.

3.     Using algorithm OBST compute wij, rij and cij, 0   [image]  i < j   [image]  4 for the identifier set (a1,a2,a3,a4) = (end, goto, print, stop) with p1 = 1/20, p2 = 1/5, p3= 1/10, p4= 1/20, qo= 1/5, q1 = 1/10, q2 = 1/5, q3 = 1/20, q4 = 1/20. Using the rij's construct the optimal binary search tree.

4.     (a) Show that the computing time of algorithm OBST is O(n2).

(b) Write an algorithm to construct the optimal binary search tree Ton given the roots rij, 0   [image]  i   [image]  j   [image]  n. Show that this can be done in time O(n).

5.     Since, often, only the approximate values of the p's and q's are known, it is perhaps just as meaningful to find a binary search tree that is nearly optimal i.e. its cost, eq. (9.1), is almost minimal for the given p's and q's. This exercise explores an O(nlogn) algorithm that results in nearly optimal binary search trees. The search tree heuristic we shall study is:

Choose the root Ak such that | w0,k - 1 - wk,n| is as small as possible. Repeat this procedure to find the left and right subtrees of Ak.

(a) Using this heuristic obtain the resulting binary search tree for the data of exercise 3. What is its cost?

(b) Write a SPARKS algorithm implementing the above heuristic. Your algorithm should have a time complexity of at most O(n log n).

An analysis of the performance of this heuristic may be found in the paper by Melhorn.

6.     (a) Convince yourself that Figures 9.11 and 9.12 take care of all the possible situations that may arise when a height balanced binary tree becomes unbalanced as a result of an insertion. Alternately come up with an example that is not covered by any of the cases in Figures 9.11 and 9.12.

(b) Complete Figure 9.12 by drawing the tree configurations for the rotations RL (a), (b) and (c).

7.     Complete algorithm AVL--INSERT by filling in the steps needed to rebalance the tree in case the imbalance was of type RL.

8.     Obtain the height balanced trees corresponding to those of Figure 9.10 using algorithm AVL--INSERT, starting with an empty tree, on the following sequence of insertions:

DECEMBER, JANUARY, APRIL, MARCH, JULY, AUGUST, OCTOBER, FEBRUARY, NOVEMBER, MAY, JUNE.

Label the rotations according to type.

9.     Assume that each node in an AVL tree T has the field LSIZE. For any node, A, LSIZE(A) is the number of nodes in its left subtree +1. Write an algorithm AVL--FINDK(T,k) to locate the kth smallest identifier in the subtree T. Show that this can be done in O(log n) time if there are n nodes in T.

10.     Rewrite algorithm AVL--INSERT with the added assumption that each node has a LSIZE field as in exercise 9. Show that the insertion time remains O(log n).

11.     Write an algorithm to list the nodes of an AVL-tree T in ascending order of IDENT fields. Show that this can be done in O(n) time if T has n nodes.

12.     Write an algorithm to delete the node with identifier X from an AVL-tree T. The resulting tree should be restructured if necessary. Show that the time required for this is O(log n) when there are n nodes in T.

13.     Do exercise 12 for the case when each node has a LSIZE field and the kth smallest identifier is to be deleted.

14.     Write an algorithm to merge the nodes of the two AVL-trees T1 and T2 together. What is the computing time of your algorithm?

15.     Write an algorithm to split an AVL tree, T, into two AVL trees T1 and T2 such that all identifiers in T1 are   [image]  X and all those in T2 are > X.

16.     Prove by induction that the minimum number of nodes in an AVL tree of height h is Nh = Fh+2 - 1, h   [image]  0.

17.     Write an algorithm to delete identifier X from a hash table which uses hash function f and linear open addressing to resolve collisions. Show that simply setting the slot previously occupied by X to 0 does not solve the problem. How must algorithm LINSRCH be modified so that a correct search is made in the situation when deletions are permitted? Where can a new identifier be inserted?

18.     (i) Show that if quadratic searching is carried out in the sequence (f(x) + q2), f(x) + (q - 1)2),...,(f(x) + 1),f(x),(f(x) - 1), ...,(f(x) - q2) with q = (b - 1)12 then the address difference mod b between successive buckets being examined is

b - 2, b - 4, b - 6, ...,5, 3, 1, 1, 3, 5, ...,b - 6, b - 4, b - 2

(ii) Write an algorithm to search a hash table HT (0,b - 1) of size b for the identifier X. Use f as the hash function and the quadratic probe scheme discussed in the text to resolve. In case X is not in the table, it is to be entered. Use the results of part (i) to reduce the computations.

19.     [Morris 1968] In random probing, the search for an identifier, X, in a hash table with b buckets is carried out by examining buckets f(x), f(x) + S(i), 1   [image]  i   [image]  b - 1 where S(i) is a pseudo random number. The random number generator must satisfy the property that every number from 1 to b - 1 must be generated exactly once. (i) Show that for a table of size 2r, the following sequence of computations generates numbers with this property:

Initialize R to 1 each time the search routine is called.

On successive calls for a random number do the following:

R   [image]  R * 5

R   [image]  low order r + 2 bits of R

S(i)   [image]  R/4

(ii) Write an algorithm, incorporating the above random number generator, to search and insert into a hash table using random probing and the middle of square hash function, fm.

It can be shown that for this method, the expected value for the average number of comparisons needed to search for X is - (1/  [image])log(1 -   [image]) for large table sizes.    is the loading factor.

20.     Write an algorithm to list all the identifiers in a hash table in lexicographic order. Assume the hash function f is f(X) = first character of X and linear probing is used. How much time does your algorithm take?

21.     Let the binary representation of identifier X be x1x2. Let | x | denote the number of bits in x and let the first bit of x1 be 1. Let | x1 | =   [image]  | x | /2   [image]  and | x2 | =   [image]  | x | /2   [image]. Consider the following hash function

f(X) = middle k bits of (x1 XOR x2)

where XOR is exclusive or. Is this a uniform hash function if identifiers are drawn at random from the space of allowable FORTRAN identifiers? What can you say about the behavior of this hash function in a real symbol table usage?

22.     [T. Gonzalez] Design a symbol table representation which allows one to search, insert and delete an identifier X in O(1) time. Assume that 1   [image]  X   [image]  m and that m + n units of space are available where n is the number of insertions to be made (Hint: use two arrays A(1:n) and B(1:m) where A(i) will be the ith identifier inserted into the table. If X is the ith identifier inserted then B(X) = i.). Write algorithms to search, insert and delete identifiers. Note that you cannot initialize either A or B to zero as this would take O(n + m) time. Note that X is an integer.

23.     [T. Gonzalez] Let S = {x1,x2, ...,xn} and T = {y1,y2, ...,yr} be two sets. Assume 1   [image]  xi   [image]  m, 1   [image]  i   [image]  n and 1   [image]  yi m, 1   [image]  i   [image]  r. Using the idea of exercise 22 write an algorithm to determine if S   [image]  T. Your algorithm should work in O(r + n) time. since S   [image]  T iff S   [image]  T and T   [image]  S, this implies that one can determine in linear time if two sets are equivalent. How much space is needed by your algorithm?

24.     [T. Gonzalez] Using the idea of exercise 22 write an O(n + m) time algorithm to carry out the function of algorithm VERIFY2 of section 7.1. How much space does your algorithm need?

25.     Complete table 9.12 by adding a column for hashing.

26.     For a fixed k, k   [image]  1 we define a height balanced tree HB(k) as below:

Definition An empty binary tree is a HB(k) tree. If T is a non-empty binary tree with TL and TR as its left and right subtrees, then T is HB(k) iff (i) TL and TR are HB(k) and (ii) | hL - hR |   [image]  k where hL and hR are the heights of TL and TR respectively.

For the case of HB(2) trees obtain the rebalancing transformations.

27.     Write an insertion algorithm for HB(2) trees.

28.     Using the notation of section 9.3.3 show that when linear open addressing is used:

Using this equation and the approximate equality:

29.     [Guttag] The following set of operations define a symbol table which handles a language with block structure. Give a set of axioms for these operations:

INIT--creates an empty table;

ENTERB--indicates a new block has been entered;

ADD--places an identifier and its attributes in the table;

LEAVEB--deletes all identifiers which are defined in the innermost block;

RETRIEVE--returns the attributes of the most recently defined identifier;

ISINB--returns true if the identifier is defined in the innermost block else false.

Go to Chapter 10     Back to Table of Contents


Fundamentals: CHAPTER 10: FILES

 Return to Table of Contents

CHAPTER 10: FILES

10.1 FILES, QUERIES AND SEQUENTIAL ORGANIZATIONS

A file, as described in earlier chapters, is a collection of records where each record consists of one or more fields. For example, the records in an employee file could contain these fields:

Employee Number (E#)

Name

Occupation

Degree (Highest Degree Obtained)

Sex

Location

Marital Status (MS)

Salary

Sample data for such a file is provided in figure 10.1.

Record  E#     Name    Occupation  Degree  Sex   Location    MS  Salary

-----------------------------------------------------------------------

  A     800  HAWKINS   programmer   B.S.    M   Los Angeles  S   10,000

  B     510  WILLIAMS  analyst      B.S.    F   Los Angeles  M   15,000

  C     950  FRAWLEY   analyst      M.S.    F   Minneapolis  S   12,000

  D     750  AUSTIN    programmer   B.S.    F   Los Angeles  S   12,000

  E     620  MESSER    programmer   B.S.    M   Minneapolis  M    9,000

Figure 10.1 Sample Data for Employee File

The primary objective of file organization is to provide a means for record retrieval and update. The update of a record could involve its deletion, changes in some of its fields or the insertion of an entirely new record. Certain fields in the record are designated as key fields. Records may be retrieved by specifying values for some or all of these keys. A combination of key values specified for retrieval will be termed a query. Let us assume that in the employee file of figure 10.1. the fields Employee Number, Occupation, Sex and Salary have been designated as key fields. Then, some of the valid queries to the file are:

Retrieve the records of all employees with

Q1: Sex = M

Q2: Salary > 9,000

Q3: Salary > average salary of all employees

Q4: (Sex = M and Occupation = Programmer) or (Employee Number > 700 and Sex = F)

One invalid query to the file would be to specify location = Los Angeles, as the location field was not designated as a key field. While it might appear that the functions one wishes to perform on a file are the same as those performed on a symbol table (chapter 9), several complications are introduced by the fact that the files we are considering in this chapter are too large to be held in internal memory. The tables of chapter 9 were small enough that all processing could be carried out without accessing external storage devices such as disks and tapes.

In this chapter we are concerned with obtaining data representations for files on external storage devices so that required functions (e.g. retrieval, update) may be carried out efficiently. The particular organization most suitable for any application will depend upon such factors as the kind of external storage device available, type of queries allowed, number of keys, mode of retrieval and mode of update. Let us elaborate on these factors.

Storage Device Types

We shall be concerned primarily with direct access storage devices, (DASD) as exemplified by disks. Some discussion of tape files will also be made.

Query Types

The examples Q1 - Q4 above typify the kinds of queries one may wish to make. The four query types are:

Q1: Simple Query: The value of a single key is specified.

Q2: Range Query: A range of values for a single key is specified

Q3: Functional Query: Some function of key values in the file is specified (e.g. average or median)

Q4: Boolean Query: A boolean combination of Q1 - Q3 using logical operators and, or, not.

Number of Keys

The chief distinction here will be between files having only one key and files with more than one key.

Mode of Retrieval

The mode of retrieval may be either real time or batched. In real time retrieval the response time for any query should be minimal (say a few seconds from the time the query is made). In a bank the accounts file has a mode of retrieval which is real time since requests to determine a balance must be satisfied quickly. Similarly, in an airline reservation system we must be able to determine the status of a flight (i.e. number of seats vacant) in a matter of a few seconds. In the batched mode of retrieval, the response time is not very significant. Requests for retrieval are batched together on a "transaction file" until either enough requests have been received or a suitable amount of time has passed. Then all queries on the transaction file are processed.

Mode of Update

The mode of update may, again, be either real time or batched. Real time update is needed, for example, in a reservation system. As soon as a seat on a flight is reserved, the file must be changed to indicate the new status. Batched update would be suitable in a bank account system where all deposits and withdrawals made on a particular day could be collected on a transaction file and the updates made at the end of the day. In the case of batched update one may consider two files: the Master File and the Transactions File. The Master File represents the file status after the previous update run. The transaction file holds all update requests that haven't yet been reflected in the master file. Thus, in the case of batched update, the master file is always 'out of date' to the extent that update requests have been batched on the transaction file. In the case of a bank file using real time retrieval and batched update, this would mean that only account balances at the end of the previous business day could be determined, since today's deposits and withdrawals haven't yet been incorporated into the master file.

The simplest situation is one in which there is only one key, the only queries allowed are of type Q1 (simple query), and all retrievals and updates are batched. For this situation tapes are an adequate storage medium. All the required functions can be carried out efficiently by maintaining the master file on a tape. The records in the file are ordered by the key field. Requests for retrieval and update are batched onto a transaction tape. When it is time to process the transactions, the transactions are sorted into order by the key field and an update process similar to algorithm VERIFY2 of section 7.1 is carried out, creating a new master file. All records in the old master file are examined, changed if necessary and then written out onto a new master file. The time required for the whole process is essentially O(n + m log m) where n and m are the number of records in the master and transaction files respectively (to be more accurate this has to be multiplied by the record length). This procedure is good only when the number of transactions that have been batched is reasonably large. If m = 1 and n = 106 then clearly it is very wasteful to process the entire master file. In the case of tapes, however, this is the best we can do since it is usually not possible to alter a record in the middle of a tape without destroying information in an adjacent record. The file organization described above for tape files will be referred to as: sequentially ordered.

In this organization, records are placed sequentially onto the storage media, (i.e., they occupy consecutive locations and in the case of a tape this would mean placing records adjacent to each other). In addition, the physical sequence of records is ordered on some key, called the primary key. For example, if the file of figure 10.1. were stored on a tape in the sequence A, B, C, D, E then we would have a sequential file. This file, however, is unordered. If the primary key is Employee Number then physical storage of the file in the sequence B, E, D, A, C would result in an ordered sequential file. For batched retrieval and update, ordered sequential files are preferred over unordered sequential files since they are easier to process (compare VERIFY2 with VERIFY1 of 7.1).

Sequential organization is also possible in the case of a DASD such as a disk. Even though disk storage is really two dimensional (cylinder x surface) it may be mapped down into a one dimensional memory using the technique of section 2.4. If the disk has c cylinders and s surfaces, one possibility would be to view disk memory sequentially as in figure 10.2. Using the notation tij to represent the j'th track of the i'th surface, the sequence is t1,1, t2,1, t3,1, ...,ts,1, t1,2, ...,ts,2 etc.

If each employee record in the file of figure 10.1 were one track long, then a possible sequential organization would store the records A,B,C,D,E onto tracks t3,4, t4,4, t5,4, t6,4 and t7,4 respectively (assuming c [image] 4 and s [image] 7). Using the interpretation of figure 10.2 the physical sequence in which the records have been stored is A,B,C,D,E. If the primary key is Employee Number then the logical sequence for the records is B,E,D,A,C as E#(B) < E#(E) < ... <E#(C). This would thus correspond to an unordered sequential file. In case the records are stored in the physical sequence B,E,D,A,C, then the file is ordered on the primary key, the logical and physical record sequences are the same, and the organization is that of a sequentially ordered file. Batched retrieval and update can be carried out essentially in the same way as for a sequentially ordered tape file by setting up input and output buffers and reading in, perhaps, one track of the master and transaction files at a time (the transaction file should be sorted on the primary key before beginning the master file processing). If updates do not change the size of records and no insertions are involved then the updated track may be written back onto the old master file. The sequential interpretation of figure 10.2 is particularly efficient for batched update

Figure 10.2 Interpreting Disk Memory as Sequential Memory

and retrieval as the tracks are to be accessed in the order: all tracks on cylinder 1 followed by all tracks on cylinder 2 etc. As a result of this the read/write heads are moved one cylinder at a time and this movement is necessitated only once for every s tracks read (s = number of surfaces). The alternative sequential interpretation (figure 10.3) would require accessing tracks in the order: all tracks on surface 1, all tracks on surface 2, etc. In this case a head movement would be necessitated for each track being read (except for tracks 1 and c).

Figure 10.3 Alternative Sequential Interpretation of Disk Memory

Since in batched update and retrieval the entire master file is scanned (typical files might contain 105 or more records), enough transactions must be batched for this to be cost effective. In the case of disks it is possible to extend this sequential ordered organization for use even in situations where the number of transactions batched is not enough to justify scanning the entire master file. First, take the case of a retrieval. If the records are of a fixed size then it is possible to use binary search to obtain the record with the desired key value. For a file containing n records, this would mean at most [log2 n] accesses would have to be made to retrieve the record. For a file with 105 records of length 300 characters this would mean a maximum of 17 accesses. On a disk with maximum seek time 1/10 sec, latency time 1/40 sec and a track density of 5000 characters/track this would mean a retrieval time of [image] arbitrary record from the same file stored on a tape with density 1600 bpi and a tape speed of 150 in/sec would in the worst case require 125 sec as the entire file must be read to access the last record (this does not include the time to cross interblock gaps and to restart tape motion etc. The actual time needed will be more than 125 sec).

When records are of variable size, binary search can no longer be used as given the address of the first and last records in a file one can no longer calculate the address of the middle record. The retrieval time can be considerably reduced by maintaining an index to guide the search. An index is just a collection of key value and address pairs. In the case of the file of figure 10.1 stored in the physical sequence B,E,D,A,C at addresses t1,2, t2,2, t3,2, t4,2 and t5,2 the index could contain five entries, one for each record in the file. The entries would be the pairs (510,t1,2) (620,t2,2) (750,t3,2) (800,t4,2) (900,t6,2). An index which contains one entry for every record in the file will be referred to as a dense index. If a dense index is maintained for the primary key then retrieval of a record with primary key = x could be carried out by first looking into the index and finding the pair (x,addr). The desired record would then be retrieved from the location addr. The total number of accesses needed to retrieve a record would now be one plus the number of accesses needed to locate the tuple (x,addr) in the index. In section 10.2 we shall look at efficient indexing techniques that allow index searches to be carried out using at most three accesses even for reasonably large files. This means that a retrieval from the file of 105 records discussed earlier could be carried out making at most four accesses rather than the seventeen accesses needed by a binary search. Since all addresses are kept in the index, it is not necessary to have fixed size records.

Sequential file organizations on a disk suffer from the same deficiencies as sequential organizatkons in internal memory. Insertion and deletion of records require moving large amounts of data in order to create space for the new record or to utilize the space used by the record being deleted. In practice these difficulties can be overcome to some extent by marking deleted records as having been deleted and not physically deleting the record. If a record is to be inserted, it can be placed into some "overflow" area rather than in its correct place in the sequentially ordered file. The entire file will have to be periodically reorganized. Reorganization will take place when a sufficient number of overflow records have accumulated and when many "deletions" have been made. While this method of handling insertions and deletions results in a degradation of system performance as far as further retrievals and updates is concerned, it is preferable to copying over large sections of the file each time an update is made. In situations where the update rate is very high or when the file size is too large to allow for periodic file reorganization (which would require processing the whole file), other organizations are called for. These will be studied in a later section.

So far, we have discussed only file organization when the number of keys is one. What happens when there is more than one key? A sequential file, clearly, can be ordered on only one key, the primary key. If an employee file is maintained on a disk using sequential organization with Employee Numbers as the primary key then how does one retrieve all records with occupation = programmer? One obvious, and inefficient, way would be to process the entire file, outputting all records that satisfy this query. Another, possibly more efficient way would be to maintain a dense index on the occupation field, search this index and retrieve the necessary records. In fact one could maintain a dense index on every key. This and other possibilities will be studied in section 10.3.

Let us summarize the ideas we have discussed. File organization is concerned with representing data records on external storage media. The choice of a representation depends on the environment in which the file is to operate, e.g., real time, batched, simple query, one key, multiple keys, etc. When there is only one key, the records may be sorted on this key and stored sequentially either on tape or disk. This results in a sequentially ordered file. This organization is good for files operating in batched retrieval and update mode when the number of transactions batched is large enough to make processing the entire file cost effective. When the number of keys is more than one or when real time responses are needed, a sequential organization in itself is not adequate. In a general situation several indexes may have to be maintained. In these cases, file organization breaks down into two more or less distinct aspects: (i) the directory (i.e. collection of indexes) and (ii) the physical organization of the records themselves. This will be referred to as the physical file. We have already discussed one possible physical organization i.e. sequential (ordered and unordered). In this general framework, processing a query or update request would proceed in two steps. First, the indexes would be interrogated to determine the parts of the physical file that are to be searched. Second, these parts of the physical file will be searched. Depending upon the kinds of indexes maintained, this second stage may involve only the accessing of records satisfying the query or may involve retrieving nonrelevant records too.

10.2 INDEX TECHNIQUES

One of the important components of a file is the directory. A directory is a collection of indexes. The directory may contain one index for every key or may contain an index for only some of the keys. Some of the indexes may be dense (i.e., contain an entry for every record) while others may be nondense (i.e., contain an entry for only some of the records). In some cases all the indexes may be integrated into one large index. Whatever the situation, the index may be thought of as a collection of pairs of the form (key value, address). If the records A,B,C,D,E of figure 10.1 are stored on disk addresses al,a2, ...,a5 respectively then an index for the key Employee Number would have entries (800,a1); (510,a2); (950,a3); (750,a4) and (620,a5). This index would be dense since it contains an entry for each of the records in the file. We shall assume that all the key values in an index are distinct. This may appear to be restrictive since several records may have the same key value as in the case of the Occupation key in figure 10.1. Records A,D,E all have the value 'programmer' for the Occupation key. This difficulty can be overcome easily by keeping in the address field of each distinct key value a pointer to another address where we shall maintain a list of addresses of all records having this value. If at address b1, we stored the list of addresses of all programmer records i.e. a1, a4 and a5, and at b2 we stored the address list for all analysts, i.e. a2 and a3 then we could achieve the effect of a dense index for Occupation by maintaining an index with entries ('programmer', b1) and ('analyst', b2). Another alternative would be to change the format of entries in an index to (key value, address 1. address 2, ...address n). In both cases, different entries will have distinct key values. The second alternative would require the use of variable size nodes. The use of variable size nodes calls for complex storage management schemes (see section 4.8), and so we would normally prefer the first alternative. An index, then, consists of pairs of the type (key value, address), the key values being distinct. The functions one wishes to perform on an index are: search for a key value; insert a new pair; delete a pair from the index; modify or update an existing entry. These functions are the same as those one had to perform on a dynamic table (section 9.2). An index differs from a table essentially in its size. While a table was small enough to fit into available internal memory, an index is too large for this and has to be maintained on external storage (say, a disk). As we shall see, the techniques for maintaining an index are rather different from those used in chapter 9 to maintain a table. The reason for this is the difference in the amount of time needed to access information from a disk and that needed to access information from internal memory. Accessing a word of information from internal memory takes typically about 10-8 seconds while accessing the same word from a disk could take about 10-1 seconds.

10.2.1 Cylinder-Surface Indexing

The simplest type of index organization is the cylinder-surface index. It is useful only for the primary key index of a sequentially ordered file. It assumes that the sequential interpretation of disk memory is that of figure 10.2 and that records are stored sequentially in increasing order of the primary key. The index consists of a cylinder index and several surface indexes. If the file requires c cylinders (1 through c) for storage then the cylinder index contains c entries. There is one entry corresponding to the largest key value in each cylinder. Figure 10.4 shows a sample file together with its cylinder index (figure 10.4(b) ). Associated with each of the c cylinders is a surface index. If the disk has s usable surfaces then each surface index has s entries. The i'th entry in the surface index for cylinder j, is the value of the largest key on the j'th track of the i'th surface. The total number of surface index entries is therefore c [image] s. Figure 10.4(c) shows the surface index for cylinder 5 of the file of figure 10.4(a). A search for a record with a particular key value X is carried out by first reading into memory the cylinder index. Since the number of cylinders in a disk is only

Figure 10.4(a) World War II aircraft

Figure 10.4(a). World War II aircraft

Abbreviations

Aus.  Australia       HF  heavy fighter

AB    attack bomber   LB  light bomber

Br    bomber          MB  medium bomber

C     cylinder        NB  night bomber

FB    fighter bomber  NF  night fighter

Fr    fighter         N   no

GB    Great Britain   S   surface

Ger.  Germany         y   yes

HB  heavy bomber

Figure 10.4(a) File of some World War II aircraft.

(Condensed from Airplanes by Enzo Angelucci, McGraw-Hill Book Co., New York, 1971.)

Cylinder/surface indexing is for a disk with four surfaces and assuming two records per track. The organization is that of a sequential file ordered on the field--Aircraft.

cylinder  highest key value

----------------------------------

    1     Bregeut 691

    2     Heinkel III H

    3     Junkers 188 E-l

    4     Messerschmitt Sturmvogel

    5     Spitfire Mk XVI

    6     Vengeance

Figure 10.4(b) Cylinder index for file of figure 10.4(a)

surface  highest key value

--------------------------

    1    Mitsubishi G 4M1

    2    Mosquito MkV1

    3    Nakajima B5N2

    4    Spitfire MkXVI

Figure 10.4(c) Surface index for cylinder 5.

a few hundred the cylinder index typically occupies only one track. The cylinder index is searched to determine which cylinder possibly contains the desired record. This search can be carried out using binary search in case each entry requires a fixed number of words. If this is not feasible, the cylinder index can consist of an array of pointers to the starting point of individual key values as in figure 10.5. In either case the search can be carried out in O(log c) time. Once the cylinder index has been searched and the appropriate cylinder determined, the surface index corresponding to that cylinder is retrieved from the disk.

Figure 10.5 Using an Array of Pointers to Permit Binary Search with Variable Length Key Values

The number of surfaces on a disk is usually very small (say 10) so that the best way to search a surface index would be to use a sequential search. Having determined which surface and cylinder is to be accessed, this track is read in and searched for the record with key X. In case the track contains only one record, the search is trivial. In the example file, a search for the record corresponding to the Japanese torpedo bomber Nakajima B5N2 which was used at Pearl Harbor proceeds as follows: the cylinder index is accessed and searched. It is now determined that the desired record is either in cylinder 5 or it is not in the file. The surface index to be retrieved is that for cylinder 5. A search of this index results in the information that the correct surface number is 3. Track t5,3 is now input and searched. The desired record is found on this track. The total number of disk accesses needed for retrieval is three (one to access the cylinder index, one for the surface index and one to get the track of records). When track sizes are very large it may not be feasible to read in the whole track. In this case the disk will usually be sector addressable and so an extra level of indexing will be needed: the sector index. In this case the number of accesses needed to retrieve a record will increase to four. When the file extends over several disks, a disk index is also maintained. This is still a great improvement over the seventeen accesses needed to make a binary search of the sequential file.

This method of maintaining a file and index is referred to as ISAM (Indexed Sequential Access Method). It is probably the most popular and simplest file organization in use for single key files. When the file contains more than one key, it is not possible to use this index organization for the remaining keys (though it can still be used for the key on which the records are sorted in case of a sequential file).

10.2.2 Hashed Indexes

The principles involved in maintaining hashed indexes are essentially the same as those discussed for hash tables in section 9.3. The same hash functions and overflow handling techniques are available. since the index is to be maintained on a disk and disk access times are generally several orders of magnitude larger than internal memory access times, much consideration must be given to hash table design and the choice of an overflow handling technique. Let us reconsider these two aspects of hash system design, giving special consideration to the fact that the hash table and overflow area will be on a disk.

Overflow Handling Techniques

The overflow handling techniques discussed in section 9.3.2 are:

(i) rehashing

(ii) open addressing (a) random

(b) quadratic

(c) linear

(iii) chaining.

The expected number of bucket accesses when s = 1 is roughly the same for methods (i), (iia) and (iib). Since the hash table is on a disk, and these overflow techniques tend to randomize the use of the hash table, we can expect each bucket access to take almost the maximum seek time. In the case of (iic), however, overflow buckets are adjacent to the home bucket and so their retrieval will require minimum seek time. While using chaining we can minimize the tendency to randomize use of the overflow area by designating certain tracks as overflow tracks for particular buckets. In this case successive buckets on a chain may be retrieved with little or no additional seek time. To the extent that this is possible, we may regard one bucket access to be equally expensive using methods (iic) and (iii). Bucket accesses using the other methods are more expensive, and since the average number of buckets retrieved isn't any better than (iii), we shall not discuss these further.

Hash Table

Let b, s, [image] and n be as defined in section 9.3. For a given [image] and n we have [image] = n/(bs), and so the product bs is determined. In the case of a hash table maintained in internal memory we chose the number of slots per bucket, s, to be 1. With this choice of s, the expected number of buckets accessed when open linear addressing is used is (2 - [image])/(2 - 2 [image]), and 1 + [image] /2 when chaining is used to resolve overflows. Since individual bucket accesses from a disk are expensive, we wish to explore the possibility of reducing the number of buckets accessed by increasing s. This would of necessity decrease b, the number of buckets, as bs is fixed. We shall assume that when chaining is used, the home bucket can be retrieved with one access. The hash table for chaining is similar to that for linear open addressing. Each bucket in the hash table, i.e. each home bucket, has s slots. Each such bucket also has a link field. This differs from the organization of section 9.3.2 where s = 0 for home buckets using chaining. Remaining buckets on individual chains have only one slot each and require additional accesses. Thus, if the key value X is in the i'th node on a chain (the home bucket being the 1st node in the chain), the number of accesses needed to retrieve X is i. In the case of linear open addressing if X is i buckets away from the home bucket, f(X), then the number of accesses to retrieve X is 1 + i.

By way of example, consider the hash function f(x) = first character of X and the values B, B1, B2, B3, A. Using a hash table with b = 6 and s = 1, the assignment of figure 10.6(a) is obtained if overflows are handled via linear open addressing. If each of the values is searched for once then the total number of bucket retrievals is 1(for A) + 1(for

Figure 10.6 Hash Tables With s = 1 and 2.

B) + 2(for B1) + 3(for B2) + 4(for B3) = 11. When the same table space is divided into 3 buckets each with two slots and f' (X) = [image] f (X)/2 [image] the assignment of key values to buckets is as in figure 10.6(b). The number of bucket retrievals needed now is 1 (for each of B and B1) + 2(for each of B2 and B3) + 3(for A) = 9. Thus the average number of buckets retrieved is reduced from 11/5 per search to 9/5 per search. The buckets of figure 10.6(b) are twice as big as those of figure 10.6(a). However, unless the bucket size becomes very large, the time to retrieve a bucket from disk will be dominated largely by the seek and latency time. Thus, the time to retrieve a bucket in each of the two cases discussed above would be approximately the same. Since the total average search time is made up of two components--first, the time, tr, to read in buckets from disk and second, the time, tp, to search each bucket in internal memory--we should choose b and s so as to minimize a (tr + tp) where a = average number of buckets retrieved. Using a sequential search within buckets, the time tp is proportional to s. Thus the average tp for figure 10.6(b) is one and a half times that for figure 10.6(a). When s is small and the hash table is on a disk, we have tr >> tp and it is sufficient to minimize a [image] tr. Let us contrast this to the situation in section 9.3 where tr and tp are comparable. The table of figure 10.6(a) can be searched with an average of 11/5 key comparisons and accesses. The table of figure 10.6(b) requires 1 comparison to find B, 2 for B1, 3 for B2, 4 for B3 and 5 for A for a total of 15 comparisons. This implies an average of 3 comparisons per search. In this case the search time has actually increased with the increase in s. But, when the table is on disk, the average times are roughly 11tr/5 and 9tr/5, and the table with s = 2 gives better performance. In general, we can conclude that increasing s while maintaining b [image] s fixed reduces a (see figure 10.7).

Figure 10.7 =.5

The table of figure 10.8 shows the results of some experiments conducted on existing hashed files maintained on disk. As is evident from this table, for a fixed [image], the average number of bucket accesses decreases with increasing s. Moreover, for s [image] 10, the average number of accesses for open linear addressing is roughly the same as for chaining. This together with our earlier observation that unless care is taken while using chaining, successive accesses will access random parts of the disk, while in the case of open linear addressing consecutive disk segments would be accessed, leads us to the conclusion that with suitable [image] and s linear addressing will outperform chaining (contrast this with the case of internal tables).

While both the data of figures 10.7 and 10.8 might suggest a choice for b = 1 (thus maximizing the value of s), such a choice for s clearly is not best. The optimal value of s will depend very much on the values of the latency time, seek time and transmission rates of the disk drive in use. We may rewrite the retrieval time per bucket, tr, as ts + tl + s [image] tt where ts, tl and tt are the seek time, latency time and transmission time per slot respectively. When the seek time is very large relative to tt the optimal s would tend to be larger than when ts is very small compared to tt (as in the case of a drum). Another consideration is the size of the input buffer available. Since the bucket has to be read into some part of internal memory, we must set aside enough space to accomodate a bucket load of data.

D = division

M = middle of square

C = chaining

L = linear open addressing

= loading factor = (number of entries)/(number of slots in hash table)

Note For the same [image] C uses more space than L as overflows are handled in a separate area.

Figure 10.8 Observed average number of bucket accesses for different and s. (Condensed from Lum, Yuen and Dodd: Key to Address Transform techniques: A fundamental performance study on large existing formatted files. CACM, Vol. 14, No. 4, Apr, 1974.)

Loading Order

As in the case of internal tables, the order in which key values are entered into the hash table is important. To the extent possible, an attempt should be made to enter these values in order of nonincreasing frequency of search. When this is done, new entries should be added to the end of the overflow chain rather than at the front.

10.2.3 Tree Indexing--B-Trees

The AVL tree of section 9.2 provided a means to search, insert and delete entries from a table of size n using at most O(log n) time. Since these same functions are to be carried out in an index, one could conceivably use AVL trees for this application too. The AVL tree would itself reside on a disk. If nodes are retrieved from the disk, one at a time, then a search of an index with n entries would require at most 1.4 log n disk accesses (the maximum depth of an AVL tree is 1.4 log n). For an index with a million entries, this would mean about 23 accesses in the worst case. This is a lot worse than the cylinder sector index scheme of section 10.2.1. In fact, we can do much better than 23 accesses by using a balanced tree based upon an m-way search tree rather than one based on a binary search tree (AVL trees are balanced binary search trees).

Definition: An m-way search tree, T, is a tree in which all nodes are of degree [image] m. If T is empty, (i.e., T = 0) then T is an m-way search tree. When T is not empty it has the following properties:

(i) T is a node of the type

n, Ao, (K1,A1), (K2,A2), ..., (Kn,An)

where the Ai, 0 [image] i [image] n are pointers to the subtrees of T and the Ki, 1 [image] i [image] n are key values; and 1 [image] n < m.

(ii) Ki < Ki+1, 1 [image] i < n

(iii) All key values in the subtree Ai are less than the key value Ki+1, 0 [image] i < n

(iv) All key values in the subtree An are greater than Kn.

(v) The subtrees Ai, 0 [image] i [image] n are also m-way search trees.

As an example of a 3-way search tree consider the tree of figure 10.9 for key values 10, 15, 20, 25, 30, 35, 40, 45 and 50. One may easily verify that it satisfies all the requirements of a 3-way search

        schematic

node    format

------------------------

  a     2,b,(20,c),(40,d)

  b     2,0,(10,0),(15,0)

  c     2,0,(25,0),(30,e)

  d     2,0,(45,0),(50,0)

  e     1,0,(35,0)

Figure 10.9 Example of a 3-way search tree.

tree. In order to search for any key value X in this tree, we first "look into" the root node T = a and determine the value of i for which Ki [image] X < Ki+1 (for convenience we use K0 = [- [image]] and Kn+1 = [+ [image] [image]] where [- [image]] is smaller than all legal key values and [+ [image]] is larger than all legal key values). In case X = Ki then the search is complete. If X [image] Ki then by the definition of an m-way search tree X must be in subtree Ai if it is in the tree. When n (the number of keys in a node) is "large," the search for the appropriate value of i above may be carried out using binary search. For "small" n a sequential search is more appropriate. In the example if X = 35 then a search in the root node indicates that the appropriate subtree to be searched is the one with root A1 = c. A search of this root node indicates that the next node to search is e. The key value 35 is found in this node and the search terminates. If this search tree resides on a disk then the search for X = 35 would require accessing the nodes a, c and e for a total of 3 disk accesses. This is the maximum number of accesses needed for a search in the tree of figure 10.9. The best binary search tree for this set of key values requires 4 disk accesses in the worst case. One such binary tree is shown in figure 10.10.

Algorithm MSEARCH searches an m-way search tree T for key value X using the scheme described above. In practice, when the search tree

Figure 10.10 Best AVL-tree for data of figure 10.9

represents an index, the tuples (Ki,Ai) in each of the nodes will really be 3-tuples (Ki,Ai,Bi) where Bi is the address in the file of the record with key Ki. This address would consist of the cylinder and surface numbers of the track to be accessed to retrieve this record (for some disks this address may also include the sector number). The Ai, 0 [image] i [image] n are the addresses of root nodes of subtrees. Since these nodes are also on a disk, the Ai are cylinder and surface numbers.

procedure MSEARCH(T,X)

//Search the m-way search tree T residing on disk for the key

value X. Individual node format is n,A0, (K1,A1), ...,(Kn,An),

n < m. A triple (P,i,j) is returned. j = 1 implies X is found

at node P, key Ki. Else j = 0 and P is the node into which

X can be inserted//

1    P    T; K0    [-  ]; Q    0         //Q is the parent of P//

2    while P    0 do

3      input node P from disk

4       Let P define n, A0,(K1,A1), ..., (Kn,An)

5       Kn+1    [+  ]

6       Let i be such that Ki    X < Ki+1

7       if X = Ki then [//X has been found// return (P,i,1)]

8       Q    P; P    Ai

9     end

//X not in T; return node into which insertion can take place//

10    return (Q,i,0)

11  end MSEARCH

Analyzing algorithm MSEARCH is fairly straightforward. The maximum number of disk accesses made is equal to the height of the tree T. since individual disk accesses are very expensive relative to the time needed to process a node (i.e. determine the next node to access, lines 4-8) we are concerned with minimizing the number of accesses needed to carry out a search. This is equivalent to minimizing the height of the search tree. In a tree of degree m and height h [image] 1 the maximum number of nodes is [image] 0 [image] i [image] h-1 mi = (mh - 1)/(m - 1). Since each node has at most m - 1 keys, the maximum number of entries in an m-way tree index of height h would be mh - 1. For a binary tree with h = 3 this figure is 7. For a 200-way tree with h = 3 we have mh - 1 = 8 X 106 - 1

Clearly, the potentials of high order search trees are much greater than those of low order search trees. To achieve a performance close to that of the best m-way search trees for a given number of entries n, it is necessary that the search tree be balanced. The particular variety of balanced m-way search trees we shall consider here is known as a B-tree. In defining a B-tree it is convenient to reintroduce the concept of failure nodes as used for optimal binary search trees in section 9.1. A failure node represents a node which can be reached during a search only if the value X being searched for is not in the tree. Every subtree with root = 0 is a point that is reached during the search iff X is not in the tree. For convenience, these empty subtrees will be replaced by hypothetical nodes called failure nodes. These nodes will be drawn square and marked with an F. The actual tree structure does not contain any such nodes but only the value 0 where such a node occurs. Figure 10.11 shows the 3-way search tree of figure 10.9 with failure nodes. Failure nodes are the only nodes that have no children.

Definition: A B-tree, T, of order m is an m-way search tree that is either empty or is of height [image] 1 and satisfies the following properties:

(i) the root node has at least 2 children

(ii) all nodes other than the root node and failure nodes have at least [image] m/2 [image] children

(iii) all failure nodes are at the same level.

The 3-way search tree of figure 10.11 is not a B-tree since it has failure nodes at levels 3 and 4. This violates requirement (iii). One possible B-tree of order 3 for the data of figure 10.9 is shown in figure 10.12. Notice that all nonfailure nodes are either of degree 2 or 3. In fact, for a B-tree of order 3, requirements (i), (ii) and the definition

Figure 10.11 Search tree of figure 10.9 with failure nodes shown.

Figure 10.12 B-tree of order 3 for data of figure 10.9.

of a m-way search tree together imply that all nonfailure nodes must be of degree 2 or 3. For this reason, B-trees of order 3 are also known as 2-3 trees.

While the total number of nonfailure nodes in a B-tree of a given order may be greater than the number of such nodes in the best possible search tree of that order (the 2-3 tree of figure 10.12 has 7 nodes while the 3-way search tree of figure 10.9 has only 5), we shall see later that it is easier to insert and delete nodes into a B-tree retaining the B-tree properties than it is to maintain the best possible m-way search tree at all times. Thus, the reasons for using B-trees rather than optimal m-way search trees for indexes, are the same as those for using AVL-trees as opposed to optimal binary search trees when maintaining dynamic internal tables.

Number of Key Values in a B-Tree

If T is a B-tree of order m in which all failure nodes are at level l + 1 then we know that the maximum number of index entries in T is ml - 1. What is the minimum number, N, of entries in T? From the definition of a B-tree we know that if l > 1, the root node jas at least 2 children. Hence, there are at least two nodes at level 2. Each of these nodes must have at least [image] m/2 [image] children. Thus there are at least 2 [image] m/2 [image] nodes at level 3. At level 4 there must be at least 2 [image] m/2 [image] 2 nodes, and continuing this argument, we see that there are at least 2 [image] m/2 [image] l-2 nodes at level l when l > 1. All of these nodes are nonfailure nodes. If the key values in the tree are K1,K2, ...,KN and Ki < Ki+1, 1 [image] i<N then the number of failure nodes is N + 1. This is so because failures occur for Ki < X < Ki+1, 0 [image] i [image] N and K0 = [- [image]], KN+1 = [+ [image]]. This results in N + 1 different nodes that one could reach while searching for a key value X not in T. Therefore, we have,

N + 1 = number of failure nodes in T

= number of nodes at level l + 1

   2  m/2   l - 1

and so,            N     2  m/2   l - 1 - 1, l    1

This in turn implies that if there are N key value in a B-tree of order m then all nonfailure nodes are at levels less than or equal to l, l [image] log[m/2] {(N + 1)/2} + 1. The maximum number of accesses that have to be made for a search is l. Using a B-tree of order m = 200, an index with N [image] 2 X 106 - 2 will have l [image] log100 {(N+ 1)/2} + l. Since l is integer, we obtain l [image] 3. For n [image] 2 X 108 - 2 we get l [image] 4. Thus, the use of a high order B-tree results in a tree index that can be searched making a very small number of disk accesses even when the number of entries is very large.

Choice of m

B-trees of high order are desirable since they result in a reduction in the number of disk accesses needed to search an index. If the index has N entries then a B-tree of order m = N + 1 would have only one level. This choice of m clearly is not reasonable, since by assumption the index is too large to fit in internal memory. Consequently, the single node representing the index cannot be read into memory and processed. In arriving at a reasonable choice for m, we must keep in mind that we are really interested in minimizing the total amount of time needed to search the B-tree for a value X. This time has two components. One, the time for reading in a node from the disk and two, the time needed to search this node for X. Let us assume that each node of a B-tree of order m is of a fixed size and is large enough to accomodate n, A0 and m - 1 values for (Ki,Ai,Bi,), 1 [image] i < m. If the Ki are at most [image] characters long and the Ai and Bi each [image] characters long, then the size of a node is approximately m( [image] + 2 [image]) characters. The time, ti, required to read in a node is therefore:

ti = ts + tl+ m(   + 2  ) tc

= a + bm

where   a = ts + tl = seek time + latency time

b = (   + 2  )tc and tc = transmission time per character

If binary search is used to carry out the search of line 6 of algorithm MSEARCH then the internal processing time per node is clog2m + d for some constants c and d. The total processing time per node is thus,

= a + bm + c log2 m + d

(10.1)

For an index with N entries, the number of levels, l, is bounded by:

The maximum search time is therefore given by (10.2).

(10.2)

We therefore desire a value of m that minimizes (10.2). Assuming that the disk drive available has a ts = 1/100 sec and tl = 1/40 sec we get a = 0.035 sec. Since d will typically be a few microseconds, we may ignore it in comparison with a. Hence, a + d [image] a = 0.035 sec. Assuming each key value is at most 6 characters long and that each Ai and Bi is 3 characters long, [image] = 6 and [image] = 3. If the transmission rate tc = 5 x 10-6 sec/character (corresponding to a track capacity of 5000 characters), b = ( [image] + 2 [image])tc = 6 X 10-5 sec. The right hand side of equation (10.2) evaluates to

(10.3)

Plotting this function gives us the graph of figure 10.13. From this plot it is evident that there is a wide range of values of m for which nearly optimal performance is achieved. This corresponds to the almost flat region m [image] [50,400]. In case the lowest value of m in this region results in a node size greater than the allowable capacity of an input buffer, the value of m will be determined by the buffer size.

Figure 10.13 Plot of (35 + .06m)/log2 m

Insertion

In addition to searching an index for a particular key value X, we also wish to insert and delete entries. We shall now focus attention on the problem of inserting a new key value X into a B-tree. The insertion is to be carried out in such a way that the tree obtained following the insertion is also a B-tree. As we shall see, the algorithm for doing this is conceptually much simpler than the corresponding insertion algorithm for AVL-trees.

In attempting to discover the operations needed to carry out the insertion, let us insert X = 38 into the 2-3 tree of figure 10.12. First, a search of the tree is carried out to determine where the 38 is to be inserted. The failure node that is reached during this search for X= 38 is the fourth from the right. Its parent node is f. The node f contains only one key value and thus has space for another. The 38 may therefore be entered here, and the tree of figure 10.14(a) is obtained. In making this insertion, the tree nodes a, c and f were accessed from the disk during the search. In addition the new node f had to be written back onto the disk. The total number of accesses is therefore four. Next, let us insert X = 55. A search into the B-tree of figure 10.14(a) indicates that this key value should go into the node g. There is no space in this node, since it already contains m - 1 key values. Symbolically inserting the new key value at its appropriate position in a node that previously had m - 1 key values would yield a node, P, with the following format

m,A0,(K1,A1) ...,(Km,Am)

and       Ki < Ki+1, 1    i < m.

This node may be split into two nodes P and P' with the following formats

node P:   m/2   - 1, A0,(K1,A1), ...,(K  m/2  -1, A  m/2  -1)

(10.4)

node P': m -   m/2  ,A  m/2  , (K  m/2  +1,A  m/2  +1), ...,(Km,Am)

The remaining value K [m/2] and the new node P' form a tuple (K [m/2],P'), and an attempt is made to insert this tuple into the parent of P. In the current example the node g splits into two nodes g and h. The tuple (50,h) is inserted into the parent of g i.e. node c. Since c has only one key value in it, this insertion can be carried out easily. This time, 3 accesses were made to determine that X = 55 should be inserted

Figure 10.14 Insertion into a B-tree of order 3

into g. Since node g was changed, it had to be written out onto disk. The node h had also to be written out onto disk. Assuming that node c is still in internal memory, another disk access is needed to write out the new node c. The total number of accesses is therefore 6. The tree obtained is that of figure 10.14(b). Insertion of X = 37, 5, and 18 results in the trees of figure 10.14(c), (d) and (e). Into this final tree let us insert X = 12. The appropriate node for insertion is node k. In determining this, the nodes a, b and k were accessed from the

Figure 10.14 continued

Figure 10.14 Insertion into a B-tree of Order 3 (continued)

disk. We shall assume that there is enough internal memory available to hold these three nodes in memory. Insertion into node k requires that it be split into two nodes k and l. The tuple (15,l) is to be inserted into the parent node b. The new nodes k and l are written out onto the disk. Insertion into b results in this node splitting into two nodes b and m. These two nodes are written onto the disk. The tuple (15, m) is to be inserted into the parent node a which also splits. The new nodes a and n are written out, leaving us to insert the tuple (30,n) into the parent of a. Node a is the root node and thus has no parent. At this point, a new root node, p, with subtrees a and n is created. The height of the B-tree increases by 1. The total number of accesses needed for this insertion is nine.

One may readily verify that the insertion transformations described above preserve the index as a B-tree and take care of all possibilities. The resulting algorithm for insertion assumes all nodes on the path from root to insertion point are stacked. If memory is unavailable then only the addresses need be stacked and PARENT(P) can be used.

procedure INSERTB(T,X)

//Key value X is inserted into the B-tree, T, of order m. T

resides on a disk.//

1   A    0; K    X  //(K,A) is tuple to be inserted//

2   (P,i,j,)    MSEARCH(T,X)    //P is node for insertion//

3   if j    0 then return   //X already in T//

4   while P    0 do

5    insert (K,A) into appropriate position in P. Let the resulting

node have the form: n,A0, (K1,A1), ..., (Kn,An).

6    if n    m - 1 then [//resulting node is not too big//

output P onto disk; return]

//P has to be split//

7       Let P and P' be defined as in equation 10.4

8       output P and P' onto the disk

9       K    K  m/2  ; A    P'; P    PARENT(P)

10    end

//new root is to be created//

11    create a new node R with format 1,T,(K,A)

12    T    R; output T onto disk

13  end INSERTB

Analysis of INSERTB

In evaluating algorithm INSERTB, we shall use as our performance measure the number of disk accesses that are made. If the B-tree T originally has l levels then the search of line 2 requires l accesses since X is not in T. Each time a node splits (line 7), two disk accesses are made (line 8). After the final splitting,an additional access is made either from line 6 or from line 12. If the number of nodes that split is k then the total number of disk accesses is l + 2k + 1. This figure assumes that there is enough internal memory available to hold all nodes accessed during the call of MSEARCH in line 2. Since at most one node can split at each of the l levels, k is always [image] l. The maximum number of accesses needed is therefore 3l + 1. Since in most practical situations l will be at most 4, this means about thirteen accesses are needed to make an insertion in the worst case.

In the case of algorithm INSERTB, this worst case figure of 3l + l accesses doesn't tell the whole story. The average number of accesses is considerably less than this. Let us obtain an estimate of the average number of accesses needed to make an insertion. If we start with an empty tree and insert N values into it then the total number of nodes split is at most p - 2 where p is the number of nonfailure nodes in the final B-tree with N entries. This upper bound of p - 2 follows from the observation that each time a node splits, at least one additional node is created. When the root node splits, two additional nodes are created. The first node created results from no splitting, and if a B-tree has more than one node then the root must have split at least once. Figure 10.15 shows that p - 2 is the best possible upper bound on the number of nodes split in the creation of a p node B-tree when p > 2 (note that there is no B-tree with p = 2). A B-tree of order m with p nodes has at least

1 + ( [image] m/2 [image] - 1)(p - 1)

key values as the root has at least one key value and remaining nodes have at least [image] m/2 [image] - 1 key values. The average number of splittings, s, may now be determined

s = (total number of splittings)/N

   (p - 2)/ {1 + (  m/2   - 1)(p - 1)}

< 1/(  m/2   - 1).

For m = 200 this means that the average number of splittings is less than 1/99 per key inserted. The average number of disk accesses is therefore only l + 2s + 1 < l + 101/99 [image] l + 1.

Figure 10.15 B-trees of order 3

Deletion

Key values may be deleted from a B-tree using an algorithm that is almost as simple as the one for insertion. To illustrate the transformations involved, let us consider the B-tree of order 3 in figure 10.16(a). First, we shall consider the problem of deleting values from leaf nodes (i.e. nodes with no children). Deletion of the key value X = 58 from node f is easy, since the removal of this key value leaves the node with 1 key value which is the minimum every nonroot node must have. Once it has been determined that X = 58 is in node f, only one additional access is needed to rewrite the new node f onto disk. This deletion leaves us with the tree of figure 10.16(b). Deletion of the key value X = 65 from node g results in the number of keys left behind falling below the minimum requirement of [image] m/2 [image] - 1. An examination of g's

Figure 10.16 Deletion from a B-tree of order 3 (Failure nodes have been dispensed with)

Figure 10.16 Deletion from a B-tree of order 3 (continued)

nearest right sibling, h, indicates that it has [image] [image] m/2 [image] keys, and the smallest key value in h, i.e., 75 is moved up to the parent node c. The value 70 is moved down to the child node g. As a result of this transformation, the number of key values in g and h is [image] [image] m/2 [image] - 1, the number in c is unchanged and the tree retains its B-tree properties. In the search for X = 65, the nodes a, c and g had to be accessed. If all three nodes are retained in memory then from c the address of h the nearest right sibling of g is readily determined. Node h is then accessed. Since three nodes are altered (g, c, and h), three more accesses are to be made to rewrite these nodes onto disk. Thus, in addition to the accesses made to search for X = 65, four more accesses need to be made to effect the deletion of X = 65. In case g did not have a nearest right sibling or if its nearest right sibling had only the minimum number of key values i.e. m/2 [image] - 1 we could have done essentially the same thing on g's nearest left sibling. In deleting X = 55 from the tree of figure 10.16(c), we see that f's nearest right sibling, g, has only [image] m/2 [image] - 1 key values and that f does not have a nearest left sibling. Consequently, a different transformation is needed. If the parent node of f is of the form n,A0, (K1,A1), ... (Kn,An) and A1 = g then we can combine the remaining keys of f with the keys of g and the key Ki into one node, g. This node will have ( [image] m/2 [image] - 2)+ ( [image] m/2 [image] - 1) + 1 = 2 [image] m/2 [image] - 2 [image] m - 1 key values which will at most fill the node. Doing this with the nodes f, g and c results in the B-tree of figure 10.16(d). Three additional accesses are needed (one to determine that g was too small, one to rewrite each of g and c). Finally, let us delete X = 40 from node e of figure 10.16(d). Removal of this key value leaves e with [image] m/2 [image] - 2 key values. Its nearest left sibling d does not have any extra key values. The keys of d are combined with those remaining in e and the key value 30 from the parent node to obtain the full node d with values 10 and 30. This however leaves b with o When the key value X being deleted is not in a leaf node, a simple transformation reduces this deletion to the case of deletion from a leaf node. Suppose that Ki = X in the node P and P is of the form n,A0, (K1,A1), ..., (Kn,An) with 1 [image] i [image] n. Since Pis not a leaf node, Ai [image] 0. We can determine Y, the smallest key value in the subtree Ai. This will be in a leaf node Q. Replace Ki in P by Y and write out thed new P. This leaves us with the problem of deleting Y from Q. Note that this retains the search properties of the tree. For example the deletion of X = 50 from the tree of figure 10.16(a) can be accomplished by replacing the 50 in node a by the 55 from node f and then proceeding to delete the 55 from node f (see figure 10.17).

The details are provided in algorithm DELETEB on page 515. To reduce the worst case performance of the algorithm, only either the nearest left or right sibling of a node is examined.

Figure 10.17 Deleting 50 from node a of figure 10.16(a)

Analysis of DELETEB

The search for X in line 1 together with the search for the leaf Q in lines 5-10 require l accesses if l is the number of levels in T. In case P is not a leaf node at line 3 then it is modified and written out in line 11. Thus, the maximum number of accesses made in lines 1-13 is l + 1. Starting from a leaf, each iteration of the while loop of lines 15-31 moves P one level up the tree. So, at most l - 1 iterations of this loop may be made. Assuming that all nodes from the root T to the leaf P are in internal memory, the only additional accesses needed are for sibling nodes and for rewriting nodes that have been changed. The worst case happens when l - 1 iterations are made and the last one results in termination at line 25 or the corresponding point in line 30. The maximum number of accesses for this loop is therefore (l - 1)

procedure DELETEB(T,X)

//delete X from the B-tree, T, of order m. It is assumed that

T reside on a disk//

1     (P,i,j)    MSEARCH (T,X)

2     if j    1 then return   //X not in T//

3     let P be of the form: n,Ao,(K1,A1), ..., (Kn,An) and Ki = X

4     if Ao    0 then   [//deletion from non leaf, find key to move up//

5                             Q    Ai        //move to right subtree//

6                             loop

7                               let Q be of the form:

n',A'0, (K'1,A'1) ..., (K'n,A'n')

8                               if A'0 = 0 then exit  //Q is a leaf

node//

9                               Q    A'0

10                             forever

11                             replace Ki in node P by K'1 and write the

altered node P onto disk.

12                             P    Q, i    1;

13                             let n,A0,(K1,A1), ...,(Kn,An) be as

defined by the new P]

//delete Ki from node P, a leaf//

14    from P: n,A0, (K1,A1), ...,(Kn,An) delete (Ki,Ai) and replace n

by n - 1

15    while n <   m/2   - 1 and P    T do

16     if P has a nearest right sibling, Y, then

17          [let Y: n",A"0, (K"1,A"1), ...,(K"n",A"n") and

18           let Z: n',A'0, (K'1,A'1),...,(K'n',A'n') be the parent of P

and Y

19           let A'j = Y and A'j-1 = P

20           if n"      m/2   then        [//redistribute key values//

21               //update P//       (Kn+1, An+1)    (K'j, A"0);

n    n + 1

22               //update Z//       K'j    K"1

23               //update Y//       (n",A"0,(K"1,A"1),...)

(n" - 1,A"1,(K"2,A"2)...)

24                                  output nodes P,Z and Y onto

disk

25                                  return]

//combine P, K'j and Y//

26           r    2  m/2   - 2

27           output r,A0,(K1,A1) ... (Kn,An),(K'j,A"0),(K"1,A"1),

... (K"n",A"n") as new node P

28    //update//         (n,A0, ...,)    (n' - 1,A.'0,...,(K'j-1,A'j-1),

(K'j+1,A'j+1)...)

29             P    Z]

else

30             [//P must have a left sibling//

this is symmetric to lines 17-29

and is left as an exercise]

31    end

32    if n    0 then [output P: (n,A0, ...,(Kn,An)]

33          else [T    A0]

34  end DELETEB

for siblings, plus (l - 2) updates at line 27 and the corresponding point in line 30, plus 3 updates at line 24 or in line 30 for termination. This figure is 2l. The maximum number of accesses required for a deletion is therefore 3l + 1.

The deletion time can be reduced at the expense of disk space and a slight increase in node size by including a delete bit, Fi, for each key value Ki in a node. Then we can set Fi= 1 if Ki has not been deleted and Fi = 0 if it has. No physical deletion takes place. In this case a delete requires a maximum of l + 1 accesses (l to locate the node containing X and 1 to write out this node with the appropriate delete bit set to 0). With this strategy, the number of nodes in the tree never decreases. However, the space used by deleted entries can be reused during further insertions (see exercises). As a result, this strategy would have little effect on search and insert times (the number of levels increases very slowly with increasing n when m is large). Insert times may even decrease slightly due to the ability to reuse deleted entry space. Such reuses would not require us to split any nodes.

For a variation of B-trees see exercises 10-14 through 10-20.

Variable Size Key Values

With a node format of the form n,A0, (K1,A1,), ..., (Kn,An), the first problem created by the use of variable size key values, Ki, is that a binary search can no longer be carried out since given the location of the first tuple (K1,A1) and n, we cannot easily determine Kn or even the location of K(1+n)/2. When the range of key value size is small, it is best to allocate enough space for the largest size key value. When the range in sizes is large, storage may be wasted and another node format may become better, i.e., the format n,A0, [image] 1, [image] 2, ..., [image] n, (K1,A1), ...,(K n,An) where [image] i is the address of Ki in internal memory i.e. Ki = memory ( [image] i). In this case a binary search of the node can still be made. The use of variable size nodes is not recommended since this would require a more complex storage management system. More importantly, the use of variable size nodes would result in degraded performance during insertion. For example, insertion of X= 85 intothe 2-3 tree of figure 10.16(e) would require us to request a larger node, j, to accomodate the new value being added to the node h. As a result, node c would also be changed since it contains a pointer to h. This pointer must be changed to j. Consequently, nodes of a fixed size should be used. The size should be such as to allow for at least m - 1 key values of the largest size. During insertions, however, we can relax the requirement that each node have [image] m - 1 key values. Instead a node will be allowed to hold as many values as can fit into it and will contain at least [image] m/2 [image] - 1 values. The resulting performance will be at least as good as that of a B-tree of order m. Another possibility is to use some kind of key sampling scheme to reduce the key value size so as not to exceed some predetermined size d. Some possibilities are prefix and suffix truncation, removing vowels, etc. Whatever the scheme used, some provision will have to be made for handling synonyms (i.e., distinct key values that have the same sampled value). Key sampling is discussed further in section 10.2.4.

10.2.4 Trie Indexing

An index structure that is particularly useful when key values are of varying size is the trie. A trie is a tree of degree m [image] 2 in which the branching at any level is determined not by the entire key value but by only a portion of it. As an example consider the trie of figure 10.18. The trie contains two types of nodes. The first type we shall call a branch node and the second an information node. In the trie of figure 10.18 each branch node contains 27 link fields. All characters in the key values are assumed to be one of the 26 letters of the alphabet. A blank is used to terminate a key value. At level 1 all key values are partitioned into 27 disjoint classes depending on their first character. Thus, LINK(T,i) points to a subtrie containing all key values beginning with the i-th letter (T is the root of the trie). On the j th level the branching is determined by the j th character. When a subtrie contains only one key value, it is replaced by a node of type information. This node contains the key value, together with other relevant information such as the address of the record with this key value, etc. In the figure, branch nodes are represented by rectangles while ovals are used for information nodes.

Searching a trie for a key value X requires breaking up X into its constituent characters and following the branching patterns determined by these characters. The algorithm TRIE assumes that P = 0 is not a branch node and that KEY(P) is the key value represented in P if P is an information node.

Analysis of Algorithm TRIE

The search algorithm for tries is very straightforward and one may readily verify that the worst case search time is O(l) where l is the number of levels in the trie (including both branch and information

procedure TRIE (T,X)

//Search a trie T for key value X. It is asssumed that branching

on the i-th level is determinded by the i-th character of the key

value.//

K    X     ' '       //concatenate a blank to the end of X//

i    1; P    T

while P is a branch node do

C    i-th character of K

P    LINK (P,C);    i + 1

end

if P = 0 or KEY (P)    X then return (0)      //X not in trie//

return (P)

end TRIE

nodes). In the case of an index, all these nodes will reside on disk and so at most l accesses will have to be made to effect the search. Given a set of key values to be represented in an index, the number of levels in the trie will clearly depend on the strategy or key sampling technique used to determine the branching at each level. This can be defined by a sampling function SAMPLE(X,i) which appropriately samples X for branching at the i-th level. In the example trie of figure 10.18 and in the search algorithm TRIE this function was

(a) SAMPLE(X,i) = i'th character of X.

Some other possible choices for this function are (X = x1 x2 ... xn)

(b) SAMPLE(X,i) = xn - i+1

(c) SAMPLE(X,i) = xr(Xi) for r(X,i) a randomization function

(d) SAMPLE(X,i) =

For each of these functions one may easily construct key value sets for which that particular function is best, i.e. it results in a trie with the fewest number of levels. The trie of figure 10.18 has 5 levels. Using the function (b) on the same key values yields the trie of figure 10.20, which has only 3 levels. An optimal sampling function for this data set will yield a trie that has only 2 levels (figure 10.21). Choosing the optimal sampling function for any particular set of values is very difficult. In a dynamic situation, with insertion and deletion, we wish to optimize average performance. In the absence of any further information on key values, probably the best choice would be (c). Even though all our examples of sampling have involved single character sampling we need not restrict ourselves to this. The key value may be interpreted as consisting of digits using any radix we desire. Using a radix of 272

Figure 10.18 Trie created using characters of key value left to right, one at a time

Figure 10.19 Trie showing need for a terminal character (in this case a blank).

would result in 2 character sampling. Other radixes would give different samplings. The maximum number of levels in a trie can be kept low by adopting a different strategy for information nodes. These nodes can be designed to hold more than one key value. If the maximum number of levels allowed is l then all key values that are synonyms up to level l - 1 are entered into the same information node. If the sampling function is chosen correctly, there will be only a few synonyms in each information node. The information node will therefore be small and can be processed in internal memory. Figure 10.22 shows the use of this strategy on the trie of figure 10.18 with l = 3. In further discussion we shall, for simplicity, assume that the sampling function in use is (a) and that no restriction is placed on the number of levels in the trie.

Insertion

Insertion into a trie is straightforward. We shall indicate the procedure by means of two examples and leave the formal writing of the algorithm as an exercise. Let us consider the trie of figure 10.18 and insert into it the two entries: bobwhite and bluejay. First, we have X = bobwhite and we attempt to search for 'bobwhite' in T. This leads us to node [image] , where we discover that LINK( [image] , 'O') = 0. Hence X is not in T and may be inserted here (see figure 10.23). Next, X = bluejay and a search of T leads us to the information node [image]. A comparison indicates

Figure 10.20 Trie constructed for data of figure 10.18 sampling one character at a time, right to left

Figure 10.21 An optimal trie for data of figure 10.18 sampling on first level done by using 4th character of key values.

Figure 10.22 Trie obtained for data of figure 10.18 when number of levels is limited to 3. Key has been sampled left to right one character at a time.

Figure 10.23 Section of trie of figure 10.18 showing changes resulting from inserting bobwhite and bluejay

that KEY( [image] ) [image] X. Both KEY( [image] ) and X will form a subtrie of [image] . The two values KEY( [image] ) and X are sampled until the sampling results in two different values. The happens when the 5th letter of KEY( [image] ) and X are compared. The resulting trie after insertion is in figure 10.23.

Deletion

Once again, we shall not present the deletion algorithm formally but we will look at two examples to illustrate some of the ideas involved in deleting entries from a trie. From the trie of figure 10.23 let us first delete 'bobwhite.' To do this we just set LINK( [image],'O') = 0. No other changes need be made. Next, let us delete 'bluejay.' This deletion leaves us with only one key value in the subtrie, [image] 3. This means that the node [image] 3 may be deleted and [image] moved up one level. The same can be done for nodes [image] 1 and [image] 2. Finally, the node [image] is reached. The subtrie with root [image] has more than one key value. Therefore [image] cannot be moved up any more levels and we set LINK( [image],'L') = [image] . In order to facilitate deletions from tries, it is useful to add a COUNT field in each branch node. This count field will at all times give us the number of information nodes in the subtree for which it is the root. See exercise 26 for more on tries.

10.3 FILE ORGANIZATIONS

10.3.1 Sequential Organizations

The problems associated with sequential organizations were discussed earlier. The most popular sequential organization scheme is ISAM, in which a cylinder-surface index is maintained for the primary key. In order to efficiently retrieve records based on other keys, it is necessary to maintain additional indexes on the remaining keys (i.e. secondary keys). The structure of these indexes may correspond to any of the alternative index techniques discussed in the previous section. The use of secondary key indexes will be discussed in greater detail in connection with inverted files.

10.3.2 Random Organization

In this organization, records are stored at random locations on disk. This randomization could be achieved by any one of several techniques. Some of these techniques are direct addressing, directory lookup and hashing.

Direct Addressing

In direct addressing with equal size records, the available disk space is divided out into nodes large enough to hold a record. The numeric value of the primary key is used to determine the node into which a particular record is to be stored. No index on this key is needed. With primary key = Employee #, the record for Employee # = 259 will be stored in node 259. With this organization, searching and deleting a record given its primary key value requires only one disk access. Updating a record requires two (one to read and another to write back the modified record). When variable size records are being used an index can be set up with pointers to actual records on disk (see figure 10.24). The number of accesses needed using this scheme is one more than for the case when memory was divided into fixed size nodes. The storage management scheme becomes more complex (section 10.4). The space efficiency of direct accessing depends on the identifier density n/T (n = number of distinct primary key values in the file, T = total number of possible primary key values). In the case of internal tables, this density is usually very low and direct addressing was very space inefficient.

Figure 10.24 Direct Access Index for Data of Figure 10.1.

Directory LookUp

This is very similar to the scheme of figure 10.24 for the case of direct addressing with variable size records. Now, however, the index is not of direct access type but is a dense index maintained using a structure suitable for index operations. Retrieving a record involves searching the index for the record address and then accessing the record itself. The storage management scheme will depend on whether fixed size or variable size nodes are being used. Except when the identifier density is almost 1, this scheme makes a more efficient utilization of space than does direct addressing. However it requires more accesses for retrieval and update, since index searching will generally require more than 1 access. In both direct addressing and directory lookup, some provision must be made for collisions (i.e. when two or more records have the same primary key value). In many applications the possibility of collisions is ruled out since the primary key value uniquely identifies a record. When this is not the case, some of the schemes to be discussed in section 10.3.3 and section 10.3.4 may be used.

Hashing

The principles of hashed file organization are essentially the same as those for a hashed index. The available file space is divided into buckets and slots. Some space may have to be set aside for an overflow area in case chaining is being used to handle overflows. When variable size records are present, the number of slots per bucket will be only a rough indicator of the number of records a bucket can hold. The actual number will vary dynamically with the size of records in a particular bucket.

Random organization on the primary key using any of the above three techniques overcomes the difficulties of sequential organizations. Insertions and deletions become relatively trivial operations. At the same time, random organizations lose the advantages of a sequential ordered organization. Batch processing of queries becomes inefficient as the records are not maintained in order of the primary key. In addition, handling of range queries becomes exceedingly inefficient except in the case of directory lookup with a suitable index structure. For example, consider these two queries: (i) retrieve the records of all employees with Employee Number > 800 and (ii) retrieve all records with 301 [image] employee number [image] 800. To do (i) we will need to know the maximum employee number in the file or else examine every node. If the maximum employee number is known then (i) becomes similar to (ii), so lets look at (ii). With direct addressing and hashing, we would have to search for records with employee number = 301, 302, ... 800, a total of 500 independent searches. The number of records satisfying the query may be much smaller (even 0). In the case of ISAM we could in three accesses locate the record (if one exists) having the smallest employee number satisfying the query and retrieve the remaining records one at a time. The same is possible when a search tree type directory (index) is maintained.

10.3.3 Linked Organization

Linked organizations differ from sequential organizations essentially in that the logical sequence of records is generally different from the physical sequence. In a sequential organization, if the i'th record of the file is at location li then the i + l'st record is in the next physical position li + c where c may be the length of the i'th record or some constant that determines the inter-record spacing. In a linked organization the next logical record is obtained by following a link value from the present record. Linking records together in order of increasing primary key value facilitates easy insertion and deletion once the place at which the insertion or deletion to be made is known. Searching for a record with a given primary key value is difficult when no index is available, since the only search possible is a sequential search. To facilitate searching on the primary key as well as on secondary keys it is customary to maintain several indexes, one for each key. An employee number index, for instance, may contain entries corresponding to ranges of employee numbers. One possibility for the example of figure 10.1 would be to have an entry for each of the ranges 501-700, 701-900 and 901-1100. All records having E # in the same range will be linked together as in figure 10.25. Using an index in this way reduces the length of the lists and thus the search time. This idea is very easily generalized to allow for easy secondary key retrieval. We just set up indexes for each key and allow records to be in more than one list. This leads to the multilist structure for file representation. Figure 10.26 shows the indexes and lists corresponding to a multilist representation of the data of figure 10.1. It is assumed that the only fields designated as keys are: E#, Occupation, Sex and Salary. Each record in the file, in addition to all the relevant information fields, has 1 link field for each key field.

The logical order of records in any particular list may or may not

Figure 10.25

Figure 10.26 Multilist representation for figure 10.1

be important depending upon the application. In the example file, lists corresponding to E#, Occupation and Sex have been set up in order of increasing E#. The salary lists have been set up in order of increasing salary within each range (record A precedes D and C even though E#(C) and E#(D) are less than E#(A)).

Notice that in addition to key values and pointers to lists, each index entry also contains the length of the corresponding list. This information is useful when retrieval on boolean queries is required. In order to meet a query of the type, retrieve all records with Sex = female and Occupation = analyst, we search the Sex and Occupation indexes for female and analyst respectively. This gives us the pointers B and B. The length of the list of analysts is less than that of the list of females, so the analyst list starting at B is searched. The records in this list are retrieved and the Sex key examined to determine if the record truly satisfies the query. Retaining list lengths enables us to reduce search time by allowing us to search the smaller list. Multilist structures provide a seemingly satisfactory solution for simple and range queries. When boolean queries are involved, the search time may bear no relation to the number of records satisfying the query. The query K1 = XX and K2 = XY may lead to a K1 list of length n and a K2 list of length m. Then, min{n,m} records will be retrieved and tested against the query. It is quite possible that none or only a very small number of these min{n,m} records have both K1 = XX and K2 = XY. This situation can be remedied to some extent by the use of compound keys. A compound key is obtained by combining two or more keys together. We could combine the Sex and Occupation keys to get a new key Sex-Occupation. The values for this key would be: female analyst, female programmer, male analyst and male programmer. With this compound key replacing the two keys Sex and Occupation, we can satisfy queries of the type, all male programmers or all programmers, by retrieving only as many records as actually satisfy the query. The index size, however, grows rapidly with key compounding. If we have ten keys K1, ...,K10, the index for Ki having ni entries, then the index for the compound key K1-K2- ... -K10 will have [image] 10i=1 ni entries while the original indexes had a total of [image] 10i=1 ni entries. Also, handling simple queries becomes more complex if the individual key indexes are no longer retained (see the exercies).

Inserting a new record into a multilist structure is easy so long as the individual lists do not have to be maintained in some order. In this case the record may be inserted at the front of the appropriate lists (see exercise 27). Deletion of a record is difficult since there are no back pointers. Deletion may be simplified at the expense of doubling the number of link fields and maintaining each list as a doubly linked list (see exercises 28 and 30). When space is at a premium, this expense may not be accpetable. An alternative is the coral ring structure described below.

Coral Rings

The coral ring structure is an adaptation of the doubly linked multilist structure discussed above. Each list is structured as a circular list with a headnode. The headnode for the list for key value Ki = X will have an information field with value X. The field for key Ki is replaced by a link field. Thus, associated with each record, Y, and key, Ki, in a coral ring there are two link fields: ALINK(Y,i) and BLINK(Y,i). The ALINK field is used to link together all records with the same value for key Ki. The ALINKs form a circular list with a headnode whose information field retains the value of Ki for the records in this ring. The BLINK field for some records is a back pointer and for others it is a pointer to the head node. To distinguish between these two cases another field FLAG(Y,i) is used. FLAG(Y,i) = 1 if BLINK(Y,i) is a back pointer and FLAG(Y,i) = 0 otherwise. In practice the FLAG and BLINK fields may be combined with BLINK(Y,i) > 0 when it is a back pointer and < 0 when it is a pointer to the head node. When the BLINK field of a record BLINK(Y,i) is used as a back pointer, it points to the nearest record, Z, preceding it in its circular list for Ki having BLINK(Z,i) also a back pointer. In any given circular list, all records with back pointers form another circular list in the reverse direction (see figure 10.27). The presence of these back pointers makes it possible to carry out a deletion without having to start at the front of each list containing the record being deleted in order to determine the preceding records in these lists (see exercise 32). Since these BLINK fields will usually be smaller than the original key fields they replace, an overall saving in space will ensue. This is, however, obtained at the expense of increased retrieval time (exercise 31). Indexes are maintained as for multilists. Index entries now point to head nodes. As in the case of multilists, an individual node may be a member of several rings on different keys.

Figure 10.27 Coral ring for analysts in a hypothetical file.

10.3.4 Inverted Files

Conceptually, inverted files are similar to multilists. The difference is that while in multilists records with the same key value are linked together with link information being kept in individual records, in the case of inverted files this link information is kept in the index itself. Figure 10.28 shows the indexes for the file of figure 10.1. A slightly different strategy has been used in the E# and salary indexes than was used in figure 10.26, though the same strategy could have been used here too. To simplify further discussion, we shall assume that the index for every key is dense and contains a value entry for each distinct value in the file. Since the index entries are variable length (the number of records with the same key value is variable), index maintenance becomes more complex than for multilists. However, several benefits accrue from this scheme. Boolean queries require only one access per record satisfying the query (plus some accesses to process the indexes). Queries of the type K1 = XX or K2 = XY can be processed by first accessing the indexes and obtaining the address lists for all records with K1 = XX and K2 = XY. These two lists are then merged to obtain a list of all records satisfying the query. K1 = XX and K2 = XY can be handled similarly by intersecting the two lists. K1 = .not. XX can be handled by maintaining a universal list, U, with the addresses of all records. Then, K1 = .not. XX is just the difference between U and the list for K1 = XX. Any complex boolean query may be handled in this way. The retrieval works in two steps. In the first step, the indexes are processed to obtain a list of records satisfying the query and in the second, these records are retrieved using this list. The number of disk accesses needed is equal to the number of records being retrieved plus the number to process the indexes. Exercise 34 explores the time savings that can result using this structure rather than multilists.

Inverted files represent one extreme of file organization in which only the index structures are important. The records themselves may be stored in any way (sequentially ordered by primary key, random,

Figure 10.28 Indexes for fully inverted file

linked ordered by primary key etc). Inverted files may also result in space saving compared with other file structures when record retrieval does not require retrieval of key fields. In this case the key fields may be deleted from the records. In the case of multilist structures, this deletion of key fields is possible only with significant loss in system retrieval performance (why?). Insertion and deletion of records requires only the ability to insert and delete within indexes.

10.3.5 Cellular Partitions

In order to reduce file search times, the storage media may be divided into cells. A cell may be an entire disk pack or it may simply be a cylinder. Lists are localised to lie within a cell. Thus if we had a multilist organization in which the list for KEY1 = PROG included records on several different cylinders then we could break this list into several smaller lists where each PROG list included only those records in the same cylinder. The index entry for PROG will now contain several entries of the type (addr, length), where addr is a pointer to the start of a list of records with KEY1 = PROG and length is the number of records on this list. By doing this, all records in the same cell (i.e. cylinder) may be accessed without moving the read/write heads. In case a cell is a disk pack then using cellular partitions it is possible to search different cells in parallel (provided the system hardware permits simultaneous reading/writing from several disk drives).

It should be noted that in any real situation a judicious combination of the techniques of this section would be called for. I.e., the file may be inverted on certain keys, ringed on others, and a simple multilist on yet other keys.

10.4 STORAGE MANAGEMENT

The functions of a storage management system for an external storage device (such as a disk) are the same as those for a similar system for internal memory. The storage management system should be able to allocate and free a contiguous block of memory of a given size. In chapter 4 we studied several memory management schemes. For fixed size nodes we used a linked stack together with the routines RET and GETNODE of section 4.3. For variable size nodes the boundary tag method of section 4.8 was good. We also studied a general purpose management scheme involving garbage collection and compaction in section 4.10. When the storage to be managed is on an external device, the management scheme used should be such as to minimize the time needed to allocate and free storage. This time will to a large extent depend on the number of external storage accesses needed to effect allocation or freeing. In this section we shall review and reevaluate the storage management schemes of chapter 4 with emphasis on the number of device accesses required.

Fixed Size Nodes

All nodes that are free may be linked together into an available space list as in section 4.3. Assuming that the value of AV is always in memory the allocate and free algorithms take the form:

procedure GETNODE(I)

if AV = 0 then [print "no more nodes;" stop]

I    AV; X    GET(AV); AV    LINK(X)

end GETNODE

procedure RET(I)

LINK(I)    AV; WRITE(I); AV    I

end RET

GET and WRITE are primitives that read from and write onto an external storage device. Each algorithm requires one access to carry out its function. In case the total number of free nodes at all times is small, this access can be eliminated altogether by maintaining the available space list in the form of a stack, STK, which contains the addresses of all free nodes. This stack would not be maintained in place as in the case of section 4.3, where the free space itself made up the stack. Instead, the stack could reside permanently in memory and no accesses would be needed to free or allocate nodes.

Variable Size Nodes

In the boundary tag method, the use of boundary tags made it possible to free a block while coalescing adjacent blocks in a constant amount of time. In order to do this it is essential that the free space be maintained in place as we need to test TAG(p - 1) and TAG(p + n)(cf. section 4.8). Such a scheme would therefore require several accesses (exercise 35) in order to free and allocate nodes. These accesses can again be eliminated by maintaining in memory a list of all blocks of storage that are free. Allocation can be made using first fit, and nodes can be freed in time O(n) if n is the number of blocks in free space (exercise 36). Since the cost of a disk access is several orders of magnitude more than the cost of internal processing, this scheme will be quicker than the boundary tag method even when free space consists of several thousand blocks. When free space contains many nodes it may not be possible to keep the list of free nodes in memory at all times. In this case the list will have to be read in from disk whenever needed and written back onto disk when modified. Still, the number of accesses will be fewer than when an in place free list is maintained (as is required by the boundary tag method).

Garbage Collection and Compaction

The principles for this technique stay essentially the same. The process was seen to be rather slow for the case of internal memory management. It is even slower for external storage because of the increased access times. In file systems that have a heavy activity rate (i.e. frequent real time update etc.), it is not possible to allocate a continuous chunk of time large enough to permit garbage collection and compaction. It is necessary, therefore, to devise these algorithms so that they can work with frequent interruptions, leaving the file in a usable state whenever interrupted.

REFERENCES

For additional material on file structures and data base management systems see:

File structures for on line systems by D. Lefkovitz, Hayden Book Co., New Jersey, 1969.

Data Management for on line systems by D. Lefkovitz, Hayden Book Co., New Jersey, 1974.

Computer data base organization by J. Martin, Prentice-Hall, Englewood Cliffs, 1975.

An introduction to data base management systems by C. Date, Addison-Wesley, Reading, Massachusetts, 1975.

Additional material on indexes can be found in:

The Art of Computer Programming: Sorting and Searching by D. Knuth, Addison-Wesley, Reading, Massachusetts, 1973.

"Binary search trees and file organization" by J. Nievergelt, ACM Computing Surveys, vol. 6, no. 3, September 1974, pp. 195-207.

EXERCISES

1. A file of employee records is being created. Each record has the following format:

E#  NAME  Occupation  Location

All four fields have been designated as keys. The file is to consist of the following 5 records:

A  10  JAMES   PROG  MPLS

B  27  SHERRY  ANAL  NY

C  39  JEAN    PROG  NY

D  50  RODNEY  KEYP  MPLS

E  75  SUSAN   ANAL  MPLS

Draw the file and index representations for the following organizations (assume that an entry in an index is a tuple (value, pointer 1, pointer 2, ...,pointer k) and these tuples are kept sequentially).

a) Multilist File

b) Fully Inverted File

c) Ring File

d) Sequential ordered by name, inverted on E# and location, ringed on occupation.

2. Write an algorithm to process a tape file in the batched mode. Assume the master file is ordered by increasing primary key value and that all such values are distinct. The transaction file contains transactions labeled: update, delete and insert. Each such transaction also contains the primary key value of the record to be updated, deleted or inserted. A new updated master file is to be created. What is the complexity of your algorithm?

3. Show that all B-trees of order 2 are full binary trees.

4. (a) Into the following 2-3 tree insert the key 62 using algorithm INSERTB.

Assuming that the tree is kept on a disk and one node may be fetched at a time, how many disk accesses are needed to make this insertion? State any assumptions you make.

(b) From the following B-tree of order 3 delete the key 30 (use algorithm DELETEB). Under the same assumptions as kn (a), how many disk accesses are needed?

5. Complete line 30 of algorithm DELETEB.

6. Write insertion and deletion algorithms for B-trees assuming that with each key value is associated an additional field F such that F = 1 iff the corresponding key value has not been deleted. Deletions should be accomplished by simply setting the corresponding F = 0 and insertions should make use of deleted space whenever possible without restructuring the tree.

7. Write algorithms to search and delete keys from a B-tree by position. I.e., SEARCH(k) finds the k-th smallest key and DELETE(k) deletes the k-th smallest key in the tree. (Hint: In order to do this efficiently additional information must be kept in each node. With each pair (Ki,Ai) keep [image] (number of key values in the subtree Aj + 1).) What is the worst case computing times of your algorithms?

8. Program the AVL tree insertion algorithm of section 9.2 and also the B-tree insertion algorithm with m = 3. Evaluate the relative performance of these two methods for maintaining dynamic internal tables when only searches and insertions are to be made. This evaluation is to be done by obtaining real computing times for various input sequences as well as by comparing storage requirements.

9. Modify algorithm INSERTB so that in case n = m in line 6 then we first check to see if either the nearest left sibling or the nearest right sibling of P has fewer than m - 1 key values. If so, then no additional nodes are created. Instead, a rotation is performed moving either the smallest or largest key in P to its parent. The corresponding key in the parent together with a subtree is moved to the sibling of P which has space for another key value.

10. [Bayer and McCreight] The idea of exercise 9 can be extended to obtain improved B-tree performance. In case the nearest sibling, P', of P already has m - 1 key values then we can split both P and P' to obtain three nodes P, P' and P\" with each node containing [image] (2m - 2)/3 [image] , [image] (2m - 1)/3 [image] and [image] 2m/3 [image] key values. Figure 10.29 below describes this splitting procedure when P' is P's nearest right sibling.

Figure 10.29 Splitting P and nearest right sibling P'

Rewrite algorithm INSERTB so that node splittings occur only as described above.

11. A B*-tree, T, of order m is a search tree that is either empty or is of height [image] 1. When T is not empty then the extended tree T (i.e., T with failure nodes added) satisfies the following conditions:

(i) The root node has at least 2 and at most 2 [image] (2m - 2)/3 [image] + 1 children.

(ii) The remaining nonfailure nodes have at most m and at least [image] (2m - 1)/3 [image] children each.

(iii) All failure nodes are on the same level.

For a B*-tree of order m and containing N key values show that if x= [image] (2m - 1)/3 [image] then

(a) The depth, d, of T satisfies:

d [image] 1 + logx{(N+1)/2}

(b) the number of nodes p in T satisfies:

p [image] 1 + (N - 1)/(x-1)

What is the average number of splittings if T is built up starting from an empty B*-tree?

12. Using the splitting technique of exercise 10 write an algorithm to insert a new key value X into a B*-tree, T, of order m. How many disk accesses are made in the worst case and on the average? Assume that T was initially of depth l and that T is maintained on a disk. Each access retrieves or writes one node.

13. Write an algorithm to delete the identifier X from the B*-tree, T, of order m. What is the maximum number of accesses needed to delete X from a B*-tree of depth l. Make the same assumptions as in exercise 12.

14. The basic idea of a B-tree may be modified differently to obtain a B'-tree. A B'-tree of order m differs from a B-tree of order m only in that in a B'-tree identifiers may be placed only in leaf nodes. If P is a nonleaf node in a B'-tree and is of degree j then the node format for P is: j, L(1), L(2), ... L(j - 1) where L(i), 1 [image] i < j is the value of the largest key in the i'th subtree of P. Figure 10.30 shows two B'-trees of order 3. Notice that in a B'-tree the key values in the leaf nodes will be increasing left to right. Only the leaf nodes contain such information as the address of records having that key value. If there are n key values in the tree then there are n leaf nodes. Write an algorithm to search for X in a B'-tree T of order 3. Show that the time for this is O(log n).

Figure 10.30 Two B'-trees of order 3

15. For a B'-tree of order 3 write an algorithm to insert X. Recall that all non leaf nodes in a B'-tree of order 3 are of degree 2 or 3. Show that the time for this is O(log n).

16. Write an algorithm to delete X from a B'-tree, T, of order 3. Since all key values are in leaf nodes, this always corresponds to a deletion from a leaf. Show that if T has n leaf nodes then this requires only O(log n) time.

17. Let T and T' be two B'-trees of order 3. Let T\" be a B'-tree of order 3 containing all key values in T and T\". Show how to construct T\" from T and T' in O(log n) time.

18. Write computer programs to insert key values into AVL trees, B-trees of order 3, B*-trees of order 3 and B'-trees of order 3. Evaluate the relative performance of these four representations of internal tables.

19. Do exercise 18 when the required operations are search for X, insert X and delete X.

20. Obtain SEARCH, INSERT and DELETE algorithms for B'-trees of order m. If the tree resides on disk, how many disk accesses are needed in the worst case for each of the three operations. Assume the tree has n leaf nodes.

21. Draw the trie obtained for the following data: Sample the keys left to right one character at a time. Using single character sampling obtain an optimal trie for the above data (an optimal trie is one with the fewest number of levels).

AMIOT, AVENGER, AVRO, HEINKEL, HELLDIVER, MACCHI, MARAUDER, MUSTANG, SPITFIRE, SYKHOI

22. Write an algorithm to insert a key value X into a trie in which the keys are sampled left to right, one character at a time.

23. Do exercise 22 with the added assumption that the trie is to have no more than 6 levels. Synonyms are to be packed into the same information node.

24. Write an algorithm to delete X from a trie T under the assumptions of exercise 22. Assume that each branch node has a count field equal to the number of information nodes in the subtrie for which it is the root.

25. Do exercise 24 for the trie of exercise 23.

26. In the trie of Figure 10.23 the nodes [image] 1 and [image] 2 each have only one child. Branch nodes with only one child may be eliminated from tries by maintaining a SKIP field with each node. The value of this field equals the number of characters to be skipped before obtaining the next character to be sampled. Thus, we can have SKIP ( [image] 3) = 2 and delete the nodes [image] 1 and [image] 2. Write algorithms to search, insert and delete from tries in which each branch node has a skip field.

In exercises 27-34 records have n keys. The i'th key value for record Z is KEY(Z,i). The link field for the i'th key is LINK(Z,i). The number of accesses required by an algorithm should be given as a function of list lengths.

27. Write an algorithm to insert a record, Z, into a multilist file with n keys. Assume that the order of records in individual lists is irrelevant. Use the primitives SEARCH(X) and UPDATE(X,A) to search and update an index. UPDATE(X,A) changes the address pointer for X to A. How many disk accesses (in addition to those needed to update the index) are needed?

28. Write an algorithm to delete an arbitrary record Z from a multilist file (see exercise 27). How many disk accesses are needed?

29. Write an algorithm to insert record Z into a multilist file assuming that individual lists are ordered by the primary key KEY(Z,1). How many accesses are needed for this (exclude index accesses)?

30. Assuming that each list in a multilist file is a doubly linked list, write an algorithm to delete an arbitrary record Z. The forward link for key i is ALINK(Z,i) while the corresponding backward link is BLINK(Z,i).

31. Write an algorithm to output all key values for record Z in a ring structure. How many accesses are needed for this? How many accesses are needed to do the same in a multilist file?

32. Write an algorithm to delete an arbitrary record Z from a ring file.

33. Describe briefly how to do the following:

(i) In a multilist organization: (a) output all records with KEY1 = PROG and KEY2 = NY. How many accesses are needed to carry this out? (b) Output all records with KEY1 = PROG or KEY2 = NY. How many accesses are needed for this. Assume that each access retrieves only one record.

(ii) If a ring organization is used instead, what complications are introduced into (a) and (b) above?

(iii) How could the following be carried out using an inverted organization:

a) output all records with KEY1 = PROG and KEY2 = NY

b) output all records with KEY1 = PROG or KEY2 = NY

c) output all records with KEY1 = PROG and KEY2 [image] NY

How many accesses are needed in each case (exclude accesses to get indexes)?

34. A 105 record file is maintained as an inverted file on a disk with track capacity 5000 characters. This disk has 200 tracks on each of its 10 surfaces. Each record in the file is 50 characters long and has five key fields. Each key is binary (i.e., has only two distinct values) and so the index for each key can be maintained as a binary bit string of length 105 bits. If 1 character is 6 bits long, then each index takes about 4 tracks. How should the 5 indexes be stored on disk so as to minimize total seek time while processing the indexes in order to determine which records satisfy a given boolean query Q? This processing involves reading in 1 track of each index and testing the query against records represented by this track. Then the next set of index tracks is input and so on. How much time does it take to process all the indexes in order to determine which records are to be retrieved? Assume a seek time of 1/10 sec and a latency time of 1/40 sec. Also assume that only the input time is significant. If k records satisfy this query, how much more time is needed to retrieve these k records? Using other file structures it may be necessary to read in the whole file. What is the minimum time needed to read in the entire file of 105 records? How does this compare with the time needed to retrieve k records using an inverted file structure?

35. Determine how many disk accesses are needed to allocate and free storage using algorithms ALLOCATE and FREE of section 4.8 for the boundary tag method when the storage being managed is disk storage.

36. Write storage management algorithms for disk storage maintaining a list of free blocks in internal memory. Use first fit for allocation. Adjacent free blocks should be coalesced when a block is freed. Show that both these tasks can be accomplished in time O(n) where n is the number of free blocks.

Go to Appendix A Back to Table of Contents


Fundamentals: PREFACE

 Return to Table of Contents

PREFACE

For many years a data structures course has been taught in computer science programs. Often it is regarded as a central course of the curriculum. It is fascinating and instructive to trace the history of how the subject matter for this course has changed. Back in the middle1960's the course was not entitled Data Structures but perhaps List Processing Languages. The major subjects were systems such as SLIP (by J. Weizenbaum), IPL-V (by A. Newell, C. Shaw, and H. Simon), LISP 1.5 (by J. McCarthy) and SNOBOL (by D. Farber, R. Griswold, and I. Polonsky). Then, in 1968, volume I of the Art of Computer Programming by D. Knuth appeared. His thesis was that list processing was not a magical thing that could only be accomplished within a specially designed system. Instead, he argued that the same techniques could be carried out in almost any language and he shifted the emphasis to efficient algorithm design. SLIP and IPL-V faded from the scene, while LISP and SNOBOL moved to the programming languages course. The new strategy was to explicitly construct a representation (such as linked lists) within a set of consecutive storage locations and to describe the algorithms by using English plus assembly language.

Progress in the study of data structures and algorithm design has continued. Out of this recent work has come many good ideas which we believe should be presented to students of computer science. It is our purpose in writing this book to emphasize those trends which we see as especially valuable and long lasting.

The most important of these new concepts is the need to distinguish between the specification of a data structure and its realization within an available programming language. This distinction has been mostly blurred in previous books where the primary emphasis has either been on a programming language or on representational techniques. Our attempt here has been to separate out the specification of the data structure from its realization and to show how both of these processes can be successfully accomplished. The specification stage requires one to concentrate on describing the functioning of the data structure without concern for its implementation. This can be done using English and mathematical notation, but here we introduce a programming notation called axioms. The resulting implementation independent specifications valuable in two ways: (i) to help prove that a program which uses this data structure is correct and (ii) to prove that a particular implementation of the data structure is correct. To describe a data structure in a representation independent way one needs a syntax. This can be seen at the end of section 1.1 where we also precisely define the notions of data object and data structure.

This book also seeks to teach the art of analyzing algorithms but not at the cost of undue mathematical sophistication. The value of an implementation ultimately relies on its resource utilization: time and space. This implies that the student needs to be capable of analyzing these factors. A great many analyses have appeared in the literature, yet from our perspective most students don't attempt to rigorously analyze their programs. The data structures course comes at an opportune time in their training to advance and promote these ideas. For every algorithm that is given here we supply a simple, yet rigorous worst case analysis of its behavior. In some cases the average computing time is also derived.

The growth of data base systems has put a new requirement on data structures courses, namely to cover the organization of large files. Also, many instructors like to treat sorting and searching because of the richness of its examples of data structures and its practical application. The choice of our later chapters reflects this growing interest.

One especially important consideration is the choice of an algorithm description language. Such a choice is often complicated by the practical matters of student background and language availability. Our decision was to use a syntax which is particularly close to ALGOL, but not to restrict ourselves to a specific language. This gives us the ability to write very readable programs but at the same time we are not tied to the idiosyncracies of a fixed language. Wherever it seemed advisable we interspersed English descriptions so as not to obscure the main pointof an algorithm. For people who have not been exposed to the IF-THEN-ELSE, WHILE, REPEAT- UNTIL and a few other basic statements, section 1.2 defines their semantics via flowcharts. For those who have only FORTRAN available, the algorithms are directly translatable by the rules given in the appendix and a translator can be obtained (see appendix A). On the other hand, we have resisted the temptation to use language features which automatically provide sophisticated data structuring facilities. We have done so on several grounds. One reason is the need to commit oneself to a syntax which makes the book especially hard to read by those as yet uninitiated. Even more importantly, these automatic featules cover up the implementation detail whose mastery remains a cornerstone of the course.

The basic audience for this book is either the computer science major with at least one year of courses or a beginning graduate student with prior training in a field other than computer science. This book contains more than one semester's worth of material and several of its chapters may be skipped without harm. The following are two scenarios which may help in deciding what chapters should be covered.

The first author has used this book with sophomores who have had one semester of PL/I and one semester of assembly language. He would cover chapters one through five skipping sections 2.2, 2.3, 3.2, 4.7, 4.11, and 5.8. Then, in whatever time was left chapter seven on sorting was covered. The second author has taught the material to juniors who have had one quarter of FORTRAN or PASCAL and two quarters of introductory courses which themselves contain a potpourri of topics. In the first quarter's data structure course, chapters one through three are lightly covered and chapters four through six are completely covered. The second quarter starts with chapter seven which provides an excellent survey of the techniques which were covered in the previous quarter. Then the material on external sorting, symbol tables and files is sufficient for the remaining time. Note that the material in chapter 2 is largely mathematical and can be skipped without harm.

The paradigm of class presentation that we have used is to begin each new topic with a problem, usually chosen from the computer science arena. Once defined, a high level design of its solution is made and each data structure is axiomatically specified. A tentative analysis is done to determine which operations are critical. Implementations of the data structures are then given followed by an attempt at verifying that the representation and specifications are consistent. The finishedalgorithm in the book is examined followed by an argument concerning its correctness. Then an analysis is done by determining the relevant parameters and applying some straightforward rules to obtain the correct computing time formula.

In summary, as instructors we have tried to emphasize the following notions to our students: (i) the ability to define at a sufficiently high level of abstraction the data structures and algorithms that are needed; (ii) the ability to devise alternative implementations of a data structure; (iii) the ability to synthesize a correct algorithm; and (iv) the abilityto analyze the computing time of the resultant program. In addition there are two underlying currents which, though not explicitly emphasized are covered throughout. The first is the notion of writing nicely structured programs. For all of the programs contained herein we have tried our best to structure them appropriately. We hope that by reading programs with good style the students will pick up good writing habits. A nudge on the instructor's part will also prove useful. The second current is the choice of examples. We have tried to use those examples which prove a point well, have application to computer programming, and exhibit some of the brightest accomplishments in computer science.

At the close of each chapter there is a list of references and selected readings. These are not meant to be exhaustive. They are a subset of those books and papers that we found to be the most useful. Otherwise, they are either historically significant or develop the material in the text somewhat further.

Many people have contributed their time and energy to improve this book. For this we would like to thank them. We wish to thank Arvind [sic], T. Gonzalez, L. Landweber, J. Misra, and D. Wilczynski, who used the book in their own classes and gave us detailed reactions. Thanks are also due to A. Agrawal, M. Cohen, A. Howells, R. Istre, D. Ledbetter, D. Musser and to our students in CS 202, CSci 5121 and 5122 who provided many insights. For administrative and secretarial help we thank M. Eul, G. Lum, J. Matheson, S. Moody, K. Pendleton, and L. Templet. To the referees for their pungent yet favorable comments we thank S. Gerhart, T. Standish, and J. Ullman. Finally, we would like to thank our institutions, the University of Southern California and the University of Minnesota, for encouraging in every way our efforts to produce this book.

Ellis Horowitz

Sartaj Sahni

Preface to the Ninth Printing

We would like to acknowledge collectively all of the individuals who have sent us comments and corrections since the book first appeared. For this printing we have made many corrections and improvements.

October 198l

Ellis Horowitz

Sartaj Sahni


Fundamentals: Table of Contents

Fundamentals of Data Structures

by Ellis Horowitz and Sartaj Sahni

PREFACE

CHAPTER 1: INTRODUCTION

CHAPTER 2: ARRAYS

CHAPTER 3: STACKS AND QUEUES

CHAPTER 4: LINKED LISTS

CHAPTER 5: TREES

CHAPTER 6: GRAPHS

CHAPTER 7: INTERNAL SORTING

CHAPTER 8: EXTERNAL SORTING

CHAPTER 9: SYMBOL TABLES

CHAPTER 10: FILES

APPENDIX A: SPARKS

APPENDIX B: ETHICAL CODE IN INFORMATION PROCESSING

APPENDIX C: ALGORITHM INDEX BY CHAPTER


DDJ Algorithms and Data Structures Articles

AUTOROUTING WITH THE A* ALGORITHM
by Randy Nevin
September 1989
Listings

SIMULATED ANNEALING
by Michael P. McLaughlin
September 1989
Listings

SELF-ADJUSTING DATA STRUCTURES
by Andrew M. Liao
February 1990
Listings

IMPLEMENTING CORDIC ALGORITHMS
by Pitts Jarvis
October 1990
Listings

ONE-WAY HASH FUNCTIONS
by Bruce Schneier
September 1991
Listings

BLOCK TRUNCATION COMPRESSION
by Anton Kruger
April 1992
Listings

AUDIO COMPRESSION
by John W. Ratcliff
July 1992
Listings
Source code

ALGORITHMS FOR STEREOSCOPIC IMAGING
by Victor J. Duvanenko and W.E. Robbins
April 1993
Listings
Source code

GENETIC ALGORITHMS AND DATABASE INDEXING
by Joe Celko
April 1993

THE IDEA ENCRYPTION ALGORITHM
by Bruce Schneier
December 1993
Listings

THE BLOWFISH ENCRYPTION ALGORITHM
by Bruce Schneier
April 1994
Listings
Data

ALGORITHM ALLEY: Rendering Circles and Ellipses
by Tim Kientzle
July 1994
Listings

ALGORITHM ALLEY: Genetic Annealing
by Kenneth V. Price
October 1994
Source code

ALGORITHM ALLEY: Truly Random Numbers
by Colin Plumb
November 1994
Listings
Source code

Real-Time Scheduling Algorithms
by Alberto Daniel Ferrari
December 1994
Listings
Source code

Morphing on Your PC
by Scott Anderson
Sourcebook, Winter 1994
Listings
Source code

The RC5 Encryption Algorithm
by Ronald L. Rivest
January 1995

ALGORITHM ALLEY: The GOST Encryption Algorithm
by Bruce Schneier
January 1995
Listings

ALGORITHM ALLEY: Faster FFTs
by J.G.G. Dobbe
February 1995
Listings
Source code

ALGORITHM ALLEY: Computing the Day of the Week
by Kim S. Larsen
April 1995

ALGORITHM ALLEY: Generating Sequential Keys in an Arbitrary Radix
by Gene Callahan
December 1995
Listings

Evaluating Data-Compression Algorithms
by G. Jason Mathews
January 1996

ALGORITHM ALLEY: Multiple Encryption: Weighing Security and Performance
by Burton S. Kaliski, Jr. and M.J.B. Robshaw
January 1996

ALGORITHM ALLEY: Binary Search
by Micha Hofri
February 1996

ALGORITHM ALLEY: A Fast Integer Square Root
by Peter Heinrich
April 1996
Listings

ALGORITHM ALLEY: Building Decision Trees with the ID3 Algorithm
by Andrew Colin
June 1996
Listings
Source code

ALGORITHM ALLEY: Minimal Perfect Hashing
by Carlo Pescio
July 1996
Listings
Source code

ALGORITHM ALLEY: Fast String Searching With Suffix Trees
by Mark R. Nelson
August 1996
Listings

ALGORITHM ALLEY: HATs: Hashed Array Trees
by Edward Sitarski
September 1996
Source code

ALGORITHM ALLEY: The Fibonacci Heap
by John Boyer
January 1997
Listings
Source code

Scheduling Algorithms and NP-Complete Problems
by Oleg Kiselyov
February 1997

Elliptic Curves and Cryptography
by Aleksandar Jurisic and Alfred J. Menezes
April 1997

ALGORITHM ALLEY: Understanding CRCs
by Tim Kientzle
April 1997
Listings

ALGORITHM ALLEY: Topological Sorting
by Jonathan Pincus and Jerry Schwarz
August 1997
Listings
Source code

The Block Cipher Square Algorithm
by Joan Daemen, Lars R. Knudsen, and Vincent Rijmen
October 1997
Listings

ALGORITHM ALLEY: Inside IMA ADPCM
by Tim Kientzle
November 1997
Listings
Source code

ALGORITHM ALLEY: Resizable Data Structures
by John Boyer
January 1998
Listings
Source code

ALGORITHM ALLEY: The Flashsort1 Algorithm
by Karl-Dietrich Neubert
February 1998
Listings

ALGORITHM ALLEY: Exponential Smoothing
by William Stallings
March 1998

ALGORITHM ALLEY: Adaptive Delta Modulation
by Gary D. Knott
April 1998
Listings

ALGORITHM ALLEY: The Cost of Recursion
by Jon Bentley
June 1998
Listings
Source code


Figure 2

Copyright © 1989, Dr. Dobb's Journal


Figure 2a

Copyright © 1989, Dr. Dobb's Journal


Figure 2b

Copyright © 1989, Dr. Dobb's Journal


Figure 2c

Copyright © 1989, Dr. Dobb's Journal


Figure 4

Copyright © 1989, Dr. Dobb's Journal


Figure 4a

Copyright © 1989, Dr. Dobb's Journal


Figure 4b

Copyright © 1989, Dr. Dobb's Journal


Figure 4c

Copyright © 1989, Dr. Dobb's Journal


Figure 5

Copyright © 1989, Dr. Dobb's Journal


SEP89: AUTOROUTING WITH THE A* ALGORITHM

AUTOROUTING WITH THE A* ALGORITHM

Searching for the best PC board layout

* This article contains the following executables: NEVIN.ZIP

Randy Nevin

Randy holds a BS in computer science from Oregon State University and an MS in computer science from the University of Washington. He has worked for Microsoft since 1983 on various programming languages and networking products. He can be reached at 1731 211th PL NE, Redmond, WA 98053.

A few years ago, a friend of mine designed an adapter board for the IBM PC. The tools he used were blue and red strips of tape, a sharp knife, large sheets of clear plastic, and a generous amount of patience. It took him several weeks, and after the first board was tested he discovered that some of the traces were incorrect and had to be cut with the knife and rerouted with a solder and wires. This caused me to start thinking about ways to use the power of the computer to simplify this tedious, error-prone job.

The design of a printed circuit board implements an electronic circuit. First you create a schematic. From this you derive a list of chips and other components that perform the required functions, and a list of the pins that need to be connected. Together, these lists are referred to as the "net-list." As long as the connections are made correctly, you usually don't care where the traces (the wires embedded in the board) are placed.

As you can imagine (or may already know), designing a PC board is a complex search problem with a seemingly infinite number of possible solutions. Luckily, there are algorithms from the field of artificial intelligence that we can use to design computer programs called "autorouters" that do this searching for you. In this article, I'll look at two algorithms: The breadth-first and A* (pronounced as "A Star") search algorithms. This article is actually based on an application I wrote to layout, view, and laser-print circuit board designs. Because of the length of that application (nearly 2500 lines of C code), I'll focus my discussion here on the pseudocode that implements the two algorithms mentioned and the source code that implements the A* algorithm. The entire C source code that implements the printed circuit board layout system is available on DDJ's Forum on CompuServe, DDJ's on-line service, and on the disks mentioned at the end of this article.

What Is Autorouting?

Autorouting is one of a class of global optimization problems that are difficult to solve. A good circuit board layout, for example, minimizes things like:

  • Physical problems (trace lengths, board size, number of routing holes, holes that transfer a trace from one side of the board to the other, also called vias)
  • Signal crosstalk
  • Number of layers
At the same time, the layout maximizes things like signal strength, reliability, and ease of debugging. The overall value of a board design is a function of all of these often conflicting variables. It is usually acceptable to find a solution that satisfies a set of constraints, because finding the globally optimal solution is infeasible for all but the most trivial problems.

Autorouting can also be viewed as a collection of search problems. The individual problems deal with finding a route and laying down a trace between two locations. There are many algorithms for solving search problems, each with different running time characteristics and data space requirements.

Autorouting search algorithms typically operate in two phases{1}, treating the board as a matrix of cells. The first phase starts at the source cell and searches for the target cell, usually by going in several directions at the same time. The algorithm builds an auxiliary data structure to keep track of how each cell was reached (this is referred to as "Pred" in the algorithms in Figure 1 and Figure 2 ). The first phase ends when the target cell has been found, and the second phase begins. If the first phase exhausts all possibilities without reaching the target cell, then no route exists between them, and there is no reason to do the second phase.

Figure 1: Pseudocode for the breadth-first algorithm

  BFS Algorithm (* breadth-first search *)
      (* Search a graph or state space, depending on the problem
         definition.  *)
      (* S is the start node, T is the goal node.  *)
      (* Open is an ordered list of nodes (ordered by arrival time;
         nodes enter at the tail and leave at the head), also called a
         queue.  Closed is a set of nodes (order doesn't matter).  In
         general, nodes that need to be searched are put on Open.  As they
         are searched, they are removed from Open and put in Closed.  *)
      (* Pred is defined for each node, and is a list of "came from"
         indications, so when we finally reach T, we traverse Pred to
         construct a path to S. *)

  1 Open <- {S} (* a list of one element *)
    Closed <- {} (* the empty set *)
    Pred [S] <- NULL, found <- FALSE
    WHILE Open <> {} and not found DO
  5   x <- the first node on Open
      Open <- Open - {x} (* remove x from Open *)
      Closed <- Closed + {x} (* put x in Closed *)
      IF x = T THEN found <- TRUE (* we're done *)
      ELSE (* continue search through node x *)
  10   let R be the set of neighboring nodes of x
       FOR each y in R DO
           IF y is not on Open or in Closed THEN
              Pred[y] <- x (* remember where we came from *)
              Open <- Open + {y} (* put y on Open (at the tail) *)
  15 IF found THEN
        use Pred[T] to find Pred[Pred[T]] and so on until S is reached
        (* this traces out the solution path in reverse *)
     ELSE T cannot be reached from S

The second phase uses the auxiliary data structure to trace the route from the target cell back to the source cell, actually laying down the electrical connections. The second phase is identical for the breadth-first and A* search algorithms. But the first phase is different, and it is this phase that gives these algorithms different behaviors.

The main data structures used in the first phase are the Open queue and the Closed set, which hold cell coordinates. Because a cell's coordinates uniquely identify it, we'll say that the Open queue and Closed set contain cells. Cell coordinates will be represented as r2c3s1 for the cell at row 2, column 3, side 1, or as r2c3 when it is understood that all cells are on the same side of the board. To remind ourselves that Open is a queue and Closed is a set, when we talk about adding cells to them, we will put the cells "on" the queue and "in" the set. Initially, the Open queue contains the source cell and the Closed set is empty.

The first phase is a loop, which removes an element from the head of the Open queue, puts it in the Closed set (which indicates that it has been searched), and checks to see if it is the target cell. If so, the first phase is done. Otherwise, the neighbors of the cell (those adjacent to it) are placed on the Open queue, and the loop continues. As we'll see later, the essential difference in the breadth-first and A* search algorithms is the order in which the neighbors are placed on the Open queue.

Breadth-First Search

Figure 1 contains pseudocode for the breadth-first search algorithm. This algorithm works by processing a first in/first out (FIFO) queue of open cells; that is, cells that have been reached, but not yet searched. Initially, the open queue contains only the source cell. A cell is removed from the head of the open queue, placed in the set of closed cells (cells that have been searched), and checked to see if it is the target cell. If not, its neighbors are placed at the tail of the open queue. Neighboring cells that have already been reached are ignored. (If a cell's coordinates are on the open queue or in the closed set, then it has been reached, otherwise, it has not.) This continues until one of two things happens:

  • The target cell has been found
  • The open queue is empty, in which case the target cannot be reached from the source cell
A version of breadth-first search known as Lee's algorithm{2} has served as the basis for some autorouters since the early 1960s. The original algorithm does not consider diagonally adjacent cells as neighbors, and consequently, the back-tracking phase can create only horizontal and vertical traces. We'll enhance the algorithm so that diagonally adjacent cells are neighbors, thus enabling it to produce diagonal traces. Unfortunately, Lee's algorithm suffers from a behavior inherent in the breadth-first search technique, which limits its application to problems of relatively small size. As the distance between the source and target cells increases by a factor of N, the number of cells processed by Lee's algorithm -- and therefore processing time -- increases by the square of N.

Figure 2 shows the behavior of Lee's algorithm while searching for a path between the source cell S (r5c5) and the target cell T (r8c8). Lee's algorithm does not specify the order in which neighboring cells are placed on the open queue, but we'll use the compass directions north, east, south, and west, followed by northeast, southeast, southwest, and northwest. This order tends to produce traces with a minimal number of turns.

In Figure 2a, the source cell (r5c5) has been searched, and its eight neighbors have been placed on the open queue. The arrows indicate the direction from which each cell was reached, and correspond to the Pred data structure. After the first eight cells on the open queue have been reached and moved to the closed set, the algorithm searches the configuration in Figure 2b, where there are sixteen cells on the open queue. Once these sixteen cells have been searched, the configuration in Figure 2c is reached. Now the target cell (r8c8) is fourth from the end on the open queue, and a solution is imminent. Searching r8c8, the algorithm recognizes it as the target cell, and uses the Pred data structure to construct a trace back to the source cell.

You can see that the search progresses outward from the source cell in all directions, like ripples when you throw a pebble into the water. If we double the size of the problem so that S and T are six cells apart, the number of cells searched and therefore the processing time will be about four times as great. If we triple the size of the problem, the number of cells searched will be roughly nine times more. Thus, the behavior of Lee's algorithm is quadratic in the size of the problem, which makes it infeasible for large problems.

A* Search

Figure 3 gives pseudocode for the A* search algorithm, while Listing One shows it implemented in C. This method also works by processing a queue of open cells, which initially contains only the source cell. But this is a priority queue, which means cells are inserted according to the estimated distance to the target{3}, not just at the end. Cells that are on the shortest estimated path from source to target go to the head of the queue. The A* algorithm removes the cell from the head of the open queue and checks to see if it's the target. If not, the neighboring cells are put on the open queue at the proper position. The algorithm checks neighboring cells that have already been searched to see if the new path between them and the source is shorter than the previous one. If it is, they are repositioned on the open queue according to the new estimated path length from source to target. As in breadth-first search, this continues until the target cell has been found or the open queue is empty.

Figure 3: Pseudocode for the A* algorithm

  A* Algorithm (* heuristic search *)
     (* Search a graph or state space, depending on the problem
        definition.  *)
     (* S is the start node, T is the goal node.  *)
     (* Open is an ordered list of nodes (ordered by lowest F value;
        see below), also called a priority queue.  Closed is a set of
        nodes (order doesn't matter).  In general, nodes that need to be
        searched are put on Open (at the proper position).  As they are
        searched, they are removed from Open and put in Closed.
        Occasionally a newer, better route will be found to a node after
        it has already been searched, in which case we remove it from
        Closed and put it back on Open to be reconsidered.  *)
     (* G[x] is the distance already traveled to get from S to node x,
        and is known exactly.  H(x) is a function (heuristic) which
        returns an estimate of the distance from node x to T.  F[x] is
        the estimated distance from S to T by going through node x, and
        is computed by F[x] = G[x] + H(x).  H(x) can be calculated for
        any node, but F[x] and G[x] only become defined when node x is
        visited.  *)
     (* Pred is defined for each node, and is a list of "came from"
        indications, so when we finally reach T, we traverse Pred to
        construct a path to S.  *)
     (* Distance (x,y) is a function for calculating the distance
        between two neighboring nodes.  *)
  1 Open <- {S} (* a list of one element *)
    Closed <- {} (* the empty set *)
    G[S] <- 0, F[S] <- 0, Pred[S] <- NULL, found <- FALSE
    WHILE Open <> {} and not found DO
  5    x <- the first node on Open (* node with smallest F value *)
       Open <- Open - {x} (* remove x from Open *)
       Closed <- Closed + {x} (* put x in Closed *)
       IF x = T THEN found <- TRUE (* we're done *)
       ELSE (* continue search through node x *)
  10           let R be the set of neighboring nodes of x
               FOR each y in R DO
                  IF y is not on Open or in Closed THEN
                     G[y] <- G[x] + Distance (x,y)
                     F[y] <- G[y] + H(y) (* estimate solution path
                      length *)
  15                 Pred[y] <- x (* remember where we came from *)
                     Open <- Open + {y} (* put y on Open *)
                  ELSE (* y is on Open or in Closed *)
                     IF (G[x] + Distance (x,y)) < G[y] THEN
                        (* we've found a better route to y *)
  20                    G[y] <- G[x] + Distance (x,y)
                        F[y] <- G[y] + H(y)
                        Pred[y] <- x (* remember where we came from *)
                        IF y is on Open THEN
                           reposition y according to F[y]
  25                    ELSE (* y is in Closed *)
                           Closed <- Closed - {y} (* remove y from Closed *)
                           Open <- Open + {y} (* put y on Open *)
     IF found THEN
        use Pred[T] to find Pred[Pred[T]] and so on until S is reached
  30    (* this traces out the solution path in reverse *)
     ELSE T cannot be reached from S

A* depends on being able to estimate the distance between a cell and the target cell. In the case of autorouting, a simple measure of this distance is available, and this helps A* to concentrate the search in the direction most likely to succeed. The more accurate the estimate, the faster the search.

In practice, A* does not suffer from the quadratic behavior of Lee's algorithm, it solves similar problems faster and can be applied to larger problems where Lee's algorithm performs poorly. As the distance between the source and target cells increases, the number of cells processed by A* increases, but not as dramatically as with Lee's algorithm.

Figure 4 shows the behavior of the A* search algorithm. A* does not specify whether new cells go in front of or behind cells already on the open queue that evaluate to identical estimated path lengths. We use the convention that they are placed in front. This minimizes the time to insert a cell on the open queue.

In Figure 4a, the source cell (r3c3) has been searched, and its eight neighbors are on the open queue. Each cell on the open queue also includes the estimated length of the shortest path from S to T that goes through that cell. After the first cell (r4c4) has been searched and moved to the closed set, the configuration in Figure 4b is reached, where there are 12 cells on the open queue. After searching the next cell (r5c5), the algorithm reaches the configuration in Figure 4c. Now the target cell (r6c6) is at the head of the open queue, and a solution will be found on the next iteration of the loop. Searching r6c6, A* recognizes it as the target and uses the Pred data structure to construct a trace back to the source cell.

You can see that the search progresses more directly toward the target cell. The target draws the search much as the earth's gravity pulls objects toward the center of mass. If we double the size of the problem, the search will process about twice as many cells, and if we triple its size, the search will run through three times as many. This linear behavior makes A* more attractive for autorouting than the quadratic Lee's algorithm. With the incorporation of the heuristic -- the rule, that guides the search in the direction most likely to succeed -- it is difficult to estimate worst case behavior. However, A* will never take more time than Lee's algorithm, and it will never search any cells that Lee's algorithm could avoid.

Optimizations and Generalizations

The algorithms in Figures 1 and 3 solve the general search problem. When these algorithms are implemented and customized to a particular application, there are ways to speed them up.

The A* algorithm in Figure 3 recomputes the heuristic H(y) when it discovers a better way to reach a cell. Depending on how difficult this heuristic is to compute, you can probably save some work at the expense of complicating the algorithm. When lines 20 and 21 of Figure 3 are executed, the previous values of G[y] and F[y] are destroyed. But F[y] = G[y] + H(y), so you could save F[y] - G[y] (which is H(y)) in a temporary variable, and use that variable instead of recomputing H(y) on line 21. Also, the common sub-expression G[x] + Distance(x,y) should be placed in a temporary variable, instead of being computed twice (lines 18 and 20).

Often, rather than searching for a path between two individual cells, what is really desired is a path between one of a set of source cells and one of a set of target cells (as when connecting power and ground pins). Both algorithms can be modified by adding the entire set of source cells to the initial open queue, and checking for a member of the set of target cells on each iteration. When this is done, the heuristic used by the A* algorithm becomes more complicated. It must estimate the minimum distance from the current cell to any one of the target cells.

For breadth-first search, once the target cell is placed on the open queue, it is pointless to add any more cells to the open queue because when this happens, the problem is solved. An appropriate shortcut would be to insert a check before line 13 in Figure 1 to see if y is the target cell. If it is, use Pred[y] to construct the trace back to the source cell, and return.

Memory Requirements

Both search algorithms use quite a bit of memory to solve problems of non-trivial size. The breadth-first search algorithm needs memory to represent the board, the predecessor structure, and the closed set. The A* search algorithm needs these also, plus structures for F[x] and G[x] (see Figure 3). In addition, both algorithms dynamically allocate memory for the open cell queue.

The board is represented as a pair of two-dimensional arrays -- one for the front side, the other for the back -- in which the dimensions are the number of rows and columns of cells. Not counting holes and traces relating to holes (Figure 5, groups A, B, and C), there are 30 possible cell contents, which can be represented with 5 bits per cell.

The hole-related cells are more difficult to enumerate; they can be combined in many ways. If we simply assign 1 bit to each of the eight traces in groups B and C, and add one more bit to indicate a hole, 14 bits will be sufficient to represent any cell. On a board of N rows and M columns, we'll need N*M*28 bits total.

The predecessor structure is also a pair of two-dimensional arrays, where an entry must be able to represent one of the eight compass directions or an indication for the opposite side of the board. This takes 4 bits per cell, or N*M*8 bits total.

The closed set can be represented by a pair of two-dimensional, single-bit arrays, where a bit is one if the corresponding cell has been searched, and zero otherwise. This will take N*M*2 bits total.

F[x] and G[x] will be similar to the board arrays, but they must contain a 16-bit integer for each cell, requiring N*M*64 bits total. Note that if memory usage needs to be minimized at the cost of increased processing time, we could omit the F[x] arrays, and calculate the F values as they are needed from the G[x] arrays and the heuristic function, H(x).

Breadth-first search thus requires N*M*38 bits, and A* needs N*M*102 bits of static memory. For a printed circuit board 4 x 13 inches (80 cells x 260 cells), breadth-first search will need 98,800 bytes and A* will need 265,200 bytes. Different algorithms that solve the same problem often trade off memory against processing time to achieve better performance. This is the case with A* versus the breadth-first search.</entry>

More Details.

Locality of Reference

Despite the fact that A* requires more memory than breadth-first search, A* exhibits better memory usage patterns. This is because it shows better locality of reference than breadth-first search. Locality of reference deals with the sequence in which memory locations are used, and consists of two rules of thumb: 1. The memory location currently being referenced is likely to be referenced again in the near future, and 2. Memory locations near the one currently being referenced are likely to be referenced in the near future.

When the first rule holds true for a given program, that program can probably benefit from a memory cache. When the second rule holds true, the program can probably benefit from a virtual memory environment with a least-recently-used page preemption policy. Most computer systems with virtual memory and caches apply them to both code and data, so programs that exhibit good locality of reference should benefit from both rules.

This becomes a factor when solving large problems (say, routing a printed circuit board that is 10 inches in both dimensions). In a virtual memory environment, improved locality of reference can minimize swapping. In an environment with cache memory, improved locality of reference can increase the cache hit rate. Both of these tend to decrease the total running time.

The memory references in the breadth-first search algorithm go around and around in circles of constantly increasing size, and do not reflect a common locality of reference. Thus, the breadth-first search algorithm is not able to take good advantage of virtual memory or a memory cache. The memory references of A* tend to be from the same area of the printed circuit board for extended periods of time, taking better advantage of these mechanisms. In a large problem, this helps to offset the extra memory that A* requires by adding speed beyond that provided by the basic algorithm. Improved locality of reference by itself may not be a sufficient reason to select A* over breadth-first search, but it is icing on the cake.

References

    1. Stephen E. Belter, "Computer-aided Routing of Printed Circuit Boards: an Examination of Lee's Algorithm and Possible Enhancements," BYTE, (June 1987), 199 - 208.

    2. C.Y. Lee, "An Algorithm for Path Connections and Its Applications," IRE Transactions on Electronic Computers," (September 1961), 346 - 365.

    3. Steven L. Tanimoto, The Elements of Artificial Intelligence, (1987, Rockville, Maryland: Computer Science Press), 148 - 164. This covers the breadth-first and A* search algorithms.

Distance Calculations

The A* search algorithm uses a heuristic to estimate the distance between the current cell and the target cell. As implemented in the autorouting program, the heuristic is a simple geometric distance approximation. Figure 5 illustrates all the possible cell types used to construct a trace, grouped by type. For each group, the distance of that cell type is also given. These distances are calculated based on a cell size of 50 x 50 mils. (A mil is 1/1000 inch, so the autorouter uses 20 cells/inch. A typical full-length adapter board for an IBM PC is 4-inches high and 13-inches long, or 80-cell rows x 260-cell columns.)

The group B and C traces can coexist in the same cell, so a hole can have up to 16 traces connecting it with other cells (eight on each side of the board). Also, the parallel traces of group F can coexist in the same cell (on the same side of the board), as shown by group J. This allows the routing of two traces through the same cell, providing the higher density required by some circuits (memory arrays, for example). Aside from these exceptions, cells can only contain one trace type (on each side of the board).

To determine the approximate distance of a trace that will connect two holes, view the board as a matrix, the differences in cell coordinates are three rows and five columns. The shortest path between them will use a diagonal trace across three cells and a horizontal trace across two cells. Using the cell types in Figure 5, the length of the trace will be 23 + (2 * 71) + 60 + 50 + 12 = 287 mils. A trace that uses a routing hole to go from one side of the board to the other covers a greater distance than one that goes diagonally across a cell (group E in Figure 5) and stays on the same side of the board because part of its path goes around the edge of a circle.

A typical hole is 25 mils in diameter, and is at the center of a cell. To calculate the distance of a trace through a routing hole, measure the section of the hole between the two connecting traces. For instance, an entering trace can connect to a hole at a point A, and possible exiting traces on the opposite side of the board at points B, C, D, and E. The distances between A and each of these points are 10, 20, 29, and 39 mils, respectively. To calculate these, use the geometric formula Circumference = PI * Diameter (approximately 78.5 mils) and divide by 8 (a 1/8 section of a hole is approximately 9.8 mils), add 1, 2, 3, and 4 of these sections, then round off to an integer.

The heuristic in the autorouting program includes a penalty when a trace takes a turn or switches to the other side of the board through a routing hole because turns are often the weak points in a circuit, and traces are more likely to break at a turn than in a straight part. The penalty encourages A* to use straight lines, and even allows a slightly longer trace to be selected over one with too many turns. The amount of penalty depends on the kind of turn; sharper turns are assessed a larger penalty. Routing holes incur a significant penalty, since overusing them early in the routing process can make later traces more difficult or even impossible to construct because a routing hole dedicates a cell exclusively to a single trace, for both sides of the board. Such a cell is not available to later routing, thus reducing the total number of cells that can be used. -- R.N.

_AUTOROUTING WITH THE A* ALGORITHM_ by Randy Nevin * PC BOARD LAYOUT SYSTEM TO ACCOMPANY _AUTOROUTING WITH THE A* ALGORITHM_ BY RANDY NEVIN IS ATTACHED TO THE END OF THE FOLLOWING LISTINGS [LISTING ONE]

/*
** printed circuit board autorouter, Copyright (C) Randy Nevin 1989.
** you may give this software to anyone, make as many copies as you
** like, and post it on public computer bulletin boards and file
** servers. you may not sell it or charge any fee for distribution
** (except for media and postage), remove this comment or the
** copyright notice from the code, or claim that you wrote this code
** or anything derived from it.
** the author's address is: Randy Nevin, 1731 211th PL NE, Redmond,
** WA 98053. this code is available directly from the author; just
** send a floppy and a self-addressed floppy mailer with sufficient
** postage. however, you should first attempt to get a copy of this
** software package from the Dr. Dobb's Journal BBS.
*/
/* the low-order bit indicates a hole */
#define HOLE       0x00000001L /* a conducting hole */
/* traces radiating outward from a hole to a side or corner */
#define HOLE_NORTH    0x00000002L /* upward             */
#define HOLE_NORTHEAST    0x00000004L /* upward and right   */
#define HOLE_EAST    0x00000008L /* to the right       */
#define HOLE_SOUTHEAST    0x00000010L /* downward and right */
#define HOLE_SOUTH    0x00000020L /* downward           */
#define HOLE_SOUTHWEST    0x00000040L /* downward and left  */
#define HOLE_WEST    0x00000080L /* to the left        */
#define HOLE_NORTHWEST    0x00000100L /* upward and left    */

/* straight lines through the center */
#define LINE_HORIZONTAL    0x00000002L /* left-to-right line */
#define LINE_VERTICAL    0x00000004L /* top-to-bottom line */

/* lines cutting across a corner, connecting adjacent sides */
#define CORNER_NORTHEAST 0x00000008L /* upper right corner */
#define CORNER_SOUTHEAST 0x00000010L /* lower right corner */
#define CORNER_SOUTHWEST 0x00000020L /* lower left corner  */
#define CORNER_NORTHWEST 0x00000040L /* upper left corner  */

/* diagonal lines through the center */
#define DIAG_NEtoSW    0x00000080L /* northeast to southwest */
#define DIAG_SEtoNW    0x00000100L /* southeast to northwest */

/* 135 degree angle side-to-far-corner lines */
#define BENT_NtoSE    0x00000200L /* north to southeast */
#define BENT_NtoSW    0x00000400L /* north to southwest */
#define BENT_EtoSW    0x00000800L /* east to southwest  */
#define BENT_EtoNW    0x00001000L /* east to northwest  */
#define BENT_StoNW    0x00002000L /* south to northwest */
#define BENT_StoNE    0x00004000L /* south to northeast */
#define BENT_WtoNE    0x00008000L /* west to northeast  */
#define BENT_WtoSE    0x00010000L /* west to southeast  */

/* 90 degree corner-to-adjacent-corner lines */
#define ANGLE_NEtoSE    0x00020000L /* northeast to southeast */
#define ANGLE_SEtoSW    0x00040000L /* southeast to southwest */
#define ANGLE_SWtoNW    0x00080000L /* southwest to northwest */
#define ANGLE_NWtoNE    0x00100000L /* northwest to northeast */

/* 45 degree angle side-to-near-corner lines */
#define SHARP_NtoNE    0x00200000L /* north to northeast */
#define SHARP_EtoNE    0x00400000L /* east to northeast  */
#define SHARP_EtoSE    0x00800000L /* east to southeast  */
#define SHARP_StoSE    0x01000000L /* south to southeast */
#define SHARP_StoSW    0x02000000L /* south to southwest */
#define SHARP_WtoSW    0x04000000L /* west to southwest  */
#define SHARP_WtoNW    0x08000000L /* west to northwest  */
#define SHARP_NtoNW    0x10000000L /* north to northwest */

/* directions the cell can be reached from (point to previous cell) */
#define FROM_NORTH   1
#define FROM_NORTHEAST   2
#define FROM_EAST   3
#define FROM_SOUTHEAST   4
#define FROM_SOUTH   5
#define FROM_SOUTHWEST   6
#define FROM_WEST   7
#define FROM_NORTHWEST   8
#define FROM_OTHERSIDE   9

#define   TOP   0
#define BOTTOM   1
#define EMPTY   0
#define ILLEGAL   -1

/* visit neighboring cells in this order
** (where [9] is on the other side):
**   +---+---+---+
**   | 1 | 2 | 3 |
**   +---+---+---+
**   | 4 |[9]| 5 |
**   +---+---+---+
**   | 6 | 7 | 8 |
**   +---+---+---+
*/

static int delta[8][2] = { /* for visiting neighbors on the same side */
    {  1, -1 }, /* northwest   */
    {  1,  0 }, /* north      */
    {  1,  1 }, /* northeast   */
    {  0, -1 }, /* west      */
    {  0,  1 }, /* east      */
    { -1, -1 }, /* southwest   */
    { -1,  0 }, /* south      */
    { -1,  1 }  /* southeast   */
   };

static int ndir[8] = { /* for building paths back to source */
    FROM_SOUTHEAST, FROM_SOUTH, FROM_SOUTHWEST,
    FROM_EAST,                  FROM_WEST,
    FROM_NORTHEAST, FROM_NORTH, FROM_NORTHWEST
   };

/* blocking masks for neighboring cells */
#define BLOCK_NORTHEAST   ( DIAG_NEtoSW | BENT_StoNE | BENT_WtoNE \
         | ANGLE_NEtoSE | ANGLE_NWtoNE \
         | SHARP_NtoNE | SHARP_EtoNE )
#define BLOCK_SOUTHEAST   ( DIAG_SEtoNW | BENT_NtoSE | BENT_WtoSE \
         | ANGLE_NEtoSE | ANGLE_SEtoSW \
         | SHARP_EtoSE | SHARP_StoSE )
#define BLOCK_SOUTHWEST   ( DIAG_NEtoSW | BENT_NtoSW | BENT_EtoSW \
         | ANGLE_SEtoSW | ANGLE_SWtoNW \
         | SHARP_StoSW | SHARP_WtoSW )
#define BLOCK_NORTHWEST   ( DIAG_SEtoNW | BENT_EtoNW | BENT_StoNW \
         | ANGLE_SWtoNW | ANGLE_NWtoNE \
         | SHARP_WtoNW | SHARP_NtoNW )
struct block {
   int r1, c1;  long b1, h1;
   int r2, c2;  long b2, h2;
   };
static struct block blocking[8] = { /* blocking masks */
    {  0, -1, BLOCK_NORTHEAST, HOLE_NORTHEAST,
       1,  0, BLOCK_SOUTHWEST, HOLE_SOUTHWEST },
    {  0,  0, 0, 0, 0, 0, 0, 0 },
    {  1,  0, BLOCK_SOUTHEAST, HOLE_SOUTHEAST,
       0,  1, BLOCK_NORTHWEST, HOLE_NORTHWEST },
    {  0,  0, 0, 0, 0, 0, 0, 0 },
    {  0,  0, 0, 0, 0, 0, 0, 0 },
    {  0, -1, BLOCK_SOUTHEAST, HOLE_SOUTHEAST,
      -1,  0, BLOCK_NORTHWEST, HOLE_NORTHWEST },
    {  0,  0, 0, 0, 0, 0, 0, 0 },
    { -1,  0, BLOCK_NORTHEAST, HOLE_NORTHEAST,
       0,  1, BLOCK_SOUTHWEST, HOLE_SOUTHWEST }
   };
static int selfok[5][8] = { /* mask for self-blocking corner effects */
    { 1, 1, 1, 1, 1, 1, 1, 1 },
    { 0, 0, 0, 0, 1, 0, 1, 0 },
    { 0, 0, 0, 1, 0, 0, 1, 0 },
    { 0, 1, 0, 0, 1, 0, 0, 0 },
    { 0, 1, 0, 1, 0, 0, 0, 0 }
   };
static long newmask[5][8] = { /* patterns to mask in neighbor cells */
    { 0, 0, 0, 0, 0, 0, 0, 0 },
    { 0, 0, 0, 0, CORNER_NORTHEAST | CORNER_SOUTHEAST, 0,
      CORNER_SOUTHEAST | CORNER_SOUTHWEST, 0 },
    { 0, 0, 0, CORNER_SOUTHWEST | CORNER_NORTHWEST, 0, 0,
      CORNER_SOUTHEAST | CORNER_SOUTHWEST, 0 },
    { 0, CORNER_NORTHEAST | CORNER_NORTHWEST, 0, 0,
      CORNER_NORTHEAST | CORNER_SOUTHEAST, 0, 0, 0 },
    { 0, CORNER_NORTHEAST | CORNER_NORTHWEST, 0,
      CORNER_SOUTHWEST | CORNER_NORTHWEST, 0, 0, 0, 0 }
   };
/* board dimensions */
extern int Nrows;
extern int Ncols;

void Solve () { /* route all traces */
   int r1, c1, r2, c2, r, c, s, d, a, nr, nc, skip;
   register int i;
   char far *n1;
   char far *n2;
   long curcell, newcell, buddy;
   int newdist, olddir, success, self;

   /* go until no more work to do */
   for (GetWork( &r1, &c1, &n1, &r2, &c2, &n2 ); r1 != ILLEGAL;
         GetWork( &r1, &c1, &n1, &r2, &c2, &n2 )) {
      if (r1 == r2 && c1 == c2) /* already routed */
         continue;
      success = 0;
      InitQueue(); /* initialize the search queue */
      /* get rough estimate of trace distance */
      a = GetApxDist( r1, c1, r2, c2 );
      SetQueue( r1, c1, TOP, 0, a, r2, c2 );
      SetQueue( r1, c1, BOTTOM, 0, a, r2, c2 );
      /* search until success or we exhaust all possibilities */
      for (GetQueue( &r, &c, &s, &d, &a ); r != ILLEGAL;
         GetQueue( &r, &c, &s, &d, &a )) {
         if (r == r2 && c == c2) { /* success! */
            /* lay traces */
            Retrace( r1, c1, r2, c2, s );
            success++;
            break;
            }
         curcell = GetCell( r, c, s );
         if (curcell & CORNER_NORTHWEST)
            self = 1;
         else if (curcell & CORNER_NORTHEAST)
            self = 2;
         else if (curcell & CORNER_SOUTHWEST)
            self = 3;
         else if (curcell & CORNER_SOUTHEAST)
            self = 4;
         else
            self = 0;
         /* consider neighbors */
         for (i = 0; i < 8; i++) {
            /* check self-block */
            if (!selfok[self][i])
               continue;
            if ((nr = r+delta[i][0]) < 0
               || nr >= Nrows
               || (nc = c+delta[i][1]) < 0
               || nc >= Ncols)
               /* off the edge */
               continue;
            newcell = GetCell( nr, nc, s );
            /* check for non-target hole */
            if (newcell & HOLE) {
               if (nr != r2 || nc != c2)
                  continue;
               }
            else {
               newcell &= ~(newmask[self][i]);
               /* check for traces */
               if (newcell)
                  continue;
               }
            /* check blocking on corner neighbors */
            if (delta[i][0] && delta[i][1]) {
               /* check first buddy */
               buddy = GetCell( r+blocking[i].r1,
                  c+blocking[i].c1, s );
               if (buddy & HOLE) {
                  if (buddy & (blocking[i].h1))
                     continue;
                  }
               else if (buddy & (blocking[i].b1))
                  continue;
               /* check second buddy */
               buddy = GetCell( r+blocking[i].r2,
                  c+blocking[i].c2, s );
               if (buddy & HOLE) {
                  if (buddy & (blocking[i].h2))
                     continue;
                  }
               else if (buddy & (blocking[i].b2))
                  continue;
               }
            olddir = GetDir( r, c, s );
            newdist = d+CalcDist( ndir[i], olddir,
               (olddir == FROM_OTHERSIDE)
               ? GetDir( r, c, 1-s ) : 0 );
            /* if not visited yet, add it to queue */
            if (!GetDir( nr, nc, s )) {
               SetDir( nr, nc, s, ndir[i] );
               SetDist( nr, nc, s, newdist );
               SetQueue( nr, nc, s, newdist,
                  GetApxDist( nr, nc, r2, c2 ),
                  r2, c2 );
               }
            /* we might have found a better path */
            else if (newdist < GetDist( nr, nc, s )) {
               SetDir( nr, nc, s, ndir[i] );
               SetDist( nr, nc, s, newdist );
               ReSetQueue( nr, nc, s, newdist,
                  GetApxDist( nr, nc, r2, c2 ),
                  r2, c2 );
               }
            }
         /* consider other side of board */
         /* check for holes or traces on other side */
         if (newcell = GetCell( r, c, 1-s ))
            continue;
         skip = 0;
         /* check for nearby holes */
         for (i = 0; i < 8; i++) {
            if ((nr = r+delta[i][0]) < 0
               || nr >= Nrows
               || (nc = c+delta[i][1]) < 0
               || nc >= Ncols)
               /* off the edge */
               continue;
            if (GetCell( nr, nc, s ) & HOLE) {
               /* neighboring hole */
               skip = 1;
               break;
               }
            }
         if (skip) /* neighboring hole? */
            continue; /* yes, can't drill one here */
         olddir = GetDir( r, c, s );
         newdist = d+CalcDist( FROM_OTHERSIDE, olddir,
            (olddir == FROM_OTHERSIDE)
            ? GetDir( r, c, 1-s ) : 0 );
         /* if not visited yet, add it to queue */
         if (!GetDir( r, c, 1-s )) {
            SetDir( r, c, 1-s, FROM_OTHERSIDE );
            SetDist( r, c, 1-s, newdist );
            SetQueue( r, c, 1-s, newdist,
               a, r2, c2 );
            }
         /* we might have found a better path */
         else if (newdist < GetDist( r, c, 1-s )) {
            SetDir( r, c, 1-s, FROM_OTHERSIDE );
            SetDist( r, c, 1-s, newdist );
            ReSetQueue( r, c, 1-s, newdist,
               a, r2, c2 );
            }
         }
      if (!success)
         printf( "\t*!* UNSUCCESSFUL *!*\n" );
      /* clear direction flags */
      for (r = 0; r < Nrows; r++) {
         for (c = 0; c < Ncols; c++) {
            SetDir( r, c, TOP, EMPTY );
            SetDir( r, c, BOTTOM, EMPTY );
            }
         }
      }
   }
/* this table drives the retracing phase */
static long bit[8][9] = { /* OT=Otherside */
    /* N, NE, E, SE, S, SW, W, NW, OT */
/* N */  { LINE_VERTICAL, BENT_StoNE, CORNER_SOUTHEAST, SHARP_StoSE,
      0, SHARP_StoSW, CORNER_SOUTHWEST, BENT_StoNW,
      (HOLE | HOLE_SOUTH) },
/* NE */ { BENT_NtoSW, DIAG_NEtoSW, BENT_EtoSW, ANGLE_SEtoSW,
      SHARP_StoSW, 0, SHARP_WtoSW, ANGLE_SWtoNW,
      (HOLE | HOLE_SOUTHWEST) },
/* E */  { CORNER_NORTHWEST, BENT_WtoNE, LINE_HORIZONTAL, BENT_WtoSE,
      CORNER_SOUTHWEST, SHARP_WtoSW, 0, SHARP_WtoNW,
      (HOLE | HOLE_WEST) },
/* SE */ { SHARP_NtoNW, ANGLE_NWtoNE, BENT_EtoNW, DIAG_SEtoNW,
      BENT_StoNW, ANGLE_SWtoNW, SHARP_WtoNW, 0,
      (HOLE | HOLE_NORTHWEST) },
/* S */  { 0, SHARP_NtoNE, CORNER_NORTHEAST, BENT_NtoSE,
      LINE_VERTICAL, BENT_NtoSW, CORNER_NORTHWEST, SHARP_NtoNW,
      (HOLE | HOLE_NORTH) },
/* SW */ { SHARP_NtoNE, 0, SHARP_EtoNE, ANGLE_NEtoSE, BENT_StoNE,
      DIAG_NEtoSW, BENT_WtoNE, ANGLE_NWtoNE,
      (HOLE | HOLE_NORTHEAST) },
/* W */  { CORNER_NORTHEAST, SHARP_EtoNE, 0, SHARP_EtoSE,
      CORNER_SOUTHEAST, BENT_EtoSW, LINE_HORIZONTAL, BENT_EtoNW,
      (HOLE | HOLE_EAST) },
/* NW */ { BENT_NtoSE, ANGLE_NEtoSE, SHARP_EtoSE, 0, SHARP_StoSE,
           ANGLE_SEtoSW, BENT_WtoSE, DIAG_SEtoNW,
      (HOLE | HOLE_SOUTHEAST) }
   };

void Retrace ( rr1, cc1, rr2, cc2, s )
   /* work from target back to source, laying down the trace */
   int rr1, cc1, rr2, cc2, s; /* start on side s */
   {
   int r0, c0, s0, r1, c1, s1, r2, c2, s2;
   register int x, y;
   long b;

   r1 = rr2;
   c1 = cc2;
   s1 = s;
   r0 = c0 = s0 = ILLEGAL;
   do {
      /* find where we came from to get here */
      switch (x = GetDir( r2 = r1, c2 = c1, s2 = s1 )) {
      case FROM_NORTH:   r2++;      break;
      case FROM_EAST:         c2++;   break;
      case FROM_SOUTH:   r2--;      break;
      case FROM_WEST:         c2--;   break;
      case FROM_NORTHEAST:   r2++;   c2++;   break;
      case FROM_SOUTHEAST:   r2--;   c2++;   break;
      case FROM_SOUTHWEST:   r2--;   c2--;   break;
      case FROM_NORTHWEST:   r2++;   c2--;   break;
      case FROM_OTHERSIDE:   s2 = 1-s2;   break;
      default:
         fprintf( stderr, "internal error\n" );
         exit( -1 );
         break;
         }
      if (r0 != ILLEGAL)
         y = GetDir( r0, c0, s0 );
      /* see if target or hole */
      if ((r1 == rr2 && c1 == cc2) || (s1 != s0)) {
         switch (x) {
         case FROM_NORTH:
           OrCell( r1, c1, s1, HOLE_NORTH );  break;
         case FROM_EAST:
           OrCell( r1, c1, s1, HOLE_EAST );  break;
         case FROM_SOUTH:
           OrCell( r1, c1, s1, HOLE_SOUTH );  break;
         case FROM_WEST:
           OrCell( r1, c1, s1, HOLE_WEST );  break;
         case FROM_NORTHEAST:
           OrCell( r1, c1, s1, HOLE_NORTHEAST );  break;
         case FROM_SOUTHEAST:
           OrCell( r1, c1, s1, HOLE_SOUTHEAST );  break;
         case FROM_SOUTHWEST:
           OrCell( r1, c1, s1, HOLE_SOUTHWEST );  break;
         case FROM_NORTHWEST:
           OrCell( r1, c1, s1, HOLE_NORTHWEST );  break;
         case FROM_OTHERSIDE:
         default:
            fprintf( stderr, "internal error\n" );
            exit( -1 );
            break;
            }
         }
      else {
         if ((y == FROM_NORTH
            || y == FROM_NORTHEAST
            || y == FROM_EAST
            || y == FROM_SOUTHEAST
            || y == FROM_SOUTH
            || y == FROM_SOUTHWEST
            || y == FROM_WEST
            || y == FROM_NORTHWEST)
            && (x == FROM_NORTH
            || x == FROM_NORTHEAST
            || x == FROM_EAST
            || x == FROM_SOUTHEAST
            || x == FROM_SOUTH
            || x == FROM_SOUTHWEST
            || x == FROM_WEST
            || x == FROM_NORTHWEST
            || x == FROM_OTHERSIDE)
            && (b = bit[y-1][x-1])) {
            OrCell( r1, c1, s1, b );
            if (b & HOLE)
               OrCell( r2, c2, s2, HOLE );
            }
         else {
            fprintf( stderr, "internal error\n" );
            exit( -1 );
            }
         }
      if (r2 == rr1 && c2 == cc1) { /* see if source */
         switch (x) {
         case FROM_NORTH:
           OrCell( r2, c2, s2, HOLE_SOUTH );  break;
         case FROM_EAST:
           OrCell( r2, c2, s2, HOLE_WEST );  break;
         case FROM_SOUTH:
           OrCell( r2, c2, s2, HOLE_NORTH );  break;
         case FROM_WEST:
           OrCell( r2, c2, s2, HOLE_EAST );  break;
         case FROM_NORTHEAST:
           OrCell( r2, c2, s2, HOLE_SOUTHWEST );  break;
         case FROM_SOUTHEAST:
           OrCell( r2, c2, s2, HOLE_NORTHWEST );  break;
         case FROM_SOUTHWEST:
           OrCell( r2, c2, s2, HOLE_NORTHEAST );  break;
         case FROM_NORTHWEST:
           OrCell( r2, c2, s2, HOLE_SOUTHEAST );  break;
         case FROM_OTHERSIDE:
         default:
            fprintf( stderr, "internal error\n" );
            exit( -1 );
            break;
            }
         }
      /* move to next cell */
      r0 = r1; c0 = c1; s0 = s1;
      r1 = r2; c1 = c2; s1 = s2;
      } while (!(r2 == rr1 && c2 == cc1));
   }

int GetApxDist ( r1, c1, r2, c2 ) /* calculate approximate distance */
   int r1, c1, r2, c2;
   {
   register int d1, d2; /* row and column deltas */
   int d0; /* temporary variable for swapping d1 and d2 */
   /* NOTE: the -25 used below is because we are not going */
   /* from the center of (r1,c1) to the center of (r2,c2), */
   /* we are going from the edge of a hole at (r1,c1) to   */
   /* the edge of a hole at (r2,c2). holes are 25 mils in  */
   /* diameter (12.5 mils in radius), so we back off by 2  */
   /* radii.                                               */
   if ((d1 = r1-r2) < 0) /* get absolute row delta */
      d1 = -d1;
   if ((d2 = c1-c2) < 0) /* get absolute column delta */
      d2 = -d2;
   if (!d1) /* in same row? */
      return( (d2*50)-25 ); /* 50 mils per cell */
   if (!d2) /* in same column? */
      return( (d1*50)-25 ); /* 50 mils per cell */
   if (d1 > d2) { /* get smaller into d1 */
      d0 = d1;
      d1 = d2;
      d2 = d0;
      }
   d2 -= d1; /* get non-diagonal part of approximate route */
   return( (d1*71)+(d2*50)-25 ); /* 71 mils diagonally per cell */
   }
/* distance to go through a cell */
static int dist[10][10] = { /* OT=Otherside, OR=Origin (source) cell */
    /* N, NE,  E, SE,  S, SW,  W, NW,   OT, OR */
/* N  */ { 50, 60, 35, 60, 99, 60, 35, 60,   12, 12 },
/* NE */ { 60, 71, 60, 71, 60, 99, 60, 71,   23, 23 },
/* E  */ { 35, 60, 50, 60, 35, 60, 99, 60,   12, 12 },
/* SE */ { 60, 71, 60, 71, 60, 71, 60, 99,   23, 23 },
/* S  */ { 99, 60, 35, 60, 50, 60, 35, 60,   12, 12 },
/* SW */ { 60, 99, 60, 71, 60, 71, 60, 71,   23, 23 },
/* W  */ { 35, 60, 99, 60, 35, 60, 50, 60,   12, 12 },
/* NW */ { 60, 71, 60, 99, 60, 71, 60, 71,   23, 23 },

/* OT */ { 12, 23, 12, 23, 12, 23, 12, 23,   99, 99 },
/* OR */ { 99, 99, 99, 99, 99, 99, 99, 99,   99, 99 }
   };

/* distance around (circular) segment of hole */
static int circ[10][10] = { /* OT=Otherside, OR=Origin (source) cell */
    /* N, NE,  E, SE,  S, SW,  W, NW,   OT, OR */
/* N  */ { 39, 29, 20, 10,  0, 10, 20, 29,   99, 0 },
/* NE */ { 29, 39, 29, 20, 10,  0, 10, 20,   99, 0 },
/* E  */ { 20, 29, 39, 29, 20, 10,  0, 10,   99, 0 },
/* SE */ { 10, 20, 29, 39, 29, 20, 10,  0,   99, 0 },
/* S  */ {  0, 10, 20, 29, 39, 29, 20, 10,   99, 0 },
/* SW */ { 10,  0, 10, 20, 29, 39, 29, 20,   99, 0 },
/* W  */ { 20, 10,  0, 10, 20, 29, 39, 29,   99, 0 },
/* NW */ { 29, 20, 10,  0, 10, 20, 29, 39,   99, 0 },

/* OT */ { 99, 99, 99, 99, 99, 99, 99, 99,   99, 0 },
/* OR */ { 99, 99, 99, 99, 99, 99, 99, 99,   99, 0 }
   };

/* penalty for routing holes and turns, scaled by sharpness of turn */
static int penalty[10][10] = { /* OT=Otherside, OR=Origin (source) cell */
    /* N, NE,  E, SE,  S, SW,  W, NW,   OT, OR */
/* N  */ {  0,  5, 10, 15, 20, 15, 10,  5,   50, 0 },
/* NE */ {  5,  0,  5, 10, 15, 20, 15, 10,   50, 0 },
/* E  */ { 10,  5,  0,  5, 10, 15, 20, 15,   50, 0 },
/* SE */ { 15, 10,  5,  0,  5, 10, 15, 20,   50, 0 },
/* S  */ { 20, 15, 10,  5,  0,  5, 10, 15,   50, 0 },
/* SW */ { 15, 20, 15, 10,  5,  0,  5, 10,   50, 0 },
/* W  */ { 10, 15, 20, 15, 10,  5,  0,  5,   50, 0 },
/* NW */ {  5, 10, 15, 20, 15, 10,  5,  0,   50, 0 },

/* OT */ { 50, 50, 50, 50, 50, 50, 50, 50,  100, 0 },
/* OR */ {  0,  0,  0,  0,  0,  0,  0,  0,    0, 0 }
   };

/*
** x is the direction to enter the cell of interest.
** y is the direction to exit the cell of interest.
** z is the direction to really exit the cell, if y=FROM_OTHERSIDE.
** return the distance of the trace through the cell of interest.
** the calculation is driven by the tables above.
*/

int CalcDist ( x, y, z )
   /* calculate distance of a trace through a cell */
   int x, y, z;
   {
   int adjust;

   adjust = 0; /* set if hole is encountered */
   if (x == EMPTY)
      x = 10;
   if (y == EMPTY)
      y = 10;
   else if (y == FROM_OTHERSIDE) {
      if (z == EMPTY)
         z = 10;
      adjust = circ[x-1][z-1] + penalty[x-1][z-1];
      }
   return( dist[x-1][y-1] + penalty[x-1][y-1] + adjust );
   }

Figure 1: Pseudo-code for the breadth-first search algorith

BFS Algorithm  (* breadth-first search *)
   (* Search a graph or state space, depending on the problem definition. *)
   (* S is the start node, T is the goal node. *)
   (* Open is an ordered list of nodes (ordered by arrival time; nodes enter
      at the tail and leave at the head), also called a queue. Closed is a set
      of nodes (order doesn't matter). In general, nodes that need to be
      searched are put on Open. As they are searched, they are removed from
      Open and put in Closed. *)
   (* Pred is defined for each node, and is a list of "came from" indications,
      so when we finally reach T, we traverse Pred to construct a path to S. *)
1  Open <- {S}  (* a list of one element *)
   Closed <- {}  (* the empty set *)
   Pred[S] <- NULL,  found <- FALSE
   WHILE Open <> {} and not found DO
5     x <- the first node on Open
      Open <- Open - {x}  (* remove x from Open *)
      Closed <- Closed + {x}  (* put x in Closed *)
      IF x = T THEN found <- TRUE  (* we're done *)
      ELSE  (* continue search through node x *)
10       let R be the set of neighboring nodes of x
         FOR each y in R DO
            IF y is not on Open or in Closed THEN
               Pred[y] <- x  (* remember where we came from *)
               Open <- Open + {y}  (* put y on Open (at the tail) *)
15 IF found THEN
      use Pred[T] to find Pred[Pred[T]] and so on until S is reached
      (* this traces out the solution path in reverse *)
   ELSE T cannot be reached from S

Figure 3: Pseudo-code for the A* algorithm

A* Algorithm  (* heuristic search *)
   (* Search a graph or state space, depending on the problem definition. *)
   (* S is the start node, T is the goal node. *)
   (* Open is an ordered list of nodes (ordered by lowest F value; see below),
      also called a priority queue. Closed is a set of nodes (order doesn't
      matter). In general, nodes that need to be searched are put on Open (at
      the proper position). As they are searched, they are removed from Open
      and put in Closed. Occasionally a newer, better route will be found to a
      node after it has already been searched, in which case we remove it from
      Closed and put it back on Open to be reconsidered. *)
   (* G[x] is the distance already traveled to get from S to node x, and is
      known exactly. H(x) is a function (heuristic) which returns an estimate
      of the distance from node x to T. F[x] is the estimated distance from S
      to T by going through node x, and is computed by F[x] = G[x] + H(x).
      H(x) can be calculated for any node, but F[x] and G[x] only become
      defined when node x is visited. *)
   (* Pred is defined for each node, and is a list of "came from" indications,
      so when we finally reach T, we traverse Pred to construct a path to
      S. *)
   (* Distance(x,y) is a function for calculating the distance between two
      neighboring nodes. *)
1  Open <- {S}  (* a list of one element *)
   Closed <- {}  (* the empty set *)
   G[S] <- 0,  F[S] <- 0,  Pred[S] <- NULL,  found <- FALSE
   WHILE Open <> {} and not found DO
5     x <- the first node on Open  (* node with smallest F value *)
      Open <- Open - {x}  (* remove x from Open *)
      Closed <- Closed + {x}  (* put x in Closed *)
      IF x = T THEN found <- TRUE  (* we're done *)
      ELSE  (* continue search through node x *)
10       let R be the set of neighboring nodes of x
         FOR each y in R DO
            IF y is not on Open or in Closed THEN
               G[y] <- G[x] + Distance(x,y)
               F[y] <- G[y] + H(y)  (* estimate solution path length *)
15             Pred[y] <- x  (* remember where we came from *)
               Open <- Open + {y}  (* put y on Open *)
            ELSE  (* y is on Open or in Closed *)
               IF (G[x] + Distance(x,y)) < G[y] THEN
                  (* we've found a better route to y *)
20                G[y] <- G[x] + Distance(x,y)
                  F[y] <- G[y] + H(y)
                  Pred[y] <- x  (* remember where we came from *)
                  IF y is on Open THEN
                     reposition y according to F[y]
25                ELSE  (* y is in Closed *)
                     Closed <- Closed - {y}  (* remove y from Closed *)
                     Open <- Open + {y}  (* put y on Open *)
   IF found THEN
      use Pred[T] to find Pred[Pred[T]] and so on until S is reached
30    (* this traces out the solution path in reverse *)
   ELSE T cannot be reached from S

Copyright © 1989, Dr. Dobb's Journal

Back to Table of Contents


Figure 1

Copyright © 1989, Dr. Dobb's Journal


Figure 3

Copyright © 1989, Dr. Dobb's Journal


SEP89: SIMULATED ANNEALING

SIMULATED ANNEALING

This algorithm may be one of the best solutions to the problem of combinatorial optimization

Michael P. McLaughlin

Dr. McLaughlin is a member of the technical staff, Air Transportation Systems, for Mitre Corp. He can be reached at 1740 Westwind Way, McLean, VA 22102.

In science and engineering, it is a truism that, in any real system, everything is connected. Indeed, you could do far worse than to define science as the search for such connections. Scientists accorded the highest esteem, those names nearly everyone knows, are generally individuals whose insight allowed them either to forge links between a large number of separate concepts or to find a connection that no one had suspected.

Occasionally, something similar happens on a smaller scale. This article concerns the application of statistical mechanics to the solution of a difficult problem in computer science. The problem is combinatorial optimization; the technique is called "simulated annealing."

Combinatorial optimization is the task of taking a finite number of indivisible "objects" and arranging them in a configuration that is "best" according to some stipulated criteria. For instance, how should 15 swimmers be selected from a team of 30, for an upcoming meet, if no one is allowed to enter more than one event? Or how should aircraft, preparing to land, be sequenced in order to minimize delays? What is the best routing for a telephone call from Boston to San Francisco at any given moment? What is the best parse for a given English-language sentence? What these and similar problems have in common are 1. an extremely large number of discrete choices, 2. a complicated set of optimizing criteria and 3. numerous constraints. In such circumstances, simulated annealing has proven to be an excellent tool for seeking out the best solution.

Simulated annealing is a relatively new form of stochastic search, familiar primarily to VLSI chip designers. As you can imagine, determining the best geometrical arrangement for thousands of circuits is a formidable task. To take an oversimplified example, suppose you have to partition 1000 circuits into five regions on a microchip. Further suppose that permutations within a region do not matter and that every chip configuration can be given a numerical score ased upon an appropriate set of (possibly conflicting) criteria. Prior to assessing constraints on placement, you have five choices for each circuit for a total of 51000 (about 10699) configurations.

With numbers such as these, it really doesn't matter what kind of computer is available; checking each possibility is out of the question. Nevertheless, the task remains and neither guessing nor giving up are viable options. An answer is usually found with the help of a little computer assistance, often using simulated annealing. Before looking at the usefulness of this simulated annealing, however, let's examine its main competitor.

The Greedy Algorithm

Perhaps the simplest general method for trying to locate a global optimum (maximum or minimum) in a discrete space is a technique often referred to as the "greedy" algorithm. Described in English, it goes something like this:

  1. You randomize the starting configuration.
  2. If possible, you find a better configuration by making repeated (usually random) changes to the current configuration.
  3. Make this better configuration the current one, then go back to step 2.
  4. If a better configuration is not found after a reasonable amount of effort, conclude that you have obtained a local optimum. Save it in the variable OPTIMUM.
  5. Repeat from step 1, replacing OPTIMUM if appropriate, until your patience or computing resources are exhausted. Hope that the final value of OPTIMUM is global.
Until you try a few experiments with this algorithm on large problems with known answers, there is a strong temptation to believe that it might actually work. However, unless the size of the problem (the number of elements to be configured) is small or the configuration space exceptionally smooth, the chances of finding a global optimum in this fashion are remote.

The deficiencies of the greedy algorithm are not obscure. Imagine a mountain goat traipsing about the Rocky Mountains looking for the deepest valley (where the best grass is). This goat is very hungry (greedy?) and, consequently, never takes an upward step. Unless the goat starts out on a smooth slope of the deepest valley, it will never reach that valley but will, instead, come to a dead end in some higher valley, that is, in a local minimum, which may be much higher than the global minimum. Moreover, a space as rough as the Rockies contains thousands of valleys, some only a few feet deep, and repeating the exercise (step 5, above) may yield improvement but probably not the best result.

A Great Deal of Difficulty

The difficulties of combinatorial optimization can be illustrated in the game of poker solitaire where the object is to rearrange the 25 cards of the tableau, like that in Figure 1, so that the 12 hands of straight poker formed from the 5 rows, 5 columns, and 2 diagonals give the highest total score. Various scoring schemes are popular, but I shall use the one listed in Table 1 in which each hand of a pair or better has a value inversely proportional to its probability in a standard deck, normalized to that of a single pair. All hands of a given type (for example, full house) are considered equal, and an ace can be high or low.

Table 1: Scoring table

  Hand                  Score
  ---------------------------

  Straight flush       27,456

  Four of a kind        1,760

  Full house              293

  Flush                   215

  Straight                108

  Three of a kind          20

  Two pair                  9

  Pair                      1

Compared to the task of optimizing circuit placement on a microchip, this is a small problem indeed. There are only 25! (about 1025) configurations and even fewer if symmetry (rotations, reflections, and so on) is taken into account. Nevertheless, this is such a huge number that, even if a computer could evaluate a million configurations per second without repetition, the execution time required to test all of them would be comparable to the present age of the universe. To accommodate this situation, both simulated annealing and the greedy algorithm employ stochastic techniques, evaluating only a fraction of the possible configurations. Consequently, these algorithms provide no absolute guarantee that they will find the global optimum every time. For large problems, it is usually sufficient that a technique provides a reasonable chance of coming close to the global optimum nearly every time. Then, a few runs will yield as good an answer as you are likely to require.

The tableau in Figure 1 contains 7 pairs for a score of 7, although there are obviously 4 aces and 4 kings, so it is reasonable to expect that a good score would be at least 4000. Finding a really high score is not easy, however. The 12 poker hands are tightly interlocked, and even a small change in configuration can result in a drastic change in score. This is another way of saying that the configuration space is very rough. Informal experiments have shown it to be more than a match for a human player, and it provides a good test for any combinatorial optimization technique.

Simulated Annealing

The term annealing originally referred to a process employed in the fabrication of objects constructed of metal or glass. When these materials are shaped, small, often microscopic, regions of stress develop in response to deformations at the atomic level and cause the object to be prone to fracture. Annealing corrects this defect.

From a chemical standpoint, regions of stress have relatively high energy and thus are "almost fractured" already. An object would be more stable (of lower energy) if the stress were absent. However, atoms and molecules in a solid at room temperature do not have enough energy, on average, to move about and relieve the stress. (If they did, the object wouldn't be very solid.) When an object is annealed, it is first heated enough to provide the constituent atoms with sufficient energy to relax any stress but not enough to initiate melting. It is then cooled very slowly. During the cooling phase, the atoms are gradually "frozen" in place. If the annealing is done properly, the resulting object will be without stress. If the cooling is too rapid, there will be insufficient time for the atomic structure to relax completely, and the object will end up with excess energy and fracture too easily.

In simulated annealing, score is associated with energy such that minimizing the energy means optimizing the score. In the poker solitaire example, high score signifies low energy. The tableau in Figure 1 can be thought of as a collection of "atoms" frozen into a configuration that has too much energy. Proceeding with this analogy, the tableau may be subjected to simulated annealing by raising its "temperature" until it is almost "melted," then cooling it slowly until it is again "frozen." Although the connection between real and simulated annealing appears to be little more than a metaphor, this system (25 playing cards) responds in the same way as does a physical system -- and with the same results.

More Details.

In any physical system, the average </entry> energy, <E>, of the constituent atoms is a function of the absolute temperature, T. As the temperature of a system decreases, the average energy of its atoms also decreases. A typical cooling curve is shown in Figure 2. The slope of this curve, d<E>/dT, is the heat capacity, C, of the system. In regions where the curve is steep, the heat capacity is high. This usually indicates a phase change -- water into ice, for example. Quite often, these regions also display large values of C/T. This quantity is equal to dS/dT, the rate of change of entropy, S, with temperature, which is related to a change in the orderliness of the atoms. Therefore, regions of a cooling curve that are steep usually imply a significant reconfiguration of the atoms (or molecules) at the corresponding temperatures. In an annealing process where temperature is being regulated, you would have to slow down the cooling in such regions in order to allow the atoms time to rearrange themselves and to permit their average energy to decrease gradually. Annealing works best when this is done.

Simulated annealing also produces cooling curves much like that of Figure 2. In fact, the literature of simulated annealing demonstrates that these analogies run very deep and it is not at all easy to decide just where to draw the line between metaphor and methodology.

The Connection

The logical link between combinatorial optimization and annealing is the Boltzmann distribution, one of the basic laws of statistical mechanics. Up to now, I've referred to the average energy of a collection of atoms. In most large populations, there is significant variation from individual to individual. So it is with atoms and molecules. They are constantly moving, and their mutual collisions result in continual transfers of energy from one atom (or molecule) to another. Much of the science of statistical mechanics involves techniques for extracting macroscopic properties of a system from the collective behavior of enormous numbers of atoms and molecules. This information may be obtained, to a large extent, by determining how energy is partitioned among the atoms and molecules making up the sample.

At any temperature, there are generally more atoms at lower energies than at higher energies. The Boltzmann distribution, (see Example 1, equation 1) describes the relative number of atoms (or molecules) populating any two energy states, Ehi > Elo, at some temperature, T, where Nx is the number of atoms with energy Ex, T is the absolute temperature (degrees Kelvin), and k is Boltzmann's constant. When equation 1 is adapted to the problem of combinatorial optimization, the constant is ignored, energy is equated to score (cost), and the left-hand side of equation 1 is interpreted as a probability. Simulated annealing uses such probabilities to direct a stochastic search of the configuration space by employing equation 2, Example 1 where Sx is a score and T is a scaling factor.

Example 1: The Boltzmann distribution in chemistry and computer science

  Equation 1: N hi/N lo = exp ((E lo-E hi)/kT)
 Equation 2: Prob (accept worse score) = exp ((S lo-S hi)/T)

In other words, you construct a new configuration at random and determine its score. If this score is at least as good as that of the current configuration, then the old configuration is forgotten and the new configuration becomes the current configuration, as in the greedy algorithm. If the new score is worse (here, lower), however, then equation 2 is used to generate the probability of accepting the new configuration anyhow. A uniform random number in the interval (0,1) is then generated, and if it is less than the probability computed with equation 2, the new configuration is made the current configuration. The net effect is to allow the system to extricate itself from local optima (that is, to let the goat climb uphill sometimes).

As equation 2 suggests, choosing a worse configuration is a function of temperature. The higher the temperature, the higher the "energy" of the system. The higher the energy of the system, the more readily it will do the work of moving in an unfavorable direction -- just like real atoms and molecules. As the temperature decreases, however, any given expenditure of energy becomes less likely. Consequently, simulated annealing automatically adapts to the magnitude of the features of a good configuration. The most profitable features are found first and the smaller, more subtle, features later. Starting with the tableau in Figure 1, simulated annealing first finds the four kings and four aces. If the temperature is very high, these features may be forgotten from time to time, but as the temperature decreases, eventually they are frozen into every acceptable tableau. As the temperature continues to fall, hands of lower and lower scores are successively frozen until, finally, the entire tableau is frozen. Simulated annealing is terminated at this point.

It is clear that the annealing schedule -- the program for reducing the temperature -- is critical; a large drop in temperature at the wrong time may freeze in undesirable features. (Ideally, all reconfigurations should be reversible.) The annealing schedule answers the following four questions:

    1. What should be the value of the starting temperature?

    2. When should the temperature be changed?

    3. How should the temperature be changed?

    4. When should the search be stopped?

A quick-and-dirty procedure is to pick a starting temperature high enough so that virtually all scores are deemed acceptable, hold that and each successive temperature for (100*SIZE) reconfigurations or (10*SIZE) successful reconfigurations, whichever comes first, then decrease the temperature by a constant factor -- for example, T(new) = 0.9*T(old) -- stopping when T reaches some very low value (say, 0.1). Such a schedule is often employed for an exploratory run on a new problem in order to determine the shape of the cooling curve. Given the cooling curve, you can then create a proper annealing schedule by arranging for the temperature to decrease most slowly where the curve is steepest.

Implementation Details

The ANNEAL.C program (see Listing One) addresses both the poker solitaire problem and computes the average and standard deviation of the scores accepted at any temperature. These statistics help in answering the questions posed above.

To choose a starting temperature, begin by picking one that seems high relative to the scores expected and run the program for that temperature only. There should be very few reconfigurations rejected. After equilibrium is reached at this temperature (see below), statistics will be output, one of which is SIGMA, the standard deviation of the acceptable scores. As the temperature approaches infinity, SIGMA approaches SIGMA(inf), the standard deviation of a large set of random configurations. Set the starting temperature equal to 20*SIGMA(inf).

There is no point in trying further reconfigurations at a given temperature once a system has reached thermal equilibrium. (The method used here for recognizing equilibrium is adapted from a procedure published by Huang, Romeo, and Sangiovanni-Vincentelli.)

ANNEAL.C keeps track of the number of reconfigurations accepted (nsuccesses) and rejected (nfailures) at a given temperature as well as the number of acceptable scores less than or more than SIGMA/2 from the average acceptable score (incount and outcount, respectively). The code in Listing Two is then used to decide whether or not equilibrium has been reached. (Uppercase variables are parameters.)

Once equilibrium has been established, the temperature is decreased, provided that the current configuration has a score no worse than SIGMA/2 from the average score. The temperature is not changed until this is true. The idea is to avoid transmitting a poor outlier to a new, lower temperature. As shown, there is no explanation for delaying the transition to a lower temperature.

The simplest and most flexible method for computing a new temperature is to use a table, matched to the cooling curve, listing the desired sequence of temperature ratios, T(new)/T(old), and the temperatures at which they take effect. Typically, these ratios fall in the range (0.5, 1.0). ANNEAL.C permits up to ten values for these ratios in addition to the initial value. Table 2 gives the actual ratios used in this example.</entry>

Table 2: Temperature ratios

  Temperature       t_ratio
   Less Than
  -------------------------

       --             0.7
      360             0.8
      215             0.7
       85             0.8
       60             0.9
       30             0.95
       15             0.9
        7             0.8
        3             0.7
        0             0.7

The configuration is considered frozen if, at equilibrium, the largest single change in score observed at the current temperature is equal to the difference between the best and worst scores at this temperature. This is taken to indicate that all accessible scores are of comparable value and there is no further need for simulated annealing. In ANNEAL.C, this test is carried out only after the temperature reaches a preset limit, t_low. As a failsafe, another parameter, t_min, is stipulated as the temperature at which the configuration is always assumed to be frozen. Once the configuration is frozen, it is "quenched" by setting the temperature equal to zero (greedy algorithm).

In the interest of expediency, the best configuration found is always remembered separately from the current configuration. Ideally, the final configuration should be the best. This is often not the case, however. At the end of the program, this best-found configuration is also quenched.

Giant Steps and Baby Steps

Although a proper annealing schedule is crucial to the success of simulated annealing, there is another consideration that is nearly as important --namely, the manner in which you make a random change to the current configuration. This is true for simulated annealing and, to a lesser extent, for the greedy algorithm. To find a global optimum, any search technique must be able to access the entire configuration space. To do this effectively requires both "giant steps" and "baby steps," the former for moving quickly to a distant part of the space and the latter for fine-tuning.

In this example, baby steps are easy -- just interchange two cards. This is the smallest reconfiguration possible. Finding good giant steps is much more difficult. Of course, you could simply execute several baby steps at once but experience shows that this strategy yields poor results. One reason is that, in making a giant step, it is essential to retain as much of the current score as possible. At moderate to low temperatures, any arbitrary giant step would almost certainly cause a substantial worsening of the score and the new configuration would simply be rejected. All rejected moves are wasted and an efficient algorithm must try to keep such moves to a minimum.

In ANNEAL.C, a giant step consists of interchanging 2 of the 12 hands of the tableau, selected at random from a uniform distribution. A baby step, on the other hand, is implemented using a "rotation." A random integer in the range [2,5] is chosen from an exponential distribution (see Table 3). This number of cards is then selected from the tableau using a uniform distribution. The first card selected is temporarily set aside and the resulting hole filled with card number 2, and so on. The last hole is filled with card number 1. The ratio of giant steps to baby steps is a parameter, exg_cutoff, here set equal to 0.2. Using only giant steps or baby steps generally gives poor performance.</entry>

Table 3: Rotation distribution

    # of Cards      Probability
    to Rotate
  -----------------------------

        2             0.33333
        3             0.27018
        4             0.21899
        5             0.17750

Experimental Results

Simulated annealing and the greedy algorithm are both stochastic techniques. Therefore, they do not give the same results every time. The proper way to evaluate a stochastic technique is to apply it several times and determine its average behavior (actually, its behavior distribution). One of the reasons such a small example was chosen for this article was precisely to allow such a test to be done in a reasonable amount of time.

The simulated annealing algorithm was carried out 30 times, starting with the tableau in Figure 1 each time, using a random-number generator with a period much longer than that required by the experiment. The greedy algorithm was then applied to the same tableau for the same amount of CPU time. Apart from the rule for accepting or rejecting a new configuration, the two programs are essentially identical. In particular, all the relevant parameter values are the same (see A.DAT, Listing Three). The best result from either technique is a score of 4516 (see Figure 3). This apparent optimum, found once by the greedy algorithm and twice by simulated annealing, is degenerate (that is, there are several tableaux with this score, which are not symmetry-related). The distribution of results for simulated annealing is shown in Figure 4 and that for the greedy algorithm in Figure 5.

The number of iterations used to generate Figure 4 and Figure 5 should be enough to give an approximate indication of the average performance of the two techniques. No serious attempt was made to find the combination of parameters that would optimize the simulated annealing results because this would have constituted a bias against the greedy algorithm and such a combination would have applied too specifically to this particular initial tableau. The parameter values in A.DAT appear to be adequate for the poker solitaire problem in general although the annealing schedule will have to be altered when the cards are changed, especially if the new initial tableau can produce a straight flush (see accompanying box). For other combinatorial optimization problems, both the parameter values and the annealing schedule will have to be changed. The big questions, of course, are, "Does simulated annealing work?" and "Is it worth the effort?" Taken together, the results shown in Figure 4 and Figure 5 suggest that simulated annealing is definitely worth the effort. Although you could argue that, given equal amounts of computation time, both techniques find what appears to be the global optimum, this would be overly simplistic. For stochastic techniques, the important criterion is their average behavior.</entry>

More Details.

For this example, the probability that any given iteration of the greedy algorithm yields a score greater than or equal to the average simulated annealing result is 6/1583. Thus, 183 iterations of the greedy algorithm afford a 50 percent chance of doing at least as well as simulated annealing (607 iterations for a 90 percent chance). The greedy algorithm, however, converges only 53 times faster in this example than does simulated annealing. Clearly, there is a higher payoff with simulated annealing.

Final Thoughts

Although I've compared simulated annealing to the well-known greedy algorithm, it would be incorrect to suggest that the latter is the only alternative. Recently, neural networks and various genetic algorithms have been applied to combinatorial optimization problems with some success. If the problems are small enough, then classical integer programming techniques (that is, branch-and-bound) may also prove fruitful. Moreover, when evaluating the material presented here, there are several points to bear in mind. First, not every application of simulated annealing will exhibit better performance than a greedy algorithm to the extent observed in this case, so you should not extrapolate too much from this single example.

Another important consideration is that simulated annealing is very time-consuming, and almost any serious application of this sort is best written in assembly language. The program ANNEAL.C was coded in C solely in the interests of pedagogy and portability.

Finally, both algorithms described in this article are stochastic in nature. Even though the tableau in Figure 1 was subjected to hundreds of simulated annealing runs, there is still no guarantee that the optimum found is, in fact, the global optimum. It's possible that there exists a tableau with a score greater than 4516 but that the path through the configuration space leading to it is so narrow that even the millions of reconfigurations carried out are insufficient to find it.

In this regard, it's interesting that all the tableaus found having scores of 4516 contained only 11 scoring hands, not 12. This being the case, there may be "room" for an even higher score. Better yet, what if a 13th hand were defined, consisting of the 4 corners plus the center card. Would there be any point in adding the extra degree of freedom? This intriguing thought is left as an exercise for DDJ readers.

Bibliography

Huang, M. D.; Romeo, F.; and Sangiovanni-Vincentelli, S. "An Efficient Cooling Schedule for Simulated Annealing." Proc. Int. Conf. Computer-Aided Design (ICCAD86) (1986): 381 - 384.

Kirkpatrick, S.; Gelatt, C. D. Jr.; and Vecchi, M. P. "Optimization by Simulated Annealing." Science, vol. 220, no. 4598 (1983): 671 - 680.

Metropolis, N.; Rosenbluth, A;, Rosenbluth, M;, Teller, A;, and Teller, E. "Equations of State Calculations by Fast Computing Machines." J. Chemical Physics, vol. 21 (1953): 1087 - 1091.

Nash, L.K. Elements of Statistical Thermodynamics. Reading, Mass.: Addison-Wesley, 1968.

Press, W. H.; Flannery, B. P.; Teukolsky, S. A.; and Vetterling, W. T. Numerical Recipes, The Art of Scientific Computing. New York: Cambridge University Press, 1986: 326 - 334.

Sechen, C. and Sangiovanni-Vincentelli, A. "TimberWolf 3.2: A New Standard Cell Placement and Global Routing Package." Proc. 23rd Design Automation Conf. (1986): 432 - 439.

Availability

All source code for articles in this issue is available on a single disk. To order, send $14.95 (Calif. residents add sales tax) to Dr. Dobb's Journal, 501 Galveston Dr., Redwood City, CA 94063, or call 800-356-2002 (from inside Calif.) or 800-533-4372 (from outside Calif.). Please specify the issue number and format (MS-DOS, Macintosh, Kaypro).</entry>

ANNEAL.C Data Structures, Functions, and Variables

The playing cards are represented as 2-byte integers. The card values, 2 to Ace (high), are coded as the numbers 2 to 14 and the suits {spades, hearts, diamonds, clubs} are coded as {2048, 1024, 512, 256}, respectively. The tableau is intrinsically a two-dimensional array but is implemented as a one-dimensional array to avoid the computational burden of double indexing (see HAND[12][5] below). ANNEAL.C ignores the symmetry of the tableau with respect to the scoring function because it is, computationally, more trouble than it is worth.

rand1( ) -- takes a 31-bit integer, SEED, and returns another one. This generator yields a full-cycle sequence without repeating any numbers.

uni(z) -- returns a double-precision float in the range (O,z) via rand1( ).

initialize( ) -- reads in the starting data from A.DAT and sets up the data structures, and so on.

init( ) -- resets a number of global counters and variables, chiefly to aid in the collection of the statistics necessary to detect equilibrium.

get_new_temperature( ) --computes a temperature ratio, if necessary, and a new temperature (see Table 2).

check_equilibrium( ) -- uses the code given in the article to detect the equilibrium point.

update( ) -- computes those statistics, which need not be found until equilibrium has been achieved. (see SCALE).

selectwr( ) -- is a general routine to perform N selections without replacement.

reconfigure(direction) -- makes a new configuration, or restores the tableau to its previous configuration if the last move was rejected. (Typically, more than 90 percent of the reconfigurations are rejected.) This routine uses giant steps or baby steps.

get_score(crds) -- is called by evaluate1( ) and returns the score of a single hand as defined in HAND[12][5].

evaluate1( ) -- passes each hand, one by one, to get_score(crds) and tallies up the total SCOR (which becomes the current SCORE, if the reconfiguration is accepted). Along with rand1( ) and reconfigure( ), the functions evaluate1( ) and get_score( ) are all written in MC68020 assembly language in the working code. This code runs at a speed of approximately 2035 attempted reconfigurations/sec. on an AMIGA 1000 equipped with a 68020/68881 chip set, plus 32-bit RAM (a standard Amiga, executing ANNEAL.C, does 70/sec.), and finishes an average simulated annealing run in less than 3 minutes with the results as shown in Figure 4. Without this sort of speed, the long series of experiments needed to obtain good parameter values, and to compare with the greedy algorithm, would have required more patience than this author has ever possessed.

decide( ) -- answers the question "To accept or not to accept?" and uses the formula shown Example 1, equation 2. It also accumulates statistics like INCOUNT and OUTCOUNT that must be computed at every move.

report( ) -- produces the output statistics and the final tableaux.

try_new_temperature( ) -- is a calling function that uses a newly computed temperature. It also checks that the tableau passed on to a new temperature is not a low outlier.

main( ) -- q.v. HAND[12][5] -- stipulates the 12 hands (5 cards each) in the TABLEAU and describes the various hands in terms of the appropriate indices of TABLEAU.

PARTITION[6], OVERLAP[66], HEX [12], CINDEX[25] -- are used to generate a new configuration. PARTITION implements Table 3 in long integers (pre-multiplied by MODULUS). OVERLAP is a triangular array giving the index, in TABLEAU, of the intersection of two HANDs (-1 if no overlap). HINDEX and CINDEX are arrays used to perform selection without replacement and to remember those selections should the new configuration be rejected.

SCORES[18] -- is an implementation of Table 1. After evaluate1( ) has determined the index to SCORES, SCOR is assigned the corresponding value.

TEMPERATURE, T_LOW, T_MIN --is the scaling factor. T_LOW is the temperature below, which the tableau can be tested to see if it is FROZEN or not. If the temperature ever goes below T_MIN, the tableau is automatically FROZEN.

AVG_SCORE, SIGMA -- are the average and standard deviation of the scores of all the tableaux accepted at the current temperature.

SCALE -- is a typical SCORE when the temperature is getting cold. This variable has no function in the algorithm per se, but is incorporated so that SIGMA may be computed using single-precision arithmetic. (This is especially important in the absence of a floating-point chip.) By subtracting off a good score before accumulating the square-of-the-score, the number of digits to be squared is reduced. If this is not done, then a single-precision number will lose too much precision at a time in the annealing process when the average-of-the-squares and the square-of-the-average are nearly equal. When this happens, it is possible to get SIGMA = 0 erroneously. This has a deleterious effect on the program.

T_RATIO -- T(new)/T(old). The initial value (here, 0.7) and 10 other values are listed in A.DAT.

EXG_CUTOFF -- is the ratio Prob (giant step)/Prob(baby step).

SUCC_MIN, INCOUNT_LIMIT, OUT-COUNT_LIMIT, FIRST_LIMIT, ULTIMATE_LIMIT -- are parameters used to determine the equilibrium point.

-- M.M.

Cellular Automata: A New Way of Simulation

Jean F. Coppola and Dr. Francis T. Marchese

Jean Coppola is a master's degree student in computer science and Dr. Francis Marchese is an associate professor in the computer science department at Pace University. They can be reached at Pace University, 1 Pace Plaza, New York, NY 10038.

In the annealing process, glass, certain metals, and alloys are heated and cooled in a controlled way to allow the atoms to arrange themselves into a stable state. Normally, the process of annealing is simulated by means of a set of complex differential equations. Alternatively, annealing can be simulated by an intuitively simpler method known as "cellular automata."

Cellular automata are discrete dynamical systems in which the behavior of the system as a whole is defined in terms of a set of local rules. The simplest and best-known cellular automata is the game of LIFE conceived by John Conway in the early 1970s. LIFE is a computer simulation that consists of a grid of cells upon which a collection of creatures are born, live and die. The life/death process is governed by a neighborhood rule that defines the behavior of any creature at any cell located in terms of the behavior of its neighbors. This is a significant concept because it implies that for collective systems, no member of that collection of entities (or creatures) can exist independently of the rest. Moreover, the rules that govern the system as a whole are only the rules of the local neighborhoods. Hence, for example, in a two-dimensional life game consisting of a 100 x 150 grid, the rules are only defined for each local 3 x 3 matrix! Even though this is the case, the complexity of the life process produced can be quite profound.

As a result, cellular automata has been applied in areas such as physics, chemistry, and computer science. For example, particle dynamics, galactic evolution, molecular kinetics, or parallel distributed processing are all fields where local laws dictate global behavior. At Pace University, we've applied cellular automata to the real-time study of the dynamics of crystal growth. Because of the large number of local interactions that must be accommodated in these simulations, even a mainframe cannot provide the computing power required for real-time visualization. However, assisting in the simulations, is an IBM PC-compatible, high-performance hardware option called CAM-6 (Cellular Automata Machine) that runs under MS-DOS. CAM-6 is a special-purpose coprocessor designed specifically for cellular automata simulations by researchers at the MIT Laboratory for computer science. Its pipelined architecture allows software to run at speeds comparable to that of a CRAY-1, therefore allowing the CAM board to update and display a 256 x 256 array of cells 1/60 of a second, thus producing real-time animation. Each cell in CAM is represented by a pixel, and cells can assume 1 of 16 possible states signified by 16 different colors.

The CAM-6 machine is programmable in a high-level language called "CAM Forth," a special version of Forth-83. Because Forth was originally developed for real-time applications and meets the demands of cellular automata by its code efficiency.

We have used this combination of hardware and software to simulate crystal growth. Presently, we wish to discover the local rules which govern the growth of sodium chloride (common table salt). This is accomplished by programming neighborhoods that work with the four major neighbors on all the planes. The difficult part in programming "rules" is keeping in mind that one cannot change the state of a neighbor but only the cell currently in question. For example, you cannot indicate if a cell has three or more neighbors at state 1, and turn all neighbors at state 0 to 1. Therefore, programming has to be done around this constraint, which at times can be extremely difficult. We have also simulated complex solid liquid equilibria. This involved programming minor neighborhoods which allowed states to be compared from a second CAM machine that was programmed as a random noise generator.

By watching and studying these simulations, we hope to gain insight into the synthesis of crystals and fundamental patterns of molecular growth. The models of cellular automata could one day be a major breakthrough in the understanding of nature's rules, which in turn could affect every aspect of life.

--J.C., F.T.M.

_SIMULATED ANNEALING_
by Michael P. McLaughlin

[LISTING ONE]

/* ANNEAL.C -- Author: Dr. Michael P. McLaughlin
*/
#include <stdio.h>
#include <math.h>

#define ABS(x) ((x<0)?(-(x)):(x))
#define BOOLEAN int
#define TRUE 1
#define FALSE 0
#define FORWARD 1
#define BACK 0

struct cardtype {
   char card[3];
   int code;
   } cards[25];
float ratios[10][2];
int tableau[25],best_tableau[25];
float temperature,best_temperature,t_low,t_min,t_ratio;
long seed,score,best_score,tot_successes,tot_failures,scor,exg_cutoff,
     report_time,report_interval,modulus = 2147483399,step = 1;
int next;
BOOLEAN equilibrium,frozen = FALSE,quench=FALSE,final=FALSE;

/* variables needed to assess equilibrium */
float totscore,totscore2,avg_score,sigma,half_sigma,score_limit;
long bscore,wscore,max_change,nsuccesses,nfailures,scale,
     incount,outcount;
int incount_limit,outcount_limit,succ_min,first_limit,ultimate_limit;

/* poker hands in tableau */
int hand[12][5] = {0,1,2,3,4,      /* rows */
                   5,6,7,8,9,
                   10,11,12,13,14,
                   15,16,17,18,19,
                   20,21,22,23,24,
                   0,5,10,15,20,   /* columns */
                   1,6,11,16,21,
                   2,7,12,17,22,
                   3,8,13,18,23,
                   4,9,14,19,24,
                   0,6,12,18,24,   /* diagonals */
                   4,8,12,16,20};
                /* 0,4,12,20,24     = corners */

long rand1()
/* Get uniform 31-bit integer -- Ref. CACM, June 1988, pg. 742 */
{
   register long k;
   k = seed/52774;
   seed = 40692*(seed-k*52774)-k*3791;
   if (seed < 0)
      seed += modulus;
   return(seed);
}
double uni(z)
int z;
{
   return((double) z*rand1()/modulus);
}
void initialize()
/* Set up entire simulated annealing run. */
{
   char label[20];
   double exgc;
   register i;
   FILE *fp;
   fp = fopen("a.dat","r");
   fscanf(fp,"%f %s\n",&temperature,label);
   fscanf(fp,"%f %s\n",&t_low,label);
   fscanf(fp,"%f %s\n",&t_min,label);
   fscanf(fp,"%f %s\n",&avg_score,label);
   fscanf(fp,"%ld %s\n",&scale,label);
   fscanf(fp,"%f %s\n",&t_ratio,label);
   fscanf(fp,"%f %s\n",&sigma,label);
   fscanf(fp,"%lf %s\n",&exgc,label);
   fscanf(fp,"%d %s\n",&succ_min,label);
   fscanf(fp,"%d %s\n",&incount_limit,label);
   fscanf(fp,"%d %s\n",&outcount_limit,label);
   fscanf(fp,"%d %s\n",&first_limit,label);
   fscanf(fp,"%d %s\n",&ultimate_limit,label);
   fscanf(fp,"%ld %s\n",&report_interval,label);
   fscanf(fp,"%ld %s\n",&seed,label);
   half_sigma = 0.5*sigma; report_time = report_interval;
   score_limit = 20; exg_cutoff = modulus*exgc;
   for (i=0;i<10;i++)
      fscanf(fp,"%f %f\n",&ratios[i][0],&ratios[i][1]);
   for (i=0;i<25;i++) {
      fscanf(fp,"%d %s\n",&cards[i].code,cards[i].card);
      tableau[i] = cards[i].code;
   }
   fclose(fp);
   printf("                   TOTAL     TOTAL  CURRENT AVERAGE\n");
   printf("STEP TEMPERATURE SUCCESSES FAILURES  SCORE   SCORE ");
   printf(" SIGMA\n\n");
}
void init()
/* set up for new temperature */
{
   nsuccesses = 0; nfailures = 0; equilibrium = FALSE; incount = 0;
   outcount = 0; bscore = 0; wscore = 100000; max_change = 0;
   totscore = 0; totscore2 = 0;
   if (temperature < t_min)
      frozen = TRUE;
}
void get_new_temperature()
{
   if (temperature < ratios[next][0]) {
      t_ratio = ratios[next][1];
      next++;
   }
   temperature = t_ratio*temperature;
}
void check_equilibrium()
/* Determine whether equilibrium has been reached. */
{
   if ((nsuccesses+nfailures) > ultimate_limit)
      equilibrium = TRUE;
   else if (nsuccesses >= succ_min) {
      if (incount > incount_limit)
         equilibrium = TRUE;
      else {
         if (outcount > outcount_limit) {
            if (nsuccesses > first_limit)
               equilibrium = TRUE;
            else {
               incount = 0;
               outcount = 0;
            }
         }
      }
   }
}
void update()
/* Compute statistics, etc. */
{
   float ascore,s;
   if (nsuccesses > 0) {
      ascore = totscore/nsuccesses;
      avg_score = ascore+scale;
      s = totscore2/nsuccesses-ascore*ascore;
      if (s > 0.0) {
         sigma = sqrt(s); half_sigma = 0.5*sigma;
         score_limit = avg_score-half_sigma;
      }
   }
   if ((temperature < t_low)&&((bscore-wscore) == max_change))
      frozen = TRUE;
}
void selectwr(array,mode,nchoices)
/* Select from array without replacement. */
int *array,mode,nchoices;
{
   int i,temp,size,index;
   size = mode==1 ? 12 : 25;
   for (i=0;i<nchoices;i++) {
      index = uni(size);
      temp = array[--size];
      array[size] = array[index];
      array[index] = temp;
   }
}
void reconfigure(direction)
/* If direction is FORWARD, make a new move; if it is BACK, restore
   the tableau to its previous configuration. */
int direction;
{
   static long partition[6] = {0,0,715827799,1296031795,1766307855,
                               2147483400};
   static int hindex[12] = {0,1,2,3,4,5,6,7,8,9,10,11};
   static int cindex[25] = {0,1,2,3,4,5,6,7,8,9,10,11,12,13,
                            14,15,16,17,18,19,20,21,22,23,24};
   static int overlap[66] = {-1,-1,-1,-1,-1,-1,-1,-1,-1,-1,0,5,10,15,
                             20,1,6,11,16,21,-1,2,7,12,17,22,-1,-1,3,
                             8,13,18,23,-1,-1,-1,4,9,14,19,24,-1,-1,-1,
                             -1,0,6,12,18,24,0,6,12,18,24,4,8,12,16,20,
                             20,16,12,8,4,12};
   static int mode,nchoices,common,last;   /* remember changes */
   int temp,hi,lo,i,j,k;
   if (direction == FORWARD) {
      if (rand1() < exg_cutoff) {   /* giant step */
         mode = 1; nchoices = 2;
         selectwr(hindex,mode,nchoices);
         hi = hindex[11]; lo = hindex[10];   /* order hand indices */
         if (hi < lo) {
            hi = lo;
            lo = hindex[11];
         }
         common = overlap[hi*(hi-1)/2+lo];   /* triangular matrix */
      }
      else {   /* baby step */
         mode = 0; nchoices = 2;
         rand1();   /* How many cards to rotate? */
         while (seed > partition[nchoices])
            nchoices++;
         selectwr(cindex,mode,nchoices);
         temp = tableau[cindex[24]];   /* rotate forward */
         for (i=1,j=24;i<nchoices;i++,j--)
            tableau[cindex[j]] = tableau[cindex[j-1]];
         tableau[cindex[j]] = temp;
         last = j;
      }
   }
   if (mode == 1) {   /* swapping hands is symmetrical */
      register first,second,c;
      first = hindex[11]; second = hindex[10];
      k = (common<0) ? 5 : 4;
      for (c=0,i=0,j=0;c<k;c++,i++,j++) {
         if (hand[first][i] == common)
            i++;
         if (hand[second][j] == common)
            j++;
         temp = tableau[hand[first][i]];
         tableau[hand[first][i]] = tableau[hand[second][j]];
         tableau[hand[second][j]] = temp;
      }
   }
   else if (direction == BACK) {   /* rotation is not */
      temp = tableau[cindex[last]];
      for (i=1,j=last;i<nchoices;i++,j++)
         tableau[cindex[j]] = tableau[cindex[j+1]];
      tableau[cindex[24]] = temp;
   }
}
long get_score(crds)
/* Return score of one hand. */
int *crds;
{
   static long scores[18] = {0,1,1,20,1,9,20,1760,1,9,9,
                             293,20,293,1760,108,215,27456};
   int flush,i,index = 0;
   flush = crds[0]&crds[1]&crds[2]&crds[3]&crds[4]&0xff00;
   for (i=0;i<5;i++)
      crds[i] = crds[i]&0xff;   /* ignore suits */
   {
      register b,k,t;   /* sort cards from high to low */
      for (b=4;b>0;b--)
         for (k=0;k<b;k++)
            if (crds[k] < crds[k+1]) {
               t = crds[k];
               crds[k] = crds[k+1];
               crds[k+1] = t;
            }
   }
   if (!flush) {   /* multiple cards ? */
      if (crds[0] == crds[1]) index += 8;
      if (crds[1] == crds[2]) index += 4;
      if (crds[2] == crds[3]) index += 2;
      if (crds[3] == crds[4]) index += 1;
   }
   if (!index) {   /* straight ? */
      if (((crds[0]-crds[4]) == 4)||((crds[0] == 14)&&(crds[1] == 5)))
         index = 15;
      if (flush)
         index = (index == 15) ? 17 : 16;
   }
   return(scores[index]);
}
long evaluate1()
/* Return total score of tableau (no corners). */
{
   int temp[5],h,c;
   long scr = 0;
   for (h=0;h<12;h++) {   /* h<13 for corners */
      for (c=0;c<5;c++)
         temp[c] = tableau[hand[h][c]];
      scr += get_score(temp);
   }
   return(scr);
}
void decide()
/* Either accept new configuration or restore old configuration */
{
   float pdt1,pdt2,sscor;
   long s;
   BOOLEAN acceptable = FALSE;
   scor = evaluate1();
   if (scor >= score) {
      if (scor > best_score) {
         register i;
         best_score = scor;
         best_temperature = temperature;
         for (i=0;i<25;i++)
            best_tableau[i] = tableau[i];
      }
      acceptable = TRUE;
   }
   else if (temperature > 0.0) {   /* uphill movement? */
      if (uni(1) < exp((scor-score)/temperature))   /* Equation 2 */
         acceptable = TRUE;
   }
   if (acceptable) {   /* statistics, etc. */
      s = ABS(score-scor);
      if (s > max_change)
         max_change = s;
      score = scor;
      sscor = scor-scale;   /* to aid precision of sigma */
      totscore += sscor;
      totscore2 += sscor*sscor;
      nsuccesses++;
      if (ABS(avg_score-score) < half_sigma)
         incount++;
      else
         outcount++;
      if (score > bscore)   /* maximization */
         bscore = score;
      else if (score < wscore)
         wscore = score;
   }
   else {   /* unacceptable */
      reconfigure(BACK);
      nfailures++;
   }
}
void report()
{
   tot_successes += nsuccesses; tot_failures += nfailures;
   printf("%3ld %10.1f %9ld %9ld %7ld %7.0f  %5.1f\n",step,temperature,
          tot_successes,tot_failures,score,avg_score,sigma);
   report_time += report_interval;
   if (frozen) {
      int temp[25],k,kk;
      register i,j;
      BOOLEAN ok;
      if (final)
         printf("\nFINAL -- ");
      else if (quench)
         printf("\nQUENCH -- ");
      else
         printf("\nFROZEN -- ");
      printf("BEST SCORE = %5ld  BEST TEMPERATURE = %5.1f\n\n",
              best_score,best_temperature);
      for (i=0;i<25;i++) {
         k = best_tableau[i];
         j = 0;
         ok = FALSE;
         while (!ok) {
            if (cards[j].code == k) {
               temp[i] = j;
               ok = TRUE;
            }
            else j++;
         }
      }
      for (i=0;i<25;i+=5) {
         for (j=i;j<(i+5);j++) {
            k = 4-strlen(cards[temp[j]].card);
            for (kk=0;kk<k;kk++) printf(" ");
            printf("%s",cards[temp[j]].card);
         }
         printf("\n");
      }
      printf("\n");
   }
}
void try_new_temperature()
{
   init();
   while (!equilibrium) {
      reconfigure(FORWARD);
      decide();
      check_equilibrium();
   }
   update();
   if (!quench) {   /* ratchet */
      while ((float) score < score_limit) {   /* maximization */
         reconfigure(FORWARD);
         decide();
      }
   }
}
main()
{
   initialize();
   while (!frozen) {
      try_new_temperature();
      update();
      if ((step == report_time)||(frozen))
         report();
      step++;
      if (temperature > 0.0)
         get_new_temperature();
   }
/* Quench frozen configuration. */
   temperature = 0;
   quench = TRUE;
   try_new_temperature();
   update();
   report();
   step++;
/* Quench best configuration (rarely useful). */
   {
      register i;
      final = TRUE;
      for (i=0;i<25;i++)
         tableau[i] = best_tableau[i];
   }
   try_new_temperature();
   update();
   report();
   printf("SEED = %ld\n",seed);   /* seed of next run */
}

[LISTING TWO]

     if ((nsuccesses+nfailures) > ULTIMATE_LIMIT)
       equilibrium = true;
     else if (nsuccesses >= SUCC_MIN) {
       if (incount > INCOUNT_LIMIT)
        equilibrium = true;
       else {
        if (outcount > OUTCOUNT_LIMIT) {
         if (nsuccesses > FIRST_LIMIT)
          equilibrium = true;
         else {
          incount = 0;
          outcount = 0;
         }
        }
       }
     }

[LISTING THREE]

2150 temperature
1.5 t_low
0.1 t_min
40 avg_score
4400 scale
0.7 t_ratio
150 sigma
0.2 exg_cutoff
200 succ_min
250 incount_limit
400 outcount_limit
2700 first_limit
10000 ultimate_limit
1 report_interval
1234567890 seed
360 0.8
215 0.7
85 0.8
60 0.9
30 0.95
15 0.9
7 0.8
3 0.7
0 0.7
0 0.7
1032 8H
2061 KS
270 AC
526 AD
522 10D
1029 5H
265 9C
1035 JH
1037 KH
525 KD
515 3D
263 7C
2058 10S
2062 AS
518 6D
1036 QH
267 JC
259 3C
2053 5S
269 KC
520 8D
1028 4H
1038 AH
519 7D
1026 2H

[EXAMPLE 1]

Equation 1:  N /su/hi//N /su/lo/ = exp((E lo-E hi)/kT)

Equation 2:  Prob(accept worse score) = exp((S /su/lo/ -S /su/hi/)/T)

Example 1: Equations that describe the Boltzman distribution

Copyright © 1989, Dr. Dobb's Journal

Back to Table of Contents


FEB90: SELF-ADJUSTING DATA STRUCTURES

SELF-ADJUSTING DATA STRUCTURES

Use self-adjusting heuristics to improve the performance of your applications

Andrew M. Liao

Andrew received his master's degree in computer science from RPI in Troy, New York. He can be reached through his Bitnet address, which is aliao%eagle @wesleyan.bitnet. You can also reach him through the DDJ office.

Application programs are often developed using standard data structure techniques such as stacks, queues, and balanced trees with the goal of limiting worst case performance. Such programs, however, normally carry out many operations on a given data structure. This means that you may be able to trade off the individual worst case cost of each operation for that of the worst case cost over a sequence of operations. In other words, any one particular operation may be slow, but the average time over a sufficiently large number of operations is fast. This is an intuitive definition of amortized time, a way of measuring the complexity of an algorithm. In this case, the algorithms to be concerned with are those that carry out operations on data structures.

The heuristic I'll discuss in this article is called the "structural self-adjusting heuristic." To illustrate what I mean by self-adjusting, consider the following example: Suppose you're running an information warehouse and your task is to distribute information to people who request it. The information in this warehouse could be stored in a fixed order, such as the order of information in a library. You quickly notice, however, that certain pieces of information are requested more often than others. You could make the job easier by moving the most often requested information close to the service counter. This means that instead of having to search through the depths of the warehouse at any given time, you have a good portion of the most requested information nearby.

As this example suggests, self-adjusting heuristic algorithms are ideally suited to lists, binary search trees, and priority queues (heaps). In lists, the heuristic attempts to keep the most frequently accessed items as close to the front as possible. In binary search trees, the heuristic attempts to keep the most frequently accessed items as close to the root as possible, while preserving the symmetric ordering. Finally, in heaps, the heuristic attempts to minimize the cost of modifying the structure, and partially orders the heap in a simple, uniform way. To illustrate how these algorithms can be implemented, I've provided sample Pascal source code.

Self-Adjusting Lists

A singly linked list is a group of records where each record contains one field that holds an individual piece of user data, and another field that holds a pointer to the next record in the list. An initial pointer that indicates which record starts the list is (or should be) kept. This pointer enables you to search, insert, and delete operations.

Move-to-Front Singly Linked Lists

To understand how the move-to-front (MTF) approach works, consider a situation in which a particular application uses an open hash table with a linked list that is associated with each array location. Suppose that the hashing routine for this application is as good as it can possibly be. If you wish to improve the search performance without unduly complicating the supporting code, however, you might examine the performance of the search performed on the lists. Chances are that certain elements are accessed more often than others. The use of either a transpose or a frequency count heuristic (two other common access approaches) does not appear to be a good idea because of the search overhead involved with each approach. Both methods require either a local exchange operation or extra searching in order to reinsert an accessed item into the correct part of the list. Also, the count method requires a change in the list: The addition of an integer field that maintains the access count. All three heuristics are effective in that they search less than half the list.

One reason why the MTF heuristic performs better than the transpose method is that the transpose heuristic causes the list to converge more slowly to its optimal ordering. In the case of MTF, an element is brought to the front of the list. Furthermore, such an element quickly "sinks" to the end of the list over the course of a sequence of accesses if that element is not a sufficiently wanted item. Essentially, MTF may be viewed as an optimistic method in the sense that the method "believes" that an accessed item will be accessed again. Analogously, the transpose heuristic may be viewed as pessimistic in that it "doubts" that an accessed item will be accessed again. The count method is a compromise between the two.

As Figure 1 and Listing One (page 105) illustrate, searching is the key operation for the MTF heuristic. This search operation is very much like a normal search on a singly linked list, except that an extra pointer is kept to the current predecessor of the list node that is currently being examined. Once a given item is found, the pointer to its predecessor node is used to alter the predecessor node's link field so that the link field points to the successor node of the accessed item. The link field in the desired node is then altered to point to the first element in the list, and the head-of-list pointer is set to point to the new front-of-the-list item. For all intents and purposes, the insert operation is a push operation -- the new item is immediately put at the front of the list. Finally, an MTF search is used to perform a delete. If the item to be deleted is located at the front of the list, that item is removed from the front of the list.

Self-Adjusting Heaps

A "heap" is a tree-based data structure in which each record node keeps a key, along with pointers to the record node's successors. The heap maintains an ordering of items such that every node has a key less than, or equal to, the keys of the node's successors. This last description is the concept of "heap-ordering."

There are a number of classical priority queue structures (such as 2 - 3 trees, leftist heaps, and binomial heaps) that are amenable to fast merging operations. Of These, the simplest scheme for maintaining a heap with fast merge operations is the "leftist heap," which was developed to maintain partially ordered lists with a logarithmic time-merge operation. A leftist heap is based upon a binary tree node that contains rank, weight, and two pointer fields (to indicate left and right children). The rank field is defined to be 0 if a node is a leaf. Otherwise, the rank field is defined as one more than the minimum value of both the rank of the leftchild and the rank of the rightchild.

A binary tree is a leftist heap if it is heap-ordered and if the rank of a given leftchild is greater than, or equal to, the rank of its rightchild sibling. The problem with maintaining leftist heaps is that the configuration of the data structure is based upon the rank definition. All operations are heavily dependent upon the value kept in the rank field of a given node. To illustrate the point, I'll describe the leftist heap merge operation.

The leftist heap merge operation is made possible by a modified version of an "enqueue" process (which takes a heap and a queue pointer's record as parameters). This particular enqueue operation saves the root of the heap and moves the front queue pointer down to the rightchild of the root just saved. You then break the link between the saved root and its rightchild. If the queue pointers are both empty, point the front and rear pointers to the root node that was just saved, and set the rightchild pointer field of the root to empty. Otherwise, point the rightchild pointer to the node currently pointed to by the rear queue pointer, and point the rear queue pointer to this newly obtained node.

Implement the merge with the following steps: While neither of the two heaps being merged is empty, call enqueue with the currently minimum key and with the queue pointer's record. Next, while the "first" heap is not empty, call enqueue with that heap and with the queue pointer's record. Perform the same steps for the other heap. These three processes merge the right path. Complete the process with a bottom-up traversal of the right path in order to fix up the rank fields and to perform any necessary swaps to maintain the structural invariant.

Now point to the current two bottommost nodes on the merge path. If there is no left sibling of the bottommost node, make the rightchild a leftchild and set its parent's rank field to 0. Next, set the rightchild's rightchild pointer field to empty. If the bottommost node has a left sibling, compare the two children and swap them when the rank of the left sibling is less than that of the right sibling. In any event (given this case), set the parent's rank field to 1+ rank of the rightchild. Also note which nodes are the next two bottommost nodes on the merge path at this point, and make sure that the parent node before this step points to the rightchild. This process continues until the root is reached when the root of the new heap is returned. Once the merge operation for leftist heaps has been described, the other heap operations are easy to implement.

This description of the merge operation suggests that two passes are required over the merge path. The question remains: How do you improve performance without unduly complicating the algorithms that maintain the heap? This can be done with a restructuring method that essentially exchanges every node on the result heap's right path with the node's left sibling. The version of the technique presented here also has a feature in which one top-down pass completes the merge. The resulting structure, called a "top-down-skew heap," is a self-adjusting analog of the leftist heap.

Top-Down-Skew Heaps

A skew heap is based upon a simple binary tree node that contains a weight plus pointer fields to left and right children. The process of merging is made possible by another modified version of the enqueue algorithm. In this case it's not necessary to maintain the rank/balance field in order to obtain logarithmic, amortized performance.

This particular enqueue operation saves the root of the heap and moves the front queue pointer down to the rightchild of the root just saved. You then break the link between the saved root and its rightchild by changing the current leftchild into a rightchild. If the queue pointers are both empty, point the front and rear pointers to the root node just saved, and set the root's leftchild pointer to empty. Otherwise, the newly obtained node becomes the leftchild of the node that is currently indicated by the rear queue pointer, after which the rear queue pointer is changed to indicate the newly obtained node. (See Figure 2.)

The following steps implement the merge: While neither of the two heaps being merged is empty, call enqueue with the heap that contains the current minimum key and the queue pointer's record. Next, while the "first" heap is not empty, call enqueue with that heap and with the queue pointer's record. Follow the same process for the other heap. (This approach is analogous to Tarjan and Sleator's conceptual noting that the left and right children of every node on the merge path are swapped. The implementation used here, however, is a variation.) Once either of the two heaps being merged becomes empty, merely attach the remaining heap to the bottom of the result heap's left path. Again, the rest of the heap operations are easy to define.

Pairing Heaps

Much like the leftist heap, the binomial heap has an analogous self-adjusting counterpart. This new structure, called the "pairing heap," is a recent development in heaps that supports the decreaseKey operation. The essential definition of the pairing heap, like that of the skew heap, is based upon a simple binary tree record node that contains at least weight plus three pointer fields (to indicate the parent and the left and right siblings). Like most heaps, the pairing heap depends upon a merge operation, but has a less complicated scheme than its classical counterpart.

In the case of the binomial heap, you need to maintain a forest of trees where each tree contains a number of nodes equal to a non-negative integer power of two. Thus, a binomial heap of n items can be represented by a forest in which each tree corresponds one-to-one with the one bit that represents the value of n in binary. (This eventually leads to the fact that all of the binomial heap operations are, in the worst case, logarithmic time.) Needless to say, the code needed to implement a binomial heap merge operation is complicated and difficult to maintain.

The merge operation for pairing heaps begins by determining which of the two heaps has the minimal weight at the root. The heap with the non-minimum key at the root then becomes the child at the root of the other heap. The heap that is being made into a subtree points its root node right sibling pointer to the child of the root of the other heap. Furthermore, the first heap's parent pointer is set to the new heap root, and the new heap root points its leftchild pointer to the root of the heap that is being made into a subtree. The merge operation returns the root of the new heap. (See Figure 3.)

Given the above definition of the merge operation, the DeleteMin operation (see Figure 4 and Listing Three, page 106) is easy to describe. I will describe the front-back one-pass variation here. To begin, save the root node and keep a pointer to the leftchild. Next, empty the pointer to the root. While subtrees are linked to the leftchild of the root, remove trees in pairs (beginning with the leftchild) and merge the trees, then merge the result to the heap pointed to by the root pointer. Repeat this step until there are no more trees. (The pairing heap derives its name from the restructuring operation that takes place during a DeleteMin.)

Describing the DecreaseKey operation for pairing heaps (see Figure 5) is just as easy. This operation assumes that you have direct access to the node whose weight field is being decreased, to a root to the heap that contains the node, and to the value by which you wish to decrease the weight. Go to the parent of the node that is being operated on, and then go to the leftchild of that parent. Scan along the right sibling list to find the predecessor of the node that will be operated upon. When the predecessor is located, clip out the tree rooted at the node upon which you wish to carry out the actual Decrease-Key operation. To clip out the tree, link around the node in question. If the node is a leftchild, make its right sibling the new leftchild. Now decrease the weight and merge the tree that is rooted at the node with the root of the pairing heap.

The simple local restructuring heuristics presented here provide an elegant approach to the development of heap structures. In fact, these heaps are simpler to understand and to implement than either the leftist or binomial heaps. Furthermore, indications are that self-adjusting heaps are just as competitive in practice as their classical counterparts. In any event, I've presented two very different (though effective) local restructuring heuristics. The first heuristic reorganizes lists in order to make frequently requested list items more accessible. The second heuristic applies a simple local restructuring method (in place of maintaining balance/accounting data and resolving special structural cases) in order to quickly maintain both the structure and the partial ordering of a heap.

Now let's consider an efficient self-adjusting heuristic for binary search trees. This algorithm makes frequently requested items in the tree more easily accessible, and quickly maintains both the structure and the sorted ordering of the tree.

Self-Adjusting Binary Search Trees

In a "binary search tree," each node keeps a key along with two pointers to the node's successors. The ordering is such that if a node has key K, every node in that node's left subtree must have keys less than K, and every node in its right subtree must have keys greater than K. This is known as "symmetric ordering." The performance costs of generic binary search tree operations are, in the worst case, logarithmic time (if the input data is sufficiently random). Such a tree may also degenerate as a result of insertions and deletions, and yield steadily poorer performance.

The process of tree degeneration has led to the development of various height/weight balanced trees and B-tree schemes. Although these various schemes guarantee logarithmic worst case times per operation, some of the schemes are not as efficient as possible under nonuniform access patterns. Furthermore, many of these schemes require extra space for balance/accounting information, and the need to keep this information current tends to complicate maintenance of the data structure. Certain cases must be checked on each update, thus incurring a large overhead.

Rotation is the key technique that makes some of the balanced and previous self-adjusting tree schemes possible. In fact, rotation plays a part in the implementation of the splay tree. Before this discussion continues, it is necessary to understand how a right rotation and a left rotation at any node of a binary tree are performed.

As Listing Two (page 105) shows, you implement a right rotation with the following steps: If the pointer to a starting root of some tree is not empty and that node has a left subtree, save pointer to the root and then save pointer to the right subtree of the initial left subtree. Then make the pointer to the initial left subtree the new starting root pointer, and let the original root be the rightchild of the new root. Finally, designate the pointer to the saved rightchild of the original leftchild as a leftchild of the new root's rightchild (which is the original root).

Implement a left rotation in a similar manner. If the pointer to a starting root of some tree isn't empty, and that no has a right subtree, save the pointer the root and then save the pointer the left subtree of the initial right subtree. Then, designate the pointer to the initial right subtree as the new starting root pointer, and let the original root be the leftchild of the new root. Finally designate the pointer to the save leftchild of the original rightchild as rightchild of the new root's leftchild (which is the original root).

The drawbacks of many of the efficient search tree techniques motivate the development of the splay tree. Because binary search trees maintain sorted sets, the question arose as to whether the speed of the search process could be improved if certain item had a higher request frequency than others. In an attempt to improve performance, Allen, Munro, and Bitner proposed two self-adjusting techniques on search trees during the late 1970s. The gist of the first scheme is a single rotation of the item accessed towards the root. The second scheme involves multiple rotations of the accessed item all the way to the root. The techniques are analogous to the variations of the transpose methods for singly linked lists. Neither heuristic is efficient in the amortized sense, since long access sequences exist where the time per access is linear. It is thus clear that the search paths to frequently accessed items need to be as short as possible. Tarjan and Sleator's proposed self-adjusting heuristic halves the depth of each node on the path to an accessed item when the item is finally moved to the root. A splay tree is a binary search tree that employs this heuristic.

Splay Trees

The proposed self-adjusting heuristic has two versions. The "bottom-up splay" is appropriate if you already have direct access to the node that is to be moved to the root. Heuristic restructuring occurs during the second pass back up the search path (assuming that the first pass down the tree is performed in order to find the item). The second version of the proposed self-adjusting heuristic, called a "top-down splay," is an efficient variation of the process used to carry out Tarjan and Sleator's self-adjusting heuristic. This variant requires a pointer to the tree (call it T), that points to the current node to be examined during the search. This heuristic also requires two double pointer records, called L and R (for left and right subtrees), that point to all items less than or greater than the node at T. Figure 6 describes this step.

Figure 6: The top-down splay step

  Case 1  (Top-Down Zig):
          If x is the root of T and y is the
          accessed item (with x=parent(y)
          then if y is a leftchild, then break
          the link from x to y and add x
          to the bottom left of R (else if y
          is a rightchild, add x to the
          bottom right of L) and now T points
          to y, the new root.

  Case 2  (Top-Down Zig-Zig):
          If x is the root of T and y
          (child(x)=y=parent(z))and z are
          both leftchildren (or both
          right-children) then do a rotation to
          the right (or symmetrically, a
          rotation to the left), break the
          link from y to z and attach y to
          the bottom left of R (else
          bottom right of R) and now T points
          to z, the new root.

  Case 3  (Top-Down Zig-Zag):
          If x is the root of T and y is a
          leftchild of x and z s a rightchild
          of y (or y is a rightchild of x and
          z s a leftchild of y), break the
          link from x to y and attach x to
          the bottom left of R (else the
          bottom right of L), break the
          link from y to z and attach y to
          the bottom right of L (else the
          bottom left of R) and now T
          points to z, the new root.

As illustrated in Figure 7, the splaying process repeatedly applies one of the appropriate cases until there are no more cases to apply. At this point, the leftchild of the remaining root is attached to the bottom right of L, and the rightchild is attached to the bottom left of R. The final step points the leftchild of the final remaining root to the subtrees kept by L, and points the rightchild to the subtrees kept by R. (Unlike the bottom-up variation, the top-down heuristic includes the splay step.) When the search/access for a requested node fails, change the last node on the search path into the root of the tree. This step makes the definition of all of the other operations very easy.

To search for a node, simply apply the searching process as described earlier. The process of insertion involves searching for the key V to be inserted. If V is less than the key at the root, make the node that contains V point its leftchild pointer to the leftchild of the root (which breaks the link from the root to that leftchild), and point the rightchild pointer to the root. Otherwise, the leftchild pointer of the node that contains V points to the root, and the rightchild pointer points to the rightchild of the root (and the link from the root to the rightchild is broken). The insertion step is completed by designating the node that contains V as the root.

The split operation is essentially the process of breaking the tree at the root in the manner described in the description of the insertion process. The process of deletion is just as easy. Perform the splay search from the key to be deleted. If the root does not contain the node with the key to be deleted, nothing happens. If the root does contain the node with the key to be deleted, keep pointers to the two subtrees at the root, perform a splay search for the maximum key in the left subtree (and designate the root of the left subtree as the new root), and point the rightchild pointer of the root to the right subtree. The join operation of two binary search trees (assuming that all items in Tree 1 are less than those in Tree 2) is simply the non-no operation of the delete algorithm just described.

The self-adjusting heuristic provides an alternative to the standard balancing/accounting worst-case asymptotic solutions used to develop efficient programs -- and may, in fact, be the method of choice. Furthermore, the algorithms to maintain these self-adjusting data structures are both conceptually easy to understand and simple to implement in practice.

In which applications could these data structures be used to improve performance? Some possibilities are symbol table management applications (MTF lists, splay trees, and possibly in conjunction with hashing schemes), graph algorithms (skew and pairing heaps, particularly with respect to finding minimum spanning trees and the shortest paths in graphs), and other network optimization algorithms (such as splay trees, particularly in maximum/minimum network flow algorithms). Recent work by Jones, Bern, and de Carvalho, as well as my own work, indicates that some of the self-adjusting data structures do seem to perform better in practice than do conventional data structures.

Bibliography

Aho, A.; Hopcroft, J.; and Ullman, J. The Design and Analysis of Computer Algorithms. Reading, Mass.: Addison-Wesley, 1974.

Allen, B. and Munro, I. "Self-Organizing Search Trees." Journal of the ACM 25 (1978).

Bentley, J.L. and McGeoch, C.C. "Amortized Analyses of Self-Organizing Sequential Search Heuristics." Communications of the ACM 28 (1985).

Bern, M. and de Carvalho, M. "A Greedy Heuristic for the Rectilinear Steiner Tree Problem." Report No. UCB/CSD 87/306, Computer Science Division. Berkeley: UC Berkeley (1987).

Bitner, J.R. "Heuristics That Dynamically Organize Data Structures." SIAM Journal of Computing 8 (1979).

Brown, M.R. "Implementation And Analysis Of Binomial Queue Algorithms." SIAM Journal of Computing 7 (1978).

Dietz, P. and Sleator, D. "Two Algorithms for Maintaining Order in a List." Proceedings of the 19th ACM Symposium on Theory of Computing (1987).

Fredman, M.L. and Tarjan, R.E. "Fibonacci Heaps and Their Uses in Improved Network Optimization Algorithms." Proceedings of the 25th Annual IEEE Foundation of Computer Science (1984).

Fredman, M.L. et al. "The Pairing Heap: A New Form of Self-Adjusting Heap." Algorithmica 1 (1986).

Jones, D.W. "An Empirical Comparison of Priority Queue and Event Set Implementations." Communications of the ACM 29 (1986).

Knuth, D.E. The Art Of Computer Programming, Vol. 3: Searching And Sorting, 2nd ed. Reading, Mass.: Addison-Wesley, 1973.

Liao, A.M. "Three Priority Queue Applications Revisited." Submitted to Algorithmica (1988).

Sedgewick, R. Algorithms. Reading, Mass.: Addison-Wesley, 1983.

Sleator, D.D. and Tarjan, R.E. "Self-Adjusting Binary Trees." Proceedings of the 15th ACM Symposium on Theory of Computing (1983).

Sleator, D.D. and Tarjan, R.E. "Self-Adjusting Binary Search Trees." Journal of the ACM 32 (1985).

Sleator, D.D., and Tarjan, R.E. "Self-Adjusting Heaps." SIAM Journal of Computing 15 (1986).

Tarjan, R.E. "Data Structures And Network Algorithms." CBMS Regional Conference Series In Applied Mathematics 44. Philadelphia: SIAM (1983).

Tarjan, R.E. "Amortized Computational Complexity." SIAM Journal of Algebraic Discreet Methods 6 (1985).

Vuillemin, J. "A Data Structure For Manipulating Priority Queues." Communications of the ACM 21 (1978).

Wirth, N. Algorithms + Data Structures = Programs. Englewood Cliffs, New Jersey: Prentice Hall, 1976.

_SELF-ADJUSTING DATA STRUCTURES_ by Andrew M. Liao [LISTING ONE]

{*** Singly linked move-to-the front list ***}
{*** Contents: "LInsert", "Mtffind" ***}

{ Data Structure:
  ptr=^node;
  node=RECORD rec:item; next:ptr; END;  }

PROCEDURE LInsert(arg:item; VAR root:ptr);
    VAR p:ptr;                          { To generate storage }
    BEGIN
      NEW(p);                           { Allocate }
      p^.rec:=arg;                      { Add data }
      p^.next:=root;                    { Place at front of list }
      root:=p;                          { Point to new front of list }
    END;

FUNCTION Mtffind(arg:item; VAR root:ptr;):boolean;
    VAR temp1,temp2:ptr;                   { Search pointers }
        found:boolean;                     { TRUE iff found }
    BEGIN
      temp1:=root;                         { Get a copy of starting location }
      temp2:=root;                         { Secondary copy }
      found:=false;                        { Nothing found yet }

      WHILE (temp1<>NIL) AND (NOT found) DO
          BEGIN
            IF temp1^.rec<>arg THEN        { Found it? }
                BEGIN                      { Nope... }
                  temp2:=temp1;            { Move trailing pointer }
                  temp1:=temp1^.next;      { Move search pointer }
                END
            ELSE found:=true;              { Yup... }
          END;

      IF found THEN                        { Move item to front of list }
          BEGIN
            temp2^.next:=temp1^.next;
            IF temp1<>root THEN temp1^.next:=root;
            root:=temp1;
          END;

      Mtffind:=found;
    END;
[LISTING TWO]

{*** Move To The Front Splay Tree ***}
{*** Contents: SplaySearch, BSInsert, BSDelete ***}

{ Data Structure:
  ptr=^node;
  node=RECORD data:key; left,right:ptr; END;    }

FUNCTION SplaySearch(x:key; VAR p:ptr):boolean;
TYPE noderec=RECORD                     { Temporary Tree Pointer Def. }
               left,right:ptr;
             END;
VAR l,r:noderec;                        { Temporary Trees }
    done:boolean;                       { TRUE if NIL encountered in search }

  PROCEDURE RRot(VAR p:ptr);
    VAR temp,temp1:ptr;                 { Temporary pointers }
    BEGIN
      IF p<>NIL THEN                    { Don't rotate if nothing's there }
        IF p^.left<>NIL THEN            { No left edge - don't rotate }
          BEGIN
            temp:=p; temp1:=p^.left^.right;  { Copy root & 2ndary child }
            p:=temp^.left; p^.right:=temp;   { Rotate root }
            temp^.left:=temp1;         { Reattach 2ndary child }
          END;
    END;

  PROCEDURE LRot(VAR p:ptr);
    VAR temp,temp1:ptr;                 { Temporary pointers }
    BEGIN
      IF p<>NIL THEN                    { Don't rotate if nothing's there }
        IF p^.right<>NIL THEN           { No right edge - don't rotate }
          BEGIN
            temp:=p; temp1:=p^.right^.left;  { Copy root & 2ndary child }
            p:=temp^.right; p^.left:=temp;   { Rotate root }
            temp^.right:=temp1;        { Reattach 2ndary child }
          END;
    END;

  PROCEDURE LnkRight(VAR p:ptr; VAR r:noderec);
    VAR temp:ptr;                       { Temporary pointer }
    BEGIN
      IF p^.left<>NIL THEN              { No left child - don't cut & link }
        BEGIN
          temp:=p^.left; p^.left:=NIL;  { Remember left child & break link }
          IF r.left=NIL THEN            { Attach to temporary tree }
            BEGIN r.left:=p; r.right:=p;END { Empty tree? }
          ELSE                          { Just add to bottom leftmost }
            BEGIN r.right^.left:=p; r.right:=r.right^.left; END;
          p:=temp;                      { New root is left child }
        END;
    END;

  PROCEDURE LnkLeft(VAR p:ptr; VAR l:noderec);
    VAR temp:ptr;                       { Temporary pointer }
    BEGIN
      IF p^.right<>NIL THEN             { No right child - don't cut & link }
        BEGIN
          temp:=p^.right; p^.right:=NIL;{ Remember right child & break link }
          IF l.left=NIL THEN            { Attach to temporary tree }
            BEGIN l.left:=p; l.right:=p;END { Empty tree? }
          ELSE                          { Just add to bottom rightmost }
            BEGIN l.right^.right:=p; l.right:=l.right^.right; END;
          p:=temp;                      { New root is right child }
        END;
    END;

  PROCEDURE Assemble(VAR p:ptr; VAR l,r:noderec);
    VAR temp,temp1:ptr;
    BEGIN
      temp:=p^.left; temp1:=p^.right;   { Hold onto subtrees }
      IF l.left<>NIL THEN
        BEGIN
          p^.left:=l.left;              { Attach temporary left subtree }
          l.right^.right:=temp;         { Reattach orginal left subtree }
        END;
      IF r.left<>NIL THEN
        BEGIN
          p^.right:=r.left;             { Attach temporary right subtree }
          r.right^.left:=temp1;         { Reattach original right subtree }
        END;
    END;

  BEGIN
    l.left:=NIL; l.right:=NIL;          { Initialize temp trees }
    r.left:=NIL; r.right:=NIL;
    done:=false;                        { Init to "item maybe there" }
    IF p<>NIL THEN                      { No search if tree's empty }
      BEGIN
      REPEAT
        IF (x<p^.data) THEN             { Item on left subtree? }
          IF (p^.left<>NIL) THEN
            BEGIN
              IF x=p^.left^.data THEN LNKRIGHT(p,r)
              ELSE
              IF x<p^.left^.data THEN BEGIN RRot(p); LNKRIGHT(p,r); END
              ELSE
              IF x>p^.left^.data THEN BEGIN LNKRIGHT(p,r);LNKLEFT(p,l);END;
            END ELSE done:=TRUE
        ELSE
        IF (x>p^.data) THEN             { Item on right subtree? }
          IF (p^.right<>NIL) THEN
            BEGIN
              IF x=p^.right^.data THEN LNKLEFT(p,l)
              ELSE
              IF x>p^.right^.data THEN BEGIN LRot(p); LNKLEFT(p,l); END
              ELSE
              IF x<p^.right^.data THEN BEGIN LNKLEFT(p,l);LNKRIGHT(p,r);END;
            END ELSE done:=TRUE;
      UNTIL (x=p^.data) OR DONE;
      ASSEMBLE(p,l,r); SplaySearch:=(x=p^.data);
    END ELSE SplaySearch:=FALSE;
  END;

PROCEDURE BSInsert(x:key; VAR root:ptr);
VAR p:ptr;
BEGIN
  NEW(p);
  p^.data:=x;
  p^.left:=NIL; p^.right:=NIL;
  IF root=NIL THEN root:=p                      { No tree, just insert }
  ELSE
  BEGIN
    IF NOT SplaySearch(x,root) THEN             { Is it already there? }
      IF x<root^.data THEN                      { Less than? }
        BEGIN
          p^.right:=root;                       { Root item greater than }
          p^.left:=root^.left;                  { Link up left child }
          root^.left:=NIL; root:=p;             { Break link; root=new item }
        END
      ELSE
      IF x>root^.data THEN                      { Greater than? }
        BEGIN
          p^.left:=root;                        { Root item less than }
          p^.right:=root^.right;                { Link up right child }
          root^.right:=NIL; root:=p;            { Break link; root=new item }
        END;
  END;
END;

PROCEDURE BSDelete(x:key; VAR root:ptr);
VAR temp1,temp2,temp4:ptr;
    temp3:key;
    flg:boolean;
BEGIN
  IF SplaySearch(x,root) THEN
    BEGIN
      temp1:=root^.left; temp2:=root^.right;    { Save subtrees }
      IF temp1<>NIL THEN                        { Is there a left subtree? }
        BEGIN
          temp4:=temp1;
          WHILE temp4^.right<>NIL DO            { MTF max left tree element }
            temp4:=temp4^.right;
          temp3:=temp4^.right^.data;
          flg:=SplaySearch(temp3,temp1);
          temp1^.right:=temp2;                  { Attach right subtree }
        END ELSE temp1:=temp2;                  { Just attach right tree }
       dispose(root);
       root:=temp1;                             { Return new tree }
    END;
END;
[LISTING THREE]

{*** Self-adjusting heap ***}
{*** Contents: Merge, Min, Insert, DeleteMin routines ***}

{ Data Structure:
  ptr=^node;
  node=RECORD data:item; left,right:ptr; END;   }

FUNCTION Merge(q1,q2:ptr):ptr;
  TYPE Qrec=RECORD
                front,rear:ptr;
            END;
  VAR Q:Qrec;
  PROCEDURE Enqueue(VAR q1:ptr; VAR Q:Qrec);
    VAR temp:ptr;
    BEGIN
      temp:=q1;                         { Save top of heap }
      q1:=q1^.right;                    { Point to next top of heap }
      temp^.right:=temp^.left;          { Swap right child to left }
      temp^.left:=NIL;                  { Make sure left link's broken }
      IF q.front=NIL THEN               { Empty merge queue }
        BEGIN
          q.front:=temp; q.rear:=temp;
        END
      ELSE                              { Oops, just add to last leftchild }
        BEGIN
          q.rear^.left:=temp; q.rear:=temp;
        END;
    END;
  BEGIN
    q.front:=NIL; q.rear:=NIL;          { Init merge queue }
    WHILE (q1<>NIL) AND (q2<>NIL) DO    { Pairwise compare and merge }
        IF q1^.data<=q2^.data THEN Enqueue(q1,q)
        ELSE Enqueue(q2,q);

    IF (q1<>NIL) AND (q2=NIL) THEN
      BEGIN
        IF q.rear<>NIL THEN q.rear^.left:=q1
        ELSE q.front:=q1;
      END
    IF (q1=NIL) AND (q2<>NIL) THEN
      BEGIN
        IF q.rear<>NIL THEN q.rear^.left:=q2
        ELSE q.front:=q2;
      END;
    Merge:=q.front;
  END;

FUNCTION Min(q1:ptr; VAR x:ptr):boolean;
  BEGIN
    x:=q1;
    Min:=(q1<>NIL);
  END;

PROCEDURE Insert(x:item; VAR q:ptr);
  VAR p:ptr;
  BEGIN
    NEW(p);                             { Allocate }
    p^.data:=x;                         { Fill it! }
    p^.left:=NIL; p^.right:=NIL;        { No children }
    q:=Merge(q,p);                      { Add it to heap }
  END;

FUNCTION DeleteMin(q:ptr; VAR x:ptr):ptr;
  BEGIN
    IF Min(q,x) THEN                    { Is there a min to delete? }
      DeleteMin:=Merge(q^.left,q^.right)
    ELSE DeleteMin:=NIL;                { Nothing at all }
  END;

{ Pairing Heaps as described by Tarjan, et al from Algorithmica:
  Data Structure:
  TYPE hptr=^node;
       node=RECORD
              wt:integer;
              parent,left,right:hptr;
            END;                        }

FUNCTION Merge(arg1,arg2:hptr):hptr;
BEGIN
  IF (arg1<>NIL) AND (arg2<>NIL) THEN           { 2 Queues to merge? }
    BEGIN
      IF arg1^.wt<arg2^.wt THEN         { Which is minimal? }
        BEGIN
          arg2^.parent:=arg1;                   { Who's the parent? }
          arg2^.right:=arg1^.left;              { Point to arg1's child }
          arg1^.left:=arg2;                     { It's officially a child }
          Merge:=arg1;
        END
      ELSE
        BEGIN
          arg1^.parent:=arg2;                   { Who's the parent? }
          arg1^.right:=arg2^.left;              { Point to arg2's child }
          arg2^.left:=arg1;                     { It's officially a child }
          Merge:=arg2;
        END;
     END
   ELSE
   IF (arg1<>NIL) THEN Merge:=arg1              { Just arg1's queue }
   ELSE Merge:=arg2                             { Anything else }
END;

PROCEDURE Insert(a1,a2,x:integer; VAR root:hptr);
VAR p:hptr;
BEGIN
  New(p);                                       { Allocate }
  p^.v1:=a1; p^.v2:=a2;
  p^.wt:=x;  p^.parent:=NIL;                    { Set key }
  p^.left:=NIL; p^.right:=NIL;                  { Set pointers }
  root:=Merge(p,root);                          { Add it... }
END;

FUNCTION Min(root:hptr; VAR minitem:hptr):boolean;
BEGIN
  minitem:=root;                                { What's at the root? }
  Min:=(minitem<>NIL);                          { Anything there? }
END;

FUNCTION DeleteMin(root:hptr; VAR minitem:hptr):hptr;
VAR arg1,arg2,p1:hptr;
BEGIN
  IF Min(root,minitem) THEN
    BEGIN
      root:=NIL;                                { ReInit root }
      p1:=minitem^.left;                        { Save kids }
      WHILE p1<>NIL DO                          { For all subtrees }
        BEGIN
          arg1:=p1;                             { First Subtree }
          p1:=p1^.right;                        { Move along }
          arg2:=p1;                             { Next potential subtree }
          IF p1<>NIL THEN p1:=p1^.right;        { If not NIL, move on }
          root:=Merge(Merge(arg1,arg2),root);   { Merge result with current }
        END;
       IF root<>NIL THEN root^.right:=NIL;
       DeleteMin:=root;
    END ELSE DeleteMin:=NIL;
END;

FUNCTION LinkSearch(p:hptr):hptr;
VAR temp:hptr;
BEGIN
  temp:=p^.parent^.left;
  WHILE (temp<>p) AND (temp^.right<>p) AND (temp^.right<>NIL) DO
    temp:=temp^.right;
  LinkSearch:=temp;
END;

FUNCTION DecreaseKey(change:integer; p,root:hptr):hptr;
VAR temp:hptr;
BEGIN
  IF (p<>NIL) AND (root<>NIL) THEN
    BEGIN
      p^.wt:=p^.wt-ABS(change);
      IF p=root THEN DecreaseKey:=root
      ELSE
      BEGIN
        temp:=LinkSearch(p);
        IF temp=p THEN p^.parent^.left:=p^.parent^.left^.right
        ELSE temp^.right:=p^.right;
        DecreaseKey:=Merge(p,root);
      END;
    END;
END;

FUNCTION Delete(p,root:hptr):hptr;
VAR temp:hptr;
BEGIN
  IF (p<>NIL) AND (root<>NIL) THEN
    BEGIN
      IF p=root THEN Delete:=DeleteMin(root,temp)
      ELSE
      BEGIN
        temp:=LinkSearch(p);
        IF temp=p THEN p^.parent^.left:=p^.parent^.left^.right
        ELSE temp^.right:=p^.right;
        Delete:=Merge(DeleteMin(p,temp),root);
      END;
    END ELSE Delete:=root;
END;

Back to Table of Contents


Example 1(f)

 s


OCT90: IMPLEMENTING CORDIC ALGORITHMS

IMPLEMENTING CORDIC ALGORITHMS

A single compact routine for computing transcendental functions

Pitts Jarvis

Pitts Jarvis is a senior engineer at 3Com Corporation. He can be reached at 1275 Martin Ave., Palo Alto, CA 94301.

Efficiently computing sines, cosines, and other transcendental functions is a process about which many programmers are blissfully ignorant. When these values are called for in a graphics or CAD program, we usually rely on a call to the compiler's run-time library. The library either derives the necessary values in some mysterious manner or calls the floating-point coprocessor to assist in the task.

The CORDIC (COordinate, Rotation DIgital Computer) family of algorithms is an elegant, efficient, and compact way to compute sines, cosines, exponentials, logarithms, and associated transcendental functions using one core routine. These truly remarkable algorithms compute these functions with n bits of accuracy in n iterations -- where each iteration requires only a small number of shifts and additions. Furthermore, these routines use only fixed-point arithmetic. Using these algorithms, you can cast your entire graphics application into fixed-point, and thus avoid the cost of run-time conversion from fixed- to floating-point representation and back.

Even if you don't plan on recasting your application into fixed-point, you just might be curious how your floating-point coprocessor works. The Intel numerics family (8087, 80287, and 80387) all use Cordic algorithms, in a form slightly different than described here, to compute circular functions. The Intel implementations are described by R. Nave1 and A. K. Yuen2.

The implementations may be contemporary, but the algorithms are not new. J. E. Volder3 coined the name in 1959. He applied these algorithms to build a special-purpose digital computer for real-time airborne navigation. D. S. Cochran4 identifies their use in the HP-35 calculator in 1972 to calculate the transcendental functions.

Mathematical Manipulation

If we have a vector [x, y], we can rotate it through an angle a by multiplying it by the matrix Ra, defined in Example 1(a). Explicitly doing the matrix multiplication yields the equation in Example 1(b).

If we choose x = 1 and y = 0 and multiply that vector by Ra we are left with the vector [cos a, sin a].

Multiplying by two successive rotation matrices; Ra and Rb rotates the vector through the angle a + b, or more formally RaRb - Ra+b. If we choose to represent the angle a as a sum of angles ai for i = 0 through n (see Example 1(c) , then we can rotate the vector through the angle a by multiplying a series of rotation matrices Rao, Ra1, . . . Ran.

By picking the ai carefully, we can simplify the arithmetic. Notice that we can rewrite the rotation matrix by factoring out cos a as shown in Example 1(d). If we pick ai such that tan ai = 2-i for i = 0 through n, all of the multiplications by tan ai become right shifts by i bits.

Now we need to specify an algorithm so that we can represent a as the sum of the ai. Initialize a variable, z, to a. This z will be a residue quantity, which we are trying to drive to zero by adding or subtracting ai at the i-th step. At the first step, i= 0. At the i-th step, if z [image] 0 then subtract ai from z. Otherwise add ai to z. At the last step i = n, and z is the error in our representation of a. Notice that in Example 1(e), for large i, each additional step yields one more bit of accuracy in our representation of a.

Figure 1 shows the relative magnitudes of the incremental angles, ai. Figure 2 gives an example of the convergence process with an initial angle of 0.65. Notice that successive iterations do not necessarily reduce the absolute error in the representation of the angle. Also notice that the error does not oscillate about zero.

At each step as we decompose a into the sum or difference of the ai. Figure 2 gives an example of the convergence pro's we could also multiply our vector [x, y] by the appropriate Rai. Figure 2 gives an example of the convergence pro or R-ai. Figure 2 gives an example of the convergence pro depending on whether we add or subtract ai. Figure 2 gives an example of the convergence pro. Remember, these multiplications are nothing more than shifts. We must also multiply in the still embarrassing factor cos ai. Figure 2 gives an example of the convergence pro. However, cosine is an even function and has the property that cos ai - cos (-a1). It does not matter whether we add or subtract the angle - we always multiply by the same factor! Because all of the cos ai can be factored out and grouped together, we can treat their product as a constant and compute it only once, along with all the ai = tan-12-i.

Not all angles can be represented as the sum of ai. There is a domain of convergence outside of which we cannot reduce the angle to within an-1 of zero. See Example 1(f). For the algorithm to work, we must start with a such that |a| [image] amax [image] 1.74. This conveniently falls just outside the first quadrant. If we are given an angle outside the first quadrant, we can scale it by dividing by [image] /2 obtaining a quotient Q and a remainder D where |D| < [image] /2 < a?? Since the algorithm computes both sine and cosine of D, we pick the appropriate value and sign depending on the value of Q.

What about angles within the domain of convergence? It's not obvious that the strange set we've picked (see Example 1(g)) can represent all angles within the domain of convergence to within a~~ But, using mathematical induction. Walther proves that the scheme works.

The Circular Functions

One variation of the Cordic algorithm computes the circular functions -- sin, cos, tan, and so on. This algorithm is shown in pseudocode in Example 2(a).

First, start with [x,y,z] The x and y are as before z is the quantity that we drive to zero with an initial value of angle a. The first step in the loop decides whether to add or subtract ai from the residue z. The variable s is correspondingly positive or negative The second step reduces the magnitude of z and effects the multiplications by the tan a~ The expression ~~~~ means shift y right by ~ bits.

Example 2: (a) The basic algorithm; (b) the inverse algorithm

  (a)
  for i from 0 to n do
  {
          if (z    0) then s:= 1 else s:= -1;
          [x,y,z] := [x - s y>>i, y + s x>>i, z - s a1]
  }

  (b)
  for i from 0 to n do
  {
          if (y    0) then s:= 1 else s:= -1;
          [x,y,z] := [x + s y>>i, y - s x>>i, z - s a1]
  }

When you start the algorithm with [x,y,z] and then drive z to zero as specified by Example 2(a). We are left with the quantities in Example 3(a). Where K is a constant. It is just the product of the cos ai as in Example 3(b).

For the curious, K [image] 0.607. The value of K can be precomputed by setting [x,y,z] to [1,0,0] and running the algorithm as before. The result is shown in Example 3(c). Take the reciprocal of the final x and we have K. Therefore to compute sin a and cos a, set [x,y,z] to [K, O, ~] and run the algorithm, Example 3(d) shows the result. In effect, we start with a vector, [x,y] and rotate it through a given ~~ a my driving z to zero. Running the algorithm with the special case where the vector initially lies along the x axis and is of length K, rotates the vector by angle a and leaves behind cos a and sin a. This relationship is shown in Figure 3.

To compute tan 1a instead of z, we could choose to drive y to zero. Driv. ing y to zero rotates the vector through the angle a, the angle subtended by the vector and the x axis, leaving the vector lying along the x axis. Start with the vector anywhere in the first or fourth quadrant and an initial value of zero in z. The first or fourth quadrant is used because almost all vectors in the second or third quadrant will not converge. At the i-th step, if y >- 0, the vector lies in the first quadrant, subtract ai from z. Move the vector closer to the x axis; rotate it by negative ai by multiplying by the rotation matrix R-ai. If y < 0, the vector lies in the fourth quadrant, add ai to z and multiply the vector by Rai. At the end, z has the negative of the angle of the original vector [x, y], tan-1 y/x = tan-1a.

Changing the sign of ai has no effect on the computed values of x and y and leaves the original angle a in z rather than its negative. With this change, the inverse algorithm to drive y to zero becomes the expression shown as in the algorithm in Example 2(b).

Starting with [x, y, z] and then driving y to zero using the inverse algorithm leaves behind the quantities in Example 3(e).

Hyperbolic Functions

The hyperbolic functions (sinh, cosh, and so on) are similar to the circular functions. The correspondences between these two types of functions are shown in Table 1.

Table 1: Hyperbolic functions

Hyperbolic Function               Circular Function
---------------------------------------------------

              ex+e-x                    eix + e-ix
  cosh x = ----------         cos x = --------------
               2                            2
              ex-e-x                    eix - e-ix
  sinh x = ----------         sin x = --------------
               2                       2i
       [cosh a sinh a]           [cosh a -sin a]
  Ha = [sinh a cosh a]      Ra = [sin a cos a]

  HaHb = Ha+b           RaRb = Ra+b

  ex = cosh x + sinh x

                    x-1
  In x = 2 tanh-1 ---
                    x+1

By analogy, use Ha as the rotation matrix and represent a using the set ai = tanh-1 2-i for i = 1 to n. Notice that for hyperbolics, a0 is infinity.

Given the change in the ai, can we still represent any angle a within the domain of convergence the same way we did for the circular functions? Unfortunately, the answer is no! Walther points out that repeating an occasional term makes the representation converge in the hyperbolic case. Repeating the terms as shown in Example 4(a) does the trick.

Except for the repeated terms and some changes of sign, the algorithms for hyperbolic functions are identical to the circular functions. Listing One, page 157, shows this in detail.

For hyperbolic functions, we start with [x,y,z] and then drive z to zero. This yields the quantities in Example 4(b). Starting with x,y,z and then driving y to zero gives the quantity shown in Example 4(c). For hyperbolics, K [image] 1.21.

Some interesting special cases include the exponential, square root, and natural logarithm. The exponential case is in Example 4(d) while the square root and logarithm cases are in Example 4(e).

Calculating the Constants

The algorithm to compute the circular and hyperbolic functions requires several precomputed constants. These include the scaling constant, K, for both circular and hyperbolic functions, and the sets shown in Example 5(a) and Example 5(b), respectively. Listing One illustrates this.

The program, written in C, uses fixed point arithmetic for all calculations. All constants and variables used to calculate functions are declared as the type long. The code assumes that a long is at least 32 bits. I have decided to represent numbers in the range - 4 <= x < 4; this lets me represent e as a fixed point number. The high order bit is for the sign. The low order fractionBits (a constant defined as 29) bits hold the fractional part of the number. The remainder of the bits between the sign bit and the fractional part hold the integer part of the number. Figure 4 shows the fixed point format in graphic form.

I use power series to calculate the incremental angles ai, as shown in Example 5(c) and Example 5(d), respectively. How do we know the number of terms necessary to evaluate tan-1 and tanh-1 to 32 bits of precision? First consider the value of x for which tan-1 x = x to 32 bits of precision. A theorem of numerical analysis states that for an alternating sum where the absolute values of the terms decrease monotonically, the error is less than the absolute value of the first neglected term. Solving the equation x3/3 = 2-32 for x yields x = 3 [image] (6 * 2-11); therefore for i [image] 11, tan-1 2-i = 2-i with 32 bits of precision.

For the higher powers of two, we need to solve the relation 2-in/n = 2-32 for n for each of the cases i = 1 to 10. We do not even attempt the calculation for i = 0. The series for tan-1 1 converges very slowly, even after 500 terms the third digit is still changing. Fortunately, we know that the answer is [image] /4. Computing the rest is not as much work. The array terms has the gory details.

As usual, tanh-1 is more perverse. It is not an alternating sum and does not meet the conditions of the theorem used above. Consider the second neglected term of tanh-1 1/2. It is less than 1/4 of the first neglected term because the series includes only every other power of two. All of the other neglected terms can have no effect on the 33rd bit. The series for the other arguments, 1/4, 1/8, ..., converges even faster. Therefore, the number of terms calculated for tan-1 works just as well for tanh-1 for 32-bit accuracy.

Before computing the power series, we still need to compute the coefficients, 1/k, for each term k = 1, 3, 5, ... 27. We fill the coefficient array long a[28] with odd indices by calling the routine Reciprocal, which takes two arguments and returns a long. The first argument is the integer for the desired reciprocal. The second specifies the desired precision for the fractional part of the result. Reciprocal uses a simple as can be restoring division; it is the algorithm we all learned in grade school for long division. The elements of the array a with even indices get OL because there are no terms in the power series with even exponents.

Everything is ready to fill the arrays atan[fractionBits+ 1] and atanh[fractionBits+1].

The routine Poly2 evaluates the power series for the specified number of terms for the specified power of two using Horner's rule. The coefficients come from the array a, which we just carefully filled. Horner's rule is the recommended method for evaluating polynomials. A polynomial as in Example 5(e) can be rewritten as in Example 5(f). This simple recursive formula evaluates the polynomial with n multiplications and n additions. We compute the prescaling constants K by using the method explained above; in the program we call these XOC and XOH, for the circular and hyperbolic constants, respectively. Program output to this point is shown in Listing Two, page 158.

The routines Circular, InvertCircular, Hyperbolic, and InvertHyperbolic are the C implementations of the algorithms described above. They all take as arguments the initial values for [x,y,z]; they leave their results in the global variables X, Y, and Z. Considering their versatility and the wide range of functions they compute, these routines are compact and elegant!

References

    1. R. Nave. "Implementation of Transcendental Functions on a Numerics Processor." Microprocessing and Microprogramming, vol. 11, num. 3 - 4, pp. 221 - 225, March - April 1983.

    2. A. K. Yuen. "Intel's Floating-Point Processors." Electro/88 Conference Record, pp. 48/5/1 - 7, 1988.

    3. J. E. Volder. "The Cordic TrigonometricComputing Technique." IRE Transactions Electronic Computers, vol. EC - 8, pp. 330 - 334, September 1959.

    4. D. S. Cochran. "Algorithms and Accuracy in the HP-35." HewlettPackard Journal, pp. 10 - 11, June 1972.

    5. J. S. Walther. "A Unified Algorithm for Elementary Functions." In 1971 Proceedings of the Joint Spring Computer Conference, pp. 379 - 385, 1971.

_IMPLEMENTING CORDIC ALGORITHMS_ by Pitts Jarvis [LISTING ONE]

/* cordicC.c -- J. Pitts Jarvis, III
 * cordicC.c computes CORDIC constants and exercises the basic algorithms.
 * Represents all numbers in fixed point notation.  1 bit sign,
 * longBits-1-n bit integral part, and n bit fractional part.  n=29 lets us
 * represent numbers in the interval [-4, 4) in 32 bit long.  Two's
 * complement arithmetic is operative here.
 */

#define fractionBits 29
#define longBits 32
#define One    (010000000000>>1)
#define HalfPi (014441766521>>1)

/* cordic algorithm identities for circular functions, starting with [x, y, z]
 *  and then
 *  driving z to 0 gives: [P*(x*cos(z)-y*sin(z)), P*(y*cos(z)+x*sin(z)), 0]
 *  driving y to 0 gives: [P*sqrt(x^2+y^2), 0, z+atan(y/x)]
 *  where K = 1/P = sqrt(1+1)* . . . *sqrt(1+(2^(-2*i)))
 *  special cases which compute interesting functions
 *  sin, cos      [K, 0, a] -> [cos(a), sin(a), 0]
 *  atan          [1, a, 0] -> [sqrt(1+a^2)/K, 0, atan(a)]
 *      [x, y, 0] -> [sqrt(x^2+y^2)/K, 0, atan(y/x)]
 * for hyperbolic functions, starting with [x, y, z] and then
 * driving z to 0 gives: [P*(x*cosh(z)+y*sinh(z)), P*(y*cosh(z)+x*sinh(z)), 0]
 * driving y to 0 gives: [P*sqrt(x^2-y^2), 0, z+atanh(y/x)]
 * where K = 1/P = sqrt(1-(1/2)^2)* . . . *sqrt(1-(2^(-2*i)))
 *  sinh, cosh    [K, 0, a] -> [cosh(a), sinh(a), 0]
 *  exponential   [K, K, a] -> [e^a, e^a, 0]
 *  atanh         [1, a, 0] -> [sqrt(1-a^2)/K, 0, atanh(a)]
 *         [x, y, 0] -> [sqrt(x^2-y^2)/K, 0, atanh(y/x)]
 *  ln            [a+1, a-1, 0] -> [2*sqrt(a)/K, 0, ln(a)/2]
 *  sqrt          [a+(K/2)^2, a-(K/2)^2, 0] -> [sqrt(a), 0, ln(a*(2/K)^2)/2]
 *  sqrt, ln      [a+(K/2)^2, a-(K/2)^2, -ln(K/2)] -> [sqrt(a), 0, ln(a)/2]
 * for linear functions, starting with [x, y, z] and then
 *  driving z to 0 gives: [x, y+x*z, 0]
 *  driving y to 0 gives: [x, 0, z+y/x]
 */

long X0C, X0H, X0R; /* seed for circular, hyperbolic, and square root */
long OneOverE, E;   /* the base of natural logarithms */
long HalfLnX0R;       /* constant used in simultanous sqrt, ln computation */

/* compute atan(x) and atanh(x) using infinite series
 *    atan(x) =  x - x^3/3 + x^5/5 - x^7/7 + . . . for x^2 < 1
 *     atanh(x) = x + x^3/3 + x^5/5 + x^7/7 + . . . for x^2 < 1
 * To calcuate these functions to 32 bits of precision, pick
 * terms[i] s.t. ((2^-i)^(terms[i]))/(terms[i]) < 2^-32
 * For x <= 2^(-11), atan(x) = atanh(x) = x with 32 bits of accuracy  */
unsigned terms[11]= {0, 27, 14, 9, 7, 5, 4, 4, 3, 3, 3};static long a[28], atan[fractionBits+1], atanh[fractionBits+1], X, Y, Z;
#include <stdio.h>   /* putchar is a marco for some */

/* Delta is inefficient but pedagogical */
#define Delta(n, Z) (Z>=0) ? (n) : -(n)
#define abs(n) (n>=0) ? (n) : -(n)

/* Reciprocal, calculate reciprocol of n to k bits of precision
 *  a and r form integer and fractional parts of the dividend respectively  */
long
Reciprocal(n, k) unsigned n, k;
{
  unsigned i, a= 1; long r= 0;
  for (i= 0; i<=k; ++i) {r += r; if (a>=n) {r += 1; a -= n;}; a += a;}
  return(a>=n? r+1 : r); /* round result */
}

/* ScaledReciprocal, n comes in funny fixed point fraction representation */
long
ScaledReciprocal(n, k) long n; unsigned k;
{
  long a, r=0; unsigned i;
  a= 1L<<k;
  for (i=0; i<=k; ++i) {r += r; if (a>=n) {r += 1; a -= n;}; a += a;};
  return(a>=n? r+1 : r); /* round result */
}

/* Poly2 calculates polynomial where the variable is an integral power of 2,
 *   log is the power of 2 of the variable
 *   n is the order of the polynomial
 *   coefficients are in the array a[]   */
long
Poly2(log, n) int log; unsigned n;
{
  long r=0; int i;
  for (i=n; i>=0; --i) r= (log<0? r>>-log : r<<log)+a[i];
  return(r);
}
WriteFraction(n) long n;
{
  unsigned short i, low, digit; unsigned long k;
  putchar(n < 0 ? '-' : ' '); n = abs(n);
  putchar((n>>fractionBits) + '0'); putchar('.');
  low = k = n << (longBits-fractionBits); /* align octal point at left */
  k   >>=  4;   /* shift to make room for a decimal digit */
  for (i=1; i<=8; ++i)
  {
    digit = (k *= 10L) >> (longBits-4);
    low = (low & 0xf) * 10;
    k += ((unsigned long) (low>>4)) - ((unsigned long) digit << (longBits-4));
    putchar(digit+'0');
  }
}
WriteRegisters()
{  printf("  X: "); WriteVarious(X);
  printf("  Y: "); WriteVarious(Y);
  printf("  Z: "); WriteVarious(Z);
}
WriteVarious(n) long n;
{
  WriteFraction(n); printf("  0x%08lx 0%011lo\n", n, n);
}
Circular(x, y, z) long x, y, z;
{
  int i;
  X = x; Y = y; Z = z;
  for (i=0; i<=fractionBits; ++i)
  {
    x= X>>i; y= Y>>i; z= atan[i];
    X -= Delta(y, Z);
    Y += Delta(x, Z);
    Z -= Delta(z, Z);
  }
}
InvertCircular(x, y, z) long x, y, z;
{
  int i;
  X = x; Y = y; Z = z;
  for (i=0; i<=fractionBits; ++i)
  {
    x= X>>i; y= Y>>i; z= atan[i];
    X -= Delta(y, -Y);
    Z -= Delta(z, -Y);
    Y += Delta(x, -Y);
  }
}
Hyperbolic(x, y, z) long x, y, z;
{
  int i;
  X = x; Y = y; Z = z;
  for (i=1; i<=fractionBits; ++i)
  {
    x= X>>i; y= Y>>i; z= atanh[i];
    X += Delta(y, Z);
    Y += Delta(x, Z);
    Z -= Delta(z, Z);
    if ((i==4)||(i==13))
    {
      x= X>>i; y= Y>>i; z= atanh[i];
      X  +=  Delta(y, Z);
      Y += Delta(x, Z);
      Z  -=  Delta(z, Z);
    }
  }
}
InvertHyperbolic(x, y, z) long x, y, z;
{
  int i;
  X = x; Y = y; Z = z;  for (i=1; i<=fractionBits; ++i)
  {
    x= X>>i; y= Y>>i; z= atanh[i];
    X += Delta(y, -Y);
    Z -= Delta(z, -Y);
    Y += Delta(x, -Y);
    if ((i==4)||(i==13))
    {
      x= X>>i; y= Y>>i; z= atanh[i];
      X += Delta(y, -Y);
      Z -= Delta(z, -Y);
      Y += Delta(x, -Y);
    }
  }
}
Linear(x, y, z) long x, y, z;
{
  int i;
  X = x; Y = y; Z = z; z= One;
  for (i=1; i<=fractionBits; ++i)
  {
    x  >>= 1; z  >>= 1; Y += Delta(x, Z); Z -= Delta(z, Z);
  }
}
InvertLinear(x, y, z) long x, y, z;
{
  int i;
  X = x; Y = y; Z = z; z= One;
  for (i=1; i<=fractionBits; ++i)
  {
     Z -= Delta(z  >>= 1, -Y); Y += Delta(x  >>= 1, -Y);
  }
}

/*********************************************************/
main()
{
  int i; long r;
  /*system("date");*//* time stamp the log for UNIX systems */
  for (i=0; i<=13; ++i)
  {
    a[2*i]= 0; a[2*i+1]= Reciprocal(2*i+1, fractionBits);
  }
  for (i=0; i<=10; ++i) atanh[i]= Poly2(-i, terms[i]);
  atan[0]= HalfPi/2; /* atan(2^0)= pi/4 */
  for (i=1; i<=7; ++i) a[4*i-1]= -a[4*i-1];
  for (i=1; i<=10; ++i) atan[i]= Poly2(-i, terms[i]);
  for (i=11; i<=fractionBits; ++i) atan[i]= atanh[i]= 1L<<(fractionBits-i);
  printf("\natanh(2^-n)\n");
  for (i=1; i<=10; ++i){printf("%2d ", i); WriteVarious(atanh[i]);}
  r= 0;
  for (i=1; i<=fractionBits; ++i)
     r +=  atanh[i];
  r +=  atanh[4]+atanh[13];
  printf("radius of convergence"); WriteFraction(r);  printf("\n\natan(2^-n)\n");
  for (i=0; i<=10; ++i){printf("%2d ", i); WriteVarious(atan[i]);}
  r= 0; for (i=0; i<=fractionBits; ++i) r +=  atan[i];
  printf("radius of convergence"); WriteFraction(r);

  /* all the results reported in the printfs are calculated with my HP-41C */
  printf("\n\n--------------------circular functions--------------------\n");
    printf("Grinding on [1, 0, 0]\n");
    Circular(One, 0L, 0L); WriteRegisters();
    printf("\n  K: "); WriteVarious(X0C= ScaledReciprocal(X, fractionBits));
    printf("\nGrinding on [K, 0, 0]\n");
    Circular(X0C, 0L, 0L); WriteRegisters();
    printf("\nGrinding on [K, 0, pi/6] -> [0.86602540, 0.50000000, 0]\n");
    Circular(X0C, 0L, HalfPi/3L); WriteRegisters();
    printf("\nGrinding on [K, 0, pi/4] -> [0.70710678, 0.70710678, 0]\n");
    Circular(X0C, 0L, HalfPi/2L); WriteRegisters();
    printf("\nGrinding on [K, 0, pi/3] -> [0.50000000, 0.86602540, 0]\n");
    Circular(X0C, 0L, 2L*(HalfPi/3L)); WriteRegisters();
  printf("\n------Inverse functions------\n");
    printf("Grinding on [1, 0, 0]\n");
    InvertCircular(One, 0L, 0L); WriteRegisters();
    printf("\nGrinding on [1, 1/2, 0] -> [1.84113394, 0, 0.46364761]\n");
    InvertCircular(One, One/2L, 0L); WriteRegisters();
    printf("\nGrinding on [2, 1, 0] -> [3.68226788, 0, 0.46364761]\n");
    InvertCircular(One*2L, One, 0L); WriteRegisters();
    printf("\nGrinding on [1, 5/8, 0] -> [1.94193815, 0, 0.55859932]\n");
    InvertCircular(One, 5L*(One/8L), 0L); WriteRegisters();
    printf("\nGrinding on [1, 1, 0] -> [2.32887069, 0, 0.78539816]\n");
    InvertCircular(One, One, 0L); WriteRegisters();
  printf("\n--------------------hyperbolic functions--------------------\n");
    printf("Grinding on [1, 0, 0]\n");
    Hyperbolic(One, 0L, 0L); WriteRegisters();
    printf("\n  K: "); WriteVarious(X0H= ScaledReciprocal(X, fractionBits));
    printf("  R: "); X0R= X0H>>1; Linear(X0R, 0L, X0R); WriteVarious(X0R= Y);
    printf("\nGrinding on [K, 0, 0]\n");
    Hyperbolic(X0H, 0L, 0L); WriteRegisters();
    printf("\nGrinding on [K, 0, 1] -> [1.54308064, 1.17520119, 0]\n");
    Hyperbolic(X0H, 0L, One); WriteRegisters();
    printf("\nGrinding on [K, K, -1] -> [0.36787944, 0.36787944, 0]\n");
    Hyperbolic(X0H, X0H, -One); WriteRegisters();
    OneOverE = X; /* save value ln(1/e) = -1 */
    printf("\nGrinding on [K, K, 1] -> [2.71828183, 2.71828183, 0]\n");
    Hyperbolic(X0H, X0H, One); WriteRegisters();
    E = X; /* save value ln(e) = 1 */
  printf("\n------Inverse functions------\n");
    printf("Grinding on [1, 0, 0]\n");
    InvertHyperbolic(One, 0L, 0L); WriteRegisters();
    printf("\nGrinding on [1/e + 1, 1/e - 1, 0] -> [1.00460806, 0,
                                                             -0.50000000]\n");
    InvertHyperbolic(OneOverE+One,OneOverE-One, 0L); WriteRegisters();
    printf("\nGrinding on [e + 1, e - 1, 0] -> [2.73080784, 0, 0.50000000]\n");
    InvertHyperbolic(E+One, E-One, 0L); WriteRegisters();
    printf("\nGrinding on (1/2)*ln(3) -> [0.71720703, 0, 0.54930614]\n");
    InvertHyperbolic(One, One/2L, 0L); WriteRegisters();
    printf("\nGrinding on [3/2, -1/2, 0] -> [1.17119417, 0, -0.34657359]\n");    InvertHyperbolic(One+(One/2L), -(One/2L), 0L); WriteRegisters();
    printf("\nGrinding on sqrt(1/2) -> [0.70710678, 0, 0.15802389]\n");
    InvertHyperbolic(One/2L+X0R, One/2L-X0R, 0L); WriteRegisters();
    printf("\nGrinding on sqrt(1) -> [1.00000000, 0, 0.50449748]\n");
    InvertHyperbolic(One+X0R, One-X0R, 0L); WriteRegisters();
    HalfLnX0R = Z;
    printf("\nGrinding on sqrt(2) -> [1.41421356, 0, 0.85117107]\n");
    InvertHyperbolic(One*2L+X0R, One*2L-X0R, 0L); WriteRegisters();
    printf("\nGrinding on sqrt(1/2), ln(1/2)/2 -> [0.70710678, 0,
                                                              -0.34657359]\n");
    InvertHyperbolic(One/2L+X0R, One/2L-X0R, -HalfLnX0R); WriteRegisters();
    printf("\nGrinding on sqrt(3)/2, ln(3/4)/2 -> [0.86602540, 0,
                                                              -0.14384104]\n");
    InvertHyperbolic((3L*One/4L)+X0R, (3L*One/4L)-X0R, -HalfLnX0R);
    WriteRegisters();
    printf("\nGrinding on sqrt(2), ln(2)/2 -> [1.41421356, 0, 0.34657359]\n");
    InvertHyperbolic(One*2L+X0R, One*2L-X0R, -HalfLnX0R);
    WriteRegisters();
    exit(0);
}
[LISTING TWO]

atanh (2^-n)

1   0.54930614   0x1193ea7a 002144765172
2   0.25541281   0x082c577d 001013053575
3   0.12565721   0x04056247 000401261107
4   0.06258157   0x0200ab11 000200125421
5   0.03126017   0x01001558 000100012530
6   0.01562627   0x008002aa 000040001252
7   0.00781265   0x00400055 000020000125
8   0.00390626   0x0020000a 000010000012
9   0.00195312   0x00100001 000004000001
10  0.00097656   0x00080000 000002000000

radius of convergence 1.11817300

atan (26-n)

0   0.78539816   0x1921fb54 003110375524
1   0.46364760   0x0ed63382 001665431602
2   0.24497866   0x07d6dd7e 000765556576
3   0.12435499   0x03fab753 000376533523
4   0.06241880   0x01ff55bb 000177652673
5   0.03123983   0x00ffeaad 000077765255
6   0.01562372   0x007ffd55 000037776525
7   0.00781233   0x003fffaa 000017777652
8   0.00390622   0x001ffff5 000007777765
9   0.00195312   0x000ffffe 000003777776
10  0.00097656   0x0007ffff 000001777777

radius of convergence 1.74328660

Back to Table of Contents


SEP91: ONE-WAY HASH FUNCTIONS

ONE-WAY HASH FUNCTIONS

Using cryptographic algorithms for hashing

Bruce Schneier

Bruce has an MS in Computer Science and has worked in computer and data security for a number of public and private concerns. He can be reached at 730 Fair Oaks Ave., Oak Park, IL 60302.

Pattern matching is usually a problem more suited for elementary programming classes than Dr. Dobb's Journal. But what if the strings are 100 Kbytes large? What if you have to search through a million of them looking for matches? And what if you have to try and match 1000 of these strings each day? With a simple string comparison algorithm, you will have to compare one billion strings, each requiring from 1 to 100,000 individual byte comparisons. And don't forget the hellacious storage and retrieval requirements. There has to be a better way.

Probabilistic algorithms are useful for jobs like this. They work -- most of the time -- and they have been applied to finding large random prime numbers, solutions to particularly convoluted graph theory problems, and pattern matching. The algorithm I am going to present only fails in one out of 2{128} (that's about 3 x 10{38}) tries. There isn't a piece of commercial software on the market that fails less often. You're much more likely to get a disk error reading the program, or a power surge that fries your whole system, than you are to have this algorithm fail. By the time this algorithm fails, our species will most likely have long killed itself off, or at least mutated into some different species that has better things to do with its time than sit around comparing a million 100-Kbyte strings.

We're going to take the computer analog of fingerprints; something small that uniquely identifies something much larger. Hash functions, which are functions that take a large input string and produce a smaller output string, perform the task nicely.

Hash Functions

One of the simplest hash functions is to take the XOR of every 128-bit block in the file, as in Example 1. This works well with random data; each 128-bit hash value is equally likely. However, with more rigidly formatted data, such as ASCII, the results are less than ideal. In most normal text files, the high-order bit of each byte is always 0. Therefore, 8 bits in the hash will always be 0. This means that the effective number of possible hashes is far less than 2{128} (or at least that each possible hash value is not equally likely), and different files are more likely to produce identical hashes.

Example 1: A simple hash function that uses the XOR of every 128-bit block in the file

  main (int argc, char *argv[]) {
      unsigned long hash[4] = {0, 0, 0, 0}, data [4];
      FILE *fp;
      int i;
      if ((fp == fopen (argv [1], "rb")) ! = NULL) {
         while ((fread (data, 4, 4, fp) ! = NULL)
            for (i=0; i<4; i++)

                hash[i] ^= data[i];
         fclose (fp);
         for (i=0; i<4; i++)

              printf ("%081x", hash [i]);
         printf ("\n");
      }
  }

An easy way to get around this is to roll the hash value 1 bit after each XOR. This way each bit is more likely to be throughly randomized by the time the entire file has been hashed, regardless of how the data looks beforehand. The program in Example 2 rolls each longword 1 bit to the right.

Example 2: Rolling each longword 1 bit to the right

      main (int argc, char *argv[]) {
         unsigned long hash [4] = {0, 0, 0, 0}, data [4];

         FILE *fp;
         int i;
         if ((fp == fopen (argv[1], "rb")) != NULL) {
          while ((fread (data, 4, 4, fp) != NULL)
              for (i=O; i<4; i++) {
              hash[i] ^= data[i];
              hash[i] = hash[i]>>1 ^ hash[i]<<31;
               }
          fclose (fp);
          for (i=0; i<4; i++)
               printf ("%081x", hash[i]);
          printf ("\n");

If the hash function is only going to deal with ASCII data, a further improvement might be to ignore spaces. This way, "HELLO WORLD" hashes to the same value as "HELLOWORLD." Of course, it also means that "The house holds one at seven bells" hashes to the same value as "The household son eats even bells," but nothing is perfect. Maybe the program should count the first space, but ignore any additional ones. Additional modifications that ignore punctuation and case might also be useful, or they might be detrimental. It all depends on the type of data involved.

These hash functions work great as long as there isn't a malicious user trying to gum up the works. Creating a file that hashes to a particular value is very easy with any of these functions. It's only slightly harder with more complicated functions. Usually, all that is required is to calculate the hash of any particular file, and then append a 128-bit string at the end of it to force the new file to hash to whatever value you'd like.

In certain applications this problem is very serious, but it can be solved by using a "one-way hash function." By "one-way," I mean that given an input file, it is very easy to calculate the hash value. However, given a particular hash value, it is, for all practical purposes, impossible to produce an input file that hashes to that value. The function has a trap door: It's easy to go one way, but hard to go back the other unless you know the "secret."

The MD5 Algorithm

In theory, any cryptographic algorithm can be used as a one-way hash function. For example, the Digital Encryption Standard (DES) algorithm (see the "C Programming" column, DDJ, September 1990), endorsed by the National Security Agency and used by countless government agencies and business interests, works quite well. Encrypt the entire file using the DES block-chaining mode, and the two final 64-bit encrypted blocks make up the hash. (ANSI X 9.9 suggests using only the last 32 bits of the last encrypted block, but that seems inadequate.) It is random, not reversible, and every bit of the hash is dependent on every bit of the input file. But in software, DES is slow; the algorithm is not at all optimized for this task. And even worse, if the DES key is compromised, then everyone knows the secret to the trap door and the one-way property of the hash function is lost.

An alternate algorithm, shown in Listing One (page 150), is called MD5. (Listing Two, page 151, is the header file for MD5.) Cryptographer Ron Rivest of MIT (he's the "R" in the RSA public-key algorithm) invented MD5 specifically as a one-way hash function. His personal needs for the algorithm revolve more around secure digital signatures than pattern matching, but we will come back to that later.

MD5 produces a 128-bit hash (or "Message Digest," hence the name) of an input file. The algorithm was designed for speed, simplicity, and compactness on 32-bit architectures. It is based on a simple set of primitive operations (that is, LOAD, ADD, XOR) on 32-bit operands. The algorithm also works on 8- and 16-bit microprocessors, albeit significantly slower.

The heart of the algorithm is MD5Update, which processes the input file in 512-bit chunks. Transform performs the basic MD5 algorithm. MD5Init kicks off the whole process. MD5Final is primarily concerned with the partial block at the end of the input file (only the rare input file can be divided exactly into 512-bit blocks). The final block is padded with a single 1 and a string of 0s, with the last word reserved for a binary representation of the number of bits in the original input file. After the algorithm churns away, the resultant hash is stored in mdContext.

It is straightforward to implement MD5 to solve the pattern matching problem at the beginning of this article. First, create and store a hash of each of the million 100-Kbyte input strings. Each one is only 16 bytes long, so you only need 16 Mbytes of storage space instead of 100 gigabytes. Every time you have a new test string, compare its hashes with those stored. Not only are the storage requirements four orders of magnitude smaller, you only have to make one 128-bit comparison per string pair. You'll get a false match every 10{27} years or so, so you might want to add code that explicitly checks strings that hash to the same value. I wouldn't even bother.

Putting One-Way Hash Functions to Work

One-way hash functions can be used to detect viruses, which make their living by surreptitiously modifying programs. Numerous security programs take simple hashes called Cyclic Redundancy Checks (CRCs). These programs hash clean files, and then compare the stored hash with a new hash of the file. If a virus infected the file, its hash would be different. The problem with most of these virus-detection programs is that a clever virus can make sure any virus-infected file hashes to the same value as the clean file. It would be harder to do this with CRC fingerprinters than with the simple hash programs listed earlier, but it would still be possible. MD5 makes it practically impossible. Even if the virus had complete knowledge of the MD5 algorithm, it could not modify itself so that the infected program would hash to the same value. All it could do would be to randomly append bits to the file and try to duplicate the hash by brute force. But even if we allow the virus to try a million possible hashes per second, it would take 10{25} years for it to successfully infect the file. I'm willing to live with the possibility of one successful virus infection on my system between now and when the sun goes nova.

As I mentioned, one-way hash functions have uses in digital signatures. A digital signature scheme is where one party can "sign" a document in such a way that no one except the signer could sign the document, and anyone can verify that the signer actually did sign the document. Most digital signature schemes involve public-key cryptography; they are computational nightmares and often painfully slow. Speed increases drastically if the digital signature algorithm operates on a hash of the document rather than the document itself. Because the chances of two documents having the same hash are only 1 in 2{128}, anyone can safely equate a signature of the hash with a signature of the document. If a conventional hash function were used, it would be a trivial matter to invent multiple documents that hashed to the same value, so that anyone signing a particular document would be duped into signing a multitude of documents. The protocol just could not work without one-way hash functions.

One-way hash functions can also be used by an archival system to verify the existence of documents without storing their contents. The central database could just store the hashes of files. It doesn't have to see the files at all; users submit their hashes to the database, and the database time stamps the submissions and stores them. If there is any disagreement in the future about who created a document and when, the database could resolve it by finding the hash in its files. Enhancements to this scheme, such as having the database use a digital signature protocol and sign a concatenation of the hash and a date/time code, would make this protocol even more useful. This has vast implications with regard to privacy: Someone could copyright a document but still keep it secret. Only if he wished to prove his copyright would he have to make it public.

The source code has been successfully compiled and run under MS-DOS, Unix, and Macintosh OS. Because of space constraints, I've included with this article only the C source that implements the MD5 algorithm; the include files, makefiles, and sample test routines are available electronically as described on page 3. RSA Data Security Inc. has put the MD5 algorithm into the public domain so that anyone can use it. Their only requirement is that any copies of their source code or documentation (but not different implementations of the mathematical algorithm) retain the copyright notice at the beginning of Listing One. Rivest cautions that while the algorithm improves over MD4, it is still new and has not been completely cryptoanalyzed. It is always possible that some clever person will find a way to break it, but over the years as more clever people try and fail, confidence in the algorithm will grow.

_ONE-WAY HASH FUNCTIONS_
by Bruce Schneier
[LISTING ONE]

/***********************************************************************
 ** md5.c -- the source code for MD5 routines                         **
 ** RSA Data Security, Inc. MD5 Message-Digest Algorithm              **
 ** Created: 2/17/90 RLR                                              **
 ** Revised: 1/91 SRD,AJ,BSK,JT Reference C Version                   **
 ** Revised (for MD5): RLR 4/27/91                                    **
 ***********************************************************************
 ** Copyright (C) 1990, RSA Data Security, Inc. All rights reserved.  **
 ** License to copy and use this software is granted provided that    **
 ** it is identified as the "RSA Data Security, Inc. MD5 Message-     **
 ** Digest Algorithm" in all material mentioning or referencing this  **
 ** software or this function.                                        **
 ** License is also granted to make and use derivative works          **
 ** provided that such works are identified as "derived from the RSA  **
 ** Data Security, Inc. MD5 Message-Digest Algorithm" in all          **
 ** material mentioning or referencing the derived work.              **
 ** RSA Data Security, Inc. makes no representations concerning       **
 ** either the merchantability of this software or the suitability    **
 ** of this software for any particular purpose.  It is provided "as  **
 ** is" without express or implied warranty of any kind.              **
 ** These notices must be retained in any copies of any part of this  **
 ** documentation and/or software.                                    **
 **********************************************************************/

#include "md5.h"

/* forward declaration */
static void Transform ();

static unsigned char PADDING[64] = {
  0x80, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00,
  0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00,
  0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00,
  0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00,
  0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00,
  0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00,
  0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00,
  0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00
};

/* F, G, H and I are basic MD5 functions */
#define F(x, y, z) (((x) & (y)) | ((~x) & (z)))
#define G(x, y, z) (((x) & (z)) | ((y) & (~z)))
#define H(x, y, z) ((x) ^ (y) ^ (z))
#define I(x, y, z) ((y) ^ ((x) | (~z)))

/* ROTATE_LEFT rotates x left n bits */
#define ROTATE_LEFT(x, n) (((x) << (n)) | ((x) >> (32-(n))))

/* FF, GG, HH, and II transformations for rounds 1, 2, 3, and 4 */
/* Rotation is separate from addition to prevent recomputation */
#define FF(a, b, c, d, x, s, ac) \
  {(a) += F ((b), (c), (d)) + (x) + (UINT4)(ac); \
   (a) = ROTATE_LEFT ((a), (s)); \
   (a) += (b); \
  }
#define GG(a, b, c, d, x, s, ac) \
  {(a) += G ((b), (c), (d)) + (x) + (UINT4)(ac); \
   (a) = ROTATE_LEFT ((a), (s)); \
   (a) += (b); \
  }
#define HH(a, b, c, d, x, s, ac) \
  {(a) += H ((b), (c), (d)) + (x) + (UINT4)(ac); \
   (a) = ROTATE_LEFT ((a), (s)); \
   (a) += (b); \
  }
#define II(a, b, c, d, x, s, ac) \
  {(a) += I ((b), (c), (d)) + (x) + (UINT4)(ac); \
   (a) = ROTATE_LEFT ((a), (s)); \
   (a) += (b); \
  }

/* The routine MD5Init initializes the message-digest context
   mdContext. All fields are set to zero.  */
void MD5Init (mdContext)
MD5_CTX *mdContext;
{
  mdContext->i[0] = mdContext->i[1] = (UINT4)0;

  /* Load magic initialization constants.
   */
  mdContext->buf[0] = (UINT4)0x67452301;
  mdContext->buf[1] = (UINT4)0xefcdab89;
  mdContext->buf[2] = (UINT4)0x98badcfe;
  mdContext->buf[3] = (UINT4)0x10325476;
}

/* The routine MD5Update updates the message-digest context to
   account for the presence of each of the characters inBuf[0..inLen-1]
   in the message whose digest is being computed.  */
void MD5Update (mdContext, inBuf, inLen)
MD5_CTX *mdContext;
unsigned char *inBuf;
unsigned int inLen;
{
  UINT4 in[16];
  int mdi;
  unsigned int i, ii;

  /* compute number of bytes mod 64 */
  mdi = (int)((mdContext->i[0] >> 3) & 0x3F);

  /* update number of bits */
  if ((mdContext->i[0] + ((UINT4)inLen << 3)) < mdContext->i[0])
    mdContext->i[1]++;
  mdContext->i[0] += ((UINT4)inLen << 3);
  mdContext->i[1] += ((UINT4)inLen >> 29);

  while (inLen--) {
    /* add new character to buffer, increment mdi */
    mdContext->in[mdi++] = *inBuf++;

    /* transform if necessary */
    if (mdi == 0x40) {
      for (i = 0, ii = 0; i < 16; i++, ii += 4)
        in[i] = (((UINT4)mdContext->in[ii+3]) << 24) |
                (((UINT4)mdContext->in[ii+2]) << 16) |
                (((UINT4)mdContext->in[ii+1]) << 8) |
                ((UINT4)mdContext->in[ii]);
      Transform (mdContext->buf, in);
      mdi = 0;
    }
  }
}

/* The routine MD5Final terminates the message-digest computation and
   ends with the desired message digest in mdContext->digest[0...15].  */
void MD5Final (mdContext)
MD5_CTX *mdContext;
{
  UINT4 in[16];
  int mdi;
  unsigned int i, ii;
  unsigned int padLen;

  /* save number of bits */
  in[14] = mdContext->i[0];
  in[15] = mdContext->i[1];

  /* compute number of bytes mod 64 */
  mdi = (int)((mdContext->i[0] >> 3) & 0x3F);

  /* pad out to 56 mod 64 */
  padLen = (mdi < 56) ? (56 - mdi) : (120 - mdi);
  MD5Update (mdContext, PADDING, padLen);

  /* append length in bits and transform */
  for (i = 0, ii = 0; i < 14; i++, ii += 4)
    in[i] = (((UINT4)mdContext->in[ii+3]) << 24) |
            (((UINT4)mdContext->in[ii+2]) << 16) |
            (((UINT4)mdContext->in[ii+1]) << 8) |
            ((UINT4)mdContext->in[ii]);
  Transform (mdContext->buf, in);

  /* store buffer in digest */
  for (i = 0, ii = 0; i < 4; i++, ii += 4) {
    mdContext->digest[ii] = (unsigned char)(mdContext->buf[i] & 0xFF);
    mdContext->digest[ii+1] =
      (unsigned char)((mdContext->buf[i] >> 8) & 0xFF);
    mdContext->digest[ii+2] =
      (unsigned char)((mdContext->buf[i] >> 16) & 0xFF);
    mdContext->digest[ii+3] =
      (unsigned char)((mdContext->buf[i] >> 24) & 0xFF);
  }
}

/* Basic MD5 step. Transforms buf based on in.  */
static void Transform (buf, in)
UINT4 *buf;
UINT4 *in;
{
  UINT4 a = buf[0], b = buf[1], c = buf[2], d = buf[3];

  /* Round 1 */
#define S11 7
#define S12 12
#define S13 17
#define S14 22
  FF ( a, b, c, d, in[ 0], S11, 0xd76aa478); /* 1 */
  FF ( d, a, b, c, in[ 1], S12, 0xe8c7b756); /* 2 */
  FF ( c, d, a, b, in[ 2], S13, 0x242070db); /* 3 */
  FF ( b, c, d, a, in[ 3], S14, 0xc1bdceee); /* 4 */
  FF ( a, b, c, d, in[ 4], S11, 0xf57c0faf); /* 5 */
  FF ( d, a, b, c, in[ 5], S12, 0x4787c62a); /* 6 */
  FF ( c, d, a, b, in[ 6], S13, 0xa8304613); /* 7 */
  FF ( b, c, d, a, in[ 7], S14, 0xfd469501); /* 8 */
  FF ( a, b, c, d, in[ 8], S11, 0x698098d8); /* 9 */
  FF ( d, a, b, c, in[ 9], S12, 0x8b44f7af); /* 10 */
  FF ( c, d, a, b, in[10], S13, 0xffff5bb1); /* 11 */
  FF ( b, c, d, a, in[11], S14, 0x895cd7be); /* 12 */
  FF ( a, b, c, d, in[12], S11, 0x6b901122); /* 13 */
  FF ( d, a, b, c, in[13], S12, 0xfd987193); /* 14 */
  FF ( c, d, a, b, in[14], S13, 0xa679438e); /* 15 */
  FF ( b, c, d, a, in[15], S14, 0x49b40821); /* 16 */

  /* Round 2 */
#define S21 5
#define S22 9
#define S23 14
#define S24 20
  GG ( a, b, c, d, in[ 1], S21, 0xf61e2562); /* 17 */
  GG ( d, a, b, c, in[ 6], S22, 0xc040b340); /* 18 */
  GG ( c, d, a, b, in[11], S23, 0x265e5a51); /* 19 */
  GG ( b, c, d, a, in[ 0], S24, 0xe9b6c7aa); /* 20 */
  GG ( a, b, c, d, in[ 5], S21, 0xd62f105d); /* 21 */
  GG ( d, a, b, c, in[10], S22,  0x2441453); /* 22 */
  GG ( c, d, a, b, in[15], S23, 0xd8a1e681); /* 23 */
  GG ( b, c, d, a, in[ 4], S24, 0xe7d3fbc8); /* 24 */
  GG ( a, b, c, d, in[ 9], S21, 0x21e1cde6); /* 25 */
  GG ( d, a, b, c, in[14], S22, 0xc33707d6); /* 26 */
  GG ( c, d, a, b, in[ 3], S23, 0xf4d50d87); /* 27 */
  GG ( b, c, d, a, in[ 8], S24, 0x455a14ed); /* 28 */
  GG ( a, b, c, d, in[13], S21, 0xa9e3e905); /* 29 */
  GG ( d, a, b, c, in[ 2], S22, 0xfcefa3f8); /* 30 */
  GG ( c, d, a, b, in[ 7], S23, 0x676f02d9); /* 31 */
  GG ( b, c, d, a, in[12], S24, 0x8d2a4c8a); /* 32 */

  /* Round 3 */
#define S31 4
#define S32 11
#define S33 16
#define S34 23
  HH ( a, b, c, d, in[ 5], S31, 0xfffa3942); /* 33 */
  HH ( d, a, b, c, in[ 8], S32, 0x8771f681); /* 34 */
  HH ( c, d, a, b, in[11], S33, 0x6d9d6122); /* 35 */
  HH ( b, c, d, a, in[14], S34, 0xfde5380c); /* 36 */
  HH ( a, b, c, d, in[ 1], S31, 0xa4beea44); /* 37 */
  HH ( d, a, b, c, in[ 4], S32, 0x4bdecfa9); /* 38 */
  HH ( c, d, a, b, in[ 7], S33, 0xf6bb4b60); /* 39 */
  HH ( b, c, d, a, in[10], S34, 0xbebfbc70); /* 40 */
  HH ( a, b, c, d, in[13], S31, 0x289b7ec6); /* 41 */
  HH ( d, a, b, c, in[ 0], S32, 0xeaa127fa); /* 42 */
  HH ( c, d, a, b, in[ 3], S33, 0xd4ef3085); /* 43 */
  HH ( b, c, d, a, in[ 6], S34,  0x4881d05); /* 44 */
  HH ( a, b, c, d, in[ 9], S31, 0xd9d4d039); /* 45 */
  HH ( d, a, b, c, in[12], S32, 0xe6db99e5); /* 46 */
  HH ( c, d, a, b, in[15], S33, 0x1fa27cf8); /* 47 */
  HH ( b, c, d, a, in[ 2], S34, 0xc4ac5665); /* 48 */

  /* Round 4 */
#define S41 6
#define S42 10
#define S43 15
#define S44 21
  II ( a, b, c, d, in[ 0], S41, 0xf4292244); /* 49 */
  II ( d, a, b, c, in[ 7], S42, 0x432aff97); /* 50 */
  II ( c, d, a, b, in[14], S43, 0xab9423a7); /* 51 */
  II ( b, c, d, a, in[ 5], S44, 0xfc93a039); /* 52 */
  II ( a, b, c, d, in[12], S41, 0x655b59c3); /* 53 */
  II ( d, a, b, c, in[ 3], S42, 0x8f0ccc92); /* 54 */
  II ( c, d, a, b, in[10], S43, 0xffeff47d); /* 55 */
  II ( b, c, d, a, in[ 1], S44, 0x85845dd1); /* 56 */
  II ( a, b, c, d, in[ 8], S41, 0x6fa87e4f); /* 57 */
  II ( d, a, b, c, in[15], S42, 0xfe2ce6e0); /* 58 */
  II ( c, d, a, b, in[ 6], S43, 0xa3014314); /* 59 */
  II ( b, c, d, a, in[13], S44, 0x4e0811a1); /* 60 */
  II ( a, b, c, d, in[ 4], S41, 0xf7537e82); /* 61 */
  II ( d, a, b, c, in[11], S42, 0xbd3af235); /* 62 */
  II ( c, d, a, b, in[ 2], S43, 0x2ad7d2bb); /* 63 */
  II ( b, c, d, a, in[ 9], S44, 0xeb86d391); /* 64 */

  buf[0] += a;
  buf[1] += b;
  buf[2] += c;
  buf[3] += d;
}

/***********************************************************************
 ** End of md5.c                                                      **
 **********************************************************************/
[LISTING TWO]

/***********************************************************************
 ** md5.h -- header file for implementation of MD5                    **
 ** RSA Data Security, Inc. MD5 Message-Digest Algorithm              **
 ** Created: 2/17/90 RLR                                              **
 ** Revised: 12/27/90 SRD,AJ,BSK,JT Reference C version               **
 ** Revised (for MD5): RLR 4/27/91                                    **
 **********************************************************************/
/* typedef a 32-bit type */
typedef unsigned long int UINT4;

/* Data structure for MD5 (Message-Digest) computation */
typedef struct {
  UINT4 i[2];                   /* number of _bits_ handled mod 2^64 */
  UINT4 buf[4];                                    /* scratch buffer */
  unsigned char in[64];                              /* input buffer */
  unsigned char digest[16];     /* actual digest after MD5Final call */
} MD5_CTX;

void MD5Init ();
void MD5Update ();
void MD5Final ();

/***********************************************************************
 ** End of md5.h                                                      **
 **********************************************************************/

Copyright © 1991, Dr. Dobb's Journal

Back to Table of Contents


Figure 1

Copyright © 1992, Dr. Dobb's Journal


Figure 1(a)

Copyright © 1992, Dr. Dobb's Journal


Figure 1(b)

Copyright © 1992, Dr. Dobb's Journal


Figure 1(c)

Copyright © 1992, Dr. Dobb's Journal


Example 1(a)

Copyright © 1992, Dr. Dobb's Journal


Example 1(b)

Copyright © 1992, Dr. Dobb's Journal


Example 1(c)

Copyright © 1992, Dr. Dobb's Journal


Example 1(d)

Copyright © 1992, Dr. Dobb's Journal


Example 1(e)

Copyright © 1992, Dr. Dobb's Journal


Example 1(f)

Copyright © 1992, Dr. Dobb's Journal


Figure 1(d)

Copyright © 1992, Dr. Dobb's Journal


Figure 1(e)

Copyright © 1992, Dr. Dobb's Journal


Figure 2(a)

Copyright © 1992, Dr. Dobb's Journal


Figure 2(b)

Copyright © 1992, Dr. Dobb's Journal


APR92: BLOCK TRUNCATION COMPRESSION

BLOCK TRUNCATION COMPRESSION

An efficient algorithm for image compression

Anton Kruger

Anton works for Truda Software, a Fortran and C consulting firm. He can be reached through the DDJ office.

Data compression algorithms can be divided into two groups. The first, error-free, or lossless compression algorithms, such as Huffman and LZW, incurs no loss of information during the compress/decompress cycle. This is an important consideration in many applications. For example, it is normally disastrous to have even 1 bit wrong in an executable file. Error-free compression is also imperative with the vast majority of text files. You don't want a compression program changing the source code of a computer program or introducing spelling mistakes into a word-processing document.

On the other hand, in some applications involving images, the error-free requirement can be dropped. When dealing with photographs of people's faces, natural scenes, and so on, some image degradation is often tolerable. This second group of compression algorithms is known as "lossy," and is well-suited for such applications. Examples of this group are the Fourier and cosine transform compression algorithms, as well as Block Truncation Compression (BTC), the focus of this article.

Block Truncation Compression

The idea behind BTC is to preserve local image statistics, typically over 4x4 blocks of the image, though larger blocks may also be used. One advantage of the BTC algorithm is that it does not require global image statistics (such as the image histogram). The implementation described in this article operates on 4x4 blocks, so that only four lines of the image are in memory at any given time. The algorithm is fairly simple and quite fast. For example, it takes about 7.25 seconds to compress a 256x256 8-bit (256-level) monochrome image, and about 2 seconds to decompress the resulting file on a 20-MHz AT clone. The most important property of Block Truncation Compression, however, is the compression ratio. The algorithm described in this article compresses monochrome image files down to 2 bits/pixel, so that the compressed images require 25 percent of their former disk space.

How does this compare with lossless compression algorithms? Two image files, "baboon" and "Valerie" were used as test files. Each contains 256x256 8-bit monochrome images in pure binary format: There are no file headers or any other information, and the images are stored line-after-line in the files. Both the popular compression program PKZIP and a DOS version of the UNIX compress utility failed to reduce the size of the two test images, and simply stored the files. Another popular file format for images is CompuServe's GIF format. A GIF-encoded version of "baboon" is about 20 percent larger, and a GIF-encoded version of "Valerie" is about 1 percent smaller than the original files. This is not too surprising, because the UNIX compress program and the GIF encoder both employ an LZW algorithm. (Sample GIF files are available eletronically; see page 3.) The files were also run through a Huffman compression program. The size of both files stayed practically the same.

In this instance we have two files that cannot easily be compressed with lossless compression algorithms, but can be compressed to 25 percent of their original size with a lossy compression algorithm. While the specific example may be extreme, it is generally true that much higher compression ratios are attainable with lossy compression algorithms than with lossless algorithms. (The price is a reduction in image quality.) Furthermore, as will hopefully become clear, this compression ratio is guaranteed with BTC, and is independent of the image properties. In fact, compression speed, decompression speed, and compression ratio are all essentially independent of the image, in contrast with error-free compression algorithms.

BTC is described in terms of Figure 1, adapted from Rosenfeld and Kak's Digital Picture Processing. Figure 1(a) shows a 4X4 block of pixels, labeled P0, P1 ,..., P15. The BTC algorithm operates by replacing the pixels in the block with only one of two judiciously chosen values, which we shall call a and b; see Figure 1(b). If a and b are made available to the decompressor, then the block of pixels can be represented with 1s and Os, as in Figure 1(c). This 16-bit bit plane can be coded into 2 bytes. Add to this 1 byte each for a and b, giving a total of 4 bytes required for the 4X4 block. Without compression, the block requires 16 bytes of storage. The savings is therefore 4/16 or 25 percent. This procedure is called "Block Truncation Compression" because the digital representation of the pixels in a block is truncated to 1 bit. The block truncation code (code) as well as a and b are needed to recover the block of pixels.

What values a and b are used to represent the pixels in the block? It was stated earlier that the idea behind the BTC algorithm is to preserve local image statistics. In particular, the first and second moments of a block of pixels are to be preserved. The first moment is simply the mean of the pixels in the block, as in Example 1(a), where n (=16) is the number of pixels in the block. The second moment is the mean of the squares of the pixel values; see Example 1(b). Let q be the number of pixels in the block greater or equal to the mean. When only a and b are used to represent the pixels in the block, the mean becomes that shown in Example 1(c), and the second moment becomes that shown in Example 1(d).

Because we want to preserve the mean and the second moment, we require these values to be the same as the original values. Thus, the equation in Example 1(a) is equated to that in 1(c) and the equation in 1(b) is equated to 1(d). When the two sets of equations are solved for a and b, we find the values in Example 1(e), where d and [image] are given by Example 1(f).

The BTC algorithm now works as follows. First, divide the image into blocks of 4X4 pixels. For each block, compute the mean and second moments from the equations in Example 1(a) and Example 1(b). Second, construct a bit plane by replacing each pixel that has a value less than the mean with a 0, and pixels with values greater or equal to the mean with a 1. Third, pack the resulting bit plane into 2 bytes, which becomes the block truncation code. Finally, write a, b, and the 2-byte code to the output file.

Decompression is straightforward. First, read a, b, and a 2-byte block truncation code from the compressed file. Second, examine each bit in the 2-byte code. If a bit is clear, then display (or write to an output file) the value a. If a bit is set, display the value b. Repeat this until no more codes are in the file. Of course, the decompressed file differs from the original, which is why BTC is not an error-free compression algorithm. However, some local image statistics (the first and second moments) over 4X4 blocks of pixels are intact.

The following is a numerical example, taken from the references; refer to Figure 1(d). With the pixel values shown, it follows that n=16, m1=98.75, m2=18.39x103, and q=7. From this and the equations in Example 1(e) and Example 1(f), it follows that [image] =92.95, d=0.882, a=17, and b=204, where a and b were rounded to the closest integer. If all pixels smaller than the mean are set to a, and the rest are set to b, the result is Figure 1(b). The bit-plane representation is given in Figure 1(c). One way of packing the bit plane is shown in Figure 1(d), and the block truncation code for this block is therefore 256* 136+227= 35043. Thus, for this block we write a=17 (1 byte), b=204 (1 byte), and code=35043 (2 bytes) to the output file.

Two special cases must be considered. The first is when q= n= 16, which occurs when all the pixel values are identical. In this instance one cannot compute d--see Example 1(f)--because a divide-by-zero error will result. This is handled by setting a= b= m1. The second case is when the computed values of a or b fall outside the closed interval [0...255]. This is handled by clamping values outside the interval so that they fall in the desired range.

Implementation

Listing One (page 104) shows code for compressing a 256x256 8-bit image with the BTC algorithm, and Listing Two (page 104) contains code for decompressing the compressed file. In this particular implementation, the image size, number of gray levels, and block size are all fixed and hard-coded into the algorithm. This was done for two reasons. The first is simplicity and clarity, and the second is that it is quite common to compress a large number of similar and related images on a regular basis. For example, a compression algorithm can be employed to compress the output of a scanner used to enter images into a personnel database. In any event, it should not be difficult to alter the code to suit other image sizes.

Listing One starts with a driver routine that calls btc4x4, the routine that does the actual compression. This routine first calls GetBlock, which fetches the next 4X4 block of pixels from the image file, and next calls GetStats, which computes the statistics a, b, and m1 (the mean) for each block according to the equations in Example 1(f). Next, each pixel in the block is examined and the appropriate bit in variable code is set, after which a, b, and code are written to the output file with a call to PutCode. The routine GetStats is quite straightforward. Floating-point calculations are used for simplicity, and rounding to the nearest integer is performed when required. The routine GetBlock maintains an internal four-line buffer that normally contains four lines of the image. A buffer pointer, bp, keeps track of the current location in the buffer. On each call, 16 pixels are copied from the four-line buffer, and bp is updated. After 64 calls, bp reaches the end of the four-line buffer, and is filled again until the end of the file is reached.

Listing Two starts with a driver for the decompression routine, btd4x4. This routine calls GetCode, which reads an a, b, and a BTC code from the compressed file. Each bit in the code is then examined, and a or b is stored in the variable block, depending on whether the bit is clear or not. It stands to reason that the same byte ordering, as well as the numbering of the bits must be followed by both the compression and decompression routines; see Figure 1(e). When block is full, PutBlock is called to write the block of pixels to an output file via a four-line buffer, quite similar to GetBlock. In other applications, PutBlock will display the block of pixels on a video display unit, or send the pixels to a printer, and so on.

As a final note on the implementation, the programs should be quite portable, though they were tested only on a PC under DOS. Two typedefs were used to create byte (1 byte) and word (2 byte) data-type names. You may need to change these to correspond to the proper variables on other systems.

Executable versions of both listings and the sample data files are available electronically.

Results and Discussion

How well does BTC work? This depends to some extent on the image, but the results are generally quite good for 4X4 block sizes. Figure 2(a) shows "Valerie," a 256x256 8-bit monochrome image. (The display was a VGA-compatible system with 6-bit 64-level gray scale capabilities.) In the BTC-compressed Figure 2(b), some block-iness is visible, especially around the baby's nose and chin, as well as the baby's right cheek. While it may not be clear on the photograph, there is also some false contouring on the baby's left cheek. This is typical of BTC, where errors show up as ragged edges and false contours in low-contrast areas. In any event, the image is acceptable, considering that it requires four times less disk space than the original.

Actually, some of the artifacts are due to the fact that a VGA-compatible system can only display 64 gray levels, and some block-iness and rough edges are present in the original Valerie image. When an image is going to be displayed on a VGA-compatible system, some additional savings are possible. A VGA-compatible system is only capable of displaying 64 gray levels, so we can save 2 bits each on a and b. When this is done, we need 1.6 bits/pixel, which implies 20 percent of the original disk space. However, implementation is a little bit more complex, because a, b, and the block truncation code no longer make up an integral number of bytes.

In the current implementation, we save a and b, computed from Example 1(e), along with the block truncation code to the output file. An alternative is to save the mean (m1) and standard deviation ( [image]), and then compute a and b during decompression. This is a slightly slower approach.

Finally, what happens when the BTC algorithm is applied to an image that has gone through the compress/decompress cycle once before--is there further reduction in image quality? The answer is generally no, which can be confirmed by working an example. Of course, if the image is compressed with another lossy compression algorithm, more image degradation will normally result.

References

Delp, E.J. and O.R. Mitchell. "Image Compression Using Block Truncation Coding." IEEE Transactions on Communications (September 1979).

Rosenfeld, A. and A.C. Kak. Digital Picture Processing, Second Edition. San Diego, Calif.: Academic Press, 1982.

Copyright © 1992, Dr. Dobb's Journal

Back to Table of Contents


JUL92: AUDIO COMPRESSION

AUDIO COMPRESSION

Digitized sound requires its own compression algorithms

This article contains the following executables: ACOMP.ARC

John W. Ratcliff

John is president of THE Audio Solution and director of product development for Milliken Publishing. He is the author of 688 Attack Sub and can be contacted at 747 Napa Lane, St. Charles, MO 63303.

As we create more powerful applications software, making audio data part of the user interface becomes more and more desirable. Voice e-mail, spoken context-sensitive online help, training systems, educational software, and games are all prime applications of this technology.

However, the amount of disk storage space digitized sound requires is a barrier to the widespread adoption of sound. Standard compression techniques using tools such as ARC or ZIP, for instance, will achieve only about a 10 percent compression rate on audio data. Clearly, compressing digitized sound requires algorithms crafted to match the nature of audio data itself. Consequently, this article presents a compression algorithm called "ACOMP" that yields better than 6:1 compression on human voice, and between 1.5:1 and 3:1 on music, while maintaining almost full fidelity.

Digitized sound is produced by a process called "analog-to-digital" (A/D) conversion, in which sound waves are converted into a digital number that represents the volume (amplitude) of the sound. On a computer, a single digital sample is usually recorded in one byte as an 8-bit number between 0 and 255, where 0 is the quietest and 255 the loudest; a good sound sample should just fit within this range. Many repeated samples must be taken to record an appreciable amount of sound. Acceptable human voice, for example, requires between 7000 and 9000 samples per second (7-9 KHz). Note that this is the same method by which sound is stored on a compact disc. However, a compact disc stores 44,000 samples per second, each 16 bits in resolution. At this resolution, one minute of sound takes four Mbytes of storage! Even with the much lower resolution of 9-KHz, 8-bit sound, one minute of speech takes over half a megabyte. ACOMP lets you store that same minute of sound in as little as 100K.

How Audio Compression Works

The human ear doesn't recognize sound by volume. You can take an entire sound sample, round it off to the closest multiple of 16 (reduce it to 4-bit data), and having achieved 2:1 compression, still hear it. What the human ear does hear is the frequency response within that data. A compression algorithm that closely approximates the data points' original values, but which cannot track frequencies within that sound, will distort the frequency response, rendering it virtually unrecognizable to the human ear.

Almost all audio-compression algorithms incorporate two basic techniques: silence encoding and delta modulation. Silence encoding is extremely important for human speech. If you examine a waveform of human speech, you will see long, relatively flat pauses between the spoken words. Silence encoding represents these pauses in a single byte of data as a pause duration rather than storing each individual data point over and over. When performing silence encoding, the user usually specifies an acceptable threshold value, typically +/-1 or 2. As long as the data samples fluctuate by no more than this amount, the whole section is considered a pause. This helps achieve extremely high data-compression rates for human voice. The trade-off is that you lose some of the highest-frequency sounds. Some of the "s"s, for example, begin to lose clarity.

The other method of compressing audio data uses a technique called "delta modulation." While ordinary compression algorithms seek an efficient method of recording the input data stream itself, delta modulation tries to find an efficient method of recording the changes in the data stream. By modeling the changes from one data sample to the next, we accurately maintain the frequency response of the input data, while balancing exact accuracy of amplitude data. (By modeling the changes in the data, we are taking the first derivative of the waveform, which contains velocity information.) Any small errors introduced into the reconstructed data are heard as slight static. The less accurately we model the data stream, the more static we hear. However, the frequency response of the input data stream is never lost, because whenever the original sound sample goes up, we go up, and when it goes down, we go down.

How Delta Modulation Works

Delta modulation is a simple concept. In an 8-bit input data stream which yields numbers between 0 and 255, you would need numbers between -255 and +255 to store the values that represent each data-point change. This would require nine bits--one more than the original sound sample! However, you don't really need nine bits. Instead, you can use four bits to represent numbers from -8 to +8 (skipping 0). If one sample were 0 and the next 230, you might store a +7 with a multiplier of 32. This would produce a value of 224. This isn't the 230 value it should be, but the exact value is less important than the frequency response, which has been maintained. So you have stored an 8-bit value in just four bits, but this gives you only a 2:1 compression rate. It doesn't model the waveform well because each data point can be off by as much as +/-31 from its actual value. The result is a lot of static.

When working with delta modulation, you can control two variables: bit resolution and the multiplier.

Bit resolution is the number of bits used to represent the delta modulation. The fewer you use, the better compression rate you achieve. With 1-bit delta modulation, you achieve 8:1 compression but may do a poor job of modeling the data.

The multiplier is the constant multiplier that controls the size of the delta mod. Ideally, the constant multiplier should roughly match the mean value the input data stream is changing from one sample to the next, divided by the total bit resolution.

After writing a number of audio-compression routines that use different bit resolutions and multipliers, I concluded that the best audio-compression routine should use the best bit size and the best multiplier value that best matches the input data stream at any given time. (In short, rather than picking one set of rules to apply on the source data, the algorithm should adapt its modeling technique to match the input data stream itself). When no bit size or multiplier will model the data as accurately as you wish, then you store that data out in raw format, causing a resynchronization to occur, and assuring that the sound quality will not be compromised. When using ACOMP, the user specifies the mean acceptable error. ACOMP will then try every multiplier and bit size possible at every point in the input data stream, ensuring that the mean error never exceed that value. The result yields high compression rates with almost no static. If your application can live with a little more static, you can pump the mean error value up to 6 or 10 and get vastly higher compression rates. (The mean error value indicates that on the average, no reconstructed data sample is off by more than +/- that amount from the original data sample.)

To accomplish this task, ACOMP needs to represent the following:

  • Silence Encoding, the duration of a pause in the original sample.
  • Resynchronization, a raw data sample to avoid introducing error.
  • Bit Size, the number of bits used to represent the delta mod (1, 2, or 4).
  • Multiplier, the size of the delta mod from one to the next, valid 1-16.
Additionally, the ACOMP header needs to indicate the frequency, of the sound sample and the frame size used for delta modulation. Figure 1 details the ACOMP data format, while Figure 2 describes ACOMP in pseudocode. The program AC.ASM (Listing One, page 96) is the source to the compression routine itself. The program UC.ASM (Listing Two, page 99) contains all the code necessary to decompress an ACOMP-compressed audio sample. This decompression algorithm involves a lot of bit-twiddling, something almost all assembly-languages excel at. (Available electronically are assembly language macro headers; a C-prototype header file for UC.ASM; a linkable, object-module version of UC.ASM; a C-prototype header for DOSCALLS.H; C-callable procedures into DOS INT 21 functions; a linkable, object-module version of DOSCALLS.ASM; executable versions of ACOMP and UCOMP; documentation of the ACOMP file format and extended ACOMP file definitions; demonstration batch files; demonstration sound files; a C version of the decompression program, and an IBM internal-speaker sound driver.)

Figure 1: ACOMP file format.

  BYTE 0-1: Length of audio sample, decompressed size, in 8086 low/high
             format
  BYTE 2-3: Recording frequency of audio sample, low/high format
  BYTE 4:   Frame size, 8-248
  BYTE 5:   Squelch size used at compression time
  BYTE 6-7: Maximum error used at compression time
  BYTE 8:   Initial audio sample
  BYTE 9:   First header byte

            Bit 7:   RESYNC.  If bit 7 is on, then the bottom 7 bits of
                     this byte represent a resynchronization value,
                     rounded to the closest multiple of 2.  Take the
                     bottom 7 bits, left shift them once, and this 8-bit
                     value represents the resynchronized data-stream value.
            Bit 6:   SQUELCH.  If bit 5 is on and bit 7 is off, then this
                     is a silence-squelching command byte.  The bottom 6
                     bits represent a repeat count of the current data
                     value.
            Bit 4-5: Delta-mod bit size.  If bit 7 and bit 6 are off,
                     then these 2 bits represent the delta-mod size used
                     to delta modulate the frame of data.  01 represents
                     1-bit modulation, 10 represents 2-bit delta
                     modulation, and 11 represents 4-bit delta
                     modulation.
            Bit 0-3: Represents delta-mod multiplier value.  Equal to the
                     value of this nibble plus 1.  Valid multipliers are
                     1-16.

            If you receive a delta-modulation header byte, record the
            delta-modulation bit size and the multiplier value.  Then grab
            bytes of delta-mod data until a full frame of data samples has
            been exhausted.  The format for delta modulation values are as
            follows:

            1-bit delta mod:     0 --> -1*multiplier
                                 1 --> +1*multiplier
            2-bit delta mod:    00 --> -2*multiplier
                                01 --> -1*multiplier
                                10 --> +1*multiplier
                                11 --> +2*multiplier
            4-bit delta mod:  0000 --> -8*multiplier
                              0001 --> -7*multiplier
                              0010 --> -6*multiplier
                              0011 --> -5*multiplier
                              0100 --> -4*multiplier
                              0101 --> -3*multiplier
                              0110 --> -2*multiplier
                              0111 --> +1*multiplier
                              1000 --> +1*multiplier
                              1001 --> +2*multiplier
                              1010 --> +3*multiplier
                              1011 --> +4*multiplier
                              1100 --> +5*multiplier
                              1101 --> +6*multiplier
                              1110 --> +7*multiplier
                              1111 --> +8*multiplier

For each data sample, the delta-mod value is added to its predecessor. You must make certain that the byte doesn't under- or overflow. Once the frame of data samples has been exhausted, you go back up to the top, looking for the next header byte. You do this until the entire audio file has been decompressed.

Figure 2: Pseudocode for the ACOMP algorithm.

  While there are data samples left to process:

  If current data can be silence encoded, then do so.
              else
              (For 1 Frame)

              Try every possible multiplier value 1-16 using 1-bit
              modulation, keeping the best.

              If the best is within the error threshold, store frame
              as 1 bit.
              else
              Try every possible multiplier value 1-16 using 2-bit
              modulation, keeping the best.

              If the best is within the error threshold, store frame as
              2 bit.
              else
              Try every possible multiplier value 1-16 using 4-bit
              modulation, keeping the best.

              If the best is within the error threshold, store frame
              as 4 bit.
              else
              Store a resynchronization value.

Note that ACOMP is far from a real-time compression algorithm. Because it does an exhaustive search for the best possible bit resolution and delta-modulation value for every frame of data, it can take a substantial amount of time to compress a sound file. However, you can decompress the resulting audio data in near real-time. A faster compression algorithm could be developed by using some frequency analysis on the input data stream rather than performing an exhaustive search.

The data definition for ACOMP-compressed audio supports compressed audio in sizes no greater than 64K in length. Because sound files can exceed this, an extended ACOMP file format has been defined; see Figure 3. This format is composed of a collection of individual ACOMP-compressed data frames. The default buffer size is 64K, but you can select a different buffer size using the command-line switch /b. If the input data extreme exceeds the current size, then an extended ACOMP audio file (comprised of n frames of audio data) is produced. One advantage to this technique is that the decompression code need only allocate memory in the size of this buffer to decompress even a very large audio file.

Figure 3: Extended ACOMP file format: Supports n frames of ACOMP-compressed data.

  // ABX file format:
  // Bytes 0-1: int TotalFrames;       Total number of ACOMP frames in
                                        file.
  // Bytes 2-5: long int TotalSize;    Total size of original source file.
  // Bytes 6-7: unsigned int bufsize;  Frame buffer size used to compress
                                         in.
  // Bytes 8-9: Unsigned int freq;     Playback frequency of audio file.
  // ....    ABH HEADERS[TotalFrames]  Array of headers indicating all
  //                   audio frame data.
  typedef struct
  {
  long int fileaddress;          // Address in file of this audio section.
  unsigned int fsize;            // compressed file size.
  unsigned int usize;            // uncompressed file size.
  } ABH;

A standardized file format for compressed digitized sound is a valuable resource. ACOMP is efficient enough at compressing human speech that it actually becomes practical to send voice data over a normal modem communication channel. Applications for this kind of audio compression extend to anywhere you would want to send voice data across a communications channel, or incorporate voice data into a computer game, online help system, or multimedia application.

_AUDIO COMPRESSION_
by John W. Ratcliff
[LISTING ONE]

;; AC.ASM -> ACOMP assembly language compressor. Written by John W. Ratcliff,
;; 1991. Uses Turbo Assembler IDEAL mode and makes HEAVE use of macros. This
;; algorithm performs an exhaustive search for the best delta mod for each
;; section of the waveform. It is very CPU intensive, and the algorithm can
;; be a little difficult to follow in assembly language.

    IDEAL           ; Enter Turbo Assembler's IDEAL mode.
    JUMPS           ; Allow automatic jump sizing.
    INCLUDE "prologue.mac"  ; Include usefull assembly lanugage macro file.

SEGMENT _TEXT   BYTE PUBLIC 'CODE'
    ASSUME  CS:_TEXT
;; This macro computes the amount of error in a frame of data, at this
;; bit resolution and this delta mod location.
Macro   DoError BITS
    LOCAL   @@NO
    mov bx,[COMP]       ; Current delta mod value.
    mov bh,BITS         ; Current bit resolution.
    push    [MINERR]        ; Pass minimum error so far.
    push    [PREV]          ; Pass previous data point.
    call    ComputeError        ; Compute error for frame.
    add sp,4            ; Balance stack.
    cmp dx,[MINERR]     ; Less than previous minimum?
    jge @@NO            ; no, don't update.
    mov [MINERR],dx     ; Save as new miniume
    mov [BESTPREV],ax       ; Best previous location.
    xor ah,ah           ;
    mov al,bl           ; Get best delta modution.
    mov [BEST1],ax      ; save it.
    mov al,bh           ; Get best bits.
    mov [BEST2],ax      ; Save it.
@@NO:
    endm

SQLCH   equ 64      ; Squelch bit.
RESYNC  equ 128     ; Resynchronization bit.

DELTAMOD equ    00110000b   ; Bit mask for delta mod bits.

ONEBIT  equ 00010000b   ; Bit pattern for one bit delta mod.
TWOBIT  equ 00100000b   ; Bit pattern for two bit delta mod.
FOURBIT equ 00110000b   ; Bit pattern for two bit delta mod.

;; This macro echos a message to the text screen, so that we can
;; monitor the progress of the compression algorithm.
Macro   Note    MSG
    push    ax
    lea ax,[MSG]
    push    ax
    call    Notify
    add sp,2
    pop ax
    endm

    public  _CompressAudio
;;This the the ACOMP compression procedure.
;;int far CompressAudio(unsigned char far *shand,
;;                                           Address of audio data to compress.
;;          unsigned char far *dhand,
;;                                      Destination address of compressed data.
;;          unsigned int slen, Length of audio data to compress.
;;          int squelch,       Squelch value allowed.
;;          int freq,      Playback frequency of audio data.
;;          int frame,     Frame size.
;;          int maxerr);       Maximum error allowed.
Proc    _CompressAudio  far
        ARG     SHAN:DWORD,DHAN:DWORD,SLEN:WORD,SQUELCH:WORD,FREQ:WORD,
                                                         FRAME:WORD,MAXERR:WORD
    LOCAL   PREV:WORD,COMP:WORD,MINERR:WORD,BEST1:WORD,BEST2:
                                                WORD,BESTPREV:WORD = LocalSpace
    PENTER  LocalSpace
    PushCREGS

    lds si,[SHAN]       ; Get source address.
    les di,[DHAN]       ; Get destination address.
    mov cx,[SLEN]       ; Get length of audio data.

    mov ax,cx       ; Get length of audio sample into AX
    stosw           ; Store it.
    mov ax,[FREQ]   ; Get frequency of recording.
    stosw           ; Store it.
    mov ax,[FRAME]  ; Get the frame size.
    stosb           ; Store it.
    mov ax,[SQUELCH]    ; Get squelch size
    stosb           ; Store it.
    mov ax,[MAXERR] ; Get maximum error allowed.
    stosw           ; Save it.
    xor ax,ax
    lodsb           ; Get first data sample
    mov [PREV],ax
    stosb           ; Store first data sample.
    dec cx      ; Decrement sample size count.
    jz  @@DONE

@@SQU:  mov ah,0Bh      ; Test keyboard status.
    int 21h
    or  al,al
    jz  @@NOK
    mov ah,08h      ; If a key was pressed get that key
    int 21h     ; value, and see if it was the
    cmp al,27       ; escape key.
    jne @@NOK
    xor ax,ax       ; If escape, return to caller, with abort.
    jmp @@EXIT
@@NOK:
    xor ax,ax
    mov dx,[SQUELCH]    ; Get squelch value.
    push    cx      ; Save remaining data count.
    push    si      ; Save si.
@@CK1:  lodsb           ; Get data byte.
    sub ax,[PREV]   ; Difference from previous data sample?
    jns @@CK2       ; if positive leave it alone.
    neg ax      ; Make it positive.
@@CK2:  cmp ax,dx       ; Is it within the squelch range?
    jg  @@NOS       ; yes, keep checking!
    loop    @@CK1       ; Keep going.
    inc si      ; Plus one, this last one counts.
@@NOS:  pop ax      ; Get back start SI
    mov dx,si       ; DX contains current address.
    sub dx,ax       ; Compute number of squelch bytes encountered.
    dec dx      ; Less, last non squelch byte.
    cmp dx,3        ; At least three?
    jle @@NOSQ      ; no, don't squelch it.
@@SQS:  cmp dx,63       ; Is it under 63?
    jle @@SEND
    mov ax,(63 + SQLCH)  ; Send count.
    sub dx,63       ; Less the 63 count we just sent.
    stosb           ; Write squelch byte out.
    jmp short @@SQS     ; Continue.
@@SEND: mov ax,dx       ; Get remaining count.
    or  ax,SQLCH    ; Or squelch bit on.
    stosb           ; Send squelch count out.
    dec si      ; Back up to last data point.
    pop ax      ; Pull CX off of stack, use current count.
    Note    msg0
    jmp short @@NXT
@@NOSQ: mov si,ax       ; Replace where source was.
    pop cx      ; Get back remaining data count.

@@NXT:  jcxz    @@DONE      ; Exit if done.
    cmp cx,[FRAME]  ; Below current frame size?
    jae @@GO        ; no, go ahead.
@@FIN:  lodsb           ; Get raw sample.
    shr al,1        ; Down to closest aproximated value.
    or  al,RESYNC   ; Add resync bit to it.
    stosb           ; Store out.
    loop    @@FIN       ; Keep sending final bytes.
    jmp @@DONE      ; exit, after sending final bytes.

@@GO:   mov [MINERR],07FFFh
    push    cx
    mov cx,[FRAME]  ; Set CX to frame size.

    mov [COMP],1
@@ALL1: DoError 1       ; Try one bit mode, +/-1.
    inc [COMP]
    cmp [COMP],17   ; Try delta comp values clean up to 16!!
    jne @@ALL1

    mov ax,[MINERR]
    cmp ax,[MAXERR]
    jle @@BCMP      ; Not good enough...
    mov [COMP],1
@@ALL2: DoError 2       ; Try two bit mode, +/-1.
    inc [COMP]
    cmp [COMP],17   ; Try delta comp values clean up to 16!!
    jne @@ALL2

    mov ax,[MINERR]
    cmp ax,[MAXERR]
    jle @@BCMP
    mov [COMP],1
@@ALL4: DoError 8       ; Try four bit mode, +/-1.
    inc [COMP]
    cmp [COMP],17   ; Try delta comp values clean up to 16!!
    jne @@ALL4

    mov ax,[MINERR] ; Get what the minimum error was.
    cmp ax,[MAXERR] ; Minimum error > maximum error?
    jle @@BCMP      ; no, then send frame.
    pop cx      ; Get back CX
    lodsb           ; Get data sample.
    and al,(NOT 1)  ; Strip off bottom bit.
    xor ah,ah
    mov [PREV],ax   ; New previous.
    shr al,1        ; /2
    or  al,RESYNC   ; Or resync bit on.
    stosb           ; Store it out into data stream.
    Note    msg1
    loop    @@SQU       ; Go check squelching.
    jmp @@DONE      ; Done, if this was last data sample.
@@BCMP: mov bx,[BEST1]  ; Get best comp.
    mov ax,[BEST2]  ; Get best bit size.
    mov bh,al       ; Into BH
    mov ax,32000
    push    ax
    push    [PREV]      ; Pass prev.
    call    ComputeError    ; Re-compute error term.
    add sp,4
    mov [PREV],ax   ; New previous.
;; Now time to store results!
    mov bx,[BEST1]  ; Get best comp.
    cmp [BEST2],1   ; 1 bit?
    jne @@NXT1
    call    Fold1Bit    ; Fold 1 bit data.
    Note    msg2
    jmp short @@IN      ; Reenter.
@@NXT1: cmp [BEST2],2   ; 2 bit data?
    jne @@NXT2
    call    Fold2Bit
    Note    msg3
    jmp short @@IN
@@NXT2:
    call    Fold4Bit
    Note    msg4
@@IN:   mov ax,[FRAME]
    pop cx      ; Get back CX
    add si,ax       ; Advance source
    sub cx,ax       ; Decrement data count.
    jnz @@SQU       ; Continue, if not at end.

@@DONE:
    mov ax,di       ; Size of compressed file.
    les di,[DHAN]
    sub ax,di       ; Difference.

@@EXIT:
    PopCREGS
    PLEAVE
    ret
    endp

;; Compute error:  Registers on entry are:
;;         DS:SI -> source data.
;;         CX    -> number of bytes to compute error term in.
;;         DX    -> total error incurred.
;;         BL    -> delta comp size.
;;         BH    -> maximum bit size value, positive or negative.
;; Exit: CX,DS:SI stay the same.
;;   DX -> total error term.
;;   AX -> new previous.
Proc    ComputeError    near
    ARG PREV:WORD,MINERR:WORD
    LOCAL   CUR:WORD = LocalSpace
    PENTER  LocalSpace

    push    cx
    push    si
    push    di      ; Save destination address.
    xor dx,dx       ; Initally no error.

@@CERR: lodsb           ; Get a data byte.
    xor ah,ah       ; Zero high byte.
    mov [CUR],ax    ; Save as current sample.
    sub ax,[PREV]
    cmp bl,1
    je  @@ND
    idiv    bl      ; Divided by delta mod size.
@@ND:   or  al,al
    js  @@DON       ; Do negative side.
    jnz @@CNT       ; If not zero then continue.
    inc al      ; Can't represent a zero, make it one.
@@CNT:  cmp al,bh       ; > max representative size?
    jle @@OK        ; no, it fit as is.
    mov al,bh       ; Make it the max representative size.
    jmp short @@OK      ;
@@DON:  neg al      ; Make it positive.
    cmp al,bh       ; > max representative size?
    jbe @@K2        ; no, use it.
    mov al,bh       ; Make it the max representative size.
@@K2:   neg al      ; Make it negative again.
@@OK:
    stosb           ; Store data value out.
    imul    bl      ; Times delta comp value.
    add ax,[PREV]   ; Add to previous data point.
    js  @@CS        ; Do signed case.
    cmp ax,255      ; Did it over flow?
    jle @@K3        ; No, then it fit byte sized.
    mov ax,255      ; Make it byte sized.
    jmp short @@K3      ; Re-enter
@@CS:   xor ax,ax       ; Close as we can get, underflow.
@@K3:   mov [PREV],ax   ; This is our new aproximated value.
    sub ax,[CUR]    ; Less actual value.
    jns @@K4        ; if positive then fine.
    neg ax      ; Take absolute value.
@@K4:   add dx,ax       ; Add into total error.
    cmp dx,[MINERR] ; Greater than minimum error allowed?
    jg  @@OUT
    loop    @@CERR
@@OUT:  mov ax,[PREV]   ; Current previous data point.
    pop di      ; Restore destination address.
    pop si      ; Reset SI back to start.
    pop cx      ; Reset CX back to start.
    PLEAVE
    ret
    endp
Macro   BuildByte
    LOCAL   @@HOP1,@@HOP2
    lodsb
    or  al,al       ; Is it signed?
    jns @@HOP1
    shl ah,1        ; Rotate.
    jmp short @@HOP2
@@HOP1: stc
    rcl ah,1
@@HOP2:
    endm
;; Fold 1 bit data.
;; ES:DI -> points to data ready to fold out.
;; CX-> frame size.
;; BL-> contains delta size.
Proc    Fold1Bit    near
    push    ds
    push    si
    push    di      ; Header byte address.
    push    es
    pop ds      ; DS=ES
    mov si,di       ; Source and dest.
    inc di      ; skip past header byte.
@@FOLD: xor ah,ah       ; Dest byte to be built, zero it.
    BuildByte
    BuildByte
    BuildByte
    BuildByte
    BuildByte
    BuildByte
    BuildByte
    BuildByte
    mov al,ah
    stosb           ; Store it out.
    sub cx,8        ; Less the 8 samples just folded up.
    jnz @@FOLD      ; Continue.

    pop si      ; Get back header byte address.
    mov al,bl       ; Get delta comp size.
    dec al      ; Less one.
    or  al,ONEBIT   ; Or the One Bit mode flag on.
    mov [ds:si],al  ; Store header byte.

    pop si
    pop ds
    ret
    endp

;; 2 Bit Format:  00 -> -2
;;        01 -> -1
;;        10 -> +1
;;        11 -> +2
Macro   BByte
    LOCAL   @@HOP1,@@HOP2
    lodsb
    or  al,al       ; Is it signed?
    jns @@HOP1
    add al,2        ; Adjust it.
    jmp short @@HOP2
@@HOP1: inc al      ; Plus 1 to fit into format size.
@@HOP2: shl ah,1
    shl ah,1
    or  ah,al       ; Place bits into byte being built.
    endm
;; Fold 2 bit data.
;; ES:DI -> points to data ready to fold out.
;; CX-> frame size.
;; BL-> contains delta size.
Proc    Fold2Bit    near
    push    ds
    push    si
@@F2:
    push    di      ; Header byte address.

    push    es
    pop ds      ; DS=ES
    mov si,di       ; Source and dest.
    inc di      ; skip past header byte.
@@FOLD: xor ah,ah       ; Dest byte to be built, zero it.
    BByte
    BByte
    BByte
    BByte
    mov al,ah
    stosb           ; Store it out.
    sub cx,4        ; Folded up 4 samples.
    jnz @@FOLD      ; Continue.

    pop si      ; Get back header byte address.
    mov al,bl       ; Get delta comp size.
    dec al      ; Less one.
    or  al,TWOBIT   ; Or the One Bit mode flag on.
    mov [ds:si],al  ; Store header byte.

    pop si
    pop ds
    ret
    endp
;; Four bit format:
;; 0 -> -8
;; 1 -> -7
;; 2 -> -6
;; 3 -> -5
;; 4 -> -4
;; 5 -> -3
;; 6 -> -2
;; 7 -> -1
;; 8 -> +1
;; 9 -> +2
;;10 -> +3
;;11 -> +4
;;12 -> +5
;;13 -> +6
;;14 -> +7
;;15 -> +8
Macro   Adjust4bit
    LOCAL   @@HOP1,@@HOP2
    lodsb
    or  al,al
    jns @@HOP1
    add al,8        ; Adjust it.
    jmp short @@HOP2
@@HOP1: add al,7        ; Adjust it.
@@HOP2:
    endm
;; Fold 4 bit data.
;; ES:DI -> points to data ready to fold out.
;; CX-> frame size.
;; BL-> contains delta size.
Proc    Fold4Bit    near
    push    ds
    push    si

    push    di      ; Header byte address.

    push    es
    pop ds      ; DS=ES
    mov si,di       ; Source and dest the same.
    inc di      ; skip past header byte.
@@FOLD: Adjust4bit      ; Get first sample.
    ShiftL  al,4        ; Into high nibble.
    mov ah,al       ; Into AH
    Adjust4bit      ; Get next nibble.
    or  al,ah       ; One whole byte.
    stosb           ; Store it out.
    sub cx,2        ; Folded up 4 samples.
    jnz @@FOLD      ; Continue.

    pop si      ; Get back header byte address.
    mov al,bl       ; Get delta comp size.
    dec al      ; Less one.
    or  al,FOURBIT  ; Or the One Bit mode flag on.
    mov [ds:si],al  ; Store header byte.

    pop si
    pop ds
    ret
    endp
msg0    db  "SQUELCH"
msg1    db  "RESYNC "
msg2    db  "1 BIT  "
msg3    db  "2 BIT  "
msg4    db  "4 BIT  "

Proc    Notify  near
    ARG MSG:WORD
    PENTER  0
    PushAll

    push    cs
    pop ds
    mov ax,0B800h
    mov es,ax
    mov si,[MSG]
    xor di,di
    mov ah,1Fh
    mov cx,7
@@SND:  lodsb
    stosw
    loop    @@SND

    PopAll
    PLEAVE
    ret
    endp

    ENDS
    END
[LISTING TWO]

;; UC.ASM      -> Uncompress ACOMP compressed audio data.
;;            Written by John W. Ratcliff, 1991.
;;            Uses Turbo Assembler IDEAL mode.

   IDEAL      ; Enter Turbo Assembler IDEAL mode.
   JUMPS      ; Allow automatic jump sizing.

   INCLUDE "prologue.mac"  ; Include common useful assembly macros.

SMALL_MODEL   equ   0   ;: true only if trying to generate near calls

   SETUPSEGMENT         ; Setup _TEXT segment.

Macro   CPROC   name
   public   _&name
IF   SMALL_MODEL
Proc   _&name   near
ELSE
Proc   _&name   far
ENDIF
   endm

SQLCH   equ   64      ; Squelch byte flag
RESYNC   equ   128      ; Resync byte flag.

DELTAMOD equ   00110000b   ; Bit mask for delta mod bits.

ONEBIT   equ   00010000b   ; Bit pattern for one bit delta mod.
TWOBIT   equ   00100000b   ; Bit pattern for two bit delta mod.
FOURBIT equ   00110000b   ; Bit pattern for two bit delta mod.

base   dw   ?      ; Base address inside translate table.

TRANS   db   -8,-7,-6,-5,-4,-3,-2,-1,1,2,3,4,5,6,7,8
   db   -16,-14,-12,-10,-8,-6,-4,-2,2,4,6,8,10,12,14,16
   db   -24,-21,-18,-15,-12,-9,-6,-3,3,6,9,12,15,18,21,24
   db   -32,-28,-24,-20,-16,-12,-8,-4,4,8,12,16,20,24,28,32
   db   -40,-35,-30,-25,-20,-15,-10,-5,5,10,15,20,25,30,35,40
   db   -48,-42,-36,-30,-24,-18,-12,-6,6,12,18,24,30,36,42,48
   db   -56,-49,-42,-35,-28,-21,-14,-7,7,14,21,28,35,42,49,56
   db   -64,-56,-48,-40,-32,-24,-16,-8,8,16,24,32,40,48,56,64
   db   -72,-63,-54,-45,-36,-27,-18,-9,9,18,27,36,45,54,63,72
   db   -80,-70,-60,-50,-40,-30,-20,-10,10,20,30,40,50,60,70,80
   db   -88,-77,-66,-55,-44,-33,-22,-11,11,22,33,44,55,66,77,88
   db   -96,-84,-72,-60,-48,-36,-24,-12,12,24,36,48,60,72,84,96
   db   -104,-91,-78,-65,-52,-39,-26,-13,13,26,39,52,65,78,91,104
   db   -112,-98,-84,-70,-56,-42,-28,-14,14,28,42,56,70,84,98,112
   db   -120,-105,-90,-75,-60,-45,-30,-15,15,30,45,60,75,90,105,120
   db   -128,-112,-96,-80,-64,-48,-32,-16,16,32,48,64,80,96,112,127

CPROC   GetFreq       ; Report playback frequency for an ACOMP file.
   ARG   SOURCE:DWORD
   PENTER   0
   push   es
   les   bx,[SOURCE]
   mov   ax,[es:bx+2]
   pop   es
   PLEAVE
   ret
   endp

;; DX contains PREVIOUS.
;; AH contains bit mask being rotated out.
;; BX up/down 1 bit value.
Macro   Delta1
   LOCAL   @@UP,@@STORE
   shl   ah,1   ; Rotate bit mask out.
   jc   @@UP
   sub   dx,bx
   jns   @@STORE
   xor   dx,dx   ; Zero it out.
   jmp short @@STORE
@@UP:   add   dx,bx
   or   dh,dh
   jz   @@STORE
   mov   dx,255
@@STORE:mov   al,dl      ; Store result.
   stosb
   endm

;; BX-> base address of translate table.
;; DX-> previous.
;; AL-> index.
Macro   DeModulate
   LOCAL   @@HIGH,@@OK
   xlat   [cs:bx] ; Translate into lookup table.
   cbw      ; Make it a signed word.
   add   dx,ax   ; Do word sized add, into previous.
   jns   @@HIGH
   xor   dx,dx   ; Underflowed.
@@HIGH: or   dh,dh   ; Did it overflow?
   jz   @@OK
   mov   dx,255   ; Maxed out.
@@OK:   mov   al,dl
   stosb
   endm

;;unsigned int     far UnCompressAudio(unsigned char far *source,unsigned char far *dest);
;; UnCompressAudio will decompress data which was compressed using ACOMP
;; into the destination address provided.  UnCompressAudio returns the
;; total size, in bytes, of the uncompressed audio data.
CPROC   UnCompressAudio
   ARG   SHAN:DWORD,DHAN:DWORD
   LOCAL   SLEN:WORD,FREQ:WORD,FRAME:WORD,BITS:WORD = LocalSpace
   PENTER   LocalSpace
   PushCREGS

   lds   si,[SHAN]      ; Get source segment
   les   di,[DHAN]      ; Get destination segment

   lodsw            ; Get length.
   mov   [SLEN],ax      ; Save length.
   mov   cx,ax         ; Into CX
   lodsw            ; Frequency.
   mov   [FREQ],ax      ; Save frequency
   lodsb            ; Get frame size.
   xor   ah,ah         ; Zero high byte
   mov   [FRAME],ax      ; Save it.
   lodsb            ; Get squelch, and skip it.
   lodsw            ; Get maximum error, and skip it.
   lodsb            ; Get initial previous data point.
   stosb            ; Store it.
   xor   ah,ah         ; zero high byte.
   mov   dx,ax      ; Save into previous word.
   dec   cx      ; Decrement total by one.
   jz   @@DONE      ; Exit
   mov   ah,al      ; AH, always the previous.
@@DCMP: lodsb         ; Get sample.
   test   al,RESYNC   ; Resync byte?
   jz   @@NOTR      ; no, skip.
   shl   al,1      ; Times two.
   mov   dl,al      ; Into previous.
   xor   dh,dh      ; Zero high word.
   stosb         ; Store it.
   loop   @@DCMP      ; Next one.
   jmp   @@DONE

@@NOTR: test   al,SQLCH   ; Squelch byte?
   jz   @@FRAM      ; no, then it is a frame.
   and   al,00111111b   ; Leave just the count.
   push   cx      ; Save current countdown counter.
   mov   cl,al      ; get repeat count
   xor   ch,ch      ; zero high byte of CX
   mov   bx,cx      ; Repeat count in DX
   mov   al,dl      ; Repeat of previous.
   rep   stosb      ; Repeat it.
   pop   cx      ; Get back remaining count.
   sub   cx,bx      ; Less.
   jnz   @@DCMP      ; Keep going.
   jmp   @@DONE

@@FRAM:
   mov   bx,ax      ; command byte into BX
   and   bx,0Fh      ; Multiplier being used.
   ShiftL   bx,4      ; Times 16.
   add   bx,offset TRANS ; Plus address of translate table.
   and   al,DELTAMOD   ; Leave just delta mod.
   push   cx
   mov   cx,[FRAME]   ; Get frame size.
   cmp   al,ONEBIT   ; In one bit delta mod?
   jne   @@NEXT1    ; no, try other.
   ShiftR   cx,3      ; /8
   mov   bl,[cs:bx+8]   ; Get up amount
   xor   bh,bh      ; Zero high byte.
@@GO:   lodsb
   xchg   al,ah      ; Place prev in AL, Bit mask in AH
   Delta1
   Delta1
   Delta1
   Delta1
   Delta1
   Delta1
   Delta1
   Delta1
   mov   ah,al
   loop   @@GO
   jmp   @@RENTER

@@NEXT1:cmp   al,TWOBIT   ; In two bit delta mod mode?
   jne   @@NEXT2
   add   bx,6      ; Point at +- 2 bit's in table.
   shr   cx,1
   shr   cx,1      ; 4 samples per byte.
@@GOGO: lodsb
   ShiftR   al,6
   DeModulate
   mov   al,[ds:si-1]
   ShiftR   al,4
   and   al,3
   DeModulate
   mov   al,[ds:si-1]
   ShiftR   al,2
   and   al,3
   DeModulate
   mov   al,[ds:si-1]
   and   al,3
   DeModulate
   loop   @@GOGO
   jmp short @@RENTER
@@NEXT2:shr   cx,1      ; Two samples per byte.
@@GO2:   lodsb         ; Get sample.
   ShiftR   al,4
   DeModulate
   mov   al,[ds:si-1]
   and   al,0Fh
   DeModulate
   loop   @@GO2

@@RENTER:
   pop   cx
   sub   cx,[FRAME]
   jnz   @@DCMP      ; Continue decompress

@@DONE:
   mov   ax,[SLEN]   ; Uncompressed length.

   PopCREGS
   PLEAVE
   ret
   endp

   ENDS
   END

Copyright © 1992, Dr. Dobb's Journal

Back to Table of Contents


Figure 1

Copyright © 1993, Dr. Dobb's Journal


Figure 2

Copyright © 1993, Dr. Dobb's Journal


Figure 3

Copyright © 1993, Dr. Dobb's Journal


APR93: ALGORITHMS FOR STEREOSCOPIC IMAGING

ALGORITHMS FOR STEREOSCOPIC IMAGING

True 3-D is easier than it looks

This article contains the following executables: STEREO.ARC

Victor J. Duvanenko and W.E. Robbins

Victor is a member of the technical staff at Truevision in Indianapolis, Indiana. He can be contacted at victor@truevision.com.

Humans, as well as many other animals, have two eyes that enable us to detect a certain range of frequencies reflected off or transmitted from surrounding objects. In other words, we have two sensors, offset horizontally by about 2.5 inches, that allow us to see and judge distances (depth) fairly precisely.

The ability to precisely judge depth comes in handy when manipulating objects with our hands. It's also beneficial to have a bit of redundancy in the visual system. Thus, our brain sees the world through our two eyes from two slightly offset perspectives. We learn to merge the two disparate views into a single image with added depth detail via triangulation.

The mechanism for extraction of depth information from two offset inputs is illustrated in Figure 1. It's easy to see that the further away an object is, the less disparate the two views (one from each eye) become. This is easy to verify by placing a finger close to your eyes, and then quickly opening and closing left and right eyes in alternating fashion. The finger appears to move a lot. However, when you fully extend your arm, moving the finger and repeat the procedure, your finger doesn't seem to move as much. Our brain learns to interpret the amount of disparity between the two views as distance (or depth). Thus, only objects that are very far away will be perceived to have little or no disparity. This ability is commonly referred to as "stereoscopic vision" (or, more formally, "binocular disparity").

The ability to present distance or depth information in a natural fashion is absent from most current-generation computer-graphics environments, which usually present only a single view of the world, as if the viewer had only one eye. Thus, the visual sense is not fully exploited, and stereoscopic skills are not utilized. The amount of information that can be effectively communicated is reduced, although many applications would greatly benefit from an increased bandwidth.

Depth information is especially useful when dealing with three-dimensional objects. By presenting depth information, a more natural human-computer interaction occurs, with higher communication bandwidth and improved understanding. In fact, the need for object rotation and movement becomes less critical, since it's no longer necessary for understanding of depth relationships, but is only needed to reveal obscured sections. Depth is an integral part of the three-dimensional space we live in. Only when depth is presented to the viewer will the visual experience offered by computers be indistinguishable from that of the real world.

True 3-D vs. Conventional 3-D

Stereoscopic, or true 3-D image-generation methods differ from conventional 3-D techniques. The 3-D techniques most computer-graphics books describe are really less than 3-D, because much depth information is lost when the 3-D image is projected onto a 2-D screen. The only depth cues that remain are the relative order of objects from the viewer (that is, one object obscures another that is behind it), limited depth information in the perspective sizing (for example, objects and their features get smaller the further they are from the viewer) and shadows. These cues provide limited depth information, which in many cases must be augmented by motion to furnish more detail about the objects. The viewer then gathers depth information from movement of the objects and constructs a mental picture of the presented scene.

For example, if you're presented with several nonoverlapping objects of various sizes, you can't tell whether the smaller ones are further away or are simply smaller. It's also impossible to tell whether the objects are 20 inches or 20 feet away from you. This effect is especially pronounced with spherical objects (planets of the solar system), since there's little, if any, perceived perspective sizing. The only way to understand such a scene is to view it in motion from an angle. For complex scenes, motion is difficult to achieve and, unless real-time motion is used, the results aren't very useful or interesting.

True 3-D, on the other hand, provides more-detailed depth information, augmenting perspective sizing and object occlusion. When several objects are presented, you immediately, and almost instinctively, surmise the depth relationships. This additional insight is particularly useful when viewing objects not common in everyday life, or when objects exhibit uncommon, and possibly confusing, relationships. In other words, if what is being viewed is common or easily understood, monoscopic techniques may be sufficient. More complex objects call for more complex viewing techniques to minimize confusion and maximize comprehension. In any case, monoscopic 3-D techniques are less than true 3-D, and are referred to as "2 1/2-D" by stereoscopic enthusiasts.

In this article, we'll explore stereoscopic presentation, providing an overview of both the hardware and software requirements. We'll then present algorithms--and implement them in C--for generating the left-eye and right-eye views fundamental to stereoscopic viewing. This article is based on our work with the scanning-tunneling microscope (STM) data from North Carolina State University's Precision Engineering Center. The STM is capable of measuring properly prepared samples of materials at atomic levels and can sense differences in vertical height of 0.01 angstrom. A sample of material is scanned by the STM, providing the vertical-height information at every regularly spaced sample point. The result is an array of 3-tuples (x,y,z) that represents the surface of a sample at the atomic level.

The application that reads and displays an STM file was written in mid-1990 on a uVAX under VMS, displayed images on a LEX/90 24-bit/pixel stereo-capable display/accelerator, and used a menu-driven, text user interface. The complete source code that implements the system is available electronically; see "Availability," page 5.

Stereoscopic Hardware

Stereoscopic hardware is necessary for the proper delivery of the two disparate (left- and right-eye) images to the respective eyes. One approach time multiplexes the left- and right-eye views (that is, alternates between them on every frame) on a single display. The views are demultiplexed (separated) through a pair of shutter glasses worn by the user. While the left-eye image is shown on the display, the left-eye lens is transparent, and the right-eye lens is opaque. When the right-eye image is shown, the opposite occurs. This technique requires the glasses to be synchronized with the computer display. StereoGraphics (San Rafael, California) manufactures infrared-controlled LCD shutter glasses that allow several people to view the display simultaneously.

Another technique is to place a polarizing LCD panel in front of the screen. This polarizes the right-eye frame in one direction and the left-eye frame 90 degrees off. The user wears passive glasses that have the left-eye lens polarized 90 degrees from the right-eye lens. Tektronix (Beaverton, Oregon) manufactures such LCD panels.

With both techniques, the user's left eye sees only the left-eye image, whereas the right eye sees only the right-eye image. If the images are displayed fast enough, the human brain fuses the two views, and the depth information becomes naturally evident.

Both techniques use a single monitor capable of displaying frames at rates of 90 to 120 frames per second. This ensures that each eye is presented with a refresh rate high enough to avoid flicker, since each eye sees its corresponding image at half the total rate. However, by running at higher vertical rates, you lose resolution, because most display hardware has an upper limit on pixel-output bandwidth. Thus, when the vertical refresh rate doubles for stereoscopic viewing (from 60 to 120 Hz), the resolution has to be halved to keep the pixel-output bandwidth constant. Also, the apparent brightness of the image is reduced due both to passing through the glasses and to each eye seeing an image for only half of the time and seeing nothing for the other half.

Another approach uses a head-mounted display (HMD) consisting of active LCD display glasses that display the left-eye view on the left lens of the glasses and right-eye view on the right lens.

One low-cost hardware configuration for stereoscopic viewing consists of a display capable of high vertical-refresh rates (preferably 120 frames/second), a pair of active LCD shutter glasses with synchronization hardware, and a stereo-ready graphics card capable of displaying 120 frames per second, switching between two images during every vertical screen retrace, and sending a synchronization signal to the controller of the LCD glasses. Stereo-ready graphics cards are available from Truevision (Indianapolis, Indiana) and other vendors.

Stereoscopic Software

Many rendering software packages are currently available. As long as the software is capable of generating a single-perspective view of a scene, it is sufficient for generating stereoscopic views. However, a particular package may not provide enough information or have enough flexibility to generate proper stereoscopic views. Then the process becomes that of trial and error, which may lead to improper results. There's actually no need to create or use a full rendering package--wire-frame images produce very dramatic results--since the addition of depth information enhances the outcome. (If you succeed at displaying wire-frame images, try closing one eye to see what you have been missing!)

Once the two views have been rendered, both must be loaded into display memory, but only one of them should be visible at a time. The software must then be capable of toggling from one image to the other during the vertical blanking period. Most graphics cards allow for two images to be loaded at once and for fast toggling between them. The software must be capable of detecting the vertical retrace interval. This can be accomplished by either servicing a vertical retrace interrupt or by polling a line counter in the display controller. During the vertical blanking period, the software must flip to the other-eye image and synchronize the glasses. The software must know which is the left-eye image and which is the right-eye image, and allow you to see only the left-eye image when the left-eye lens is transparent. If the two images are displayed backwards, you'll most likely be incapable of fusing them into one and be very uncomfortable. (The effect is equivalent to swapping the left and right eye on your head.)

Stereoscopic Algorithms

At this point, we'll discuss the steps, algorithms, and mathematics necessary for generating stereoscopic images. Most of the methods are based on those described by Larry Hodges in his PhD thesis, "Technologies, Applications, Hardcopy and Perspective Transformations for True Three-Dimensional CRT-Based Display" (North Carolina State University, 1988). Keep in mind that the data set we're working with is a uniform 2-D 200 x 200 grid with a Z value (height) given at each grid point.

To generate perspective stereoscopic views of the scene, we'll perform these steps:

  1. Make the viewing transformation matrix.
  2. Compute shading values and normals of each triangle.
  3. Transform interocular distance from the physical coordinate system to the viewing coordinate system.
  4. Make the left- and right-eye transformation matrices.
  5. Concatenate all matrices needed for the left-eye transformation into a single matrix.
  6. Transform, homogenize, clip, and render the left-eye view.
  7. Repeat steps #5 and #6 for the right-eye view.
  8. Display alternating views during successive frames.

Viewing Transformation

To simplify perspective projection, clipping, depth sorting, and stereoscopic-image generation, you need a viewing transformation that places the viewer reference point (VRP) at the origin, viewing down the negative Z-axis. In other words, the first step is to compute the transformation matrix necessary to move the viewer down to the origin, looking down the negative Z-axis. This transformation is explained as a four-step process by Foley and van Dam in Fundamentals of Interactive Computer Graphics (Addison-Wesley, 1984, pp. 258-261, 274-276). The technique is not efficient, but it is clear and performed only once per image. (See make_composite_viewing_matrix() in Listing Two (page 76) for implementation details.) This computation need be performed only when the viewer moves.

Homogeneous matrix form (4x4) was used for simplicity and efficiency. This method is beautiful, as it allows for accumulation of a series of transformations into a single matrix. For example, it's possible to construct a single matrix that not only performs the viewing transformation, but also the scaling to the device/display coordinates. Thus, each sample point in the original image needs to be multiplied by a 4x4 matrix only once to transform it all the way to the device/display coordinates.

Some computational optimizations were performed here. For instance, instead of handling each triangle separately, the entire image was transformed. This reduced the amount of computations by six, since every sample point generates two triangles, and all of the vertices are shared between neighboring triangles. The multiplication of a vector (the sample point, 1x4) by a matrix (4x4) is computationally intensive: 16 multiplications and 12 additions per sample point (a total of 640,000 multiplications and 480,000 additions). Also, three divisions, necessary to bring a point back to 3-D from the homogeneous space, have been replaced by one reciprocal and three multiplications. This should execute faster on most floating-point platforms.

More Details.

Shading

Every triangle in the uniform STM grid was flat shaded (but other shading algorithms can easily be used). For every sample/grid point (the grid is 200X200), two triangles are created. The intensity/color of each triangle is based on the angle between its normalized normal and the normalized negative light-source direction vector (a normalized vector from the origin to the light-source position). Since both vectors have been normalized (and thus have a magnitude of unity), the angle between them is obtained by computing the dot product. The result will be between -1 and 1. For simplicity, all negative values were set to 0, implying that all triangles facing away from the light source were rendered as dark as possible. Thus, the resulting triangle, shade values were between 0 and 1. The range of possible display intensities was known to be between 0 and 255. Therefore, the final triangle-shading value was obtained by multiplying the shade value with 255.

For a 200x200 grid of sample points (or a total of 40,000 samples ), 80,000 triangle normals must be computed. Each triangle normal is obtained by computing the cross product of two adjacent edge vectors. (This is how a normal to a plane is found, and triangles are guaranteed to be planar, since it takes a minimum of three points to define a plane.) For example, for a grid point P[i,i] the normal to the first triangle is obtained by computing (P[i,i] P[i+1,i])X(P[i,i]P[i+1,i+1]); and for the second triangle by computing (P[i,i]P[i+1,i+1])X(P[i,i]P[i,i+1]). A general cross-product formula would require performing three subtractions per edge to compute the edge vectors and then six multiplications and three subtractions, for a total of nine subtractions and six multiplications. Thus, for 80,000 triangles 720,000 subtractions and 480,000 multiplications are required, which would take a noticeable amount of time even on the best of current workstations.

Several computational optimizations are possible due to the restrictive structure of the STM database. The grid of samples is uniform in both X and Y, and is X- and Y-axis aligned. Thus, the X and Y components of the vectors are known in advance, and need not be computed for each triangle. Only the Z component has to be computed. Also, for vectors between successive-row grid points (for instance, P[i,i]P[i+1,i]), the X component is 0; for vectors between successive-column grid points (for example, P[i,i]P[i,i+1]), the Y component is 0. This leads to several simplifications in the cross-product computations, as the 0 components do not contribute to the result and can be eliminated. The result is that only five subtractions and four multiplications are performed for every pair of triangles (or a total of 200,000 subtractions and 160,000 multiplications--a 3.6-times reduction in subtractions and 3.0-times reduction in multiplications). See procedure compute_normals() in Listing Three, page 76. However, these optimizations produce results that are not general (that is, specific for the STM database structure).

Several floating-point operations per triangle are also necessary to normalize the normal vectors: three multiplications, two additions, three divisions, and one square root. To compute the dot product between the normalized light-source direction vector and the normalized triangle, normal requires three multiplications and two additions. It would be an interesting exercise in vector mathematics to try and reduce the number of operations required.

Transforming Interocular Distance

Once a scene can be generated from a single viewpoint, the move to stereoscopic scene generation is not difficult--two scenes must be generated from two slightly different perspectives. The trick is to decide how far apart to place the two viewpoints--the two eyes.

On average, the distance between human eyes (interocular distance) is 2.5 inches. This implies that when you look at any object, the views that each eye sees will differ by no more than 2.5 inches. This interocular distance stays nearly constant for each of us. We get used to it and derive depth information based on it. The further away an object is, the more similar the two views are; the closer an object is, the more disparate the two views.

Most computer-graphics books describe the rendering pipeline as a sequence of steps that ends with drawing a pixel on the display device. Pixel aspect ratio, gamma, and the color/shade capabilities of the display device may be the only other parameters considered. Several additional parameters must be considered when generating stereoscopic images. These parameters move beyond the pixels of the display and into your physical world.

Two images (a "stereopair") must be generated, one for each eye. However, your interocular distance (eye-to-eye spacing) is known only in the physical-coordinate system (about 2.5 inches). This distance needs to be transformed into (brought into) the viewing-coordinate system where the object being viewed lies. The result will be the interocular distance in the viewing-coordinate system, which will correspond to 2.5 inches in the physical-coordinate system.

Foley and van Dam refer to the transformation from the viewing-coordinate system to the device/display-coordinate system as "window-to-viewport scaling." The transformation from the device-coordinate system to the physical-coordinate system depends on the pixel size of the physical display. Since the interocular distance is measured along the X-axis, only the pixel width is significant (and this is obviously monitor dependent).

To map the interocular distance from the physical-coordinate system to the viewing-coordinate system, it must first be mapped to the device-coordinate system (pixels or dots). In other words, inches are mapped to pixels (and dots per inch, or dot pitch, varies between monitors). For instance, the dot pitch of the LEX/90 monitor is 0.021 inches/dot. Thus, 2.5 inches are equivalent to 119 dots (pixels). The 119 pixels is the interocular distance in the device-coordinate system. The 119 pixels must now be mapped to the viewing-coordinate system. The horizontal scale factor is the window width divided by the viewport width. Multiplying the 119 pixels by this scale factor will transform the 2.5 inches in the physical-coordinate system to the viewing-coordinate system. This computation is shown in set_per_projection_plane() in Listing Six, page 78.

Of course, your physical distance from the display needs to be transformed into the viewing-coordinate system as well. This is done in set_per_projection_plane(). These two transformations bring you from the physical world into the viewing-coordinate system (once again, where the object is). The set_per_projection_plane() procedure suggests the distance of the projection plane in the viewing-coordinate system (which corresponds to 18 inches in the physical world).

Now that we have the interocular distance and your distance from the display transformed into the viewing-coordinate system, the left- and right-eye view transformation matrices can be formed. Larry Hodges presents a simple technique whereby you first look down the Z-axis, and your eyes are centered around the origin (along the X-axis). Thus, the left eye is at coordinate (e/2, 0, 0) and the right eye is at (-e/2, 0, 0), where e is the interocular distance in the viewing-coordinate system. Secondly, the two views are generated by using the TPT{-1} model (translate, project, translate back). By using homogeneous (4x4) matrices, a single (4x4) transformation matrix results. The left-eye view transformation consists of a concatenation of three operations: translation to the right by e/2 distance, projection, and translation to the left by the same amount (T[R]PT[L]); see Figure 2. The display_3D_data() procedure (see Listing One, page 76) follows this model by creating first a left-eye (4x4) transformation matrix, then a right-eye matrix.

Concatenating All Matrices into One

The display_3D_data() procedure then combines the left-eye matrix with the viewing-transformation (4x4) matrix to produce a single (4x4) matrix that contains the complete transformation that each vertex of the 2OOx2OO grid must undergo. Once a single (4x4) transformation matrix has been obtained, transform_and_homogenize_image() (Listing Four, page 78) is called, transforming the 2OOx2OO grid (or 40,000 vertices). The result is homogenized (X, Y, and Z are divided by W) to bring the points back into 3-D from the homogenous space.

At this point, render_image() (Listing Five, page 78) is called, which takes transformed grid with shading information for each of the 80,000 triangles, clips the triangles to the viewport, and renders it (utilizing the LEX/90 polygon command). Clipping is done very simplistically--the triangle is rendered only if it is entirely inside the viewport.

Displaying Alternating Views

The LEX/90 has two display buffers (double-buffered). It is capable of displaying either of the two buffers and of alternating between the buffers on each frame. The left-eye image was rendered into display-buffer one, and the right-eye image was rendered into display-buffer two. Then LEX/90 is set into the alternating buffer mode, and--stereo! The LEX/90 displays the left-eye image in one frame, then displays the right-eye image, then the left-eye image, and so on, 60 times per second.

To view this image, we used a crusty pair of piezoelectric shutter glasses synchronized with the vertical sync pulse of the LEX/90 display. Thus, the left eye saw only the left-eye image and the right eye saw only the right-eye image. The display flickered a little, since each eye saw a picture at 30 Hz (frames/second), but the result was sufficient to get quite a few "Wow!"s.

Rolling Your Own Stereopair Images

Barr E. Bauer

Barr is a research scientist at Arris Pharmaceutical and can be contacted at 385 Oyster Point Blvd., South San Francisco, CA 94080.

In my work as a pharmaceutical researcher, I find stereo imaging to be not only useful, but essential because true 3-D presentation is often the only way to clearly present results. In other words, when viewed in conventional 3-D or 2-D, DNA structures (such as that on the cover of this magazine) can lose much of their structural content.

Workstations like Silicon Graphics' (Mountain View, California) Indigo provide real-time 3-D effects directly on the monitor. Vector objects can be displayed depth-cued, whereby uniform fading along the screen Z-axis gives a good sense of front to back. The pseudo-3-D effect is magnified when the object is rotated or when the fading is dramatic. Interior objects can be highlighted by making them less bright than objects in front, maintaining a clear what's-in-front distinction. The depth-cued display sometimes inverts in the mind's eye, making the bright part of the display appear to be in the rear. This curious side effect occurs at random and can complicate presentations. The Indigo can display solid-shaded objects, too, which also gives a good pseudo-3-D effect.

CrystalEyes from StereoGraphics (San Rafael, California) uses a special monitor that alternates the display of left- and right-eye views to achieve real-time stereo. Liquid-crystal goggles synched to the monitor through an infrared emitter separate the displays so that each eye sees only the image appropriate for it. The stereo effect is startling: Objects appear to float in front of the screen and can be rotated in real time. On the downside, the hardware is costly, significant programming must be done to support the graphics, the picture quality is degraded due to fewer display lines, and the manipulation of 2-D menus presents a programming challenge. Despite all this, CrystalEyes can show stereo to up to four people in front of a monitor and up to 20 people with projection.

Stereo slides enable stereo presentation to larger crowds or when maximum picture quality is desired. Separate left- and right-eye views are photographed, then projected through polarized filters and viewed with polarized glasses, one eye at 90 degrees with respect to the other. The images are rotated six degrees with respect to each other along the screen's Y-axis. Internal coordinate frames must be normalized to the screen frame before the two images are rotated and photographed, otherwise the resulting projected stereo images can be very distressing to look at. The result is a static image that looks good but requires careful image alignment, and it often takes a moment for the audience to "see" the stereo effect. The polarizing filters steal light intensity the farther you sit from the central axis of the screen, making the image appear dim to those sitting in the front corners of the room. The screen can fool camera exposure control often enough that multiple exposures and careful intensity matching give the best stereo image. If one of the pair is noticeably different in exposure, the stereo effect becomes more difficult to see. Regardless, this method is inexpensive and portable. I combine depth-cuing with stereo slides for a double effect that accentuates the stereo.

Finally, stereo images can be put in print as stereo pairs, even with line art. The simplest picture that minimizes unnecessary overlap provides the clearest 3-D image. The pairs can be viewed with glasses, or by defocusing your eyes and merging the two images into one. Figure 3, for instance, is a stereo triplet from AI software used to recognize 3-D molecular shape developed at Arris Pharmaceutical Corp. To see in stereo, follow these steps:

  • Hold the page flat under uniform light at arm's length.
  • Look at the center and right images, cross your eyes, and attempt to merge the new images formed into a single 3-D image. When the structures merge, the image will snap into 3-D.
  • If you can't see the stereo, relax, have a glass of wine, and try again.
Most people cross their eyes; if you are "wall-eyed" (eyes turned outward), look instead at the left and center images. Beware, however, that this can give you a severe headache if you look long enough.

_ALGORITHMS FOR STEREOSCOPIC IMAGES_
by Victor Duvanenko and W.E. Robbins
[LISTING ONE]

display_3D_data( proj )
int proj;
{
   double  Tw[4][4], S[4][4], Td[4][4], Ry[4][4], Per[4][4], tmp[4][4];
   double  Left[4][4], Right[4][4];
   double  e_v,      /* interocular distance mapped back to the view  coord. */
           e_w;      /* interocular distance mapped back to the world coord. */
   printf( "Computing normals for shading.  " );
   compute_normals();      /* and the dot products for each triangle */
   printf( "Done.\n" );
   /* Perspective projection must use three steps:
      1)  Compute normals and project,
      2)  Divide by W (homogenize)
      3)  Transform into the device coordinates. */
   if ( proj == PERSPECTIVE )
   {
      /* map physical interocular distance into the view port coordinates. */
      e_v = INTEROCULAR_DISTANCE / PIXEL_WIDTH;      /* e_v == pixels */
      /* map from the viewport coordinate system to the world. */
      e_w = e_v / (( x_right_d - x_left_d ) / ( x_right - x_left ));
      set_to_identity( Left,  4 );
      set_to_identity( Right, 4 );

      /* Use the Translate, Project, Translate back model. */

      /* Create the Left eye transformation matrix. */
      set_to_identity( Tw, 4 );    /* translate the left eye to the origin */
      Tw[3][0] = -e_w / 2.0;
      matrix_mult( Left, 4, 4, Tw, 4, 4, tmp );
      /* Create the perspective projection matrix. */
      set_to_identity( Per, 4 );
      Per[2][3] = 1.0 / proj_plane;          /* 1/d */
      Per[3][3] = 0.0;
      matrix_mult( tmp, 4, 4, Per, 4, 4, Left );

      Tw[3][0] = e_w / 2.0;             /* translate back */
      matrix_mult( Left, 4, 4, Tw, 4, 4, tmp );
      copy_matrix( tmp, Left, 4, 4 );

      /* Create the Right eye transformation matrix. */
      set_to_identity( Tw, 4 );    /* translate the right eye to the origin */
      Tw[3][0] = e_w / 2.0;
      matrix_mult( Right, 4, 4, Tw, 4, 4, tmp );
      /* Create the perspective projection matrix. */
      set_to_identity( Per, 4 );
      Per[2][3] = 1.0 / proj_plane;          /* 1/d */
      Per[3][3] = 0.0;
      matrix_mult( tmp, 4, 4, Per, 4, 4, Right );

      Tw[3][0] = -e_w / 2.0;             /* translate back */
      matrix_mult( Right, 4, 4, Tw, 4, 4, tmp );
      copy_matrix( tmp, Right, 4, 4 );
#if 0
      printf( "Transforming, projecting and homogenizing the image.  " );
      transform_and_homogenize_image( image, tr_image, Tm );
      printf( "Done.\n" );
#endif
   }
   /* Create the world to device transformation matrix. */
   /* Create the translation matrix. Translate only in X and Y. */
   set_to_identity( Tw, 4 );
   Tw[3][0] = -x_left;   Tw[3][1] = -y_bottom;   Tw[3][2] = 0.0;
   /* Create a uniform scale matrix. */
   set_to_identity( S, 4 );
   S[0][0] = ( x_right_d - x_left_d   ) / ( x_right - x_left   );
   S[1][1] = ( y_top_d   - y_bottom_d ) / ( y_top   - y_bottom );
   S[2][2] = ( z_back_d  - z_front_d  ) / ( z_back  - z_front  );

   matrix_mult( Tw, 4, 4, S, 4, 4, tmp );
   copy_matrix( tmp, Tw, 4, 4 );

   /* Create the translation matrix. Translate only in X and Y. */
   set_to_identity( Td, 4 );
   Td[3][0] = x_left_d;   Td[3][1] = y_bottom_d;   Td[3][2] = 0.0;
   matrix_mult( Tw, 4, 4, Td, 4, 4, tmp );
   copy_matrix( tmp, Tw, 4, 4 );

   /* Since device/screen origin on LEX/90 is in upper left, we need to reflect
      Y and translate by screen height to place device origin in bottom left.*/
   set_to_identity( Ry, 4 );
   Ry[1][1] = -1.0;
   matrix_mult( Tw, 4, 4, Ry, 4, 4, tmp );
   copy_matrix( tmp, Tw, 4, 4 );

   set_to_identity( Td, 4 );
   Td[3][1] = SCREEN_HEIGHT;
   matrix_mult( Tw, 4, 4, Td, 4, 4, tmp );
   copy_matrix( tmp, Tw, 4, 4 );
   /* Now, Tw has the world to device/screen transformation matrix. */
   if ( proj == PARALLEL )
   {
      /* Beautiful!!!  Perform a single transformation of the image (for
         parallel projection). */
      printf( "Transforming, projecting and mapping the image onto screen." );
      matrix_mult( Tm, 4, 4, Tw, 4, 4, tmp );
      copy_matrix( tmp, Tm, 4, 4 );
      show_matrix( Tm, 4, 4 );
      transform_and_homogenize_image( image, tr_image, Tm );
      printf( "Done.\n" );
      printf( "Rendering the image.  " );
      render_image( LEFT_EYE );        printf( "Done.\n" );
   }
   if ( proj == PERSPECTIVE )
   {
      printf( "Transforming, projecting, homogenizing and mapping the " );
      printf( "Left image onto screen.  " );
      matrix_mult( Tm, 4, 4, Left, 4, 4, tmp );
      matrix_mult( tmp, 4, 4, Tw, 4, 4, Left );
      printf( "Left eye transformation matrix.\n" );
      show_matrix( Left, 4, 4 );
      transform_and_homogenize_image( image, tr_image, Left );
      printf( "Done.\n" );
      printf( "Rendering the Left eye image.  " );
      render_image( LEFT_EYE );        printf( "Done.\n" );

      printf( "Transforming, projecting, homogenizing and mapping the " );
      printf( "Right image onto screen.  " );
      matrix_mult( Tm, 4, 4, Right, 4, 4, tmp );
      matrix_mult( tmp, 4, 4, Tw, 4, 4, Right );
      /* Move the right eye view into the lower half of the buffer. */
      set_to_identity( Tw, 4 );
      Tw[3][1] = SCREEN_HEIGHT;                 /* move in device coord */
      matrix_mult( Right, 4, 4, Tw, 4, 4, tmp );
      copy_matrix( tmp, Right, 4, 4 );
      printf( "Right eye transformation matrix.\n" );
      show_matrix( Right, 4, 4 );
      transform_and_homogenize_image( image, tr_image, Right );
      printf( "Done.\n" );
      dszom( &0, &512, &1 );                   /* look at the lower half */
      printf( "Rendering the Right eye image.  " );
      render_image( RIGHT_EYE );        printf( "Done.\n" );
   }
}
[LISTING TWO]

make_composite_viewing_matrix( proj )
int proj;      /* PARALLEL or PERSPECTIVE */
{
   double p1[4], p2[4], p3[4], p_tmp[4];
   double T[4][4], Rx[4][4], Ry[4][4], Rz[4][4], C_tmp1[4][4], C_tmp2[4][4];
   double Per[4][4], d1, d2, d12, cos_ang, sin_ang;
   /* Initialize the three points */
   p1[0] = eye_pt.x;  p1[1] = eye_pt.y;  p1[2] = eye_pt.z;  p1[3] = 1.0;
   p2[0] = p2[1] = p2[2] = 0.0;  p2[3] = 1.0;
   p3[0] = p1[0] + vup.x;  p3[1] = p1[1] + vup.y;  p3[2] = p1[2] + vup.z;
                           p3[3] = 1.0;
   /* Magnitude of vector p1->p2 */
   d12 = sqrt( p1[0] * p1[0] + p1[1] * p1[1] + p1[2] * p1[2] );
   /* Create the translation matrix. */
   set_to_identity( T, 4 );
   T[3][0] = -p1[0];   T[3][1] = -p1[1];   T[3][2] = -p1[2];
   /* Translate the three points p1, p2, and p3 to the origin. */
   matrix_mult( p1, 1, 4, T, 4, 4, p_tmp );
   copy_matrix( p_tmp, p1, 1, 4 );
   matrix_mult( p2, 1, 4, T, 4, 4, p_tmp );
   copy_matrix( p_tmp, p2, 1, 4 );
   matrix_mult( p3, 1, 4, T, 4, 4, p_tmp );
   copy_matrix( p_tmp, p3, 1, 4 );

   d1 = sqrt( p2[0] * p2[0] + p2[2] * p2[2] );      /* length of projection */
   cos_ang = -p2[2] / d1;
   sin_ang =  p2[0] / d1;
   /* Create the rotation about Y-axis matrix. */
   set_to_identity( Ry, 4 );
   Ry[0][0] = cos_ang;  Ry[0][2] = -sin_ang;
   Ry[2][0] = sin_ang;  Ry[2][2] =  cos_ang;
   /* Rotate the three points p2, and p3 about the Y-axis. */
   /* p1 is at the origin after translation => no need to rotate. */
   matrix_mult( p2, 1, 4, Ry, 4, 4, p_tmp );
   copy_matrix( p_tmp, p2, 1, 4 );
   matrix_mult( p3, 1, 4, Ry, 4, 4, p_tmp );
   copy_matrix( p_tmp, p3, 1, 4 );

   cos_ang = -p2[2] / d12;
   sin_ang = -p2[1] / d12;

   /* Create the rotation about X-axis matrix. */
   set_to_identity( Rx, 4 );
   Rx[1][1] = cos_ang;   Rx[1][2] = sin_ang;
   Rx[2][1] = -sin_ang;  Rx[2][2] = cos_ang;
   /* Rotate the three points p2, and p3 about the X-axis. */
   matrix_mult( p2, 1, 4, Rx, 4, 4, p_tmp );
   copy_matrix( p_tmp, p2, 1, 4 );
   matrix_mult( p3, 1, 4, Rx, 4, 4, p_tmp );
   copy_matrix( p_tmp, p3, 1, 4 );
   /* Sanity check. */
   printf( "The view vector should be [ %lf %lf %lf %lf ]\n", 0.0, 0.0, -d12,
                                                              1.0 );
   printf( "The view vector is [ %lf %lf %lf %lf ]\n", p2[0], p2[1],
                                                       p2[2], p2[3] );
   d2 = sqrt( p3[0] * p3[0] + p3[1] * p3[1] );
   cos_ang = p3[1] / d2;
   sin_ang = p3[0] / d2;
   /* Create the rotation about Z-axis matrix. */
   set_to_identity( Rz, 4 );
   Rz[0][0] = cos_ang;   Rz[0][1] = sin_ang;
   Rz[1][0] = -sin_ang;  Rz[1][1] = cos_ang;
   /* At this point the translation, and all rotation matrices are known
      and need to be combined into a single transformaation matrix. */
   matrix_mult( T, 4, 4, Ry, 4, 4, C_tmp1 );
   matrix_mult( C_tmp1, 4, 4, Rx, 4, 4, C_tmp2 );
   matrix_mult( C_tmp2, 4, 4, Rz, 4, 4, C_tmp1 );
   copy_matrix( C_tmp1, Tm, 4, 4 );
}
[LISTING THREE]

void compute_normals()
{
   register i, j;
   point_3D_t p11_p21, p11_p22, p11_p12, cross_a, cross_b;
   double d, dx;              /* stepping distance (dx = dy) */
   double dx_sqrd;            /* dx^2 */
   int num_lines;
   point_3D_t na, nb;         /* normals to triangle A and B */
   dx = scan_sz / num_samples;

   num_lines = MIN( num_samples, MAX_IMAGE_SIZE );
   num_lines--;
   dx_sqrd = dx * dx;
   for( i = 0; i < num_lines; i++ )
      for( j = 0; j < num_lines; j++ )
      {
#if DEBUG
         printf( "P11 %f %f %f\n", image[i][j].x, image[i][j].y,
                                   image[i][j].z );
         printf( "P21 %f %f %f\n", image[i+1][j].x, image[i+1][j].y,
                                   image[i+1][j].z );
         printf( "P12 %f %f %f\n", image[i][j+1].x, image[i][j+1].y,
                                   image[i][j+1].z );
         printf( "P22 %f %f %f\n", image[i+1][j+1].x, image[i+1][j+1].y,
                                   image[i+1][j+1].z );
#endif
         p11_p21.z = image[ i + 1 ][ j ].z - image[ i ][ j ].z;
         p11_p22.z = image[ i + 1 ][ j + 1 ].z - image[ i ][ j ].z;
         p11_p12.z = image[ i ][ j + 1 ].z - image[ i ][ j ].z;
#if DEBUG
         printf( "dz11_21 = %f,  dz11_22 = %f,  dz11_12 = %f\n",
                 p11_p21.z, p11_p22.z, p11_p12.z );
#endif
         /* It's possible to eliminate one more multiplication in the
            computations below. */
         na.x = dx * ( p11_p21.z - p11_p22.z );
         na.y = dx * p11_p21.z;
         na.z = dx_sqrd;
#if DEBUG
         printf( "Na %f %f %f\n", na.x, na.y, na.z );
#endif
         nb.x = (-dx) * p11_p12.z;
         nb.y = dx * ( p11_p22.z - p11_p12.z );
         nb.z = dx_sqrd;
#if DEBUG
         printf( "Nb %f %f %f\n", nb.x, nb.y, nb.z );
#endif
         /* Normalize the normal vectors, since the intensity will be
            proportional to the angle between light source and the normal. */
         d = sqrt((double)( na.x * na.x + na.y * na.y + na.z * na.z ));
         na.x = na.x / d;
         na.y = na.y / d;
         na.z = na.z / d;
#if DEBUG
         printf( "Na %f %f %f\n", na.x, na.y, na.z );
#endif
         d = sqrt((double)( nb.x * nb.x + nb.y * nb.y + nb.z * nb.z ));
         nb.x = nb.x / d;
         nb.y = nb.y / d;
         nb.z = nb.z / d;
#if DEBUG
         printf( "Nb %f %f %f\n", nb.x, nb.y, nb.z );
#endif
         /* Compute the dot product between the light source vector and
            the normals (== to the angle between two unit vectors ).
            -1 <= cos( theta ) <= 1, which will be very useful. */
         image[ i ][ j ].sha = light_source.x * na.x + light_source.y * na.y +
                               light_source.z * na.z;
         image[ i ][ j ].shb = light_source.x * nb.x + light_source.y * nb.y +
                               light_source.z * nb.z;
      }
}
[LISTING FOUR]

transform_and_homogenize_image( s, d, tm )
point_3D_ex_t  s[][ MAX_IMAGE_SIZE ], d[][ MAX_IMAGE_SIZE ];
double *tm;         /* transformation matrix */
{
   register i, j;
   int  num_lines;
   double  p[4];      /* the point to be transformed */
   double  t[4], inv_W;

   num_lines = MIN( num_samples, MAX_IMAGE_SIZE );
   for( i = 0; i < num_lines; i++ )
      for( j = 0; j < num_lines; j++ )
      {
         p[0] = s[i][j].x;   p[1] = s[i][j].y;
         p[2] = s[i][j].z;   p[3] = 1.0;
         matrix_mult( p, 1, 4, tm, 4, 4, t );
         if ( t[3] != 1.0 )        /* divide by W (homogenize) */
         {
            inv_W = 1.0 / t[3];
            t[0] *= inv_W;   t[1] *= inv_W;  t[2] *= inv_W;
         }
         d[i][j].x = t[0];   d[i][j].y = t[1];   d[i][j].z = t[2];
         d[i][j].sha = s[i][j].sha;    d[i][j].shb = s[i][j].shb;
      }
}
[LISTING FIVE]

render_image( y )
int  y;
{
   register i, j;
   int    num_lines;
   short  v[6], intensity;

   num_lines = MIN( num_samples, MAX_IMAGE_SIZE );
   num_lines--;
   for( i = 0; i < num_lines; i++ )
      for( j = 0; j < num_lines; j++ )
      {
         v[0] = ROUND( tr_image[ i     ][ j     ].x );
         v[1] = ROUND( tr_image[ i     ][ j     ].y );
         v[2] = ROUND( tr_image[ i + 1 ][ j     ].x );
         v[3] = ROUND( tr_image[ i + 1 ][ j     ].y );
         v[4] = ROUND( tr_image[ i + 1 ][ j + 1 ].x );
         v[5] = ROUND( tr_image[ i + 1 ][ j + 1 ].y );
         /* Render triangle A */
         intensity = ROUND( tr_image[ i ][ j ].sha * (double)(NUM_SHADES - 1));
         if ( intensity > ( NUM_SHADES - 1 ))
            intensity = NUM_SHADES - 1;      /* saturate */
         if ( intensity < 0 )
         {
#if 0
            printf( "Triangle A,  intensity = %d\n", intensity );
            printf( "v11.x = %f, v11.y = %f, v11.z = %f\n",
                    image[i][j].x, image[i][j].y, image[i][j].z );
            printf( "v21.x = %f, v21.y = %f, v21.z = %f\n",
                    image[i+1][j].x, image[i+1][j].y, image[i+1][j].z );
            printf( "v22.x = %f, v22.y = %f, v22.z = %f\n",
                    image[i+1][j+1].x, image[i+1][j+1].y, image[i+1][j+1].z );
#endif
            intensity = 0;
         }
         if ( clip_to_viewport( v, 6, y ) == ACCEPT )
            dspoly( &intensity, v, &6 );

         v[2] = ROUND( tr_image[ i     ][ j + 1 ].x );
         v[3] = ROUND( tr_image[ i     ][ j + 1 ].y );
         /* Render triangle B */
         intensity = ROUND( tr_image[ i ][ j ].shb * (double)( NUM_SHADES-1));
         if ( intensity > ( NUM_SHADES - 1 ))
            intensity = NUM_SHADES - 1;      /* saturate */
         if ( intensity < 0 )   intensity = 0;
         if ( clip_to_viewport( v, 6, y ) == ACCEPT )
            dspoly( &intensity, v, &6 );
      }
}
[LISTING SIX]

display_3D_data( proj )
int proj;
{
   double  Tw[4][4], S[4][4], Td[4][4], Ry[4][4], Per[4][4], tmp[4][4];
   double  Left[4][4], Right[4][4];
   double  e_v,      /* interocular distance mapped back to the view  coord. */
           e_w;      /* interocular distance mapped back to the world coord. */

   printf( "Computing normals for shading.  " );
   compute_normals();      /* and the dot products for each triangle */
   printf( "Done.\n" );

Copyright © 1993, Dr. Dobb's Journal

Back to Table of Contents


Example 1

Copyright © 1993, Dr. Dobb's Journal


APR93: GENETIC ALGORITHMS AND DATABASE INDEXING

GENETIC ALGORITHMS AND DATABASE INDEXING

Finding the best set of indexes

Joe Celko

Joe is a columnist for DBMS magazine and a member of the ANSI X3H2 Database Standards Committee. He can be reached through the DDJ offices.

Imagine you've just declared all the tables for a SQL database, and now have to declare primary and secondary indexes. Most SQL implementations automatically create primary indexes (which assure uniqueness of primary keys) from the table declarations. However, secondary indexes which optimize update, delete, insert, and query search time in the database must be explicitly created and can also be unique, although they're most often not.

The easy way out is to not declare secondary indexes because small test databases run fine without them. But when the test database grows into a large production system, you're hit with a big performance issue. At this point, the solution might seem to swing to the other extreme and index everything.

But while query performance will almost certainly improve since the query optimizer can find an index whenever it needs it, performance of inserts, updates, and deletes will suffer. This is because every time the database changes, so do the indexes. At times, hashing algorithms will even have to take care of collisions unless the algorithm produces a relatively even distribution. This means that a change to the database will cost one table access plus one or more accesses per index, which can slow performance in databases with millions of rows.

For instance, given a table of (n) columns, there are actually more than (n) possible indexes on it. An index can be built on more than one column, and the order of the columns within the index is important. The formula for the number of possible indexes is the sum of all permutations of (n) columns; see Example 1.

Now consider a table with three columns, T(a,b,c). The 16 possible indexes on table T are (a,b,c), (a,b), (a,c,b), (a,c), (a), (b,a,c), (b,a), (b,c,a), (b,c), (b), (c,a,b), (c,a), (c,b,a), (c,b), (c), or none at all. The optimal solution is to declare only the indexes you absolutely need to balance query time against change times. This means that, as always, you also need to know something about the transactions that will run against the database. If the job is all queries with no changes on only one column in a table, that column needs to be indexed. Any other indexing would not be used by the queries or updates and only take up space. This is commonly called a "lookup table." However, since the indexed column is probably the primary key of the lookup table, it's indexed automatically. (Note that many databases have a column that is used extensively for access and that isn't the primary key.) Likewise, if no queries or changes use a particular column, then it doesn't need to be indexed. The secondary index is often used to establish a one-to-many relationship or many-to-many relationship with another table.

Real database problems lie somewhere between these two extremes.

Finding the Best Set of Indexes

To make life easier, let's consider only queries. Given a database schema and a set of queries, we want to find the best possible set of indexes no bigger than some limit, say (i).

Finding the optimal indexing arrangement is known to be NP-complete. This means that the work to find a solution grows much faster as the number of items in the problem increases. As a rule of thumb, if you have to inspect all possible combinations of something, the problem will likely be difficult to crack.

This doesn't mean you can't optimize indexing for a particular database schema and set of input queries, but it does mean you can't write a program which will do it for all possible relational databases and query sets.

The query information is usually given as a statistical model of the expected inputs. For example, you might be told that 80 percent of the queries will use the primary key and 20 percent will use another column picked at random. This is about what you'd know in real-world situations, since most of the accesses will be done by production programs with embedded SQL, with only a small percentage of ad hoc queries.

However, NP-complete problems do have efficient, near-optimal algorithms. Farshad Fotouhi and Carlos E. Galarce of Wayne State University Computer Science Department, for instance, have proposed using genetic algorithms to search for near-optimal indexing.

The idea behind genetic algorithms is to mirror natural selection to evolve a better algorithm. You create a set of "chromosomes" that can be modified by a set of operators based on feedback from their environment. A group of "genes" form a chromosome, which is handled as a unit. Each generation, the survival quotient of a chromosome is measured by some adaptive plan. The starting point is usually a random mixture of chromosomes, and the experiment is run for a fixed number of generations.

Fotouhi and Galarce's approach uses a single table of library-book information (classification, ISBN, title, subject, author). The gene is a binary vector with a position for each of the five attributes. A 1 means the column is indexed; 0 means it's not. For example, the gene (01011) would mean that there are indexes on ISBN, subject, and author only. Notice that the ISBN is the primary key, but no attempt is made to accept only genes with the ISBN indexed. The idea is to let the genetic process find a solution without any help.

This same chromosome pattern can also be used to represent a type of query. A 1 in a "query chromosome" means that the corresponding column is to be returned; 0 means it's not. For example, the query, "find I, THE JURY by Mickey Spillane" has the genes (00101) because the title and author are given.

This correspondence makes it simple to simulate query runs. The payoff formula is based on hitting or missing an index in a query. The optimal score is to ask for only indexed columns, which makes sense because there's a chance that a nonindexed column would require a sequential search of the table.

Fotouhi and Galarce ran a series of random queries with a known statistical distribution against the test database of one million rows. The genes with the highest scores were saved from ("survived") that test run and used to build the next test run ("generation"). The performance of the system was measured in terms of average query-response times. The system leveled out in about ten generations with a 5-bit chromosome, but took longer with a 10-bit chromosome.

The Fotouhi-Galarce experiment was based on a single table, a rare occurrence in the real world. Tables are built based on a set of functional dependencies (FDs). An FD between two data items means that if I know one value, I can determine the second. This is written "A->B" and read "A determines B." If I know the part's stock number, I can look up its weight in the inventory file.

The way I combine columns to make tables uses normal forms built from FDs. A first-normal-form (1NF) table is simply a table; you cannot avoid it in SQL. A second-normal-form (2NF) table is a table with at least one key. A third-normal-form (3NF) has only one key and no transitive dependencies. A transitive dependency would be a table with columns A, B, and C such that (A->B) and (B->C), which implies (A->C).

There are higher normal forms, but most databases have to get to at least 3NF to work without anomalies. A quick example shows why this is important. Consider a table T(department, advisor, student) for faculty advisors at a college. If Dr. Celko in computer science is deleted, so are all his students (deletion anomaly); we cannot start a department until we have an advisor for it (insertion anomaly); if Dr. Celko switches to the math department, so do all his students (update anomaly). What we needed was two tables, T1(department, advisor) and T2(student, department), to get rid of the transitive dependency.

The bad news is that it's possible to develop more than one 3NF schema from the same set of FDs.

An Example

Let's say an imaginary airline has a database for scheduling flights and pilots. Most of the relationships are obvious: Flights have only one departure time and one destination, and they can get a different pilot and be assigned to a different gate each day of the week.

The functional dependencies for the database are given in Example 2 . Example 3, provides five possible answers, although there can be many more. The query chromosome structure would have six genes (day, destination, flight, gate, hour, and pilot). To show each possible 3NF schema, we first build tables where functional dependencies for the database are the key. Once the tables are defined, we apply queries against the whole database schema, not just one table, as we did before.

Example 2: The functional dependencies for the database.

  flight -> destination
  flight -> hour
  (day, flight) -> gate
  (day, flight) -> pilot
  (day, hour, gate)-> destination
  (day, hour, gate)-> flight
  (day, hour, gate) -> pilot
  (day, hour, pilot)-> destination
  (day, hour, pilot)-> flight
  (day, hour, pilot)-> gate

Example 3: Pseudo-SQL notation for creating tables within a database schema. Type declarations and constraints are not shown, just the table names, column names, and primary keys.

  CREATE SCHEMA Normal_1
  CREATE TABLE Departures
    (flight, destination, hour, PRIMARY KEY (flight)):
  CREATE TABLE WeeklyRoster
    (day, hour, gate, flight, pilot,
     PRIMARY KEY (day, hour, gate));

  CREATE SCHEMA Normal_2;
  CREATE TABLE Departures
    (flight, destination, hour, PRIMARY KEY (flight));
  CREATE TABLE Weekly Roster
    (day, hour, pilot, flight, gate,
     PRIMARY KEY (day, hour, pilot));

  CREATE SCHEMA Normal_3;
  CREATE TABLE Departures
    (flight, destination, hour, PRIMARY KEY (flight))
  CREATE TABLE GatePilotSchedule
    (day, flight, gate, pilot, PRIMARY KEY (day, flight));
  CREATE TABLE GateFlightSchedule
    (day, hour, gate, flight, PRIMARY KEY (day, hour, gate));
  CREATE TABLE PilotFlightSchedule
    (day, hour, pilot, flight, PRIMARY KEY (day, hour, pilot));

  CREATE SCHEMA Normal_4;
  CREATE TABLE Departures
    (destination, hour, flight, PRIMARY KEY (flight));
  CREATE TABLE GateFlightSchedule
    (day, flight, gate, PRIMARY KEY (day, flight));
  CREATE TABLE GatePilotSchedule
    (day, hour, gate, pilot, PRIMARY KEY (day, hour gate));
  CREATE TABLE PilotFlightSchedule
    (day, hour, pilot, flight, PRIMARY KEY (day, hour, pilot));

  CREATE SCHEMA Normal_5;
  CREATE TABLE Departures
    (destination, hour, flight, PRIMARY KEY (flight));
  CREATE TABLE DutyRoster
    (day, flight, pilot, PRIMARY KEY (day, flight ));
  CREATE TABLE GateFlightSchedule
    (day, hour, gate, flight, PRIMARY KEY (day, hour, gate));
  CREATE TABLE GatePilotSchedule
    (day, hour, pilot, gate, PRIMARY KEY (day, hour, pilot));

A table chromosome is made up of a subset of the ten original functional dependencies. Two rules have to be obeyed by the tables. First, no combination that violates the 3NF condition is allowed; it's a "lethal mutation" and is rejected at once. Second, all columns must be present in some table in the schema; this assumes that you won't put information into a database if you aren't going to look at it.

The database schemas will be made up of more than one table, and we'll be mutating one or more tables at a time. The payoff function will have to consider joins between tables (an expensive operation), the number of tables accessed, and so forth.

A join is how SQL relates data in one table to that in another. A join query builds a result table, which is made up of columns from other tables. Given tables X1(x,b,c) and X2(x,d,e), the syntax in SQL for a simple join would look like this:

  SELECT X1.x, b, c, d, e   FROM X1, X2   WHERE (X1.x = X2.x)

This builds a result table with five columns by taking each row in X1 and concatenating it to each row in X2 (a Cartesian product), then keeping only those new rows for which the join condition (X1.x = X2.x) is true. The <table>.<column> notation is used when columns in different tables have the same name.

Notice that indexes are not mentioned in the SQL SELECT statement, unlike Xbase and other nonrelational database languages. The SQL engine decides which indexes (if any) to use and builds a query plan behind the scenes. This means that indexes can be dropped and created from the schema without changing the queries; but you do have to recompile the query plans. Indexes can both help and hurt the performance of joins. Hopefully, the query optimizer will pick the fastest search, but it can be fooled. It might pick an index when a sequential read of the table would have been faster.

For example, if I want to use the flight to find both the day and the destination, I have to join TABLE Departures to TABLE GateFlightSchedule in the Normal_4 schema.

The operations on the schema will now be more complex than those we used for indexes on a single table. We have to allow tables to combine or split. The goal is to have the smallest number of tables used in the queries to avoid the cost of joins. Once the tables are determined for the set of queries, we can apply the index genetic algorithm to the tables.

Conclusion

To implement a system like this using existing system facilities, start with the query optimizer in most mainframe SQL packages that has the name of the indexes used in its query plans. The system tracks the CPU usage of each job on several different parameters--time, resources, working storage used, and so on. Most shops have repeated workloads which are close to identical in size and mix on weekly or monthly cycles. The statistics are fairly simple; the real trick is deciding on a payoff function.

Every large-scale database package has a utility program to trap statistics--the number of accesses to a table, resources used by a job, and so on. Next, have the indexes automatically appear or disappear as needed, a process that's easy in SQL using the commands DROP <index name>; and CREATE INDEX <index name> ON <table>(<column list>);, usually with vendor extensions on the CREATE INDEX statement. Remember to recompile the execution plans.

You can't reorganize the database schema because the queries would not work. Of course, as the input changes the old schema might not be the best possible, so you would have to run the genetic system on a regular basis or whenever the performance of the system declined dramatically.

References

Comer, D. "The Difficulty of Optimum Index Selection." ACM Transactions on Database Systems (vol. 3, 1978).

Fotouhi, Farshad and Carlos E. Galarce. "Genetic Algorithms and the Search for Optimal Database Index Selection," in Lecture Notes on Computer Science #507. Berlin: Springer-Verlag, 1991.

Goldberg, D.E. Genetic Algorithms. Reading, MA: Addison-Wesley, 1989.

Paitetsky-Shapiro, G. "The Optimal Selection of Secondary Indexes in NP-Complete." SIGMOD Record (vol. 13-N2, 1983).

Copyright © 1993, Dr. Dobb's Journal

Back to Table of Contents


Table 1

Copyright © 1993, Dr. Dobb's Journal


DEC93: The IDEA Encryption Algorithm

The IDEA Encryption Algorithm

An advanced block-cipher approach to encryption

Bruce Schneier

Bruce is author of Applied Cryptography (John Wiley & Sons, 1993) and can be contacted at 730 Fair Oaks Ave., Oak Park, IL 60302.

For the past 15 years, the best security most of have heard about is the Digital Encryption Standard (DES). It's a good algorithm, and secure against mid-seventies technology. With the computing power of today's machines, however, DES's small key size has made the algorithm vulnerable, leading cryptographers to look for, and propose new, stronger algorithms.

One such proposal, the International Data Encryption Algorithm (IDEA) invented in Switzerland by Xuejia Lai and James Massey, may be one of most secure block algorithms available to the public today. IDEA is a block cipher, operating on 64-bit data blocks: 64 bits of plaintext go in one end and 64-bits of ciphertext come out the other. The key is 128 bits long. The same algorithm is used for both encryption and decryption.

The design philosophy behind the algorithm is one of "mixing operations from different algebraic groups." There are three algebraic groups whose operations are being mixed, all of which are easily implemented in both hardware and software:

  • XOR.
  • Addition modulo 216 (addition, ignoring any overflow).
  • Multiplication modulo 216+1 (multiplication, ignoring any overflow).
All these operations (and these are the only operations in the algorithm--there are no permutations) operate on 16-bit subblocks, making it efficient even on 16-bit processors.

Figure 1 shows a diagram of IDEA. The 64-bit data block is divided into four 16-bit subblocks, which are X1, X2, X3, and X4. These four subblocks become the input to the first round of the algorithm. There are eight rounds total. In each round, the four subblocks are XORed, added, and multiplied with each other and six 16-bit subblocks of key material. Between each round, the second and third subblocks are swapped.

The sequence of events in each round is as follows:

  • 1 Multiply X1 and the first key subblock.
  • 2 Add X2 and the second key subblock.
  • 3 Add X3 and the third key subblock.
  • 4 Multiply X4 and the fourth key subblock.
  • 5 XOR the results of step #1 and #3.
  • 6 XOR the results of step #2 and #4.
  • 7 Multiply the results of step #5 with the fifth key subblock.
  • 8 Add the results of step #6 and #7.
  • 9 Multiply the results of step #8 with the sixth key subblock.
  • 10 Add the results of step #7 and #9.
  • 11 XOR the results of step #1 and #9.
  • 12 XOR the results of step #3 and #9.
  • 13 XOR the results of steps #2 and #10.
  • 14 XOR the results of steps #4 and #10.
That's it. The output of the round is the four subblocks that result from steps 11, 13, 12, and 14. Swap the two inner blocks (except for the last round), and that's the input to the next round. After the eighth round, there's a final output transform:

  • 1 Multiply X1 and the first key subblock.
  • 2 Add X2 and the second key subblock.
  • 3 Add X3 and the third key subblock.
  • 4 Multiply X4 and the fourth key subblock.
Finally, the four subblocks are reattached to produce the ciphertext.

Creating the key subblocks is also easy. The algorithm uses 52 of them (six for each of the eight rounds, and four more for the output transform.) First, the 128-bit key is divided into eight 16-bit subkeys. These are the first eight subkeys for the algorithm (the six for the first round, and the first two for the second round). Then, the key is rotated 25 bits to the left and again divided into eight subkeys. The first four are used in round two; the last four are used in round three. The key is rotated another 25 bits to the left for the next eight subkeys, and so on until the end of the algorithm.

Decryption is the same, except that the key subblocks are reversed and slightly different. The decryption key subblocks are either the additive or multiplicative inverses of the encryption-key subblocks. (For the purposes of IDEA, the multiplicative inverse of 0 is 0.) Calculating these takes some doing, but you only have to do it once for each decryption key. Table 1 shows the encryption key subblocks and the corresponding decryption key subblocks.

Current software implementations of IDEA are faster than DES (about 1.5 to 2 times as fast). On a 33-MHz 386 machine, IDEA encrypts data at 880 kbps. (An equivalent DES implementation encrypts data at only 584 kbps.) On a VAX 9000, the speed is almost four times greater. Listing One (page 105) is an implementation of IDEA in C.

A VLSI implementation of IDEA (developed at ETH Zurich) consisting of 251,000 transistors on a chip 107.8mm on a side, encrypts data using IDEA at a 177 Mbits/second data rate when clocked at 25 MHz.

IDEA can work within any block-cipher mode discussed in relation to the DES algorithm: Electronic Codebook (ECB), Cipher Block Chaining (CBC), Output Feedback (OFB), and Cipher Feedback (CFB). ECB (see Figure 2) is the simplest mode. The plaintext is encrypted in blocks of 64 bits. The first 64 bits of plaintext are encrypted to become the first 64 bits of ciphertext; the second 64 bits of plaintext are encrypted to become the second 64 bits of ciphertext; and so on. Each block is encrypted independently of all other blocks. Decryption is the reverse.

The problem with ECB mode is that, if a cryptanalyst has the plaintext and ciphertext for several messages, he can start to compile a codebook without knowing the key. In most real-world situations, fragments of messages tend to repeat. One message may have bit sequences in common with another. Computer-generated messages (such as electronic mail) may have a regular structure. Messages may have parameters that take only a few values, or long strings of zeros or spaces. Other computer-generated messages may always have important data in the same place.

If the cryptanalyst learns that the plaintext block "5e081bc5" encrypts to the ciphertext block "7ea593a4", he can immediately decrypt that ciphertext block whenever he sees it in another message. If the application encrypts messages with many redundancies, and these redundancies tend to show up in the same places in the message, this can be a very powerful attack.

The other three modes defend against this sort of attack. In CBC mode, the plaintext is XORed with the previous ciphertext block before it is encrypted. Figure 3 shows CBC in action. After a plaintext block is encrypted, the resulting ciphertext is also stored in a feedback register. Before the next plaintext block is encrypted, it is XORed with the feedback register to become the next input to the encrypting routine. The resultant ciphertext is again stored in the feedback register, to be XORed with the next plaintext block. And so on until the end of the message. The encryption of each block depends on all the previous blocks.

Decryption is just as straightforward. A ciphertext block is decrypted normally, and also saved in a feedback register. After the next block is decrypted, it is XORed with the results of the feedback register. Then the next ciphertext block is stored in the feedback register, and so on, until the end of the message.

Mathematically, this appears thus:

Ci = EK(Pi XOR Ci-1)

Pi = Ci-1 XOR DK(Ci)

OFB and CFB (described in standard cryptography books) are two other feedback mechanisms.

IDEA Security

IDEA's key length is 128 bits--over twice as long as the DES key. Assuming that testing every possible key (a brute-force attack) is the most efficient way to break the algorithm, it would require 2128 encryptions to recover the key. DES only requires 256 encryptions to break; a million chips capable of testing a million keys a second can break DES in 20 hours. IDEA is much stronger. Design a chip that can test a billion keys per second, and throw a billion of them at the problem, and it will still take 1013 years to break IDEA. An array of 1024 such chips can find the key in a day, but it questionable whether there are enough silicon atoms in the universe to build such a machine.

Unless, of course, brute force isn't the best way to attack IDEA. The algorithm is still too new for any definitive statements about its security. The designers have done their best to make the algorithm immune to all known cryptanalytic attacks (including a new and powerful attack called "differential cryptanalysis").

While IDEA appears to be significantly more secure than DES, it isn't always easy to substitute one for the other in an existing application. If your database and message templates are hardwired to accept a 64-bit key, it may be impossible to implement IDEA's 128-bit key.

For those applications, generate a 128-bit key by concatenating the 64-bit key with itself. Remember, however, that IDEA is weakened considerably by this modification.

Conclusions

IDEA is a new algorithm that looks secure. While it appears to be resistant to differential and related-key cryptanalysis, secure-looking algorithms have fallen to new forms of cryptanalysis time and time again. Several academic and military groups currently cryptanalyzing IDEA. None of them have succeeded yet, but who knows what they might come up with tomorrow.

The IDEA block cipher is patented in Europe, and the patent is pending in the United States. The patent is held by Ascom-Tech AG. No license fee is required for noncommercial use. Commercial users interested in licensing the algorithm should contact Dr. Dieter Profos, Ascom Tech AG, Solothurn Lab, Postfach 151, 4502 Solothurn, Switzerland (Tel: +41 65 242-885 or fax: +41 65 235-761).

References

Lai, X., J. Massey, and S. Murphy. "Markov Ciphers and Differential Cryptanalysis." Advances in Cryptology--EUROCRYPT '91 Proceedings. Springer-Verlag, 1991.

Massey, J.L. and X. Lai, "Device for Converting a Digital Block and the Use Thereof." International Patent PCT/CH91/00117, November 28, 1991.

Schneier, Bruce. Applied Cryptography. New York: John Wiley & Sons, 1993.

Figure 1: Diagram of the IDEA approach to cryptography.

Table 1: IDEA encryption- and decryption-key subblocks.

Figure 2: Electronic codebook (ECB) mode.

Figure 3: Cipher-block chaining (CBC) mode.

[LISTING ONE]

/* idea.h - header file for idea.c */

#include "usuals.h"  /* typedefs for byte, word16, boolean, etc. */
#define IDEAKEYSIZE 16
#define IDEABLOCKSIZE 8
void initcfb_idea(word16 iv0[4], byte key[16], boolean decryp);
void ideacfb(byteptr buf, int count);
void close_idea(void);
void init_idearand(byte key[16], byte seed[8], word32 tstamp);
byte idearand(void);
void close_idearand(void);

/* prototypes for passwd.c */
/* GetHashedPassPhrase -get pass phrase from user, hashes it to an IDEA key. */
int GetHashedPassPhrase(char *keystring, char *hash, boolean noecho);

/* hashpass - Hash pass phrase down to 128 bits (16 bytes). */
void hashpass (char *keystring, int keylen, byte *hash);

/********************************IDEA.C*******************************/
/* idea.c - C source code for IDEA block cipher. IDEA (International Data
 * Encryption Algorithm), formerly known as IPES (Improved Proposed Encryption
 * Standard). Algorithm developed by Xuejia Lai and James L. Massey, of ETH
 * Zurich. This implementation modified and derived from original C code
 * developed by Xuejia Lai. Zero-based indexing added, names changed from IPES
 * to IDEA. CFB functions added. Random number routines added. Optimized for
 * speed 21 Oct 92 by Colin Plumb <colin@nsq.gts.org>. This code assumes that
 * each pair of 8-bit bytes comprising a 16-bit word in the key and in the
 * cipher block are externally represented with the Most Significant Byte
 * (MSB) first, regardless of internal native byte order of the target CPU.  */

#include "idea.h"

#ifdef TEST
#include <stdio.h>
#include <time.h>
#endif

#define ROUNDS      8               /* Don't change this value, should be 8 */
#define KEYLEN      (6*ROUNDS+4)    /* length of key schedule */

typedef word16 IDEAkey[KEYLEN];

#ifdef IDEA32/* Use >16-bit temporaries */
#define low16(x) ((x) & 0xFFFF)
typedef unsigned int uint16;        /* at LEAST 16 bits, maybe more */
#else
#define low16(x) (x)                /* this is only ever applied to uint16's */
typedef word16 uint16;
#endif

#ifdef _GNUC_
/* __const__ simply means there are no side effects for this function,
 * which is useful info for the gcc optimizer */
#define CONST __const__
#else
#define CONST
#endif

static void en_key_idea(word16 userkey[8], IDEAkey Z);
static void de_key_idea(IDEAkey Z, IDEAkey DK);
static void cipher_idea(word16 in[4], word16 out[4], CONST IDEAkey Z);
/* Multiplication, modulo (2**16)+1. Note that this code is structured like
 * this on the assumption that untaken branches are cheaper than taken
 * branches, and the compiler doesn't schedule branches. */
#ifdef SMALL_CACHE
CONST static uint16 mul(register uint16 a, register uint16 b)
{
       register word32 p;
       if (a)
       {     if (b)
             {      p = (word32)a * b;
                    b = low16(p);
                    a = p>>16;
                    return b - a + (b < a);
             }
             else
             {      return 1-a;
             }
       }
       else
       {     return 1-b;
       }
}        /* mul */
#endif /* SMALL_CACHE */
/* Compute multiplicative inverse of x, modulo (2**16)+1, using Euclid's GCD
 * algorithm. It is unrolled twice to avoid swapping the meaning of the
 * registers each iteration; some subtracts of t have been changed to adds.  */
CONST static uint16 inv(uint16 x)
{
       uint16 t0, t1;
       uint16 q, y;
       if (x <= 1)
             return x;    /* 0 and 1 are self-inverse */
       t1 = 0x10001 / x;  /* Since x >= 2, this fits into 16 bits */
       y = 0x10001 % x;
       if (y == 1)
             return low16(1-t1);
       t0 = 1;
       do
       {     q = x / y;
             x = x % y;
             t0 += q * t1;
             if (x == 1)
                    return t0;
             q = y / x;
             y = y % x;
             t1 += q * t0;
       } while (y != 1);
       return low16(1-t1);
} /* inv */
/*     Compute IDEA encryption subkeys Z */
static void en_key_idea(word16 *userkey, word16 *Z)
{
       int i,j;
       /* shifts */
       for (j=0; j<8; j++)
             Z[j] = *userkey++;
       for (i=0; j<KEYLEN; j++)
       {     i++;
             Z[i+7] = Z[i & 7] << 9 | Z[i+1 & 7] >> 7;
             Z += i & 8;
             i &= 7;
       }
}        /* en_key_idea */
/*     Compute IDEA decryption subkeys DK from encryption subkeys Z */
/* Note: these buffers *may* overlap! */
static void de_key_idea(IDEAkey Z, IDEAkey DK)
{
       int j;
       uint16 t1, t2, t3;
       IDEAkey T;
       word16 *p = T + KEYLEN;
       t1 = inv(*Z++);
       t2 = -*Z++;
       t3 = -*Z++;
       *--p = inv(*Z++);
       *--p = t3;
       *--p = t2;
       *--p = t1;
       for (j = 1; j < ROUNDS; j++)
       {
             t1 = *Z++;
             *--p = *Z++;
             *--p = t1;
             t1 = inv(*Z++);
             t2 = -*Z++;
             t3 = -*Z++;
             *--p = inv(*Z++);
             *--p = t2;
             *--p = t3;
             *--p = t1;
       }
       t1 = *Z++;
       *--p = *Z++;
       *--p = t1;
       t1 = inv(*Z++);
       t2 = -*Z++;
       t3 = -*Z++;
       *--p = inv(*Z++);
       *--p = t3;
       *--p = t2;
       *--p = t1;
/* Copy and destroy temp copy */
       for (j = 0, p = T; j < KEYLEN; j++)
       {
             *DK++ = *p;
             *p++ = 0;
       }
} /* de_key_idea */
/* MUL(x,y) computes x = x*y, modulo 0x10001. Requires two temps, t16 and t32.
 * x must me a side-effect-free lvalue. y may be anything, but unlike x, must
 * be strictly 16 bits even if low16() is #defined. All of these are
 * equivalent; see which is faster on your machine.  */
#ifdef SMALL_CACHE
#define MUL(x,y) (x = mul(low16(x),y))
#else
#ifdef AVOID_JUMPS
#define MUL(x,y) (x = low16(x-1), t16 = low16((y)-1), \
             t32 = (word32)x*t16+x+t16+1, x = low16(t32), \
             t16 = t32>>16, x = x-t16+(x<t16) )
#else
#define MUL(x,y) ((t16 = (y)) ? (x=low16(x)) ? \
        t32 = (word32)x*t16, x = low16(t32), t16 = t32>>16, \
        x = x-t16+(x<t16) : \
        (x = 1-t16) : (x = 1-x))
#endif
#endif
/* IDEA encryption/decryption algorithm . In/out can be the same buffer */
static void cipher_idea(word16 in[4], word16 out[4], register CONST IDEAkey Z)
{
       register uint16 x1, x2, x3, x4, t1, t2;
       register uint16 t16;
       register word32 t32;
       int r = ROUNDS;
       x1 = *in++;  x2 = *in++;
       x3 = *in++;  x4 = *in;
       do
       {
             MUL(x1,*Z++);
             x2 += *Z++;
             x3 += *Z++;
             MUL(x4, *Z++);
             t2 = x1^x3;
             MUL(t2, *Z++);
             t1 = t2 + (x2^x4);
             MUL(t1, *Z++);
             t2 = t1+t2;
             x1 ^= t1;
             x4 ^= t2;
             t2 ^= x2;
             x2 = x3^t1;
             x3 = t2;
       } while (--r);
       MUL(x1, *Z++);
       *out++ = x1;
       *out++ = x3 + *Z++;
       *out++ = x2 + *Z++;
       MUL(x4, *Z);
       *out = x4;
} /* cipher_idea */
/*-------------------------------------------------------------*/
#ifdef TEST
/* Number of Kbytes of test data to encrypt. Defaults to 1 MByte. */
#ifndef KBYTES
#define KBYTES 1024
#endif
void main(void)
{      /* Test driver for IDEA cipher */
       int i, j, k;
       IDEAkey Z, DK;
       word16 XX[4], TT[4], YY[4];
       word16 userkey[8];
       clock_t start, end;
       long l;
       /* Make a sample user key for testing... */
       for(i=0; i<8; i++)
             user
key[i] = i+1;
       /* Compute encryption subkeys from user key... */
       en_key_idea(userkey,Z);
       printf("\nEncryption key subblocks: ");
       for(j=0; j<ROUNDS+1; j++)
       {
             printf("\nround %d:   ", j+1);
             if (j==ROUNDS)
                    for(i=0; i<4; i++)
                          printf(" %6u", Z[j*6+i]);
             else
                    for(i=0; i<6; i++)
                          printf(" %6u", Z[j*6+i]);
       }
       /* Compute decryption subkeys from encryption subkeys... */
       de_key_idea(Z,DK);
       printf("\nDecryption key subblocks: ");
       for(j=0; j<ROUNDS+1; j++)
       {
             printf("\nround %d:   ", j+1);
             if (j==ROUNDS)
                    for(i=0; i<4; i++)
                          printf(" %6u", DK[j*6+i]);
             else
                    for(i=0; i<6; i++)
                          printf(" %6u", DK[j*6+i]);
       }
       /* Make a sample plaintext pattern for testing... */
       for (k=0; k<4; k++)
             XX[k] = k;
       printf("\n Encrypting %d KBytes (%ld blocks)...", KBYTES, KBYTES*64l);
       fflush(stdout);
       start = clock();
       cipher_idea(XX,YY,Z);         /* encrypt plaintext XX, making YY */
       for (l = 1; l < 64*KBYTES; l++)
             cipher_idea(YY,YY,Z);   /* repeated encryption */
       cipher_idea(YY,TT,DK);        /* decrypt ciphertext YY, making TT */
       for (l = 1; l < 64*KBYTES; l++)
             cipher_idea(TT,TT,DK);  /* repeated decryption */
       end = clock() - start;
       l = end * 1000. / CLOCKS_PER_SEC + 1;
       i = l/1000;
       j = l%1000;
       l = KBYTES * 1024. * CLOCKS_PER_SEC / end;
       printf("%d.%03d seconds = %ld bytes per second\n", i, j, l);
       printf("\nX %6u   %6u  %6u  %6u \n",
         XX[0], XX[1],  XX[2], XX[3]);
       printf("Y %6u   %6u  %6u  %6u \n",
         YY[0], YY[1],  YY[2], YY[3]);
       printf("T %6u   %6u  %6u  %6u \n",
         TT[0], TT[1],  TT[2], TT[3]);
       /* Now decrypted TT should be same as original XX */
       for (k=0; k<4; k++)
             if (TT[k] != XX[k])
             {
                    printf("\n\07Error!  Noninvertable encryption.\n");
                    exit(-1);      /* error exit */
             }
       printf("\nNormal exit.\n");
       exit(0);/* normal exit */
}        /* main */
#endif       /* TEST */
/*************************************************************************/
/* xorbuf - change buffer via xor with random mask block. Used for Cipher
 * Feedback (CFB) or Cipher Block Chaining (CBC) modes of encryption. Can be
 *  applied for any block encryption algorithm, with any block size, such as
 * the DES or the IDEA cipher.  */
static void xorbuf(register byteptr buf, register byteptr mask,
       register int count)
/*     count must be > 0 */
{
       if (count)
             do
                    *buf++ ^= *mask++;
             while (--count);
}      /* xorbuf */
/* cfbshift - shift bytes into IV for CFB input. Used only for Cipher Feedback
 * (CFB) mode of encryption. Can be applied for any block encryption algorithm
 * with any block size, such as the DES or the IDEA cipher.  */
static void cfbshift(register byteptr iv, register byteptr buf,
             register int count, int blocksize)
/* iv is initialization vector. buf is buffer pointer. count is number of bytes
 * to shift in...must be > 0. blocksize is 8 bytes for DES or IDEA ciphers. */
{
       int retained;
       if (count)
       {
             retained = blocksize-count;   /* number bytes in iv to retain */
         /* left-shift retained bytes of IV over by count bytes to make room */
             while (retained--)
             {
                    *iv = *(iv+count);
                    iv++;
             }
             /* now copy count bytes from buf to shifted tail of IV */
             do     *iv++ = *buf++;
             while (--count);
       }
}      /* cfbshift */
/* Key schedules for IDEA encryption and decryption */
static IDEAkey Z, DK;
static word16 *iv_idea;        /* pointer to IV for CFB or CBC */
static boolean cfb_dc_idea;    /* TRUE iff CFB decrypting */
/* initkey_idea initializes IDEA for ECB mode operations */
void initkey_idea(byte key[16], boolean decryp)
{
       word16 userkey[8]; /* IDEA key is 16 bytes long */
       int i;
       /* Assume each pair of bytes comprising a word is ordered MSB-first. */
       for (i=0; i<8; i++)
       {
             userkey[i] = (key[0]<<8) + key[1];
             key++; key++;
       }
       en_key_idea(userkey,Z);
       if (decryp)
       {
             de_key_idea(Z,Z);   /* compute inverse key schedule DK */
       }
       for (i=0; i<8; i++)/* Erase dangerous traces */
             userkey[i] = 0;
} /* initkey_idea */
/* Run a 64-bit block thru IDEA in ECB (Electronic Code Book) mode, using the
 * currently selected key schedule. */
void idea_ecb(word16 *inbuf, word16 *outbuf)
{
       /* Assume each pair of bytes comprising a word is ordered MSB-first. */
#ifndef HIGHFIRST   /* If this is a least-significant-byte-first CPU */
       word16 x;
       /* Invert the byte order for each 16-bit word for internal use. */
       x = inbuf[0]; outbuf[0] = x >> 8 | x << 8;
       x = inbuf[1]; outbuf[1] = x >> 8 | x << 8;
       x = inbuf[2]; outbuf[2] = x >> 8 | x << 8;
       x = inbuf[3]; outbuf[3] = x >> 8 | x << 8;
       cipher_idea(outbuf, outbuf, Z);
       x = outbuf[0]; outbuf[0] = x >> 8 | x << 8;
       x = outbuf[1]; outbuf[1] = x >> 8 | x << 8;
       x = outbuf[2]; outbuf[2] = x >> 8 | x << 8;
       x = outbuf[3]; outbuf[3] = x >> 8 | x << 8;
#else  /* HIGHFIRST */
       /* Byte order for internal and external representations is the same. */
       cipher_idea(inbuf, outbuf, Z);
#endif /* HIGHFIRST */
} /* idea_ecb */
/* initcfb - Initializes IDEA key schedule tables via key; initializes Cipher
 * Feedback mode IV. References context variables cfb_dc_idea and iv_idea.  */
void initcfb_idea(word16 iv0[4], byte key[16], boolean decryp)
/* iv0 is copied to global iv_idea, buffer will be destroyed by ideacfb. key is
 * pointer to key buffer. decryp is TRUE if decrypting, FALSE if encrypting. */
{
       iv_idea = iv0;
       cfb_dc_idea = decryp;
       initkey_idea(key,FALSE);
} /* initcfb_idea */
/* ideacfb - encipher a buffer with IDEA enciphering algorithm, using Cipher
 *  Feedback (CFB) mode. Assumes initcfb_idea has already been called.
 * References context variables cfb_dc_idea and iv_idea.  */
void ideacfb(byteptr buf, int count)
/* buf is input, output buffer, may be more than 1 block. count is byte count
 * is byte count of buffer.  May be > IDEABLOCKSIZE. */
{
       int chunksize;                /* smaller of count, IDEABLOCKSIZE */
       word16 temp[IDEABLOCKSIZE/2];

       while ((chunksize = min(count,IDEABLOCKSIZE)) > 0)
       {
             idea_ecb(iv_idea,temp);  /* encrypt iv_idea, making temp. */
             if (cfb_dc_idea)/* buf is ciphertext */
                    /* shift in ciphertext to IV... */
                    cfbshift((byte *)iv_idea,buf,chunksize,IDEABLOCKSIZE);
             /* convert buf via xor */
             xorbuf(buf,(byte *)temp,chunksize);/* buf has enciphered output */
             if (!cfb_dc_idea)/* buf was plaintext, is now ciphertext */
                    /* shift in ciphertext to IV... */
                    cfbshift((byte *)iv_idea,buf,chunksize,IDEABLOCKSIZE);
             count -= chunksize;
             buf += chunksize;
       }
} /* ideacfb */
/* close_idea function erases all the key schedule information when we are
 * done with a set of operations for a particular IDEA key context. This is to
 * prevent any sensitive data from being left around in memory. */
void close_idea(void)     /* erase current key schedule tables */
{
       short i;
       for (i = 0; i < KEYLEN; i++)
             Z[i] = 0;
}      /* close_idea() */
/********************************************************************/
/* These buffers are used by init_idearand, idearand, and close_idearand. */
static word16 dtbuf_idea[4] = {0};     /* buffer for enciphered timestamp */
static word16 randseed_idea[4] = {0};  /* seed for IDEA random # generator */
static word16 randbuf_idea[4] = {0};   /* buffer for IDEA random # generator */
static byte randbuf_idea_counter = 0;  /* random bytes left in randbuf_idea */
/* init_idearand - initialize idearand, IDEA random number generator. Used for
 *  generating cryptographically strong random numbers. Much of design comes
 * from Appendix C of ANSI X9.17. key is pointer to IDEA key buffer. seed is
 * pointer to random number seed buffer. tstamp is a 32-bit timestamp */
void init_idearand(byte key[16], byte seed[8], word32 tstamp)
{
       int i;
       initkey_idea(key, FALSE);      /* initialize IDEA */
       for (i=0; i<4; i++)            /* capture timestamp material */
       {     dtbuf_idea[i] = tstamp;  /* get bottom word */
             tstamp = tstamp >> 16;   /* drop bottom word */
             /* tstamp has only 4 bytes-- last 4 bytes will always be 0 */
       }
       /* Start with enciphered timestamp: */
       idea_ecb(dtbuf_idea,dtbuf_idea);
       /* initialize seed material */
       for (i=0; i<8; i++)
             ((byte *)randseed_idea)[i] = seed[i];
       randbuf_idea_counter = 0; /* # of random bytes left in randbuf_idea */
} /* init_idearand */
/* idearand - IDEA pseudo-random number generator. Used for generating
 * cryptographically strong random numbers. Much of design comes from Appendix
 * C of ANSI X9.17.  */
byte idearand(void)
{
       int i;
       if (randbuf_idea_counter==0)         /* if random buffer is spent...*/
       {     /* Combine enciphered timestamp with seed material: */
             for (i=0; i<4; i++)
                    randseed_idea[i] ^= dtbuf_idea[i];
             idea_ecb(randseed_idea,randbuf_idea);   /* fill new block */
             /* Compute new seed vector: */
             for (i=0; i<4; i++)
                    randseed_idea[i] = randbuf_idea[i] ^ dtbuf_idea[i];
             idea_ecb(randseed_idea,randseed_idea);    /* fill new seed */
             randbuf_idea_counter = 8;   /* reset counter for full buffer */
       }
       /* Take a byte from randbuf_idea: */
       return(((byte *)randbuf_idea)[--randbuf_idea_counter]);
} /* idearand */
void close_idearand(void)
{      /* Erase random IDEA buffers and wipe out IDEA key info */
       int i;
       for (i=0; i<4; i++)
       {     randbuf_idea[i] = 0;
             randseed_idea[i] = 0;
             dtbuf_idea[i] = 0;
       }
       close_idea();/* erase current key schedule tables */
}      /* close_idearand */
/* end of idea.c */

Copyright © 1993, Dr. Dobb's Journal

Back to Table of Contents


Figure 1

Copyright © 1994, Dr. Dobb's Journal


Figure 2

Copyright © 1994, Dr. Dobb's Journal


APR94: The Blowfish Encryption Algorithm

The Blowfish Encryption Algorithm

Bruce Schneier

Bruce is the author of Applied Cryptography: Protocols, Algorithms, and Source Code in C (John Wiley, 1994). This article is based on a paper he presented at the Cambridge Algorithms Conference. Bruce can be contacted at schneier@ chinet.com.

Blowfish is a block-encryption algorithm designed to be fast (it encrypts data on large 32-bit microprocessors at a rate of 26 clock cycles per byte), compact (it can run in less than 5K of memory), simple (the only operations it uses are addition, XOR, and table lookup on 32-bit operands), secure (Blowfish's key length is variable and can be as long as 448 bits), and robust (unlike DES, Blowfish's security is not diminished by simple programming errors).

The Blowfish block-cipher algorithm, which encrypts data one 64-bit block at a time, is divided into key-expansion and a data-encryption parts. Key expansion converts a key of at most 448 bits into several subkey arrays totaling 4168 bytes. Data encryption consists of a simple function iterated 16 times. Each iteration, called a "round," consists of a key-dependent permutation and a key- and data-dependent substitution.

Subkeys

Blowfish uses a large number of subkeys that must be precomputed before any data encryption or decryption. The P-array consists of 18 32-bit subkeys, P1, P2...P18, and there are four 32-bit S-boxes with 256 entries each: S1,0, S1,1... S1,255; S2,0, S2,1...S2,255; S3,0, S3,1...S3,255; S4,0, S4,1...S4,255.

Encryption

Blowfish is a Feistel network consisting of 16 rounds; see Figure 1. The input is a 64-bit data element, x. Divide x into two 32-bit halves: xL and xR. Then, for i=1 to 16:

xL=xL XOR Pi
xR=F(xL) XOR xR
Swap xL and xR

After the sixteenth iteration, swap xL and xR to undo the last swap. Then xR=xR XOR P17 and xL=xL XOR P18. Finally, recombine xL and xR to get the ciphertext.

Function F looks like this: Divide xL into four eight-bit quarters: a, b, c, and d. F(xL)=((S1,a+S2,b mod 232)XOR S3,c)+ S4,d mod 232; see Figure 2.

Decryption is exactly the same as encryption, except that P1, P2_P18 are used in the reverse order.

Implementations of Blowfish that require the fastest speeds should unroll the loop and ensure that all subkeys are stored in cache. For the purposes of illustration, I've implemented Blowfish in C; Listing One (page 98) is blowfish.h, and Listing Two (page 98) is blowfish.c. A required data file is available electronically; see "Availability," page 3.

Generating the Subkeys

The subkeys are calculated using the Blowfish algorithm, as follows:

  1. Initialize first the P-array and then the four S-boxes, in order, with a fixed random string. This string consists of the hexadecimal digits of [pi].
  2. XOR P1 with the first 32 bits of the key, XOR P2 with the second 32 bits of the key, and so on for all bits of the key (up to P18). Cycle through the key bits repeatedly until the entire P-array has been XORed.
  3. Encrypt the all-zero string with the Blowfish algorithm, using the subkeys described in steps #1 and #2.
  4. Replace P1 and P2 with the output of step #3.
  5. Encrypt the all-zero string using the Blowfish algorithm with the modified subkeys.
  6. Replace P3 and P4 with the output of step #4.
  7. Continue the process, replacing all elements of the P-array and then all four S-boxes in order, with the output of the continuously changing Blowfish algorithm.
In total, 521 iterations are required to generate all required subkeys. Applications can store the subkeys rather than re-executing this derivation process.

Design Decisions

The underlying philosophy behind Blowfish is that simplicity of design yields an algorithm that is easier both to understand and to implement. Hopefully, the use of a streamlined Feistel network (the same structure used in DES, IDEA, and many other algorithms), a simple S-box substitution, and a simple P-box substitution, will minimize design flaws.

For details about the design decisions affecting the security of Blowfish, see "Requirements for a New Encryption Algorithm" (by B. Schneier and N. Fergusen) and "Description of a New Variable-Length Key, 64-Bit Block Cipher (Blowfish)" (by B. Schneier), both to be included in Fast Software Encryption, to be published by Springer-Verlag later this year as part of their Lecture Notes on Computer Science series. The algorithm is designed to be very fast on 32-bit microprocessors. Operations are all based on a 32-bit word and are one-instruction XORs, ADDs, and MOVs. There are no branches (assuming you unravel the main loop). The subkey arrays and the instructions can fit in the on-chip caches of both the Pentium and the PowerPC. Furthermore, the algorithm is designed to be resistant to poor implementation and programmer errors.

I'm considering several simplifications to the algorithm, including fewer and smaller S-boxes, fewer rounds, and on-the-fly subkey calculation.

Conclusions

At this early stage, I don't recommend implementing Blowfish in security systems. More analysis is needed. I conjecture that the most efficient way to break Blowfish is through exhaustive search of the keyspace. I encourage all cryptanalytic attacks, modifications, and improvements to the algorithm.

However, remember one of the basic rules of cryptography: The inventor of an algorithm is the worst person to judge its security. I am publishing the details of Blowfish so that others may have a chance to analyze it.

Blowfish is unpatented and will remain so in all countries. The algorithm is hereby placed in the public domain and can be freely used by anyone.

Figure 1: Blowfish is a Feistel network consisting of 16 rounds.

Figure 2: Blowfish function F.

DDJ's Blowfish Cryptanalysis Contest

The only way to inspire confidence in a cryptographic algorithm is to let people analyze it. It is in this spirit that DDJ is pleased to announce the Blowfish Cryptanalysis Contest, our third reader contest in recent years.

We'd like you to cryptanalyze Bruce Schneier's Blowfish algorithm presented in this issue. Give it your best shot. Break it, beat on it, cryptanalyze it. The best attack received by April 1, 1995 wins the contest.

The contest rules are simple. It's open to any individual or organization. Governments are encouraged to enter. Even the NSA can compete and win the prize (their budget isn't what it used to be; they can probably use the money). But since we will publish the results, classified entries will not be permitted. To officially enter the contest, your entry must be accompanied by a completed and signed entry form. These are available electronically (see "Availability," page 3) or we'll be glad to mail or fax you a hardcopy.

We're not going to publish messages encrypted in Blowfish and some random key, because we think that would be too difficult.

Partial results--those attacks that don't break the algorithm but instead prove that it isn't as strong as we thought it was--are just as useful and can be entered.

Your entry does not have to consist of code. Instead, your entry can be a paper describing the attack. The attack does not have to completely break the Blowfish algorithm, it can simply be more efficient than a brute-force attack. The attack can be against either the complete algorithm or a simplified version of the algorithm (fewer rounds, smaller block size, simpler S-boxes, and the like).

We'll select a winner based on the following criteria:

  • Success of the attack. How much more efficient is the attack than brute force?
  • Type of attack. Is it ciphertext only, known plaintext, or chosen plaintext?
  • Type of algorithm. Is the attack against full Blowfish or a simplified version of the algorithm?
Bruce Schneier, frequent DDJ contributor, author of Applied Cryptography, and inventor of the Blowfish algorithm will referee the contest.

The contest results will be published in the September 1995 issue of Dr. Dobb's Journal, in which we'll discuss and summarize the winning programs, the weaknesses of the Blowfish algorithm, and any modifications of the algorithm.

We'll be providing a number of awards for the winners. The grand-prize winner will receive a $750 honorarium. Honorariums of $250 to the second-place winner and $100 to the third-place winner will also be awarded.

--editors

[LISTING ONE]

/* Blowfish.h */

#define MAXKEYBYTES 56              /* 448 bits */

short opensubkeyfile(void);
unsigned long F(unsigned long x);
void Blowfish_encipher(unsigned long *xl, unsigned long *xr);
void Blowfish_decipher(unsigned long *xl, unsigned long *xr);
short InitializeBlowfish(char key[], short keybytes);

[LISTING TWO]

/* Blowfish.c */

#include <dos.h>
#include <graphics.h>
#include <io.h>
#include <math.h>
#include <stdio.h>
#include <stdlib.h>
#include <time.h>
#include <alloc.h>
#include <ctype.h>
#include <dir.h>
#include <bios.h>
#include <Types.h>

#define  N  16
#define  noErr 0
#define  DATAERROR -1
#define  KEYBYTES 8
#define  subkeyfilename "Blowfish.dat"

static unsigned long P[18];
static unsigned long S[4,256];
static FILE*         SubkeyFile;

short opensubkeyfile(void) /* read only */
{
   short error;
   error = noErr;
   if((SubkeyFile = fopen(subkeyfilename,"rb")) == NULL) {
      error = DATAERROR;
   }
   return error;
}
unsigned long F(unsigned long x)
{
   unsigned short a;
   unsigned short b;
   unsigned short c;
   unsigned short d;
   unsigned long  y;

   d = x & 0x00FF;
   x >>= 8;
   c = x & 0x00FF;
   x >>= 8;
   b = x & 0x00FF;
   x >>= 8;
   a = x & 0x00FF;

   y = ((S[0, a] + (S[1, b] % 32)) ^ S[2, c]) + (S[3, d] % 32);
   /* Note: There is a good chance that the following line will execute faster */
   /* y = ((S[0,a] + (S[1, b] & 0x001F)) ^ S[2, c]) + (S[3,d] & 0x001F); */
   return y;
}
void Blowfish_encipher(unsigned long *xl, unsigned long *xr)
{
   unsigned long  Xl;
   unsigned long  Xr;
   unsigned long  temp;
   short          i;

   Xl = *xl;
   Xr = *xr;
   for (i = 0; i < N; ++i) {
      Xl = Xl ^ P[i];
      Xr = F(Xl) ^ Xr;

      temp = Xl;
      Xl = Xr;
      Xr = temp;
   }
   temp = Xl;
   Xl = Xr;
   Xr = temp;

   Xr = Xr ^ P[N];
   Xl = Xl ^ P[N + 1];

   *xl = Xl;
   *xr = Xr;
}
void Blowfish_decipher(unsigned long *xl, unsigned long *xr)
{
   unsigned long  Xl;
   unsigned long  Xr;
   unsigned long  temp;
   short          i;

   Xl = *xl;
   Xr = *xr;

   for (i = N + 1; i > 1; --i) {
      Xl = Xl ^ P[i];
      Xr = F(Xl) ^ Xr;

      /* Exchange Xl and Xr */
      temp = Xl;
      Xl = Xr;
      Xr = temp;
   }
   /* Exchange Xl and Xr */
   temp = Xl;
   Xl = Xr;
   Xr = temp;

   Xr = Xr ^ P[1];
   Xl = Xl ^ P[0];

   *xl = Xl;
   *xr = Xr;
}
short InitializeBlowfish(char key[], short keybytes)
{
   short          i;
   short          j;
   short          k;
   short          error;
   short          numread;
   unsigned long  data;
   unsigned long  datal;
   unsigned long  datar;
   /* First, open the file containing the array initialization data */
   error = opensubkeyfile();
   if (error == noErr) {
      for (i = 0; i < N + 1; ++i) {
         numread = fread(&data, 4, 1, SubkeyFile);
         printf("%d : %d : %.4s\n", numread, i, &data);
         if (numread != 1) {
            return DATAERROR;
         } else {
            P[i] = data;
         }
      }
      for (i = 0; i < 4; ++i) {
         for (j = 0; j < 256; ++j) {
            numread = fread(&data, 4, 1, SubkeyFile);
            printf("[%d, %d] : %.4s\n", i, j, &data);
            if (numread != 1) {
               return DATAERROR;
            } else {
               S[i, j] = data;
            }
         }
      }
      fclose(SubkeyFile);
      j = 0;
      for (i = 0; i < 18; ++i) {
         data = 0x00000000;
         for (k = 0; k < 4; ++k) {
            data = (data << 8) | key[j];
            j = j + 1;
            if (j > keybytes) {
               j = 0;
            }
         }
         P[i] = P[i] ^ data;
      }
      datal = 0x00000000;
      datar = 0x00000000;
      for (i = 0; i < 18; i += 2) {
         Blowfish_encipher(&datal, &datar);

         P[i] = datal;
         P[i + 1] = datar;
      }
      for (j = 0; i < 4; ++j) {
         for (i = 0; i < 256; i += 2) {
            Blowfish_encipher(&datal, &datar);

            S[j, i] = datal;
            S[j, i + 1] = datar;
         }
      }
   } else {
      printf("Unable to open subkey initialization file : %d\n", error);
   }
   return error;
}

Copyright © 1994, Dr. Dobb's Journal

Back to Table of Contents


JUL94: ALGORITHM ALLEY

ALGORITHM ALLEY

Rendering Circles and Ellipses

Tim Kientzle

Tim, who has a PhD in mathematics from the University of California, Berkeley, is the author of numerous commercial, shareware, and public-domain programs for graphics and serial communication. Tim can be contacted at kientzle@netcom.com.

In the July 1990 issue of Dr. Dobb's Journal, Tim Paterson presented the article "Circles and the Digital Differential Analyzer." While Paterson's algorithm does not accumulate error, his explanation involves a method for solving the differential equation dy/dx=-x/y which does accumulate error.

In a subsequent letter to the editor ("Letters," DDJ, July 1991), V. Venkataraman pointed out that Paterson's algorithm plots points on or just within the ideal circle, and suggests the desirability of a method that plots the points closest to the circle, even if they are outside it. Venkataraman presented a minor change to Paterson's algorithm, but still failed to consistently choose the optimal point.

Based on this exchange, I developed a circle algorithm which satisfies both Paterson's interest in speed and Venkataraman's interest in plotting the closest points to the circle. The corresponding algorithm for drawing ellipses is faster than the line algorithm that inspired it.

Lining Up

The digital differential analyzer (DDA), also known as "Bresenham's algorithm," is based on a simple idea: Instead of picking points on the ideal line, you should pick points on the screen that are close to the line. To do this, increment one coordinate while updating the other so that you pick the integer closest to the line. Since the endpoints of the lines have integer coordinates, the slope is always rational, so you can obtain a fast, exact algorithm by using rational arithmetic.

For example, assume you're drawing a line from (0,0) to (3,2). Whenever you increment the x-coordinate, you need to increase the y-coordinate by 2/3. If you let the variable y hold the integral part of the y-coordinate and add a new variable to hold the numerator of the fractional part, you have the algorithm in Listing One .

The problem with this approach is that by simply using y to plot the point, you are truncating the exact value rather than rounding. Since rounding involves adding 1/2 and then truncating, in this algorithm it suffices to initialize the exact y-coordinate to 1/2 before you start, as in Listing Two .

Going in Circles

Drawing circles using the DDA algorithm requires more-involved geometry than drawing lines, but it uses the same approach: Step the x-coordinate through successive values and update the approximate y-coordinate based on some additional information. The trick is to figure out what that additional information should be.

You need the following simplifications: First, deal with points on the circle as "offsets" from the center of the circle (this is required by the algebra); and second, only compute the pixels in 1/8 of the circle, taking advantage of symmetry. Reflection to the remaining octants of the circle is handled by the routines in Listings Three and Four .

To get a first cut at a circle algorithm, you compute y from sqrt(r2-x2) (see Listing Five). Although it may not look like it, the algorithm in Listing Five is almost the same as Paterson's. The difference is that his approach replaced an expensive, repeated square-root calculation with a series of additions and subtractions. This is similar to the "strength reduction" performed by many optimizing compilers that replace an expensive operation within a loop with a series of less-expensive operations, usually adding a new variable in the process. Of course, strength reduction of a square-root operation is considerably beyond what can be handled automatically by today's compilers.

Then, reduce the square-root calculation to calculation of squares. In the octant you're computing (the part of the circle from the top to the top-left), the y-coordinate of the next point is always the same or one less than the previous y-coordinate. This means that you can simply decrement y whenever y2 > radius2 - x2, or equivalently, whenever x2+y2-radius2>0.

Next, remove the calculation of the squares. Keep the square of the radius in a temporary variable. You can then use the algebraic simplification (x+1)2 = x2+2x+1 to keep track of x2 and y2 as you change x and y. This involves adding new variables xSquared and ySquared, adding 2x+1 before incrementing x and subtracting 2y-1 from y2 before decrementing y.

Finally, because the only place you need all these squares is in the condition to decrement y, you can combine them into a single error variable kept equal to xSquared+ySquared-radiusSquared. The earlier condition then simplifies to error>0, which gives exactly the algorithm Paterson describes; see Listing Six . This explanation of the variable error should make clear why this algorithm works: Whenever the current y gives a point outside the circle (whenever x2+y2>radius2), you reduce y by 1 to remain within the circle. As Venkataraman pointed out, you'd get a smoother result if you could instead choose the points closest to the circle; that is, if you could only reduce y when the lower point would be closer to the circle.

To accomplish this, take a careful look at how the algorithm decides when to decrement y. You want a condition that will tell you whether (x,y) or (x,y-1) is closer to the circle. Algebraically, you want to decrement y whenever the inequality in Figure 1(a) is true. Changing the absolute values to squares gives something equivalent, but easier to simplify. The simplification is still tedious, but you eventually end up with something useful. The left side of Figure 1(b) is simply the value error from earlier attempts. The value in brackets is always positive and less than 1. This means that the right side of the inequality is less than y but greater than y-1. Since everything else is an integer, the inequality simplifies to error >= y.

Figure 1 (a) You need to decrement y whenever the inequality is true; (b) the left side is simply the value error from earlier attempts.

This results in Listing Seven , which is close to Venkataraman's suggestion, but always gives the point closest to the circle. To test this code, replace the plot8 routine with one that computes the radius of the points being plotted and the radius of points just above and below them. You'll see that the algorithm always picks the y-coordinate that results in a radius closest to the circle. You may be able to speed up Listing Seven slightly by changing error to be x2+y2-r2-y, which allows the condition to simply test error >= 0.

Handling Ellipses and Nonsquare Pixels

If you're only interested in drawing perfect circles on screens with square pixels, then Listing Seven serves the purpose quite nicely. However, if the pixels are not square, you have to draw an ellipse in order to get a circular result.

The biggest problem in drawing ellipses is that you lose the eight-fold symmetry used to simplify the circle routine. Where before you had eight identical octants, you now have two sets of octants: two each at the top and bottom of the circle that are identical to each other, and four at the left and right that are identical to each other. So, you need to run the earlier algorithm twice, once for each set.

The formula for the ellipse that fits within a rectangle of width 2a and height 2b is b2x2+a2y2=a2b2. Adjusting the calculation of error to match this gives us the algorithm in Listing Eight for drawing the ellipse. The least-obvious part of this is how to determine where each octant stops. Remember that the algorithm depends on y either staying the same or decreasing by 1. This means that each loop has to continue until you reach a place in the ellipse where the slope is 1 or -1. This happens exactly when b2x==a2y; see Listing Eight.

Clearly, a few more things could be done to speed up this routine. By adding variables to keep track of b2x and a2y, you could replace all of the multiplications with additions. You could also alter the definition of error as discussed earlier to simplify the condition test. Listing Nine incorporates those optimizations, plus a few others. In assembly language, you should be able to optimize this to a maximum of eight addition/subtraction/comparison operations per loop iteration, not counting the operations in the plot4 routine. Counting those, you get a maximum of 16 such operations for four points on the ellipse, or a mere four additions per point, which compares quite favorably to the three additions per point for the less-general circle routine developed earlier, and the six additions per point for the line algorithm. In fact, this shows that drawing the ellipse directly is faster than computing several points on the ellipse and using DDA lines to connect them!

Of course, it might be faster to undo some of these optimizations to reduce the number of variables required. By doing that, you may be able to fit all the variables in registers, thus speeding up the algorithm. As is usual for such low-level graphics operations, optimizing for the specific processor and video hardware is critical.

Listing One

void line_1(int x0, int y0, int x1, int y1 )
{
    int x = x0,y = y0;          /* Current point on line */
    int deltaX = x1-x0;         /* Change in x from x0 to x1 */
    int deltaY = y1-y0;         /* Change in y from y0 to y1 */
    int numerator = 0;          /* Numerator of fractional part of y */

    while ( x <= x1 )
    {
        plot( x,y );
        x += 1;

        numerator += deltaY;        /* Increase fractional part of y */
        if (numerator >= deltaX)    /* If fraction is 1 or more */
        {
            numerator -= deltaY;    /* Reduce fraction */
            y += 1;                 /* Increase whole part of y */
        }
    }
}

Listing Two

#define abs(x)  (((x)>=0)?(x):-(x))  /* Absolute value */
void line_2(int x0, int y0, int x1, int y1 )
{
    int x = x0,y = y0;
    int deltaX = 2*abs(x1-x0);
    int deltaY = 2*abs(y1-y0);
    int numerator = deltaX/2;      /* Initialize y-coordinate to 1/2 */
    while ( x <= x1 )
    {
        plot( x,y );
        x += 1;

        numerator += deltaY;
        if (numerator >= deltaX)
        {
            numerator -= deltaY;
            y += 1;
        }
    }
}

Listing Three

void plot4( int xOrigin, int yOrigin, int xOffset, int yOffset)
{
    plot( xOrigin + xOffset, yOrigin + yOffset );
    plot( xOrigin + xOffset, yOrigin - yOffset );
    plot( xOrigin - xOffset, yOrigin + yOffset );
    plot( xOrigin - xOffset, yOrigin - yOffset );
}

Listing Four

void plot8( int xOrigin, int yOrigin, int xOffset, int yOffset)
{
    plot4( xOrigin, yOrigin, xOffset, yOffset );
    plot4( xOrigin, yOrigin, yOffset, xOffset );
}

Listing Five

void circle_1(int xOrigin, int yOrigin, int radius)
{
   int x = 0;                           /* Exact x coordinate */
   int y = radius;                      /* Approximate y coordinate */

   while (x <= y)                       /* Just one octant */
   {
       plot8( xOrigin, yOrigin, x, y );
       x += 1;
       y = sqrt(radius*radius - x*x);
   }
}

Listing Six

void circle_2(int xOrigin, int yOrigin, int radius)
{
    int x = 0;                         /* Exact x coordinate */
    int y = radius;                    /* Approximate y coordinate */
    int error = 0;                     /* x^2 + y^2 - r^2 */

    while (x <= y)
    {
        plot8( xOrigin, yOrigin, x, y );

        error += 2 * x + 1;
        x += 1;

        if (error > 0)
        {
            error -= 2 * y - 1;

            y -= 1;
        }
    }
}

Listing Seven

void circle_3(int xOrigin, int yOrigin, int radius)
{
    int x = 0;                  /* Exact x coordinate */
    int y = radius;             /* Approximate y coordinate */
    int error = 0;              /* x^2 + y^2 - r^2 */

    while (x <= y)
    {
        plot8( xOrigin, yOrigin, x, y );

        error += 2 * x + 1;
        x += 1;

        if (error >= y)
        {
            error -= 2 * y - 1;
            y -= 1;
        }
    }
}

Listing Eight

void ellipse_1(int xOrigin, int yOrigin, int a, int b)
{
    {    /* Plot the octant from the top to the top-right */
        int x = 0;
        int y = b;
        int error = 0;/* b^2 x^2 + a^2 y^2 - a^2 b^2 */

        while (x * b *b <= y * a * a)
        {
            plot4( xOrigin, yOrigin, x, y );

            error += 2 * b*b * x + b*b;
            x += 1;

            if (error >= y * a*a)
            {
                error -= 2 * a*a * y - a*a;
                y -= 1;
            }
        }
    }

    {    /* Plot the octant from right to top-right */
        int x = a;
        int y = 0;
        int error = 0;/* b^2 x^2 + a^2 y^2 - a^2 b^2 */

        while (x * b * b > y * a * a)
        {
            plot4( xOrigin, yOrigin, x, y );

            error += 2 * a*a * y + a*a;
            y += 1;

            if (error >= x * b*b)
            {
                error -= 2 * b*b * x - b*b;
                x -= 1;
            }
        }
    }
}

Listing Nine

void ellipse_2(int xOrigin, int yOrigin, int a, int b)
{
    int aSquared = a*a;
    int bSquared = b*b;
    int twoASquared = 2 * aSquared;
    int twoBSquared = 2 * bSquared;

    {    /* Plot the octant from the top to the top-right */
        int x = 0;
        int y = b;
        int twoXTimesBSquared = 0;
        int twoYTimesASquared = y * twoASquared;

        int error = -y* aSquared;  /* b^2 x^2 + a^2 y^2 - a^2 b^2 - a^2y */

        while (twoXTimesBSquared <= twoYTimesASquared )
        {
            plot4( xOrigin, yOrigin, x, y );
            x += 1;
            twoXTimesBSquared += twoBSquared;
            error += twoXTimesBSquared - bSquared;
            if (error >= 0)
            {
                y -= 1;
                twoYTimesASquared -= twoASquared;
                error -= twoYTimesASquared;
            }
        }
    }
    {    /* Plot the octant from right to top-right */
        int x = a;
        int y = 0;
        int twoXTimesBSquared = x * twoBSquared;
        int twoYTimesASquared = 0;
        int error = -x* bSquared;  /* b^2 x^2 + a^2 y^2 - a^2 b^2 - b^2x */

        while (twoXTimesBSquared > twoYTimesASquared)
        {
            plot4( xOrigin, yOrigin, x, y );
            y += 1;
            twoYTimesASquared += twoASquared;
            error += twoYTimesASquared - aSquared;
            if (error >= 0)
            {
                x -= 1;
                twoXTimesBSquared -= twoBSquared;
                error -= twoXTimesBSquared;
            }
        }
    }
}

Copyright © 1994, Dr. Dobb's Journal

Back to Table of Contents


Figure 3

Copyright © 1994, Dr. Dobb's Journal


Figure 5

Copyright © 1994, Dr. Dobb's Journal


OCT94: ALGORITHM ALLEY

ALGORITHM ALLEY

Genetic Annealing

Kenneth V. Price

Ken holds a BS in physics from Rensselear Polytechnic Institute. He is currently engaged in research on artificial intelligence and system modeling. Ken can be reached at kprice@solano.community.net.

Introduction

by Bruce Schneier

How do you determine the best machine for a job?

Step #1. Express all the parameters of your machine using some kind of coding machine. Each particular machine will then be expressed as a list of these parameters.

Step #2. Generate some machines with random parameters.

Step #3. Test the machines against each other and select the few best ones.

Step #4. Generate new machines by combining parameters from the ones selected in Step #3, occasionally making random modifications in some of the parameters.

Step #5. Repeat Steps #3 and #4 until you're tired of watching the show.

What I've just described is a genetic algorithm, and it's done a pretty good job with life on this planet. Generation after generation of iterations, selecting the best few to reproduce and occasionally throwing in the odd mutation, has resulted in millions of different species. The end result is that there are lifeforms suited for every niche in every environment.

Genetic algorithms might not be the fastest way to generate the "best" machine for a job (it would be a lot easier to design a sentient lifeform from scratch than to wait for evolution to stumble across a human), but their main advantage is that they require no knowledge about the system. For instance, the traditional way of generating an aircraft wing is to study aerodynamics and spend months calculating airflows, lifts, stresses, and the like. Alternatively, you could just express a generic wing as a string of parameters (size, shape, weight, and so on), create some random wings, test them against each other, and let the fittest reproduce. A weekend of iteration on a supercomputer would probably yield a pretty good design.

Solving problems such as these falls into the category of "combinatorial" or "global" optimization. Other approaches to finding the "best" (according to some predefined criteria) configuration involve simulated annealing, a stochastic-search technique familiar to chip designers, who must determine the best geometrical arrangement for millions of circuits on microprocessors.

DDJ has examined both genetic algorithms and simulated annealing before (see "Genetic Algorithms" by Mike Morrow, April 1991 and "Simulated Annealing" by Michael P. McLaughlin, September 1989). In this month's column, however, Kenneth Price combines the two techniques, resulting in an approach called "genetic annealing" which takes the best from both the genetic and annealing worlds.

Genetic annealing is a hybrid, random-search technique that fuses simulated annealing and genetic methodologies into a more efficient algorithm. Like its genetic counterpart, the genetic-annealing algorithm creates new solutions by exchanging genetic material between members of a population. Instead of using a competitive tournament to decide which members will survive into the next generation, however, genetic annealing uses an adaptive, thermodynamic criterion based on a simple feedback scheme. With this strategy, the genetic-annealing algorithm can outperform the algorithms whose principles it embodies even though it is hardly more complex than running a suite of greedy algorithms in parallel.

Bipartitioning a Necklace

To evaluate the effectiveness of a random-search algorithm, it helps to use a problem whose global minimum is already known. To this end, I have chosen a simple graph-bipartitioning problem to illustrate how genetic annealing works. A graph is a collection of vertices connected by edges. Bipartitioning is the process of assigning each vertex to one of two sets. The goal of the bipartitioning problem is to assign half of a graph's vertices to each set so that the number of edges connecting the two sets is minimized. In many ways, this problem resembles the real-world task of the circuit designer, who wants to divide a set of components equally between two chips so that the number of lines needed to connect the chips is minimal.

The particular graph used in this example is commonly called a "necklace" because of its obvious resemblance to neckwear; see Figure 1(a). An [M,2] necklace is a ring of M vertices ("beads") each of which is connected radially to a companion vertex for a total of 2M vertices. As Figure 1(b) illustrates, any diameter across the necklace constitutes a bipartition since it divides the necklace into two sets, each of which contains M vertices. The bipartition generated by a diameter is optimal because only two edges cross the diameter to connect opposing sides. Because of its rotational symmetry, an [M,2] necklace has M optimal bipartitions in all.

For the purposes of computation, you can represent an [M,2] necklace as a string of 2M bits. Bits at even positions are the beads on the ring itself, and bits at odd positions represent the dangling beads; see Figure 2. The actual binary value assigned to a bead indicates the set to which it belongs.

To count the number of edges that span sets, perform an XOR operation on every pair of bits that represent connected vertices and sum the results. When the vertices at the ends of an edge belong to the same set, the bits they contain will be identical, and the XOR operation will return a 0. Conversely, a pair of connected vertices with different binary values represents an edge with a vertex in each set. The XOR operation counts these spans by returning a 1.

To simulate a heat bath, you will need an entire population of bit strings. The number of bit strings depends on the problem: for this example a population of 20 strings is adequate. For convenience, you can arrange the bit strings into a two-dimensional array so that one coordinate locates a string within the population while the other coordinate gives you the position of a bit within a string.

Next, you initialize the bit strings by filling each one with an equal number of 1s and 0s. In a genetic-annealing experiment, you must randomly select the initial population to ensure that the system starts out like a "white-hot" thermodynamic ensemble. Starting from this condition makes it less likely that the population will prematurely cool to a suboptimal minimum. To randomize a bit string while keeping its populations of 1s and 0s equal, try swapping each bit with another bit from a randomly selected location within the same string.

The number of edges having a vertex in each set measures the quality of a bipartition as a solution. In the context of a genetic algorithm, this number reflects the "fitness" of a bit string to survive as a "gene" in a competitive selection procedure. By contrast, simulated annealing uses the fitness of a configuration-like energy in order to exploit the laws of statistical mechanics. Genetic annealing adopts the annealing metaphor, treating a configuration's fitness-like energy, even though the annealing process itself is driven by population dynamics.

Acceptance Criteria

Like its component algorithms, genetic annealing is a random-search technique. This class of algorithm tries to improve an existing configuration by subjecting it to trial mutations. A mutation is any procedure that alters either the structure of a configuration or the data that it holds. One of the keys to a successful random search is knowing when a mutation produces an acceptable improvement. For example, under the "greedy" criterion, a mutant is deemed acceptable whenever its energy is less than or equal to the energy of the configuration from which it was derived. If it is accepted, the mutant supplants its progenitor and becomes the target for subsequent mutations. This way, each improved mutant lowers the acceptability criterion until configurations with lower energies fail to turn up.

In the genetic-annealing approach, you assign an energy threshold to each bit string. Initially, each threshold equals the energy of the randomized bit string to which it is assigned. Unlike the greedy criterion, a bit string's threshold, not its energy, determines which trial mutations constitute acceptable improvements. If the energy of a mutant exceeds the threshold of the bit string that spawned it, you reject the mutant and move on to the next bit string. However, if its energy is less than or equal to the threshold, you accept the mutant as a replacement for its progenitor.

While the greedy algorithm accepts a better configuration without regard to how much better it is, the genetic-annealing algorithm uses this information to drive the annealing process. Genetic annealing uses an "energy bank," represented by the real variable DE, to keep track of the energy liberated by successful mutants. Whenever a mutant passes the threshold test, you add the difference between the threshold and the mutant's energy to DE for temporary storage. Once you account for this quantum of heat, you reset the threshold so that it equals the energy of the accepted mutant and then move on to the next bit string.

Reheating

After each bit string has been subjected to a random mutation, you "reheat" the population by raising each threshold a little. The size of the increase depends both on the amount of energy accumulated in the energy bank and on the rate at which you want to cool the population. If you let N equal the number of bit strings in the population, then the average contribution to the energy bank is just DE/N. To fully reheat the population, all you need to do is add DE/N to each threshold. Annealing results from repeated cycles of collecting energy from successful mutants (spontaneous cooling) and then redistributing nearly all of it by raising the threshold energy of each population member equally (uniform reheating).

After they have been reheated, thresholds are higher than the energies of the bit strings to which they have been assigned. This means that sometimes you are forced to accept a mutant even though its energy is not as low as the energy of the bit string it replaces. Replacing a bit string with a worse one may seem counter-productive, but these occasional reversals of fortune provide floundering bit strings with a helpful energy boost. In essence, the entire population acts like a giant heat reservoir that exchanges energy among its members. Less successful bit strings can escape suboptimal configurations by borrowing the energy they need from the more successful versions.

The Cooling Constant

To relax the bit strings into their optimal condition, they must be cooled very slowly. In a genetic-annealing program, you control the rate of cooling with the cooling constant, C--a real number in the closed interval [0,1] that represents the fraction of DE that is returned to the population. For example, C=1 holds the population at a constant "temperature" by using 100 percent of the energy stored in DE to reheat thresholds. By contrast, C=0 releases all of DE's stored energy from the system and leaves thresholds unaltered. In effect, C=0 sets up a suite of greedy algorithms since each threshold is always equal to the energy of the bit string to which it's assigned. Most problems of consequence require very slow cooling. Typically, C ranges from 0.9 to 0.99 and beyond, although for a given problem, the optimal choice of C depends on a variety of factors. Pseudo-code for the genetic-annealing algorithm is shown in Figure 4.

You can also use the genetic-annealing algorithm to "cool" a population of configurations to a condition of maximum energy. All you need to do is replace the "greater than" symbol with a "less than" symbol in the portion of code that determines whether or not a mutant is acceptable. This has the effect of reversing the sign of the spontaneous energy so that during the reheating cycle each threshold is lowered by the amount dE=C*DE/N. This feature lets you use a problem's natural measure of fitness without having to invert it. Figure 3 shows what the maximum energy solution to the necklace problem looks like for M=8.

This energy bank approach to simulated annealing can substantially reduce the time you need to design and execute difficult combinatorial optimizations. Traditional annealing methods invoke the venerable Metropolis algorithm to forge a link between the laws of statistical mechanics and combinatorial optimization. Despite its great utility, the Metropolis algorithm is computationally expensive because the decision of whether or not to accept a mutant usually requires that you generate a random number and compare it to an exponential term. Configurations that are not much worse than their progenitors may be rejected depending on the random number generated. The genetic-annealing algorithm accepts every configuration that is not much worse than its progenitor based on the result of a simple comparison that requires neither a random-number generator nor the use of acceptance probabilities.

Traditional annealing methods also rely on an empirically derived "annealing schedule" to control the rate at which the Metropolis temperature should be lowered. The advantage of using thresholds to track the time-averaged loss of spontaneous energy from individual configurations is that you can maintain equilibrium at any temperature without using an annealing schedule just by restoring energy to each threshold at the same average rate that the ensemble loses it. In most cases, this reduces the researcher's challenge to finding the smallest value of C that will allow the population to maintain equilibrium as it anneals.

Mutations

Choosing the right mutation scheme is just as important to the success of a random search as determining which mutants are acceptable. A mutation can be an elementary operation like flipping a bit, or it can be a more complex procedure like a symmetry operation or crossbreeding. In general, each problem will have its own menu of mutations. Frequently, mutations are subject to constraints based on a problem's symmetries. For example, mutations used in the bipartitioning problem must preserve the equal number of 1s and 0s in each bit string.

Of all the ways you can alter a bit string without changing the number of 1s (or 0s) that it contains, swapping bits is perhaps the easiest. To swap a pair of bits, just randomly select two bits with different binary values and replace each with its one's complement.

While it may be the simplest form of mutation, swapping a single pair of bits provides you with only a limited search capability. You can explore remote regions of solution space more effectively if you occasionally swap more than one pair of bits at a time. For example, you can usually enhance the efficiency of a random search by drawing the number of elements involved in a mutation from an exponential distribution. You can characterize an exponential distribution by its decay constant, EX, where EX is a real number in the half-open interval [0,1). In the case of swapping bits, EX=.5 means that a single pair of bits is swapped half of the time, two pairs of bits are swapped one quarter of the time, three pairs of bits are swapped one eighth of the time, and so on. EX=0 means that you never swap more than one pair of bits. With the right decay constant, an exponential distribution of mutation sizes makes distant points in solution space more accessible without sacrificing the ability to efficiently fine tune a configuration.

Symmetry Operations as Mutations

In most problems, a well-chosen symmetry operation can make a valuable addition to your mutation scheme. When appropriate, operations like rotations and reflections can generate large-scale variations of a configuration without des-troying crucial local relationships. For example, a reflection operation in which the first four bits of the sequence: 0000111100001111 are exchanged with the second four bits, produces the optimal configuration: 0000000011111111. Despite altering the target string on a relatively large scale, the reflection procedure not only conserves the number of 1s, but also maintains the XOR relationships between the bits within the reflected substring. To accomplish the same transformation in a series of random swap operations would require considerable good fortune. Despite their utility, symmetry mutations reduce the novelty of a "random" search, so you should not rely on them exclusively.

Splicing

Perhaps the greatest advantage that cooling a population in parallel affords you is the opportunity to employ crossbreeding as a form of mutation. Actually, instead of mating pairs of bit strings in a separate cross-breeding procedure (as you do in the genetic algorithm), the genetic-annealing algorithm transfers genetic information between bit strings by "splicing" it. Splicing tentatively replaces a portion of the target string with the corresponding section from another bit string chosen at random from the population. In the splicing scenario, the randomly chosen string remains unaltered while it donates a copy of a segment of its "genetic material" for the target string to use as a trial mutation; see Figure 5. By drawing upon the success of other configurations in the population, splicing brings to the genetic-annealing algorithm all of the problem-solving power that crossbreeding imparts to the genetic algorithm.

You must be careful when implementing a splicing procedure for the bipartitioning problem because the number of 1s in the substring being donated must be the same as the number of 1s in the target substring. To ensure that 1s are conserved, the size of the donated substring is allowed to grow until the number of 1s it contains equals the number of 1s in the corresponding target substring and until the two substrings differ in at least one bit position, or until the whole string is used.

The mutation scheme in GENNEAL.C (described later) includes all three forms of mutation: random bit swapping, a reflection operation, and splicing. PS, PR, and PX are the probabilities that a mutation will swap random bits, reflect a substring, or splice a substring, respectively. Of course, PS+PR+PX=1. The size of a swap mutation is controlled by EXS, while EXR determines the size of a reflection mutation.

Putting it All Together

Along with the mutation probabilities and their respective decay constants, other control variables include the population size, N, and the cooling constant, C. Given a necklace of size M (for a minimum energy of 2, make M even), what combination of these variables will repeatedly produce an optimal partition with a minimum of computational effort?

Since run times on a sequential computer are proportional to population size, you will usually want to reduce N until further reductions begin to jeopardize the population's ability to simulate a heat bath. A population containing anywhere from 10 to 40 configurations should be sufficient for most problems. Smaller populations hamper the annealing process by failing to provide energy when it is needed, while larger populations increase execution times while enhancing the thermal simulation only incrementally.

The robust flexibility of the genetic-annealing algorithm makes it easy to experiment with a variety of computational approaches to find the one that works best. You can transform the genetic-annealing algorithm into a greedy algorithm, a suite of annealing programs, or a genetic-style algorithm simply by changing a few control constants. When you approach a new problem, try starting with N=20 and run a suite of greedy algorithms in parallel by setting C=0 and PX=0 (no annealing or splicing). These results will provide you with a convenient performance benchmark.

Results

Although it occasionally turns up an optimal configuration, the greedy algorithm performs poorly because it is not consistently successful. The true performance of the greedy algorithm, like any stochastic search, is more reliably gauged by the average and variance of an ensemble of results.

In a series of ten trials with N=20 and M=80, the greedy version of the genetic-annealing algorithm (in which C, EXS, PR, and PX are all zero), produced an average minimum energy of 6.62 and a variance of 4.63. In each trial, the program was allowed to run for 256,000 generations. When each mutation swapped only one pair of bits (EXS=0), just four out of the total of 200 configurations (10 trialsx20 strings per trial) reached the optimal energy of 2. Increasing EXS produced a modest improvement. A series of ten trials that used EXS=.35 turned up six optimal bit strings and dropped the average final energy to 6.37. For the sake of simplicity, the random-swap procedure uses a constant value of EXS=.35 throughout the remainder of this demonstration.

As expected, results improved significantly once the population was annealed. With N and C both greater than 0 and no splicing, this version of the genetic-annealing algorithm resembles a suite of annealing algorithms running in parallel. Initial trials, which used C=.84, drove the average energy down to 4.83, reduced the variance to 2.61, and produced 21 optimal bit strings. The next set of trials used C=.92, which cooled the population at about half the rate that C=.84 did, because annealing times are roughly proportional to 1/(1--C). With C=.92, the average final energy dropped to 4.51, and 28 bit strings found their way to an optimal state. Cutting the cooling rate in half again gave still better results, but the returns were clearly diminishing. C=.96 lowered the average final energy to 4.43 and produced a total of 38 optimal bit strings. When the cooling rate was halved yet again to C=.98, the average final energy rose to 4.52, indicating that cooling was so slow that even 256,000 generations were not enough to completely quench the population.

In the absence of the reflection mutation, schemes that used only a minuscule fraction of splicing performed best. If splicing is used too frequently, the population will misconverge. Annealing, however, can overcome misconvergence. For example, the "greedy-genetic" scheme: C=0, PX=.0002, PS=.9998 produced optimal results in only five of ten trials with convergence occurring at about 100,000 generations. Annealing with C=.84 took a little longer but misconvergence abated. By 120,000 generations, all 200 bit strings had found an optimal state.

The real surprise comes when you use a scheme that uses both long reflections and splicing. In particular, the scheme EXR=.95, PR=.3, PX=.7 not only produced perfect results in ten out of ten trials without annealing, but also converged on average in only 280 generations! This level of performance is somewhat atypical, due primarily to the reflection mutation's ability to speed equilibration across a bit string when large values of EXR are used. Some annealing is needed when you bipartition larger necklaces and/or use a less effective mutation scheme.

Computational Synergy

In the genetic-annealing approach, a synergy exists between splicing and annealing. Annealing controls the rate of convergence, tolerates error, and can endure high rates of random mutation. All these factors help alleviate the tendency of the genetic algorithm to misconverge. Furthermore, splicing genetic material empowers the annealing algorithm by providing the means to efficiently compare competing bit strings that are widely separated in solution space. Run times benefit from this synthesis, since you can exploit the superior search capabilities of splicing without having to maintain a large population to ensure genetic diversity. Add to this the increase in computational speed made possible by using thresholds instead of the Metropolis algorithm, and you can improve run times substantially when compared to traditional genetic and annealing techniques. Of course, you can also improve performance by running the genetic-annealing algorithm on a parallel computer, especially one using the SIMD architecture, since each configuration can then be assigned its own processor.

The GENNEAL.C Program

GENNEAL.C is a bipartitioning program available electronically; see "Availability," page 3. I've kept code simple for the sake of clarity and as a result, several routines have not been optimized. For instance, GENNEAL.C computes the energy of a mutant by performing an XOR operation on every pair of connected vertices, even though most mutations affect only a small part of the target string. It is faster to compute the energy of a mutant by reevaluating just those links of the target string that have been altered. Similarly, if a mutation fails, you need to restore only that part of the target string affected by the mutation. GENNEAL.C copies the whole bit string and then restores it in its entirety, if necessary.

Conclusion

You should now have enough information about how genetic annealing works to experiment on your own. If you currently use either a traditional genetic or simulated annealing algorithm, give genetic annealing a try. I would enjoy hearing from those of you who do the comparison.

I would also like to acknowledge Margaret E. Burwell for her valuable assistance in the preparation of this article.

Figure 1 (a) The "necklace" graph; (b) any diameter across the necklace constitutes an optimal bipartition.

Figure 2 Bits at even positions are the beads on the ring itself; bits at odd positions represent the dangling beads.

Figure 3 Maximum-energy solution to the necklace problem for M=8.

Figure 4:The genetic-annealing algorithm (minimization version).

 1. Randomly select an initial population of N configurations.
 2. For i=1 to N: Initialize the ith threshold,
    Th[i], with the energy of the ith configuration.
 3. DE=0                       /* Empty the energy bank */.
 4. For i=1 to N:              /* Begin cooling loop */.
 5. Splice or randomly mutate the ith configuration.
 6. Compute the energy, E, of the resulting mutant.
 7. If E>Th[i] then restore the old configuration.
 8. If E <= Th[i] then:
    a.DE=DE+Th[i]--E   /* Add energy difference to DE */.
    b.Th[i]=E                  /* Reset threshold */.
    c. Replace old configuration with successful mutant.
 9. Mutate next configuration or end cooling loop.
10.dE=DE*C/N                   /* Compute reheating increment, dE. 0<C<1 */
11.For i=1 to N:               /* Begin reheating loop */.
12.Th[i]+dE                    /* Add dE to each threshold */.
13.Return to step 3 once all thresholds have been reheated.

Figure 5 The splicing scenario.

Copyright © 1994, Dr. Dobb's Journal

Back to Table of Contents


NOV94: ALGORITHM ALLEY

ALGORITHM ALLEY

Truly Random Numbers

Colin Plumb

Colin, a student from Toronto, was introduced to modern cryptography by the Pretty Good Privacy package. He can be contacted at colin@nyx.cs.du.edu.

Introduction

by Bruce Schneier

Why do we need random numbers? There are a lot of reasons. You might be designing a communications protocol and need random timing parameters to prevent system lockups. You might be conducting a massive Monte Carlo simulation and need random numbers for various parameters. Or you might be designing a computer game and need random numbers to determine the results of different actions.

As common as they are, random numbers can be infuriatingly hard to generate on a computer. The very nature of a computer--a deterministic, digital Turing machine--is contrary to the notion of randomness.

One application where random numbers are essential is in cryptography. The security of a cryptographic system often hinges on the randomness of its keys. In this month's "Algorithm Alley," Colin Plumb discusses the random-number generator in the Pretty Good Privacy (PGP) e-mail security program. Colin is one of the designers and programmers of PGP, and has spent a lot of time thinking about this problem. His solution is elegant, efficient, effective, and has applications well beyond e-mail security.

The ANSI C rand() function does not return random numbers. This is not a bug; it's required by the ANSI C standard. Instead, the values returned are determined by the seed supplied to srand(). If you run the same program with the same seed, you get the same "random" numbers. The pattern may not be obvious to the casual observer, but if Las Vegas ran this way, there'd be fewer bright lights in the big city.

John von Neumann once said that "anyone who considers arithmetical methods of producing random digits is, of course, in a state of sin." Sometimes you want truly random numbers. Any number of security and cryptographic applications require them. When it comes to checking for viruses, for example, CRCs are convenient and fast, but you can easily fake out a known polynomial. The Strongbox secure loader from Carnegie Mellon University, however, uses a random polynomial to achieve security while keeping the speed advantages of a CRC (see Camelot and Avalon: A Distributed Transaction Facility, edited by Jeffrey L. Eppinger).

There are ways to produce random bits in hardware; sampling a quantum effect such as radioactive decay, for instance. However, this is hard to calibrate and involves special hardware.

On the other hand, software in a properly working computer is deterministic--the antithesis of random. Still, a computer generally has to interact and receive input from real-world events, so it is possible to make use of a very unpredictable part of most computer systems: the person typing at the keyboard.

Although keystrokes are somewhat random, compression utilities illustrate just how predictable most text is. While it would be foolish to ignore this entropy, anticipating what someone types is akin to password guessing: difficult, but if you have the computational horsepower, conceivable.

A more fruitful source is timing. Many computers can time events down to the microsecond. And while typing patterns on familiar words or phrases are repeatable enough to be used for identification, there is still a large window of available noise. Our basic source for entropy comes from sampling system timers on every keystroke.

The problem that remains is to turn these timer values, which have a nonuniform distribution, into uniformly distributed random bits. This is where the software comes in.

Theory of Operation

The file randpool.c (see Listing Four) uses cryptographic techniques to "distill" the essential randomness from an arbitrary amount of sort-of-random seed material. As the file name suggests, the program maintains a pool of hopefully random bits, into which additional information is "stirred." The goal here is that if you have n bits of entropy in the pool ("Shannon information," if you're familiar with information theory), any n bits of the output are truly random.

The stirring operation (actually nothing more than an encryption pass) is central. If you know the key and initial vector, you can reverse it to get back the initial state. Since the encryption is reversible, stirring the pool obviously does not lose information. So all the information in the initial state is there, it's just masked by the encryption.

Since we don't need to reverse the encryption, the key is then destroyed. The information is then reinitialized with data taken from the pool that was just stirred. This makes it essentially impossible to determine the previous state of the random-number pool from what is left in memory.

The cipher is Peter Gutmann's Message Digest Cipher using MD5 as a base. This is fast and simple (as strong ciphers go), especially on 32-bit machines. In this application, the large key size also helps efficiency. (For another application of this cipher, see Gutmann's shareware MS-DOS disk encryptor, "SFS." Every commercial MS-DOS disk encryptor I've seen--Norton Diskreet, for example--has appalling cryptography. Their only advantage is that you can get the data back with a few weeks' work if you lose the key. If you lose the key with SFS, it's lost.)

The output of the generator is taken from the pool, starting after the 64 bytes used for the next stirring key. If you reach the end of the pool, stir again and restart. After that, it is theoretically possible to examine the output and determine the key, which would reveal the complete state of the generator and let you predict its output forever. That, however, would require breaking the cipher by deriving the key from the data before and after encryption, an adequate guarantee of security.

Input is more interesting. To ensure that each bit of seed material affects the entire pool, the seed material is added (using XOR) to the key buffer. When you reach the end of the key buffer, stir the pool and start over. The difficulty of cryptanalysis (deriving the key from the prior and following states of the pool) ensures that regularities in the seed material do not produce regularities in the pool.

Adding bytes to the key sets the take position to the end of the pool, so the newly added data will be stirred in before any bytes are returned. Thus, you can add and remove bytes in any order.

The code mostly works with bytes, but since MD5 works with 32-bit words, a standard byte ordering is used. This way you can use it as a pseudorandom number generator seeded with a passphrase.

If you want to use the hash directly, md5.c (see Listing Two) includes a full implementation of the MD5 algorithm. (It is similar to the hash presented in "SHA: The Secure Hash Algorithm," by William Stallings, DDJ, April 1994.) If you have a large amount of low-grade seed material, you can use MD5 to pre-reduce it. For example, you can feed mouse-position reports into MD5, then periodically add the resultant 16-byte digest to the pool. Even faster algorithms are possible--based on CRCs and scrambler polynomials--if you have real-time constraints.

Practice of Operation

The file noise.c (see Listing Three) samples a variety of system timers and adds them to the random-number pool. It also returns the number of highest-resolution ticks since the previous call, which you can use to estimate the entropy of this sample. On an IBM PC, only 16 bits are returned; this underestimates the result if the calls are more than 1/18.2 seconds apart, but that is not a security problem.

The code also works under UNIX. You may have to find the frequency of a timer that only returns ticks; noiseTickSize() finds the resolution of a timer (the gettimeofday() function) that only returns seconds.

The main driver is in randtest.c (see Listing One). A flash effect is provided by funnyprint(). Of more value is randRange(), which illustrates a way to generate uniformly distributed random numbers in a range not provided by the generator. The problem is akin to generating numbers from 1 to 5 using a six-sided die. The solution amounts to rerolling if you get a 6.

The most interesting part is rand-Accum(), which accumulates a specified amount of entropy from the keyboard. It uses the number of ticks returned by the noise() function to estimate the entropy. It assumes that inter-keystroke times vary pretty uniformly over a range of 15 percent or so. Thus, it divides the tick count by 6 to get the fraction of the interval that is random, then takes the logarithm to get the number of bits of entropy.

The integer number of bits comes from normalizing the number and counting the shifts. The entropy is kept to four fractional bits using a few iterations of an integer-logarithm algorithm.

Weaknesses

I don't know of any exploitable holes in this approach to generating random numbers, but in cryptography, only a fool is sure he has a good algorithm. I believe the following points need further examination:

  • The divide-by-six approximation in randAccum(). This was chosen so a machine with only a 60-Hz clock would produce at least one bit per keystroke; not a very good reason. A much better technique is suggested by Ueli Maurer's paper from Crypto '90, "A Universal Statistical Test For Random Bit Generators." However, this technique is slow to decide that the input is trustworthy and requires large tables.
  • The "leakage" rate of information from the pool. Because the stirring key is drawn from the pool itself, collisions are possible. These are states of the pool which, after stirring, result in the same output state. This reduces the information content of the pool.
  • The use of MD5 as a cipher. If you are using this as a cryptographic PRNG and producing large amounts of output from a smaller seed, the cipher at the heart of the stirring may be broken. The amount of known plaintext available from any given stirring is quite low (a few hundred bytes), the key space is dauntingly large (512 bits), and no such attacks on MD5 have appeared in the civilian literature; however, MD5 was not designed for use as a cipher and has had less study in this mode.

References

Eppinger, Jeffrey L., Lily B. Mummert, and Alfred Z. Spector, eds. Camelot and Avalon: A Distributed Transaction Facility. San Mateo, CA: Morgan Kaufman, 1991.

Maurer, Ueli M. "A Universal Statistical Test for Random Bit Generators," in Advances in Cryptology--Crypto '90. Berlin: Springer-Verlag, 1991.

Davis, Donald T., Ross Ihaka, and Philip Fenstermacher. "Cryptographic Randomness From Air Turbulence in Disk Drives," in Advances in Cryptology--Crypto '94. Berlin: Springer-Verlag, 1994.

Knuth, Donald E. The Art of Computer Programming, Volume 2: Seminumerical Algorithms. Reading, MA: Addison-Wesley, 1981.

Listing One

/* usuals.h -- Useful typedefs */
#ifndef USUALS_H
#define USUALS_H
#include <limits.h>
#if UCHAR_MAX == 0xFF
typedef unsigned char byte; /* 8-bit byte */
#else
#error No 8-bit type found
#endif
#if UINT_MAX == 0xFFFFFFFF
typedef unsigned int word32;    /* 32-bit word */
#elif ULONG_MAX == 0xFFFFFFFF
typedef unsigned long word32;
#else
#error No 32-bit type found
#endif
#endif /* USUALS_H */

/* randtest.c -- Test application for the random-number routines */
#include <stdio.h>
#include <string.h>
#include <stdlib.h> /* For rand(), srand() and RAND_MAX */
#include "randpool.h"
#include "noise.h"

/* This function returns pseudo-random numbers uniformly from 0..range-1. */
static unsigned
randRange(unsigned range)
{
    unsigned result, div = ((unsigned)RAND_MAX+1)/range;
    while ((result = rand()/div) == range)
        /* retry */ ;
    return result;
}
/* Cute Wargames-like random effect thrown in for fun */
static void
funnyprint(char const *string)
{
    static const char alphabet[] =
"ABCDEFGHIJKLMNOPWRSTUVWXYZabcdefghijklmnopqrstuvwxyz1234567890";
    char c, flag[80] = {0}; /* 80 is maximum line length */
    unsigned i, tumbling = 0, len = strlen(string);
    /* We don't need good random numbers, so just use a good seed */
    randPoolGetBytes((byte *)i, sizeof(i));
    srand(i);   /* Sometimes simple PRNGs are useful! */
    /* Truncate longer strings (unless you have a better idea) */
    if (len > sizeof(flag))
        len = sizeof(flag);
    /* Count letters that we can tumble (letters in the alphabet) */
    for (i = 0; i < len; i++) {
        if (strchr(alphabet, string[i])) {
            flag[i] = 1;    /* Increase this for more tumbling */
            tumbling++;
        }
    }
    /* Print until all characters are stable. */
    do {
        putchar('\r');
        for (i = 0; i < len; i++) {
            if (flag[i]) {
                c = alphabet[randRange(sizeof(alphabet)-1)];
                if (c == string[i] && --flag[i] == 0)
                    tumbling--;
            } else {
                c = string[i];
            }
            putchar(c);
        }
        fflush(stdout);
    } while (tumbling);
    putchar('\n');
}
#include <conio.h>  /* For getch() */
#define FRACBITS 4  /* We count in 1/16 of a bit increments. */
/* Gather entropy from keyboard timing.  This is currently MS-DOS specific. */
static void
randAccum(int bits)
{
    word32 delta;
    int c, oldc = 0, olderc = 0;
    if (bits > RANDPOOLBITS)
        bits = RANDPOOLBITS;
    bits <<= FRACBITS;
    puts("We are generating some truly random bits by timing your\n"
         "keystrokes. Please type until the counter reaches 0.\n");
    while (bits > 0) {
        printf("\r%4d ", (bits-1 >> FRACBITS) + 1);
        c = getch();
        delta = noise()/6;  /* Add time of keystroke */
        if (c == 0)
            c = 0x100 + getch();    /* Handle function keys */
        randPoolAddBytes((byte const *)&c, sizeof(c));
        /* Normal typing has double letters, but discard triples */
        if (c == oldc && c == olderc)
            continue;
        olderc = oldc;
        oldc = c;
        if (delta) {    /* Subtract log2(delta) from bits */
            /* Integer bits first, normalizing */
            bits -= 31<<FRACBITS;
            while (delta < 1ul<<31) {
                bits += 1<<FRACBITS;
                delta <<= 1;
            }
            /* Fractional bits, using integer log algorithm */
            for (c = 1 << FRACBITS-1; c; c >>= 1) {
                delta >>= 16;
                delta *= delta;
                if (delta >= 1ul<<31)
                    bits -= c;
                else
                    delta <<= 1;
            }
        }
    }
    puts("\r    0 Thank you, that's enough.");
}
/* When invoked with the argument "foo", this should start with:
 * Adding "foo\0" to pool. Pseudo-random bytes:
 * 4c 9d 41 ba 44 41 63 a1 db 1c ab 3f 52 a1 a2 84 c3 e5 dc bc 57 4c d9 f3 38
 * d7 45 50 f9 94 36 96 a3 df 90 ff 23 e5 ec 3c 76 1f ce 1c bc d6 79 8b 5e e7
 * aa 97 16 c0 50 c6 95 0b c1 62 42 e5 5b 8f d7 bd d7 70 1f c6 60 6a 5f f3 74
 * 8d 35 ad 51 5a 4a 0c 02 cd d5 36 7e d4 c2 d9 f0 d3 49 ed 2d fa 4e 2b 70 3f
 */
int
main(int argc, char **argv)
{
    int i;
    while (--argc) {
        printf("Adding \"%s\\0\" to the pool.\n", *++argv);
        randPoolAddBytes((byte const *)*argv, strlen(*argv)+1);
    }
    puts("\nPseudo-random bytes:");
    i = 100;
    while (i--)
        printf("%02x%c", randPoolGetByte(), i % 25 ? ' ' : '\n');
    putchar('\n');
    funnyprint("This will be deterministic on a given system.");
    putchar('\n');
    noise();    /* Establish a baseline for the deltas */
    randAccum(800); /* 800 random bits = 100 random bytes */
    puts("\nTruly random bytes:");
    i = 100;
    while (i--)
        printf("%02x%c", randPoolGetByte(), i % 25 ? ' ' : '\n');
    putchar('\n');
    funnyprint("This will be unpredictable.");
    return 0;
}

Listing Two

/* md5.h -- declarations for md5.c */
#ifndef MD5_H
#define MD5_H
#include "usuals.h"
struct MD5Context {
    word32 hash[4];
    word32 bytes[2];
    word32 input[16];
};
void byteSwap(word32 *buf, unsigned words);
void MD5Init(struct MD5Context *context);
void MD5Update(struct MD5Context *context, byte const *buf, unsigned len);
void MD5Final(byte digest[16], struct MD5Context *context);
void MD5Transform(word32 hash[4], word32 const input[16]);
#endif /* !MD5_H */

/* md5.c -- An implementation of Ron Rivest's MD5 message-digest algorithm.
 * Written by Colin Plumb in 1993, no copyright is claimed. This code is in the
 * public domain; do with it what you wish. Equivalent code is available from
 * RSA Data Security, Inc.  This code does not oblige you to include legal
 * boilerplate in the documentation. To compute the message digest of a string
 * of bytes, declare an MD5Context structure, pass it to MD5Init, call
 * MD5Update as needed on buffers full of bytes, and then call MD5Final, which
 * will fill a supplied 16-byte array with the digest.
 */
#include <string.h> /* for memcpy() */
#include "md5.h"

/* Byte-swap an array of words to little-endian. (Byte-sex independent) */
void
byteSwap(word32 *buf, unsigned words)
{
    byte *p = (byte *)buf;
    do {
        *buf++ = (word32)((unsigned)p[3]<<8 | p[2]) << 16 |
                 ((unsigned)p[1]<<8 | p[0]);
        p += 4;
    } while (--words);
}
/* Start MD5 accumulation. */
void
MD5Init(struct MD5Context *ctx)
{
    ctx->hash[0] = 0x67452301;  ctx->hash[1] = 0xefcdab89;
    ctx->hash[2] = 0x98badcfe;  ctx->hash[3] = 0x10325476;
    ctx->bytes[1] = ctx->bytes[0] = 0;
}
/* Update ctx to reflect the addition of another buffer full of bytes. */
void
MD5Update(struct MD5Context *ctx, byte const *buf, unsigned len)
{
    word32 t = ctx->bytes[0];
    if ((ctx->bytes[0] = t + len) < t)  /* Update 64-bit byte count */
        ctx->bytes[1]++;    /* Carry from low to high */
    t = 64 - (t & 0x3f);    /* Bytes available in ctx->input (>= 1) */
    if (t > len) {
        memcpy((byte *)ctx->input+64-t, buf, len);
        return;
    }
    /* First chunk is an odd size */
    memcpy((byte *)ctx->input+64-t, buf, t);
    byteSwap(ctx->input, 16);
    MD5Transform(ctx->hash, ctx->input);
    buf += t;
    len -= t;
    /* Process data in 64-byte chunks */
    while (len >= 64) {
        memcpy(ctx->input, buf, 64);
        byteSwap(ctx->input, 16);
        MD5Transform(ctx->hash, ctx->input);
        buf += 64;
        len -= 64;
    }
    /* Buffer any remaining bytes of data */
    memcpy(ctx->input, buf, len);
}
/* Final wrapup - pad to 64-byte boundary with the bit pattern
 * 1 0* (64-bit count of bits processed, LSB-first) */
void
MD5Final(byte digest[16], struct MD5Context *ctx)
{
    int count = ctx->bytes[0] & 0x3F;   /* Bytes mod 64 */
    byte *p = (byte *)ctx->input + count;
    /* Set the first byte of padding to 0x80.  There is always room. */
    *p++ = 0x80;
    /* Bytes of zero padding needed to make 56 bytes (-8..55) */
    count = 56 - 1 - count;
    if (count < 0) {    /* Padding forces an extra block */
        memset(p, 0, count+8);
        byteSwap(ctx->input, 16);
        MD5Transform(ctx->hash, ctx->input);
        p = (byte *)ctx->input;
        count = 56;
    }
    memset(p, 0, count);
    byteSwap(ctx->input, 14);
    /* Append 8 bytes of length in *bits* and transform */
    ctx->input[14] = ctx->bytes[0] << 3;
    ctx->input[15] = ctx->bytes[1] << 3 | ctx->bytes[0] >> 29;
    MD5Transform(ctx->hash, ctx->input);
    byteSwap(ctx->hash, 4);
    memcpy(digest, ctx->hash, 16);
    memset(ctx, 0, sizeof(*ctx));   /* In case it's sensitive */
}
/* The four core functions */
#define F1(x, y, z) (z ^ (x & (y ^ z)))
#define F2(x, y, z) F1(z, x, y)
#define F3(x, y, z) (x ^ y ^ z)
#define F4(x, y, z) (y ^ (x | ~z))
/* This is the central step in the MD5 algorithm. */
#define MD5STEP(f,w,x,y,z,in,s) (w += f(x,y,z)+in, w = (w<<s | w>>32-s) + x)
/* The heart of the MD5 algorithm. */
void
MD5Transform(word32 hash[4], word32 const input[16])
{
    register word32 a = hash[0], b = hash[1], c = hash[2], d = hash[3];

    MD5STEP(F1, a, b, c, d, input[ 0]+0xd76aa478,  7);
    MD5STEP(F1, d, a, b, c, input[ 1]+0xe8c7b756, 12);
    MD5STEP(F1, c, d, a, b, input[ 2]+0x242070db, 17);
    MD5STEP(F1, b, c, d, a, input[ 3]+0xc1bdceee, 22);
    MD5STEP(F1, a, b, c, d, input[ 4]+0xf57c0faf,  7);
    MD5STEP(F1, d, a, b, c, input[ 5]+0x4787c62a, 12);
    MD5STEP(F1, c, d, a, b, input[ 6]+0xa8304613, 17);
    MD5STEP(F1, b, c, d, a, input[ 7]+0xfd469501, 22);
    MD5STEP(F1, a, b, c, d, input[ 8]+0x698098d8,  7);
    MD5STEP(F1, d, a, b, c, input[ 9]+0x8b44f7af, 12);
    MD5STEP(F1, c, d, a, b, input[10]+0xffff5bb1, 17);
    MD5STEP(F1, b, c, d, a, input[11]+0x895cd7be, 22);
    MD5STEP(F1, a, b, c, d, input[12]+0x6b901122,  7);
    MD5STEP(F1, d, a, b, c, input[13]+0xfd987193, 12);
    MD5STEP(F1, c, d, a, b, input[14]+0xa679438e, 17);
    MD5STEP(F1, b, c, d, a, input[15]+0x49b40821, 22);

    MD5STEP(F2, a, b, c, d, input[ 1]+0xf61e2562,  5);
    MD5STEP(F2, d, a, b, c, input[ 6]+0xc040b340,  9);
    MD5STEP(F2, c, d, a, b, input[11]+0x265e5a51, 14);
    MD5STEP(F2, b, c, d, a, input[ 0]+0xe9b6c7aa, 20);
    MD5STEP(F2, a, b, c, d, input[ 5]+0xd62f105d,  5);
    MD5STEP(F2, d, a, b, c, input[10]+0x02441453,  9);
    MD5STEP(F2, c, d, a, b, input[15]+0xd8a1e681, 14);
    MD5STEP(F2, b, c, d, a, input[ 4]+0xe7d3fbc8, 20);
    MD5STEP(F2, a, b, c, d, input[ 9]+0x21e1cde6,  5);
    MD5STEP(F2, d, a, b, c, input[14]+0xc33707d6,  9);
    MD5STEP(F2, c, d, a, b, input[ 3]+0xf4d50d87, 14);
    MD5STEP(F2, b, c, d, a, input[ 8]+0x455a14ed, 20);
    MD5STEP(F2, a, b, c, d, input[13]+0xa9e3e905,  5);
    MD5STEP(F2, d, a, b, c, input[ 2]+0xfcefa3f8,  9);
    MD5STEP(F2, c, d, a, b, input[ 7]+0x676f02d9, 14);
    MD5STEP(F2, b, c, d, a, input[12]+0x8d2a4c8a, 20);

    MD5STEP(F3, a, b, c, d, input[ 5]+0xfffa3942,  4);
    MD5STEP(F3, d, a, b, c, input[ 8]+0x8771f681, 11);
    MD5STEP(F3, c, d, a, b, input[11]+0x6d9d6122, 16);
    MD5STEP(F3, b, c, d, a, input[14]+0xfde5380c, 23);
    MD5STEP(F3, a, b, c, d, input[ 1]+0xa4beea44,  4);
    MD5STEP(F3, d, a, b, c, input[ 4]+0x4bdecfa9, 11);
    MD5STEP(F3, c, d, a, b, input[ 7]+0xf6bb4b60, 16);
    MD5STEP(F3, b, c, d, a, input[10]+0xbebfbc70, 23);
    MD5STEP(F3, a, b, c, d, input[13]+0x289b7ec6,  4);
    MD5STEP(F3, d, a, b, c, input[ 0]+0xeaa127fa, 11);
    MD5STEP(F3, c, d, a, b, input[ 3]+0xd4ef3085, 16);
    MD5STEP(F3, b, c, d, a, input[ 6]+0x04881d05, 23);
    MD5STEP(F3, a, b, c, d, input[ 9]+0xd9d4d039,  4);
    MD5STEP(F3, d, a, b, c, input[12]+0xe6db99e5, 11);
    MD5STEP(F3, c, d, a, b, input[15]+0x1fa27cf8, 16);
    MD5STEP(F3, b, c, d, a, input[ 2]+0xc4ac5665, 23);

    MD5STEP(F4, a, b, c, d, input[ 0]+0xf4292244,  6);
    MD5STEP(F4, d, a, b, c, input[ 7]+0x432aff97, 10);
    MD5STEP(F4, c, d, a, b, input[14]+0xab9423a7, 15);
    MD5STEP(F4, b, c, d, a, input[ 5]+0xfc93a039, 21);
    MD5STEP(F4, a, b, c, d, input[12]+0x655b59c3,  6);
    MD5STEP(F4, d, a, b, c, input[ 3]+0x8f0ccc92, 10);
    MD5STEP(F4, c, d, a, b, input[10]+0xffeff47d, 15);
    MD5STEP(F4, b, c, d, a, input[ 1]+0x85845dd1, 21);
    MD5STEP(F4, a, b, c, d, input[ 8]+0x6fa87e4f,  6);
    MD5STEP(F4, d, a, b, c, input[15]+0xfe2ce6e0, 10);
    MD5STEP(F4, c, d, a, b, input[ 6]+0xa3014314, 15);
    MD5STEP(F4, b, c, d, a, input[13]+0x4e0811a1, 21);
    MD5STEP(F4, a, b, c, d, input[ 4]+0xf7537e82,  6);
    MD5STEP(F4, d, a, b, c, input[11]+0xbd3af235, 10);
    MD5STEP(F4, c, d, a, b, input[ 2]+0x2ad7d2bb, 15);
    MD5STEP(F4, b, c, d, a, input[ 9]+0xeb86d391, 21);

    hash[0] += a;   hash[1] += b;   hash[2] += c;   hash[3] += d;
}

A NAME="01d6_009f">Listing Three<

/* noise.h -- get environmental noise for RNG */
#include "usuals.h"
word32 noise(void);

/* noise.c -- Get environmental noise.
 * This is adapted from code in the Pretty Good Privacy (PGP) package.
 * Written by Colin Plumb. */
#include <time.h>
#include "usuals.h"
#include "randpool.h"
#include "noise.h"

#if defined(MSDOS) || defined(__MSDOS__)    /* Use 1.19 MHz PC timer */
#include <dos.h>    /* for enable() and disable() */
#include <conio.h>  /* for inp() and outp() */

/* This code gets as much information as possible out of 8253/8254 timer 0,
 * which ticks every .84 microseconds.  There are three cases:
 * 1) Original 8253.  15 bits available, as the low bit is unused.
 * 2) 8254, in mode 3.  The 16th bit is available from the status register.
 * 3) 8254, in mode 2.  All 16 bits of the counters are available.
 *    (This is not documented anywhere, but I've seen it!)
 * This code repeatedly tries to latch the status (ignored by an 8253) and
 * sees if it looks like xx1101x0.  If not, it's definitely not an 8254.
 * Repeat this a few times to make sure it is an 8254. */
static int
has8254(void)
{
    int i, s1, s2;
    for (i = 0; i < 5; i++) {
        disable();
        outp(0x43, 0xe2);   /* Latch status for timer 0 */
        s1 = inp(0x40);     /* If 8253, read timer low byte */
        outp(0x43, 0xe2);   /* Latch status for timer 0 */
        s2 = inp(0x40);     /* If 8253, read timer high byte */
        enable();
        if ((s1 & 0x3d) != 0x34 || (s2 & 0x3d) != 0x34)
            return 0;   /* Ignoring status latch; 8253 */
    }
    return 1;   /* Status reads as expected; 8254 */
}
static unsigned
read8254(void)
{
    unsigned status, count;
    disable();
    outp(0x43, 0xc2);   /* Latch status and count for timer 0 */
    status = inp(0x40);
    count = inp(0x40);
    count |= inp(0x40) << 8;
    enable();
    /* The timer is usually in mode 3, but some BIOSes use mode 2. */
    if (status & 2)
        count = count>>1 | (status & 0x80)<<8;
    return count;
}
static unsigned
read8253(void)
{
    unsigned count;
    disable();
    outp(0x43, 0x00);   /* Latch count for timer 0 */
    count = (inp(0x40) & 0xff);
    count |= (inp(0x40) & 0xff) << 8;
    enable();
    return count >> 1;
}
#endif /* MSDOS || __MSDOS__ */

#ifdef UNIX
#include <sys/types.h>
#include <sys/time.h>   /* For gettimeofday() */
#include <sys/times.h>  /* for times() */
#include <stdlib.h> /* For qsort() */
#define N 15    /* Number of deltas to try (at least 5, preferably odd) */
/* Function needed for qsort() */
static int
noiseCompare(void const *p1, void const *p2)
{   return *(int const *)p1 - *(int const *)p2; }
/* Find the resolution of the gettimeofday() clock */
static unsigned
noiseTickSize(void)
{
    int i = 0, j = 0, d[N];
    struct timeval tv0, tv1, tv2;
    gettimeofday(&tv0, (struct timezone *)0);
    tv1 = tv0;
    do {
        gettimeofday(&tv2, (struct timezone *)0);
        if (tv2.tv_usec > tv1.tv_usec+2) {
            d[i++] = tv2.tv_usec - tv0.tv_usec +
                1000000 * (tv2.tv_sec - tv0.tv_sec);
            tv0 = tv2;
            j = 0;
        } else if (++j > 10000) /* Always getting <= 2 us, */
            return 2;   /* so assume 2us ticks */
        tv1 = tv2;
    } while (i < N);
    /* Return average of middle 5 values (rounding up) */
    qsort(d, N, sizeof(d[0]), noiseCompare);
    return (d[N/2-2]+d[N/2-1]+d[N/2]+d[N/2+1]+d[N/2+2]+4)/5;
}
#endif /* UNIX */
/* Add as much time-dependent random noise to the randPool as possible. */
word32
noise(void)
{
    static word32 lastcounter;
    word32 delta;
    time_t tnow;
    clock_t cnow;
#if defined(MSDOS) || defined(__MSDOS__)
    static unsigned deltamask = 0;
    unsigned t;
    if (deltamask == 0)
        deltamask = has8254() ? 0xffff : 0x7fff;
    t = (deltamask & 0x8000) ? read8254() : read8253();
    randPoolAddBytes((byte const *)&t, sizeof(t));
    delta = deltamask & (t - (unsigned)lastcounter);
    lastcounter = t;
#elif defined(VMS)
    word32 t[2];
    SYS$GETTIM(t);  /* VMS hardware clock increments by 100000 per tick */
    randPoolAddBytes((byte const *)t, sizeof(t));
    delta = (t[0]-lastcounter)/100000;
    lastcounter = t[0];
#elif defined(UNIX)
    static unsigned ticksize = 0;
    struct timeval tv;
    struct tms tms;
    gettimeofday(&tv, (struct timezone *)0);
    randPoolAddBytes((byte const *)&tv, sizeof(tv));
    cnow = times(&tms);
    randPoolAddBytes((byte const *)&tms, sizeof(tms));
    randPoolAddBytes((byte const *)&cnow, sizeof(cnow));
    tv.tv_usec += tv.tv_sec * 1000000; /* Unsigned, so wrapping is okay */
    if (!ticksize)
        ticksize = noiseTickSize();
    delta = (tv.tv_usec-lastcounter)/ticksize;
    lastcounter = tv.tv_usec;
#else
#error Unknown operating system
#endif
    cnow = clock();
    randPoolAddBytes((byte const *)&cnow, sizeof(cnow));
    tnow = time((time_t *)0);       /* Read slowest clock last */
    randPoolAddBytes((byte const *)&tnow, sizeof(tnow));
    return delta;
}

Listing Four

/* randpool.h -- declarations for randpool.c */
#include "usuals.h"
#define RANDPOOLBITS 3072   /* Whatever size you need (must be > 512) */
void randPoolStir(void);
void randPoolAddBytes(byte const *buf, unsigned len);
void randPoolGetBytes(byte *buf, unsigned len);
byte randPoolGetByte(void);

/* randpool.c -- True random number computation and storage
 * This is adapted from code in the Pretty Good Privacy (PGP) package.
 * Written by Colin Plumb.  */
#include <stdlib.h>
#include <string.h>
#include "randpool.h"
#include "md5.h"

#define RANDKEYWORDS 16 /* This is a parameter of the the MD5 algorithm */
/* The pool must be a multiple of the 16-byte (128-bit) MD5 block size */
#define RANDPOOLWORDS ((RANDPOOLBITS+127 & ~127) >> 5)
#if RANDPOOLWORDS <= RANDKEYWORDS
#error Random pool too small - please increase RANDPOOLBITS in randpool.h
#endif
/* Must be word-aligned, so make it words.  Cast to bytes as needed. */
static word32 randPool[RANDPOOLWORDS];  /* Random pool */
static word32 randKey[RANDKEYWORDS];    /* Next stirring key */
static unsigned randPoolGetPos = sizeof(randPool); /* Position to get from */
static unsigned randKeyAddPos = 0;  /* Position to add to */
/* "Stir in" any random seed material before removing any random bytes. */
void
randPoolStir(void)
{
    int i;
    word32 iv[4];
    byteSwap(randPool, RANDPOOLWORDS);  /* convert to word32s */
    byteSwap(randKey, RANDKEYWORDS);
    /* Start IV from last block of randPool */
    memcpy(iv, randPool+RANDPOOLWORDS-4, sizeof(iv));
    /* CFB pass */
    for (i = 0; i < RANDPOOLWORDS; i += 4) {
        MD5Transform(iv, randKey);
        iv[0] = randPool[i  ] ^= iv[0];
        iv[1] = randPool[i+1] ^= iv[1];
        iv[2] = randPool[i+2] ^= iv[2];
        iv[3] = randPool[i+3] ^= iv[3];
    }
    memset(iv, 0, sizeof(iv));  /* Wipe IV from memory */
    byteSwap(randPool, RANDPOOLWORDS);  /* Convert back to bytes */
    memcpy(randKey, randPool, sizeof(randKey)); /* Get new key */
    randKeyAddPos = 0;  /* Set up pointers for future use. */
    randPoolGetPos = sizeof(randKey);
}
/* Make a deposit of information (entropy) into the pool. */
void
randPoolAddBytes(byte const *buf, unsigned len)
{
    byte *p = (byte *)randKey+randKeyAddPos;
    unsigned t = sizeof(randKey) - randKeyAddPos;
    while (len > t) {
        len -= t;
        while (t--)
            *p++ ^= *buf++;
        randPoolStir();     /* sets randKeyAddPos to 0 */
        p = (byte *)randKey;
        t = sizeof(randKey);
    }
    if (len) {
        randKeyAddPos += len;
        do
            *p++ ^= *buf++;
        while (--len);
        randPoolGetPos = sizeof(randPool); /* Force stir on get */
    }
}
/* Withdraw some bits from the pool. */
void
randPoolGetBytes(byte *buf, unsigned len)
{
    unsigned t;
    while (len > (t = sizeof(randPool) - randPoolGetPos)) {
        memcpy(buf, (byte const *)randPool+randPoolGetPos, t);
        buf += t;
        len -= t;
        randPoolStir();
    }
    memcpy(buf, (byte const *)randPool+randPoolGetPos, len);
    randPoolGetPos += len;
}
/* Get a single byte */
byte
randPoolGetByte(void)
{
    if (randPoolGetPos == sizeof(randPool))
        randPoolStir();
    return ((byte const *)randPool)[randPoolGetPos++];
}

Copyright © 1994, Dr. Dobb's Journal

Back to Table of Contents


Figure 4

Copyright © 1994, Dr. Dobb's Journal


DEC94: Real-Time Scheduling Algorithms

Real-Time Scheduling Algorithms

Achieving predictability for critical applications

Alberto Daniel Ferrari

Alberto is on leave from the Laboratorio de Controle e Microinformatica at the Universidade Federal de Santa Catarina in Florianopolis, Brazil. He can be reached at alberto@uncu.edu.ar.

In a real-time computer system, correctness depends on the time at which the results are given. In practice, this means that for real-time systems to behave properly, some critical subset of a system's tasks should complete their processing before their deadlines. Failure to do so can lead to human, environmental, and/or economic damages. Compounding the problem is that emerging systems which are distributed, dynamic, and adaptive will put even more stringent demands on real-time systems. The success of teams of robots working in hazardous environments, on-board space-shuttle systems, and underwater or outer-space autonomous vehicles, for instance, will all be strongly dependent on the timeliness of their computational results.

The predictability of the real-time system is fundamental to achieve this temporal correctness. A predictable system has known temporal bounds for all its actions and can therefore be formally analyzed. Automated techniques can be applied in advance to guarantee the system's critical set. This way, you can have an early warning of a system's inability to meet its timing requirements, and so take appropriate corrective actions. In other words, real-time is not synonymous with "fast," but with "predictable."

Most commercial real-time executives are based in a priority-driven scheme. Current real-time practices for uniprocessor multitasking systems rely on the importance the designer subjectively perceives they have. However, there's no guarantee that critical tasks will meet their deadlines just because the "ready" process with higher priority is the one that executes at every moment; consequently, the temporal correctness of the system remains uncertain. Furthermore, ad hoc priority schemes are not effective for predictable access to shared resources due to the unbounded priority inversion phenomenon, in which a higher-priority task is prevented from executing by a lower-priority task for an indefinite period of time.

To alleviate these problems, real-time scheduling algorithms have been devised that force the system's scheduler to base its decisions explicitly on the temporal characteristics of the task set. For most of these algorithms, simple mathematical conditions verify the schedulability of the critical set; satisfying the conditions guarantees the set, even without the designer knowing precisely when any task will be run. This way, you isolate temporal from logical system correctness.

With real-time algorithms, you trade additional information about your system for temporal predictability. To get a more dependable system, you must know about the temporal qualities of the target environment, application, and tasks--then make this information available to the scheduler.

Algorithms for Real-Time Use

When applying real-time algorithms, you must distinguish the algorithm's heuristic--used for scheduling the system--from its schedulability test, which guarantees a given task set. That is, you can use every heuristic that comes to mind to schedule a task group, but you won't have any guarantee of meeting the set's temporal constraints unless you apply a scheduling test.

The algorithm heuristic is the criterion according to which we define a figure of merit for a task. The task with highest-merit value among all ready tasks at every moment is the task that executes (ties are broken arbitrarily). Schedulability tests are valid for a well-defined task model. In this article, I'll examine the rate-monotonic (RM), earliest deadline (EDF), least-laxity dynamic (LLF), and maximum-urgency-first (MUF) algorithms. The temporal model for the algorithms discussed in this article assumes that each task:

  • Repeatedly executes at a known fixed rate (its "period").
  • Must end before the beginning of its next period (its "deadline").
  • Does not need to synchronize with others in order to execute.
  • Can be interrupted at any point in time and replaced by another task in the CPU.
  • Does not suspend voluntarily.
  • Has zero preemption cost (task-switch times and scheduling-algorithm execution load are neglected).
  • Is ready while its assigned processing time is not exhausted. After running out of execution units, the task blocks until its next period.
Since the task set is periodic, the base timeline will repeat itself after the LCM (least-common multiple) of the periods of the involved tasks.

A scheduling algorithm is said static if the task merit's value is fixed throughout its execution, and dynamic otherwise. Static algorithms have lower run-time costs but lack flexibility for adapting to a changing environment; they also need temporal information from all tasks before they're actually run.

Rate monotonic (RM). Developed in 1973 by Liu and Leyland, this is one of the most well-known and often-used algorithms for real-time applications. Its scheduling heuristic is shortest-period-first, so it always runs the ready task with shortest period. Since the period of a task is fixed, RM is a static algorithm. The condition for a task set to always meet its deadlines is shown in Figure 1, where Ci is task i's execution time, Ti is its period, and n is the number of tasks. The left term is the sum of the individual task loads; the right term defines a limit that goes from 100 percent for one task, to 69 percent for a large number of tasks. In practice, 88 percent is a more realistic value, although when task periods are harmonically related, it could still reach 100 percent. Even when the total task load is higher than the limit, the set may still be scheduled using RM: You just have to prove that every task meets its first deadline when all tasks are started simultaneously.

RM is said to be "stable" because one subset of tasks remains guaranteed when the processor is overloaded. Having ordered the task set by increasing periods, its stable set is composed of the first n tasks whose combined load is below the limit. The RM analytical model has been extended to handle complex situations, such as task synchronization (priority-ceiling algorithms) and aperiodic task service (through appropriate servers). RM is also optimum in the sense that it can always schedule a task set if another static algorithm can.

Earliest deadline (EDF). This algorithm schedules the task with closer deadline first. Because the task with the nearest deadline varies with time, this is considered a dynamic algorithm. In this case, a task set is schedulable if the total task load is under 100 percent. Its task model was extended for the same cases as RM, too. EDF is optimal in that if a task set can be scheduled by any algorithm, it can also be scheduled using EDF. It also features the lowest number of task switches but is not stable.

Least laxity (LLF). The merit of the least-laxity (LLF) dynamic algorithm is the lesser "flexibility" to be scheduled in time. This laxity is measured as the difference between the time to deadline and the remaining computation time to finish. A task with negative laxity won't meet its deadline, so laxity provides early detection of temporal failures. This is important if specific actions (other than aborting the faulty task or warning the user) should be taken due to a timing fault: You must ensure that the exception handler has time to execute its code within a given deadline. Again, for a task set to be schedulable, its total load must be under 100 percent; LLF is also optimal, in the same sense as EDF.

Maximum-urgency-first (MUF). Developed at Carnegie Mellon University, this mixed-scheduling algorithm combines the best features of the others: predictability under overload conditions and a scheduling bound of 100 percent for its critical set. The static part of a task's urgency is a user-defined criticality (high/ low), which has higher precedence than its dynamic part.

First, tasks are ordered by increasing periods: The first n highly critical tasks with combined load under 100 percent form the critical set. The highly critical ready task with the least laxity is chosen to run at every moment. If no critical tasks are ready, then tasks with low criticality are selected, again with the least-laxity heuristic. Ties are broken through an optional user priority.

The period of any taski may be modified without dropping it from the critical set, provided its load (Ci/Ti) remains unchanged. This is useful in dynamic environments, where the system needs flexibility to adapt to changing situations.

Table 1 provides a summary of the RM, EDF, LLF, and MUF algorithms. All four may accept new periodic tasks at run time, but a reschedule operation is necessary to guarantee the former critical tasks.

Figure 2 shows execution timelines for a task set scheduled by each of these algorithms, with a critical load of 58.3 percent. The resulting schedules differ in the number of (potentially costly) context switches--13, 11, 13, and 13, respectively. Any timeline repeats itself every 24 time units.

For nonstable algorithms such as EDF and LLF, any overload that raises the total load above 100 percent will cause at least one task to miss its deadline. We don't know which one may fail; it could be even a critical task. In RM (as in MUF), tasks outside the critical set may safely overload without any critical task losing a deadline; highly critical tasks may overload the example in up to (78.0/58.3--1)=33.8 percent and still remain guaranteed for RM. In the case of MUF, this increases to (100/58.3--1)=71.5 percent, a certainly higher value. The actual RM-scheduling limit for this case is higher than the value yielded by the formula in Figure 1 (78.0 percent).

Increasing Task B's CPU time from 3 to 5 raises the critical load in 64.3 percent (95.8/58.3--1); see Figure 3. RM cannot handle the critical set anymore, but MUF still guarantees it. Since the timeline is periodic with period 24, no task belonging to the critical set (A and B) ever loses a deadline.

Implementation

I've included with this article a program that implements a generic execution machine that simulates a system's temporal behavior under a specified scheduling algorithm. Configuration files describe the intended task set and system parameters; see Listing One . The program is composed of a series of modules which are available electronically; see "Availability," page 3.

Listing Two is the code for the simulation machine. The selected algorithm module is called upon entry and exit of the program, and on every time tick of the system when it returns the task; in the latter case, it returns the task chosen to run next.

The system is not priority driven but managed through linked lists. The deadline_list holds current instances ordered by increasing deadlines and is searched for failed tasks: When a deadline is reached, its owner should be idle; otherwise, there is a deadline failure and the instance is aborted. Request_list contains the time for future task invocations; since the tasks are periodic, a request for the next instance is made upon starting. Finally, merit_list contains current instances ordered by merit value.

Default_dispatcher(), the sched_alg() function for RM and EDF, returns the first ready task from merit_list. If the selected task is different from the current task but has the same merit, the current task continues to execute, to save one context switch.

The first three algorithms are in Listing Three . All relevant processing for RM is done during its initialization; after that, the generic dispatcher is called, according to its static nature. Since EDF's merit list is deadline_list, it is always modified as instances develop and execute. As the tasks' laxities vary with time, I preferred not to reorder the merit list of LLF on every tick; instead, merit_list is linearly searched for the task with least value. Apart from this, least_laxity() has almost the same code as default_dispatcher.

Finally, Listing Four shows the code of the MUF algorithm. Tasks are first ordered by increasing periods; then the critical set is defined. The code returns the highly critical task with the least laxity, if it exists, or a task with lower criticality and least laxity otherwise. Ties are broken through user priority, which is determined by the order in which tasks appear in the configuration file (preceding tasks have higher user priority than a given task).

Figure 4 shows a sample timeline, plus the program output generated. When a task executes continuously, you capture just its beginning and not its subsequent time periods: This is the case for a, which executes from the beginning of 5 to the end of 7. There exists a special task, idle_task, whose execution represents the processing of background tasks or tasks without hard deadlines. When a task terminates normally, it blocks until the beginning of its next period.

For linked-list management, I adapted Pugh's skip lists, described and coded in the article "Skip Lists" by Bruce Schneier (DDJ, January 1994). The program supports an interface to this library that modifies some calls, mainly because the original doesn't have search-by-value functions for duplicated items. Consequently, I combined the key and the task_id into one new key and didn't allow for duplicates.

Conclusion

The main difficulties in applying real-time scheduling techniques stem from the restricted task model for which they're valid. Those models often miss communication relations, precedence orders, and mutual-exclusion problems among tasks that exist in real-world situations. As tasks interact, integrated resource scheduling is also necessary. Algorithms exist that support special cases, in which decisions deal with imprecise results, task-completion value, and so on. However, no algorithm is good for all cases: You'd have to look for the one most suitable to your system's task model.

Additionally, you must know the precise temporal attributes of each process in advance. State-of-the-art real time lacks tools for getting them automatically. The problem is exacerbated when modules synchronize or share resources (such as data). Modern hardware, with cache, pipelining, and the like, also adds temporal uncertainty to the system.

Real-time scheduling may seem unnecessary, but as the project's complexity and size increase, it's the only way to guarantee proper system behavior. It is certainly more predictable than ad hoc techniques. If you're still not convinced, just imagine scheduling 20--30 processes by hand. Even worse, imagine modifying the resulting schedule to accommodate a change in the specifications of one or two processes. Remember, however, that the benefits of real-time scheduling depend on the quality of the temporal information (actual execution-time upper limit of each task, among others) on which you base your analysis.

Bibliography

Liu, C. and J. Layland. "Scheduling Algorithms for Multiprogramming in a Hard-Real-Time Environment." Journal of the ACM (January 1973).

Natarajan, S. and W. Zhao. "Issues in Building Dynamic Real-Time Systems." IEEE Software (September 1992).

Sha, L. and J. Goodenough. "Real-Time Scheduling Theory and Ada." IEEE Computer (April 1990).

Stankovic, J. "Misconceptions about Real-Time Computing." IEEE Computer (October 1988).

Stewart, D. and P. Koshla. "Real-Time Scheduling of Sensor-Based Control Systems." Proceedings of Eighth IEEE Workshop on Real-Time Operating Systems and Software (May 1991).

Figure 1 RM schedulability test for a set of n periodic, independent, preemptive tasks. The right term of the equation defines a limit which the total load should not surpass for the task set to be guaranteed. Figure 2 Task set scheduled using: RM, EDF, LLF, MUF. Although total load (83 percent) is higher than the RM limit for this case (78 percent), RM can schedule the set satisfactorily.

Table 1: Summary of RM, EDF, LLF, and MUF scheduling algorithms.

Algorithm           Type      Schedulability                   Comment
                              Test

Rate monotonic      Static    Total load under limit defined   Stable
  (RM)                          by algorithm task model.
                                Meeting of first deadline.
Earliest deadline   Dynamic   Total load under 100%            Least number
  (EDF)                                                          of context
                                                                 switches
Least laxity        Dynamic   Total load under 100%            Early
  (LLF)                                                          detection
                                                                 of timing
                                                                 failures
Maximum urgency     Mixed     Critical load under 100%         Stable;
  first (MUF)                                                    100%
                                                                 schedule
                                                                 bound

Figure 3 The same as Figure 2, but with a critical overload of 64.3 percent. Note that MUF is the only algorithm that gets the critical set (A and B) scheduled in time.

Figure 4 Sample timeline and its program's representation. Background processing (or soft) real-time tasks are symbolized by (Z); (b) program output for the example in Figure 2, scheduled with RM.

Listing One

configuration file: TASK DESCRIPTION FILE
; Task set descriptions, from which tasks are instantiated. Keywords
; are at the line's beginning, and end with ':'.
; Everything in the line after the keywords or values is ignored. Lines
; beginning with '*' are also ignored. No line can be
; longer than MAX_STRING characters, and no name longer than
; MAX_NAME_LENGTH. Please, maintain the order of the parameters in the
; task descriptions.
; start:

test set: Article's Example

MAXTIME= 27    /* timeline's upper limit (starting at 0) */
Number of Application Tasks=3

APPLICATION TASKS DESCRIPTION:
Name       Criticality  Period      Execution_time
Task A,       HIGH,        6,           2.
Task B,       HIGH,        8,           2.
Task C,        LOW,       12,           3.
end.

Listing Two

/***** RTALGS.C -- Algorithms for Real-Time Scheduling  *****
 ***** Alberto Ferrari -- aferrari@uncu.edu.ar          *****/
/* Simulates the execution of a task set in a multitasking environment,
under several real-time scheduling algorithms. The objective is to
obtain a timeline of the execution, and to show if tasks meet their
deadlines or not. Tasks are assumed to be hard real-time, preemptive,
periodic, with deadline equal to the next instance's arrival
time, and independent (they do not need to syncronize with others in
order to execute). They also do not suspend its execution voluntarily.
All tasks start execution at the same time in the simulation.
Available algorithms for scheduling are Rate Monotonic (RM),
Earliest-Deadline-First (EDF), Least-Laxity-First (LLF), and
Maximum-Urgency-First (MUF), selected in command line. Also selected
in command line is the configuration file, containing the task set
description and the system parameters. Usage:
rtalgs -[e|E|l|L|m|M|r|R] <testvect.name> where: e,
Earliest-Deadline-First (EDF); l, Least-Laxity-First (LLF); m,
Maximum-Urgency-First (MUF), r, Rate Monotonic (RM) *****/
void main( int argc, char *argv[])
{
    node n;
    task_t *task, *new;
  init( argc, argv);
  printf( "\nSelected Scheduling Algorithm: %s,\n", labels[ alg]);
  (sched_alg_init)();
  /* select which task to run next */
  for( sys_time=0;
      (merit_list->header->forward[0]!=NIL || request_list->
                                                     header->forward[0]!=NIL)
          &&  sys_time <= max_time;
      sys_time++){

            /* and if current task emptied its allocated time... */
            if( current!= idle_task  &&  -- current->remaining == 0){
             current->state=DEAD;
             current->cycles++;
             delete_task( deadline_list, current->deadline, current);
             current= idle_task;
          }
          /* Look out for deadline failures */
          while(  key_of( n=first_node_of( deadline_list)) <= sys_time){
              if( (task= n->v)->state != DEAD){
                  printf( "At %d: task %c (\"%s\"),
                                   instance %d, Deadline Failure%s\n",
                      sys_time, task->sys_id, task->name,
                                          task->instance, bell);
              }
              delete( deadline_list, n->key);
          }
          /* if it is time to launch a task... */
          while(  key_of( n=first_node_of( request_list)) <= sys_time){
              task_init( (task= n->v) );
              delete( request_list, n->key);
              insert_task( deadline_list, task->deadline,task);
              insert_task(  request_list, task->deadline, task);
          }
          new = (sched_alg)();
          /* swap and register who's using the processor */
          if( current!=new){
              context_switches++;
              current->state=READY;
              current=new;
              current->state=RUNNING;
          }
          timeline.history[ sys_time]= current->sys_id;
          #ifdef DEBUG
          printf( "%d: %s\n", sys_time, timeline.history);
          #endif
  }
  (sched_alg_end)();

  draw_timeline();
}
  ......
task_t *default_dispatcher( void)
{
    task_t *task;
    if(  (task=first_ready( merit_list))==NULL)
        return( idle_task);
    else if( current== idle_task)
        return( task);
    else /* current task prevails other tasks with same merit */
        return(  (*task->merit == *current->merit)? current : task);
}

Listing Three

/***** Rate Monotonic (RM) Algorithm *****/
void monotonic_rate_init( void)
{
    int i;
    node n;
    task_t *task;      /* 'task' is the task with 'lesser' period */
    float task_load=0.0, critical_task_load=0.0, schedulability_bound;
    /* in RM case, 'deadline_list' is different from the 'merit_list' */
    deadline_list= newList(); deadline_list ->sys_id= 'D';
    /* calculate n*(2^1/n - 1) */
    schedulability_bound= num_tasks * ( pow( 2.0, 1.0/num_tasks) -1.0);
    printf( "which has a schedulability bound of %.1f%% for %d tasks.\n",
            100.0 * schedulability_bound, num_tasks);
    /* insert tasks in merit_list by increasing periods */
    /* If two tasks with equal period, order them by original sequence */
    for( i=1; i<=num_tasks; i++){
        task->merit= &(task=task_set+i)->period;
        insert_task( merit_list, *task->merit, task);
        insert_task( request_list, 0, task);
    }
    puts( "Critical set is composed of");
    for( task= (n= merit_list->header->forward[0])->v; n!=NIL;
         task= (n= n->forward[0])->v){
        task_load+= (float )task->cpu_time / (float )task->period;
        if( task_load <schedulability_bound){
            critical_task_load= task_load;
            printf( "\t%s,\n", task->name);
        }
    }
    printf( "which accounts for a critical load of %.1f%%, over a total
      system load of %.1f%%\n", 100.0 * critical_task_load, 100.0 * task_load);

    if( task_load<=schedulability_bound) printf( "So, the whole task set IS");
    else if( task_load>1.0) printf( "WARNING: the whole task set IS NOT");
    else printf( "WARNING: the whole task set MAY NOT be");
    printf( " schedulable under RM\n\n");
}
void monotonic_rate_end( void){}
/***** Earliest-Deadline-First (EDF) algorithm *****/
void earliest_deadline_init( void)
{
    task_t *task;
    float task_load=0.0;
    int i;
    printf( "which has a schedulability bound of 100%%\n");
    /* in the EDF case, 'deadline_list' is the same as 'merit_list' */
    deadline_list= merit_list;
    /* insert tasks in merit_list by increasing deadlines */
    for( i=1; i<=num_tasks; i++){
        task->merit= &(task=task_set+i)->deadline;
        task_load+= (float )task->cpu_time / (float )task->period;
        insert_task( request_list, 0, task);
    }
    printf( "Total system task load = %.1f%%\n", 100.0 * task_load);
    if( task_load<=1.0) printf( "So, the whole task set IS");
    else printf( "WARNING: the whole task set IS NOT");
    printf( " schedulable under EDF\n\n");
}
void earliest_deadline_end( void) {}
/***** Least Laxity Algorithm (LLA)  *****/
void least_laxity_init( void)
{
    task_t *task;
    float task_load=0.0;
    int i;
    printf( "which has a schedulability bound of 100%%\n");
    /* in the LLF case, 'deadline_list' is not the same as 'merit_list' */
    deadline_list= newList(); deadline_list ->sys_id= 'D';
    for( i=1; i<=num_tasks; i++){
        task->merit= &(task=task_set+i)->laxity;
        task_load+= (float )task->cpu_time / (float )task->period;
        insert_task( merit_list, *task->merit, task);
        insert_task( request_list, 0, task);
    }
    printf( "Total system task load = %.1f%%\n", 100.0 * task_load);
    if( task_load<=1.0) printf( "So, the whole task set IS");
    else printf( "WARNING: the whole task set IS NOT");
    printf( " schedulable under LLF\n\n");
}
task_t *least_laxity( void)
{
    task_t *least;
    /* all tasks (except 'current') now have one less 'laxity' unit */
    if(  (least=update_laxity_and_get_least( merit_list)) ==idle_task)
        return( idle_task);
    else if( current== idle_task)
        return( least);
    else /* current task prevails other tasks with same merit */
        return( (*least->merit == *current->merit)? current : least);
}
void least_laxity_end( void){}
/* returns idle_task if 'l' is empty */
task_t *update_laxity_and_get_least( list l)
{
    task_t *task, *least;
    node n;
    least= idle_task;
    for( task= (n= l->header->forward[0])->v; n!=NIL;
         task= (n= n->forward[0])->v){
        /* task->laxity(t) = task->deadline - t - task->remaining(t);
         * but now(t)= now(t-1)+1,
         * and task->remaining(t)=  task->remaining(t-1), if task!=current,
         * ==> task->laxity(t) = task->laxity(t-1) -1;
         *****************************************************************/
        /* look out! task->laxity is decremented only if its state is
                                                        READY, because of && */
        if( task->state ==READY  &&  -- task->laxity<0){
                                                      /* if it's eligible... */
            printf( "At %d: task %c (\"%s\"), instance %d, will
                            lose its deadline at %d%s\n",
                sys_time, task->sys_id, task->name,
                            task->instance, task->deadline, bell);
            task->state=BLOCKED;
        }
        if( (task->state ==READY  ||  task->state ==RUNNING)
         &&  task->laxity < least->laxity)
                   least=task;
    }
    return( least);
}

Listing Four

/***** Maximum-Urgency-First (MUF) Algorithm *****/
list high_crit_l, low_crit_l;
task_t *first;
void maximum_urgency_first_init( void)
{
    node n;
    list temp_list;
    task_t *task;
    float critical_task_load=0.0, task_load=0.0, temp=0.0, load;
    int i, critical_set=TRUE;
    printf( "which has a schedulability bound of 100%%\n");
    /* in the MUF case, 'deadline_list' is not the same as 'merit_list' */
    deadline_list= newList(); deadline_list->sys_id= 'D';
    temp_list= newList(); temp_list->sys_id= 'T';
    high_crit_l= merit_list; high_crit_l->sys_id= 'H';
    low_crit_l= newList(); low_crit_l->sys_id= 'L';
    for( i=1; i<=num_tasks; i++){
        task=task_set+i;
        task->merit= &task->laxity;
        /* use temp_list to order tasks by increasing periods */
        insert_task( temp_list, task->period, task);
        insert_task( request_list, 0, task);
    }
    /* insert tasks in both (high_crit_l and low_crit_l) lists */
    puts( "Critical set is composed of"); /* the first 'n' tasks in
                             'high_crit_l' with combined load less than 100% */
    for( task= (n= temp_list->header->forward[0])->v; n!=NIL;
                                                  task= (n=n->forward[0])->v){
        task_load+=(load=(float )task->cpu_time/(float )task->period);
        if( task->criticality ==HIGH){
            if(  (temp+=load)<=1.0  &&  critical_set==TRUE){
                critical_task_load= temp;
                printf( "\t%s,\n", task->name);
                insert_task( high_crit_l, task->period, task);
            }else{
                critical_set= FALSE;
                printf( "WARNING at %d: Highly critical task
                                    %c (\"%s\"),\ found NOT Schedulable!!%s",
                                    now(), task->sys_id, task->name, bell);
                printf( "\nContinue anyway? (y/[N]) ");
                if( (i=getchar()) == '\n' || i=='n' || i=='N')
                                                                      exit(0);
                else insert_task(low_crit_l,task->period,task);
            }
        }else{    /* task->criticality ==LOW */
            insert_task( low_crit_l, task->period, task);
        }
    }
    freeList( temp_list);
    printf( "which accounts for a critical load of %.1f%%, over a total
                        system load of %.1f%%\n",
                        100.0 * critical_task_load,
                        100.0 * task_load);
    if( task_load<=1.0) printf( "So, the whole task set MAY BE");
    else printf( "WARNING: the whole task set IS NOT");
    printf( " schedulable under MUF\n\n");
}
task_t *maximum_urgency_first( void)
{
    task_t *least, *leasth, *leastl;
    /* all tasks (except 'current') now have one less 'laxity' unit */
    leasth= update_laxity_and_get_least( high_crit_l);
    leastl= update_laxity_and_get_least( low_crit_l);
    least= (leasth==idle_task)? leastl : leasth;
    /* all tasks (except 'current') have one less 'laxity' time unit */
    if( least==idle_task)
        return( idle_task);
    else if( current== idle_task)
        return( least);
    else /* current task prevails other tasks with same merit */
        return( (*least->merit == *current->merit)? current : least);
}
void maximum_urgency_first_end( void){}

Copyright © 1994, Dr. Dobb's Journal

Back to Table of Contents


Example 1(a)

Copyright © 1994, Dr. Dobb's Journal


Example 1(b)

Copyright © 1994, Dr. Dobb's Journal


Example 1(c)

Copyright © 1994, Dr. Dobb's Journal


Example 1(d)

Copyright © 1994, Dr. Dobb's Journal


Figure 8

Copyright © 1994, Dr. Dobb's Journal


Example 1(e)

Copyright © 1994, Dr. Dobb's Journal


Example 1(f)

Copyright © 1994, Dr. Dobb's Journal


Example 1(g)

Copyright © 1994, Dr. Dobb's Journal


SP 94: Morphing on Your PC

Morphing on Your PC

An easy-to-use algorithm for animation and morphing

Scott Anderson

Scott is the president of Wild Duck, a software publishing and development company in Rohnert Park, California. He is also the author of the animation program Fantavision, and the recently released book Morphing Magic from SAMS. He can be reached at 73710,1055 on CompuServe.

Morphing has become almost as ubiquitous as bubble-gum under theater seats. As the July 1993 issue of Dr. Dobb's Journal illustrated, from autos turning into tigers to bulls into bears, we see examples of morphing everywhere on television and in movies. While the effect is compelling, morphing is useful beyond the magical transformations we've grown used to seeing.

This article examines the history and mathematics behind morphing, and includes three utility programs that allow you to run full-screen animated morphs on your PC.

Warping, Tweening, and Dissolving

Morphing is a blend of three separate algorithms: warping, tweening, and dissolving. Warping is the mathematical trick of stretching and squashing an image as if it were painted on rubber. This article discusses an implementation of one of the latest warping routines, based on the method used by Pacific Data Images (PDI) to make the Michael Jackson video "Black or White." The algorithm uses lines to control the warping. Lines make the warping specification much easier than using points, giving the artist a break.

Tweening is short for in-betweening, the interpolation of two images to yield a smooth-flowing animation. Tweening is typically done with points, lines, or polygons. Since this warping algorithm is line oriented, tweening fits right in. By tweening the position of the control lines, you can smoothly warp an image around. With warping and tweening alone, you can create photorealistic animations from single photographs. And it's simple to do. (For more details, see my book Morphing Magic, SAMS, 1993.)

Dissolving, or cross-dissolving, is Hollywood-speak for fading out one scene while fading in the next. In the middle, you have a double exposure. This powerful effect was used in the early Wolfman movies. While poor Lon Chaney, Jr. stuck his head in a vice, makeup people swarmed about, adding hair and putty. After each little change in makeup was completed, another few frames of film were squeezed off. Each short take was cross-dissolved with the previous one to complete the illusion.

When you put all three of these effects together, you end up with morphing. Here's how it works. Let's say you want to morph yourself into a tiger. On the first frame of your self-portrait, you mark the key areas with lines. You might place a control line from one eye to the other, down the nose, and across the lips. These three lines capture the most important aspects of a face. Then, on the tiger's ferocious countenance, you would draw control lines on the same features: eyes, nose, and mouth. That's enough information for the computer to morph the two. To create a single morph in the middle, the control lines are tweened halfway and then the images are warped to follow the new lines. The algorithm warps your face midway to the tiger, and warps the tiger midway to you. This is the fifty-fifty interpolation between the two sets of control lines--the central tween. It is also simply the average of the two sets.

After creating the two warped images, the routine cross-dissolves them, finally producing the morph. If you use more tweens, say ten interpolations between you and the tiger, you can make a smoothly animated sequence of you turning into the tiger. The first frame uses 10 percent of the tiger mixed with 90 percent of you. The second frame has 20 percent of the tiger with 80 percent of you. By the ninth frame, there is 90 percent tiger, and you have faded out to 10 percent.

Warping Algorithms

There are as many ways to warp as there are sort routines. The method employed here is a variation on the PDI line method. As I mentioned before, this approach is friendly to the artist. Other routines use dots to warp the image. These changes are local and act only over a short distance, so you need hundreds of them. Misplacing some of these dots and ruining the warp isn't at all difficult. With the new method, each line represents the string of points that compose it, so a few short lines can stand in for hundreds of points. As you move a control line, the pixels around it are pulled as well.

One of the interesting peculiarities of this algorithm is the global nature of the warping. A single line can specify any combination of scaling, translation, and rotation. This can be a nice effect in itself. Just draw, say, a vertical line, rotate it 90 degrees, and the image will also be rotated. When there are more lines (as there usually are), they will each compete for influence, but the effect will still be global. The downside is that it takes a little longer to perform the warping.

In the algorithm presented here, the influence of the line falls off as 1/distance2. This is just like the influence of gravity, but you can select any variation on this that you desire. This formulation was chosen based largely on speed considerations. A mathematical discussion of the warping algorithm is in the text box entitled, "Warping with Control Lines." The basic warping algorithm is as follows:

  1. Find the distance between a pixel and the source line by dropping the perpendicular d (see Figure 1).
  2. Find the fractional distance f along the source line to the perpendicular. The fraction is normalized to go from 0 to 1.
  3. Move the point to a spot that is the same distance from, and fraction of, the target line.
This algorithm is carried out by two routines, sumLines and getSourceLoc; see Listing One, page 75. In getSourceLoc, the warped pixel is determined for each line, while sumLines adds up the influence of each line. GetSourceLoc performs the mathematics given in the text box entitled, "Warping with Control Lines." SumLines uses the distance returned by getSourceLoc to determine the weight contributed by each control line. In a loop through the lines, SumLines calculates the individual weights, as shown in Figure 2. You can experiment with the weight calculation (.001 is added to avoid division-by-zero errors). Try equations that depend on the length of the line, or inversely on the cube of the distance. The method used here is simple and fast, but don't be afraid to try something new. You never know what will happen!

Tweening Algorithm

The second piece of the morphing trio is tweening. Compared with warping, tweening is a piece of cake. In the simplest case of linear tweening, all you do is interpolate. Linear interpolation is just finding a point on the line connecting two others. Although there are dozens of ways to interpolate, for our purposes linear tweening is just fine. All we do is take the lines describing the source image and tween them into the lines describing the target. For each intermediate tween, we warp the images accordingly.

For speed, divide the distance between the points into as many steps as you desire, say ten. You will have an X and a Y component for the length of these segments, called "deltas." For a regular division of the line, all these deltas will be equal to one-tenth of the line distance, so you only need to calculate it once and then simply add it to the starting point. After ten additions of this delta, you will have arrived at the target point. The deltas are given by the simple equations in Figure 3. Adding a delta in this fashion is fast, but care must be taken to avoid round-off errors.

Dissolving Algorithm

Now we are at the third and final part of morphing, the dissolve. This is where you combine the two images you are warping and tweening. Mix any two colors just the way you might expect. The color midway between two colors is given by the formulas in Figure 4. On a 24-bit color system, the color you get from this calculation will always be another color you can display. Not so with color-mapped graphics cards. You can easily derive a color that isn't in the puny list of displayable colors called the "palette." The best most VGA cards can offer is 256 colors, which is pretty sad for this application.

To fix this you need to calculate all the colors as if you had 24 bits to play with. Then, you need to create a new palette that uses the 256 most popular colors. The rest of the colors must be forced into the closest color available from those 256. This process is referred to as "palette mapping."

So dissolving, which should be the smallest part of this whole algorithm, suddenly swells into an unsightly carbuncle on this otherwise straightforward code. Those are the basics of morphing. There are many different algorithms for performing this terrific effect, but this should serve as a good jumping-off point for further experimentation.

The Morphing Programs

So that you can experiment, I've included three programs: MORPH.C (Listing Three, page 77), LOAD.C (Listing Four, page 78), and FIX.C (Listing Five, page 78). The header containing the #defines for all three of these utilities is in Listing Two, page 76. MORPH gets the user input and creates the morphing sequence, FIX finds a good palette and LOAD plays the sequence back from memory. All the programs work with PCX files having 256 colors and 320x200 resolution. You can capture your own PCX files with any screen-capture program, or you can find GIF images on CompuServe and convert them with a program such as ZSoft's Paint Shop. Remember to convert the number of colors to 256. If you have access to a scanner, then your troubles are over. You can scan in your own photos and magazine clips and go from there.

When specifying a PCX file, you don't need to type in the file extension. The programs automatically append ".PCX" to the end of each filename, saving you the trouble.

Some of these programs produce a numbered sequence of output files. These are also PCX files. The maximum sequence size is 99. The sequence number, from 1 to 99, is appended to the root name of the file sequence. This takes two characters away from the already-tiny DOS allotment of eight. Therefore, the <OutFile> name must be six characters or less.

Figure 5 shows the syntax used on the command line to call the MORPH program. MORPH first reads the two input files: the source and target PCX images. Then it looks on the disk for any existing control lines that might be associated with the source or target image. If the user okays it, these line files are loaded in. Otherwise, the user creates a new set of control lines from scratch, and these are saved to the disk. Morph metamorphoses between the two in the given number of steps, to make a sequence of files. The files are numbered and can be displayed by the LOAD program. As an example, issuing the command line MORPH JIM BOB will cause MORPH to load the files JIM.PCX and BOB.PCX. It will create the middle tween and display it. No files are created. However, MORPH JIM BOB 7 will load the files JIM.PCX and BOB.PCX, create a sequence of seven frames, and display them. No files are created. Finally, MORPH JIM BOB 7 JIMBOB will also load JIM.PCX and BOB.PCX and create a sequence of seven frames. Each frame is displayed while it is saved to the disk. The files will be named JIMBOB1.PCX, JIMBOB2.PCX, and so on.

Figure 6 shows the syntax used to call the LOAD program. LOAD reads a sequence of PCX files indicated by the given name. It reads as many as it can fit into memory, then it displays them sequentially. You can control the playback rate by pressing a number from 1 (slow) to 9 (fast). To single-step through the pictures, press Spacebar. To continue playback, press Enter. To quit, press Esc. The number of images that can be animated depends on how much free RAM you have. Get rid of your TSRs and use the new memory features of DOS 6.0. The best you can do in 640K is about eight images.

The command-line syntax used to call FIX is in Figure 7. FIX takes a sequence created by MORPH and forces each picture in the sequence to have the same palette. FIX then writes the sequence out under the new name. The command FIX JIM FXJIM reads the sequence JIM1.PCX, JIM2.PCX, and so on, outputting the same number of files named FXJIM1.PCX, FXJIM2.PCX, and so. Also, FIX JIM JIM reads the JIM sequence and writes over the original files. Be careful! Make sure you don't need the originals before you do this.

Compiling

These programs were all compiled with Microsoft C/C++ 7.0. If you didn't include the graphics library when you installed your compiler, you will need to link that library in at compile time. I've included a makefile, which is available electronically; see "Availability," page 2. The makefile is set up assuming that you haven't installed the graphics library. If you have installed the graphics library, then you'll have to modify this makefile accordingly.

To make the programs work with other C compilers, you will probably need to make some changes to the non-ANSI library calls. In Microsoft C, these library calls are prefixed with the underscore character. The non-ANSI library calls are all in IO.C (available electronically) and LINECALC.C (Listing One).

Conclusion

The most obvious upgrade to this program is the ability to use extended RAM, so you can hold more frames in memory. If you have access to some of the new software on the market that attempts to leap the 640K barrier, use it. If you can work with smaller images, say, quarter-screen, 160x100 resolution, you should be able to quadruple the number of frames that can be held in memory, not to mention speeding up the animation. The display-picture routines you need to change are in Listing Five. Currently, they slam a continuous string of bytes into memory. For smaller screens, you will need to move the image a line at a time.

If you have a true-color card, you don't need the agonizing color-collapsing routines from Listing Six (page 79). Just yank them all out. You will need to save the files in 24-bit color mode, which can present some data problems, so look at the drivers and documentation for your particular graphics card.

With these programs, you can create some amazing morphs. With a video capture board and a computer capable of displaying 30 frames per second, you can output to videotape. Now you can play with some of the same toys they have at Industrial Light and Magic and Pacific Data Images. Have fun!

Figure 1: A given point P has a relationship to the line AB that gets mapped to the warping line A'B'.

Figure 2: Calculating individual weights.

distance      = getSourceLoc(&orig,origline,warp,warpline);
weight      = 1/(.001+distance*distance);
deltaSumX      += (orig.x - warp->x) * weight;
deltaSumY      += (orig.y - warp->y) * weight;

Figure 3: Equations to calculate the interpolation deltas.

DeltaX = (TargetX - SourceX) / (Tweens + 1)
DeltaY = (TargetY - SourceY) / (Tweens + 1)

Figure 4: Calculating the color midway between two colors.

NewRed = (SourceRed + TargetRed) / 2
NewGreen = (SourceGreen + TargetGreen) / 2
NewBlue = (SourceBlue + TargetBlue) / 2

Figure 5: Command-line syntax for calling MORPH.

MORPH <File1> <File2> [<Steps> [<OutFile>]]

<File1> is the name of the source PCX file to morph.
<File2> is the name of the target PCX file.
<Steps> is the optional number of output files.

If you don't specify the number of steps in your sequence, the program produces one in-between morph. You must specify <Steps> if you want to output files.
<OutFile> is the root name of the optional output files. The value of <Steps> determines the number of files to output. The step number is appended to the root name, so the <OutFile> name must be six characters or less.

Figure 6: Using the LOAD program.

LOAD <File>

<File> is the root name of a PCX sequence to display.

Figure 7: Command-line syntax used to invoke the FIX program.

FIX <InFile> <OutFile>

<InFile> is the root name of the PCX sequence to fix.
<OutFile> is the root name of the fixed PCX sequence.

A History of Warping

Warping's history reaches back to the '60s space program, when NASA was snapping shots of the earth like an eager tourist. But when it came time to put all the pictures together, NASA discovered that nothing quite lined up correctly because of the different altitudes, angles, times, and optics for each shot. Consequently, researchers at NASA developed algorithms that treated the digitized data as points on a polynomial surface that could be stretched to fit a set of reference points. They called their efforts image registration. Those marvelous Landsat pictures were thus stretched and pulled into a big quilt of pictures blanketing the globe. This was a great start for a little warping routine.

Warping was dusted off in the mid-seventies, when Viking 2 landed on Mars. Unfortunately, Viking came to rest at an angle, with its camera pointing down. The landscape it depicted looked like the Big Valley, with the horizon strongly curved. All the images of Mars were pushed through a warping algorithm that corrected for the odd optics before the public ever saw them.

Warping has found its way into the medical world, too. In a procedure called "digital subtraction angiography," an X-ray of a patient is taken before and after the injection of a dye into the area of interest. By subtracting the first image from the second, everything is removed but the dyed arteries. This uncluttered image is of great value to the diagnostician. Unfortunately, if the patient moves, the effect is wrecked. And, except for tree surgeons, moving patients are the norm. As you might have guessed, warping is the perfect tool to ensure registration.

-- S.A.

Warping with Control Lines

Given point P in the source image, you want to deduce point P' in the target image (see Figure 1). The distance d is the projection of vector AP on the perpendicular, which yields |AP| cos a, where |AP| represents the magnitude of the vector. The dot product of vector AP with the perpendicular to AB (denoted ^AB) is defined in Example 1(a), where |AP| and |AB| are the magnitudes, and the subscripted variables are the x and y components of the vectors. You can see that this solution in terms of components, (APx*^ABx+APy*^ABy), doesn't involve any trigonometry. Example 1(b) solves for d.

The magnitude of the line itself, |AB| is used instead of the magnitude of the perpendicular, |^AB|. The reason is that these two are equal, and you can reuse this number. The vector dot product allows you to calculate the desired values without computing any angles or using any transcendental functions. That makes it both simple and fast. From Figure 1, note that the distance represented by f is the projection of AP on AB itself, |AP| cos b, which is the quantity: (AP_AB)/|AB|. We want the fractional part of line AB, so we divide by the length of AB again as shown in Example 1(c).

Examples 1(b) and 1(c) represent a translation into a new-scaled orthogonal coordinate system based on d and f, instead of x and y. Now you are ready to transfer these two important relationships over to the new line, A'B'. The fractional part, the "f-axis," is simply the fractional part of the new line, A'B': f*A'B' Next, apply the distance to the "d-axis," which is perpendicular to the new line. The unit d-vector is (^A'B')/|A'B'|. So, with the new origin at A', the source pixel P (represented as a vector from the origin) is transformed into the destination pixel P'; see Example 1(d). This algorithm uses the relationship a point has with the original line and applies it to the new line. To be of real use, however, you need more lines. They all have to compete for pixels. As mentioned in the text, this implementation uses a weighting proportional to one over the distance squared.

In the calculation for two control lines, the distance to the point is computed from each line, and a new pixel is calculated as before. But this time there are two lines and therefore two warped pixels, so some further work is needed. As shown in Figure 8, from the original point P to the new points P1' and P2' there are two displacements, D1 and D2; see Example 1(e). The routine calculates the weighted average of the two displacements to arrive at the final position, P'; see Example 1(f). If there are three lines, there are three displacements to average, and so on. Finally, the general equation for calculating with n lines is shown in Example 1(g).

--S.A.

Example 1(a): Dot product of vector AP with the perpendicular to AB; (b) solving for d; (c) calculating the fractional part of line AB; (d) transforming the source pixel P into the destination pixel P'; (e) determining the displacements D1 and D2; (f) weighted average of the two displacements; (g) solving for the general case using n lines.

Figure 8: Where two lines contribute influence, the warped point is the weighted average of the displacements D1 and D2.

[LISTING ONE]

/****************************************************************
* FILE: linecalc.c
* DESC: Warping calculations and line-handling functions.
* HISTORY:  Created   3/11/1993   LAST CHANGED:  5/ 6/1993
*   Copyright (c) 1993 by Scott Anderson
****************************************************************/

/* ----------------------INCLUDES----------------------------- */
#include <conio.h>
#include <stdio.h>
#include <io.h>
#include <math.h>
#include <graph.h>
#include <malloc.h>
#include <memory.h>
#include <string.h>
#include "define.h"

/* -----------------------MACROS------------------------------ */
#define PIXEL(p,x,y)    (p->pixmap[y * (long) p->wide + x])
#define SQUARE(x)       (((long) x)*(x))

/* ----------------------PROTOTYPES--------------------------- */
/**** line routines ****/
int     xorLine(int x1, int y1, int x2, int y2);
int     getLine(int *argx1, int *argy1, int *argx2, int *argy2);
int     findPoint(LINE_LIST *lineList, int * line, int * point, int x, int y);
int     movePoint();
/**** warping and morphing routines ****/
int     sumLines(PICTURE *picture, COLOR *color, LINE *origline,
                                                 POINT *warp, LINE *warpline);
float   getSourceLoc(POINT *orig, LINE *origline, POINT *warp, LINE *warpline);
int     setLength(LINE *line);
void    setupScreen(PICTURE *pic, int editFlag);

/* ----------------------EXTERNALS---------------------------- */
/* set from last picture loaded */
extern int      Xmin, Ymin, Xmax, Ymax;
extern int      NumLines;
extern LINE     SrcLine[MAX_LINES];
extern LINE     DstLine[MAX_LINES];

/* ----------------------GLOBAL DATA-------------------------- */
int     TargFlag=0;
/********   These are the basic warping calculations   **********/
* FUNC: int sumLines(PICTURE *picture, COLOR *color,
*               LINE *origline, POINT *warp, LINE *warpline)
* DESC: Sum and weight the contribution of each warping line
*****************************************************************/
int
sumLines(PICTURE *picture, COLOR *color, LINE *origline,
                                                   POINT *warp, LINE *warpline)
{
    int     x, y;
    float   weight, weightSum;
    float   distance;
    int     line;
    POINT   orig;
    int     paletteIndex;
    float   deltaSumX = 0.0;
    float   deltaSumY = 0.0;
    /* if no control lines, get an unwarped pixel */
    if (NumLines == 0)
        orig = *warp;
    else {
        weightSum = 0.0;
        for (line = 0; line < NumLines; line++, origline++, warpline++) {
            distance = getSourceLoc(&orig,origline,warp,warpline);
            weight = 1/(.001+distance*distance);
            deltaSumX += (orig.x - warp->x) * weight;
            deltaSumY += (orig.y - warp->y) * weight;
            weightSum += weight;
        }
        orig.x = warp->x + deltaSumX / weightSum + .5;
        orig.y = warp->y + deltaSumY / weightSum + .5;
    }
    /* clip it to the nearest border pixel */
    x = clip(orig.x, Xmin, Xmax);
    y = clip(orig.y, Ymin, Ymax);
    paletteIndex = PIXEL (picture, x, y);
    color->r = picture->pal.c[paletteIndex].r;
    color->g = picture->pal.c[paletteIndex].g;
    color->b = picture->pal.c[paletteIndex].b;
    return (paletteIndex);
}
/*****************************************************************
* FUNC: float getSourceLoc(POINT *orig, LINE *origline,
*                                 POINT *warp, LINE *warpline)
* DESC: For a given line, locate the corresponding warped pixel
*****************************************************************/
float
getSourceLoc(POINT *orig, LINE *origline, POINT *warp, LINE *warpline)
{
    float fraction, fdist;
    int dx, dy;
    float distance;
    dx = warp->x - warpline->p[0].x;
    dy = warp->y - warpline->p[0].y;
    fraction = (dx * (long) warpline->delta_x + dy
             * (long) warpline->delta_y) / (float) (warpline->length_square);
    fdist = (dx * (long) -warpline->delta_y + dy
             * (long) warpline->delta_x) / (float) warpline->length;
    if (fraction <= 0 )
        distance = sqrt(dx*(long) dx + dy * (long) dy);
    else if (fraction >= 1) {
        dx = warp->x - warpline->p[1].x;
        dy = warp->y - warpline->p[1].y;
        distance = sqrt(dx*(long) dx + dy * (long) dy);
    }
    else if (fdist >= 0)
        distance =  fdist;
    else
        distance = -fdist;
    orig->x = origline->p[0].x + fraction * origline->delta_x -
                fdist * origline->delta_y / (float) origline->length + .5;
    orig->y = origline->p[0].y + fraction * origline->delta_y +
                fdist * origline->delta_x / (float) origline->length + .5;
    return distance;
}
/*****************************************************************
* FUNC: int setLength(LINE *line)
* DESC: Set the deltas, the length and the length squared for a given line.
*****************************************************************/
int
setLength (LINE *line)
{
    line->delta_x = line->p[1].x - line->p[0].x;
    line->delta_y = line->p[1].y - line->p[0].y;
    line->length_square = SQUARE(line->delta_x) + SQUARE(line->delta_y);
    line->length = sqrt(line->length_square);
}
/*********************  The line routines  **********************/
* FUNC: int xorLine(int x1, int y1, int x2, int y2)
* DESC: Draw a line on the screen using the XOR of the screen index.
*****************************************************************/
int
xorLine(int x1, int y1, int x2, int y2)
{
    int oldcolor = _getcolor();
    _setcolor(WHITE);   /* Use white as the xor color */
    _setwritemode(_GXOR);
    _moveto (x1,y1);
    _lineto (x2,y2);
    _setcolor(oldcolor);    /* restore the old color */
}
/*****************************************************************
* FUNC: int getLine(int *argx1, int *argy1, int *argx2, int*argy2)
* DESC: Input a line on the screen with the mouse.
*****************************************************************/
int
getLine (int *argx1, int *argy1, int *argx2, int *argy2)
{
    int     x1,y1, x2,y2;
    int     oldx, oldy;
    int     input;
    /* save the current mode */
    short   old_mode = _getwritemode();
    /* get input until we have a real line, not just a point */
    do {
        /* wait for button or key press */
        while (!(input = mousePos (&x1, &y1)));
        if (input & KEYPRESS) {
            _setwritemode(old_mode);
            return 1;
        }
        oldx=x1, oldy=y1;
        hideMouse();
        /* prime the pump with this dot */
        xorLine (x1, y1, oldx, oldy);
        showMouse();
        while (input = mousePos (&x2, &y2)) {
            /* rubber band a line while the mouse is dragged */
            if (x2 != oldx || y2 != oldy)
            {
                hideMouse();
                xorLine (x1, y1, oldx, oldy);
                xorLine (x1, y1, x2, y2);
                showMouse();
                oldx=x2, oldy=y2;
            }
        }
    } while (x1 == x2 && y1 == y2);
    *argx1 = x1, *argy1 = y1;
    *argx2 = x2, *argy2 = y2;
    _setwritemode(old_mode);        /* get out of XOR mode */
    return (0);
}
/*****************************************************************
* FUNC: int findPoint(LINE_LIST *lineList,int * line,int * point,int x, int y)
* DESC: loop thru dstline and find point within GRAB_DISTANCE,
*       return 1 if found, 0 otherwise.
*****************************************************************/
int
findPoint (LINE_LIST *lineList, int * line, int * point, int x, int y)
{
    int l, p;
    int minl, minp;
    long length;
    long minlength = SQUARE(640) + SQUARE(480);
    for (l = 0; l < lineList->number; l++) {
        for (p = 0; p <= 1; p++) {
            length = SQUARE(lineList->line[l].p[p].x - x)
                   + SQUARE(lineList->line[l].p[p].y - y);
            if (length < minlength) {
                minlength = length;
                minl = l;
                minp = p;
            }
        }
    }
    if (minlength > GRAB_DISTANCE)
        return 0;
    *line = minl;
    *point = minp;
    return 1;
}
/*****************************************************************
* FUNC: int movePoint(LINE_LIST *lineList)
* DESC: Grab a point and move it. Return 1 when key is pressed, else return 0.
*****************************************************************/
int
movePoint(LINE_LIST *lineList)
{
    int     stuckx, stucky, movex,movey;
    int     oldx, oldy;
    int     input;
    int     line, point;
    /* save the current mode */
    short   old_mode = _getwritemode();
    do {
        /* keep getting input until we have a mouse button */
        while (!(input = mousePos (&movex, &movey)));
        if (input & KEYPRESS) {
            _setwritemode(old_mode);
            return 1;
        }
        if (!findPoint(lineList, &line, &point, movex, movey)) {
            _setwritemode(old_mode);
            return 0;
        }
        /* establish fixed end point */
        stuckx = lineList->line[line].p[1-point].x;
        stucky = lineList->line[line].p[1-point].y;
        oldx=movex, oldy=movey;
        hideMouse();
        /* erase the old line */
        xorLine (stuckx, stucky,
                lineList->line[line].p[point].x,
                lineList->line[line].p[point].y);
        /* and prime the pump with the new line */
        xorLine (stuckx, stucky, oldx, oldy);
        showMouse();
        while (input = mousePos (&movex, &movey)) {
            /* rubber band a line while the mouse is dragged */
            if (movex != oldx || movey != oldy) {
                hideMouse();
                xorLine (stuckx, stucky, oldx, oldy);
                xorLine (stuckx, stucky, movex, movey);
                showMouse();
                oldx=movex, oldy=movey;
            }
        }
    } while (stuckx == movex && stucky == movey);
    lineList->line[line].p[point].x = movex;
    lineList->line[line].p[point].y = movey;
    _setwritemode(old_mode);        /* get out of XOR mode */
    return (0);
}
/*****************************************************************
* FUNC: void    createLines(PICTURE *pic, LINE_LIST *lineList)
* DESC: create a list of line segments for a picture
*****************************************************************/
void
createLines(PICTURE *pic, LINE_LIST *lineList)
{
    setupScreen(pic, 0);    /* set for enter prompt */
    initMouse();
    showMouse();
    for (lineList->number = 0;lineList->number < MAX_LINES;
                                                        lineList->number++) {
        if (getLine(&lineList->line[lineList->number].p[0].x,
            &lineList->line[lineList->number].p[0].y,
            &lineList->line[lineList->number].p[1].x,
            &lineList->line[lineList->number].p[1].y))
            break;
    }
    hideMouse();
}
/*****************************************************************
* FUNC: void    editLines(PICTURE *pic, LINE_LIST *lineList)
* DESC: move around some existing lines
*****************************************************************/
void
editLines(PICTURE *pic, LINE_LIST *lineList)
{
    int segment;
    setupScreen(pic, 1);    /* set for edit prompt */
    initMouse();
    for (segment = 0; segment < lineList->number; segment++) {
       xorLine(lineList->line[segment].p[0].x, lineList->line[segment].p[0].y,
               lineList->line[segment].p[1].x, lineList->line[segment].p[1].y);
    }
    showMouse();
    /* move the endpoints around */
    while(!movePoint(lineList));
    hideMouse();
}
/*****************************************************************
* FUNC: void    setupScreen(PICTURE *pic, int editFlag)
* DESC: Print a message introducing the screen, wait for input,
*       then set the graphics mode and display the screen.
*****************************************************************/
void
setupScreen(PICTURE *pic, int editFlag)
{
    static char *editMess[2] = {"enter", "edit"};
    static char *targMess[2] = {"source", "target"};
    setTextMode();
    _settextposition(VTAB, HTAB);
    printf("When you are ready to %s the control lines", editMess[editFlag]);
    _settextposition(VTAB+2, HTAB);
    printf("for the %s image, press any key.", targMess[TargFlag]);
    waitForKey();
    setGraphicsMode();
    displayPicture(pic);
}

[LISTING TWO]

/****************************************************************
* FILE: define.h
* DESC: These are the main defines for dissolve, warp,  morph, load and fix.
* HISTORY:  Created  1/11/1993   LAST CHANGED:  5/ 6/1993
*   Copyright (c) 1993 by Scott Anderson
****************************************************************/

/* ----------------------DEFINES------------------------------ */
#define ON              1
#define OFF             0
#define MAX_TWEENS      99      /* Maximum tweens (2 digits) */
/* minus 2 digit tween# appended to end */
#define MAX_NAME_SIZE   (8-2)
#define HEADER_LEN      128     /* PCX header length */
/* Number of colors in the palette */
#define COLORS          256
/* bytes in palette (COLORS*3) */
#define PALETTE_SIZE    (3*COLORS)
/* Maximum number of morphing lines */
#define MAX_LINES       32
/* max number of pixels wide we handle */
#define MAX_WIDE        320
/* max number of pixels tall we handle */
#define MAX_TALL        200
/* Size of screen buffer */
#define MAX_BYTES       (MAX_WIDE*(long) MAX_TALL)
/* Number of components per color (RGB) */
#define COMPS           3
/* largest color component value */
#define MAX_COMP        32
/* the midpoint of the colors - for gray */
#define MID_COMP        (MAX_COMP/2)
/* enough to handle about 10 different palettes */
#define MAX_FREQ        1023
#define MAX_FILES       10
/* length of a file name including directory */
#define MAX_PATHLEN             80
#define ENTER           13  /* Keyboard values */
#define ESC         27
#define HTAB            18  /* Position for text messages */
#define VTAB            8
/* The mouse button & keyboard constants */
#define NO_BUTTON       0
#define LEFT_BUTTON     1
#define RIGHT_BUTTON            2
#define KEYPRESS        4
/* the square of min dist for grabbing pt */
#define GRAB_DISTANCE   25
/* Some of the graphics colors */
#define BLACK           0
#define WHITE           255
#define EXT_PCX         ".PCX"  /* pcx file extension */
/* primary line file holder extension */
#define EXT_LINE1       ".LN1"
#define EXT_LINE2       ".LN2"  /* aux file for warp lines */
#define ERROR           -1  /* General-purpose error code */

typedef enum {
    NO_ERROR,              /* first entry means everything is ok */
    MEMORY_ERR,            /* Not enough memory */
    READ_OPEN_ERR,         /* Couldn't open file for reading */
    READ_ERR,              /* Trouble reading the file */
    WRITE_OPEN_ERR,        /* Couldn't open the file for writing */
    WRITE_ERR,             /* Couldn't write the file */
    MOUSE_ERR,             /* No mouse driver found */
    WRONG_PCX_FILE,        /* PCX file format not supported yet */
    READ_CONTENTS_ERR      /* error in .LN file */
}
ERR;
/* -----------------------MACROS------------------------------ */
#define MIN(a,b)        (((a)<(b)) ? (a) : (b))
#define PIXEL(p,x,y)    (p->pixmap[y * (long) p->wide + x])
#define SQUARE(x)       (((long) x)*(x))
/* ----------------------TYPEDEFS----------------------------- */
typedef struct {
    int x,y;        /* the screen coordinates of the point */
}
POINT;
typedef struct {
    POINT p[2];
}
LINE_SEGMENT;
typedef struct {
    int number;     /* number of segments to follow */
    LINE_SEGMENT line[MAX_LINES];
    char *filename; /* name of file holding the line list */
}
LINE_LIST;
typedef struct {
    POINT p[2];             /* the endpoints */
    int delta_x, delta_y;   /* x & y displacement */
    float length;           /* the precalculated length of the line */
    long length_square;     /* the length squared */
}
LINE;
typedef struct {
    /* red, green, and blue color components */
    unsigned char r, g, b;
}
COLOR;
typedef struct {
    COLOR c[COLORS];        /* a 256 entry palette */
}
PALETTE;
typedef struct {
    int xmin, ymin;         /* the upper left corner */
    int xmax, ymax;         /* the lower right corner */
    int wide, tall;         /* the width and height */
    int pal_id;         /* an ID number for each palette */
    PALETTE pal;            /* the actual palette is here */
    unsigned char far *pixmap;  /* a pointer to the pixel map */
}
PICTURE;
typedef struct linko {
    struct linko *next;
    char        *str;
}
LINKED_LIST;
/* ----------------------PROTOTYPES--------------------------- */
/**** file handling routines ****/
extern PICTURE  *loadPicture(char *filename);
extern int      loadPalette(FILE *fp, PALETTE *palette);
extern int      getBlock (unsigned char *byte, int *count, FILE *fp);
extern int      mustRead(FILE *fp, char *buf, int n);
extern int      saveScreen(PALETTE *pal);
extern int      putBlock(unsigned char num, unsigned char color, FILE *fp);
extern int      writeByte(unsigned char *byte, FILE *fp);
/**** screen and color routines ****/
extern int      defaultPalette(PALETTE *palette);
extern int      setPalette(PALETTE *palette);
extern int      displayPicture(PICTURE *picture);
extern int      displayNoPal(PICTURE *picture);
extern int      freePicture(PICTURE *pic);
/**** mouse routines ****/
extern int      initMouse();
extern int      hideMouse();
extern int      showMouse();
extern int      mousePos(int *x, int *y);
/**** general purpose routines ****/
extern int      clip(int num, int min, int max);
extern int      quitCheck();
extern void     quit(int err, char *name);
extern int      wait(int count);
extern int      waitForKey();
extern char     lineAsk(char *name);
/* ----------------------GLOBAL DATA-------------------------- */
extern int      TargFlag;
extern int      Key;

[LISTING THREE]

/****************************************************************
* FILE: morph.c
* DESC: Create a metamorphosing sequence between two given images. This
*     program lets you specify two files to morph, then prompts you for
*     control lines. It uses the lines to warp the underlying images a step
*     at a time, combine them, and optionally save them as numbered PCX files.
* HISTORY:  Created  1/13/1993    LAST CHANGED:  5/ 6/1993
*   Copyright (c) 1993 by Scott Anderson
****************************************************************/

/* ----------------------INCLUDES----------------------------- */
#include <conio.h>
#include <stdio.h>
#include <io.h>
#include <math.h>
#include <graph.h>
#include <malloc.h>
#include <memory.h>
#include <string.h>
#include "define.h"

/* ----------------------DEFINES------------------------------ */
#define MORPH_TWEENS    1

/* ----------------------PROTOTYPES--------------------------- */
int     tweenMorph(PICTURE *src, PICTURE *dst);

/* ----------------------EXTERNALS---------------------------- */
/**** color routines ****/
extern int  closestColor(int r, int g, int b, PALETTE *palPtr);
extern void collapseColors(PALETTE *palPtr);
/**** line routines ****/
extern int  setLength(LINE *line);
extern int  sumLines(PICTURE *picture, COLOR *color,
                LINE *origline, POINT *warp, LINE *warpline);
/**** io routines ****/
extern LINE_LIST    *loadLines(char *filename, char *extension);
extern void          saveLines(char *filename,
                    LINE_LIST *lineList, char *extension);

/***** variables used to compute intermediate images ****/
/* number of colors in tweened image before reduction*/
extern int  Ncolors;
/* r, g, b frequency counter array */
extern unsigned int far Freq[MAX_COMP][MAX_COMP][MAX_COMP];
/* tweened images red, grn, and blu components*/
extern unsigned char far Red[MAX_WIDE][MAX_TALL];
extern unsigned char far Grn[MAX_WIDE][MAX_TALL];
extern unsigned char far Blu[MAX_WIDE][MAX_TALL];
extern PALETTE TweenPal;            /* resulting palette */

/**** other variables ****/
extern char     *OutFilename;
/* set from last picture loaded */
extern int      Xmin, Ymin, Xmax, Ymax;
/* ID of palette currently being displayed */
extern int      CurrentPal;

/* ----------------------GLOBAL DATA-------------------------- */
PICTURE *Src;       /* source & destination picture pointers */
PICTURE *Dst;
LINE    SrcLine[MAX_LINES];
LINE    DstLine[MAX_LINES];
int     Tweens;
int     NumLines;

/*****************************************************************
* FUNC: main (int argc, char *argv[])
* DESC: Read in a filename to load
*****************************************************************/
main (int argc, char *argv[])
{
    int     segment;
    LINE_LIST *lineSrcList;
    LINE_LIST *lineDstList;
    char    answer;
    /* load the pcx file if one is given */
    if ((3 > argc) || (argc > 5)) {
        printf("Usage: morph <source><dest> [<steps> [<output>]]\n\n");
        printf("Where: <source> is the source PCX filename\n");
        printf("       <dest>   is the destination filename\n");
        printf("       <steps>  is the optional sequence size\n");
        printf("                (the max is %d, the default is %d)\n",
                        MAX_TWEENS, MORPH_TWEENS+2);
        printf("       <output> is the optional output filename\n");
        printf("                (defaults to no output)\n\n");
        printf("Note:  The output filename can be at most %d
                                         characters long.\n", MAX_NAME_SIZE);
        printf("       The PCX extension is added automatically,
                                                                 so don't\n");
        printf("       include it in the filename.\n");
        printf("       Morph only accepts PCX files with %d X %d
                                            resolution\n", MAX_WIDE, MAX_TALL);
        printf("       and %d colors.\n", COLORS);
        exit(0);
    }
    if (argc > 3) {
        /* subtract two from the series count to get the tweens
         * since the starting and ending frame are included. */
        Tweens = clip (atoi(argv[3]) - 2, 1, MAX_TWEENS);
        if (argc > 4)
            OutFilename = argv[4];
    }
    else
        Tweens = MORPH_TWEENS;
    printf("Loading the file %s\n", argv[1]);
    Src = loadPicture(argv[1]);
    if (Src == NULL)
        quit(MEMORY_ERR, "");
    printf("Loading the file %s\n", argv[2]);
    Dst = loadPicture(argv[2]);
    if (Dst == NULL)
        quit(MEMORY_ERR, "");
    lineSrcList = loadLines(argv[1], EXT_LINE1);
    if (lineSrcList->number != 0) {
        if (lineAsk(argv[1]) == 'N')
            createLines(Src, lineSrcList);
        else
            editLines(Src, lineSrcList);
    }
    else
        createLines(Src, lineSrcList);

    TargFlag = 1;   /* For the screen intro message */
    NumLines = lineSrcList->number;
    if (NumLines) {
        lineDstList = loadLines(argv[2], EXT_LINE1);
            /* inconsistent warp target*/
        if (lineDstList->number !=  NumLines)
            lineDstList->number = 0;
        if (lineDstList->number) {  /* ask what he wants to do */
            if (lineAsk(argv[2]) == 'N')
                lineDstList->number = 0;
        }
        if (lineDstList->number == 0) { /* create a warp target */
            /* copy the source lines */
            lineDstList->number = NumLines;
            for (segment = 0; segment < NumLines; segment++)
                lineDstList->line[segment] = lineSrcList->line[segment];
        }
        editLines(Dst, lineDstList);
        saveLines(argv[1], lineSrcList, EXT_LINE1);
        saveLines(argv[2], lineDstList, EXT_LINE1);
        beep();
        for (segment = 0; segment < NumLines; segment++) {
            DstLine[segment].p[0]=lineDstList->line[segment].p[0];
            DstLine[segment].p[1]=lineDstList->line[segment].p[1];
            setLength(&DstLine[segment]);
            SrcLine[segment].p[0]=lineSrcList->line[segment].p[0];
            SrcLine[segment].p[1]=lineSrcList->line[segment].p[1];
            setLength(&SrcLine[segment]);
        }
    }
    tweenMorph(Src, Dst);
    setTextMode();
}
/*****************************************************************
* FUNC: int tweenMorph(PICTURE *src, PICTURE *dst)
* DESC: calculate a pixel to plot, from the warping function
*****************************************************************/
#define TOTAL_WEIGHT        (100)   /* Good for up to 99 tweens */
tweenMorph(PICTURE *src, PICTURE *dst)
{
    int color;
    POINT warp;
    int x,y;
    COLOR scolor, dcolor;
    LINE warpLine[MAX_LINES];
    int t, i, p;
    int r, g, b;
    unsigned int srcweight, srcpaletteindex;
    unsigned int dstweight, dstpaletteindex;
    displayPicture(src);
    saveScreen(&src->pal);
    /* src is on screen, now tween to the target */
    for (t = 1; t <= Tweens; t++) {
        /* Tween the lines used to warp the images */
        for (i = 0; i < NumLines; i++) {
            for (p = 0; p < 2; p++) {
                warpLine[i].p[p].x = SrcLine[i].p[p].x +
                  ((DstLine[i].p[p].x - SrcLine[i].p[p].x) * t)
                  /(Tweens+1);
                warpLine[i].p[p].y = SrcLine[i].p[p].y +
                  ((DstLine[i].p[p].y - SrcLine[i].p[p].y) * t)
                  /(Tweens+1);
            }
            setLength(&warpLine[i]);
        }
        dstweight = t * TOTAL_WEIGHT / (Tweens+1);
        srcweight = TOTAL_WEIGHT - dstweight;
        /* Zero out the buffers */
        initFreq();
        /* set background to black */
        _fmemset(Red, 0, sizeof Red);
        _fmemset(Grn, 0, sizeof Grn);
        _fmemset(Blu, 0, sizeof Blu);
        /* Go through the screen and get warped source pixels */
        for (warp.y = Ymin; warp.y <= Ymax; warp.y++)   {
            if (quitCheck())
                quit(0, "");
            for (warp.x = Xmin; warp.x <= Xmax; warp.x++)   {
                 sumLines(src, &scolor, SrcLine, &warp, warpLine);
                 sumLines(dst, &dcolor, DstLine, &warp, warpLine);
                 r = (scolor.r * srcweight + dcolor.r * dstweight)
                            / TOTAL_WEIGHT;
                 g = (scolor.g * srcweight + dcolor.g * dstweight)
                            / TOTAL_WEIGHT;
                 b = (scolor.b * srcweight + dcolor.b * dstweight)
                            / TOTAL_WEIGHT;
                 if (Freq[r][g][b] == 0)    /* A new color */
                    Ncolors++;
                /* Keep it to one byte */
                if (Freq[r][g][b] < MAX_FREQ)
                    Freq[r][g][b]++;
                /* put RGB components into temporary buffer */
                Red[warp.x][warp.y] = r;
                Grn[warp.x][warp.y] = g;
                Blu[warp.x][warp.y] = b;
            }
        }
        collapseColors(&TweenPal);
        setPalette(&TweenPal);
        for (y = Ymin; y <= Ymax; y++)  {
            if (quitCheck())
                quit(0, "");
            for (x = Xmin; x <= Xmax; x++)  {
                color = closestColor(   Red[x][y], Grn[x][y],
                                        Blu[x][y], &TweenPal);
                _setcolor (color);
                _setpixel (x, y);
            }
        }
        /* no output file name on command line */
        if (!OutFilename) {
            beep();
            waitForKey();   /* so pause to enjoy the pictures */
        }
        else
            saveScreen(&TweenPal);
    }
    if (OutFilename) {  /* save the last pic in this series */
        CurrentPal = 0;         /* force a new palette */
        displayPicture(dst);
        saveScreen(&dst->pal);
    }
}

[LISTING FOUR]

/****************************************************************
* FILE: load.c
* DESC: This program loads a PCX file or a list of them. It crams as
*       many into memory as it can, then it flips quickly through them.
* HISTORY:  Created  1/13/1993   LAST CHANGED:  3/20/1993
*   Copyright (c) 1993 by Scott Anderson
****************************************************************/
/* ----------------------INCLUDES----------------------------- */
#include <conio.h>
#include <stdio.h>
#include <io.h>
#include <math.h>
#include <graph.h>
#include <string.h>
#include "define.h"
/* ----------------------EXTERNALS---------------------------- */
/* External functions */
extern int  quitCheck();
extern LINKED_LIST  *rootSequence(int argc, char *argv[]);
/* External variables */
extern int  Wait;
extern int  Key;
extern int  EndWait;
/* ----------------------GLOBAL DATA-------------------------- */
PICTURE *Src[MAX_FILES];            /* source picture pointer */
/*****************************************************************
* FUNC: main (int argc, char *argv[])
* DESC: Display the file or sequence passed on the command line. Read in as
*       many files as will fit in memory, then display them in a loop.
*****************************************************************/
main (int argc, char *argv[])
{
    int file, fileNum;
    int direction;
    int i;
    LINKED_LIST *pcxList;
    LINKED_LIST *pcxListHead;
    if (argc == 1) {
        printf("Usage: load <name>\n\n");
        printf("Where: <name> is the root name of a sequence\n");
        exit(23);
    }
    setGraphicsMode();
    file = 1;
    pcxListHead =  rootSequence(argc, argv);
    for (pcxList=pcxListHead; pcxList; pcxList = pcxList->next) {
        Src[file] = loadPicture(pcxList->str);
        if (Src[file] == NULL)
            break;
        displayPicture(Src[file++]);
    }
    fileNum = file - 1;
    if (fileNum == 1)   /* there's only one file */
        waitForKey();   /* so wait for the user to quit */
    else if (fileNum > 1) {
        file = 1;
        direction = 1;
        while (!(quitCheck())) {
            if ((file += direction) >= fileNum)
                direction = -1;
            if (file <= 1)
                direction = 1;
            displayPicture(Src[file]);
            if (EndWait && (file == 1 || file == fileNum))
                wait(Wait);
        }
    }
    /* Reset to original mode, then quit */
    setTextMode();
}

[LISTING FIVE]

/****************************************************************
* FILE: fix.c
* DESC: This program inputs a list of pictures, creates a best
*       fit palette, remaps the pictures, and writes them out.
* HISTORY:  Created  1/13/1993   LAST CHANGED: 3/10/1993
*   Copyright (c) 1993 by Scott Anderson
****************************************************************/
/* ----------------------INCLUDES----------------------------- */
#include <conio.h>
#include <stdio.h>
#include <io.h>
#include <math.h>
#include <graph.h>
#include <malloc.h>
#include <memory.h>
#include <string.h>
#include "define.h"
/* ----------------------EXTERNALS---------------------------- */
extern LINKED_LIST  *rootSequence(int argc, char *argv[]);
/**** color routines ****/
extern int      closestColor(int r, int g, int b, PALETTE *pal);
extern void     collapseColors(PALETTE *palPtr);
extern int      mergePalette(PICTURE *pic);
extern int      remapPicture(PICTURE *picPtr, PALETTE *palPtr);
/**** line routines ****/
extern int      getLine(int *argx1, int *argy1, int *argx2, int *argy2);
extern int      movePoint();
extern int      setLength(LINE *line);
/**** other variables ****/
extern char     *OutFilename;
/* set from last picture loaded */
extern int      Xmin, Ymin, Xmax, Ymax;
/* ----------------------GLOBAL DATA-------------------------- */
PICTURE *Src;       /* source & destination picture pointers */
/***** variables used to compute intermediate images ****/
/* number of colors in tweened image before reduction*/
extern int  Ncolors;
/* r, g, b frequency counter array */
extern unsigned int far Freq[MAX_COMP][MAX_COMP][MAX_COMP];
/* tweened images red, grn, and blu components*/
extern unsigned char far Red[MAX_WIDE][MAX_TALL];
extern unsigned char far Grn[MAX_WIDE][MAX_TALL];
extern unsigned char far Blu[MAX_WIDE][MAX_TALL];
extern PALETTE TweenPal;            /* resulting palette */
/*****************************************************************
* FUNC: main (int argc, char *argv[])
* DESC: Read in a list of filenames to load, change their palettes
*       to the best-fit palette, and write them out.
*****************************************************************/
main (int argc, char *argv[])
{
    int     file;
    LINKED_LIST *pcxList, *pcxListHead;
    /* load the pcx file if one is given */
    if (argc < 3) {
        printf("Usage: fix <infile> <outfile>\n\n");
        printf("Where: <infile>  is the input sequence name\n");
        printf("       <outfile> is the output sequence name\n");
        exit(0);
    }
    OutFilename = argv[argc-1];
    initFreq();
    pcxListHead =  rootSequence(argc-1, argv);
    for (pcxList = pcxListHead; pcxList; pcxList=pcxList->next) {
        printf("Loading the file %s\n", pcxList->str);
        Src = loadPicture(pcxList->str);
        if (Src == NULL)
            quit(MEMORY_ERR, "");
        mergePalette(Src);
        freePicture(Src);
    }
    collapseColors(&TweenPal);
    setGraphicsMode();
    setPalette(&TweenPal);
    for (pcxList = pcxListHead; pcxList; pcxList=pcxList->next) {
        Src = loadPicture(pcxList->str);
        if (Src == NULL)
            quit(MEMORY_ERR, "");
        remapPicture(Src, &TweenPal);
        displayNoPal(Src);
        saveScreen(&TweenPal);
        freePicture(Src);
    }
    setTextMode();
}

[LISTING SIX]

/****************************************************************
* FILE: color.c
* DESC: This file contains the color routines used by morph, dissolve and fix.
* HISTORY:  Created  3/18/1993  LAST CHANGED: 5/ 6/1993
*   Copyright (c) 1992 by Scott Anderson
****************************************************************/
#include <stdio.h>
#include <memory.h>
#include "define.h"
/* ----------------------DEFINES------------------------------ */
/* ----------------------TYPEDEFS/STRUCTS--------------------- */
/* ----------------------PROTOTYPES--------------------------- */
int     closestColor(int r, int g, int b, PALETTE *palPtr);
void    collapseColors(PALETTE *palPtr);
int     mergePalette(PICTURE *pic);
int     remapPicture(PICTURE *picPtr, PALETTE *palPtr);
int     initFreq();
/* ----------------------EXTERNALS---------------------------- */
/* set from last picture loaded */
extern int      Xmin, Ymin, Xmax, Ymax;
/* ----------------------GLOBAL DATA-------------------------- */
/* number of colors in tweened image before reduction*/
int     Ncolors;
/* r, g, b frequency counter array */
unsigned int far Freq[MAX_COMP][MAX_COMP][MAX_COMP];
/* tweened images red, grn, and blu components*/
unsigned char far Red[MAX_WIDE][MAX_TALL];
unsigned char far Grn[MAX_WIDE][MAX_TALL];
unsigned char far Blu[MAX_WIDE][MAX_TALL];
PALETTE TweenPal;           /* resulting palette */
/*****************************************************************
* FUNC: void    collapseColors(PALETTE *palPtr)
* DESC: Collapse the colors in the Freq table until
*       Ncolors < COLORS, then put it in the given color palette.
*****************************************************************/
void
collapseColors(PALETTE *palPtr)
{
    int freqCutoff;
    int r, g, b;
    int index;
    int ncolors;
    static int freqCount[MAX_FREQ+1];
    memset(freqCount, 0, sizeof freqCount);
    for (r = 0; r < MAX_COMP; r++)
        for (g = 0; g < MAX_COMP; g++)
            for (b = 0; b < MAX_COMP; b++)
                freqCount[Freq[r][g][b]]++;
    ncolors = 0;
    for (freqCutoff = COLORS-1; freqCutoff > 1; freqCutoff--) {
        ncolors += freqCount[freqCutoff];
        if (ncolors > COLORS) break;
    }
    /* Collapse color space to 256 colors */
    r = g = b = 0;
    while (Ncolors >= COLORS) {
        for (; r < MAX_COMP; r++, g=0) {
            for (; g < MAX_COMP; g++, b=0) {
                for (; b < MAX_COMP; b++) {
                    if (Freq[r][g][b] && Freq[r][g][b]
                        <= freqCutoff)
                goto castOut;   /* the ultimate no no */
                }
            }
        }
        r = g = b = 0;
        freqCutoff++;
        continue;
    castOut:
        Freq[r][g][b] = 0;  /* just remove this low freq color */
        Ncolors--;
    }
    /* build a palette out of all the remaining non zero freq's */
    index = 0;
    for (r = 0; r < MAX_COMP; r++)
        for (g = 0; g < MAX_COMP; g++)
            for (b = 0; b < MAX_COMP; b++)
                /* we have a color we need to map */
                if (Freq[r][g][b]) {
                    palPtr->c[index].r = r;
                    palPtr->c[index].g = g;
                    palPtr->c[index].b = b;
                    /* remember index in palette */
                    Freq[r][g][b] = index;
                    index++;
                }
}

/*****************************************************************
* FUNC: int closestColor(int r, int g, int b, PALETTE *palPtr)
* DESC: return the palette index of the color closest to rgb.
*****************************************************************/
int
closestColor(int r, int g, int b, PALETTE *palPtr)
{
    int index;
    int distance;
    int min_distance = 3200;    /* a big number */
    int min_index;
    /* The value in Freq is now the index into the color table */
    if (Freq[r][g][b]) return Freq[r][g][b];
    /* If zero, search for the closest color */
    for (index = 1; index < Ncolors; index++) {
        /* this is really the distance squared, but it works */
        distance =  SQUARE (r - palPtr->c[index].r) +
                    SQUARE (g - palPtr->c[index].g) +
                    SQUARE (b - palPtr->c[index].b);
        if (distance < min_distance) {
            min_distance = distance;
            min_index = index;
            if (distance <= 2) break;   /* close enough! */
        }
    }
    /* New index - for future reference */
    Freq[r][g][b] = min_index;
    return min_index;
}
/*****************************************************************
* FUNC: int mergePalette(PICTURE *picPtr)
* DESC: Merge a palette into Freq count table.
*****************************************************************/
int
mergePalette(PICTURE *picPtr)
{
    int     r, g, b;
    unsigned int    pos;
    unsigned char   index;
    PALETTE *palPtr = &picPtr->pal;
    unsigned char far *bufPtr = picPtr->pixmap;
    for (pos = 0; pos < MAX_BYTES; pos++) {
        index = *bufPtr++;
        r = palPtr->c[index].r;
        g = palPtr->c[index].g;
        b = palPtr->c[index].b;
        if (Freq[r][g][b] == 0)     /* A new color */
            Ncolors++;
        if (Freq[r][g][b] < MAX_FREQ)   /* Keep it managable */
            Freq[r][g][b]++;
    }
}
/*****************************************************************
* FUNC: int remapPicture(PICTURE *picPtr, PALETTE *palPtr)
* DESC: Remap a picture with a different palette.
*****************************************************************/
int
remapPicture(PICTURE *picPtr, PALETTE *palPtr)
{
    int     x, y;
    int     index;
    int     r, g, b;
    unsigned int    pos;
    unsigned char   lookup[COLORS];
    unsigned char far *bufPtr;
    /* Create the cross-reference lookup table */
    for (index = 0; index < COLORS; index++) {
        r = picPtr->pal.c[index].r;
        g = picPtr->pal.c[index].g;
        b = picPtr->pal.c[index].b;
        lookup[index] = closestColor(r, g, b, palPtr);
    }
    /* Save the new palette in the picture's palette */
    for (index = 0; index < COLORS; index++) {
        picPtr->pal.c[index].r = palPtr->c[index].r;
        picPtr->pal.c[index].g = palPtr->c[index].g;
        picPtr->pal.c[index].b = palPtr->c[index].b;
    }
    /* Remap the individual pixels to point to the new colors */
    for (bufPtr = picPtr->pixmap, pos = 0; pos < MAX_BYTES;
                        bufPtr++, pos++)
        *bufPtr = lookup[*bufPtr];
}
/*****************************************************************
* FUNC: int initFreq()
* DESC: zero out the frequency color space table
*****************************************************************/
int
initFreq()
{
    int bytes = (sizeof Freq) / 2;
    _fmemset(Freq, 0, bytes);
    /* divide because of element size */
    _fmemset(Freq+(bytes/sizeof *Freq), 0, bytes);
    /* Guarantee a black color */
    Freq[0][0][0] = MAX_FREQ;
    /* a grey color */
    Freq[MID_COMP-1][MID_COMP-1][MID_COMP-1] = MAX_FREQ;
    /* and a white color */
    Freq[(long)MAX_COMP-1][MAX_COMP-1][MAX_COMP-1] = MAX_FREQ;
    Ncolors = 3;
}

Copyright © 1994, Dr. Dobb's Journal

Back to Table of Contents


Example 1

Copyright © 1995, Dr. Dobb's Journal


JAN95: The RC5 Encryption Algorithm

The RC5 Encryption Algorithm

A fast, symmetric block cipher that may replace DES

Ronald L. Rivest

Ron is associate director of the MIT Laboratory for Computer Science, a coinventor of the RSA public-key cryptosystem, and a cofounder of RSA Data Security Inc. He can be contacted at rivest@theory.lcs.mit.edu. RC5 and RSA-RC5 are trademarks of RSA Data Security Inc. Patent pending.

The RC5 encryption algorithm is a fast, symmetric block cipher suitable for hardware or software implementations. A novel feature of RC5 is the heavy use of data-dependent rotations. RC5 has a variable-length secret key, providing flexibility in its security level.

RC5 is a parameterized algorithm, and a particular RC5 algorithm is designated as RC5-w/r/b. The parameters are as follows:

  • w is the word size, in bits. The standard value is 32 bits; allowable values are 16, 32, and 64. RC5 encrypts two-word blocks: plaintext and ciphertext blocks are each 2w bits long.
  • r is the number of rounds. Allowable values are 0, 1_255.
  • The number of bytes in the secret key K. Allowable values of b are 0, 1_255.
RC5 uses an "expanded key table," S, derived from the user's supplied secret key K. The size t of table S depends on the number r of rounds: S has t=2(r+1) words.

RC5 is not intended to be secure for all possible parameter values. On the other hand, choosing the maximum parameter values would be overkill for most applications.

We provide a variety of parameter settings so that users may select an encryption algorithm whose security and speed are optimized for their application, while providing an evolutionary path for adjusting their parameters as necessary in the future.

For example, RC5-32/16/7 is an RC5 algorithm with the number of rounds and the length of key equivalent to DES. Unlike unparameterized DES, however, an RC5 user can upgrade the choice for a DES replacement to an 80-bit key by moving to RC5-32/16/10.

As technology improves, and as the true strength of RC5 algorithms becomes better understood through analysis, the most appropriate parameters can be chosen. We propose RC5-32/12/16 as providing a "nominal" choice of parameters. Further analysis is needed to analyze the security of this choice.

Overview of the Algorithm

RC5 consists of three algorithms, one each for key expansion, encryption, and decryption. These algorithms use the following three primitive operations (and their inverses).

  • Two's complement addition of words, denoted by "+". This is modulo-2w addition.
  • Bit-wise exclusive-OR of words, denoted by xor.
  • A left-rotation (or "left-spin") of words: the rotation of word x left by y bits is denoted x <<< y. Only the lg(w) low-order bits of y are used to determine the rotation amount, so that y is interpreted modulo w.
The key-expansion routine expands the user's key K to fill the expanded key array S, so S resembles an array of t random binary words determined by the user's secret key K. The array S is first initialized using a linear congruential generator modulo 2w determined by some "magic constants." Then, S is mixed with the secret key K in three passes by both the + and <<< operations.

The key-expansion function has a certain amount of "one-wayness": It is not so easy to determine K from S.

For encryption, we assume that the input block is given in two w-bit registers, A and B, and the ouput is also placed in the registers A and B. Example 1 is a pseudocode version of the encryption algorithm. The output is in the registers A and B. The decryption routine is easily derived from the encryption routine.

Speed and Security

The encryption algorithm is very compact, and can be coded efficiently in assembly language on most processors. The table S is accessed sequentially, minimizing issues of cache size. The RC5 encryption speeds obtainable are yet to be fully determined. For RC5-32/12/16 on a 90-MHz Pentium, a preliminary C++ implementation compiled with the Borland C++ compiler (in 16-bit mode) performs a key setup in 220 mu sec and performs an encryption in 22 mu sec (equivalent to 360,000 bytes/sec). These timings can presumably be improved by more than an order of magnitude using a 32-bit compiler and/or assembly language--an assembly-language routine for the 486 can perform each round in eight instructions.

A distinguishing feature of RC5 is its heavy use of data-dependent rotations--the amount of rotation performed is dependent on the input data, and is not predetermined.

The use of variable rotations should help defeat differential and linear cryptanalysis since bits are rotated to "random" positions in each round.

I invite the reader to help determine the strength of RC5.

Acknowledgments

I'd like to thank Burt Kaliski, Lisa Yin, Paul Kocher, and everyone else at RSA Laboratories for their comments and constructive criticism.

References

Biham, E. and A. Shamir. A Differential Cryptanalysis of the Data Encryption Standard. Berlin: Springer-Verlag, 1993.

Matsui, Mitsuru. "The First Experimental Cryptanalysis of the Data Encryption Standard" in Proceedings CRYPTO '93. Berlin: Springer, 1994.

Example 1 Pseudocode of RC5 encryption algorithm.

Copyright © 1995, Dr. Dobb's Journal

Back to Table of Contents


Figure 1

Copyright © 1995, Dr. Dobb's Journal


Figure 2

Copyright © 1995, Dr. Dobb's Journal


JAN95: ALGORITHM ALLEY

ALGORITHM ALLEY

The GOST Encryption Algorithm

Bruce Schneier

As a cryptographer, I sometimes get to see some really exciting technology. Last year, for example, I was introduced to "GOST," my first encryption algorithm from the former Soviet Union. The algorithm, whose formal name is "Cryptographic Transformation Algorithm--GOST 28147-89," was published in 1989 by the National Soviet Bureau of Standards (also known as "GOST"). The algorithm was not classified, which implies it was for protection of civilian, not military, data. Don't ask me how it leaked out of the country.

In this month's column, I'll describe GOST and its relation to DES, and examine its security. GOST is still new to Western cryptographers, and more-detailed information about its security is sure to emerge as more people get the chance to study it.

A Description of GOST

GOST is a secret-key algorithm, similar in construction to DES. I believe it was invented by Soviet cryptographers who had examined DES. Its publication in 1989 as a Soviet standard occurred well after DES became public, but there is no way of knowing when it was developed and for how long it was used within the Soviet government.

Also like DES, GOST is a Feistel network; the algorithm iterates a simple encryption algorithm for multiple rounds. The text is first broken up into a left half, L, and a right half, R. K is the subkey for that particular round. A round, i, of either algorithm looks like this:

Li=Ri-1
Ri=Li-1 XOR f(Ri-1,Ki)

DES has 16 rounds; GOST has 32. Both algorithms can be reversed easily: Decryption is the same as encryption with the order of the K reversed.

Figure 1 is a single round of GOST. First, the right half and the ith subkey are added modulo 232 (that's just 32-bit addition, ignoring any carry). The result is then broken into eight 4-bit chunks, each of which becomes the input to a different S-box. There are eight different S-boxes in GOST; the first four bits go in the first S-box, the second four bits go in the second S-box, and so on. Each S-box is a permutation of the numbers 0 through 15. For example, an S-box might be: 7, 10, 2, 4, 15, 9, 0, 3, 6, 12, 5, 13, 1, 8, 11. In this case, if the input to the S-box is 0, the output is 7. If the input is 1, the output is 10, and so on. All eight S-boxes are different, and they are considered additional key material. The S-boxes are to be kept secret.

The outputs of the eight S-boxes are recombined into a 32-bit word, and then the entire word undergoes an 11-bit circular shift left, towards the higher-order bits. Finally, the result is added modulo 232 to the left half to become the new right half, and the right half becomes the new left half. Do this 32 times and you're done.

The subkeys are generated simply. The 256-bit key is divided into eight 32-bit blocks. The first block is used in the first round, the second block is used in the second round, and so on. At the ninth round and again at the 17th round, the cycle starts again: The first block is used in the ninth and 17th rounds, the second block is used in the 10th and 18th rounds, and so on. For the 25th through 32nd rounds, the order is reversed: The eighth block is used in the 25th round, the seventh block is used in the 26th round, the sixth block is used in the 26th round, and on and on.

Comparison with DES

For comparison, Figure 2 is a single round of DES. (DES also has an initial and final permutation; these add nothing to the algorithm's security, and are omitted for this discussion.) First, the right half is expanded from 32 to 48 bits by a fixed permutation. The result is XORed with the ith subkey, and then broken into eight 6-bit chunks. Each chunk becomes the input to a different S-box. DES has eight different S-boxes; the first six bits go in the first S-box, the second six bits go in the second S-box, and so on. Each S-box is four permutations of the numbers 0 through 15. In DES, the S-boxes are fixed and public; they are part of the standard and are known by everyone.

The outputs of the eight S-boxes are recombined into a 32-bit word, and then the entire word is permuted by another fixed permutation. Finally, the result is added modulo 232 to the left half to become the new right half, and the right half becomes the new left half. This is repeated 16 times.

DES's subkey-generation process is complicated. The 56-bit key is first divided in half. Then, each 28-bit half is circularly shifted to the left by either one or two digits, depending on the round. After the shift, 48 bits are selected by a fixed permutation.

Here again are the major differences between DES and GOST:

  • DES has a complicated procedure for generating the subkeys from the keys. GOST has a very simple procedure.
  • DES has a 56-bit key; GOST has a 256-bit key. If you add in the secret S-box permutations, GOST has a total of about 610 bits of secret information.
  • The S-boxes in DES have 6-bit inputs and 4-bit outputs; the S-boxes in GOST have 4-bit inputs and outputs. Both algorithms have eight S-boxes, but an S-box in GOST is one fourth the size of an S-box in DES.
  • DES has an irregular permutation, called a "P-box;" GOST uses an 11-bit left circular shift.
  • DES uses XOR to add the key to the right half and to add the right half to the left half; GOST uses addition modulo 232 for both of those operations.
  • DES has 16 rounds; GOST has 32.

Generating GOST's S-Boxes

The GOST standard does not discuss how to generate the S-boxes, only that they are somehow supplied. Recent discoveries of differential and linear cryptanalysis show that the choice of S-boxes greatly affects security; there are good S-boxes and bad S-boxes. The same holds true for GOST, leading some to speculate that the erstwhile Soviets would supply good S-boxes to those organizations it liked and bad S-boxes to those organizations it wished to eavesdrop on. This may very well be true, but further conversations with a GOST chip manufacturer within Russia offered another alternative. He generated the S-box permutations himself, using a random-number generator. The S-boxes in the accompanying source code were used in a Soviet banking application.

Is it Secure?

Is GOST secure? The short answer is that no one knows. Here is a longer answer:

Assuming there is no better way to break GOST than brute force, it is certainly much more secure than DES. GOST has a 256-bit key--longer if you count the secret S-boxes. Against differential and linear cryptanalysis, GOST is probably stronger. Although the random S-boxes in GOST are probably weaker than the fixed S-boxes in DES, they are part of the key. The secrecy of GOST's S-boxes adds to its resistance against differential and linear attacks. Also, both of these attack are very much dependent on the number of rounds in the algorithm: The more rounds, the harder the attack. GOST has twice as many rounds as DES, and this probably makes both differential and linear cryptanalysis infeasible.

The other building blocks are either on par or worse. GOST doesn't have the same expansion permutation that DES has. It is known that deleting this permutation weakens DES by reducing the avalanche effect; it is reasonable to believe that GOST is weaker for not having it. GOST's use of addition instead of exclusive-OR is probably no different.

The most serious change seems to be after the S-boxes: GOST's cyclic shift instead of a permutation. The DES permutation is designed to increase the avalanche effect: the rate in which a single-bit change in the input affects bits in the output. In GOST a change in one input bit affects one S-box in one round, which then affects two S-boxes in the next round, three the round after that, and so on. GOST requires eight rounds before a single change in an input affects every output bit; DES only requires five rounds. This is certainly a weakness. But remember that GOST has 32 rounds to DES's 16.

Conclusions

GOST's designers tried to achieve a balance between efficiency and security. They followed the same basic design of DES, but made some modifications to the algorithm. As Listing One shows, the result is an algorithm that is better suited for software implementation (no irritating bit twiddling). They seem to have been less sure of their algorithm's security and have tried to compensate by making the key length very large, keeping the S-boxes secret, and doubling the number of iterations. Whether or not their efforts have resulted in an algorithm more secure than DES remains to be seen.

Figure 1 One round of GOST (O+=addition modulo 232).

Figure 2 One round of DES (_=exclusive-OR).

Listing One

/* The GOST 28147-89 cipher  by Colin Plumb */

/* If you read the standard, it belabors the point of copying corresponding
 * bits from point A to point B quite a bit.  It helps to understand that
 * the standard is uniformly little-endian, although it numbers bits from
 * 1 rather than 0, so bit n has value 2^(n-1).  The least significant bit
 * of the 32-bit words that are manipulated in the algorithm is the first,
 * lowest-numbered, in the bit string.
 */

/* A 32-bit data type */
#ifdef __alpha  /* Any other 64-bit machines? */
typedef unsigned int word32;
#else
typedef unsigned long word32;
#endif

static unsigned char const k8[16] = {
    1, 15, 13, 0,  5, 7, 10,  4,  9, 2,  3, 14,  6,  11,  8,  12 };
static unsigned char const k7[16] = {
    13, 11,  4, 1,  3, 15,  5,  9,  0,  10,  14, 7, 6,  8,  2, 12 };
static unsigned char const k6[16] = {
    4,  11,  10, 0,  7,  2, 1,  13,  3, 6, 8,  5, 9,  12,  15,  14 };
static unsigned char const k5[16] = {
    6, 12, 7,  1,  5,  15,  13, 8,  4,  10,  9,  14, 0, 3,  11, 2 };
static unsigned char const k4[16] = {
    7, 13,  10,  1,  0, 8, 9,  15,  14,  4,  6, 12, 11,  2, 5,  3 };
static unsigned char const k3[16] = {
    5,  8, 1, 13,  10,  3,  4,  2,  14, 15,  12,  7, 6,  0,  9, 11 };
static unsigned char const k2[16] = {
    14, 11,  4, 12, 6,  13,  15, 10,  2, 3,  8,  1,  0, 7,  5,  9 };
static unsigned char const k1[16] = {
    4,  10,  9,  2,  13, 8, 0,  14,  6,  11,  1, 12,  7, 15, 5,  3 };

/* Byte-at-a-time substitution boxes */
static unsigned char k87[256];
static unsigned char k65[256];
static unsigned char k43[256];
static unsigned char k21[256];

/* Build byte-at-a-time subtitution tables.
 * This must be called once for global setup.
 */
void
kboxinit(void)
{
    int i;
    for (i = 0; i < 256; i++) {
        k87[i] = k8[i >> 4] << 4 | k7[i & 15];
        k65[i] = k6[i >> 4] << 4 | k5[i & 15];
        k43[i] = k4[i >> 4] << 4 | k3[i & 15];
        k21[i] = k2[i >> 4] << 4 | k1[i & 15];
    }
}

/* Do the substitution and rotation that are the core of the operation, like
 * the expansion, substitution and permutation of the DES. It would be possible
 * to perform DES-like optimisations and store the table entries as 32-bit
 * words, already rotated, but the efficiency gain is questionable.
 * This should be inlined for maximum speed
 */
#if __GNUC__
__inline__
#endif
static word32
f(word32 x)
{
    /* Do substitutions */
#if 0
    /* This is annoyingly slow */
    x = k8[x>>28 & 15] << 28 | k7[x>>24 & 15] << 24 |
        k6[x>>20 & 15] << 20 | k5[x>>16 & 15] << 16 |
        k4[x>>12 & 15] << 12 | k3[x>> 8 & 15] <<  8 |
        k2[x>> 4 & 15] <<  4 | k1[x     & 15];
#else
    /* This is faster */
    x = k87[x>>24 & 255] << 24 | k65[x>>16 & 255] << 16 |
        k43[x>> 8 & 255] <<  8 | k21[x & 255];
#endif

    /* Rotate left 11 bits */
    return x<<11 | x>>(32-11);
}

/* The GOST standard defines the input in terms of bits 1..64, with
 * bit 1 being the lsb of in[0] and bit 64 being the msb of in[1].
 * The keys are defined similarly, with bit 256 being the msb of key[7].
 */
void
gostcrypt(word32 const in[2], word32 out[2], word32 const key[8])
{
    register word32 n1, n2; /* As named in the GOST */

    n1 = in[0];
    n2 = in[1];

    /* Instead of swapping halves, swap names each round */
    n2 ^= f(n1+key[0]);
    n1 ^= f(n2+key[1]);
    n2 ^= f(n1+key[2]);
    n1 ^= f(n2+key[3]);
    n2 ^= f(n1+key[4]);
    n1 ^= f(n2+key[5]);
    n2 ^= f(n1+key[6]);
    n1 ^= f(n2+key[7]);

    n2 ^= f(n1+key[0]);
    n1 ^= f(n2+key[1]);
    n2 ^= f(n1+key[2]);
    n1 ^= f(n2+key[3]);
    n2 ^= f(n1+key[4]);
    n1 ^= f(n2+key[5]);
    n2 ^= f(n1+key[6]);
    n1 ^= f(n2+key[7]);

    n2 ^= f(n1+key[0]);
    n1 ^= f(n2+key[1]);
    n2 ^= f(n1+key[2]);
    n1 ^= f(n2+key[3]);
    n2 ^= f(n1+key[4]);
    n1 ^= f(n2+key[5]);
    n2 ^= f(n1+key[6]);
    n1 ^= f(n2+key[7]);

    n2 ^= f(n1+key[7]);
    n1 ^= f(n2+key[6]);
    n2 ^= f(n1+key[5]);
    n1 ^= f(n2+key[4]);
    n2 ^= f(n1+key[3]);
    n1 ^= f(n2+key[2]);
    n2 ^= f(n1+key[1]);
    n1 ^= f(n2+key[0]);

    /* There is no swap after the last round */
    out[0] = n2;
    out[1] = n1;
}

/* The key schedule is somewhat different for decryption. (The key table is
 * used once forward and three times backward.) You could define an expanded
 * key, or just write the code twice, as done here.
 */
void
gostdecrypt(word32 const in[2], word32 out[2], word32 const key[8])
{
    register word32 n1, n2; /* As named in the GOST */

    n1 = in[0];
    n2 = in[1];

    n2 ^= f(n1+key[0]);
    n1 ^= f(n2+key[1]);
    n2 ^= f(n1+key[2]);
    n1 ^= f(n2+key[3]);
    n2 ^= f(n1+key[4]);
    n1 ^= f(n2+key[5]);
    n2 ^= f(n1+key[6]);
    n1 ^= f(n2+key[7]);

    n2 ^= f(n1+key[7]);
    n1 ^= f(n2+key[6]);
    n2 ^= f(n1+key[5]);
    n1 ^= f(n2+key[4]);
    n2 ^= f(n1+key[3]);
    n1 ^= f(n2+key[2]);
    n2 ^= f(n1+key[1]);
    n1 ^= f(n2+key[0]);

    n2 ^= f(n1+key[7]);
    n1 ^= f(n2+key[6]);
    n2 ^= f(n1+key[5]);
    n1 ^= f(n2+key[4]);
    n2 ^= f(n1+key[3]);
    n1 ^= f(n2+key[2]);
    n2 ^= f(n1+key[1]);
    n1 ^= f(n2+key[0]);

    n2 ^= f(n1+key[7]);
    n1 ^= f(n2+key[6]);
    n2 ^= f(n1+key[5]);
    n1 ^= f(n2+key[4]);
    n2 ^= f(n1+key[3]);
    n1 ^= f(n2+key[2]);
    n2 ^= f(n1+key[1]);
    n1 ^= f(n2+key[0]);

    out[0] = n2;
    out[1] = n1;
}

/* The GOST "Output feedback" standard. It seems closer morally to the counter
 * feedback mode some people have proposed for DES.
 *
 * The IV is encrypted with the key to produce the initial counter value.
 * Then, for each output block, a constant is added, modulo 2^32-1 (0 is
 * represented as all-ones, not all-zeros), to each half of the counter, and
 * the counter is encrypted to produce the value to XOR with the output.
 *
 * Len is the number of blocks. Sub-block encryption is left as an exercise
 * for the user. Remember that the standard defines everything in a
 * little-endian manner, so you want to use the low bit of gamma[0] first.
 *
 * OFB is, of course, self-inverse, so there is only one function.
 */

/* The constants for addition */
#define C1 0x01010104
#define C2 0x01010101

void
gostofb(word32 const *in, word32 *out, int len,
    word32 const iv[2], word32 const key[8])
{
    word32 temp[2];         /* Counter */
    word32 gamma[2];        /* Output XOR value */

    /* Compute starting value for counter */
    gostcrypt(iv, temp, key);

    while (len--) {
        temp[0] += C2;
        if (temp[0] < C2)       /* Wrap modulo 2^32? */
            temp[0]++;      /* Make it modulo 2^32-1 */
        temp[1] += C1;
        if (temp[1] < C1)       /* Wrap modulo 2^32? */
            temp[1]++;      /* Make it modulo 2^32-1 */

        gostcrypt(temp, gamma, key);

        *out++ = *in++ ^ gamma[0];
        *out++ = *in++ ^ gamma[1];
    }
}

/*
 * The CFB mode is just what you'd expect.  Each block of ciphertext y[] is
 * derived from the input x[] by the following pseudocode:
 * y[i] = x[i] ^ gostcrypt(y[i-1])
 * x[i] = y[i] ^ gostcrypt(y[i-1])
 * Where y[-1] is the IV.
 *
 * The IV is modified in place.  Again, len is in *blocks*.
 */

void
gostcfbencrypt(word32 const *in, word32 *out, int len,
           word32 iv[2], word32 const key[8])
{
    while (len--) {
        gostcrypt(iv, iv, key);
        iv[0] = *out++ ^= iv[0];
        iv[1] = *out++ ^= iv[1];
    }
}
void
gostcfbdecrypt(word32 const *in, word32 *out, int len,
           word32 iv[2], word32 const key[8])
{
    word32 t;
    while (len--) {
        gostcrypt(iv, iv, key);
        t = *out;
        *out++ ^= iv[0];
        iv[0] = t;
        t = *out;
        *out++ ^= iv[1];
        iv[1] = t;
    }
}

/* The message suthetication code uses only 16 of the 32 rounds. There *is*
 * a swap after the 16th round. The last block should be padded to 64 bits
 * with zeros. len is the number of *blocks* in the input.
 */
void
gostmac(word32 const *in, int len, word32 out[2], word32 const key[8])
{
    register word32 n1, n2; /* As named in the GOST */

    n1 = 0;
    n2 = 0;

    while (len--) {
        n1 ^= *in++;
        n2 = *in++;

        /* Instead of swapping halves, swap names each round */
        n2 ^= f(n1+key[0]);
        n1 ^= f(n2+key[1]);
        n2 ^= f(n1+key[2]);
        n1 ^= f(n2+key[3]);
        n2 ^= f(n1+key[4]);
        n1 ^= f(n2+key[5]);
        n2 ^= f(n1+key[6]);
        n1 ^= f(n2+key[7]);

        n2 ^= f(n1+key[0]);
        n1 ^= f(n2+key[1]);
        n2 ^= f(n1+key[2]);
        n1 ^= f(n2+key[3]);
        n2 ^= f(n1+key[4]);
        n1 ^= f(n2+key[5]);
        n2 ^= f(n1+key[6]);
        n1 ^= f(n2+key[7]);
    }
    out[0] = n1;
    out[1] = n2;
}

#ifdef TEST

#include <stdio.h>
#include <stdlib.h>

/* Designed to cope with 15-bit rand() implementations */
#define RAND32 ((word32)rand() << 17 ^ (word32)rand() << 9 ^ rand())

int
main(void)
{
    word32 key[8];
    word32 plain[2];
    word32 cipher[2];
    int i, j;

    kboxinit();

    printf("GOST 21847-89 test driver.\n");

    for (i = 0; i < 1000; i++) {
        for (j = 0; j < 8; j++)
            key[j] = RAND32;
        plain[0] = RAND32;
        plain[1] = RAND32;

        printf("%3d\r", i);
        fflush(stdout);

        gostcrypt(plain, cipher, key);
        for (j = 0; j < 99; j++)
            gostcrypt(cipher, cipher, key);
        for (j = 0; j < 100; j++)
            gostdecrypt(cipher, cipher, key);

        if (plain[0] != cipher[0] || plain[1] != cipher[1]) {
            fprintf(stderr, "\nError! i = %d\n", i);
            return 1;
        }
    }
    printf("All tests passed.\n");
    return 0;
}
#endif /* TEST */

Copyright © 1995, Dr. Dobb's Journal

Back to Table of Contents


Example 2

Copyright © 1995, Dr. Dobb's Journal


Example 3

Copyright © 1995, Dr. Dobb's Journal


Figure 3

Copyright © 1995, Dr. Dobb's Journal


Example 5

Copyright © 1995, Dr. Dobb's Journal


FEB95: ALGORITHM ALLEY

ALGORITHM ALLEY

Faster FFTs

J.G.G. Dobbe

Iwan is a hardware and software developer in the department of medical bio-engineering at the Academic Medical Center in Amsterdam, Holland.

Introduction

by Bruce Schneier

The Fourier transform is an essential tool of modern applied mathematics. Named after the French mathematician and physicist Jean-Baptiste Joseph Fourier (1768--1830), it has found uses in all areas where information or signals are transmitted or analyzed: concert-hall acoustics, computer vision, speech analysis, aerodynamics, fluid control, broadcasting, and process control.

What Fourier devised is a way to express periodic functions as an infinite series of sine and cosine functions. These sorts of expressions are a way of putting all periodic functions within the same reference. Even though they are all different, you can think about them in the same way. The Fourier transform is the method for deriving the various coefficients for the sine and cosine functions.

The Fourier transform assumes that the function is periodic even though in the real world, functions are most often not. However, most applications only care about the behavior of a function within a certain interval. The trick is to assume that the function is periodic outside that interval, then just not worry about it. The Fourier transform of a sampled function that is assumed periodic outside the interval is called a "Discrete Fourier Transform" (DFT).

Getting a computer to do a DFT requires an incredible number of calculations and is generally not feasible. Consequently, mathematicians have invented the "Fast Fourier Transform" (FFT)--a smarter, divide-and-conquer algorithm that skips most of the DFT's unnecessary calculations and makes all of this stuff accessible. Think of the Fourier transform as a mathematical tool, and the FFT as a computational tool for mathematical applications.

In this month's "Algorithm Alley," Iwan describes two implementations of a radix 2 FFT algorithm, one written in C and the other in assembler. The assembler version can be linked to C or Pascal programs and can run a 1024-point FFT on a 486/66 in 36 msec, or in just 12 msec on a 60-MHz Pentium. This performance is comparable to what you can get from a dedicated digital signal processing (DSP) board--a 1024-point FFT in 1--4 msec--and is much cheaper. Figure 1 provides various performance results on a variety of processors.

When computers are involved in signal processing, an analog-to-digital converter must be used to digitize the incoming analog signal. This results in an array of samples representing the original analog signal. It is possible to perform a Discrete Fourier Transform (DFT) on the array of samples resulting in the discrete-density spectrum in Example 1, where fn is the nth sample in time domain. Because the number of complex spectral components (N) in the frequency domain equals the number of signal samples, the same information is available in both domains, and direct inverse transformation is permissible, regaining the original samples. This also eases programming because one array can be used for both time and frequency samples.

When computing an N-point DFT, N2 multiplications are required. Some of those factors appear several times throughout the calculation. Using a smarter computation method called the "Fast Fourier Transform" (FFT), it is possible to skip duplicate calculations, saving a lot of execution time.

The secret behind the FFT algorithm is splitting an N-point DFT into two N/2-point DFTs. Computing an N/2-point DFT takes N2/4 multiplications, so two N/2-point DFTs can be calculated with N2/2 multiplications. Note that this is half the number of calculations required to perform an N-point DFT. While some combining of the N/2-point DFTs is required to obtain the original N-point DFT, computation time is reduced almost by a factor of 2. By continuing this halving process, finally 2-point DFTs remain to be calculated. When the combination method is disregarded, FFT execution time is the result of the equation in Example 2. F(k) can be written as a combination of two N/2 point DFTs, F1(k) and F2(k), holding the even and odd samples of F(k), respectively; see Example 3. Note that the sequence of Q-factors is the same for both equations of F(k). This characteristic is used to reduce execution time. The equation for finding F(k) for one Q-factor is graphically represented by the so-called FFT "Butterfly" in Figure 2.

The computation of an N-point FFT can be graphically represented as a whole network of butterflies for calculating an 8-point FFT, as in Figure 3. The network in Figure 3 consists of sections that combine two N/2-point FFTs. If you read from right to left, the butterflies become smaller, resulting in the following 2-point DFTs:

F(0)=f(0)+f(1)
F(1)=f(0)-f(1)

Shuffle Samples

While splitting the N-point DFT into two N/2-point DFTs, one DFT consists of the even samples and the other of the odd samples. When each N/2-point DFT is split into two N/2-point DFTs, the first N/4-point DFT consists of the even samples of the N/2-point DFT, while the odd samples give the second N/4-point DFT.

By shuffling the FFT input samples in a structured way, the output data is ordered after performing the FFT. When N is a power of two, this shuffle algorithm is relatively simple. The index of an array sample in binary format can be reversed to obtain the index of the array sample for exchange. When the first half of the array is shuffled (index 0..3), the other half automatically comes in the right place; see Table 1. To find the array index for exchanging elements, a compact and fast piece of code like Example 4 can be written in assembler in which AX holds the array index for exchange with the original element (OldIndex). (The file PCFFT.ASM includes code for calculating shuffle indexes and is available electronically; see page 3.)

Speeding Up FFT Performance

The FFT function prototype void Fft(float *Re, float *Im, int Pwr, int Dir); (where Re and Im are the base addresses of the real and imaginary arrays) shows the complex array split into its real and imaginary parts. During the FFT execution, elements in the arrays are reached by adding the right offset to the base addresses. By splitting the complex array in two, the same offset applies for indexing an element in both real and imaginary arrays. This results in saving execution time, both during FFT calculation and when calculating the power spectrum from the real and imaginary elements.

Size of Complex Numbers

An FFT can be performed on any type and size of variables. Integer numbers are often small in size so less time is spent on memory access. By using floating-point numbers, accuracy of integer FFTs can be improved at the cost of execution time. Coprocessors can work with IEEE floating-point numbers of different size--short-real (32 bits), long-real (64 bits), and a so-called "temp-real" format (80 bits). However, the more bits involved, the longer it will take the coprocessor to access memory. In short, when working with floating-point numbers, the fastest algorithm can be obtained with the short-real type. Note that some Pascal compilers convert the REAL type into a 48-bits (6-byte) variant not supported by the coprocessor. In such cases, the coprocessor is emulated in software; however, execution slows down in the process.

Improving the Goniometry

When the computer executes an FFT, it must calculate the factor Qk=cos( alpha k)+i*sin( alpha k), with alpha =-2 pi /N and k=0,1,_, N-1, over and over again. Because sin and cos are time-consuming instructions (even for coprocessors), they slow down FFT performance. It is possible, however, to use a table of constants holding start values alpha for each N-point FFT and derive cos( alpha k) and sin( alpha k) from the start value by faster multiplication and subtraction; see Example 5. Therefore, Qk+1 can be calculated from Qk and the table values for cos( alpha) and sin( alpha). Note that cos( alpha) and sin( alpha) for finding the first Qk for k=0 can also be found in the tables.

Using a Coprocessor

Math coprocessors are typically stack-based. The operands must be loaded onto the stack first, before operations can be released to them. You can reduce FFT execution time by reducing the number of bus accesses. When writing coprocessor assembler code, you can leave numbers on the coprocessor stack that can be used later on during the calculation. Not all compilers are this farsighted. The calculation of sin( alpha (k+1)) and cos(A alpha (k+1)) can be computed with a very compact piece of code like that in PCFFT.ASM. With languages such as C or Pascal, the efficiency of the code is at the mercy of the compiler and often is not easy to see through and improve.

Faster FFT Code

I've implemented the FFT algorithm described here in both C and assembler. PCFFT.C (Listing One) is the C version, while PCFFT.ASM (available electronically) is written in assembler. Listing Two is the header file for PCFFT.C. Both C and assembler versions use the same function prototype: void Fft(float *Re, float *Im, int Pwr, int Dir);, where Re, Im are pointers to arrays of (32-bit) floating-point numbers holding the real and imaginary part of the input, respectively; Pwr holds the size of the arrays as a power of two (for example, when a 1024-point FFT is to be calculated, Pwr should be equal to 10); and Dir determines whether an FFT (Dir [image] 1) or an inverse FFT (Dir [image] 0) should be performed.

PCFFTest.C (Listing Three) includes the PCFFT.H header file and shows how the FFT function must be called. You can determine whether to use the C or assembler version by linking the right object file.

As Figure 1 illustrates, the assembler version is the fastest and makes direct use of your system's coprocessor. Set your compiler to generate the fastest code (80x86); when using the assembler version, define the COPROC287 symbol at the assembler command line--if your system has a 80x87 coprocessor--for maximum performance.

The assembler version can also be used to link with Borland Pascal, by defining the PSCL symbol at the assembler command line. When this is done, the correct parameter passing sequence is applied. With Borland Turbo Assembler (TASM 3.0), use the command lines in Example 6 to generate the object file (PCFFT.OBJ) of your choice.

References

Kaldewaij, A. and J. van Tiel. Voortgezette Wiskunde; Fourier-theorie en systeemtheorie. Utrecht, The Netherlands: Scheltema & Holkema, 1983.

Nieland, H.M. "Fourier Analyse." Natuur en Techniek (no. 3, 1990).

Lawrence, R.R., and B. Gold. Theory and Application of Digital Signal Processing. Englewood Cliffs, NJ: Prentice Hall, 1975.

Pettit, F. Fourier Transforms in Action. Chartwell-Bratt Ltd. (Publishing and Training), Bromley, U.K., 1985.

Press, William et al. Numerical Recipes in C, Second Edition. Cambridge: Cambridge University Press, 1992.

Example 1 Performing a DFT on an array of samples results in the discrete density spectrum. Example 2 When the combination method is disregarded, FFT execution time is the result of this equation. Example 3 F(k) can be written as a combination of two N/2-point DFTs, F1(k) and F2(k), holding the even and odd samples of F(k), respectively. Figure 1 FFT execution time on PC using Borland C++ 3.1 and assembler. The assembler code reduces execution time by about another 35 percent. Figure 2 FFT "Butterfly," where the "hub" represents a summation/ subtraction point, while the arrow represents a multiplication.

Table 1: Before performing the FFT, array elements are shuffled. The shuffle index for element X is found by placing its binary format in reversed order.

    Old     Binary   Reversed   New
    Index   Format   Binary     Index
                     Format
    0       000      000        0
    1       001      100        4
    2       010      010        2
    3       011      110        6
    4       100      001        1
    5       101      101        5
    6       110      011        3
    7       111      111        7

Figure 3 8-point FFT in "Butterfly" notation. Reading the graph from left to right shows how N/2-point FFTs are combined. Reading from right to left shows how an N-point FFT is reduced to the calculation of 2-point DFTs.

Example 4: Assembler code to find shuffle index.

    MOV    BX, OldIndex   ; BX = Old index
    MOV    AX, 0000H      ; AX = 0
    MOV    CX, WordLength ; CX = Wordlength in bits
    CLC                   ; Clear carry flag
NextBit:
    RCR    BX,1           ; Rotate BX through carry right 1 bit
    RCL    AX,1           ; Rotate AX left using carry 1 bit
    LOOP   NextBit        ; Repeat till whole word reversed

Example 5 Deriving cos ( alpha (k+1)) and sin ( alpha (k+1)) from a table of constants.

Example 6: Command lines for (a) C-linkable code and (b) Pascal-linkable code.

(a)
For XT: TASM pcfft.asm
For AT: TASM /dCOPROC287=1 pcfft.asm
(b)
For XT: TASM /dPSCL=1 pcfft.asm
For AT: TASM /dPSCL=1 /dCOPROC287=1 pcfft.asm

Listing One

/* PCFFT.C  -- by J.G.G. Dobbe -- Performs an FFT on two arrays (Re, Im) of
               type float (can be changed). This unit is written in C and
               doesn't call assembler routines.
*/

/* --------------------- Include directive ------------------------ */
#include "pcfft.h"

/* --------------------- Local variables -------------------------- */
static float CosArray[28] =
{ /* cos{-2pi/N} for N = 2, 4, 8, ... 16384 */
 -1.00000000000000,  0.00000000000000,  0.70710678118655,
  0.92387953251129,  0.98078528040323,  0.99518472667220,
  0.99879545620517,  0.99969881869620,  0.99992470183914,
  0.99998117528260,  0.99999529380958,  0.99999882345170,
  0.99999970586288,  0.99999992646572,
  /* cos{2pi/N} for N = 2, 4, 8, ... 16384 */
 -1.00000000000000,  0.00000000000000,  0.70710678118655,
  0.92387953251129,  0.98078528040323,  0.99518472667220,
  0.99879545620517,  0.99969881869620,  0.99992470183914,
  0.99998117528260,  0.99999529380958,  0.99999882345170,
  0.99999970586288,  0.99999992646572
};
static float SinArray[28] =
{ /* sin{-2pi/N} for N = 2, 4, 8, ... 16384 */
  0.00000000000000, -1.00000000000000, -0.70710678118655,
 -0.38268343236509, -0.19509032201613, -0.09801714032956,
 -0.04906767432742, -0.02454122852291, -0.01227153828572,
 -0.00613588464915, -0.00306795676297, -0.00153398018628,
 -0.00076699031874, -0.00038349518757,
  /* sin{2pi/N} for N = 2, 4, 8, ... 16384 */
  0.00000000000000,  1.00000000000000,  0.70710678118655,
  0.38268343236509,  0.19509032201613,  0.09801714032956,
  0.04906767432742,  0.02454122852291,  0.01227153828572,
  0.00613588464915,  0.00306795676297,  0.00153398018628,
  0.00076699031874,  0.00038349518757
};

/* --------------------- Function implementations ----------------- */
/* --------------------- ShuffleIndex ----------------------------- */
static unsigned int ShuffleIndex(unsigned int i, int WordLength)

/* Function     : Finds the shuffle index of array elements. The array length
                  must be a power of two; The power is stored in "WordLength".
   Return value : With "i" the source array index, "ShuffleIndex"
                  returns the destination index for shuffling.
   Comment      : -
*/
{
  unsigned int  NewIndex;
  unsigned char BitNr;
  NewIndex = 0;
  for (BitNr = 0; BitNr <= WordLength - 1; BitNr++)
  {
    NewIndex = NewIndex << 1;
    if ((i & 1) != 0) NewIndex = NewIndex + 1;
    i = i >> 1;
  }
  return NewIndex;
}
/* --------------------- Shuffle2Arr ------------------------------ */
static void Shuffle2Arr(float *a, float *b, int bitlength)
/* Function     : Shuffles both arrays "a" and "b". This function is called
                 before performing the actual FFT so the array elements
                 are in the right order after FFT.
  Return value : -
  Comment      : -
*/
{
  unsigned int  IndexOld, IndexNew;
  float         temp;
  unsigned int  N;
  int bitlengthtemp;

  bitlengthtemp = bitlength;                  /* Save for later use */
  N = 1;                                       /* Find array-length */
  do
  {
    N = N * 2;
    bitlength = bitlength - 1;
  } while (bitlength > 0) ;
                                            /* Shuffle all elements */
  for (IndexOld = 0; IndexOld <= N - 1; IndexOld++)
  {                              /* Find index to exchange elements */
    IndexNew = ShuffleIndex(IndexOld, bitlengthtemp);
    if (IndexNew > IndexOld)
    {                                         /* Exchange elements: */
      temp = a[IndexOld];                             /* Of array a */
      a[IndexOld] = a[IndexNew];
      a[IndexNew] = temp;
      temp = b[IndexOld];                             /* Of array a */
      b[IndexOld] = b[IndexNew];
      b[IndexNew] = temp;
    }
  }
}
/* --------------------- Fft -------------------------------------- */
void Fft(float *Re, float *Im, int Pwr, int Dir)

/* Function     : Actual FFT algorithm. "Re" and "Im" point to start of real
                 and imaginary arrays of numbers, "Pwr" holds the array sizes
                 as a power of 2 while "Dir" indicates whether an FFT (Dir>=1)
                 or an inverse FFT must be performed (Dir<=0).
  Return value : The transformed information is returned by "Re"
                 and "Im" (real and imaginary part respectively).
  Comment      : -
*/
{
  int pwrhelp;
  int N;
  int Section;
  int AngleCounter;
  int FlyDistance;
  int FlyCount;
  int index1;
  int index2;
  float tempr, tempi;
  float Re1, Re2, Im1, Im2;
  float c, s;
  float scale;
  float sqrtn;
  float temp;
  float Qr, Qi;

  Shuffle2Arr(Re, Im, Pwr);                /* Shuffle before (i)FFT */
  pwrhelp = Pwr;                          /* Determine size of arrs */
  N = 1;
  do
  {
    N = N * 2;
    pwrhelp--;
  } while (pwrhelp > 0) ;

  if (Dir >= 1) AngleCounter = 0;                            /* FFT */
  else          AngleCounter = 14;                   /* Inverse FFT */
  Section = 1;
  while (Section < N)
  {
    FlyDistance = 2 * Section;
    c = CosArray[AngleCounter];
    s = SinArray[AngleCounter];
    Qr = 1; Qi = 0;
    for (FlyCount = 0; FlyCount <= Section - 1; FlyCount++)
    {
      index1 = FlyCount;
      do
      {
        index2 = index1 + Section;
                                            /* Perform 2-Point DFT  */
        tempr = 1.0 * Qr * Re[index2] - 1.0 * Qi * Im[index2];
        tempi = 1.0 * Qr * Im[index2] + 1.0 * Qi * Re[index2];

        Re[index2] = Re[index1] - tempr;             /* For Re-part */
        Re[index1] = Re[index1] + tempr;
        Im[index2] = Im[index1] - tempi;             /* For Im-part */
        Im[index1] = Im[index1] + tempi;

        index1 = index1 + FlyDistance;
      } while (index1 <= (N - 1));

      /*                 k                                  */
      /*  Calculate new Q = cos(ak) + j*sin(ak) = Qr + j*Qi */
      /*          -2*pi                                     */
      /*  with: a = -----                                   */
      /*            N                                       */
      temp = Qr;
      Qr   = Qr*c - Qi*s;
      Qi   = Qi*c + temp*s;
    }
    Section = Section * 2;
    AngleCounter = AngleCounter + 1;
  }
  if (Dir <= 0)                                    /* Normalize for */
  {                                             /* inverse FFT only */
    scale = 1.0/N;
    for (index1 = 0; index1 <= N - 1; index1++)
    {
      Re[index1] = scale * Re[index1];
      Im[index1] = scale * Im[index1];
    }
  }
}
/* ---------------------------------------------------------------- */

Listing Two

/* PCFFT.H -- by J.G.G. Dobbe -- Headers for PCFFT.C */

#ifndef PCFFT_H                /* If not defined yet, use this file */
#define PCFFT_H

/* --------------------- External function ------------------------ */
void Fft(float *Re, float *Im, int Pwr, int Dir);
/* ---------------------------------------------------------------- */
#endif

Listing Three

/* PCFFTest.C -- by J.G.G. Dobbe -- Test program written in Turbo C/Borland C
                 that uses the C or Assembler version of FFT. It depends on
                 the type of FFT object file whether the C- or ASM-version is
                 linked in. In both cases, the same FFT.H file is used.
*/

/* --------------------- Include directives ----------------------- */
#include <stdio.h>
#include <conio.h>
#include "pcfft.h"

/* --------------------- Constant definition ---------------------- */
#define SIZE 16                     /* Size of data arrays (re, im) */

/* --------------------- Variable definitions --------------------- */
float re[SIZE];                  /* Array holding Real part of data */
float im[SIZE];             /* Array holding Imaginary part of data */

/* --------------------- Function implementations ----------------- */
/* --------------------- DispArr ---------------------------------- */
void DispArr(char *Txt)
{
  int i;                                            /* Loop counter */
  clrscr();                                         /* Clear screen */
  printf("\n%s:\n", Txt);                         /* Display header */
  for (i = 0; i < SIZE; i++)                 /* Display data points */
    printf("i = %4d: Re = %8.2f, Im = %8.2f\n", i, re[i], im[i]);
  printf("Press <ENTER> to continue\n");         /* Display message */
  getch();                                    /* Wait for keystroke */
}
/* --------------------- main ------------------------------------- */
int main()
{
  int i;
  for (i = 0; i < SIZE; i++)                   /* Clear data arrays */
  {
    re[i] = 0.0;
    im[i] = 0.0;
  }
  re[0] = 100.0;                    /* Fill array with pulse signal */
  DispArr("Input Time Data");   /* Display time data (pulse signal) */
  Fft(re, im, 4, 1);                          /* FFT on data points */
  DispArr("Frequency Response (after FFT)");   /* Display freq data */
  Fft(re, im, 4, -1);                   /* Inverse FFT on freq data */
  DispArr("Time Response (after inverse FFT)");
                                /* Display time data (pulse signal) */
  return 0;
}
/* ---------------------------------------------------------------- */

Copyright © 1995, Dr. Dobb's Journal

Back to Table of Contents


Table 2

Copyright © 1995, Dr. Dobb's Journal


APR95: ALGORITHM ALLEY

ALGORITHM ALLEY

Computing the Day of the Week

Kim S. Larsen

Kim has a PhD in computer science and is primarily interested in databases, algorithmics, and data structures. He can be contacted at Ærtebjerggårdvej 31, DK-5270 Odense N, Denmark or at kslarsen@imada.ou.dk.

Introduction

by Bruce Schneier

Thirty days have September, April, June, and November.... So the song goes. Or did you learn to count the months forward and backwards on your fingers, with even-numbered fingers being the short months? Mnemonics such as these work well for people (I recited that silly song well into adulthood) but are less intuitive for computers. Of course, you can write a program that manually computes the day of the week for any date, using a whole lot of If-Then-Elses, or a few Case statements.

However, I like the technique Kim Larsen presents in this month's "Algorithm Alley" because it approaches the problem from another direction. There isn't really a mathematical formula for computing the day of the week for any given date, but maybe we can cobble one together. The formula works, and you have an easy-to-program mathematical formula for computing the day of the week automatically.

By the way, if you've developed a clever new algorithm, or come up with a new twist on an old idea, I'd love to hear from you. Please contact me at schneier@chinet.com, or just drop a note to me at the DDJ offices.

Have you ever wondered how your computer knows that today is Wednesday? Even if your machine has been down and you specify a new date when starting it up again, it immediately knows which day of the week it is.

When you were a kid, you probably saw tables with entries for the date, the month, and the year. You added up a few numbers and another table gave you the day of that date. Of course, such tables could be hardwired into your computer's operating system. However, there exists a simple formula for computing the correct day of the week. This formula takes up very little space, whereas a collection of tables covering just a few hundred years would take up quite a bit.

If your machine is not capable of computing the day of the week, then you can use this formula in your own programs and applications.

Creating a Formula

The starting point for the formula is a date represented by the variables D, M, and Y. For example, for the date March 1, 1994, D=1, M=3, and Y=1994. Our goal is to compute a number between 0 and 6, where 0 represents Monday, 1 represents Tuesday, 2 represents Wednesday, and so on.

It turns out that March 1, 1994 is a Tuesday, so the formula D mod 7 would actually work for the rest of the month of March. For example, the 18th is a Friday, and 18 mod 7=4, which represents Friday. (Remember that integer division and modulo are closely connected. For example, 26 divided by 7 is 3 with the remainder 5. This means that the integer division of 26 by 7 equals 3, and 26 modulo 7 (abbreviated 26 mod 7) equals 5. This also implies that 19 mod 7=12 mod 7=5 mod 7=5. In fact, this works similarly for negative numbers, so --2 mod 7=5, --9 mod 7=5, and so on.

More formally, it can be shown that for any integers n and k, n can be written as n=qk+r in exactly one way, where q and r are also integers and 0 [image] Table 1 lists the shift information for all months. Note that in order to obtain as much regularity as possible, the short month of February (and, hence, also January) has been moved to the end.

Example 1(a) is a formula that imitates the pattern depicted in the shift column. The division is integer division, so the result is rounded down to the nearest integer. The interesting values for this function are given in Table 2. Intuitively, when M increases with one (going from one month to the next), 2M increases with two, and 3(M+1)/5 increases three out of five times, which is what we need to imitate this repeated pattern 3,2,3,2,3 of shifts (indicated by the curly brackets to the right of the table). Notice that since we are working modulo 7, going from 6 to 2 is an increase of 3 (counting 6,0,1,2).

We have now found a formula to adjust our calculations correctly when we go from one month to the next, and we want to add this formula to our first attempt, namely D mod 7. The only problem is getting it to start out right. Again using March 1, 1994 (that is, M=3), notice that in Example 1(b) 8 mod 7=1, so we must subtract 1 when the whole formula is put together. Working modulo 7, this is the same as adding 6, since -1 mod 7=6 mod 7=6.

Now the formula in Example 1(c) will work for the rest of the year. In fact, since we have placed January and February at the end of Table 1, the formula will also work for these two months in 1995, provided that we refer to these as months 13 and 14. This is because they start a new, though incomplete, 3,2,3,2,3 sequence. To make a nicer formula, we adopt the convention from now on of treating January and February as the months 13 and 14 of the previous year.

Incorporating the Year

In going from one year to the next, we observe that March 1, 1995 is a Wednesday. This means that changing to a new year should have the effect of adding one to our formula. That is easy: We simply add Y to what we already have. Again, we have to make sure things start out right. Since 1994 mod 7=6, we must subtract 6 when we combine Y with the formula we already have. Example 2(a) then becomes a new and better formula.

Our next problem occurs with 1996--a leap year. March 1 is a Friday, not a Thursday, as our formula would currently predict. So we need to add one every time we enter a leap year. The rule is that a year is a leap year if it is divisible by four, except that years divisible by 100 are only leap years if they are also divisible by 400. In effect, we add Y/4--Y/100+Y/400 to what we already have. Again, we must make sure that we start out correctly. Since (1994/4--1994/100+1994/400) mod 7=(498--19+4) mod 7=483 mod 7=0, no adjustments need to be made, and Example 2(b) is our final formula. This formula works indefinitely (unless we change calendar systems). As an example, let us try July 4, 2000: (4+2*7+3(7+1)/5+2000+2000/4--2000/100+2000/400) mod7=(4+14+4+2000+500--20+5) mod 7=2507 mod 7=1, so it is a Tuesday.

This also works backwards in time; however, we switched to the current calendar system on Thursday, September 14, 1752, so it does not work for dates earlier than this. But if we try the standard "where were you when..." date of November 22, 1963, we find: (22+2*11+3(11+1)/5+1963+1963/4--1963/100+1963/400)mod7=(22+22+7+1963+490--19+4)mod 7=2489 mod 7=4, which is a Friday.

These formulas have been implemented in Example 3, a C program for computing the day of the week automatically.

Table 1: Shift information for each month.

Month   # Days  Shift
March   31       --
April   30        3
May     31        2
June    30        3
July    31        2
Aug.    31        3
Sept.   30        3
Oct.    31        2
Nov.    30        3
Dec.    31        2
Jan.    31        3
Feb.    28        3

Table 2 A function for imitating the shift pattern. Example 1 Creating a formula that works for the year 1994. Example 2 Extending the formula to account for different years.

Example 3: The formula expressed as a C program.

/* Computing day of the week from the date. It is assumed that input */
/* represents a correct date.                                      */
#include <stdio.h>
char *name[] = { "Monday",
                 "Tuesday",
                 "Wednesday",
                "Thursday",
                "Friday",
                "Saturday",
                "Sunday"
               };
void main(){
  int D,M,Y,A;
  printf("Day: "); fflush(stdout);
  scanf("%d",&D);
  printf("Month: "); fflush(stdout);
  scanf("%d",&M);
  printf("Year: "); fflush(stdout);
  scanf("%d",&Y);
/* January and February are treated as month 13 and 14, */
/* respectively, from the year before.                  */
  if ((M == 1) || (M == 2)){
    M += 12;
    Y--;
  }
  A = (D + 2*M + 3*(M+1)/5 + Y + Y/4 - Y/100 + Y/400) % 7;
  printf("It's a %s.\n",name[A]);
}

Copyright © 1995, Dr. Dobb's Journal

Back to Table of Contents


DEC95: ALGORITHM ALLEY

ALGORITHM ALLEY

Generating Sequential Keys in an Arbitrary Radix

Gene Callahan

Gene is president of St. George Technologies and the developer of software such as Managing Your Money and the HyperWriter Autolinker. He can be contacted at ecallah@ibm.net.

Recently I had to expand the range of 4-byte keys that could be sent to a client database, as we were on the verge of running out of keys. Our existing internal library functions dealt adequately with things like incrementing a key to generate the next key in sequence. But these functions were hardwired to operate in base-36, using all 10 digits and 26 lowercase letters to represent the key. Obviously, I could have "changed the wiring" to use base-62 keys (by including uppercase letters, which the client system could accept). But I hesitated to recast an algorithm hardcoded to handle a radix of 36 into one hardcoded to handle a radix of 62.

I had seen a similar problem many years before. Working on a dBase III system, I had to generate as many sequential MS-DOS filenames as possible, varying only the last four characters of the name. (The first four indicated the type of file, and all of the extensions would be ".db3".) Certainly a problem like this, which I had seen occur in several contexts, was worthy of a general solution!

I set about looking for one. I consulted my algorithm books and found nothing. I asked half a dozen other programmers if they had ever needed to write code to handle this situation. Five said that they had. When asked what they had done, it turned out that each had written an ad hoc solution suitable only for the radix and character representation used in that specific situation. I wanted one that could handle any reasonable radix, any length key, and any ordering of characters within the base representation.

The solution I present here achieves these goals and also illustrates a couple of important programming principles, to which I was first alerted by Jon Bentley's book, Programming Pearls (Addison-Wesley, 1986):

  • Often, solving a general problem is easier than solving a specific instance of that problem. The algorithm employed here is easier to understand than a hardwired one, and shouldn't take any longer to get right.
  • The key to solving many problems is getting the right data structures--here, given the data structures, the code follows naturally.

The Data

First, I looked at the data I had to deal with. Many languages have built-in routines for processing strings into numbers. C, however, has no routine that, when fed a string in an arbitrary base, will convert it to an internal representation suitable for numeric processing. Clearly, such a facility would make the problem as trivial as it would be if we were using octal, decimal, or hexadecimal numbers in C. In such cases, we simply use standard library routines such as sprintf(), atoi(), and sscanf().

Because the programs in the system I was working on had to perform these conversions thousands of times a day, they had to be fast. Space, however, was not at a premium--our machine had 512 megabytes of RAM. This immediately suggested two table structures, one mapping from characters in a string to their numeric value in a particular base, and the other, back again, by simple array lookup.

In creating the data structure for the conversion from string to number, I took advantage of the happy circumstance that C considers characters valid array indexes. What could be simpler than to use the character itself as the index to find its value in a base? Since the characters are merely signals in an arbitrary code (the key), I didn't have to worry about such niceties as whether some user will want a different, larger alphabet appearing on screen. My strings were for internal consumption only!

To convert from native numeric representation back to a string representation in base-X, a string laying out the ordering of characters in base-X is the natural mapping mechanism. The base-X digits of the native number itself, "peeled off" modulus the powers of the radix X, can serve as indexes into the mapping string. This takes advantage of the relationship between a base and the modulus and division operations--for any base X and number N, repeated modulus and division operations on N by X will produce the value of each base-X digit of N from right to left. The structures I came up with, along with a representative instance, appear in Listing One.

The base representation in the listing has long runs of consecutive values, but this need not be the case. If the application involved something like encryption, the values assigned to different characters could just as well be randomly generated.

The Functions

To map from string to number, BaseXToLong() (see Listing Two) moves through the string representing the number from right to left, indexing by the character at each position into the structure NumValues. That character's value in the current base is stored in NumValues and then multiplied by the power of the radix equivalent to that position in the string. In other words, the numeric value of the last character in the string is multiplied by the 0th power of the radix; the next to last, by the first; the next, by the square of the radix; and so on.

LongToBaseX() maps in the other direction, using the structure CharValues. Again, the function operates from right to left, filling the last character of the string first. To determine the character in the last place, I take the long argument (named "number") modulus the base, and index CharValues with the result. For the next character, I divide number by the radix, essentially dropping off the last digit, and repeat the modulus step; see Listing Three.

Finally, I had to implement the increment function itself. I wrote it as a single statement using the primitives discussed earlier. The code has now gained a level of abstraction. The caller of LongToBaseX() has to worry about details like the length of the return value and where to store it. But at the level of increment abstraction, I knew the answers to these questions--the length is the length of the original key, and it is stored back in the original buffer. I left the low-level routines flexible about the answer to these questions, and built another layer of abstraction to hide these implementation details from the layers above it; see Example 1.

I wrote this function as a single statement because it concisely states my algorithm. The calls to BaseXToLong() and strlen() are made only for their return values, not for any side effect such as I/O or assignment; hence their position inside the call to LongXToBase(). Since I won't otherwise use any intermediate results, storing them would be deceptive. Also, side effects complicate the proof of a function's correctness and impede operations such as shipping the calculation of each of a function's arguments off to separate processors--one function may depend on the side effect of another. By making explicit the precedence of calls to BaseXToLong(), strlen(), and LongXToBase(), this form suggests a macro (or template, in C++) implementation of a family of BaseXIncr functions. These functions add one to a macro argument, rather than the return of BaseXToLong(). Example 2 is a function that allows key comparison, even of keys stored in different bases.

This basic idea can be implemented in several different ways. For instance, the radix could be passed as an argument instead of being in the BaseRep structure; thus, one BaseRep structure could work for many different radices. The base representation could be read in from a file for further flexibility. C++ users should easily be able to turn this into a class, packaging together the BaseRep pointer and the functions operating on it. You might write the class so that the radix could be reset, and perhaps BaseXToLong() and LongToBaseX() would be protected virtual members, so that you could override them in descendant classes.

Conclusion

In working out this problem, I found that once I discovered adequate data structures, I was nearly done. The rest of the implementation consists of only a few lines of code--it is much shorter, in fact, than the version hardcoded to base-36, and it expresses the solution to the problem in a more straightforward way. Changing the radix or the character representation of the base requires only filling in tables, not error-prone recoding. And by isolating general-purpose primitives, I was able to keep all "messy" details out of my application-level functions.

Example 1: Hiding details from the above layers.

char* BaseXIncr(char* number, BaseRep* br)
{
     return LongToBaseX(BaseXToLong
(number, br) + 1,
          number, strlen(number), br);
}
Example 2: Function that allows key comparison, even of keys stored in different bases.
int LessThan(char* num1, BaseRep* br1,
             char* num2, BaseRep* br2)
{
     return BaseXToLong(num1, br1)
          < BaseXToLong(num2, br2);
}

Listing One

/* given an ASCII character, what is its value in the base? */
#define MAX_BASE     256
typedef struct baserep
{
     int  Radix;
     int  NumValues[MAX_BASE];
     char CharValues[MAX_BASE + 1];
} BaseRep;
BaseRep Base36DigitsLower = {
     { 36 },
/* 0 - 47 */
     {-1, -1, -1, /* ...and so on, until 48 negative ones in all*/
/* 48 ('0') - 57 ('9') */
     0, 1, 2, 3, 4, 5, 6, 7, 8, 9,
/* 58 - 96: another run of -1 */
/* 97 ('a') - 122 ('z') */
     10, 11, 12, 13, 14, 15, 16, 17, 18, 19, 20, 21, 22,
23, 24, 25, 26, 27, 28, 29, 30, 31, 32, 33, 34, 35,
/* 123 - 255: whole bunch o' negative ones go here */},
     {"0123456789abcdefghijklmnopqrstuvwxyz"}
};

Listing Two

long BaseXToLong(char* number, BaseRep* br)
{
     long PowerOfRadix;
     long ret = 0;
     int  i;
     int  digits = strlen(number);
     for(i = digits - 1, PowerOfRadix = 1;
          i >= 0;
          i--, PowerOfRadix *= br->Radix)
          ret += (br->NumValues[(int)number[i]] * PowerOfRadix);
     return ret;
}

Listing Three

char* LongToBaseX(long number, char* buf,
                  int buf_len, BaseRep* br)
{
     int i;
     for(i = buf_len - 1; i >= 0; i--)
     {
          buf[i] = br->CharValues[number % br->Radix];
          number /= br->Radix;
     }
     if(number) return 0;  /* null: error return */
     return buf;  /* so function will generate rvalue */
}

Copyright © 1995, Dr. Dobb's Journal

Back to Table of Contents


JAN96: Evaluating Data-Compression Algorithms

Evaluating Data-Compression Algorithms

Finding the right tool for the right job

G. Jason Mathews

Jason is a computer engineer at the National Space Science Data Center Interoperable Systems Office. He can be contacted at mathews@nssdc.gsfc.nasa.gov.

The large amount of data collected by NASA and other scientific organizations makes data compression a necessary part of archival and data-management systems. For instance, data compression is being evaluated for incorporation into the Common Data Format (CDF), a data-management package developed and maintained by the National Space Science Data Center of the NASA/Goddard Space Flight Center (GSFC) for storing, manipulating, and accessing multidimensional data sets. CDF is capable of storing any type of data, including scalar data items, vectors, and multidimensional arrays of data.

Regardless of dimensionality, all data types break down into a sequence of bytes that can be fed into a compression function. The diverse data types and high-performance speed requirements necessitate a general-purpose, fast, simple data-compression algorithm. As a member of the team designing and implementing CDF, I compared several data-compression approaches, searching for a suitable, general-purpose algorithm. In this article, I'll present the results of my examinations. While specifically targeted at CDF, the results are applicable to a variety of platforms.

An ideal data-compression algorithm would compress all data at the highest compression rate in the least amount of time. No such algorithm exists, however, since the measurement criteria are both data and application dependent. For example, one developer might want a high compression ratio regardless of time, while another might need fast compression with some sacrifice of compression ratio. There are trade-offs among the following: time to compress and decompress the data, compression rates, memory requirements (hash tables, frequency counts, temporary buffers, and so on), and algorithmic complexity.

CDF is a programming interface for applications whose goal is to provide fast access to the data; therefore, the compression algorithm should be fast. There isn't much need for compression if most of the data cannot be compressed, so we needed a general-purpose compression algorithm that compresses most data with a good compression ratio. The compression ratio is defined as the percentage reduction (1--compressed data size/uncompressed data size) x 100. One hundred percent compression is not possible unless the compressed size is zero, which occurs only if the uncompressed data size is zero as well. Therefore, the upper limit is 100 percent, and the ratios will approach this limit. If the compressed data size is larger than the uncompressed data size, the data expanded with negative compression. Unless you set a lower limit, data can expand infinitely. If you set the lower limit to zero, files that fail to compress will retain a compression ratio of 0 instead of expanding. A good compression ratio is defined as a significant data reduction for all test data as a whole for a particular compression algorithm. Furthermore, the CDF interface is a library and is linked into all CDF applications, so it's also important that the compression algorithm not dramatically increase the application size.

Finally, portability is important, since both the CDF software and data sets are portable to DEC Alpha, DECstation, HP 9000, PC, RS/6000, Macintosh, NeXT, SGI, Sun, and VAX platforms. The hallmark of the CDF concept is its data-set independence, so a compression implementation with minimum architecture- and machine-level dependencies is preferable:

  • A fast algorithm that minimizes compression/decompression time.
  • A good algorithm with a high compression rate.
  • A simple algorithm that has a small code size.

Data Characteristics

CDF can handle data conceptually organized as scalar and multidimensional with arrays of up to ten dimensions. It supports the data types available with C and Fortran compilers on most systems, which include 1-, 2-, and 4-byte signed and unsigned integers, 4-byte single-precision floating point, 8-byte double-precision floating point, and 1-byte signed and unsigned character types. For example, there may be data defined as a single scalar 1-byte integer and another as a 512x512x16 array of double-precision floating-point numbers. Although it is not very productive to compress a single scalar byte, compressing a large array or stream of bytes has vastly different results. Similarly, any data value with two or three bytes cannot be reduced. However, a 4-byte quantity could possibly be reduced to three bytes, so the minimum file size of four bytes is selected. Since most data stored in CDFs typically have fewer than three dimensions (for example, scalar data types, three-element vectors, and two-dimensional images), the sizes of the sample data files are arbitrarily chosen to range from 4 bytes to 144 KB. Although larger files achieve higher compression ratios, smaller data files better represent the typical size of data stored as components within a CDF.

To evaluate a general-purpose compression algorithm, I needed a representative collection of sample data. I turned to the graphics, text, executables, and sound-data samples presented in Dr. Dobb's Journal; see "DDJ Data Compression Contest" (DDJ, February 1991) and "DDJ Data Compression Contest Results," by Mark Nelson (DDJ, November 1991). While it is unlikely that MS-DOS executable programs will be stored in CDFs, these files have their own structure, and since the goal was to find a general-purpose compressor for all types of data, I included all of the files used by the DDJ tests. In addition, a subset of scientific data from the International Solar-Terrestrial Physics (ISTP) Key Parameter Data CD-ROM was included, since it represents the typical values and types of data that CDF actually stores (data for ISTP spacecraft orbit, attitude, magnetic field and particle measurements, and images).

The compression programs I evaluated included implementations for variations of the LZ77, LZ78, LZW, Huffman, run-length encoding, and arithmetic compression algorithms. Table 1 lists the programs included for the MS-DOS compression tests.

CHURN, the compression utility program that accompanies Mark Nelson's Data Compression Book, Second Edition (M&T Books, 1995), is capable of running compression and decompression programs on all data files in a specified disk volume for MS-DOS. CHURN also measures the elapsed compression and decompression times, and verifies that the contents of the decompressed files match the original data files. CHURN is called with three arguments: a drive letter and pathname to recursively search for files to compress, a compression command, and an output filename. The compression command tells CHURN how to compress the input file to a file called TEST.CMP. CHURN executes the compression command by passing the command line to DOS using the system() function call. The program inserts the filename into the compression command by calling sprintf() with the filename as an argument. This means that if the compression command has a %s anywhere in it, the name of the input file will be substituted. Finally, the third argument on the command line is the command that CHURN needs to decompress TEST.CMP to TEST.OUT; for example, CHURN D:\DATA\ "LZSS-C %%s test.cmp" "LZSS-D test.cmp test.out". The double % symbols defeat variable substitution under some command-line interpreters such as 4DOS. A more complicated example which tests PKZIP might be CHURN D:\DATA\ "PKZIP.BAT %%s" "PKUNZIP TEST.CMP", where PKZIP.BAT has two lines:

COPY %1 TEST.OUT
PKZIP -M TEST.CMP TEST.OUT
CHURN also creates a file called CHURN.LOG containing a summary of compression results. This can be used for further analysis by other programs. This file is reformatted and used to generate the graphic output in Figure 1, where the numbers correspond to those in Table 1. CHURN and the corresponding compression programs were tested on a 486/33 PC running MS-DOS 6.20.

The DOS COPY command (item 14 in Table 1) measures the baseline time to copy a file byte-by-byte with no compression. No compression program should be able to read the entire input file, analyze the data, and compress the results in less time than a straightforward copy, but some compression programs actually approach this time with significant compression. Therefore, the overhead for compressing data won't impact performance if the faster compression algorithms are used.

The DCT program appears to have the highest compression rate, but this is a lossy algorithm included in the test suite only for comparison since lossy algorithms do not meet the requirement for full reconstruction of the data. However, the COMPRESS, GZIP, LZRW1, and LZRW3A programs all achieve high compression rates with fast execution times.

I also conducted tests on the six algorithms listed in Table 2, using the Computerized Reduction Using Selected Heuristics (CRUSH) data-compression program. CRUSH is a portable, multimethod utility that incorporates different algorithms into a common interface for VMS and UNIX systems. CRUSH takes a different approach than CHURN: It links different compression and decompression algorithms into one program, rather than calling external programs. CRUSH is flexible because you can either compress files using a compression algorithm or select automatic mode, which tries all algorithms and selects the best method. Automatic mode ensures that the data have the maximum compression given the available algorithms, but at a high cost of time. The algorithms included in CRUSH were developed by Ian H. Witten, Radford M. Neal, and John G. Cleary (Department of Computer Science, University of Calgary, Canada); David M. Abrahamson (Trinity College, Dublin, Ireland); Ross N. Williams (Renaissance Software, Adelaide, Australia); Robert F. Rice (JPL); and Pen-Shu Yeh and Warner Miller (GSFC). The source code for CRUSH (which is in the public domain) is available via anonymous ftp from dftnic.gsfc.nasa.gov (128.183.115.71) in the software/unix/crushv3 directory.

I ran tests on a Sun 4 machine under SunOS 4.1. This machine is not dedicated, so the elapsed times may have been slightly affected by other activity on the system. To minimize this, the tests were run at off-hours. CRUSH compresses each data file using all six algorithms. For tests of an individual data file, the best compression algorithm varied; the overall results are in Figure 2. LZC has the highest compression ratio and one of the lowest times. The next-best class of algorithms with a high compression/time ratio is LZRW3A and LZRW1. WNC and ADAP both offer comparable compression ratios, and ADAP is even slightly better than LZC, but both take much longer. Overall, the RICE algorithm does not work as well as a general-purpose compressor, but it works best on several individual files.

CRUSH has a single interface, with each compression algorithm having equivalent functionality, so the code complexity can be estimated using the size of the corresponding object-code files. This is a rough approximation, but, along with the other measurements, it gives some idea of how much is involved in the algorithm. Since the compression programs tested on the IBM PC each have different functionality, error handling, and other bells and whistles, the object and executable sizes are not very reliable measures of their complexity. For this reason, all complexity estimates will include only the algorithms available through the CRUSH interface, which has the equivalent interface.

Table 3 ranks compression ratio, speed, and complexity from 1 to 4 (1=worst in its category; 4=most desirable and best in its category). The compression-ratio and speed (total elapsed time) rankings are extracted from the results of running CHURN with the test data and scaled appropriately.

The overall score is a weighted sum (10 x compression rank + 10 x time rank + 5 x complexity rank) with a maximum of 100. The complexity weight is lower than the other two because it is the least reliable measurement. The weights can be adjusted according to the measurements' priority.

Another study that compares 47 compression programs over various large collections of data was conducted by Greg Flint of Purdue University. The raw test results are available as arctst03.zip via anonymous ftp from any SimTel archive site, such as oak.oakland.edu (141.210.10.117) in the SimTel/msdos/info directory. Upon preliminary examination these results were consistent with those presented in this article.

Results

According to my tests, it appears that the best class of programs is based on the Lempel-Ziv adaptive dictionary-based algorithm. Arithmetic compression has a high compression ratio but is the slowest. GZIP and LZC (UNIX Compress) achieve the highest compression ratios and lowest elapsed time. They are the best external, stand-alone compression programs, and they are portable to MS-DOS, UNIX, VMS, and Macintosh platforms. The two algorithms that incorporate small, simple functions into a much larger data-management package are LZRW1 and LZRW3A.

Unfortunately, the LZC implementation, which is based on the LZW algorithm, is covered by Unisys patents (U.S. 4,464,650 and 4,558,302). While Unisys has stated it will license the algorithm to modem manufacturers, the company has not expressly said how the patent applies to other products. This is one reason for the CompuServe GIF/Unisys controversy. The LZRW algorithms are also covered by one or more patents. (The author of these two algorithms believes that the patents are discouraging implementation of the compression techniques.) The GZIP algorithm (a variation of LZ77), however, is free software and is free of patents. GZIP is fast and good but the code is neither simple nor straightforward for incorporating into a large data-management package.

CDF is distributed worldwide as public-domain software and as part of several commercial data-analysis and visualization packages, such as RSI's Interactive Data Language (IDL) and IBM's Visualization Data Explorer; thus, there may be legal complications that are beyond the scope of this article. Aside from these issues, these four LZ-based algorithms (GZIP, LZC, LZRW1, and LZRW3A) are recommended as good, general-purpose data- compression algorithms.

Future Plans

The future plans for CDF are to incorporate the LZC and possibly another data-compression algorithm into a test version of the CDF data-management package, where compression and decompression will be transparent to CDF applications. Various techniques will be tried to determine what to compress to allow efficient access and minimize the impact on performance.

References

Goucher, G.W. and G.J. Mathews. A Comprehensive Look at CDF, NSSDC/WDCA-R&S 94-07. August 1994. (http://nssdc.gsfc.nasa.gov/cdf/cdf_home.html).

Nelson, Mark R. "DDJ Data Compression Contest Results." Dr. Dobb's Journal (November 1991).

------. The Data Compression Book, Second Edition. San Mateo, CA: M&T Books, 1995.

Ross, E., "A Simple Data Compression Technique." C/C++ User's Journal (October 1992).

Teague, M. "ISTP Key Parameter Available on CD-ROM." Solar-Terrestrial Energy Program (STEP) International, Vol. 4, No. 7. July 1994.

Williams, R.N. "An Extremely Fast Ziv-Lempel Data Compression Algorithm." Proceedings Data Compression Conference '91.

Ziv, J. and A. Lempel. "A Universal Algorithm for Sequential Data Compression." IEEE Transactions on Information Theory, Vol. 23, No. 3, May 1977.

Table 1: Data-compression programs tested with CHURN. *Source/executables available on SimTel anonymous ftp sites (for example, oak.oakland.edu). **Source available on disks provided with Mark Nelson's Data Compression Book.

     Item     Program              Description

       1      ahuff**              Adaptive Huffman coding
       2      ar**                 Haruhiki Okumura's archiver
       3      arith**              Arithmetic coding
       4      arith1**             Order-1 arithmetic coding
       5      arj*                 Robert Jung's ARJ, v2.41a
       6      arth1e**             Order-1 arithmetic coding
       7      cmp**                Linear-conversion scheme for sound
       8      compress*            UNIX compress uses LZW variant
       9      cmp40-14             Compress w/COMP40 option + 12 bit
      10      comp40               Compress with COMP40 option
      11      comprs12             Compress 12 bit
      12      comprs14             Compress 14 bit
      13      comp_opt             Modified/optimized compress
      14      copy     			   MS-DOS COPY command
      15      dct**                Discrete cosine transformation
      16      dogzip*              gzip-like LZ77 variant
      17      doz*                 UNIX-like LZW
      18      gzip*                Gnu ZIP uses LZ77 variant
      19      huff*                Huffman coding
      20      lha*                 Harayasu Yoshizahi's LHA v2.13
      21      lzari*               LZSS with adaptive arithmetic coding
      22      lzhuf*               LZSS with adaptive Huffman coding
      23      lzrw1                Ross Williams's LZ variant
      24      lzrw3a               Ross Williams's LZ variant
      25      lzss*                Storer-Szymanski modified LZ77
      26      lzss**               LZ77 with 12-bit sliding window
      27      lzw12**              12-bit LZW compression
      28      lzwcom*              Kent Williams's LZW variant
      29      pkarc*               PKARC v3.6
      30      pkzip (default)*     PKZIP v2.04g LZ77 variant
      31      pkzip -ex*           PKZIP using maximal compression
      32      rdc                  Ross Data Compression uses RLE
      33      snd**                Silence compression coding
      34      zoo*                 Rahul Dhesi's Zoo archiver, v2.1

Table 2: Data-compression algorithms tested with CRUSH.

     Method Name     Algorithm Description

     ADAP            Adaptive dependency WNC method.
     LZC             Lempel/Ziv/Welch UNIX compress.
     LZRW1           Fast Lempel-Ziv method.
     LZRW3A          Fast Lempel-Ziv method.
     RICE            Rice machine.
     WNC             Witten/Neal/Cleary arithmetic code.

Table 3: Ranking of six algorithms tested with CRUSH.

         Compression                 Complexity       Overall Score
Method      Ratio         Speed    (Obj. Code Size)     (Max=100)

LZC           4             3             2                 80
LZRW1         2             4             4                 80
LZRW3A        3             3             3                 75
ADAP          4             1             3                 65
WNC           2             1             2                 40
RICE          1             2             2                 40

Figure 1: CHURN compression results for 30 programs.

Figure 2: CRUSH compression results.

Back to Table of Contents


JAN96: ALGORITHM ALLEY

ALGORITHM ALLEY

Multiple Encryption: Weighing Security and Performance

Burton S. Kaliski, Jr. and M.J.B. Robshaw

Burt and Matt are scientists at RSA Laboratories. They can be contacted at burt@rsa.com and matt@rsa.com, respectively.

As time passes, encryption algorithms age and become vulnerable to new or previously infeasible attacks. In particular, interest is often paid to the size of the key used for encryption, since, in the absence of any other weaknesses, this is one way to compare the strength of ciphers. Key lengths once considered adequate are now vulnerable to adversaries with only modest resources.

While old, well-trusted ciphers approach the end of their useflife, new ciphers are continually being proposed. But new ciphers should be greeted with extreme caution and deep skepticism. They should be subjected to prolonged cryptanalysis (by people other than the designers) before they are considered safe for use by the community at large. Several ciphers are currently earning their credentials as part of this process; IDEA is perhaps the foremost among them (see "The IDEA Encryption Algorithm," by Bruce Schneier, DDJ, December 1993).

To span this period of uncertainty, you should try to use trusted ciphers in more secure ways. As a first step, it is natural to suppose that encrypting twice is better than encrypting once; that the more times you encrypt data which has already been encrypted (perhaps with a different cipher), the safer you must be.

For the most part, this might be true. However, you don't get something for nothing. More encryption means more work, and you need to know exactly how much extra security is gained from the additional work.

In this article, we will examine these issues, paying particular attention to repeated encryption with the same cipher. We will then consider some issues involved in the mixed use of different block ciphers.

Double Encryption

Triple encryption (encrypting three times) is mentioned frequently. But whatever happened to double encryption? Destined to remain the perfect tutorial example, double encryption increases security so little that the extra effort cannot be justified.

With double encryption, the same block cipher is used twice with two different keys. If we denote block-cipher encryption of a message m under a key k1 by E(m,k1), then double encryption might be written as E(E(m,k1),k2). One argument for the effort required to mount an exhaustive search is that if each key is k bits long and k1 and k2 are independent, then the best brute-force search for the keys used in double encryption requires 22k operations.

But what if the opponent is willing to invest in memory? Very often, the time required to mount an attack is reduced by memory; hence the "time-memory" trade-off. In the case of double encryption, a brute-force attack requires 2k table entries, where a table entry consists of a key value and an intermediate data value, and 2k+1 operations, where an operation is encryption under the particular cipher.

To mount this "meet-in-the-middle" attack, you encrypt some known plaintext m under every possible key k1 and store a table with 2k entries, where each resulting intermediate block is indexed by the choice of first key. Then you use every possible key k2 to decrypt the ciphertext; if there is a match in the table, then you have a guess (k1*,k2*) for the pair of keys that were actually used. (Figure 1 shows this attack.) The cryptanalyst might need additional known-plaintext/ciphertext pairs to check that the guess is correct (depending on block and key size, there may well be false alarms).

While such an attack still seems to have large requirements--a table of size 2k, for instance--if the attacker can choose the plaintext, only one table must be compiled for all keys. (If the plaintext is only known rather than chosen, then the table needs to be prepared for that particular plaintext.) Thus, while encryption of each plaintext block requires twice as much work, the cryptanalyst with resources to invest in memory is only faced with twice the computational effort required for single encryption. This is certainly not the 22k operations we might have expected.

Triple Encryption

Unlike double encryption, triple encryption can provide the kind of security we want at a price we are willing to pay, and in several different ways.

In two-key triple encryption we can write the ciphertext equivalent of some plaintext m as E(D(E(m,k1),k2),k1), where D( [image] ,k2) denotes decryption under key k2. When k1=k2, this mode of triple encryption reduces to the single encryption E(m,k1). This useful property is referred to as "backward compatibility." For shorthand, let's call this version of triple encryption with two distinct keys "EDE2"; obvious changes will be made to this notation to accommodate other variants.

Two-key triple encryption is vulnerable to some forms of attack. While the attacks we will describe require large amounts of time and memory, they show that two-key triple encryption does not provide as much security as we might have hoped. Nevertheless, two-key triple encryption is used quite widely in banking; we are not suggesting any practical problem with two-key triple encryption, but there are benefits in using three keys.

In their paper "On the Security of Multiple Encryption" (Communications of the ACM, July 1981), R.C. Merkle and M.E. Hellman showed how two-key triple encryption is vulnerable to a chosen-plaintext attack that requires the encryption of 2k chosen plaintexts, 2k words of memory, and about 2k+1 operations. This is essentially the same work effort required to tackle double encryption, though the plaintext requirements have been vastly increased. Merkle and Hellman were the first to point out the essential impracticality of their own attack, but they referred to the attack as a "certificational weakness" in two-key triple encryption.

This somewhat prophetic view was later upheld. In the paper "A Known-plaintext Attack on Two-key Triple Encryption" (Advances in Cryptology: Eurocrypt '90, Springer-Verlag, 1991), P.C. van Oorschot and M.J. Wiener described a known-plaintext attack that requires 2120/n operations when given n known plaintext blocks, and memory requirements that grow linearly as n increases. While this attack may also be considered impractical, anyone using two-key triple encryption must consider it seriously.

Instead, it is recommended that three independent keys be used. Thus, we can write the ciphertext equivalent to some plaintext encrypted using three-key triple encryption as E(D(E(m,k1),k2),k3), where backward compatibility is provided by putting k3=k2 or k1=k2.

Three-key triple encryption is still vulnerable to a meet-in-the-middle attack requiring 22k words of memory and about 2k+1 operations. However, for any cipher with a reasonably sized key, this brute-force attack will not be a substantial threat.

All these brute-force attacks are applicable, at least to some degree, to any cipher. However, there are some technical issues to resolve in adapting the attack. These attacks were devised with DES in mind, and they make assumptions about key size compared to block size. The sizes can vary considerably between different ciphers.

Finally, one potential property of a block cipher that would jeopardize the security of triple and even higher multiples of encryption is the possibility that the underlying block cipher generates a "group." If so, then for each set of keys k1, k2, and k3, there would be a fourth key k4, such that E(D(E(m,k1),k2),k3)=E(m,k4). Clearly, this would be disastrous. Unfortunately, for most ciphers it's still unknown whether or not the cipher forms a group. DES, however, is not a group, and triple encryption is not vulnerable to attack in this way (see the accompanying text box, "The Data Encryption Standard").

Modes of Triple Encryption

So far, we have only considered the triple-encryption cipher as an "electronic code book," where each plaintext block is encrypted independently of the others. Previously, triple encryption has been used to encrypt valuable information such as other encryption keys (see American National Standard X9.17: Financial Institution Key Management, 1985), and this information can often be accommodated within a single block.

For longer messages, however, it is customary to "chain" the series of encryptions in a mode known as "cipher block chaining" (CBC). With single-encryption CBC, the previous ciphertext is XORed with the plaintext prior to block-cipher encryption.

With triple encryption, we can use one of two CBC modes (see Figure 2):

  • Inner-CBC, where each individual component is used in CBC mode.
  • Outer-CBC, where the three stages of encryption are considered a single unit and the ciphertext is fed back to the plaintext.
Before considering the security issues, let's examine the performance of these CBC modes.

All triple-encryption modes obviously require more resources than single encryption: more hardware or more time. But it would be nice if, given sufficient hardware, the throughput were the same as for single encryption. (For software implementations, the different triple-encryption modes all take roughly the same amount of time.)

Throughput is the main performance distinction between the inner- and outer-CBC modes. In the inner-CBC modes, feedback is around one operation. Three chips, each implementing a single cipher operation, can be kept busy continually, each feeding back to itself, so the throughput is the same as for single encryption. (Latency, the amount of time it takes to process a particular block, is nevertheless three times longer than in single encryption.)

In outer-CBC, feedback is around all three operations. This means that even with three chips, the throughput is only one third that of single encryption; the first chip must wait until the other two are done to move on to the next block. Of course, the chip can process blocks from other messages during its idle time. By interleaving the processing of three messages, the chips can be kept busy all the time. This changes the encryption protocol at a higher level, but may be appropriate for some applications.

With three encryption chips then, inner-CBC is intrinsically more efficient than outer-CBC, unless three messages are interleaved for the latter.

It appears that inner-CBC offers more security than outer-CBC, since the values of the internal feedbacks are never revealed. In an EDE configuration the usual feedback around the middle decryption unit actually becomes a feedforward. This hinders meet-in-the-middle attacks, and inner-CBC appears to offer considerable security against brute-force attacks.

The use of three keys is obviously beneficial, and any additional operational overhead it incurs is insignificant. So, three-key inner-CBC EDE seems to be the mode of choice against brute-force attacks.

But in "Cryptanalysis of Multiple Modes of Operation," Advances in Cryptology: Asiacrypt '94 (Springer-Verlag, 1995), E. Biham showed that inner-CBC modes are vulnerable to a class of sophisticated attacks. The trouble is that the use of feedback around each unit in triple encryption allows controlled changes in the ciphertext to be propagated as changes to the internal and unseen data.

By controlling these changes in the ciphertext, powerful differential and linear cryptanalysis techniques can be used on individual components (see Differential Cryptanalysis of the Data Encryption Standard, by E. Biham and A. Shamir, Springer-Verlag, 1993). Depending on the exact form of triple encryption--two- or three-key, EDE, or EEE--different attacks can be mounted.

In essence, Biham has demonstrated that the use of feedback in the internals of an encipherment procedure is dangerous. It is better to use one substantial encryption transformation and to employ any feedback around its entirety, rather than to consider the encryption as a succession of small, and essentially less secure, transformations, each with its own feedback.

When using triple encryption with chaining, we recommend three-key, outer-CBC triple encryption with an EDE configuration for backward compatibility.

Mixing Ciphers

One frequent suggestion is to mix the use of different ciphers. Perhaps we should encrypt the plaintext with one cipher, the result with a second, incompatible cipher, and so on. This might well be the basis of a strong design, although the difficulty and cost of implementation must be considered, and potentially intricate solutions to backward compatibility must be provided.

Of course, this more-general question could have been the theme of this entire article; after all, multiple encryption with a single cipher is merely one restricted case where all the ciphers are the same. But more-practical issues have influenced our emphasis. Multiple encryption with the same cipher is standardized and widely used, whereas the use of different ciphers remains, at best, merely a proposal. In "Cascade Ciphers: The Importance of Being First," Proceeding of the IEEE Symposium on Information Theory, 1990, U.M. Maurer and J.L. Massey demonstrated that a cascade of ciphers, where the key used at each stage is chosen independently of the others, is at least as strong as its first component. This property is often casually referred to as "the importance of being first." When the chain of ciphers can be equivalently rewritten with any of the ciphers first (technically, the ciphers are said to "commute"), then the cascade is as strong as its strongest component.

Others have addressed related questions, and it seems reasonable to suppose that the successive use of different and incompatible ciphers can only provide additional protection against more-sophisticated methods of cryptanalysis. The basic brute-force and meet-in-the-middle attacks can always be mounted, whatever the cipher, in one way or another. (There are some technical issues to resolve when the key size either differs between the ciphers or becomes larger than the block size.) But there appears to be some reluctance in establishing a precedent; as yet, no one has offered a good choice of ciphers or basic results on the security of a particular mix.

Less-Computation-Intensive Alternatives

As we add encryption operations to the chain of multiple encryptions, we increase the work required to encrypt a block of data by another factor. Is there some less-computation-intensive way to increase the security of our basic block cipher without resorting to multiple encryptions?

With regard to DES, there have been a few proposals that might easily be used on other block ciphers. In one variant, which we call "DES-SES" (see "Foiling an Exhaustive Key-search Attack," by F. Rubin, Cryptologia, April 1987), a secret substitution operation S is used on the plaintext before encryption using single DES. We'll focus on DES-XEX, another variant where an XOR is used both before and after single encryption with DES; hence the -XEX. Thus, the plaintext m would be encrypted as E(m [image] s1, k1) [image] s2, where k1 is the usual secret DES key and s1 and s2 are secret 64-bit quantities which may be the same (DES-XEX2) or different (DES-XEX3).

Against brute-force attacks, both DES-XEX2 and DES-XEX3 offer a marked improvement over single-encryption DES. The effort required to attack DES-XEX2 is 2120/n operations with n known plaintexts; to attack DES-XEX3, 2121 DES operations are required when three plaintext blocks are known.

The additional protection offered against differential and linear cryptanalysis is not as dramatic but still significant; DES-XEX2 or DES-XEX3 offer the same protection as DES with so-called independent subkeys. Such attacks, while impractical (they require more than 261 chosen plaintexts for differential and 260 known plaintexts for linear cryptanalysis; see Biham's "Cryptanalysis of Multiple Modes of Operation"), offer only a small improvement over the requirements for single-DES.

But DES-XEX is cheap--only an additional two XORs are required to encrypt each plaintext block. DES-XEX or similar variants may offer a very realistic balance between increased encryption effort and increased security.

As an exercise, we pose the following question: Is DES-EXE--that is, E(E(m,k1) [image] s1,k2), where k1 and k2 are independent DES keys and s1 is a secret value--stronger or weaker than DES-XEX?

Final Thoughts

While there may be significant advantages in using several types of ciphers, the approach is essentially untried and has significant operational implications. Instead, it is most commonly proposed to use multiple iterations of the same cipher, often DES. In particular, three-key triple-DES in EDE mode is just starting to be widely accepted for the encryption of data streams.

The use of triple encryption raises an interesting question: Surely we need something stronger than triple encryption to transmit the keys used for triple encryption?

When considering triple-DES in a system, perhaps you should examine higher multiples of DES encryption. In quintuple-DES, if a three-key variant is used, then E(D(E(D(E(m,k1),k2),k3),k2),k1) would be among the strongest key sequences against meet-in-the-middle attacks. Backward compatibility with triple-DES is provided by putting k2=k3, and backward compatibility with single-DES is provided by putting k1=k2=k3. Of course, a five-key variant would have obvious additional advantages.

Whatever the future with DES, the issue of multiple encryption will have to be considered for the block cipher of the day. We now have a sufficient understanding of current cryptanalytic techniques to provide a fairly accurate estimate for the additional security provided in return for the increase in work effort and key size.

The Data-Encryption Standard

Designed in the early 1970s, the Data Encryption Standard (DES) is, in the 1990s, still arguably the world's most trusted block cipher, and with good reason. Its resistance to all published cryptanalytic attacks bears continuing testament to the exceptional quality of its design. In the absence of any new major cryptanalytic breakthrough, the declining security offered by DES is essentially a symptom of technological aging--as a basic, primitive building block, it can still be cryptographically useful.

DES is a block cipher that transforms plaintext blocks of 64 bits to 64 bits of ciphertext. The transformation is controlled by a 56-bit key. DES is an iterated cipher consisting of 16 rounds, each using a 48-bit subkey derived from the 56-bit key chosen by the user. Sometimes a variant of DES is described as using independent subkeys. In this case, the subkeys used in each of the 16 rounds are unrelated, rather than being derived from the same user-provided 56-bit quantity.

Apart from a potential halving of the search space gained by using the so-called complementation property (this results from the way the subkey is used in each round), no known shortcuts allow a reduction in the complexity of a brute-force attack. Compared to some more-recent variants, DES is remarkably resilient against sophisticated attacks such as differential and linear cryptanalysis. The best differential attack requires 247 chosen plaintexts, and the best linear cryptanalytic attack requires 243 known plaintexts. In fact, in an experiment lasting 50 days, 12 workstations were used to recover the key used for the DES encryption of 243 known plaintexts (see "The First Experimental Cryptanalysis of the Data Encryption," by M. Matsui, Advances in Cryptology: Crypto '94, Springer-Verlag, 1994). This attack is impractical, however; 40 of the 50 days were spent just encrypting the data.

More significantly, in a Crypto '93 Rump Session entitled "Efficient DES Key Search," M.J. Wiener estimated that for $1 million, a machine can be built which would search exhaustively for the key used in any DES encryption. It is estimated that this machine could find the correct key in an average time of 3.5 hours.

While DES is still practically secure to those without $1 million to invest, it is time to move to stronger methods of encryption, be they new ciphers or multiple iterations of an old favorite.

--B.S.K. & M.J.B.R.

Figure 1: Double encryption using two independent keys and a meet-in-the-middle attack.

Figure 2: Two different ways of implementing the CBC mode of triple encryption.

Back to Table of Contents


FEB96: ALGORITHM ALLEY

ALGORITHM ALLEY

Binary Search

Micha Hofri

Micha, a member of the computer science department at Rice University, is the author of Analysis of Algorithms: Mathematical Methods, Computational Tools (Oxford University Press, 1995). He can be contacted at hofri@cs.rice.edu.

Introduction

by Bruce Schneier

Binary searches are among those algorithmic staples that have uses everywhere. If you want to locate an entry in a sorted array, a binary search is the most efficient way to do so. I can't think of a college course in basic algorithms that doesn't cover binary searches.

The technique isn't perfect, however. Records must all be the same length and must be stored in a static array. If the entries are of variable length or in some kind of dynamic data structure, other less-efficient search techniques--search trees, interpolation searches, hashing, and the like--take over. But since the entry can be a key pointing to a more-complex record, binary-search techniques can be used in many situations.

This month, Micha Hofri sees how efficient he can make a basic binary-search algorithm. Why are we bothering with something so basic? Micha's analysis is interesting not so much in what it reveals about binary search, but in how the algorithm works. Good programmers do this kind of analysis with any algorithm that has significant effects on the performance of the system. (At least, they do when they have the time and budget to program right, not when the deadline is in five hours and management doesn't care what it looks like as long as it works.) Micha's trade-offs--iteration versus recursion, more simple steps versus fewer complex steps--are the type that can be made almost everywhere. Even in a world of ever-increasing processor power and clock speeds, a finely crafted algorithm is still a thing of beauty.

By the way, I am still interested in hearing your column ideas, whether you want to write a column yourself or you'd like to see a certain topic explored. You can contact me at schneier@winternet.com.

Binary search is the method of choice for searching a list for a given key value. In fact, it is the optimal comparison-based search algorithm (hashing methods can do better--with some trade-offs). To use binary search, the lists to be searched must be sorted (we assume in increasing order), of known length, and indexable in an array. This implies that the keys must be all of the same size, and that binary search can not be used to search a linked list.

These conditions mean that the ith smallest key can be directly accessed, as Ai (or A[i], using C-like notation). Binary search can be used to find a given key or to check whether or not the list contains one. The sorting implies, for example, that if the key value 6 (assuming all the keys are integers) is followed by the key value 10, then 7, 8, and 9 are not in the list.

I'll present here a basic form of binary search, which I refer to as "BS"; I'll also discuss "BS1" and "BS2." Much of the following discussion is based on Gilles Brassard and Paul Bratley's book Algorithmics: Theory and Practice (Prentice-Hall, 1988). Since BS is a "divide-and-conquer" algorithm, its form is naturally recursive; see Example 1(a). Note in line 3 the comment [image] (i+j+1)/2 [image]. (The term [image] a [image] is the "floor" of a; it is the largest integer that does not exceed a.) In particular, on entry, when i=0, and j=n-1, k= [image] n/2 [image]. To adapt binary search to check for missing values, you simply change line 2 to Example 1(b).

Analyzing Binary Search

When implementing BS, a few design options are possible. All considerations are based on computing the run time. The purpose here is not just to learn about BS, but to show the power--and limitations--of this type of analysis.

The first question to resolve is the cost unit. Each basic operation in a source language (such as C) is taken as one unit, as is each comparison in BS. However, line 3 in Example 1(a), for instance, needs three operations (although some compilers can do better than that). Q denotes the cost of a function call and return; therefore, if BS terminates in line 2, its cost is Q+1; otherwise it is Q+1+3+1+another call.

To be more precise, suppose A has n entries and x is equally likely to match each of them. If T(n) denotes the total cost of such a call, then T(1)=Q+1 and for ngt;1, T(n)=Q+5+T(r), where r depends on the result of the comparison in line 4 and is either k= [image] n/2 [image] or n-k= [image] n/2 [image] (the "ceiling" of n/2 or the smallest integer gt;/=n/2).

Although T(n) is a random variable, we deal here only with its expected value, t(n). (For a deeper analysis, see my book Analysis of Algorithms: Mathematical Methods, Computational Tools, Oxford University Press, 1995.) Since x is equally likely to be each of the entries, it is with probability of [image] n/2 [image] /2n in the lower part and [image] n/2 [image] /n in the upper part. Figure 1 (a) presents an equation--here, a recurrence--for the mean running time. This kind of equation is typical for divide-and-conquer algorithms. The successful approach to solving this type of equation is:

    1. Simplify.

    2. Look for a sequence of argument values where the equation can be solved by standard analytic means.

    3. Use this as a guide to guess a solution from a table of values produced by the recurrence. Then use substitution to prove that the guess satisfies the equation.

Simplifying means replacing Q+1 by a, Q+5 by b, multiplying by n, and defining u(n)=nt(n). This leaves us with the equation in Figure 1 (b). We then choose values of n for which the troublesome floor and ceiling functions relent: powers of 2, or n=2m. The result is in Figure 1 (c).

The relation n/2=2m-1 suggests a change of notation, so we define v(m)=u(2m) as in Figure 1 (d). This is finally a standard, first-order recurrence, which can be solved by the simple iteration in Figure 1 (e).

So much for these special arguments; writing m=lg n (binary logarithm), we have u(n)=n( [image] lg n+ [image]). Clearly, n( [image] lg n+ [image]) cannot be right for all values of n, since the recurrence generates integer coefficients and lg n is not integral unless n is a power of 2. We must return to the original recurrence for u(n), in Figure 1 (b), and use it--floors, ceilings, and all--to generate several values, compare them with the special solution, and look for a pattern. Here, after computing about ten values, everything falls into place; when n=2m+r, 0 lt;/= rlt;2m, where m= [image] lg n [image], then the solution in Figure 1 (f) fits all the generated values. When r=0, we get back to the special solution. Testing by substitution into Figure 1 (b) is successful, so t(n)=u(n)/n= [image] + [image] ( [image] lg n [image] +2r/n). Since r/n lt; 1, this result supports the standard claim that "BS runs in logarithmic time."

So much for the expected run time. Determining its variability requires a different kind of computation, one that addresses the probabilistic structure directly. I show this in my book; a partial result is that the number of iterations has only two possible values, [image] lg n [image] and [image] lg n [image] (if n=2m, these are both m).

Recursion Elimination

Having computed the expected cost of BS, let's try to reduce it. One obvious source of cost is the recursive call at each level of the procedure. This is "tail recursion," where the last instruction performed at each level is a single recursive call. The call is easy to replace using a single loop; see Example 2. The logic is identical to that of BS, except that a new call is not initiated at each level. Instead, either the left or right end of the interval is adjusted.

The logical structure of such a program parallels that of a recursive program, but the analysis is entirely different. In BS, every instruction was done (at most) once at each level; not so in BS1. That's why the left-most column is added in Example 2: The number of times each line is repeated is given as a function of the searched interval size. The symbol a(n) denotes the number of times that the main instructions (lines 4 and 5) are executed. We define p1 as the probability that the condition in line 5 is satisfied and q1=1-p1. We make the crucial assumption that p1 depends in the same way on the interval size each time line 5 is reached. This is the precise analog of the assumption we made when analyzing BS--and for the same reason. Regardless of which subinterval still needs to be searched, the assumption that x is equally likely to be in any position translates without change: The previous comparisons delimit the subinterval to be searched, but provide no other information on the location of x. Hence p1 always has the same functional form as at the first iteration, [image] n/2 [image] /n, and q1= [image] n/2 [image] /n.

If [image] ' is the cost of the lines performed once--1, 2, 3, and 8--then [image] '=Q+2. ' counts the cost that recurs a(n) times--lines 3, 4, 5, 6, or 7--so [image] '=6. The equation to determine a(n) follows from a description of the search: Starting with an interval of size n, one iteration leaves us with an interval of the size [image] n/2 [image] (in probability [image] n/2 [image] /n) or [image] n/2 [image] (with the complementary probability [image] n/2 [image] /n); hence the equation in Figure 2 (a). Up to a factor of [image], this is the same as Figure 1 (a), so the solution is familiar: n=2 [image] lg n [image] +r yields the result in Figure 2 (b). The amount saved by this modification is t(n)-t1(n), and writing it in full yields the equation in Figure 2 (c).

The last relation allows us to estimate Q (without reading arcane compiled code). Table 1 represents the average time per search (in [image] s) on a standard workstation running under a UNIX-like operating system. Consider for the moment only the left four columns; compute for each n the value [image] lg n [image] +2r/n and perform a linear regression between these values and the column labeled BS-BS1. This yields the intercept -0.248 and slope 0.383. Since the intercept (in admittedly flexible units) is given in Figure 2 (c) as -1, the value of Q in these units is approximately 2.54--about two and a half basic operations.

The cost of recursion is not trivial for a procedure with such a short body. Still, it is hardly onerous. However, this evaluation disregards two important facts:

  • Q depends on the number of variables pushed into (or popped from) the stack; here 3(i,j,x). This is actually two addresses and x. It would differ for calls with other argument vectors.
  • Q's value is materially influenced by details in the implementation of stack operations and cache and system-storage management. On an older, slower minicomputer with essentially the same operating system, Qapproximately equals 1.9. This approach to estimating Q failed on a different UNIX-like system, when BS suffered penalties for recursions of depth beyond 5 that raised the running time by an unexpected factor of 5 to 8! (Yet another experiment suggested that Q without stack-management overhead is somewhat less than 2.) This was the highest price I saw paid for a flexible, dynamic memory-management policy.

Three-Way Comparison

The next modification was prompted by the observation that T(n) is essentially constant, while BS and BS1 sometimes "locate" the desired key early in the search (BS and BS1 are unaware of this, since equality is never tested, only inequality). The value x could be in position [image] n/2 [image] (located at the first stab) but the routines would just go chugging along, only to return there lg n steps later. Example 3 adds just such a test. While BS1 always makes two comparisons per iteration, BS2 needs two in about half of its iterations, and three in the other half--but presumably fewer iterations. Is this a good idea? Analysis should provide the answer: We find in BS2 the same probability p1 used before, Pr(xlt;A[k])= [image] n/2 [image] /n, In addition, we have p2, defined as the probability that x=A[k], given that x is not in the first k positions; hence p2=1/(n-k). The dependence of the exact number of operations on the results of the comparisons is somewhat more complex here. It is evaluated by considering the average number of iterations BS2 will perform, again assuming that x is in the array and is equally likely to be any element. Reading the evolution of s(n) from the code in Example 3 yields the equation in Figure 3 (a). This equation is more complex then those presented so far; for example, two initial values are required to drive the recursion. They too can be deduced by reading BS2: s(1) is obviously 0, and s(2) is always 1. Since k= [image] n/2 [image], then n-k-1= [image] n/2 [image] -1.

Define the function b(n)ns(n), to get the equation in Figure 3 (b), where the initial values are b(1)=0 and b(2)=2. Picking n as a power of 2 does not help as before; instead, we pick n=2m-1, then [image] n/2 [image] -1= [image] n/2 [image] =2m-1-1. We find a first-order recursion, similar to the equation in Figure 1 (c). This is shown in Figure 3 (c). Using the initial value for m=1, which is b(2-1)=1 [image] s(1)=0, a simple unreeling reveals the solution in Figure 3 (d). Trying to guess the general solution is more difficult (I needed more than 30 terms to see the pattern) because the increments change as n reaches 3/2 n* (where n* represents 2 [image] lg n [image] ). Figure 4 (a) shows the guessed solution; [image] (n) is given in Figure 4 (b). Substitution confirms this as a solution to the equation in Figure 3 (b).

Finally we compare the costs t1(n) and t2(n). First, how many iterations, on average, does BS2 save over BS1? The difference is a(n)-s(n), and is given in Figure 5 (a) for the range n* [image] nlt;3/2n*.

As n increases from n* to 3/2n*, this difference varies from 1.5 to 0.66 (approximately). In the rest of the range, the equation in Figure 5 (b) varies correspondingly, approximately from 1.33 to 1.5. The savings are modest and are essentially independent of the size of n, but dependent on the ratio n/n*. Hence the difference between the running times can be roughly set to Figure 5 (c).

Allow the same cost C for any operation (comparison, arithmetic, or assignment) so that the two cost factors in Figure 5 (c) are nearly 6C and C (for two operations, at about half the iterations), respectively. Using only lg n*, the leading term from the solution for s(n), shows that BS1 is better when the equation in Figure 5 (d) applies.

If we take 4/3 as a representative value for d n, we can expect the extra check in BS2 to pay off unless the array is more than about 200 entries long, with a relatively important dependence on the ratio n/n*. Table 1 bears this out. Since the logarithm of n* appears in this relation, the critical length is sensitive to minor variations in the implementation. Higher precision requires consideration of the compiler and the machine code it generates, as well as details of the machine architecture, such as availability of increment/decrement instructions, separate data and instruction caches, and the like.

Note that in some machines, condition codes (with computed branches) allow for a three-way comparison at no added cost. If a compiler is designed to take advantage of this and can use a single comparison for lines 5 and 7 of BS2, then it will outperform BS1 on the average for any value of n. Another possibility is to code it directly in assembly language. Since binary search is used so frequently, this may be a reasonable approach.

Example 1: BS, recursive form of binary search.

(a)
1. BS(A[i..j],x)
2.    if (i==j) return i;                   /* This is it! */
3.    k=(i+j+1)/2;                          /* Computes   (i+j+1)/2   */
4.    if (xlt;A[k])return BS(A[i..k-1],x); /* When x is below, */
5.          else return BS(A[k..j],x);     /* or above the midpoint */

(b)
2. if (i==j) {if (x==A[i])  return i; else return -1;}

Example 2: BS1, first iterative form of BS.

(1)           1. BS1(A[n],x)
(1)           2.     i=0; j=n-1;
(a(n)+1)     3.     while ilt;j
(a(n))       4.     {k=(i+j+1)/2;
(a(n))       5.       if xlt;A[k]
(p1 a(n))   6.         j=k-1;
(q1 a(n))   7.       else i=k;
(1)           8.     }  return i;

Example 3: BS2, second iterative form of BS.

(1)           1. BS2(A[n],x)
(1)           2.     i=0; j=n-1;
(s(n)+1)     3.     while ilt;j
(s(n))       4.     {k=(i+j+1)/2;
(s(n))       5.       if xlt;A[k]
(p1s(n))    6.          j=k-1;
(q1s(n))    7.       else if (xgt;A[k])
(q1q2s(n)) 8.         i=k+1;
(q1p2)      9.       else i=j=k;
(1)          10.     } return i;

Figure 1: Analyzing BS.

Figure 2: Recursion elimination.

Figure 3: Three-way comparison.

Figure 4: Form of the guessed solution.

Figure 5: Determining what BS2 saves over BS1.

Table 1: Average time per search (in ms)

n        BS     BS1  BS-BS1   BS2  BS1-BS2

10      4.22   2.99   1.23   2.76    0.23
100     7.13   4.93   2.20   4.71    0.22
1000   10.18   6.88   3.30   6.99   -0.11
10000  14.96   9.87   5.09  10.45   -4.51

Back to Table of Contents


APR96: ALGORITHM ALLEY

ALGORITHM ALLEY

A Fast Integer Square Root

Peter Heinrich

Peter is a video and computer game programmer who has worked on products for Amiga, PC, Sega, 3DO, and Macintosh. He's currently working for Starwave and can be contacted at peterh@starwave.com.

Complex calculation has always frustrated speed-conscious programmers, since mathematical formulas often form bottlenecks in programs that rely on them. To cope with this problem, three primary tactics have evolved: eliminate, simplify, and be tricky.

Rarely will a programmer eliminate a calculation completely. (If a program operates without it, why was it there in the first place?) Instead, integer or fixed-point may replace expensive floating-point math. At the same time, a simpler version of the formula may be sought--one which is easier to compute but gives roughly the same result.

If this proves difficult (as it often does), a tricky solution may provide the answer. This approach requires almost as much luck as programming skill, and is definitely the most difficult. Then again, the fun is in the challenge.

Trick or Treat

The square-root function certainly qualifies as a complex calculation, as anyone who has actually computed one by hand will readily attest. In general, square roots are avoided in speed-critical code, and rank even higher than division on the list of things to avoid. The technique I present here is an iterative approach to finding [image] [image] N [image], the largest integer less than or equal to the square root of N. Like many tricky solutions, it's also simple, fast, and elegant.

Before attacking the actual algorithm, it might be useful to look briefly at two other iterative methods for computing the square root. Example 1(a) simply applies Newton's Method, a straightforward way to zero in on a value given an initial guess. This method is theoretically fast, having order O(log2N). Unfortunately, it uses a lot of multiplication, which may form a bottleneck in itself.

Example 1(b) uses a different approach, summing terms until they exceed N. The number of terms summed to that point is the square root of N. While this method eliminates the multiplication, it has a higher order of O( [image] N).

It would be nice to find a practical algorithm that also is efficient, that is, one which requires only elementary operations but also is of low order. The Binomial Theorem suggests a possible approach. Assume [image] N is the sum of two numbers, u and v. Then N=(u+v)2=u2+2uv+v2. Choosing u and v carefully may simplify calculation of the quadratic expansion. But what constitutes a good choice?

Finding Your Roots

For any number N, it's easy to determine [image] log2N [image] --simply find the position of the highest set bit. Similarly, [image] log2 [image] N [image] = [image] log2 N1/2 [image] = [image] 1/2 log2N [image] indicates the position of highest bit set in result, [image] [image] N [image]. Now the problem just entails finding which of the remaining (less significant) bits, if any, also are set in [image] [image] N [image].

Let u=2 [image] 1/2 log2 N [image]; that is, let u take the value of the highest bit set in the result, [image] [image] N [image] . It isn't known if the next-lower bit is also set in the result, so let v take its value, then solve u2+2uv+v2. This calculation is easy because each term is a simple shift. Since v is known to be a power of two, even the middle term, 2uv, reduces to a shift operation.

If the sum of all three terms is less than or equal to N, the next-lower bit must be set. In that case, the result just computed will be used for u2 and u=u+v for the next iteration. If the sum is greater than N, the next lower bit isn't set, so u remains unchanged. In either case, move on to the next-lower bit and repeat the process until there are no more bits to test.

Example 2(a) implements (in C) an algorithm that appears to satisfy both design goals. It uses only elementary operations (addition and shift) and is extremely efficient, weighing in at O(log2 [image] N). However, a few minor optimizations still can be performed: determining [image] 1/2 log2 N [image] can be improved; v doesn't have to be recomputed from scratch every iteration; and noticing that 2uv+v2=v(2u+v) simplifies some computation inside the loop. Example 2(b) is the final result.

Actually, many assembly languages make the first optimization moot. In fact, two of the three assembler listings presented here use a shortcut. Only the ARM processor lacks a specialized instruction to find the highest set bit in a number (but it's a RISC chip, after all). Listings One through Three present implementations of the optimized algorithm for the Motorola 68020, Intel 80386, and ARM family of processors, respectively.

Conclusion

For programmers developing high-performance code, complex mathematical calculation is not always practical. Some may spurn floating-point math altogether, especially if a math coprocessor isn't guaranteed to be present on the target platform. The algorithm I present here computes an integer square root suitable for just such situations. Even as hardware speeds increase, programs demand more and more. Fast and elegant little tricks like this one can still be useful.

Example 1: (a) Newton's Method; (b) summing terms.

(a)
//  Newton's Method -- O( log2 N )
unsigned long sqroot( unsigned long N )
{
    unsigned long n, p, low, high;
    if( 2 > N )
        return( N );
    low  = 0;
    high = N;
    while( high > low + 1 )
    {
        n = (high + low) / 2;
        p = n * n;
        if( N < p )
            high = n;
        else if( N > p )
            low = n;
        else
            break;
    }
    return( N == p ? n : low );
}

(b)
//  Summing terms -- O( sqrt N )
unsigned long sqroot( unsigned long N )
{
    unsigned long n, u, v;
    if( 2 > N )
        return( N );
    u = 4;
    v = 5;
    for( n = 1; u <= N; n++ )
    {
        u += v;
        v += 2;
    }
    return( n );
}

Example 2: (a) Binomial theorem; (b) optimized binomial theorem.

(a)
//  Binomial Theorem -- O( 1/2 log2 N )
unsigned long sqroot( unsigned long N )
{
    unsigned long l2, u, v, u2, v2, uv2, n;
    if( 2 > N )
        return( N );
    u  = N;
    l2 = 0;
    while( u >>= 1 )
        l2++;
    l2 >>= 1;
    u  = 1L << l2;
    u2 = u << l2;
    while( l2-- )
    {
        v   = 1L << l2;
        v2  = v << l2;
        uv2 = u << (l2 + 1);
        n   = u2 + uv2 + v2;
        if( n <= N )
        {
            u  += v;
            u2  = n;
        }
    }
    return( u );
}

(b)
//  Optimized Binomial Theorem
unsigned long sqroot( unsigned long N )
{
    unsigned long l2, u, v, u2, n;
    if( 2 > N )
        return( N );
    u  = N;
    l2 = 0;
    while( u >>= 2 )
        l2++;
    u  = 1L << l2;
    v  = u;
    u2 = u << l2;
    while( l2-- )
    {
        v >>= 1;
        n   = (u + u + v) << l2;
        n  += u2;
        if( n <= N )
        {
            u += v;
            u2 = n;
        }
    }
    return( u );
}

Listing One

           MACHINE MC68020
           EXPORT  sqroot
;;  unsigned long sqroot( unsigned long N ).
;;  This routine assumes standard standard Macintosh C calling conventions,
;;  so it expects argument N to be passed on the stack. Macintosh C register
;;  conventions specify that d0-d1/a0-a1 are scratch.
sqroot      PROC
           ; If N < 2, return N; otherwise, save non-scratch registers.
           move.l      4(sp),d0                ; just past the return address
           cmpi.l      #2,d0
           bcs.b       done
           movem.l d2-d3,-(sp)
           ; Compute the position of the highest bit set in the root.
           ; Using a loop instead of BFFFO will make this code run
           ; on any 680x0 processor.
           movea.l     d0,a0                    ; preserve N for later
           bfffo       d0{0:0},d3
           neg.l       d3
           addi.l      #31,d3
           lsr.l       #1,d3
           ; Determine the initial values of u, u^2, and v.
           moveq.l     #1,d0
           lsl.l       d3,d0                   ; u
           move.l      d0,d1                   ; v starts equal to u
           movea.l     d0,a1
           lsl.l       d3,d1                   ; u^2
           exg.l       d1,a1
           ; Process bits until there are no more.
checkBit   dbf.w       d3,nextBit
           movem.l     (sp)+,d2-d3
done       rts
           ; Solve the equation u^2 + 2uv + v^2.
nextBit    lsr.l       #1,d1                   ; v = next lower bit
           move.l      d1,d2
           add.l       d0,d2
           add.l       d0,d2                   ; n = 2u + v
           lsl.l       d3,d2
           add.l       a1,d2                   ; n = u^2 + v(2u + v)
                                               ;   = u^2 + 2uv + v^2
           ; If n <= N, the bit v is set.
           cmpa.l      d2,a0
           bcs.b       checkBit
           add.l       d1,d0                   ; u += v
           movea.l     d2,a1                   ; u^2 = n
           bra.b       checkBit
           END

Listing Two

           NAME        sqroot
           PUBLIC      _sqroot
;;  unsigned long sqroot( unsigned long N ).
;;  This routine assumes the argument N is passed on the stack, and eax-edx
;;  are scratch registers.
TEXT       SEGMENT     PUBLIC 'CODE'
           ASSUME      CS:TEXT
           P386
_sqroot    PROC        FAR
           ; If 2 > N, return N; otherwise, save the non-scratch registers.
           mov         eax,[esp+4]             ; just past the return address
           cmp         eax,2
           jb          short done
           push        edi
           push        esi
           ; Compute position of the highest set bit in the root. It's just
           ; half the position of the highest bit set in N.
           mov         esi,eax                 ; preserve N for later
           bsr         ecx,eax
           shr         ecx,1
           ; Determine the initial values of u, u^2, and v.
           mov         eax,1
           shl         eax,cl                  ; u
           mov         ebx,eax                 ; v starts equal to u
           mov         edx,eax
           shl         edx,cl                  ; u^2
           ; Process bits until there are no more.
checkBit   dec         ecx
           js          short restore
           ; Solve the equation u^2 + 2uv + v^2.
           shr         ebx,1                   ; v = next lower bit
           mov         edi,eax
           add         edi,eax
           add         edi,ebx                 ; n = 2u + v
           shl         edi,cl
           add         edi,edx                 ; n = u^2 + v(2u + v)
                                               ;   = u^2 + 2uv + v^2
           ; If n <= N, the bit v is set.
            cmp        edi,esi
            ja          short checkBit
            add         eax,ebx                 ; u += v
            mov         edx,edi                 ; u^2 = n
            jmp         short checkBit
restore     pop         esi
            pop         edi
done        ; Return to caller.
            mov         edx,eax
            shr         edx,16                  ; necessary, but seems silly...
            retf
_sqroot     ENDP
TEXT        ENDS
            END

Listing Three

            AREA        object,CODE
            EXPORT      sqroot
;;  unsigned long sqroot( unsigned long N ).
;;  This routine observes the ARM Procedure Call Standard (APCS), so it expects
;;  the argument N to appear in r0 (referred to as a1 by the APCS). Likewise,
;;  the first four registers, r0-r3 (a1-a4 in the APCS), are treated as scratch.
sqroot      ROUT
            ; If N < 2, return N; otherwise, save non-scratch registers.
            cmp         a1,#2
            movcc       pc,lr
            stmfd       sp!,{v1,v2,lr}
            ; Compute position of the highest bit set in root. It's just
            ; half the position of the highest bit set in N.
            mov         a2,a1                   ; preserve N for later
            mov         a3,a1
            mov         v1,#0
findlog2    movs        a3,a3,LSR #2
            addne       v1,v1,#1
            bne         findlog2
            ; Determine the initial values of u, u^2, and v.
            mov         a1,#1
            mov         a1,a1,LSL v1            ; u
            mov         a3,a1                   ; v starts equal to u
            mov         a4,a1,LSL v1            ; u^2
            ; Process bits until there are no more.
checkBit    cmp         v1,#0
            ldmeqfd     sp!,{v1,v2,pc}
            sub         v1,v1,#1
            ; Solve the equation u^2 + 2uv + v^2.
            mov         a3,a3,LSR #1            ; v = next lower bit
            add         v2,a3,a1,LSL #1         ; n = 2u + v
            add         v2,a4,v2,LSL v1         ; n = u^2 + v(2u + v)
                                                ;   = u^2 + 2uv + v^2
            ; If n <= N, the bit v is set.
            cmp         v2,a2
            addls       a1,a1,a3                ; u += v
            ldmeqfd sp!,{v1,v2,pc}              ; exit early if n == N
            movls       a4,v2                   ; u^2 = n
            b          checkBit

            END

Back to Table of Contents


JUN96: ALGORITHM ALLEY

ALGORITHM ALLEY

Building Decision Trees with the ID3 Algorithm

Andrew Colin

Andrew is a mathematician and proprietary trader in the Treasury, Commonwealth Bank of Australia, Sydney. He can be reached at 100036.1647@compuserve.com.

From network routers to games, decision making is a crucial part of many programs. In this column, Andrew deals with decision making at two levels. Most obviously, he is interested in constructing binary decision trees. Perhaps more interestingly, the algorithm Andrew discusses uses a metric to decide how to build the tree. This particular metric is grounded in the mathematics of information theory, but other decision algorithms use more ad hoc measurements, such as those used to evaluate moves in game-playing programs. If you'd like to write an article about an interesting metric or other decision method, contact me at Dr. Dobb's Journal.

--Tim Kientzle

A decision tree is a series of nested If/Then rules that users apply to data to elicit a result. For instance, a decision tree to assist in the diagnosis of automobile engine trouble might start out with general questions ("Does the engine start?") and use the outcome to determine which subsequent questions are asked:

    1. Did the engine turn over? No.

    2. Is the battery charged? Yes.

    3. Is there a loose battery connection? Yes.

A decision tree is usually a straightforward tool to use. However, inducing rules from sets of past decisions to build such decision trees is a difficult problem, especially when we want to reach a useful answer in as few questions as possible.

In this article, I'll present one way of building such decision trees using the ID3 algorithm, originally developed by J. Ross Quinlan of the University of Sydney. This algorithm was first presented in 1975 (J.R. Quinlan, Machine Learning, vol 1. no. 1) and has proven to be so powerful that it has found its way into a number of commercial rule-induction packages. However, the underlying concepts are simple enough for ID3 to be implemented in only a few pages of code.

How does ID3 work? Suppose you have the animals ostrich, raven, albatross, and koala. For each one, you know the following attribute values: warm blooded, has feathers, has fur, can swim underwater, lays eggs. Based on these attributes, you are asked to find a set of rules to determine whether the animal lays eggs. In this case, a classifying attribute is readily apparent: If the animal has feathers, then it lays eggs. Consequently, the decision tree has just one question: "Does the animal have feathers?" If you add dolphin and crocodile to the list, this rule is incorrect, since the crocodile does not have feathers, but does lay eggs. By inspecting Table 1, you can amend this to the rules in Figure 1(a).

However, there are a number of ways to elicit the correct answer from a dataset. In Figure 1(b), the first question in the tree ("Does the animal have fur?") is poorly chosen, since it gives you little information about the answer. Clearly, some questions are more useful in this context than others. For more realistic datasets--where there are tens of attributes and thousands of samples--a well-chosen set of questions is of critical importance to the effectiveness of the tree.

The ID3 algorithm searches through the attributes of a dataset for the one that conveys the most information about the desired target. If the attribute classifies the target perfectly, then you stop. Otherwise, the process is repeated recursively on the two subsets generated by setting this "best" attribute to their on/off states.

Information theory provides one way to measure the amount of information conveyed by a particular attribute. The best attribute is the one with the lowest "negentropy," as defined in Figure 2. The ID3 algorithm picks the attribute with the lowest total negentropy.

To illustrate, I'll calculate the negentropies generated by Table 1. For example, there are five animals in the sample that are warm blooded, of which three lay eggs. The negentropy of the "on" (true) attribute is NE_on = -(3/5) log2 (3/5) - (2/5) log2 (2/5) = 0.970951. There is one animal in the sample that is not warm blooded but does lay eggs. The negentropy of the "off" (false) attribute is NE_off = -(1/1) log2(1) - 0 = 0. The combined negentropy is the weighted sum (5 * NE_on + 1 * NE_off) / (5 + 1) = 0.809125. Table 2 shows the negentropies for all of the attributes.

None of the attributes has a negentropy of zero, so none classifies all cases correctly. However, the "feathers" attribute has the lowest negentropy and hence conveys the most information. This is what we use as the root attribute for our decision tree.

The process is now repeated recursively. All animals that have feathers lay eggs, so the negentropy for this subset is zero and there are no further questions to ask. When the animal does not have feathers, you compute the negentropies using only those animals to produce Table 3. The attribute to use for the subtree is therefore "warm blooded," which has the lowest negentropy in this sample. The zero value indicates that all samples are now classified correctly.

The algorithm can also be used for real-valued, rather than binary-valued, attributes. At the outset of processing, you scan all of the variables to select an attribute i and a value r so that the condition Attribute(i) >= r yields the lowest negentropy possible. The "yes/no" question is now of the form "Is attribute greater than or equal to r?" If the negentropy arising from this partition is zero, then the data was classified perfectly, and the calculation ends. Otherwise, the dataset is partitioned into two sets--one for which the first condition is true, and the other for false.

The process is repeated recursively over the two subsets until all negentropies are zero (perfect classification) or until no more splitting is possible and the negentropy is nonzero. In this case, classification is not possible and you must solicit more data from the user. Such a situation could occur when all attributes in two records have the same values, but the outcomes differ. More data must then be supplied to the program to allow it to discriminate between the two cases.

Tree Pruning

Most real-world datasets do not have the convenient properties shown here. Instead, noisy data with measurement errors or incorrect classification for some examples can lead to very bushy trees in which the rule tree has many special cases to classify small numbers of uninteresting samples.

One way to address this problem is to use "rule-tree pruning." Instead of stopping when the negentropy reaches zero, you stop when it reaches some sufficiently small value, indicating that you are near the end of a branch. This pruning leaves a small number of examples incorrectly classified, but the overall structure of the decision tree will be preserved. Finding the exact, nonzero cutoff value will be a matter of experimentation.

Implementing Decision Trees

In implementing the decision tree described here, I've represented it within the program (see Listing One, as a binary tree, constructed of NODE structures pointing to other NODEs or to NULLs for terminal nodes.

Rather than copying the data table for each partition, I pass the partially formed data tree to the routine that calculates negentropy, allowing the program to exclude records that are not relevant for that part of the tree. Negentropy of a partition is calculated in routine negentropy (see Listing Two), which is called for all attribute/threshold combinations by routine ID3 (Listing Three).

The ability to use real-valued as well as binary-valued attributes comes at a price. To ensure the correct value of r, we scan through all attribute values in the dataset--a process that can be quite computationally intensive for large datasets.

No claims are made for the efficiency of this implementation. For cases where many sample attribute values are the same, or where a mixture of real-valued and binary-valued attributes is to be considered, the user is probably better advised to sort the attributes into a list and to eliminate repeated values. I've also not considered the case where a question can have more than two outcomes.

Two illustrative datasets are available electronically; see "Availability," page 3. The first is a set of sample data from a botanical classification problem, in which a type of flower, an iris, is to be classified into one of two subgenera (Virgin, Setosa) according to the dimensions of sample pistils and stamens. The data is taken from M. James' book Classification Algorithms (Collins, 1985). Figure 3 shows the resulting decision tree.

The second dataset is a torture test for the algorithm. Given the attributes {John, Paul, George, Ringo}, which are random numbers between 0 and 1, and a target attribute that takes random values from {0, 1}, the program returns a large and complex tree that classifies 100 examples. On a 486/75, the calculation took about 220 seconds to run to completion. Most real-world datasets will produce much simpler decision trees than Listing Four.

The code (available electronically) includes the files ID3.C, ID3.H, PROTO.H. To build this program with Borland's Turbo C compiler, enter tcc id3.c at the DOS prompt. The code has also been compiled under Watcom C and Microsoft Visual C, so it should be fairly portable. To run the program, two datasets are required. The first contains the sample data in ASCII form, with values for the target attribute in the last column (0 or 1). The second dataset contains the names of the attributes, again in ASCII. For the iris dataset, these files have the names IRIS.DAT and IRIS.TAG. Enter id3 iris at the command prompt to run the program.

Conclusion

ID3 is a conceptually simple but powerful classification algorithm. Used in conjunction with other statistical and machine-learning techniques, this algorithm will form a valuable addition to your armory of data exploration tools.

Table 1: Various animal attributes.

Animal   Warm blooded  Feathers  Fur  Swims  Lays eggs

Ostrich       Yes         Yes    No    No       Yes

Crocodile     No          No     No    Yes      Yes

Raven         Yes         Yes    No    No       Yes

Albatross     Yes         Yes    No    No       Yes

Dolphin       Yes         No     No    Yes      No

Koala         Yes         No     Yes   No       No

Table 2: Negentropies for each attribute from Table 1.

Attribute    on_ctr  Hits  Off_ctr  Hits  Negentropy

Warmblooded    5       3      1       1    0.809125

Feathers       3       3      3       2    0.459148

Fur            1       0      5       4    0.601607

Swims          2       1      4       3    0.874185

Table 3: Negentropies of each attribute for animals that do not have feathers.

Attribute    on_ctr  Hits  Off_ctr  Hits  Negentropy

Warmblooded    2       0      1       0    0.0

Feathers       0       0      3       1    0.918296

Fur            1       0      2       1    0.666666

Swims          2       1      1       0    0.666666

Figure 1: Two different decision trees to answer the question: Does the animal lay eggs?

(a)  Does animal have feathers?
        Yes: Lays eggs (raven, albatross, ostrich)
        No : Is animal warmblooded?
               Yes: Does not lay eggs (koala, dolphin)
               No: Lays eggs (crocodile)
(b)  Does animal have fur?
        Yes: Does not lay eggs (koala)
        No:  Does animal have feathers?
                Yes: Lays eggs (raven, albatross, ostrich)
                No: Is animal warmblooded?
                        Yes: Does not lay eggs (dolphin)
                        No: Lays eggs (crocodile)

Figure 2: Definition of negentropy. p(ON) and p(OFF) are the measured probabilities of an answer being true or false.

If p(ON) and p(OFF) are both nonzero:
      -p(ON) log2 p(ON) - p(OFF)
                          log2 p(OFF)
Otherwise:
      0

Figure 3: Decision tree for irises.

Is the pistil width >= 1.80?
       Yes: class Setosa
      No : Is the pistil length >= 5.60?
               Yes: class Setosa
            No : class Virgin

Listing One

typedef struct node {
   UINT idx;            /* ID code for attribute */
   REAL threshold;      /* Numerical threshold for attribute test */
   struct node *on;     /* Address of 'on' node */
   struct node *off;        /* Address of 'off' node */
   struct node *parent;     /* Addess of parent node */
} NODE;

Listing Two

NEGENTROPY negentropy ( REAL **data, UINT n_samples, NODE *local, UINT target)
{
   /* Calculates the entropy of classification of an attribute, given a table
    * of attributes already used, the attribute on which splitting is to be
    * taken, and the target attribute. Entropy is calculated in bits, so logs
    * are taken to base 2 by dividing by LN_2.
    * The returned value always lies in the (closed) range [0, 1].
    */
   NEGENTROPY ret_val;
   NODE *_node, *_parent;
   UINT on_ctr, off_ctr, p1, p2, i, _match;
   REAL p_on, p_off, negentropy_on, negentropy_off;
   on_ctr = off_ctr = p1 = p2 = 0;
   /* Scan through all supplied data samples */
   for (i=0; i<n_samples; i++) {
      /* If pattern matches the current position in the decision tree, then use
       * this vector. The match is determined by passing up the decision tree
       * and checking whether 'data[idx] >= threshold' matches at each step,
       * where idx and threshold are taken from each node in turn.
       */
      _match = 1;
      _node = local;
      while (_node->parent != NULL) { /* If at root node, all entries match */
         _parent = _node->parent;
         if (_node == _parent->on) { /* if parent node is ON */
            if (data[i][_parent->idx] < _parent->threshold)
               _match = 0;
     }
         else
         if (_node == _parent->off) { /* if parent node is OFF */
            if (data[i][_parent->idx] >= _parent->threshold)
              _match = 0;
     }
      _node = _parent;
      }
      if (_match) {
         if (data[i][local->idx] >= local->threshold) {
            on_ctr++;
              if (data[i][target] >= 0.5)
               p1++;
         }
         else {
            off_ctr++;
            if (data[i][target] >= 0.5)
               p2++;
         }
      }
   }   /* for (i=0; i<n_samples; i++) */
   /* 1: Entropy of subtable with activation ON */
   /* We now have the numbers of samples that match this part of the decision
    * tree, & the number of samples for which the supplied condition are true.
    * From these quantities we can find the negentropy of this partition.
    */
   if (on_ctr > 0)
   {
      p_on  = (REAL) p1 / (REAL) on_ctr;
      p_off = 1 - p_on;
      negentropy_on = -entropy (p_on) - entropy (p_off);
   }
   else
      negentropy_on = 0.0;
   /* 2: Entropy of subtable with activation OFF */
   if (off_ctr > 0)
   {
      p_on  = (REAL) p2 / (REAL) off_ctr;
      p_off = 1 - p_on;
      negentropy_off = -entropy (p_on) - entropy (p_off);
   }
   else
      negentropy_off = 0.0;
   ret_val.ne = (negentropy_on * on_ctr + negentropy_off * off_ctr);
   ret_val.ne /= (on_ctr + off_ctr);
   /* If all values examined were the same, set 'ret_val.status' to
    * the target value since this will be an end-of-branch node
    */
   if ((p1 == on_ctr) && (p2 == off_ctr))
      ret_val.status = ON;
   else if ((p1 == 0) && (p2 == 0))
      ret_val.status = OFF;
   else
      ret_val.status = INACTIVE;
   return ret_val;
}

Listing Three

NODE* ID3 ( MATRIX *matrix, NODE* parent, UINT target, UINT state)
/* Routine to build a decision tree, based on Quinlan's ID3 algorithm. */
{
   NEGENTROPY negentropy_struct;
   NODE *node;
   UINT n_vars = matrix->width, n_samples = matrix->height, i, j, split;
   REAL **data = matrix->data;
   REAL best_threshold, min_negentropy, _negentropy;
   /* Allocate memory for this node */
   node = (NODE*) malloc (sizeof(NODE));
   if (!node)
      err_exit (__FILE__, __LINE__);
   /* Set up links in decision tree */
   node->parent = parent;  /* Set address of parent node */
   if (parent != NULL) /* parent to child; not relevant for root node */
   {
      /* Pass address of this node to the parent node */
      if (state == ON)
     parent->on = node;
      else
      if (state == OFF)
     parent->off = node;
   }
   /* Select attribute with lowest negentropy for splitting. Scan through
    * ALL attributes (except target) and ALL data samples. This is inefficient
    * for data sets with repeated values, but will do for illustrative purposes
    */
   min_negentropy = 1.0;
   for (i=0; i<n_vars; i++) {
      for (j=0; j<n_samples; j++) {
         if (i != target) {
            /* Set trial values for this node... */
            node->idx = i;
            node->threshold = data[j][i];
            /* ...and calculate the negentropy of this partition */
            negentropy_struct = negentropy (data, n_samples, node, target);
            _negentropy = negentropy_struct.ne;
        /* If this negentropy is lower than any other, retain the
               index and threshold for future use */
            if (_negentropy < min_negentropy) {
           min_negentropy = _negentropy;
               split = i;
               best_threshold = data[j][i];
            }
         } /*if (i != target)*/
      } /*for (j=0; j<n_samples; j++)*/
   } /*for (i=0; i<n_vars; i++)*/
   /* Save the combination of best attribute and threshold value */
   node->idx = split;
   node->threshold = best_threshold;
   /* If the negentropy routine found itself at an end-of-branch
    * for the decision tree, the 'status' flag in 'negentropy_struct'
    * is set to ON or OFF and the node labelled accordingly. Otherwise,
    * ID3 continues to call itself until all end-of-branch nodes are found.
    */
   if  (negentropy_struct.status != INACTIVE) {
      node->on = node->off = NULL;
      node->idx = negentropy_struct.status;
   }
   else
   {
      node->on  = ID3 (matrix, node, target, ON);
      node->off = ID3 (matrix, node, target, OFF);
   }
   return node;
}

Listing Four

Welcome to ID3
Last compiled on Dec 28 1995, 09:15:09
if { Ringo >= 0.33 then if { George >= 0.63 then if { Paul >= 0.58 then
if { George >= 0.99 then OFF else ON }  else if { Ringo >= 0.77 then
if { Paul >= 0.13 then ON else OFF }  else if { John >= 0.52 then
if { Ringo >= 0.57 then ON else if { John >= 0.79 then ON
else OFF } }  else OFF }  }  }  else ON }  else if { George >= 0.34
then if { John >= 0.52 then if { Paul >= 0.43 then if { George >= 0.76
then OFF else if { John >= 0.65 then ON else if { John >= 0.52
then if { Paul >= 0.79 then ON else OFF }  else ON }  }  }  else OFF }
else OFF }  else if { John >= 0.49
then ON else if { Ringo >= 0.08 then if { John >= 0.01
then ON else OFF }  else OFF }  }  }  }

Back to Table of Contents


July 1996: Algorithm Alley

Minimal Perfect Hashing

Carlo Pescio

Dr. Carlo Pescio is a freelance consultant in Savona, Italy. He specializes in object technologies and is a member of the IEEE Computer Society and the ACM. You can contact him at pescio@ mbox.vol.it.

Hashing is one of the oldest, simplest, and most powerful of all computer-science techniques. That doesn't keep it from being subtle, though. The choice of a particular hash function, or the key that you use for a hash, can have a phenomenal impact on performance. In his article "Hashing Rehashed," Andrew Binstock (DDJ, April 1996) pointed out the impacts of different kinds of collision resolution. This month, we'll see that you can sometimes completely eliminate collisions.

-Tim Kientzle

Hashing is a powerful and simple technique for accessing information: Each key is mathematically converted into a number, which is then used as an index into a table. In typical applications, it's common for two different keys to map to the same index, requiring some strategy for collision resolution. However, many applications use a static set of records; for instance, the table of keywords for a compiler, the contents of a dictionary, or the topics in a help file. In these cases, it's nice to have a perfect hashing function, one that maps different keys into different numbers. With a perfect function, you could retrieve the record (or detect that no record exists) in exactly one probe, without worrying about collisions. A special case of perfect hashing is minimal perfect hashing, where no empty locations in the table exist. Minimal perfect hashing satisfies the ideal of fast performance and low memory usage. However, finding perfect hashing functions is difficult. In this article, I'll present a simple algorithm to find perfect (and optionally minimal) hashing functions for a static set of data.

Chichelli's Functions

A family of perfect hashing functions was introduced by R. Chichelli in "Minimal Perfect Hashing Made Simple" (Communications of the ACM, January 1980). Chichelli suggests using an exhaustive search to find such functions and discusses some heuristics to speed the search. I have extended the heuristics to obtain a general implementation and speed up the original algorithm by more than an order of magnitude.

Chichelli's hashing functions are very simple. Given word w, the hash value is Value(First_Letter(w)) + Value(Last_Letter(w)) + Length(w). The mapping function Value() is normally implemented as a table that maps characters into numbers. Figure 1 is an example of table and hash values of a minimal perfect-hashing function for the days of the week. Note that you only need to keep a small table for Value(), since only first and last letters need to be mapped. Calculating the hash value requires only two table accesses and two additions.

Chichelli's Algorithm

The original algorithm uses an exhaustive approach with backtracking. Every possible mapping is tried for Value(), and whenever an assignment results in a collision, the search path is pruned and another solution is tried. The algorithm is easy to code but can take exponential time (in the number of words) to calculate.

Chichelli suggested two ways to speed up this approach. One way is to sort the keys: Compute the frequency of each first and last letter, then sort the words according to the sum of the frequency for the first and last letters. This guarantees that the most common letters are assigned values early, and increases the probability that collisions will be found when the depth of the search tree is still small. This reduces the cost of backtracking.

This isn't always sufficient, however. Consider the set of keys:

{aa,ab,ac,ad,ae,af,ag,bd,cd,ce,de,fg},

which sorts to

{aa,ad,ac,ae,ab,af,ag,cd,de,bd,ce,fg}

Now, the value of the key cd is determined as soon as ac is given a value, but a test for collision on cd is not done until five more branches in the search tree are taken. This makes a collision on cd much more costly than it ought to be.

Chichelli suggested a second heuristic to address this problem. If a key in this sorted list has letters that have already occurred, the key is placed immediately after the one that completes its mapping. In the aforementioned example, cd will be moved immediately after ac. While doing so, you must maintain the original sorted order whenever possible. Listing One is a snippet of my C++ implementation.

There are five steps to Chichelli's algorithm:

1.Calculate the frequency of first and last letters.

2.Order the keys in decreasing order of the sum of the frequencies.

3.Reorder the keys from the beginning of the list, so that if a key's first and last letters have already appeared, that key is placed next to the completing key in the list.

4.Search cycle: For each key, if both values for first and last letters are already assigned, try to place the key in the table (step 5); otherwise, if only one value needs to be assigned, try all possible assignments. If both values must be assigned, use two nested loops to try all the possible assignments for the two letters.

5.Given the key, the length, and the Value mapping, compute the address in the table. Backtrack to step 4 if there is a collision; otherwise place the key in the table. If it is the last letter, a perfect hashing has been found; otherwise, recursively go into step 4 for the next key.

The table size is not fixed. If the table size is equal to the number of keys, you are looking for minimal perfect hashing; otherwise, some empty buckets remain at the end of the process.

Improvements

Chichelli did not suggest any way to restrict the range of values to try for each letter. This range plays an important role in the algorithm. A small range may result in no solutions, and an overly large range may increase the width of the search tree unmanageably.

I've developed a general rule that works fine for most sets, resulting in a moderately small range that does not exclude solutions. Since you add the length of the word and you want to obtain small index values (including zero), most of the letters will be assigned to negative values. Also, the letter values will never be very large because the size of the table is an upper bound for the index.

I chose a range of [-MaxLen,TableSize-MinLen), where MaxLen is the maximum length of a key belonging to the set; MinLen, the minimum length of a key; and TableSize , the size of the hash table. The range is normally small (for the keys in Figure 1, assuming minimal hashing, it is equal to [-9,1)), but still effective.

This can be further improved by adapting the range dynamically. Consider what happens when one of the two letters has already been assigned a value. This occurs twice in the algorithm: when only one value must be assigned and in the inner of the two nested loops. In this case, you already know one of the values (say, v) and the length of the word, so you can recalculate the range as [-v - Len,TableSize -v - Len], where Len is the length of the key to which you are trying to assign an index. This optimization alone produced more than an order-of-magnitude improvement in the performance of the algorithm. I've frequently seen a speed up of 40 times or more.

Note that this dynamic range may occasionally extend the original, static one. For instance, in Figure 1, y is assigned a value of 3, outside the static range. Further optimization of the dynamic range can be obtained by considering which buckets are free. If you keep track of the first and last free buckets in the hash table, the dynamic range can be restricted to [FirstFree -v - Len, LastFree - v - Len]. Listing Two is my C++ implementation of steps 4 and 5. I also added some logic to stop when the first solution is found.

The result is a fairly quick algorithm. A minimal perfect hash for the 32 keywords of ANSI C (see Figure 2) can be found in less than one second on a Pentium/90, and the 46 noncolliding keywords of C++ need about four seconds. The complete C++ implementation of my algorithm is available electronically . It has been compiled and tested on several different architectures and operating systems, and it should work without changes on most systems.

Collision Resolution and Extensions

Some words, such as the C++ keywords double and delete, result in a collision no matter which assignment is selected. There are several ways to overcome this problem: The first is to enhance the algorithm to consider letter positions other than the first and last. Another is to consider more than two letters. For instance, in "Finding Minimal Perfect Hash Functions," Haggard and Karplus (ACM SICCSE Bulletin, February 1986) tried using all letters of the word, with encouraging results.

Another idea is to partition the set of words into two or more sets. This is also effective when there are many words and the algorithm takes hours (or days) to calculate a perfect hash. The best idea would be to partition the keys into disjoint sets of first and last letters. This allows you to use a single hash table simply by adding an offset to each partition. Unfortunately, the distribution of letters usually does not allow such a partition, and you must often resort to multiple tables and, consequently, to multiple probes.

I'm currently investigating two promising ideas, one based on the concept of an articulation point in a graph, which should reduce backtracking, and another based on a look-ahead technique to prune dead paths more quickly. Both techniques should reduce the computational complexity of the algorithm in the average case, so that larger sets of keys can be conveniently mapped. If you are interested in the results of my research, drop me an e-mail message.

Figure 1: Minimal perfect hashing for days of the week.

      Letter      Value
      f           -8
      m           -4
      s           -9
      t           7
      w           -6
      y           3

      Word        Hash
      sunday      0
      monday      5
      tuesday     3
      wednesday   6
      thursday    4
      friday      1
      saturday    2

Figure 2: Minimal perfect hashing for ANSI C keywords.

      Letter      Value      Letter      Value
      a           -5         l           29
      b           -8         m           -4
      c           -7         n           -1
      d           -10        o           22
      e           4          r           5
      f           12         s           7
      g           -7         t           10
      h           4          u           26
      i           2          v           19
      k           31         w           2

      Word      Hash      Word      Hash
      auto      21        int       15
      break     28        long      26
      case      1         register  18
      char      2         return    10
      const     8         short     22
      continue  5         signed    3
      default   7         sizeof    25
      do        14        static    6
      double    0         struct    23
      else      12        switch    17
      enum      4         typedef   29
      extern    9         union     30
      float     27        unsigned  24
      for       20        void      13
      goto      19        volatile  31
      if        16        while     11

Listing One

void WordSet :: Reorder()
  {
  // reorder items in the set, so that if an item first and last letter
  // occurred before, the item is moved upward in the list. This heuristic
  // maximizes the likelihood that collisions are found early in the
  // assignment of values.

  const unsigned NO_OCCURRENCE = MAXINT ;
  for( unsigned i = 1; i < card; i++ )
    {
    const Word& w = *(set[ i ]) ;
    char first = w.First() ;
    char last = w.Last() ;
    unsigned firstOccurredIndex = NO_OCCURRENCE ;
    unsigned lastOccurredIndex = NO_OCCURRENCE ;
    for( unsigned j = 0; j < i; j++ )
      {
      const Word& prev = *(set[ j ]) ;
      if( lastOccurredIndex == NO_OCCURRENCE &&
          ( prev.Last() == last || prev.First() == last ) )
        lastOccurredIndex = j ;
      if( firstOccurredIndex == NO_OCCURRENCE &&
          ( prev.First() == first || prev.Last() == first ) )
        firstOccurredIndex = j ;
      if( firstOccurredIndex != NO_OCCURRENCE &&
          lastOccurredIndex != NO_OCCURRENCE )
        {
        // both letters occurred before: move the item upward, by pushing down
        // the ones between the maximum occurence index and the item index
        // itself. This preserve the original ordering as much as possible
        unsigned firstToMoveDown =
           max( firstOccurredIndex, lastOccurredIndex ) + 1 ;
        unsigned lastToMoveDown = i - 1 ;
        Word* itemToMoveUp = set[ i ] ;
        for( unsigned k = lastToMoveDown; k >= firstToMoveDown; k-- )
          set[ k + 1 ] = set[ k ] ;
        set[ firstToMoveDown ] = itemToMoveUp ;
        break ; // item 'i' handled
        }
      }
    }
  }
Listing Two

bool HashFinder :: Search( unsigned index )
  {
  const Word& w = words[ index ] ;
  if( value[ w.First() ] != NO_VALUE && value[ w.Last() ] != NO_VALUE )
    {
    // try to insert into the table     int h = Hash( w ) ;
    if( h < 0 || h >= maxTableSize || ! buckets.IsFree( h ) )
      return( FALSE ) ;
    else
      {
      buckets.MarkUsed( h ) ;
      index++ ;
      int card = words.Card() ;
      if( index == card )
        {
        cout << "------------\n" ;
        cout << "perfect hashing found\n" ;
        for( int i = 0; i < CHAR_MAX; i++ )
          if( value[ i ] != NO_VALUE )
            cout << (char)i << " = " << value[ i ] << "  " ;
        cout << endl ;
        for( i = 0; i < card; i++ )
          cout << words[ i ] << " = " << Hash( words[ i ] ) << endl ;
        // calculate and print loading factor
        int used = 0 ;
        for( i = 0; i < maxTableSize; i++ )
          if( ! buckets.IsFree( i ) )
            used++ ;
        cout << "loading factor: " << (float)used/maxTableSize << endl;
alues for the one without values
    // must find valid vtime
      found = found ||  res ;
       int valToAssign ;
    int valAssigned ;
    if( value[ w.First() ] == NO_VALUE )
      {
      valToAssign = w.First() ;
      valAssigned = w.Last() ;
      }
    else
      {
      valToAssign = w.Last() ;
      valAssigned = w.First() ;
      }

    // see above for heuristics used here
    int minVal2 = buckets.FirstFree() - value[ valAssigned ] - w.Len();
    int maxVal2 = buckets.LastFree() -value[valAssigned] - w.Len() + 1;

    assert( value[ valToAssign ] == NO_VALUE ) ;
    bool found = FALSE ;
    for( int i = minVal2; i < maxVal2 && ! stopNow; i++ )
      {
      value[ valToAssign ] = i ;
      bool res = Search( index ) ;  // will take first branch this
        // decide if it is time to stop now
        stopNow = stopFirst || ( used == maxTableSize && stopMinimal );

        buckets.MarkFree( h ) ; // restore for backtracking
        return( TRUE ) ;
        }
      else
        {
        bool res = Search( index ) ;
        buckets.MarkFree( h ) ; // restore for backtracking
        return( res ) ;
        }
      }
    }
  else if( value[ w.First() ] == NO_VALUE &&
           value[ w.Last() ] == NO_VALUE )
    {
    // must find values for both first and last;

    int firstValToAssign = w.First() ;
    int lastValToAssign = w.Last() ;
    bool found = FALSE ;
    for( int i = minVal; i < maxVal && ! stopNow; i++ )
      {
      value[ firstValToAssign ] = i ;
     // here we can change the heuristics: we already assigned one of
     // the values, so we have modified the boundaries.
     // Now we must have FirstFree <= to-assign + 'i' + len <= LastFree
      int minVal2 = buckets.FirstFree() - i - w.Len() ;
      int maxVal2 = buckets.LastFree() - i - w.Len() + 1 ;
for( int j = minVal2; j < maxVal2 && ! stopNow; j++ )
        {
        value[ lastValToAssign ] = j ;
        bool res = Search( index ) ;
        found = found ||  res ;
        }
      }
    value[ firstValToAssign ] = NO_VALUE ; // restore for backtracking
    value[ lastValToAssign ] = NO_VALUE ;
    return( found ) ;
    }
  else
    {
   }
    value[ valToAssign ] = NO_VALUE ; // restore for backtracking
    return( found ) ;
    }
  }

Back to Table of Contents


August 1996: Algorithm Alley

Fast String Searching With Suffix Trees

Mark R. Nelson

Mark is the author of the C++ Programmer's Guide to the Standard Template Library (IDG Books, 1995) and The Data Compression Book, Second Edition (M&T Books, 1995). He can be contacted at http://web2.airmail.net/markn.

In the January/February, 1996 issue of Dr. Dobb's Sourcebook, Michael Abrash described how to greatly accelerate three-dimensional graphics by precomputing the correct information. The suffix trees Mark Nelson describes here are similar in concept-precalculating this one data structure makes future searches very quick. Since string searching is an important component of modern compression algorithms, a faster string search has some interesting repercussions. If you haven't heard about suffix trees, that's because no one knew how to build one quickly. At least, not until last year....

-Tim Kientzle

Imagine you've just been hired as a programmer for a DNA sequencing project. Researchers are busy slicing and dicing viral genetic material, producing fragmented sequences of nucleotides. They send these sequences to your server, which is then expected to locate the sequences in a database of genomes. The genome for a given virus can have hundreds of thousands of nucleotide bases, and you have hundreds of viruses in your database. Clearly, a simple brute-force search won't be fast enough to satisfy impatient PhDs.

Since the database that you are testing against is invariant, you can preprocess it to simplify the search. One preprocessing approach is to build a search trie-a type of tree that has N possible branches from each node, where N is the number of characters in the alphabet. For searching through input text, a straightforward approach to a search trie yields a suffix trie. "Suffix" refers to the fact that the trie contains all of the suffixes of a given block of text (perhaps a viral genome).

Figure 1 shows a suffix trie for the word BANANAS. Note that, starting at the root node, each of the suffixes of BANANAS is found in the trie, starting with BANANAS, ANANAS, NANAS, and finishing up with a solitary S. Because of this organization, you can search for any substring of the word by starting at the root and following matches down the tree until exhausted.

What makes the suffix trie a nice construct is that if you have an input text of length N and search string of length M, a traditional brute-force search will take as many as N*M character comparisons to complete. Optimized searching techniques (such as the Boyer-Moore algorithm) can guarantee searches that require no more than M+N comparisons, with even better average performance. But the suffix trie requires only M character comparisons, regardless of the length of the text being searched! This means you could determine whether or not the word BANANAS was in the collected works of William Shakespeare by performing just seven character comparisons!

The catch is that suffix tries require O(N2) time and space to construct. This quadratic performance rules out the use of suffix tries where they are needed most-to search through long blocks of data.

Under the Spreading Suffix Tree

Edward McCreight gave a definitive illustration of the data structure used to circumvent this problem in his 1976 paper, "A Space-economical Suffix Tree Construction Algorithm" (Journal of the ACM, 23:262-272, 1976). The suffix tree retains the same topology as the suffix trie, but eliminates nodes that have only a single descendant. This process, known as "path compression," means that individual edges in the tree now represent sequences of text instead of single characters.

Figure 2 shows what the suffix trie from Figure 1 looks like when converted to a suffix tree. You can see that the tree still has the same general shape, just far fewer nodes. By eliminating every node with just a single descendant, the node count is reduced from 23 to 11.

The reduction in the number of nodes, in concert with an efficient algorithm, ensures that the time and space requirements for constructing a suffix tree are reduced from O(N2) to O(N). In the worst case, a suffix tree can be built with a maximum of 2N nodes, where N is the length of the input text. So for a one-time investment proportional to the length of the input text, you can create a tree that turbocharges your string searches.

McCreight's original algorithm for constructing a suffix tree had a few disadvantages. Principal among them was the requirement that the tree be built right to left; by adding the longest suffixes first. This ruled the algorithm out for online processing, making it much more difficult to use for applications such as data compression.

Twenty years later, Esko Ukkonen developed a slightly modified version of the algorithm that works from left to right. Both my sample code and the descriptions that follow are based on Ukkonen's work, described in "On-line Construction of Suffix Trees" (Algorithmica, September 1995).

For a given string of text, T, Ukkonen's algorithm starts with an empty tree, then adds each prefix of T to the suffix tree. For example, when creating the suffix tree for BANANAS, B is inserted into the tree, then BA, then BAN, and so on. When BANANAS is finally inserted, the tree is complete.

Adding a new prefix to the tree is done by walking through the tree and visiting each of the current tree's suffixes. You start at the longest suffix (BAN in Figure 3), and work down to the shortest suffix, which is the empty string. Each suffix ends at a node that consists of one of these three types:

  • A leaf node. In Figure 4, the nodes labeled 1, 2, 4, and 5 are leaf nodes.
  • An explicit node. The nonleaf nodes that are labeled 0 and 3 in Figure 4 are explicit nodes. They represent a point on the tree where two or more edges part ways.
  • An implicit node. In Figure 4, prefixes such as BO, BOO, and OO all end in the middle of an edge. These positions are referred to as implicit nodes. They would represent nodes in the suffix trie, but path compression eliminated them. As the tree is built, implicit nodes are sometimes converted to explicit nodes.

In Figure 4, there are five suffixes in the tree (including the empty string) after adding BOOK to the structure. Adding the next prefix, BOOKK, to the tree means visiting each of the suffixes in the existing tree, and adding the letter K to the end of the suffix.

The first four suffixes-BOOK, OOK, OK, and K-all end at leaf nodes. Because of the path compression applied to suffix trees, adding a new character to a leaf node will always just add to the string on that node. It will never create a new node, regardless of the letter being added.

After all of the leaf nodes have been updated, you still need to add K to the empty string, which is found at node 0. Since there is already an edge leaving node 0 that starts with the letter K, you don't have to do anything. The newly added suffix K will be found at node 0 and will end at the implicit node found one character down along the edge leading to node 2. Figure 5 shows the final shape of the resulting tree.

Things get Knotty

Updating the tree in Figure 4 was relatively easy, involving two types of updates: simply extending an edge, and performing an implicit update. Adding BOOKKE to the tree in Figure 5 demonstrates the two other types of updates. In the first type, a new node is created to split an existing edge at an implicit node, followed by the addition of a new edge. The second type of update consists of adding a new edge to an explicit node.

When adding BOOKKE to the tree in Figure 5, you once again start with the longest suffix, BOOKK, and work your way to the shortest-the empty string. Updating the longer suffixes is trivial as long as you are updating leaf nodes. In Figure 5, the suffixes that end in leaf nodes are BOOKK, OOKK, OKK, and KK. The first tree in Figure 6 shows what the tree looks like after these suffixes have been updated using the simple string extension.

The first suffix in Figure 5 that doesn't terminate at a leaf node is K. When updating a suffix tree, the first nonleaf node is defined as the active point of the tree. All of the suffixes that are longer than the suffix defined by the active point will end in leaf nodes. None of the suffixes shorter than it will terminate in leaf nodes.

The suffix K terminates in an implicit node part way down the edge defined by KKE. When testing nonleaf nodes, you need to see if they have any descendants that match the new character being appended; in this case, that would be E.

A quick look at the first K in KKE shows that it only has a single descendant: K. This means you have to add a descendent to represent E. This is a two-step process. First, split the edge holding the arc so that it has an explicit node at the end of the suffix being tested. The middle tree in Figure 6 shows what the tree looks like after the split.

Once the edge has been split and the new node has been added, you have a tree that looks like that in the third position of Figure 6. Note that the K node, which has now grown to be KE, has become a leaf node.

Updating an Explicit Node

After updating the suffix K, you have to update the next-shorter suffix, which is the empty string. The empty string ends at explicit node 0, so just check to see if it has a descendant that starts with the letter E. A quick look at the tree in Figure 6 shows that node 0 doesn't have this descendent, so another leaf node is added, which yields the tree in Figure 7.

Generalizing the Algorithm

By taking advantage of a few of the characteristics of the suffix tree, you generate a fairly efficient algorithm. The first important trait is "once a leaf node, always a leaf node." Any node created as a leaf will never be given a descendant; it will only be extended through character concatenation. More importantly, every time a new suffix is added to the tree, you are going to automatically extend the edges leading into every leaf node by a single character. That character will be the last character in the new suffix.

This makes management of the edges leading into leaf nodes easy. Any time you create a new leaf node, just set its edge to represent all the characters from its starting point to the end of the input text. Even though you don't know what those characters are, you know they will be added to the tree eventually. Because of this, once a leaf node is created, just forget about it. If the edge is split, its starting point may change, but it will still extend all the way to the end of the input text.

This means that you only have to worry about updating explicit and implicit nodes starting at the active point, which was the first nonleaf node. Given this, you progress from the active point to the empty string, testing each node for update eligibility.

However, you can save some time by stopping the update earlier. As you walk through the suffixes, add a new edge to each node that doesn't have a descendant edge starting with the correct character. When you finally do reach a node that has the correct character as a descendant, simply stop updating. Knowing how the construction algorithm works, you can see that if you find a certain character as a descendent of a particular suffix, you are bound to also find it as a descendent of every smaller suffix.

The point where you find the first matching descendent is called the "end point" and it has a particularly useful feature. Since you add leaves to every suffix between the active point and the end point, you know that every suffix longer than the end point will soon be a leaf node. This means the end point will turn into the active point on the next pass over the tree!

By confining updates to the suffixes between the active point and end point, you can cut the processing required to update the tree. And by keeping track of the end point, you automatically know what the active point will be on the next pass. Example 1 is a first pass at the update algorithm using this information.

The Suffix Pointer

The pseudocode in Example 1 is more or less accurate, but glosses over one difficulty-as you navigate through the tree, you repeatedly call next_smaller_suffix(). This routine has to find the implicit or explicit node corresponding to a particular suffix.

If you do this by simply walking down the tree to find the correct node, the algorithm isn't going to run in linear time. To get around this, add one additional pointer to the tree-the suffix pointer. The suffix pointer is found at each internal node. Each internal node represents a sequence of characters that start at the root. The suffix pointer points to the node that is the first suffix of that string. So if a particular string contains characters 0 through N of the input text, the suffix pointer for that string will point to the node that is the termination point for the string starting at the root that represents characters 1 through N of the input text.

Figure 8 shows the suffix tree for the string ABABABC. The first suffix pointer is found at the node that represents ABAB, and points to BAB. Likewise, BAB has its own suffix pointer, which points to the node for AB.

The suffix pointers are built as the tree is updated. As I move from the active point to the end point, I keep track of the parent node of each of the new leaves I create. Each time I create a new edge, I also create a suffix pointer from the parent node of the last leaf edge I created to the current parent edge. (Obviously, I can't do this for the first edge created in the update, but I can for all the remaining edges.)

With the suffix pointers in place, navigating from one suffix to the next is simply a matter of following a pointer. This critical addition to the algorithm is what reduces it to an O(N) algorithm.

Tree Houses

To illustrate suffix trees, I wrote STREE.CPP, a program that reads in a string of text from standard input and builds a suffix tree. A second version, STREED.CPP, has extensive debug output. Both programs are available electronically and from my home page.

Understanding STREE.CPP is just a matter of understanding the workings of the data structures it contains. The most important data structure is the Edge object; see Example 2.

Each time a new edge in the suffix tree is created, a new Edge object is created to represent it. The four data members of the object are:

  • first_char_index and last_char_index. Each edge in the tree has a sequence of characters from the input text associated with it. To ensure that the storage size of each edge is identical, I just store two indices into the input text to represent the sequence.
  • start_node. The number of the node that represents the starting node for this edge. Node 0 is the root of the tree.
  • end_node. The number of the node that represents the end node for this edge. Each time an edge is created, a new end node is created as well. The end node for every edge will not change over the life of the tree, so this can be used as an edge id as well.
A frequent task performed when building the suffix tree is a search for the edge emanating from a particular node based on the first character in its sequence. On a byte-oriented computer, there could be as many as 256 edges originating at a single node. To make the search reasonably quick and easy, I store the edges in a hash table, using a hash key based on their starting node number and the first character of their substring. The Insert() and Remove() member functions are used to manage the transfer of edges in and out of the hash table.

The second important data structure used when building the suffix tree is the Suffix object. Remember that updating the tree is done by working through all of the suffixes of the string currently stored in the tree, starting with the longest and ending at the end point. A Suffix is simply a sequence of characters that starts at node 0 and ends at some point in the tree.

It makes sense that we can then safely represent any suffix by defining the position in the tree of its last character, since we know the first character starts at node 0, the root. The Suffix object (see Example 3) defines a given suffix using that system.

The Suffix object defines the last character in a string by starting at a specific node, then following the string of characters in the input sequence pointed to by the first_char_index and last_char_index members. For example, in Figure 8, the longest suffix, ABABABC, would have an origin_node of 0, first_char_index of 0, and last_char_index of 6.

Ukkonen's algorithm requires that you work with these Suffix definitions in canonical form. The Canonize() function is called to perform this transformation any time a Suffix object is modified. The canonical representation of the suffix simply requires that the origin_node in the Suffix object be the closest parent to the end point of the string. This means that the suffix string represented by the pair (0, "ABABABC"), would be canonized by moving first to (1, "ABABC"), then (4, "ABC"), and finally (8,"").

When a suffix string ends on an explicit node, the canonical representation will use an empty string to define the remaining characters in the string. An empty string is defined by setting first_char_ index to be greater than last_char_index. When this is the case, you know that the suffix ends on an explicit node. If first_char_index is less than or equal to last_char_index, the suffix string ends on an implicit node.

Given these data structure definitions, you will find the code in STREE.CPP to be a straightforward implementation of the Ukkonen algorithm. For additional clarity, use STREED.CPP to dump copious debug information out at run time.

Acknowledgment

I was convinced to tackle suffix-tree construction by reading Jesper Larsson's paper for the 1996 IEEE Data Compression Conference. Jesper was also kind enough to provide me with sample code and pointers to Ukkonen's paper.

Figure 1: The suffix trie representing BANANAS.

Figure 2: A suffix tree representing BANANAS.

Figure 3: Progressively building the suffix tree.

Figure 4: BOOKKEEPER after adding BOOK.

Figure 5: The same tree after adding BOOKK.

Figure 6: The split and add update.

Figure 8: The suffix tree for ABABABC with suffix pointers are shown as curved and solid lines. The normal edges of the tree are shown as dashed/dotted lines.

Figure 7: Suffix tree for BOOKKE.

Example 1: Updating the suffix tree (in C-like pseudocode).

Update( new_suffix )
{
  current_suffix = active_point
  test_char = last_char in new_suffix
  done = false;
  while ( !done ) {
    if current_suffix ends at an explicit node {
      if the node has no descendant edge starting with test_char
        create new leaf edge starting at the explicit node
      else
        done = true;
    } else {
      if the implicit node's next char isn't test_char {
        split the edge at the implicit node
        create new leaf edge starting at the split in the edge
      } else
        done = true;
    }
    if current_suffix is the empty string
      done = true;
    else
       current_suffix = next_smaller_suffix( current_suffix )
  }
  active_point = current_suffix
}

Example 2: The Edge class.

class Edge {
 public :
    int first_char_index;
    int last_char_index;
    int end_node;
    int start_node;
    void Insert();
    void Remove();
    Edge();
    Edge( int init_first_char_index,
          int init_last_char_index,
          int parent_node );
    int SplitEdge( Suffix &s );
    static Edge Find( int node, int c );
    static int Hash( int node, int c );
};

Example 3: The Suffix class.

class Suff ix {
 public :
   int origin_node;
   int first_char_index;

   int last_char_index;
   Suff ix( int node, int start, int stop );
   int Explicit();
   int Implicit();
   void Canonize();
};

Back to Table of Contents


September 1996: Algorithm Alley

HATs: Hashed Array Trees

Fast variable-length arrays

Edward Sitarski

Edward is vice president of R&D at Numetrix Ltd. He can be reached at ed@ tor.numetrix.com.

Arrays are the most natural and convenient data structure for a great many applications. They provide constant access time to their elements and are the structure of choice for many practical algorithms. Most commercial C++ toolkits offer a variable-length array container class, and probably millions of C++ programs use variable-length arrays as their core data structures. Increasing the size of a variable-length array by appending to the end is a common operation; if we knew the size of the array at the start then there would be no reason to use a variable-length array.

I became painfully aware of the limitations of most implementations of variable-length arrays when investigating some code that read the results of a database query. There was no way to determine the number of rows returned from the query beforehand, and the code used a commercial variable-length array class to store the data. The profiler told me that the array class was doing an incredible amount of element copying when resizing. This copying was taking much more time than the slow data transfer from the database!

You might argue that another data structure, such as a linked list, would have been more appropriate for this application, but it was too difficult to fix all the old code. I decided to try to implement a better performing array class that had the same interface as the old one.

Many implementations of variable-length arrays leave much to be desired. Specifically, the function that appends to the end of the array can cause a large performance problem. Example 1 shows just about the worst way to do it. There are two main reasons why this implementation is so bad.

  • Excessive element copying. If you add 1000 elements to the array, the internal loop will copy the old elements into newTArray each time the array expands, a total of over 500,000 copy operations. This seemingly innocent code is doing about 500 times more assignments than necessary for the 1000 elements, and it only gets worse as the array grows. This implementation is O(N2) (the same time complexity as Bubblesort) and highly inefficient even for small arrays.
  • Excessive memory usage. During the copy operation, both the existing array and the new array must be present in memory at the same time, doubling the memory requirement of the array. Worse, each time the array grows, it creates a memory fragment: the old array. This fragment cannot be reused the next time the array grows because the new array is larger and will not fit, as shown in Figure 1. This can increase the memory requirements by almost another factor of two.

It is interesting to note that C's realloc() function does not have such severe problems. The reason is that once the array is large enough, it will be allocated from the end of the heap. realloc can then expand the buffer by increasing the size of the memory block without having to copy the data. Unfortunately, this does not work with C++ because realloc cannot call constructors on the expanded memory. C++ requires us to copy the elements explicitly.

You might try to address these problems by growing the array by more than one element at a time. However, this does not significantly reduce the memory usage. Even though it reduces the total number of copies, it is still an O(N2) algorithm-all you've done is decrease the running constant. For example, if you add 1000 elements and grow the array by 10 elements whenever it's full, you are still performing 10+20+..+1000 = 50,500 copy operations, which is still O(N2). Although this is 10 times less copying than before, it is still 50 times more copying than is necessary.

HATs Usually Append in O(1) Time

To overcome the limitations of variable-length arrays, I created a data structure that has fast constant access time like an array, but mostly avoids copying elements when it grows. I call this new structure a "Hashed-Array Tree" (HAT) because it combines some of the features of hash tables, arrays, and trees.

Although used to implement one-dimensional arrays, HATs are really two-dimensional; see Figure 2. A HAT consists of a Top array and a number of Leaves which are pointed to by the Top array. The number of pointers in the Top array and the number of elements in each Leaf is the same, and is always a power of 2.

Because the Top and Leaf arrays are powers of 2, you can efficiently find an element in a HAT using bit operations; see Example 2. Usually, appending elements is very fast since the last leaf may have empty space. Less frequently, you'll need to add a new leaf, which is also very fast and requires no copying.

When the Top array is full, it becomes a bit more interesting. My implementation first computes the correct size (Top and Leaf arrays are the same size, both a power of 2), then copies the elements into a new HAT structure, freeing the old leaves and allocating new leaves as it goes.

This approach dramatically avoids most of the element copying performed by the previous array implementation. Recopying only occurs when the Top array is full, and this only happens when the number of elements jusrt exceeds the square of a power of 2. If N=4n, then the total amount of recopying is 1+4+16+64+256+...+N. Using the identity (x(n+1)-1)=(x-1) (1+x+x2+x3+... + xn) you have 1 +4+42+43+...+4n = (4(n+1) -1)/(4-1) = (4N-1)/3, or about 4/3 N. This means that the average number of extra copy operations is O(N) for sequentially appending N elements, not O(N2).

The technique of reallocating and copying each leaf one at a time also significantly reduces the memory overhead. Instead of needing memory for a complete copy of the HAT, you only need enough memory for an extra leaf. Because the leaf size depends on the square root of N, the extra memory required during resizing decreases dramatically as a percentage of N. For example, if you added 1,000,000 elements to the HAT, the extra memory needed for the last resize would be 1024 elements or about 0.1 percent of the memory already used for the array's data.

In addition, memory fragmentation is reduced. Since the Top and Leaf sizes always increase to the next power of 2, the heap manager may be able to combine two freed leaves from the smaller HAT to allocate a leaf in the larger one.

Available electronically is code for a HAT template that includes some further optimizations to eliminate copying when the HAT grows and shrinks across resize boundaries. This is achieved at some expense of potential memory fragmentation, but results in smoother performance. The implementation allows the Top array to increase as a multiple of its former ideal size until an up or down threshold is reached.

Even if resizing is not an issue, this implementation has other advantages. For example, in a 16-bit environment, no object on the heap can exceed 64 KB. A HAT can manage arrays much larger than this without any single leaf exceeding 64 KB. You could also extend HATs to automatically swap leaves to and from disk storage to support truly huge arrays with a minimal memory footprint. Finally, there may be advantages to extending HATs to three or more levels rather than the two levels I've used.

Memory Overhead

I've already suggested that HATs use much less memory than the standard approach of reallocating and copying the entire array. To see how much less, I'll compute the actual memory overhead of a HAT resize.

The worst case happens when the elements in the HAT are the same size as pointers and the number of elements is one greater than a resize value. If the elements are the same size as pointers, you can add the unused portion of the Top array to any wasted memory in the Leaves, thus maximizing the amount of wasted memory as a percentage of the data in the array; see Table 1. You can see that as the HAT grows, the percentage of wasted memory in the worst case drops dramatically. Generally, the worst-case memory waste is (top+leaf-1) ~= 2*sqrt(N) = O(sqrt(N)). If you expect the last leaf to be half full, the expected memory waste drops to (top + leaf/2) ~= 1.5*sqrt(N), which is still O(sqrt(N)). This overhead compares well with other data structures that can add elements in expected O(1) time. For example, singly linked lists require O(N) memory overhead (one pointer for each element).

Performance

In a built-in C++ array, the address of the element is computed by adding the index of the element, times its size, to the start of the array. By comparison, finding the address of an element in a HAT requires two such array lookups, a pointer dereference, and the topIndex and leafIndex calculations. By precomputing (1power)-1, this becomes two array lookups, a pointer dereference, a bit shift, and an & operation. On modern processors, the last three can be done very quickly. This means that HAT elements can be dereferenced in about twice the time required for a C++ array.

Figure 3 shows the results of a quicksort implementation comparing standard C++ arrays and HATs. This algorithm uses array references very heavily, and supports the estimate that HATs are about twice as slow as regular arrays.

Figure 4 compares the two types of arrays under slightly different circumstances. In this case, N random integers are appended to the array and then sorted. Even though the HAT is slower for array references, the faster append makes the overall operation of this test significantly faster.

The benchmarks compare HATs to a resizable-array class implemented in the standard way (reallocate and copy) that grows by 64 elements whenever it runs out of space. The quicksort algorithm is a template that accepts either resizable-array class. All benchmarks were run on a 100Mhz HP 700 series PA-RISC workstation running HPUX 9.01 with 256 MB of memory.

Conclusion

HATs are a practical and efficient way to implement variable-length arrays. They offer highly desirable O(N) performance to add N elements to an empty array and only require O(sqrt(N)) memory overhead. They provide all the ease of use and standard features of normal arrays, including random access to elements. They maintain their performance and memory-usage characteristics over any number of elements and require no special application performance tuning.

Although Hashed-Array Trees require more time to access their elements, this drawback is often far outweighed by their fast resizing, efficient memory utilization, and high programmer convenience. They offer strong advantages over other methods of implementing variable-length arrays.

References

Cline, M.P. and G.A. Lomow, C++ FAQs, Reading, MA: Addison-Wesley, 1995.

Cormen, T.H., C.E. Leiserson, and R.L. Rivest. Introduction to Algorithms, Cambridge, MA: MIT Press, 1990.

Example 1: A bad way to resize an array.

void vArray::append( T &el )
{
     // Extremely bad implementation of
     // variable length array.
     T *newTArray = new T [numElements+1];
     for(size_t j = 0; j < numElements; j++)
         newTArray[j]  =     tArray[j];
     delete tArray;
     tArray = newTArray;
     tArray[maxElements++] = el;
}

Example 2: Fast indexing with bit operations.

inline size_t Hat::topIndex(const size_t j ) const
{
    // Get the high index.
    return j  power;
}
inline size_t Hat::leafIndex(const size_t j ) const
{
    // Get the low index.
    return j & ((1power)-1);
}
inline T &Hat::operator[](const size_t j) const
{
    // Return an element in the HAT. Do no bounds checking.
    return top[topIndex(j)][leafIndex(j)];
}

Table 1: Worst-case memory overhead.

Top/Leaf     Worst     Worst Waste
  Size         N       (top+leaf-1)     Percent of Data

    2           3            1+2=3             100
    4           5            4+3=7             140
    8          17           8+7=15              80
   16          65         16+15=31              48
   32         257         32+31=63              25
   64        1025        64+63=127               8
  128        4097      128+127=255               6
  256       16385      256+255=511               3

Figure 1: Memory usage during array resizing.

Figure 2: HAT data structure.

Figure 4: Append and Sort with standard arrays and Hats.

Figure 3: QuickSort with standard arrays and HATs.

Back to Table of Contents


Dr. Dobb's Journal January 1997: Algorithm Alley

The Fibonacci Heap

By John Boyer

In September 1996, you saw how a simple tree structure can improve the performance of a resizable-array class. The idea was not to make every operation fast, but to keep the average cost low by making expensive operations infrequent. That same idea appears again this month, as John shows us how a roughly balanced tree of circularly linked lists can be used to implement a very fast heap.

--Tim Kientzle

John is a software development manager at UWI Unisoft Wares Inc. in Victoria, British Columbia and is a part-time graduate student at the University of Victoria. He can be contacted at jboyer@uwi.com.

Heaps are simple, useful containers. A heap allows you to insert elements and extract the least element. Many heap implementations also allow you to decrease an element already in the heap, and there are a variety of less-common operations. Table 1 shows the operations supported by the Fibonacci heap implementation I'm going to discuss in this article.

One of the first uses for a heap was to implement a priority queue. In fact, before researchers began realizing the wealth of algorithms to which a heap is well suited, the heap was actually called a "priority queue." Priority queues are used to keep a dynamic list of tasks of differing priorities. The Insert() operation adds a new job to the queue. The ExtractMin() operation extracts the highest-priority task. If a job suddenly required a higher priority, the DecreaseKey() operation would be used.

A heap can be used to implement a worst-case O(n log) sorting algorithm, which is the best-possible rating for a sort that uses only key comparisons. You simply insert all elements into the heap, then ExtractMin() until the heap is empty. The items will be extracted in ascending order. Heaps can also be used to construct optimal binary trees for Huffman compression, as shown in Mark Nelson's article "Priority Queues and the STL" (Dr. Dobb's Journal, January 1996).

Function | Description
Insert() | Inserts a new node into the heap; the new node must contain a key value.
ExtractMin() | Returns the node of minimum value after removing it from the heap.
DecreaseKey() | Assigns a new, smaller key value to a node; the node may need to be repositioned in the heap so that it is extracted when there are no nodes with lesser values than the NEW key value. A pointer to the node must be given because heaps don't support an efficient search operation.
Union() | Creates a new heap by joining two heaps given as input.
Minimum() | Returns a reference to the node containing the minimum key value or some indication of what the key value is.
Delete() | Deletes any node from the heap. A pointer to the node must be given.
Table 1: Heap operations.

One important algorithm that uses a heap is Dijkstra's shortest-path algorithm. This algorithm finds the least expensive path through a weighted graph (also called a "network").

Dijkstra's algorithm sets the cost of each vertex (except the starting vertex) to infinity and puts all the vertices onto a heap. You then extract the cheapest vertex from the heap -- call it M -- and examine each vertex A adjacent to M. If the cost of M plus the cost of the edge joining M to A is cheaper than the current cost of A (that is, if there's a cheap path to A through M), you create a link from A to M and decrease A's key to represent the new cost. You continue extracting successive nodes until you reach T, the target vertex. The value of T is the cost of the shortest path. The links from T back to the starting vertex indicate the shortest path.

Input:  Graph G, vertices S (start), T (terminate)
Declare:  H (initially empty heap)
1: For all vertices v
2:    if v == S then v.cost := 0
3:    else v.cost := infinity
3:    Insert v into H
4: Repeat
5:    M := ExtractMin(H)
6:    For each vertex A attached to M
7:       w := cost of edge from M to A
8:       if (M.cost + w < A.cost)
9:          DecreaseKey(A,M.cost + w)
10:         A.backlink := M
11: Until M = T
12: Output T.cost
13: Output vertices on chain of backlinks from T to S
Figure 1: Dijkstra's shortest-path algorithm.

As you can see in Figure 1, the DecreaseKey() on line 9 is the most time-consuming operation of the inner loop. Since Dijkstra's algorithm is important in network routing and other applications, it would be nice to find a heap implementation that makes this operation as fast as possible. This is the primary motivation for the Fibonacci heap.

The Fibonacci Heap

Heaps are usually implemented using binary trees, with the property that for every subtree, the root is the minimum item. In 1984, Michael Fredman and Robert Tarjan described a new way to implement heaps. In particular, the Fibonacci heap, or F-heap, is exceptionally fast at the DecreaseKey() and Insert() operations. For a typical binary heap, these operations require logarithmic time, while a Fibonacci heap requires only constant time. Table 2 shows how other operations compare.

There are some caveats, however. First, a Fibonacci heap uses more memory than a binary heap, since it requires a number of additional data elements to keep track of the items. Second, the time efficiency of the different operations is based on "amortized analysis." With the binary heap, the time reflects the total time for that operation. For an F-heap, a particular call to ExtractMin(), for example, might take a very long time because it's doing leftover work from other operations. With amortized analysis, this additional work is accounted for in the operations that cause the work.

Behind the Scenes

The Fibonacci heap stores all elements in a collection of circular, doubly linked lists. In addition to the Left and Right pointers used to implement the lists, each node has a Child pointer that allows it to be the parent of another circular list and a corresponding Parent pointer. There is also, of course, the application-specific key value and three variables called Degree, Mark, and Negative Infinity.

At any given time, there is a MinRoot pointer that points to the smallest item on the top list. In addition, if any item has a child list, that item is smaller than anything on the child list. Insertion is simple: You add the new item beside the current MinRoot and possibly adjust MinRoot if the new item is smaller. This always takes constant time.

The Union() operation is used by other heap operations. It combines two heaps by linking the top-level lists into a single circular list, much the way that two soap bubbles join at a point then expand to form one larger bubble. Again, this linkage can always be done in constant time.

The ExtractMin() operation is the real workhorse of the Fibonacci heap. After extracting the current MinRoot (and unioning its child list into the top level), the entire top-level list is traversed, both to find the new MinRoot and to rearrange the entire heap into a more efficient structure. Each node has a Degree variable that indicates the number of direct children of that node. The heap is rearranged so that no two nodes on the top list have the same degree. One interesting property is that the total number of descendants of any node will be about 2Degree. For example, suppose you insert 12 nodes into a Fibonacci heap. The heap then looks like a single circular list. After you extract the minimum element, there will be 11 nodes left. Three of these will be on the root list, with degrees of 0, 1, and 3. The node of degree three would be the root of a subtree with a total of eight nodes.

The detailed operation of ExtractMin() involves traversing the top-level loop and keeping an array with one element for each possible degree value; see Listing One. As you traverse the loop, you put a pointer to that element into the array. If there's already an element of that degree, you add the larger element to the child list of the smaller, which increases the degree of the smaller. This smaller element is then put back into the array, which may require repeating the process. Figure 2 shows how this works.

DecreaseKey() specifies a node in the heap and a new, smaller key value. If the key value is larger than the parent, then that node must be moved up in the heap. The Fibonacci heap simply removes the node from its current parent (reducing the parent's degree), and inserts it into the root list, where it could become the new MinRoot. The Fibonacci heap then does some additional work to try to maintain the overall balance of the heap; see Listing Two.

The Fibonacci heap relies on maintaining a certain rough balance to its structure, and moving many items from deep in the heap to the top can upset that balance. This is the purpose of the Mark field. Whenever a child is moved out of a parent during a DecreaseKey() operation, the parent's Mark field is set. If the Mark field is already set, it indicates a second child being lost, so the parent is also moved to the top. This "cascading cut" must then consider the next node up, which is now losing a child. This process stops when it reaches a node that is not Marked, or when it reaches a node on the root list (a node that has no parent).

To see why the DecreaseKey() operation manages to have constant amortized time, even with this cascading cut, consider a Fibonacci heap with N elements, and suppose you perform N DecreaseKey() operations. Although any particular operation could move a large number of subtrees, I want to show that the total number of subtrees moved is proportional to N. How many times can each node be visited during a cascading cut? The first time it's visited, it will be marked. The second time, it will be moved to the root list. So, each node can be visited twice before it becomes a root node. Since the cascading cut always stops when it hits a root node, the total number of root node visits can't be more than the total number of cascading cuts. Thus, there are at most 3N node visitations required for N DecreaseKey() operations.

Although the full analysis requires more complex methods to account for mixing the various operations, this gives you some idea of how DecreaseKey() can achieve constant amortized time.

The Delete() operation is actually remarkably simple. Each node has a special Negative Infinity flag that forces it to have a value smaller than any other node. To delete a node, you effectively decrease the key to negative infinity, then extract it. Since ExtractMin() is O(log), so is Delete().

A Fix for the Expandable Binary Heap

I've used the same basic design to build a dynamic binary heap. This is an unusual thing to do. Usually, binary heaps are declared to be of a certain size and don't grow or shrink. An expandable binary heap must be constructed with great care to ensure that the Insert() and ExtractMin() operations don't degrade to O(n) behavior. Specifically, you must only grow the heap log times, doubling the heap array size each time an expansion is necessary. This will ensure that the nodes will be copied to a new array a maximum of 1+2+4+...+n= 2n-1 times, so the cost of expansion is constant per insertion. Likewise, the heap array should be reduced to half its size when it is only one-quarter full. If it is reduced to half size at precisely the point when it is half full, then a sequence of Insert() and ExtractMin() operations could alternate right at that moment, causing expansion and shrinking at every operation.

How to Use the Code

The source code for this Fibonacci heap implementation (available electronically; see "Availability," page 3) defines two classes: FibHeapNode and FibHeap. The FibHeap class provides the heap operations such as Insert() and ExtractMin(). It is intended to be used as is; no subclassing is required. The FibHeapNode class should be subclassed to store your particular data and to redefine the virtual functions used by FibHeap.

The file FIBTEST.CPP shows how this subclassing should be done. In particular, when overriding the assignment, equality, and less-than functions, you must call the corresponding protected base-class function first. For example, FHN_Cmp() handles the Negative Infinity test, so you should not do your own test if the base class function indicates a meaningful value.

Binary Heap versus F-Heap

The F-heap provides a great enhancement to Dijkstra's algorithm and to other algorithms that can use DecreaseKey() effectively. The big surprise, though, is that the F-heap is even competitive on regular heap algorithms like heap sorting. The test programs generate random test data and compare the output against qsort() to make sure the heap is operating correctly. To determine the overall time complexity, you can compare the time for 2048-element and 1024-element data sets. If the process is O(n log), the ratio should be 2.2 to 1. The test program averages a ratio of 2.17 for a binary heap and 2.07 for a Fibonacci heap. The decreased ratio is due to the fact that more of the F-heap operations execute in linear time, including the Insert() and DecreaseKey() operations. In addition, the Fibonacci heap test runs about 15 percent faster than the binary heap test. Even if you remove the DecreaseKey() operation from the test, the Fibonacci heap is still over 10 percent faster. For algorithms like Dijkstra's, the difference can be two to ten times faster on a 1024-vertex graph.

References

Cormen, T., C. Leiserson, and R. Rivest. Introduction to Algorithms. Cambridge, MA: MIT Press, 1990.

Fredman, M. and Tarjan, R. "Fibonacci Heaps and Their Uses in Improved Network Optimization Algorithms." Journal of the Association for Computing Machinery, vol. 34, no. 3, July 1987.

DDJ

Listing One

//=====================================================================// ExtractMin() - O(n) worst-case actual; O(lg n) amortized
//=====================================================================

FibHeapNode *FibHeap::ExtractMin()
{
FibHeapNode *Result;
FibHeap *ChildHeap = NULL;

// Remove minimum node and set MinRoot to next node
     if ((Result = Minimum()) == NULL)
          return NULL;
     MinRoot = Result->Right;
     Result->Right->Left = Result->Left;
     Result->Left->Right = Result->Right;
     Result->Left = Result->Right = NULL;

     NumNodes --;
     if (Result->Mark) {
         NumMarkedNodes --;
         Result->Mark = 0;
     }
     Result->Degree = 0;

// Attach child list of Minimum node to the root list of the heap
// If there is no child list, then do no work
     if (Result->Child == NULL) {
         if (MinRoot == Result)
             MinRoot = NULL;
     }
// If MinRoot==Result then there was only one root tree, so the root list is
// the child list of that node (NULL if this is the last node in the list)
     else if (MinRoot == Result)
         MinRoot = Result->Child;
// If MinRoot is different, then the child list is pushed into a new temporary
// heap, which is then merged by Union() onto the root list of this heap.
     else {
         ChildHeap = new FibHeap();
         ChildHeap->MinRoot = Result->Child;
     }
// Complete the disassociation of the Result node from the heap
     if (Result->Child != NULL)
         Result->Child->Parent = NULL;
     Result->Child = Result->Parent = NULL;
// If there was a child list, then we now merge it with rest of the root list
     if (ChildHeap)
         Union(ChildHeap);
// Consolidate heap to find new minimum and do reorganize work
     if (MinRoot != NULL)
         _Consolidate();
// Return the minimum node, which is now disassociated with the heap
// It has Left, Right, Parent, Child, Mark and Degree cleared.
     return Result;
}
//====================================================================
// Consolidate(). Internal function that reorganizes heap as part of an
// ExtractMin(). We must find new minimum in heap, which could be anywhere
// along the root list. The search could be O(n) since all nodes could be on
// the root list. So, we reorganize the tree into more of a binomial forest
// structure, and then find the new minimum on the consolidated O(lg n) sized
// root list, and in the process set each Parent pointer to NULL,
// and count the number of resulting subtrees.
// After a list of n inserts, there will be n nodes on the root list,
// so the first ExtractMin() will be O(n) regardless of whether or not we
// consolidate. However, the consolidation causes subsequent ExtractMin()
// operations to be O(lg n). Furthermore, the extra cost of the first
// ExtractMin() is covered by the higher amortized cost of Insert(), which is
// the real governing factor in how costly the first ExtractMin() will be.
//=====================================================================
void FibHeap::_Consolidate()
{
FibHeapNode *x, *y, *w;
FibHeapNode *A[1+8*sizeof(long)]; // 1+lg(n)
int  I=0, Dn = 1+8*sizeof(long);
short d;

// Initialize the consolidation detection array
     for (I=0; I < Dn; I++)
          A[I] = NULL;
// We need to loop through all elements on root list.  When a collision of
// degree is found, the two trees are consolidated in favor of the one with
// the lesser element key value. We first need to break the circle so that we
// can have a stopping condition (we can't go around until we reach the tree
// we started with because all root trees are subject to becoming a child
// during the consolidation).
     MinRoot->Left->Right = NULL;
     MinRoot->Left = NULL;
     w = MinRoot;

     do {
        x = w;
        d = x->Degree;
        w = w->Right;
// We need another loop here because the consolidated result
// may collide with another large tree on the root list.
        while (A[d] != NULL) {
             y = A[d];
             if (*y < *x)
                 _Exchange(x, y);
             if (w == y) w = y->Right;
             _Link(y, x);
             A[d] = NULL;
             d++;
        }
    A[d] = x;
     } while (w != NULL);
// Now we rebuild the root list, find the new minimum, set all root list
// nodes' parent pointers to NULL and count the number of subtrees.
     MinRoot = NULL;
     NumTrees = 0;
     for (I = 0; I < Dn; I++)
          if (A[I] != NULL)
              _AddToRootList(A[I]);
}
//=====================================================================
void FibHeap::_AddToRootList(FibHeapNode *x)
{
     if (x->Mark) NumMarkedNodes --;
     x->Mark = 0;
     NumNodes--;
     Insert(x);
}
//==== Node y is moved from the root list to become a subtree of node x. ====
void FibHeap::_Link(FibHeapNode *y, FibHeapNode *x)
{
// Remove node y from root list
     if (y->Right != NULL)
        y->Right->Left = y->Left;
     if (y->Left != NULL)
         y->Left->Right = y->Right;
     NumTrees--;
// Make node y a singleton circular list with a parent of x
     y->Left = y->Right = y;
     y->Parent = x;
// If node x has no children, then list y is its new child list
     if (x->Child == NULL)
        x->Child = y;
// Otherwise, node y must be added to node x's child list
     else {
         y->Left = x->Child;
         y->Right = x->Child->Right;
         x->Child->Right = y;
         y->Right->Left = y;
     }
// Increase the degree of node x because it's now a bigger tree
     x->Degree ++;
// Node y has just been made a child, so clear its mark
     if (y->Mark) NumMarkedNodes--;
         y->Mark = 0;
}

Back to Article

Listing Two

//=====================================================================// DecreaseKey() - O(lg n) actual; O(1) amortized
// The O(lg n) actual cost stems from the fact that the depth, and
// therefore the number of ancestor parents, is bounded by O(lg n).
//=====================================================================

int  FibHeap::DecreaseKey(FibHeapNode *theNode, FibHeapNode& NewKey)
{
FibHeapNode *theParent;

     if (theNode==NULL || *theNode < NewKey)
         return NOTOK;
     *theNode = NewKey;

     theParent = theNode->Parent;
     if (theParent != NULL && *theNode < *theParent) {
         _Cut(theNode, theParent);
         _CascadingCut(theParent);
     }
     if (*theNode < *MinRoot)
         MinRoot = theNode;
     return OK;
}
//==== Remove node x from the child list of its parent node y ====
void FibHeap::_Cut(FibHeapNode *x, FibHeapNode *y)
{
     if (y->Child == x)
         y->Child = x->Right;
     if (y->Child == x)
         y->Child = NULL;
     y->Degree --;

     x->Left->Right = x->Right;
     x->Right->Left = x->Left;

     _AddToRootList(x);
}
//=====================================================================
// Cuts each node in parent list, putting successive ancestor nodes on the root
// list until we either arrive at the root list or until we find an ancestor
// that is unmarked. When a mark is set (which only happens during a cascading
// cut), it means that one child subtree has been lost; if a second subtree is
// lost later during another cascading cut, then we move the node to the root
// list so that it can be re-balanced on the next consolidate.
//=====================================================================

void FibHeap::_CascadingCut(FibHeapNode *y)
{
FibHeapNode *z = y->Parent;

     while (z != NULL) {
         if (y->Mark == 0) {
             y->Mark = 1;
             NumMarkedNodes++;
             z = NULL;
         } else {
             _Cut(y, z);
             y = z;
         z = y->Parent;
         }
     }
}

Back to Article

Copyright © 1997, Dr. Dobb's Journal

Back to Table of Contents


Dr. Dobb's Journal January 1997: Algorithm Alley, Figure 2

The Fibonacci Heap

By John Boyer

Figure 2: The ExtractMin() consolidation process. (a) The heap after inserting values 0-12 and extracting 0. Now, start from 11 and consolidate the heap; (b) 10 and 11 both had degree zero, so were joined; (c) 8 and 9 were joined; (d) 8 and 10 both had degree one; (e) 7,6,5, and 4 were consolidated. (f) Eight and 4 both had degree two; (g) Finished. One has degree zero, 2 has degree one, 4 has degree three. Note that every element is smaller than all of its children.

BACK TO ARTICLE
Copyright © 1997, Dr. Dobb's Journal


Dr. Dobb's Journal January 1997: Algorithm Alley, Table 2

The Fibonacci Heap

By John Boyer

Table 2: Comparing the binary and Fibonacci heaps. means that the operation's time complexity is bounded both above and below--it won't take longer, but it won't take significantly less time, either. The O means that the time complexity is only bounded above: The operation won't take longer, but it may (under certain best-case conditions) take less than the stated time.

BACK TO ARTICLE
Copyright © 1997, Dr. Dobb's Journal


Dr. Dobb's Journal February 1997: Scheduling Algorithms and NP-Complete Problems

Scheduling Algorithms and NP-Complete Problems

Dr. Dobb's Journal February 1997

By Oleg Kiselyov

Oleg is a computer scientist/software developer with Computer Science Corp. (CSC) in Monterey, California. http:// pobox.com/~oleg/ftp/.

The term "NP-complete" refers to a family of problems that are all roughly equivalent; if you could find a fast algorithm to solve one of these problems, the others would quickly follow. NP-complete problems are both extremely common and frustratingly difficult. Many problems of the form "find the best/cheapest/shortest way to do X" are NP-complete, and a fast solution to NP-complete problems would revolutionize such disparate fields as compiler optimization and airline scheduling. However, in lieu of such a breakthrough, the best way to deal with an NP-complete problem is usually to find a trick that will give you a "good enough" approximate solution. This month, Oleg discusses an NP-complete problem that he encountered in trying to schedule a chess tournament, and the approximate solution that made it feasible.

-- Tim Kientzle

Many years ago, when I was still a chemist, I was approached by a scientist of the "old school" who didn't like computers. To my surprise, he asked me for help in using computers to schedule a traditional New Year "blitz" chess tournament. That year, quite a few participants were expected, prompting a switch to a Swiss system. In this method of conducting a tournament, the total number of games to be played is specified in advance and is smaller than n [image] (n-1)/2, where n is the number of participants. Consequently, not everybody would end up playing with everybody. To be fair, the scheduler would have to match partners of equal skill, provided they hadn't played already. Keeping track of who played whom and (at the same time) trying to find partners of similar strength would quickly become a pain. That's why he thought a computer might be of some help.

In the actual setting, there were about 20 participants, and 10 rounds were played. In each round, partners played two five-minute games (one person plays white, another black, then they switch). So, one can score 0 (lost both games), 1/2 (lost one, tied the other), and so on up to 2 points (won both). After a round was over, the results were entered into a computer that had to come up with the pairings for the next round. The players scheduled in the next round must not have played each other in previous rounds; moreover, their scores should be as close as possible.

For example, consider the tournament in Table 1: Eight people who have already played three rounds. In the first round of the tournament, there are no constraints: No one has played a game, and everyone has the same score (zero). Consequently, any schedule will do; for example, player 2i-1 plays against player 2i, with the numbers being assigned randomly. After the round is played, a few leaders emerge. They're pitted against each other in the next round, and so on. After the third round, the leaders are #1 and #5, but they've already played each other. Aiming to match scores as closely as possible, you could pair players #1-#4, #5-#7, and #3-#8. This leaves #6 and #2, but they have already played! You can see the kind of conflicts you have to resolve. With 20 players, the situation becomes very tangled very quickly.

Handling the Problem

You can think about the scheduling problem in terms of graphs. Consider a graph where each vertex represents a potential pair. For example, vertex 1-2 represents a pairing of players #1 and #2. Initially there are n [image] (n-1)/2 vertices where n is the number of players. As the tournament progresses, you remove vertices that have already played. Two vertices are connected if the two pairs can both appear in the same round; for example, vertices 1-2 and 1-3 are not connected, since player #1 can't play player #2 and player #3 at the same time. Vertices 1-2 and 3-4 are connected. There is a cost associated with each vertex: a disparity of scores of the two players. This cost is updated after each round. Figure 1, for example, shows the graph corresponding to the situation after the third round in our sample tournament. The scheduling problem is reduced to finding a clique of a minimal cost in the graph. That is, find a set of n/2 vertices that are all connected with each other (a clique of order n/2) such that the total cost (of all vertices in the clique) is minimal. It is obvious that a clique of order n/2 means that all n participants will play in the round.

Finding a clique in a graph is, by itself, an NP-complete problem. But it pales in comparison with the requirement of finding a clique of minimal cost. A naive solution would be to consider all possible subsets of n/2 vertices of the graph, check to see if they all are connected with each other, compute the total cost, and pick up a subset with the minimal cost. The complexity of this obvious solution is far more than exponential in n. For example, in a tournament with 20 participants, you have to choose 10 pairs from 190 possible pairs. There are over 1016 possible combinations to consider, a formidable number for even the most powerful supercomputer.

Even if you want the exact optimal solution, however, you can do better than this. Assume for a moment that player #1 has stepped out, and we have to schedule a round without him. Obviously, some other player has to take the round off, too. Suppose that for each i=2..n, you have come up with optimal schedules involving all players but i and 1. In graph terms, you found optimal cliques of order (n/2)-1 that don't contain vertices mentioning 1 and i. The cost of that clique would be cost(all-{1,i}). Suppose player #1 has shown up and wants to play the round after all. Rather than rebuilding the schedule from scratch, you can use the already-found partial optimal schedules, add players #1 and i to them, and select the one with the smallest total cost. For each player i, the total cost is cost(all) = cost(all-{1,i}) + cost({1,i}), where cost({1,i}) is the disparity in scores between players i and #1. If players i and #1 have already played, put cost({1,i}) to infinity.

This technique is easy to justify: If the optimal schedule matches player #1 against player i, then the schedule without these players also has to be optimal (with respect to the remaining players). Otherwise, you can lower the cost of the original schedule, which would contradict the premise that it is optimal. To find the best partial schedules, you can repeat this algorithm recursively. This dynamical programming algorithm -- similar to the one used to solve the traveling-salesman problem (TSP) -- reduces the complexity somewhat, to the order of (n-1)2n-1. The origin of this estimate is easy to see: You need to consider all even subsets of the set of all players. Determining an optimal schedule for a subset (given optimal schedules for smaller subsets computed previously) takes no more than (n-1) operations: adding a cost of a new pair and selecting the smallest term. For n=20 players, it should take no more than 107 operations to schedule a round, immensely faster than the aforementioned naive algorithm.

However, this algorithm is fast because it caches the partial schedules. This cache is quite large, several megabytes for n=20. At the time, I only had a lowly PDP-11 to work with, so I wasn't able to use even this improved algorithm.

Fast Approximation

In practice, you do not need the perfect schedule -- a reasonably good approximation suffices. To speed things up, I used a simpler approach. I first sorted all players by their scores, then tried to pick partners that were neighbors in this sorted list. I started with the player at the top of the list, and walked down the list, picking the first available partner, providing they hadn't played already. Then I'd pick the player with the next-highest score (if still available) and find him a partner, and so on. Since the list is sorted, picking close players from this list gives pairs with similar scores.

This greedy strategy of scheduling by adding pairs of players with the closest score may not necessarily lead to the lowest overall cost. For example, after picking all but one pair, you have no choice for the final pair. It may happen that their scores are so different that the total cost is enormous. Worse, this last pair may have already played: Then the entire schedule crumbles. If this is the case, you can back off and try changing the arrangement for the next-to-last pair, hoping that the last pair would be suitable this time. Although building the complete schedule this way may not be optimal, it is close.

Note that the list of players is sorted in descending order, and I start pairing from the top. Most of the backtracking and fiddling is then done with the players near the bottom of the list. That is, even if the final solution may not be optimal as a whole, you err by mismatching losers rather than leaders. The possible inaccuracy pays off handsomely in terms of complexity: It takes only n [image] (n-1)/2 operations to go through the entire list and arrange pairs. Backtracking adds to the complexity, and this could be significant; however, in reality, the convergence is fast. The memory requirements are bounded and small.

Once the schedule is complete, I evaluate its total cost (called measure in the code, available electronically; see "Availability," page 3). Then I backtrack once again, trying to find another complete schedule. I repeat this process up to five times, and pick the schedule with the smallest cost. Although the algorithm is based on a greedy strategy, it still tries to find a global optimum. The whole idea is similar to that of approximate TSP algorithms, even though at the time I only had a very vague understanding of TSP.

Implementation

This fast algorithm was used to schedule the tournament in real time. That is, after a round was finished, the results were entered into the computer. It had to come up with a schedule for the new round while the participants were taking a break. The cumulative tournament results (history) were kept in a matrix Has_played and a vector Scores, with obvious meaning. The code had provisions to save/restore this history in a file, a precaution against the event of a power surge during the tournament (it happened a few times). Note that the players are numbered 1 through n (so index 0 has a special meaning). A schedule is represented by a vector allo: allo[i], if not zero, tells who player i is to play with; zero allo[i] means that player i is available for allocation. The first step of scheduling is sorting the list of players according to their scores in array Sorted_indices, so that Scores[Sorted_indices[i]] >= Scores[Sorted_indices[j]] for all i>j. The rest of the algorithm can be written as in Example 1.

Function try_allocate() (also available electronically) follows this idea almost literally, with the exception that it is not recursive. The first version of the code was written in DEC Basic on the PDP-11, which doesn't support recursion. Even when I later rewrote the program in C, I stuck with the faster iterative approach.

The only variables that define an arrangement of a pair are the indices i and j of the players in the sorted list. You can keep these indices for each arranged pair in special arrays, si and sj, accessed through an index level. This makes it easy to back off the arrangement to the previous level(s). The scheduling process is made of a series of actions: Pick up a suitable i, pick up j, arrange the next pair (that is, advance the "level"), or back off and try the next j; after that, try the next i. The action to execute depends on the previous action and some tests (availability of players). This is exactly how a finite automaton runs. The scheduler function in the listing implements this automaton literally.

The C compiler I initially used was pre-ANSI C. I have recently beautified the code, mainly adding void. The code runs, and even the user interface for a VT100 terminal works. The program had been used for several years and, I think, helped convince my professor that computers aren't all evil.

DDJ

Copyright © 1997, Dr. Dobb's Journal

Back to Table of Contents


Dr. Dobb's Journal February 1997: Scheduling Algorithms and NP-Complete Problems

Scheduling Algorithms and NP-Complete Problems

By Oleg Kiselyov

Dr. Dobb's Journal February 1997

/* schedule a pair 'pair_no', and then the others */
int schedule_pair(const int pair_no, const int i_start)
{
  if( pair_no >= N_players/2 ) return TRUE;
  for(i=i_start; i<=N_players; i++)
    for(j=i+1; j<=N_players; j++) {
      int ti = Sorted_indices[i], tj = Sorted_indices[j];
      if( allo[ti] == 0 && allo[tj] == 0 && Has_played[ti][tj] == 0 ) {
        /* Try pairing ti and tj */
        allo[ti] = tj; allo[tj] = ti;
        if(schedule_pair(pair_no+1,i+1))
          return TRUE; /* It worked! */
        else
          allo[ti] = allo[tj] = 0; /* It didn't work. */
      }
    }
  return FALSE; /* No valid pairing was found */
}

Example 1: Call schedule_pair(0,1) to schedule the entire tournament.

Back to Article

Copyright © 1997, Dr. Dobb's Journal


Dr. Dobb's Journal February 1997: Scheduling Algorithms and NP-Complete Problems

Scheduling Algorithms and NP-Complete Problems

By Oleg Kiselyov

Dr. Dobb's Journal February 1997

Figure 1: Scheduling choice graph corresponding to Table 1. Vertex cost is in parentheses.

Back to Article

Copyright © 1997, Dr. Dobb's Journal


Dr. Dobb's Journal February 1997: Scheduling Algorithms and NP-Complete Problems

Scheduling Algorithms and NP-Complete Problems

By Oleg Kiselyov

Dr. Dobb's Journal February 1997

Table 1: Three rounds of a sample tournament.

Back to Article

Copyright © 1997, Dr. Dobb's Journal


Dr. Dobb's Journal April 1997: Elliptic Curves and Cryptography

Elliptic Curves and Cryptography

Dr. Dobb's Journal April 1997

Strong digital signature algorithms

By Aleksandar Jurisic and Alfred J. Menezes

Aleksandar conducts cryptographic research for Certicom and can be contacted at ajurisic@certicom.ca. Alfred is a coauthor of Handbook of Applied Cryptography (CRC Press, 1997; http://www.dms.auburn.edu/hac/) and author of Elliptic Curve Public Key Cryptosystems (Kluwer Academic Publishers, 1993). He is a professor of mathematics at Auburn University and consults for Certicom. He can be reached at menezal@mail.auburn.edu.

Originally pursued for purely aesthetic reasons, elliptic curves have recently been used in algorithms for factoring integers, primality proving, and public-key cryptography. In this article, we will examine elliptic curve cryptosystems, and demonstrate why they provide relatively small block sizes, high-speed software and hardware implementations, and offer the highest strength-per-key-bit of any known public-key scheme.

Since the introduction of the concept of public-key cryptography by Whit Diffie and Martin Hellman in 1976, the cryptographic importance of the apparent intractability of the well-studied discrete logarithm problem has been recognized. Taher ElGamal first described how this problem could be utilized in public-key encryption and digital-signature schemes. ElGamal's methods have been refined and incorporated into various protocols, one of which forms the basis for the U.S. government's Digital Signature Algorithm (DSA).

We'll begin by introducing some basic mathematical terminology. A "group," for instance, is an abstract mathematical object consisting of a set G together with an operation * defined on pairs of elements of G; the "order" of the group is the number of elements in G. The operation must have certain properties, similar to those with which we are familiar from ordinary integer arithmetic. For example, the integers modulo n, namely [z] n = {0,1,2,...,n-1}, form a group under the operation of addition modulo n. If p is a prime number, then the nonzero elements of [z] p, namely [z] *p = {1,2,...,p-1}, form a group under the operation of multiplication modulo p. The order of a group element g [image] G is the least positive integer n such that gn = 1. For example, in the group [z] *11, the element g = 3 has order 5, since

31 [image] 3 (mod 11)
32 [is equivalent to] 9 (mod 11)
33 [is equivalent to] 5 (mod 11)
34 [is equivalent to] 4 (mod 11)
35 [is equivalent to] 1 (mod 11)

The discrete logarithm problem, as first employed by Diffie and Hellman in their key agreement protocol, was defined explicitly as the problem of finding logarithms in the group [image] *p: given an element g [image] [z] *p of order n, and given h [image] [z] *p, find an integer x, 0 [less than or equal to] x [less than or equal to] n-1, such that gx [is equivalent to] h (mod p), provided that such an integer exists. The integer x is called the "discrete logarithm" of h to the base g. For example, consider p =17. Then g =10 is an element of order n =16 in [z] *17. If h = 11, then the discrete logarithm of h to the base g is 13 because 1013 [is equivalent to] 11 (mod 17).

These concepts can be extended to arbitrary groups. Let G be a group of order n, and let [alpha] be an element of G. The "discrete logarithm problem" for G is: Given elements [alpha] and [beta] [image] G, find an integer x, 0 [less than or equal to] x [less than or equal to] n -1, such that [alpha] x = [beta], provided that such an integer exists.

For two primary reasons, a variety of groups have been proposed for cryptographic use. First, the operation in some groups may be easier to implement in software or hardware than the operation in other groups. Second, the discrete logarithm problem in a group may be harder than the discrete logarithm problem in [z] *p. Consequently, you could use a group G that is smaller than [z] *p while maintaining the same level of security. This results in smaller key sizes, bandwidth savings, and faster implementations. These features are especially attractive for security applications, where computational power and integrated circuit space is limited (smart cards, PC cards, wireless devices, and the like). Such is the case with elliptic curve groups, which were independently proposed for cryptographic use by Neal Koblitz and Victor Miller in 1985.

The Digital Signature Algorithm

The Digital Signature Algorithm (DSA) was proposed in August of 1991 by the U.S. National Institute of Standards and Technology (NIST) and became a U.S. Federal Information Processing Standard (FIPS 186) in 1993. It was the first digital signature scheme to be accepted as legally binding by a government. The algorithm is a variant of the ElGamal signature scheme. It exploits small subgroups in [z] *p in order to decrease the size of signatures. Figure 1 shows the key-generation, signature-generation, and signature-verification procedures for DSA.

Since r and s are each integers less than q, DSA signatures are 320 bits in length. The security of the DSA relies on two distinct (but related) discrete logarithm problems. One is the discrete logarithm problem in [z] *p where the number field sieve algorithm applies. Since p is a 1024-bit prime, the DSA is currently not vulnerable to this attack. The second discrete logarithm problem works to the base g: given p, q, g, and y, find x such that y [is equivalent to] gx (mod p). For large p (1024 bits, for example), the best-known algorithm for this problem is the Pollard rho-method, which takes about [image] steps. Since q [image] 2160, the DSA is not vulnerable to this attack.

Background in Elliptic Curves

We'll now turn to the fascinating theory of elliptic curves. For simplicity, we'll restrict our discussion to elliptic curves over [image] p, where p is a prime greater than 3. Keep in mind, though, that elliptic curves can more generally be defined over any finite field. In particular, the "characteristic two finite fields" [image] 2m are of special interest since they lead to the most efficient implementation of the elliptic curve arithmetic.

An elliptic curve E over [image] p is defined by an equation of the form

y2 = x3 + ax + b,

in Figure 2, where a,b [image] [image] p, and 4a3+27b2 [image] 0 (mod p), together with a special point [image], called the "point at infinity." The set E( [image] p) consists of all points (x,y), x [image] [image] p, y [image] [image] p, which satisfy the equation in Figure 2, together with [image].

For example, let p =23 and consider the elliptic curve E: y2=x3+x+1 defined over [image] 23. (In the notation of Figure 2, we have a =1 and b =1.) Note that 4a3+27b2 =4+4=8 [image] 0, so E is indeed an elliptic curve. The points in E( [image] 23) are [image] and those listed in Figure 3.

There is a rule for adding two points on an elliptic curve E( [image] p) to give a third elliptic curve point. Together with this addition operation, the set of points E( [image] p) forms a group with [image] serving as its identity. It is this group that is used in the construction of elliptic curve cryptosystems. The addition rule, which can be explained geometrically, is presented in Figure 4 as a sequence of algebraic formulae.

Examples of the addition rule are given in Figure 5. For historical reasons, the group operation for an elliptic curve E( [image] p) has been called "addition." By contrast, the group operation in [image] *p is "multiplication." The differences in the resulting additive notation and multiplicative notation can sometimes be confusing. Table 1 shows the correspondence between notation used for the two groups [image] *p and E( [image] p).

The Elliptic Curve Digital Signature Algorithm (ECDSA)

ECDSA is the elliptic curve analogue of the DSA. That is, instead of working in a subgroup of order q in [image] *p, you work in an elliptic curve group E( [image] p). The ECDSA is currently being standardized by the ANSI X9F1 and IEEE P1363 standards committees. Table 2 shows the correspondence between some math notation used [image] in DSA and ECDSA. Tables 1 and 2 should make the analogies between the DSA and ECDSA more apparent (see Figure 6).

The key-generation, signature-generation, and signature-verification procedures for ECDSA are given in Figure 6. The only significant difference between ECDSA and DSA is in the generation of r. The DSA does this by taking the random element (gk mod p) and reducing it modulo q, thus obtaining an integer in the interval [1,q -1]. The ECDSA generates the integer r in the interval [1,n -1] by taking the x-coordinate of the random point kP and reducing it modulo n.

To obtain a security level similar to that of the DSA (with 160-bit q and 1024-bit p), the parameter n should have about 160 bits. If this is the case, then DSA and ECDSA signatures have the same bit length (320 bits).

Instead of each entity generating its own elliptic curve, the entities may elect to use the same curve E and point P of order n. In this case, an entity's public key consists only of the point Q. This results in public keys of smaller sizes. Additionally, there are point compression techniques whereby the point Q = (xQ, yQ) can be efficiently constructed from its x-coordinate xQ and a specific bit of the y-coordinate yQ. Thus, for example, if p [image] 2160 (so elements in [image] p are 160-bit strings), then public keys can be represented as 161-bit strings.

Security Issues

The basis for the security of elliptic curve cryptosystems such as the ECDSA is the apparent intractability of this elliptic curve discrete logarithm problem (ECDLP): given an elliptic curve E defined over [image] p, a point P [image] E( [image] p) of order n, and a point Q [image] E( [image] p), determine the integer l, 0 [image] l [image] n -1, such that Q=lP, provided that such an integer exists.

Over the past 12 years, the ECDLP has received considerable attention from leading mathematicians, and no significant weaknesses have been reported. An algorithm by Pohlig and Hellman reduces the determination of l to the determination of l modulo each of the prime factors of n. Hence, to achieve the maximum possible security level, n should be prime. The best-known algorithm to date for the ECDLP in general is the Pollard rho-method, which takes about [image] steps, where a step is an elliptic curve addition. In 1993, Paul van Oorschot and Michael Wiener showed how the Pollard rho-method can be parallelized so that if r processors are used, the expected number of steps by each processor before a single discrete logarithm is obtained is [image].

In considering software attacks on ECDLP, we assume that a million instructions per second (MIPS) machine can perform 43104 elliptic curve additions per second. (This estimate is indeed conservative -- an application-specific integrated circuit for performing elliptic curve operations over the field [image] 2155 described in "An Implementation of Elliptic Curve Cryptosystems over [image] 2155," by G. Agnew, R. Mullin, and S. Vanstone [IEEE Journal on Selected Areas in Communications, 11, 1993] has a 40-MHz clock rate and can perform roughly 40,000 elliptic additions per second.) Then the number of elliptic curve additions that can be performed by a 1 MIPS machine in one year is

(4 [image] 104) [image] (60 [image] 60 [image] 24 [image] 365) [image] 240

Table 3 shows, for various values of n, the computing power required to compute a single discrete logarithm using the Pollard rho-method. A MIPS year is equivalent ot the computational power of a computer that is rated at 1 MIPS and utilized for one year.

For instance, if 10,000 computers each rated at 1000 MIPS are available, and n [image] 2150, then an elliptic curve discrete logarithm can be computed in 3800 years. Andrew Odlyzko has estimated that if 0.1 percent of the world's computing power were available for one year to work on a collaborative effort to break some challenge cipher, then the computing power available would be 108 MIPS years in 2004 and 1010 -1011 MIPS years in 2014.

To put the numbers in Table 3 in some perspective, Table 4 (see A. Odlyzko's "The Future of Integer Factorization," CryptoBytes: The Technical Newsletter of RSA Laboratories, Summer 1995; also available at http://www.rsa.com/) shows the estimated computing power required to factor integers with current versions of the general number field sieve.

A more promising attack (for well-funded attackers) on elliptic curve systems would be to build special-purpose hardware for a parallel search using the Pollard rho-method. In "Parallel Collision Search with Cryptanalytic Applications" (to appear in Journal of Cryptology), P. van Oorschot and M. Wiener provide a detailed study of such a possibility. They estimated that if n [image] 1036 [image] 2120, then a machine with m =325,000 processors that could be built for about $10 million would compute a single discrete logarithm in about 35 days.

It should be pointed out that in the software and hardware attacks just described, computation of a single elliptic curve discrete logarithm has the effect of revealing a single user's private key. The same effort must be repeated to determine another user's private key.

In "Minimal Key Lengths for Symmetric Ciphers to Provide Adequate Commercial Security" (January 1996, available from http://theory.lcs.mit.edu/~rivest/publications.html), M. Blaze, W. Diffie, R. Rivest, B. Schneier, T. Shimomura, E. Thompson, and M. Wiener reported on the minimum key lengths required for secure symmetric-key encryption schemes (such as DES and IDEA). Their report comes to the following conclusion:

To provide adequate protection against the most serious threats -- well-funded commercial enterprises or government intelligence agencies -- keys used to protect data today should be at least 75 bits long. To protect information adequately for the next 20 years in the face of expected advances in computing power, keys in newly-deployed systems should be at least 90 bits long.

Extrapolating these conclusions to the case of elliptic curves, you see that n should be at least 150 bits for short-term security and 180 bits for medium-term security. This extrapolation is justified by the following considerations:

  • Exhaustive search through a k-bit symmetric-key cipher takes about the same time as the Pollard rho algorithm applied to an elliptic curve having a 2k-bit parameter n.
  • Exhaustive searches with a symmetric-key cipher and the Pollard rho algorithm can be parallelized with a linear speedup.
  • A basic operation with elliptic curves (addition of two points) is computationally more expensive than a basic operation in a symmetric-key cipher (encryption of one block).
  • In both symmetric-key ciphers and elliptic curve systems, a "break" has the same effect: It will recover a single private key.

Implementation Issues

Since the elliptic curve discrete logarithm problem appears to be harder than the discrete logarithm problem in [image] *p (or the problem of factoring a composite integer n), you can use an elliptic curve group that is significantly smaller than [image] *p (respectively, n). For example, an elliptic curve E( [image] p) with a point P [image] E( [image] p) whose order is a 160-bit prime offers approximately the same level of security as DSA with a 1024-bit modulus p and RSA with a 1024-bit modulus n.

To get a rough idea of the computational efficiency of elliptic curve systems, compare the times to compute Figure 7(a) with 7(b).

Assume that a field multiplication in [image] p, where log2 p =l, takes l2 bit operations; then a modular multiplication in Figure 7(b) takes (1024/160)2 [image] 41 times longer than a field multiplication in Figure 7(a). Computing kP by repeated doubling and adding requires on average 80 elliptic curve additions and 160 elliptic curve doublings. From the addition formula, you see that an elliptic curve addition or doubling requires one field inversion and two field multiplications. (The cost of field addition is negligible, as is the cost of a field squaring if the field [image] 2m is used instead of [image] p.) Assume also that the time to perform a field inversion is roughly equivalent to that of three field multiplications (this is what has been reported in practice for the case of [image] 2m). Then, computing kP requires the equivalent of 1200 field multiplications, or 1200/41 [image] 29 1024-bit modular multiplications. On the other hand, computing gk mod p by repeated squaring and multiplying requires on average 240 1024-bit modular multiplications. Thus, the operation in Figure 7(a) can be expected to be about 8 times faster than the operation in Figure 7(b). It must be emphasized that such a comparison is indeed very rough, not taking into account the various enhancements that are possible for each system. Since multiplication in [image] 2m is in fact substantially faster than modular multiplication in [image] *p, even more impressive speedups can be realized in practice.

Another important consequence of using a smaller group in elliptic curve systems is that low-cost implementations are feasible in restricted computing environments, such as smart cards and wireless devices. For example, an ASIC built for performing elliptic curve operations over the field [image] 2155 has only 12,000 gates and would occupy less that 5 percent of the area typically designated for a smart-card processor (see "An implementation of elliptic curve cryptosystems over [image] 2155," by G. Agnew, R. Mullin, and S. Vanstone, IEEE Journal on Selected Areas in Communications, 11, 1993). By comparison, a chip designed to do modular multiplication of 512-bit numbers (see "An Ultra-high Speed Public Key Encryption Processor," by P. Ivey, S. Walker, J. Stern, and S. Davidson. Proceedings of IEEE Custom Integrated Circuits Conference, Boston, 1992) has about 50,000 gates, while the chip designed to do field multiplications in [image] 2593 has about 90,000 gates (see "An Implementation for a Fast Public-Key Cryptosystem," by G. Agnew, R. Mullin, I. Onyszchuk, and S. Vanstone. Journal of Cryptology, 3, 1991).

Another advantage of elliptic curve systems is that the underlying field ( [image] p or [image] 2m) and a representation for its elements can be selected so that the field arithmetic (addition, multiplication, and inversion) can be optimized. This is not the case for systems based on discrete log (respectively, integer factorization), where the prime modulus p (respectively, the composite modulus n) should not be chosen to have a special form because this might render the underlying problem easy.

Standards Activities

The primary objectives of industry standards are to promote interoperability and facilitate widespread use of well-accepted techniques. Standards for elliptic curve systems are being drafted by various accredited standards bodies around the world.

  • Elliptic curve systems are being drafted in two work items by the American National Standards Institute (ANSI) ASC X9 (Financial Services): ANSI X9.62, The Elliptic Curve Digital Signature Algorithm (ECDSA); and ANSI X9.63, Elliptic Curve Key Agreement and Transport Protocols.
  • Elliptic curves are in the draft IEEE P1363 standard (Standard for Public Key Cryptography), which includes encryption, signature, and key agreement mechanisms. Elliptic curves over [image] p and over [image] 2m are both supported. For the case of [image] 2m, polynomial bases and normal bases of [image] 2m over an arbitrary subfield [image] 2l are supported. P1363 also specifies discrete log systems (in subgroups of the multiplicative group of the integers modulo a prime) and RSA encryption and signatures. The latest drafts are available at http://stdsbbs.ieee.org/.
  • The OAKLEY Key Determination Protocol of the Internet Engineering Task Force (IETF) describes a key agreement protocol that is a variant of the Diffie-Hellman protocol. It allows for a variety of groups to be used, including elliptic curves over [image] p and [image] 2m. The document makes specific mention of elliptic curve groups over the fields [image] 2155 and [image] 2210. A draft is available at http://www.ietf .cnri.reston.va.us/.
  • The draft document ISO/IEC 14888: Digital signature with appendix -- Part 3: Certificate-based mechanisms specifies elliptic curve analogues of some ElGamal-like signature algorithms.
  • The ATM Forum Technical Committee's Phase I ATM Security Specification draft document aims to provide security mechanisms for ATM (Asynchronous Transfer Mode) networks. Security services provided include confidentiality, authentication, data integrity, and access control. A variety of systems are supported, including RSA, DSA, and elliptic curve systems.

As these drafts become officially adopted by the appropriate standards bodies, you can expect elliptic curve systems to be widely used by providers of information security.

Conclusion

Elliptic curve cryptosystems offer the highest strength-per-key-bit of any known public-key system. With a 160-bit modulus, an elliptic curve system offers the same level of cryptographic security as DSA or RSA with 1024-bit moduli. The smaller key sizes result in smaller system parameters, smaller public-key certificates, bandwidth savings, faster implementations, lower power requirements, and smaller hardware processors.

Further Reading

For an accessible introduction to all aspects of cryptography, refer to Bruce Schneier's Applied Cryptography: Protocols, Algorithms, and Source Code in C, second edition (John Wiley & Sons, 1996). Douglas Stinson's Cryptography: Theory and Practice, (CRC Press, 1995) is an excellent textbook. Handbook of Applied Cryptography (CRC Press, 1997), by A. Menezes, P. van Oorschot, and S. Vanstone, is an extensive source book on cryptography for practitioners.

Elliptic curve cryptosystems were introduced in "Elliptic Curve Cryptosystems," Mathematics of Computation, 48 (1987), by N. Koblitz, and "Uses of Elliptic Curves in Cryptography," Advances in Cryptology: CRYPTO '85, Lecture Notes in Computer Science 218 (Springer-Verlag, 1986), by V. Miller. Chapter six of N. Koblitz's A Course in Number Theory and Cryptography, second edition (Springer-Verlag, 1994), provides an introduction to elliptic curves and elliptic curve systems. Koblitz's book also covers the relevant number theory algorithms including the Pollard rho-method, and gives an overview of the number field sieve for integer factorization. The parallelization of the Pollard rho-method is described by P. van Oorschot and M. Wiener in "Parallel Collision Search with Cryptanalytic Applications," to appear in Journal of Cryptology. For a more detailed account on various implementation and security issues, see Elliptic Curve Public Key Cryptosystems (Kluwer Academic Publishers, 1993) by A. Menezes.

Finally, the Information Security Classroom at Certicom's web site (http:// www.certicom.ca) is designed to instruct people of various mathematical backgrounds, and includes some nifty Java applets that illustrate the theory of elliptic curves. If you do not have access to the Web, you can request this information by sending e-mail to info@certicom.ca.

DDJ

Copyright © 1997, Dr. Dobb's Journal

Back to Table of Contents


Dr. Dobb's Journal April 1997: Elliptic Curves and Cryptography

Elliptic Curves and Cryptography

By Aleksandar Jurisic and Alfred J. Menezes

Dr. Dobb's Journal April 1997

Figure 1: (a) DSA key generation; (b) DSA signature generation; (c) DSA signature verification.

Back to Article

Copyright © 1997, Dr. Dobb's Journal


Dr. Dobb's Journal April 1997: Elliptic Curves and Cryptography

Elliptic Curves and Cryptography

By Aleksandar Jurisic and Alfred J. Menezes

Dr. Dobb's Journal April 1997

Figure 2: Defining an elliptic curve over p.

Back to Article

Copyright © 1997, Dr. Dobb's Journal


Dr. Dobb's Journal April 1997: Elliptic Curves and Cryptography

Elliptic Curves and Cryptography

By Aleksandar Jurisic and Alfred J. Menezes

Dr. Dobb's Journal April 1997

Figure 3: Points on the elliptic curve y2=x3+x+1 over 23.

Back to Article

Copyright © 1997, Dr. Dobb's Journal


Dr. Dobb's Journal April 1997: Elliptic Curves and Cryptography

Elliptic Curves and Cryptography

By Aleksandar Jurisic and Alfred J. Menezes

Dr. Dobb's Journal April 1997

Figure 4: The addition rule.

Back to Article

Copyright © 1997, Dr. Dobb's Journal


Dr. Dobb's Journal April 1997: Elliptic Curves and Cryptography

Elliptic Curves and Cryptography

By Aleksandar Jurisic and Alfred J. Menezes

Dr. Dobb's Journal April 1997

Figure 5: Examples of elliptic curve addition on the curve y2=x3+x+1 over 23.

Back to Article

Copyright © 1997, Dr. Dobb's Journal


Dr. Dobb's Journal April 1997: Elliptic Curves and Cryptography

Elliptic Curves and Cryptography

By Aleksandar Jurisic and Alfred J. Menezes

Dr. Dobb's Journal April 1997

Figure 6: (a) ECDSA key generation; (b) ECDSA signature generation; (c) ECDSA signature verification.

Back to Article

Copyright © 1997, Dr. Dobb's Journal


Dr. Dobb's Journal April 1997: Elliptic Curves and Cryptography

Elliptic Curves and Cryptography

By Aleksandar Jurisic and Alfred J. Menezes

Dr. Dobb's Journal April 1997

Figure 7: Comparing compute times.

Back to Article

Copyright © 1997, Dr. Dobb's Journal


Dr. Dobb's Journal April 1997: Elliptic Curves and Cryptography

Elliptic Curves and Cryptography

By Aleksandar Jurisic and Alfred J. Menezes

Dr. Dobb's Journal April 1997

Table 1: Correspondence between p*and E( p)notation.

Back to Article

Copyright © 1997, Dr. Dobb's Journal


Dr. Dobb's Journal April 1997: Elliptic Curves and Cryptography

Elliptic Curves and Cryptography

By Aleksandar Jurisic and Alfred J. Menezes

Dr. Dobb's Journal April 1997

Table 2: Correspondence between DSA and ECDSA notation.

Back to Article

Copyright © 1997, Dr. Dobb's Journal


Dr. Dobb's Journal April 1997: Elliptic Curves and Cryptography

Elliptic Curves and Cryptography

By Aleksandar Jurisic and Alfred J. Menezes

Dr. Dobb's Journal April 1997

Table 3: Computing power to compute elliptic curve logarithms with the Pollard rho-method.

Back to Article

Copyright © 1997, Dr. Dobb's Journal


Dr. Dobb's Journal April 1997: Elliptic Curves and Cryptography

Elliptic Curves and Cryptography

By Aleksandar Jurisic and Alfred J. Menezes

Dr. Dobb's Journal April 1997

Table 4: Computing power required to factor integers using the general number field sieve.

Back to Article

Copyright © 1997, Dr. Dobb's Journal


Dr. Dobb's Journal April 1997: Algorithm Alley

Understanding CRCs

Dr. Dobb's Journal April 1997

By Tim Kientzle

Tim is an editor for DDJ and author of the forthcoming Developer's Guide to Sound Programming (Addison-Wesley, 1997). He can be reached at kientzle@ddj.com.

From XModem to TCP, most communications protocols use some form of error-detecting signature. The sender computes a special value and "signs" the packet of data with that value. When the receiver sees the data, it computes its own signature and compares it with the one received from the sender. An error in transferring the data will hopefully cause these two values to disagree. The two most popular error-detecting signatures are checksums and cyclic redundancy checks (CRCs).

Although simple, checksums lack the robust mathematical properties of CRCs. Checksums are the easiest signatures to understand and may well be the most widely used type of signature. A checksum simply adds up the values of each byte of data to obtain a signature value. Unfortunately, checksums aren't particularly effective.

To detect a large number of errors, the signature values must be uniformly distributed -- on random data any signature value should be about as likely as any other. Unfortunately, checksums do not have this property, due to a statistical fact known as the "Law of Large Numbers." (This is the same principle that says seven is more likely than two when you roll a pair of dice.) To get well-distributed signatures, you need to look elsewhere.

What You Need to Know About CRCs

There are many different CRCs, all computed in essentially the same fashion. The most important aspect of a CRC is its length. Longer CRCs are more effective than short ones at detecting errors. A 12-bit CRC will miss only one error out of 212, while a 32-bit CRC will miss only one error in 232.

CRCs are characterized by a string of bits called the "generating polynomial." Each different generating polynomial produces a different CRC. There are also a number of minor differences in how CRCs are computed. If you want to compute a compatible CRC, you need to understand these differences.

CRCs can be computed nearly as quickly as checksums, using table-driven algorithms (such as those presented here). These implementations are both compact and fast. CRCs are also easy to compute in hardware.

Because CRCs are easily implemented, they are widely used. They are used to detect errors in floppy- and hard-disk interfaces, network software and hardware, program loaders, backup software, revision-control systems, and the like.

Implementing CRCs

The generating polynomial is written either as a sum of powers of x or as a string of bits. Table 1 lists three popular generating polynomials. It's not important at this point that you know what a generating polynomial is, just that different generating polynomials give different CRCs.

The relationship between the binary and polynomial forms is just that the powers of x indicate which bits in the binary form are set, with the convention that 1 in the polynomial form is thought of as x0. So, for the generating polynomial x16+x12+x5+1, bits 16, 12, 5, and 0 are set.

Computing CRCs Slowly

CRCs are calculated on a stream of bits. I'll start with a simple calculator that computes the 16-bit CRC recommended by the CCITT, which creates international communications standards. The calculator in Listing One isn't efficient, but it will help explain those that follow.

CRC calculations consist of alternating two operations: collect the next bit(s), and reduce the value so far. In Listing One, the function is called once for each bit of data. The shift and exclusive-or puts this bit into the longcrc variable, and the if statement handles the reduction. Listing Two shows a modification (checking the top bit before the shift rather than after) that eliminates the need for a long temporary variable.

Table-Driven CRC Calculation

If you look at this operation over the eight bits in a byte, the eight reductions don't depend on the bits being shifted into the low-order end -- they depend only on what was already in the high-order byte. Thus, you can rearrange the calculation to collect eight bits, then do eight reductions (or vice versa). By precomputing the reductions in a table, you end up with the very efficient algorithm shown in Listing Three. InitCrc16 precomputes the reductions for all 256 possible byte values, using exactly the algorithm in the previous section.

Using the precomputed values in the table, Crc16() can rapidly update the CRC value. The table itself is small enough to comfortably fit in the L1 cache on most processors. To make Crc16() even faster, it could be modified to accept a pointer and a count, removing the need for a function call for each byte of data.

Real CRC Calculators

If you compare the previous code with the CCITT-CRC16 computations for YModem and ZModem, you'll notice that the table is identical, but the CRC calculation itself (Listing Four) is slightly rearranged. There's an interesting mathematical reason for this, but here's a relevant nonmathematical observation: In the first version, if you initialize the crc value to zero, the first two bytes simply get shifted directly into the CRC (the zero entry in the table is always zero). That is, the CRC16 of a 2-byte buffer is just those two bytes. Listing Four doesn't have this property. You can get the exact same result from Listing Three by adding two zero bytes to the end of your data and including those in the CRC calculation.

The CRC16 in Listing Five is another variation. Kermit computes its CRC backwards, shifting the CRC to the right rather than left. This reversed direction means you test the least-significant bit rather than the most-significant bit, and the CRC constant is reversed from 0×1021 to 0×8408.

32-Bit CRCs

Other CRCs can be computed similarly. Listing Six shows how to compute a common 32-bit CRC. Notice that the left shift shifts by 8, because it's computing the CRC one byte at a time. The right shift aligns the high-order byte, and therefore is a shift by 24. With minor modifications, this code can be used for any CRC of at least eight bits. Note that ZModem computes its 32-bit CRC backwards from the Crc32 function here, just as the Kermit CRC16 is backwards from the X/Y/ZModem CRC16.

CRC Variations

Common CRC calculators show a number of other minor variations, none of which affect the mathematical effectiveness of the CRC:

  • Multibyte CRCs can be sent in any byte order.
  • The CRC may be initialized to all 0s or all 1s.
  • The final CRC is often complemented before being used.

Understanding CRCs

CRCs are based on a technique for dividing polynomials that's routinely discussed in high-school algebra classes. The key point is to think of the data being transferred as a single long list of bits. By writing this list as a polynomial, a chunk of data becomes a single mathematical entity that can be manipulated as a whole. Put differently, polynomials provide a way of computing that treats any large block of binary data as a single mathematical object.

Long Division of Polynomials

Grade-school math classes teach a technique for division referred to as "long division" which looks at the dividend (the part under the division symbol) a few digits at a time. High-school math classes teach this same method for dividing polynomials. As you may recall, division of polynomials is even simpler than division of integers, because only the first term matters. For example, the first step in dividing 2x4-2x3-12x2+12x-6 by x2-3x+2 is to divide 2x4 by x2, yielding 2x2. The rest of the process (see Figure 1) is the same as dividing integers. At each step, you divide the leading terms to obtain a new term in the quotient, then multiply and subtract.

If you look back at the original bit-at-a-time CRC calculator, you'll see these same steps: Bring another term (bit) into consideration (shift), divide the leading terms (test the high-order bit), multiply (by 0 or 1), and subtract (exclusive OR). Repeating this procedure for each term (bit) eventually leaves a remainder (CRC).

Because you're dividing by a second-degree polynomial in Figure 1, you always consider three terms at a time. After subtracting, you're left with two terms, so you "shift" another term into consideration, just as the CRC algorithm shifts the next bit into the CRC variable. Also, the CRC algorithm only tests a single bit at each loop, much as the division algorithm divides only the leading terms.

The reduction is the part that doesn't make sense at first glance. The subtraction in the normal division algorithm somehow has been mutated into an exclusive-or operation. Clearly, these aren't ordinary polynomials...

Polynomials Modulo 2

The number system of integers modulo 2 has only two numbers -- 0 or 1. This number system allows all four of the basic arithmetic operations according to Figure 2. Just as in normal arithmetic, division by zero is undefined.

Using integers modulo 2 to create polynomials gives you some peculiar properties. For example, since 1+1=0 in modulo 2, you know that (x+1)2= x2+x+x+1=x2+1, which is very different from what you get using the standard integers.

Messages as Polynomials

Now consider the word "dog" in ASCII. As a string of bits, it is 0110 0100 0110 1111 0110 0111. By considering these bits as the coefficients, you have a 22nd degree message polynomial: x22+x21+x18+x14+x13+x11+x10+x9+x8+x6+x5+x2+x+1.

In a similar way, even a multimegabyte data file can be viewed as a single large polynomial. Fortunately, because the division algorithm only requires you to look at a few bits at a time, you can compute with these very large polynomials quite easily.

Be forewarned that many people are tempted to equate these polynomials with binary numbers by "substituting 2." That doesn't work. A moment ago you saw that (x+1)2=x2+1 in modulo 2. If you "substitute 2," this would mean that (2+1)2=32=9 is the same as 22+1=4+1=5. It's more appropriate to think of a term such as x22 as meaning that there is a one bit in position 22. The polynomial is then just a list of which bit positions hold ones.

CRCs Defined

Now, I'm ready to define a cyclic redundancy check: A CRC is the remainder obtained by dividing the message polynomial by the generating polynomial. The key to translating this into the code I presented earlier is to realize that adding and subtracting polynomials in modulo 2 is really just the exclusive-or operation, as reflected in the addition and subtraction tables in Figure 2. The initial bit-at-a-time CRC calculator is just the direct realization of the long division algorithm for polynomials modulo 2.

From Mathematics into C

Because CRCs use polynomials modulo 2, there are only two possibilities at each step of the long division algorithm -- either it divides or it doesn't. For example, a 16th degree polynomial divides any other 16th degree polynomial. That's the key to the bit-at-a-time CRC calculator. Checking the high-order bit is simply checking to see if a polynomial is 16th degree. If it is, the exclusive-or does the subtraction.

In symbols, if the message polynomial so far is m, and the generating polynomial is g, then m/g can be written in terms of a quotient q and a remainder r: m/g=q+r/g.

Assume that's already been done. In particular, you know r, which is the CRC of the message before this bit. If the next bit is b, the new message is mx+b. (Multiplying m by x raises each power by one; that is, it performs a left shift.) The previous remainder then tells you something about the new remainder: (mx+b)/g=mx/g+b/g=qx+(rx+b)/g.

This result tells us that the remainder from (mx+b)/g is the same as the remainder from (rx+b)/g. To find out the new remainder (new CRC), just shift the new bit b into the old remainder, and then take the remainder from dividing that by g. Because the old remainder was at most 15th degree, the new remainder cannot be more than 16th degree, so the division is just one bit test and a possible exclusive-or. If you look back at the original bit-at-a-time CRC calculator, you'll see that's exactly what it's doing. The nice part about only needing the old remainder to compute the new one is that it's not necessary to look at the entire file to compute the CRC; you only need to keep track of the CRC thus far as you scan through the bits in the file.

Instead of looking at one bit b at a time, you could consider the next eight bits at a time, which reveals the table-driven algorithm. Just let B=b0x7+b1x6+... +b6x+b7 (B is the next byte of the message). Then, as before, the old remainder r becomes the basis for computing the new remainder: (mx8+B)/g=qx8+(rx8+B)/g. Since B has a smaller degree than g, this simplifies to rx8/g+B/g. The table is just the precomputed remainder from rx8/g.

Why CRCs Work

To understand why CRCs work, start by examining what happens when there's an error -- when the data you receive is different than what was sent. If you subtract the two different message polynomials (the one that was sent and the one you received), you have a polynomial with a one bit at each error location.

This "error polynomial" is the key. If the CRC doesn't detect the error, then the error polynomial is divisible by the generating polynomial. If the generating polynomial isn't divisible by x (the least-significant bit is a one), then the first and last wrong bits must be at least as far apart as the degree of the generating polynomial. This is the often-quoted fact that a CRC will detect any burst error shorter than the generating polynomial.

By selecting the factors of the generating polynomial, you can bias the CRC to detect certain errors. For example, every multiple of x+1 has even parity, so the CRC-16 (which has a generating polynomial divisible by x+1) will catch every error which affects an odd number of bits. (The error polynomial will have odd parity, and hence can't be a multiple of the generating polynomial.) Conversely, if an error affects an even number of bits, the CRC-16 is that much less likely to detect it, so most generating polynomials cannot be factored.

Conclusion

I originally wrote this description of the CRC after reading many articles that claimed CRCs relied on "properties of prime numbers." Such near-misses are all the more frustrating because the complete story is only slightly more complex, and it gives practical insight into how to compute a wide variety of different CRCs.

For more information on CRCs, see Andy Lowry's algorithm for computing the 16-bit Kermit CRC in the Kermit protocol reference, available via anonymous FTP from kermit.columbia.edu. Likewise, Terry Ritter provided a good description of CRCs in "The Great CRC Mystery" (DDJ, February 1986).

DDJ

Listing One

short BitwiseCrc16(int bit, short crc) { long longcrc = crc;
 longcrc = (longcrc << 1) ^ bit;  /* next bit */
 if (longcrc & 0x10000)
   longcrc ^= 0x11021; /* reduce */
 return(longcrc & 0xFFFF);
}

Back to Article

Listing Two

short BitwiseCrc16(int bit, short crc) { if (crc & 0x8000)
   crc = (crc << 1) ^ bit ^ 0x1021 ;
 else crc = (crc << 1) ^ bit;
 return (crc & 0xffff);
}

Back to Article

Listing Three

static short Crc16Table[256];

void InitCrc16() {
  short i, j, crc;
  for(i=0; i < 256; i++) {
    crc = (i << 8); /* Put i into MSB */
    for(j=0; j < 8; j++) /* Do 8 reductions */
      crc = (crc << 1) ^((crc & 0x8000)? 0x1021:0);
    Crc16Table[i] = crc & 0xFFFF;
  }
}
short Crc16(int ch, short crc) {
    crc = Crc16Table[((crc >> 8) & 255)] ^ (crc << 8) ^ ch;
    return crc & 0xFFFF;
}

Back to Article

Listing Four

/* InitCrc16() is same as Listing Three */short XYZModemCrc16(int ch, short crc) {
  crc = Crc16Table[((crc >> 8) ^ ch) & 255] ^ (crc << 8);
  return crc & 0xFFFF;
}

Back to Article

Listing Five

static short KermitCrc16Table[256];void KermitInitCrc16() {
  int i, j, crc;
  for(i=0; i <256; i++) {
    crc = i;  /* Start with i in low-order byte */
    for(j=0; j < 8; j++)
      crc = (crc >> 1) ^ ((crc & 1) ? 0x8408 : 0);
    KermitCrc16Table[i] = crc & 0xFFFF;
  }
}
short KermitCrc16(int ch, short crc) {
  crc = KermitCrc16Table[(crc ^ ch) & 255)] ^ (crc >> 8);
  return crc & 0xFFFF;
}

Back to Article

Listing Six

static long Crc32Table[256];InitCrc32() {
  int i, j;  long crc;
  for(i=0; i <256; i++) {
    crc = (i << 24); /* Put i into MSB */
    for(j=0; j < 8; j++) /* 8 reductions */
      crc = (crc << 1) ^ ((crc & 0x80000000L) ? 0x04c11db7L : 0);
    Crc32Table[i] = crc;
  }
}
long Crc32(int ch, long crc) {
  crc = Crc32Table[((crc >> 24) ^ ch) & 255] ^ (crc << 8);
  return crc & 0xFFFFFFFF;
}

Back to Article

Copyright © 1997, Dr. Dobb's Journal

Back to Table of Contents


Dr. Dobb's Journal April 1997: Understanding CRCs

Understanding CRCs

By Tim Kientzle

Dr. Dobb's Journal April 1997

Figure 1: A complete division example.

Back to Article

Copyright © 1997, Dr. Dobb's Journal


Dr. Dobb's Journal April 1997: Understanding CRCs

Understanding CRCs

By Tim Kientzle

Dr. Dobb's Journal April 1997

Figure 2: Arithmetic modulo 2.

Back to Article

Copyright © 1997, Dr. Dobb's Journal


Dr. Dobb's Journal April 1997: Understanding CRCs

Understanding CRCs

By Tim Kientzle

Dr. Dobb's Journal April 1997

Table 1: Popular generating polynomials.

Back to Article

Copyright © 1997, Dr. Dobb's Journal


Dr. Dobb's Journal August 1997: Algorithm Alley

Topological Sorting

Dr. Dobb's Journal August 1997

By Jonathan Pincus and Jerry Schwarz

Jonathan is the chief technical officer and one of the founders of Intrinsa Corp. Jerry is a member of the technical staff at Intrinsa and a veteran of the ANSI C++ Standards Committee. They can be reached at jon@intrinsa.com and jerry@intrinsa.com, respectively.

An old adage claims that "Anyone can make a bridge that stands, but it takes an engineer to make one that just barely stands." This is as true of software as bridges. It takes considerable skill to design efficient software that only does as much work as necessary. For example, traditional sorting may be overkill if you really only have a few conditions you need to satisfy. This month, Jonathan and Jerry show how topological sorting does only enough work to satisfy a few constraints. They then extend the basic algorithm to provide good results even when those constraints are contradictory.

-- Tim Kientzle

In developing our PREfix software component simulator, we realized that to get the best results, we should analyze functions in a particular order. For example, if Foo() calls Bar(), it's best to analyze Bar() before analyzing Foo(). Given an entire program, we needed to sort the various functions so that a called function was analyzed before the function that called it. Note that there are many different orders that might work; we just needed to find one.

This is an example of a "topological sort." Most sorting techniques require a "total order," where you have to be able to compare any two elements. A topological sort uses a "partial order" -- you may know that A precedes both B and C, but not know (or care) whether B precedes C or C precedes B. Topological sorting is a useful technique in many different domains, including software tools, dependency analysis, constraint analysis, and CAD.

Topological sorting works well in certain situations. If there are very few relations (the partial order is "sparse"), then a topological sort is likely to be faster than a standard sort. Given n objects and m relations, a topological sort's complexity is O(n+m) rather than the O(n log n) of a standard sort. In our case, most functions typically call a handful of other functions, meaning the total number of relations (caller/callee pairs) is relatively small, so topological sorting makes sense.

Topological sorts can also deal gracefully with cycles. For example, imagine that two functions X and Y are mutually recursive: X calls Y and Y calls X. In this case, it is useful to detect the cycle and the specific relations that cause the cycle. Standard sorting algorithms, however, will simply fail in this situation. We developed an extension to topological sorting that can produce a "best" order, even in the presence of cycles.

In this article, we present a basic topological sorting algorithm and implementation, then extend the algorithm and implementation to deal with cycles. The implementations presented here make heavy use of, the C++ Standard Template Library (STL). Much of the work involves manipulating vectors and queues, and STL's primitives make it easier. Furthermore, packaging the various functions as STL algorithms allows them to be reused on different types of data.

The Basic Algorithm

For simplicity, we'll describe the algorithm in terms of a generic "precedes" relation: If X precedes Y, we say that X is the predecessor and Y is the successor. Our terminology and discussion follows Donald Knuth's presentation in The Art of Computer Programming, Volume 1 (Addison-Wesley, 1997).

For example, Figure 1 illustrates the following set of relations on the input set { A, B, C, D }: A precedes B, C precedes D, and B precedes D. For these relations, there are several valid orderings:

{ A, B, C, D }

{ A, C, B, D }

{ C, A, B, D }

However, { C, D, A, B } would not be valid, because it violates the relation that B precedes D.

One way of ensuring a correct order is to simply copy the data, making sure that whenever we copy one item, we know that all of its predecessors have already been copied. As a shorthand for this notion of "already been copied," we'll define a predecessor X as an "active" predecessor of Y if X precedes Y and X has not (yet) been copied. Then this basic algorithm can be summed up quite succinctly:

While there is a member that has no active predecessors, repeat:

1. Pick a member X with no active predecessors.

2. Copy X to the output.

Clearly, the output will indeed be ordered. What is not so obvious is that -- assuming there are no cycles in the input -- the while loop will not terminate until every item has been output.

The key to an efficient implementation of this algorithm is to be able to pick members with no active predecessors without having to rescan the entire list at each iteration. One way of accomplishing this is to associate a count of predecessors and a list of successors to each input item, and use an auxiliary queue Q as interim storage of nodes that have no active predecessors but have not yet been output. In fact, Knuth uses the topological sorting algorithm as an example of using a linked list as a queue.

Assume we have an array, count, which stores the predecessor count for each member. Using the basic queue operations add (at the end of the queue) and pop (from the front of the queue), we can refine the algorithm as in Figure 2.

The while loop terminates when there are no items left on the queue, either because everything in the input list has been processed, or because there is a cycle.

Complexity

The overall time complexity of this basic algorithm is O(n+m). The O(n) comes from the number of times that the while loop (and initial for loop) is executed, and the O(m) from the nested for loop.

Although there is no way to calculate how many times the inner loop will be executed on any one iteration of the outer loop, it will only be executed once for each successor of each member, which means that the total number of times that it will be executed is the total number of successors of all the members -- or the total number of relations.

Space complexity is also O(n+m). The O(n) component comes from the predecessor count information stored for each member, and the maximum length of the auxiliary queue. The O(m) comes from storing the successors for each member; once again, the total number of successors is the number of relations, so O(m).

Cycles

The basic algorithm's usefulness may be limited because of its inability to give detailed information about cycles. In practice, there are often one or more cycles in real-world problems. In program analysis, mutually recursive functions cause cycles.

For example, Figure 3 illustrates the set of relations on the set { A, B, C, D, E, F, G }: A precedes B, B precedes C, C precedes D, C precedes B, D precedes B, D precedes E, D precedes F, F precedes G, and G precedes F. Note that there is a cycle involving B, C, and D; and another cycle involving F and G. There is also a cycle involving just B and C, but we can ignore that because of the B, C, D cycle; we are only interested in the maximal cycles.

In The Design and Analysis of Computer Algorithms (Addison Wesley, 1974), Aho, Hopcroft, and Ullman present a complete -- and efficient -- algorithm for identifying strongly connected components in a graph, which we can use for finding cycles.

The basic approach is to iterate through all the members of the set and perform a depth-first search of the relations from each. As we visit each member for the first time, we number it (Aho et al. refer to this as the "depth-first number") and push it onto a stack.

For each member, we also keep a record of the lowest-numbered potential root of a cycle involving that member. Initially, this is simply the member itself; when we are done with the depth-first search from a member, if any of its successors has a lower-numbered potential root, we update the member's potential root.

A member is potentially the base of a cycle if its lowest-numbered potential root is itself. Because the search is depth-first, any cycle will end up at the end of the stack, so we can find the cycle by popping the stack repeatedly until we get to the base node. (Unlike the Aho et al. algorithm, we ignore members that do not have any members following them on the stack; these members are not in any cycle.) Since each member is popped off the stack only once, it is in at most one cycle. The file topsort.h (available electronically; see "Availability," page 3) implements this algorithm.

Dealing with Cycles

In a dependency-analysis tool, it might be enough just to identify the cycles so that you know which dependencies overconstrain the problem. In other cases, however, what you want is a "best" ordering -- an ordering that preserves as much of the information as possible. In our case, we want to warn the user of the approximation caused by mutually recursive functions, but still proceed with bottom-up analysis.

In Figure 3, there is no definitive ordering between B, C, and D, or between F and G. However, the first cycle precedes the second cycle because D precedes F (but there is no member of the second cycle preceding any member of the first cycle). Similarly, A precedes the first cycle, and the first cycle precedes E.

Possible "best" orderings for Figure 3 include { A, B, C, D, E, F, G } and { A, D, C, B, F, G, E }, but not { A, E, B, C, D, F, G }.

You can extend the topological sorting algorithm to deal with cycles by first finding the cycles of the set, then creating a set where all members of a cycle are replaced by a single placeholder. Next, topologically sort this smaller set. Finally, replace each placeholder with all the members of the corresponding cycle. Figure 4 shows this extended topological sorting algorithm.

The key point is that the topological sort in step 5 will not encounter any cycles, since we have replaced them all in step 2.

Complexity

The Aho, Hopcroft, and Ullman cycle-finding algorithm is O(n+m). Since the worst-case number of cycles is O(n), and the number of relations between cycles is m (or fewer, if relations between the same cycle can be weeded out efficiently), the extended algorithm is also O(n+m).

Oddly, the worst case for the extended algorithm is the situation where there are no cycles, and it performs a topological sort on all n members. If this is the typical case for some data, it may be more efficient to first perform a topological sort with the basic algorithm, and only fall back on the advanced algorithm if cycles actually appear.

In some real-world problems, cycles of length 2 may be much more common than longer cycles. Mutual recursion is a good example of this; pairs of mutually recursive functions are not infrequent, but larger combinations are rarer (although they certainly occur in practice). In this case, you might optimize the basic topological sorting algorithm by making an O(m) initial pass through the members. Simply look for mutually recursive pairs; when both members in a mutually recursive pair's predecessor counts are reduced to 1, that pair is eligible to be put on the output queue. While this computation remains O(m), the constant factor may be high enough that it is not worth doing.

Implementation

We've tested the implementation on Windows 95/NT with Microsoft Visual C++ 4.2, and on Solaris with the Sparcworks compiler and ObjectSpace's STL library.

The file topsort.h (available electronically; see "Availability," page 3) is the complete implementation of the topsort, cycles, and topsortWithCycles algorithms presented here. The file driver.cpp (also available electronically) illustrates different ways of calling the functions. Example 1 is driver.cpp's output for the example illustrated in Figure 3.

Packaging these algorithms completely in a header file is not ideal, since they define auxiliary types, which include some noninline functions; however, it makes the clearest explanation.

The topsort algorithm produces a linear order for its input obeying the partial order specified by the relations. It returns true if there is a cycle, and false otherwise (but does not give any information about the cycle). The cycles algorithm produces a list of cycles in the input. The topsortWithCycles algorithm extends topsort to return an order whether or not cycles are present.

The functions are templatized to allow them to operate on arbitrary types. Example 2 shows the declaration of topsort.

For efficiency, the underlying GraphInfo class operates on integer indices, rather than the iterators that are used in the external interface. The functions topsort, cycles, and topsortWithCycles first convert the relations to relations between integer indices, and use GraphInfo to operate on the integers, then use the result to order their output.

While this may appear unnecessary, it is actually important to ensure the time complexity of algorithm. The aforementioned complexity analysis assumed that the time to index into the count array and get the list of successors of a member is O(1), and that the time to assign a result is also O(1). If strings were being used as indices, for example, array lookups would be O(log n) complexity, rather than O(1), and so the overall complexity would be O((n+m) · log n). For integers, both assignments and lookups are indeed O(1).

One additional implementation note: topsort checks for relations between an integer and itself, and simply ignores them. Strictly speaking, such a relation is a cycle of length 1, but such a cycle does not affect the linear ordering. It is easy and efficient to deal with this as a special case.

topsortWithCycles illustrates the combination of cycle-finding and topological sorting to produce an approximate topological sort in the presence of cycles. For simplicity, this implementation always searches for cycles; if cycles are relatively rare, it may be more efficient to first perform a topological sort, and then find and expand cycles from the remaining relations only if there is at least one cycle.

DDJ

Copyright © 1997, Dr. Dobb's Journal

Back to Table of Contents


Dr. Dobb's Journal August 1997: Topological Sorting

Topological Sorting

By Jonathan Pincus and Jerry Schwarz

Dr. Dobb's Journal August 1997

Example 1: Output of the test program.

Back to Article

Copyright © 1997, Dr. Dobb's Journal


Dr. Dobb's Journal August 1997: Topological Sorting

Topological Sorting

By Jonathan Pincus and Jerry Schwarz

Dr. Dobb's Journal August 1997

Example 2: Declaration of the topsort template function.

Back to Article

Copyright © 1997, Dr. Dobb's Journal


Dr. Dobb's Journal August 1997: Topological Sorting

Topological Sorting

By Jonathan Pincus and Jerry Schwarz

Dr. Dobb's Journal August 1997

Figure 1: One possible order is {C, A, B, D}, but there are many others.

Back to Article

Copyright © 1997, Dr. Dobb's Journal


Dr. Dobb's Journal August 1997: Topological Sorting

Topological Sorting

By Jonathan Pincus and Jerry Schwarz

Dr. Dobb's Journal August 1997

Figure 2: The refined algorithm.

Back to Article

Copyright © 1997, Dr. Dobb's Journal


Dr. Dobb's Journal August 1997: Topological Sorting

Topological Sorting

By Jonathan Pincus and Jerry Schwarz

Dr. Dobb's Journal August 1997

Figure 3: The cycles in this diagram require special consideration.

Back to Article

Copyright © 1997, Dr. Dobb's Journal


Dr. Dobb's Journal August 1997: Topological Sorting

Topological Sorting

By Jonathan Pincus and Jerry Schwarz

Dr. Dobb's Journal August 1997

Figure 4: The extended topological sorting algorithm.

Back to Article

Copyright © 1997, Dr. Dobb's Journal


Dr. Dobb's Journal October 1997: The Block Cipher Square Algorithm

The Block Cipher Square Algorithm

Dr. Dobb's Journal October 1997

A fast block cipher that uses a 128-bit key

By Joan Daemen, Lars R. Knudsen, and Vincent Rijmen

Joan is an employee of Banksys, Brussels. He can be contacted at Daemen.J@banksys.be. Lars and Vincent are with the COSIC research group of the Department of Electrical Engineering of the Katholieke Universiteit Leuven, Belgium. They can be contacted at lars.knudsen@esat.kuleuven.ac.be and vincent.rijmen@esat.kuleuven.ac.be, respectively.
Sidebar: The Galois Field GF(28)

Square is a new fast block cipher that encrypts data in blocks of 128 bits, using a 128-bit key. Most cipher designs are optimized for one specific implementation platform. Square's structure (that is, its building blocks and their interaction) has been carefully chosen to allow very efficient implementations on a wide range of processors. This versatility is achieved through the use of operations in the finite field GF(28), which is a new approach in block cipher design. A reference implementation was written in C by Paulo Barreto (pbarreto@uninet.com.br) and runs at 2.63 MB on a 100 Mhz Pentium (available electronically; see "Availability," page 3). By optimizing in assembler, this speed was doubled.

Structure

Square is an iterated, but not a Feistel, cipher. Instead, every round acts upon all 16 bytes of its the input. One round of Square consists of four different operations. These operations can be efficiently combined in a single set of table lookups and XOR operations. The round operations are easiest to visualize if you think of the 16 bytes as being arranged in a 4×4 array; see Figure 1. An intermediate value a can then be represented by the 16 bytes ai,j,0<=i,j< 4. The four operations are:

  • A linear transformation that operates separately on the four rows of four bytes each. [theta] is determined by four coefficients cj,0<=j<4. b= [theta] (a) corresponds to
    In this formula, additions and multiplications are performed in the Galois field GF(28); see the accompanying text box entitled "The Galois Field GF(28)."
  • A nonlinear transformation [gamma] that uses one invertible 8-bit S-box. Each of the 16 bytes is transformed independently by the S-box.
  • A byte permutation [pi] that rearranges the bytes by interchanging rows and columns (that is, it transposes the array).
  • Key addition. Exclusive-Or with the round key kt is denoted [sigma] [kt] (Exclusive-or corresponds to addition in the field GF(28)).
  • The round operation is denoted [rho] [kt] and defined as
    Square is defined as eight rounds preceded by an extra key addition and by q-1: Square[k]=

Cryptanalysis

Square was designed following the Wide Trail Design Strategy (see "Cipher and Hash Function Design Strategies based on Linear and Differential Cryptanalysis," by Joan Daemen, doctoral dissertation, March 1995, K.U. Leuven), which separates the functionality of the block cipher round operation in separate layers -- the nonlinear substitution, linear diffusion layer, and key addition. Strict criteria are imposed on each layer to build ciphers that resist linear and differential cryptanalysis.

For a description of the basic principles of linear and differential cryptanalysis, refer to "Differential and Linear Cryptanalysis," by Bruce Schneier (DDJ, January 1996). In short, differential cryptanalysis works by applying small changes to the input of the algorithm and predicting the resulting differences at various intermediate stages of the algorithm. Resistance against differential cryptanalysis can be improved by ensuring that small differences at the input cause large differences at the output (and intermediate) values. In Square, this diffusion of differences is ensured by the two linear transformations: [theta] and [pi].

q takes care of the intrarow diffusion. If one byte of a row at the input is different, all four bytes of this row in the output are different. For example, if n bytes of a row at the input are different (1<=n<=4), at least 5-n output bytes of this row are different.

[pi] produces interrow diffusion. After [pi], the four bytes of a row are spread over the four different rows.

Together, [theta] and [pi] cause a very quick spread of differences; see Figure 2. Suppose two bytes of the same row at the input of the first round are different, then after [theta], at least three bytes of this row are different. [rho] distributes these bytes to three different rows at the input of the next round. Then 3×4 of the output bytes of [theta] are different. The differences are distributed by [rho] over the four rows, and so on.

An immediate consequence of the mathematical structure of Square is that the resistance against linear cryptanalysis can be treated in an analogous way. Square is resistant to linear and differential cryptanalysis after six of its eight rounds.

There exists, however, a dedicated attack that breaks six rounds of Square (see our paper "The Block Cipher Square," in Fast Software Encryption, E. Biham, editor, Springer-Verlag 1995). Since Square has eight rounds, this dedicated attack poses no problems.

Implementations

To illustrate how Square can be used, we have written a smart card application for the Motorola M68HC05 microcontroller. (Smart cards are becoming important as a platform for implementing cryptography and security.) The machine code occupies 547 bytes of ROM and needs 36 bytes of RAM. One execution of Square (including the key schedule) takes about 7500 cycles. This corresponds to less than 2 milliseconds with a 4-MHz clock. In comparison, a typical DES implementation occupies about 1 KB of ROM. One DES operation encrypts only 64 bits and needs about 16,000 cycles.

Writing out the consecutive transformations of the cipher for 32-bit processors, the succession of steps

occurs seven times. These transformations can be combined into a single set of table lookups. An intermediate value of the text can be represented by four 32-bit words, each containing a row [ai]. Its transpose is denoted by [ai]T. For
we have Figure 3(a). We define the tables M and T as Figure 3(b). T and M have 256 entries of four bytes each. The table M implements the multiplications in GF(28). T combines the nonlinear substitution with the multiplications. Define rotrj([ai]) as the rotation of the row [ai] with j bytes to the right. This yields Figure 3(c). We conclude that
can be done with 16 table lookups, 12 rotations, and 16 XORs of 32-bit words. This implementation needs the table T, with 256 entries of four bytes -- 1 KB in total.

Further Reading

More details about Square are available in our paper "The Block Cipher Square," which has been published in Fast Software Encryption (E. Biham, editor, Springer-Verlag 1995) and at http://www .esat.kuleuven.ac.be/rijmen/square/.

DDJ

Copyright © 1997, Dr. Dobb's Journal

Back to Table of Contents


Dr. Dobb's Journal October 1997: The Galois Field GF(28)

The Galois Field GF(28)

By Joan Daemen, Lars R. Knudsen, and Vincent Rijmen

Dr. Dobb's Journal October 1997

Figure 1: Geometrical representation of the basic operations of Square. It consists of four parallel linear diffusion mappings; g consists of 16 separate substitutions and p is a transposition.

Back to Article

Copyright © 1997, Dr. Dobb's Journal


Dr. Dobb's Journal October 1997: The Galois Field GF(28)

The Galois Field GF(28)

By Joan Daemen, Lars R. Knudsen, and Vincent Rijmen

Dr. Dobb's Journal October 1997

Figure 2: Spreading of differences through four consecutive rounds. The left-side squares stand for the input values of the rounds. The sum of the numbers of bytes that are changed at the inputs of the four rounds equals 28. We can show that, in Square, this sum over four consecutive rounds is always at least 25.

Back to Article

Copyright © 1997, Dr. Dobb's Journal


Dr. Dobb's Journal October 1997: The Galois Field GF(28)

The Galois Field GF(28)

By Joan Daemen, Lars R. Knudsen, and Vincent Rijmen

Dr. Dobb's Journal October 1997

Figure 3: Implementing Square.

Back to Article

Copyright © 1997, Dr. Dobb's Journal


Dr. Dobb's Journal October 1997: The Galois Field

Dr. Dobb's Journal October 1997

The Galois Field GF(28)

Two byte values can be combined in many different ways to result in a third value. In general, a rule to combine two values to obtain a third value is called a "composition law." Some obvious examples of composition laws that are applicable to byte values are modular addition and multiplication, and the bitwise logical OR, AND, and XOR operations. The combination of a set and two composition laws is called an "algebra." If the laws obey certain rules, one of the laws can be seen as "additive" and the other as "multiplicative." Mathematicians have developed powerful tools to derive and prove nonobvious properties for these algebras. A particularly relevant property of such an algebra is the ability to solve x from the equation a+b [dot] x=0 for any value of a and any nonzero value of b. An algebra exhibiting these properties is called a field.

Finite Field

A field with a finite number of elements is called a "finite field" or "Galois Field" (GF()), after the mathematician Evariste Galois.

The most obvious candidates for the two composition laws on bytes are addition modulo 256 and multiplication modulo 256. Unfortunately, the resulting algebra is not a field. For instance, the equation 1+2 [dot] x=0 has no solution.

Still, 28 is a prime power and it is therefore possible to define two composition laws such that the resulting algebra is the field GF(28). For the additive law we take the bitwise XOR. The multiplicative is more involved, and is constructed as follows: The bits in the bytes are considered as coefficients of polynomials of degree smaller than 8 and the composition law is the multiplication of polynomials modulo some irreducible polynomial (with no other divisors than 1 and itself, similar to prime number). In practice, the multiplication can be easily implemented using table lookup methods.

-- J.D., L.K., and V.R.

Back to Article

Copyright © 1997, Dr. Dobb's Journal


Dr. Dobb's Journal November 1997: Algorithm Alley

Inside IMA ADPCM

Dr. Dobb's Journal November 1997

By Tim Kientzle

Tim is senior technical editor for DDJ and author of the forthcoming Programmer's Guide to Sound (Addison-Wesley, 1998). He can be reached at kientzle@ddj.com.

Audio compression is a tricky business. Although the 10:1 or better compression offered by MPEG audio and Dolby AC-2 is impressive, the CPU requirements are, unfortunately, equally impressive. This is fine if you can afford to devote a high-speed CPU entirely to audio decompression, but that's not always an option. Frequently, you'll want to juggle animation or other CPU-intensive actions at the same time.

Intel's DVI video compression engine includes a method for audio compression that offers moderate compression with very low CPU requirements. This engine was later adopted by the Interactive Multimedia Association (IMA), and the audio algorithm has become widely known as "IMA ADPCM." In this article, I'll describe how IMA ADPCM works, and discuss two specific implementations of it, one by Microsoft and the other by Apple.

IMA ADPCM

IMA ADPCM stores four bits of information for each sample. The compressor takes the difference between two samples, divides it by the current step size, and uses that as the next four-bit compressed output. Conversely, the decompressor takes this four-bit value, multiplies it by the current step size, and adds the result to the previous sample to create the next sample.

To keep the amount of state information low, the step size is not stored directly. Rather, there is a table of 88 possible step sizes (Example 1), and the decompressor maintains an index into this table. These step sizes roughly follow an exponential curve. Thanks to this table, the total amount of state information is only 23 bits: seven bits for the index into the step size table and 16 bits for the current decompressed value. In embedded applications, it may be possible to keep the entire state in registers, requiring no RAM at all.

The important part of IMA ADPCM is how it adjusts the step size. Each four-bit nybble is a signed-magnitude value; the high-order bit is the sign, the remaining bits are the value from zero to seven. (Unlike the more familiar twos-complement format, signed magnitude format distinguishes between +0 and -0.)

Since the decompressor doesn't know the next sample, it has to choose a step size based on previous values. If the four-bit scaled difference is large, the next might be larger, overflowing the four-bit value. If the four-bit scaled difference is small, then you're not taking advantage of the four bits of information you have. As a result, the step size is decreased when the scaled difference gets closer to zero, and is increased when the scaled difference gets large. Example 2 shows the table used by the decompressor to update the step size. It uses the four-bit code as an index into this table, and adds the result to the current step size index.

Implementation

You now have the rough outline of the decoder; for each four-bit sample, multiply by the current step size, add the result to the previous output value to get the next output value, and finally, update the current step size.

There's one additional trick in the actual implementation: Rather than directly multiplying, a series of bit operations are used that are roughly -- but not quite -- the same. The advantage of this approach is that similar bit operations can be used in the compressor, and these operations are faster than a full integer division on many processors.

Listing One shows the full decompressor. You simply feed it a nybble at a time (IMA ADPCM specifies that the low-order nybble precedes the high-order nybble), and it returns the decoded sample.

Listing Two is the compressor. To simplify the implementation, ImaAdpcmEncode simply calls ImaAdpcmDecode directly to update the state information.

What IMA Forgot

Both Microsoft and Apple use IMA ADPCM. Microsoft uses it with WAVE files and Apple uses it with AIFF files. One of the key advantages to IMA ADPCM is that it requires so little state information. This allows you to simply dump the complete compressor state at the beginning of each packet. However, IMA failed to specify the length or format of such packets, and as a result the Microsoft and Apple codecs are completely incompatible.

Microsoft's IMA ADPCM Variant

Microsoft uses several fields in the WAVE file header to specify the length of packets, and uses a different packet format for mono and stereo sounds. The "Block Alignment" field specifies the number of bytes per packet. The "Number of Channels" field specifies the number of channels, hence, the precise packet format. In addition, WAVE IMA ADPCM files add two bytes of compressor-specific information, which are used to specify the number of samples stored in the packet. Note that the "Bits per Sample" field is always set to four.

Table 1 illustrates the mono packet's format. It begins with a four-byte header, followed by the compressed data. When you decompress a packet, you emit one sample for the header itself, then one sample for each nybble. For example, if the Block Alignment is 256, the packet will contain 505 samples (504 samples in 252 bytes plus one sample for the four-byte header).

Microsoft resets the decompressor at the beginning of each packet. Although the header has space for the complete 16-bit current sample value, the mono files I've inspected (compressed with the IMA ADPCM compressor included with Windows 95) always set the low-order byte to zero. (The stereo files do store the complete 16-bit current sample for each channel.)

A stereo packet looks very much like two separate mono packets that have been interleaved by alternating 32-bit words, as shown in Table 2. In particular, the extended field in the WAVE header contains the number of samples contained in a packet for each channel. For example, a packet size of 2048 bytes corresponds to 2041 samples in the WAVE header (one for the four-byte decompressor state plus 2040 samples from 1020 compressed bytes).

My implementation is contained in the class DecompressImaAdpcmMs (see Listing Three; the complete source code is available electronically; see "Availability," page 3). To use it, read the number of samples per packet and the number of channels from the WAVE header, and use that to create a DecompressImaAdpcmMs object. You can then call GetSamples on that object to read decompressed data. Internally, GetSamples calls the NextPacket method to read and decode the next packet whenever the two buffers are empty. NextPacket in turn calls ReadBytes to obtain additional data from the file; you'll need to implement ReadBytes appropriately.

Apple's IMA ADPCM Variant

Although Apple and Microsoft used the exact same low-level compression, their packet formats are quite different:

  • Apple uses the same packet format for either mono or stereo. Stereo files alternate one packet of left data with one packet of right data.
  • Microsoft emits one sample for the header, Apple does not. (A Microsoft packet always contains an odd number of samples. An Apple packet always contains an even number of samples.)
  • The Microsoft packet header uses eight bits for the step size index and stores eight or 16 bits of the current sample. Apple uses only seven bits for the step size index and stores the upper nine bits of the current sample.
  • Microsoft's decompressor completely resets at the start of each packet, Apple's does not.
  • The Microsoft packet header contains a zero byte for detecting synchronization errors. Apple does not provide a means for detecting these errors (apart from checking that the seven-bit index value falls in the range 0-88).
  • An Apple packet is always 34 bytes (two bytes of header and 64 compressed samples). Microsoft allows the compressor to specify the packet size; it stores the number of samples per packet in the file header.

With this laundry list in hand, it's not too hard to adapt my Microsoft IMA ADPCM compressor to handle Apple IMA ADPCM files. I've done so in my DecompressImaAdpcmApple class, which I use the same way as the DecompressImaAdpcmMs class.

As Listing Four shows, there's one interesting trick in implementing the Apple-compatible decompressor. Because the packet header only stores the upper nine bits of the current state, Apple's decompressor is careful to only reset the decompressor state when necessary. My implementation of NextPacket is equally careful: If the nine-bit value given matches the upper nine bits of my internal state, I keep my state, since it's more accurate.

Comparing Microsoft's and Apple's IMA ADPCM Codecs

From a coding point of view, Apple's implementation of IMA ADPCM is simpler, mostly because you don't need to worry about two different packet formats. It also provides slightly better quality; Microsoft throws away eight bits of decompressor state at the beginning of each mono packet, which results in a small jump in sample value at the start of each packet.

Microsoft's approach is more compact. Assuming 2048-byte stereo packets, Microsoft's approach incurs only eight bytes of overhead for each 2041 bytes of compressed data (less than 0.4 percent overhead), while Apple incurs two bytes for each 32 bytes of compressed data (over six percent overhead). It also provides lower latency; if stereo data is being compressed and decompressed simultaneously, Microsoft only incurs an eight-sample delay, while Apple must wait for a complete 64-sample packet before it can have stereo data available. However, this is not a very serious issue even at low sampling rates; at 11,025 samples per second, 64 samples is less than six milliseconds.

Conclusion

Although originally defined to be a platform-independent standard for audio-file compression, the IMA ADPCM format doesn't quite meet that requirement. The low-level compression is not a concern; it's independent of byte order and is defined using only low-level bit manipulations, which give it good performance even on processors that don't have fast integer division.

However, the differences between the Microsoft and Apple versions make it impossible to mix their codecs. You can't extract IMA ADPCM compressed data from an AIFF-C file and feed it to the Microsoft codec, nor can you feed IMA ADPCM WAVE data to the Apple codec. Fortunately, IMA ADPCM codecs are compact; the code I've provided here should allow you to easily support both common variants, and you should also be able to adapt this code to support any other IMA ADPCM variants you may encounter.

Acknowledgments

The code presented here is partially based on an implementation of Intel's DVI ADPCM algorithm developed by Jack Jansen at the Stichting Mathematisch Centrum, Amsterdam, The Netherlands.

Listing One

short ImaAdpcmDecode(unsigned char deltaCode, ImaState &state) {   // Get the current step size
   int step = _stepSizeTable[state.index];
   // Construct the difference by scaling the current step size
   // This is approximately: difference = (deltaCode+.5)*step/4
   int difference = step>>3;
   if ( deltaCode & 1 ) difference += step>>2;
   if ( deltaCode & 2 ) difference += step>>1;
   if ( deltaCode & 4 ) difference += step;
   if ( deltaCode & 8 ) difference = -difference;
   // Build the new sample
   state.previousValue += difference;
   if (state.previousValue > 32767) state.previousValue = 32767;
   else if (state.previousValue < -32768) state.previousValue = -32768;

   // Update the step for the next sample
   state.index += indexAdjustTable[deltaCode];
   if (state.index < 0) state.index = 0;
   else if (state.index > 88) state.index = 88;

   return state.previousValue;
}

Back to Article

Listing Two

unsigned char ImaAdpcmEncode(short sample, ImaState &state) {   int diff = sample - state.previousValue;
   int step = _stepSizeTable[state.index];
   int deltaCode = 0;

   // Set sign bit
   if (diff < 0) { deltaCode = 8; diff = -diff; }

   // This is essentially deltaCode = (diff<<2)/step,
   // except the roundoff is handled differently.
   if ( diff >= step ) {  deltaCode |= 4;  diff -= step;  }
   step >>= 1;
   if ( diff >= step ) {  deltaCode |= 2;  diff -= step;  }
   step >>= 1;

   if ( diff >= step ) {  deltaCode |= 1;  diff -= step;  }

   ImaAdpcmDecode(deltaCode,state);  // update state
   return deltaCode;
}

Back to Article

Listing Three

#ifndef IMAADPCM_H_INCLUDED#define IMAADPCM_H_INCLUDED

struct ImaState {
   int index;    // Index into step size table
   int previousValue; // Most recent sample value
};
// Decode/Encode a single sample and update state
short ImaAdpcmDecode(unsigned char deltaCode, ImaState &);
unsigned char ImaAdpcmEncode(short sample, ImaState &);
// Microsoft-compatible decoder
class DecompressImaAdpcmMs {
private:
   int  _channels;
   short *_samples[2];  // Left and right sample buffers
   short *_samplePtr[2]; // Pointers to current samples
   size_t   _samplesRemaining; // Samples remaining in each channel
   size_t   _samplesPerPacket; // Total samples per packet
public:
   DecompressImaAdpcmMs(int packetLength, int channels);
   ~DecompressImaAdpcmMs();
   size_t GetSamples(short *outBuff, size_t len);
private:
   unsigned char *_packet;   // Temporary buffer for packets
   size_t   _bytesPerPacket; // Size of a packet
   size_t  NextPacket();
   size_t ReadBytes(void *buffer, size_t request);
};
// Apple-compatible decoder
class DecompressImaAdpcmApple {
private:
   int _channels;
   ImaState _state[2];
   short _samples[2][64];
   short *_samplePtr[2];
   size_t   _samplesRemaining;
   size_t  NextPacket(short *sampleBuffer, ImaState &state);
public:
   DecompressImaAdpcmApple(int channels);
   size_t GetSamples(short *outBuff, size_t len);
   size_t ReadBytes(void *buffer, size_t request);
};
#endif

Back to Article

Listing Four

size_t  DecompressImaAdpcmApple::NextPacket(short *sampleBuffer, ImaState &state) {
   unsigned char _packet[34];
   // Pull in the packet and check the header
   size_t bytesRead = ReadBytes(_packet,34);
   if (bytesRead < 34) return 0;

   // Check the decompressor state
   state.index = _packet[1] & 0x7f;
   if (state.index > 88) {
      cerr << "Synchronization error.\n";
      exit(1);
   }
   // Recover upper nine bits of last sample
   short lastSample =
      (static_cast<signed char>(_packet[0])*0x100 +
       static_cast<signed char>(_packet[1])) & 0xff80;
   // If ours doesn't match, reset to the value in the file
   if ((state.previousValue & 0xff80) != lastSample)
      state.previousValue = lastSample;
   // Decompress nybbles
   for(int i=0; i<32; i++) {
      *sampleBuffer++ = ImaAdpcmDecode(_packet[i+2] & 0xf, state);
      *sampleBuffer++ = ImaAdpcmDecode((_packet[i+2]>>4) & 0xf, state);
   }
   return 64;
};

Back to Article

DDJ

Copyright © 1997, Dr. Dobb's Journal

Back to Table of Contents


Dr. Dobb's Journal November 1997: Inside IMA ADPCM

Inside IMA ADPCM

By Tim Kientzle

Dr. Dobb's Journal November 1997

static const int _stepSizeTable[89] = {
   7, 8, 9, 10, 11, 12, 13, 14, 16, 17, 19, 21, 23, 25, 28, 31, 34,
   37, 41, 45, 50, 55, 60, 66, 73, 80, 88, 97, 107, 118, 130, 143,
   157, 173, 190, 209, 230, 253, 279, 307, 337, 371, 408, 449, 494,
   544, 598, 658, 724, 796, 876, 963, 1060, 1166, 1282, 1411, 1552,
   1707, 1878, 2066, 2272, 2499, 2749, 3024, 3327, 3660, 4026,
   4428, 4871, 5358, 5894, 6484, 7132, 7845, 8630, 9493, 10442,
   11487, 12635, 13899, 15289, 16818, 18500, 20350, 22385, 24623,
   27086, 29794, 32767
};

Example 1: IMA ADPCM step size table.

Back to Article

Copyright © 1997, Dr. Dobb's Journal


Dr. Dobb's Journal November 1997: Inside IMA ADPCM

Inside IMA ADPCM

By Tim Kientzle

Dr. Dobb's Journal November 1997

static const int indexAdjustTable[16] = {
   -1, -1, -1, -1,  // +0 - +3, decrease the step size
    2, 4, 6, 8,     // +4 - +7, increase the step size
   -1, -1, -1, -1,  // -0 - -3, decrease the step size
    2, 4, 6, 8,     // -4 - -7, increase the step size
};

Example 2: Adjusting the step size.

Back to Article

Copyright © 1997, Dr. Dobb's Journal


Dr. Dobb's Journal November 1997: Inside IMA ADPCM

Inside IMA ADPCM

By Tim Kientzle

Dr. Dobb's Journal November 1997

Table 1: Microsoft IMA ADPCM single-channel packet format.

Back to Article

Copyright © 1997, Dr. Dobb's Journal


Dr. Dobb's Journal November 1997: Inside IMA ADPCM

Inside IMA ADPCM

By Tim Kientzle

Dr. Dobb's Journal November 1997

Table 2: Microsoft IMA ADPCM stereo packet format.

Back to Article

Copyright © 1997, Dr. Dobb's Journal


Dr. Dobb's Journal January 1998: Algorithm Alley

Resizable Data Structures

Dr. Dobb's Journal January 1998

By John Boyer

John is a software development manager at UWI Unisoft Wares Inc. (http://www .uwi.com/) and is also a part-time theoretical computer science graduate student at the University of Victoria. He can be contacted at jboyer@ uwi.com.

Most textbooks discuss basic data structures such as heaps and hash tables by starting with a fixed-size array. Although they are fast, fixed-size arrays are rarely sufficient in practice -- modern software has to handle whatever data is thrown at it, and presized data structures aren't flexible enough to meet such requirements. Conversely, linked structures are flexible, but are more complex, and memory fragmentation can degrade performance. This month, John Boyer explains an easy, fast way to resize arrays and the data structures that use them.

-- Tim Kientzle

Edward Sitarski described how Hashed Array Trees (HATs), coupled with some sophisticated programming techniques, can be used for growing and shrinking arrays in his September 1996 "Algorithm Alley" (available electronically; see "Resource Center, page 3). In this article, I'll show how to do the same thing using techniques that are so simple you can immediately incorporate them into your programming repertoire. Furthermore, you can apply these techniques to existing programs without changing data structures or accidentally introducing new bugs.

I developed these techniques while adding a resizable hash table to the Universal Forms Description Language (http:// www.uwi.com/). The source code for a resizable hash table that uses resizable arrays for its collision lists is available electronically. You can also get code for a resizing binary heap from my January 1997 "Algorithm Alley" on Fibonacci heaps (also available electronically).

Building Some Suspense

As a hash table becomes overcrowded, the benefit of hashing diminishes. But resizing the table requires reinserting all of the elements, since the final operation on a hash key is a modulus by the table size. Without some tricks, your program will suffer every time you have to enlarge or reduce the table.

I recently wrote a lexical analyzer as part of a new parser generator. This analyzer had to be able to read quoted strings of any length. Since the syntax of many languages does not allow the size of a quoted string literal to be specified, it is necessary to grow the string dynamically until its end is found.

As part of my January 1997 Fibonacci heap article, I built a binary heap for comparison with the F-heap. Since F-heaps can grow arbitrarily large, I built the binary heap to reallocate its array when it was full. Binary heaps are assumed to have an O(log n) insert, but if you look closely, this is based on the assumption that space for the new element can be procured in O(log n) time or less. Most textbooks use a fixed array for storage, so there is no overhead in procuring space for new elements. However, it occurred to me that my expansion code would cause O(n) behavior per insertion.

The Extensible Array Problem

These problems can all be solved with an array that can grow dynamically. The obvious approach is to add k elements each time you need more space. Unfortunately, the C function realloc() may copy the entire array to the new storage. If you use C++ new/delete operators, then you must manually perform a reallocation, which guarantees that your array contents must be moved on expansion. So, as your array grows up to some arbitrary size n, you will be moving your array elements a total of k+2k+3k+4k+...+n-k times. The result is O(n2) overall, or O(n) per element.

Consider the aforementioned examples. The lexical analyzer would take O(n2) time to read an n character string, which is unacceptable for parsers (which must achieve O(n) performance to be viable). The binary heap insert was slowed from O(log n) down to O(n), and the hash table insert goes from O(1) down to a pitiful O(n). You can reduce the hidden constant by increasing the expansion size k, but the overall result is still O(n) per array element.

The Solution

The solution is to double the size of the array whenever it must be expanded. Listing One is lex_growbuffer(), which is typical of resizable arrays. The buffer size is doubled rather than being increased by a constant value. Also, this example reallocates the array down to the actual string size once the string terminator is found.

In C++, you have no choice but to copy the array to change its size. As the array size doubles 2,4,8,...,n, you must copy arrays of sizes 1,2,4,...n/2. This is a total of n-1 elements being copied. You expand to size n when you insert the element (n/2+1). Thus, the amortized cost of copies during reallocation never exceeds two time units, which is constant amortized time per element.

At the worst moments, the array will be twice as big as necessary, but this trade-off is more than accounted for by the increased speed of reallocation, and the utter simplicity of using this technique in both new and existing code.

What About Shrinking?

In some applications, it may be necessary to have a delete operation for the resizable array. The binary heap is a good example. After any random sequence of insert and delete operations, it would be helpful if the size of the heap's array were bounded to be a linear factor of the number of elements in the heap. This would ensure that a local "spike" of insert operations couldn't cause the heap to continue occupying unneeded memory.

One approach is to shrink the array by half precisely when it becomes half full. In general, this is the right idea, but it has a weakness. Suppose the random sequence happens to be 1024 inserts followed by an alternating sequence of insert and delete operations. The first insert in the alternating sequence will cause the heap array to grow to 2048 elements in size, since this is the smallest power of two that is greater than 1025. The first delete will reduce the table size to 1024 elements, which would cause an array resize to 1024. The next insert would grow the array again, and the next delete would shrink the array again. Since each operation is resizing the array, each operation will have O(n) performance.

Although most random sequences of insert and delete operations will not cause this problem, I want to guarantee that the worst case will be amortized constant time per operation. To do this, I shrink the array to half size when it is one-quarter full. The worst case sequence would then only alternate the array size between 2k and 2(k+1) by using inserts and deletes to alternate the number of elements between 2(k-1) and 2k+1. When you reach 2k+1 elements, you move 2k elements to a new array before inserting the newest element. You then need a sequence of at least 2(k-1)+1 deletes to cause an array shrink, which will move the 2(k-1) remaining elements to a smaller array. There must be 2(k-1)+1 more inserts to get back up to size 2k+1 and cause an array growth. Thus, each sequence of 2(k-1)+1 deletes followed by 2(k-1)+1 inserts will cause shrinking and growth. This resizing results in 2(k-1) moves after the deletion phase and 2k moves after the insert. This is clearly still constant time per operation since you do fewer than 2k+2k-1 element moves for each 2k+2 operations.

Thus, even with deletes, you can maintain constant time per operation and the data structure will never use more than four times the required memory -- even if there are large spikes of insert operations in the sequence.

The Hash Table

I was introduced to resizable hash tables in Introduction to Algorithms, by T. Cormen, C. Leiserson, and R. Rivest (MIT Press, 1990). It was only while building the binary heap included in my Fibonacci heap article that I realized the applicability of the doubling technique to all extensible array problems.

The key to resizable hash tables is simply creating a new array as described earlier, copying the existing elements to it using the hash function for insertion, then destroying the old array. Rebuilding the entire table may seem like brute force, but as far as amortized time complexity goes, there is no difference because copying an element is constant time and hash-table inserting is "expected" constant time. Using a hash function might take longer than a memory copy, but this is simply a larger constant.

realloc() is not appropriate for resizing a hash table because two differently sized tables will store many elements in completely different places. Thus, you are forced to manually move objects from the old array into the new array.

Java and the STL

The Java hash-table implementation uses array doubling to increase the hash table size, but it doesn't shrink the hash table when it becomes smaller. The C++ Standard Template Library (STL) vector class, which is used as the basis for the priority queue (binary heap), also uses array doubling to increase the size of the vector, but a request to reserve an amount of memory smaller than the current capacity of the container is ignored. This should be changed because data structures (such as heaps used in long-running programs like servers) must be able to reduce their memory allocation after a spike of data.

The current STL hash-table implementation does not automatically increase or decrease the number of buckets in the hash table (each bucket grows to accept collisions, which increases the load on the hash table). There is a resize() function that will rebuild the hash table with the number of buckets given as a parameter. So, if you are using the STL hash table, then use the resize() function only to double or halve the number of buckets when the hash table is full or one-quarter full, respectively. This will ensure an expected O(1) collision list size.

Good Hashing Functions

When using this technique of varying the number of buckets in the hash table to keep the load down, it is important to have a good hashing function that achieves high element dispersion regardless of table size. The efficiency of a hash function is measured by dividing the number of buckets by the number of elements in the hash table. A perfect hash has 100 percent efficiency because each element has its own bucket.

Listing Two is a simple hash function that performs calculations using the binary codes of the characters in a key. The record's position in the table is calculated based only on the key value itself, not how many elements are in the table. This is why hash-table inserts, deletes, and searches are rated constant time.

HashJB alternates addition and multiplication. Addition is much faster, but multiplication causes the hash value to grow quickly (so that it can reach virtually any position in the table regardless of how big it gets). Also, notice that I perform the modulus Key%N using Key&(N-1). Because the array size doubles, N is always a power of two, so the modulus can be replaced with the much faster subtract and bitwise-and.

Truncating the resulting key value is a problem, since it does throw away information in the upper bits of the hash value. I later added two lines that mix the upper bits into the lower bits, which reduced the number of collisions in my tests.

I also investigated other hash functions. When I tried Peter J. Weinberger's hash function (see "Hashing Rehashed," by Andrew Binstock, DDJ, April 1996, available electronically), my test program slowed noticeably. Further investigation revealed that hash tables generated with HashPJW() contained many more collisions than those generated with HashJB(). This was slowing down the UFDL form-load time because, after parsing the form and building the hash table, the compute engine makes extensive use of the hash table. Why was HashPJW() hashing so poorly? Like the first draft of HashJB(), it doesn't seem to account for the information lost during the modulus truncation of the hash value. When I applied the code to mix the higher bits into the lower bits, the efficiency of HashPJW() increased dramatically -- nearly as high as HashJB() on my sample forms.

More recently, Bob Jenkins presented a hashing function ("Algorithm Alley," DDJ, September 1997, also available electronically) that performs a complex mixing operation using 12 bytes at a time from the input key. By design, the bytes of the key have some affect on all regions of the hash value. Thus, the hash function suffers little or no information loss when the hash value is truncated to the table size. As Table 1 shows, the efficiency of Jenkins' hash function was far greater than the original HashPJW(), and even slightly better than HashJB() and the new HashPJW(). The run time using HashJB() is equivalent to HashJenkins() because the former uses slightly less CPU time to compute hash values.

Conclusion

In over half a dozen projects, I've found that the array-doubling/halving technique was responsible for an order of magnitude performance increase when compared to reallocation by a constant size. Furthermore, the idea was typically applied to large existing projects in minutes, rather than the extensive redesign and recoding efforts that might be required by new data structures. The technique does use extra memory; sometimes this can be eliminated (as with the lexical analyzer that downsizes strings). But even when the extra memory usage cannot be eliminated (as with a hash table), the trade-off is usually more than accounted for by the need to increase execution speed and the utter simplicity of using this technique.

DDJ

Listing One

char *lex_growbuffer(char *Buffer, long *pBufferSize){
char    *TempP;
long    NewSize;
    if (*pBufferSize==0) NewSize = 16;
    else NewSize = *pBufferSize * 2;
    TempP = (char *) realloc(Buffer, NewSize+1);
    if (TempP != NULL) *pBufferSize = NewSize;
    return TempP;
}
  ... /* Snippet to add a character to the lex buffer */

   if (BufferLen == BufferSize)
        Buffer = lex_growbuffer(Buffer, &BufferSize);
    if (Buffer == NULL) return NULL;
    Buffer[BufferLen++] = NewChar;
    Buffer[BufferLen] = '\0';
    return Buffer;
  ...
/* After string is read, reduce to exact size */
    Buffer = (char *) realloc(Buffer, BufferLen+1);

Back to Article

Listing Two

typedef unsigned long ulong;typedef unsigned char uchar;
ulong HashJB(uchar *KeyStr, ulong N)
{
ulong Hash=0, I=0;
uchar Ch;
    for (;Ch=*KeyStr++; I^=1)
        if (I)
            Hash *= Ch;
        else    Hash += Ch;
    Hash += ((Hash&0xFFFF0000)>>16);
    Hash += ((Hash&0x0000FF00)>>8);
    return Hash & (N-1);
}

Back to Article

Copyright © 1998, Dr. Dobb's Journal

Back to Table of Contents


Dr. Dobb's Journal January 1998: Resizable Data Structures

Resizable Data Structures

By John Boyer

Dr. Dobb's Journal January 1998

Table 1: Comparison of hash functions on typical UFDL forms. 2150 objects were placed in the hash table. *Efficiency is the number of buckets/N. **Time is the number of seconds to load the sample form and run all computes in it.

Back to Article

Copyright © 1998, Dr. Dobb's Journal


Dr. Dobb's Journal February 1998: The Flashsort1 Algorithm

The Flashsort1 Algorithm

Dr. Dobb's Journal February 1998

By Karl-Dietrich Neubert

Karl does basic research in solid-state and atomic physics, biophysical systems, and computer algorithms. He can be reached at karl-dietrich.neubert@usa.net.

Many programmers are familiar with the theoretical result that "no sorting algorithm can be faster than O(n log n) on average." But this result assumes that you only have limited information; it assumes you only know how to compare two items. However, what if your keys are the integers from 0 to 100? In that case, you can simply place each item directly into the correct location -- no comparisons are necessary. There are many other situations where you can guess the final position of each item. This month, Karl shows how this basic idea can be used to fashion an O(n) sorting algorithm.

-- Tim Kientzle

Most discussions of sorting algorithms are based on comparing elements. Although it's faster to classify elements, algorithms based on classification suffer because they require considerable auxiliary memory. The Flashsort1 algorithm I present here addresses this problem. It sorts in O(n) time and requires less than 0.1n auxiliary memory to sort n elements. I achieve this by using classification to do the long-range ordering with in-place permutation, then use a simple comparison method for the final short-range ordering of each class. I assume here that the elements are approximately evenly distributed real numbers, although the algorithm can be adapted to other types of data.

The algorithm (see Listing One) consists of three logical blocks: classification, permutation, and straight insertion. Classification determines the size of each class of elements. Permutation does long-range reordering to collect elements of each class together, and straight insertion does the final short-range ordering.

Classification

The whole idea of a classification sort is simple: Assuming the elements A(i) are about evenly distributed, you can compute the approximate final position directly from the element value, with no comparisons. If the maximal element is Amax and the minimal element is Amin, you can compute:

K(A(i))=1+INT((m-1)(A(i)-Amin)/(Amax-Amin))

The result will be a value from 1 to m, which I'll call the class of A(i). Note that, on average, you have approximately n/m elements in each class. (In fact, the class K=m only has elements equal to Amax, and the remaining classes are slightly larger.)

Since the main part of the algorithm relies on it, I start by computing the actual number of elements in each class. This requires simply scanning the data and counting the total number of elements.

Since I know the size of each class, I can predict where each class will appear in the final array. The vector L is used to track this information. Initially, each L(K) indicates the upper end of the section that will contain the elements in class K. In particular, L(1) is simply the number of elements in class 1, and the final L(m) is equal to n. As I rearrange things, each L(K) is updated.

The first part of the classification is O(n), since it requires scanning the input to find the size of each class. Then there's an O(m) part, which adds up the class sizes to determine the initial L(K) values.

Permutation

The next step is to move elements into the correct class. Obviously, whenever I move one element into its new location, I have to move the old element from that location. My code uses a single temporary variable to hold the evicted element. I can then compute its new location and continue until I'm forced to stop.

The first problem is determining where to put each element. Rather than try to place an element A(i) exactly into its final location, I simply compute its class index K and place it into the location indicated by L(K). I then decrement L(K). As I move elements into their appropriate classes, these class pointers grow downward.

This process ends when the first class has been filled up. At that time, there are generally still elements that need to be moved because permutations usually consist of several cycles. Therefore, the second problem arises: How do you find those elements that are cycle leaders? Fortunately, it's easy to find such elements, since they're located in the wrong class. I keep a variable j in my implementation and use j to search through the array to find the next item that satisfies the condition j<L(K(A(j))). The corresponding item A(j) serves as the new cycle leader.

Each item is moved exactly once. I can simply count the total number of moves and stop when I've moved every element. (In fact, I can stop before that; if there's only one element unmoved, it must already be in the right place.)

This process is all O(n). Each item is moved once, and j scans through the entire array at most once.

Short-Range Ordering

Once the permutation step has moved everything into approximately the correct place, you have a partially sorted array to work with. Following Robert Sedgewick (see "Implementing Quicksort Programs," Communications of the ACM 21, 1978), I use a straight-insertion sort over the entire array. Assuming the classes are about the same size, this requires O((n/m)2) time for each class.

Minimizing Run Time

By adjusting the number of classes, you can fine tune the overall running time of the algorithm. If you decrease the number of classes, the classification time decreases, but the time for the straight-insertion sort increases. To estimate the optimal number of classes, I start by estimating the total time of the algorithm:

  • The permutation step is O(n), as is part of the classification. This contributes a1 [multiplication dot] n to the total time.
  • The rest of the classification is O(m), contributing a2 [multiplication dot] m to the total.
  • The straight-insertion sort contributes a3 [multiplication dot] m [multiplication dot] (n/m)2.

Here, a1, a2, and a3 are some numbers that I'll determine later.

I can use simple calculus to determine the optimum value. From differentiating t=a1 [multiplication dot] n+a2 [multiplication dot] m+a3 [multiplication dot] m [multiplication dot] (n/m)2, I get dt/dm=a2-a3(n/m)2=0. Solving for m, I have m/n= [square root] (a3/a2). In particular, I don't need to estimate a1.

By assuming shifts were twice as expensive as comparisons, Nicolas Wirth ("Algorithm and Datenstrukturen," B.G. Teubner, 1983) arrived at a3=3/8 (using some arbitrary unit of time) for the time constant of straight-insertion sorting. Following his lead, I estimated a2=2, which gives an optimal m of 0.43n.

Figure 1 shows experimental values for the time of the various procedures as a function of m/n. As expected, the permutation part takes the same amount of time, regardless of m. Between about m=0.2n and m=0.5n, there is a broad minimum of the overall time. For larger classes, the straight insertion becomes excessive; and for smaller classes, it takes longer to perform the initial classification. In particular, if m>n, then many classes are empty, and straight insertion gains you no additional time.

Figure 2 shows the experimental frequency distribution of the run time for the different procedures at fixed n and fixed m/n. The time needed for classification is, of course, constant. The permutation time varies because of the varying time required to locate the next cycle leader. The straight-insertion sort varies depending on the size of each class. The distribution for the total run time reflects the combination of these two distributions.

The total time for Flashsort1 does not depend on the original order; partially ordered or reverse ordered elements are sorted just as fast as randomly ordered ones. However, if the values are not evenly distributed, the run time can increase unpredictably. One way to address this may be to use another classification and permutation step to sort large classes before the final straight insertion.

Comparison to Other Sorts

Figure 3 shows some experimental results on the relative run times. The separate curves indicate the running time of Flashsort1, Heapsort, and Quickersort using sequences of uniform random numbers as test numbers. As expected, the run time of Flashsort1 increases linearly with n. For m=0.1n, Flashsort1 is always faster than Heapsort, and is faster than Quickersort if n>80. For n=10000, Flashsort1 is approximately twice as fast as Quickersort. For m=0.2n, Flashsort1 is about 15 percent faster than for m=0.1n; for m=0.05n, it is about 30 percent slower, but if n>200, it is still significantly faster than Quickersort.

Discussion

Flashsort1 sorts by permutation in-situ in O(n) time with only a short auxiliary pointer vector. In "Mathematical Analysis of Algorithms," Donald Knuth remarked "...that research on computational complexity is an interesting way to sharpen our tools for the more routine problems we face from day to day." As he points out, time effective in-situ permutation is inherently connected with the problem of finding the cycle leaders, and in-situ permutation could easily be performed in O(n) time if we were allowed to manipulate n extra "tag" bits specifying how much of the permutation has been carried out at any time. Without such tag bits, he concludes "it seems reasonable to conjecture that every algorithm for in-situ permutation will require at least n log n steps on the average." In this article, I showed how n elements can be put into m property classes with m being only a small fraction of n. Instead of n tag bits, I need only an auxiliary vector of length m to sort the elements into m classes. As a final step, small numbers of partially distinguishable elements are sorted locally within their classes by a conventional sort algorithm.

Acknowledgment

I'd like to express my thanks to Christopher McManus of the Veterans Affairs Medical Center in Washington, D.C., for stimulating discussions and helpful comments on the manuscript.

References

Duccin, F. "Tri par addressage direct," R.A.I.R.O. Informatique/Computer Science 13, (1979), 225-259.

Feurzeig, W. "Algorithm 23: Math Sort," Communications of the ACM 3, (1960), 601.

Gamzon, E., C.F. Picard. "Algorithme de tri par addressage direct," C. R. Acad. Sc. Paris 269, (1969), 38-41.

Knuth, D.E. "Mathematical Analysis of Algorithms," Information Processing 71, North Holland Publishing, 1972.

Macleod, I.D.G. "An Algorithm for In Situ Permutation," The Australian Computer Journal 2, (1970), 16-19.

Scowen, R.S. "Algorithm 271: Quickersort," Communications of the ACM 8, (1965), 669-670.

Williams, J.W.J. "Algorithm 232: Heapsort," Communications of the ACM 7, (1964), 347-348.

DDJ

Listing One

      SUBROUTINE FLASH1 (A. N. L. M)C     SORTS ARRY A WITH N ELEMENTS BY USE OF INDEX VECTOR L
C     OF DIMENSION M WITH M ABOUT 0.1 N.
C     COPYRIGHT (C)  K. - D. NEUBERT  1997

      DIMENSION A(1),L(1)
C     ============================ CLASS FORMATION =====
      ANMIN=A(1)
      NMAX=1
      DO I=1,N
         IF( A(I).LT.ANMIN) ANMIN=A(I)
         IF( A(I).GT.A(NMAX)) NMAX=I
      END DO
      IF (ANMIN.EQ.A(NMAX)) RETURN
      C1=(M - 1) / (A(NMAX) - ANMIN))
      DO K=1,M
         L(K)=0
      END DO
      DO I=1,N
         K=1 + INT(C1 * (A(I) - ANMIN))
         L(K)=L(K) + 1
      END DO
      DO K=2,M
         L(K)=L(K) + L(K - 1)
      END DO
      HOLD=A(NMAX)
      A(NMAX)=A(1)
      A(1)=HOLD
C     =============================== PERMUTATION =====
      NMOVE=0
      J=1
      K=M
      DO WHILE (NMOVE.LT.N - 1)
         DO WHILE (J.GT.L(K))
            J=J + 1
            K=1 + INT(C1 * (A(J) - ANMIN))
         END DO
         FLASH=A(J)
         DO WHILE (.NOT.(J.EQ.L(K) + 1))
            K=1 + INT(C1 * (FLASH - ANMIN))
            HOLD=A(L(K))
            A(L(K))=FLASH
            FLASH=HOLD
            L(K)=L(K) - 1
            NMOVE=NMOVE + 1
         END DO
      END DO
C     ========================= STRAIGHT INSERTION =====
      DO I=N-2,1,-1
         IF  (A(I + 1).LT.A(I)) THEN
            HOLD=A(I)
            J=I
            DO WHILE  (A(J + 1).LT.HOLD)
               A(J)=A(J + 1)
               J=J + 1
            END DO
            A(J)=HOLD
         ENDIF
      END DO
C     =========================== RETURN,END FLASH1 =====
      RETURN
      END

Back to Article

Copyright © 1998, Dr. Dobb's Journal

Back to Table of Contents


Dr. Dobb's Journal February 1998: The Flashsort1 Algorithm

The Flashsort1 Algorithm

By Karl-Dietrich Neubert

Dr. Dobb's Journal February 1998

Figure 1: The time contribution of the various procedures to the total run time as a function of m/n. P=permutation time, C=classification time, S=straight-insertion time, F=total time for Flashsort1.

Back to Article

Copyright © 1998, Dr. Dobb's Journal


Dr. Dobb's Journal February 1998: The Flashsort1 Algorithm

The Flashsort1 Algorithm

By Karl-Dietrich Neubert

Dr. Dobb's Journal February 1998

Figure 2: Experimental frequency distribution of the various procedures at fixed n and fixed m/n.

Back to Article

Copyright © 1998, Dr. Dobb's Journal


Dr. Dobb's Journal February 1998: The Flashsort1 Algorithm

The Flashsort1 Algorithm

By Karl-Dietrich Neubert

Dr. Dobb's Journal February 1998

Figure 3: Experimental results on the relative run time for several sorting algorithms as a function of n.

Back to Article

Copyright © 1998, Dr. Dobb's Journal


Dr. Dobb's Journal March 1998: Algorithm Alley

Exponential Smoothing

Dr. Dobb's Journal March 1998

By William Stallings

William is the author of High-Speed Networks: TCP/IP and ATM Design Principles (Prentice Hall, 1998). He can be reached at http://www.shore.net/~ws/ or ws@shore.net.

Whenever your software interfaces with hardware or other software, it's hard to know what to expect. How much memory will be required? How long will it take? The most robust systems measure past performance, then use that to predict future response. In this way, your software can smoothly adapt to varying situations.

In "Improving Kermit Performance" (DDJ, February 1996; available electronically, see "Resource Center," page 3), I used a "decaying average" to predict future delays, then used those predictions to optimize error handling. This month, William explores this technique -- also known as "exponential smoothing" -- in more detail and explains a variety of uses.

-- Tim Kientzle

You often need to guess the next value in a time series based on previous values in the time series. For example, the nonpreemptive scheduling policy known as "shortest process next" (SPN) selects the process with the shortest expected processing time (before it is blocked by an I/O or system call) to run next. For this purpose, the operating system keeps a running average of all of the previous processing bursts for each process, and uses these averages to estimate the next bursts. Running averages for estimating future values also show up in many areas of communications protocol design.

Smoothing Techniques

Suppose you have a series of measured values V(1), V(2), V(3),..., where V(i) is the value observed at the ith observation. Assume you want to predict (estimate) the value V(K+1) based on the value up through V(K). One approach would be simply to take the average of observed values, as in Figure 1(a), where AV(K) is the average of the first K values. The estimate for the next value in the series, V(K+1), is equal to AV(K).

Figure 1(b) shows the expression reformulated so that it is not necessary to recalculate the entire summation each time.

Each term in the summation is given equal weight; that is, each term is multiplied by the same constant 1/K. Typically, you would like to give greater weight to more recent instances because they are more likely to reflect future behavior. A common technique for predicting the next value on the basis of a series of past values is known as "exponential averaging," or "exponential smoothing." Figure 2(a) shows how you compute SV(K), the "smoothed estimate."

Compare Figure 2(a) with Figure 1(b). By using a constant value of [alpha] (0< [alpha] <1), independent of the number of past observations, you have a circumstance in which all past values are considered, but more distant ones have less weight. To see this more clearly, consider Figure 2(b). Since both [alpha] and (1- [alpha]) are less than 1, each successive term in the preceding equation is smaller. For example, for a=0.8, the expansion is Figure 2(c). The older the observation, the less it is counted in the average.

Figure 3 shows the size of the coefficient as a function of its position in the expansion. The smaller the value of [alpha], the greater the weight given to the more-recent observations. For a=0.5, virtually all of the weight is given to the four or five most recent observations, whereas for [alpha] =0.875, the averaging is effectively spread out over the ten or so most recent observations. The advantage of using a small value of [alpha] is that the estimate will quickly reflect a rapid change in the observed quantity. The disadvantage is that brief surges in the value of the observed quantity will result in jerky changes in the smoothed average.

Figure 4 compares simple averaging with exponential averaging (for two different values of [alpha]). In Figure 4(a), the observed value begins at 1, grows gradually to a value of 10, and then stays there. In Figure 4(b), the observed value begins at 20, declines gradually to 10, and then stays there. In both cases, you start out with an estimate of SV(1)=0. Exponential averaging tracks changes in process behavior faster than does simple averaging and the smaller value of a results in a more rapid reaction to the change in the observed value.

TCP Retransmission Timer

In the Transmission Control Protocol (TCP), two TCP entities exchange segments over a TCP connection. Each side retains a copy of each segment that it sends and, if an acknowledgment is not received within a retransmission timeout (RTO) interval, resends the segment. Thus, if a segment is lost in transit, it will automatically be resent. If RTO is set too small, then a TCP entity may retransmit a segment for which an acknowledgment is on its way but delayed because of network congestion; such unnecessary retransmissions actually worsen the congestion. On the other hand, if RTO is too large, the protocol will be sluggish in responding to a lost segment.

Initially, TCP used a smoothed average to estimate the round-trip time. However, this didn't work well if the round-trip time varied significantly. To cope with this, Van Jacobson proposed a refinement in the standard algorithm that has now been officially adopted for TCP. Figure 5 summarizes this algorithm.

The Van Jacobson algorithm provides an improvement by estimating both the round-trip time (RTT) and the standard deviation of the RTT. The RTO value is then set to the estimated value of RTT plus a constant multiplied by the estimated standard deviation. This enables the use of more reasonable values of the retransmission timer. Standard implementations of TCP use the parameters g=1/8=0.125, h=1/4=0.25, and f=4. Experience has shown that Van Jacobson's algorithm can significantly improve TCP performance.

ATM ABR Service

Support for bursty data traffic on ATM networks, such as traffic generated by TCP/IP-based applications, has traditionally been carried on the so-called Unspecified Bit Rate (UBR) service. UBR is designed to make use of available capacity on the ATM network not consumed by time-sensitive traffic such as voice and video. All of this unused capacity could be made available for the UBR service. This service is suitable for applications that can tolerate variable delays and some cell losses, which is typically true of TCP-based traffic. With UBR, cells are forwarded on a first-in-first-out (FIFO) basis using the capacity not consumed by other services; both delays and variable losses are possible. No initial commitment is made to a UBR source and no feedback concerning congestion is provided; this is referred to as a "best-effort service."

To improve the service provided to bursty sources that would otherwise use UBR, the Available Bit Rate (ABR) service has been defined. An application using ABR specifies a peak cell rate (PCR) that it will use and a minimum cell rate (MCR) that it requires. The network allocates resources so that all ABR applications receive at least their MCR capacity. Any unused capacity is then shared in a fair and controlled fashion among all ABR sources. The ABR mechanism uses explicit feedback to sources to assure that capacity is fairly allocated. Any capacity not used by ABR sources remains available for UBR traffic.

To determine how much capacity to allow to each connection, each ATM switch must monitor the traffic through itself. One of the techniques recommended by the ATM Forum for this purpose uses the equation MACR(I)=(1- [alpha])&times;MACR(I-1) + [alpha] &times;CCR(I) for each virtual connection through the switch, where CCR is a measure of the current cell rate on a given connection, and MACR (mean allowed cell rate) is an estimate of the cell rate for the connection in the next sampling period. Typically, [alpha] =1/16, so that more weight is given to past values of CCR than to the current value. Based on the estimates for each connection, if a switch experiences congestion, it restricts the cell rate of the connections passing through it in a manner that is proportional to the estimated demand from each connection.

Real-Time Transport Protocol (RTP)

Real-time transport protocol (RTP) is designed to support both point-to-point and multicast real-time applications. RTP overcomes three deficiencies in TCP for real-time applications:

  1. TCP is a point-to-point protocol that sets up a connection between two end points. Therefore, it is not suitable for multicast distribution.
  2. TCP includes mechanisms for retransmission of lost segments, which then arrive out of order. Such segments are not usable in most real-time applications.
  3. TCP contains no convenient mechanism for associating timing information with segments, which is another real-time requirement.

One key requirement for receivers of real-time traffic is to estimate the amount of delay variation, or jitter, experienced on a connection in order to determine buffering requirements. In essence, the receiver would like to pass incoming traffic through a delay buffer that will smooth out delay variability so that the received data has the same timing characteristics as the transmitted data. There is no simple way to measure this quantity at the receiver, but it is possible to estimate the average jitter in the following way. At a particular receiver, you define the following parameters for a given source:

  • S(I) = Timestamp from RTP data packet I.
  • R(I) = Time of arrival for RTP data packet I, expressed in RTP timestamp units. The receiver must use the same clock frequency (increment interval) as the source, but need not synchronize time values with the source.
  • D(I) = The difference between the interarrival time at the receiver and the spacing between adjacent RTP data packets leaving the source.
  • J(I) = Estimated average interarrival jitter up to the receipt of RTP data packet I.

The value of D(I) is calculated as D(I)=(R(I)-R(I-1))-(S(I)-S(I-1)). Thus, D(I) measures how much the spacing between arriving packets differs from the spacing between transmitted packets. In the absence of jitter, the spacings will be the same and D(I) will have a value of 0. The interarrival jitter J(I) is calculated continuously as each data packet I is received, according to the formula in Figure 6. J(I) is calculated as an exponential average of observed values of D(I). Only a small weight is given to the most recent observation, so that temporary fluctuations do not invalidate the estimate. The estimated value of jitter is reported to other participants in the connection. The jitter measure may provide a warning of increasing congestion before it leads to packet loss.

Internet Congestion Control

When routers in a network become congested to the point of buffer saturation, it becomes necessary to discard incoming or buffered packets. It may be desirable for routers to begin to discard packets before total saturation occurs. In this way, TCP connections receive early warning of increased congestion as the result of a few lost packets. The TCP connections can then back off the volume that they generate to avoid more catastrophic congestion. The most important example of a proactive packet discard scheme is Random Early Detection (RED), introduced by Sally Floyd. RED has been implemented by a number of router vendors.

In essence, RED monitors the queue length for an output port on the router. Associated with each output buffer are two thresholds THmax and THmin. When the queue length is less than THmin, no discard occurs; when it is greater than THmax, any arriving packet is discarded. Between these two thresholds, an incoming packet has an increasing probability of being discarded as the queue length grows from the lower to the higher threshold. The use of two thresholds and a sliding probability scale affords the opportunity to fine-tune the algorithm, and experience has shown that it can be effective in anticipating congestion without wasteful discard.

However, rather than use the actual queue length for the calculation, RED uses a smoothed average queue length SQ(K)=(1-W)&times;SQ(K-1)+W&times;Q(K), where Q is the actual queue length, SQ is the smoothed average, and W is a weighting factor.

You might wonder why an average queue size is used when it would be simpler to use the actual queue size. The purpose of using an average queue size is to filter out transient congestion at the router. The weight W determines how rapidly SQ changes in response to changes in actual queue size. Floyd recommends a quite small value of 0.002. As a result, SQ lags considerably behind changes in actual queue size. The use of this small weight prevents the algorithm from reacting to short bursts of congestion.

Summary

Exponential averaging is used in a surprisingly wide variety of protocols and congestion-control algorithms. By varying the weighting factor used in the average, more or less weight can be given to recent observations compared to more distant ones. Anyone needing a simple technique for making estimates based on past observations should be aware of this versatile tool.

References

The algorithm used by TCP to adjust its retransmission timeout was described by Van Jacobson in "Congestion Avoidance and Control." Proceedings, SIGCOMM '88, Computer Communication Review, August 1988. The article was reprinted in Computer Communication Review, January 1995. A revised version is available at ftp.ee.lbl.gov/papers/congavoid.ps.Z.

The RED technique was described by Sally Floyd and Van Jacobson in "Random Early Detection Gateways for Congestion Avoidance," IEEE/ACM Transactions on Networking, August 1993.

DDJ

Copyright © 1998, Dr. Dobb's Journal

Back to Table of Contents


Dr. Dobb's Journal March 1998: Exponential Smoothing

Exponential Smoothing

By William Stallings

Dr. Dobb's Journal March 1998

Figure 1: (a) Computing the average; (b) this version doesn't require a summation.

Back to Article

Copyright © 1998, Dr. Dobb's Journal


Dr. Dobb's Journal March 1998: Exponential Smoothing

Exponential Smoothing

By William Stallings

Dr. Dobb's Journal March 1998

Figure 2: SV(K) is the smoothed average. Note that 0< [alpha] < 1.

Back to Article

Copyright © 1998, Dr. Dobb's Journal


Dr. Dobb's Journal March 1998: Exponential Smoothing

Exponential Smoothing

By William Stallings

Dr. Dobb's Journal March 1998

Figure 3: Weighting of recent values for different values of [alpha].

Back to Article

Copyright © 1998, Dr. Dobb's Journal


Dr. Dobb's Journal March 1998: Exponential Smoothing

Exponential Smoothing

By William Stallings

Dr. Dobb's Journal March 1998

Figure 4: Comparing (a) simple and (b) exponential averaging.

Back to Article

Copyright © 1998, Dr. Dobb's Journal


Dr. Dobb's Journal March 1998: Exponential Smoothing

Exponential Smoothing

By William Stallings

Dr. Dobb's Journal March 1998

Figure 5: The Van Jacobson TCP retransmission algorithm.

Back to Article

Copyright © 1998, Dr. Dobb's Journal


Dr. Dobb's Journal March 1998: Exponential Smoothing

Exponential Smoothing

By William Stallings

Dr. Dobb's Journal March 1998

Figure 6: Formula for calculating the interarrival jitter J(I) continuously as each data packet I is received.

Back to Article

Copyright © 1998, Dr. Dobb's Journal


Algorithm Alley

Adaptive Delta Modulation

Dr. Dobb's Journal April 1998

Algorithms for audio compression

By Gary D. Knott

Gary works at Civilized Software and can be contacted at knott@civilized.com.

One common method of data compression is delta modulation, a technique that's typically used for sound and/or video digital radio transmissions and recordings as well as other types of signals. Delta modulation is a data-encoding technique that also compresses the sequence of sampled function values being encoded. It is particularly useful for transmitting fewer bits (over a network, for instance), and for archival storage. In this article, I'll review the basic delta-modulation algorithm, then examine variations that implement "adaptive delta modulation." For an assembly-language implementation of delta modulation, see "Audio Compression," by John W. Ratcliff (DDJ, July 1992).

The idea behind the delta modulation representation of a signal f is quite simple. Given the initial value, h=f(s0), a sampling interval [alpha], and an increment value [delta], you can interpret a string of bits b1, b2,...bn to obtain a specification of estimated signal values [f with hat] (t1), [f with hat] (t2),... [f with hat] (tn), where ti=s0+i [alpha] for 0 [less than or equal to] i [less than or equal to] n, as follows: Each bit bi indicates adding the constant [delta] if bi=1 and subtracting [delta] if bi=0 so that the equation in Example 1 holds. For example, consider f(t)=sin(t)+2cos(3t)exp(-t) if t [less than or equal to] 6; and f(t)=-.27614(1-exp(-2(t-6))) otherwise. Figure 1 shows a graph of f on the interval [0,11].

Let s1=tn. Choosing the parameter values s0=0, s1=11, [delta] =0.1, and [alpha] =0.1, you have h=2 and the delta modulation representation is the binary digital sequence

00000000000000111111000000000000000000
10000101010101111011111101010010101010
101010101010101010101010101010101.

As Figure 2 illustrates, this sequence specifies [f with hat] drawn with the traditional step-function interpolation that would be seen on an oscilloscope. The smooth curve is the true signal, and the stepped curve is the step-function form of [f with hat]. Thus the function [f with hat] on the interval 0 to 11 is represented with 110 bits.

The encoding process computes the ith bit, bi, by predicting f(ti) to be some value Pi. Then, bi is taken as 1 if f(ti)>Pi and 0 otherwise. The choice of Pi used here is merely the previous estimate value [f with hat] (ti-1).

In general, to obtain a reasonable estimate for f, the sampling interval, [alpha], must be such that (s1-s0)/ [alpha] is greater than the Nyquist frequency, which is twice the frequency of the highest-frequency component present in the signal f, and [delta] must be small enough to track high-frequency oscillations without undue shape distortion. A practical way to choose [alpha] and [delta] is to choose [delta] as the largest absolute error by which [f with hat] may deviate from f, then choose [alpha] = [delta] /W, where W=maxs0 [less than or equal to] t [less than or equal to] s1|df(t)/dt|, the maximum absolute is slope of f. In Figure 2, [alpha] is clearly too large for the chosen [delta]; the result is the large error in the initial part of the signal, called "slope overload error," where the slope is too large in magnitude for [delta] =0.1 and [alpha] =0.1 to track the signal.

Even when the signal is being appropriately followed, the estimate oscillates about it. This "granular" noise is unavoidable, although its size is controlled by [delta]. Note that the error characteristics of the estimator are given by |f(t)- [f with hat] (t)|< [delta] for s0 [less than or equal to] t [less than or equal to] s1, assuming a is small enough. This is an absolute error criterion rather than a relative error criterion, and [f with hat] behaves like a Chebychev approximation to f.

A delta modulation signal is sensitive to transmission error. Changing a burst of a dozen bits or so during transmission can destroy the validity of the remaining bits. However, higher sampling rates mean short burst errors are less harmful, and methods to periodically restart the delta-modulation process can be included in a practical transmission system. In general, delta modulation is a very efficient way to encode a signal. It is not clear how to define the notion of the efficiency of an approximation (as opposed to an exact encoding) in a precise information-theoretic manner, but this is an intriguing direction for investigation.

Extending Delta Modulation

You can extend the idea of basic delta modulation in several ways. One approach is to let the increment [delta] assume various values, depending upon the past tracking of the signal. If you output m 1s or 0s in a row (indicating a region of large absolute slope), you can increase [delta], replacing [delta] with 3 [delta] /2, for example. If you output m alternating 1s and 0s, you can then decrease [delta], say, to 2 [delta] /3. The new value of [delta] applies to the current bit being output, which forms the mth bit of the change-triggering pattern. This approach is called "adaptive delta modulation." Changing [delta], however, is not always an improvement. Indeed, the signal may be such that a closely tracking, but lagging, estimate becomes an oscillating estimate with greater absolute error when adaptive delta modulation is employed. For the signal in Figure 2 with [alpha] =0.1 (too large), and [delta] varying within the limits 0.05 to 0.28 based on m=2, so that two zeros or ones in a row increase [delta], while two different values decrease [delta] -- you obtain the approximation in Figure 3.

Another approach is to allow the sampling interval, [alpha], to change. This is not very useful for hardware, which is more conveniently designed to use a fixed clock rate, but for data compression for digital storage purposes, varying [alpha] may allow fewer bits to be used to encode a given curve. You can increase [alpha] when the previous m bits have alternated in value, but when m 1s or 0s in a row occur, you reduce [alpha] to reestablish the fastest sampling rate. This permits large steps in slowly varying regions, but it allows relatively large deviations in the estimate to occur at turning points where f changes from being flat to sharply rising or falling. Choosing m=2 minimizes this effect, but it is still noticeable. Lagging tracking at turning points is the major flaw in most delta-modulation schemes. The step-function estimate of the signal is shown in Figure 4, where I replaced [alpha] by 1.6 [alpha] up to a limit of 0.41 whenever the previous two bits were the same, and reset [alpha] to 0.05 otherwise. I fixed [delta] =0.1 (which is too small to be completely reasonable for the range of [alpha]). I now have as estimated points: (s0,f(s0)), (t1, [f with hat] (t1)),...,(tn, [f with hat] (tn)) for some irregular mesh s0<t1<...<tn. If you allow [delta] and [alpha] to both vary as described, with [delta] in the interval [0.05,0.28] and [alpha] in the interval [0.05,0.41], you obtain the approximation in Figure 5.

To compute the bit bi, which determines the point (ti, [f with hat] (ti)) when encoding the signal, f, you form an estimate of f(ti), called "Pi," where Pi predicts f(ti), given the previous estimate points (t0, [f with hat] (t0)), (t1, [f with hat] (t1)),...,(ti-1, [f with hat] (ti-1)). Then if Pi is less than f(ti), you output bi=1; otherwise, for Pi [greater than or equal to] f(ti), bi is output as 0.

This same predictor must be used in decoding the bitstring, b, to compute [f with hat]; this is why Pi depends on [f with hat] values, and not on f-values. In this discussion, I've used the simple predictor Pi= [f with hat] (ti-1). Other predictor schemes are possible and may provide better performance, allowing smaller [delta] and/or larger [alpha] values to be used. Of course, other predictors do not necessarily have the property that bi=1 iff [f with hat] (ti-1)<f(ti).

In general, then, the decoding calculation for obtaining [f with hat] (ti) is [f with hat] (ti)=Pi+ [delta] i(2bi-1) for 1 [less than or equal to] i [less than or equal to] n, where [delta] i is the particular value of the increment used when bi is computed; [delta] i is a function of b1,b2,...,bi-1.

You could choose Pi to be determined by extrapolation on the linear or quadratic curve passing through the previous two or three points of the estimate, but experimentation shows the error in this form of predictor to be worse than the simple constant form. A more promising approach is to use an extrapolation on a polynomial that fits the previous k estimate points in the least squares (or better yet, L1-norm) sense. This works adequately, although the predictor is slow to track f on turning intervals. A more elaborate filter predictor might be worth exploring, but weighted averages of the previous k points and weighted averages of the linear predictors based on the k previous points taken two at a time also perform no better than the simple constant predictor. Thus, finding a better predictor seems difficult and the constant form seems to be the most practical choice as well as the simplest.

If a particularly good predictor is used, however, the adaptive variation of [delta] and [alpha] becomes less useful. Indeed, with a perfect predictor Pi=f(ti), the output is all 0 bits, and [alpha] will stay at its minimum, while [delta] stays at its maximum. The resulting [f with hat] curve tracks below f with the maximum allowable error. Even using a merely good predictor means you should sharply decrease [delta] and perhaps increase [alpha].

Examples 2(a) and 2(b) present algorithms for encoding and decoding an adaptive delta-modulation signal with m=2 and with the simple constant predictor function, Pi= [f with hat] (ti-1), previously used. The constants C, [delta] min, [delta] max, g, [alpha] min, and [alpha] max that appear there are global parameters that may be "tuned" to the situation. I have used C=1.5, [delta] min=0.05, [delta] max=0.28, g=1.6, [alpha] min=0.05, and [alpha] max=0.41 in the examples.

There are many variations possible, based on different ranges for [delta] and [alpha], and different formulas for changing them. For example signal f, I actually do about as well with even fewer bits than used above when I let [delta] assume just the values 0.1 and 0.2 and let [alpha] assume just the values 0.1 and 0.2. Another idea is to compute b by the recipe: if f(t)>P-a(log(1+|f1(t)|)/k) then b<-1 else b<-0. This use of slope information can perhaps be marginally useful, even though it produces "lies" about the location of f. Some suggestions have been made that an "intelligent" encoder could hold m signal values in a memory, and carefully compute the best block of one or more output bits based on this look ahead. Perhaps just provisional output bits could be held, so that you would hold m bits in a local memory and output each bit based upon the m-1 future bits that have been provisionally computed and stored, but it seems difficult to make a useful scheme out of this idea.

Also, when you use m>2 to adaptively change [delta] and/or [alpha], you could use a 2m-bit decision tree to pick carefully chosen [delta] and [alpha] modifications; this scheme does work well, but at a high cost in complexity.

The graphs presented here were produced with the Modeling Laboratory (MLAB), a program from Civilized Software for mathematical and statistical modeling (see http://www.civilized.com/). Originally developed at the National Institutes of Health, MLAB includes curve fitting, differential equations, statistics, and graphics as some of its major capabilities. Listing One includes the statements required to do such computer experiments with MLAB. When invoked with an appropriate MLAB Do command, Listing One produces matrices suitable for graphing.

It is worth noting that the step-function approximation form of drawing [f with hat] is somewhat deceiving. A simple straight-line interpolation is a more reasonable presentation. For example, the ( [delta], [alpha])-varying estimate shown in Figure 5 is generated again in Figure 6 using linear connecting segments. Viewing a picture such as this suggests that you might estimate f more exactly if you compute the midpoints of the line segments in the graph of f that cross the graph of [f with hat] . But this set of crossing points is only marginally useful when filtering is used. Generally, when possible, the input, f, should be prefiltered to pass only frequencies below an appropriate cutoff point. In any event, the output points, (t, [f with hat] (t)), have both sampling and encoding error, and the output should be filtered to remove some of this noise. The filtering can be done with a distance-weighted smoothing transformation in software, or with an appropriate low-pass filter in hardware.

Figure 7 shows the smoothed variable [delta] and [alpha] estimate. A doubly smoothed estimate would be even better in regions of slowly changing slope.

DDJ

Listing One

FUNCTION F(T)=IF T<=6 THEN SIN(T)+2*COS(3*T)*EXP(-T) ELSE = J-J*EXP(-2*(T-6));
J = -.27614; MINA = .05; MAXA = .41; MAXD = .28; MIND = .05;
T0 = 0; T1 = 11; D = MIND; A = MINA; G = 1.6; C = 1.5;
TYPE "SET T0,T1,A,D, ETC. AS DESIRED."; DO CONSOLE;

FUNCTION ADM(I)= IF T+A<=T1 THEN (B:=((PV:=P(ME[I]:=OLDP(ADM)))<=F(MT[I+1]:=
(T:=T+A))))+0*(A:=NEWA(X1:=X2,X2:=B))+0*(D:=NEWD()) ELSE 1000-I;

FUNCTION OLDP(B)=PV+D*(2*B-1);
FUNCTION P(H1)=H1;

FUNCTION NEWA(X1,X2)=IF X1=X2 THEN MINA ELSE MIN(G*A,MAXA);
FUNCTION NEWD()=IF X1=X2 THEN MIN(D*C,MAXD) ELSE MAX(D/C,MIND);

X2 = 1; ADM =.5; T = T0; IF T1 <= T0 THEN TYPE ("null interval"); = PV = F(T0);
"PRE-ALLOCATE THE ARRAYS MT, ME.";
MT = 0^^360; ME[360] = 0; MT[1] = T0;MB = ADM ON 1:360;

N = MAXI(MB); IF N >=360 THEN TYPE "TOO MANY POINTS.";

ME(N) = OLDP(B); ME = ME ROW 1:N; MT = MT ROW 1:N; MB = MB ROW = 1:(N-1);
SME = MESH(MT,MT)&'ROTATE(MESH(ME,ME),1); DELETE SME ROW (1,N+N);

ME = MT&'ME; MF = POINTS(F,MT);
"MB IS THE DELTA-MODULATION BIT-VECTOR, MF IS THE SAMPLED SIGNAL POINTS, ME IS
THE ESTIMATED SIGNAL POINTS, AND SME IS THE STEP-FUNCTION ESTIMATE.";

Back to Article

Copyright © 1998, Dr. Dobb's Journal

Back to Table of Contents


Algorithm Alley

Adaptive Delta Modulation

By Gary D. Knott

Dr. Dobb's Journal April 1998

Example 1: Delta-modulation basics.

Back to Article

Copyright © 1998, Dr. Dobb's Journal


Algorithm Alley

Adaptive Delta Modulation

By Gary D. Knott

Dr. Dobb's Journal April 1998

Example 2: Algorithms for encoding and decoding an adaptive delta-modulation signal.

Back to Article

Copyright © 1998, Dr. Dobb's Journal


Algorithm Alley

Adaptive Delta Modulation

By Gary D. Knott

Dr. Dobb's Journal April 1998

Figure 1: Graph of f on the interval [0,11].

Back to Article

Copyright © 1998, Dr. Dobb's Journal


Algorithm Alley

Adaptive Delta Modulation

By Gary D. Knott

Dr. Dobb's Journal April 1998

Figure 2: Specifying [f with hat].

Back to Article

Copyright © 1998, Dr. Dobb's Journal


Algorithm Alley

Adaptive Delta Modulation

By Gary D. Knott

Dr. Dobb's Journal April 1998

Figure 3: Obtaining an approximation.

Back to Article

Copyright © 1998, Dr. Dobb's Journal


Algorithm Alley

Adaptive Delta Modulation

By Gary D. Knott

Dr. Dobb's Journal April 1998

Figure 4: Step-function estimate of the signal.

Back to Article

Copyright © 1998, Dr. Dobb's Journal


Algorithm Alley

Adaptive Delta Modulation

By Gary D. Knott

Dr. Dobb's Journal April 1998

Figure 5: Obtaining an approximation.

Back to Article

Copyright © 1998, Dr. Dobb's Journal


Algorithm Alley

Adaptive Delta Modulation

By Gary D. Knott

Dr. Dobb's Journal April 1998

Figure 6: Generating an approximation using linear connecting segments.

Back to Article

Copyright © 1998, Dr. Dobb's Journal


Algorithm Alley

Adaptive Delta Modulation

By Gary D. Knott

Dr. Dobb's Journal April 1998

Figure 7: Smoothed variable [delta] and [alpha] estimate.

Back to Article

Copyright © 1998, Dr. Dobb's Journal


Algorithm Alley

The Cost of Recursion

Dr. Dobb's Journal June 1998

By Jon Bentley

Jon is a Member of Technical Staff at Bell Labs. He can be contacted at jlb@ research.bell-labs.com.
Sidebar: A Real Program

Starting this month, Jon Bentley will be a regular contributor to "Algorithm Alley" Jon is best known for the "Programming Pearls" column he wrote for Communications of the ACM and later turned into a series of often-recommended books.

This month, Jon takes a hard look at a common tradeoff: Should you optimize your code to be easy to read or to be efficient? Recursive algorithms are often elegant and easy to express, but are less efficient than iterative techniques. Or are they?

As Jon shows, the truth is much more complicated than you might think.

-- Tim Kientzle

Too often, I find myself wrestling with a piece of code, beating at it with while and for loops, all to no avail. The code gets bigger, buggier, and uglier. After hours (or days) of frustration, I finally try recursion and a clean function jumps out in a dozen lines of code.

So why do I make the same mistake over and over again, and blithely ignore recursion? I'll fall back on the old excuse of the way I was raised. I grew up programming in Fortran without recursion. And during my early years of recurring, it was notoriously expensive.

On modern systems, though, recursion can be almost free. It can also be much more expensive (relatively) than it was in the bad old days. I recently confronted this ghost of my childhood by studying the cost of recursion. In this month's "Algorithm Alley," I'll tell the story of my experiments, starting with some examples, and ending with substantial speedups for a new data structure. It is sprinkled along the way with exercises so you can download the programs and try them (details on the program follow shortly).

This analysis has been a wonderfully liberating experience for me. New insights into the cost of recursion allow me to use this powerful tool more comfortably. And when recursion is too expensive, simple techniques let me retrieve the performance while maintaining much of the elegance.

Strings

I'll start with two important functions on strings. The strlen function returns the number of characters in a null-terminated string. The typical implementation is usually along the lines of Example 1(a). The loop increments the pointer s until it finds the null character, incrementing the integer n along the way.

As Example 1(b) shows, you can implement the same function recursively (the leading r is for recursion). The length of a null string is zero; the length of a nonnull string is one more than the length of the substring that omits the first character. The two programs perform a similar sequence of computations: test *s for null, add 1 to s, and repeat. The different mechanisms for repetition can have very different overheads, though.

Exercise 1. Estimate the run times of the two strlen functions on your machine. Assume that the input is a very long string. How will the run times vary across hardware, compilers, and optimization level?

The first two lines of Table 1 show the run time of the two functions across three 200-MHz machines, three different compilers, and two extremes of optimization. The first entry on the first line says that on a Pentium Pro with no optimization, the istrlen function takes about 30n nanoseconds on an input of length n; the second entry says that the optimized version is twice as fast. The two rows together show that recursion is expensive: from factor of four on an (optimized) Pentium Pro to a whopping factor of 29 on an (optimized) UltraSPARC. Caveat hackor!

The next experiment (see Example 2 (a)) implements the strcmp function that compares two strings. The recursive version (Example 2 (b)) peels off the first character of each before recurring.

The third and fourth lines of Table 1 give the coefficient in nanoseconds for comparing two equal strings of length n. That data shows that recursion without optimization is more expensive than iteration by about a factor of five. With the highest level of optimization, though, recursion has exactly the same cost as iteration. The clever compilers noticed that the recursion is always the final call in the function (usually called "tail recursion"), and transformed it to iteration.

When I first implemented the recursive length function, I used C's "?:" conditional expression to yield more succinct (but not necessarily clearer) code; see Example 2 (c).

Exercise 2. Will using a conditional expression change the run time of the program?

I had assumed the answer was no. The final line in Table 1 shows that on the Pentium, the times are similar. On the other systems, though, the difference is enormous. I had originally written most of my recursive functions using conditional expressions; I rewrote the experiments to avoid this vexing anomaly.

Playing with the Program

All of the experiments in this column (and many others not reported here) are contained in the 300-line C program called "recit.c" (for recursion and iteration), which is available electronically; see "Resource Center," page 3. After compiling the program, run it with the single parameter n (500 is a fine starting value). The program produces a single column of Table 1 (reflecting your machine and the optimization level you used on your compiler). The program displays the raw clicks for each operation and the computed coefficient in nanoseconds; double the value of n until the clicks are smooth and the coefficients converge.

Exercise 3. Estimate the cost on your system of the string functions strlen and strcmp. After writing down your guesses (no cheating!), download the program and run it on your machine, the experiment with different levels of optimizations.

Binary Search Trees

The simple functions you've seen so far have illustrated some of the costs of recursion. They show little of its expressive power, though: The problems are easily solved iteratively. I'll move on now to a data structure that needs recursion.

Each node in a binary search tree contains an integer value and two pointers to subtrees; see Example 3 (a). The nodes in the lo subtree have values less than val, while those in the hi subtree have values greater than or equal to val. The root of the tree is Tptr root;. The recursive search function returns 1 if the value t is in the tree and (for similarity to the array search) -1 if the value is not present; see Example 3 (b). The iterative version ( Example 3 (c) is similar.

The first two lines of Table 2 are similar to other functions that employ tail recursion. (The nodes of the tree were inserted in increasing order to form a degenerate tree that requires linear time to search; the entries in the table are still the coefficients in nanoseconds.)

The glory of recursion becomes evident in the insertion function, which is originally called as root = rinsert(root, t);. If p is nonnull, the element is inserted in the correct branch (ties go high). When p is null, it makes a new node, as is shown in Example 3 (d). This recursion gracefully solves the problem of remembering whether to store the new subtree into a lo, hi, or root pointer. To appreciate that service, try doing the job yourself.

Exercise 4. Write an iterative function to insert a new node into a binary search tree.

Example 4, the simplest program I could come up with for this task, is faster than the recursive version, but I find it much less clear. The recursive version ran the first time; the iterative version had several bugs.

In situations like this, an old C trick combines the best of both worlds.

Exercise 5. Use a pointer to a pointer to keep track of where to store the pointer to the new node.

Tree Traversal

As a final exercise, I'll turn to the problem of traversing a binary tree. I'll just scratch the surface of this fascinating topic. If you're interested in more detail, read Donald Knuth's classic "Structured Programming With goto Statements" (Computing Surveys 6, 4, December 1974, pp. 261-301). The rest of this section is a brief sketch of Knuth's study.

Some traversals report the values in sorted order; others verify that the tree is sorted. To keep our traversal simple, we'll just count the nodes in the tree. Example 5(a) is a recursive function for the job.

Since you know that recursion is expensive, you can reduce the number of calls from 2n to n by guarding the recursive call; see Example 5(b). The original rcount first calls the function and then tests whether p is null; rcount2 first tests for null, and then makes the call. For one extra if statement, you get a speedup of between six and 26 percent.

Exercise 6. Function rcount2 is faster than function rcount, but it is potentially buggy. Explain what the problem is and how to avoid it.

The final call in rcount2 is tail recursive. You can therefore replace it with an assignment and a while loop; as in Example 6 (a). The systems usually found this tail recursion; it was a 15 percent speedup on just one system.

Knuth uses several straightforward transformations on rcount3 to remove the first recursion. The resulting icount function is shown in Example 6 (b). It uses its own stack. I found this version much more difficult to understand, so I was delighted that it was never significantly faster than the (singly) recursive function.

Principles

These little experiments have illustrated some big truths about recursion.

  • Recursion is expressive. Some of the recursive tree functions are much clearer than their iterative cousins.
  • Recursion can be very expensive.
  • Tail recursion is sometimes cheap. Smart compilers convert it to iteration.
  • Tail recursion is sometimes very expensive. Converting it by hand to iteration can be very effective.
  • Guarding recursive calls can be effective.
  • Handwritten stacks require lots of code yet yield little (if any) run time. (Guy Steele said it best: "If the programmer can simulate a construct faster than the construct itself, then the compiler writer has blown it badly.")
  • Optimization is usually effective. Enabling compiler optimization can make some little functions an order of magnitude faster. Then again, it makes some functions a bit slower.
  • Timing is hard. The factor of 20 in a conditional expression is just one of many mines in this very large field.

The text box, entitled "A Real Program," shows how these principles were applied to speed up search algorithms for a new data structure.

Solutions to Selected Exercises

The following are solutions for a couple of the exercises: Exercise 5 -- Function iinsertpp uses p as a pointer to the pointer that is assigned a new node. Note that it is almost as fast as the iterative version; see Listing One. Exercise 6 -- You must guard the first call: if (root) rcount2(root);.

DDJ

Listing One

  void iinsertpp(int t)
  {   Tptr *p = &root;
      while (*p) {
          if (t < (*p)->val)
              p = &((*p)->lo);
          else
              p = &((*p)->hi);
      }
      *p = (Tptr) malloc(sizeof(Tnode));
      (*p)->lo = (*p)->hi = 0;
      (*p)->val = t;
   }
Copyright © 1998, Dr. Dobb's Journal

Back to Table of Contents


Algorithm Alley

The Cost of Recursion

By Jon Bentley

Dr. Dobb's Journal June 1998

(a)
int istrlen(char *s)
{   int n = 0;
    for ( ; *s; s++)
        n++;
    return n;
}
(b)
int rstrlen(char *s)
{   if (*s) return 1+rstrlen(s+1);
    else return 0;
}

Example 1: (a) Typical implementation of strings; (b) implementing the function recursively.

Back to Article

Copyright © 1998, Dr. Dobb's Journal


Algorithm Alley

The Cost of Recursion

By Jon Bentley

Dr. Dobb's Journal June 1998

(a)
int istrcmp(char *s, char *t)
{   for ( ; *s == *t; s++, t++)
    if (*s == 0)
        return 0;
    return *s - *t;
}
(b)
int rstrcmp(char *s, char *t)
{   if (*s == *t) {
        if (*s)
            return rstrcmp(s+1, t+1);
        else
            return 0;
    } else
        return (*s - *t);
}
(c)
int rcstrcmp(char *s, char *t)
{   return (*s == *t)
           ? (*s ? rcstrcmp(s+1, t+1) : 0)
           : *s - *t;
}

Example 2: (a) strcmp function that compares two strings; (b) recursive version; (c) implementing the recursive length.

Back to Article

Copyright © 1998, Dr. Dobb's Journal


Algorithm Alley

The Cost of Recursion

By Jon Bentley

Dr. Dobb's Journal June 1998

(a)
typedef struct tnode *Tptr;
typedef struct tnode {
    int val;
    Tptr lo, hi;
} Tnode;
(b)
int rbstsearch(Tptr p, int t)
{  if (!p)
       return -1;
   if (t == p->val)
       return 1;
   if (t < p->val)
       return rbstsearch(p->lo, t);
   return rbstsearch(p->hi, t);
}
(c)
int ibstsearch(int t)
{  Tptr p;
   for (p=root; p; ) {
       if (t == p->val)
           return 1;
       p=(t<p->val) ? p->lo : p->hi;
   }
   return -1;
}
(d)
Tptr rinsert(Tptr p, int t)
{   if (p) {
        if (t < p->val)
            p->lo = rinsert(p->lo, t);
        else
            p->hi = rinsert(p->hi, t);
    } else {
        p = (Tptr) malloc(sizeof(Tnode));
        p->lo = p->hi = 0;
        p->val = t;
    }
    return p;
}

Example 3: (a) Each node in a binary search tree contains an integer value and two pointers to subtrees; (b) recursive search function returns 1 if the value t is in the tree, -1 if the value is not present; (c) iterative version; (d) when p is null, it makes a new node.

Back to Article

Copyright © 1998, Dr. Dobb's Journal


Algorithm Alley

The Cost of Recursion

By Jon Bentley

Dr. Dobb's Journal June 1998

void iinsert(int t)
{   Tptr p, q;
    Tptr next = (Tptr) malloc(sizeof(Tnode));
    if (!root)
        root = next;
    else {
        for (q = root; q; ) {
            p = q;
            q = (t < p->val) ? q->lo : q->hi;
        }
        if (t < p->val)
            p->lo = next;
        else
            p->hi = next;
    }
    next->lo = next->hi = 0;
    next->val = t;
}

Example 4: Iterative function to insert a new node into a binary search tree.

Back to Article

Copyright © 1998, Dr. Dobb's Journal


Algorithm Alley

The Cost of Recursion

By Jon Bentley

Dr. Dobb's Journal June 1998

(a)
void rcount(Tptr p)
{   if (!p) return;
    rcount(p->lo);
    count++;
    rcount(p->hi);
}
(b)
void rcount2(Tptr p)
{   if (p->lo) rcount2(p->lo);
    count++;
    if (p->hi) rcount2(p->hi);
}

Example 5: (a) Counting nodes in the tree; (b) reducing number of calls from 2n to n.

Back to Article

Copyright © 1998, Dr. Dobb's Journal


Algorithm Alley

The Cost of Recursion

By Jon Bentley

Dr. Dobb's Journal June 1998

(a)
void rcount3(Tptr p)
{   while (p) {
        if (p->lo) rcount3(p->lo);
        count++;
        p = p->hi;
    }
}
(b)
void icount(Tptr p)
{   Tptr s[MAXN];
    int sp = 0;
    for (;;) {
        while (p) {
            s[sp++] = p;
            p = p->lo;
        }
        if (sp == 0) break;
        p = s[--sp];
        count++;
        p = p->hi;
    }
}

Example 6: (a) Final call in rcount3 is tail recursive; (b) resulting icount function uses its own stack.

Back to Article

Copyright © 1998, Dr. Dobb's Journal


A Real Program

Robert Sedgewick and I described ternary search trees in the April 1998 issue of DDJ. When we worked on the data structure, we were both pleasantly surprised to find that a recursive search function was as fast as an iterative function (under optimizing compilers). The run times for the two functions are shown in the first two rows of Table 3. We therefore assumed that our other recursive search functions were competitive with the iterative versions. After performing the experiments in this article, I decided to go back and test the assumption.

The pmsearch function in (Listing Two) performs a partial-match search in a tree. The query can include dots as "don't care" characters, so ".at" matches bat, cat, and a dozen other three-letter words. pmsearch calls itself in three distinct places, and returns to do more work after two of those calls.

The function pmsearch2 is in Listing Three. It is almost twice as long as function pmsearch, but only a bit harder to understand. The new code has two primary blocks, depending on whether the current character is a dot. Both of the cases ended with a tail recursion, so I explicitly transformed that to a while loop. The speedups ranged from a third to a factor of three, which I felt was worth a dozen lines of code and a couple hours of programming. I took a similar approach to a function for near-neighbor searching in ternary search trees, with similar results in code length and run time. The code for these experiments is available electronically; see "Resource Center," page 3.

-- J.B.

Back to Main Article

Listing Two

  void pmsearch(Tptr p, char *s)
  {   if (!p) return;
      nodecnt++;
      if (*s == '.' || *s < p->splitchar)
          pmsearch(p->lokid, s);
      if (*s == '.' || *s == p->splitchar)
          if (p->splitchar && *s)
              pmsearch(p->eqkid, s+1);
      if (*s == 0 && p->splitchar == 0)
          srcharr[srchtop++] =
              (char *) p->eqkid;
      if (*s == '.' || *s > p->splitchar)
          pmsearch(p->hikid, s);
  }

Back to Text

Listing Three

 void pmsearch2(Tptr p, char *s)
 {   while (p) {
         nodecnt++;
         if (*s == '.') {
             pmsearch2(p->lokid, s);
             pmsearch2(p->hikid, s);
             if (p->splitchar == 0) return;
             p = p->eqkid;
             ++s;
         } else {
             if (*s < p->splitchar)
                 p = p->lokid;
             else if (*s > p->splitchar)
                 p = p->hikid;
             else /* *s == p->splitchar */ {
                 if (*s == 0) {
                     if (p->splitchar == 0)
                         srcharr[srchtop++] =
                             (char *) p->eqkid;
                     return;
                 }
                 p = p->eqkid;
                 ++s;
             }
         }
     }
 }
Copyright © 1998, Dr. Dobb's Journal


Algorithm Alley

The Cost of Recursion

By Jon Bentley

Dr. Dobb's Journal June 1998

Table 1: Run times of string functions in nanoseconds.

Back to Article

Copyright © 1998, Dr. Dobb's Journal


Algorithm Alley

The Cost of Recursion

By Jon Bentley

Dr. Dobb's Journal June 1998

Table 2: Run times of binary search tree functions in nanoseconds.

Back to Article

Copyright © 1998, Dr. Dobb's Journal


Algorithm Alley

The Cost of Recursion

By Jon Bentley

Dr. Dobb's Journal June 1998

Table 3: Run times of ternary search tree functions.

Back to Sidebar

Copyright © 1998, Dr. Dobb's Journal


Data Structures Using C: Index

! Terms

A Terms

B Terms

C Terms

D Terms

E Terms

F Terms

G Terms

H Terms

I Terms

J Terms

K Terms

L Terms

M Terms

N Terms

O Terms

P Terms

Q Terms

R Terms

S Terms

T Terms

U Terms

V Terms

W Terms

X Terms

Y Terms

Z Terms


INDEX: ! Terms

No ! Terms


INDEX: A Terms

Abstraction

Abstraction

Abstraction

Abstraction

Access

Access

direct

indexed sequential

random

random

sequential

sequential

sequential

access_succ

Adaptable

addrecord

addrecord

Address

calculation without access

Adelson-Velskii, G.M.

Aho, A.V.

Algorithm

Algorithm

Allocation

Allocation

Allocation

Allocation

Alphabetic codes

Analysis of programs

anotherprimes

anotherrecord

anotherrecord

anotherrecord

anotherrecord

Antisymmetric array

Array

Array

antisymmetric array

diagonal entries

index

of pointer variable

of pointer variable

symmetric

symmetric

tridiagonal

two-dimensional

two-dimensional

ASCII

Asymmetry

Asymptotic analysis

avail

avail

avail

avail

avail

avail

avail

avail

Available space list

Available space list

Availist

Average

amount of compression

depth

length

search time

AVL trees


INDEX: B Terms

B-tree

Backtracking

Backtracking

Backtracking

Bag

baginitialization

bal

Balance

Balance

access by order

Balanced binary trees

Balanced binary trees

Barron, D.W.

Basic operation

Baumert, L.D.

Bayer, R.

Bell, J.R.

belongs

belongs

belongs

belongs

Ben-Ari, M.

Benchmark figure

Berry, J. T.

Best fit

Binary relation

Binary search trees

Binary search trees

extended

greedy

nearly optimal

optimal

simple

Binary search

Binary search

timing

Binary trees

Binary trees

2-3 tree

balanced

complete

full

implementation

link inverted

optimal

Robson

threaded

traversal

binarysearch

Bird, R.S.

Blocks

Bowling scores algorithm

Bowling scores

Branch and bound traversal

Branches

Breadth-first traversal

Bubble sort

Bubble sort

timing

bubblesort

Bucket

Buddies

Buddy system

Buffer area

Buffer

Buneman, P.


INDEX: C Terms

Case studies

bowling scores

checking sequences for proper nesting

information retrieval

merging and the perfect shuffle

stable marriage

Chaining

Chaining

Chains

Changing programs

check

check

check

check

check

check

Circular lists

Circular lists

Circular lists

Coalesced chaining

Coalescing

Coding

alphabetic

ASCII

EBCDIC

fixed-length

Huffman

Morse

variable-length

collection.c

collection.c

collection.h

collection.h

Collision resolution policy

Collision

Columnwise representation

Command languages

Communication between functions

Compaction

Complete binary tree

Complex record

complex

complex

Composition of filters

Compression of text

Consistent ranking

Constant time

Construction of a solution

Construction of a solution

Construction of a solution

Construction of a solution

Construction of a solution

Constructs

complex

decision

decision

loop

loop

modification

modularization

sequence

sequence

copy

in heapsort

in topsort

list

list

Copying a list

copylist

count

Counting squares

countprimes

Crane, C. A.

create

in binary tree representation

list-structures

primes

primes

primes

primes

primes

create_priorities

create_priorities

create_womens_mates

create_womens_mates

createnode

createnode

createnode

createrecord

createtree

Cylinder

Cylinder

index


INDEX: D Terms

Dahl, O.

Dangling reference

Data abstraction

Data abstraction

Data abstraction

Data abstraction

Data abstraction

Data abstraction

Data abstraction

Data abstraction

Data abstraction

Data abstraction

Data abstraction

Data abstraction

Data for recursive programs

Data representation

Data structure

Data structure

Data structure

Data structure

Data type

Debugging

Decision construct

Decision construct

decrease

Deen, S.M.

Defensive programming

delete

lists

primes

primes

primes

primes

primes

Deletion

B-tree

balanced binary search tree

binary search tree(simple)

binary tree

hash table

lists

lists

depth

depth

Depth-first traversal

Depth-first traversal

Depth

of a binary tree

of a binary tree

of a recursive program

versus capacity

Deque

determine

determine

Diagonal entries

Dijkstra, E.W.

Dijkstra, E.W.

Direct access

Direct recursion

Directories

Disk pack

Displaced linear probing

Distribution

Distributive partitioning

Dobosiewicz, W.

Documentation

Double hashing

Driver

Dummy record

Dynamic memory

Dynamic memory

Dynamic memory

Dynamic memory

Dynamic memory

list

Dynamic storage management

Dynamic storage management

Dynamic structure

Dynamic structure


INDEX: E Terms

EBCDIC

Echo print

Elementary searches

Elementary searches

binary search

efficiency comparisons

interpolation search

linear search

Elementary sorts

Elementary sorts

attempted improvements

bubble

insertion

maximum entry

modified insertion

Elements of style

empty, queue

empty, stack array implementation

empty, stack array implementation

in binary tree creation

in list-structure creation

in n-queens

in towers

in towers

in towers

empty, stack list implementation

empty, stack list implementation

in towers

emptybag

Encapsulation

Entire pairing

Entries

EOF

evaluate

Evaluation of expressions

Execution of recursive programs

Execution of recursive programs

Execution of recursive programs

Execution tree

Execution tree

exit

expand

Expression

completely parenthesized

evaluation

translation

Extended binary search tree

Extern

External fragmentation

External memory

External node


INDEX: F Terms

f

f

Fagin, R.

FALSE

Family relationships

fclose

Feasible

feasible

Feasible

Fibonacci

generalized numbers

numbers

numbers

trees

trees

Fibonaccitree

Fibonaccitree

Field

Field

offset

FIFO

File pointer

FILE

File

File

modularity

object code

organization

sequential

single source code

Filter

Finalization

finalize

finalize

finalize

finalize

First fit

Fixed-length code

Floppy disk

Flow of control

fopen

fprint

Fragmentation

framecalc

framecalc

fread

free

free

fscanf

Full binary tree

Functional modularization

Functional modularization

Functional modularization

Functional modularization

Functional modularization

Functional modularization

Functional modularization

Functional modularization

fwrite


INDEX: G Terms

gamecalc

gamecalc

Garbage

Garbage

collection

generation

Garwick, J.V.

General list traversal

Generalized Fibonacci numbers

Generalized inorder traversal

getnextrecord

getnextrecord

Global variable

Global variable

Golomb, S.W.

Graph

Greedy algorithm

Greedy binary search tree

Greedy construction

Greedy implementation


INDEX: H Terms

Hard disk

Harrison, S.

Hash table

Hash trees

Hashing function

Header record

Heap

Heap

Heap

creation

implementation

management

heapsort

Heapsort

Hierarchical relationship

High-level computer

High-level input/output

High-level languages

Hoare, C.A.R.

Hopcroft, J.E.

Horibe, Y.

Horowitz, E.

Horowitz, E.

Huffman

algorithm

coding

proof

trees

Hull, M.


INDEX: I Terms

Imbalance

Implementable

Implementation independence

Implementing a data abstraction

Implementing recursive programs

Inclusion relationship

increase

Independence

Index

array

cylinder

master

Indexed sequential access

Indexing methods

Indirect recursion

Infix expressions

Infix expressions

Infix expressions

Infix notation

info

info

Information field

Information retrieval

initial

initial

Initialization

initialize

lists

lists

lists

lists

lists

lists

lists

permutations

permutations

Inorder traverse

Input buffer area

Input validation module(topsort)

Input/output

insert

lists

primes

primes

primes

primes

queues

topsort

insertavl

nonrecursive

recursive

insertinorder

Insertion sort

Insertion sort

modified

timing

Insertion

B-tree

balanced binary search tree

binary search tree(simple)

hash table

in binary tree

lists

prime numbers

priority queues

queues

insertionsort

insertnode

interchange

Internal fragmentation

Internal memory

Internal node

Interpolation search

Irreflexivity

ISAM

iszero

item, array implementation

item, array implementation

in binary tree creation

in n-queens

Iterative


INDEX: J Terms

Job control languages


INDEX: K Terms

k-ary trees

Kelley, A.

Kernigan, B.W.

Known nodes

Knuth, D.E.

Knuth, D.E.

Knuth, D.E.

Knuth, D.E.

Knuth, D.E.

Knuth, D.E.

Knuth, D.E.

Knuth, D.E.

Knuth, D.E.

Korsh, J.F.

Korsh, J.F.


INDEX: L Terms

Landis, Y.M.

Lapin, J.

Larson, P.

last

last

Latency time

Leaf node

Left successor

left

left

left

left

left

left

Leftist trees

leftparen

leftparen

length

Length

of a list

record

Levy, L.

LIFO

LIFO

Linear probing

Linear probing

Linear search

linearsearch

Link field

Link inversion

Linked representation

List pointers

List reversal

List

List

List-structure

List-structure

implementation

representation for binary trees

List

chain

circular

circular

circular

complex

example

head

implementation

in arrays

in languages without records

name

of available records

of available space

of available space

two-way

versus sequential arrays

why needed

Litwin, W.

ll

ll

Local copy

Local minimal triple

Local variable

Local variable

Loop construct

Loop construct

Low-level input/output

lr

lr


INDEX: M Terms

Magnetic disk

Magnetic disk

Magnetic tape

Main memory

main

AVL tree creation

binary tree creation

bowling scores

bowling scores

driver for check

driver for perfect shuffle

driver for perfect shuffle

driver for permutations

driver for permutations

driver for towers

driver for towers

driver for towers

files

list creation

list creation

list-structure creation

merging files

n-queens

primes

primes

primes

primes

stabilitycheck

stabilitycheck

topological sort

Maintenance

malloc

malloc

malloc

Master index

match

match

Matched characters, pairs

Matched characters, pairs

mate

mate

mate

Max heap

max

maxentrysort

Maximum entry sort

maxloc

McCreight, E.M.

Mehlhorn, K.

Member

Memory hierarchy

Merge sort

Merge sort

natural

polyphase

replacement-selection

straight

merge

Merging

Merging

Merging

Methodology

Methodology

Miller, L.

Min heap

Mod

Modification

Modified insertion sort

Modularization

Modularization

Modularization

Modularization

Modularization

Module

object

source

Morse code

mostpreferred

mostpreferred

mostpreferred

move

move

Multiarray implementation


INDEX: N Terms

n-queens

Natural merge

Nearly optimal binary search tree

Nearly optimal binary search tree

Nearly(reverse) sorted

Negative infinity

Nemetz, T.

Nesting

next

list creation

list-structure creation

lists

lists

permutations

permutations

permutations

permutations

permutations

permutations

permutations

topsort

nextpair

nextpreferred

nextpreferred

nextpreferred

nextsubtree

nextsubtree

nextsubtree

nextsubtree

Nievergelt, J.

Node

Node

node.p

Node

external

internal

leaf

terminal

Nonlocal variable

nonprimes

nonprimes

nonprimes

nonprimes

Normalized

Notation

infix, prefix, postfix

infix, prefix, postfix

parenthesis-free

Null list

Null pointer

NULL

NULL


INDEX: O Terms

O( )

Object code file

obst

Offset field record

omit

omit

omit

omit

omit

Open addressing

Operating systems

Optimal binary search tree

Optimal binary search tree

Optimal binary tree

Optimal binary tree

Order m(B-tree)

Order( )

Ordered forest

Ordered hash table

Ordered trees

Ordered trees

Ordered trees

orderedinsert

original

original

Overflow

area

overflow

in binary tree creation

in list structure creation

in n-queens

in towers

in towers

in towers

in towers

Overflow

stack


INDEX: P Terms

Packaging

Packaging

compilation

compilation

compilation

compilation

linker

linker

linker

Paired parentheses

Paired parentheses

Palindrome

paragraphinsert

paragraphinsert

Parameters, passing

Parentheses-free notation

Partial order

Partition function

partition

Partitioning entry

Path length

of a node

of a node

weighted

Perfect shuffle

Perfect shuffle

array implementation

list implementation

perfectshuffle

perfectshuffle

Permutations

permutations

permutations

permutations

Pippenger, N.

Plus infinity

Pohl, I.

Pointer

Pointer

Pointer

array

array

array

null

relative

variable

variable

Polyphase

pop, array implementation

pop, array implementation

pop, array implementation

in binary tree creation

in list-structure creation

in n-queens

in towers

in towers

in towers

pop, list implementation

pop, list implementation

in towers

in towers

in towers

Postfix expression

Postfix expression

Postorder traverse

binary tree

tree

Postponed obligations

Postponed obligations

power

Pratt, T.W.

Precedence values

precedes

Predecessor

predinitialization

prefers

prefers

prefers

prefers

Prefix

Preorder traverse

binary tree

tree

preorder, nonrecursive version

saving path to the root

saving right pointers

preorder, recursive version

preorder, recursive version

preorder, stackless traversals

Primary clustering

Primary key field

Prime area

Prime number

primes

primes

primes

primes

primes

primes

primes

complete program

complete program

print

print

print

print

printfield

printnode

printrecord

printrecord

printrecord

printtree

printtree

Priorities

Priority queue

Probe sequence

process

binary trees

binary trees

binary trees

list-structures

list-structures

lists

lists

lists

lists

lists

lists

n-queens

permutations

permutations

topsort

processrecord

Program

adaptable

documentation

maintenance

recursive

reliable

segments

structure

structure

structure

understandable

Programming structure

Programming

debugging

defensive

environment

methodology

methodology

structured

style

style

style

top-down

push, array implementation

push, array implementation

push, array implementation

in binary trees

in list-structure creation

in n-queens

in towers

in towers

in towers

stackless traversals

stackless traversals

stackless traversals

push, list implementation

push, list implementation

in towers


INDEX: Q Terms

Quadratic residue

queens

Queue

Queue

Queue

implementation

implementation

priority

Quicksort

quicksort

iterative

recursive

Quilici, A.


INDEX: R Terms

Random access memory

Random access structure

Random access

Random access

Random hashing

ranf

Read/write head

read_mens_mate

read_mens_mate

read_preferences

read_preferences

readchar

readlist

Rebalance

reclaim

Reclamation

Reclamation

Record

Record

complex

complexity

offset

representation in arrays

simple

storing

union

union

variable-length

variable-length

Recursion

Recursion

Recursive language

Recursive programs

call

data

depth

execution

execution

execution

implementation

scratchpad

simulation

verification

Reference counter method

Reference counters

Reference counters

Refine

Refine

Refinement

Refinement

Refinement

Reheaping

Reheaping

Relative pointer

Reliable

remove

primes

primes

primes

primes

primes

primes

queue

topological sort

remove_driver

replacefirstterminal

Replacement-selection

replacenode

Representation

Huffman trees

pointer array

records in arrays

two-dimensional arrays

reset1balance

reset2balances

resetpathbalances

restore

restore

restore

restore

Right successor

right

right

right

right

right

right

rightparen

rightparen

Ritchie, D.

rl

rl

rng

Robson traversal

Root

rotateleft

rotateleft

rotateright

rotateright

rotateright

Rotation(left and right)

Rowwise representation

rr

rr

Run length


INDEX: S Terms

s_tack

s_tack

s_tack

s_tack

Sahni, S.

Scratchpad

Search key

search

Searching

Searching

Searching

B-tree

binary search trees

binary search trees

binary

comparative

deletion

distributive

efficiency

hash table

hash table

insertion

interpolation

linear

ternary

Secondary clusters

Secondary clusters

Secondary storage

Sedgewick, R.

Seek time

Selection

Selection

Self-reference

Separate chaining

Sequence construct

Sequence construct

Sequential access

Sequential access

in external memory

in external memory

Sequential array methods

Sequential arrays versus lists

Sequential file

Sequential representation

set

set

set

set

set

setinfo

setinfo

binary tree creation

binary tree creation

lists

lists

lists

topsort

towers

setleft

setleft

setlink

lists

lists

lists

lists

topsort

setnull

binary trees

lists

lists

lists

lists

stackless traversals

stackless traversals

stackless traversals

topsort

setqueue

setright

setstack, array implementation

setstack, array implementation

in binary tree creation

in list-structure creation

in n-queens

in stackless traversals

in stackless traversals

in stackless traversals

in towers

in towers

in towers

setstack, list implementation

setstack, list implementation

in towers

setsublist

setvar

setvar

setvar

setvar

Shared storage

Shared storage

Shared storage

Sharing

shift

Siblings

Side effect

Simple record

simple

Simulation

Simulation

Simulation

Simulation

sorts

Single source code file

Single-array implementation

Sorenson, P.G.

Sorenson, P.G.

Sort key

Sort

Sorting

Sorting

bubble

distributive partitioning

efficiency

heapsort

insertion

maximum entry

merge

natural merge

polyphase

quicksort

replacement-selection

simulation

straight merge

tape files

topological

Sparse binary tree

stabilitycheck

stabilitycheck

Stable marriages

Stable marriages

Stable marriages

Stack

Stack

Stack

Stack

array implementation

example

list implementation

overflow

underflow

Stackless traversals

Standish, T.A.

Standish, T.A.

Static structure

Static structure

Steele, G.

Stone, H.

Storage

limits

management

reclamation

reclamation

requirements

shared

shared

shared

Straight merge

Strong, R.H.

Strongly typed languages

Structure

Structure

Structure

Structure

dynamic

program

program

program

record

static

Structured programming

Structured programming

Stub

Style

Style

Sublist members

sublist

sublist

Subtrees

Subtrees

most recent

Successor

succinitialization

Symmetric array


INDEX: T Terms

Task

Terminal node

terminalcount

Ternary search

Threaded binary trees

Threads

Time

constant

latency

limits

requirements

seek

Timing

binary search

bubble sort

heap creation

insertion sort

quicksort

reheaping

Top-down design

Top-down programming

Top-down programming

Topological sorting

Topological sorting

analysis

better implementation

better refinement

construction-based algorithm

first refinement

initial implementation

search-based algorithm

topsort

Towers of Hanoi

Towers of Hanoi

Towers of Hanoi

iterative algorithm

recursive algorithm

towers

towers

towers

towers

towers

towers

towers

towers

towers

implementation

Trading storage for time

Transitivity

Translation

of expressions

of infix to postfix

Traversal

2-3 tree

arrays

arrays

arrays

arrays

B-tree

backtracking

backtracking

backtracking

binary trees

branch and bound

breadth-first

depth-first

depth-first

link inversion

list-structure

lists

lists

lists

lists

postorder

postorder

preorder, inorder

preorder, inorder

preorder, inorder

Robson

stackless

threaded

trees

trees

variable-length records

variable-length records

traverse

array of variable length records

array of variable length records

list-structures

list-structures

list-structures

lists

lists

lists

Trees

Trees

Trees

as binary trees

as binary trees

as binary trees

balanced

balanced

binary search

binary

Fibonacci

Fibonacci

general

greedy

leftist

link inverted

name

nearly optimal binary search

optimal binary search

optimal binary

ordered

ordered

ordered

Robson

threaded

traversal

traversal

treesort

Tremblay, J.P.

Tremblay, J.P.

Tridiagonal array

Triple

Triple

local minimal

TRUE

Two-dimensional array

Two-dimensional array

Two-dimensional array

Two-way circular list

Two-way list

Type


INDEX: U Terms

Ullman, J.D.

Ullman, J.D.

underflow

in binary tree creation

in list-structure creation

in n-queens

in towers

in towers

in towers

in towers

Union

Union

UNlX-type input/output

update

update

update

updatencount

Usage ratio


INDEX: V Terms

Valid sequence

Valid sequence

Valid sequence

Value of a node

Variable-length

code

records

records

Variables

Variables

global

global

local

local

nonlocal

Verification

Verification

Verification

Virtual sequential access

VSAM


INDEX: W Terms

Walsh, T.R.

Weighted path length

Wirth, N.

Wirth, N.

Wirth, N.


INDEX: X Terms

No X Terms


INDEX: Y Terms

No Y Terms


INDEX: Z Terms

No Z Terms


Data Structures Using C: PREFACE

PREFACE

One must learn by doing the thing, for though you can think you know it you have no certainty until you try it.

Sophocles

Data Structures, Algorithms, and Program Style Using C is intended for use in a second programming course, such as CS2 of the ACM curriculum, as well as in a Data Structures course. It will also be invaluable for computer professionals who wish to develop applications using the C programming language. All programs appear in C, although the concepts and methods presented are independent of the language used. The book emphasizes current techniques for the creation and testing of programs and stresses the importance of writing programs that are general enough to be readily adapted for use in the solution of complex problems. Data abstraction and modularity are stressed throughout. Complex concepts are presented in a concise, informal way; theory is coupled with concrete examples.

The book is written to provide a significant amount of flexibility in the order in which the material is covered, as illustrated in the accompanying chart. For a second programming/introductory data structures class, the fundamental concepts of structured programming and data structures are presented in Chapters 1 through 7. These chapters, combined with selected topics from Chapters 8 and 9, provide a challenging course. Chapters 8 through 13 deal with the application of the concepts to more complex problems and also introduce some advanced data structures. Where data structures are taught after the second programming course, selected material from Chapters 1 through 8 can serve as a quick review of the basics, followed by an in-depth study of Chapters 9 through 13. Care has been taken throughout to include sets of exercises that are graduated by degree of difficulty and complexity. Also included are suggested assignments that incorporate the essential concepts of each chapter.

Just as knowing the grammar of a natural language does not ensure quality writing, knowing the grammar of a programming language does not ensure quality programs. Learning to program well entails utilizing principles of good style and requires skill in the use of a wide variety of tools. It is an advanced task, beyond simply learning the grammar of a language. A master mechanic, upon opening the hood of an automobile, does not see a jumble of wires and metal, but rather essential components linked together and aligned in a systematic fashion. A knowledgeable programmer must learn not only to write programs but to read them in the same fashion as a master mechanic looks at an automobile engine. He or she has to be able to see the major components and linkages of the program. Program structure can enhance or obscure a program's intelligibility.

This book explains and demonstrates the tools needed to craft good programs and to solve substantial problems. These tools, although few in number, must be used with care and forethought. The tools include the basic paradigms for problem solving, for structuring programs, and for structuring data. Their proper use involves data abstraction, modularization, and the appropriate choice of data structures. Because the book emphasizes the entire program development process, readers will learn how to write well-designed programs and how to recognize them. They will, at the same time, develop insight into program analysis and learn how to analyze programs in order to determine their correctness and efficiency.

* Heapsort requires Chapter 7.

Many individuals contributed to this book. Special thanks are due to Sharon Doherty, Mary Helen Fein, Giorgio Ingargiola, and Paul LaFollette for their help, as well as to the students who labored through the early versions of the manuscript. The staff of PWS-KENT Publishing Company, including Bob Prior and Elise Kaiser, all provided valuable support and guidance. We would also like to thank David Hoyt for help during the production process. It is also a delight to acknowledge the painstaking and helpful work of the reviewers: James Calhoun,Western Illinois University; Michael Driscoll, DePaul University; Terry Hamberger, York College of Pennsylvania; Marek Holynski, Boston University; Sam C. Hsu, Florida Atlantic University; Paul W. Ross, Millersville University of Pennsylvania; Shai Simonson, University of Illinois at Chicago; John B. Tappen, University of Southern Colorado; and Therril Valentine, McNeese State University. Finally, and certainly most important, we deeply appreciate and thank our wives Nina and Judy and our children, who suffered from us and with us, avoided us and supported us, during the time this book was being written.

James F. Korsh

Leonard J. Garrett


Data Structures Using C: CHAPTER 1: PROGRAMMING STRUCTURE

CHAPTER 1: PROGRAMMING STRUCTURE

Emphasizes the importance of

structuring programs

using an explicit problem solving methodology and top-down design to create programs

good program style

Introduces and illustrates the components of good programming style

data abstraction

functional modularization

verification and debugging

documentation

Begins the discussion of program time and storage requirements and how to determine them

Case study�scoring bowling games

illustrates the concepts introduced in the chapter

1.1: Structure: Data and Program

1.2: A Look Ahead

1.3: The Need for Structure

1.4: Basic Constructs

1.5: Top Down Design

1.6: High-Level Languages

1.7: Elements of Style

1.8: Time and Storage Requirements

1.9: Case Study: Bowling Scores

1.10: Summary

Exercises

Suggested Assignments


Data Structures Using C: 1.1: Structure: Data and Program

1.1: Structure: Data and Program

Two programs doing exactly the same task may look entirely different. One may be highly readable, concise, and easily adjusted to carry out related tasks. The other may be impenetrable, lengthy, and difficult to modify. The same two may differ so much in execution time and storage needs that one program executes while the other aborts, having exceeded time or space restrictions. Experience has shown such differences to be traceable to choice of program structure and data structure.

Programs are written to solve real problems. Program structure breaks the problem and its solution down into simpler, more easily understood parts. The information to be processed is stored in data structures (arrays, records, lists, stacks, trees, and files) provided by the programming language. A data structure groups data. The right data structure for an operation can make it simple and efficient; the wrong one can make the operation cumbersome and inefficient.

Data structures contain information and are operated on during the execution of a program. Programs are said to process information when, in fact, they process data structures. Thus it is not surprising that data and program structure are important and need to be properly related to achieve the goal of successful programming. How this is done is the subject of this book. The story is fascinating and sometimes involved, but it has a happy ending: you master practical programming techniques. These techniques provide the programmer with the means to keep a tight rein on programs so that they remain understandable, maintainable, and efficient, even as they grow in size.

There is structure, then, in both the data and the program itself, and both program and data structure must be appropriate to their tasks. This book presents data structures, not as an isolated theoretical subject, but as an essential object of the problem-solving process leading to the creation of a good program.Three main themes are stressed.

1. Data structures and their impact on programs

2. Programming methodology and its impact on programs

3. Consolidation and extension of knowledge of a high-level language

Data structures are important because the way the programmer chooses to represent data significantly affects the clarity, conciseness, speed of execution, and storage requirements of the program. The text shows how to use particular data structures to create correct and efficient programs. You will see that the operations to be performed on the data determine the best data structures to use.

Developing programs is difficult, especially when done without good guidelines. Creating programs entails solving problems. Top-down programming, another name for structured programming, makes program development easier and leads to well-structured programs. It is the controlled breaking down of a complex task into simpler ones whose individual solutions easily combine to carry out the original task. This process gives structure to the development of programs (hence its name). The top-down, or structured, approach, combined with good programming style, is emphasized in this text. Elements of good programming style include the use of data abstractions and functional modularization, which significantly enhance the readability and changeability of programs.

Data abstraction treats a collection of data by abstracting its important aspects while ignoring as much detail as possible. A data abstraction reduces data to a collection, and to those processing operations directly allowed on that collection. The effect is as though the collection were in an impenetrable black box, the only access to the contents of the box being through the invocation of one or more of the allowed operations. How the collection is actually stored within the box, and how the operations are actually performed, become irrelevant details. Such details determine the efficiency of a program but do not affect its logic.

Functional modularization is the use of a special program unit to carry out a task. In FORTRAN such a unit would be a function or subroutine, in Pascal a function or procedure, in COBOL a paragraph, and in C a function. These program units are used to carry out meaningful processing tasks. C programmers use the word modularization in two ways. One means functional modularization, and the other has to do with packaging a data abstraction using one or more related files. When the first meaning is intended, this book uses functional modularization. The second interpretation appears only in Chapter 5 and will be addressed there.

Practice may not make perfect but seems to be the best way to make problem solvers and programmers out of computer science students. As a student you must repeatedly practice creating programs, because repetition is what facilitates learning. As in learning to dance, draw, or drive, no matter how closely you watch others perform, you can develop skills only by practice.

Finally, whatever high-level programming language you use, studying and understanding the material in this text will enhance your knowledge of that language and your ability to use it effectively. It is assumed that you are familiar with some high-level programming language. Such languages share many features, but each is unique. Although this text occasionally indulges in comparative investigations, all programs herein are written in VAX C, version 2.2, and run on a VAX 11/780 under VMS.

Horowitz [1983] is a collection of reprints of the landmark papers in programming languages. For background in C, see the introductory texts by Kelley and Pohl [1986], Miller and Quilici [1986], and the classic book by Kernighan and Ritchie [1978]. A good reference is Harbison and Steele [1987]. Advanced topics can be found in Berry [1986] and Lapin [1987]. For interesting reading in structured programming, the reader should study the texts by Dahl, Dijkstra, and Hoare [1972] and Wirth [1973].


Data Structures Using C: 1.2: A Look Ahead

1.2: A Look Ahead

A nucleus of commonly used data structures will be introduced. These include arrays and records, which are static structures that change only their values, and stack, lists, trees, and files, which are dynamic structures whose size and shape change as well as their values. Specific ways to implement each structure will be explained and typical and important operations on them investigated. These operations include selection, traversal, insertion, deletion, searching, and sorting. Although particular methodologies and data structures are important, it is more important to acquire skill in applying a methodology and in determining which data structures are appropriate for a given problem.

Programs are built from data structures. Yet the methodology an expert programmer uses makes the data structures invisible and makes the program structure clear and understandable. Neither good program structure nor good data structure selection alone produces good programs. Both must be done well, but they cannot be done independently. Obtaining a good program requires a well-designed solution, expressed first as an algorithm. The final step is writing the program in a programming language.

Determining what information is required to implement the algorithm and how to store it for efficient use and processing are perhaps the most crucial aspects of that facet of problem solving that is commonly called computer programming. The aim of this text is to illustrate good algorithm design, data structure selection, and programs.

If you do not know how to solve a given problem, then you cannot magically write a program to do so. In other words, if you don't know where you are going, then it doesn't matter how you get there. Moreover, as H. L. Mencken said, "Every complex problem has a simple easy-to-understand wrong answer." So before you begin writing a program, you will need to understand the problem and write a step-by-step solution, or algorithm.

Unfortunately, there is no algorithm for finding algorithms. There isn't even an algorithm for checking a candidate to see if it really is an algorithm. But there are some helpful techniques for developing algorithms. For example, it is useful to recognize when a known algorithm or program can be modified to give a solution to a new problem. One way to enhance this recognition is to solve problems in general or abstract terms. Sometimes a problem requires finding an object stored in computer memory that has certain properties. One technique is to search through the set of all objects until one with the desired properties is found. Alternatively, objects without the desired properties may be eliminated. The latter approach is used to develop a prime number algorithm in Section 1.3. At other times it may be possible to avoid searching altogether by constructing the desired object directly. Thinking up an algorithm is usually more difficult than programming it.


Data Structures Using C: 1.3: The Need for Structure

1.3: The Need for Structure

Imagine that you are a census taker charged with visiting all the homes in Spokesville and Squaresville (Figure 1.1). In each town, you must start and finish at home 1, after visiting each home once. All roads are the same length, and there are the same number of homes in each town. Nonetheless, in Spokesville you will travel a total distance exactly twice the distance required in Squaresville! The difference is due to structure.

It is easy to find the telephone number of a company whose name you know, even in the telephone directory of a large city. But finding the name of a company when you know only its telephone number is a tedious and time-consuming chore.The information in the telephone directory is structured in such a way that one task is easy while the other causes tension headache and eyestrain!

Two maps of the continental United States appear in Figure 1.2; part (a) gives a clear overview, while (b) gives a more detailed view. Structure clarifies, and detail obscures. Still, both are needed for a complete picture. Notice that drawing a border around and naming each state in the overview, in the absence of other distracting details, makes the composition of the country discernible. It is the plethora of detail and subordination of structure that is overwhelming in the other map. The less detailed map is analogous to a functionally modularized program, the other to an unmodularized program.

An algorithm for a given problem specifies a series of operations to be performed that eventually leads to a correct result. Algorithms express solutions step by step. In stating an algorithm, care must be taken to avoid ambiguity and to be sure that it is clear how each step is to be done. When the solution is translated into a programming language for execution by a computer, even more care and attention to detail are needed. But first, the solution is usually expressed in English or flowchart form, before the details required by the programming language are introduced. The next example illustrates the effect of structure in this context.

(a) Spokesville

(b) Squaresville

Figure 1.1 A Tale of Two Cities

An integer larger than 1 is a prime number if it cannot be written as the product of two integers, each larger than 1. Suppose we must find a way to print all prime numbers between 2 and some integer n. Thus if n is 12, the primes are 2, 3, 5, 7, and 11. Compare the following two algorithms for accomplishing this task.

                 primes(n)                          anotherprimes(n)

1. Create a collection called candidates,    1. Set X1, X2, . . . , Xn to

   consisting of all integers from 2 to n.      zero.

2. Remove all nonprime integers from         2. Set k to 2.

   candidates.

3. Print all integers that remain in         3. If Xk is 1, go to step 7.

   candidates.

                                             4. Print k.

                                             5. Set m to the integer part of

                                                n/k.

                                             6. Set Xk, X2k, . . . , Xmk to 1.

                                             7. Set k to k + 1.

                                             8. If k � sqrt n, go to

                                                step 3.

                                             9. If Xk is zero, print k.

                                            10. Set k to k - 1.

                                            11. If k � n, go to step 9.

(a) Overview

Figure 1.2(a) Courtesy: Hammond Incorporated, Maplewood, NJ 07040.

(b) Detailed View

Figure 1.2(b) Courtesy: Hammond Incorporated, Maplewood, NJ 07040.

It is not difficult to see that if each step of the first algorithm can be carried out, the correct result will be printed. It is considerably more difficult to see that the same is also true of the second algorithm. The first is structured to provide clarity, while the second is poorly structured and provides too much detail. These examples indicate why structure is needed in each stage of program design. Structured solutions are easier to understand, expand, and alter, and structured data make the operations of the algorithm easier and faster to perform.

1.3.1 Using Structure

1.3.2 A Prime Example


Data Structures Using C: 1.3.1 Using Structure

1.3.1 Using Structure

Well-designed, clear programs are based on the explicit underlying methodology called structured programming. As problems and their solutions become more sophisticated, the brute-force or ad hoc method of programming becomes much less effective. Compilers, operating systems, business management systems, and word processing systems consist of many thousands of program statements. Just as writing a lengthy book is several magnitudes more difficult than writing a paragraph, so is the process of developing a large programming system much more involved than writing a short program. There must be an overall structure.

An example of a large and complex program is an operating system. Many years ago it was thought that operating systems could not be implemented without errors. The immensity of the task was believed to cause inherent problems that could not be isolated and debugged in advance of system implementation. Hence system failure seemed inevitable and getting the system up and running meant a long trial-and-error period of testing and debugging. A new methodology of system design based on the concept of structured programming has demonstrated, however, that error-free operating systems are possible.

Since the programs you see as a student may be relatively trivial, the need for a well-founded methodology of programming may not be evident. However, imagine a large program (twenty pages long, and over a thousand lines of program statements) that you did not write. If you were assigned the task of modifying the program to add a new feature, how or where would you start? Certainly you must have a full understanding of what the program does, how it does it, and how it relates to other programs that use it or are used by it. Anyone who intends to read, write, modify, or correct any programming systems with more than a few dozen statements should use structured programming.


Data Structures Using C: 1.3.2 A Prime Example

1.3.2 A Prime Example

The example of prime numbers is presented here in more detail to illustrate the process and benefits of structuring and the concept of data abstraction. To do this a group of functions is developed from the first prime number algorithm:

primes(n)

1. Create a collection called candidates, consisting of all integers from 2 to n.

2. Remove all nonprime integers from candidates.

3. Print all integers that remain in candidates.

Consider first a simple function called primes.

primes(n)

int n;

{

collection candidates;

create(n,candidates);

remove(n,candidates);

print(n,candidates);

}

Primes may be obtained directly from the prime number algorithm when create, remove, and print are interpreted as functions carrying out steps 1, 2, and 3 of the algorithm, respectively. Note that no assumptions about candidates have been made. At this point it is of type collection, which has not yet been defined. Later in the process of developing the algorithm and program, collection, and thereby candidates, will be defined, after the basic operations on candidates become clear. Note that collection is a type, whereas candidates is a variable whose type must be declared. The next step is to refine and verify each function of primes. Refine means to express or define in more detail; verify means to confirm that a refinement does indeed do its job. To refine and verify create and print is straightforward enough, but to refine and verify remove is more difficult. If you want to know more about a state than you see in the first map of Figure 1.2, you may consult another map containing more information on the state's geography. Similarly, if you want to know how to carry out remove, a more detailed description may be available. If not, as is the case here, remove may be treated as a new problem whose solution must be found. One solution is this:

remove(n, candidates)

Set factor to the first prime.

While (nonprimes remain in candidates),

Delete from candidates all integers that are multiples of factor, and

Increase factor by 1.

It is again easy to see that, as long as each step is done correctly, the desired effect is obtained.

Figure 1.3 PRIMES Modularized

Still, more examination is needed to determine whether "nonprimes remain in candidates." It can be shown that nonprimes may remain in candidates only when factor is less than or equal to the square root of n. Consequently the condition (nonprimes remain in candidates) may be replaced by the condition [image]. But doing this will make remove more difficult to understand. It is better to consider the evaluation of the condition as a new task that must be refined. We are now ready to write the function for remove and to verify it. The three functions appear in Figure 1.3, where set initializes the collection candidates to empty, insert adds the integer i to the collection, nonprimes is a function assumed to return the value true when [image] and false otherwise, delete removes all integers from candidates that are multiples of factor, and belongs returns the value true if i is in candidates and false otherwise.

Delete is refined next (Figure 1.3). You should verify that this function correctly performs its task as long as omit takes nextmultiple out of candidates. It may help to simulate the execution of delete when n is 40 and factor is 2, and then 3.

Each task of function primes has now been defined and is, in fact, a complete C function. All other tasks, except for set, insert, belongs, and omit, have also been defined. Primes and all other tasks have been written modularly (functionally), meaning that for each task there is a function. For example, new functions have been introduced for each of the three tasks of primes: create, remove, and print. Had primes along with remove and delete not been functionally modularized, it would appear as follows:

primes(n)

int n;

{

collection candidates;

int factor,i,nextmultiple;

int firstprime;

firstprime = 2;

double sqrt();

>Comment

set(n,candidates);

>Comment

for (i=2;i<=n;i++)

insert(i,candidates);

factor = firstprime;

while (factor <= sqrt((double)n))

>Comment

{

nextmultiple = 2 * factor;

while (nextmultiple <= n)

{

omit(nextmultiple,candidates);

nextmultiple = nextmultiple + factor;

}

factor++

}

>Comment

for (i=2;i<=n;i++)

if(belongs(i,candidates))

printf("\n %d\n",i);

}

This version is also a refinement of primes, but instead of defining new functions for its three tasks, the tasks have been replaced by program segments. Thus there are two basic ways to refine any task: (1) define a new function for the task, or (2) replace the task directly by its refinement expressed as a program segment in code. Compare this version with the functionally modularized form, the simple function primes given earlier. Notice that functional modularization suppresses detail.

Deciding how to modularize a program functionally is often a matter of individual taste, but as this text unfolds, the process will become clearer, and you will be able to develop your own style. Functional modularization is intended to highlight and isolate important tasks. The advantage of using functions may not be apparent with this simple prime number problem but will be striking in more complex problems.

Finally, the low-level tasks set, insert, omit, and belongs must be coded. To do this requires deciding how to implement candidates�that is, deciding which data structures shall be used to store candidates and how to write the basic functions that operate on candidates. Recall that data structures such as candidates contain information and are operated on during the execution of a program. No matter how candidates is implemented, none of the functions already defined needs to be changed. They are independent of candidates implementation (except for the data declaration). As long as set, insert, omit, and belongs do their tasks correctly, no function invoking them needs to be changed. This is because we have written them treating candidates as a data abstraction. That is, we have assumed that candidates and the basic operations on it are available via functions (set, insert, omit, and belongs) and further, that any other operations on candidates have been expressed in terms of these functions. For instance, delete uses omit, and create uses insert. If, instead of calls to set, insert, omit, and belongs, the code defining them had been used, all functions containing these codes would be dependent on the implementation. Later, if we changed to a better implementation, it would be necessary to hunt through each function, find any code that would be affected by the change, and modify that code appropriately. A data abstraction consists of a data structure, to store data, and allowed operations on the data structure, to manipulate it. As a result, wherever one of these operations is invoked in a program, the data structure is being affected. Just as important, the data structure is not being affected unless one of these operations is invoked.

The use of data abstraction and functional modularization not only enhances understanding, but also simplifies maintaining a program. Maintenance is a crucial aspect of programming. It involves changing a program to do something new, to stop doing something, or to do better what it now does.

Using data abstractions as outlined above hides implementation details and makes the program independent of those details. On the other hand, the program's speed of execution and storage requirements will be greatly influenced by the implementation of the data abstraction. Consequently, the selection of an implementation should be based on the data abstraction's operations. Sometimes the choice that minimizes execution time and storage will be evident. More typically, conflicts arise, and the choice will involve trade-offs between time and storage. How to make this choice properly is a major focus of this text.

Consider two ways to implement candidates. It might be implemented as an integer array with the ith element of the array containing the value true if i is a prime and false if it is not. Alternatively, candidates could be implemented as an integer that is made up of contributions from each number in candidates. If the number i is in candidates, it contributes 2i. Thus if candidates consisted of 2, 3, 5, 7, and 9, its value would be 684(4 + 8 + 32 + 128 + 512). The integer is stored in an integer array of size one. Candidates could also be implemented as a dynamic list or tree and, indeed, one of these might be preferred. You might want to return to this problem after studying lists and trees, to verify that it can be converted to these structures. However, at this stage, arrays that are more familiar to you will be used. The code below defines the operations on candidates for the two array implementations. Each implementation should have its type declaration for collection global to primes and all its functions, as follows:

   Candidates Implemented          Candidates Implemented

         as an Array                    as an Integer

typedef int collection[MAXN];  typedef int collection[1];

set(n,c)                       set(n,c)

/* Sets c to empty */          /* Sets c to empty */

int n;                         int n;

collection c;                  collection c;

{                              {

   int i;                         c[0] = 0;

   for (i=1;i<=n;i++)          }

      c[i] = FALSE;

}

insert(i,c)                    insert(i,c)

/* Inserts i into c */         /* Inserts i into c */

int i;                         int i;

collection c;                  collection c;

{                              {

   c[i] = TRUE;                   int p,q;

    >Comment

}                                 p = power(2,i);

                                  q = (c[0]/p)%2;

                                  c[0] = c[0] + (1-q)*p;

                                }

omit(i,c)                       omit(i,c)

/* Removes i from c */          /* Removes i from c */

int i;                          int i;

collection c;                   collection c;

{                               {

   c[i] = FALSE;                   int p,q;

    >Comment

}                                  p = power(2,i);

                                   q = (c[0]/p)%2;

                                   c[0] = c[0] - q*p;

                                }

belongs(i,c)                    belongs(i,c)

/* Returns true only            /* Returns true ony

   if i is in c                    if i is in c

*/                              */

int i;                          int i;

collection c;                   collection c;

{                               {

   return(c[i]);                   return((c[0]/power(2,i))%2);

}                               }

                                power (x,y)

                                /* Returns x to the power y */

                                int x,y;

                                {

                                   int i,p;

                                      p = 1;

                                   for (i=1;i<=y;++i)

                                      p = p*x;

                                   return(p);

                                }

Note that the terms TRUE and FALSE are used in the boolean version. Since C does not have a type boolean, TRUE and FALSE must be defined before they are used in the program. In C, a true statement returns a non-zero value, and a false statement returns a zero. For consistency, write

# define TRUE 1

# define FALSE 0

TRUE and FALSE will be used for programming style in many places in the text. Programs that use them merely require the above statements. In the second version the function power (2,i) returns 2i.

Each implementation specifies a data structure for candidates and defines the same four basic operations allowed on it. Each entails some explicit restriction on the size of candidates. Such restrictions can be avoided by using the dynamic data structures introduced in later chapters.

The first implementation of candidates allows insert, omit, and belongs to be written so that they take constant time to execute. However, the amount of storage required will depend directly on n. The second implementation for candidates increases the execution times of the basic operations on candidates but reduces its storage requirement. In general, the choice of implementation for a data abstraction may be critical, determining whether a program will run to completion or abort because of insufficient time or storage. Section 1.8 discusses the selection of program and data structure in more detail. In any case, once the decision is made, the data declaration for candidates must appear in primes. The final result is a well-structured program. A complete program to run primes using the array implementation follows.

#include <stdio.h>

    >Comment

main()

{

int n;

printf("\n n = ? \n");

scanf("%d",&n);

primes(n);

}

    >Comment

#define TRUE 1

#define FALSE 0

#define MAXN 500

typedef int collection[MAXN];

set(n,c)

/* Sets c to empty */

int n;

collection c;

{

int i;

for (i=1;i<=n;i++)

c[i] = FALSE;

}

insert(i,c)

/* Inserts i in c */

int i;

collection c;

{

c[i] = TRUE;

}

omit(i,c)

/* Removes i from c */

int i;

collection c;

{

c[i] = FALSE;

}

belongs(i,c)

/* Returns true only

if i is in c

*/

int i;

collection c;

{

return(c[i]);

}

    >Comment

primes(n)

/* Prints all prime numbers between 2 and n */

int n;

{

collection candidates;

create(n,candidates);

remove(n,candidates);

print(n,candidates);

}

create(n,c)

/* Creates a collection of integers between 2 and n */

int n;

collection c;

{

int i;

    >Comment

set(n,c);

    >Comment

for (i=2;i<=n;i++)

insert(i,c);

}

remove(n,c)

/* Removes all nonprimes from c */

int n;

collection c;

{

int firstprime;

int factor;

firstprime = 2;

factor = firstprime;

    >Comment

while (nonprimes(factor,n))

{

delete(factor,n,c);

factor++;

}

}

nonprimes(factor,n)

/* Returns true only if non

primes may remain in c

*/

int factor,n;

double sqrt();

{

    >Comment

return(factor <= sqrt((double)n));

}

delete(factor,n,c)

/* Deletes multiples of factor from c */

int factor,n;

collection c;

{

int nextmultiple;

nextmultiple = 2 * factor;

    >Comment

while (nextmultiple <= n)

{

omit(nextmultiple,c);

nextmultiple = nextmultiple + factor;

}

}

print(n,c)

/* Prints integers in c */

int n;

collection c;

{

int i;

    >Comment

for (i=2;i<=n;i++)

if (belongs(i,c))

printf("\n %d\n",i);

}

As noted, the first implementation for collection is used in the program. Notice that the declarations and definitions needed for the collection appear together in the program. Simply replacing that portion of the program with another implementation is all that is required to change to the new implementation. In Chapter 5 another way to package the collection and its operations for this program will be presented.

Abstraction allows us to focus on concepts rather than concrete details. The architects planning a building assume an elevator as part of the plan. They know the elevator will hold passengers and freight. They know the elevator will convey its contents between floors, respond to call buttons, and be warm in winter and cool in summer. The initial building plans can proceed using only these essential concepts regarding the elevator. Later, the detailed choices necessary to build the elevator will be made. They will determine its size, shape, and structure, as well as its speed, responsiveness, and reliability. Notice that, as long as the elevator is treated as an abstraction, any specific implementation can be substituted for another without requiring changes to the plan.

Similarly, data abstraction allows us to focus on what is needed rather than how it will be assembled and packaged. Planning a program, we assume that we have specific means to store and operate on data. Only later must an implementation be selected. The implementation determines how well the job is done, but does not affect its logical design. The longer this choice can be delayed in program development, the more independent of this specific choice our program will be. This is highly desirable, making program changes much easier to effect.


Data Structures Using C: 1.4: Basic Constructs

1.4: Basic Constructs

The richness and power of a language are related to the variety and ease of expression that the language allows. Thus high-level computer languages provide more expressive power than more limited languages, such as assembly languages. However, it is theoretically possible to state any algorithm using only three basic constructs and the rules for their combination. In this discussion three basic constructs have been chosen for the sake of conciseness, but others are used freely when appropriate.

The three constructs indicate different types of flow of control. Each construct determines the order in which its tasks are processed, and each was chosen for its logic and clarity.

The first is the sequence construct, in which each task follows the previous one without altering the order of the tasks. Each task is executed once. Notice that primes (Figure 1.3) is based on this construct. The second is the decision construct, which allows only one of two tasks to be carried out, depending on a controlling logical condition. The appropriate task is executed once. It is the true task when the condition is true and the false task when the condition is false. The final construct is the loop, which enables a task to be repeated until a controlling condition no longer holds. Obviously, the loop must at some point end so that the next task can be executed. Failure to change the value of the controlling condition in order to end the loop task results in the infamous "infinite loop." Examples of algorithms formulated according to the three basic constructs appear in Figure 1.4 as structured flowcharts, along with their structured English equivalents. The corresponding versions are obvious.

Each construct has exactly one entry and one exit point. The constructs may be combined only by making the exit arrow of one the entrance arrow of another. This results in complex constructs that are sequences of constructs. The only allowed modification of a construct is the replacement of any of its tasks by one of the three basic constructs. This amounts to incorporating the task's refinements into the construct, producing a more complex construct.

Figure 1.4 The Basic Control Constructs and Their English Equivalents

There are two different ways to express a solution using these constructs and rules for their combination. One is to start with a basic construct and repeatedly apply modifications to it. Each modification corresponds to the use of a program segment for its implementation. This results in a final unmodularized flowchart, English equivalent, or program. See the unmodularized version of primes for an example. The second is to start with a basic construct, and, instead of modifying it, express the solution for each of its tasks as a separate function. This amounts to functionally modularizing each task's refinement, with the complete solution consisting of the collection of individual solutions. This results in a final collection of functionally modularized flowcharts, their English equivalents, or program components. Usually a combination of the two is better than using one method only. It is wise to keep each part of a solution to a reasonable size, say one page. This can serve as a guide to whether or not functional modularization (the use of a function) or modification (the use of a program segment) is appropriate. But remember: the primary goal is clarity.


Data Structures Using C: 1.5: Top Down Design

1.5: Top Down Design

The purpose of structured programming is to create programs that are

n Understandable, meaning the logic is readily followed

n Reliable, so the program performs as intended

n Adaptable, so modifications can be made with reasonable effort

Structured programming is a method of approaching a problem rather than adhering to a rigid set of rules.

As Dijkstra [1972] observed, the "art of programming is the art of organizing complexity, of mastering multitude and avoiding its bastard chaos as effectively as possible." The essential concept of structured programming is that dealing with complexity requires simplicity.

Regarding terminology, note that the tasks performed by a program solve problems. In this book the words task and problem are used interchangeably. Each programming task entails defining the input information to be processed, defining the output information to be produced, and stating clearly the precise instructions as to what, but not how, it is to be done.

Simplicity and clarity are achieved in a complex program by focusing attention on just one task at a time. To show how the task is done, it is broken down into smaller meaningful subtasks. The combination of smaller tasks to perform the given task is called a refinement of that task. The next step is verifying that the refinement is correct�that is, that the refinement indeed accomplishes the task, as long as each of its components does its job. This verification may be done even without knowing how each component works. This is important. How the components of a refinement work is irrelevant to verification. The goal is to ensure that, if they work correctly, then the refinement itself works correctly.

Top-down design begins with the original problem and involves a loop: repeated refining and verifying of the problem and its refinements until only implementable problems remain. Implementable means simple enough to be directly coded in a programming language. Review the development in Section 1.3.2 for primes to see that this loop was, in fact, being used. With this approach the programmer can be sure the final program is correct even though he or she has considered only one manageable piece at a time.

If the total number of tasks needed to complete a detailed solution is proportional to the complexity of the initial problem (as we hope), then the effort to create, understand, and verify a solution to a complex problem will also be proportional to its complexity. This is a very desirable situation. If the cost of a solution increased more drastically with complexity, then the complexity of problems we could solve would be much more severely limited.

Think about building a jigsaw puzzle. You probably use the top-down approach. You break up the picture conceptually into components�say, the border and recognizable objects with distinct shapes and colors. Then you build each component and combine them all. Surely the most difficult puzzles are those with no distinguishable objects. They prevent refinement, forcing one to deal with all the complexity at one time. Perhaps this is why all the problems humans solve have structure; they are the only ones we can handle.

Two caveats must be added:

1. At each level of refinement, keep the number of component tasks small and keep them independent. This is critical because human beings can deal well with only a few things at a time, and can do so more readily when the things are not intertwined. Psychological studies suggest a "magic number": a maximum of seven tasks, plus or minus two. Independence means that how one task is accomplished need not be a consideration in how another task is done. Sometimes independence must be given up for other considerations, such as efficiency. These trade-offs should be considered carefully.

2. Control carefully how tasks of a refinement are combined. Simplicity requires structure that makes the logic of a program easily discernible. Certain combinations or constructs foster such clarity. Some programs present a bewildering panorama of program statements, but what is needed are programs that appear as a logical sequence of tasks to be executed. Some ways of expressing the order in which tasks are to be executed are more understandable than others.Constructs that are well designed allow anyone working with a program to grasp quickly what is intended. This is the hallmark of the basic constructs�sequence, decision, and loop�introduced in Section 1.4.


Data Structures Using C: 1.6: High-Level Languages

1.6: High-Level Languages

High-level languages provide basic data structures and operations and the capability of using these to build more complex data structures and operations. With some languages it can be hard to realize this capability, while with others it is relatively effortless. In addition, languages have different idiosyncrasies and subtle shortcomings. In this text, the focus is not on the language but on the more general topics of program structure and data structure selection. Still, whatever background you have in high-level languages may be helpful.

A program written in a high-level language specifies a series of operations to be performed on data structures. Think of a high-level language as having an associated high-level computer that carries out those operations and stores those data structures in its memory. Such high-level computers are just a concept�normally they are not built. The conventional computer is not specifically a C, Cobol, Pascal, or FORTRAN machine; we rely on compilers to translate programs from high-level languages into the languages of existing computers. The translated version is then executed by the conventional computer. The compiler, in effect, makes the conventional computer appear to be a high-level computer for the high-level language.

To do this translation, the compiler must find a way to represent the program's data structures, using those in the conventional computer's language. lt must also apply operations of the conventional computer's language to these representations to simulate the program's operations on its data structures. How this is done, or what the actual computer is, does not affect the correctness of programs written in the high-level language. This is data abstraction par excellence! However, the way this is done will influence the storage requirements and execution time of the program. It can also cause problems for the unwary user.

1.6.1 Data Types

1.6.2 Data Representation


Data Structures Using C: 1.6.1 Data Types

1.6.1 Data Types

Programs use variables to refer to storage elements in the memory of the high-level computer. Each variable has a name, which refers to its storage elements, and a value, which is the content of those elements. For example, the C assignment statement

sum = sum + increment;

uses two variables with names sum and increment. Individual variables, which correspond to a single storage element, are the simplest type of data structure. More complex structures are built from variables. In turn, operations on the more complex structures are built from operations on the simpler structures. High-level languages differ in the kinds of basic data structures they provide. For example, FORTRAN includes individual variables and dimensioned arrays. C adds records and pointers and, significantly, allows the definition of new data types and data structures. The Ada language allows new basic operations to be introduced as well.

Each variable must be defined by a declaration that gives its name and data type. The data type indicates the actual values that may by assumed by the variable. For example, the C declaration

int sum, increment;

char check;

specifies that sum and increment are the names of variables whose values will be integers, and that check is the name of a variable whose values will be a character.


Data Structures Using C: 1.6.2 Data Representation

1.6.2 Data Representation

Data types must be specified. An example will show why. Suppose a compiler represents integer and real variables (floats) using seven decimal digits. Then in storage that is designated type integer, the value 1230045 represents the integer number 1,230,045. In storage that is designated type float the value 1230045 represents the real number 0.30045 x 10l2. In general, the compiler interprets

d1 d2 . . . d7 of type integer as the integer number dl d2 . . . d7.

and

d1 d2 . . . d7 of type float as the real number 0.d3 d4 . . . d7 x 10d1d2

Given this situation, suppose the compiler must cause the contents of sum = 1230045 to be added to the contents of increment = 1250047,where they are both of type integer. The compiler might apply the usual column-by-column addition algorithm:

sum        1230045

+

increment  1250047

           枛枛枛-

           2480092

Here, 2480092, when stored in sum, represents the correct result. If sum and increment are of type real, then the content of sum represents the real number 0.30045 x 1012 and the content of increment represents the real number 0.50047 x 1012. Their sum is 0.80092 x 1012. However, if the above column-by-column addition algorithm is applied and the result stored in sum, then its contents will still be 2480092, but this now represents the real number 0.80092 x 1024. This is incorrect by a factor of 1012. To represent the correct result, sum should contain 1280092. The difficulty is, of course, that the same algorithm should not be applied for the addition of numbers of both type integer and type float. Two distinct algorithms are needed, because two distinct representations are used. In general, the algorithms used must depend on the data types of the variables involved. The compiler can choose the correct algorithm only when it knows the data types.

Thus it is essential to declare the data type of each variable so that the compiler can choose the proper algorithm to achieve the intended result. Type declarations also allow high-level language compilers to build in checks for validity. Such checks may generate error messages during compilation or during execution of the program. Languages with more restrictive rules for the manipulation of different data types, called strongly typed languages, allow more checking. For example, Pascal's strong typing prevents the contents of a variable of one type from being copied into a variable of a different type. C, FORTRAN, and COBOL allow this to be done in some cases but not in others.

The C data types integer, float, and char or their equivalents are now standard in most high-level languages. The values they may assume, and the operations that may be performed on variables of these types, are specified in each language.


Data Structures Using C: 1.7: Elements of Style

1.7: Elements of Style

Individual writers, even whole magazines or journals, have their own writing style. Often a glimpse at a few paragraphs can give clues to the author or the type of publication. Still, there are general rules to follow for acceptable style; entire books have been devoted to this subject. Style will always be in style.

Dress, carriage, and response are parts of personal style, as are straightness of teeth, polished or scuffed shoes, and a fast or slow gait. Program style is determined by overall appearance as well as by details of implementation. We want to see the logical structure of a program clearly; too many details cloud the picture. Data abstraction and functional modularization are tools for sharpening the picture. Hiding information, so that only parts of a program that need it can see it, also provides better focus. People react differently to unexpected situations: some become erratic while others remain calm, going cheerfully about their business. Programs should react to unexpected data with aplomb, letting us know about errors, but getting on with the job when possible. Details of a program should not affect its logical structure, only its efficiency. Details can make an otherwise stylish program sloppy, slow, and in need of repair, or they can finish the design so it is elegant, sleek, and well-tuned. What we see isn't always what we get, but stylish programs suggest a better thought-out design and product.

Since good programs must be written with style, guidelines are offered here for this purpose. A good program is correct, easy to read, and changeable through a reasonable amount of effort. Whether or not it should be efficient depends on the application. In some cases an efficient program may be written with the same effort as an inefficient one. In other cases it will be necessary to put in more effort to achieve efficiency, so the additional programming may be done only if it will bring substantial (or required) savings in computation time.

1.7.1 Expressing Purpose

1.7.2 Communication between Functions

1.7.3 Ensuring against Data Errors: Defensive Programming

1.7.4 Changing Programs

1.7.5 Verification and Debugging

1.7.6 In a Nutshell


Data Structures Using C: 1.7.1 Expressing Purpose

1.7.1 Expressing Purpose

What's in a name? "A rose is a rose is a . . . ." "There's more form than substance." Perhaps these sentiments are true, but in a well-presented program there is a great deal in a name. One function is not another function, and understanding of substance is enhanced by form.

Indentation and proper spacing serve to delineate significant components of a program. Choosing names to convey and delineate purpose is also extremely important. Even without the descriptions given for the functions of Figure 1.3, the names of each function, variable, and constant almost provide, by themselves, enough information to see what each does. On the other hand, inappropriate names can be very misleading and distracting.

Reading identifiers to understand what a program does requires reading the entire program. This is too demanding. After all, when selecting a book, no one wants to be forced to read from cover to cover just to see what it is about. The same holds for a program, program documentation must be provided for this purpose. Good name selection is not enough. The main program and each of its functions must start with comments that allow users (including its author) to understand readily what it does and what its inputs, outputs, and parameters are. The main program's documentation then tells immediately what it does. No further reading is necessary unless someone wants to know how it is done. Similar comments can precede program segments that implement tasks of some complexity. To a great extent, the structure of a function, coupled with the documentation of each of its tasks, explains how the function does what it does. In this sense, structured programs (with proper comments) are self-documenting. Functional modularization and data abstraction help to elucidate structure by obscuring detail. Some programmers are free to develop their own style for delineation and documentation; for others it may be specified rigidly. The case study presented later gives a complete program with documentation as a guide for annotating your own programs. As a rule, complete documentation is not included in programs appearing in this book, since in effect the text itself serves as documentation, discussing completely each program presented. Still, programs will be annotated for clarity. Proper documentation is a prominent aspect of every good program and should become a habit of the programmer. Since the top-down approach requires a description for each task, the documentation for a task may be done as it is introduced, even before it is written.


Data Structures Using C: 1.7.2 Communication between Functions

1.7.2 Communication between Functions

High-level languages allow the definition of three kinds of variables: local, nonlocal and global. A C program consists of functions. A function of a program may reference

1. Local variables, to which it alone has access

2. Nonlocal variables, to which it and some other functions of the program have access

3. Global variables, to which all functions of the program have access

A variable is referenced by a function in order to copy, modify, or simply look at its value.

High-level languages treat variables in differing ways. FORTRAN, for example, considers variables to be local to a function unless they appear as formal parameters of that function (in which case they are nonlocal). Variables stored in special common areas of memory are global. C determines whether a variable is local, nonlocal, or global by using block rules. In Figure 1.3 candidates is local to primes, factor is local to remove, and nextmultiple is local to delete, while n is nonlocal or global to each.

Each high-level language has its own rules to give one function access to variables of another. Merely using the same variable name in two functions will not normally allow one to reference a variable in another. For example, i in create and i in print (Figure 1.3) do not represent the same variable; each is local to its function. If this were not so, functions could not be written independently. Once a complex problem has been refined, different individuals or groups should be able to develop components independently. Otherwise, each time a group used a new variable name, it would have to check that no other group used the same name.

The issue here is really communication between functions of a program. Communication between functions�allowing one to reference variables also used by another�occurs in two basic ways: using nonlocal or global variables, or using formal parameters in defining functions.

When a function changes the value of a variable used by another, and the variable is not communicated (or passed) between them as a parameter, a side effect has occurred. Side effects make it extremely difficult to understand what functions do. It is easier to see what a function does when we can be sure that only the parameters and the local variables will be affected by the execution of the function. The communication structure of a program is clearer when information is communicated explicitly by the use of parameters. This shows "who is doing what to whom." For example, in delete (Figure 1.3), no side effects can occur, since the only nonlocal variables factor, n, and candidates all appear as parameters. Thus the only variables not local to delete that it can reference are factor, n, and candidates.

At one extreme, it is possible to make all variables global�every function may then reference any variable. Making all variables global requires extremely careful adherence to the use of variable names and increases the probability of side effects. COBOL programs are typically written this way. However, such programs use well-defined file structures, deal with high-volume input and output, and use relatively simple algorithms, so this method may not cause difficulty.

The other extreme is to use only parameters between functions. This may not always be possible. For example, storage considerations may intrude, as with arrays in Pascal.

Communication between functions by using parameters may be a one-way or a two-way street. In some high-level languages, like FORTRAN, any change in the value of a parameter in a called function is passed or reflected back to the calling function. Other languages, such as C, allow one component to communicate information to another yet never reflect changes made to the parameter's value. Upon return, the parameter's value is the same as prior to the call. Also, upon return from a call, different languages treat local variables of functions differently. Some preserve their values between calls, while others do not. In FORTRAN it is possible to have local variables retain their values between invocations of a function; in Pascal it is not. In C either case may apply. Values may be preserved or erased as desired.


Data Structures Using C: 1.7.3 Ensuring against Data Errors: Defensive

1.7.3 Ensuring against Data Errors: Defensive Programming

Functions receive and output data. In primes, what happens if n is negative or so large that it exceeds the storage capacity allocated to the data structure implementing candidates? Think how upset a company's personnel would be if no one received paychecks because the input data for one employee was not valid. Or suppose each employee received an output check equal to the entire payroll. Defensive programming is ensuring that your programs guard against such outcomes by checking for invalid data, incoming or outgoing, and doing something reasonable when such data are discovered. What is reasonable can vary from signaling an error and stopping further processing to simply outputting an error message, ignoring the invalid data, and continuing processing. Sometimes the correct data may be inferred from the invalid data, but this approach must be taken with great caution. The important point is to ensure that errors are discovered. It is safer to have no output than output that is in error but thought to be correct. Invalid data is a very serious problem, regardless of whether it is due to design or to happenstance. Validation functions for input and output should be used, even though it is not always easy to see how to guard against all possible errors. This text does not have space to dwell on the issue of data validation, but it should always be in your mind.


Data Structures Using C: 1.7.4 Changing Programs

1.7.4 Changing Programs

Consider a new problem: count and print the number of primes that are no greater than n. Instead of immediately attempting a solution, can you think of a program that already solves a related problem? You've already seen one: primes. Perhaps it can be changed to help with the solution. Looking at the refinement of primes (Figure 1.3), you can see that replacing function print by a count function that prints the proper count does the trick. The solution countprimes, using the changed primes and count, is as follows.

countprimes (n)

/* This function counts the prime

numbers between 2 and n

and prints the count.

*/

int n;

{

collection candidates;

create (n, candidates);

remove(n,candidates);

count(n,candidates);

}

count(n,c)

/* Prints the number of integers

in c between 2 and n.

*/

int n;

collection c;

{

int i,counter;

counter = 0;

for (i = 2; i<=n;i++)

if (belongs(i,c))

counter++;

printf ("\n The number of primes between 2 and %d is %d\n",

n, counter);

}

Solutions are rarely obtained so easily. Yet this example teaches us something. Programs that have been designed wisely, with an eye to generality, may often be used as tools in the solution of related problems. When beginning the development of a solution to a problem, see first if a program for a related problem is already known, and if so whether it can be adapted. This can save significant time, effort, and money compared to writing a new program or function.

It was easy to see what to change and how to change it in our example, because primes is functionally modular, with narrowly defined tasks and independent functions. Changes are rarely as straightforward as this; for contrast, consider changing the algorithm anotherprimes to a program for countprimes. In general, though, isolating functions that carry out a single narrowly defined task tends to generate lower-level functions that are clear, simple, and of moderate size. At higher levels, such single-task functions need not be simple. For example, a function to "Select the best next chess move" is such a function, but it is not simple. The lower the level of a function that needs changing, the less the effort required. However, the higher the level of the task for which one can adapt a known program, the greater the savings, since much refinement is avoided. In an ideal situation, changes require little effort and benefits are great.

Often a programmer's assignment or goal is to increase the efficiency of a program, and this will involve changing the way data are stored. If the program has been written using data abstractions, it is much easier to see exactly which functions need changing. For example, in primes, to change the implementation of candidates involves changing only its declaration and set, insert, omit, and belongs.


Data Structures Using C: 1.7.5 Verification and Debugging

1.7.5 Verification and Debugging

Verifying a solution means checking that program segments and functions do their tasks correctly. It is difficult to verify programs, although some formal methods are known. This text will illustrate how a program can be verified informally, but rigorously, for appropriate examples. For most of the small examples, verification is left to the reader.

Although only proper verification can ensure the correctness of a program, debugging can help in locating mistakes. Your goal should be not to run or even code programs until you are sure the algorithm is correct. Even if you are sure of the algorithm, debugging tools should be employed. Then, when your program fails to work correctly (as you know by now it will), you can track down the cause without the help of a Holmes or Watson.

In addition to debugging aids provided by your computer system and compiler, good programming techniques facilitate debugging. For example, it is important to learn to display appropriate error messages and information. The top-down approach includes debugging individual functions or program segments as they are refined, so that debugging proceeds as the program is developed. Verifying, debugging, and documenting all require basic understanding of the program and much the same information. To a large extent, then, they should be done concurrently with program development. During development you have the concepts and information fresh in your mind.

The main program and the other functions may each be debugged independently, just as they may be verified and documented independently. This requires stubs and drivers.

A stub takes the place of a function. It should have the same name and parameters as the function and, when called, should output a message indicating it was called. For example,

remove (n,c)

/* This is a stub for remove */

int n;

collection c;

{

printf ("\n remove was called \n");

}

A driver plays the role of a function that calls another function, and is used to debug the called function. The driver should set up test values for the parameters used by the function it calls, output those values, call the function, and output the returned values. For example,

removedriver ()

/* This is a driver for remove */

{

collection candidates;

int n;

int sentinel = -1;

printf("\n enter a value for n between 2 and 27 \n");

scanf("%2d",&n);

while (n != sentinel)

{

printf("\n remove has been called with n = %d\n",n);

create(n,candidates);

print(n,candidates);

remove(n,candidates);

printf("\n remove returns n = %d ", n);

printf("and candidates: \n");

print(n,candidates);

printf("\n enter a new value for n or -1 to stop \n");

scanf("%2d",&n);

}

}

Using drivers and stubs allows any individual function to be tested and debugged even though the functions that call it and are called by it have not yet been written. Within the function, program segments that correspond to algorithmic tasks can be isolated. A statement can be inserted before and after each of these segments to output a message indicating that the segment was reached during execution, giving the values of variables it worked on, specifying that it was exited during execution, and giving the modified values of the variables.

In this way, you can see the progress of a program during its execution by studying the output of the drivers, stubs, and function being tested. With this approach, you never find yourself with a program that has not run correctly and for which you have no output, or output that gives no clue as to what went wrong or where. Suppose function M has just been refined and is to be tested and debugged. Its refinement may include calls to other functions not yet written or debugged. To test and debug M you must

1. Write a driver to call M and feed it test data on which to work

2. Replace each function of M's refinement by its own stub

For example, to test and debug remove (Figure 1.3) requires

1. A driver to call remove and feed it test values for n and candidates

2. Replacing nonprimes and delete by their individual stubs

This assumes that primes (which calls remove) and noprimes and delete (which remove calls) have not yet been verified or even written.

Instead, suppose primes has already been debugged. Primes, in conjunction with the driver used to debug it, could then be taken as the basis for remove's driver. Next, this driver plus remove could be used as the basis for delete's driver. This way of proceeding parallels the top-down development of the program. It amounts to inserting one new function at a time into an already debugged program.

To illustrate the proper way to use debugging statements, a complete listing and output is given for an execution of the case study program later in the chapter.


Data Structures Using C: 1.7.6 In a Nutshell

1.7.6 In a Nutshell

The discussion of good programming style may be summarized with the following guidelines.

1. Take care to present programs well.

Don't choose a name without a purpose.

Don't leave a component's purpose unspecified.

Descriptions should be simple even if tasks are not.

2. Communicate well.

Store data only where it is needed (use local variables).

Pass information explicitly when needed elsewhere (use parameters).

Avoid side effects.

3. Use data abstraction.

This is the basic way to make programs independent of data implementation, and it makes implementation changes easier and more reliable.

4. Functionally modularize programs.

Functions should execute a simple, narrowly defined task.

It should be possible to determine what a function does by looking only at it.

Changes to a function should not necessitate changes in other functions.

Prevent unreasonable input and output data by defensive programming.

5. Use debugging aids.

Also verify your programs.

Of course, strict adherence to rules may not always be possible or even yield the best results. Knowing how or to what extent to apply programming rules comes only from experience in using them and in studying their effects.


Data Structures Using C: 1.8: Time and Storage Requirements

1.8: Time and Storage Requirements

Sometimes the time of execution or the storage required by a program, or both, are limited. Analysis helps pinpoint parts of a program whose efficiency may be worth trying to improve, because they are executed frequently or require large amounts of storage. Such analysis can be very difficult, and simulation of the program may be needed for estimates. Simulation means that the program is executed repeatedly for appropriately chosen data, and the results are averaged or otherwise statistically analyzed. Sometimes it is possible to analyze how these requirements increase as the amount or size of the data increases, even though we cannot say what happens exactly for smaller amounts or sizes. This is called asymptotic analysis. When exact analysis cannot be done, we settle for bounds on the time and storage required.

1.8.1 Analyzing the Prime Example

1.8.2 Limits on Time and Storage


Data Structures Using C: 1.8.1 Analyzing the Prime Example

1.8.1 Analyzing the Prime Example

The nonmathematically inclined reader may want to skip the following analysis of primes (Figure 1.3), which demonstrates how to go about informally analyzing a program and shows how trade-offs arise. Even such simple programs do not necessarily have simple analyses.

When a statement is said to take constant time, it means that the statement is always executed in the same amount of time, whatever the current values of the variables it processes. Clearly, the execution time of primes is given by the sum of the execution times of its three components. In what follows it is assumed that insert, belongs, and omit all take constant time (notice that for the array implementation for candidates (Section 1.3.2) this will be true, but set takes time proportional to n).

Create executes set once and insert exactly n - 1 times. Similarly, print executes its if statement n - 1 times. The amount of time the if statement takes depends on whether belongs is true or false but requires, at most, the time for belongs plus the time for printf.

Remove is more complex. Its initial assignment statement takes constant time, to which must be added the loop time. The loop is executed [image] times, so the loop condition is evaluated [image] times. Assuming, for simplicity, that sqrt (n) takes constant time C, then [image] is the total time for the evaluation of the condition. If the loop task took constant time, then the total loop time would be the product of this constant and [image]. Unfortunately delete's time varies, depending on the value of factor. Nonetheless, the total time for delete can still be derived.

Delete's own loop is executed for each multiple of factor between 2 x factor and m x factor, where m is the largest number of times that factor divides into n. m is the integer part of n/factor denoted by �n/factor�. Hence delete's loop is executed �n/factor� times for this value of factor. Since factor takes on values [image], the total number of delete's loop executions is

or

But 1/2 + 1/3 + . . . + 1/n is roughly the natural logarithm of n, denoted by ln n.Thus the entire time for the loop of delete is a constant (for omit and the assignment statement) times [image]. The condition of this loop contributes roughly its own constant times [image]. The initialization of nextmultiple in delete contributes its constant times [image] .

Adding everything, we get a good execution time estimate of

where C1, C2, C3, and C4 come from the times for condition evaluation, assignment statement execution, and set, insert, omit, and belongs.

For large n, this expression is dominated by the term C4n1n n. In fact, for large enough values of n, the expression will be no greater than C n1n n for an appropriate value of C. We describe this concisely by saying the expression is O(n1n n), read "order n1n n"; this notation will be used throughout the text.

By the storage requirements of primes is meant the total storage required for its data structures. Except for a few variables, the storage is required primarily for candidates and will be determined by its implementation. The analysis reveals that the bulk of the execution time for primes is spent executing the loop in delete. Consequently, to improve its time we speed up the time of the loop itself, or reduce the number of times it needs to be executed. For the array implementation, the storage required will be proportional to n, and this is where the bulk of the storage required is needed.

Primes can be improved in a number of obvious ways. Except for 2, no primes are even. Hence all other even numbers can be eliminated from consideration. This means that factor need never take on even values, so it can be increased by 2 instead of 1 in remove and can also be initialized to 3. Delete can be changed so that nextmultiple takes on only odd multiples of factor. With a little more work, candidates can be cut to roughly half its size by eliminating even integers. This will necessitate changes to set as well as changes to the for loops in create and print, but it reduces the storage needed by about one-half and the time by more than one-half.

Perhaps a less obvious improvement is to modify the increase of factor in remove so that, instead of increasing by 1 (or 2), factor increases to the value of the next prime in candidates (see Exercise 8). To realize this saving fully, it is necessary to use a new data structure, the list or the balanced binary tree, for the collection. Lists are discussed in Chapter 3, balanced binary trees in Chapters 7 and 9. These would save not only time but storage as well. Borrowing a result from mathematics about the distribution of prime numbers, we can state that any version of primes that stores all the primes from 2 to n in a collection will need n/ln n storage locations, since there are about that many such primes. Thus we will run out of storage before we run out of time!

Primes may be written so that its execution time increases but its storage needs decrease, or vice versa. Thus time is traded for storage, as is often done. Primes represents one extreme where all the primes are saved. A version of primes in which no primes are saved is at the other extreme (see Exercise 10). This version would require significantly more time.


Data Structures Using C: 1.8.2 Limits on Time and Storage

1.8.2 Limits on Time and Storage

Programs carry out certain basic operations. For example, primes executes assignment statements, evaluates conditions, and executes set, insert, omit, and belongs. In fact, its total execution time equals the number of times each of the operations is executed multiplied by the time taken by that operation. Determining the number of basic operations performed by a program to solve a problem allows evaluation of its practicality. For illustration, suppose that 1,000,000 basic operations can be carried out in 1 second by a computer. Then the number of basic operations that it can carry out in 1 year is

1,000,000 basic operations/second � 60 seconds/minute � 60 minutes/hour � 24 hours/day � 365 days/year

This is fewer than 1013 basic operations/year, and this is assuming that the computer is available for the exclusive use of the program and that it will never fail during the year. Thus 1013 can be taken as a benchmark figure�any program that requires a number of operations on this order to solve a problem is not feasible!

How much storage is available is determined by your computer system. The amount needed is surely not feasible when your program aborts during execution with a message indicating insufficient storage, or when the compiler tells you there is insufficient storage. If you run out of time or storage, you must then abandon the problem (and perhaps your job), find a more efficient program, or trade time for storage.


Data Structures Using C: 1.9: Case Study: Bowling Scores

1.9: Case Study: Bowling Scores

This section illustrates the way a flowchart and a program are developed for a problem connected with the game of bowling. The way in which the spirit of the structured programming philosophy is followed is more important than the particular development.

Bowling, as you may know, consists of ten frames. In each frame, there is a chance to knock down ten pins by throwing one or two balls down the bowling alley. To start, you roll the first ball of a given frame. If all ten pins are knocked down, a strike is achieved for that frame, and you earn a frame score of 10 plus the total number of pins knocked down on the next two rolls. If any pins remain standing, you roll a second ball for the frame. After this roll, if all pins are down, you have a spare, and a frame score of 10 plus the number of pins knocked down on the next roll. If any pins remain standing after the second roll, then the result is an open frame, and the frame score obtained is given by the total number of pins knocked down on the two rolls of the frame. A strike or spare occurring on the last (the tenth), results in two or one additional rolls, respectively. The total score for a game is the sum of all the individual frame scores.

Getting twelve strikes in a row results in the maximum attainable score of 300. In this case, twelve rolls are made. The least number of rolls possible is eleven and the greatest is twenty-one. In fact there is a correlation between the number of rolls in a game and the game score. Many rolls correspond to low scores and few rolls to high scores. The better a bowler you are, the fewer rolls you get!

Example 1.1

The problem is to print the total score for each of a series of games, as well as the minimum, average, and maximum scores. Assume the input is given as a series of roll scores for each game and includes at least one game.n

1.9.1 Algorithm

1.9.2 Program

1.9.3 Debugging


Data Structures Using C: 1.9.1 Algorithm

1.9.1 Algorithm

An initial flowchart might be as shown in Figure 1.5(a). If the three tasks can be carried out correctly, in the sequence indicated, then the problem is solved. The flowchart breaks down the original problem into three smaller problems. Obtaining their solutions and sequencing them correctly provides the solution to the original problem. This is the top-down approach to problem solving. Next, it is applied to each of the three problems.

(a) Initial Flowchart

(b) The Refined Flowchart

(c) A Loop for Processing the Games

Figure 1.5 Flowcharts of the Bowling Scores Algorithm

Problem 1 is transparent enough. Problem 3 is clearly solved by the expanded or refined flowchart shown in Figure 1.5(b).

Problem 2 needs to be refined. It seems clear that the basic structure for its solution is a loop, within which each game is processed. When all games have been processed, the loop should be exited. (See Figure 1.5(c).)

The new problem introduced as task 4 must be solved next. The input for this new problem will be the roll scores that must be processed for the new game. A basic looping structure can accomplish the task. The loop task will be executed ten times, once for each frame. Each execution calculates the score for the current frame and updates the game score and a frame counter, frame. Prior to entering the loop, f rame must be initialized to 1, and gamescore to 0. Other initialization may be required. After the loop is exited, the game score must be printed, and proper updating must occur. This refinement is shown in Figure 1.6.

Suppose that each time the programmer expands a task, the expanded version of the task does in fact solve the problem introduced by that task. Then the programmer knows that, whatever the current structured flowchart, it must represent a solution to the original problem as long as each of its tasks can be carried out correctly. The detailed structured flowchart of Figure 1.7 is thus a solution to the problem, as long as each of its tasks can be carried out correctly. It is detailed enough so that we know how to carry out (or program) each of its tasks except for 5, 6, 7, and 8. However, if the high-level language chosen for the program allows direct implementation of those tasks, then no more detail is needed to complete the solution.

Figure 1.6 Refinement of Task 4

Figure 1.7 Refinement of Initial Flowchart

Instead of increasing the complexity of the flowchart of Figure 1.7, tasks 5-8 can be functionally modularized.

Functions for tasks 7 and 8 appear in Figures 1.8 and 1.9. The solution to the original problem will then consist of Figure 1.7 plus the functions for tasks 5-8. This illustrates the application of structured programming to a relatively simple problem. Note again the importance of choosing variable names that convey information about the meaning and use of the variables. Good names make the algorithm clearer. Note that C compilers normally treat names as distinct only when they differ in the first eight characters, so some care must be taken. In this text, however, this constraint is ignored for clarity.

Figure 1.8 Refinement of Task 7

Figure 1.9 Refinement of Task 8

When task 7 is entered, the variables rollscore and r1 must contain, respectively, the first roll score of the current frame and the next roll score. Consequently, before entering task 6, the first two roll scores of the game must be read into rollscore and r1. This is the "other initialization" required of task 5.

The conditions "strike" and "spare" (Figure 1.8) are actually implemented as "rollscore equals 10" and "(rollscore + r1) equals 10," respectively. To emphasize the fact that complexity in a flowchart should be limited, Figure 1.10 is the complete detailed flowchart for the solution as it could appear without functional modularization. Descriptions do not appear; the chart simply indicates the structure of the solution. The flowchart can be derived by adding the detailed expansions of tasks 5-8 to Figure 1.7.

Figure 1.10 Too Detailed a Refinement


Data Structures Using C: 1.9.2 Program

1.9.2 Program

The program now follows directly from the flowcharts. Notice that its structure parallels the flowcharts', that we have attempted to minimize side effects by passing information between components explicitly as parameters, and that information needed only by a component is kept local to that component.

#include <stdio.h>

#define TRUE 1

#define FALSE 0

main()

/*

Bowling Scores calculates and prints the game scores

for a series of bowling games. It also prints the

number of games bowled, the minimum, average, and

maximum of all game scores. The input consists, for

each game, of a sequence of roll scores. Each roll

represents the number of pins knocked down on the

corresponding roll. Its input must consist of at least

one game.

Main Variables

average - average of all game scores

gamescore - score for the current game

max - current maximum game score

min - current minimum game score

number - current number of games played

total - accumulated total of all game scores

*/

{

int gamescore;

int number = 0, max = 0, min = 300;

int newgame = TRUE;

float average;

while (newgame)

{

gamescore = gamecalc(gamescore);

update(gamescore,&number,&min,&max,&total);

printf("If you want to bowl another game enter a 1\n");

printf("If you wish to stop enter a 0\n");

scanf("%d",&newgame);

}

average = ((float)total)/number;

printf("Minimum game %d\n",min);

printf("Average game %f\n",average);

printf("Maximum game %d\n",max);

printf("Number of games %d\n",number);

}

gamecalc (gamescore)

/* calculates gamescore for the current game.

Main variables

frame - the current frame

gamescore - score for the current game

rollscore - first roll score of the current frame

r1 - succeeding roll score after rollscore

*/

int gamescore;

{

int frame, rollscore, r1;

gamescore = 0;

frame = 1;

printf("Please enter the number of pins down on this roll\n");

scanf("%d",&rollscore);

printf("Please enter the number of pins down on this roll\n");

scanf("%d",&r1);

while (frame <= 10)

framecalc(&gamescore,&frame,&rollscore,&r1);

return(gamescore);

}

framecalc(pgamescore,pframe,prollscore,pr1)

/* Calculates the framescore for the current game,

adds it to gamescore, increases frame by 1 and

updates rollscore and r1.

Main variables

pframe - pointer to the current frame

pgamescore - pointer to the score for the current game

prollscore - pointer to first rollscore of current frame

pr1 - pointer to next rollscore of current frame

framescore - score for current frame

r2 - rollscore after r1

*/

int *pgamescore,*pframe,*prollscore,*pr1;

{

int framescore, r2;

if (*prollscore == 10) /* it is a strike */

{

printf("Please enter the number of pins down on this");

printf("roll\n");

scanf("%d",&r2);

framescore = 10 + *pr1 + r2;

*prollscore = *pr1;

*pr1 = r2;

}

else if ((*prollscore + *pr1) == 10) /* it is a spare */

{

printf("Please enter the number of pins down on this");

printf(" roll\n");

scanf("%d",&r2);

framescore = 10 + r2;

*prollscore = r2;

if (*pframe < 10)

{

printf("Please enter the number of pins down");

printf("on this roll\n");

scanf("%d",pr1);

}

}

else /* it is an open frame */

{

framescore = *prollscore + *pr1;

if (*pframe < 10)

{

printf("Please enter the number of pins");

printf("down on this roll\n");

scanf("%d",prollscore);

printf("Please enter the number of pins");

printf("down on this roll\n");

scanf("%d",pr1);

}

}

*pgamescore = *pgamescore + framescore;

*pframe = *pframe + 1;

}

update(gamescore,pnumber,pmin,pmax,ptotal)

/* This prints gamescore, and updates number, min, max,

and total to reflect gamescore.

Main variables

gamescore - current game score

pnumber - pointer to current number of games

pmin - pointer to current minimum game score

pmax - pointer to current maximum game score

ptotal - pointer to total of all game scores

*/

int gamescore, *pnumber, *pmin, *pmax;

int *ptotal;

{

printf("gamescore \n");

printf("%d\n",gamescore);

*pnumber = *pnumber + 1;

*ptotal = *ptotal + gamescore;

if (gamescore < *pmin)

*pmin = gamescore;

if (gamescore > *pmax)

*pmax = gamescore;

}


Data Structures Using C: 1.9.3 Debugging

1.9.3 Debugging

The following is a complete listing and output of a version of the Bowling Scores program that contains statements used as debugging aids (the boxed lines). This listing demonstrates how to use such statements and what they can tell us as we peruse the output. The roll scores of the two games that served as input are

game 1�5, 4, 6, 4, 2, 4, 1, 4, 3, 4, 8, 1, 9, 0, 10, 10, 9, 0

game 2�9, 1, 10, 7, 3, 9, 0, 10, 10, 5, 4, 6, 4, 10, 10, 10, 8

The prompts for input are in the program. They have been omitted from the output.

#include <stdio.h>

#define TRUE 1

#define FALSE 0

main()

/* Bowling Scores calculates and prints the game scores

for a series of bowling games. It also prints the

number of games bowled, the minimum, average, and

maximum of all game scores. The input consists, for

each game, of a sequence of roll scores. Each roll

represents the number of pins knocked down on the

corresponding roll. Its input must consist of at least

one game.

Main Variables

average - average of all game scores

gamescore - score for the current game

max - current maximum game score

min - current minimum game score

number - current number of games played

total - accumulated total of all game scores

*/

{

int gamescore;

int number = 0, max = 0, total = 0, min = 300;

int newgame = TRUE;

float average;

while (newgame)

{

printf("\n CALLING GAMECALC FOR GAME #%2d",number+1);

printf("\n-----------------------------\n");

gamescore = gamecalc(gamescore);

printf("-===>RETURNING FROM GAMECALC WITH GAMESCORE");

printf(" =%3d FOR GAME #%2d\n",gamescore, number+1);

printf("CALLING UPDATE WITH; MIN TOTAL MAX NUMBER\n");

printf("%26d%5d%5d%5d\n",min,total,max,number);

update(gamescore,&number,&min,&max,&total);

printf("If you want to bowl another game enter a 1 \n");

printf("If you wish to stop enter a 0 \n");

scanf("%d",&newgame);

printf("-===>RETURNING FROM UPDATE: MIN TOTAL MAX");

printf("NUMBER\n");

printf("%33d%6d%6d%5d\n",min,total,max,number);

}

average = ((float)total)\number;

printf("FINAL RESULTS FOR ALL GAMES FOLLON:\n");

printf("***********************************\n");

printf("  ACCUMULATED TOTAL = %5d\n\n",total);

printf("    MINIMUM GAME = %5d\n",min);

printf("    AVERAGE GAME = %10.4f\n",average);

printf("    MAXIMUM GAME = %5d\n\n",max);

printf("NUMBER OF GAMES PLAYED = %3d\n",number);

}

gamecalc(gamescore)

/* calculates gamescore for the current game.

Main variables

frame - the current frame

gamescore - score for the current game

rollscore - first roll score of the current frame

r1 - succeeding roll score after rollscore

*/

int gamescore;

{

int frame, rollscore, r1;

gamescore = 0;

frame = 1;

printf("Please enter the number of pins down on this roll\n");

scanf("%d",&rollscore);

printf("Please enter the number of pins down on this roll\n");

scanf("%d",&r1);

while (frame <= 10)

{

printf(" IN GAMECALC, GOING TO FRAMECALC WITH: GAMESCORE");

printf(" FRAME ROLLSCORE R1\n");

printf("%49d%9d%9d%7d\n",gamescore,frame,rollscore, r1);

framecalc(&gamescore,&frame,&rollscore,&r1);

printf(" IN GAMECALC, RETURNING FROM FRAMECALC WITH:");

printf("GAMESCORE FRAME ROLLSCORE R1\n");

printf("%53d%9d%9d%7d\n\n",gamescore,frame,rollscore,r1);

}

return(gamescore);

}

framecalc(pgamescore,pframe,prollscore,pr1)

/* Calculates the framescore for the current game,

adds it to gamescore, increases frame by 1 and

updates rollscore and r1.

Main variables

pframe - pointer to the current frame

pgamescore - pointer to the score for the current game

prollscore - pointer to first rollscore of current frame

pr1 - pointer to next rollscore of current frame

framescore - score for current frame

r2 - rollscore after r1

*/

int *pgamescore,*pframe,*prollscore,*pr1;

{

int framescore, r2;

printf(" STARTING FRAMECALC WITH: GAMESCORE FRAME");

printf(" ROLLSCORE R1\n");

printf("%36d%9d%9d%7d\n",*pgamescore,*pframe,*prollscore,*pr1);

if(*prollscore == 10) /* it is a strike */

{

printf("Please enter the number of pins down on this");

printf(" roll\n");

scanf("%d",&r2);

framescore = 10 + *pr1 + r2;

*prollscore = *pr1;

*pr1 = r2;

printf("AFTER STRIKE, FRAMESCORE = %2d\n", framescore);

}

else if ((*prollscore + *pr1) == 10) /* it is a spare */

{

printf("Please enter the number of pins down on this");

printf(" roll\n");

scanf("%d",&r2);

framescore = 10 + r2;

printf("AFTER SPARE, FRAMESCORE= %2d\n", framescore);

*prollscore = r2;

if (*pframe < 10)

{

printf("Please enter the number of pins down");

printf(" on this roll\n");

scanf("%d",pr1);

}

}

else /* it is an open frame */

{

framescore = *prollscore + *pr1;

printf("OPEN FRAME, FRAMESCORE %2d\n",framescore);

if (*pframe < 10)

{

printf("Please enter the number of pins");

printf(" down on this roll\n");

scanf("%d",prollscore);

printf("Please enter the number of pins");

printf(" down on this roll\n");

scanf("%d",pr1);

}

}

*pgamescore = *pgamescore + framescore;

*pframe = *pframe + 1;

printf(" LEAVING FRAMECALC WITH: GAMESCORE FRAME");

printf(" ROLLSCORE R1\n");

printf("%35d%9d%9d%7d\n\n",*pgamescore,*pframe,*prollscore,*pr1);

}

update(gamescore,pnumber,pmin,pmax,ptotal)

/* This prints gamescore, and updates number, min, max

and total to reflect gamescore.

Main variables

gamescore - current game score

pnumber - pointer to current number of games

pmin - pointer to current minimum game score

pmax - pointer to current maximum game score

ptotal - pointer to total of all game scores

*/

int gamescore, *pnumber, *pmin, *pmax;

int *ptotal;

{

printf("\n BEGINNING TO UPDATE WITH: GAMESCORE");

printf(" NUMBER MIN MAX TOTAL\n");

printf("%37d%8d%8d%4d%6d\n",

gamescore, *pnumber, *pmin, *pmax, *ptotal);

*pnumber = *pnumber + 1;

*ptotal = *ptotal + gamescore;

if (gamescore < *pmin)

*pmin = gamescore;

if (gamescore > *pmax)

*pmax = gamescore;

printf(" LEAVING UPDATE WITH: GAMESCORE NUMBER");

printf(" MIN MAX TOTAL\n");

printf("%35d%8d%8d%5d%6d\n\n",

gamescore,*pnumber,*pmin,*pmax,*ptotal);

}

The output of the program is as follows.

CALLING GAMECALC FOR GAME # 1

-----------------------------

  IN GAMECALC, GOING TO FRAMECALC WITH:  GAMESCORE  FRAME  ROLLSCORE  R1

                                             0        1        5       4

  STARTING FRAMECALC WITH:  GAMESCORE  FRAME  ROLLSCORE  R1

                                0        1        5       4

OPEN FRAME, FRAMESCORE = 9

  LEAVING FRAMECALC WITH:  GAMESCORE  FRAME  ROLLSCORE  R1

                               9         2       6       4

  IN GAMECALC, RETURNING FROM FRAMECALC WITH:  GAMESCORE  FRAME  ROLLSCORE  R1

                                                   9        2        6       4

  IN GAMECALC, GOING TO FRAMECALC WITH:  GAMESCORE  FRAME  ROLLSCORE  R1

                                             9        2        6       4

  STARTING FRAMECALC WITH:  GAMESCORE  FRAME  ROLLSCORE  R1

                                9        2        6       4

AFTER SPARE, FRAMESCORE = 12

  LEAVING FRAMECALC WITH:  GAMESCORE  FRAME  ROLLSCORE  R1

                               21       3        2       4

  IN GAMECALC, RETURNING FROM FRAMECALC WITH:  GAMESCORE  FRAME  ROLLSCORE  R1

                                                  21        3        2       4

  IN GAMECALC, GOING TO FRAMECALC WITH:  GAMESCORE  FRAME  ROLLSCORE  R1

                                             21       3        2       4

  STARTING FRAMECALC WITH:  GAMESCORE  FRAME  ROLLSCORE  R1

                                21       3        2       4

OPEN FRAME, FRAMESCORE = 6

  LEAVING FRAMECALC WITH:  GAMESCORE  FRAME  ROLLSCORE  R1

                               27       4        1       4

  IN GAMECALC, RETURNING FROM FRAMECALC WITH:  GAMESCORE  FRAME  ROLLSCORE  R1

                                                   27       4        1       4

  IN GAMECALC, GOING TO FRAMECALC WITH:  GAMESCORE  FRAME  ROLLSCORE  R1

                                             27       4        1       4

  STARTING FRAMECALC WITH:  GAMESCORE  FRAME  ROLLSCORE  R1

                                27       4        1       4

OPEN FRAME, FRAMESCORE = 5

  LEAVING FRAMECALC WITH:  GAMESCORE  FRAME  ROLLSCORE  R1

                               32       5        3       4

  IN GAMECALC, RETURNING FROM FRAMECALC WITH:  GAMESCORE  FRAME  ROLLSCORE  R1

                                                   32       5        3       4

  IN GAMECALC, GOING TO FRAMECALC WITH:  GAMESCORE  FRAME  ROLLSCORE  R1

                                             32       5        3       4

  STARTING FRAMECALC WITH:  GAMESCORE  FRAME  ROLLSCORE  R1

                                32       5        3       4

OPEN FRAME, FRAMESCORE = 7

  LEAVING FRAMECALC WITH:    GAMESCORE  FRAME  ROLLSCORE  R1

                                 39       6        8       1

  IN GAMECALC, RETURNING FROM FRAMECALC WITH:  GAMESCORE  FRAME  ROLLSCORE  R1

                                                   39       6        8       1

  IN GAMECALC, GOING TO FRAMECALC WITH:  GAMESCORE  FRAME  ROLLSCORE  R1

                                             39       6        8       1

  STARTING FRAMECALC WITH:  GAMESCORE  FRAME  ROLLSCORE  R1

                                39       6        8       1

OPEN FRAME, FRAMESCORE = 9

  LEAVING FRAMECALC WITH:    GAMESCORE  FRAME  ROLLSCORE  R1

                                 48       7        9       0

  IN GAMECALC, RETURNING FROM FRAMECALC WITH:  GAMESCORE  FRAME  ROLLSCORE  R1

                                                   48       7        9       0

  IN GAMECALC, GOING TO FRAMECALC WITH:  GAMESCORE  FRAME  ROLLSCORE  R1

                                            48        7        9       0

  STARTING FRAMECALC WITH:  GAMESCORE  FRAME  ROLLSCORE  R1

                                48       7        9       0

OPEN FRAME, FRAMESCORE = 9

  LEAVING FRAMECALS WITH:  GAMESCORE  FRAME  ROLLSCORE  R1

                              57        8        10     10

  IN GAMECALC, RETURNING FROM FRAMECAL WITH:  GAMESCORE  FRAME  ROLLSCORE  R1

                                                 57        8        10     10

  IN GAMECALC, GOING TO FRAMECALC WITH:  GAMESCORE  FRAME  ROLLSCORE  R1

                                             57       8        10     10

  STARTING FRAMECALC WITH:  GAMESCORE  FRAME  ROLLSCORE  R1

                               57        8        10     10

AFTER STRIKE, FRAMESCORE = 29

  LEAVING FRAMECALC WITH:  GAMESCORE  FRAME  ROLLSCORE  R1

                              86        9        10      9

  IN GAMECALC, RETURNING FROM FRAMECALC WITH:  GAMESCORE  FRAME  ROLLSCORE  R1

                                                  86        9        10      9

  IN GAMECALC, GOING TO FRAMECALC WITH:  GAMESCORE  FRAME  ROLLSCORE  R1

                                            86        9        10      9

  STARTING FRAMECALC WITH:  GAMESCORE  FRAME  ROLLSCORE  R1

                               86        9        10      9

AFTER STRIKE, FRAMESCORE = 19

  LEAVING FRAMECALC WITH:  GAMESCORE  FRAME  ROLLSCORE  R1

                              105       10       9       0

  IN GAMECALC, RETURNING FROM FRAMECALC WITH:  GAMESCORE  FRAME  ROLLSCORE  R1

                                                   105      10       9       0

  IN GAMECALC, GOING TO FRAMECALC WITH:  GAMESCORE  FRAME  ROLLSCORE  R1

                                            105       10       9       0

  STARTING FRAMECALC WITH:  GAMESCORE  FRAME  ROLLSCORE  R1

                               105       10       9       0

OPEN FRAME, FRAMESCORE = 9

  LEAVING FRAMECALC WITH:  GAMESCORE  FRAME  ROLLSCORE  R1

                              114       11       9       0

  IN GAMECALC, RETURNING FROM FRAMECALC WITH:  GAMESCORE  FRAME  ROLLSCORE  R1

                                                  114      11        9       0

- = = = > RETURNING FROM GAMECALC WITH GAMESCORE = 114 FOR GAME # 1

CALLING UPDATE WITH;  MIN  TOTAL  MAX  NUMBER

                      300    0     0     0

  BEGINNING TO UPDATE WITH:  GAMESCORE  NUMBER  MIN  MAX  TOTAL

                                 114      0     300   0     0

    LEAVING UPDATE WITH:  GAMESCORE  NUMBER  MIN  MAX  TOTAL

                             114        1    114  114   114

- = = = > RETURNING FROM UPDATE:  MIN  TOTAL  MAX  NUMBER

                                  114   114   114     1

CALLING GAMECALC FOR GAME # 2

-----------------------------

  IN GAMECALC, GOING TO FRAMECALC WITH:  GAMESCORE  FRAME  ROLLSCORE  R1

                                             0        1        9       1

  STARTING FRAMECALC WITH:  GAMESCORE  FRAME  ROLLSCORE  R1

                                0        1        9       1

AFTER SPARE, FRAMESCORE = 20

  LEAVING FRAMECALC WITH:  GAMESCORE  FRAME  ROLLSCORE  R1

                               20       2        10      7

  IN GAMECALC, RETURNING FROM FRAMECALC WITH:  GAMESCORE  FRAME  ROLLSCORE  R1

                                                   20       2        10      7

  IN GAMECALC, GOING TO FRAMECALC WITH:  GAMESCORE  FRAME  ROLLSCORE  R1

                                             20       2        10      7

  STARTING FRAMECALC WITH:  GAMESCORE  FRAME  ROLLSCORE  R1

                                20       2        10      7

AFTER STRIKE, FRAMESCORE = 20

  LEAVING FRAMECAL WITH:  GAMESCORE  FRAME  ROOLSCORE  R1

                              40       3        7       3

  IN GAMECALC, RETURNING FROM FRAMECALC WITH:  GAMESCORE  FRAME  ROLLSCORE  R1

                                                   40       3        7       3

  IN GAMECALC, GOING TO FRAMECALC WITH:  GAMESCORE  FRAME  ROLLSCORE  R1

                                             40       3        7       3

  STARTING FRAMECALC WITH:  GAMESCORE  FRAME  ROLLSCORE  R1

                                40       3        7       3

AFTER SPARE, FRAMESCORE = 19

  LEAVING FRAMECALC WITH:  GAMESCORE  FRAME  ROLLSCORE  R1

                               59       4        9       0

  IN GAMECALC, RETURNING FROM FRAMECALC WITH:  GAMESCORE  FRAME  ROLLSCORE  R1

                                                   59       4        9       0

  IN GAMECALC, GOING TO FRAMECALC WITH:  GAMESCORE  FRAME  ROLLSCORE  R1

                                             59       4        9       0

  STARTING FRAMECALS WITH:  GAMESCORE  FRAME  ROLLSCORE  R1

                                59       4        9       0

OPEN FRAME, FRAMESCORE = 9

  LEAVING FRAMECALC WITH:  GAMESCORE  FRAME  ROLLSCORE  R1

                               68       5        10     10

  IN GAMECALC, RETURNING FROM FRAMECALC WITH:  GAMESCORE  FRAME  ROLLSCORE  R1

                                                   68       5        10     10

  IN GAMECALC, GOING TO FRAMECALC WITH:  GAMESCORE  FRAME  ROLLSCORE  R1

                                             68       5           10     10

  STARTING FRAMECALC WITH:  GAMESCORE  FRAME  ROLLSCORE  R1

                               68        5        10     10

AFTER STRIKE, FRAMESCORE = 25

  LEAVING FRAMECALC WITH:  GAMESCORE  FRAME  ROLLSCORE  R1

                              93        6        10      5

  IN GAMECALC, RETURNING FROM FRAMECALC WITH:  GAMESCORE  FRAME  ROLLSCORE  R1

                                                  93        6        10      5

  IN GAMECALC, GOING TO FRAMECALC WITH:  GAMESCORE  FRAME  ROLLSCORE  R1

                                            93        6        10      5

  STARTING FRAMECALC WITH:  GAMESCORE  FRAME  ROLLSCORE  R1

                                93       6        10      5

AFTER STRIKE, FRAMESCORE = 19

  LEAVING FRAMECALC WITH:  GAMESCORE  FRAME  ROLLSCORE  R1

                              112       7        5       4

  IN GAMECALC, RETURNING FROM FRAMECALC WITH:  GAMESCORE  FRAME  ROLLSCORE  R1

                                                  112       7        5       4

  IN GAMCALC, RETURNING FROM FRAMECALC WITH:  GAMESCORE  FRAME  ROLLSCORE  R1

                                                 112       7        5       4

  STARTING FRAMECALC WITH:  GAMESCORE  FRAME  ROLLSCORE  R1

                               112       7        5       4

OPEN FRAME, FRAMESCORE = 9

  LEAVING FRAMECALC WITH:  GAMESCORE  FRAME  ROLLSCORE  R1

                              121       8        6       4

  IN GAMECALC, RETURNING FROM FRAMECALC WITH:  GAMESCORE  FRAME  ROLLSCORE  R1

                                                  121       8        6       4

  IN GAMECALC, GOING TO FRAMECALC WITH:  GAMESCORE  FRAME  ROLLSCORE  R1

                                            121       8        6       4

  STARTING FRAMECALC WITH:  GAMESCORE  FRAME  ROLLSCORE  R1

                               121        8       6       4

AFTER SPARE, FRAMESCORE = 20

  LEAVING FRAMECALC WITH:  GAMESCORE  FRAME  ROLLSCORE  R1

                              141       9        10     10

  IN GARAMECALC, RETURNING FROM FRAMECALC WITH: GAMESCORE FRAME ROLLSCORE R1

                                                   141      9       10    10

  IN GAMECALC, GOING TO FRAMECALC WITH:  GAMESCORE  FRAME  ROLLSCORE  R1

                                            141       9        10     10

  STARTING FRAMECALC WITH:  GAMESCORE  FRAME  ROLLSCORE  R1

                               141       9        10     10

AFTER STRIKE, FRAMESCORE = 30

  LEAVING FRAMECALC WITH:  GAMESCORE  FRAME  ROLLSCORE  R1

                              171       10       10     10

  IN GAMECALC, RETURNING FROM FRAMECALC WITH:  GAMESCORE  FRAME  ROLLSCORE  R1

                                                  171       10       10     10

  IN GAMECALC, GOING TO FRAMECALC WITH:  GAMESCORE  FRAME  ROLLSCORE  R1

                                            171       10       10     10

  STARTING FRAMECALC WITH:  GAMESCORE  FRAME  ROLLSCORE  R1

                               171       10       10     10

AFTER STRIKE, FRAMESCORE = 28

  LEAVING FRAMECALC WITH:  GAMESCORE  FRAME  ROLLSCORE  R1

                              199      11        10      8

  IN GAMECALC, RETURNING FROM FRAMECALC WITH:  GAMESCORE  FRAME  ROLLSCORE  R1

                                                  199      11        10      8

- = = = > RETURNTING FROM GAMECALC WITH GAMESCORE = 199 FOR GAME # 2

CALLING UPDATE WITH:  MIN  TOTAL  MAX  NUMBER

                      114   114   114    1

  BEGINNING TO UPDATE WITH:  GAMESCORE  NUMBER  MIN  MAX  TOTAL

                                 199      1     114  114   114

    LEAVING UPDATE WITH:  GAMESCORE  NUMBER  MIN  MAX  TOTAL

                              199      2     114  199   313

- = = = > RETURNING FROM UPDATE:  MIN  TOTAL  MAX  NUMBER

                                  114   313   199    2

FINAL RESULTS FOR ALL GAMES FOLLOW:

***********************************

     ACCUMULATED TOTAL =  313

          MINIMUM GAME =  114

         AVERAGE GAME  =  156.5000

          MAXIMUM GAME =  199

NUMBER OF GAMES PLAYED =  2


Data Structures Using C: 1.10: Summary

1.10: Summary

Structure is important for algorithms, programs, and data. Structured programming is a philosophy as well as a defined methodology to guide the construction of algorithms and programs. It involves a top-down approach: starting with a given problem, then using stepwise refinement, and breaking problems into smaller component problems that then may be solved independently and subsequently combined. The overall goal is to write programs that are easily understood, debugged, and changed. Data abstraction and functional modularity are important elements of programming style to be incorporated into this approach. Algorithm and program structure have been stressed in this chapter and will continue to be emphasized throughout the book, as will the equally important choice of data structures. The next chapter discusses record and array information and pointers.


Data Structures Using C: Exercises

Exercises

1. In Figure 1.1 suppose there are n = 100 houses and you must visit houses 3, 10, 50, 7, and 82 (in this order). How much distance must you travel in each town?

2. Why do no more nonprimes remain in candidates when factor > [image] ?

3. Explain the difference between a function and program segment implementation of a refinement.

4. a. What determines the largest value of n for which each of the two array implementations for candidates will work?

b. Discuss any significant differences between set, insert, omit, and belongs for the two implementations of candidates.

c. Modify the second implementation so that it works for larger n�say 20 times its current limit.

d. Write a function copy with parameters c1 and c2. It must treat them as data abstractions with the operations set, insert, omit, and belongs and is to return with c2's contents identical to c1's and c1's contents unaltered.

5. Change primes so that even numbers are not considered as values for factor or nextmultiple.

6. Change your solution to Exercise 5 so that even numbers, except 2, do not appear in candidates. Use only enough storage for 2 and the odd numbers between 2 and n.

7. This is the same as Exercise 6 except that candidates must use only enough storage for 2 and the odd primes between 2 and n.

8. Change remove so that instead of increasing factor by 1, factor is increased to the value of the next prime in candidates. You may choose to do this by starting with remove from your solution to Exercise 7.

9. Analyze the time and storage required for your solutions to Exercises 5 to 8.

10. For the prime problem, find a solution that stores no primes.

11. Explain why we must know the data type of the information stored in variables A and B in order to multiply their contents correctly.

12. Suppose that integers are represented as in the example of Section 1.3.2. Write an algorithm, in English, that will allow someone who knows addition tables up to 9 to compute the correct sum of the contents of A and B. The algorithm must yield the correct result no matter what integers are stored in A and B.

13. This is the same as Exercise 12 except that reals are represented, and the algorithm should yield the correct sum represented as a real number.

14. Express the Bowling Scores program just as the prime number solution is expressed in Figure 1.3. What are its local, nonlocal, and global variables?

15. Modify the flowcharts and program for the bowling problem to obtain flowcharts and a program solving the same problem but also printing out the number of rolls for each game.

16. What functions might be inserted in Bowling Scores to check the validity of its input and output? Where would they go in the program?

17. What elements of style have been incorporated in the programs of Sections 1.9.2 and 1.9.3?


Data Structures Using C: Suggested Assignments

Suggested Assignments

1. Write and run a program to print all perfect numbers no greater than n. A number is perfect if it is equal to the sum of its factors; thus 6 is a perfect number (6 = 1 + 2 + 3). The program is to create a collection candidates of all candidates for perfect numbers, remove those that are not perfect from candidates, and print the numbers remaining in it. The program must treat the collection with operations set, insert, omit, and belongs as a data abstraction.

2. Write a program to read in integers, store them in a collection, then read in more integers, remove each of these from it, and finally print all the integers remaining in it. The program is to treat the collection of integers as a data abstraction with the operations of set, insert, omit, and belongs.

3. a. Why does the Bowling Scores program require that the input consist of at least one game?

b. Change the program so that it handles the case of no games as input in a reasonable way.


Data Structures Using C: CHAPTER 2: RECORDS, ARRAYS, AND POINTERS

CHAPTER 2: RECORDS, ARRAYS, AND POINTERS

Presents the first serious consideration of data structuring

Focuses on

two data structures for grouping related information

records and arrays

two important operations on arrays

randomly accessing an array element

traversing an array

pointers and their use

as indices to arrays

to access information stored in dynamic memory

pointer arrays

their application and benefits

representations of two-dimensional arrays

to show different implementations

to evaluate the benefits of these implementations

Case study�stable marriages

highlights the benefits of carefully selecting both the information to be stored

and the data structure in which to store it

2.1: Bricks

2.2: Mortar

2.3: Representations of Two-Dimensional Arrays

2.4: Advantages and Disadvantages of the Techniques

2.5: Case Study: Stable Marriages

Exercises

Suggested Assignments


Data Structures Using C: 2.1: Bricks

2.1: Bricks

Now that the context in which problems will be solved and programs written has been presented, it is time to start studying seriously the effect of choice of data structure on the problem solution. Before we explore some simple examples, the two basic data structures�records and arrays�must be understood, along with the two basic operations on them�selection and traversal.

A record is a data structure made up of a fixed number of information items, not necessarily of the same type. In C records are of the type structure. In the text the two words record and structure will be used interchangeably. An array is a data structure made up of a fixed number of information items in sequence; all items are required to be of the same type. Records and arrays have boundless applications. Records are useful when the items are to be referred to by name, while arrays are useful when the items are to be referred to by their positions in the sequence. The fundamental importance of the array was recognized quite early in the development of programming languages; the structure, or record, has only recently been incorporated in many languages. Pointers are variables that

1. Refer or point to memory locations for variables

2. Point to data structures, or

3. Link the components of a data structure together

Records and arrays are the bricks out of which more complex data structures are built, and pointers are the mortar holding them together.

Just as a builder asks for bricks and mortar, programmers in a high-level language ask for data structures to help build their solutions. A data structure is specified by declaring a variable name to be of a specific type. Some languages restrict the choice of type to individual variables or to arrays of type int, float, or char. C includes these, plus other types such as records, pointers, and unions. However, it does not have type boolean, which is found in other languages. C does allow the user to give new names to types.

Records and arrays allow related items of information to be treated as a group. The individual pieces of information in a record (structure) are called members in C. In most languages they are termed fields, but this word has a special meaning in C. Since that meaning is never utilized in this text, both member and field will be used for the elements of a record. In the case of arrays the individual pieces of information are called entries. The merit of records and arrays lies in the fact that they allow random access to any one of their fields or entries. Random access is accomplished by direct location of the desired piece of information, without regard to sequence. This means that access to an item may be accomplished in a fixed amount of time, no matter what item was previously accessed. This contrasts with sequential access, where the time required to access an item depends on the number of items stored between that item and the last item accessed, since each of the intermediate items must be accessed. With random access, any member of a record or any entry of an array may be accessed in a fixed amount of time, typically a few millionths or billionths of a second.

Both records and arrays provide the means to treat related items as a unit, and both generalize the idea of variables containing one item to variables containing many items. Records will be discussed next, then arrays.

2.1.1 Structures or Records in C

2.1.2 Arrays in C

2.1.3 Two-Dimensional Arrays

2.1.4 Representation of Records in Arrays

2.1.5 Variable-Length Records

2.1.6 The Use of Unions in C

2.1.7 Address Calculation without Access


Data Structures Using C: 2.1.1 Structures or Records in C

2.1.1 Structures or Records in C

In C, memory can be thought of as consisting of storage elements of different length and complexity. Their complexity is determined by their type. The simplest storage elements have one of the three basic types, int, float, and char. Up one level in complexity are storage elements that have type structure. Among these, the simplest have exactly one member, which in turn has one of the basic types. A storage element of type structure for a book would be more complex; it might include members for its title, author, cost, and number of pages. Members are not restricted to one of the basic types but may themselves have type structure.

Consider the following C declarations:

float amount;     struct book               struct book text;

{

char title[35];

struct name author;

float cost;

int pages;

};

struct balance    struct name

{                 {

    >Comment

float amount;     char last[15];

}bankbalance;         char middleinit;

char first[15];

}

Storage for the three variables amount, bankbalance, and text may be pictured as in Figure 2.1. From the figure it is clear that storage must differ in complexity for different variables. The simplest is obviously the one that stores only one piece of information, as does the storage element for the variable amount. The structure bankbalance is obviously more complex; it could also be made longer by adding more fields such as lastdeposit and lastwithdrawal. Of course, more memory is needed to store the variable text than to store bankbalance. The type structure generalizes the idea of an individual variable containing only one piece of information to variables containing related pieces of information. The amount of storage required determines the length of a record, while the record's structure determines its complexity.

It is now possible to write C statements that refer to the entire record text, its author field, or its last field, using the names text,text.author, and text.author.last, respectively. In earlier versions of C it was not possible to write statements that referred to entire structures.

Figure 2.1 Three Variables


Data Structures Using C: 2.1.2 Arrays in C

2.1.2 Arrays in C

Arrays provide another way to group either individual items or records. An array stores a sequence of entries; an entry in an array is referred to and located by its position in the sequence. An array has a fixed size and is homogeneous (that is, its entries must all be of the same type�say, of type integer or book). This contrasts with a record, whose fields may be of different types. An array entry is identified by qualifying the array name by the entry's position. To identify a member in a structure, the structure name is qualified by the member name. (Note that arrays can have entries of type structure, and structures can have members of type array.)

Consider the following declarations:

int book_number[80];

struct name author[40];

float prices[80];

char library[30];

The declaration specifies the name of each variable (e.g., book_number) and its type for each of four variables. It also specifies the range of values for an array index for each of the four arrays. An array index refers to a position of the array. Book_number,author,prices, and library have array indexes that may take on values in the range 0 to 79, 0 to 39, 0 to 79, and 0 to 29, respectively. In C the first position in an array is always the zero position. Book_number[3], author[9], and prices[5] refer to the fourth, tenth, and sixth entries, respectively, and library[12] refers to the thirteenth character in the character string library.

It has already been mentioned that arrays are very useful when entries must be accessed randomly. This occurs when the order in which requests for access to entries cannot be predicted. Suppose an array stores the initial mileage for each car of a fleet of rental cars. As a car is returned, its initial mileage is needed to prepare a bill based on mileage used. The next car to be returned can be any rented car. To access its initial mileage directly requires the random access capability of arrays. Processing time is saved, since the information entry is located directly�no time is lost searching through other entries to find it. When information is accessed directly, in a constant time, it is said to be selected. Thus any array entry whose position is known can be selected.

Instead of accessing a single array entry, a programmer may wish to process an entire array by accessing and processing each entry in turn. Moving sequentially through the entries is a traverse. In this case, the program is said to traverse the array. Such a traversal would be required in order to print out, in order, the initial mileage of each car in the fleet. A complete traversal is accomplished when each entry has been processed exactly once. A simple loop, starting at the first entry and finishing at the last, will accomplish this. Another use of traversal is to find an array entry whose position is unknown. In this case, the loop is terminated when the entry is found. If the traversal is completed, it means the desired entry was not present.

Selection and traversal are two ways to access an array entry. If the entry's position is known, selection is the efficient way to access it. When successive array positions must be examined to find the entry, selection is not possible, but traversal may be used.

If the traversal terminates at the ith entry (because it is the desired one), then a time proportional to i is required, since the i - 1 preceding entries must be accessed before the ith entry is obtained. Selection, on the other hand, takes constant time. This is an important distinction between these two basic operations on arrays. Selection means you can go directly to what is wanted; traversal means you must rummage around to find it. Use selection when one or more accesses must be made in arbitrary order and traversal when each entry is to be accessed in sequence.

Example 2.1

Suppose that data, such as integers in the range 1 to 100, are stored in an array a, as shown in Figure 2.2. Given any valid index value in the variable i, print the number stored in the ith position of the array a. n

Figure 2.2 The Array for Example 2.1

This may be done by executing the C statement

printf("%d\n",a[i]);

This is a concise, easy-to-understand solution to Example 2.1. It takes a constant amount of time to execute, since it selects the desired entry of a. Note that when the value of the index i is 7, then the array entry with the value 87 is printed; when the value of i is 5, then the entry with the value 3 is printed. The index i is avariable that points to a particular entry of a. The current value of the index i determines the entry currently selected. Changing the value of i changes the entry to which it points. Thus the solution can be interpreted as the command "Print the entry of a to which the index i currently points."

Example 2.2

Let a be an array of length n, and let k be a variable containing an integer in the range 1 to 100. If there is some value of i for which a[i] contains k, then print that value of i; if there is more than one such value, print any one of them. If the value of k is nowhere in a, then print "-1." n

Array a might represent the information kept by a car rental agency. For each i, array entry a[i] contains the account number, k, of the individual or firm currently renting car i. Example 2.1 thus requires finding the account now leasing car i. Example 2.2 requires finding the car being used by account k.

Example 2.2 could certainly be solved by traversing the array a, searching for an entry whose value equals the value of k. If one is found, its position in a would be printed. If the traversal is completed without success, "-l" would be printed. This solution is easy to code and could be written as follows:

found = FALSE;

loc = 0;

>Comment

while (found != TRUE && loc <= n)

if (a[loc] != k)

loc++;

else

found = TRUE;

if (found)

printf("\n %d\n",loc);

else

printf("\n - 1 \n");

Although this program segment for array traversal is simple, it is more complex than that for Example 2.1. Its execution time is proportional to the number of entries searched before the number in k is found, and if it is not found, the execution time is proportional to n.

Is it possible to find another way to solve this problem, one that will be as concise and clear as that for Example 2.1 and also execute in constant time? The answer is yes, if the information stored in a is represented in such a way that the program can select the correct value to print.

Such a program would start with creation of an array pointer of length 101, as follows. For each integer k (account number) between 1 and 100, if k does not appear in a, place "-1" in pointer[k]. If k does appear in a (it is an active account), place in pointer[k] the position of an entry of a that contains k. That is, place in pointer[k] the index value of an entry in a containing k. Thus, in Figure 2.3, pointer[3] contains 5, since a[5] contains 3.

With the array pointer available, the solution to Example 2.2 may be written as

printf("%d\n",pointer[k]);

This is the concise, easy-to-understand solution desired, and it executes in constant time. Again, the way in which the relevant information was represented in the program�that is, the way in which the data was structured�allows this solution to be achieved. Even in such a simple example, the way data is represented has a substantial impact on the resulting program.

Note that the use of the pointer array instead of a may cause some information available in a to be lost. This is because pointer captures only one of a's positions that contains a specific number, even though that number may appear in a more than once. Finally, a price in execution time must be paid to create pointer from a in the first place. However, if the solution to Example 2.2 needed to be executed repeatedly, for many values of k, then it probably would be worth the price. It is frequently the case that prior processing to extract or restructure data saves later processing time.

Figure 2.3 A Pointer Array for Example 2.2

In terms of storage required, a takes up n storage entries, and pointer takes up 101 storage entries. If n is much larger than 101, then pointer even saves storage; if n is much smaller, it requires more storage. Thus it is possible to trade time for storage. It will become apparent that this is generally true: saving execution time requires more storage, and vice versa.

When the choice of data structure was the array a, Example 2.1 was solved using selection, but traversal was required to find a solution to Example 2.2. When the pointer array was chosen to represent information, selection could be used to solve Example 2.2 more efficiently. However, if we had wanted to know all the cars leased by account k, pointer would not have been sufficient. It would have been necessary to traverse a to find all the occurrences of account number k. Can you find a way to avoid this traversal?


Data Structures Using C: 2.1.3 Two-Dimensional Arrays

2.1.3 Two-Dimensional Arrays

Until now we have discussed one-dimensional arrays. High-level languages also provide arrays of more than one dimension. Thus the C declaration

int a[5][3];

defines a two-dimensional array of 5 � 3 or 15 entries, each of type integer. This array a can also be viewed as a collection of 5 one-dimensional arrays, each of length 3, or 3 one-dimensional arrays, each of length 5. Again, we can think of two variables i and j of the correct index type as pointing to an entry of the array. Thus a[i][j] refers to the entry of the ith row and jth column, or to the entry of a currently pointed to by i and j. We assume that the (i,j)th entry can be accessed in constant time, no matter what the value of i or j . That is, the (i,j)th entry may be selected. Traversal of two-dimensional arrays is also easy.

A two-dimensional array is symmetric if a[i][j] and a[j][i] contain the same value for all i and j between 0 and n - 1. Suppose a collection of n2 numbers, representing the distances between n(�50) cities, is stored in a two-dimensional array a. Thus a[i][j] contains the distance between city i and city j. For this example, which involves distances, the array must be symmetric. However, the array a shown in Figure 2.4 is not symmetric, because a[2][3] does not equal a[3][2]. The n entries a[0][0], a[1][1],..., a[n - 1][n - 1] are called its diagonal entries. In the city-distance array, the diagonal entries must, of course, be zero.

Figure 2.4 An Asymmetric Two-Dimensional Array

Example 2.3

Suppose a programmer wished, perhaps as part of an input validation function for the data of an array, to check the array for symmetry. The programmer might choose to write a function check to return the value true if the array is symmetric and false otherwise. A simple traversal, as follows, will do the checking. n

>Comment

typedef int collection[50][50];

check(a,n)

/* Returns true only if the first n

rows and columns of a represent

a symmetric array

*/

>Comment

collection a;

int n;

{

>Comment

int i,j,ck;

ck = TRUE;

>Comment

for (i = 0; i<n; i++)

for (j = 0; j<n; j++)

if (a[i][j] != a[j][i])

ck = FALSE;

return(ck);

}

This program does a complete traversal of the appropriate n entries of the array a. However, it is inefficient as a solution for a number of reasons. First, it does not exit immediately from the loops, even though it may have just found that a was not symmetric. Second, it checks the diagonal elements, which will always satisfy a[i][j] = a[j][i]. Finally, it double-checks each off-diagonal entry. For instance, when i = 2 and j = 3, it checks a[2][3] against a[3][2], and when i = 3 and j = 2, it checks a[3][2] against a[2][3] again. A better version would be a modified traversal, such as

typedef int collection[50][50];

check(a,n)

/* Returns true only if the first n

rows and columns of a represent

a symmetric array

*/

collection a;

int n;

{

int i,j,ck;

ck = TRUE;

>Comment

for(i=1; i<n && ck; i++)

for(j=0; j <i && ck; j++)

if(a[i][j]! = a [j][i])

ck = FALSE;

return(ck);

}

Be sure you understand why the modified lines of code eliminate the inefficiencies.


Data Structures Using C: 2.1.4 Representation of Records in Arrays

2.1.4 Representation of Records in Arrays

In C the record, or structure, is a basic data structure. In some languages the record is not a built-in feature but must be constructed from the data structures (such as arrays) that are available. To see how this construction might be done, assume C designers had provided the integer, float and char data types, and only one-dimensional arrays. Not only would records have to be constructed, but so would arrays of more dimensions if they were needed. There are two reasons for looking at ways to build data structures.

1. It affords insight into how compilers implement records and higher-dimensional arrays.

2. It demonstrates how programmers build new structures from those that are given in order to tailor data structures to specific needs.

Moreover, even though records and higher-dimensional arrays are available in C, there are times when programmers may find it preferable to use these construction techniques.

Let us consider how to represent n records, each composed of m fields of the same fixed type. Suppose m is 3. A natural solution is to take an array of length 3�n, of the same type as the fields, and store the records in the array. The first record takes up the first three entries, the second record the next three entries, and so on, as in Figure 2.5. Each field of a record will be stored in the corresponding one of its three entries.

The ith record begins in position 3*(i - 1) of the array. Thus we may select the ith record. To access a specific field of the ith record (say, the third) requires that position 3*(i - 1) + 2 of the array be accessed. Although this is the (3*(i - 1) + 2)th array element, it contains the third field value of the ith record. Thus we may select any field of any record too.

It is also easy to traverse the records, using a loop. For example, the following function traverses the n records stored in the array a and prints the value in their kth field, where k may be 1, 2, or 3.

>Comment

typedef int fieldtype;

array is the name for the type: array of fieldtype of size 30

typedef fieldtype array [30];

printfield(a,n,k)

/* Prints the contents of the

kth field of each of the first

n records stored in array a.

*/

>Comment

array a;

int n,k;

{

>Comment

int length = 3, i;

for(i=0; i<n; i++)

printf("\n %d\n",a[length * i + (k-1)]);

}

In C, since arrays must be homogeneous (consisting of entries of the same type), this method is applicable only to homogeneous records�those whose fields are all of the same type. How then to handle the more general case of records whose fields are not homogeneous? The following example demonstrates the general concept and then how it is achieved in C.

Example 2.4

Consider Figure 2.6, which represents an array cars containing a record for each car of an automobile rental agency. The record for each car has an entry for the make (characters), monthly rate (a real number), current mileage (an integer), and the rentee (characters). As noted, this array, with entries of different types, could not be constructed in C. For the sake of simplicity, the following discussion does not deal with the actual storage taken by a character, float, or integer variable, but assumes that each takes one element of memory. All the records in the array are 42 elements long, using 15 elements for the make, 1 element each for the monthly rate and current mileage, and 25 elements for the rentee field. n

Figure 2.5 An Array of Records (of Length 3)

Figure 2.6 Records of a Car Rental Agency Stored in an Array CARS

An offset gives the position of a data item with respect to a given reference point. When you say, "the third house from the corner," you are using an offset reference point. Each field has a field offset from the beginning of the record. In this case, the field offsets are 0, 15, 16, and 17, respectively, for make, monthly rate, current mileage, and rentee. The ith record has a record offset of 42�(i - 1) from the beginning of the array. The offset for its rentee field from the beginning of the array is 42�(i - 1) + 17, the sum of the record offset for the ith record and the field offset for the rentee.

In order to reference, for example, the current mileage field of the ith record, the programmer may refer to cars[42*(i-1)+16], assuming the first record starts in element zero of the array. Only the fields may be referred to by the programmer; the record itself cannot be dealt with directly as a unit. In C the task is simplified for the programmer by the use of structures and arrays of structures. Carrecord is defined as a structure and cars[100] as an array of 100 carrecords.

struct carrecord

{

char make[15];

float rate;

int mileage;

char rentee[25];

}cars[100];

Then the C programmer may refer to the (i + 1)th record as a unit as cars[i], and to its mileage field by cars[i].mileage. Treating a record as a unit allows intentions to be stated more clearly in a program. It also allows for the contents of one record to be assigned to the storage for another record more efficiently. This is analogous to being able to refer to an automobile and say "Move that automobile from here to there," rather than specifying the automobile or the move by referring to each of its parts individually. Languages that do not provide the record data structure prevent the programmer from referring to a record as a unit in this way.

Note that the record referred to during the execution of a program may be changed merely by changing the value of i, but the field referred to cannot be changed, since field names may not be changed during execution. The variable names cars[i].mileage shows this. The value of array index i can be changed, but the name cars cannot be. This means that code referring to the ith position of an array, such as cars[i], may refer to different array entries as aprogram executes, by computing a new value for i. Code using a field name will always refer to that particular field.

This example gives a technique for sorting records in languages without strong typing constraints. Even in languages with such constraints, the compiler might implement cars in the actual computer memory as illustrated in Figure 2.6. In this case, if the initial array address in memory is 100 rather than 0, then the offset (or relative position) for the (i + 1)th record is added to 100 instead of to 0. The compiler can produce commands that calculate the actual position in memory of any field of the ith record by adding the record offset and field offset to 100. Then a programmer's reference to cars[1] is interpreted as a reference to the 42 consecutive locations 142 to 183. Similarly, cars [1].rate refers to the entry in location 157. This assumes the array starts at memory location 100. The offsets are independent of the starting location. When applied to homogeneous records, this method requires that cars be defined to be the same type as the record's fields.

Observe that the cars array preserves the homogeneity of type that is required of all arrays in C. Its entries are now all of the same specific record type. The array entries just happen to be of type carrecord, even though a record itself is not homogeneous. C allows operations to be performed directly on structures, treating them as a unit. Of course, the compiler hides the details of the records' implementation from readers of the program and provides facility in handling records.

The method just discussed is a single-array implementation. The second method of representing the n records is a multiarray implementation, shown in Figure 2.7. The method simply uses a different array for each field. Each array then contains information of the appropriate type only. There is no direct way to refer to the collective notion cars[2]. Instead, its data are given by make[2],rate[2], mileage[2], and rentee[2]. Selection of the ith record and traversal through the records is also possible for this method.

The single-array technique may cause problems when all entries are not of the same type. As noted, however, Example 2.4 shows offset calculations of the type that may be used by a compiler but are invisible to the programmer. The multiarray technique is useful in any language that provides arrays. Both techniques fail to give the programmer means to refer directly to, or to manipulate as a unit, an entire record composed of more than one field. The C approach provides the record data structure and makes the underlying details transparent to the C programmer, thus providing an elegant solution to the problem of treating information of different types as a unit.

Figure 2.7 A Multiarray Storage of Records of a Car Rental Agency


Data Structures Using C: 2.1.5 Variable-Length Records

2.1.5 Variable-Length Records

When the records of a collection are not necessarily of the same length, they are called variable-length records. Looking back at the single-array and the multiarray representations for a collection of records, it is clear that selection of the ith record was possible only under the tacit assumption that the records were all the same length.

Suppose a field is added to carrecord for each previous rentee. Surely some newer cars will have few previous rentees, while older cars will have a long history of rentees. The programmer could decide to represent each car's record by using a single-length record large enough to accommodate the largest possible number of past rentees. This would aid in processing, as uniformity always does, but would waste a great deal of storage if many records actually have short histories.

With variable-length records, shorter records use fewer locations than longer records, as shown in Figure 2.8. Details of the implementation of such a collection of records must somehow specify where one record ends and the next starts. Separators between records would do, or a field associated with each record could give its total length. Using separators, the last element of the storage for a record would contain a special value denoting the end of the record's storage. If associated fields are used, the first storage element of each record would contain the record's length and would allow a determination of the last element of its storage. In either case, the ith record may no longer be selected, because its location cannot be calculated. However, given the position of the ith record, one can calculate the position of the (i + 1)th record.

Figure 2.8 Variable-Length Records Stored in Array A

Assuming the first element of each record contains its length, as in Figure 2.8, the following function accomplishes the traversal. A record of zero length is used as a sentinel to indicate that there are no more records in the array.

#define SENTINEL 0

>Comment

traverse(a)

*/ Traverses an array a */

int a[];

{

int i;

i = 0;

>Comment

while(a[i] != SENTINEL)

{

process(a,i);

i =i+a[i];

}

}

The record array here is simply an array of individual elements. The programmer imposes the record structure. Each field of the record must thus have the same type in C. Process is a function that does whatever processing is called for. Thus, traversal through the records can be used to access the ith record, but this takes time proportional to i.


Data Structures Using C: 2.1.6 The Use of Unions in C

2.1.6 The Use of Unions in C

No facility is provided in C to handle variable length records directly, though unions do afford a means of handling some variability. A union in C may contain data that can have different types and sizes at different times. Thus unions provide a means of using a single storage area for distinct purposes. Unions are declared in the same manner as structures. The compiler allocates enough memory to hold the largest member that can be stored in the union. Unions are consequently of fixed size but allow variation in the type of data stored. The programmer has to keep track of which type is currently stored in the union. Unions would not be of significant help in Example 2.4 with the list of rentees. In this case there can be more than a few distinct rentees for each car, and unions do not fit this type of variability. Lists, the topic of the next chapter, would be used for this example.

union book_lookup

{

float location;

char title[50];

};

Variables of type book_lookup could contain either the title of a book or its location in a library (based on the Dewey Decimal System). Unions thus give some measure of variability in records. However, the records do not have variable lengths; they are of fixed length.

Unions may occur in structures, and structures may occur in unions.

struct

{

int library_id;

union book_lookup information;

}catalog[100];

Each individual structure of the catalog array can contain an "i.d." number and either a real number or character strings of length 50 in any of its entries. Then, for example, catalog[40].information.title refers to the member title in entry 40 of catalog.


Data Structures Using C: 2.1.7 Address Calculation without Access

2.1.7 Address Calculation without Access

Calculating the address of an item of data is not the same thing as accessing the item of data. The address of an item is its storage position, not its value. To access the item means to read its current value from memory or to store a value in its memory location.

Many authors associate the process of selection with the idea of access to an item, rather than with the calculation of the storage address of the item. They say that the ith item of a structure can be selected if, given the value of i, the program can access the item in a time that is independent of i. Such a structure is called a random access structure. This definition served well for early computers, in which arrays could reside in a single homogeneous memory. However, current practice includes radically different memory architectures.

Modern computers have memory hierarchies�that is, different kinds of memory�whose access times vary by one or two orders of magnitude from one level of the hierarchy to the next. It may well be the case that some portion of an array is stored in fast memory, such as high-speed cache memory, and some portion in slower central memory. Then the access time for an array element would depend on the index of the element (that is, its position in the sequence of the array), but the time required to compute the storage address would still be independent of the value of the index. Consequently, the process of address calculation should be thought of as separate from the process of access to the data. Selection thus denotes address calculation without access. Although it is important to grasp this distinction between address calculation and actual access, it is generally ignored in the explanations here, as in other texts, for convenience of exposition.

Our interest in the effect of data structures on program efficiency concerns the timing of various algorithms. Because access times may vary, depending on features of the physical computer that are completely hidden from the language, time will be measured here as if each program were executed on a computer that has a single homogeneous central memory.

The special techniques necessary to write efficient programs for computers with hierarchical memories are not within the scope of this text. Some computers even allow the exploitation of any parallelism inherent in an algorithm, which means that more than one operation may be carried out at a time. This text also largely ignores this possibility of parallel processing, although it is currently a topic of computer research.

If this array of material leaves you in disarray, keep going. Pointers will point the way!


Data Structures Using C: 2.2: Mortar

2.2: Mortar

2.2.1 Pointers

2.2.2 Pointer Variables and Dynamic Memory


Data Structures Using C: 2.2.1 Pointers

2.2.1 Pointers

In C, records can be stored in arrays. They can also be stored in an area of memory called the dynamic memory. Dynamic memory is managed automatically for the programmer, who can request and release storage as needed. Pointers are variables that point to data items. Those that point to array entries, as noted earlier, are called array indexes. Those that point to individual data items, or structures, are called pointer variables. By following a pointer, you can access a record. Sometimes it is necessary to access a record by following a sequence of pointers.

Pointers and pointer variables afford flexibility in structuring data but must be used with great care, since they can introduce subtle errors that cause difficulty in the debugging of a program. The errors are problems that arise when collections of records can expand or contract in unpredictable ways. Storage for such records must be managed carefully.

To understand pointers and pointer variables, it is practical to examine them in a more static context. They are useful even then, in solving problems involving variable-length records and the representation of arrays. A good way to become familiar with the workings of pointers and pointer variables is to solve the following promblem: Is there a way to store n variable-length records in a one-dimensional array so that traversal through the records is possible, and, more important, so that the (i + 1)th one can be selected?

The solution is not difficult conceptually. Suppose a pointer array pa is created, as in Figure 2.9. The information in pa[i] should state where the base, or origin, of the (i + 1)th record is in a. Thus pa [3] is 13, because the fourth record begins in a[13]. One describes this by saying that pa[i] contains a pointer to the (i + 1)th record in a. The variable i can be thought of as pointing indirectly to a record in a; to get to it requires an access to pa along the way.

Recall that records are represented by a sequence of consecutive entries, so pa[i] points to the first entry of the (i + 1)th record, its base. Once the base of a record and its field offsets are known, any field of the record can be selected. This technique of using pointer arrays is an old and very important technique in computer science. This kind of pointer is just an array index; it is also known as a relative pointer, because it indicates the position of the record in relation to the other records within the same array. This is different from the other type of pointer, the pointer variable, which refers to actual memory locations.

Figure 2.9 Pointer Array PA, with Pointers Stored in A

Selecting the (i + 1)th record is now possible by accessing pa[i] and following the pointer found there to the entry of a in which the (i + 1)th record begins. The base of this record has the name a[pa[i]]. Note that selection is being applied first to array pa, and then to array a. Each takes constant time, which is why we say the (i + 1)th record may be selected. It is also easy to traverse through the records using pa, as shown by the following program.

traverse(a,pa,n)

/* Traverses through the n

records stored in a which

are pointed to by indices

stored in the pointer array pa

*/

int a[],pa[],n;

{

int i;

for (i=0; i<n; i++)

process(a,pa,i);

}

Using the pointer array costs additional storage but solves the problem posed for variable-length records. Relative pointers to records stored in an array are easy to use when language typing constraints are not violated.


Data Structures Using C: 2.2.2 Pointer Variables and Dynamic Memory

2.2.2 Pointer Variables and Dynamic Memory

As a programmer you may need to store a collection of records that are not all of the same type. In C they cannot be kept in a single array, since that would violate the array typing constraint, but there is a way out: simply declare each different type to correspond to one of the members of a union occurring in the record. With this trick, C treats records of different types as records of the same type. Another difficulty in C is how to store a collection of records of the same type when the maximum number of such records is not known in advance. Storing them in an array requires specifying the length of the array, but the programmer may not know what this should be. A practical solution to this problem is provided by the dynamic memory.

So far, by grouping records in arrays, it has been possible to refer to a record by its position in the array. This eliminates the need to attach a separate name to each record of the group. Dynamic memory may be viewed as a collection of storage elements that are on call for the storage of information. However, records stored in dynamic memory do not have individual names or array positions to be used for their reference. Instead, pointer variables are used to point to an individual record. A record in dynamic memory can be accessed only through its pointer variable. A pointer is like a string attached to a record: follow it and you get to the record. The pointers previously examined point to a record's position in an array, whereas pointer variables point to a record's location in dynamic memory. Both play the same role: they lead to a record.

Declaring a record type for records to be stored in dynamic memory is the same as for individual records or records stored in arrays. However, an associated variable of type pointer must be declared and used to refer to the record. For example, to store a record of type carrecord to be referenced by the pointer variable pcar 1 requires the following declaration:

struct carrecord *pcar1,car2;

An important distinction exists between the variable pcar1, which is a pointer, and car2 of type carrecord. The variable car2 is not stored in dynamic memory. Storage is allocated to car2 as soon as the component of the program in which it is declared is executed. The programmer need do nothing other than declare it to ensure this allocation of storage; statements assigning values to it may immediately be written.

This is not the case for the record pointed to by pcar1, which is to be stored in dynamic memory; pcar1 itself is not stored in dynamic memory, but the record it will point to is. This is because pcar1 was declared to be a pointer. The prefix, *, denotes this. No storage is allocated for the record that pcar1 will point to until a specific request made by the programmer is executed. This is done by an invocation of the C function malloc. Thus,

pcar1 = malloc(sizeof(struct carrecord));

when executed, places a pointer value into pcar1 and also allocates storage in dynamic memory for a record of type carrecord to which that pointer points. Until this is done, pcar1's value is undefined. After the function is executed, the situation may be pictured as in Figure 2.10.

The record may be referenced by pcar1 and its members referenced by pcar1->make, pcar1->rate, pcar1->mileage, and pcar1->rentee. At this point, though, the contents of the fields are undefined. Storage has simply been allocated for them in dynamic memory. They may be assigned values by using the assignment statement. Each field of the record pointed to by pcar1 can be given a value. Thus

pcar1->rate = 20.25;

assigns a value of 20.25 to the rate field of the record pointed to by pcar 1. You must be sure before assigning values to a record stored in dynamic memory that you have assigned a value to the pointer variable pointing to that record.

One advantage of pointer variables in C is that the value of a pointer can be printed. It means that a programmer can see the value of a pointer for debugging purposes. Other advantages of using pointer variables (that is, of storing records in dynamic memory) will soon become apparent.

Selection and traversal of a collection of records stored in dynamic memory may be accomplished using an array of pointers. The array entries are pointers that point to records in dynamic memory.

Example 2.5

The definition

struct carrecord *pcar[100];

establishes an array of 100 pointers that can address 100 records stored in dynamic memory. A reference to a member of a record is done by first specifying the pointer to the record, pcar[i], and then the desired member of the record, say make. The resultant reference would then be written as pcar[i]->make. This references the make member of the (i + 1)th record. n

Arrays of pointers to records in dynamic memory are not as easy to use as relative pointers to records in an array, but they provide significant flexibility. You shall see, though, that considerable care is needed to use them properly.

Figure 2.10 Dynamic Memory


Data Structures Using C: 2.3: Representations of Two-Dimensional Array

2.3: Representations of Two-Dimensional Arrays

It is time to return to the problem of how to represent higher-dimensional arrays in terms of one-dimensional arrays. For simplicity, only two-dimensional arrays are discussed, but the techniques can be readily extended for higher dimensions. Two techniques will be studied: 1) rowwise and columnwise representation and 2) pointer array representation.

What exactly is meant by representing a two-dimensional array a in terms of a one-dimensional array (say, data), and why is it done? The idea is that the problem to be solved may involve the use of a, but the language being used may not provide two-dimensional arrays, or the programmer may choose not to use them for some other reason. Instead, the entries of a are stored in data. This must be done so that whenever reference is to be made to the entry in position (i,j) of a in the program, the corresponding position of data in which it is stored is referenced instead. For example, the check functions of Example 2.3 referred to a two-dimensional array a. If its information were stored in data, then the functions would have to be rewritten, replacing each reference to a[i][j] with a reference to the correct entry in data that corresponds to it. Of course, the declarations involving a would also have to be replaced by declarations involving data.

2.3.1 Rowwise and Columnwise Representation

2.3.2 Symmetric Array Representation

2.3.3 Pointer Array Representation


Data Structures Using C: 2.3.1 Rowwise and Columnwise Representation

2.3.1 Rowwise and Columnwise Representation

Suppose a has r rows and c columns, with row index range 0 . . . (r - 1) and column index range 0 . . . (c - 1). A rowwise representation for a uses a data array of length r � c and stores the first row of a in the first c consecutive entries of data, the second row in the next c entries, and so on.

Rowwise representation in arrays is similar to the first method discussed for storing records. In fact, it is logically the same as viewing each row as a record of length c. Thus the offset of the entry in position (i,j) of a is i * c + j in data. This is because rows 0, 1, . . . , i - 1 precede the ith row in data, and take up i � c entries. The jth position of the ith row is offset an additional j entries.

For the array a shown in Figure 2.11, with c = 5, the offset of a[2][3]is 2 * 5 + 3, or 13, and the third row of a starts at data[10]. a[2][3] is represented by data[13]. In C the offset and index are the same, since index values start at zero.

With this representation, function check of Example 2.3 would be written as

#define TRUE 1

#define FALSE 0

check(data,n)

/* Returns true only if an nxn

two-dimensional array stored in

rowwise representation in array

data is symmetric

*/

>Comment

int data[];

int n;

{

>Comment

int i,j,ck;

ck = TRUE;

for(i=1; i<n && ck ; i++)

for(j=0; j<i && ck ; j++)

>Comment

if(data[(i*n+j)] != data[(j*n+i)])

ck =FALSE;

return(ck);

}

Figure 2.11 Representing a Two-Dimensional Array as a One-DimensionalArray

Obviously, this representation in data allows a to be traversed, as well as allowing its (i,j)th entry to be selected.

The columnwise representation of a two-dimensional array is similar, except that the columns rather than the rows of a are stored sequentially in data. The offset for the entry in position (i,j) with the columnwise representation is j*r + i.


Data Structures Using C: 2.3.2 Symmetric Array Representation

2.3.2 Symmetric Array Representation

Example 2.6

Suppose a program involves an n � n array a that is always symmetric. In other words, it is initially symmetric, and any processing performed on a does not disturb its symmetry. When n is small, both the rowwise and columnwise representations are feasible, even though roughly one-half of the information being stored is superfluous. For large values of n (say, 400), the computer system may not provide the required 160,000 storage elements, so the program could not execute. One remedy may be to store only the diagonal entries of a and the entries below the diagonal (those whose row index i exceeds the column index j). This requires ((n � n) + n)/2 entries, or 80,200 instead of 160,000.

Although high-level languages could have provided a data-type, symmetric array to be used in this way, no language does. Instead, the programmer can construct a data array of length ((n � n) + n)/2 to store the entries. The idea is the same as for a rowwise representation, except that only those entries of a row in the lower half or on the diagonal of a are stored. Thus for the row in position i, entries a[i-1][0], a[i-1][1], . . . , a[i][i] are stored in consecutive entries of data, each row's entries following those of the preceding row. n

To use this representation of a, the programmer needs to be able to refer, for given i and j, to the proper position of data that corresponds to a[i][j], just as was done in the check function. Finding the offset formula is a little more complex than for the rowwise representation but is done in the same way.

Preceding the entries for the row in position i are the entries for rows in positions 0, 1, . . . , i - 1. The entry in position j of this row is then an additional j entries down in data. Thus the offset for a[i] [j] is 1 + 2 + 3 +. . . + i + j . Any reference we want to make to a[i] [j] is made, instead, by a reference to data [1 + 2 + ... + i + j]. This, of course, is true only for i � j, since a[i] [j], when i < j, is not stored in data directly. Its symmetric entry in a, a[j][i], is stored in data [1 + 2 + ... + j + i]. So if i < j, a reference to a[i] [j] must be achieved by referring to a[j] [i].

The polynomial (1 + 2 + . . . + i + j) (or ((1 + 2 + . . . + i) + j) appears at first glance to require a loop for its calculation, requiring time proportional to i. However,

The division is integer division, so the polynomial may be evaluated in constant time. This means that the entry in position (i,j) of a can be selected using the index,

Traversal may also be accomplished easily. Consider that the entries are stored in data for each row as records of variable length, yet they can be selected and traversed conveniently with no pointer array. The reason is that their lengths, and the order in which they are stored, are regular enough that the requisite calculations for positions can be made. In the general case of variable-length records, there is no way to do this.


Data Structures Using C: 2.3.3 Pointer Array Representation

2.3.3 Pointer Array Representation

The second technique for representing a two-dimensional array in terms of one-dimensional arrays uses a pointer array p. Pointer arrays have pointers as their entries. The entries of the pointer array point to the rows of a, which may be stored in one of two ways: in data (a one-dimensional array) or in the dynamic memory. The number of rows, r, of a must be known. Then p, data, and r represent a when the array is used, and p and r represent a when dynamic memory is used. In the array case, p is of integer type, as in Section 2.2.1. In fact, this is just an application of that method.

Clearly, there is no need to store the rows one after another in data; they can be anywhere in the array as long as they do not overlap. This feature provides considerable flexibility in the use of storage.

A variation of the array technique would treat the rows stored in data as row records. In the dynamic memory case, p is of type pointer, and the row records are stored in dynamic memory. Selection and traversal are thus both easily achieved. The corresponding versions (storing rows in data, storing row records in data, storing row records in dynamic memory) of the function check would appear as follows.

Version 1�Storing Rows in Data

#define TRUE 1

#define FALSE 0

>Comment

typedef int pointerarray[50];

typedef int dataarray[2500];

check(p,data,r) /* storing rows in data */

/* Returns true only if the rxr

two-dimensional array, whose rows

are stored in data and are pointed to

by pointers stored in p, is symmetric.

*/

    >Comment

pointerarray p;

dataarray data;

int r;

{

    >Comment

int i,j,ck;

ck = TRUE;

for(i=1;i<r && ck;i++)

for(j=0;j<i && ck;j++)

    >Comment

if(data [p[i]+j]! = data[p[j]+i])

ck = FALSE;

return (ck);

}

Version 2�Storing Row Records in Data

#define TRUE l

#define FALSE 0

    >Comment

typedef struct

{

int row[50];

}rowrecord;

typedef int pointerarray[50];

typedef rowrecord dataarray[50];

check (p, data, r) /* storing row records in data */

/* Returns true only if the rxr

two-dimensional array, whose rows

are stored in row records in data and

are pointed to by pointers stored

in p, is symmetric.

*/

    >Comment

pointerarray p;

dataarray data;

int r;

{

    >Comment

int i,j,ck;

ck = TRUE;

for(i=1;i<r && ck;i++)

for(j=0;j<i && ck;j++)

    >Comment

if(data[p[i]].row[j] !=

data[p[j]].row[i])

ck=FALSE;

return(ck);

}

The third version (storing row records in dynamic memory) is a complete program with a driver for check(p,r). It illustrates reading and printing of r and the array represented by r and p. The input and output from its execution might appear as follows.

Typical Input and Output

Enter an integer between 1 and 50 for r

4

Enter the next rowrecord

  3   4  -67   8

Enter the next rowrecord

  4   5   23  57

Enter the next rowrecord

-67  23   -6   4

Enter the next rowrecord

  8  57    4   0

r is 4

The row records are:

  3   4  -67   8

  4   5   23  57

-67  23   -6   4

  8  57    4   0

The array is symmetric

The program reads the two-dimensional input array, represents it, checks it for symmetry, and outputs the result.

Version 3�Storing Row Records in Dynamic Memory

#include <stdio.h>

#define TRUE 1

#define FALSE 0

    >Comment

typedef struct

{

int row[50];

}rowrecord, *rowpointer;

typedef rowpointer pointerarray[50];

main()

/* Driver for check(p,r) */

{

    >Comment

pointerarray p;

int r;

    >Comment

int i,j;

    >Comment

rowpointer malloc();

printf("Enter an integer between 1 and 50 for r\n");

scanf("%d",&r);

for(i=0;i<r;i++)

{

    >Comment

p[i]=malloc(sizeof(rowrecord));

printf("Enter the next rowrecord\n");

    >Comment

for(j=0;j<r;j++)

scanf("%d",&(p[i]->row[j]));

}

printf("\n r is %d \n",r);

printf("The row records are: \n");

for (i=0;i<r;i++)

{

printf("\n");

    >Comment

for (j=0;j<r;j++)

printf("%3d",p[i]->row[j]);

}

if(check(p,r))

printf("\n The array is symmetric \n");

else

printf("\n The array is not symmetric \n");

}

check(p,r) /* Storing row records in dynamic memory */

/* Returns true only if the rxr

two-dimensional array, whose rows

are stored in row records in dynamic

memory and are pointed to by pointers

stored in p, is symmetric.

*/

pointerarray p;

int r;

{

    >Comment

int i,j,ck;

ck=TRUE;

for(i=1;i<r && ck;i++)

for(j=0;j<i && ck;j++)

    >Comment

if (p[i]->row[j] != p[j]->row[i])

ck=FALSE;

return (ck);

}

The pointer array technique may be used storing columns instead of rows. Then p's entries point to the column records.

Example 2.7

It is often desirable to store information in an array in a specific order. Algorithms to rearrange the information to achieve the order may require repeated interchange of rows. Suppose the programmer wants to interchange the ith and jth rows of the array a when it is represented using the pointer array p. This can be accomplished quickly, taking advantage of the pointers, by executing

temp = p[i];

p[i] = p[j];

p[j] = temp;

Note that the data in each row are not moved; only the pointers change. The time it takes the computer to switch rows is independent of their length. Contrast this with the processing involved for the rowwise method of representing a (Section 2.3.1). Even if a actually is available directly as a two-dimensional array, the pointer array representation is more efficient. Swapping columns, of course, is a much more involved task, unless the programmer has chosen to store columns rather than rows. n

The pointer array method offers the advantage of convenience in storing rows of different lengths. Thus it can be used to construct arrays with rows of different lengths easily. Also, if many rows of a are the same, they need not be duplicated in data or in dynamic memory. Their pointer entries in pointer array p need merely point to the same record, resulting in significant storage savings. The means should fit the ends as much as possible! In programming, the ends (the desired operations) determine the means (the data structures to be used).


Data Structures Using C: 2.4: Advantages and Disadvantages of the Tech

2.4: Advantages and Disadvantages of the Techniques

If you think of the rows (or columns) of a two-dimensional array as records, then what has been demonstrated amounts to five basic ways to store records so that selection and traversal are possible. The methods for storing a collection of records are as follows:

1. Fixed-length records of the same type are stored sequentially in a single array and accessed by calculating record and member offsets.

2. In the C structure, each array entry is a record, perhaps including unions.

3. Fixed-length records of the same type are stored in multiarrays, each of appropriate type to match a member type.

4. Use relative pointer arrays to records stored in an array.

5. Use pointer variable arrays to records stored in the dynamic memory.

The first approach is enlightening because it makes explicit the offset calculations that are usually provided invisibly by the compiler. A severe disadvantage of this approach is that the representation of different type records causes difficulties�awkward in some languages, and impossible in others.

The second approach is the elegant form of this representation. It is possible in a language that provides the record data structure in which calculations are provided by the compiler. The disadvantage of the first approach then disappears.

The method of multiple arrays is applicable to all languages that provide arrays, whether or not they have strong typing or the record data structure. This method, and the first, sacrifice the ability to refer to an entire record by a single term, as provided by the newer C compilers.

The two pointer-oriented methods work for collections of variable-length records but are convenient only when the number of distinct lengths is small. (The more general case is treated in the next chapter.) Relative pointers are easy to use but have the drawback that embedding records in an array raises the problems of data types clashing.

Pointers to dynamic memory are more difficult and dangerous to use, for reasons mentioned later, but they provide extra flexibility. In any event, storing records in an array or in dynamic memory eliminates the need to declare a variable name for them at the outset of a program; records can thus be created dynamically during program execution.

The array is limited in the number of records it can store by its fixed length, which must be declared prior to program execution. The capacity of dynamic memory can be much larger and need not be declared a priori, but it is determined by the computer system. As will become more apparent, dynamic memory allows more efficient use of available storage.

In solving a problem, the programmer is free to use any data abstraction that is helpful. In deciding which data structure to use for its implementation, one must consider such factors as the possibility of carrying out the required operations, storage efficiency, execution time, and proneness to error. There is often a trade-off between storage efficiency and execution time. For example, pointers use extra storage and require some time to maintain, but they typically give fast access. Developing your appreciation of such trade-offs in programming is one of the goals of this text.


Data Structures Using C: 2.5: Case Study: Stable Marriages

2.5: Case Study: Stable Marriages

To bring together the concepts presented in this chapter, a case study on stable marriages will now be analyzed. The aims of this study are to

1. Show how a program is designed using the top-down approach

2. Show that storing the "right" information in the "right" way has an important effect on the design of a program, on the execution time, and on the memory requirements

3. Demonstrate the concept of data abstraction by treating data and basic operations on it as a unit

4. Illustrate the treatment of arrays and array indices

Consider Table 2.1. It represents the preferences that each of five men and five women has expressed for the five members of the opposite sex. For example, the first man has ranked the women in the order 2, 4, 3, 1, 5.

Table 2.1 Preferences

       Men's              Women's

------------------------------------

M1  2  4  3  1  5  W1  1  5  3  4  2

M2  3  5  2  1  4  W2  4  5  3  2  1

M3  1  5  4  2  3  W3  3  1  4  2  5

M4  3  1  2  5  4  W4  4  2  5  1  3

M5  2  5  1  4  3  W5  1  3  2  4  5

Now suppose the group has paired off and married, as in Table 2.2. Mi is said to be stable with respect to Wj if there is no other woman preferred by Mi to Wj who, in turn, prefers Mi to her mate. Similarly, Wj is stable with respect to Mi if there is no other man Wj prefers to Mi who prefers Wj to his mate. Any pair (say, Mi-Wj) is stable if each is stable with respect to the other. An entire pairing is stable if all its pairs are stable; an entire pairing is not stable if there exist a man and a woman who are not mates but who prefer each other to their mates.

Table 2.2 Pairings

Men  Women

----------

 1     4

 2     3

 3     1

 4     2

 5     5

Consider the pair M3-W1 of Table 2.2. Since M3 prefers W1 to all others, he is stable with respect to W1. W1 prefers M1 and M5 to M3. M1 is married to W4 and does not prefer W1 to W4. M5 is married to W5 and does not prefer W1 to W5. Thus W1 is also stable with respect to M3. Hence the pair is stable. The entire pairing is not stable. For instance, M2 and W3 are paired, but W3 prefers M4 to her mate, and M4 prefers W3 to his mate, so M2-W3 is not a stable pair. Determining whether or not an entire pairing such as that of Table 2.2 is stable may appear frivolous as stated, but there are other applications. Instead of using the terms men, women, and marriage preferences, we might consider tasks, employees, the employees' preferences for performing the tasks, and the ranking of employees by how well they perform the tasks. Or students might provide their preferences for majors while each department gives its preferences for each student.

2.5.1 The ProblemStability of an Entire Pairing

2.5.2 The Algorithm

2.5.3 The Program

2.5.4 Review of the Program's Development

2.5.5 Generating Stable Pairings


Data Structures Using C: 2.5.1 The ProblemStability of an Entire Pairi

2.5.1 The Problem�Stability of an Entire Pairing

Suppose the problem is to write a program to check whether or not a given entire pairing is stable for a given set of preferences. After this is done the problem of generating a stable entire pairing from a set of preferences will be considered. Assume that there are n men and n women.


Data Structures Using C: 2.5.2 The Algorithm

2.5.2 The Algorithm

To determine whether an entire pairing is stable, the solution must traverse through all the pairs, testing each for stability. The following high-level algorithm does this.

Set stable to true.

For each man, as long as stable is true

set stable to false if the man is not stable with respect to his wife or the wife is not stable with respect to her husband.

The next question is how to determine whether a person is stable. A person is stable if every individual preferred by the person to the person's mate prefers his or her mate to the person. Let stable hold the result of this determination and refine the task:

1. Set individual to whoever is most preferred by person.

2. While (stable and individual � person's mate)

If the individual prefers person to individual's mate, then

set stable to false

else

set individual to whoever is next most preferred by person.

This assumes stable is initially true. Note that the refinement requires

Determining the mate of any person

Choosing the individual most preferred by a person

Determining whether an individual prefers a person to the individual's mate

Finding the individual next most preferred by a person

To follow the dictates of good programming style, the pairings and preference tables should be treated as data abstractions, with their implementation put off until the basic operations on them have become clear. These operations correspond to the four tasks required by the refinement. They will be developed, respectively, as the functions mate, mostpreferred, prefers, and nextpreferred. Before these functions can be written, the implementation of the pairings and the preference data must be decided upon.

Task 1 is to determine a person's mate. This requires the data for the entire pairing, which is given as input to the program and must be read in. However, how should this data be stored? If the mens' pairings are kept in an array mpairs, relating a man to his mate, then the mate of a man can be selected from the array. For the data of Table 2.2, mpairs would be

mpairs

------

  4

  3

  1

  2

  5

Thus the mate of man 3 is woman 1. More generally, the mate of man i is found in mpairs[i], the position of mpairs indexed by i. To find the mate of a woman, however, requires a traversal of mpairs. For example, the mate of woman 1 is found by traversing mpairs until an entry with value 1 is located. The array index of this entry, 3, gives the woman's mate. Rather than spending time traversing whenever a woman's mate is needed, storage can be traded for time. To do this, create another array, wpairs, so selection may be used in this case too. Of course, one traversal will still be necessary to generate wpairs, but this is done just once. The wpairs generated from mpairs would be

wpairs

------

  3

  4

  2

  1

  5

Task 2 is to find out who is most preferred by a person. Suppose the preferences are implemented as two-dimensional arrays mpref and wpref. If the given preferences are those of Table 2.1, then mpref and wpref would be

   mpref           wpref

-----------------------------

2  4  3  1  5   1  5  3  4  2

3  5  2  1  4   4  5  3  2  1

1  5  4  2  3   3  1  4  2  5

3  1  2  5  4   4  2  5  1  3

2  5  1  4  3   1  3  2  4  5

Task 2 is then straightforward and can also be done by selection. In each row of these arrays the first entry is the most preferred mate and the last entry is the least preferred. Thus for man 3, mpref[3][1] contains his most preferred mate.

To perform task 3, determining if an individual prefers a person to the individual's mate, the row for that individual in the proper preference array (mpref for a man and wpref for a woman) may be traversed until either the person or the individual's mate is encountered. If the person is encountered first, prefers must return true, and false otherwise.

Finally, task 4, finding the next preferred individual, can be done by using an index to point to the current individual and, after increasing the index by 1, selecting the (person, index) entry from the appropriate preference array.

The task, determining whether a person is stable, can now be further refined.

1. Set individual to mostpreferred (person, pref).

2. Set index to 1.

3. While (stable and individual � person's mate)

a. set individual's mate to mate (individual, pairs),

b. if prefers (individual, person, individual's mate, pref), then

set stable to false

else

i. Increase index by 1,

ii. Set individual to nextpreferred (person, index, pref1).

This assumes that the person's mate is initially set correctly. The implementation for the functions used follows.

>Comment

mate(person,pairs)

/* Returns person's mate

as specified by pairs.

*/

int person;

pairings pairs;

{

>Comment

return(pairs[person]);

}

    >Comment

mostpreferred(person,pref)

/* Returns the mate most

preferred by person as

specified by pref.

*/

int person;

preferences pref;

{

    >Comment

return(pref[person][1]);

}

    >Comment

prefers(individual,person,individualsmate,pref)

/* Returns true only if individual prefers person

to the individual's mate as specified by pref.

*/

int individual,person,individualsmate;

preferences pref;

{

    >Comment

int index,next;

index = 1;

next = pref[individual][1];

    >Comment

while(next != person && next != individualsmate)

{

index++;

next = pref[individual][index];

}

if(next == person)

return (TRUE);

else

return(FALSE);

}

    >Comment

nextpreferred(person,index,pref)

/* Returns the mate next most

preferred by person as specified

by pref.

*/

int person,index;

preferences pref;

{

    >Comment

return(pref[person][index]);

}

To get an idea of the time required for this solution, note that the while loop used in determining stable can require a traversal through a row of pref. This loop may thus be executed (n - 1) times, since rows of pref are of length n (recall that there are n men and n women). During each of these executions, prefers is invoked and can traverse through a row of pref, which is also of length n. The determination of stable can thus take O(n2) time. Since stable may need to be calculated 2n times (twice for each pair), the total time of this solution could be O(n3). We can do better!

The traversal through pref in prefers can be eliminated. It is done simply to determine if the current individual prefers person to individual's mate, and this information can be ferreted out beforehand from pref and summarized in a priority array. This needs to be done only once. Thus the priority array can be produced before the algorithm is invoked, or can be created initially in the main function. One priority array must be created from mpref and another from wpref. The mpriority array can be created from mpref by traversing mpref and processing its (i, j)th entry by setting mpriority [i] [mpref [i] [j]] to j. Similarly, wpriority can be obtained from wpref. Then an individual prefers person to individual's mate if

priority [individual][person] < priority [individual][individual's mate]

For our example, these arrays are as follows:

  mpriority       wpriority

-----------------------------

4  1  3  2  5   1  5  3  4  2

4  3  1  5  2   5  4  3  1  2

1  4  5  3  2   2  4  1  3  5

2  3  1  5  4   4  2  5  1  3

3  1  5  4  2   1  3  2  4  5

This version of prefers may be expressed as follows:

    >Comment

prefers(individual,person,individualsmate,priority);

/* Returns true only if individual prefers person

to the individual's mate as specified by priority.

*/

int individual,person,individualsmate;

preferences priority;

{

    >Comment

return(priority[individual][person]<

priority[individual][individualsmate]);

}

When person is male, the parameter priority must be wpriority, and when person is female, priority must be mpriority. The priority arrays result in O(n) time for determining stable, since one traversal is avoided. This reduces the total time of the solution to O(n2). Again, storage has been traded for time. Extracting precisely the relevant information and implementing it properly, just as was done in Example 2.2, is what produced this time saving. Also, this version is clearer and more concise.


Data Structures Using C: 2.5.3 The Program

2.5.3 The Program

The algorithm, expressed as a function stabilitycheck, is implemented in the following program. The program first inputs n. Next it reads in mpairs, creates wpairs from it, reads in mpref and wpref, then creates mpriority and wpriority, calls stabilitycheck to check the pairings, and finally prints the result along with the pairings.

#include <stdio.h>

    >Comment

#define MAXN 50

typedef int pairings[MAXN];

mate(person,pairs)

/* Returns person's mate

as specified by pairs.

*/

int person;

pairings pairs;

{

return(pairs[person]);

}

    >Comment

read_mens_mates(n,pairs)

/* Reads the n mates of the

men into pairs.

*/

int n;

pairings pairs;

{

int i;

printf("\n mpairs = ? \n");

for(i=1;i<=n;i++)

scanf("%d",&pairs[i]);

}

    >Comment

create_womens_mates(n,pairs1,pairs2)

/* Creates in pairs2 the women's mates

based on the men's mates as specified

by pairs1.

*/

int n;

pairings pairs1,pairs2;

{

int i,j;

for(i=1;i<=n;i++)

{

for(j=1;pairs1[j] != i;j++)

{

}

pairs2[i] = j;

}

}

    >Comment

typedef int preferences[MAXN] [MAXN];

mostpreferred(person,pref)

/* Returns the mate most

preferred by person as

specified by pref.

*/

int person;

preferences pref;

{

return(pref[person][1]);

}

nextpreferred(person,index,pref)

/* Returns the mate next most

preferred by person as specified

by pref.

*/

int person,index;

preferences pref;

{

return (pref[person][index]);

}

prefers(individual,person, individualsmate,priority)

/* Returns true only if individual prefers person

to the individual's mate as specified by priority.

*/

int individual,person,individualsmate;

preferences priority;

{

return(priority[individual][person]

< priority[individual][individualsmate]);

}

read_preferences(n,pref)

/* Reads n rows of

preferences into pref.

*/

int n;

preferences pref;

{

int i,j;

for(i=1;i<=n;i++)

for(j=1;j<=n;j++)

scanf("%d",&pref[i][j]);

}

    >Comment

create_priorities(n,pref,priority)

/* Creates n individual priorities in

priority based on the individual

preferences specified by pref.

*/

int n;

preferences pref,priority;

{

int i,j;

for(i=1;i<=n;i++)

for(j=1;j<=n;j++)

priority[i][pref[i][j]] = j;

}

#define TRUE 1

#define FALSE 0

main()

{

    >Comment

pairings mpairs,wpairs;

preferences mpref,wpref,mpriority,wpriority;

int i,j,n;

    >Comment

printf("\n n=?\n);

scanf("%d",&n);

    >Comment

read_mens_mates(n,mpairs);

    >Comment

create_womens_mates(n,mpairs,wpairs);

printf("\n Enter mens preferences \n");

    >Comment

read_preferences (n,mpref);

printf("\n Enter womens

preferences \n");

    >Comment

read_preferences (n,wpref);

    >Comment

create_priorities(n,mpref,mpriority);

    >Comment

create_priorities(n,wpref,wpriority);

    >Comment

if(stabilitycheck(n,mpairs,wpairs,mpref,wpref,

mpriority,wpriority))

printf("The pairings are stable \n");

else

printf("The pairings are not stable \n");

printf("The pairings are: \n");

printf("man wife \n");

for(i=1;i<=n;i++)

printf("%d %d \n",i,mate(i,mpairs));

}

stabilitycheck(n,mpairs,wpairs,mpref,wpref,mpriority,wpriority)

/* Checks the entire pairing specified by

the parameters and returns true only

if it is stable.

*/

int n;

pairings mpairs,wpairs;

preferences mpref,wpref,mpriority,wpriority;

{

int man,wife,individual,individualsmate,index,stable;

stable = TRUE;

    >Comment

for(man=1;(stable && man<=n);man++)

{

wife = mate(man,mpairs);

individual = mostpreferred(man,mpref);

index = 1;

    >Comment

while(stable &&(individual != wife))

{

individualsmate = mate(individual,

wpairs);

if(prefers(individual,man,

individualsmate,wpriority))

stable = FALSE;

else

{

index++;

individual = nextpreferred(man,

index,mpref);

}

}

individual = mostpreferred(wife,wpref);

index = 1;

    >Comment

while(stable &&(individual != man))

{

individualsmate = mate(individual,

mpairs);

if(prefers(individual,wife,

individualsmate,mpriority))

stable = FALSE;

else

{

index++;

individual = nextpreferred(wife,

index,wpref);

}

}

}

return(stable);

}


Data Structures Using C: 2.5.4 Review of the Program's Development

2.5.4 Review of the Program's Development

At this point let's review the process followed in creating the program. First an algorithm was conceived and refined, moving from the original problem to the subtasks into which it was broken down: the four tasks for operating on the data abstractions, plus the task of determining stable. The refinement for this last task was specified, treating the pairings and preferences as data abstractions. Next the functions embodying the operations on the data abstractions were specified. In doing so, the trade-off between storage (adding the priority arrays) and execution time (requiring a traversal of rows of mpref or wpref if there were no priority arrays) came into play. This led to the "right" information to be stored and the "right" choice of data structure in which to store it�the priority arrays.

Finally, the solution was packaged into a main program that (1) created and input data to the data structures chosen; (2) called stabilitycheck to check the entire pairing. The functions main and stabilitycheck were written, treating the pairs and preferences as data abstractions. As a result, should their implementations need to be changed, the changes are localized to the data structure definitions and the functions operating on them.


Data Structures Using C: 2.5.5 Generating Stable Pairings

2.5.5 Generating Stable Pairings

As explained earlier, the program only checks a particular entire pairing for stability. It does not generate, for given preferences, an entire pairing that is stable. The task of generating such a stable entire pairing is known as the stable marriage problem.

Some simple cases allow a stable entire pairing to be written by inspection:

1. People have a unique way of looking at the world. This case comes up when the first column of mpref has n distinct entries: every man most prefers a different woman. Simply pair each man with his heart's desire. The case where wpref has this property is similar.

2. People are all the same. This case occurs when the rows of mpref are all exactly alike: every man has exactly the same preferences. Pair the most preferred woman with the man she most prefers. Pair the next most preferred woman with the man she most prefers among those not yet paired, and so on. For example:

   mpref      wpref

----------------------

2  4  3  1  3  1  2  4

2  4  3  1  3  1  4  2

2  4  3  1  4  2  1  3

2  4  3  1  4  3  2  1

Pair W2 with M3, W4 with M4, W3 with M2, and W1 with M1. The case where wpref has this property has a similar solution.

Not only is an algorithm for generating a stable entire pairing not apparent; it is not even evident that one always exists. The following algorithm does construct one; you might try your hand at convincing yourself that it always does so.

1. Set man to 1.

2. While (man is not paired)

a. find the woman most preferred by man who is not yet paired, or, if paired, prefers man to her current mate

b. if the woman is not yet paired, then

i. pair man to woman,

ii. set man to an unpaired man if there is one

else

i. break the pair between the woman and her current mate,

ii. pair woman to man,

iii. set man to her ex-mate

To illustrate this algorithm, apply it to the preferences of Table 2.1. The sequence of pairs generated by the algorithm is as shown in Table 2.3. Asterisks indicate conflicts that cause breaks when they occur. The final pairings are then 3-1, 4-3, 2-5, 5-2, 1-4.

This is a good example of a correct algorithm that surely requires both a proof of correctness (produces the proper result) and a proof that it will eventually stop. These are the essential ingredients of an algorithm. (The proofs are left as exercises.)

Table 2.3 Sequence of Pairs

     Man  Active Pairs

--------------------------

      1    1         2

  1   2    2         3*

      3    3         1

      4    4         3*

--------------------------

           1         2*

  2        3         1

           4         3

      2    2         5

      5    5         2*

--------------------------

           3         1

           4         3

  3        2         5

           5         2

      1    1         4


Data Structures Using C: Exercises

Exercises

1. a. Do users of the subway system of a large city reach their destinations by a process analogous to selection or a process analogous to traversal? Why?

b. Do users of the telephone system of a large city reach their parties by a process analogous to selection or one like traversal? Why?

2. Suppose you are interested in processing English text in the following way:

a. Input is in the form of an English sentence.

b. The program is to check the sentence looking for violations of the spelling rule, "�i'" before �e' except after �c'." (Do not worry about exceptions such as neighbor, weigh, etc.) Would an array be a convenient data structure to use in your program, and would traversal be a relevant process?

3. Write a program to print out how many elements of an integer array (of length n) contain even integers. Did you use the process of traversal?

4. In Example 2.2 suppose the pointer array were stored in memory and there were no a itself. If you are told that a[5] and a[9] have been interchanged, then what changes would you have to make to pointer to reflect this interchange? What if a[2] and a[5] were interchanged?

5. Write a function to update the pointer array of Example 2.2 to reflect the fact that the ith and jth columns of a have been interchanged.

6. a. Suppose the cars array (Example 2.4) represents the fleet information for the cars of one rental agency. For a group of several such agencies, would the record data structure be convenient to represent all the information?

b. Assume the information is represented in an array that stores records, with each record containing all the information for a particular agency. In general, are these records fixed or variable length?

7. For Example 2.4 assume the alternate multiarray implementation. Write a function to interchange the ith and jth records.

8. Create a function that prints out the rentees of all cars that have been driven more than 50,000 miles for the multiarray implementation of Example 2.4.

9. Write a function to read in a sequence of at most twenty characters into an array. They are all digits (0, or 1, or 2, . . . , or 9) except for one character that will be a decimal point. After execution of the function the variable dp should point to the array element that contains the decimal point, and variable 1 should point to the last digit of the input sequence. The decimal point will always have at least one digit on each side.

10. Produce a function similar to that of Exercise 2.3 except that the array of integers is two-dimensional.

11. Suppose a checkerboard is represented as a two-dimensional array. Write a function that is given the current configuration of the board and the move to be made by the player whose turn it is. The function is to return the value true if the move is legal and false otherwise. (Do not forget that a move can consist of many jumps.)

12. a. Is there an analogy between tabs in your home telephone directory and pointers?

b. If a song on a tape cassette is like a record data structure type, can a particular song be indexed, or must a traverse be used to find it? How about a song on a phonograph record?

13. Explain the differences between a record stored in an array and in dynamic memory.

14. Define a function to copy the array of Example 2.5 into an array indexed by a pointer array of integers.

15. Write a function to interchange two records of the array of Example 2.5.

16. Assume the row representation for two-dimensional arrays has been extended to three-dimensional arrays. Determine the formula for the offset of a[i][j][k].

17. The array a with r rows and c columns is stored using the rowwise representation, with a[0][0] corresponding to the actual base location of a, abase. Given an integer number k (abase � k � abase + r � (c - 1)), write formulas that will yield the row and column of a to which actual location k corresponds. That is, if a [i] [j] is stored in k, then your formula for the row will yield i and for the column will yield j.

18. An antisymmetric array a is a square array that satisfies the condition that a[i][j] = -a[j][i] for all 0 � i, j < n. Hence the diagonal elements must be zero. Storing the antisymmetric array can be done just as storing the symmetric array was done, except that now the diagonal entries can be omitted, since they are known. Determine the formula for the offset of a[i] [j].

19. An array a is said to be tridiagonal if the nonzero entries of a fall along the three diagonals a[i] [i], a[i] [i+1], and a[i] [i-1], where i goes, respectively, from 0 to n - 1, 0 to n - 2, and 1 to n - 1.

a. Construct a rowwise representation of the nonzero entries of a, and find a formula for the offset of a[i] [j] in this array.

b. Construct a diagonal representation of the nonzero entries of a tridiagonal array, and find a formula for the offset of a[i] [j] in the array.

20. Write a function that takes a two-dimensional array represented in terms of p and data, as in Section 2.3.3, and changes p so that any duplicate rows of a are stored only once in data. Thus, if rows 2 and 7 are identical, the contents of p[1] and p[6] are made equal.

21. Define a function to output the n diagonal elements of the array a of Section 2.3.3 when the array is represented in terms of p and data.

22. Produce a function to print out the jth column of array a when it is represented in terms of p and data as in Section 2.3.3.

23. Write a function to interchange a[i][j] and a [j][i], for all i, j, when a is represented in terms of p and data as in Section 2.3.3.

24. Create a function that takes any array a of integers with at most twenty entries and creates a pointer array p that has in p[i] a pointer to the (i + l)th largest entry of a. Thus p[0] points to the largest entry of a.

25. Do the same as in Exercise 24, except the array a will be two-dimensional and the ith record in p must contain two pointers, one for the row and one for the column of a's ith largest entry.

26. Do Exercises 20 to 25, except use pointers and dynamic memory. Print the pointers.

27. What pairs are not stable in the example pairings?

28. Give a function to create wpairs from mpairs.

29. Write a function to create a priority array, given a pref array.

30. What function will create a pref array given a priority array?

31. a. Give a function to produce stable marriages.

b. How much time does your answer to Exercise 31a require?

32. a. Why must the algorithm to produce stable marriages terminate eventually?

b. Why does it produce stable marriages?


Data Structures Using C: Suggested Assignments

Suggested Assignments

1. a. Implement the collection data abstraction of Chapter 1 using an array to store the actual integers in the collection. That is, implement the operations set, insert, omit, and belongs.

b. Implement a collection c of carrecords as a data abstraction with the operations set (c), insert (p,c), omit (i,c), and printrecord (i,c). Use the carrecords defined in Section 2.1.3 but add an additional field for an identification number for each car. Then, insert (p,c) adds the carrecord pointed to by p to the collection c, omit (i,c) deletes the carrecord with the identifier i from c, and then printrecord (i,c) prints the contents of the carrecord with identifier i.

c. This is the same as (b), except the records are to be stored in dynamic memory with a pointer array used to contain pointers to them.

2. Suppose the amount of precipitation has been recorded over a region at n locations. In general each record consists of one to five numbers. You are to write and run six programs. Each is to produce the average precipitation. For example, if the input is

3.6

2.1

2.7

record sentinel

.67

1.2

record sentinel

1.9

record sentinel

final sentinel

the output should be the sum [(3.6 + 2.1 + 2.7) + (.67 + 1.2) + 1.9] divided by 6. Always echo print the input (that is, output the input as it is read in), and annotate all output so that you, or anyone, will know what the output means. Assume n � 100.

a. Two of the programs should assume all records are of the same length (say, 3). Program 1 will represent the records as fixed-length homogeneous records stored in a single array (method 1 of Section 2.4). Program 3 will represent them as fixed-length records stored in multiarrays (method 3 of Section 2.4). For method 3, assume an additional record field giving the specific location where it was recorded.

b. Three of the programs should assume the general case of C structures that contain a union and store them in an array (method 2 of Section 2.4). Program 4 will use a relative pointer array whose entries point to records stored in an array (method 4 of Section 2.4). The fifth program uses pointer variable arrays whose entries point to records stored in dynamic memory. For methods 4 and 5, output the contents of the pointer arrays and the records to which each pointer points. Your programs 4 and 5 should work correctly no matter where the records are stored in the one-dimensional array.

c. Finally, one program should assume all records are of length 1 and represent entries in an n � n symmetric array. This program uses the symmetric array representation of Example 2.6. No two-dimensional array should be used in your solution.

The input and output for each program might appear as follows:

                Input                                Output

------------------------------------------------------------------------------

Programs A  1.    3.6            Echo print of the precipitation records:

                  2.1                3.6, 2.1, 2.7

                  2.7                .67, 1.2, 1.9

                  RS             The average precipitation of the 6 values is

                 .67             2.028 inches. The number of records is 2.

                  1.2

                  1.9

                  RS

                  FS

            3. White House       Echo print of the precipitation records:

                  3.6                White House 3.6, 2.1, 2.7

                  2.1                Lincoln Memorial .67, 1.2 , 1.9

                  2.7            The average precipitation of the 6 values is

                  RS             2.028 inches. The number of records is 2.

               Lincoln Memorial

                  .67

                  1.2

                  1.9

                  RS

                  FS

                Input                                 Output

-----------------------------------------------------------------------------

Programs B  2.    3.6             Echo print of the precipitation records:

                  2.1                3.6, 2.1, 2.7

                  2.7                .67, 1.2

                  RS                 1.9

                  1.2

                  RS              The average precipitation of the 6 values is

                  1.9             2.028 inches. The number of records is 3.

                  RS

                  FS

            4.    3.6             Echo print of the precipitation records:

                  2.1                3.6, 2.1, 2.7

                  2.7                .67, 1.2

                  RS                 1.9

                 .67              Pointer array  Data array

                  1.2                1    10     10    3.6

                  RS                 2     1     11    2.1

                  1.9                3    16     12    2.7

                  RS

                  FS                              1     .67

                                                  2    1.2

                                                 16    1.9

                                  -------------------------

                                  The average precipitation of the 6 values is

                                  2.028 inches. The number of records is 3.

            5. This is the same   Records in dynamic storage

               as for 4, except      3.6      2.1   2.7

               it will print the      .67     1.2

               pointer array         1.9

               (that is, the

               actual pointers).

Programs C     3.6   2.1  2.7     Echo print of the precipitation records as a

                .67  1.2  1.9     3 � 3 array:

               0.0   1.1   .15        3.6     2.1   2.7

                                       .67    1.2   1.9

                                      0.0     1.1    .15

                                  The average precipitation of the 9 values is

                                  2.028 inches.

3.     Modify the program of Section 2.5.2 so that its input consists of the men's and women's preferences and then a series of entire pairings. For each of the pairings, it is to call stabilitycheck and print the result for the pairing.


Data Structures Using C: CHAPTER 3: LISTS

CHAPTER 3: LISTS

Presents the list data structure for

efficient traversals

arbitrary insertions and deletions

imposing an ordering on the collection of information it holds

Discusses the limitations of lists for random access

Compares the advantages and disadvantages of lists and arrays for storing data

Discusses the implementation of lists using

records

arrays

pointers

dynamic memory

Stresses the importance of writing functional modules to solve general programming problems and

uses list traversal to convey these techniques

shows how such functional modules can be used as tools and adapted to solve related problems

Explains dynamic storage and its management

Case study�the perfect shuffle

illustrates the use of lists and arrays

3.1: Why Are Lists Needed?

3.2: Keeping Track of List Pointers

3.3: Expanding and Contracting Lists

3.4: Traversal of Lists

3.5: Using the Traverse Function for Lists

3.6: Implementing Lists

3.7: Keeping Track of Available Records

3.8: Sequential Arrays versus Lists for Record Storage

3.9: Case Study: Merging and the Perfect Shuffle

Exercises

Suggested Assignments


Data Structures Using C: 3.1: Why Are Lists Needed?

3.1: Why Are Lists Needed?

Imagine you are a financial wizard and want to store information about your stock portfolio as records. Table 3.1 lists the information. How should these records be stored? The book so far has shown how to store them in

Individual storage elements with individual names

An array, so they can be referenced by position or from a pointer array

Dynamic memory, where they can be referenced by individual pointer variables or from an array of pointer variables

Having more than a few records rules out using individual names, a method that would also be inadequate because it does not allow the records to be treated as a group. The same is true for individual pointer variables.

The other three methods, two kinds of arrays and dynamic memory, allow not only grouping but also selection of records, as well as traversal through all the records. Figure 3.1 represents storage of the stock portfolio in each of these three ways. The three methods are called (a) sequential arrays, (b) sequential pointer arrays, and (c) sequential pointer variable arrays. Collectively they are known as sequential array methods for record storage.

The sequential array methods are convenient for recording the daily transactions common in the buying and selling of stocks, but only when records are kept in arbitrary order. A new record can then be inserted directly as the new last record. This means that in the method of sequential arrays (Figure 3.1(a)), a new record is inserted after Sears. In the method of sequential pointer arrays (Figure 3.1(b)), it can be placed in any unused slot, but the corresponding pointer goes in the seventh position of p. For the method of sequential pointer variable arrays (Figure 3.1(c)), the function malloc provides dynamic storage that is available to store the record, but again, its associated pointer variable value also goes in position 7 of p.

Table 3.1 A Stock Portfolio

Name     # Shares    Value     Date Bought

Apple      300     $ 7,512.46    3/15/84

CBS        100       8,000.55    6/16/83

Digital    200      18,400.00    9/20/83

IBM        100      11,213.25    1/20/83

IBM        200      21,400.00    4/17/84

Sears      100       3,100.00    2/10/82

Sears      100       3,200.00     4/6/84

Usually order does matter. In reality the stocks are kept in alphabetical order. In this case the sequential array methods provide neither easy nor rapid insertion or deletion capability for the records. Suppose Chrysler stock is bought, and its new record must be inserted into the portfolio; Chrysler must be inserted in its proper place as the new third record. In sequential arrays this means that all five records below its new position must be moved down. In the two pointer array methods, this means that the five pointers below its new location (position 2) must be moved down. The pointer array methods afford flexibility of record storage position, but they still require shifting, although it is pointers that are moved rather than records. Shifting takes time proportional to the number of records moved. If n records are to be inserted, then the total time can be proportional, not to n, but to n2. For example, suppose the n records each require insertion at the top. Even if there are no records stored initially, insertion of n records requires 0 + 1 + 2 + . . . + (n - 1) moves. The time adds up to n(n - 1)/2, and so the time is O(n2). Even if each insertion requires only half the information to be shifted, the time is reduced merely by a factor of 2; it is still O(n2). When n is large, this can take significant time. Of course, for one trader, n will be small, but for larger brokerage houses n might be hundreds of thousands. If they were not designed to be efficient, these insertions might take hours or even days ((300,000)2 operations � 10-6 seconds per operation = 25 hours).

(a) Sequential Array

(b) Sequential Pointer Array

(c) Sequential Pointer Variable Array

Figure 3.1 Sequential Methods of Storing Records in Arrays

Selling a stock, say Digital, requires that its record be deleted in the sequential array (a) or that its pointer entries be set to - 1 or null in the pointer arrays (b) and (c). Either course would leave gaps of unused storage, which is undesirable. Leaving gaps soon uses up all the slots in an array, and unless wasting storage is tolerable and affordable, this is impractical. Instead, the gap would be filled by moving up all the records below a deleted record. But again, this could take O(n2) time for the deletion of n records.

What is needed is a way to store records so that their number can grow or shrink in response to an arbitrary number of insertions and deletions, in such a way that storage is not wasted and the time for insertion and deletion is reduced. The list data structure introduced in this chapter provides one solution to this problem. A list is a collection of records linked by pointers. Even with lists, it is still necessary, as with the sequential array methods, for the program to spend time deciding where to make an insertion or deletion in the first place. To make these operations efficient when large numbers of records are involved requires more advanced data structures (introduced later). For a moderate number of records, the time taken to locate the point of insertion or deletion is tolerable. Lists called stacks and queues, where insertion and deletion occur only at the beginning and end of the lists, are especially useful even when there are many records. These will be considered in the next chapter.

Chapter 2 showed that the key advantage of the sequential array methods is selection in constant time. As just shown in this chapter, the disadvantage is that inserting or deleting entries from the interior of an array is costly. Another drawback is that arrays have fixed lengths that must be declared to the compiler. Thus, the maximum number of items to be stored in the array must be known in advance of execution. No such a priori limit need be declared for the number of items on a list. Lists provide a way to insert new items or to delete existing ones at a known point of the list, in constant time. However, selection of an arbitrary ith list item will take time proportional to i.


Data Structures Using C: 3.2: Keeping Track of List Pointers

3.2: Keeping Track of List Pointers

The pointers in lists are what is manipulated by programs, because lists are collections of records linked by pointers. A special element, called the head or name of the list, points to the beginning of the list, and each record in the list has a pointer to the next one. The final record, which has no successor, has an empty link, denoted by a special pointer value called the null pointer.

Figure 3.2 Graphic Representation of the List STOCKS

The stock portfolio example is represented in a list in Figure 3.2. The variable labeled head contains a pointer to the first record in the list. Each record is composed of two fields: 1) an information field, which may itself be made up of subfields such as name, shares, value, datebought (or accountnumber, balance, address); and 2) a link field, which contains the pointer to the next record. The x in the link field of the last record denotes the null pointer and indicates that this is the last record on the list. The list with no records is called the null list, and is denoted by a head containing the null pointer.

It does no good to know that a phone number (say, 642-3715) contains the digits 1, 2, 3, 4, 5, 6, and 7, unless the order 6, 4, 2, 3, 7, 1, 5 is retained as well. Notice that the list data structure, besides being a new way to store information, allows the information to be remembered in a specific order�the order in which the records appear on the list. Arrays provide this ability as well. The order in which records (or the pointers to them) are stored does this, but the operations that make lists interesting and powerful are insertion and deletion.

In effect, a list works as if the pointers from a pointer array were incorporated into the records to which they point. This is the crucial advantage of lists compared to arrays. The price of the added convenience is the loss of the capability to select an arbitrary record. Instead of access to the list records being through the pointer array, the list records must now be accessed via the pointer to the first list record (in head or name). Knowing only the head of a list means that only the first record is immediately visible. To access another list record, the chain of pointers linking list records must be followed until the desired record is encountered.

Consider the simple list l in Figure 3.3(a). Two situations are depicted in Figure 3.3(b,c): (b) is the case when the list head l is known, and (c) the case when a pointer, ptr, to a list record is known. In each case only the record to which the known pointer points is visible.

When only one pointer points to a record, if that pointer value is changed, then the record cannot be accessed. It is as if the programmer had stored something somewhere but had no hope of remembering where. If that pointer happens to be in the head of a list, then access to the list is lost. Suppose the programmer must start with the situation depicted in Figure 3.3(a) and must create two lists. One, l, is to consist of all records of the original list, from the record pointed to by ptr to the last record. Another, newl, is to consist of the original list records, from the first to the record prior to that pointed to by ptr. Simply copying the value of ptr into l creates the correct modified version of l. However, newl cannot be created since the pointer to the original first record that had been stored in l has been lost. Such a predicament must be avoided.

Figure 3.3 Simple Lists

Keeping track of pointers in processing lists is extremely important. One reason is that more than one pointer can point to a record. The only way to see if two pointers point to the same record is to see if their pointer values are equal.


Data Structures Using C: 3.2.1 Insertion and Deletion of Records in Li

3.2.1 Insertion and Deletion of Records in Lists

Consider the effects of inserting and deleting on a conceptual list. The effect of inserting a new record is shown in Figure 3.4. The new record is inserted between the records labeled predecessor and successor. This requires changing predecessor's link to point to the new record, and setting the new record's link to point to successor.

Figure 3.4 Insertion of a New Record in a Conceptual List

Figure 3.5 Deletion of a Record from a Conceptual List

The deletion of a record from a list is shown in Figure 3.5. The record labeled removed is deleted from the list by arranging predecessor's link to point to successor. The rearrangement of links for insertion or deletion is independent of where in the list the operation occurs. Each operation can be performed in constant time. Lists, unlike arrays, readily accommodate change; they exhibit no growing pains (or contraction pains).


Data Structures Using C: 3.3: Expanding and Contracting Lists

3.3: Expanding and Contracting Lists

3.3.1 Insertion

3.3.2 Special Cases

3.3.3 Header Records

3.3.4 Deletion


Data Structures Using C: 3.3.1 Insertion

3.3.1 Insertion

Suppose you wish to insert a new record for CHRYSLER. It is always a good idea to sketch an image like Figure 3.6(a) to show both the situation at the outset and the situation desired after the task is carried out. Since only the head of the list, stocks, is assumed known, normally each record of the list must be accessed in turn, starting from the first, in order to determine where to make the insertion. For now, assume there is a given pointer predecessor that points to the record after which the new record is to be inserted. In the example, this is the second record. The value of its link field must be copied into the link field of the new record, so the new record's link field points to the successor of the predecessor record. The link field of the predecessor record must end up pointing to the new record, so the value of newpointer must be copied into the predecessor's link field. The code to accomplish this may be written as

setlink(newpointer,next(predecessor));

setlink(predecessor,newpointer);

where setlink is a function that copies the value of its second parameter into the link field of the record pointed to by its first parameter, and next is a function whose value is a copy of the link field value of the record pointed to by its parameter. Why is it incorrect to interchange these two statements?

Notice that no assumptions have been made about the implementation of the list. The code is thus independent of how the list is implemented. Of course, setlink can be written only when the implementation is known, and it will depend on the details. However, this code will not. The list is thus a data abstraction with insertion as one of its operations; deletion is another.

Figure 3.6 Insertion of a New Record (a) in the Interior of a List and (b) at the Front of a List


Data Structures Using C: 3.3.2 Special Cases

3.3.2 Special Cases

It is evident that this code is correct for insertion into the interior of the list or at its end. Special cases must always be sniffed out and dealt with. By definition, a special case means a situation requiring code different from the current solution.The existing solution must be modified to handle special cases. The special case here, shown in Figure 3.6(b), represents insertion at the front of the list. Obviously there is no predecessor record, so the solution cannot properly handle this case. It is reasonable to assume that this special condition is represented by a null value for predecessor. Then it is the value of head that must be copied into the link field of the new record. Also, it is the head itself that must receive the copy of newpointer and end up pointing to the new record. The correct function to handle all situations is as follows:

>Comment

insert(phead,predecessor,newpointer)

/* Inserts, in list head, the record pointed to

by newpointer as the successor of the

record pointed to by predecessor.

*/

listpointer *phead,predecessor,newpointer;

{

listpointer setnull(),next();

if(predecessor == setnull())

{

>Comment

setlink(newpointer,*phead);

*phead = newpointer;

}

else

{

>Comment

setlink(newpointer,next(predecessor));

setlink(predecessor,newpointer);

}

}

Note: listpointer is a type which is explained in Sections 3.6.1 and 3.6.2.

where setnull returns the null value. What is used for the null value is determined by the list implementation (Section 3.6). When insert is invoked, its actual first parameter, corresponding to phead, must be a pointer to the head of the list. Thus to have insert work on a list l, the call would be insert (&1,predecessor,newpointer). Whenever a parameter of a function must return a value (be modified by the function) the convention in this text is to prefix the name of the parameter in the function definition with a p. Thus, since the value of the pointer to head in insert must be changed, it is referred to as phead. Notice that the code for insert refers to *phead. Think of *phead as another name for head. We say that head has been passed to insert by pointer.

This version of insert is independent of how the list is actually implemented. In practice, insert might be made to execute more efficiently by writing it so that its code does depend on the list implementation. Insert might then be the level at which the list implementation is hidden instead of in setnull, setlink, and next. However, the version above is clearer and can be readily modified if desired.


Data Structures Using C: 3.3.3 Header Records

3.3.3 Header Records

An alternative that avoids the special case of inserting a new first record assigns a special record, called a header record, to every list as its first record. This record is a dummy record. The dummy is so called because it is not part of the conceptual list of records being created. Used to make processing easier, the dummy record is never deleted, and no record is ever inserted in front of it. Then the special case never occurs, and the original code, requiring no check for the special case, can be used. Actually the header record can be a smart dummy, used to contain important information about the list, such as the number of records. When this alternative is used, the null list can no longer be recognized by a null value in the list head. It is recognized by a null pointer in the link field of the dummy record.

Figure 3.7 Deletion of a Record (a) from the Interior of a List and (b) from the Front of a List


Data Structures Using C: 3.3.4 Deletion

3.3.4 Deletion

To delete the record following predecessor (Figure 3.7(a)), only the link field value of its successor must be copied into the predecessor's link field:

setlink(predecessor,next(next(predecessor)));

Thus next (next(predecessor)) is the value that must be copied into the predecessor's link field. It is clearer to write the code as

successor = next(predecessor);

setlink(predecessor,next(successor));

even if it is slower, requiring an additional memory access. Again, either code is independent of any implementation for the list. After checking for special cases (Figure 3.7(b)), the correct function is found to be

>Comment

delete(phead,predecessor)

/* Deletes, in list head, the successor of

the record pointed to by predecessor.

*/

listpointer *phead,predecessor;

{

listpointer successor,null,setnull(),next();

null = setnull();

>Comment

if(*phead != null)

if(predecessor == null)

>Comment

*phead = next(*phead);

else

{

>Comment

successor = next(predecessor);

if(successor != null)

setlink(predecessor,next(successor));

}

{

Sometimes, instead of having a pointer to the predecessor of a record to be deleted, only a pointer to the record itself is available. Deletion can still be accomplished; all that is required is to copy the entire successor record (the successor to the record to be deleted, that is) into the record to be deleted. It is necessary to assume, however, that the last list record is never deleted. Why? Whether or not this is better than keeping track of the predecessor depends on how much time is required to effect this copying operation. The larger the record, the greater the time. However, if the information field of a record is kept elsewhere, and a pointer to it stored in the record itself, then the operation reduces to copying just this pointer and the value of the link field.

Are you listing from all this list information?


Data Structures Using C: 3.4: Traversal of Lists

3.4: Traversal of Lists

Traversing a list's records is another operation performed on the data abstraction called the list. Traversal is useful when the records are to be processed one after another in the same order in which they appear in the list. A traversal may be accomplished by following the pointer in the list's head to the first record and processing it, then following the first record's link field pointer to the next record and processing it, and so on, until the last record has been processed.

3.4.1 List Reversal Using a Loop

3.4.2 A General List Traversal Using a Loop

3.4.3 The Merits of Functional Modularization


Data Structures Using C: 3.4.1 List Reversal Using a Loop

3.4.1 List Reversal Using a Loop

Example 3.1

Consider the list l in Figure 3.8(a). The task is to write a function reverse, to reverse such a list. This means that when reverse is applied to l, the result should be as shown in Figure 3.8(b). n

At first glance, this may seem to be a complicated task involving a great deal of record and pointer manipulation. However, there is a simple solution: the information fields of each record need not be considered at all. Instead, as in Figure 3.8(c), the link field of F can be set to null, the link field of O can be set to point to F, the link field of R can be set to point to O, and so on. Finally, l itself can be set to point to D. The result would be as sketched in Figure 3.8(c), the correct reversed list. This procedure can be stated more generally:

1. Set the link field of the first record to null.

2. Set the link field of each record to point to its original predecessor.

3. Finally, set the list head to point to the original last record.

Even without the details of implementing the list and carrying out the procedure,it is easy to be certain that this procedure, when successfully carried out, does solve the problem. This is so except for the case of a null list; the procedure does not specify what is to be done, since there is no first or last record in a null list. Since the first record of any list has no predecessor, the procedure can be specified somewhat differently.

(a) The Original List

(b) The Reversed List

(c) The Procedure for Reversing the List

Figure 3.8 A List before, after, and during Reversal

1. Set the link field of each record to point to its original predecessor.

2. Then set the list head to point to the original last record, or to predecessor if the list is null.

Now we have a procedure that is correct for all cases, the first record's predecessor being assumed null.

The next step is to detail how this procedure is actually to be done. The basic structure involved is a loop, since the same process is to be carried out repeatedly, but on a different record each time. In general, loops need a beginning (initialization) and an end (finalization). The finalization here is specified explicitly in the procedure: namely, set the list head to point to the original last record, or to predecessor if the list is null. The initialization, as usual, cannot be specified until the loop is specified in more detail. Refining task 1 makes the algorithm read as follows:

1. Initialization

While (there is a current record)

a. set the link field of the current record to point to its original predecessor, and

b. update the current record.

2. Set the head of the list to point to the original last record, or to predecessor if the list is null.

To implement the loop requires a pointer (call it recordpointer) to keep track of the current record to be processed within the loop by its loop task. Imagine that the first three records have already been processed correctly and the fourth is about to be processed. It is pointed to by recordpointer, as depicted in Figure 3.9(a).

Figure 3.9 Achieving Proper Loop Implementation

Processing W requires setting its link field to point to W's predecessor R. To do this requires knowing where its predecessor R is located. Thus another pointer, predecessor, is needed. It is so named because it will contain a pointer to the predecessor of the current record to be processed. (See Figure 3.9(b).)

Suppose someone attempts to process W now, copying predecessor into W's nextpointer field to set it properly. The result is shown in Figure 3.9(c).

Predecessor and recordpointer must now be updated so that they point to W and A, respectively. Then A can be processed next when the loop is repeated. However, the link field of W no longer points to A, but instead to R. Thus the location of A has been lost. Care is required when changing pointer values so that such important information is not lost. To remedy this, introduce another pointer hold in which the pointer to A can be saved before changing W's link field value. The processing of the record pointed to by recordpointer may now be done by sequentially

1. copying its link field value into hold,

2. copying predecessor into recordpointer's link field

3. copying hold into recordpointer.

This produces the result depicted in Figure 3.9(d).

The test or condition for another repetition of the loop becomes (recordpointer ! = null). The initialization involves setting recordpointer to 1, to make it point to the first record, and setting predecessor to null. Why is it unnecessary to initialize hold?

When the loop is exited, predecessor will point to the original last record, and the finalization can be completed by copying predecessor into l. Refining again:

1. Set predecessor to null.

Set recordpointer to l.

While (recordpointer is not null),

a. set hold to the link field value of the record pointed to by recordpointer, and

set recordpointer's link field to predecessor,

b. set predecessor to recordpointer, and

set recordpointer to hold.

2. Set l to predecessor.

This algorithm actually involves a complete traversal through the records of the list l. As it is carried out, recordpointer accesses each record of l in turn and processes it within the while loop. The processing is done by the loop task. In order for the loop to work correctly, some initialization had to be done, and some finalization was required after each record of the list had been processed.


Data Structures Using C: 3.4.2 A General List Traversal Using a Loop

3.4.2 A General List Traversal Using a Loop

Our intention now is to write a general function traverse, to carry out a traversal of a list, given its head, listname. The code for traverse should be independent of the list implementation, as was the code for insertion and deletion. To make traverse independent of the particular application as well, theprocessing to be done on each record will be functionally modularized in a function process. Its details are thus irrelevant; simply invoke process within traverse. However, to do its job, this function may need access to the list head, listname. It will surely need to know what record it is currently to process; recordprinter will be used to point to this record.

Often, the traversal must proceed through all the list records. To allow for the possibility of a partial traversal, however, a variable done can be introduced. Done may be set by process so that the traversal does not continue beyond the current record. Finally, process cannot retain any information between calls unless that information is kept nonlocal to it, is passed as one of its parameters, or is a static variable. So process is given the parameters listname, recordpointer, done, and other. Other denotes those parameters whose values must be preserved between calls to process.

The heart of a traversal is repeated access of the record pointed to by recordpointer, processing of the record, and updating of recordpointer to point to the next record. Clearly, a loop is its main construct. The function for a general traversal of a list is as follows:

>Comment

traverse(listname)

listpointer listname;

{

listpointer recordpointer,next();

moreparameters other;

int done;

>Comment

initialize(listname,&other);

recordpointer = listname;

done = FALSE;

while(!done&&anotherrecord(listname,

recordpointer))

{

>Comment

process(listname,recordpointer,

&other,&done);

>Comment

recordpointer =

next(recordpointer);

}

>Comment

finalize(listname,other);

}

Initialize may be used to set the initial value of any parameters that require initialization. Anotherrecord is a function returning true if recordpointer points to the next record to be processed, and returning false if the last record has already been processed. Finalize is used to carry out any tasks required after all list records have been processed�for instance, to print an appropriate message.

It is a good idea to check a program for correctness, in lieu of a formal proof, by seeing if it handles all special cases (and the typical case) correctly. You should check traverse in this way for the case of a null list, a one-record list, and a list with more than one record. For example, if the list is null, the while loop is never executed as long as anotherrecord is correct (which is assumed), and the program terminates properly as long as finalize is correct (which is assumed). So traverse does work correctly for the special case of a null list.


Data Structures Using C: 3.4.3 The Merits of Functional Modularization

3.4.3 The Merits of Functional Modularization

Traverse has been written in this general way so that it can be used as a tool readily adaptable to the solution of a large class of problems. How to adapt this function to the solution of seemingly disparate problems is the purpose of the next section. It is important to recognize the basic technique used to render the program independent of the list implementation: embedding those details in lower level functions. This is the reason for anotherrecord and next, initialize, process, and finalize; these functions allow traverse to be adapted to specific applications.

All the functions that must reflect implementation and application details in their definitions are lower-level functions. Changes to them do not affect the correctness of traverse, but they do affect what traverse does. The functions call attention to what is being done instead of how it is being done. Furthermore, if the programmer consistently uses a function such as next or process to carry out specific tasks, then no matter how many places the task is required in a program, the definition of that task is localized in one place, and it is clear where to look for the code that implements it. The functions may then all be written independently to suit the immediate purpose. As long as they are correct, traverse must be correct too.


Data Structures Using C: 3.5: Using the Traverse Function for Lists

3.5: Using the Traverse Function for Lists

Sometimes a solution to a new problem can be obtained by modifying a known program that solves a related problem. Also, maintenance or adaptation of programs frequently involves small changes to current programs. Changes are easier, quicker, and more likely to be error-free if exactly that function or task that requires change has been localized as a program function.

Traverse adheres to two important elements of programming style.

1. High-level descriptions and programs should be as independent of the details of data abstraction implementation and of the application as possible.

2. High-level descriptions and programs should be functionally modular so that special lower-level functions or tasks carried out within the high-level description or program may be easily isolated.

To illustrate the use of traverse, and the effect of these principles, this section presents four examples. Although each could be solved in isolation, from scratch, a different approach is taken here. In each case it should be easy to see that traverse may be adapted to provide an almost immediate solution. This will be done by writing specific versions of traverse's functions for each application, so that these functions turn traverse into a solution. This means that it will then do the required task when executed. Of course, the functions, when invoked by traverse, will do exactly what the programmer specifies in their definition.

Example 3.2

A sentence is stored in a list, with each record storing one character. Thus the sentence, THIS SENTENCE HAS EXACTLY 60 CHARACTERS COUNTING BLANKS AND., is stored in a list with 60 records. For simplicity, assume the period character "." does not occur within a sentence (as it does in this one). The task is to write a function to print the length of the sentence. n

The solution can be achieved by traversing the list and counting the number of records. Initialize can set a counter, length, to zero, and process can increment it by 1 each time it is invoked. Finalize can do the printing. The functions are readily defined.

>Comment

initialize(listname,plength)

listpointer listname;

moreparameters *plength;

{

*plength = 0;

}

>Comment

process(listname,recordpointer,plength)

listpointer listname,recordpointer;

moreparameters *plength;

{

(*plength)++;

}

    >Comment

finalize(listname,length)

listpointer lastname;

moreparameters length;

{

if(length != 0)

printf ("\n The sentence has length %d \n",length);

else

printf("\n There is no sentence \n");

}

Either anotherrecord and next must be supplied to the programmer, or the details of the list implementation must be specified such that the programmer can write them. The task of the example is then carried out by the invocation traverse (sentence). In this example traverse must be modified to the following function.

#define TRUE 1

#define FALSE 0

typedef int moreparameters;

traverse(listname)

/* Prints the length of the sentence

stored in the list, listname.

*/

listpointer listname;

{

listpointer recordpointer,next();

moreparameters other;

int done;

    >Comment

initialize(listname,&other)

recordpointer = listname;

done = FALSE;

while(!done&&anotherrecord(listname,recordpointer))

{

    >Comment

process(listname,recordpointer,&other);

recordpointer = next(recordpointer);

}

finalize(listname,other);

}

There are two ways to view traverse. One is as a program that can be used but not modified in any way. This would require that the functions be written as they appear and the type moreparameters be declared prior to traverse. The second is as a program that may be used or changed in any desirable way. In this case, for clarity, the function names can be changed: traverse to countcharacters, initialize to zerocounter, process to incrementcounter, finalize to printcounter, and anotherrecord to anothercharacter, for example. It may be desirable to remove parameters that are not needed. Since listname is not needed by initialize, process, or finalize, and recordpointer is not needed by process they may be omitted from the definitions and invocations of these functions. Other may be changed to length. Finally, done may be eliminated, since it is not needed. Having made this distinction, the programmer is free to take either view, or a view that falls between the two. The one chosen should be clear from the context.

Suppose the sentence were stored in the list with three characters per record instead of just one per record. The solution would require a new version of process to reflect this change, but initialize and finalize remain as is. Next might have to reflect the new record format. The new process is

    >Comment

process(listname,recordpointer,plength)

listpointer listname,recordpointer;

moreparameters *plength;

{

listpointer next();

int lastrecord;

char char1(),char2();

lastrecord = !anotherrecord(listname,next(recordpointer));

    >Comment

if(!lastrecord)

*plength = *plength + 3;

    >Comment

else if(char1(recordpointer) == '.')

(*plength)++;

else if(char2(recordpointer) == '.')

    >Comment

*plength = *plength + 2;

else

    >Comment

*plength = *plength + 3;

}

where char1 returns a copy of the first character in the record pointed to by its parameters, and char2 returns a copy of the second character in the record pointed to by its parameters. Like next and anotherrecord, they must be supplied, or the programmer must write them given list implementation details.

Finally, suppose the list is implemented as a circular list (Figure 3.10). In circular lists, the null pointer of the last record is replaced by a pointer that points to the first record. To illustrate the difference in the implementations between a standard list and a circular list, the two distinct versions required for anotherrecord are presented.

Figure 3.10 A Circular List

Standard List

anotherrecord(listname,recordpointer)

listpointer listname,recordpointer;

{

listpointer setnull();

    >Comment

return(recordpointer != setnull());

}

Circular List

anotherrecord(listname,recordpointer)

listpointer listname,recordpointer;

{

    >Comment

return(listname != recordpointer);

}

Actually, the solution is not as general as possible. Suppose the list were implemented using a dummy header record. Then process must recognize when it is processing this record (the first) and not change length, since no characters are stored in the header record. You should modify process to account for this possibility. It is necessary to assume that the header may be distinguished from the other records in some way.

Example 3.3

Earlier in the chapter a way to reverse a list was developed from scratch. We now build a program to reverse a list utilizing traverse. Traverse may be turned into a solution for the list reversal by defining initialize, process, and finalize, as follows:

    >Comment

initialize(listname,psave,ppredecessor)

listpointer listname,*psave,*ppredecessor;

{

listpointer setnull();

*psave = setnull();

*ppredecessor = setnull();

}

    >Comment

process(plistname,recordpointer,psave,

ppredecessor)

listpointer *plistname,recordpointer,*psave,

*ppredecessor;

{

listpointer setnull();

if(*ppredecessor != setnull())

setlink(*ppredecessor,*psave);

*psave = *ppredecessor;

*ppredecessor = recordpointer:

}

    >Comment

finalize(plistname,save,predecessor)

listpointer *plistname,save,predecessor;

{

listpointer setnull();

if(predecessor != setnull())

{

setlink(predecessor,save);

*plistname = predecessor;

}

}

    >Comment

traverse(plistname)

/*Reverses the list listname.*/

listpointer *plistname;

{

listpointer recordpointer,*psave,*ppredecessor,next();

initialize(listname,&psave,&ppredecessor);

recordpointer = *plistname;

while(anotherrecord(*plistname,recordpointer))

{

process(*plistname,recordpointer,&psave,&ppredecessor);

recordpointer = next(recordpointer);

}

finalize(&plistname,*psave,*ppredecessor);

}

Notice that the parameter of traverse is now a pointer to the listname. This is necessary since listname may need to be changed by traverse. Here process works on the record preceding the record pointed to by recordpointer, while the original solution worked on the record pointed to by recordpointer. This was necessary because that solution changed recordpointer's link field before updating recordpointer. n

Example 3.4

Insert a new record into a list stocks, which is kept in alphabetical order by name. Assume the data for the new record may be accessed by a function getnextrecord, which returns a pointer to the new record. n

To turn traverse into a solution requires simply that process compare the new record's name field value with that of the current record. If it precedes the current record's name field value, it must be inserted before the current record and done set to true. To make the insertion conveniently, a pointer, predecessor, to the predecessor of the current record will be kept by process and updated each time process is invoked. Initialize must allocate storage for the new record, set its data field values, and set predecessor to null. Assume that a function avail is given, returning a pointer to a storage element that may be used to store the new record. Finalize must insert the new record in the event that the list is null. The solution for these functions follows:

    >Comment

initialize(pnewpointer,ppredecessor)

listpointer *pnewpointer,*ppredecessor;

{

listpointer avail(),setnull(),getnextrecord();

*pnewpointer = avail();

setinfo(*pnewpointer,getnextrecord());

*ppredecessor = setnull();

}

where setinfo copies the contents of the record pointed to by its second parameter into the information field of the record pointed to by its first parameter, so it depends on the list implementation details.

    >Comment

finalize(plistname,predecessor,newpointer)

listpointer *plistname,predecessor,newpointer;

{

listpointer setnull();

if(*plistname == setnull())

insert(plistname,predecessor,newpointer);

}

    >Comment

process(plistname,recordpointer,newpointer,ppredecessor,

pdone)

listpointer *plistname,recordpointer,newpointer,

*ppredecessor;

int *pdone;

{

listpointer setnull(),next();

if(precedes(newpointer,recordpointer))

{

    >Comment

insert(plistname,*ppredecessor,newpointer);

*pdone = TRUE;

}

else if(next(recordpointer) == setnull())

{

    >Comment

insert(plistname,recordpointer,newpointer);

*pdone = TRUE;

}

    >Comment

else

*ppredecessor = recordpointer;

}

where precedes returns true if its first parameter points to a record whose name precedes, in alphabetical order, the name in the record pointed to by its second parameter. To insert a new record, invoke traverse (&stocks). However, traverse must be modified in a way similar to Example 3.2 as follows. A better name for this modified traverse would be orderedinsert, so we use it.

    >Comment

orderedinsert(plistname)

/*Inputs and inserts a new record

in the list listname in alphabetical

order.

*/

listpointer *plistname;

{

listpointer recordpointer,newpointer,predecessor,next();

int done;

initialize(&newpointer,&predecessor);

recordpointer = *plistname;

done = FALSE;

while(!done&&anotherrecord(*plistname,recordpointer))

{

process(plistname,recordpointer,newpointer,

&predecessor,&done);

recordpointer = next(recordpointer);

}

finalize(plistname,predecessor,newpointer);

}

Suppose you wanted to insert the new record into a list of stocks kept in order by date bought, instead of by name. Rather than keep another list distinct from stocks, you could add another link field to its records. Call it the datelink field, and assume each record's datelink field points to the correct successor as in Figure 3.11. Stockdate is a new head, pointing to the first record of that list when date bought is used as the criterion for ordering.

Figure 3.11 Two Orders for a List Using Two Heads and Two Link Fields

To solve this problem: 1) define precedes so that it compares dates instead of names; 2) replace each reference in insert to the setlink function by a reference to the setdatebought function and define it to work on the datelink field instead of the link field; and 3) redefine next to return a copy of the datelink field of the record pointed to by its parameter. A call to orderedinsert(&stocksdate) then carries out the task.

Example 3.5

Create a list stocks, to store a portfolio of stock records. n

To solve this problem, assume the data are already stored on some input medium and getnextrecord gives access to each record in turn, as it is invoked. The desired list may be created by traversing the records stored on the input medium. The job of process is to fill in the information field of the storage allocated for the new record, and fill in the link field of the new record with a pointer to storage allocated for the next new record. However, it is assumed that the last input data is a sentinel value that indicates there are no more records. When the sentinel is input, getnextrecord returns a null pointer value. Consequently, process must first check for this case and, when it occurs, simply return after setting done to true and the link field of the predecessor of the new record to null, unless the list is null. If it is null, process must set done to true and listname to null. Initialize must allocate storage for the first record and set listname to it. Finalize is not needed. We have

    >Comment

initialize(plistname)

listpointer *plistname;

{

listpointer avail();

*plistname = avail();

}

    >Comment

process(plistname,recordpointer,pdone)

listpointer *plistname,recordpointer;

int *pdone;

{

static listpointer predecessor;

listpointer setnull(),avail(),getnextrecord(),

pointer;

pointer = getnextrecord();

if(pointer != SENTINEL)

{

setinfo(recordpointer,pointer);

setlink(recordpointer,avail());

predecessor = recordpointer;

}

    >Comment

else if (*plistname != recordpointer)

{

setlink(predecessor,setnull());

*pdone = TRUE;

}

else

{

    >Comment

*plistname = setnull();

*pdone = TRUE;

}

}

Invoking a correctly modified traverse(&stocks) creates the list. Notice that this traverse will copy a list whenever getnextrecord obtains its information from the next record of the list to be copied. A better name for this function would be readlist or copylist. At this point you may want to see a complete program that creates the stocks list and also prints it. The program is in Section 3.6.1.

Suppose the records are given in arbitrary order, but the stocks list must be created with the records appearing in alphabetical order. To turn traverse into a solution to this problem, initialize, finalize, and anotherrecord are not needed; neither are the variables recordpointer and other. This problem is solved by the following version of traverse, which we call expand. The name expand was chosen because, when called to work on an existing ordered list, it simply expands the list by adding the new inputs to the list in the proper places. When the call expand(&listname) is made with listname set to null, it creates the ordered list.

    >Comment

expand (plistname)

/*Inputs a series of records and

inserts them in alphabetical

order in the ordered list listname.

*/

listpointer *plistname;

{

int done;

done = FALSE;

while(!done)

    >Comment

processrecord(plistname,&done);

}

    >Comment

processrecord(plistname,pdone)

listpointer *plistname;

int *pdone;

{

listpointer pointer,getnextrecord();

pointer = getnextrecord();

if(pointer != SENTINEL)

    >Comment

insertinorder(plistname,pointer);

else

*pdone = TRUE;

}

Notice that expand has no need for the functions initialize, finalize, and anotherrecord, nor for the variables recordpointer and other.

Processrecord must obtain the next input record and insert it in the list at the correct place. It invokes insertinorder, which has the task of creating and inserting the new record into its proper place in listname; pointer points to the record whose content is to be placed in the new record's information field. Insertinorder itself requires a traversal of listname. The version below is a modification of orderedinsert, obtained by adding a second parameter, pointer. Orderedinsert uses getnextrecord to obtain the next input record and then inserts it, while insertinorder is given a pointer to the new record and needs only insert it. The additional task of obtaining the new input record is handled outside of insertinorder. Consequently its process and finalize functions can be used directly in insertinorder. Only its initialize function needs changing, but all are given below for easy reference.

    >Comment

insertinorder(plistname,pointer)

/*Inserts the record pointed to by pointer

in the ordered list listname.

*/

listpointer *plistname,pointer;

{

listpointer recordpointer,predecessor,newpointer,

next();

int done;

    >Comment

initialize(&newpointer,&predecessor,pointer);

recordpointer = *plistname;

done = FALSE;

while(!done&&anotherrecord(*plistname,

recordpointer))

{

    >Comment

process(plistname,recordpointer,newpointer,

&predecessor,&done);

recordpointer = next(recordpointer);

}

    >Comment

finalize(plistname,predecessor,pointer);

}

    >Comment

initialize(pnewpointer,ppredecessor,pointer)

listpointer *pnewpointer,*ppredecessor,pointer;

{

listpointer setnull(),avail();

*pnewpointer = avail();

setinfo(*pnewpointer,pointer);

*ppredecessor = setnull();

}

    >Comment

process(plistname,recordpointer,newpointer,

ppredecessor,pdone)

listpointer *plistname,recordpointer,

newpointer,*ppredecessor;

int *pdone;

{

listpointer setnull(),next();

if(precedes(newpointer,recordpointer))

{

    >Comment

insert(plistname,*ppredecessor,newpointer);

*pdone = TRUE;

}

else if(next(recordpointer) == setnull())

{

    >Comment

insert(plistname,recordpointer,newpointer);

*pdone = TRUE;

}

else

    >Comment

*ppredecessor = recordpointer;

}

    >Comment

finalize(plistname,predecessor,newpointer)

listpointer *plistname,predecessor,newpointer;

{

listpointer setnull();

if(*plistname == setnull())

insert(plistname,predecessor,newpointer);

}


Data Structures Using C: 3.5.1 Why Traverse Was Useful

3.5.1 Why Traverse Was Useful

It is very important to recognize that traverse of Section 3.4.2 is a tool that could be used in solving the example problems. The tool was applicable because

It was written using data abstractions that allowed writing or using required implementation functions without changing traverse.

It was written using functional modularization, which allowed isolation of functions for relevant tasks so that they could be written to satisfy specific needs.

Recognizing and applying tools is a major aspect of programming methodology. For a tool to be convenient to use, it must be written according to the two elements of programming style just stated. Of course, in the simple examples using the tool may not save much time, effort, or cost, but for complex tasks the savings can be great.

Note that the use of getnextrecord makes the solutions to Examples 3.4 and 3.5 independent of the input medium or the format of the input. Thus any changes in the input are not reflected in traverse; only getnextrecord and SENTINEL can require modification. In fact, the same should be done for any output. It is better to use a function rather than specify input and output in the basic functions, in case the input or output medium is changed.


Data Structures Using C: 3.6: Implementing Lists

3.6: Implementing Lists

So far this chapter has covered lists in the abstract. Lists are implemented in two basic ways, differentiated by where the records of the lists are stored: 1) in an array of records or 2) in dynamic memory. These are similar, but dynamic memory allows for more efficient use of storage and is easier to use.

3.6.1 Lists Stored in Dynamic Memory

3.6.2 Lists Stored in an Array of Records

3.6.3 Lists Stored in Languages without Records


Data Structures Using C: 3.6.1 Lists Stored in Dynamic Memory

3.6.1 Lists Stored in Dynamic Memory

The simplest implementation is to store all list records in the dynamic memory. The declarations required are illustrated below for the example of the stock portfolio. For storage in dynamic memory, the records of a list are defined as usual, and an additional field is added to contain the link field pointer. Since it will point to a successor record stored in dynamic memory, it must be defined as a pointer. In this case, the null value for a pointer is always given by the value , with having the value 0. Variables of type listpointer will contain pointers to records of type stockrecord.

#define MAXSIZE 5

#define NULL 0

#define SENTINEL -1

typedef struct

{

int month;

int day;

int year;

}date;

typedef struct

{

char name[MAXSIZE];

int shares;

float value;

date datebought;

}infofield;

typedef struct listrecord

{

infofield info;

struct listrecord *link;

}stockrecord,*listpointer;

The following complete program reads in and creates a list of stockrecords as in Example 3.5 and, in addition, prints out the information fields of the list records. Both the creation and printing portion of the code are appropriately modified versions of traverse. The initialize and process functions of Example 3.5 are used in the creation portion, but process has been given the name addrecord. The program treats the list as a data abstraction. The data abstraction list, with its allowed operations�next, setlink, setinfo, avail, anotherrecord, getnextrecord, and printrecord, appear together in the program.

Table 3.2 Sample Input

                                Explanation                   Actual Input

--------------------------------------------------------------------------

Record 1         Number of shares                              100

                 Stock name                                    abc

                 Price of a share                              56.25

                 Month day year of purchase                    1 23 1978

Record 2         Number of shares                              50

                 Stock name                                    wxyz

                 Price of a share                              5.50

                 Month day year of purchase                    11 4 1986

                 .

                 .

                 .

Terminal record  Sentinel (where next number of shares would   -1

                   otherwise appear)

Typical input to the program is shown in Table 3.2.

The program that follows reads in the records of a list, creates the list, and prints the records in the list.

>Comment

#define MAXSIZE 5

#define NULL 0

#define SENTINEL -1

typedef struct

{

int month;

int day;

int year;

}date;

typedef struct

{

char name[MAXSIZE];

int share;

float value;

date datebought;

}infofield;

typedef struct listrecord

{

infofield info;

struct listrecord *link;

}stockrecord,*listpointer;

listpointer next(pointer)

/* Returns a copy of the

link field in the record

pointed to by pointer

*/

listpointer pointer;

{

return(pointer->link);

}

listpointer setnull()

/* Returns a null pointer */

{

return(NULL);

}

setlink(pointer1,pointer2)

/* Sets the link field of the

record pointed to by pointer1

to the contents of pointer2

*/

listpointer pointer1,pointer2;

{

pointer1->link = pointer2;

}

setinfo(pointer1,pointer2)

/* Sets the infofield of the record

pointed to by pointer1 to the

infofield of the record

pointed to by pointer2

*/

listpointer pointer1,pointer2;

{

int i;

for(i=0;i<MAXSIZE;i++)

pointer1->info.name[i] = pointer2->

info.name[i];

pointer1->info.shares = pointer2->

info.shares;

pointer1->info.value = pointer2->info.value;

pointer1->info.datebought.month = pointer2->

info.datebought.month;

pointer1->info.datebought.day = pointer2->

info.datebought.day;

pointer1->info.datebought.year = pointer2->

info.datebought.year;

}

listpointer avail()

/* Returns a pointer to storage

allocated for a record of type

stockrecord

*/

{

listpointer malloc();

return(malloc(sizeof(stockrecord)));

}

anotherrecord(recordpointer)

/* Returns true if there is

another record

*/

listpointer recordpointer;

{

listpointer setnull();

return(recordpointer != setnull());

}

listpointer getnextrecord()

/* Inputs the data for the new

record and returns a pointer

to a record containing the data

in its infofield, or, if there

are no new records,returns

a null pointer

*/

{

listpoiner setnull();

static stockrecord nextrecord;

printf("Enter number of shares\n");

scanf("%d",&(nextrecord.info.shares));

if(nextrecord.info.shares != SENTINEL)

{

printf(" Enter the stock name - less than

%d characters\n",MAXSIZE);

scanf("%s",nextrecord.info.name);

printf(" Enter the price of one share of

the stock\n");

scanf("%f",&(nextrecord.info.value));

printf(" Enter the month day year of the

stock purchase\n");

scanf("%d %d %d",&(nextrecord.info.

datebought.month),

&(nextrecord.info.datebought.day),

&(nextrecord.info.datebought.year));

return(&nextrecord);

}

else

return(setnull());

}

printrecord(recordpointer)

/* Prints the contents of the infofield

of the record pointed to by recordpointer

*/

listpointer recordpointer;

{

printf(" The stock is %s\n",

recordpointer->info.name);

printf(" The number of shares is %d\n",

recordpointer->info.shares);

printf(" The value of a share is %f\n",

recordpointer->info.value);

printf(" The date of purchase is %d %d %d\n\n",

recordpointer->info.datebought.month,

recordpointer->info.datebought.day,

recordpointer->info.datebought.year);

}

    >Comment

#include <stdio.h>

#define TRUE 1

#define FALSE 0

main()

/* Inputs a series of stockrecords,

creates a list of the records, and

prints the contents of each list

record's infofield

*/

{

listpointer stocks,recordpointer,next();

    >Comment

int done;

initialize(&stocks);

recordpointer = stocks;

done = FALSE;

while(!done&&anotherrecord(recordpointer))

{

    >Comment

addrecord(&stocks,recordpointer,&done);

recordpointer = next(recordpointer);

}

    >Comment

recordpointer=stocks;

while(anotherrecord(recordpointer))

{

    >Comment

printrecord(recordpointer);

recordpointer = next(recordpointer);

}

}

initialize(plistname)

/* Allocates storage for the first record

and sets listname to point to it

*/

listpointer *plistname;

{

listpointer avail();

*plistname = avail();

}

addrecord(plistname,recordpointer,pdone)

/* Fills in the current record's data, and allocates

storage for the next record and adds it to the

list. If there is no data for the current record

it sets the link field of the last list record

to null, or the list head to null, and done to

true.

*/

listpointer *plistname,recordpointer;

int *pdone;

{

static listpointer predecessor;

listpointer setnull(),avail(),getnextrecord(),

pointer;

pointer =getnextrecord();

if(pointer != setnull())

{

    >Comment

setinfo(recordpointer,pointer);

setlink(recordpointer,avail());

predecessor = recordpointer;

}

else if(*plistname != recordpointer)

{

    >Comment

setlink(predecessor,setnull());

*pdone = TRUE;

}

else

{

    >Comment

*plistname = setnull();

*pdone = TRUE;

}

}


Data Structures Using C: 3.6.2 Lists Stored in an Array of Records

3.6.2 Lists Stored in an Array of Records

Instead of being stored in dynamic memory, list records may be stored in an array of records. The declarations required are illustrated for the example of stocks.

#define MAXSIZE 5

#define MAX 100

#define NULL -1

typedef struct

{

int month;

int day;

int year;

}date;

typedef struct

{

char name[MAXSIZE];

int shares;

float value;

date datebought;

}infofield;

typedef struct listrecord

{

infofield info;

>Comment

int link;

}stockrecord;

stockrecord records[MAX];

>Comment

typedef in listpointer;

For this implementation the null pointer value is -1. Here, variables of type listpointer contain integer array indices. The implementation for some of the functions operating on the list are

next(pointer)

/* Returns a copy of the

link field in the record

pointed to by pointer

*/

listpointer pointer;

{

return(records[pointer].link);

}

setnull()

/* Returns a null pointer */

{

return(-1);

}

setlink(pointerl,pointer2)

/* Sets the link field of the

record pointed to by pointer1

to the contents of pointer2

*/

listpointer pointer1,pointer2;

{

records[pointer1].link = pointer2;

}

This implementation assumes that records is declared nonlocally to these functions, so that they may reference it. With this implementation, the records array plays the role of the dynamic memory. It is necessary to write an avail function that must keep track of those slots in records that are not in use, and hence can be allocated when storage for a list record is requested.


Data Structures Using C: 3.6.3 Lists Stored in Languages without Recor

3.6.3 Lists Stored in Languages without Records

In some languages, such as FORTRAN, there is no dynamic memory, nor are there pointer variables or records. In such a language, the records themselves must be implemented by the programmer. They may be stored in single or multiple arrays, as discussed in Chapter 2. Since the list records now have an additional link field indicating their successor, they need not be stored sequentially and may be located anywhere in the array or multiarrays, as long as they do not overlap. An additional variable is needed for the head of the list. Once these arrays are set up, the programmer may write functions to access each field of a record and functions to copy one record's value into another record, or to copy a record's field value into another record's field. These might include

next(pointer)

Returns a copy of the link field value of the record pointed to by pointer.

setlink(pointer1,pointer2)

Sets the value of the link field of the record pointed to by pointer1 to pointer2.

setlinfo(pointer1,pointer2)

Sets the value of the information field of the record pointed to by pointer1 to the value of the information field of the record pointed to by pointer2.

In FORTRAN, the array or multiarray in which the records are kept could be declared to be stored in the FORTRAN common area or else must be passed as parameters to these functions. Once they are written, for all intents and purposes the language has records. These records can be accessed, copied, or modified by using these functions. Although the programmer must do all this work initially, from this point on the situation is conceptually the same as in Section 3.6.2. That is, the actual implementation of the records may be forgotten; they are treated as if stored in an array of records, just as in C. The problem of writing an avail function still remains, and it is addressed in the next section.

In FORTRAN, although storage for records kept in lists must be managed, the programmer does not need to manage storage for individual elements or for arrays of elements. To refer in a program to an element n, or an array data, simply declare n to be an element, and data to be an array. Then use the operations available in FORTRAN to enter specific values into n or data. This works because individual elements and arrays are data structures directly available in FORTRAN. Lists, on the other hand, are not directly available and must be built up explicitly by storing their records in arrays. Consequently, the programmer must determine where storage is available to create a record when the need arises.

In C it is not necessary to implement records of lists explicitly in arrays. Instead, dynamic data structures may be used. A data structure is dynamic if it is built during execution of a program and storage is allocated for its components at that time. Dynamic data structures are built simply by asking for storage to be allocated for a record in dynamic memory. A pointer will then be assigned to point to that storage. Using the pointer, appropriate values may be assigned to the fields of the new record. These fields may be referred to directly, using naming conventions in the language. Special functions to store or access information from the fields of the record need not be written.

Records still need to be inserted properly in a dynamic data structure. Commands directly available in the language may be used to create or reclaim records, or reclamation may be left to the implicit storage reclamation provided by the language. Thus the definition, creation, reclamation, and manipulation of dynamic data structures become easier.


Data Structures Using C: 3.7: Keeping Track of Available Records

3.7: Keeping Track of Available Records

Keeping track of and allocating storage that is needed during the execution of a program is called dynamic storage management. In some languages, notably FORTRAN, storage for arrays is allocated before the program's execution, and no dynamic storage management is present. In languages like C, storage is allocated to the local variables of a function at the start of its execution and remains assigned until the function (or block) completes its execution. The space can then be reassigned to the next executing function (or block). In effect, it disappears. This is why values of local variables are not preserved between invocations of a function, unless they are declared as static, in which case they are preserved between calls. Management of this storage during the execution of a program is not what is meant by dynamic storage management. Contrast this situation to dynamic memory, which is assigned or taken away from records in dynamic memory during the execution of a function itself. This captures the true flavor of dynamic storage allocation.

If the array in which records are to be kept (as in Sections 3.6.2 and 3.6.3) is to function as dynamic memory, then allocating storage in the array for a record requires dynamic storage management. In fact, this amounts to an implementation of a C-like dynamic memory. Although storage can thus be managed explicitly for a specific program, dynamic memory can manage storage for many programs and is hence more efficient use of storage. The difference is that one large array may function as storage to be used by all the programs, instead of dedicating separate smaller arrays to each program.

3.7.1 Why Dynamic Memory?

3.7.2 Using Dynamic Memory


Data Structures Using C: 3.7.1 Why Dynamic Memory?

3.7.1 Why Dynamic Memory?

Why is storage efficiency important? Why not find storage for a new record in the array simply by placing it sequentially below the last new record's storage? After all, this would be very easy to do and would certainly use the storage as efficiently as possible. The answer is that this simple solution is a great idea�provided that all the storage needed for the array to be of sufficient length is available. The catch is that this proviso is rarely the case. Often, the amount required isn't known in advance; even when it is known, not enough storage may be available.

Reconsider the prime number example of Chapter 1. It is probably evident by now that the list is a good data structure to use for collection. The program to generate all the primes between 2 and n uses very little storage besides that needed for the list, but how large should you make the array in which to store the list? It is possible to modify the program so that it need not actually store all integers between 2 and n in the list, but only those that are prime. Still, how many primes are there no greater than n? It is even possible to write such a program so that it efficiently generates and prints all primes � n yet stores only those [image] n in the list, but again, how many of these are there?

Suppose you are a programmer asked to store the contents of ten manuscripts in ten lists. If each can have a maximum length of 50,000 words, then storage for 500,000 words is needed. How long is the longest word? You see the problem. Even if the amount of storage could be estimated, sufficient storage may not be available. You might conclude that the task cannot be done, or you might use secondary storage (magnetic tape, disks). However, in secondary storage access is relatively slow, and it requires techniques different from those described so far. One possibility remains. Maybe the manuscripts to be stored, perhaps even more than 10, are compatible (in the sense that they are not all of great length simultaneously). Perhaps the lengths will vary as the manuscripts are processed by a program, so that at any one time the manuscripts being stored can fit. This means that they may be growing and shrinking in length at different points in the program. If the storage no longer needed for one manuscript is relinquished and reused for another, then the total array length may not ever be exceeded. In other words, if storage can be allocated, then reused for new allocations when no longer needed for its original purpose, problems may be solved that are otherwise out of range. This is the reason to manage storage.

This also explains why dynamic memory makes better use of storage than individual program management using arrays of records. Each program may, to ensure its execution, ask for a large amount of storage to be used for its array. The total amount needed by all the programs may not be at hand. It is better to take all available storage and dole it out to each program as needed, the hope being that whenever some programs need a great deal of storage, the others need little. Instead of dedicating amounts to each user for each one's worst case, the programmer manages the storage and gives out only what is needed at any moment.


Data Structures Using C: 3.7.2 Using Dynamic Memory

3.7.2 Using Dynamic Memory

Having taken this relevant digression, let's return to the job of managing dynamic memory. This will show how dynamic memory itself might function, and give us the means to create our own when desirable. In fact, you will see when this might be advantageous.

Pointers are used in C to eliminate the problem of dynamic storage management for the programmer. Pointers allow the declaration of a variable of a complex data type and allocate storage for it by merely assigning to its pointer a value that points to that storage. The C function malloc is used for this purpose. This section develops a function avail that will have the same effect as malloc but will manage storage in the records array for Section 3.6.2.

In the general case of list processing, records need to be inserted in and deleted from lists. It is deletion, you may recall, that makes the solution of larger problems feasible. When records are deleted, their storage in the records array may be reused. Reclaiming storage in this way allows larger problems to be solved than would otherwise be possible, by allowing the program to execute to completion.

Records of one or more lists may be stored in the records array at any moment during the execution of a program. Assume that all these records have the same length, and that no sharing of storage occurs. No sharing means that no list record has more than one pointer to it at any time. The more typical case of variable-length records and sharing of storage is more complex and will be dealt with in a later chapter. The issue is now how to implement avail.

Suppose that, by looking at a record, you can tell whether or not it is in use. This would be the case if, whenever a record were deleted from a list, it were marked unused. An available record could be found by traversing the records array until a record marked unused were found. Avail could then return a pointer to that record, after marking it as now in use. In Figure 3.12, this would require the first three records to be accessed before avail could return to point to position 2. The time to carry out this procedure is proportional to the length of the records array in the worst case.

A very significant improvement would be to keep track of the available records so that the search may be avoided and the time for avail to do its task reduced to a constant. This is achieved by using the link fields of each unused record to create a list of available records. Initially all entries of records are unused, so the list should contain every record of the array. Figure 3.13 shows how this list might appear for the configuration of Figure 3.12, with availist taken as its head. Avail now carries out its task by simply deleting the first record from availist and returning a pointer to the deleted record. This yields the desired constant time for avail.

Figure 3.12 Typical Configuration of Used and Unused List Records in the RECORDS Array

Figure 3.13 The List of Available Records Corresponding to Figure 3.12

When records are deleted from a list in programs, it is convenient to keep track of their storage for future use. This requires inserting the deleted records' storage on availist. Again, this operation can be done in constant time if the newly available record is inserted at the front of availist�as its new first record. The function that carries out this task will be called reclaim. What time is required for insertion if the record is inserted in the list other than at the front?

The programmer invokes p = avail() to request storage for a record to be stored in records, and p points to that storage when avail returns. The programmer invokes reclaim(p) to request that the storage in records pointed to by p be inserted on availist. By employing avail and reclaim appropriately, the programmer can dynamically allocate and reclaim storage for lists. This means that records can be viewed as the repository of all list records, thus acting as a miniature dynamic memory. Such routines are the heart of storage management algorithms. When the records of a list are not all of the same length, more complex management techniques must be used. When records can be shared, still more complexity is added to the storage management problem, as will be illustrated subsequently.

Malloc is the general C allocation function provided for the programmer's use, so that dynamic memory appears as an abstract facility for storing records. When no memory is available to be allocated, malloc returns a NULL value. Consequently, on each call to malloc the value returned should be checked to be certain it is not the NULL value. Avail may be written to abort or take whatever action is necessary for your application in the event memory is not available.

The C function free is provided to carry out the general task of reclaim. Free deallocates memory that was previously allocated by malloc. The argument to free has to be a pointer previously returned by malloc that has not already been freed. Once memory is freed, it can be used to satisfy new requests through malloc.

In languages such as FORTRAN, which do not provide dynamic memory, you can implement it through avail and reclaim. Once avail and reclaim are written, for all intents and purposes the language has dynamic memory for record storage. You have tailored it to your needs.


Data Structures Using C: 3.8: Sequential Arrays versus Lists for Recor

3.8: Sequential Arrays versus Lists for Record Storage

Arrays allow records to be stored contiguously and to be accessed randomly. With pointer arrays, random access is possible even for variable-length records. Lists are also data structures containing records. Lists provide the flexibility, when needed, to separate records�enhancing the ability to insert or delete, but reducing the capability of accessing records at random. Stored records can be traversed using either the sequential array or list implementation. The time to carry out the traversal need not be significantly different for the two methods.

Suppose that instead of processing the records in sequential order, as in a traversal, your program must access the records in arbitrary order. (In an airplane reservation system, for example, seat reservations must be processed immediately, in random order.) Having to access the records in arbitrary order means that the program must access and process the records even though the programmer cannot predict the order in which this must be done. The sequential representation, because it allows selection for fixed-length records, allows this random accessing to be accomplished in constant time. The list representation requires that random accessing be done by traversing to the needed record, starting from the first and accessing each succeeding record until the desired one is reached. If the required record is the ith, traversal will take time proportional to i. Thus randomly accessing records takes constant time for each record accessed with the sequential implementation, but time proportional to the desired record number for the list representation. This can result in significantly greater processing time for the list implementation when you are randomly processing records.

To insert one record in the collection (say, after the ith record) when using the sequential implementation, all succeeding records (the (i + 1)th, (i + 2)th, . . . , nth) must be moved down in the data array. This operation will take time proportional to the number of succeeding records (n - 1). Adding a record as the new first record gives the worst-case time, which is proportional to n. Inserting a new record using the list implementation, again assuming you have determined where it is to be inserted, takes a constant time�the time needed to change two pointers.

To delete one record, say the ith, using the sequential representation, requires that the succeeding i - 1 records be moved up. Thus the time required is proportional to i - 1. Using the list implementation requires a constant time for this deletion�the time needed to change one pointer. The time for deletion with a sequential implementation in the worst case is also proportional to n. It is obvious that insertion and deletion of one record can result in much greater execution time for the sequential implementation as compared to the list implementation.

The contrast is especially striking when record accesses, insertions, or deletions are performed on m records. The worst-case time to randomly access an entire collection of m records becomes proportional to m and m2, for the sequential and list implementations, respectively. For random insertions and deletions, these are reversed in favor of lists. This means that if m is doubled or tripled, the respective times increase fourfold or ninefold instead of merely doubling or tripling.

When selecting an implementation, the basic considerations are comparative execution times and relative frequencies for random access, insertion, or deletion.

The list implementation will require at least as much memory as the sequential implementation. For instance, if each record requires one element for the information field, and the list implementation requires another element for the link field, then lists would require twice as much storage. It is possible that part of the storage used to implement a record would be left unused in any case, so that using it for the link field requires no extra storage.


Data Structures Using C: 3.9: Case Study: Merging and the Perfect Shuf

3.9: Case Study: Merging and the Perfect Shuffle

Consider two sequences of numbers that are ordered, largest to smallest:

100 80 65 30 90 85 50 10

They may be merged to produce a new sequence containing all the numbers in order. For example, compare the two largest and place the larger one first in the final sequence. Continue comparing the two current largest and placing the larger in the next spot in the final sequence until all numbers have been placed. The result, of course, is

100 90 85 80 65 50 30 10

This merging algorithm takes 2n basic operations when there are n numbers in each original sequence. It is possible to merge the sequences using a different algorithm, which, when parallel processing is possible, can reduce the time to O(lg 2n) rather than O(2n) when n is a power of 2. It is not obvious how to do this. The solution involves

1. A "perfect shuffle" operation, for which we next develop a program

2. A compare-and-exchange operation, which we will discuss after the perfect shuffle

3.9.1 The Perfect Shuffle

3.9.2 Array Implementation of the Perfect Shuffle

3.9.3 List Implementation of the Perfect Shuffle

3.9.4 Merging Sequences of Entries


Data Structures Using C: 3.9.1 The Perfect Shuffle

3.9.1 The Perfect Shuffle

Using gambling terminology, a perfect shuffle of 2n cards occurs when the deck of cards is split into two halves and the shuffle interleafs the two halves perfectly. An original configuration of eight cards, with the configurations after two consecutive perfect shuffles, is shown in Figure 3.14 The cards are numbered one through eight. A third perfect shuffle would produce the original configuration.

Figure 3.14 The Result of Two Perfect Shuffles

Suppose a sequence of perfect shuffles is made for 2n cards. Eventually the original configuration must reappear. How many shuffles will it take before this happens? One way to find out is to execute code that performs such a sequence of shuffles, keeping track of how many are made, and printing this total when the original configuration occurs again. A program segment to do this can be written as follows:

initial(configuration,n);

numbershuffles = 1;

perfectshuffle (configuration,n);

while(!original(configuration,n))

{

perfectshuffle(configuration,n);

numbershuffles++;

}

Initial initializes configuration. Original returns true if configuration is the original, and false otherwise. Perfectshuffle changes configuration so that it reflects the new configuration resulting from a perfect shuffle. As written, this code is independent of how the configuration is actually implemented.


Data Structures Using C: 3.9.2 Array Implementation of the Perfect Shu

3.9.2 Array Implementation of the Perfect Shuffle

Suppose we decide to implement configuration as an array and use n as a pointer to the card in position n. Then initial and original may be defined, respectively by

>Comment

for(i=1;i<=(2*n);i++)

configuration[i] = i;

and

>Comment

temporiginal = TRUE;

i = 1;

while((i<n+1)&&temporiginal)

if(configuration[i] != i)

temporiginal = FALSE;

else

i++;

original = temporiginal;

Perfectshuffle is somewhat more complex. The first and last cards, 1 and 2n, never change position in configuration. After the shuffle, cards 1 and n + 1, 2 and n + 2, 3 and n + 3, . . . , i and n + i . . . , and n and 2n will appear in consecutive positions in the resultant configuration. An array, result, is used to hold the new configuration. In traversing the relevant n positions of configuration (the 0th is ignored), its ith entry is processed by copying configuration[i] and configuration[i+n] into their proper new consecutive positions in result. I serves as a pointer to the current entry of configuration being processed, and newi will point to the new position of that entry in result. The situation, when n is 6, is depicted in Figure 3.15(a) just before the fourth entry is to be processed.

If the following statements are executed

>Comment

result[newi] = configuration[i];

result[newi+1] = configuration[i+n];

i++;

newi = newi + 2;

then the program will have properly processed the ith and (i+n)th cards. It will also have updated i and newi so that they are pointing to the proper new positions, in configuration and result, respectively. The situation after the fourth element has been processed is as shown in Figure 3.15(b).

To complete the shuffle, result can be copied into configuration. A program segment for perfectshuffle might be as follows.

>Comment

i = 1;

while(i < n)

{

result[newi] = configuration[i];

result[newi+1] = configuration[i+n];

i++;

newi = newi + 2;

}

for(i=1;i<=(2*n);i++)

configuration[i] = result[i];

Notice that the solution involves a number of array traversals, each traversal processing accessed elements in a different way.

(a) Situation Just before the Fourth Entry Is Processed

(b) Situation Just after the Fourth Entry Is Processed

Figure 3.15 Two Arrays Used for a Perfect Shuffle of Six Elements


Data Structures Using C: 3.9.3 List Implementation of the Perfect Shuf

3.9.3 List Implementation of the Perfect Shuffle

Suppose you decided to implement configuration as a list of records stored in dynamic memory, and to use n as a pointer to the nth record of the configuration list. Figure 3.16 depicts these assumptions graphically for ten cards.

Figure 3.16 The List Implementation for CONFIGURATION

Perfectshuffle must, consecutively, move 6 before 2, 7 before 3, 8 before 4, and finally 9 before 5. Two pointers, l1 and n, are shown in Figure 3.16. Traversing configuration and processing the records pointed to by l1 and n will accomplish the movements, if the processing deletes the successor of the record pointed to by n, inserts this deleted record after the record pointed to by l1, and updates l1 to point to it. Move carries out this processing task.

Perfectshuffle may now be implemented by

>Comment

configurationpointer l1,newn;

l1 = configuration;

newn = l1;

while(l1 != n)

{

move(l1,n);

l1 =l1->link;

newn = newn->link;

}

n = newn;

Perfectshuffle in this form is a refinement of traverse. Move plays the role of process. It uses local variables hold1 and hold2 to keep track of the successors needed:

>Comment

move(pl1,n)

configurationpointer *pl1,n;

{

configurationpointer hold1,hold2;

hold1 = l1->link;

hold2 = n->link;

11->link = hold2;

n->link = hold2->link;

hold2->link = hold1;

l1 = l1->link;

}

Two complete programs that output the number of shuffles are presented at the end of this paragraph. The first is based on an array, and the second on a list implementation. In this case there is nothing to recommend the list implementation over the array implementation, since the configuration does not grow or shrink (except for the artificial limit on the array size that would be required by the array implementation). The two versions of perfectshuffle are both useful for the application to which we return in the section that follows the programs.

Array Implementation

#define TRUE 1

#define FALSE 0

>Comment

typedef int arraytype[53];

main() /* Perfect shuffle using arrays */

/* Drives the perfectshuffle */

{

arraytype configuration;

int numbershuffles,n;

    >Comment

printf("\n Enter an integer between 1 and 26 \n");

scanf("%d",&n);

initial(configuration,n);

numbershuffles = 1;

    >Comment

perfectshuffle(configuration,n);

    >Comment

while(!original(configuration,n))

{

perfectshuffle(configuration,n);

numbershuffles++;

}

printf("\n Number of shuffles = %d \n",numbershuffles);

}

initial(configuration,n)

/* Creates the initial configuration */

arraytype configuration;

int n;

{

int i;

for(i=1;i<=(2*n);i++)

configuration[i] = i;

}

    >Comment

perfectshuffle(configuration,n)

/* Carries out a perfect shuffle */

arraytype configuration;

int n;

{

arraytype result;

int i,newi;

i = 1;

newi = 1;

while(i < (n+1))

{

    >Comment

result[newi] = configuration[i];

result[newi+1] = configuration[i+n];

    >Comment

i++;

newi = newi + 2;

}

for(i=1;i<=(2*n);i++)

configuration[i] = result[i];

}

original(configuration,n)

/* Returns true if the original

configuration has recurred

*/

arraytype configuration;

int n;

}

int i,temporiginal;

i = 1;

temporiginal = TRUE;

while((i < (n+1))&&temporiginal)

if(configuration[i] != i)

temporiginal = FALSE;

else

i++;

return(temporiginal);

}

List Implementation

#define TRUE 1

#define FALSE 0

#define NULL 0

typedef struct configrec

{

int info;

struct configrec *link;

    >Comment

}configrecord,*configptr;

int num;

main() /* perfect shuffle using lists */

/* Drives the perfectshuffle */

{

configptr configuration,n;

int numbershuffles;

    >Comment

printf("\n Enter an integer between 1 and 26\n");

scanf("%d",&num);

initial(&configuration,&n);

numbershuffles = 1;

    >Comment

perfectshuffle(configuration,&n);

    >Comment

while(!original(configuration,n))

{

perfectshuffle(configuration,&n);

numbershuffles++;

}

printf("\n The number of shuffles = %d\n",numbershuffles);

}

initial(pconfiguration,pn)

/* Creates the initial configuration */

configptr *pconfiguration,*pn;

{

configptr p,q;

int i;

*pconfiguration = malloc(sizeof(configrecord));

p = *pconfiguration;

i = 1;

while(i != (2*num+1))

{

p->info = i;

if(i == num)

*pn = p;

i++;

p->link = malloc(sizeof(configrecord));

q = p;

p = p->link;

}

q->link = NULL;

}

    >Comment

perfectshuffle(configuration,pn)

/* Carries out a perfect shuffle */

configptr configuration,*pn;x

{

configptr l1,newn;

l1 = configuration;

newn = l1;

while(l1 != *pn)

{

    >Comment

move(&l1,*pn);

    >Comment

l1 = l1->link;

newn = newn->link;

}

*pn = newn;

}

original(configuration,n)

/* Returns true if the original

configuration has recurred

*/

configptr configuration,n;

{

int temporiginal,i;

configptr p;

temporiginal = TRUE;

i = 1;

p = configuration;

while((p != n->link)&&temporiginal)

if(p->info != i)

temporiginal = FALSE;

else

{

i++;

p = p->link;

}

return(temporiginal);

}

move(pl1,n)

/* Processes a record of configuration */

configptr *pl1,n;

{

configptr hold1,hold2;

hold1 = (*pl1)->link;

hold2 = n->link;

(*pl1)->link = hold2;

n->link = hold2->link;

hold2->link = hold1;

(*pl1) = (*pl1)->link;

}


Data Structures Using C: 3.9.4 Merging Sequences of Entries

3.9.4 Merging Sequences of Entries

Now that the perfect shuffle has been developed, let's return to the merging of two sequences. Set the first n entries of configuration to one of the sequences to be merged, and the second n entries of configuration to the other, but enter the second in reverse order.

Consider the following algorithm for merging two sequences of ordered (largest to smallest) entries, each of length n:

Apply perfectshuffle and compareexchange to configuration a total of 2n times.

Compareexchange is a function that compares each of the n pairs of adjacent configuration entries and exchanges their values whenever the second is larger than the first. Table 3.3 shows the results obtained as this algorithm is applied to our sample sequences of length 4. Since lg 2n = lg 8 = 3, the algorithm terminates after the third perfect shuffle and compare-and-exchange operation. Notice that this results in configuration containing the correct merged sequence.

When 2n is a power of 2, say 2k, then the k iterations always produce the correct merge. This is certainly not obvious, and it is not proven here, but one point is important. If this merge is implemented as a for loop and carried out conventionally (that is, sequentially), it will take time O(n lg n). The straightforward sequential merge discussed earlier takes only O(n) time. However, if the n moves of perfectshuffle and the n pair comparisons and interchanges of compareexchange are all done in parallel (at the same time), then the total time will be just O(lg n).

Table 3.3 The Result of Merging Two Sequences of Ordered Entries of Length 4

    Initial       After First      After First      After Second

 Configuration  Perfect Shuffle  Compare Exchange  Perfect Shulle

-----------------------------------------------------------------

      100               100              100             100

       80                10               10              85

       65                80               80              10

       30                50               50              65

       10                65               85              80

       50                85               65              90

       85                30               90              50

       90                90               30              30

  After Second      After Third      After Third

Compare Exchange  Perfect Shuffle  Compare Exchange

---------------------------------------------------

      100               100              100

       85                90               90

       65                85               85

       10                80               80

       90                65               65

       80                50               50

       50                10               30

       30                30               10

Parallel processing is beyond the scope of this text, but this application illustrates its power and some of the difficulties involved. Doing parallel processing correctly is tricky, since care must be taken to do the right thing at the right time. For instance, one could dress in parallel fashion, but certain constraints must be observed. Socks and shirt can be put on at the same time (a valet is needed of course, since a person has only two hands) followed by pants, with shoes and jacket next and in parallel. But shoes cannot go on before socks.

The perfect shuffle (and the topic of parallel processing) is pursued further in Stone [1971, 1980] and Ben-Ari [1982]. An application of these ideas to the stable marriage problem appears in Hull [1984].


Data Structures Using C: Exercises

Exercises

1. What record of a list has no pointer pointing to it from another list record? What record of a list has a null pointer in its link field?

2. If each record of a list has, in addition to its information and link fields, a "pred" field, then the list is called a two-way list. The pred field contains a pointer to the preceding record. The first record's pred field contains a null pointer. Write insertion and deletion functions corresponding to those of Section 3.3 for a two-way list.

3. A circular list is a list whose last record's link field contains a pointer to the first list record rather than a null pointer. Write insertion and deletion routines corresponding to those of Section 3.3 for a circular list.

4. Modify the solution to Example 3.1 so it is correct for

a. two-way list

b. A circular list

5. Write a function to interchange the records of a list that are pointed to by p1 and p2.

6. The solution developed for Example 3.1 essentially cycles pointers from one list record to the next. Instead, a solution could cycle information field values from one record to the next. Modify the solution so that it does this and compare the two execution times. What if the information field contains only a pointer to the actual information field value?

7. Consider the three arrays below�data, p, and s.

   DATA   P   S

---------------

0    1   -1   5

1   12    4   6

2   18    6   8

3    4    5   9

4    7    9   1

5    3    0   3

6   12    1   2

7  100    8  -1

8   20    2   7

9    5    3   4

An ordering of the integers in data is specified by p and by s . Arrays p[i] and s[i] point to the predecessor and successor of the integer stored in data[i]. A minus one indicates no predecessor or successor. Assume the first i elements of data, p, and s are properly set, as above, to reflect the usual ordering of integers. Suppose a new integer is placed into data[i+1].

a. If i is 10 and the new integer is 15, what will the new p and s arrays be after they are properly updated to reflect the correct ordering among the eleven integers? The data array, except for 15 being placed in position 10, is to be unchanged.

b. Write a program segment to update p and s after a new integer is placed in data. You might want to assume that first and last point, respectively, to the element of data containing the first and last integers in the ordering.

8. a. What does traverse (Section 3.4) do when process interchanges the record that recordpointer points to with its successor record?

b. Same as Exercise 8(a) except that, in addition, process then sets recordpointer to the link field value of the record to which it points.

9. Write a function to delete all the records from list l1 that also appear on the list 12.

In Exercises 10-13 you must turn traverse of Section 3.4 into a solution to the exercise by defining its functions properly.

10. Write a function to delete all duplicate records from an alphabetical list such as that of Example 3.4.

11. Write a function to insert a new record in front of the record pointed to by ptr. You should not keep track of a predecessor nor traverse the list to find a predecessor. A "trick" is involved here (see Section 3.3.4).

12. This is the same as Example 3.4, but assume the list is implemented as a two-way list.

13. Create a new list that is the same as l except that each record has been duplicated, with the duplicate inserted as the new successor of the record it duplicates.

14. Create a list l that consists of all the records of the list l1 followed by all the records of the list 12.

15. Write a function to print out the number of words for a list such as that of Example 3.2.

16. Write the required functions for traverse of Section 3.4 when list records are stored in dynamic memory and the list is implemented as

a. A two-way list

b. A circular list

17. A hospital has 100 beds. The array records contains records that represent, by floor and bed number, those beds that are not occupied. The first, second, and third elements of a record contain the floor number, bed number, and link value, respectively. How many beds are not occupied and what is the lowest floor with an available bed?

18. Change the appropriate pointers for Exercise 17 so that the records are kept in order by floor and bed number.

19. Show what the arrays floor, bed, and next might have in their entries if the beds list of Exercise 17 were implemented using three corresponding words of these arrays.

20. How would the records array of Exercise 17 change if bed 2 on the fifth floor became occupied?

21. Suppose bed 10 on the sixth floor became empty. Assuming that the records array is used only for the beds list, what changes might be made to it to reflect this new information?

22. This is the same as Exercise 16 except that the list records are stored in an array records .

23. A palindrome is a sequence of characters that reads the same from front to back as from back to front. ABCDDCBA, ABCBA, and NOON are palindromes. Write a function palindrome to return the value true if the list it is called to check represents a palindrome, and the value false otherwise. The list records are stored in dynamic memory.

24. Suppose four records, 13, 7, 1, and 2, were deleted from lists whose records are stored in records of Figure 3.13. Then three records were inserted. If reclaim returns records at the front of availist, depict the availist and records after this processing.

25. Do the same as in Exercise 24, but assume that reclaim returns records at the rear of availist.

26. a. Why isn't it reasonable for avail to delete a record from anywhere on availist except the front or rear?

b. Why isn't it reasonable for reclaim to insert a record anywhere on availist except the front or rear?

27. Suppose a record is deleted from a list stored in records, but reclaim is not invoked to return it to availist. Is it ever possible to reuse the storage relinquished by that record?

28. Suppose records with different lengths are stored in records using the techniques of Chapter 2. The availist contains the available records in arbitrary order. Describe how avail and reclaim might be implemented.

29. This is the same as Exercise 28, except that the availist contains the available records ordered by length.

30. Write a function to create a list whose records are defined by

typedef struct node

{

whatevertype info;

struct node *linkptr;

struct node *prevptr;

}listnode;

The input should consist of a sequence of information field values given in the order they should appear in the list. Linkptr points to the next record on the list, and prevptr points to the preceding record on the list.

31. Explain why pointer variables and the functions malloc and free remove the need for the programmer to manage storage for list records.

32. Both the sequential array and list representations for ordered records use arrays. Why are they different?

33. Suppose you are asked to store records that represent an inventory of books that are currently signed out from the library. Assume that frequent requests are made for output that includes the book title and the current borrower in alphabetical order by the borrower's name. Of course, books are frequently taken out and returned. Should a sequential array be used for the records or a list representation? Why?

34. Suppose records are stored using the sequential array implementation. Initially 1,000 records are stored. You are asked to delete every other record, starting with the first. After this deletion you must insert records before every third record, starting with the first record. Find an expression for the total number of shifts of records that must take place to accomplish these insertions and deletions.

35. Do Exercise 34, but instead of assuming a sequential implementation assume a list implementation.

36. Suppose you must access records in the order tenth, first, twelfth, thirtieth, and fiftieth. How much time will the five accesses take with a sequential representation, and how much time will they take with a list representation?

37. Describe a real-life situation in which the sequential array representation of ordered records is clearly more desirable, another in which the list representation is clearly favored, and a third situation that involves trade-offs between the two representations.

38. a. Write the initial and original routines for the array implementation of the perfect shuffle and for the list implementation of the perfect shuffle in Section 3.9.

b. Can original be reduced to checking only the nth card to see if it has returned to the initial configuration?


Data Structures Using C: Suggested Assignments

Suggested Assignments

1. Write and execute a program to input a series of lists, each of which contains a number of sentences. After inputting all the lists, the program should invoke functions shorten and printlist to work on each list. Shorten is to print each sentence of the list it is to deal with whose length exceeds 25, and delete such sentences from the list; printlist is to print the resultant list. You should write shorten by defining the functions of traverse so as to turn traverse into shorten. Use dynamic memory for record storage. Don't forget to echo print all input and annotate all output. Make up some fun sentences. Store one character in each list record, and, for simplicity, assume all sentences end with a period.

2. Modify your program (and run the new version) when three characters (instead of one) are stored per record. Hint: within a list, assume the first character of each sentence is stored in a new record rather than in the record containing the period of the previous sentence, even when it has the room.


Data Structures Using C: CHAPTER 4: INTRODUCTION TO RECURSION, STACKS,

CHAPTER 4: INTRODUCTION TO RECURSION, STACKS, AND QUEUES

Presents recursion as a special case of the top-down approach to problem solving

Explains recursive programs

how to understand them

how to simulate them

how to write them

Shows how recursive programs may be translated by a compiler or written in a non-recursive language

Introduces two more data structures

the stack

the queue

Illustrates the use of a stack with operations

setstack

empty

push

pop

Case study梒hecking sequences for proper nesting

uses the stack and treats it as a data abstraction

4.1: What Is Recursion?

4.2: Using Recursion

4.3: A Close Look at the Execution of Recursive Programs

4.4: Implementing Recursive Programs

4.5: Stacks

4.6: Evaluation and Translation of Expressions

4.7: Queues

4.8: Case Study: Checking Sequences for Proper Nesting

Exercises

Suggested Assignments


Data Structures Using C: 4.1: What Is Recursion?

4.1: What Is Recursion?

People can't help but use themselves�their own life experiences�as a point of reference for understanding the world and the experiences of others. Definitions, algorithms, and programs may also refer to themselves, but for a different purpose: to express their meaning or intention more clearly and concisely. They are then said to be recursive. Languages such as C that allow functions to call themselves are called recursive languages. When no self-reference is allowed, languages are said to be nonrecursive or iterative.

Self-reference is not without its pitfalls. You have most likely encountered the frustration of circular definitions in the dictionary, such as "Fashion pertains to style" and "Style pertains to fashion." From this you conclude that "Fashion pertains to fashion." This kind of self-reference conveys no information; it is to be avoided. "This sentence is false" is another type of self-reference to be avoided, for it tells us nothing. If true it must be false, and if false it must be true. Nevertheless, self-reference is an important technique in finding and expressing solutions to problems.

Recall that the top-down approach to problem solving entails breaking down an initial problem into smaller component problems. These components need not be related except in the sense that putting their solutions together yields a solution to the original problem. If any of these component problems is identical in structure to the initial problem, the solution is said to be recursive and the problem is said to be solved by recursion. If the solution and its components are functionally modularized, then the solution will refer to itself. Recursion is thus a special case of the top-down design methodology.

Suppose a traveler asks you, "How do I get there from here?" You might respond with a complete set of directions, but if the directions are too complex, or you are not sure, your response might be: "Go to the main street, turn left, continue for one mile, and then ask, "How do I get there from here?" This is an example of recursion. The traveler wanted directions to a destination. In solving the problem, you provided an initial small step leading toward the traveler's goal. After taking that step, the traveler will be confronted with a new version of the original problem. This new problem, while identical in form to the original, involves a new starting location closer to the destination.

Cursing is often used to belittle a problem; recursing has the same effect! Recursion, when appropriate, can give relatively easy-to-understand and concise descriptions for complex tasks. Still, care must be taken in its use, or else efficiency of storage and execution time will be sacrificed.


Data Structures Using C: 4.2: Using Recursion

4.2: Using Recursion

Applying the recursive method is just like applying the top-down approach, but it can be tricky; it presents conceptual difficulties for many students. The purpose of this section is to show how to develop and understand recursive solutions. Some illustrative examples are provided to help you get the idea. Practice will help you become skilled in the use of recursion.

4.2.1 The Towers of Hanoi

4.2.2 Verifying and Simulating a Recursive Program

4.2.3 The Length of a List

4.2.4 Copying a List

4.2.5 Local and Global Variables and Parameters

4.2.6 Counting Squares

4.2.7 Permutations


Data Structures Using C: 4.2.1 The Towers of Hanoi

4.2.1 The Towers of Hanoi

A game called the Towers of Hanoi was purportedly played by priests in the Temple of Brahma, who believed that completion of the game's central task would coincide with the end of the world. The task involves three pegs. The version considered here has n disks initially mounted on peg 1 (the priests dealt with 64 disks). The n disks increase in size from top to bottom, the top disk being the smallest. The problem is to relocate the n disks to peg 3 by moving one at a time. Only the top disk on a peg can be moved. Each time a disk is moved it may be placed only on an empty peg or on top of a pile of disks of larger size. The task is to develop an algorithm specifying a solution to this problem, no matter what the value of n.

Consider the case when n = 4, shown in Figure 4.1(a). The following sequence of instructions provides a solution.

1. Move the top disk from peg l to peg 2.

2. Move the top disk from peg 1 to peg 3.

3. Move the top disk from peg 2 to peg 3.

4. Move the top disk from peg 1 to peg 2.

5. Move the top disk from peg 3 to peg 1.

6. Move the top disk from peg 3 to peg 2.

7. Move the top disk from peg 1 to peg 2.

8. Move the top disk from peg 1 to peg 3.

9. Move the top disk from peg 2 to peg 3.

10. Move the top disk from peg 2 to peg 1.

11. Move the top disk from peg 3 to peg 1.

12. Move the top disk from peg 2 to peg 3.

13. Move the top disk from peg 1 to peg 2.

14. Move the top disk from peg 1 to peg 3.

15. Move the top disk from peg 2 to peg 3.

(a) The Initial Problem

(b) After Component 1

(c) After Component 2

(d) After Component 3 � the Solution!

Figure 4.1 The Towers of Hanoi Problem

It is not difficult to arrive at such a solution for a specific small value of n. Try to find a solution for n = 5. You will find that, even though you succeed after some trial and error, creating a solution for n = 25 is indeed a formidable task. You may even be too old to care by the time you finish. If a particular value of n gives so much trouble, how does one specify a solution for any n, when one does not even know what value n will have?

One answer is to apply the technique of recursion. This means attempting to break down the problem into component problems that can be solved directly or that are identical to the original problem but involve a smaller number of disks. The three component problems below fit the requirements of this framework.

1. Move the top n - 1 disks from peg 1 to peg 2.

2. Move the top disk from peg 1 to peg 3.

3. Move the top n - 1 disks from peg 2 to peg 3.

If these three problems can be solved, and their solutions are applied in sequence to the initial situation, then the result is a solution to the original problem. This can be seen from parts (a)-(d) of Figure 4.1, which show the consecutive situations after each is applied.

We now generalize the original problem:

Move the top n disks from peg i to peg f.

i denotes the initial peg on which the top n disks to be relocated reside, and f denotes the final peg to which they are to be relocated. Let a denote the third available peg. Solving the original problem, which has only n disks, is equivalent to solving the generalized problem with i taken as 1 and f taken as 3. This is because the moves that accomplish its solution will also work for the generalized problem, since any disks below the top n are larger than they are, so their existence doesn't render any of the moves illegal. Solving the first component problem is equivalent to solving the generalized problem with n replaced by n - 1, i by 1, and � by 2. It is now apparent that the original problem, and the components 1, 2, and 3, have structure identical to the generalized problem, but the components involve fewer disks. They also involve different initial, available, and final pegs. This is why it was necessary to generalize the original problem.

Suppose you have found a solution to the generalization and denote it by towers(n,i,a,f). Then, as just shown, the application of the solutions to components 1, 2, and 3, in sequence, will be a solution to the original problem. A recursive definition for towers(n,i,a,f) can now be expressed as follows. Note that moves in towers(n,i,a,f) are from peg i to peg f�that is, from the second parameter to the fourth parameter. The first parameter specifies the number of top disks, on the peg given by the second parameter, that are to be moved.

>Comment

To obtain towers(n,i,a,f):

If n = 1, then

>Comment

move the top disk from peg i to peg f

else

>Comment

apply towers(n - 1,i,f,a)

moves the top disks from peg i to peg f

>Comment

apply towers(n - 1,a,i,f).

This is the recursive algorithm for the Towers of Hanoi problem. Notice that the definition gives an explicit solution for the special case when n = l and an implicit solution for the case of n > 1.

To use the method of recursion we must recognize when it is applicable and find a correct way to generalize the original problem and then to break down the generalization. We have been successful and have found a recursive algorithm. Because no further refinement is needed, the corresponding recursive program can be written directly using functional modularization:

towers(n,i,a,f)

/* Moves the top n disks from peg i to peg f

*/

int n,i,a,f;

{

if(n == 1)

printf("\n %d -> %d\n",i,f);

else

{

towers(n-1,i,f,a);

[1]

printf("\n %d -> %d\n",i,f);

towers(n-1,a,i,f);

[2]

}

}

The labels [1] and [2] are not part of the program but are used later as references when explaining the program. It is essential to see that this function has been written just like any other; whenever a component task is available as a function, we invoke it to carry out that task. The feature that distinguishes it and makes it a recursive function is that the required functions happen to be the function itself. The references within towers to itself are allowed in recursive languages.

Anytime a function calls itself during its execution, a recursive call has been made. The two invocations of towers within towers represent such recursive calls. In general, when a recursive program executes on a computer, or when its execution is simulated, many recursive calls will be made. Simulation of a program or function call means carrying out its instructions by hand and keeping track of the values of all its variables. These are referred to as the first, second, etc., recursive calls. The initial call to the function precedes the first recursive call.


Data Structures Using C: 4.2.2 Verifying and Simulating a Recursive Pr

4.2.2 Verifying and Simulating a Recursive Program

The verification of a recursive program can be done in two ways, as is true for a nonrecursive program. One is by formal proof, the other by checking all cases. With recursion, the cases given explicitly are checked first. A case is given explicitly when the action taken by the program in that case does not require a recursive call. Also, it is assumed that each function individually works correctly. For towers(n,i,a,f), the only explicit case is when n = 1 and the function clearly outputs the proper result and terminates.

For the case when n exceeds l, Figure 4.1 has already confirmed that as long as the three component functions work properly, then towers(n,i,a,f) will also. Note that these components correspond to smaller values of n. Mathematically inclined readers may recognize that this really amounts to a proof of correctness for towers using mathematical induction on n. One of the advantages of recursive programs is that they may be proven correct more easily in this way.

In order to understand how a recursive program actually executes, it is necessary to simulate its execution. Below are the first few steps of the simulation for a call to towers(n,i,a,f) with the values 4, 1, 2, 3 -- that is, towers(4,1,2,3). It will soon become apparent that considerable bookkeeping is involved, requiring an organized way to do it.

Since n is greater than 1, the procedure requires the sequential execution of

towers(3,1,3,2)

printf("\n %d -> %d\n",1,3);

and

towers(3,2,1,3)

To do this requires suspending the execution of towers(4,1,2,3) at this point in order to simulate the first recursive call, towers(3,1,3,2). Note that after this recursive call is completed, 1�3 is to be output. (1 � 3 means "Move the top disk from peg 1 to peg 3.") Then another recursive call must be simulated, towers(3,2,1,3).

Dealing with towers(3,1,3,2), since n > 1, the commands

towers(2,1,2,3)

printf("\n %d -> %d\n",1,2);

and

towers(2,3,1,2)

must be carried out sequentially. Now the execution of towers(3,1,3,2) must be suspended in order to simulate the second recursive call, towers(2,1,2,3). Note that after this recursive call is completed, 1 � 2 is to be output, and then another recursive call must be simulated: towers(2,3,1,2).

Dealing with towers(2,1,2,3), since n > 1,

towers(1,1,3,2)

printf("\n %d -> %d\n",1,3);

and

towers(1,2,1,3)

must be carried out sequentially. These two calls to towers represent the third and fourth recursive calls. They both complete without generating any new recursive calls. The three statements result in the outputting of l � 2, 1� 3, and 2 � 3, respectively. This completes the second recursive call (towers(2,1,2,3)) so we must pick up the simulation at the proper point, which is to output l � 2 and then make the fifth recursive call to simulate towers(2,3,1,2).

Figure 4.2 Bookkeeping for the Simulation of a Recursive Program

Enough is enough! At this point you may feel like a juggler with too few hands and too many recursive calls. Figure 4.2 represents a convenient way to keep track of the complete simulation; it may be generated as the simulation proceeds. The figure traces the recursive calls made to towers as the initial call to towers(4,1,2,3) is executed to completion. Each recursive call is numbered to show whether it is the first, second, . . . , or fourteenth. The bracketed return values indicate the labeled statement of procedure towers at which the preceding recursive call is to resume when the next call is completed. For example, when towers(3,1,3,2) is completed, towers(4,1,2,3) resumes at [l]. Our simulation stopped just as the fifth recursive call was to be carried out. The figure uses shorthand for the output. For example, l � 2 means, "move the top disk from peg l to peg 2." The solution is the same as our earlier one.

Figure 4.2 indicates very clearly what information must be available at each point in the simulation (or execution of the program) in order to carry it out. Specifically, it is necessary to know the following information:

n The four parameter values of the call currently being carried out

n Where to resume when the call currently being carried out is completed

For example, when the third recursive call is completed, resumption must be at the printf statement of the second recursive call; when the fifth recursive call is completed, resumption must be at the end of the first recursive call; and finally, after the initial recursive call is completed, resumption must be at the proper point in the program that called towers originally.

Notice that the last three parameters are never modified by towers, but the first parameter, which corresponds to the number of disks for the current problem, is modified. For instance, the first recursive call to towers(3,1,3,2) has modified the first parameter, making it 3 rather than 4 as it was initially. It is important that this modification be kept local to towers(3,1,3,2), not reflected back to towers(4,1,2,3), the initial call.

It is not always possible to find a recursive solution to a problem, nor is it always desirable to use one when available. Sometimes a recursive solution is the only one we can find. For the Towers of Hanoi problem it is difficult to find a nonrecursive solution, but a number are known. Recursive solutions normally make it easier to see why they are correct, probably because to achieve them requires that you make the structure inherent in the problem more explicit.

To make this point clearly, the recursive algorithm and a nonrecursive algorithm are repeated here. Convince yourself that the nonrecursive algorithm solves the Towers of Hanoi problem correctly (see Exercise 8). The nonrecursive program is taken from Walsh [1982]; another nonrecursive program appears in Buneman and Levy [1980].

A Recursive Solution for the Towers of Hanoi

To obtain towers(n,i,a,f):

If n = 1, then

move the top disk from i to f

else

apply towers (n - 1,i,f,a)

move the top disk from i to f

apply towers(n - 1,a,i,f).

A Nonrecursive Algorithm for the Towers of Hanoi

l. Label the disks l, 2, . . . , n in increasing order of size.

2. Label the pegs i, a, and f, n + 1, n + 2, and n + 3, respectively.

3. Move the smallest disk onto another disk with an even label or onto an empty peg with an even label.

4. While all disks are not on the same peg

a. move the second smallest top disk onto the peg not containing the smallest disk

b. move the smallest disk onto another disk with an even label or onto an empty peg with an even label.

The Tower of Babel failed for lack of communication. The construction workers did not speak the same language. While recursion allows the solution to the Towers of Hanoi to be specified conveniently, it is execution time that causes trouble. The diagram of Figure 4.1 shows that the construction of the new tower of disks will take 24 - 1 or fifteen moves. It is not difficult to generalize the diagram to n disks to conclude that 2n - 1 moves will be needed for n disks (see Exercise 7). Even with Olympic-caliber construction workers, it would take over one thousand years to complete the tower for 50 disks.


Data Structures Using C: 4.2.3 The Length of a List

4.2.3 The Length of a List

We will now present a series of functions to further illustrate recursive programs. The functions

1. Determine the length of a list

2. Create a list that is a copy of another list

3. Count checkerboard placements

4. Generate all permutations of n integers and process each one. (This is applied to the stable marriage problem.)

In addition to providing repetition so that you can become more familiar with and understand recursion better, the examples provide a background to show how local and global variables and parameters are treated in recursive programs and why recursive solutions are not always the best. A list may be thought of as being either a null list or as one record followed by another list (the link field of the record plays the role of the head of this other list). This is nothing but a way to define a list recursively. Consider the recursive function defined by

length(listname)

/* Returns the number of records

in the list pointed to by listname.

*/

listpointer listname;

{

listpointer setnull(),next();

if(listname == setnull())

>Comment

return(0);

else

return(1 + length(next(listname)));

}

This function returns the length of the list, listname. To verify that this is correct we must check

1. The null list. It returns with length = 0.

2. A one-record list followed by another list. It returns length = 1 + the length of the list after the first record.

It is correct, since its value is correct for each case. Of course it is correct�after all, the length of a null list is zero, and the length of a list with at least one record is 1 for the record, plus the length of the rest of the list beyond the first record ! This was how length was written in the first place.


Data Structures Using C: 4.2.4 Copying a List

4.2.4 Copying a List

Consider the function copy, which creates a list second that is a copy of a list first:

copy(first,psecond)

/* Creates a new list which is a copy

of the list pointed to by first and

whose head is second

*/

listpointer first,*psecond;

{

listpointer null, rest,setnull(),avail(),next();

null = setnull();

if(first == null)

>Comment

*psecond = null;

else

{

>Comment

*psecond = avail();

    >Comment

setinfo(*psecond,first);

    >Comment

copy(next(first),&rest);

    >Comment

setlink(*psecond,rest);

}

}

When first is the null list, copy faithfully terminates with the copy also a null list. When first is a one-record list, copy sets second to point to storage allocated for a record and sets the information field value of this record to the value of the information field in first's first record. The recursive call to copy(next(first),&rest) is then an invocation of copy to copy the null list, since next(first) is null in this case. Assuming it performs its task correctly, rest will be null when it returns. The link field of the record pointed to by second is then set to the value of rest (the null pointer), and the initial call to copy terminates correctly.

Similarly, had there been more than one record in first's list, the recursive call to copy would have returned with rest pointing to a correct copy of the list pointed to by next(first). Setlink would correctly place a pointer to that correct copy into the link field of the record pointed to by second, and the initial call would terminate correctly. This verifies that copy is correct in all cases. Again, to copy the null list is easy. To copy a list with at least one record, simply copy that record, copy the rest of the list beyond the first record, and append that copy to the copy of the first record. This is how copy was written!

If you are confronted with a recursive program and want to understand it, approach it in the same way as you would a nonrecursive program. Assume that any functions that are called perform their functions correctly, and then attempt to see what the program itself does. Sometimes this will be apparent, as it was for towers and copy, and sometimes a simulation of the program helps to increase comprehension. Imagine, for example, that you were not told what copy did. It wouldn't have been so easy to find out without simulating some simple cases.


Data Structures Using C: 4.2.5 Local and Global Variables and Paramete

4.2.5 Local and Global Variables and Parameters

In order to understand what happens as a recursive program executes, it is necessary to review how local and global variables are treated, and also how parameters are treated. This must also be known to understand what happens when any program executes.

In nonrecursive programs, references to global and to static variables always refer to the current actual storage assigned to them. Hence any changes to their values during execution of the program will be reflected back in their values. Local automatic variables are treated differently. Local copies are made, and these are referred to during the execution of the function to which they are local. If the local variables are automatic, the storage for the local copies disappears when the function terminates or returns. If the local variables are static, the storage remains after the function is terminated, and the variable retains its value for the next time the function is called.

This situation holds for recursion also. Each recursive call to a function produces its own copies of automatic local variables. Thus there may be many different values for the same local variable�each applicable to its own invocation of the function. As with nonrecursion, there is only one storage allocation for a global or a static variable, and it holds the last value of the variable.

Parameters passed in the argument lists of functions are treated as variables local to the function. They are passed by value. New storage is established for each parameter. The storage is initially set to the value of the corresponding actual parameter when the function is invoked. In effect, this storage represents a local copy of the actual parameter. Any reference in the function is interpreted as a reference to the local copy only. This implies that any change to a parameter during the execution of the function will not be reflected back to the storage used for the actual parameter; it can not have its value changed by the function.

The only way to change the original value of a variable used as a parameter is to pass the address of the parameter instead of its value. This is done automatically in the case of arrays and character strings. When an array is passed as a parameter, the address of the first storage location in the array is passed. This negates the need to make local copies of all the values of the array in the function and thus waste memory. Addresses for parameters can also be passed as pointers A copy is made in the function and is local to the function. Using the pointer, the value of the parameter can be changed.

Here is a simple example.

simple (x,py)

int x, *py;

{

x = 1;

*py = 2;

*py = x + *py;

}

Suppose that simple is invoked with actual parameters a and b, containing 0 and 1 respectively. It is invoked by the statement, simple (a,&b). Within the function, x initially has a copy of a as its value�0 in this case. y refers to the storage for b�that is, y is a pointer to the storage for b. The first statement of simple causes x to assume the value l, the second statement causes the storage for b to contain 2, and the third statement sets the storage for b to 3. When the calling program continues, a will still be 0, but b will be 3.

Parameters of recursive programs are treated in just this way. Any time cursive call is made within a recursive program, the recursive call acts just like a call to any function. The only difference is that, since a recursive function may generate many recursive calls during its execution, many local copies of its parameters may be produced. References to a parameter by value during the execution of each recursive call will then mean the current local copy of that parameter. Reference to a parameter passed by a pointer to it always means the same actual parameter.

In the recursive version of copy, note that

n Null and rest are local variables of the initial call to copy and remain local variables in all subsequent recursive calls. Local copies of null and rest are thus made on any call to copy.

n First is passed by value and second by pointer.

n Whenever copy is executing and generates a recursive call to copy(next(first),&rest), the actual first parameter of copy is the current copy of next(first)�i.e., the copy associated with the currently executing call to copy. Any references to the first parameter during the generated recursive call refer to the new local copy associated with that call.

When first points to the list as shown in Figure 4.3, and copy(first, second) is executed, the sequence of events and memory configurations that will take place appear in Figure 4.4. First0, first1, first2, first3 refer to the local copies of the actual first parameter associated with the initial, first, second, and third recursive calls to copy, respectively. Similarly for the other variables. A question mark indicates an as yet undefined value. Be sure you understand Figure 4.4 and how it represents the execution of copy.

Figure 4.3 The List for COPY to Process

An alternative version of copy that returns a pointer to the copy of the list first would be as shown on page 160.

(a) The Situation after SETINF0 (*PSECOND, FIRST0)

(b) After SETINF0(REST1, FIRST1)

(c) After SETINF0(REST2, FIRST 2)

(d) After REST2:= NULL 3; i.e., SECOND:= NULL

(e) After SETLINK(REST1, REST2)

(f) After SETLINK(REST0, REST1)

Figure 4.4 Simulation of COPY(FIRST,&SECOND)

listpointer copy(first)

/* Returns a pointer to the first record of a new list which is a

copy of the list first.

*/

listpointer first;

{

listpointer null,second,setnull(),avail(),next();

null = setnull();

if (first == null)

>Comment

second = null;

else

{

>Comment

second = avail();

    >Comment

setinfo(second,first);

    >Comment

setlink(second,copy(next(first)));

}

return (second);

}

This version differs from the original version in two ways. One, it has only one parameter, the list to be copied, and two, it returns a pointer to the new copy rather than setting a second parameter to point to the new copy.


Data Structures Using C: 4.2.6 Counting Squares

4.2.6 Counting Squares

The third example relates to counting squares. Consider an n � n checkerboard and determine the number of ways a 4 � 4 checkerboard can be placed on it. For example, if n = 5, there are four ways to place a 4 � 4 checkerboard on the 5 � 5 board, as shown below.

For this particular example the solution is easy to see by inspection. However, how do you state an algorithm that produces the result in all cases? This can be done handily using recursion. Suppose you know the number of ways, f(n - 1), in which the 4 � 4 board can be placed on an (n - 1) � (n - 1) board. An additional (n - 3) + (n - 4) placements result from an n � n board, because of the n cells

Figure 4.5 An n x n Checkerboard

added at the top and left sides of the (n - 1) � (n - 1) board, as shown in Figure 4.5. There are a total of f(n) = f(n - 1) + (n - 3) + (n - 4) placements for the n � n board, when n � 5. If n = 4, f(4) = 1. So the following is true.

f(n) is

1 if n = 4

f(n - 1) + (n - 3) + (n - 4) if n � 5

It is easy to write a recursive function f, with parameter n, to return the correct result for any n � 4.

f(n)

int n;

{

if (n == 4)

>Comment

return (1);

else

>Comment

return(f(n-1)+(n-3)+(n-4));

}

If n = 7, the program would execute as follows:

    >Comment

f(7) = f(6) + (7 - 3) + (7 - 4)

    >Comment

f(6) = f(5) + (6 - 3) + (6 - 4)

    >Comment

f(5) = f(4) + (5 - 3) + (5 - 4)

    >Comment

f(4) = 1

    >Comment

f(5) = 1 + 2 + 1 = 4

    >Comment

f(6) = 4 + 3 + 2 = 9

    >Comment

f(7) = 9 + 4 + 3 = 16

A simpler, more direct solution can be obtained by noticing that the top left corner of the 4 x 4 board can be placed in the first, second, . . . , (n - 3)th position of the top row of the n � n board, resulting in (n - 3) different placements. The second row also allows (n - 3) placements. In fact, each of the first (n - 3) rows allows (n - 3) placements, for a total of (n - 3) � (n - 3) distinct placements.Thus f(n) = (n - 3) � (n - 3) for n � 4. The program for this solution would be

f(n)

int n;

{

return((n-3)*(n-3));

}

The recursive solution will take time and storage O(n), whereas the nonrecursive solution requires constant time and storage. Obviously, this solution is better in every way than the recursive solution to this problem. Hence recursion does not always yield the best solution. Just as with any tool, it must be used judiciously. You wouldn't use a bulldozer to dig a two-foot hole in your yard.


Data Structures Using C: 4.2.7 Permutations

4.2.7 Permutations

The final example returns to the stable marriage problem. Suppose you are asked to find a stable pairing for the n men and women of the stable marriage problem in Chapter 2, but you aren't aware of the algorithm discussed in Section 2.5. Probably the most straightforward approach is to search through all the possible pairings until a stable pairing is found. Each possible pairing may be thought of as a specific permutation of the n women. Thus the example pairing of Table 2.2 corresponds to the permutation (or listing) 4, 3, 1, 2, 5. In general there are n ! such permutations (where n! means n factorial).

This approach yields a program that traverses the collection of permutations, processing each one and halting when a stable permutation is encountered. Stable may be conveniently invoked to check a permutation for stability. In contrast, the algorithm of Section 2.5 avoids such an exhaustive search by eliminating some permutations from consideration altogether. Still, many problems require just such an exhaustive search and contain a component to produce the permutations. The problem is to write a program to carry out the traversal. The function permutations does this:

permutations(n)

/* Generates and processes each

permutation of the integers

1 to n.

*/

int n;

{

int done;

done = FALSE;

>Comment

initialize(n);

while (!done)

{

>Comment

process(n);

    >Comment

next(n,&done);

}

}

where next (n,&done) must return, having updated the current permutation to the next permutation, unless the current permutation is the last one, when it simply sets done to true.

The problem is now to refine next. Suppose n = 4 and the first permutation is 3 2 1. Inserting 4 in all possible positions among 1, 2, and 3 gives the first four permutations of 1, 2, 3, and 4.

4 3 2 1

3 4 2 1

3 2 4 1

3 2 1 4

Removing 4 leaves 3 2 1. The next four permutations for n = 4 can be obtained by first obtaining the next permutation after 3 2 1 for n = 3. But this next permutation for n = 3 can similarly be obtained by shifting 3 to the right one place, yielding 2 3 1. Then the next four permutations for n = 4 are

4 2 3 1

2 4 3 1

2 3 4 1

2 3 1 4

Removing 4 leaves 2 3 1. Again, the next permutation after 2 3 1, 2 1 3, is obtained by shifting 3 right. The next four permutations for n = 4 are

4 2 1 3

2 4 1 3

2 1 4 3

2 1 3 4

Continuing this process will produce, in turn,

4 3 1 2    4 1 3 2           4 1 2 3

3 4 1 2    1 4 3 2           1 4 2 3

3 1 4 2    1 3 4 2           1 2 4 3

3 1 2 4    1 3 2 4    and    1 2 3 4

Generalizing, we see that, given a permutation of the n integers from 1 to n, the next permutation can be generated by shifting n to the right whenever possible. The only time this is not possible is if n is already at the right end. The next permutation must then be generated by removing n and replacing the leftmost n - 1 integers, which must be a permutation of the integers from 1 to n - 1, by their next permutation. If n is 1, then the last permutation has been produced. This provides a recursive definition for next.

next (n,pdone)

If n is not 1, then

If n is not at the right end of the current permutation of 1, . . , n then

shift it right one position

else

remove "n" from the current permutation of 1, . . , n

next(n - 1,pdone)

Insert "n" at the left end of the current permutation of 1, . . , n - 1

else

set done to true.

In order to proceed, a data structure to store the permutation must be chosen. To illustrate the effect of this selection, we will use two different structures�arrays and lists. We start by storing the permutation in an array p. Searching for the location of n in p will be avoided by keeping an array l of pointers. Both p and l are defined globally. l[n] points to the location of n in p when next(n,pdone) is invoked. Initialize must set p[i] to n - i + 1 and l[i] to 1 for 1 � i � n. Next may be defined as

next(n,pdone)

/* Generates the next permutation unless

the current one is the last. In this case

done is set to true.

*/

int n,*pdone;

{

int i;

    >Comment

if (n > 1)

if (l[n] < n)

{

    >Comment

p[l[n]] = p[l[n]+1];

p[l[n]+1] = n;

l[n] = l[n]+1;

}

    >Comment

else

{

    >Comment

next(n-1,pdone);

    >Comment

for (i=n-1;i>=1;i--)

p[i+1] = p[i];

p[1] = n;

l[n] = 1;

}

    >Comment

else

*pdone = TRUE;

}

Figure 4.6(a) simulates the execution of next(4,pdone) when the current permutation is 2 1 3 4.

Now suppose p is implemented as a list whose records are stored in dynamic memory. We need the pointer arrays ptr and pred, which are defined globally, as is the record pointer p. When next(n,pdone) is invoked, ptr[n] will contain a pointer to the record containing n in its information field, and pred[n] will contain a pointer to the predecessor of that record in p. To simplify list shifts and the insertion of a record at the front of p, a dummy record is used as the first record of p. Pred[k] will initially contain a pointer to the dummy record, and ptr[k] will initially contain a pointer to the record of p that contains k, for l � k � n. Now next may be defined as follows:

next(n,pdone)

/* Generates the next permutation unless

the current one is the last. In this case

done is set to true.

*/

int n,*pdone;

{

    >Comment

if (n > 1)

if (ptr[n]->link != NULL)

{

    >Comment

pred[n]->link = ptr[n]->link;

pred[n] = pred[n]->link;

ptr[n]->link = pred[n]->link;

pred[n]->link = ptr[n];

}

    >Comment

else

{

pred[n]->link = NULL;

    >Comment

next(n-l,pdone);

    >Comment

ptr[n]->link = p->link;

p->link = ptr[n];

pred[n] = p;

}

    >Comment

else

*pdone = TRUE;

}

Figure 4.6(b) simulates the execution of next(4,pdone) when the current permutation is 2 l 3 4.

(a) Array Implementation for P

(b) List Implementation for P

Figure 4.6 Simulation of NEXT (4,&DONE) When P is 2134

In comparing the two implementations notice that the advantage of the list implementation is that the for loop required by the array implementation is no longer needed. Inserting n at the left end of the current permutation thus takes only two pointer changes rather than time proportional to n. Other permutation algorithms are explored in Sedgewick [1977].

To further illustrate the differences, two complete programs using function permutations, one for the array and one for the list implementation, are given below. The process function prints each permutation. To adapt these programs to obtain a solution for the stable marriage problem, the function process must test for stability. Another modification would be to abort the execution when a stable pairing is found. This requires that process be given access to done so that it can set done to true at that point.

Array Implementation

#include <stdio.h>

#define TRUE 1

#define FALSE 0

    >Comment

typedef int arraytype[21];

    >Comment

arraytype p,l;

main()

/* Driver for permutations. */

{

int n;

printf("\n n = ?\n");

scanf("%d",&n);

permutations(n);

}

permutations(n)

/* Generates and processes each

permutation of the integers

1 to n.

*/

int n;

{

int done;

done = FALSE;

    >Comment

initialize(n);

while (!done)

{

    >Comment

process(n);

    >Comment

next(n,&done);

}

}

initialize(n)

/* Creates initial permutation */

int n;

{

int i;

    >Comment

for (i = 1;i<=n;i++)

{

p[i] = n - i + 1;

l[i] = 1;

}

}

process(n)

/* Prints the current permutation */

int n;

{

int i;

    >Comment

for (i=1;i<=n;i++)

printf("\n%d\n",p[i]);

printf("\n");

}

next(n,pdone)

/* Generates the next permutation unless

the current one is the last. In this case

done is set to true.

*/

int n,*pdone;

{

int i;

if (n > 1)

if (l[n] < n)

{

p[l[n]] = p[l[n]1];

p[l[n]+1] = n;

l[n] = l[n]+1;

}

else

{

next(n-1,pdone);

for (i=n-1;i>=1;i--)

p[i+1] = p[i];

p[1] = n;

l[n] = 1;

}

else

*pdone = TRUE;

}

List Implementation

#include <stdio.h>

#define TRUE l

#define FALSE 0

#define NULL 0

    >Comment

typedef struct record

{

int info;

struct record *link;

}listrecord,*listpointer;

typedef listpointer pointerarray[21];

    >Comment

pointerarray ptr,pred;

    >Comment

listpointer p;

main()

/* Driver for permutations. */

{

int n;

printf("\n n = ?\n");

scanf("%d",&n);

permutations(n);

}

permutations(n)

/* Generates and processes each

permutation of the integers 1 to n.

*/

int n;

{

int done;

done = FALSE;

    >Comment

initialize(n);

while (!done)

{

    >Comment

process(n);

    >Comment

next(n,&done);

}

}

initialize(n)

/* Creates initial permutation */

int n;

{

int i;

listpointer q;

p = malloc(sizeof(listrecord));

q = P;

    >Comment

for (i=1;i<=n;i++)

{

q->link = malloc(sizeof(listrecord));

q = q->link;

q->info = n - i + 1;

ptr[n-i+1] = q;

pred[i] = p;

}

q->link = NULL;

}

process(n)

/* Prints the current permutation */

int n;

{

listpointer q;

q = p->link;

    >Comment

while (q != NULL)

{

printf("\n%d\n",q->info);

q = q->link;

}

printf("\n");

}

next(n,pdone)

/* Generates the next permutation unless

the current one is the last. In this case

done is set to true.

*/

int n,*pdone;

{

if (n > 1)

if (ptr[n]->link != NULL)

{

pred[n]->link = ptr[n]->link;

pred[n] = pred[n]->link;

ptr[n]->link = pred[n]->link;

pred[n]->link = ptr[n];

}

else

{

pred[n]->link = NULL;

next(n-1,pdone);

ptr[n]->link = p->link;

p->link = ptr[n];

pred[n] = p;

}

else

*pdone = TRUE;

}


Data Structures Using C: 4.3: A Close Look at the Execution of Recursi

4.3: A Close Look at the Execution of Recursive Programs

When any program invokes one of its functions, it is because the task performed by that component must now be carried out. Global variables and variables passed by address (pointers) specify storage that is not local to the component but that it may reference and change. It may also reference and change static variables. All such storage represents the data the component is to process. Any changes by the component to the data are reflected back in the data's storage.

Parameters passed to a function by value also specify storage that is not local to it, but these parameters are treated differently. When the component is invoked, their values are copied into storage that is local to the component. It is these local copies that are referenced, and perhaps changed, by the component as it executes, but such changes are not reflected back to the original values. In this way the information passed by value is made accessible to the component, which may even change the local copies, but such changes are kept local; the original nonlocal values are untouched.

In effect, these copies are treated as if they were additional local variables of the component. The actual local variables of the component represent temporary information that the component uses to do its job. Both the local copies and local variables disappear when the task is completed. This storage for local copies and local variables serves as a "scratchpad" for use by the component. The scratchpad contains information used temporarily by the component to perform its task on the data. This view of the data and scratchpad of a component is depicted in Figure 4.7.

The component may, in turn, invoke another component, which may invoke still another, and so on. Each time this is done, the calling component suspends its operation, so all components of the sequence, except the last, are waiting.

When the last component completes its task, the component that invoked it then continues. In order for this to happen, each component must remember from whence it was called. Also, upon return to the correct place, the scratchpad of the calling component must be the same as before the call. For nonrecursive programs, the important point is that each component must retain only one returning point, and the scratchpad of a component is automatically retained when it invokes another component.

With recursive programs the situation is considerably different, as evidenced by Figures 4.2, 4.4, and 4.6. First, when a recursive function is invoked, this may be the initial, first, second, or nth recursive call to it. If it is the nth recursive call, there may be as many as (n + 1) returning places to be retained, not just one. Second, there may be (n + l) scratchpads to be retained. How many there are is determined by the number of calls that are currently suspended.

Figure 4.7 Data and Scratchpad for a Component

Suppose that this number is k, that the return for the initial call is handled separately, and also that the scratchpad for the currently executing call is associated with the function. The number of sets of additional returns and scratchpads that must be retained is then k. The depth of a recursive program is given by the maximum value of k. It is the depth that determines the amount of storage required by the function. The following examples illustrate these ideas.

Example 4.1

In Figure 4.2, when the tenth recursive call (k = 3) is made to towers, there are three returns and three sets of scratchpads to remember in addition to the initial call's return.

[2] and n = 4, i = l, a = 2, f = 3 for the initial call

[1] and n = 3, i = 2, a = 1, f = 3 for the eighth recursive call

[1] and n = 2, i = 2, a = 3, f = 1 for the ninth recursive call

n = 1, i = 2, a = 1, f = 3 for the tenth recursive call are stored, by our assumption, local to towers. In general, the depth of recursion will be n - l. Using our terminology, there are no data. n

Example 4.2

In Figure 4.4, when the third recursive call is made to copy, there are three returns and three sets of scratchpads to be retained beside the initial call's return:

first0, null0, rest0 for the initial call

first1, null1, rest1 for the first recursive call

first2, null2, rest2 for the second recursive call

No returns are shown since they all return to the same place: the statement setlink(*psecond, rest). The depth of the recursion is one less than the length of the list to be copied. The data for the initial call is second, while for the (n + 1)th recursive call it is restn. A change to rest in the (n + 1)th call is reflected in the scratchpad for the nth call. Storage could be saved by making null global to copy. n

Example 4.3

In Figure 4.6 the situation corresponds to a point in the initial call when the second and then the third recursive calls to next are made. The local variable i is not shown, and the three scratchpad values are

n = 4 for the initial call

n = 3 for the second recursive call

n = 2 for the third recursive call

Again, the returns are all the same, to the statement following the recursive call to next, so they are not shown. The depth of the recursion is n - 1. The data consist of p, ptr, pred, and done. n

It should by now be clear that the storage required by a recursive function is proportional to its depth. The execution time, on the other hand, has to do with the total number of recursive calls that are required to complete the initial call. This explains why the recursive solution to Counting Squares (Section 4.2.6) is so inefficient both in time and in storage.


Data Structures Using C: 4.4: Implementing Recursive Programs

4.4: Implementing Recursive Programs

Recursion is a powerful tool for the development of algorithms and leads to concise, clear recursive programs that can be easier to check for correctness. Complex algorithms may be expressed using relatively few program statements, which at the same time emphasize the structure of the algorithm. This will become even more apparent later in the text, as more difficult problems are solved.

A function that invokes itself is directly recursive. A function is indirectly recursive if any sequence of calls to other functions can lead to a call to the original function. Compilers for recursive languages must translate recursive as well as nonrecursive programs. It is not necessary to understand how a recursive program is translated (any more than it is necessary to know how a nonrecursive program is translated) in order to understand or to write the program itself. But it is necessary to know how the program will execute. This amounts to being able to simulate its execution.

The programmer constrained to a nonrecursive language can still use the power of recursion. This is done by writing a program as if the language allowed recursion, and then translating the program into an equivalent nonrecursive program. This is what compilers for recursive languages do and is one reason for spending time finding out how to translate (or implement) a recursive program.

As shown in the discussion of Counting Squares, some problems should not be solved using recursion under any circumstances. Sometimes, as in the Towers of Hanoi problem, we may only be able to think of a recursive solution. Often we can find both a recursive and a nonrecursive solution, which may be compared for trade-offs in clarity, conciseness, storage, and execution time.

Even though a recursive solution may not be best, it may suggest an approach to the problem that otherwise might have been missed and that may turn out to be the most desirable. In fact, the translated version of a recursive solution can often be helpful in finding desirable modifications. This is another reason for looking at the translation process.

Normally, if a recursive approach is right for a problem, and is applied intelligently, further refinements of the program do not improve its efficiency significantly. Breaking up a problem as required by the recursive approach does, however, create the possibility of doing certain tasks in parallel. For instance, the three components of the Towers of Hanoi solution may all be done at the same time with parallel processing. Languages and computer systems that allow concurrent processing will surely become more prevalent. When they do, recursive solutions will have another advantage.

If each successive call to a recursive function were implemented so that during execution a new copy of the function is created with its own storage for the return and its scratchpad values, then, in effect, the function could be treated like any function in a nonrecursive program. With the advent of cheaper and larger storage, this may well occur in the future, but until then, another means of managing the generated returns and scratchpad values is needed.

The essential difference between nonrecursive (or iterative) programs and recursive programs is the storage needed for the returns and scratchpad values of the latter. Figures 4.2, 4.4, and 4.6 actually represent the bookkeeping needed to simulate the execution of a recursive program. The basic idea in translating such programs is to implement this bookkeeping procedure as follows:

1. Each time a recursive call is made to the function,

a. save the proper return place and the current scratchpad values,

b. set the scratchpad values to their new values.

2. Each time a recursive call is completed,

a. restore the current return and scratchpad values to those that were last saved (and save them no longer),

b. return to the proper place in the recursive function, or to the original calling component if it is the initial call that has just finished.

This procedure does not apply to a function that returns a value, but it may be modified to do so. Or the function may be modified to return the value in an additional parameter passed by pointer. As an illustration of this, the first version of copy might be obtained by such a modification of the second.

Notice that the information to be retained (return and scratchpad values) must be recalled in the order opposite to the order in which it is generated. This is no different for nonrecursive programs, except that then the information is saved locally with each component. It is helpful to think of return and scratchpad values as the entries of a record. Then the collection of records corresponding to each suspended recursive call must be stored in some data structure. The data structure must retain not only these records, but also the correct order.

In step 1 above, the record to be saved must become the new first record in the order. In step 2, the current first record (the last saved) must be removed from the data structure. A data structure for storing information so that new information can be inserted, so that a deletion always removes the last piece of information inserted, and which can be tested to see if it is empty (contains no information) is called a stack. Stacks will be discussed in more detail later in the chapter; for now they will simply be used.

4.4.1 A Sample Implementation of Towers

4.4.2 A Sample Implementation for Permutations


Data Structures Using C: 4.4.1 A Sample Implementation of Towers

4.4.1 A Sample Implementation of Towers

The stack is the most convenient data abstraction to use for the bookkeeping recursion entails. To illustrate the translation, and the possibility of enhancing efficiency, the recursive functions towers and next will now be translated into nonrecursive functions.

In towers there are two recursive calls. The idea in implementing it is to write a nonrecursive program that executes just as the recursive program would, while managing the additional records generated by the recursion. This amounts to implementing the procedure described for simulating towers. Each recursive call will be replaced by code to insert the proper return indicator onto a stack, as well as the current scratchpad values. Whenever the end of a recursive call is reached, the proper return indicator and the scratchpad values last put on the stack will be removed and restored to the program's local storage. The earlier recursive version and its implementation as a nonrecursive function are as follows .

Recursive Version

towers(n,i,a,f)

/* Moves the top n disks

from peg i to peg f

*/

int n,i,a,f;

{

if(n == 1)

printf("\n %d -> %d\n",i,f);

else

{

>Comment

towers(n-1,i,f,a);

printf("\n %d -> %d\n",i,f)

    >Comment

towers(n-1,a,i,f);

}

}

Nonrecursive Version

towers(n,i,a,f)

/* Moves the top n disks

from peg i to peg f

*/

int n,i,a,f;

{

    >Comment

int retrn;

    >Comment

stack s;

    >Comment

setstack(&s);

    >Comment

one: if(n == 1)

printf("\n %d -> %d\n",i,f);

else

{

    >Comment

s_tack(1,n,i,a,f &s);

    >Comment

setvar1(&n,&i,&a,&f);

    >Comment

goto one;

}

    >Comment

two: if(!empty(&s))

{

    >Comment

restore(&retrn,&n,&i,&a,&f,&s);

switch(retrn)

{

    >Comment

case 1:

printf("\n %d -> %d\n",

i,f)

s_tack(2,n,i,a,f,&s);

setvar2(&n,&i,&a,&f);

goto one;

    >Comment

case 2:

goto two;

}

}

}

S_tack is a routine that inserts its arguments, as fields of a record, onto the stack. The name s_tack is used to be sure the compiler can distinguish this routine from the type name stack that also appears in the function. Setstack ensures that the stack will initially contain zero entries; empty returns true if the stack contains no records and false otherwise. The setvar routines do the proper setting of the current scratchpad values for the ensuing recursive call. Restore(&retrn,&n,&i,&a,&f,&s) restores the proper scratchpad values when a return is to be made after completion of some recursive call.

Setvar1(&n,&i,&a,&f) sets n to n - 1 and interchanges the values of f and a.

Setvar2(&n,&i,&a,&f) sets n to n - 1 and interchanges the values of a and i.

The "implicit" exit of the recursive program (after the second recursive call) has been replaced in the implementation by a complex "explicit" sequence of statements. This is because the exit must, in effect, transfer control to the proper place in the calling program. If the stack itself is empty at this point, this means that it is the initial call to towers that has been completed. Return should then be to the calling program. Otherwise, it is some recursive call to towers that has been completed. At this point, the top stack record contains the four scratchpad values to be restored and the return location in towers.

You should simulate the execution of this implementation to see that it really works correctly. The statement labeled "one" corresponds to the first executable statement of the recursive program; this is the statement that is invoked in the implementation to represent a recursive call. It is only invoked, however, after the scratchpad values of the current call and proper return location have been saved on the stack, and the scratchpad values for the upcoming recursive call have been correctly set.

Notice that the implementation requires a stack that can hold the stored information for up to n - 1 recursive calls. The stack must be initialized to empty. Its contents must be preserved between calls. We will not dwell on its implementation details here, but a complete program to test this version is given in Section 4.5.1.

This example highlights the fact that considerable overhead is involved in executing recursive programs. Time is consumed because variables must be stacked and unstacked for each recursive call and its return. It is often possible to discover a more efficient nonrecursive algorithm for a problem, even though the recursive algorithm is evident, as was done in Counting Squares. Nonetheless, the recursive version is frequently clearer, easier to understand, and more concise, as well as easier to verify. Therefore some programmers prefer recursive solutions to problems, even if they are less efficient than the nonrecursive versions.

Another approach is to attempt to increase the efficiency of the iterative program (the implementation) that is translated from a recursive program. This attempt can be made whether or not there is a competing nonrecursive routine.

How can we create a more efficient version of a translated program such as towers? The reason for the stack in the first place is to save return codes and to save scratchpad values that are needed when the suspended call is ready to resume (after the saved values have been unstacked and restored, and the proper program segment is to be executed). If the programmer can determine where the return should be, or what the scratchpad values that are to be restored should be, then this information does not have to be stacked. It may not be necessary to save the values of some other variables. It may also be possible to eliminate some return codes. If no variables need to be saved, and there is only one return code, then the stack itself is not needed. Let us attempt to reach this goal for the implementation of the recursive towers routine.

Notice that in the case statement for retrn at 1, the call to s_tack(2,n,i,a,f,&s) saves the values of the current variables in the suspended call, in order to make the recursive call to towers(n-1,a,i,f). When this new recursive call is completed, these current values will be restored, and the retrn will be 2. The statement labeled "two" will then be executed.

If the stack is empty, the initial call is complete. Otherwise restore will be called immediately and control transferred to the new value of retrn. The current values saved by s_tack(2,n,i,a,f,&s) will never be used, and so need not be saved at all. Thus the call to s_tack( 2,n,i,a,f,&s ) may be eliminated from the program. This means that the only return code appearing will be l in the call to s_tack(1,n,i,a,f,&s). However, the value of retrn after restore is called will always be 1. Therefore, no return code needs to be stored on the stack; the case structure may be removed. Retrn is not needed at all. This new program then looks like the following.

Second Nonrecursive Version

towers(n,i,a,f)

/* Moves the top n disks

from peg i to peg f

*/

int n,i,a,f;

{

stack s;

setstack(&s);

one: if(n == 1)

printf("\n %d -> %d\n",i,f);

else

{

s_tack(n,i,a,f,&s);

setvar1(&n,&i,&a,&f);

goto one;

}

if(!empty(&s))

{

    >Comment

restore(&n,&i,&a,&f,&s);

printf("\n %d -> %d\n",i,f);

setvar2(&n,&i,&a,&f);

goto one;

}

}

Another improvement can be made by changing the if-else structure to a while loop. Doing so improves the clarity and eliminates the goto:

one: if(n == 1)                    one: while(n > 1)

printf("\n %d -> %d\n",i,f);          {

else                                          s_tack(n,i,a,f,&s);

{                                         setvar1(&n,&i,&a,&f);

s_tack(n,i,a,f,&s);                 }

setvar1(&n,&i,&a,&f);            printf("\n %d -> %d\n",i,f);

goto one;

}

This yields

Third Nonrecursive Version

towers(n,i,a,f)

/* Moves the top n disks

from peg i to peg f

*/

int n,i,a,f;

{

stack s;

one: while(n > 1)

5 while loop replaces if-else construct

{

s_tack(n,i,a,f,&s);

setvar1(&n,&i,&a,&f);

}

printf("\n %d -> %d\n",i,f);

if(!empty(&s))

{

restore(&n,&i,&a,&f,&s);

printf("\n %d -> %d\n",i,f);

setvar2(&n,&i,&a,&f);

goto one;

}

}

Finally, the function can be written in more structured form, as follows:

Final Nonrecursive Version

towers(n,i,a,f)

/* Moves the top n disks

from peg i to peg f

*/

int n,i,a,f;

{

stack s;

int done

done = FALSE;

while(!done)

{

while(n > 1)

{

s_tack(n,i,a,f,&s);

setvar1(&n,&i,&a,&f);

}

printf("\n %d -> %d\n",i,f);

if(!empty(&s))

{

restore(&n,&i,&a,&f,&s);

printf("\n %d -> %d\n",i,f);

setvar2(&n,&i,&a,&f);

}

else

done = TRUE;

}

This structured version is more efficient than the compilerlike direct translation because the number of stack operations has been reduced. It is also structured for clarity in reading. Both changes make it a better implementation.


Data Structures Using C: 4.4.2 A Sample Implementation for Permutation

4.4.2 A Sample Implementation for Permutations

The same approach can now be applied to develop a nonrecursive translation for next.

Nonrecursive Version

next(n,pdone)

/* Generates the next permutation unless

the current one is the last. In this case

done is set to true.

*/

int n,*pdone;

{

>Comment

int i,retrn;

    >Comment

stack s;

    >Comment

one: if(n > 1)

if(l[n] < n)

{

p[l[n]] = p[l[n] + 1];

p[l[n] + 1] = n;

l[n] = l[n] + 1;

}

else

{

    >Comment

s_tack(1,n,i,&s);

    >Comment

setvar(&n,&i);

    >Comment

goto one;

}

else

*pdone = TRUE;

    >Comment

two: if(!empty(&s))

{

    >Comment

restore(&retrn,&n,&i,&s);

switch(retrn)

{

    >Comment

case 1:

"explicit" exit must be made

{

for(i=n-1;i>=1;i--)

p[i+1] = p[i];

p[1] = n;

l[n] = 1;

    >Comment

goto two;

{

{

{

{

where setvar(&n,&i) simply sets n to n-1.

The same program can be written in a more structured way. First notice that the statement labeled "two" and the goto "two" statement create a while loop. Introducing that while loop but retaining the structure of "one," the overall structure of the program becomes as follows

    >Comment

one: if c1 then

    >Comment

if c2 then

task c1 and c2 both true

else

task for c1 true but c2 false

goto one

else

task for c1 false

    >Comment

while not empty(&s)

loop task

Next, the structure of "one" can also be improved, as follows.

    >Comment

while (c1 and not c2)

task for c1 true but c2 false

    >Comment

if c1, then

task for c1 and c2 both true

else

task for c1 false

    >Comment

while not empty (&s)

loop task.

The logic of this version is much easier to follow. You can see at a glance that the segment control flows from the top down. It consists of a loop, followed by a decision, followed by a loop. The iterative version of next, translated from the recursive next, and using the improved structure, now becomes as follows:

Second Nonrecursive Version

    >Comment

next(n,pdone)

/* Generates the next permutation unless

the current one is the last. In this case

done is set to true.

*/

int n,*pdone

{

int i,retrn;

stack s;

while((n > 1) && !(l[n] < n))

{

s_tack(1,n,i,&s);

setvar(&n,&i);

}

if(n > 1)

{

p[l[n]] = p[l[n] + 1];

p[l[n] + 1] = n;

l[n] = l[n] + 1;

}

else

*pdone = TRUE;

while(!empty(&s))

{

restore(&retrn,&n,&i,&s);

switch(retrn)

{

case 1:

{

for(i=n-1;i>=1;i--)

p[i+1] = p[i];

p[1] = n;

l[n] = 1;

}

}

}

}

In general, when the value of a scratchpad variable does not change or does not need to be maintained between recursive calls, it need not be stacked. This is the case with the variable i of next. Also, when there is only one return place from a recursive call, there is no need to stack a return indicator. This is the situation with next. Finally, the value of n varies in a predictable fashion. Its restored value will always be 1 greater than its current value. Thus there is no need to stack n. Each time a recursive call returns, n may be correctly restored by adding 1 to its current value.

Therefore, the stack is unnecessary, since no information must be retained on it. The s_tack function can be removed, setvar can be replaced by the statement n = n-1, and restore can be replaced by the statement n = n+1. Still, the stack controls the second while loop in our translated version. Even though no information needs to be stacked, it is still necessary to know when the stack would have been empty so that the loop may be properly exited.

Notice that the first while loop decreases n by l every time a stack entry would have been made, and the second while increases n by l every time an entry would have been deleted from the stack. Hence, if an integer variable s is set to n initially, prior to the first while loop, the test to see if the stack is empty in the second while loop becomes s = n. The final version of next is an iterative program that has no stack. It is an example of the translated version giving us a better solution. This four-step solution is efficient and is also clear.

1. Set s to n.

2. Set n to the rightmost entry in the permutation that can still be shifted right.

3. If that entry is not l, then

Shift it right

else

Set done to true

4. Place the s-n entries that could not be moved to the right in the proper order at the left end of the permutation.

The final program is as follows:

Final Nonrecursive Version

next(n,pdone)

/* Generates the next permutation unless

the current one is the last. In this case

done is set to true.

*/

int n,*pdone

{

int i,s;

1. s = n;

2. while((n > 1) && !(l[n] < n))

n--;

3. if(n > 1)

{

p[l[n]] = p[l[n] + 1];

p[l[n] + 1] = n;

l[n] = l[n] + 1;

}

else

*pdone = TRUE;

4. while(s > n)

{

n++

for(i=n-1;i>=1;i--)

p[i+1] = p[i];

p[1] = n;

l[n] = 1;

}

}

By translating the recursive solution and analyzing it, a highly efficient nonrecursive solution was built. A little insight can help a lot. For more on recursion, see Barron [1968] and Bird [1977a, 1977b]. It may now be time for you to make a recursive call on all preceding material!


Data Structures Using C: 4.5: Stacks

4.5: Stacks

Earlier in this chapter the notion of a stack was introduced and used in the nonrecursive implementations of towers and next. Stacks are useful in situations requiring the retention and recall of information in a specific order梟amely, when information is removed in a last-in, first-out order (LIFO for short). Removing tennis balls from a can is LIFO. In programming, stacks are useful when a program must postpone obligations, which must later be fulfilled in reverse order from that in which they were incurred.

Information is kept in order in a stack. Additional information may be inserted on the stack, but only at the front or top end. Thus the information at the top of the stack always corresponds to the most recently incurred obligation. Information may be recalled from the stack, but only by removing the current top piece of information. The stack acts like a pile of playing cards to which we can only add a card at the top and from which we can only remove the top card. Of course, this is not what is meant by a "stacked deck."

The stack, with the operations of front-end insertion and deletion, initialization, and an empty check, is a data abstraction. Stacks used to be called "pushdown stacks"; insertion of data onto the stack was referred to as "pushing" the stack, and deleting data was known as "popping" the stack. So far we have written programs that treat the stack with the four operations as a data abstraction. Stacks will be used frequently in algorithms developed throughout the rest of this book, and it is now time to consider this implementation. Of the many implementations possible for the stack, the following subsections illustrate the three most straightforward.

4.5.1 Array Implementation of a Stack

4.5.2 List Implementation of the Stack with an Array of Records

4.5.3 List Implementation of the Stack with Records in Dynamic Memory


Data Structures Using C: 4.5.1 Array Implementation of a Stack

4.5.1 Array Implementation of a Stack

A stack may be implemented using arrays with the following declarations:

>Comment

typedef struct

{

whatever info;

}stackrecord;

>Comment

typedef struct

{

stackrecord stackarray [LIMIT];

int top;

}stack;

>Comment

stack s;

The definitions must be preceded by a typedef for "whatever." Let us assume LIMIT has been set in a define statement fixing the maximum number of items the stack can hold. Suppose 6, 45, 15, 32, and 18 were pushed onto the stack. Figure 4.8 illustrates the meaning attached to the declaration. Figure 4.8(a) is a conceptualization of the stack. Since 6 was the first entry on the stack, it is on the bottom. Likewise, 18 was the last entry, so it is on top. Figure 4.8(b) is the way the data is stored in the actual stack s. The stackarray holds the data and top contains an index to the current entry on the top of the stack. Although 6 is at the top of the array, it is the bottom stack entry; while 18 is at the bottom of the array, it is the top stack entry. Thus top has the value 4.

Addition or insertion of a new element to the stack will be accomplished by a function push. Removal or deletion of the current top stack element will be accomplished by a function pop. Executing push (&newrecord,&s) results in the contents of newrecord being placed on the stack as a new first element. A call to the function pop(&s,&value) causes the removal of the current top stack element, with value containing a copy of that value. A routine setstack sets the stack to empty so that it initially contains no entries.

Example 4.4

Suppose we execute

pop(&s,&y);

pop(&s,&z);

>Comment

push(&newrecord,&s);

If Figure 4.8 indicates the current stack, then Figure 4.9 shows the new situation after this execution. n

(a) The Stack Data Abstraction

(b) Array Implementation of the Stack

Figure 4.8 Stack before Insertion of a New Element

(a) The Stack Data Abstraction

(b) Array Implementation of the Stack

Figure 4.9 Stack after Insertion of a New Element

Stack underflow occurs when a program attempts to remove an element from an empty stack. It normally indicates either an error or the termination of a task. Stack overflow occurs when a program attempts to add an element to a full stack. Although, conceptually, the stack as a data structure has no limit, an actual implemented stack does have a limit. In the present implementation it is termed. LIMIT. Push and pop should test, respectively, for stack overflow and underflow and take appropriate action whenever these conditions occur. Such precautionary measures are an example of defensive programming.

It is also necessary to test the stack to determine whether or not it is empty. This may be accomplished by invoking a function empty, which will return the value true when the stack is empty and false otherwise.

These routines may be implemented as follows:

setstack (ps)

/* Initializes the stack s to empty */

stack *ps;

{

>Comment

(*ps).top = -1;

}

empty (ps)

/* Returns true only if the stack is empty */

stack *ps;

{

    >Comment

return ((*ps).top == -1);

}

push (pnewrecord,ps)

/* Inserts the record newrecord

at the top of the stack s

*/

stackrecord *pnewrecord,

stack *ps;

{

    >Comment

if ((*ps).top == (LIMIT - 1))

overflow(ps);

else

{

    >Comment

(*ps).top = (*ps).top + 1;

(*ps).stackarray[(*ps).top].info =

(*pnewrecord).info;

}

}

pop(ps,pvalue)

/* Copies the contents of the top record

of stack s into value and removes the

record from the stack

*/

stack *ps;

stackrecord *pvalue;

{

    >Comment

if(empty(ps))

underflow(ps);

else

{

    >Comment

(*pvalue).info = (*ps).stackarray

[(*ps).top].info;

(*ps).top = (*ps).top - 1;

}

}

The procedures overflow and underflow are assumed to take appropriate action when they are invoked. Notice that the elements of the stack will be of "whatever" type. We have taken the liberty here of assuming that the C compiler being used allows the contents of one structure to be assigned to another. Otherwise, for example, the assignment

(*ps).stackarray[(*ps).top].info =

(*pnewrecord).info;

that occurs in push could not be used. Instead, each field of info would have to be individually assigned.

The implementation discussed so far applies when records are to be stored on the stack. Instead, if only the contents of simple variables of type such as int, float, char, or listpointer are to be stored, then the declarations and definitions of push and pop may be simplified, as follows.

    >Comment

typedef struct

{

whatever stackarray [LIMIT];

int top;

}stack;

push(value,ps)

/* Inserts the new value

at the top of the stack s

*/

whatever value;

stack *ps;

{

if((*ps).top == (LIMIT-1))

overflow(ps);

else

{

(*ps).top = (*ps).top + 1;

    >Comment

(*ps).stackarray[(*ps).top] = value;

}

}

pop(ps,pvalue)

/* Copies the contents of the top entry

of stack s into value and removes it

from the stack

*/

stack *ps;

whatever *pvalue;

{

if(empty(ps))

underflow(ps);

else

{

    >Comment

*pvalue = (*ps).stackarray[(*ps).top];

(*ps).top = (*ps).top - 1;

}

}

"Whatever" will, of course, be declared of type int, or float, etc. The same simplification can be made, when appropriate, to the implementations of Sections 4.5.2, 4.5.3, 4.7.1, 4.7.2, and 4.7.3.

The idea of functional modularity is to isolate or localize the code of a program that is invoked in a particular task, such as an operation on a data structure. The implementation of the stack data abstraction discussed so far has achieved this modularity, with respect to the push, pop, setstack, and empty operations on the stack, by embodying them in individual routines. Another kind of modularity may be achieved by placing the type definitions and the declarations of these operations together in one file. This will be discussed further in the next chapter.

An application of the stack to the nonrecursive towers of Section 4.4.1 follows. It illustrates the use of the stack and its treatment as a data abstraction.

#include <stdio.h>

    >Comment

#define LIMIT 50

typedef struct

{

int n;

int i;

int a;

int f;

}whatever;

typedef struct

{

whatever info;

}stackrecord;

typedef struct

{

stackrecord stackarray [LIMIT];

int top;

}stack;

setstack(ps)

/* Initializes the stack s to empty */

stack *ps;

{

(*ps).top = -1;

}

empty(ps)

/* Returns true only if the stack s is empty */

stack *ps;

{

return((*ps).top == -1);

}

push(pnewrecord,ps)

/* Inserts the record newrecord

at the top of the stack s

*/

stackrecord *pnewrecord;

stack *ps;

{

if ((*ps).top == (LIMIT-1))

overflow(ps);

else

{

(*ps).top = (*ps).top + 1;

(*ps).stackarray[(*ps).top].info =

(*pnewrecord).info;

}

}

pop(ps,pvalue)

/* Copies the contents of the top record

of stack s into value and removes the

record from the stack

*/

stack *ps;

stackrecord *pvalue;

{

if (empty(ps))

underflow(ps);

else

{

(*pvalue).info =

(*ps).stackarray[(*ps).top].info;

(*ps).top = (*ps).top - 1;

}

}

overflow(ps);

/* Prints a message that the stack has overflowed */

stack *ps;

{

printf(" The stack has overflowed \n");

}

underflow(ps);

/* Prints a message that the stack has underflowed */

stack *ps;

{

printf(" The stack has underflowed \n");

}

main()

/* Driver for towers */

{

int n;

printf("\n enter a value for n\n");

scanf("%d",&n);

towers(n,1,2,3);

}

    >Comment

towers(n,i,a,f)

/* Moves the top n disks

from peg i to peg f

*/

int n,i,a,f;

{

    >Comment

stack s;

int done;

setstack(&s);

done = FALSE;

while(!done)

{

while (n > 1)

{

s_tack(n,i,a,f,&s);

setvar1(&n,&i,&a,&f);

}

printf("\n %d -> %d\n",i,f);

if(!empty(&s))

{

restore(&n,&i,&a,&f,&s);

printf("\n %d -> %d\n",i,f);

setvar2(&n,&i,&a,&f);

}

else

done = TRUE;

}

setvar1(pn,pi,pa,pf)

/* Sets n to n-1 and

interchanges f and a

*/

int *pn,*pi,*pa,*pf;

{

int t;

*pn = *pn-1;

t = *pf;

*pf = *pa;

*pa = t;

}

setvar2(pn,pi,pa,pf)

/* Sets n to n-1 and interchanges a and i

*/

int *pn,*pi,*pa,*pf;

{

int t;

*pn = *pn-1;

t = *pa;

*pa = *pi;

*pi = t;

}

    >Comment

s_tack(n,i,a,f,ps)

/* Creates a record containing

n,i,a, and f and inserts it

on stack s

*/

int n,i,a,f;

stack *ps;

{

    >Comment

stackrecord newrecord;

newrecord.info.n = n;

newrecord.info.i = i;

newrecord.info.a = a;

newrecord.info.f= f;

push(&newrecord,ps);

}

    >Comment

restore(pn,pi,pa,pf,ps)

/* Removes the top record from

stack s and copies its contents

into n,i,a,f

*/

int *pn,*pi,*pa,*pf;

stack *ps;

{

stackrecord value;

pop(ps,&value);

*pn = value.info.n;

*pi = value.info.i;

*pa = value.info.a;

*pf = value.info.f;

}

In review of the program, notice that it treates the stack as a data abstraction. The type definitions and function definitions for the stack are placed at the top of the program. The functions setvar, s_tack, and restore are not basic stack operations; they are used by towers. Thus they have been placed after towers. Similarly, towers has been placed after main. Also, s_tack and restore are dependent on the stack implementation, since they must know the structure of a stack record, which is, in turn, determined by the needs of towers for its scratchpad.


Data Structures Using C: 4.5.2 List Implementation of the Stack with a

4.5.2 List Implementation of the Stack with an Array of Records

In addition to the array implementation, a stack can be implemented as a list. Since we have discussed two ways to implement lists, we know two ways to implement stacks as lists. One is to store the stack records in an array; the other is to store them in dynamic memory. We have already given an implementation of a stack using an array of records in Chapter 3梬here availist (Section 3.7) actually acts as a stack. When avail is invoked, it removes the top record from the availist, in addition to returning a pointer to the record. Thus it has effectively popped the record. Likewise, reclaim may return a record to the front of the availist. Doing so, reclaim is essentially pushing a new record onto the availist. Availist serves as the head of a list, which itself serves as a list implementation of a stack. Of course, appropriate overflow and underflow checking must be built into avail and reclaim, respectively.


Data Structures Using C: 4.5.3 List Implementation of the Stack with R

4.5.3 List Implementation of the Stack with Records in Dynamic Memory

The third stack implementation is also as a list梠ne that involves dynamic storage allocation of the records. That is, the records are stored in the storage area set aside by C to store records referenced by pointer variables. First, the following declaration must be made.

>Comment

typedef struct record

{

struct whatever info;

struct record *link;

}stackrecord,*stack;

If we then define s as type stack, code can be written that is independent of its implementation. Setstack must initialize the stack to empty. This means setting s, the head of the list implementing the stack, to null. Push must now use malloc to allocate storage for the entry to be pushed onto the stack before setting its info field and inserting that record as the new first record on the stack. Pop must copy the info field value of the top record into value and then delete that record from the stack.

These stack operations are implemented as follows.

setstack(ps)

/* Initializes the stack s to empty */

stack *ps;

{

>Comment

*ps = NULL;

}

empty(ps)

/* Returns true only if the stack s is empty */

stack *ps;

{

    >Comment

return(*ps == NULL);

}

push(pnewrecord,ps)

/* Inserts the record newrecord

at the top of the stack s

*/

stackrecord *pnewrecord;

stack *ps;

{

stackrecord *newentry;

    >Comment

newentry = malloc(sizeof(stackrecord));

newentry->info = pnewrecord->info;

newentry->link = *ps;

*ps = newentry;

}

pop(ps,pvalue)

/* Copies the contents of the top record

of stack s into value and removes the

record from the stack

*/

stack *ps;

stackrecord *pvalue;

{

    >Comment

if(!empty(ps))

{

    >Comment

pvalue->info = (*ps)->info;

*ps = (*ps)->link;

}

else

underflow(ps);

}

Notice that this implementation removes the artificial restriction imposed by LIMIT in the first implementation using an array. This is one of the advantages of using dynamic allocation by means of pointer variables.

It is very important to see that no matter what the implementation, as long as the stack is treated as a data abstraction when writing programs, the programs need not be changed, even if the programmer decides to change the implementation of the stack. Only the declarations and definitions related to the stack need be changed, as is evident in the following nonrecursive towers that uses the list implementation. It should be compared to the program of Section 4.5.1, in which the stack was implemented using an array.

    >Comment

#include <stdio.h>

#define NULL 0

typedef struct

{

int n;

int i;

int a;

int f;

}whatever;

typedef struct record

{

whatever info;

struct record *link;

}stackrecord, *stack;

setstack(ps)

/* Initializes the stack s to empty */

stack *ps;

{

*ps = NULL;

}

empty(ps)

stack *ps;

{

return(*ps == NULL);

}

push(pnewrecord,ps)

stackrecord *pnewrecord;

stack *ps;

{

stackrecord *newentry;

newentry = malloc(sizeof(stackrecord));

newentry->info = pnewrecord->info;

newentry->link = *ps;

*ps = newentry;

}

pop(ps,pvalue)

/* Copies the contents of the top record

of stack s into value and removes the

record from the stack

*/

stack *ps;

stackrecord *pvalue;

{

if(!empty(ps))

{

pvalue->info = (*ps)->info;

*ps = (*ps)->link;

}

else

underflow(ps);

}

overflow(ps);

/* Prints a message that the stack has overflowed */

stack *ps;

{

printf(" The stack has overflowed \n");

}

underflow(ps);

/* Prints a message that the stack has underflowed */

stack *ps;

{

printf(" The stack has underflowed \n");

}

    >Comment

main()

/* Driver for towers */

{

int n;

printf("\n enter a value for n \n");

scanf("%d",&n);

towers(n,1,2,3);

}

towers(n,i,a,f)

1 final nonrecursive version

/* Moves the top n disks

from peg i to peg f

*/

int n,i,a,f;

{

stack s;

1 allocates storage for s (an instance of type stack)

int done

setstack(&s);

done = FALSE;

while(!done)

{

while (n > 1)

{

s_tack(n,i,a,f,&s);

setvar 1(&n,&i,&a,&f);

}

printf("\n %d -> %d\n",i,f);

if(!empty(&s))

{

restore(&n,&i,&a,&f,&s);

printf("\n %d -> %d\n",i,f);

setvar2(&n,&i,&a,&f);

}

else

done = TRUE;

}

setvar1(pn,pi,pa,pf)

/* Sets n to n - 1 and

interchanges f and a

*/

int *pn,*pi,*pa,*pf;

{

int t;

*pn = *pn - 1;

t = *pf;

*pf = *pa;

*pa = t;

}

setvar2(pn,pi,pa,pf)

/* Sets n to n - 1 and

interchanges a and i

*/

int *pn,*pi,*pa,*pf;

{

int t;

*pn = *pn - 1;

t = *pa;

*pa = *pi;

*pi = t;

}

    >Comment

s_tack(n,i,a,f,ps)

/* Creates a record containing

n, i, a, and f and inserts it

on stack s

*/

int n,i,a,f;

stack *ps;

{

    >Comment

stackrecord newrecord;

newrecord.info.n = n;

newrecord.info.i = i;

newrecord.info.a = a;

newrecord.info.f = f;

push(&newrecord,ps);

}

    >Comment

restore(pn,pi,pa,pf,ps)

/* Removes the top record from

stack s and copies its contents

into n,i,a,f

*/

int *pn,*pi,*pa,*pf;

stack *ps;

{

stackrecord value;

pop(ps,&value);

*pn = value.info.n;

*pi = value.info.i;

*pa = value.info.a

*pf = value.info.f;

}

The problem of implementing many stacks simultaneously is introduced in the suggested assignment at the end of this chapter. More sophisticated algorithms for the management of the stacks are explored in Garwick [1964], Knuth [1973], and Korsh [1983].


Data Structures Using C: 4.6: Evaluation and Translation of Expression

4.6: Evaluation and Translation of Expressions

To illustrate the use of stacks, let us consider the problem of evaluating arithmetic expressions. People are accustomed to writing arithmetic expressions in infix notation, where an operator appears between its operands. Consequently, programming languages generally use infix notation to be "user friendly." Thus the programmer can write

(((a + (b/c * d)) + (a/w)) * b)/(((a * c) + d) + a)

Table 4.1 Notations for Two

Arithmetic Expressions

          a + (b * c)   (a + b) * c

Prefix    + a * bc      * + abc

Infix     a + b * c     a + b * c

Postfix   abc * +       ab + c *

Even humans must scan such an expression carefully, more than once, before discovering that its basis syntactic structure is (operand 1)/(operand 2). A serious problem for the compiler writers was how to translate such expressions without making many scans back and forth through them. The problem was compounded further because programmers were not required to write fully parenthesized expressions; this created the additional dilemma of how to resolve ambiguity. As will be shown below, ambiguity may be resolved by using a notation other than infix.

Consider the arithmetic expressions of Table 4.1. These arithmetic expressions are written in prefix, infix, and postfix notation. Prefix means that the operator appears to the left, before its operands, and postfix means that the operator appears to the right, after its operands. Although infix notation is what programmers normally use, there are contexts where prefix (or postfix) notation is used. For instance we write sqrt (x), f(x,y), and minimum (x,y)

Note that the prefix and postfix expressions of the first column are equivalent ways to express the infix expression a + (b * c), while the prefix and postfix expressions of the second column are equivalent to (a + b) * c. Unfortunately, the infix expressions of both columns are identical. This means that, unless parentheses are used, there is no way to distinguish between a + (b * c) and (a + b) * c using infix notation. Yet we can choose the proper prefix or postfix notation to make clear which of the two is intended, even though no parentheses (or special conventions) are used. This holds true in general for arithmetic expressions. For this reason, prefix and postfix notations are called parentheses-free notations. Compilers can easily process prefix and postfix expressions. Since postfix expressions are generally used, they are the main focus of the discussion that follows.

4.6.1 Postfix Expressions

4.6.2 Infix Expressions

4.6.3 Translating Infix to Postfix


Data Structures Using C: 4.6.1 Postfix Expressions

4.6.1 Postfix Expressions

Given an infix expression that is completely parenthesized, such as

((a + b) * (c/d))

you know how to evaluate it. This is because you have grown accustomed to the rules for its evaluation, although you might be hard pressed to state those rules completely and precisely. In any case, parentheses signal how the operators are to be applied to the operands, indicating that (a + b) and (c/d) must be evaluated first before applying the * operator. Removing all parentheses yields

a + b * c/d

Unless conventions or rules are given, this expression is ambiguous. Its five distinct interpretations correspond to the ways in which parentheses may be placed:

((a + b) * (c/d))

(((a + b) * c)/d)

((a + (b * c))/d)

(a + ((b * c)/d))

(a + (b * (c/d)))

A postfix expression such as

ab + cd/*

makes little sense, for you are not used to it, having little experience in interpreting or evaluating it. The rule to be used is as follows:

1. Scan the postfix expression from left to right.

2. Whenever an operator is encountered,

replace it, and the immediately preceding number of operands required by that operator, with the result of applying that operator to those operands.

3. Treat this replacement as an operand, and continue scanning from this point.

Thus the postfix expression is equivalent to the infix expression

((a + b) * (c/d))

The rule, in generating this expression, produces the two intermediate expressions: (a + b)cd/*, (a + b)(c/d)*.

The following list shows the five fully parenthesized infix expressions representing the interpretations of a + b * c/d and their corresponding postfix expressions.

((a + b) * (c/d)) .... ab + cd/*

(((a + b) * c)/d) .... ab + c * d/

((a + (b * c))/d) .... abc* + d/

(a + ((b * c)/d)) .... abc*d/+

(a + (b * (c/d))) .... abcd/*+

These examples emphasize two points:

1. The rule for evaluating postfix expressions allows no ambiguity. It determines precisely the meaning or interpretation of the postfix expression.

2. An infix expression, unless fully parenthesized, may be ambiguous in its meaning or interpretation.

It is because of statement 1 that postfix notation, or postfix expressions, are called parentheses-free. A similar rule allows prefix notation, or prefix expression, to be parentheses-free .

Furthermore, evaluation of a postfix expression is easily accomplished by scanning the expression from left to right and using a stack to retain operands and

Table 4.2 Evaluation of a Postfix Expression

Input      Operand Stack

  a        a

  b        b

           a

  c        c

           b

           a

  *        (b * c)

           a

  d        d

           (b * c)

           a

  /        ((b * c)/d)

           a

  +        (a + ((b * c)/d)

results of intermediate evaluations. Any operand that is encountered is placed on an operand stack. When an operator is encountered, its operands are taken from the top of the stack, the operator is applied, and the result is put on the top of the stack. The process is illustrated in the following example.

Example 4.5

Table 4.2 shows the contents of the operand stack after each input is processed, simulating the procedure for evaluation of a postfix expression. The expression is ABC*D/+. n

Since the procedure for the evaluation of an arithmetic expression using a stack is relatively straightforward and requires no parentheses, why don't programmers write arithmetic expressions in postfix notation only? It is because they are so used to infix notation that they don't want to change. As a result, high-level languages accept infix expressions rather than postfix. This leads back to the question of how infix expressions may be evaluated.


Data Structures Using C: 4.6.2 Infix Expressions

4.6.2 Infix Expressions

The rule to be used for a completely parenthesized infix expression is

1. Scan the infix expression from left to right.

2. When a "matched" pair of ''left" and "right" parentheses is encountered,

apply the operator to the operands enclosed within the pair of parentheses, and replace the paired parentheses and the contained operator and operands by the result. Continue scanning from this point.

One problem remains: How do we evaluate infix expressions that are not completely parenthesized? Recall that there are standard rules for interpreting infix expressions in this case. These rules are also incorporated in high-level languages. They are necessary because parentheses are not always available to signal what is to be done next. For example, if we evaluate

a + b * c/d

then when the * is encountered, the stacks appear as

Operand Stack     Operator Stack

--------------------------------

      b                 +

      a

The operator * signals that, had the expressions been completely parenthesized, a left parenthesis would have occurred just before the b, or else a right parenthesis would have occurred just after the b. In other words, the appearance of the operator requires a decision as to whether b should be considered an operand of *or an operand of +. The standard rules resolve such ambiguities by assigning priorities or precedence values to operators, assuming a left-to-right scan of the expression. For the binary operators, +, -, *, /, and ** (for exponentiation), the priorities are as follows:

1. Exponentiation (**) has the highest priority.

2. Multiplication and division (*, /) have the same priority, which is higher than that of addition or subtraction (+, -).

3. Addition and subtraction (+, -) have the same priority.

A left parenthesis also needs a priority, the lowest.

Whether b is an operand of * or + can then be resolved by taking b as an operand of the highest priority operator (* in this case). Again, parentheses may be used to override priorities. For example, in

(a + b) * c/d

b is to be associated with the + operator.

Example 4.6

In order to see how any infix expression is evaluated using priorities, and to develop an algorithm for this evaluation, consider the expression,

a + (b * c/d ** e) *f n

To evaluate it, when the first * is encountered the situation will be

Operand Stack      Operator Stack

--------------------------------

     b                   (

     a                   +

In this case, since the left parenthesis is on top of the operator stack, the * should be placed on top of the operator stack, and scanning continued. After c, division(/) is encountered and the stacks become

Operand Stack      Operator Stack

---------------------------------

      c                  *

      b                  (

      a                  +

At this point a decision must be made with respect to c: either it is to be associated with * (at the top of the stack) or with / (the current operator being processed). The standard rules associate c with the operator of highest priority. In this case, since * and / have equal priorities, associate c with the leftmost of the two operators in the input�that is, the top stack operator. Thus, the * must be popped, c and b must be popped, the * must be applied to b and c, and the result, (b * c), placed on top of the operand stack. Then the / must be placed on the operator stack. Had the current operator been of higher priority (say, **), then the current operator should simply be placed on the operator stack.

In the example, after / and d are processed, the stacks are as follows:

Operand Stack      Operator Stack

---------------------------------

      d                  /

   (b * c)               (

      a                  +

The ** becomes the current operator, and d must be associated with it or with /. Since ** has higher priority, place it on the operator stack; the result after processing e is

Operand Stack      Operator Stack

---------------------------------

      e                  **

      d                  /

   (b * c)               (

      a                  +

The current input character being scanned is now a right parenthesis. This "matches" the topmost left parenthesis on the stack. The next steps are to pop the operator stack; pop the operand stack to obtain the top two operands; apply **to them, and place the result back on the top of the operand stack. Then pop the operator stack; pop the operand stack to obtain the top two operands; apply / to them and place the result back on top of the operand stack; pop the stack again; and notice that a left parenthesis has appeared, signaling completion of the process for the current input character. The result is

 Operand Stack       Operator Stack

----------------------------------

((b * c)/(d ** e))         +

        a

Finally, the last * is processed. It has higher priority than +, so it is placed on the operator stack. f is then processed and placed on the operand stack, yielding

 Operand Stack        Operator Stack

-----------------------------------

       f                    *

((b * c)/(d ** e))          +

       a

Since the input is exhausted, the final steps are, for each remaining operator on the operator stack, to pop the operator stack, remove the required number of top operands from the operand stack, apply the operator, and place the result back on top of the operand stack. The final result appears at the top of the operand stack.

The algorithm for the evaluation of any infix expression can now be written. The algorithm, which follows, assumes that a character, when read, will be an entire operand or operator. This means, for example, that ** should be interpreted as a "character." The algorithm returns the value of the expression in evaluate.

1. Initialize the operand stack to empty.

2. Initialize the operator stack to empty.

3. While there is another input character,

read the current character

if the current character is "(", then

push it on the operator stack;

else if the current character is an operand, then

push it on the operand stack;

else if the current character is an operator, then

if the operator stack is not empty, then

if the current character does not have greater priority than the top

operator stack character, then

pop the operator stack, remove the required number of operands

from the operand stack, apply the operator to them, and push the

result on the operand stack

push the current character onto the operator stack

else if the current character is ")", then

pop the operator stack and set topchar to the popped value

while topchar is not (

remove the number of operands required by topchar from the operand

stack, apply topchar to them, and

push the result onto the operand stack

pop the operator stack and set topchar to the popped value

4.While the operator stack is not empty,

pop the operator stack, remove the required number of operands from the

operand stack, apply the operator to them, and push the result on the operand

stack

5.Pop the operand stack and set evaluate to the popped value

Notice that the evaluation of an infix expression requires two stacks rather than the one stack required for the evaluation of a postfix expression. Other operators may be introduced and given appropriate priorities. For example, the assignment operator, =, would be given a higher priority than **.


Data Structures Using C: 4.6.3 Translating Infix to Postfix

4.6.3 Translating Infix to Postfix

Suppose, rather than evaluating an infix expression, a programmer wished to translate it and produce the corresponding postfix expression as output. The reason is that postfix expressions can be executed directly, whereas infix expressions are tricky to evaluate. Moreover, if the translation can be done, then the postfix expression may easily be embellished to produce a machine language program for the evaluation of the original infix expression.

The following modification of the infix evaluation algorithm results in such a translation.

1. Eliminate the operand stack. Instead, any pushing onto the operand stack should be changed to outputting the same value.

2. Instead of applying operators to operands, simply output the operators.


Data Structures Using C: 4.7: Queues

4.7: Queues

The queue, like the stack, is a useful data abstraction for retaining information in a specific order. A queue is a data structure with restricted insertion and deletion operations. Information may be inserted on the queue, but only at the rear end, or bottom. Information may be removed from the queue, but only at the front end, or top.

The queue acts like the pile of Community Chest cards in Monopoly. The top card is the only one that can be removed, and insertions to the pile can be made only at the bottom. If we think of the information stored as representing postponed obligations, then the queue gives access to the obligations in the order in which they were incurred. A popular mnemonic for helping to remember this characteristic is FIFO�first in, first out. Like LIFO, this is a common term in accounting and inventory control systems.

4.7.1 Array and Circular Array Implementation of the Queue

4.7.2 List Implementation of the Queue with an Array of Records

4.7.3 List Implementation of the Queue with Records in Dynamic Memory


Data Structures Using C: 4.7.1 Array and Circular Array Implementation

4.7.1 Array and Circular Array Implementation of the Queue

Queues can be implemented in the same basic ways as stacks�as an array of records, as a list with its records stored in an array, and as a list with its records stored in dynamic memory. As in the case of stacks, these implementations are discussed in that order.

A queue may be implemented using an array of records by first declaring

>Comment

typedef struct

{

whatever info;

}queuerecord;

>Comment

typedef struct

{

queuerecord queuearray [LIMIT];

int front, rear;

}queue;

>Comment

queue q;

As with stacks, overflow and underflow should be checked in a general implementation. A typical queue situation is depicted in Figure 4.10(a) for the array implementation. Figure 4.10(b) shows the situation after 18, 32, and 15 have been removed and 17 and 20 have been inserted.

Storage for insertion of more records is available at the top of the queuearray, but not at the bottom. To ensure that all the relocated storage for queue is used before overflow occurs, programs "wrap around." That is, the queue is implemented so that the queuearray is viewed as a circle of records, with 0 following the LIMIT, 7. Front is used to point to the entry at the front of the queue, and rear is used to point to the array location in which the next

(a) Initial Configuration

(b) After Three Records Are Removed and Two Records Are Inserted

Figure 4.10 Typical Queue Configuration for an Array Implementation

inserted record is to be placed. As entries are inserted, rear moves down until it "falls off" the array, then it must be "wrapped around." As entries are deleted, front moves down until it "falls off" the array, then it must "wrap around."

The insertion operation may then be implemented as follows:

insert(pnewrecord,pq)

/* Inserts newrecord at the rear of queue q */

queuerecord *pnewrecord;

queue *pq;

{

>Comment

if ((*pq).front == -1)

{

    >Comment

(*pq).front = 0;

(*pq).queuearray[0].info = (*pnewrecord).info;

(*pq).rear = 1;

}

else

    >Comment

if((*pq).rear ==  (*pq).front)

printf("\n overflow \n");

else

{

    >Comment

if((*pq).rear <= (LIMIT - 1))

{

    >Comment

(*pq).queuearray[(*pq).rear].info =

(*pnewrecord).info;

(*pq).rear = (*pq).rear + 1;

}

else

    >Comment

{

if((*pq).front == 0)

printf("\n overflow \n");

else

{

4 there is room to wrap around, so the new entry is inserted at the start of the array and rear set to the next location

(*pq).queuearray[0].info =

(*pnewrecord).info;

(*pq).rear = 1;

}

}

}

}

Setqueue must set front and rear to minus one (-1). Empty will return the value true when front = -1, and false otherwise. The removal of a record can be implemented by a function remove, which returns in value the record that was deleted from the queue.

remove (pq, pvalue)

/* Deletes the front entry from

queue q and copies its contents

into value

*/

queue *pq;

queuerecord *pvalue;

{

    >Comment

if(empty(pq))

printf("\n underflow \n");

else

{

    >Comment

(*pvalue).info = (*pq).queuearray

[(*pq).front].info;

    >Comment

(*pq).front = (*pq).front + 1;

if((*pq).front == (*pq).rear)

2 if queue is empty, then reinitialize it

setqueue(pq);

    >Comment

if((*pq).front > (LIMIT - 1))

(*pq).front = 0;

}

}


Data Structures Using C: 4.7.2 List Implementation of the Queue with a

4.7.2 List Implementation of the Queue with an Array of Records

Now we can turn to implementing a queue as a list with records stored in an array. The availist of Section 3.7 actually acts as a list implementation of a queue if reclaim returns a record to the rear of the availist. Thus queue can be implemented as a list of records stored in an array.


Data Structures Using C: 4.7.3 List Implementation of the Queue with R

4.7.3 List Implementation of the Queue with Records in Dynamic Memory

The third queue implementation is also a list梠ne involving dynamic storage allocation of the records in dynamic memory. First, the following declaration must be made:

typedef struct record

{

whatever info;

struct record *link;

}queuerecord,*queue;

Setqueue must set queue to null. Insert will first request storage for the new record to be inserted by invoking malloc. It will then set its info data and insert it as the new last record in the queue. Remove must delete the first record of the queue and return its info data.

Your deck has now been "stacked," you've been "queued" into queues, and you mustn't postpone your obligation to use them!


Data Structures Using C: 4.8: Case Study: Checking Sequences for Prope

4.8: Case Study: Checking Sequences for Proper Nesting

To bring together the major ideas of this chapter, a case study to check program structure for proper nesting is presented. Its main features are:

1. The top-down approach

2. Demonstration of the use of the stack, treating it as a data abstraction

3. Checking program structure for proper nesting by developing a nonrecursive and a recursive function

4. The different perspectives required to develop the two functions

Every computer programmer has at one time or another gotten error messages saying "incomplete loop structure" or "a missing parenthesis." You now have enough background to write a program that checks for such errors. Let's see how it can be done.

Arithmetic expressions, as well as a program text, consist of sequences of symbols. Complex arithmetic expressions may be composed by properly combining simpler arithmetic expressions, delineated by matching left and right parentheses. The simpler expressions are said to be nested within the more complex expressions that contain them. The complex expression (a * (b + [c /d])) + b) contains two pairs of nested expressions. C programs contain compound statements delimited by matching { (begin) and } (end) pairs. Ignoring all other symbols, these matching pairs must satisfy rules of nesting to be syntactically correct.

Nesting means that one matching pair is wholly contained within another. When pairs are nested, they may not overlap, as do the two pairs in ( [ ) ]. The rules of nesting are the same for any sequence that is built by combining simpler sequences. They require that the start and end of each component be specified. Checking complex sequences to determine that these begin-end pairs do not overlap is an important operation. It provides an interesting application of the stack data abstraction, as well as of recursion. An intimate connection exists among such sequences, stacks, recursions, and trees; it will be explored more fully in later chapters.

Two abstractions of this problem will be considered. The second is a generalization of the first. To begin, assume only one kind of left parenthesis and one kind of matching right parenthesis, and consider sequences composed of these. Thus, at first, expressions such as ([]) are not allowed, since they involve more than one kind of parenthesis.

Given a sequence of left and right parentheses, determine if they may be paired so that they satisfy two conditions:

i. Each pair consists of a left and right parenthesis, with the left parenthesis preceding the right parenthesis.

ii. Each parenthesis occurs in exactly one pair.

If a sequence has a pairing satisfying both of these conditions, the sequence and pairing are valid. The problem is to determine whether a given sequence is valid or not. No sequence starting with a right parenthesis, ")", can be valid, since there is no preceding left parenthesis, "(", to pair with it. Any valid sequence must therefore begin with some series of n "(" parentheses preceding the first appearance of a ")" parenthesis, such as Figure 4.11(a), for which n is 3. Suppose we remove this adjacent pair, the nth "(" parenthesis, and the immediately following first ")" parenthesis. This would be pair 1 in Figure 4.11(a). If the new sequence thus obtained is valid, then adding the removed pair to it results in a valid pairing. Thus the original sequence is valid.

Suppose you find that the resultant sequence is not valid. Can you conclude that the original sequence is not valid? In other words, has constraining one of the pairs to be the first adjacent "(" and ")" parentheses made a pairing of the original sequence satisfying (i) and (ii) impossible? The answer is no. To see this, assume that there is some valid pairing of the original sequence, in which the first ")" parenthesis is not paired with the immediately preceding "(" parenthesis, but there is no valid pairing in which it is. In general, the first ")" parenthesis must then be paired with an earlier "(" parenthesis, and the immediately preceding "("parenthesis must have been paired with a later ")" parenthesis, as in Figure 4.11(b). These two pairs can then be rearranged by interchanging their "(" parentheses. The resultant pairing satisfies conditions (i) and (ii), yet it has the first adjacent "(" parenthesis paired with the immediately following ")" parenthesis, but this is the nesting constraint that has been specified. This contradicts the assumption that no valid pairing satisfies the constraint.

Consequently, the conclusion is that a solution to the problem is obtained as follows:

1. Pair the first adjacent left, "(", and right, ")", parentheses and remove the pair.

2. If the resultant sequence is valid, then so is the original sequence; otherwise the original sequence is not valid.

Think of a left parenthesis as denoting an incurred obligation, and a right parenthesis as the fulfilling of the most recently incurred obligation. This is the precise situation for which a stack is useful. As the program scans a sequence

(a) Four Pairs Satisfying Conditions i and ii

(b) Four Other Pairs Satisfying Conditions i and ii

(c) Four Pairs Not Satisfying Conditions i and ii; No Other Four Pairs Will

Figure 4.11 Valid and Invalid Nesting of Parentheses

from left to right, it pushes each "(" parenthesis onto a stack, representing an incurred obligation. Task 1 of the solution may then be carried out by popping the stack when the first ")" parenthesis is encountered, to fulfill the most recently incurred obligation.

Carrying out task 2 requires first determining if the resultant sequence is valid. Notice, however, that had the resultant sequence been processed in this same way, the contents of the stack would be exactly what they are now. Consequently, the program can continue to scan the original sequence (which is exactly what would be seen if it were scanning the resultant sequence), pushing whenever it encounters a "(" parenthesis, and popping whenever it comes to a ")" parenthesis. If the stack is empty when the entire scan is completed, then we have determined that the original sequence is valid; otherwise it is not. During the scan, any attempt to pop an empty stack means that no "(" parenthesis is available to be paired with the ")" parenthesis that caused the attempt (that is, the sequence is invalid).

This solution may be implemented as in the function determine, which follows this paragraph. Note that the stack is treated as a data abstraction . Readchar is a function that reads into its parameter the next character of the input sequence and returns true, unless the character read was a specialchar used as a sentinel, and then it returns false. When determine returns, the entire input sequence has been read. Valid will return true if a pairing satisfying (i) and (ii) exists, and false otherwise.

Nonrecursive Version

determine(pvalid)

/* Returns true in valid only if the

input sequence is properly nested

*/

int *pvalid;

{

char currentchar;

struct

{

char topchar;

}topcharacter;

>Comment

stack s;

*pvalid = TRUE;

>Comment

setstack(&s);

>Comment

while(readchar(&currentchar))

    >Comment

if(leftparen(currentchar))

push(&currentchar,&s);

    >Comment

else if(rightparen(currentchar))

if(!empty(&s))

{

pop(&s,&topcharacter);

    >Comment

if(!match(topcharacter.topchar,

currentchar))

*pvalid = FALSE;

}

else

*pvalid = FALSE;

else

    >Comment

process(currentchar,&s);

    >Comment

if(!empty(&s))

*pvalid = FALSE;

}

Leftparen returns the value true if its parameter is a left parenthesis, and false otherwise. Rightparen is defined similarly. Match returns the value true if topchar and currentchar are left and right parentheses, respectively.

One of the aims of the text is to show how general functions can be adapted to solve particular problems. Determine is such a general function. It can be adapted to evaluate a completely parenthesized infix expression. We repeat the rule of Section 4.6.2 to be used to evaluate such an expression:

1. Scan the infix expression from left to right.

2. When a "matched" pair of "left" and "right" parentheses is encountered,

apply the operator to the operands enclosed within the pair of parentheses, and replace the paired parentheses and the contained operator and operands by the result. Continue scanning from this point.

The matched pairs referred to in this rule are exactly the same as in this case study. Studying the rule, you can see that a right parenthesis signals that an operator and its operands are enclosed between the right parenthesis and its matching left parenthesis.

Determine may be modified to obtain a function evaluate, by defining the process routine as follows:

1. When an operand is encountered, place the operand on top of an operand stack.

2. When an operator is encountered, place the operator on top of the operator stack.

It is also necessary, when a "right" parenthesis is encountered, and the operator stack is not empty, to pop the operator stack, remove the required number of operands from the top of the operand stack, apply the operator to those operands, and place the result on the top of the operand stack. This must be done just before the pop(&s,&topcharacter) statement. Determine's stack, s, plays the role of the operator stack. An operand stack must be introduced and must be available to both the evaluate and process functions.

The general function determine is actually more complex than need be for "paired parentheses." In fact, a stack is not even necessary. All that is really needed is to keep a tally of "(" parentheses, decreasing the tally by 1 every time a ")" parenthesis is read. If the tally ever becomes negative, then determine should, from that point, simply read in the rest of the input sequence and return false. Otherwise, if the tally is zero after the entire input sequence has been read, it should return true. Determine is implemented as shown in the nonrecursive function listed above because it will serve as the solution to the next problem, for which "paired parentheses" will be a special case.

Suppose, instead of containing only "(" and ")" parentheses, the input sequence could consist of any characters. Assume that each character could be either a left or a right character but not both, and also that each left character has a unique right character with which it matches. For instance, a, (, [, and 1 might be "left" characters and the unique "right" characters with which they match z, ), ], and r, respectively.

Given a sequence of left and right characters, determine if they may be paired so that

i. Each pair consists of a left and a right matching character, with the left character preceding the right character.

ii. Each character occurs in exactly one pair.

iii. The sequence of characters between the left and right characters of each pair must also satisfy (i) and (ii).

If a sequence of left and right characters satisfies conditions (i), (ii), and (iii), the sequence and the pairing are valid. Condition (iii) then says that the sequence of characters between the left and right characters of each pair must be valid.

Let's see if this definition makes sense. Consider the above sample "matching" involving a, z, (, ), [, ], l, and r and the sequences of Figure 4.12.

Both the pairings of Figure 4.12(a) and those of Figure 4.12(b) satisfy (i) and (ii). However, only the pairing of (a) satisfies (iii). The pairing of (b) fails because the left and right characters of pair 3 are a and z, and the sequence of characters between them is ])1r. This sequence does not satisfy conditions (i), (ii), and (iii), because none of the six possible ways to form two pairs will be valid. It is also apparent that the pairing of (b) fails, because the left and right characters of pair 2 are "[" and "]" and the sequence of characters between them, a, is not valid. One would have to be sure no other pairings for (b) are valid before concluding that (b)is not a valid sequence.

It is not immediately apparent that this problem generalizes the first problem, even if the characters are restricted to be just left and right parentheses. This is because condition (iii) does not appear in the first problem. However, the reasoning used there shows that adding the condition does not change the problem. In fact, if the left and right matching characters are thought of as different kinds of parentheses, the same reasoning applied in the first problem can be applied to the present problem.

(a) Five pairs Satisfying Conditions i-iii

(b) Five Pairs Not Satisfying Conditions i-iii; No Other Five Pairs Will

Figure 4.12 Valid and Invalid Nesting of Matched Characters

Again, any valid sequence must start with some series of n > 0 left parenthese preceding the first appearance of a right parenthesis. Suppose its matching left parenthesis does not immediately precede it. This is the situation, for example, in Figure 4.12(b), where there are four left parentheses, a, (, [, a, appearing before "]," the first right parenthesis. In order for this first right parenthesis to satisfy (i) and (ii), it must then be paired with a previous matching left parenthesis. But then the immediately preceding left parenthesis is the last character in the sequence between that pair of parentheses. In order to satisfy (iii), this sequence must be valid. This sequence cannot be valid since this last left parenthesis has no matching right parenthesis in this sequence, as indicated in Figure 4.13.

In conclusion, a solution may be obtained as follows:

1. Pair the first adjacent left and right parentheses that match, and remove the pair.

2. If the resultant sequence is valid, then so is the original sequence; otherwise the original sequence is not valid.

Figure 4.13 An Invalid Nesting Sequence

Function determine is also an implementation of this solution using a stack, assuming leftparen, rightparen, and match reflect the different left parentheses, right parentheses, and matched pairs, respectively. The stack is needed in order to implement this solution. It is no longer simply a question of increasing or decreasing a tally, depending on whether the program reads a "(" or a ")"parenthesis, and making sure it never goes below zero and ends at zero. It is not even enough to keep a distinct tally for each of the different kinds of "left" and "right" parentheses. This would not result in Figure 4.12(b), for example, being recognized as invalid, because each of the four numbers would satisfy the requirements just mentioned. It is condition (iii), and more than one kind of left and right parentheses, that makes the stack mandatory. Consider a pair satisfying conditions (i) and (ii). To satisfy condition (iii), the sequence of characters between the "left" and "right" parentheses of the pair must also be valid. In effect, a new set of tallies must be kept for that sequence, and they must satisfy the three conditions. The stack is actually holding all of these tallies. Each time a "left" parenthesis appears, it becomes a character in many sequences, in general, and the tally for it, associated with each of those sequences, must be increased by one. Pushing it onto the stack, in effect, increases all these tallies by one simultaneously. Similarly, popping the stack decreases them all simultaneously.

So far the solution has been nonrecursive and has required looking at the input as simply a sequence of parentheses satisfying certain local constraints; a left parenthesis must encounter a matching right parenthesis before any other right parenthesis appears. Any global structure inherent in the input has been ignored. Development of a recursive solution, however, requires that a valid input be defined recursively. The recursive definition of a valid sequence may be restated to show its structure more clearly.

A valid sequence is

n     A null sequence

n     A left and right matching pair of parentheses with a valid sequence between them

n     A valid sequence followed by a valid sequence

In other words, a valid sequence is

n     A null sequence,

n     One valid sequence surrounded by a pair of matching parentheses, or

n     A series of valid sequences, each surrounded by a pair of matching parentheses

The last case would appear as

(valid sequence)(valid sequence) . . . (valid sequence)

The following program is a recursive version of determine based on the recursive definition.

Recursive Version

determine(pvalid,last)

/* Returns true in valid only if the input

sequence is properly nested; last must

initially contain a special character

used as the input terminator.

*/

int *pvalid;

char last;

{

char currentchar;

int done;

    >Comment

static int more;

done = FALSE;

more = TRUE;

while(!done && more)

17 loops through each input character

{

    >Comment

more = readchar(&currentchar);

if(leftparen(currentchar))

    >Comment

determine(pvalid,currentchar);

else if(rightparen(currentchar))

    >Comment

if(!match (last,currentchar))

*pvalid = FALSE;

else

{

    >Comment

done = TRUE;

    >Comment

last = SPECIALCHAR;

}

else

    >Comment

process(last,currentchar);

}

    >Comment

if(last != SPECIALCHAR)

*pvalid = FALSE;

}

Determine must be called initially with last set to specialchar. It is used as an input terminator and should not occur in the sequence being checked. When a valid sequence surrounded by a pair of matching parentheses has been encountered, the variable done is set to true. At that point the program continues looking for a next such sequence. However, last must be reinitialized to the specialchar. The variable more is used to signal whether or not more characters remain to be read in the input string. It is declared to be static because it must retain its value between any recursive calls to determine.

Both this recursive solution and the iterative solution work from the inside out. The iterative program looks for adjacent matching left and right parentheses in the interior of the sequence. The recursive program looks for a valid sequence surrounded by a matching pair of parentheses. Thus the iterative solution sees the input

[ ( [ ] ) ] ( ) { [ ] } #

as

[ ( [ 1 ] ) ] ( 4 ) { [ 5 ] } #     (6 pairs of matching parentheses)

-   2   -    -   6   -

-  - 3   - -

and the recursive solution sees it as

[ ( [ ] ) ] ( 2 ) { [ ] } #    (3 valid sequences each one surrounded by a pair of

- -  1 - -         - 3  -       matching parentheses)

where # denotes the specialchar.

Another recursive solution could be written to work from the outside in, looking for a rightmost mate to the first left parenthesis and then checking the sequence between them for validity. This would lead to a less efficient solution, because parentheses could be considered more than once. Recursion often involves this dual possibility, so both should be explored.


Data Structures Using C: Exercises

Exercises

1. Set i and j be two integers and define q(i, j) by q(i, j) = 0 if i < j and q(i - j, j) + 1 if i � j

a. What is the value of q(7,2)?

b. Can we determine q(0,0)?

c. Can we determine q(-3, -4)?

2. Let a be an array of type float and i and n be positive integers. The function

m(a,i,n) is defined by

m(a,i,n) = a[i] if i = n and max(m(a,i + 1,n) ,a[i]) if i < n

a. Find m(a,i,n) if i = 1, n = 6, and a is

0.   6.8

1.   3.2

2.  -5.0

3.  14.6

4.   7.8

5.   9.6

6.   3.2

7.   4.0

b. What does m(a,i,n) do?

3. Let a be an array of type real, and i and n be positive integers. The function m(a,i,n) is defined by

�(i + n)/2� means, "Do an integer divide of (i + n)/2."

a. Find m(a,i,n) for i = 1, n = 6, and a the array of Exercise 2.

b. What does m(a,i,n) do?

4. Let i be a nonnegative integer. Find b(4) where

b(0) = 1

b(n) = b(0) � b(n - 1) + b(1) � b(n - 2)

+ b(2) � b(n - 3) + . . . + b(n - 1) � b(0) n>0

5. Write a recursive function max corresponding to Exercise 2.

6. Write a recursive function max corresponding to Exercise 3.

7. a. Why does the recursive solution to the Towers of Hanoi problem use the minimum number of moves?

b. What is the number of moves it uses?

8. Consider the following nonrecursive algorithm for the Towers of Hanoi problem.

1. Label the disks 1, 2, . . ., n in increasing order of size.

2. Label the pegs i, a, and �, respectively, n + 1, n + 2, and n + 3.

3. Move the smallest disk onto another disk with an even label or onto an empty peg with an even label.

4. If all disks are on the same peg, then terminate. Move the second smallest top disk onto the peg not containing the smallest disk. Go to 1.

a. Convince yourself that this correctly solves the Towers of Hanoi problem.

b. Is the recursive solution or this nonrecursive version easier to understand.

9. Suppose a fourth peg is added to the Towers of Hanoi problem.

a. Write a recursive solution for this problem.

b. Can you find a recursive solution that uses the minimum number of moves for this problem?

10. Mark any n distinct points on the circumference of a circle. How many triangles may be drawn whose vertexes lie on three of the n points?

11. How many ways may k distinct points be selected from the n points of Exercise 10?

12. Given a permutation of 1, 2, . . ., n, let ni be the number of integers to the right of i which exceed i. For the permutation 2134, n 1 = 2, n 2 = 3, n 3 = 1, n 4 = 4. Write a process function that will turn permutations into a program that prints n1, n2, . . ., nn for each permutation.

13. Suppose 1 is implemented as a two-way list. Write a recursive function to reverse 1.

14. Implement the function of Exercise 5.

15. Implement the function of Exercise 6.

16. Compare the requirements for the stack storage required during execution for the implementations of Exercises 5 and 6.

17. Why are the more efficient versions of Section 4.4 more efficient?

18. The Fibonacci sequence is defined by

�(0) = 0, �(1) = 1

�(n) = f(n - 1) + �(n - 2) for n = 2,3,4 . . .

a. Write a recursive function fibonacci(n) to compute and print �(n).

b. Analyze its time and storage requirements.

19. a. Translate the program of Exercise 18 and obtain as efficient a version as you can.

b. Analyze its time and storage requirements.

20. Write a nonrecursive program fibonacci(n), which takes O(n) time and uses only four local variables.

21. a. Write a function element with parameter k, which returns the value of the kth element on the stack of integers implemented as in Figure 4.8. For instance, if k is 1, it returns the value of the current first element. It does not disturb the stack at all. Assume you do not have access to the stack directly, but only through the push, pop, and empty routines.

b. Same as Exercise 21(a), except assume you do have direct access to the stack.

22. Two stacks are implemented in one array of length 1,000 as shown. The tops of the stacks are pointed to by top1 and top2 and grow toward each other. Write a procedure push that pushes the contents of v onto the stack indicated by s. s will have a value of 1 or 2. If the stack to which v is to be added is full, then push should not add it, but should return with s set to 0.

23. Write a function maxstack that returns in m the maximal element on a stack of integers implemented with pointer variables. Maxstack will return true when the stack is not empty, and false otherwise.

24 What does the following do?

pop(&s,&value);

push(&value,&s)

25. Why are parentheses not needed when prefix or postfix notation is used?

26. How do conventions eliminate ambiguity in infix notation?

27. Evaluate the postfix expression 3 6 + 6 2 / *8+.

28. Simulate the execution of the algorithm of Section 4.6.2 for evaluation of an infix expression on (a * b/(c + d)) + e ** �.

29. Simulate the execution of the translation algorithm of Section 4.6.3 on (a * b/(c + d)) + e ** �.

30. What priority must be given to the assignment operator, =, if the evaluation and translation algorithms for infix to postfix are to work on assignment statements as well as arithmetic expressions?

31. Write the function evaluate.

32. Describe how evaluate must be modified if arithmetic expressions are allowed to include function calls.

33. What will the pairing generated by the function determine be for the input sequence aa([[]()])zz?

34. Modify the function determine so that it outputs the pairs as follows:

1 2 3 4 5 6 7 8

if ( ( ( ) ) ( ) ) is the input sequence,

then the output will be the pairs 3 4, 2 5, 6 7, 1 8.

35. Modify function determine so that it also outputs ( ( ( ) ) ( ) ), for example, if the input is ( ( ( ) ) ( ) ).

4 2 1 1 2 3 3 4

36. How must determine be modified if some left characters may also be right characters, and they will always match each other? For example, let the left characters be a, (, [,b, with the "right" characters being a, ), ], b. Here a is matched with a, b with b, "("with ")," and "[" with "]." Then a[a] is not valid, but a[CC](aa)a and ab()aab[]a are.

37. Write the remove function for the array implementation of a queue.

38. Write the insert function for the pointer variable implementation of a queue.

39. Write the remove function for the pointer variable implementation of a queue.

40. Suppose function buffer has three parameters, q of type queue, e of type whatever, and op of type character. If op is r, then buffer returns after removing the top record of the queue and copying its value into e. If op is n, then buffer returns after inserting the value in e at the bottom of the queue. Write an implementation of buffer, assuming that access to the queue is only through the empty, remove, and insert routines.

41. A deque (pronounce like "deck") is a data abstraction that retains ordered information, but allows insertion and deletion at both its front and rear.

a. Develop an array-based implementation of a deque and write corresponding setdeque, insertf, insertr, removef, remover, and empty routines.

b. Do the same as in Exercise 41(a), except use pointer variables with dynamic storage allocation for the deque implementation.

42. Suppose there is a program to keep track of the current board configuration in a game of Monopoly by using a circular array of records, each record representing a board position (such as Boardwalk), and queues for Community Chest and Chance. Discuss how the records should be defined, what pointers to records will be convenient, and how these pointers and the circular array should be processed on each turn.


Data Structures Using C: Suggested Assignments

Suggested Assignments

1. Write and simulate a recursive program to reverse a list.

2. You might enjoy finding an algorithm for the following generalization of the Towers of Hanoi problem. The disks are each labelled with a positive integer. A configuration of disks on a peg is "legal" if each disk's integer is no larger than the sum of the integers on all disks below it, or there are no disks below it. How can the disks be moved so that any legal original configuration of n disks on peg 1 ends up on peg 3, with all intermediate configurations being legal?

3. Write the permutation program so it treats the permutation as a data abstraction.

4. In order to use storage efficiently when n stacks are needed during the execution of a program, it may be desirable to implement all the stacks in one array as shown here for n= 4.

The Bi's and Ti's point to the bottoms and tops of the stacks, which grow to the right as indicated. Think of the total storage currently allocated to stack i as being from B(i) to B(i + 1). Stack i is actually using T(i) - B(i + 1) + 1 positions of the array. If B1 = 1, B2 = 10, B3 = 15, B4 = 25, B5 = size of the array = 35, T1 = 3, T2 = 12, T3 = 20, and T4 = 33, then the stacks are using 3, 3, 6, and 9 entries, respectively, while they are allocated 9, 5, 10, and 10 entries.

When an entry is to be inserted into a stack that is currently using all its allocated storage, the stack is said to have overflowed. Stack i overflows if Ti = B(i + 1) - 1. Write a function to manage the storage provided by the array. Whenever a stack overflows, the function is to be invoked. If there is any unused space in the array, then the stack with an unused but allocated position which is "closest" to the overflow stack is to give up one of its unused positions. This position is then used by the overflow stack to accommodate its entry. This function must be written to shift stacks around. If no unused space exists in the array, then the procedure should print an "out of stack storage" message.


Data Structures Using C: CHAPTER 5: PACKAGING DATA ABSTRACTIONS

CHAPTER 5: PACKAGING DATA ABSTRACTIONS

Introduces the concept of modules and file modularization to structure programs

With this type of program structure data abstractions are placed in files separate from the program file, and this hides the details of the data abstraction from the programmer using them easing the programming task

increases efficiency as it enables the data abstractions to be compiled separately and stored in object code (separate compilation is possible)

To illustrate how programs are structured using modules and file modularization four examples, from previous chapters, are presented in detail

primes

stablemarriages

creation and printing of a list

nonrecursive Towers of Hanoi

5.1: Data Abstraction Revisited

5.2: The Prime Example

5.3: Stable Marriages

5.4 : A List Example

5.5: A Stack Example

5.6: Summary

Exercises

Suggested Assignments


Data Structures Using C: 5.1: Data Abstraction Revisited

5.1: Data Abstraction Revisited

The first step in creating a program is to develop the algorithm that will produce the solution. The algorithm must detail operations that will be performed on the given data. A decision must also be made about how the data will be organized. Typically data is best split into distinct groups, or collections, based on the basic operations required for each collection. The data structure utilized for each collection can be tailored to the specific operations to be performed on it. Each collection, with its operations, can then be treated individually as a data abstraction.

A data abstraction is a collection of data upon which only a set of specific operations are allowed. The technique of data abstraction has been used in this text as a tool in creating programs that are understandable and maintainable. Insight is required to determine the particular collection to use and the operations to act upon it. The same collection may be useful in a number of different problems. This may be so with the same set of allowable operations or with different sets of operations. As an example, recall that the stack data abstraction was used in both towers and next with the same operations.

Once the data abstractions to be used in a solution are determined, a decision must be made on how to implement them. This requires deciding what data structure is to be used to store the collection and how the operations on it will be defined. In this text each operation has been incorporated in a function using the tool of functional modularization. Treating the collection and its operations as a data abstraction means that the program can only process the collection using those functions. This guarantees that the collection is accessed only when one of the functions is invoked. Proper packaging of the data abstraction can thus ensure that it is used appropriately. How this packaging is done is the topic of this chapter.

The discussion is presented only now for several reasons. First, preceding chapters have introduced a number of data abstractions that can serve as vehicles to present different approaches to packaging. Next, packaging introduces another level of abstraction, which can complicate the understanding of the basic material. Time was needed to absorb the beginning concepts. Finally, packaging is language-dependent, and the aim of this book is to be as language-independent as possible yet still present programs in the C programming language. Given the language-dependence of packaging and the belief that students should first concentrate on basics, the topic is confined to this chapter. This does not mean it is unimportant梠nly that it might cloud the other concepts presented. Consequently the material in this chapter may be skipped or, alternatively, imposed on the programs in the later chapters. The C approaches to packaging are straightforward; should you desire, the programs can be readily modified to the style presented here.


Data Structures Using C: 5.2: The Prime Example

5.2: The Prime Example

Consider the program below from Chapter 1 for calculating prime numbers.

#include <stdio.h>

    >Comment

main()

{

int n;

printf("\n n = ?\n");

scanf("%d",&n);

primes(n);

}

    >Comment

#define TRUE 1

#define FALSE 0

#define MAXN 500

typedef int collection[MAXN];

set(n,c)

/* Sets c to empty */

int n;

collection c;

{

int i;

for (i=1;i<=n;i++)

c[i] = FALSE;

}

insert(i,c)

/* Inserts i in c */

int i;

collection c;

{

c[i] = TRUE;

}

omit(i,c)

/* Removes i from c */

int i;

collection c;

{

c[i] = FALSE;

}

belongs(i,c)

/* Returns true only

if i is in c

*/

int i;

collection c;

{

return(c[i]);

}

    >Comment

primes(n)

/* Prints all prime numbers between 2 and n */

int n;

{

collection candidates;

create(n,candidates);

remove(n,candidates);

print(n,candidates);

}

create(n,c)

/* Creates a collection of integers between 2 and n */

int n;

collection c;

{

int i;

    >Comment

set(n,c);

    >Comment

for (i=2;i<=n;i++)

insert(i,c);

}

remove(n,c)

/* Removes all nonprimes from c */

int n;

collection c;

{

int firstprime;

int factor;

firstprime = 2;

    >Comment

factor = firstprime;

while (nonprimes(factor,n))

{

delete(factor,n,c);

factor++;

}

}

nonprimes(factor,n)

/* Returns true only if non

primes may remain in c

*/

int factor,n;

double sqrt();

{

    >Comment

return(factor <= sqrt((double)n));

}

delete(factor,n,c)

/* Deletes multiples of factor from c */

int factor,n;

collection c;

{

int nextmultiple;

    >Comment

nextmultiple = 2 * factor;

while (nextmultiple <= n)

{

omit(nextmultiple,c);

nextmultiple = nextmultiple + factor;

}

}

print(n,c)

/* Prints integers in c */

int n;

collection c;

{

int i;

    >Comment

for (i=2;i<=n;i++)

if (belongs(i,c))

printf("\n %d\n",i);

}

This program is structured with the data abstraction collection and its operations set, insert, omit, and belongs placed together. This facilitates finding its data structure and function declarations and definitions, whenever modifications are desired. Changes to collection, set, insert, omit, and belongs may be made, as long as they still do their tasks correctly without changing any other part of the program.

The program, however, is more cluttered and longer than necessary. It is burdened with exhibiting details of the data abstraction that are irrelevant to it. One way to avoid this is to use the include capability of C. Using C's include statement, we can

n     Create a file called collection that contains all the statements specifying the data abstraction, and then

n     Insert these into the program with one statement梩he include statement

This is done in the following version of the program.

When this program is compiled, the C preprocessor will replace the include "collection" statement in the program file with the source code from the file collection. Thus the object code produced by the compiler for this version will be identical to that produced by the first version.

#include <stdio.h>

    >Comment

main()

{

int n;

printf("\n n = ?\n");

scanf("%d",&n);

primes(n);

}

    >Comment

#include "collection"

    >Comment

primes(n)

/* Prints all prime numbers between 2 and n */

int n;

{

collection candidates;

create(n,candidates);

remove(n,candidates);

print(n,candidates);

}

create(n,c)

/* Creates a collection of integers between 2 and n */

int n;

collection c;

{

int i;

    >Comment

set(n,c);

    >Comment

for (i=2;i<=n;i++)

insert(i,c);

}

remove(n,c)

/* Removes all nonprimes from c */

int n;

collection c;

{

int firstprime;

int factor;

firstprime = 2;

    >Comment

factor = firstprime;

while (nonprimes(factor,n))

{

delete(factor,n,c);

factor++;

}

}

nonprimes(factor,n)

/* Returns true only if non

primes may remain in c

*/

int factor,n;

double sqrt();

{

    >Comment

return(factor <= sqrt((double)n));

}

delete(factor,n,c)

/* Deletes multiples of factor from c */

int factor,n;

collection c;

{

int nextmultiple;

    >Comment

nextmultiple = 2 * factor;

while (nextmultiple <= n)

{

omit(nextmultiple,c);

nextmultiple = nextmultiple + factor;

}

}

print(n,c)

/* Prints integers in c */

int n;

collection c;

{

int i;

    >Comment

for (i=2;i<=n;i++)

if (belongs(i,c))

printf("\n %d\n",i);

}

The file collection, which can be stored separately, contains:

/* The data abstraction collection

and its allowed operations */

    >Comment

#define TRUE 1

#define FALSE 0

#define MAXN 500

typedef int collection [MAXN];

set(n,c)

/* Sets c to empty */

int n;

collection c;

{

int i;

for (i=1;i<=n;i++)

c[i] = FALSE;

}

insert(i,c)

/* Inserts i in c */

int i;

collection c;

{

c[i] = TRUE;

}

omit(i,c)

/* Removes i from c */

int i;

collection c;

{

c[i] = FALSE;

}

belongs(i,c)

/* Returns true only

if i is in c

*/

int i;

collection c;

{

return(c[i]);

}

So far, the programs in this text have been written so that they are compiled as a single source code file. This is still true for the two versions of primes. The first was written as a single source file. In the second, the include statement causes the precompiler to incorporate collection into the program file, producing a single source code file.

It is also possible in C to locate the functions of a program in more than one file. Each file of source code can be separately compiled, producing a corresponding file of object code. This ability to allocate the functions of a program to different files is a tool that can be used to achieve another kind of modularity besides functional modularity. This new modularity, which we call file modularity, allocates functions to files, whereas functional modularity allocates tasks to functions. In keeping with this definition of file modularity, the term module will be used to mean one or more files, to be separately compiled, used for the implementation of a data abstraction. Before compilation they are source modules; after compilation they are object modules.

To create a module for collection, two files in addition to the program file will be needed. They are called collection.h and collection.c. Collection.h, the header file, contains all definitions needed for collection's data structure and all declarations for its functions. The header file must be included in any file using this data abstraction. Collection.c, which contains definitions of all the functions of the data abstraction that implement its operations, must also include the header file, as in the following example.

The two files are shown below. In collection.h a standard way of declaring a function is presented. It uses the extern declaration, as in the following example.

extern set(/* int,collection */);

The extern declaration informs the compiler that the function is defined elsewhere, in another file. The comment /* int,collection */ identifies the parameters of the function�since it is a comment, it disappears in compilation.

With this version, the program file and collection are compiled separately into object modules. The linker must combine the two object modules to create an executable program.

The File collection.h

/* header for collection of integers */

#define TRUE 1

#define FALSE 0

#define MAXN 500

typedef int collection[MAXN];

extern set(/* int,collection */);

/* Sets collection to empty */

extern insert(/* int,collection */);

/* Adds int to collection */

extern omit(/* int,collection */);

/* Takes int out of collection */

extern belongs(/*int,collection */);

/* Returns true only if int is in collection */

The File collection.c

/* operations for collection of integers */

    >Comment

#include "collection.h"

set(n,c)

/* Sets c to empty */

int n;

collection c;

{

int i;

for (i=1;i<=n;i++)

c[i] = FALSE;

}

insert(i,c)

/* Inserts i in c */

int i;

collection c;

{

c[i] = TRUE;

}

omit(i,c)

/* Removes i from c */

int i;

collection c;

{

c[i] = FALSE;

}

belongs(i,c)

/* Returns true only

if i is in c

*/

int i;

collection c;

{

return(c[i]);

}

Final Version: The File primestest.c

#include <stdio.h>

    >Comment

main()

{

int n;

printf("\n n = ?\n");

scanf("%d",&n);

primes(n);

}

    >Comment

#nclude "collection.h"

    >Comment

primes(n)

/* Prints all prime numbers between 2 and n */

int n;

{

collection candidates;

create(n,candidates);

remove(n,candidates);

print(n,candidates);

}

create(n,c)

/* Creates a collection of integers between

2 and n */

int n;

collection c;

{

int i;

    >Comment

set(n,c);

    >Comment

for (i=2;i<=n;i++)

insert(i,c);

}

remove(n,c)

/* Removes all nonprimes from c */

int n;

collection c;

{

int firstprime;

int factor;

firstprime = 2;

    >Comment

factor = firstprime;

while (nonprimes(factor,n))

{

delete(factor,n,c);

factor++;

}

}

nonprimes(factor,n)

/* Returns true only if non

primes may remain in c

*/

int factor,n;

double sqrt();

{

    >Comment

return(factor <= sqrt((double)n));

}

delete(factor,n,c)

/* Deletes multiples of factor from c */

int factor,n;

collection c;

{

int nextmultiple;

    >Comment

nextmultiple = 2 * factor;

while (nextmultiple <= n)

{

omit(nextmultiple,c);

nextmultiple = nextmultiple + factor;

}

}

print(n,c)

/* Prints integers in c */

int n;

collection c;

{

int i;

    >Comment

for (i=2;i<=n;i++)

if (belongs(i,c))

printf("\n %d\n",i);

}

Although this third version may seem similar to the second, it differs in an important way. Its functions are distributed among three files. Two of these, collection.h and collection.c, are used by the module for collection, the data abstraction. This allows the details of the implementation to be hidden, removes the need for the program itself to change whenever the data abstraction implementation is changed, and also shortens the program. The second version does all this as well, but the third version is even better. It allows collection.c to be compiled separately from file primestest.c, the file containing main, primes, create, and so on. More precisely, primestest.c must be compiled, and then the object code for primestest.c, along with the object code from the separately compiled file collection.c, can be linked by the linker. Finally, the resultant executable file produced by the linker can be run.

Once collection.c has been successfully compiled and verified, its object module can be stored in a library for use in the creation of programs such as primes. In this way C fosters the building up of programs by putting together previous solutions. In the creation or modification of large programs, the use of already compiled modules can significantly reduce time for compiling and program development.

Whenever the implementation of the data abstraction collection changes, changes may occur in the source code of collection.h or of collection.c. When they occur in collection.h, any file that includes it must be recompiled. In our example, this means that collection.c and primestest.c must be recompiled. To run primestest.c, its compiled version, along with the compiled version of the new collection.c, must be linked; then the resultant executable program file can be run. The new object code produced by compiling the new version of collection.c should replace the old version in the library. When the changes occur in collection.c only, it must be recompiled followed by relinking with the object code for primestest.c before the new executable program file can be run.


Data Structures Using C: 5.3: Stable Marriages

5.3: Stable Marriages

As a second example of file modularization, consider the stable marriage program of Section 2.5.2. That program is structured with the data abstractions pairings, with its operations mate, read_mens_mates, and create_womens_mates placed together, and preferences, with its operations mostpreferred, nextpreferred, prefers, read_preferences, and create_priorities also placed together. Grouping each data abstraction facilitates finding its data structures, function declarations, and definitions, whenever modifications to them are required. Such changes will not necessitate changing any other part of the program, since it was written treating pairings and preferences as data abstractions. Also, changes to one data abstraction do not require changes in the other. The program in Section 2.5.2 is written as a single source file. We now place the two data abstractions in modules. The modules and the program file structure of the program using file modularization are printed below. The headers, operations and program file are, respectively, pairings.h, pairings.c, preferences.h, preferences.c, and stablemarriages.c.

The File pairings.h

/* header for pairings */

#define MAXN 50

typedef int mates[MAXN];

extern mate(/* int, pairings */);

/* Returns the mate of person int */

extern read_mens_mates(* int, pairings */);

/* Reads the mates of the men(int of them)

into pairings

*/

extern create_womens_mates(/* int,pairings_1,pairings_2 */);

/* Creates the pairings_2 of the

men(int of them), based on

the mens' mates specified by

pairings_1

*/

The File pairings.c

/* operations allowed on pairings */

    >Comment

#include "pairings.h"

mate (person,pairs)

/* Returns persons mate

as specified by pairs */

int person;

pairings pairs;

{

return(pairs[person]);

}

read_mens_mates(n,pairs)

/* Reads the n wives into pairs */

int n;

pairings pairs;

{

int i;

printf("\n mpairs = ? \n");

for (i=1;i <=n;i++)

scanf("%d",&pairs[i]);

}

create_womens_mates(n,pairs1,pairs2)

/* Pairs 1 specifies the n wives, one

for each man. Pairs2 is set to

specify the n men, one for each woman.

*/

int n;

pairings pair;

{

int i,j;

for (i=1;i<=n;i++)

{

for (j=1;pairs1[j] != i;j++)

;

pairs2 [i] = j;

}

}

The File preferences.h

/* header file for preferences */

#define MAXN 50

typedef int preferences[MAXN][MAXN];

extern mostpreferred(/* int,preferences */);

/* Returns the mate most preferred by

person int as specified by pref */

extern nextpreferred(/* int_1,int_2,pref */);

/* Returns the mate preferred most, after

the mate given by index int_2, by person

int_1 as specified by pref

*/

extern prefers(/* int_1,int_2,int_3,pref */);

/* Returns true if individual int_1 prefers

person int_2 to individualsmate int_3

as specified by pref

*/

extern read_preferences(/* int,pref */);

/* Reads int number of rows of

preferences into pref

*/

extern create_priorities(/* int,pref_1,pref_2 */);

/* Creates int number of individual

priorities in pref_2 based on the

individual preferences specified

by pref_1

*/

The File preferences.c

/* operations allowed on preferences */

    >Comment

#include "preferences.h"

mostpreferred (person,pref)

/* Returns the mate most preferred by

person as specified by pref */

int person;

preferences pref;

{

return(pref[person][1]);

}

nextpreferred(person,index,pref)

/* Returns the mate preferred most, after

the mate given by index, by person

as specified by pref

*/

int person,index;

preferences pref;

{

return(pref[person][index]);

}

prefers(individual,person,individualsmate,priority)

/* Returns true if individual prefers person to

individuals mate as specified by priority

*/

int individual,person,individualsmate;

preferences priority;

{

return(priority[individual][person] <

priority [individual] [individualsmate]);

}

read_preferences (n,pref)

/* Reads n rows of

preferences into pref

*/

int n;

preferences pref;

{

int i,j;

for (i=1;i<=n;i++)

for (j=1;j<=n;j++)

scanf ("%d",&pref[i][j]);

}

create_priorities(n, pref, priority)

/* Creates n individual priorities

in priority based on the

individual preferences specified

by pref

*/

int n;

preferences pref,priority;

{

int i,j;

for (i=1,i<=n;i++)

for (j=1;j<=n;j++)

priority[i][pref[i][j]] = j;

}

The File stablemarriages.c

#include <stdio.h>

    >Comment

#include "pairings.h"

    >Comment

#include "preferences.h"

#define TRUE 1

#define FALSE 0

main()

{

pairings mpairs,wpairs;

preferences mpref,wpref,mpriority,wpriority;

int i,j,n;

printf("\n n = ?\n");

scanf("%d",&n);

read_mens_mates(n,mpairs);

create_womens_mates(n,mpairs,wpairs);

printf("\n Enter mens preferences \n");

read_preferences(n,mpref);

printf("\n Enter womens preferences \n");

read_preferences(n,wpref);

create_priorities(n,mpref,mpriority);

create_priorities(n,wpref,wpriority);

if (stabilitycheck(n,mpairs,wpairs,mpref,wpref,mpriority,

wpriority))

printf(" The pairings are stable \n");

else

printf(" The pairings are not stable \n");

printf(" The pairings are: \n");

printf(" man wife \n");

for (i=1;i<=n;i++)

printf("%d %d \n",i,mate (i,mpairs));

}

stabilitycheck(n,mpairs,wpairs,mpref,wpref,mpriority,wpriority)

int n;

pairings mpairs,wpairs

preferences mpref,wpref,mpriority,wpriority;

{

int man,wife,individual,individualsmate,index,stable;

stable = TRUE;

for(man=1;(stable && man<=n);man++)

{

wife = mate(man,mpairs);

individual = mostpreferred(man,mpref);

index = 1;

while (stable && (individual != wife))

{

individualsmate = mate(individual, wpairs);

if(prefers(individual,man,individualsmate,

wpriority))

stable = FALSE;

else

{

index++;

individual = nextpreferred(man,index,mpref);

}

}

individual = mostpreferred(wife,wpref);

index = 1;

while(stable && (individual != man))

{

individualsmate = mate(individual,mpairs);

if(prefers(individual,wife,individualsmate,

mpriority))

stable = FALSE;

else

{

index++;

individual = nextpreferred(wife,index,wpref);

}

}

}

return(stable);

}

The include "pairings.h" and include "preferences.h" were put at the top of the program file because their contents are needed from that point in the file. In the program file for primestest.c, the include "collection.h" appears just before the primes function, since its contents are only needed from that point on in the file.

The files pairings.c, preferences.c, and stablemarriages.c can be compiled separately. To run stable marriages.c, they must first all be linked by the linker.

Any time changes are made in pairings.h, the files pairings.c and stablemarriages.c must be recompiled. Changes in pairings.c also require that pairings.c be recompiled . The same applies for preferences.h and preferences.c. However, changes to only one of pairings.c, preferences.c, or stablemarriages.c require only its individual recompilation. Relinking is always required before execution.


Data Structures Using C: 5.4 : A List Example

5.4 : A List Example

Let us look at another example of file modularization of a program. Consider the program of Section 3.6.1 that reads in, creates, and prints the information fields of each record on a list of stock records. That version is in a single file and is functionally modularized. The data abstraction list, with its operations next, setlink, setinfo, avail, getnextrecord, printrecord, and anotherrecord, appear together.

The version of the program that follows uses a module for the list data abstraction with the header file, list.h, and the operations file, list.c. The file, list.c, and the program file, listexample.c, are compiled separately.

The file list.h:

The File list.h

/* header for list */

#define MAXSIZE 5

#define NULL 0

#define SENTINEL -1

typedef struct

{

int month;

int day;

int year;

}date;

typedef struct

{

char name[MAXSIZE];

int shares;

float value;

date datebought;

}infofield;

typedef struct listrecord

{

infofield info;

struct listrecord *link;

}stockrecord,*listpointer;

extern listpointer next(/* listpointer */);

/* Returns a copy of listpointer in link field of

record pointed to by listpointer */

extern listpointer setnull();

/* Returns a null listpointer value */

extern setlink(/* listpointer_1,listpointer_2 */);

/* Sets the link field of the record pointed to

by listpointer_1 to the value of

listpointer_2

*/

extern setinfo(/* listpointer_1,listpointer_2 */);

/* Sets the info field of the record pointed to

by listpointer_1 to the contents of the

info field of the record pointed to by

listpointer_2

*/

extern listpointer avail();

/* Returns a pointer to storage for

a record of type struct stockrecord */

extern listpointer getnextrecord();

/* Returns a pointer to a record

containing the next input records'

information, or a null pointer if

the next input is the sentinel

*/

extern printrecord(/* listpointer */);

/* Prints the information in the record

pointed to by listpointer */

extern anotherrecord(/* listpointer */);

/* Returns true if there are no

more records and false otherwise */

The File list.c

/* operations allowed on a list */

    >Comment

#include "list.h"

listpointer next(pointer)

/* Returns a copy of the

link field in the record

pointed to by pointer

*/

listpointer pointer;

{

return (pointer->link);

}

listpointer setnull()

/* Returns a null pointer */

{

return(NULL);

}

setlink(pointer1,pointer2)

/* Sets the link field of the

record pointed to by pointer1

to the contents of pointer2

*/

listpointer pointer1, pointer2;

{

pointer1->link = pointer2;

}

setinfo(pointer1,pointer2)

/* Sets the infofield of the record

pointed to by pointer1 to the

infofield of the record

pointed to by pointer2

*/

listpointer pointer1,pointer2;

{

int i;

for (i=0;i<MAXSIZE;i++)

pointer1->info.name[i] = pointer2->info.name[i];

pointer1->info.shares = pointer2->info.shares;

pointer1->info.value = pointer2->info.value;

pointer1->info.datebought.month = pointer2->info.datebought.

month;

pointer1->info.datebought.day = pointer2->info.datebought.day;

pointer1->info.datebought.year = pointer2->info.datebought.year;

}

listpointer avail()

*/ Returns a pointer to storage

allocated for a record of type

stockrecord

*/

{

listpointer malloc();

return(malloc(sizeof(stockrecord)));

}

anotherrecord(recordpointer)

/* Returns true if there is

another record

*/

listpointer recordpointer;

{

listpointer setnull();

return(recordpointer != setnull());

}

listpointer getnextrecord()

/* Inputs the data for the new

record and returns a pointer

to a record containing the data

in its infofield, or, if there

are no new records, returns

a null pointer

*/

{

listpointer setnull();

static stockrecord nextrecord;

printf(" Enter number of shares \n");

scanf("%d",&(nextrecord.info.shares));

if(nextrecord.info.shares != SENTINEL)

{

printf(" Enter the stock name - less than %d characters\n",

MAXSIZE);

scanf("%s",nextrecord.info.name);

printf(" Enter the price of one share of the stock \n");

scanf("%f",&(nextrecord.info.value));

printf(" Enter the month day year of the stock purchase

\n");

scanf("%d %d %d",&(nextrecord.info.datebought.month),

&(nextrecord.info.datebought.day),

&(nextrecord.info.datebought.year));

return(&nextrecord);

}

else

return(setnull());

}

printrecord(recordpointer)

/* Prints the contents of the infofield

of the record pointed to by recordpointer

*/

listpointer recordpointer;

{

printf(" The stock is %s \n",recordpointer->info.name);

print(" The number of shares is %d \n",recordpointer->info.

shares);

printf(" The value of a share is %f\n",recordpointer->info.

value);

printf(" The date of purchase is %d %d %d\n\n",

recordpointer->info.datebought.month,

recordpointer->info.datebought.day, recordpointer->

info.datebought.year);

}

Finally, here is the program file to create a list and print its contents. It treats the list as a data abstraction and uses file modularization for the data abstraction.

The File listexample.c

    >Comment

#include "list.h"

    >Comment

#include <stdio.h>

#define TRUE 1

#define FALSE 0

main()

/* Inputs a series of stockrecords,

creates a list of the records, and

prints the contents of each list

record's infofield

*/

{

listpointer stocks,recordpointer,next();

int done;

    >Comment

initialize(&stocks);

recordpointer = stocks;

done = FALSE;

while(!done&&anotherrecord(recordpointer))

{

    >Comment

addrecord(&stocks,recordpointer,&done);

recordpointer = next(recordpointer);

}

    >Comment

recordpointer=stocks;

while (anotherrecord(recordpointer))

{

    >Comment

printrecord(recordpointer);

recordpointer = next(recordpointer);

}

}

initialize(plistname)

/* Allocates storage for the first record

and sets listname to point to it

*/

listpointer *plistname;

{

listpointer avail();

*plistname = avail();

}

addrecord(plistname,recordpointer,pdone)

/* Fills in the current record's data, and

allocates storage for the next record and

adds it to the list. If there is no data for

the current record it sets the link field of

the last list record to null, or the list

head, to null, and done to true.

*/

listpointer *plistname,recordpointer;

int *pdone;

{

static listpointer predecessor;

listpointer setnull(),avail(),

getnextrecord(),pointer;

pointer = getnextrecord();

if (pointer != setnull())

{

    >Comment

setinfo(recordpointer,pointer);

setlink(recordpointer,avail());

predecessor = recordpointer;

}

else if (*plistname != recordpointer)

{

    >Comment

setlink(predecessor,setnull());

*pdone = TRUE;

}

else

{

    >Comment

*plistname = setnull();

*pdone = TRUE;

}

}

The files list.h and list.c now embody the list data abstraction. Note that, should the format of the list change, these changes must be reflected in the header file and also in the operations file. In the later file, list.c, the changes would be localized to setinfo, getnextrecord, and printrecord. Instead, were the format to remain the same but the records stored in an array of records rather than the dynamic memory, more extensive changes would be required in both lists.h and list.c.


Data Structures Using C: 5.5: A Stack Example

5.5: A Stack Example

This last example involves a stack and its application in the nonrecursive towers program of Section 4.5.3. The stack is implemented as a list. The program treats the stack as a data abstraction and is structured so the stack and its operations appear together.

We give the header file, stack.h, the operations file, stack.c, and the program file, towerstest.c, that can be used to achieve file modularization and separate compilation.

The File stack.h

/* header for stack */

#define NULL 0

typedef struct

{

int n;

int i;

int a;

int f;

}whatever;

typedef struct record

{

whatever info;

struct record *link;

}stackrecord,*stack;

extern setstack(/* pstack */);

/* Sets stack to empty */

extern empty (/* pstack */);

/* Returns true only if

the stack is empty */

extern push(/* pstackrecord,pstack */);

/* Makes stackrecord the new

top entry on stack*/

extern pop(/* pstack,pstackrecord */);

/* Removes the top entry from

stack and returns with its

contents in stackrecord

*/

extern overflow(/* pstack */);

/* Prints a warning when

the stack overflows */

extern underflow(/* pstack */);

/* Prints a warning when the stack underflows */

The File stack.c

/* operations for the stack */

    >Comment

#include "stack.h"

setstack(ps)

/* Sets stack s to empty */

stack *ps;

{

*ps = NULL;

}

empty(ps)

/* Returns true only if

the stack s is empty */

stack *ps;

{

return(*ps == NULL);

}

push(pnewrecord,ps)

/* Makes stackrecord the new

top entry on stack s */

stackrecord *pnewrecord;

stack *ps;

{

stackrecord *newentry;

newentry = malloc(sizeof(stackrecord));

newentry->info = pnewrecord->info;

newentry->link = *ps;

*ps = newentry;

}

pop(ps,pvalue)

/* Removes the top entry from

stack s and returns with its

contents in stackrecord

*/

stack *ps;

stackrecord *pvalue;

{

if(!empty(ps))

{

pvalue->info = (*ps)->info;

*ps = (*ps)->link;

}

else

underflow(ps);

}

overflow(ps);

/* Prints a warning when

the stack overflows */

stack *ps;

{

printf (" The stack has overflowed /n");

}

underflow(ps);

/* Prints a warning when

the stack underflows */

stack *ps;

{

printf(" The stack has underflowed /n");

}

The File towerstest.c

#include <stdio.h>

main()

{

int n;

printf("\n enter a value for n \n");

scanf("%d",&n);

towers(n,1,2,3);

}

    >Comment

#include "stack.h"

towers(n,i,a,f)

/* Moves the top n disks

from peg i to peg f

*/

int n,i,a,f;

{

stack s;

int done

setstack(&s);

done = FALSE;

while(!done)

{

while(n > 1)

{

s_tack(n,i,a,f,&s);

setvar 1(&n,&i,&a,&f);

}

printf("\n %d -> %d\n",i,f);

if(!empty(&s))

{

restore(&n,&i,&a,&f,&s);

printf("\n %d -> %d\n",i,f);

setvar2(&n,&i,&a,&f);

}

else

done = TRUE;

}

setvar1(pn,pi,pa,pf)

/* Sets n to n - 1 and

interchanges f and a

*/

int *pn,*pi,*pa,*pf;

{

int t;

*pn = *pn - 1;

t = *pf;

*pf = *pa;

*pa = t;

}

setvar2(pn,pi,pa,pf)

/* Sets n to n - 1 and

interchanges a and i

*/

int *pn,*pi,*pa,*pf;

{

int t;

*pn = *pn - 1;

t = *pa;

*pa = *pi;

*pi = t;

}

s_tack(n,i,a,f,ps)

/* Creates a record containing

n, i, a, and f and inserts it

on stack s

*/

int n,i,a,f;

stack *ps;

{

stackrecord newrecord;

newrecord.info.n = n;

newrecord.info.i = i;

newrecord.info.a = a;

newrecord.info.f = f;

push(&newrecord,ps);

}

restore(pn,pi,pa,pf,ps)

/* Removes the top record from

stack s and copies its contents into n,i,a,f

*/

int *pn,*pi,*pa,*pf;

stack *ps;

{

stackrecord value;

pop(ps,&value);

*pn = value.info.n;

*pi = value.info.i;

*pa = value.info.a;

*pf = value.info.f;

}

Notice that the operations push and pop in stack.c have been written under the assumption that the C compiler allows the entire contents of one record to be copied into another record of the same type using just one assignment statement. In order to achieve greater portability, these operations could be written so that they do not depend on this assumption by using individual assignment statements for each field of the records. However, this can also be accomplished by restructuring the program. Moreover, restructuring can also achieve another level of data abstraction with respect to the stack records and their operations (in this example only one will be needed). This will require changes to stack.h and stack.c but will make the new stack module independent of the implementation of the stack records. The current stack module, consisting of the files stack.h and stack.c, must be changed and recompiled whenever the records being stacked have a different format from that which is represented by whatever. After the restructuring this will not be necessary. Any changes in the stack record format will be localized to the stack record module.

To do the restructuring, first create a module for the data abstraction stackrecord and its operations. The module consists of the files stack_record.h and stack_record.c.

The File stack_record.h

/* header for stackrecord */

typedef struct

{

int n;

int i;

int a;

int f;

}whatever;

typedef struct record

{

whatever info;

struct record *link;

}stackrecord,*stackrecordpointer;

extern setinfo(/* stackrecordpointer_1,stackrecordpointer_2 */);

/* Sets the info field of the record pointed to

by stackrecordpointer_1 to the contents of the

info field of the record pointed to by stackrecordpointer_2

*/

The File stack_record.c

/* operations for stackrecord */

    >Comment

#include "stack_record.h"

setinfo(pointer1,pointer2)

/* Copies the contents of the record

pointed to by pointer 1 into

the record pointed to by pointer2

*/

stackrecordpointer pointer1,pointer2;

{

pointer1->info.retrn = pointer2->info.retrn;

pointer1->info.n = pointer2->info.n;

pointer1->info.i = pointer2->info.i;

pointer1->info.a = pointer2->info.a;

pointer1->info.f = pointer2->info.f;

}

Here is the new stack module.

The File stack.h

/* header for stack */

    >Comment

#include "stack_record.h"

#define NULL 0

stackrecord *stack;

extern setstack(/* pstack */);

/* Sets stack to empty */

extern empty(/* pstack */);

/* Returns true only if the stack is empty */

extern push(/* pstackrecord,pstack */);

/* Makes stackrecord the new top entry on stack*/

extern pop(/* pstack,pstackrecord */);

/* Removes the top entry from

stack and returns with its

contents in stackrecord

*/

extern overflow(/* pstack */);

/* Prints a warning when

the stack overflows */

extern underflow(/* pstack */);

/* Prints a warning when

the stack underflows */

File stack.c

/* operations for the stack */

    >Comment

#include "stack.h"

setstack(ps)

/* Sets stack to empty */

stackrecordpointer *ps;

{

*ps = NULL;

}

empty(ps)

/* Returns true only if

the stack is empty */

stackrecordpointer *ps;

{

return(*ps == NULL);

}

push(pnewrecord,ps)

/* Makes stackrecord the new

top entry on stack*/

stackrecord *pnewrecord;

stackrecordpointer *ps;

{

stackrecord *newentry;

newentry = malloc(sizeof

(stackrecord));

    >Comment

setinfo(newentry,pnewrecord);

newentry->link = *ps;

*ps = newentry;

}

pop(ps,pvalue)

/* Removes the top entry from

stack and returns with its

contents in stackrecord

*/

stackrecordpointer *ps;

stackrecord *pvalue;

{

if(!empty(ps))

{

    >Comment

setinfo(pvalue,(*ps));

*ps = (*ps)->link;

}

else

underflow(ps);

}

overflow(ps);

/* Prints a warning when

the stack overflows */

stackrecordpointer *ps;

{

printf (" The stack has overflowed /n");

}

underflow(ps);

/* Prints a warning when

the stack underflows */

stackrecordpointer *ps;

{

printf (" The stack has underflowed /n");

}

As promised, the new stack modules are not dependent on the details of the stack records. The program file, towerstest.c, does not need to be changed from its original version. To run it, stack_record.c and stack.c must be compiled. Their object files, along with the previously generated object file for towerstest.c, can be linked, and then towerstest.c can be run. Any future changes in stack_record.h mean that stack_record.c, stack.c, and towerstest.c must be recompiled, but changes to only stack_record.c or stack.c will require just their recompilation. Similarly, changes to just towerstest.c require only its recompilation.

Using this technique it is possible, as demonstrated, to structure programs so that modules for data abstractions can be built using modules for other data abstractions. The examples presented, for the sake of simplicity, have dealt with small to moderate-sized programs. With programs of this size the overall impact of file modularity and modules may not appear to be great. The real power of the technique becomes quickly apparent when large programs are written by teams of programmers. The efficiency and control gained through modularity in such cases is significant.


Data Structures Using C: 5.6: Summary

5.6: Summary

Data abstraction and functional and file modularization are important concepts whose proper application to program development and program structure enhance program readability and maintainability. They aid the programmer in establishing libraries of useful functions that can be readily selected for use in creating new programs for new problems.

The ability to compile parts of a program separately can make debugging an easier task and can save time. In C this can be achieved by placing appropriate parts of a program in distinct files. When this is done it is necessary to keep track of which files must be compiled and linked in order to execute the program file.


Data Structures Using C: Exercises

Exercises

1. Why is it not necessary to place the include "collection.h" before the primes function in primestest.c?

2. Why is primestest.c independent of the implementation of collection?

3. Why must primestest.c and collection.c be recompiled when changes are made in collection.h?

4. Change preferences.h and preferences.c so that the preferences and priorities arrays are implemented using a pointer array pointing to their rows, which are stored as row records in dynamic memory (see Section 2.3.3).

5. Run stablemarriages.c with your solution to Exercise 4.

6. What must be changed in list.h and list.c if you want to add another field (say, dividends) to the list records?

7. Change the files of Section 5.4 so that the name field of the stockrecords, instead of being an array, is a list. Treat that list as a data abstraction.

8. Write and compile a module for the queue data abstraction using the circular array implementation of Section 4.6.1.

9. Write and compile a module for the queue data abstraction using the list implementation of Section 4.6.3.

10. Create a program file ordlistexample.c. The program is to do what listexample.c does, except the list that is created is to have the records appear in alphabetical order by name. The list is to be treated as a data abstraction using list.h and list.c.

11. Compile, link, and run your program created in Exercise l0.

12. Define file list_record.h and list_record.c, and modify list.h and list.c to reflect these changes. The result should make list.h and list.c independent of details of the listrecords (as was done with the stack in Section 5.5). To accomplish this, you will have to remove and modify some operations from list.c, placing their modified versions in list_record.c, and some declarations from list.h, placing their modified versions in list_record.h.


Data Structures Using C: Suggested Assignments

Suggested Assignments

1. Replace collection.h and collection.c with new files that implement collection as an array that contains the actual integers currently in the collection as its entries.

2. a. One of the two complete programs of Section 3.9.3 implements the configuration for the perfect shuffle as an array and the other implements it as a list. They do not treat the configuration as a data abstraction; write and run a single source file program that does.

b. Rewrite and run the program developed for (a) using file modularization to hide the details of the configuration's implementation and so that separate compilation can be done.

3. Change the stack implementation of Section 5.5 so that it is implemented as in Section 4.5.1. You can change only the files stack.h and stack.c; the program file, towerstest.c, cannot be changed.

4. Modify the implementation of the stack data abstraction in Section 5.5 so that instead of the stack containing records, it contains pointers to the records. Again, this should be achieved with changes only to stack.h and stack.c.


Data Structures Using C: CHAPTER 6: MORE COMPLEX LISTS

CHAPTER 6: MORE COMPLEX LISTS

Considers lists of lists connected by pointers to

impose more structure on data

reduce search time

Emphasizes

liste-structures, a special case of such lists

the traversal of list-structures as a general tool

the implementation of list-structures

Case study�information retrieval

illustrates the use and advantages of more complex lists

6.1: Imposing List Structure on Data

6.2: Traversal of List-structures

6.3: Using a Traverse Function for List-structures

6.4: Implementing List-structures

6.5: Case Study: Information Retrieval

Exercises

Suggested Assignment


Data Structures Using C: 6.1: Imposing List Structure on Data

6.1: Imposing List Structure on Data

So far in this book, records in lists have had information members and link members. These are simple records. The lists themselves have looked like chains of records. Such lists may be used to store the records only or to store them and also impose an ordering on them. This chapter presents complex records, which contain members that are themselves names of lists. The members are called sublist members. Lists composed of such records have more complex structure than chains and are known as complex lists. Several such lists are depicted schematically in Figure 6.1. The following examples illustrate the use of complex lists and show how they provide one means of organizing data to capture certain inherent relations among the list records.

Example 6.1

The accounts list in Figure 6.1(a) consists of three records. The first two are complex, the third simple. The first record points to a sublist consisting of two records; the first of these is complex and the second is simple. Accounts might actually represent three companies' user accounts. The first user account is billed for two distinct computer services, $100 for last month's computing and $75 for this month's software consulting. The second user account owes $200 for the month's computing and $250 for the month's consulting. The third user account has a credit of $46. The X in the sublist member of the complex record "200" means its sublist is null. n

Example 6.2

A programmer asked to simulate a card game with n players might choose to represent the current configuration of the n individual hands by using a list composed of complex records. A typical configuration is shown in Figure 6.1(b) for four poker players. The sublist that makes up each complex record contains all the cards of a particular player's hand, each card represented as a record. Insertions and deletions into an individual hand require insertions and deletions from the appropriate sublist. n

(a) ACCOUNTS List

(b) Poker Hands for n = 4 Players

(c) BOOK List

Figure 6.1 More Complex Lists

(d) List P, a Polynomial Expression

(e) Brokerage Data Base

Example 6.3

A book might be represented as the book list shown in Figure 6.1(c). Records for the individual lines of text would be simple; the records for paragraph, chapter,and page would be complex.

The data processing for word processing or text editing might involve inserting and deleting chapters, paragraphs, or lines. Creating an index for the book would require even more complex processing. This could be done by making a special "dictionary" that lists the key words relevant to the book's topic, then searching the text for all instances of key words. If the book were a computer science text, then the dictionary would include all key technical computer science words. The index could be created by accessing and processing each page. This processing would involve checking the page for occurrences of the key words and keeping track, for each such word in the dictionary, of the pages on which it occurred. (Unfortunately, such automatic indexing creates too large an index and one that may not be very useful, since not every mention of a key term is especially relevant.) n

Example 6.4

A polynomial in x, y, and z may be represented using a list-structure. Suppose the polynomial P(x, y, z) is 3zx5 + 5z3yx + 20. We can write this as

P(x, y, z) = z3(y(x(5))) + z(x5(3)) + 20

This, in turn, can be represented as p of Figure 6.1(d).

(a) General Form of a Record

(b) Main-List Record

(c) Last Record on Main List

Figure 6.2 Records of the List-structure for a Polynomial

Each record of p has four fields, as shown in Figure 6.2(a). A tag field value of 1 indicates that the record is complex, while a tag field of 0 indicates that the record is simple.

Each sublist, including the main list, has a header record and represents a polynomial. The main list, for example, is a polynomial in z composed of three terms. The record (Figure 6.2(b)) on the main list indicates that this term contains z to the power 3 and a coefficient that is pointed to by the pointer in the coeff field. Its successor record is similarly expressed. The last record on the main list (Figure 6.2(c)) is simple, meaning that the value in the coeff field is to be interpreted as a constant. Sublists are interpreted in the same way.n

Example 6.5

Each record on brokerage's main list in Figure 6.1(e) represents a sales representative (a financial advisor) of a brokerage firm. The sublist of each salesperson contains information about each of his or her clients. Each client's sublist is the client's stock portfolio. n

6.1.1 A Definition of List-structures


Data Structures Using C: 6.1.1 A Definition of List-structures

6.1.1 A Definition of List-structures

Each of the example lists organizes information by collecting related data on sublists. They are a step up in complexity from the chains of Chapter 3 but do not illustrate the most complex lists, which allow sharing of records and sublists. Sharing means that more than one record can point to another record or sublist. Records can also have additional pointer fields linking records to form intertwined lists. However, this chapter concentrates primarily on list-structures. They may be defined formally by either of two equivalent definitions:

A list-structure is a collection of simple and complex records linked by pointers in the following way.

The link and sublist fields of each record contain pointers to records of the collection.

Each record of the collection has no more than one pointer to it from any record of the collection.

Or recursively:

A list-structure is a list of simple and complex records whose complex record's sublist fields point to distinct list-structures called sublists. The predecessor of each record must be unique.

The first definition treats list-structures as a collection of linked records but does not convey their inherent structure. The recursive definition emphasizes this structure, making clear how a list-structure is itself built from other list-structures.

It is evident that each list of Figure 6.1 satisfies the first definition. To see that they also satisfy the recursive definition, consider the accounts list. The main list consists of a list of simple and complex records, so it is necessary to check that the complex record's sublists are themselves list-structures. The first main-list record's sublist is a list-structure since the sublist's main list satisfies the definition and its complex record points to a list-structure, a single list record. The second main-list record's sublist is itself a list whose complex record's sublist is a list-structure, the null list-structure.

(a) List with Shared Sublists

(b) List with a Loop

Figure 6.3 Two Lists That Are Not List-structures

The lists sam and sally in Figure 6.3 are not list-structures, since they fail to satisfy the definitions. Sam has shared sublists; sally has a loop.


Data Structures Using C: 6.2: Traversal of List-structures

6.2: Traversal of List-structures

Inserting or deleting a record from a list-structure is done using the same procedures as in Chapter 3. Determining where to insert or delete involves a traversal through the records to locate the desired position. Unlike the chain, for which there is a natural order in which to traverse the records, list-structures have no simple natural order. Instead, the order in which records are to be accessed and processed is defined as follows:

To traverse a list-structure,

Access each record on the main list in order and

1. Process the record

2. If the record is complex, then traverse the complex record's sublist.

Notice that the definition is actually recursive, since a complex record's sublist is itself a list-structure. To see how to apply it, consider formula in Figure 6.4. The records are numbered for ease of reference.

To determine the order in which records are accessed and processed in a traversal of the list-structure formula, apply the definition. It specifies that records 1 and 2 on the main list be accessed and processed in sequence. Since record 2 is complex, sublist 1 must be traversed. At the finish of traversal of sublist 1, it will be necessary to continue where processing left off on the main list. To know where to resume, save P1. To traverse sublist 1, apply the definition to it, and access and process records 3 and 4 in order. Since record 4 is complex, traverse sublist 2. Save P3 to know where to resume when its traversal is complete.

Figure 6.4 The List-structure FORMULA

To traverse sublist 2, apply the definition of traversal to it, and so access and process records 5 and 6. Since the link field of record 6 is null, no records remain on this sublist, so its traversal is complete.

Which task should be done now? Step 2 of the traversal definition has just been applied to sublist 2. The next record of some sublist must now be accessed. What record? It is the successor record of the complex record梤ecord 4梬hose sublist traversal was just completed. P3, which was saved for just this moment, points to record 7, so let the program follow it. Since P3 is no longer needed, discard it. Since record 7 is complex, traverse sublist 3. Continuing in this way, the program would access and process the records in the order in which they are numbered.

From this example it can be seen that whenever the link field of the record just accessed and processed is null, the traversal of some sublist has been completed and the program must determine where to go next. Each time the processing of a complex record was finished, the content of its link field was saved before traversing its sublist. In formula, this leads to saving P1, then P3. After record 6 comes a null link field signifying the completion of the traversal of sublist 2. P3 then tells the program where to go next. Once followed, it can be forgotten. Pl is still remembered, and P5, P7, and P9 need to be remembered in turn. After dealing with record 14, the program encounters its null link value, which signifies completion of sublist 4's traversal. Where does it go? It follows P9 and needs to remember only P7, P5, and P1, and so on.

Each pointer that is saved represents an obligation that must be postponed. It is the obligation to apply steps 1 and 2 of the definition of traversal of list-structures to successive records on the list pointed to by the pointer. The definition requires that the corresponding complex record's sublist be traversed immediately. In carrying out the traversal of that sublist, further obligations may be incurred and subsequently fulfilled. Each time step 2 is completed, the most recently saved pointer provides the information to determine what record to deal with next. When the record is selected for processing, the pointer is discarded. This implies that pointers saved during the complex records' traversal will be discarded when its traversal is complete. Consequently, the most recently saved pointer must point to the record to be processed each time step 2 is completed. If there is no remaining saved pointer, it means that the entire traversal is complete.

Traversing a list-structure thus requires retention, recall, and deletion of information in a specific order. The pointer information retained represents postponed obligations. These obligations must be fulfilled in reverse order from that in which they were incurred. This is the precise situation in which the stack data abstraction is very convenient. Notice how a stack can be used to store the pointers so that the list-structure can be traversed correctly. Each pointer saved is pushed on the stack and popped when needed. It is a typical LIFO situation.

It is important to recognize that the definition of a list-structure traversal does not merely explain how to traverse a specific list-structure, such as formula, but describes a procedure that may be applied to any list-structure. This is, of course, true for any algorithm or program. Programs and algorithms specify a procedure for any input, not just for a specific input. The definition of traversal of list-structures is really an algorithm, an initial description of a procedure for list-structure traversal.

6.2.1 An Iterative Function

6.2.2 A Recursive Function


Data Structures Using C: 6.2.1 An Iterative Function

6.2.1 An Iterative Function

We will now develop a program traverse to traverse a list-structure ls. In applying the definition it became clear that such a traversal may be described by the following algorithm.

1. Initialize the current record.

2. Initialize the stack of postponed obligations to "empty."

3. While there is a current record or a postponed obligation,

if there is a current record, then

a. process the current record, and

b. update the current record to the next record to be processed

else

c. remove the top obligation from the stack and update the current record to it.

To update the current record, the loop task 3b needs to be specified in greater detail. If the record just processed was simple, then the next record to be processed is the successor of the current record. Otherwise梚f the record just processed was complex梚ts successor record represents a postponed obligation that must be placed on the stack. In that case the next record to be processed is the first record on the current record's sublist. This gives a refinement for task 3b that is incorporated in the revised algorithm.

1. Initialize the current record.

2. Initialize the stack of postponed obligations to "empty."

3. While there is a current record or a postponed obligation,

if there is a current record, then

a. process the current record, and

b. if the current record is complex, then

i. save an obligation to its successor record on the stack

ii. update the current record to the first record on its sublist,

else

update the current record to its successor record

else

c. remove the top obligation from the stack and update the current record to it.

A variable (call it ptr) is needed to contain a pointer to the current record. The stack can be treated as a data abstraction with the usual operations set-stack, empty, push, and pop as was done in Chapters 4 and 5. The details for implementing the stack are built into the routines implementing these operations.

Assume that during the traverse each record is processed by calling a routine process. It will clearly need the pointer to the record to be processed, ptr. It will also need access to the list name ls . For example, if its task involves deleting the first record of ls, then it must be able to change the contents of ls. To treat the list-structure as a data abstraction, we need the basic functions complex, next, and sublist, each with ptr as a parameter. Complex returns a value true if the record to which ptr points is complex, the value false otherwise. Next and sublist return, respectively, a copy of the link and sublist field value of the record pointed to by ptr.

A more detailed refinement of the algorithm can now be written.

1. Set ptr to ls

2. Setstack(s)

3. While ptr is not null or not empty(s)

if ptr is not null, then

a. Process(ls,ptr)

b. If complex(ptr), then

i. Push(next(ptr),s)

ii. Set ptr to sublist(ptr)

else

Set ptr to next(ptr)

else

c. Pop(s,ptr)

Notice that if all records of ls are simple, then tasks (i), (ii), and (c) will never be invoked. This procedure is then essentially the same as the list traversal of Chapter 3, as it should be. As written, it does not provide for partial traversal (done is not involved in the loop control) but could be easily modified when necessary.

Traverse may be implemented as in the following iterative (nonrecursive) function. This implementation uses data abstraction and has been written using functional modularly.

Nonrecursive Version

traverse (pls)

/* Accesses and processes each record

of the list-structure ls.

*/

liststructurepointer *pls;

{

liststructurepointer ptr,null,setnull(),next(),sublist();

>Comment

stack s;

null = setnull();

>Comment

ptr = *pls;

    >Comment

setstack(&s);

    >Comment

while((ptr ! = null) | | (!empty(&s)))

    >Comment

if(ptr ! = null)

{

    >Comment

process (pls, ptr);

    >Comment

if (complex(ptr))

{

    >Comment

push next(ptr),&s);

ptr = sublist(ptr);

}

else

    >Comment

ptr = next(ptr);

}

else

    >Comment

pop(&s,&ptr);

}

It is important to see that the program is functionally modular and independent of the implementation of the data abstractions to which it refers. Data abstractions for the stack with its operations, and for the list-structure and its operations (which must include complex, next, sublist, and setnull) are needed in order to use traverse. The programmer must either be given these or else must write them.


Data Structures Using C: 6.2.2 A Recursive Function

6.2.2 A Recursive Function

A recursive implementation can also be developed for traverse. Begin with a recursive algorithm for traversing a list-structure; this is obtained directly from the recursive definition of a list-structure traversal.

1. Initialize the current record.

2. While there is a current record,

a. process the current record;

b. if the current record is complex, then

i. traverse the current record's sublist;

c. update the current record to its successor record.

Once more we need the variable ptr to point to the current record, and the variable nextsublist to point to a sublist. Again, the list-structure is treated as a data abstraction. Also, to modularize functionally, assume that a function traverse is available to implement task (i). The algorithm can now be refined:

1. Set ptr to ls

2. While ptr is not null,

a. Process z(ls, ptr);

b. if complex (ptr), then

i. Set nextsublist to sublist (ptr),

traverse (nextsublist);

c. Set ptr to next (ptr).

This refinement is implemented in the following recursive procedure.

Recursive Version

traverse (pls)

/* Accesses and processes each record

of the list-structure ls.

*/

liststructurepointer *pls;

{

liststructurepointer ptr,nextsublist,null,

setnull(),next (),sublist();

null = setnull();

>Comment

ptr = *pls;

>Comment

while(ptr != null)

{

>Comment

process(pls,ptr);

>Comment

if(complex(ptr))

{

>Comment

nextsublist = sublist(ptr);

>Comment

traverse(&nextsublist);

}

>Comment

ptr = next(ptr);

}

}

Notice that this program, at the point when it encounters a complex record and hence must traverse its sublist, does so by simply invoking a module that does that traversal梚tself. After that traversal is complete, it merely resumes where it left off. In contrast, the nonrecursive version must perform its own bookkeeping to keep track of where to resume each time it encounters a complex record and then retrieve that information when it is needed.

A number of comments are in order about the two implementations of traverse. First, imagine that the recursive procedure traverse is presented to you with the statement that it is a refined version of the definition of traversal of a list-structure. It is easy to see that this is so. Contrast this with the effort involved in seeing that the nonrecursive version of traverse is also a refined version of the same defining algorithm. Second, the recursive version is more concise than the nonrecursive. Third, the recursive traverse is almost a direct translation of the defining algorithm. It required very little effort to achieve it in this case, because the definition is itself a recursive description.

These comments usually hold for algorithms that process data structures that can be defined recursively, such as list-structures. This is not an accident but can be expected when we deal with complex data structures. Recursive definitions allow concise descriptions of complex algorithms and data structures. Concise descriptions allow us to deal more easily with their complexity. ln fact, it is difficult to develop a definition for a list-structure traversal that has no vagueness, is applicable to any list-structure, and does not involve recursion. Recursive definitions are especially useful to the programmer dealing with complex data structures, because they lead systematically and directly to recursive programs for their processing.


Data Structures Using C: 6.3: Using a Traverse Function for List-struc

6.3: Using a Traverse Function for List-structures

Use of the general list-structure traversal functions just developed is illustrated in the following examples. The examples are intended to convey the flavor of list-structure applications and their processing.

Example 6.6

Print the information field of each stock record of the list-structure brokerage (shown in Figure 6.1(e)) which was bought prior to a specified date and which we store in date. A solution may be obtained by traversing brokerage and printing the relevant stock records. To do this requires only that we write the process function invoked by traverse. Either the recursive or the nonrecursive version may be used. The function process must be written to turn traverse into a solution to this problem.

The task is straightforward except for the determination, within process, of whether the record it is to process is currently a stock record. Since only stock records are simple, a test for complexity of the current record will provide this nformation. Process may be written as where prior returns true if datebought of the record pointed to by ptr is prior to date, and false otherwise, and printinfo prints the information field value of the record pointed to by ptr. n

process (pls, ptr)

/* Print the stock record of ls

pointed to by ptr if it was

bought prior to date

*/

liststructurepointer *pls, ptr;

{

>Comment

if(!complex(ptr) && prior(ptr,&date))

printinfo (ptr);

}

Example 6.7

Consider again the list-structure brokerage in Figure 6.1(e). The task now is to print, for each client, the total value of his or her portfolio. n

In detail, what has to be done is

1. To traverse brokerage, and each time a client's record is encountered

2. Print the name

3. Then accumulate the total value of the portfolio as it is being traversed

4. Print this total when the portfolio has been completely traversed

While this seems straightforward and simple enough, some details require attention. We can apply the traverse function to this task. To do so, process must be written to turn traverse into a solution. Thus process must be able to recognize when the record it is to work on is a client's record. At this point, it must initialize the accumulated total to zero. For each ensuing record of the client's portfolio, it must then update the accumulated total by the value of that record's stock. Finally, process must recognize the end of the portfolio list and print the accumulated total.

If total is used to store the accumulated value, then it must be preserved between calls to process. This can be achieved by making total a static variable in process or by making total global. We choose the former. Process may be written as follows.

process(ptr)

/* If the current record points to a

client's portfolio record that client's

total portfolio value is updated.

If the record is the last record on

the client's portfolio list its total

value is printed.

*/

liststructurepointer null,setnull(),

sublist(),next();

float value();

>Comment

static float total;

null = setnull();

>Comment

if(complex(ptr))

if(sublist(ptr) != null)

if(!complex(sublist(ptr)))

{

    >Comment

total = 0.0;

    >Comment

printname(ptr);

}

    >Comment

else

{

    >Comment

total = total + value(ptr);

    >Comment

if(next(ptr) == null)

printf("\n Portfolio value is

%12.2f \n",total);

}

}

The if conditions are used to determine if this is the first call to process for a portfolio and, if so, to initialize and print the client's name or, after updating a client's total, to see if the record is the last of a portfolio. Assume null is nonlocal to process. Process will work with either version of traverse.

Example 6.8

Suppose book, the list-structure shown in Figure 6.1(c), stores the current version of a text that you wish to modify by replacing the current paragraph p of page pg of chapter c with a new paragraph. The new paragraph is stored as a list newp. To solve this problem we will write a new nonrecursive traverse function, called paragraphinsert, tailored to the situation. Since not all records of book need to be accessed, we do not want to traverse the entire list-structure as we have done in the previous two examples. Two versions of the function are presented. In both versions the idea is to

1.     Traverse the main list searching for chapter c

2.     Traverse that chapter's sublist searching for page pg

3.     Traverse that page's sublist searching for paragraph p

4.     Finally, replace that paragraph with newp                         n

The first version assumes that records representing chapters, pages, or paragraphs have a number field. In the case of a chapter record the field contains the number of that chapter; for a page record it contains the number of that page; and for a paragraph it contains the number of that paragraph. These field values are shown in Figure 6.1(c). The program keeps track of whether it is currently looking for a chapter, a page, or a paragraph by using a variable currentlist. The value stored in currentlist will be CH (denoting that the search is now for a chapter), PAGE (denoting that the search is for a page), or PAR (denoting the search is for a paragraph) . Similarly, a variable currentnumber will keep track of the particular chapter, page, or paragraph number currently being sought by the search. A variable ptr keeps track of the record currently being processed. Once the search for a chapter or a page is complete, currentlist, currentnumber, and ptr must be updated. Once the search for a paragraph has completed, the searching terminates and the new paragraph replaces the old.

Version 1

    >Comment

#define CH 1

    >Comment

#define PAGE 2

    >Comment

#define PAR 3

#define TRUE 1

#define FALSE 0

paragraphinsert(book,newp,c,pg,p)

/* Replaces paragraph p of page pg

of chapter c of book by the

paragraph pointed to by newp.

Assumes the records of book contain

number fields.

*/

liststructurepointer book,newp;

int c,pg,p;

{

int currentnumber,currentlist,found;

liststructurepointer null,ptr,setnull(),next(),sublist();

null = setnull();

    >Comment

ptr = book;

    >Comment

currentnumber = c;

    >Comment

currentlist = CH;

found = FALSE;

    >Comment

while(!found && (ptr != null))

{

    >Comment

if(currentnumber != number(ptr))

ptr = next(ptr);

    >Comment

else

    >Comment

switch(currentlist)

{

    >Comment

case CH:

    >Comment

currentlist = PAGE;

currentnumber = pg;

ptr = sublist(ptr);

break;

    >Comment

case PAGE:

    >Comment

currentlist = PAR;

currentnumber = p;

ptr = sublist(ptr);

break;

    >Comment

case PAR:

found = TRUE;

}

}

    >Comment

if(found)

    >Comment

setsublist(ptr,newp);

    >Comment

else

printf("\n Paragraph not found \n");

1 print message

}

Setsublist sets the sublist field of the record pointed to by its first parameter to the value of its second parameter. Just before setsublist is invoked, code (not shown here) could be inserted to reclaim the storage used by the replaced paragraph. This is also true for the second version.

The second version assumes no number field, so the records are indistinguishable. Hence they must be kept in order�that is, chapters are in order on the main list, pages are in order on their lists, etc. Note that this ordering is not required by the first version. Now a count of list records must be maintained, to determine when the appropriate record number has been encountered. In the second version below only the differences between it and the first version have been annotated. Notice that this second program requires less storage for the list-structure (as the number fields are not needed), but it does somewhat more processing. Also, the sublists must be kept in order. This is another example of trading storage for time.

Version 2

#define CH 1

#define PAGE 2

#define PAR 3

#define TRUE 1

#define FALSE 0

paragraphinsert(book,newp,c,pg,p)

/* Replaces paragraph p of page pg

of chapter c of book by the

paragraph pointed to by newp.

Assumes the sublists of book are

in order.

*/

liststructurepointer book,newp;

int c,pg,p;

{

int currentnumber,count,currentlist,found;

liststructurepointer null,ptr,setnull(),next(),sublist();

null = setnull();

ptr = book;

currentnumber = c;

currentlist = CH;

found = FALSE;

    >Comment

count = 1;

    >Comment

while(!found && (ptr != null))

{

    >Comment

if(currentnumber != count)

{

    >Comment

count++;

ptr = next(ptr);

}

else

switch(currentlist)

{

case CH:

currentlist = PAGE;

currentnumber = pg;

    >Comment

count = 1;

ptr = sublist(ptr);

break;

case PAGE:

currentlist = PAR:

currentnumber = p;

    >Comment

count = 1;

ptr = sublist(ptr);

break;

case PAR:

found = TRUE;

}

}

if(found)

setsublist(ptr,newp);

else

printf("\n Paragraph not found \n");

}

Example 6.9

Consider again the list-structure formula shown in Figure 6.4. It may be interpreted as representing the formula   [image]. This happens to be the root of the quadratic equation ax2 + bx + c = 0. In formula the symbols QT, PS, TS, MS, DIFF, and SQ stand for the operations of division, addition, multiplication, changing sign, taking the difference, and taking the square root, respectively. The square root and sign change operations apply to one operand. The others apply to two operands. The main list (and each sublist) represents an arithmetic expression. Their first records contain an arithmetic operator such as QT, PS, TS, MS, and DIFF. Their remaining list records represent the operand values of that arithmetic operator. The expression represented by such a list is to be evaluated by applying the operation to its operands. Thus sublist 4 would be evaluated by multiplying the operands in records 13 and 14 (b and b respectively), together. n

Notice that each complex record's sublist is an arithmetic expression that must be evaluated. Thus the operand each represents must be obtained before the operator represented by the list on which it appears may be evaluated. For example, the sublists of complex records 2 and 22 must be evaluated before QT of record 1 can be applied to them.

The task is to write a function evaluate to return the value of an arithmetic expression represented by a list-structure. As with formula, the evaluation can be accomplished by traversing the expression's main list and, whenever a complex record is encountered, evaluating its sublist. When the last list record has been processed in this way, the operator and each of its operands are known. The final result is obtained by applying the operator to its known operands. Of course, in evaluating an operand, more complex records may be encountered, such as record 4, and they must then be evaluated before continuing. A process function can be written to turn either the recursive or the nonrecursive traversal program into evaluate.

One important detail remains: how to store the operands that are generated during the traversal so that they are available when needed. Suppose each operand encountered by process is stored in an operand stack, and each operator encountered is stored in an operator stack. If the current record being processed is the last record of a list (its link value is null), then the operator currently at the top of the stack will have its operands at the top of the operand stack. Process can then pop both these stacks, apply the operator to the operands, and place the result back onto the operand stack. If the record being processed represents the last record on the main list (the operator stack will then be empty), instead of placing the result on the operand stack it should copy it into evaluate.

A detailed description of process follows.

If next(ptr) is not null and the record pointed to by ptr is simple, then

if it contains an operator, then

place the operator on the operator stack

else

place the operand on the operand stack

else

pop the operator stack, remove the top k operands from the operand stack (k is

the number of operands of the operator just popped), and apply the operator to

the k operands.

If the operator stack is not empty then

push the result onto the operand stack

else

set evaluate to the result.

This solution may also be applied to the evaluation of logical or boolean expressions. In fact, it forms the basis of the interpretive language LISP.

We have now seen a number of applications of traverse to examples involving list-structures. The example, presented next, is not discussed in detail but points out another possible application.

Example 6.10

Suppose polynomials are represented as p is in Figure 6.1(d). It is then possible to write functions to perform important algebraic operations on polynomials such as the addition and multiplication of two polynomials. Even symbolic differentiation can be performed. Note that traversal of the list-structures representing the polynomials would form the heart of such functions. n


Data Structures Using C: 6.4: Implementing List-structures

6.4: Implementing List-structures

The graphic depictions of list-structures mirror the conceptual image of them. To implement list-structures in a language such as C, you may store their records in dynamic memory or in an array of records. Wherever the records are stored, it must be possible to distinguish simple records from complex ones. The main difference in the format of the records, compared to those of a chain, is that an additional field becomes necessary for the sublist pointer when the record is complex. We also assume that the record contains a field whose value will establish whether the record is simple or complex. The actual implementation of a list-structure record may be done in three ways:

1. Declare two types of records:

one to be used for simple records

another to be used for complex records

2. Declare one record type that consists of a fixed part and a union, which has

one format for simple records

another format for complex records

3. Declare one record type, used for both simple and complex records, but containing a kind or tag field to indicate whether the record stored is simple or complex

The first way is undesirable because it requires that a program deal with two distinct pointer types, one for simple records and another for complex records, so we dismiss it. The second method is feasible and also requires a kind field, which we can put in the fixed part. However, even though the two distinct union formats may have significant differences in the amount of storage they require, C will always allocate the larger amount. Thus this method does not save storage over the third way, which we now choose to use.

To implement the accounts list-structure of Figure 6.1(a), the following definitions could be used when the records are to be stored in dynamic memory.

>Comment

typedef struct record

{

int kind;

int info;

struct record *sublist;

struct record *link;

}listrecord, *liststructurepointer;

When the records are to be stored in an array of records, the implementation should be changed to

>Comment

typedef struct record

{

int kind;

int info;

>Comment

int sublist;

>Comment

int link;

}listrecord;

>Comment

typedef int liststructurepointer;

This change is needed because now listpointer, link, and sublistptr will each be an integer index into the array of records rather than a pointer variable containing an absolute memory address.

As a second illustration, consider the records for the polynomials of Figure 6.1(e). Notice that a complex record in that list-structure has two possible formats, depending on whether the record is a header or not. If it is a header record, the second field contains a character; otherwise it contains a sublist pointer. Using a union, the polynomial records can be implemented as follows.

>Comment

typedef union

{

char unknown;

polyrecord *sublist;

int value;

}varying;

typedef struct record

{

int tag;

varying secondfield;

int exponent;

struct record *nextterm;

}polyrecord, *liststructurepointer;

The tag field indicates whether the record is simple or complex. In this implementation the union allows the same storage to be used to hold the unknown (for a header record) or a sublist pointer. It was used here because a complex record has two distinct possible second fields and the union allows them to share the same storage. The alternative is to avoid the use of the union and keep separate fields for both the unknown and the sublist pointer in each record. This just wastes storage.

When the records are to be stored in an array of records, the implementation should be changed to

    >Comment

typedef union

{

char unknown;

    >Comment

int sublist;

int value;

}varying;

typedef struct

{

int tag;

varying secondfield;

int exponent;

    >Comment

int nextterm;

}polyrecord;

    >Comment

typedef liststructurepointer;

It is also possible to implement the main list and sublists of a list-structure as two-way lists. This is convenient, just as for chains, when it is necessary to go back and forth during the processing of a list-structure or to make the predecessor of any record easily accessible. In this case an additional field would be needed for the backward pointer.

The following program creates and prints the records of a list-structure to illustrate its implementation with records stored in dynamic memory. It also demonstrates two applications of the traverse function. The input consists, first, of an integer to indicate whether the list being created is null, and, second, a sequence of data for each record. This sequence is assumed to be in the order in which the records are accessed in a traversal of the liststructure. For example, input corresponding to accounts of Figure 6.1(a), would be as follows. (Program output is not shown, nor are program prompts for input.)

1    --  indicates a nonnull list-structure

175  --  information field value

1    --  indicates record is complex

11   --  indicates the link and sublist fields are not null

0    --  information field value

1    --  indicates record is complex

11   --  indicates the link and sublist fields are not null

100  --  information field value

0    --  indicates record is simple

0    --  indicates the link field is null

75   --  information field value

0    --  indicates record is simple

0    --  indicates the link field is null

450  --  information field value

1    --  indicates record is complex

11   --  indicates the link and sublist fields are not null

200  --  information field value

1    --  indicates record is complex

10   --  indicates the link field is not null but the sublist field is

250  --  information field value

0    --  indicates record is simple

0    --  indicates the link field is null

-46  --  information field value

0    --  indicates record is simple

0    --  indicates the link field is null

The Program

Reads in list-structure records,

Creates the list-structure, and

Prints all its records.

#include <stdio.h>

    >Comment

typedef struct record

{

int info;

int kind;

struct record *sublistptr;

struct record *link;

}examplerecord,*liststructurepointer;

    >Comment

#define NULL 0

    >Comment

liststructurepointer next(p)

/* Returns a copy of the link field

value of record pointed to by p.

*/

liststructurepointer p;

{

return(p->link);

}

liststructurepointer sublist(p)

/* Returns a copy of the sublistpointer

field value of record pointed to by p.

*/

liststructurepointer p;

{

return(p->sublistptr);

}

liststructurepointer setnull()

/* Returns a null pointer */

{

return(NULL);

}

liststructurepointer avail()

/* Returns a pointer to storage allocated

for a liststructure record of type

examplerecord.

*/

{

return(malloc(sizeof(examplerecord)));

}

complex(p)

/* Returns true only if the record pointed

to by p is complex.

*/

liststructurepointer p;

{

    >Comment

return(p->kind == 1);

}

createrecord(l,ptr)

/* Reads in the data for the current record

and sets its fields appropriately.

*/

liststructurepointer 1,ptr;

{

int link,subptr,value,kind;

liststructurepointer setnull(),avail();

printf("\n enter info \n");

    >Comment

scanf("%d",&value);

printf("\n enter 0 for simple 1

for complex \n");

    >Comment

scanf("%d",&kind);

    >Comment

if (kind == 0)

{

printf("\n enter link \n");

    >Comment

scanf("%d",&link);

    >Comment

if (link == 0)

    >Comment

ptr->link = setnull();

    >Comment

else

    >Comment

ptr->link = avail();

}

    >Comment

else

{

printf("\n enter link & subptr \n");

    >Comment

scanf("%d %d",&link,&subptr);

    >Comment

if (link == 0)

    >Comment

ptr->link = setnull();

    >Comment

else

    >Comment

ptr->link = avail();

    >Comment

if (subptr == 0)

    >Comment

ptr->sublistptr = setnull();

    >Comment

else

    >Comment

ptr->sublistptr = avail();

}

    >Comment

ptr->info = value;

    >Comment

ptr->kind = kind:

}

printrecord(pls,ptr)

/* Prints the contents of the info field

of the record pointed to by ptr.

*/

liststructurepointer *pls,ptr;

{

printf("\n %d \n",ptr->info);

}

main()

/* Reads in liststructure records, creates

the liststructure, and prints all its records.

*/

{

int n;

    >Comment

liststructurepointer l,avail();

printf("\n enter liststructure: 0 for\

null list 1 otherwise \n");

scanf("%d",&n);

printf("\n remember - the data for the\

records must be in traversal order\

\n");

    >Comment

if (n == 0)

    >Comment

l = NULL;

    >Comment

else

{

    >Comment

l = avail()

    >Comment

create(1);

}

    >Comment

traverse(&l);

    >Comment

if (l == NULL)

printf("\n The list is null \n");

}

    >Comment

#define LIMIT 50

typedef liststructurepointer whatever;

typedef struct

{

whatever stackarray[LIMIT];

int top;

}stack;

    >Comment

setstack(ps)

/* Sets stack s to empty */

stack *ps;

{

(*ps).top = -1;

}

empty(ps)

/* Returns true only if

stack s is empty.

*/

stack *ps;

{

return((*ps).top == -1);

}

push(value,ps)

/* Inserts contents of value as

top entry of stack s.

*/

whatever value;

stack *ps;

{

if ((*ps).top == (LIMIT - 1))

overflow(ps);

else

{

(*ps).top = (*ps).top + 1;

(*ps).stackarray[(*ps).top] = value;

}

}

pop(ps,pvalue)

/* Removes the top entry of stack s

and places its contents in value.

*/

stack *ps;

whatever *pvalue;

{

if (empty(ps))

underflow(ps);

else

{

*pvalue = (*ps).stackarray[(*ps).top];

(*ps).top = (*ps).top - 1;

}

}

overflow(ps)

/* Prints a message when

the stack overflows.

*/

stack *ps;

{

printf("\n stack overflow \n");

}

underflow(ps)

/* Prints a message when

the stack underflows.

*/

stack *ps;

{

printf("\n stack underflow \n");

}

    >Comment

traverse(pls)

/* Prints the info fields of all

records of the liststructure l.

*/

liststructurepointer *pls;

{

liststructurepointer ptr,null,setnull(),

next(),sublist();

stack s1;

null = setnull();

ptr = *pls;

setstack(&s1);

while ((ptr != null)||(!empty(&s1)))

if (ptr != null)

{

    >Comment

printrecord(pls,ptr);

if (complex(ptr))

{

push(next(ptr),&s1);

ptr = sublist(ptr);

}

else

ptr = next(ptr);

}

else

pop(&s1,&ptr);

}

    >Comment

create(l)

/* Inputs records and creates the list l */

liststructurepointer l;

{

liststructurepointer ptr,null,setnull(),

next(),sublist();

stack s2;

null = setnull();

ptr = l;

setstack(&s2);

while ((ptr != null)||(!empty(&s2)))

if (ptr != null)

{

    >Comment

createrecord(l,ptr);

if (complex(ptr))

{

push(next(ptr),&s2);

ptr = sublist(ptr);

}

else

ptr = next(ptr);

}

else

pop(&s2,&ptr);

}

Although the stacks have been given distinct names, since s1 is local to create and s2 is local to traverse, they could have both been named s. Note that either create or traverse could be replaced by recursive versions.

6.4.1 Sharing Storage


Data Structures Using C: 6.4.1 Sharing Storage

6.4.1 Sharing Storage

List-structures that share list records as well as sharing sublists are frequently needed to fit the demands of a problem. These structures are more complicated than those presented thus far and may require large amounts of storage for their implementation. A natural idea is to avoid the duplication of identical sublists or records. At times it is convenient to use additional fields linking records or sublists so that the resultant structure contains records with more than one pointer to them, and the structure may even contain loops. That is, by following pointers from a record we can arrive back at the record. You will see examples of this in the case study that follows. Whenever these possibilities occur, the structures created are still lists but are no longer list-structures. These more general lists provide extensive flexibility but lead to difficult storage management problems, which are considered later in the text.


Data Structures Using C: 6.5: Case Study: Information Retrieval

6.5: Case Study: Information Retrieval

We now present a case study to illustrate the versatility of lists intertwined with pointers. When the name of a record or a pointer to it is known, retrieval of the record is simple. It may be directly accessed and processed to suit our needs. Slightly more difficult is the retrieval of a record that can be recognized when encountered but whose location is unknown. In order to retrieve it the records must be traversed until the desired one is encountered.

When large collections of records are involved, traversing all the records to find one is not feasible. The problem is not so much that retrieving one desired record takes too long but that many retrievals in rapid succession cause too much delay for the later retrievals. Imagine 1,000 computer terminals linked to a central computer. The banking terminals known as automatic tellers are an example. Suppose 200 requests occur in rapid succession for the retrieval of records. If 1 million records need to be traversed, the retrieval of an individual record could easily take a second, so the last of the 200 requests to be processed would be honored only after more than 3 minutes of elapsed time. No one wants to wait at a terminal for 3 minutes before each request is answered. In the meantime, even more requests become backlogged.

The basic technique to avoid traversing all stored records is to focus the search on a smaller group of records known to contain the desired one. This is the essence of most faster retrieval algorithms. In this case study, lists will be used to organize the data so that the search may be localized in this way. In Chapters 8 and 9 the search problem is presented in a more general context for information stored in random access internal memory, and in Chapter 10 for information stored in external memory.

6.5.1 Family Relationships

6.5.2 A First Solution

6.5.3 A Better Solution

6.5.4 Updating the System

6.5.5 Retrieving Information from the System

6.5.6 Other System Components

6.5.7 Case Study Overview


Data Structures Using C: 6.5.1 Family Relationships

6.5.1 Family Relationships

The central problem in this case study concerns information about a collection of individuals. The collection might represent everyone who graduated from a specific university, all registered voters in a given region, or all individuals known to have contracted a particular disease that is being researched.

The input information for each individual is given as a record with the individual's name, sex, birthdate, father's name, and mother's name as fields. Other relevant fields may be present but are ignored for our purposes here.

To simplify, assume that no duplicate records appear, no two individuals have exactly the same name, all siblings have the same parents, all parents are married, there are no divorces, and no one remarries. This is truly a noteworthy collection of people living in an idyllic world! The violation of these assumptions would create problems that could not be ignored. However, their consideration would detract from our immediate aims.

The task is to input a sequence of these records and represent the collection in memory so that questions involving certain relationships among individuals may be answered. Examples of possible questions are

How many children does John Smith have?

Who are the uncles of John Smith?

Who are the siblings of John Smith?


Data Structures Using C: 6.5.2 A First Solution

6.5.2 A First Solution

Perhaps the most straightforward approach to a solution for this information retrieval problem is to represent the collection of information by its image in memory as an array of records. To answer the question, "How many children does John Smith have?" requires a simple traversal of the records. Have a program look at the father's name field in each record and, whenever it comes to John Smith, increment a counter that represents the number of John Smith's children. When the traversal is complete, the program would print the counter value. To answer the question, "Who are the uncles of John Smith?", the program must first traverse, looking at the name field of each record, to find John Smith. It would note the father's name, which is in the fathersname field of that record. Then it would traverse again, looking at the fathersname fields. Each time John Smith's father's name appears, and "male" is listed as the corresponding sex field value, the program must print the name that appears in the corresponding name field. The same sequence would be repeated for John Smith's mother.

The execution time for these traversals can be proportional to the total number of records. This makes the solution time proportional to the amount of information represented in the system. For large systems, this would be impractical, as noted earlier.

What is the least amount of time that could be taken by any solution for these kinds of questions? Clearly, identifying the brothers of John Smith requires printing the names of the brothers. To display who the uncles of John Smith are, the program must print the names of each of his uncles. The least amount of time required for any solution would thus be proportional to the length of the answer to any given question. Length in the case of the uncles is the number of uncles. In the next section, you will see that a different way of representing the information allows a solution that is closer in execution time to this length. This is much better than having it proportional to the entire amount of information stored.


Data Structures Using C: 6.5.3 A Better Solution

6.5.3 A Better Solution

To find answers to relationship questions quickly, you must be able to localize the search and get to the relevant records without considering all of them. You can, if you somehow link records together in a meaningful way. The problem deals with family relationships. Suppose you set up a record in memory to correspond to each individual name in the system, each record having seven fields, five of the fields containing pointer information. The seven fields of each individual's record will be nameptr, fnameptr, mnameptr, siblingptr, childrenlist, sex, and birthdate.

You will also have a table of names, called nametable. Each record of nametable will have a name and a recordptr field. The recordptr field contains a pointer to the individual record of the person whose name appears in the name field. By searching the nametable for a name, we get the pointer to the corresponding individual's record. Following the trails (pointers) from the record, we can retrieve the data needed to answer questions.

Figure 6.5 depicts the form of the two kinds of records. In memory the collection of nametable records will be stored separately from the collection of individual records.

The memory would appear as shown in Figure 6.6, after the data has been input. The childrenlist field of a record f contains a pointer to the first record on the list of children of the individual represented by f. The siblingptr field plays the role of a link field for records on the list of siblings of f. The different lists are intertwined as shown in Figure 6.7.

In Figure 6.7 records f and m are husband and wife, so their children-list pointers are identical. It might be convenient to keep the lists of children in order by birthdate or to make them circular, so that the null pointer signifying the last record would instead point to the first record of the list.

(a) Nametable Record

(b) Individual's Record

Figure 6.5 Format of the Nametable and Individual Records

If the assumptions about an "ideal" world were violated, it would be possible to have shared records, and loops could even appear. With the assumptions, the lists are list-structures. The main list of any sublist consists of siblings. All records are complex, with sublist fields (childrenlist fields) pointing to sublists representing children of the individual represented by the complex record. Sublist sharing, however, does occur because the fathers and mothers share the lists of children.

Figure 6.6 Memory in the Family Relationships Problem


Data Structures Using C: 6.5.4 Updating the System

6.5.4 Updating the System

How are the collections of individual records and of nametable records created originally? Suppose you have already dealt with the first hundred records and that the situation in memory accurately reflects all the information they contain. That is, the two collections are correct and as complete as possible, given this input. You now attempt to input the next record, the one hundred first. Recall that the

input consists of the individual's name, sex, birthdate, father's name, and mother's name. Input data for John Smith would appear as

JOHN SMITH  M  1/2/79  GEORGE SMITH  ALICE RICKARD

The program should check the record for validity (input validation). If valid, the two collections in memory must be updated to reflect the new information. Then the program should read the next record, and so on until there are no more to enter. An important component of the solution is an update function, which will create and add new records to the collections when necessary and enter new values into the proper fields. It must function as follows.

Figure 6.7 Typical Configuration of Individual Records

It is possible that one or more of the three names in the name fields of the newly input record are already in the nametable, with individual records having been created for them. For instance, the father's name might have been dealt with earlier when it appeared in the name field of another record. Assume first that each distinct name appearing in any field of the previously processed records has been entered into the nametable and has had an individual record created for it. Thus the nametable entry for each of these distinct names has the correct name in its name field, and a pointer in its recordptr field to its corresponding individual record. This individual record also has a pointer in its name field back to the record in the nametable it represents. Some fields of the individual record may not yet have been filled with appropriate information.

We cannot, however, assume that any name in the new input record has been seen before. Suppose John Smith is in the name field of the new record. Search the nametable to see if there is an entry for John Smith. If there is, remember where it is by placing a pointer to it in nt_ptr and then copy the pointer in the recordptr field into a variable ir_nptr. Then ir_nptr reveals where the individual record is that corresponds to John Smith. Nt_ptr is a pointer to John Smith's nametable record, and ir_nptr is a pointer to John Smith's individual record. Ir_nptr will be used later when we have to update John Smith's individual record. If there is no entry for John Smith in the nametable, you know that there is not yet an individual record created for John Smith. Create such a record, and also create an entry in the nametable corresponding to it. Store John Smith's name and a pointer to his corresponding record in the recordptr field of the nametable entry. Also place a copy of the pointer to this entry in the nameptr field of the individual record. Then copy the pointer to the individual record into ir_nptr.

At this point, whether or not information on John Smith was already in the system, both an entry in the nametable and an individual record for him exist. The nametable record is completely filled in, and the individual record has at least its nameptr field filled in. Also, ir_nptr points to the individual record.

We would follow the same process for John Smith's father and mother, creating the following four pointers.

ir_fptr

A pointer to the individual record representing John Smith's father

ir_mptr

A pointer to the individual record representing John Smith's mother

nt_fptr

A pointer to the nametable entry for John Smith's father

nt_mptr

A pointer to the nametable entry for John Smith's mother

How do we now complete the update properly? That is, which individual records and nametable entries must be modified, and how are the modifications done?

Consider the individual record for John Smith. The sex and birthdate fields can easily be filled in, as well as the fnameptr field, copying nt_fptr into it.The same can be done for mnameptr.

Now the individual record for the father of John Smith must be updated. Fill in the sex field (you know he is male). There is no additional information on his birthdate. (It may already be known. Why?) John Smith's record must be added to the list of children of his father. Assuming that the children are not kept in any special order, add his record at the front of the list. This requires copying the childrenlist pointer of his father's record into the siblingptr of John Smith's record, and then copying the pointer in ir_nptr into the childrenlist field of his father. This implies that whenever individual records are created, they must have their childrenlist and siblingptr fields set to null.

The mother's record can be treated similarly, except we may simply copy the father's childrenlist pointer (or ir_nptr) into her childrenlist field. This completes the update, which consisted of a search of the nametable and the updating of pointers in nine fields after the nametable entries and individual records were created. These nine fields were

fnameptr, mnameptr, sex, birthdate, siblingptr梖or the individual record of John Smith

sex, childrenlist�for the individual record of George Smith and Alice Rickard, the father and mother

Example 6.11

Let us now trace this process with some data and show how the records would appear. Suppose the input data so far consisted of the information shown in Table 6.1, and suppose that we use different arrays to hold the individual records and the nametable entries. (Later, in Chapters 8 and 9, more appropriate implementations for collections such as nametable are introduced.) The situation in memory will then be as shown in Table 6.2. n

The next input is

JOHN SMITH  M  1/2/79  GEORGE SMITH  ALICE RICKARD

Since none of these names appears in the nametable, an entry in the nametable and an individual record must be created for each of the names.

After this has been done, the new nametable entries, individual records, and saved pointers will appear as shown in Figure 6.8(a). Updating the nine fields of the individual records yields the result shown in Figure 6.8(b).

Table 6.1 Input Data for Example 6.11

       Name       Sex  Birthdate    Fathersname             Mothersname

-----------------------------------------------------------------------------

Albert Einstein    M   3/14/1879  Hermann Einstein       Paulina Koch

Leo Tolstoy        M    9/9/1828  Nikolai Tolstoy        Marie Volkronskii

George Eliot       F  11/22/1819  Robert Evans           Christina Pearson

Margaret Truman    F   2/17/1924  Harry Truman           Bess Truman

Wilhelm Bach       M  11/22/1710  Johann S. Bach         Maria Bach

Jimmy Carter       M   10/1/1924  James Earl Carter      Lillian Carter

Sir Isaac Newton   M  12/25/1642  Isaac Newton           Hannah Newton

Harry Truman       M    5/8/1884  John Truman            Martha Young

Billy Carter       M   3/29/1937  James Earl Carter      Lillian Carter

Johann S. Bach     M   3/21/1685  Johann Ambrosius Bach  Elizabeth L鋗merhirt

Johann Gottfried

 Bach              M   5/11/1715  Johann S. Bach         Maria Bach

Table 6.2 The nametable and Individual Records (for Example 6.11) after Initial Input Data Are Processed

   Nametable                                                          Individual

    Records                                                            Records

      Name                Recordptr      Nameptr  Fnameptr  Mnameptr  Siblingptr   Childrenlist  Sex  Birthdate

---------------------------------------------------------------------------------------------------------------

 1 Albert Einstein            1       1     1         2         3          0             0        M   03141879

 2 Hermann Einstein           2       2     2                              0             1        M

 3 Paulina Koch               3       3     3                              0             1        F

 4 Leo Tolstoy                4       4     4         5         6          0             0        M   09091828

 5 Nikolai Tolstoy            5       5     5                              0             4        M

 6 Marie Volkronskii          6       6     6                              0             4        F

 7 George Eliot               7       7     7         8         9          0             0        F   11221819

 8 Robert Evans               8       8     8                              0             7        M

 9 Christina Pearson          9       9     9                              0             7        F

10 Margaret Truman           10      10    10        11        12          0             0        F   02171924

11 Harry Truman              11      11    11                              0            10        M

12 Bess Truman               12      12    12                              0            10        F

13 Wilhelm Bach              13      13    13        14        15          0             0        M   11221710

14 Johann S. Bach            14      14    14        25        26          0            27        M   03211685

15 Maria Bach                15      15    15                              0            27        F

16 Jimmy Carter              16      16    16        17        18          0             0        M   10011924

17 James Earl Carter         17      17    17                              0            16        M

18 Lilian Carter             18      18    18                              0            16        F

19 Sir Isaac Newton          19      19    19        20        21          0             0        M   12251642

20 Isaac Newton              20      20    20                              0            19        M

21 Hannah Newton             21      21    21                              0            19        F

22 John Truman               22      22    22                                           11        M

23 Martha Young              23      23    23                                           11        F

24 Billy Carter              24      24    24        17        18         16             0        M   03291937

25 Johann Ambrosius Bach     25      25    25                              0            14        M

26 Elizabeth L鋗merhirt      26      26    26                              0            14        F

27 Johann Gottfried Bach     27      27    27        14        15         13             0        M   05111715

(a) With Three New Entries

(b) Updated

(c) With One Additional Entry

(d) Updated Again

Figure 6.8 Nametable and Individual Records with Saved Pointers

The next input is

CARL BACH M  3/8/1714  JOHANN S. BACH  MARIA BACH

Johann S. Bach and Maria Bach already appear in the nametable and have individual records. An entry in the nametable and an individual record must, however, be created for Carl Bach.

When this has been completed, the nametable entries, i`ndividual records, and saved pointers related to this input appear as in Figure 6.8(c). Updating the nine fields of the individual records yields the result shown in Figure 6.8(d). The time required to complete an update is the constant time for the pointer manipulations plus the time to search the nametable.

The following is an algorithm for updating the information base given another input record.

1. Search the nametable.

a. Create entries in the nametable and create individual records for any of the three input names not present in nametable.

b. Set their recordptr and nameptr fields.

c. Set their childrenlist and siblingptr fields to null.

d. Set nt_ptr, nt_fptr, nt_mptr, ir_nptr, ir_fptr, and ir_mptr.

2. Update the following fields of the individual record pointed to by ir_nptr:

sex to input sex

birthdate to input birthdate

fnameptr to nt_fptr

mnameptr to nt_mptr

3. Update:

sex.fptr (the sex field of the record pointed to by ir_fptr) to m

siblingptr.ir_nptr to childrenlist.ir_fptr

childrenlist.ir_fptr to ir_nptr

sex.ir_mptr to f

childrenlist.ir_mptr to ir_nptr


Data Structures Using C: 6.5.5 Retrieving Information from the System

6.5.5 Retrieving Information from the System

Now that we have built the collections, how do we answer questions about family relationships?

Example 6.12

How do we determine who the children of John Smith are? Search the nametable for John Smith. If the name is not there, print an appropriate message. If it is there, follow the pointer to John Smith's individual record. Traverse his list of children and count the number of records on it, or output the individual names. In any case, ignoring the search of nametable, the time required is proportional to the length of the answer�that is, it is proportional to the number of children of John Smith. n

Example 6.13

How do we find all the uncles of John Smith? Search the nametable, and follow its pointer to the record for John Smith. Then, follow the fnameptr pointer to the entry in the nametable for his father and follow its recordptr back to the father's record. Do this again for the father to get to the grandfather's record. Once we have the grandfather's record we can traverse his children's list to find the uncles on John Smith's father's side. Repeat the process for John Smith's mother to find the uncles on her side. n

Simply following the sibling pointer of the father's record doesn't work, because the traversal may have begun in the middle of this sibling list. To ensure starting at the beginning of the sibling list, it is best to use the children's list of the grandfather's record. If the sibling lists were circular, this would not be necessary; it would suffice to traverse the grandfather's list of children (John Smith's father's siblings) and then output the name of each male. The same processing must be done for John Smith's mother, to output all his uncles on his mother's side.

The total time, again ignoring the nametable search, is proportional to the answer, the number of uncles. Excluding the nametable search, this is certainly close to the minimal time that would be desirable. You will see in the chapters on searching and sorting how long it would take to search the nametable!


Data Structures Using C: 6.5.6 Other System Components

6.5.6 Other System Components

To complete design of the system, in addition to the nametable implementation and search function, components are needed for deletion of records. We will not pursue this topic here, but clearly the issues of storage reclamation and allocation would have to be considered.

Beyond this, there are some more basic unresolved problems. Exactly how are questions asked of the system?

One way is for anyone wanting answers to questions to bring them to the system's designers. The designers then write programs to output the solutions. This is feasible in certain situations; in fact, many organizations do information retrieval in this way. A related solution would be to give the data representation to questioners and have them write their own programs.

Another way is to design a simple "menu," or list, of questions that includes at least those most likely to be asked. Users could then "check off" their questions if they appear on the menu, and previously written programs could be run to answer them. This solution proves satisfactory as long as the questions people want answered are on the menu.

A more sophisticated scheme would be to design a general query language. Questioners could then ask their questions in this language. A built-in language translator would interpret the question and automatically produce a program to print the answer. Achieving this solution can be quite complex and might challenge the state of the art at the present time.

The issue of security has been ignored. It would be of major concern in any retrieval system. Users' access to specific information and ability to modify it would have to be limited.


Data Structures Using C: 6.5.7 Case Study Overview

6.5.7 Case Study Overview

This case study illustrates the use of many of the concepts this book has stressed. These include the basic concepts of top-down design, data abstraction, functional modularization, and the use of appropriate data structures to achieve clear, concise, and efficient programs.

It also shows the versatility of list-structures and the way in which pointers contribute to this versatility. List-structures provide a means of interconnecting data to cut retrieval time significantly. To do so, however, means that we have to organize the data to fit the desired retrievals. As usual, we are faced with higher costs to organize the data, that is, to build the links we will need, but gain reduced costs in search time for retrieving information. This is the classic trade-off.


Data Structures Using C: Exercises

Exercises

1. Draw a picture like Figure 6.1(b), corresponding to Example 6.2 for an actual configuration of hands for the game of poker with four players.

2. Draw pictures, similar to p of Figure 6.1(d), for the polynomials

z2(y2(x(8))) + z(y(3x)) + y(x) + 18 and x3yz + 2xz + y2xz3

3. Give a recursive definition for a chain.

4. Why do the two definitions of list-structures not allow more than one pointer to a record, or loops, to occur in a list-structure?

5. Why does a null ptr and an empty stack indicate the end of the traversal of a list-structure?

6. What will be in the stack when record 20 of formula (Figure 6.4) is accessed?

7. Simulate the execution of the iterative implementation of traverse on p of Figure 6.1(d).

8. Simulate the execution of the recursive implementation of traverse on p of Figure 6.1(d).

9. Give an example of a list containing loops for which the traversal functions fail because they do not terminate.

10. a. Write a process function to turn the iterative traversal into a program that prints the number of sublists of a list-structure.

b. Will your solution also work for the recursive traversal?

11. a. Write a function to return a pointer to the record of a list-structure that has the maximum information field value.

b. Modify your solution to return a pointer to a list. The list should contain pointers to all records whose information field value coincides with the maximum value.

12. Give an algorithm to determine whether two list-structures whose records have the same format are identical.

13. Modify your solution to Exercise 12 so that the algorithm determines whether the two list-structures have the same "structure," independent of the information values stored.

14. Each complex record of a list-structure has a field pointing to the record's sublist. The sublist is composed of a main list (composed of the chain of records linked by their link fields). Suppose each complex record of a list-structure also has a field called number that contains the number of records on the complex records sublist. Write a process function to turn the traverse functions into a program to insert a record after the record whose information field value is value. The program must, of course, also correctly update the number field for the appropriate sublist.

15. Write a function to implement evaluate (Example 6.9).

16. Modify your solution to Exercise 15 so that it outputs the evaluation of each operation as it occurs.

17. Modify your solution to Exercise 16 so that it prints out an appropriate message, such as "too many operands for the operator PS of the sublist of record 7" or "missing operands for the operator PS of the sublist of record 7," whenever the list-structure does not represent a valid arithmetic expression.

18. Assume that p points to a record of Example 6.9 containing an operator or operand. Suppose ptr points to a record of a sublist after which the record pointed to by p is to be inserted. It is to be inserted as the first record of a sublist if it is an operator, and as the next record on the sublist if it is an operand. Write an insertion function to accomplish this. For example, suppose the list-structure, ls, is currently:

If ptr points to DIFF, and the record pointed to by p contains PS, then after insertion ls will be

If the record pointed to by p contained 432.713, then after insertion ls would have been

19. Insertion and deletion of records in list-structures is similar to their insertion and deletion in chains. However, the deletion of a sublist has ramifications for the reclaim function of Chapter 3. Discuss these, and explain how they might be resolved by modifying the definition of reclaim or by increasing the responsibility of the programmer. In either case, the idea is to be sure all unused records are returned to availist.

20. Modify paragraphinsert (Example 6.8) so that all deleted records are returned to the availist.

21. Let ptr and pv point, respectively, to a record of a list-structure and to its preceding record. Write a function to delete the record pointed to by ptr if it is simple, and to replace it with the list-structure to which it points if it is complex. For example, consider the following list-structure.

If ptr points to 17, then the function produces

If ptr points to the complex record following 17, then the function produces

22. Give an implementation of next and sublist of the function traverse (Section 6.2.1) for each declaration of the polynomial p.

23. Give a declaration for the records of Example 6.8 using dynamic memory and write implementations for all the functions of traverse and process that depend on this implementation.

24. Draw a picture, corresponding to Table 6.2, for the complete input data (31 names) of Example 6.11.

25. Modify the answer to Exercise 24 to reflect the lists of children being kept as circular lists.

26. Show the complete situation in the arrays of your solution to Exercise 24 after the input below is processed.

JOHANN 0AMBROSIUS BACH  M  2/22/1695  CHRISTOPH BACH

MARIA MAGDALENA GRABLER

27. Show the complete situation after the input of Exercise 26 is processed, when circular lists are used for the lists of children.

28. Suppose the records of Example 6.11 were to be kept in alphabetical order by the name in the name field. What would the arrays look like after all the input data were processed?

29. How does the algorithm for updating the information base given another input record (p. 287) have to be modified if inputs are allowed for which one or more field values are unknown?

30. Write two functions for Example 6.11, children(person) and brothers(person). They should print out the names of all children and all brothers of person, respectively. Simulate the behavior of the functions on the data of the example.

31. Modify your solutions to Exercise 30 so that the functions will work on circular lists of children.

32. Suppose the nametable is implemented as a list. How will your solutions to Exercises 30 and 31 be modified?

33. Write an algorithm to print out the names of individuals who were born before 1900. How much time will your algorithm take?

34. Can you find a way to implement the information system so that a faster solution can be obtained for Exercise 33? If you find a faster solution, then determine how its update time will compare to that for Exercise 33.

35. Write a detailed description for a routine that will delete an individual record and its nametable entry from the system. This implies that all relevant pointers will be properly modified and that storage will be reclaimed.


Data Structures Using C: Suggested Assignment

Suggested Assignment

Consider the following list-structure for a book. Book might represent the current structure of a textbook. Suppose we want to print this information as a table of contents. It would appear as follows:

CHAPTER 1

1.1  ARRAYS

1.2  RECORDS

1.2.1 POINTERS

1.2.2 EXAMPLE

1.3  SUMMARY

CHAPTER 2

2.1  LISTS

2.2  USES

CHAPTER 3

3.1  APPLICATIONS

3.1.1  BACKGROUND

3.1.2  RANKINGS

3.1.2.1 FLOWCHART

This problem can be abstracted to the printing out of a "picture" of a list-structure. Assume all records are declared as

typedef struct listrecord

{

whatever info;

int recordtype;

struct listrecord *link,*sublist;

}bookrecord,*liststructurepointer;

Write a function printlist that will print out a "picture" of the list-structure pointed to by ls. Consider, for example, the list-structure accounts of Section 6.1. The list-structure is

INFO = 175

RECORDTYPE = COMPLEX

INFO   = 0

RECORDTYPE = COMPLEX

INFO   = 100

RECORDTYPE = SIMPLE

INFO  = 75

RECORDTYPE = SIMPLE

INFO  = 450

RECORDTYPE = COMPLEX

INFO   = 200

RECORDTYPE = COMPLEX

INFO   = 250

RECORDTYPE = SIMPLE

INFO  = -46

RECORDTYPE = SIMPLE

Your program should read in and echo print three list-structures. One should be null. It should then call on printlist to print out the "picture" for each of the three list-structures.

The input data for accounts might be

1     indicates a nonnull list-structure; 0 would indicate a null list

       structure

175  CMPLX

1    1   indicates the link and sublist fields of "175" are not null

0    CMPLX

1    1

100  SMPL

0     indicates the link field of "100" is null; it is thus the last record of

       a sublist

75   SMPL

0

450  CMPLX

1    1

200  CMPLX

1    0     indicates the link field of "200" is not null, but its sublist

            field is null

250  SMPL

0

-46  SMPL

0

Notice that the input is given in traversal order.

The printlist function is to have almost exactly the same detailed refinement as either the iterative traverse (Section 6.2.1 ) or the recursive traverse (Section 6.2.2). You must write the proper process routine, say process2, to turn traverse into printlist. You may add parameters to process2 if necessary.

For extra credit, do the reading in and echo printing of the three list-structures in the program by calling a function readechoprint. It must also be obtained by writing the proper process routine, say process1, to turn traverse into a function for readechoprint.

Of course, it will be necessary for you also to write the stack operation routines setstack, push, pop, and empty, as well as next, sublist, and complex.


Data Structures Using C: CHAPTER 7: TREES

CHAPTER 7: TREES

Introduces three new data structures

binary trees

trees

ordered trees

Shows

how to apply them to store data

that they are useful as conceptual aids to problem solving and uses them in the n-queens problem

how to implement them

Emphasizes the basic operations for

binary trees

preorder, inorder, and postorder traversals

trees

preorder and postorder traversals

backtracking

depth-first and breadth-first traversals

Clarifies the relation between recursion, trees, and stacks

Discusses recursive solutions to show

when they are efficient

how to improve them

7.1: Branching Out

7.2: Trees of Records

7.3: Using the Traverse Functions for Binary Trees

7.4: Implementing Binary Trees

7.5: Trees

7.6: Traversals of Trees

7.7: More on Recursion, Trees, and Stacks

Exercises

Suggested Assignments


Data Structures Using C: 7.1: Branching Out

7.1: Branching Out

All data structures provide a way to organize data. Different structures serve different purposes. Complex lists, as shown by the information retrieval case study of Chapter 6, can result in considerable intertwining and sharing of lists and data. Everyone is familiar with the concept of a family tree. In fact, a nice exercise would be to print the family tree for an individual whose family history is stored in the data base of that case study. The family tree is actually embedded within the lists containing the family history. Trees may be viewed as a special case of lists in which no intertwining or sharing takes place. They are the basis for structures and operations used in all aspects of programming, from the structure of programs for compilers, to work in data processing, information retrieval, and artificial intelligence. Trees are ubiquitous; they seem to sprout everywhere�even in the field of computer science!

Formally, trees are collections of nodes. Trees derive both their appearance and their name from the lines, known as branches, that connect the nodes. Understanding how to represent trees, and how to perform certain operations on them, opens the door to an appreciation of many elegant algorithms and useful techniques in computer science.

Trees play a significant role in the organization of data for efficient information retrieval and are ideal candidates for fast searches, insertions, deletions, and sequential access. Efficiency and speed are possible because trees "spread out" the data they store, so that different paths in the tree lead quickly to the relevant data. They are also convenient for conceptualizing algorithms.

Many problems can be solved by imagining all the possible answers, represented as a tree. The algorithm or program is then viewed as a traversal through the tree. The general procedure for a traversal through a tree, which will be developed in this chapter, can then be adapted and used as the basis for the program. This often results in easily created and efficient programs. In such cases the tree itself serves as a conceptual aid and need not actually be "grown" or stored in memory. Nevertheless, it provides the framework for obtaining the algorithm.

The mathematics of trees has always been of interest and is discussed quite thoroughly by Knuth [1973a]. In this chapter we consider fundamentals and introduce special trees as they are needed in subsequent chapters.

As indicated in Figure 7.1, there are many ways to depict inclusion or hierarchical relationships. Each part of the figure conveys the same information about the relations between objects A to J. Figure 7.1(a) uses indentation reminiscent of program structure, or Pascal, COBOL, and C record structure, to convey inclusion. Figure 7.1(b) uses nested blocks or sets, while Figure 7.1(c) has nested parentheses. The last, Figure 7.1(d), uses lines that branch out like a tree turned upside down. This is how the data structure that is the topic of this chapter, the tree, is depicted.

Figure 7.1 Capturing Inclusion Relationships

If Figure 7.1(a) is being used to represent the structure of a C record, then the structure shows that record A includes three fields, B, C, and D, or that D includes two subfields, I and J. The tree of Figure 7.1(d) captures this inclusion information. To a C compiler, it is also important that field B comes before C and that C precedes D. When such ordering information is needed, we use ordered trees. To capture order, as well as inclusion information, Figure 7.1(d) must be an ordered tree.

Much of the terminology for the tree data structure is taken from botany and is therefore familiar and colorful. For instance, A of the tree in Figure 7.1(d) is at the root of the tree. The tree consists of three subtrees, whose roots contain B, C, and D. The entries in the tree, A to J, are each stored at a node of the tree, connected by lines called branches. The tree is ordered if each node has a specified first, second, or nth subtree. The order itself may be specified in many ways. In this example, the order is specified by taking the subtrees in alphabetical order, determined by their root entries. Rearranging the order of the subtrees in Figure 7.1(d) results in the same tree but not the same ordered tree, if we assume the ordering is specified for each node from left to right by the way the tree is pictured. A distinction will be made between ordered trees and trees throughout this text. Keep in mind that these terms refer to two distinct kinds of structures.

Besides these two kinds of trees, there are binary trees. A binary tree is either

n The null binary tree, or

n A root node and its left and right subtrees, which are themselves binary trees with no common nodes

This recursive definition emphasizes the fact that binary trees with at least one node are composed of a root and two other binary trees, called the left and right subtrees, respectively. These subtrees, in turn, have the same structure. All binary trees consist of nodes, except for the null binary tree, which has no nodes. The null binary tree, just like the null list, cannot be "drawn." Both are used to make definitions more uniform and easier, and both represent the smallest data structure of their kind. There is exactly one binary tree with one node, its root. This tree has null left and right subtrees. The two binary trees with exactly two nodes are pictured as in Figure 7.2. One has a root and a nonnull left subtree, the other a root and a nonnull right subtree. Figure 7.3 pictures a larger binary tree.

The lines used to indicate a left and right subtree are called branches. Branches show the relationship between nodes and their subtrees. In Figure 7.3 the nodes are numbered 1 to 20 for easy reference. Node 5 has a predecessor (node 2), a left successor (node 10), and a right successor (node 11). Every node has at most two successors, and every node except the node labeled root has exactly one predecessor. Predecessors are also known as "parents," successors can be called "children," and nodes with the same parent are (what else?) "siblings." As far as is known, trees do not exhibit sibling rivalry.

(a) Left Subtree

(b) Right Subtree

Figure 7.2 Two Two-Node Binary Trees

Notice that, starting from any node in the tree, there is exactly one path from the node back to the root. This path is traveled by following the sequence of predecessors of the original node until the root is reached. For node 18, the sequence of predecessors is 9, 4, 2, and 1. There is always exactly one path from the root to any node.

Follow the branch from node 1 to node 2 or 3 and you come to the left or right subtree of node 1. Its right subtree is shown in Figure 7.4. Similarly, each node has a left and a right subtree. Nodes with at least one successor are called internal nodes. Those with no successors are called terminal nodes (also known as "external" nodes or "leaf" nodes). The left and right subtrees of a terminal node are always null binary trees.

By definition, the root of a binary tree is at depth 1 . A node whose path to the root contains exactly d - 1 predecessor nodes has depth d. For example, node 10 in Figure 7.3 is at depth 4. This is the path length of the node. The longest path length is called the depth of the binary tree; it is 5 for the tree of Figure 7.3. For many applications it is the depth of a binary tree, like the depth of a list-structure, that determines the required storage and execution time.

Figure 7.3 A Binary Tree

Figure 7.4 The Right Subtree of the Root of Figure 7.3

Suppose your program must count the number of occurrences of each integer appearing in an input sequence of integers known to be between zero and 100. This is easy. Store the count for each integer of the allowed range in an array and increment the appropriate entry by 1 for each integer as it is input. The code would be array [i] + +, where i is the variable containing the integer just read. Note that the efficiency of this solution derives from the selection operation for an entry, which is supported by the array data structure.

What if the range of the integers was not known? Sufficient storage is not available for an array to store counts for all possible integers. What is required is a data structure that can be dynamically extended to accommodate each new integer and its count as the integer is encountered in the input. This data structure must be searched to see if the new integer already appears. If it does, its count is increased by 1 . Should the integer not appear, it must be inserted with its count set to 1. A special tree, a binary search tree introduced in Chapter 9, is convenient to use for this data structure. This tree allows the search for an integer to consider only nodes along a particular path. At worst, this path must be followed from the root to a terminal node. Thus the longest search time will be determined by the depth of the tree. The same data structure and the depth measure are important for the related problem, "Produce an index for a book." Now, instead of storing integer counts, the position of each occurrence of a word must be retained. Do you see the possibility of storing dictionaries of modest size in binary search trees?

Figure 7.5 A Sparse Binary Tree

Binary trees need not have such a compact look as that of Figure 7.3 but can appear sparse, as in Figure 7.5. A compact binary tree like Figure 7.3 is called a complete binary tree. To be more precise, a binary tree of depth d is a complete binary tree when it has terminal nodes only at the two greatest depths. d and d -1. Further, any nodes at the greatest depth occur to the left, and any nodes at depth less than d - 1 have two successors. In Figure 7.3, nodes 16 to 20 are the nodes at greatest depth; they occur to the left. The sparse tree has the same number of nodes but twice the depth.

7.1.1 Depth versus Capacity*


Data Structures Using C: 7.1.1 Depth versus Capacity*

7.1.1 Depth versus Capacity*

*This section can be omitted on first reading.

Let us take a moment here to derive two important relations between the depth, d and the number of nodes, n, in a binary tree. First note that in any binary tree there is at most one node at depth 1, at most two nodes at depth 2, and, in general, at most 2k-1 nodes at depth k, since a node can give rise to at most two nodes at the next depth. The total number of nodes, n, is then at most 1 + 21 + 22 + . . . + 2d-1. This sum is 2d - 1. So

n � 2d - 1

The minimum possible depth, dmin, for a binary tree with n nodes will be given by

dmin = �lg (n + 1)�

where �x� indicates the smallest integer that equals or exceeds x. If n = 20, then dmin = �lg 21� = �4.39� = 5; and if n = 31, then dmin = �lg 32� =5. The binary tree corresponding to dmin may always be taken as a complete binary tree. Consequently, the depth of a complete binary tree will never exceed (lg n) + 1. This contrasts sharply with the depth of the sparsest binary tree with n nodes, which is n . For instance, if n is 1,000, the complete binary tree has a depth of only 10, while the sparsest has depth 1,000. This has important implications for how trees are constructed and for information retrieval, as shown in Chapter 9.


Data Structures Using C: 7.2: Trees of Records

7.2: Trees of Records

The nodes of binary trees may be used to store information. For example, the nametable of Section 6.5 could be implemented by storing each of its records at a node of a binary tree. Just as lists are referred to by name, the tree can be named, say nametable. Obviously, nametable must have records inserted into it, deleted from it, and searched for in it. Insertion, deletion, search, and traversal are the basic tree operations.

Balanced trees and binary search trees (Chapter 9) and heaps (Chapter 8) are all binary trees with special properties that make them useful for storage of information. The properties that make them useful favor efficient searching, inserting, and deleting. Each of these types of binary tree requires its own algorithms for these operations, in order to take advantage of or to preserve their characteristics. For now, to get the flavor of these operations, consider some simple examples.

7.2.1 Insertion and Deletion of Records in Binary Trees

7.2.2 Climbing Binary Trees

7.2.3 Three Preorder Traversals of a Binary Tree


Data Structures Using C: 7.2.1 Insertion and Deletion of Records in Bi

7.2.1 Insertion and Deletion of Records in Binary Trees

Assume that each node of a binary tree is represented by a record containing an information field, a left pointer field, and a right pointer field. The notation node.p stands for the record or node pointed to by the variable p.

To accomplish an insertion, storage must first be allocated for a new record. Next, its fields must be set correctly; this includes entering data into the record and setting its pointers to the succeeding left and right nodes, or to null if it is a leaf. Finally the proper pointer adjustments must be made in the tree to the predecessor of the new entry. Consider Figure 7.6, which shows a binary tree t before and after the insertion of a new entry C, which replaces predecessor O's null right subtree. The information for the new record is assumed to reside in value, so C is stored in value. Thus the new record must have its information field set to value and its left and right pointer fields set to null. The predecessor's right pointer field must be set to point to the new record. This is accomplished by the following.

>Comment

p = avail();

>Comment

setinfo(p,&value);

>Comment

setleft(p,null);

>Comment

setright(p,null);

>Comment

setright(predecessor,p);

Setinfo copies the contents of the storage pointed to by its second parameter into the info field of the record pointed to by its first parameter. Note that it is defined somewhat differently from the setinfo used in previous chapters. Setleft and setright copy the contents of their second parameter into the proper field of the record pointed to by their first parameter. Notice that setleft is used to set the left pointer of the new record; similarly for setright. However, setright is also used to change the right pointer of the predecessor record so that it points to the new record, which is now its right node. Thus setleft and setright can be used to set pointers for any node in the tree. Together with setinfo and avail, these functions reflect the binary tree's implementation details. The code for the insertion itself is independent of the implementation.

Figure 7.6 A Binary Tree T before and after the Insertion of C

If a new record is to replace a nonnull subtree (say, if C were to replace U in t), then the new record's left and right pointer fields would have to be adjusted accordingly. The left and right pointers from U would have to be copied into the new record's (C's) pointer fields. Furthermore, the storage for the replaced record U might be reclaimable. If so, care is required to save the pointer to it in its predecessor's left or right field so that its storage can be reclaimed. In this example, after saving the pointer to U, the right pointer of U's predecessor, L, would be replaced by a pointer to C, its new right node. Then U's storage can be reclaimed. The code below does this correctly, assuming predecessor points to a record whose nonnull right subtree is to be replaced.

>Comment

q = right(predecessor);

>Comment

p = avail();

>Comment

setinfo(p,&value);

>Comment

setleft(p,left(q));

    >Comment

setright(p,right(q));

    >Comment

setright(predecessor,p);

    >Comment

reclaim(q);

Right returns a copy of the right pointer field value of the record pointed to by its parameter. This code is for illustrative purposes. If the record is really reclaimable, it would be more efficient simply to replace its info field by the new info value. The statement setinfo(q,&value) would suffice. However, if the replaced record were needed for another purpose, say for insertion in another binary tree, then this would not work.

The deletion of a terminal node pointed to by p, whose predecessor is pointed to by predecessor, may be accomplished by the following.

    >Comment

if(left(predecessor) == p)

    >Comment

setleft(predecessor,null);

else

1 otherwise

setright(predecessor, null);

1 set its predecessor's right pointer to null

Left is analogous to right. The decision is required in order to know whether it is predecessor's left or right successor that is to be deleted. Again, if the deleted node's storage may be reclaimed, care must be taken not to lose the pointer to it. Deletion of an internal node must take into account what is to be done with the node's subtrees.

The wary reader may have qualms about the code developed for insertion and deletion. Remember�special cases must always be checked. When insertion or deletion occurs at the root node, indicated by a null value in predecessor, it is the name of the tree whose pointer must be adjusted. The illustrative code fails in this case. You should modify the program segments so that they also account for this special case.


Data Structures Using C: 7.2.2 Climbing Binary Trees

7.2.2 Climbing Binary Trees

We have seen examples whose solution includes a traversal through a collection of records in an array or a list. A traversal through a collection of records stored in a binary tree is also the basis for the solution to many problems. We will consider three distinct ways to traverse a binary tree: (1) preorder, (2) inorder, and (3)postorder.

Pre-, in-, and postorder traversals are often referred to mnemonically as root-left-right, left-root-right, and left-right-root traversals. The prefixes pre, in, and post refer to whether the root is accessed prior to, in between, or after the corresponding traversals of the left and right subtrees. In all cases, the left subtree is traversed before the right subtree. When nodes are listed in the order in which they are processed, all nodes in the left subtree appear before all nodes in the right subtree, no matter which traversal is used. However, the type of traversal determines where the root appears. Since we defined binary trees recursively, it is convenient to give recursive definitions for these traversals.

To preorder traverse a binary tree:

if the binary tree is not the null binary tree, then

1. access and process its root

2. traverse its left subtree in preorder

3. traverse its right subtree in preorder

To inorder traverse a binary tree:

if the binary tree is not the null binary tree, then

1. traverse its left subtree in inorder

2. access and process its root

3. traverse its right subtree in inorder

To postorder traverse a binary tree:

if the binary tree is not the null binary tree, then

1. traverse its left subtree in postorder

2. traverse its right subtree in postorder

3. access and process its root

Let's apply these definitions to Figure 7.6(b).

To do a preorder traversal of t, since it is not null, requires that its root, L, be accessed and processed first. Next, the left subtree of t must be preorder traversed. Applying the definition to it, its root, O, is accessed and processed. Since O's left subtree is null, its right subtree is traversed next in preorder. This results in C being accessed and processed, terminating the preorder traversal of O's right subtree and also of t's left subtree. Completing t's traversal requires a preorder traversal of its right subtree, resulting in U, S, and T being accessed and processed in turn.

An inorder traversal of t requires its left subtree to be inorder traversed first. Applying the definition to this left subtree requires its null left subtree to be inorder traversed (note how easy that is), its root O to be accessed and processed next, and then its right subtree to be inorder traversed, which results in C being accessed and processed. This terminates the inorder traversal of t's left subtree. T's root, L, must then be accessed and processed, followed by an inorder traversal of t's right subtree. This causes S, U, and T to be accessed and processed in turn, completing the inorder traversal of t.

Try applying the postorder traversal definition to t yourself. The sequences below represent the order in which the records stored at the nodes of t are accessed and processed for each of the traversals.

>Comment

locust

>Comment

oclsut

>Comment

costul

Notice that the terminal nodes C, S, and T are dealt with in the same order for all three traversals. This is not just a coincidence but will always happen.

The trees of Figure 7.7 represent a table of contents, an arithmetic expression, and a program that invokes functions that, in turn, invoke other functions. Traversing these trees, respectively, in pre-, in-, and postorder results in the order of access to nodes shown in Figure 7.7(d-f). Each is a natural application of one of the three traversals: a listing of the table of contents, an arithmetic expression in the usual notation, and a listing of the order in which the functions are needed to test another function (for example, P1 and P2 are needed to test P3).

Not all data structures need to be traversed. For example, stacks, queues, heaps, and the hash tables to be introduced in Chapter 9 are not normally traversed. Just as applications of arrays and, even more so, lists often reduce to traversals, applications of binary trees often do as well. Therefore procedures for their traversal merit study.


Data Structures Using C: 7.2.3 Three Preorder Traversals of a Binary T

7.2.3 Three Preorder Traversals of a Binary Tree

Writing a recursive function to traverse a binary tree based on the recursive definition for preorder traversal is straightforward. The same is true for an inorder and postorder traversal.

Recursive Version

preorder(ptree)

/* Preorder traverses the

binary tree tree.

*/

binarytreepointer *ptree;

{

binarytreepointer 1,r,setnull(),left(),right();

if(*ptree != setnull())

{

>Comment

process(ptree);

>Comment

1 = left(*ptree);

>Comment

preorder(&l);

r = right(*ptree);

preorder(&r);

}

}

To convince yourself it works, simulate it on the tree of Figure 7.6(b).

Figure 7.7 Trees for (a) a Table of Contents, (b) an Arithmetic Expression, and (c) Components of a Program. Order of Access to Nodes is (d) for Preorder Traversal of (a), (e) for Inorder Traversal of (b), and (f) for Postorder Traversal of (c)

We will now develop an iterative procedure for contrast with the recursive one. It will also be useful if the programming language you are using is not recursive. Recall that the application of the definition of preorder traversal to Figure 7.6(b) required keeping track of the left subtree about to be traversed and remembering at which node to resume when that left subtree traversal terminated. For example, in Figure 7.6(b), after node L is processed, its left subtree must be preorder traversed. We must remember, after that traversal is complete, to resume the traversal by traversing L's right subtree. Put another way, it was necessary to suspend the current traversal, incurring its completion as a postponed obligation, and begin a traversal of the proper left subtree. These postponed obligations must be recalled and resumed in reverse order to the order in which they were incurred. That is, in a last-in first-out fashion. Obviously, the ubiquitous stack is called for. It seems to pop up everywhere.

To accomplish this let p point to the root of the subtree about to be traversed. After the processing of node.p, a pointer to its right subtree can be pushed onto the stack before traversal of its left subtree. At the time that node.p's right subtree is to be processed (when its left subtree traversal terminates), the stack will contain pointers to all right subtrees whose traversals have been postponed (right subtrees of nodes already accessed), but the top stack entry will point to node.p's right subtree. Thus popping the stack and setting p to the popped value correctly updates p to point to the correct next record to be accessed.

Nonrecursive Version Saving Right Pointers on a Stack

preorder(ptree)

/* Preorder traverses the

binary tree tree.

*/

binarytreepointer *ptree;

{

binarytreepointer p, null, setnull(), left(), right();

>Comment

stack s;

>Comment

null = setnull();

>Comment

setstack(&s);

>Comment

p = *ptree;

>Comment

while ((p != null) || !empty(&s))

>Comment

if (p != null)

{

>Comment

process (ptree,p);

>Comment

push (right(p),&s);

>Comment

p = left(p);

}

>Comment

else

>Comment

pop (&s,&p);

}

The while loop condition is true whenever the traversal is not yet completed. A nonnull p means that a record is pointed to by p and is the next one to be accessed and processed. A null p value means no record is pointed to by p, and some left subtree traversal must have just been completed. If, at that point, the stack is not empty, then more needs to be done; but if it is empty, then no right subtrees remain to be traversed, and the traversal is done. Again, to convince yourself the program works, simulate it on the tree of Figure 7.6(b).

The recursive procedure reflects the structure of binary trees and emphasizes that they are made up of a root with binary trees as subtrees. It sees the special cases, such as a null tree, and the substructures of the recursive definition, such as left and right subtrees. In order to develop the nonrecursive version, it is necessary to think of the binary tree as a collection of individual records. Writing the function then involves providing for the correct next record to be accessed and processed within the while loop. This is a general strategy for attempting to find nonrecursive solutions. The nonrecursive version requires that the structure be viewed as a collection of its individual parts, with each part treated appropriately and in the correct order. This explains why recursive versions are clearer.

An alternative nonrecursive function is based on saving on a stack the path from the root to the predecessor of the node currently being processed. It is useful if the processing of a node requires accessing its predecessor node. This alternative approach guarantees that at the time the node is processed its predecessor is at the top of the stack. However, this algorithm requires some searching to find the proper right subtree to traverse next when a subtree traversal has terminated. Such a termination is indicated by p being null. Figure 7.8 shows a situation when such a termination has occurred. P is null, having been set to the null right pointer value of node 10 after node 10's left subtree has been traversed. Looking back along the path from node 10 to the root, we find the path nodes 8, 6, 5, 1. It is not difficult to see that node 5's right subtree is where the traversal must resume. Precisely this path back toward the root must be followed until a node on this return path (node 6) is the left successor of its predecessor. P must then be set to the right subtree of that node (node 12) and the traversal resumed. If no node which is the left successor of its predecessor is found on the return path, the entire tree has been traversed.

Figure 7.8 Node 5's Left Subtree Has Just Been Traversed

The following function is a nonrecursive preorder traversal that saves the path to the root on a stack. Item returns a copy of the pointer at the top of the stack without disturbing the stack.

Nonrecursive Version Saving the Path to the Root on a Stack

preorder(ptree)

/* Preorder traverses the

binary tree tree.

*/

binarytreepointer *ptree;

{

binarytreepointer null,p,q,rightpointer;

binarytreepointer setnull(),left(),right(),item();

>Comment

stack s;

    >Comment

null = setnull();

    >Comment

setstack(&s);

    >Comment

p = *ptree;

    >Comment

while((p != null) || !empty(&s))

if(p != null)

1 if there is a current node

{

    >Comment

process(ptree,p);

    >Comment

if(left(p) != null)

{

    >Comment

push(p,&s);

    >Comment

p = left(p);

}

    >Comment

else

{

    >Comment

push(p,&s);

    >Comment

p = right(p);

}

}

else

1 otherwise

{

    >Comment

do

{

    >Comment

pop(&s,&q);

    >Comment

if (!empty(&s))

    >Comment

rightpointer = right(item(&s));

    >Comment

else

    >Comment

rightpointer = null;

    >Comment

}while(!empty(&s)&&(q==rightpointer));

    >Comment

if(q != rightpointer)

    >Comment

p = rightpointer;

}

}

Notice that the first two versions of the preorder traversal do not retain information about the depth of a node or about a node's predecessor, although they could be modified to do so. The third version retains this information, since the depth of the current node is equal to the number of stack entries plus one, and the top stack entry always points to the current node's predecessor. When this information is needed, the last version provides it directly, especially if a basic stack operation that returns the size of the stack is available.

Each of the three traversal functions can be modified to be a partial traversal when required, by including a variable done to signal termination, just as was done for the list traversals. This would be the case, for example, if you were traversing only to find a desired record. Similar function can be written for the inorder and postorder traversals.


Data Structures Using C: 7.3: Using the Traverse Functions for Binary

7.3: Using the Traverse Functions for Binary Trees

In this section a number of examples are presented to show how the traverse functions developed thus far can be applied to develop solutions to programming problems.

Example 7.1

Print the number of terminal nodes in a binary tree tree. n

The nonrecursive preorder traversal that saves right subtree pointers on a stack will be used to perform this task. The other versions could have been adapted as readily. The approach taken is to leave the traverse function as it is and write process to turn preorder into a solution to the problem.

To accomplish this, process must test each node it deals with to see if it is a terminal node. If so a counter, tncount, should be increased by 1. Making tncount a static variable in process allows its values to be preserved between calls to process. The first time process is called during the traversal it must initialize tncount to zero, and the last time it is called, in addition to updating tncount, process must print its final value.

If p points to the current record to be processed, then when p points to the root of the traversed tree tree, process has been called for the first time, so it must set tncount to zero. But how can process recognize when p points to the last node so that tncount can be printed? To do this, process must have access to the stack s. When s contains no nonnull pointers and node.p is a terminal node, then node.p is the last node. Assume that a function last returns true if s contains no nonnull pointers and false otherwise. Consequently s must be a parameter of, or global to, process. Finally, we must assume that only nonnull binary trees are to be considered since process is invoked by preorder only when tree is not null. Process may now be defined as follows.

process(ptree,p,ps)

/* Prints the number of terminal nodes in

binary tree tree when p points to the last

node in a preorder traverse of tree which

saves right pointers on stack s.

*/

binarytreepointer *ptree,p;

stack *ps;

{

binarytreepointer null,setnull(),left(),right();

static int tncount;

null = setnull();

>Comment

if(p == *ptree)

>Comment

tncount = 0;

>Comment

if((left(p) == null) && (right(p) == null))

{

>Comment

tncount++;

>Comment

if(last (ps))

printf("\n The number of terminal nodes is %d",tncount);

}

}

It would certainly be more efficient to make null global so that setnull would not be invoked for each node. Similarly, setting tncount to zero in preorder could remove the need for the (p == *ptree) test for each node. In fact, it would be easier to take a second approach to finding a solution. Namely, modify preorder so that it sets tncount to zero initially, and after its while loop is exited, simply print tncount. This second solution is considerably more efficient, because the test performed by last is not needed. The second approach is more efficient, but it involves changing a traversal, which may be already compiled. Which approach to use depends on what alternatives are available and just how important the extra efficiency is.

Example 7.2

The second example concerns the problem of inserting a new node in the binary tree tree in place of the first terminal node encountered in a preorder traversal of tree. The information field value of the new node is to be copied from that of the record value. n

In this case process will be written to make a solution out of the nonrecursive version of preorder that stores the path from the root to the node being processed. This is convenient, since the predecessor of the node to be processed is required by process when the insertion is made, and it will be the top stack entry at that time.

process(ptree,p,null,ps,value)

/* If p points to the first terminal node, encountered

in a preorder traversal of binary tree tree that saves

the path to the root on stack s, then replaces that node by

a new node whose info field is set to the contents of value.

*/

binarytreepointer *ptree,p,null;

stack *ps;

int value;

{

binarytreepointer q,ptr,avail(),

left(),right();

>Comment

if((left(p) == null)&&(right(p) == null)

{

>Comment

ptr = avail();

>Comment

setinfo(ptr,&value);

>Comment

setleft(ptr,null);

>Comment

setright(ptr,null);

>Comment

if(p != *ptree)

{

>Comment

q = item(ps);

>Comment

if(left(q) == p)

setleft(q,ptr);

else

setright(q,ptr);

}

>Comment

else

>Comment

*ptree = ptr;

>Comment

setstack(ps);

}

}

When process is called, it tests whether the node is a terminal node. Since it will cause the traversal to be terminated after the replacement, this must be the first terminal node. Storage is then obtained for the new record and its fields filled in properly. Since it will be a terminal node, its pointer fields are set to null. To insert the new record, the predecessor of the terminal node must have one of its pointer fields set to point to the new record. The correct one is determined by whether the replaced record was its left or right successor. The special case of the insertion occurring at the root requires only that tree be set to point to the new record. Finally, setting the stack to empty ensures that the traversal will be aborted.

Value is passed by preorder to process. Item simply returns a copy of the top stack entry.

Example 7.3

The final example involves writing a function depth to return the depth of a binary tree as its value.n

The nonrecursive version of preorder that saves the path to the root on the stack can be turned into such a function. When a node is being processed, its depth is one more than the number of stack entries at that moment. To determine the depth, initialize a variable d to 1 and size to zero before the while loop, increase size by 1 after push, and decrease it by 1 after pop. Give process access to d and size. Process must simply compare size+1 to d, and when greater, set d to size+1. After termination of the traversal, depth must return d.

Another solution for depth can be obtained by recursion. This requires a way to express the depth recursively. The depth of a binary tree t is

n Zero if t is null, else

n One more than the maximum of the depths of t's left and right subtrees

Now the function can be written directly as follows:

depth(tree)

/* Returns the depth of

the binary tree tree.

*/

binarytreepointer tree;

{

binarytreepointer setnull(),left(),right();

>Comment

if(tree == setnull())

>Comment

return(0);

>Comment

else

>Comment

return(1 + max(depth(left(tree)),

depth(right(tree))));

}

where max returns the maximum of its parameters as its value.


Data Structures Using C: 7.4: Implementing Binary Trees

7.4: Implementing Binary Trees

This section introduces three ways to implement a binary tree.

1. Using an array of records with just an information field

2. Using records with an information field and left and right pointer fields linked together via their pointer fields

3. Using a list-structure

Chapter 8 contains an application of the first implementation. A program using the second implementation, with the records stored in dynamic memory, is included. The program specifies the details of this important representation and illustrates how to create and process binary trees and how to test functions developed in the earlier examples.

7.4.1 Sequential Representation

7.4.2 Linked Representation

7.4.3 List-structure Representation


Data Structures Using C: 7.4.1 Sequential Representation

7.4.1 Sequential Representation

The sequential representation uses an array to store information corresponding to each node of a tree. Consider the binary tree shown in Figure 7.9. Its depth is 5 and it has fourteen nodes. The nodes are numbered in a special way. This numbering may be obtained by preorder traversing the tree and, as each node is accessed, assigning it a node number. The root is assigned 1, each left successor of a node is assigned twice the number of its predecessor, and each right successor is assigned one more than twice the number of its predecessor. With nodes numbered in this way, the predecessor of node i is numbered i/2 (integer divide), its left successor is numbered 2*i, and its right successor 2*i+1.

Figure 7.9 Sequential Representation of a Binary Tree

The sequential representation of a binary tree is obtained by storing the record corresponding to node i of the tree as the ith record in an array of records, as shown in Figure 7.9. A node not in the tree can be recognized by a special value in its corresponding array position (such as node 8, which has - as its value) or by a node number (like 24) that exceeds the highest node number in the tree (20 in this example). Given a pointer p to a node in the array, it is easy to determine the position of its predecessor (p/2), its left successor (2*p), and its right successor (2*p+1).

A binary tree with n nodes will require an array length between n and 2n - 1. The smallest length (n) is sufficient when the binary tree is complete, and the greatest length (2n - 1) corresponds to the case of an extremely skewed binary tree whose nodes have only right successors. No space in the array is unused for a complete binary tree, and the array length is proportional to the number of nodes in the tree. However, a sparse tree can waste a great deal of array space. The worst case requires 2n - 1 positions but uses only n. Thus the sequential representation is convenient for dealing with complete static binary trees. Such trees arise, for example, when you must put student records in alphabetical order by student name. How to do such ordering is considered in the next chapter (Section 8.4). For cases when the tree may change size or shape drastically, the sequential representation is not appropriate. An example of this situation is found in the integer or word counting problem of Section 7.1.


Data Structures Using C: 7.4.2 Linked Representation

7.4.2 Linked Representation

A binary tree data structure can also be implemented by representing each node of the tree as a record with three fields:

1. A left pointer field,

2. An information field

3. A right pointer field

(called here the leftptr, info, and rightptr fields). This is the most common representation. The left and right pointer fields link records of the tree, just as the link and sublist fields of lists link their records. In effect, the pointer fields specify a node's left and right subtrees, just as the sublist field of a list specifies its sublist. The leftptr and rightptr fields of a node point to the record that represents, respectively, the root of the node's left and right subtree. The info field contains the information stored at the node. A variable referred to as the name of the tree points to the record representing the root of the tree, and a null pointer in the name denotes the null tree.

We can store each record of a binary tree using an array of records as shown in Figure 7.10, just as was done for the records of lists. The binary tree fred, for example, has its root stored in position 5 of the array, while the root of t is in position 2. the left successor of fred's root is stored in position 8, and its right successor in position 0. A minus one is used as the null pointer.

Figure 7.10 Linked Representation for Binary Trees with Records Stored in an Array

The definitions for the records and the array declaration are as follows.

>Comment

typedef struct

{

whatever info;

int leftptr;

int rightptr;

}binarytreenode;

>Comment

typedef int binarytreepointer;

    >Comment

# define MAX 50

    >Comment

typedef binarytreenode recordsarray[MAX];

    >Comment

recordsarray records;

Instead of using an array and array indices, we may store the records of the tree in dynamic memory and use pointers. The record declarations then become

    >Comment

typedef struct treenode

{

whatever info;

struct treenode *leftptr;

struct treenode *rightptr;

    >Comment

}binarytreenode,*binarytreepointer;

Such linked representations require storage proportional to the number of nodes of the binary tree represented, no matter what the shape of the binary tree. Contrast this with the sequential representation of binary trees. If a linked representation uses three elements per record, whereas a sequential representation uses one element per record, then for complete binary trees three times as much storage is used in a linked representation as in a sequential one. If the binary tree is not complete, then the sequential representation may require storage proportional to 2d - 1, where d is the depth of the binary tree. This could result in significant amounts of wasted space; there may not even be enough storage. Consequently, the sequential representation is more practical only when dealing with complete or nonsparse binary trees.

Using the linked representation, it is obviously possible to insert or delete records, once we know where to do so, by making changes in only a few pointer fields.

The following program illustrates the binary tree implementation with records stored in dynamic memory. It is a program that could be used to test all the functions of Section 7.3 (although they have been renamed). The names are self-explanatory. The input consists of

An integer to indicate whether the binary tree being created is null,

A sequence of data for each record

A value for the info field of a new node

The sequence is assumed to be in the order in which the records are accessed in a preorder traversal of the binary tree being input. For example, if the input corresponded to the right subtree of Figure 7.9 and the new value replacing the first terminal node were 50, then the input would be as follows. (Program output and prompts are not shown.)

Typical Input

1    indicates nonnull binary tree

3    data for the root;

11   both its subtrees are nonnull

6    data for the next record in preorder;

11   both its subtrees are nonnull1

12   data for the next record in preorder;

00   both its subtrees are null1

13   data for the next record in preorder;

00   both its subtrees are null

7    data for the next record in preorder;

00   both its subtrees are null

50   the new value

The Program

Reads the data for nodes of a binary tree, creates

the tree and prints how many terminal nodes it

has and its depth, reads the value for the new node,

replaces the first terminal node in preorder access

by the new node, and then prints all the final binary

tree's nodes, its terminal count, and its depth.

#include <stdio.h>

    >Comment

typedef struct treerecord

{

int info;

struct treerecord *leftptr;

struct treerecord *rightptr;

}binarytreerecord,*binarytreepointer;

#define NULL 0

binarytreepointer left(p)

/* Returns a copy of the left

pointer of the node p points to.

*/

binarytreepointer p;

{

return(p->leftptr);

}

binarytreepointer right(p)

/* Returns a copy of the right

pointer of the node p points to.

*/

binarytreepointer p;

{

return(p->rightptr);

}

binarytreepointer setnull()

/* Returns a null pointer */

{

return(NULL);

}

binarytreepointer avail()

/* Returns a pointer to storage

allocated for a new node.

*/

{

return(malloc(sizeof(binarytreerecord)));

}

setinfo(p,pvalue)

/* Copies the contents of value

into the record p points to.

*/

binarytreepointer p;

int *pvalue;

{

p->info = *pvalue;

}

setleft(p,q)

/* Copies q into the left pointer

of the record p points to.

*/

binarytreepointer p,q;

{

p->leftptr = q;

}

setright(p,q)

/* Copies q into the right pointer

of the record p points to.

*/

binarytreepointer p,q;

{

p->rightptr = q;

}

printnode(pl,ptr)

/* Prints the info field of the

record ptr points to.

*/

binarytreepointer *pl,ptr;

{

printf("\n %d",ptr->info);

}

    >Comment

main()

/* Reads the data for nodes of a binary tree, creates

the tree and prints how many terminal nodes it

has and its depth, reads the value for the new node,

replaces the first terminal node in preorder access

by the new node, and then prints all the final

binary tree's nodes, its terminal count, and its

depth.

*/

{

int n,value;

binarytreepointer t,setnull(),avail();

printf("\n enter binarytree 0 for null tree 1\

otherwise \n");

scanf("%d",&n);

    >Comment

if (n == 0)

t = setnull();

else

{

    >Comment

t = avail();

createtree(&t);

}

    >Comment

printtree(&t);

    >Comment

terminalcount(&t);

    >Comment

printf("\n The depth is %d",depth(t));

printf("\n enter value ");

    >Comment

scanf("%d",&value);

    >Comment

replacefirstterminal(&t,value);

    >Comment

printtree(&t);

    >Comment

terminalcount(&t);

    >Comment

printf("\n The depth is %d",depth(t));

}

    >Comment

createtree(ptree)

/* Read the data and create the binary tree tree. */

binarytreepointer *ptree;

{

    >Comment

binarytreepointer l,r,setnull(),left(),right();

if (*ptree != setnull())

{

    >Comment

createnode(ptree);

l = left(*ptree);

    >Comment

createtree(&l);

r = right(*ptree);

    >Comment

createtree(&r);

}

}

createnode(pptr)

/* Reads the data, and fills in the fields

of the node pointed to by ptr.

*/

binarytreepointer *pptr;

{

int leftlink,rightlink,value;

binarytreepointer setnull(),avail();

printf("\n enter info \n");

    >Comment

scanf("%d",&value);

    >Comment

setinfo(*pptr,&value);

    >Comment

printf("\n enter enter left & right ptrs \n");

scanf("%d %d",&leftlink,&rightlink);

    >Comment

if (leftlink == 0)

setleft(*pptr,setnull());

else

setleft(*pptr,avail());

if (rightlink == 0)

setright(*pptr,setnull();

else

setright(*pptr,avail());

}

    >Comment

#define LIMIT 50

    >Comment

typedef binarytreepointer whatever;

typedef struct

{

whatever stackarray[LIMIT];

int top;

}stack;

#define TRUE 1

#define FALSE 0

setstack(ps)

/* Sets the stack s to empty. */

stack *ps;

{

(*ps).top = -1;

}

empty(ps)

/* Returns true only if stack s is empty.*/

stack *ps;

{

return((*ps).top == -1);

}

push(value,ps)

/* Inserts value at the top of stack s. */

whatever value;

stack *ps;

{

if ((*ps).top == (LIMIT - 1))

overflow(ps);

else

{

(*ps).top = (*ps).top + 1;

(*ps).stackarray[(*ps).top] = value;

}

}

pop(ps,pvalue)

/* Removes the top entry of stack s

and copies its contents into value.

*/

stack *ps;

whatever *pvalue;

{

if (empty(ps))

underflow(ps);

else

{

*pvalue = (*ps).stackarray[(*ps).top];

(*ps).top = (*ps).top - 1;

}

}

whatever item(ps)

/* Returns a copy of the top entry on stack s. */

stack *ps;

{

return((*ps).stackarray[(*ps).top]);

}

last(ps)

/* Returns true only if stack s

contains no non null pointers.

*/

stack *ps;

{

int i,temp;

binarytreepointer null,setnull();

null = setnull();

temp = TRUE;

for (i=0;i<=(*ps).top;i++)

if ((*ps).stackarray[i] != null)

temp = FALSE;

return(temp);

}

overflow(ps)

/* Prints a message if the stack overflows. */

stack *ps;

{

printf("\n stack overflow ");

}

underflow(ps)

/* Prints a message if the stack underflows. */

stack *ps;

{

printf("\n stack underflow ");

}

    >Comment

    >Comment

printtree(ptree)

/* Prints the info field values of all

nodes of the binary tree tree.

*/

binarytreepointer *ptree;

{

binarytreepointer null,p,q,rightpointer;

binarytreepointer setnull(),left(),right(),item();

stack s1;

null = setnull();

setstack(&s1);

p = *ptree;

while ((p != null) || !empty(&s1))

if (p != null)

{

    >Comment

printnode(ptree,p);

if (left(p) != null)

{

push(p,&s1);

p = left(p);

}

else

{

push(p,&s1);

p = right(p);

}

}

else

{

do

{

pop(&s1,&q);

if (!empty(&s1))

rightpointer = right(item(&s1));

else

rightpointer = null;

}while(!empty(&s1) &&

(q == rightpointer));

if (q != rightpointer)

p = rightpointer;

}

}

    >Comment

terminalcount(ptree)

/* Prints the number of terminal nodes

in the binary tree tree.

*/

binarytreepointer *ptree;

{

binarytreepointer p,null,setnull(),left(),right();

stack s2;

null = setnull();

setstack(&s2);

p = *ptree;

while ((p != null) || !empty(&s2))

if (p != null)

{

    >Comment

updatetncount(ptree,p,&s2);

push(right(p),&s2);

p = left(p);

}

else

pop(&s2,&p);

}

updatetncount(pt,p,ps)

/* Prints the number of terminal nodes in

binary tree tree when p points to the last

node in a preorder traverse of tree which

saves right pointers on stack s.

*/

binarytreepointer *pt,p;

stack *ps;

{

binarytreepointer null,setnull(),left(),right();

static int tncount;

null = setnull();

if (p == *pt)

tncount = O;

if ((left(p) == null) && (right(p) == null))

{

tncount++;

if (last(ps))

printf("\n The number of terminal nodes\

is %d",tncount);

}

}

    >Comment

replacefirstterminal(ptree,value)

/* Replaces the first terminal node, encountered

in a preorder traversal of binary tree tree, by a

new node with its info field set to the contents

of value.

*/

binarytreepointer *ptree;

int value;

{

binarytreepointer null,p,q,rightpointer;

binarytreepointer setnull(),left(),right(),item();

stack s3;

null = setnull();

setstack(&s3);

p = *ptree;

while ((p != null) | | !empty(&s3))

if (p != null)

{

replacenode(ptree,p,null,&s$,value);

if (left (p) != null)

{

push(p,&s3);

p = left(p);

}

else

{

push(p,&s3);

p = right(p);

}

}

else

{

do

{

pop(&s3,&q);

if (!empty(&s3))

rightpointer = right(item(&s3));

else

rightpointer = null;

}while (!empty(&s3) &&

(q == rightpointer));

if (q != rightpointer)

p = right pointer;

}

}

replacenode(ptree,p,null,ps,value)

/* If p points to the first terminal node, encountered

in a preorder traversal of binary tree tree that

saves the path to the root on stack s, then

replaces that node by a new node whose info field is

set to the contents of value.

*/

binarytreepointer *ptree,p,null;

stack *ps;

int value;

{

binarytreepointer q,ptr,avail(),left(),right();

if ((left(p) == null) && (right(p) == null))

{

ptr = avail();

setinfo(ptr,&value);

setleft(ptr,null);

setright(ptr,null);

if (p != *ptree)

{

q = item(ps);

if (left(q) == p)

setleft(q,ptr);

else

setright(q,ptr);

}

else

*ptree = ptr;

setstack(ps);

}

}

depth(tree)

/* Returns the depth of the binary tree tree. */

binarytreepointer tree;

{

binarytreepointer setnull(),left(),right();

if (tree == setnull())

return(0);

else

return(1 + max(depth(left(tree)),

depth(right(tree))));

}

max(i,j)

/* Returns the maximum of i and j. */

int i,j;

{

if (i >= j)

return(i);

else

return(j);

}


Data Structures Using C: 7.4.3 List-structure Representation

7.4.3 List-structure Representation

Another way to represent a binary tree is indicated in Figure 7.11. Each node's record is made complex. A complex record's sublist points to the representation of the node's left subtree. The result is that btree is implemented as the list structure lstree. Abstractly, binary trees are a special case of list-structures, but there are list-structures that do not correspond to binary trees. However, in computer science the term tree usually implies zero or restricted sharing of storage, whereas the term list-structure or list implies the possibility of more general sharing of storage.


Data Structures Using C: 7.5: Trees

7.5: Trees

So far we have considered only binary trees, but trees may have nodes with more than two successors. This is the case in the tree of Figure 7.1, where the nodes have up to three successors. It would be natural to assume that binary trees are a special case of trees, but the tree literature does not treat them this way. The formal definition of a tree, as opposed to a binary tree, is as follows: A tree is a root node with subtrees t1, t2, . . . , tn that are themselves trees and have no nodes in common.

Formally there are no null trees (although there are null binary trees), so each tree has at least one node, its root. It is convenient nonetheless to refer to null trees, so the terminology applied to binary trees also applies to trees. The important distinction is that no order is imposed on the subtrees of a tree, whereas for binary trees there is a distinct left and right subtree. Consequently, when trees are depicted, it makes no difference in what order subtrees are drawn; any order represents the same tree. When order among the subtrees is to be imposed, they are called ordered trees. For example the structures in Figure 7.12, when viewed as binary trees, represent two distinct binary trees, since their left and right subtrees differ, but they represent the same tree when viewed as trees.

Figure 7.11 (a) List-structure Representation of (b) a Binary Tree

Figure 7.12 Two Distinct Binary Trees or One General Tree

7.5.1 Implementing Trees as Binary Trees


Data Structures Using C: 7.5.1 Implementing Trees as Binary Trees

7.5.1 Implementing Trees as Binary Trees

When a constraint is imposed limiting the number of successors of any node to at most k, the tree is called a k-ary tree. Such trees may be implemented by generalizing the sequential representation for binary trees. The nodes are numbered as before, except the jth successor of the node numbered i is now assigned node number k � (i - 1) + (j + 1). Thus, when k = 5, the fourth successor (j = 4) of the node numbered i (i = 3) is assigned the number 5*(3 - 1) + (4 + 1) = 15. A 5-ary tree with nodes numbered in this way is

This implementation will not be pursued further but is similar in character to the sequential implementation for binary trees. This means that formulas can be derived to compute the array position of a node's predecessor and successors, which are important requirements for tree processing. Also, this representation is convenient for essentially static, healthy-looking (nonparse) trees.

Another representation is based on a natural generalization of the linked representation for binary trees. Simply expand each record so that it has as many pointer fields as it has subtrees. However, this leads to variable-length records, which are usually not desirable. The variable-length records occur because nodes need not all have the same number of successors. Further, it may not even be possible to predict in advance what the greatest number of successors will be.When it is known that the tree will be a k-ary tree, then all records could be made the same length, large enough to accommodate k subtree pointers. This gives a fixed record size but can result in considerable wasted storage for unused pointer fields within records. This is particularly true for sparse trees.

Surprisingly, there is a way to represent a tree as a binary tree, which then allows all records to be of fixed length. This is an important result. Since any tree can be represented by a binary tree, studying only binary trees is really no restriction at all.

Example 7.4

Let us now look at how to represent a specific tree as a binary tree. Consider the tree of Figure 7.13(a). How can this tree be represented as a binary tree? n

The binary tree is obtained as follows: First, create a root A of the binary tree to correspond to the root A of the tree. Next, pick any successor of A in the tree�say, B�and create a node B in the binary tree as A's left successor. Then take the remaining siblings of B, which are C and D in this case, and let C be B's right successor and D be C's right successor, in the binary tree. So far this yields the binary tree shown in Figure 7.13(b). If we view this procedure as the processing of node A of the tree, then the binary tree is completed by repeating this procedure for every other node of the tree. For example, after node B is processed (that is, its successors have been added to the binary tree), the result is Figure 7.13(c). The final binary tree created using this procedure appears in Figure 7.13(d).

(a) Initial Tree

(b) Binary Tree for Nodes A - D

(c) Binary Tree for Nodes A - G

(d) Final Binary Tree

Figure 7.13 Representation of a General Tree as a Binary Tree

Applying this procedure, given any tree, a corresponding binary tree can be constructed. The specific binary tree generated will depend on the order in which successors are taken in the tree. For instance, had C been selected initially as the first successor of A, then the resultant binary tree would be different. If the ordering of each node's successors is specified and they are taken in the specified order, then the construction creates a unique binary tree for each tree. In any case, a tree with n nodes will correspond to a binary tree with n nodes. It is not difficult to find an algorithm that, given a binary tree with no right subtree, can reverse the construction and generate the tree it represents. This means that when the tree is ordered, there is a unique binary tree that can be constructed to represent it. Conversely, given a binary tree with no right subtree, there is a unique tree that it represents, which can also be constructed. Hence there are exactly as many trees with n nodes as there are binary trees with n nodes and no right subtrees.

If we define an ordered forest as an ordered collection of ordered trees, then each tree has its unique binary tree representation. Can you see how to append these binary trees to the binary tree representing the first ordered tree of the ordered forest to obtain a binary tree representing the forest? (Hint: Use the fact that the binary trees have no right subtrees.) For those of you who wish to count trees, pursuing this should lead to the conclusion that there are exactly as many ordered forests as there are binary trees.


Data Structures Using C: 7.6: Traversals of Trees

7.6: Traversals of Trees

In this section algorithms are developed for traversing trees and are applied to two examples. The second example should be studied carefully, because it illustrates an important modified traversal of trees that has many applications. Assuming the trees are ordered, it makes sense to refer to a tree's first subtree, second subtree, and so on. If an order is not given, the programmer can impose one.

When trees are used to store data at their nodes, then tree traversals give a means to access all the data. However, a tree might be used to represent a maze or all routes from Philadelphia to San Francisco. Traversal of the tree then provides the means to find exits from the maze or the shortest route across the country. The maze and the routes must not intersect. When they do, a more general data structure, a graph, is needed for their representation. Unlike trees, a graph allows more than one path between its nodes. * Should a tree contain family relationships by storing an individual's record at a node, a traversal allows the parent (predecessor) or the children (successors) of an individual to be found.

* It is not difficult to generalize tree traversal algorithms to obtain graph traversal algorithms.

The preorder and postorder traversals of binary trees generalize quite naturally to ordered trees.

To preorder traverse an ordered tree:

1. Access and process its root, then

2. Preorder traverse its subtrees in order.

To postorder traverse an ordered tree:

1. Postorder traverse its subtrees in order, then

2. Access and process its root.

There is no natural inorder traversal of trees. Preorder and postorder traversals of a tree correspond, respectively, to preorder and inorder traversals of its corresponding binary tree. The preorder and postorder access sequences for the tree of Figure 7.13(a) are ABEFGCHDIKLMJ and EFGBHCKLMIJDA. You should confirm that a preorder and an inorder traversal of the binary tree of Figure 7.13 result in these respective access sequences. A preorder traversal is also called a depth-first traversal.

7.6.1 Obtaining the Binary Tree for a Tree

7.6.2 Backtracking: The n-queens Problem

7.6.3 Depth-first

7.6.4 Breadth-first

7.6.5 Branch and Bound


Data Structures Using C: 7.6.1 Obtaining the Binary Tree for a Tree

7.6.1 Obtaining the Binary Tree for a Tree

It should now be clear that confining attention to binary trees, their representation, and operations on them is not actually a restriction. That is, since trees can be represented as binary trees, any operations can be performed on them by performing equivalent operations on their binary tree representations. In this sense no generality is lost in dealing with binary trees.

Example 7.5

Let the task for this example be the development of an algorithm to generate the binary tree corresponding to a general tree. n

In order to do this we must be able to traverse a general tree. As it is traversed, we can generate the binary tree by properly processing each node as it is accessed. A nonrecursive algorithm to preorder traverse a tree t follows.

Nonrecursive Tree Preorder Traversal

1. Set p to t

2. Setstack_t(ts)

3. While p is not null or not empty_t(ts),

if p is not null, then

process(t,p)

For each successor pointer ps of node.p,

except next(p),

push_t(ps,ts)

Update p to next(p)

else

pop_t(ts,p)

This algorithm is a generalization of the nonrecursive preorder traversal of a binary tree that saves right pointers. In the binary tree case, the next subtree is the left subtree, and only one successor of the node just processed must be saved�the right successor. The difference is that now all the subtree pointers of the node just processed, except for the next one, must be saved.

Now that we have a general tree traversal, it will be adapted to our task. Let bt point to the binary tree to be generated by the algorithm. Recall the list-structure representation of a binary tree. Think of the leftpointer field of each node of bt as pointing to a sublist. This sublist contains all the successors of the corresponding node in t. Thus, in Figure 7.13(d), node A of the binary tree representation has a leftpointer field pointing to the sublist consisting of nodes B, C, and D. These are the successors of A in the general tree, Figure 7.13(a). Similarly, D has a leftpointer field pointing to the sublist consisting of I and J, the successors of D in the general tree.

Consequently, to produce bt, when a node of t is processed we must

Create a sublist consisting of all its successors

Set the leftpointer field of the corresponding node in bt to point to this sublist

Figure 7.14 Binary Tree BT Generated from General Tree T

Copy the info field of the node in t into the info field of the corresponding node in bt

The rightpointer fields of these nodes in bt serve as the link fields of the sublist.

Recall that, as we preorder traverse a general tree t, we are preorder traversing its corresponding binary tree, bt. That is, the order in which the nodes of t are accessed is exactly the same as the order in which the corresponding nodes of bt are accessed. Thus we can achieve a solution by preorder traversing both t and bt at the same time, and processing the corresponding nodes as described above. For example, if this were carried out on the general and final binary tree of Figure 7.13, and nodes A, B, E, F, G, and C had been processed, we would have generated the binary tree shown as Figure 7.14(b).

Suppose p points to the node currently being processed in t. Let pb point to the corresponding node in bt. In the example, p would now point to D of t and pb to D of bt.

After processing the node to which p points, we can update p to point to one of its successors, next(p), and stack on ts all pointers to the remaining successors. These pointers represent the latest postponed obligations for the preorder traversal of t. Next (p) is a function returning a pointer to the successor of p that is not stacked, or a null pointer if there are no successors. At the same time, to keep track of the postponed obligations for bt's preorder traversal, we will keep a parallel stack, bts. Only nonnull pointers will be pushed onto bts. When the top pointer in stack ts is popped, the top pointer in bts will also be popped. These pointers will then point to corresponding nodes�one in t, the other in bt. Storage can be created for the nodes of bt as the traversal proceeds by using avail to return a pointer to the available storage for a node, since a pointer variable implementation is used for the binary tree.

One last detail remains. In order to do the processing required on node.p and node.pb, process will be invoked. As already noted, process must create a sublist consisting of new nodes corresponding to the successors of node.p in t, set leftptr.pb to point to that sublist, and copy info.p into info.pb. We assume that a procedure create, given p, creates the required sublist, and returns with first pointing to the first node of that sublist. The preorder traversal algorithm to create the binary tree corresponding to a general tree is as follows:

Nonrecursive Algorithm to Generate a Binary Tree Corresponding to a General Tree

1. Set p to t

If p is null

Set bt to null

else

bt = avail()

Set pb to bt

2. Setstack_t(ts)

Setstack_bt(bts)

3. While p is not null or not empty_t(ts),

if p is not null, then

process(t,bt,p,pb).

For each successor pointer ps of node.p, except next(p),

push_t(ps,ts)

If right(pb) is not null

push_bt(right(pb),bts)

p = next(p)

pb = left(pb)

else

pop_t(ts,p)

pop_bt(bts,pb).

The process algorithm to be used in the preorder traversal is:

Create(p,first)

Set leftptr.pb to first

Set info.pb to info.p

Notice that the traversal of the binary tree bt is embedded in the traversal of the general tree t. Also, suffixes have been used to distinguish the stack storing pointers to nodes of the general tree from the stack storing pointers to nodes of the binary tree. Finally, since the tree t is not ordered, the function next is free to pick any successor. The order in which the rest of the successors are placed on the stack is arbitrary but must determine the order in which create places them on the sublist. You should simulate this algorithm on the tree of Figure 7.13(a) to see how it works.

The correspondence between an ordered tree and its binary tree can now be defined more formally. In this case the binary tree corresponding to a tree is unique. An ordered tree t may be represented by a binary tree bt as follows:

Each node of t corresponds to a node of bt,

If node n of t corresponds to node nb of bt, then the leftmost successor of n in t corresponds to the left successor of nb in bt;

All other siblings of n in t form a right chain from the node in bt corresponding to the leftmost successor of n in t.

We may now give a recursive definition specifying an algorithm for generating the binary tree bt representing the general ordered tree t.

To generate a binary tree corresponding to an ordered tree:

If t is null

Set bt to null

else

Create the root node of bt corresponding to the root of t.

If t has a subtree, then generate the binary tree bt.left corresponding to the leftmost subtree of t,t.leftmost, and make the generated binary tree the left subtree of bt.

For every other subtree of t, in order, generate the binary tree corresponding to the subtree, and insert it at the end of the right chain of bt.left.

For illustration this algorithm is applied to the ordered tree in t in Figure 7.15(a).

First create the root of BT corresponding to the root of t shown in Figure 7.15(b). To carry out statement 2, note that t has three subtrees with B at the root of its leftmost subtree. We must thus generate bt.left to orrespond to Figure 7.15(c). To do this, apply the definition to this tree and obtain Figure 7.15(d) as its corresponding binary tree. Completing the first part of statement 2 for t yields the binary tree shown in Figure 7.15(e).

Figure 7.15 Generation of Binary Tree BT from an Ordered Tree T

To complete the second part of statement 2, generate the binary tree for the next subtree of t, with C at its root, and then insert it at the end of the right chain of B to obtain Figure 7.15(f). Finally, do the same for the last subtree of A, with D at its root, and complete the application to t, to obtain the result shown in Figure 7.15(g).

This is another example for which we found a recursive solution. Compare the clarity and conciseness of the recursive solution with that of the nonrecursive solution.


Data Structures Using C: 7.6.2 Backtracking: The n-queens Problem

7.6.2 Backtracking: The n-queens Problem

In the last section a tree traversal was developed and used to generate a binary tree. Here and in the following two sections we will develop a modified tree traversal that employs backtracking. Backtracking is a useful technique in solving many problems. First we apply it to the n-queens problem.

Chess is played on a square board with eight rows and eight columns, just like a checkerboard. We can think of the board as a two-dimensional array, board (Figure 7.16).

A queen located in the jth row and kth column may move to a new position in one of the following ways:

n To the right, or left, along its row

n Up, or down, in its column

n Up, or down, along its d1 diagonal

n Up, or down, along its d2 diagonal

Figure 7.16 A Partial Chessboard Represented as a Two-Dimensional Array, BOARD

The n-queens problem is to find all ways to place n queens on an n � n checkerboard so that no queen may take another. As an example, the eight-queens problem is to find all ways to place eight queens on the chessboard so that no queen may take any other梩hat is, move into a position on its next move that is occupied by any other queen.

The question is how to solve the n-queens problem. At first glance this problem has little relation to trees, but let us see. It is not even clear that a solution exists for all n.

Since a queen can be moved along one of its rows, columns, or diagonals, a solution clearly is achieved by specifying how to place the n queens so that exactly one queen appears in each row and column, and no more than one queen appears on any diagonal of the board. One could attempt to find a solution by searching through all possible placements of the n queens satisfying these restrictions. There are, however, n! such placements. Even for a moderate n, this is obviously not a practical solution. Instead we must construct a solution. The idea behind the construction is to determine if the placements obtained for the first i queens can lead to a solution. If they cannot, abandon that construction and try another, since placing the remaining n - i queens will not be fruitful. This may avoid the need to consider all possible constructions.

To construct a solution involves testing a partial construction to determine whether or not it can lead to a solution or must be abandoned as a dead end. This testing is the next question.

Consider a general tree, with each of its nodes representing a sequence of decisions on the placement of queens. Take the root to represent the initial situation, in which no decisions have yet been made. The root will have n successors. Each successor corresponds to a choice of row for the placement of the first queen. Each of these successor nodes, in turn, has n successors, corresponding to a choice of row for the placement of the second queen, and so on. The tree for the four-queens problem is shown in Figure 7.17.

Since exactly one queen must appear in each column in any solution, assume the ith queen is placed in column i. Hence, each path, from the root to a node at depth i in the tree, specifies a partial construction in which the first i queens have been placed on the board. The path indicated in the four-queens tree specifies a partial construction in which the first, second, and third queens have been placed in rows 2, 4, and 1 (and columns 1, 2, and 3), respectively. The tree helps us visualize all possible solutions and provides a framework in which to view what we are doing.

Suppose a path has been found to a node at depth k. The path is feasible if none of the k queens whose positions are specified by the path can take any other. If the node at depth k is a terminal node, and the path is feasible, then a solution has been found and may be printed. If the node at depth k is not terminal, and the path is feasible, we want to extend the path to a node at depth k + 1. Let p point to such a node. Then node.p must be a successor of the node on the path at depth k.

Figure 7.17 Tree for the Four-queens Problem

P is feasible if the position that node. p specifies for the (k + 1)th queen does not allow it to take any of the other k queens. If p is feasible, the path can be extended downward in the tree to include node.p, so there is now a feasible path of depth k + 1. If p is not feasible, then another successor of the node at depth k must be tried until a feasible one is found to extend the path, or until all have been tried and none is feasible. In the latter case, the choice of node at depth k cannot be extended to a solution, and the computer backs up in the tree to a node representing a shorter path that has not yet been explored. This procedure to extend the new path is repeated until a solution is found or until all possibilities have been tried.

The procedure described amounts to a modified general tree traversal called backtracking. A different approach would be to create an algorithm for the problem by applying a general tree preorder traversal that uses a process routine. The process routine would check each node to be processed to see whether the node is terminal and represents a feasible path. If so, it prints the solution. This approach amounts to an exhaustive search of all n! possibilities, since the preorder traversal backs up only when the preorder traversal of a subtree has been completed�that is, after a terminal node has been processed. The backtracking traversal need not access and process all nodes. It backs up when the traversal of a subtree has been completed or when it becomes known that no path involving the root of a subtree can be extended to a solution. The backtracking procedure generates only feasible paths�not all paths. In effect, it prunes the tree by ignoring all nodes that cannot lead to a solution.


Data Structures Using C: 7.6.3 Depth-first

7.6.3 Depth-first

A backtracking algorithm can be derived by modifying the general tree preorder traversal algorithm. The "p not null" test in the loop body of the preorder traversal algorithm must be replaced by two tests. One to see if p points to an existing node and another "p feasible" test to prune the tree. How this feasibility test is implemented is what determines the amount of pruning that occurs and hence the efficiency of the solution. The process routine called in the loop body must determine if node.p is terminal, and, if so, print the solution. If these were the only changes made, then nothing at all would be printed when no solution exists.

You, as the programmer, do not know whether a solution exists for every n. Therefore, initialize a flag, found, to false, and have process set found to true if it finds a solution. Upon exiting from the loop body, test found, and if it is false print "No solution." The resultant algorithm for a backtracking preorder traversal is as follows:

Backtracking Preorder Tree Traversal

1. Set p to the root of t.

2. Set found to false.

3. Set the stack to empty.

4. While p is not null or the stack is not empty,

if p is not null, then

if p is feasible, then

process(p),

push all successors of node.p onto the stack except next(p),

move down in t by setting p to next(p),

else

backtrack in t by popping the stack and setting p to the popped value,

else

backtrack in t by popping the stack and setting p to the popped value;

if not found then

print no solutions.

The function next(p) is assumed to return a null value if node.p has no successors. If only one solution is to be found and printed, only the test of the while loop must be changed. It should become "(p not null or stack not empty) and (not found)." The process algorithm to be used with the backtracking preorder traversal is as follows:

If node.p is a terminal node, then

1. Print the solution, and

2. Set found to true.

The algorithms developed in this section are quite general. They are applicable to any problem that can be reduced to a backtracking traversal (like the n-queens problem). They should be viewed as general tools for problem solving. We must now specialize them to the n-queens problem so that they are detailed enough to implement as a function. Data structure decisions, specifically the implementation of the current board configuration, must be made along the way.

The tree is not actually stored in memory; it was used as a conceptual aid in formulating the problem abstractly. As a result, the problem has been recognized as one that may be solved by a general tree backtracking procedure. It is not necessary to refer to the tree at all, but we have done so here to place the problem in this more general context, and we continue with this approach.

Suppose p points to a node at depth k. The depth of the node specifies the placement for the kth queen, column k. P itself determines the choice of row. In this way p corresponds to a particular row and col pair and vice versa. A nonnull value for p corresponds to a column and row value <n + 1. Initializing p to the first successor of the root corresponds to setting row to 1 and col to 1.

Next must have row and col as parameters; it simply increases col by 1 and sets row to 1. As the traversal proceeds, row and col vary in a regular way. This can be used to advantage. Instead of pushing all successors of node.p onto the stack, which would involve n - 1 entries, simply push the current value of row. Then the backtracking task can be carried out by adding 1 to the current value of row when this value is less than n. If it equals n, the stack must be popped, row set to the popped value plus 1, and col set to col-1.

To implement the crucial test, "p feasible," we use a function feasible to return the value true when p is feasible and false otherwise. Feasible must have row and col as parameters and must have access to information about the current board configuration. It must determine if placing a queen in the position specified by row and col will allow it to "take" one of the currently placed queens. If a queen may be taken, it must return "false," otherwise "true."

We could use an n � n two-dimensional array board to keep track of the current board configuration. If we did this, feasible would have to traverse through the row, column, and diagonals specified by row and col in order to determine what value to return. This checking must involve all entries in many rows and columns. Since feasible is invoked in the loop body of the backtracking preorder traversal algorithm, it is executed once for every node of the tree that is accessed. Operations that appear in loop bodies should be done efficiently since they have a significant effect on the overall efficiency of the implementation.

To do its job, feasible must know whether or not the row, col, and the diagonals specified by row and col are occupied. If this information can be efficiently extracted and made available, feasible will be more efficient. We now see exactly what information is required and proceed to the details involved in efficiently extracting it.

Consider Figure 7.16. Notice that entries [j, k], [j + 1, k - 1], and [j - 1, k +1] all lie along the d1 diagonal. Notice also that j+ k = (j+ 1) + (k - 1) = (j- 1) + (k + 1). This means that all entries on the diagonal d1, determined by j and k, have the same row + col sum: j + k. The same is true of differences along d2. That is, all entries on the diagonal d2, determined by j and k, have the same row minus col difference: j - k. Suppose we keep information about the current board configuration as follows:

r[j]

Is true if the current board configuration has no queen in the jth row, and is false otherwise.

d1[j + k]

Is true if the current board configuration has no queen along the diagonal d1 with row + col sum j + k, and false otherwise.

d2[j - k]

Is true if the current board configuration has no queen along the diagonal d2 with row - col sum j - k, and false otherwise.

Feasible can now simply return the value of (r[j] and d1 [j + k] and d2[j - k]). The arrays r, d1, and d2 require storage for n, 2(n - 1), and 2(n - 1) entries, respectively, and must be properly initialized, but feasible now takes constant time to execute. Also, when row and col are changed, these changes must be reflected in r, d1, and d2. What we have done amounts to finding a way to store the current board configuration so that the operations performed on it by the algorithm are done efficiently. R, d1, and d2 have indices that range from 1 to n, 2 to 2n, and -(n - 1) to (n - 1), respectively. Since C does not allow such array indices, it is necessary to refer in the program below to corresponding entries in arrays whose indices will range from 0 to n - 1, 0 to 2(n - 1), and 0 to 2(n - 1), respectively.

Finally, we must implement the printing of a solution in process . This can be done by printing the sequence of values stored on the stack. They specify the rows in which queens 1 to n were placed. To do this we will need a stack operation item. It returns a copy of the ith stack entry, which represents the row in which the (i+1)th placed queen (the queen in column i) appears. When i is zero, it returns a copy of the top entry.

The backtracking traversal algorithm that was developed and applied to this problem is a general tool for problem solving. It may be adapted, for example, to solving a maze, finding good game-playing strategies, and translating high-level language programs.

The nonrecursive solution may be written as follows.

Nonrecursive n-queens Program

# include <stdio.h>

main()

/* Reads in n and prints all solutions

to the n-queens problem.

*/

{

int n;

printf("\n n = ?");

scanf("%d",&n);

queens(n);

}

    >Comment

#define NLIMIT 20

typedef int rowcheck[NLIMIT];

typedef int diagonalcheck1[2*NLIMIT-1];

typedef int diagonalcheck2[2*NLIMIT-1];

    >Comment

#define SLIMIT 20

typedef int whatever;

typedef struct

{

whatever stackarray[SLIMIT�1];

int top;

}stack;

setstack(ps)

/* Sets stack s to empty. */

stack *ps;

{

(*ps).top = �1;

}

empty(ps)

/* Returns true only if stack s is empty. */

stack *ps;

{

return((*ps).top == -1);

}

push(value,ps)

/* Inserts contents of value as

the top entry on stack s.

/*

whatever value;

stack *ps;

{

if((*ps).top == (SLIMIT-1))

overflow(ps);

else

{

(*ps).top = (*ps).top + 1;

(*ps).stackarray[(*ps).top] = value;

}

}

pop(ps,pvalue)

/* Removes the top entry of stack s

and copies its contents into value.

*/

stack *ps;

whatever *pvalue;

{

if(empty(ps))

underflow(ps);

else

{

*pvalue = (*ps).stackarray[(*ps).top];

(*ps).top = (*ps).top - 1;

}

}

whatever item(i,ps)

/* Returns a copy of the ith

entry in stack s. When i is

zero it returns a copy of

the top entry.

*/

stack *ps;

{

return((*ps).stackarray[(*ps).top-i]);

}

overflow(ps)

/* Prints a message if the stack overflows. */

stack *ps;

{

printf("\n stack overflow ");

}

underflow(ps)

/* Prints a message if the stack underflows. */

stack *ps;

{

printf("\n stack underflow ");

}

queens(n)

1 a backtracking preorder traversal

/* Prints all solutions to the n-queens problem */

int n;

{

stack s;

#define TRUE 1

#define FALSE 0

int i,row,col,found;

rowcheck r;

3 r, d1, and d2 will contain the board configuration

diagonalcheck1 d1;

diagonalcheck2 d2;

col = 1;

2 col and row determine the next placement attempt

row = 1;

    >Comment

for(i=0;i<=n-1;i++)

r[i] = TRUE;

for(i=0;i<=2*n-2;i++)

d1[i] = TRUE;

for(i=0;i<=2*n-2;i++)

d2[i] = TRUE;

found = FALSE;

setstack(&s);

    >Comment

while(((col < n+1)&&(row < n+1)) || !empty(&s))

    >Comment

if((col < n+1)&&(row < n+1))

    >Comment

if(feasible(row,col,r,d1,d2,n))

{

    >Comment

process(row,col,&found,&s,n);

    >Comment

push(row,&s);

    >Comment

r[row-1] = FALSE;

d1[row+col-2] = FALSE;

d2[row-col+n-1] = FALSE;

    >Comment

col = col + 1;

    >Comment

row = 1;

}

    >Comment

else

    >Comment

row = row + 1;

    >Comment

else

{

    >Comment

pop(&s,&row);

col = col - 1;

    >Comment

r[row-1] = TRUE

d1[row+col-2] = TRUE;

d2[row-col+n-1] = TRUE;

    >Comment

row = row + 1;

}

if(!found)

printf("\n NO SOLUTIONS");

}

feasible(row,col,r,d1,d2,n)

/* Returns true only if the placement of the

next queen in position col,row does not

allow any queen to take another under

the current board configuration given

by r,d1,d2.

*/

int row,col, n;

rowcheck r;

diagonalcheck1 d1;

diagonalcheck2 d2;

{

return(r[row-1]&&d1[row+col-2]&&d2[row-col+n-1]);

}

process(row,col,pfound,ps,n)

/* If the partial solution is a complete

solution then it is printed and

found is set to true.

*/

int row,col,*pfound,n;

stack *ps;

{

int i;

whatever item();

if(col == n)

{

for(i=n-1;i>=1;i--)

    >Comment

printf("\n COL %d ROW is %d----",

n-1,item(i-1,ps));

    >Comment

printf("\n COL %d ROW is %d----\n",

n,row);

*pfound = TRUE;

}

}

You may be tempted to try a more analytic approach to this problem so as to eliminate the need for all this searching. Be forewarned that famous mathematicians also attempted to analyze this problem but had difficulty solving it other than by backtracking! Actually, there are no solutions for n = 1, 2, and 3, and at least one solution for all n�4. For the curious, the number of solutions for n up to 15 are:

    N      4      5          6    7      8       9         10

Solutions  2      10         4    40    92      352        724

    N        11        12         13        14          15

Solutions  2,680     14,200     73,712    365,596    2,279,184


Data Structures Using C: 7.6.4 Breadth-first

7.6.4 Breadth-first

One final point about the backtracking traversal algorithm. Suppose the while loop task is replaced by the task:

if p is not null, then

if p is feasible, then

process(p)

push all successors of node.p onto the stack

pop(sp)

else

pop(s,p).

This implementation also represents a preorder tree traversal. The stack is used to store and recall postponed obligations. It can be further modified by changing the stack to a queue and replacing empty, push, and pop by the corresponding queue operations. Pop must return null if called when the queue is empty. The result is another modified general tree traversal, a backtracking breadth-first traversal. It accesses the nodes in level order. In contrast, the backtracking preorder traversal presented earlier might be called a backtracking depth-first traversal. Removing the "p is feasible" test of the backtracking breadth-first traversal turns it into a breadth-first traversal. You should convince yourself that this version leads to the nodes of Figure 7.13(a) being accessed and processed in the order A, B, C, D, E, F, G, H, I, J, K, L, M. Keep in mind that in this application the root node was never considered. The difference between a backtracking traversal and a traversal is that backtracking tests for feasibility. This allows tree paths to be ignored when it can be determined that they cannot lead to a solution.


Data Structures Using C: 7.6.5 Branch and Bound

7.6.5 Branch and Bound

There is another type of modified tree traversal which is often found useful. It is called a branch and bound traversal. An algorithm for this traversal is as follows:

Nonrecursive Branch and Bound Traversal

1. Set p to t.

2. Set bag to empty.

3. While p is not null

if p is feasible, then

i. process(t,p), and

ii. add each successor pointer of p to the bag;

Select a pointer from the bag and set p to it.

This algorithm uses a data abstraction consisting of a bag data structure and operations on the bag that set it to empty, test it for emptiness, select (and remove) a pointer from it, and add a pointer to it. The idea is to define the select operation so that it removes the pointer from the bag that points to the node most likely to lead to a solution. The execution time required by the branch and bound traversal is directly related to how well the select function does the job of picking the next node to process and how well the feasible function does its job of pruning the tree. The better the selection and pruning the better the execution time. The select function should return null if the bag is empty.

If the select function always selects the node most recently added to the bag, then the data abstraction becomes a stack, and the algorithm is turned into a backtracking depth-first traversal. If the select function always selects the node that was added earliest to the bag, then the data abstraction becomes a queue, and the algorithm is turned into a backtracking breadth-first traversal. Thus backtracking may be viewed as a special case of branch and bound.

Horowitz and Sakni [1978] contains extensive discussions of backtracking and branch and bound traversals, with many applications worked out in detail. Also see Golomb and Baumert [1965]. Wirth [1976] deals with these techniques for problem solving; in particular, the stable marriage problem and the eight-queens problems are solved recursively.


Data Structures Using C: 7.7: More on Recursion, Trees, and Stacks

7.7: More on Recursion, Trees, and Stacks

Now that we are steeped in trees, we can look back at Figures 4.2 and 4.4 and notice that they are pictures of trees. We shall call them the execution trees of their corresponding recursive programs. In fact, any recursive program can be viewed as generating such a tree as it executes. The recursive program can be thought of as executing by traversing its corresponding execution tree. The processing done at each node in the course of this traversal is the carrying out of the program's task on a specific problem specified by its data and current scratchpad. Now that you know a good deal about tree traversals, is it any wonder that the stack appears in the translation of a recursive program? It is clearly the number of nodes in the tree that determines the execution time of the algorithm, and it is the depth of the tree that determines the storage requirements. This point can become somewhat muddled when recursion is applied to trees themselves, but you should realize that this is the case even when no trees are apparent in the nature of the problem, as in the Towers of Hanoi problem in Chapter 4.

Further, we can now see that recursion clarifies because it hides the bookkeeping required by the tree traversal and because it elucidates the structure that must be inherent in the problem in order for a recursive solution to be found in the first place. Although it is an extremely powerful tool for these reasons, it must be applied with care. The purpose of this section is to provide some insight into the nature of those problems where a more skeptical approach is wise.

(a) Balanced

(b) Unbalanced

Figure 7.18 Balanced and Unbalanced Binary Trees

7.7.1 Balanced Binary Trees

7.7.2 Trading Storage for Time


Data Structures Using C: 7.7.1 Balanced Binary Trees

7.7.1 Balanced Binary Trees

In a balanced binary tree, no matter what node of the tree is considered, the left and right subtrees of that node have depths that differ by at most 1.

The binary tree of Figure 7.18(a) is balanced; the binary tree of (b) is not. Balanced trees are important in information retrieval applications; they will be discussed fully in Chapter 9. In information retrieval, the data are stored at the tree's nodes. The depth of the tree determines the search times for retrieval and, with balanced trees, the tree's depth is controllable and can be limited, as will be shown in Chapter 9. Here we consider the "worst case" of such trees.

A binary tree is a Fibonacci tree if it is balanced and has the minimum number of nodes among all balanced binary trees with its depth. Fibonacci trees represent the worst case for balanced trees, since they contain the fewest nodes (and hence the least storage capacity) among all balanced trees with a specified depth. The binary tree in Figure 7.19 is a Fibonacci tree of depth 4. This is a Fibonacci tree because it is balanced, and no balanced binary tree with a depth of 4 has fewer than 7 nodes. Any complete binary tree of depth 4, for example, will not be a Fibonacci tree because, although balanced, it will have more than the minimum number of nodes (7) for this depth.

Figure 7.19 A Fibonacci Tree of Depth 4

The Fibonacci numbers are a well-known sequence of integers defined by the following recursive definition:

F0 = 0,   F1 = 1,

Fn = Fn-1 + Fn-2    for n > 1

The first few are 0, 1, 1, 2, 3, 5, 8, 13, 21, 34, . . . . These numbers seem to appear in the most unexpected places. In computer science they often arise in the analysis of algorithms. The number of nodes in a Fibonacci tree of depth d can be shown to be Fd+2 - 1 (see Exercise 34b). For example, if d is 4, then Fd+2 - 1 = F6 - 1 = 8 - 1 = 7.

Example 7.6

The task now is to develop an algorithm to generate a Fibonacci tree of depth d. If possible, apply the method of recursion to find a recursive definition of the solution. n

Clearly, if d = 0, there is only one Fibonacci tree of depth d, the null binary tree. If d = 1, there is only one Fibonacci tree of depth d, the tree with one node�its root.

Any subtree of a balanced binary tree must also be balanced, or else the tree itself would not be balanced. Any subtree of a Fibonacci tree must therefore be balanced and must also be a Fibonacci tree. Otherwise it could be replaced by a Fibonacci tree of that depth with fewer nodes. This is the key to the recursive solution. We can now construct a Fibonacci tree of depth d > 1 as follows: Since d > 1, the Fibonacci tree must have a root, and its two subtrees must be Fibonacci trees. One (say, the right) must have depth d - 1 (so the tree itself has depth d). The other, the left subtree, must be a Fibonacci tree and must differ in depth from d - 1 by at most 1. Hence it must have depth d - 2 or d - 1. It has just been demonstrated that a Fibonacci tree of depth d - 1 has at least one subtree of depth d - 2. Hence the Fibonacci tree of depth d - 1 has more nodes than the Fibonacci tree of depth d - 2. Thus the left subtree of the Fibonacci tree being constructed must have depth d - 2. (Why?) This gives the complete recursive solution.

To construct a Fibonacci tree of depth d:

1. If d = 0, then construct the null tree,

2. else if d = 1, then construct the tree with one node, its root,

3. else if d > 1, then

a. construct a root,

b. construct a Fibonacci tree of depth d - 2 and make it the left subtree of the root, and

c. construct a Fibonacci tree of depth d - 1 and make it the right subtree of the root.

The explicit constructions are given by the algorithm's statements 1 and 2 for d = 0 and d = 1. Statement 3 gives the implicit construction for d > 1. You should apply this definition to the case d = 4, and confirm that it constructs the example given earlier of a Fibonacci tree of depth 4. It should also be clear that other Fibonacci trees of depth 4 exist.

The recursive program can be written directly from the definition.

Recursive Version 1�Linked Representation with Records Stored in an Array

fibonaccitree (d, ptree)

/* Creates a Fibonacci tree, tree, of depth d. */

int d;

binarytreepointer *ptree;

{

binarytreepointer tl, tr, avail ( );

>Comment

*ptree = avail ( );

if (d == 0)

>Comment

*ptree = NULL;

else if (d == 1)

{

>Comment

records [*ptree].leftptr = NULL;

records [*ptree].rightptr = NULL;

}

else

{

>Comment

fibonaccitree (d-2,&tl);

records [*ptree].leftptr = tl;

fibonaccitree (d-1,&tr);

records [*ptree].rightptr = tr;

}

}

For concreteness, we have chosen to implement the constructed tree using a linked representation. The tree itself will be stored in the records array (Section 7.4.2) with the fields of a record being info, leftptr, and rightptr.Records is assumed to be a global variable. When the routine returns, tree will point to its root record in the array representation.

Using implementation with pointer variables for the binary tree, the procedure becomes the following.

Recursive Version 2�Linked Representation with Records Stored in Dynamic Memory

fibonaccitree(d,ptree)

/* Creates a Fibonacci tree, tree, of depth d. */

int d;

binarytreepointer *ptree;

{

binarytreepointer tl, tr;

>Comment

*ptree = malloc (sizeof(binarytreerecord));

if (d == 0)

>Comment

*ptree = NULL;

else if (d == 1)

{

>Comment

(*ptree)->leftptr = NULL;

(*ptree)->rightptr = NULL;

}

else

{

>Comment

fibonaccitree (d-2, &tl);

(*ptree)->leftptr = tl;

fibonaccitree (d-1, &tr);

(*ptree)->rightptr = tr;

}

}

Finally, for contrast, the function may be written treating the tree as a data abstraction, as follows.

Recursive Version 3�The Tree Is Treated as a Data Abstraction

fibonaccitree(d,ptree)

/* Creates a Fibonacci tree, tree, of depth d. */

int d;

binarytreepointer *ptree;

{

binarytreepointer null,tl,tr,avail( ),setnull( );

null = setnull( );

*ptree = avail( );

if (d == 0)

>Comment

*ptree = null;

else if (d == 1)

{

>Comment

setleft(*ptree, null);

setright(*ptree, null);

}

else

{

>Comment

fibonaccitree(d-2,&tl);

setleft(*ptree, tl);

fibonaccitree(d-1,&tr);

setleft(*ptree, tr);

}

}


Data Structures Using C: 7.7.2 Trading Storage for Time

7.7.2 Trading Storage for Time

Consider the execution tree for these functions when d = 6, which is given in Figure 7.20. Each node of the tree represents a call to the recursive function Fibonaccitree. Nodes with the same d-value thus correspond to requests of the recursive function to carry out identical tasks, that is, to create identical trees.

Figure 7.20 Execution Tree for Depth d = 6

Figure 7.20 shows a rather healthy-looking (nonsparse) tree with depth d = 6. This holds true for any value of d and means that while the storage is not excessive, the number of nodes in the tree will be O(f d), where f = (1 + [image])/2.* This is because

The execution tree for Fibonaccitree will always "resemble" the tree it is constructing.

The depth of an execution tree determines the size of the underlying stack needed for the program to execute properly.

The number of nodes in a Fibonacci tree of depth d is Fd+2 - 1.

Hence the execution time will increase exponentially with d. A close inspection reveals that many identical tasks or problems are solved repeatedly during its execution. For example, d = 0, 1, 2, 3, and 4 are solved 5, 8, 5, 3, and 2 times, respectively. This is what causes the excessive execution time and illustrates the situation to be wary of. When the same problems occur repeatedly, the efficiency can be increased if the algorithm can be modified to save the results for these problems so they may be used directly when needed instead of being regenerated by recursive calls. Of course, if this requires excessive storage, then such an approach won't do. Excessive storage is needed when an excessive number of distinct problems must be solved and their results saved. Still, this represents a viable strategy when applicable, as is the case here.

* In fact, the number of nodes will be 2Fd+1 - 1.

Notice that a Fibonacci tree of depth d � 2 can be constructed even if only Fibonacci trees of depth d - 2 and d - 1 are known. All that is necessary is to create a root node and append the tree of depth d - 2 as its left subtree and the tree of depth d - 1 as its right subtree. Consequently, the Fibonacci tree of depth d can be obtained by constructing the trees for depth 0, 1, 2 . . . , d in turn, while retaining only the lastest two. This gives the nonrecursive solution:

An Efficient Nonrecursive Version

If d = 0, then

set t to null;

else if d = 1, then

a. create a root node

b. set t to point to the root node;

else

c. set tl to null,

d. create a root node,

e. set tr to point to the root node, and

f. for k from 2 to d,

i. create a root node with tl its left subtree and tr its right subtree

ii. set t to point to the root node

iii. set tl to tr and tr to t.

This solution is efficient in both execution time and storage (see Exercises 16 and 17 of Chapter 4 for a related example).

Although it was possible to cut down drastically on the number of Fibonacci trees retained in this solution, such a saving not always possible. Sometimes solutions to all the problems generated by recursive calls will need to be retained. Even so, when this number is not excessive and an appropriate data structure can be found in which to store and efficiently access a solution when needed, then execution time can be improved. This is another instance of what we have frequently seen: time may be traded for storage.

It is hoped that you do not feel we have been barking up the wrong trees! Perhaps you even agree with the humorist Ogden Nash:

I think that I shall never see

A billboard lovely as a tree.

Indeed, unless the billboards fall

I'll never see a tree at all.


Data Structures Using C: Exercises

Exercises

1. a. What is the minimum number of nodes that a binary tree of depth d can have?

b. What is the maximum number of nodes that a binary tree of depth d can have?

2. Suppose t is a binary tree with n internal nodes and m terminal nodes. T is a full binary tree if each of its nodes has zero or two successors. What is the relation between m and n when t is a fully binary tree?

3. There are 0, 1, 2, and 5 distinct binary trees with, respectively, 0, 1, 2, and 3 nodes.

a. how many distinct binary trees are there with 4 nodes?

b. Can you find a formula for the number of distinct binary trees with n nodes? This formula bt(n) may be expressed in terms of bt(k) for k < n.

4. Modify the program segments of Section 7.2.1 so that they work correctly in all cases.

5. a. Suppose the binary tree that follows is preorder, inorder, and postorder traversed. List the order in which the nodes will be accessed for each.

b. Do you notice anything striking about the inorder listing?

6. Prove that all terminal nodes will always be accessed in the same relative order for all three travelsals.

7. a. Suppose the stack used in a pre-, in-, or postorder traversal is implemented as an array. What is the requirement for limit in terms of the depth of the binary tree, to guarantee that the stack is large enough for the traversal to work?

b. What is the necessary value for limit in order to guarantee that the traversal will work with the stack, for any binary tree with n nodes?

8. Suppose the following two binary trees are preorder and postorder traversed. The preorder and postorder listings are given for each tree.

Both binary trees have identical listings. This cannot happen with preorder and inorder listings. That is, we cannot find two distinct binary trees that have identical preorder and inorder listings. The same is true for postorder and inorder listings. Find an algorithm that will create the unique binary tree that is thus determined by its preorder and inorder listings.

9. a. Can you find a way to use the left and right pointer fields of terminal and internal nodes that contain null pointers in a binary tree to create "threaded binary trees" so that they can be traversed without using a stack? (Hint: Consider pointers that point to the predecessor and successor of a terminal node in an inorder traversal.)

b. What about for inorder and postorder traversals?

10. Write inorder and postorder traversal routines (nonrecursive and recursive).

11. a. Suppose you wish to use a linked representation for binary trees with an array implementation as in Figure 7.10. Suppose tht input is given as a name value followed by a sequence of records, each record representing a node of a binary tree. For the binary tree fred of Figure 7.10, the input would be

 5

 8  L   0

-1  O  -1

 3  U   4

-1  S  -1

-1  T  -1

Three records appear in the input in the order in which they would be accessed by a preorder traversal. Assume the head value is read into fred, and preorder traverse is called. Write a process routine that will turn preorder traverse into a routine that reads the records of the binary tree into the leftptr, info, and rightptr fields properly.

b. Same as Exercise 11(a), but assume the records are given in the order in which they would be accessed in a postorder traversal.

12. The binary tree sample is stored using a linked representation. List the order in which its nodes will be accessed in a preorder traversal.

          leftptr  info  rightptr

           1    0    R       0

           2    3    W       4

           3    5    R       0

sample 7   4    0    C       0

           5    4    E       6

           6    0    T       0

           7   10    C       3

           8    6    A       3

           9    1    B       4

          10    1    O       0

13. What will be in the stack during a preorder traversal of sample in Exercise 12 when its rightmost node is accessed? What about when it is inorder and postorder traversed?

14. a. Suppose every subtree of sample of Exercise 12 is interchanged with its right subtree. What will leftptr, info, and rightptr look like?

b. What would leftptr, info, and rightptr look like if all the terminal nodes of sample were deleted in Exercise 12?

15. Suppose a binary tree is stored using the records array implementation as in Figure 7.10. If tree points to its root node, write a function to print out the sequence of pointers that specify the path in the tree to the right most node in tree. For example, in Figure 7.10 the rightmost node of fred is 1, and the sequence of pointers to be printed out for fred would be 6, 1, 5.

16. a. If the binary tree of Figure 7.5 were represented sequentially, what numbers would be assigned to its nodes?

b. What binary tree with n nodes will require the largest array for its sequential representation?

17. When dealing with arbitrary binary trees, would you use linked or sequential representation? Why?

18. a. Write a function that will return, given a binary tree t, with a pointer to a terminal node of t containing the largest value in its information field.

b. Write a function that, given a binary tree t and two pointers p and q to nodes of t, returns a value true if node.q is a successor of node.p and a value false otherwise.

c. Write a function that does the same thing as Exercise 18(b), except that the function returns a value true if node.q is in the subtree of node.p and a value false otherwise.

19. Consider the binary tree that was produced by the transformation from a general tree to a binary tree as a general tree, and find the binary tree produced by its transformation.

20. Consider again the solution to the information retrieval problem of Chapter 6. Suppose you create a record, r, in memory. R will have a pointer field pointing to the record for a father. You can then think of r as a record representing the node of a binary tree.

a. Describe this binary tree.

b. What general tree would have r as its binary tree under the transformation of Section 7.5.1?

21. a. Write a function to append a binary tree t1 to a binary tree t2 by replacing some terminal node in t2 at least depth by the whole tree t1. For example, see the figure. Use a linked representation with pointer variables.

b. What does your function do if t1 and t2 are the same tree?

c. How will your solution change if a sequential representation is used?

(a) Input

(b) Output

22. a. Write a routine that deletes all the terminal nodes of a binary tree.

b. Write a routine that deletes node.q if it is a successor of node.p in a binary tree.

23. a. Write a function to interchange all left and right subtrees of a binary tree.

b. Create a function to interchange all left and right subtrees, except that the interchanged version of the original tree will be created as a new tree, leaving the original tree unchanged. Assume pointer variables are used for its implementation.

24. a. Modify the preorder traversal so that it returns, in a variable identical, a value true if t1 and t2 point to identical trees, and a value false otherwise. That is, the value is true if the two trees have the same structure and the same information field values.

b. Do the same task, except that the binary trees need have only the same structure.

25. a. Suppose you are given a maze with the structure of a binary tree. The maze has a starting point and an exit. A node is associated with the starting point, with the exit, and with every point in the maze where one of two paths must be taken. The maze is given as a two-dimensional array a. A [i,j] is 1 if there is a path from node i to node j in the maze, and is 0 otherwise. There are n points in the maze. Adapt the preorder traversal to achieve an algorithm for solving the maze�that is, for printing out the path from the starting node to the exit node. Your solution should be detailed enough to write a function corresponding to it directly.

b. If the maze has loops, what modifications must be made in your solution so that the algorithm works correctly?

26. a. Modify the preorder traversal algorithm for creating a binary tree from a general tree (p. 332) and the process algorithm so the preorder traversal returns, in the variable transform, a value "true" only if a binary tree bt is the transformed version of a general tree t.

b. Modify the preorder traversal algorithm so that it produces the transformed tree by processing the nodes of t in breadth-first order. (Hint: Consider the queue.)

27. Write the function create used in the process algorithm.

28. Modify the backtracking preorder traversal algorithm so that it does a backtracking breadth-first traversal.

29. Suppose you are given a "map" of n cities and the distances between them. Can you find a way to modify the preorder traversal algorithm to determine the shortest distance between two cities? What is a feasible test to use to allow feasible to limit the execution time of the solution?

30. a. Suppose a general tree is given as follows:

(A(B(E, F, G), C(H), D(I(K, L, M), J)))

This corresponds to the general tree:

Suppose that input is given by such a parenthesized expression. Can you find an algorithm that will create the general tree from it? Assume that the general tree will be represented in memory as a collection of lists of successors, with one successor list for each node.

b. What is the connection between parenthesized expressions and a listing of nodes in the order in which they are accessed by a preorder traversal?

31. T is a binary tree. Result(t) is defined by

Left(t) and right(t) are, respectively, the left and right subtrees of t.

a. If t is the binary tree that follows, then what is result(t)?

b. What does result do?

32. T1 and t2 point to binary trees represented using pointer variables.

check (t1,t2)

binarytreepointer t1,t2;

{

binarytreepointer null,setnull();

null = setnull();

if(((t1 == null) && (t2 == null)) || ((t1 != null)

&& (t2 != null)))

return(check(t1->leftptr,t2->leftptr)

&& check(t1->rightptr,t2->rightptr));

else

return (FALSE);

}

a. Find the function value if it is invoked for trees t1 and t2.

b. What does check do?

33. a. Find b(4) where

b(0) = 1

b(n) = b(0) � b(n - 1) + b(1) � b(n - 2) + b(2)b(n - 3) + . . .

+ b(n - 1) x b(0) n > 0

b. Find a connection between the number of distinct binary trees with n nodes and b(n).

34. a. Give a recursive definition of balanced binary trees.

b. Find a recursive definition for F(d), the number of nodes of a Fibonacci tree of depth d.

35. Write a recursive function result corresponding to Exercise 31.

36. a. Write a recursive function count to return the number of terminal nodes of a binary tree.

b. Write a recursive function terminal to delete all terminal nodes of a binary tree using pointer variables.

37. Write a recursive routine partialexchange to interchange all subtrees of a binary tree whose left subtree info field value exceeds the right subtree value. Assume the binary tree is represented using pointer variables.


Data Structures Using C: Suggested Assignments

Suggested Assignments

1. Write a function to print the family tree for an individual whose family history is stored in the data base of the case study of Chapter 6. The individual is at the root of the family tree, and its subtrees are the family trees of all the individual's children. Print the tree using indentation as in Figure 7.1(a). Assume individual records are stored in dynamic memory and that the appropriate functions are available for searching and accessing the nametables. If this function is to also be executed, then this would be a good assignment for a group project.

Also discuss the connection between the lists used in the case study and the binary tree representation of a general tree.

2. Consider the problem of finding a stable pairing, given preferences, as discussed in Chapter 2. Describe a tree that represents all possible pairings, and write a backtracking traversal function to produce all stable pairings. Base your solution on the backtracking preorder traversal algorithm given in the text. Compare this solution with the solution given in Chapter 2.

3. Formulate a solution to the stable pairing problem as a traversal through all permutations of the men. Use the permutation traversal program of Chapter 4 as the basis for your algorithm. Compare this solution to your solution to Assignment 2 and also to the solution given in Chapter 2.

4. Write an efficient recursive function fibonaccicheck(t,d,flag), which is to return in d the depth of the binary tree t, and with flag true if t is a Fibonacci tree and false otherwise.


Data Structures Using C: CHAPTER 8: INTRODUCTION TO SEARCHING AND SORT

CHAPTER 8: INTRODUCTION TO SEARCHING AND SORTING

Considers how collections of information may be stored to allow

efficient searches

efficient traversals

Develops efficient in-memory sorting algorithms

to illustrate the principles of good design in algorithm development

to show how sorting is used to facilitate searching

Worst-case and average times are developed for

linear search

binary search

maximum entry sort

bubble sort

insertion sort

heapsort

quicksort

Illustrates the use of simulation in analyzing the execution time of a program

8.1: Overview

8.2: Elementary Searches

8.3: Elementary Sorts

8.4: Heapsort: A Faster Sort

8.5: Quicksort: Another Fast Sort

8.6: Simulation of an Algorithm

8.7: Simulation Results for the Sorts

8.8: Synopsis of Search and Sort Efficiencies

Exercises

Suggested Assignments


Data Structures Using C: 8.1: Overview

8.1: Overview

So far in this book, arrays, lists, binary trees, and trees�the basic data sructures�have been introduced and applied to a variety of problems. Selecting the appropriate data structure requires knowing what operations are to be performed on it. The frequency with which these operations are to be performed, the characteristics of the data to be stored, and the average and worst-case time and storage requirements all combine to determine this choice. Other considerations, such as clarity and adaptability, may also serve as criteria for deciding which data structures to use. It is rarely possible to pick a data structure that is best for all criteria. Instead, trade-offs must be made.

In the case study of family relationships in Section 6.5, one of the important operations was a search of the nametable data base. At that time no decisions were made as to how the nametable data base would be structured to facilitate the search. This chapter and the next will show that manyimplementations are possible and that the appropriate choice is determined by the kinds of operations to be performed on the data base. Nametable might be organized in many ways. It can be implemented as an array, a list, a binary search tree, a balanced binary search tree, or a hash table. The techniques used to create and search these structures are introduced in this and the next chapter.

Searching and sorting masses of data to retrieve specific information and organize it for manipulation and presentation are nothing new. These basic operations were performed and studied before the advent of computers.Volumes have been written on these topics, yet they are still being researched. Even the general public and the press are evidently interested in searching, as you can see from the following letter and response:

Dear Ann Landers:

Why is it that whenever I lose anything, it is always

in the last place I look?

�Dizzy Lizzy

Dear Liz:

Because when you find it you stop looking�and

that's the last place you looked.

Although searching and sorting may appear mundane, the speed with which they can be carried out largely determines whether a particular application of the computer is practical (and "cost-effective'") or not. Estimates indicate that about one-fourth of the processing time in computer centers is spent just sorting. The relation between sorting and searching will become apparent in this chapter. If one can be done faster, so can the other. The choice of data structures is always important in algorithm design and is the key to the evolution of good (that is, efficient) searching and sorting procedures. The basic data structures studied in the preceding chapters (arrays, lists, and binary trees) are used in the design of efficient procedures.

Normally, the objects stored, searched for, and sorted are records. In sorting, one field of the records, called the sort key, is chosen as the basis for the sort. Thus, in a payroll application, an employee name field might be the key for an alphabetical ordering, while a salary field might serve as the basis for a sort to determine an ordering by amount of earnings. At times the sort key can be a combination of fields. As an example, we may desire to sort with name as the primary key and zip code as the secondary key. For simplicity it will be assumed in this chapter that the objects being sorted are integer numbers. However, any variable type could have been assumed. For sorting purposes, the only requirement is that, given any two objects, it is possible to tell which precedes the other.

Searching normally involves comparing records to a given search key value. The goal is to determine whether there exists among the records of the data base a record with key equal to that of the search key. If the key value can be matched, then a pointer to the record with the matching value must be returned. The basic principle for shortening search times is to organize the data base so that the search can be focused quickly on the part of the data base that will contain the desired record, if it is present.

The chapter first explores simple linear, binary, and interpolation searches and a number of elementary sorting algorithms, such as the bubble sort and the insertion sort. These lead to more advanced methods for sorting based on a special binary tree called a heap. Next, an application of recursion yields another fast sort, quicksort.

Some of the algorithms developed in the text are fairly complex and difficult, if not impossible, to analyze mathematically. For this reason the last topic considered in the chapter is the use of simulation as a tool in discovering the behavior and efficiency of such complex algorithms.

The chapter concludes with a summary of the strengths and weaknesses of each approach introduced. Knuth [1973b] and Wirth [1976] discuss many searching and sorting algorithms that are not dealt with in this text. Knuth also contains mathematical analyses of many of these algorithms.


Data Structures Using C: 8.2: Elementary Searches

8.2: Elementary Searches

Three simple searches are applicable when records are stored in arrays: (1) linear search, (2) binary search, and (3) interpolation search. You probably have all used these methods in everyday activities. Understanding their limitations and how they work will help you see the need for the more complex structures and algorithms of later sections.

8.2.1 Linear Search

8.2.2 Saving Time: A Neat Trick

8.2.3 Binary Search

8.2.4 Timing the Binary Search

8.2.5 Interpolation Search

8.2.6 Efficiency Comparisons


Data Structures Using C: 8.2.1 Linear Search

8.2.1 Linear Search

A linear search is so named because it proceeds in sequence, linearly through an array. Suppose data in Figure 8.1 is an array of integers.* The task is to determine whether the number stored in the variable key is in the array, and if so, what its index is.

* To simplify the explanation in this chapter, the 0th array position will be ignored, and data stored in arrays will start in position 1 unless otherwise noted.

Figure 8.1 An Array of Integers to Be Searched for the Key Value 25

One obvious method would be to traverse the data array, checking each element of data to see if it contains the same value as key. If an element currently being processed did contain the same value as key, its index would simply be returned, and the search would be complete. Arriving at the end of data without having found the key value would mean that no element with that value is stored in the array. This procedure is easily implemented using a loop structure (as in the following function). The worst-case time required for its execution will be of the order n, written O(n), where n is the length of the data array. In particular, if the value being sought is in element i of the array, it will take O(i) time to find it. The linear search is implemented as follows:

#define MAX 100

typedef struct

{

int key;

}record;

typedef record recordarray[MAX];

    >Comment

linearsearch(data,n,key,pfound,ploc)

/* Searches the n records stored in

the array data for the search key.

Found will be true only if a record

is present with key field equal to

the search key value and then loc

will contain its array index.

*/

recordarray data;

int n,key,*pfound,*ploc;

{

    >Comment

*ploc = 1;

    >Comment

while ((*ploc <= n) && (data[*ploc].key != key))

(*ploc)++;

    >Comment

*pfound = (*ploc < n + 1);

}

A special but common case is when the search is always for a key known to be stored in the array, and each key stored is distinct and equally likely to be the search key. This is like standing in front of a locked door with a bunch of keys. You need to find the key that opens the door from among n keys in arbitrary order on your key-ring. The average time for such a search would be O(n/2). That is, on the average, you would look at half the keys, or half the elements of the array, before finding the right one.

Thus the procedure called linear search is so named, not only because it proceeds linearly through an array, but also because its processing time increases linearly with n; when n is doubled, the time required to complete the search is doubled.


Data Structures Using C: 8.2.2 Saving Time: A Neat Trick

8.2.2 Saving Time: A Neat Trick

Occasionally programming tricks can be used to enhance efficiency. In this case paying attention to some details produces a trick that can pay off by saving processing time. The straightforward implementation of the linear search requires a test (loc � n) for the end of the array before checking the next element. This is true whether the implementation is a for loop or a while loop. However, it is possible to avoid performing this test.

Place the search key in element n + 1 of data. Then, in implementing the linear search, there is no need to test for the end of the array before checking the next element. Since the array is now considered to be (n + 1) elements in length, we are certain to find the search key before we "fall off the end." After the loop is exited, note the element in which key was found, and test to see if it was the (n + 1)th element, which means that the search was unsuccessful. Thus only one test (data[loc].key ! = key) must be performed. This saves n tests in unsuccessful searches, and i tests in successful searches, when the search key appears in element i of data. This simple modification of the basic linear search can easily produce significant time savings (20 to 50 percent). In general, whenever loops are modified to reduce the tests or processing they perform, execution time is saved each time through the loop. The more efficient linear search implementation appears as follows:

#define MAX 100

typedef struct

{

int key;

}record;

typedef record recordarray[MAX];

    >Comment

linearsearch(data,n,key,pfound,ploc)

/* Searches the n records stored in

the array data for the search key.

Found will be true only if a record

is present with key field equal to

the search key value and then loc

will contain its array index.

*/

recordarray data;

int n,key,*pfound,*ploc;

{

*ploc = 1;

    >Comment

data[n + 1].key = key;

    >Comment

while (data[*ploc].key != key)

(*ploc)++;

*pfound = (*ploc < n + 1);

}

Another way to save time in a linear search is to order the records in the array. One way to do this is to store them in decreasing order of key value. The linear search can then be modified by terminating the search as soon as a record is reached whose key value is less than the search key. The desired record cannot be in the array past that point. Although this results in extra comparisons for successful searches, time is saved overall when unsuccessful searches occur frequently enough.

It is also possible, if the frequency with which records will be retrieved is known, to store the records in decreasing order of retrieval frequency. The most frequently retrieved record will then be at the top of the array. This approach significantly improves search efficiency when the desired record is likely to be near the top of the array. In fact, if the frequencies decrease rapidly enough, the average search time for the linear search will be a small constant.


Data Structures Using C: 8.2.3 Binary Search

8.2.3 Binary Search

Certainly no one does a linear search to look for a phone number in the telephone directory. Imagine trying to do so in the directory for New York City or any other large city! But there are ways to carry out the search so that the name is found in a few seconds, even from among a few million names. The structure of the telephone directory is the clue to the solution, which takes advantage of the alphabetical ordering of the entries in the phone book.

Assume that the numbers stored in data are arranged in decreasing order, as in Figure 8.2. They are said to be sorted in decreasing order. Since a telephone book is sorted in alphabetical order, one of the things most people do by habit when using the telephone directory is to flip the directory open toward the front, middle, or back depending on the location in the alphabet of the name being sought. If the page opened to has names farther down in the alphabet, the search name cannot be on that page or anywhere in the directory to its right. It can only be to the left of the selected page. This process of elimination is behind the search technique called binary search. It is not difficult to see why this search technique is called binary. Every time a test is made in the search, there are two choices. In the telephone book example, search either half of the remaining pages for the name.

A binary search is also a good strategy for winning the game of twenty questions, which requires guessing the identity of an object after finding out if it is in the animal, the mineral, or the vegetable category. Twenty question with "yes" or "no" answers are allowed. Selecting each question so that about half the remaining possibilities are eliminated, no matter what the answer, is best. (For example, given the animal category, "Is the object male or female"?) In this way 220, or over a million, objects can be distinguished.

In twenty questions it is difficult to select a question that makes close to a 50-50 split of the remaining possibilities. However, if you are asked to determine a number known to lie between 1 and 1024 by asking yes-no questions, it is easy to do so. Ask, "Is it greater than 512?" and continue splitting the remaining possibilities in half. No more than 10 such questions are needed to find the number from among the 210 (1024) initial possibilities. Similarly, when searching for a specific key value among 2n records, at most n key comparisons are needed to determine whether it is present, as long as each comparison splits the remaining collection of records into two parts that differ in size by at most one.

Figure 8.2 Array of Integers Sorted in Decreasing Order

When searching data stored in arbitrary order in an array, there is no reason to expect the search key to be in any particular region of the array. However, when the data have been stored in sorted order, the situation is different. It is possible to check the middle element to see if the search key is there. If it is, the program would note its location and terminate the search. If the key is not there, then if the search key is greater than the key in the middle element, eliminate the middle element and all elements below it from consideration. If the desired record is in data at all, it must be above the middle element. Similarly, if the search key is less than the key in the middle element, eliminate the middle element and all elements above it. In any event, at most half the elements remain to be considered, and the same procedure can then be applied to the appropriate half: either elements 1 to [mid-1] or elements [mid+1] to n. Mid is a variable that points to the middle element.

For example, if mid is 10, then the search key value, 25, is not greater than data[mid], 78. Hence, if it is in data at all, it must be between positions mid+1 and n�11 and 20. Taking 15 as the middle element of these, the new mid value is 15. Key is less than data [mid], 32. Hence we need to search between mid+1 and 20-16 and 20. With a new mid value of 18, key is greater than data [mid], 13. Now search only between 16 and 17. The next mid value is 16, and the key is found.

The basic structure of the binary search algorithm is a loop, as shown in the following function.

#define TRUE 1

#define FALSE 0

#define MAX 100

typedef struct

{

int key;

}record;

typedef record recorddarray[MAX];

binarysearch(data,n,key,pfound,ploc)

/* Searches the n records stored in order by

key field value(largest first) in the array

data for the search key. Found will be true

only if a record is present with key field

equal to the search key value and then loc

will contain its array index.

*/

recordarray data;

int n,key,*pfound,*ploc;

{

int top, mid,bottom;

    >Comment

top = 1;

bottom = n;

*pfound = FALSE;

*ploc = 0;

    >Comment

while ((top <= bottom) && !(*pfound))

{

    >Comment

mid = (top + bottom)/2

    >Comment

if (data[mid].key == key)

{

    >Comment

*pfound = TRUE

    >Comment

*ploc = mid;

}

    >Comment

else if (data[mid].key < key)

bottom = mid - 1;

    >Comment

else

top = mid + 1;

}

}

Mid is initialized to the middle element of data, top to 1, and bottom to n. Top and bottom are variables that point to the top and bottom elements of the consecutive elements left to be searched, all others having been eliminated from consideration. In the loop body, test for key equal to data [mid]. If the equality is true, the program exits the loop and the search is done, since key has been found at element mid. If not, top and bottom must be updated to reflect the new sequence of elements that are to be searched. This updating must set bottom to mid whenever key is greater than data [mid], and top to mid + 1 otherwise. The test for continuing the loop involves checking whether key was found, and whether any elements are left to be searched between top and bottom. The search is terminated as soon as the desired entry is found or as soon as it is concluded that the entry is not present. Loc will be zero if the key value was not found and will index its location in data if found.


Data Structures Using C: 8.2.4 Timing the Binary Search

8.2.4 Timing the Binary Search

How much time is required by the binary search algorithm in the worst case? Each pass through the loop requires at most a constant amount of work. The time to execute the algorithm will then be determined by the number of passes through the loop, plus, of course, a constant time for initialization and finalization. What is the greatest number of loop iterations executed?

In each pass through the loop, the search involves at most half the elements that were under consideration during the preceding pass through the loop. Starting with n elements, after one pass through the loop at most 1/2n elements are left, after two passes through the loop at most 1/2(1/2)n elements, after three passes at most 1/2(1/2)2n elements, and after k passes at most 1/2(1/2)k-1n or 1/2kn elements. This number, 1/2kn, is less than 1 when -k + 1g n < 0 or when k > 1g n. This means that the loop cannot be executed more than �lg n� times. If n is 20, for example, then 1g 20 = 4.32, and �4.32� is 5. Thus in the worst case, the binary search is O(1gn). If each record stored is searched with equal frequency, then the average time to find a stored record is actually about (1g n) - 1.(The proof is not given here.)


Data Structures Using C: 8.2.5 Interpolation Search

8.2.5 Interpolation Search

Sometimes, telephone directories, like dictionaries and encyclopedias, have tabs so that users can flip open the book to the beginning of the section containing entries with a specific first letter and gauge whether to search toward the front, middle, or back pages of the section. The tabs have been inserted or interpolated within the data to guide the search. This represents a generalization of the binary search called the interpolation search. The topic will not be pursued in detail, but note that this technique allows the elimination of greater fractions of the remaining names than does a binary search. Not surprisingly, this tends to reduce the execution time.


Data Structures Using C: 8.2.6 Efficiency Comparisons

8.2.6 Efficiency Comparisons

When n is large, the binary search takes much less time, in the worst case, than the linear search. This is because it makes 1g n rather than n comparisons, going through its loop body 1g n rather than n times. However, the steps performed in the loop body of the binary search are more complex and more time-consuming than the steps performed in the loop body for the linear search. Hence it is possible, for smaller values of n, that the linear search will be faster. Similarly, the interpolation search will need to perform more complex steps in its loop body, even though it may need to loop fewer times than the binary search.

Of course the binary and extrapolation searches require sorted data. When only a few searches are to be made or when n is small, it may not pay to spend time to sort the data first. For repeated searching or large n, the efficiencies of the binary and extrapolation searches easily overcome the cost of sorting.


Data Structures Using C: 8.3: Elementary Sorts

8.3: Elementary Sorts

We now turn our attention to three simple sorting methods applicable to records stored in arrays: (1) maximum entry sort, (2) bubble sort, and (3) insertion sort. They are also applicable to records stored in lists. Understanding their limitations and how they work sets the stage for the more complex structures and algorithms of the next section.

8.3.1 Maximum Entry Sort

8.3.2 Bubble Sort

8.3.3 Timing the Bubble Sort

8.3.4 Insertion Sort

8.3.5 Timing the Insertion Sort

8.3.6 Attempted Improvements


Data Structures Using C: 8.3.1 Maximum Entry Sort

8.3.1 Maximum Entry Sort

Perhaps the most straightforward way to sort numbers in an array into decreasing* order is to begin by placing the largest number in element 1. The next largest number is then placed into element 2, and so on. The maximum entry sort repeatedly finds the next largest array entry and places it in its proper position in the array. Suppose the point has been reached in this procedure where the k largest numbers have been properly placed into the first k elements of the array data. Assume a variable next points to the element of data into which the next largest element is to be placed. Next must now be updated to next + 1 so that it contains the value k + 1, as indicated in Figure 8.3.

* The sorting programs assume that decreasing order is desired. They can easily be modified to sort in increasing order.

Assume that a function maxloc exists, with parameters data, next, and n, the array length. Maxloc returns with its value indexing the largest element between the nextth and the nth elements of data. The sort may then be obtained by a loop in which maxloc is invoked (n - 1 times, with next taking on values 1, 2, . . . , n - 1, respectively, on each call. Within the loop, when each call to maxloc is made, an exchange between data [next] and data[maxloc] occurs.

Maxloc might be implemented by traversing the elements next to n of data. Each time a new element is accessed, it is compared to a variable, largest, containing the largest entry seen so far in this traversal. If the new element is greater than largest, largest is set to the new element value, and maxloc is set to point to the location of the new element in data.

This algorithm is probably similar to what most peole do instinctively when confronted with a small sorting problem. It may be implemented as in the procedure maxentrysort.

Figure 8.3 Ready to Determine the Next Largest

#define MAX 100

typedef struct

{

int key;

}record;

typedef record recordarray[MAX];

maxloc (data,next,n)

/* Returns the array index of the

largest entry in data between the

entries indexed by next and n.

*/

recordarray data;

int next,n;

{

int largest,i,tmaxloc;

tmaxloc = next;

largest = data[next].key;

for (i = next + 1; i <= n; i++)

    >Comment

if (data[i].key > largest)

{

tmaxloc = i;

largest = data[i].key;

return (tmaxloc);

}

}

maxentrysort (data, n)

/* Sorts the n records stored in array data

in descending order.

*/

recordarray data;

int n;

{

int next, loc;

for (next=1; next<=n - 1; next++)

{

loc = maxloc (data,next,n);

    >Comment

interchange (data, next, loc);

}

}

For the simple record used here for illustration, the interchange function is as follows.

interchange (data, i, j)

/* Interchanges the entries of

array data indexed by i and j.

*/

recordarray data;

int i,j;

{

record tempdata;

tempdata.key = data[i].key;

data[i].key = data[j].key;

data[j].key = tempdata.key;

}

How much time does this implementation of the maximum entry sort take? One interchange is made on each call to maxloc for a total of n - 1 interchanges. The number of comparisons is (n - k) on the call to maxloc, when next has the value k. The total number of comparisons is (n - 1) + (n - 2) + . . . + 2 + 1, which sums to [n(n - 1)]/2. Thus the time is determined by the time of the (n - 1) interchanges plus the time of the [n(n - 1)]/2 comparisons. Each interchange takes a constant amount of time, and each comparison takes, at most, a constant amount of time. The result is O(n2) time. This means that if n is doubled or tripled, the time will increase by a factor of about 22, or 4, and 32, or 9, respectively, when n is large. For n = 105, the time is roughly proportional to 1/2 1010, which is within a factor of 100 of 1013, the guide for feasibility presented in Chapter 1. Quicker algorithms are obviously needed for large n, and even perhaps for small n. The next elementary sort technique offers some improvement.


Data Structures Using C: 8.3.2 Bubble Sort

8.3.2 Bubble Sort

The bubble sort gets its name from the way smaller entries "sink" to the bottom of the array during the sort while larger entries "bubble up" to the top. To carry out the bubble sort on the array shown in Figure 8.4(a), traverse the array, comparing two adjacent elements, starting with the first two, and ending with the last two. If the [i + 1]th element value exceeds the ith, interchange the two element values.When a traversal is completed, it is called a pass through the array. Continue making passes until a pass occurs in which no interchanges are required.

In the example, after pass 1, the array looks like Figure 8.4(b). Since at least one interchange was made, another pass is made, yielding Figure 8.4(c). Again, at least one interchange was made, so the program must make another pass.

Will this process continue forever making passes, or eventually stop? Clearly, once it stops since no interchanges were made on the last pass, the array is sorted. If any number were out of order, at least one interchange would have been made during that pass.

Notice that after the first pass the smallest entry, 3, was at the bottom of the array. After the second pass the two smallest entries, 10 and 3, were in order at the bottom of the array. This was not accidental. Notice that the two smallest entries will never be moved, and the third smallest element (13 in this case) will be encountered on the next pass. Once it is encountered, it will always be interchanged with the adjacent lower record to which it is compared. It will eventually come to rest just above the second smallest entry.

(a) Initial Array

(b) After Pass 1

(c) After Pass 2

Figure 8.4 Bubble Sort

In general, after the kth pass, the k smallest entries will be in order in the bottom k elements of the array. Of course, more than the k smallest may be in order at this point, but only k can be guaranteed. Hence the array will be in sorted order after at most n - 1 passes. The procedure can therefore be modified to compare down to only the [n - k]th element on the kth pass. The bubble sort is implemented with the following function.

#define TRUE 1

#define FALSE 0

#define MAX 100

typedef struct

{

int key;

}record;

typedef record recordarray[MAX];

bubblesort (data, n)

/* Sorts the n records stored in array data

in descending order.

*/

recordarray data;

int n;

{

int i,done;

done = FALSE;

while (!done)

{

done = TRUE;

for (i=1;i<=n-1;i++)

if (data [i +1].key > data [i].key)

{

interchange (data, i, i + 1);

done = FALSE;

}

}

}


Data Structures Using C: 8.3.3 Timing the Bubble Sort

8.3.3 Timing the Bubble Sort

If the array were initially given in sorted order, only one pass would be required, and the time would be proportional to n. If the array were initially given in reverse sorted order, then n - 1 passes would be required, with (n - 1 - k) interchanges and comparisons. This is the worst case, and takes time determined by the [image] (n - k) interchanges and comparisons. This sum is (n - 1) + (n - 2) + � � � + 1 or [n(n - 1)]/2 interchanges and comparisons. The worst-case time for the bubble sort is O(n2), just as for the maximum entry sort. However, the maximum entry sort will always take time O(n2), since the number of comparisons it makes is not dependent on the data. The time taken by the bubble sort thus varies from O(n) for sorted data to O(n2) for reverse sorted data. The more ordered the initial data, the shorter the time.


Data Structures Using C: 8.3.4 Insertion Sort

8.3.4 Insertion Sort

The next sort to be considered is insertion sort. An insertion sort works by assuming that all entries in the array are already in sorted order and inserting the next entry in its proper place in the order. Figure 8.5 presents an example of an insertion sort.

In this example the same array, data, is used as for the bubble sort. Each record is processed in turn, starting from the second. Figure 8.5(a) represents the situation after processing of the first six records. The seventh is yet to be processed. It is processed, as are all the other records, by assuming the preceding records are already in sorted order. Notice that records 1�6 are in order. To insert the seventh record, 75, into its proper place among the first seven records, compare record 7 with its predecessor, record 6. Since 75 exceeds 10, it should go somewhere higher up in the array, so compare it to the predecessor of record 6, record 5. Since 75 exceeds 32, compare it to the predecessor of record 5, record 4.

(a) Before Processing Record 7

(b) After Processing Record 7

Figure 8.5 Insertion Sort

Again, 75 exceeds that element's value of 32, so compare it to its predecessor's value, 65. The last comparison is with 78. Since 75 does not exceed 78, record 7 must be inserted between record 2, where 78 is now, and record 3. To do this, move records 3 to 6 down one element, respectively, and place 75 in record 3. The result is Figure 8.5(b).

K has been updated by 1, so it points to the next record to be inserted. This algorithm for an insertion sort can be described by a loop structure in which k varies from 2 to n, as in the following function.

#define TRUE 1

#define FALSE 0

#define MAX 100

typedef struct

{

int key;

}record;

typedef record recordarray [MAX];

insertionsort (data, n)

/* Sorts the n records stored in array data

in descending order.

*/

recordarray data;

int n;

{

int k, i, inserted;

for (k=2;k<=n; k++)

{

i = k;

    >Comment

inserted = FALSE;

while ((i > 1) && !inserted)

if (data[i - l].key < data[i].key)

{

interchange(data,i,i - 1);

i--;

}

else

inserted = TRUE;

}

}

Within the loop, process the kth record by inserting it properly among the top k elements. This may be accomplished by another loop, in which record k is compared to each consecutive predecessor until it does not exceed the current predecessor record. It is inserted at that point in the array. As an exercise, you should modify the insertionsort program to incorporate the programming trick of Section 8.2.2. In Table 8.1 (see Section 8.7), this is referred to as the modified insertion sort.


Data Structures Using C: 8.3.5 Timing the Insertion Sort

8.3.5 Timing the Insertion Sort

If the initial array were given in sorted order, the insertion of the kth record would require one comparison and no insertions. The total time would be O(n - 1). If the initial array were given in reverse sorted order, the insertion of the kth record would require (k - 1) comparisons, and the kth record would have to be inserted at the top of the array. To accomplish this, each of the (k - 1) records must be moved down one element. The total time will be the time for the 1 + 2 + � � � + (n � 1) comparisons, plus the time for the 1 + 2 + � � � + (n � 1) shifts down. Consequently, the total time is O(n2). This represents the worst case. Again we see that the time can vary with n2 for large n. The insertion sort, like the bubble sort, performs better the closer the initial data is to being sorted. If each of the n! possible initial orderings of the array with n distinct entries is equally likely to occur, then the average time for all three of the sorts considered so far is also O(n2).

In the three implementations for the sorting algorithms, it was assumed that the interchanges taking place were actually interchanges of records. When records are large, it takes appreciably more time to move a record than it does to move a single pointer to the record. Consequently significant amounts of time can be saved by keeping an array, p, of indexes or pointers to the records. The entries of this array can then be kept in order, so that p[i] indexes or points to the ith record in sorted order. Interchanging indexes or pointers takes less time than for records but uses more memory (which is an added expense). This is another example of trading time for storage. In the procedures shown for maximum entry sort, bubble sort, and insertion sort, the interchange operations would then apply to the array p, and the comparison operations would apply to the actual records indexed or pointed to by i.


Data Structures Using C: 8.3.6 Attempted Improvements

8.3.6 Attempted Improvements

The next question is, can we improve the insertion sort? We could try to improve it by changing how the search is done for the correct location of the kth entry. In effect, the insertion sort traversed linearly through records [k - 1] to 1 to insert the kth record. Instead, we could do a binary search of the consecutive elements 1 to [k - 1]. We will then know where to insert the kth record. This will reduce the 1 + 2 + � � � + (n - 1) comparisons to 1g 2 + 1g 3 + � � � + 1g (n - 1), which is about n 1g n. It does not, however, alleviate the need for the 1 + 2 + � � � + (n - 1) shifts due to the insertions, so the time is still O(n2).

What else can be tried to improve the insertion sort? Aha, lists! Recall that lists are convenient for insertions, so simply represent the records in a list rather than an array. Then, once you know where to insert a record, merely change two pointers. Thus the 1 + 2 + � � � + (n - 1) or [n(n - 1)]1/2 insertion time is reduced to time O(n). The list, coupled with the binary search reduction in comparison time to n 1g n, seems to give a new implementation of the insertion sort with worst-case time n 1g n. The fly in this ointment is that the ability to do a binary search has disappeared. It is no longer possible to locate the middle record by selection, which is what makes the binary search with sorted arrays so fast. Just the first record's location is known; the middle record can be found only by traversing the list and counting records until the middle one is reached.

Figure 8.6 Selection of Records by Means of a Pointer Array

Wait! We are not yet defeated in our quest for a faster insertion sort. Use a pointer array, assuming enough storage is available, to tell where all the records are on the list. This allows selection of the list records as in Chapter 2. The situation is then as shown in Figure 8.6.

Numberlist is the head of the list of records in the illustration. Suppose the first six records have been properly processed as in the insertion sort, and the seventh is to be processed next. A binary search can now be done, since the middle record, among 1 to 6, will be indexed by the middle entry of elements 1 to 6 of the pointer array. Thus it is possible, in 1g 6 time, to determine where the seventh record is to be inserted. The insertion will require the changing of two pointers. But here's the rub! To maintain the pointer array, it must have its seventh element inserted in the corresponding position of the pointer array. We have merely succeeded in transferring the insertion problem to the pointer array; we have not eliminated it!

Although these attempts to find a faster sorting algorithm have so far not been successful, they provide insight into the binary search and sorting problems. First, notice that except for the binary search, all the searching and sorting algorithms so far considered may be applied to records stored in lists as well as in arrays. Observe that to achieve both a search in binary search time (lg n) and fast insertions and deletions, a linked type structure is needed, as well as some way to eliminate from consideration a significant fraction of the records. A second observation is that the original insertion sort would be faster if the length of the array in which each insertion is made were shorter. We must somehow break up the array of records to be sorted, so that the individual insertions do not involve all of the previous records. These observations lead to the ideas behind heapsort and quicksort, two faster sorting algorithms.


Data Structures Using C: 8.4: Heapsort: A Faster Sort

8.4: Heapsort: A Faster Sort

In this section and the next, two less obvious but faster sorts are considered. These two sorts, heapsort and quicksort, were developed from different points of view and as a result have different advantages.

8.4.1 Heapsorts

8.4.2 Creating a Heap

8.4.3 Timing the Heap Creation

8.4.4 Better Heap Creation

8.4.5 Some Details of Heap Implementation

8.4.6 Reheaping

8.4.7 An Implementation for Heapsort

8.4.8 Heapsort is Uniformly Fast


Data Structures Using C: 8.4.1 Heapsorts

8.4.1 Heapsorts

Heapsorts use the concept of a heap. A heap is a binary tree with the property that all records stored in nodes are in sorted order along the path from each node back to the root. The tree of Figure 8.7 is not a heap. Its records are not in sorted order along the path from each node to the root. For example, the path from node 10 (where 123 is stored) to the root is not in sorted order. That path has records 123, 10, 32 and 100. (The nodes in this chapter are numbered like the nodes on trees in Chapter 7 and are referred to by their number.) Notice that the tree has the same information as the data array, but the information has been spread out along different paths.

Figure 8.8(a) is a heap storing the same information. It is one of many heaps that can be created to store the information. Since the path from every node to the root is in sorted order, the record at the root of a heap must be as great as any other in the tree. The same holds for all subtrees. The record at the root of any subtree must be at least as great as any other record in the subtree. The root record may then be taken as the first record in a sorted output of the information stored in the tree.

Figure 8.7 Not a Heap

(a) A Heap

(b) After Removing 536 and Reheaping

(c) After Adding a New Record to Heap (a)

Figure 8.8 Heaps

To sort, output 536 first, and then remove it from the tree. Now suppose that somehow the records can be rearranged in the tree to create a new heap�a process called reheaping. The result might look like Figure 8.8(b). This tree has one less node than the original. Clearly, this process of removing the root can be repeated, outputting it next in the sorted output, and reheaping. When all the records have been output in sorted order, the remaining tree will be the null binary tree (the heap will be empty), and a sort will have been obtained. The algorithm is stated simply:

To perform a heapsort:

1. Create a heap.

2. While the heap is not empty

a. output the record at the root of the heap,

b. remove it from the heap, and

c. reheap.

As stated, the algorithm is not detailed enough. It does not specify how to create a heap (step 1) or how to reheap (step 2).


Data Structures Using C: 8.4.2 Creating a Heap

8.4.2 Creating a Heap

Consider the heap of Figure 8.8(a) with one new record as in Figure 8.8(c). Notice that all paths are in order, except for the path from the new record at node 21 to the root. This prevents this tree from being a heap. To make it a heap, the insertion sort idea may be useful. Insert the new record at its proper place along the already sorted path from its predecessor node to the root, and allow 425 to work its way up along the path as far as it can go the reach its proper place. This can be done by comparing 425 to its predecessor and interchanging if it exceeds its predecessor's value. In this case, it does exceed that value, so interchange to obtain the tree shown in Figure 8.9(a).

All other paths remain sorted, as before. Now compare 425 at node 10 with its predecessor. Since it exceeds 123, interchange again to obtain Figure 8.9(b). Finally, compare 425 with its predecessor's value, and interchange to obtain Figure 8.9(c).

At this point, when 425 is compared to its predecessor's value, the 425 has reached its proper place. All paths must now be in order, since the path from node10 to the root is in order, and the interchanges have not disturbed the order along any other path.

This gives us the means to create a heap:

Start with records stored in the complete binary tree and process each node in turn, starting with node 2 and ending with node n.

The processing of the ith node is exactly the same processing as was done on the new record 425. Since nodes 1 to i - 1 will already have been processed prior to processing node i, the binary tree consisting of nodes 1 to i - 1 will already be a heap. Processing node i then, in effect, treats the record at node i as a new record and creates a new heap. For the example of twenty records, this procedure leads to the heap of Figure 8.8(a).

Figure 8.9 Heap Creation


Data Structures Using C: 8.4.3 Timing the Heap Creation

8.4.3 Timing the Heap Creation

In order to analyze the time required, recall that the number of nodes n and the depth of a complete binary tree are related by

d = �lg2(n + 1)�

This is important. Since the heaps dealt with initially in our heapsort algorithm are complete, their depth is O(lg (n)).

We now analyze the time required to create a heap by this algorithm. Starting from a complete binary tree is important. For instance, if we started with the binary tree shown in Figure 8.10, this algorithm would be equivalent to the original insertion sort of Section 8.3.4.

The heap creation algorithm does a total amount of work determined by the sum of the number of comparisons and interchanges needed in processing each node. A node at depth k cannot require more than k comparisons and k interchanges in its processing. The worst-case time will be achieved for any initial tree in which the numbers stored at nodes 1, 2, 3, . . . , n form an increasing sequence. For instance, if the integer i is stored at node i, for i = 1, 2, . . . , n, we obtain such an increasing sequence. If n is 15, the tree will be as shown in Figure 8.11.

Figure 8.10 An Incomplete Binary Tree

It can be shown that the total time for such a tree to be turned into a heap by the algorithm is proportional to [image], where n is the number of nodes at depth k, d is the depth, and k is the number of comparisons and interchanges required for each node at depth k. This is because every record will work its way to the root of the tree when it is processed. Since nk = 2k-1, this sum will be proportional to n lg n. The efficiency of this method can be increased somewhat by first determining where a record to be processed will end up along its path to the root, and then shifting everything down before inserting it. This efficiency could also have been achieved with the insertion sort.

Figure 8.11 Binary Tree with Numbers Forming an Increasing Sequence


Data Structures Using C: 8.4.4 Better Heap Creation

8.4.4 Better Heap Creation

For a tree of depth d, a better procedure for creating a heap is to start at node number 2d-1 - 1, the rightmost node at depth d - 1. Even better, start at �n/2�, the integer part of n/2, the first nonterminal node at depth d - 1. Process each node in turn, from node �n/2� to node 1. The processing at each node i is to turn the tree with node i as its root into a heap. This can be done, assuming that its left and right subtrees are already heaps, as in the following example.

Consider the initial complete binary tree in Figure 8.12(a). Start at node 5 ( = �11/2�) (83 is stored there) and compare its successors 94 and 7, determining that 94 is the larger. Compare 94 with 83; since 94 exceeds 83, interchange 94 and 83 to obtain Figure 8.12(b).

The subtree with node 5 (94) as root is now a heap. Move to node 4, and the similar comparisons reveal that it is already a heap. Move to node 3, compare its successors 29 and 30, and then compare the larger to 16. Since 30 exceeds 16, interchange 30 and 16 to obtain Figure 8.12(c).

The subtree of node 3 is a heap, so move to node 2. Compare its successors 57 and 94, and then compare the larger to 14. Interchange 94 and 14 to obtain Figure 8.12(d).

At this point, notice that all nodes processed so far must be the roots of subtrees that remain heaps, except perhaps for the root of the subtree, where 14 now appears after the interchange. This subtree need not be a heap and, in fact, is not in this case. You can make it a heap by applying the same process to the node where 14 is currently stored, node 5. Compare its two successors and then compare the larger of those with 14. Then interchange 14 and 83 to obtain Figure 8.12(e).

Figure 8.12 A Fast Heap Creation

Figure 8.12

The subtree with root node 2 is now a heap, and the processing of node 2 is complete. Notice that all nodes processed so far are roots of subtrees that are heaps. The last node to be processed is now node 1. Compare its successors, 94 and 30. Since the larger, 94, exceeds 18, interchange to obtain Figure 8.12(f).

The 18 from the node being processed, node 1, has now moved down to node 2. Hence its subtree may no longer be a heap. Apply the process to node 2. The 18 can move down again and will continue to move down until it either becomes situated at a terminal node or becomes the root of a subtree that remains a heap. In this case, it should be compared with 83, the larger of its two successors, and interchanged to get Figure 8.12(g).

At this point, you can see that 18 is the root of a heap by comparing it to the larger of its two successors. Processing of node 1 has been completed, and the heap has been created. The heap obtained from the original example with this algorithm is Figure 8.12(h). Notice that this heap differs from that in Figure 8.8(a). As mentioned earlier, more than one heap can be constructed to store the same data.

Without analyzing this heap creation algorithm in detail, the result can be stated. This procedure is actually only O(n). This heap creation algorithm is faster than that of the last section because few nodes must move down long paths. In the other algorithm, many nodes may move up long paths. A use for this linear time heap creation algorithm will be demonstrated in another context in Section 9.2, Priority Queues.


Data Structures Using C: 8.4.5 Some Details of Heap Implementation

8.4.5 Some Details of Heap Implementation

One way to implement the heap creation algorithm is to make use of a function, or routine, that works on a binary tree whose left and right subtrees are already heaps. The function does the "shifting" down of the root record until it becomes the root of a subtree that remains a heap, or until it reaches a terminal node. This "shift" routine need not do the interchanging along the way. Instead it can determine the final position of the root being shifted down, moving each record along that path to its predecessor, finally placing the root where it belongs.

Notice that the first heap creation algorithm requires testing the record being inserted along a path to the root against its current predecessor. Before this can be done, the program must test for whether or not there is a predecessor�that is, whether the record has already reached the root. This test must precede every comparison. Similarly, in the second heap creation algorithm, the record being shifted down must be tested against its successors. Before doing this, it is necessary to test whether there are successors.

These tests can be avoided by always putting "plus infinity" at the root of the initial complete binary tree for the first algorithm, and "negative infinity" at the subtrees of all terminal nodes for the second algorithm. This is similar in effect to adding the search key to the end of the array in the linear search algorithm. Plus infinity and negative infinity stand for, respectively, any number larger or smaller than the numbers appearing in the binary tree. In this way, a record being inserted along a path in the first algorithm will never reach the root (so it will always have a predecessor with which it may be compared), and a record being shifted down in the second algorithm will always have successor nodes (since it will never become a terminal node).


Data Structures Using C: 8.4.6 Reheaping

8.4.6 Reheaping

Think of step 1 of the heap sorting algorithm as phase I, in which a heap is created. The execution of step 2 until the records are sorted is phase II. In phase II, the root element of the heap is removed and output. The program must then reheap. This may be done in two ways.

One is to allow the "gap" created at the root by removal of its record to shift down by comparing its two successors and interchanging the larger of the two successors with the gap, then repeating this process until the gap is at a terminal node. A convenient way to do this is to replace the record removed from the root by a "negative infinity" and invoke the shift down function referred to earlier for the second algorithm.

A second way to reheap is to replace the gap with the rightmost record at greatest depth. (Actually, any record with no successors will do.) The same shift function may be invoked to reheap.

Since the length of the longest path in a complete binary tree is O(lg n), the reheaping after each record is removed can take time at most O(lg n). Because n records will ultimately be removed, the total time required for phase II will be at most O(n lg n). For either implementation of phase I, the total heapsort time is O(n lg n). This was achieved by "spreading out" the records along paths of a complete binary tree, and, in effect, doing insertions only along the path from a record to the root.

We have been dealing with a max heap, in which the largest record is at the top. A min heap, with the smallest record at the top, is dealt with similarly. The only difference is that interchanging occurs when a successor's key is smaller, rather than larger.


Data Structures Using C: 8.4.7 An Implementation for Heapsort

8.4.7 An Implementation for Heapsort

A detailed version of the heapsort algorithm can now be developed. You have now seen a number of ways to create a heap and to reheap. We will use the second heap creation algorithm for phase I and the second reheaping algorithm for phase II. Both algorithms use the shift function.

To proceed we must decide on an implementation of the binary tree data structure. The binary tree used in heapsort is initially complete. The required processing involves predecessors and successors of nodes. Sequential representation for the binary tree is ideal, so we use it. The binary tree is stored in the data array and consists of n records.

Storage is needed for the sorted version of data. Although another array may seem necessary for this purpose, we will use only the data array. The algorithm, after creating a heap (in data), continually takes the root record as the next record in the sorted output and then reheaps. Before reheaping, the rightmost record at the lowest level of the heap is placed at the root. The storage vacated by that rightmost record will no longer be needed. We choose to store the output record there. You should convince yourself that this will result in the records appearing in data in reverse sorted order upon termination. To remedy this, we write shift so that it produces min heaps. The records will then appear in correct sorted order upon termination.

Shift has parameters data, root, and last. Root points to the root of a subtree, and last points to the node of the original heap that is currently the rightmost node at greatest depth. The nodes last+1 to n are currently storing the already sorted n-last output records. Whenever shift is invoked, the left and right subtrees of root (excluding nodes last+1 to n) are assumed to be heaps. Shift returns after reheaping, by allowing the root record to move down to its final position. This is done by first storing the key value and record of the initial root record in keyvalue and recordvalue, respectively. Ptr points to the node currently being considered as the final position for the initial root record. This will be the final position if either of the following two conditions is met.

Figure 8.13 (a) Initial Heap and (b) Heap after First Three Entries Have Been Output

1. The successor record of ptr with minimum key value has a key value greater than or equal to keyvalue, or

2. Ptr points to a terminal node or to a node whose successors are nodes among last+1 to n.

Condition 2 can be recognized by the test (succ<=last), where succ points to the left successor node of ptr. Condition 1 requires finding the minimum key value for the successors of ptr. Ptr may not have a right successor, or that successor may represent an output record. This can be tested for by the test (succ<last).

Copy copies the contents of the record pointed to by its second parameter into the record pointed to by its first parameter. When shift determines that ptr is not the final position, it moves the record at ptr to its predecessor node. This is why the original record at root is saved. When the final position is found, the saved record is placed there. The initial heap and the situation after the first three entries have been output are shown in Figure 8.13 for the tree of Figure 8.7.

The implementation for heapsort follows.

#define MAX 100

typedef struct

{

int key;

}record;

typedef record recordarray[MAX];

heapsort (data,n)

/* Sorts the n records stored in array data

in descending order.

*/

recordarray data;

int n;

{

int root,last;

    >Comment

root = n/2;

last = n;

while (root > 0)

{

    >Comment

shift(data,root,last);

root--;

}

root = 1;

while (last > 1)

{

    >Comment

interchange(data,1,last);

last--;

shift(data,root,last);

}

}

shift (data,root,last)

/* Makes the subtree of root into

a heap (min heap). The left and

right subtrees of root must be

heaps. All subtrees are stored

in array data in sequential

representation. Only the nodes

between root and last are

considered to be in the subtrees.

*/

recordarray data;

int root,last;

{

    >Comment

int ptr,succ;

int keyvalue;

record recordvalue;

    >Comment

ptr = root;

    >Comment

succ = 2 * ptr;

    >Comment

if (succ < last)

if (data[succ + 1].key < data[succ].key)

    >Comment

succ++;

at this point, succ points to the smallest of ptr's successors

    >Comment

keyvalue = data[root].key;

    >Comment

copy(&recordvalue,&data[root]);

the purpose of the while loop is to determine the proper place for the original root record; ptr will

point to that location when the loop is exited

    >Comment

while((succ <= last) && (data[succ].key < keyvalue))

{

    >Comment

copy(&data[ptr],&data[succ]);

    >Comment

ptr = succ;

    >Comment

succ = 2 * ptr

if (succ < last)

if (data[succ + 1].key < data[succ].key)

succ++;

}

    >Comment

copy(&data[ptr],&recordvalue);

}


Data Structures Using C: 8.4.8 Heapsort is Uniformly Fast

8.4.8 Heapsort is Uniformly Fast

As presented, heapsort is efficient. Unlike the other sorts considered, which are array- or list-oriented, it is tree-oriented. They are O(n2) in the worst case, whereas heapsort is O(n lg n). Moreover, we shall see in Section 8.7 that it gives uniformly good sort times, no matter how ordered or unordered the original data. Although times are somewhat faster when the data is initially either fairly well ordered or reverse ordered, the range of these times is small compared to the other sorts. This is to be expected, since the construction of the heap takes little time and the reheaping tends to destroy any order present in the data. In effect it appears to be in random order. Actually, during reheaping, when shift works on larger entries at the root, it takes O(lg n) time, since they tend toward the bottom of the tree, while the opposite is true for smaller entries. Because most entries are near the bottom of the tree and are larger, their shifting makes up the bulk of the sorting time. Thus this time should be relatively insensitive to initial order in the data. This uniformity of sort times is one of its main virtues, since it eliminates the great variation in time from O(n) to O(n2) of the other sorts. In addition, it is fast.


Data Structures Using C: 8.5: Quicksort: Another Fast Sort

8.5: Quicksort: Another Fast Sort

As a final example of a sorting method, we consider an algorithm called quicksort. This algorithm is appropriately named, since its average execution is very quick. Its worst-case execution time is slow, but the worst case should occur rarely.

One way to develop quicksort is to attempt a recursive solution to the problem of sorting an array data of n records. The general recursive method immediately prescribes guidelines�break up the original problem into smaller versions of the original problem and then achieve the solution from their solutions. Here, this dictates that the original array be broken down into component arrays, which can be sorted independently and from which the final sorted array can be derived.

Thus data is to be separated into two components to be sorted, an upper array and a lower array. We then have two options for proceeding. One is to combine or merge the resultant two sorted component arrays to achieve a final sorted solution. This method is the merge sort, which is not pursued here but is applied to the sorting of files in Chapter 11. The second approach, which is taken here, is to ensure that the resultant two sorted component arrays require no further processing, but yield the final sorted solution directly.

Merely splitting data into two components and then sorting each component independently is not sufficient to guarantee that the resultant array is sorted, with no further processing required. This is because some entries in the upper component may be smaller than some entries in the lower component. For example, suppose the initial array is data, shown in Figure 8.14(a), and the upper and lower components consist, respectively, of entries 1 to 4 and 5 to 8. Sorting these components independently yields Figure 8.14(b), which is clearly not sorted.

Figure 8.14 Sorting an Array by Component

Notice, however, that if all entries in the upper component were no less than all entries in the lower component, then sorting each component independently would always lead to a result that is sorted. For example, suppose the initial array is Figure 8.14(c). Taking the upper and lower components as entries 1 to 4, and 5 to 8, respectively, and independently sorting them produces Figure 8.14(d). This array is now sorted.

Note that the fifth entry, 13, occupies the same position in the sorted array as in the initial array. This is because all entries in the upper component are no less, and all entries in the lower component are no greater, than 13. The key value 13 actually occupied its proper position in the sorted version of data before the two components were sorted. Hence it need not be included in either component and may serve as a separator or partitioning entry between the two components to be sorted.

To carry out this procedure for an arbitrary array requires finding a partitioning entry. Normally a partitioning entry, such as 13 in position 5, will not occur in the initial array to be sorted. For example, no such partitioning entry (separating larger entries above from smaller entries below) occurs in Figure 8.14(a). This means that the initial array must be rearranged so that a partitioning entry may be found and subsequently used to separate the components.

In keeping with the top-down approach, assume that this task (rearranging the initial array and determining a partitioning entry) is carried out by the function partition(i,j). The arguments i and j contain pointers to entries of the array, as shown in Figure 8.15. Partition (i,j) returns an index pointer to a partitioning entry of the array consisting of all entries between i and j. Thus, if partition (i,j) returns a value p, then all entries between i and p�1 are at least as great as data [p].key, and all entries between p+1 and j are no greater than data [p].key. The pointer p separates the upper and lower components of i,j.

Figure 8.15 Array with a Partitioning Entry P Separating Upper and Lower Components

8.5.1 Two Quicksort Procedures

8.5.2 The Partition Function

8.5.3 Analyzing Iterative Quicksort

8.5.4 The Worst and Best Cases

8.5.5 Distributive Partitioning


Data Structures Using C: 8.5.1 Two Quicksort Procedures

8.5.1 Two Quicksort Procedures

The procedure quicksort may now be written, recursively, as follows.

Recursive Version

quicksort(i,j)

/* Sorts the records stored in array data

between positions indexed by i and j in

descending order.

*/

int i,j;

{

int p;

if(i < j)

{

>Comment

p = partition(i,j);

    >Comment

quicksort(i,p-1);

    >Comment

quicksort(p+1,j);

}

}

To sort a global array of n records, quicksort(1,n) is invoked. The crux of the algorithm is the partition function, which will be discussed shortly.

Every time a call is made to quicksort, two new obligations are incurred, and a current obligation is postponed. For instance, the initial call to quicksort generates the initial obligation: sort the n entries of data. During the execution of this call, partition determines a partitioning entry, and two new obligations are incurred. These are represented by the two recursive calls to quicksort, one for each component. The current obligation must then be postponed to carry out these new obligations. Each recursive call also generates two new obligations, requiring the execution of the current recursive call to be postponed.

As we have seen previously, what can be done recursively can also be done iteratively. Quicksort can be written nonrecursively by using a stack to store one of the new obligations, while the other can be processed immediately. It turns out that the choice of which of the two obligations to stack, and which to deal with immediately, is critical to the algorithm's storage requirements. The partition function does not necessarily split an array into two equal parts, as evidenced by our earlier example. Selecting the larger of the two components to stack results in a worst-case stack depth of O (lg n). An arbitrary choice instead leads to a worst-case stack depth of O (n). The difference in storage needed for the stack can be significant.

The structure of the nonrecursive algorithm amounts to a loop in which the smaller of the two new component arrays is processed and the larger is stacked. The procedure may be written as follows:

Nonrecursive Version

quicksort(i,j)

/* Sorts the records stored in array data

between positions indexed by i and j in

descending order.

*/

int i,j;

{

int p,top,bottom;

stack s;

    >Comment

setstack(&s);

    >Comment

push(j,&s);

push(i&s);

    >Comment

while(!empty(&s))

{

    >Comment

pop(&s,&top);

pop(&s,&bottom);

while(top < bottom)

{

    >Comment

p = partition(top,bottom);

    >Comment

if((p - top) > (bottom - p))

{

push(p-l,&s);

push(top,&s);

top = p + l

}

else

{

push(bottom,&s);

push(p+1,&s);

bottom = p - 1;

}

}

}

}


Data Structures Using C: 8.5.2 The Partition Function

8.5.2 The Partition Function

We must now return to the partitioning task on which the efficiency of both versions of quicksort depends. To determine a partitioning entry, and a corresponding rearrangement, the first array entry is selected as a basis. This is a somewhat arbitrary selection and will be discussed more fully later. In general, this first array entry will not be in its proper sorted position. It will need to be shifted, and the array rearranged, so that all larger entries appear in the upper component and all smaller entries appear in the lower component.

A clever way to do this requires two pointers, upper and lower. Initially, upper is set to i, the first array index, and lower to j, the last array index. The initial situation is shown in Figure 8.16(a).

Lower is now moved upward until an entry larger than data [upper].key is encountered. In this case, 15 will be that entry. It is then interchanged with data[upper]; see Figure 8.16(b). At this point, attention is shifted to upper, which is moved downward until an entry smaller than data[lower].key is encountered. This entry will be 10, and it is interchanged with data[lower], yielding Figure 8.16(c).

Figure 8.16 Sorting an Array by Means of a Partitioning Function

Attention now shifts back to lower, which is again moved upward until an entry greater than data[upper].key is encountered. This will be 30, and it is interchanged with data[upper], resulting in Figure 8.16(d).

Upper is then moved downward until an entry smaller than data[lower].key is encountered. In this case no such entry appears, and when upper reaches lower, no such entry can appear. This is so because, at all times, any entries above upper are greater than the original first entry (13 in the example), and any entries below lower are less than the original first entry. Hence, whichever pointer is being moved, only entries between upper and lower can result in an interchange. When upper and lower coincide, no entries are left to interchange. Furthermore, when upper and lower coincide, they must be pointing to the original first entry or basis, since at all times at least one of them points to it. The final situation, then, is as shown in Figure 8.16(e).

This procedure clearly carries out the task of partition(i,j), which is to return the value of upper(or lower). Notice that, whether upper or lower is the focus of attention, it is always the original first entry (13 here) that is the basis for comparison.

Figure 8.17 Partitioning an Array by Means of a Temporary Variable, SAVERECORD

A slight modification makes partition more efficient. Instead of interchanging the basis with the larger or smaller entry that has been found, simply move the larger or smaller entry to the position pointed to by the current stationary pointer. This is the position that would have been occupied by the basis (13 in the example). However, the basis does not now appear in the array but is saved in a temporary variable, saverecord. For the example, this would generate the sequence shown in Figure 8.17. When upper and lower meet, the record stored in saverecord must be copied into data[upper]. This version of partition is faster because a move is carried out much faster than an interchange, and many such operations are required as the algorithm executes. It is therefore the algorithm implemented below.

partition(i,j)

/* Rearranges the entries in array

data between positions indexed by

i and j and returns an array index

which guarantees that all entries

above it are larger than the entry

it indexes, and all entries below it

are smaller.

*/

int i,j;

{

int upper,lower;

record saverecord;

>Comment

upper = i;

lower = j;

>Comment

copy(&saverecord,&data[i]);

    >Comment

while(upper != lower)

{

    >Comment

while((upper < lower) && (saverecord.key >=

data[lower].key))

lower--;

if(upper != lower)

    >Comment

copy(&data[upper],&data[lower]);

    >Comment

while((upper < lower) && (saverecord.key <=

data[upper].key))

upper++;

if(upper != lower)

    >Comment

copy(&data[lower],&data[upper]);

}

    >Comment

copy(&data[upper],&saverecord);

    >Comment

return(upper);

}

It is important to recognize that whenever partition (i,j) is invoked, it processes only entries between i and j of the data array and deals with each of these entries exactly once. Each entry is compared with saverecord, and perhaps copied into data [upper] or data [lower]. Consequently, a call to partition (i,j) takes time O(j - i + 1), since there are j - i + 1 entries between i and j.


Data Structures Using C: 8.5.3 Analyzing Iterative Quicksort

8.5.3 Analyzing Iterative Quicksort

We now want to analyze the nonrecursive version of quicksort to determine its time and storage requirements. To provide some insight into its behavior, we will simulate its application to our initial array, focusing on the changes that occur in this array and in the stack during execution.

Initially, the stack is empty, and the array is as shown in Figure 8.18(a). When quicksort (1,8) is invoked, it calls partition (1,8), which returns with p pointing to the partitioning entry and the rearranged array of Figure 8.18(b).

The upper component array is the larger, so (1,4) is placed on the stack, and the lower component array is processed. This leads to a call to partition (6,8). When partition returns, the situation is Figure 8.18(c).

Figure 8.18 Iterative Quicksort

The current upper component array is larger, so (6,7) is placed on the stack, and the lower component array is processed. Since the lower component contains no entries, (6,7) is removed from the stack and partition(6,7) is invoked, returning the situation depicted in Figure 8.18(d).

Now (6,6) is placed on the stack, and the lower component is processed. Again, this component has no entries, so (6,6) is removed from the stack. Since top (= 6) is not less than bottom (= 6), the outer while loop body is now executed, causing (1,4) to be removed from the stack. At this time the stack is empty, and the array is as shown in Figure 8.18(e). Partition(1,4) is now invoked, and returns, yielding Figure 8.18(f).

Now (1,3) is placed on the stack. The lower component again contains no entries, so (1,3) is removed from the stack, and partition(1,3) is invoked. The result is Figure 8.18(g).

At this point (2,3) is placed on the stack. Since the upper component contains no entries, (2,3) is removed from the stack, and partition(2,3) is invoked. It returns with Figure 8.18(h).

Now (2,2) is placed on the stack. The lower component has no entries, so (2,2) is removed from the stack. Since top (= 2) and bottom (= 2), the outer while loop tests for an empty stack. As the stack is empty, quicksort terminates.

It should not be clear that each time the inner loop body of quicksort is executed, and top < bottom, p is set to point to an entry of the array that has been correctly placed in its final sorted position. If top = bottom, then the one entry is already in its proper sorted position. The inner loop body is thus executed at most n times. Ignoring the time taken by partition, the inner loop body takes a constant time to execute. Each time the inner loop body executes, it generates one new problem that is placed on the stack and another new problem that is processed immediately. As soon as this latter problem has fewer than two entries (top � bottom), the inner loop is exited. Unless the stack is empty, the outer loop removes the next problem from the stack, and the inner loop then functions again. Consequently, each entry that is placed in its proper position requires, at most, an execution of the inner loop body or an execution of the inner loop body plus the additional constant time required by the outer loop test plus removal of a problem from the stack. This means that the total time is, at most, c1 + c2 x n plus the total time taken by partition, for some constants c1 and c2.


Data Structures Using C: 8.5.4 The Worst and Best Cases

8.5.4 The Worst and Best Cases

To analyze the total time of partition, recall that each execution takes time O(length of the component array it is called to work on). Since the length of a component is at most n, with partition being invoked at most n times, total time is at most O(n2). Hence quicksort has a worst-case time O(n2). You should convince yourself that the worst case occurs whenever the initial array is in sorted or reverse sorted order.

The best case (fastest execution) actually occurs when the partitioning entry returned by partition splits the array in half. In this case, partition is called to work on one component array of length n, two component arrays of length n/2, four component arrays of length n/4, . . . , 2k component arrays of length n/2k, etc. By an argument similar to the worst-case analysis of the binary search, about 1g n such calls will be made to partition. Its total time will then be O(n 1g n).

While the worst-case time for quicksort is poor, its best-case time is fast�in fact, better than heapsort. The average time for quicksort is also O(n 1g n). This assumes that each ordering of the n entries is equally likely to appear in the initial array. Evidently, cases differing significantly from the best occur rarely, and this is why quicksort is aptly named.

The storage requirements for quicksort, beyond the initial array, are a few variables plus the stack depth. Since the larger of each generated component array problem is stacked, the next problem to be stacked must have length at most one-half the current top stack entry length. For example, in the preceding section, note that when (6,7) was placed on the stack, its length was no greater than half the length of the preceding stack component (1,4). Again, by an argument similar to the worst-case binary search analysis, at most O(1g n) such problems can appear on the stack at any time, so the storage requirements for quicksort are O(1g n). Thus, quicksort, while faster on the average than heapsort, does not have the guaranteed worst-case O(n 1g n) time of heapsort. Moreover, quicksort requires additional storage O(1g n) to achieve its faster average time.

For smaller values of n, the simpler sorts, like the insertion sort, will be even faster than quicksort. Of course, if the initial array is known to be nearly sorted, the insertion sort will be faster anyway. Clearly, one way to speed up quicksort is to modify it by using a faster sort, such as insertion sort, when small component arrays are encountered. It is also possible to enhance the performance of quicksort by taking more care in the selection of the entry to be used as a basis by partition. Instead of taking the first entry of a component as the basis, the median of three entries may be selected, or an entry may be selected at random.


Data Structures Using C: 8.5.5 Distributive Partitioning

8.5.5 Distributive Partitioning

Although sorting has been extensively researched, a new generalization of the quicksort, distributive partitioning, has recently been discovered [Dobosiewicz,1976]. It appears to be even faster than quicksort. Recall that the quicksort was based on the idea of rearranging the initial array so that it could be split into two component arrays to be sorted independently. Each of these arrays was then recursively sorted in the same way. Distributive partitioning carries this idea to its logical limit, by splitting the initial array into n component arrays, to be sorted independently and recursively. The price paid by this new algorithm is that additional storage of size O(n 1g n) is required. Note that all the sorting algorithms considered sort in place, except the quicksort and distributive partitioning.Quicksort and distributive partitioning require additional storage, limiting their use for large n. Since quicksort's additional storage increases 1/nth as fast as that of distributive partitioning, it will not be as severely limited.


Data Structures Using C: 8.6: Simulation of an Algorithm

8.6: Simulation of an Algorithm

It is often difficult, if not impossible, to analyze an algorithm mathematically and determine its time and storage requirements. Simulation is a useful basic tool for determining the behavior of an algorithm for this purpose. To illustrate, an algorithm for a simulation of the heapsort is constructed in this section.

The storage required by the heapsort algorithm in the implementation is O(n). A reasonable measure for the time required is given by the number of interchanges and comparisons necessary to produce a sort. Assume that a procedure treesort has been written that is one of the detailed implementations discussed previously for heapsort. It has parameters n, a, sa, int1, int2, comp1, and comp2. These represent, respectively, the number of records, the input array of records to be sorted, the output array of sorted records, the number of interchanges required in phases I and II, and the number of comparisons required in phases I and II.

The simulation program to be written must explore the time requirements, as measured by int1, int2, comp1, and comp2, for various values of n, such as 20, 40, 60, 80, and 100. This will suggest how the execution time depends on n. Alternatively, the actual execution times could be measured. Intuitively, the algorithm will have fastest execution time when the input is nearly sorted, the slowest time when the input is nearly reverse sorted. For random initial orderings, the time can be expected to fall between these two extremes. This may not be the case but seems likely a priori. Consequently, we want to generate a number of samples from each of these three distinct classes of input, execute the treesort routine for each sample, and collect statistics on the numbers of interchanges and comparisons based on the samples taken.

We will choose twenty-five samples of each type of input, for a total number of executions of treesort of (twenty-five samples for each input class and value of n) x 3 input classes x 5 values of n, or 375 runs. One way to see if twenty-five samples is enough to achieve good statistics is to run an additional twenty-five. If the results do not vary significantly, then the twenty-five samples, or perhaps even fewer, are sufficient. If necessary, we can use a larger number of samples. Remember, the twenty-five samples are taken from a possible number of n! samples. It may seem the height of foolishness to base estimates on such a ridiculously small fraction of the possibilities. Thank goodness for statistics (when properly applied).

                          n = 40

                       -------------

  Input Class          Min  Ave  Max

------------------------------------

"Nearly sorted"  int1   *    *    *

                 int2   *    *    *

                 comp1  *    *    *

                 comp2  *    *    *

"Random"         int1   *    *    *

                 int2   *    *    *

                 comp1  *    *    *

                 comp2  *    *    *

"Nearly reverse  int1   *    *    *

  sorted"        int2   *    *    *

                 comp1  *    *    *

                 comp2  *    *    *

For each of the twenty-five samples for a given input class, the task is to find the following statistics for int1, int2, comp1, and comp2: their minimum, average, and maximum over all twenty-five sample values. Other statistics, such as the range and standard deviation, could be calculated if needed. We will have the simulation program print this information for each value of n, as indicated for n = 40 in the chart on the previous page. The *'s represent actual numeric values that will be output. A total of thirty-six values will be printed for each n; 5 � 36 =180 statistics.

The program will keep statistics tables, one table for each input class for the current value of n. An algorithm for the simulation program is as follows:

1. Set n to 20.

2. While n � 100

a.Set ns to 1.

b. Initialize statistics tables.

c. While ns � 25,

i. Generate a sample from each input class, call treesort to work on each input, and then update the appropriate statistics tables.

Set ns to ns+1.

d. Output the statistics tables for the current value of n and for each input class.

e. Set n to n+20.

Task (i) may be carried out as follows:

a. Generate a "random" sample for a of size n and call treesort(n, a, sa, int1, int2, comp1, comp2).

b. Update the statistics table for "random" input.

c. Generate a "nearly sorted" input for a of size n by making 20 percent of n random interchanges on sa. Copy sa into a.

d. Generate a "nearly reverse sorted" input for a of size n by copying sa in reverse order in ra. Call treesort(n, a, sa, int1, int2, comp1, comp2).

e. Update the statistics tables for "nearly sorted" input.

f. Call treesort(n, ra, sa, int1, int2, comp1, comp2).

g. Update the statistics table for "nearly reverse sorted" input.

h. Set ns to ns+1.

At this point we can explain in more detail what is meant by the three classes of input. Assume that all input arrays, a, will contain integers between 1 and 1,000. Random inputs for a of size n are generated by placing the n integers into a in such a way that all orderings of the n numbers are equally likely. Consider the two arrays shown in Figure 8.19.

Array (b) has been obtained from array (a), which is sorted, by making two interchanges. These were interchanging a1[2] with al[6], and interchanging a1[4] with a1[9] . These two interchanges represent 20 percent of n interchanges, since n is 10. A nearly sorted input for a of size n is obtained by making 20 percent of n such interchanges in an initially sorted array. The interchanges are to be selected at random. This means that each interchange is obtained by picking a value for i at random, so that each of the locations 1 to n of sa is equally likely to be selected. An integer j may be similarly obtained. Then sa[i] and sa[j] are interchanged. The only details still left to implement in the simulation algorithm are how to generate the required random integers to be placed in a and used for interchange locations, how to update the statistics tables, and how to finalize the tables before printing them.

Figure 8.19 Array (b) Obtained from Sorted Array (a) by Two Interchanges

Assume that a function rng is available that returns a real number as its value. This real number will be greater than 0.0 and less than 1.0. Whenever a sequence of these numbers is generated by consecutive calls to rng, the numbers appear, from a statistical point of view, to have been selected independently. Also assume that the returned value will lie in a segment of length l between 0 and 1, with probability l. For example, if l is of length 0.25, then the returned value will lie in that interval (say, between 0.33 and 0.58) 25 percent of the time. The random number generator ranf of FORTRAN is an example of such a function.

To generate an integer between 1 and 1,000, take the value rng returns, multiply it by 1,000, take the integer part of the result, and add 1 to it. For example, if the value returned is 0.2134, multiplying by 1,000 gives 213.4. Its integer part is 213; adding 1 yields 214. In fact, if any number strictly between 0.213 and 0.214 had been returned, 214 would have been generated. Hence 214 would be generated with probability 1/1,000, the length of the interval from 0.213 to 0.214. The same is true for each integer between 1 and 1,000. Carrying out this procedure n times will fill the array a to generate a "random" input (See Exercise 23 for a more exact approach). An alternative method in C to generate an integer at random between integers low and high uses the library function rand. Evaluating rand()%(high-low+1) + low yields the desired integer between low and high.

To generate a nearly sorted or nearly reverse sorted input, select each of the 20 percent of n interchanges as follows. Call rng twice. Multiply the two returned values by n, take the integer part, and add 1. Or evaluate rand()%(high-low+1) + low twice with high set to n and low set to 1. Either method gives two integers, i and j, between 1 and n. Interchange sa[i] with sa[j].

The statistics tables can keep current running minimums, maximums, and the accumulated sum of the appropriate statistic. When each call to treesort returns, the minimum, maximum, and accumulated sum may be updated for each statistic, int1,int2,comp1, and comp2. When the inner loop of the algorithm is exited, and task d is to be carried out, the statistics tables are ready to be printed, except for the twelve averages. They must be finalized in task d by dividing the accumulated sums by 25.

The algorithm presented in this section can be used directly to obtain simulation results for treesort. Simply replacing the call to treesort in task (i) by a call to suitably modified versions of the other sort programs will produce their simulation results as well. To use simulation in the analysis of other programs, not necessarily sorts, requires some insight into what input is appropriate to generate and what statistics are meaningful to collect. This is not always so straightforward as in our example, but the general procedure and goal for the simulation of an algorithm should now be clear.


Data Structures Using C: 8.7: Simulation Results for the Sorts

8.7: Simulation Results for the Sorts

Table 8.1 gives simulation results (in actual execution time measured in 1/1000 second intervals). It should be carefully studied. Note that the O(n) and O(n lg n) sorts may be easily identified. The best and worst input cases are also evident, as well as the relatively uniform behavior of heapsort.

Each entry in the table is the average of 20 runs on a VAX 730 under VMS Version 4.0. For each n the five values correspond to input that is sorted, half sorted�half random, random, half reverse sorted�half random, and reverse sorted. If the entries for heapsort seem counter to your intuition, remember that its implementation sees sorted input as reverse sorted and reverse sorted input as sorted. This is because it uses a min heap to produce descending order.


Data Structures Using C: 8.8: Synopsis of Search and Sort Efficiencie

8.8: Synopsis of Search and Sort Efficiencies

We have consistently viewed the record as the basic unit of data. Collections of records usually appear in problems, and their processing typically reduces to traversing through the records of the collection, inserting or deleting records, randomly accessing records, accessing records in sorted order, or searching the collection for a given record. The basic data structures are available for storage of the collection, and each supports some operations well at the expense of others. Arrays are excellent for random access and traversal but poor for general insertions and deletions. Lists support traversal, insertions, and deletions well but are poor for random access.

Linear searches of arrays and binary searches of sorted arrays can be performed, respectively, in O(n) and O(1g n) time. Under special conditions, linear searches can be fast.

Simple sorting algorithms such as the bubble and insertion sort have worst-case time O(n2) but are O(n) when dealing with almost sorted data. Quicksort gives very fast average times and heapsort does sorting in worst-case time O(n 1g n).

Table 8.1 Simulation Results in Actual Execution Time (milliseconds)*

 n    Bubble Sort                        Insertion Sort                     Modified Insertion Sort

-----------------------------------------------------------------------------------------------------------

 10    1       2       6       8      9     1      2      5      7      7     0     2       4      6      9

 20    1       8      23      33     36     1      5     16     28     30     1     6      17     25     30

 40    2      31     101     138    148     2     16     67    110    121     5    16      65    107    120

100    4     217     646     863    923     7     89    412    678    761     6    87     402    660    744

200    8     966    2653    3463   3724    12    364   1657   2706   3058    11   354    1612   2639   2993

400   17    3933   11277   14732  15858    24   1583   7645  12489  14595    20  1373    6352  10581  12157

-----------------------------------------------------------------------------------------------------------

     srt'd  half   rnd'm    half   rev    srt'd  half   rnd'm  half    rev   srtd  half   rndm   half   rev

            srt'd           rev   srt'd         srt'd          rev   srt'd        srt'd          rev  srt'd

             half          srt'd                 half         srt'd                half         srt'd

            rnd'm          half                 rnd'm         half                rnd'm          half

                           rnd'm                              rnd'm                            rnd'm

 n    Heapsort                          Iterative Quicksort                  Recursive Quicksort

-----------------------------------------------------------------------------------------------------------

 10     9      9      8      8      7     10      9     9      8     10      5     5      3      3      4

 20    18     19     18     18     15     21     19    15     17     23     13    11     10     11     14

 40    43     43     41     40     37     51     47    33     38     55     35    31     21     25     36

100   130    126    122    119    115    196    171    94    106    202    156   133     65     74    158

200   291    282    277    264    256    626    521   202    231    633    581   484    149    169    602

400   640    624    610    587    572   2175   1750   432    492   2195   2042  1644    318    372   2056

-----------------------------------------------------------------------------------------------------------

    srt'd   half  rnd'm    half    rev  srt'd   half  rnd'm   half   rev   srtd   half   rndm   half   rev

           srt'd           rev    srt'd        srt'd          rev   srt'd        srt'd          rev  srt'd

            half          srt'd                 half         srt'd                half         srt'd

           rnd'm           half                rnd'm          half               rnd'm          half

                          rnd'm                              rnd'm                             rnd'm

-----------------------------------------------------------------------------------------------------------

* These results were produced by Sharon Doherty.

It is important to recognize that in this chapter we have been considering comparative searches and sorts. This means that we have assumed no special knowledge of the key values involved and have used only comparisons between key values in each search or sort algorithm. It is possible to prove that such searches and sorts have worst-case times O(lg n) and O(n lg n), respectively.

Other kinds of search and sort algorithms are possible. Some may take significantly less time. For example, suppose you must sort n distinct keys whose values are known to be integers between 1 and 1,000. Simply store the record with key value i as the ith record in an array of records. This places the records in the array in (decreasing) sorted order and takes O(n) time. Sorts that place records in groups, depending on their actual key values, and then attempt to sort further within the groups, are called distributive sorts. These are not considered in this text. We will, however, consider a very important noncomparative search in Section 9.3.

Simulation is a basic tool for the analysis of the storage and timing requirements of algorithms although we have used it only for analyzing sorts.

As searching as this chapter may have been, it is hoped that it didn't leave you out of sorts!


Data Structures Using C: Exercises

Exercises

1. a. If all comparisons and assignment statements take the same amount of time to execute, then compare the execution times of the first linear search procedure and the procedure revised for greater efficiency.

b. For what values of n will the binary search be faster than the linear search of the improved procedure in the worst case?

2. When you look up a book by author in the library, what kind of a search are you doing?

3. In a ternary search of a sorted array, the array is divided into three "nearly" equal parts and searched. Key values are examined "near" 1/3 and "near" 2/3 and compared with the search key to determine in which third to search further. Determine the worst-case time for this search and compare it to the worst-case time for a binary search.

4. Suppose a sorted array stores 1,000 integers between 0 and 1,000,000. A pointer array indicates where the least integer greater than 1,000 x (i - 1) but not greater than 1,000 x i appears in the array. P[i] contains that pointer for i = 1, 2, . . . , 1,000. If no such integer is in the array, then p[i] = 0. Write a function to do an interpolation search, coupled with a binary search.

5. Assuming all comparisons and assignments take the same time to execute, do the following.

a. Determine the worst-case times for the sorts of the procedures given in the text for the maximum entry sort, the bubble sort, and the insertion sort.

b. For each value of n, determine which of the three sorts has least worst-case time.

6. What input order produces the worst case time for

a. The bubble sort?

b. The insertion sort?

7. Write a function to carry out the bubble sort when the integers are stored in a chain.

8. Write a function to carry out the insertion sort when the integers are stored in a chain.

9. Modify the insertionsort function so that it does a binary search for the proper insertion position, and then does the insertion.

10. Modify the function so that it does not deal with the last i elements after the ith pass.

11. Suppose input is given in the order

16, 18, 22, 15, 4, 50, 17, 31, 4, 90, 6, 25

Create the heap produced by

a. The first heap creation algorithm of Section 8.4.

b. The second heap creation algorithm of Section 8.4.

c. How many comparisons and interchanges were required in Exercise 11(a) and Exercise 11(b)?

12. For each heap of Exercise 11(a) and 11(b), produce

a. The reheaped heap after the root element is removed using the first reheaping algorithm of Section 8.4.

b. The reheaped heap after the root element is removed using the second reheaping algorithm of Section 8.4.

c. How many comparisons and interchanges were required in Exercises 12(a) and12(b)?

13. Why does the pointer array of Section 8.3.6 fail to help?

14. Show the entries of the data array, for the input of Exercise 11, when the heapsort function is applied after the first three elements have been output.

15. Modify the heapsort function so that it counts the number of interchanges and comparisons in each phase and returns their values as parameters.

16. Modify the heapsort function so it sorts a data array of pointers to records instead of records themselves.

17. What input sequence will cause the second heap creation algorithm to take worst-case time? Calculate the number of comparisons and interchanges it requires.

18. Determine the number of comparisons and copies required by quicksort for an array of n records, which is (a) in sorted order; (b) in reverse sorted order; (c) in order so that partition always produces two component arrays whose lengths differ by at most 1.

19. Modify partition so that it selects an entry at random to be used as the basis entry.

20. Modify partition so that it selects the median of the first, middle, and last entries as a basis entry.

21. Explain why the implementation given for quicksort requires a stack depth O(lg n).

22. Modify quicksort so that it sorts all component arrays of length at most 20 by using insertionsort.

23. Generation of n integers to fill the array a in Section 8.6 randomly was really only an approximation of a random input. This is because duplicate values might occur. While the effect of this duplication may be negligible for the simulation results, a "true" random input can be generated. You can fill array a as follows: For i from 1 to n, set a[i] to i.Then, for i from 1 to n, pick an integer int at random between i and n, and interchange a[i] and a[int].

a. Convince yourself that this will generate a "'true" random input.

b. Write a procedure to generate this "true" random input.

24. Consider the "perfect shuffle" example of Chapter 3. When 2n - 1 is a prime number p, then the number of shuffles required is p - 1. The initial configuration is in sorted order, and so is the final configuration after p - 1 shuffles. The configurations produced along the way represent different permutations of the initial input. With successive reshufflings, the configurations seem more and more like a "random" arrangement, and then less and less like a "random" arrangement. Take p = 101 and use the one hundred configurations, produced by the hundred shuffles required to reproduce the original, as inputs to the heapsort algorithm. Output the statistics table based on one hundred samples of an input of size 101.


Data Structures Using C: Suggested Assignments

Suggested Assignments

1. Write and run the heapsort function of Section 8.4.7. A main program should read in examples for the input array a. For each example, n and the array a should be printed, and heapsort should be called to sort a. When heapsort returns, the output array, sa, should be printed along with the values for int1,int2,comp1, and comp2. At least two examples for small n should be worked through by hand to test the correctness of heapsort.

2. Write an efficient recursive function minheaps (t,count,flag) to return in count the number of subtrees of the binary tree pointed to by t that are minheaps, and to return in flag true if t points to a minheap and false otherwise.


Data Structures Using C: CHAPTER 9: MORE SEARCHING: INSERTION AND DEL

CHAPTER 9: MORE SEARCHING: INSERTION AND DELETION

Deals with dynamic data structures, which

can grow and contract in size

support efficient searches

The data structures are

priority queues, which

support insertions and deletions

hash tables, which

support insertions and deletions

have excellent average search time

do not support ordered traversals

binary search trees, which

support searching, inserting, and deleting

have good average times for these operations

support ordered traversing

balanced binary search trees, which

support all the operations of binary search trees

also support fast access to information in an arbitrary position in a sorted

ordering of the information stored

have good worst-case time for all the operations

Worst-case and average times are given for the operations on each data structure

9.1: Overview

9.2: Priority Queues

9.3: Hash Tables

9.4: Binary Search Trees

9.5: A Brief Review

Exercises

Suggested Assignments


Data Structures Using C: 9.1: Overview

9.1: Overview

In the previous chapter, techniques were presented for ordering and storing data so that it could be efficiently searched and traversed when desired. Those methods work well as long as the collection of data is essentially static�meaning that relatively few insertions and deletions are required. This is a general characteristic of arrays, which were the basis for the faster binary search: they support efficient search but not efficient insertion and deletion. Lists can be used for more efficient insertion and deletion, but then a binary search is not possible.

In this chapter the focus is on achieving efficient search and efficient insertions and deletion. Sometimes traversals through the data in sorted order by key value are also desirable. Four data structures that are appropriate for dealing with dynamic data are presented. First, priority queues are discussed. They provide the means to deal with a special case of insertion and deletion, as well as serving as an application of heaps. Next, hash tables are considered. Hash tables are important data structures that have excellent average search, insertion, and deletion times. Finally, binary search trees and AVL trees are introduced. These data structures support all the operations�search, insertion, deletion, and traversal�quite well.


Data Structures Using C: 9.2: Priority Queues

9.2: Priority Queues

It often happens that there are a number of tasks, n, each with an associated priority. The n tasks are to be carried out in order of priority. When studying for examinations, most students review and prepare in an order of priority determined by the date of the exam. Most people read the chapters of a book in order, with priority determined by chapter numbering. In this case the situation is static, since the number of chapters does not change. Computer centers process submitted programs by attaching a priority to each and processing them in order of priority. Here the number of programs does change. This represents the usual case, where it is necessary to add new tasks with their priorities to the current collection. The general situation is covered by a priority queue.

A priority queue is a data structure with objects, their priorities, and two operations, insertion and deletion. Insertion in a priority queue means that an object with its priority is added to the collection of objects. Deletion from a priority queue means that an object with highest priority is deleted from the collection. A stack may be thought of as a priority queue with the highest priority object the one most recently added. A queue may be thought of as a priority queue with the highest priority object the earliest added.

High-level programming languages do not have priority queues directly available in their memories. It is important to be able to implement such queues, because they are of use in their own right and (as will be shown later) often appear as components of algorithms.

9.2.1 Simple Implementations

9.2.2 Using a Heap

9.2.3 Using Leftist Trees


Data Structures Using C: 9.2.1 Simple Implementations

9.2.1 Simple Implementations

To implement a priority queue, first represent its objects and priorities as records with priority fields. Two ways to store the records come immediately to mind. The records can be stored in an array or in a list. In either case, the programmer may choose to store the records in arbitrary order or in order of priority. Table 9.1 gives the worst-case insertion and deletion times for each of the four resultant implementations, when n records are present in the priority queue. For example, when using the ordered array implementation, the top array record will have highest priority; thus deletion can be performed in constant time by using a pointer top to the current top record. Insertion of a record requires a search of the array to find its proper position. A binary search will allow this position to be found quickly, but to insert it requires O(n) time, because all records below it must be shifted down.

Table 9.1 Worst-Case Insertion and Deletion Times for the Priority Queue

           Implementation  Insertion  Deletion

----------------------------------------------

           Array           Constant   O(n)

Unordered  List            Constant   O(n)

           Array           O(n)       Constant

Ordered    List            O(n)       Constant

The table shows that if insertions are more frequent than deletions, it is better to use unordered records. In addition to considering the time for insertion and deletion, the amount of storage required for the list and array implementations must be evaluated. Arrays take less storage.


Data Structures Using C: 9.2.2 Using a Heap

9.2.2 Using a Heap

Another way to implement a priority queue is to use a heap. You have seen how to build a heap efficiently so that storage is minimal, heap creation takes at most O(n) time, and reheaping takes at most O(lg n) time. To use a heap for the priority queue, the records are stored at the nodes of a heap that is ordered by priority field values. To insert a new record, place it as a new rightmost entry at the greatest depth of the heap. Then let it work its way up in the heap as in the first heap creation algorithm. Adding the new record in this way will take time at most O(lg n). To delete a record, simply delete the record at the root of the heap, and then reheap by invoking shift. This takes time at most O(lg n). The time to create the heap initially will be at most O(n) if the second heap creation algorithm is used. This is also the time that would be required to create an initial unordered array or unordered list of records. To create sorted arrays or lists takes time at most O(n lg n). Referring to Table 9.1, you can see that the heap implementation affords a compromise solution. With arrays and lists, one operation can be done quickly (in constant time) and the other in O(n) time. With the heap, both operations can be done in at most O(lg n) time.


Data Structures Using C: 9.2.3 Using Leftist Trees

9.2.3 Using Leftist Trees

Still another way to implement a priority queue is as a leftist tree, which is a null binary tree or a binary tree with the following properties:

1. It is a heap.

2. Its left and right subtrees are leftist trees.

3. The path that is obtained by starting from the root and branching to the right until a node with no right successor is reached is the shortest path from the root to any node with at most one successor.

For example, the tree (a) of Figure 9.1 violates condition 3, while the trees (b) and (c) are leftist trees.

Figure 9.1 One Nonleftist and Two Leftist Trees

As an exercise, you should prove by induction on the number of nodes, n, in a leftist tree, that the shortest path in the tree, from the root to a node with at most one successor, contains at most �lg (n + 1)� nodes. Such a shortest path can always be obtained by traversing the right chain from the root.

Leftist trees are implemented using a linked representation. (See Figure 9.2) Each node's record has an additional field, the distance field, containing the number of nodes along the right chain.

A leftist tree implementation of a priority queue has a number of pluses. Both additions and deletions can be done in constant time when the priority queue is acting as a stack. The time for these operations is O(lg n) for the general case. Also, two priority queues can be merged into one in O(lg n) time, where n is the total number of entries in the two priority queues to be merged. This is significantly faster merging than can be done with any of the previous methods, except for unordered lists. Crane [1972] introduced leftist trees as a data structure that can be used to implement priority queues. (See Knuth [1973b] for more details.)

Figure 9.2 The Merging of Leftist Trees Implemented in Linked Representation

The algorithm for merging two priority queues represented as leftist trees in O(lg n) time is as follows:

1. Make node.p the root of the merged tree, and merge q with p's right subtree.

2. Update the distance field of node.p.

3. If the distance of the root of the left subtree is not at least as great as the distance of the root of its right subtree, then interchange the two subtrees.

P is assumed to point to that root of the two leftist trees to be merged which contains the maximum key value, and q points to the other root. The result of merging the two trees p and q is shown as the tree r. The distance field values appear in parentheses. Notice in Figure 9.2 that, in effect, the two right chains of p and q are merged, with the other subtrees coming along for the ride. It is because only these two shortest paths participate in the algorithm that the time is O(lg n).


Data Structures Using C: 9.3: Hash Tables

9.3: Hash Tables

The next data structure we consider is the hash table. A hash table is a data structure for the storage of records. It provides a means for rapid searching for a record with a given key value and adapts well to the insertion and deletion of records. Many compilers, for example, use hash tables to implement their symbol tables. They are also used extensively in systems where the primary objective is information retrieval. Thus nametable of Chapter 6 or a dictionary of words might be implemented in a hash table.

To illustrate the need for hash tables, suppose we want to store 1,000 records representing 1,000 customer accounts. Assume that each record has an account number as its key and, since we control account numbers, they range from 1 to 1,000. The solution is simple. Store the record with key value i in element i of an array data. Then, given a key value to search for (say, 836), go directly to element 836 of data to find it. What could be a quicker search? We are given a record and know immediately where to go to find it. The search time is a constant, independent of the number of records stored. However, this kind of search requires the ability to assign key values so that all keys are within a range of array indices (1 to 1,000 in this case). It also requires that we have enough storage to accommodate all records.

Now suppose we are dealing with records representing information about our employees, and that the keys are social security numbers. Even with only 1,000 employees, there are 109 possible social security numbers. It is not possible to know in advance which of the 109 social security numbers our 1,000 employees will actually have. If we were to allocate storage using social security numbers as addresses, we would run into two problems. First, there would not be enough storage. Second, if we did have sufficient storage, 999,999,000 (109 - 103) storage elements would be wasted! Nonetheless, it is extremely attractive to be able to look at the search key value and to know where to find it. This is the goal of hash tables and hash searching�to find a record directly, given its key value.

9.3.1 Building a Hash Table

9.3.2 Searching a Hash Table

9.3.3 Random Hashing

9.3.4 Other Collision Resolution Policies: Open Addressing and Chaining

9.3.5 Searching in Ordered Hash Tables

9.3.6 Deletion from Hash Tables

9.3.7 Hashing Overview


Data Structures Using C: 9.3.1 Building a Hash Table

9.3.1 Building a Hash Table

This section deals with a scaled-down version of the social security example. There are eighteen records, each with a key value that is a three-digit number. The records and their key values will be given in some order, normally unpredictable. We are to build a table, implemented in an array, in which the eighteen key values will be stored, so that searching can be done efficiently. One way to do this is to build a special kind of table�a hash table. This table will then be searched for given key values. To build a hash table requires a hashing function and a collision resolution policy. A hashing function is a method of calculating an array or table address for any key. To be useful, the address must be quickly calculable.

The hash function should have other properties, as will become evident. The hash function used here assigns to any key the remainder after division of the key by the table size. If the table size is m and the key value is k, then k mod m denotes this remainder. The starting table or array size is 23.* The eighteen key values and their order are given in Table 9.2, along with the hashing function values assigned to each.

* With division-type hashing functions, the divisor, or table size, is usually a prime number greater than the number of entries to be placed in the table. We have taken this prime to be 23.

Table 9.2 Hash Addresses

               Hashing                Hashing

Key Value      Address  Key Value     Address

---------------------------------------------

   019     -->     19       468     -->    08

   392     -->     01       814     -->    09

   179     -->     18       720     -->    07

   359     -->     14       260     -->    07

   663     -->     19       802     -->    20

   262     -->     09       364     -->    19

   639     -->     18       976     -->    10

   321     -->     22       774     -->    15

   097     -->     05       566     -->    14

The hashing address for 019 is found, for example, by dividing 019 by the table size 23, to obtain 019 = 0 � 23 + 19. The remainder, 19, is the assigned hashing address. As each key value is input, its hashing address is calculated, and the key value is placed into that element of the table. After the first four key values are entered, the table is as shown in Figure 9.3(a).

When the fifth key value, 663, is input, its hashing address is 19, but element 19 of the table is already occupied by another key value. This event is called a collision. The collision resolution policy is the method used to determine where to store a value that has undergone collision. The most straightforward policy for resolving the conflict is called linear probing. It entails proceeding through the table from the collision element and placing 663, the colliding entry, into the first unoccupied element found. The hash table is assumed to have been initialized, so that each element was "unoccupied" or "empty." If the search for an empty element reaches the end of the table, it circles to the top of the table. In this case, 663 is entered at element 20. The final table is shown in Figure 9.3(b). A total of eight initial collisions occurred in building it.

Figure 9.3 Hash Table (a) after Four Entries and (b) Finally

Each time a location of the table is accessed in an attempt to insert a key value, one probe has been made. To enter the first key value required one probe; to enter the fifth key value required two probes. To enter keys 1 through 15 required, respectively, 1, 1, 1, 1, 2, 1, 4, 1, 1, 1, 2, 1, 5, 4, 7, 3, 1, 3 probes. You can build the table and confirm these probe numbers.


Data Structures Using C: 9.3.2 Searching a Hash Table

9.3.2 Searching a Hash Table

How would you search a hash table when asked to find a given key, such as 642? If the key were stored in the table, it would either be found at its hashing address or at the end of a linear probe. If its hashing address were empty, or if the search came to an empty element in the linear probe before finding the key, you might conclude that it was not stored in the table. In this case, the hash address of 642 is 21. Element 21 is not empty but does not have 642 as its value. If you do a linear probe, you will eventually come to element 3, which is empty, without finding 642; thus you would conclude that 642 is not in the table.

If the key searched for were 802, you could go to its hash address (20), find that element occupied with a different key, and start the linear probe, finding 802 at element 0.

Notice that if the search key value is present in the table, the search will require exactly as many probes to find it as were required to enter it into the table. The search traces out the same path. Instead, if the search is for a key value that is not present, the search will take exactly as many probes as would be required to enter it into the current table.

A search that finds a key value is called successful, and one that fails to find a key value is unsuccessful. Theoretically, one can calculate the number of probes required for every possible key value. Knowing the frequency with which each possible key value (there are 1,000 in our example) will be searched for, one can determine the average number of probes required to search the table. The maximum number of probes can always be determined given the table. In the example, a successful search would require at most seven probes, while an unsuccessful search (for example, for key value 248) could require nine probes. If one never had to search for a key not stored, and all stored keys were searched for with equal frequency, the average number of probes would be

  1/18 � 1 + 1/18 � 1 + 1/18 � 1 + 1/18 � 1 + 1/18 � 2 + 1/18 � 1

+ 1/18 � 4 + 1/18 � 1 + 1/18 � 1 + 1/18 � 1 + 1/18 � 2 + 1/18 � 1

+ 1/18 � 5 + 1/18 � 4 + 1/18 � 7 + 1/18 � 3 + 1/18 � 1 + 1/18 � 3

or 40/18 = 2.22 probes. This calculation involves multiplying the frequency of search for each record by the number of probes required to enter it, and then adding all these products. In this case, each frequency was 1/18, and the required probes were noted when the hash table was built.

If no collisions occurred in building the table, every successful search would take exactly one probe. If the hash function actually assigned every key value not stored in the table to an empty address, exactly one probe would be required for any search. This would be the ideal situation. It is apparent now that a desirable hashing function will "scatter" or distribute the key values throughout the table, so that relatively few collisions ensue.

The hash table was of length 23 and stored eighteen key values. Its usage ratio is 18/23; the table is 78 percent full. Intuitively, the lower this ratio, the greater the storage wasted, but the lower the likelihood of a collision. Appropriate hashing functions must be selected with care in real applications, but it is possible to achieve very quick searches using hash tables.


Data Structures Using C: 9.3.3 Random Hashing

9.3.3 Random Hashing

A great deal of literature in computer science is devoted to the selection of hashing functions and collision resolution policies. To give some idea of what can be achieved, this section states results for an idealized scheme that assumes the hashing function to be perfectly random. Given any key value, the scheme assigns an address to that key value by picking randomly from among all the addresses of the table. This random selection is made so that each of the m addresses in a table of size m is equally likely to be selected. The table is built by inserting, in turn, each given key value. A key value is stored by determining its hash address. If that address is empty, the key value is stored there. If that address is occupied, the collision resolution policy works as follows. Another random hash function again selects an address at random. If that address is empty, the key value is stored there. If it is occupied, this procedure is repeated until the key value is finally entered. Thus an insertion of a key value may require the generation of a sequence of hash addresses ending with the actual final storage address.

After the table is built, searches are performed as follows. Trace the sequence of hash addresses that were generated to store the search key (if it is in the table) or that would have been generated (had the key been stored). The basic assumption made here is that the hash functions generate precisely the same sequence of hash addresses for the search as for building the table. This may seem like a strange way for random hash functions to work, but it is our assumption nevertheless. Table 9.3 shows the theoretical results for such a scheme, when n keys are stored in the table of size m.

The successful and unsuccessful search columns are given approximately by the functions (1/a )1n 1/(1-a) and 1/(1-a) where a = n/m. The remarkable fact is that the average search time does not depend on n, the number of records stored in the table, but only on the usage ratio, n/m. Thus storage can be traded for speed in hash table searches. For example, if n is 10,000 and m is 11,112, a is about 0.90, and an average of only 2.56 probes will be required for a successful search. Contrast this with the 5,000 comparisons or probes needed for the linear search in the random case, or the 12.29 comparisons or probes needed for a binary search. Actual hashing functions and collision resolution policies can come close to achieving this kind of average behavior. The worst-case times for hash table searching can, however, be quite large.

Hash tables are not a searching panacea. A major disadvantage is that entries will not be in order by key. The only way to access the records in sorted order is to sort them. However, if this isn't important, hashing is often a good approach.

Table 9.3 Average Number of Probes for Searches

                 Successful  Unsuccessful

Usage Ratio n/m   Search       Search

-----------------------------------------

      .25          1.15         1.33

      .50          1.39         2.00

      .75          1.85         4.00

      .80          2.01         5.00

      .90          2.56        10.00

      .95          3.15        20.00


Data Structures Using C: 9.3.4 Other Collision Resolution Policies: Op

9.3.4 Other Collision Resolution Policies: Open Addressing and Chaining

The linear probing collision resolution policy is a special case of open addressing. The idea of open addressing is to resolve collisions by following a specific probe sequence whenever a collision occurs, in order to determine the insertion location for a new key. To be specific, let ki denote the key stored in the ith location of a hash table of size m. Assume that at least one table location is unoccupied and set all unoccupied locations to the "null" key value. This prevents "falling off" the table when searching for a key and reduces the search time, because an end-of-table search test need not be made.

Let h(k) denote the hash function value for search key k. The sequence of indexes determined by

[h(k) + i � p(k)]mod m* for i = 0, 1, 2, . . . , m - 1

is a probe sequence if the indexes are all distinct. This implies that, as i goes from 0 to m - 1, the generated indexes form a permutation of the indexes 0, 1, 2, . . . , m - 1. Consequently, in attempting a new key insertion, all locations of the hash table will be accessed in some order before deciding that the key cannot be inserted. It also ensures that, in attempting to search for a key, all locations of the table will be accessed in some order before concluding that the search key is not in the table, or until a null key is encountered.

* The formula states that the ith index in the probe sequence is found by calculating [i + p(k)]mod m. However, if the previous index, the (i - 1)th, is known, then simply adding p(k) to it and taking the result mod m will also yield the ith index.

Linear probing is the special case obtained by taking p(k) = 1 for all keys k. In general, a key is inserted by accessing the table locations given by its probe sequence, determined by [h(k) + i � p(k)]mod m as i goes from 0 to m - 1, until a null key is encountered. The key is then inserted in place of that null key. A search is done the same way. The algorithm for a hash table search using a probe sequence is as follows:

1. Set i to h(k)

2. While ki is not the search key k and ki is not the null key

set i to [i + p(k)]mod m.

3. If ki = k, then

set found to true

else

set found to false.

Notice that p(k) represents a displacement from the current probe address.

With linear probing, large clusters of keys stored in adjacent locations of the hash table tend to get larger as insertions are made. This is because large clusters have a greater probability of the new keys' hash address falling within the cluster, assuming a "random" hash function and "random" keys. A cluster gets larger when a new key is inserted at either end of the cluster, or when two separated clusters become connected because of the new key insertion. Both of these events occurred in the scaled-down social security number example. This phenomenon is called primary clustering. As the table usage ratio increases, primary clustering leads to greater search and insertion times. This is clearly undesirable. It happens because the large clusters must be traversed, with the average time being proportional to half the cluster length when keys are searched with equal frequency. Thus the insertion, or search for a key, begins to look like a linear search through a cluster.

To avoid primary clustering, try taking the next probe to be in a position removed from the current one instead of being adjacent to it. This is open addressing with displaced linear probing. Taking p(k) = a for some integer a that is not a factor of m will give such a probe sequence. To see this, take m = 13 and a = 5. The probe sequence is then generated by [h(k) + i � 5] mod 13 for i = 0, 1, 2, . . . , 12. If h(k) = 7, this gives the probe sequence

7, 12, 4, 9, 1, 6, 11, 8, 0, 5, 10, 2

Note that any key k whose hash address is 7 will have exactly the same probe sequence. Even worse, suppose a key's hash address is 11, and 11 is occupied upon its insertion. The next probe for this key will be in location [11 + 1 � 5] mod 13, which is location 3. A collision at 3 will result in a further probe at [3 + 1 � 5] mod 13 or 8, then 0, 5, etc. Thus the key that originally hashed to address 11 picks up at the intersection of 11 with the probe sequence for any key hashing to 7, and follows that same probe sequence from 11 onward. Similarly, any key that collides, upon insertion, with a key already occupying its hash address, will continue along this exact probe sequence from its point of entry.

In effect, there is exactly one probe sequence for all keys; they just enter, or merge into it, at different positions specified by their hash addresses. This means that, as a "cluster" builds up along this probe sequence (say, at 11, 3, 8, 0, and 5, as they become occupied), it has the same tendency to grow as a cluster of physically adjacent occupied addresses did with linear probing. This is just primary clustering, although the clusters do not appear in adjacent locations.

We need effective techniques for dealing with this problem. Figure 9.4 gives the hash tables that result from their application to our example. The hash tables produced by linear probing, and the displaced linear probing just discussed, appear in tables (a) and (b) of the figure, corresponding to a = 1 and a = 4, respectively. In addition, the number of probes required for the insertion of each key is indicated for each table. You should create these tables yourself to confirm their correctness and to aid in understanding the discussion.

The average number of probes for a successful search of this table is 1/18 � (11 � 1 + 5 � 2 + 1 � 3 + 1 � 12) = 36/18 or 2.00, which is an improvement over linear probing for this example. However, for the reasons stated previously, the theoretical average number of probes in a successful search will be the same as for linear probing.

How can we eliminate or break up large clusters to improve these results? It can be done using open addressing with secondary clustering. Consider the following analogy. Suppose there are m stores, each selling a certain specialty item, and each with exactly one such item in stock. Assume n people decide to buy that item the same day, and they, in turn, each select one of the m stores at random. If a customer finds the item in stock at the shop of his or her choice, he or she buys it. However, if someone has already bought the item, it will be out of stock, and the person may ask the store manager for directions to another store. Let the stores be numbered from 0 to m - 1. Whenever this occurs, suppose the manager of the ith shop sends the customer to the (i + 1)th shop. Of course, the manager of the (m - 1)th store refers people to the 0th store. The result of all this feverish activity is that all customers arriving at a particular out-of-stock store are sent to the same next store. Again, at that store, they are all sent to the same next store whenever the item has been sold. It seems clear that this results in primary clusters of customers being formed. The physical locations of the stores is irrelevant to this process.

       Hash         Hash       Hash       Hash       Hash       Hash

Keys  Address     Table (a)  Table (b)  Table (c)  Table (d)  Table (e)

-----------------------------------------------------------------------

019     19     0  802   (4)  663   (2)  364   (5)  364   (4)  392   (1)

392     01     1  392   (1)  392   (1)  392   (1)  392   (1)

179     18     2  364   (7)

359     14     3             321   (2)

663     19     4             364   (3)  566   (3)

262     09     5  097   (1)  097   (1)  097   (1)  097   (1)  097   (1)

639     18     6             720   (3)  260   (3)  663   (3)

321     22     7  720   (1)  720   (1)  639   (2)  720   (1)  720   (1)

097     05     8  468   (1)  468   (1)  468   (1)  468   (1)  468   (1)

468     08     9  262   (1)  262   (1)  262   (1)  262   (1)  262   (1)

814     09    10  814   (2)  976   (1)  976   (1)  814   (2)  976   (1)

720     07    11  260   (5)  260   (2)             976   (2)  364   (2)

260     07    12  976   (3)  566  (12)  814   (3)             321   (2)

802     20    13             814   (2)             566   (3)

364     19    14  359   (1)  359   (1)  359   (1)  359   (1)  359   (1)

976     10    15  774   (1)  774   (1)  774   (1)  774   (1)  774   (1)

774     15    16  566   (3)                                   566   (3)

566     14    17                        260   (4)  639   (3)  260   (4)

              18  179   (1)  179   (1)  179   (1)  179   (1)  179   (1)

              19  019   (1)  019   (1)  019   (1)  019   (1)  019   (1)

              20  663   (2)  802   (1)  663   (2)  663   (2)  806   (1)

              21  639   (4)             802   (2)  802   (2)  814   (2)

              22  321   (1)  639   (2)  321   (1)  321   (1)  639   (2)

                   linear    displaced  secondary   quadratic  double

                   probing   linear     clustering  residue    hashing

                             probing

Average number      2.22       2.00       1.89       1.72      1.61

of probes for a

succesful search

Figure 9.4 Typical Hash Tables for the Five Open-Addressing Techniques Considered (Usage Ratio 78 Percent)

Suppose each store manager, instead of giving identical directions to each would-be purchaser, sends customers to another store depending on which store they tried originally and on the number of stores they have already been to. The effect will be to disperse the unhappy customers. Rather than all these people being sent on to the same sequence of shops, only those with the same initial shop and the same number of previous attempts at buying are given identical directions. Clusters should still appear, but they should tend to be smaller than the primary clusters. This is, in fact, what happens, and the clusters that appear after such dispersion are called secondary clusters.

Translating this remedy to the problem involves choosing p(k) so that it becomes a function of h(k), the initial hash address of key k. This implies that only keys hashing to the same address will follow identical probe sequences. Unlike linear or displaced linear probing, keys whose probe sequences intersect do not merge into the same probe sequence at intersection. They go their separate ways. Again, to ensure that p[h(k)] generates a probe sequence, it is necessary to choose it carefully. As long as p[h(k)] is not a factor of m, for each key k, a probe sequence will result. If m is a prime number, this will always be the case when p[h(k)] is between 0 and m - 1.

As an example, take

Using our h(k) = k mod m and m = 13, any key with hash address 7 (say, key 137) has the probe sequence

7, 5, 3, 1, 12, 10, 8, 6, 4, 2, 0, 11, 9

Any key hashing to 11 now (say, key 258) has the probe sequence

11, 0, 2, 4, 6, 8, 10, 12, 1, 3, 5, 7, 9

Notice that a collision at 11, for such a key, does not result in the probe sequence for keys with hash address 7 being followed from its intersection with 11. It follows the sequence 11, 0, 2, . . . instead of 11, 9, 7, 5, . . . .

The hash table for this secondary clustering remedy is hash table (c) of Figure 9.4. The average number of probes for a successful search in this table is 1/18 � [(11 � 1) + (3 � 2) + (3 � 3) + (1 � 5)] = 31/18 or 1.72. This gives an improvement over hash tables (a) and (b), as expected. It occurs because the secondary clusters tend to be shorter than the primary clusters. Notice that the "physically adjacent" clusters in the table are deceiving; they are not relevant. It is the clustering along the probe sequences that is important. In linear and displaced linear probing there is, in effect, only one probe sequence. With the secondary clustering techniques, there is a distinct probe sequence for each distinct hash address.

Open addressing with the quadratic residue technique attempts to break up primary clusters in a similar way, although it generates a probe sequence somewhat differently. The probe sequence for key k starts with h(k) and follows with

[h(k) + i2]mod m, [h(k) - i2]mod m for i = 1, 2, . . . , (m - 1)/2

Again, to ensure that m distinct indexes are generated, care must be taken in the choice of m. If m is prime and of the form (4 � j) + 3, then m distinct indexes are guaranteed. Since m = 23 in our example, j = 5 and (m - 1)/2 = 11. The probe sequence for key k is then

h(k),

[h(k) + 12] mod 23, [h(k) - 12] mod 23,

[h(k) + 22] mod 23, [h(k) - 22] mod 23, . . . ,

[h(k) + 112] mod 23, [h(k) - 112] mod 23

If k is 364, h(k) = 19, and the probe sequence is

19, 20, 18, 0, 15, 5, 10, 12, 3, 21, 17, 9, 6, 22, 16, 12, 1, 8, 7, 4, 11, 2, 13

This is really a secondary clustering technique, since keys that hash to the same address will still have identical probe sequences. The hash table for this technique appears as hash table (d) of Figure 9.4, which gives an average number of probes of 1/18 � [(10 � 1) + (4 � 2 ) + (3 � 3) + (1 � 4)] = 31/18 or 1.72 for a successful search. When a key is inserted with these secondary clustering techniques, the result may be the lengthening or connecting of two clusters. These events may occur simultaneously along more than one probe sequence when the key is inserted.

Returning to the example of the out-of-stock specialty item, we might look for a remedy that avoids even these secondary clusters, since they also tend to increase search and insertion times. It seems reasonable to try to disperse these secondary clusters by giving each would-be customer a seperate set of directions. This may be done by having store managers randomly select the next store for a customer to try. This idea is known as open addressing with double hashing. The technique uses a second hash function for p(k). To be most effective, this should be as "random" as the initial hash function and should assign a value to key k by selecting this value independently of the value assigned to key k by the initial hash function.

The theoretical analysis for this technique has not been carried out as fully as for other methods, but the results indicate that double hashing gives the best results among the open addressing techniques. Empirically, it gives results that are the same as those for "random" hashing (see Table 9.3). This is somewhat suprising, since random hashing corresponds to our store remedy. Double hashing, instead of picking a store randomly, over and over again, until a customer is satisfied, merely picks an individual displacement at random for each key. This displacement then determines the probe sequence for a given hash address.

Care must be exercised in the selection of the second hash function so that a desirable probe sequence is generated. If m is a prime, then one possibility that ensures the generation of such a probe sequence is

For instance, p(657) = 11, since 657 = 50 � 13 + 7 when m = 13. Hence the probe sequence for 657 is

7, 5, 3, 1, 12, 10, 8, 6, 4, 2, 0, 11, 9

Key 137 also has hash address 7, but p(137) = 10, since 137 = 10 � 13 + 7. Its probe sequence, however, is distinct from 657's:

7, 4, 1, 11, 8, 5, 2, 12, 9, 6, 3, 0, 10

Now, even keys with the same hash address are unlikely to have the same probe sequence. The idea is that, statistically, probe sequences should appear to be approximately independently and randomly generated for two distinct keys. The hash table produced by this technique is given as hash table (e) of Figure 9.4, and the average number of successful probes is 1/18 � [(11 � 1) + (4 � 2) + (2 � 3) + (1 � 4)] = 29/18 or 1.61

As a further illustration, suppose key 582 is to be inserted in the hash tables of Figure 9.4. Its hash address, 582 mod 23, is 7, since 582 = 25 � 23 + 7. The sequence of probed locations and the number of probes required for each table are given below.

Linear probing    7, 8, 9, 10, 11, 12             (7)

Displaced linear

  probing         7, 11, 15, 19, 0, 4, 8, 12, 16  (9)

Secondary

  clustering      7, 18, 6, 17, 5, 16             (6)

Quadratic

  residue         7, 8, 6, 11, 3                  (5)

Double hashing    7, 9, 11, 13                    (4)

It is interesting that the average number of probes required to make an insertion is the same as the average number of probes required to carry out an unsuccessful search. This fact is useful in the analysis of hashing. The results of the example are typical of the behavior of open addressing techniques. As the usage ratio of the table increases, the secondary clustering and quadratic residue methods provide considerable improvement over the primary clustering methods. The double hashing techniques improve the results even more. For lower usage ratios, the improvements are much less noticeable. These improvements in average numbers of probes do not translate directly into similar improvements in average times, which depend on implementation details.

Chaining is another method of collision resolution. The idea is to link together all records that collide at a hash address. These may be kept separately or within the hash table itself. Separate chaining treats each location of the hash table as the head of a list�the list of all keys that collide at that location. Separate storage is used to store records of the list, which consist of key values and link pointer values. When a collision occurs, an insertion is made at the front of the corresponding list. Using this technique for our example yields the hash table shown in Figure 9.5 with corresponding numbers of probes (key comparisons). The table's average number of probes for a successful search is 1/18 � [(12 � 1) + (5 � 2) + (1 � 3)] = 25/18 or 1.39.

In contrast to open addressing, separate chaining allows clusters to form only when keys hash to the same address. Hence the lenghts of the lists should be similar to the lengths of clusters formed with secondary clustering techniques. A comparison of the 1.39 probes to the 1.89 or 1.72 probes with secondary clustering is somewhat unfair, since the hash table with separate chaining really uses additional storage. The actual amount of additional storage depends on the record lenghts. When they are small, and storage that would otherwise be unused is available for the link pointer values, then the usage ratio should more properly be measured as n/(n + m). This is 18/41, or 44 percent, in the example. When the link field takes up the same amount of storage as a record, then the usage ratio should more properly be measured as n/(2n + m). This is 18/59, or 31 percent, in the example.

Figure 9.5 Hash Table for Separate Chaining

Coalesced chaining maintains lists of records whose keys hash to the same address. These lists are stored within the hash table itself. Whenever two such lists "intersect," they coalesce. This is analogous to two separated clusters becoming connected when a new key is inserted in the location that separated them. Each hash table entry is now a record containing a key value and a link value to the next record on its associated list.

A key is inserted into the table by going to its hash address and traversing its list until a null key is found, the key itself is found, or no records remain on the list. If the null key is found, the new key is inserted in the record stored at that location, and its link field is set to null. If the key itself is found, no insertion is necessary. If no records remain on the list, then an unoccupied location must be found, the new key inserted there, and its link field set to null. This new record must then be appended at the end of the traversed list.

In order to create this new record to be appended, a pointer p is kept updated. Initially, p points to location m (which is just off the table). Whenever a new record must be found, p is decremented by 1 until the location it points to is unoccupied. If p "falls off" the table (becomes less than 0), then the table is full and no insertion is possible.

Figure 9.6 illustrates coalesced chaining for the example. P is initially 23 and the first four keys are inserted directly into their hash addresses. Key 663 collides at its hash address, 19. P is then decremented by 1 and, since location 22 is empty, 663 and a null next pointer are inserted there. This new record is then appended at the end of the list beginning in 19, by setting the link field of location 19 to 22. Key 262 is inserted directly into its hash address, 09. The next entry is 639, but its hash address, 18, contains an entry. P is then decremented by 1, and since location 21 is empty, 639 and a null link value is inserted there. This record is then appended at the end of the list beginning in 18 by setting the link field of location 18 to 21.

Figure 9.6 Hash Table for Coalesced Chaining

Figure 9.6 shows the resultant hash table, the number of probes for each key insertion, and the value of p after all eighteen keys are entered. The average number of probes for a successful search of this table is 1/18 � [(9 � 1) + (6 � 2) +(1 � 3) + (1 � 4) + (1 � 5)] = 33/18 or 1.81. If extra storage is available for the link fields, then n/m is a reasonable measure of the table usage here. These 1.81 probes are not as good as the 1.61 or 1.72 probes resulting from double hashing and quadratic residues in the example. Theoretical results, however, show that coalesced chaining does yield better average results than "random" hashing when storage is available for link pointer fields. You may expect coalesced chaining to give results that are not as good as separate chaining, better than linear or displaced linear probing, not as good as "random" or double hashing, but perhaps comparable to the secondary clustering techniques. If the link fields take additional storage, it should be taken into account in the usage factors for comparison. Note that coalesced chaining generally yields more but shorter lists (or clusters) than linear or displaced probing, because connecting or coalescing of clusters occurs to a lesser extent with coalesced chaining.

A third method of collision resolution will be discussed in Chapter 10 in connection with external memory, or secondary storage, where it is especially appropriate and useful.


Data Structures Using C: 9.3.5 Searching in Ordered Hash Tables

9.3.5 Searching in Ordered Hash Tables

If keys are entered into a hash table in sorted order, open-addressing and chaining techniques will always result in searches along a probe sequence or along a list, in which the keys appear in the same or reverse order, respectively. In both cases, the search algorithm may be easily modified so that whenever a key is encountered that precedes the search key in the relevant ordering, the search is immediately terminated unsuccessfully. This can significantly decrease search times for unsuccessful searches but will not affect successful search times. You can generate the new hash tables (a) through (g) of Figures 9.4 to 9.6 to see the effect on the tables of the keys appearing in sorted order.

Of course, one cannot normally control the order in which keys to be inserted appear. It is not difficult, however, to modify the insertion algorithms so that the resultant hash table is exactly the same as the hash table resulting when keys are given in sorted order. The idea of the algorithm is clever and simple. For open addressing, simply insert a key as usual unless, in following a probe sequence, a key is encountered that precedes the new key in the ordering. Then the search key is inserted in that position, and the insertion proceeds as though the replaced key were being inserted using the displaced key instead of the new key. For separate chaining, it is necessary to traverse the list at which a collision occurs, and insert the new key in its proper position of the list.

This technique does not work directly for coalesced chaining, because the insertion of a new key in a list may start from the interior of the list when a collision occurs. In order to keep coalesced lists in order, it is necessary to find the first list record or be able to traverse the list in either direction. To achieve this, the lists might be implemented as circular lists or two-way lists. Circular lists take more traversal time but less storage than two-way lists.

Keeping ordered hash tables yields significant improvements in the average number of probes required for unsuccessful searches with open addressing. These improvements are especially striking for high usage ratios.


Data Structures Using C: 9.3.6 Deletion from Hash Tables

9.3.6 Deletion from Hash Tables

Deletion in hash tables using separate chaining is straightforward, since it amounts to the deletion of a record from a list. With open-addressing and coalesced chaining, the deletion of a key is not so straightforward, since a key of the table will be needed to reach another key when it is on that key's probe sequence or list. Deleting it would break this path. One solution to this problem is to mark a deleted key location as "deleted," so that the path is not broken. When a significant fraction of keys has been deleted, the actual usage ratio of the resultant table is low, but search times will still reflect the original higher usage ratio. New insertions, when encountering a "deleted" location, may recapture this wasted storage by occupying such "deleted" locations. However, if "deleted" locations build up, without being recaptured by insertions, search times deteriorate to a linear search for unsuccessful searches and are larger than necessary for successful searches. The only remedy is to rebuild the hash table. Although the algorithm is not given here, this may be done in place, by using only storage allocated to the table in order to carry out the creation of a new table. Hash tables can also be expanded or decreased in size, and recreated. Again, this can be done in place.


Data Structures Using C: 9.3.7 Hashing Overview

9.3.7 Hashing Overview

Let us review the advantages and disadvantages of hashing. First, note that some estimate is needed in advance for the maximum number of entries to be stored in a hash table, since its size must be declared. This is important; too high an estimate wastes space, and too low an estimate means a large usage ratio and poorer performance. Second, the hash function must be carefully chosen so that it can be computed quickly and generates relatively few collisions.

The main characteristic of hashing is its excellent average search, insertion, and deletion time. However, when deletions are frequent, care is required in deciding which collision resolution policy to use. The major disadvantages of hashing are the possibility of large worst-case times and the lack of support for traversal through the records in sorted order. Also, the only information obtained from an unsuccessful search is that the record isn't in the table. In the data structure considered next, more information may be obtained.

Choosing the appropriate hashing technique involves consideration of time and storage as well as the relative frequency with which successful searches, unsuccessful searches, and deletions will occur. However, some generalizations can be made.

Using open addressing to resolve collisions can be quite efficient for moderate usage ratios. Although the average times for successful searches depend on whether simple linear probing or more complex probing such as double hashing is used, the differences are small. These differences are more significant for unsuccessful searches, so more complex probing is desirable when they will occur frequently. Ordered hash tables can significantly reduce the search times in this case. For larger usage ratios the chaining methods have superior search times.

With open addressing, when deletions are frequent, the table can actually be storing relatively few entries, with many table positions simply marking deleted entries. Thus the table performance is governed by a usage ratio that does not reflect the actual, much smaller, number of positions used to store records. Separate chaining uses storage more efficiently in this case, and deletion is easy. Separate chaining also has better average time behavior than the other methods.

When records are small, the links required by chaining take significant extra storage compared to open addressing, whereas the opposite is true when records are large. When record size is moderate, the extra storage for links using separate chaining must be weighed against the reduced usage ratio that could be obtained if this storage were used in open addressing.


Data Structures Using C: 9.4: Binary Search Trees

9.4: Binary Search Trees

We turn now to the next data structure: binary search trees. A binary search tree is a binary tree with the property that its left subtree contains records no larger than the root record, its right subtree contains records no smaller than the root record, and its left and right subtrees are also binary search trees. This is a recursive definition.

A binary tree fails to be a binary search tree if even one node contains a record in its left subtree or its right subtree that is, respectively, larger or smaller than the record stored at the node. Figure 9.7 has three examples of binary search trees containing the same fifteen records.

Note that a binary search tree need not be a heap, and that a heap need not be a binary search tree. In general, they are different kinds of binary trees. There are many binary trees that can store the same records. Any binary tree with n nodes can store n records in such a way that the tree is a binary search tree. (Just do an inorder traversal of the binary tree and insert the records in sorted order.)

9.4.1 Searching the Search Trees

9.4.2 Growing the Search Tree "Simply"

9.4.3 The Shape of Simple Binary Search Trees

9.4.4 Deleting a Record Simply

9.4.5 A Balancing Act

9.4.6 Maintaining Balance

9.4.7 Access By Order in AVL Trees

9.4.8 Binary Search Tree Overview


Data Structures Using C: 9.4.1 Searching the Search Trees

9.4.1 Searching the Search Trees

In Chapter 7, three important traversals through the nodes of any binary tree were defined: 1) preorder, 2) inorder, and 3) postorder. In order to search a binary tree to determine if a particular key value is stored, any one of those traversals could be used to gain access to each node in turn, and the node tested to see if the search key value is there. This would be equivalent to a linear search.

Instead, the special nature of binary search trees facilitates highly efficient search. We simply compare the search key with the root key. If they match, the search is complete. If not, determine whether the search key is smaller than the key at the root. If so, then if it is in the tree at all, it must be in the left subtree. If it is larger than the root key, it must be in the right subtree if it is in the tree at all. Thus either the left or right subtree is eliminated from consideration. This procedure is repeated for the subtree not eliminated. Eventually, the procedure finds the key in the tree or comes to a null subtree and concludes that the key is not stored in the tree.

Figure 9.7 Three Binary Search Trees Containing the Same Data

Assume that nodes of the binary search tree are represented as follows.

typedef struct treenode

{

whatever info;

struct treenode *leftptr;

struct treenode *rightptr;

}binarytreenode, *binarytreepointer;

This special search may be implemented as follows:

>Comment

found = FALSE;

p = t;

pred = NULL;

while ((p != NULL) && (!found))

if (keyvalue == p->key)

found = TRUE;

else if (keyvalue < p->key)

{

pred = p;

p = p->leftptr;

}

else

{

pred = p;

p = p->rightptr:

}

Found indicates whether the search key value was present in the tree. If so, p points to the node in which the search key was found, and pred points to its predecessor. If the search key was not found, then pred points to the node that would have been its predecessor had it been in the tree. Note that this allows additional information to be obtained, if desired, beyond the fact that the search key was not found. The key less than the search key and the key greater than the search key can easily be determined, so that records closest to the desired one are known.

Notice that when (b) of Figure 9.7 is searched in this way, the search is equivalent to a linear search of an array in which the records are stored in sorted order. This search algorithm, applied to (c), would be equivalent to a binary search of an array in which the records are stored in sorted order. A search of (a) would fall somewhere between these two extremes.

What allows the search of Figure 9.7(c) to be equivalent to a binary search? It is the fact that the tree is "scrunched up" or near minimum possible depth, and not as straggly (unnecessarily deep) as the others. It has an intuitive kind of "balance," so that we eliminate from consideration about half the records each time we make a comparison. We are back to the game of "twenty questions.''


Data Structures Using C: 9.4.2 Growing the Search Tree "Simply"

9.4.2 Growing the Search Tree "Simply"

This section develops an algorithm for creating or growing a binary search tree that stores records at its nodes. Imagine that you have already processed some records and created a binary search tree for them. Given the next input record, search the current tree for its key value. Assume no duplicates will occur in the tree, although this need not create any real problem, since a node can contain a pointer to a list, stored separately, of records with identical key values. Eventually, since the new record is not in the tree, a null subtree will be reached. However, if this null subtree is replaced by a node that stores this record, the resultant tree will still be a binary search tree. For instance, inserting 45 by this procedure in Figure 9.7(c) produces Figure 9.8. This gives an algorithm for growing a binary search tree. Start with the first input record at the root. For each additional input, search the tree. The search will reach a node whose left or right successor must be followed, but the successor is null. The record replaces that null subtree.

Figure 9.8 Binary Search Tree Created from Figure 9.7

This can be accomplished by using pred from the search implementation, and then invoking the following.

>Comment

p = malloc(sizeof(binarytreenode));

p->key = keyvalue;

p->leftptr = NULL;

p->rightptr = NULL;

if (keyvalue < pred->key)

pred->leftptr = p;

else

pred->rightptr = p;

We now have algorithms for growing, searching, and inserting records into a binary search tree.


Data Structures Using C: 9.4.3 The Shape of Simple Binary Search Trees

9.4.3 The Shape of Simple Binary Search Trees

The binary search tree grown by the algorithm of the preceding section will have its shape determined by the input order of the records to be stored. If they are input in sorted order or reverse sorted order, the result is Figure 9.7(b) or its mirror image. If they are input in the order 55, 40, 83, 35, 51, 80, 85, 17, 37, 50, 52, 70, 81, 84, 100, the result is Figure 9.7(c). There are n! possible input orderings of n distinct keys. Some of these may give rise to the same binary search tree. Recall that the tree grown cannot have depth greater than n nor less than �lg(n + 1)�. Suppose the input is selected so that each of the n! possible orderings is equally likely to occur. This would be the case if the first input record is selected at random from among the n records, the second is selected at random from among the remaining n - 1 records, and so on. What will be the average depth of trees grown by the algorithm in this case? To calculate the average depth directly would mean to generate, for each of the n! orderings of input records, the binary search tree grown and note its depth. Then add up the n! depths and divide by n! to obtain the average depth.

It is difficult to analyze this problem using probability theory. However, the average depth has been shown to be O(c lg n), where c is about 4. The average search time can be calculated for a randomly grown binary search tree. It is the average time required to search for a record stored in the binary search tree, assuming each record is equally likely to be searched. This average search time is O(1.39 lg n). In the case of random input, the average search of the binary search tree grown by the algorithm will be only 39 percent more than the smallest possible average search time! Moreover, the algorithm for growing the tree will take an average time at most O(nlg n). If the records are given in sorted order instead of in random order, it will take the algorithm O(n2) time to grow the tree.

It is possible to grow "balanced" binary search trees, also called AVL trees (after the discoverers of these trees, Adelson-Velskii and Landis [1962]). The algorithm for growing AVL trees is slower and more complex, as is the insertion and deletion of records. However, they are guaranteed to have depth no greater than about 1.44 lg n. This is the worst-case depth. Consequently, AVL trees may be searched in time at most O(1.44 lg n). Such trees appear quite healthy and are never sparse. However it takes great care to grow healthy trees, as will be seen later.


Data Structures Using C: 9.4.4 Deleting a Record Simply

9.4.4 Deleting a Record "Simply"

At this point all the important operations on binary search trees have been considered except deletion. The purpose of this section is to derive an algorithm for deleting a record from a binary search tree so that the remaining tree is still a binary search tree. To delete a terminal record, simply replace it by a null tree. To delete a record with only one successor, simply insert the successor's subtree at the position occupied by the deleted node. For instance, in the tree shown in Figure 9.9(a), deleting 51 and 71 yields the tree shown in Figure 9.9(b).

Consider Figure 9.9 again. A difficulty occurs if you wish to delete 80, a record with two successors. The node 80 must be replaced by a record that will allow the tree to retain its binary search tree property. This means that the record replacing 80 must be at least as large as any record in the left subtree of 80, and no larger than any record in the right subtree of 80. There are two choices: replace 80 with the largest record in its left subtree or the smallest record in its right subtree.

Notice that the largest record in any binary search tree is always found by going to its root and following right successors until a null tree is reached. The smallest record is found by going to its root and following left successors until a null tree is reached. Thus, in the example, we may take either 77's or 81's record to replace 80's record. The record selected, say 77, must then be inserted in place of 80's record, which is then effectively deleted. Taking 77 requires that 74's record replace it. The result is Figure 9.9(c). It should be clear that searching, inserting, or deleting, using the procedures in a search tree of depth d, will take worst-case time proportional to d.

Binary search trees are treated in this chapter as abstract objects. A sequential representation for deep but sparse trees requires array lengths proportional to 2d - 1. Deletions for this representation require a great deal of shifting of records. For instance, to delete 85 requires shifting 84, 83, and 81. To save storage, and to achieve faster insertions and deletions, requires using a linked representation.

(a) Before Deletion of Records

(b) After Deletion of 51 and 71

(c) After Deletion of 80

Figure 9.9 A Binary Search Tree before and after Deletion of Records


Data Structures Using C: 9.4.5 A Balancing Act

9.4.5 A Balancing Act

The "simple" algorithms for growing binary search trees and inserting and deleting records within them are relatively simple and have excellent time and storage requirements�on the average�when randomly done. These randomly grown trees can be fine for compiler symbol tables or for small data base applications such as dictionaries, but there are situations where good average behavior is not enough. Suppose your rocket ship's computer must search a binary search tree for retro-rocket firing instructions and must complete the search within a specific interval. This is not the time to find that the tree is out of shape and that a much longer than average time will ensue.

One answer is the AVL tree, which is discussed in this section. An AVL tree is a balanced binary tree that is also a binary search tree. (Balanced binary trees were defined in Chapter 7 in the discussion of Fibonacci trees.)

AVL trees have excellent worst-case time and storage requirements and allow access to the ith record to be performed quickly. The algorithms for the growth of such trees and insertion and deletion within them are considerably more complex than for "simple" binary search trees. Balanced trees are very useful whenever all four operations�searching, inserting, deleting, and accessing the ith record�are needed and worst-case rather than average time is important.

Because we want to use AVL trees to store records, it is desirable for an AVL tree to have as small a depth as possible and as many nodes (in which to store records) as possible for that depth. The "best" AVL tree of depth d is thus complete and has 2d - 1 nodes. A balanced binary search tree (AVL tree) of depth d, which has the least number of nodes among all AVL trees of depth d, represents the "worst" AVL tree of depth d. Such a tree will be a Fibonacci tree that is also a binary search tree. These trees represent the worst-case AVL trees. A Fibonacci AVL tree of depth d will store the minimum possible number of records.

Suppose nd represents the number of nodes of such a tree. Then any AVL tree with depth d must have a number of nodes n that is at least nd. It is possible to prove that nd is related to the (d + 2)th Fibonacci number, Fd+2. In fact, nd = Fd+2 - 1. A great deal is known about the Fibonacci numbers. In particular, [image], where [image]. It follows from this that [image]. Also, from Chapter 7, n � 2d - 1. After some manipulations we may conclude that

d < 1.44 1g (n + 2) - 0.328

This means that the depth of any AVL tree with n nodes is never greater than 1.44 1g (n + 2). In other words, an AVL tree storing n records has a worst-case depth O(lg n). Since the best possible depth of a binary tree storing n records is �lg (n + 1)�, the worst depth of an AVL tree storing n records is at most 44 percent more than the best achievable. Contrast this with the "simple" binary search tree for which the average depth is at least 39 percent more than the best achievable; it is clearly very desirable to be able to create and use AVL trees.


Data Structures Using C: 9.4.6 Maintaining Balance

9.4.6 Maintaining Balance

Consider the AVL tree of Figure 9.10(a). To insert a new record into this tree, let us first do a "simple" insertion. The dashed branches indicate the sixteen possible insertion positions for a new node. You should confirm that, of these, only an insertion of a new node as a successor of 45, 53, 81, or 100 will cause the new tree to fail to remain an AVL tree. An insertion at node 45 causes the subtrees with roots 55, 40, and 70 to become unbalanced. For example, the subtree with root 40 would have a left subtree of depth 3, while its right subtree would have depth 5. These differ by more than 1. Notice that 55, 40, and 70 lie along the search path from the root to the new node inserted as a successor of 45. If 43 were inserted, then the result would be the tree of Figure 9.10(b). Thus the "simple" insertion discussed earlier will not retain the balance of a search tree.

This imbalance is remedied by rearranging the subtree with root 55 as shown in Figure 9.10(c). Notice that the subtrees with roots 40 and 70 are no longer out of balance. In fact, the tree of Figure 9.10(c) is now an AVL tree. Notice also that the subtree in Figure 9.10(c) replacing the subtree of 55 in Figure 9.10(b) has exactly the same depth as the original subtree of 55 in Figure 9.10(a). If any of the other insertions that would have produced an imbalance in Figure 9.10(a) had occurred, they could have been remedied by similar rearrangements. You should confirm this by trying them all and finding the suitable rearrangements. Remember, the rearrangements must not only rebalance but also preserve the binary search tree property.

It is possible to analyze the general situation to determine exactly when an insertion will cause an imbalance and what its remedy will be. An insertion into an AVL tree with zero or one node cannot cause an imbalance. When the tree has more than one node, the types of imbalance, and their corresponding remedies, reduce to the four indicated in Figures 9.11 and 9.12.

Each rectangle in Figures 9.11 and 9.12 represents an AVL tree. Any of these may be null. An X represents the position of the inserted new record that has caused imbalance. Three X's appear in imbalances LR and RL, actually representing three possibilities, although only one will have taken place. The LR and RL imbalances are meant to include the special case where B is actually the inserted node. This is why the X appears next to node B. Subtrees 1, 2, 3, and 4 must then be null. This fact is noted here, since it is not clear from the figures. In cases LL and RR, the subtree with root B represents the node closest to the inserted node, along the path from the inserted node to the root, where an imbalance occurs. In cases LR and RL the subtree with root C represents this node. In all remedies, the subtrees represented by the rectangles retain their relative positions. Note that all remedies preserve the binary search tree property. It is very important to see that, in each case, the depth of the remedy is exactly the same as the depth of the original subtree before the new insertion was made. Consequently, no nodes on the path from the subtree root, B in cases LL and RR, and C in cases LR and RL, imbalanced by the new insertion, will be imbalanced after the remedy is applied. It is also important to see that these remedies are applied locally; they do not involve the creation of a new AVL tree in its entirety.

Figure 9.10 An AVL Tree before and after Insertion

In Figure 9.11, when 43 was inserted, case LL occurred. The roles of B and A were taken by 55 and 50, respectively. The three rectangles, 1, 2, 3 of Figure 9.11 (a and b), represented, respectively, the subtrees shown in Figure 9.13. The X represented 43.

(a) Imbalance LL

(b) Remedy for LL

(c) Imbalance LR

(d) Remedy for LR

Figure 9.11 Two Imbalances and Their Remedies

Consider imbalance LL. The remedy looks as though A has moved up to the root and B has moved to the right. This is called a rotation to the right of the subtree with B at its root. The remedy for imbalance RR would be described as a rotation to the left of the subtree with B at its root. To rectify imbalance LR, perform a rotation to the left of the subtree with root A to obtain Figure 9.14(a). A further rotation to the right of the subtree rooted at C produces (b) of Figure 9.14. Assume that rotations are accomplished by the procedures rotateleft and

(a) Imbalance RR

(b) Remedy for RR

(c) Imbalance RL

(d) Remedy for RL

Figure 9.12 Two More Imbalances and Their Remedies

Figure 9.13 Rectangles 1-3 of Figure 9.12(a) and 9.12(b)

Figure 9.14 The Double Rotation Producing Remedy LR

rotateright, each with a parameter that points to the root of the subtree to be rotated. Then the remedies for each imbalance may be obtained by

LL:  rotateright(plast)             where last points to B

RR:  rotateleft(plast)              where last points to B

LR:  rotateleft(p(last.leftptr))    where last points to C

     rotateright(plast)

RL:  rotateright(p(last.rightptr))  where last points to C

     rotateleft(plast)

The procedure for rotateright, for example, might be as follows:

rotateright(plocalroot)

/* Performs a rotation to the right

of the subtree whose root is

pointed to by localroot.

*/

binarytreepointer *plocalroot;

{

binarytreepointer q;

q = (*plocalroot).leftptr;

(*plocalroot).leftptr = q�>rightptr;

q->rightptr = *plocalroot;

*plocalroot = q;

}

Note that the procedure also changes the pointer in localroot, so it points to the root of the new subtree. It is assumed that nodes of the AVL tree are represented as follows:

typedef struct treenode

{

whatever key;

struct treenode *leftptr;

struct treenode *rightptr;

int balance;

}binarytreenode,*binarytreepointer;

The balance field stores the node's right subtree depth minus the node's left subtree depth. In an AVL tree it will have values - 1, 0, or + 1.

Consider the binary tree shown in Figure 9.15(a), which was balanced until the new node, marked by X, was inserted. The initial balance values are also shown. Because of the insertion, the root node and node C have become unbalanced. Since node C is closest to the inserted node, we remedy this imbalance by applying the remedy for RL to obtain Figure 9.15(b). The tree itself is now balanced, but the balances of nodes A, B, and C are now incorrect, as are the balances of the three nodes between X and A. The last node, however, does have the correct balance value.

In general, after rebalancing, one of the nodes involved in the remedy will be the new root of the subtree that was rebalanced (B in the example). If the remedy was LL or RR, all nodes along the path from this node to the inserted node, starting with its successor, need their balances reset. If the remedy was LR or RL, then all nodes along this path, starting with its successor's successor (the successor of A in the example), need their balances reset. The node not on this path involved in the remedy (C in the example) must have its balance reset, and, finally, the new root of the rebalanced subtree must have its balance set to 0. The special case when the inserted node is at B requires only that C's balance be set to 0. All nodes above this new root need not be reset; they retain their original values. Any time the root node is involved, special processing is required, since the head of the tree must be modified rather than a left or right subtree pointer of a node.

The algorithm for inserting a node into an AVL tree can now be written.

1. Search for the keyvalue.

2. If it is not found, then

a. Create a new node for it and set its field values.

b. Insert the new node into the tree.

c. If the tree has more than one node, then

If it is not necessary to rebalance, then reset last's balance and the balances of nodes on he search path between last and the new node.

If it is necessary to rebalance, then

i. apply the appropriate remedy to rebalance, reset the balance of the node involved in the remedy but not on the new search path, reset the proper new search path balances, set the balance of the root of the subtree at which rebalancing occurred to 0.

ii. If this root was the root of the tree itself, then

set the head of the tree to this root; otherwise

set its predecessor to this root;

else

reset the balance of the predecessor of the inserted node.

It has already been noted that, when rebalancing is necessary, once the subtree whose root is nearest the newly inserted node has been rebalanced with the correct remedy, all other subtrees along the path to the root of the entire tree will also be back in balance. Again, this is because the smallest subtree causing the imbalance has been modified by the remedy to have its original depth. All the subtrees were originally balanced when it had that depth, so they must be balanced now. During the search in step 1 for the insertion key, suppose you keep a pointer last. Last is updated to point to the node that may become imbalanced on the search path nearest the insertion node. Only nodes with balance values of + 1 or -1 may become imbalanced due to the insertion. It is necessary to rebalance only if last.balance was -1 and the search path went left at last, or last.balance was + 1 and the search path went right at last, and p is not t, and pred.balance = 0. Only nodes on the search path, between the inserted node and last, will need to have their balance fields updated.

(a) Unbalanced

(b) Balanced

Figure 9.15 Rebalancing a Binary Tree

During the search, an updated pointer to the predecessor node of last should be kept in predlast. P will point to the node whose key field is currently being compared with the search key, and predp points to its predecessor. An implementation of the AVL tree insertion is given in the insertavl function. For clarity, the procedures createnode and search are then given separately, followed by procedures insertnode and resetpathbalances.

Nonrecursive AVL Insertion

#define TRUE 1

#define FALSE 0

#define NULL 0

insertavl(keyvalue,pt)

/* Inserts a new record with key equal to

keyvalue in the AVL tree t, if a record

with this keyvalue is not present in t.

After insertion t will still be an AVL tree.

*/

whatever keyvalue;

binarytreepointer *pt;

{

binarytreepointer p,predp,last,predlast,q;

int found;

    >Comment

search(keyvalue,*pt,&p,&predp,

&last,&predlast,&found);

    >Comment

if(!found)

{

    >Comment

createnode(keyvalue,&p);

    >Comment

insertnode(keyvalue,p,predp,pt);

    >Comment

if(p != *pt)

if(predp->balance == 0)

if(keyvalue < last->key)

if(last->balance == 0)

    >Comment

{

last->balance = -1;

resetpathbalances(keyvalue,

last->leftptr,p);

}

else if(last->balance == +1)

    >Comment

{

last->balance = 0;

resetpathbalances(keyvalue,

last�>leftptr,p);

}

else

    >Comment

{

    >Comment

q = last->leftptr;

if(keyvalue < q->key)

ll(keyvalue,&last,p);

else

lr(keyvalue,&last,&q,p);

last->balance = 0;

    >Comment

if(predlast == NULL)

*pt = last;

else if(keyvalue < predlast->key)

predlast->leftptr = last;

else

predlast->rightptr = last;

}

else

if(last->balance == 0)

    >Comment

{

last->balance = +1;

resetpathbalances(keyvalue,

last->rightptr,p);

}

else if(last->balance == -1)

    >Comment

{

last->balance = 0;

resetpathbalances(keyvalue,

last->rightptr,p);

}

else

    >Comment

{

    >Comment

q = last->rightptr;

if(keyvalue < q->key)

rr(keyvalue,&last,p);

else

rl(keyvalue,&last,&q,p);

last->balance = 0;

    >Comment

if(predlast == NULL)

*pt = last;

else if(keyvalue < predlast->key)

predlast->leftptr = last;

else

predlast->rightptr = last;

}

else

    >Comment

predp->balance = 0;

}

}

ll(keyvalue,plast,p)

/* Applies remedy ll, resets the balance of the node

involved in the remedy but not on the new search

path, and resets the proper new search path balances.

*/

whatever keyvalue;

binarytreepointer *plast,p;

{

rotateright(plast);

((*plast)->rightptr)->balance = 0;

resetpathbalances(keyvalue,(*plast)->leftptr,p);

}

lr(keyvalue,plast,pq,p)

/* Applies remedy lr, resets the balance of the node

involved in the remedy but not on the new search

path, and resets the proper new search path balances.

*/

whatever keyvalue;

binarytreepointer *plast,*pq,p;

{

rotateleft(&((*plast)->leftptr));

rotateright(plast);

if(keyvalue < (*plast)->key)

{

(*pq).balance = 0;

((*plast)->rightptr)->balance = +1;

resetpathbalances(keyvalue,

((*plast)->leftptr)->rightptr,p);

}

else if(keyvalue > (*plast)->key)

{

(*pq)->balance = -1;

((*plast)->rightptr)->balance = 0;

resetpathbalances(keyvalue,

((*plast)->rightptr)->leftptr,p);

}

else

((*plast)->rightptr)->balance = 0;

}

rr(keyvalue,plast,p)

/* Applies remedy rr, resets the balance

of the node involved in the remedy but

not on the new search path, and resets

the proper new search path balances.

*/

whatever keyvalue;

binarytreepointer *plast,p;

{

rotateleft(plast);

((*plast)->leftptr)->balance = 0;

resetpathbalances(keyvalue,(*plast)->rightptr,p);

}

rl(keyvalue,plast,pq,p)

/* Applies remedy rl, resets the balance

of the node involved in the remedy but

not on the new search path, and resets

the proper new search path balances.

*/

whatever keyvalue;

binarytreepointer *plast,*pq,p;

{

rotateright(&((*plast)->rightptr));

rotateleft(plast);

if(keyvalue > (*plast)->key)

{

(*pq)->balance = 0;

((*plast)->leftptr)->balance = -1;

resetpathbalances(keyvalue,((*plast)->rightptr)->leftptr,p);

}

else if(keyvalue < (*plast)->key)

{

(*pq)->balance = +1;

((*plast)->leftptr)->balance = 0;

resetpathbalances(keyvalue,((*plast)->leftptr)->rightptr,p);

}

else

((*plast)->leftptr)->balance = 0;

}

rotateright(plocalroot)

/* Performs a rotation to the right

of the subtree whose root is

pointed to by localroot.

*/

binarytreepointer *plocalroot;

{

binarytreepointer q;

q = (*plocalroot)->leftptr;

(*plocalroot)->leftptr = q->rightptr;

q->rightptr = *plocalroot;

*plocalroot = q;

}

rotateleft(plocalroot)

/* Performs a rotation to the left

of the subtree whose root is

pointed to by localroot.

*/

binarytreepointer *plocalroot;

{

binarytreepointer q;

q = (*plocalroot)->rightptr;

(*plocalroot)->rightptr = q->leftptr;

q->leftptr = *plocalroot;

*plocalroot = q;

}

resetpathbalances(keyvalue,start,p)

/* Resets the balances of all nodes on

search path between the nodes

pointed to by start and p.

*/

whatever keyvalue;

binarytreepointer start,p;

{

binarytreepointer q;

q = start;

while(q !=p)

if(keyvalue < q->key)

{

q-balance = -1;

q = q->leftptr;

}

else

{

q->balance = +1;

q = q->rightptr;

}

}

createnode(keyvalue,pp)

/* Returns with p pointing to a new node

record with key equal to keyvalue,

left and right pointer fields null, and

balance field zero.

*/

whatever keyvalue;

binarytreepointer *pp;

{

*pp = malloc(sizeof(binarytreenode));

(*pp)->key = keyvalue;

(*pp)->leftptr = NULL;

(*pp)->rightptr = NULL;

(*pp)->balance = 0;

}

search(keyvalue,t,pp,ppredp,plast,ppredlast,pfound)

/* Returns with found true only if the binary search

tree t has a node with its key equal to keyvalue.

If found is true

p and predp point respectively, to

the found node and its predecessor,

else

predp, and last, will point, respectively,

to the node that will be the new node's

predecessor, and the node along the search

path closest to the new node's insertion point

that may become unbalanced.

*/

whatever keyvalue;

binarytreepointer t,*pp,*ppredp,*plast,*ppredlast;

int *pfound;

{

*pfound = FALSE;

*pp = t;

*plast = t;

*ppredp = NULL;

*ppredlast = NULL;

while((*pp != NULL) && (!*pfound))

if(keyvalue < (*pp)->key)

{

if((*pp)->balance != 0)

{

*ppredlast = *ppredp;

*plast = *pp;

}

*ppredp = *pp;

*pp = (*pp)->leftptr;

}

else if(keyvalue > (*pp)->key)

{

if((*pp)->balance != 0)

{

*ppredlast = *ppredp;

*plast = *pp;

}

*ppredp = *pp;

*pp = (*pp)->rightptr;

}

else

*pfound = TRUE;

}

insertnode(keyvalue,p,predp,pt)

/* Inserts the new node, pointed to

by p, as the proper successor

of predp.

*/

whatever keyvalue;

binarytreepointer p,predp,*pt;

{

if(predp == NULL)

*pt = p;

else

if(predp == *pt)

if(keyvalue < (*pt)->key)

(*pt)->leftptr = p;

else

(*pt)->rightptr = p;

else

if(keyvalue < predp->key)

predp->leftptr = p;

else

predp->rightptr = p;

}

As another illustration of recursion, a recursive version is presented next. It is based on the following formulation.

To insert a node in t:

If t is null then

create the new node, set its field values and set t to it

else

if keyvalue < t.key, then

insert the node in t.leftptr

if t is balanced, then

reset its balance

else

rebalance t and reset the balances of the nodes involved in the

remedy

else if keyvalue > t.key then

insert the node in t.rightptr

if t is balanced, then

reset its balance

else

rebalance t and reset the balances of the nodes involved in the

remedy.

The recursive insertavl procedure is as follows:

Recursive AVL Insertion

#define TRUE 1

#define FALSE 0

#define NULL 0

insertavl(keyvalue,pt,pincrease)

/* Inserts a new record with key value to

keyvalue in the AVL tree t, if a record

with this keyvalue is not present in t.

After insertion t will still be an AVL tree.

Increase must be zero when the function is invoked.

*/

whatever keyvalue;

binarytreepointer *pt;

int *pincrease;

{

binarytreepointer q;

    >Comment

if(*pt == NULL)

{

    >Comment

createnode(keyvalue,pt);

    >Comment

*pincrease = 1;

}

    >Comment

else if(keyvalue < (*pt)->key)

{

    >Comment

insertavl(keyvalue,

&((*pt)->leftptr),pincrease);

    >Comment

if(*pincrease) == 1)

if((*pt)->balance == 0)

    >Comment

(*pt)->balance = -1;

else if((*pt)->balance == +1)

{

    >Comment

(*pt)->balance = 0;

    >Comment

*pincrease = 0;

}

else

{

    >Comment

q = (*pt)->leftptr;

if(keyvalue < q->key)

ll(pt);

else

lr (pt,q);

    >Comment

(*pt)->balance = 0;

*pincrease = 0;

1 t's depth did not increase

}

}

    >Comment

else if(keyvalue > (*pt)->key)

{

    >Comment

insertavl(keyvalue,

&((*pt)->rightptr),pincrease);

    >Comment

if(*pincrease == 1)

if((*pt)->balance == 0)

    >Comment

(*pt)->balance = +1;

else if((*pt)->balance == -1)

{

    >Comment

(*pt)->balance = 0;

    >Comment

*pincrease = 0;

}

else

{

    >Comment

q = (*pt)->rightptr;

if(keyvalue > q->key)

rr (pt)

else

rl(pt,q);

    >Comment

(*pt)->balance = 0;

    >Comment

*pincrease = 0;

}

}

}

ll(pt)

/* Apply remedy ll and reset the

balance of the node involved in

the remedy.

*/

binarytreepointer *pt;

{

rotateright(pt);

reset1balance((*pt)->rightptr);

}

lr(pt,q)

/* Apply remedy lr and reset the balances

of the two nodes involved in

the remedy.

*/

binarytreepointer *pt,q;

{

rotateleft(&((*pt)->leftptr));

rotateright(pt);

reset2balances(q,*pt,(*pt)->rightptr);

}

rr(pt)

/* Apply remedy rr and reset the

balance of the node involved in

the remedy.

*/

binarytreepointer *pt;

{

rotateleft(pt);

reset1balance((*pt)->leftptr);

}

rl(pt,q)

/* Apply remedy rl and reset the balances

of the two nodes involved in the remedy.

*/

binarytreepointer *pt,q;

{

rotateright(&((*pt)->rightptr));

rotateleft(pt);

reset2balances(q,*pt,(*pt)->leftptr);

}

rotateright(plocalroot)

/* Performs a rotation to the right

of the subtree whose root is

pointed to by localroot.

*/

binarytreepointer *plocalroot;

{

binarytreepointer q;

q = (*plocalroot)->leftptr;

(*plocalroot)->leftptr = q->rightptr;

q->rightptr = *plocalroot;

*plocalroot = q;

}

rotateleft(plocalroot)

/* Performs a rotation to the left

of the subtree whose root is

pointed to by localroot.

*/

binarytreepointer *plocalroot;

{

binarytreepointer q;

q = (*plocalroot)->rightptr;

(*plocalroot)->rightptr = q->leftptr;

q->leftptr = *plocalroot;

*plocalroot = q;

}

createnode(keyvalue,pp)

/* Returns with p pointing to a new node

record with key equal to keyvalue,

left and right pointer fields null, and balance field zero.

*/

whatever keyvalue;

binarytreepointer *pp;

{

*pp = malloc(sizeof(binarytreenode));

(*pp)->key = keyvalue;

(*pp)->leftptr = NULL;

(*pp)->rightptr = NULL;

(*pp)->balance = 0;

}

In order to determine the balance of t upon return from the recursive calls (these are underlined in the algorithm), we will need to know if the subtree in which the node was inserted has increased in depth. A flag increase, with values 1 or 0, indicates an increase. Increase may be nonlocal to, or a parameter of, the function, but it must be initialized to 0 before the function is invoked. We make it a parameter.

The functions for resetting balances are

reset1balance(q)

/* Sets the balance of node

q to zero.

*/

binarytreepointer q;

{

q->balance = 0;

}

reset2balances(q,t,p)

/* Resets the balance of nodes

p and q determined by the

balance of t.

*/

binarytreepointer q,t,p;

{

if(t->balance == -1)

p->balance = +1;

else

p->balance = 0;

if(t->balance == +1)

q->balance = -1;

else

q->balance = 0;

}

Suppose you wanted to create an AVL tree by reading a sequence of key values and to test the AVL insertion. The code below, when insertavl and the functions it invokes are included, creates the AVL tree using the recursive version of insertavl. It prints the key field values of the resultant tree nodes in preorder access order as the tree is created, and prints both the key and balance values in preorder access order for the final tree. The preorder traversal routine calls a process routine, as in Chapter 7, that prints out the key and balance field values of a node it is called to work on.

#include <stdio.h>

#define NULL 0

typedef int whatever;

typedef struct treenode

{

whatever key;

struct treenode *leftptr;

struct treenode *rightptr;

int balance;

}binarytreenode,*binarytreepointer;

binarytreepointer left(p)

/* Returns a copy of the left pointer

of the node pointed to by p.

*/

binarytreepointer p;

{

return(p->leftptr);

}

binarytreepointer right(p)

/* Returns a copy of the right pointer

of the node pointed to by p.

*/

binarytreepointer p;

{

return(p->rightptr);

}

info(l)

/* Returns the key field value of

the node pointed to by l.

*/

binarytreepointer l;

{

return(l->key);

}

bal(l)

/* Returns the balance field value of

the node pointed to by l.

*/

binarytreepointer l;

{

return(l->balance);

}

binarytreepointer setnull()

/* Returns a null pointer. */

{

return(NULL);

}

The function insertavl and all its invoked functions should go here, since they depend on the tree's implementation.

#define SENTINEL -1

main()

/* Reads record information, creates an AVL tree

storing the records, prints the key field values

of the tree records in preorder access order

after each record is inserted in the tree, and

prints the key & balance field values of the

final tree in preorder access order.

*/

{

binarytreepointer t,setnull();

int keyvalue,increase;

    >Comment

t = setnull();

printf(\" enter keyvalue or sentinel \n");

    >Comment

scanf("%d",&keyvalue);

while(keyvalue != SENTINEL)

}

increase = 0;

    >Comment

insertavl(keyvalue,&t,&increase);

printf("\n enter keyvalue or sentinel \n");

    >Comment

scanf("%d",&keyvalue);

    >Comment

printtree(t);

}

    >Comment

preorder (t);

}

printtree(t)

/* Prints the keys of t in

preorder access order.

*/

binarytreepointer t,left(),right();

{

printf("\n %d \n",info(l));

printtree(left(t));

printtree(right(t));

}

preorder(t)

/* Prints the keys & balances of

t in preorder access order.

*/

binarytreepointer t,left(),right(),setnull();

{

if(t == setnull())

printf("\n The tree is null \n");

else

{

process(t);

preorder(left(t));

preorder(right(t));

}

}

process(t)

/* Prints the key and balance fields of

the node pointed to by t.

*/

binarytreepointer t;

{

printf("\n key:%d balance:%d \n",info(t),bal(t));

}

Notice that all components of the program except insertavl and its associated functions are written treating the binary tree as a data abstraction.

Deletion procedures for AVL trees are not developed in detail here, but the basic ideas will be illustrated. Consider the AVL tree of Figure 9.16(a). The tree is a Fibonacci tree of depth 5. Suppose the rightmost node of its right subtree, 12, is deleted, as in a simple binary search tree. The result is Figure 9.16(b).

Figure 9.16  A Fibonacci Tree of Depth 5 and Deletion

Figure 9.16

Node 11, the predecessor of 12, is now unbalanced. Rebalancing can be achieved by applying remedy LR at node 11 to obtain Figure 9.16(c). Node 8, the predecessor of 11 before rebalancing, is now unbalanced. Rebalancing can be done by applying remedy LL at node 8 to obtain Figure 9.16(d).

This tree is finally an AVL tree. Deleting the rightmost node from any such Fibonacci tree will always lead to a rebalancing at every node along the search path.

The basic algorithm for deletion of a node from an AVL tree is first to delete, as in a simple binary tree. Then the search path must be retraced, from the deleted node back to the root. If an imbalance occurs, it must be remedied. Deletion can require O(1g n) rebalances, while insertion can require at most one rebalance.

A recursive deletion algorithm is sketched below.

To delete a node from t:

If t is not null, then

if keyvalue<t.key, then

delete the node from t.leftptr

if t is balanced, then

reset its balance

else

rebalance t and reset balances

else if keyvalue>t.key, then

delete the node from t.rightptr

if t is balanced, then

reset its balance

else

rebalance t and reset balances

else

delete the node with the largest key value in t.leftptr

replace the root of t by that deleted node

if t is balanced, then

reset its balance

else

rebalance t and reset balances.


Data Structures Using C: 9.4.7 Access By Order in AVL Trees

9.4.7 Access By Order in AVL Trees

Given a key value to find, it is easy to search the AVL tree for it. However, consider the records stored at the nodes as ordered by their processing positions in an inorder traversal of the AVL tree. For instance, in the AVL tree of Figure 9.10, the ordering is

1.  17   9.  60

2.  35  10.  70

3.  37  11.  80

4.  40  12.  81

5.  45  13.  83

6.  50  14.  85

7.  53  15.  100

8.  55

By access by order to a node is meant that, given an index (say, 10) produce the tenth record (key value 70 above). The difficulty is that it is not immediately apparent where node 10 is nor what its key value is. One way to access this node is simply to inorder traverse the tree, counting nodes. Arriving at the tenth node, produce it. This will take time O(i) if the index is i. In the worst case this is O(n), if n nodes are stored in the tree. Since AVL trees with n nodes have maximum depth 1.44 lg n, any key search, insertion, or deletion can be done in at most O(1g n) time. There is also a simple way to access by order, or index, in at most O(1g n) time. To accomplish this, add an index field to each node. It will contain an integer that is 1 plus the number of nodes in its left subtree. The index field thus contains the nodes' index. For example, the index field value of 70 would be 10. It is then possible to search the tree for an index value, much as it is searched for a key value. The index search thus takes time at most O(1g n). Of course, the insertion and deletion algorithms must be modified to keep the index fields updated correctly.

Creating an AVL tree is done by growing it as was done for the simple binary search tree: start with a null tree and insert each input record, one at a time. The difference is that a "simple" insertion is used to grow the simple binary search tree, whereas the insertavl procedure is used to grow an AVL tree. The AVL tree may be grown in O(n lg n) time. Also, since it may then be inorder traversed in O(n) time, we have another way to sort in O(n lg n) time.


Data Structures Using C: 9.4.8 Binary Search Tree Overview

9.4.8 Binary Search Tree Overview

Easily constructed binary search trees allow efficient insertion and deletion and can also be searched in average time O(1g n). More complex construction, insertion, and deletion algorithms for balanced binary search trees guarantee search, insertion, and deletion in worst-case time O(1g n). Binary search trees allow easy access to records in sorted order, and balanced binary search trees also allow O(lg i) access to the ith record in sorted order. It is not necessary to estimate in advance how many entries will be made in a binary search tree, as it is for hash tables. Hash tables can be searched very quickly, on the average, when properly constructed, but they do not allow records to be easily enumerated in sorted order.


Data Structures Using C: 9.5: A Brief Review

9.5: A Brief Review

This chapter and the last have provided examples of the effective use of appropriate data structures with their appropriate implementations. These include the use of a heap for priority queues, the use of a binary tree for a heap, the use of a heap for sorting, and the use of hash tables and binary search trees for searching. Previously we found that trees find numerous applications, ranging from storing data to aiding in the conceptualization of algorithms. Binary trees are not only useful themselves but can serve as the representation for general trees. You have seen that stacks and queues underlie the important tree traversal operations. The virtues and shortcomings of arrays and lists have also been explored.

With respect to problem solving methodology, the top-down approach, with recursion as a special case, has been consistently adhered to. A number of basic strategies have also been illustrated repeatedly. These included searching through the collection of all possible solutions, constructing a solution, and adapting a previously written algorithm to a new but related problem. The rest of this book deals with the same concepts but applies them to more advanced problems.


Data Structures Using C: Exercises

Exercises

1. Justify all the entries of Table 9.1 for the priority queue implementations.

2. Write two functions to insert and delete, respectively, records from a priority queue implemented as a heap.

3. a. Build a hash table for the twenty-seven names of the nametable of Chapter 6. Do not store duplicates. The table will be of size 31. Assign to each letter of this alphabet the integer corresponding to its position in the alphabet. For example A is assigned 1, E is assigned 5. Add the integers corresponding to every letter of a name and divide the sum by 31. The remainder is then the hash address of the name. For example, Jimmy Carter has the sum (10 + 9 + 13 + 13 + 25 + 3 + 1 + 18 + 20 + 5 + 18) = 135. Its hash address is 11. Use linear probing when a collision occurs.

b. How many probes were needed to insert each name?

4. Criticize the choice of hash function in Exercise 3.

5. Search the hash table of Exercise 3 for John Smith, and write down the probe addresses needed.

6. Show the state of hash tables (a) through (g) of Section 9.3 when key 019 is deleled.

7. Suppose that the eighteen keys are entered into the hash table of Section 9.3 in order, from largest to smallest. As you search for a key, you conclude that it is not in the table whenever you encounter a key value smaller than the search key as you trace out the linear probe path. Convince yourself that this is true, and that it will result in fewer probes for an unsuccessful search.

8. Write a function to build a hash table for each of the open-addressing and chaining techniques of Section 9.3.

9. Run each function of Exercise 8 with the input data and hash function of Exercise 3.

10. What is the maximum number of probes for an unsuccessful search of each of the hash tables (a)-(g) of Figures 9.4, 9.5, and 9.6?

11. Can you find a way to build a hash table as in Exercise 7 so that the sorted search can be done even if the input was not actually in sorted order?

12. How might the average number of probes be studied for a given hash function as in Exercise 3?

13. For m = 107, generate n random key values between 000 and 999 and build (by executing a program) hash tables (a) to (g) of Figures 9.4, 9.5, and 9.6 for these keys. Do this twenty-five times for each n and output the average number of probes to insert the n keys over the twenty-five samples. Do this repeatedly for a series of n values corresponding to usage ratios of approximately 0.50, 0.55, 0.60, . . ., 0.90. 0.95. Compare results of this simulation to your expectations.

14. Take the nametable names of Chapter 6 and grow a simple binary search tree (as in Section 9.4) when the names are input in the order in which they appear in that figure. The order is determined alphabetically by last name, first name, and then middle initial. Thus Robert Evans precedes Harry Truman. If a duplicate name occurs, do not enter it again in the tree.

15. Delete Paulina Koch from the binary search tree of Exercise 14, using the simple deletion algorithm of Section 9.4.

16. Discuss the advantages and disadvantages of storing the nametable entries of Chapter 6 in a sorted array, sorted chain, binary search tree, AVL tree, and hash table.

17. Write a function, search, with parameters keyvalue, p, and t, a pointer to the root node of a binary search tree. Search is to return with p pointing to the node of t in which keyvalue equals p�key. If no such node exists, p should return with its value null.

18. Write a function, insertbst, with parameters keyvalue and t. It is to insert a node with keyvalue into the simple binary search tree t if no node exists with that key.

19. Write a function, de1etebst, with parameters keyvalue and t. It is to delete a node whose key equals keyvalue in the simple binary search tree t, if such a node exists.

20. Should a linked or sequential implementation be used for binary search trees grown "simply" when they will contain fifty integers picked at random from the first 1,000 integers? Why?

21. What is the minimum depth of a binary search tree storing 137 entries, and how may it be constructed in time O(n) for n entries?

22. Should a linked or sequential implementation be used for a binary search tree grown as an AVL tree when it will contain 50 integers picked at random from the first 1,000 integers? Why? (Cf. Exercise 20.)

23. Follow the same directions as in Exercise 14, but grow a binary search tree that is an AVL tree. It should be the same AVL tree that would result if it were grown by invoking insertavl of Section 9.4.6 for each input name.

24. Create a binary search tree, with the names of Exercise 23 in it, which has minimum possible depth.

25. Delete the rightmost node of the AVL tree grown in Exercise 23, using the recursive deletion algorithm of Section 9.4.6.

26. Modify the nonrecursive function insertavl so that indexing can be done as described in Section 9.4.6.

27. Modify the recursive function insertavl so that indexing can be done as described in Section 9.4.6.

28. Why does growing and then inorder traversing an AVL tree correspond to an O(n lg n) sort?

29. Write a function search, with parameters indexvalue, p, and t. It is to return with p pointing to the node, if it exists, in the AVL tree t, whose index field is equal to indexvalue. P should be null if there is no such node.

30. How might we estimate the relation between the average depth of a simple binary search tree grown randomly and n, the number of inputs?

31. Suppose AVL trees are grown randomly. That is, each of the n ! arrangements of n distinct inputs is equally likely to occur. Starting with the null tree, insertavl is invoked for each input. How might the average depth of such AVL trees be estimated as a function of n?


Data Structures Using C: Suggested Assignments

Suggested Assignments

1. This is the same as the suggested assignment 2 of Chapter 2, except the integers for part (a) and the records for parts (b) and (c) are to be stored in a binary search tree.

2. Design the nametable for the case study of Chapter 6. Discuss the relative merits of arrays, lists, binary search trees, AVL trees, and hash tables for its implementation.

3. Write an efficient check (t,d,flag) to return in d the depth of the binary tree pointed to by t, and to return in flag true if t points to a balanced binary tree and false otherwise.


Data Structures Using C: CHAPTER 10: FILES

CHAPTER 10: FILES

Introduces the file data structure

treats communication with the outside world

input, output, and secondary storage

Discusses sorting of files using

straight merge

natural merge

replacement selection

polyphase sort

Considers file organization to support sequential and random access

Explains the physical basis for the file organizations using

direct access

indexed sequential access

B-trees

Illustrates how the basic operations on B-trees are done and what their time requirements are

creating

inserting

deleting

searching

traversing

10.1: The File Data Structure

10.2: Internal and External Memory

10.3: Organization of Files

10.4: Sequential Access in External Memory

10.5: Sorting Tape Files

10.6: Direct Access in External Memory

10.7: Sequential and Random Access of Disk Files

10.8: Summary

Exercises

Suggested Assignment


Data Structures Using C: 10.1: The File Data Structure

10.1: The File Data Structure

A file is a collection of records. This is a loose definition of a file. Almost all of the data structures discussed in the preceding chapters satisfy it. The term file, however, is usually reserved for large collections of information stored on devices outside the computer's internal memory. It usually implies that the records are stored in secondary storage in the computer's external memory, on tapes or disks. As a result, the ways in which the file must be organized so that operations on it can be carried out efficiently are dependent on the characteristics of the secondary storage devices used to implement the file. The basic operations on a file are to insert and delete records, process or update records, and search for or retrieve records. These operations are the same as the basic operations on arrays, lists, trees, list-structures, and more complex lists, which are stored in the computer's internal memory. In designing algorithms that use files stored in external memory, programmers must weigh the trade-offs between the time and the storage required to support these operations. They must do this just as they would for data structures stored in internal memory. Time spent in organizing information results in faster operations, but this efficiency must be weighed against the increased care required to nurture and maintain the better-organized data structure. For example, balanced trees are faster to use in the worst case but harder to grow. Similar effects pertain to file manipulations.


Data Structures Using C: 10.2: Internal and External Memory

10.2: Internal and External Memory

Computer memories are normally thought of as falling into two basic categories, internal and external. Which memory is used to store data depends on the trade-off of efficiency of access to data versus the cost of that efficiency.

The data structures and algorithms considered thus far are all appropriate to the internal memory of a computer. This memory is also referred to as main memory or random access memory. A program can process only information stored in its internal memory. If the program is to process information stored in external memory, that information must first be read into its internal memory. Except for the instructions involved in this transfer of information, all program instructions refer to variables that name internal memory locations.

By means of random access, it is possible to access any individual internal memory element in exactly the same length of time�typically one-millionth of a second or less. This access time is independent of the past history of accessed elements. However, random access memory is expensive and, although available in increasingly larger sizes, is still relatively small in size compared to the storage needs of many applications.

External memory, also referred to as secondary storage, is so called because it is physically outside the computer itself. It is relatively inexpensive and is available in very large amounts, but access to an arbitrary record is slow compared to internal memory. Magnetic tapes, floppy disks, and magnetic disks (also known as hard disks) are the usual devices for external storage of data. The characteristics of these devices that are essential for efficient use of data structures are discussed in this chapter. Since the characteristics of secondary storage devices are different from those of internal memory, the data structures and algorithms required to process the data they store are distinct from those for internal memory. However, many of the same concepts and techniques of good design and implementation of algorithms are still applicable and form the basis for representing and operating on information stored in external memory.

10.2.1 Operating Systems

10.2.2 Filters


Data Structures Using C: 10.2.1 Operating Systems

10.2.1 Operating Systems

Programming languages such as C do not ordinarily allow direct manipulation of external storage elements. Instead, as with internal storage, they provide commands that allow programmers to deal with external memory conceptually. C provides very limited kinds of operations to be performed on its conceptual external memory, whereas COBOL provides very extensive operations. FORTRAN falls between these extremes. Whatever high-level language is used, the operating system of a computer installation provides commands to make files available for processing by a program. Operating systems allow permanent storage of files generated by a program during its execution. The stored files can then be retrieved for later use.

You may have used the DOS, MVS, VM, or NOS operating system on a mainframe computer; RT-11, RSTS, RSX, TSX, VAX/VMS, or UNIX on a mini-computer; or MS-DOS, PC-DOS, UNIX, or CP/M on a micro - or personal computer. In effect, operating systems provide repositories of files that may be added to, or referenced by, programs. Such files are used to store data or information to be processed. They are also used to store algorithms (such as programs to be executed), data base management systems to organize and maintain large data bases, compilers to be invoked, or even other operating systems to be used when needed. Thus operating systems support the development of libraries of data and algorithms that programmers use, add to, or delete from. They support the general programming strategy recommended and demonstrated in this book; in solving a problem, the programmer begins by attempting to apply an already known algorithm, either directly or after appropriate modification.


Data Structures Using C: 10.2.2 Filters

10.2.2 Filters

Frequently, a solution to a problem is readily found by starting with one or more files of data and repeatedly applying available programs to these files. These programs, in turn, produce output data in the form of files, which then become the input files for other programs. The basic idea behind this kind of solution is a filter�a program whose input is one file and whose output is another file. A solution to a problem may often be found by starting with an input file and applying a sequence of filters. Each filter takes as its input the output file of the preceding filter. System commands allow this composition of filters, so that complex programs can be generated with few commands.

Example 10.1

Suppose an input file contained the roll scores for the bowling problem of Chapter 1. The problem is to structure the solution so that processing this file produces an output file of statistics. n

Suppose gamescores is a filter that takes the input file and produces an output file of game scores. Once the game scores are available, a statistical package in the computer's operating system library might be used to process the game scores as an input file (as data), producing appropriate statistics. Applying this program, or filter, to the game scores file would result in an output file containing the required information. This file could be stored for future use or processed by a printing routine, producing printed output directly.

The two filters, gamescores and statistics, applied sequentially, produce the desired output file. Note that this solution is written so that the calculation of the game scores is done first, independently of the calculations producing the statistics. In the solution of Chapter 1 these tasks were intertwined.

The idea behind filters is to allow the construction of complex programs by sequencing simpler but powerful programs. The next example illustrates this more forcefully.

Example 10.2

Use filters to produce a simplified index for a book. The index should consist of the important words of the book in alphabetical order and the pages on which they appear. n

One way to achieve a solution is to apply the following filters in sequence.

1. Word-count references

2. Truncate

3. Sort

The first takes a file consisting of the book text (organized as a sequence of pages) and produces an output file of records. Each record contains a word or sequence of words, the number of pages on which it appears, and the pages on which it appears. Truncate takes a file of records as its input and outputs a file that is identical to its input file, except that all records with a designated key field value greater than n are deleted. The key field contains the number of pages on which the word or sequence appears. A reasonable value for n might be 10. This should eliminate frequently occurring but not relevant words such as "the," "a," "and," and so on. Sort takes an input file of records and creates an output file in which the records appear in sorted order by designated key field value�in this case, the word field. The sequencing of these three relatively simple filters provides a quick and effective solution.

Using these system capabilities to store and access files of data or programs provides a framework in which to approach the entire task of problem solving. In effect, a computer installation provides a programming environment that determines the tools available for program construction. Such tools include text editors, data base management systems, file management utilities, ways to create a new program from existing programs already stored as files, and debugging aids. The enhancement of such programming environments is currently one of the major research and development areas of computer science. The command languages, also known as job control languages, provided by operating systems are used to operate on general files.


Data Structures Using C: 10.3: Organization of Files

10.3: Organization of Files

During the execution of high-level language programs, operations typically deal only with files treated as data. Operations include traversing and processing all the records of a file, randomly accessing and processing individual records selected in random order, or some combination of these two modes. The basic function of a file system, in addition to providing storage facilities, is to allow files to be searched efficiently and conveniently, so that records of an entire file may be sequentially retrieved or a portion of the file's records (perhaps only one record) may be randomly retrieved. In Example 10.1, after the bowling game scores file has been obtained, one may want to retrieve the scores of a particular game or individual.

This chapter briefly discusses the file processing facilities provided by C. It shows a number of ways in which more flexible and powerful file processing may be implemented. The literature on files would itself require a large file for its storage, and file storage and manipulation is currently an active area of research.This book considers a very small portion of that activity. The discussion deals only with the case of fixed-length records that are to be retrieved on the basis of a single field value, the primary key field. A primary key uniquely identifies a record. It is the field normally used as the search key, and the key by which records are sorted.

Just as with data structures that are stored in internal memory, the programmer must take great care to keep related records of files stored in external memory physically contiguous or linked by pointers. In internal memory, the time requirement of an algorithm is measured by the number of operations performed. These might be arithmetic operations or assignments, comparisons between two records (keys), or comparisons between two variables. When dealing with external memory, the time requirements are measured by the number of accesses to secondary storage.


Data Structures Using C: 10.3.1 Sequential Files

10.3.1 Sequential Files

C has no built-in ability to perform input/output. Input/output is done by using the standard C library <stdio> or by developing a user library. The printf and scanf functions used in the programs of the book are in this standard library.The <stdio> library contains a number of other functions that also perform input/output.

C input/output is character-oriented. All reads or writes to or from keyboards, screens, tapes, or disks are character-oriented. C reads or writes in bytes.The functions that read or write numbers or strings in turn use calls to the basic character-oriented input/output functions.

There are two ways to read or write files in C. They are termed high-level and low-level input/output. In high-level input/output, reading and writing is done a character at a time. The input/output is buffered, and the programmer does not have to consider buffer sizes or other characteristics of the operating system. The buffers, however, are not available to the programmer. In low-level input/output, which is essentially UNIX-type input/output to provide a UNIX interface, the buffers are not provided; the programmer must provide buffers and pointers to control the input/output process. In the following discussion only high-level input/output will be considered.

The simplest organization for a file is sequential. A sequential file is a sequence of records. The records may or may not be kept in sorted order in the sequence. In standard C input/output all files are sequential files. A record of a file is not necessarily declared to be of type structure. Although file records are typically of type structure, a file record may also be declared to be of type integer, float, character, or any other C type. All records of a file need not be the same type.

Example 10.3

To illustrate how records composed of structures are treated, suppose you are asked to keep a master file, bookinventory, consisting of records representing all books currently in stock in a book store. Each record is to contain a title, author, publisher, price, and currentstock field. These declarations may be used:

typedef struct

{

char title[MAXTITLE];

char author[MAXAUTHOR];

char publisher[MAXPUBLISHER];

float price;

int currentstock;

}bookrecords;

bookrecords bookinventory; n

The name of a file is a variable within a C program. Unlike other variables, it represents stored information that may have been available before the execution of the program or that may be available after the execution of the program. A file named f is depicted in Figure 10.1.

Figure 10.1 File F and F �

Associated with each file is a file pointer, which always points to a record or is positioned prior to the first record or after the last record. File pointers must be declared and are of type FILE. For example,

FILE *f_ptr;

declares f_ptr as a file pointer.

A record may be read from a file or written to a file. A read or write statement refers to the record indicated by the current position of the file pointer. Each file also has a portion of main memory, called the buffer, associated with it. When the file pointer is pointing to a file record, the file buffer contains a copy of that record's contents.

To attach a file to a program, it is necessary to execute an open statement. For reading the records in the file containing the bookinventory records, the following code is used.

>Comment

f_ptr = fopen("bookinventory", "r");

if(f_ptr == NULL)

printf("Can not open the file");

F_ptr is the pointer to the file bookinventory. The r signifies that the file will be opened for reading only and that the file is already in existence.

Executing the first statement causes the file pointer f_ptr to be positioned to the first record. If any error occurs and the file can not be opened, the file pointer will returns a NULL value. For this reason, it is a wise policy always to test for the NULL pointer when opening a file, as was done above.

If the open is executed with a w (instead of an r) it opens the file for writing. If no file by the name used exists, it creates a new file and places the file pointer at the beginning of the file. If a file already exists, it places the file pointer at the beginning of the file, in effect erasing or truncating the file. If a file is opened with an a, it creates a new file or, if one already exists, positions the pointer at the end of the file for appending items. A similar set of values, r+, w+, and a+ is used to open files for both reading and writing.

To read a single record or numbers of records from a file the fread( ) function is used. The general form is

fread(  pointer to the location of the ,  size of ,  number of,  pointer to  )

        memory in which the records       record to  records to  the file to

        read are to be placed             be read    be read     be read

For example, to read a single record from the opened file:

fread(&bookinventory,sizeof(bookrecords),1,f_ptr);

Executing this statement will cause the value of the record pointed to by the file pointer to be copied into memory starting at the memory location pointed to by &bookinventory. Also, fread returns the number of records read, or zero when the end of file is reached.

To write a record to a file that is open for writing:

fwrite(&bookinventory,sizeof(bookrecords),1,f_ptr);

Executing this statement will cause the value of the record starting at the location pointed to by &bookinventory to be copied into the storage pointed to by the file pointer, and the file pointer moved to point to the storage to be used for the next record.

Reading from a file thus causes information in secondary storage (the file) to be transferred to variables whose values are stored in internal memory. Writing to a file causes the information stored in internal memory to be transferred to secondary storage.

Example 10.4

To illustrate how files composed of single variables such as integers are treated and the effect of the file operations, consider the following program. The file prime will be created initially to consist of the first 10 primes. Note that the reading and writing are done with fscanf and fprintf. They are like scanf and printf except that they work on files of individual variables. In contrast, fread and fwrite work on files of records. Also, fscanf returns EOF when the end of the file is reached or an error occurs. n

#define NULL 0

#include <stdio>

main()

/* Creates a file consisting of the first 10 prime

numbers, and a file containing these primes in

reversed order, and prints the contents of both

the files.

*/

{

/* This section creates file prime and

stores the first ten primes in it.

*/

int a[20],i,dummy;

FILE *f_ptr,*fr_ptr,*fopen();

    >Comment

f_ptr = fopen("prime","w");

    >Comment

fprintf(f_ptr,"%d",2);

fprintf(f_ptr,"%d",3);

fprintf(f_ptr,"%d",5);

fprintf(f_ptr,"%d",7);

fprintf(f_ptr,"%d",11);

fprintf(f_ptr,�%d�,13);

fprintf(f_ptr,"%d",17);

fprintf(f_ptr,"%d",19);

fprintf(f_ptr,"%d",23);

fprintf(f_ptr,"%d",29);

    >Comment

fclose(f_ptr);

/* This section reads from the prime file,

fills a with its contents, and creates

reversedprime which will contain

the contents of a in reversed order.

*/

i = 0;

    >Comment

(1) f_ptr = fopen("prime","r");

    >Comment

if(f_ptr == NULL)

{

printf("Can not open");

exit(-1);

}

    >Comment

while(fscanf(f_ptr,"%d",&dummy) != EOF)

{

    >Comment

a[i] = dummy;

i++;

}

    >Comment

i--;

    >Comment

(2) fr_ptr = fopen("reversedprime","w");

    >Comment

if(fr_ptr == NULL)

{

printf("Can not open");

exit(-1);

}

    >Comment

while(i >= 0)

{

    >Comment

fprintf(fr_ptr,"%d",a[i]);

i--;

}

    >Comment

fclose(f_ptr);

    >Comment

fclose(fr_ptr);

/* This section prints, at the terminal,

the contents of prime and then of

reversedprime.

*/

    >Comment

f_ptr = fopen("prime","r");

    >Comment

while(fscanf(f_ptr,"%d",&dummy) != EOF)

printf("\n %d\n",dummy);

    >Comment

fr_ptr = fopen("reversedprime","r");

    >Comment

while(fscanf(fr_ptr,"%d",&dummy) != EOF)

printf("\n %d\n",dummy);

    >Comment

fclose(f_ptr);

    >Comment

fclose(fr_ptr);

}

Prime and reversedprime are files of integers and a is an array of integers.

After (1) fopen is executed, the file called prime and its file pointer can be depicted as in Figure 10.2(a). After nine executions of the first while loop body, the situation will be as shown in Figure 10.2(b).

The tenth execution sets a [9] to 29 and moves the file pointer beyond the last file record. The EOF test yields the value true, so the loop is exited with i set at 10.

After (2) fopen is executed, reversedprime appears as in Figure 10.3(a). After the second while loop body has been executed nine times, reversedprime looks like Figure 10.3(b). The last execution yields the file shown in Figure 10.3(c).

The standard function, exit, closes any open files, flushes any buffered output, signals what has happened, and terminates program execution. A zero parameter value, by convention, signals that all is well, while other values may have different meanings. The -1 is used here to signal that a file could not be opened. Files can also be closed by simply invoking the function fclose, whose parameter is the file pointer of the file to be closed. It is important to close all files appropriately.

Suppose you wished to append the eleventh prime, 31, to prime after 29. This can be done directly in C by opening the file prime to its end so that items can be appended. To do this, as noted earlier, the file need only be opened with the a to position the file pointer

f_ptr = fopen("prime","a");

fprintf(f_ptr,"%d",31);

(a) After FOPEN Is Executed

(b) After Nine Executions of the While Loop

Figure 10.2 File PRIME

(a) After FOPEN Is Executed

(b) After Nine Executions of the Second While Loop

(c) After Last Execution

Figure 10.3 File REVERSEDPRIME

Example 10.5

Suppose there are two files with records of the same type in sorted order based on key value. Merge the two files. n

Producing a new file consisting of all the records of the two files in sorted order is called merging the two files. The merging of two files is a basic component of efficient methods of external sorting. The task in Example 10.5 is to write a function to produce a merged file assuming records are ordered with the smallest key value appearing in the first record.

The basic idea is straightforward. Read a record from each sorted file, and append the record with the smaller key value to the merged file. Read the next record of the file that had the record with the smaller key value. Compare it with the record that has not yet been appended. Append the record with the smaller key value to the merged file. Repeat this process of reading and comparing records from the two files until one of the files becomes empty. Then append the records on the remaining file to the merged file.

Assume that the files to be merged are not empty and have been opened with filepointers f1_ptr,f2_ptr, and f3_ptr. Filepointers f1_ptr and f2_ptr point to the input files to be merged. Filepointer f3_ptr points to the resultant merged file. The files are made up of records of typedeffilerecords, with key the field of filerecords on which merging is to be based. The merge may be written as follows:

merge(f1_ptr,f2_ptr,f3_prt)

/* Merges the files pointed to by f1_ptr

and f2_ptr. The merged file is pointed

to by f3_prt. The files must be opened before the merge is invoked.

*/

FILE *f1_ptr,*f2_ptr,*f3_ptr;

{

struct filerecords record1,record2;

int flag,length;

flag = 0;

length = sizeof(filerecords);

    >Comment

fread(&record1,length,1,f1_ptr);

fread(&record2,length,1,f2_ptr);

    >Comment

while(flag == 0)

    >Comment

if(record1.key < record2.key

{

    >Comment

fwrite(&record1,length,1,f3_ptr);

    >Comment

if(fread(&record1,length,1,f1_ptr) == 0)

{

flag = 1;

    >Comment

fwrite(&record2,1ength,1,f3_ptr);

}

}

    >Comment

else

{

    >Comment

fwrite(&record2,length,1,f3_ptr);

    >Comment

if(fread(&record2,length,1,f2_ptr) == 0)

{

flag = 2;

    >Comment

fwrite (&record1,length,1,f3_ptr);

}

}

    >Comment

if(flag == 1)

    >Comment

while(fread(&record2,length,1,f2_ptr) != 0)

fwrite(&record2,length,1,f3_ptr);

    >Comment

else

    >Comment

while(fread(&record1,1ength,1,f1_ptr) != 0)

fwrite(&record1,1ength,1,f3_ptr);

}

The files must be opened before the merge is invoked.

The following program, when the merge is included, merges two files and prints the resultant file.

#define NULL 0

#include <stdio>

typedef struct

{

int key;

}filerecords;

main()

/* Merges two files and prints the merged file.

*/

{

filerecords record;

FILE *f1_ptr,*f2_ptr,*f3_ptr;

    >Comment

f1_ptr = fopen("prime","r");

if(f1_ptr == NULL)

{

printf("Can not open file1");

exit(-1);

}

f2_ptr = fopen("other","r");

if(f2_ptr == NULL)

{

printf("Can not open file2");

exit(-1);

}

f3_ptr = fopen("result","w");

if(f3_ptr == NULL)

{

printf("Can not open file3");

exit(-1);

}

    >Comment

merge(f1_ptr,f2_ptr,f3_ptr);

    >Comment

fclose(f1_ptr);

fclose(f2_ptr);

fclose(f3_ptr);

    >Comment

f3_ptr = fopen("result","r");

    >Comment

while(fread(&record,sizeof(filerecords),1,f3_ptr) != 0)

printf("\n %d\n",record.key);

    >Comment

fclose(f3_ptr);

}


Data Structures Using C: 10.4: Sequential Access in External Memory

10.4: Sequential Access in External Memory

A range of devices is now available for use as secondary storage, the most common being magnetic tapes and disks. Tape cassettes like those used with stereo equipment are prototypical of the magnetic tape used for secondary storage. If a song is being played on such a tape, the time required to access another song on the tape depends on how far away from the current position that song is. This accessibility is the same for magnetic tapes used as secondary storage. Such a tape can be pictured as containing a sequence of fixed-length blocks (Figure 10.4). Each block contains a fixed number of records of a file (say, n per block). One or more input buffer areas (reserved storage areas) are set aside in internal memory, each large enough to hold one block of records. These buffers are required in order to allow for efficient reading and writing of files.

Figure 10.4 Sequential Access on Magnetic Tape

Suppose one buffer is set aside. To read a block of records into the buffer, the tape must be moved at sufficient speed past a read head. This speed determines the rate at which data are transferred to the buffer. If a sequential file is stored on magnetic tape, executing the open statement causes the buffer storage to be allocated, the tape to be rewound so that the first block may be read, and the first block to be read into the buffer. After these operations have occurred, the buffer and the file pointer can be visualized as in Figure 10.5.

If a read is executed, the first record is copied from the buffer into its memory location, and the file pointer is advanced to the next buffer position. Consecutive executions of a read for this file cause similar actions. Finally, when n such reads have occurred, the next block of records from the tape is placed into the buffer, the file pointer is automatically repositioned at the first record of the buffer, and the process is repeated. Had two buffers been set aside, as the current buffer is being read the other would be filled with the next block of file records. A similar procedure takes place for the writes with respect to an output buffer for the file. In this way the buffer is being filled, when reading from a file (or emptied, when writing to a file), at the same time that the computer is carrying out other commands of a program. Record blocking is done because n records can be read or written as a block much faster than individually, since it reduces tape starts and stops. Today's computers, whether mainframe or personal, can execute on the order of 100,000 commands in the time it takes for a block to be read into or written from the buffers. A program may actually process n records of the file while the next block is being transferred to the buffer. For example, the records may be sorted or individually processed during this transfer time. In this case, the time required to process the entire file is determined by the time required to scan the file in this fashion. Thus the number of accesses to secondary storage determines the execution time of the program. This may take minutes, which is a very long time in computer terms.

Figure 10.5 Input Buffer for the File

It should now be clear why sequential files should not be used when records must be accessed in an order other than the order in which they appear on the file. Attempting to access the records in random order may mean spending great amounts of computer time, since minutes may be required for each access. Instead, the typical use for sequential files is when records are to be processed in turn, starting from the first record. In other words, when the processing to be done can be accomplished by a traversal through the records of the file, then sequential files are appropriate. This is typical of external sorting, copying, updating, and printing (such as end-of-the-month billings).

As a practical example, suppose you are given the master file, bookinventory, of Example 10.3, and another transaction file, booktransactions. Booktransactions consists of records representing the arrival of new books for inventory and the sales of books in the last week. If both files are sorted on the same key (say, title), it is possible to traverse each of the files once and update each master file record to reflect new arrivals and sales. A new file must be created for this updated master file. This step would actually be advantageous for security purposes, since the old master file and booktransactions file could be saved. If anything were to happen to the new master file, the procedure could be repeated and the file reconstructed.


Data Structures Using C: 10.5: Sorting Tape Files

10.5: Sorting Tape Files

Once again, one picture�or in this case, a good example�is worth a thousand words. The different merging techniques demonstrated for the following example illustrate some of the complexities of sorting files stored on magnetic tape, when all records of the file cannot fit into internal memory at the same time. Tape sorts are used here, since they highlight the concepts of sorting, but these concepts are also applicable to disk sorts. Detailed analysis of the sorting algorithms discussed here, and others, may be found in Knuth [1973b]. We begin with an extreme case.

Example 10.6

Sort an original file of twenty-one records that are stored sequentially on a tape file. The records have integer key values. Assume that only two records can be kept in internal memory at any one time. Storage for the two records is in addition to input and output buffers. n

Suppose the sequence of records in the original file is as follows:

2 12 17 16 14 30 17 2 50 65 20 32 48 58 16 20 15 10 30 45 16

It is not possible simply to read all the records into internal memory and then apply one of the internal sort algorithms of Chapter 8, since available internal storage is assumed to be only enough to hold two records. Instead, the general strategy, given that two sorted files can always be merged as shown in Section 10.3, is to create sorted subfiles repeatedly and merge them repeatedly until a final merged file containing all the original records is produced. The literature contains many ingenious algorithms for external sorting. This chapter gives only the flavor of the solutions. One major constraint on a solution is the number of tapes available for use in the sort. Assume three available tapes.

10.5.1 Straight Merge

10.5.2 Natural Merge

10.5.3 Replacement-Selection

10.5.4 Polyphase Sort


Data Structures Using C: 10.5.1 Straight Merge

10.5.1 Straight Merge

Perhaps the most straightforward sorting technique is the straight merge. The straight merge distributes the initial records onto two tapes, a and b, so that they contain the same number of records, or so that one tape contains only one more record than the other. If this is done for the original file in Example 10.5, then the configuration of tapes a and b will appear as the initial distribution.

a   2|17|14|17|50|20|48|16|15|30|16

b  12|16|30| 2|65|32|58|20|10|45|

Now a and b can be thought of as being composed of subfiles, each of which is sorted and of length 1. A subfile is a run of length r if it consists of r records sorted in order. Here, a contains eleven subfiles, or runs, of length 1, and b contains ten runs of length 1.

Apply a merge procedure to each pair of runs, each pair containing a run from tape a and the corresponding run from tape b. Thus

 2 17 14

12 16 30, etc.

are "paired." The merge procedure consecutively merges each pair (consisting of a sorted subfile of length 1 from a, and a corresponding sorted subfile of length 1 from b). The resultant merged files are written consecutively to tape c. The result of the first merge of a and b to c is

c 2 12|16 17|14 30|2 17|50 65|20 32|48 58|16 20|10 15|30 45|16

Think of c as containing ten subfiles or runs of length 2 and one run of length 1 and distribute these to a and b equally to obtain the second distribution.

a  2 12|14 30|50 65|48 58|10 15|16

b 16 17| 2 17|20 32|16 20|30 45|

By now there are five paired runs of length 2 and one run of length 1. These may again be consecutively merged, and written to c to obtain the second merge of a and b to c.

c 2 12 16 17|2 14 17 30|20 32 50 65|16 20 48 58|10 15 30 45|16

Another distribution to a and b yields the third distribution.

a 2 12 16 17|20 32 50 65|10 15 30 45

b 2 14 17 30|16 20 48 58|16

These pairs of length 4 can now be merged to obtain the third merge of a and b to c.

c 2 2 12 14 16 17 17 30|16 20 20 32 48 50 58 65|10 15 16 30 45

It takes two more distributions and merges to complete the sort. The fourth distribution

a  2  2 12 14 16 17 17 30|10 15 16 30 45

b 16 20 20 32 48 50 58 65|

is followed by the fourth merge.

c 2 2 12 14 16 16 17 17 20 20 30 32 48 50 58 65|10 15 16 30 45

The fifth distribution is next. It is followed by the fifth merge.

a  2  2 12 14 16 16 17 17 20 20 30 32 48 50 58 65

b 10 15 16 30 45

c  2  2 10 12 14 15 16 16 16 17 17 20 20 30 32 45 48 50 58 65

This completes the sort, which has taken five distributions and five merges. The sort is described as a five-pass sort, with five distribution phases and five merge phases, using three tapes. Programming this process is not a trivial task. Ignoring details, the basic time requirements can be seen to be determined by the number of times files must be rewound or rewritten and the number of records that must be accessed from secondary storage (tapes). This assumes that the internal memory processing will be done so quickly that it does not add to the total time for the sort. Before the original file, c, can be distributed (rewritten) to tapes a and b. those tapes must be rewound and the original file must be rewound. Before a and b can be merged (and written) to c, they must be rewound, and c must be rewound. Also, in each phase, whether distribution or merge, each record must be accessed from secondary storage. Thus the total time is proportional to the number of passes (five here). If the access time of a record is Ta, and the rewind time Tr, then the total time is 5 � (2 � 21 x Ta + 3 � Tr). Note the factor of 2 associated with the twenty-one records and the access time. The factor is 2 because there are two phases, distribution and merge.


Data Structures Using C: 10.5.2 Natural Merge

10.5.2 Natural Merge

The straight merge sort does not take advantage of any natural ordering that may exist in the original file. Even if that file were given in sorted order, the straight merge would proceed as before. A natural merge sort distributes the records of the original file to a and b, so that sorted subfiles or runs are preserved and treated as a unit. Such a distribution would assign 2 12 17 to a, since they form a run. The 16 terminates this run and goes to b. The 14 terminates this run and goes to a as the beginning of its second run, followed by 30. So far the result is

a  2 12 17|14 30

b 16

The 17 now terminates the 14 30 run and goes to b. Note that this extends the first run of b to 16 17. At this point, 2 terminates this run with the following result.

a  2 12 17|14 30

b 16 17

If balance is to be maintained, each file must end up with a number of runs differing by no more than 1 from the other files. Therefore the next run (starting with 2) must go to b. Subtleties of this kind must be carefully monitored in the actual implementation of merge sorting algorithms. The resultant distribution will be the initial distribution.

a  2 12 17|14 30   |20 32 48 58|15 16

b 16 17   | 2 50 65|16 20      |10 30 45

Merging these pairs of runs and writing the merged results to c yields the first merge.

c 2 12 16 17 17|2 14 30 50 65|16 20 20 32 48 58|10 15 16 30 45

Another distribution and merge gives the second distribution,

a 2 12 16 17 17|16 20 20 32 48 48

b 2 14 30 50 65|10 15 16 30 45

which is followed by the second merge.

c 2 2 12 14 16 17 17 30 50 65|10 15 16 16 20 20 30 32 45 48 58

A third and final distribution and merge complete the sort. The third distribution yields

a  2  2 12 14 16 17 17 30 50 65

b 10 15 16 16 20 20 30 32 45 48 58

The result of the third merge is as follows:

c 2 2 10 12 14 15 16 16 16 17 17 20 20 30 30 32 45 48 50 58 65

This is a three-pass, two-phase sort, using three tapes. The total time is 3 � (2 � 21 � Ta + 3 x Tr). Any inherent ordering of the initial data allows the number of passes to be reduced, as compared to the straight merge sort.

Note that to merge runs embedded in a file, the procedure of Section 10.3.1 requires modification.


Data Structures Using C: 10.5.3 Replacement-Selection

10.5.3 Replacement-Selection

Since there is typically room for a reasonable number of records in internal memory, both the straight and natural merge sorts can be speeded up. For instance, if m records can be sorted in internal memory, the straight merge sort can start with the distribution of runs of length m on each of the tapes a and b.

There is a clever way to produce runs for distribution (for example, with the natural merge sort) without actually storing the m records. This is the replacement-selection sort, which uses a heap of size m and works as follows. Start with the first m keys of the original file of Example 10.6 in internal memory and create a heap to obtain Figure 10.6(a) (m = 7).

Remove the root key from the heap and output it to an output file. The next input file key, 2, is inserted at the root of the heap, and the heap is reheaped, giving Figure 10.6(b).

Figure 10.6 Replacement-Selection Sort

Again, the root key is removed and written to the output file. The next record key of the input file, 50, is inserted at the root and reheaping occurs. Figure10.6(c) results.

Continuing in this way generates a run on the output file of length 8, which would be terminated when 16 from the input file is inserted at the root of the heap. The situation at that point is shown in Figure 10.6(d). Instead of terminating the run at length 8, treat 16 as associated with the next run, since 16 is less than the last output value, 20. This means that 16 must be distinguished from the heap entries for the current run; it is shaded in Figure 10.6(d). In reheaping, current run entries are considered smaller than next run entries. Figure 10.6(e) shows the heap after reheaping. Each input smaller than the last output is handled in this way. Eventually the heap becomes filled with entries of the next run, and the process repeats.

Notice that the length of the output run will always be at least m (with the possible exception of the last run generated). Continuing this process, called replacement-selection, we obtain the output file,

2 2 12 14 16 17 17 20 30 32 48 50 58 65|10 15 16 16 20 30 45

The original file consisted of ten runs; the output file after replacement-selection consists of two runs. With randomly ordered input files, this replacement-selection generates runs whose average length is 2m. The example exhibits this behavior in an exemplary fashion.

Notice that both the straight and natural merges spend half their time accessing records for the distribution phase. Spending this time can be avoided by using another tape. By this technique, each time two runs are merged, they can be written, alternately, to tapes a and b. This eliminates the need for extra, basically unproductive, time spent in distribution from tape c.


Data Structures Using C: 10.5.4 Polyphase Sort

10.5.4 Polyphase Sort

Rather than using more tapes, it is possible to incorporate the distribution phase (except for the initial distribution) into the merge phase. Such a sort is called a polyphase sor t. In this type of sort, the tapes being merged, and the tape to which the merged subfiles are written, vary continuously throughout the sort. In this technique, the concept of a pass through records is not as clear-cut as in the straight or the natural merge. A distribution, a merge, and a pass all blend together. This might have more properly been called an amorphous sort, except that it has considerable character and is one of the better sorts.

The polyphase sort starts with an initial distribution of runs on tapes a and b. The initial distribution is critical to its proper execution. For the case of the original file of twenty-one records in Example 10.6, the initial distribution should be thirteen runs to tape a, and 8 runs to tape b, each of length 1. It is the number, not the length, of the runs on each tape that is critical. The number of records for the example is twenty-one because the first few Fibonacci numbers are 0, 1, 1, 2, 3, 5, 8, 13, 21. The Fibonacci numbers are crucial to the polyphase sort because the number of records must be a Fibonacci number to make the polyphase sort work.

When a three-tape polyphase sort is used, the original number of runs should be a Fibonacci number, with the distribution of runs between tapes a and b the two preceding Fibonacci numbers, which add to the original number of runs. In the present case, 8 + 13 = 21. When more tapes are used, generalized Fibonacci numbers, which specify the required number of records, become the basis for the initial run distribution among the tapes. No programs will be shown for the implementation of the details of the external sorting algorithms discussed in this section. Therefore it is unnecessary to resolve in detail the question of how to proceed when the actual number of runs is not a Fibonacci number. The remedy, though, involves adding an appropriate number of "dummy" runs [Knuth, 1973b] .

Getting back to the workings of the polyphase sort, the initial situation after distribution will be an initial distribution.

a   2�12�17�16�14�30�17�  2�50�65�20�32�48  13 runs of length 1

b  58�16�20�15�10�30�45� 16�                 8 runs of length 1

c

This distribution required twenty-one record accesses.

We now proceed by merging the first eight runs of a and the eight runs of b to c to obtain the merge of the first eight runs of a and b to c.

a  50�65�20�32�48                                 5 runs

b

c   2 58�12 16�17 20�15 16�10 14�30 30�17 45�2 16  8 runs

Notice that b is now empty, a total of 8 + 8 = 16 records have been accessed, and a and c contain 5 + 8 = 13 runs. All three tapes had to be rewound before the initial distribution, but only b need be rewound now. It is possible to continue merging the first five runs of c and the five runs of a to b. This results in a merge of the first five runs of a and c to b.

a

b  2 50 58�12 16 65�17 20 20�15 16 32�10 14 48  5 runs

c    30 30�17 45   � 2 16   �                   3 runs

Notice that a is empty, a total of 5 + 5 � 2 = 15 records have been accessed, and b and c contain 5 + 3 = 8 runs. Tape a is now rewound, and the first three runs of b are merged with the three runs of c. The result is the merge of the first three runs of b and c to a.

a  2 30 50 58�12 16 17 45 65�2 16 17 20 20  3 runs

b 15 16 32�10 14 48                         2 runs

c

Tape c is now empty and must be rewound. A total of (3 � 3) + (3 � 2) = 15 records were accessed, and a total of 2 + 3 = 5 runs remain. Merging the first two runs of a and b to c yields

a  2 16 17 20 20                                   1 run

b

c  2 15 16 30 30 32 50 58�10 12 14 16 17 45 48 65  2 runs

This leaves b empty, so we rewind it. A total of (2 � 3) + (2 � 5) = 16 records were accessed, and 1 + 2 = 3 runs remain. Merging the run of a and the first run of c to b yields.

a

b  2 2 15 16 16 17 20 20 30 30 32 50 58  1 run

c 10 12 14 16 17 45 48 65                1 run

A total of (1 � 5) + (1 � 8) = 13 record accesses occurred, and 1 + 1 = 2 runs are left. Tape a is rewound. Then a final merge produces

a  2 2 10 12 14 15 16 16 16 17 17 20 20 30 30 32 45 48 50 58 65

b

c

This final merge of b and c to a took 13 + 8 = 21 record accesses. A total of 117 ( = 21 + 16 + 15 + 15 + 16 + 13 + 21) record accesses is required. These 117 record accesses represent an effective 117/21, or 5 4/7, passes. The total time for this algorithm is 117 x Ta + 8 x Tr. The straight merge sort and the natural merge sort required 210 � Ta + 15 � Tr and 126 � Ta + 9 � Tr time, respectively.

The polyphase sort, until completion, results in exactly one empty tape, so that the remaining tapes may be partially merged to it. After each merge, the distribution of runs is made up of two consecutive Fibonacci numbers (13 + 8, 8 + 5, 5 + 3, 3 + 2, 2 + 1, 1 + 1, 1 + 0) . With more tapes , this is also true , but generalized Fibonacci numbers appear.


Data Structures Using C: 10.6: Direct Access in External Memory

10.6: Direct Access in External Memory

Accessing a song on a phonograph record is different from accessing a song on a tape cassette. Think of the songs as stored along the grooves of a record rather than along the tape of a cassette. Instead of going through each song between the current arm position and the location of the desired song (and scratching the record!), the arm is moved directly to the proper groove. Once the arm is placed, it is necessary to wait until the record revolves, so the beginning of the song appears under the arm. The movable arm eliminates the need to pass through all songs that intervene between its current and desired positions. The magnetic disk works the same way. It is the reading head that moves to the desired storage location, thus saving access time compared to the magnetic tape.

Magnetic disks and disk packs are examples of direct access devices used for secondary storage. In this context, direct access means that a program can go straight to desired information stored on a given surface and cylinder. This access time is not constant. It is slower than random access in internal memory, but generally much faster than the sequential access time for tapes. A floppy disk has similar access capability but is slower and has much less storage capacity. Thus in terms of access time, direct access storage devices fall between sequential access devices, such as magnetic tapes, and random access memory. Hard disks are generally used with mainframes and minicomputers, whereas microcomputers or personal computers vary from no disks through floppy disks to hard disks.

A random access memory allows very fast access to any element of the memory. The access time is the same for all stored elements, no matter which element was last accessed. Sequential devices allow fast access "locally," but all elements between the currently accessed element and the next desired element must be passed through before the next element may be accessed. Consequently, the time to access the next desired element is proportional to its distance from the currently accessed element. Direct access devices are intermediate between these two extremes. They act like sequential devices with tabs. Just as using the alphabetical tabs in a dictionary allows a person seeking a word to find it quickly, so do "tabs" allow one to close in, relatively quickly, on the desired area of the sequential storage. In direct access devices, the "tabs" are the movable read/write heads.

Sequential devices are convenient for tasks that may be accomplished by traversing records. However, when the order in which the records must be processed cannot be predicted, sequential access devices are not efficient. Tasks requiring random access to records cannot be accomplished efficiently using sequential memory devices. The introduction of economical direct access devices made random access feasible. As a result, airline reservation systems, interactive computing, and instant updating of bank accounts have become commonplace. This was made possible by the development of appropriate technology for direct access memory.

A magnetic disk is a flat, circular platter on whose surfaces information can be stored and accessed. It can be pictured as a phonograph tone arm and record, with its concentric tracks corresponding to the grooves of the phonograph record and its movable read/write head corresponding to the tone arm. A disk pack consists of a pile of such magnetic disks and a read/write head for each record surface�that is, one head for each side of a record (see Figure 10.7). The read/ write heads are typically coupled so that they all move at the same time. They are positioned over the same track on their respective surfaces, although only one head is actually reading or writing at any moment. There may be twenty magnetic disks and a hundred tracks on each disk.

Figure 10.7 A Disk Pack

Each track of a surface in a disk pack stores records in blocks, similar to the blocking of records on magnetic tapes. Each track behaves as a sequential access device, like a magnetic tape. That is, each block must be passed through, as the disks rotate, before another block farther along the track can be accessed by the surface read/write head.

If we focus on the same track on each surface, then the pile of tracks forms a conceptual cylinder in space, as shown in Figure 10.8. Assuming tracks are numbered (say, from 1 to 100), a cylinder consists of all tracks with the same number. Thus the disk pack can be viewed as composed of one hundred concentric cylinders.

A cylinder is important because it allows any block of records stored on a surface of the cylinder to be accessed without moving the read/write heads. Accessing a block stored on a different cylinder (or track) requires moving the read/write heads. The time required to move the heads from one cylinder to another is called seek time; it is proportional to the distance between the cylinders. This time typically takes a significant fraction of a second (1/10th to 1/20th). It is the dominant factor in dealing with time requirements for accessing blocks on magnetic disks.

Once the read/write heads are positioned over the desired cylinder, the time required for the desired block to rotate under the heads is called latency time. It can be on the order of one hundredth of a second.

Figure 10.8 Conceptual Cylinder


Data Structures Using C: 10.7: Sequential and Random Access of Disk Fi

10.7: Sequential and Random Access of Disk Files

Often large amounts of information, too much to fit into internal memory, must be organized so that random requests for specific pieces of information can be readily satisfied. The methods of organization that are useful in internal memory are not adequate for data stored on disks in external memory.

10.7.1 Sequential Access

10.7.2 Random Access

10.7.3 Indexed Sequential Access

10.7.4 B-trees for Sequential and Random Access


Data Structures Using C: 10.7.1 Sequential Access

10.7.1 Sequential Access

There is a natural way to store records sequentially on magnetic disk to minimize total seek time for a traversal through the records. This storage technique is based on the cylinder concept. It starts by filling blocks of the top surface track of the outermost cylinder with records. The records are in sequence within the blocks. When the track is full, the same process is repeated with the track of the next surface, and the next, and the next, until the lowest surface track is filled. At that time the outermost cylinder has been filled. Continuing in the same way, records are stored on each of the inner cylinders, starting with the next adjacent cylinder. This is similar to storing records sequentially on magnetic tape. In fact, we can picture the surfaces and cylinders laid out linearly as in Figure 10.9. This arrangement has all the inherent drawbacks of magnetic tape, whenever random accessing of records is required.

We can adapt disk storage to random accessing by taking advantage of the tab feature of the disk, which allows the head to go directly to any cylinder and surface. The problem is how to determine exactly where to go. If one knew that the record to be retrieved were the ith record, then it would be easy to calculate on which cylinder and surface it would be located. This is not often the case, since the search is conducted by means of key value, not by order in the sequence.

Figure 10.9 Sequential Access on a Magnetic Disk Pack

A linear search of the file could be done but might take O(n) time, when n records are stored in the file. This could take minutes. If the records are stored in sorted order by key value, a binary search of the file might be done. The difficulty here is that 1g n accesses are required. A million-record file would require twenty accesses, and each access might require a seek. While the binary search is better, it still might take seconds. This is reasonable for isolated requests, but during this time there may be many requests for desired retrievals. Waiting requests may build up interminally, with unlimited waiting time before all are processed. The need to insert or delete records further compounds the problem.

Just as in the case of internal memory, keeping records in blocks with pointers to succeeding blocks is a possibility, based on the concept of lists. This may avoid insertion and deletion difficulties, but it does not alleviate the problem of searching. We encountered this same problem with lists in internal memory. What about binary search trees? Even if balanced, a million records may need as much as 20 � c seeks, where c is a small constant.


Data Structures Using C: 10.7.2 Random Access

10.7.2 Random Access

We still have hash tables in our arsenal. Using hash tables in secondary storage requires some accommodation to its access capabilities. Buckets, an extension of the chaining policy for collision resolution with internal memory, are appropriate. The idea is to create a hash table based on key values. The hash table will contain pointers to lists. Each list, referred to as a bucket, contains all records that hash to the same hash table address. The lists are stored in blocks on the disk, each block holding a fixed number of records and a pointer to the next block containing other records on the list. When a key is inserted into the hash table, the program determines its hash address and then inserts the key into the bucket pointed to by that address. The key may be inserted into the bucket by traversing the list of linked blocks until a block is encountered with room for the record. If no such block is found, a new block is appended to the list, and the record is inserted in that block. Care must be taken to avoid wasting seek time (and latency time) when these lists are being traversed. This means that the blocks of a bucket should be on the same cylinder. Searching for a key is similar to making an insertion. Deletion involves the usual deletion from a list, with care taken to release unneeded blocks when possible. When small enough, the hash table itself is kept in internal memory to avoid one disk access.

In practice, such hash tables can be devised so that one or two disk accesses are required for either successful or unsuccessful searches. This is accomplished by selecting a good hash function, an appropriate fixed number of records per block, and an appropriate number of buckets. The usage factor of such a hash table is given by n/bk, where n is the number of records stored, b is the number of buckets, and k the fixed number of records per bucket. If the performance of the table degrades as insertions are made, the table may be reorganized.

Other interesting and important external hashing methods are presented in Larson [1976], Litwin [1960], and Fagin, Nievergelt, Pippenger, and Strong [1979].


Data Structures Using C: 10.7.3 Indexed Sequential Access

10.7.3 Indexed Sequential Access

While the hash table solution may yield fast average access times, there is no guarantee for the worst case, nor any convenient way to access records in sorted order by key value. To achieve the ability to access records in random order or in sorted order, as necessary, we must incorporate an old idea familiar to all. Suppose a visitor from outer space wanted to find the location of a particular street, say Bond Street in London. The visitor might go to a world atlas and determine first where London's country, England, is situated. The intergalactic traveler might then find a map of the country, and locate London. Finally, a city map of London gives the creature the exact location of the street. On a more mundane level, suppose a student wants to locate a particular book in the university library. The student would go to the file cabinet of authors and find the drawer that contains authors' last names with the appropriate beginning initial. She would then search for the card with the correct author and title, which gives a library call number. Next she would consult a library directory to learn the floor location of those call numbers. Finally, she would go to that location and search for the book with that particular call number. The critical concept in such searches is the availability of one or more directories, to be used to determine quickly the location of the particular item desired.

The magnetic disk supports the use of such a directory to focus the search. It can be used to find a particular cylinder and surface relatively quickly, eliminating the need for traversing sequentially through intervening records. Until recently, when B-trees (see next section) were introduced, the most popular method of creating and maintaining such a directory used the natural sequential storage of records, so that they appear in sorted order by key value. This is known as the indexed sequential access method (ISAM); it may be implemented in various ways, typically using three or four levels of directories. A first directory, known as the master index, may be searched to determine the cylinder on which a particular search key appears. If a disk pack has two hundred cylinders (tracks), then the master index has two hundred entries, one for each cylinder. The entry for the ith track gives the highest key value appearing on that track. A search of the master index then readily determines the cylinder on which a search key may reside. That cylinder is accessed; it contains a cylinder index directory. If a disk pack has ten surfaces, the cylinder index has ten entries, one for each surface. The entry for the ith surface gives the highest key value appearing on that surface. A search of the cylinder index determines the surface on which a search key may reside. Finally, the track of the cylinder on that surface may be searched sequentially for the desired key, resulting in a successful or unsuccessful search. With such an implementation, unused storage is initially left in blocks so that the insertion of new records may be more easily accommodated. If a block is full, records can be shifted to the right to other blocks, so that the new record is inserted in proper order.

Overflow areas must also be available and maintained on disk, in case the track storage is exhausted due to insertions. Tracks and cylinders are set aside for this purpose. Insertions may require the master or cylinder index to be updated. Similarly, deletions of records, leaving vacated storage to be reclaimed, may require updating of directories and shifting of records. Deletions may also be treated simply by marking a record as deleted, but this may eventually lead to the need for reclamation of this storage, degrading the efficiency of the implementation. Because of overflow from insertions, performance may degrade, necessitating reorganization, allocation of more disk storage, and reconstitution of the file.

The overflow areas may be maintained as lists of records in sorted order, linked by pointers. When the proper cylinder index is searched, the pointer to be followed may point to an overflow area, in which case the search amounts to a traversal of a list of records. Otherwise, the record is said to reside in the prime area, and the search amounts to a sequential traversal of records on a track. The prime area consists of reserved tracks on cylinders designated as prime cylinders.

For a given number of records to be stored, there is no guarantee that overflow will not degrade the system, unless a high price is paid by dedicating large amounts of track storage to reduce this possibility.

Each cylinder index actually has additional entries giving the highest key value in its overflow area for each surface and a pointer to the first record of the overflow list for the surface. Conceptually, we may visualize the organization of such implementations as shown in Figure 10.10.


Data Structures Using C: 10.7.4 B-trees for Sequential and Random Acce

10.7.4 B-trees for Sequential and Random Access

A more recent method allows for achieving both the random access of records and a traversal through the records in sorted order. This method allows a more flexible use of the disk and guarantees excellent worst-case performance for a given number of records, with an acceptable amount of dedicated disk storage. It is known as the virtual sequential access method (VSAM). To see how it might be implemented, we study its basis B-trees. VSAM and many data base management systems use modified B-trees in their implementation. The basic reference for B-trees is the classic paper by Bayer and McCreight [1972]. Hash trees are an alternative to B-trees for the organization of large files of data. They are discussed and compared to B-trees by Bell and Deen [1984]. Tremblay and Sorenson [1984] discuss many file structure organizations. See Ullman [1982] for more on data base systems.

Figure 10.10 shows that the directories (the master index and the cylinder indexes) form a tree structure that guides the search for a record. Programmers try to find specialized trees leading to efficient searches, efficient use of storage, and relatively easy maintenance. Since the depth of the tree determines the worst-case search times, the goal is to find trees with short paths to terminal nodes. This implies that the nodes of the tree generally will have many successors. B-trees satisfy these requirements.

Figure 10.10 Organization of Indexed Sequential Files

A B-tree of order m has the following four properties:

1. Every node of the tree, except the root and terminal nodes, has at least �1/2 m� subtrees.

2. Every terminal node has the same depth.

3. The root node has at least two subtrees, unless it is a terminal node.

4. No node has more than m subtrees.

Each node of a B-tree will have the form shown in Figure 10.11. This represents storage for m - 1 keys and m pointers, although the storage may not all be used at a given time. If n is less than m, not all m keys are present in the node. Each pointer Pi points to a subtree of the node. The keys K1, K2, . . . , Kn are kept in sorted order, so K1 < K2 < � � � < Kn within the node. The subtree pointed to by Pi+1 contains all records of the file whose key values are less than Ki+1 and greater than or equal to Ki. P1 and Pn+1 point to subtrees containing records with key values less than K1 and greater than or equal to Kn, respectively. For example, if the keys are

1,3,5,6,7,8,13,20,21,23,24,30,37,40,56,60,62,63,70,80,83

then the tree of Figure 10.12(a) is a B-tree of order 4 storing these keys and their records.

Figure 10.11 Form of Each Node on a B-Tree

The terminal nodes of the B-tree in Figure 10.12 are not shown. Instead, the pointers to them have been replaced by null pointers. This reflects the way the tree might be represented in external storage. Notice that, in general, null pointers will appear at the same depth in the tree�depth one less than the terminal nodes. A generalized inorder traversal (see Chapter 7) of the B-tree accesses the keys in sorted order. Associate with each null pointer the key value accessed immediately after the pointer is considered in such a traversal. A unique key is then associated with every null pointer except the last, and each key has a unique null pointer associated with it. For example, the first three null pointers have keys 1, 3, and 5 associated with them. The fourth null pointer (to the right of 5) has key 6 associated with it. Consequently, the number of null pointers, and the number of terminal nodes, is exactly one greater than the number of keys stored in the tree.

Any B-tree has a root node with at least two successors. Each successor has at least �1/2 m� successors. Thus there are at least 2 �1/2 m� d-2 nodes at depth d > 3 of any B-tree. A B-tree of depth d > 3 has at least 2 �1/2 m� d-2 terminal nodes, so

n + 1 > 2 �1/2 m� d-2

This implies that d < 2 + lgx�(n + 1)/2�, where lgx denotes logarithm with base x and x = �1/2 m�. If n = 224 - 1 = 16,777,215 and m = 2 � 28 = 512, then (n + 1)/2 = 223, and lgx [(n + 1)/2] = lg256(223) < 3.

A B-tree storing 16,777,215 records will therefore have depth �4. We shall see that d - 1 is the maximum number of nodes that must be accessed when searching a B-tree for a search key. Thus we are guaranteed that no more than three node accesses are required to search for a key in the B-tree with 16,777,215 records!

A B-tree may be viewed as a generalization of a binary search tree, with a special imposed balance requirement to ensure against too great a depth. To search a B-tree for a search key value (say, 22) entails accessing its root node and traversing the node's keys until 22 is encountered, a key that exceeds 22 is reached, or the last key stored in the node is passed. If 22 is found, the search is complete. In Figure 10.12(a) the key value 40 is reached. The pointer to its left points to a subtree that must contain the search key, if it is in the tree at all. That subtree is then searched in the same way. Its root node is accessed, and 30 is reached. Following its left pointer, the final subtree root is accessed. Its key value, 23, is reached. Its left pointer is null, which means that the search terminates unsuccessfully.

Property 1 ensures that, except for the root and terminal nodes (which don't appear anyway), every node of a B-tree will be at least half full. Property 2 ensures that the B-tree will not be scraggly. All paths to terminal nodes must have the same length. The idea is to store information of a node on blocks of a cylinder on the disk, so that accessing a node requires one seek time and all keys of the node may be read into internal memory. The search through the keys of a node, to determine the next node to access, then takes place rapidly in internal memory, using an appropriate search technique.

(a) Initial B-tree

(b) After Insertion of 35

Figure 10.12 A B-Tree of Order 4 with Twenty-one Records

(c) After Splitting, Due to Insertion of 28

(d) After 7 is Deleted and 5 is Borrowed

Figure 10.12

(e) After Deletion of 83 and Merging

Figure 10.12

After an insertion or deletion, of course, the tree must be maintained as a B-tree. This may be done making local modifications that are simpler than those required for AVL trees. For insertion, the key to be inserted is searched until a null pointer is to be followed. This assumes that only one record with a given key value appears in the tree and that it is similar to the initial insertion in AVL trees. If the node containing the null pointer has unused storage, then the key is simply inserted at the proper place in this node. This may require the shifting of keys in the node, but it is accomplished quickly in internal memory. For example, if key 35 is inserted in the tree of Figure 10.12(a), the result is the tree of Figure 10.12(b).

All insertions will be made in this way, in nodes at the lowest level, until the node becomes full. For example, if 28 is to be inserted next in the tree of Figure 10.12(b), no room exists in the node x 21 x 23 x 24 x . The m + 1 key values (the m old keys and the 1 new key) can be split by leaving �m/2� - 1 in the old node and moving the remaining records, except the middle one, the �1/2m�th, to a newly created node. The middle record is inserted in the predecessor (parent) of the old node. The result is the B-tree of Figure 10.12(c).

Had the predecessor been full, it would have been split in the same way in turn. Splitting may be necessary at every node of the search path, back to and including the root. If this occurs, the depth of the resultant B-tree is one greater than the original B-tree.

Deletion is done similarly. A key is simply deleted from a lowest-level node, and the keys shifted over, if the resultant number of keys satisfies the constraint that the node contain at least �m/2� - 1 keys. If the constraint is violated, a key can be borrowed from a "neighbor" node, provided this may be done without violating the constraint for that node. In this case, the predecessor or successor key of the borrowed key moves into the deletion node, and the borrowed key takes its place in the predecessor node. For example, if key 7 is deleted from the tree of Figure 10.12(b), key 5 is borrowed from its neighbor (13 cannot be borrowed). The result is Figure 10.12(d).

If no neighbor can spare a key, the total number of keys between a neighbor and the deletion node will be less than m. These nodes may then be merged, along with a key from their predecessor. In the tree of Figure 10.12(d), if 83 is to be deleted, this situation will occur. The result is Figure 10.12(e).

It is possible for the predecessor to become too sparse, although this did not happen in our example. It is then treated as though its key were deleted, and so on, until the entire process is complete. This may go all the way to the root, in which case the root disappears and the depth of the tree decreases by 1.

If the key to be deleted were not in a lowest-level node, it would be replaced by its successor and the successor node treated as if the successor were being deleted.

These insertion and deletion procedures preserve the B-tree properties. The B-tree is an external memory data structure that plays the same role as AVL trees in internal memory. In practice, a modification of the basic B-tree is used to ensure better storage utilization and to increase the efficiency of traversal through the records in sorted order for VSAM. With B-trees the generalized inorder traversal must be used for sequential access. This is time-consuming and requires stack storage. Instead of storing some key records in higher-level nodes, the idea is to store all records only in lowest-level nodes and chain the lowest-level nodes by pointers. For example, in Figure 10.12(a) all nodes at depth 3 could be linked in a chain, with the leftmost node first and the rightmost node last on the list. A traversal then amounts to a traversal of this chain.

It is clear that storage management is needed for external memory, just as for internal memory. This problem is not discussed here, but the internal memory techniques are applicable, when properly modified to take into account the structure of the disks.


Data Structures Using C: 10.8: Summary

10.8: Summary

Files are stored in secondary storage and provide powerful facilities for the temporary and permanent storage of large amounts of data. Magnetic tapes and magnetic disks are two important devices on which to store files. These are typical of sequential and direct access devices, respectively.

Because of the structure and resultant access capabilities of these devices, internal memory techniques must be modified in order to deal efficiently and conveniently with files stored on them. It is possible to sort efficiently, even with sequential files, by using external sorting techniques. Primarily, these are merge sorts, such as straight, natural, and polyphase sorts. Other merge sorts may also decrease sorting time. In general, sequential files are suitable for tasks that can be accomplished by traversing the records of the file. These tasks include sorting and updating master files.

It is possible to achieve efficient random accessing of records with direct access devices when access is by a specific key. Hash tables are useful for this purpose. To achieve a combined capability of random plus sequential access in sorted order by a specific key requires considerable file organization. This organization involves the creation of a file directory, which may consist of a number of levels. The directory guides the search efficiently to the desired record with specific key value. A number of directory implementations are possible; directories based on B-trees are becoming increasingly popular. Such directories may be searched relatively quickly, allowing efficient insertions and deletions while utilizing storage well.

At this point, you should have filed away the substance of this book. It is hoped that you have done so to allow quick retrieval, while maintaining room for insertions. Deletions may be necessary but should be minimal.


Data Structures Using C: Exercises

Exercises

1. Suppose a function reads in each record of Example 10.4 and creates a file consisting of every other prime number on the file. Show the state of the input and output files and their file pointers after the fourth record of the output file has been written.

2. Write a function to create the output file of Exercise 1.

3. Write a function to create reversedprime of Example 10.4, assuming the array a may only be of length 10 while reversedprime may contain many more entries.

4. How many writes and reads will be executed by the function merge if file 1 and file 2 are, respectively,

1 2 8 10 13 15 40

3 7 10 12 17 20 45 50 60

5. Write a function to create a sequential file that is the same as file 1, except that all records on file 1 that are also on file 2 do not appear.

a. Assume file 1 and file 2 records appear in sorted order based on the key field.

b. Assume file 1 and file 2 are not sorted, and you do not know how to sort a file.

6. Write a function to produce a new file that is the same as an input file except that all blanks have been deleted from it.

7. Write a function that takes two sequential files and appends the second file to the first.

8. Why are the sort procedures of Chapter 8 inappropriate for files?

9. Why does the number of passes required by a straight sort not depend on any initial ordering among records?

10. Write a function for the straight merge.

11. Modify the merge function of Section 10.3 so that it merges two runs embedded in files.

12. Write a function to distribute records as required by the natural merge.

13. Write a function for the natural merge.

14. Simulate the behavior of replacement-selection to verify that the average length run produced is 2m.

15. Why must Fibbonacci numbers govern the number of initial runs for three tapes in the polyphase sort?

16. a. If four tapes are used in a polyphase sort, what is the relation determining the initial distribution of runs?

b. Can you generalize to n tapes?

17. Write a function for the polyphase sort using three tapes.

18. Why do the movable read/write heads and surfaces act as tabs for a disk?

19. Suppose the master index of Section 10.7.3 does not fit on one cylinder. How may another directory level be created to serve as an index to the master index?

20. Why do B-trees allow more flexible use of a disk than directories implemented using prime and overflow areas?

21. Create a B-tree of order 4 to store the nametable records of Chapter 6. The records are to be ordered alphabetically so that the B-tree can be searched with the name field as key.

22. Write a function to insert a record into a B-tree of order m that is stored in internal memory.

23. Can you write the function of Exercise 22 so that it is independent of whether the B-tree is stored in internal or external memory? If so, write it. If not, explain why not.

24. Write a function to delete a record from a B-tree of order m that is stored in internal memory.

25. Why does the chaining of lowest-level nodes discussed in Section 10.7.4 allow more efficient traversal through a file?

26. Write a function to create a B-tree stored in internal memory.

27. Suppose we want to store variable length records in a B-tree. How may this be done?

28. Why is it not feasible to store all records in the root node of a B-tree so that only one seek is required?

29. A B-tree of order 3 is called a 2-3 tree, since each node has two or three successors.Although not useful for external storage, such a tree may rival AVL trees for an internal memory structure in which to store records. Compare its search, insertion, and deletion algorithms and its time requirements with those for AVL trees.

30. Write a search function for a B-tree of order m.

31. Write a function to traverse through the records of a B-tree in order by key value.

32. Suppose the individual records of Figure 6.5 are to be stored on a disk with the recordptr field of the nametable pointing to the address of an individual record on the disk. The address is a cylinder and surface specification. What is a reasonable way to allocate disk storage to the individual records?

33. Suppose records of a file were stored in a hash table, with buckets for random access by key, and linked in sequential order by pointers for sequential access. Compare the insertion and retrieval times of such an implementation with the use of B-trees.


Data Structures Using C: Suggested Assignment

Suggested Assignment

This assignment is appropriate for a group of students to do together. It involves a task of reasonable size, which must be partitioned so that each member of the group knows just what to do and so that, when all parts are put together, the final solution works correctly. The idea is to produce an "index" for a book. This involves

1. The creation of files, using the sequential files available in C.

2. The creation of appropriate functions that will allow the records of a file to be accessed randomly, or in sequential order, by key.

One file, the book file, is to consist of records that represent the pages of a book. Each page contains one hundred lines, and each line consists of one hundred characters. The key consists of a page number and a line number. A second file, the dictionary file, is to consist of records representing words to appear in an index for the book represented on the book file. When the solution is run, it should use a scaled-down version of the book file, which will consist of a few pages of ten lines and twenty characters each, with the dictionary containing relatively few words (say, twenty-five).

The dictionary file is to be a sequential file, with the file implemented using B-trees. Instead of a disk, assume that enough internal memory exists to store the book file. The index for the book should be output so that the words of the index appear in alphabetical order, followed by the page and line numbers on which the word appears in the book. Also, an input file consisting of page and line numbers appearing in random order should result in an output consisting of the text for each record of this input file.


Data Structures Using C: CHAPTER 11: TOPOLOGICAL SORTING: AN ARCHETYP

CHAPTER 11: TOPOLOGICAL SORTING: AN ARCHETYPAL SOLUTION

Introduces topological sorting

as an example of the application of the problem solving methodology and programming style presented in this text

background for the problem is given first

each step and decision is explained as the solution evolves

Emphasis is on

top-down design

data abstraction

functional modularization

intelligent choice of information to store and data structure in which to store it

11.1: Background

11.2: A Searching Solution

11.3: A Constructed Solution

11.4: Analysis of Topsort

11.5: Behavior for Replicated Pairs or Loops

11.6: Final Comments on Topsort

11.7: Reviewing Methodology

Exercises

Suggested Assignments


Data Structures Using C: 11.1: Background

11.1: Background

Topological sorting requires ranking a set of objects subject to constraints on the resultant topology�that is, on the placement of the objects. It occurs in many practical situations. For example, textbooks are often written so that each chapter builds on material covered earlier and cannot be understood without this base of information. Determining the order in which to present topics to ensure that all prerequisite material has been covered reduces to finding a topological sort of the topics. Other examples involving topological sorting are presented later.

The approach to the topological sorting problem presented in this chapter and the characteristics of the final program are intended to serve as prototypes. The problem is substantial and the algorithm developed important, but it is the process by which it is achieved that is most important. Developing an efficient topological sorting program is a complex process and demonstrates the general problem solving methodology used in this text. It requires the application of structured programming concepts, during which the selection of data structures plays a significant role.

The resultant topological sorting program is a model solution, representing nearly the best programmers can hope to achieve. Few solutions to problems have all its characteristics. It is concise and clear; correctness is easy to determine; and the storage and execution time requirements can be analyzed. Invalid input data are not difficult to deal with. What more can we ask?

The following are some examples of problems whose solutions require topological sorting.

Example 11.1

A large skyscraper is to be constructed. Imagine that you must write a program to prescribe the order in which all the tasks are to be performed. The tasks include T3�install windows; T50�install floors; T8�install window sashes; T1000�clear the ground; T7�build the foundation; T9�install doors. For any two tasks, it is possible to specify whether one should be done before the other. Clearly, T8 must be done before T3, and T1000 must be done before T7, while it makes no difference whether or not T3 is done before T9. The constraint that Ti be done before Tj is represented as the pair (i, j). n

Example 11.2

Imagine that you want to write a book on data structures. All technical words used must be defined. First write down all the relevant words. If word wj uses wi in its definition, represent this constraint by the pair (i, j). n

Example 11.3

As a student you must take a number of courses to graduate. Represent the fact that course cj has course ci as a prerequisite by the pair (i, j). n

Example 11.4

A number of books must be read for a course you are taking. Represent the fact that you prefer to read book i before book j by the pair (i, j). n

Suppose that an order is given in which the construction tasks of Example 11.1 are to be carried out. If you proceed to carry out the tasks, you may not be able to do the next task, since tasks that must precede it have not yet been finished. The same situation can arise in Examples 11.2, 11.3, and 11.4, when an order is specified in which words are to be defined, courses to be taken, and books to be read, respectively. This is because words used in the next word's definition have not yet been defined, all prerequisites for the next course may not have been taken, or you will not read the next available book because you want to honor your preferences. A topological sort gives an order in which to proceed so that such difficulties will never be encountered.

11.1.1 Binary Relations and Partial Orders

11.1.2 Graphic Representation of Partial Orders

11.1.3 Topological Sorts: Consistent Rankings


Data Structures Using C: 11.1.1 Binary Relations and Partial Orders

11.1.1 Binary Relations and Partial Orders

Some mathematical concepts and terminology must be defined before the topological sorting problem can be stated and solved in abstract terms. As with the solution to any problem, stating this solution abstractly is desirable so that it will not be tied to a particular application.

You readily understand what is meant by the statements: "Fred is related to Sam"; "John is taller than Sue"; "3 is the square root of 9." The phrases each specify a connection or relation between two objects. In the first two cases the relations refer to two people. In the third case the relation refers to numbers. Each is a binary relation on a set of objects.

Mathematics abstracts from such examples to define a binary relation R on a set S. S is taken as the set or collection of objects referred to by the relation. R contains information, for any two of these objects, on whether or not they are related. R is a set of pairs of objects (a, b), where a and b belong to S. If the pair (a, b) belongs to R, then this is interpreted to mean that a is related to b. This is written as aRb. If (a, b) does not belong to R, then a is not related to b. This is written as [image].

For instance, specify S to be the set of integers between 2 and 10, so S = {2, 3, . . . , 10}; and take R = {(4, 2), (6, 2), (8, 2), (8, 4), (9, 3), (10, 2), (6, 3), (10, 5)}.Then R captures the "is divided evenly by but is not equal to" relation on S, since a pair (a, b) will belong to R only if a "is divided evenly by but is not equal to" b. By looking at R, we can determine if aRb or [image] for any a and b in S.

Of particular interest are the kinds of binary relations called partial orders. A binary relation R on S is called a partial order on S if it has the following properties:

n Irreflexivity For any a in S,

n Asymmetry For any two distinct a and b in S, aRb implies

n Transitivity For any three distinct a, b, and c in S, aRb and bRc implies aRc

These properties specify that whenever certain pairs are in R, then other pairs must necessarily be included in R or excluded from R. The "is divided evenly by but is not equal to" relation on S in the preceding paragraph fulfills all three properties of a partial order.

It is irreflexive, since a is divided evenly by but is equal to a for all a in S.

It is asymmetric, since if a "is divided evenly by" b, then b cannot be divided evenly by a when a is not equal to b.

It is transitive, since whenever a is divided evenly by b and b is divided evenly by c, then a must also be divided evenly by c.

To show that a relation does not have a property requires only one counter-example.

Two extreme binary relations on a set S are the null relation, which contains no pairs at all, and the universal relation, which contains all possible pairs. The null relation is a partial order. This is because no counterexamples can be found. The universal relation will never be irreflexive (unless S has no objects in it) or asymmetric (unless S has only one object), but it will always be transitive.


Data Structures Using C: 11.1.2 Graphic Representation of Partial Orde

11.1.2 Graphic Representation of Partial Orders

One can represent any binary relation R on a set of n objects graphically, as in Figure 11.1. Let S consist of ten objects, the integers 1, 2, 3, . . . , 10. Take R to contain the pairs (1, 2), (8, 3), (7, 6), (2, 7), (5, 4), (7, 8), (2, 4), (4, 7), (2, 8), (8, 5). Create a node for each object in S and draw arrows between two nodes for each pair in R. For example, an arrow will appear from 7 to 8 for the pair (7, 8).

Any such diagram can be interpreted as representing a binary relation on a set S. For such a diagram to represent a partial order requires that

n No arrows appear from any node to itself (irreflexivity)

n No arrows appear from any node i to another node j and back from j to i (asymmetry)

n If arrows appear from i to j and from j to k, then an arrow must appear from i to k (transitivity)

In depicting partial orders, it is unnecessary to draw all the arrows needed to represent transitivity, because they are always assumed. With this understanding, it is not difficult to see that a diagram represents a partial order if and only if it contains no loops. This means that no node can be found such that one can follow a series of arrows starting from that node and ending at that node. In Figure 11.1, there is a loop involving nodes 5, 4, 7, and 8. Hence the diagram does not represent a partial order. Note that if one starts with a diagram representing a partial order (no loops), erasing a node and all arrows emanating from it results in a diagram that represents a new partial order on the set of remaining nodes.

Figure 11.1 Binary Relation R on a Set of n = 10 Objects


Data Structures Using C: 11.1.3 Topological Sorts: Consistent Rankings

11.1.3 Topological Sorts: Consistent Rankings

Partial orders are important. They are meant to capture abstractly any concrete example in which the pairs of R (or arrows of the diagram) mean "is greater than," "precedes," "is ranked higher than," or "comes before." A special case of a partial order is one whose diagram looks like the following:

3 � 5 � 1 � 2 � 4 � 6 � 8 � 7

For such a diagram, it seems clear what would be meant by a listing or ranking of the eight objects that is consistent with the given partial order. Such a ranking would be 3, 5, 1, 2, 4, 6, 8, 7. No other ranking would be consistent with the given partial order. This can be generalized to any partial order by interpreting the arrows as constraints on the ordering of objects in a ranking. For example, an arrow from 3 to 5 would mean that 3 must be ranked higher than 5. Then, a ranking consistent with a partial order would mean a ranking of the n objects that violates no constraints (that is, a ranking that violates no arrows or pairs).

Example 11.5

Consider the partial order on the set of nine integers represented by Figure 11.2. n

The ranking 2, 3, 4, 5, 6, 7, 8, 9, 10 would not be consistent with this partial order, because it violates at least one of the constraints�namely, that 6 be ranked higher than 3, or that 10 be ranked higher than 2. This diagram actually represents the "is divided evenly by but is not equal to" relation introduced earlier.

Figure 11.2 A Partial Order on the Integers 2, . . . , 9

The definition of topological sorting can now be stated more formally than at the outset of the chapter. A topological sort is a ranking of the n objects of S that is consistent with the given partial order. Let us try to solve the following topological sorting problem.

Example 11.6

Given a partial order on a set S of n objects, produce a topological sort of the n objects, if one exists. If no such ranking exists, then print out a message saying that none exists.* n

* If a relation is asymmetric and transitive but not irreflexive, then one can eliminate all pairs of the form (x, x) from the relation or, equivalently, erase all self-loops from its diagram. The resulting relation is a partial order, and it still makes sense to find a topological sort with respect to it. The topological sort then satisfies the constraints of the original relation involving distinct objects.

For simplicity assume that S is always the set of the first n integers 1, 2, 3, . . . , n, and that the partial order is given as m pairs of integers representing arrows in the corresponding diagram. Assume no pairs are replicated. For concreteness, take the input to have the following form:

10       value of n

7 3

5 3

2 10

6 8

5 6      m = 9 pairs

4 8

7 1

4 2

7 6

This particular instance of the topological sorting problem will be referred to repeatedly throughout the chapter. Is there an obvious algorithm for this problem? Try to think of one before continuing.


Data Structures Using C: 11.2: A Searching Solution

11.2: A Searching Solution

Consider the specific ranking 4, 8, 1, 2, 7, l0, 9, 5, 3, 6 of the 10 objects of the example. It is easy to determine whether or not this ranking violates any constraints of the partial order by checking each one. Since this ranking violates two of the nine constraints, 6 � 8 and 7 � 1, it is not a topological sort. Any ranking may be checked in this way. This leads to an obvious algorithm based on the generation of all rankings of the n objects.

A search-based algorithm:

If a ranking is found which violates no constraints, then the ranking is a topological sort, print it and stop. If all possible rankings have been tried and no topological sort found, then print out the message that none exists, and stop.

To implement this algorithm in a programming language requires finding a way to generate the rankings so that all the distinct rankings are eventually generated. The procedure permutations of Chapter 4 does this. Any implementation of the algorithm could take time proportional at least to the number of distinct rankings of n objects. A particular ranking can be constructed by selecting any one of the n objects to be first in the ranking, then selecting any one of the remaining n - 1 objects to be second in the ranking, and so on. Thus there are n � (n - 1) � (n - 2) � � � � � 2 � 1 distinct rankings of n objects. This product is n factorial. Consequently this solution can deal with values of n no greater than 16.* For larger n, an algorithm must be found that need not consider all rankings.

* See Section 1.8.2.


Data Structures Using C: 11.3: A Constructed Solution

11.3: A Constructed Solution

The first solution to Example 1l.6 searched through all possible rankings. Another approach is to attempt the construction of the required ranking. To this end consider a ranking, obj1, obj2, . . . , objn, of the n objects that is a topological sort with respect to a given diagram. What can be said about obj1, the highest-ranked object? Clearly there must be no objects that are constrained by the diagram to appear higher in the ranking than obj1. Otherwise, those constraints would be violated, and the ranking would not be a topological sort. In terms of the diagram, this means that no arrows must point to obj1. We describe this by saying that obj1 must have no predecessors.

Now suppose the node corresponding to obj1 is erased, as well as all the arrows emanating from obj1. As noted earlier, this results in a new diagram that represents a new partial order on the remaining n - 1 objects. Consider the ranking of these n - 1 objects obtained by taking the original topological sort of the n objects and removing obj1. That is, consider obj2, obj3, . . . , objn. It must be a topological sort of the remaining n - 1 objects in the new diagram with respect to the new partial order. This is because any constraint in the new diagram that is violated by this ranking would have been violated in the original ranking of the n objects. Note that if any topological sort of the remaining n - 1 objects were taken with respect to the new partial order, and obj1 were placed in front of it, the ranking of n objects thus obtained would be a topological sort with respect to the original partial order. Placing obj1 first in this ranking cannot violate any of the original constraints because they did not require that any object precede obj1.

We have just discovered how to construct a topological sort:

A construction-based algorithm:

While S is not empty

1. Select any object with no predecessors.

2. Place it in the next position of the output ranking.

3. Remove that object from S, and

4. Remove its arrows from the partial order.

The output ranking produced will always be a topological sort.

11.3.1 Correctness

11.3.2 An Initial Implementation

11.3.3 A Better Implementation


Data Structures Using C: 11.3.1 Correctness

11.3.1 Correctness

One difficulty remains. How can the programmer be certain the algorithm can actually be carried out? It may be impossible to carry out because at some point there may be no objects with zero predecessors. That this is indeed an algorithm requires showing that there always is at least one object with no predecessors.

One method of proof for any proposition is to assume it is not true and then show that this assumption leads to a contradiction. This method of proof is applied here. The goal is to prove that every time the loop task is to be carried out, at least one object with no predecessors can be found among those remaining. To assume that this is not true means that at some point the task cannot be carried out because every object left has at least one predecessor. Recall that the original diagram, representing a partial order, had no loops. It will be shown that the assumption implies a loop in the diagram. This will be the contradiction.

Suppose i1 is one of the remaining objects. Under the assumption, every object has at least one predecessor, so i1 has a predecessor (say, i2). Note that i2 cannot be the same object as i1, or the irreflexivity property would not hold. Again, i2 must have a predecessor (say, i3). The object i3 cannot be i1 or i2. If it were i1 or i2, then a loop involving i2 and i1 or a loop with just i2 would exist. Continuing with this argument, i1, i2, i3, i4, . . . must all be distinct objects. But since there are only n objects in total, eventually an object must repeat. This implies that the repeated object is the starting and ending object of a loop. The conclusion is that the assumption was false. Therefore what we wanted to prove must be true, and we never get bogged down�there will always be an object with no predecessors.

This algorithm yields additional insight into the topological sorting problem. Since it is clear that the loop can always be carried out to completion, it will always produce a topological sort of the n objects. In other words, a topological sort always exists. This was certainly not obvious before.

To be sure the algorithm is understood, it is applied next to Example 11.6, depicted again in Figure 11.3. Initially (see Figure 11.3(a)), 4, 7, 5, and 9 have zero predecessors.

The first step is to select and remove 5 (either 4, 7, or 9 could also have been removed) and its successor arrows, to obtain Figure 11.3(b). S is not empty, so the loop must be repeated. Now, 4, 7, and 9 have no predecessors. Select and remove 7 and its successor arrows, and obtain Figure 11.3(c). S is still not empty, and now 4, 9, 1, 3, and 6 have no predecessors. Select 4. Updating produces Figure 11.3(d).

Figure 11.3 Graphic Depiction of Example 11.6

S is still not empty, but 9, 1, 3, 6, and 2 now have zero predecessors. Continuing in this way, removing 1, 3, 6, 8, 9, 2, and finally 10 in succeeding iterations yields the final ranking: 5, 7, 4, 1, 3, 6, 8, 9, 2, 10.

Notice that every time an object is selected and removed, the diagram is updated to reflect the new S and the new partial order. As a result, some additional objects may end up with zero predecessors. Once an object has zero predecessors, it will remain in each succeeding diagram, always with zero predecessors, until it is removed.

The first refinement describes an implementation of the algorithm.

First refinement:

1. Read n.

2. Initialize for phase I.

3. While there is another input pair,

a. record the information in the pair i, j phase I

4. Initialize for phase II. phase II

5. While the set S is not empty,

a. select an object with no predecessors, and output it as the next object in the output ranking;

b. update the records to reflect the removal of the output objects.

Phase I reads into memory the information about the number of objects and the partial order. Every time an i, j pair is read in, the new information that this represents must be incorporated into a representation in memory. The "record the information" of task 3a refers to the modification of this representation to reflect the new arrow, i � j.

Phase II describes the processing required by our algorithm. Task 5a involves the selection, removal, and outputting of an object. Task 5b involves modifying the partial order representation and S, to reflect the removal of an object and its arrows, and outputting it in the ranking. After task 5b is carried out, that representation should reflect the new set S and the new partial order resulting from the object's removal.


Data Structures Using C: 11.3.2 An Initial Implementation

11.3.2 An Initial Implementation

A straightforward representation of the partial order would be to keep an image in memory of all input arrows. We might use two arrays, first and second, for this purpose. First would contain the first integer, and second would contain the second integer of an input pair. The "record the information" task would then involve putting i and j into the next available locations of first and second, respectively. For Example 11.6, after phase I was executed, the result would be the arrays shown in Figure 11.4.

Suppose the while loop of phase II is being executed, and task 5a is to be carried out. An object with no predecessors must be selected. How is such an object found? Any object that has no predecessors will not appear in the second array. Somehow the second array must be searched to select an object with no predecessors. This can be accomplished by using an array S, traversing second, and marking location i of S when object i appears in second. Any location in S left unmarked corresponds to an object with no predecessors.

Suppose this is done, and object 5 is selected. Object 5 can then be output to the next place in the ranking. To accomplish task 5b, the "updating of the records," then requires that the array first be traversed. As it is traversed, each time a 5 is encountered, the program must mark the location of first in which it appears, and the corresponding location of second. The mark signifies that the corresponding arrow emanating from 5 has been removed from the representation.

Figure 11.4 Result of Phase I of the First Refinement

This completes one execution of the loop. This same processing must be repeated for each repetition of that loop. The loop will be repeated n times, once for each of the n objects. The first and second arrays must be traversed each time. This will take time proportional to their length, which, in general, will be m. Hence this implementation of the algorithm will take time proportional to n � m. The storage required is roughly 2m locations for first and second, and n locations for S.


Data Structures Using C: 11.3.3 A Better Implementation

11.3.3 A Better Implementation

It is apparent that a lot of time is spent finding the information needed to carry out the processing in phase II. Efficiency can be improved by focusing on the following important question. Precisely what information is needed to carry out the processing?

Notice that each time through the loop, the program must select an object with no predecessors. To do this requires kdnowing which of the objects left in S have zero predecessors. Suppose that it is known, before and after each execution of the loop body, exactly how many predecessors each remaining object has. With this knowledge, an object with zero predecessor count can be selected. Once an object becomes eligible to be selected (has zero predecessors), it remains eligible until it is actually selected. Why not collect all those initially eligible in the initialization for phase II? The program could then select among them arbitrarily. Think of this collection as kept in a bag from which the object selected for output is pulled.

As the rest of the loop body is executed, the records must be updated to reflect the removal of this object and its arrows. Removing its arrows amounts to reducing the predecessor count of each successor of the object removed.Whenever the predecessor count of one of these successors becomes zero, that successor becomes eligible for selection on the next repetition of the loop. That successor is immediately added to the bag. Consequently, at the start of each loop repetition, the bag will contain exactly those objects eligible for selection, thus eliminating the need to search for them. In fact, the "S not empty" test is now equivalent to asking if the "bag is not empty." Keeping track of the successors of each object eliminates the need to search for them when that object is output. Incidently, we now know two operations that will be needed on the count collection, decrease(i,count) and iszero(i,count). The first reduces the count of the ith object by one, and the second returns true only if the count of the ith object is zero. The precise information needed for processing has now been isolated. What must be known are the predecessor counts and successors for each object, and what's in the bag.

The loop body of phase I must update predecessor counts and successors.The input pair i, j contains the information that j has one more predecessor than currently reflected in j's predecessor count, and that i has one more successor, j, than currently reflected by i's successors. The loop body of phase II must update the predecessor counts of all successors of the removed object. It must also update the bag. We have developed the following better refinement of the algorithm.

A better refinement:

1. Read n.

2. Initialize

a. Initialize the predecessor counts for each of the n objects to zero.

b. Initialize current successors for each of the n objects to zero.

3. While there is another input pair,

a. increase the predecessor count of j by l;

b. add j to the current successors of i.

4. Place all objects with zero predecessor count into the bag.

5. While the bag is not empty,

a. remove an object from the bag and output it in the output ranking;

b. for each successor of the output object, decrease its predecessor count by l; if the predecessor count becomes zero, add the successor to the bag.

The implementation treats the predecessor counts, the collections of successor counts, the bag, and the output ranking as data abstractions. In order to refine the solution further, we must decide how to store the predecessor counts, the successors of each object, the bag, and the output ranking. These are data structure decisions. They should be based on the kinds of operations to be performed in tasks 1 through 5. First we functionally modularize the solution.

1. read(n);

2. a. predinitialization(n,count);

b. succinitialization(n,successor);

3. while nextpair(i,j)

a. increase(j,count);

b. insert(i,j,successor)

4. baginitialization(bag,n,count);

5. while not emptybag(bag)

a. remove(bag,obj);

b. update(successor,obj,bag,count)

Every operation performed in tasks 1 through 5 that involves the bag, any of the n predecessor counts or collections of successors, or the output ranking has been implemented by a corresponding function. Note the use of a prefix to designate the structure involved in some of these operations. When written in this form, the program is independent of the specific details of implementation of each of these structures. This means that even if changes in the data structure implementations are ultimately adopted, this version of the solution will never need to be modified.

The next step is to decide on specific data structures for implementation of the data abstractions. This choice must be based on the functions that operate on the data structures. These functions include predinitialization,succinitialization, increase, decrease, iszero, insert, baginitialization, emptybag, and remove.

At some future time a decision may be made to change the implementation of any data abstraction. This requires finding all declarations and definitions of the data structures and functions involved and replacing them with the new versions.This modification may be made more easily and more reliably when the originals have been localized in the program. Such localization is called encapsulation. Not all high-level languages provide appropriate tools for achieving the same degree of encapsulation. In the ideal final program the data abstractions are defined in precisely one place; even more important, no operations are allowed on the corresponding data structures except those specified by the functions of the data abstraction. In the implementation above, this means, for example, that for the bag only the baginitialization, emptybag, remove, plus any other operations we take as basic to the bag will appear together. This makes it easier to determine which operations may be affected by a new implementation of the bag.The greater the encapsulation and modularization, the more expeditious the process of achieving correct modifications and adaptations. How to do this in C was discussed in Chapter 5.

We are actually in a position now to make a decision on the implementations for count, successor, and bag. The programmer does not know, in advance, the order in which objects will be output. In task 5b of the better refinement, after an object is output, its successors must be accessed and their predecessor counts updated. This means that the predecessor counts and the collections of successors must be accessed in arbitrary order. Because arrays support random access to their entries, we choose to store the predecessor counts in an array count and pointers to each collection of successors in an array succ. Count[i] and succ [i] will then contain, respectively, the predecessor count and a pointer to the successors of object i. This allows the predecessor count and collection of successors of any object to be selected, taking constant time. Since we shall need to access the collection of successors of objects, we define access_succ(i,succ) as a basic operation that returns a pointer to the collection of successors of object i.

There are n collections of successors, one for each object. These may be stored separately or may share storage. This choice has important ramifications, which will be discussed later. For now, it is best to share storage and to implement each collection of successors as a list of records. The records consist of two integer fields, succobj, containing a successor object, and link, containing a pointer to the next list record. Either dynamic memory or an array may be used to store the records. Selecting the array for storage means the programmer must manage its allocation himself or herself. Dynamic memory is probably a more natural choice, but an array of records is used to illustrate its simple management in this case and to make the later discussion comparing shared or separate storage for the list records more concrete. Lists will be the array for record storage, and a variable t will keep track of the next available record in lists for task 3b. Since only insertions will be made into the lists, the entries of lists will be allocated one after another as needed, starting with the first. Thus t must be initialized correctly.

The apparent choice for the output ranking is an integer array rank, with a variable next specifying where the next output object is to be placed in the rank array. For convenience, we use the record ranking with the two fields rank and next.

The bag could be implemented similarly, in an array with bag pointing to the next available element for a new bag entry. When an object is to be removed from the bag, we can store its value in obj and then decrease bag by 1. Thus an object can be added or removed from the bag in constant time. This implementation is permissible because objects may be selected from the bag in any order. However, implementing the bag as in Figure 11.5 affords an advantage. By allowing the bag and the output ranking to share storage, it saves time. However, this selection is made primarily to emphasize a point to be made later about the independence of modules. In order to simplify the example, only the array entries starting with 1 are used. As you know, in C they start with zero; 0th array positions will not be used in this example so that ranking and array positions will be identical.

Figure 11.5 Data Structure Implementation of Bag and Output-Ranking Data Abstractions

The X's represent nine objects placed in the rank array. Next, pointing to position 6, indicates that the next object to be selected from the bag will be the one in rank[next],rank[6] in this case. Bag, pointing to 10, indicates that when a successor is to be added to the bag, it should be placed in rank[bag],rank[10] in this case. In other words, the bag contains all objects in rank between next and bag-1.

The bag is initialized by traversing the count array and placing any object with zero count into the location of the rank array to which bag points. Bag, of course, must then be updated by 1 to move its pointer down. Bag must initially be 1, and next must initially be 1. Objects are then output in the order in which they are placed into the bag, easing processing and keeping the time constant for addition or removal of an object. Each time an object is selected, next must be increased by 1 to move it down. The bag is empty when next equals bag.

Before proceeding, phase I is illustrated by applying it to Example 11.6. For this example, count, succ, t, and list would appear as in Figure 11.6, after the first eight pairs have been input. Figure 11.7 graphically depicts the successor lists.

At the start of phase I, no information was processed yet on predecessors or successors for any object. The first pair, 7 3, contains the information that 3 has one more predecessor than was known, and that 7 has one more successor than was known, namely 3. One must be added to the count for 3, and 3 to the list of successors of 7, and so on for the other seven pairs.

Figure 11.6 Data Structures Involved in Phase I after Input of First Eight Pairs

The detailed processing of an input pair is illustrated for the last pair of this example. It is the ninth pair, 7 6. To update the count for 6, add 1 to count[6]. To create a record to contain the successor 6, place 6 into lists[9].succobj and add this new record to the current successor list for 7 by changing two pointers. First, copy the pointer from succ[7], the head of the list to which the new record is to be added, into the link field of the new record lists[9].link. Then place a pointer into succ[7] so that it points to the new record. In this case, 9 is placed into succ[7]. Since successors need not be kept in any special order, always adding records at the front of successor lists saves processing time. Adding the record anywhere else on the list would require traversal time from the head of the list to the insertion place. T must be incremented by 1 so that it points to the next available record in lists. The final situation after all nine input pairs have been read and processed is shown in Figures 11.8 and 11.9. An X indicates a null pointer.

Figure 11.7 Graphic Depiction of Successor Lists

Figure 11.8 Data Structures of Figure11.7 after Processing Last Input Pair of Phase I

We are now ready to consider the actual implementations for the functions of the data abstractions. Following is the code that might be used for those functions whose implementation may not be apparent.

Figure 11.9 Graphic Depiction of Successor Lists of Figure 11.8

increase

count[j] = count[j] + 1;

decrease

count[i] = count[i] - 1;

iszero

return(count[i] == 0);

avail

>Comment

static int t=0;

t++;

return(t);

insert

    >Comment

pointer = avail();

    >Comment

setinfo(pointer,j);

    >Comment

setlink(pointer,succ[i]);

succ[i] = pointer;

succinitialization

    >Comment

for (i=1;i<=n;i++)

succ[i] = setnull();

emptybag

    >Comment

emptybag =(bag == ranking.next);

remove

    >Comment

*pobj = ranking.rank[ranking.next];

ranking.next = ranking.next + 1;

Note that avail must correctly initialize and update t.

Task 5b of the better (or "bag") refinement of the algorithm could be done by writing the procedure update from scratch. Instead, it is done here using a tool that is available, the traverse procedure of Chapter 3. First, traverse is modified by adding bag and count as additional parameters of both traverse and process, and changing its name to update.

Process, called by update, must now be implemented so that update does its job. Its code is

    >Comment

currentsucc = info(recordpointer);

decrease(currentsucc,count);

if(iszero(currentsucc,count))

{

ranking.rank[*pbag] = currentsucc;

*pbag = *pbag + 1;

}

A complete program using topsort follows. Notice that topsort has two parameters. It is the same as our version, except it copies the rank field of ranking into the output array as a last step.

Because of the implementation, some functions must have access to specific variables. C requires these variables to be either passed as pointers or declared as global. For example, process must have access to rank, and emptybag to next. Incidentally, as written, topsort does not depend on the medium assumed for the input pairs. Normally, we want programs to be independent of the form of the input. This has been accomplished here for the pairs by using the function nextpair(i,j), which inputs the next pair and returns true if it is a nonsentinel pair. The program uses 0 0 as the sentinel pair.

    >Comment

#defineLIMIT 21

    >Comment

typedef int outputarray[LIMIT];

main()

/* Reads the number of objects n and the input pairs

specifying a partial order on the objects, and prints

a topological sort of the n objects. Each pair should

not occur more than once in the input and n should

not exceed twenty. The sentinel pair is 0 0.

*/

{

int n,i;

    >Comment

outputarray topologicalsort;

    >Comment

topsort(&n,topologicalsort);

    >Comment

printf("\n A TOPOLOGICAL SORT IS");

for (i=1;i>=n;1++)

printf("\n %d",topologicalsort[i]);

}

    >Comment

typedef int countcollection[LIMIT];

predinitialization(n,count)

/* Initializes all n counts to zero. */

int n;

countcollection count;

{

int i;

for (i=1;i<=n;i++)

count[i] = 0;

}

increase(j,count)

/* Increases the jth count by one. */

int j;

countcollection count;

{

count[j] = count[j] + 1;

}

decrease(j,count)

/* Decreases the jth count by one. */

int j;

countcollection count;

{

count[j] = count[j] - 1;

}

iszero(i,count)

/* Returns true only if the ith count is zero. */

int i;

countcollection count;

{

return(count[i] == 0);

}

    >Comment

#define RECORDLIMIT 191

/* RECORDLIMIT should be at least

(((LIMIT-1)*(LIMIT-2))/2)+1

*/

#define NULL - 1

typedef int listpointer;

typedef struct

{

int succobj;

listpointer link;

}listrecords,recordsarray[RECORDLIMIT];

    >Comment

recordsarray lists;

listpointer setnull()

/* Returns a null pointer. */

{

return(NULL);

}

anotherrecord(recordpointer)

/* Returns true only if recordpointer

points to a record.

*/

listpointer recordpointer;

{

return(recordpointer != NULL);

}

info(pointer)

/* Returns the contents of the succobj field

of the record pointed to by pointer.

*/

listpointer pointer;

{

return(lists[pointer].succobj);

}

listpointer next(pointer)

/* Returns the link field value of

the record pointed to by pointer.

*/

listpointer recordpointer;

{

return(lists[recordpointer].link);

}

setinfo(pointer,value)

/* Copies value into the succobj field

of the record pointed to by pointer.

*/

listpointer pointer;

int value;

{

lists[pointer].succobj = value;

}

setlink(pointer1,pointer2)

/* Copies pointer 2 into the link field

of the record pointed to by pointer 1.

*/

listpointer pointer1,pointer2;

{

lists[pointer 1].link = pointer2;

}

listpointer avail()

/* Returns a pointer to storage allocated

for a list record.

*/

{

static int t=0;

t++;

return(t);

}

    >Comment

typedef listpointer succ_collection[LIMIT];

insert(i,j,succ)

/* Add j into the ith collection in succ. */

int i,j;

succ_collection succ;

{

listpointer pointer,avail();

pointer = avail();

setinfo(pointer,j);

setlink(pointer,succ[i]);

succ[i] = pointer;

}

succinitialization(n,succ)

/* Initializes all n collections of succ to empty. */

int n;

succ_collection succ;

{

int i;

listpointer setnull();

for (i=1;i<=n;i++)

succ[i] = setnull();

}

listpointer access_succ(i,succ)

/* Returns a pointer to the i collection in succ. */

int i;

succ_collection succ;

{

return(succ[i]);

}

    >Comment

typedef struct

{

outputarray rank;

int next;

}rankingrecord;

rankingrecord ranking;

1 allocates storage for ranking

copy(n,topologicalsort,ranking)

/* Copies the rank field of ranking

into topologicalsort.

*/

int n;

outputarray topologicalsort;

rankingrecord ranking;

{

int i;

for (i=1;i<=n;i++)

topologicalsort[i] = ranking.rank[i];

}

    >Comment

typedef int bagcollection;

baginitialization(pbag,n,count)

/* Initializes the bag so it contains

only objects whose counts are zero.

*/

bagcollection *pbag;

int n;

countcollection count;

{

int i;

ranking.next = 1;

*pbag = 1;

for(i=1;i<=n;i++)

if(iszero(i,count))

{

ranking.rank[*pbag] = i;

*pbag = *pbag + 1;

}

}

emptybag(bag)

/* Returns true only if bag is empty. */

bagcollection bag;

{

return(bag == ranking.next);

}

remove(pbag,pobj)

/* Sets obj to an object to be removed

from bag, and removes it.

*/

bagcollection *pbag;

int *pobj;

{

*pobj = ranking.rank[ranking.next];

ranking.next = ranking.next + 1;

}

process(listname,recordpointer,pbag,count)

/* Decreases the count of the successor pointed

to by recordpointer and adds it to the bag if

its count has become zero.

*/

listpointer listname,recordpointer;

bagcollection *pbag;

countcollection count;

{

int currentsucc;

currentsucc = info(recordpointer);

decrease(currentsucc,count);

if (iszero(currentsucc,count))

{

ranking.rank[*pbag] = currentsucc;

*pbag = *pbag + 1;

}

}

    >Comment

topsort(pn,topologicalsort)

/* Inputs the number of objects n and the

partial order and generates a solution in

topologicalsort.

*/

int *pn;

outputarray topologicalsort;

{

    >Comment

countcollection count;

succ_collection succ;

bagcollection bag;

int i,j,obj;

    >Comment

printf("\n enter n between 1 and %d\n",LIMIT-1);

scanf("%d",pn);

    >Comment

predinitialization(*pn,count);

    >Comment

succinitialization(*pn,succ);

    >Comment

while (nextpair(&i,&j))

{

    >Comment

increase(j,count);

    >Comment

insert(i,j,succ);

{

    >Comment

baginitialization(&bag,*pn,count);

    >Comment

while(!emptybag(bag))

{

    >Comment

remove(&bag,&obj);

    >Comment

update(succ,obj,&bag,count);

}

    >Comment

copy(*pn,topologicalsort,ranking);

}

nextpair(pi,pj)

int*pi,*pj;

/* Input the next pair and return true

only if it was not the sentinel pair.

*/

{

print("\n enter a pair \n");

scanf("%d %d",pi,pj);

return(!((*pi == 0) && (*pj == 0)));

}

    >Comment

update(succ,obj,pbag,count)

/* Update the counts of all successors of obj

and place any whose counts become zero

into bag.

*/

succ_collection succ;

int obj;

bagcollection *pbag;

countcollection count;

{

listpointer listname,recordpointer,access_succ();

listname = access_succ(obj,succ);

recordpointer = listname;

while(anotherrecord(recordpointer))

{

process(listname,recordpointer,pbag,count);

recordpointer = next(recordpointer);

}

}

Note that the ranking and bag implementations are not independent. Thus, if it is desirable to change the implementation of one, the other is also affected. As written, the bag acts as a stack, but it may be important to pick the entry to be removed from the bag more selectively. One way to do this is to implement the bag as a priority queue. In any case, to make ranking and bag independent requires changing their declarations, definitions, and basic operations.


Data Structures Using C: 11.4: Analysis of Topsort

11.4: Analysis of Topsort

We will now analyze the time requirements of this implementation. Refer to the refinement shown in the implementation that used data abstractions on page 503. Reading n takes constant time. The count and succ array initializations take time proportional to n. The while loop read and test takes constant time. Reading the next input pair and tasks 3a and 3b, which involve executing five instructions, take constant time. Each repetition of the loop thus takes constant time. Since the loop is executed once for each of the m input pairs, the total loop time will be proportional to m. Phase I thus takes some constant time, plus time proportional to n, plus time proportional to m.

Initializing the bag at the start of phase II takes time proportional to n, since it involves traversing the count array and placing each object whose count is zero into the bag. The while loop body of phase II, unlike that of phase I, does not take the same amount of time on every repetition. Task 5a takes constant time, but task 5b depends on the number of successors of the object that was removed by task 5a. The loop itself is executed n times, once for each object output.

How can we determine the total loop time required? Notice that each of the n objects will eventually be output in some loop execution. We do not know in what order the objects will be output. However, the time taken in its particular loop execution for each object output is the same. This time is at most a constant plus time proportional to the number of successors of the object. Thus, the total time for the while loop in phase II is the sum of the time required to output each object. The total time is then a constant times n plus time proportional to the sum

[(number of successors of object 1) plus (number of successors of object 2) . . . plus the (number of successors of object n)]

This sum is just the total number of successors, m. Hence the total time for phase II is made up of the same kinds of components as the total time for phase I. We conclude that the total time for this implementation has the same form.

Note that the search-based algorithm could take time O(n!). The straight-forward implementation of the construction-based algorithm could take time O(n � m). Topsort takes time of the form c1 + c2n + c3m and represents a very substantial improvement. It was made possible because we were able to determine precisely, and to obtain efficiently, the information required to do the processing steps of the algorithm. The topological sort problem is more complex than earlier ones in this book, but it illustrates the same theme. The processing to be done determines the data structures that are most appropriate. Clearly, any implementation of an algorithm that does topological sorting must take time of the form an + bm, since it is necessary to read in all m input pairs and output all n objects.

How much storage does the implementation require? Suppose the program is to run correctly for values of n between 1 and 20. The only difficulty in determining the actual amount of storage required is in deciding what length to declare for lists. This depends on how large m may be. In general, if there are n objects, each object can have no more than n - 1 successors�that is, an arrow to every other object. There are then at most n(n - 1) possible successors. Each successor requires one record of the lists array. Actually, because of the asymmetry property, at most one-half of the n(n - 1) possibilities can appear. Hence we need a total of at most 1/2 � n(n - 1) records for lists. If n is 20, then (20 � 19)/2 records will do. The storage required is proportional to n2.

Knuth [1973a] and Wirth [1976] give different implementations of the algorithm for the topological sorting problem. See Aho, Hopcroft, and Ullman [1983] for a somewhat different point of view on its solution.


Data Structures Using C: 11.5: Behavior for Replicated Pairs or Loops

11.5: Behavior for Replicated Pairs or Loops

In the solution to the topological sorting problem, it is assumed that each distinct input pair appears only once and also that the input contains no loops. Either assumption might be violated. Of course, more obvious errors could occur. For example, n or one of the input pair members could be negative or out of range. It is not difficult to add a validation routine to check for these kinds of input errors. Subtler errors, such as replication of pairs or the occurrence of loops, are not as obviously remedied.

What does happen if pairs are replicated or loops occur in the input ? How would you recognize loops? Replication of an input pair i, j causes the count of j to be 1 larger than it should be. It also causes the successor j to appear on the list of successors of i one more time. When object i is eventually removed, its list of successors is traversed. This results in the count of j being reduced one additional time, so that the extra increase due to the i, j replication is cancelled. The implementation will thus work correctly as long as there is room for the extra successor records to appear in lists. If there is no room to accommodate these extra records, it is not possible to predict what will occur when the implementation is executed.

Loops, on the other hand, result in some objects never being output. Objects that are involved in a loop, or that are the successors of an object in a loop, will never have their counts go to zero, and will never appear in the bag of objects. In fact, had the test for completion been "S not empty" or "n objects output," loops in the input would cause an infinite loop in the program. It should now be clear that next-1, when the loop in phase II is exited, will be the actual number of objects that were output by the loop. If this is less than n, then loops occurred in the input. This can provide a simple test for loops.


Data Structures Using C: 11.6: Final Comments on Topsort

11.6: Final Comments on Topsort

As already noted, the implementation of the output ranking and the bag are coupled because they share storage. As a result they are not independent. This means that a change in the choice of implementation for one will affect the other. In general, such a situation is to be avoided�and, as pointed out earlier, could have been avoided. Certainly the small saving in time and storage did not warrant the added complexity.

There are, however, some important storage considerations that were alluded to earlier. Topsort will run out of storage before it runs out of time, since storage requirements grow as n2. The bulk of the storage needed is taken by the lists array. Suppose a maximum size, based on the available storage of the computer system, is declared for the lists array. Say its length is 50,000. Assuming each of its records takes two entries, this provides enough storage to guarantee the solution of any problem with no more than 25,000 successors (m � 25,000). It makes no difference how the successors are distributed among the objects; only the total number is relevant.

Suppose, instead, that the decision had been to represent each collection of successors by dedicating an array to the collection. Since the programmer does not know in advance how the successors will be distributed, each of the required n arrays should have the same length. Lists are no longer necessary, since the successors can be placed sequentially in the proper array. Of course, a variable will be needed to keep track of the last successor in each array. An additional array may be used for these pointers. If n is to be no greater than (say) 500, then each of the arrays containing the successors can be declared of length 50,000/500, or 100. The upshot is that the solution may now be guaranteed for a different class of problems. These problems must have n � 500, and each object may have no more than 100 successors. Even though the total number of successors may now be 50,000 (twice as many as before), their distribution is critical. When storage is at a premium, such considerations are important.

Had dynamic memory been used instead of the lists array for storage of the successor records, the program would contain no explicit storage limitation (other than the declarations for the length of count, succ, and rank). Still, the maximum dynamic memory size would have imposed a limit on the value of m for which the program would execute.


Data Structures Using C: 11.6.1 An Input Validation

11.6.1 An Input Validation

One final point involving time and storage trade-offs. Suppose an input validation must be added to topsort just prior to the processing of the current i, j pair read in phase I. The function is to check whether i, j is a duplicate of an earlier input pair. If so, it is to be ignored; if not, it is to be processed. One way to accomplish this, which requires no additional storage, is to traverse the current list of successors of object i. If j appears on the list as a successor, then i, j is a duplicate; otherwise it is not. This takes time proportional to the current number of successors of i and thus adds a total time to phase I that is proportional to m2. Instead, the function can work in constant time if n(n - 1) additional storage is available. Use the storage for an n(n - 1) array that is initialized to all zeros. When i, j is read, simply test the (i, j)th array entry. If it is zero, the pair is not a duplicate. Then set the entry to 1 to indicate that this i, j pair has been processed. In this way, an entry of 1 will mean that the pair is a duplicate from now on. This adds total time to phase I of O(m) but requires n(n - 1) additional storage.


Data Structures Using C: 11.7: Reviewing Methodology

11.7: Reviewing Methodology

A topological sort produces a ranking of objects that satisfies constraints on their allowable positions within the ranking. The obvious algorithm for finding a topological sort, searching through all rankings until one satisfying the constraints is found, is not feasible. A feasible algorithm was developed by constructing a ranking that satisfied the constraints. The initial implementation merely produced an image of the input data in memory. It was improved significantly by using an array to contain predecessor counts, creating lists of successors, and introducing a bag to hold objects whose predecessor counts became zero. This solution was achieved by a process of stepwise refinement, stressing data abstraction, encapsulation, modularity, and the proper choice of data structures. The functional modularity of the final solution allowed immediate application of a routine developed earlier�the list traversal routine. Again, because of functional modularity, it was not difficult to determine where and how to build in appropriate checks of the input data.

More generally, the analysis of the storage and time requirements of a fairly complex program was demonstrated. Even though this cannot always be accomplished, or may be difficult for complex programs, it serves as an example of what we hope to be able to do.


Data Structures Using C: Exercises

Exercises

1. Within a textbook, chapters cover specific topics. The information required to understand each topic is expected to have appeared before the topic appears in the book. Suppose you are given a list of topics and the other topics on which each depends. How might you select an ordering of the topics for their appearance in the book?

2. Which of these binary relations is a partial order on S?

a. "Square root of" on the set S of all real numbers greater than 1

b. "Is older than" on the set S of all your relatives

c. "Sits in front of" on the set S of all students in a class

d. "Lives diagonally across from" on the set S of all people

3. Write down the graphic representation for the binary relation R on the first 10 integers. R = (3, 2), (4, 6), (7, 6), (8, 1), (9, 7), (9, 8).

4. Write a detailed modification of the algorithm of Section 11.2 that will check if a given ranking is consistent with the given partial order.

5. Write a program to implement the refinement of Exercise 4 and determine its worst-case time and storage requirements.

6. Apply the construction-based algorithm of Section 11.3 to the following data to obtain a consistent ranking. Always select the smallest integer for output.

S = 1, 2, 3, . . . , 12

R = (3, 2), (4, 6), (7, 9), (12, 10), (5, 6), (2, 4), (8, 9), (2, 6), (3, 6), (7, 12), (3, 4), (7, 10)

7. If the second implementation of the construction-based algorithm is applied to the input of Exercise 6, what integers are in the bag after 3 is removed?

8. State clearly and concisely what purpose the bag serves in the better "bag" refinement and why it was introduced.

9. Write an expanded version of the better refinement that will output integers from the bag by selecting the smallest possible integer first.

10. What will the succ, lists and count arrays have in them after phase I, when the input pairs of the running example appear in the order 7 6, 5 3, 2 10, 6 8, 5 6, 7 3, 4 8, 7 1, 4 2?

11. What will the count, succ, and lists arrays look like after phase II as compared to after phase I?

12. Suppose the better implementation were modified so that no count array were stored in memory, and only the succ and lists arrays were available after phase I. Write a new phase II refinement under this constraint.

13. Suppose a solution to the topological sort problem had been constructed by determining what object should be at the bottom of the ranking. How would the better refinement be changed to reflect this new solution?

14. a. Suppose the bag is implemented in a separate record bag instead of using the ranking record. How must the data abstraction implementations change?

b. Suppose the output ranking is not kept in the ranking record but, instead, is simply printed. How must the program and data abstraction implementations change?

15. The function process is itself dependent on the ranking and bag implementations. Modify it so that process becomes independent of these implementations.

16. Suppose task 5b of the better refinement is further expanded to:

5b.i. For each successor of the output object, decrease its predecessor count by 1.

5b.ii. For each successor of the output object, if its predecessor count is zero, add the successor to the bag.

a. ow should the implementation of the data abstraction algorithm (on p. 503) be modified to reflect this refinement?

b. rite implementations for tasks 5b.i and 5b.ii that use traverse as a tool. This should include writing the corresponding two process functions.

c. ow will this expansion of task 5b affect the time required for the algorithm?

17. Suppose the input for Exercise 10 had the pairs below appended after 4 2. What would the graphic depictions of the successor lists be after phase I of topsort?

7 3, 5 6, 4 2, 7 3, 6 8

18. For the input of Exercise 17, what would be in the count array after phase I?

19. For the input of Exercise 17, show what will be in the count array, the successor lists, and the bag after 7 is output.

20. Take as input the Example 11.6 with pairs 10 4 and 10 1 appended after 4 2. What will the second implementation of topsort output in the rank array? What will be the value of next after phase II?


Data Structures Using C: Suggested Assignments

Suggested Assignments

1. Write and run a program whose input will be a series of topological sorting problems. For each problem the program should execute a slight modification of the topsort function called newtopsort. Newtopsort has parameters n and topologicalsort. Newtopsort is to read in and echo print n and all input pairs for the example it has been called to work on. It should print out the relevant portions of the count, succ and lists arrays as they appear after phase I. When newtopsort returns, the program prints the topological sort contained in the topologicalsort array. Except for the modifications needed to do the printing, newtopsort corresponds exactly to the sample implementation of topsort, with one exception: Implement the bag as an array separate from rank. Newtopsort should work correctly for valid input and for any number of objects between 1 and 20. You should make up four input sample problems to use. Two should be valid. One should include replications, and one should contain loops.

2. Suppose, instead of using the sample implementation, you do the following to keep successor information: Declare lists to be an n(n - 1) two-dimensional array and keep the successors of object i in the ith row of lists. For the example, after phase I, lists would be This two-dimensional lists array takes up the same total amount of storage as the one-dimensional lists array of the better implementation. For both implementations, the limiting factor on whether or not the program will run might then be the value of n beyond which available storage is exhausted. Suppose you are willing to give up the guarantee that topsort will always run correctly. This means you may attempt to run problems whose n is large enough that you are not sure whether the implementation of lists can hold all successors. Discuss the relative advantages of the implementations.

3. This is the same as Assignment 1 except, instead of the lists array, use dynamic memory to store the successor lists.

4. Assume your program for Assignment 1 treated the bag with the operations empty-bag, remove, and baginitialization and count with the operations predinitialization and increase as data abstractions.

a. What does this mean?

b. Write a function, number, that also treats them as data abstractions and returns the number of objects in the bag when it is invoked. When it returns, the bag must contain exactly the same objects as before number was invoked. Assume number does not have access to information about the output ranking.


Data Structures Using C: CHAPTER 12: HUFFMAN CODING AND OPTIMAL AND NE

CHAPTER 12: HUFFMAN CODING AND OPTIMAL AND NEARLY OPTIMAL BINARY SEARCH TREES

Three illustrations of the critical effect on program behavior of data structure selection are presented

Huffman coding, a text compression and coding technique, is used to show

good algorithm development

the benefits of good data structure selection

Optimal binary search trees, a related topic, are useful for storing static collections of information to be randomly accessed and traversed in sorted order; they demonstrate

the use of recursion

how to develop efficient programs based on recursion

Nearly optimal binary search trees are considered to emphasize program development

12.1: Techniques for Compressing Text or Storing Records

12.2: Weighted Path Length

12.3: Huffman Coding

12.4: Optimal Binary Search Trees

12.5: Nearly Optimal Binary Search Trees

12.6: Conclusion

Exercises

Suggested Assignments


Data Structures Using C: 12.1: Techniques for Compressing Text or Stor

12.1: Techniques for Compressing Text or Storing Records

Many important techniques are available for the compression of text. Text compression is important when dealing with large collections of information, since internal memory is scarce and access to external memory is slow. Huffman coding is an elegant method that can be used profitably when characters do not appear with equal frequency in a text. This is the case with the letters of the alphabet in English text.

Binary trees can be used as the basis for encoding text and for the creation of Huffman codes. Binary search trees can also be used to generate more quickly codes of comparable efficiency.

The application of "simple" and balanced binary search trees for record storage and retrieval by key was demonstrated in Chapter 9. As shown there, "simple" binary search trees provide excellent average search times with random input when keys are accessed equally often. Also, AVL trees guarantee excellent worst-case search time. We shall see in this chapter that optimal binary search trees are useful when the keys have known, even unequal, frequencies, and when the trees, once grown, remain unchanged. Words in English text occur with known and unequal frequencies. An optimal binary search tree could be applied, for example, as a dictionary storing the 500 most frequently used English words, along with their definitions and their common misspellings. A search of the tree would then yield the definitions and misspellings. Once built, this tree would remain unchanged.

This chapter shows that the problems of finding Huffman codes and of finding optimal or nearly optimal binary search trees are related. The solutions illustrate the use of trees, heaps, and list-structures in the construction of good algorithms. The chapter also introduces the "greedy" heuristic. This method leads to an optimal solution for finding Huffman codes and to a nearly optimal solution for finding binary search trees. In both cases, the proper selection of data structures plays a significant role.


Data Structures Using C: 12.2: Weighted Path Length

12.2: Weighted Path Length

An important measure associated with binary trees is the weighted path length. This measure underlies the entire development in this chapter. Suppose a numeric weight is assigned to each node of a binary tree T. The meaning of these weights is dependent on the intended application. When the application of the binary tree is to text compression or text encoding, only the weights assigned to terminal nodes have meaning. Each terminal node corresponds to a different character that can appear in the text. The weight assigned to the node represents the relative number of occurrences of the character in the text. When the binary tree is used for record storage, each node stores a record with a particular key. The weight assigned to a node represents the frequency of search requests for the key corresponding to the node. These weights must be estimated or determined empirically for each specific application.

The weighted path length of a node in T is the product of its depth and its assigned weight. The weighted path length of T, W(T), is the sum of the weighted path lengths of all nodes of T. As an example, consider the binary tree in Figure 12.1(a), with assigned weights indicated for each node. Its weighted path length is

(1 � 3) + (2 � 17) + (2 � 10) + (3 � 15) + (3 � 0)

 + (3 � 20) + (4 � 31) + (4 � 4)

or 302. This was calculated directly from the definition of the weighted path length.

Another way to determine the weighted path length is to determine first the value of each node of the tree. The value of a node is the sum of the assigned weights of all nodes in the subtree with that node as root. Thus the root of a tree is assigned a value that is the sum of all the weights of the tree. The example would have values assigned to each node as indicated in Figure 12.1(b). The weighted path length of the tree is then calculated by adding up the values of the nodes.Thus, from Figure 12.1(b), the weighted path length is

100 + 67 + 30 + 15 + 35 + 20 + 31 + 4 = 302

It is easy to see that adding the node values must always yield the weighted path length, since the weight of any node is counted once for every subtree in which it appears. For example, the weight 31 is counted in the value attached to each node from the node to which it is assigned to the root. This is a total of four nodes, exactly the number of times that the node's weight should count in the weighted path length.

(a) Nodes Labeled with Assigned Weights

(b) Nodes Labeled with Values Derived by Adding Assigned Weights

Figure 12.1 Binary Tree T

The weighted path length can be viewed in still another way. It is given by the value of the root plus the weighted path length of the two subtrees of the root node. For the example shown in Figure 12.1(b), this is given by 100 + 152 + 50, where 100 is the root's value and 152 and 50 are the respective weighted path lengths of the left and right subtrees. Thus there are three ways in which the weighted path length of a binary tree can be calculated.


Data Structures Using C: 12.3: Huffman Coding

12.3: Huffman Coding

With this background, let us turn to Huffman coding itself. First we will look at how information may be coded based on a binary tree and how it is coded in computers. Consider a binary tree with n terminal nodes. Assign a zero to every left successor branch and a one to every right successor branch of the tree. Encode each terminal node with the sequence of 0's and l's encountered when following the path from the root to the terminal node. Since this path is unique, the sequence or code assigned to each node must be unique. For example, in binary tree T1 (shown in Figure 12.2) the twenty-six letters of the alphabet and the space character designated by D were assigned to the terminal nodes. Thus the code for the leftmost terminal node (N) in Figure 12.2 is the sequence 00000. Any character, word, or information associated with a terminal node is encoded with the unique code for that node.

The code assigned to the letter S in T1 is the sequence 00011 of length 5, one less than the depth of the encoded node to which S is assigned. In this way, a code is automatically generated for the twenty-seven characters associated with the terminal nodes of T1. Any string of these characters can then be encoded into a string of 0's and 1's. This string is simply the sequence of the combined sequences of 0's and l's that constitute the codes for each of the characters in the string. For instance, the string of six characters "ADTREE" will be encoded

100101001100011111

of length 18. Again, because each path is unique, given any two code words, one can never be an extension of another. Consequently, no other sequence of code words will yield exactly this sequence of eighteen 0's and 1's, and any sequence starting with these eighteen 0's and 1's must start with ADTREE.

The tree T1 generates a variable-length code, since it does not assign the same length sequences of 0's and 1's to each character. A fixed-length code assigns equal-length strings of 0's and 1's to each character. A fixed-length code would be generated by a tree if each terminal node of the tree had the same depth.

Figure 12.2 Binary Tree T1

Since elements of computer memory actually store only sequences of 0's and 1's (by design), character information must be encoded. One way to encode the character set of a computer is by means of a standard fixed-length code, which uses six 0's and 1's per character.

The character set of a computer is the alphabet of characters that it can accept. For example, not all computers accept braces (that is, {}). Two important standard character sets are ASCII (American Standard Code for Information Interchange) and EBCDIC (Extended Binary Coded Decimal Interchange Codes). Both use fixed-length codes. ASCII uses a seven-bit code and EBCDIC uses eight bits. An n-bit fixed-length code can distinguish 2n characters, and a sequence of l characters would then be represented by a sequence of l � n 0's and 1's. Variable-length codes are not constrained to using the same number of 0's and l's for each character.

Using a variable-length code, a text in which the characters occur with different frequencies can be compressed. Shorter-length sequences would be assigned to characters used more frequently, and longer-length sequences to characters used less frequently. This is the idea behind the Morse code. Morse, in fact, estimated the relative frequencies of characters by looking at the frequency with which printer's boxes containing the different characters of the alphabet needed to be refilled.

The use of fixed-length codes makes it easy to decode a sequence of 0's and 1's to recover the original sequence of characters or text. It is not so easy for variable-length codes. A variable-length code must be chosen with care, if it is to allow proper decoding and avoid ambiguity. One way to achieve such a code is to use the scheme outlined for Figure 12.2, which associates a code to each binary tree. Decoding the code generated by a given binary tree would then proceed as follows:

1. Start at the root of a binary tree that is identical to the one used for encoding the original text.

2. Follow the branches according to the dictates of the sequence of 0's and l's to be decoded梘o left for a 0 and right for a 1.

3. When a terminal node is reached, the sequence of 0's and 1's that led to it is decoded as the character associated with that terminal node.

4. Start again at the root to decode the remainder of the sequence in the same way.

The relative frequency with which each of twenty-six characters of the alphabet and the blank space between words occur in English text is as follows:

 D   A   B   C   D    E   F   G   H   I   J  K  L

186  64  13  22  32  103  21  15  47  57  1  5  32

M   N   O   P   Q  R   S   T   U   V  W   X  Y   Z

20  57  63  15  1  48  51  80  23  8  18  1  16  1

These relative frequencies are used to assign weights to the nodes of a tree such as T1. Each internal node is assigned weight 0, and each terminal node is assigned the relative frequency of the character associated with it. Thus, node S of T1 is assigned weight 51. The average amount of compression is measured by the weighted path length of the tree. In this way, the amount of compression achieved with different codes can be compared, since each tree has associated with it a unique code. The average length of the assigned codes is obtained by subtracting 1 from the weighted path length when the weights are normalized to sum to 1. Weights are normalized by dividing each weight by the sum of all the weights. The designer of a compression scheme can make this calculation directly or write a program for this purpose.

The weighted path length of T1 can be improved. Take the subtree with root at depth 5, and I and S as its successors, and interchange it with T to obtain binary tree T2 (shown in Figure 12.3). The weights of I and S, 57 and 51, will now each contribute one less time to the weighted path length of T2 than to T1, while T, with weight 80, will contribute one more time to the weighted path length of T2 than to T1. The result is that the weighted path length of T2 will be (57 + 51) - 80 less than that of T1.

Figure 12.3 Binary Tree T2, Obtained by Interchanging Two Subtrees

The programmer's goal should be to find a binary tree, called the optimal binary tree, which has minimum weighted path length when the characters of the alphabet are stored at its n terminal nodes. Incidentally, the references here are to characters associated with the terminal nodes, but more generally they can be words or messages. A straightforward algorithm for finding this optimal binary tree is to generate each binary tree with n terminal nodes, associate the alphabetic characters with these n nodes, and calculate the weighted path length of the tree. A tree that yields the minimal weighted path length is optimal. Unfortunately, this algorithm is feasible only for small n. If n = 13, there are more than 1012 such trees, and the number of trees increases by a factor of almost 4 each time n increases by 1. We have to find a better way.

12.3.1 The Huffman Algorithm

12.3.2 Representation of Huffman Trees

12.3.3 Implementation

12.3.4 A Proof


Data Structures Using C: 12.3.1 The Huffman Algorithm

12.3.1 The Huffman Algorithm

An optimal binary tree generates what is called a Huffman code. Fortunately, the searching just described is not needed to find it; it can be constructed as follows.

1. Start with a collection of n trees, each with just a root. Assign an alphabetic character to each of these nodes. The weight of each node is the weight associated with its alphabetic character. The value of each of these root nodes is equivalent to its weight. (Value in this case equals the weight of the root itself, since these trees have no other nodes.)

2. Find the two trees with smallest values. Combine them as the successors of a tree whose root has a value that is the sum of their values. Remove the two combined trees and add the new tree obtained to the current collection of trees. Repeat the second step n - 2 times to obtain exactly one tree. This will be the optimal tree and will give the assignments of characters to terminal nodes.

When this algorithm is applied to the twenty-seven characters with their weights reflecting their frequency of occurrence in English, the first combination made is J with Q, since each has weight 1. The resultant tree has value 2. Next X and Z, also each of weight 1, are combined, yielding another tree with value 2. These two trees are combined, yielding a tree with value 4. The current collection now consists of twenty-four trees. The next combination involves weights 4 and 5 and yields value 9. Following this procedure, a final optimal tree that might result from the construction is shown in Figure 12.4. Other optimal trees could have been constructed, but all would have the same weighted path length of 5,124. This means that the average length (number) of 0's and 1's needed per character will be 4.124. The value 4.124 is obtained by normalizing the weighted path length (dividing it by the sum of the weights) and subtracting one. Thus 5124/1000 - 1 = 4.124. This saves almost 20 percent over a fixed-length code of length 5 (the shortest fixed-length code that can distinguish twenty-seven characters).

This construction procedure is the Huffman algorithm. A proof is given later showing that it does, in fact, produce an optimal binary tree. Notice that a cumulative total weight can be kept as the algorithm is carried out. To do this, start with the variable weight set at the sum of all the weights of the alphabetic characters. Every time two trees are combined, add the value of the resultant tree to weight. When the algorithm terminates, after making (n - 1) combinations, weight will contain the weighted path length of the optimal binary tree constructed.

The algorithm chooses the combination of current trees that yields a minimal value for the combined trees. It is in this sense that it is a "greedy" algorithm. It doesn't consider future ramifications of the current choice of subtrees to combine but takes the current two "best" subtrees. In this instance, best means "of smallest value." The algorithm makes a "local" best or greedy decision. It is an example of the greedy method of algorithm design, which involves making choices on the basis of the immediate best alternative. This may lead to later choices being constrained to such poor alternatives as to lead to an overall suboptimal result. Being less greedy now may lead to better choices later, which in turn may yield better overall results. Hence it is not obvious that taking the best possible next combination, as done here, must lead to an optimal tree.

Figure 12.4 A Final Optimal Tree

The greedy method does not yield optimal solutions in all situations. A traveler who applied the greedy method to selecting a route might always choose the next road because it is the shortest of all the current possibilities. This method will clearly not always yield the shortest overall route. When making change in U.S. currency using our system of 1-, 5-, 10-, 25-, and 50-cent pieces and dollar bills, we normally use a greedy algorithm. For instance, to give $3.70 change most people would give three $1 bills, one 50-cent piece, and two dimes (not seven 50-cent pieces, one dime, and two nickels). The algorithm is greedy because it always takes as many pieces of currency of the next largest value as possible. This always minimizes the total number of pieces of currency used to make change in our system. If a system had 1-, 5-, and 11-cent pieces and dollar bills, then making change "greedily" to give $3.70 would require three $1 bills, six 11-cent pieces, and four pennies. Fewer pieces of currency result from using three dollar bills, five 11-cent pieces, and three nickels, Fortunately, the greedy method does yield optimal binary trees.


Data Structures Using C: 12.3.2 Representation of Huffman Trees

12.3.2 Representation of Huffman Trees

Let us now consider how the binary tree constructed by the Huffman algorithm should be stored in memory. To this end we consider an example that shows that even though the weighted path length is as small as possible for a given set of weights, the depth of the tree can still be great.

Example 12.1

Suppose that the first n Fibonacci* numbers are assigned as weights to characters of an alphabet represented as terminal nodes in a tree. The task is to generate a Huffman code. n

*The Fibonacci sequence is the sequence of integers F 0, F 1, F 2, . . . that starts with F 0 = 0 and F 1 = 1, with each succeeding integer found by adding its two predecessors. Thus the sequence is 0, 1, 1, 2, 3, 5, 8, 13, 21, 34, 55, . . . , Fn�1, . . . , where F 0 = 0, F 1 = 1, Fn+2 = Fn+1 + F n, n � 0.

The Huffman algorithm would construct the tree shown in Figure 12.5. This tree has depth n. Hence the Huffman algorithm may generate trees whose depth is O(n). If a sequential representation were used for such trees, considerable storage would be wasted. For large n, the storage would not even be available. A good alternative is a linked representation of the tree.

Figure 12.5 Tree Constructed by the Huffman Algorithm for Fibonacci Weights


Data Structures Using C: 12.3.3 Implementation

12.3.3 Implementation

In this section the Huffman algorithm is implemented so that the Huffman tree and its weighted path length are generated. For a list of n records, a linked representation of the tree is illustrated in Figure 12.6(a) for n = 5 and weights 4, 3, 10, 2, 6.

The X's in the figure represent null pointers. The information field of each list record contains a pointer to a record representing the root of a binary tree. The records of the tree might be stored using pointer variables or in arrays. The records have leftptr, info, and rightptr fields. The info field contains the value assigned to the record's subtree by the Huffman algorithm. Thus each list record represents a tree. Weight is a variable initially set to the sum of the n weights and incremented by the value attached to each combined tree.

The implementation could proceed by traversing the list to find the trees with the two smallest values. These trees must be combined as dictated by step 2 of the algorithm into a new tree whose value is the sum of their values. The two trees that were combined must be deleted from the list, and the new tree added to the list. The result of these operations is shown in Figure 12.6(b) after one combination has taken place.

Every time step 2 is repeated, two trees are combined and the length of the list decreases by 1. The final list would contain one record; its information field would point to the root of the generated Huffman tree. Weight would contain its weighted path length. For this example, the resultant list would be as shown in Figure 12.6(c).

The list traversal required to find the two smallest value trees results in a worst-case time proportional to the length of the list for each traversal. Since the tree must be traversed n - 1 times, the worst-case time is O(n2). We can do better.

To improve the implementation, the two smallest values whose trees are to be combined must be found quickly without traversing the entire list. Recall that a heap, in this case a min heap, always has its smallest record at its root. Suppose the records of the binary tree are kept in a heap, or better, suppose the pointers are kept in the heap. Each pointer points to a corresponding record of the binary tree. The initial heap is shown in Figure 12.7.

(a) List of Five Records Pointing to Five Initial Trees of the Huffman Algorithm

(b) List after Combining Two Trees with Smallest Values

(c) Final List and Huffman Tree after Repeated Combinings of Values

Figure 12.6 A Linked List Representing the Trees Constructed by the Huffman Algorithm

To find the two smallest values, simply remove the top heap pointer T4, reheap, and again remove the top pointer, T2. The records corresponding to the two smallest values can be combined. The new pointer to the combined record can then be placed at the top of the heap (which was left vacant), and the heap reheaped. Figure 12.7(b) shows the heap obtained by removing T4, reheaping, and removing T2. The two trees removed are combined, and reheaping is done to obtain the heap shown in Figure 12.7(c).

(a) Initial Heap

(b) Incomplete Heap

(c)Heap after Two Trees Are Combined and Reheaping Done

Figure 12.7 Heap of Pointers of the Huffman Tree

With this implementation the computer time consumed in finding the two smallest values, combining their corresponding records, and reheaping is O(lg n). Since the entire process is repeated (n - 1) times, the total time will be O(n lg n). It was the proper choice of data structure that made this improvement possible!


Data Structures Using C: 12.3.4 A Proof

12.3.4 A Proof

For those readers who would like a proof of the Huffman construction, the following is an interesting example of the use of mathematical induction applied to trees. It can easily be adopted to prove that a recursive program for the Huffman algorithm is correct.

Assume the weights are w1, w2, . . . , wn and are indexed so that w1 � w2 � . . . � wn. You should convince yourself that an optimal binary tree must be a full binary tree.* An optimal tree can always be found, since the number of full binary trees with n terminal nodes is finite. It is clear that the Huffman construction yields optimal trees for n = 1 and 2. This is the basis for the induction, which will be on the number of weights or terminal nodes, n. The induction hypothesis is that the Huffman construction yields optimal trees for any n - 1 weights. We must show that as a result, it must generate optimal trees for n weights.

*See Exercise 5.

Given any tree with n terminal nodes, Tn, in which Figure 12.8(a) appears as a subtree, let T�n - 1 be the tree with n - 1 terminal nodes obtained from Tn by replacing this subtree by the terminal node shown in Figure 12.8(b). Then the weighted path length of Tn equals the weighted path length of T�n - 1 + (w1 + w2).

Given T�n-1, inversely, Tn can be obtained from it. Therefore, Tn must be optimal with respect to weights w1, w2 . . . , wn, if and only if T�n-1 is optimal with respect to weights (w1 + w2), w3, . . . , wn. This is because if only one tree were not optimal, it could be replaced by an optimal tree, thus improving the weighted path length of the corresponding version of the other.

By the induction hypothesis, the Huffman construction yields a tree, T�n-1, with n - 1 terminal nodes, that is optimal with respect to weights (w1 + w2), w3, . . . , wn. The corresponding tree, Tn, with n terminal nodes, is just the tree given by the Huffman construction for weights w1, w2 , . . . , wn . Since T�n-1 is optimal, so is Tn. This completes the proof.

(a) Subtree

(b)Terminal Node after Combining Weights

Figure 12.8 Subtree Replacement


Data Structures Using C: 12.4: Optimal Binary Search Trees

12.4: Optimal Binary Search Trees

So far we have been considering binary trees with nonzero weights assigned only to terminal nodes. Information (a character or message) has been associated only with their terminal nodes. We now consider extended binary search trees, which have keys stored at their internal nodes.

Suppose n keys, k1, k2, . . . , kn, are stored at the internal nodes of a binary search tree. It is assumed that the keys are given in sorted order, so that k1 < k2 < . . . < kn. An extended binary search tree is obtained from the binary search tree by adding successor nodes to each of its terminal nodes as indicated in Figure 12.9 by �'s.

Although the programming goal in this chapter is to find optimal binary search trees, extended binary search trees are used along the way. In the extended tree, the �'s represent terminal nodes, while the other nodes are internal nodes. These terminal nodes represent unsuccessful searches of the tree for key values. Such searches do not end successfully because the search key is not actually stored in the tree. They actually terminate when a null subtree pointer is encountered along the search path.

In general, the terminal node in the extended tree that is the left successor of k1 can be interpreted as representing all key values that are not stored and are less than k1. Similarly, the terminal node in the extended tree that is the right successor of kn represents all key values not stored in the tree that are greater than kn. The terminal node that is accessed between ki and ki+1 in an inorder traversal represents all key values not stored that lie between ki and ki+1. For example, in the extended tree in Figure 12.9(b), if the possible key values are 0,1, 2, 3, . . . ,100, then the terminal node labeled 0 represents the missing key values 0,1, and 2 if k1 is 3. The terminal node labeled 3 represents the key values between k3 and k4. If k3 is 17 and k4 is 21, then the terminal node labeled 3 represents the missing key values 18,19, and 20. If k6 is 90, then terminal node 6 represents the missing key values 91 through 100.

Assuming that the relative frequency with which each key value is accessed is known, weights can be assigned to each node of the extended tree. The weights represent the relative frequencies of searches terminating at each node. The weighted path length of the extended tree is then a natural measure for the average time to search the binary search tree for a key.

(a) Binary Search Tree

(b) Extended Binary Search Tree

Figure 12.9 Extension of a Binary Search Tree

Example 12.2

Find the extended binary search tree that has the minimal weighted path length. The optimal binary search tree is obtained from this tree simply by omitting the extended nodes. n

Before dealing with this problem, we compare it to the Huffman coding problem that has already been solved. When the weights attached to the internal nodes of the extended binary search tree are zero, this problem is similar to the Huffman coding problem. The difference is that the task now is to find the binary search tree with minimal weighted path length, whereas the Huffman algorithm finds the binary tree with minimal weighted path length. Since the binary search trees are only a subset of all binary trees, the Huffman tree will yield a value of the minimal weighted path length that is never larger than that of the optimal binary search tree. The Huffman algorithm will not solve the current problem unless, by chance, the tree that it constructs is a binary search tree or can be converted to one with no increase in the weighted path length. An algorithm, the Hu-Tucker algorithm, has been developed for this special case, and requires, as does the Huffman algorithm, time O(n lg n) and O(n) storage. Initially, its time was thought to be O(n2), but Knuth showed how to reduce this time by selecting the appropriate data structure. We now proceed to the general case in which the weights of the internal nodes need not be zero.


Data Structures Using C: 12.4.1 Finding Optimal Binary Search Trees

12.4.1 Finding Optimal Binary Search Trees

An obvious way to find an optimal binary search tree is to generate each possible binary search tree for the keys, calculate its weighted path length, and keep that tree with the smallest weighted path length. This search through all possible solutions is not feasible except for small n, since the number of such trees grows exponentially with n. A feasible alternative would be a recursive algorithm based on the structure inherent in optimal binary search trees. Such an algorithm can be developed in a way similar to that used for the construction of Fibonacci trees in Chapter 7.

How do we decompose the optimal search tree problem in Example 12.2 into components with the same structure? Consider the characteristics of any optimal tree. Of course, it has a root and two subtrees. A moment's reflection should convince you that both subtrees must themselves be optimal binary search trees with respect to their keys and weights. First, any subtree of any binary search tree must be a binary search tree. Second, the subtrees must also be optimal. Otherwise, they could be replaced by optimal subtrees with smaller weighted path lengths, which would imply that the original tree could not have been optimal.

Since there are n possible keys as candidates for the root of the optimal tree, the recursive solution must try them all. For each candidate key as root, all keys less than that key must appear in its left subtree, and all keys greater than it must appear in its right subtree. To state the recursive algorithm based on these observations requires some notation.

Denote the weights assigned to the n stored keys by b's and the weights assigned to the terminal nodes by a's. The weight assigned to ki is b i for i = 1, 2, . . . , n, and that assigned to the external node labeled i is a i for i = 0, 1, 2, . . . , n. Let obst(i,j) denote the optimal binary search tree containing keys ki, . . . , kj, and let w(i,j) denote its weighted path length. Obst(i,j) will involve weights ai-1, bi, . . . , bi, aj.

The optimal tree with root constrained to be kk and containing keys ki, . . . , kk, . . . , kj must then have obst(i,k-1) as its left subtree and obst(k+1,...,j) as its right subtree. Its weighted path length, w(i, j), is given by sw(i, j) + w(i, k - 1) + w(k + 1, j), where sw(i, j) is the sum of the weights ai-1, bi, . . . , bj, aj. Sw(i, j) is the value assigned to the root of obst(i,j), the sum of its weights.

Finally the algorithm can be stated.

For each k(k) as root, k = 1, 2, . . . , n

find obst(1,k-1) and obst(k+1,n).

Find a k that minimizes [w(1, k - 1) + w(k + 1, n)] over k = 1, 2, . . . , n.

Obst(1,n) is given by the tree with root kk, obst(1,k-1) as its left subtree, and

obst(k+1,n) as its right subtree.

The weighted path length of obst(1,n) is w(1, n).

w(1, n) = sw(1, n) + w(1, k-1) + w(k+1, n).

The execution tree for obst(1,4) is given in Figure 12.10. Apparently this algorithm exhibits the same proliferation of recursive calls for identical component problems as did the algorithm for the Fibonacci tree construction in Chapter 7. These are summarized in Table 12.1. This inefficiency can be avoided by using a bottom-up construction, just as was done for Fibonacci trees. To do this requires a clear understanding of exactly which optimal subtrees must be retained for later reference.

All possible optimal subtrees are not required. Those that are needed consist of sequences of keys that are immediate successors of the smallest key in the

Table 12.1 Components Replicated by Recursive Calls

Replicated Component  Occurrences

----------------------------------

 (a2, b3, a3, b4, a4)       2

 (a0, b1, a1, b2, a2)       2

 (a1, b2, a2, b3, a3)       2

 (a0, b1, a1)               4

 (a1, b2, a2)               5

 (a2, b3, a3)               5

 (a3, b4, a4)               4

 (a0)                       4

 (a1)                       3

 (a2)                       4

 (a3)                       3

 (a4)                       4

Figure 12.10 The Execution Tree for n = 4

subtree, successors in the sorted order for the keys. The final solution tree contains all n keys. There will be two with n - 1 keys, three with n - 2 keys, four with n - 3 keys, and in general k + 1 with n - k keys, for k = 0, 1, 2, . . . , n. Consequently there are a total of (n + 1) � (n + 2)/2 optimal subtrees to be retained. Notice that to determine obst(i,j) requires knowing the optimal smaller subtrees involving weights:

(ai�1) and (ai, bi+1, . . . , bj, aj)

i.e., obst(i,i�1) and obst(i+1,j)

(ai-1, bi, ai) and (ai+1, bi+2, . . . , Bj, aj)

i.e., obst(i,i) and obst(i+2,j)

(ai-1, bi, . . . , bj-1, aj-1) and (aj)

i.e., obst(i,j-1) and obst(j+1,j)

The bottom-up approach generates all smallest required optimal subtrees first, then all next smallest, and so on until the final solution involving all the weights is found. Since the algorithm requires access to each subtree's weighted path length, these weighted path lengths must also be retained to avoid their recalculation. Finally, the root of each subtree must also be stored for reference.

This algorithm requires O(n3) execution time and O(n2) storage. Knuth, who developed this solution, noticed an important fact that reduces the execution time to O(n2). He was able to show that the minimization need not be over all k from 1 to n. Instead, the minimization may be taken only over all k between r(i,j - 1) and r(i + 1,j), where r(i,j) denotes the root of obst(i,j). See Knuth [1973b].

Intuitively, this is plausible because obst(i,j-1) is the optimal binary search tree containing keys ki, . . . , kr(i,j-1), . . . , kj-1. Thus taking kr(i,j-1) as its root gives the tree just the "balance" it needs to be optimal, distributing keys to its left and right subtrees in the optimal way. Obst(i,j) includes kj as well. If its root is also kr(i,j-1), then kj will appear in the right subtree of this optimal tree adding more weight to its right subtree. Think of the optimal tree as "balanced." Its root should not be a predecessor of kr(i,j-1), since this would cause even more weight to appear in its right subtree. Reasoning this way, it is not unreasonable to expect the root of obst(i,j) to be kr(i,j-1) or one of its successors. Thus looking only at k where r(i, j - 1) � k should suffice. A symmetrical argument should make it equally reasonable that looking only at k, k � r(i + 1, j) should suffice. The actual proof is complex and adds little insight, so it is omitted here.

The bottom-up algorithm, incorporating the new minimization limits, may now be written.

1. For i from 0 to n

a. Set w(i + 1, i) to 0 and sw(i + 1, i) to a i

b. For j from i + 1 to n

Set sw(i + 1, j) to sw(i + 1, j - 1) + b j + a j

2. For j from 1 to n

a. Set w(j, j) to sw(j, j) and r(j, j) to j.

(This initializes all obst's containing 0 keys and 1 key.)

3. For l from 2 to n

a. For j from l to n,

i. set i to j - l + 1;

ii set w(i, j) to sw(i, j) + minimum[w(i, k - 1) + w(k + 1,j)], the minimum to be over k satisfying r(i, j - 1) � k � r(i + 1, j) and set r(i, j) to a value of k corresponding to the minimum.

Actually, the w(i,j)'s will differ from the weighted path length of obst(i,j) by the sum of a i-1, ai, . . . , a j. This does not affect the final tree but means that to obtain the true weighted path length, a0 + a1 + . . . + a n must be added to w(1, n).

Arrays for w, ws, and r seem to be a natural choice for implementing this algorithm and provide rapid access to the information required for the processing involved.

Example 12.3

Find the optimal binary search tree for n = 6 and weights b1= 10, b2= 3, b3= 9, b4= 2, b5= 0, b6= 10, a0= 5, a1= 6, a2= 4, a3= 4, a4= 3, a5= 8, a6= 0. n

Figure 12.11(a) shows the arrays as they would appear after the initialization, and 12.11(b) gives their final disposition. The actual optimal tree is shown in Figure 12.12. It has a weighted path length of 188.

Since this algorithm requires O(n2 ) time and O(n2) storage, as n increases, it will run out of storage even before it runs out of time. The storage needed can be reduced by almost half, at the expense of a slight increase in time, by implementing the two-dimensional arrays as one-dimensional arrays using the technique of Chapter 2. Using one-dimensional arrays may enable problems to be solved that otherwise will abort because of insufficient storage. In any case, the algorithm is feasible only for moderate values of n.

(a) Initial Array Values

(b) Final Array Values

Figure 12.11 Arrays Used to Implement the Bottom-up Recursive Algorithm for Finding Optimal Binary Search Trees

Figure 12.12 The Optimal Binary Search Tree Found by Means of the Bottom-up Recursive Algorithm

In order to understand exactly how the algorithm creates the optimal tree, you should simulate the algorithm to reproduce the arrays of Figure 12.11.


Data Structures Using C: 12.5: Nearly Optimal Binary Search Trees

12.5: Nearly Optimal Binary Search Trees

To date, no one knows how to construct the optimal binary search tree using less time or storage in the general case of nonzero a's and b's. For this reason it is important to develop algorithms that produce good, but not necessarily optimal, solutions with less time and storage being required. Algorithms have been found that construct nearly optimal binary search trees. The weighted path lengths of such trees cannot differ by more than a constant factor from the optimal value.

12.5.1 Greedy Binary Search Trees

12.5.2 Greedy Construction

12.5.3 Implementation

12.5.4 Alphabetic Codes


Data Structures Using C: 12.5.1 Greedy Binary Search Trees

12.5.1 Greedy Binary Search Trees

One such algorithm produces greedy binary search trees. These trees are constructed by an algorithm using the greedy method. Unlike the Huffman algorithm, which is also based on the greedy method, they are not guaranteed to be optimal, but they are guaranteed to be nearly optimal. Other distinct, nearly optimal, tree constructions are found in Mehlhorn [1975, 1977] and in Horibe and Nemetz [1979]. A comparison of all these and other algorithms for the generation of nearly optimal binary search trees appears in Korsh [1982]. The proof that the following greedy method yields nearly optimal trees is in Korsh [1981].

In constructing a nearly optimal binary search tree by the greedy method, the programmer must ensure that the solution is a binary search tree. Looking at the optimal tree (Figure 12.12) found by the preceding section's algorithm will provide the idea behind the greedy tree construction. Think of the tree being generated by creating the subtrees shown in Figure 12.13, one after another, starting with the subtree with k4 at its root and ending with the final optimal tree. Each subtree has its value (the sum of its weights) associated with it.

Figure 12.13 Trees Created from Subtrees (Circled Numbers Are the Trees' Values)

Each subtree is formed by taking for its root a key that has not yet appeared and adjoining a left and right subtree. These subtrees may correspond to an extended node (with weight some a i) or to an already produced subtree. The value of the subtree formed is given by the b i of its key plus the value of its left and right subtrees. These subtrees must be selected in a constrained fashion, or the resultant tree will not be a binary search tree. These constraints will be delineated shortly, but for now, the main point is that the weighted path of the optimal tree is the sum of the values of each of the subtrees. The greedy trees are built by always selecting the next subtree to generate as the one with smallest value from among all those satisfying the constraints. This is exactly how the Huffman construction works, except there are no constraints on the choice of subtrees since the resultant tree need not be a binary search tree.


Data Structures Using C: 12.5.2 Greedy Construction

12.5.2 Greedy Construction

To start the development, we use the a's and b's of Example 12.3 and array them as shown in Figure 12.14(a). In an inorder traversal of any binary search tree with the six keys, the weights assigned to the accessed nodes would occur in the order a0 b 1 a1 b2 a2 b 3, and so on. This sequence looks like triples, as shown in Figure12.14(b). A value can be associated with each triple by summing up its weights. Thus the triples just shown have values 21, 13, 17, 9, 11, and 18 respectively. A greedy tree is constructed by always finding an allowed triple with minimal value and generating its corresponding subtree next. The weighted path length of the resultant tree is the sum of the weights of the (n - 1) triples thus created. The greedy construction procedure is as follows. The allowable triples shown in the array have values 21, 13, 17, 9, 11, and 18. The triple with the lowest value (9) is

        2

     4    3

Combining the corresponding key and terminal nodes yields the subtree shown in Figure 12.14(c). Its assigned value is the triple value 9. Delete that triple from the array, and replace it by a terminal node value of 9, as shown in Figure 12.14(d).

Now repeat this process. The next minimal value is 13, corresponding to the triple This gives the subtree shown in Figure 12.14(e). The condensed array is shown in Figure 12.14(f). Repeating the process finds the minimal triple whose value is 17. The corresponding subtree is Figure 12.14(g), and the new array is shown in Figure 12.14(h). Continuing in this fashion produces the arrays shown in Figure 12.14(i), (j), and (k).

        3

     6    4

             b1     b2       b3     b4     b5      b6

             10     3        9     2      0       10

        5        6       4      4       3      8      0

        a0       a1      a2     a3      a4     a5      a6

(a) Original Array of a's and b's from Example 12.3

(b) A Sequence of Triples

(c) Subtree with Value 9

(d) First Condensed Array

(e) Subtree with Value 13

(f) Second Condensed Array

Figure 12.14 Construction of Greedy Binary Search Tree

(g) Subtree with Value 17

(h) Third Condensed Array

(i) Fourth Condensed Array

(j) Fifth Condensed Array

(k) Final Array

Figure 12.14

The greedy tree thus constructed by always taking a minimal triple is shown in Figure 12.15. Its weighted path length can be found by keeping an accumulated sum, as in the Huffman algorithm. In this case the greedy algorithm actually yields the optimal tree.


Data Structures Using C: 12.5.3 Implementation

12.5.3 Implementation

Just as in the Huffman tree construction, if we are not careful we will end up with an O(n2) time implementation. Instead, we can use a heap similar to that for the Huffman construction to obtain an O(n 1g n) time requirement and O(n) storage. However, we can do even better.

Figure 12.15 Greedy Binary Search Tree Constructed as Illustrated in Figure 12.14

The preceding method leads to a sequence of n - 1 combined triples with values v1, v2, . . . , vn-1. In this sequence v1 � v2 � . . . � vn-1. That is, the triples are generated in order, from the smallest to the highest value. The improved algorithm, introduced now, also generates a sequence of n - 1 combined triples, but their values will not necessarily be the same as v1, . . . , v2, vn-1, nor will they necessarily be generated in order by size. The improved algorithm, however, will produce the greedy tree that would have been obtained had they been combined in order by size when it is unique. A unique tree exists when there is exactly one smallest triple at each step. In general, the resultant tree will be greedy, but it will not necessarily be the same greedy tree generated by v1, . . . , v2, vn-1, nor will it necessarily have the same weighted path length.

We start with a list-structure illustrated in Figure 12.16. Each record represents a potential triple to be combined in the construction of a greedy tree. Traverse the list starting at the left and search for a locally minimal triple -- that is, a triple whose value does not exceed the value of its successor triple. No triple to the left of this locally minimal triple can be combined before it in any greedy tree construction. In addition, any future combining of locally minimal triples to its right cannot cause the resultant tree to fail to be greedy. In other words, combining the minimal triple always allows a greedy tree to be constructed with the combined minimal triple as a subtree. Combine the locally minimal triple (say it is the ith), and obtain the new list-structure shown in Figure 12.16(b).

Here a�i-1 and a�i were both set to the value of the combined triple a�i�1 + bi + a�i, the ith record was deleted, and the pointers changed as shown. The left-to-right traversal is continued, starting now at the (i - 2)th record, combining as before for each locally minimal triple encountered. Details involving the ends of the list-structure can be handled by special checking or by introducing artificial records with large weights at the ends. This results in the construction of a greedy tree. This procedure is reminiscent of the evaluation of an infix expression of Chapter 4. That algorithm is greedy in the sense that we look for the highest local priority operator to evaluate first in a left-to-right scan, evaluate it, and continue from that point.

(a) Starting List Structure

(b) List Structure after Combining the ith Locally Minimal Triple

(c) Starting List Structure for the Weights of Example 12.3

(d) Result of Combining the Triple 6 3 4

(e) Result of Combining 4 2 3

(f) Result of Combining 5 10 13

(g) Result of Combining 9 0 8

(h) Result of Combining 17 10 0

(i) Result of Combining 28 9 27

Figure 12.16 Implementation of the Greedy Algorithm

Applying this improved algorithm to Example 12.3, the starting list-structure is as shown in Figure 12.16(c). The first locally minimal triple found is 6 3 4. The list-structure resulting from combining it is shown in Figure 12.16(d). Starting again at the first record, 4 2 3 is the next minimal triple, yielding the list shown in Figure 12.16(e). Starting again, the first record here is found to be the locally minimal triple to obtain the list shown in Figure 12.16(f).

Starting again at the next first record, 9 0 8, the next locally minimal triple, gives the list shown in Figure 12.16(g). Once again, we start at the first record. The next locally minimal triple is 17 10 0. Combining it results in Figure 12.16(h). Finally the triple 28 9 27 is combined to obtain the result shown in Figure 12.16(i).

We have constructed the same tree as before (in Figures 12.14 and 12.15). Again, this need not always occur. However, even when it does, as here, the order in which the subtrees were formed need not be the same. We generated subtrees with values 13, 9, 28, 17, 27, and finally 64, rather than in order from the smallest to largest as in the other method. Both trees in this example have the same weighted path length of 188.

This implementation takes time and storage only O(n)! This was made possible by the new method, coupled with the proper choice of data structure.


Data Structures Using C: 12.5.4 Alphabetic Codes

12.5.4 Alphabetic Codes

Consider the twenty-seven characters with relative frequencies given in Section 12.3 as weights. Order them as D < A < B < C < � � � < Z. Suppose we want to construct an extended binary search tree with these characters associated with the terminal nodes. We take the weights given there as the a's of this problem and take the b's to be zero. The extended binary tree will then generate what is called an alphabetic code. The optimal binary search tree will have weighted path length 5,200, and the greedy binary search tree will have weighted path length 5,307. Recall that the Huffman tree had weighted path length 5,124. Requiring use of a binary search tree rather than a binary tree results in an increase relative to 5,124 of only (5,200 - 5,124)/5,124 or 1.5 percent. The greedy construction yields an increase relative to 5,200 of (5,307 - 5,200)/5,200 or 2 percent.


Data Structures Using C: 12.6: Conclusion

12.6: Conclusion

Binary trees are useful data structures for encoding and decoding information. The Huffman algorithm generates optimal binary trees for this purpose and can do so efficiently. Binary search trees are useful data structures for storing information that is to be searched; they also produce good codes.

Greedy binary search trees, which are nearly optimal, can be generated very efficiently. The greedy method of algorithm design does not yield optimal solutions but may still yield very good solutions. Again it was demonstrated that careful selection of data structures can significantly change the time requirements of an algorithm.


Data Structures Using C: Exercises

Exercises

1. a. Using its definition, calculate the weighted path length of the following tree. The weight of a node appears at the node.

b. Calculate the weighted path length of the tree using the other method of assigning a "value" to each node and adding up all the "values."

c. Calculate the weighted path length of the same tree by adding the sum of all its weights to the weighted path length of its left subtree plus the weighted path of its right subtree.

2. Write a process function to turn a postorder traversal into a function for determining the weighted path length of a binary tree T based on the method of Exercise 1(b).

3. Determine the Huffman code (tree) for weights 1, 2, 3, 4, 5.

4. Determine the weighted path length for your solution to Exercise 3.

5. The Huffman tree for n weights will always have (n - 1) internal nodes and n terminal nodes. This is because it is a full binary tree. That is, every internal node has exactly 0 or exactly two successors. Prove, by induction on the total number of nodes of a full binary tree, that this is true.

6. Write a nonrecursive function, huffman, with parameters n, weights, tree, and code. Weights is an array containing n weights, code is an array returned containing the n codes (sequences of 0's and 1's) assigned to each weight by the Huffman code, and tree points to the root of the corresponding Huffman tree upon return. Your function should take O(n 1g n) time.

7. Write a recursive version of the function of Exercise 6.

8. Simulate the proof (by induction on n) that the Huffman tree yields a binary tree with minimal weighted path length for n weights for Exercise 3.

9. Write a function decode with parameters n, alpha, and tree. Alpha is a sequence of n 0's and 1's, and tree points to a Huffman tree for the twenty-seven characters of Section 12.3 (determined by their weights). Decode is to decode the sequence of n 0's and 1's into the corresponding characters. The sequence represents a coded version of a message composed of the characters.

10. Suppose you know the Huffman tree for the twenty-seven characters of Section 12.3 (determined by their weights). Write a function encode to encode a message composed of characters into the Huffman code. To do this you might consider using the following data structures:

a. The Huffman tree.

b. An array with twenty-seven records kept in sorted order by character, each record containing a character and its corresponding Huffman code.

c. A hash table of records, each record consisting of a character and its Huffman code; the key is the character.

Discuss the relative advantages and disadvantages for each of these in the function encode.

11. Write a function encode to encode a message in a Huffman code.

12. Let a0 = 10, a1 = 6, a2 = 8, a3 = 4, a4 = 2, b 1 = 2, b2 = 5, b 3 = 7, b4 = 6. Determine the optimal binary search tree by generating all possible binary search trees storing the four keys with weights 2, 5, 7, and 6, and calculating their weighted path lengths.

13. The recursive approach of Section 12.4.1 is based on the idea that an optimal binary search tree must have an optimal left and an optimal right subtree. Can you use this idea to show that some of the trees you considered in Exercise 12 could have been ignored in searching for the optimal tree?

14. Simulate the bottom-up algorithm of Section 12.4.1 to reproduce the arrays of Figure12.11.

15. Why do we want to find nearly optimal binary search trees rather than optimal binary search trees for large numbers of keys?

16. Find the greedy binary search tree for the weights of Exercise 12 using the definition of a greedy tree. Determine the relative difference between its weighted path length and the optimal value.

17. Find the greedy binary search tree for the weights of Exercise 12 using the implementation of the greedy tree construction. Is its weighted path length the same as the greedy tree of Exercise 16?

18. Write a nonrecursive function greedy to implement the greedy tree construction of Section 12.5.3.

19. Give a recursive definition of a greedy binary search tree.

20. Write a recursive function greedy to implement the greedy tree construction of Section 12.5.3.

21. Another method of constructing a nearly optimal binary search tree is to select the key to be placed at the root to be a key that minimizes the difference between the weights of its left and right subtrees. Having obtained the root, apply this procedure recursively to construct its left and right subtrees. This is a top-down construction, as opposed to the greedy tree construction, which is bottom-up. What tree does it generate for the weights of Exercise 12 and for the example of Section 12.5.2?

22. Write a recursive function for the construction method of Exercise 21. How much time does your function require?

23. Can you find a way to implement the procedure of Exercise 21 in O(n) time, where n is the number of internal keys to be stored?

24. What will a greedy tree look like if the n b's are zero and the n + 1 a's are given by the first n + 1 Fibonacci numbers? Give two answers梠ne when the greedy tree definition is used, and one when the O(n) implementation is used.


Data Structures Using C: Suggested Assignments

Suggested Assignments

1. a. Write programs to find and output optimal greedy trees and their weighted path lengths.

b. Explain why the greedy tree implementation with the list structure is linear in time. (Hint: When the next locally minimal triple is to be found, only list-structure records not previously accessed will be considered, as well as two records that were previously accessed.)

2. a. Write a function to read a file composed of the twenty-six letters of the alphabet, plus the blank character, the comma, and the period; produce an output file that is the encoded version of the input file. The Huffman code is to be used, with weights based on the number of times each character appears in the input text.

b. This is the same as (a) except that the "words" of the input text are taken as the objects to be encoded, and their number of occurrences is the basis for the code. Use any reasonable definition of a word. For example, take commas, periods, blanks, and any sequence of other characters to be words.

Note: For both parts, use a function getnexttoken. In part (a) it must return the next character in the input or EOF, while in part (b) it must return the next word or EOF. Also, there must be a table in which the tokens (characters or words) are stored. This table must be searched for a token, inserted into, and its entries updated. Your function must treat the table with these operations as a data abstraction, so the solutions to both parts should look the same except that the definitions of certain functions should change from one solution to the other.

c. Run your program for part (b) with the table implemented as

i. a binary search tree

ii. a hash table using double hashing

iii. a table composed of two parts梐n optimal binary search tree storing the 25 most frequently used English words (take the a's to be zero) and a binary search storing any other words that occur in the input. The optimal binary search tree is to be built in advance of the input.

d. Write a function to read a text file consisting of the twenty-six letters of the alphabet, plus the blank and period characters, and produce an output file consisting of the text in Huffman code. Use the heap implementation in generating the Huffman tree.


Data Structures Using C: CHAPTER 13: SOME POINTERS ON STORAGE MANAGEME

CHAPTER 13: SOME POINTERS ON STORAGE MANAGEMENT

Applies the basic data structures to the management of dynamic memory

Presents alternative ways to structure the collection of unused memory locations

two-way circular lists

the buddy system

indexing methods

Discusses problems that arise when structures share storage

Explains fundamental techniques for reclaiming unused storage

Illustrates data abstraction and functional modularization in the development of three stackless binary tree traversals

Calls attention to the problems of

garbage generation

dangling references

13.1: The Need for Storage Management

13.2: The Heap or Dynamic Memory

13.3: The List of Available Space for Variable-Length Records

13.4: Shared Storage

13.5: Storage Reclamation Techniques

13.6: Stackless Traversals

13.7: Pitfalls: Garbage Generation and Dangling References

13.8: Conclusion

Exercises

Suggested Assignments


Data Structures Using C: 13.1: The Need for Storage Management

13.1: The Need for Storage Management

In languages such as C, arrays are static data structures. They do not change size or shape during the execution of a program. Chains, however, do change size, though they don't change shape. They are dynamic. List-structures, more complex lists, and binary and general trees can change both size and shape and thus are also dynamic data structures. The basic distinction between static and dynamic data structures relates to whether a fixed amount of memory is allocated in advance of program execution. Static structures can have a fixed amount of memory allocated for their sole use prior to program execution. For dynamic structures, the amount of memory required is not known in advance and thus cannot be allocated in this way.

In FORTRAN and COBOL, the total amount of storage needed by the variables of a program is known by the compiler and can be set aside before program execution. This means that a fixed amount of storage is allocated to variables for the entire execution of the program. This is not done in C, which is more versatile. It uses both static and dynamic storage allocation. Some storage is allocated for the entire execution of the program, while other storage is allocated when needed and is reused when no longer needed. This requires storage management during program execution. Dynamic structures such as lists and trees need even more complex storage management during program execution.

Consider a program that maintains a data base of stock portfolios for the clients of a brokerage house. Suppose each client's stocks are kept in a binary search tree ordered by stock name, with each node of the tree represented as a record containing information about the stock. It is not possible to predict in advance how much storage is needed, since how many stocks each client will buy is not known. Experience may show that all clients' stocks will never number more than 2,000, but the distribution of these stocks among the clients varies throughout the year. Thus it will be necessary to have storage available for 2,000 records. At a given moment this storage will be distributed among the clients' binary search trees. Whenever a stock is sold for one client, its storage must be deleted from that client's tree. It may be needed later to store the record of a stock just bought for another client. This allocation and reclaiming of record storage during program execution requires dynamic storage management. Some languages, such as C, provide needed storage management. Others do not, so the programmer must provide it.

This chapter discusses the fundamentals of storage management techniques. It offers insight into how programming languages manage storage and provides a better appreciation of the pitfalls that lie below the surface features of a language. Such insight is especially important when the programmer employs complex data structures that share storage.

You have already seen in Chapter 3 how the simple case of fixed-length records with no sharing might be handled. Problems arise when records have variable lengths and when data structures share memory. The remedies for these problems involve the creation and processing of data structures tailored to these needs. Sophisticated traversal algorithms have been developed to conserve storage. They are introduced here for three reasons. First, they are essential for the implementation of those data structures. Second, they are interesting in themselves. Finally, they serve as a good example of the use of data abstraction.

Although storage management is viewed here from the perspective of a single program's needs, these same techniques are employed by operating systems to deal with the allocation and reclamation of storage for a group of active programs.

In some problems, an upper bound on the problem size is not known in advance. One way to deal with this situation is to pick a reasonably large size and declare static data structures of this size. If the problem's memory requirement exceeds this size, an error message may be printed. When the number of available memory elements is fixed, there will generally be some problem size for which programs fail. However, it is possible that the program uses several data structures. During execution, when one data structure is large the others may be small, so that the total amount of storage required for all the structures may actually be available at all times. With static data structures, where memory is allocated and remains dedicated to a particular structure, storage must be allocated to accommodate the worst case. The worst case calls for the sum of the maximum storages needed for each data structure of the program. If, instead, the available memory can be used for the data structure that currently requires it and reclaimed for later use when that structure no longer needs it, then larger problems can be solved. This point is illustrated in the following examples.

Example 13.1

The lists array used to store successor records for the topological sort of Chapter 11 allows all successor lists to share storage. No storage is dedicated to any individual list. As long as the total number of successors on all lists does not exceed the length of lists, the program will execute correctly. This contrasts with the alternate collection of n arrays discussed in Section 11.6. With the latter approach, storage is dedicated to each list; although twice as many total successors may be accommodated, no list can contain more than its dedicated amount. n

Example 13.2

Suppose five stacks are each implemented, as in Chapter 4, using static array structures, each with a maximum size of 5,000 elements. This requires 25,000 total memory locations. However, if the number of entries in all the stacks at any one time never exceeds 10,000, proper sharing of storage would allow a reduction of up to 15,000 in the amount of storage needed. n

Example 13.3

A program can store, in linked representation, a number of binary trees that must be traversed using stacks. If the trees must be stored and traversed simultaneously, it is possible that available storage for the records of the tree nodes and for the entries of the stacks will not be sufficient for the worst case. However, if the available storage can be shared, so that it is used for tree records or for stack entries when needed, then the program might execute correctly. n

In Example 13.1, the records sharing storage all have the same size. In Example 13.2, the five stacks can all store records of the same size, or they can store records of different lengths. Example 13.3 needs shared storage for records, which may or may not be of the same length for each binary tree, and for stack entries. Normally the record lengths would be significantly greater than that of the stack entries. One fundamental distinction between storage management techniques involves whether or not the storage allocated will all be the same size. When dealing with the fixed-size case, it is convenient to use the techniques for storage allocation and storage reclamation, discussed in Chapter 3, which involve a list of available space. When dealing with more general data structures or with variable-size records, additional complications arise.


Data Structures Using C: 13.2: The Heap or Dynamic Memory

13.2: The Heap or Dynamic Memory

As one or more programs execute, they may require storage for the creation of new records, which typically are to be inserted into dynamic data structures. How can a fixed amount of storage that is made available for the creation of these records be managed efficiently?

The records, or more generally, the components of dynamic data structures, are referred to as nodes. The region of memory consisting of contiguous elements that is dedicated to the storage of these nodes is called the heap or dynamic memory. We will use the term heap in the further discussion. In the heap all nodes need not be the same length, and nodes may appear as part of more than one data structure. Associated with each node is a length field, indicating the size of the node (that is, the number of memory elements it uses). If all nodes were the same size, there would be no need to store size information within each node. All dynamic data structures reside in the heap. At all times, the heap may contain three kinds of nodes:

n Nodes that are in use because they appear in dynamic data structures

n Nodes that are not in use and appear in no dynamic data structure but are on the current list of available space

n Nodes that are not in use because they are in no dynamic data structure but are not on the list of available space

Nodes on the list of available space are said to be known. Nodes that are available but not on the list of available space are called garbage. All heads (names) of dynamic data structures, which point to their first records, are outside the heap and are kept track of in some way. This means that at all times the program knows what dynamic data structures are in existence, since their names are known.

Example 13.4

Suppose av, 1, and t are, respectively, the heads of the available space list, a list, and a binary tree. A heap containing these dynamic data structures may be visualized as in Figure 13.1. Each node has been labeled to indicate its current status:

1梚n use since the node appears on l or t

2�not in use but on the available space list

3�not in use and not on the available space list, hence garbage n

Storage allocation now involves three tasks:

1. Decide which node on the list of available space will be made available when a request for creation of a node is made.

2. Delete that node from the list of available space.

3. Set a pointer value to point to the node.

An avail-like function, discussed in Chapter 3, will perform these three tasks.


Data Structures Using C: 13.3: The List of Available Space for Variabl

13.3: The List of Available Space for Variable-Length Records

When dealing with variable-size records, which is the usual situation, storage management is complex because of fragmentation. The fragmentation problem is as follows. When all records being allocated dynamically are the same size, then as long as unused storage finds its way to the list of available space, it can be reused. However, it is possible with variable-size records that storage for a large record is needed but is not immediately in existence. Suppose each available block of consecutive elements of storage is kept on the list of available space as an individual record, with a field of each record used to store its length. It is possible for records of length 10, 15, 3, 20, and 45 to appear on the list when a record of length 65 must be created, but no individual record is large enough to be allocated for it. However, a total of 93 elements is unused. This is the external fragmentation problem. Enough storage is unused to satisfy the need, but no record currently on the list of available space is, by itself, large enough to be used. The problem may be resolved by invoking a compaction procedure, which moves the field values in used nodes to contiguous elements of memory at one end of the heap. All unused elements then also become contiguous and can represent a node large enough that the needed storage can be taken.

Figure 13.1 A Heap Containing the Available Space List (AV), a List (L), and a Binary Tree (T) and Nodes Labeled 1, 2, or 3 to Show Status

Figure 13.2 The Heap of Example 13.4 (Figure13.1) after Compaction

Example 13.5

Suppose the compaction procedure is applied to the heap of Example 13.4, which was illustrated in Figure 13.1. The result may be visualized as in Figure 13.2. n

Compaction, like garbage collection, is a time-consuming process that the programmer or computer system should only invoke infrequently. To overcome this, it is possible to organize the list of available space in an attempt to minimize fragmentation and thus the need for compaction. However, in limiting fragmentation the program should not spend too much computer time determining which node to allocate each time a request is made. Organizing the list of available space and using a reasonable policy to select the node to be allocated helps to reduce fragmentation. However, additional time is required to search the list to select the node to be allocated and to place a reclaimed node in proper position on the list to maintain its organization.

Fragmentation can also be reduced by creating larger nodes, by adding the freed storage to storage adjacent to it that is already free. This process is called coalescing. These policies and ways of organizing must be addressed by designers of compilers or operating systems and by programmers when they create storage management schemes for their own use.

Recall that the more structured or organized the data, the faster the access to required information. At the same time a price must be paid to maintain the organization. The primary considerations for a storage management scheme are the time to satisfy a request for a new node, the time to insert a node on the list of available space, the storage utilization, and the likelihood of satisfying a request. How well these characteristics turn out for a particular storage management scheme depends on the sequence of requests and returns. Mathematical analysis is generally not possible, and simulation must be used to evaluate different schemes. To provide some idea of the trade-offs and possibilities, three fundamental organizations for the available space list and appropriate allocation and reclamation policies are presented next.

13.3.1 Two-Way Circular Lists

13.3.2 The Buddy System

13.3.3 Indexing Methods


Data Structures Using C: 13.3.1 Two-Way Circular Lists

13.3.1 Two-Way Circular Lists

The simplest organization for the list of available space is a two-way circular list, with nodes kept in arbitrary order, in order by size, or in order by actual memory address. Two-way lists are convenient for making insertions and deletions. They allow the list to be traversed starting from any list record, and in either direction. A request for a node of size l then requires a traversal of the list, starting from some initial node.

Two simple policies for selection of the list node to be deleted to satisfy a request are first-fit and best-fit. First-fit selects the first node encountered whose size is sufficient. Best-fit does a complete traversal and selects that node of smallest size that is sufficient to satisfy the request. Of course, if an exact fit is found, the traversal need not continue. For both policies, any leftover storage is retained as a node on the list.

When a node is returned to the list, its neighbors (in terms of storage addresses) are coalesced with it if free, and the resultant node properly inserted on the list. All nodes require tag and size fields, with the list nodes also needing backward and forward link fields. The tag field distinguishes nodes in use from those not in use, depending on which of two values it contains.

Allowing the starting node to circulate around the list, always being set to the successor of the deleted (or truncated) node, typically improves performance.This circulation provides time for the smaller nodes left behind it to coalesce so that, by the next time around, larger nodes have been formed.

Intuitively, spending more time fitting the size of the selected node to the size of the requested node should cut down on fragmentation effects. Surprisingly, this need not always happen, but it is the normal situation.


Data Structures Using C: 13.3.2 The Buddy System

13.3.2 The Buddy System

Another organization, called the buddy system, provides faster request and return time responses than two-way circular lists. The buddy system requires the heap to be of length 2m for some integer m, occupying addresses 0 to 2m - 1. Over time, the heap is split into nodes of varying lengths. These lengths are constrained to be powers of 2, and nodes are kept on separate lists, each list containing nodes of a specific length. An array a of length m contains the heads of these lists, so a[k] points to the first node on the list corresponding to length 2k. Initially, a[m] points to the first address of the heap; all other array entries are null, since all other lists are empty.

To satisfy a request for a node of length n, k is set to �lg n�. If a[k] is not null, a node is deleted from its list to satisfy the request. Otherwise, the next nonnull list is found. Its nodes will be of length 2r, where r > k. A node is deleted from that list and split into two nodes, each of length 2r-1. That node created by the split with the greater address is placed on the appropriate list, while the other is used to satisfy the request if r - 1 = k. Otherwise it is split and processed in the same way. Eventually a node of length k will be generated to satisfy the request.

During this iterated splitting process, each time a node is split, it is split into two special nodes called buddies. Buddies bear a unique relation to each other. Only buddies are allowed to coalesce. Although this can result in unused nodes being contiguous but not coalescing because they are not buddies, it is a rare occurrence. This restriction allows very fast coalescing of buddies, as will be shown.

When a node is to be returned to the list of available space (a misnomer now), its buddy is first checked to see if it is not in use and, if it is not, the buddies are coalesced. This node is, in turn, coalesced with its buddy when possible, and so on, until no further coalescing is possible. This coalescing process is the reverse of the splitting process that created the buddies in the first place.

At this point you must be wondering how a node's buddy is found. The scheme ensures that it may be done quickly because a node of size 2k with starting address

has as its buddy a node of size 2k with starting address

In general, given the address of a node of size 2k to be returned, its (k + 1)th bit (addresses are expressed in binary notation) is changed from 1 to 0 or 0 to 1, and the result is the address of its buddy. A tag field is used to indicate whether or not a node is in use.

Example 13.6

How does the buddy system respond when m is 6 and a sequence of requests is made for nodes of size 8, 16, 4, and 6? n

(a) After a Sequence of Requests Is Made for Nodes of Size 8, 16, 4, 6

(b) After the Node of Length 16 Is Returned

Figure 13.3 Sequence of Memory Configurations for Example 13.6

The sequence of memory configurations is as shown in Figure 13.3(a). lf the node of length 16 is now returned, the configuration becomes as shown in Figure13.3(b)

A[4] is now the head of a list with two nodes, each of length 16. The released node of length 16 starting at address 16 (10000 in binary) has the node of length 16 starting at address 0 (00000 in binary) as its buddy. Since it is not free, it cannot coalesce with it. Neither can it coalesce with the node of length 4 starting at address 12, since that node is not its buddy.

The distribution of nodes can be mirrored by a binary tree. This tree is a kind of derivation tree for nodes currently in existence and expands and contracts as nodes are split and coalesced. The tree shown in Figure 13.4 reflects the configuration for Example 13.6 before the node of length 16 is returned.

Buddies appear as successors of the same parent. Thus [12, 15] and [16, 31], although contiguous in memory, are not buddies and therefore may not be coalesced.

With the buddy system, lists are never actually traversed to find storage to allocate, although time is spent finding the proper list from which to take the storage. The result is quick response to requests and returns. However, in contrast to two-way lists, this scheme produces significant internal fragmentation. Internal fragmentation occurs when allocated nodes are larger than requested, resulting in wasted storage residing within a node. This is one of the drawbacks of the buddy system. Studies have been done on similar schemes that allow a greater variety of node sizes to appear. They also tend to be fast and to reduce storage loss due to internal fragmentation.

Figure 13.4 Binary Tree Reflecting the Memory Configuration for Example 13.6 Just before Return of Node of Length 16


Data Structures Using C: 13.3.3 Indexing Methods

13.3.3 Indexing Methods

At this point each of the two approaches, circular lists and buddy systems, seems to solve half the problem. Circular lists using the best-fit method utilize storage very efficiently, causing little internal fragmentation. Buddy systems produce significant internal fragmentation but offer fast allocation by narrowly focusing the search for a proper size node. The question is whether it is possible to achieve good storage utilization and allocation time by combining their features. As you would guess, it is. It can be done by using the appropriate structures. The means to do it is called the indexing approach. The indexing approach breaks up the list of available space into collections of two-way circular lists. Each of these circular lists contains nodes of just one size. Each node on the circular list represents a block of consecutive memory locations available for the storage of a record of the size of the node. Assuming that requests for nodes of sizes (say, 1 to 4) are very frequent, an array a contains pointers to each of the four lists corresponding to these sizes. The last array entry contains a pointer to an AVL tree. The nodes of this tree are circular lists, each list corresponding to nodes of specific size that are requested much less frequently. Thus a[2] contains a pointer to a circular list containing four blocks or nodes, each of size 2. In the AVL tree pointed to by a[5], the root node is a circular list containing three blocks of size 10. The tree is ordered on the basis of node sizes. A typical configuration might be as shown in Figure 13.5.

In this way, a specific request for a node of length less than 5 can be quickly satisfied if the exact matching list is not empty. If it is empty, the next nonempty greater-size list can be used, with one of its nodes truncated to satisfy the request. Any leftover is inserted on the appropriate list. For instance, with the situation of Figure 13.5, a request for a node of size 3 will be satisfied by deleting a node of size 4 from the list pointed to by a[4]. This node will be split into a node of size 3 to satisfy the request, and the leftover node of size 1 inserted on the list pointed to by a[1]. In the infrequent case when the requested node size is 5 or greater, the AVL tree must be searched similarly. It is clear that this scheme can cause internal fragmentation, but the search times should be fast on the average. In fact the indexing approach is a compromise. It is not as fast as the buddy system but faster than simple circular lists. Likewise, the fragmentation is less than with the buddy system but more than with simple circular lists.

Figure 13.5 An Indexing Array Containing Pointers to Circular Lists and an AVL Tree Storing Circular Lists

Thus choosing the appropriate scheme of storage management is a difficult task. It requires insight into the kind of request and return distribution involved in a given application. In the worst case all schemes may fail miserably. Intuitively, good performance should result when the distribution of node sizes on the list of available space reflects the distribution of requests and returns. In turn, this distribution should be reflected in the distribution of node sizes actually in use. This is analogous to the situation mentioned in Chapter 12 for the distribution of letters in printer's boxes. The distribution of letter requests is reflected in actual text, and the printer's ordering policy for letters should have caused the contents of the letter boxes to reflect this distribution.

The three techniques considered in this section, and other techniques, are compared extensively in Knuth [1973a] and Standish [1980]. See also Tremblay and Sorenson [1984].


Data Structures Using C: 13.4: Shared Storage

13.4: Shared Storage

When components of data structures are identical and require significant amounts of storage, it is only natural to attempt to avoid component replication. This means that identical components will all share the same storage, so references to each will always be to the same place. As an example, consider Figure 13.6. The two list-structures ls1 and ls2 share a common sublist. As a result, p1, p3, and p4 point to the shared sublist.

Figure 13.6 A Shared Sublist

As usual, what we gain in one area we pay for in another. Here, storage is saved, but the price is paid in more complex processing. In particular, suppose the sublist pointed to by p3 is to be deleted from ls2 by setting p3 to null. Since p1 and p4 still point to it, it is not valid to return its records' storage to the list of available space, as would be the case without sharing. Deciding when and how to reclaim deleted records or sublists by insertion on the list of available space requires more sophisticated reclamation algorithms than those of Chapter 3.


Data Structures Using C: 13.5: Storage Reclamation Techniques

13.5: Storage Reclamation Techniques

Saying that a node or component of a data structure is to be reclaimed for use means that the storage used for its records is to become known to the avail function. That is, the storage no longer needed by its records is to be inserted on the list of available space. There are two basic methods of reclaiming released nodes, garbage collection and reference counters. Other methods are essentially combinations of these. These methods are the two extremes with respect to the time at which storage is reclaimed. Garbage collection does not make storage available for reuse until it is needed, putting off reclamation as long as possible. Reference counters make storage available for reuse as soon as it actually becomes reclaimable.

13.5.1 Garbage Collection

13.5.2 The Reference Counter Method

13.5.3 Overview of Storage Reclamation


Data Structures Using C: 13.5.1 Garbage Collection

13.5.1 Garbage Collection

The garbage collection method of reclamation is used when the available space list is empty or when the amount of storage that is known to avail becomes less than some given amount. Until that time, whenever a sublist is to be deleted from a data structure, the deletion is accomplished by updating appropriate pointers, but the storage vacated by the sublist's deletion is not made known. Initially, all records are assumed to be unmarked. The garbage collection routine works by traversing each data structure that is in use and "marking" each one of its records as being in use.

To accomplish this properly, the garbage collector program must know the names of all data structures in use. We have assumed that these are accessible through variables outside the heap. After marking, every record of the heap that is not marked will be made known (that is, placed on the available space list). Every record that is marked will be reset to unmarked so that the garbage collector will work properly on its next call. The marking phase of the garbage collector works by traversing all data structures in use. Thus marking routines generally take execution time proportional to the number of records in use and are therefore slow. Similarly, the collection routine, which inserts the unmarked records on the list of available space, takes time proportional to the heap size.

Example 13.7

Apply garbage collection to the heap of Example 13.4. n

In the marking phase, the garbage collection would traverse l and t, marking all their nodes. The collection phase would then traverse the heap and collect all unmarked nodes on the available space list. Assuming the garbage collector recognizes when two unmarked nodes are adjacent and combines them into a larger node, the resultant heap would appear as shown in Figure 13.7. A better approach might be to garbage collect and compact together, thus producing an available list consisting of one large block, reducing fragmentation.

Figure 13.7 Heap from Example 13.4 after Garbage Collection


Data Structures Using C: 13.5.2 The Reference Counter Method

13.5.2 The Reference Counter Method

The reference counter method assumes that the first record of every sublist in use has information giving the number of pointers that point to it. This number is called the reference count of the sublist. The pointers can be stored in the name variables or in sublist fields of complex records. A "smart" header record may be used for this purpose. It is assumed here that only sublists may be shared. Each time a new pointer refers to a sublist, the reference count of the sublist must be increased by 1. Each time a pointer that refers to a sublist is changed so that it no longer points to that sublist, the reference count of the sublist must be decreased by 1. After a decrease that leaves the reference count at zero, the storage for records of the sublist may be made known. Again, a data structure traversal may be used to determine which storage elements are to be made known. No traversal algorithm has been given in this text for data structures that allow sublist sharing. Such algorithms differ from the list-structure traversal algorithm because of the possibility of looping or revisiting records. As an exercise you should modify the list-structure traversals of Chapter 6 to work for this more general case. The idea is to mark each accessed record and make sure that marked records are not revisited.

Deletion of a sublist whose reference count has become zero is similar to the traversal of successor lists for an object that has been removed from the bag in the topological sort algorithm. The sublist must be traversed and, in addition to the storage for all records being inserted on the list of available space, each sublist pointed to by a record must have its reference count decremented by 1.

Example 13.8

Associate reference counts with the first record of each of the sublists of Figure 13.6. The list-structures with associated reference counts would appear as shown in Figure 13.8. n

Suppose p1 is set to null. The reference count of its sublist would be reduced to 2. Since it has not gone to zero, no storage is available for reclamation. If p6 is then set to null, the reference count of its sublist would be reduced to 0. Its storage could be reclaimed. This would require that p3's sublist have its reference count reduced by 1. The result would be as shown in Figure 13.8(b).

A problem can occur with the reference counter method whenever loops appear, as demonstrated in the following example.

Example 13.9

Consider the loop in ls1 of Figure 13.9. What happens when p1, p2, and p3 of Figure 13.6 no longer reference the sublist? Then this sublist has no variables referencing it. It cannot be accessed, since we do not know where it is. It is serving no useful purpose, yet its storage was not reclaimed, since the reference count of records 1 and 2 never went to zero. This problem is analogous to the existence of loops in the input for a topological sort. Such unused storage, or garbage, can accumulate and clog the memory. n

It is possible to share individual records as well as sublists. In this case, garbage collection will still work, as will the reference counter method. Now, however, the individual records (not just the sublists) must have reference counters associated with them, requiring additional storage and updating overhead. Loops may cause the same problems for shared records as for shared sublists .

(a) Initial List Structures with Reference Counts

(b) Final List Structures with Reference Counts

Figure 13.8 List-structures of Figure 13.6 with Associated Reference Counts

Figure 13.9 A Lost Loop


Data Structures Using C: 13.5.3 Overview of Storage Reclamation

13.5.3 Overview of Storage Reclamation

There is a spectrum of reclamation techniques, but the two basic underlying approaches are garbage collection and reference counters. Garbage collection represents one extreme. Nodes that are no longer needed in a dynamic data structure are deleted from the structure, thus becoming garbage. They are not returned at that time to the list of available space, even though they are then reusable. Reference counters, at the other extreme, return such unused nodes to the list of available space as soon as possible, when their reference counters become zero. These methods of reclamation are done implicitly by the programming language in conjunction with the operating system.

In addition, programmer control allows the programmer to use commands signaling that nodes are no longer needed by a dynamic data structure. The nodes may then be returned to the list of available space directly, left for garbage collection, or, if reference counters are used, their reference counts will be decremented and tested for possible return to the list of available space. In C the standard library function free can be used for such signaling.

In the design of a specific reclamation system, a number of trade-offs are possible. For example, reference counters require additional storage and additional time to increment, decrement, or test when insertions and deletions occur. However, they do not always work. The reason is that garbage can accumulate when loops are present in the dynamic data structures (as in Example 13.9). If the amount of garbage generated in this way is significant, it can limit the size of the problem that can be solved. Still, reference counters allow the reclamation process to be spread over time. Garbage collection, in contrast, occasionally requires a relatively large block of time. Garbage collection is a more complex technique, involving more sophisticated algorithms. Recall that when garbage collection is invoked, even though many available nodes may exist, they are not known. It is not possible simply to look at a node of the heap to determine if it is in use.

To summarize, garbage collection works in two phases. The first phase, or marking phase, requires the traversal of all dynamic data structures in existence. Recall that the heads of such structures are assumed to be known. Initially, whether in use or not, every node has a field that is marked "unused." As each node of a dynamic data structure is accessed in the traversal, it is marked "used." When the marking phase is completed, the heap then consists of nodes that are marked "unused" and nodes that are marked "used." Consequently, it is now possible to look at a node and determine its status. The second phase, or collection phase, involves a traversal through the nodes of the heap. Each node marked "unused" is inserted on the list of available space. Each node marked "used" is merely marked "unused" so that the garbage collector will work correctly the next time it is invoked.

The marking algorithms known are quite elegant. The traversal of the dynamic data structures cannot simply use a stack, since no nodes are known in which to implement the stack. Instead, these algorithms allow the traversals to be accomplished by using tag fields, usually one bit, associated with each node. It is even possible to do the traversals with no tag fields, with a fixed amount of additional storage required. Of course, if we had an infinite amount of storage available, no reclamation would be necessary. Storage management would be trivial! The next section features three traversal algorithms for use in phase I of the garbage collector, or in other applications where storage is at a premium. See Pratt [1975] for more on storage management. The algorithms to be discussed next and generalizations are also given in Standish [1980].


Data Structures Using C: 13.6: Stackless Traversals

13.6: Stackless Traversals

Garbage collection routines require the traversal of dynamic data structures that are more general than binary trees, such as list-structures and more complex lists. Algorithms for their traversal are generalizations of those we consider. For increased clarity, we will restrict our discussion to binary tree traversals and present three techniques that do not use extra storage for a stack.

These algorithms may be written assuming a particular binary tree implementation, such as arrays or pointer variables. Instead, to emphasize data abstraction and functional modularity, the following function, which is independent of the tree implementation, has been chosen. The function is a preorder traversal saving the path from the root on the stack.

preorder(pt)

/* Preorder traverses the binary tree. */

binarytreepointer *pt;

{

binarytreepointer null,p,lptr,rptr;

binarytreepointer setnull(),left(),right(),nextsubtree();

stack s;

null = setnull();

>Comment

p = *pt;

setstack(&s);

>Comment

while(p != null)

{

>Comment

process(pt,p);

>Comment

lptr = left(p);

>Comment

if(lptr != null)

{

>Comment

push(p,&s);

>Comment

p = lptr;

}

>Comment

else

{

>Comment

rptr = right(p);

>Comment

if(rptr != null)

{

>Comment

push(p,&s);

>Comment

p = rptr;

}

>Comment

else

>Comment

p = nextsubtree(&s,&p);

}

}

}

For each traversal, the basic concept is described. Then the appropriate implementation for the routines of the procedure is described.

Consider the binary tree of Figure 13.10, and suppose node.p has just been processed during the traversal using the preorder procedure. The stack will contain pointers p1, p2, p3, p4, p5, p6, and p7. At this point, the traversal of the left subtree of node.p4 has been completed. The purpose of the nextsubtree(&s,&p) function in the preorder traversal algorithms is to find node.p4, to allow the traversal to pick up at its right subtree. In general, whenever the stack is not empty, a null p value signifies the completion of the traversal of the left subtree of some node. The traversal must then pick up at that node's right subtree. There may be more than one such node; it is the most recent that must be found. The stack allows this node, and its right subtree, to be found conveniently. Nextsubtree(&s,&p) is called after the terminal node pointed to by p has been processed. It finds the nonnull right subtree that must be traversed next, returns a pointer to it, and updates the stack to reflect the path from that subtree's root to the root of the tree. If no such nonnull right subtree exists, then it returns a null value. Each of the tree traversals we now consider actually uses stack information, but it is, in effect, stored in the tree, eliminating the requirement for additional stack storage. The preorder procedure treats the tree as a data abstraction, so it is independent of the tree implementation.

Figure 13.10 A Binary Tree T

13.6.1 Threaded Trees

13.6.2 Link Inversion

13.6.3 Robson Traversal


Data Structures Using C: 13.6.1 Threaded Trees

13.6.1 Threaded Trees

A threaded tree implementation of the preorder traversal procedure is shown in Figure 13.11. It resulted from the replacement of the null left and right pointer field values of each node t by a pointer to the predecessor and to the successor, respectively, of that node in an inorder traversal of t. These pointers are called threads. The "leftmost" node of t does not have a predecessor, and the "rightmost" node of t does not have a successor. Their left and right null pointers, respectively, are left unchanged and are not considered threads. We assume that a field is associated with each node to indicate whether the node contains threads in its left link field, in its right link field, in both, or in neither.

To preorder traverse threaded trees, proceed as before, but whenever a node with threads is reached, follow its right pointer. This pointer leads directly to the node whose right subtree must then be preorder traversed. Notice, for example, that the right thread of node.p points to node.p4. The traversal is complete when a null right thread is encountered. Threaded trees may also be easily inorder and postorder traversed.

Procedure preorder may be used for the traversal of a threaded tree, with stack declared to be of type binarytreepointer, by implementing its routines as follows:

      Routine                            Task

--------------------------------------------------------------------

setnull( )          determined by the implementation (array or pointer

                  variables)

setstack(&s)       sets s to null

push(p,&s)        sets s to p

left(p)            returns a copy of leftptr.p if it is not a thread, and a

                  null value otherwise

right(p)            similar to left(p)

nextsubtree(&s,&p)  set s to p

                while (rightptr.s is a thread),

                 set s to rightptr.s

                return a copy of rightptr.s


Data Structures Using C: 13.6.2 Link Inversion

13.6.2 Link Inversion

Now consider the tree of Figure 13.12. It represents the binary tree t of Figure 13.10 during a link-inversion traversal after node.p has been processed. Notice that a path exists in the tree from predp back to the root. Each node has a tag field associated with it, and all tag values are initially zero. Preadp always points to the predecessor of p in t, and every node on the path from predp to the root has its right pointer field pointing to its predecessor if its tag field is 1, and its left pointer field pointing to its predecessor if its tag field is 0. The root of the subtree whose left subtree traversal is complete after the processing of p may be found by following the backward links from predp until a tag value of 0 is encountered. As the path back is followed, a tag of 1 at a predecessor node means we are coming from the node's right subtree. A tag of 0 means we are coming from the left subtree. Again, for node.p this leads to node.p4.

Figure 13.11 A Threaded Implementation of the Binary Tree T

Using the backward links with appropriate tag values to preserve the path to the root is the essential concept of a link-inversion traversal. During the traversal, the creation of appropriate links must be handled. For example, as the path to node.p4 is traversed, after p has been processed, the nodes with tag values of 1 must have their correct right pointer values restored. Then when node.p4 is found (tag = 0), its correct left pointer value must be restored, its tag field set to 1, its right pointer field set to its predecessor, and predp and p correctly updated before node.p4's right subtree is traversed. The traversal is complete when the backtracking leads to a null predp value. Figure 13.13 shows the state of the tree just before node.p4's right subtree is traversed. Node.p must now be processed, then its left pointer field must be set to its predecessor, predp updated to p, and p updated to the left pointer value of node.p. The traversal then continues with the left subtree of p. Initially, p must be set to t and predp to null.

Figure 13.12 A Link-inverted Binary Tree T

Procedure preorder may be used for the link-inversion traversal if stack is declared to be of type binarytreepointer, and its routines are implemented as follows.

    Routine                            Task

-----------------------------------------------------------------------------

setnull( )          determined by the implementation

setstack(&s)       set s to null

push(p,&s)          if leftptr.p is not null, then

                      set leftptr.p to s and s to p

                  else

                      set tag(p) to 1, set rightptr.p to s and s to p

left(p)            returns a copy of leftptr.p

right(p)           returns a copy of rightptr.p

nextsubtree(&s,&p)  set found to false

                while((not found) and (s not null))

                    if tag(s) = 1, then

                        set pred to s

                        set tag(s) to 0, s to rightptr.s,

                        rightptr.pred to p, and p to pred

                    else

                        set pred to leftptr.s and leftptr.s to p

                        if rightptr.s null, then

                            set found to true, tag(s) to 1, return

                            rightptr.s, and set rightptr.s to

                            pred

                        else

                            set p to s, set s to pred

                if s = null, then

                     return null.

In this implementation s plays the role of predp.

In a stack traversal, the storage required for the stack is proportional to the depth of the traversed tree. In the threaded-tree and link-inversion traversals, storage is not required for the stack but is required for the tag fields. This storage may actually be available, but unused, because of the implementation of the tree records. In this case, using it for the tag fields costs nothing. Otherwise, the additional storage is proportional to the number of nodes in the traversed tree. For the stack traversal, the constant of proportionality is given by the length of the pointer fields, while in the threaded-tree and link-inversion traversals it is given by the length of the tag fields.

The threaded-tree traversal is simpler than the link-inversion traversal and does not disturb the tree itself during the traversal. This allows more than one program to access the tree concurrently. However, the insertion and deletion of nodes in a threaded tree is more complex and time-consuming. Inorder and postorder traversals may also be done using link inversion.

Figure 13.13 New State before Subtree Pointed to by P Is Traversed in Binary Tree T


Data Structures Using C: 13.6.3 Robson Traversal

13.6.3 Robson Traversal

It is remarkable that an algorithm has been found which does not require a stack or even tag fields. This is the Robson traversal, the final stackless traversal we consider. Again, we give a preorder version, although it may be used for inorder and postorder traversals as well.

Consider the tree of Figure 13.14. In the link-inversion traversal, for every descent to the left from node.p (that is, every traversal of the left subtree of node.p), the left pointer field of node.p was set to point to node.p's predecessor. For every descent to the right (to traverse the right subtree of node.p), the leftpointer of node.p was restored to its original value, the tag of node.p was set to 1, and the right pointer field of node.p was set to point to node.p's predecessor. Consequently, having completed the traversal of the left subtree of some node q, it was possible to follow the backward pointing pointers to get to q. Q was the first node encountered with a tag value of 0, on the path back.

The Robson traversal does not have tag fields available for this purpose. Instead, a stack is kept in the tree itself, which will allow node q to be found. Q is the root of the most recent subtree, whose nonnull left subtree has already been traversed and whose nonnull right subtree is in the process of being traversed. A variable top is used, and kept updated, to point to the current node q. A variable stack is used, and kept updated, to point to the next most recent subtree whose nonnull left subtree has already been traversed and whose nonnull right subtree is in the process of being traversed. Variables p and predp are used, as in link inversion, to point to the node currently being processed and to its predecessor, respectively. Each stack entry, except the current top entry, is kept in the "rightmost" terminal node of one of the subtrees whose nonnull left subtree has been completely traversed, and whose nonnull right subtree is in the process of being traversed. In Figure 13.14, when node.p is being processed, there are four subtrees whose left subtrees have already been traversed and whose right subtrees are currently being traversed. Their roots are labeled 1 to 4 in Figure 13.14. Nodes 1, 3, and 4 have nonnull left subtrees. Each of these nodes has its left pointer field pointing to its predecessor and its right pointer pointing to the root of its left subtree. Node 2, which has a null left subtree, has its original null value, and its right pointer field is pointing to its predecessor. Nodes 1, 3, and 4 are thus the roots that must have stack entries. Top points to the most recent, node 1. Stack points to the stack entry corresponding to the next most recent root, node 3. This stack entry is kept in the rightmost terminal node of node 1's left subtree. This rightmost node has its left pointer field pointing to the node that contains the next stack entry, and its right pointer field pointing to its corresponding root (node 3 in this case). All stack entries (except the first, pointed to by top) have this same format. In Figure 13.14, the last stack entry points to node 4.

Figure 13.14 Robson Traversal Configuration of the Binary Tree T

Notice that a path back to the root always exists from node.predp. In Figure 13.14, after node.p is processed, the traversal of node 1's right subtree has been completed. This also completes the traversal of the right subtrees of nodes 2 and 3 and the left subtree of node 3's predecessor. The Robson traversal now proceeds by following the backward path, restoring appropriate left or right subtrees along the way, until the predecessor of node 3 is encountered. At this point, its right subtree must be traversed. Figure 13.15 shows the situation just before that traversal . During the traversal of the backward path, if node.predp has a null right subtree, or top=predp, then we are coming from the left subtree of node.predp, otherwise from the right subtree. Thus the stack entries allow node q to be found even though no tag fields are used.

Figure 13.15 Robson Traversal before P Is Processed in the Binary Tree T

During the backtracking, the stack is popped, and top is updated as necesssary. Top and stack are initialized to null, predp to null, and p to t. An additional variable avail is used and updated to point to a terminal node every time one is encountered, so that its pointer fields are available if a new stack entry must be created. In this case, the available node must retain a pointer to node.top in its right pointer field and a pointer to the current entry to which stack points in its left pointer field; stack must be updated to point to the new entry; and top must be updated to point to the new top root. At all times, the path from predp back to the root is stored in the tree, as well as the stack of entries pointing to roots of subtrees (with nonnull left subtrees), whose left subtree traversals have completed, and whose right subtrees are being traversed. Each node along the path from predp to the root is in one of the following states:

If the node's left subtree is being traversed, then

its left pointer is pointing to its predecessor, and its right pointer is undisturbed.

If the node's right subtree is being traversed, then

if the node has a null left subtree, then

its right pointer is pointing to its predecessor

else

its left pointer is pointing to its predecessor, and its right pointer is pointing to

the root of the node's left subtree.

The following routines are to be used with the preorder traversal so that it implements the Robson traversal. Stack is declared to be of the same type as binarytreepointer, and s plays the role of predp.

   Routine                                   Task

-----------------------------------------------------------------------------

setnull( )           determined by the implementation

setstack(&s)        sets top, stack, and s to null

push(p,&s)         if leftptr.p is not null then

                     set leftptr.p to s and s to p

                 else

                     set rightptr.p to s and s to p

left(p)             returns a copy of leftptr.p

right(p)            returns a copy of rightptr.p

nextsubtree(&s, &p)  set avail to p

                     set found to false

                     while((not found) and (s not null))

                         if top = s, then

                             save stack in hold, pop the stack by setting

                             top to rightptr.stack, stack to leftptr.

                             stack, and leftptr.hold and

                             rightptr.hold to null. Save leftptr.s in

                             pred, restore leftptr.s by setting it to

                             rightptr.s, restore rightptr.s by setting it

                             to p, set p to s, and s to pred

                         else

                             if leftptr.s = null, then

                                 save rightptr.s in pred, restore

                                 rightptr.s by setting it to p, set p to s,

                                 and s to pred

                             else

                                 if rightptr.s null, then

                                     push an entry for s onto the stack by

                                     setting leftptr.avail to stack,

                                     rightptr.avail to top, stack to

                                     avail, and top to s. Return the value

                                     of rightptr.s, set rightptr.s to p,

                                     and set found to true

                                 else

                                     save leftptr.s in pred, restore

                                     leftptr.s by setting it to p, set p to s,

                                     and s to pred.

                     if s = null, then

                          return null.


Data Structures Using C: 13.7: Pitfalls: Garbage Generation and Dangli

13.7: Pitfalls: Garbage Generation and Dangling References

This chapter has shown how a list of available space can be maintained to keep track of storage known to be currently unused. Storage may be allocated dynamically, during program execution, by deleting appropriate elements from this list when needed.

The need and means for storage reclamation have also been discussed. Garbage collection and reference counter techniques (or some combination) may be used for this purpose. When a node of a binary tree, a list, or other dynamic data structure is not needed, the programmer frequently knows at what point this occurs. Some high-level languages allow commands to be used by the programmer to signal this occurrence. For example, C provides the function free. Like malloc (and other allocation functions), its parameter will be a pointer type variable. When p points to a node that the programmer knows is no longer needed, free(p) may be invoked. This signals that the storage pointed to by p may be relinquished and reclaimed for use for another dynamically allocated data structure. Exactly how free uses this information is determined by the compiler.

High-level languages are usually designed with particular storage management schemes in mind. However, the language specifications, or standards, do not normally specify that these schemes must be used. It is possible for a compiler to use garbage collection or reference counters, to leave the reclamation entirely to the programmer, or to use some combination. If necessary programmers may even write their own programs to manage storage. We will not discuss these problems further except to illustrate two potential dangers.

Consider the following disaster segment as an illustration of these dangers.

(1) p = q;

(2) r = malloc ( sizeof(binarytreerecord));

(3) r -> leftptr = tree;

(4) free(tree);

(5) process(&s);

(6) s -> leftptr = lefttree;

Suppose p, q, r, s, lefttree, and tree are all pointer variables pointing to nodes of the same type. Before the segment is executed, the situation might be as shown in Figure 13.16(a). Statement (1), when executed, results in the situation sketched in Figure 13.16(b).

The tree, pointed to by p originally, now has no pointer pointing to it. Hence there is no way the program can access any of its nodes. Also, the storage allocated to this tree, while no longer in use, has not been returned to the list of available space. Unless the storage management technique uses garbage collection or reference counters, that storage is lost. It cannot be accessed and will never be returned to the list of available space. It truly has become garbage. If free(p) had been invoked before statement (1), this might have been avoided, depending on whether free recognizes that the other nodes of the tree may also be returned. If free merely returns the storage for the node to which p points, this could have been avoided only by the programmer traversing the tree pointed to by p and "freeing" each of its nodes.

Suppose, instead, that free uses reference counters and, recognizing that the tree pointed to by p has no other references, returns its storage to the list of available space. Now statement (2) is executed. The result is shown in Figure 13.16(c).

After statement (4) is executed, assuming that free simply returns the storage allocated to the node pointed to by tree to the list of available space, we get Figure 13.16(e). However, the node pointed to by r�leftptr is now on the list of available space. R�leftptr is called a dangling reference because it points to a node that is on the list of available space. When process is invoked in statement (5), it is possible that it invokes malloc, so storage is allocated to s. Then we have the situation shown in Figure 13.16(f).

(a) Initial Situation

(b) Result of Statement (1)

(c) Result of Statement (2)

(d) Result of Statement (3)

(e) Result of Statement (4)

(f) Result of Statement (5)

(g) Result of Statement (6)

Figure 13.16 Simulation of the Disaster Segment

Statement (6) produces Figure 13.16(g). The binary tree that r now points to has been changed, because of the dangling reference. The programmer, unaware that this has occurred, may find the program's output incorrect and be unable to determine what went wrong. This is especially difficult to discover, since the next time the program executes, process may not need to invoke malloc. Even if it does, the storage allocated by malloc to s might not cause this problem. Still, the programmer may not realize that an error has occurred.

Of the two problems, garbage generation and dangling reference generation, the latter is more serious. Garbage generation may cause longer execution time, or, at worst, program failure. Dangling references, however, may cause serious errors undetected by the programmer.


Data Structures Using C: 13.8: Conclusion

13.8: Conclusion

Dynamic data structures require dynamic storage allocation and reclamation. This may be accomplished by the programmer or may be done implicitly by a high-level language. It is important to understand the fundamentals of storage management because these techniques have significant impact on the behavior of programs.The basic idea is to keep a pool of memory elements that may be used to store components of dynamic data structures when needed.

Allocated storage may be returned to the pool when no longer needed. In this way, it may be used and reused. This contrasts sharply with static allocation, in which storage is dedicated for the use of static data structures. It cannot then be reclaimed for other uses, even when not needed for the static data structure. As a result, dynamic allocation makes it possible to solve larger problems that might otherwise be storage-limited.

Garbage collection and reference counters are two basic techniques for implementing storage management. Combinations of these techniques may also be designed. Explicit programmer control is also possible. Potential pitfalls of these techniques are garbage generation, dangling references, and fragmentation.

High-level language may take most of the burden for storage management from the programmer. The concept of pointers or pointer variables underlies the use of these facilities, and complex algorithms are required for their implementation.


Data Structures Using C: Exercises

Exercises

1. a. Consider the Suggested Assignments of Chapter 11. Suppose only 200 memory elements are available for lists to store successors. Given an example of a problem with fewer than twenty objects for which the implementation of Assignment 1 will work but the implementation of Assignment 2 will fail. Assume count and succ have length 20, the lists array of (1) has length 200, and the lists array of (2) is 20 � 10.

b. Can you find such an example for which Assignment 2 works but Assignment 1 fails?

2. a. Consider two implementations. First, a total of 50 memory elements in an array are to be dynamically allocated and reclaimed so that a binary tree (in linked representation) as well as a stack may be stored in the array. Second, an array of length 35 is to hold the binary tree, and another array of length 15 is to implement the stack. Each array is dynamically managed. The binary tree is to be preorder traversed using the stack. Give an example of a binary tree, with fewer than 35 nodes, that may be successfully traversed with the first implementation but not the second.

b. Suppose that after a node is accessed and processed, its storage may be reclaimed. Give an example of a binary tree with fewer than 35 nodes that may be successfully traversed with the first implementation but not the second.

3. In Exercise 1, the lists array of (1) functioned as a heap. Why was its storage management so simple?

4. In Exercise 2, the arrays functioned as heaps. Why was the first implementation able to do traversals that the second could not do?

5. In the first implementation of Exercise 2(b), suppose that, after a node is accessed and processed, it is simply put onto the available space list.

a. Is the head of the tree a dangling reference?

b. Are fixed-size or variable-size nodes being dealt with?

c. After your answer to Exercise 2(b) is traversed, what will be garbage, and what will be on the list of available space?

6. This is the same as Exercise 5, except that, after a node is accessed and processed, it is not put onto the available space list. After the binary tree has been traversed, the head of the tree is set to null.

7. Why is compaction not necessary with fixed-size nodes?

8. Suppose Example 13.3 deals with three binary trees. Each node has an integer valued info, leftptr, and rightptr field. The stack needs elements that are also integer valued. An integer array of length l is used as a heap in which nodes of the binary trees and the stack are stored. Assume one additional field is used for each node to indicate its length. Design a storage allocation scheme and write an appropriate avail function.

9. a. Suppose in Exercise 8 that when a node of a binary tree is deleted, its predecessor node will be changed to point to the left successor of the deleted node. Write a function delete that carries out this deletion and also returns the storage used by the deleted node to a list of available space.

b. How would the delete function change if reference counters were used as a reclamation scheme?

10. Write avail for the two-way circular list implementation for

a. Randomly ordered list entries.

b. List entries in order by size

c. List entries in order by address

11. Write avail for the buddy system implementation.

12. Write a reclaim function for the buddy system.

13. Why will garbage collection take a longer time when many nodes are in use and a shorter time when few nodes are in use?

14. What are the disadvantages of reference counters compared to garbage collection?

15. Write a function to traverse a list-structure with shared sublists to reclaim storage of a sublist whose reference count has gone to zero.

16. Write a function to insert a node in a threaded binary tree t. The node is to be inserted as the node accessed next in an inorder traversal of t after node.pred is accessed.

17. Write a function to delete a node in a threaded binary tree t. The deleted node is to be node.p.

18. Write a function to return a pointer to the predecessor of node.p in a threaded binary tree t.

19. Write a function to read information into the info fields of a binary tree t, implemented using pointer variables. The info fields will contain character strings of up to twenty characters. The input is given in the order that the info fields would be accessed in a preorder traversal of t.

20. How will your solution to Exercise 19 change if the binary tree t is implemented as a threaded tree?

21. Implement the transformation of Section 7.6.1 using a list implementation for the stack. The binary tree is implemented using pointer variables as a threaded tree. Assume the general tree is given by a two-dimensional array of size 50 � 50. If the [i,j] entry of the array is 1, then node j is a successor of node i; if the [i,j] entry is 0, then node j is not a successor of node i.

22. This is the same as Exercise 21 except that the general tree will be given by a list of successors for each node. These lists should be pointed to by pointer variables, one for each node of the general tree. The binary tree should be a link-inversion tree.

23. Suppose input, representing a general tree, is given as a sequence of pairs. A pair i, j implies that node j is a successor of node i. A sentinel pair has its i, j values equal. For example, the general tree of Figure 7.13 might be input as B,F A,C D,J I,K I,L C,H B,E I,M A,D D,I A,B B,G X,X. Write a function that will create the successor lists of Exercise 22.

24. Simulate the preorder traversal implementation for the link-inversion traversal of the binary tree of Figure 13.10.

25. Simulate the preorder traversal implementation for the Robson traversal of the binary tree of Figure 13.10.

26. Why will there always be enough storage in the binary tree, during a Robson traversal, for the stack of root nodes whose left subtrees are nonnull, whose traversal has been completed, and whose right subtrees are being traversed?

27. What purpose does the stack of Exercise 26 serve in the Robson traversal?


Data Structures Using C: Suggested Assignments

Suggested Assignments

1. Write and run a function to copy a list-structure with shared sublists. Use pointer variables.

2. Explain how storage may be reclaimed if all nodes of the copied list-structure that have nonnull sublists are deleted.

3. Why would we not write recursives versions of the three stackless traversals?

4. Simulate the different implementations for the list of available space of Section 13.3, choosing a number of request and return distributions.


Data Structures Using C: BIBLIOGRAPHY

BIBLIOGRAPHY

Adelson-Velskii, G. M., and Landis, Y. M. [1962]. "An Algorithm for the Organization of Information." Doklady Akademia Nauk. 146(2), 263-266; Soviet Mathematics. 3, 1259-1263.

Aho, A. V., Hopcroft, J. E., and Ullman, J. D. [1983]. Data Structures and Algorithms. Reading, Mass.: Addison-Wesley.

Aho, A. V., and Ullman, J. D. [1972]. The Theory of Parsing, Translation, and Compiling, Vol. 1. Englewood Cliffs, N.J.: Prentice-Hall.

Aho, A. V., and Ullman, J. D. [1977]. Principles of Compiler Design. Reading, Mass.: Addison-Wesley.

Amble, O., and Knuth, D. E. [1974]. "Ordered Hash Tables." Computer Journal 17(2):135-142.

Austing, R. H., et al., eds. [1979]. "Curriculum '78: Recommendations for the Undergraduate Program in Computer Science�A Report of the ACM Curriculum Committee on Computer Science." Communications of the Association for Computing Machinery 22(3): 147-166.

Baase, S. [1978]. Computer Algorithms: Introduction to Design and Analysis. Reading, Mass.: Addison-Wesley.

Barnes, J. G. P. [1982]. Programming in Ada. Reading, Mass.: Addison-Wesley.

Barron, D. W. [1968]. Recursive Techniques in Programming. New York: American-Elsevier.

Bayer, R., and McCreight, E. M. [1972]. "Organization and Maintenance of Large Ordered Indices." Acta Informatica 1(3): 173-189.

Bays, C. [1977]. "A Comparison of Next-Fit, First-Fit, and Best-Fit." Communications of the Association for Computing Machinery 20(3):191-192.

Bell, D. A., and Deen, S. M. [1984]. "Hash Trees Versus B-Trees." Computer Journal 27(3):218-224.

Bell, J. R. [1970]. "The Quadratic Quotient Method: A Hash Code Eliminating Secondary Clustering." Communications of the Association for Computing Machinery 13(2): 107-109.

Ben-Ari, M. [1982]. Principles of Concurrent Programming. London: Prentice-Hall International.

Bentley, J. L. [1982]. Writing Efficient Programs. Englewood Cliffs, N.J.: Prentice-Hall.

Berry, J. T. [1986]. Advanced C Programming. New York: Prentice-Hall Press.

Bird, R. S. [1977a]. "Improving Programs by the Introduction of Recursion." Communications of the Association for Computing Machinery 20(11):856-863.

Bird, R. S. [1977b]. "Notes on Recursion Elimination." Communications of the Association for Computing Machinery 20(6): 434-439.

Bitner, J. R., and Reingold, E. M. [1975]. "Backtrack Programming Techniques." Communications of the Association for Computing Machinery 18:651-656.

Brooks, F. P. [1974]. The Mythical Man Month. Reading, Mass.: Addison-Wesley.

Bunneman, P., and Levy, L. [1980]. "The Towers of Hanoi Problem." Information Processing Letters 10:243-244.

Comer, D. [1979]. "The Ubiquitous B-Tree." Computing Surveys 11:121-137.

Crane, C. A. [1972]. Linear Lists and Priority Queues as Balanced Binary Trees. Unpublished Ph.D. Dissertation. Stanford University, Stanford, Calif.

Dahl, O.-J., Dijkstra, E. W., and Hoare, C. A. R. [1972]. Structured Programming. New York: Academic Press.

Dijkstra, E. W. [1972]. Notes on Structured Programming. London: Academic Press.

Dobosiewicz, W. [1978]. "Sorting by Distributive Partitioning." Information Processing Letters 7(1): 1-6.

Earley, J. [1970]. "An Efficient Context-Free Parsing Algorithm." Communications of the Association for Computing Machinery 13(2):94-102.

Elson, M. [1975]. Data Structures. Chicago, Ill.: Science Research Associates.

Fagin, R., Nievergelt, J., Pippenger, N., and Strong, R. H. [1978]. Extendible Hashing�A Fast Access Method for Dynamic Files. Research Rept. RJ2305. IBM Research Division, Yorktown Heights, N.Y.

Feller, W. [1975]. An Introduction to Probability Theory and Its Applicalions. 2nd ed. New York: Wiley.

Floyd, R. W. [1964]. "Algorithm 245: Treesort3." Communications of the Association for Computing Machinery 7(12):701.

Floyd, R. W. [1979]. "The Paradigms of Programming," Communications of the Association for Computing Machinery 22(8):455-460.

Fredkin, E. H. [1960]. "Trio Memory." Communications of the Association for Computing Machinery 3(9):490-500.

Garwick, J. V. [1964]. "Data Storage in Compilers." BIT 4:137-140.

Golomb, S. W., and Baumert, L. D. [1965]. "Backtrack Programming." Journal of the Association for Computing Machinery 12(4):516-524.

Greene, D. H., and Knuth, D. E. [1981]. Mathematics for the Analysis of Algorithms. Cambridge, Mass.: Birkhauser, Boston.

Harrison, S. P., and Steele, G. L., Jr. [1987]. C: A Reference Manual. Englewood Cliffs, N.J.: Prentice-Hall.

Hirschberg, D. S. [1973]. "A Class of Dynamic Memory Allocation Algorithms." Communications of the Association for Computing Machinery 16(10):615-618.

Hoare, C. A. R. [1962]. "Quicksort." ComputerJournal 5(1):10-15.

Horbibe, Y., and Nemetz, T. [1979]. "On the Max-Entropy Rule for a Binary Search Tree." Acta Informatica 12:63-72.

Horowitz, E. [1983]. Programming Languages: A Grand Tour. Rockville, Md.: Computer Science Press.

Horowitz, E., and Sahni, S. [1976]. Fundamentals of Data Structures. Woodland Hills, Calif.: Computer Science Press.

Horowitz, E., and Sahni, S. [1978]. Fundamentals of Computer Algorithms. Potomac, Md.: Computer Science Press.

Hu, T. C., and Tucker, A. C. [1971]. "Optimal Computer Search Trees and Variable-Length Alphabetic Codes." SIAM Journal of Applied Mathematics 21(4):514-532.

Huffman, D. A. [1952]. "A Method for the Construction of Minimum-Redundancy Codes." Proceedings of the Institute of Radio Engineers 40: 1098-1101.

Hull, M. E. C. [1984]. "A Parallel View of Stable Marriages." Information Processing Letters 18:63-66.

Johnson, D. B. [1975]. "Priority Queues with Update and Finding Minimum Spanning Trees." Information Processing Letters 4(3):53-57.

Jones, W. B. [1982]. Programming Concepts: A Second Course. Englewood Cliffs, N.J.: Prentice-Hall.

Kelley, A., and Pohl, I. [1987]. C by Dissection. Menlo Park, Calif.: Benjamin/Cummings.

Kernighan, B. W., and Plauger, P. J. [1978]. The Elements of Programming Style. 2nd ed. New York: McGraw-Hill.

Kernighan, B. W., and Plauger, P. J. [1981]. Software Tools in Pascal. Reading, Mass.: Addison-Wesley.

Kernighan, B. W., and Ritchie, D. M. [1978]. The C Programming Language. Englewood Cliffs, N.J.: Prentice-Hall.

Knuth, D. E. [1971]. "Optimum Binary Search Trees." Acta Informatica 1(1):14-25.

Knuth, D. E. [1973a]. The Art of Computer Programming. Vol. 1: Fundamental Algorithms. Reading, Mass.: Addison-Wesley.

Knuth, D. E. [1973b]. The Art of Computer Programming. Vol. 3: Searching and Sorting. Reading, Mass.: Addison-Wesley.

Koffman, E. B., Stemple, D., and Wardle, C. E. [1985]. "Recommended Curriculum for CSE, 1984." Communications of the Association for Computing Machinery 28(8):815-818.

Korsh, J. F. [1981]. "Greedy Binary Search Trees Are Nearly Optimal." Information Processing Letters 13(1):16-19.

Korsh, J. F. [1982]. "Growing Nearly Optimal Binary Search Trees." Information Processing Letters 14(3): 139-143.

Korsh, J. F., and Laison, G. [1983]. "A Multiple-Stack Manipulation Procedure." Communications of the Association for Computing Machinery 26(11):921-923.

Kruse, R. L. [1984]. Data Structures and Program Design. Englewood Cliffs, N.J.: Prentice-Hall.

Lapin, J. E. [1987]. Portable C and UNIX System Programming. Englewood Cliffs, N.J.: Prentice-Hall.

Larson, P. [1978]. "Dynamic Hashing." BIT 18(2):184-201.

Lin, S., and Kernighan, B. W. [1973]. "A Heuristic Algorithm for the Traveling Salesman Problem." Operations Research 21:498-516.

Linger, R. C., Mills, H. D., and Witt, B. I. [1979]. Structured Programming Theory and Practice. Reading, Mass.: Addison-Wesley.

Litwin, W. [1980]. "Linear Hashing: A New Tool for File and Table Addressing." Proceedings of the Sixth International Conference on Very Large Data Bases. Montreal, pp. 212-223.

Martin, W. A. [1971]. "Sorting." Computing Surveys 3:148-174.

Maurer, H. A., and Williams, M. R. [1972]. A Collection of Programming Problems and Techniques. Englewood Cliffs, N. J.: Prentice-Hall.

Maurer, W. D. [1968]. "An Improved Hash Code for Scatter Storage." Communications of the Association for Computing Machinery 11(1):35-37.

Maurer, W. D., and Lewis, T. G. [1975]. "Hash Table Methods." Computing Surveys 7(1):5-20.

Mehlhorn, K. [1975]. "Nearly Optimal Binary Search Trees." Acta Informatica 5:287-295.

Mehlhorn, K. [1977]. "A Best Possible Bound for the Weighted Path Length of Binary Search Trees." Society of Industrial and Applied Mathematics, Journal of Computers 6(2):235-239.

Miller, L. H., and Quilici, A. E. [1986]. Programming in C. New York: Wiley.

Morris, R. [1968]. "Scatter Storage Techniques." Communications of the Association for Computing Machinery 11(1):35-44.

Naur, P. (ed.). [1963]. "Revised Report on the Algorithmic Language ALGOL60," Communications of the Association for Computing Machinery 6:1-17.

Nievergelt, J. [1974]. "Binary Search Trees and File Organization." Computing Surveys 6(3):195-207.

Peterson, J. L. [1980]. Design of a Spelling Program: An Experiment in Program Design. Lecture Notes in Computer Science. New York: Springer-Verlag.

Peterson, J. L., and Norman, T. A. [1977] . "Buddy Systems." Communications of the Association for Computing Machinery 20(6):431-435.

Pratt, T. W. [1975]. Programming Languages: Design and Implementation. Englewood Cliffs, N.J.: Prentice-Hall.

Reingold, E. M., and Hansen, W. J. [1983]. Boston, Mass.: Little, Brown.

Robson, J. M. [1973]. "An Improved Algorithm for Traversing Binary Trees without Auxiliary Stack." Information Processing Letters 2:12-14.

Robson, J. M. [1977]. "A Bounded Storage Algorithm for Copying Cyclic Structures." Communications of the Association for Computing Machinery 20(6):431-433.

Robson, J. M. [1979]. "The Height of Binary Search Trees." Austrian Computer Journal 11(4):151-153.

Schorr, H., and Waite, W. M. [1967]. "An Efficient Machine-Independent Procedure for Garbage Collection in Various List Structures." Communications of the Association for Computing Machinery 10(8):501-506.

Sedgewick, R. [1977]. "Permutation Generation Methods." Computing Surveys 9:137-164.

Sedgewick, R. [1983]. Algorithms. Reading, Mass.: Addison-Wesley.

Shell, D. L. [1959]. "A High-Speed Sorting Procedure." Communications of the Association for Computing Machinery 2:30-32.

Standish, T. A. [1980]. Data Structure Techniques. Reading, Mass.: Addison-Wesley.

Stanfel, G. L. [1970]. "Tree Structures for Optimal Searching." Journal of the Association for Computing Machinery 19(3):508-517.

Stone, H. S. [1971]. "Parallel Processing with the Perfect Shuffle." IEEE Transactions on Computers C-20: 153-161.

Stone, H. S. [1972]. Introduction to Computer Organization and Data Structures. New York: Mc-Graw-Hill.

Stone, H. S. [1980]. Introduction to Computer Architecture. 2nd ed. Chicago, Ill.: Science Research Associates.

Sussenguth, E. H. Jr. [1963]. "Use of Tree Structures for Processing Files." Communications of the Association for Computing Machinery 6(5):272-279.

Tenenbaum, A. M., and Augenstein, M. J. [1981]. Data Structures Using Pascal. Englewood Cliffs, N.J.: Prentice-Hall.

Tremblay, J. P., and Sorenson, P. G. [1984]. An Introduction to Data Structures with Applications. 2nd ed. New York: McGraw-Hill.

Ullman, J. D. [1982]. Principles of Database Systems. Rockville, Md.: Computer Science Press.

Van Wijngaarden, A., et al. [1975]. "Revised Report on the Algorithmic Language ALGOL 68." Acta Informatica 5:1-236.

Walsh, T. R. [1982]. "The Towers of Hanoi Revisited: Moving the Rings by Counting the Moves." Information Processing Letters 15(2):64-67.

Weinberg, G. M. [1971]. The Psychology of Computer Programming. New York: Van Nostrand.

Wexelblat, R. L. [1981]. History of Programming Languages. New York: Academic Press.

Williams, J. W. J. [1964]. "Algorithm 232: Heapsort." Communications of the Association for Computing Machinery 7(6):347-348.

Wirth, N. [1973]. Systematic Programming: An Introduction. Englewood Cliffs, N.J.: Prentice-Hall.

Wirth, N. [1976]. Algorithms + Data Structures = Programs. Englewood Cliffs, N.J.: Prentice-Hall.


Comment on this Routine

initialize candidates to empty

set(n,candidates);


Comment on this Routine

inserts integers 2 to n into candidates

for (i=2;i<=n;i++)

insert(i,candidates);


Comment on this Routine

removes nonprimes from candidates

{

nextmultiple = 2 * factor;

while (nextmultiple <= n)


Comment on this Routine

prints the primes

for (i=2;i<=n;i++)

if(belongs(i,candidates))

printf("\n %d\n",i);


Comment on this Routine

if i is not in c, adds 2i to c[0]

}                                 p = power(2,i);

                                  q = (c[0]/p)%2;

                                  c[0] = c[0] + (1-q)*p;


Comment on this Routine

if i is in c, subtracts 2i from c[0]

}                                  p = power(2,i);

                                   q = (c[0]/p)%2;

                                   c[0] = c[0] - q*p;


Comment on this Routine

reads in an integer n and calls primes to print all prime numbers � n

main()

{

int n;

printf("\n n = ? \n");

scanf("%d",&n);

primes(n);


Comment on this Routine

These definitions and functions give a specific implementation of a data abstraction: collection with operations set, insert, omit, and belongs allowed. This is the only portion of the program that needs to be changed when the implementation of the data abstraction is changed. MAXN sets a limit on the size of arrays of type collection, so it sets a limit for n.

#define TRUE 1

#define FALSE 0

#define MAXN 500

typedef int collection[MAXN];

set(n,c)

/* Sets c to empty */

int n;

collection c;

{

int i;

for (i=1;i<=n;i++)

c[i] = FALSE;

}

insert(i,c)

/* Inserts i in c */

int i;

collection c;

{

c[i] = TRUE;

}

omit(i,c)

/* Removes i from c */

int i;

collection c;

{

c[i] = FALSE;

}

belongs(i,c)

/* Returns true only

if i is in c

*/

int i;

collection c;

{

return(c[i]);

}


Comment on this Routine

This portion of the program is independent of the specific implementation of the data abstraction. It does not have to be changed if the implementation of the data abstraction changes. Candidates is an instance of the data structure of type collection.

primes(n)

/* Prints all prime numbers between 2 and n */

int n;

{

collection candidates;

create(n,candidates);

remove(n,candidates);

print(n,candidates);

}

create(n,c)

/* Creates a collection of integers between 2 and n */

int n;

collection c;

{

int i;

set(n,c);

for (i=2;i<=n;i++)

insert(i,c);

}

remove(n,c)

/* Removes all nonprimes from c */

int n;

collection c;

{

int firstprime;

int factor;

firstprime = 2;

factor = firstprime;

while (nonprimes(factor,n))

{

delete(factor,n,c);

factor++;

}

}

nonprimes(factor,n)

/* Returns true only if non

primes may remain in c

*/

int factor,n;

double sqrt();

{

return(factor <= sqrt((double)n));

}

delete(factor,n,c)

/* Deletes multiples of factor from c */

int factor,n;

collection c;

{

int nextmultiple;

nextmultiple = 2 * factor;

while (nextmultiple <= n)

{

omit(nextmultiple,c);

nextmultiple = nextmultiple + factor;

}

}

print(n,c)

/* Prints integers in c */

int n;

collection c;

{

int i;

for (i=2;i<=n;i++)

if (belongs(i,c))

printf("\n %d\n",i);

}


Comment on this Routine

sets c to empty

set(n,c);


Comment on this Routine

inserts integers 2 to n in c

for (i=2;i<=n;i++)

insert(i,c);


Comment on this Routine

For all values of factor f rom 2 to n, invokes delete to remove multiples

while (nonprimes(factor,n))

{

delete(factor,n,c);

factor++;

}

}


Comment on this Routine

returns true as long as factor is not larger than the square root of n

return(factor <= sqrt((double)n));


Comment on this Routine

invokes omit to remove each nonprime from c that is a multiple of factor

while (nextmultiple <= n)

{

omit(nextmultiple,c);

nextmultiple = nextmultiple + factor;

}

}


Comment on this Routine

invokes belongs for each integer from 2 to n to determine if it is a prime, and if so prints it

for (i=2;i<=n;i++)

if (belongs(i,c))

printf("\n %d\n",i);

}


Comment on this Routine

note that, for example, char last[15] denotes a character string 15 characters long

float amount;     char last[15];


Comment on this Routine

traverses the array until an entry containing account k is found or until the last array entry has been considered

while (found != TRUE && loc <= n)

if (a[loc] != k)

loc++;

else

found = TRUE;


Comment on this Routine

collection is the type name for an integer array of size 50 � 50

typedef int collection[50][50];


Comment on this Routine

a is an instance of collection, an integer array of size 50 � 50

collection a;


Comment on this Routine

i is the row and j the column index

int i,j,ck;


Comment on this Routine

transverses the first n rows and columns of a and sets ck to false if any a[i][j] is not equal to a[j][i]

for (i = 0; i<n; i++)

for (j = 0; j<n; j++)

if (a[i][j] != a[j][i])

ck = FALSE;


Comment on this Routine

the traversal is aborted when a is known not to be symmetric; it makes only necessary comparisons

for(i=1; i<n && ck; i++)

for(j=0; j <i && ck; j++)

if(a[i][j]! = a [j][i])

ck = FALSE;


Comment on this Routine

fieldtype is now another name for the type: integer�it is used to clarify its intended use;

typedef int fieldtype;


Comment on this Routine

a is an instance of type array

array a;


Comment on this Routine

length is initialized to the number of fields in each record

int length = 3, i;


Comment on this Routine

a is the array of records to be traversed

traverse(a)


Comment on this Routine

traverses the records until the last has been processed; does the necessary processing of the (i + 1)th record

while(a[i] != SENTINEL)

{

process(a,i);

i =i+a[i];

}


Comment on this Routine

data is a one-dimensional array

int data[];


Comment on this Routine

compares the entries in positions [i][j] and [j][i] of the n � n two-dimensional array being checked by comparing their values, which are stored in data[i*n+j] and in data[j*n+i]

if(data[(i*n+j)] != data[(j*n+i)])


Comment on this Routine

type names for array of pointers to rows and array containing the rows

typedef int pointerarray[50];

typedef int dataarray[2500];


Comment on this Routine

p is an instance of type pointerarray and data is an instance of type data-array

pointerarray p;

dataarray data;


Comment on this Routine

the entries of the r � r two-dimensional array checked are stored in positions p[i] + j and p[j] + i of data

if(data [p[i]+j]! = data[p[j]+i])


Comment on this Routine

type names for row records, for array of pointers to row records, and array containing the row records

typedef struct

{

int row[50];

}rowrecord;

typedef int pointerarray[50];

typedef rowrecord dataarray[50];


Comment on this Routine

p is an instance of type pointerarray, and data is an instance of type data-array

pointerarray p;

dataarray data;


Comment on this Routine

the entries of the r � r two-dimensional array checked are stored in field row in records in positions p[i] and p[j] of data

if(data[p[i]].row[j] !=


Comment on this Routine

type names for row records, for pointers to row records, and an array of pointers to row records

typedef struct

{

int row[50];

}rowrecord, *rowpointer;

typedef rowpointer pointerarray[50];


Comment on this Routine

p is an instance of type pointerarray

pointerarray p;


Comment on this Routine

i is the row and j the column index

int i,j;


Comment on this Routine

malloc must return a pointer to storage of type rowpointer

rowpointer malloc();


Comment on this Routine

sets p[i] to storage for next row record

p[i]=malloc(sizeof(rowrecord));


Comment on this Routine

reads in the next row record

for(j=0;j<r;j++)

scanf("%d",&(p[i]->row[j]));


Comment on this Routine

prints the next row record

for (j=0;j<r;j++)

printf("%3d",p[i]->row[j]);


Comment on this Routine

the entries of the r � r two-dimensional array checked are stored in the field row of the records in dynamic memory pointed to by the pointers in p[i] and p[j]

if (p[i]->row[j] != p[j]->row[i])


Comment on this Routine

Task 1

mate(person,pairs)


Comment on this Routine

selects the mate from mpairs or wpairs

return(pairs[person]);


Comment on this Routine

Task 2

mostpreferred(person,pref)


Comment on this Routine

selects the mate most preferred from mpref or wpref

return(pref[person][1]);


Comment on this Routine

Task 3

prefers(individual,person,individualsmate,pref)


Comment on this Routine

index is a column index for mpref or wpref

int index,next;


Comment on this Routine

traverses the row for individual in mpref or wpref until person or individual's mate is found and returns true only if person is found first

while(next != person && next != individualsmate)

{

index++;

next = pref[individual][index];

}


Comment on this Routine

Task 4

nextpreferred(person,index,pref)


Comment on this Routine

selects next most preferred from mpref or wpref

return(pref[person][index]);


Comment on this Routine

Task 3�more efficient implementation

prefers(individual,person,individualsmate,priority);


Comment on this Routine

selects from mpriority or wpriority; traversal not needed here

return(priority[individual][person]<

priority[individual][individualsmate]);


Comment on this Routine

data abstraction pairings and the allowed operations on it�mate, read_mens_mates, and create_womens_mates

#define MAXN 50

typedef int pairings[MAXN];

mate(person,pairs)

/* Returns person's mate

as specified by pairs.

*/

int person;

pairings pairs;

{

return(pairs[person]);

}

read_mens_mates(n,pairs)

/* Reads the n mates of the

men into pairs.

*/

int n;

pairings pairs;

{

int i;

printf("\n mpairs = ? \n");

for(i=1;i<=n;i++)

scanf("%d",&pairs[i]);

}

create_womens_mates(n,pairs1,pairs2)

/* Creates in pairs2 the women's mates

based on the men's mates as specified

by pairs1.

*/

int n;

pairings pairs1,pairs2;

{

int i,j;

for(i=1;i<=n;i++)

{

for(j=1;pairs1[j] != i;j++)

{

}

pairs2[i] = j;

}

}


Comment on this Routine

used to input data and to build mpairs

read_mens_mates(n,pairs)


Comment on this Routine

used to create wpairs from mpairs

create_womens_mates(n,pairs1,pairs2)


Comment on this Routine

data abstraction preferences and the allowed operations on it� mostpreferred, next preferred, prefers, read_preferences, and create_priorities

typedef int preferences[MAXN] [MAXN];

mostpreferred(person,pref)

/* Returns the mate most

preferred by person as

specified by pref.

*/

int person;

preferences pref;

{

return(pref[person][1]);

}

nextpreferred(person,index,pref)

/* Returns the mate next most

preferred by person as specified

by pref.

*/

int person,index;

preferences pref;

{

return (pref[person][index]);

}

prefers(individual,person, individualsmate,priority)

/* Returns true only if individual prefers person

to the individual's mate as specified by priority.

*/

int individual,person,individualsmate;

preferences priority;

{

return(priority[individual][person]

< priority[individual][individualsmate]);

}

read_preferences(n,pref)

/* Reads n rows of

preferences into pref.

*/

int n;

preferences pref;

{

int i,j;

for(i=1;i<=n;i++)

for(j=1;j<=n;j++)

scanf("%d",&pref[i][j]);

}

create_priorities(n,pref,priority)

/* Creates n individual priorities in

priority based on the individual

preferences specified by pref.

*/

int n;

preferences pref,priority;

{

int i,j;

for(i=1;i<=n;i++)

for(j=1;j<=n;j++)

priority[i][pref[i][j]] = j;

}


Comment on this Routine

used to create mpriority and wpriority

create_priorities(n,pref,priority)


Comment on this Routine

mpairs and wpairs are instances of type pairings; mpref, wpref, mpriority, and wpriority instances of type preferences

pairings mpairs,wpairs;

preferences mpref,wpref,mpriority,wpriority;


Comment on this Routine

inputs the number of men (and women)

printf("\n n=?\n);

scanf("%d",&n);


Comment on this Routine

reads into mpairs

read_mens_mates(n,mpairs);


Comment on this Routine

creates wpairs

create_womens_mates(n,mpairs,wpairs);


Comment on this Routine

reads into mpreferences

read_preferences (n,mpref);


Comment on this Routine

reads into wpreferences

read_preferences (n,wpref);


Comment on this Routine

creates mpriority

create_priorities(n,mpref,mpriority);


Comment on this Routine

creates wpriority

create_priorities(n,wpref,wpriority);


Comment on this Routine

checks stability of the entire pairing and outputs the result; outputs the entire pairing checked

if(stabilitycheck(n,mpairs,wpairs,mpref,wpref,

mpriority,wpriority))

printf("The pairings are stable \n");

else

printf("The pairings are not stable \n");

printf("The pairings are: \n");

printf("man wife \n");

for(i=1;i<=n;i++)

printf("%d %d \n",i,mate(i,mpairs));

}


Comment on this Routine

determines whether the entire pairing is stable

for(man=1;(stable && man<=n);man++)

{

wife = mate(man,mpairs);

individual = mostpreferred(man,mpref);

index = 1;

while(stable &&(individual != wife))

{

individualsmate = mate(individual,

wpairs);

if(prefers(individual,man,

individualsmate,wpriority))

stable = FALSE;

else

{

index++;

individual = nextpreferred(man,

index,mpref);

}

}

individual = mostpreferred(wife,wpref);

index = 1;

while(stable &&(individual != man))

{

individualsmate = mate(individual,

mpairs);

if(prefers(individual,wife,

individualsmate,mpriority))

stable = FALSE;

else

{

index++;

individual = nextpreferred(wife,

index,wpref);

}

}

}


Comment on this Routine

determines if the man in the pair man-wife is stable

while(stable &&(individual != wife))

{

individualsmate = mate(individual,

wpairs);

if(prefers(individual,man,

individualsmate,wpriority))

stable = FALSE;

else

{

index++;

individual = nextpreferred(man,

index,mpref);

}

}


Comment on this Routine

determines if the wife in the pair man-wife is stable

while(stable &&(individual != man))

{

individualsmate = mate(individual,

mpairs);

if(prefers(individual,wife,

individualsmate,mpriority))

stable = FALSE;

else

{

index++;

individual = nextpreferred(wife,

index,wpref);


Comment on this Routine

inserts a new record into a list

insert(phead,predecessor,newpointer)


Comment on this Routine

inserts as a new first record

setlink(newpointer,*phead);

*phead = newpointer;


Comment on this Routine

inserts in the body of the list

setlink(newpointer,next(predecessor));

setlink(predecessor,newpointer);


Comment on this Routine

deletes a record from a list

delete(phead,predecessor)


Comment on this Routine

if the list is not null

if(*phead != null)


Comment on this Routine

deletes the first record

*phead = next(*phead);


Comment on this Routine

deletes from the body of the list

successor = next(predecessor);

if(successor != null)

setlink(predecessor,next(successor));


Comment on this Routine

a general list traversal function

traverse(listname)


Comment on this Routine

does the needed initialization for process

initialize(listname,&other);


Comment on this Routine

processes a record

process(listname,recordpointer,


Comment on this Routine

updates recordpointer to point to the next record

recordpointer =

next(recordpointer);


Comment on this Routine

does the finalization needed after all records have been processed

finalize(listname,other);


Comment on this Routine

initializes length to zero

initialize(listname,plength)


Comment on this Routine

adds one to length to count the next record

process(listname,recordpointer,plength)


Comment on this Routine

prints the result

finalize(listname,length)


Comment on this Routine

other corresponds to length

initialize(listname,&other)


Comment on this Routine

the only modification�done is not needed

process(listname,recordpointer,&other);


Comment on this Routine

updates length to reflect the number of characters in the record pointed to by recordpointer

process(listname,recordpointer,plength)


Comment on this Routine

not the last record, so it contains three characters

if(!lastrecord)

*plength = *plength + 3;


Comment on this Routine

the last record has only a period

else if(char1(recordpointer) == '.')

(*plength)++;


Comment on this Routine

has two characters

*plength = *plength + 2;


Comment on this Routine

has three characters

*plength = *plength + 3;


Comment on this Routine

when recordpointer is not null, there is another record

return(recordpointer != setnull());


Comment on this Routine

when recordpointer does not point to the first record, or is not null, there is another record

return(listname != recordpointer);


Comment on this Routine

initializes save and predecessor

initialize(listname,psave,ppredecessor)


Comment on this Routine

sets the link field of the record pointed to by predecessor to save, which points to that record's predecessor and updates save and predecessor

process(plistname,recordpointer,psave,


Comment on this Routine

when the list has more than one record, reverses the link of the last record and sets the list name to point to the original last record

finalize(plistname,save,predecessor)


Comment on this Routine

the parameter of traverse has been modified to a pointer to listname, since it may need to be changed; save and predecessor are now parameters of initialize, process, and finalize

traverse(plistname)


Comment on this Routine

sets newpointer to point to storage for the new record, reads in its data and places it into the new record, initializes predecessor

initialize(pnewpointer,ppredecessor)


Comment on this Routine

inserts the new record if the list is null

finalize(plistname,predecessor,newpointer)


Comment on this Routine

inserts the record if the proper place is after predecessor and sets done to true, else just updates predecessor

process(plistname,recordpointer,newpointer,ppredecessor,


Comment on this Routine

insert in list body

insert(plistname,*ppredecessor,newpointer);

*pdone = TRUE;


Comment on this Routine

insert as new last record

insert(plistname,recordpointer,newpointer);

*pdone = TRUE;


Comment on this Routine

not the place to insert

else

*ppredecessor = recordpointer;


Comment on this Routine

inserts a new record in the proper place in a list

orderedinsert(plistname)


Comment on this Routine

allocates storage for the list and sets listname to point to it

initialize(plistname)


Comment on this Routine

reads the information for the new record, fills in its information and link fields, and updates predecessor, so that on the next call it can access the new record's predecessor when necessary; special handling is needed when the input is the sentinel

process(plistname,recordpointer,pdone)


Comment on this Routine

the sentinel was input and the list is not null, so the link field of the new record's predecessor (the last list record) must be set to null and done to true

else if (*plistname != recordpointer)

{

setlink(predecessor,setnull());

*pdone = TRUE;

}


Comment on this Routine

the sentinel was input and the list is null, so the list name must be set to null and done to true

*plistname = setnull();

*pdone = TRUE;


Comment on this Routine

this modified traverse inserts all records of an arbitrarily ordered input list of records into an ordered list, listname, maintaining ordering

expand (plistname)

/*Inputs a series of records and

inserts them in alphabetical

order in the ordered list listname.

*/

listpointer *plistname;

{

int done;

done = FALSE;

while(!done)

processrecord(plistname,&done);

}


Comment on this Routine

processes the next record of the input list

processrecord(plistname,&done);


Comment on this Routine

gets the next record of the input and inserts it into listname

processrecord(plistname,pdone)


Comment on this Routine

inserts the record pointed to by pointer into the list, listname

insertinorder(plistname,pointer);


Comment on this Routine

this modification of traverse inserts the input record at the correct place

insertinorder(plistname,pointer)


Comment on this Routine

creates the new record

initialize(&newpointer,&predecessor,pointer);


Comment on this Routine

processes the new record

process(plistname,recordpointer,newpointer,


Comment on this Routine

deals with insertion in a null list

finalize(plistname,predecessor,pointer);


Comment on this Routine

creates the new record and initializes predecessor

initialize(pnewpointer,ppredecessor,pointer)


Comment on this Routine

inserts the record if the proper place is after predecessor and sets done to true; else just updates predecessor

process(plistname,recordpointer,newpointer,


Comment on this Routine

not the place to insert

*ppredecessor = recordpointer;


Comment on this Routine

This portion of the program specifies the implemetation of the list and its allowed operations. It is the only part of the program that needs to be changed should the list implementation change.

#define MAXSIZE 5

#define NULL 0

#define SENTINEL -1

typedef struct

{

int month;

int day;

int year;

}date;

typedef struct

{

char name[MAXSIZE];

int share;

float value;

date datebought;

}infofield;

typedef struct listrecord

{

infofield info;

struct listrecord *link;

}stockrecord,*listpointer;

listpointer next(pointer)

/* Returns a copy of the

link field in the record

pointed to by pointer

*/

listpointer pointer;

{

return(pointer->link);

}

listpointer setnull()

/* Returns a null pointer */

{

return(NULL);

}

setlink(pointer1,pointer2)

/* Sets the link field of the

record pointed to by pointer1

to the contents of pointer2

*/

listpointer pointer1,pointer2;

{

pointer1->link = pointer2;

}

setinfo(pointer1,pointer2)

/* Sets the infofield of the record

pointed to by pointer1 to the

infofield of the record

pointed to by pointer2

*/

listpointer pointer1,pointer2;

{

int i;

for(i=0;i<MAXSIZE;i++)

pointer1->info.name[i] = pointer2->

info.name[i];

pointer1->info.shares = pointer2->

info.shares;

pointer1->info.value = pointer2->info.value;

pointer1->info.datebought.month = pointer2->

info.datebought.month;

pointer1->info.datebought.day = pointer2->

info.datebought.day;

pointer1->info.datebought.year = pointer2->

info.datebought.year;

}

listpointer avail()

/* Returns a pointer to storage

allocated for a record of type

stockrecord

*/

{

listpointer malloc();

return(malloc(sizeof(stockrecord)));

}

anotherrecord(recordpointer)

/* Returns true if there is

another record

*/

listpointer recordpointer;

{

listpointer setnull();

return(recordpointer != setnull());

}

listpointer getnextrecord()

/* Inputs the data for the new

record and returns a pointer

to a record containing the data

in its infofield, or, if there

are no new records,returns

a null pointer

*/

{

listpoiner setnull();

static stockrecord nextrecord;

printf("Enter number of shares\n");

scanf("%d",&(nextrecord.info.shares));

if(nextrecord.info.shares != SENTINEL)

{

printf(" Enter the stock name - less than

%d characters\n",MAXSIZE);

scanf("%s",nextrecord.info.name);

printf(" Enter the price of one share of

the stock\n");

scanf("%f",&(nextrecord.info.value));

printf(" Enter the month day year of the

stock purchase\n");

scanf("%d %d %d",&(nextrecord.info.

datebought.month),

&(nextrecord.info.datebought.day),

&(nextrecord.info.datebought.year));

return(&nextrecord);

}

else

return(setnull());

}

printrecord(recordpointer)

/* Prints the contents of the infofield

of the record pointed to by recordpointer

*/

listpointer recordpointer;

{

printf(" The stock is %s\n",

recordpointer->info.name);

printf(" The number of shares is %d\n",

recordpointer->info.shares);

printf(" The value of a share is %f\n",

recordpointer->info.value);

printf(" The date of purchase is %d %d %d\n\n",

recordpointer->info.datebought.month,

recordpointer->info.datebought.day,

recordpointer->info.datebought.year);

}


Comment on this Routine

this portion of the program is independent of the list implementation

#include <stdio.h>

#define TRUE 1

#define FALSE 0

main()

/* Inputs a series of stockrecords,

creates a list of the records, and

prints the contents of each list

record's infofield

*/

{

listpointer stocks,recordpointer,next();

int done;

initialize(&stocks);

recordpointer = stocks;

done = FALSE;

while(!done&&anotherrecord(recordpointer))

{

addrecord(&stocks,recordpointer,&done);

recordpointer = next(recordpointer);

}

recordpointer=stocks;

while(anotherrecord(recordpointer))

{

printrecord(recordpointer);

recordpointer = next(recordpointer);

}

}

initialize(plistname)

/* Allocates storage for the first record

and sets listname to point to it

*/

listpointer *plistname;

{

listpointer avail();

*plistname = avail();

}

addrecord(plistname,recordpointer,pdone)

/* Fills in the current record's data, and allocates

storage for the next record and adds it to the

list. If there is no data for the current record

it sets the link field of the last list record

to null, or the list head to null, and done to

true.

*/

listpointer *plistname,recordpointer;

int *pdone;

{

static listpointer predecessor;

listpointer setnull(),avail(),getnextrecord(),

pointer;

pointer =getnextrecord();

if(pointer != setnull())

{

setinfo(recordpointer,pointer);

setlink(recordpointer,avail());

predecessor = recordpointer;

}

else if(*plistname != recordpointer)

{

setlink(predecessor,setnull());

*pdone = TRUE;

}

else

{

*plistname = setnull();

*pdone = TRUE;

}


Comment on this Routine

this is the traversal to input data and create the list

int done;

initialize(&stocks);

recordpointer = stocks;

done = FALSE;

while(!done&&anotherrecord(recordpointer))

{

addrecord(&stocks,recordpointer,&done);

recordpointer = next(recordpointer);

}


Comment on this Routine

fills in the current record, allocates storage for the next and adds it to the list, and if there is no data for the current record, terminates the list and sets done to true

addrecord(&stocks,recordpointer,&done);


Comment on this Routine

this is the traversal to print the list

recordpointer=stocks;

while(anotherrecord(recordpointer))

{

printrecord(recordpointer);

recordpointer = next(recordpointer);

}

}


Comment on this Routine

prints the record

printrecord(recordpointer);


Comment on this Routine

there is data for the current record

setinfo(recordpointer,pointer);

setlink(recordpointer,avail());

predecessor = recordpointer;


Comment on this Routine

there is no data for the current record, and the list is not null

setlink(predecessor,setnull());

*pdone = TRUE;


Comment on this Routine

there is no data for the current record, and the list is null

*plistname = setnull();

*pdone = TRUE;


Comment on this Routine

link is now an array index

int link;


Comment on this Routine

listpointer is now an array index

typedef in listpointer;


Comment on this Routine

initial

for(i=1;i<=(2*n);i++)

configuration[i] = i;


Comment on this Routine

original

temporiginal = TRUE;

i = 1;

while((i<n+1)&&temporiginal)

if(configuration[i] != i)

temporiginal = FALSE;

else

i++;

original = temporiginal;


Comment on this Routine

process card i and i+n and update i and newi

result[newi] = configuration[i];

result[newi+1] = configuration[i+n];

i++;

newi = newi + 2;


Comment on this Routine

perfectshuffle

i = 1;

while(i < n)

{

result[newi] = configuration[i];

result[newi+1] = configuration[i+n];

i++;

newi = newi + 2;

}

for(i=1;i<=(2*n);i++)

configuration[i] = result[i];


Comment on this Routine

perfectshuffle

configurationpointer l1,newn;

l1 = configuration;

newn = l1;

while(l1 != n)

{

move(l1,n);

l1 =l1->link;

newn = newn->link;

}

n = newn;


Comment on this Routine

move

move(pl1,n)

configurationpointer *pl1,n;

{

configurationpointer hold1,hold2;

hold1 = l1->link;

hold2 = n->link;

11->link = hold2;

n->link = hold2->link;

hold2->link = hold1;

l1 = l1->link;

}


Comment on this Routine

the array must have 53 entries, since the 0th is not used

typedef int arraytype[53];


Comment on this Routine

enter the number of cards in half a deck

printf("\n Enter an integer between 1 and 26 \n");

scanf("%d",&n);


Comment on this Routine

first shuffle

perfectshuffle(configuration,n);


Comment on this Routine

shuffle until original configuration recurs, and keep count

while(!original(configuration,n))

{

perfectshuffle(configuration,n);

numbershuffles++;

}


Comment on this Routine

traverses the configuration and processes each entry

perfectshuffle(configuration,n)


Comment on this Routine

does the processing of an entry

result[newi] = configuration[i];

result[newi+1] = configuration[i+n];


Comment on this Routine

updates

i++;

newi = newi + 2;


Comment on this Routine

variables of type configptr will point to a record of the list containing the configuration

}configrecord,*configptr;


Comment on this Routine

enter the number of cards in half a deck

printf("\n Enter an integer between 1 and 26\n");

scanf("%d",&num);


Comment on this Routine

first shuffle

perfectshuffle(configuration,&n);


Comment on this Routine

shuffle until the original configuration recurs, and keep count

while(!original(configuration,n))

{

perfectshuffle(configuration,&n);

numbershuffles++;

}


Comment on this Routine

traverses the configuration and processes each record

perfectshuffle(configuration,pn)


Comment on this Routine

processes each record

move(&l1,*pn);


Comment on this Routine

updates

l1 = l1->link;

newn = newn->link;


Comment on this Routine

moves top n disk from peg i to peg f

To obtain towers(n,i,a,f):

If n = 1, then

move the top disk from peg i to peg f

else

apply towers(n - 1,i,f,a)

moves the top disks from peg i to peg f

apply towers(n - 1,a,i,f).


Comment on this Routine

special case n is 1

move the top disk from peg i to peg f


Comment on this Routine

moves top n � 1 disks from i to a

apply towers(n - 1,i,f,a)


Comment on this Routine

moves top n - 1 disks from a to f

apply towers(n - 1,a,i,f).


Comment on this Routine

returns zero for a null list

return(0);


Comment on this Routine

second should be set to null when first is null

*psecond = null;


Comment on this Routine

sets second to storage for a copy of first's first record

*psecond = avail();


Comment on this Routine

copies its info field into that storage's info field

setinfo(*psecond,first);


Comment on this Routine

creates a copy of the rest of first and makes rest its head

copy(next(first),&rest);


Comment on this Routine

appends the list rest to the list second

setlink(*psecond,rest);


Comment on this Routine

first is null, so the returned pointer must be null

second = null;


Comment on this Routine

sets second to storage for a copy of first's first record

second = avail();


Comment on this Routine

copies its info field into that storage's info field

setinfo(second,first);


Comment on this Routine

copy returns with a pointer to a copy to the rest of first, and setlink appends that copy to second

setlink(second,copy(next(first)));


Comment on this Routine

explicit case n = 1

return (1);


Comment on this Routine

uses the result for an (n - 1) � (n - 1) board and adds the new possibilities due to another column and row

return(f(n-1)+(n-3)+(n-4));


Comment on this Routine

calls f(n) with n = 6

f(7) = f(6) + (7 - 3) + (7 - 4)


Comment on this Routine

calls f(n) with n = 5

f(6) = f(5) + (6 - 3) + (6 - 4)


Comment on this Routine

calls f(n) with n = 4

f(5) = f(4) + (5 - 3) + (5 - 4)


Comment on this Routine

evaluates f(4) directly

f(4) = 1


Comment on this Routine

evaluates f(5) using f(4)

f(5) = 1 + 2 + 1 = 4


Comment on this Routine

evaluates f(6) using f(5)

f(6) = 4 + 3 + 2 = 9


Comment on this Routine

evaluates f(7) using f(6)

f(7) = 9 + 4 + 3 = 16


Comment on this Routine

creates the initial permutation

initialize(n);


Comment on this Routine

does the processing on the current permutation�for the stable marriage problem this would be a stability check

process(n);


Comment on this Routine

generates the next permutation

next(n,&done);


Comment on this Routine

if there is another permutation

if (n > 1)


Comment on this Routine

the integer n is not at the right end, so shift it

p[l[n]] = p[l[n]+1];

p[l[n]+1] = n;

l[n] = l[n]+1;


Comment on this Routine

n is at the right end

else


Comment on this Routine

recursive call to get the next permutation of 1 to n-1

next(n-1,pdone);


Comment on this Routine

put n at the left end of that permutation

for (i=n-1;i>=1;i--)

p[i+1] = p[i];

p[1] = n;

l[n] = 1;


Comment on this Routine

no next permutation, so done

else

*pdone = TRUE;


Comment on this Routine

the integer n is not at the right end, so shift it

pred[n]->link = ptr[n]->link;

pred[n] = pred[n]->link;

ptr[n]->link = pred[n]->link;

pred[n]->link = ptr[n];


Comment on this Routine

recursive call to get the next permutation of 1 to n-l

next(n-l,pdone);


Comment on this Routine

put n at the left end of that permutation

ptr[n]->link = p->link;

p->link = ptr[n];

pred[n] = p;


Comment on this Routine

no next permutation, so done becomes true

else

*pdone = TRUE;


Comment on this Routine

global definition of arraytype

typedef int arraytype[21];


Comment on this Routine

arrays p and 1 declared globally

arraytype p,l;


Comment on this Routine

creates initial permutation

initialize(n);


Comment on this Routine

processes the current permutation

process(n);


Comment on this Routine

traverses and fills the array p and 1

for (i = 1;i<=n;i++)

{

p[i] = n - i + 1;

l[i] = 1;


Comment on this Routine

traverses the array p

for (i=1;i<=n;i++)

printf("\n%d\n",p[i]);

printf("\n");


Comment on this Routine

global definitions for listrecord, listpointer, pointerarray types

typedef struct record

{

int info;

struct record *link;

}listrecord,*listpointer;

typedef listpointer pointerarray[21];


Comment on this Routine

pointerarrays ptr and pred declared globally

pointerarray ptr,pred;


Comment on this Routine

listpointer p declared globally

listpointer p;


Comment on this Routine

creates the list p and fills pointer arrays ptr and pred

for (i=1;i<=n;i++)

{

q->link = malloc(sizeof(listrecord));

q = q->link;

q->info = n - i + 1;

ptr[n-i+1] = q;

pred[i] = p;

}

q->link = NULL;


Comment on this Routine

traverses the permutation list

while (q != NULL)

{

printf("\n%d\n",q->info);

q = q->link;

}


Comment on this Routine

in the nonrecursive version, a retrn value of 1 means this call has completed

towers(n-1,i,f,a);


Comment on this Routine

in the nonrecursive version, a retrn value of 2 means this call has completed; in this version an "implicit" exit then occurs; this must be made explicit in the implementation

towers(n-1,a,i,f);


Comment on this Routine

retrn holds the place to continue after a recursive call completes

int retrn;


Comment on this Routine

storage allocated for the stack s

stack s;


Comment on this Routine

initializes stack s to empty

setstack(&s);


Comment on this Routine

the start of towers

one: if(n == 1)


Comment on this Routine

saves the return and n, i, a and f on stack s

s_tack(1,n,i,a,f &s);


Comment on this Routine

sets scratchpad values: n to n-1, interchanges f and a

setvar1(&n,&i,&a,&f);


Comment on this Routine

makes a recursive call

goto one;


Comment on this Routine

this is the start of the "explicit" exit; when the stack is empty, the original call to towers is complete; otherwise the return point must be determined

two: if(!empty(&s))


Comment on this Routine

sets retrn and the scratchpad to their proper values

restore(&retrn,&n,&i,&a,&f,&s);


Comment on this Routine

after executing the printf statement, a recursive call must be made, so return and n, i, a, f are saved on s and the scratchpad values are set: n to n-1, a and i are interchanged, and the recursive call is made

case 1:

printf("\n %d -> %d\n",

i,f)

s_tack(2,n,i,a,f,&s);

setvar2(&n,&i,&a,&f);

goto one;


Comment on this Routine

a recursive call has completed; the "explicit" exit must now be made, so go to the "explicit" exit

case 2:

goto two;


Comment on this Routine

retrn, the switch, and one call to s_tack have been eliminated; also, s_tack has one less parameter

restore(&n,&i,&a,&f,&s);

printf("\n %d -> %d\n",i,f);

setvar2(&n,&i,&a,&f);

goto one;

}


Comment on this Routine

retrn holds the place to continue after a recursive call completes

int i,retrn;


Comment on this Routine

the start of next

one: if(n > 1)


Comment on this Routine

saves the return, n, and i on stack s

s_tack(1,n,i,&s);


Comment on this Routine

sets scratchpad values: n to n-1

setvar(&n,&i);


Comment on this Routine

this is the start of the "explicit" exit; when the stack is empty, the original call to next is complete; otherwise the return point must be determined

two: if(!empty(&s))


Comment on this Routine

sets retrn and the scratchpad to their proper values

restore(&retrn,&n,&i,&s);


Comment on this Routine

after executing the for loop, a recursive call has completed; insert n at the left end, and the

case 1:

{

for(i=n-1;i>=1;i--)

p[i+1] = p[i];

p[1] = n;

l[n] = 1;


Comment on this Routine

go to the "explicit" exit

goto two;


Comment on this Routine

c1 is (n > 1)

one: if c1 then


Comment on this Routine

c2 is (1[n] < n)

if c2 then


Comment on this Routine

the new while loop

while not empty(&s)


Comment on this Routine

another loop

while (c1 and not c2)


Comment on this Routine

a decision

if c1, then


Comment on this Routine

the new while loop

while not empty (&s)


Comment on this Routine

this differs from the first version only in its structure, not in its operations

next(n,pdone)


Comment on this Routine

the type stackrecord is a record with one member info of type whatever

typedef struct

{

whatever info;

}stackrecord;


Comment on this Routine

the type stack is a record with members an array of type stackrecord and top of type int

typedef struct

{

stackrecord stackarray [LIMIT];

int top;

}stack;


Comment on this Routine

s is an instance of type stack, and storage is allocated for it

stack s;


Comment on this Routine

assume newrecord contains the number 5

push(&newrecord,&s);


Comment on this Routine

sets top to - 1, ti signify empty

(*ps).top = -1;


Comment on this Routine

top = - 1 signifies empty

return ((*ps).top == -1);


Comment on this Routine

overflow test

if ((*ps).top == (LIMIT - 1))


Comment on this Routine

no overflow, so updates top and inserts the new record where top points

(*ps).top = (*ps).top + 1;

(*ps).stackarray[(*ps).top].info =

(*pnewrecord).info;


Comment on this Routine

underflow test

if(empty(ps))


Comment on this Routine

no underflow, so copies the top record's contents into value and updates top to reflect its removal

(*pvalue).info = (*ps).stackarray

[(*ps).top].info;

(*ps).top = (*ps).top - 1;


Comment on this Routine

the stackarray now no longer holds records, just simple variables of type whatever

typedef struct

{

whatever stackarray [LIMIT];

int top;

}stack;


Comment on this Routine

now only a simple variable must be inserted

(*ps).stackarray[(*ps).top] = value;


Comment on this Routine

now only a simple variable must be copied

*pvalue = (*ps).stackarray[(*ps).top];


Comment on this Routine

Definitions of types and operations for the data abstraction stack: whatever, stackrecord, stack and setstack, empty, push, pop, overflow, and underf1ow

#define LIMIT 50

typedef struct

{

int n;

int i;

int a;

int f;

}whatever;

typedef struct

{

whatever info;

}stackrecord;

typedef struct

{

stackrecord stackarray [LIMIT];

int top;

}stack;

setstack(ps)

/* Initializes the stack s to empty */

stack *ps;

{

(*ps).top = -1;

}

empty(ps)

/* Returns true only if the stack s is empty */

stack *ps;

{

return((*ps).top == -1);

}

push(pnewrecord,ps)

/* Inserts the record newrecord

at the top of the stack s

*/

stackrecord *pnewrecord;

stack *ps;

{

if ((*ps).top == (LIMIT-1))

overflow(ps);

else

{

(*ps).top = (*ps).top + 1;

(*ps).stackarray[(*ps).top].info =

(*pnewrecord).info;

}

}

pop(ps,pvalue)

/* Copies the contents of the top record

of stack s into value and removes the

record from the stack

*/

stack *ps;

stackrecord *pvalue;

{

if (empty(ps))

underflow(ps);

else

{

(*pvalue).info =

(*ps).stackarray[(*ps).top].info;

(*ps).top = (*ps).top - 1;

}

}

overflow(ps);

/* Prints a message that the stack has overflowed */

stack *ps;

{

printf(" The stack has overflowed \n");

}

underflow(ps);

/* Prints a message that the stack has underflowed */

stack *ps;

{

printf(" The stack has underflowed \n");

}


Comment on this Routine

final nonrecursive version

towers(n,i,a,f)


Comment on this Routine

allocates storage for s, an instance of type stack

stack s;


Comment on this Routine

depends on the implementation of the stack, only because it must know the structure of the stack records

s_tack(n,i,a,f,ps)


Comment on this Routine

allocates storage for newrecord (an instance of type stackrecord), fills in its fields, and puts it on s

stackrecord newrecord;

newrecord.info.n = n;

newrecord.info.i = i;

newrecord.info.a = a;

newrecord.info.f= f;

push(&newrecord,ps);


Comment on this Routine

depends on the implementation of the stack, only because it must know the structure of the stack records

restore(pn,pi,pa,pf,ps)


Comment on this Routine

the type stackrecord is a record with members info of type whatever and link of type pointer to a stackrecord; the type stack is a pointer to a stackrecord

typedef struct record

{

struct whatever info;

struct record *link;

}stackrecord,*stack;


Comment on this Routine

a null pointer in s now signifies an empty stack

*ps = NULL;


Comment on this Routine

the stack is empty if s contains a null pointer

return(*ps == NULL);


Comment on this Routine

puts a pointer to storage allocated for the new record into newentry and inserts the new record on stack s; this version does not check for overflow but could do so (how to do it depends on what malloc does when it runs out of storage)

newentry = malloc(sizeof(stackrecord));

newentry->info = pnewrecord->info;

newentry->link = *ps;

*ps = newentry;

}


Comment on this Routine

tests for underflow

if(!empty(ps))


Comment on this Routine

no underflow, so removes the top record on stacks s and copies its contents into value

pvalue->info = (*ps)->info;

*ps = (*ps)->link;


Comment on this Routine

definitions of types and operations for the data abstraction stack: whatever, stackrecord, stack and setstack, empty, push, pop, and underflow; these differ from those in the corresponding program of Section 4.5.1

#include <stdio.h>

#define NULL 0

typedef struct

{

int n;

int i;

int a;

int f;

}whatever;

typedef struct record

{

whatever info;

struct record *link;

}stackrecord, *stack;

setstack(ps)

/* Initializes the stack s to empty */

stack *ps;

{

*ps = NULL;

}

empty(ps)

stack *ps;

{

return(*ps == NULL);

}

push(pnewrecord,ps)

stackrecord *pnewrecord;

stack *ps;

{

stackrecord *newentry;

newentry = malloc(sizeof(stackrecord));

newentry->info = pnewrecord->info;

newentry->link = *ps;

*ps = newentry;

}

pop(ps,pvalue)

/* Copies the contents of the top record

of stack s into value and removes the

record from the stack

*/

stack *ps;

stackrecord *pvalue;

{

if(!empty(ps))

{

pvalue->info = (*ps)->info;

*ps = (*ps)->link;

}

else

underflow(ps);

}

overflow(ps);

/* Prints a message that the stack has overflowed */

stack *ps;

{

printf(" The stack has overflowed \n");

}

underflow(ps);

/* Prints a message that the stack has underflowed */

stack *ps;

{

printf(" The stack has underflowed \n");

}


Comment on this Routine

the rest of the program is exactly the same as the corresponding part of the program of Section 4.5.1

main()

/* Driver for towers */

{

int n;

printf("\n enter a value for n \n");

scanf("%d",&n);

towers(n,1,2,3);

}

towers(n,i,a,f)

1 final nonrecursive version

/* Moves the top n disks

from peg i to peg f

*/

int n,i,a,f;

{

stack s;

1 allocates storage for s (an instance of type stack)

int done

setstack(&s);

done = FALSE;

while(!done)

{

while (n > 1)

{

s_tack(n,i,a,f,&s);

setvar 1(&n,&i,&a,&f);

}

printf("\n %d -> %d\n",i,f);

if(!empty(&s))

{

restore(&n,&i,&a,&f,&s);

printf("\n %d -> %d\n",i,f);

setvar2(&n,&i,&a,&f);

}

else

done = TRUE;

}

setvar1(pn,pi,pa,pf)

/* Sets n to n - 1 and

interchanges f and a

*/

int *pn,*pi,*pa,*pf;

{

int t;

*pn = *pn - 1;

t = *pf;

*pf = *pa;

*pa = t;

}

setvar2(pn,pi,pa,pf)

/* Sets n to n - 1 and

interchanges a and i

*/

int *pn,*pi,*pa,*pf;

{

int t;

*pn = *pn - 1;

t = *pa;

*pa = *pi;

*pi = t;

}

s_tack(n,i,a,f,ps)

/* Creates a record containing

n, i, a, and f and inserts it

on stack s

*/

int n,i,a,f;

stack *ps;

{

stackrecord newrecord;

newrecord.info.n = n;

newrecord.info.i = i;

newrecord.info.a = a;

newrecord.info.f = f;

push(&newrecord,ps);

}

restore(pn,pi,pa,pf,ps)

/* Removes the top record from

stack s and copies its contents

into n,i,a,f

*/

int *pn,*pi,*pa,*pf;

stack *ps;

{

stackrecord value;

pop(ps,&value);

*pn = value.info.n;

*pi = value.info.i;

*pa = value.info.a


Comment on this Routine

allocates storage for newrecord, an instance of type stackrecord, fills in its field, and puts it on s

stackrecord newrecord;

newrecord.info.n = n;

newrecord.info.i = i;

newrecord.info.a = a;

newrecord.info.f = f;

push(&newrecord,ps);


Comment on this Routine

depends on the implementation of the stack only because it must know the structure of the stack records

restore(pn,pi,pa,pf,ps)


Comment on this Routine

the type queuerecord is a record with one member info of type whatever

typedef struct

{

whatever info;

}queuerecord;


Comment on this Routine

the type queue is a record with members an array of type queuerecord and front and rear of type int

typedef struct

{

queuerecord queuearray [LIMIT];

int front, rear;

}queue;


Comment on this Routine

q is an instance of type queue, and storage is allocated for it

queue q;


Comment on this Routine

front= -1 signifies queue empty

if ((*pq).front == -1)


Comment on this Routine

empty, so update front to point to the start of the array, insert the new entry there and set rear to the next location

(*pq).front = 0;

(*pq).queuearray[0].info = (*pnewrecord).info;

(*pq).rear = 1;


Comment on this Routine

test for overflow which occurs when the queue is full, signified by front = rear

if((*pq).rear ==  (*pq).front)

printf("\n overflow \n");


Comment on this Routine

rear <= LIMIT - 1 signifies that rear must wrap around

if((*pq).rear <= (LIMIT - 1))


Comment on this Routine

add new entry and move rear down

(*pq).queuearray[(*pq).rear].info =

(*pnewrecord).info;

(*pq).rear = (*pq).rear + 1;


Comment on this Routine

rear must now wrap around to the start of the array; if front = 0, then the array is full, so it cannot do so

{

if((*pq).front == 0)

printf("\n overflow \n");


Comment on this Routine

underflow test

if(empty(pq))


Comment on this Routine

copies front record

(*pvalue).info = (*pq).queuearray

[(*pq).front].info;


Comment on this Routine

deletes it by moving front down

(*pq).front = (*pq).front + 1;


Comment on this Routine

front > LIMIT - 1 signifies that it must wrap around

if((*pq).front > (LIMIT - 1))

(*pq).front = 0;


Comment on this Routine

s is an instance of type stack, and storage is allocated to it

stack s;


Comment on this Routine

s initialized to empty

setstack(&s);


Comment on this Routine

reads into and tests currentchar

while(readchar(&currentchar))


Comment on this Routine

if a "(", then puts it on the stack

if(leftparen(currentchar))

push(&currentchar,&s);


Comment on this Routine

if a ")" and if the stack is not empty, then pops the stack into topcharacter; now two "adjacent" parentheses have been removed

else if(rightparen(currentchar))

if(!empty(&s))

{

pop(&s,&topcharacter);


Comment on this Routine

test for a match

if(!match(topcharacter.topchar,

currentchar))


Comment on this Routine

do desired processing of the nonparenthesis

process(currentchar,&s);


Comment on this Routine

if s is not empty when all input is complete, then unmatched parentheses remain

if(!empty(&s))

*pvalid = FALSE;


Comment on this Routine

static to retain its value between calls

static int more;


Comment on this Routine

reads into currentchar, updates more

more = readchar(&currentchar);


Comment on this Routine

determines if the input sequence between this left parenthesis and the matching "right parenthesis is valid; the left parenthesis is saved in last

determine(pvalid,currentchar);


Comment on this Routine

test if the left parenthesis (in last) in fact matches the right parenthesis just read (which is now in currentchar)

if(!match (last,currentchar))


Comment on this Routine

set done to signify a match

done = TRUE;


Comment on this Routine

reinitialize last

last = SPECIALCHAR;


Comment on this Routine

process nonparenthesis

process(last,currentchar);


Comment on this Routine

last ! = SPECIALCHAR means left parentheses remain unmatched

if(last != SPECIALCHAR)

*pvalid = FALSE;


Comment on this Routine

reads in an integer n and calls primes to print all prime numbers �n

main()


Comment on this Routine

these definitions and functions give a specific implementation of a data abstraction: collection with operations set, insert, omit, and belongs allowed. This is the only portion of the program that needs to be changed when the implementation of the data abstraction is changed. MAXN sets a limit on the size of arrays of type collection, so it sets a limit for n.

#define TRUE 1

#define FALSE 0

#define MAXN 500

typedef int collection[MAXN];

set(n,c)

/* Sets c to empty */

int n;

collection c;

{

int i;

for (i=1;i<=n;i++)

c[i] = FALSE;

}

insert(i,c)

/* Inserts i in c */

int i;

collection c;

{

c[i] = TRUE;

}

omit(i,c)

/* Removes i from c */

int i;

collection c;

{

c[i] = FALSE;

}

belongs(i,c)

/* Returns true only

if i is in c

*/

int i;

collection c;

{

return(c[i]);

}


Comment on this Routine

this portion of the program is independent of the specific implementation of the data abstraction. It does not have to be changed if the implementation of the data abstraction changes. Candidates is an instance of the data structure of type collection.

primes(n)

/* Prints all prime numbers between 2 and n */

int n;

{

collection candidates;

create(n,candidates);

remove(n,candidates);

print(n,candidates);

}

create(n,c)

/* Creates a collection of integers between 2 and n */

int n;

collection c;

{

int i;

set(n,c);

for (i=2;i<=n;i++)

insert(i,c);

}

remove(n,c)

/* Removes all nonprimes from c */

int n;

collection c;

{

int firstprime;

int factor;

firstprime = 2;

factor = firstprime;

while (nonprimes(factor,n))

{

delete(factor,n,c);

factor++;

}

}

nonprimes(factor,n)

/* Returns true only if non

primes may remain in c

*/

int factor,n;

double sqrt();

{

return(factor <= sqrt((double)n));

}

delete(factor,n,c)

/* Deletes multiples of factor from c */

int factor,n;

collection c;

{

int nextmultiple;

nextmultiple = 2 * factor;

while (nextmultiple <= n)

{

omit(nextmultiple,c);

nextmultiple = nextmultiple + factor;

}

}

print(n,c)

/* Prints integers in c */

int n;

collection c;

{

int i;

for (i=2;i<=n;i++)

if (belongs(i,c))

printf("\n %d\n",i);

}


Comment on this Routine

for all values of factor from 2 to n, invokes delete to remove multiples of factor from c

factor = firstprime;

while (nonprimes(factor,n))

{

delete(factor,n,c);

factor++;

}


Comment on this Routine

invokes omit to remove each nonprime from c that is a multiple of factor

nextmultiple = 2 * factor;

while (nextmultiple <= n)

{

omit(nextmultiple,c);

nextmultiple = nextmultiple + factor;

}


Comment on this Routine

invokes belongs for each integer from 2 to n, to determine if it is a prime, and if so prints it

for (i=2;i<=n;i++)

if (belongs(i,c))

printf("\n %d\n",i);


Comment on this Routine

replaced by the contents of the file collection

#include "collection"


Comment on this Routine

for all values of factor from 2 to n, invokes delete to remove multiples of factor from c

factor = firstprime;

while (nonprimes(factor,n))

{

delete(factor,n,c);

factor++;

}

}


Comment on this Routine

These definitions and functions give a specific implementation of a data abstraction: collection with operations set, insert, omit, and belongs allowed. This is the only portion of the program that needs to be changed when the implementation of the data abstraction is changed. MAXN sets a limit on the size of arrays of type collection so it sets a limit for n.

#define TRUE 1

#define FALSE 0

#define MAXN 500

typedef int collection [MAXN];

set(n,c)

/* Sets c to empty */

int n;

collection c;

{

int i;

for (i=1;i<=n;i++)

c[i] = FALSE;

}

insert(i,c)

/* Inserts i in c */

int i;

collection c;

{

c[i] = TRUE;

}

omit(i,c)

/* Removes i from c */

int i;

collection c;

{

c[i] = FALSE;

}

belongs(i,c)

/* Returns true only

if i is in c

*/

int i;

collection c;

{

return(c[i]);

}


Comment on this Routine

replaced by the contents of the file collection.h

#include "collection.h"


Comment on this Routine

replaced by the contens of the file collection.h

#nclude "collection.h"


Comment on this Routine

This portion of the program is independent of the specific implementation of the data abstraction. It does not have to be changed if the implementation of the data abstraction changes. Candidates is an instance the data structure of type collection.

primes(n)

/* Prints all prime numbers between 2 and n */

int n;

{

collection candidates;

create(n,candidates);

remove(n,candidates);

print(n,candidates);

}

create(n,c)

/* Creates a collection of integers between

2 and n */

int n;

collection c;

{

int i;

set(n,c);

for (i=2;i<=n;i++)

insert(i,c);

}

remove(n,c)

/* Removes all nonprimes from c */

int n;

collection c;

{

int firstprime;

int factor;

firstprime = 2;

factor = firstprime;

while (nonprimes(factor,n))

{

delete(factor,n,c);

factor++;

}

}

nonprimes(factor,n)

/* Returns true only if non

primes may remain in c

*/

int factor,n;

double sqrt();

{

return(factor <= sqrt((double)n));

}

delete(factor,n,c)

/* Deletes multiples of factor from c */

int factor,n;

collection c;

{

int nextmultiple;

nextmultiple = 2 * factor;

while (nextmultiple <= n)

{

omit(nextmultiple,c);

nextmultiple = nextmultiple + factor;

}

}

print(n,c)

/* Prints integers in c */

int n;

collection c;

{

int i;

for (i=2;i<=n;i++)

if (belongs(i,c))

printf("\n %d\n",i);

}


Comment on this Routine

replaced by the contents of file pairings.h

#include "pairings.h"


Comment on this Routine

replaced by the contents of file preferences.h

#include "preferences.h"


Comment on this Routine

replaced by the contents of file list.h

#include "list.h"


Comment on this Routine

this portion of the program is independent of the list implementation

#include <stdio.h>

#define TRUE 1

#define FALSE 0

main()

/* Inputs a series of stockrecords,

creates a list of the records, and

prints the contents of each list

record's infofield

*/

{

listpointer stocks,recordpointer,next();

int done;

initialize(&stocks);

recordpointer = stocks;

done = FALSE;

while(!done&&anotherrecord(recordpointer))

{

addrecord(&stocks,recordpointer,&done);

recordpointer = next(recordpointer);

}

recordpointer=stocks;

while (anotherrecord(recordpointer))

{

printrecord(recordpointer);

recordpointer = next(recordpointer);

}

}

initialize(plistname)

/* Allocates storage for the first record

and sets listname to point to it

*/

listpointer *plistname;

{

listpointer avail();

*plistname = avail();

}

addrecord(plistname,recordpointer,pdone)

/* Fills in the current record's data, and

allocates storage for the next record and

adds it to the list. If there is no data for

the current record it sets the link field of

the last list record to null, or the list

head, to null, and done to true.

*/

listpointer *plistname,recordpointer;

int *pdone;

{

static listpointer predecessor;

listpointer setnull(),avail(),

getnextrecord(),pointer;

pointer = getnextrecord();

if (pointer != setnull())

{

setinfo(recordpointer,pointer);

setlink(recordpointer,avail());

predecessor = recordpointer;

}

else if (*plistname != recordpointer)

{

setlink(predecessor,setnull());

*pdone = TRUE;

}

else

{

*plistname = setnull();

*pdone = TRUE;

}


Comment on this Routine

this is the traversal to input data and create the list

initialize(&stocks);

recordpointer = stocks;

done = FALSE;

while(!done&&anotherrecord(recordpointer))

{

addrecord(&stocks,recordpointer,&done);

recordpointer = next(recordpointer);

}


Comment on this Routine

fills in the current record, allocates storage for the next and adds it to the list, and, if there is no data for the current record, terminates the list and sets done to true

addrecord(&stocks,recordpointer,&done);


Comment on this Routine

this is the traversal to print the list

recordpointer=stocks;

while (anotherrecord(recordpointer))

{

printrecord(recordpointer);

recordpointer = next(recordpointer);

}


Comment on this Routine

replaced by the contents of file stack.h

#include "stack.h"


Comment on this Routine

replaced by the contents of file stack_record.h

#include "stack_record.h"


Comment on this Routine

new call to stack record operation

setinfo(newentry,pnewrecord);


Comment on this Routine

new call to stack record operation

setinfo(pvalue,(*ps));


Comment on this Routine

storage allocated for stack s

stack s;


Comment on this Routine

current record pointer set to the first record of ls

ptr = *pls;


Comment on this Routine

initialize stack s

setstack(&s);


Comment on this Routine

test for a current record to be processed or for more saved pointers

while((ptr ! = null) | | (!empty(&s)))


Comment on this Routine

ptr ! = null signifies there is a current record to be processed

if(ptr ! = null)


Comment on this Routine

process it

process (pls, ptr);


Comment on this Routine

if current record is complex

if (complex(ptr))


Comment on this Routine

save pointer to its successor and update current record pointer to the first record of its sublist

push next(ptr),&s);

ptr = sublist(ptr);


Comment on this Routine

update current record pointer to its successor record

ptr = next(ptr);


Comment on this Routine

the stack contains the pointer to the next record to access, so set current record pointer to it and pop the stack

pop(&s,&ptr);


Comment on this Routine

test for current record to be processed

while(ptr != null)


Comment on this Routine

process it

process(pls,ptr);


Comment on this Routine

if the current record has a sublist

if(complex(ptr))


Comment on this Routine

set nextsublist to it

nextsublist = sublist(ptr);


Comment on this Routine

traverse it

traverse(&nextsublist);


Comment on this Routine

current record not complex means it contains a stock record, so print its contents if bought early enough

if(!complex(ptr) && prior(ptr,&date))

printinfo (ptr);


Comment on this Routine

total will retain its value between calls

static float total;


Comment on this Routine

if the current record is complex, has a non-null sublist, and the first record of that sublist is not complex, then the current record must be a record of a client, so

if(complex(ptr))

if(sublist(ptr) != null)

if(!complex(sublist(ptr)))


Comment on this Routine

total must be initialized

total = 0.0;


Comment on this Routine

the client's name printed

printname(ptr);


Comment on this Routine

otherwise it must be a record on the client's portfolio list, so

else


Comment on this Routine

total must be increased

total = total + value(ptr);


Comment on this Routine

if this is the last porfolio, print the total

if(next(ptr) == null)

printf("\n Portfolio value is

%12.2f \n",total);


Comment on this Routine

currentlist = 1 means the search is now for a chapter

#define CH 1


Comment on this Routine

currentlist = 2 means the search is now for a page

#define PAGE 2


Comment on this Routine

currentlist = 3 means the search is now for a paragraph

#define PAR 3


Comment on this Routine

set ptr to the first chapter

ptr = book;


Comment on this Routine

set chapter searched for to c

currentnumber = c;


Comment on this Routine

indicate that the search is now for a chapter

currentlist = CH;


Comment on this Routine

test for completion of present search or no more records of desired type

while(!found && (ptr != null))


Comment on this Routine

if current record isn't the one being sought, update ptr

if(currentnumber != number(ptr))

ptr = next(ptr);


Comment on this Routine

otherwise

else


Comment on this Routine

determine what type is being sought

switch(currentlist)


Comment on this Routine

chapter found, so update

case CH:


Comment on this Routine

a page is to be found, its number is pg; start looking in the chapter's sublist

currentlist = PAGE;

currentnumber = pg;

ptr = sublist(ptr);

break;


Comment on this Routine

page found, so update

case PAGE:


Comment on this Routine

a paragraph is to be found, number p; start looking in the paragraph's sublist

currentlist = PAR;

currentnumber = p;

ptr = sublist(ptr);

break;


Comment on this Routine

paragraph found, so search is finished successfully; the traversal will be terminated

case PAR:

found = TRUE;


Comment on this Routine

if successful search

if(found)


Comment on this Routine

replace old paragraph with the new one

setsublist(ptr,newp);


Comment on this Routine

count initialized

count = 1;


Comment on this Routine

count will contain the number of the current search type

while(!found && (ptr != null))


Comment on this Routine

if the current record has the number sought

if(currentnumber != count)


Comment on this Routine

update count

count++;


Comment on this Routine

chapter located, so reinitialize count

count = 1;


Comment on this Routine

page located, so reinitialize count

count = 1;


Comment on this Routine

implementation for accounts records stored in dynamic memory

typedef struct record

{

int kind;

int info;

struct record *sublist;

struct record *link;

}listrecord, *liststructurepointer;


Comment on this Routine

implementation for accounts records stored in an array of records of type listrecord

typedef struct record

{

int kind;

int info;

int sublist;

int link;

}listrecord;


Comment on this Routine

sublist now an array index

int sublist;


Comment on this Routine

link now an array index

int link;


Comment on this Routine

liststructurepointer now an array index

typedef int liststructurepointer;


Comment on this Routine

implementation of polynomial records stored in dynamic memory

typedef union

{

char unknown;

polyrecord *sublist;

int value;

}varying;

typedef struct record

{

int tag;

varying secondfield;

int exponent;

struct record *nextterm;

}polyrecord, *liststructurepointer;


Comment on this Routine

implementation of polynomial records stored in an array of records of type polyrecord

typedef union

{

char unknown;

int sublist;

int value;

}varying;

typedef struct

{

int tag;

varying secondfield;

int exponent;

int nextterm;

}polyrecord;


Comment on this Routine

sublist is now an array index

int sublist;


Comment on this Routine

nextterm is now an array index

int nextterm;


Comment on this Routine

liststructurepointer is now an array index

typedef liststructurepointer;


Comment on this Routine

definitions for the liststructure data abstraction

typedef struct record

{

int info;

int kind;

struct record *sublistptr;

struct record *link;

}examplerecord,*liststructurepointer;

#define NULL 0

liststructurepointer next(p)

/* Returns a copy of the link field

value of record pointed to by p.

*/

liststructurepointer p;

{

return(p->link);

}

liststructurepointer sublist(p)

/* Returns a copy of the sublistpointer

field value of record pointed to by p.

*/

liststructurepointer p;

{

return(p->sublistptr);

}

liststructurepointer setnull()

/* Returns a null pointer */

{

return(NULL);

}

liststructurepointer avail()

/* Returns a pointer to storage allocated

for a liststructure record of type

examplerecord.

*/

{

return(malloc(sizeof(examplerecord)));

}

complex(p)

/* Returns true only if the record pointed

to by p is complex.

*/

liststructurepointer p;

{

return(p->kind == 1);

}

createrecord(l,ptr)

/* Reads in the data for the current record

and sets its fields appropriately.

*/

liststructurepointer 1,ptr;

{

int link,subptr,value,kind;

liststructurepointer setnull(),avail();

printf("\n enter info \n");

scanf("%d",&value);

printf("\n enter 0 for simple 1

for complex \n");

scanf("%d",&kind);

if (kind == 0)

{

printf("\n enter link \n");

scanf("%d",&link);

if (link == 0)

ptr->link = setnull();

else

ptr->link = avail();

}

else

{

printf("\n enter link & subptr \n");

scanf("%d %d",&link,&subptr);

if (link == 0)

ptr->link = setnull();

else

ptr->link = avail();

if (subptr == 0)

ptr->sublistptr = setnull();

else

ptr->sublistptr = avail();

}

ptr->info = value;

ptr->kind = kind:

}

printrecord(pls,ptr)

/* Prints the contents of the info field

of the record pointed to by ptr.

*/

liststructurepointer *pls,ptr;

{

printf("\n %d \n",ptr->info);


Comment on this Routine

the null pointer will be 0

#define NULL 0


Comment on this Routine

basic operations on liststructure

liststructurepointer next(p)

/* Returns a copy of the link field

value of record pointed to by p.

*/

liststructurepointer p;

{

return(p->link);

}

liststructurepointer sublist(p)

/* Returns a copy of the sublistpointer

field value of record pointed to by p.

*/

liststructurepointer p;

{

return(p->sublistptr);

}

liststructurepointer setnull()

/* Returns a null pointer */

{

return(NULL);

}

liststructurepointer avail()

/* Returns a pointer to storage allocated

for a liststructure record of type

examplerecord.

*/

{

return(malloc(sizeof(examplerecord)));

}

complex(p)

/* Returns true only if the record pointed

to by p is complex.

*/

liststructurepointer p;

{

return(p->kind == 1);

}

createrecord(l,ptr)

/* Reads in the data for the current record

and sets its fields appropriately.

*/

liststructurepointer 1,ptr;

{

int link,subptr,value,kind;

liststructurepointer setnull(),avail();

printf("\n enter info \n");

scanf("%d",&value);

printf("\n enter 0 for simple 1

for complex \n");

scanf("%d",&kind);

if (kind == 0)

{

printf("\n enter link \n");

scanf("%d",&link);

if (link == 0)

ptr->link = setnull();

else

ptr->link = avail();

}

else

{

printf("\n enter link & subptr \n");

scanf("%d %d",&link,&subptr);

if (link == 0)

ptr->link = setnull();

else

ptr->link = avail();

if (subptr == 0)

ptr->sublistptr = setnull();

else

ptr->sublistptr = avail();

}

ptr->info = value;

ptr->kind = kind:

}

printrecord(pls,ptr)

/* Prints the contents of the info field

of the record pointed to by ptr.

*/

liststructurepointer *pls,ptr;

{

printf("\n %d \n",ptr->info);

}


Comment on this Routine

kind = 1 will signify a complex record

return(p->kind == 1);


Comment on this Routine

reads information to go in the info field

scanf("%d",&value);


Comment on this Routine

reads the information to go in the kind field

scanf("%d",&kind);


Comment on this Routine

if a simple record

if (kind == 0)


Comment on this Routine

read link data

scanf("%d",&link);


Comment on this Routine

if link is null

if (link == 0)


Comment on this Routine

set it to null

ptr->link = setnull();


Comment on this Routine

set it to storage allocated for the next record

ptr->link = avail();


Comment on this Routine

read link and sublist data

scanf("%d %d",&link,&subptr);


Comment on this Routine

if sublistptr is null

if (subptr == 0)


Comment on this Routine

set it to null

ptr->sublistptr = setnull();


Comment on this Routine

set it to storage allocated for the first sublist record

ptr->sublistptr = avail();


Comment on this Routine

set the info field

ptr->info = value;


Comment on this Routine

set the kind field

ptr->kind = kind:


Comment on this Routine

l is the name of the liststructure created

liststructurepointer l,avail();


Comment on this Routine

if liststructure input is null

if (n == 0)


Comment on this Routine

set l to null

l = NULL;


Comment on this Routine

set l to allocated storage

l = avail()


Comment on this Routine

create l

create(1);


Comment on this Routine

this traverse prints all records

traverse(&l);


Comment on this Routine

prints message to indicate that l is the null liststructure

if (l == NULL)

printf("\n The list is null \n");


Comment on this Routine

definitions for the stack data abstraction�its entries will be pointers to liststructure records

#define LIMIT 50

typedef liststructurepointer whatever;

typedef struct

{

whatever stackarray[LIMIT];

int top;

}stack;

setstack(ps)

/* Sets stack s to empty */

stack *ps;

{

(*ps).top = -1;

}

empty(ps)

/* Returns true only if

stack s is empty.

*/

stack *ps;

{

return((*ps).top == -1);

}

push(value,ps)

/* Inserts contents of value as

top entry of stack s.

*/

whatever value;

stack *ps;

{

if ((*ps).top == (LIMIT - 1))

overflow(ps);

else

{

(*ps).top = (*ps).top + 1;

(*ps).stackarray[(*ps).top] = value;

}

}

pop(ps,pvalue)

/* Removes the top entry of stack s

and places its contents in value.

*/

stack *ps;

whatever *pvalue;

{

if (empty(ps))

underflow(ps);

else

{

*pvalue = (*ps).stackarray[(*ps).top];

(*ps).top = (*ps).top - 1;

}

}

overflow(ps)

/* Prints a message when

the stack overflows.

*/

stack *ps;

{

printf("\n stack overflow \n");

}

underflow(ps)

/* Prints a message when

the stack underflows.

*/

stack *ps;

{

printf("\n stack underflow \n");

}


Comment on this Routine

basic operations for the stack

setstack(ps)

/* Sets stack s to empty */

stack *ps;

{

(*ps).top = -1;

}

empty(ps)

/* Returns true only if

stack s is empty.

*/

stack *ps;

{

return((*ps).top == -1);

}

push(value,ps)

/* Inserts contents of value as

top entry of stack s.

*/

whatever value;

stack *ps;

{

if ((*ps).top == (LIMIT - 1))

overflow(ps);

else

{

(*ps).top = (*ps).top + 1;

(*ps).stackarray[(*ps).top] = value;

}

}

pop(ps,pvalue)

/* Removes the top entry of stack s

and places its contents in value.

*/

stack *ps;

whatever *pvalue;

{

if (empty(ps))

underflow(ps);

else

{

*pvalue = (*ps).stackarray[(*ps).top];

(*ps).top = (*ps).top - 1;

}

}

overflow(ps)

/* Prints a message when

the stack overflows.

*/

stack *ps;

{

printf("\n stack overflow \n");

}

underflow(ps)

/* Prints a message when

the stack underflows.

*/

stack *ps;

{

printf("\n stack underflow \n");


Comment on this Routine

this nonrecursive traversal uses a stack s1

traverse(pls)

/* Prints the info fields of all

records of the liststructure l.

*/

liststructurepointer *pls;

{

liststructurepointer ptr,null,setnull(),

next(),sublist();

stack s1;

null = setnull();

ptr = *pls;

setstack(&s1);

while ((ptr != null)||(!empty(&s1)))

if (ptr != null)

{

printrecord(pls,ptr);

if (complex(ptr))

{

push(next(ptr),&s1);

ptr = sublist(ptr);

}

else

ptr = next(ptr);

}

else

pop(&s1,&ptr);


Comment on this Routine

plays the role of process for this traversal

printrecord(pls,ptr);


Comment on this Routine

this nonrecursive traversal uses a stack s2

create(l)

/* Inputs records and creates the list l */

liststructurepointer l;

{

liststructurepointer ptr,null,setnull(),

next(),sublist();

stack s2;

null = setnull();

ptr = l;

setstack(&s2);

while ((ptr != null)||(!empty(&s2)))

if (ptr != null)

{

createrecord(l,ptr);

if (complex(ptr))

{

push(next(ptr),&s2);

ptr = sublist(ptr);

}

else

ptr = next(ptr);

}

else

pop(&s2,&ptr);

}


Comment on this Routine

plays the role of process for this traversal

createrecord(l,ptr);


Comment on this Routine

set p to allocated storage

p = avail();


Comment on this Routine

set the new info field

setinfo(p,&value);


Comment on this Routine

set the left pointer

setleft(p,null);


Comment on this Routine

set the right pointer

setright(p,null);


Comment on this Routine

set the predecessor's pointer ( the right pointer in this case)

setright(predecessor,p);


Comment on this Routine

saves the pointer to the old node

q = right(predecessor);


Comment on this Routine

sets p to storage for the new node

p = avail();


Comment on this Routine

sets the info field of the new node

setinfo(p,&value);


Comment on this Routine

copies the left pointer of the old node into the left pointer of the new

setleft(p,left(q));


Comment on this Routine

copies the right pointer of the old node into the right pointer of the new

setright(p,right(q));


Comment on this Routine

sets the right pointer of the new node's predecessor to point to the new node

setright(predecessor,p);


Comment on this Routine

reclaims the old node's storage

reclaim(q);


Comment on this Routine

if the deleted node is a left successor

if(left(predecessor) == p)


Comment on this Routine

set its predecessor's left pointer to null

setleft(predecessor,null);


Comment on this Routine

preorder

locust


Comment on this Routine

inorder

oclsut


Comment on this Routine

postorder

costul


Comment on this Routine

process the root node of tree

process(ptree);


Comment on this Routine

set l to tree's left subtree

1 = left(*ptree);


Comment on this Routine

call preorder to traverse tree's left subtree; similarly for tree's right subtree

preorder(&l);

r = right(*ptree);

preorder(&r);


Comment on this Routine

null set to null pointer value

null = setnull();


Comment on this Routine

current node pointer set to tree

p = *ptree;


Comment on this Routine

test for completion

while ((p != null) || !empty(&s))


Comment on this Routine

if there is a current node

if (p != null)


Comment on this Routine

process it

process (ptree,p);


Comment on this Routine

save the pointer to its right subtree on stack s

push (right(p),&s);


Comment on this Routine

set current node pointer to its left subtree root

p = left(p);


Comment on this Routine

since the stack s cannot be empty, pop it, updating the current node pointer

pop (&s,&p);


Comment on this Routine

test for completion

while((p != null) || !empty(&s))


Comment on this Routine

process it

process(ptree,p);


Comment on this Routine

if there is a left subtree

if(left(p) != null)


Comment on this Routine

save a pointer to the current node on the stack s

push(p,&s);


Comment on this Routine

update the current node pointer to the left subtree root

p = left(p);


Comment on this Routine

update the current node pointer to the right subtree root

p = right(p);


Comment on this Routine

search along the return path for the next right subtreee at which the traversal is to resume

do

{

pop(&s,&q);

if (!empty(&s))

rightpointer = right(item(&s));

else

rightpointer = null;

}while(!empty(&s)&&(q==rightpointer));


Comment on this Routine

pop the stack and set q to the popped node

pop(&s,&q);


Comment on this Routine

if the popped node is not the root of tree

if (!empty(&s))


Comment on this Routine

set rightpointer to its predecessor's right successor

rightpointer = right(item(&s));


Comment on this Routine

set it to null

rightpointer = null;


Comment on this Routine

test for completion of traversal or for q containing a pointer to the right subtree to be trasversed next; note that p will be null if the tranversal is done

}while(!empty(&s)&&(q==rightpointer));


Comment on this Routine

if not done then

if(q != rightpointer)


Comment on this Routine

update the current node pointer to the correct next right subtree

p = rightpointer;


Comment on this Routine

if first time called

if(p == *ptree)


Comment on this Routine

set tncount to zero

tncount = 0;


Comment on this Routine

if terminal node

if((left(p) == null) && (right(p) == null))


Comment on this Routine

update tncount

tncount++;


Comment on this Routine

test for last node

if(last (ps))


Comment on this Routine

if terminal node, then, for the new node

if((left(p) == null)&&(right(p) == null)


Comment on this Routine

allocate storage

ptr = avail();


Comment on this Routine

set the info field

setinfo(ptr,&value);


Comment on this Routine

set the left pointer

setleft(ptr,null);


Comment on this Routine

set the right pointer

setright(ptr,null);


Comment on this Routine

if the replacement is not for the root node

if(p != *ptree)


Comment on this Routine

get its predecessor

q = item(ps);


Comment on this Routine

determine if the predecessor's left or right pointer is to point to the new node and set it

if(left(q) == p)

setleft(q,ptr);

else


Comment on this Routine

set tree to point to the new node

*ptree = ptr;


Comment on this Routine

setting the stack to empty will cause the traversal to terminate

setstack(ps);


Comment on this Routine

if the binary tree is null

if(tree == setnull())


Comment on this Routine

its depth is zero

return(0);


Comment on this Routine

use depth to find the depth of its left and right subtrees; its depth is then one plus their maximum

return(1 + max(depth(left(tree)),


Comment on this Routine

a binary tree record type

typedef struct

{

whatever info;

int leftptr;

int rightptr;


Comment on this Routine

pointers will be array indices

typedef int binarytreepointer;


Comment on this Routine

the array can hold 50 records

# define MAX 50


Comment on this Routine

the array type

typedef binarytreenode recordsarray[MAX];


Comment on this Routine

storage is allocated for the array records that will hold the binary trees

recordsarray records;


Comment on this Routine

a binary tree record type

typedef struct treenode

{

whatever info;

struct treenode *leftptr;

struct treenode *rightptr;

}binarytreenode,*binarytreepointer;


Comment on this Routine

pointers point to records stored in dynamic memory

}binarytreenode,*binarytreepointer;


Comment on this Routine

the binary tree data abstraction

typedef struct treerecord

{

int info;

struct treerecord *leftptr;

struct treerecord *rightptr;

}binarytreerecord,*binarytreepointer;

#define NULL 0

binarytreepointer left(p)

/* Returns a copy of the left

pointer of the node p points to.

*/

binarytreepointer p;

{

return(p->leftptr);

}

binarytreepointer right(p)

/* Returns a copy of the right

pointer of the node p points to.

*/

binarytreepointer p;

{

return(p->rightptr);

}

binarytreepointer setnull()

/* Returns a null pointer */

{

return(NULL);

}

binarytreepointer avail()

/* Returns a pointer to storage

allocated for a new node.

*/

{

return(malloc(sizeof(binarytreerecord)));

}

setinfo(p,pvalue)

/* Copies the contents of value

into the record p points to.

*/

binarytreepointer p;

int *pvalue;

{

p->info = *pvalue;

}

setleft(p,q)

/* Copies q into the left pointer

of the record p points to.

*/

binarytreepointer p,q;

{

p->leftptr = q;

}

setright(p,q)

/* Copies q into the right pointer

of the record p points to.

*/

binarytreepointer p,q;

{

p->rightptr = q;

}

printnode(pl,ptr)

/* Prints the info field of the

record ptr points to.

*/

binarytreepointer *pl,ptr;

{

printf("\n %d",ptr->info);

}


Comment on this Routine

from here on the program is independent of the binary tree implementation

main()

/* Reads the data for nodes of a binary tree, creates

the tree and prints how many terminal nodes it

has and its depth, reads the value for the new node,

replaces the first terminal node in preorder access

by the new node, and then prints all the final

binary tree's nodes, its terminal count, and its

depth.

*/

{

int n,value;

binarytreepointer t,setnull(),avail();

printf("\n enter binarytree 0 for null tree 1\

otherwise \n");

scanf("%d",&n);

if (n == 0)

t = setnull();

else

{

t = avail();

createtree(&t);

}

printtree(&t);

terminalcount(&t);

printf("\n The depth is %d",depth(t));

printf("\n enter value ");

scanf("%d",&value);

replacefirstterminal(&t,value);

printtree(&t);

terminalcount(&t);

printf("\n The depth is %d",depth(t));

}

createtree(ptree)

/* Read the data and create the binary tree tree. */

binarytreepointer *ptree;

{

binarytreepointer l,r,setnull(),left(),right();

if (*ptree != setnull())

{

createnode(ptree);

l = left(*ptree);

createtree(&l);

r = right(*ptree);

createtree(&r);

}

}

createnode(pptr)

/* Reads the data, and fills in the fields

of the node pointed to by ptr.

*/

binarytreepointer *pptr;

{

int leftlink,rightlink,value;

binarytreepointer setnull(),avail();

printf("\n enter info \n");

scanf("%d",&value);

setinfo(*pptr,&value);

printf("\n enter enter left & right ptrs \n");

scanf("%d %d",&leftlink,&rightlink);

if (leftlink == 0)

setleft(*pptr,setnull());

else

setleft(*pptr,avail());

if (rightlink == 0)

setright(*pptr,setnull();

else

setright(*pptr,avail());

}

#define LIMIT 50

typedef binarytreepointer whatever;

typedef struct

{

whatever stackarray[LIMIT];

int top;

}stack;

#define TRUE 1

#define FALSE 0

setstack(ps)

/* Sets the stack s to empty. */

stack *ps;

{

(*ps).top = -1;

}

empty(ps)

/* Returns true only if stack s is empty.*/

stack *ps;

{

return((*ps).top == -1);

}

push(value,ps)

/* Inserts value at the top of stack s. */

whatever value;

stack *ps;

{

if ((*ps).top == (LIMIT - 1))

overflow(ps);

else

{

(*ps).top = (*ps).top + 1;

(*ps).stackarray[(*ps).top] = value;

}

}

pop(ps,pvalue)

/* Removes the top entry of stack s

and copies its contents into value.

*/

stack *ps;

whatever *pvalue;

{

if (empty(ps))

underflow(ps);

else

{

*pvalue = (*ps).stackarray[(*ps).top];

(*ps).top = (*ps).top - 1;

}

}

whatever item(ps)

/* Returns a copy of the top entry on stack s. */

stack *ps;

{

return((*ps).stackarray[(*ps).top]);

}

last(ps)

/* Returns true only if stack s

contains no non null pointers.

*/

stack *ps;

{

int i,temp;

binarytreepointer null,setnull();

null = setnull();

temp = TRUE;

for (i=0;i<=(*ps).top;i++)

if ((*ps).stackarray[i] != null)

temp = FALSE;

return(temp);

}

overflow(ps)

/* Prints a message if the stack overflows. */

stack *ps;

{

printf("\n stack overflow ");

}

underflow(ps)

/* Prints a message if the stack underflows. */

stack *ps;

{

printf("\n stack underflow ");

}

printtree(ptree)

/* Prints the info field values of all

nodes of the binary tree tree.

*/

binarytreepointer *ptree;

{

binarytreepointer null,p,q,rightpointer;

binarytreepointer setnull(),left(),right(),item();

stack s1;

null = setnull();

setstack(&s1);

p = *ptree;

while ((p != null) || !empty(&s1))

if (p != null)

{

printnode(ptree,p);

if (left(p) != null)

{

push(p,&s1);

p = left(p);

}

else

{

push(p,&s1);

p = right(p);

}

}

else

{

do

{

pop(&s1,&q);

if (!empty(&s1))

rightpointer = right(item(&s1));

else

rightpointer = null;

}while(!empty(&s1) &&

(q == rightpointer));

if (q != rightpointer)

p = rightpointer;

}

}

terminalcount(ptree)

/* Prints the number of terminal nodes

in the binary tree tree.

*/

binarytreepointer *ptree;

{

binarytreepointer p,null,setnull(),left(),right();

stack s2;

null = setnull();

setstack(&s2);

p = *ptree;

while ((p != null) || !empty(&s2))

if (p != null)

{

updatetncount(ptree,p,&s2);

push(right(p),&s2);

p = left(p);

}

else

pop(&s2,&p);

}

updatetncount(pt,p,ps)

/* Prints the number of terminal nodes in

binary tree tree when p points to the last

node in a preorder traverse of tree which

saves right pointers on stack s.

*/

binarytreepointer *pt,p;

stack *ps;

{

binarytreepointer null,setnull(),left(),right();

static int tncount;

null = setnull();

if (p == *pt)

tncount = O;

if ((left(p) == null) && (right(p) == null))

{

tncount++;

if (last(ps))

printf("\n The number of terminal nodes\

is %d",tncount);

}

}

replacefirstterminal(ptree,value)

/* Replaces the first terminal node, encountered

in a preorder traversal of binary tree tree, by a

new node with its info field set to the contents

of value.

*/

binarytreepointer *ptree;

int value;

{

binarytreepointer null,p,q,rightpointer;

binarytreepointer setnull(),left(),right(),item();

stack s3;

null = setnull();

setstack(&s3);

p = *ptree;

while ((p != null) | | !empty(&s3))

if (p != null)

{

replacenode(ptree,p,null,&s$,value);

if (left (p) != null)

{

push(p,&s3);

p = left(p);

}

else

{

push(p,&s3);

p = right(p);

}

}

else

{

do

{

pop(&s3,&q);

if (!empty(&s3))

rightpointer = right(item(&s3));

else

rightpointer = null;

}while (!empty(&s3) &&

(q == rightpointer));

if (q != rightpointer)

p = right pointer;

}

}

replacenode(ptree,p,null,ps,value)

/* If p points to the first terminal node, encountered

in a preorder traversal of binary tree tree that

saves the path to the root on stack s, then

replaces that node by a new node whose info field is

set to the contents of value.

*/

binarytreepointer *ptree,p,null;

stack *ps;

int value;

{

binarytreepointer q,ptr,avail(),left(),right();

if ((left(p) == null) && (right(p) == null))

{

ptr = avail();

setinfo(ptr,&value);

setleft(ptr,null);

setright(ptr,null);

if (p != *ptree)

{

q = item(ps);

if (left(q) == p)

setleft(q,ptr);

else

setright(q,ptr);

}

else

*ptree = ptr;

setstack(ps);

}

}

depth(tree)

/* Returns the depth of the binary tree tree. */

binarytreepointer tree;

{

binarytreepointer setnull(),left(),right();

if (tree == setnull())

return(0);

else

return(1 + max(depth(left(tree)),

depth(right(tree))));

}

max(i,j)

/* Returns the maximum of i and j. */

int i,j;

{

if (i >= j)

return(i);


Comment on this Routine

a zero indicates that the input is a null binary tree, so t is set to null

if (n == 0)

t = setnull();


Comment on this Routine

the input is not a null binary tree, so allocate storage for t and create the binary tree t

t = avail();

createtree(&t);


Comment on this Routine

print all the tree's nodes

printtree(&t);


Comment on this Routine

print the number of terminal nodes

terminalcount(&t);


Comment on this Routine

print t's depth

printf("\n The depth is %d",depth(t));


Comment on this Routine

read the new value

scanf("%d",&value);


Comment on this Routine

replace the first terminal node

replacefirstterminal(&t,value);


Comment on this Routine

this is a recursive preorder traversal

createtree(ptree)

/* Read the data and create the binary tree tree. */

binarytreepointer *ptree;

{

binarytreepointer l,r,setnull(),left(),right();

if (*ptree != setnull())

{

createnode(ptree);

l = left(*ptree);

createtree(&l);

r = right(*ptree);

createtree(&r);

}

}


Comment on this Routine

allocates storage for l & r, the left and right subtrees

binarytreepointer l,r,setnull(),left(),right();


Comment on this Routine

fills the root node's fields

createnode(ptree);


Comment on this Routine

creates the left subtree

createtree(&l);


Comment on this Routine

creates the right subtree

createtree(&r);


Comment on this Routine

reads the info value

scanf("%d",&value);


Comment on this Routine

sets the info field

setinfo(*pptr,&value);


Comment on this Routine

reads left and right pointer data

printf("\n enter enter left & right ptrs \n");

scanf("%d %d",&leftlink,&rightlink);


Comment on this Routine

sets the left and right pointers properly

if (leftlink == 0)

setleft(*pptr,setnull());

else

setleft(*pptr,avail());

if (rightlink == 0)

setright(*pptr,setnull();

else

setright(*pptr,avail());

}


Comment on this Routine

the stack data abstraction

#define LIMIT 50

typedef binarytreepointer whatever;

typedef struct

{

whatever stackarray[LIMIT];

int top;

}stack;

#define TRUE 1

#define FALSE 0

setstack(ps)

/* Sets the stack s to empty. */

stack *ps;

{

(*ps).top = -1;

}

empty(ps)

/* Returns true only if stack s is empty.*/

stack *ps;

{

return((*ps).top == -1);

}

push(value,ps)

/* Inserts value at the top of stack s. */

whatever value;

stack *ps;

{

if ((*ps).top == (LIMIT - 1))

overflow(ps);

else

{

(*ps).top = (*ps).top + 1;

(*ps).stackarray[(*ps).top] = value;

}

}

pop(ps,pvalue)

/* Removes the top entry of stack s

and copies its contents into value.

*/

stack *ps;

whatever *pvalue;

{

if (empty(ps))

underflow(ps);

else

{

*pvalue = (*ps).stackarray[(*ps).top];

(*ps).top = (*ps).top - 1;

}

}

whatever item(ps)

/* Returns a copy of the top entry on stack s. */

stack *ps;

{

return((*ps).stackarray[(*ps).top]);

}

last(ps)

/* Returns true only if stack s

contains no non null pointers.

*/

stack *ps;

{

int i,temp;

binarytreepointer null,setnull();

null = setnull();

temp = TRUE;

for (i=0;i<=(*ps).top;i++)

if ((*ps).stackarray[i] != null)

temp = FALSE;

return(temp);

}

overflow(ps)

/* Prints a message if the stack overflows. */

stack *ps;

{

printf("\n stack overflow ");

}

underflow(ps)


Comment on this Routine

the stack will contain pointers to nodes of a binary tree

typedef binarytreepointer whatever;


Comment on this Routine

from here on the program is independent of both the binary tree and the stack implementation

printtree(ptree)

/* Prints the info field values of all

nodes of the binary tree tree.

*/

binarytreepointer *ptree;

{

binarytreepointer null,p,q,rightpointer;

binarytreepointer setnull(),left(),right(),item();

stack s1;

null = setnull();

setstack(&s1);

p = *ptree;

while ((p != null) || !empty(&s1))

if (p != null)

{

printnode(ptree,p);

if (left(p) != null)

{

push(p,&s1);

p = left(p);

}

else

{

push(p,&s1);

p = right(p);

}

}

else

{

do

{

pop(&s1,&q);

if (!empty(&s1))

rightpointer = right(item(&s1));

else

rightpointer = null;

}while(!empty(&s1) &&

(q == rightpointer));

if (q != rightpointer)

p = rightpointer;

}

}

terminalcount(ptree)

/* Prints the number of terminal nodes

in the binary tree tree.

*/

binarytreepointer *ptree;

{

binarytreepointer p,null,setnull(),left(),right();

stack s2;

null = setnull();

setstack(&s2);

p = *ptree;

while ((p != null) || !empty(&s2))

if (p != null)

{

updatetncount(ptree,p,&s2);

push(right(p),&s2);

p = left(p);

}

else

pop(&s2,&p);

}

updatetncount(pt,p,ps)

/* Prints the number of terminal nodes in

binary tree tree when p points to the last

node in a preorder traverse of tree which

saves right pointers on stack s.

*/

binarytreepointer *pt,p;

stack *ps;

{

binarytreepointer null,setnull(),left(),right();

static int tncount;

null = setnull();

if (p == *pt)

tncount = O;

if ((left(p) == null) && (right(p) == null))

{

tncount++;

if (last(ps))

printf("\n The number of terminal nodes\

is %d",tncount);

}

}

replacefirstterminal(ptree,value)

/* Replaces the first terminal node, encountered

in a preorder traversal of binary tree tree, by a

new node with its info field set to the contents

of value.

*/

binarytreepointer *ptree;

int value;

{

binarytreepointer null,p,q,rightpointer;

binarytreepointer setnull(),left(),right(),item();

stack s3;

null = setnull();

setstack(&s3);

p = *ptree;

while ((p != null) | | !empty(&s3))

if (p != null)

{

replacenode(ptree,p,null,&s$,value);

if (left (p) != null)

{

push(p,&s3);

p = left(p);

}

else

{

push(p,&s3);

p = right(p);

}

}

else

{

do

{

pop(&s3,&q);

if (!empty(&s3))

rightpointer = right(item(&s3));

else

rightpointer = null;

}while (!empty(&s3) &&

(q == rightpointer));

if (q != rightpointer)

p = right pointer;

}

}

replacenode(ptree,p,null,ps,value)

/* If p points to the first terminal node, encountered

in a preorder traversal of binary tree tree that

saves the path to the root on stack s, then

replaces that node by a new node whose info field is

set to the contents of value.

*/

binarytreepointer *ptree,p,null;

stack *ps;

int value;

{

binarytreepointer q,ptr,avail(),left(),right();

if ((left(p) == null) && (right(p) == null))

{

ptr = avail();

setinfo(ptr,&value);

setleft(ptr,null);

setright(ptr,null);

if (p != *ptree)

{

q = item(ps);

if (left(q) == p)

setleft(q,ptr);

else

setright(q,ptr);

}

else

*ptree = ptr;

setstack(ps);

}

}

depth(tree)

/* Returns the depth of the binary tree tree. */

binarytreepointer tree;

{

binarytreepointer setnull(),left(),right();

if (tree == setnull())

return(0);

else

return(1 + max(depth(left(tree)),

depth(right(tree))));

}

max(i,j)

/* Returns the maximum of i and j. */

int i,j;

{

if (i >= j)

return(i);

else

return(j);

}


Comment on this routine

this is the nonrecursive traversal that saves the path to the root

printtree(ptree)
/* Prints the info field values of all
nodes of the binary tree tree.
*/
binarytreepointer *ptree;
{
binarytreepointer null,p,q,rightpointer;
binarytreepointer setnull(),left(),right(),item();
stack s1;
null = setnull();
setstack(&s1);
p = *ptree;
while ((p != null) || !empty(&s1))
if (p != null)
{
printnode(ptree,p);
if (left(p) != null)
{
push(p,&s1);
p = left(p);
}
else
{
push(p,&s1);
p = right(p);
}
}
else
{
do
{
pop(&s1,&q);
if (!empty(&s1))
rightpointer = right(item(&s1));
else
rightpointer = null;
}while(!empty(&s1) &&
(q == rightpointer));
if (q != rightpointer)
p = rightpointer;
}
}


Comment on this Routine

prints the info field of the node p points to

printnode(ptree,p);


Comment on this Routine

this is a nonrecursive traversal that saves right pointers

terminalcount(ptree)

/* Prints the number of terminal nodes

in the binary tree tree.

*/

binarytreepointer *ptree;

{

binarytreepointer p,null,setnull(),left(),right();

stack s2;

null = setnull();

setstack(&s2);

p = *ptree;

while ((p != null) || !empty(&s2))

if (p != null)

{

updatetncount(ptree,p,&s2);

push(right(p),&s2);

p = left(p);

}

else

pop(&s2,&p);

}


Comment on this Routine

updates the terminal count appropriately and prints it after the last node

updatetncount(ptree,p,&s2);


Comment on this Routine

this is a nonrecursive traversal that saves the path to the root

replacefirstterminal(ptree,value)

/* Replaces the first terminal node, encountered

in a preorder traversal of binary tree tree, by a

new node with its info field set to the contents

of value.

*/

binarytreepointer *ptree;

int value;

{

binarytreepointer null,p,q,rightpointer;

binarytreepointer setnull(),left(),right(),item();

stack s3;

null = setnull();

setstack(&s3);

p = *ptree;

while ((p != null) | | !empty(&s3))

if (p != null)

{

replacenode(ptree,p,null,&s$,value);

if (left (p) != null)

{

push(p,&s3);

p = left(p);

}

else

{

push(p,&s3);

p = right(p);

}

}

else

{

do

{

pop(&s3,&q);

if (!empty(&s3))

rightpointer = right(item(&s3));

else

rightpointer = null;

}while (!empty(&s3) &&

(q == rightpointer));

if (q != rightpointer)

p = right pointer;

}

}


Comment on this Routine

definitions for types used to store the board configuration

#define NLIMIT 20

typedef int rowcheck[NLIMIT];

typedef int diagonalcheck1[2*NLIMIT-1];

typedef int diagonalcheck2[2*NLIMIT-1];


Comment on this Routine

the stack data abstraction

#define SLIMIT 20

typedef int whatever;

typedef struct

{

whatever stackarray[SLIMIT�1];

int top;

}stack;

setstack(ps)

/* Sets stack s to empty. */

stack *ps;

{

(*ps).top = �1;

}

empty(ps)

/* Returns true only if stack s is empty. */

stack *ps;

{

return((*ps).top == -1);

}

push(value,ps)

/* Inserts contents of value as

the top entry on stack s.

/*

whatever value;

stack *ps;

{

if((*ps).top == (SLIMIT-1))

overflow(ps);

else

{

(*ps).top = (*ps).top + 1;

(*ps).stackarray[(*ps).top] = value;

}

}

pop(ps,pvalue)

/* Removes the top entry of stack s

and copies its contents into value.

*/

stack *ps;

whatever *pvalue;

{

if(empty(ps))

underflow(ps);

else

{

*pvalue = (*ps).stackarray[(*ps).top];

(*ps).top = (*ps).top - 1;

}

}

whatever item(i,ps)

/* Returns a copy of the ith

entry in stack s. When i is

zero it returns a copy of

the top entry.

*/

stack *ps;

{

return((*ps).stackarray[(*ps).top-i]);

}

overflow(ps)

/* Prints a message if the stack overflows. */

stack *ps;

{

printf("\n stack overflow ");

}


Comment on this Routine

initialization of the board; no queens have been placed

for(i=0;i<=n-1;i++)

r[i] = TRUE;

for(i=0;i<=2*n-2;i++)

d1[i] = TRUE;

for(i=0;i<=2*n-2;i++)

d2[i] = TRUE;


Comment on this Routine

test for completion

while(((col < n+1)&&(row < n+1)) || !empty(&s))


Comment on this Routine

if col, row position is on the board

if((col < n+1)&&(row < n+1))


Comment on this Routine

if potential partial solution

if(feasible(row,col,r,d1,d2,n))


Comment on this Routine

if the partial solution is complete, it is printed and found set to true

process(row,col,&found,&s,n);


Comment on this Routine

stores the row position of the queen just placed

push(row,&s);


Comment on this Routine

updates the board configuration

r[row-1] = FALSE;

d1[row+col-2] = FALSE;

d2[row-col+n-1] = FALSE;


Comment on this Routine

moves the col over

col = col + 1;


Comment on this Routine

sets row back to 1, its first possible value

row = 1;


Comment on this Routine

try the next row

row = row + 1;


Comment on this Routine

backtract to the last col and row placement

pop(&s,&row);

col = col - 1;


Comment on this Routine

update the board configuration

r[row-1] = TRUE

d1[row+col-2] = TRUE;

d2[row-col+n-1] = TRUE;


Comment on this Routine

the rows for the queens are on the stack; the row for col n-1 is at the top

printf("\n COL %d ROW is %d----",

n-1,item(i-1,ps));


Comment on this Routine

the row position of the queen in col n was not stacked

printf("\n COL %d ROW is %d----\n",

n,row);


Comment on this Routine

allocates storage for tree

*ptree = avail ( );


Comment on this Routine

tree must be null

*ptree = NULL;


Comment on this Routine

tree has only a root node

records [*ptree].leftptr = NULL;

records [*ptree].rightptr = NULL;


Comment on this Routine

tree is constructed by creating its left and right subtrees as proper Fibonacci trees

fibonaccitree (d-2,&tl);

records [*ptree].leftptr = tl;

fibonaccitree (d-1,&tr);

records [*ptree].rightptr = tr;

}


Comment on this Routine

allocates storage for tree

*ptree = malloc (sizeof(binarytreerecord));


Comment on this Routine

tree has only a root node

(*ptree)->leftptr = NULL;

(*ptree)->rightptr = NULL;


Comment on this Routine

tree is constructed by creating its left and right subtrees as proper Fibonacci trees

fibonaccitree (d-2, &tl);

(*ptree)->leftptr = tl;

fibonaccitree (d-1, &tr);

(*ptree)->rightptr = tr;

}


Comment on this Routine

tree must be null

*ptree = null;


Comment on this Routine

tree has only a root node

setleft(*ptree, null);

setright(*ptree, null);


Comment on this Routine

tree is constructed by creating its left and right subtrees as proper Fibonacci trees

fibonaccitree(d-2,&tl);

setleft(*ptree, tl);

fibonaccitree(d-1,&tr);

setleft(*ptree, tr);

}


Comment on this Routine

traverses through the n records in the data array

linearsearch(data,n,key,pfound,ploc)


Comment on this Routine

sets loc to index the first record

*ploc = 1;


Comment on this Routine

tests for another record to be searched and if so, whether that record has the search key value

while ((*ploc <= n) && (data[*ploc].key != key))


Comment on this Routine

found should be true only if loc < n + 1

*pfound = (*ploc < n + 1);


Comment on this Routine

faster traversal

linearsearch(data,n,key,pfound,ploc)


Comment on this Routine

stores the search key as a "last" record in data

data[n + 1].key = key;


Comment on this Routine

no need to test for another record

while (data[*ploc].key != key)


Comment on this Routine

top and bottom initialized

top = 1;

bottom = n;


Comment on this Routine

test for more records to be searched and search key not yet found

while ((top <= bottom) && !(*pfound))


Comment on this Routine

update mid

mid = (top + bottom)/2


Comment on this Routine

if search key found

if (data[mid].key == key)


Comment on this Routine

set found

*pfound = TRUE


Comment on this Routine

set loc

*ploc = mid;


Comment on this Routine

else if entry pointed to by mid is less than search key, move bottom up to eliminate lower half from consideration

else if (data[mid].key < key)

bottom = mid - 1;


Comment on this Routine

else move top down to eliminate upper half from consideration

else

top = mid + 1;


Comment on this Routine

comparison

if (data[i].key > largest)


Comment on this Routine

interchange

interchange (data, next, loc);


Comment on this Routine

traversal to "insert" the kth record

inserted = FALSE;

while ((i > 1) && !inserted)

if (data[i - l].key < data[i].key)

{

interchange(data,i,i - 1);

i--;

}

else

inserted = TRUE;

}


Comment on this Routine

create the heap

root = n/2;

last = n;

while (root > 0)

{

shift(data,root,last);

root--;


Comment on this Routine

make the subtree of root a heap and update root

shift(data,root,last);

root--;


Comment on this Routine

output and reheap

interchange(data,1,last);

last--;

shift(data,root,last);


Comment on this Routine

ptr keeps track of the location of the root entry as it shifts down

int ptr,succ;


Comment on this Routine

initialize ptr to the root

ptr = root;


Comment on this Routine

set succ to ptr's left successor

succ = 2 * ptr;


Comment on this Routine

if the node indexed by ptr has a right successor and it is less than ptr's left successor

if (succ < last)

if (data[succ + 1].key < data[succ].key)


Comment on this Routine

set succ to it

succ++;


Comment on this Routine

save the root key value

keyvalue = data[root].key;


Comment on this Routine

save the root record

copy(&recordvalue,&data[root]);


Comment on this Routine

test for ptr at a terminal node and, if not, whether ptr must be shifted down

while((succ <= last) && (data[succ].key < keyvalue))


Comment on this Routine

move succ's record up to ptr's location

copy(&data[ptr],&data[succ]);


Comment on this Routine

update ptr to succ, moving it down

ptr = succ;


Comment on this Routine

set succ, as above, to the smallest successor of ptr

succ = 2 * ptr

if (succ < last)

if (data[succ + 1].key < data[succ].key)

succ++;


Comment on this Routine

insert the original root record where it belongs

copy(&data[ptr],&recordvalue);


Comment on this Routine

when partition returns, all entries above the pth entry are larger than it, and those below are smaller

p = partition(i,j);


Comment on this Routine

sorts entries above the pth

quicksort(i,p-1);


Comment on this Routine

sorts entries below the pth

quicksort(p+1,j);


Comment on this Routine

stores the original task on stack s

push(j,&s);

push(i&s);


Comment on this Routine

test for remaining tasks

while(!empty(&s))


Comment on this Routine

removes the top task from stack s

pop(&s,&top);

pop(&s,&bottom);


Comment on this Routine

generates two new tasks

p = partition(top,bottom);


Comment on this Routine

puts the larger of the two tasks on stack s and updates top or bottom

if((p - top) > (bottom - p))

{

push(p-l,&s);

push(top,&s);

top = p + l

}

else

{

push(bottom,&s);

push(p+1,&s);

bottom = p - 1;

}


Comment on this Routine

initialize upper and lower

upper = i;

lower = j;


Comment on this Routine

save the basis, data [i]

copy(&saverecord,&data[i]);


Comment on this Routine

test for completion

while(upper != lower)


Comment on this Routine

move lower up, looking for a record whose key exceeds the basis key

while((upper < lower) && (saverecord.key >=

data[lower].key))

lower--;


Comment on this Routine

copy the basis into lower's position

copy(&data[upper],&data[lower]);


Comment on this Routine

move upper down, looking for a record whose key is less than the basis

while((upper < lower) && (saverecord.key <=

data[upper].key))

upper++;


Comment on this Routine

copy the basis into upper's position

copy(&data[lower],&data[upper]);


Comment on this Routine

copy the basis into its final position

copy(&data[upper],&saverecord);


Comment on this Routine

return the index of the basis

return(upper);


Comment on this Routine

This code sets found to true only if a record whose key is equal to keyvalue is in the binary search tree t. If found is true, then pred contains a pointer to the predecessor of that record, and p contains a pointer to that record. If found is false, then pred points to the node that would have been its predecessor had it been in the tree.

found = FALSE;

p = t;

pred = NULL;

while ((p != NULL) && (!found))

if (keyvalue == p->key)

found = TRUE;

else if (keyvalue < p->key)

{

pred = p;

p = p->leftptr;

}

else

{

pred = p;

p = p->rightptr:

}


Comment on this Routine

This code creates and inserts a new record with key equal to keyvalue as the successor of the record pointed to by pred. It replaces the correct null subtree of pred by the new record, which becomes a terminal node.

p = malloc(sizeof(binarytreenode));

p->key = keyvalue;

p->leftptr = NULL;

p->rightptr = NULL;

if (keyvalue < pred->key)

pred->leftptr = p;

else

pred->rightptr = p;


Comment on this Routine

searches for keyvalue

search(keyvalue,*pt,&p,&predp,

&last,&predlast,&found);


Comment on this Routine

if not in the tree

if(!found)


Comment on this Routine

create a new node

createnode(keyvalue,&p);


Comment on this Routine

inserts it

insertnode(keyvalue,p,predp,pt);


Comment on this Routine

if t has more than one node

if(p != *pt)


Comment on this Routine

no need to rebalance, so reset last's and appropriate path balances

{

last->balance = -1;

resetpathbalances(keyvalue,

last->leftptr,p);

}


Comment on this Routine

no need to rebalance, so reset last's and appropriate path balances

{

last->balance = 0;

resetpathbalances(keyvalue,

last�>leftptr,p);

}

else


Comment on this Routine

need to rebalance

{

q = last->leftptr;

if(keyvalue < q->key)

ll(keyvalue,&last,p);

else

lr(keyvalue,&last,&q,p);

last->balance = 0;

if(predlast == NULL)

*pt = last;

else if(keyvalue < predlast->key)

predlast->leftptr = last;

else

predlast->rightptr = last;

}


Comment on this Routine

task (i) of the algorithm

q = last->leftptr;

if(keyvalue < q->key)

ll(keyvalue,&last,p);

else

lr(keyvalue,&last,&q,p);

last->balance = 0;


Comment on this Routine

task (ii) of the algorithm

if(predlast == NULL)

*pt = last;

else if(keyvalue < predlast->key)

predlast->leftptr = last;

else

predlast->rightptr = last;


Comment on this Routine

no need to rebalance, so reset last's and appropriate path balances

{

last->balance = +1;

resetpathbalances(keyvalue,

last->rightptr,p);

}


Comment on this Routine

no need to rebalance, so reset last's and appropriate path balances

{

last->balance = 0;

resetpathbalances(keyvalue,

last->rightptr,p);

}


Comment on this Routine

need to rebalance

{

q = last->rightptr;

if(keyvalue < q->key)

rr(keyvalue,&last,p);

else

rl(keyvalue,&last,&q,p);

last->balance = 0;

if(predlast == NULL)

*pt = last;

else if(keyvalue < predlast->key)

predlast->leftptr = last;

else

predlast->rightptr = last;

}


Comment on this Routine

task (i) of the algorithm

q = last->rightptr;

if(keyvalue < q->key)

rr(keyvalue,&last,p);

else

rl(keyvalue,&last,&q,p);

last->balance = 0;


Comment on this Routine

reset the balance of the inserted node's predecessor

predp->balance = 0;


Comment on this Routine

if t is null

if(*pt == NULL)


Comment on this Routine

create and make the new node t's root

createnode(keyvalue,pt);


Comment on this Routine

t's depth has increased

*pincrease = 1;


Comment on this Routine

else if the new node is to be inserted in t's left subtree

else if(keyvalue < (*pt)->key)


Comment on this Routine

insert it there

insertavl(keyvalue,

&((*pt)->leftptr),pincrease);


Comment on this Routine

if the left subtree depth increased

if(*pincrease) == 1)


Comment on this Routine

reset t's balance

(*pt)->balance = -1;


Comment on this Routine

reset t's balance

(*pt)->balance = 0;


Comment on this Routine

t's depth did not increase

*pincrease = 0;


Comment on this Routine

rebalance t and reset the balances of nodes involved in the remedy

q = (*pt)->leftptr;

if(keyvalue < q->key)

ll(pt);

else

lr (pt,q);


Comment on this Routine

set t's balance to zero

(*pt)->balance = 0;


Comment on this Routine

else if the new node is to be inserted in t's right subtree

else if(keyvalue > (*pt)->key)


Comment on this Routine

insert it there

insertavl(keyvalue,

&((*pt)->rightptr),pincrease);


Comment on this Routine

if the right subtree depth increased

if(*pincrease == 1)


Comment on this Routine

reset t's balance

(*pt)->balance = +1;


Comment on this Routine

rebalance t and reset the balances of nodes involved in the remedy

q = (*pt)->rightptr;

if(keyvalue > q->key)

rr (pt)

else

rl(pt,q);


Comment on this Routine

sets the tree t to null

t = setnull();


Comment on this Routine

inputs first record's key

scanf("%d",&keyvalue);


Comment on this Routine

inserts the record into t

insertavl(keyvalue,&t,&increase);


Comment on this Routine

inputs next record's key

scanf("%d",&keyvalue);


Comment on this Routine

prints t's keys in preorder access order

printtree(t);


Comment on this Routine

prints the final tree's keys and balances in preorder access order

preorder (t);


Comment on this Routine

opens the file for reading and returns a pointer to the first file record unless an error has occured, when it returns a null pointer

f_ptr = fopen("bookinventory", "r");

if(f_ptr == NULL)

printf("Can not open the file");


Comment on this Routine

open prime for writing

f_ptr = fopen("prime","w");


Comment on this Routine

write the first 10 primes

fprintf(f_ptr,"%d",2);

fprintf(f_ptr,"%d",3);

fprintf(f_ptr,"%d",5);

fprintf(f_ptr,"%d",7);

fprintf(f_ptr,"%d",11);

fprintf(f_ptr,�%d�,13);

fprintf(f_ptr,"%d",17);

fprintf(f_ptr,"%d",19);

fprintf(f_ptr,"%d",23);

fprintf(f_ptr,"%d",29);


Comment on this Routine

close prime

fclose(f_ptr);


Comment on this Routine

open prime for reading

(1) f_ptr = fopen("prime","r");


Comment on this Routine

if an error has occurred, take proper action

if(f_ptr == NULL)

{

printf("Can not open");

exit(-1);


Comment on this Routine

read and test the next file entry

while(fscanf(f_ptr,"%d",&dummy) != EOF)


Comment on this Routine

store it in a[i] and update array index i

a[i] = dummy;

i++;


Comment on this Routine

decrease i so it points to the last prime read

i--;


Comment on this Routine

open reversedprime for writing

(2) fr_ptr = fopen("reversedprime","w");


Comment on this Routine

if an error has occurred, take proper action

if(fr_ptr == NULL)

{

printf("Can not open");

exit(-1);

}


Comment on this Routine

while array a contains more primes

while(i >= 0)


Comment on this Routine

write the next prime to the file and update array index i

fprintf(fr_ptr,"%d",a[i]);

i--;


Comment on this Routine

close the prime file

fclose(f_ptr);


Comment on this Routine

close the reversedprime file

fclose(fr_ptr);


Comment on this Routine

open file prime for reading

f_ptr = fopen("prime","r");


Comment on this Routine

read and test each file entry and print it

while(fscanf(f_ptr,"%d",&dummy) != EOF)

printf("\n %d\n",dummy);


Comment on this Routine

open file reversedprime for reading

fr_ptr = fopen("reversedprime","r");


Comment on this Routine

read and test each file entry and print it

while(fscanf(fr_ptr,"%d",&dummy) != EOF)

printf("\n %d\n",dummy);


Comment on this Routine

close file prime

fclose(f_ptr);


Comment on this Routine

close file reversedprime

fclose(fr_ptr);


Comment on this Routine

read the first record from each input file

fread(&record1,length,1,f1_ptr);

fread(&record2,length,1,f2_ptr);


Comment on this Routine

test for nonempty input files

while(flag == 0)


Comment on this Routine

if file f1 record has the smaller key value

if(record1.key < record2.key


Comment on this Routine

append it to f3

fwrite(&record1,length,1,f3_ptr);


Comment on this Routine

read into record1 from f1; if f1 empty, signify by setting flag to 1

if(fread(&record1,length,1,f1_ptr) == 0)

{

flag = 1;


Comment on this Routine

append record2 to f3

fwrite(&record2,1ength,1,f3_ptr);


Comment on this Routine

append record2 to f3

fwrite(&record2,length,1,f3_ptr);


Comment on this Routine

read into record2 from f2; if f2 empty, signify by setting flag to 2

if(fread(&record2,length,1,f2_ptr) == 0)

{

flag = 2;


Comment on this Routine

append record1 to f3

fwrite (&record1,length,1,f3_ptr);


Comment on this Routine

if f1 empty

if(flag == 1)


Comment on this Routine

append the rest of f2 to f3

while(fread(&record2,length,1,f2_ptr) != 0)

fwrite(&record2,length,1,f3_ptr);


Comment on this Routine

append the rest of f1 to f3

while(fread(&record1,1ength,1,f1_ptr) != 0)

fwrite(&record1,1ength,1,f3_ptr);


Comment on this Routine

opens and tests the input files for reading and the output file for writing

f1_ptr = fopen("prime","r");

if(f1_ptr == NULL)

{

printf("Can not open file1");

exit(-1);

}

f2_ptr = fopen("other","r");

if(f2_ptr == NULL)

{

printf("Can not open file2");

exit(-1);

}

f3_ptr = fopen("result","w");

if(f3_ptr == NULL)

{

printf("Can not open file3");

exit(-1);

}


Comment on this Routine

invokes merge to produce the output file

merge(f1_ptr,f2_ptr,f3_ptr);


Comment on this Routine

closes the files

fclose(f1_ptr);

fclose(f2_ptr);

fclose(f3_ptr);


Comment on this Routine

opens the merged file for reading

f3_ptr = fopen("result","r");


Comment on this Routine

reads and prints the merged file

while(fread(&record,sizeof(filerecords),1,f3_ptr) != 0)

printf("\n %d\n",record.key);


Comment on this Routine

closes the merged file

fclose(f3_ptr);


Comment on this Routine

t is an array index that points to the next available record in lists; it must be preserued between calls to avail

static int t=0;

t++;

return(t);


Comment on this Routine

get storage for the new record

pointer = avail();


Comment on this Routine

copy j into its succobj field

setinfo(pointer,j);


Comment on this Routine

copy the head of the successor list of i into its link field and set the head to point to the new record

setlink(pointer,succ[i]);

succ[i] = pointer;


Comment on this Routine

set all heads of the successor lists to null

for (i=1;i<=n;i++)

succ[i] = setnull();


Comment on this Routine

the bag is empty when bag equals next

emptybag =(bag == ranking.next);


Comment on this Routine

set obj to an object to be removed from the bag and remove it

*pobj = ranking.rank[ranking.next];

ranking.next = ranking.next + 1;


Comment on this Routine

decrease the count of the successor pointed to by recordpointer, and if the count becomes zero, add the successor to the bag

currentsucc = info(recordpointer);

decrease(currentsucc,count);

if(iszero(currentsucc,count))

{

ranking.rank[*pbag] = currentsucc;

*pbag = *pbag + 1;

}


Comment on this Routine

sets the maximum value of n

#defineLIMIT 21


Comment on this Routine

global definition of type outputarray

typedef int outputarray[LIMIT];


Comment on this Routine

storage allocated for the solution

outputarray topologicalsort;


Comment on this Routine

topsort inputs n and the partial order and places the solution in topologicalsort

topsort(&n,topologicalsort);


Comment on this Routine

prints the solution

printf("\n A TOPOLOGICAL SORT IS");

for (i=1;i>=n;1++)

printf("\n %d",topologicalsort[i]);


Comment on this Routine

implementation of the data abstraction countcollection and its basic operations

typedef int countcollection[LIMIT];

predinitialization(n,count)

/* Initializes all n counts to zero. */

int n;

countcollection count;

{

int i;

for (i=1;i<=n;i++)

count[i] = 0;

}

increase(j,count)

/* Increases the jth count by one. */

int j;

countcollection count;

{

count[j] = count[j] + 1;

}

decrease(j,count)

/* Decreases the jth count by one. */

int j;

countcollection count;

{

count[j] = count[j] - 1;

}

iszero(i,count)

/* Returns true only if the ith count is zero. */

int i;

countcollection count;

{

return(count[i] == 0);

}


Comment on this Routine

implementation of the list data abstraction and its basic operations

#define RECORDLIMIT 191

/* RECORDLIMIT should be at least

(((LIMIT-1)*(LIMIT-2))/2)+1

*/

#define NULL - 1

typedef int listpointer;

typedef struct

{

int succobj;

listpointer link;

}listrecords,recordsarray[RECORDLIMIT];

recordsarray lists;

listpointer setnull()

/* Returns a null pointer. */

{

return(NULL);

}

anotherrecord(recordpointer)

/* Returns true only if recordpointer

points to a record.

*/

listpointer recordpointer;

{

return(recordpointer != NULL);

}

info(pointer)

/* Returns the contents of the succobj field

of the record pointed to by pointer.

*/

listpointer pointer;

{

return(lists[pointer].succobj);

}

listpointer next(pointer)

/* Returns the link field value of

the record pointed to by pointer.

*/

listpointer recordpointer;

{

return(lists[recordpointer].link);

}

setinfo(pointer,value)

/* Copies value into the succobj field

of the record pointed to by pointer.

*/

listpointer pointer;

int value;

{

lists[pointer].succobj = value;

}

setlink(pointer1,pointer2)

/* Copies pointer 2 into the link field

of the record pointed to by pointer 1.

*/

listpointer pointer1,pointer2;

{

lists[pointer 1].link = pointer2;

}

listpointer avail()

/* Returns a pointer to storage allocated

for a list record.

*/

{

static int t=0;

t++;

return(t);

}


Comment on this Routine

storage allocated for the array lists to hold records of all lists

recordsarray lists;


Comment on this Routine

implementation of the succ_collection data abstraction and its basic operations

typedef listpointer succ_collection[LIMIT];

insert(i,j,succ)

/* Add j into the ith collection in succ. */

int i,j;

succ_collection succ;

{

listpointer pointer,avail();

pointer = avail();

setinfo(pointer,j);

setlink(pointer,succ[i]);

succ[i] = pointer;

}

succinitialization(n,succ)

/* Initializes all n collections of succ to empty. */

int n;

succ_collection succ;

{

int i;

listpointer setnull();

for (i=1;i<=n;i++)

succ[i] = setnull();

}

listpointer access_succ(i,succ)

/* Returns a pointer to the i collection in succ. */

int i;

succ_collection succ;

{

return(succ[i]);

}


Comment on this Routine

implementation of the ranking-record data abstraction and its basic operations

typedef struct

{

outputarray rank;

int next;

}rankingrecord;

rankingrecord ranking;

1 allocates storage for ranking

copy(n,topologicalsort,ranking)

/* Copies the rank field of ranking

into topologicalsort.

*/

int n;

outputarray topologicalsort;

rankingrecord ranking;

{

int i;

for (i=1;i<=n;i++)

topologicalsort[i] = ranking.rank[i];

}

typedef int bagcollection;

baginitialization(pbag,n,count)

/* Initializes the bag so it contains

only objects whose counts are zero.

*/

bagcollection *pbag;

int n;

countcollection count;

{

int i;

ranking.next = 1;

*pbag = 1;

for(i=1;i<=n;i++)

if(iszero(i,count))

{

ranking.rank[*pbag] = i;

*pbag = *pbag + 1;

}

}

emptybag(bag)

/* Returns true only if bag is empty. */

bagcollection bag;

{

return(bag == ranking.next);

}

remove(pbag,pobj)

/* Sets obj to an object to be removed

from bag, and removes it.

*/

bagcollection *pbag;

int *pobj;

{

*pobj = ranking.rank[ranking.next];

ranking.next = ranking.next + 1;

}

process(listname,recordpointer,pbag,count)

/* Decreases the count of the successor pointed

to by recordpointer and adds it to the bag if

its count has become zero.

*/

listpointer listname,recordpointer;

bagcollection *pbag;

countcollection count;

{

int currentsucc;

currentsucc = info(recordpointer);

decrease(currentsucc,count);

if (iszero(currentsucc,count))

{

ranking.rank[*pbag] = currentsucc;

*pbag = *pbag + 1;

}


Comment on this Routine

implementation of the bag data abstraction and its basic operations; notice that it shares storage with ranking-record, so bag and ranking-record are interdependent

typedef int bagcollection;

baginitialization(pbag,n,count)

/* Initializes the bag so it contains

only objects whose counts are zero.

*/

bagcollection *pbag;

int n;

countcollection count;

{

int i;

ranking.next = 1;

*pbag = 1;

for(i=1;i<=n;i++)

if(iszero(i,count))

{

ranking.rank[*pbag] = i;

*pbag = *pbag + 1;

}

}

emptybag(bag)

/* Returns true only if bag is empty. */

bagcollection bag;

{

return(bag == ranking.next);

}

remove(pbag,pobj)

/* Sets obj to an object to be removed

from bag, and removes it.

*/

bagcollection *pbag;

int *pobj;

{

*pobj = ranking.rank[ranking.next];

ranking.next = ranking.next + 1;

}

process(listname,recordpointer,pbag,count)

/* Decreases the count of the successor pointed

to by recordpointer and adds it to the bag if

its count has become zero.

*/

listpointer listname,recordpointer;

bagcollection *pbag;

countcollection count;

{

int currentsucc;

currentsucc = info(recordpointer);

decrease(currentsucc,count);

if (iszero(currentsucc,count))

{

ranking.rank[*pbag] = currentsucc;

*pbag = *pbag + 1;

}

}


Comment on this Routine

from here on, the program is independent of the implementations of the data abstractions

topsort(pn,topologicalsort)

/* Inputs the number of objects n and the

partial order and generates a solution in

topologicalsort.

*/

int *pn;

outputarray topologicalsort;

{

countcollection count;

succ_collection succ;

bagcollection bag;

int i,j,obj;

printf("\n enter n between 1 and %d\n",LIMIT-1);

scanf("%d",pn);

predinitialization(*pn,count);

succinitialization(*pn,succ);

while (nextpair(&i,&j))

{

increase(j,count);

insert(i,j,succ);

{

baginitialization(&bag,*pn,count);

while(!emptybag(bag))

{

remove(&bag,&obj);

update(succ,obj,&bag,count);

}

copy(*pn,topologicalsort,ranking);

}

nextpair(pi,pj)

int*pi,*pj;

/* Input the next pair and return true

only if it was not the sentinel pair.

*/

{

print("\n enter a pair \n");

scanf("%d %d",pi,pj);

return(!((*pi == 0) && (*pj == 0)));

}

update(succ,obj,pbag,count)

/* Update the counts of all successors of obj

and place any whose counts become zero

into bag.

*/

succ_collection succ;

int obj;

bagcollection *pbag;

countcollection count;

{

listpointer listname,recordpointer,access_succ();

listname = access_succ(obj,succ);

recordpointer = listname;

while(anotherrecord(recordpointer))

{

process(listname,recordpointer,pbag,count);

recordpointer = next(recordpointer);

}

}


Comment on this Routine

storage allocated for count, succ_collection, and bag

countcollection count;

succ_collection succ;


Comment on this Routine

input n,the number of objects

printf("\n enter n between 1 and %d\n",LIMIT-1);

scanf("%d",pn);


Comment on this Routine

initialize the n counts to zero

predinitialization(*pn,count);


Comment on this Routine

initialize the n succ_collections to empty

succinitialization(*pn,succ);


Comment on this Routine

input the partial order

while (nextpair(&i,&j))

{

increase(j,count);

insert(i,j,succ);

{


Comment on this Routine

increase the jth count by one

increase(j,count);


Comment on this Routine

add j to the ith collection in succ

insert(i,j,succ);


Comment on this Routine

set bag to contain only those objects whose count is zero

baginitialization(&bag,*pn,count);


Comment on this Routine

generate a solution in ranking

while(!emptybag(bag))

{

remove(&bag,&obj);

update(succ,obj,&bag,count);

}

copy(*pn,topologicalsort,ranking);

}


Comment on this Routine

set obj to an object to be removed from the bag, and remove it

remove(&bag,&obj);


Comment on this Routine

decrease the count and place in the bag any successors of the removed object whose count has become zero

update(succ,obj,&bag,count);


Comment on this Routine

copy the solution into topologicalsort

copy(*pn,topologicalsort,ranking);


Comment on this Routine

this is a modification of the general list traversal function of Chapter 3

update(succ,obj,pbag,count)

/* Update the counts of all successors of obj

and place any whose counts become zero

into bag.

*/

succ_collection succ;

int obj;

bagcollection *pbag;

countcollection count;

{

listpointer listname,recordpointer,access_succ();

listname = access_succ(obj,succ);

recordpointer = listname;

while(anotherrecord(recordpointer))

{

process(listname,recordpointer,pbag,count);

recordpointer = next(recordpointer);

}

}


Comment on this Routine

initializes the current subtree to be traversed to t

p = *pt;


Comment on this Routine

test for completion of the traversal

while(p != null)


Comment on this Routine

process the current node p

process(pt,p);


Comment on this Routine

set lptr to the left subtree of p

lptr = left(p);


Comment on this Routine

if the left subtree is not null

if(lptr != null)


Comment on this Routine

save p on the stack

push(p,&s);


Comment on this Routine

set p to the left subtree

p = lptr;


Comment on this Routine

set rptr to the right subtree of p

rptr = right(p);


Comment on this Routine

if it is not null

if(rptr != null)


Comment on this Routine

set p to the right subtree

p = rptr;


Comment on this Routine

set p to the next nonnull right subtree to be traversed

p = nextsubtree(&s,&p);


Data Structures Using C: BIBLIOGRAPHY

 Return to Table of Contents

BIBLIOGRAPHY

Adelson-Velskii, G. M., and Landis, Y. M. [1962]. "An Algorithm for the Organization of Information." Doklady Akademia Nauk. 146(2), 263-266; Soviet Mathematics. 3, 1259-1263.

Aho, A. V., Hopcroft, J. E., and Ullman, J. D. [1983]. Data Structures and Algorithms. Reading, Mass.: Addison-Wesley.

Aho, A. V., and Ullman, J. D. [1972]. The Theory of Parsing, Translation, and Compiling, Vol. 1. Englewood Cliffs, N.J.: Prentice-Hall.

Aho, A. V., and Ullman, J. D. [1977]. Principles of Compiler Design. Reading, Mass.: Addison-Wesley.

Amble, O., and Knuth, D. E. [1974]. "Ordered Hash Tables." Computer Journal 17(2):135-142.

Austing, R. H., et al., eds. [1979]. "Curriculum '78: Recommendations for the Undergraduate Program in Computer Science--A Report of the ACM Curriculum Committee on Computer Science." Communications of the Association for Computing Machinery 22(3): 147-166.

Baase, S. [1978]. Computer Algorithms: Introduction to Design and Analysis. Reading, Mass.: Addison-Wesley.

Barnes, J. G. P. [1982]. Programming in Ada. Reading, Mass.: Addison-Wesley.

Barron, D. W. [1968]. Recursive Techniques in Programming. New York: American-Elsevier.

Bayer, R., and McCreight, E. M. [1972]. "Organization and Maintenance of Large Ordered Indices." Acta Informatica 1(3): 173-189.

Bays, C. [1977]. "A Comparison of Next-Fit, First-Fit, and Best-Fit." Communications of the Association for Computing Machinery 20(3):191-192.

Bell, D. A., and Deen, S. M. [1984]. "Hash Trees Versus B-Trees." Computer Journal 27(3):218-224.

Bell, J. R. [1970]. "The Quadratic Quotient Method: A Hash Code Eliminating Secondary Clustering." Communications of the Association for Computing Machinery 13(2): 107-109.

Ben-Ari, M. [1982]. Principles of Concurrent Programming. London: Prentice-Hall International.

Bentley, J. L. [1982]. Writing Efficient Programs. Englewood Cliffs, N.J.: Prentice-Hall.

Berry, J. T. [1986]. Advanced C Programming. New York: Prentice-Hall Press.

Bird, R. S. [1977a]. "Improving Programs by the Introduction of Recursion." Communications of the Association for Computing Machinery 20(11):856-863.

Bird, R. S. [1977b]. "Notes on Recursion Elimination." Communications of the Association for Computing Machinery 20(6): 434-439.

Bitner, J. R., and Reingold, E. M. [1975]. "Backtrack Programming Techniques." Communications of the Association for Computing Machinery 18:651-656.

Brooks, F. P. [1974]. The Mythical Man Month. Reading, Mass.: Addison-Wesley.

Bunneman, P., and Levy, L. [1980]. "The Towers of Hanoi Problem." Information Processing Letters 10:243-244.

Comer, D. [1979]. "The Ubiquitous B-Tree." Computing Surveys 11:121-137.

Crane, C. A. [1972]. Linear Lists and Priority Queues as Balanced Binary Trees. Unpublished Ph.D. Dissertation. Stanford University, Stanford, Calif.

Dahl, O.-J., Dijkstra, E. W., and Hoare, C. A. R. [1972]. Structured Programming. New York: Academic Press.

Dijkstra, E. W. [1972]. Notes on Structured Programming. London: Academic Press.

Dobosiewicz, W. [1978]. "Sorting by Distributive Partitioning." Information Processing Letters 7(1): 1-6.

Earley, J. [1970]. "An Efficient Context-Free Parsing Algorithm." Communications of the Association for Computing Machinery 13(2):94-102.

Elson, M. [1975]. Data Structures. Chicago, Ill.: Science Research Associates.

Fagin, R., Nievergelt, J., Pippenger, N., and Strong, R. H. [1978]. Extendible Hashing--A Fast Access Method for Dynamic Files. Research Rept. RJ2305. IBM Research Division, Yorktown Heights, N.Y.

Feller, W. [1975]. An Introduction to Probability Theory and Its Applicalions. 2nd ed. New York: Wiley.

Floyd, R. W. [1964]. "Algorithm 245: Treesort3." Communications of the Association for Computing Machinery 7(12):701.

Floyd, R. W. [1979]. "The Paradigms of Programming," Communications of the Association for Computing Machinery 22(8):455-460.

Fredkin, E. H. [1960]. "Trio Memory." Communications of the Association for Computing Machinery 3(9):490-500.

Garwick, J. V. [1964]. "Data Storage in Compilers." BIT 4:137-140.

Golomb, S. W., and Baumert, L. D. [1965]. "Backtrack Programming." Journal of the Association for Computing Machinery 12(4):516-524.

Greene, D. H., and Knuth, D. E. [1981]. Mathematics for the Analysis of Algorithms. Cambridge, Mass.: Birkhauser, Boston.

Harrison, S. P., and Steele, G. L., Jr. [1987]. C: A Reference Manual. Englewood Cliffs, N.J.: Prentice-Hall.

Hirschberg, D. S. [1973]. "A Class of Dynamic Memory Allocation Algorithms." Communications of the Association for Computing Machinery 16(10):615-618.

Hoare, C. A. R. [1962]. "Quicksort." ComputerJournal 5(1):10-15.

Horbibe, Y., and Nemetz, T. [1979]. "On the Max-Entropy Rule for a Binary Search Tree." Acta Informatica 12:63-72.

Horowitz, E. [1983]. Programming Languages: A Grand Tour. Rockville, Md.: Computer Science Press.

Horowitz, E., and Sahni, S. [1976]. Fundamentals of Data Structures. Woodland Hills, Calif.: Computer Science Press.

Horowitz, E., and Sahni, S. [1978]. Fundamentals of Computer Algorithms. Potomac, Md.: Computer Science Press.

Hu, T. C., and Tucker, A. C. [1971]. "Optimal Computer Search Trees and Variable-Length Alphabetic Codes." SIAM Journal of Applied Mathematics 21(4):514-532.

Huffman, D. A. [1952]. "A Method for the Construction of Minimum-Redundancy Codes." Proceedings of the Institute of Radio Engineers 40: 1098-1101.

Hull, M. E. C. [1984]. "A Parallel View of Stable Marriages." Information Processing Letters 18:63-66.

Johnson, D. B. [1975]. "Priority Queues with Update and Finding Minimum Spanning Trees." Information Processing Letters 4(3):53-57.

Jones, W. B. [1982]. Programming Concepts: A Second Course. Englewood Cliffs, N.J.: Prentice-Hall.

Kelley, A., and Pohl, I. [1987]. C by Dissection. Menlo Park, Calif.: Benjamin/Cummings.

Kernighan, B. W., and Plauger, P. J. [1978]. The Elements of Programming Style. 2nd ed. New York: McGraw-Hill.

Kernighan, B. W., and Plauger, P. J. [1981]. Software Tools in Pascal. Reading, Mass.: Addison-Wesley.

Kernighan, B. W., and Ritchie, D. M. [1978]. The C Programming Language. Englewood Cliffs, N.J.: Prentice-Hall.

Knuth, D. E. [1971]. "Optimum Binary Search Trees." Acta Informatica 1(1):14-25.

Knuth, D. E. [1973a]. The Art of Computer Programming. Vol. 1: Fundamental Algorithms. Reading, Mass.: Addison-Wesley.

Knuth, D. E. [1973b]. The Art of Computer Programming. Vol. 3: Searching and Sorting. Reading, Mass.: Addison-Wesley.

Koffman, E. B., Stemple, D., and Wardle, C. E. [1985]. "Recommended Curriculum for CSE, 1984." Communications of the Association for Computing Machinery 28(8):815-818.

Korsh, J. F. [1981]. "Greedy Binary Search Trees Are Nearly Optimal." Information Processing Letters 13(1):16-19.

Korsh, J. F. [1982]. "Growing Nearly Optimal Binary Search Trees." Information Processing Letters 14(3): 139-143.

Korsh, J. F., and Laison, G. [1983]. "A Multiple-Stack Manipulation Procedure." Communications of the Association for Computing Machinery 26(11):921-923.

Kruse, R. L. [1984]. Data Structures and Program Design. Englewood Cliffs, N.J.: Prentice-Hall.

Lapin, J. E. [1987]. Portable C and UNIX System Programming. Englewood Cliffs, N.J.: Prentice-Hall.

Larson, P. [1978]. "Dynamic Hashing." BIT 18(2):184-201.

Lin, S., and Kernighan, B. W. [1973]. "A Heuristic Algorithm for the Traveling Salesman Problem." Operations Research 21:498-516.

Linger, R. C., Mills, H. D., and Witt, B. I. [1979]. Structured Programming Theory and Practice. Reading, Mass.: Addison-Wesley.

Litwin, W. [1980]. "Linear Hashing: A New Tool for File and Table Addressing." Proceedings of the Sixth International Conference on Very Large Data Bases. Montreal, pp. 212-223.

Martin, W. A. [1971]. "Sorting." Computing Surveys 3:148-174.

Maurer, H. A., and Williams, M. R. [1972]. A Collection of Programming Problems and Techniques. Englewood Cliffs, N. J.: Prentice-Hall.

Maurer, W. D. [1968]. "An Improved Hash Code for Scatter Storage." Communications of the Association for Computing Machinery 11(1):35-37.

Maurer, W. D., and Lewis, T. G. [1975]. "Hash Table Methods." Computing Surveys 7(1):5-20.

Mehlhorn, K. [1975]. "Nearly Optimal Binary Search Trees." Acta Informatica 5:287-295.

Mehlhorn, K. [1977]. "A Best Possible Bound for the Weighted Path Length of Binary Search Trees." Society of Industrial and Applied Mathematics, Journal of Computers 6(2):235-239.

Miller, L. H., and Quilici, A. E. [1986]. Programming in C. New York: Wiley.

Morris, R. [1968]. "Scatter Storage Techniques." Communications of the Association for Computing Machinery 11(1):35-44.

Naur, P. (ed.). [1963]. "Revised Report on the Algorithmic Language ALGOL60," Communications of the Association for Computing Machinery 6:1-17.

Nievergelt, J. [1974]. "Binary Search Trees and File Organization." Computing Surveys 6(3):195-207.

Peterson, J. L. [1980]. Design of a Spelling Program: An Experiment in Program Design. Lecture Notes in Computer Science. New York: Springer-Verlag.

Peterson, J. L., and Norman, T. A. [1977] . "Buddy Systems." Communications of the Association for Computing Machinery 20(6):431-435.

Pratt, T. W. [1975]. Programming Languages: Design and Implementation. Englewood Cliffs, N.J.: Prentice-Hall.

Reingold, E. M., and Hansen, W. J. [1983]. Boston, Mass.: Little, Brown.

Robson, J. M. [1973]. "An Improved Algorithm for Traversing Binary Trees without Auxiliary Stack." Information Processing Letters 2:12-14.

Robson, J. M. [1977]. "A Bounded Storage Algorithm for Copying Cyclic Structures." Communications of the Association for Computing Machinery 20(6):431-433.

Robson, J. M. [1979]. "The Height of Binary Search Trees." Austrian Computer Journal 11(4):151-153.

Schorr, H., and Waite, W. M. [1967]. "An Efficient Machine-Independent Procedure for Garbage Collection in Various List Structures." Communications of the Association for Computing Machinery 10(8):501-506.

Sedgewick, R. [1977]. "Permutation Generation Methods." Computing Surveys 9:137-164.

Sedgewick, R. [1983]. Algorithms. Reading, Mass.: Addison-Wesley.

Shell, D. L. [1959]. "A High-Speed Sorting Procedure." Communications of the Association for Computing Machinery 2:30-32.

Standish, T. A. [1980]. Data Structure Techniques. Reading, Mass.: Addison-Wesley.

Stanfel, G. L. [1970]. "Tree Structures for Optimal Searching." Journal of the Association for Computing Machinery 19(3):508-517.

Stone, H. S. [1971]. "Parallel Processing with the Perfect Shuffle." IEEE Transactions on Computers C-20: 153-161.

Stone, H. S. [1972]. Introduction to Computer Organization and Data Structures. New York: Mc-Graw-Hill.

Stone, H. S. [1980]. Introduction to Computer Architecture. 2nd ed. Chicago, Ill.: Science Research Associates.

Sussenguth, E. H. Jr. [1963]. "Use of Tree Structures for Processing Files." Communications of the Association for Computing Machinery 6(5):272-279.

Tenenbaum, A. M., and Augenstein, M. J. [1981]. Data Structures Using Pascal. Englewood Cliffs, N.J.: Prentice-Hall.

Tremblay, J. P., and Sorenson, P. G. [1984]. An Introduction to Data Structures with Applications. 2nd ed. New York: McGraw-Hill.

Ullman, J. D. [1982]. Principles of Database Systems. Rockville, Md.: Computer Science Press.

Van Wijngaarden, A., et al. [1975]. "Revised Report on the Algorithmic Language ALGOL 68." Acta Informatica 5:1-236.

Walsh, T. R. [1982]. "The Towers of Hanoi Revisited: Moving the Rings by Counting the Moves." Information Processing Letters 15(2):64-67.

Weinberg, G. M. [1971]. The Psychology of Computer Programming. New York: Van Nostrand.

Wexelblat, R. L. [1981]. History of Programming Languages. New York: Academic Press.

Williams, J. W. J. [1964]. "Algorithm 232: Heapsort." Communications of the Association for Computing Machinery 7(6):347-348.

Wirth, N. [1973]. Systematic Programming: An Introduction. Englewood Cliffs, N.J.: Prentice-Hall.

Wirth, N. [1976]. Algorithms + Data Structures = Programs. Englewood Cliffs, N.J.: Prentice-Hall.


Data Structures Using C: CHAPTER 1: PROGRAMMING STRUCTURE

 Return to Table of Contents

CHAPTER 1: PROGRAMMING STRUCTURE

Emphasizes the importance of

structuring programs

using an explicit problem solving methodology and top-down design to create programs

good program style

Introduces and illustrates the components of good programming style

data abstraction

functional modularization

verification and debugging

documentation

Begins the discussion of program time and storage requirements and how to determine them

Case study--scoring bowling games

illustrates the concepts introduced in the chapter

1.1: Structure: Data and Program

Two programs doing exactly the same task may look entirely different. One may be highly readable, concise, and easily adjusted to carry out related tasks. The other may be impenetrable, lengthy, and difficult to modify. The same two may differ so much in execution time and storage needs that one program executes while the other aborts, having exceeded time or space restrictions. Experience has shown such differences to be traceable to choice of program structure and data structure.

Programs are written to solve real problems. Program structure breaks the problem and its solution down into simpler, more easily understood parts. The information to be processed is stored in data structures (arrays, records, lists, stacks, trees, and files) provided by the programming language. A data structure groups data. The right data structure for an operation can make it simple and efficient; the wrong one can make the operation cumbersome and inefficient.

Data structures contain information and are operated on during the execution of a program. Programs are said to process information when, in fact, they process data structures. Thus it is not surprising that data and program structure are important and need to be properly related to achieve the goal of successful programming. How this is done is the subject of this book. The story is fascinating and sometimes involved, but it has a happy ending: you master practical programming techniques. These techniques provide the programmer with the means to keep a tight rein on programs so that they remain understandable, maintainable, and efficient, even as they grow in size.

There is structure, then, in both the data and the program itself, and both program and data structure must be appropriate to their tasks. This book presents data structures, not as an isolated theoretical subject, but as an essential object of the problem-solving process leading to the creation of a good program.Three main themes are stressed.

1. Data structures and their impact on programs

2. Programming methodology and its impact on programs

3. Consolidation and extension of knowledge of a high-level language

Data structures are important because the way the programmer chooses to represent data significantly affects the clarity, conciseness, speed of execution, and storage requirements of the program. The text shows how to use particular data structures to create correct and efficient programs. You will see that the operations to be performed on the data determine the best data structures to use.

Developing programs is difficult, especially when done without good guidelines. Creating programs entails solving problems. Top-down programming, another name for structured programming, makes program development easier and leads to well-structured programs. It is the controlled breaking down of a complex task into simpler ones whose individual solutions easily combine to carry out the original task. This process gives structure to the development of programs (hence its name). The top-down, or structured, approach, combined with good programming style, is emphasized in this text. Elements of good programming style include the use of data abstractions and functional modularization, which significantly enhance the readability and changeability of programs.

Data abstraction treats a collection of data by abstracting its important aspects while ignoring as much detail as possible. A data abstraction reduces data to a collection, and to those processing operations directly allowed on that collection. The effect is as though the collection were in an impenetrable black box, the only access to the contents of the box being through the invocation of one or more of the allowed operations. How the collection is actually stored within the box, and how the operations are actually performed, become irrelevant details. Such details determine the efficiency of a program but do not affect its logic.

Functional modularization is the use of a special program unit to carry out a task. In FORTRAN such a unit would be a function or subroutine, in Pascal a function or procedure, in COBOL a paragraph, and in C a function. These program units are used to carry out meaningful processing tasks. C programmers use the word modularization in two ways. One means functional modularization, and the other has to do with packaging a data abstraction using one or more related files. When the first meaning is intended, this book uses functional modularization. The second interpretation appears only in Chapter 5 and will be addressed there.

Practice may not make perfect but seems to be the best way to make problem solvers and programmers out of computer science students. As a student you must repeatedly practice creating programs, because repetition is what facilitates learning. As in learning to dance, draw, or drive, no matter how closely you watch others perform, you can develop skills only by practice.

Finally, whatever high-level programming language you use, studying and understanding the material in this text will enhance your knowledge of that language and your ability to use it effectively. It is assumed that you are familiar with some high-level programming language. Such languages share many features, but each is unique. Although this text occasionally indulges in comparative investigations, all programs herein are written in VAX C, version 2.2, and run on a VAX 11/780 under VMS.

Horowitz [1983] is a collection of reprints of the landmark papers in programming languages. For background in C, see the introductory texts by Kelley and Pohl [1986], Miller and Quilici [1986], and the classic book by Kernighan and Ritchie [1978]. A good reference is Harbison and Steele [1987]. Advanced topics can be found in Berry [1986] and Lapin [1987]. For interesting reading in structured programming, the reader should study the texts by Dahl, Dijkstra, and Hoare [1972] and Wirth [1973].

1.2: A Look Ahead

A nucleus of commonly used data structures will be introduced. These include arrays and records, which are static structures that change only their values, and stack, lists, trees, and files, which are dynamic structures whose size and shape change as well as their values. Specific ways to implement each structure will be explained and typical and important operations on them investigated. These operations include selection, traversal, insertion, deletion, searching, and sorting. Although particular methodologies and data structures are important, it is more important to acquire skill in applying a methodology and in determining which data structures are appropriate for a given problem.

Programs are built from data structures. Yet the methodology an expert programmer uses makes the data structures invisible and makes the program structure clear and understandable. Neither good program structure nor good data structure selection alone produces good programs. Both must be done well, but they cannot be done independently. Obtaining a good program requires a well-designed solution, expressed first as an algorithm. The final step is writing the program in a programming language.

Determining what information is required to implement the algorithm and how to store it for efficient use and processing are perhaps the most crucial aspects of that facet of problem solving that is commonly called computer programming. The aim of this text is to illustrate good algorithm design, data structure selection, and programs.

If you do not know how to solve a given problem, then you cannot magically write a program to do so. In other words, if you don't know where you are going, then it doesn't matter how you get there. Moreover, as H. L. Mencken said, "Every complex problem has a simple easy-to-understand wrong answer." So before you begin writing a program, you will need to understand the problem and write a step-by-step solution, or algorithm.

Unfortunately, there is no algorithm for finding algorithms. There isn't even an algorithm for checking a candidate to see if it really is an algorithm. But there are some helpful techniques for developing algorithms. For example, it is useful to recognize when a known algorithm or program can be modified to give a solution to a new problem. One way to enhance this recognition is to solve problems in general or abstract terms. Sometimes a problem requires finding an object stored in computer memory that has certain properties. One technique is to search through the set of all objects until one with the desired properties is found. Alternatively, objects without the desired properties may be eliminated. The latter approach is used to develop a prime number algorithm in Section 1.3. At other times it may be possible to avoid searching altogether by constructing the desired object directly. Thinking up an algorithm is usually more difficult than programming it.

1.3: The Need for Structure

Imagine that you are a census taker charged with visiting all the homes in Spokesville and Squaresville (Figure 1.1). In each town, you must start and finish at home 1, after visiting each home once. All roads are the same length, and there are the same number of homes in each town. Nonetheless, in Spokesville you will travel a total distance exactly twice the distance required in Squaresville! The difference is due to structure.

It is easy to find the telephone number of a company whose name you know, even in the telephone directory of a large city. But finding the name of a company when you know only its telephone number is a tedious and time-consuming chore.The information in the telephone directory is structured in such a way that one task is easy while the other causes tension headache and eyestrain!

Two maps of the continental United States appear in Figure 1.2; part (a) gives a clear overview, while (b) gives a more detailed view. Structure clarifies, and detail obscures. Still, both are needed for a complete picture. Notice that drawing a border around and naming each state in the overview, in the absence of other distracting details, makes the composition of the country discernible. It is the plethora of detail and subordination of structure that is overwhelming in the other map. The less detailed map is analogous to a functionally modularized program, the other to an unmodularized program.

An algorithm for a given problem specifies a series of operations to be performed that eventually leads to a correct result. Algorithms express solutions step by step. In stating an algorithm, care must be taken to avoid ambiguity and to be sure that it is clear how each step is to be done. When the solution is translated into a programming language for execution by a computer, even more care and attention to detail are needed. But first, the solution is usually expressed in English or flowchart form, before the details required by the programming language are introduced. The next example illustrates the effect of structure in this context.

(a) Spokesville

(b) Squaresville

Figure 1.1 A Tale of Two Cities

An integer larger than 1 is a prime number if it cannot be written as the product of two integers, each larger than 1. Suppose we must find a way to print all prime numbers between 2 and some integer n. Thus if n is 12, the primes are 2, 3, 5, 7, and 11. Compare the following two algorithms for accomplishing this task.

                 primes(n)                          anotherprimes(n)

1. Create a collection called candidates,    1. Set X1, X2, . . . , Xn to

   consisting of all integers from 2 to n.      zero.

2. Remove all nonprime integers from         2. Set k to 2.

   candidates.

3. Print all integers that remain in         3. If Xk is 1, go to step 7.

   candidates.

                                             4. Print k.

                                             5. Set m to the integer part of

                                                n/k.

                                             6. Set Xk, X2k, . . . , Xmk to 1.

                                             7. Set k to k + 1.

                                             8. If k    sqrt n, go to

                                                step 3.

                                             9. If Xk is zero, print k.

                                            10. Set k to k - 1.

                                            11. If k    n, go to step 9.

(a) Overview

Figure 1.2(a) Courtesy: Hammond Incorporated, Maplewood, NJ 07040.

(b) Detailed View

Figure 1.2(b) Courtesy: Hammond Incorporated, Maplewood, NJ 07040.

It is not difficult to see that if each step of the first algorithm can be carried out, the correct result will be printed. It is considerably more difficult to see that the same is also true of the second algorithm. The first is structured to provide clarity, while the second is poorly structured and provides too much detail. These examples indicate why structure is needed in each stage of program design. Structured solutions are easier to understand, expand, and alter, and structured data make the operations of the algorithm easier and faster to perform.

1.3.1 Using Structure

Well-designed, clear programs are based on the explicit underlying methodology called structured programming. As problems and their solutions become more sophisticated, the brute-force or ad hoc method of programming becomes much less effective. Compilers, operating systems, business management systems, and word processing systems consist of many thousands of program statements. Just as writing a lengthy book is several magnitudes more difficult than writing a paragraph, so is the process of developing a large programming system much more involved than writing a short program. There must be an overall structure.

An example of a large and complex program is an operating system. Many years ago it was thought that operating systems could not be implemented without errors. The immensity of the task was believed to cause inherent problems that could not be isolated and debugged in advance of system implementation. Hence system failure seemed inevitable and getting the system up and running meant a long trial-and-error period of testing and debugging. A new methodology of system design based on the concept of structured programming has demonstrated, however, that error-free operating systems are possible.

Since the programs you see as a student may be relatively trivial, the need for a well-founded methodology of programming may not be evident. However, imagine a large program (twenty pages long, and over a thousand lines of program statements) that you did not write. If you were assigned the task of modifying the program to add a new feature, how or where would you start? Certainly you must have a full understanding of what the program does, how it does it, and how it relates to other programs that use it or are used by it. Anyone who intends to read, write, modify, or correct any programming systems with more than a few dozen statements should use structured programming.

1.3.2 A Prime Example

The example of prime numbers is presented here in more detail to illustrate the process and benefits of structuring and the concept of data abstraction. To do this a group of functions is developed from the first prime number algorithm:

primes(n)

1. Create a collection called candidates, consisting of all integers from 2 to n.

2. Remove all nonprime integers from candidates.

3. Print all integers that remain in candidates.

Consider first a simple function called primes.

primes(n)

int n;

{

collection candidates;

create(n,candidates);

remove(n,candidates);

print(n,candidates);

}

Primes may be obtained directly from the prime number algorithm when create, remove, and print are interpreted as functions carrying out steps 1, 2, and 3 of the algorithm, respectively. Note that no assumptions about candidates have been made. At this point it is of type collection, which has not yet been defined. Later in the process of developing the algorithm and program, collection, and thereby candidates, will be defined, after the basic operations on candidates become clear. Note that collection is a type, whereas candidates is a variable whose type must be declared. The next step is to refine and verify each function of primes. Refine means to express or define in more detail; verify means to confirm that a refinement does indeed do its job. To refine and verify create and print is straightforward enough, but to refine and verify remove is more difficult. If you want to know more about a state than you see in the first map of Figure 1.2, you may consult another map containing more information on the state's geography. Similarly, if you want to know how to carry out remove, a more detailed description may be available. If not, as is the case here, remove may be treated as a new problem whose solution must be found. One solution is this:

remove(n, candidates)

Set factor to the first prime.

While (nonprimes remain in candidates),

Delete from candidates all integers that are multiples of factor, and

Increase factor by 1.

It is again easy to see that, as long as each step is done correctly, the desired effect is obtained.

Figure 1.3 PRIMES Modularized

Still, more examination is needed to determine whether "nonprimes remain in candidates." It can be shown that nonprimes may remain in candidates only when factor is less than or equal to the square root of n. Consequently the condition (nonprimes remain in candidates) may be replaced by the condition [image]. But doing this will make remove more difficult to understand. It is better to consider the evaluation of the condition as a new task that must be refined. We are now ready to write the function for remove and to verify it. The three functions appear in Figure 1.3, where set initializes the collection candidates to empty, insert adds the integer i to the collection, nonprimes is a function assumed to return the value true when [image] and false otherwise, delete removes all integers from candidates that are multiples of factor, and belongs returns the value true if i is in candidates and false otherwise.

Delete is refined next (Figure 1.3). You should verify that this function correctly performs its task as long as omit takes nextmultiple out of candidates. It may help to simulate the execution of delete when n is 40 and factor is 2, and then 3.

Each task of function primes has now been defined and is, in fact, a complete C function. All other tasks, except for set, insert, belongs, and omit, have also been defined. Primes and all other tasks have been written modularly (functionally), meaning that for each task there is a function. For example, new functions have been introduced for each of the three tasks of primes: create, remove, and print. Had primes along with remove and delete not been functionally modularized, it would appear as follows:

primes(n)

int n;

{

collection candidates;

int factor,i,nextmultiple;

int firstprime;

firstprime = 2;

double sqrt();

>Comment

set(n,candidates);

>Comment

for (i=2;i<=n;i++)

insert(i,candidates);

factor = firstprime;

while (factor <= sqrt((double)n))

>Comment

{

nextmultiple = 2 * factor;

while (nextmultiple <= n)

{

omit(nextmultiple,candidates);

nextmultiple = nextmultiple + factor;

}

factor++

}

>Comment

for (i=2;i<=n;i++)

if(belongs(i,candidates))

printf("\n %d\n",i);

}

This version is also a refinement of primes, but instead of defining new functions for its three tasks, the tasks have been replaced by program segments. Thus there are two basic ways to refine any task: (1) define a new function for the task, or (2) replace the task directly by its refinement expressed as a program segment in code. Compare this version with the functionally modularized form, the simple function primes given earlier. Notice that functional modularization suppresses detail.

Deciding how to modularize a program functionally is often a matter of individual taste, but as this text unfolds, the process will become clearer, and you will be able to develop your own style. Functional modularization is intended to highlight and isolate important tasks. The advantage of using functions may not be apparent with this simple prime number problem but will be striking in more complex problems.

Finally, the low-level tasks set, insert, omit, and belongs must be coded. To do this requires deciding how to implement candidates--that is, deciding which data structures shall be used to store candidates and how to write the basic functions that operate on candidates. Recall that data structures such as candidates contain information and are operated on during the execution of a program. No matter how candidates is implemented, none of the functions already defined needs to be changed. They are independent of candidates implementation (except for the data declaration). As long as set, insert, omit, and belongs do their tasks correctly, no function invoking them needs to be changed. This is because we have written them treating candidates as a data abstraction. That is, we have assumed that candidates and the basic operations on it are available via functions (set, insert, omit, and belongs) and further, that any other operations on candidates have been expressed in terms of these functions. For instance, delete uses omit, and create uses insert. If, instead of calls to set, insert, omit, and belongs, the code defining them had been used, all functions containing these codes would be dependent on the implementation. Later, if we changed to a better implementation, it would be necessary to hunt through each function, find any code that would be affected by the change, and modify that code appropriately. A data abstraction consists of a data structure, to store data, and allowed operations on the data structure, to manipulate it. As a result, wherever one of these operations is invoked in a program, the data structure is being affected. Just as important, the data structure is not being affected unless one of these operations is invoked.

The use of data abstraction and functional modularization not only enhances understanding, but also simplifies maintaining a program. Maintenance is a crucial aspect of programming. It involves changing a program to do something new, to stop doing something, or to do better what it now does.

Using data abstractions as outlined above hides implementation details and makes the program independent of those details. On the other hand, the program's speed of execution and storage requirements will be greatly influenced by the implementation of the data abstraction. Consequently, the selection of an implementation should be based on the data abstraction's operations. Sometimes the choice that minimizes execution time and storage will be evident. More typically, conflicts arise, and the choice will involve trade-offs between time and storage. How to make this choice properly is a major focus of this text.

Consider two ways to implement candidates. It might be implemented as an integer array with the ith element of the array containing the value true if i is a prime and false if it is not. Alternatively, candidates could be implemented as an integer that is made up of contributions from each number in candidates. If the number i is in candidates, it contributes 2i. Thus if candidates consisted of 2, 3, 5, 7, and 9, its value would be 684(4 + 8 + 32 + 128 + 512). The integer is stored in an integer array of size one. Candidates could also be implemented as a dynamic list or tree and, indeed, one of these might be preferred. You might want to return to this problem after studying lists and trees, to verify that it can be converted to these structures. However, at this stage, arrays that are more familiar to you will be used. The code below defines the operations on candidates for the two array implementations. Each implementation should have its type declaration for collection global to primes and all its functions, as follows:

   Candidates Implemented          Candidates Implemented

         as an Array                    as an Integer

typedef int collection[MAXN];  typedef int collection[1];

set(n,c)                       set(n,c)

/* Sets c to empty */          /* Sets c to empty */

int n;                         int n;

collection c;                  collection c;

{                              {

   int i;                         c[0] = 0;

   for (i=1;i<=n;i++)          }

      c[i] = FALSE;

}

insert(i,c)                    insert(i,c)

/* Inserts i into c */         /* Inserts i into c */

int i;                         int i;

collection c;                  collection c;

{                              {

   c[i] = TRUE;                   int p,q;

    >Comment

}                                 p = power(2,i);

                                  q = (c[0]/p)%2;

                                  c[0] = c[0] + (1-q)*p;

                                }

omit(i,c)                       omit(i,c)

/* Removes i from c */          /* Removes i from c */

int i;                          int i;

collection c;                   collection c;

{                               {

   c[i] = FALSE;                   int p,q;

    >Comment

}                                  p = power(2,i);

                                   q = (c[0]/p)%2;

                                   c[0] = c[0] - q*p;

                                }

belongs(i,c)                    belongs(i,c)

/* Returns true only            /* Returns true ony

   if i is in c                    if i is in c

*/                              */

int i;                          int i;

collection c;                   collection c;

{                               {

   return(c[i]);                   return((c[0]/power(2,i))%2);

}                               }

                                power (x,y)

                                /* Returns x to the power y */

                                int x,y;

                                {

                                   int i,p;

                                      p = 1;

                                   for (i=1;i<=y;++i)

                                      p = p*x;

                                   return(p);

                                }

Note that the terms TRUE and FALSE are used in the boolean version. Since C does not have a type boolean, TRUE and FALSE must be defined before they are used in the program. In C, a true statement returns a non-zero value, and a false statement returns a zero. For consistency, write

# define TRUE 1

# define FALSE 0

TRUE and FALSE will be used for programming style in many places in the text. Programs that use them merely require the above statements. In the second version the function power (2,i) returns 2i.

Each implementation specifies a data structure for candidates and defines the same four basic operations allowed on it. Each entails some explicit restriction on the size of candidates. Such restrictions can be avoided by using the dynamic data structures introduced in later chapters.

The first implementation of candidates allows insert, omit, and belongs to be written so that they take constant time to execute. However, the amount of storage required will depend directly on n. The second implementation for candidates increases the execution times of the basic operations on candidates but reduces its storage requirement. In general, the choice of implementation for a data abstraction may be critical, determining whether a program will run to completion or abort because of insufficient time or storage. Section 1.8 discusses the selection of program and data structure in more detail. In any case, once the decision is made, the data declaration for candidates must appear in primes. The final result is a well-structured program. A complete program to run primes using the array implementation follows.

#include <stdio.h>

    >Comment

main()

{

int n;

printf("\n n = ? \n");

scanf("%d",&n);

primes(n);

}

    >Comment

#define TRUE 1

#define FALSE 0

#define MAXN 500

typedef int collection[MAXN];

set(n,c)

/* Sets c to empty */

int n;

collection c;

{

int i;

for (i=1;i<=n;i++)

c[i] = FALSE;

}

insert(i,c)

/* Inserts i in c */

int i;

collection c;

{

c[i] = TRUE;

}

omit(i,c)

/* Removes i from c */

int i;

collection c;

{

c[i] = FALSE;

}

belongs(i,c)

/* Returns true only

if i is in c

*/

int i;

collection c;

{

return(c[i]);

}

    >Comment

primes(n)

/* Prints all prime numbers between 2 and n */

int n;

{

collection candidates;

create(n,candidates);

remove(n,candidates);

print(n,candidates);

}

create(n,c)

/* Creates a collection of integers between 2 and n */

int n;

collection c;

{

int i;

    >Comment

set(n,c);

    >Comment

for (i=2;i<=n;i++)

insert(i,c);

}

remove(n,c)

/* Removes all nonprimes from c */

int n;

collection c;

{

int firstprime;

int factor;

firstprime = 2;

factor = firstprime;

    >Comment

while (nonprimes(factor,n))

{

delete(factor,n,c);

factor++;

}

}

nonprimes(factor,n)

/* Returns true only if non

primes may remain in c

*/

int factor,n;

double sqrt();

{

    >Comment

return(factor <= sqrt((double)n));

}

delete(factor,n,c)

/* Deletes multiples of factor from c */

int factor,n;

collection c;

{

int nextmultiple;

nextmultiple = 2 * factor;

    >Comment

while (nextmultiple <= n)

{

omit(nextmultiple,c);

nextmultiple = nextmultiple + factor;

}

}

print(n,c)

/* Prints integers in c */

int n;

collection c;

{

int i;

    >Comment

for (i=2;i<=n;i++)

if (belongs(i,c))

printf("\n %d\n",i);

}

As noted, the first implementation for collection is used in the program. Notice that the declarations and definitions needed for the collection appear together in the program. Simply replacing that portion of the program with another implementation is all that is required to change to the new implementation. In Chapter 5 another way to package the collection and its operations for this program will be presented.

Abstraction allows us to focus on concepts rather than concrete details. The architects planning a building assume an elevator as part of the plan. They know the elevator will hold passengers and freight. They know the elevator will convey its contents between floors, respond to call buttons, and be warm in winter and cool in summer. The initial building plans can proceed using only these essential concepts regarding the elevator. Later, the detailed choices necessary to build the elevator will be made. They will determine its size, shape, and structure, as well as its speed, responsiveness, and reliability. Notice that, as long as the elevator is treated as an abstraction, any specific implementation can be substituted for another without requiring changes to the plan.

Similarly, data abstraction allows us to focus on what is needed rather than how it will be assembled and packaged. Planning a program, we assume that we have specific means to store and operate on data. Only later must an implementation be selected. The implementation determines how well the job is done, but does not affect its logical design. The longer this choice can be delayed in program development, the more independent of this specific choice our program will be. This is highly desirable, making program changes much easier to effect.

1.4: Basic Constructs

The richness and power of a language are related to the variety and ease of expression that the language allows. Thus high-level computer languages provide more expressive power than more limited languages, such as assembly languages. However, it is theoretically possible to state any algorithm using only three basic constructs and the rules for their combination. In this discussion three basic constructs have been chosen for the sake of conciseness, but others are used freely when appropriate.

The three constructs indicate different types of flow of control. Each construct determines the order in which its tasks are processed, and each was chosen for its logic and clarity.

The first is the sequence construct, in which each task follows the previous one without altering the order of the tasks. Each task is executed once. Notice that primes (Figure 1.3) is based on this construct. The second is the decision construct, which allows only one of two tasks to be carried out, depending on a controlling logical condition. The appropriate task is executed once. It is the true task when the condition is true and the false task when the condition is false. The final construct is the loop, which enables a task to be repeated until a controlling condition no longer holds. Obviously, the loop must at some point end so that the next task can be executed. Failure to change the value of the controlling condition in order to end the loop task results in the infamous "infinite loop." Examples of algorithms formulated according to the three basic constructs appear in Figure 1.4 as structured flowcharts, along with their structured English equivalents. The corresponding versions are obvious.

Each construct has exactly one entry and one exit point. The constructs may be combined only by making the exit arrow of one the entrance arrow of another. This results in complex constructs that are sequences of constructs. The only allowed modification of a construct is the replacement of any of its tasks by one of the three basic constructs. This amounts to incorporating the task's refinements into the construct, producing a more complex construct.

Figure 1.4 The Basic Control Constructs and Their English Equivalents

There are two different ways to express a solution using these constructs and rules for their combination. One is to start with a basic construct and repeatedly apply modifications to it. Each modification corresponds to the use of a program segment for its implementation. This results in a final unmodularized flowchart, English equivalent, or program. See the unmodularized version of primes for an example. The second is to start with a basic construct, and, instead of modifying it, express the solution for each of its tasks as a separate function. This amounts to functionally modularizing each task's refinement, with the complete solution consisting of the collection of individual solutions. This results in a final collection of functionally modularized flowcharts, their English equivalents, or program components. Usually a combination of the two is better than using one method only. It is wise to keep each part of a solution to a reasonable size, say one page. This can serve as a guide to whether or not functional modularization (the use of a function) or modification (the use of a program segment) is appropriate. But remember: the primary goal is clarity.

1.5: Top Down Design

The purpose of structured programming is to create programs that are

      Understandable, meaning the logic is readily followed

      Reliable, so the program performs as intended

      Adaptable, so modifications can be made with reasonable effort

Structured programming is a method of approaching a problem rather than adhering to a rigid set of rules.

As Dijkstra [1972] observed, the "art of programming is the art of organizing complexity, of mastering multitude and avoiding its bastard chaos as effectively as possible." The essential concept of structured programming is that dealing with complexity requires simplicity.

Regarding terminology, note that the tasks performed by a program solve problems. In this book the words task and problem are used interchangeably. Each programming task entails defining the input information to be processed, defining the output information to be produced, and stating clearly the precise instructions as to what, but not how, it is to be done.

Simplicity and clarity are achieved in a complex program by focusing attention on just one task at a time. To show how the task is done, it is broken down into smaller meaningful subtasks. The combination of smaller tasks to perform the given task is called a refinement of that task. The next step is verifying that the refinement is correct--that is, that the refinement indeed accomplishes the task, as long as each of its components does its job. This verification may be done even without knowing how each component works. This is important. How the components of a refinement work is irrelevant to verification. The goal is to ensure that, if they work correctly, then the refinement itself works correctly.

Top-down design begins with the original problem and involves a loop: repeated refining and verifying of the problem and its refinements until only implementable problems remain. Implementable means simple enough to be directly coded in a programming language. Review the development in Section 1.3.2 for primes to see that this loop was, in fact, being used. With this approach the programmer can be sure the final program is correct even though he or she has considered only one manageable piece at a time.

If the total number of tasks needed to complete a detailed solution is proportional to the complexity of the initial problem (as we hope), then the effort to create, understand, and verify a solution to a complex problem will also be proportional to its complexity. This is a very desirable situation. If the cost of a solution increased more drastically with complexity, then the complexity of problems we could solve would be much more severely limited.

Think about building a jigsaw puzzle. You probably use the top-down approach. You break up the picture conceptually into components--say, the border and recognizable objects with distinct shapes and colors. Then you build each component and combine them all. Surely the most difficult puzzles are those with no distinguishable objects. They prevent refinement, forcing one to deal with all the complexity at one time. Perhaps this is why all the problems humans solve have structure; they are the only ones we can handle.

Two caveats must be added:

1. At each level of refinement, keep the number of component tasks small and keep them independent. This is critical because human beings can deal well with only a few things at a time, and can do so more readily when the things are not intertwined. Psychological studies suggest a "magic number": a maximum of seven tasks, plus or minus two. Independence means that how one task is accomplished need not be a consideration in how another task is done. Sometimes independence must be given up for other considerations, such as efficiency. These trade-offs should be considered carefully.

2. Control carefully how tasks of a refinement are combined. Simplicity requires structure that makes the logic of a program easily discernible. Certain combinations or constructs foster such clarity. Some programs present a bewildering panorama of program statements, but what is needed are programs that appear as a logical sequence of tasks to be executed. Some ways of expressing the order in which tasks are to be executed are more understandable than others.Constructs that are well designed allow anyone working with a program to grasp quickly what is intended. This is the hallmark of the basic constructs--sequence, decision, and loop--introduced in Section 1.4.

1.6: High-Level Languages

High-level languages provide basic data structures and operations and the capability of using these to build more complex data structures and operations. With some languages it can be hard to realize this capability, while with others it is relatively effortless. In addition, languages have different idiosyncrasies and subtle shortcomings. In this text, the focus is not on the language but on the more general topics of program structure and data structure selection. Still, whatever background you have in high-level languages may be helpful.

A program written in a high-level language specifies a series of operations to be performed on data structures. Think of a high-level language as having an associated high-level computer that carries out those operations and stores those data structures in its memory. Such high-level computers are just a concept--normally they are not built. The conventional computer is not specifically a C, Cobol, Pascal, or FORTRAN machine; we rely on compilers to translate programs from high-level languages into the languages of existing computers. The translated version is then executed by the conventional computer. The compiler, in effect, makes the conventional computer appear to be a high-level computer for the high-level language.

To do this translation, the compiler must find a way to represent the program's data structures, using those in the conventional computer's language. lt must also apply operations of the conventional computer's language to these representations to simulate the program's operations on its data structures. How this is done, or what the actual computer is, does not affect the correctness of programs written in the high-level language. This is data abstraction par excellence! However, the way this is done will influence the storage requirements and execution time of the program. It can also cause problems for the unwary user.

1.6.1 Data Types

Programs use variables to refer to storage elements in the memory of the high-level computer. Each variable has a name, which refers to its storage elements, and a value, which is the content of those elements. For example, the C assignment statement

sum = sum + increment;

uses two variables with names sum and increment. Individual variables, which correspond to a single storage element, are the simplest type of data structure. More complex structures are built from variables. In turn, operations on the more complex structures are built from operations on the simpler structures. High-level languages differ in the kinds of basic data structures they provide. For example, FORTRAN includes individual variables and dimensioned arrays. C adds records and pointers and, significantly, allows the definition of new data types and data structures. The Ada language allows new basic operations to be introduced as well.

Each variable must be defined by a declaration that gives its name and data type. The data type indicates the actual values that may by assumed by the variable. For example, the C declaration

int sum, increment;

char check;

specifies that sum and increment are the names of variables whose values will be integers, and that check is the name of a variable whose values will be a character.

1.6.2 Data Representation

Data types must be specified. An example will show why. Suppose a compiler represents integer and real variables (floats) using seven decimal digits. Then in storage that is designated type integer, the value 1230045 represents the integer number 1,230,045. In storage that is designated type float the value 1230045 represents the real number 0.30045 x 10l2. In general, the compiler interprets

d1 d2 . . . d7 of type integer as the integer number dl d2 . . . d7.

and

d1 d2 . . . d7 of type float as the real number 0.d3 d4 . . . d7 x 10d1d2

Given this situation, suppose the compiler must cause the contents of sum = 1230045 to be added to the contents of increment = 1250047,where they are both of type integer. The compiler might apply the usual column-by-column addition algorithm:

sum        1230045

+

increment  1250047

           -------

           2480092

Here, 2480092, when stored in sum, represents the correct result. If sum and increment are of type real, then the content of sum represents the real number 0.30045 x 1012 and the content of increment represents the real number 0.50047 x 1012. Their sum is 0.80092 x 1012. However, if the above column-by-column addition algorithm is applied and the result stored in sum, then its contents will still be 2480092, but this now represents the real number 0.80092 x 1024. This is incorrect by a factor of 1012. To represent the correct result, sum should contain 1280092. The difficulty is, of course, that the same algorithm should not be applied for the addition of numbers of both type integer and type float. Two distinct algorithms are needed, because two distinct representations are used. In general, the algorithms used must depend on the data types of the variables involved. The compiler can choose the correct algorithm only when it knows the data types.

Thus it is essential to declare the data type of each variable so that the compiler can choose the proper algorithm to achieve the intended result. Type declarations also allow high-level language compilers to build in checks for validity. Such checks may generate error messages during compilation or during execution of the program. Languages with more restrictive rules for the manipulation of different data types, called strongly typed languages, allow more checking. For example, Pascal's strong typing prevents the contents of a variable of one type from being copied into a variable of a different type. C, FORTRAN, and COBOL allow this to be done in some cases but not in others.

The C data types integer, float, and char or their equivalents are now standard in most high-level languages. The values they may assume, and the operations that may be performed on variables of these types, are specified in each language.

1.7: Elements of Style

Individual writers, even whole magazines or journals, have their own writing style. Often a glimpse at a few paragraphs can give clues to the author or the type of publication. Still, there are general rules to follow for acceptable style; entire books have been devoted to this subject. Style will always be in style.

Dress, carriage, and response are parts of personal style, as are straightness of teeth, polished or scuffed shoes, and a fast or slow gait. Program style is determined by overall appearance as well as by details of implementation. We want to see the logical structure of a program clearly; too many details cloud the picture. Data abstraction and functional modularization are tools for sharpening the picture. Hiding information, so that only parts of a program that need it can see it, also provides better focus. People react differently to unexpected situations: some become erratic while others remain calm, going cheerfully about their business. Programs should react to unexpected data with aplomb, letting us know about errors, but getting on with the job when possible. Details of a program should not affect its logical structure, only its efficiency. Details can make an otherwise stylish program sloppy, slow, and in need of repair, or they can finish the design so it is elegant, sleek, and well-tuned. What we see isn't always what we get, but stylish programs suggest a better thought-out design and product.

Since good programs must be written with style, guidelines are offered here for this purpose. A good program is correct, easy to read, and changeable through a reasonable amount of effort. Whether or not it should be efficient depends on the application. In some cases an efficient program may be written with the same effort as an inefficient one. In other cases it will be necessary to put in more effort to achieve efficiency, so the additional programming may be done only if it will bring substantial (or required) savings in computation time.

1.7.1 Expressing Purpose

What's in a name? "A rose is a rose is a . . . ." "There's more form than substance." Perhaps these sentiments are true, but in a well-presented program there is a great deal in a name. One function is not another function, and understanding of substance is enhanced by form.

Indentation and proper spacing serve to delineate significant components of a program. Choosing names to convey and delineate purpose is also extremely important. Even without the descriptions given for the functions of Figure 1.3, the names of each function, variable, and constant almost provide, by themselves, enough information to see what each does. On the other hand, inappropriate names can be very misleading and distracting.

Reading identifiers to understand what a program does requires reading the entire program. This is too demanding. After all, when selecting a book, no one wants to be forced to read from cover to cover just to see what it is about. The same holds for a program, program documentation must be provided for this purpose. Good name selection is not enough. The main program and each of its functions must start with comments that allow users (including its author) to understand readily what it does and what its inputs, outputs, and parameters are. The main program's documentation then tells immediately what it does. No further reading is necessary unless someone wants to know how it is done. Similar comments can precede program segments that implement tasks of some complexity. To a great extent, the structure of a function, coupled with the documentation of each of its tasks, explains how the function does what it does. In this sense, structured programs (with proper comments) are self-documenting. Functional modularization and data abstraction help to elucidate structure by obscuring detail. Some programmers are free to develop their own style for delineation and documentation; for others it may be specified rigidly. The case study presented later gives a complete program with documentation as a guide for annotating your own programs. As a rule, complete documentation is not included in programs appearing in this book, since in effect the text itself serves as documentation, discussing completely each program presented. Still, programs will be annotated for clarity. Proper documentation is a prominent aspect of every good program and should become a habit of the programmer. Since the top-down approach requires a description for each task, the documentation for a task may be done as it is introduced, even before it is written.

1.7.2 Communication between Functions

High-level languages allow the definition of three kinds of variables: local, nonlocal and global. A C program consists of functions. A function of a program may reference

1.     Local variables, to which it alone has access

2.     Nonlocal variables, to which it and some other functions of the program have access

3.     Global variables, to which all functions of the program have access

A variable is referenced by a function in order to copy, modify, or simply look at its value.

High-level languages treat variables in differing ways. FORTRAN, for example, considers variables to be local to a function unless they appear as formal parameters of that function (in which case they are nonlocal). Variables stored in special common areas of memory are global. C determines whether a variable is local, nonlocal, or global by using block rules. In Figure 1.3 candidates is local to primes, factor is local to remove, and nextmultiple is local to delete, while n is nonlocal or global to each.

Each high-level language has its own rules to give one function access to variables of another. Merely using the same variable name in two functions will not normally allow one to reference a variable in another. For example, i in create and i in print (Figure 1.3) do not represent the same variable; each is local to its function. If this were not so, functions could not be written independently. Once a complex problem has been refined, different individuals or groups should be able to develop components independently. Otherwise, each time a group used a new variable name, it would have to check that no other group used the same name.

The issue here is really communication between functions of a program. Communication between functions--allowing one to reference variables also used by another--occurs in two basic ways: using nonlocal or global variables, or using formal parameters in defining functions.

When a function changes the value of a variable used by another, and the variable is not communicated (or passed) between them as a parameter, a side effect has occurred. Side effects make it extremely difficult to understand what functions do. It is easier to see what a function does when we can be sure that only the parameters and the local variables will be affected by the execution of the function. The communication structure of a program is clearer when information is communicated explicitly by the use of parameters. This shows "who is doing what to whom." For example, in delete (Figure 1.3), no side effects can occur, since the only nonlocal variables factor, n, and candidates all appear as parameters. Thus the only variables not local to delete that it can reference are factor, n, and candidates.

At one extreme, it is possible to make all variables global--every function may then reference any variable. Making all variables global requires extremely careful adherence to the use of variable names and increases the probability of side effects. COBOL programs are typically written this way. However, such programs use well-defined file structures, deal with high-volume input and output, and use relatively simple algorithms, so this method may not cause difficulty.

The other extreme is to use only parameters between functions. This may not always be possible. For example, storage considerations may intrude, as with arrays in Pascal.

Communication between functions by using parameters may be a one-way or a two-way street. In some high-level languages, like FORTRAN, any change in the value of a parameter in a called function is passed or reflected back to the calling function. Other languages, such as C, allow one component to communicate information to another yet never reflect changes made to the parameter's value. Upon return, the parameter's value is the same as prior to the call. Also, upon return from a call, different languages treat local variables of functions differently. Some preserve their values between calls, while others do not. In FORTRAN it is possible to have local variables retain their values between invocations of a function; in Pascal it is not. In C either case may apply. Values may be preserved or erased as desired.

1.7.3 Ensuring against Data Errors: Defensive Programming

Functions receive and output data. In primes, what happens if n is negative or so large that it exceeds the storage capacity allocated to the data structure implementing candidates? Think how upset a company's personnel would be if no one received paychecks because the input data for one employee was not valid. Or suppose each employee received an output check equal to the entire payroll. Defensive programming is ensuring that your programs guard against such outcomes by checking for invalid data, incoming or outgoing, and doing something reasonable when such data are discovered. What is reasonable can vary from signaling an error and stopping further processing to simply outputting an error message, ignoring the invalid data, and continuing processing. Sometimes the correct data may be inferred from the invalid data, but this approach must be taken with great caution. The important point is to ensure that errors are discovered. It is safer to have no output than output that is in error but thought to be correct. Invalid data is a very serious problem, regardless of whether it is due to design or to happenstance. Validation functions for input and output should be used, even though it is not always easy to see how to guard against all possible errors. This text does not have space to dwell on the issue of data validation, but it should always be in your mind.

1.7.4 Changing Programs

Consider a new problem: count and print the number of primes that are no greater than n. Instead of immediately attempting a solution, can you think of a program that already solves a related problem? You've already seen one: primes. Perhaps it can be changed to help with the solution. Looking at the refinement of primes (Figure 1.3), you can see that replacing function print by a count function that prints the proper count does the trick. The solution countprimes, using the changed primes and count, is as follows.

countprimes (n)

/* This function counts the prime

numbers between 2 and n

and prints the count.

*/

int n;

{

collection candidates;

create (n, candidates);

remove(n,candidates);

count(n,candidates);

}

count(n,c)

/* Prints the number of integers

in c between 2 and n.

*/

int n;

collection c;

{

int i,counter;

counter = 0;

for (i = 2; i<=n;i++)

if (belongs(i,c))

counter++;

printf ("\n The number of primes between 2 and %d is %d\n",

n, counter);

}

Solutions are rarely obtained so easily. Yet this example teaches us something. Programs that have been designed wisely, with an eye to generality, may often be used as tools in the solution of related problems. When beginning the development of a solution to a problem, see first if a program for a related problem is already known, and if so whether it can be adapted. This can save significant time, effort, and money compared to writing a new program or function.

It was easy to see what to change and how to change it in our example, because primes is functionally modular, with narrowly defined tasks and independent functions. Changes are rarely as straightforward as this; for contrast, consider changing the algorithm anotherprimes to a program for countprimes. In general, though, isolating functions that carry out a single narrowly defined task tends to generate lower-level functions that are clear, simple, and of moderate size. At higher levels, such single-task functions need not be simple. For example, a function to "Select the best next chess move" is such a function, but it is not simple. The lower the level of a function that needs changing, the less the effort required. However, the higher the level of the task for which one can adapt a known program, the greater the savings, since much refinement is avoided. In an ideal situation, changes require little effort and benefits are great.

Often a programmer's assignment or goal is to increase the efficiency of a program, and this will involve changing the way data are stored. If the program has been written using data abstractions, it is much easier to see exactly which functions need changing. For example, in primes, to change the implementation of candidates involves changing only its declaration and set, insert, omit, and belongs.

1.7.5 Verification and Debugging

Verifying a solution means checking that program segments and functions do their tasks correctly. It is difficult to verify programs, although some formal methods are known. This text will illustrate how a program can be verified informally, but rigorously, for appropriate examples. For most of the small examples, verification is left to the reader.

Although only proper verification can ensure the correctness of a program, debugging can help in locating mistakes. Your goal should be not to run or even code programs until you are sure the algorithm is correct. Even if you are sure of the algorithm, debugging tools should be employed. Then, when your program fails to work correctly (as you know by now it will), you can track down the cause without the help of a Holmes or Watson.

In addition to debugging aids provided by your computer system and compiler, good programming techniques facilitate debugging. For example, it is important to learn to display appropriate error messages and information. The top-down approach includes debugging individual functions or program segments as they are refined, so that debugging proceeds as the program is developed. Verifying, debugging, and documenting all require basic understanding of the program and much the same information. To a large extent, then, they should be done concurrently with program development. During development you have the concepts and information fresh in your mind.

The main program and the other functions may each be debugged independently, just as they may be verified and documented independently. This requires stubs and drivers.

A stub takes the place of a function. It should have the same name and parameters as the function and, when called, should output a message indicating it was called. For example,

remove (n,c)

/* This is a stub for remove */

int n;

collection c;

{

printf ("\n remove was called \n");

}

A driver plays the role of a function that calls another function, and is used to debug the called function. The driver should set up test values for the parameters used by the function it calls, output those values, call the function, and output the returned values. For example,

removedriver ()

/* This is a driver for remove */

{

collection candidates;

int n;

int sentinel = -1;

printf("\n enter a value for n between 2 and 27 \n");

scanf("%2d",&n);

while (n != sentinel)

{

printf("\n remove has been called with n = %d\n",n);

create(n,candidates);

print(n,candidates);

remove(n,candidates);

printf("\n remove returns n = %d ", n);

printf("and candidates: \n");

print(n,candidates);

printf("\n enter a new value for n or -1 to stop \n");

scanf("%2d",&n);

}

}

Using drivers and stubs allows any individual function to be tested and debugged even though the functions that call it and are called by it have not yet been written. Within the function, program segments that correspond to algorithmic tasks can be isolated. A statement can be inserted before and after each of these segments to output a message indicating that the segment was reached during execution, giving the values of variables it worked on, specifying that it was exited during execution, and giving the modified values of the variables.

In this way, you can see the progress of a program during its execution by studying the output of the drivers, stubs, and function being tested. With this approach, you never find yourself with a program that has not run correctly and for which you have no output, or output that gives no clue as to what went wrong or where. Suppose function M has just been refined and is to be tested and debugged. Its refinement may include calls to other functions not yet written or debugged. To test and debug M you must

1.     Write a driver to call M and feed it test data on which to work

2.     Replace each function of M's refinement by its own stub

For example, to test and debug remove (Figure 1.3) requires

1.     A driver to call remove and feed it test values for n and candidates

2.     Replacing nonprimes and delete by their individual stubs

This assumes that primes (which calls remove) and noprimes and delete (which remove calls) have not yet been verified or even written.

Instead, suppose primes has already been debugged. Primes, in conjunction with the driver used to debug it, could then be taken as the basis for remove's driver. Next, this driver plus remove could be used as the basis for delete's driver. This way of proceeding parallels the top-down development of the program. It amounts to inserting one new function at a time into an already debugged program.

To illustrate the proper way to use debugging statements, a complete listing and output is given for an execution of the case study program later in the chapter.

1.7.6 In a Nutshell

The discussion of good programming style may be summarized with the following guidelines.

1.     Take care to present programs well.

Don't choose a name without a purpose.

Don't leave a component's purpose unspecified.

Descriptions should be simple even if tasks are not.

2.     Communicate well.

Store data only where it is needed (use local variables).

Pass information explicitly when needed elsewhere (use parameters).

Avoid side effects.

3.     Use data abstraction.

This is the basic way to make programs independent of data implementation, and it makes implementation changes easier and more reliable.

4.     Functionally modularize programs.

Functions should execute a simple, narrowly defined task.

It should be possible to determine what a function does by looking only at it.

Changes to a function should not necessitate changes in other functions.

Prevent unreasonable input and output data by defensive programming.

5.     Use debugging aids.

Also verify your programs.

Of course, strict adherence to rules may not always be possible or even yield the best results. Knowing how or to what extent to apply programming rules comes only from experience in using them and in studying their effects.

1.8: Time and Storage Requirements

Sometimes the time of execution or the storage required by a program, or both, are limited. Analysis helps pinpoint parts of a program whose efficiency may be worth trying to improve, because they are executed frequently or require large amounts of storage. Such analysis can be very difficult, and simulation of the program may be needed for estimates. Simulation means that the program is executed repeatedly for appropriately chosen data, and the results are averaged or otherwise statistically analyzed. Sometimes it is possible to analyze how these requirements increase as the amount or size of the data increases, even though we cannot say what happens exactly for smaller amounts or sizes. This is called asymptotic analysis. When exact analysis cannot be done, we settle for bounds on the time and storage required.

1.8.1 Analyzing the Prime Example

The nonmathematically inclined reader may want to skip the following analysis of primes (Figure 1.3), which demonstrates how to go about informally analyzing a program and shows how trade-offs arise. Even such simple programs do not necessarily have simple analyses.

When a statement is said to take constant time, it means that the statement is always executed in the same amount of time, whatever the current values of the variables it processes. Clearly, the execution time of primes is given by the sum of the execution times of its three components. In what follows it is assumed that insert, belongs, and omit all take constant time (notice that for the array implementation for candidates (Section 1.3.2) this will be true, but set takes time proportional to n).

Create executes set once and insert exactly n - 1 times. Similarly, print executes its if statement n - 1 times. The amount of time the if statement takes depends on whether belongs is true or false but requires, at most, the time for belongs plus the time for printf.

Remove is more complex. Its initial assignment statement takes constant time, to which must be added the loop time. The loop is executed   [image]  times, so the loop condition is evaluated   [image]  times. Assuming, for simplicity, that sqrt (n) takes constant time C, then   [image]  is the total time for the evaluation of the condition. If the loop task took constant time, then the total loop time would be the product of this constant and   [image]. Unfortunately delete's time varies, depending on the value of factor. Nonetheless, the total time for delete can still be derived.

Delete's own loop is executed for each multiple of factor between 2 x factor and m x factor, where m is the largest number of times that factor divides into n. m is the integer part of n/factor denoted by   [image] n/factor  [image]. Hence delete's loop is executed   [image] n/factor  [image]  times for this value of factor. Since factor takes on values   [image], the total number of delete's loop executions is

or

But 1/2 + 1/3 + . . . + 1/n is roughly the natural logarithm of n, denoted by ln n.Thus the entire time for the loop of delete is a constant (for omit and the assignment statement) times   [image]. The condition of this loop contributes roughly its own constant times   [image]. The initialization of nextmultiple in delete contributes its constant times   [image] .

Adding everything, we get a good execution time estimate of

where C1, C2, C3, and C4 come from the times for condition evaluation, assignment statement execution, and set, insert, omit, and belongs.

For large n, this expression is dominated by the term C4n1n n. In fact, for large enough values of n, the expression will be no greater than C n1n n for an appropriate value of C. We describe this concisely by saying the expression is O(n1n n), read "order n1n n"; this notation will be used throughout the text.

By the storage requirements of primes is meant the total storage required for its data structures. Except for a few variables, the storage is required primarily for candidates and will be determined by its implementation. The analysis reveals that the bulk of the execution time for primes is spent executing the loop in delete. Consequently, to improve its time we speed up the time of the loop itself, or reduce the number of times it needs to be executed. For the array implementation, the storage required will be proportional to n, and this is where the bulk of the storage required is needed.

Primes can be improved in a number of obvious ways. Except for 2, no primes are even. Hence all other even numbers can be eliminated from consideration. This means that factor need never take on even values, so it can be increased by 2 instead of 1 in remove and can also be initialized to 3. Delete can be changed so that nextmultiple takes on only odd multiples of factor. With a little more work, candidates can be cut to roughly half its size by eliminating even integers. This will necessitate changes to set as well as changes to the for loops in create and print, but it reduces the storage needed by about one-half and the time by more than one-half.

Perhaps a less obvious improvement is to modify the increase of factor in remove so that, instead of increasing by 1 (or 2), factor increases to the value of the next prime in candidates (see Exercise 8). To realize this saving fully, it is necessary to use a new data structure, the list or the balanced binary tree, for the collection. Lists are discussed in Chapter 3, balanced binary trees in Chapters 7 and 9. These would save not only time but storage as well. Borrowing a result from mathematics about the distribution of prime numbers, we can state that any version of primes that stores all the primes from 2 to n in a collection will need n/ln n storage locations, since there are about that many such primes. Thus we will run out of storage before we run out of time!

Primes may be written so that its execution time increases but its storage needs decrease, or vice versa. Thus time is traded for storage, as is often done. Primes represents one extreme where all the primes are saved. A version of primes in which no primes are saved is at the other extreme (see Exercise 10). This version would require significantly more time.

1.8.2 Limits on Time and Storage

Programs carry out certain basic operations. For example, primes executes assignment statements, evaluates conditions, and executes set, insert, omit, and belongs. In fact, its total execution time equals the number of times each of the operations is executed multiplied by the time taken by that operation. Determining the number of basic operations performed by a program to solve a problem allows evaluation of its practicality. For illustration, suppose that 1,000,000 basic operations can be carried out in 1 second by a computer. Then the number of basic operations that it can carry out in 1 year is

1,000,000 basic operations/second X 60 seconds/minute X 60 minutes/hour X 24 hours/day X 365 days/year

This is fewer than 1013 basic operations/year, and this is assuming that the computer is available for the exclusive use of the program and that it will never fail during the year. Thus 1013 can be taken as a benchmark figure--any program that requires a number of operations on this order to solve a problem is not feasible!

How much storage is available is determined by your computer system. The amount needed is surely not feasible when your program aborts during execution with a message indicating insufficient storage, or when the compiler tells you there is insufficient storage. If you run out of time or storage, you must then abandon the problem (and perhaps your job), find a more efficient program, or trade time for storage.

1.9: Case Study: Bowling Scores

This section illustrates the way a flowchart and a program are developed for a problem connected with the game of bowling. The way in which the spirit of the structured programming philosophy is followed is more important than the particular development.

Bowling, as you may know, consists of ten frames. In each frame, there is a chance to knock down ten pins by throwing one or two balls down the bowling alley. To start, you roll the first ball of a given frame. If all ten pins are knocked down, a strike is achieved for that frame, and you earn a frame score of 10 plus the total number of pins knocked down on the next two rolls. If any pins remain standing, you roll a second ball for the frame. After this roll, if all pins are down, you have a spare, and a frame score of 10 plus the number of pins knocked down on the next roll. If any pins remain standing after the second roll, then the result is an open frame, and the frame score obtained is given by the total number of pins knocked down on the two rolls of the frame. A strike or spare occurring on the last (the tenth), results in two or one additional rolls, respectively. The total score for a game is the sum of all the individual frame scores.

Getting twelve strikes in a row results in the maximum attainable score of 300. In this case, twelve rolls are made. The least number of rolls possible is eleven and the greatest is twenty-one. In fact there is a correlation between the number of rolls in a game and the game score. Many rolls correspond to low scores and few rolls to high scores. The better a bowler you are, the fewer rolls you get!

Example 1.1

The problem is to print the total score for each of a series of games, as well as the minimum, average, and maximum scores. Assume the input is given as a series of roll scores for each game and includes at least one game.

1.9.1 Algorithm

An initial flowchart might be as shown in Figure 1.5(a). If the three tasks can be carried out correctly, in the sequence indicated, then the problem is solved. The flowchart breaks down the original problem into three smaller problems. Obtaining their solutions and sequencing them correctly provides the solution to the original problem. This is the top-down approach to problem solving. Next, it is applied to each of the three problems.

(a) Initial Flowchart

(b) The Refined Flowchart

(c) A Loop for Processing the Games

Figure 1.5 Flowcharts of the Bowling Scores Algorithm

Problem 1 is transparent enough. Problem 3 is clearly solved by the expanded or refined flowchart shown in Figure 1.5(b).

Problem 2 needs to be refined. It seems clear that the basic structure for its solution is a loop, within which each game is processed. When all games have been processed, the loop should be exited. (See Figure 1.5(c).)

The new problem introduced as task 4 must be solved next. The input for this new problem will be the roll scores that must be processed for the new game. A basic looping structure can accomplish the task. The loop task will be executed ten times, once for each frame. Each execution calculates the score for the current frame and updates the game score and a frame counter, frame. Prior to entering the loop, f rame must be initialized to 1, and gamescore to 0. Other initialization may be required. After the loop is exited, the game score must be printed, and proper updating must occur. This refinement is shown in Figure 1.6.

Suppose that each time the programmer expands a task, the expanded version of the task does in fact solve the problem introduced by that task. Then the programmer knows that, whatever the current structured flowchart, it must represent a solution to the original problem as long as each of its tasks can be carried out correctly. The detailed structured flowchart of Figure 1.7 is thus a solution to the problem, as long as each of its tasks can be carried out correctly. It is detailed enough so that we know how to carry out (or program) each of its tasks except for 5, 6, 7, and 8. However, if the high-level language chosen for the program allows direct implementation of those tasks, then no more detail is needed to complete the solution.

Figure 1.6 Refinement of Task 4

Figure 1.7 Refinement of Initial Flowchart

Instead of increasing the complexity of the flowchart of Figure 1.7, tasks 5-8 can be functionally modularized.

Functions for tasks 7 and 8 appear in Figures 1.8 and 1.9. The solution to the original problem will then consist of Figure 1.7 plus the functions for tasks 5-8. This illustrates the application of structured programming to a relatively simple problem. Note again the importance of choosing variable names that convey information about the meaning and use of the variables. Good names make the algorithm clearer. Note that C compilers normally treat names as distinct only when they differ in the first eight characters, so some care must be taken. In this text, however, this constraint is ignored for clarity.

Figure 1.8 Refinement of Task 7

Figure 1.9 Refinement of Task 8

When task 7 is entered, the variables rollscore and r1 must contain, respectively, the first roll score of the current frame and the next roll score. Consequently, before entering task 6, the first two roll scores of the game must be read into rollscore and r1. This is the "other initialization" required of task 5.

The conditions "strike" and "spare" (Figure 1.8) are actually implemented as "rollscore equals 10" and "(rollscore + r1) equals 10," respectively. To emphasize the fact that complexity in a flowchart should be limited, Figure 1.10 is the complete detailed flowchart for the solution as it could appear without functional modularization. Descriptions do not appear; the chart simply indicates the structure of the solution. The flowchart can be derived by adding the detailed expansions of tasks 5-8 to Figure 1.7.

Figure 1.10 Too Detailed a Refinement

1.9.2 Program

The program now follows directly from the flowcharts. Notice that its structure parallels the flowcharts', that we have attempted to minimize side effects by passing information between components explicitly as parameters, and that information needed only by a component is kept local to that component.

#include <stdio.h>

#define TRUE 1

#define FALSE 0

main()

/*

Bowling Scores calculates and prints the game scores

for a series of bowling games. It also prints the

number of games bowled, the minimum, average, and

maximum of all game scores. The input consists, for

each game, of a sequence of roll scores. Each roll

represents the number of pins knocked down on the

corresponding roll. Its input must consist of at least

one game.

Main Variables

average - average of all game scores

gamescore - score for the current game

max - current maximum game score

min - current minimum game score

number - current number of games played

total - accumulated total of all game scores

*/

{

int gamescore;

int number = 0, max = 0, min = 300;

int newgame = TRUE;

float average;

while (newgame)

{

gamescore = gamecalc(gamescore);

update(gamescore,&number,&min,&max,&total);

printf("If you want to bowl another game enter a 1\n");

printf("If you wish to stop enter a 0\n");

scanf("%d",&newgame);

}

average = ((float)total)/number;

printf("Minimum game %d\n",min);

printf("Average game %f\n",average);

printf("Maximum game %d\n",max);

printf("Number of games %d\n",number);

}

gamecalc (gamescore)

/* calculates gamescore for the current game.

Main variables

frame - the current frame

gamescore - score for the current game

rollscore - first roll score of the current frame

r1 - succeeding roll score after rollscore

*/

int gamescore;

{

int frame, rollscore, r1;

gamescore = 0;

frame = 1;

printf("Please enter the number of pins down on this roll\n");

scanf("%d",&rollscore);

printf("Please enter the number of pins down on this roll\n");

scanf("%d",&r1);

while (frame <= 10)

framecalc(&gamescore,&frame,&rollscore,&r1);

return(gamescore);

}

framecalc(pgamescore,pframe,prollscore,pr1)

/* Calculates the framescore for the current game,

adds it to gamescore, increases frame by 1 and

updates rollscore and r1.

Main variables

pframe - pointer to the current frame

pgamescore - pointer to the score for the current game

prollscore - pointer to first rollscore of current frame

pr1 - pointer to next rollscore of current frame

framescore - score for current frame

r2 - rollscore after r1

*/

int *pgamescore,*pframe,*prollscore,*pr1;

{

int framescore, r2;

if (*prollscore == 10) /* it is a strike */

{

printf("Please enter the number of pins down on this");

printf("roll\n");

scanf("%d",&r2);

framescore = 10 + *pr1 + r2;

*prollscore = *pr1;

*pr1 = r2;

}

else if ((*prollscore + *pr1) == 10) /* it is a spare */

{

printf("Please enter the number of pins down on this");

printf(" roll\n");

scanf("%d",&r2);

framescore = 10 + r2;

*prollscore = r2;

if (*pframe < 10)

{

printf("Please enter the number of pins down");

printf("on this roll\n");

scanf("%d",pr1);

}

}

else /* it is an open frame */

{

framescore = *prollscore + *pr1;

if (*pframe < 10)

{

printf("Please enter the number of pins");

printf("down on this roll\n");

scanf("%d",prollscore);

printf("Please enter the number of pins");

printf("down on this roll\n");

scanf("%d",pr1);

}

}

*pgamescore = *pgamescore + framescore;

*pframe = *pframe + 1;

}

update(gamescore,pnumber,pmin,pmax,ptotal)

/* This prints gamescore, and updates number, min, max,

and total to reflect gamescore.

Main variables

gamescore - current game score

pnumber - pointer to current number of games

pmin - pointer to current minimum game score

pmax - pointer to current maximum game score

ptotal - pointer to total of all game scores

*/

int gamescore, *pnumber, *pmin, *pmax;

int *ptotal;

{

printf("gamescore \n");

printf("%d\n",gamescore);

*pnumber = *pnumber + 1;

*ptotal = *ptotal + gamescore;

if (gamescore < *pmin)

*pmin = gamescore;

if (gamescore > *pmax)

*pmax = gamescore;

}

1.9.3 Debugging

The following is a complete listing and output of a version of the Bowling Scores program that contains statements used as debugging aids (the boxed lines). This listing demonstrates how to use such statements and what they can tell us as we peruse the output. The roll scores of the two games that served as input are

game 1--5, 4, 6, 4, 2, 4, 1, 4, 3, 4, 8, 1, 9, 0, 10, 10, 9, 0

game 2--9, 1, 10, 7, 3, 9, 0, 10, 10, 5, 4, 6, 4, 10, 10, 10, 8

The prompts for input are in the program. They have been omitted from the output.

#include <stdio.h>

#define TRUE 1

#define FALSE 0

main()

/* Bowling Scores calculates and prints the game scores

for a series of bowling games. It also prints the

number of games bowled, the minimum, average, and

maximum of all game scores. The input consists, for

each game, of a sequence of roll scores. Each roll

represents the number of pins knocked down on the

corresponding roll. Its input must consist of at least

one game.

Main Variables

average - average of all game scores

gamescore - score for the current game

max - current maximum game score

min - current minimum game score

number - current number of games played

total - accumulated total of all game scores

*/

{

int gamescore;

int number = 0, max = 0, total = 0, min = 300;

int newgame = TRUE;

float average;

while (newgame)

{

printf("\n CALLING GAMECALC FOR GAME #%2d",number+1);

printf("\n-----------------------------\n");

gamescore = gamecalc(gamescore);

printf("-===>RETURNING FROM GAMECALC WITH GAMESCORE");

printf(" =%3d FOR GAME #%2d\n",gamescore, number+1);

printf("CALLING UPDATE WITH; MIN TOTAL MAX NUMBER\n");

printf("%26d%5d%5d%5d\n",min,total,max,number);

update(gamescore,&number,&min,&max,&total);

printf("If you want to bowl another game enter a 1 \n");

printf("If you wish to stop enter a 0 \n");

scanf("%d",&newgame);

printf("-===>RETURNING FROM UPDATE: MIN TOTAL MAX");

printf("NUMBER\n");

printf("%33d%6d%6d%5d\n",min,total,max,number);

}

average = ((float)total)\number;

printf("FINAL RESULTS FOR ALL GAMES FOLLON:\n");

printf("***********************************\n");

printf("  ACCUMULATED TOTAL = %5d\n\n",total);

printf("    MINIMUM GAME = %5d\n",min);

printf("    AVERAGE GAME = %10.4f\n",average);

printf("    MAXIMUM GAME = %5d\n\n",max);

printf("NUMBER OF GAMES PLAYED = %3d\n",number);

}

gamecalc(gamescore)

/* calculates gamescore for the current game.

Main variables

frame - the current frame

gamescore - score for the current game

rollscore - first roll score of the current frame

r1 - succeeding roll score after rollscore

*/

int gamescore;

{

int frame, rollscore, r1;

gamescore = 0;

frame = 1;

printf("Please enter the number of pins down on this roll\n");

scanf("%d",&rollscore);

printf("Please enter the number of pins down on this roll\n");

scanf("%d",&r1);

while (frame <= 10)

{

printf(" IN GAMECALC, GOING TO FRAMECALC WITH: GAMESCORE");

printf(" FRAME ROLLSCORE R1\n");

printf("%49d%9d%9d%7d\n",gamescore,frame,rollscore, r1);

framecalc(&gamescore,&frame,&rollscore,&r1);

printf(" IN GAMECALC, RETURNING FROM FRAMECALC WITH:");

printf("GAMESCORE FRAME ROLLSCORE R1\n");

printf("%53d%9d%9d%7d\n\n",gamescore,frame,rollscore,r1);

}

return(gamescore);

}

framecalc(pgamescore,pframe,prollscore,pr1)

/* Calculates the framescore for the current game,

adds it to gamescore, increases frame by 1 and

updates rollscore and r1.

Main variables

pframe - pointer to the current frame

pgamescore - pointer to the score for the current game

prollscore - pointer to first rollscore of current frame

pr1 - pointer to next rollscore of current frame

framescore - score for current frame

r2 - rollscore after r1

*/

int *pgamescore,*pframe,*prollscore,*pr1;

{

int framescore, r2;

printf(" STARTING FRAMECALC WITH: GAMESCORE FRAME");

printf(" ROLLSCORE R1\n");

printf("%36d%9d%9d%7d\n",*pgamescore,*pframe,*prollscore,*pr1);

if(*prollscore == 10) /* it is a strike */

{

printf("Please enter the number of pins down on this");

printf(" roll\n");

scanf("%d",&r2);

framescore = 10 + *pr1 + r2;

*prollscore = *pr1;

*pr1 = r2;

printf("AFTER STRIKE, FRAMESCORE = %2d\n", framescore);

}

else if ((*prollscore + *pr1) == 10) /* it is a spare */

{

printf("Please enter the number of pins down on this");

printf(" roll\n");

scanf("%d",&r2);

framescore = 10 + r2;

printf("AFTER SPARE, FRAMESCORE= %2d\n", framescore);

*prollscore = r2;

if (*pframe < 10)

{

printf("Please enter the number of pins down");

printf(" on this roll\n");

scanf("%d",pr1);

}

}

else /* it is an open frame */

{

framescore = *prollscore + *pr1;

printf("OPEN FRAME, FRAMESCORE %2d\n",framescore);

if (*pframe < 10)

{

printf("Please enter the number of pins");

printf(" down on this roll\n");

scanf("%d",prollscore);

printf("Please enter the number of pins");

printf(" down on this roll\n");

scanf("%d",pr1);

}

}

*pgamescore = *pgamescore + framescore;

*pframe = *pframe + 1;

printf(" LEAVING FRAMECALC WITH: GAMESCORE FRAME");

printf(" ROLLSCORE R1\n");

printf("%35d%9d%9d%7d\n\n",*pgamescore,*pframe,*prollscore,*pr1);

}

update(gamescore,pnumber,pmin,pmax,ptotal)

/* This prints gamescore, and updates number, min, max

and total to reflect gamescore.

Main variables

gamescore - current game score

pnumber - pointer to current number of games

pmin - pointer to current minimum game score

pmax - pointer to current maximum game score

ptotal - pointer to total of all game scores

*/

int gamescore, *pnumber, *pmin, *pmax;

int *ptotal;

{

printf("\n BEGINNING TO UPDATE WITH: GAMESCORE");

printf(" NUMBER MIN MAX TOTAL\n");

printf("%37d%8d%8d%4d%6d\n",

gamescore, *pnumber, *pmin, *pmax, *ptotal);

*pnumber = *pnumber + 1;

*ptotal = *ptotal + gamescore;

if (gamescore < *pmin)

*pmin = gamescore;

if (gamescore > *pmax)

*pmax = gamescore;

printf(" LEAVING UPDATE WITH: GAMESCORE NUMBER");

printf(" MIN MAX TOTAL\n");

printf("%35d%8d%8d%5d%6d\n\n",

gamescore,*pnumber,*pmin,*pmax,*ptotal);

}

The output of the program is as follows.

CALLING GAMECALC FOR GAME # 1

-----------------------------

  IN GAMECALC, GOING TO FRAMECALC WITH:  GAMESCORE  FRAME  ROLLSCORE  R1

                                             0        1        5       4

  STARTING FRAMECALC WITH:  GAMESCORE  FRAME  ROLLSCORE  R1

                                0        1        5       4

OPEN FRAME, FRAMESCORE = 9

  LEAVING FRAMECALC WITH:  GAMESCORE  FRAME  ROLLSCORE  R1

                               9         2       6       4

  IN GAMECALC, RETURNING FROM FRAMECALC WITH:  GAMESCORE  FRAME  ROLLSCORE  R1

                                                   9        2        6       4

  IN GAMECALC, GOING TO FRAMECALC WITH:  GAMESCORE  FRAME  ROLLSCORE  R1

                                             9        2        6       4

  STARTING FRAMECALC WITH:  GAMESCORE  FRAME  ROLLSCORE  R1

                                9        2        6       4

AFTER SPARE, FRAMESCORE = 12

  LEAVING FRAMECALC WITH:  GAMESCORE  FRAME  ROLLSCORE  R1

                               21       3        2       4

  IN GAMECALC, RETURNING FROM FRAMECALC WITH:  GAMESCORE  FRAME  ROLLSCORE  R1

                                                  21        3        2       4

  IN GAMECALC, GOING TO FRAMECALC WITH:  GAMESCORE  FRAME  ROLLSCORE  R1

                                             21       3        2       4

  STARTING FRAMECALC WITH:  GAMESCORE  FRAME  ROLLSCORE  R1

                                21       3        2       4

OPEN FRAME, FRAMESCORE = 6

  LEAVING FRAMECALC WITH:  GAMESCORE  FRAME  ROLLSCORE  R1

                               27       4        1       4

  IN GAMECALC, RETURNING FROM FRAMECALC WITH:  GAMESCORE  FRAME  ROLLSCORE  R1

                                                   27       4        1       4

  IN GAMECALC, GOING TO FRAMECALC WITH:  GAMESCORE  FRAME  ROLLSCORE  R1

                                             27       4        1       4

  STARTING FRAMECALC WITH:  GAMESCORE  FRAME  ROLLSCORE  R1

                                27       4        1       4

OPEN FRAME, FRAMESCORE = 5

  LEAVING FRAMECALC WITH:  GAMESCORE  FRAME  ROLLSCORE  R1

                               32       5        3       4

  IN GAMECALC, RETURNING FROM FRAMECALC WITH:  GAMESCORE  FRAME  ROLLSCORE  R1

                                                   32       5        3       4

  IN GAMECALC, GOING TO FRAMECALC WITH:  GAMESCORE  FRAME  ROLLSCORE  R1

                                             32       5        3       4

  STARTING FRAMECALC WITH:  GAMESCORE  FRAME  ROLLSCORE  R1

                                32       5        3       4

OPEN FRAME, FRAMESCORE = 7

  LEAVING FRAMECALC WITH:    GAMESCORE  FRAME  ROLLSCORE  R1

                                 39       6        8       1

  IN GAMECALC, RETURNING FROM FRAMECALC WITH:  GAMESCORE  FRAME  ROLLSCORE  R1

                                                   39       6        8       1

  IN GAMECALC, GOING TO FRAMECALC WITH:  GAMESCORE  FRAME  ROLLSCORE  R1

                                             39       6        8       1

  STARTING FRAMECALC WITH:  GAMESCORE  FRAME  ROLLSCORE  R1

                                39       6        8       1

OPEN FRAME, FRAMESCORE = 9

  LEAVING FRAMECALC WITH:    GAMESCORE  FRAME  ROLLSCORE  R1

                                 48       7        9       0

  IN GAMECALC, RETURNING FROM FRAMECALC WITH:  GAMESCORE  FRAME  ROLLSCORE  R1

                                                   48       7        9       0

  IN GAMECALC, GOING TO FRAMECALC WITH:  GAMESCORE  FRAME  ROLLSCORE  R1

                                            48        7        9       0

  STARTING FRAMECALC WITH:  GAMESCORE  FRAME  ROLLSCORE  R1

                                48       7        9       0

OPEN FRAME, FRAMESCORE = 9

  LEAVING FRAMECALS WITH:  GAMESCORE  FRAME  ROLLSCORE  R1

                              57        8        10     10

  IN GAMECALC, RETURNING FROM FRAMECAL WITH:  GAMESCORE  FRAME  ROLLSCORE  R1

                                                 57        8        10     10

  IN GAMECALC, GOING TO FRAMECALC WITH:  GAMESCORE  FRAME  ROLLSCORE  R1

                                             57       8        10     10

  STARTING FRAMECALC WITH:  GAMESCORE  FRAME  ROLLSCORE  R1

                               57        8        10     10

AFTER STRIKE, FRAMESCORE = 29

  LEAVING FRAMECALC WITH:  GAMESCORE  FRAME  ROLLSCORE  R1

                              86        9        10      9

  IN GAMECALC, RETURNING FROM FRAMECALC WITH:  GAMESCORE  FRAME  ROLLSCORE  R1

                                                  86        9        10      9

  IN GAMECALC, GOING TO FRAMECALC WITH:  GAMESCORE  FRAME  ROLLSCORE  R1

                                            86        9        10      9

  STARTING FRAMECALC WITH:  GAMESCORE  FRAME  ROLLSCORE  R1

                               86        9        10      9

AFTER STRIKE, FRAMESCORE = 19

  LEAVING FRAMECALC WITH:  GAMESCORE  FRAME  ROLLSCORE  R1

                              105       10       9       0

  IN GAMECALC, RETURNING FROM FRAMECALC WITH:  GAMESCORE  FRAME  ROLLSCORE  R1

                                                   105      10       9       0

  IN GAMECALC, GOING TO FRAMECALC WITH:  GAMESCORE  FRAME  ROLLSCORE  R1

                                            105       10       9       0

  STARTING FRAMECALC WITH:  GAMESCORE  FRAME  ROLLSCORE  R1

                               105       10       9       0

OPEN FRAME, FRAMESCORE = 9

  LEAVING FRAMECALC WITH:  GAMESCORE  FRAME  ROLLSCORE  R1

                              114       11       9       0

  IN GAMECALC, RETURNING FROM FRAMECALC WITH:  GAMESCORE  FRAME  ROLLSCORE  R1

                                                  114      11        9       0

- = = = > RETURNING FROM GAMECALC WITH GAMESCORE = 114 FOR GAME # 1

CALLING UPDATE WITH;  MIN  TOTAL  MAX  NUMBER

                      300    0     0     0

  BEGINNING TO UPDATE WITH:  GAMESCORE  NUMBER  MIN  MAX  TOTAL

                                 114      0     300   0     0

    LEAVING UPDATE WITH:  GAMESCORE  NUMBER  MIN  MAX  TOTAL

                             114        1    114  114   114

- = = = > RETURNING FROM UPDATE:  MIN  TOTAL  MAX  NUMBER

                                  114   114   114     1

CALLING GAMECALC FOR GAME # 2

-----------------------------

  IN GAMECALC, GOING TO FRAMECALC WITH:  GAMESCORE  FRAME  ROLLSCORE  R1

                                             0        1        9       1

  STARTING FRAMECALC WITH:  GAMESCORE  FRAME  ROLLSCORE  R1

                                0        1        9       1

AFTER SPARE, FRAMESCORE = 20

  LEAVING FRAMECALC WITH:  GAMESCORE  FRAME  ROLLSCORE  R1

                               20       2        10      7

  IN GAMECALC, RETURNING FROM FRAMECALC WITH:  GAMESCORE  FRAME  ROLLSCORE  R1

                                                   20       2        10      7

  IN GAMECALC, GOING TO FRAMECALC WITH:  GAMESCORE  FRAME  ROLLSCORE  R1

                                             20       2        10      7

  STARTING FRAMECALC WITH:  GAMESCORE  FRAME  ROLLSCORE  R1

                                20       2        10      7

AFTER STRIKE, FRAMESCORE = 20

  LEAVING FRAMECAL WITH:  GAMESCORE  FRAME  ROOLSCORE  R1

                              40       3        7       3

  IN GAMECALC, RETURNING FROM FRAMECALC WITH:  GAMESCORE  FRAME  ROLLSCORE  R1

                                                   40       3        7       3

  IN GAMECALC, GOING TO FRAMECALC WITH:  GAMESCORE  FRAME  ROLLSCORE  R1

                                             40       3        7       3

  STARTING FRAMECALC WITH:  GAMESCORE  FRAME  ROLLSCORE  R1

                                40       3        7       3

AFTER SPARE, FRAMESCORE = 19

  LEAVING FRAMECALC WITH:  GAMESCORE  FRAME  ROLLSCORE  R1

                               59       4        9       0

  IN GAMECALC, RETURNING FROM FRAMECALC WITH:  GAMESCORE  FRAME  ROLLSCORE  R1

                                                   59       4        9       0

  IN GAMECALC, GOING TO FRAMECALC WITH:  GAMESCORE  FRAME  ROLLSCORE  R1

                                             59       4        9       0

  STARTING FRAMECALS WITH:  GAMESCORE  FRAME  ROLLSCORE  R1

                                59       4        9       0

OPEN FRAME, FRAMESCORE = 9

  LEAVING FRAMECALC WITH:  GAMESCORE  FRAME  ROLLSCORE  R1

                               68       5        10     10

  IN GAMECALC, RETURNING FROM FRAMECALC WITH:  GAMESCORE  FRAME  ROLLSCORE  R1

                                                   68       5        10     10

  IN GAMECALC, GOING TO FRAMECALC WITH:  GAMESCORE  FRAME  ROLLSCORE  R1

                                             68       5           10     10

  STARTING FRAMECALC WITH:  GAMESCORE  FRAME  ROLLSCORE  R1

                               68        5        10     10

AFTER STRIKE, FRAMESCORE = 25

  LEAVING FRAMECALC WITH:  GAMESCORE  FRAME  ROLLSCORE  R1

                              93        6        10      5

  IN GAMECALC, RETURNING FROM FRAMECALC WITH:  GAMESCORE  FRAME  ROLLSCORE  R1

                                                  93        6        10      5

  IN GAMECALC, GOING TO FRAMECALC WITH:  GAMESCORE  FRAME  ROLLSCORE  R1

                                            93        6        10      5

  STARTING FRAMECALC WITH:  GAMESCORE  FRAME  ROLLSCORE  R1

                                93       6        10      5

AFTER STRIKE, FRAMESCORE = 19

  LEAVING FRAMECALC WITH:  GAMESCORE  FRAME  ROLLSCORE  R1

                              112       7        5       4

  IN GAMECALC, RETURNING FROM FRAMECALC WITH:  GAMESCORE  FRAME  ROLLSCORE  R1

                                                  112       7        5       4

  IN GAMCALC, RETURNING FROM FRAMECALC WITH:  GAMESCORE  FRAME  ROLLSCORE  R1

                                                 112       7        5       4

  STARTING FRAMECALC WITH:  GAMESCORE  FRAME  ROLLSCORE  R1

                               112       7        5       4

OPEN FRAME, FRAMESCORE = 9

  LEAVING FRAMECALC WITH:  GAMESCORE  FRAME  ROLLSCORE  R1

                              121       8        6       4

  IN GAMECALC, RETURNING FROM FRAMECALC WITH:  GAMESCORE  FRAME  ROLLSCORE  R1

                                                  121       8        6       4

  IN GAMECALC, GOING TO FRAMECALC WITH:  GAMESCORE  FRAME  ROLLSCORE  R1

                                            121       8        6       4

  STARTING FRAMECALC WITH:  GAMESCORE  FRAME  ROLLSCORE  R1

                               121        8       6       4

AFTER SPARE, FRAMESCORE = 20

  LEAVING FRAMECALC WITH:  GAMESCORE  FRAME  ROLLSCORE  R1

                              141       9        10     10

  IN GARAMECALC, RETURNING FROM FRAMECALC WITH: GAMESCORE FRAME ROLLSCORE R1

                                                   141      9       10    10

  IN GAMECALC, GOING TO FRAMECALC WITH:  GAMESCORE  FRAME  ROLLSCORE  R1

                                            141       9        10     10

  STARTING FRAMECALC WITH:  GAMESCORE  FRAME  ROLLSCORE  R1

                               141       9        10     10

AFTER STRIKE, FRAMESCORE = 30

  LEAVING FRAMECALC WITH:  GAMESCORE  FRAME  ROLLSCORE  R1

                              171       10       10     10

  IN GAMECALC, RETURNING FROM FRAMECALC WITH:  GAMESCORE  FRAME  ROLLSCORE  R1

                                                  171       10       10     10

  IN GAMECALC, GOING TO FRAMECALC WITH:  GAMESCORE  FRAME  ROLLSCORE  R1

                                            171       10       10     10

  STARTING FRAMECALC WITH:  GAMESCORE  FRAME  ROLLSCORE  R1

                               171       10       10     10

AFTER STRIKE, FRAMESCORE = 28

  LEAVING FRAMECALC WITH:  GAMESCORE  FRAME  ROLLSCORE  R1

                              199      11        10      8

  IN GAMECALC, RETURNING FROM FRAMECALC WITH:  GAMESCORE  FRAME  ROLLSCORE  R1

                                                  199      11        10      8

- = = = > RETURNTING FROM GAMECALC WITH GAMESCORE = 199 FOR GAME # 2

CALLING UPDATE WITH:  MIN  TOTAL  MAX  NUMBER

                      114   114   114    1

  BEGINNING TO UPDATE WITH:  GAMESCORE  NUMBER  MIN  MAX  TOTAL

                                 199      1     114  114   114

    LEAVING UPDATE WITH:  GAMESCORE  NUMBER  MIN  MAX  TOTAL

                              199      2     114  199   313

- = = = > RETURNING FROM UPDATE:  MIN  TOTAL  MAX  NUMBER

                                  114   313   199    2

FINAL RESULTS FOR ALL GAMES FOLLOW:

***********************************

     ACCUMULATED TOTAL =  313

          MINIMUM GAME =  114

         AVERAGE GAME  =  156.5000

          MAXIMUM GAME =  199

NUMBER OF GAMES PLAYED =  2

1.10: Summary

Structure is important for algorithms, programs, and data. Structured programming is a philosophy as well as a defined methodology to guide the construction of algorithms and programs. It involves a top-down approach: starting with a given problem, then using stepwise refinement, and breaking problems into smaller component problems that then may be solved independently and subsequently combined. The overall goal is to write programs that are easily understood, debugged, and changed. Data abstraction and functional modularity are important elements of programming style to be incorporated into this approach. Algorithm and program structure have been stressed in this chapter and will continue to be emphasized throughout the book, as will the equally important choice of data structures. The next chapter discusses record and array information and pointers.

Exercises

1.     In Figure 1.1 suppose there are n = 100 houses and you must visit houses 3, 10, 50, 7, and 82 (in this order). How much distance must you travel in each town?

2.     Why do no more nonprimes remain in candidates when factor >   [image] ?

3.     Explain the difference between a function and program segment implementation of a refinement.

4.     a.     What determines the largest value of n for which each of the two array implementations for candidates will work?

b.     Discuss any significant differences between set, insert, omit, and belongs for the two implementations of candidates.

c.     Modify the second implementation so that it works for larger n--say 20 times its current limit.

d.     Write a function copy with parameters c1 and c2. It must treat them as data abstractions with the operations set, insert, omit, and belongs and is to return with c2's contents identical to c1's and c1's contents unaltered.

5.     Change primes so that even numbers are not considered as values for factor or nextmultiple.

6.     Change your solution to Exercise 5 so that even numbers, except 2, do not appear in candidates. Use only enough storage for 2 and the odd numbers between 2 and n.

7.     This is the same as Exercise 6 except that candidates must use only enough storage for 2 and the odd primes between 2 and n.

8.     Change remove so that instead of increasing factor by 1, factor is increased to the value of the next prime in candidates. You may choose to do this by starting with remove from your solution to Exercise 7.

9.     Analyze the time and storage required for your solutions to Exercises 5 to 8.

10.     For the prime problem, find a solution that stores no primes.

11.     Explain why we must know the data type of the information stored in variables A and B in order to multiply their contents correctly.

12.     Suppose that integers are represented as in the example of Section 1.3.2. Write an algorithm, in English, that will allow someone who knows addition tables up to 9 to compute the correct sum of the contents of A and B. The algorithm must yield the correct result no matter what integers are stored in A and B.

13.     This is the same as Exercise 12 except that reals are represented, and the algorithm should yield the correct sum represented as a real number.

14.     Express the Bowling Scores program just as the prime number solution is expressed in Figure 1.3. What are its local, nonlocal, and global variables?

15.     Modify the flowcharts and program for the bowling problem to obtain flowcharts and a program solving the same problem but also printing out the number of rolls for each game.

16.     What functions might be inserted in Bowling Scores to check the validity of its input and output? Where would they go in the program?

17.     What elements of style have been incorporated in the programs of Sections 1.9.2 and 1.9.3?

Suggested Assignments

1.     Write and run a program to print all perfect numbers no greater than n. A number is perfect if it is equal to the sum of its factors; thus 6 is a perfect number (6 = 1 + 2 + 3). The program is to create a collection candidates of all candidates for perfect numbers, remove those that are not perfect from candidates, and print the numbers remaining in it. The program must treat the collection with operations set, insert, omit, and belongs as a data abstraction.

2.     Write a program to read in integers, store them in a collection, then read in more integers, remove each of these from it, and finally print all the integers remaining in it. The program is to treat the collection of integers as a data abstraction with the operations of set, insert, omit, and belongs.

3.     a.     Why does the Bowling Scores program require that the input consist of at least one game?

b.     Change the program so that it handles the case of no games as input in a reasonable way.

Go to Chapter 2      Return to Table of Contents


Data Structures Using C: CHAPTER 2: RECORDS, ARRAYS, AND POINTERS

 Return to Table of Contents

CHAPTER 2: RECORDS, ARRAYS, AND POINTERS

Presents the first serious consideration of data structuring

Focuses on

two data structures for grouping related information

records and arrays

two important operations on arrays

randomly accessing an array element

traversing an array

pointers and their use

as indices to arrays

to access information stored in dynamic memory

pointer arrays

their application and benefits

representations of two-dimensional arrays

to show different implementations

to evaluate the benefits of these implementations

Case study--stable marriages

highlights the benefits of carefully selecting both the information to be stored

and the data structure in which to store it

2.1: Bricks

Now that the context in which problems will be solved and programs written has been presented, it is time to start studying seriously the effect of choice of data structure on the problem solution. Before we explore some simple examples, the two basic data structures--records and arrays--must be understood, along with the two basic operations on them--selection and traversal.

A record is a data structure made up of a fixed number of information items, not necessarily of the same type. In C records are of the type structure. In the text the two words record and structure will be used interchangeably. An array is a data structure made up of a fixed number of information items in sequence; all items are required to be of the same type. Records and arrays have boundless applications. Records are useful when the items are to be referred to by name, while arrays are useful when the items are to be referred to by their positions in the sequence. The fundamental importance of the array was recognized quite early in the development of programming languages; the structure, or record, has only recently been incorporated in many languages. Pointers are variables that

1. Refer or point to memory locations for variables

2. Point to data structures, or

3. Link the components of a data structure together

Records and arrays are the bricks out of which more complex data structures are built, and pointers are the mortar holding them together.

Just as a builder asks for bricks and mortar, programmers in a high-level language ask for data structures to help build their solutions. A data structure is specified by declaring a variable name to be of a specific type. Some languages restrict the choice of type to individual variables or to arrays of type int, float, or char. C includes these, plus other types such as records, pointers, and unions. However, it does not have type boolean, which is found in other languages. C does allow the user to give new names to types.

Records and arrays allow related items of information to be treated as a group. The individual pieces of information in a record (structure) are called members in C. In most languages they are termed fields, but this word has a special meaning in C. Since that meaning is never utilized in this text, both member and field will be used for the elements of a record. In the case of arrays the individual pieces of information are called entries. The merit of records and arrays lies in the fact that they allow random access to any one of their fields or entries. Random access is accomplished by direct location of the desired piece of information, without regard to sequence. This means that access to an item may be accomplished in a fixed amount of time, no matter what item was previously accessed. This contrasts with sequential access, where the time required to access an item depends on the number of items stored between that item and the last item accessed, since each of the intermediate items must be accessed. With random access, any member of a record or any entry of an array may be accessed in a fixed amount of time, typically a few millionths or billionths of a second.

Both records and arrays provide the means to treat related items as a unit, and both generalize the idea of variables containing one item to variables containing many items. Records will be discussed next, then arrays.

2.1.1 Structures or Records in C

In C, memory can be thought of as consisting of storage elements of different length and complexity. Their complexity is determined by their type. The simplest storage elements have one of the three basic types, int, float, and char. Up one level in complexity are storage elements that have type structure. Among these, the simplest have exactly one member, which in turn has one of the basic types. A storage element of type structure for a book would be more complex; it might include members for its title, author, cost, and number of pages. Members are not restricted to one of the basic types but may themselves have type structure.

Consider the following C declarations:

float amount;     struct book               struct book text;

{

char title[35];

struct name author;

float cost;

int pages;

};

struct balance    struct name

{                 {

    >Comment

float amount;     char last[15];

}bankbalance;         char middleinit;

char first[15];

}

Storage for the three variables amount, bankbalance, and text may be pictured as in Figure 2.1. From the figure it is clear that storage must differ in complexity for different variables. The simplest is obviously the one that stores only one piece of information, as does the storage element for the variable amount. The structure bankbalance is obviously more complex; it could also be made longer by adding more fields such as lastdeposit and lastwithdrawal. Of course, more memory is needed to store the variable text than to store bankbalance. The type structure generalizes the idea of an individual variable containing only one piece of information to variables containing related pieces of information. The amount of storage required determines the length of a record, while the record's structure determines its complexity.

It is now possible to write C statements that refer to the entire record text, its author field, or its last field, using the names text,text.author, and text.author.last, respectively. In earlier versions of C it was not possible to write statements that referred to entire structures.

Figure 2.1 Three Variables

2.1.2 Arrays in C

Arrays provide another way to group either individual items or records. An array stores a sequence of entries; an entry in an array is referred to and located by its position in the sequence. An array has a fixed size and is homogeneous (that is, its entries must all be of the same type--say, of type integer or book). This contrasts with a record, whose fields may be of different types. An array entry is identified by qualifying the array name by the entry's position. To identify a member in a structure, the structure name is qualified by the member name. (Note that arrays can have entries of type structure, and structures can have members of type array.)

Consider the following declarations:

int book_number[80];

struct name author[40];

float prices[80];

char library[30];

The declaration specifies the name of each variable (e.g., book_number) and its type for each of four variables. It also specifies the range of values for an array index for each of the four arrays. An array index refers to a position of the array. Book_number,author,prices, and library have array indexes that may take on values in the range 0 to 79, 0 to 39, 0 to 79, and 0 to 29, respectively. In C the first position in an array is always the zero position. Book_number[3], author[9], and prices[5] refer to the fourth, tenth, and sixth entries, respectively, and library[12] refers to the thirteenth character in the character string library.

It has already been mentioned that arrays are very useful when entries must be accessed randomly. This occurs when the order in which requests for access to entries cannot be predicted. Suppose an array stores the initial mileage for each car of a fleet of rental cars. As a car is returned, its initial mileage is needed to prepare a bill based on mileage used. The next car to be returned can be any rented car. To access its initial mileage directly requires the random access capability of arrays. Processing time is saved, since the information entry is located directly--no time is lost searching through other entries to find it. When information is accessed directly, in a constant time, it is said to be selected. Thus any array entry whose position is known can be selected.

Instead of accessing a single array entry, a programmer may wish to process an entire array by accessing and processing each entry in turn. Moving sequentially through the entries is a traverse. In this case, the program is said to traverse the array. Such a traversal would be required in order to print out, in order, the initial mileage of each car in the fleet. A complete traversal is accomplished when each entry has been processed exactly once. A simple loop, starting at the first entry and finishing at the last, will accomplish this. Another use of traversal is to find an array entry whose position is unknown. In this case, the loop is terminated when the entry is found. If the traversal is completed, it means the desired entry was not present.

Selection and traversal are two ways to access an array entry. If the entry's position is known, selection is the efficient way to access it. When successive array positions must be examined to find the entry, selection is not possible, but traversal may be used.

If the traversal terminates at the ith entry (because it is the desired one), then a time proportional to i is required, since the i - 1 preceding entries must be accessed before the ith entry is obtained. Selection, on the other hand, takes constant time. This is an important distinction between these two basic operations on arrays. Selection means you can go directly to what is wanted; traversal means you must rummage around to find it. Use selection when one or more accesses must be made in arbitrary order and traversal when each entry is to be accessed in sequence.

Example 2.1

Suppose that data, such as integers in the range 1 to 100, are stored in an array a, as shown in Figure 2.2. Given any valid index value in the variable i, print the number stored in the ith position of the array a.

Figure 2.2 The Array for Example 2.1

This may be done by executing the C statement

printf("%d\n",a[i]);

This is a concise, easy-to-understand solution to Example 2.1. It takes a constant amount of time to execute, since it selects the desired entry of a. Note that when the value of the index i is 7, then the array entry with the value 87 is printed; when the value of i is 5, then the entry with the value 3 is printed. The index i is avariable that points to a particular entry of a. The current value of the index i determines the entry currently selected. Changing the value of i changes the entry to which it points. Thus the solution can be interpreted as the command "Print the entry of a to which the index i currently points."

Example 2.2

Let a be an array of length n, and let k be a variable containing an integer in the range 1 to 100. If there is some value of i for which a[i] contains k, then print that value of i; if there is more than one such value, print any one of them. If the value of k is nowhere in a, then print "-1."

Array a might represent the information kept by a car rental agency. For each i, array entry a[i] contains the account number, k, of the individual or firm currently renting car i. Example 2.1 thus requires finding the account now leasing car i. Example 2.2 requires finding the car being used by account k.

Example 2.2 could certainly be solved by traversing the array a, searching for an entry whose value equals the value of k. If one is found, its position in a would be printed. If the traversal is completed without success, "-l" would be printed. This solution is easy to code and could be written as follows:

found = FALSE;

loc = 0;

    >Comment

while (found != TRUE && loc <= n)

if (a[loc] != k)

loc++;

else

found = TRUE;

if (found)

printf("\n %d\n",loc);

else

printf("\n - 1 \n");

Although this program segment for array traversal is simple, it is more complex than that for Example 2.1. Its execution time is proportional to the number of entries searched before the number in k is found, and if it is not found, the execution time is proportional to n.

Is it possible to find another way to solve this problem, one that will be as concise and clear as that for Example 2.1 and also execute in constant time? The answer is yes, if the information stored in a is represented in such a way that the program can select the correct value to print.

Such a program would start with creation of an array pointer of length 101, as follows. For each integer k (account number) between 1 and 100, if k does not appear in a, place "-1" in pointer[k]. If k does appear in a (it is an active account), place in pointer[k] the position of an entry of a that contains k. That is, place in pointer[k] the index value of an entry in a containing k. Thus, in Figure 2.3, pointer[3] contains 5, since a[5] contains 3.

With the array pointer available, the solution to Example 2.2 may be written as

printf("%d\n",pointer[k]);

This is the concise, easy-to-understand solution desired, and it executes in constant time. Again, the way in which the relevant information was represented in the program--that is, the way in which the data was structured--allows this solution to be achieved. Even in such a simple example, the way data is represented has a substantial impact on the resulting program.

Note that the use of the pointer array instead of a may cause some information available in a to be lost. This is because pointer captures only one of a's positions that contains a specific number, even though that number may appear in a more than once. Finally, a price in execution time must be paid to create pointer from a in the first place. However, if the solution to Example 2.2 needed to be executed repeatedly, for many values of k, then it probably would be worth the price. It is frequently the case that prior processing to extract or restructure data saves later processing time.

Figure 2.3 A Pointer Array for Example 2.2

In terms of storage required, a takes up n storage entries, and pointer takes up 101 storage entries. If n is much larger than 101, then pointer even saves storage; if n is much smaller, it requires more storage. Thus it is possible to trade time for storage. It will become apparent that this is generally true: saving execution time requires more storage, and vice versa.

When the choice of data structure was the array a, Example 2.1 was solved using selection, but traversal was required to find a solution to Example 2.2. When the pointer array was chosen to represent information, selection could be used to solve Example 2.2 more efficiently. However, if we had wanted to know all the cars leased by account k, pointer would not have been sufficient. It would have been necessary to traverse a to find all the occurrences of account number k. Can you find a way to avoid this traversal?

2.1.3 Two-Dimensional Arrays

Until now we have discussed one-dimensional arrays. High-level languages also provide arrays of more than one dimension. Thus the C declaration

int a[5][3];

defines a two-dimensional array of 5 X 3 or 15 entries, each of type integer. This array a can also be viewed as a collection of 5 one-dimensional arrays, each of length 3, or 3 one-dimensional arrays, each of length 5. Again, we can think of two variables i and j of the correct index type as pointing to an entry of the array. Thus a[i][j] refers to the entry of the ith row and jth column, or to the entry of a currently pointed to by i and j. We assume that the (i,j)th entry can be accessed in constant time, no matter what the value of i or j . That is, the (i,j)th entry may be selected. Traversal of two-dimensional arrays is also easy.

A two-dimensional array is symmetric if a[i][j] and a[j][i] contain the same value for all i and j between 0 and n - 1. Suppose a collection of n2 numbers, representing the distances between n(  [image] 50) cities, is stored in a two-dimensional array a. Thus a[i][j] contains the distance between city i and city j. For this example, which involves distances, the array must be symmetric. However, the array a shown in Figure 2.4 is not symmetric, because a[2][3] does not equal a[3][2]. The n entries a[0][0], a[1][1],..., a[n - 1][n - 1] are called its diagonal entries. In the city-distance array, the diagonal entries must, of course, be zero.

Figure 2.4 An Asymmetric Two-Dimensional Array

Example 2.3

Suppose a programmer wished, perhaps as part of an input validation function for the data of an array, to check the array for symmetry. The programmer might choose to write a function check to return the value true if the array is symmetric and false otherwise. A simple traversal, as follows, will do the checking.

    >Comment

typedef int collection[50][50];

check(a,n)

/* Returns true only if the first n

rows and columns of a represent

a symmetric array

*/

    >Comment

collection a;

int n;

{

    >Comment

int i,j,ck;

ck = TRUE;

    >Comment

for (i = 0; i<n; i++)

for (j = 0; j<n; j++)

if (a[i][j] != a[j][i])

ck = FALSE;

return(ck);

}

This program does a complete traversal of the appropriate n entries of the array a. However, it is inefficient as a solution for a number of reasons. First, it does not exit immediately from the loops, even though it may have just found that a was not symmetric. Second, it checks the diagonal elements, which will always satisfy a[i][j] = a[j][i]. Finally, it double-checks each off-diagonal entry. For instance, when i = 2 and j = 3, it checks a[2][3] against a[3][2], and when i = 3 and j = 2, it checks a[3][2] against a[2][3] again. A better version would be a modified traversal, such as

typedef int collection[50][50];

check(a,n)

/* Returns true only if the first n

rows and columns of a represent

a symmetric array

*/

collection a;

int n;

{

int i,j,ck;

ck = TRUE;

    >Comment

for(i=1; i<n && ck; i++)

for(j=0; j <i && ck; j++)

if(a[i][j]! = a [j][i])

ck = FALSE;

return(ck);

}

Be sure you understand why the modified lines of code eliminate the inefficiencies.

2.1.4 Representation of Records in Arrays

In C the record, or structure, is a basic data structure. In some languages the record is not a built-in feature but must be constructed from the data structures (such as arrays) that are available. To see how this construction might be done, assume C designers had provided the integer, float and char data types, and only one-dimensional arrays. Not only would records have to be constructed, but so would arrays of more dimensions if they were needed. There are two reasons for looking at ways to build data structures.

1.     It affords insight into how compilers implement records and higher-dimensional arrays.

2.     It demonstrates how programmers build new structures from those that are given in order to tailor data structures to specific needs.

Moreover, even though records and higher-dimensional arrays are available in C, there are times when programmers may find it preferable to use these construction techniques.

Let us consider how to represent n records, each composed of m fields of the same fixed type. Suppose m is 3. A natural solution is to take an array of length 3Xn, of the same type as the fields, and store the records in the array. The first record takes up the first three entries, the second record the next three entries, and so on, as in Figure 2.5. Each field of a record will be stored in the corresponding one of its three entries.

The ith record begins in position 3*(i - 1) of the array. Thus we may select the ith record. To access a specific field of the ith record (say, the third) requires that position 3*(i - 1) + 2 of the array be accessed. Although this is the (3*(i - 1) + 2)th array element, it contains the third field value of the ith record. Thus we may select any field of any record too.

It is also easy to traverse the records, using a loop. For example, the following function traverses the n records stored in the array a and prints the value in their kth field, where k may be 1, 2, or 3.

    >Comment

typedef int fieldtype;

array is the name for the type: array of fieldtype of size 30

typedef fieldtype array [30];

printfield(a,n,k)

/* Prints the contents of the

kth field of each of the first

n records stored in array a.

*/

    >Comment

array a;

int n,k;

{

    >Comment

int length = 3, i;

for(i=0; i<n; i++)

printf("\n %d\n",a[length * i + (k-1)]);

}

In C, since arrays must be homogeneous (consisting of entries of the same type), this method is applicable only to homogeneous records--those whose fields are all of the same type. How then to handle the more general case of records whose fields are not homogeneous? The following example demonstrates the general concept and then how it is achieved in C.

Example 2.4

Consider Figure 2.6, which represents an array cars containing a record for each car of an automobile rental agency. The record for each car has an entry for the make (characters), monthly rate (a real number), current mileage (an integer), and the rentee (characters). As noted, this array, with entries of different types, could not be constructed in C. For the sake of simplicity, the following discussion does not deal with the actual storage taken by a character, float, or integer variable, but assumes that each takes one element of memory. All the records in the array are 42 elements long, using 15 elements for the make, 1 element each for the monthly rate and current mileage, and 25 elements for the rentee field.

Figure 2.5 An Array of Records (of Length 3)

Figure 2.6 Records of a Car Rental Agency Stored in an Array CARS

An offset gives the position of a data item with respect to a given reference point. When you say, "the third house from the corner," you are using an offset reference point. Each field has a field offset from the beginning of the record. In this case, the field offsets are 0, 15, 16, and 17, respectively, for make, monthly rate, current mileage, and rentee. The ith record has a record offset of 42X(i - 1) from the beginning of the array. The offset for its rentee field from the beginning of the array is 42X(i - 1) + 17, the sum of the record offset for the ith record and the field offset for the rentee.

In order to reference, for example, the current mileage field of the ith record, the programmer may refer to cars[42*(i-1)+16], assuming the first record starts in element zero of the array. Only the fields may be referred to by the programmer; the record itself cannot be dealt with directly as a unit. In C the task is simplified for the programmer by the use of structures and arrays of structures. Carrecord is defined as a structure and cars[100] as an array of 100 carrecords.

struct carrecord

{

char make[15];

float rate;

int mileage;

char rentee[25];

}cars[100];

Then the C programmer may refer to the (i + 1)th record as a unit as cars[i], and to its mileage field by cars[i].mileage. Treating a record as a unit allows intentions to be stated more clearly in a program. It also allows for the contents of one record to be assigned to the storage for another record more efficiently. This is analogous to being able to refer to an automobile and say "Move that automobile from here to there," rather than specifying the automobile or the move by referring to each of its parts individually. Languages that do not provide the record data structure prevent the programmer from referring to a record as a unit in this way.

Note that the record referred to during the execution of a program may be changed merely by changing the value of i, but the field referred to cannot be changed, since field names may not be changed during execution. The variable names cars[i].mileage shows this. The value of array index i can be changed, but the name cars cannot be. This means that code referring to the ith position of an array, such as cars[i], may refer to different array entries as aprogram executes, by computing a new value for i. Code using a field name will always refer to that particular field.

This example gives a technique for sorting records in languages without strong typing constraints. Even in languages with such constraints, the compiler might implement cars in the actual computer memory as illustrated in Figure 2.6. In this case, if the initial array address in memory is 100 rather than 0, then the offset (or relative position) for the (i + 1)th record is added to 100 instead of to 0. The compiler can produce commands that calculate the actual position in memory of any field of the ith record by adding the record offset and field offset to 100. Then a programmer's reference to cars[1] is interpreted as a reference to the 42 consecutive locations 142 to 183. Similarly, cars [1].rate refers to the entry in location 157. This assumes the array starts at memory location 100. The offsets are independent of the starting location. When applied to homogeneous records, this method requires that cars be defined to be the same type as the record's fields.

Observe that the cars array preserves the homogeneity of type that is required of all arrays in C. Its entries are now all of the same specific record type. The array entries just happen to be of type carrecord, even though a record itself is not homogeneous. C allows operations to be performed directly on structures, treating them as a unit. Of course, the compiler hides the details of the records' implementation from readers of the program and provides facility in handling records.

The method just discussed is a single-array implementation. The second method of representing the n records is a multiarray implementation, shown in Figure 2.7. The method simply uses a different array for each field. Each array then contains information of the appropriate type only. There is no direct way to refer to the collective notion cars[2]. Instead, its data are given by make[2],rate[2], mileage[2], and rentee[2]. Selection of the ith record and traversal through the records is also possible for this method.

The single-array technique may cause problems when all entries are not of the same type. As noted, however, Example 2.4 shows offset calculations of the type that may be used by a compiler but are invisible to the programmer. The multiarray technique is useful in any language that provides arrays. Both techniques fail to give the programmer means to refer directly to, or to manipulate as a unit, an entire record composed of more than one field. The C approach provides the record data structure and makes the underlying details transparent to the C programmer, thus providing an elegant solution to the problem of treating information of different types as a unit.

Figure 2.7 A Multiarray Storage of Records of a Car Rental Agency

2.1.5 Variable-Length Records

When the records of a collection are not necessarily of the same length, they are called variable-length records. Looking back at the single-array and the multiarray representations for a collection of records, it is clear that selection of the ith record was possible only under the tacit assumption that the records were all the same length.

Suppose a field is added to carrecord for each previous rentee. Surely some newer cars will have few previous rentees, while older cars will have a long history of rentees. The programmer could decide to represent each car's record by using a single-length record large enough to accommodate the largest possible number of past rentees. This would aid in processing, as uniformity always does, but would waste a great deal of storage if many records actually have short histories.

With variable-length records, shorter records use fewer locations than longer records, as shown in Figure 2.8. Details of the implementation of such a collection of records must somehow specify where one record ends and the next starts. Separators between records would do, or a field associated with each record could give its total length. Using separators, the last element of the storage for a record would contain a special value denoting the end of the record's storage. If associated fields are used, the first storage element of each record would contain the record's length and would allow a determination of the last element of its storage. In either case, the ith record may no longer be selected, because its location cannot be calculated. However, given the position of the ith record, one can calculate the position of the (i + 1)th record.

Figure 2.8 Variable-Length Records Stored in Array A

Assuming the first element of each record contains its length, as in Figure 2.8, the following function accomplishes the traversal. A record of zero length is used as a sentinel to indicate that there are no more records in the array.

#define SENTINEL 0

    >Comment

traverse(a)

*/ Traverses an array a */

int a[];

{

int i;

i = 0;

    >Comment

while(a[i] != SENTINEL)

{

process(a,i);

i =i+a[i];

}

}

The record array here is simply an array of individual elements. The programmer imposes the record structure. Each field of the record must thus have the same type in C. Process is a function that does whatever processing is called for. Thus, traversal through the records can be used to access the ith record, but this takes time proportional to i.

2.1.6 The Use of Unions in C

No facility is provided in C to handle variable length records directly, though unions do afford a means of handling some variability. A union in C may contain data that can have different types and sizes at different times. Thus unions provide a means of using a single storage area for distinct purposes. Unions are declared in the same manner as structures. The compiler allocates enough memory to hold the largest member that can be stored in the union. Unions are consequently of fixed size but allow variation in the type of data stored. The programmer has to keep track of which type is currently stored in the union. Unions would not be of significant help in Example 2.4 with the list of rentees. In this case there can be more than a few distinct rentees for each car, and unions do not fit this type of variability. Lists, the topic of the next chapter, would be used for this example.

union book_lookup

{

float location;

char title[50];

};

Variables of type book_lookup could contain either the title of a book or its location in a library (based on the Dewey Decimal System). Unions thus give some measure of variability in records. However, the records do not have variable lengths; they are of fixed length.

Unions may occur in structures, and structures may occur in unions.

struct

{

int library_id;

union book_lookup information;

}catalog[100];

Each individual structure of the catalog array can contain an "i.d." number and either a real number or character strings of length 50 in any of its entries. Then, for example, catalog[40].information.title refers to the member title in entry 40 of catalog.

2.1.7 Address Calculation without Access

Calculating the address of an item of data is not the same thing as accessing the item of data. The address of an item is its storage position, not its value. To access the item means to read its current value from memory or to store a value in its memory location.

Many authors associate the process of selection with the idea of access to an item, rather than with the calculation of the storage address of the item. They say that the ith item of a structure can be selected if, given the value of i, the program can access the item in a time that is independent of i. Such a structure is called a random access structure. This definition served well for early computers, in which arrays could reside in a single homogeneous memory. However, current practice includes radically different memory architectures.

Modern computers have memory hierarchies--that is, different kinds of memory--whose access times vary by one or two orders of magnitude from one level of the hierarchy to the next. It may well be the case that some portion of an array is stored in fast memory, such as high-speed cache memory, and some portion in slower central memory. Then the access time for an array element would depend on the index of the element (that is, its position in the sequence of the array), but the time required to compute the storage address would still be independent of the value of the index. Consequently, the process of address calculation should be thought of as separate from the process of access to the data. Selection thus denotes address calculation without access. Although it is important to grasp this distinction between address calculation and actual access, it is generally ignored in the explanations here, as in other texts, for convenience of exposition.

Our interest in the effect of data structures on program efficiency concerns the timing of various algorithms. Because access times may vary, depending on features of the physical computer that are completely hidden from the language, time will be measured here as if each program were executed on a computer that has a single homogeneous central memory.

The special techniques necessary to write efficient programs for computers with hierarchical memories are not within the scope of this text. Some computers even allow the exploitation of any parallelism inherent in an algorithm, which means that more than one operation may be carried out at a time. This text also largely ignores this possibility of parallel processing, although it is currently a topic of computer research.

If this array of material leaves you in disarray, keep going. Pointers will point the way!

2.2: Mortar

2.2.1 Pointers

In C, records can be stored in arrays. They can also be stored in an area of memory called the dynamic memory. Dynamic memory is managed automatically for the programmer, who can request and release storage as needed. Pointers are variables that point to data items. Those that point to array entries, as noted earlier, are called array indexes. Those that point to individual data items, or structures, are called pointer variables. By following a pointer, you can access a record. Sometimes it is necessary to access a record by following a sequence of pointers.

Pointers and pointer variables afford flexibility in structuring data but must be used with great care, since they can introduce subtle errors that cause difficulty in the debugging of a program. The errors are problems that arise when collections of records can expand or contract in unpredictable ways. Storage for such records must be managed carefully.

To understand pointers and pointer variables, it is practical to examine them in a more static context. They are useful even then, in solving problems involving variable-length records and the representation of arrays. A good way to become familiar with the workings of pointers and pointer variables is to solve the following promblem: Is there a way to store n variable-length records in a one-dimensional array so that traversal through the records is possible, and, more important, so that the (i + 1)th one can be selected?

The solution is not difficult conceptually. Suppose a pointer array pa is created, as in Figure 2.9. The information in pa[i] should state where the base, or origin, of the (i + 1)th record is in a. Thus pa [3] is 13, because the fourth record begins in a[13]. One describes this by saying that pa[i] contains a pointer to the (i + 1)th record in a. The variable i can be thought of as pointing indirectly to a record in a; to get to it requires an access to pa along the way.

Recall that records are represented by a sequence of consecutive entries, so pa[i] points to the first entry of the (i + 1)th record, its base. Once the base of a record and its field offsets are known, any field of the record can be selected. This technique of using pointer arrays is an old and very important technique in computer science. This kind of pointer is just an array index; it is also known as a relative pointer, because it indicates the position of the record in relation to the other records within the same array. This is different from the other type of pointer, the pointer variable, which refers to actual memory locations.

Figure 2.9 Pointer Array PA, with Pointers Stored in A

Selecting the (i + 1)th record is now possible by accessing pa[i] and following the pointer found there to the entry of a in which the (i + 1)th record begins. The base of this record has the name a[pa[i]]. Note that selection is being applied first to array pa, and then to array a. Each takes constant time, which is why we say the (i + 1)th record may be selected. It is also easy to traverse through the records using pa, as shown by the following program.

traverse(a,pa,n)

/* Traverses through the n

records stored in a which

are pointed to by indices

stored in the pointer array pa

*/

int a[],pa[],n;

{

int i;

for (i=0; i<n; i++)

process(a,pa,i);

}

Using the pointer array costs additional storage but solves the problem posed for variable-length records. Relative pointers to records stored in an array are easy to use when language typing constraints are not violated.

2.2.2 Pointer Variables and Dynamic Memory

As a programmer you may need to store a collection of records that are not all of the same type. In C they cannot be kept in a single array, since that would violate the array typing constraint, but there is a way out: simply declare each different type to correspond to one of the members of a union occurring in the record. With this trick, C treats records of different types as records of the same type. Another difficulty in C is how to store a collection of records of the same type when the maximum number of such records is not known in advance. Storing them in an array requires specifying the length of the array, but the programmer may not know what this should be. A practical solution to this problem is provided by the dynamic memory.

So far, by grouping records in arrays, it has been possible to refer to a record by its position in the array. This eliminates the need to attach a separate name to each record of the group. Dynamic memory may be viewed as a collection of storage elements that are on call for the storage of information. However, records stored in dynamic memory do not have individual names or array positions to be used for their reference. Instead, pointer variables are used to point to an individual record. A record in dynamic memory can be accessed only through its pointer variable. A pointer is like a string attached to a record: follow it and you get to the record. The pointers previously examined point to a record's position in an array, whereas pointer variables point to a record's location in dynamic memory. Both play the same role: they lead to a record.

Declaring a record type for records to be stored in dynamic memory is the same as for individual records or records stored in arrays. However, an associated variable of type pointer must be declared and used to refer to the record. For example, to store a record of type carrecord to be referenced by the pointer variable pcar 1 requires the following declaration:

struct carrecord *pcar1,car2;

An important distinction exists between the variable pcar1, which is a pointer, and car2 of type carrecord. The variable car2 is not stored in dynamic memory. Storage is allocated to car2 as soon as the component of the program in which it is declared is executed. The programmer need do nothing other than declare it to ensure this allocation of storage; statements assigning values to it may immediately be written.

This is not the case for the record pointed to by pcar1, which is to be stored in dynamic memory; pcar1 itself is not stored in dynamic memory, but the record it will point to is. This is because pcar1 was declared to be a pointer. The prefix, *, denotes this. No storage is allocated for the record that pcar1 will point to until a specific request made by the programmer is executed. This is done by an invocation of the C function malloc. Thus,

pcar1 = malloc(sizeof(struct carrecord));

when executed, places a pointer value into pcar1 and also allocates storage in dynamic memory for a record of type carrecord to which that pointer points. Until this is done, pcar1's value is undefined. After the function is executed, the situation may be pictured as in Figure 2.10.

The record may be referenced by pcar1 and its members referenced by pcar1->make, pcar1->rate, pcar1->mileage, and pcar1->rentee. At this point, though, the contents of the fields are undefined. Storage has simply been allocated for them in dynamic memory. They may be assigned values by using the assignment statement. Each field of the record pointed to by pcar1 can be given a value. Thus

pcar1->rate = 20.25;

assigns a value of 20.25 to the rate field of the record pointed to by pcar 1. You must be sure before assigning values to a record stored in dynamic memory that you have assigned a value to the pointer variable pointing to that record.

One advantage of pointer variables in C is that the value of a pointer can be printed. It means that a programmer can see the value of a pointer for debugging purposes. Other advantages of using pointer variables (that is, of storing records in dynamic memory) will soon become apparent.

Selection and traversal of a collection of records stored in dynamic memory may be accomplished using an array of pointers. The array entries are pointers that point to records in dynamic memory.

Example 2.5

The definition

struct carrecord *pcar[100];

establishes an array of 100 pointers that can address 100 records stored in dynamic memory. A reference to a member of a record is done by first specifying the pointer to the record, pcar[i], and then the desired member of the record, say make. The resultant reference would then be written as pcar[i]->make. This references the make member of the (i + 1)th record.

Arrays of pointers to records in dynamic memory are not as easy to use as relative pointers to records in an array, but they provide significant flexibility. You shall see, though, that considerable care is needed to use them properly.

Figure 2.10 Dynamic Memory

2.3: Representations of Two-Dimensional Arrays

It is time to return to the problem of how to represent higher-dimensional arrays in terms of one-dimensional arrays. For simplicity, only two-dimensional arrays are discussed, but the techniques can be readily extended for higher dimensions. Two techniques will be studied: 1) rowwise and columnwise representation and 2) pointer array representation.

What exactly is meant by representing a two-dimensional array a in terms of a one-dimensional array (say, data), and why is it done? The idea is that the problem to be solved may involve the use of a, but the language being used may not provide two-dimensional arrays, or the programmer may choose not to use them for some other reason. Instead, the entries of a are stored in data. This must be done so that whenever reference is to be made to the entry in position (i,j) of a in the program, the corresponding position of data in which it is stored is referenced instead. For example, the check functions of Example 2.3 referred to a two-dimensional array a. If its information were stored in data, then the functions would have to be rewritten, replacing each reference to a[i][j] with a reference to the correct entry in data that corresponds to it. Of course, the declarations involving a would also have to be replaced by declarations involving data.

2.3.1 Rowwise and Columnwise Representation

Suppose a has r rows and c columns, with row index range 0 . . . (r - 1) and column index range 0 . . . (c - 1). A rowwise representation for a uses a data array of length r X c and stores the first row of a in the first c consecutive entries of data, the second row in the next c entries, and so on.

Rowwise representation in arrays is similar to the first method discussed for storing records. In fact, it is logically the same as viewing each row as a record of length c. Thus the offset of the entry in position (i,j) of a is i * c + j in data. This is because rows 0, 1, . . . , i - 1 precede the ith row in data, and take up i X c entries. The jth position of the ith row is offset an additional j entries.

For the array a shown in Figure 2.11, with c = 5, the offset of a[2][3]is 2 * 5 + 3, or 13, and the third row of a starts at data[10]. a[2][3] is represented by data[13]. In C the offset and index are the same, since index values start at zero.

With this representation, function check of Example 2.3 would be written as

#define TRUE 1

#define FALSE 0

check(data,n)

/* Returns true only if an nxn

two-dimensional array stored in

rowwise representation in array

data is symmetric

*/

    >Comment

int data[];

int n;

{

    >Comment

int i,j,ck;

ck = TRUE;

for(i=1; i<n && ck ; i++)

for(j=0; j<i && ck ; j++)

    >Comment

if(data[(i*n+j)] != data[(j*n+i)])

ck =FALSE;

return(ck);

}

Figure 2.11 Representing a Two-Dimensional Array as a One-DimensionalArray

Obviously, this representation in data allows a to be traversed, as well as allowing its (i,j)th entry to be selected.

The columnwise representation of a two-dimensional array is similar, except that the columns rather than the rows of a are stored sequentially in data. The offset for the entry in position (i,j) with the columnwise representation is j*r + i.

2.3.2 Symmetric Array Representation

Example 2.6

Suppose a program involves an n X n array a that is always symmetric. In other words, it is initially symmetric, and any processing performed on a does not disturb its symmetry. When n is small, both the rowwise and columnwise representations are feasible, even though roughly one-half of the information being stored is superfluous. For large values of n (say, 400), the computer system may not provide the required 160,000 storage elements, so the program could not execute. One remedy may be to store only the diagonal entries of a and the entries below the diagonal (those whose row index i exceeds the column index j). This requires ((n X n) + n)/2 entries, or 80,200 instead of 160,000.

Although high-level languages could have provided a data-type, symmetric array to be used in this way, no language does. Instead, the programmer can construct a data array of length ((n X n) + n)/2 to store the entries. The idea is the same as for a rowwise representation, except that only those entries of a row in the lower half or on the diagonal of a are stored. Thus for the row in position i, entries a[i-1][0], a[i-1][1], . . . , a[i][i] are stored in consecutive entries of data, each row's entries following those of the preceding row.

To use this representation of a, the programmer needs to be able to refer, for given i and j, to the proper position of data that corresponds to a[i][j], just as was done in the check function. Finding the offset formula is a little more complex than for the rowwise representation but is done in the same way.

Preceding the entries for the row in position i are the entries for rows in positions 0, 1, . . . , i - 1. The entry in position j of this row is then an additional j entries down in data. Thus the offset for a[i] [j] is 1 + 2 + 3 +. . . + i + j . Any reference we want to make to a[i] [j] is made, instead, by a reference to data [1 + 2 + ... + i + j]. This, of course, is true only for i   [image]  j, since a[i] [j], when i < j, is not stored in data directly. Its symmetric entry in a, a[j][i], is stored in data [1 + 2 + ... + j + i]. So if i < j, a reference to a[i] [j] must be achieved by referring to a[j] [i].

The polynomial (1 + 2 + . . . + i + j) (or ((1 + 2 + . . . + i) + j) appears at first glance to require a loop for its calculation, requiring time proportional to i. However,

The division is integer division, so the polynomial may be evaluated in constant time. This means that the entry in position (i,j) of a can be selected using the index,

Traversal may also be accomplished easily. Consider that the entries are stored in data for each row as records of variable length, yet they can be selected and traversed conveniently with no pointer array. The reason is that their lengths, and the order in which they are stored, are regular enough that the requisite calculations for positions can be made. In the general case of variable-length records, there is no way to do this.

2.3.3 Pointer Array Representation

The second technique for representing a two-dimensional array in terms of one-dimensional arrays uses a pointer array p. Pointer arrays have pointers as their entries. The entries of the pointer array point to the rows of a, which may be stored in one of two ways: in data (a one-dimensional array) or in the dynamic memory. The number of rows, r, of a must be known. Then p, data, and r represent a when the array is used, and p and r represent a when dynamic memory is used. In the array case, p is of integer type, as in Section 2.2.1. In fact, this is just an application of that method.

Clearly, there is no need to store the rows one after another in data; they can be anywhere in the array as long as they do not overlap. This feature provides considerable flexibility in the use of storage.

A variation of the array technique would treat the rows stored in data as row records. In the dynamic memory case, p is of type pointer, and the row records are stored in dynamic memory. Selection and traversal are thus both easily achieved. The corresponding versions (storing rows in data, storing row records in data, storing row records in dynamic memory) of the function check would appear as follows.

Version 1--Storing Rows in Data

#define TRUE 1

#define FALSE 0

    >Comment

typedef int pointerarray[50];

typedef int dataarray[2500];

check(p,data,r) /* storing rows in data */

/* Returns true only if the rxr

two-dimensional array, whose rows

are stored in data and are pointed to

by pointers stored in p, is symmetric.

*/

    >Comment

pointerarray p;

dataarray data;

int r;

{

    >Comment

int i,j,ck;

ck = TRUE;

for(i=1;i<r && ck;i++)

for(j=0;j<i && ck;j++)

    >Comment

if(data [p[i]+j]! = data[p[j]+i])

ck = FALSE;

return (ck);

}

Version 2--Storing Row Records in Data

#define TRUE l

#define FALSE 0

    >Comment

typedef struct

{

int row[50];

}rowrecord;

typedef int pointerarray[50];

typedef rowrecord dataarray[50];

check (p, data, r) /* storing row records in data */

/* Returns true only if the rxr

two-dimensional array, whose rows

are stored in row records in data and

are pointed to by pointers stored

in p, is symmetric.

*/

    >Comment

pointerarray p;

dataarray data;

int r;

{

    >Comment

int i,j,ck;

ck = TRUE;

for(i=1;i<r && ck;i++)

for(j=0;j<i && ck;j++)

    >Comment

if(data[p[i]].row[j] !=

data[p[j]].row[i])

ck=FALSE;

return(ck);

}

The third version (storing row records in dynamic memory) is a complete program with a driver for check(p,r). It illustrates reading and printing of r and the array represented by r and p. The input and output from its execution might appear as follows.

Typical Input and Output

Enter an integer between 1 and 50 for r

4

Enter the next rowrecord

  3   4  -67   8

Enter the next rowrecord

  4   5   23  57

Enter the next rowrecord

-67  23   -6   4

Enter the next rowrecord

  8  57    4   0

r is 4

The row records are:

  3   4  -67   8

  4   5   23  57

-67  23   -6   4

  8  57    4   0

The array is symmetric

The program reads the two-dimensional input array, represents it, checks it for symmetry, and outputs the result.

Version 3--Storing Row Records in Dynamic Memory

#include <stdio.h>

#define TRUE 1

#define FALSE 0

    >Comment

typedef struct

{

int row[50];

}rowrecord, *rowpointer;

typedef rowpointer pointerarray[50];

main()

/* Driver for check(p,r) */

{

    >Comment

pointerarray p;

int r;

    >Comment

int i,j;

    >Comment

rowpointer malloc();

printf("Enter an integer between 1 and 50 for r\n");

scanf("%d",&r);

for(i=0;i<r;i++)

{

    >Comment

p[i]=malloc(sizeof(rowrecord));

printf("Enter the next rowrecord\n");

    >Comment

for(j=0;j<r;j++)

scanf("%d",&(p[i]->row[j]));

}

printf("\n r is %d \n",r);

printf("The row records are: \n");

for (i=0;i<r;i++)

{

printf("\n");

    >Comment

for (j=0;j<r;j++)

printf("%3d",p[i]->row[j]);

}

if(check(p,r))

printf("\n The array is symmetric \n");

else

printf("\n The array is not symmetric \n");

}

check(p,r) /* Storing row records in dynamic memory */

/* Returns true only if the rxr

two-dimensional array, whose rows

are stored in row records in dynamic

memory and are pointed to by pointers

stored in p, is symmetric.

*/

pointerarray p;

int r;

{

    >Comment

int i,j,ck;

ck=TRUE;

for(i=1;i<r && ck;i++)

for(j=0;j<i && ck;j++)

    >Comment

if (p[i]->row[j] != p[j]->row[i])

ck=FALSE;

return (ck);

}

The pointer array technique may be used storing columns instead of rows. Then p's entries point to the column records.

Example 2.7

It is often desirable to store information in an array in a specific order. Algorithms to rearrange the information to achieve the order may require repeated interchange of rows. Suppose the programmer wants to interchange the ith and jth rows of the array a when it is represented using the pointer array p. This can be accomplished quickly, taking advantage of the pointers, by executing

temp = p[i];

p[i] = p[j];

p[j] = temp;

Note that the data in each row are not moved; only the pointers change. The time it takes the computer to switch rows is independent of their length. Contrast this with the processing involved for the rowwise method of representing a (Section 2.3.1). Even if a actually is available directly as a two-dimensional array, the pointer array representation is more efficient. Swapping columns, of course, is a much more involved task, unless the programmer has chosen to store columns rather than rows.

The pointer array method offers the advantage of convenience in storing rows of different lengths. Thus it can be used to construct arrays with rows of different lengths easily. Also, if many rows of a are the same, they need not be duplicated in data or in dynamic memory. Their pointer entries in pointer array p need merely point to the same record, resulting in significant storage savings. The means should fit the ends as much as possible! In programming, the ends (the desired operations) determine the means (the data structures to be used).

2.4: Advantages and Disadvantages of the Techniques

If you think of the rows (or columns) of a two-dimensional array as records, then what has been demonstrated amounts to five basic ways to store records so that selection and traversal are possible. The methods for storing a collection of records are as follows:

1.     Fixed-length records of the same type are stored sequentially in a single array and accessed by calculating record and member offsets.

2.     In the C structure, each array entry is a record, perhaps including unions.

3.     Fixed-length records of the same type are stored in multiarrays, each of appropriate type to match a member type.

4.     Use relative pointer arrays to records stored in an array.

5.     Use pointer variable arrays to records stored in the dynamic memory.

The first approach is enlightening because it makes explicit the offset calculations that are usually provided invisibly by the compiler. A severe disadvantage of this approach is that the representation of different type records causes difficulties--awkward in some languages, and impossible in others.

The second approach is the elegant form of this representation. It is possible in a language that provides the record data structure in which calculations are provided by the compiler. The disadvantage of the first approach then disappears.

The method of multiple arrays is applicable to all languages that provide arrays, whether or not they have strong typing or the record data structure. This method, and the first, sacrifice the ability to refer to an entire record by a single term, as provided by the newer C compilers.

The two pointer-oriented methods work for collections of variable-length records but are convenient only when the number of distinct lengths is small. (The more general case is treated in the next chapter.) Relative pointers are easy to use but have the drawback that embedding records in an array raises the problems of data types clashing.

Pointers to dynamic memory are more difficult and dangerous to use, for reasons mentioned later, but they provide extra flexibility. In any event, storing records in an array or in dynamic memory eliminates the need to declare a variable name for them at the outset of a program; records can thus be created dynamically during program execution.

The array is limited in the number of records it can store by its fixed length, which must be declared prior to program execution. The capacity of dynamic memory can be much larger and need not be declared a priori, but it is determined by the computer system. As will become more apparent, dynamic memory allows more efficient use of available storage.

In solving a problem, the programmer is free to use any data abstraction that is helpful. In deciding which data structure to use for its implementation, one must consider such factors as the possibility of carrying out the required operations, storage efficiency, execution time, and proneness to error. There is often a trade-off between storage efficiency and execution time. For example, pointers use extra storage and require some time to maintain, but they typically give fast access. Developing your appreciation of such trade-offs in programming is one of the goals of this text.

2.5: Case Study: Stable Marriages

To bring together the concepts presented in this chapter, a case study on stable marriages will now be analyzed. The aims of this study are to

1.     Show how a program is designed using the top-down approach

2.     Show that storing the "right" information in the "right" way has an important effect on the design of a program, on the execution time, and on the memory requirements

3.     Demonstrate the concept of data abstraction by treating data and basic operations on it as a unit

4.     Illustrate the treatment of arrays and array indices

Consider Table 2.1. It represents the preferences that each of five men and five women has expressed for the five members of the opposite sex. For example, the first man has ranked the women in the order 2, 4, 3, 1, 5.

Table 2.1 Preferences

       Men's              Women's

------------------------------------

M1  2  4  3  1  5  W1  1  5  3  4  2

M2  3  5  2  1  4  W2  4  5  3  2  1

M3  1  5  4  2  3  W3  3  1  4  2  5

M4  3  1  2  5  4  W4  4  2  5  1  3

M5  2  5  1  4  3  W5  1  3  2  4  5

Now suppose the group has paired off and married, as in Table 2.2. Mi is said to be stable with respect to Wj if there is no other woman preferred by Mi to Wj who, in turn, prefers Mi to her mate. Similarly, Wj is stable with respect to Mi if there is no other man Wj prefers to Mi who prefers Wj to his mate. Any pair (say, Mi-Wj) is stable if each is stable with respect to the other. An entire pairing is stable if all its pairs are stable; an entire pairing is not stable if there exist a man and a woman who are not mates but who prefer each other to their mates.

Table 2.2 Pairings

Men  Women

----------

 1     4

 2     3

 3     1

 4     2

 5     5

Consider the pair M3-W1 of Table 2.2. Since M3 prefers W1 to all others, he is stable with respect to W1. W1 prefers M1 and M5 to M3. M1 is married to W4 and does not prefer W1 to W4. M5 is married to W5 and does not prefer W1 to W5. Thus W1 is also stable with respect to M3. Hence the pair is stable. The entire pairing is not stable. For instance, M2 and W3 are paired, but W3 prefers M4 to her mate, and M4 prefers W3 to his mate, so M2-W3 is not a stable pair. Determining whether or not an entire pairing such as that of Table 2.2 is stable may appear frivolous as stated, but there are other applications. Instead of using the terms men, women, and marriage preferences, we might consider tasks, employees, the employees' preferences for performing the tasks, and the ranking of employees by how well they perform the tasks. Or students might provide their preferences for majors while each department gives its preferences for each student.

2.5.1 The Problem--Stability of an Entire Pairing

Suppose the problem is to write a program to check whether or not a given entire pairing is stable for a given set of preferences. After this is done the problem of generating a stable entire pairing from a set of preferences will be considered. Assume that there are n men and n women.

2.5.2 The Algorithm

To determine whether an entire pairing is stable, the solution must traverse through all the pairs, testing each for stability. The following high-level algorithm does this.

Set stable to true.

For each man, as long as stable is true

set stable to false if the man is not stable with respect to his wife or the wife is not stable with respect to her husband.

The next question is how to determine whether a person is stable. A person is stable if every individual preferred by the person to the person's mate prefers his or her mate to the person. Let stable hold the result of this determination and refine the task:

1.     Set individual to whoever is most preferred by person.

2.     While (stable and individual   [image]  person's mate)

If the individual prefers person to individual's mate, then

set stable to false

else

set individual to whoever is next most preferred by person.

This assumes stable is initially true. Note that the refinement requires

Determining the mate of any person

Choosing the individual most preferred by a person

Determining whether an individual prefers a person to the individual's mate

Finding the individual next most preferred by a person

To follow the dictates of good programming style, the pairings and preference tables should be treated as data abstractions, with their implementation put off until the basic operations on them have become clear. These operations correspond to the four tasks required by the refinement. They will be developed, respectively, as the functions mate, mostpreferred, prefers, and nextpreferred. Before these functions can be written, the implementation of the pairings and the preference data must be decided upon.

Task 1 is to determine a person's mate. This requires the data for the entire pairing, which is given as input to the program and must be read in. However, how should this data be stored? If the mens' pairings are kept in an array mpairs, relating a man to his mate, then the mate of a man can be selected from the array. For the data of Table 2.2, mpairs would be

mpairs

------

  4

  3

  1

  2

  5

Thus the mate of man 3 is woman 1. More generally, the mate of man i is found in mpairs[i], the position of mpairs indexed by i. To find the mate of a woman, however, requires a traversal of mpairs. For example, the mate of woman 1 is found by traversing mpairs until an entry with value 1 is located. The array index of this entry, 3, gives the woman's mate. Rather than spending time traversing whenever a woman's mate is needed, storage can be traded for time. To do this, create another array, wpairs, so selection may be used in this case too. Of course, one traversal will still be necessary to generate wpairs, but this is done just once. The wpairs generated from mpairs would be

wpairs

------

  3

  4

  2

  1

  5

Task 2 is to find out who is most preferred by a person. Suppose the preferences are implemented as two-dimensional arrays mpref and wpref. If the given preferences are those of Table 2.1, then mpref and wpref would be

   mpref           wpref

-----------------------------

2  4  3  1  5   1  5  3  4  2

3  5  2  1  4   4  5  3  2  1

1  5  4  2  3   3  1  4  2  5

3  1  2  5  4   4  2  5  1  3

2  5  1  4  3   1  3  2  4  5

Task 2 is then straightforward and can also be done by selection. In each row of these arrays the first entry is the most preferred mate and the last entry is the least preferred. Thus for man 3, mpref[3][1] contains his most preferred mate.

To perform task 3, determining if an individual prefers a person to the individual's mate, the row for that individual in the proper preference array (mpref for a man and wpref for a woman) may be traversed until either the person or the individual's mate is encountered. If the person is encountered first, prefers must return true, and false otherwise.

Finally, task 4, finding the next preferred individual, can be done by using an index to point to the current individual and, after increasing the index by 1, selecting the (person, index) entry from the appropriate preference array.

The task, determining whether a person is stable, can now be further refined.

1.     Set individual to mostpreferred (person, pref).

2.     Set index to 1.

3.     While (stable and individual   [image]  person's mate)

a.     set individual's mate to mate (individual, pairs),

b.     if prefers (individual, person, individual's mate, pref), then

set stable to false

else

i.     Increase index by 1,

ii.     Set individual to nextpreferred (person, index, pref1).

This assumes that the person's mate is initially set correctly. The implementation for the functions used follows.

    >Comment

mate(person,pairs)

/* Returns person's mate

as specified by pairs.

*/

int person;

pairings pairs;

{

    >Comment

return(pairs[person]);

}

    >Comment

mostpreferred(person,pref)

/* Returns the mate most

preferred by person as

specified by pref.

*/

int person;

preferences pref;

{

    >Comment

return(pref[person][1]);

}

    >Comment

prefers(individual,person,individualsmate,pref)

/* Returns true only if individual prefers person

to the individual's mate as specified by pref.

*/

int individual,person,individualsmate;

preferences pref;

{

    >Comment

int index,next;

index = 1;

next = pref[individual][1];

    >Comment

while(next != person && next != individualsmate)

{

index++;

next = pref[individual][index];

}

if(next == person)

return (TRUE);

else

return(FALSE);

}

    >Comment

nextpreferred(person,index,pref)

/* Returns the mate next most

preferred by person as specified

by pref.

*/

int person,index;

preferences pref;

{

    >Comment

return(pref[person][index]);

}

To get an idea of the time required for this solution, note that the while loop used in determining stable can require a traversal through a row of pref. This loop may thus be executed (n - 1) times, since rows of pref are of length n (recall that there are n men and n women). During each of these executions, prefers is invoked and can traverse through a row of pref, which is also of length n. The determination of stable can thus take O(n2) time. Since stable may need to be calculated 2n times (twice for each pair), the total time of this solution could be O(n3). We can do better!

The traversal through pref in prefers can be eliminated. It is done simply to determine if the current individual prefers person to individual's mate, and this information can be ferreted out beforehand from pref and summarized in a priority array. This needs to be done only once. Thus the priority array can be produced before the algorithm is invoked, or can be created initially in the main function. One priority array must be created from mpref and another from wpref. The mpriority array can be created from mpref by traversing mpref and processing its (i, j)th entry by setting mpriority [i] [mpref [i] [j]] to j. Similarly, wpriority can be obtained from wpref. Then an individual prefers person to individual's mate if

priority [individual][person] < priority [individual][individual's mate]

For our example, these arrays are as follows:

  mpriority       wpriority

-----------------------------

4  1  3  2  5   1  5  3  4  2

4  3  1  5  2   5  4  3  1  2

1  4  5  3  2   2  4  1  3  5

2  3  1  5  4   4  2  5  1  3

3  1  5  4  2   1  3  2  4  5

This version of prefers may be expressed as follows:

    >Comment

prefers(individual,person,individualsmate,priority);

/* Returns true only if individual prefers person

to the individual's mate as specified by priority.

*/

int individual,person,individualsmate;

preferences priority;

{

    >Comment

return(priority[individual][person]<

priority[individual][individualsmate]);

}

When person is male, the parameter priority must be wpriority, and when person is female, priority must be mpriority. The priority arrays result in O(n) time for determining stable, since one traversal is avoided. This reduces the total time of the solution to O(n2). Again, storage has been traded for time. Extracting precisely the relevant information and implementing it properly, just as was done in Example 2.2, is what produced this time saving. Also, this version is clearer and more concise.

2.5.3 The Program

The algorithm, expressed as a function stabilitycheck, is implemented in the following program. The program first inputs n. Next it reads in mpairs, creates wpairs from it, reads in mpref and wpref, then creates mpriority and wpriority, calls stabilitycheck to check the pairings, and finally prints the result along with the pairings.

#include <stdio.h>

    >Comment

#define MAXN 50

typedef int pairings[MAXN];

mate(person,pairs)

/* Returns person's mate

as specified by pairs.

*/

int person;

pairings pairs;

{

return(pairs[person]);

}

    >Comment

read_mens_mates(n,pairs)

/* Reads the n mates of the

men into pairs.

*/

int n;

pairings pairs;

{

int i;

printf("\n mpairs = ? \n");

for(i=1;i<=n;i++)

scanf("%d",&pairs[i]);

}

    >Comment

create_womens_mates(n,pairs1,pairs2)

/* Creates in pairs2 the women's mates

based on the men's mates as specified

by pairs1.

*/

int n;

pairings pairs1,pairs2;

{

int i,j;

for(i=1;i<=n;i++)

{

for(j=1;pairs1[j] != i;j++)

{

}

pairs2[i] = j;

}

}

    >Comment

typedef int preferences[MAXN] [MAXN];

mostpreferred(person,pref)

/* Returns the mate most

preferred by person as

specified by pref.

*/

int person;

preferences pref;

{

return(pref[person][1]);

}

nextpreferred(person,index,pref)

/* Returns the mate next most

preferred by person as specified

by pref.

*/

int person,index;

preferences pref;

{

return (pref[person][index]);

}

prefers(individual,person, individualsmate,priority)

/* Returns true only if individual prefers person

to the individual's mate as specified by priority.

*/

int individual,person,individualsmate;

preferences priority;

{

return(priority[individual][person]

< priority[individual][individualsmate]);

}

read_preferences(n,pref)

/* Reads n rows of

preferences into pref.

*/

int n;

preferences pref;

{

int i,j;

for(i=1;i<=n;i++)

for(j=1;j<=n;j++)

scanf("%d",&pref[i][j]);

}

    >Comment

create_priorities(n,pref,priority)

/* Creates n individual priorities in

priority based on the individual

preferences specified by pref.

*/

int n;

preferences pref,priority;

{

int i,j;

for(i=1;i<=n;i++)

for(j=1;j<=n;j++)

priority[i][pref[i][j]] = j;

}

#define TRUE 1

#define FALSE 0

main()

{

    >Comment

pairings mpairs,wpairs;

preferences mpref,wpref,mpriority,wpriority;

int i,j,n;

    >Comment

printf("\n n=?\n);

scanf("%d",&n);

    >Comment

read_mens_mates(n,mpairs);

    >Comment

create_womens_mates(n,mpairs,wpairs);

printf("\n Enter mens preferences \n");

    >Comment

read_preferences (n,mpref);

printf("\n Enter womens

preferences \n");

    >Comment

read_preferences (n,wpref);

    >Comment

create_priorities(n,mpref,mpriority);

    >Comment

create_priorities(n,wpref,wpriority);

    >Comment

if(stabilitycheck(n,mpairs,wpairs,mpref,wpref,

mpriority,wpriority))

printf("The pairings are stable \n");

else

printf("The pairings are not stable \n");

printf("The pairings are: \n");

printf("man wife \n");

for(i=1;i<=n;i++)

printf("%d %d \n",i,mate(i,mpairs));

}

stabilitycheck(n,mpairs,wpairs,mpref,wpref,mpriority,wpriority)

/* Checks the entire pairing specified by

the parameters and returns true only

if it is stable.

*/

int n;

pairings mpairs,wpairs;

preferences mpref,wpref,mpriority,wpriority;

{

int man,wife,individual,individualsmate,index,stable;

stable = TRUE;

    >Comment

for(man=1;(stable && man<=n);man++)

{

wife = mate(man,mpairs);

individual = mostpreferred(man,mpref);

index = 1;

    >Comment

while(stable &&(individual != wife))

{

individualsmate = mate(individual,

wpairs);

if(prefers(individual,man,

individualsmate,wpriority))

stable = FALSE;

else

{

index++;

individual = nextpreferred(man,

index,mpref);

}

}

individual = mostpreferred(wife,wpref);

index = 1;

    >Comment

while(stable &&(individual != man))

{

individualsmate = mate(individual,

mpairs);

if(prefers(individual,wife,

individualsmate,mpriority))

stable = FALSE;

else

{

index++;

individual = nextpreferred(wife,

index,wpref);

}

}

}

return(stable);

}

2.5.4 Review of the Program's Development

At this point let's review the process followed in creating the program. First an algorithm was conceived and refined, moving from the original problem to the subtasks into which it was broken down: the four tasks for operating on the data abstractions, plus the task of determining stable. The refinement for this last task was specified, treating the pairings and preferences as data abstractions. Next the functions embodying the operations on the data abstractions were specified. In doing so, the trade-off between storage (adding the priority arrays) and execution time (requiring a traversal of rows of mpref or wpref if there were no priority arrays) came into play. This led to the "right" information to be stored and the "right" choice of data structure in which to store it--the priority arrays.

Finally, the solution was packaged into a main program that (1) created and input data to the data structures chosen; (2) called stabilitycheck to check the entire pairing. The functions main and stabilitycheck were written, treating the pairs and preferences as data abstractions. As a result, should their implementations need to be changed, the changes are localized to the data structure definitions and the functions operating on them.

2.5.5 Generating Stable Pairings

As explained earlier, the program only checks a particular entire pairing for stability. It does not generate, for given preferences, an entire pairing that is stable. The task of generating such a stable entire pairing is known as the stable marriage problem.

Some simple cases allow a stable entire pairing to be written by inspection:

1. People have a unique way of looking at the world. This case comes up when the first column of mpref has n distinct entries: every man most prefers a different woman. Simply pair each man with his heart's desire. The case where wpref has this property is similar.

2. People are all the same. This case occurs when the rows of mpref are all exactly alike: every man has exactly the same preferences. Pair the most preferred woman with the man she most prefers. Pair the next most preferred woman with the man she most prefers among those not yet paired, and so on. For example:

   mpref      wpref

----------------------

2  4  3  1  3  1  2  4

2  4  3  1  3  1  4  2

2  4  3  1  4  2  1  3

2  4  3  1  4  3  2  1

Pair W2 with M3, W4 with M4, W3 with M2, and W1 with M1. The case where wpref has this property has a similar solution.

Not only is an algorithm for generating a stable entire pairing not apparent; it is not even evident that one always exists. The following algorithm does construct one; you might try your hand at convincing yourself that it always does so.

1.     Set man to 1.

2.     While (man is not paired)

a.     find the woman most preferred by man who is not yet paired, or, if paired, prefers man to her current mate

b.     if the woman is not yet paired, then

i.     pair man to woman,

ii.     set man to an unpaired man if there is one

else

i.     break the pair between the woman and her current mate,

ii.     pair woman to man,

iii.     set man to her ex-mate

To illustrate this algorithm, apply it to the preferences of Table 2.1. The sequence of pairs generated by the algorithm is as shown in Table 2.3. Asterisks indicate conflicts that cause breaks when they occur. The final pairings are then 3-1, 4-3, 2-5, 5-2, 1-4.

This is a good example of a correct algorithm that surely requires both a proof of correctness (produces the proper result) and a proof that it will eventually stop. These are the essential ingredients of an algorithm. (The proofs are left as exercises.)

Table 2.3 Sequence of Pairs

     Man  Active Pairs

--------------------------

      1    1         2

  1   2    2         3*

      3    3         1

      4    4         3*

--------------------------

           1         2*

  2        3         1

           4         3

      2    2         5

      5    5         2*

--------------------------

           3         1

           4         3

  3        2         5

           5         2

      1    1         4

Exercises

1.     a.     Do users of the subway system of a large city reach their destinations by a process analogous to selection or a process analogous to traversal? Why?

b.     Do users of the telephone system of a large city reach their parties by a process analogous to selection or one like traversal? Why?

2.     Suppose you are interested in processing English text in the following way:

a.     Input is in the form of an English sentence.

b.     The program is to check the sentence looking for violations of the spelling rule, "'i'" before 'e' except after 'c'." (Do not worry about exceptions such as neighbor, weigh, etc.) Would an array be a convenient data structure to use in your program, and would traversal be a relevant process?

3.     Write a program to print out how many elements of an integer array (of length n) contain even integers. Did you use the process of traversal?

4.     In Example 2.2 suppose the pointer array were stored in memory and there were no a itself. If you are told that a[5] and a[9] have been interchanged, then what changes would you have to make to pointer to reflect this interchange? What if a[2] and a[5] were interchanged?

5.     Write a function to update the pointer array of Example 2.2 to reflect the fact that the ith and jth columns of a have been interchanged.

6.     a.     Suppose the cars array (Example 2.4) represents the fleet information for the cars of one rental agency. For a group of several such agencies, would the record data structure be convenient to represent all the information?

b.     Assume the information is represented in an array that stores records, with each record containing all the information for a particular agency. In general, are these records fixed or variable length?

7.     For Example 2.4 assume the alternate multiarray implementation. Write a function to interchange the ith and jth records.

8.     Create a function that prints out the rentees of all cars that have been driven more than 50,000 miles for the multiarray implementation of Example 2.4.

9.     Write a function to read in a sequence of at most twenty characters into an array. They are all digits (0, or 1, or 2, . . . , or 9) except for one character that will be a decimal point. After execution of the function the variable dp should point to the array element that contains the decimal point, and variable 1 should point to the last digit of the input sequence. The decimal point will always have at least one digit on each side.

10.     Produce a function similar to that of Exercise 2.3 except that the array of integers is two-dimensional.

11.     Suppose a checkerboard is represented as a two-dimensional array. Write a function that is given the current configuration of the board and the move to be made by the player whose turn it is. The function is to return the value true if the move is legal and false otherwise. (Do not forget that a move can consist of many jumps.)

12.     a.     Is there an analogy between tabs in your home telephone directory and pointers?

b.     If a song on a tape cassette is like a record data structure type, can a particular song be indexed, or must a traverse be used to find it? How about a song on a phonograph record?

13.     Explain the differences between a record stored in an array and in dynamic memory.

14.     Define a function to copy the array of Example 2.5 into an array indexed by a pointer array of integers.

15.     Write a function to interchange two records of the array of Example 2.5.

16.     Assume the row representation for two-dimensional arrays has been extended to three-dimensional arrays. Determine the formula for the offset of a[i][j][k].

17.     The array a with r rows and c columns is stored using the rowwise representation, with a[0][0] corresponding to the actual base location of a, abase. Given an integer number k (abase   [image]  k   [image]  abase + r X (c - 1)), write formulas that will yield the row and column of a to which actual location k corresponds. That is, if a [i] [j] is stored in k, then your formula for the row will yield i and for the column will yield j.

18.     An antisymmetric array a is a square array that satisfies the condition that a[i][j] = -a[j][i] for all 0   [image]  i, j < n. Hence the diagonal elements must be zero. Storing the antisymmetric array can be done just as storing the symmetric array was done, except that now the diagonal entries can be omitted, since they are known. Determine the formula for the offset of a[i] [j].

19.     An array a is said to be tridiagonal if the nonzero entries of a fall along the three diagonals a[i] [i], a[i] [i+1], and a[i] [i-1], where i goes, respectively, from 0 to n - 1, 0 to n - 2, and 1 to n - 1.

a.     Construct a rowwise representation of the nonzero entries of a, and find a formula for the offset of a[i] [j] in this array.

b.     Construct a diagonal representation of the nonzero entries of a tridiagonal array, and find a formula for the offset of a[i] [j] in the array.

20.     Write a function that takes a two-dimensional array represented in terms of  p and data, as in Section 2.3.3, and changes p so that any duplicate rows of a are stored only once in data. Thus, if rows 2 and 7 are identical, the contents of p[1] and p[6] are made equal.

21.     Define a function to output the n diagonal elements of the array a of Section 2.3.3 when the array is represented in terms of p and data.

22.     Produce a function to print out the jth column of array a when it is represented in terms of p and data as in Section 2.3.3.

23.     Write a function to interchange a[i][j] and a [j][i], for all i, j, when a is represented in terms of p and data as in Section 2.3.3.

24.     Create a function that takes any array a of integers with at most twenty entries and creates a pointer array p that has in p[i] a pointer to the (i + l)th largest entry of a. Thus p[0] points to the largest entry of a.

25.     Do the same as in Exercise 24, except the array a will be two-dimensional and the ith record in p must contain two pointers, one for the row and one for the column of a's ith largest entry.

26.     Do Exercises 20 to 25, except use pointers and dynamic memory. Print the pointers.

27.     What pairs are not stable in the example pairings?

28.     Give a function to create wpairs from mpairs.

29.     Write a function to create a priority array, given a pref array.

30.     What function will create a pref array given a priority array?

31.     a.     Give a function to produce stable marriages.

b.     How much time does your answer to Exercise 31a require?

32.     a.     Why must the algorithm to produce stable marriages terminate eventually?

b.     Why does it produce stable marriages?

Suggested Assignments

1.     a.     Implement the collection data abstraction of Chapter 1 using an array to store the actual integers in the collection. That is, implement the operations set, insert, omit, and belongs.

b.     Implement a collection c of carrecords as a data abstraction with the operations set (c), insert (p,c), omit (i,c), and printrecord (i,c). Use the carrecords defined in Section 2.1.3 but add an additional field for an identification number for each car. Then, insert (p,c) adds the carrecord pointed to by p to the collection c, omit (i,c) deletes the carrecord with the identifier i from c, and then printrecord (i,c) prints the contents of the carrecord with identifier i.

c.     This is the same as (b), except the records are to be stored in dynamic memory with a pointer array used to contain pointers to them.

2.     Suppose the amount of precipitation has been recorded over a region at n locations. In general each record consists of one to five numbers. You are to write and run six programs. Each is to produce the average precipitation. For example, if the input is

3.6

2.1

2.7

record sentinel

.67

1.2

record sentinel

1.9

record sentinel

final sentinel

the output should be the sum [(3.6 + 2.1 + 2.7) + (.67 + 1.2) + 1.9] divided by 6. Always echo print the input (that is, output the input as it is read in), and annotate all output so that you, or anyone, will know what the output means. Assume n   [image]  100.

a.     Two of the programs should assume all records are of the same length (say, 3). Program 1 will represent the records as fixed-length homogeneous records stored in a single array (method 1 of Section 2.4). Program 3 will represent them as fixed-length records stored in multiarrays (method 3 of Section 2.4). For method 3, assume an additional record field giving the specific location where it was recorded.

b.     Three of the programs should assume the general case of C structures that contain a union and store them in an array (method 2 of Section 2.4). Program 4 will use a relative pointer array whose entries point to records stored in an array (method 4 of Section 2.4). The fifth program uses pointer variable arrays whose entries point to records stored in dynamic memory. For methods 4 and 5, output the contents of the pointer arrays and the records to which each pointer points. Your programs 4 and 5 should work correctly no matter where the records are stored in the one-dimensional array.

c.     Finally, one program should assume all records are of length 1 and represent entries in an n X n symmetric array. This program uses the symmetric array representation of Example 2.6. No two-dimensional array should be used in your solution.

The input and output for each program might appear as follows:

                Input                                Output

------------------------------------------------------------------------------

Programs A  1.    3.6            Echo print of the precipitation records:

                  2.1                3.6, 2.1, 2.7

                  2.7                .67, 1.2, 1.9

                  RS             The average precipitation of the 6 values is

                 .67             2.028 inches. The number of records is 2.

                  1.2

                  1.9

                  RS

                  FS

            3. White House       Echo print of the precipitation records:

                  3.6                White House 3.6, 2.1, 2.7

                  2.1                Lincoln Memorial .67, 1.2 , 1.9

                  2.7            The average precipitation of the 6 values is

                  RS             2.028 inches. The number of records is 2.

               Lincoln Memorial

                  .67

                  1.2

                  1.9

                  RS

                  FS

                Input                                 Output

-----------------------------------------------------------------------------

Programs B  2.    3.6             Echo print of the precipitation records:

                  2.1                3.6, 2.1, 2.7

                  2.7                .67, 1.2

                  RS                 1.9

                  1.2

                  RS              The average precipitation of the 6 values is

                  1.9             2.028 inches. The number of records is 3.

                  RS

                  FS

            4.    3.6             Echo print of the precipitation records:

                  2.1                3.6, 2.1, 2.7

                  2.7                .67, 1.2

                  RS                 1.9

                 .67              Pointer array  Data array

                  1.2                1    10     10    3.6

                  RS                 2     1     11    2.1

                  1.9                3    16     12    2.7

                  RS

                  FS                              1     .67

                                                  2    1.2

                                                 16    1.9

                                  -------------------------

                                  The average precipitation of the 6 values is

                                  2.028 inches. The number of records is 3.

            5. This is the same   Records in dynamic storage

               as for 4, except      3.6      2.1   2.7

               it will print the      .67     1.2

               pointer array         1.9

               (that is, the

               actual pointers).

Programs C     3.6   2.1  2.7     Echo print of the precipitation records as a

                .67  1.2  1.9     3 X 3 array:

               0.0   1.1   .15        3.6     2.1   2.7

                                       .67    1.2   1.9

                                      0.0     1.1    .15

                                  The average precipitation of the 9 values is

                                  2.028 inches.

3.     Modify the program of Section 2.5.2 so that its input consists of the men's and women's preferences and then a series of entire pairings. For each of the pairings, it is to call stabilitycheck and print the result for the pairing.

Go to Chapter 3      Return to Table of Contents


Data Structures Using C: CHAPTER 3: LISTS

 Return to Table of Contents

CHAPTER 3: LISTS

Presents the list data structure for

efficient traversals

arbitrary insertions and deletions

imposing an ordering on the collection of information it holds

Discusses the limitations of lists for random access

Compares the advantages and disadvantages of lists and arrays for storing data

Discusses the implementation of lists using

records

arrays

pointers

dynamic memory

Stresses the importance of writing functional modules to solve general programming problems and

uses list traversal to convey these techniques

shows how such functional modules can be used as tools and adapted to solve related problems

Explains dynamic storage and its management

Case study--the perfect shuffle

illustrates the use of lists and arrays

3.1: Why Are Lists Needed?

Imagine you are a financial wizard and want to store information about your stock portfolio as records. Table 3.1 lists the information. How should these records be stored? The book so far has shown how to store them in

Individual storage elements with individual names

An array, so they can be referenced by position or from a pointer array

Dynamic memory, where they can be referenced by individual pointer variables or from an array of pointer variables

Having more than a few records rules out using individual names, a method that would also be inadequate because it does not allow the records to be treated as a group. The same is true for individual pointer variables.

The other three methods, two kinds of arrays and dynamic memory, allow not only grouping but also selection of records, as well as traversal through all the records. Figure 3.1 represents storage of the stock portfolio in each of these three ways. The three methods are called (a) sequential arrays, (b) sequential pointer arrays, and (c) sequential pointer variable arrays. Collectively they are known as sequential array methods for record storage.

The sequential array methods are convenient for recording the daily transactions common in the buying and selling of stocks, but only when records are kept in arbitrary order. A new record can then be inserted directly as the new last record. This means that in the method of sequential arrays (Figure 3.1(a)), a new record is inserted after Sears. In the method of sequential pointer arrays (Figure 3.1(b)), it can be placed in any unused slot, but the corresponding pointer goes in the seventh position of p. For the method of sequential pointer variable arrays (Figure 3.1(c)), the function malloc provides dynamic storage that is available to store the record, but again, its associated pointer variable value also goes in position 7 of p.

Table 3.1 A Stock Portfolio

Name     # Shares    Value     Date Bought

Apple      300     $ 7,512.46    3/15/84

CBS        100       8,000.55    6/16/83

Digital    200      18,400.00    9/20/83

IBM        100      11,213.25    1/20/83

IBM        200      21,400.00    4/17/84

Sears      100       3,100.00    2/10/82

Sears      100       3,200.00     4/6/84

Usually order does matter. In reality the stocks are kept in alphabetical order. In this case the sequential array methods provide neither easy nor rapid insertion or deletion capability for the records. Suppose Chrysler stock is bought, and its new record must be inserted into the portfolio; Chrysler must be inserted in its proper place as the new third record. In sequential arrays this means that all five records below its new position must be moved down. In the two pointer array methods, this means that the five pointers below its new location (position 2) must be moved down. The pointer array methods afford flexibility of record storage position, but they still require shifting, although it is pointers that are moved rather than records. Shifting takes time proportional to the number of records moved. If n records are to be inserted, then the total time can be proportional, not to n, but to n2. For example, suppose the n records each require insertion at the top. Even if there are no records stored initially, insertion of n records requires 0 + 1 + 2 + . . . + (n - 1) moves. The time adds up to n(n - 1)/2, and so the time is O(n2). Even if each insertion requires only half the information to be shifted, the time is reduced merely by a factor of 2; it is still O(n2). When n is large, this can take significant time. Of course, for one trader, n will be small, but for larger brokerage houses n might be hundreds of thousands. If they were not designed to be efficient, these insertions might take hours or even days ((300,000)2 operations X 10-6 seconds per operation = 25 hours).

(a) Sequential Array

(b) Sequential Pointer Array

(c) Sequential Pointer Variable Array

Figure 3.1 Sequential Methods of Storing Records in Arrays

Selling a stock, say Digital, requires that its record be deleted in the sequential array (a) or that its pointer entries be set to - 1 or null in the pointer arrays (b) and (c). Either course would leave gaps of unused storage, which is undesirable. Leaving gaps soon uses up all the slots in an array, and unless wasting storage is tolerable and affordable, this is impractical. Instead, the gap would be filled by moving up all the records below a deleted record. But again, this could take O(n2) time for the deletion of n records.

What is needed is a way to store records so that their number can grow or shrink in response to an arbitrary number of insertions and deletions, in such a way that storage is not wasted and the time for insertion and deletion is reduced. The list data structure introduced in this chapter provides one solution to this problem. A list is a collection of records linked by pointers. Even with lists, it is still necessary, as with the sequential array methods, for the program to spend time deciding where to make an insertion or deletion in the first place. To make these operations efficient when large numbers of records are involved requires more advanced data structures (introduced later). For a moderate number of records, the time taken to locate the point of insertion or deletion is tolerable. Lists called stacks and queues, where insertion and deletion occur only at the beginning and end of the lists, are especially useful even when there are many records. These will be considered in the next chapter.

Chapter 2 showed that the key advantage of the sequential array methods is selection in constant time. As just shown in this chapter, the disadvantage is that inserting or deleting entries from the interior of an array is costly. Another drawback is that arrays have fixed lengths that must be declared to the compiler. Thus, the maximum number of items to be stored in the array must be known in advance of execution. No such a priori limit need be declared for the number of items on a list. Lists provide a way to insert new items or to delete existing ones at a known point of the list, in constant time. However, selection of an arbitrary ith list item will take time proportional to i.

3.2: Keeping Track of List Pointers

The pointers in lists are what is manipulated by programs, because lists are collections of records linked by pointers. A special element, called the head or name of the list, points to the beginning of the list, and each record in the list has a pointer to the next one. The final record, which has no successor, has an empty link, denoted by a special pointer value called the null pointer.

Figure 3.2 Graphic Representation of the List STOCKS

The stock portfolio example is represented in a list in Figure 3.2. The variable labeled head contains a pointer to the first record in the list. Each record is composed of two fields: 1) an information field, which may itself be made up of subfields such as name, shares, value, datebought (or accountnumber, balance, address); and 2) a link field, which contains the pointer to the next record. The x in the link field of the last record denotes the null pointer and indicates that this is the last record on the list. The list with no records is called the null list, and is denoted by a head containing the null pointer.

It does no good to know that a phone number (say, 642-3715) contains the digits 1, 2, 3, 4, 5, 6, and 7, unless the order 6, 4, 2, 3, 7, 1, 5 is retained as well. Notice that the list data structure, besides being a new way to store information, allows the information to be remembered in a specific order--the order in which the records appear on the list. Arrays provide this ability as well. The order in which records (or the pointers to them) are stored does this, but the operations that make lists interesting and powerful are insertion and deletion.

In effect, a list works as if the pointers from a pointer array were incorporated into the records to which they point. This is the crucial advantage of lists compared to arrays. The price of the added convenience is the loss of the capability to select an arbitrary record. Instead of access to the list records being through the pointer array, the list records must now be accessed via the pointer to the first list record (in head or name). Knowing only the head of a list means that only the first record is immediately visible. To access another list record, the chain of pointers linking list records must be followed until the desired record is encountered.

Consider the simple list l in Figure 3.3(a). Two situations are depicted in Figure 3.3(b,c): (b) is the case when the list head l is known, and (c) the case when a pointer, ptr, to a list record is known. In each case only the record to which the known pointer points is visible.

When only one pointer points to a record, if that pointer value is changed, then the record cannot be accessed. It is as if the programmer had stored something somewhere but had no hope of remembering where. If that pointer happens to be in the head of a list, then access to the list is lost. Suppose the programmer must start with the situation depicted in Figure 3.3(a) and must create two lists. One, l, is to consist of all records of the original list, from the record pointed to by ptr to the last record. Another, newl, is to consist of the original list records, from the first to the record prior to that pointed to by ptr. Simply copying the value of ptr into l creates the correct modified version of l. However, newl cannot be created since the pointer to the original first record that had been stored in l has been lost. Such a predicament must be avoided.

Figure 3.3 Simple Lists

Keeping track of pointers in processing lists is extremely important. One reason is that more than one pointer can point to a record. The only way to see if two pointers point to the same record is to see if their pointer values are equal.

3.2.1 Insertion and Deletion of Records in Lists

Consider the effects of inserting and deleting on a conceptual list. The effect of inserting a new record is shown in Figure 3.4. The new record is inserted between the records labeled predecessor and successor. This requires changing predecessor's link to point to the new record, and setting the new record's link to point to successor.

Figure 3.4 Insertion of a New Record in a Conceptual List

Figure 3.5 Deletion of a Record from a Conceptual List

The deletion of a record from a list is shown in Figure 3.5. The record labeled removed is deleted from the list by arranging predecessor's link to point to successor. The rearrangement of links for insertion or deletion is independent of where in the list the operation occurs. Each operation can be performed in constant time. Lists, unlike arrays, readily accommodate change; they exhibit no growing pains (or contraction pains).

3.3: Expanding and Contracting Lists

3.3.1 Insertion

Suppose you wish to insert a new record for CHRYSLER. It is always a good idea to sketch an image like Figure 3.6(a) to show both the situation at the outset and the situation desired after the task is carried out. Since only the head of the list, stocks, is assumed known, normally each record of the list must be accessed in turn, starting from the first, in order to determine where to make the insertion. For now, assume there is a given pointer predecessor that points to the record after which the new record is to be inserted. In the example, this is the second record. The value of its link field must be copied into the link field of the new record, so the new record's link field points to the successor of the predecessor record. The link field of the predecessor record must end up pointing to the new record, so the value of newpointer must be copied into the predecessor's link field. The code to accomplish this may be written as

setlink(newpointer,next(predecessor));

setlink(predecessor,newpointer);

where setlink is a function that copies the value of its second parameter into the link field of the record pointed to by its first parameter, and next is a function whose value is a copy of the link field value of the record pointed to by its parameter. Why is it incorrect to interchange these two statements?

Notice that no assumptions have been made about the implementation of the list. The code is thus independent of how the list is implemented. Of course, setlink can be written only when the implementation is known, and it will depend on the details. However, this code will not. The list is thus a data abstraction with insertion as one of its operations; deletion is another.

Figure 3.6 Insertion of a New Record (a) in the Interior of a List and (b) at the Front of a List

3.3.2 Special Cases

It is evident that this code is correct for insertion into the interior of the list or at its end. Special cases must always be sniffed out and dealt with. By definition, a special case means a situation requiring code different from the current solution.The existing solution must be modified to handle special cases. The special case here, shown in Figure 3.6(b), represents insertion at the front of the list. Obviously there is no predecessor record, so the solution cannot properly handle this case. It is reasonable to assume that this special condition is represented by a null value for predecessor. Then it is the value of head that must be copied into the link field of the new record. Also, it is the head itself that must receive the copy of newpointer and end up pointing to the new record. The correct function to handle all situations is as follows:

>Comment

insert(phead,predecessor,newpointer)

/* Inserts, in list head, the record pointed to

by newpointer as the successor of the

record pointed to by predecessor.

*/

listpointer *phead,predecessor,newpointer;

{

listpointer setnull(),next();

if(predecessor == setnull())

{

>Comment

setlink(newpointer,*phead);

*phead = newpointer;

}

else

{

>Comment

setlink(newpointer,next(predecessor));

setlink(predecessor,newpointer);

}

}

Note: listpointer is a type which is explained in Sections 3.6.1 and 3.6.2.

where setnull returns the null value. What is used for the null value is determined by the list implementation (Section 3.6). When insert is invoked, its actual first parameter, corresponding to phead, must be a pointer to the head of the list. Thus to have insert work on a list l, the call would be insert (&1,predecessor,newpointer). Whenever a parameter of a function must return a value (be modified by the function) the convention in this text is to prefix the name of the parameter in the function definition with a p. Thus, since the value of the pointer to head in insert must be changed, it is referred to as phead. Notice that the code for insert refers to *phead. Think of *phead as another name for head. We say that head has been passed to insert by pointer.

This version of insert is independent of how the list is actually implemented. In practice, insert might be made to execute more efficiently by writing it so that its code does depend on the list implementation. Insert might then be the level at which the list implementation is hidden instead of in setnull, setlink, and next. However, the version above is clearer and can be readily modified if desired.

3.3.3 Header Records

An alternative that avoids the special case of inserting a new first record assigns a special record, called a header record, to every list as its first record. This record is a dummy record. The dummy is so called because it is not part of the conceptual list of records being created. Used to make processing easier, the dummy record is never deleted, and no record is ever inserted in front of it. Then the special case never occurs, and the original code, requiring no check for the special case, can be used. Actually the header record can be a smart dummy, used to contain important information about the list, such as the number of records. When this alternative is used, the null list can no longer be recognized by a null value in the list head. It is recognized by a null pointer in the link field of the dummy record.

Figure 3.7 Deletion of a Record (a) from the Interior of a List and (b) from the Front of a List

3.3.4 Deletion

To delete the record following predecessor (Figure 3.7(a)), only the link field value of its successor must be copied into the predecessor's link field:

setlink(predecessor,next(next(predecessor)));

Thus next (next(predecessor)) is the value that must be copied into the predecessor's link field. It is clearer to write the code as

successor = next(predecessor);

setlink(predecessor,next(successor));

even if it is slower, requiring an additional memory access. Again, either code is independent of any implementation for the list. After checking for special cases (Figure 3.7(b)), the correct function is found to be

>Comment

delete(phead,predecessor)

/* Deletes, in list head, the successor of

the record pointed to by predecessor.

*/

listpointer *phead,predecessor;

{

listpointer successor,null,setnull(),next();

null = setnull();

>Comment

if(*phead != null)

if(predecessor == null)

>Comment

*phead = next(*phead);

else

{

>Comment

successor = next(predecessor);

if(successor != null)

setlink(predecessor,next(successor));

}

{

Sometimes, instead of having a pointer to the predecessor of a record to be deleted, only a pointer to the record itself is available. Deletion can still be accomplished; all that is required is to copy the entire successor record (the successor to the record to be deleted, that is) into the record to be deleted. It is necessary to assume, however, that the last list record is never deleted. Why? Whether or not this is better than keeping track of the predecessor depends on how much time is required to effect this copying operation. The larger the record, the greater the time. However, if the information field of a record is kept elsewhere, and a pointer to it stored in the record itself, then the operation reduces to copying just this pointer and the value of the link field.

Are you listing from all this list information?

3.4: Traversal of Lists

Traversing a list's records is another operation performed on the data abstraction called the list. Traversal is useful when the records are to be processed one after another in the same order in which they appear in the list. A traversal may be accomplished by following the pointer in the list's head to the first record and processing it, then following the first record's link field pointer to the next record and processing it, and so on, until the last record has been processed.

3.4.1 List Reversal Using a Loop

Example 3.1

Consider the list l in Figure 3.8(a). The task is to write a function reverse, to reverse such a list. This means that when reverse is applied to l, the result should be as shown in Figure 3.8(b).

At first glance, this may seem to be a complicated task involving a great deal of record and pointer manipulation. However, there is a simple solution: the information fields of each record need not be considered at all. Instead, as in Figure 3.8(c), the link field of F can be set to null, the link field of O can be set to point to F, the link field of R can be set to point to O, and so on. Finally, l itself can be set to point to D. The result would be as sketched in Figure 3.8(c), the correct reversed list. This procedure can be stated more generally:

1. Set the link field of the first record to null.

2. Set the link field of each record to point to its original predecessor.

3. Finally, set the list head to point to the original last record.

Even without the details of implementing the list and carrying out the procedure,it is easy to be certain that this procedure, when successfully carried out, does solve the problem. This is so except for the case of a null list; the procedure does not specify what is to be done, since there is no first or last record in a null list. Since the first record of any list has no predecessor, the procedure can be specified somewhat differently.

(a) The Original List

(b) The Reversed List

(c) The Procedure for Reversing the List

Figure 3.8 A List before, after, and during Reversal

1. Set the link field of each record to point to its original predecessor.

2. Then set the list head to point to the original last record, or to predecessor if the list is null.

Now we have a procedure that is correct for all cases, the first record's predecessor being assumed null.

The next step is to detail how this procedure is actually to be done. The basic structure involved is a loop, since the same process is to be carried out repeatedly, but on a different record each time. In general, loops need a beginning (initialization) and an end (finalization). The finalization here is specified explicitly in the procedure: namely, set the list head to point to the original last record, or to predecessor if the list is null. The initialization, as usual, cannot be specified until the loop is specified in more detail. Refining task 1 makes the algorithm read as follows:

1. Initialization

While (there is a current record)

a. set the link field of the current record to point to its original predecessor, and

b. update the current record.

2. Set the head of the list to point to the original last record, or to predecessor if the list is null.

To implement the loop requires a pointer (call it recordpointer) to keep track of the current record to be processed within the loop by its loop task. Imagine that the first three records have already been processed correctly and the fourth is about to be processed. It is pointed to by recordpointer, as depicted in Figure 3.9(a).

Figure 3.9 Achieving Proper Loop Implementation

Processing W requires setting its link field to point to W's predecessor R. To do this requires knowing where its predecessor R is located. Thus another pointer, predecessor, is needed. It is so named because it will contain a pointer to the predecessor of the current record to be processed. (See Figure 3.9(b).)

Suppose someone attempts to process W now, copying predecessor into W's nextpointer field to set it properly. The result is shown in Figure 3.9(c).

Predecessor and recordpointer must now be updated so that they point to W and A, respectively. Then A can be processed next when the loop is repeated. However, the link field of W no longer points to A, but instead to R. Thus the location of A has been lost. Care is required when changing pointer values so that such important information is not lost. To remedy this, introduce another pointer hold in which the pointer to A can be saved before changing W's link field value. The processing of the record pointed to by recordpointer may now be done by sequentially

1. copying its link field value into hold,

2. copying predecessor into recordpointer's link field

3. copying hold into recordpointer.

This produces the result depicted in Figure 3.9(d).

The test or condition for another repetition of the loop becomes (recordpointer ! = null). The initialization involves setting recordpointer to 1, to make it point to the first record, and setting predecessor to null. Why is it unnecessary to initialize hold?

When the loop is exited, predecessor will point to the original last record, and the finalization can be completed by copying predecessor into l. Refining again:

1. Set predecessor to null.

Set recordpointer to l.

While (recordpointer is not null),

a. set hold to the link field value of the record pointed to by recordpointer, and

set recordpointer's link field to predecessor,

b. set predecessor to recordpointer, and

set recordpointer to hold.

2. Set l to predecessor.

This algorithm actually involves a complete traversal through the records of the list l. As it is carried out, recordpointer accesses each record of l in turn and processes it within the while loop. The processing is done by the loop task. In order for the loop to work correctly, some initialization had to be done, and some finalization was required after each record of the list had been processed.

3.4.2 A General List Traversal Using a Loop

Our intention now is to write a general function traverse, to carry out a traversal of a list, given its head, listname. The code for traverse should be independent of the list implementation, as was the code for insertion and deletion. To make traverse independent of the particular application as well, theprocessing to be done on each record will be functionally modularized in a function process. Its details are thus irrelevant; simply invoke process within traverse. However, to do its job, this function may need access to the list head, listname. It will surely need to know what record it is currently to process; recordprinter will be used to point to this record.

Often, the traversal must proceed through all the list records. To allow for the possibility of a partial traversal, however, a variable done can be introduced. Done may be set by process so that the traversal does not continue beyond the current record. Finally, process cannot retain any information between calls unless that information is kept nonlocal to it, is passed as one of its parameters, or is a static variable. So process is given the parameters listname, recordpointer, done, and other. Other denotes those parameters whose values must be preserved between calls to process.

The heart of a traversal is repeated access of the record pointed to by recordpointer, processing of the record, and updating of recordpointer to point to the next record. Clearly, a loop is its main construct. The function for a general traversal of a list is as follows:

>Comment

traverse(listname)

listpointer listname;

{

listpointer recordpointer,next();

moreparameters other;

int done;

>Comment

initialize(listname,&other);

recordpointer = listname;

done = FALSE;

while(!done&&anotherrecord(listname,

recordpointer))

{

>Comment

process(listname,recordpointer,

&other,&done);

>Comment

recordpointer =

next(recordpointer);

}

>Comment

finalize(listname,other);

}

Initialize may be used to set the initial value of any parameters that require initialization. Anotherrecord is a function returning true if recordpointer points to the next record to be processed, and returning false if the last record has already been processed. Finalize is used to carry out any tasks required after all list records have been processed--for instance, to print an appropriate message.

It is a good idea to check a program for correctness, in lieu of a formal proof, by seeing if it handles all special cases (and the typical case) correctly. You should check traverse in this way for the case of a null list, a one-record list, and a list with more than one record. For example, if the list is null, the while loop is never executed as long as anotherrecord is correct (which is assumed), and the program terminates properly as long as finalize is correct (which is assumed). So traverse does work correctly for the special case of a null list.

3.4.3 The Merits of Functional Modularization

Traverse has been written in this general way so that it can be used as a tool readily adaptable to the solution of a large class of problems. How to adapt this function to the solution of seemingly disparate problems is the purpose of the next section. It is important to recognize the basic technique used to render the program independent of the list implementation: embedding those details in lower level functions. This is the reason for anotherrecord and next, initialize, process, and finalize; these functions allow traverse to be adapted to specific applications.

All the functions that must reflect implementation and application details in their definitions are lower-level functions. Changes to them do not affect the correctness of traverse, but they do affect what traverse does. The functions call attention to what is being done instead of how it is being done. Furthermore, if the programmer consistently uses a function such as next or process to carry out specific tasks, then no matter how many places the task is required in a program, the definition of that task is localized in one place, and it is clear where to look for the code that implements it. The functions may then all be written independently to suit the immediate purpose. As long as they are correct, traverse must be correct too.

3.5: Using the Traverse Function for Lists

Sometimes a solution to a new problem can be obtained by modifying a known program that solves a related problem. Also, maintenance or adaptation of programs frequently involves small changes to current programs. Changes are easier, quicker, and more likely to be error-free if exactly that function or task that requires change has been localized as a program function.

Traverse adheres to two important elements of programming style.

1. High-level descriptions and programs should be as independent of the details of data abstraction implementation and of the application as possible.

2. High-level descriptions and programs should be functionally modular so that special lower-level functions or tasks carried out within the high-level description or program may be easily isolated.

To illustrate the use of traverse, and the effect of these principles, this section presents four examples. Although each could be solved in isolation, from scratch, a different approach is taken here. In each case it should be easy to see that traverse may be adapted to provide an almost immediate solution. This will be done by writing specific versions of traverse's functions for each application, so that these functions turn traverse into a solution. This means that it will then do the required task when executed. Of course, the functions, when invoked by traverse, will do exactly what the programmer specifies in their definition.

Example 3.2

A sentence is stored in a list, with each record storing one character. Thus the sentence, THIS SENTENCE HAS EXACTLY 60 CHARACTERS COUNTING BLANKS AND., is stored in a list with 60 records. For simplicity, assume the period character "." does not occur within a sentence (as it does in this one). The task is to write a function to print the length of the sentence.

The solution can be achieved by traversing the list and counting the number of records. Initialize can set a counter, length, to zero, and process can increment it by 1 each time it is invoked. Finalize can do the printing. The functions are readily defined.

>Comment

initialize(listname,plength)

listpointer listname;

moreparameters *plength;

{

*plength = 0;

}

>Comment

process(listname,recordpointer,plength)

listpointer listname,recordpointer;

moreparameters *plength;

{

(*plength)++;

}

    >Comment

finalize(listname,length)

listpointer lastname;

moreparameters length;

{

if(length != 0)

printf ("\n The sentence has length %d \n",length);

else

printf("\n There is no sentence \n");

}

Either anotherrecord and next must be supplied to the programmer, or the details of the list implementation must be specified such that the programmer can write them. The task of the example is then carried out by the invocation traverse (sentence). In this example traverse must be modified to the following function.

#define TRUE 1

#define FALSE 0

typedef int moreparameters;

traverse(listname)

/* Prints the length of the sentence

stored in the list, listname.

*/

listpointer listname;

{

listpointer recordpointer,next();

moreparameters other;

int done;

    >Comment

initialize(listname,&other)

recordpointer = listname;

done = FALSE;

while(!done&&anotherrecord(listname,recordpointer))

{

    >Comment

process(listname,recordpointer,&other);

recordpointer = next(recordpointer);

}

finalize(listname,other);

}

There are two ways to view traverse. One is as a program that can be used but not modified in any way. This would require that the functions be written as they appear and the type moreparameters be declared prior to traverse. The second is as a program that may be used or changed in any desirable way. In this case, for clarity, the function names can be changed: traverse to countcharacters, initialize to zerocounter, process to incrementcounter, finalize to printcounter, and anotherrecord to anothercharacter, for example. It may be desirable to remove parameters that are not needed. Since listname is not needed by initialize, process, or finalize, and recordpointer is not needed by process they may be omitted from the definitions and invocations of these functions. Other may be changed to length. Finally, done may be eliminated, since it is not needed. Having made this distinction, the programmer is free to take either view, or a view that falls between the two. The one chosen should be clear from the context.

Suppose the sentence were stored in the list with three characters per record instead of just one per record. The solution would require a new version of process to reflect this change, but initialize and finalize remain as is. Next might have to reflect the new record format. The new process is

    >Comment

process(listname,recordpointer,plength)

listpointer listname,recordpointer;

moreparameters *plength;

{

listpointer next();

int lastrecord;

char char1(),char2();

lastrecord = !anotherrecord(listname,next(recordpointer));

    >Comment

if(!lastrecord)

*plength = *plength + 3;

    >Comment

else if(char1(recordpointer) == '.')

(*plength)++;

else if(char2(recordpointer) == '.')

    >Comment

*plength = *plength + 2;

else

    >Comment

*plength = *plength + 3;

}

where char1 returns a copy of the first character in the record pointed to by its parameters, and char2 returns a copy of the second character in the record pointed to by its parameters. Like next and anotherrecord, they must be supplied, or the programmer must write them given list implementation details.

Finally, suppose the list is implemented as a circular list (Figure 3.10). In circular lists, the null pointer of the last record is replaced by a pointer that points to the first record. To illustrate the difference in the implementations between a standard list and a circular list, the two distinct versions required for anotherrecord are presented.

Figure 3.10 A Circular List

Standard List

anotherrecord(listname,recordpointer)

listpointer listname,recordpointer;

{

listpointer setnull();

    >Comment

return(recordpointer != setnull());

}

Circular List

anotherrecord(listname,recordpointer)

listpointer listname,recordpointer;

{

    >Comment

return(listname != recordpointer);

}

Actually, the solution is not as general as possible. Suppose the list were implemented using a dummy header record. Then process must recognize when it is processing this record (the first) and not change length, since no characters are stored in the header record. You should modify process to account for this possibility. It is necessary to assume that the header may be distinguished from the other records in some way.

Example 3.3

Earlier in the chapter a way to reverse a list was developed from scratch. We now build a program to reverse a list utilizing traverse. Traverse may be turned into a solution for the list reversal by defining initialize, process, and finalize, as follows:

    >Comment

initialize(listname,psave,ppredecessor)

listpointer listname,*psave,*ppredecessor;

{

listpointer setnull();

*psave = setnull();

*ppredecessor = setnull();

}

    >Comment

process(plistname,recordpointer,psave,

ppredecessor)

listpointer *plistname,recordpointer,*psave,

*ppredecessor;

{

listpointer setnull();

if(*ppredecessor != setnull())

setlink(*ppredecessor,*psave);

*psave = *ppredecessor;

*ppredecessor = recordpointer:

}

    >Comment

finalize(plistname,save,predecessor)

listpointer *plistname,save,predecessor;

{

listpointer setnull();

if(predecessor != setnull())

{

setlink(predecessor,save);

*plistname = predecessor;

}

}

    >Comment

traverse(plistname)

/*Reverses the list listname.*/

listpointer *plistname;

{

listpointer recordpointer,*psave,*ppredecessor,next();

initialize(listname,&psave,&ppredecessor);

recordpointer = *plistname;

while(anotherrecord(*plistname,recordpointer))

{

process(*plistname,recordpointer,&psave,&ppredecessor);

recordpointer = next(recordpointer);

}

finalize(&plistname,*psave,*ppredecessor);

}

Notice that the parameter of traverse is now a pointer to the listname. This is necessary since listname may need to be changed by traverse. Here process works on the record preceding the record pointed to by recordpointer, while the original solution worked on the record pointed to by recordpointer. This was necessary because that solution changed recordpointer's link field before updating recordpointer.

Example 3.4

Insert a new record into a list stocks, which is kept in alphabetical order by name. Assume the data for the new record may be accessed by a function getnextrecord, which returns a pointer to the new record.

To turn traverse into a solution requires simply that process compare the new record's name field value with that of the current record. If it precedes the current record's name field value, it must be inserted before the current record and done set to true. To make the insertion conveniently, a pointer, predecessor, to the predecessor of the current record will be kept by process and updated each time process is invoked. Initialize must allocate storage for the new record, set its data field values, and set predecessor to null. Assume that a function avail is given, returning a pointer to a storage element that may be used to store the new record. Finalize must insert the new record in the event that the list is null. The solution for these functions follows:

    >Comment

initialize(pnewpointer,ppredecessor)

listpointer *pnewpointer,*ppredecessor;

{

listpointer avail(),setnull(),getnextrecord();

*pnewpointer = avail();

setinfo(*pnewpointer,getnextrecord());

*ppredecessor = setnull();

}

where setinfo copies the contents of the record pointed to by its second parameter into the information field of the record pointed to by its first parameter, so it depends on the list implementation details.

    >Comment

finalize(plistname,predecessor,newpointer)

listpointer *plistname,predecessor,newpointer;

{

listpointer setnull();

if(*plistname == setnull())

insert(plistname,predecessor,newpointer);

}

    >Comment

process(plistname,recordpointer,newpointer,ppredecessor,

pdone)

listpointer *plistname,recordpointer,newpointer,

*ppredecessor;

int *pdone;

{

listpointer setnull(),next();

if(precedes(newpointer,recordpointer))

{

    >Comment

insert(plistname,*ppredecessor,newpointer);

*pdone = TRUE;

}

else if(next(recordpointer) == setnull())

{

    >Comment

insert(plistname,recordpointer,newpointer);

*pdone = TRUE;

}

    >Comment

else

*ppredecessor = recordpointer;

}

where precedes returns true if its first parameter points to a record whose name precedes, in alphabetical order, the name in the record pointed to by its second parameter. To insert a new record, invoke traverse (&stocks). However, traverse must be modified in a way similar to Example 3.2 as follows. A better name for this modified traverse would be orderedinsert, so we use it.

    >Comment

orderedinsert(plistname)

/*Inputs and inserts a new record

in the list listname in alphabetical

order.

*/

listpointer *plistname;

{

listpointer recordpointer,newpointer,predecessor,next();

int done;

initialize(&newpointer,&predecessor);

recordpointer = *plistname;

done = FALSE;

while(!done&&anotherrecord(*plistname,recordpointer))

{

process(plistname,recordpointer,newpointer,

&predecessor,&done);

recordpointer = next(recordpointer);

}

finalize(plistname,predecessor,newpointer);

}

Suppose you wanted to insert the new record into a list of stocks kept in order by date bought, instead of by name. Rather than keep another list distinct from stocks, you could add another link field to its records. Call it the datelink field, and assume each record's datelink field points to the correct successor as in Figure 3.11. Stockdate is a new head, pointing to the first record of that list when date bought is used as the criterion for ordering.

Figure 3.11 Two Orders for a List Using Two Heads and Two Link Fields

To solve this problem: 1) define precedes so that it compares dates instead of names; 2) replace each reference in insert to the setlink function by a reference to the setdatebought function and define it to work on the datelink field instead of the link field; and 3) redefine next to return a copy of the datelink field of the record pointed to by its parameter. A call to orderedinsert(&stocksdate) then carries out the task.

Example 3.5

Create a list stocks, to store a portfolio of stock records.

To solve this problem, assume the data are already stored on some input medium and getnextrecord gives access to each record in turn, as it is invoked. The desired list may be created by traversing the records stored on the input medium. The job of process is to fill in the information field of the storage allocated for the new record, and fill in the link field of the new record with a pointer to storage allocated for the next new record. However, it is assumed that the last input data is a sentinel value that indicates there are no more records. When the sentinel is input, getnextrecord returns a null pointer value. Consequently, process must first check for this case and, when it occurs, simply return after setting done to true and the link field of the predecessor of the new record to null, unless the list is null. If it is null, process must set done to true and listname to null. Initialize must allocate storage for the first record and set listname to it. Finalize is not needed. We have

    >Comment

initialize(plistname)

listpointer *plistname;

{

listpointer avail();

*plistname = avail();

}

    >Comment

process(plistname,recordpointer,pdone)

listpointer *plistname,recordpointer;

int *pdone;

{

static listpointer predecessor;

listpointer setnull(),avail(),getnextrecord(),

pointer;

pointer = getnextrecord();

if(pointer != SENTINEL)

{

setinfo(recordpointer,pointer);

setlink(recordpointer,avail());

predecessor = recordpointer;

}

    >Comment

else if (*plistname != recordpointer)

{

setlink(predecessor,setnull());

*pdone = TRUE;

}

else

{

    >Comment

*plistname = setnull();

*pdone = TRUE;

}

}

Invoking a correctly modified traverse(&stocks) creates the list. Notice that this traverse will copy a list whenever getnextrecord obtains its information from the next record of the list to be copied. A better name for this function would be readlist or copylist. At this point you may want to see a complete program that creates the stocks list and also prints it. The program is in Section 3.6.1.

Suppose the records are given in arbitrary order, but the stocks list must be created with the records appearing in alphabetical order. To turn traverse into a solution to this problem, initialize, finalize, and anotherrecord are not needed; neither are the variables recordpointer and other. This problem is solved by the following version of traverse, which we call expand. The name expand was chosen because, when called to work on an existing ordered list, it simply expands the list by adding the new inputs to the list in the proper places. When the call expand(&listname) is made with listname set to null, it creates the ordered list.

    >Comment

expand (plistname)

/*Inputs a series of records and

inserts them in alphabetical

order in the ordered list listname.

*/

listpointer *plistname;

{

int done;

done = FALSE;

while(!done)

    >Comment

processrecord(plistname,&done);

}

    >Comment

processrecord(plistname,pdone)

listpointer *plistname;

int *pdone;

{

listpointer pointer,getnextrecord();

pointer = getnextrecord();

if(pointer != SENTINEL)

    >Comment

insertinorder(plistname,pointer);

else

*pdone = TRUE;

}

Notice that expand has no need for the functions initialize, finalize, and anotherrecord, nor for the variables recordpointer and other.

Processrecord must obtain the next input record and insert it in the list at the correct place. It invokes insertinorder, which has the task of creating and inserting the new record into its proper place in listname; pointer points to the record whose content is to be placed in the new record's information field. Insertinorder itself requires a traversal of listname. The version below is a modification of orderedinsert, obtained by adding a second parameter, pointer. Orderedinsert uses getnextrecord to obtain the next input record and then inserts it, while insertinorder is given a pointer to the new record and needs only insert it. The additional task of obtaining the new input record is handled outside of insertinorder. Consequently its process and finalize functions can be used directly in insertinorder. Only its initialize function needs changing, but all are given below for easy reference.

    >Comment

insertinorder(plistname,pointer)

/*Inserts the record pointed to by pointer

in the ordered list listname.

*/

listpointer *plistname,pointer;

{

listpointer recordpointer,predecessor,newpointer,

next();

int done;

    >Comment

initialize(&newpointer,&predecessor,pointer);

recordpointer = *plistname;

done = FALSE;

while(!done&&anotherrecord(*plistname,

recordpointer))

{

    >Comment

process(plistname,recordpointer,newpointer,

&predecessor,&done);

recordpointer = next(recordpointer);

}

    >Comment

finalize(plistname,predecessor,pointer);

}

    >Comment

initialize(pnewpointer,ppredecessor,pointer)

listpointer *pnewpointer,*ppredecessor,pointer;

{

listpointer setnull(),avail();

*pnewpointer = avail();

setinfo(*pnewpointer,pointer);

*ppredecessor = setnull();

}

    >Comment

process(plistname,recordpointer,newpointer,

ppredecessor,pdone)

listpointer *plistname,recordpointer,

newpointer,*ppredecessor;

int *pdone;

{

listpointer setnull(),next();

if(precedes(newpointer,recordpointer))

{

    >Comment

insert(plistname,*ppredecessor,newpointer);

*pdone = TRUE;

}

else if(next(recordpointer) == setnull())

{

    >Comment

insert(plistname,recordpointer,newpointer);

*pdone = TRUE;

}

else

    >Comment

*ppredecessor = recordpointer;

}

    >Comment

finalize(plistname,predecessor,newpointer)

listpointer *plistname,predecessor,newpointer;

{

listpointer setnull();

if(*plistname == setnull())

insert(plistname,predecessor,newpointer);

}

3.5.1 Why Traverse Was Useful

It is very important to recognize that traverse of Section 3.4.2 is a tool that could be used in solving the example problems. The tool was applicable because

It was written using data abstractions that allowed writing or using required implementation functions without changing traverse.

It was written using functional modularization, which allowed isolation of functions for relevant tasks so that they could be written to satisfy specific needs.

Recognizing and applying tools is a major aspect of programming methodology. For a tool to be convenient to use, it must be written according to the two elements of programming style just stated. Of course, in the simple examples using the tool may not save much time, effort, or cost, but for complex tasks the savings can be great.

Note that the use of getnextrecord makes the solutions to Examples 3.4 and 3.5 independent of the input medium or the format of the input. Thus any changes in the input are not reflected in traverse; only getnextrecord and SENTINEL can require modification. In fact, the same should be done for any output. It is better to use a function rather than specify input and output in the basic functions, in case the input or output medium is changed.

3.6: Implementing Lists

So far this chapter has covered lists in the abstract. Lists are implemented in two basic ways, differentiated by where the records of the lists are stored: 1) in an array of records or 2) in dynamic memory. These are similar, but dynamic memory allows for more efficient use of storage and is easier to use.

3.6.1 Lists Stored in Dynamic Memory

The simplest implementation is to store all list records in the dynamic memory. The declarations required are illustrated below for the example of the stock portfolio. For storage in dynamic memory, the records of a list are defined as usual, and an additional field is added to contain the link field pointer. Since it will point to a successor record stored in dynamic memory, it must be defined as a pointer. In this case, the null value for a pointer is always given by the value , with  having the value 0. Variables of type listpointer will contain pointers to records of type stockrecord.

#define MAXSIZE 5

#define NULL 0

#define SENTINEL -1

typedef struct

{

int month;

int day;

int year;

}date;

typedef struct

{

char name[MAXSIZE];

int shares;

float value;

date datebought;

}infofield;

typedef struct listrecord

{

infofield info;

struct listrecord *link;

}stockrecord,*listpointer;

The following complete program reads in and creates a list of stockrecords as in Example 3.5 and, in addition, prints out the information fields of the list records. Both the creation and printing portion of the code are appropriately modified versions of traverse. The initialize and process functions of Example 3.5 are used in the creation portion, but process has been given the name addrecord. The program treats the list as a data abstraction. The data abstraction list, with its allowed operations--next, setlink, setinfo, avail, anotherrecord, getnextrecord, and printrecord, appear together in the program.

Table 3.2 Sample Input

                                Explanation                   Actual Input

--------------------------------------------------------------------------

Record 1         Number of shares                              100

                 Stock name                                    abc

                 Price of a share                              56.25

                 Month day year of purchase                    1 23 1978

Record 2         Number of shares                              50

                 Stock name                                    wxyz

                 Price of a share                              5.50

                 Month day year of purchase                    11 4 1986

                 .

                 .

                 .

Terminal record  Sentinel (where next number of shares would   -1

                   otherwise appear)

Typical input to the program is shown in Table 3.2.

The program that follows reads in the records of a list, creates the list, and prints the records in the list.

    >Comment

#define MAXSIZE 5

#define NULL 0

#define SENTINEL -1

typedef struct

{

int month;

int day;

int year;

}date;

typedef struct

{

char name[MAXSIZE];

int share;

float value;

date datebought;

}infofield;

typedef struct listrecord

{

infofield info;

struct listrecord *link;

}stockrecord,*listpointer;

listpointer next(pointer)

/* Returns a copy of the

link field in the record

pointed to by pointer

*/

listpointer pointer;

{

return(pointer->link);

}

listpointer setnull()

/* Returns a null pointer */

{

return(NULL);

}

setlink(pointer1,pointer2)

/* Sets the link field of the

record pointed to by pointer1

to the contents of pointer2

*/

listpointer pointer1,pointer2;

{

pointer1->link = pointer2;

}

setinfo(pointer1,pointer2)

/* Sets the infofield of the record

pointed to by pointer1 to the

infofield of the record

pointed to by pointer2

*/

listpointer pointer1,pointer2;

{

int i;

for(i=0;i<MAXSIZE;i++)

pointer1->info.name[i] = pointer2->

info.name[i];

pointer1->info.shares = pointer2->

info.shares;

pointer1->info.value = pointer2->info.value;

pointer1->info.datebought.month = pointer2->

info.datebought.month;

pointer1->info.datebought.day = pointer2->

info.datebought.day;

pointer1->info.datebought.year = pointer2->

info.datebought.year;

}

listpointer avail()

/* Returns a pointer to storage

allocated for a record of type

stockrecord

*/

{

listpointer malloc();

return(malloc(sizeof(stockrecord)));

}

anotherrecord(recordpointer)

/* Returns true if there is

another record

*/

listpointer recordpointer;

{

listpointer setnull();

return(recordpointer != setnull());

}

listpointer getnextrecord()

/* Inputs the data for the new

record and returns a pointer

to a record containing the data

in its infofield, or, if there

are no new records,returns

a null pointer

*/

{

listpoiner setnull();

static stockrecord nextrecord;

printf("Enter number of shares\n");

scanf("%d",&(nextrecord.info.shares));

if(nextrecord.info.shares != SENTINEL)

{

printf(" Enter the stock name - less than

%d characters\n",MAXSIZE);

scanf("%s",nextrecord.info.name);

printf(" Enter the price of one share of

the stock\n");

scanf("%f",&(nextrecord.info.value));

printf(" Enter the month day year of the

stock purchase\n");

scanf("%d %d %d",&(nextrecord.info.

datebought.month),

&(nextrecord.info.datebought.day),

&(nextrecord.info.datebought.year));

return(&nextrecord);

}

else

return(setnull());

}

printrecord(recordpointer)

/* Prints the contents of the infofield

of the record pointed to by recordpointer

*/

listpointer recordpointer;

{

printf(" The stock is %s\n",

recordpointer->info.name);

printf(" The number of shares is %d\n",

recordpointer->info.shares);

printf(" The value of a share is %f\n",

recordpointer->info.value);

printf(" The date of purchase is %d %d %d\n\n",

recordpointer->info.datebought.month,

recordpointer->info.datebought.day,

recordpointer->info.datebought.year);

}

    >Comment

#include <stdio.h>

#define TRUE 1

#define FALSE 0

main()

/* Inputs a series of stockrecords,

creates a list of the records, and

prints the contents of each list

record's infofield

*/

{

listpointer stocks,recordpointer,next();

    >Comment

int done;

initialize(&stocks);

recordpointer = stocks;

done = FALSE;

while(!done&&anotherrecord(recordpointer))

{

    >Comment

addrecord(&stocks,recordpointer,&done);

recordpointer = next(recordpointer);

}

    >Comment

recordpointer=stocks;

while(anotherrecord(recordpointer))

{

    >Comment

printrecord(recordpointer);

recordpointer = next(recordpointer);

}

}

initialize(plistname)

/* Allocates storage for the first record

and sets listname to point to it

*/

listpointer *plistname;

{

listpointer avail();

*plistname = avail();

}

addrecord(plistname,recordpointer,pdone)

/* Fills in the current record's data, and allocates

storage for the next record and adds it to the

list. If there is no data for the current record

it sets the link field of the last list record

to null, or the list head to null, and done to

true.

*/

listpointer *plistname,recordpointer;

int *pdone;

{

static listpointer predecessor;

listpointer setnull(),avail(),getnextrecord(),

pointer;

pointer =getnextrecord();

if(pointer != setnull())

{

    >Comment

setinfo(recordpointer,pointer);

setlink(recordpointer,avail());

predecessor = recordpointer;

}

else if(*plistname != recordpointer)

{

    >Comment

setlink(predecessor,setnull());

*pdone = TRUE;

}

else

{

    >Comment

*plistname = setnull();

*pdone = TRUE;

}

}

3.6.2 Lists Stored in an Array of Records

Instead of being stored in dynamic memory, list records may be stored in an array of records. The declarations required are illustrated for the example of stocks.

#define MAXSIZE 5

#define MAX 100

#define NULL -1

typedef struct

{

int month;

int day;

int year;

}date;

typedef struct

{

char name[MAXSIZE];

int shares;

float value;

date datebought;

}infofield;

typedef struct listrecord

{

infofield info;

    >Comment

int link;

}stockrecord;

stockrecord records[MAX];

    >Comment

typedef in listpointer;

For this implementation the null pointer value is -1. Here, variables of type listpointer contain integer array indices. The implementation for some of the functions operating on the list are

next(pointer)

/* Returns a copy of the

link field in the record

pointed to by pointer

*/

listpointer pointer;

{

return(records[pointer].link);

}

setnull()

/* Returns a null pointer */

{

return(-1);

}

setlink(pointerl,pointer2)

/* Sets the link field of the

record pointed to by pointer1

to the contents of pointer2

*/

listpointer pointer1,pointer2;

{

records[pointer1].link = pointer2;

}

This implementation assumes that records is declared nonlocally to these functions, so that they may reference it. With this implementation, the records array plays the role of the dynamic memory. It is necessary to write an avail function that must keep track of those slots in records that are not in use, and hence can be allocated when storage for a list record is requested.

3.6.3 Lists Stored in Languages without Records

In some languages, such as FORTRAN, there is no dynamic memory, nor are there pointer variables or records. In such a language, the records themselves must be implemented by the programmer. They may be stored in single or multiple arrays, as discussed in Chapter 2. Since the list records now have an additional link field indicating their successor, they need not be stored sequentially and may be located anywhere in the array or multiarrays, as long as they do not overlap. An additional variable is needed for the head of the list. Once these arrays are set up, the programmer may write functions to access each field of a record and functions to copy one record's value into another record, or to copy a record's field value into another record's field. These might include

next(pointer)

Returns a copy of the link field value of the record pointed to by pointer.

setlink(pointer1,pointer2)

Sets the value of the link field of the record pointed to by pointer1 to pointer2.

setlinfo(pointer1,pointer2)

Sets the value of the information field of the record pointed to by pointer1 to the value of the information field of the record pointed to by pointer2.

In FORTRAN, the array or multiarray in which the records are kept could be declared to be stored in the FORTRAN common area or else must be passed as parameters to these functions. Once they are written, for all intents and purposes the language has records. These records can be accessed, copied, or modified by using these functions. Although the programmer must do all this work initially, from this point on the situation is conceptually the same as in Section 3.6.2. That is, the actual implementation of the records may be forgotten; they are treated as if stored in an array of records, just as in C. The problem of writing an avail function still remains, and it is addressed in the next section.

In FORTRAN, although storage for records kept in lists must be managed, the programmer does not need to manage storage for individual elements or for arrays of elements. To refer in a program to an element n, or an array data, simply declare n to be an element, and data to be an array. Then use the operations available in FORTRAN to enter specific values into n or data. This works because individual elements and arrays are data structures directly available in FORTRAN. Lists, on the other hand, are not directly available and must be built up explicitly by storing their records in arrays. Consequently, the programmer must determine where storage is available to create a record when the need arises.

In C it is not necessary to implement records of lists explicitly in arrays. Instead, dynamic data structures may be used. A data structure is dynamic if it is built during execution of a program and storage is allocated for its components at that time. Dynamic data structures are built simply by asking for storage to be allocated for a record in dynamic memory. A pointer will then be assigned to point to that storage. Using the pointer, appropriate values may be assigned to the fields of the new record. These fields may be referred to directly, using naming conventions in the language. Special functions to store or access information from the fields of the record need not be written.

Records still need to be inserted properly in a dynamic data structure. Commands directly available in the language may be used to create or reclaim records, or reclamation may be left to the implicit storage reclamation provided by the language. Thus the definition, creation, reclamation, and manipulation of dynamic data structures become easier.

3.7: Keeping Track of Available Records

Keeping track of and allocating storage that is needed during the execution of a program is called dynamic storage management. In some languages, notably FORTRAN, storage for arrays is allocated before the program's execution, and no dynamic storage management is present. In languages like C, storage is allocated to the local variables of a function at the start of its execution and remains assigned until the function (or block) completes its execution. The space can then be reassigned to the next executing function (or block). In effect, it disappears. This is why values of local variables are not preserved between invocations of a function, unless they are declared as static, in which case they are preserved between calls. Management of this storage during the execution of a program is not what is meant by dynamic storage management. Contrast this situation to dynamic memory, which is assigned or taken away from records in dynamic memory during the execution of a function itself. This captures the true flavor of dynamic storage allocation.

If the array in which records are to be kept (as in Sections 3.6.2 and 3.6.3) is to function as dynamic memory, then allocating storage in the array for a record requires dynamic storage management. In fact, this amounts to an implementation of a C-like dynamic memory. Although storage can thus be managed explicitly for a specific program, dynamic memory can manage storage for many programs and is hence more efficient use of storage. The difference is that one large array may function as storage to be used by all the programs, instead of dedicating separate smaller arrays to each program.

3.7.1 Why Dynamic Memory?

Why is storage efficiency important? Why not find storage for a new record in the array simply by placing it sequentially below the last new record's storage? After all, this would be very easy to do and would certainly use the storage as efficiently as possible. The answer is that this simple solution is a great idea--provided that all the storage needed for the array to be of sufficient length is available. The catch is that this proviso is rarely the case. Often, the amount required isn't known in advance; even when it is known, not enough storage may be available.

Reconsider the prime number example of Chapter 1. It is probably evident by now that the list is a good data structure to use for collection. The program to generate all the primes between 2 and n uses very little storage besides that needed for the list, but how large should you make the array in which to store the list? It is possible to modify the program so that it need not actually store all integers between 2 and n in the list, but only those that are prime. Still, how many primes are there no greater than n? It is even possible to write such a program so that it efficiently generates and prints all primes   [image]  n yet stores only those   [image]  n in the list, but again, how many of these are there?

Suppose you are a programmer asked to store the contents of ten manuscripts in ten lists. If each can have a maximum length of 50,000 words, then storage for 500,000 words is needed. How long is the longest word? You see the problem. Even if the amount of storage could be estimated, sufficient storage may not be available. You might conclude that the task cannot be done, or you might use secondary storage (magnetic tape, disks). However, in secondary storage access is relatively slow, and it requires techniques different from those described so far. One possibility remains. Maybe the manuscripts to be stored, perhaps even more than 10, are compatible (in the sense that they are not all of great length simultaneously). Perhaps the lengths will vary as the manuscripts are processed by a program, so that at any one time the manuscripts being stored can fit. This means that they may be growing and shrinking in length at different points in the program. If the storage no longer needed for one manuscript is relinquished and reused for another, then the total array length may not ever be exceeded. In other words, if storage can be allocated, then reused for new allocations when no longer needed for its original purpose, problems may be solved that are otherwise out of range. This is the reason to manage storage.

This also explains why dynamic memory makes better use of storage than individual program management using arrays of records. Each program may, to ensure its execution, ask for a large amount of storage to be used for its array. The total amount needed by all the programs may not be at hand. It is better to take all available storage and dole it out to each program as needed, the hope being that whenever some programs need a great deal of storage, the others need little. Instead of dedicating amounts to each user for each one's worst case, the programmer manages the storage and gives out only what is needed at any moment.

3.7.2 Using Dynamic Memory

Having taken this relevant digression, let's return to the job of managing dynamic memory. This will show how dynamic memory itself might function, and give us the means to create our own when desirable. In fact, you will see when this might be advantageous.

Pointers are used in C to eliminate the problem of dynamic storage management for the programmer. Pointers allow the declaration of a variable of a complex data type and allocate storage for it by merely assigning to its pointer a value that points to that storage. The C function malloc is used for this purpose. This section develops a function avail that will have the same effect as malloc but will manage storage in the records array for Section 3.6.2.

In the general case of list processing, records need to be inserted in and deleted from lists. It is deletion, you may recall, that makes the solution of larger problems feasible. When records are deleted, their storage in the records array may be reused. Reclaiming storage in this way allows larger problems to be solved than would otherwise be possible, by allowing the program to execute to completion.

Records of one or more lists may be stored in the records array at any moment during the execution of a program. Assume that all these records have the same length, and that no sharing of storage occurs. No sharing means that no list record has more than one pointer to it at any time. The more typical case of variable-length records and sharing of storage is more complex and will be dealt with in a later chapter. The issue is now how to implement avail.

Suppose that, by looking at a record, you can tell whether or not it is in use. This would be the case if, whenever a record were deleted from a list, it were marked unused. An available record could be found by traversing the records array until a record marked unused were found. Avail could then return a pointer to that record, after marking it as now in use. In Figure 3.12, this would require the first three records to be accessed before avail could return to point to position 2. The time to carry out this procedure is proportional to the length of the records array in the worst case.

A very significant improvement would be to keep track of the available records so that the search may be avoided and the time for avail to do its task reduced to a constant. This is achieved by using the link fields of each unused record to create a list of available records. Initially all entries of records are unused, so the list should contain every record of the array. Figure 3.13 shows how this list might appear for the configuration of Figure 3.12, with availist taken as its head. Avail now carries out its task by simply deleting the first record from availist and returning a pointer to the deleted record. This yields the desired constant time for avail.

Figure 3.12 Typical Configuration of Used and Unused List Records in the RECORDS Array

Figure 3.13 The List of Available Records Corresponding to Figure 3.12

When records are deleted from a list in programs, it is convenient to keep track of their storage for future use. This requires inserting the deleted records' storage on availist. Again, this operation can be done in constant time if the newly available record is inserted at the front of availist--as its new first record. The function that carries out this task will be called reclaim. What time is required for insertion if the record is inserted in the list other than at the front?

The programmer invokes p = avail() to request storage for a record to be stored in records, and p points to that storage when avail returns. The programmer invokes reclaim(p) to request that the storage in records pointed to by p be inserted on availist. By employing avail and reclaim appropriately, the programmer can dynamically allocate and reclaim storage for lists. This means that records can be viewed as the repository of all list records, thus acting as a miniature dynamic memory. Such routines are the heart of storage management algorithms. When the records of a list are not all of the same length, more complex management techniques must be used. When records can be shared, still more complexity is added to the storage management problem, as will be illustrated subsequently.

Malloc is the general C allocation function provided for the programmer's use, so that dynamic memory appears as an abstract facility for storing records. When no memory is available to be allocated, malloc returns a NULL value. Consequently, on each call to malloc the value returned should be checked to be certain it is not the NULL value. Avail may be written to abort or take whatever action is necessary for your application in the event memory is not available.

The C function free is provided to carry out the general task of reclaim. Free deallocates memory that was previously allocated by malloc. The argument to free has to be a pointer previously returned by malloc that has not already been freed. Once memory is freed, it can be used to satisfy new requests through malloc.

In languages such as FORTRAN, which do not provide dynamic memory, you can implement it through avail and reclaim. Once avail and reclaim are written, for all intents and purposes the language has dynamic memory for record storage. You have tailored it to your needs.

3.8: Sequential Arrays versus Lists for Record Storage

Arrays allow records to be stored contiguously and to be accessed randomly. With pointer arrays, random access is possible even for variable-length records. Lists are also data structures containing records. Lists provide the flexibility, when needed, to separate records--enhancing the ability to insert or delete, but reducing the capability of accessing records at random. Stored records can be traversed using either the sequential array or list implementation. The time to carry out the traversal need not be significantly different for the two methods.

Suppose that instead of processing the records in sequential order, as in a traversal, your program must access the records in arbitrary order. (In an airplane reservation system, for example, seat reservations must be processed immediately, in random order.) Having to access the records in arbitrary order means that the program must access and process the records even though the programmer cannot predict the order in which this must be done. The sequential representation, because it allows selection for fixed-length records, allows this random accessing to be accomplished in constant time. The list representation requires that random accessing be done by traversing to the needed record, starting from the first and accessing each succeeding record until the desired one is reached. If the required record is the ith, traversal will take time proportional to i. Thus randomly accessing records takes constant time for each record accessed with the sequential implementation, but time proportional to the desired record number for the list representation. This can result in significantly greater processing time for the list implementation when you are randomly processing records.

To insert one record in the collection (say, after the ith record) when using the sequential implementation, all succeeding records (the (i + 1)th, (i + 2)th, . . . , nth) must be moved down in the data array. This operation will take time proportional to the number of succeeding records (n - 1). Adding a record as the new first record gives the worst-case time, which is proportional to n. Inserting a new record using the list implementation, again assuming you have determined where it is to be inserted, takes a constant time--the time needed to change two pointers.

To delete one record, say the ith, using the sequential representation, requires that the succeeding i - 1 records be moved up. Thus the time required is proportional to i - 1. Using the list implementation requires a constant time for this deletion--the time needed to change one pointer. The time for deletion with a sequential implementation in the worst case is also proportional to n. It is obvious that insertion and deletion of one record can result in much greater execution time for the sequential implementation as compared to the list implementation.

The contrast is especially striking when record accesses, insertions, or deletions are performed on m records. The worst-case time to randomly access an entire collection of m records becomes proportional to m and m2, for the sequential and list implementations, respectively. For random insertions and deletions, these are reversed in favor of lists. This means that if m is doubled or tripled, the respective times increase fourfold or ninefold instead of merely doubling or tripling.

When selecting an implementation, the basic considerations are comparative execution times and relative frequencies for random access, insertion, or deletion.

The list implementation will require at least as much memory as the sequential implementation. For instance, if each record requires one element for the information field, and the list implementation requires another element for the link field, then lists would require twice as much storage. It is possible that part of the storage used to implement a record would be left unused in any case, so that using it for the link field requires no extra storage.

3.9: Case Study: Merging and the Perfect Shuffle

Consider two sequences of numbers that are ordered, largest to smallest:

100   80   65   30       90   85   50   10

They may be merged to produce a new sequence containing all the numbers in order. For example, compare the two largest and place the larger one first in the final sequence. Continue comparing the two current largest and placing the larger in the next spot in the final sequence until all numbers have been placed. The result, of course, is

100   90   85   80   65   50   30   10

This merging algorithm takes 2n basic operations when there are n numbers in each original sequence. It is possible to merge the sequences using a different algorithm, which, when parallel processing is possible, can reduce the time to O(lg 2n) rather than O(2n) when n is a power of 2. It is not obvious how to do this. The solution involves

1.     A "perfect shuffle" operation, for which we next develop a program

2.     A compare-and-exchange operation, which we will discuss after the perfect shuffle

3.9.1 The Perfect Shuffle

Using gambling terminology, a perfect shuffle of 2n cards occurs when the deck of cards is split into two halves and the shuffle interleafs the two halves perfectly. An original configuration of eight cards, with the configurations after two consecutive perfect shuffles, is shown in Figure 3.14 The cards are numbered one through eight. A third perfect shuffle would produce the original configuration.

Figure 3.14 The Result of Two Perfect Shuffles

Suppose a sequence of perfect shuffles is made for 2n cards. Eventually the original configuration must reappear. How many shuffles will it take before this happens? One way to find out is to execute code that performs such a sequence of shuffles, keeping track of how many are made, and printing this total when the original configuration occurs again. A program segment to do this can be written as follows:

initial(configuration,n);

numbershuffles = 1;

perfectshuffle (configuration,n);

while(!original(configuration,n))

{

perfectshuffle(configuration,n);

numbershuffles++;

}

Initial initializes configuration. Original returns true if configuration is the original, and false otherwise. Perfectshuffle changes configuration so that it reflects the new configuration resulting from a perfect shuffle. As written, this code is independent of how the configuration is actually implemented.

3.9.2 Array Implementation of the Perfect Shuffle

Suppose we decide to implement configuration as an array and use n as a pointer to the card in position n. Then initial and original may be defined, respectively by

    >Comment

for(i=1;i<=(2*n);i++)

configuration[i] = i;

and

    >Comment

temporiginal = TRUE;

i = 1;

while((i<n+1)&&temporiginal)

if(configuration[i] != i)

temporiginal = FALSE;

else

i++;

original = temporiginal;

Perfectshuffle is somewhat more complex. The first and last cards, 1 and 2n, never change position in configuration. After the shuffle, cards 1 and n + 1, 2 and n + 2, 3 and n + 3, . . . , i and n + i . . . , and n and 2n will appear in consecutive positions in the resultant configuration. An array, result, is used to hold the new configuration. In traversing the relevant n positions of configuration (the 0th is ignored), its ith entry is processed by copying configuration[i] and configuration[i+n] into their proper new consecutive positions in result. I serves as a pointer to the current entry of configuration being processed, and newi will point to the new position of that entry in result. The situation, when n is 6, is depicted in Figure 3.15(a) just before the fourth entry is to be processed.

If the following statements are executed

    >Comment

result[newi] = configuration[i];

result[newi+1] = configuration[i+n];

i++;

newi = newi + 2;

then the program will have properly processed the ith and (i+n)th cards. It will also have updated i and newi so that they are pointing to the proper new positions, in configuration and result, respectively. The situation after the fourth element has been processed is as shown in Figure 3.15(b).

To complete the shuffle, result can be copied into configuration. A program segment for perfectshuffle might be as follows.

    >Comment

i = 1;

while(i < n)

{

result[newi] = configuration[i];

result[newi+1] = configuration[i+n];

i++;

newi = newi + 2;

}

for(i=1;i<=(2*n);i++)

configuration[i] = result[i];

Notice that the solution involves a number of array traversals, each traversal processing accessed elements in a different way.

(a) Situation Just before the Fourth Entry Is Processed

(b) Situation Just after the Fourth Entry Is Processed

Figure 3.15 Two Arrays Used for a Perfect Shuffle of Six Elements

3.9.3 List Implementation of the Perfect Shuffle

Suppose you decided to implement configuration as a list of records stored in dynamic memory, and to use n as a pointer to the nth record of the configuration list. Figure 3.16 depicts these assumptions graphically for ten cards.

Figure 3.16 The List Implementation for CONFIGURATION

Perfectshuffle must, consecutively, move 6 before 2, 7 before 3, 8 before 4, and finally 9 before 5. Two pointers, l1 and n, are shown in Figure 3.16. Traversing configuration and processing the records pointed to by l1 and n will accomplish the movements, if the processing deletes the successor of the record pointed to by n, inserts this deleted record after the record pointed to by l1, and updates l1 to point to it. Move carries out this processing task.

Perfectshuffle may now be implemented by

    >Comment

configurationpointer l1,newn;

l1 = configuration;

newn = l1;

while(l1 != n)

{

move(l1,n);

l1 =l1->link;

newn = newn->link;

}

n = newn;

Perfectshuffle in this form is a refinement of traverse. Move plays the role of process. It uses local variables hold1 and hold2 to keep track of the successors needed:

    >Comment

move(pl1,n)

configurationpointer *pl1,n;

{

configurationpointer hold1,hold2;

hold1 = l1->link;

hold2 = n->link;

11->link = hold2;

n->link = hold2->link;

hold2->link = hold1;

l1 = l1->link;

}

Two complete programs that output the number of shuffles are presented at the end of this paragraph. The first is based on an array, and the second on a list implementation. In this case there is nothing to recommend the list implementation over the array implementation, since the configuration does not grow or shrink (except for the artificial limit on the array size that would be required by the array implementation). The two versions of perfectshuffle are both useful for the application to which we return in the section that follows the programs.

Array Implementation

#define TRUE 1

#define FALSE 0

    >Comment

typedef int arraytype[53];

main() /* Perfect shuffle using arrays */

/* Drives the perfectshuffle */

{

arraytype configuration;

int numbershuffles,n;

    >Comment

printf("\n Enter an integer between 1 and 26 \n");

scanf("%d",&n);

initial(configuration,n);

numbershuffles = 1;

    >Comment

perfectshuffle(configuration,n);

    >Comment

while(!original(configuration,n))

{

perfectshuffle(configuration,n);

numbershuffles++;

}

printf("\n Number of shuffles = %d \n",numbershuffles);

}

initial(configuration,n)

/* Creates the initial configuration */

arraytype configuration;

int n;

{

int i;

for(i=1;i<=(2*n);i++)

configuration[i] = i;

}

    >Comment

perfectshuffle(configuration,n)

/* Carries out a perfect shuffle */

arraytype configuration;

int n;

{

arraytype result;

int i,newi;

i = 1;

newi = 1;

while(i < (n+1))

{

    >Comment

result[newi] = configuration[i];

result[newi+1] = configuration[i+n];

    >Comment

i++;

newi = newi + 2;

}

for(i=1;i<=(2*n);i++)

configuration[i] = result[i];

}

original(configuration,n)

/* Returns true if the original

configuration has recurred

*/

arraytype configuration;

int n;

}

int i,temporiginal;

i = 1;

temporiginal = TRUE;

while((i < (n+1))&&temporiginal)

if(configuration[i] != i)

temporiginal = FALSE;

else

i++;

return(temporiginal);

}

List Implementation

#define TRUE 1

#define FALSE 0

#define NULL 0

typedef struct configrec

{

int info;

struct configrec *link;

    >Comment

}configrecord,*configptr;

int num;

main() /* perfect shuffle using lists */

/* Drives the perfectshuffle */

{

configptr configuration,n;

int numbershuffles;

    >Comment

printf("\n Enter an integer between 1 and 26\n");

scanf("%d",&num);

initial(&configuration,&n);

numbershuffles = 1;

    >Comment

perfectshuffle(configuration,&n);

    >Comment

while(!original(configuration,n))

{

perfectshuffle(configuration,&n);

numbershuffles++;

}

printf("\n The number of shuffles = %d\n",numbershuffles);

}

initial(pconfiguration,pn)

/* Creates the initial configuration */

configptr *pconfiguration,*pn;

{

configptr p,q;

int i;

*pconfiguration = malloc(sizeof(configrecord));

p = *pconfiguration;

i = 1;

while(i != (2*num+1))

{

p->info = i;

if(i == num)

*pn = p;

i++;

p->link = malloc(sizeof(configrecord));

q = p;

p = p->link;

}

q->link = NULL;

}

    >Comment

perfectshuffle(configuration,pn)

/* Carries out a perfect shuffle */

configptr configuration,*pn;x

{

configptr l1,newn;

l1 = configuration;

newn = l1;

while(l1 != *pn)

{

    >Comment

move(&l1,*pn);

    >Comment

l1 = l1->link;

newn = newn->link;

}

*pn = newn;

}

original(configuration,n)

/* Returns true if the original

configuration has recurred

*/

configptr configuration,n;

{

int temporiginal,i;

configptr p;

temporiginal = TRUE;

i = 1;

p = configuration;

while((p != n->link)&&temporiginal)

if(p->info != i)

temporiginal = FALSE;

else

{

i++;

p = p->link;

}

return(temporiginal);

}

move(pl1,n)

/* Processes a record of configuration */

configptr *pl1,n;

{

configptr hold1,hold2;

hold1 = (*pl1)->link;

hold2 = n->link;

(*pl1)->link = hold2;

n->link = hold2->link;

hold2->link = hold1;

(*pl1) = (*pl1)->link;

}

3.9.4 Merging Sequences of Entries

Now that the perfect shuffle has been developed, let's return to the merging of two sequences. Set the first n entries of configuration to one of the sequences to be merged, and the second n entries of configuration to the other, but enter the second in reverse order.

Consider the following algorithm for merging two sequences of ordered (largest to smallest) entries, each of length n:

Apply perfectshuffle and compareexchange to configuration a total of 2n times.

Compareexchange is a function that compares each of the n pairs of adjacent configuration entries and exchanges their values whenever the second is larger than the first. Table 3.3 shows the results obtained as this algorithm is applied to our sample sequences of length 4. Since lg 2n = lg 8 = 3, the algorithm terminates after the third perfect shuffle and compare-and-exchange operation. Notice that this results in configuration containing the correct merged sequence.

When 2n is a power of 2, say 2k, then the k iterations always produce the correct merge. This is certainly not obvious, and it is not proven here, but one point is important. If this merge is implemented as a for loop and carried out conventionally (that is, sequentially), it will take time O(n lg n). The straightforward sequential merge discussed earlier takes only O(n) time. However, if the n moves of perfectshuffle and the n pair comparisons and interchanges of compareexchange are all done in parallel (at the same time), then the total time will be just O(lg n).

Table 3.3 The Result of Merging Two Sequences of Ordered Entries of Length 4

    Initial       After First      After First      After Second

 Configuration  Perfect Shuffle  Compare Exchange  Perfect Shulle

-----------------------------------------------------------------

      100               100              100             100

       80                10               10              85

       65                80               80              10

       30                50               50              65

       10                65               85              80

       50                85               65              90

       85                30               90              50

       90                90               30              30

  After Second      After Third      After Third

Compare Exchange  Perfect Shuffle  Compare Exchange

---------------------------------------------------

      100               100              100

       85                90               90

       65                85               85

       10                80               80

       90                65               65

       80                50               50

       50                10               30

       30                30               10

Parallel processing is beyond the scope of this text, but this application illustrates its power and some of the difficulties involved. Doing parallel processing correctly is tricky, since care must be taken to do the right thing at the right time. For instance, one could dress in parallel fashion, but certain constraints must be observed. Socks and shirt can be put on at the same time (a valet is needed of course, since a person has only two hands) followed by pants, with shoes and jacket next and in parallel. But shoes cannot go on before socks.

The perfect shuffle (and the topic of parallel processing) is pursued further in Stone [1971, 1980] and Ben-Ari [1982]. An application of these ideas to the stable marriage problem appears in Hull [1984].

Exercises

1.     What record of a list has no pointer pointing to it from another list record? What record of a list has a null pointer in its link field?

2.      If each record of a list has, in addition to its information and link fields, a "pred" field, then the list is called a two-way list. The pred field contains a pointer to the preceding record. The first record's pred field contains a null pointer. Write insertion and deletion functions corresponding to those of Section 3.3 for a two-way list.

3.     A circular list is a list whose last record's link field contains a pointer to the first list record rather than a null pointer. Write insertion and deletion routines corresponding to those of Section 3.3 for a circular list.

4.     Modify the solution to Example 3.1 so it is correct for

a.     two-way list

b.     A circular list

5.     Write a function to interchange the records of a list that are pointed to by p1 and p2.

6.     The solution developed for Example 3.1 essentially cycles pointers from one list record to the next. Instead, a solution could cycle information field values from one record to the next. Modify the solution so that it does this and compare the two execution times. What if the information field contains only a pointer to the actual information field value?

7.     Consider the three arrays below--data, p, and s.

   DATA   P   S

---------------

0    1   -1   5

1   12    4   6

2   18    6   8

3    4    5   9

4    7    9   1

5    3    0   3

6   12    1   2

7  100    8  -1

8   20    2   7

9    5    3   4

An ordering of the integers in data is specified by p and by s . Arrays p[i] and s[i] point to the predecessor and successor of the integer stored in data[i]. A minus one indicates no predecessor or successor. Assume the first i elements of data, p, and s are properly set, as above, to reflect the usual ordering of integers. Suppose a new integer is placed into data[i+1].

a.     If i is 10 and the new integer is 15, what will the new p and s arrays be after they are properly updated to reflect the correct ordering among the eleven integers? The data array, except for 15 being placed in position 10, is to be unchanged.

b.     Write a program segment to update p and s after a new integer is placed in data. You might want to assume that first and last point, respectively, to the element of data containing the first and last integers in the ordering.

8.     a.     What does traverse (Section 3.4) do when process interchanges the record that recordpointer points to with its successor record?

b.     Same as Exercise 8(a) except that, in addition, process then sets recordpointer to the link field value of the record to which it points.

9.     Write a function to delete all the records from list l1 that also appear on the list 12.

In Exercises 10-13 you must turn traverse of Section 3.4 into a solution to the exercise by defining its functions properly.

10.     Write a function to delete all duplicate records from an alphabetical list such as that of Example 3.4.

11.     Write a function to insert a new record in front of the record pointed to by ptr. You should not keep track of a predecessor nor traverse the list to find a predecessor. A "trick" is involved here (see Section 3.3.4).

12.     This is the same as Example 3.4, but assume the list is implemented as a two-way list.

13.     Create a new list that is the same as l except that each record has been duplicated, with the duplicate inserted as the new successor of the record it duplicates.

14.     Create a list l that consists of all the records of the list l1 followed by all the records of the list 12.

15.     Write a function to print out the number of words for a list such as that of Example 3.2.

16.     Write the required functions for traverse of Section 3.4 when list records are stored in dynamic memory and the list is implemented as

a.     A two-way list

b.     A circular list

17.     A hospital has 100 beds. The array records contains records that represent, by floor and bed number, those beds that are not occupied. The first, second, and third elements of a record contain the floor number, bed number, and link value, respectively. How many beds are not occupied and what is the lowest floor with an available bed?

18.     Change the appropriate pointers for Exercise 17 so that the records are kept in order by floor and bed number.

19.     Show what the arrays floor, bed, and next might have in their entries if the beds list of Exercise 17 were implemented using three corresponding words of these arrays.

20.     How would the records array of Exercise 17 change if bed 2 on the fifth floor became occupied?

21.     Suppose bed 10 on the sixth floor became empty. Assuming that the records array is used only for the beds list, what changes might be made to it to reflect this new information?

22.     This is the same as Exercise 16 except that the list records are stored in an array records .

23.     A palindrome is a sequence of characters that reads the same from front to back as from back to front. ABCDDCBA, ABCBA, and NOON are palindromes. Write a function palindrome to return the value true if the list it is called to check represents a palindrome, and the value false otherwise. The list records are stored in dynamic memory.

24.     Suppose four records, 13, 7, 1, and 2, were deleted from lists whose records are stored in records of Figure 3.13. Then three records were inserted. If reclaim returns records at the front of availist, depict the availist and records after this processing.

25.     Do the same as in Exercise 24, but assume that reclaim returns records at the rear of availist.

26.     a.     Why isn't it reasonable for avail to delete a record from anywhere on availist except the front or rear?

b.     Why isn't it reasonable for reclaim to insert a record anywhere on availist except the front or rear?

27.     Suppose a record is deleted from a list stored in records, but reclaim is not invoked to return it to availist. Is it ever possible to reuse the storage relinquished by that record?

28.     Suppose records with different lengths are stored in records using the techniques of Chapter 2. The availist contains the available records in arbitrary order. Describe how avail and reclaim might be implemented.

29.     This is the same as Exercise 28, except that the availist contains the available records ordered by length.

30.     Write a function to create a list whose records are defined by

typedef struct node

{

whatevertype info;

struct node *linkptr;

struct node *prevptr;

}listnode;

The input should consist of a sequence of information field values given in the order they should appear in the list. Linkptr points to the next record on the list, and prevptr points to the preceding record on the list.

31.     Explain why pointer variables and the functions malloc and free remove the need for the programmer to manage storage for list records.

32.     Both the sequential array and list representations for ordered records use arrays. Why are they different?

33.     Suppose you are asked to store records that represent an inventory  of books that are currently signed out from the library. Assume that frequent requests are made for output that includes the book title and the current borrower in alphabetical order by the borrower's name. Of course, books are frequently taken out and returned. Should a sequential array be used for the records or a list representation? Why?

34.     Suppose records are stored using the sequential array implementation. Initially 1,000 records are stored. You are asked to delete every other record, starting with the first. After this deletion you must insert records before every third record, starting with the first record. Find an expression for the total number of shifts of records that must take place to accomplish these insertions and deletions.

35.     Do Exercise 34, but instead of assuming a sequential implementation assume a list implementation.

36.     Suppose you must access records in the order tenth, first, twelfth, thirtieth, and fiftieth. How much time will the five accesses take with a sequential representation, and how much time will they take with a list representation?

37.     Describe a real-life situation in which the sequential array representation of ordered records is clearly more desirable, another in which the list representation is clearly favored, and a third situation that involves trade-offs between the two representations.

38.     a.     Write the initial and original routines for the array implementation of the perfect shuffle and for the list implementation of the perfect shuffle in Section 3.9.

b.     Can original be reduced to checking only the nth card to see if it has returned to the initial configuration?

Suggested Assignments

1.     Write and execute a program to input a series of lists, each of which contains a number of sentences. After inputting all the lists, the program should invoke functions shorten and printlist to work on each list. Shorten is to print each sentence of the list it is to deal with whose length exceeds 25, and delete such sentences from the list; printlist is to print the resultant list. You should write shorten by defining the functions of traverse so as to turn traverse into shorten. Use dynamic memory for record storage. Don't forget to echo print all input and annotate all output. Make up some fun sentences. Store one character in each list record, and, for simplicity, assume all sentences end with a period.

2.     Modify your program (and run the new version) when three characters (instead of one) are stored per record. Hint: within a list, assume the first character of each sentence is stored in a new record rather than in the record containing the period of the previous sentence, even when it has the room.

Go to Chapter 4      Return to Table of Contents


Data Structures Using C: CHAPTER 4: INTRODUCTION TO RECURSION, STACKS,

 Return to Table of Contents

CHAPTER 4: INTRODUCTION TO RECURSION, STACKS, AND QUEUES

Presents recursion as a special case of the top-down approach to problem solving

Explains recursive programs

how to understand them

how to simulate them

how to write them

Shows how recursive programs may be translated by a compiler or written in a non-recursive language

Introduces two more data structures

the stack

the queue

Illustrates the use of a stack with operations

setstack

empty

push

pop

Case study--checking sequences for proper nesting

uses the stack and treats it as a data abstraction

4.1: What Is Recursion?

People can't help but use themselves--their own life experiences--as a point of reference for understanding the world and the experiences of others. Definitions, algorithms, and programs may also refer to themselves, but for a different purpose: to express their meaning or intention more clearly and concisely. They are then said to be recursive. Languages such as C that allow functions to call themselves are called recursive languages. When no self-reference is allowed, languages are said to be nonrecursive or iterative.

Self-reference is not without its pitfalls. You have most likely encountered the frustration of circular definitions in the dictionary, such as "Fashion pertains to style" and "Style pertains to fashion." From this you conclude that "Fashion pertains to fashion." This kind of self-reference conveys no information; it is to be avoided. "This sentence is false" is another type of self-reference to be avoided, for it tells us nothing. If true it must be false, and if false it must be true. Nevertheless, self-reference is an important technique in finding and expressing solutions to problems.

Recall that the top-down approach to problem solving entails breaking down an initial problem into smaller component problems. These components need not be related except in the sense that putting their solutions together yields a solution to the original problem. If any of these component problems is identical in structure to the initial problem, the solution is said to be recursive and the problem is said to be solved by recursion. If the solution and its components are functionally modularized, then the solution will refer to itself. Recursion is thus a special case of the top-down design methodology.

Suppose a traveler asks you, "How do I get there from here?" You might respond with a complete set of directions, but if the directions are too complex, or you are not sure, your response might be: "Go to the main street, turn left, continue for one mile, and then ask, "How do I get there from here?" This is an example of recursion. The traveler wanted directions to a destination. In solving the problem, you provided an initial small step leading toward the traveler's goal. After taking that step, the traveler will be confronted with a new version of the original problem. This new problem, while identical in form to the original, involves a new starting location closer to the destination.

Cursing is often used to belittle a problem; recursing has the same effect! Recursion, when appropriate, can give relatively easy-to-understand and concise descriptions for complex tasks. Still, care must be taken in its use, or else efficiency of storage and execution time will be sacrificed.

4.2: Using Recursion

Applying the recursive method is just like applying the top-down approach, but it can be tricky; it presents conceptual difficulties for many students. The purpose of this section is to show how to develop and understand recursive solutions. Some illustrative examples are provided to help you get the idea. Practice will help you become skilled in the use of recursion.

4.2.1 The Towers of Hanoi

A game called the Towers of Hanoi was purportedly played by priests in the Temple of Brahma, who believed that completion of the game's central task would coincide with the end of the world. The task involves three pegs. The version considered here has n disks initially mounted on peg 1 (the priests dealt with 64 disks). The n disks increase in size from top to bottom, the top disk being the smallest. The problem is to relocate the n disks to peg 3 by moving one at a time. Only the top disk on a peg can be moved. Each time a disk is moved it may be placed only on an empty peg or on top of a pile of disks of larger size. The task is to develop an algorithm specifying a solution to this problem, no matter what the value of n.

Consider the case when n = 4, shown in Figure 4.1(a). The following sequence of instructions provides a solution.

1. Move the top disk from peg l to peg 2.

2. Move the top disk from peg 1 to peg 3.

3. Move the top disk from peg 2 to peg 3.

4. Move the top disk from peg 1 to peg 2.

5. Move the top disk from peg 3 to peg 1.

6. Move the top disk from peg 3 to peg 2.

7. Move the top disk from peg 1 to peg 2.

8. Move the top disk from peg 1 to peg 3.

9. Move the top disk from peg 2 to peg 3.

10. Move the top disk from peg 2 to peg 1.

11. Move the top disk from peg 3 to peg 1.

12. Move the top disk from peg 2 to peg 3.

13. Move the top disk from peg 1 to peg 2.

14. Move the top disk from peg 1 to peg 3.

15. Move the top disk from peg 2 to peg 3.

(a) The Initial Problem

(b) After Component 1

(c) After Component 2

(d) After Component 3 -- the Solution!

Figure 4.1 The Towers of Hanoi Problem

It is not difficult to arrive at such a solution for a specific small value of n. Try to find a solution for n = 5. You will find that, even though you succeed after some trial and error, creating a solution for n = 25 is indeed a formidable task. You may even be too old to care by the time you finish. If a particular value of n gives so much trouble, how does one specify a solution for any n, when one does not even know what value n will have?

One answer is to apply the technique of recursion. This means attempting to break down the problem into component problems that can be solved directly or that are identical to the original problem but involve a smaller number of disks. The three component problems below fit the requirements of this framework.

1. Move the top n - 1 disks from peg 1 to peg 2.

2. Move the top disk from peg 1 to peg 3.

3. Move the top n - 1 disks from peg 2 to peg 3.

If these three problems can be solved, and their solutions are applied in sequence to the initial situation, then the result is a solution to the original problem. This can be seen from parts (a)-(d) of Figure 4.1, which show the consecutive situations after each is applied.

We now generalize the original problem:

Move the top n disks from peg i to peg f.

i denotes the initial peg on which the top n disks to be relocated reside, and f denotes the final peg to which they are to be relocated. Let a denote the third available peg. Solving the original problem, which has only n disks, is equivalent to solving the generalized problem with i taken as 1 and f taken as 3. This is because the moves that accomplish its solution will also work for the generalized problem, since any disks below the top n are larger than they are, so their existence doesn't render any of the moves illegal. Solving the first component problem is equivalent to solving the generalized problem with n replaced by n - 1, i by 1, and â by 2. It is now apparent that the original problem, and the components 1, 2, and 3, have structure identical to the generalized problem, but the components involve fewer disks. They also involve different initial, available, and final pegs. This is why it was necessary to generalize the original problem.

Suppose you have found a solution to the generalization and denote it by towers(n,i,a,f). Then, as just shown, the application of the solutions to components 1, 2, and 3, in sequence, will be a solution to the original problem. A recursive definition for towers(n,i,a,f) can now be expressed as follows. Note that moves in towers(n,i,a,f) are from peg i to peg f--that is, from the second parameter to the fourth parameter. The first parameter specifies the number of top disks, on the peg given by the second parameter, that are to be moved.

>Comment

To obtain towers(n,i,a,f):

If n = 1, then

>Comment

move the top disk from peg i to peg f

else

>Comment

apply towers(n - 1,i,f,a)

moves the top disks from peg i to peg f

>Comment

apply towers(n - 1,a,i,f).

This is the recursive algorithm for the Towers of Hanoi problem. Notice that the definition gives an explicit solution for the special case when n = l and an implicit solution for the case of n > 1.

To use the method of recursion we must recognize when it is applicable and find a correct way to generalize the original problem and then to break down the generalization. We have been successful and have found a recursive algorithm. Because no further refinement is needed, the corresponding recursive program can be written directly using functional modularization:

towers(n,i,a,f)

/* Moves the top n disks from peg i to peg f

*/

int n,i,a,f;

{

if(n == 1)

printf("\n %d -> %d\n",i,f);

else

{

towers(n-1,i,f,a);

[1]

printf("\n %d -> %d\n",i,f);

towers(n-1,a,i,f);

[2]

}

}

The labels [1] and [2] are not part of the program but are used later as references when explaining the program. It is essential to see that this function has been written just like any other; whenever a component task is available as a function, we invoke it to carry out that task. The feature that distinguishes it and makes it a recursive function is that the required functions happen to be the function itself. The references within towers to itself are allowed in recursive languages.

Anytime a function calls itself during its execution, a recursive call has been made. The two invocations of towers within towers represent such recursive calls. In general, when a recursive program executes on a computer, or when its execution is simulated, many recursive calls will be made. Simulation of a program or function call means carrying out its instructions by hand and keeping track of the values of all its variables. These are referred to as the first, second, etc., recursive calls. The initial call to the function precedes the first recursive call.

4.2.2 Verifying and Simulating a Recursive Program

The verification of a recursive program can be done in two ways, as is true for a nonrecursive program. One is by formal proof, the other by checking all cases. With recursion, the cases given explicitly are checked first. A case is given explicitly when the action taken by the program in that case does not require a recursive call. Also, it is assumed that each function individually works correctly. For towers(n,i,a,f), the only explicit case is when n = 1 and the function clearly outputs the proper result and terminates.

For the case when n exceeds l, Figure 4.1 has already confirmed that as long as the three component functions work properly, then towers(n,i,a,f) will also. Note that these components correspond to smaller values of n. Mathematically inclined readers may recognize that this really amounts to a proof of correctness for towers using mathematical induction on n. One of the advantages of recursive programs is that they may be proven correct more easily in this way.

In order to understand how a recursive program actually executes, it is necessary to simulate its execution. Below are the first few steps of the simulation for a call to towers(n,i,a,f) with the values 4, 1, 2, 3 -- that is, towers(4,1,2,3). It will soon become apparent that considerable bookkeeping is involved, requiring an organized way to do it.

Since n is greater than 1, the procedure requires the sequential execution of

towers(3,1,3,2)

printf("\n %d -> %d\n",1,3);

and

towers(3,2,1,3)

To do this requires suspending the execution of towers(4,1,2,3) at this point in order to simulate the first recursive call, towers(3,1,3,2). Note that after this recursive call is completed, 1 [image] 3 is to be output. (1 [image] 3 means "Move the top disk from peg 1 to peg 3.") Then another recursive call must be simulated, towers(3,2,1,3).

Dealing with towers(3,1,3,2), since n > 1, the commands

towers(2,1,2,3)

printf("\n %d -> %d\n",1,2);

and

towers(2,3,1,2)

must be carried out sequentially. Now the execution of towers(3,1,3,2) must be suspended in order to simulate the second recursive call, towers(2,1,2,3). Note that after this recursive call is completed, 1 [image] 2 is to be output, and then another recursive call must be simulated: towers(2,3,1,2).

Dealing with towers(2,1,2,3), since n > 1,

towers(1,1,3,2)

printf("\n %d -> %d\n",1,3);

and

towers(1,2,1,3)

must be carried out sequentially. These two calls to towers represent the third and fourth recursive calls. They both complete without generating any new recursive calls. The three statements result in the outputting of l [image] 2, 1 [image] 3, and 2 [image] 3, respectively. This completes the second recursive call (towers(2,1,2,3)) so we must pick up the simulation at the proper point, which is to output l [image] 2 and then make the fifth recursive call to simulate towers(2,3,1,2).

Figure 4.2 Bookkeeping for the Simulation of a Recursive Program

Enough is enough! At this point you may feel like a juggler with too few hands and too many recursive calls. Figure 4.2 represents a convenient way to keep track of the complete simulation; it may be generated as the simulation proceeds. The figure traces the recursive calls made to towers as the initial call to towers(4,1,2,3) is executed to completion. Each recursive call is numbered to show whether it is the first, second, . . . , or fourteenth. The bracketed return values indicate the labeled statement of procedure towers at which the preceding recursive call is to resume when the next call is completed. For example, when towers(3,1,3,2) is completed, towers(4,1,2,3) resumes at [l]. Our simulation stopped just as the fifth recursive call was to be carried out. The figure uses shorthand for the output. For example, l [image] 2 means, "move the top disk from peg l to peg 2." The solution is the same as our earlier one.

Figure 4.2 indicates very clearly what information must be available at each point in the simulation (or execution of the program) in order to carry it out. Specifically, it is necessary to know the following information:

The four parameter values of the call currently being carried out

Where to resume when the call currently being carried out is completed

For example, when the third recursive call is completed, resumption must be at the printf statement of the second recursive call; when the fifth recursive call is completed, resumption must be at the end of the first recursive call; and finally, after the initial recursive call is completed, resumption must be at the proper point in the program that called towers originally.

Notice that the last three parameters are never modified by towers, but the first parameter, which corresponds to the number of disks for the current problem, is modified. For instance, the first recursive call to towers(3,1,3,2) has modified the first parameter, making it 3 rather than 4 as it was initially. It is important that this modification be kept local to towers(3,1,3,2), not reflected back to towers(4,1,2,3), the initial call.

It is not always possible to find a recursive solution to a problem, nor is it always desirable to use one when available. Sometimes a recursive solution is the only one we can find. For the Towers of Hanoi problem it is difficult to find a nonrecursive solution, but a number are known. Recursive solutions normally make it easier to see why they are correct, probably because to achieve them requires that you make the structure inherent in the problem more explicit.

To make this point clearly, the recursive algorithm and a nonrecursive algorithm are repeated here. Convince yourself that the nonrecursive algorithm solves the Towers of Hanoi problem correctly (see Exercise 8). The nonrecursive program is taken from Walsh [1982]; another nonrecursive program appears in Buneman and Levy [1980].

A Recursive Solution for the Towers of Hanoi

To obtain towers(n,i,a,f):

If n = 1, then

move the top disk from i to f

else

apply towers (n - 1,i,f,a)

move the top disk from i to f

apply towers(n - 1,a,i,f).

A Nonrecursive Algorithm for the Towers of Hanoi

l. Label the disks l, 2, . . . , n in increasing order of size.

2. Label the pegs i, a, and f, n + 1, n + 2, and n + 3, respectively.

3. Move the smallest disk onto another disk with an even label or onto an empty peg with an even label.

4. While all disks are not on the same peg

a. move the second smallest top disk onto the peg not containing the smallest disk

b. move the smallest disk onto another disk with an even label or onto an empty peg with an even label.

The Tower of Babel failed for lack of communication. The construction workers did not speak the same language. While recursion allows the solution to the Towers of Hanoi to be specified conveniently, it is execution time that causes trouble. The diagram of Figure 4.1 shows that the construction of the new tower of disks will take 24 - 1 or fifteen moves. It is not difficult to generalize the diagram to n disks to conclude that 2n - 1 moves will be needed for n disks (see Exercise 7). Even with Olympic-caliber construction workers, it would take over one thousand years to complete the tower for 50 disks.

4.2.3 The Length of a List

We will now present a series of functions to further illustrate recursive programs. The functions

1. Determine the length of a list

2. Create a list that is a copy of another list

3. Count checkerboard placements

4. Generate all permutations of n integers and process each one. (This is applied to the stable marriage problem.)

In addition to providing repetition so that you can become more familiar with and understand recursion better, the examples provide a background to show how local and global variables and parameters are treated in recursive programs and why recursive solutions are not always the best. A list may be thought of as being either a null list or as one record followed by another list (the link field of the record plays the role of the head of this other list). This is nothing but a way to define a list recursively. Consider the recursive function defined by

length(listname)

/* Returns the number of records

in the list pointed to by listname.

*/

listpointer listname;

{

listpointer setnull(),next();

if(listname == setnull())

>Comment

return(0);

else

return(1 + length(next(listname)));

}

This function returns the length of the list, listname. To verify that this is correct we must check

1. The null list. It returns with length = 0.

2. A one-record list followed by another list. It returns length = 1 + the length of the list after the first record.

It is correct, since its value is correct for each case. Of course it is correct--after all, the length of a null list is zero, and the length of a list with at least one record is 1 for the record, plus the length of the rest of the list beyond the first record ! This was how length was written in the first place.

4.2.4 Copying a List

Consider the function copy, which creates a list second that is a copy of a list first:

copy(first,psecond)

/* Creates a new list which is a copy

of the list pointed to by first and

whose head is second

*/

listpointer first,*psecond;

{

listpointer null, rest,setnull(),avail(),next();

null = setnull();

if(first == null)

>Comment

*psecond = null;

else

{

>Comment

*psecond = avail();

    >Comment

setinfo(*psecond,first);

    >Comment

copy(next(first),&rest);

    >Comment

setlink(*psecond,rest);

}

}

When first is the null list, copy faithfully terminates with the copy also a null list. When first is a one-record list, copy sets second to point to storage allocated for a record and sets the information field value of this record to the value of the information field in first's first record. The recursive call to copy(next(first),&rest) is then an invocation of copy to copy the null list, since next(first) is null in this case. Assuming it performs its task correctly, rest will be null when it returns. The link field of the record pointed to by second is then set to the value of rest (the null pointer), and the initial call to copy terminates correctly.

Similarly, had there been more than one record in first's list, the recursive call to copy would have returned with rest pointing to a correct copy of the list pointed to by next(first). Setlink would correctly place a pointer to that correct copy into the link field of the record pointed to by second, and the initial call would terminate correctly. This verifies that copy is correct in all cases. Again, to copy the null list is easy. To copy a list with at least one record, simply copy that record, copy the rest of the list beyond the first record, and append that copy to the copy of the first record. This is how copy was written!

If you are confronted with a recursive program and want to understand it, approach it in the same way as you would a nonrecursive program. Assume that any functions that are called perform their functions correctly, and then attempt to see what the program itself does. Sometimes this will be apparent, as it was for towers and copy, and sometimes a simulation of the program helps to increase comprehension. Imagine, for example, that you were not told what copy did. It wouldn't have been so easy to find out without simulating some simple cases.

4.2.5 Local and Global Variables and Parameters

In order to understand what happens as a recursive program executes, it is necessary to review how local and global variables are treated, and also how parameters are treated. This must also be known to understand what happens when any program executes.

In nonrecursive programs, references to global and to static variables always refer to the current actual storage assigned to them. Hence any changes to their values during execution of the program will be reflected back in their values. Local automatic variables are treated differently. Local copies are made, and these are referred to during the execution of the function to which they are local. If the local variables are automatic, the storage for the local copies disappears when the function terminates or returns. If the local variables are static, the storage remains after the function is terminated, and the variable retains its value for the next time the function is called.

This situation holds for recursion also. Each recursive call to a function produces its own copies of automatic local variables. Thus there may be many different values for the same local variable--each applicable to its own invocation of the function. As with nonrecursion, there is only one storage allocation for a global or a static variable, and it holds the last value of the variable.

Parameters passed in the argument lists of functions are treated as variables local to the function. They are passed by value. New storage is established for each parameter. The storage is initially set to the value of the corresponding actual parameter when the function is invoked. In effect, this storage represents a local copy of the actual parameter. Any reference in the function is interpreted as a reference to the local copy only. This implies that any change to a parameter during the execution of the function will not be reflected back to the storage used for the actual parameter; it can not have its value changed by the function.

The only way to change the original value of a variable used as a parameter is to pass the address of the parameter instead of its value. This is done automatically in the case of arrays and character strings. When an array is passed as a parameter, the address of the first storage location in the array is passed. This negates the need to make local copies of all the values of the array in the function and thus waste memory. Addresses for parameters can also be passed as pointers A copy is made in the function and is local to the function. Using the pointer, the value of the parameter can be changed.

Here is a simple example.

simple (x,py)

int x, *py;

{

x = 1;

*py = 2;

*py = x + *py;

}

Suppose that simple is invoked with actual parameters a and b, containing 0 and 1 respectively. It is invoked by the statement, simple (a,&b). Within the function, x initially has a copy of a as its value--0 in this case. y refers to the storage for b--that is, y is a pointer to the storage for b. The first statement of simple causes x to assume the value l, the second statement causes the storage for b to contain 2, and the third statement sets the storage for b to 3. When the calling program continues, a will still be 0, but b will be 3.

Parameters of recursive programs are treated in just this way. Any time cursive call is made within a recursive program, the recursive call acts just like a call to any function. The only difference is that, since a recursive function may generate many recursive calls during its execution, many local copies of its parameters may be produced. References to a parameter by value during the execution of each recursive call will then mean the current local copy of that parameter. Reference to a parameter passed by a pointer to it always means the same actual parameter.

In the recursive version of copy, note that

      Null and rest are local variables of the initial call to copy and remain local variables in all subsequent recursive calls. Local copies of null and rest are thus made on any call to copy.

      First is passed by value and second by pointer.

      Whenever copy is executing and generates a recursive call to copy(next(first),&rest), the actual first parameter of copy is the current copy of next(first)--i.e., the copy associated with the currently executing call to copy. Any references to the first parameter during the generated recursive call refer to the new local copy associated with that call.

When first points to the list as shown in Figure 4.3, and copy(first, second) is executed, the sequence of events and memory configurations that will take place appear in Figure 4.4. First0, first1, first2, first3 refer to the local copies of the actual first parameter associated with the initial, first, second, and third recursive calls to copy, respectively. Similarly for the other variables. A question mark indicates an as yet undefined value. Be sure you understand Figure 4.4 and how it represents the execution of copy.

Figure 4.3 The List for COPY to Process

An alternative version of copy that returns a pointer to the copy of the list first would be as shown on page 160.

(a) The Situation after SETINF0 (*PSECOND, FIRST0)

(b) After SETINF0(REST1, FIRST1)

(c) After SETINF0(REST2, FIRST 2)

(d) After REST2:= NULL 3; i.e., SECOND:= NULL

(e) After SETLINK(REST1, REST2)

(f) After SETLINK(REST0, REST1)

Figure 4.4 Simulation of COPY(FIRST,&SECOND)

listpointer copy(first)

/* Returns a pointer to the first record of a new list which is a

copy of the list first.

*/

listpointer first;

{

listpointer null,second,setnull(),avail(),next();

null = setnull();

if (first == null)

    >Comment

second = null;

else

{

    >Comment

second = avail();

    >Comment

setinfo(second,first);

    >Comment

setlink(second,copy(next(first)));

}

return (second);

}

This version differs from the original version in two ways. One, it has only one parameter, the list to be copied, and two, it returns a pointer to the new copy rather than setting a second parameter to point to the new copy.

4.2.6 Counting Squares

The third example relates to counting squares. Consider an n X n checkerboard and determine the number of ways a 4 X 4 checkerboard can be placed on it. For example, if n = 5, there are four ways to place a 4 X 4 checkerboard on the 5 X 5 board, as shown below.

For this particular example the solution is easy to see by inspection. However, how do you state an algorithm that produces the result in all cases? This can be done handily using recursion. Suppose you know the number of ways, f(n - 1), in which the 4 X 4 board can be placed on an (n - 1) X (n - 1) board. An additional (n - 3) + (n - 4) placements result from an n X n board, because of the n cells

Figure 4.5 An n x n Checkerboard

added at the top and left sides of the (n - 1) X (n - 1) board, as shown in Figure 4.5. There are a total of f(n) = f(n - 1) + (n - 3) + (n - 4) placements for the n X n board, when n   [image]  5. If n = 4, f(4) = 1. So the following is true.

f(n) is

1 if n = 4

f(n - 1) + (n - 3) + (n - 4) if n    5

It is easy to write a recursive function f, with parameter n, to return the correct result for any n   [image]  4.

f(n)

int n;

{

if (n == 4)

    >Comment

return (1);

else

    >Comment

return(f(n-1)+(n-3)+(n-4));

}

If n = 7, the program would execute as follows:

    >Comment

f(7) = f(6) + (7 - 3) + (7 - 4)

    >Comment

f(6) = f(5) + (6 - 3) + (6 - 4)

    >Comment

f(5) = f(4) + (5 - 3) + (5 - 4)

    >Comment

f(4) = 1

    >Comment

f(5) = 1 + 2 + 1 = 4

    >Comment

f(6) = 4 + 3 + 2 = 9

    >Comment

f(7) = 9 + 4 + 3 = 16

A simpler, more direct solution can be obtained by noticing that the top left corner of the 4 x 4 board can be placed in the first, second, . . . , (n - 3)th position of the top row of the n X n board, resulting in (n - 3) different placements. The second row also allows (n - 3) placements. In fact, each of the first (n - 3) rows allows (n - 3) placements, for a total of (n - 3) X (n - 3) distinct placements.Thus f(n) = (n - 3) X (n - 3) for n   [image]  4. The program for this solution would be

f(n)

int n;

{

return((n-3)*(n-3));

}

The recursive solution will take time and storage O(n), whereas the nonrecursive solution requires constant time and storage. Obviously, this solution is better in every way than the recursive solution to this problem. Hence recursion does not always yield the best solution. Just as with any tool, it must be used judiciously. You wouldn't use a bulldozer to dig a two-foot hole in your yard.

4.2.7 Permutations

The final example returns to the stable marriage problem. Suppose you are asked to find a stable pairing for the n men and women of the stable marriage problem in Chapter 2, but you aren't aware of the algorithm discussed in Section 2.5. Probably the most straightforward approach is to search through all the possible pairings until a stable pairing is found. Each possible pairing may be thought of as a specific permutation of the n women. Thus the example pairing of Table 2.2 corresponds to the permutation (or listing) 4, 3, 1, 2, 5. In general there are n ! such permutations (where n! means n factorial).

This approach yields a program that traverses the collection of permutations, processing each one and halting when a stable permutation is encountered. Stable may be conveniently invoked to check a permutation for stability. In contrast, the algorithm of Section 2.5 avoids such an exhaustive search by eliminating some permutations from consideration altogether. Still, many problems require just such an exhaustive search and contain a component to produce the permutations. The problem is to write a program to carry out the traversal. The function permutations does this:

permutations(n)

/* Generates and processes each

permutation of the integers

1 to n.

*/

int n;

{

int done;

done = FALSE;

    >Comment

initialize(n);

while (!done)

{

    >Comment

process(n);

    >Comment

next(n,&done);

}

}

where next (n,&done) must return, having updated the current permutation to the next permutation, unless the current permutation is the last one, when it simply sets done to true.

The problem is now to refine next. Suppose n = 4 and the first permutation is 3 2 1. Inserting 4 in all possible positions among 1, 2, and 3 gives the first four permutations of 1, 2, 3, and 4.

4 3 2 1

3 4 2 1

3 2 4 1

3 2 1 4

Removing 4 leaves 3 2 1. The next four permutations for n = 4 can be obtained by first obtaining the next permutation after 3 2 1 for n = 3. But this next permutation for n = 3 can similarly be obtained by shifting 3 to the right one place, yielding 2 3 1. Then the next four permutations for n = 4 are

4 2 3 1

2 4 3 1

2 3 4 1

2 3 1 4

Removing 4 leaves 2 3 1. Again, the next permutation after 2 3 1, 2 1 3, is obtained by shifting 3 right. The next four permutations for n = 4 are

4 2 1 3

2 4 1 3

2 1 4 3

2 1 3 4

Continuing this process will produce, in turn,

4 3 1 2    4 1 3 2           4 1 2 3

3 4 1 2    1 4 3 2           1 4 2 3

3 1 4 2    1 3 4 2           1 2 4 3

3 1 2 4    1 3 2 4    and    1 2 3 4

Generalizing, we see that, given a permutation of the n integers from 1 to n, the next permutation can be generated by shifting n to the right whenever possible. The only time this is not possible is if n is already at the right end. The next permutation must then be generated by removing n and replacing the leftmost n - 1 integers, which must be a permutation of the integers from 1 to n - 1, by their next permutation. If n is 1, then the last permutation has been produced. This provides a recursive definition for next.

next (n,pdone)

If n is not 1, then

If n is not at the right end of the current permutation of 1, . . , n then

shift it right one position

else

remove "n" from the current permutation of 1, . . , n

next(n - 1,pdone)

Insert "n" at the left end of the current permutation of 1, . . , n - 1

else

set done to true.

In order to proceed, a data structure to store the permutation must be chosen. To illustrate the effect of this selection, we will use two different structures--arrays and lists. We start by storing the permutation in an array p. Searching for the location of n in p will be avoided by keeping an array l of pointers. Both p and l are defined globally. l[n] points to the location of n in p when next(n,pdone) is invoked. Initialize must set p[i] to n - i + 1 and l[i] to 1 for 1   [image]  i   [image]  n. Next may be defined as

next(n,pdone)

/* Generates the next permutation unless

the current one is the last. In this case

done is set to true.

*/

int n,*pdone;

{

int i;

    >Comment

if (n > 1)

if (l[n] < n)

{

    >Comment

p[l[n]] = p[l[n]+1];

p[l[n]+1] = n;

l[n] = l[n]+1;

}

    >Comment

else

{

    >Comment

next(n-1,pdone);

    >Comment

for (i=n-1;i>=1;i--)

p[i+1] = p[i];

p[1] = n;

l[n] = 1;

}

    >Comment

else

*pdone = TRUE;

}

Figure 4.6(a) simulates the execution of next(4,pdone) when the current permutation is 2 1 3 4.

Now suppose p is implemented as a list whose records are stored in dynamic memory. We need the pointer arrays ptr and pred, which are defined globally, as is the record pointer p. When next(n,pdone) is invoked, ptr[n] will contain a pointer to the record containing n in its information field, and pred[n] will contain a pointer to the predecessor of that record in p. To simplify list shifts and the insertion of a record at the front of p, a dummy record is used as the first record of p. Pred[k] will initially contain a pointer to the dummy record, and ptr[k] will initially contain a pointer to the record of p that contains k, for l   [image]  k   [image]  n. Now next may be defined as follows:

next(n,pdone)

/* Generates the next permutation unless

the current one is the last. In this case

done is set to true.

*/

int n,*pdone;

{

    >Comment

if (n > 1)

if (ptr[n]->link != NULL)

{

    >Comment

pred[n]->link = ptr[n]->link;

pred[n] = pred[n]->link;

ptr[n]->link = pred[n]->link;

pred[n]->link = ptr[n];

}

    >Comment

else

{

pred[n]->link = NULL;

    >Comment

next(n-l,pdone);

    >Comment

ptr[n]->link = p->link;

p->link = ptr[n];

pred[n] = p;

}

    >Comment

else

*pdone = TRUE;

}

Figure 4.6(b) simulates the execution of next(4,pdone) when the current permutation is 2 l 3 4.

(a) Array Implementation for P

(b) List Implementation for P

Figure 4.6 Simulation of NEXT (4,&DONE) When P is 2134

In comparing the two implementations notice that the advantage of the list implementation is that the for loop required by the array implementation is no longer needed. Inserting n at the left end of the current permutation thus takes only two pointer changes rather than time proportional to n. Other permutation algorithms are explored in Sedgewick [1977].

To further illustrate the differences, two complete programs using function permutations, one for the array and one for the list implementation, are given below. The process function prints each permutation. To adapt these programs to obtain a solution for the stable marriage problem, the function process must test for stability. Another modification would be to abort the execution when a stable pairing is found. This requires that process be given access to done so that it can set done to true at that point.

Array Implementation

#include <stdio.h>

#define TRUE 1

#define FALSE 0

    >Comment

typedef int arraytype[21];

    >Comment

arraytype p,l;

main()

/* Driver for permutations. */

{

int n;

printf("\n n = ?\n");

scanf("%d",&n);

permutations(n);

}

permutations(n)

/* Generates and processes each

permutation of the integers

1 to n.

*/

int n;

{

int done;

done = FALSE;

    >Comment

initialize(n);

while (!done)

{

    >Comment

process(n);

    >Comment

next(n,&done);

}

}

initialize(n)

/* Creates initial permutation */

int n;

{

int i;

    >Comment

for (i = 1;i<=n;i++)

{

p[i] = n - i + 1;

l[i] = 1;

}

}

process(n)

/* Prints the current permutation */

int n;

{

int i;

    >Comment

for (i=1;i<=n;i++)

printf("\n%d\n",p[i]);

printf("\n");

}

next(n,pdone)

/* Generates the next permutation unless

the current one is the last. In this case

done is set to true.

*/

int n,*pdone;

{

int i;

if (n > 1)

if (l[n] < n)

{

p[l[n]] = p[l[n]1];

p[l[n]+1] = n;

l[n] = l[n]+1;

}

else

{

next(n-1,pdone);

for (i=n-1;i>=1;i--)

p[i+1] = p[i];

p[1] = n;

l[n] = 1;

}

else

*pdone = TRUE;

}

List Implementation

#include <stdio.h>

#define TRUE l

#define FALSE 0

#define NULL 0

    >Comment

typedef struct record

{

int info;

struct record *link;

}listrecord,*listpointer;

typedef listpointer pointerarray[21];

    >Comment

pointerarray ptr,pred;

    >Comment

listpointer p;

main()

/* Driver for permutations. */

{

int n;

printf("\n n = ?\n");

scanf("%d",&n);

permutations(n);

}

permutations(n)

/* Generates and processes each

permutation of the integers 1 to n.

*/

int n;

{

int done;

done = FALSE;

    >Comment

initialize(n);

while (!done)

{

    >Comment

process(n);

    >Comment

next(n,&done);

}

}

initialize(n)

/* Creates initial permutation */

int n;

{

int i;

listpointer q;

p = malloc(sizeof(listrecord));

q = P;

    >Comment

for (i=1;i<=n;i++)

{

q->link = malloc(sizeof(listrecord));

q = q->link;

q->info = n - i + 1;

ptr[n-i+1] = q;

pred[i] = p;

}

q->link = NULL;

}

process(n)

/* Prints the current permutation */

int n;

{

listpointer q;

q = p->link;

    >Comment

while (q != NULL)

{

printf("\n%d\n",q->info);

q = q->link;

}

printf("\n");

}

next(n,pdone)

/* Generates the next permutation unless

the current one is the last. In this case

done is set to true.

*/

int n,*pdone;

{

if (n > 1)

if (ptr[n]->link != NULL)

{

pred[n]->link = ptr[n]->link;

pred[n] = pred[n]->link;

ptr[n]->link = pred[n]->link;

pred[n]->link = ptr[n];

}

else

{

pred[n]->link = NULL;

next(n-1,pdone);

ptr[n]->link = p->link;

p->link = ptr[n];

pred[n] = p;

}

else

*pdone = TRUE;

}

4.3: A Close Look at the Execution of Recursive Programs

When any program invokes one of its functions, it is because the task performed by that component must now be carried out. Global variables and variables passed by address (pointers) specify storage that is not local to the component but that it may reference and change. It may also reference and change static variables. All such storage represents the data the component is to process. Any changes by the component to the data are reflected back in the data's storage.

Parameters passed to a function by value also specify storage that is not local to it, but these parameters are treated differently. When the component is invoked, their values are copied into storage that is local to the component. It is these local copies that are referenced, and perhaps changed, by the component as it executes, but such changes are not reflected back to the original values. In this way the information passed by value is made accessible to the component, which may even change the local copies, but such changes are kept local; the original nonlocal values are untouched.

In effect, these copies are treated as if they were additional local variables of the component. The actual local variables of the component represent temporary information that the component uses to do its job. Both the local copies and local variables disappear when the task is completed. This storage for local copies and local variables serves as a "scratchpad" for use by the component. The scratchpad contains information used temporarily by the component to perform its task on the data. This view of the data and scratchpad of a component is depicted in Figure 4.7.

The component may, in turn, invoke another component, which may invoke still another, and so on. Each time this is done, the calling component suspends its operation, so all components of the sequence, except the last, are waiting.

When the last component completes its task, the component that invoked it then continues. In order for this to happen, each component must remember from whence it was called. Also, upon return to the correct place, the scratchpad of the calling component must be the same as before the call. For nonrecursive programs, the important point is that each component must retain only one returning point, and the scratchpad of a component is automatically retained when it invokes another component.

With recursive programs the situation is considerably different, as evidenced by Figures 4.2, 4.4, and 4.6. First, when a recursive function is invoked, this may be the initial, first, second, or nth recursive call to it. If it is the nth recursive call, there may be as many as (n + 1) returning places to be retained, not just one. Second, there may be (n + l) scratchpads to be retained. How many there are is determined by the number of calls that are currently suspended.

Figure 4.7 Data and Scratchpad for a Component

Suppose that this number is k, that the return for the initial call is handled separately, and also that the scratchpad for the currently executing call is associated with the function. The number of sets of additional returns and scratchpads that must be retained is then k. The depth of a recursive program is given by the maximum value of k. It is the depth that determines the amount of storage required by the function. The following examples illustrate these ideas.

Example 4.1

In Figure 4.2, when the tenth recursive call (k = 3) is made to towers, there are three returns and three sets of scratchpads to remember in addition to the initial call's return.

[2] and n = 4, i = l, a = 2, f = 3 for the initial call

[1] and n = 3, i = 2, a = 1, f = 3 for the eighth recursive call

[1] and n = 2, i = 2, a = 3, f = 1 for the ninth recursive call

n = 1, i = 2, a = 1, f = 3 for the tenth recursive call are stored, by our assumption, local to towers. In general, the depth of recursion will be n - l. Using our terminology, there are no data.

Example 4.2

In Figure 4.4, when the third recursive call is made to copy, there are three returns and three sets of scratchpads to be retained beside the initial call's return:

first0, null0, rest0 for the initial call

first1, null1, rest1 for the first recursive call

first2, null2, rest2 for the second recursive call

No returns are shown since they all return to the same place: the statement setlink(*psecond, rest). The depth of the recursion is one less than the length of the list to be copied. The data for the initial call is second, while for the (n + 1)th recursive call it is restn. A change to rest in the (n + 1)th call is reflected in the scratchpad for the nth call. Storage could be saved by making null global to copy.

Example 4.3

In Figure 4.6 the situation corresponds to a point in the initial call when the second and then the third recursive calls to next are made. The local variable i is not shown, and the three scratchpad values are

n = 4 for the initial call

n = 3 for the second recursive call

n = 2 for the third recursive call

Again, the returns are all the same, to the statement following the recursive call to next, so they are not shown. The depth of the recursion is n - 1. The data consist of p, ptr, pred, and done.

It should by now be clear that the storage required by a recursive function is proportional to its depth. The execution time, on the other hand, has to do with the total number of recursive calls that are required to complete the initial call. This explains why the recursive solution to Counting Squares (Section 4.2.6) is so inefficient both in time and in storage.

4.4: Implementing Recursive Programs

Recursion is a powerful tool for the development of algorithms and leads to concise, clear recursive programs that can be easier to check for correctness. Complex algorithms may be expressed using relatively few program statements, which at the same time emphasize the structure of the algorithm. This will become even more apparent later in the text, as more difficult problems are solved.

A function that invokes itself is directly recursive. A function is indirectly recursive if any sequence of calls to other functions can lead to a call to the original function. Compilers for recursive languages must translate recursive as well as nonrecursive programs. It is not necessary to understand how a recursive program is translated (any more than it is necessary to know how a nonrecursive program is translated) in order to understand or to write the program itself. But it is necessary to know how the program will execute. This amounts to being able to simulate its execution.

The programmer constrained to a nonrecursive language can still use the power of recursion. This is done by writing a program as if the language allowed recursion, and then translating the program into an equivalent nonrecursive program. This is what compilers for recursive languages do and is one reason for spending time finding out how to translate (or implement) a recursive program.

As shown in the discussion of Counting Squares, some problems should not be solved using recursion under any circumstances. Sometimes, as in the Towers of Hanoi problem, we may only be able to think of a recursive solution. Often we can find both a recursive and a nonrecursive solution, which may be compared for trade-offs in clarity, conciseness, storage, and execution time.

Even though a recursive solution may not be best, it may suggest an approach to the problem that otherwise might have been missed and that may turn out to be the most desirable. In fact, the translated version of a recursive solution can often be helpful in finding desirable modifications. This is another reason for looking at the translation process.

Normally, if a recursive approach is right for a problem, and is applied intelligently, further refinements of the program do not improve its efficiency significantly. Breaking up a problem as required by the recursive approach does, however, create the possibility of doing certain tasks in parallel. For instance, the three components of the Towers of Hanoi solution may all be done at the same time with parallel processing. Languages and computer systems that allow concurrent processing will surely become more prevalent. When they do, recursive solutions will have another advantage.

If each successive call to a recursive function were implemented so that during execution a new copy of the function is created with its own storage for the return and its scratchpad values, then, in effect, the function could be treated like any function in a nonrecursive program. With the advent of cheaper and larger storage, this may well occur in the future, but until then, another means of managing the generated returns and scratchpad values is needed.

The essential difference between nonrecursive (or iterative) programs and recursive programs is the storage needed for the returns and scratchpad values of the latter. Figures 4.2, 4.4, and 4.6 actually represent the bookkeeping needed to simulate the execution of a recursive program. The basic idea in translating such programs is to implement this bookkeeping procedure as follows:

1. Each time a recursive call is made to the function,

     a. save the proper return place and the current scratchpad values,

     b. set the scratchpad values to their new values.

2. Each time a recursive call is completed,

     a. restore the current return and scratchpad values to those that were last saved (and save them no longer),

     b. return to the proper place in the recursive function, or to the original calling component if it is the initial call that has just finished.

This procedure does not apply to a function that returns a value, but it may be modified to do so. Or the function may be modified to return the value in an additional parameter passed by pointer. As an illustration of this, the first version of copy might be obtained by such a modification of the second.

Notice that the information to be retained (return and scratchpad values) must be recalled in the order opposite to the order in which it is generated. This is no different for nonrecursive programs, except that then the information is saved locally with each component. It is helpful to think of return and scratchpad values as the entries of a record. Then the collection of records corresponding to each suspended recursive call must be stored in some data structure. The data structure must retain not only these records, but also the correct order.

In step 1 above, the record to be saved must become the new first record in the order. In step 2, the current first record (the last saved) must be removed from the data structure. A data structure for storing information so that new information can be inserted, so that a deletion always removes the last piece of information inserted, and which can be tested to see if it is empty (contains no information) is called a stack. Stacks will be discussed in more detail later in the chapter; for now they will simply be used.

4.4.1 A Sample Implementation of Towers

The stack is the most convenient data abstraction to use for the bookkeeping recursion entails. To illustrate the translation, and the possibility of enhancing efficiency, the recursive functions towers and next will now be translated into nonrecursive functions.

In towers there are two recursive calls. The idea in implementing it is to write a nonrecursive program that executes just as the recursive program would, while managing the additional records generated by the recursion. This amounts to implementing the procedure described for simulating towers. Each recursive call will be replaced by code to insert the proper return indicator onto a stack, as well as the current scratchpad values. Whenever the end of a recursive call is reached, the proper return indicator and the scratchpad values last put on the stack will be removed and restored to the program's local storage. The earlier recursive version and its implementation as a nonrecursive function are as follows .

Recursive Version

towers(n,i,a,f)

/* Moves the top n disks

from peg i to peg f

*/

int n,i,a,f;

{

if(n == 1)

printf("\n %d -> %d\n",i,f);

else

{

    >Comment

towers(n-1,i,f,a);

printf("\n %d -> %d\n",i,f)

    >Comment

towers(n-1,a,i,f);

}

}

Nonrecursive Version

towers(n,i,a,f)

/* Moves the top n disks

from peg i to peg f

*/

int n,i,a,f;

{

    >Comment

int retrn;

    >Comment

stack s;

    >Comment

setstack(&s);

    >Comment

one: if(n == 1)

printf("\n %d -> %d\n",i,f);

else

{

    >Comment

s_tack(1,n,i,a,f &s);

    >Comment

setvar1(&n,&i,&a,&f);

    >Comment

goto one;

}

    >Comment

two: if(!empty(&s))

{

    >Comment

restore(&retrn,&n,&i,&a,&f,&s);

switch(retrn)

{

    >Comment

case 1:

printf("\n %d -> %d\n",

i,f)

s_tack(2,n,i,a,f,&s);

setvar2(&n,&i,&a,&f);

goto one;

    >Comment

case 2:

goto two;

}

}

}

S_tack is a routine that inserts its arguments, as fields of a record, onto the stack. The name s_tack is used to be sure the compiler can distinguish this routine from the type name stack that also appears in the function. Setstack ensures that the stack will initially contain zero entries; empty returns true if the stack contains no records and false otherwise. The setvar routines do the proper setting of the current scratchpad values for the ensuing recursive call. Restore(&retrn,&n,&i,&a,&f,&s) restores the proper scratchpad values when a return is to be made after completion of some recursive call.

Setvar1(&n,&i,&a,&f) sets n to n - 1 and interchanges the values of f and a.

Setvar2(&n,&i,&a,&f) sets n to n - 1 and interchanges the values of a and i.

The "implicit" exit of the recursive program (after the second recursive call) has been replaced in the implementation by a complex "explicit" sequence of statements. This is because the exit must, in effect, transfer control to the proper place in the calling program. If the stack itself is empty at this point, this means that it is the initial call to towers that has been completed. Return should then be to the calling program. Otherwise, it is some recursive call to towers that has been completed. At this point, the top stack record contains the four scratchpad values to be restored and the return location in towers.

You should simulate the execution of this implementation to see that it really works correctly. The statement labeled "one" corresponds to the first executable statement of the recursive program; this is the statement that is invoked in the implementation to represent a recursive call. It is only invoked, however, after the scratchpad values of the current call and proper return location have been saved on the stack, and the scratchpad values for the upcoming recursive call have been correctly set.

Notice that the implementation requires a stack that can hold the stored information for up to n - 1 recursive calls. The stack must be initialized to empty. Its contents must be preserved between calls. We will not dwell on its implementation details here, but a complete program to test this version is given in Section 4.5.1.

This example highlights the fact that considerable overhead is involved in executing recursive programs. Time is consumed because variables must be stacked and unstacked for each recursive call and its return. It is often possible to discover a more efficient nonrecursive algorithm for a problem, even though the recursive algorithm is evident, as was done in Counting Squares. Nonetheless, the recursive version is frequently clearer, easier to understand, and more concise, as well as easier to verify. Therefore some programmers prefer recursive solutions to problems, even if they are less efficient than the nonrecursive versions.

Another approach is to attempt to increase the efficiency of the iterative program (the implementation) that is translated from a recursive program. This attempt can be made whether or not there is a competing nonrecursive routine.

How can we create a more efficient version of a translated program such as towers? The reason for the stack in the first place is to save return codes and to save scratchpad values that are needed when the suspended call is ready to resume (after the saved values have been unstacked and restored, and the proper program segment is to be executed). If the programmer can determine where the return should be, or what the scratchpad values that are to be restored should be, then this information does not have to be stacked. It may not be necessary to save the values of some other variables. It may also be possible to eliminate some return codes. If no variables need to be saved, and there is only one return code, then the stack itself is not needed. Let us attempt to reach this goal for the implementation of the recursive towers routine.

Notice that in the case statement for retrn at 1, the call to s_tack(2,n,i,a,f,&s) saves the values of the current variables in the suspended call, in order to make the recursive call to towers(n-1,a,i,f). When this new recursive call is completed, these current values will be restored, and the retrn will be 2. The statement labeled "two" will then be executed.

If the stack is empty, the initial call is complete. Otherwise restore will be called immediately and control transferred to the new value of retrn. The current values saved by s_tack(2,n,i,a,f,&s) will never be used, and so need not be saved at all. Thus the call to s_tack( 2,n,i,a,f,&s ) may be eliminated from the program. This means that the only return code appearing will be l in the call to s_tack(1,n,i,a,f,&s). However, the value of retrn after restore is called will always be 1. Therefore, no return code needs to be stored on the stack; the case structure may be removed. Retrn is not needed at all. This new program then looks like the following.

Second Nonrecursive Version

towers(n,i,a,f)

/* Moves the top n disks

from peg i to peg f

*/

int n,i,a,f;

{

stack s;

setstack(&s);

one: if(n == 1)

printf("\n %d -> %d\n",i,f);

else

{

s_tack(n,i,a,f,&s);

setvar1(&n,&i,&a,&f);

goto one;

}

if(!empty(&s))

{

    >Comment

restore(&n,&i,&a,&f,&s);

printf("\n %d -> %d\n",i,f);

setvar2(&n,&i,&a,&f);

goto one;

}

}

Another improvement can be made by changing the if-else structure to a while loop. Doing so improves the clarity and eliminates the goto:

one: if(n == 1)                    one: while(n > 1)

printf("\n %d -> %d\n",i,f);          {

else                                          s_tack(n,i,a,f,&s);

{                                         setvar1(&n,&i,&a,&f);

s_tack(n,i,a,f,&s);                 }

setvar1(&n,&i,&a,&f);            printf("\n %d -> %d\n",i,f);

goto one;

}

This yields

Third Nonrecursive Version

towers(n,i,a,f)

/* Moves the top n disks

from peg i to peg f

*/

int n,i,a,f;

{

stack s;

one: while(n > 1)

5 while loop replaces if-else construct

{

s_tack(n,i,a,f,&s);

setvar1(&n,&i,&a,&f);

}

printf("\n %d -> %d\n",i,f);

if(!empty(&s))

{

restore(&n,&i,&a,&f,&s);

printf("\n %d -> %d\n",i,f);

setvar2(&n,&i,&a,&f);

goto one;

}

}

Finally, the function can be written in more structured form, as follows:

Final Nonrecursive Version

towers(n,i,a,f)

/* Moves the top n disks

from peg i to peg f

*/

int n,i,a,f;

{

stack s;

int done

done = FALSE;

while(!done)

{

while(n > 1)

{

s_tack(n,i,a,f,&s);

setvar1(&n,&i,&a,&f);

}

printf("\n %d -> %d\n",i,f);

if(!empty(&s))

{

restore(&n,&i,&a,&f,&s);

printf("\n %d -> %d\n",i,f);

setvar2(&n,&i,&a,&f);

}

else

done = TRUE;

}

This structured version is more efficient than the compilerlike direct translation because the number of stack operations has been reduced. It is also structured for clarity in reading. Both changes make it a better implementation.

4.4.2 A Sample Implementation for Permutations

The same approach can now be applied to develop a nonrecursive translation for next.

Nonrecursive Version

next(n,pdone)

/* Generates the next permutation unless

the current one is the last. In this case

done is set to true.

*/

int n,*pdone;

{

    >Comment

int i,retrn;

    >Comment

stack s;

    >Comment

one: if(n > 1)

if(l[n] < n)

{

p[l[n]] = p[l[n] + 1];

p[l[n] + 1] = n;

l[n] = l[n] + 1;

}

else

{

    >Comment

s_tack(1,n,i,&s);

    >Comment

setvar(&n,&i);

    >Comment

goto one;

}

else

*pdone = TRUE;

    >Comment

two: if(!empty(&s))

{

    >Comment

restore(&retrn,&n,&i,&s);

switch(retrn)

{

    >Comment

case 1:

"explicit" exit must be made

{

for(i=n-1;i>=1;i--)

p[i+1] = p[i];

p[1] = n;

l[n] = 1;

    >Comment

goto two;

{

{

{

{

where setvar(&n,&i) simply sets n to n-1.

The same program can be written in a more structured way. First notice that the statement labeled "two" and the goto "two" statement create a while loop. Introducing that while loop but retaining the structure of "one," the overall structure of the program becomes as follows

    >Comment

one: if c1 then

    >Comment

if c2 then

task c1 and c2 both true

else

task for c1 true but c2 false

goto one

else

task for c1 false

    >Comment

while not empty(&s)

loop task

Next, the structure of "one" can also be improved, as follows.

    >Comment

while (c1 and not c2)

task for c1 true but c2 false

    >Comment

if c1, then

task for c1 and c2 both true

else

task for c1 false

    >Comment

while not empty (&s)

loop task.

The logic of this version is much easier to follow. You can see at a glance that the segment control flows from the top down. It consists of a loop, followed by a decision, followed by a loop. The iterative version of next, translated from the recursive next, and using the improved structure, now becomes as follows:

Second Nonrecursive Version

    >Comment

next(n,pdone)

/* Generates the next permutation unless

the current one is the last. In this case

done is set to true.

*/

int n,*pdone

{

int i,retrn;

stack s;

while((n > 1) && !(l[n] < n))

{

s_tack(1,n,i,&s);

setvar(&n,&i);

}

if(n > 1)

{

p[l[n]] = p[l[n] + 1];

p[l[n] + 1] = n;

l[n] = l[n] + 1;

}

else

*pdone = TRUE;

while(!empty(&s))

{

restore(&retrn,&n,&i,&s);

switch(retrn)

{

case 1:

{

for(i=n-1;i>=1;i--)

p[i+1] = p[i];

p[1] = n;

l[n] = 1;

}

}

}

}

In general, when the value of a scratchpad variable does not change or does not need to be maintained between recursive calls, it need not be stacked. This is the case with the variable i of next. Also, when there is only one return place from a recursive call, there is no need to stack a return indicator. This is the situation with next. Finally, the value of n varies in a predictable fashion. Its restored value will always be 1 greater than its current value. Thus there is no need to stack n. Each time a recursive call returns, n may be correctly restored by adding 1 to its current value.

Therefore, the stack is unnecessary, since no information must be retained on it. The s_tack function can be removed, setvar can be replaced by the statement n = n-1, and restore can be replaced by the statement n = n+1. Still, the stack controls the second while loop in our translated version. Even though no information needs to be stacked, it is still necessary to know when the stack would have been empty so that the loop may be properly exited.

Notice that the first while loop decreases n by l every time a stack entry would have been made, and the second while increases n by l every time an entry would have been deleted from the stack. Hence, if an integer variable s is set to n initially, prior to the first while loop, the test to see if the stack is empty in the second while loop becomes s = n. The final version of next is an iterative program that has no stack. It is an example of the translated version giving us a better solution. This four-step solution is efficient and is also clear.

1. Set s to n.

2. Set n to the rightmost entry in the permutation that can still be shifted right.

3. If that entry is not l, then

Shift it right

else

Set done to true

4. Place the s-n entries that could not be moved to the right in the proper order at the left end of the permutation.

The final program is as follows:

Final Nonrecursive Version

next(n,pdone)

/* Generates the next permutation unless

the current one is the last. In this case

done is set to true.

*/

int n,*pdone

{

int i,s;

1. s = n;

2. while((n > 1) && !(l[n] < n))

n--;

3. if(n > 1)

{

p[l[n]] = p[l[n] + 1];

p[l[n] + 1] = n;

l[n] = l[n] + 1;

}

else

*pdone = TRUE;

4. while(s > n)

{

n++

for(i=n-1;i>=1;i--)

p[i+1] = p[i];

p[1] = n;

l[n] = 1;

}

}

By translating the recursive solution and analyzing it, a highly efficient nonrecursive solution was built. A little insight can help a lot. For more on recursion, see Barron [1968] and Bird [1977a, 1977b]. It may now be time for you to make a recursive call on all preceding material!

4.5: Stacks

Earlier in this chapter the notion of a stack was introduced and used in the nonrecursive implementations of towers and next. Stacks are useful in situations requiring the retention and recall of information in a specific order--namely, when information is removed in a last-in, first-out order (LIFO for short). Removing tennis balls from a can is LIFO. In programming, stacks are useful when a program must postpone obligations, which must later be fulfilled in reverse order from that in which they were incurred.

Information is kept in order in a stack. Additional information may be inserted on the stack, but only at the front or top end. Thus the information at the top of the stack always corresponds to the most recently incurred obligation. Information may be recalled from the stack, but only by removing the current top piece of information. The stack acts like a pile of playing cards to which we can only add a card at the top and from which we can only remove the top card. Of course, this is not what is meant by a  "stacked deck."

The stack, with the operations of front-end insertion and deletion, initialization, and an empty check, is a data abstraction. Stacks used to be called "pushdown stacks"; insertion of data onto the stack was referred to as "pushing" the stack, and deleting data was known as "popping" the stack. So far we have written programs that treat the stack with the four operations as a data abstraction. Stacks will be used frequently in algorithms developed throughout the rest of this book, and it is now time to consider this implementation. Of the many implementations possible for the stack, the following subsections illustrate the three most straightforward.

4.5.1 Array Implementation of a Stack

A stack may be implemented using arrays with the following declarations:

    >Comment

typedef struct

{

whatever info;

}stackrecord;

    >Comment

typedef struct

{

stackrecord stackarray [LIMIT];

int top;

}stack;

    >Comment

stack s;

The definitions must be preceded by a typedef for "whatever." Let us assume LIMIT has been set in a define statement fixing the maximum number of items the stack can hold. Suppose 6, 45, 15, 32, and 18 were pushed onto the stack. Figure 4.8 illustrates the meaning attached to the declaration. Figure 4.8(a) is a conceptualization of the stack. Since 6 was the first entry on the stack, it is on the bottom. Likewise, 18 was the last entry, so it is on top. Figure 4.8(b) is the way the data is stored in the actual stack s. The stackarray holds the data and top contains an index to the current entry on the top of the stack. Although 6 is at the top of the array, it is the bottom stack entry; while 18 is at the bottom of the array, it is the top stack entry. Thus top has the value 4.

Addition or insertion of a new element to the stack will be accomplished by a function push. Removal or deletion of the current top stack element will be accomplished by a function pop. Executing push (&newrecord,&s) results in the contents of newrecord being placed on the stack as a new first element. A call to the function pop(&s,&value) causes the removal of the current top stack element, with value containing a copy of that value. A routine setstack sets the stack to empty so that it initially contains no entries.

Example 4.4

Suppose we execute

pop(&s,&y);

pop(&s,&z);

    >Comment

push(&newrecord,&s);

If Figure 4.8 indicates the current stack, then Figure 4.9 shows the new situation after this execution.

(a) The Stack Data Abstraction

(b) Array Implementation of the Stack

Figure 4.8 Stack before Insertion of a New Element

(a) The Stack Data Abstraction

(b) Array Implementation of the Stack

Figure 4.9 Stack after Insertion of a New Element

Stack underflow occurs when a program attempts to remove an element from an empty stack. It normally indicates either an error or the termination of a task. Stack overflow occurs when a program attempts to add an element to a full stack. Although, conceptually, the stack as a data structure has no limit, an actual implemented stack does have a limit. In the present implementation it is termed. LIMIT. Push and pop should test, respectively, for stack overflow and underflow and take appropriate action whenever these conditions occur. Such precautionary measures are an example of defensive programming.

It is also necessary to test the stack to determine whether or not it is empty. This may be accomplished by invoking a function empty, which will return the value true when the stack is empty and false otherwise.

These routines may be implemented as follows:

setstack (ps)

/* Initializes the stack s to empty */

stack *ps;

{

    >Comment

(*ps).top = -1;

}

empty (ps)

/* Returns true only if the stack is empty */

stack *ps;

{

    >Comment

return ((*ps).top == -1);

}

push (pnewrecord,ps)

/* Inserts the record newrecord

at the top of the stack s

*/

stackrecord *pnewrecord,

stack *ps;

{

    >Comment

if ((*ps).top == (LIMIT - 1))

overflow(ps);

else

{

    >Comment

(*ps).top = (*ps).top + 1;

(*ps).stackarray[(*ps).top].info =

(*pnewrecord).info;

}

}

pop(ps,pvalue)

/* Copies the contents of the top record

of stack s into value and removes the

record from the stack

*/

stack *ps;

stackrecord *pvalue;

{

    >Comment

if(empty(ps))

underflow(ps);

else

{

    >Comment

(*pvalue).info = (*ps).stackarray

[(*ps).top].info;

(*ps).top = (*ps).top - 1;

}

}

The procedures overflow and underflow are assumed to take appropriate action when they are invoked. Notice that the elements of the stack will be of "whatever" type. We have taken the liberty here of assuming that the C compiler being used allows the contents of one structure to be assigned to another. Otherwise, for example, the assignment

(*ps).stackarray[(*ps).top].info =

(*pnewrecord).info;

that occurs in push could not be used. Instead, each field of info would have to be individually assigned.

The implementation discussed so far applies when records are to be stored on the stack. Instead, if only the contents of simple variables of type such as int, float, char, or listpointer are to be stored, then the declarations and definitions of push and pop may be simplified, as follows.

    >Comment

typedef struct

{

whatever stackarray [LIMIT];

int top;

}stack;

push(value,ps)

/* Inserts the new value

at the top of the stack s

*/

whatever value;

stack *ps;

{

if((*ps).top == (LIMIT-1))

overflow(ps);

else

{

(*ps).top = (*ps).top + 1;

    >Comment

(*ps).stackarray[(*ps).top] = value;

}

}

pop(ps,pvalue)

/* Copies the contents of the top entry

of stack s into value and removes it

from the stack

*/

stack *ps;

whatever *pvalue;

{

if(empty(ps))

underflow(ps);

else

{

    >Comment

*pvalue = (*ps).stackarray[(*ps).top];

(*ps).top = (*ps).top - 1;

}

}

"Whatever" will, of course, be declared of type int, or float, etc. The same simplification can be made, when appropriate, to the implementations of Sections 4.5.2, 4.5.3, 4.7.1, 4.7.2, and 4.7.3.

The idea of functional modularity is to isolate or localize the code of a program that is invoked in a particular task, such as an operation on a data structure. The implementation of the stack data abstraction discussed so far has achieved this modularity, with respect to the push, pop, setstack, and empty operations on the stack, by embodying them in individual routines. Another kind of modularity may be achieved by placing the type definitions and the declarations of these operations together in one file. This will be discussed further in the next chapter.

An application of the stack to the nonrecursive towers of Section 4.4.1 follows. It illustrates the use of the stack and its treatment as a data abstraction.

#include <stdio.h>

    >Comment

#define LIMIT 50

typedef struct

{

int n;

int i;

int a;

int f;

}whatever;

typedef struct

{

whatever info;

}stackrecord;

typedef struct

{

stackrecord stackarray [LIMIT];

int top;

}stack;

setstack(ps)

/* Initializes the stack s to empty */

stack *ps;

{

(*ps).top = -1;

}

empty(ps)

/* Returns true only if the stack s is empty */

stack *ps;

{

return((*ps).top == -1);

}

push(pnewrecord,ps)

/* Inserts the record newrecord

at the top of the stack s

*/

stackrecord *pnewrecord;

stack *ps;

{

if ((*ps).top == (LIMIT-1))

overflow(ps);

else

{

(*ps).top = (*ps).top + 1;

(*ps).stackarray[(*ps).top].info =

(*pnewrecord).info;

}

}

pop(ps,pvalue)

/* Copies the contents of the top record

of stack s into value and removes the

record from the stack

*/

stack *ps;

stackrecord *pvalue;

{

if (empty(ps))

underflow(ps);

else

{

(*pvalue).info =

(*ps).stackarray[(*ps).top].info;

(*ps).top = (*ps).top - 1;

}

}

overflow(ps);

/* Prints a message that the stack has overflowed */

stack *ps;

{

printf(" The stack has overflowed \n");

}

underflow(ps);

/* Prints a message that the stack has underflowed */

stack *ps;

{

printf(" The stack has underflowed \n");

}

main()

/* Driver for towers */

{

int n;

printf("\n enter a value for n\n");

scanf("%d",&n);

towers(n,1,2,3);

}

    >Comment

towers(n,i,a,f)

/* Moves the top n disks

from peg i to peg f

*/

int n,i,a,f;

{

    >Comment

stack s;

int done;

setstack(&s);

done = FALSE;

while(!done)

{

while (n > 1)

{

s_tack(n,i,a,f,&s);

setvar1(&n,&i,&a,&f);

}

printf("\n %d -> %d\n",i,f);

if(!empty(&s))

{

restore(&n,&i,&a,&f,&s);

printf("\n %d -> %d\n",i,f);

setvar2(&n,&i,&a,&f);

}

else

done = TRUE;

}

setvar1(pn,pi,pa,pf)

/* Sets n to n-1 and

interchanges f and a

*/

int *pn,*pi,*pa,*pf;

{

int t;

*pn = *pn-1;

t = *pf;

*pf = *pa;

*pa = t;

}

setvar2(pn,pi,pa,pf)

/* Sets n to n-1 and interchanges a and i

*/

int *pn,*pi,*pa,*pf;

{

int t;

*pn = *pn-1;

t = *pa;

*pa = *pi;

*pi = t;

}

    >Comment

s_tack(n,i,a,f,ps)

/* Creates a record containing

n,i,a, and f and inserts it

on stack s

*/

int n,i,a,f;

stack *ps;

{

    >Comment

stackrecord newrecord;

newrecord.info.n = n;

newrecord.info.i = i;

newrecord.info.a = a;

newrecord.info.f= f;

push(&newrecord,ps);

}

    >Comment

restore(pn,pi,pa,pf,ps)

/* Removes the top record from

stack s and copies its contents

into n,i,a,f

*/

int *pn,*pi,*pa,*pf;

stack *ps;

{

stackrecord value;

pop(ps,&value);

*pn = value.info.n;

*pi = value.info.i;

*pa = value.info.a;

*pf = value.info.f;

}

In review of the program, notice that it treates the stack as a data abstraction. The type definitions and function definitions for the stack are placed at the top of the program. The functions setvar, s_tack, and restore are not basic stack operations; they are used by towers. Thus they have been placed after towers. Similarly, towers has been placed after main. Also, s_tack and restore are dependent on the stack implementation, since they must know the structure of a stack record, which is, in turn, determined by the needs of towers for its scratchpad.

4.5.2 List Implementation of the Stack with an Array of Records

In addition to the array implementation, a stack can be implemented as a list. Since we have discussed two ways to implement lists, we know two ways to implement stacks as lists. One is to store the stack records in an array; the other is to store them in dynamic memory. We have already given an implementation of a stack using an array of records in Chapter 3--where availist (Section 3.7) actually acts as a stack. When avail is invoked, it removes the top record from the availist, in addition to returning a pointer to the record. Thus it has effectively popped the record. Likewise, reclaim may return a record to the front of the availist. Doing so, reclaim is essentially pushing a new record onto the availist. Availist serves as the head of a list, which itself serves as a list implementation of a stack. Of course, appropriate overflow and underflow checking must be built into avail and reclaim, respectively.

4.5.3 List Implementation of the Stack with Records in Dynamic Memory

The third stack implementation is also as a list--one that involves dynamic storage allocation of the records. That is, the records are stored in the storage area set aside by C to store records referenced by pointer variables. First, the following declaration must be made.

    >Comment

typedef struct record

{

struct whatever info;

struct record *link;

}stackrecord,*stack;

If we then define s as type stack, code can be written that is independent of its implementation. Setstack must initialize the stack to empty. This means setting s, the head of the list implementing the stack, to null. Push must now use malloc to allocate storage for the entry to be pushed onto the stack before setting its info field and inserting that record as the new first record on the stack. Pop must copy the info field value of the top record into value and then delete that record from the stack.

These stack operations are implemented as follows.

setstack(ps)

/* Initializes the stack s to empty */

stack *ps;

{

    >Comment

*ps = NULL;

}

empty(ps)

/* Returns true only if the stack s is empty */

stack *ps;

{

    >Comment

return(*ps == NULL);

}

push(pnewrecord,ps)

/* Inserts the record newrecord

at the top of the stack s

*/

stackrecord *pnewrecord;

stack *ps;

{

stackrecord *newentry;

    >Comment

newentry = malloc(sizeof(stackrecord));

newentry->info = pnewrecord->info;

newentry->link = *ps;

*ps = newentry;

}

pop(ps,pvalue)

/* Copies the contents of the top record

of stack s into value and removes the

record from the stack

*/

stack *ps;

stackrecord *pvalue;

{

    >Comment

if(!empty(ps))

{

    >Comment

pvalue->info = (*ps)->info;

*ps = (*ps)->link;

}

else

underflow(ps);

}

Notice that this implementation removes the artificial restriction imposed by LIMIT in the first implementation using an array. This is one of the advantages of using dynamic allocation by means of pointer variables.

It is very important to see that no matter what the implementation, as long as the stack is treated as a data abstraction when writing programs, the programs need not be changed, even if the programmer decides to change the implementation of the stack. Only the declarations and definitions related to the stack need be changed, as is evident in the following nonrecursive towers that uses the list implementation. It should be compared to the program of Section 4.5.1, in which the stack was implemented using an array.

    >Comment

#include <stdio.h>

#define NULL 0

typedef struct

{

int n;

int i;

int a;

int f;

}whatever;

typedef struct record

{

whatever info;

struct record *link;

}stackrecord, *stack;

setstack(ps)

/* Initializes the stack s to empty */

stack *ps;

{

*ps = NULL;

}

empty(ps)

stack *ps;

{

return(*ps == NULL);

}

push(pnewrecord,ps)

stackrecord *pnewrecord;

stack *ps;

{

stackrecord *newentry;

newentry = malloc(sizeof(stackrecord));

newentry->info = pnewrecord->info;

newentry->link = *ps;

*ps = newentry;

}

pop(ps,pvalue)

/* Copies the contents of the top record

of stack s into value and removes the

record from the stack

*/

stack *ps;

stackrecord *pvalue;

{

if(!empty(ps))

{

pvalue->info = (*ps)->info;

*ps = (*ps)->link;

}

else

underflow(ps);

}

overflow(ps);

/* Prints a message that the stack has overflowed */

stack *ps;

{

printf(" The stack has overflowed \n");

}

underflow(ps);

/* Prints a message that the stack has underflowed */

stack *ps;

{

printf(" The stack has underflowed \n");

}

    >Comment

main()

/* Driver for towers */

{

int n;

printf("\n enter a value for n \n");

scanf("%d",&n);

towers(n,1,2,3);

}

towers(n,i,a,f)

1 final nonrecursive version

/* Moves the top n disks

from peg i to peg f

*/

int n,i,a,f;

{

stack s;

1 allocates storage for s (an instance of type stack)

int done

setstack(&s);

done = FALSE;

while(!done)

{

while (n > 1)

{

s_tack(n,i,a,f,&s);

setvar 1(&n,&i,&a,&f);

}

printf("\n %d -> %d\n",i,f);

if(!empty(&s))

{

restore(&n,&i,&a,&f,&s);

printf("\n %d -> %d\n",i,f);

setvar2(&n,&i,&a,&f);

}

else

done = TRUE;

}

setvar1(pn,pi,pa,pf)

/* Sets n to n - 1 and

interchanges f and a

*/

int *pn,*pi,*pa,*pf;

{

int t;

*pn = *pn - 1;

t = *pf;

*pf = *pa;

*pa = t;

}

setvar2(pn,pi,pa,pf)

/* Sets n to n - 1 and

interchanges a and i

*/

int *pn,*pi,*pa,*pf;

{

int t;

*pn = *pn - 1;

t = *pa;

*pa = *pi;

*pi = t;

}

    >Comment

s_tack(n,i,a,f,ps)

/* Creates a record containing

n, i, a, and f and inserts it

on stack s

*/

int n,i,a,f;

stack *ps;

{

    >Comment

stackrecord newrecord;

newrecord.info.n = n;

newrecord.info.i = i;

newrecord.info.a = a;

newrecord.info.f = f;

push(&newrecord,ps);

}

    >Comment

restore(pn,pi,pa,pf,ps)

/* Removes the top record from

stack s and copies its contents

into n,i,a,f

*/

int *pn,*pi,*pa,*pf;

stack *ps;

{

stackrecord value;

pop(ps,&value);

*pn = value.info.n;

*pi = value.info.i;

*pa = value.info.a

*pf = value.info.f;

}

The problem of implementing many stacks simultaneously is introduced in the suggested assignment at the end of this chapter. More sophisticated algorithms for the management of the stacks are explored in Garwick [1964], Knuth [1973], and Korsh [1983].

4.6: Evaluation and Translation of Expressions

To illustrate the use of stacks, let us consider the problem of evaluating arithmetic expressions. People are accustomed to writing arithmetic expressions in infix notation, where an operator appears between its operands. Consequently, programming languages generally use infix notation to be "user friendly." Thus the programmer can write

(((a + (b/c * d)) + (a/w)) * b)/(((a * c) + d) + a)

Table 4.1 Notations for Two

Arithmetic Expressions

          a + (b * c)   (a + b) * c

Prefix    + a * bc      * + abc

Infix     a + b * c     a + b * c

Postfix   abc * +       ab + c *

Even humans must scan such an expression carefully, more than once, before discovering that its basis syntactic structure is (operand 1)/(operand 2). A serious problem for the compiler writers was how to translate such expressions without making many scans back and forth through them. The problem was compounded further because programmers were not required to write fully parenthesized expressions; this created the additional dilemma of how to resolve ambiguity. As will be shown below, ambiguity may be resolved by using a notation other than infix.

Consider the arithmetic expressions of Table 4.1. These arithmetic expressions are written in prefix, infix, and postfix notation. Prefix means that the operator appears to the left, before its operands, and postfix means that the operator appears to the right, after its operands. Although infix notation is what programmers normally use, there are contexts where prefix (or postfix) notation is used. For instance we write sqrt (x), f(x,y), and minimum (x,y)

Note that the prefix and postfix expressions of the first column are equivalent ways to express the infix expression a + (b * c), while the prefix and postfix expressions of the second column are equivalent to (a + b) * c. Unfortunately, the infix expressions of both columns are identical. This means that, unless parentheses are used, there is no way to distinguish between a + (b * c) and (a + b) * c using infix notation. Yet we can choose the proper prefix or postfix notation to make clear which of the two is intended, even though no parentheses (or special conventions) are used. This holds true in general for arithmetic expressions. For this reason, prefix and postfix notations are called parentheses-free notations. Compilers can easily process prefix and postfix expressions. Since postfix expressions are generally used, they are the main focus of the discussion that follows.

4.6.1 Postfix Expressions

Given an infix expression that is completely parenthesized, such as

((a + b) * (c/d))

you know how to evaluate it. This is because you have grown accustomed to the rules for its evaluation, although you might be hard pressed to state those rules completely and precisely. In any case, parentheses signal how the operators are to be applied to the operands, indicating that (a + b) and (c/d) must be evaluated first before applying the * operator. Removing all parentheses yields

a + b * c/d

Unless conventions or rules are given, this expression is ambiguous. Its five distinct interpretations correspond to the ways in which parentheses may be placed:

((a + b) * (c/d))

(((a + b) * c)/d)

((a + (b * c))/d)

(a + ((b * c)/d))

(a + (b * (c/d)))

A postfix expression such as

ab + cd/*

makes little sense, for you are not used to it, having little experience in interpreting or evaluating it. The rule to be used is as follows:

1. Scan the postfix expression from left to right.

2. Whenever an operator is encountered,

replace it, and the immediately preceding number of operands required by that  operator, with the result of applying that operator to those operands.

3. Treat this replacement as an operand, and continue scanning from this point.

Thus the postfix expression is equivalent to the infix expression

((a + b) * (c/d))

The rule, in generating this expression, produces the two intermediate expressions: (a + b)cd/*, (a + b)(c/d)*.

The following list shows the five fully parenthesized infix expressions representing the interpretations of a + b * c/d and their corresponding postfix expressions.

((a + b) * (c/d)) .... ab + cd/*

(((a + b) * c)/d) .... ab + c * d/

((a + (b * c))/d) .... abc* + d/

(a + ((b * c)/d)) .... abc*d/+

(a + (b * (c/d))) .... abcd/*+

These examples emphasize two points:

1.     The rule for evaluating postfix expressions allows no ambiguity. It determines precisely the meaning or interpretation of the postfix expression.

2.     An infix expression, unless fully parenthesized, may be ambiguous in its meaning or interpretation.

It is because of statement 1 that postfix notation, or postfix expressions, are called parentheses-free. A similar rule allows prefix notation, or prefix expression, to be parentheses-free .

Furthermore, evaluation of a postfix expression is easily accomplished by scanning the expression from left to right and using a stack to retain operands and

Table 4.2 Evaluation of a Postfix Expression

Input      Operand Stack

  a        a

  b        b

           a

  c        c

           b

           a

  *        (b * c)

           a

  d        d

           (b * c)

           a

  /        ((b * c)/d)

           a

  +        (a + ((b * c)/d)

results of intermediate evaluations. Any operand that is encountered is placed on an operand stack. When an operator is encountered, its operands are taken from the top of the stack, the operator is applied, and the result is put on the top of the stack. The process is illustrated in the following example.

Example 4.5

Table 4.2 shows the contents of the operand stack after each input is processed, simulating the procedure for evaluation of a postfix expression. The expression is ABC*D/+.

Since the procedure for the evaluation of an arithmetic expression using a stack is relatively straightforward and requires no parentheses, why don't programmers write arithmetic expressions in postfix notation only? It is because they are so used to infix notation that they don't want to change. As a result, high-level languages accept infix expressions rather than postfix. This leads back to the question of how infix expressions may be evaluated.

4.6.2 Infix Expressions

The rule to be used for a completely parenthesized infix expression is

1. Scan the infix expression from left to right.

2. When a "matched" pair of ''left" and "right" parentheses is encountered,

apply the operator to the operands enclosed within the pair of parentheses, and replace the paired parentheses and the contained operator and operands by the result. Continue scanning from this point.

One problem remains: How do we evaluate infix expressions that are not completely parenthesized? Recall that there are standard rules for interpreting infix expressions in this case. These rules are also incorporated in high-level languages. They are necessary because parentheses are not always available to signal what is to be done next. For example, if we evaluate

a + b * c/d

then when the * is encountered, the stacks appear as

Operand Stack     Operator Stack

--------------------------------

      b                 +

      a

The operator * signals that, had the expressions been completely parenthesized, a left parenthesis would have occurred just before the b, or else a right parenthesis would have occurred just after the b. In other words, the appearance of the operator requires a decision as to whether b should be considered an operand of *or an operand of +. The standard rules resolve such ambiguities by assigning priorities or precedence values to operators, assuming a left-to-right scan of the expression. For the binary operators, +, -, *, /, and ** (for exponentiation), the priorities are as follows:

1.     Exponentiation (**) has the highest priority.

2.     Multiplication and division (*, /) have the same priority, which is higher than that of addition or subtraction (+, -).

3.     Addition and subtraction (+, -) have the same priority.

A left parenthesis also needs a priority, the lowest.

Whether b is an operand of * or + can then be resolved by taking b as an operand of the highest priority operator (* in this case). Again, parentheses may be used to override priorities. For example, in

(a + b) * c/d

b is to be associated with the + operator.

Example 4.6

In order to see how any infix expression is evaluated using priorities, and to develop an algorithm for this evaluation, consider the expression,

a + (b * c/d ** e) *f

To evaluate it, when the first * is encountered the situation will be

Operand Stack      Operator Stack

--------------------------------

     b                   (

     a                   +

In this case, since the left parenthesis is on top of the operator stack, the * should be placed on top of the operator stack, and scanning continued. After c, division(/) is encountered and the stacks become

Operand Stack      Operator Stack

---------------------------------

      c                  *

      b                  (

      a                  +

At this point a decision must be made with respect to c: either it is to be associated with * (at the top of the stack) or with / (the current operator being processed). The standard rules associate c with the operator of highest priority. In this case, since * and / have equal priorities, associate c with the leftmost of the two operators in the input--that is, the top stack operator. Thus, the * must be popped, c and b must be popped, the * must be applied to b and c, and the result, (b * c), placed on top of the operand stack. Then the / must be placed on the operator stack. Had the current operator been of higher priority (say, **), then the current operator should simply be placed on the operator stack.

In the example, after / and d are processed, the stacks are as follows:

Operand Stack      Operator Stack

---------------------------------

      d                  /

   (b * c)               (

      a                  +

The ** becomes the current operator, and d must be associated with it or with /. Since ** has higher priority, place it on the operator stack; the result after processing e is

Operand Stack      Operator Stack

---------------------------------

      e                  **

      d                  /

   (b * c)               (

      a                  +

The current input character being scanned is now a right parenthesis. This "matches" the topmost left parenthesis on the stack. The next steps are to pop the operator stack; pop the operand stack to obtain the top two operands; apply **to them, and place the result back on the top of the operand stack. Then pop the operator stack; pop the operand stack to obtain the top two operands; apply / to them and place the result back on top of the operand stack; pop the stack again; and notice that a left parenthesis has appeared, signaling completion of the process for the current input character. The result is

 Operand Stack       Operator Stack

----------------------------------

((b * c)/(d ** e))         +

        a

Finally, the last * is processed. It has higher priority than +, so it is placed on the operator stack. f is then processed and placed on the operand stack, yielding

 Operand Stack        Operator Stack

-----------------------------------

       f                    *

((b * c)/(d ** e))          +

       a

Since the input is exhausted, the final steps are, for each remaining operator on the operator stack, to pop the operator stack, remove the required number of top operands from the operand stack, apply the operator, and place the result back on top of the operand stack. The final result appears at the top of the operand stack.

The algorithm for the evaluation of any infix expression can now be written. The algorithm, which follows, assumes that a character, when read, will be an entire operand or operator. This means, for example, that ** should be interpreted as a "character." The algorithm returns the value of the expression in evaluate.

1. Initialize the operand stack to empty.

2. Initialize the operator stack to empty.

3. While there is another input character,

read the current character

if the current character is "(", then

push it on the operator stack;

else if the current character is an operand, then

push it on the operand stack;

else if the current character is an operator, then

if the operator stack is not empty, then

if the current character does not have greater priority than the top

operator stack character, then

pop the operator stack, remove the required number of operands

from the operand stack, apply the operator to them, and push the

result on the operand stack

push the current character onto the operator stack

else if the current character is ")", then

pop the operator stack and set topchar to the popped value

while topchar is not (

remove the number of operands required by topchar from the operand

stack, apply topchar to them, and

push the result onto the operand stack

pop the operator stack and set topchar to the popped value

4.While the operator stack is not empty,

pop the operator stack, remove the required number of operands from the

operand stack, apply the operator to them, and push the result on the operand

stack

5.Pop the operand stack and set evaluate to the popped value

Notice that the evaluation of an infix expression requires two stacks rather than the one stack required for the evaluation of a postfix expression. Other operators may be introduced and given appropriate priorities. For example, the assignment operator, =, would be given a higher priority than **.

4.6.3 Translating Infix to Postfix

Suppose, rather than evaluating an infix expression, a programmer wished to translate it and produce the corresponding postfix expression as output. The reason is that postfix expressions can be executed directly, whereas infix expressions are tricky to evaluate. Moreover, if the translation can be done, then the postfix expression may easily be embellished to produce a machine language program for the evaluation of the original infix expression.

The following modification of the infix evaluation algorithm results in such a translation.

1. Eliminate the operand stack. Instead, any pushing onto the operand stack should be changed to outputting the same value.

2. Instead of applying operators to operands, simply output the operators.

4.7: Queues

The queue, like the stack, is a useful data abstraction for retaining information in a specific order. A queue is a data structure with restricted insertion and deletion operations. Information may be inserted on the queue, but only at the rear end, or bottom. Information may be removed from the queue, but only at the front end, or top.

The queue acts like the pile of Community Chest cards in Monopoly. The top card is the only one that can be removed, and insertions to the pile can be made only at the bottom. If we think of the information stored as representing postponed obligations, then the queue gives access to the obligations in the order in which they were incurred. A popular mnemonic for helping to remember this characteristic is FIFO--first in, first out. Like LIFO, this is a common term in accounting and inventory control systems.

4.7.1 Array and Circular Array Implementation of the Queue

Queues can be implemented in the same basic ways as stacks--as an array of records, as a list with its records stored in an array, and as a list with its records stored in dynamic memory. As in the case of stacks, these implementations are discussed in that order.

A queue may be implemented using an array of records by first declaring

    >Comment

typedef struct

{

whatever info;

}queuerecord;

    >Comment

typedef struct

{

queuerecord queuearray [LIMIT];

int front, rear;

}queue;

    >Comment

queue q;

As with stacks, overflow and underflow should be checked in a general implementation. A typical queue situation is depicted in Figure 4.10(a) for the array implementation. Figure 4.10(b) shows the situation after 18, 32, and 15 have been removed and 17 and 20 have been inserted.

Storage for insertion of more records is available at the top of the queuearray, but not at the bottom. To ensure that all the relocated storage for queue is used before overflow occurs, programs "wrap around." That is, the queue is implemented so that the queuearray is viewed as a circle of records, with 0 following the LIMIT, 7. Front is used to point to the entry at the front of the queue, and rear is used to point to the array location in which the next

(a) Initial Configuration

(b) After Three Records Are Removed and Two Records Are Inserted

Figure 4.10 Typical Queue Configuration for an Array Implementation

inserted record is to be placed. As entries are inserted, rear moves down until it "falls off" the array, then it must be "wrapped around." As entries are deleted, front moves down until it "falls off" the array, then it must "wrap around."

The insertion operation may then be implemented as follows:

insert(pnewrecord,pq)

/* Inserts newrecord at the rear of queue q */

queuerecord *pnewrecord;

queue *pq;

{

    >Comment

if ((*pq).front == -1)

{

    >Comment

(*pq).front = 0;

(*pq).queuearray[0].info = (*pnewrecord).info;

(*pq).rear = 1;

}

else

    >Comment

if((*pq).rear ==  (*pq).front)

printf("\n overflow \n");

else

{

    >Comment

if((*pq).rear <= (LIMIT - 1))

{

    >Comment

(*pq).queuearray[(*pq).rear].info =

(*pnewrecord).info;

(*pq).rear = (*pq).rear + 1;

}

else

    >Comment

{

if((*pq).front == 0)

printf("\n overflow \n");

else

{

4 there is room to wrap around, so the new entry is inserted at the start of the array and rear set to the next location

(*pq).queuearray[0].info =

(*pnewrecord).info;

(*pq).rear = 1;

}

}

}

}

Setqueue must set front and rear to minus one (-1). Empty will return the value true when front = -1, and false otherwise. The removal of a record can be implemented by a function remove, which returns in value the record that was deleted from the queue.

remove (pq, pvalue)

/* Deletes the front entry from

queue q and copies its contents

into value

*/

queue *pq;

queuerecord *pvalue;

{

    >Comment

if(empty(pq))

printf("\n underflow \n");

else

{

    >Comment

(*pvalue).info = (*pq).queuearray

[(*pq).front].info;

    >Comment

(*pq).front = (*pq).front + 1;

if((*pq).front == (*pq).rear)

2 if queue is empty, then reinitialize it

setqueue(pq);

    >Comment

if((*pq).front > (LIMIT - 1))

(*pq).front = 0;

}

}

4.7.2 List Implementation of the Queue with an Array of Records

Now we can turn to implementing a queue as a list with records stored in an array. The availist of Section 3.7 actually acts as a list implementation of a queue if reclaim returns a record to the rear of the availist. Thus queue can be implemented as a list of records stored in an array.

4.7.3 List Implementation of the Queue with Records in Dynamic Memory

The third queue implementation is also a list--one involving dynamic storage allocation of the records in dynamic memory. First, the following declaration must be made:

typedef struct record

{

whatever info;

struct record *link;

}queuerecord,*queue;

Setqueue must set queue to null. Insert will first request storage for the new record to be inserted by invoking malloc. It will then set its info data and insert it as the new last record in the queue. Remove must delete the first record of the queue and return its info data.

Your deck has now been "stacked," you've been "queued" into queues, and you mustn't postpone your obligation to use them!

4.8: Case Study: Checking Sequences for Proper Nesting

To bring together the major ideas of this chapter, a case study to check program structure for proper nesting is presented. Its main features are:

1.     The top-down approach

2.     Demonstration of the use of the stack, treating it as a data abstraction

3.     Checking program structure for proper nesting by developing a nonrecursive and a recursive function

4.     The different perspectives required to develop the two functions

Every computer programmer has at one time or another gotten error messages saying "incomplete loop structure" or "a missing parenthesis." You now have enough background to write a program that checks for such errors. Let's see how it can be done.

Arithmetic expressions, as well as a program text, consist of sequences of symbols. Complex arithmetic expressions may be composed by properly combining simpler arithmetic expressions, delineated by matching left and right parentheses. The simpler expressions are said to be nested within the more complex expressions that contain them. The complex expression (a * (b + [c /d])) + b) contains two pairs of nested expressions. C programs contain compound statements delimited by matching { (begin) and } (end) pairs. Ignoring all other symbols, these matching pairs must satisfy rules of nesting to be syntactically correct.

Nesting means that one matching pair is wholly contained within another. When pairs are nested, they may not overlap, as do the two pairs in ( [ ) ]. The rules of nesting are the same for any sequence that is built by combining simpler sequences. They require that the start and end of each component be specified. Checking complex sequences to determine that these begin-end pairs do not overlap is an important operation. It provides an interesting application of the stack data abstraction, as well as of recursion. An intimate connection exists among such sequences, stacks, recursions, and trees; it will be explored more fully in later chapters.

Two abstractions of this problem will be considered. The second is a generalization of the first. To begin, assume only one kind of left parenthesis and one kind of matching right parenthesis, and consider sequences composed of these. Thus, at first, expressions such as ([]) are not allowed, since they involve more than one kind of parenthesis.

Given a sequence of left and right parentheses, determine if they may be paired so that they satisfy two conditions:

i. Each pair consists of a left and right parenthesis, with the left parenthesis preceding the right parenthesis.

ii. Each parenthesis occurs in exactly one pair.

If a sequence has a pairing satisfying both of these conditions, the sequence and pairing are valid. The problem is to determine whether a given sequence is valid or not. No sequence starting with a right parenthesis, ")", can be valid, since there is no preceding left parenthesis, "(", to pair with it. Any valid sequence must therefore begin with some series of n "(" parentheses preceding the first appearance of a ")" parenthesis, such as Figure 4.11(a), for which n is 3. Suppose we remove this adjacent pair, the nth "(" parenthesis, and the immediately following first ")" parenthesis. This would be pair 1 in Figure 4.11(a). If the new sequence thus obtained is valid, then adding the removed pair to it results in a valid pairing. Thus the original sequence is valid.

Suppose you find that the resultant sequence is not valid. Can you conclude that the original sequence is not valid? In other words, has constraining one of the pairs to be the first adjacent "(" and ")" parentheses made a pairing of the original sequence satisfying (i) and (ii) impossible? The answer is no. To see this, assume that there is some valid pairing of the original sequence, in which the first ")" parenthesis is not paired with the immediately preceding "(" parenthesis, but there is no valid pairing in which it is. In general, the first ")" parenthesis must then be paired with an earlier "(" parenthesis, and the immediately preceding "("parenthesis must have been paired with a later ")" parenthesis, as in Figure 4.11(b). These two pairs can then be rearranged by interchanging their "(" parentheses. The resultant pairing satisfies conditions (i) and (ii), yet it has the first adjacent "(" parenthesis paired with the immediately following ")" parenthesis, but this is the nesting constraint that has been specified. This contradicts the assumption that no valid pairing satisfies the constraint.

Consequently, the conclusion is that a solution to the problem is obtained as follows:

1. Pair the first adjacent left, "(", and right, ")", parentheses and remove the pair.

2. If the resultant sequence is valid, then so is the original sequence; otherwise the original sequence is not valid.

Think of a left parenthesis as denoting an incurred obligation, and a right parenthesis as the fulfilling of the most recently incurred obligation. This is the precise situation for which a stack is useful. As the program scans a sequence

(a) Four Pairs Satisfying Conditions i and ii

(b) Four Other Pairs Satisfying Conditions i and ii

(c) Four Pairs Not Satisfying Conditions i and ii; No Other Four Pairs Will

Figure 4.11 Valid and Invalid Nesting of Parentheses

from left to right, it pushes each "(" parenthesis onto a stack, representing an incurred obligation. Task 1 of the solution may then be carried out by popping the stack when the first ")" parenthesis is encountered, to fulfill the most recently incurred obligation.

Carrying out task 2 requires first determining if the resultant sequence is valid. Notice, however, that had the resultant sequence been processed in this same way, the contents of the stack would be exactly what they are now. Consequently, the program can continue to scan the original sequence (which is exactly what would be seen if it were scanning the resultant sequence), pushing whenever it encounters a "(" parenthesis, and popping whenever it comes to a ")" parenthesis. If the stack is empty when the entire scan is completed, then we have determined that the original sequence is valid; otherwise it is not. During the scan, any attempt to pop an empty stack means that no "(" parenthesis is available to be paired with the ")" parenthesis that caused the attempt (that is, the sequence is invalid).

This solution may be implemented as in the function determine, which follows this paragraph. Note that the stack is treated as a data abstraction . Readchar is a function that reads into its parameter the next character of the input sequence and returns true, unless the character read was a specialchar used as a sentinel, and then it returns false. When determine returns, the entire input sequence has been read. Valid will return true if a pairing satisfying (i) and (ii) exists, and false otherwise.

Nonrecursive Version

determine(pvalid)

/* Returns true in valid only if the

input sequence is properly nested

*/

int *pvalid;

{

char currentchar;

struct

{

char topchar;

}topcharacter;

    >Comment

stack s;

*pvalid = TRUE;

    >Comment

setstack(&s);

    >Comment

while(readchar(&currentchar))

    >Comment

if(leftparen(currentchar))

push(&currentchar,&s);

    >Comment

else if(rightparen(currentchar))

if(!empty(&s))

{

pop(&s,&topcharacter);

    >Comment

if(!match(topcharacter.topchar,

currentchar))

*pvalid = FALSE;

}

else

*pvalid = FALSE;

else

    >Comment

process(currentchar,&s);

    >Comment

if(!empty(&s))

*pvalid = FALSE;

}

Leftparen returns the value true if its parameter is a left parenthesis, and false otherwise. Rightparen is defined similarly. Match returns the value true if topchar and currentchar are left and right parentheses, respectively.

One of the aims of the text is to show how general functions can be adapted to solve particular problems. Determine is such a general function. It can be adapted to evaluate a completely parenthesized infix expression. We repeat the rule of Section 4.6.2 to be used to evaluate such an expression:

1. Scan the infix expression from left to right.

2. When a "matched" pair of "left" and "right" parentheses is encountered,

apply the operator to the operands enclosed within the pair of parentheses, and replace the paired parentheses and the contained operator and operands by the result. Continue scanning from this point.

The matched pairs referred to in this rule are exactly the same as in this case study. Studying the rule, you can see that a right parenthesis signals that an operator and its operands are enclosed between the right parenthesis and its matching left parenthesis.

Determine may be modified to obtain a function evaluate, by defining the process routine as follows:

1. When an operand is encountered, place the operand on top of an operand stack.

2. When an operator is encountered, place the operator on top of the operator stack.

It is also necessary, when a "right" parenthesis is encountered, and the operator stack is not empty, to pop the operator stack, remove the required number of operands from the top of the operand stack, apply the operator to those operands, and place the result on the top of the operand stack. This must be done just before the pop(&s,&topcharacter) statement. Determine's stack, s, plays the role of the operator stack. An operand stack must be introduced and must be available to both the evaluate and process functions.

The general function determine is actually more complex than need be for "paired parentheses." In fact, a stack is not even necessary. All that is really needed is to keep a tally of "(" parentheses, decreasing the tally by 1 every time a ")" parenthesis is read. If the tally ever becomes negative, then determine should, from that point, simply read in the rest of the input sequence and return false. Otherwise, if the tally is zero after the entire input sequence has been read, it should return true. Determine is implemented as shown in the nonrecursive function listed above because it will serve as the solution to the next problem, for which "paired parentheses" will be a special case.

Suppose, instead of containing only "(" and ")" parentheses, the input sequence could consist of any characters. Assume that each character could be either a left or a right character but not both, and also that each left character has a unique right character with which it matches. For instance, a, (, [, and 1 might be "left" characters and the unique "right" characters with which they match z, ), ], and r, respectively.

Given a sequence of left and right characters, determine if they may be paired so that

i. Each pair consists of a left and a right matching character, with the left character preceding the right character.

ii. Each character occurs in exactly one pair.

iii. The sequence of characters between the left and right characters of each pair must also satisfy (i) and (ii).

If a sequence of left and right characters satisfies conditions (i), (ii), and (iii), the sequence and the pairing are valid. Condition (iii) then says that the sequence of characters between the left and right characters of each pair must be valid.

Let's see if this definition makes sense. Consider the above sample "matching" involving a, z, (, ), [, ], l, and r and the sequences of Figure 4.12.

Both the pairings of Figure 4.12(a) and those of Figure 4.12(b) satisfy (i) and (ii). However, only the pairing of (a) satisfies (iii). The pairing of (b) fails because the left and right characters of pair 3 are a and z, and the sequence of characters between them is ])1r. This sequence does not satisfy conditions (i), (ii), and (iii), because none of the six possible ways to form two pairs will be valid. It is also apparent that the pairing of (b) fails, because the left and right characters of pair 2 are "[" and "]" and the sequence of characters between them, a, is not valid. One would have to be sure no other pairings for (b) are valid before concluding that (b)is not a valid sequence.

It is not immediately apparent that this problem generalizes the first problem, even if the characters are restricted to be just left and right parentheses. This is because condition (iii) does not appear in the first problem. However, the reasoning used there shows that adding the condition does not change the problem. In fact, if the left and right matching characters are thought of as different kinds of parentheses, the same reasoning applied in the first problem can be applied to the present problem.

(a) Five pairs Satisfying Conditions i-iii

(b) Five Pairs Not Satisfying Conditions i-iii; No Other Five Pairs Will

Figure 4.12 Valid and Invalid Nesting of Matched Characters

Again, any valid sequence must start with some series of n > 0 left parenthese preceding the first appearance of a right parenthesis. Suppose its matching left parenthesis does not immediately precede it. This is the situation, for example, in Figure 4.12(b), where there are four left parentheses, a, (, [, a, appearing before "]," the first right parenthesis. In order for this first right parenthesis to satisfy (i) and (ii), it must then be paired with a previous matching left parenthesis. But then the immediately preceding left parenthesis is the last character in the sequence between that pair of parentheses. In order to satisfy (iii), this sequence must be valid. This sequence cannot be valid since this last left parenthesis has no matching right parenthesis in this sequence, as indicated in Figure 4.13.

In conclusion, a solution may be obtained as follows:

1. Pair the first adjacent left and right parentheses that match, and remove the pair.

2. If the resultant sequence is valid, then so is the original sequence; otherwise the original sequence is not valid.

Figure 4.13 An Invalid Nesting Sequence

Function determine is also an implementation of this solution using a stack, assuming leftparen, rightparen, and match reflect the different left parentheses, right parentheses, and matched pairs, respectively. The stack is needed in order to implement this solution. It is no longer simply a question of increasing or decreasing a tally, depending on whether the program reads a "(" or a ")"parenthesis, and making sure it never goes below zero and ends at zero. It is not even enough to keep a distinct tally for each of the different kinds of "left" and "right" parentheses. This would not result in Figure 4.12(b), for example, being recognized as invalid, because each of the four numbers would satisfy the requirements just mentioned. It is condition (iii), and more than one kind of left and right parentheses, that makes the stack mandatory. Consider a pair satisfying conditions (i) and (ii). To satisfy condition (iii), the sequence of characters between the "left" and "right" parentheses of the pair must also be valid. In effect, a new set of tallies must be kept for that sequence, and they must satisfy the three conditions. The stack is actually holding all of these tallies. Each time a "left" parenthesis appears, it becomes a character in many sequences, in general, and the tally for it, associated with each of those sequences, must be increased by one. Pushing it onto the stack, in effect, increases all these tallies by one simultaneously. Similarly, popping the stack decreases them all simultaneously.

So far the solution has been nonrecursive and has required looking at the input as simply a sequence of parentheses satisfying certain local constraints; a left parenthesis must encounter a matching right parenthesis before any other right parenthesis appears. Any global structure inherent in the input has been ignored. Development of a recursive solution, however, requires that a valid input be defined recursively. The recursive definition of a valid sequence may be restated to show its structure more clearly.

A valid sequence is

      A null sequence

      A left and right matching pair of parentheses with a valid sequence between them

      A valid sequence followed by a valid sequence

In other words, a valid sequence is

      A null sequence,

      One valid sequence surrounded by a pair of matching parentheses, or

      A series of valid sequences, each surrounded by a pair of matching parentheses

The last case would appear as

(valid sequence)(valid sequence) . . . (valid sequence)

The following program is a recursive version of determine based on the recursive definition.

Recursive Version

determine(pvalid,last)

/* Returns true in valid only if the input

sequence is properly nested; last must

initially contain a special character

used as the input terminator.

*/

int *pvalid;

char last;

{

char currentchar;

int done;

    >Comment

static int more;

done = FALSE;

more = TRUE;

while(!done && more)

17 loops through each input character

{

    >Comment

more = readchar(&currentchar);

if(leftparen(currentchar))

    >Comment

determine(pvalid,currentchar);

else if(rightparen(currentchar))

    >Comment

if(!match (last,currentchar))

*pvalid = FALSE;

else

{

    >Comment

done = TRUE;

    >Comment

last = SPECIALCHAR;

}

else

    >Comment

process(last,currentchar);

}

    >Comment

if(last != SPECIALCHAR)

*pvalid = FALSE;

}

Determine must be called initially with last set to specialchar. It is used as an input terminator and should not occur in the sequence being checked. When a valid sequence surrounded by a pair of matching parentheses has been encountered, the variable done is set to true. At that point the program continues looking for a next such sequence. However, last must be reinitialized to the specialchar. The variable more is used to signal whether or not more characters remain to be read in the input string. It is declared to be static because it must retain its value between any recursive calls to determine.

Both this recursive solution and the iterative solution work from the inside out. The iterative program looks for adjacent matching left and right parentheses in the interior of the sequence. The recursive program looks for a valid sequence surrounded by a matching pair of parentheses. Thus the iterative solution sees the input

[ ( [ ] ) ] ( ) { [ ] } #

as

[ ( [ 1 ] ) ] ( 4 ) { [ 5 ] } #     (6 pairs of matching parentheses)

-   2   -    -   6   -

-  - 3   - -

and the recursive solution sees it as

[ ( [ ] ) ] ( 2 ) { [ ] } #    (3 valid sequences each one surrounded by a pair of

- -  1 - -         - 3  -       matching parentheses)

where # denotes the specialchar.

Another recursive solution could be written to work from the outside in, looking for a rightmost mate to the first left parenthesis and then checking the sequence between them for validity. This would lead to a less efficient solution, because parentheses could be considered more than once. Recursion often involves this dual possibility, so both should be explored.

Exercises

1.     Set i and j be two integers and define q(i, j) by q(i, j) = 0 if i < j and q(i - j, j) + 1 if i   [image]  j

a. What is the value of q(7,2)?

b. Can we determine q(0,0)?

c. Can we determine q(-3, -4)?

2.     Let a be an array of type float and i and n be positive integers. The function

m(a,i,n) is defined by

m(a,i,n) = a[i] if i = n and max(m(a,i + 1,n) ,a[i]) if i < n

a. Find m(a,i,n) if i = 1, n = 6, and a is

0.   6.8

1.   3.2

2.  -5.0

3.  14.6

4.   7.8

5.   9.6

6.   3.2

7.   4.0

b. What does m(a,i,n) do?

3.     Let a be an array of type real, and i and n be positive integers. The function m(a,i,n) is defined by

  (i + n)/2   means, "Do an integer divide of (i + n)/2."

a. Find m(a,i,n) for i = 1, n = 6, and a the array of Exercise 2.

b. What does m(a,i,n) do?

4.     Let i be a nonnegative integer. Find b(4) where

b(0) = 1

b(n) = b(0) X b(n - 1) + b(1) X b(n - 2)

+ b(2) X b(n - 3) + . . . + b(n - 1) X b(0) n>0

5.     Write a recursive function max corresponding to Exercise 2.

6.     Write a recursive function max corresponding to Exercise 3.

7.     a. Why does the recursive solution to the Towers of Hanoi problem use the minimum number of moves?

b. What is the number of moves it uses?

8.     Consider the following nonrecursive algorithm for the Towers of Hanoi problem.

1.     Label the disks 1, 2, . . ., n in increasing order of size.

2.     Label the pegs i, a, and   [image] , respectively, n + 1, n + 2, and n + 3.

3.     Move the smallest disk onto another disk with an even label or onto an empty peg with an even label.

4.     If all disks are on the same peg, then terminate. Move the second smallest top disk onto the peg not containing the smallest disk. Go to 1.

a. Convince yourself that this correctly solves the Towers of Hanoi problem.

b. Is the recursive solution or this nonrecursive version easier to understand.

9.     Suppose a fourth peg is added to the Towers of Hanoi problem.

a. Write a recursive solution for this problem.

b. Can you find a recursive solution that uses the minimum number of moves for this problem?

10.     Mark any n distinct points on the circumference of a circle. How many triangles may be drawn whose vertexes lie on three of the n points?

11.     How many ways may k distinct points be selected from the n points of Exercise 10?

12.     Given a permutation of 1, 2, . . ., n, let ni be the number of integers to the right of i which exceed i. For the permutation 2134, n 1 = 2, n 2 = 3, n 3 = 1, n 4 = 4. Write a process function that will turn permutations into a program that prints n1, n2, . . ., nn for each permutation.

13.     Suppose 1 is implemented as a two-way list. Write a recursive function to reverse 1.

14.     Implement the function of Exercise 5.

15.     Implement the function of Exercise 6.

16.     Compare the requirements for the stack storage required during execution for the implementations of Exercises 5 and 6.

17.     Why are the more efficient versions of Section 4.4 more efficient?

18.     The Fibonacci sequence is defined by

  (0) = 0,   (1) = 1

  (n) = f(n - 1) +   (n - 2) for n = 2,3,4 . . .

a. Write a recursive function fibonacci(n) to compute and print   [image] (n).

b. Analyze its time and storage requirements.

19.     a. Translate the program of Exercise 18 and obtain as efficient a version as you can.

b. Analyze its time and storage requirements.

20.     Write a nonrecursive program fibonacci(n), which takes O(n) time and uses only four local variables.

21.     a. Write a function element with parameter k, which returns the value of the kth element on the stack of integers implemented as in Figure 4.8. For instance, if k is 1, it returns the value of the current first element. It does not disturb the stack at all. Assume you do not have access to the stack directly, but only through the push, pop, and empty routines.

b. Same as Exercise 21(a), except assume you do have direct access to the stack.

22.     Two stacks are implemented in one array of length 1,000 as shown. The tops of the stacks are pointed to by top1 and top2 and grow toward each other. Write a procedure push that pushes the contents of v onto the stack indicated by s. s will have a value of 1 or 2. If the stack to which v is to be added is full, then push should not add it, but should return with s set to 0.

23.     Write a function maxstack that returns in m the maximal element on a stack of integers implemented with pointer variables. Maxstack will return true when the stack is not empty, and false otherwise.

24     What does the following do?

pop(&s,&value);

push(&value,&s)

25.     Why are parentheses not needed when prefix or postfix notation is used?

26.     How do conventions eliminate ambiguity in infix notation?

27.     Evaluate the postfix expression 3 6 + 6 2 / *8+.

28.     Simulate the execution of the algorithm of Section 4.6.2 for evaluation of an infix expression on (a * b/(c + d)) + e **   [image] .

29.     Simulate the execution of the translation algorithm of Section 4.6.3 on (a * b/(c + d)) + e **   [image] .

30.     What priority must be given to the assignment operator, =, if the evaluation and translation algorithms for infix to postfix are to work on assignment statements as well as arithmetic expressions?

31.     Write the function evaluate.

32.     Describe how evaluate must be modified if arithmetic expressions are allowed to include function calls.

33.     What will the pairing generated by the function determine be for the input sequence aa([[]()])zz?

34.     Modify the function determine so that it outputs the pairs as follows:

1 2 3 4 5 6 7 8

if ( ( ( ) ) ( ) ) is the input sequence,

then the output will be the pairs 3 4, 2 5, 6 7, 1 8.

35.     Modify function determine so that it also outputs (  (  (  )  ) (  )  ), for example, if the input is ( ( ( ) ) ( ) ).

4 2 1 1 2 3 3 4

36.     How must determine be modified if some left characters may also be right characters, and they will always match each other? For example, let the left characters be a, (, [,b, with the "right" characters being a, ), ], b. Here a is matched with a, b with b, "("with ")," and "[" with "]." Then a[a] is not valid, but a[CC](aa)a and ab()aab[]a are.

37.     Write the remove function for the array implementation of a queue.

38.     Write the insert function for the pointer variable implementation of a queue.

39.     Write the remove function for the pointer variable implementation of a queue.

40.     Suppose function buffer has three parameters, q of type queue, e of type whatever, and op of type character. If op is r, then buffer returns after removing the top record of the queue and copying its value into e. If op is n, then buffer returns after inserting the value in e at the bottom of the queue. Write an implementation of buffer, assuming that access to the queue is only through the empty, remove, and insert routines.

41.     A deque (pronounce like "deck") is a data abstraction that retains ordered information, but allows insertion and deletion at both its front and rear.

a. Develop an array-based implementation of a deque and write corresponding setdeque, insertf, insertr, removef, remover, and empty routines.

b. Do the same as in Exercise 41(a), except use pointer variables with dynamic storage allocation for the deque implementation.

42.     Suppose there is a program to keep track of the current board configuration in a game of Monopoly by using a circular array of records, each record representing a board position (such as Boardwalk), and queues for Community Chest and Chance. Discuss how the records should be defined, what pointers to records will be convenient, and how these pointers and the circular array should be processed on each turn.

Suggested Assignments

1.     Write and simulate a recursive program to reverse a list.

2.     You might enjoy finding an algorithm for the following generalization of the Towers of Hanoi problem. The disks are each labelled with a positive integer. A configuration of disks on a peg is "legal" if each disk's integer is no larger than the sum of the integers on all disks below it, or there are no disks below it. How can the disks be moved so that any legal original configuration of n disks on peg 1 ends up on peg 3, with all intermediate configurations being legal?

3.     Write the permutation program so it treats the permutation as a data abstraction.

4.     In order to use storage efficiently when n stacks are needed during the execution of a program, it may be desirable to implement all the stacks in one array as shown here for n= 4.

The Bi's and Ti's point to the bottoms and tops of the stacks, which grow to the right as indicated. Think of the total storage currently allocated to stack i as being from B(i) to B(i + 1). Stack i is actually using T(i) - B(i + 1) + 1 positions of the array. If B1 = 1, B2 = 10, B3 = 15, B4 = 25, B5 = size of the array = 35, T1 = 3, T2 = 12, T3 = 20, and T4 = 33, then the stacks are using 3, 3, 6, and 9 entries, respectively, while they are allocated 9, 5, 10, and 10 entries.

When an entry is to be inserted into a stack that is currently using all its allocated storage, the stack is said to have overflowed. Stack i overflows if Ti = B(i + 1) - 1. Write a function to manage the storage provided by the array. Whenever a stack overflows, the function is to be invoked. If there is any unused space in the array, then the stack with an unused but allocated position which is "closest" to the overflow stack is to give up one of its unused positions. This position is then used by the overflow stack to accommodate its entry. This function must be written to shift stacks around. If no unused space exists in the array, then the procedure should print an "out of stack storage" message.

Go to Chapter 5      Return to Table of Contents


Data Structures Using C: CHAPTER 5: PACKAGING DATA ABSTRACTIONS

 Return to Table of Contents

CHAPTER 5: PACKAGING DATA ABSTRACTIONS

Introduces the concept of modules and file modularization to structure programs

With this type of program structure data abstractions are placed in files separate from the program file, and this hides the details of the data abstraction from the programmer using them easing the programming task increases efficiency as it enables the data abstractions to be compiled separately and stored in object code (separate compilation is possible)

To illustrate how programs are structured using modules and file modularization four examples, from previous chapters, are presented in detail

primes

stablemarriages

creation and printing of a list

nonrecursive Towers of Hanoi

5.1: Data Abstraction Revisited

The first step in creating a program is to develop the algorithm that will produce the solution. The algorithm must detail operations that will be performed on the given data. A decision must also be made about how the data will be organized. Typically data is best split into distinct groups, or collections, based on the basic operations required for each collection. The data structure utilized for each collection can be tailored to the specific operations to be performed on it. Each collection, with its operations, can then be treated individually as a data abstraction.

A data abstraction is a collection of data upon which only a set of specific operations are allowed. The technique of data abstraction has been used in this text as a tool in creating programs that are understandable and maintainable. Insight is required to determine the particular collection to use and the operations to act upon it. The same collection may be useful in a number of different problems. This may be so with the same set of allowable operations or with different sets of operations. As an example, recall that the stack data abstraction was used in both towers and next with the same operations.

Once the data abstractions to be used in a solution are determined, a decision must be made on how to implement them. This requires deciding what data structure is to be used to store the collection and how the operations on it will be defined. In this text each operation has been incorporated in a function using the tool of functional modularization. Treating the collection and its operations as a data abstraction means that the program can only process the collection using those functions. This guarantees that the collection is accessed only when one of the functions is invoked. Proper packaging of the data abstraction can thus ensure that it is used appropriately. How this packaging is done is the topic of this chapter.

The discussion is presented only now for several reasons. First, preceding chapters have introduced a number of data abstractions that can serve as vehicles to present different approaches to packaging. Next, packaging introduces another level of abstraction, which can complicate the understanding of the basic material. Time was needed to absorb the beginning concepts. Finally, packaging is language-dependent, and the aim of this book is to be as language-independent as possible yet still present programs in the C programming language. Given the language-dependence of packaging and the belief that students should first concentrate on basics, the topic is confined to this chapter. This does not mean it is unimportant--only that it might cloud the other concepts presented. Consequently the material in this chapter may be skipped or, alternatively, imposed on the programs in the later chapters. The C approaches to packaging are straightforward; should you desire, the programs can be readily modified to the style presented here.

5.2: The Prime Example

Consider the program below from Chapter 1 for calculating prime numbers.

#include <stdio.h>

    >Comment

main()

{

int n;

printf("\n n = ?\n");

scanf("%d",&n);

primes(n);

}

    >Comment

#define TRUE 1

#define FALSE 0

#define MAXN 500

typedef int collection[MAXN];

set(n,c)

/* Sets c to empty */

int n;

collection c;

{

int i;

for (i=1;i<=n;i++)

c[i] = FALSE;

}

insert(i,c)

/* Inserts i in c */

int i;

collection c;

{

c[i] = TRUE;

}

omit(i,c)

/* Removes i from c */

int i;

collection c;

{

c[i] = FALSE;

}

belongs(i,c)

/* Returns true only

if i is in c

*/

int i;

collection c;

{

return(c[i]);

}

    >Comment

primes(n)

/* Prints all prime numbers between 2 and n */

int n;

{

collection candidates;

create(n,candidates);

remove(n,candidates);

print(n,candidates);

}

create(n,c)

/* Creates a collection of integers between 2 and n */

int n;

collection c;

{

int i;

    >Comment

set(n,c);

    >Comment

for (i=2;i<=n;i++)

insert(i,c);

}

remove(n,c)

/* Removes all nonprimes from c */

int n;

collection c;

{

int firstprime;

int factor;

firstprime = 2;

    >Comment

factor = firstprime;

while (nonprimes(factor,n))

{

delete(factor,n,c);

factor++;

}

}

nonprimes(factor,n)

/* Returns true only if non

primes may remain in c

*/

int factor,n;

double sqrt();

{

    >Comment

return(factor <= sqrt((double)n));

}

delete(factor,n,c)

/* Deletes multiples of factor from c */

int factor,n;

collection c;

{

int nextmultiple;

    >Comment

nextmultiple = 2 * factor;

while (nextmultiple <= n)

{

omit(nextmultiple,c);

nextmultiple = nextmultiple + factor;

}

}

print(n,c)

/* Prints integers in c */

int n;

collection c;

{

int i;

    >Comment

for (i=2;i<=n;i++)

if (belongs(i,c))

printf("\n %d\n",i);

}

This program is structured with the data abstraction collection and its operations set, insert, omit, and belongs placed together. This facilitates finding its data structure and function declarations and definitions, whenever modifications are desired. Changes to collection, set, insert, omit, and belongs may be made, as long as they still do their tasks correctly without changing any other part of the program.

The program, however, is more cluttered and longer than necessary. It is burdened with exhibiting details of the data abstraction that are irrelevant to it. One way to avoid this is to use the include capability of C. Using C's include statement, we can

      Create a file called collection that contains all the statements specifying the data abstraction, and then

      Insert these into the program with one statement--the include statement

This is done in the following version of the program.

When this program is compiled, the C preprocessor will replace the include "collection" statement in the program file with the source code from the file collection. Thus the object code produced by the compiler for this version will be identical to that produced by the first version.

#include <stdio.h>

    >Comment

main()

{

int n;

printf("\n n = ?\n");

scanf("%d",&n);

primes(n);

}

    >Comment

#include "collection"

    >Comment

primes(n)

/* Prints all prime numbers between 2 and n */

int n;

{

collection candidates;

create(n,candidates);

remove(n,candidates);

print(n,candidates);

}

create(n,c)

/* Creates a collection of integers between 2 and n */

int n;

collection c;

{

int i;

    >Comment

set(n,c);

    >Comment

for (i=2;i<=n;i++)

insert(i,c);

}

remove(n,c)

/* Removes all nonprimes from c */

int n;

collection c;

{

int firstprime;

int factor;

firstprime = 2;

    >Comment

factor = firstprime;

while (nonprimes(factor,n))

{

delete(factor,n,c);

factor++;

}

}

nonprimes(factor,n)

/* Returns true only if non

primes may remain in c

*/

int factor,n;

double sqrt();

{

    >Comment

return(factor <= sqrt((double)n));

}

delete(factor,n,c)

/* Deletes multiples of factor from c */

int factor,n;

collection c;

{

int nextmultiple;

    >Comment

nextmultiple = 2 * factor;

while (nextmultiple <= n)

{

omit(nextmultiple,c);

nextmultiple = nextmultiple + factor;

}

}

print(n,c)

/* Prints integers in c */

int n;

collection c;

{

int i;

    >Comment

for (i=2;i<=n;i++)

if (belongs(i,c))

printf("\n %d\n",i);

}

The file collection, which can be stored separately, contains:

/* The data abstraction collection

and its allowed operations */

    >Comment

#define TRUE 1

#define FALSE 0

#define MAXN 500

typedef int collection [MAXN];

set(n,c)

/* Sets c to empty */

int n;

collection c;

{

int i;

for (i=1;i<=n;i++)

c[i] = FALSE;

}

insert(i,c)

/* Inserts i in c */

int i;

collection c;

{

c[i] = TRUE;

}

omit(i,c)

/* Removes i from c */

int i;

collection c;

{

c[i] = FALSE;

}

belongs(i,c)

/* Returns true only

if i is in c

*/

int i;

collection c;

{

return(c[i]);

}

So far, the programs in this text have been written so that they are compiled as a single source code file. This is still true for the two versions of primes. The first was written as a single source file. In the second, the include statement causes the precompiler to incorporate collection into the program file, producing a single source code file.

It is also possible in C to locate the functions of a program in more than one file. Each file of source code can be separately compiled, producing a corresponding file of object code. This ability to allocate the functions of a program to different files is a tool that can be used to achieve another kind of modularity besides functional modularity. This new modularity, which we call file modularity, allocates functions to files, whereas functional modularity allocates tasks to functions. In keeping with this definition of file modularity, the term module will be used to mean one or more files, to be separately compiled, used for the implementation of a data abstraction. Before compilation they are source modules; after compilation they are object modules.

To create a module for collection, two files in addition to the program file will be needed. They are called collection.h and collection.c. Collection.h, the header file, contains all definitions needed for collection's data structure and all declarations for its functions. The header file must be included in any file using this data abstraction. Collection.c, which contains definitions of all the functions of the data abstraction that implement its operations, must also include the header file, as in the following example.

The two files are shown below. In collection.h a standard way of declaring a function is presented. It uses the extern declaration, as in the following example.

extern set(/* int,collection */);

The extern declaration informs the compiler that the function is defined elsewhere, in another file. The comment /* int,collection */ identifies the parameters of the function--since it is a comment, it disappears in compilation.

With this version, the program file and collection are compiled separately into object modules. The linker must combine the two object modules to create an executable program.

The File collection.h

/* header for collection of integers */

#define TRUE 1

#define FALSE 0

#define MAXN 500

typedef int collection[MAXN];

extern set(/* int,collection */);

/* Sets collection to empty */

extern insert(/* int,collection */);

/* Adds int to collection */

extern omit(/* int,collection */);

/* Takes int out of collection */

extern belongs(/*int,collection */);

/* Returns true only if int is in collection */

The File collection.c

/* operations for collection of integers */

    >Comment

#include "collection.h"

set(n,c)

/* Sets c to empty */

int n;

collection c;

{

int i;

for (i=1;i<=n;i++)

c[i] = FALSE;

}

insert(i,c)

/* Inserts i in c */

int i;

collection c;

{

c[i] = TRUE;

}

omit(i,c)

/* Removes i from c */

int i;

collection c;

{

c[i] = FALSE;

}

belongs(i,c)

/* Returns true only

if i is in c

*/

int i;

collection c;

{

return(c[i]);

}

Final Version: The File primestest.c

#include <stdio.h>

    >Comment

main()

{

int n;

printf("\n n = ?\n");

scanf("%d",&n);

primes(n);

}

    >Comment

#nclude "collection.h"

    >Comment

primes(n)

/* Prints all prime numbers between 2 and n */

int n;

{

collection candidates;

create(n,candidates);

remove(n,candidates);

print(n,candidates);

}

create(n,c)

/* Creates a collection of integers between

2 and n */

int n;

collection c;

{

int i;

    >Comment

set(n,c);

    >Comment

for (i=2;i<=n;i++)

insert(i,c);

}

remove(n,c)

/* Removes all nonprimes from c */

int n;

collection c;

{

int firstprime;

int factor;

firstprime = 2;

    >Comment

factor = firstprime;

while (nonprimes(factor,n))

{

delete(factor,n,c);

factor++;

}

}

nonprimes(factor,n)

/* Returns true only if non

primes may remain in c

*/

int factor,n;

double sqrt();

{

    >Comment

return(factor <= sqrt((double)n));

}

delete(factor,n,c)

/* Deletes multiples of factor from c */

int factor,n;

collection c;

{

int nextmultiple;

    >Comment

nextmultiple = 2 * factor;

while (nextmultiple <= n)

{

omit(nextmultiple,c);

nextmultiple = nextmultiple + factor;

}

}

print(n,c)

/* Prints integers in c */

int n;

collection c;

{

int i;

    >Comment

for (i=2;i<=n;i++)

if (belongs(i,c))

printf("\n %d\n",i);

}

Although this third version may seem similar to the second, it differs in an important way. Its functions are distributed among three files. Two of these, collection.h and collection.c, are used by the module for collection, the data abstraction. This allows the details of the implementation to be hidden, removes the need for the program itself to change whenever the data abstraction implementation is changed, and also shortens the program. The second version does all this as well, but the third version is even better. It allows collection.c to be compiled separately from file primestest.c, the file containing main, primes, create, and so on. More precisely, primestest.c must be compiled, and then the object code for primestest.c, along with the object code from the separately compiled file collection.c, can be linked by the linker. Finally, the resultant executable file produced by the linker can be run.

Once collection.c has been successfully compiled and verified, its object module can be stored in a library for use in the creation of programs such as primes. In this way C fosters the building up of programs by putting together previous solutions. In the creation or modification of large programs, the use of already compiled modules can significantly reduce time for compiling and program development.

Whenever the implementation of the data abstraction collection changes, changes may occur in the source code of collection.h or of collection.c. When they occur in collection.h, any file that includes it must be recompiled. In our example, this means that collection.c and primestest.c must be recompiled. To run primestest.c, its compiled version, along with the compiled version of the new collection.c, must be linked; then the resultant executable program file can be run. The new object code produced by compiling the new version of collection.c should replace the old version in the library. When the changes occur in collection.c only, it must be recompiled followed by relinking with the object code for primestest.c before the new executable program file can be run.

5.3: Stable Marriages

As a second example of file modularization, consider the stable marriage program of Section 2.5.2. That program is structured with the data abstractions pairings, with its operations mate, read_mens_mates, and create_womens_mates placed together, and preferences, with its operations mostpreferred, nextpreferred, prefers, read_preferences, and create_priorities also placed together. Grouping each data abstraction facilitates finding its data structures, function declarations, and definitions, whenever modifications to them are required. Such changes will not necessitate changing any other part of the program, since it was written treating pairings and preferences as data abstractions. Also, changes to one data abstraction do not require changes in the other. The program in Section 2.5.2 is written as a single source file. We now place the two data abstractions in modules. The modules and the program file structure of the program using file modularization are printed below. The headers, operations and program file are, respectively, pairings.h, pairings.c, preferences.h, preferences.c, and stablemarriages.c.

The File pairings.h

/* header for pairings */

#define MAXN 50

typedef int mates[MAXN];

extern mate(/* int, pairings */);

/* Returns the mate of person int */

extern read_mens_mates(* int, pairings */);

/* Reads the mates of the men(int of them)

into pairings

*/

extern create_womens_mates(/* int,pairings_1,pairings_2 */);

/* Creates the pairings_2 of the

men(int of them), based on

the mens' mates specified by

pairings_1

*/

The File pairings.c

/* operations allowed on pairings */

    >Comment

#include "pairings.h"

mate (person,pairs)

/* Returns persons mate

as specified by pairs */

int person;

pairings pairs;

{

return(pairs[person]);

}

read_mens_mates(n,pairs)

/* Reads the n wives into pairs */

int n;

pairings pairs;

{

int i;

printf("\n mpairs = ? \n");

for (i=1;i <=n;i++)

scanf("%d",&pairs[i]);

}

create_womens_mates(n,pairs1,pairs2)

/* Pairs 1 specifies the n wives, one

for each man. Pairs2 is set to

specify the n men, one for each woman.

*/

int n;

pairings pair;

{

int i,j;

for (i=1;i<=n;i++)

{

for (j=1;pairs1[j] != i;j++)

;

pairs2 [i] = j;

}

}

The File preferences.h

/* header file for preferences */

#define MAXN 50

typedef int preferences[MAXN][MAXN];

extern mostpreferred(/* int,preferences */);

/* Returns the mate most preferred by

person int as specified by pref */

extern nextpreferred(/* int_1,int_2,pref */);

/* Returns the mate preferred most, after

the mate given by index int_2, by person

int_1 as specified by pref

*/

extern prefers(/* int_1,int_2,int_3,pref */);

/* Returns true if individual int_1 prefers

person int_2 to individualsmate int_3

as specified by pref

*/

extern read_preferences(/* int,pref */);

/* Reads int number of rows of

preferences into pref

*/

extern create_priorities(/* int,pref_1,pref_2 */);

/* Creates int number of individual

priorities in pref_2 based on the

individual preferences specified

by pref_1

*/

The File preferences.c

/* operations allowed on preferences */

    >Comment

#include "preferences.h"

mostpreferred (person,pref)

/* Returns the mate most preferred by

person as specified by pref */

int person;

preferences pref;

{

return(pref[person][1]);

}

nextpreferred(person,index,pref)

/* Returns the mate preferred most, after

the mate given by index, by person

as specified by pref

*/

int person,index;

preferences pref;

{

return(pref[person][index]);

}

prefers(individual,person,individualsmate,priority)

/* Returns true if individual prefers person to

individuals mate as specified by priority

*/

int individual,person,individualsmate;

preferences priority;

{

return(priority[individual][person] <

priority [individual] [individualsmate]);

}

read_preferences (n,pref)

/* Reads n rows of

preferences into pref

*/

int n;

preferences pref;

{

int i,j;

for (i=1;i<=n;i++)

for (j=1;j<=n;j++)

scanf ("%d",&pref[i][j]);

}

create_priorities(n, pref, priority)

/* Creates n individual priorities

in priority based on the

individual preferences specified

by pref

*/

int n;

preferences pref,priority;

{

int i,j;

for (i=1,i<=n;i++)

for (j=1;j<=n;j++)

priority[i][pref[i][j]] = j;

}

The File stablemarriages.c

#include <stdio.h>

    >Comment

#include "pairings.h"

    >Comment

#include "preferences.h"

#define TRUE 1

#define FALSE 0

main()

{

pairings mpairs,wpairs;

preferences mpref,wpref,mpriority,wpriority;

int i,j,n;

printf("\n n = ?\n");

scanf("%d",&n);

read_mens_mates(n,mpairs);

create_womens_mates(n,mpairs,wpairs);

printf("\n Enter mens preferences \n");

read_preferences(n,mpref);

printf("\n Enter womens preferences \n");

read_preferences(n,wpref);

create_priorities(n,mpref,mpriority);

create_priorities(n,wpref,wpriority);

if (stabilitycheck(n,mpairs,wpairs,mpref,wpref,mpriority,

wpriority))

printf(" The pairings are stable \n");

else

printf(" The pairings are not stable \n");

printf(" The pairings are: \n");

printf(" man wife \n");

for (i=1;i<=n;i++)

printf("%d %d \n",i,mate (i,mpairs));

}

stabilitycheck(n,mpairs,wpairs,mpref,wpref,mpriority,wpriority)

int n;

pairings mpairs,wpairs

preferences mpref,wpref,mpriority,wpriority;

{

int man,wife,individual,individualsmate,index,stable;

stable = TRUE;

for(man=1;(stable && man<=n);man++)

{

wife = mate(man,mpairs);

individual = mostpreferred(man,mpref);

index = 1;

while (stable && (individual != wife))

{

individualsmate = mate(individual, wpairs);

if(prefers(individual,man,individualsmate,

wpriority))

stable = FALSE;

else

{

index++;

individual = nextpreferred(man,index,mpref);

}

}

individual = mostpreferred(wife,wpref);

index = 1;

while(stable && (individual != man))

{

individualsmate = mate(individual,mpairs);

if(prefers(individual,wife,individualsmate,

mpriority))

stable = FALSE;

else

{

index++;

individual = nextpreferred(wife,index,wpref);

}

}

}

return(stable);

}

The include "pairings.h" and include "preferences.h" were put at the top of the program file because their contents are needed from that point in the file. In the program file for primestest.c, the include "collection.h" appears just before the primes function, since its contents are only needed from that point on in the file.

The files pairings.c, preferences.c, and stablemarriages.c can be compiled separately. To run stable marriages.c, they must first all be linked by the linker.

Any time changes are made in pairings.h, the files pairings.c and stablemarriages.c must be recompiled. Changes in pairings.c also require that pairings.c be recompiled . The same applies for preferences.h and preferences.c. However, changes to only one of pairings.c, preferences.c, or stablemarriages.c require only its individual recompilation. Relinking is always required before execution.

5.4 : A List Example

Let us look at another example of file modularization of a program. Consider the program of Section 3.6.1 that reads in, creates, and prints the information fields of each record on a list of stock records. That version is in a single file and is functionally modularized. The data abstraction list, with its operations next, setlink, setinfo, avail, getnextrecord, printrecord, and anotherrecord, appear together.

The version of the program that follows uses a module for the list data abstraction with the header file, list.h, and the operations file, list.c. The file, list.c, and the program file, listexample.c, are compiled separately.

The file list.h:

The File list.h

/* header for list */

#define MAXSIZE 5

#define NULL 0

#define SENTINEL -1

typedef struct

{

int month;

int day;

int year;

}date;

typedef struct

{

char name[MAXSIZE];

int shares;

float value;

date datebought;

}infofield;

typedef struct listrecord

{

infofield info;

struct listrecord *link;

}stockrecord,*listpointer;

extern listpointer next(/* listpointer */);

/* Returns a copy of listpointer in link field of

record pointed to by listpointer */

extern listpointer setnull();

/* Returns a null listpointer value */

extern setlink(/* listpointer_1,listpointer_2 */);

/* Sets the link field of the record pointed to

by listpointer_1 to the value of

listpointer_2

*/

extern setinfo(/* listpointer_1,listpointer_2 */);

/* Sets the info field of the record pointed to

by listpointer_1 to the contents of the

info field of the record pointed to by

listpointer_2

*/

extern listpointer avail();

/* Returns a pointer to storage for

a record of type struct stockrecord */

extern listpointer getnextrecord();

/* Returns a pointer to a record

containing the next input records'

information, or a null pointer if

the next input is the sentinel

*/

extern printrecord(/* listpointer */);

/* Prints the information in the record

pointed to by listpointer */

extern anotherrecord(/* listpointer */);

/* Returns true if there are no

more records and false otherwise */

The File list.c

/* operations allowed on a list */

    >Comment

#include "list.h"

listpointer next(pointer)

/* Returns a copy of the

link field in the record

pointed to by pointer

*/

listpointer pointer;

{

return (pointer->link);

}

listpointer setnull()

/* Returns a null pointer */

{

return(NULL);

}

setlink(pointer1,pointer2)

/* Sets the link field of the

record pointed to by pointer1

to the contents of pointer2

*/

listpointer pointer1, pointer2;

{

pointer1->link = pointer2;

}

setinfo(pointer1,pointer2)

/* Sets the infofield of the record

pointed to by pointer1 to the

infofield of the record

pointed to by pointer2

*/

listpointer pointer1,pointer2;

{

int i;

for (i=0;i<MAXSIZE;i++)

pointer1->info.name[i] = pointer2->info.name[i];

pointer1->info.shares = pointer2->info.shares;

pointer1->info.value = pointer2->info.value;

pointer1->info.datebought.month = pointer2->info.datebought.

month;

pointer1->info.datebought.day = pointer2->info.datebought.day;

pointer1->info.datebought.year = pointer2->info.datebought.year;

}

listpointer avail()

*/ Returns a pointer to storage

allocated for a record of type

stockrecord

*/

{

listpointer malloc();

return(malloc(sizeof(stockrecord)));

}

anotherrecord(recordpointer)

/* Returns true if there is

another record

*/

listpointer recordpointer;

{

listpointer setnull();

return(recordpointer != setnull());

}

listpointer getnextrecord()

/* Inputs the data for the new

record and returns a pointer

to a record containing the data

in its infofield, or, if there

are no new records, returns

a null pointer

*/

{

listpointer setnull();

static stockrecord nextrecord;

printf(" Enter number of shares \n");

scanf("%d",&(nextrecord.info.shares));

if(nextrecord.info.shares != SENTINEL)

{

printf(" Enter the stock name - less than %d characters\n",

MAXSIZE);

scanf("%s",nextrecord.info.name);

printf(" Enter the price of one share of the stock \n");

scanf("%f",&(nextrecord.info.value));

printf(" Enter the month day year of the stock purchase

\n");

scanf("%d %d %d",&(nextrecord.info.datebought.month),

&(nextrecord.info.datebought.day),

&(nextrecord.info.datebought.year));

return(&nextrecord);

}

else

return(setnull());

}

printrecord(recordpointer)

/* Prints the contents of the infofield

of the record pointed to by recordpointer

*/

listpointer recordpointer;

{

printf(" The stock is %s \n",recordpointer->info.name);

print(" The number of shares is %d \n",recordpointer->info.

shares);

printf(" The value of a share is %f\n",recordpointer->info.

value);

printf(" The date of purchase is %d %d %d\n\n",

recordpointer->info.datebought.month,

recordpointer->info.datebought.day, recordpointer->

info.datebought.year);

}

Finally, here is the program file to create a list and print its contents. It treats the list as a data abstraction and uses file modularization for the data abstraction.

The File listexample.c

    >Comment

#include "list.h"

    >Comment

#include <stdio.h>

#define TRUE 1

#define FALSE 0

main()

/* Inputs a series of stockrecords,

creates a list of the records, and

prints the contents of each list

record's infofield

*/

{

listpointer stocks,recordpointer,next();

int done;

    >Comment

initialize(&stocks);

recordpointer = stocks;

done = FALSE;

while(!done&&anotherrecord(recordpointer))

{

    >Comment

addrecord(&stocks,recordpointer,&done);

recordpointer = next(recordpointer);

}

    >Comment

recordpointer=stocks;

while (anotherrecord(recordpointer))

{

    >Comment

printrecord(recordpointer);

recordpointer = next(recordpointer);

}

}

initialize(plistname)

/* Allocates storage for the first record

and sets listname to point to it

*/

listpointer *plistname;

{

listpointer avail();

*plistname = avail();

}

addrecord(plistname,recordpointer,pdone)

/* Fills in the current record's data, and

allocates storage for the next record and

adds it to the list. If there is no data for

the current record it sets the link field of

the last list record to null, or the list

head, to null, and done to true.

*/

listpointer *plistname,recordpointer;

int *pdone;

{

static listpointer predecessor;

listpointer setnull(),avail(),

getnextrecord(),pointer;

pointer = getnextrecord();

if (pointer != setnull())

{

    >Comment

setinfo(recordpointer,pointer);

setlink(recordpointer,avail());

predecessor = recordpointer;

}

else if (*plistname != recordpointer)

{

    >Comment

setlink(predecessor,setnull());

*pdone = TRUE;

}

else

{

    >Comment

*plistname = setnull();

*pdone = TRUE;

}

}

The files list.h and list.c now embody the list data abstraction. Note that, should the format of the list change, these changes must be reflected in the header file and also in the operations file. In the later file, list.c, the changes would be localized to setinfo, getnextrecord, and printrecord. Instead, were the format to remain the same but the records stored in an array of records rather than the dynamic memory, more extensive changes would be required in both lists.h and list.c.

5.5: A Stack Example

This last example involves a stack and its application in the nonrecursive towers program of Section 4.5.3. The stack is implemented as a list. The program treats the stack as a data abstraction and is structured so the stack and its operations appear together.

We give the header file, stack.h, the operations file, stack.c, and the program file, towerstest.c, that can be used to achieve file modularization and separate compilation.

The File stack.h

/* header for stack */

#define NULL 0

typedef struct

{

int n;

int i;

int a;

int f;

}whatever;

typedef struct record

{

whatever info;

struct record *link;

}stackrecord,*stack;

extern setstack(/* pstack */);

/* Sets stack to empty */

extern empty (/* pstack */);

/* Returns true only if

the stack is empty */

extern push(/* pstackrecord,pstack */);

/* Makes stackrecord the new

top entry on stack*/

extern pop(/* pstack,pstackrecord */);

/* Removes the top entry from

stack and returns with its

contents in stackrecord

*/

extern overflow(/* pstack */);

/* Prints a warning when

the stack overflows */

extern underflow(/* pstack */);

/* Prints a warning when the stack underflows */

The File stack.c

/* operations for the stack */

    >Comment

#include "stack.h"

setstack(ps)

/* Sets stack s to empty */

stack *ps;

{

*ps = NULL;

}

empty(ps)

/* Returns true only if

the stack s is empty */

stack *ps;

{

return(*ps == NULL);

}

push(pnewrecord,ps)

/* Makes stackrecord the new

top entry on stack s */

stackrecord *pnewrecord;

stack *ps;

{

stackrecord *newentry;

newentry = malloc(sizeof(stackrecord));

newentry->info = pnewrecord->info;

newentry->link = *ps;

*ps = newentry;

}

pop(ps,pvalue)

/* Removes the top entry from

stack s and returns with its

contents in stackrecord

*/

stack *ps;

stackrecord *pvalue;

{

if(!empty(ps))

{

pvalue->info = (*ps)->info;

*ps = (*ps)->link;

}

else

underflow(ps);

}

overflow(ps);

/* Prints a warning when

the stack overflows */

stack *ps;

{

printf (" The stack has overflowed /n");

}

underflow(ps);

/* Prints a warning when

the stack underflows */

stack *ps;

{

printf(" The stack has underflowed /n");

}

The File towerstest.c

#include <stdio.h>

main()

{

int n;

printf("\n enter a value for n \n");

scanf("%d",&n);

towers(n,1,2,3);

}

    >Comment

#include "stack.h"

towers(n,i,a,f)

/* Moves the top n disks

from peg i to peg f

*/

int n,i,a,f;

{

stack s;

int done

setstack(&s);

done = FALSE;

while(!done)

{

while(n > 1)

{

s_tack(n,i,a,f,&s);

setvar 1(&n,&i,&a,&f);

}

printf("\n %d -> %d\n",i,f);

if(!empty(&s))

{

restore(&n,&i,&a,&f,&s);

printf("\n %d -> %d\n",i,f);

setvar2(&n,&i,&a,&f);

}

else

done = TRUE;

}

setvar1(pn,pi,pa,pf)

/* Sets n to n - 1 and

interchanges f and a

*/

int *pn,*pi,*pa,*pf;

{

int t;

*pn = *pn - 1;

t = *pf;

*pf = *pa;

*pa = t;

}

setvar2(pn,pi,pa,pf)

/* Sets n to n - 1 and

interchanges a and i

*/

int *pn,*pi,*pa,*pf;

{

int t;

*pn = *pn - 1;

t = *pa;

*pa = *pi;

*pi = t;

}

s_tack(n,i,a,f,ps)

/* Creates a record containing

n, i, a, and f and inserts it

on stack s

*/

int n,i,a,f;

stack *ps;

{

stackrecord newrecord;

newrecord.info.n = n;

newrecord.info.i = i;

newrecord.info.a = a;

newrecord.info.f = f;

push(&newrecord,ps);

}

restore(pn,pi,pa,pf,ps)

/* Removes the top record from

stack s and copies its contents into n,i,a,f

*/

int *pn,*pi,*pa,*pf;

stack *ps;

{

stackrecord value;

pop(ps,&value);

*pn = value.info.n;

*pi = value.info.i;

*pa = value.info.a;

*pf = value.info.f;

}

Notice that the operations push and pop in stack.c have been written under the assumption that the C compiler allows the entire contents of one record to be copied into another record of the same type using just one assignment statement. In order to achieve greater portability, these operations could be written so that they do not depend on this assumption by using individual assignment statements for each field of the records. However, this can also be accomplished by restructuring the program. Moreover, restructuring can also achieve another level of data abstraction with respect to the stack records and their operations (in this example only one will be needed). This will require changes to stack.h and stack.c but will make the new stack module independent of the implementation of the stack records. The current stack module, consisting of the files stack.h and stack.c, must be changed and recompiled whenever the records being stacked have a different format from that which is represented by whatever. After the restructuring this will not be necessary. Any changes in the stack record format will be localized to the stack record module.

To do the restructuring, first create a module for the data abstraction stackrecord and its operations. The module consists of the files stack_record.h and stack_record.c.

The File stack_record.h

/* header for stackrecord */

typedef struct

{

int n;

int i;

int a;

int f;

}whatever;

typedef struct record

{

whatever info;

struct record *link;

}stackrecord,*stackrecordpointer;

extern setinfo(/* stackrecordpointer_1,stackrecordpointer_2 */);

/* Sets the info field of the record pointed to

by stackrecordpointer_1 to the contents of the

info field of the record pointed to by stackrecordpointer_2

*/

The File stack_record.c

/* operations for stackrecord */

    >Comment

#include "stack_record.h"

setinfo(pointer1,pointer2)

/* Copies the contents of the record

pointed to by pointer 1 into

the record pointed to by pointer2

*/

stackrecordpointer pointer1,pointer2;

{

pointer1->info.retrn = pointer2->info.retrn;

pointer1->info.n = pointer2->info.n;

pointer1->info.i = pointer2->info.i;

pointer1->info.a = pointer2->info.a;

pointer1->info.f = pointer2->info.f;

}

Here is the new stack module.

The File stack.h

/* header for stack */

    >Comment

#include "stack_record.h"

#define NULL 0

stackrecord *stack;

extern setstack(/* pstack */);

/* Sets stack to empty */

extern empty(/* pstack */);

/* Returns true only if the stack is empty */

extern push(/* pstackrecord,pstack */);

/* Makes stackrecord the new top entry on stack*/

extern pop(/* pstack,pstackrecord */);

/* Removes the top entry from

stack and returns with its

contents in stackrecord

*/

extern overflow(/* pstack */);

/* Prints a warning when

the stack overflows */

extern underflow(/* pstack */);

/* Prints a warning when

the stack underflows */

File stack.c

/* operations for the stack */

    >Comment

#include "stack.h"

setstack(ps)

/* Sets stack to empty */

stackrecordpointer *ps;

{

*ps = NULL;

}

empty(ps)

/* Returns true only if

the stack is empty */

stackrecordpointer *ps;

{

return(*ps == NULL);

}

push(pnewrecord,ps)

/* Makes stackrecord the new

top entry on stack*/

stackrecord *pnewrecord;

stackrecordpointer *ps;

{

stackrecord *newentry;

newentry = malloc(sizeof

(stackrecord));

    >Comment

setinfo(newentry,pnewrecord);

newentry->link = *ps;

*ps = newentry;

}

pop(ps,pvalue)

/* Removes the top entry from

stack and returns with its

contents in stackrecord

*/

stackrecordpointer *ps;

stackrecord *pvalue;

{

if(!empty(ps))

{

    >Comment

setinfo(pvalue,(*ps));

*ps = (*ps)->link;

}

else

underflow(ps);

}

overflow(ps);

/* Prints a warning when

the stack overflows */

stackrecordpointer *ps;

{

printf (" The stack has overflowed /n");

}

underflow(ps);

/* Prints a warning when

the stack underflows */

stackrecordpointer *ps;

{

printf (" The stack has underflowed /n");

}

As promised, the new stack modules are not dependent on the details of the stack records. The program file, towerstest.c, does not need to be changed from its original version. To run it, stack_record.c and stack.c must be compiled. Their object files, along with the previously generated object file for towerstest.c, can be linked, and then towerstest.c can be run. Any future changes in stack_record.h mean that stack_record.c, stack.c, and towerstest.c must be recompiled, but changes to only stack_record.c or stack.c will require just their recompilation. Similarly, changes to just towerstest.c require only its recompilation.

Using this technique it is possible, as demonstrated, to structure programs so that modules for data abstractions can be built using modules for other data abstractions. The examples presented, for the sake of simplicity, have dealt with small to moderate-sized programs. With programs of this size the overall impact of file modularity and modules may not appear to be great. The real power of the technique becomes quickly apparent when large programs are written by teams of programmers. The efficiency and control gained through modularity in such cases is significant.

5.6: Summary

Data abstraction and functional and file modularization are important concepts whose proper application to program development and program structure enhance program readability and maintainability. They aid the programmer in establishing libraries of useful functions that can be readily selected for use in creating new programs for new problems.

The ability to compile parts of a program separately can make debugging an easier task and can save time. In C this can be achieved by placing appropriate parts of a program in distinct files. When this is done it is necessary to keep track of which files must be compiled and linked in order to execute the program file.

Exercises

1.     Why is it not necessary to place the include "collection.h" before the primes function in primestest.c?

2.     Why is primestest.c independent of the implementation of collection?

3.     Why must primestest.c and collection.c be recompiled when changes are made in collection.h?

4.     Change preferences.h and preferences.c so that the preferences and priorities arrays are implemented using a pointer array pointing to their rows, which are stored as row records in dynamic memory (see Section 2.3.3).

5.     Run stablemarriages.c with your solution to Exercise 4.

6.     What must be changed in list.h and list.c if you want to add another field (say, dividends) to the list records?

7.     Change the files of Section 5.4 so that the name field of the stockrecords, instead of being an array, is a list. Treat that list as a data abstraction.

8.     Write and compile a module for the queue data abstraction using the circular array implementation of Section 4.6.1.

9.     Write and compile a module for the queue data abstraction using the list implementation of Section 4.6.3.

10.     Create a program file ordlistexample.c. The program is to do what listexample.c does, except the list that is created is to have the records appear in alphabetical order by name. The list is to be treated as a data abstraction using list.h and list.c.

11.     Compile, link, and run your program created in Exercise l0.

12.     Define file list_record.h and list_record.c, and modify list.h and list.c to reflect these changes. The result should make list.h and list.c independent of details of the listrecords (as was done with the stack in Section 5.5). To accomplish this, you will have to remove and modify some operations from list.c, placing their modified versions in list_record.c, and some declarations from list.h, placing their modified versions in list_record.h.

13.     Create a module for a data abstraction list of stacks.

Suggested Assignments

1.     Replace collection.h and collection.c with new files that implement collection as an array that contains the actual integers currently in the collection as its entries.

2.     a. One of the two complete programs of Section 3.9.3 implements the configuration for the perfect shuffle as an array and the other implements it as a list. They do not treat the configuration as a data abstraction; write and run a single source file program that does.

b.     Rewrite and run the program developed for (a) using file modularization to hide the details of the configuration's implementation and so that separate compilation can be done.

3.     Change the stack implementation of Section 5.5 so that it is implemented as in Section 4.5.1. You can change only the files stack.h and stack.c; the program file, towerstest.c, cannot be changed.

4.     Modify the implementation of the stack data abstraction in Section 5.5 so that instead of the stack containing records, it contains pointers to the records. Again, this should be achieved with changes only to stack.h and stack.c.

Go to Chapter 6      Return to Table of Contents


Data Structures Using C: CHAPTER 6: MORE COMPLEX LISTS

 Return to Table of Contents

CHAPTER 6: MORE COMPLEX LISTS

Considers lists of lists connected by pointers to

impose more structure on data

reduce search time

Emphasizes

liste-structures, a special case of such lists

the traversal of list-structures as a general tool

the implementation of list-structures

Case study--information retrieval

illustrates the use and advantages of more complex lists

6.1: Imposing List Structure on Data

So far in this book, records in lists have had information members and link members. These are simple records. The lists themselves have looked like chains of records. Such lists may be used to store the records only or to store them and also impose an ordering on them. This chapter presents complex records, which contain members that are themselves names of lists. The members are called sublist members. Lists composed of such records have more complex structure than chains and are known as complex lists. Several such lists are depicted schematically in Figure 6.1. The following examples illustrate the use of complex lists and show how they provide one means of organizing data to capture certain inherent relations among the list records.

Example 6.1

The accounts list in Figure 6.1(a) consists of three records. The first two are complex, the third simple. The first record points to a sublist consisting of two records; the first of these is complex and the second is simple. Accounts might actually represent three companies' user accounts. The first user account is billed for two distinct computer services, $100 for last month's computing and $75 for this month's software consulting. The second user account owes $200 for the month's computing and $250 for the month's consulting. The third user account has a credit of $46. The X in the sublist member of the complex record "200" means its sublist is null.

Example 6.2

A programmer asked to simulate a card game with n players might choose to represent the current configuration of the n individual hands by using a list composed of complex records. A typical configuration is shown in Figure 6.1(b) for four poker players. The sublist that makes up each complex record contains all the cards of a particular player's hand, each card represented as a record. Insertions and deletions into an individual hand require insertions and deletions from the appropriate sublist.

(a) ACCOUNTS List

(b) Poker Hands for n = 4 Players

(c) BOOK List

Figure 6.1 More Complex Lists

(d) List P, a Polynomial Expression

(e) Brokerage Data Base

Example 6.3

A book might be represented as the book list shown in Figure 6.1(c). Records for the individual lines of text would be simple; the records for paragraph, chapter,and page would be complex.

The data processing for word processing or text editing might involve inserting and deleting chapters, paragraphs, or lines. Creating an index for the book would require even more complex processing. This could be done by making a special "dictionary" that lists the key words relevant to the book's topic, then searching the text for all instances of key words. If the book were a computer science text, then the dictionary would include all key technical computer science words. The index could be created by accessing and processing each page. This processing would involve checking the page for occurrences of the key words and keeping track, for each such word in the dictionary, of the pages on which it occurred. (Unfortunately, such automatic indexing creates too large an index and one that may not be very useful, since not every mention of a key term is especially relevant.)

Example 6.4

A polynomial in x, y, and z may be represented using a list-structure. Suppose the polynomial P(x, y, z) is 3zx5 + 5z3yx + 20. We can write this as

P(x, y, z) = z3(y(x(5))) + z(x5(3)) + 20

This, in turn, can be represented as p of Figure 6.1(d).

(a) General Form of a Record

(b) Main-List Record

(c) Last Record on Main List

Figure 6.2 Records of the List-structure for a Polynomial

Each record of p has four fields, as shown in Figure 6.2(a). A tag field value of 1 indicates that the record is complex, while a tag field of 0 indicates that the record is simple.

Each sublist, including the main list, has a header record and represents a polynomial. The main list, for example, is a polynomial in z composed of three terms. The record (Figure 6.2(b)) on the main list indicates that this term contains z to the power 3 and a coefficient that is pointed to by the pointer in the coeff field. Its successor record is similarly expressed. The last record on the main list (Figure 6.2(c)) is simple, meaning that the value in the coeff field is to be interpreted as a constant. Sublists are interpreted in the same way.

Example 6.5

Each record on brokerage's main list in Figure 6.1(e) represents a sales representative (a financial advisor) of a brokerage firm. The sublist of each salesperson contains information about each of his or her clients. Each client's sublist is the client's stock portfolio.

6.1.1 A Definition of List-structures

Each of the example lists organizes information by collecting related data on sublists. They are a step up in complexity from the chains of Chapter 3 but do not illustrate the most complex lists, which allow sharing of records and sublists. Sharing means that more than one record can point to another record or sublist. Records can also have additional pointer fields linking records to form intertwined lists. However, this chapter concentrates primarily on list-structures. They may be defined formally by either of two equivalent definitions:

A list-structure is a collection of simple and complex records linked by pointers in the following way.

The link and sublist fields of each record contain pointers to records of the collection.

Each record of the collection has no more than one pointer to it from any record of the collection.

Or recursively:

A list-structure is a list of simple and complex records whose complex record's sublist fields point to distinct list-structures called sublists. The predecessor of each record must be unique.

The first definition treats list-structures as a collection of linked records but does not convey their inherent structure. The recursive definition emphasizes this structure, making clear how a list-structure is itself built from other list-structures.

It is evident that each list of Figure 6.1 satisfies the first definition. To see that they also satisfy the recursive definition, consider the accounts list. The main list consists of a list of simple and complex records, so it is necessary to check that the complex record's sublists are themselves list-structures. The first main-list record's sublist is a list-structure since the sublist's main list satisfies the definition and its complex record points to a list-structure, a single list record. The second main-list record's sublist is itself a list whose complex record's sublist is a list-structure, the null list-structure.

(a) List with Shared Sublists

(b) List with a Loop

Figure 6.3 Two Lists That Are Not List-structures

The lists sam and sally in Figure 6.3 are not list-structures, since they fail to satisfy the definitions. Sam has shared sublists; sally has a loop.

6.2: Traversal of List-structures

Inserting or deleting a record from a list-structure is done using the same procedures as in Chapter 3. Determining where to insert or delete involves a traversal through the records to locate the desired position. Unlike the chain, for which there is a natural order in which to traverse the records, list-structures have no simple natural order. Instead, the order in which records are to be accessed and processed is defined as follows:

To traverse a list-structure,

Access each record on the main list in order and

1. Process the record

2. If the record is complex, then traverse the complex record's sublist.

Notice that the definition is actually recursive, since a complex record's sublist is itself a list-structure. To see how to apply it, consider formula in Figure 6.4. The records are numbered for ease of reference.

To determine the order in which records are accessed and processed in a traversal of the list-structure formula, apply the definition. It specifies that records 1 and 2 on the main list be accessed and processed in sequence. Since record 2 is complex, sublist 1 must be traversed. At the finish of traversal of sublist 1, it will be necessary to continue where processing left off on the main list. To know where to resume, save P1. To traverse sublist 1, apply the definition to it, and access and process records 3 and 4 in order. Since record 4 is complex, traverse sublist 2. Save P3 to know where to resume when its traversal is complete.

Figure 6.4 The List-structure FORMULA

To traverse sublist 2, apply the definition of traversal to it, and so access and process records 5 and 6. Since the link field of record 6 is null, no records remain on this sublist, so its traversal is complete.

Which task should be done now? Step 2 of the traversal definition has just been applied to sublist 2. The next record of some sublist must now be accessed. What record? It is the successor record of the complex record--record 4--whose sublist traversal was just completed. P3, which was saved for just this moment, points to record 7, so let the program follow it. Since P3 is no longer needed, discard it. Since record 7 is complex, traverse sublist 3. Continuing in this way, the program would access and process the records in the order in which they are numbered.

From this example it can be seen that whenever the link field of the record just accessed and processed is null, the traversal of some sublist has been completed and the program must determine where to go next. Each time the processing of a complex record was finished, the content of its link field was saved before traversing its sublist. In formula, this leads to saving P1, then P3. After record 6 comes a null link field signifying the completion of the traversal of sublist 2. P3 then tells the program where to go next. Once followed, it can be forgotten. Pl is still remembered, and P5, P7, and P9 need to be remembered in turn. After dealing with record 14, the program encounters its null link value, which signifies completion of sublist 4's traversal. Where does it go? It follows P9 and needs to remember only P7, P5, and P1, and so on.

Each pointer that is saved represents an obligation that must be postponed. It is the obligation to apply steps 1 and 2 of the definition of traversal of list-structures to successive records on the list pointed to by the pointer. The definition requires that the corresponding complex record's sublist be traversed immediately. In carrying out the traversal of that sublist, further obligations may be incurred and subsequently fulfilled. Each time step 2 is completed, the most recently saved pointer provides the information to determine what record to deal with next. When the record is selected for processing, the pointer is discarded. This implies that pointers saved during the complex records' traversal will be discarded when its traversal is complete. Consequently, the most recently saved pointer must point to the record to be processed each time step 2 is completed. If there is no remaining saved pointer, it means that the entire traversal is complete.

Traversing a list-structure thus requires retention, recall, and deletion of information in a specific order. The pointer information retained represents postponed obligations. These obligations must be fulfilled in reverse order from that in which they were incurred. This is the precise situation in which the stack data abstraction is very convenient. Notice how a stack can be used to store the pointers so that the list-structure can be traversed correctly. Each pointer saved is pushed on the stack and popped when needed. It is a typical LIFO situation.

It is important to recognize that the definition of a list-structure traversal does not merely explain how to traverse a specific list-structure, such as formula, but describes a procedure that may be applied to any list-structure. This is, of course, true for any algorithm or program. Programs and algorithms specify a procedure for any input, not just for a specific input. The definition of traversal of list-structures is really an algorithm, an initial description of a procedure for list-structure traversal.

6.2.1 An Iterative Function

We will now develop a program traverse to traverse a list-structure ls. In applying the definition it became clear that such a traversal may be described by the following algorithm.

1. Initialize the current record.

2. Initialize the stack of postponed obligations to "empty."

3. While there is a current record or a postponed obligation,

if there is a current record, then

a. process the current record, and

b. update the current record to the next record to be processed

else

c. remove the top obligation from the stack and update the current record to it.

To update the current record, the loop task 3b needs to be specified in greater detail. If the record just processed was simple, then the next record to be processed is the successor of the current record. Otherwise--if the record just processed was complex--its successor record represents a postponed obligation that must be placed on the stack. In that case the next record to be processed is the first record on the current record's sublist. This gives a refinement for task 3b that is incorporated in the revised algorithm.

1. Initialize the current record.

2. Initialize the stack of postponed obligations to "empty."

3. While there is a current record or a postponed obligation,

if there is a current record, then

a. process the current record, and

b. if the current record is complex, then

i. save an obligation to its successor record on the stack

ii. update the current record to the first record on its sublist,

else

update the current record to its successor record

else

c. remove the top obligation from the stack and update the current record to it.

A variable (call it ptr) is needed to contain a pointer to the current record. The stack can be treated as a data abstraction with the usual operations set-stack, empty, push, and pop as was done in Chapters 4 and 5. The details for implementing the stack are built into the routines implementing these operations.

Assume that during the traverse each record is processed by calling a routine process. It will clearly need the pointer to the record to be processed, ptr. It will also need access to the list name ls . For example, if its task involves deleting the first record of ls, then it must be able to change the contents of ls. To treat the list-structure as a data abstraction, we need the basic functions complex, next, and sublist, each with ptr as a parameter. Complex returns a value true if the record to which ptr points is complex, the value false otherwise. Next and sublist return, respectively, a copy of the link and sublist field value of the record pointed to by ptr.

A more detailed refinement of the algorithm can now be written.

1. Set ptr to ls

2. Setstack(s)

3. While ptr is not null or not empty(s)

if ptr is not null, then

a. Process(ls,ptr)

b. If complex(ptr), then

i. Push(next(ptr),s)

ii. Set ptr to sublist(ptr)

else

Set ptr to next(ptr)

else

c. Pop(s,ptr)

Notice that if all records of ls are simple, then tasks (i), (ii), and (c) will never be invoked. This procedure is then essentially the same as the list traversal of Chapter 3, as it should be. As written, it does not provide for partial traversal (done is not involved in the loop control) but could be easily modified when necessary.

Traverse may be implemented as in the following iterative (nonrecursive) function. This implementation uses data abstraction and has been written using functional modularly.

Nonrecursive Version

traverse (pls)

/* Accesses and processes each record

of the list-structure ls.

*/

liststructurepointer *pls;

{

liststructurepointer ptr,null,setnull(),next(),sublist();

>Comment

stack s;

null = setnull();

>Comment

ptr = *pls;

    >Comment

setstack(&s);

    >Comment

while((ptr ! = null) | | (!empty(&s)))

    >Comment

if(ptr ! = null)

{

    >Comment

process (pls, ptr);

    >Comment

if (complex(ptr))

{

    >Comment

push next(ptr),&s);

ptr = sublist(ptr);

}

else

    >Comment

ptr = next(ptr);

}

else

    >Comment

pop(&s,&ptr);

}

It is important to see that the program is functionally modular and independent of the implementation of the data abstractions to which it refers. Data abstractions for the stack with its operations, and for the list-structure and its operations (which must include complex, next, sublist, and setnull) are needed in order to use traverse. The programmer must either be given these or else must write them.

6.2.2 A Recursive Function

A recursive implementation can also be developed for traverse. Begin with a recursive algorithm for traversing a list-structure; this is obtained directly from the recursive definition of a list-structure traversal.

1.     Initialize the current record.

2.     While there is a current record,

a.     process the current record;

b.     if the current record is complex, then

i.     traverse the current record's sublist;

c.     update the current record to its successor record.

Once more we need the variable ptr to point to the current record, and the variable nextsublist to point to a sublist. Again, the list-structure is treated as a data abstraction. Also, to modularize functionally, assume that a function traverse is available to implement task (i). The algorithm can now be refined:

1.     Set ptr to ls

2.     While ptr is not null,

a.     Process z(ls, ptr);

b.     if complex (ptr), then

i.     Set nextsublist to sublist (ptr),

traverse (nextsublist);

c.     Set ptr to next (ptr).

This refinement is implemented in the following recursive procedure.

Recursive Version

traverse (pls)

/* Accesses and processes each record

of the list-structure ls.

*/

liststructurepointer *pls;

{

liststructurepointer ptr,nextsublist,null,

setnull(),next (),sublist();

null = setnull();

    >Comment

ptr = *pls;

    >Comment

while(ptr != null)

{

    >Comment

process(pls,ptr);

    >Comment

if(complex(ptr))

{

    >Comment

nextsublist = sublist(ptr);

    >Comment

traverse(&nextsublist);

}

    >Comment

ptr = next(ptr);

}

}

Notice that this program, at the point when it encounters a complex record and hence must traverse its sublist, does so by simply invoking a module that does that traversal--itself. After that traversal is complete, it merely resumes where it left off. In contrast, the nonrecursive version must perform its own bookkeeping to keep track of where to resume each time it encounters a complex record and then retrieve that information when it is needed.

A number of comments are in order about the two implementations of traverse. First, imagine that the recursive procedure traverse is presented to you with the statement that it is a refined version of the definition of traversal of a list-structure. It is easy to see that this is so. Contrast this with the effort involved in seeing that the nonrecursive version of traverse is also a refined version of the same defining algorithm. Second, the recursive version is more concise than the nonrecursive. Third, the recursive traverse is almost a direct translation of the defining algorithm. It required very little effort to achieve it in this case, because the definition is itself a recursive description.

These comments usually hold for algorithms that process data structures that can be defined recursively, such as list-structures. This is not an accident but can be expected when we deal with complex data structures. Recursive definitions allow concise descriptions of complex algorithms and data structures. Concise descriptions allow us to deal more easily with their complexity. ln fact, it is difficult to develop a definition for a list-structure traversal that has no vagueness, is applicable to any list-structure, and does not involve recursion. Recursive definitions are especially useful to the programmer dealing with complex data structures, because they lead systematically and directly to recursive programs for their processing.

6.3: Using a Traverse Function for List-structures

Use of the general list-structure traversal functions just developed is illustrated in the following examples. The examples are intended to convey the flavor of list-structure applications and their processing.

Example 6.6

Print the information field of each stock record of the list-structure brokerage (shown in Figure 6.1(e)) which was bought prior to a specified date and which we store in date. A solution may be obtained by traversing brokerage and printing the relevant stock records. To do this requires only that we write the process function invoked by traverse. Either the recursive or the nonrecursive version may be used. The function process must be written to turn traverse into a solution to this problem.

The task is straightforward except for the determination, within process, of whether the record it is to process is currently a stock record. Since only stock records are simple, a test for complexity of the current record will provide this nformation. Process may be written as where prior returns true if datebought of the record pointed to by ptr is prior to date, and false otherwise, and printinfo prints the information field value of the record pointed to by ptr.

process (pls, ptr)

/* Print the stock record of ls

pointed to by ptr if it was

bought prior to date

*/

liststructurepointer *pls, ptr;

{

    >Comment

if(!complex(ptr) && prior(ptr,&date))

printinfo (ptr);

}

Example 6.7

Consider again the list-structure brokerage in Figure 6.1(e). The task now is to print, for each client, the total value of his or her portfolio.

In detail, what has to be done is

1.     To traverse brokerage, and each time a client's record is encountered

2.     Print the name

3.     Then accumulate the total value of the portfolio as it is being traversed

4.     Print this total when the portfolio has been completely traversed

While this seems straightforward and simple enough, some details require attention. We can apply the traverse function to this task. To do so, process must be written to turn traverse into a solution. Thus process must be able to recognize when the record it is to work on is a client's record. At this point, it must initialize the accumulated total to zero. For each ensuing record of the client's portfolio, it must then update the accumulated total by the value of that record's stock. Finally, process must recognize the end of the portfolio list and print the accumulated total.

If total is used to store the accumulated value, then it must be preserved between calls to process. This can be achieved by making total a static variable in process or by making total global. We choose the former. Process may be written as follows.

process(ptr)

/* If the current record points to a

client's portfolio record that client's

total portfolio value is updated.

If the record is the last record on

the client's portfolio list its total

value is printed.

*/

liststructurepointer null,setnull(),

sublist(),next();

float value();

    >Comment

static float total;

null = setnull();

    >Comment

if(complex(ptr))

if(sublist(ptr) != null)

if(!complex(sublist(ptr)))

{

    >Comment

total = 0.0;

    >Comment

printname(ptr);

}

    >Comment

else

{

    >Comment

total = total + value(ptr);

    >Comment

if(next(ptr) == null)

printf("\n Portfolio value is

%12.2f \n",total);

}

}

The if conditions are used to determine if this is the first call to process for a portfolio and, if so, to initialize and print the client's name or, after updating a client's total, to see if the record is the last of a portfolio. Assume null is nonlocal to process. Process will work with either version of traverse.

Example 6.8

Suppose book, the list-structure shown in Figure 6.1(c), stores the current version of a text that you wish to modify by replacing the current paragraph p of page pg of chapter c with a new paragraph. The new paragraph is stored as a list newp. To solve this problem we will write a new nonrecursive traverse function, called paragraphinsert, tailored to the situation. Since not all records of book need to be accessed, we do not want to traverse the entire list-structure as we have done in the previous two examples. Two versions of the function are presented. In both versions the idea is to

1.     Traverse the main list searching for chapter c

2.     Traverse that chapter's sublist searching for page pg

3.     Traverse that page's sublist searching for paragraph p

4.     Finally, replace that paragraph with newp

The first version assumes that records representing chapters, pages, or paragraphs have a number field. In the case of a chapter record the field contains the number of that chapter; for a page record it contains the number of that page; and for a paragraph it contains the number of that paragraph. These field values are shown in Figure 6.1(c). The program keeps track of whether it is currently looking for a chapter, a page, or a paragraph by using a variable currentlist. The value stored in currentlist will be CH (denoting that the search is now for a chapter), PAGE (denoting that the search is for a page), or PAR (denoting the search is for a paragraph) . Similarly, a variable currentnumber will keep track of the particular chapter, page, or paragraph number currently being sought by the search. A variable ptr keeps track of the record currently being processed. Once the search for a chapter or a page is complete, currentlist, currentnumber, and ptr must be updated. Once the search for a paragraph has completed, the searching terminates and the new paragraph replaces the old.

Version 1

    >Comment

#define CH 1

    >Comment

#define PAGE 2

    >Comment

#define PAR 3

#define TRUE 1

#define FALSE 0

paragraphinsert(book,newp,c,pg,p)

/* Replaces paragraph p of page pg

of chapter c of book by the

paragraph pointed to by newp.

Assumes the records of book contain

number fields.

*/

liststructurepointer book,newp;

int c,pg,p;

{

int currentnumber,currentlist,found;

liststructurepointer null,ptr,setnull(),next(),sublist();

null = setnull();

    >Comment

ptr = book;

    >Comment

currentnumber = c;

    >Comment

currentlist = CH;

found = FALSE;

    >Comment

while(!found && (ptr != null))

{

    >Comment

if(currentnumber != number(ptr))

ptr = next(ptr);

    >Comment

else

    >Comment

switch(currentlist)

{

    >Comment

case CH:

    >Comment

currentlist = PAGE;

currentnumber = pg;

ptr = sublist(ptr);

break;

    >Comment

case PAGE:

    >Comment

currentlist = PAR;

currentnumber = p;

ptr = sublist(ptr);

break;

    >Comment

case PAR:

found = TRUE;

}

}

    >Comment

if(found)

    >Comment

setsublist(ptr,newp);

    >Comment

else

printf("\n Paragraph not found \n");

1 print message

}

Setsublist sets the sublist field of the record pointed to by its first parameter to the value of its second parameter. Just before setsublist is invoked, code (not shown here) could be inserted to reclaim the storage used by the replaced paragraph. This is also true for the second version.

The second version assumes no number field, so the records are indistinguishable. Hence they must be kept in order--that is, chapters are in order on the main list, pages are in order on their lists, etc. Note that this ordering is not required by the first version. Now a count of list records must be maintained, to determine when the appropriate record number has been encountered. In the second version below only the differences between it and the first version have been annotated. Notice that this second program requires less storage for the list-structure (as the number fields are not needed), but it does somewhat more processing. Also, the sublists must be kept in order. This is another example of trading storage for time.

Version 2

#define CH 1

#define PAGE 2

#define PAR 3

#define TRUE 1

#define FALSE 0

paragraphinsert(book,newp,c,pg,p)

/* Replaces paragraph p of page pg

of chapter c of book by the

paragraph pointed to by newp.

Assumes the sublists of book are

in order.

*/

liststructurepointer book,newp;

int c,pg,p;

{

int currentnumber,count,currentlist,found;

liststructurepointer null,ptr,setnull(),next(),sublist();

null = setnull();

ptr = book;

currentnumber = c;

currentlist = CH;

found = FALSE;

    >Comment

count = 1;

    >Comment

while(!found && (ptr != null))

{

    >Comment

if(currentnumber != count)

{

    >Comment

count++;

ptr = next(ptr);

}

else

switch(currentlist)

{

case CH:

currentlist = PAGE;

currentnumber = pg;

    >Comment

count = 1;

ptr = sublist(ptr);

break;

case PAGE:

currentlist = PAR:

currentnumber = p;

    >Comment

count = 1;

ptr = sublist(ptr);

break;

case PAR:

found = TRUE;

}

}

if(found)

setsublist(ptr,newp);

else

printf("\n Paragraph not found \n");

}

Example 6.9

Consider again the list-structure formula shown in Figure 6.4. It may be interpreted as representing the formula   [image]. This happens to be the root of the quadratic equation ax2 + bx + c = 0. In formula the symbols QT, PS, TS, MS, DIFF, and SQ stand for the operations of division, addition, multiplication, changing sign, taking the difference, and taking the square root, respectively. The square root and sign change operations apply to one operand. The others apply to two operands. The main list (and each sublist) represents an arithmetic expression. Their first records contain an arithmetic operator such as QT, PS, TS, MS, and DIFF. Their remaining list records represent the operand values of that arithmetic operator. The expression represented by such a list is to be evaluated by applying the operation to its operands. Thus sublist 4 would be evaluated by multiplying the operands in records 13 and 14 (b and b respectively), together.

Notice that each complex record's sublist is an arithmetic expression that must be evaluated. Thus the operand each represents must be obtained before the operator represented by the list on which it appears may be evaluated. For example, the sublists of complex records 2 and 22 must be evaluated before QT of record 1 can be applied to them.

The task is to write a function evaluate to return the value of an arithmetic expression represented by a list-structure. As with formula, the evaluation can be accomplished by traversing the expression's main list and, whenever a complex record is encountered, evaluating its sublist. When the last list record has been processed in this way, the operator and each of its operands are known. The final result is obtained by applying the operator to its known operands. Of course, in evaluating an operand, more complex records may be encountered, such as record 4, and they must then be evaluated before continuing. A process function can be written to turn either the recursive or the nonrecursive traversal program into evaluate.

One important detail remains: how to store the operands that are generated during the traversal so that they are available when needed. Suppose each operand encountered by process is stored in an operand stack, and each operator encountered is stored in an operator stack. If the current record being processed is the last record of a list (its link value is null), then the operator currently at the top of the stack will have its operands at the top of the operand stack. Process can then pop both these stacks, apply the operator to the operands, and place the result back onto the operand stack. If the record being processed represents the last record on the main list (the operator stack will then be empty), instead of placing the result on the operand stack it should copy it into evaluate.

A detailed description of process follows.

If next(ptr) is not null and the record pointed to by ptr is simple, then

if it contains an operator, then

place the operator on the operator stack

else

place the operand on the operand stack

else

pop the operator stack, remove the top k operands from the operand stack (k is

the number of operands of the operator just popped), and apply the operator to

the k operands.

If the operator stack is not empty then

push the result onto the operand stack

else

set evaluate to the result.

This solution may also be applied to the evaluation of logical or boolean expressions. In fact, it forms the basis of the interpretive language LISP.

We have now seen a number of applications of traverse to examples involving list-structures. The example, presented next, is not discussed in detail but points out another possible application.

Example 6.10

Suppose polynomials are represented as p is in Figure 6.1(d). It is then possible to write functions to perform important algebraic operations on polynomials such as the addition and multiplication of two polynomials. Even symbolic differentiation can be performed. Note that traversal of the list-structures representing the polynomials would form the heart of such functions.

6.4: Implementing List-structures

The graphic depictions of list-structures mirror the conceptual image of them. To implement list-structures in a language such as C, you may store their records in dynamic memory or in an array of records. Wherever the records are stored, it must be possible to distinguish simple records from complex ones. The main difference in the format of the records, compared to those of a chain, is that an additional field becomes necessary for the sublist pointer when the record is complex. We also assume that the record contains a field whose value will establish whether the record is simple or complex. The actual implementation of a list-structure record may be done in three ways:

1.     Declare two types of records:

one to be used for simple records

another to be used for complex records

2.     Declare one record type that consists of a fixed part and a union, which has

one format for simple records

another format for complex records

3.     Declare one record type, used for both simple and complex records, but containing a kind or tag field to indicate whether the record stored is simple or complex

The first way is undesirable because it requires that a program deal with two distinct pointer types, one for simple records and another for complex records, so we dismiss it. The second method is feasible and also requires a kind field, which we can put in the fixed part. However, even though the two distinct union formats may have significant differences in the amount of storage they require, C will always allocate the larger amount. Thus this method does not save storage over the third way, which we now choose to use.

To implement the accounts list-structure of Figure 6.1(a), the following definitions could be used when the records are to be stored in dynamic memory.

    >Comment

typedef struct record

{

int kind;

int info;

struct record *sublist;

struct record *link;

}listrecord, *liststructurepointer;

When the records are to be stored in an array of records, the implementation should be changed to

    >Comment

typedef struct record

{

int kind;

int info;

    >Comment

int sublist;

    >Comment

int link;

}listrecord;

    >Comment

typedef int liststructurepointer;

This change is needed because now listpointer, link, and sublistptr will each be an integer index into the array of records rather than a pointer variable containing an absolute memory address.

As a second illustration, consider the records for the polynomials of Figure 6.1(e). Notice that a complex record in that list-structure has two possible formats, depending on whether the record is a header or not. If it is a header record, the second field contains a character; otherwise it contains a sublist pointer. Using a union, the polynomial records can be implemented as follows.

    >Comment

typedef union

{

char unknown;

polyrecord *sublist;

int value;

}varying;

typedef struct record

{

int tag;

varying secondfield;

int exponent;

struct record *nextterm;

}polyrecord, *liststructurepointer;

The tag field indicates whether the record is simple or complex. In this implementation the union allows the same storage to be used to hold the unknown (for a header record) or a sublist pointer. It was used here because a complex record has two distinct possible second fields and the union allows them to share the same storage. The alternative is to avoid the use of the union and keep separate fields for both the unknown and the sublist pointer in each record. This just wastes storage.

When the records are to be stored in an array of records, the implementation should be changed to

    >Comment

typedef union

{

char unknown;

    >Comment

int sublist;

int value;

}varying;

typedef struct

{

int tag;

varying secondfield;

int exponent;

    >Comment

int nextterm;

}polyrecord;

    >Comment

typedef liststructurepointer;

It is also possible to implement the main list and sublists of a list-structure as two-way lists. This is convenient, just as for chains, when it is necessary to go back and forth during the processing of a list-structure or to make the predecessor of any record easily accessible. In this case an additional field would be needed for the backward pointer.

The following program creates and prints the records of a list-structure to illustrate its implementation with records stored in dynamic memory. It also demonstrates two applications of the traverse function. The input consists, first, of an integer to indicate whether the list being created is null, and, second, a sequence of data for each record. This sequence is assumed to be in the order in which the records are accessed in a traversal of the liststructure. For example, input corresponding to accounts of Figure 6.1(a), would be as follows. (Program output is not shown, nor are program prompts for input.)

1    --  indicates a nonnull list-structure

175  --  information field value

1    --  indicates record is complex

11   --  indicates the link and sublist fields are not null

0    --  information field value

1    --  indicates record is complex

11   --  indicates the link and sublist fields are not null

100  --  information field value

0    --  indicates record is simple

0    --  indicates the link field is null

75   --  information field value

0    --  indicates record is simple

0    --  indicates the link field is null

450  --  information field value

1    --  indicates record is complex

11   --  indicates the link and sublist fields are not null

200  --  information field value

1    --  indicates record is complex

10   --  indicates the link field is not null but the sublist field is

250  --  information field value

0    --  indicates record is simple

0    --  indicates the link field is null

-46  --  information field value

0    --  indicates record is simple

0    --  indicates the link field is null

The Program

Reads in list-structure records,

Creates the list-structure, and

Prints all its records.

#include <stdio.h>

    >Comment

typedef struct record

{

int info;

int kind;

struct record *sublistptr;

struct record *link;

}examplerecord,*liststructurepointer;

    >Comment

#define NULL 0

    >Comment

liststructurepointer next(p)

/* Returns a copy of the link field

value of record pointed to by p.

*/

liststructurepointer p;

{

return(p->link);

}

liststructurepointer sublist(p)

/* Returns a copy of the sublistpointer

field value of record pointed to by p.

*/

liststructurepointer p;

{

return(p->sublistptr);

}

liststructurepointer setnull()

/* Returns a null pointer */

{

return(NULL);

}

liststructurepointer avail()

/* Returns a pointer to storage allocated

for a liststructure record of type

examplerecord.

*/

{

return(malloc(sizeof(examplerecord)));

}

complex(p)

/* Returns true only if the record pointed

to by p is complex.

*/

liststructurepointer p;

{

    >Comment

return(p->kind == 1);

}

createrecord(l,ptr)

/* Reads in the data for the current record

and sets its fields appropriately.

*/

liststructurepointer 1,ptr;

{

int link,subptr,value,kind;

liststructurepointer setnull(),avail();

printf("\n enter info \n");

    >Comment

scanf("%d",&value);

printf("\n enter 0 for simple 1

for complex \n");

    >Comment

scanf("%d",&kind);

    >Comment

if (kind == 0)

{

printf("\n enter link \n");

    >Comment

scanf("%d",&link);

    >Comment

if (link == 0)

    >Comment

ptr->link = setnull();

    >Comment

else

    >Comment

ptr->link = avail();

}

    >Comment

else

{

printf("\n enter link & subptr \n");

    >Comment

scanf("%d %d",&link,&subptr);

    >Comment

if (link == 0)

    >Comment

ptr->link = setnull();

    >Comment

else

    >Comment

ptr->link = avail();

    >Comment

if (subptr == 0)

    >Comment

ptr->sublistptr = setnull();

    >Comment

else

    >Comment

ptr->sublistptr = avail();

}

    >Comment

ptr->info = value;

    >Comment

ptr->kind = kind:

}

printrecord(pls,ptr)

/* Prints the contents of the info field

of the record pointed to by ptr.

*/

liststructurepointer *pls,ptr;

{

printf("\n %d \n",ptr->info);

}

main()

/* Reads in liststructure records, creates

the liststructure, and prints all its records.

*/

{

int n;

    >Comment

liststructurepointer l,avail();

printf("\n enter liststructure: 0 for\

null list 1 otherwise \n");

scanf("%d",&n);

printf("\n remember - the data for the\

records must be in traversal order\

\n");

    >Comment

if (n == 0)

    >Comment

l = NULL;

    >Comment

else

{

    >Comment

l = avail()

    >Comment

create(1);

}

    >Comment

traverse(&l);

    >Comment

if (l == NULL)

printf("\n The list is null \n");

}

    >Comment

#define LIMIT 50

typedef liststructurepointer whatever;

typedef struct

{

whatever stackarray[LIMIT];

int top;

}stack;

    >Comment

setstack(ps)

/* Sets stack s to empty */

stack *ps;

{

(*ps).top = -1;

}

empty(ps)

/* Returns true only if

stack s is empty.

*/

stack *ps;

{

return((*ps).top == -1);

}

push(value,ps)

/* Inserts contents of value as

top entry of stack s.

*/

whatever value;

stack *ps;

{

if ((*ps).top == (LIMIT - 1))

overflow(ps);

else

{

(*ps).top = (*ps).top + 1;

(*ps).stackarray[(*ps).top] = value;

}

}

pop(ps,pvalue)

/* Removes the top entry of stack s

and places its contents in value.

*/

stack *ps;

whatever *pvalue;

{

if (empty(ps))

underflow(ps);

else

{

*pvalue = (*ps).stackarray[(*ps).top];

(*ps).top = (*ps).top - 1;

}

}

overflow(ps)

/* Prints a message when

the stack overflows.

*/

stack *ps;

{

printf("\n stack overflow \n");

}

underflow(ps)

/* Prints a message when

the stack underflows.

*/

stack *ps;

{

printf("\n stack underflow \n");

}

    >Comment

traverse(pls)

/* Prints the info fields of all

records of the liststructure l.

*/

liststructurepointer *pls;

{

liststructurepointer ptr,null,setnull(),

next(),sublist();

stack s1;

null = setnull();

ptr = *pls;

setstack(&s1);

while ((ptr != null)||(!empty(&s1)))

if (ptr != null)

{

    >Comment

printrecord(pls,ptr);

if (complex(ptr))

{

push(next(ptr),&s1);

ptr = sublist(ptr);

}

else

ptr = next(ptr);

}

else

pop(&s1,&ptr);

}

    >Comment

create(l)

/* Inputs records and creates the list l */

liststructurepointer l;

{

liststructurepointer ptr,null,setnull(),

next(),sublist();

stack s2;

null = setnull();

ptr = l;

setstack(&s2);

while ((ptr != null)||(!empty(&s2)))

if (ptr != null)

{

    >Comment

createrecord(l,ptr);

if (complex(ptr))

{

push(next(ptr),&s2);

ptr = sublist(ptr);

}

else

ptr = next(ptr);

}

else

pop(&s2,&ptr);

}

Although the stacks have been given distinct names, since s1 is local to create and s2 is local to traverse, they could have both been named s. Note that either create or traverse could be replaced by recursive versions.

6.4.1 Sharing Storage

List-structures that share list records as well as sharing sublists are frequently needed to fit the demands of a problem. These structures are more complicated than those presented thus far and may require large amounts of storage for their implementation. A natural idea is to avoid the duplication of identical sublists or records. At times it is convenient to use additional fields linking records or sublists so that the resultant structure contains records with more than one pointer to them, and the structure may even contain loops. That is, by following pointers from a record we can arrive back at the record. You will see examples of this in the case study that follows. Whenever these possibilities occur, the structures created are still lists but are no longer list-structures. These more general lists provide extensive flexibility but lead to difficult storage management problems, which are considered later in the text.

6.5: Case Study: Information Retrieval

We now present a case study to illustrate the versatility of lists intertwined with pointers. When the name of a record or a pointer to it is known, retrieval of the record is simple. It may be directly accessed and processed to suit our needs. Slightly more difficult is the retrieval of a record that can be recognized when encountered but whose location is unknown. In order to retrieve it the records must be traversed until the desired one is encountered.

When large collections of records are involved, traversing all the records to find one is not feasible. The problem is not so much that retrieving one desired record takes too long but that many retrievals in rapid succession cause too much delay for the later retrievals. Imagine 1,000 computer terminals linked to a central computer. The banking terminals known as automatic tellers are an example. Suppose 200 requests occur in rapid succession for the retrieval of records. If 1 million records need to be traversed, the retrieval of an individual record could easily take a second, so the last of the 200 requests to be processed would be honored only after more than 3 minutes of elapsed time. No one wants to wait at a terminal for 3 minutes before each request is answered. In the meantime, even more requests become backlogged.

The basic technique to avoid traversing all stored records is to focus the search on a smaller group of records known to contain the desired one. This is the essence of most faster retrieval algorithms. In this case study, lists will be used to organize the data so that the search may be localized in this way. In Chapters 8 and 9 the search problem is presented in a more general context for information stored in random access internal memory, and in Chapter 10 for information stored in external memory.

6.5.1 Family Relationships

The central problem in this case study concerns information about a collection of individuals. The collection might represent everyone who graduated from a specific university, all registered voters in a given region, or all individuals known to have contracted a particular disease that is being researched.

The input information for each individual is given as a record with the individual's name, sex, birthdate, father's name, and mother's name as fields. Other relevant fields may be present but are ignored for our purposes here.

To simplify, assume that no duplicate records appear, no two individuals have exactly the same name, all siblings have the same parents, all parents are married, there are no divorces, and no one remarries. This is truly a noteworthy collection of people living in an idyllic world! The violation of these assumptions would create problems that could not be ignored. However, their consideration would detract from our immediate aims.

The task is to input a sequence of these records and represent the collection in memory so that questions involving certain relationships among individuals may be answered. Examples of possible questions are

How many children does John Smith have?

Who are the uncles of John Smith?

Who are the siblings of John Smith?

6.5.2 A First Solution

Perhaps the most straightforward approach to a solution for this information retrieval problem is to represent the collection of information by its image in memory as an array of records. To answer the question, "How many children does John Smith have?" requires a simple traversal of the records. Have a program look at the father's name field in each record and, whenever it comes to John Smith, increment a counter that represents the number of John Smith's children. When the traversal is complete, the program would print the counter value. To answer the question, "Who are the uncles of John Smith?", the program must first traverse, looking at the name field of each record, to find John Smith. It would note the father's name, which is in the fathersname field of that record. Then it would traverse again, looking at the fathersname fields. Each time John Smith's father's name appears, and "male" is listed as the corresponding sex field value, the program must print the name that appears in the corresponding name field. The same sequence would be repeated for John Smith's mother.

The execution time for these traversals can be proportional to the total number of records. This makes the solution time proportional to the amount of information represented in the system. For large systems, this would be impractical, as noted earlier.

What is the least amount of time that could be taken by any solution for these kinds of questions? Clearly, identifying the brothers of John Smith requires printing the names of the brothers. To display who the uncles of John Smith are, the program must print the names of each of his uncles. The least amount of time required for any solution would thus be proportional to the length of the answer to any given question. Length in the case of the uncles is the number of uncles. In the next section, you will see that a different way of representing the information allows a solution that is closer in execution time to this length. This is much better than having it proportional to the entire amount of information stored.

6.5.3 A Better Solution

To find answers to relationship questions quickly, you must be able to localize the search and get to the relevant records without considering all of them. You can, if you somehow link records together in a meaningful way. The problem deals with family relationships. Suppose you set up a record in memory to correspond to each individual name in the system, each record having seven fields, five of the fields containing pointer information. The seven fields of each individual's record will be nameptr, fnameptr, mnameptr, siblingptr, childrenlist, sex, and birthdate.

You will also have a table of names, called nametable. Each record of nametable will have a name and a recordptr field. The recordptr field contains a pointer to the individual record of the person whose name appears in the name field. By searching the nametable for a name, we get the pointer to the corresponding individual's record. Following the trails (pointers) from the record, we can retrieve the data needed to answer questions.

Figure 6.5 depicts the form of the two kinds of records. In memory the collection of nametable records will be stored separately from the collection of individual records.

The memory would appear as shown in Figure 6.6, after the data has been input. The childrenlist field of a record f contains a pointer to the first record on the list of children of the individual represented by f. The siblingptr field plays the role of a link field for records on the list of siblings of f. The different lists are intertwined as shown in Figure 6.7.

In Figure 6.7 records f and m are husband and wife, so their children-list pointers are identical. It might be convenient to keep the lists of children in order by birthdate or to make them circular, so that the null pointer signifying the last record would instead point to the first record of the list.

(a) Nametable Record

(b) Individual's Record

Figure 6.5 Format of the Nametable and Individual Records

If the assumptions about an "ideal" world were violated, it would be possible to have shared records, and loops could even appear. With the assumptions, the lists are list-structures. The main list of any sublist consists of siblings. All records are complex, with sublist fields (childrenlist fields) pointing to sublists representing children of the individual represented by the complex record. Sublist sharing, however, does occur because the fathers and mothers share the lists of children.

Figure 6.6 Memory in the Family Relationships Problem

6.5.4 Updating the System

How are the collections of individual records and of nametable records created originally? Suppose you have already dealt with the first hundred records and that the situation in memory accurately reflects all the information they contain. That is, the two collections are correct and as complete as possible, given this input. You now attempt to input the next record, the one hundred first. Recall that the

input consists of the individual's name, sex, birthdate, father's name, and mother's name. Input data for John Smith would appear as

JOHN SMITH  M  1/2/79  GEORGE SMITH  ALICE RICKARD

The program should check the record for validity (input validation). If valid, the two collections in memory must be updated to reflect the new information. Then the program should read the next record, and so on until there are no more to enter. An important component of the solution is an update function, which will create and add new records to the collections when necessary and enter new values into the proper fields. It must function as follows.

Figure 6.7 Typical Configuration of Individual Records

It is possible that one or more of the three names in the name fields of the newly input record are already in the nametable, with individual records having been created for them. For instance, the father's name might have been dealt with earlier when it appeared in the name field of another record. Assume first that each distinct name appearing in any field of the previously processed records has been entered into the nametable and has had an individual record created for it. Thus the nametable entry for each of these distinct names has the correct name in its name field, and a pointer in its recordptr field to its corresponding individual record. This individual record also has a pointer in its name field back to the record in the nametable it represents. Some fields of the individual record may not yet have been filled with appropriate information.

We cannot, however, assume that any name in the new input record has been seen before. Suppose John Smith is in the name field of the new record. Search the nametable to see if there is an entry for John Smith. If there is, remember where it is by placing a pointer to it in nt_ptr and then copy the pointer in the recordptr field into a variable ir_nptr. Then ir_nptr reveals where the individual record is that corresponds to John Smith. Nt_ptr is a pointer to John Smith's nametable record, and ir_nptr is a pointer to John Smith's individual record. Ir_nptr will be used later when we have to update John Smith's individual record. If there is no entry for John Smith in the nametable, you know that there is not yet an individual record created for John Smith. Create such a record, and also create an entry in the nametable corresponding to it. Store John Smith's name and a pointer to his corresponding record in the recordptr field of the nametable entry. Also place a copy of the pointer to this entry in the nameptr field of the individual record. Then copy the pointer to the individual record into ir_nptr.

At this point, whether or not information on John Smith was already in the system, both an entry in the nametable and an individual record for him exist. The nametable record is completely filled in, and the individual record has at least its nameptr field filled in. Also, ir_nptr points to the individual record.

We would follow the same process for John Smith's father and mother, creating the following four pointers.

ir_fptr

A pointer to the individual record representing John Smith's father

ir_mptr

A pointer to the individual record representing John Smith's mother

nt_fptr

A pointer to the nametable entry for John Smith's father

nt_mptr

A pointer to the nametable entry for John Smith's mother

How do we now complete the update properly? That is, which individual records and nametable entries must be modified, and how are the modifications done?

Consider the individual record for John Smith. The sex and birthdate fields can easily be filled in, as well as the fnameptr field, copying nt_fptr into it.The same can be done for mnameptr.

Now the individual record for the father of John Smith must be updated. Fill in the sex field (you know he is male). There is no additional information on his birthdate. (It may already be known. Why?) John Smith's record must be added to the list of children of his father. Assuming that the children are not kept in any special order, add his record at the front of the list. This requires copying the childrenlist pointer of his father's record into the siblingptr of John Smith's record, and then copying the pointer in ir_nptr into the childrenlist field of his father. This implies that whenever individual records are created, they must have their childrenlist and siblingptr fields set to null.

The mother's record can be treated similarly, except we may simply copy the father's childrenlist pointer (or ir_nptr) into her childrenlist field. This completes the update, which consisted of a search of the nametable and the updating of pointers in nine fields after the nametable entries and individual records were created. These nine fields were

fnameptr, mnameptr, sex, birthdate, siblingptr--for the individual record of John Smith

sex, childrenlist--for the individual record of George Smith and Alice Rickard, the father and mother

Example 6.11

Let us now trace this process with some data and show how the records would appear. Suppose the input data so far consisted of the information shown in Table 6.1, and suppose that we use different arrays to hold the individual records and the nametable entries. (Later, in Chapters 8 and 9, more appropriate implementations for collections such as nametable are introduced.) The situation in memory will then be as shown in Table 6.2.

The next input is

JOHN SMITH  M  1/2/79  GEORGE SMITH  ALICE RICKARD

Since none of these names appears in the nametable, an entry in the nametable and an individual record must be created for each of the names.

After this has been done, the new nametable entries, individual records, and saved pointers will appear as shown in Figure 6.8(a). Updating the nine fields of the individual records yields the result shown in Figure 6.8(b).

Table 6.1 Input Data for Example 6.11

       Name       Sex  Birthdate    Fathersname             Mothersname

-----------------------------------------------------------------------------

Albert Einstein    M   3/14/1879  Hermann Einstein       Paulina Koch

Leo Tolstoy        M    9/9/1828  Nikolai Tolstoy        Marie Volkronskii

George Eliot       F  11/22/1819  Robert Evans           Christina Pearson

Margaret Truman    F   2/17/1924  Harry Truman           Bess Truman

Wilhelm Bach       M  11/22/1710  Johann S. Bach         Maria Bach

Jimmy Carter       M   10/1/1924  James Earl Carter      Lillian Carter

Sir Isaac Newton   M  12/25/1642  Isaac Newton           Hannah Newton

Harry Truman       M    5/8/1884  John Truman            Martha Young

Billy Carter       M   3/29/1937  James Earl Carter      Lillian Carter

Johann S. Bach     M   3/21/1685  Johann Ambrosius Bach  Elizabeth Lämmerhirt

Johann Gottfried

 Bach              M   5/11/1715  Johann S. Bach         Maria Bach

Table 6.2 The nametable and Individual Records (for Example 6.11) after Initial Input Data Are Processed

   Nametable                                                          Individual

    Records                                                            Records

      Name                Recordptr      Nameptr  Fnameptr  Mnameptr  Siblingptr   Childrenlist  Sex  Birthdate

---------------------------------------------------------------------------------------------------------------

 1 Albert Einstein            1       1     1         2         3          0             0        M   03141879

 2 Hermann Einstein           2       2     2                              0             1        M

 3 Paulina Koch               3       3     3                              0             1        F

 4 Leo Tolstoy                4       4     4         5         6          0             0        M   09091828

 5 Nikolai Tolstoy            5       5     5                              0             4        M

 6 Marie Volkronskii          6       6     6                              0             4        F

 7 George Eliot               7       7     7         8         9          0             0        F   11221819

 8 Robert Evans               8       8     8                              0             7        M

 9 Christina Pearson          9       9     9                              0             7        F

10 Margaret Truman           10      10    10        11        12          0             0        F   02171924

11 Harry Truman              11      11    11                              0            10        M

12 Bess Truman               12      12    12                              0            10        F

13 Wilhelm Bach              13      13    13        14        15          0             0        M   11221710

14 Johann S. Bach            14      14    14        25        26          0            27        M   03211685

15 Maria Bach                15      15    15                              0            27        F

16 Jimmy Carter              16      16    16        17        18          0             0        M   10011924

17 James Earl Carter         17      17    17                              0            16        M

18 Lilian Carter             18      18    18                              0            16        F

19 Sir Isaac Newton          19      19    19        20        21          0             0        M   12251642

20 Isaac Newton              20      20    20                              0            19        M

21 Hannah Newton             21      21    21                              0            19        F

22 John Truman               22      22    22                                           11        M

23 Martha Young              23      23    23                                           11        F

24 Billy Carter              24      24    24        17        18         16             0        M   03291937

25 Johann Ambrosius Bach     25      25    25                              0            14        M

26 Elizabeth Lämmerhirt      26      26    26                              0            14        F

27 Johann Gottfried Bach     27      27    27        14        15         13             0        M   05111715

(a) With Three New Entries

(b) Updated

(c) With One Additional Entry

(d) Updated Again

Figure 6.8 Nametable and Individual Records with Saved Pointers

The next input is

CARL BACH M  3/8/1714  JOHANN S. BACH  MARIA BACH

Johann S. Bach and Maria Bach already appear in the nametable and have individual records. An entry in the nametable and an individual record must, however, be created for Carl Bach.

When this has been completed, the nametable entries, i`ndividual records, and saved pointers related to this input appear as in Figure 6.8(c). Updating the nine fields of the individual records yields the result shown in Figure 6.8(d). The time required to complete an update is the constant time for the pointer manipulations plus the time to search the nametable.

The following is an algorithm for updating the information base given another input record.

1.     Search the nametable.

a.     Create entries in the nametable and create individual records for any of the three input names not present in nametable.

b.     Set their recordptr and nameptr fields.

c.     Set their childrenlist and siblingptr fields to null.

d.     Set nt_ptr, nt_fptr, nt_mptr, ir_nptr, ir_fptr, and ir_mptr.

2.     Update the following fields of the individual record pointed to by ir_nptr:

sex to input sex

birthdate to input birthdate

fnameptr to nt_fptr

mnameptr to nt_mptr

3.     Update:

sex.fptr (the sex field of the record pointed to by ir_fptr) to m

siblingptr.ir_nptr to childrenlist.ir_fptr

childrenlist.ir_fptr to ir_nptr

sex.ir_mptr to f

childrenlist.ir_mptr to ir_nptr

6.5.5 Retrieving Information from the System

Now that we have built the collections, how do we answer questions about family relationships?

Example 6.12

How do we determine who the children of John Smith are? Search the nametable for John Smith. If the name is not there, print an appropriate message. If it is there, follow the pointer to John Smith's individual record. Traverse his list of children and count the number of records on it, or output the individual names. In any case, ignoring the search of nametable, the time required is proportional to the length of the answer--that is, it is proportional to the number of children of John Smith.

Example 6.13

How do we find all the uncles of John Smith? Search the nametable, and follow its pointer to the record for John Smith. Then, follow the fnameptr pointer to the entry in the nametable for his father and follow its recordptr back to the father's record. Do this again for the father to get to the grandfather's record. Once we have the grandfather's record we can traverse his children's list to find the uncles on John Smith's father's side. Repeat the process for John Smith's mother to find the uncles on her side.

Simply following the sibling pointer of the father's record doesn't work, because the traversal may have begun in the middle of this sibling list. To ensure starting at the beginning of the sibling list, it is best to use the children's list of the grandfather's record. If the sibling lists were circular, this would not be necessary; it would suffice to traverse the grandfather's list of children (John Smith's father's siblings) and then output the name of each male. The same processing must be done for John Smith's mother, to output all his uncles on his mother's side.

The total time, again ignoring the nametable search, is proportional to the answer, the number of uncles. Excluding the nametable search, this is certainly close to the minimal time that would be desirable. You will see in the chapters on searching and sorting how long it would take to search the nametable!

6.5.6 Other System Components

To complete design of the system, in addition to the nametable implementation and search function, components are needed for deletion of records. We will not pursue this topic here, but clearly the issues of storage reclamation and allocation would have to be considered.

Beyond this, there are some more basic unresolved problems. Exactly how are questions asked of the system?

One way is for anyone wanting answers to questions to bring them to the system's designers. The designers then write programs to output the solutions. This is feasible in certain situations; in fact, many organizations do information retrieval in this way. A related solution would be to give the data representation to questioners and have them write their own programs.

Another way is to design a simple "menu," or list, of questions that includes at least those most likely to be asked. Users could then "check off" their questions if they appear on the menu, and previously written programs could be run to answer them. This solution proves satisfactory as long as the questions people want answered are on the menu.

A more sophisticated scheme would be to design a general query language. Questioners could then ask their questions in this language. A built-in language translator would interpret the question and automatically produce a program to print the answer. Achieving this solution can be quite complex and might challenge the state of the art at the present time.

The issue of security has been ignored. It would be of major concern in any retrieval system. Users' access to specific information and ability to modify it would have to be limited.

6.5.7 Case Study Overview

This case study illustrates the use of many of the concepts this book has stressed. These include the basic concepts of top-down design, data abstraction, functional modularization, and the use of appropriate data structures to achieve clear, concise, and efficient programs.

It also shows the versatility of list-structures and the way in which pointers contribute to this versatility. List-structures provide a means of interconnecting data to cut retrieval time significantly. To do so, however, means that we have to organize the data to fit the desired retrievals. As usual, we are faced with higher costs to organize the data, that is, to build the links we will need, but gain reduced costs in search time for retrieving information. This is the classic trade-off.

Exercises

1.     Draw a picture like Figure 6.1(b), corresponding to Example 6.2 for an actual configuration of hands for the game of poker with four players.

2.     Draw pictures, similar to p of Figure 6.1(d), for the polynomials

z2(y2(x(8))) + z(y(3x)) + y(x) + 18 and x3yz + 2xz + y2xz3

3.     Give a recursive definition for a chain.

4.     Why do the two definitions of list-structures not allow more than one pointer to a record, or loops, to occur in a list-structure?

5.     Why does a null ptr and an empty stack indicate the end of the traversal of a list-structure?

6.     What will be in the stack when record 20 of formula (Figure 6.4) is accessed?

7.     Simulate the execution of the iterative implementation of traverse on p of Figure 6.1(d).

8.     Simulate the execution of the recursive implementation of traverse on p of Figure 6.1(d).

9.     Give an example of a list containing loops for which the traversal functions fail because they do not terminate.

10.     a.     Write a process function to turn the iterative traversal into a program that prints the number of sublists of a list-structure.

b.     Will your solution also work for the recursive traversal?

11.     a.     Write a function to return a pointer to the record of a list-structure that has the maximum information field value.

b.     Modify your solution to return a pointer to a list. The list should contain pointers to all records whose information field value coincides with the maximum value.

12.     Give an algorithm to determine whether two list-structures whose records have the same format are identical.

13.     Modify your solution to Exercise 12 so that the algorithm determines whether the two list-structures have the same "structure," independent of the information values stored.

14.     Each complex record of a list-structure has a field pointing to the record's sublist. The sublist is composed of a main list (composed of the chain of records linked by their link fields). Suppose each complex record of a list-structure also has a field called number that contains the number of records on the complex records sublist. Write a process function to turn the traverse functions into a program to insert a record after the record whose information field value is value. The program must, of course, also correctly update the number field for the appropriate sublist.

15.     Write a function to implement evaluate (Example 6.9).

16.     Modify your solution to Exercise 15 so that it outputs the evaluation of each operation as it occurs.

17.     Modify your solution to Exercise 16 so that it prints out an appropriate message, such as "too many operands for the operator PS of the sublist of record 7" or "missing operands for the operator PS of the sublist of record 7," whenever the list-structure does not represent a valid arithmetic expression.

18.     Assume that p points to a record of Example 6.9 containing an operator or operand. Suppose ptr points to a record of a sublist after which the record pointed to by p is to be inserted. It is to be inserted as the first record of a sublist if it is an operator, and as the next record on the sublist if it is an operand. Write an insertion function to accomplish this. For example, suppose the list-structure, ls, is currently:

If ptr points to DIFF, and the record pointed to by p contains PS, then after insertion ls will be

If the record pointed to by p contained 432.713, then after insertion ls would have been

19.     Insertion and deletion of records in list-structures is similar to their insertion and deletion in chains. However, the deletion of a sublist has ramifications for the reclaim function of Chapter 3. Discuss these, and explain how they might be resolved by modifying the definition of reclaim or by increasing the responsibility of the programmer. In either case, the idea is to be sure all unused records are returned to availist.

20.     Modify paragraphinsert (Example 6.8) so that all deleted records are returned to the availist.

21.     Let ptr and pv point, respectively, to a record of a list-structure and to its preceding record. Write a function to delete the record pointed to by ptr if it is simple, and to replace it with the list-structure to which it points if it is complex. For example, consider the following list-structure.

If ptr points to 17, then the function produces

If ptr points to the complex record following 17, then the function produces

22.     Give an implementation of next and sublist of the function traverse (Section 6.2.1) for each declaration of the polynomial p.

23.     Give a declaration for the records of Example 6.8 using dynamic memory and write implementations for all the functions of traverse and process that depend on this implementation.

24.     Draw a picture, corresponding to Table 6.2, for the complete input data (31 names) of Example 6.11.

25.     Modify the answer to Exercise 24 to reflect the lists of children being kept as circular lists.

26.     Show the complete situation in the arrays of your solution to Exercise 24 after the input below is processed.

JOHANN 0AMBROSIUS BACH  M  2/22/1695  CHRISTOPH BACH

MARIA MAGDALENA GRABLER

27.     Show the complete situation after the input of Exercise 26 is processed, when circular lists are used for the lists of children.

28.     Suppose the records of Example 6.11 were to be kept in alphabetical order by the name in the name field. What would the arrays look like after all the input data were processed?

29.     How does the algorithm for updating the information base given another input record (p. 287) have to be modified if inputs are allowed for which one or more field values are unknown?

30.     Write two functions for Example 6.11, children(person) and brothers(person). They should print out the names of all children and all brothers of person, respectively. Simulate the behavior of the functions on the data of the example.

31.     Modify your solutions to Exercise 30 so that the functions will work on circular lists of children.

32.     Suppose the nametable is implemented as a list. How will your solutions to Exercises 30 and 31 be modified?

33.     Write an algorithm to print out the names of individuals who were born before 1900. How much time will your algorithm take?

34.     Can you find a way to implement the information system so that a faster solution can be obtained for Exercise 33? If you find a faster solution, then determine how its update time will compare to that for Exercise 33.

35.     Write a detailed description for a routine that will delete an individual record and its nametable entry from the system. This implies that all relevant pointers will be properly modified and that storage will be reclaimed.

Suggested Assignment

Consider the following list-structure for a book. Book might represent the current structure of a textbook. Suppose we want to print this information as a table of contents. It would appear as follows:

CHAPTER 1

1.1  ARRAYS

1.2  RECORDS

1.2.1 POINTERS

1.2.2 EXAMPLE

1.3  SUMMARY

CHAPTER 2

2.1  LISTS

2.2  USES

CHAPTER 3

3.1  APPLICATIONS

3.1.1  BACKGROUND

3.1.2  RANKINGS

3.1.2.1 FLOWCHART

This problem can be abstracted to the printing out of a "picture" of a list-structure. Assume all records are declared as

typedef struct listrecord

{

whatever info;

int recordtype;

struct listrecord *link,*sublist;

}bookrecord,*liststructurepointer;

Write a function printlist that will print out a "picture" of the list-structure pointed to by ls. Consider, for example, the list-structure accounts of Section 6.1. The list-structure is

INFO = 175

RECORDTYPE = COMPLEX

INFO   = 0

RECORDTYPE = COMPLEX

INFO   = 100

RECORDTYPE = SIMPLE

INFO  = 75

RECORDTYPE = SIMPLE

INFO  = 450

RECORDTYPE = COMPLEX

INFO   = 200

RECORDTYPE = COMPLEX

INFO   = 250

RECORDTYPE = SIMPLE

INFO  = -46

RECORDTYPE = SIMPLE

Your program should read in and echo print three list-structures. One should be null. It should then call on printlist to print out the "picture" for each of the three list-structures.

The input data for accounts might be

1     indicates a nonnull list-structure; 0 would indicate a null list

       structure

175  CMPLX

1    1   indicates the link and sublist fields of "175" are not null

0    CMPLX

1    1

100  SMPL

0     indicates the link field of "100" is null; it is thus the last record of

       a sublist

75   SMPL

0

450  CMPLX

1    1

200  CMPLX

1    0     indicates the link field of "200" is not null, but its sublist

            field is null

250  SMPL

0

-46  SMPL

0

Notice that the input is given in traversal order.

The printlist function is to have almost exactly the same detailed refinement as either the iterative traverse (Section 6.2.1 ) or the recursive traverse (Section 6.2.2). You must write the proper process routine, say process2, to turn traverse into printlist. You may add parameters to process2 if necessary.

For extra credit, do the reading in and echo printing of the three list-structures in the program by calling a function readechoprint. It must also be obtained by writing the proper process routine, say process1, to turn traverse into a function for readechoprint.

Of course, it will be necessary for you also to write the stack operation routines setstack, push, pop, and empty, as well as next, sublist, and complex.

Go to Chapter 7      Return to Table of Contents


Data Structures Using C: CHAPTER 7: TREES

 Return to Table of Contents

CHAPTER 7: TREES

Introduces three new data structures

binary trees

trees

ordered trees

Shows

how to apply them to store data

that they are useful as conceptual aids to problem solving and uses them in the n-queens problem

how to implement them

Emphasizes the basic operations for

binary trees

preorder, inorder, and postorder traversals

trees

preorder and postorder traversals

backtracking

depth-first and breadth-first traversals

Clarifies the relation between recursion, trees, and stacks

Discusses recursive solutions to show

when they are efficient

how to improve them

7.1: Branching Out

All data structures provide a way to organize data. Different structures serve different purposes. Complex lists, as shown by the information retrieval case study of Chapter 6, can result in considerable intertwining and sharing of lists and data. Everyone is familiar with the concept of a family tree. In fact, a nice exercise would be to print the family tree for an individual whose family history is stored in the data base of that case study. The family tree is actually embedded within the lists containing the family history. Trees may be viewed as a special case of lists in which no intertwining or sharing takes place. They are the basis for structures and operations used in all aspects of programming, from the structure of programs for compilers, to work in data processing, information retrieval, and artificial intelligence. Trees are ubiquitous; they seem to sprout everywhere--even in the field of computer science!

Formally, trees are collections of nodes. Trees derive both their appearance and their name from the lines, known as branches, that connect the nodes. Understanding how to represent trees, and how to perform certain operations on them, opens the door to an appreciation of many elegant algorithms and useful techniques in computer science.

Trees play a significant role in the organization of data for efficient information retrieval and are ideal candidates for fast searches, insertions, deletions, and sequential access. Efficiency and speed are possible because trees "spread out" the data they store, so that different paths in the tree lead quickly to the relevant data. They are also convenient for conceptualizing algorithms.

Many problems can be solved by imagining all the possible answers, represented as a tree. The algorithm or program is then viewed as a traversal through the tree. The general procedure for a traversal through a tree, which will be developed in this chapter, can then be adapted and used as the basis for the program. This often results in easily created and efficient programs. In such cases the tree itself serves as a conceptual aid and need not actually be "grown" or stored in memory. Nevertheless, it provides the framework for obtaining the algorithm.

The mathematics of trees has always been of interest and is discussed quite thoroughly by Knuth [1973a]. In this chapter we consider fundamentals and introduce special trees as they are needed in subsequent chapters.

As indicated in Figure 7.1, there are many ways to depict inclusion or hierarchical relationships. Each part of the figure conveys the same information about the relations between objects A to J. Figure 7.1(a) uses indentation reminiscent of program structure, or Pascal, COBOL, and C record structure, to convey inclusion. Figure 7.1(b) uses nested blocks or sets, while Figure 7.1(c) has nested parentheses. The last, Figure 7.1(d), uses lines that branch out like a tree turned upside down. This is how the data structure that is the topic of this chapter, the tree, is depicted.

Figure 7.1 Capturing Inclusion Relationships

If Figure 7.1(a) is being used to represent the structure of a C record, then the structure shows that record A includes three fields, B, C, and D, or that D includes two subfields, I and J. The tree of Figure 7.1(d) captures this inclusion information. To a C compiler, it is also important that field B comes before C and that C precedes D. When such ordering information is needed, we use ordered trees. To capture order, as well as inclusion information, Figure 7.1(d) must be an ordered tree.

Much of the terminology for the tree data structure is taken from botany and is therefore familiar and colorful. For instance, A of the tree in Figure 7.1(d) is at the root of the tree. The tree consists of three subtrees, whose roots contain B, C, and D. The entries in the tree, A to J, are each stored at a node of the tree, connected by lines called branches. The tree is ordered if each node has a specified first, second, or nth subtree. The order itself may be specified in many ways. In this example, the order is specified by taking the subtrees in alphabetical order, determined by their root entries. Rearranging the order of the subtrees in Figure 7.1(d) results in the same tree but not the same ordered tree, if we assume the ordering is specified for each node from left to right by the way the tree is pictured. A distinction will be made between ordered trees and trees throughout this text. Keep in mind that these terms refer to two distinct kinds of structures.

Besides these two kinds of trees, there are binary trees. A binary tree is either

The null binary tree, or

A root node and its left and right subtrees, which are themselves binary trees with no common nodes

This recursive definition emphasizes the fact that binary trees with at least one node are composed of a root and two other binary trees, called the left and right subtrees, respectively. These subtrees, in turn, have the same structure. All binary trees consist of nodes, except for the null binary tree, which has no nodes. The null binary tree, just like the null list, cannot be "drawn." Both are used to make definitions more uniform and easier, and both represent the smallest data structure of their kind. There is exactly one binary tree with one node, its root. This tree has null left and right subtrees. The two binary trees with exactly two nodes are pictured as in Figure 7.2. One has a root and a nonnull left subtree, the other a root and a nonnull right subtree. Figure 7.3 pictures a larger binary tree.

The lines used to indicate a left and right subtree are called branches. Branches show the relationship between nodes and their subtrees. In Figure 7.3 the nodes are numbered 1 to 20 for easy reference. Node 5 has a predecessor (node 2), a left successor (node 10), and a right successor (node 11). Every node has at most two successors, and every node except the node labeled root has exactly one predecessor. Predecessors are also known as "parents," successors can be called "children," and nodes with the same parent are (what else?) "siblings." As far as is known, trees do not exhibit sibling rivalry.

(a) Left Subtree

(b) Right Subtree

Figure 7.2 Two Two-Node Binary Trees

Notice that, starting from any node in the tree, there is exactly one path from the node back to the root. This path is traveled by following the sequence of predecessors of the original node until the root is reached. For node 18, the sequence of predecessors is 9, 4, 2, and 1. There is always exactly one path from the root to any node.

Follow the branch from node 1 to node 2 or 3 and you come to the left or right subtree of node 1. Its right subtree is shown in Figure 7.4. Similarly, each node has a left and a right subtree. Nodes with at least one successor are called internal nodes. Those with no successors are called terminal nodes (also known as "external" nodes or "leaf" nodes). The left and right subtrees of a terminal node are always null binary trees.

By definition, the root of a binary tree is at depth 1 . A node whose path to the root contains exactly d - 1 predecessor nodes has depth d. For example, node 10 in Figure 7.3 is at depth 4. This is the path length of the node. The longest path length is called the depth of the binary tree; it is 5 for the tree of Figure 7.3. For many applications it is the depth of a binary tree, like the depth of a list-structure, that determines the required storage and execution time.

Figure 7.3 A Binary Tree

Figure 7.4 The Right Subtree of the Root of Figure 7.3

Suppose your program must count the number of occurrences of each integer appearing in an input sequence of integers known to be between zero and 100. This is easy. Store the count for each integer of the allowed range in an array and increment the appropriate entry by 1 for each integer as it is input. The code would be array [i] + +, where i is the variable containing the integer just read. Note that the efficiency of this solution derives from the selection operation for an entry, which is supported by the array data structure.

What if the range of the integers was not known? Sufficient storage is not available for an array to store counts for all possible integers. What is required is a data structure that can be dynamically extended to accommodate each new integer and its count as the integer is encountered in the input. This data structure must be searched to see if the new integer already appears. If it does, its count is increased by 1 . Should the integer not appear, it must be inserted with its count set to 1. A special tree, a binary search tree introduced in Chapter 9, is convenient to use for this data structure. This tree allows the search for an integer to consider only nodes along a particular path. At worst, this path must be followed from the root to a terminal node. Thus the longest search time will be determined by the depth of the tree. The same data structure and the depth measure are important for the related problem, "Produce an index for a book." Now, instead of storing integer counts, the position of each occurrence of a word must be retained. Do you see the possibility of storing dictionaries of modest size in binary search trees?

Figure 7.5 A Sparse Binary Tree

Binary trees need not have such a compact look as that of Figure 7.3 but can appear sparse, as in Figure 7.5. A compact binary tree like Figure 7.3 is called a complete binary tree. To be more precise, a binary tree of depth d is a complete binary tree when it has terminal nodes only at the two greatest depths. d and d -1. Further, any nodes at the greatest depth occur to the left, and any nodes at depth less than d - 1 have two successors. In Figure 7.3, nodes 16 to 20 are the nodes at greatest depth; they occur to the left. The sparse tree has the same number of nodes but twice the depth.

7.1.1 Depth versus Capacity*

*This section can be omitted on first reading.

Let us take a moment here to derive two important relations between the depth, d and the number of nodes, n, in a binary tree. First note that in any binary tree there is at most one node at depth 1, at most two nodes at depth 2, and, in general, at most 2k-1 nodes at depth k, since a node can give rise to at most two nodes at the next depth. The total number of nodes, n, is then at most 1 + 21 + 22 + . . . + 2d-1. This sum is 2d - 1. So

n    2d - 1

The minimum possible depth, dmin, for a binary tree with n nodes will be given by

dmin =   lg (n + 1)

where [image] x [image] indicates the smallest integer that equals or exceeds x. If n = 20, then dmin = [image] lg 21 [image] = [image] 4.39 [image] = 5; and if n = 31, then dmin = [image] lg 32 [image] =5. The binary tree corresponding to dmin may always be taken as a complete binary tree. Consequently, the depth of a complete binary tree will never exceed (lg n) + 1. This contrasts sharply with the depth of the sparsest binary tree with n nodes, which is n . For instance, if n is 1,000, the complete binary tree has a depth of only 10, while the sparsest has depth 1,000. This has important implications for how trees are constructed and for information retrieval, as shown in Chapter 9.

7.2: Trees of Records

The nodes of binary trees may be used to store information. For example, the nametable of Section 6.5 could be implemented by storing each of its records at a node of a binary tree. Just as lists are referred to by name, the tree can be named, say nametable. Obviously, nametable must have records inserted into it, deleted from it, and searched for in it. Insertion, deletion, search, and traversal are the basic tree operations.

Balanced trees and binary search trees (Chapter 9) and heaps (Chapter 8) are all binary trees with special properties that make them useful for storage of information. The properties that make them useful favor efficient searching, inserting, and deleting. Each of these types of binary tree requires its own algorithms for these operations, in order to take advantage of or to preserve their characteristics. For now, to get the flavor of these operations, consider some simple examples.

7.2.1 Insertion and Deletion of Records in Binary Trees

Assume that each node of a binary tree is represented by a record containing an information field, a left pointer field, and a right pointer field. The notation node.p stands for the record or node pointed to by the variable p.

To accomplish an insertion, storage must first be allocated for a new record. Next, its fields must be set correctly; this includes entering data into the record and setting its pointers to the succeeding left and right nodes, or to null if it is a leaf. Finally the proper pointer adjustments must be made in the tree to the predecessor of the new entry. Consider Figure 7.6, which shows a binary tree t before and after the insertion of a new entry C, which replaces predecessor O's null right subtree. The information for the new record is assumed to reside in value, so C is stored in value. Thus the new record must have its information field set to value and its left and right pointer fields set to null. The predecessor's right pointer field must be set to point to the new record. This is accomplished by the following.

>Comment

p = avail();

>Comment

setinfo(p,&value);

>Comment

setleft(p,null);

>Comment

setright(p,null);

>Comment

setright(predecessor,p);

Setinfo copies the contents of the storage pointed to by its second parameter into the info field of the record pointed to by its first parameter. Note that it is defined somewhat differently from the setinfo used in previous chapters. Setleft and setright copy the contents of their second parameter into the proper field of the record pointed to by their first parameter. Notice that setleft is used to set the left pointer of the new record; similarly for setright. However, setright is also used to change the right pointer of the predecessor record so that it points to the new record, which is now its right node. Thus setleft and setright can be used to set pointers for any node in the tree. Together with setinfo and avail, these functions reflect the binary tree's implementation details. The code for the insertion itself is independent of the implementation.

Figure 7.6 A Binary Tree T before and after the Insertion of C

If a new record is to replace a nonnull subtree (say, if C were to replace U in t), then the new record's left and right pointer fields would have to be adjusted accordingly. The left and right pointers from U would have to be copied into the new record's (C's) pointer fields. Furthermore, the storage for the replaced record U might be reclaimable. If so, care is required to save the pointer to it in its predecessor's left or right field so that its storage can be reclaimed. In this example, after saving the pointer to U, the right pointer of U's predecessor, L, would be replaced by a pointer to C, its new right node. Then U's storage can be reclaimed. The code below does this correctly, assuming predecessor points to a record whose nonnull right subtree is to be replaced.

>Comment

q = right(predecessor);

>Comment

p = avail();

>Comment

setinfo(p,&value);

>Comment

setleft(p,left(q));

    >Comment

setright(p,right(q));

    >Comment

setright(predecessor,p);

    >Comment

reclaim(q);

Right returns a copy of the right pointer field value of the record pointed to by its parameter. This code is for illustrative purposes. If the record is really reclaimable, it would be more efficient simply to replace its info field by the new info value. The statement setinfo(q,&value) would suffice. However, if the replaced record were needed for another purpose, say for insertion in another binary tree, then this would not work.

The deletion of a terminal node pointed to by p, whose predecessor is pointed to by predecessor, may be accomplished by the following.

    >Comment

if(left(predecessor) == p)

    >Comment

setleft(predecessor,null);

else

1 otherwise

setright(predecessor, null);

1 set its predecessor's right pointer to null

Left is analogous to right. The decision is required in order to know whether it is predecessor's left or right successor that is to be deleted. Again, if the deleted node's storage may be reclaimed, care must be taken not to lose the pointer to it. Deletion of an internal node must take into account what is to be done with the node's subtrees.

The wary reader may have qualms about the code developed for insertion and deletion. Remember--special cases must always be checked. When insertion or deletion occurs at the root node, indicated by a null value in predecessor, it is the name of the tree whose pointer must be adjusted. The illustrative code fails in this case. You should modify the program segments so that they also account for this special case.

7.2.2 Climbing Binary Trees

We have seen examples whose solution includes a traversal through a collection of records in an array or a list. A traversal through a collection of records stored in a binary tree is also the basis for the solution to many problems. We will consider three distinct ways to traverse a binary tree: (1) preorder, (2) inorder, and (3)postorder.

Pre-, in-, and postorder traversals are often referred to mnemonically as root-left-right, left-root-right, and left-right-root traversals. The prefixes pre, in, and post refer to whether the root is accessed prior to, in between, or after the corresponding traversals of the left and right subtrees. In all cases, the left subtree is traversed before the right subtree. When nodes are listed in the order in which they are processed, all nodes in the left subtree appear before all nodes in the right subtree, no matter which traversal is used. However, the type of traversal determines where the root appears. Since we defined binary trees recursively, it is convenient to give recursive definitions for these traversals.

To preorder traverse a binary tree:

if the binary tree is not the null binary tree, then

1.     access and process its root

2.     traverse its left subtree in preorder

3.     traverse its right subtree in preorder

To inorder traverse a binary tree:

if the binary tree is not the null binary tree, then

1.     traverse its left subtree in inorder

2.     access and process its root

3.     traverse its right subtree in inorder

To postorder traverse a binary tree:

if the binary tree is not the null binary tree, then

1.     traverse its left subtree in postorder

2.     traverse its right subtree in postorder

3.     access and process its root

Let's apply these definitions to Figure 7.6(b).

To do a preorder traversal of t, since it is not null, requires that its root, L, be accessed and processed first. Next, the left subtree of t must be preorder traversed. Applying the definition to it, its root, O, is accessed and processed. Since O's left subtree is null, its right subtree is traversed next in preorder. This results in C being accessed and processed, terminating the preorder traversal of O's right subtree and also of t's left subtree. Completing t's traversal requires a preorder traversal of its right subtree, resulting in U, S, and T being accessed and processed in turn.

An inorder traversal of t requires its left subtree to be inorder traversed first. Applying the definition to this left subtree requires its null left subtree to be inorder traversed (note how easy that is), its root O to be accessed and processed next, and then its right subtree to be inorder traversed, which results in C being accessed and processed. This terminates the inorder traversal of t's left subtree. T's root, L, must then be accessed and processed, followed by an inorder traversal of t's right subtree. This causes S, U, and T to be accessed and processed in turn, completing the inorder traversal of t.

Try applying the postorder traversal definition to t yourself. The sequences below represent the order in which the records stored at the nodes of t are accessed and processed for each of the traversals.

    >Comment

locust

    >Comment

oclsut

    >Comment

costul

Notice that the terminal nodes C, S, and T are dealt with in the same order for all three traversals. This is not just a coincidence but will always happen.

The trees of Figure 7.7 represent a table of contents, an arithmetic expression, and a program that invokes functions that, in turn, invoke other functions. Traversing these trees, respectively, in pre-, in-, and postorder results in the order of access to nodes shown in Figure 7.7(d-f). Each is a natural application of one of the three traversals: a listing of the table of contents, an arithmetic expression in the usual notation, and a listing of the order in which the functions are needed to test another function (for example, P1 and P2 are needed to test P3).

Not all data structures need to be traversed. For example, stacks, queues, heaps, and the hash tables to be introduced in Chapter 9 are not normally traversed. Just as applications of arrays and, even more so, lists often reduce to traversals, applications of binary trees often do as well. Therefore procedures for their traversal merit study.

7.2.3 Three Preorder Traversals of a Binary Tree

Writing a recursive function to traverse a binary tree based on the recursive definition for preorder traversal is straightforward. The same is true for an inorder and postorder traversal.

Recursive Version

preorder(ptree)

/* Preorder traverses the

binary tree tree.

*/

binarytreepointer *ptree;

{

binarytreepointer 1,r,setnull(),left(),right();

if(*ptree != setnull())

{

    >Comment

process(ptree);

    >Comment

1 = left(*ptree);

    >Comment

preorder(&l);

r = right(*ptree);

preorder(&r);

}

}

To convince yourself it works, simulate it on the tree of Figure 7.6(b).

Figure 7.7 Trees for (a) a Table of Contents, (b) an Arithmetic Expression, and (c) Components of a Program. Order of Access to Nodes is (d) for Preorder Traversal of (a), (e) for Inorder Traversal of (b), and (f) for Postorder Traversal of (c)

We will now develop an iterative procedure for contrast with the recursive one. It will also be useful if the programming language you are using is not recursive. Recall that the application of the definition of preorder traversal to Figure 7.6(b) required keeping track of the left subtree about to be traversed and remembering at which node to resume when that left subtree traversal terminated. For example, in Figure 7.6(b), after node L is processed, its left subtree must be preorder traversed. We must remember, after that traversal is complete, to resume the traversal by traversing L's right subtree. Put another way, it was necessary to suspend the current traversal, incurring its completion as a postponed obligation, and begin a traversal of the proper left subtree. These postponed obligations must be recalled and resumed in reverse order to the order in which they were incurred. That is, in a last-in first-out fashion. Obviously, the ubiquitous stack is called for. It seems to pop up everywhere.

To accomplish this let p point to the root of the subtree about to be traversed. After the processing of node.p, a pointer to its right subtree can be pushed onto the stack before traversal of its left subtree. At the time that node.p's right subtree is to be processed (when its left subtree traversal terminates), the stack will contain pointers to all right subtrees whose traversals have been postponed (right subtrees of nodes already accessed), but the top stack entry will point to node.p's right subtree. Thus popping the stack and setting p to the popped value correctly updates p to point to the correct next record to be accessed.

Nonrecursive Version Saving Right Pointers on a Stack

preorder(ptree)

/* Preorder traverses the

binary tree tree.

*/

binarytreepointer *ptree;

{

binarytreepointer p, null, setnull(), left(), right();

    >Comment

stack s;

    >Comment

null = setnull();

    >Comment

setstack(&s);

    >Comment

p = *ptree;

    >Comment

while ((p != null) || !empty(&s))

    >Comment

if (p != null)

{

    >Comment

process (ptree,p);

    >Comment

push (right(p),&s);

    >Comment

p = left(p);

}

    >Comment

else

    >Comment

pop (&s,&p);

}

The while loop condition is true whenever the traversal is not yet completed. A nonnull p means that a record is pointed to by p and is the next one to be accessed and processed. A null p value means no record is pointed to by p, and some left subtree traversal must have just been completed. If, at that point, the stack is not empty, then more needs to be done; but if it is empty, then no right subtrees remain to be traversed, and the traversal is done. Again, to convince yourself the program works, simulate it on the tree of Figure 7.6(b).

The recursive procedure reflects the structure of binary trees and emphasizes that they are made up of a root with binary trees as subtrees. It sees the special cases, such as a null tree, and the substructures of the recursive definition, such as left and right subtrees. In order to develop the nonrecursive version, it is necessary to think of the binary tree as a collection of individual records. Writing the function then involves providing for the correct next record to be accessed and processed within the while loop. This is a general strategy for attempting to find nonrecursive solutions. The nonrecursive version requires that the structure be viewed as a collection of its individual parts, with each part treated appropriately and in the correct order. This explains why recursive versions are clearer.

An alternative nonrecursive function is based on saving on a stack the path from the root to the predecessor of the node currently being processed. It is useful if the processing of a node requires accessing its predecessor node. This alternative approach guarantees that at the time the node is processed its predecessor is at the top of the stack. However, this algorithm requires some searching to find the proper right subtree to traverse next when a subtree traversal has terminated. Such a termination is indicated by p being null. Figure 7.8 shows a situation when such a termination has occurred. P is null, having been set to the null right pointer value of node 10 after node 10's left subtree has been traversed. Looking back along the path from node 10 to the root, we find the path nodes 8, 6, 5, 1. It is not difficult to see that node 5's right subtree is where the traversal must resume. Precisely this path back toward the root must be followed until a node on this return path (node 6) is the left successor of its predecessor. P must then be set to the right subtree of that node (node 12) and the traversal resumed. If no node which is the left successor of its predecessor is found on the return path, the entire tree has been traversed.

Figure 7.8 Node 5's Left Subtree Has Just Been Traversed

The following function is a nonrecursive preorder traversal that saves the path to the root on a stack. Item returns a copy of the pointer at the top of the stack without disturbing the stack.

Nonrecursive Version Saving the Path to the Root on a Stack

preorder(ptree)

/* Preorder traverses the

binary tree tree.

*/

binarytreepointer *ptree;

{

binarytreepointer null,p,q,rightpointer;

binarytreepointer setnull(),left(),right(),item();

    >Comment

stack s;

    >Comment

null = setnull();

    >Comment

setstack(&s);

    >Comment

p = *ptree;

    >Comment

while((p != null) || !empty(&s))

if(p != null)

1 if there is a current node

{

    >Comment

process(ptree,p);

    >Comment

if(left(p) != null)

{

    >Comment

push(p,&s);

    >Comment

p = left(p);

}

    >Comment

else

{

    >Comment

push(p,&s);

    >Comment

p = right(p);

}

}

else

1 otherwise

{

    >Comment

do

{

    >Comment

pop(&s,&q);

    >Comment

if (!empty(&s))

    >Comment

rightpointer = right(item(&s));

    >Comment

else

    >Comment

rightpointer = null;

    >Comment

}while(!empty(&s)&&(q==rightpointer));

    >Comment

if(q != rightpointer)

    >Comment

p = rightpointer;

}

}

Notice that the first two versions of the preorder traversal do not retain information about the depth of a node or about a node's predecessor, although they could be modified to do so. The third version retains this information, since the depth of the current node is equal to the number of stack entries plus one, and the top stack entry always points to the current node's predecessor. When this information is needed, the last version provides it directly, especially if a basic stack operation that returns the size of the stack is available.

Each of the three traversal functions can be modified to be a partial traversal when required, by including a variable done to signal termination, just as was done for the list traversals. This would be the case, for example, if you were traversing only to find a desired record. Similar function can be written for the inorder and postorder traversals.

7.3: Using the Traverse Functions for Binary Trees

In this section a number of examples are presented to show how the traverse functions developed thus far can be applied to develop solutions to programming problems.

Example 7.1

Print the number of terminal nodes in a binary tree tree.

The nonrecursive preorder traversal that saves right subtree pointers on a stack will be used to perform this task. The other versions could have been adapted as readily. The approach taken is to leave the traverse function as it is and write process to turn preorder into a solution to the problem.

To accomplish this, process must test each node it deals with to see if it is a terminal node. If so a counter, tncount, should be increased by 1. Making tncount a static variable in process allows its values to be preserved between calls to process. The first time process is called during the traversal it must initialize tncount to zero, and the last time it is called, in addition to updating tncount, process must print its final value.

If p points to the current record to be processed, then when p points to the root of the traversed tree tree, process has been called for the first time, so it must set tncount to zero. But how can process recognize when p points to the last node so that tncount can be printed? To do this, process must have access to the stack s. When s contains no nonnull pointers and node.p is a terminal node, then node.p is the last node. Assume that a function last returns true if s contains no nonnull pointers and false otherwise. Consequently s must be a parameter of, or global to, process. Finally, we must assume that only nonnull binary trees are to be considered since process is invoked by preorder only when tree is not null. Process may now be defined as follows.

process(ptree,p,ps)

/* Prints the number of terminal nodes in

binary tree tree when p points to the last

node in a preorder traverse of tree which

saves right pointers on stack s.

*/

binarytreepointer *ptree,p;

stack *ps;

{

binarytreepointer null,setnull(),left(),right();

static int tncount;

null = setnull();

    >Comment

if(p == *ptree)

    >Comment

tncount = 0;

    >Comment

if((left(p) == null) && (right(p) == null))

{

    >Comment

tncount++;

    >Comment

if(last (ps))

printf("\n The number of terminal nodes is %d",tncount);

}

}

It would certainly be more efficient to make null global so that setnull would not be invoked for each node. Similarly, setting tncount to zero in preorder could remove the need for the (p == *ptree) test for each node. In fact, it would be easier to take a second approach to finding a solution. Namely, modify preorder so that it sets tncount to zero initially, and after its while loop is exited, simply print tncount. This second solution is considerably more efficient, because the test performed by last is not needed. The second approach is more efficient, but it involves changing a traversal, which may be already compiled. Which approach to use depends on what alternatives are available and just how important the extra efficiency is.

Example 7.2

The second example concerns the problem of inserting a new node in the binary tree tree in place of the first terminal node encountered in a preorder traversal of tree. The information field value of the new node is to be copied from that of the record value.

In this case process will be written to make a solution out of the nonrecursive version of preorder that stores the path from the root to the node being processed. This is convenient, since the predecessor of the node to be processed is required by process when the insertion is made, and it will be the top stack entry at that time.

process(ptree,p,null,ps,value)

/* If p points to the first terminal node, encountered

in a preorder traversal of binary tree tree that saves

the path to the root on stack s, then replaces that node by

a new node whose info field is set to the contents of value.

*/

binarytreepointer *ptree,p,null;

stack *ps;

int value;

{

binarytreepointer q,ptr,avail(),

left(),right();

    >Comment

if((left(p) == null)&&(right(p) == null)

{

    >Comment

ptr = avail();

    >Comment

setinfo(ptr,&value);

    >Comment

setleft(ptr,null);

    >Comment

setright(ptr,null);

    >Comment

if(p != *ptree)

{

    >Comment

q = item(ps);

    >Comment

if(left(q) == p)

setleft(q,ptr);

else

setright(q,ptr);

}

    >Comment

else

    >Comment

*ptree = ptr;

    >Comment

setstack(ps);

}

}

When process is called, it tests whether the node is a terminal node. Since it will cause the traversal to be terminated after the replacement, this must be the first terminal node. Storage is then obtained for the new record and its fields filled in properly. Since it will be a terminal node, its pointer fields are set to null. To insert the new record, the predecessor of the terminal node must have one of its pointer fields set to point to the new record. The correct one is determined by whether the replaced record was its left or right successor. The special case of the insertion occurring at the root requires only that tree be set to point to the new record. Finally, setting the stack to empty ensures that the traversal will be aborted.

Value is passed by preorder to process. Item simply returns a copy of the top stack entry.

Example 7.3

The final example involves writing a function depth to return the depth of a binary tree as its value.

The nonrecursive version of preorder that saves the path to the root on the stack can be turned into such a function. When a node is being processed, its depth is one more than the number of stack entries at that moment. To determine the depth, initialize a variable d to 1 and size to zero before the while loop, increase size by 1 after push, and decrease it by 1 after pop. Give process access to d and size. Process must simply compare size+1 to d, and when greater, set d to size+1. After termination of the traversal, depth must return d.

Another solution for depth can be obtained by recursion. This requires a way to express the depth recursively. The depth of a binary tree t is

   Zero if t is null, else

      One more than the maximum of the depths of t's left and right subtrees

Now the function can be written directly as follows:

depth(tree)

/* Returns the depth of

the binary tree tree.

*/

binarytreepointer tree;

{

binarytreepointer setnull(),left(),right();

    >Comment

if(tree == setnull())

    >Comment

return(0);

    >Comment

else

    >Comment

return(1 + max(depth(left(tree)),

depth(right(tree))));

}

where max returns the maximum of its parameters as its value.

7.4: Implementing Binary Trees

This section introduces three ways to implement a binary tree.

1.     Using an array of records with just an information field

2.     Using records with an information field and left and right pointer fields linked together via their pointer fields

3.     Using a list-structure

Chapter 8 contains an application of the first implementation. A program using the second implementation, with the records stored in dynamic memory, is included. The program specifies the details of this important representation and illustrates how to create and process binary trees and how to test functions developed in the earlier examples.

7.4.1 Sequential Representation

The sequential representation uses an array to store information corresponding to each node of a tree. Consider the binary tree shown in Figure 7.9. Its depth is 5 and it has fourteen nodes. The nodes are numbered in a special way. This numbering may be obtained by preorder traversing the tree and, as each node is accessed, assigning it a node number. The root is assigned 1, each left successor of a node is assigned twice the number of its predecessor, and each right successor is assigned one more than twice the number of its predecessor. With nodes numbered in this way, the predecessor of node i is numbered i/2 (integer divide), its left successor is numbered 2*i, and its right successor 2*i+1.

Figure 7.9 Sequential Representation of a Binary Tree

The sequential representation of a binary tree is obtained by storing the record corresponding to node i of the tree as the ith record in an array of records, as shown in Figure 7.9. A node not in the tree can be recognized by a special value in its corresponding array position (such as node 8, which has - as its value) or by a node number (like 24) that exceeds the highest node number in the tree (20 in this example). Given a pointer p to a node in the array, it is easy to determine the position of its predecessor (p/2), its left successor (2*p), and its right successor (2*p+1).

A binary tree with n nodes will require an array length between n and 2n - 1. The smallest length (n) is sufficient when the binary tree is complete, and the greatest length (2n - 1) corresponds to the case of an extremely skewed binary tree whose nodes have only right successors. No space in the array is unused for a complete binary tree, and the array length is proportional to the number of nodes in the tree. However, a sparse tree can waste a great deal of array space. The worst case requires 2n - 1 positions but uses only n. Thus the sequential representation is convenient for dealing with complete static binary trees. Such trees arise, for example, when you must put student records in alphabetical order by student name. How to do such ordering is considered in the next chapter (Section 8.4). For cases when the tree may change size or shape drastically, the sequential representation is not appropriate. An example of this situation is found in the integer or word counting problem of Section 7.1.

7.4.2 Linked Representation

A binary tree data structure can also be implemented by representing each node of the tree as a record with three fields:

1.     A left pointer field,

2.     An information field

3.     A right pointer field

(called here the leftptr, info, and rightptr fields). This is the most common representation. The left and right pointer fields link records of the tree, just as the link and sublist fields of lists link their records. In effect, the pointer fields specify a node's left and right subtrees, just as the sublist field of a list specifies its sublist. The leftptr and rightptr fields of a node point to the record that represents, respectively, the root of the node's left and right subtree. The info field contains the information stored at the node. A variable referred to as the name of the tree points to the record representing the root of the tree, and a null pointer in the name denotes the null tree.

We can store each record of a binary tree using an array of records as shown in Figure 7.10, just as was done for the records of lists. The binary tree fred, for example, has its root stored in position 5 of the array, while the root of t is in position 2. the left successor of fred's root is stored in position 8, and its right successor in position 0. A minus one is used as the null pointer.

Figure 7.10 Linked Representation for Binary Trees with Records Stored in an Array

The definitions for the records and the array declaration are as follows.

    >Comment

typedef struct

{

whatever info;

int leftptr;

int rightptr;

}binarytreenode;

    >Comment

typedef int binarytreepointer;

    >Comment

# define MAX 50

    >Comment

typedef binarytreenode recordsarray[MAX];

    >Comment

recordsarray records;

Instead of using an array and array indices, we may store the records of the tree in dynamic memory and use pointers. The record declarations then become

    >Comment

typedef struct treenode

{

whatever info;

struct treenode *leftptr;

struct treenode *rightptr;

    >Comment

}binarytreenode,*binarytreepointer;

Such linked representations require storage proportional to the number of nodes of the binary tree represented, no matter what the shape of the binary tree. Contrast this with the sequential representation of binary trees. If a linked representation uses three elements per record, whereas a sequential representation uses one element per record, then for complete binary trees three times as much storage is used in a linked representation as in a sequential one. If the binary tree is not complete, then the sequential representation may require storage proportional to 2d - 1, where d is the depth of the binary tree. This could result in significant amounts of wasted space; there may not even be enough storage. Consequently, the sequential representation is more practical only when dealing with complete or nonsparse binary trees.

Using the linked representation, it is obviously possible to insert or delete records, once we know where to do so, by making changes in only a few pointer fields.

The following program illustrates the binary tree implementation with records stored in dynamic memory. It is a program that could be used to test all the functions of Section 7.3 (although they have been renamed). The names are self-explanatory. The input consists of

An integer to indicate whether the binary tree being created is null,

A sequence of data for each record

A value for the info field of a new node

The sequence is assumed to be in the order in which the records are accessed in a preorder traversal of the binary tree being input. For example, if the input corresponded to the right subtree of Figure 7.9 and the new value replacing the first terminal node were 50, then the input would be as follows. (Program output and prompts are not shown.)

Typical Input

1    indicates nonnull binary tree

3    data for the root;

11   both its subtrees are nonnull

6    data for the next record in preorder;

11   both its subtrees are nonnull1

12   data for the next record in preorder;

00   both its subtrees are null1

13   data for the next record in preorder;

00   both its subtrees are null

7    data for the next record in preorder;

00   both its subtrees are null

50   the new value

The Program

Reads the data for nodes of a binary tree, creates

the tree and prints how many terminal nodes it

has and its depth, reads the value for the new node,

replaces the first terminal node in preorder access

by the new node, and then prints all the final binary

tree's nodes, its terminal count, and its depth.

#include <stdio.h>

    >Comment

typedef struct treerecord

{

int info;

struct treerecord *leftptr;

struct treerecord *rightptr;

}binarytreerecord,*binarytreepointer;

#define NULL 0

binarytreepointer left(p)

/* Returns a copy of the left

pointer of the node p points to.

*/

binarytreepointer p;

{

return(p->leftptr);

}

binarytreepointer right(p)

/* Returns a copy of the right

pointer of the node p points to.

*/

binarytreepointer p;

{

return(p->rightptr);

}

binarytreepointer setnull()

/* Returns a null pointer */

{

return(NULL);

}

binarytreepointer avail()

/* Returns a pointer to storage

allocated for a new node.

*/

{

return(malloc(sizeof(binarytreerecord)));

}

setinfo(p,pvalue)

/* Copies the contents of value

into the record p points to.

*/

binarytreepointer p;

int *pvalue;

{

p->info = *pvalue;

}

setleft(p,q)

/* Copies q into the left pointer

of the record p points to.

*/

binarytreepointer p,q;

{

p->leftptr = q;

}

setright(p,q)

/* Copies q into the right pointer

of the record p points to.

*/

binarytreepointer p,q;

{

p->rightptr = q;

}

printnode(pl,ptr)

/* Prints the info field of the

record ptr points to.

*/

binarytreepointer *pl,ptr;

{

printf("\n %d",ptr->info);

}

    >Comment

main()

/* Reads the data for nodes of a binary tree, creates

the tree and prints how many terminal nodes it

has and its depth, reads the value for the new node,

replaces the first terminal node in preorder access

by the new node, and then prints all the final

binary tree's nodes, its terminal count, and its

depth.

*/

{

int n,value;

binarytreepointer t,setnull(),avail();

printf("\n enter binarytree 0 for null tree 1\

otherwise \n");

scanf("%d",&n);

    >Comment

if (n == 0)

t = setnull();

else

{

    >Comment

t = avail();

createtree(&t);

}

    >Comment

printtree(&t);

    >Comment

terminalcount(&t);

    >Comment

printf("\n The depth is %d",depth(t));

printf("\n enter value ");

    >Comment

scanf("%d",&value);

    >Comment

replacefirstterminal(&t,value);

    >Comment

printtree(&t);

    >Comment

terminalcount(&t);

    >Comment

printf("\n The depth is %d",depth(t));

}

    >Comment

createtree(ptree)

/* Read the data and create the binary tree tree. */

binarytreepointer *ptree;

{

    >Comment

binarytreepointer l,r,setnull(),left(),right();

if (*ptree != setnull())

{

    >Comment

createnode(ptree);

l = left(*ptree);

    >Comment

createtree(&l);

r = right(*ptree);

    >Comment

createtree(&r);

}

}

createnode(pptr)

/* Reads the data, and fills in the fields

of the node pointed to by ptr.

*/

binarytreepointer *pptr;

{

int leftlink,rightlink,value;

binarytreepointer setnull(),avail();

printf("\n enter info \n");

    >Comment

scanf("%d",&value);

    >Comment

setinfo(*pptr,&value);

    >Comment

printf("\n enter enter left & right ptrs \n");

scanf("%d %d",&leftlink,&rightlink);

    >Comment

if (leftlink == 0)

setleft(*pptr,setnull());

else

setleft(*pptr,avail());

if (rightlink == 0)

setright(*pptr,setnull();

else

setright(*pptr,avail());

}

    >Comment

#define LIMIT 50

    >Comment

typedef binarytreepointer whatever;

typedef struct

{

whatever stackarray[LIMIT];

int top;

}stack;

#define TRUE 1

#define FALSE 0

setstack(ps)

/* Sets the stack s to empty. */

stack *ps;

{

(*ps).top = -1;

}

empty(ps)

/* Returns true only if stack s is empty.*/

stack *ps;

{

return((*ps).top == -1);

}

push(value,ps)

/* Inserts value at the top of stack s. */

whatever value;

stack *ps;

{

if ((*ps).top == (LIMIT - 1))

overflow(ps);

else

{

(*ps).top = (*ps).top + 1;

(*ps).stackarray[(*ps).top] = value;

}

}

pop(ps,pvalue)

/* Removes the top entry of stack s

and copies its contents into value.

*/

stack *ps;

whatever *pvalue;

{

if (empty(ps))

underflow(ps);

else

{

*pvalue = (*ps).stackarray[(*ps).top];

(*ps).top = (*ps).top - 1;

}

}

whatever item(ps)

/* Returns a copy of the top entry on stack s. */

stack *ps;

{

return((*ps).stackarray[(*ps).top]);

}

last(ps)

/* Returns true only if stack s

contains no non null pointers.

*/

stack *ps;

{

int i,temp;

binarytreepointer null,setnull();

null = setnull();

temp = TRUE;

for (i=0;i<=(*ps).top;i++)

if ((*ps).stackarray[i] != null)

temp = FALSE;

return(temp);

}

overflow(ps)

/* Prints a message if the stack overflows. */

stack *ps;

{

printf("\n stack overflow ");

}

underflow(ps)

/* Prints a message if the stack underflows. */

stack *ps;

{

printf("\n stack underflow ");

}

    >Comment

    >Comment

printtree(ptree)

/* Prints the info field values of all

nodes of the binary tree tree.

*/

binarytreepointer *ptree;

{

binarytreepointer null,p,q,rightpointer;

binarytreepointer setnull(),left(),right(),item();

stack s1;

null = setnull();

setstack(&s1);

p = *ptree;

while ((p != null) || !empty(&s1))

if (p != null)

{

    >Comment

printnode(ptree,p);

if (left(p) != null)

{

push(p,&s1);

p = left(p);

}

else

{

push(p,&s1);

p = right(p);

}

}

else

{

do

{

pop(&s1,&q);

if (!empty(&s1))

rightpointer = right(item(&s1));

else

rightpointer = null;

}while(!empty(&s1) &&

(q == rightpointer));

if (q != rightpointer)

p = rightpointer;

}

}

    >Comment

terminalcount(ptree)

/* Prints the number of terminal nodes

in the binary tree tree.

*/

binarytreepointer *ptree;

{

binarytreepointer p,null,setnull(),left(),right();

stack s2;

null = setnull();

setstack(&s2);

p = *ptree;

while ((p != null) || !empty(&s2))

if (p != null)

{

    >Comment

updatetncount(ptree,p,&s2);

push(right(p),&s2);

p = left(p);

}

else

pop(&s2,&p);

}

updatetncount(pt,p,ps)

/* Prints the number of terminal nodes in

binary tree tree when p points to the last

node in a preorder traverse of tree which

saves right pointers on stack s.

*/

binarytreepointer *pt,p;

stack *ps;

{

binarytreepointer null,setnull(),left(),right();

static int tncount;

null = setnull();

if (p == *pt)

tncount = O;

if ((left(p) == null) && (right(p) == null))

{

tncount++;

if (last(ps))

printf("\n The number of terminal nodes\

is %d",tncount);

}

}

    >Comment

replacefirstterminal(ptree,value)

/* Replaces the first terminal node, encountered

in a preorder traversal of binary tree tree, by a

new node with its info field set to the contents

of value.

*/

binarytreepointer *ptree;

int value;

{

binarytreepointer null,p,q,rightpointer;

binarytreepointer setnull(),left(),right(),item();

stack s3;

null = setnull();

setstack(&s3);

p = *ptree;

while ((p != null) | | !empty(&s3))

if (p != null)

{

replacenode(ptree,p,null,&s  ,value);

if (left (p) != null)

{

push(p,&s3);

p = left(p);

}

else

{

push(p,&s3);

p = right(p);

}

}

else

{

do

{

pop(&s3,&q);

if (!empty(&s3))

rightpointer = right(item(&s3));

else

rightpointer = null;

}while (!empty(&s3) &&

(q == rightpointer));

if (q != rightpointer)

p = right pointer;

}

}

replacenode(ptree,p,null,ps,value)

/* If p points to the first terminal node, encountered

in a preorder traversal of binary tree tree that

saves the path to the root on stack s, then

replaces that node by a new node whose info field is

set to the contents of value.

*/

binarytreepointer *ptree,p,null;

stack *ps;

int value;

{

binarytreepointer q,ptr,avail(),left(),right();

if ((left(p) == null) && (right(p) == null))

{

ptr = avail();

setinfo(ptr,&value);

setleft(ptr,null);

setright(ptr,null);

if (p != *ptree)

{

q = item(ps);

if (left(q) == p)

setleft(q,ptr);

else

setright(q,ptr);

}

else

*ptree = ptr;

setstack(ps);

}

}

depth(tree)

/* Returns the depth of the binary tree tree. */

binarytreepointer tree;

{

binarytreepointer setnull(),left(),right();

if (tree == setnull())

return(0);

else

return(1 + max(depth(left(tree)),

depth(right(tree))));

}

max(i,j)

/* Returns the maximum of i and j. */

int i,j;

{

if (i >= j)

return(i);

else

return(j);

}

7.4.3 List-structure Representation

Another way to represent a binary tree is indicated in Figure 7.11. Each node's record is made complex. A complex record's sublist points to the representation of the node's left subtree. The result is that btree is implemented as the list structure lstree. Abstractly, binary trees are a special case of list-structures, but there are list-structures that do not correspond to binary trees. However, in computer science the term tree usually implies zero or restricted sharing of storage, whereas the term list-structure or list implies the possibility of more general sharing of storage.

7.5: Trees

So far we have considered only binary trees, but trees may have nodes with more than two successors. This is the case in the tree of Figure 7.1, where the nodes have up to three successors. It would be natural to assume that binary trees are a special case of trees, but the tree literature does not treat them this way. The formal definition of a tree, as opposed to a binary tree, is as follows: A tree is a root node with subtrees t1, t2, . . . , tn that are themselves trees and have no nodes in common.

Formally there are no null trees (although there are null binary trees), so each tree has at least one node, its root. It is convenient nonetheless to refer to null trees, so the terminology applied to binary trees also applies to trees. The important distinction is that no order is imposed on the subtrees of a tree, whereas for binary trees there is a distinct left and right subtree. Consequently, when trees are depicted, it makes no difference in what order subtrees are drawn; any order represents the same tree. When order among the subtrees is to be imposed, they are called ordered trees. For example the structures in Figure 7.12, when viewed as binary trees, represent two distinct binary trees, since their left and right subtrees differ, but they represent the same tree when viewed as trees.

Figure 7.11 (a) List-structure Representation of (b) a Binary Tree

Figure 7.12 Two Distinct Binary Trees or One General Tree

7.5.1 Implementing Trees as Binary Trees

When a constraint is imposed limiting the number of successors of any node to at most k, the tree is called a k-ary tree. Such trees may be implemented by generalizing the sequential representation for binary trees. The nodes are numbered as before, except the jth successor of the node numbered i is now assigned node number k X (i - 1) + (j + 1). Thus, when k = 5, the fourth successor (j = 4) of the node numbered i (i = 3) is assigned the number 5*(3 - 1) + (4 + 1) = 15. A 5-ary tree with nodes numbered in this way is

This implementation will not be pursued further but is similar in character to the sequential implementation for binary trees. This means that formulas can be derived to compute the array position of a node's predecessor and successors, which are important requirements for tree processing. Also, this representation is convenient for essentially static, healthy-looking (nonparse) trees.

Another representation is based on a natural generalization of the linked representation for binary trees. Simply expand each record so that it has as many pointer fields as it has subtrees. However, this leads to variable-length records, which are usually not desirable. The variable-length records occur because nodes need not all have the same number of successors. Further, it may not even be possible to predict in advance what the greatest number of successors will be.When it is known that the tree will be a k-ary tree, then all records could be made the same length, large enough to accommodate k subtree pointers. This gives a fixed record size but can result in considerable wasted storage for unused pointer fields within records. This is particularly true for sparse trees.

Surprisingly, there is a way to represent a tree as a binary tree, which then allows all records to be of fixed length. This is an important result. Since any tree can be represented by a binary tree, studying only binary trees is really no restriction at all.

Example 7.4

Let us now look at how to represent a specific tree as a binary tree. Consider the tree of Figure 7.13(a). How can this tree be represented as a binary tree?

The binary tree is obtained as follows: First, create a root A of the binary tree to correspond to the root A of the tree. Next, pick any successor of A in the tree--say, B--and create a node B in the binary tree as A's left successor. Then take the remaining siblings of B, which are C and D in this case, and let C be B's right successor and D be C's right successor, in the binary tree. So far this yields the binary tree shown in Figure 7.13(b). If we view this procedure as the processing of node A of the tree, then the binary tree is completed by repeating this procedure for every other node of the tree. For example, after node B is processed (that is, its successors have been added to the binary tree), the result is Figure 7.13(c). The final binary tree created using this procedure appears in Figure 7.13(d).

(a) Initial Tree

(b) Binary Tree for Nodes A - D

(c) Binary Tree for Nodes A - G

(d) Final Binary Tree

Figure 7.13 Representation of a General Tree as a Binary Tree

Applying this procedure, given any tree, a corresponding binary tree can be constructed. The specific binary tree generated will depend on the order in which successors are taken in the tree. For instance, had C been selected initially as the first successor of A, then the resultant binary tree would be different. If the ordering of each node's successors is specified and they are taken in the specified order, then the construction creates a unique binary tree for each tree. In any case, a tree with n nodes will correspond to a binary tree with n nodes. It is not difficult to find an algorithm that, given a binary tree with no right subtree, can reverse the construction and generate the tree it represents. This means that when the tree is ordered, there is a unique binary tree that can be constructed to represent it. Conversely, given a binary tree with no right subtree, there is a unique tree that it represents, which can also be constructed. Hence there are exactly as many trees with n nodes as there are binary trees with n nodes and no right subtrees.

If we define an ordered forest as an ordered collection of ordered trees, then each tree has its unique binary tree representation. Can you see how to append these binary trees to the binary tree representing the first ordered tree of the ordered forest to obtain a binary tree representing the forest? (Hint: Use the fact that the binary trees have no right subtrees.) For those of you who wish to count trees, pursuing this should lead to the conclusion that there are exactly as many ordered forests as there are binary trees.

7.6: Traversals of Trees

In this section algorithms are developed for traversing trees and are applied to two examples. The second example should be studied carefully, because it illustrates an important modified traversal of trees that has many applications. Assuming the trees are ordered, it makes sense to refer to a tree's first subtree, second subtree, and so on. If an order is not given, the programmer can impose one.

When trees are used to store data at their nodes, then tree traversals give a means to access all the data. However, a tree might be used to represent a maze or all routes from Philadelphia to San Francisco. Traversal of the tree then provides the means to find exits from the maze or the shortest route across the country. The maze and the routes must not intersect. When they do, a more general data structure, a graph, is needed for their representation. Unlike trees, a graph allows more than one path between its nodes. * Should a tree contain family relationships by storing an individual's record at a node, a traversal allows the parent (predecessor) or the children (successors) of an individual to be found.

* It is not difficult to generalize tree traversal algorithms to obtain graph traversal algorithms.

The preorder and postorder traversals of binary trees generalize quite naturally to ordered trees.

To preorder traverse an ordered tree:

1. Access and process its root, then

2. Preorder traverse its subtrees in order.

To postorder traverse an ordered tree:

1. Postorder traverse its subtrees in order, then

2. Access and process its root.

There is no natural inorder traversal of trees. Preorder and postorder traversals of a tree correspond, respectively, to preorder and inorder traversals of its corresponding binary tree. The preorder and postorder access sequences for the tree of Figure 7.13(a) are ABEFGCHDIKLMJ and EFGBHCKLMIJDA. You should confirm that a preorder and an inorder traversal of the binary tree of Figure 7.13 result in these respective access sequences. A preorder traversal is also called a depth-first traversal.

7.6.1 Obtaining the Binary Tree for a Tree

It should now be clear that confining attention to binary trees, their representation, and operations on them is not actually a restriction. That is, since trees can be represented as binary trees, any operations can be performed on them by performing equivalent operations on their binary tree representations. In this sense no generality is lost in dealing with binary trees.

Example 7.5

Let the task for this example be the development of an algorithm to generate the binary tree corresponding to a general tree.

In order to do this we must be able to traverse a general tree. As it is traversed, we can generate the binary tree by properly processing each node as it is accessed. A nonrecursive algorithm to preorder traverse a tree t follows.

Nonrecursive Tree Preorder Traversal

1.     Set p to t

2.     Setstack_t(ts)

3.     While p is not null or not empty_t(ts),

if p is not null, then

process(t,p)

For each successor pointer ps of node.p,

except next(p),

push_t(ps,ts)

Update p to next(p)

else

pop_t(ts,p)

This algorithm is a generalization of the nonrecursive preorder traversal of a binary tree that saves right pointers. In the binary tree case, the next subtree is the left subtree, and only one successor of the node just processed must be saved--the right successor. The difference is that now all the subtree pointers of the node just processed, except for the next one, must be saved.

Now that we have a general tree traversal, it will be adapted to our task. Let bt point to the binary tree to be generated by the algorithm. Recall the list-structure representation of a binary tree. Think of the leftpointer field of each node of bt as pointing to a sublist. This sublist contains all the successors of the corresponding node in t. Thus, in Figure 7.13(d), node A of the binary tree representation has a leftpointer field pointing to the sublist consisting of nodes B, C, and D. These are the successors of A in the general tree, Figure 7.13(a). Similarly, D has a leftpointer field pointing to the sublist consisting of I and J, the successors of D in the general tree.

Consequently, to produce bt, when a node of t is processed we must

Create a sublist consisting of all its successors

Set the leftpointer field of the corresponding node in bt to point to this sublist

Figure 7.14 Binary Tree BT Generated from General Tree T

Copy the info field of the node in t into the info field of the corresponding node in bt

The rightpointer fields of these nodes in bt serve as the link fields of the sublist.

Recall that, as we preorder traverse a general tree t, we are preorder traversing its corresponding binary tree, bt. That is, the order in which the nodes of t are accessed is exactly the same as the order in which the corresponding nodes of bt are accessed. Thus we can achieve a solution by preorder traversing both t and bt at the same time, and processing the corresponding nodes as described above. For example, if this were carried out on the general and final binary tree of Figure 7.13, and nodes A, B, E, F, G, and C had been processed, we would have generated the binary tree shown as Figure 7.14(b).

Suppose p points to the node currently being processed in t. Let pb point to the corresponding node in bt. In the example, p would now point to D of t and pb to D of bt.

After processing the node to which p points, we can update p to point to one of its successors, next(p), and stack on ts all pointers to the remaining successors. These pointers represent the latest postponed obligations for the preorder traversal of t. Next (p) is a function returning a pointer to the successor of p that is not stacked, or a null pointer if there are no successors. At the same time, to keep track of the postponed obligations for bt's preorder traversal, we will keep a parallel stack, bts. Only nonnull pointers will be pushed onto bts. When the top pointer in stack ts is popped, the top pointer in bts will also be popped. These pointers will then point to corresponding nodes--one in t, the other in bt. Storage can be created for the nodes of bt as the traversal proceeds by using avail to return a pointer to the available storage for a node, since a pointer variable implementation is used for the binary tree.

One last detail remains. In order to do the processing required on node.p and node.pb, process will be invoked. As already noted, process must create a sublist consisting of new nodes corresponding to the successors of node.p in t, set leftptr.pb to point to that sublist, and copy info.p into info.pb. We assume that a procedure create, given p, creates the required sublist, and returns with first pointing to the first node of that sublist. The preorder traversal algorithm to create the binary tree corresponding to a general tree is as follows:

Nonrecursive Algorithm to Generate a Binary Tree Corresponding to a General Tree

1.     Set p to t

If p is null

Set bt to null

else

bt = avail()

Set pb to bt

2.     Setstack_t(ts)

Setstack_bt(bts)

3.     While p is not null or not empty_t(ts),

if p is not null, then

process(t,bt,p,pb).

For each successor pointer ps of node.p, except next(p),

push_t(ps,ts)

If right(pb) is not null

push_bt(right(pb),bts)

p = next(p)

pb = left(pb)

else

pop_t(ts,p)

pop_bt(bts,pb).

The process algorithm to be used in the preorder traversal is:

Create(p,first)

Set leftptr.pb to first

Set info.pb to info.p

Notice that the traversal of the binary tree bt is embedded in the traversal of the general tree t. Also, suffixes have been used to distinguish the stack storing pointers to nodes of the general tree from the stack storing pointers to nodes of the binary tree. Finally, since the tree t is not ordered, the function next is free to pick any successor. The order in which the rest of the successors are placed on the stack is arbitrary but must determine the order in which create places them on the sublist. You should simulate this algorithm on the tree of Figure 7.13(a) to see how it works.

The correspondence between an ordered tree and its binary tree can now be defined more formally. In this case the binary tree corresponding to a tree is unique. An ordered tree t may be represented by a binary tree bt as follows:

Each node of t corresponds to a node of bt,

If node n of t corresponds to node nb of bt, then the leftmost successor of n in t corresponds to the left successor of nb in bt;

All other siblings of n in t form a right chain from the node in bt corresponding to the leftmost successor of n in t.

We may now give a recursive definition specifying an algorithm for generating the binary tree bt representing the general ordered tree t.

To generate a binary tree corresponding to an ordered tree:

If t is null

Set bt to null

else

Create the root node of bt corresponding to the root of t.

If t has a subtree, then generate the binary tree bt.left corresponding to the leftmost subtree of t,t.leftmost, and make the generated binary tree the left subtree of bt.

For every other subtree of t, in order, generate the binary tree corresponding to the subtree, and insert it at the end of the right chain of bt.left.

For illustration this algorithm is applied to the ordered tree in t in Figure 7.15(a).

First create the root of BT corresponding to the root of t shown in Figure 7.15(b). To carry out statement 2, note that t has three subtrees with B at the root of its leftmost subtree. We must thus generate bt.left to orrespond to Figure 7.15(c). To do this, apply the definition to this tree and obtain Figure 7.15(d) as its corresponding binary tree. Completing the first part of statement 2 for t  yields the binary tree shown in Figure 7.15(e).

Figure 7.15 Generation of Binary Tree BT from an Ordered Tree T

To complete the second part of statement 2, generate the binary tree for the next subtree of t, with C at its root, and then insert it at the end of the right chain of B to obtain Figure 7.15(f). Finally, do the same for the last subtree of A, with D at its root, and complete the application to t, to obtain the result shown in Figure 7.15(g).

This is another example for which we found a recursive solution. Compare the clarity and conciseness of the recursive solution with that of the nonrecursive solution.

7.6.2 Backtracking: The n-queens Problem

In the last section a tree traversal was developed and used to generate a binary tree. Here and in the following two sections we will develop a modified tree traversal that employs backtracking. Backtracking is a useful technique in solving many problems. First we apply it to the n-queens problem.

Chess is played on a square board with eight rows and eight columns, just like a checkerboard. We can think of the board as a two-dimensional array, board (Figure 7.16).

A queen located in the jth row and kth column may move to a new position in one of the following ways:

      To the right, or left, along its row

      Up, or down, in its column

      Up, or down, along its d1 diagonal

      Up, or down, along its d2 diagonal

Figure 7.16 A Partial Chessboard Represented as a Two-Dimensional Array, BOARD

The n-queens problem is to find all ways to place n queens on an n X n checkerboard so that no queen may take another. As an example, the eight-queens problem is to find all ways to place eight queens on the chessboard so that no queen may take any other--that is, move into a position on its next move that is occupied by any other queen.

The question is how to solve the n-queens problem. At first glance this problem has little relation to trees, but let us see. It is not even clear that a solution exists for all n.

Since a queen can be moved along one of its rows, columns, or diagonals, a solution clearly is achieved by specifying how to place the n queens so that exactly one queen appears in each row and column, and no more than one queen appears on any diagonal of the board. One could attempt to find a solution by searching through all possible placements of the n queens satisfying these restrictions. There are, however, n! such placements. Even for a moderate n, this is obviously not a practical solution. Instead we must construct a solution. The idea behind the construction is to determine if the placements obtained for the first i queens can lead to a solution. If they cannot, abandon that construction and try another, since placing the remaining n - i queens will not be fruitful. This may avoid the need to consider all possible constructions.

To construct a solution involves testing a partial construction to determine whether or not it can lead to a solution or must be abandoned as a dead end. This testing is the next question.

Consider a general tree, with each of its nodes representing a sequence of decisions on the placement of queens. Take the root to represent the initial situation, in which no decisions have yet been made. The root will have n successors. Each successor corresponds to a choice of row for the placement of the first queen. Each of these successor nodes, in turn, has n successors, corresponding to a choice of row for the placement of the second queen, and so on. The tree for the four-queens problem is shown in Figure 7.17.

Since exactly one queen must appear in each column in any solution, assume the ith queen is placed in column i. Hence, each path, from the root to a node at depth i in the tree, specifies a partial construction in which the first i queens have been placed on the board. The path indicated in the four-queens tree specifies a partial construction in which the first, second, and third queens have been placed in rows 2, 4, and 1 (and columns 1, 2, and 3), respectively. The tree helps us visualize all possible solutions and provides a framework in which to view what we are doing.

Suppose a path has been found to a node at depth k. The path is feasible if none of the k queens whose positions are specified by the path can take any other. If the node at depth k is a terminal node, and the path is feasible, then a solution has been found and may be printed. If the node at depth k is not terminal, and the path is feasible, we want to extend the path to a node at depth k + 1. Let p point to such a node. Then node.p must be a successor of the node on the path at depth k.

Figure 7.17 Tree for the Four-queens Problem

P is feasible if the position that node. p specifies for the (k + 1)th queen does not allow it to take any of the other k queens. If p is feasible, the path can be extended downward in the tree to include node.p, so there is now a feasible path of depth k + 1. If p is not feasible, then another successor of the node at depth k must be tried until a feasible one is found to extend the path, or until all have been tried and none is feasible. In the latter case, the choice of node at depth k cannot be extended to a solution, and the computer backs up in the tree to a node representing a shorter path that has not yet been explored. This procedure to extend the new path is repeated until a solution is found or until all possibilities have been tried.

The procedure described amounts to a modified general tree traversal called backtracking. A different approach would be to create an algorithm for the problem by applying a general tree preorder traversal that uses a process routine. The process routine would check each node to be processed to see whether the node is terminal and represents a feasible path. If so, it prints the solution. This approach amounts to an exhaustive search of all n! possibilities, since the preorder traversal backs up only when the preorder traversal of a subtree has been completed--that is, after a terminal node has been processed. The backtracking traversal need not access and process all nodes. It backs up when the traversal of a subtree has been completed or when it becomes known that no path involving the root of a subtree can be extended to a solution. The backtracking procedure generates only feasible paths--not all paths. In effect, it prunes the tree by ignoring all nodes that cannot lead to a solution.

7.6.3 Depth-first

A backtracking algorithm can be derived by modifying the general tree preorder traversal algorithm. The "p not null" test in the loop body of the preorder traversal algorithm must be replaced by two tests. One to see if p points to an existing node and another "p feasible" test to prune the tree. How this feasibility test is implemented is what determines the amount of pruning that occurs and hence the efficiency of the solution. The process routine called in the loop body must determine if node.p is terminal, and, if so, print the solution. If these were the only changes made, then nothing at all would be printed when no solution exists.

You, as the programmer, do not know whether a solution exists for every n. Therefore, initialize a flag, found, to false, and have process set found to true if it finds a solution. Upon exiting from the loop body, test found, and if it is false print "No solution." The resultant algorithm for a backtracking preorder traversal is as follows:

Backtracking Preorder Tree Traversal

1.     Set p to the root of t.

2.     Set found to false.

3.     Set the stack to empty.

4.     While p is not null or the stack is not empty,

if p is not null, then

if p is feasible, then

process(p),

push all successors of node.p onto the stack except next(p),

move down in t by setting p to next(p),

else

backtrack in t by popping the stack and setting p to the popped value,

else

backtrack in t by popping the stack and setting p to the popped value;

if not found then

print no solutions.

The function next(p) is assumed to return a null value if node.p has no successors. If only one solution is to be found and printed, only the test of the while loop must be changed. It should become "(p not null or stack not empty) and (not found)." The process algorithm to be used with the backtracking preorder traversal is as follows:

If node.p is a terminal node, then

1.     Print the solution, and

2.      Set found to true.

The algorithms developed in this section are quite general. They are applicable to any problem that can be reduced to a backtracking traversal (like the n-queens problem). They should be viewed as general tools for problem solving. We must now specialize them to the n-queens problem so that they are detailed enough to implement as a function. Data structure decisions, specifically the implementation of the current board configuration, must be made along the way.

The tree is not actually stored in memory; it was used as a conceptual aid in formulating the problem abstractly. As a result, the problem has been recognized as one that may be solved by a general tree backtracking procedure. It is not necessary to refer to the tree at all, but we have done so here to place the problem in this more general context, and we continue with this approach.

Suppose p points to a node at depth k. The depth of the node specifies the placement for the kth queen, column k. P itself determines the choice of row. In this way p corresponds to a particular row and col pair and vice versa. A nonnull value for p corresponds to a column and row value <n + 1. Initializing p to the first successor of the root corresponds to setting row to 1 and col to 1.

Next must have row and col as parameters; it simply increases col by 1 and sets row to 1. As the traversal proceeds, row and col vary in a regular way. This can be used to advantage. Instead of pushing all successors of node.p onto the stack, which would involve n - 1 entries, simply push the current value of row. Then the backtracking task can be carried out by adding 1 to the current value of row when this value is less than n. If it equals n, the stack must be popped, row set to the popped value plus 1, and col set to col-1.

To implement the crucial test, "p feasible," we use a function feasible to return the value true when p is feasible and false otherwise. Feasible must have row and col as parameters and must have access to information about the current board configuration. It must determine if placing a queen in the position specified by row and col will allow it to "take" one of the currently placed queens. If a queen may be taken, it must return "false," otherwise "true."

We could use an n X n two-dimensional array board to keep track of the current board configuration. If we did this, feasible would have to traverse through the row, column, and diagonals specified by row and col in order to determine what value to return. This checking must involve all entries in many rows and columns. Since feasible is invoked in the loop body of the backtracking preorder traversal algorithm, it is executed once for every node of the tree that is accessed. Operations that appear in loop bodies should be done efficiently since they have a significant effect on the overall efficiency of the implementation.

To do its job, feasible must know whether or not the row, col, and the diagonals specified by row and col are occupied. If this information can be efficiently extracted and made available, feasible will be more efficient. We now see exactly what information is required and proceed to the details involved in efficiently extracting it.

Consider Figure 7.16. Notice that entries [j, k], [j + 1, k - 1], and [j - 1, k +1] all lie along the d1 diagonal. Notice also that j+ k = (j+ 1) + (k - 1) = (j- 1) + (k + 1). This means that all entries on the diagonal d1, determined by j and k, have the same row + col sum: j + k. The same is true of differences along d2. That is, all entries on the diagonal d2, determined by j and k, have the same row minus col difference: j - k. Suppose we keep information about the current board configuration as follows:

r[j]

Is true if the current board configuration has no queen in the jth row, and is false otherwise.

d1[j + k]

Is true if the current board configuration has no queen along the diagonal d1 with row + col sum j + k, and false otherwise.

d2[j - k]

Is true if the current board configuration has no queen along the diagonal d2 with row - col sum j - k, and false otherwise.

Feasible can now simply return the value of (r[j] and d1 [j + k] and d2[j - k]). The arrays r, d1, and d2 require storage for n, 2(n - 1), and 2(n - 1) entries, respectively, and must be properly initialized, but feasible now takes constant time to execute. Also, when row and col are changed, these changes must be reflected in r, d1, and d2. What we have done amounts to finding a way to store the current board configuration so that the operations performed on it by the algorithm are done efficiently. R, d1, and d2 have indices that range from 1 to n, 2 to 2n, and -(n - 1) to (n - 1), respectively. Since C does not allow such array indices, it is necessary to refer in the program below to corresponding entries in arrays whose indices will range from 0 to n - 1, 0 to 2(n - 1), and 0 to 2(n - 1), respectively.

Finally, we must implement the printing of a solution in process . This can be done by printing the sequence of values stored on the stack. They specify the rows in which queens 1 to n were placed. To do this we will need a stack operation item. It returns a copy of the ith stack entry, which represents the row in which the (i+1)th placed queen (the queen in column i) appears. When i is zero, it returns a copy of the top entry.

The backtracking traversal algorithm that was developed and applied to this problem is a general tool for problem solving. It may be adapted, for example, to solving a maze, finding good game-playing strategies, and translating high-level language programs.

The nonrecursive solution may be written as follows.

Nonrecursive n-queens Program

# include <stdio.h>

main()

/* Reads in n and prints all solutions

to the n-queens problem.

*/

{

int n;

printf("\n n = ?");

scanf("%d",&n);

queens(n);

}

    >Comment

#define NLIMIT 20

typedef int rowcheck[NLIMIT];

typedef int diagonalcheck1[2*NLIMIT-1];

typedef int diagonalcheck2[2*NLIMIT-1];

    >Comment

#define SLIMIT 20

typedef int whatever;

typedef struct

{

whatever stackarray[SLIMIT--1];

int top;

}stack;

setstack(ps)

/* Sets stack s to empty. */

stack *ps;

{

(*ps).top = --1;

}

empty(ps)

/* Returns true only if stack s is empty. */

stack *ps;

{

return((*ps).top == -1);

}

push(value,ps)

/* Inserts contents of value as

the top entry on stack s.

/*

whatever value;

stack *ps;

{

if((*ps).top == (SLIMIT-1))

overflow(ps);

else

{

(*ps).top = (*ps).top + 1;

(*ps).stackarray[(*ps).top] = value;

}

}

pop(ps,pvalue)

/* Removes the top entry of stack s

and copies its contents into value.

*/

stack *ps;

whatever *pvalue;

{

if(empty(ps))

underflow(ps);

else

{

*pvalue = (*ps).stackarray[(*ps).top];

(*ps).top = (*ps).top - 1;

}

}

whatever item(i,ps)

/* Returns a copy of the ith

entry in stack s. When i is

zero it returns a copy of

the top entry.

*/

stack *ps;

{

return((*ps).stackarray[(*ps).top-i]);

}

overflow(ps)

/* Prints a message if the stack overflows. */

stack *ps;

{

printf("\n stack overflow ");

}

underflow(ps)

/* Prints a message if the stack underflows. */

stack *ps;

{

printf("\n stack underflow ");

}

queens(n)

1 a backtracking preorder traversal

/* Prints all solutions to the n-queens problem */

int n;

{

stack s;

#define TRUE 1

#define FALSE 0

int i,row,col,found;

rowcheck r;

3 r, d1, and d2 will contain the board configuration

diagonalcheck1 d1;

diagonalcheck2 d2;

col = 1;

2 col and row determine the next placement attempt

row = 1;

    >Comment

for(i=0;i<=n-1;i++)

r[i] = TRUE;

for(i=0;i<=2*n-2;i++)

d1[i] = TRUE;

for(i=0;i<=2*n-2;i++)

d2[i] = TRUE;

found = FALSE;

setstack(&s);

    >Comment

while(((col < n+1)&&(row < n+1)) || !empty(&s))

    >Comment

if((col < n+1)&&(row < n+1))

    >Comment

if(feasible(row,col,r,d1,d2,n))

{

    >Comment

process(row,col,&found,&s,n);

    >Comment

push(row,&s);

    >Comment

r[row-1] = FALSE;

d1[row+col-2] = FALSE;

d2[row-col+n-1] = FALSE;

    >Comment

col = col + 1;

    >Comment

row = 1;

}

    >Comment

else

    >Comment

row = row + 1;

    >Comment

else

{

    >Comment

pop(&s,&row);

col = col - 1;

    >Comment

r[row-1] = TRUE

d1[row+col-2] = TRUE;

d2[row-col+n-1] = TRUE;

    >Comment

row = row + 1;

}

if(!found)

printf("\n NO SOLUTIONS");

}

feasible(row,col,r,d1,d2,n)

/* Returns true only if the placement of the

next queen in position col,row does not

allow any queen to take another under

the current board configuration given

by r,d1,d2.

*/

int row,col, n;

rowcheck r;

diagonalcheck1 d1;

diagonalcheck2 d2;

{

return(r[row-1]&&d1[row+col-2]&&d2[row-col+n-1]);

}

process(row,col,pfound,ps,n)

/* If the partial solution is a complete

solution then it is printed and

found is set to true.

*/

int row,col,*pfound,n;

stack *ps;

{

int i;

whatever item();

if(col == n)

{

for(i=n-1;i>=1;i--)

    >Comment

printf("\n COL %d ROW is %d----",

n-1,item(i-1,ps));

    >Comment

printf("\n COL %d ROW is %d----\n",

n,row);

*pfound = TRUE;

}

}

You may be tempted to try a more analytic approach to this problem so as to eliminate the need for all this searching. Be forewarned that famous mathematicians also attempted to analyze this problem but had difficulty solving it other than by backtracking! Actually, there are no solutions for n = 1, 2, and 3, and at least one solution for all n  [image] 4. For the curious, the number of solutions for n up to 15 are:

    N      4      5          6    7      8       9         10

Solutions  2      10         4    40    92      352        724

    N        11        12         13        14          15

Solutions  2,680     14,200     73,712    365,596    2,279,184

7.6.4 Breadth-first

One final point about the backtracking traversal algorithm. Suppose the while loop task is replaced by the task:

if p is not null, then

if p is feasible, then

process(p)

push all successors of node.p onto the stack

pop(sp)

else

pop(s,p).

This implementation also represents a preorder tree traversal. The stack is used to store and recall postponed obligations. It can be further modified by changing the stack to a queue and replacing empty, push, and pop by the corresponding queue operations. Pop must return null if called when the queue is empty. The result is another modified general tree traversal, a backtracking breadth-first traversal. It accesses the nodes in level order. In contrast, the backtracking preorder traversal presented earlier might be called a backtracking depth-first traversal. Removing the "p is feasible" test of the backtracking breadth-first traversal turns it into a breadth-first traversal. You should convince yourself that this version leads to the nodes of Figure 7.13(a) being accessed and processed in the order A, B, C, D, E, F, G, H, I, J, K, L, M. Keep in mind that in this application the root node was never considered. The difference between a backtracking traversal and a traversal is that backtracking tests for feasibility. This allows tree paths to be ignored when it can be determined that they cannot lead to a solution.

7.6.5 Branch and Bound

There is another type of modified tree traversal which is often found useful. It is called a branch and bound traversal. An algorithm for this traversal is as follows:

Nonrecursive Branch and Bound Traversal

1.     Set p to t.

2.     Set bag to empty.

3.     While p is not null

if p is feasible, then

i.     process(t,p), and

ii.     add each successor pointer of p to the bag;

Select a pointer from the bag and set p to it.

This algorithm uses a data abstraction consisting of a bag data structure and operations on the bag that set it to empty, test it for emptiness, select (and remove) a pointer from it, and add a pointer to it. The idea is to define the select operation so that it removes the pointer from the bag that points to the node most likely to lead to a solution. The execution time required by the branch and bound traversal is directly related to how well the select function does the job of picking the next node to process and how well the feasible function does its job of pruning the tree. The better the selection and pruning the better the execution time. The select function should return null if the bag is empty.

If the select function always selects the node most recently added to the bag, then the data abstraction becomes a stack, and the algorithm is turned into a backtracking depth-first traversal. If the select function always selects the node that was added earliest to the bag, then the data abstraction becomes a queue, and the algorithm is turned into a backtracking breadth-first traversal. Thus backtracking may be viewed as a special case of branch and bound.

Horowitz and Sakni [1978] contains extensive discussions of backtracking and branch and bound traversals, with many applications worked out in detail. Also see Golomb and Baumert [1965]. Wirth [1976] deals with these techniques for problem solving; in particular, the stable marriage problem and the eight-queens problems are solved recursively.

7.7: More on Recursion, Trees, and Stacks

Now that we are steeped in trees, we can look back at Figures 4.2 and 4.4 and notice that they are pictures of trees. We shall call them the execution trees of their corresponding recursive programs. In fact, any recursive program can be viewed as generating such a tree as it executes. The recursive program can be thought of as executing by traversing its corresponding execution tree. The processing done at each node in the course of this traversal is the carrying out of the program's task on a specific problem specified by its data and current scratchpad. Now that you know a good deal about tree traversals, is it any wonder that the stack appears in the translation of a recursive program? It is clearly the number of nodes in the tree that determines the execution time of the algorithm, and it is the depth of the tree that determines the storage requirements. This point can become somewhat muddled when recursion is applied to trees themselves, but you should realize that this is the case even when no trees are apparent in the nature of the problem, as in the Towers of Hanoi problem in Chapter 4.

Further, we can now see that recursion clarifies because it hides the bookkeeping required by the tree traversal and because it elucidates the structure that must be inherent in the problem in order for a recursive solution to be found in the first place. Although it is an extremely powerful tool for these reasons, it must be applied with care. The purpose of this section is to provide some insight into the nature of those problems where a more skeptical approach is wise.

(a) Balanced

(b) Unbalanced

Figure 7.18 Balanced and Unbalanced Binary Trees

7.7.1 Balanced Binary Trees

In a balanced binary tree, no matter what node of the tree is considered, the left and right subtrees of that node have depths that differ by at most 1.

The binary tree of Figure 7.18(a) is balanced; the binary tree of (b) is not. Balanced trees are important in information retrieval applications; they will be discussed fully in Chapter 9. In information retrieval, the data are stored at the tree's nodes. The depth of the tree determines the search times for retrieval and, with balanced trees, the tree's depth is controllable and can be limited, as will be shown in Chapter 9. Here we consider the "worst case" of such trees.

A binary tree is a Fibonacci tree if it is balanced and has the minimum number of nodes among all balanced binary trees with its depth. Fibonacci trees represent the worst case for balanced trees, since they contain the fewest nodes (and hence the least storage capacity) among all balanced trees with a specified depth. The binary tree in Figure 7.19 is a Fibonacci tree of depth 4. This is a Fibonacci tree because it is balanced, and no balanced binary tree with a depth of 4 has fewer than 7 nodes. Any complete binary tree of depth 4, for example, will not be a Fibonacci tree because, although balanced, it will have more than the minimum number of nodes (7) for this depth.

Figure 7.19 A Fibonacci Tree of Depth 4

The Fibonacci numbers are a well-known sequence of integers defined by the following recursive definition:

F0 = 0,   F1 = 1,

Fn = Fn-1 + Fn-2    for n > 1

The first few are 0, 1, 1, 2, 3, 5, 8, 13, 21, 34, . . . . These numbers seem to appear in the most unexpected places. In computer science they often arise in the analysis of algorithms. The number of nodes in a Fibonacci tree of depth d can be shown to be Fd+2 - 1 (see Exercise 34b). For example, if d is 4, then Fd+2 - 1 = F6 - 1 = 8 - 1 = 7.

Example 7.6

The task now is to develop an algorithm to generate a Fibonacci tree of depth d. If possible, apply the method of recursion to find a recursive definition of the solution.

Clearly, if d = 0, there is only one Fibonacci tree of depth d, the null binary tree. If d = 1, there is only one Fibonacci tree of depth d, the tree with one node--its root.

Any subtree of a balanced binary tree must also be balanced, or else the tree itself would not be balanced. Any subtree of a Fibonacci tree must therefore be balanced and must also be a Fibonacci tree. Otherwise it could be replaced by a Fibonacci tree of that depth with fewer nodes. This is the key to the recursive solution. We can now construct a Fibonacci tree of depth d > 1 as follows: Since d > 1, the Fibonacci tree must have a root, and its two subtrees must be Fibonacci trees. One (say, the right) must have depth d - 1 (so the tree itself has depth d). The other, the left subtree, must be a Fibonacci tree and must differ in depth from d - 1 by at most 1. Hence it must have depth d - 2 or d - 1. It has just been demonstrated that a Fibonacci tree of depth d - 1 has at least one subtree of depth d - 2. Hence the Fibonacci tree of depth d - 1 has more nodes than the Fibonacci tree of depth d - 2. Thus the left subtree of the Fibonacci tree being constructed must have depth d - 2. (Why?) This gives the complete recursive solution.

To construct a Fibonacci tree of depth d:

1.     If d = 0, then construct the null tree,

2.     else if d = 1, then construct the tree with one node, its root,

3.     else if d > 1, then

a.     construct a root,

b.     construct a Fibonacci tree of depth d - 2 and make it the left subtree of the root, and

c.     construct a Fibonacci tree of depth d - 1 and make it the right subtree of the root.

The explicit constructions are given by the algorithm's statements 1 and 2 for d = 0 and d = 1. Statement 3 gives the implicit construction for d > 1. You should apply this definition to the case d = 4, and confirm that it constructs the example given earlier of a Fibonacci tree of depth 4. It should also be clear that other Fibonacci trees of depth 4 exist.

The recursive program can be written directly from the definition.

Recursive Version 1--Linked Representation with Records Stored in an Array

fibonaccitree (d, ptree)

/* Creates a Fibonacci tree, tree, of depth d. */

int d;

binarytreepointer *ptree;

{

binarytreepointer tl, tr, avail ( );

    >Comment

*ptree = avail ( );

if (d == 0)

    >Comment

*ptree = NULL;

else if (d == 1)

{

    >Comment

records [*ptree].leftptr = NULL;

records [*ptree].rightptr = NULL;

}

else

{

    >Comment

fibonaccitree (d-2,&tl);

records [*ptree].leftptr = tl;

fibonaccitree (d-1,&tr);

records [*ptree].rightptr = tr;

}

}

For concreteness, we have chosen to implement the constructed tree using a linked representation. The tree itself will be stored in the records array (Section 7.4.2) with the fields of a record being info, leftptr, and rightptr.Records is assumed to be a global variable. When the routine returns, tree will point to its root record in the array representation.

Using implementation with pointer variables for the binary tree, the procedure becomes the following.

Recursive Version 2--Linked Representation with Records Stored in Dynamic Memory

fibonaccitree(d,ptree)

/* Creates a Fibonacci tree, tree, of depth d. */

int d;

binarytreepointer *ptree;

{

binarytreepointer tl, tr;

    >Comment

*ptree = malloc (sizeof(binarytreerecord));

if (d == 0)

    >Comment

*ptree = NULL;

else if (d == 1)

{

    >Comment

(*ptree)->leftptr = NULL;

(*ptree)->rightptr = NULL;

}

else

{

    >Comment

fibonaccitree (d-2, &tl);

(*ptree)->leftptr = tl;

fibonaccitree (d-1, &tr);

(*ptree)->rightptr = tr;

}

}

Finally, for contrast, the function may be written treating the tree as a data abstraction, as follows.

Recursive Version 3--The Tree Is Treated as a Data Abstraction

fibonaccitree(d,ptree)

/* Creates a Fibonacci tree, tree, of depth d. */

int d;

binarytreepointer *ptree;

{

binarytreepointer null,tl,tr,avail( ),setnull( );

null = setnull( );

*ptree = avail( );

if (d == 0)

    >Comment

*ptree = null;

else if (d == 1)

{

    >Comment

setleft(*ptree, null);

setright(*ptree, null);

}

else

{

    >Comment

fibonaccitree(d-2,&tl);

setleft(*ptree, tl);

fibonaccitree(d-1,&tr);

setleft(*ptree, tr);

}

}

7.7.2 Trading Storage for Time

Consider the execution tree for these functions when d = 6, which is given in Figure 7.20. Each node of the tree represents a call to the recursive function Fibonaccitree. Nodes with the same d-value thus correspond to requests of the recursive function to carry out identical tasks, that is, to create identical trees.

Figure 7.20 Execution Tree for Depth d = 6

Figure 7.20 shows a rather healthy-looking (nonsparse) tree with depth d = 6. This holds true for any value of d and means that while the storage is not excessive, the number of nodes in the tree will be O(  [image] d), where   [image]  = (1 +   [image])/2.* This is because

The execution tree for Fibonaccitree will always "resemble" the tree it is constructing.

The depth of an execution tree determines the size of the underlying stack needed for the program to execute properly.

The number of nodes in a Fibonacci tree of depth d is Fd+2 - 1.

Hence the execution time will increase exponentially with d. A close inspection reveals that many identical tasks or problems are solved repeatedly during its execution. For example, d = 0, 1, 2, 3, and 4 are solved 5, 8, 5, 3, and 2 times, respectively. This is what causes the excessive execution time and illustrates the situation to be wary of. When the same problems occur repeatedly, the efficiency can be increased if the algorithm can be modified to save the results for these problems so they may be used directly when needed instead of being regenerated by recursive calls. Of course, if this requires excessive storage, then such an approach won't do. Excessive storage is needed when an excessive number of distinct problems must be solved and their results saved. Still, this represents a viable strategy when applicable, as is the case here.

* In fact, the number of nodes will be 2Fd+1 - 1.

Notice that a Fibonacci tree of depth d   [image]  2 can be constructed even if only Fibonacci trees of depth d - 2 and d - 1 are known. All that is necessary is to create a root node and append the tree of depth d - 2 as its left subtree and the tree of depth d - 1 as its right subtree. Consequently, the Fibonacci tree of depth d can be obtained by constructing the trees for depth 0, 1, 2 . . . , d in turn, while retaining only the lastest two. This gives the nonrecursive solution:

An Efficient Nonrecursive Version

If d = 0, then

set t to null;

else if d = 1, then

a.     create a root node

b.     set t to point to the root node;

else

c.     set tl to null,

d.     create a root node,

e.     set tr to point to the root node, and

f.     for k from 2 to d,

i.     create a root node with tl its left subtree and tr its right subtree

ii.     set t to point to the root node

iii.     set tl to tr and tr to t.

This solution is efficient in both execution time and storage (see Exercises 16 and 17 of Chapter 4 for a related example).

Although it was possible to cut down drastically on the number of Fibonacci trees retained in this solution, such a saving not always possible. Sometimes solutions to all the problems generated by recursive calls will need to be retained. Even so, when this number is not excessive and an appropriate data structure can be found in which to store and efficiently access a solution when needed, then execution time can be improved. This is another instance of what we have frequently seen: time may be traded for storage.

It is hoped that you do not feel we have been barking up the wrong trees! Perhaps you even agree with the humorist Ogden Nash:

I think that I shall never see

A billboard lovely as a tree.

Indeed, unless the billboards fall

I'll never see a tree at all.

Exercises

1.     a.     What is the minimum number of nodes that a binary tree of depth d can have?

b.     What is the maximum number of nodes that a binary tree of depth d can have?

2.     Suppose t is a binary tree with n internal nodes and m terminal nodes. T is a full binary tree if each of its nodes has zero or two successors. What is the relation between m and n when t is a fully binary tree?

3.     There are 0, 1, 2, and 5 distinct binary trees with, respectively, 0, 1, 2, and 3 nodes.

a.     how many distinct binary trees are there with 4 nodes?

b.     Can you find a formula for the number of distinct binary trees with n nodes? This formula bt(n) may be expressed in terms of bt(k) for k < n.

4.     Modify the program segments of Section 7.2.1 so that they work correctly in all cases.

5.     a.     Suppose the binary tree that follows is preorder, inorder, and postorder traversed. List the order in which the nodes will be accessed for each.

b.     Do you notice anything striking about the inorder listing?

6.     Prove that all terminal nodes will always be accessed in the same relative order for all three travelsals.

7.     a.     Suppose the stack used in a pre-, in-, or postorder traversal is implemented as an array. What is the requirement for limit in terms of the depth of the binary tree, to guarantee that the stack is large enough for the traversal to work?

b.     What is the necessary value for limit in order to guarantee that the traversal will work with the stack, for any binary tree with n nodes?

8.     Suppose the following two binary trees are preorder and postorder traversed. The preorder and postorder listings are given for each tree.

Both binary trees have identical listings. This cannot happen with preorder and inorder listings. That is, we cannot find two distinct binary trees that have identical preorder and inorder listings. The same is true for postorder and inorder listings. Find an algorithm that will create the unique binary tree that is thus determined by its preorder and inorder listings.

9.     a.     Can you find a way to use the left and right pointer fields of terminal and internal nodes that contain null pointers in a binary tree to create "threaded binary trees" so that they can be traversed without using a stack? (Hint: Consider pointers that point to the predecessor and successor of a terminal node in an inorder traversal.)

b.     What about for inorder and postorder traversals?

10.     Write inorder and postorder traversal routines (nonrecursive and recursive).

11.     a.     Suppose you wish to use a linked representation for binary trees with an array implementation as in Figure 7.10. Suppose tht input is given as a name value followed by a sequence of records, each record representing a node of a binary tree. For the binary tree fred of Figure 7.10, the input would be

 5

 8  L   0

-1  O  -1

 3  U   4

-1  S  -1

-1  T  -1

Three records appear in the input in the order in which they would be accessed by a preorder traversal. Assume the head value is read into fred, and preorder traverse is called. Write a process routine that will turn preorder traverse into a routine that reads the records of the binary tree into the leftptr, info, and rightptr fields properly.

b.     Same as Exercise 11(a), but assume the records are given in the order in which they would be accessed in a postorder traversal.

12.     The binary tree sample is stored using a linked representation. List the order in which its nodes will be accessed in a preorder traversal.

          leftptr  info  rightptr

           1    0    R       0

           2    3    W       4

           3    5    R       0

sample 7   4    0    C       0

           5    4    E       6

           6    0    T       0

           7   10    C       3

           8    6    A       3

           9    1    B       4

          10    1    O       0

13.     What will be in the stack during a preorder traversal of sample in Exercise 12 when its rightmost node is accessed? What about when it is inorder and postorder traversed?

14.     a.     Suppose every subtree of sample of Exercise 12 is interchanged with its right subtree. What will leftptr, info, and rightptr look like?

b.     What would leftptr, info, and rightptr look like if all the terminal nodes of sample were deleted in Exercise 12?

15.     Suppose a binary tree is stored using the records array implementation as in Figure 7.10. If tree points to its root node, write a function to print out the sequence of pointers that specify the path in the tree to the right most node in tree. For example, in Figure 7.10 the rightmost node of fred is 1, and the sequence of pointers to be printed out for fred would be 6, 1, 5.

16.     a.     If the binary tree of Figure 7.5 were represented sequentially, what numbers would be assigned to its nodes?

b.     What binary tree with n nodes will require the largest array for its sequential representation?

17.     When dealing with arbitrary binary trees, would you use linked or sequential representation? Why?

18.     a.     Write a function that will return, given a binary tree t, with a pointer to a terminal node of t containing the largest value in its information field.

b.     Write a function that, given a binary tree t and two pointers p and q to nodes of t, returns a value true if node.q is a successor of node.p and a value false otherwise.

c.     Write a function that does the same thing as Exercise 18(b), except that the function returns a value true if node.q is in the subtree of node.p and a value false otherwise.

19.     Consider the binary tree that was produced by the transformation from a general tree to a binary tree as a general tree, and find the binary tree produced by its transformation.

20.     Consider again the solution to the information retrieval problem of Chapter 6. Suppose you create a record, r, in memory. R will have a pointer field pointing to the record for a father. You can then think of r as a record representing the node of a binary tree.

a.     Describe this binary tree.

b.     What general tree would have r as its binary tree under the transformation of Section 7.5.1?

21.     a.     Write a function to append a binary tree t1 to a binary tree t2 by replacing some terminal node in t2 at least depth by the whole tree t1. For example, see the figure. Use a linked representation with pointer variables.

b.     What does your function do if t1 and t2 are the same tree?

c.     How will your solution change if a sequential representation is used?

(a) Input

(b) Output

22.     a.     Write a routine that deletes all the terminal nodes of a binary tree.

b.     Write a routine that deletes node.q if it is a successor of node.p in a binary tree.

23.     a.     Write a function to interchange all left and right subtrees of a binary tree.

b.     Create a function to interchange all left and right subtrees, except that the interchanged version of the original tree will be created as a new tree, leaving the original tree unchanged. Assume pointer variables are used for its implementation.

24.     a.     Modify the preorder traversal so that it returns, in a variable identical, a value true if t1 and t2 point to identical trees, and a value false otherwise. That is, the value is true if the two trees have the same structure and the same information field values.

b.     Do the same task, except that the binary trees need have only the same structure.

25.     a.     Suppose you are given a maze with the structure of a binary tree. The maze has a starting point and an exit. A node is associated with the starting point, with the exit, and with every point in the maze where one of two paths must be taken. The maze is given as a two-dimensional array a. A [i,j] is 1 if there is a path from node i to node j in the maze, and is 0 otherwise. There are n points in the maze. Adapt the preorder traversal to achieve an algorithm for solving the maze--that is, for printing out the path from the starting node to the exit node. Your solution should be detailed enough to write a function corresponding to it directly.

b.     If the maze has loops, what modifications must be made in your solution so that the algorithm works correctly?

26.     a.     Modify the preorder traversal algorithm for creating a binary tree from a general tree (p. 332) and the process algorithm so the preorder traversal returns, in the variable transform, a value "true" only if a binary tree bt is the transformed version of a general tree t.

b.     Modify the preorder traversal algorithm so that it produces the transformed tree by processing the nodes of t in breadth-first order. (Hint: Consider the queue.)

27.     Write the function create used in the process algorithm.

28.     Modify the backtracking preorder traversal algorithm so that it does a backtracking breadth-first traversal.

29.     Suppose you are given a "map" of n cities and the distances between them. Can you find a way to modify the preorder traversal algorithm to determine the shortest distance between two cities? What is a feasible test to use to allow feasible to limit the execution time of the solution?

30.     a.     Suppose a general tree is given as follows:

(A(B(E, F, G), C(H), D(I(K, L, M), J)))

This corresponds to the general tree:

Suppose that input is given by such a parenthesized expression. Can you find an algorithm that will create the general tree from it? Assume that the general tree will be represented in memory as a collection of lists of successors, with one successor list for each node.

b.     What is the connection between parenthesized expressions and a listing of nodes in the order in which they are accessed by a preorder traversal?

31.     T is a binary tree. Result(t) is defined by

Left(t) and right(t) are, respectively, the left and right subtrees of t.

a.     If t is the binary tree that follows, then what is result(t)?

b.     What does result do?

32.     T1 and t2 point to binary trees represented using pointer variables.

check (t1,t2)

binarytreepointer t1,t2;

{

binarytreepointer null,setnull();

null = setnull();

if(((t1 == null) && (t2 == null)) || ((t1 != null)

&& (t2 != null)))

return(check(t1->leftptr,t2->leftptr)

&& check(t1->rightptr,t2->rightptr));

else

return (FALSE);

}

a.     Find the function value if it is invoked for trees t1 and t2.

b.     What does check do?

33.     a.     Find b(4) where

b(0) = 1

b(n) = b(0) X b(n - 1) + b(1) X b(n - 2) + b(2)b(n - 3) + . . .

+ b(n - 1) x b(0) n > 0

b.     Find a connection between the number of distinct binary trees with n nodes and b(n).

34.     a.     Give a recursive definition of balanced binary trees.

b.     Find a recursive definition for F(d), the number of nodes of a Fibonacci tree of depth d.

35.     Write a recursive function result corresponding to Exercise 31.

36.     a.     Write a recursive function count to return the number of terminal nodes of a binary tree.

b.     Write a recursive function terminal to delete all terminal nodes of a binary tree using pointer variables.

37.     Write a recursive routine partialexchange to interchange all subtrees of a binary tree whose left subtree info field value exceeds the right subtree value. Assume the binary tree is represented using pointer variables.

38.     Write a recursive function using pointers to transform a general tree into its binary tree representation.

39.     Write a recursive program that produces a solution, if one exists, to the n-queens problem.

40.     Write a recursive function to produce a copy of a binary tree. Both trees should be implemented using pointers.

Suggested Assignments

1.     Write a function to print the family tree for an individual whose family history is stored in the data base of the case study of Chapter 6. The individual is at the root of the family tree, and its subtrees are the family trees of all the individual's children. Print the tree using indentation as in Figure 7.1(a). Assume individual records are stored in dynamic memory and that the appropriate functions are available for searching and accessing the nametables. If this function is to also be executed, then this would be a good assignment for a group project.

     Also discuss the connection between the lists used in the case study and the binary tree representation of a general tree.

2.     Consider the problem of finding a stable pairing, given preferences, as discussed in Chapter 2. Describe a tree that represents all possible pairings, and write a backtracking traversal function to produce all stable pairings. Base your solution on the backtracking preorder traversal algorithm given in the text. Compare this solution with the solution given in Chapter 2.

3.     Formulate a solution to the stable pairing problem as a traversal through all permutations of the men. Use the permutation traversal program of Chapter 4 as the basis for your algorithm. Compare this solution to your solution to Assignment 2 and also to the solution given in Chapter 2.

4.     Write an efficient recursive function fibonaccicheck(t,d,flag), which is to return in d the depth of the binary tree t, and with flag true if t is a Fibonacci tree and false otherwise.

Go to Chapter 8      Return to Table of Contents


Data Structures Using C: CHAPTER 8: INTRODUCTION TO SEARCHING AND SORT

 Return to Table of Contents

CHAPTER 8: INTRODUCTION TO SEARCHING AND SORTING

Considers how collections of information may be stored to allow

efficient searches

efficient traversals

Develops efficient in-memory sorting algorithms

to illustrate the principles of good design in algorithm development

to show how sorting is used to facilitate searching

Worst-case and average times are developed for

linear search

binary search

maximum entry sort

bubble sort

insertion sort

heapsort

quicksort

Illustrates the use of simulation in analyzing the execution time of a program

8.1: Overview

So far in this book, arrays, lists, binary trees, and trees--the basic data sructures--have been introduced and applied to a variety of problems. Selecting the appropriate data structure requires knowing what operations are to be performed on it. The frequency with which these operations are to be performed, the characteristics of the data to be stored, and the average and worst-case time and storage requirements all combine to determine this choice. Other considerations, such as clarity and adaptability, may also serve as criteria for deciding which data structures to use. It is rarely possible to pick a data structure that is best for all criteria. Instead, trade-offs must be made.

In the case study of family relationships in Section 6.5, one of the important operations was a search of the nametable data base. At that time no decisions were made as to how the nametable data base would be structured to facilitate the search. This chapter and the next will show that manyimplementations are possible and that the appropriate choice is determined by the kinds of operations to be performed on the data base. Nametable might be organized in many ways. It can be implemented as an array, a list, a binary search tree, a balanced binary search tree, or a hash table. The techniques used to create and search these structures are introduced in this and the next chapter.

Searching and sorting masses of data to retrieve specific information and organize it for manipulation and presentation are nothing new. These basic operations were performed and studied before the advent of computers.Volumes have been written on these topics, yet they are still being researched. Even the general public and the press are evidently interested in searching, as you can see from the following letter and response:

Dear Ann Landers:

Why is it that whenever I lose anything, it is always

in the last place I look?

--Dizzy Lizzy

Dear Liz:

Because when you find it you stop looking--and

that's the last place you looked.

Although searching and sorting may appear mundane, the speed with which they can be carried out largely determines whether a particular application of the computer is practical (and "cost-effective'") or not. Estimates indicate that about one-fourth of the processing time in computer centers is spent just sorting. The relation between sorting and searching will become apparent in this chapter. If one can be done faster, so can the other. The choice of data structures is always important in algorithm design and is the key to the evolution of good (that is, efficient) searching and sorting procedures. The basic data structures studied in the preceding chapters (arrays, lists, and binary trees) are used in the design of efficient procedures.

Normally, the objects stored, searched for, and sorted are records. In sorting, one field of the records, called the sort key, is chosen as the basis for the sort. Thus, in a payroll application, an employee name field might be the key for an alphabetical ordering, while a salary field might serve as the basis for a sort to determine an ordering by amount of earnings. At times the sort key can be a combination of fields. As an example, we may desire to sort with name as the primary key and zip code as the secondary key. For simplicity it will be assumed in this chapter that the objects being sorted are integer numbers. However, any variable type could have been assumed. For sorting purposes, the only requirement is that, given any two objects, it is possible to tell which precedes the other.

Searching normally involves comparing records to a given search key value. The goal is to determine whether there exists among the records of the data base a record with key equal to that of the search key. If the key value can be matched, then a pointer to the record with the matching value must be returned. The basic principle for shortening search times is to organize the data base so that the search can be focused quickly on the part of the data base that will contain the desired record, if it is present.

The chapter first explores simple linear, binary, and interpolation searches and a number of elementary sorting algorithms, such as the bubble sort and the insertion sort. These lead to more advanced methods for sorting based on a special binary tree called a heap. Next, an application of recursion yields another fast sort, quicksort.

Some of the algorithms developed in the text are fairly complex and difficult, if not impossible, to analyze mathematically. For this reason the last topic considered in the chapter is the use of simulation as a tool in discovering the behavior and efficiency of such complex algorithms.

The chapter concludes with a summary of the strengths and weaknesses of each approach introduced. Knuth [1973b] and Wirth [1976] discuss many searching and sorting algorithms that are not dealt with in this text. Knuth also contains mathematical analyses of many of these algorithms.

8.2: Elementary Searches

Three simple searches are applicable when records are stored in arrays: (1) linear search, (2) binary search, and (3) interpolation search. You probably have all used these methods in everyday activities. Understanding their limitations and how they work will help you see the need for the more complex structures and algorithms of later sections.

8.2.1 Linear Search

A linear search is so named because it proceeds in sequence, linearly through an array. Suppose data in Figure 8.1 is an array of integers.* The task is to determine whether the number stored in the variable key is in the array, and if so, what its index is.

* To simplify the explanation in this chapter, the 0th array position will be ignored, and data stored in arrays will start in position 1 unless otherwise noted.

Figure 8.1 An Array of Integers to Be Searched for the Key Value 25

One obvious method would be to traverse the data array, checking each element of data to see if it contains the same value as key. If an element currently being processed did contain the same value as key, its index would simply be returned, and the search would be complete. Arriving at the end of data without having found the key value would mean that no element with that value is stored in the array. This procedure is easily implemented using a loop structure (as in the following function). The worst-case time required for its execution will be of the order n, written O(n), where n is the length of the data array. In particular, if the value being sought is in element i of the array, it will take O(i) time to find it. The linear search is implemented as follows:

#define MAX 100

typedef struct

{

int key;

}record;

typedef record recordarray[MAX];

    >Comment

linearsearch(data,n,key,pfound,ploc)

/* Searches the n records stored in

the array data for the search key.

Found will be true only if a record

is present with key field equal to

the search key value and then loc

will contain its array index.

*/

recordarray data;

int n,key,*pfound,*ploc;

{

    >Comment

*ploc = 1;

    >Comment

while ((*ploc <= n) && (data[*ploc].key != key))

(*ploc)++;

    >Comment

*pfound = (*ploc < n + 1);

}

A special but common case is when the search is always for a key known to be stored in the array, and each key stored is distinct and equally likely to be the search key. This is like standing in front of a locked door with a bunch of keys. You need to find the key that opens the door from among n keys in arbitrary order on your key-ring. The average time for such a search would be O(n/2). That is, on the average, you would look at half the keys, or half the elements of the array, before finding the right one.

Thus the procedure called linear search is so named, not only because it proceeds linearly through an array, but also because its processing time increases linearly with n; when n is doubled, the time required to complete the search is doubled.

8.2.2 Saving Time: A Neat Trick

Occasionally programming tricks can be used to enhance efficiency. In this case paying attention to some details produces a trick that can pay off by saving processing time. The straightforward implementation of the linear search requires a test (loc   [image]  n) for the end of the array before checking the next element. This is true whether the implementation is a for loop or a while loop. However, it is possible to avoid performing this test.

Place the search key in element n + 1 of data. Then, in implementing the linear search, there is no need to test for the end of the array before checking the next element. Since the array is now considered to be (n + 1) elements in length, we are certain to find the search key before we "fall off the end." After the loop is exited, note the element in which key was found, and test to see if it was the (n + 1)th element, which means that the search was unsuccessful. Thus only one test (data[loc].key ! = key) must be performed. This saves n tests in unsuccessful searches, and i tests in successful searches, when the search key appears in element i of data. This simple modification of the basic linear search can easily produce significant time savings (20 to 50 percent). In general, whenever loops are modified to reduce the tests or processing they perform, execution time is saved each time through the loop. The more efficient linear search implementation appears as follows:

#define MAX 100

typedef struct

{

int key;

}record;

typedef record recordarray[MAX];

    >Comment

linearsearch(data,n,key,pfound,ploc)

/* Searches the n records stored in

the array data for the search key.

Found will be true only if a record

is present with key field equal to

the search key value and then loc

will contain its array index.

*/

recordarray data;

int n,key,*pfound,*ploc;

{

*ploc = 1;

    >Comment

data[n + 1].key = key;

    >Comment

while (data[*ploc].key != key)

(*ploc)++;

*pfound = (*ploc < n + 1);

}

Another way to save time in a linear search is to order the records in the array. One way to do this is to store them in decreasing order of key value. The linear search can then be modified by terminating the search as soon as a record is reached whose key value is less than the search key. The desired record cannot be in the array past that point. Although this results in extra comparisons for successful searches, time is saved overall when unsuccessful searches occur frequently enough.

It is also possible, if the frequency with which records will be retrieved is known, to store the records in decreasing order of retrieval frequency. The most frequently retrieved record will then be at the top of the array. This approach significantly improves search efficiency when the desired record is likely to be near the top of the array. In fact, if the frequencies decrease rapidly enough, the average search time for the linear search will be a small constant.

8.2.3 Binary Search

Certainly no one does a linear search to look for a phone number in the telephone directory. Imagine trying to do so in the directory for New York City or any other large city! But there are ways to carry out the search so that the name is found in a few seconds, even from among a few million names. The structure of the telephone directory is the clue to the solution, which takes advantage of the alphabetical ordering of the entries in the phone book.

Assume that the numbers stored in data are arranged in decreasing order, as in Figure 8.2. They are said to be sorted in decreasing order. Since a telephone book is sorted in alphabetical order, one of the things most people do by habit when using the telephone directory is to flip the directory open toward the front, middle, or back depending on the location in the alphabet of the name being sought. If the page opened to has names farther down in the alphabet, the search name cannot be on that page or anywhere in the directory to its right. It can only be to the left of the selected page. This process of elimination is behind the search technique called binary search. It is not difficult to see why this search technique is called binary. Every time a test is made in the search, there are two choices. In the telephone book example, search either half of the remaining pages for the name.

A binary search is also a good strategy for winning the game of twenty questions, which requires guessing the identity of an object after finding out if it is in the animal, the mineral, or the vegetable category. Twenty question with "yes" or "no" answers are allowed. Selecting each question so that about half the remaining possibilities are eliminated, no matter what the answer, is best. (For example, given the animal category, "Is the object male or female"?) In this way 220, or over a million, objects can be distinguished.

In twenty questions it is difficult to select a question that makes close to a 50-50 split of the remaining possibilities. However, if you are asked to determine a number known to lie between 1 and 1024 by asking yes-no questions, it is easy to do so. Ask, "Is it greater than 512?" and continue splitting the remaining possibilities in half. No more than 10 such questions are needed to find the number from among the 210 (1024) initial possibilities. Similarly, when searching for a specific key value among 2n records, at most n key comparisons are needed to determine whether it is present, as long as each comparison splits the remaining collection of records into two parts that differ in size by at most one.

Figure 8.2 Array of Integers Sorted in Decreasing Order

When searching data stored in arbitrary order in an array, there is no reason to expect the search key to be in any particular region of the array. However, when the data have been stored in sorted order, the situation is different. It is possible to check the middle element to see if the search key is there. If it is, the program would note its location and terminate the search. If the key is not there, then if the search key is greater than the key in the middle element, eliminate the middle element and all elements below it from consideration. If the desired record is in data at all, it must be above the middle element. Similarly, if the search key is less than the key in the middle element, eliminate the middle element and all elements above it. In any event, at most half the elements remain to be considered, and the same procedure can then be applied to the appropriate half: either elements 1 to [mid-1] or elements [mid+1] to n. Mid is a variable that points to the middle element.

For example, if mid is 10, then the search key value, 25, is not greater than data[mid], 78. Hence, if it is in data at all, it must be between positions mid+1 and n--11 and 20. Taking 15 as the middle element of these, the new mid value is 15. Key is less than data [mid], 32. Hence we need to search between mid+1 and 20-16 and 20. With a new mid value of 18, key is greater than data [mid], 13. Now search only between 16 and 17. The next mid value is 16, and the key is found.

The basic structure of the binary search algorithm is a loop, as shown in the following function.

#define TRUE 1

#define FALSE 0

#define MAX 100

typedef struct

{

int key;

}record;

typedef record recorddarray[MAX];

binarysearch(data,n,key,pfound,ploc)

/* Searches the n records stored in order by

key field value(largest first) in the array

data for the search key. Found will be true

only if a record is present with key field

equal to the search key value and then loc

will contain its array index.

*/

recordarray data;

int n,key,*pfound,*ploc;

{

int top, mid,bottom;

    >Comment

top = 1;

bottom = n;

*pfound = FALSE;

*ploc = 0;

    >Comment

while ((top <= bottom) && !(*pfound))

{

    >Comment

mid = (top + bottom)/2

    >Comment

if (data[mid].key == key)

{

    >Comment

*pfound = TRUE

    >Comment

*ploc = mid;

}

    >Comment

else if (data[mid].key < key)

bottom = mid - 1;

    >Comment

else

top = mid + 1;

}

}

Mid is initialized to the middle element of data, top to 1, and bottom to n. Top and bottom are variables that point to the top and bottom elements of the consecutive elements left to be searched, all others having been eliminated from consideration. In the loop body, test for key equal to data [mid]. If the equality is true, the program exits the loop and the search is done, since key has been found at element mid. If not, top and bottom must be updated to reflect the new sequence of elements that are to be searched. This updating must set bottom to mid whenever key is greater than data [mid], and top to mid + 1 otherwise. The test for continuing the loop involves checking whether key was found, and whether any elements are left to be searched between top and bottom. The search is terminated as soon as the desired entry is found or as soon as it is concluded that the entry is not present. Loc will be zero if the key value was not found and will index its location in data if found.

8.2.4 Timing the Binary Search

How much time is required by the binary search algorithm in the worst case? Each pass through the loop requires at most a constant amount of work. The time to execute the algorithm will then be determined by the number of passes through the loop, plus, of course, a constant time for initialization and finalization. What is the greatest number of loop iterations executed?

In each pass through the loop, the search involves at most half the elements that were under consideration during the preceding pass through the loop. Starting with n elements, after one pass through the loop at most 1/2n elements are left, after two passes through the loop at most 1/2(1/2)n elements, after three passes at most 1/2(1/2)2n elements, and after k passes at most 1/2(1/2)k-1n or 1/2kn elements. This number, 1/2kn, is less than 1 when -k + 1g n < 0 or when k > 1g n. This means that the loop cannot be executed more than   [image] lg n  [image]  times. If n is 20, for example, then 1g 20 = 4.32, and   [image] 4.32  [image]  is 5. Thus in the worst case, the binary search is O(1gn). If each record stored is searched with equal frequency, then the average time to find a stored record is actually about (1g n) - 1.(The proof is not given here.)

8.2.5 Interpolation Search

Sometimes, telephone directories, like dictionaries and encyclopedias, have tabs so that users can flip open the book to the beginning of the section containing entries with a specific first letter and gauge whether to search toward the front, middle, or back pages of the section. The tabs have been inserted or interpolated within the data to guide the search. This represents a generalization of the binary search called the interpolation search. The topic will not be pursued in detail, but note that this technique allows the elimination of greater fractions of the remaining names than does a binary search. Not surprisingly, this tends to reduce the execution time.

8.2.6 Efficiency Comparisons

When n is large, the binary search takes much less time, in the worst case, than the linear search. This is because it makes 1g n rather than n comparisons, going through its loop body 1g n rather than n times. However, the steps performed in the loop body of the binary search are more complex and more time-consuming than the steps performed in the loop body for the linear search. Hence it is possible, for smaller values of n, that the linear search will be faster. Similarly, the interpolation search will need to perform more complex steps in its loop body, even though it may need to loop fewer times than the binary search.

Of course the binary and extrapolation searches require sorted data. When only a few searches are to be made or when n is small, it may not pay to spend time to sort the data first. For repeated searching or large n, the efficiencies of the binary and extrapolation searches easily overcome the cost of sorting.

8.3: Elementary Sorts

We now turn our attention to three simple sorting methods applicable to records stored in arrays: (1) maximum entry sort, (2) bubble sort, and (3) insertion sort. They are also applicable to records stored in lists. Understanding their limitations and how they work sets the stage for the more complex structures and algorithms of the next section.

8.3.1 Maximum Entry Sort

Perhaps the most straightforward way to sort numbers in an array into decreasing* order is to begin by placing the largest number in element 1. The next largest number is then placed into element 2, and so on. The maximum entry sort repeatedly finds the next largest array entry and places it in its proper position in the array. Suppose the point has been reached in this procedure where the k largest numbers have been properly placed into the first k elements of the array data. Assume a variable next points to the element of data into which the next largest element is to be placed. Next must now be updated to next + 1 so that it contains the value k + 1, as indicated in Figure 8.3.

* The sorting programs assume that decreasing order is desired. They can easily be modified to sort in increasing order.

Assume that a function maxloc exists, with parameters data, next, and n, the array length. Maxloc returns with its value indexing the largest element between the nextth and the nth elements of data. The sort may then be obtained by a loop in which maxloc is invoked (n - 1 times, with next taking on values 1, 2, . . . , n - 1, respectively, on each call. Within the loop, when each call to maxloc is made, an exchange between data [next] and data[maxloc] occurs.

Maxloc might be implemented by traversing the elements next to n of data. Each time a new element is accessed, it is compared to a variable, largest, containing the largest entry seen so far in this traversal. If the new element is greater than largest, largest is set to the new element value, and maxloc is set to point to the location of the new element in data.

This algorithm is probably similar to what most peole do instinctively when confronted with a small sorting problem. It may be implemented as in the procedure maxentrysort.

Figure 8.3 Ready to Determine the Next Largest

#define MAX 100

typedef struct

{

int key;

}record;

typedef record recordarray[MAX];

maxloc (data,next,n)

/* Returns the array index of the

largest entry in data between the

entries indexed by next and n.

*/

recordarray data;

int next,n;

{

int largest,i,tmaxloc;

tmaxloc = next;

largest = data[next].key;

for (i = next + 1; i <= n; i++)

    >Comment

if (data[i].key > largest)

{

tmaxloc = i;

largest = data[i].key;

return (tmaxloc);

}

}

maxentrysort (data, n)

/* Sorts the n records stored in array data

in descending order.

*/

recordarray data;

int n;

{

int next, loc;

for (next=1; next<=n - 1; next++)

{

loc = maxloc (data,next,n);

    >Comment

interchange (data, next, loc);

}

}

For the simple record used here for illustration, the interchange function is as follows.

interchange (data, i, j)

/* Interchanges the entries of

array data indexed by i and j.

*/

recordarray data;

int i,j;

{

record tempdata;

tempdata.key = data[i].key;

data[i].key = data[j].key;

data[j].key = tempdata.key;

}

How much time does this implementation of the maximum entry sort take? One interchange is made on each call to maxloc for a total of n - 1 interchanges. The number of comparisons is (n - k) on the call to maxloc, when next has the value k. The total number of comparisons is (n - 1) + (n - 2) + . . . + 2 + 1, which sums to [n(n - 1)]/2. Thus the time is determined by the time of the (n - 1) interchanges plus the time of the [n(n - 1)]/2 comparisons. Each interchange takes a constant amount of time, and each comparison takes, at most, a constant amount of time. The result is O(n2) time. This means that if n is doubled or tripled, the time will increase by a factor of about 22, or 4, and 32, or 9, respectively, when n is large. For n = 105, the time is roughly proportional to 1/2 1010, which is within a factor of 100 of 1013, the guide for feasibility presented in Chapter 1. Quicker algorithms are obviously needed for large n, and even perhaps for small n. The next elementary sort technique offers some improvement.

8.3.2 Bubble Sort

The bubble sort gets its name from the way smaller entries "sink" to the bottom of the array during the sort while larger entries "bubble up" to the top. To carry out the bubble sort on the array shown in Figure 8.4(a), traverse the array, comparing two adjacent elements, starting with the first two, and ending with the last two. If the [i + 1]th element value exceeds the ith, interchange the two element values.When a traversal is completed, it is called a pass through the array. Continue making passes until a pass occurs in which no interchanges are required.

In the example, after pass 1, the array looks like Figure 8.4(b). Since at least one interchange was made, another pass is made, yielding Figure 8.4(c). Again, at least one interchange was made, so the program must make another pass.

Will this process continue forever making passes, or eventually stop? Clearly, once it stops since no interchanges were made on the last pass, the array is sorted. If any number were out of order, at least one interchange would have been made during that pass.

Notice that after the first pass the smallest entry, 3, was at the bottom of the array. After the second pass the two smallest entries, 10 and 3, were in order at the bottom of the array. This was not accidental. Notice that the two smallest entries will never be moved, and the third smallest element (13 in this case) will be encountered on the next pass. Once it is encountered, it will always be interchanged with the adjacent lower record to which it is compared. It will eventually come to rest just above the second smallest entry.

(a) Initial Array

(b) After Pass 1

(c) After Pass 2

Figure 8.4 Bubble Sort

In general, after the kth pass, the k smallest entries will be in order in the bottom k elements of the array. Of course, more than the k smallest may be in order at this point, but only k can be guaranteed. Hence the array will be in sorted order after at most n - 1 passes. The procedure can therefore be modified to compare down to only the [n - k]th element on the kth pass. The bubble sort is implemented with the following function.

#define TRUE 1

#define FALSE 0

#define MAX 100

typedef struct

{

int key;

}record;

typedef record recordarray[MAX];

bubblesort (data, n)

/* Sorts the n records stored in array data

in descending order.

*/

recordarray data;

int n;

{

int i,done;

done = FALSE;

while (!done)

{

done = TRUE;

for (i=1;i<=n-1;i++)

if (data [i +1].key > data [i].key)

{

interchange (data, i, i + 1);

done = FALSE;

}

}

}

8.3.3 Timing the Bubble Sort

If the array were initially given in sorted order, only one pass would be required, and the time would be proportional to n. If the array were initially given in reverse sorted order, then n - 1 passes would be required, with (n - 1 - k) interchanges and comparisons. This is the worst case, and takes time determined by the   [image]  (n - k) interchanges and comparisons. This sum is (n - 1) + (n - 2) +   [image]    [image]    [image]  + 1 or [n(n - 1)]/2 interchanges and comparisons. The worst-case time for the bubble sort is O(n2), just as for the maximum entry sort. However, the maximum entry sort will always take time O(n2), since the number of comparisons it makes is not dependent on the data. The time taken by the bubble sort thus varies from O(n) for sorted data to O(n2) for reverse sorted data. The more ordered the initial data, the shorter the time.

8.3.4 Insertion Sort

The next sort to be considered is insertion sort. An insertion sort works by assuming that all entries in the array are already in sorted order and inserting the next entry in its proper place in the order. Figure 8.5 presents an example of an insertion sort.

In this example the same array, data, is used as for the bubble sort. Each record is processed in turn, starting from the second. Figure 8.5(a) represents the situation after processing of the first six records. The seventh is yet to be processed. It is processed, as are all the other records, by assuming the preceding records are already in sorted order. Notice that records 1-6 are in order. To insert the seventh record, 75, into its proper place among the first seven records, compare record 7 with its predecessor, record 6. Since 75 exceeds 10, it should go somewhere higher up in the array, so compare it to the predecessor of record 6, record 5. Since 75 exceeds 32, compare it to the predecessor of record 5, record 4.

(a) Before Processing Record 7

(b) After Processing Record 7

Figure 8.5 Insertion Sort

Again, 75 exceeds that element's value of 32, so compare it to its predecessor's value, 65. The last comparison is with 78. Since 75 does not exceed 78, record 7 must be inserted between record 2, where 78 is now, and record 3. To do this, move records 3 to 6 down one element, respectively, and place 75 in record 3. The result is Figure 8.5(b).

K has been updated by 1, so it points to the next record to be inserted. This algorithm for an insertion sort can be described by a loop structure in which k varies from 2 to n, as in the following function.

#define TRUE 1

#define FALSE 0

#define MAX 100

typedef struct

{

int key;

}record;

typedef record recordarray [MAX];

insertionsort (data, n)

/* Sorts the n records stored in array data

in descending order.

*/

recordarray data;

int n;

{

int k, i, inserted;

for (k=2;k<=n; k++)

{

i = k;

    >Comment

inserted = FALSE;

while ((i > 1) && !inserted)

if (data[i - l].key < data[i].key)

{

interchange(data,i,i - 1);

i--;

}

else

inserted = TRUE;

}

}

Within the loop, process the kth record by inserting it properly among the top k elements. This may be accomplished by another loop, in which record k is compared to each consecutive predecessor until it does not exceed the current predecessor record. It is inserted at that point in the array. As an exercise, you should modify the insertionsort program to incorporate the programming trick of Section 8.2.2. In Table 8.1 (see Section 8.7), this is referred to as the modified insertion sort.

8.3.5 Timing the Insertion Sort

If the initial array were given in sorted order, the insertion of the kth record would require one comparison and no insertions. The total time would be O(n - 1). If the initial array were given in reverse sorted order, the insertion of the kth record would require (k - 1) comparisons, and the kth record would have to be inserted at the top of the array. To accomplish this, each of the (k - 1) records must be moved down one element. The total time will be the time for the 1 + 2  +   [image]    [image]    [image]  + (n - 1) comparisons, plus the time for the 1 + 2 +   [image]    [image]    [image]  + (n - 1) shifts down. Consequently, the total time is O(n2). This represents the worst case. Again we see that the time can vary with n2 for large n. The insertion sort, like the bubble sort, performs better the closer the initial data is to being sorted. If each of the n! possible initial orderings of the array with n distinct entries is equally likely to occur, then the average time for all three of the sorts considered so far is also O(n2).

In the three implementations for the sorting algorithms, it was assumed that the interchanges taking place were actually interchanges of records. When records are large, it takes appreciably more time to move a record than it does to move a single pointer to the record. Consequently significant amounts of time can be saved by keeping an array, p, of indexes or pointers to the records. The entries of this array can then be kept in order, so that p[i] indexes or points to the ith record in sorted order. Interchanging indexes or pointers takes less time than for records but uses more memory (which is an added expense). This is another example of trading time for storage. In the procedures shown for maximum entry sort, bubble sort, and insertion sort, the interchange operations would then apply to the array p, and the comparison operations would apply to the actual records indexed or pointed to by i.

8.3.6 Attempted Improvements

The next question is, can we improve the insertion sort? We could try to improve it by changing how the search is done for the correct location of the kth entry. In effect, the insertion sort traversed linearly through records [k - 1] to 1 to insert the kth record. Instead, we could do a binary search of the consecutive elements 1 to [k - 1]. We will then know where to insert the kth record. This will reduce the 1 + 2 +   [image]    [image]    [image]  + (n - 1) comparisons to 1g 2 + 1g 3 +   [image]    [image]    [image]  + 1g (n - 1), which is about n 1g n. It does not, however, alleviate the need for the 1 + 2 +   [image]    [image]    [image]  + (n - 1) shifts due to the insertions, so the time is still O(n2).

What else can be tried to improve the insertion sort? Aha, lists! Recall that lists are convenient for insertions, so simply represent the records in a list rather than an array. Then, once you know where to insert a record, merely change two pointers. Thus the 1 + 2 +   [image]    [image]    [image]  + (n - 1) or [n(n - 1)]1/2 insertion time is reduced to time O(n). The list, coupled with the binary search reduction in comparison time to n 1g n, seems to give a new implementation of the insertion sort with worst-case time n 1g n. The fly in this ointment is that the ability to do a binary search has disappeared. It is no longer possible to locate the middle record by selection, which is what makes the binary search with sorted arrays so fast. Just the first record's location is known; the middle record can be found only by traversing the list and counting records until the middle one is reached.

Figure 8.6 Selection of Records by Means of a Pointer Array

Wait! We are not yet defeated in our quest for a faster insertion sort. Use a pointer array, assuming enough storage is available, to tell where all the records are on the list. This allows selection of the list records as in Chapter 2. The situation is then as shown in Figure 8.6.

Numberlist is the head of the list of records in the illustration. Suppose the first six records have been properly processed as in the insertion sort, and the seventh is to be processed next. A binary search can now be done, since the middle record, among 1 to 6, will be indexed by the middle entry of elements 1 to 6 of the pointer array. Thus it is possible, in 1g 6 time, to determine where the seventh record is to be inserted. The insertion will require the changing of two pointers. But here's the rub! To maintain the pointer array, it must have its seventh element inserted in the corresponding position of the pointer array. We have merely succeeded in transferring the insertion problem to the pointer array; we have not eliminated it!

Although these attempts to find a faster sorting algorithm have so far not been successful, they provide insight into the binary search and sorting problems. First, notice that except for the binary search, all the searching and sorting algorithms so far considered may be applied to records stored in lists as well as in arrays. Observe that to achieve both a search in binary search time (lg n) and fast insertions and deletions, a linked type structure is needed, as well as some way to eliminate from consideration a significant fraction of the records. A second observation is that the original insertion sort would be faster if the length of the array in which each insertion is made were shorter. We must somehow break up the array of records to be sorted, so that the individual insertions do not involve all of the previous records. These observations lead to the ideas behind heapsort and quicksort, two faster sorting algorithms.

8.4: Heapsort: A Faster Sort

In this section and the next, two less obvious but faster sorts are considered. These two sorts, heapsort and quicksort, were developed from different points of view and as a result have different advantages.

8.4.1 Heapsorts

Heapsorts use the concept of a heap. A heap is a binary tree with the property that all records stored in nodes are in sorted order along the path from each node back to the root. The tree of Figure 8.7 is not a heap. Its records are not in sorted order along the path from each node to the root. For example, the path from node 10 (where 123 is stored) to the root is not in sorted order. That path has records 123, 10, 32 and 100. (The nodes in this chapter are numbered like the nodes on trees in Chapter 7 and are referred to by their number.) Notice that the tree has the same information as the data array, but the information has been spread out along different paths.

Figure 8.8(a) is a heap storing the same information. It is one of many heaps that can be created to store the information. Since the path from every node to the root is in sorted order, the record at the root of a heap must be as great as any other in the tree. The same holds for all subtrees. The record at the root of any subtree must be at least as great as any other record in the subtree. The root record may then be taken as the first record in a sorted output of the information stored in the tree.

Figure 8.7 Not a Heap

(a) A Heap

(b) After Removing 536 and Reheaping

(c) After Adding a New Record to Heap (a)

Figure 8.8 Heaps

To sort, output 536 first, and then remove it from the tree. Now suppose that somehow the records can be rearranged in the tree to create a new heap--a process called reheaping. The result might look like Figure 8.8(b). This tree has one less node than the original. Clearly, this process of removing the root can be repeated, outputting it next in the sorted output, and reheaping. When all the records have been output in sorted order, the remaining tree will be the null binary tree (the heap will be empty), and a sort will have been obtained. The algorithm is stated simply:

To perform a heapsort:

1.     Create a heap.

2.     While the heap is not empty

a.     output the record at the root of the heap,

b.     remove it from the heap, and

c.     reheap.

As stated, the algorithm is not detailed enough. It does not specify how to create a heap (step 1) or how to reheap (step 2).

8.4.2 Creating a Heap

Consider the heap of Figure 8.8(a) with one new record as in Figure 8.8(c). Notice that all paths are in order, except for the path from the new record at node 21 to the root. This prevents this tree from being a heap. To make it a heap, the insertion sort idea may be useful. Insert the new record at its proper place along the already sorted path from its predecessor node to the root, and allow 425 to work its way up along the path as far as it can go the reach its proper place. This can be done by comparing 425 to its predecessor and interchanging if it exceeds its predecessor's value. In this case, it does exceed that value, so interchange to obtain the tree shown in Figure 8.9(a).

All other paths remain sorted, as before. Now compare 425 at node 10 with its predecessor. Since it exceeds 123, interchange again to obtain Figure 8.9(b). Finally, compare 425 with its predecessor's value, and interchange to obtain Figure 8.9(c).

At this point, when 425 is compared to its predecessor's value, the 425 has reached its proper place. All paths must now be in order, since the path from node10 to the root is in order, and the interchanges have not disturbed the order along any other path.

This gives us the means to create a heap:

Start with records stored in the complete binary tree and process each node in turn, starting with node 2 and ending with node n.

The processing of the ith node is exactly the same processing as was done on the new record 425. Since nodes 1 to i - 1 will already have been processed prior to processing node i, the binary tree consisting of nodes 1 to i - 1 will already be a heap. Processing node i then, in effect, treats the record at node i as a new record and creates a new heap. For the example of twenty records, this procedure leads to the heap of Figure 8.8(a).

Figure 8.9 Heap Creation

8.4.3 Timing the Heap Creation

In order to analyze the time required, recall that the number of nodes n and the depth of a complete binary tree are related by

d =   lg2(n + 1)

This is important. Since the heaps dealt with initially in our heapsort algorithm are complete, their depth is O(lg (n)).

We now analyze the time required to create a heap by this algorithm. Starting from a complete binary tree is important. For instance, if we started with the binary tree shown in Figure 8.10, this algorithm would be equivalent to the original insertion sort of Section 8.3.4.

The heap creation algorithm does a total amount of work determined by the sum of the number of comparisons and interchanges needed in processing each node. A node at depth k cannot require more than k comparisons and k interchanges in its processing. The worst-case time will be achieved for any initial tree in which the numbers stored at nodes 1, 2, 3, . . . , n form an increasing sequence. For instance, if the integer i is stored at node i, for i = 1, 2, . . . , n, we obtain such an increasing sequence. If n is 15, the tree will be as shown in Figure 8.11.

Figure 8.10 An Incomplete Binary Tree

It can be shown that the total time for such a tree to be turned into a heap by the algorithm is proportional to   [image], where n is the number of nodes at depth k, d is the depth, and k is the number of comparisons and interchanges required for each node at depth k. This is because every record will work its way to the root of the tree when it is processed. Since nk = 2k-1, this sum will be proportional to n lg n. The efficiency of this method can be increased somewhat by first determining where a record to be processed will end up along its path to the root, and then shifting everything down before inserting it. This efficiency could also have been achieved with the insertion sort.

Figure 8.11 Binary Tree with Numbers Forming an Increasing Sequence

8.4.4 Better Heap Creation

For a tree of depth d, a better procedure for creating a heap is to start at node number 2d-1 - 1, the rightmost node at depth d - 1. Even better, start at   [image] n/2  [image] , the integer part of n/2, the first nonterminal node at depth d - 1. Process each node in turn, from node   [image] n/2  [image]  to node 1. The processing at each node i is to turn the tree with node i as its root into a heap. This can be done, assuming that its left and right subtrees are already heaps, as in the following example.

Consider the initial complete binary tree in Figure 8.12(a). Start at node 5 ( =   [image] 11/2  [image] ) (83 is stored there) and compare its successors 94 and 7, determining that 94 is the larger. Compare 94 with 83; since 94 exceeds 83, interchange 94 and 83 to obtain Figure 8.12(b).

The subtree with node 5 (94) as root is now a heap. Move to node 4, and the similar comparisons reveal that it is already a heap. Move to node 3, compare its successors 29 and 30, and then compare the larger to 16. Since 30 exceeds 16, interchange 30 and 16 to obtain Figure 8.12(c).

The subtree of node 3 is a heap, so move to node 2. Compare its successors 57 and 94, and then compare the larger to 14. Interchange 94 and 14 to obtain Figure 8.12(d).

At this point, notice that all nodes processed so far must be the roots of subtrees that remain heaps, except perhaps for the root of the subtree, where 14 now appears after the interchange. This subtree need not be a heap and, in fact, is not in this case. You can make it a heap by applying the same process to the node where 14 is currently stored, node 5. Compare its two successors and then compare the larger of those with 14. Then interchange 14 and 83 to obtain Figure 8.12(e).

Figure 8.12 A Fast Heap Creation

Figure 8.12

The subtree with root node 2 is now a heap, and the processing of node 2 is complete. Notice that all nodes processed so far are roots of subtrees that are heaps. The last node to be processed is now node 1. Compare its successors, 94 and 30. Since the larger, 94, exceeds 18, interchange to obtain Figure 8.12(f).

The 18 from the node being processed, node 1, has now moved down to node 2. Hence its subtree may no longer be a heap. Apply the process to node 2. The 18 can move down again and will continue to move down until it either becomes situated at a terminal node or becomes the root of a subtree that remains a heap. In this case, it should be compared with 83, the larger of its two successors, and interchanged to get Figure 8.12(g).

At this point, you can see that 18 is the root of a heap by comparing it to the larger of its two successors. Processing of node 1 has been completed, and the heap has been created. The heap obtained from the original example with this algorithm is Figure 8.12(h). Notice that this heap differs from that in Figure 8.8(a). As mentioned earlier, more than one heap can be constructed to store the same data.

Without analyzing this heap creation algorithm in detail, the result can be stated. This procedure is actually only O(n). This heap creation algorithm is faster than that of the last section because few nodes must move down long paths. In the other algorithm, many nodes may move up long paths. A use for this linear time heap creation algorithm will be demonstrated in another context in Section 9.2, Priority Queues.

8.4.5 Some Details of Heap Implementation

One way to implement the heap creation algorithm is to make use of a function, or routine, that works on a binary tree whose left and right subtrees are already heaps. The function does the "shifting" down of the root record until it becomes the root of a subtree that remains a heap, or until it reaches a terminal node. This "shift" routine need not do the interchanging along the way. Instead it can determine the final position of the root being shifted down, moving each record along that path to its predecessor, finally placing the root where it belongs.

Notice that the first heap creation algorithm requires testing the record being inserted along a path to the root against its current predecessor. Before this can be done, the program must test for whether or not there is a predecessor--that is, whether the record has already reached the root. This test must precede every comparison. Similarly, in the second heap creation algorithm, the record being shifted down must be tested against its successors. Before doing this, it is necessary to test whether there are successors.

These tests can be avoided by always putting "plus infinity" at the root of the initial complete binary tree for the first algorithm, and "negative infinity" at the subtrees of all terminal nodes for the second algorithm. This is similar in effect to adding the search key to the end of the array in the linear search algorithm. Plus infinity and negative infinity stand for, respectively, any number larger or smaller than the numbers appearing in the binary tree. In this way, a record being inserted along a path in the first algorithm will never reach the root (so it will always have a predecessor with which it may be compared), and a record being shifted down in the second algorithm will always have successor nodes (since it will never become a terminal node).

8.4.6 Reheaping

Think of step 1 of the heap sorting algorithm as phase I, in which a heap is created. The execution of step 2 until the records are sorted is phase II. In phase II, the root element of the heap is removed and output. The program must then reheap. This may be done in two ways.

One is to allow the "gap" created at the root by removal of its record to shift down by comparing its two successors and interchanging the larger of the two successors with the gap, then repeating this process until the gap is at a terminal node. A convenient way to do this is to replace the record removed from the root by a "negative infinity" and invoke the shift down function referred to earlier for the second algorithm.

A second way to reheap is to replace the gap with the rightmost record at greatest depth. (Actually, any record with no successors will do.) The same shift function may be invoked to reheap.

Since the length of the longest path in a complete binary tree is O(lg n), the reheaping after each record is removed can take time at most O(lg n). Because n records will ultimately be removed, the total time required for phase II will be at most O(n lg n). For either implementation of phase I, the total heapsort time is O(n lg n). This was achieved by "spreading out" the records along paths of a complete binary tree, and, in effect, doing insertions only along the path from a record to the root.

We have been dealing with a max heap, in which the largest record is at the top. A min heap, with the smallest record at the top, is dealt with similarly. The only difference is that interchanging occurs when a successor's key is smaller, rather than larger.

8.4.7 An Implementation for Heapsort

A detailed version of the heapsort algorithm can now be developed. You have now seen a number of ways to create a heap and to reheap. We will use the second heap creation algorithm for phase I and the second reheaping algorithm for phase II. Both algorithms use the shift function.

To proceed we must decide on an implementation of the binary tree data structure. The binary tree used in heapsort is initially complete. The required processing involves predecessors and successors of nodes. Sequential representation for the binary tree is ideal, so we use it. The binary tree is stored in the data array and consists of n records.

Storage is needed for the sorted version of data. Although another array may seem necessary for this purpose, we will use only the data array. The algorithm, after creating a heap (in data), continually takes the root record as the next record in the sorted output and then reheaps. Before reheaping, the rightmost record at the lowest level of the heap is placed at the root. The storage vacated by that rightmost record will no longer be needed. We choose to store the output record there. You should convince yourself that this will result in the records appearing in data in reverse sorted order upon termination. To remedy this, we write shift so that it produces min heaps. The records will then appear in correct sorted order upon termination.

Shift has parameters data, root, and last. Root points to the root of a subtree, and last points to the node of the original heap that is currently the rightmost node at greatest depth. The nodes last+1 to n are currently storing the already sorted n-last output records. Whenever shift is invoked, the left and right subtrees of root (excluding nodes last+1 to n) are assumed to be heaps. Shift returns after reheaping, by allowing the root record to move down to its final position. This is done by first storing the key value and record of the initial root record in keyvalue and recordvalue, respectively. Ptr points to the node currently being considered as the final position for the initial root record. This will be the final position if either of the following two conditions is met.

Figure 8.13 (a) Initial Heap and (b) Heap after First Three Entries Have Been Output

1.     The successor record of ptr with minimum key value has a key value greater than or equal to keyvalue, or

2.     Ptr points to a terminal node or to a node whose successors are nodes among last+1 to n.

Condition 2 can be recognized by the test (succ<=last), where succ points to the left successor node of ptr. Condition 1 requires finding the minimum key value for the successors of ptr. Ptr may not have a right successor, or that successor may represent an output record. This can be tested for by the test (succ<last).

Copy copies the contents of the record pointed to by its second parameter into the record pointed to by its first parameter. When shift determines that ptr is not the final position, it moves the record at ptr to its predecessor node. This is why the original record at root is saved. When the final position is found, the saved record is placed there. The initial heap and the situation after the first three entries have been output are shown in Figure 8.13 for the tree of Figure 8.7.

The implementation for heapsort follows.

#define MAX 100

typedef struct

{

int key;

}record;

typedef record recordarray[MAX];

heapsort (data,n)

/* Sorts the n records stored in array data

in descending order.

*/

recordarray data;

int n;

{

int root,last;

    >Comment

root = n/2;

last = n;

while (root > 0)

{

    >Comment

shift(data,root,last);

root--;

}

root = 1;

while (last > 1)

{

    >Comment

interchange(data,1,last);

last--;

shift(data,root,last);

}

}

shift (data,root,last)

/* Makes the subtree of root into

a heap (min heap). The left and

right subtrees of root must be

heaps. All subtrees are stored

in array data in sequential

representation. Only the nodes

between root and last are

considered to be in the subtrees.

*/

recordarray data;

int root,last;

{

    >Comment

int ptr,succ;

int keyvalue;

record recordvalue;

    >Comment

ptr = root;

    >Comment

succ = 2 * ptr;

    >Comment

if (succ < last)

if (data[succ + 1].key < data[succ].key)

    >Comment

succ++;

at this point, succ points to the smallest of ptr's successors

    >Comment

keyvalue = data[root].key;

    >Comment

copy(&recordvalue,&data[root]);

the purpose of the while loop is to determine the proper place for the original root record; ptr will

point to that location when the loop is exited

    >Comment

while((succ <= last) && (data[succ].key < keyvalue))

{

    >Comment

copy(&data[ptr],&data[succ]);

    >Comment

ptr = succ;

    >Comment

succ = 2 * ptr

if (succ < last)

if (data[succ + 1].key < data[succ].key)

succ++;

}

    >Comment

copy(&data[ptr],&recordvalue);

}

8.4.8 Heapsort is Uniformly Fast

As presented, heapsort is efficient. Unlike the other sorts considered, which are array- or list-oriented, it is tree-oriented. They are O(n2) in the worst case, whereas heapsort is O(n lg n). Moreover, we shall see in Section 8.7 that it gives uniformly good sort times, no matter how ordered or unordered the original data. Although times are somewhat faster when the data is initially either fairly well ordered or reverse ordered, the range of these times is small compared to the other sorts. This is to be expected, since the construction of the heap takes little time and the reheaping tends to destroy any order present in the data. In effect it appears to be in random order. Actually, during reheaping, when shift works on larger entries at the root, it takes O(lg n) time, since they tend toward the bottom of the tree, while the opposite is true for smaller entries. Because most entries are near the bottom of the tree and are larger, their shifting makes up the bulk of the sorting time. Thus this time should be relatively insensitive to initial order in the data. This uniformity of sort times is one of its main virtues, since it eliminates the great variation in time from O(n) to O(n2) of the other sorts. In addition, it is fast.

8.5: Quicksort: Another Fast Sort

As a final example of a sorting method, we consider an algorithm called quicksort. This algorithm is appropriately named, since its average execution is very quick. Its worst-case execution time is slow, but the worst case should occur rarely.

One way to develop quicksort is to attempt a recursive solution to the problem of sorting an array data of n records. The general recursive method immediately prescribes guidelines--break up the original problem into smaller versions of the original problem and then achieve the solution from their solutions. Here, this dictates that the original array be broken down into component arrays, which can be sorted independently and from which the final sorted array can be derived.

Thus data is to be separated into two components to be sorted, an upper array and a lower array. We then have two options for proceeding. One is to combine or merge the resultant two sorted component arrays to achieve a final sorted solution. This method is the merge sort, which is not pursued here but is applied to the sorting of files in Chapter 11. The second approach, which is taken here, is to ensure that the resultant two sorted component arrays require no further processing, but yield the final sorted solution directly.

Merely splitting data into two components and then sorting each component independently is not sufficient to guarantee that the resultant array is sorted, with no further processing required. This is because some entries in the upper component may be smaller than some entries in the lower component. For example, suppose the initial array is data, shown in Figure 8.14(a), and the upper and lower components consist, respectively, of entries 1 to 4 and 5 to 8. Sorting these components independently yields Figure 8.14(b), which is clearly not sorted.

Figure 8.14 Sorting an Array by Component

Notice, however, that if all entries in the upper component were no less than all entries in the lower component, then sorting each component independently would always lead to a result that is sorted. For example, suppose the initial array is Figure 8.14(c). Taking the upper and lower components as entries 1 to 4, and 5 to 8, respectively, and independently sorting them produces Figure 8.14(d). This array is now sorted.

Note that the fifth entry, 13, occupies the same position in the sorted array as in the initial array. This is because all entries in the upper component are no less, and all entries in the lower component are no greater, than 13. The key value 13 actually occupied its proper position in the sorted version of data before the two components were sorted. Hence it need not be included in either component and may serve as a separator or partitioning entry between the two components to be sorted.

To carry out this procedure for an arbitrary array requires finding a partitioning entry. Normally a partitioning entry, such as 13 in position 5, will not occur in  the initial array to be sorted. For example, no such partitioning entry (separating larger entries above from smaller entries below) occurs in Figure 8.14(a). This means that the initial array must be rearranged so that a partitioning entry may be found and subsequently used to separate the components.

In keeping with the top-down approach, assume that this task (rearranging the initial array and determining a partitioning entry) is carried out by the function partition(i,j). The arguments i and j contain pointers to entries of the array, as shown in Figure 8.15. Partition (i,j) returns an index pointer to a partitioning entry of the array consisting of all entries between i and j. Thus, if partition (i,j) returns a value p, then all entries between i and p-1 are at least as great as data [p].key, and all entries between p+1 and j are no greater than data [p].key. The pointer p separates the upper and lower components of i,j.

Figure 8.15 Array with a Partitioning Entry P Separating Upper and Lower Components

8.5.1 Two Quicksort Procedures

The procedure quicksort may now be written, recursively, as follows.

Recursive Version

quicksort(i,j)

/* Sorts the records stored in array data

between positions indexed by i and j in

descending order.

*/

int i,j;

{

int p;

if(i < j)

{

    >Comment

p = partition(i,j);

    >Comment

quicksort(i,p-1);

    >Comment

quicksort(p+1,j);

}

}

To sort a global array of n records, quicksort(1,n) is invoked. The crux of the algorithm is the partition function, which will be discussed shortly.

Every time a call is made to quicksort, two new obligations are incurred, and a current obligation is postponed. For instance, the initial call to quicksort generates the initial obligation: sort the n entries of data. During the execution of this call, partition determines a partitioning entry, and two new obligations are incurred. These are represented by the two recursive calls to quicksort, one for each component. The current obligation must then be postponed to carry out these new obligations. Each recursive call also generates two new obligations, requiring the execution of the current recursive call to be postponed.

As we have seen previously, what can be done recursively can also be done iteratively. Quicksort can be written nonrecursively by using a stack to store one of the new obligations, while the other can be processed immediately. It turns out that the choice of which of the two obligations to stack, and which to deal with immediately, is critical to the algorithm's storage requirements. The partition function does not necessarily split an array into two equal parts, as evidenced by our earlier example. Selecting the larger of the two components to stack results in a worst-case stack depth of O (lg n). An arbitrary choice instead leads to a worst-case stack depth of O (n). The difference in storage needed for the stack can be significant.

The structure of the nonrecursive algorithm amounts to a loop in which the smaller of the two new component arrays is processed and the larger is stacked. The procedure may be written as follows:

Nonrecursive Version

quicksort(i,j)

/* Sorts the records stored in array data

between positions indexed by i and j in

descending order.

*/

int i,j;

{

int p,top,bottom;

stack s;

    >Comment

setstack(&s);

    >Comment

push(j,&s);

push(i&s);

    >Comment

while(!empty(&s))

{

    >Comment

pop(&s,&top);

pop(&s,&bottom);

while(top < bottom)

{

    >Comment

p = partition(top,bottom);

    >Comment

if((p - top) > (bottom - p))

{

push(p-l,&s);

push(top,&s);

top = p + l

}

else

{

push(bottom,&s);

push(p+1,&s);

bottom = p - 1;

}

}

}

}

8.5.2 The Partition Function

We must now return to the partitioning task on which the efficiency of both versions of quicksort depends. To determine a partitioning entry, and a corresponding rearrangement, the first array entry is selected as a basis. This is a somewhat arbitrary selection and will be discussed more fully later. In general, this first array entry will not be in its proper sorted position. It will need to be shifted, and the array rearranged, so that all larger entries appear in the upper component and all smaller entries appear in the lower component.

A clever way to do this requires two pointers, upper and lower. Initially, upper is set to i, the first array index, and lower to j, the last array index. The initial situation is shown in Figure 8.16(a).

Lower is now moved upward until an entry larger than data [upper].key is encountered. In this case, 15 will be that entry. It is then interchanged with data[upper]; see Figure 8.16(b). At this point, attention is shifted to upper, which is moved downward until an entry smaller than data[lower].key is encountered. This entry will be 10, and it is interchanged with data[lower], yielding Figure 8.16(c).

Figure 8.16 Sorting an Array by Means of a Partitioning Function

Attention now shifts back to lower, which is again moved upward until an entry greater than data[upper].key is encountered. This will be 30, and it is interchanged with data[upper], resulting in Figure 8.16(d).

Upper is then moved downward until an entry smaller than data[lower].key is encountered. In this case no such entry appears, and when upper reaches lower, no such entry can appear. This is so because, at all times, any entries above upper are greater than the original first entry (13 in the example), and any entries below lower are less than the original first entry. Hence, whichever pointer is being moved, only entries between upper and lower can result in an interchange. When upper and lower coincide, no entries are left to interchange. Furthermore, when upper and lower coincide, they must be pointing to the original first entry or basis, since at all times at least one of them points to it. The final situation, then, is as shown in Figure 8.16(e).

This procedure clearly carries out the task of partition(i,j), which is to return the value of upper(or lower). Notice that, whether upper or lower is the focus of attention, it is always the original first entry (13 here) that is the basis for comparison.

Figure 8.17 Partitioning an Array by Means of a Temporary Variable, SAVERECORD

A slight modification makes partition more efficient. Instead of interchanging the basis with the larger or smaller entry that has been found, simply move the larger or smaller entry to the position pointed to by the current stationary pointer. This is the position that would have been occupied by the basis (13 in the example). However, the basis does not now appear in the array but is saved in a temporary variable, saverecord. For the example, this would generate the sequence shown in Figure 8.17. When upper and lower meet, the record stored in saverecord must be copied into data[upper]. This version of partition is faster because a move is carried out much faster than an interchange, and many such operations are required as the algorithm executes. It is therefore the algorithm implemented below.

partition(i,j)

/* Rearranges the entries in array

data between positions indexed by

i and j and returns an array index

which guarantees that all entries

above it are larger than the entry

it indexes, and all entries below it

are smaller.

*/

int i,j;

{

int upper,lower;

record saverecord;

    >Comment

upper = i;

lower = j;

    >Comment

copy(&saverecord,&data[i]);

    >Comment

while(upper != lower)

{

    >Comment

while((upper < lower) && (saverecord.key >=

data[lower].key))

lower--;

if(upper != lower)

    >Comment

copy(&data[upper],&data[lower]);

    >Comment

while((upper < lower) && (saverecord.key <=

data[upper].key))

upper++;

if(upper != lower)

    >Comment

copy(&data[lower],&data[upper]);

}

    >Comment

copy(&data[upper],&saverecord);

    >Comment

return(upper);

}

It is important to recognize that whenever partition (i,j) is invoked, it processes only entries between i and j of the data array and deals with each of these entries exactly once. Each entry is compared with saverecord, and perhaps copied into data [upper] or data [lower]. Consequently, a call to partition (i,j) takes time O(j - i + 1), since there are j - i + 1 entries between i and j.

8.5.3 Analyzing Iterative Quicksort

We now want to analyze the nonrecursive version of quicksort to determine its time and storage requirements. To provide some insight into its behavior, we will simulate its application to our initial array, focusing on the changes that occur in this array and in the stack during execution.

Initially, the stack is empty, and the array is as shown in Figure 8.18(a). When quicksort (1,8) is invoked, it calls partition (1,8), which returns with p pointing to the partitioning entry and the rearranged array of Figure 8.18(b).

The upper component array is the larger, so (1,4) is placed on the stack, and the lower component array is processed. This leads to a call to partition (6,8). When partition returns, the situation is Figure 8.18(c).

Figure 8.18 Iterative Quicksort

The current upper component array is larger, so (6,7) is placed on the stack, and the lower component array is processed. Since the lower component contains no entries, (6,7) is removed from the stack and partition(6,7) is invoked, returning the situation depicted in Figure 8.18(d).

Now (6,6) is placed on the stack, and the lower component is processed. Again, this component has no entries, so (6,6) is removed from the stack. Since top (= 6) is not less than bottom (= 6), the outer while loop body is now executed, causing (1,4) to be removed from the stack. At this time the stack is empty, and the array is as shown in Figure 8.18(e). Partition(1,4) is now invoked, and returns, yielding Figure 8.18(f).

Now (1,3) is placed on the stack. The lower component again contains no entries, so (1,3) is removed from the stack, and partition(1,3) is invoked. The result is Figure 8.18(g).

At this point (2,3) is placed on the stack. Since the upper component contains no entries, (2,3) is removed from the stack, and partition(2,3) is invoked. It returns with Figure 8.18(h).

Now (2,2) is placed on the stack. The lower component has no entries, so (2,2) is removed from the stack. Since top (= 2) and bottom (= 2), the outer while loop tests for an empty stack. As the stack is empty, quicksort terminates.

It should not be clear that each time the inner loop body of quicksort is executed, and top < bottom, p is set to point to an entry of the array that has been correctly placed in its final sorted position. If top = bottom, then the one entry is already in its proper sorted position. The inner loop body is thus executed at most n times. Ignoring the time taken by partition, the inner loop body takes a constant time to execute. Each time the inner loop body executes, it generates one new problem that is placed on the stack and another new problem that is processed immediately. As soon as this latter problem has fewer than two entries (top   [image]  bottom), the inner loop is exited. Unless the stack is empty, the outer loop removes the next problem from the stack, and the inner loop then functions again. Consequently, each entry that is placed in its proper position requires, at most, an execution of the inner loop body or an execution of the inner loop body plus the additional constant time required by the outer loop test plus removal of a problem from the stack. This means that the total time is, at most, c1 + c2 x n plus the total time taken by partition, for some constants c1 and c2.

8.5.4 The Worst and Best Cases

To analyze the total time of partition, recall that each execution takes time O(length of the component array it is called to work on). Since the length of a component is at most n, with partition being invoked at most n times, total time is at most O(n2). Hence quicksort has a worst-case time O(n2). You should convince yourself that the worst case occurs whenever the initial array is in sorted or reverse sorted order.

The best case (fastest execution) actually occurs when the partitioning entry returned by partition splits the array in half. In this case, partition is called to work on one component array of length n, two component arrays of length n/2, four component arrays of length n/4, . . . , 2k component arrays of length n/2k, etc. By an argument similar to the worst-case analysis of the binary search, about 1g n such calls will be made to partition. Its total time will then be O(n 1g n).

While the worst-case time for quicksort is poor, its best-case time is fast--in fact, better than heapsort. The average time for quicksort is also O(n 1g n). This assumes that each ordering of the n entries is equally likely to appear in the initial array. Evidently, cases differing significantly from the best occur rarely, and this is why quicksort is aptly named.

The storage requirements for quicksort, beyond the initial array, are a few variables plus the stack depth. Since the larger of each generated component array problem is stacked, the next problem to be stacked must have length at most one-half the current top stack entry length. For example, in the preceding section, note that when (6,7) was placed on the stack, its length was no greater than half the length of the preceding stack component (1,4). Again, by an argument similar to the worst-case binary search analysis, at most O(1g n) such problems can appear on the stack at any time, so the storage requirements for quicksort are O(1g n). Thus, quicksort, while faster on the average than heapsort, does not have the guaranteed worst-case O(n 1g n) time of heapsort. Moreover, quicksort requires additional storage O(1g n) to achieve its faster average time.

For smaller values of n, the simpler sorts, like the insertion sort, will be even faster than quicksort. Of course, if the initial array is known to be nearly sorted, the insertion sort will be faster anyway. Clearly, one way to speed up quicksort is to modify it by using a faster sort, such as insertion sort, when small component arrays are encountered. It is also possible to enhance the performance of quicksort by taking more care in the selection of the entry to be used as a basis by partition. Instead of taking the first entry of a component as the basis, the median of three entries may be selected, or an entry may be selected at random.

8.5.5 Distributive Partitioning

Although sorting has been extensively researched, a new generalization of the quicksort, distributive partitioning, has recently been discovered [Dobosiewicz,1976]. It appears to be even faster than quicksort. Recall that the quicksort was based on the idea of rearranging the initial array so that it could be split into two component arrays to be sorted independently. Each of these arrays was then recursively sorted in the same way. Distributive partitioning carries this idea to its logical limit, by splitting the initial array into n component arrays, to be sorted independently and recursively. The price paid by this new algorithm is that additional storage of size O(n 1g n) is required. Note that all the sorting algorithms considered sort in place, except the quicksort and distributive partitioning.Quicksort and distributive partitioning require additional storage, limiting their use for large n. Since quicksort's additional storage increases 1/nth as fast as that of distributive partitioning, it will not be as severely limited.

8.6: Simulation of an Algorithm

It is often difficult, if not impossible, to analyze an algorithm mathematically and determine its time and storage requirements. Simulation is a useful basic tool for determining the behavior of an algorithm for this purpose. To illustrate, an algorithm for a simulation of the heapsort is constructed in this section.

The storage required by the heapsort algorithm in the implementation is O(n). A reasonable measure for the time required is given by the number of interchanges and comparisons necessary to produce a sort. Assume that a procedure treesort has been written that is one of the detailed implementations discussed previously for heapsort. It has parameters n, a, sa, int1, int2, comp1, and comp2. These represent, respectively, the number of records, the input array of records to be sorted, the output array of sorted records, the number of interchanges required in phases I and II, and the number of comparisons required in phases I and II.

The simulation program to be written must explore the time requirements, as measured by int1, int2, comp1, and comp2, for various values of n, such as 20, 40, 60, 80, and 100. This will suggest how the execution time depends on n. Alternatively, the actual execution times could be measured. Intuitively, the algorithm will have fastest execution time when the input is nearly sorted, the slowest time when the input is nearly reverse sorted. For random initial orderings, the time can be expected to fall between these two extremes. This may not be the case but seems likely a priori. Consequently, we want to generate a number of samples from each of these three distinct classes of input, execute the treesort routine for each sample, and collect statistics on the numbers of interchanges and comparisons based on the samples taken.

We will choose twenty-five samples of each type of input, for a total number of executions of treesort of (twenty-five samples for each input class and value of n) x 3 input classes x 5 values of n, or 375 runs. One way to see if twenty-five samples is enough to achieve good statistics is to run an additional twenty-five. If the results do not vary significantly, then the twenty-five samples, or perhaps even fewer, are sufficient. If necessary, we can use a larger number of samples. Remember, the twenty-five samples are taken from a possible number of n! samples. It may seem the height of foolishness to base estimates on such a ridiculously small fraction of the possibilities. Thank goodness for statistics (when properly applied).

                          n = 40

                       -------------

  Input Class          Min  Ave  Max

------------------------------------

"Nearly sorted"  int1   *    *    *

                 int2   *    *    *

                 comp1  *    *    *

                 comp2  *    *    *

"Random"         int1   *    *    *

                 int2   *    *    *

                 comp1  *    *    *

                 comp2  *    *    *

"Nearly reverse  int1   *    *    *

  sorted"        int2   *    *    *

                 comp1  *    *    *

                 comp2  *    *    *

For each of the twenty-five samples for a given input class, the task is to find the following statistics for int1, int2, comp1, and comp2: their minimum, average, and maximum over all twenty-five sample values. Other statistics, such as the range and standard deviation, could be calculated if needed. We will have the simulation program print this information for each value of n, as indicated for n = 40 in the chart on the previous page. The *'s represent actual numeric values that will be output. A total of thirty-six values will be printed for each n; 5 X 36 =180 statistics.

The program will keep statistics tables, one table for each input class for the current value of n. An algorithm for the simulation program is as follows:

1.     Set n to 20.

2.     While n   [image]  100

a.Set ns to 1.

b.     Initialize statistics tables.

c.     While ns   [image]  25,

i.     Generate a sample from each input class, call treesort to work on each input, and then update the appropriate statistics tables.

Set ns to ns+1.

d.     Output the statistics tables for the current value of n and for each input class.

e.     Set n to n+20.

Task (i) may be carried out as follows:

a.     Generate a "random" sample for a of size n and call treesort(n, a, sa, int1, int2, comp1,  comp2).

b.     Update the statistics table for "random" input.

c.     Generate a "nearly sorted" input for a of size n by making 20 percent of n random interchanges on sa. Copy sa into a.

d.     Generate a "nearly reverse sorted" input for a of size n by copying sa in reverse order in ra. Call treesort(n, a, sa, int1, int2, comp1, comp2).

e.     Update the statistics tables for "nearly sorted" input.

f.     Call treesort(n, ra, sa, int1, int2, comp1, comp2).

g.     Update the statistics table for "nearly reverse sorted" input.

h.     Set ns to ns+1.

At this point we can explain in more detail what is meant by the three classes of input. Assume that all input arrays, a, will contain integers between 1 and 1,000. Random inputs for a of size n are generated by placing the n integers into a in such a way that all orderings of the n numbers are equally likely. Consider the two arrays shown in Figure 8.19.

Array (b) has been obtained from array (a), which is sorted, by making two interchanges. These were interchanging a1[2] with al[6], and interchanging a1[4] with a1[9] . These two interchanges represent 20 percent of n interchanges, since n is 10. A nearly sorted input for a of size n is obtained by making 20 percent of n such interchanges in an initially sorted array. The interchanges are to be selected at random. This means that each interchange is obtained by picking a value for i at random, so that each of the locations 1 to n of sa is equally likely to be selected. An integer j may be similarly obtained. Then sa[i] and sa[j] are interchanged. The only details still left to implement in the simulation algorithm are how to generate the required random integers to be placed in a and used for interchange locations, how to update the statistics tables, and how to finalize the tables before printing them.

Figure 8.19 Array (b) Obtained from Sorted Array (a) by Two Interchanges

Assume that a function rng is available that returns a real number as its value. This real number will be greater than 0.0 and less than 1.0. Whenever a sequence of these numbers is generated by consecutive calls to rng, the numbers appear, from a statistical point of view, to have been selected independently. Also assume that the returned value will lie in a segment of length l between 0 and 1, with probability l. For example, if l is of length 0.25, then the returned value will lie in that interval (say, between 0.33 and 0.58) 25 percent of the time. The random number generator ranf of FORTRAN is an example of such a function.

To generate an integer between 1 and 1,000, take the value rng returns, multiply it by 1,000, take the integer part of the result, and add 1 to it. For example, if the value returned is 0.2134, multiplying by 1,000 gives 213.4. Its integer part is 213; adding 1 yields 214. In fact, if any number strictly between 0.213 and 0.214 had been returned, 214 would have been generated. Hence 214 would be generated with probability 1/1,000, the length of the interval from 0.213 to 0.214. The same is true for each integer between 1 and 1,000. Carrying out this procedure n times will fill the array a to generate a "random" input (See Exercise 23 for a more exact approach). An alternative method in C to generate an integer at random between integers low  and high uses the library function rand. Evaluating rand()%(high-low+1) + low yields the desired integer between low and high.

To generate a nearly sorted or nearly reverse sorted input, select each of the 20 percent of n interchanges as follows. Call rng twice. Multiply the two returned values by n, take the integer part, and add 1. Or evaluate rand()%(high-low+1) + low twice with high set to n and low set to 1. Either method gives two integers, i and j, between 1 and n. Interchange sa[i] with sa[j].

The statistics tables can keep current running minimums, maximums, and the accumulated sum of the appropriate statistic. When each call to treesort returns, the minimum, maximum, and accumulated sum may be updated for each statistic, int1,int2,comp1, and comp2. When the inner loop of the algorithm is exited, and task d is to be carried out, the statistics tables are ready to be printed, except for the twelve averages. They must be finalized in task d by dividing the accumulated sums by 25.

The algorithm presented in this section can be used directly to obtain simulation results for treesort. Simply replacing the call to treesort in task (i) by a call to suitably modified versions of the other sort programs will produce their simulation results as well. To use simulation in the analysis of other programs, not necessarily sorts, requires some insight into what input is appropriate to generate and what statistics are meaningful to collect. This is not always so straightforward as in our example, but the general procedure and goal for the simulation of an algorithm should now be clear.

8.7: Simulation Results for the Sorts

Table 8.1 gives simulation results (in actual execution time measured in 1/1000 second intervals). It should be carefully studied. Note that the O(n) and O(n lg n) sorts may be easily identified. The best and worst input cases are also evident, as well as the relatively uniform behavior of heapsort.

Each entry in the table is the average of 20 runs on a VAX 730 under VMS Version 4.0. For each n the five values correspond to input that is sorted, half sorted--half random, random, half reverse sorted--half random, and reverse sorted. If the entries for heapsort seem counter to your intuition, remember that its implementation sees sorted input as reverse sorted and reverse sorted input as sorted. This is because it uses a min heap to produce descending order.

8.8:  Synopsis of Search and Sort Efficiencies

We have consistently viewed the record as the basic unit of data. Collections of records usually appear in problems, and their processing typically reduces to traversing through the records of the collection, inserting or deleting records, randomly accessing records, accessing records in sorted order, or searching the collection for a given record. The basic data structures are available for storage of the collection, and each supports some operations well at the expense of others. Arrays are excellent for random access and traversal but poor for general insertions and deletions. Lists support traversal, insertions, and deletions well but are poor for random access.

Linear searches of arrays and binary searches of sorted arrays can be performed, respectively, in O(n) and O(1g n) time. Under special conditions, linear searches can be fast.

Simple sorting algorithms such as the bubble and insertion sort have worst-case time O(n2) but are O(n) when dealing with almost sorted data. Quicksort gives very fast average times and heapsort does sorting in worst-case time O(n 1g n).

Table 8.1 Simulation Results in Actual Execution Time (milliseconds)*

 n    Bubble Sort                        Insertion Sort                     Modified Insertion Sort

-----------------------------------------------------------------------------------------------------------

 10    1       2       6       8      9     1      2      5      7      7     0     2       4      6      9

 20    1       8      23      33     36     1      5     16     28     30     1     6      17     25     30

 40    2      31     101     138    148     2     16     67    110    121     5    16      65    107    120

100    4     217     646     863    923     7     89    412    678    761     6    87     402    660    744

200    8     966    2653    3463   3724    12    364   1657   2706   3058    11   354    1612   2639   2993

400   17    3933   11277   14732  15858    24   1583   7645  12489  14595    20  1373    6352  10581  12157

-----------------------------------------------------------------------------------------------------------

     srt'd  half   rnd'm    half   rev    srt'd  half   rnd'm  half    rev   srtd  half   rndm   half   rev

            srt'd           rev   srt'd         srt'd          rev   srt'd        srt'd          rev  srt'd

             half          srt'd                 half         srt'd                half         srt'd

            rnd'm          half                 rnd'm         half                rnd'm          half

                           rnd'm                              rnd'm                            rnd'm

 n    Heapsort                          Iterative Quicksort                  Recursive Quicksort

-----------------------------------------------------------------------------------------------------------

 10     9      9      8      8      7     10      9     9      8     10      5     5      3      3      4

 20    18     19     18     18     15     21     19    15     17     23     13    11     10     11     14

 40    43     43     41     40     37     51     47    33     38     55     35    31     21     25     36

100   130    126    122    119    115    196    171    94    106    202    156   133     65     74    158

200   291    282    277    264    256    626    521   202    231    633    581   484    149    169    602

400   640    624    610    587    572   2175   1750   432    492   2195   2042  1644    318    372   2056

-----------------------------------------------------------------------------------------------------------

    srt'd   half  rnd'm    half    rev  srt'd   half  rnd'm   half   rev   srtd   half   rndm   half   rev

           srt'd           rev    srt'd        srt'd          rev   srt'd        srt'd          rev  srt'd

            half          srt'd                 half         srt'd                half         srt'd

           rnd'm           half                rnd'm          half               rnd'm          half

                          rnd'm                              rnd'm                             rnd'm

-----------------------------------------------------------------------------------------------------------

* These results were produced by Sharon Doherty.

It is important to recognize that in this chapter we have been considering comparative searches and sorts. This means that we have assumed no special knowledge of the key values involved and have used only comparisons between key values in each search or sort algorithm. It is possible to prove that such searches and sorts have worst-case times O(lg n) and O(n lg n), respectively.

Other kinds of search and sort algorithms are possible. Some may take significantly less time. For example, suppose you must sort n distinct keys whose values are known to be integers between 1 and 1,000. Simply store the record with key value i as the ith record in an array of records. This places the records in the array in (decreasing) sorted order and takes O(n) time. Sorts that place records in groups, depending on their actual key values, and then attempt to sort further within the groups, are called distributive sorts. These are not considered in this text. We will, however, consider a very important noncomparative search in Section 9.3.

Simulation is a basic tool for the analysis of the storage and timing requirements of algorithms although we have used it only for analyzing sorts.

As searching as this chapter may have been, it is hoped that it didn't leave you out of sorts!

Exercises

1.     a.     If all comparisons and assignment statements take the same amount of time to execute, then compare the execution times of the first linear search procedure and the procedure revised for greater efficiency.

b.     For what values of n will the binary search be faster than the linear search of the improved procedure in the worst case?

2.     When you look up a book by author in the library, what kind of a search are you doing?

3.     In a ternary search of a sorted array, the array is divided into three "nearly" equal parts and searched. Key values are examined "near" 1/3 and "near" 2/3 and compared with the search key to determine in which third to search further. Determine the worst-case time for this search and compare it to the worst-case time for a binary search.

4.     Suppose a sorted array stores 1,000 integers between 0 and 1,000,000. A pointer array indicates where the least integer greater than 1,000 x (i - 1) but not greater than 1,000 x i appears in the array. P[i] contains that pointer for i = 1, 2, . . . , 1,000. If no such integer is in the array, then p[i] = 0. Write a function to do an interpolation search, coupled with a binary search.

5.     Assuming all comparisons and assignments take the same time to execute, do the following.

a.     Determine the worst-case times for the sorts of the procedures given in the text for the maximum entry sort, the bubble sort, and the insertion sort.

b.     For each value of n, determine which of the three sorts has least worst-case time.

6.     What input order produces the worst case time for

a.     The bubble sort?

b.      The insertion sort?

7.     Write a function to carry out the bubble sort when the integers are stored in a chain.

8.     Write a function to carry out the insertion sort when the integers are stored in a chain.

9.     Modify the insertionsort function so that it does a binary search for the proper insertion position, and then does the insertion.

10.     Modify the function so that it does not deal with the last i elements after the ith pass.

11.     Suppose input is given in the order

16, 18, 22, 15, 4, 50, 17, 31, 4, 90, 6, 25

Create the heap produced by

a.     The first heap creation algorithm of Section 8.4.

b.     The second heap creation algorithm of Section 8.4.

c.     How many comparisons and interchanges were required in Exercise 11(a) and Exercise 11(b)?

12.     For each heap of Exercise 11(a) and 11(b), produce

a.     The reheaped heap after the root element is removed using the first reheaping algorithm of Section 8.4.

b.     The reheaped heap after the root element is removed using the second reheaping algorithm of Section 8.4.

c.     How many comparisons and interchanges were required in Exercises 12(a) and12(b)?

13.     Why does the pointer array of Section 8.3.6 fail to help?

14.     Show the entries of the data array, for the input of Exercise 11, when the heapsort function is applied after the first three elements have been output.

15.     Modify the heapsort function so that it counts the number of interchanges and comparisons in each phase and returns their values as parameters.

16.     Modify the heapsort function so it sorts a data array of pointers to records instead of records themselves.

17.     What input sequence will cause the second heap creation algorithm to take worst-case time? Calculate the number of comparisons and interchanges it requires.

18.     Determine the number of comparisons and copies required by quicksort for an array of n records, which is (a) in sorted order; (b) in reverse sorted order; (c) in order so that partition always produces two component arrays whose lengths differ by at most 1.

19.     Modify partition so that it selects an entry at random to be used as the basis entry.

20.     Modify partition so that it selects the median of the first, middle, and last entries as a basis entry.

21.     Explain why the implementation given for quicksort requires a stack depth O(lg n).

22.     Modify quicksort so that it sorts all component arrays of length at most 20 by using insertionsort.

23.     Generation of n integers to fill the array a in Section 8.6 randomly was really only an approximation of a random input. This is because duplicate values might occur. While the effect of this duplication may be negligible for the simulation results, a "true" random input can be generated. You can fill array a as follows: For i from 1 to n, set a[i] to i.Then, for i from 1 to n, pick an integer int at random between i and n, and interchange a[i] and a[int].

a.     Convince yourself that this will generate a "'true" random input.

b.     Write a procedure to generate this "true" random input.

24.     Consider the "perfect shuffle" example of Chapter 3. When 2n - 1 is a prime number p, then the number of shuffles required is p - 1. The initial configuration is in sorted order, and so is the final configuration after p - 1 shuffles. The configurations produced along the way represent different permutations of the initial input. With successive reshufflings, the configurations seem more and more like a "random" arrangement, and then less and less like a "random" arrangement. Take p = 101 and use the one hundred configurations, produced by the hundred shuffles required to reproduce the original, as inputs to the heapsort algorithm. Output the statistics table based on one hundred samples of an input of size 101.

Suggested Assignments

1.     Write and run the heapsort function of Section 8.4.7. A main program should read in examples for the input array a. For each example, n and the array a should be printed, and heapsort should be called to sort a. When heapsort returns, the output array, sa, should be printed along with the values for int1,int2,comp1, and comp2. At least two examples for small n should be worked through by hand to test the correctness of heapsort.

2.     Write an efficient recursive function minheaps (t,count,flag) to return in count the number of subtrees of the binary tree pointed to by t that are minheaps, and to return in flag true if t points to a minheap and false otherwise.

Go to Chapter 9      Return to Table of Contents


Data Structures Using C: CHAPTER 9: MORE SEARCHING: INSERTION AND DEL

 Return to Table of Contents

CHAPTER 9: MORE SEARCHING: INSERTION AND DELETION

Deals with dynamic data structures, which

can grow and contract in size

support efficient searches

The data structures are

priority queues, which

support insertions and deletions

hash tables, which

support insertions and deletions

have excellent average search time

do not support ordered traversals

binary search trees, which

support searching, inserting, and deleting

have good average times for these operations

support ordered traversing

balanced binary search trees, which

support all the operations of binary search trees

also support fast access to information in an arbitrary position in a sorted ordering of the information stored

have good worst-case time for all the operations

Worst-case and average times are given for the operations on each data structure

9.1: Overview

In the previous chapter, techniques were presented for ordering and storing data so that it could be efficiently searched and traversed when desired. Those methods work well as long as the collection of data is essentially static--meaning that relatively few insertions and deletions are required. This is a general characteristic of arrays, which were the basis for the faster binary search: they support efficient search but not efficient insertion and deletion. Lists can be used for more efficient insertion and deletion, but then a binary search is not possible.

In this chapter the focus is on achieving efficient search and efficient insertions and deletion. Sometimes traversals through the data in sorted order by key value are also desirable. Four data structures that are appropriate for dealing with dynamic data are presented. First, priority queues are discussed. They provide the means to deal with a special case of insertion and deletion, as well as serving as an application of heaps. Next, hash tables are considered. Hash tables are important data structures that have excellent average search, insertion, and deletion times. Finally, binary search trees and AVL trees are introduced. These data structures support all the operations--search, insertion, deletion, and traversal--quite well.

9.2: Priority Queues

It often happens that there are a number of tasks, n, each with an associated priority. The n tasks are to be carried out in order of priority. When studying for examinations, most students review and prepare in an order of priority determined by the date of the exam. Most people read the chapters of a book in order, with priority determined by chapter numbering. In this case the situation is static, since the number of chapters does not change. Computer centers process submitted programs by attaching a priority to each and processing them in order of priority. Here the number of programs does change. This represents the usual case, where it is necessary to add new tasks with their priorities to the current collection. The general situation is covered by a priority queue.

A priority queue is a data structure with objects, their priorities, and two operations, insertion and deletion. Insertion in a priority queue means that an object with its priority is added to the collection of objects. Deletion from a priority queue means that an object with highest priority is deleted from the collection. A stack may be thought of as a priority queue with the highest priority object the one most recently added. A queue may be thought of as a priority queue with the highest priority object the earliest added.

High-level programming languages do not have priority queues directly available in their memories. It is important to be able to implement such queues, because they are of use in their own right and (as will be shown later) often appear as components of algorithms.

9.2.1 Simple Implementations

To implement a priority queue, first represent its objects and priorities as records with priority fields. Two ways to store the records come immediately to mind. The records can be stored in an array or in a list. In either case, the programmer may choose to store the records in arbitrary order or in order of priority. Table 9.1 gives the worst-case insertion and deletion times for each of the four resultant implementations, when n records are present in the priority queue. For example, when using the ordered array implementation, the top array record will have highest priority; thus deletion can be performed in constant time by using a pointer top to the current top record. Insertion of a record requires a search of the array to find its proper position. A binary search will allow this position to be found quickly, but to insert it requires O(n) time, because all records below it must be shifted down.

Table 9.1 Worst-Case Insertion and Deletion Times for the Priority Queue

           Implementation  Insertion  Deletion

----------------------------------------------

           Array           Constant   O(n)

Unordered  List            Constant   O(n)

           Array           O(n)       Constant

Ordered    List            O(n)       Constant

The table shows that if insertions are more frequent than deletions, it is better to use unordered records. In addition to considering the time for insertion and deletion, the amount of storage required for the list and array implementations must be evaluated. Arrays take less storage.

9.2.2 Using a Heap

Another way to implement a priority queue is to use a heap. You have seen how to build a heap efficiently so that storage is minimal, heap creation takes at most O(n) time, and reheaping takes at most O(lg n) time. To use a heap for the priority queue, the records are stored at the nodes of a heap that is ordered by priority field values. To insert a new record, place it as a new rightmost entry at the greatest depth of the heap. Then let it work its way up in the heap as in the first heap creation algorithm. Adding the new record in this way will take time at most O(lg n). To delete a record, simply delete the record at the root of the heap, and then reheap by invoking shift. This takes time at most O(lg n). The time to create the heap initially will be at most O(n) if the second heap creation algorithm is used. This is also the time that would be required to create an initial unordered array or unordered list of records. To create sorted arrays or lists takes time at most O(n lg n). Referring to Table 9.1, you can see that the heap implementation affords a compromise solution. With arrays and lists, one operation can be done quickly (in constant time) and the other in O(n) time. With the heap, both operations can be done in at most O(lg n) time.

9.2.3 Using Leftist Trees

Still another way to implement a priority queue is as a leftist tree, which is a null binary tree or a binary tree with the following properties:

1. It is a heap.

2. Its left and right subtrees are leftist trees.

3. The path that is obtained by starting from the root and branching to the right until a node with no right successor is reached is the shortest path from the root to any node with at most one successor.

For example, the tree (a) of Figure 9.1 violates condition 3, while the trees (b) and (c) are leftist trees.

Figure 9.1 One Nonleftist and Two Leftist Trees

As an exercise, you should prove by induction on the number of nodes, n, in a leftist tree, that the shortest path in the tree, from the root to a node with at most one successor, contains at most [image] lg (n + 1) [image] nodes. Such a shortest path can always be obtained by traversing the right chain from the root.

Leftist trees are implemented using a linked representation. (See Figure 9.2) Each node's record has an additional field, the distance field, containing the number of nodes along the right chain.

A leftist tree implementation of a priority queue has a number of pluses. Both additions and deletions can be done in constant time when the priority queue is acting as a stack. The time for these operations is O(lg n) for the general case. Also, two priority queues can be merged into one in O(lg n) time, where n is the total number of entries in the two priority queues to be merged. This is significantly faster merging than can be done with any of the previous methods, except for unordered lists. Crane [1972] introduced leftist trees as a data structure that can be used to implement priority queues. (See Knuth [1973b] for more details.)

Figure 9.2 The Merging of Leftist Trees Implemented in Linked Representation

The algorithm for merging two priority queues represented as leftist trees in O(lg n) time is as follows:

1. Make node.p the root of the merged tree, and merge q with p's right subtree.

2. Update the distance field of node.p.

3. If the distance of the root of the left subtree is not at least as great as the distance of the root of its right subtree, then interchange the two subtrees.

P is assumed to point to that root of the two leftist trees to be merged which contains the maximum key value, and q points to the other root. The result of merging the two trees p and q is shown as the tree r. The distance field values appear in parentheses. Notice in Figure 9.2 that, in effect, the two right chains of p and q are merged, with the other subtrees coming along for the ride. It is because only these two shortest paths participate in the algorithm that the time is O(lg n).

9.3: Hash Tables

The next data structure we consider is the hash table. A hash table is a data structure for the storage of records. It provides a means for rapid searching for a record with a given key value and adapts well to the insertion and deletion of records. Many compilers, for example, use hash tables to implement their symbol tables. They are also used extensively in systems where the primary objective is information retrieval. Thus nametable of Chapter 6 or a dictionary of words might be implemented in a hash table.

To illustrate the need for hash tables, suppose we want to store 1,000 records representing 1,000 customer accounts. Assume that each record has an account number as its key and, since we control account numbers, they range from 1 to 1,000. The solution is simple. Store the record with key value i in element i of an array data. Then, given a key value to search for (say, 836), go directly to element 836 of data to find it. What could be a quicker search? We are given a record and know immediately where to go to find it. The search time is a constant, independent of the number of records stored. However, this kind of search requires the ability to assign key values so that all keys are within a range of array indices (1 to 1,000 in this case). It also requires that we have enough storage to accommodate all records.

Now suppose we are dealing with records representing information about our employees, and that the keys are social security numbers. Even with only 1,000 employees, there are 109 possible social security numbers. It is not possible to know in advance which of the 109 social security numbers our 1,000 employees will actually have. If we were to allocate storage using social security numbers as addresses, we would run into two problems. First, there would not be enough storage. Second, if we did have sufficient storage, 999,999,000 (109 - 103) storage elements would be wasted! Nonetheless, it is extremely attractive to be able to look at the search key value and to know where to find it. This is the goal of hash tables and hash searching--to find a record directly, given its key value.

9.3.1 Building a Hash Table

This section deals with a scaled-down version of the social security example. There are eighteen records, each with a key value that is a three-digit number. The records and their key values will be given in some order, normally unpredictable. We are to build a table, implemented in an array, in which the eighteen key values will be stored, so that searching can be done efficiently. One way to do this is to build a special kind of table--a hash table. This table will then be searched for given key values. To build a hash table requires a hashing function and a collision resolution policy. A hashing function is a method of calculating an array or table address for any key. To be useful, the address must be quickly calculable.

The hash function should have other properties, as will become evident. The hash function used here assigns to any key the remainder after division of the key by the table size. If the table size is m and the key value is k, then k mod m denotes this remainder. The starting table or array size is 23.* The eighteen key values and their order are given in Table 9.2, along with the hashing function values assigned to each.

* With division-type hashing functions, the divisor, or table size, is usually a prime number greater than the number of entries to be placed in the table. We have taken this prime to be 23.

Table 9.2 Hash Addresses

               Hashing                Hashing

Key Value      Address  Key Value     Address

---------------------------------------------

   019     -->     19       468     -->    08

   392     -->     01       814     -->    09

   179     -->     18       720     -->    07

   359     -->     14       260     -->    07

   663     -->     19       802     -->    20

   262     -->     09       364     -->    19

   639     -->     18       976     -->    10

   321     -->     22       774     -->    15

   097     -->     05       566     -->    14

The hashing address for 019 is found, for example, by dividing 019 by the table size 23, to obtain 019 = 0 X 23 + 19. The remainder, 19, is the assigned hashing address. As each key value is input, its hashing address is calculated, and the key value is placed into that element of the table. After the first four key values are entered, the table is as shown in Figure 9.3(a).

When the fifth key value, 663, is input, its hashing address is 19, but element 19 of the table is already occupied by another key value. This event is called a collision. The collision resolution policy is the method used to determine where to store a value that has undergone collision. The most straightforward policy for resolving the conflict is called linear probing. It entails proceeding through the table from the collision element and placing 663, the colliding entry, into the first unoccupied element found. The hash table is assumed to have been initialized, so that each element was "unoccupied" or "empty." If the search for an empty element reaches the end of the table, it circles to the top of the table. In this case, 663 is entered at element 20. The final table is shown in Figure 9.3(b). A total of eight initial collisions occurred in building it.

Figure 9.3 Hash Table (a) after Four Entries and (b) Finally

Each time a location of the table is accessed in an attempt to insert a key value, one probe has been made. To enter the first key value required one probe; to enter the fifth key value required two probes. To enter keys 1 through 15 required, respectively, 1, 1, 1, 1, 2, 1, 4, 1, 1, 1, 2, 1, 5, 4, 7, 3, 1, 3 probes. You can build the table and confirm these probe numbers.

9.3.2 Searching a Hash Table

How would you search a hash table when asked to find a given key, such as 642? If the key were stored in the table, it would either be found at its hashing address or at the end of a linear probe. If its hashing address were empty, or if the search came to an empty element in the linear probe before finding the key, you might conclude that it was not stored in the table. In this case, the hash address of 642 is 21. Element 21 is not empty but does not have 642 as its value. If you do a linear probe, you will eventually come to element 3, which is empty, without finding 642; thus you would conclude that 642 is not in the table.

If the key searched for were 802, you could go to its hash address (20), find that element occupied with a different key, and start the linear probe, finding 802 at element 0.

Notice that if the search key value is present in the table, the search will require exactly as many probes to find it as were required to enter it into the table. The search traces out the same path. Instead, if the search is for a key value that is not present, the search will take exactly as many probes as would be required to enter it into the current table.

A search that finds a key value is called successful, and one that fails to find a key value is unsuccessful. Theoretically, one can calculate the number of probes required for every possible key value. Knowing the frequency with which each possible key value (there are 1,000 in our example) will be searched for, one can determine the average number of probes required to search the table. The maximum number of probes can always be determined given the table. In the example, a successful search would require at most seven probes, while an unsuccessful search (for example, for key value 248) could require nine probes. If one never had to search for a key not stored, and all stored keys were searched for with equal frequency, the average number of probes would be

  1/18 X 1 + 1/18 X 1 + 1/18 X 1 + 1/18 X 1 + 1/18 X 2 + 1/18 X 1

+ 1/18 X 4 + 1/18 X 1 + 1/18 X 1 + 1/18 X 1 + 1/18 X 2 + 1/18 X 1

+ 1/18 X 5 + 1/18 X 4 + 1/18 X 7 + 1/18 X 3 + 1/18 X 1 + 1/18 X 3

or 40/18 = 2.22 probes. This calculation involves multiplying the frequency of search for each record by the number of probes required to enter it, and then adding all these products. In this case, each frequency was 1/18, and the required probes were noted when the hash table was built.

If no collisions occurred in building the table, every successful search would take exactly one probe. If the hash function actually assigned every key value not stored in the table to an empty address, exactly one probe would be required for any search. This would be the ideal situation. It is apparent now that a desirable hashing function will "scatter" or distribute the key values throughout the table, so that relatively few collisions ensue.

The hash table was of length 23 and stored eighteen key values. Its usage ratio is 18/23; the table is 78 percent full. Intuitively, the lower this ratio, the greater the storage wasted, but the lower the likelihood of a collision. Appropriate hashing functions must be selected with care in real applications, but it is possible to achieve very quick searches using hash tables.

9.3.3 Random Hashing

A great deal of literature in computer science is devoted to the selection of hashing functions and collision resolution policies. To give some idea of what can be achieved, this section states results for an idealized scheme that assumes the hashing function to be perfectly random. Given any key value, the scheme assigns an address to that key value by picking randomly from among all the addresses of the table. This random selection is made so that each of the m addresses in a table of size m is equally likely to be selected. The table is built by inserting, in turn, each given key value. A key value is stored by determining its hash address. If that address is empty, the key value is stored there. If that address is occupied, the collision resolution policy works as follows. Another random hash function again selects an address at random. If that address is empty, the key value is stored there. If it is occupied, this procedure is repeated until the key value is finally entered. Thus an insertion of a key value may require the generation of a sequence of hash addresses ending with the actual final storage address.

After the table is built, searches are performed as follows. Trace the sequence of hash addresses that were generated to store the search key (if it is in the table) or that would have been generated (had the key been stored). The basic assumption made here is that the hash functions generate precisely the same sequence of hash addresses for the search as for building the table. This may seem like a strange way for random hash functions to work, but it is our assumption nevertheless. Table 9.3 shows the theoretical results for such a scheme, when n keys are stored in the table of size m.

The successful and unsuccessful search columns are given approximately by the functions (1/ [image] )1n 1/(1- [image]) and 1/(1- [image]) where [image] = n/m. The remarkable fact is that the average search time does not depend on n, the number of records stored in the table, but only on the usage ratio, n/m. Thus storage can be traded for speed in hash table searches. For example, if n is 10,000 and m is 11,112, [image] is about 0.90, and an average of only 2.56 probes will be required for a successful search. Contrast this with the 5,000 comparisons or probes needed for the linear search in the random case, or the 12.29 comparisons or probes needed for a binary search. Actual hashing functions and collision resolution policies can come close to achieving this kind of average behavior. The worst-case times for hash table searching can, however, be quite large.

Hash tables are not a searching panacea. A major disadvantage is that entries will not be in order by key. The only way to access the records in sorted order is to sort them. However, if this isn't important, hashing is often a good approach.

Table 9.3 Average Number of Probes for Searches

                 Successful  Unsuccessful

Usage Ratio n/m   Search       Search

-----------------------------------------

      .25          1.15         1.33

      .50          1.39         2.00

      .75          1.85         4.00

      .80          2.01         5.00

      .90          2.56        10.00

      .95          3.15        20.00

9.3.4 Other Collision Resolution Policies: Open Addressing and Chaining

The linear probing collision resolution policy is a special case of open addressing. The idea of open addressing is to resolve collisions by following a specific probe sequence whenever a collision occurs, in order to determine the insertion location for a new key. To be specific, let ki denote the key stored in the ith location of a hash table of size m. Assume that at least one table location is unoccupied and set all unoccupied locations to the "null" key value. This prevents "falling off" the table when searching for a key and reduces the search time, because an end-of-table search test need not be made.

Let h(k) denote the hash function value for search key k. The sequence of indexes determined by

[h(k) + i X p(k)]mod m* for i = 0, 1, 2, . . . , m - 1

is a probe sequence if the indexes are all distinct. This implies that, as i goes from 0 to m - 1, the generated indexes form a permutation of the indexes 0, 1, 2, . . . , m - 1. Consequently, in attempting a new key insertion, all locations of the hash table will be accessed in some order before deciding that the key cannot be inserted. It also ensures that, in attempting to search for a key, all locations of the table will be accessed in some order before concluding that the search key is not in the table, or until a null key is encountered.

* The formula states that the ith index in the probe sequence is found by calculating [i + p(k)]mod m. However, if the previous index, the (i - 1)th, is known, then simply adding p(k) to it and taking the result mod m will also yield the ith index.

Linear probing is the special case obtained by taking p(k) = 1 for all keys k. In general, a key is inserted by accessing the table locations given by its probe sequence, determined by [h(k) + i X p(k)]mod m as i goes from 0 to m - 1, until a null key is encountered. The key is then inserted in place of that null key. A search is done the same way. The algorithm for a hash table search using a probe sequence is as follows:

1. Set i to h(k)

2. While ki is not the search key k and ki is not the null key

set i to [i + p(k)]mod m.

3. If ki = k, then

set found to true

else

set found to false.

Notice that p(k) represents a displacement from the current probe address.

With linear probing, large clusters of keys stored in adjacent locations of the hash table tend to get larger as insertions are made. This is because large clusters have a greater probability of the new keys' hash address falling within the cluster, assuming a "random" hash function and "random" keys. A cluster gets larger when a new key is inserted at either end of the cluster, or when two separated clusters become connected because of the new key insertion. Both of these events occurred in the scaled-down social security number example. This phenomenon is called primary clustering. As the table usage ratio increases, primary clustering leads to greater search and insertion times. This is clearly undesirable. It happens because the large clusters must be traversed, with the average time being proportional to half the cluster length when keys are searched with equal frequency. Thus the insertion, or search for a key, begins to look like a linear search through a cluster.

To avoid primary clustering, try taking the next probe to be in a position removed from the current one instead of being adjacent to it. This is open addressing with displaced linear probing. Taking p(k) = [image] for some integer [image] that is not a factor of m will give such a probe sequence. To see this, take m = 13 and [image] = 5. The probe sequence is then generated by [h(k) + i X 5] mod 13 for i = 0, 1, 2, . . . , 12. If h(k) = 7, this gives the probe sequence

7, 12, 4, 9, 1, 6, 11, 8, 0, 5, 10, 2

Note that any key k whose hash address is 7 will have exactly the same probe sequence. Even worse, suppose a key's hash address is 11, and 11 is occupied upon its insertion. The next probe for this key will be in location [11 + 1 X 5] mod 13, which is location 3. A collision at 3 will result in a further probe at [3 + 1 X 5] mod 13 or 8, then 0, 5, etc. Thus the key that originally hashed to address 11 picks up at the intersection of 11 with the probe sequence for any key hashing to 7, and follows that same probe sequence from 11 onward. Similarly, any key that collides, upon insertion, with a key already occupying its hash address, will continue along this exact probe sequence from its point of entry.

In effect, there is exactly one probe sequence for all keys; they just enter, or merge into it, at different positions specified by their hash addresses. This means that, as a "cluster" builds up along this probe sequence (say, at 11, 3, 8, 0, and 5, as they become occupied), it has the same tendency to grow as a cluster of physically adjacent occupied addresses did with linear probing. This is just primary clustering, although the clusters do not appear in adjacent locations.

We need effective techniques for dealing with this problem. Figure 9.4 gives the hash tables that result from their application to our example. The hash tables produced by linear probing, and the displaced linear probing just discussed, appear in tables (a) and (b) of the figure, corresponding to [image] = 1 and [image] = 4, respectively. In addition, the number of probes required for the insertion of each key is indicated for each table. You should create these tables yourself to confirm their correctness and to aid in understanding the discussion.

The average number of probes for a successful search of this table is 1/18 X (11 X 1 + 5 X 2 + 1 X 3 + 1 X 12) = 36/18 or 2.00, which is an improvement over linear probing for this example. However, for the reasons stated previously, the theoretical average number of probes in a successful search will be the same as for linear probing.

How can we eliminate or break up large clusters to improve these results? It can be done using open addressing with secondary clustering. Consider the following analogy. Suppose there are m stores, each selling a certain specialty item, and each with exactly one such item in stock. Assume n people decide to buy that item the same day, and they, in turn, each select one of the m stores at random. If a customer finds the item in stock at the shop of his or her choice, he or she buys it. However, if someone has already bought the item, it will be out of stock, and the person may ask the store manager for directions to another store. Let the stores be numbered from 0 to m - 1. Whenever this occurs, suppose the manager of the ith shop sends the customer to the (i + 1)th shop. Of course, the manager of the (m - 1)th store refers people to the 0th store. The result of all this feverish activity is that all customers arriving at a particular out-of-stock store are sent to the same next store. Again, at that store, they are all sent to the same next store whenever the item has been sold. It seems clear that this results in primary clusters of customers being formed. The physical locations of the stores is irrelevant to this process.

       Hash         Hash       Hash       Hash       Hash       Hash

Keys  Address     Table (a)  Table (b)  Table (c)  Table (d)  Table (e)

-----------------------------------------------------------------------

019     19     0  802   (4)  663   (2)  364   (5)  364   (4)  392   (1)

392     01     1  392   (1)  392   (1)  392   (1)  392   (1)

179     18     2  364   (7)

359     14     3             321   (2)

663     19     4             364   (3)  566   (3)

262     09     5  097   (1)  097   (1)  097   (1)  097   (1)  097   (1)

639     18     6             720   (3)  260   (3)  663   (3)

321     22     7  720   (1)  720   (1)  639   (2)  720   (1)  720   (1)

097     05     8  468   (1)  468   (1)  468   (1)  468   (1)  468   (1)

468     08     9  262   (1)  262   (1)  262   (1)  262   (1)  262   (1)

814     09    10  814   (2)  976   (1)  976   (1)  814   (2)  976   (1)

720     07    11  260   (5)  260   (2)             976   (2)  364   (2)

260     07    12  976   (3)  566  (12)  814   (3)             321   (2)

802     20    13             814   (2)             566   (3)

364     19    14  359   (1)  359   (1)  359   (1)  359   (1)  359   (1)

976     10    15  774   (1)  774   (1)  774   (1)  774   (1)  774   (1)

774     15    16  566   (3)                                   566   (3)

566     14    17                        260   (4)  639   (3)  260   (4)

              18  179   (1)  179   (1)  179   (1)  179   (1)  179   (1)

              19  019   (1)  019   (1)  019   (1)  019   (1)  019   (1)

              20  663   (2)  802   (1)  663   (2)  663   (2)  806   (1)

              21  639   (4)             802   (2)  802   (2)  814   (2)

              22  321   (1)  639   (2)  321   (1)  321   (1)  639   (2)

                   linear    displaced  secondary   quadratic  double

                   probing   linear     clustering  residue    hashing

                             probing

Average number      2.22       2.00       1.89       1.72      1.61

of probes for a

succesful search

Figure 9.4 Typical Hash Tables for the Five Open-Addressing Techniques Considered (Usage Ratio 78 Percent)

Suppose each store manager, instead of giving identical directions to each would-be purchaser, sends customers to another store depending on which store they tried originally and on the number of stores they have already been to. The effect will be to disperse the unhappy customers. Rather than all these people being sent on to the same sequence of shops, only those with the same initial shop and the same number of previous attempts at buying are given identical directions. Clusters should still appear, but they should tend to be smaller than the primary clusters. This is, in fact, what happens, and the clusters that appear after such dispersion are called secondary clusters.

Translating this remedy to the problem involves choosing p(k) so that it becomes a function of h(k), the initial hash address of key k. This implies that only keys hashing to the same address will follow identical probe sequences. Unlike linear or displaced linear probing, keys whose probe sequences intersect do not merge into the same probe sequence at intersection. They go their separate ways. Again, to ensure that p[h(k)] generates a probe sequence, it is necessary to choose it carefully. As long as p[h(k)] is not a factor of m, for each key k, a probe sequence will result. If m is a prime number, this will always be the case when p[h(k)] is between 0 and m - 1.

As an example, take

Using our h(k) = k mod m and m = 13, any key with hash address 7 (say, key 137) has the probe sequence

7, 5, 3, 1, 12, 10, 8, 6, 4, 2, 0, 11, 9

Any key hashing to 11 now (say, key 258) has the probe sequence

11, 0, 2, 4, 6, 8, 10, 12, 1, 3, 5, 7, 9

Notice that a collision at 11, for such a key, does not result in the probe sequence for keys with hash address 7 being followed from its intersection with 11. It follows the sequence 11, 0, 2, . . . instead of 11, 9, 7, 5, . . . .

The hash table for this secondary clustering remedy is hash table (c) of Figure 9.4. The average number of probes for a successful search in this table is 1/18 X [(11 X 1) + (3 X 2) + (3 X 3) + (1 X 5)] = 31/18 or 1.72. This gives an improvement over hash tables (a) and (b), as expected. It occurs because the secondary clusters tend to be shorter than the primary clusters. Notice that the "physically adjacent" clusters in the table are deceiving; they are not relevant. It is the clustering along the probe sequences that is important. In linear and displaced linear probing there is, in effect, only one probe sequence. With the secondary clustering techniques, there is a distinct probe sequence for each distinct hash address.

Open addressing with the quadratic residue technique attempts to break up primary clusters in a similar way, although it generates a probe sequence somewhat differently. The probe sequence for key k starts with h(k) and follows with

[h(k) + i2]mod m, [h(k) - i2]mod m for i = 1, 2, . . . , (m - 1)/2

Again, to ensure that m distinct indexes are generated, care must be taken in the choice of m. If m is prime and of the form (4 X j) + 3, then m distinct indexes are guaranteed. Since m = 23 in our example, j = 5 and (m - 1)/2 = 11. The probe sequence for key k is then

h(k),

[h(k) + 12] mod 23, [h(k) - 12] mod 23,

[h(k) + 22] mod 23, [h(k) - 22] mod 23, . . . ,

[h(k) + 112] mod 23, [h(k) - 112] mod 23

If k is 364, h(k) = 19, and the probe sequence is

19, 20, 18, 0, 15, 5, 10, 12, 3, 21, 17, 9, 6, 22, 16, 12, 1, 8, 7, 4, 11, 2, 13

This is really a secondary clustering technique, since keys that hash to the same address will still have identical probe sequences. The hash table for this technique appears as hash table (d) of Figure 9.4, which gives an average number of probes of 1/18 X [(10 X 1) + (4 X 2 ) + (3 X 3) + (1 X 4)] = 31/18 or 1.72 for a successful search. When a key is inserted with these secondary clustering techniques, the result may be the lengthening or connecting of two clusters. These events may occur simultaneously along more than one probe sequence when the key is inserted.

Returning to the example of the out-of-stock specialty item, we might look for a remedy that avoids even these secondary clusters, since they also tend to increase search and insertion times. It seems reasonable to try to disperse these secondary clusters by giving each would-be customer a seperate set of directions. This may be done by having store managers randomly select the next store for a customer to try. This idea is known as open addressing with double hashing. The technique uses a second hash function for p(k). To be most effective, this should be as "random" as the initial hash function and should assign a value to key k by selecting this value independently of the value assigned to key k by the initial hash function.

The theoretical analysis for this technique has not been carried out as fully as for other methods, but the results indicate that double hashing gives the best results among the open addressing techniques. Empirically, it gives results that are the same as those for "random" hashing (see Table 9.3). This is somewhat suprising, since random hashing corresponds to our store remedy. Double hashing, instead of picking a store randomly, over and over again, until a customer is satisfied, merely picks an individual displacement at random for each key. This displacement then determines the probe sequence for a given hash address.

Care must be exercised in the selection of the second hash function so that a desirable probe sequence is generated. If m is a prime, then one possibility that ensures the generation of such a probe sequence is

For instance, p(657) = 11, since 657 = 50 X 13 + 7 when m = 13. Hence the probe sequence for 657 is

7, 5, 3, 1, 12, 10, 8, 6, 4, 2, 0, 11, 9

Key 137 also has hash address 7, but p(137) = 10, since 137 = 10 X 13 + 7. Its probe sequence, however, is distinct from 657's:

7, 4, 1, 11, 8, 5, 2, 12, 9, 6, 3, 0, 10

Now, even keys with the same hash address are unlikely to have the same probe sequence. The idea is that, statistically, probe sequences should appear to be approximately independently and randomly generated for two distinct keys. The hash table produced by this technique is given as hash table (e) of Figure 9.4, and the average number of successful probes is 1/18 X [(11 X 1) + (4 X 2) + (2 X 3) + (1 X 4)] = 29/18 or 1.61

As a further illustration, suppose key 582 is to be inserted in the hash tables of Figure 9.4. Its hash address, 582 mod 23, is 7, since 582 = 25 X 23 + 7. The sequence of probed locations and the number of probes required for each table are given below.

Linear probing    7, 8, 9, 10, 11, 12             (7)

Displaced linear

  probing         7, 11, 15, 19, 0, 4, 8, 12, 16  (9)

Secondary

  clustering      7, 18, 6, 17, 5, 16             (6)

Quadratic

  residue         7, 8, 6, 11, 3                  (5)

Double hashing    7, 9, 11, 13                    (4)

It is interesting that the average number of probes required to make an insertion is the same as the average number of probes required to carry out an unsuccessful search. This fact is useful in the analysis of hashing. The results of the example are typical of the behavior of open addressing techniques. As the usage ratio of the table increases, the secondary clustering and quadratic residue methods provide considerable improvement over the primary clustering methods. The double hashing techniques improve the results even more. For lower usage ratios, the improvements are much less noticeable. These improvements in average numbers of probes do not translate directly into similar improvements in average times, which depend on implementation details.

Chaining is another method of collision resolution. The idea is to link together all records that collide at a hash address. These may be kept separately or within the hash table itself. Separate chaining treats each location of the hash table as the head of a list--the list of all keys that collide at that location. Separate storage is used to store records of the list, which consist of key values and link pointer values. When a collision occurs, an insertion is made at the front of the corresponding list. Using this technique for our example yields the hash table shown in Figure 9.5 with corresponding numbers of probes (key comparisons). The table's average number of probes for a successful search is 1/18 X [(12 X 1) + (5 X 2) + (1 X 3)] = 25/18 or 1.39.

In contrast to open addressing, separate chaining allows clusters to form only when keys hash to the same address. Hence the lenghts of the lists should be similar to the lengths of clusters formed with secondary clustering techniques. A comparison of the 1.39 probes to the 1.89 or 1.72 probes with secondary clustering is somewhat unfair, since the hash table with separate chaining really uses additional storage. The actual amount of additional storage depends on the record lenghts. When they are small, and storage that would otherwise be unused is available for the link pointer values, then the usage ratio should more properly be measured as n/(n + m). This is 18/41, or 44 percent, in the example. When the link field takes up the same amount of storage as a record, then the usage ratio should more properly be measured as n/(2n + m). This is 18/59, or 31 percent, in the example.

Figure 9.5 Hash Table for Separate Chaining

Coalesced chaining maintains lists of records whose keys hash to the same address. These lists are stored within the hash table itself. Whenever two such lists "intersect," they coalesce. This is analogous to two separated clusters becoming connected when a new key is inserted in the location that separated them. Each hash table entry is now a record containing a key value and a link value to the next record on its associated list.

A key is inserted into the table by going to its hash address and traversing its list until a null key is found, the key itself is found, or no records remain on the list. If the null key is found, the new key is inserted in the record stored at that location, and its link field is set to null. If the key itself is found, no insertion is necessary. If no records remain on the list, then an unoccupied location must be found, the new key inserted there, and its link field set to null. This new record must then be appended at the end of the traversed list.

In order to create this new record to be appended, a pointer p is kept updated. Initially, p points to location m (which is just off the table). Whenever a new record must be found, p is decremented by 1 until the location it points to is unoccupied. If p "falls off" the table (becomes less than 0), then the table is full and no insertion is possible.

Figure 9.6 illustrates coalesced chaining for the example. P is initially 23 and the first four keys are inserted directly into their hash addresses. Key 663 collides at its hash address, 19. P is then decremented by 1 and, since location 22 is empty, 663 and a null next pointer are inserted there. This new record is then appended at the end of the list beginning in 19, by setting the link field of location 19 to 22. Key 262 is inserted directly into its hash address, 09. The next entry is 639, but its hash address, 18, contains an entry. P is then decremented by 1, and since location 21 is empty, 639 and a null link value is inserted there. This record is then appended at the end of the list beginning in 18 by setting the link field of location 18 to 21.

Figure 9.6 Hash Table for Coalesced Chaining

Figure 9.6 shows the resultant hash table, the number of probes for each key insertion, and the value of p after all eighteen keys are entered. The average number of probes for a successful search of this table is 1/18 X [(9 X 1) + (6 X 2) +(1 X 3) + (1 X 4) + (1 X 5)] = 33/18 or 1.81. If extra storage is available for the link fields, then n/m is a reasonable measure of the table usage here. These 1.81 probes are not as good as the 1.61 or 1.72 probes resulting from double hashing and quadratic residues in the example. Theoretical results, however, show that coalesced chaining does yield better average results than "random" hashing when storage is available for link pointer fields. You may expect coalesced chaining to give results that are not as good as separate chaining, better than linear or displaced linear probing, not as good as "random" or double hashing, but perhaps comparable to the secondary clustering techniques. If the link fields take additional storage, it should be taken into account in the usage factors for comparison. Note that coalesced chaining generally yields more but shorter lists (or clusters) than linear or displaced probing, because connecting or coalescing of clusters occurs to a lesser extent with coalesced chaining.

A third method of collision resolution will be discussed in Chapter 10 in connection with external memory, or secondary storage, where it is especially appropriate and useful.

9.3.5 Searching in Ordered Hash Tables

If keys are entered into a hash table in sorted order, open-addressing and chaining techniques will always result in searches along a probe sequence or along a list, in which the keys appear in the same or reverse order, respectively. In both cases, the search algorithm may be easily modified so that whenever a key is encountered that precedes the search key in the relevant ordering, the search is immediately terminated unsuccessfully. This can significantly decrease search times for unsuccessful searches but will not affect successful search times. You can generate the new hash tables (a) through (g) of Figures 9.4 to 9.6 to see the effect on the tables of the keys appearing in sorted order.

Of course, one cannot normally control the order in which keys to be inserted appear. It is not difficult, however, to modify the insertion algorithms so that the resultant hash table is exactly the same as the hash table resulting when keys are given in sorted order. The idea of the algorithm is clever and simple. For open addressing, simply insert a key as usual unless, in following a probe sequence, a key is encountered that precedes the new key in the ordering. Then the search key is inserted in that position, and the insertion proceeds as though the replaced key were being inserted using the displaced key instead of the new key. For separate chaining, it is necessary to traverse the list at which a collision occurs, and insert the new key in its proper position of the list.

This technique does not work directly for coalesced chaining, because the insertion of a new key in a list may start from the interior of the list when a collision occurs. In order to keep coalesced lists in order, it is necessary to find the first list record or be able to traverse the list in either direction. To achieve this, the lists might be implemented as circular lists or two-way lists. Circular lists take more traversal time but less storage than two-way lists.

Keeping ordered hash tables yields significant improvements in the average number of probes required for unsuccessful searches with open addressing. These improvements are especially striking for high usage ratios.

9.3.6 Deletion from Hash Tables

Deletion in hash tables using separate chaining is straightforward, since it amounts to the deletion of a record from a list. With open-addressing and coalesced chaining, the deletion of a key is not so straightforward, since a key of the table will be needed to reach another key when it is on that key's probe sequence or list. Deleting it would break this path. One solution to this problem is to mark a deleted key location as "deleted," so that the path is not broken. When a significant fraction of keys has been deleted, the actual usage ratio of the resultant table is low, but search times will still reflect the original higher usage ratio. New insertions, when encountering a "deleted" location, may recapture this wasted storage by occupying such "deleted" locations. However, if "deleted" locations build up, without being recaptured by insertions, search times deteriorate to a linear search for unsuccessful searches and are larger than necessary for successful searches. The only remedy is to rebuild the hash table. Although the algorithm is not given here, this may be done in place, by using only storage allocated to the table in order to carry out the creation of a new table. Hash tables can also be expanded or decreased in size, and recreated. Again, this can be done in place.

9.3.7 Hashing Overview

Let us review the advantages and disadvantages of hashing. First, note that some estimate is needed in advance for the maximum number of entries to be stored in a hash table, since its size must be declared. This is important; too high an estimate wastes space, and too low an estimate means a large usage ratio and poorer performance. Second, the hash function must be carefully chosen so that it can be computed quickly and generates relatively few collisions.

The main characteristic of hashing is its excellent average search, insertion, and deletion time. However, when deletions are frequent, care is required in deciding which collision resolution policy to use. The major disadvantages of hashing are the possibility of large worst-case times and the lack of support for traversal through the records in sorted order. Also, the only information obtained from an unsuccessful search is that the record isn't in the table. In the data structure considered next, more information may be obtained.

Choosing the appropriate hashing technique involves consideration of time and storage as well as the relative frequency with which successful searches, unsuccessful searches, and deletions will occur. However, some generalizations can be made.

Using open addressing to resolve collisions can be quite efficient for moderate usage ratios. Although the average times for successful searches depend on whether simple linear probing or more complex probing such as double hashing is used, the differences are small. These differences are more significant for unsuccessful searches, so more complex probing is desirable when they will occur frequently. Ordered hash tables can significantly reduce the search times in this case. For larger usage ratios the chaining methods have superior search times.

With open addressing, when deletions are frequent, the table can actually be storing relatively few entries, with many table positions simply marking deleted entries. Thus the table performance is governed by a usage ratio that does not reflect the actual, much smaller, number of positions used to store records. Separate chaining uses storage more efficiently in this case, and deletion is easy. Separate chaining also has better average time behavior than the other methods.

When records are small, the links required by chaining take significant extra storage compared to open addressing, whereas the opposite is true when records are large. When record size is moderate, the extra storage for links using separate chaining must be weighed against the reduced usage ratio that could be obtained if this storage were used in open addressing.

9.4: Binary Search Trees

We turn now to the next data structure: binary search trees. A binary search tree is a binary tree with the property that its left subtree contains records no larger than the root record, its right subtree contains records no smaller than the root record, and its left and right subtrees are also binary search trees. This is a recursive definition.

A binary tree fails to be a binary search tree if even one node contains a record in its left subtree or its right subtree that is, respectively, larger or smaller than the record stored at the node. Figure 9.7 has three examples of binary search trees containing the same fifteen records.

Note that a binary search tree need not be a heap, and that a heap need not be a binary search tree. In general, they are different kinds of binary trees. There are many binary trees that can store the same records. Any binary tree with n nodes can store n records in such a way that the tree is a binary search tree. (Just do an inorder traversal of the binary tree and insert the records in sorted order.)

9.4.1 Searching the Search Trees

In Chapter 7, three important traversals through the nodes of any binary tree were defined: 1) preorder, 2) inorder, and 3) postorder. In order to search a binary tree to determine if a particular key value is stored, any one of those traversals could be used to gain access to each node in turn, and the node tested to see if the search key value is there. This would be equivalent to a linear search.

Instead, the special nature of binary search trees facilitates highly efficient search. We simply compare the search key with the root key. If they match, the search is complete. If not, determine whether the search key is smaller than the key at the root. If so, then if it is in the tree at all, it must be in the left subtree. If it is larger than the root key, it must be in the right subtree if it is in the tree at all. Thus either the left or right subtree is eliminated from consideration. This procedure is repeated for the subtree not eliminated. Eventually, the procedure finds the key in the tree or comes to a null subtree and concludes that the key is not stored in the tree.

Figure 9.7 Three Binary Search Trees Containing the Same Data

Assume that nodes of the binary search tree are represented as follows.

typedef struct treenode

{

whatever info;

struct treenode *leftptr;

struct treenode *rightptr;

}binarytreenode, *binarytreepointer;

This special search may be implemented as follows:

>Comment

found = FALSE;

p = t;

pred = NULL;

while ((p != NULL) && (!found))

if (keyvalue == p->key)

found = TRUE;

else if (keyvalue < p->key)

{

pred = p;

p = p->leftptr;

}

else

{

pred = p;

p = p->rightptr:

}

Found indicates whether the search key value was present in the tree. If so, p points to the node in which the search key was found, and pred points to its predecessor. If the search key was not found, then pred points to the node that would have been its predecessor had it been in the tree. Note that this allows additional information to be obtained, if desired, beyond the fact that the search key was not found. The key less than the search key and the key greater than the search key can easily be determined, so that records closest to the desired one are known.

Notice that when (b) of Figure 9.7 is searched in this way, the search is equivalent to a linear search of an array in which the records are stored in sorted order. This search algorithm, applied to (c), would be equivalent to a binary search of an array in which the records are stored in sorted order. A search of (a) would fall somewhere between these two extremes.

What allows the search of Figure 9.7(c) to be equivalent to a binary search? It is the fact that the tree is "scrunched up" or near minimum possible depth, and not as straggly (unnecessarily deep) as the others. It has an intuitive kind of "balance," so that we eliminate from consideration about half the records each time we make a comparison. We are back to the game of "twenty questions.''

9.4.2 Growing the Search Tree "Simply"

This section develops an algorithm for creating or growing a binary search tree that stores records at its nodes. Imagine that you have already processed some records and created a binary search tree for them. Given the next input record, search the current tree for its key value. Assume no duplicates will occur in the tree, although this need not create any real problem, since a node can contain a pointer to a list, stored separately, of records with identical key values. Eventually, since the new record is not in the tree, a null subtree will be reached. However, if this null subtree is replaced by a node that stores this record, the resultant tree will still be a binary search tree. For instance, inserting 45 by this procedure in Figure 9.7(c) produces Figure 9.8. This gives an algorithm for growing a binary search tree. Start with the first input record at the root. For each additional input, search the tree. The search will reach a node whose left or right successor must be followed, but the successor is null. The record replaces that null subtree.

Figure 9.8 Binary Search Tree Created from Figure 9.7

This can be accomplished by using pred from the search implementation, and then invoking the following.

>Comment

p = malloc(sizeof(binarytreenode));

p->key = keyvalue;

p->leftptr = NULL;

p->rightptr = NULL;

if (keyvalue < pred->key)

pred->leftptr = p;

else

pred->rightptr = p;

We now have algorithms for growing, searching, and inserting records into a binary search tree.

9.4.3 The Shape of Simple Binary Search Trees

The binary search tree grown by the algorithm of the preceding section will have its shape determined by the input order of the records to be stored. If they are input in sorted order or reverse sorted order, the result is Figure 9.7(b) or its mirror image. If they are input in the order 55, 40, 83, 35, 51, 80, 85, 17, 37, 50, 52, 70, 81, 84, 100, the result is Figure 9.7(c). There are n! possible input orderings of n distinct keys. Some of these may give rise to the same binary search tree. Recall that the tree grown cannot have depth greater than n nor less than [image] lg(n + 1) [image] . Suppose the input is selected so that each of the n! possible orderings is equally likely to occur. This would be the case if the first input record is selected at random from among the n records, the second is selected at random from among the remaining n - 1 records, and so on. What will be the average depth of trees grown by the algorithm in this case? To calculate the average depth directly would mean to generate, for each of the n! orderings of input records, the binary search tree grown and note its depth. Then add up the n! depths and divide by n! to obtain the average depth.

It is difficult to analyze this problem using probability theory. However, the average depth has been shown to be O(c lg n), where c is about 4. The average search time can be calculated for a randomly grown binary search tree. It is the average time required to search for a record stored in the binary search tree, assuming each record is equally likely to be searched. This average search time is O(1.39 lg n). In the case of random input, the average search of the binary search tree grown by the algorithm will be only 39 percent more than the smallest possible average search time! Moreover, the algorithm for growing the tree will take an average time at most O(nlg n). If the records are given in sorted order instead of in random order, it will take the algorithm O(n2) time to grow the tree.

It is possible to grow "balanced" binary search trees, also called AVL trees (after the discoverers of these trees, Adelson-Velskii and Landis [1962]). The algorithm for growing AVL trees is slower and more complex, as is the insertion and deletion of records. However, they are guaranteed to have depth no greater than about 1.44 lg n. This is the worst-case depth. Consequently, AVL trees may be searched in time at most O(1.44 lg n). Such trees appear quite healthy and are never sparse. However it takes great care to grow healthy trees, as will be seen later.

9.4.4 Deleting a Record "Simply"

At this point all the important operations on binary search trees have been considered except deletion. The purpose of this section is to derive an algorithm for deleting a record from a binary search tree so that the remaining tree is still a binary search tree. To delete a terminal record, simply replace it by a null tree. To delete a record with only one successor, simply insert the successor's subtree at the position occupied by the deleted node. For instance, in the tree shown in Figure 9.9(a), deleting 51 and 71 yields the tree shown in Figure 9.9(b).

Consider Figure 9.9 again. A difficulty occurs if you wish to delete 80, a record with two successors. The node 80 must be replaced by a record that will allow the tree to retain its binary search tree property. This means that the record replacing 80 must be at least as large as any record in the left subtree of 80, and no larger than any record in the right subtree of 80. There are two choices: replace 80 with the largest record in its left subtree or the smallest record in its right subtree.

Notice that the largest record in any binary search tree is always found by going to its root and following right successors until a null tree is reached. The smallest record is found by going to its root and following left successors until a null tree is reached. Thus, in the example, we may take either 77's or 81's record to replace 80's record. The record selected, say 77, must then be inserted in place of 80's record, which is then effectively deleted. Taking 77 requires that 74's record replace it. The result is Figure 9.9(c). It should be clear that searching, inserting, or deleting, using the procedures in a search tree of depth d, will take worst-case time proportional to d.

Binary search trees are treated in this chapter as abstract objects. A sequential representation for deep but sparse trees requires array lengths proportional to 2d - 1. Deletions for this representation require a great deal of shifting of records. For instance, to delete 85 requires shifting 84, 83, and 81. To save storage, and to achieve faster insertions and deletions, requires using a linked representation.

(a) Before Deletion of Records

(b) After Deletion of 51 and 71

(c) After Deletion of 80

Figure 9.9 A Binary Search Tree before and after Deletion of Records

9.4.5 A Balancing Act

The "simple" algorithms for growing binary search trees and inserting and deleting records within them are relatively simple and have excellent time and storage requirements--on the average--when randomly done. These randomly grown trees can be fine for compiler symbol tables or for small data base applications such as dictionaries, but there are situations where good average behavior is not enough. Suppose your rocket ship's computer must search a binary search tree for retro-rocket firing instructions and must complete the search within a specific interval. This is not the time to find that the tree is out of shape and that a much longer than average time will ensue.

One answer is the AVL tree, which is discussed in this section. An AVL tree is a balanced binary tree that is also a binary search tree. (Balanced binary trees were defined in Chapter 7 in the discussion of Fibonacci trees.)

AVL trees have excellent worst-case time and storage requirements and allow access to the ith record to be performed quickly. The algorithms for the growth of such trees and insertion and deletion within them are considerably more complex than for "simple" binary search trees. Balanced trees are very useful whenever all four operations--searching, inserting, deleting, and accessing the ith record--are needed and worst-case rather than average time is important.

Because we want to use AVL trees to store records, it is desirable for an AVL tree to have as small a depth as possible and as many nodes (in which to store records) as possible for that depth. The "best" AVL tree of depth d is thus complete and has 2d - 1 nodes. A balanced binary search tree (AVL tree) of depth d, which has the least number of nodes among all AVL trees of depth d, represents the "worst" AVL tree of depth d. Such a tree will be a Fibonacci tree that is also a binary search tree. These trees represent the worst-case AVL trees. A Fibonacci AVL tree of depth d will store the minimum possible number of records.

Suppose nd represents the number of nodes of such a tree. Then any AVL tree with depth d must have a number of nodes n that is at least nd. It is possible to prove that nd is related to the (d + 2)th Fibonacci number, Fd+2. In fact, nd = Fd+2 - 1. A great deal is known about the Fibonacci numbers. In particular, [image], where [image]. It follows from this that [image]. Also, from Chapter 7, n [image] 2d - 1. After some manipulations we may conclude that

d < 1.44 1g (n + 2) - 0.328

This means that the depth of any AVL tree with n nodes is never greater than 1.44 1g (n + 2). In other words, an AVL tree storing n records has a worst-case depth O(lg n). Since the best possible depth of a binary tree storing n records is [image] lg (n + 1) [image] , the worst depth of an AVL tree storing n records is at most 44 percent more than the best achievable. Contrast this with the "simple" binary search tree for which the average depth is at least 39 percent more than the best achievable; it is clearly very desirable to be able to create and use AVL trees.

9.4.6 Maintaining Balance

Consider the AVL tree of Figure 9.10(a). To insert a new record into this tree, let us first do a "simple" insertion. The dashed branches indicate the sixteen possible insertion positions for a new node. You should confirm that, of these, only an insertion of a new node as a successor of 45, 53, 81, or 100 will cause the new tree to fail to remain an AVL tree. An insertion at node 45 causes the subtrees with roots 55, 40, and 70 to become unbalanced. For example, the subtree with root 40 would have a left subtree of depth 3, while its right subtree would have depth 5. These differ by more than 1. Notice that 55, 40, and 70 lie along the search path from the root to the new node inserted as a successor of 45. If 43 were inserted, then the result would be the tree of Figure 9.10(b). Thus the "simple" insertion discussed earlier will not retain the balance of a search tree.

This imbalance is remedied by rearranging the subtree with root 55 as shown in Figure 9.10(c). Notice that the subtrees with roots 40 and 70 are no longer out of balance. In fact, the tree of Figure 9.10(c) is now an AVL tree. Notice also that the subtree in Figure 9.10(c) replacing the subtree of 55 in Figure 9.10(b) has exactly the same depth as the original subtree of 55 in Figure 9.10(a). If any of the other insertions that would have produced an imbalance in Figure 9.10(a) had occurred, they could have been remedied by similar rearrangements. You should confirm this by trying them all and finding the suitable rearrangements. Remember, the rearrangements must not only rebalance but also preserve the binary search tree property.

It is possible to analyze the general situation to determine exactly when an insertion will cause an imbalance and what its remedy will be. An insertion into an AVL tree with zero or one node cannot cause an imbalance. When the tree has more than one node, the types of imbalance, and their corresponding remedies, reduce to the four indicated in Figures 9.11 and 9.12.

Each rectangle in Figures 9.11 and 9.12 represents an AVL tree. Any of these may be null. An X represents the position of the inserted new record that has caused imbalance. Three X's appear in imbalances LR and RL, actually representing three possibilities, although only one will have taken place. The LR and RL imbalances are meant to include the special case where B is actually the inserted node. This is why the X appears next to node B. Subtrees 1, 2, 3, and 4 must then be null. This fact is noted here, since it is not clear from the figures. In cases LL and RR, the subtree with root B represents the node closest to the inserted node, along the path from the inserted node to the root, where an imbalance occurs. In cases LR and RL the subtree with root C represents this node. In all remedies, the subtrees represented by the rectangles retain their relative positions. Note that all remedies preserve the binary search tree property. It is very important to see that, in each case, the depth of the remedy is exactly the same as the depth of the original subtree before the new insertion was made. Consequently, no nodes on the path from the subtree root, B in cases LL and RR, and C in cases LR and RL, imbalanced by the new insertion, will be imbalanced after the remedy is applied. It is also important to see that these remedies are applied locally; they do not involve the creation of a new AVL tree in its entirety.

Figure 9.10 An AVL Tree before and after Insertion

In Figure 9.11, when 43 was inserted, case LL occurred. The roles of B and A were taken by 55 and 50, respectively. The three rectangles, 1, 2, 3 of Figure 9.11 (a and b), represented, respectively, the subtrees shown in Figure 9.13. The X represented 43.

(a) Imbalance LL

(b) Remedy for LL

(c) Imbalance LR

(d) Remedy for LR

Figure 9.11 Two Imbalances and Their Remedies

Consider imbalance LL. The remedy looks as though A has moved up to the root and B has moved to the right. This is called a rotation to the right of the subtree with B at its root. The remedy for imbalance RR would be described as a rotation to the left of the subtree with B at its root. To rectify imbalance LR, perform a rotation to the left of the subtree with root A to obtain Figure 9.14(a). A further rotation to the right of the subtree rooted at C produces (b) of Figure 9.14. Assume that rotations are accomplished by the procedures rotateleft and

(a) Imbalance RR

(b) Remedy for RR

(c) Imbalance RL

(d) Remedy for RL

Figure 9.12 Two More Imbalances and Their Remedies

Figure 9.13 Rectangles 1-3 of Figure 9.12(a) and 9.12(b)

Figure 9.14 The Double Rotation Producing Remedy LR

rotateright, each with a parameter that points to the root of the subtree to be rotated. Then the remedies for each imbalance may be obtained by

LL:  rotateright(plast)             where last points to B

RR:  rotateleft(plast)              where last points to B

LR:  rotateleft(p(last.leftptr))    where last points to C

     rotateright(plast)

RL:  rotateright(p(last.rightptr))  where last points to C

     rotateleft(plast)

The procedure for rotateright, for example, might be as follows:

rotateright(plocalroot)

/* Performs a rotation to the right

of the subtree whose root is

pointed to by localroot.

*/

binarytreepointer *plocalroot;

{

binarytreepointer q;

q = (*plocalroot).leftptr;

(*plocalroot).leftptr = q->rightptr;

q->rightptr = *plocalroot;

*plocalroot = q;

}

Note that the procedure also changes the pointer in localroot, so it points to the root of the new subtree. It is assumed that nodes of the AVL tree are represented as follows:

typedef struct treenode

{

whatever key;

struct treenode *leftptr;

struct treenode *rightptr;

int balance;

}binarytreenode,*binarytreepointer;

The balance field stores the node's right subtree depth minus the node's left subtree depth. In an AVL tree it will have values - 1, 0, or + 1.

Consider the binary tree shown in Figure 9.15(a), which was balanced until the new node, marked by X, was inserted. The initial balance values are also shown. Because of the insertion, the root node and node C have become unbalanced. Since node C is closest to the inserted node, we remedy this imbalance by applying the remedy for RL to obtain Figure 9.15(b). The tree itself is now balanced, but the balances of nodes A, B, and C are now incorrect, as are the balances of the three nodes between X and A. The last node, however, does have the correct balance value.

In general, after rebalancing, one of the nodes involved in the remedy will be the new root of the subtree that was rebalanced (B in the example). If the remedy was LL or RR, all nodes along the path from this node to the inserted node, starting with its successor, need their balances reset. If the remedy was LR or RL, then all nodes along this path, starting with its successor's successor (the successor of A in the example), need their balances reset. The node not on this path involved in the remedy (C in the example) must have its balance reset, and, finally, the new root of the rebalanced subtree must have its balance set to 0. The special case when the inserted node is at B requires only that C's balance be set to 0. All nodes above this new root need not be reset; they retain their original values. Any time the root node is involved, special processing is required, since the head of the tree must be modified rather than a left or right subtree pointer of a node.

The algorithm for inserting a node into an AVL tree can now be written.

1. Search for the keyvalue.

2. If it is not found, then

a. Create a new node for it and set its field values.

b. Insert the new node into the tree.

c. If the tree has more than one node, then

If it is not necessary to rebalance, then reset last's balance and the balances of nodes on he search path between last and the new node.

If it is necessary to rebalance, then

i. apply the appropriate remedy to rebalance, reset the balance of the node involved in the remedy but not on the new search path, reset the proper new search path balances, set the balance of the root of the subtree at which rebalancing occurred to 0.

ii. If this root was the root of the tree itself, then

set the head of the tree to this root; otherwise

set its predecessor to this root;

else

reset the balance of the predecessor of the inserted node.

It has already been noted that, when rebalancing is necessary, once the subtree whose root is nearest the newly inserted node has been rebalanced with the correct remedy, all other subtrees along the path to the root of the entire tree will also be back in balance. Again, this is because the smallest subtree causing the imbalance has been modified by the remedy to have its original depth. All the subtrees were originally balanced when it had that depth, so they must be balanced now. During the search in step 1 for the insertion key, suppose you keep a pointer last. Last is updated to point to the node that may become imbalanced on the search path nearest the insertion node. Only nodes with balance values of + 1 or -1 may become imbalanced due to the insertion. It is necessary to rebalance only if last.balance was -1 and the search path went left at last, or last.balance was + 1 and the search path went right at last, and p is not t, and pred.balance = 0. Only nodes on the search path, between the inserted node and last, will need to have their balance fields updated.

(a) Unbalanced

(b) Balanced

Figure 9.15 Rebalancing a Binary Tree

During the search, an updated pointer to the predecessor node of last should be kept in predlast. P will point to the node whose key field is currently being compared with the search key, and predp points to its predecessor. An implementation of the AVL tree insertion is given in the insertavl function. For clarity, the procedures createnode and search are then given separately, followed by procedures insertnode and resetpathbalances.

Nonrecursive AVL Insertion

#define TRUE 1

#define FALSE 0

#define NULL 0

insertavl(keyvalue,pt)

/* Inserts a new record with key equal to

keyvalue in the AVL tree t, if a record

with this keyvalue is not present in t.

After insertion t will still be an AVL tree.

*/

whatever keyvalue;

binarytreepointer *pt;

{

binarytreepointer p,predp,last,predlast,q;

int found;

    >Comment

search(keyvalue,*pt,&p,&predp,

&last,&predlast,&found);

    >Comment

if(!found)

{

    >Comment

createnode(keyvalue,&p);

    >Comment

insertnode(keyvalue,p,predp,pt);

    >Comment

if(p != *pt)

if(predp->balance == 0)

if(keyvalue < last->key)

if(last->balance == 0)

    >Comment

{

last->balance = -1;

resetpathbalances(keyvalue,

last->leftptr,p);

}

else if(last->balance == +1)

    >Comment

{

last->balance = 0;

resetpathbalances(keyvalue,

last->leftptr,p);

}

else

    >Comment

{

    >Comment

q = last->leftptr;

if(keyvalue < q->key)

ll(keyvalue,&last,p);

else

lr(keyvalue,&last,&q,p);

last->balance = 0;

    >Comment

if(predlast == NULL)

*pt = last;

else if(keyvalue < predlast->key)

predlast->leftptr = last;

else

predlast->rightptr = last;

}

else

if(last->balance == 0)

    >Comment

{

last->balance = +1;

resetpathbalances(keyvalue,

last->rightptr,p);

}

else if(last->balance == -1)

    >Comment

{

last->balance = 0;

resetpathbalances(keyvalue,

last->rightptr,p);

}

else

    >Comment

{

    >Comment

q = last->rightptr;

if(keyvalue < q->key)

rr(keyvalue,&last,p);

else

rl(keyvalue,&last,&q,p);

last->balance = 0;

    >Comment

if(predlast == NULL)

*pt = last;

else if(keyvalue < predlast->key)

predlast->leftptr = last;

else

predlast->rightptr = last;

}

else

    >Comment

predp->balance = 0;

}

}

ll(keyvalue,plast,p)

/* Applies remedy ll, resets the balance of the node

involved in the remedy but not on the new search

path, and resets the proper new search path balances.

*/

whatever keyvalue;

binarytreepointer *plast,p;

{

rotateright(plast);

((*plast)->rightptr)->balance = 0;

resetpathbalances(keyvalue,(*plast)->leftptr,p);

}

lr(keyvalue,plast,pq,p)

/* Applies remedy lr, resets the balance of the node

involved in the remedy but not on the new search

path, and resets the proper new search path balances.

*/

whatever keyvalue;

binarytreepointer *plast,*pq,p;

{

rotateleft(&((*plast)->leftptr));

rotateright(plast);

if(keyvalue < (*plast)->key)

{

(*pq).balance = 0;

((*plast)->rightptr)->balance = +1;

resetpathbalances(keyvalue,

((*plast)->leftptr)->rightptr,p);

}

else if(keyvalue > (*plast)->key)

{

(*pq)->balance = -1;

((*plast)->rightptr)->balance = 0;

resetpathbalances(keyvalue,

((*plast)->rightptr)->leftptr,p);

}

else

((*plast)->rightptr)->balance = 0;

}

rr(keyvalue,plast,p)

/* Applies remedy rr, resets the balance

of the node involved in the remedy but

not on the new search path, and resets

the proper new search path balances.

*/

whatever keyvalue;

binarytreepointer *plast,p;

{

rotateleft(plast);

((*plast)->leftptr)->balance = 0;

resetpathbalances(keyvalue,(*plast)->rightptr,p);

}

rl(keyvalue,plast,pq,p)

/* Applies remedy rl, resets the balance

of the node involved in the remedy but

not on the new search path, and resets

the proper new search path balances.

*/

whatever keyvalue;

binarytreepointer *plast,*pq,p;

{

rotateright(&((*plast)->rightptr));

rotateleft(plast);

if(keyvalue > (*plast)->key)

{

(*pq)->balance = 0;

((*plast)->leftptr)->balance = -1;

resetpathbalances(keyvalue,((*plast)->rightptr)->leftptr,p);

}

else if(keyvalue < (*plast)->key)

{

(*pq)->balance = +1;

((*plast)->leftptr)->balance = 0;

resetpathbalances(keyvalue,((*plast)->leftptr)->rightptr,p);

}

else

((*plast)->leftptr)->balance = 0;

}

rotateright(plocalroot)

/* Performs a rotation to the right

of the subtree whose root is

pointed to by localroot.

*/

binarytreepointer *plocalroot;

{

binarytreepointer q;

q = (*plocalroot)->leftptr;

(*plocalroot)->leftptr = q->rightptr;

q->rightptr = *plocalroot;

*plocalroot = q;

}

rotateleft(plocalroot)

/* Performs a rotation to the left

of the subtree whose root is

pointed to by localroot.

*/

binarytreepointer *plocalroot;

{

binarytreepointer q;

q = (*plocalroot)->rightptr;

(*plocalroot)->rightptr = q->leftptr;

q->leftptr = *plocalroot;

*plocalroot = q;

}

resetpathbalances(keyvalue,start,p)

/* Resets the balances of all nodes on

search path between the nodes

pointed to by start and p.

*/

whatever keyvalue;

binarytreepointer start,p;

{

binarytreepointer q;

q = start;

while(q !=p)

if(keyvalue < q->key)

{

q-balance = -1;

q = q->leftptr;

}

else

{

q->balance = +1;

q = q->rightptr;

}

}

createnode(keyvalue,pp)

/* Returns with p pointing to a new node

record with key equal to keyvalue,

left and right pointer fields null, and

balance field zero.

*/

whatever keyvalue;

binarytreepointer *pp;

{

*pp = malloc(sizeof(binarytreenode));

(*pp)->key = keyvalue;

(*pp)->leftptr = NULL;

(*pp)->rightptr = NULL;

(*pp)->balance = 0;

}

search(keyvalue,t,pp,ppredp,plast,ppredlast,pfound)

/* Returns with found true only if the binary search

tree t has a node with its key equal to keyvalue.

If found is true

p and predp point respectively, to

the found node and its predecessor,

else

predp, and last, will point, respectively,

to the node that will be the new node's

predecessor, and the node along the search

path closest to the new node's insertion point

that may become unbalanced.

*/

whatever keyvalue;

binarytreepointer t,*pp,*ppredp,*plast,*ppredlast;

int *pfound;

{

*pfound = FALSE;

*pp = t;

*plast = t;

*ppredp = NULL;

*ppredlast = NULL;

while((*pp != NULL) && (!*pfound))

if(keyvalue < (*pp)->key)

{

if((*pp)->balance != 0)

{

*ppredlast = *ppredp;

*plast = *pp;

}

*ppredp = *pp;

*pp = (*pp)->leftptr;

}

else if(keyvalue > (*pp)->key)

{

if((*pp)->balance != 0)

{

*ppredlast = *ppredp;

*plast = *pp;

}

*ppredp = *pp;

*pp = (*pp)->rightptr;

}

else

*pfound = TRUE;

}

insertnode(keyvalue,p,predp,pt)

/* Inserts the new node, pointed to

by p, as the proper successor

of predp.

*/

whatever keyvalue;

binarytreepointer p,predp,*pt;

{

if(predp == NULL)

*pt = p;

else

if(predp == *pt)

if(keyvalue < (*pt)->key)

(*pt)->leftptr = p;

else

(*pt)->rightptr = p;

else

if(keyvalue < predp->key)

predp->leftptr = p;

else

predp->rightptr = p;

}

As another illustration of recursion, a recursive version is presented next. It is based on the following formulation.

To insert a node in t:

If t is null then

create the new node, set its field values and set t to it

else

if keyvalue < t.key, then

insert the node in t.leftptr

if t is balanced, then

reset its balance

else

rebalance t and reset the balances of the nodes involved in the

remedy

else if keyvalue > t.key then

insert the node in t.rightptr

if t is balanced, then

reset its balance

else

rebalance t and reset the balances of the nodes involved in the

remedy.

The recursive insertavl procedure is as follows:

Recursive AVL Insertion

#define TRUE 1

#define FALSE 0

#define NULL 0

insertavl(keyvalue,pt,pincrease)

/* Inserts a new record with key value to

keyvalue in the AVL tree t, if a record

with this keyvalue is not present in t.

After insertion t will still be an AVL tree.

Increase must be zero when the function is invoked.

*/

whatever keyvalue;

binarytreepointer *pt;

int *pincrease;

{

binarytreepointer q;

    >Comment

if(*pt == NULL)

{

    >Comment

createnode(keyvalue,pt);

    >Comment

*pincrease = 1;

}

    >Comment

else if(keyvalue < (*pt)->key)

{

    >Comment

insertavl(keyvalue,

&((*pt)->leftptr),pincrease);

    >Comment

if(*pincrease) == 1)

if((*pt)->balance == 0)

    >Comment

(*pt)->balance = -1;

else if((*pt)->balance == +1)

{

    >Comment

(*pt)->balance = 0;

    >Comment

*pincrease = 0;

}

else

{

    >Comment

q = (*pt)->leftptr;

if(keyvalue < q->key)

ll(pt);

else

lr (pt,q);

    >Comment

(*pt)->balance = 0;

*pincrease = 0;

1 t's depth did not increase

}

}

    >Comment

else if(keyvalue > (*pt)->key)

{

    >Comment

insertavl(keyvalue,

&((*pt)->rightptr),pincrease);

    >Comment

if(*pincrease == 1)

if((*pt)->balance == 0)

    >Comment

(*pt)->balance = +1;

else if((*pt)->balance == -1)

{

    >Comment

(*pt)->balance = 0;

    >Comment

*pincrease = 0;

}

else

{

    >Comment

q = (*pt)->rightptr;

if(keyvalue > q->key)

rr (pt)

else

rl(pt,q);

    >Comment

(*pt)->balance = 0;

    >Comment

*pincrease = 0;

}

}

}

ll(pt)

/* Apply remedy ll and reset the

balance of the node involved in

the remedy.

*/

binarytreepointer *pt;

{

rotateright(pt);

reset1balance((*pt)->rightptr);

}

lr(pt,q)

/* Apply remedy lr and reset the balances

of the two nodes involved in

the remedy.

*/

binarytreepointer *pt,q;

{

rotateleft(&((*pt)->leftptr));

rotateright(pt);

reset2balances(q,*pt,(*pt)->rightptr);

}

rr(pt)

/* Apply remedy rr and reset the

balance of the node involved in

the remedy.

*/

binarytreepointer *pt;

{

rotateleft(pt);

reset1balance((*pt)->leftptr);

}

rl(pt,q)

/* Apply remedy rl and reset the balances

of the two nodes involved in the remedy.

*/

binarytreepointer *pt,q;

{

rotateright(&((*pt)->rightptr));

rotateleft(pt);

reset2balances(q,*pt,(*pt)->leftptr);

}

rotateright(plocalroot)

/* Performs a rotation to the right

of the subtree whose root is

pointed to by localroot.

*/

binarytreepointer *plocalroot;

{

binarytreepointer q;

q = (*plocalroot)->leftptr;

(*plocalroot)->leftptr = q->rightptr;

q->rightptr = *plocalroot;

*plocalroot = q;

}

rotateleft(plocalroot)

/* Performs a rotation to the left

of the subtree whose root is

pointed to by localroot.

*/

binarytreepointer *plocalroot;

{

binarytreepointer q;

q = (*plocalroot)->rightptr;

(*plocalroot)->rightptr = q->leftptr;

q->leftptr = *plocalroot;

*plocalroot = q;

}

createnode(keyvalue,pp)

/* Returns with p pointing to a new node

record with key equal to keyvalue,

left and right pointer fields null, and balance field zero.

*/

whatever keyvalue;

binarytreepointer *pp;

{

*pp = malloc(sizeof(binarytreenode));

(*pp)->key = keyvalue;

(*pp)->leftptr = NULL;

(*pp)->rightptr = NULL;

(*pp)->balance = 0;

}

In order to determine the balance of t upon return from the recursive calls (these are underlined in the algorithm), we will need to know if the subtree in which the node was inserted has increased in depth. A flag increase, with values 1 or 0, indicates an increase. Increase may be nonlocal to, or a parameter of, the function, but it must be initialized to 0 before the function is invoked. We make it a parameter.

The functions for resetting balances are

reset1balance(q)

/* Sets the balance of node

q to zero.

*/

binarytreepointer q;

{

q->balance = 0;

}

reset2balances(q,t,p)

/* Resets the balance of nodes

p and q determined by the

balance of t.

*/

binarytreepointer q,t,p;

{

if(t->balance == -1)

p->balance = +1;

else

p->balance = 0;

if(t->balance == +1)

q->balance = -1;

else

q->balance = 0;

}

Suppose you wanted to create an AVL tree by reading a sequence of key values and to test the AVL insertion. The code below, when insertavl and the functions it invokes are included, creates the AVL tree using the recursive version of insertavl. It prints the key field values of the resultant tree nodes in preorder access order as the tree is created, and prints both the key and balance values in preorder access order for the final tree. The preorder traversal routine calls a process routine, as in Chapter 7, that prints out the key and balance field values of a node it is called to work on.

#include <stdio.h>

#define NULL 0

typedef int whatever;

typedef struct treenode

{

whatever key;

struct treenode *leftptr;

struct treenode *rightptr;

int balance;

}binarytreenode,*binarytreepointer;

binarytreepointer left(p)

/* Returns a copy of the left pointer

of the node pointed to by p.

*/

binarytreepointer p;

{

return(p->leftptr);

}

binarytreepointer right(p)

/* Returns a copy of the right pointer

of the node pointed to by p.

*/

binarytreepointer p;

{

return(p->rightptr);

}

info(l)

/* Returns the key field value of

the node pointed to by l.

*/

binarytreepointer l;

{

return(l->key);

}

bal(l)

/* Returns the balance field value of

the node pointed to by l.

*/

binarytreepointer l;

{

return(l->balance);

}

binarytreepointer setnull()

/* Returns a null pointer. */

{

return(NULL);

}

The function insertavl and all its invoked functions should go here, since they depend on the tree's implementation.

#define SENTINEL -1

main()

/* Reads record information, creates an AVL tree

storing the records, prints the key field values

of the tree records in preorder access order

after each record is inserted in the tree, and

prints the key & balance field values of the

final tree in preorder access order.

*/

{

binarytreepointer t,setnull();

int keyvalue,increase;

    >Comment

t = setnull();

printf(\" enter keyvalue or sentinel \n");

    >Comment

scanf("%d",&keyvalue);

while(keyvalue != SENTINEL)

}

increase = 0;

    >Comment

insertavl(keyvalue,&t,&increase);

printf("\n enter keyvalue or sentinel \n");

    >Comment

scanf("%d",&keyvalue);

    >Comment

printtree(t);

}

    >Comment

preorder (t);

}

printtree(t)

/* Prints the keys of t in

preorder access order.

*/

binarytreepointer t,left(),right();

{

printf("\n %d \n",info(l));

printtree(left(t));

printtree(right(t));

}

preorder(t)

/* Prints the keys & balances of

t in preorder access order.

*/

binarytreepointer t,left(),right(),setnull();

{

if(t == setnull())

printf("\n The tree is null \n");

else

{

process(t);

preorder(left(t));

preorder(right(t));

}

}

process(t)

/* Prints the key and balance fields of

the node pointed to by t.

*/

binarytreepointer t;

{

printf("\n key:%d balance:%d \n",info(t),bal(t));

}

Notice that all components of the program except insertavl and its associated functions are written treating the binary tree as a data abstraction.

Deletion procedures for AVL trees are not developed in detail here, but the basic ideas will be illustrated. Consider the AVL tree of Figure 9.16(a). The tree is a Fibonacci tree of depth 5. Suppose the rightmost node of its right subtree, 12, is deleted, as in a simple binary search tree. The result is Figure 9.16(b).

Figure 9.16  A Fibonacci Tree of Depth 5 and Deletion

Figure 9.16

Node 11, the predecessor of 12, is now unbalanced. Rebalancing can be achieved by applying remedy LR at node 11 to obtain Figure 9.16(c). Node 8, the predecessor of 11 before rebalancing, is now unbalanced. Rebalancing can be done by applying remedy LL at node 8 to obtain Figure 9.16(d).

This tree is finally an AVL tree. Deleting the rightmost node from any such Fibonacci tree will always lead to a rebalancing at every node along the search path.

The basic algorithm for deletion of a node from an AVL tree is first to delete, as in a simple binary tree. Then the search path must be retraced, from the deleted node back to the root. If an imbalance occurs, it must be remedied. Deletion can require O(1g n) rebalances, while insertion can require at most one rebalance.

A recursive deletion algorithm is sketched below.

To delete a node from t:

If t is not null, then

if keyvalue<t.key, then

delete the node from t.leftptr

if t is balanced, then

reset its balance

else

rebalance t and reset balances

else if keyvalue>t.key, then

delete the node from t.rightptr

if t is balanced, then

reset its balance

else

rebalance t and reset balances

else

delete the node with the largest key value in t.leftptr

replace the root of t by that deleted node

if t is balanced, then

reset its balance

else

rebalance t and reset balances.

9.4.7  Access By Order in AVL Trees

Given a key value to find, it is easy to search the AVL tree for it. However, consider the records stored at the nodes as ordered by their processing positions in an inorder traversal of the AVL tree. For instance, in the AVL tree of Figure 9.10, the ordering is

1.  17   9.  60

2.  35  10.  70

3.  37  11.  80

4.  40  12.  81

5.  45  13.  83

6.  50  14.  85

7.  53  15.  100

8.  55

By access by order to a node is meant that, given an index (say, 10) produce the tenth record (key value 70 above). The difficulty is that it is not immediately apparent where node 10 is nor what its key value is. One way to access this node is simply to inorder traverse the tree, counting nodes. Arriving at the tenth node, produce it. This will take time O(i) if the index is i. In the worst case this is O(n), if n nodes are stored in the tree. Since AVL trees with n nodes have maximum depth 1.44 lg n, any key search, insertion, or deletion can be done in at most O(1g n) time. There is also a simple way to access by order, or index, in at most O(1g n) time. To accomplish this, add an index field to each node. It will contain an integer that is 1 plus the number of nodes in its left subtree. The index field thus contains the nodes' index. For example, the index field value of 70 would be 10. It is then possible to search the tree for an index value, much as it is searched for a key value. The index search thus takes time at most O(1g n). Of course, the insertion and deletion algorithms must be modified to keep the index fields updated correctly.

Creating an AVL tree is done by growing it as was done for the simple binary search tree: start with a null tree and insert each input record, one at a time. The difference is that a "simple" insertion is used to grow the simple binary search tree, whereas the insertavl procedure is used to grow an AVL tree. The AVL tree may be grown in O(n lg n) time. Also, since it may then be inorder traversed in O(n) time, we have another way to sort in O(n lg n) time.

9.4.8  Binary Search Tree Overview

Easily constructed binary search trees allow efficient insertion and deletion and can also be searched in average time O(1g n). More complex construction, insertion, and deletion algorithms for balanced binary search trees guarantee search, insertion, and deletion in worst-case time O(1g n). Binary search trees allow easy access to records in sorted order, and balanced binary search trees also allow O(lg i) access to the ith record in sorted order. It is not necessary to estimate in advance how many entries will be made in a binary search tree, as it is for hash tables. Hash tables can be searched very quickly, on the average, when properly constructed, but they do not allow records to be easily enumerated in sorted order.

9.5: A Brief Review

This chapter and the last have provided examples of the effective use of appropriate data structures with their appropriate implementations. These include the use of a heap for priority queues, the use of a binary tree for a heap, the use of a heap for sorting, and the use of hash tables and binary search trees for searching. Previously we found that trees find numerous applications, ranging from storing data to aiding in the conceptualization of algorithms. Binary trees are not only useful themselves but can serve as the representation for general trees. You have seen that stacks and queues underlie the important tree traversal operations. The virtues and shortcomings of arrays and lists have also been explored.

With respect to problem solving methodology, the top-down approach, with recursion as a special case, has been consistently adhered to. A number of basic strategies have also been illustrated repeatedly. These included searching through the collection of all possible solutions, constructing a solution, and adapting a previously written algorithm to a new but related problem. The rest of this book deals with the same concepts but applies them to more advanced problems.

Exercises

1.     Justify all the entries of Table 9.1 for the priority queue implementations.

2.     Write two functions to insert and delete, respectively, records from a priority queue implemented as a heap.

3.     a.     Build a hash table for the twenty-seven names of the nametable of Chapter 6. Do not store duplicates. The table will be of size 31. Assign to each letter of this alphabet the integer corresponding to its position in the alphabet. For example A is assigned 1, E is assigned 5. Add the integers corresponding to every letter of a name and divide the sum by 31. The remainder is then the hash address of the name. For example, Jimmy Carter has the sum (10 + 9 + 13 + 13 + 25 + 3 + 1 + 18 + 20 + 5 + 18) = 135. Its hash address is 11. Use linear probing when a collision occurs.

b.     How many probes were needed to insert each name?

4.     Criticize the choice of hash function in Exercise 3.

5.     Search the hash table of Exercise 3 for John Smith, and write down the probe addresses needed.

6.     Show the state of hash tables (a) through (g) of Section 9.3 when key 019 is deleled.

7.     Suppose that the eighteen keys are entered into the hash table of Section 9.3 in order, from largest to smallest. As you search for a key, you conclude that it is not in the table whenever you encounter a key value smaller than the search key as you trace out the linear probe path. Convince yourself that this is true, and that it will result in fewer probes for an unsuccessful search.

8.     Write a function to build a hash table for each of the open-addressing and chaining techniques of Section 9.3.

9.     Run each function of Exercise 8 with the input data and hash function of Exercise 3.

10.     What is the maximum number of probes for an unsuccessful search of each of the hash tables (a)-(g) of Figures 9.4, 9.5, and 9.6?

11.     Can you find a way to build a hash table as in Exercise 7 so that the sorted search can be done even if the input was not actually in sorted order?

12.     How might the average number of probes be studied for a given hash function as in Exercise 3?

13.     For m = 107, generate n random key values between 000 and 999 and build (by executing a program) hash tables (a) to (g) of Figures 9.4, 9.5, and 9.6 for these keys. Do this twenty-five times for each n and output the average number of probes to insert the n keys over the twenty-five samples. Do this repeatedly for a series of n values corresponding to usage ratios of approximately 0.50, 0.55, 0.60, . . ., 0.90. 0.95. Compare results of this simulation to your expectations.

14.     Take the nametable names of Chapter 6 and grow a simple binary search tree (as in Section 9.4) when the names are input in the order in which they appear in that figure. The order is determined alphabetically by last name, first name, and then middle initial. Thus Robert Evans precedes Harry Truman. If a duplicate name occurs, do not enter it again in the tree.

15.     Delete Paulina Koch from the binary search tree of Exercise 14, using the simple deletion algorithm of Section 9.4.

16.     Discuss the advantages and disadvantages of storing the nametable entries of Chapter 6 in a sorted array, sorted chain, binary search tree, AVL tree, and hash table.

17.     Write a function, search, with parameters keyvalue, p, and t, a pointer to the root node of a binary search tree. Search is to return with p pointing to the node of t in which keyvalue equals p  [image] key. If no such node exists, p should return with its value null.

18.     Write a function, insertbst, with parameters keyvalue and t. It is to insert a node with keyvalue into the simple binary search tree t if no node exists with that key.

19.     Write a function, de1etebst, with parameters keyvalue and t. It is to delete a node whose key equals keyvalue in the simple binary search tree t, if such a node exists.

20.     Should a linked or sequential implementation be used for binary search trees grown "simply" when they will contain fifty integers picked at random from the first 1,000 integers? Why?

21.     What is the minimum depth of a binary search tree storing 137 entries, and how may it be constructed in time O(n) for n entries?

22.     Should a linked or sequential implementation be used for a binary search tree grown as an AVL tree when it will contain 50 integers picked at random from the first 1,000 integers? Why? (Cf. Exercise 20.)

23.     Follow the same directions as in Exercise 14, but grow a binary search tree that is an AVL tree. It should be the same AVL tree that would result if it were grown by invoking insertavl of Section 9.4.6 for each input name.

24.     Create a binary search tree, with the names of Exercise 23 in it, which has minimum possible depth.

25.     Delete the rightmost node of the AVL tree grown in Exercise 23, using the recursive deletion algorithm of Section 9.4.6.

26.     Modify the nonrecursive function insertavl so that indexing can be done as described in Section 9.4.6.

27.     Modify the recursive function insertavl so that indexing can be done as described in Section 9.4.6.

28.     Why does growing and then inorder traversing an AVL tree correspond to an O(n lg n) sort?

29.     Write a function search, with parameters indexvalue, p, and t. It is to return with p pointing to the node, if it exists, in the AVL tree t, whose index field is equal to indexvalue. P should be null if there is no such node.

30.     How might we estimate the relation between the average depth of a simple binary search tree grown randomly and n, the number of inputs?

31.     Suppose AVL trees are grown randomly. That is, each of the n ! arrangements of n distinct inputs is equally likely to occur. Starting with the null tree, insertavl is invoked for each input. How might the average depth of such AVL trees be estimated as a function of n?

Suggested Assignments

1.     This is the same as the suggested assignment 2 of Chapter 2, except the integers for part (a) and the records for parts (b) and (c) are to be stored in a binary search tree.

2.     Design the nametable for the case study of Chapter 6. Discuss the relative merits of arrays, lists, binary search trees, AVL trees, and hash tables for its implementation.

3.     Write an efficient check (t,d,flag) to return in d the depth of the binary tree pointed to by t, and to return in flag true if t points to a balanced binary tree and false otherwise.

Go to Chapter 10      Return to Table of Contents


Data Structures Using C: CHAPTER 10: FILES

 Return to Table of Contents

CHAPTER 10: FILES

Introduces the file data structure

treats communication with the outside world

input, output, and secondary storage

Discusses sorting of files using

straight merge

natural merge

replacement selection

polyphase sort

Considers file organization to support sequential and random access

Explains the physical basis for the file organizations using

direct access

indexed sequential access

B-trees

Illustrates how the basic operations on B-trees are done and what their time requirements are

creating

inserting

deleting

searching

traversing

10.1: The File Data Structure

A file is a collection of records. This is a loose definition of a file. Almost all of the data structures discussed in the preceding chapters satisfy it. The term file, however, is usually reserved for large collections of information stored on devices outside the computer's internal memory. It usually implies that the records are stored in secondary storage in the computer's external memory, on tapes or disks. As a result, the ways in which the file must be organized so that operations on it can be carried out efficiently are dependent on the characteristics of the secondary storage devices used to implement the file. The basic operations on a file are to insert and delete records, process or update records, and search for or retrieve records. These operations are the same as the basic operations on arrays, lists, trees, list-structures, and more complex lists, which are stored in the computer's internal memory. In designing algorithms that use files stored in external memory, programmers must weigh the trade-offs between the time and the storage required to support these operations. They must do this just as they would for data structures stored in internal memory. Time spent in organizing information results in faster operations, but this efficiency must be weighed against the increased care required to nurture and maintain the better-organized data structure. For example, balanced trees are faster to use in the worst case but harder to grow. Similar effects pertain to file manipulations.

10.2: Internal and External Memory

Computer memories are normally thought of as falling into two basic categories, internal and external. Which memory is used to store data depends on the trade-off of efficiency of access to data versus the cost of that efficiency.

The data structures and algorithms considered thus far are all appropriate to the internal memory of a computer. This memory is also referred to as main memory or random access memory. A program can process only information stored in its internal memory. If the program is to process information stored in external memory, that information must first be read into its internal memory. Except for the instructions involved in this transfer of information, all program instructions refer to variables that name internal memory locations.

By means of random access, it is possible to access any individual internal memory element in exactly the same length of time--typically one-millionth of a second or less. This access time is independent of the past history of accessed elements. However, random access memory is expensive and, although available in increasingly larger sizes, is still relatively small in size compared to the storage needs of many applications.

External memory, also referred to as secondary storage, is so called because it is physically outside the computer itself. It is relatively inexpensive and is available in very large amounts, but access to an arbitrary record is slow compared to internal memory. Magnetic tapes, floppy disks, and magnetic disks (also known as hard disks) are the usual devices for external storage of data. The characteristics of these devices that are essential for efficient use of data structures are discussed in this chapter. Since the characteristics of secondary storage devices are different from those of internal memory, the data structures and algorithms required to process the data they store are distinct from those for internal memory. However, many of the same concepts and techniques of good design and implementation of algorithms are still applicable and form the basis for representing and operating on information stored in external memory.

10.2.1 Operating Systems

Programming languages such as C do not ordinarily allow direct manipulation of external storage elements. Instead, as with internal storage, they provide commands that allow programmers to deal with external memory conceptually. C provides very limited kinds of operations to be performed on its conceptual external memory, whereas COBOL provides very extensive operations. FORTRAN falls between these extremes. Whatever high-level language is used, the operating system of a computer installation provides commands to make files available for processing by a program. Operating systems allow permanent storage of files generated by a program during its execution. The stored files can then be retrieved for later use.

You may have used the DOS, MVS, VM, or NOS operating system on a mainframe computer; RT-11, RSTS, RSX, TSX, VAX/VMS, or UNIX on a mini-computer; or MS-DOS, PC-DOS, UNIX, or CP/M on a micro - or personal computer. In effect, operating systems provide repositories of files that may be added to, or referenced by, programs. Such files are used to store data or information to be processed. They are also used to store algorithms (such as programs to be executed), data base management systems to organize and maintain large data bases, compilers to be invoked, or even other operating systems to be used when needed. Thus operating systems support the development of libraries of data and algorithms that programmers use, add to, or delete from. They support the general programming strategy recommended and demonstrated in this book; in solving a problem, the programmer begins by attempting to apply an already known algorithm, either directly or after appropriate modification.

10.2.2 Filters

Frequently, a solution to a problem is readily found by starting with one or more files of data and repeatedly applying available programs to these files. These programs, in turn, produce output data in the form of files, which then become the input files for other programs. The basic idea behind this kind of solution is a filter--a program whose input is one file and whose output is another file. A solution to a problem may often be found by starting with an input file and applying a sequence of filters. Each filter takes as its input the output file of the preceding filter. System commands allow this composition of filters, so that complex programs can be generated with few commands.

Example 10.1

Suppose an input file contained the roll scores for the bowling problem of Chapter 1. The problem is to structure the solution so that processing this file produces an output file of statistics.

Suppose gamescores is a filter that takes the input file and produces an output file of game scores. Once the game scores are available, a statistical package in the computer's operating system library might be used to process the game scores as an input file (as data), producing appropriate statistics. Applying this program, or filter, to the game scores file would result in an output file containing the required information. This file could be stored for future use or processed by a printing routine, producing printed output directly.

The two filters, gamescores and statistics, applied sequentially, produce the desired output file. Note that this solution is written so that the calculation of the game scores is done first, independently of the calculations producing the statistics. In the solution of Chapter 1 these tasks were intertwined.

The idea behind filters is to allow the construction of complex programs by sequencing simpler but powerful programs. The next example illustrates this more forcefully.

Example 10.2

Use filters to produce a simplified index for a book. The index should consist of the important words of the book in alphabetical order and the pages on which they appear.

One way to achieve a solution is to apply the following filters in sequence.

1. Word-count references

2. Truncate

3. Sort

The first takes a file consisting of the book text (organized as a sequence of pages) and produces an output file of records. Each record contains a word or sequence of words, the number of pages on which it appears, and the pages on which it appears. Truncate takes a file of records as its input and outputs a file that is identical to its input file, except that all records with a designated key field value greater than n are deleted. The key field contains the number of pages on which the word or sequence appears. A reasonable value for n might be 10. This should eliminate frequently occurring but not relevant words such as "the," "a," "and," and so on. Sort takes an input file of records and creates an output file in which the records appear in sorted order by designated key field value--in this case, the word field. The sequencing of these three relatively simple filters provides a quick and effective solution.

Using these system capabilities to store and access files of data or programs provides a framework in which to approach the entire task of problem solving. In effect, a computer installation provides a programming environment that determines the tools available for program construction. Such tools include text editors, data base management systems, file management utilities, ways to create a new program from existing programs already stored as files, and debugging aids. The enhancement of such programming environments is currently one of the major research and development areas of computer science. The command languages, also known as job control languages, provided by operating systems are used to operate on general files.

10.3: Organization of Files

During the execution of high-level language programs, operations typically deal only with files treated as data. Operations include traversing and processing all the records of a file, randomly accessing and processing individual records selected in random order, or some combination of these two modes. The basic function of a file system, in addition to providing storage facilities, is to allow files to be searched efficiently and conveniently, so that records of an entire file may be sequentially retrieved or a portion of the file's records (perhaps only one record) may be randomly retrieved. In Example 10.1, after the bowling game scores file has been obtained, one may want to retrieve the scores of a particular game or individual.

This chapter briefly discusses the file processing facilities provided by C. It shows a number of ways in which more flexible and powerful file processing may be implemented. The literature on files would itself require a large file for its storage, and file storage and manipulation is currently an active area of research.This book considers a very small portion of that activity. The discussion deals only with the case of fixed-length records that are to be retrieved on the basis of a single field value, the primary key field. A primary key uniquely identifies a record. It is the field normally used as the search key, and the key by which records are sorted.

Just as with data structures that are stored in internal memory, the programmer must take great care to keep related records of files stored in external memory physically contiguous or linked by pointers. In internal memory, the time requirement of an algorithm is measured by the number of operations performed. These might be arithmetic operations or assignments, comparisons between two records (keys), or comparisons between two variables. When dealing with external memory, the time requirements are measured by the number of accesses to secondary storage.

10.3.1 Sequential Files

C has no built-in ability to perform input/output. Input/output is done by using the standard C library <stdio> or by developing a user library. The printf and scanf functions used in the programs of the book are in this standard library.The <stdio> library contains a number of other functions that also perform input/output.

C input/output is character-oriented. All reads or writes to or from keyboards, screens, tapes, or disks are character-oriented. C reads or writes in bytes.The functions that read or write numbers or strings in turn use calls to the basic character-oriented input/output functions.

There are two ways to read or write files in C. They are termed high-level and low-level input/output. In high-level input/output, reading and writing is done a character at a time. The input/output is buffered, and the programmer does not have to consider buffer sizes or other characteristics of the operating system. The buffers, however, are not available to the programmer. In low-level input/output, which is essentially UNIX-type input/output to provide a UNIX interface, the buffers are not provided; the programmer must provide buffers and pointers to control the input/output process. In the following discussion only high-level input/output will be considered.

The simplest organization for a file is sequential. A sequential file is a sequence of records. The records may or may not be kept in sorted order in the sequence. In standard C input/output all files are sequential files. A record of a file is not necessarily declared to be of type structure. Although file records are typically of type structure, a file record may also be declared to be of type integer, float, character, or any other C type. All records of a file need not be the same type.

Example 10.3

To illustrate how records composed of structures are treated, suppose you are asked to keep a master file, bookinventory, consisting of records representing all books currently in stock in a book store. Each record is to contain a title, author, publisher, price, and currentstock field. These declarations may be used:

typedef struct

{

char title[MAXTITLE];

char author[MAXAUTHOR];

char publisher[MAXPUBLISHER];

float price;

int currentstock;

}bookrecords;

bookrecords bookinventory;

The name of a file is a variable within a C program. Unlike other variables, it represents stored information that may have been available before the execution of the program or that may be available after the execution of the program. A file named f is depicted in Figure 10.1.

Figure 10.1 File F and F

Associated with each file is a file pointer, which always points to a record or is positioned prior to the first record or after the last record. File pointers must be declared and are of type FILE. For example,

FILE *f_ptr;

declares f_ptr as a file pointer.

A record may be read from a file or written to a file. A read or write statement refers to the record indicated by the current position of the file pointer. Each file also has a portion of main memory, called the buffer, associated with it. When the file pointer is pointing to a file record, the file buffer contains a copy of that record's contents.

To attach a file to a program, it is necessary to execute an open statement. For reading the records in the file containing the bookinventory records, the following code is used.

>Comment

f_ptr = fopen("bookinventory", "r");

if(f_ptr == NULL)

printf("Can not open the file");

F_ptr is the pointer to the file bookinventory. The r signifies that the file will be opened for reading only and that the file is already in existence.

Executing the first statement causes the file pointer f_ptr to be positioned to the first record. If any error occurs and the file can not be opened, the file pointer will returns a NULL value. For this reason, it is a wise policy always to test for the NULL pointer when opening a file, as was done above.

If the open is executed with a w (instead of an r) it opens the file for writing. If no file by the name used exists, it creates a new file and places the file pointer at the beginning of the file. If a file already exists, it places the file pointer at the beginning of the file, in effect erasing or truncating the file. If a file is opened with an a, it creates a new file or, if one already exists, positions the pointer at the end of the file for appending items. A similar set of values, r+, w+, and a+ is used to open files for both reading and writing.

To read a single record or numbers of records from a file the fread( ) function is used. The general form is

fread(  pointer to the location of the ,  size of ,  number of,  pointer to  )

        memory in which the records       record to  records to  the file to

        read are to be placed             be read    be read     be read

For example, to read a single record from the opened file:

fread(&bookinventory,sizeof(bookrecords),1,f_ptr);

Executing this statement will cause the value of the record pointed to by the file pointer to be copied into memory starting at the memory location pointed to by &bookinventory. Also, fread returns the number of records read, or zero when the end of file is reached.

To write a record to a file that is open for writing:

fwrite(&bookinventory,sizeof(bookrecords),1,f_ptr);

Executing this statement will cause the value of the record starting at the location pointed to by &bookinventory to be copied into the storage pointed to by the file pointer, and the file pointer moved to point to the storage to be used for the next record.

Reading from a file thus causes information in secondary storage (the file) to be transferred to variables whose values are stored in internal memory. Writing to a file causes the information stored in internal memory to be transferred to secondary storage.

Example 10.4

To illustrate how files composed of single variables such as integers are treated and the effect of the file operations, consider the following program. The file prime will be created initially to consist of the first 10 primes. Note that the reading and writing are done with fscanf and fprintf. They are like scanf and printf except that they work on files of individual variables. In contrast, fread and fwrite work on files of records. Also, fscanf returns EOF when the end of the file is reached or an error occurs.

#define NULL 0

#include <stdio>

main()

/* Creates a file consisting of the first 10 prime

numbers, and a file containing these primes in

reversed order, and prints the contents of both

the files.

*/

{

/* This section creates file prime and

stores the first ten primes in it.

*/

int a[20],i,dummy;

FILE *f_ptr,*fr_ptr,*fopen();

    >Comment

f_ptr = fopen("prime","w");

    >Comment

fprintf(f_ptr,"%d",2);

fprintf(f_ptr,"%d",3);

fprintf(f_ptr,"%d",5);

fprintf(f_ptr,"%d",7);

fprintf(f_ptr,"%d",11);

fprintf(f_ptr,\"%d\",13);

fprintf(f_ptr,"%d",17);

fprintf(f_ptr,"%d",19);

fprintf(f_ptr,"%d",23);

fprintf(f_ptr,"%d",29);

    >Comment

fclose(f_ptr);

/* This section reads from the prime file,

fills a with its contents, and creates

reversedprime which will contain

the contents of a in reversed order.

*/

i = 0;

    >Comment

(1) f_ptr = fopen("prime","r");

    >Comment

if(f_ptr == NULL)

{

printf("Can not open");

exit(-1);

}

    >Comment

while(fscanf(f_ptr,"%d",&dummy) != EOF)

{

    >Comment

a[i] = dummy;

i++;

}

    >Comment

i--;

    >Comment

(2) fr_ptr = fopen("reversedprime","w");

    >Comment

if(fr_ptr == NULL)

{

printf("Can not open");

exit(-1);

}

    >Comment

while(i >= 0)

{

    >Comment

fprintf(fr_ptr,"%d",a[i]);

i--;

}

    >Comment

fclose(f_ptr);

    >Comment

fclose(fr_ptr);

/* This section prints, at the terminal,

the contents of prime and then of

reversedprime.

*/

    >Comment

f_ptr = fopen("prime","r");

    >Comment

while(fscanf(f_ptr,"%d",&dummy) != EOF)

printf("\n %d\n",dummy);

    >Comment

fr_ptr = fopen("reversedprime","r");

    >Comment

while(fscanf(fr_ptr,"%d",&dummy) != EOF)

printf("\n %d\n",dummy);

    >Comment

fclose(f_ptr);

    >Comment

fclose(fr_ptr);

}

Prime and reversedprime are files of integers and a is an array of integers.

After (1) fopen is executed, the file called prime and its file pointer can be depicted as in Figure 10.2(a). After nine executions of the first while loop body, the situation will be as shown in Figure 10.2(b).

The tenth execution sets a [9] to 29 and moves the file pointer beyond the last file record. The EOF test yields the value true, so the loop is exited with i set at 10.

After (2) fopen is executed, reversedprime appears as in Figure 10.3(a). After the second while loop body has been executed nine times, reversedprime looks like Figure 10.3(b). The last execution yields the file shown in Figure 10.3(c).

The standard function, exit, closes any open files, flushes any buffered output, signals what has happened, and terminates program execution. A zero parameter value, by convention, signals that all is well, while other values may have different meanings. The -1 is used here to signal that a file could not be opened. Files can also be closed by simply invoking the function fclose, whose parameter is the file pointer of the file to be closed. It is important to close all files appropriately.

Suppose you wished to append the eleventh prime, 31, to prime after 29. This can be done directly in C by opening the file prime to its end so that items can be appended. To do this, as noted earlier, the file need only be opened with the a to position the file pointer

f_ptr = fopen("prime","a");

fprintf(f_ptr,"%d",31);

(a) After FOPEN Is Executed

(b) After Nine Executions of the While Loop

Figure 10.2 File PRIME

(a) After FOPEN Is Executed

(b) After Nine Executions of the Second While Loop

(c) After Last Execution

Figure 10.3 File REVERSEDPRIME

Example 10.5

Suppose there are two files with records of the same type in sorted order based on key value. Merge the two files.

Producing a new file consisting of all the records of the two files in sorted order is called merging the two files. The merging of two files is a basic component of efficient methods of external sorting. The task in Example 10.5 is to write a function to produce a merged file assuming records are ordered with the smallest key value appearing in the first record.

The basic idea is straightforward. Read a record from each sorted file, and append the record with the smaller key value to the merged file. Read the next record of the file that had the record with the smaller key value. Compare it with the record that has not yet been appended. Append the record with the smaller key value to the merged file. Repeat this process of reading and comparing records from the two files until one of the files becomes empty. Then append the records on the remaining file to the merged file.

Assume that the files to be merged are not empty and have been opened with filepointers f1_ptr,f2_ptr, and f3_ptr. Filepointers f1_ptr and f2_ptr point to the input files to be merged. Filepointer f3_ptr points to the resultant merged file. The files are made up of records of typedeffilerecords, with key the field of filerecords on which merging is to be based. The merge may be written as follows:

merge(f1_ptr,f2_ptr,f3_prt)

/* Merges the files pointed to by f1_ptr

and f2_ptr. The merged file is pointed

to by f3_prt. The files must be opened before the merge is invoked.

*/

FILE *f1_ptr,*f2_ptr,*f3_ptr;

{

struct filerecords record1,record2;

int flag,length;

flag = 0;

length = sizeof(filerecords);

    >Comment

fread(&record1,length,1,f1_ptr);

fread(&record2,length,1,f2_ptr);

    >Comment

while(flag == 0)

    >Comment

if(record1.key < record2.key

{

    >Comment

fwrite(&record1,length,1,f3_ptr);

    >Comment

if(fread(&record1,length,1,f1_ptr) == 0)

{

flag = 1;

    >Comment

fwrite(&record2,1ength,1,f3_ptr);

}

}

    >Comment

else

{

    >Comment

fwrite(&record2,length,1,f3_ptr);

    >Comment

if(fread(&record2,length,1,f2_ptr) == 0)

{

flag = 2;

    >Comment

fwrite (&record1,length,1,f3_ptr);

}

}

    >Comment

if(flag == 1)

    >Comment

while(fread(&record2,length,1,f2_ptr) != 0)

fwrite(&record2,length,1,f3_ptr);

    >Comment

else

    >Comment

while(fread(&record1,1ength,1,f1_ptr) != 0)

fwrite(&record1,1ength,1,f3_ptr);

}

The files must be opened before the merge is invoked.

The following program, when the merge is included, merges two files and prints the resultant file.

#define NULL 0

#include <stdio>

typedef struct

{

int key;

}filerecords;

main()

/* Merges two files and prints the merged file.

*/

{

filerecords record;

FILE *f1_ptr,*f2_ptr,*f3_ptr;

    >Comment

f1_ptr = fopen("prime","r");

if(f1_ptr == NULL)

{

printf("Can not open file1");

exit(-1);

}

f2_ptr = fopen("other","r");

if(f2_ptr == NULL)

{

printf("Can not open file2");

exit(-1);

}

f3_ptr = fopen("result","w");

if(f3_ptr == NULL)

{

printf("Can not open file3");

exit(-1);

}

    >Comment

merge(f1_ptr,f2_ptr,f3_ptr);

    >Comment

fclose(f1_ptr);

fclose(f2_ptr);

fclose(f3_ptr);

    >Comment

f3_ptr = fopen("result","r");

    >Comment

while(fread(&record,sizeof(filerecords),1,f3_ptr) != 0)

printf("\n %d\n",record.key);

    >Comment

fclose(f3_ptr);

}

10.4: Sequential Access in External Memory

A range of devices is now available for use as secondary storage, the most common being magnetic tapes and disks. Tape cassettes like those used with stereo equipment are prototypical of the magnetic tape used for secondary storage. If a song is being played on such a tape, the time required to access another song on the tape depends on how far away from the current position that song is. This accessibility is the same for magnetic tapes used as secondary storage. Such a tape can be pictured as containing a sequence of fixed-length blocks (Figure 10.4). Each block contains a fixed number of records of a file (say, n per block). One or more input buffer areas (reserved storage areas) are set aside in internal memory, each large enough to hold one block of records. These buffers are required in order to allow for efficient reading and writing of files.

Figure 10.4 Sequential Access on Magnetic Tape

Suppose one buffer is set aside. To read a block of records into the buffer, the tape must be moved at sufficient speed past a read head. This speed determines the rate at which data are transferred to the buffer. If a sequential file is stored on magnetic tape, executing the open statement causes the buffer storage to be allocated, the tape to be rewound so that the first block may be read, and the first block to be read into the buffer. After these operations have occurred, the buffer and the file pointer can be visualized as in Figure 10.5.

If a read is executed, the first record is copied from the buffer into its memory location, and the file pointer is advanced to the next buffer position. Consecutive executions of a read for this file cause similar actions. Finally, when n such reads have occurred, the next block of records from the tape is placed into the buffer, the file pointer is automatically repositioned at the first record of the buffer, and the process is repeated. Had two buffers been set aside, as the current buffer is being read the other would be filled with the next block of file records. A similar procedure takes place for the writes with respect to an output buffer for the file. In this way the buffer is being filled, when reading from a file (or emptied, when writing to a file), at the same time that the computer is carrying out other commands of a program. Record blocking is done because n records can be read or written as a block much faster than individually, since it reduces tape starts and stops. Today's computers, whether mainframe or personal, can execute on the order of 100,000 commands in the time it takes for a block to be read into or written from the buffers. A program may actually process n records of the file while the next block is being transferred to the buffer. For example, the records may be sorted or individually processed during this transfer time. In this case, the time required to process the entire file is determined by the time required to scan the file in this fashion. Thus the number of accesses to secondary storage determines the execution time of the program. This may take minutes, which is a very long time in computer terms.

Figure 10.5 Input Buffer for the File

It should now be clear why sequential files should not be used when records must be accessed in an order other than the order in which they appear on the file. Attempting to access the records in random order may mean spending great amounts of computer time, since minutes may be required for each access. Instead, the typical use for sequential files is when records are to be processed in turn, starting from the first record. In other words, when the processing to be done can be accomplished by a traversal through the records of the file, then sequential files are appropriate. This is typical of external sorting, copying, updating, and printing (such as end-of-the-month billings).

As a practical example, suppose you are given the master file, bookinventory, of Example 10.3, and another transaction file, booktransactions. Booktransactions consists of records representing the arrival of new books for inventory and the sales of books in the last week. If both files are sorted on the same key (say, title), it is possible to traverse each of the files once and update each master file record to reflect new arrivals and sales. A new file must be created for this updated master file. This step would actually be advantageous for security purposes, since the old master file and booktransactions file could be saved. If anything were to happen to the new master file, the procedure could be repeated and the file reconstructed.

10.5: Sorting Tape Files

Once again, one picture--or in this case, a good example--is worth a thousand words. The different merging techniques demonstrated for the following example illustrate some of the complexities of sorting files stored on magnetic tape, when all records of the file cannot fit into internal memory at the same time. Tape sorts are used here, since they highlight the concepts of sorting, but these concepts are also applicable to disk sorts. Detailed analysis of the sorting algorithms discussed here, and others, may be found in Knuth [1973b]. We begin with an extreme case.

Example 10.6

Sort an original file of twenty-one records that are stored sequentially on a tape file. The records have integer key values. Assume that only two records can be kept in internal memory at any one time. Storage for the two records is in addition to input and output buffers.

Suppose the sequence of records in the original file is as follows:

2 12 17 16 14 30 17 2 50 65 20 32 48 58 16 20 15 10 30 45 16

It is not possible simply to read all the records into internal memory and then apply one of the internal sort algorithms of Chapter 8, since available internal storage is assumed to be only enough to hold two records. Instead, the general strategy, given that two sorted files can always be merged as shown in Section 10.3, is to create sorted subfiles repeatedly and merge them repeatedly until a final merged file containing all the original records is produced. The literature contains many ingenious algorithms for external sorting. This chapter gives only the flavor of the solutions. One major constraint on a solution is the number of tapes available for use in the sort. Assume three available tapes.

10.5.1 Straight Merge

Perhaps the most straightforward sorting technique is the straight merge. The straight merge distributes the initial records onto two tapes, a and b, so that they contain the same number of records, or so that one tape contains only one more record than the other. If this is done for the original file in Example 10.5, then the configuration of tapes a and b will appear as the initial distribution.

a   2|17|14|17|50|20|48|16|15|30|16

b  12|16|30| 2|65|32|58|20|10|45|

Now a and b can be thought of as being composed of subfiles, each of which is sorted and of length 1. A subfile is a run of length r if it consists of r records sorted in order. Here, a contains eleven subfiles, or runs, of length 1, and b contains ten runs of length 1.

Apply a merge procedure to each pair of runs, each pair containing a run from tape a and the corresponding run from tape b. Thus

 2 17 14

12 16 30, etc.

are "paired." The merge procedure consecutively merges each pair (consisting of a sorted subfile of length 1 from a, and a corresponding sorted subfile of length 1 from b). The resultant merged files are written consecutively to tape c. The result of the first merge of a and b to c is

c 2 12|16 17|14 30|2 17|50 65|20 32|48 58|16 20|10 15|30 45|16

Think of c as containing ten subfiles or runs of length 2 and one run of length 1 and distribute these to a and b equally to obtain the second distribution.

a  2 12|14 30|50 65|48 58|10 15|16

b 16 17| 2 17|20 32|16 20|30 45|

By now there are five paired runs of length 2 and one run of length 1. These may again be consecutively merged, and written to c to obtain the second merge of a and b to c.

c 2 12 16 17|2 14 17 30|20 32 50 65|16 20 48 58|10 15 30 45|16

Another distribution to a and b yields the third distribution.

a 2 12 16 17|20 32 50 65|10 15 30 45

b 2 14 17 30|16 20 48 58|16

These pairs of length 4 can now be merged to obtain the third merge of a and b to c.

c 2 2 12 14 16 17 17 30|16 20 20 32 48 50 58 65|10 15 16 30 45

It takes two more distributions and merges to complete the sort. The fourth distribution

a  2  2 12 14 16 17 17 30|10 15 16 30 45

b 16 20 20 32 48 50 58 65|

is followed by the fourth merge.

c 2 2 12 14 16 16 17 17 20 20 30 32 48 50 58 65|10 15 16 30 45

The fifth distribution is next. It is followed by the fifth merge.

a  2  2 12 14 16 16 17 17 20 20 30 32 48 50 58 65

b 10 15 16 30 45

c  2  2 10 12 14 15 16 16 16 17 17 20 20 30 32 45 48 50 58 65

This completes the sort, which has taken five distributions and five merges. The sort is described as a five-pass sort, with five distribution phases and five merge phases, using three tapes. Programming this process is not a trivial task. Ignoring details, the basic time requirements can be seen to be determined by the number of times files must be rewound or rewritten and the number of records that must be accessed from secondary storage (tapes). This assumes that the internal memory processing will be done so quickly that it does not add to the total time for the sort. Before the original file, c, can be distributed (rewritten) to tapes a and b. those tapes must be rewound and the original file must be rewound. Before a and b can be merged (and written) to c, they must be rewound, and c must be rewound. Also, in each phase, whether distribution or merge, each record must be accessed from secondary storage. Thus the total time is proportional to the number of passes (five here). If the access time of a record is Ta, and the rewind time Tr, then the total time is 5 X (2 X 21 x Ta + 3 X Tr). Note the factor of 2 associated with the twenty-one records and the access time. The factor is 2 because there are two phases, distribution and merge.

10.5.2 Natural Merge

The straight merge sort does not take advantage of any natural ordering that may exist in the original file. Even if that file were given in sorted order, the straight merge would proceed as before. A natural merge sort distributes the records of the original file to a and b, so that sorted subfiles or runs are preserved and treated as a unit. Such a distribution would assign 2 12 17 to a, since they form a run. The 16 terminates this run and goes to b. The 14 terminates this run and goes to a as the beginning of its second run, followed by 30. So far the result is

a  2 12 17|14 30

b 16

The 17 now terminates the 14 30 run and goes to b. Note that this extends the first run of b to 16 17. At this point, 2 terminates this run with the following result.

a  2 12 17|14 30

b 16 17

If balance is to be maintained, each file must end up with a number of runs differing by no more than 1 from the other files. Therefore the next run (starting with 2) must go to b. Subtleties of this kind must be carefully monitored in the actual implementation of merge sorting algorithms. The resultant distribution will be the initial distribution.

a  2 12 17|14 30   |20 32 48 58|15 16

b 16 17   | 2 50 65|16 20      |10 30 45

Merging these pairs of runs and writing the merged results to c yields the first merge.

c 2 12 16 17 17|2 14 30 50 65|16 20 20 32 48 58|10 15 16 30 45

Another distribution and merge gives the second distribution,

a 2 12 16 17 17|16 20 20 32 48 48

b 2 14 30 50 65|10 15 16 30 45

which is followed by the second merge.

c 2 2 12 14 16 17 17 30 50 65|10 15 16 16 20 20 30 32 45 48 58

A third and final distribution and merge complete the sort. The third distribution yields

a  2  2 12 14 16 17 17 30 50 65

b 10 15 16 16 20 20 30 32 45 48 58

The result of the third merge is as follows:

c 2 2 10 12 14 15 16 16 16 17 17 20 20 30 30 32 45 48 50 58 65

This is a three-pass, two-phase sort, using three tapes. The total time is 3 X (2 X 21 X Ta + 3 x Tr). Any inherent ordering of the initial data allows the number of passes to be reduced, as compared to the straight merge sort.

Note that to merge runs embedded in a file, the procedure of Section 10.3.1 requires modification.

10.5.3 Replacement-Selection

Since there is typically room for a reasonable number of records in internal memory, both the straight and natural merge sorts can be speeded up. For instance, if m records can be sorted in internal memory, the straight merge sort can start with the distribution of runs of length m on each of the tapes a and b.

There is a clever way to produce runs for distribution (for example, with the natural merge sort) without actually storing the m records. This is the replacement-selection sort, which uses a heap of size m and works as follows. Start with the first m keys of the original file of Example 10.6 in internal memory and create a heap to obtain Figure 10.6(a) (m = 7).

Remove the root key from the heap and output it to an output file. The next input file key, 2, is inserted at the root of the heap, and the heap is reheaped, giving Figure 10.6(b).

Figure 10.6 Replacement-Selection Sort

Again, the root key is removed and written to the output file. The next record key of the input file, 50, is inserted at the root and reheaping occurs. Figure10.6(c) results.

Continuing in this way generates a run on the output file of length 8, which would be terminated when 16 from the input file is inserted at the root of the heap. The situation at that point is shown in Figure 10.6(d). Instead of terminating the run at length 8, treat 16 as associated with the next run, since 16 is less than the last output value, 20. This means that 16 must be distinguished from the heap entries for the current run; it is shaded in Figure 10.6(d). In reheaping, current run entries are considered smaller than next run entries. Figure 10.6(e) shows the heap after reheaping. Each input smaller than the last output is handled in this way. Eventually the heap becomes filled with entries of the next run, and the process repeats.

Notice that the length of the output run will always be at least m (with the possible exception of the last run generated). Continuing this process, called replacement-selection, we obtain the output file,

2 2 12 14 16 17 17 20 30 32 48 50 58 65|10 15 16 16 20 30 45

The original file consisted of ten runs; the output file after replacement-selection consists of two runs. With randomly ordered input files, this replacement-selection generates runs whose average length is 2m. The example exhibits this behavior in an exemplary fashion.

Notice that both the straight and natural merges spend half their time accessing records for the distribution phase. Spending this time can be avoided by using another tape. By this technique, each time two runs are merged, they can be written, alternately, to tapes a and b. This eliminates the need for extra, basically unproductive, time spent in distribution from tape c.

10.5.4 Polyphase Sort

Rather than using more tapes, it is possible to incorporate the distribution phase (except for the initial distribution) into the merge phase. Such a sort is called a polyphase sor t. In this type of sort, the tapes being merged, and the tape to which the merged subfiles are written, vary continuously throughout the sort. In this technique, the concept of a pass through records is not as clear-cut as in the straight or the natural merge. A distribution, a merge, and a pass all blend together. This might have more properly been called an amorphous sort, except that it has considerable character and is one of the better sorts.

The polyphase sort starts with an initial distribution of runs on tapes a and b. The initial distribution is critical to its proper execution. For the case of the original file of twenty-one records in Example 10.6, the initial distribution should be thirteen runs to tape a, and 8 runs to tape b, each of length 1. It is the number, not the length, of the runs on each tape that is critical. The number of records for the example is twenty-one because the first few Fibonacci numbers are 0, 1, 1, 2, 3, 5, 8, 13, 21. The Fibonacci numbers are crucial to the polyphase sort because the number of records must be a Fibonacci number to make the polyphase sort work.

When a three-tape polyphase sort is used, the original number of runs should be a Fibonacci number, with the distribution of runs between tapes a and b the two preceding Fibonacci numbers, which add to the original number of runs. In the present case, 8 + 13 = 21. When more tapes are used, generalized Fibonacci numbers, which specify the required number of records, become the basis for the initial run distribution among the tapes. No programs will be shown for the implementation of the details of the external sorting algorithms discussed in this section. Therefore it is unnecessary to resolve in detail the question of how to proceed when the actual number of runs is not a Fibonacci number. The remedy, though, involves adding an appropriate number of "dummy" runs [Knuth, 1973b] .

Getting back to the workings of the polyphase sort, the initial situation after distribution will be an initial distribution.

a   2  12  17  16  14  30  17    2  50  65  20  32  48  13 runs of length 1

b  58  16  20  15  10  30  45   16                   8 runs of length 1

c

This distribution required twenty-one record accesses.

We now proceed by merging the first eight runs of a and the eight runs of b to c to obtain the merge of the first eight runs of a and b to c.

a  50  65  20  32  48                                 5 runs

b

c   2 58  12 16  17 20  15 16  10 14  30 30  17 45  2 16  8 runs

Notice that b is now empty, a total of 8 + 8 = 16 records have been accessed, and a and c contain 5 + 8 = 13 runs. All three tapes had to be rewound before the initial distribution, but only b need be rewound now. It is possible to continue merging the first five runs of c and the five runs of a to b. This results in a merge of the first five runs of a and c to b.

a

b  2 50 58  12 16 65  17 20 20  15 16 32  10 14 48  5 runs

c    30 30  17 45      2 16                        3 runs

Notice that a is empty, a total of 5 + 5 X 2 = 15 records have been accessed, and b and c contain 5 + 3 = 8 runs. Tape a is now rewound, and the first three runs of b are merged with the three runs of c. The result is the merge of the first three runs of b and c to a.

a  2 30 50 58  12 16 17 45 65  2 16 17 20 20  3 runs

b 15 16 32  10 14 48                         2 runs

c

Tape c is now empty and must be rewound. A total of (3 X 3) + (3 X 2) = 15 records were accessed, and a total of 2 + 3 = 5 runs remain. Merging the first two runs of a and b to c yields

a  2 16 17 20 20                                   1 run

b

c  2 15 16 30 30 32 50 58  10 12 14 16 17 45 48 65  2 runs

This leaves b empty, so we rewind it. A total of (2 X 3) + (2 X 5) = 16 records were accessed, and 1 + 2 = 3 runs remain. Merging the run of a and the first run of c to b yields.

a

b  2 2 15 16 16 17 20 20 30 30 32 50 58  1 run

c 10 12 14 16 17 45 48 65                1 run

A total of (1 X 5) + (1 X 8) = 13 record accesses occurred, and 1 + 1 = 2 runs are left. Tape a is rewound. Then a final merge produces

a  2 2 10 12 14 15 16 16 16 17 17 20 20 30 30 32 45 48 50 58 65

b

c

This final merge of b and c to a took 13 + 8 = 21 record accesses. A total of 117 ( = 21 + 16 + 15 + 15 + 16 + 13 + 21) record accesses is required. These 117 record accesses represent an effective 117/21, or 5 4/7, passes. The total time for this algorithm is 117 x Ta + 8 x Tr. The straight merge sort and the natural merge sort required 210 X Ta + 15 X Tr and 126 X Ta + 9 X Tr time, respectively.

The polyphase sort, until completion, results in exactly one empty tape, so that the remaining tapes may be partially merged to it. After each merge, the distribution of runs is made up of two consecutive Fibonacci numbers (13 + 8, 8 + 5, 5 + 3, 3 + 2, 2 + 1, 1 + 1, 1 + 0) . With more tapes , this is also true , but generalized Fibonacci numbers appear.

10.6: Direct Access in External Memory

Accessing a song on a phonograph record is different from accessing a song on a tape cassette. Think of the songs as stored along the grooves of a record rather than along the tape of a cassette. Instead of going through each song between the current arm position and the location of the desired song (and scratching the record!), the arm is moved directly to the proper groove. Once the arm is placed, it is necessary to wait until the record revolves, so the beginning of the song appears under the arm. The movable arm eliminates the need to pass through all songs that intervene between its current and desired positions. The magnetic disk works the same way. It is the reading head that moves to the desired storage location, thus saving access time compared to the magnetic tape.

Magnetic disks and disk packs are examples of direct access devices used for secondary storage. In this context, direct access means that a program can go straight to desired information stored on a given surface and cylinder. This access time is not constant. It is slower than random access in internal memory, but generally much faster than the sequential access time for tapes. A floppy disk has similar access capability but is slower and has much less storage capacity. Thus in terms of access time, direct access storage devices fall between sequential access devices, such as magnetic tapes, and random access memory. Hard disks are generally used with mainframes and minicomputers, whereas microcomputers or personal computers vary from no disks through floppy disks to hard disks.

A random access memory allows very fast access to any element of the memory. The access time is the same for all stored elements, no matter which element was last accessed. Sequential devices allow fast access "locally," but all elements between the currently accessed element and the next desired element must be passed through before the next element may be accessed. Consequently, the time to access the next desired element is proportional to its distance from the currently accessed element. Direct access devices are intermediate between these two extremes. They act like sequential devices with tabs. Just as using the alphabetical tabs in a dictionary allows a person seeking a word to find it quickly, so do "tabs" allow one to close in, relatively quickly, on the desired area of the sequential storage. In direct access devices, the "tabs" are the movable read/write heads.

Sequential devices are convenient for tasks that may be accomplished by traversing records. However, when the order in which the records must be processed cannot be predicted, sequential access devices are not efficient. Tasks requiring random access to records cannot be accomplished efficiently using sequential memory devices. The introduction of economical direct access devices made random access feasible. As a result, airline reservation systems, interactive computing, and instant updating of bank accounts have become commonplace. This was made possible by the development of appropriate technology for direct access memory.

A magnetic disk is a flat, circular platter on whose surfaces information can be stored and accessed. It can be pictured as a phonograph tone arm and record, with its concentric tracks corresponding to the grooves of the phonograph record and its movable read/write head corresponding to the tone arm. A disk pack consists of a pile of such magnetic disks and a read/write head for each record surface--that is, one head for each side of a record (see Figure 10.7). The read/ write heads are typically coupled so that they all move at the same time. They are positioned over the same track on their respective surfaces, although only one head is actually reading or writing at any moment. There may be twenty magnetic disks and a hundred tracks on each disk.

Figure 10.7 A Disk Pack

Each track of a surface in a disk pack stores records in blocks, similar to the blocking of records on magnetic tapes. Each track behaves as a sequential access device, like a magnetic tape. That is, each block must be passed through, as the disks rotate, before another block farther along the track can be accessed by the surface read/write head.

If we focus on the same track on each surface, then the pile of tracks forms a conceptual cylinder in space, as shown in Figure 10.8. Assuming tracks are numbered (say, from 1 to 100), a cylinder consists of all tracks with the same number. Thus the disk pack can be viewed as composed of one hundred concentric cylinders.

A cylinder is important because it allows any block of records stored on a surface of the cylinder to be accessed without moving the read/write heads. Accessing a block stored on a different cylinder (or track) requires moving the read/write heads. The time required to move the heads from one cylinder to another is called seek time; it is proportional to the distance between the cylinders. This time typically takes a significant fraction of a second (1/10th to 1/20th). It is the dominant factor in dealing with time requirements for accessing blocks on magnetic disks.

Once the read/write heads are positioned over the desired cylinder, the time required for the desired block to rotate under the heads is called latency time. It can be on the order of one hundredth of a second.

Figure 10.8 Conceptual Cylinder

10.7: Sequential and Random Access of Disk Files

Often large amounts of information, too much to fit into internal memory, must be organized so that random requests for specific pieces of information can be readily satisfied. The methods of organization that are useful in internal memory are not adequate for data stored on disks in external memory.

10.7.1 Sequential Access

There is a natural way to store records sequentially on magnetic disk to minimize total seek time for a traversal through the records. This storage technique is based on the cylinder concept. It starts by filling blocks of the top surface track of the outermost cylinder with records. The records are in sequence within the blocks. When the track is full, the same process is repeated with the track of the next surface, and the next, and the next, until the lowest surface track is filled. At that time the outermost cylinder has been filled. Continuing in the same way, records are stored on each of the inner cylinders, starting with the next adjacent cylinder. This is similar to storing records sequentially on magnetic tape. In fact, we can picture the surfaces and cylinders laid out linearly as in Figure 10.9. This arrangement has all the inherent drawbacks of magnetic tape, whenever random accessing of records is required.

We can adapt disk storage to random accessing by taking advantage of the tab feature of the disk, which allows the head to go directly to any cylinder and surface. The problem is how to determine exactly where to go. If one knew that the record to be retrieved were the ith record, then it would be easy to calculate on which cylinder and surface it would be located. This is not often the case, since the search is conducted by means of key value, not by order in the sequence.

Figure 10.9 Sequential Access on a Magnetic Disk Pack

A linear search of the file could be done but might take O(n) time, when n records are stored in the file. This could take minutes. If the records are stored in sorted order by key value, a binary search of the file might be done. The difficulty here is that 1g n accesses are required. A million-record file would require twenty accesses, and each access might require a seek. While the binary search is better, it still might take seconds. This is reasonable for isolated requests, but during this time there may be many requests for desired retrievals. Waiting requests may build up interminally, with unlimited waiting time before all are processed. The need to insert or delete records further compounds the problem.

Just as in the case of internal memory, keeping records in blocks with pointers to succeeding blocks is a possibility, based on the concept of lists. This may avoid insertion and deletion difficulties, but it does not alleviate the problem of searching. We encountered this same problem with lists in internal memory. What about binary search trees? Even if balanced, a million records may need as much as 20   [image]  c seeks, where c is a small constant.

10.7.2 Random Access

We still have hash tables in our arsenal. Using hash tables in secondary storage requires some accommodation to its access capabilities. Buckets, an extension of the chaining policy for collision resolution with internal memory, are appropriate. The idea is to create a hash table based on key values. The hash table will contain pointers to lists. Each list, referred to as a bucket, contains all records that hash to the same hash table address. The lists are stored in blocks on the disk, each block holding a fixed number of records and a pointer to the next block containing other records on the list. When a key is inserted into the hash table, the program determines its hash address and then inserts the key into the bucket pointed to by that address. The key may be inserted into the bucket by traversing the list of linked blocks until a block is encountered with room for the record. If no such block is found, a new block is appended to the list, and the record is inserted in that block. Care must be taken to avoid wasting seek time (and latency time) when these lists are being traversed. This means that the blocks of a bucket should be on the same cylinder. Searching for a key is similar to making an insertion. Deletion involves the usual deletion from a list, with care taken to release unneeded blocks when possible. When small enough, the hash table itself is kept in internal memory to avoid one disk access.

In practice, such hash tables can be devised so that one or two disk accesses are required for either successful or unsuccessful searches. This is accomplished by selecting a good hash function, an appropriate fixed number of records per block, and an appropriate number of buckets. The usage factor of such a hash table is given by n/bk, where n is the number of records stored, b is the number of buckets, and k the fixed number of records per bucket. If the performance of the table degrades as insertions are made, the table may be reorganized.

Other interesting and important external hashing methods are presented in Larson [1976], Litwin [1960], and Fagin, Nievergelt, Pippenger, and Strong [1979].

10.7.3 Indexed Sequential Access

While the hash table solution may yield fast average access times, there is no guarantee for the worst case, nor any convenient way to access records in sorted order by key value. To achieve the ability to access records in random order or in sorted order, as necessary, we must incorporate an old idea familiar to all. Suppose a visitor from outer space wanted to find the location of a particular street, say Bond Street in London. The visitor might go to a world atlas and determine first where London's country, England, is situated. The intergalactic traveler might then find a map of the country, and locate London. Finally, a city map of London gives the creature the exact location of the street. On a more mundane level, suppose a student wants to locate a particular book in the university library. The student would go to the file cabinet of authors and find the drawer that contains authors' last names with the appropriate beginning initial. She would then search for the card with the correct author and title, which gives a library call number. Next she would consult a library directory to learn the floor location of those call numbers. Finally, she would go to that location and search for the book with that particular call number. The critical concept in such searches is the availability of one or more directories, to be used to determine quickly the location of the particular item desired.

The magnetic disk supports the use of such a directory to focus the search. It can be used to find a particular cylinder and surface relatively quickly, eliminating the need for traversing sequentially through intervening records. Until recently, when B-trees (see next section) were introduced, the most popular method of creating and maintaining such a directory used the natural sequential storage of records, so that they appear in sorted order by key value. This is known as the indexed sequential access method (ISAM); it may be implemented in various ways, typically using three or four levels of directories. A first directory, known as the master index, may be searched to determine the cylinder on which a particular search key appears. If a disk pack has two hundred cylinders (tracks), then the master index has two hundred entries, one for each cylinder. The entry for the ith track gives the highest key value appearing on that track. A search of the master index then readily determines the cylinder on which a search key may reside. That cylinder is accessed; it contains a cylinder index directory. If a disk pack has ten surfaces, the cylinder index has ten entries, one for each surface. The entry for the ith surface gives the highest key value appearing on that surface. A search of the cylinder index determines the surface on which a search key may reside. Finally, the track of the cylinder on that surface may be searched sequentially for the desired key, resulting in a successful or unsuccessful search. With such an implementation, unused storage is initially left in blocks so that the insertion of new records may be more easily accommodated. If a block is full, records can be shifted to the right to other blocks, so that the new record is inserted in proper order.

Overflow areas must also be available and maintained on disk, in case the track storage is exhausted due to insertions. Tracks and cylinders are set aside for this purpose. Insertions may require the master or cylinder index to be updated. Similarly, deletions of records, leaving vacated storage to be reclaimed, may require updating of directories and shifting of records. Deletions may also be treated simply by marking a record as deleted, but this may eventually lead to the need for reclamation of this storage, degrading the efficiency of the implementation. Because of overflow from insertions, performance may degrade, necessitating reorganization, allocation of more disk storage, and reconstitution of the file.

The overflow areas may be maintained as lists of records in sorted order, linked by pointers. When the proper cylinder index is searched, the pointer to be followed may point to an overflow area, in which case the search amounts to a traversal of a list of records. Otherwise, the record is said to reside in the prime area, and the search amounts to a sequential traversal of records on a track. The prime area consists of reserved tracks on cylinders designated as prime cylinders.

For a given number of records to be stored, there is no guarantee that overflow will not degrade the system, unless a high price is paid by dedicating large amounts of track storage to reduce this possibility.

Each cylinder index actually has additional entries giving the highest key value in its overflow area for each surface and a pointer to the first record of the overflow list for the surface. Conceptually, we may visualize the organization of such implementations as shown in Figure 10.10.

10.7.4 B-trees for Sequential and Random Access

A more recent method allows for achieving both the random access of records and a traversal through the records in sorted order. This method allows a more flexible use of the disk and guarantees excellent worst-case performance for a given number of records, with an acceptable amount of dedicated disk storage. It is known as the virtual sequential access method (VSAM). To see how it might be implemented, we study its basis B-trees. VSAM and many data base management systems use modified B-trees in their implementation. The basic reference for B-trees is the classic paper by Bayer and McCreight [1972]. Hash trees are an alternative to B-trees for the organization of large files of data. They are discussed and compared to B-trees by Bell and Deen [1984]. Tremblay and Sorenson [1984] discuss many file structure organizations. See Ullman [1982] for more on data base systems.

Figure 10.10 shows that the directories (the master index and the cylinder indexes) form a tree structure that guides the search for a record. Programmers try to find specialized trees leading to efficient searches, efficient use of storage, and relatively easy maintenance. Since the depth of the tree determines the worst-case search times, the goal is to find trees with short paths to terminal nodes. This implies that the nodes of the tree generally will have many successors. B-trees satisfy these requirements.

Figure 10.10 Organization of Indexed Sequential Files

A B-tree of order m has the following four properties:

1.     Every node of the tree, except the root and terminal nodes, has at least   [image] 1/2 m  [image]  subtrees.

2.     Every terminal node has the same depth.

3.     The root node has at least two subtrees, unless it is a terminal node.

4.     No node has more than m subtrees.

Each node of a B-tree will have the form shown in Figure 10.11. This represents storage for m - 1 keys and m pointers, although the storage may not all be used at a given time. If n is less than m, not all m keys are present in the node. Each pointer Pi points to a subtree of the node. The keys K1, K2, . . . , Kn are kept in sorted order, so K1 < K2 <   [image]    [image]    [image]  < Kn within the node. The subtree pointed to by Pi+1 contains all records of the file whose key values are less than Ki+1 and greater than or equal to Ki. P1 and Pn+1 point to subtrees containing records with key values less than K1 and greater than or equal to Kn, respectively. For example, if the keys are

1,3,5,6,7,8,13,20,21,23,24,30,37,40,56,60,62,63,70,80,83

then the tree of Figure 10.12(a) is a B-tree of order 4 storing these keys and their records.

Figure 10.11 Form of Each Node on a B-Tree

The terminal nodes of the B-tree in Figure 10.12 are not shown. Instead, the pointers to them have been replaced by null pointers. This reflects the way the tree might be represented in external storage. Notice that, in general, null pointers will appear at the same depth in the tree--depth one less than the terminal nodes. A generalized inorder traversal (see Chapter 7) of the B-tree accesses the keys in sorted order. Associate with each null pointer the key value accessed immediately after the pointer is considered in such a traversal. A unique key is then associated with every null pointer except the last, and each key has a unique null pointer associated with it. For example, the first three null pointers have keys 1, 3, and 5 associated with them. The fourth null pointer (to the right of 5) has key 6 associated with it. Consequently, the number of null pointers, and the number of terminal nodes, is exactly one greater than the number of keys stored in the tree.

Any B-tree has a root node with at least two successors. Each successor has at least   [image] 1/2 m  [image]  successors. Thus there are at least 2   [image] 1/2 m  [image]  d-2 nodes at depth d > 3 of any B-tree. A B-tree of depth d > 3 has at least 2   [image] 1/2 m  [image]  d-2 terminal nodes, so

n + 1 > 2   1/2 m   d-2

This implies that d < 2 + lgx  [image] (n + 1)/2  [image], where lgx denotes logarithm with base x and x =   [image] 1/2 m  [image]. If n = 224 - 1 = 16,777,215 and m = 2 X 28 = 512, then (n + 1)/2 = 223, and lgx [(n + 1)/2] = lg256(223) < 3.

A B-tree storing 16,777,215 records will therefore have depth   [image] 4. We shall see that d - 1 is the maximum number of nodes that must be accessed when searching a B-tree for a search key. Thus we are guaranteed that no more than three node accesses are required to search for a key in the B-tree with 16,777,215 records!

A B-tree may be viewed as a generalization of a binary search tree, with a special imposed balance requirement to ensure against too great a depth. To search a B-tree for a search key value (say, 22) entails accessing its root node and traversing the node's keys until 22 is encountered, a key that exceeds 22 is reached, or the last key stored in the node is passed. If 22 is found, the search is complete. In Figure 10.12(a) the key value 40 is reached. The pointer to its left points to a subtree that must contain the search key, if it is in the tree at all. That subtree is then searched in the same way. Its root node is accessed, and 30 is reached. Following its left pointer, the final subtree root is accessed. Its key value, 23, is reached. Its left pointer is null, which means that the search terminates unsuccessfully.

Property 1 ensures that, except for the root and terminal nodes (which don't appear anyway), every node of a B-tree will be at least half full. Property 2 ensures that the B-tree will not be scraggly. All paths to terminal nodes must have the same length. The idea is to store information of a node on blocks of a cylinder on the disk, so that accessing a node requires one seek time and all keys of the node may be read into internal memory. The search through the keys of a node, to determine the next node to access, then takes place rapidly in internal memory, using an appropriate search technique.

(a) Initial B-tree

(b) After Insertion of 35

Figure 10.12 A B-Tree of Order 4 with Twenty-one Records

(c) After Splitting, Due to Insertion of 28

(d) After 7 is Deleted and 5 is Borrowed

Figure 10.12

(e) After Deletion of 83 and Merging

Figure 10.12

After an insertion or deletion, of course, the tree must be maintained as a B-tree. This may be done making local modifications that are simpler than those required for AVL trees. For insertion, the key to be inserted is searched until a null pointer is to be followed. This assumes that only one record with a given key value appears in the tree and that it is similar to the initial insertion in AVL trees. If the node containing the null pointer has unused storage, then the key is simply inserted at the proper place in this node. This may require the shifting of keys in the node, but it is accomplished quickly in internal memory. For example, if key 35 is inserted in the tree of Figure 10.12(a), the result is the tree of Figure 10.12(b).

All insertions will be made in this way, in nodes at the lowest level, until the node becomes full. For example, if 28 is to be inserted next in the tree of Figure 10.12(b), no room exists in the node x 21 x 23 x 24 x . The m + 1 key values (the m old keys and the 1 new key) can be split by leaving   [image] m/2  [image]  - 1 in the old node and moving the remaining records, except the middle one, the   [image] 1/2m  [image] th, to a newly created node. The middle record is inserted in the predecessor (parent) of the old node. The result is the B-tree of Figure 10.12(c).

Had the predecessor been full, it would have been split in the same way in turn. Splitting may be necessary at every node of the search path, back to and including the root. If this occurs, the depth of the resultant B-tree is one greater than the original B-tree.

Deletion is done similarly. A key is simply deleted from a lowest-level node, and the keys shifted over, if the resultant number of keys satisfies the constraint that the node contain at least   [image] m/2  [image]  - 1 keys. If the constraint is violated, a key can be borrowed from a "neighbor" node, provided this may be done without violating the constraint for that node. In this case, the predecessor or successor key of the borrowed key moves into the deletion node, and the borrowed key takes its place in the predecessor node. For example, if key 7 is deleted from the tree of Figure 10.12(b), key 5 is borrowed from its neighbor (13 cannot be borrowed). The result is Figure 10.12(d).

If no neighbor can spare a key, the total number of keys between a neighbor and the deletion node will be less than m. These nodes may then be merged, along with a key from their predecessor. In the tree of Figure 10.12(d), if 83 is to be deleted, this situation will occur. The result is Figure 10.12(e).

It is possible for the predecessor to become too sparse, although this did not happen in our example. It is then treated as though its key were deleted, and so on, until the entire process is complete. This may go all the way to the root, in which case the root disappears and the depth of the tree decreases by 1.

If the key to be deleted were not in a lowest-level node, it would be replaced by its successor and the successor node treated as if the successor were being deleted.

These insertion and deletion procedures preserve the B-tree properties. The B-tree is an external memory data structure that plays the same role as AVL trees in internal memory. In practice, a modification of the basic B-tree is used to ensure better storage utilization and to increase the efficiency of traversal through the records in sorted order for VSAM. With B-trees the generalized inorder traversal must be used for sequential access. This is time-consuming and requires stack storage. Instead of storing some key records in higher-level nodes, the idea is to store all records only in lowest-level nodes and chain the lowest-level nodes by pointers. For example, in Figure 10.12(a) all nodes at depth 3 could be linked in a chain, with the leftmost node first and the rightmost node last on the list. A traversal then amounts to a traversal of this chain.

It is clear that storage management is needed for external memory, just as for internal memory. This problem is not discussed here, but the internal memory techniques are applicable, when properly modified to take into account the structure of the disks.

10.8: Summary

Files are stored in secondary storage and provide powerful facilities for the temporary and permanent storage of large amounts of data. Magnetic tapes and magnetic disks are two important devices on which to store files. These are typical of sequential and direct access devices, respectively.

Because of the structure and resultant access capabilities of these devices, internal memory techniques must be modified in order to deal efficiently and conveniently with files stored on them. It is possible to sort efficiently, even with sequential files, by using external sorting techniques. Primarily, these are merge sorts, such as straight, natural, and polyphase sorts. Other merge sorts may also decrease sorting time. In general, sequential files are suitable for tasks that can be accomplished by traversing the records of the file. These tasks include sorting and updating master files.

It is possible to achieve efficient random accessing of records with direct access devices when access is by a specific key. Hash tables are useful for this purpose. To achieve a combined capability of random plus sequential access in sorted order by a specific key requires considerable file organization. This organization involves the creation of a file directory, which may consist of a number of levels. The directory guides the search efficiently to the desired record with specific key value. A number of directory implementations are possible; directories based on B-trees are becoming increasingly popular. Such directories may be searched relatively quickly, allowing efficient insertions and deletions while utilizing storage well.

At this point, you should have filed away the substance of this book. It is hoped that you have done so to allow quick retrieval, while maintaining room for insertions. Deletions may be necessary but should be minimal.

Exercises

1.     Suppose a function reads in each record of Example 10.4 and creates a file consisting of every other prime number on the file. Show the state of the input and output files and their file pointers after the fourth record of the output file has been written.

2.     Write a function to create the output file of Exercise 1.

3.     Write a function to create reversedprime of Example 10.4, assuming the array a may only be of length 10 while reversedprime may contain many more entries.

4.     How many writes and reads will be executed by the function merge if file 1 and file 2 are, respectively,

1 2 8 10 13 15 40

3 7 10 12 17 20 45 50 60

5.     Write a function to create a sequential file that is the same as file 1, except that all records on file 1 that are also on file 2 do not appear.

a.     Assume file 1 and file 2 records appear in sorted order based on the key field.

b.     Assume file 1 and file 2 are not sorted, and you do not know how to sort a file.

6.     Write a function to produce a new file that is the same as an input file except that all blanks have been deleted from it.

7.     Write a function that takes two sequential files and appends the second file to the first.

8.     Why are the sort procedures of Chapter 8 inappropriate for files?

9.     Why does the number of passes required by a straight sort not depend on any initial ordering among records?

10.     Write a function for the straight merge.

11.     Modify the merge function of Section 10.3 so that it merges two runs embedded in files.

12.     Write a function to distribute records as required by the natural merge.

13.     Write a function for the natural merge.

14.     Simulate the behavior of replacement-selection to verify that the average length run produced is 2m.

15.     Why must Fibbonacci numbers govern the number of initial runs for three tapes in the polyphase sort?

16.     a.     If four tapes are used in a polyphase sort, what is the relation determining the initial distribution of runs?

b.     Can you generalize to n tapes?

17.     Write a function for the polyphase sort using three tapes.

18.     Why do the movable read/write heads and surfaces act as tabs for a disk?

19.     Suppose the master index of Section 10.7.3 does not fit on one cylinder. How may another directory level be created to serve as an index to the master index?

20.     Why do B-trees allow more flexible use of a disk than directories implemented using prime and overflow areas?

21.     Create a B-tree of order 4 to store the nametable records of Chapter 6. The records are to be ordered alphabetically so that the B-tree can be searched with the name field as key.

22.     Write a function to insert a record into a B-tree of order m that is stored in internal memory.

23.     Can you write the function of Exercise 22 so that it is independent of whether the B-tree is stored in internal or external memory? If so, write it. If not, explain why not.

24.     Write a function to delete a record from a B-tree of order m that is stored in internal memory.

25.     Why does the chaining of lowest-level nodes discussed in Section 10.7.4 allow more efficient traversal through a file?

26.     Write a function to create a B-tree stored in internal memory.

27.     Suppose we want to store variable length records in a B-tree. How may this be done?

28.     Why is it not feasible to store all records in the root node of a B-tree so that only one seek is required?

29.     A B-tree of order 3 is called a 2-3 tree, since each node has two or three successors.Although not useful for external storage, such a tree may rival AVL trees for an internal memory structure in which to store records. Compare its search, insertion, and deletion algorithms and its time requirements with those for AVL trees.

30.     Write a search function for a B-tree of order m.

31.     Write a function to traverse through the records of a B-tree in order by key value.

32.     Suppose the individual records of Figure 6.5 are to be stored on a disk with the recordptr field of the nametable pointing to the address of an individual record on the disk. The address is a cylinder and surface specification. What is a reasonable way to allocate disk storage to the individual records?

33.     Suppose records of a file were stored in a hash table, with buckets for random access by key, and linked in sequential order by pointers for sequential access. Compare the insertion and retrieval times of such an implementation with the use of B-trees.

Suggested Assignment

This assignment is appropriate for a group of students to do together. It involves a task of reasonable size, which must be partitioned so that each member of the group knows just what to do and so that, when all parts are put together, the final solution works correctly. The idea is to produce an "index" for a book. This involves

1.     The creation of files, using the sequential files available in C.

2.     The creation of appropriate functions that will allow the records of a file to be accessed randomly, or in sequential order, by key.

One file, the book file, is to consist of records that represent the pages of a book. Each page contains one hundred lines, and each line consists of one hundred characters. The key consists of a page number and a line number. A second file, the dictionary file, is to consist of records representing words to appear in an index for the book represented on the book file. When the solution is run, it should use a scaled-down version of the book file, which will consist of a few pages of ten lines and twenty characters each, with the dictionary containing relatively few words (say, twenty-five).

The dictionary file is to be a sequential file, with the file implemented using B-trees. Instead of a disk, assume that enough internal memory exists to store the book file. The index for the book should be output so that the words of the index appear in alphabetical order, followed by the page and line numbers on which the word appears in the book. Also, an input file consisting of page and line numbers appearing in random order should result in an output consisting of the text for each record of this input file.

Go to Chapter 11      Return to Table of Contents


Data Structures Using C: CHAPTER 11: TOPOLOGICAL SORTING: AN ARCHETYP

 Return to Table of Contents

CHAPTER 11: TOPOLOGICAL SORTING: AN ARCHETYPAL SOLUTION

Introduces topological sorting

as an example of the application of the problem solving methodology and programming style presented in this text

background for the problem is given first

each step and decision is explained as the solution evolves

Emphasis is on

top-down design

data abstraction

functional modularization

intelligent choice of information to store and data structure in which to store it

11.1: Background

Topological sorting requires ranking a set of objects subject to constraints on the resultant topology--that is, on the placement of the objects. It occurs in many practical situations. For example, textbooks are often written so that each chapter builds on material covered earlier and cannot be understood without this base of information. Determining the order in which to present topics to ensure that all prerequisite material has been covered reduces to finding a topological sort of the topics. Other examples involving topological sorting are presented later.

The approach to the topological sorting problem presented in this chapter and the characteristics of the final program are intended to serve as prototypes. The problem is substantial and the algorithm developed important, but it is the process by which it is achieved that is most important. Developing an efficient topological sorting program is a complex process and demonstrates the general problem solving methodology used in this text. It requires the application of structured programming concepts, during which the selection of data structures plays a significant role.

The resultant topological sorting program is a model solution, representing nearly the best programmers can hope to achieve. Few solutions to problems have all its characteristics. It is concise and clear; correctness is easy to determine; and the storage and execution time requirements can be analyzed. Invalid input data are not difficult to deal with. What more can we ask?

The following are some examples of problems whose solutions require topological sorting.

Example 11.1

A large skyscraper is to be constructed. Imagine that you must write a program to prescribe the order in which all the tasks are to be performed. The tasks include T3--install windows; T50--install floors; T8--install window sashes; T1000--clear the ground; T7--build the foundation; T9--install doors. For any two tasks, it is possible to specify whether one should be done before the other. Clearly, T8 must be done before T3, and T1000 must be done before T7, while it makes no difference whether or not T3 is done before T9. The constraint that Ti be done before Tj is represented as the pair (i, j).

Example 11.2

Imagine that you want to write a book on data structures. All technical words used must be defined. First write down all the relevant words. If word wj uses wi in its definition, represent this constraint by the pair (i, j).

Example 11.3

As a student you must take a number of courses to graduate. Represent the fact that course cj has course ci as a prerequisite by the pair (i, j).

Example 11.4

A number of books must be read for a course you are taking. Represent the fact that you prefer to read book i before book j by the pair (i, j).

Suppose that an order is given in which the construction tasks of Example 11.1 are to be carried out. If you proceed to carry out the tasks, you may not be able to do the next task, since tasks that must precede it have not yet been finished. The same situation can arise in Examples 11.2, 11.3, and 11.4, when an order is specified in which words are to be defined, courses to be taken, and books to be read, respectively. This is because words used in the next word's definition have not yet been defined, all prerequisites for the next course may not have been taken, or you will not read the next available book because you want to honor your preferences. A topological sort gives an order in which to proceed so that such difficulties will never be encountered.

11.1.1 Binary Relations and Partial Orders

Some mathematical concepts and terminology must be defined before the topological sorting problem can be stated and solved in abstract terms. As with the solution to any problem, stating this solution abstractly is desirable so that it will not be tied to a particular application.

You readily understand what is meant by the statements: "Fred is related to Sam"; "John is taller than Sue"; "3 is the square root of 9." The phrases each specify a connection or relation between two objects. In the first two cases the relations refer to two people. In the third case the relation refers to numbers. Each is a binary relation on a set of objects.

Mathematics abstracts from such examples to define a binary relation R on a set S. S is taken as the set or collection of objects referred to by the relation. R contains information, for any two of these objects, on whether or not they are related. R is a set of pairs of objects (a, b), where a and b belong to S. If the pair (a, b) belongs to R, then this is interpreted to mean that a is related to b. This is written as aRb. If (a, b) does not belong to R, then a is not related to b. This is written as [image].

For instance, specify S to be the set of integers between 2 and 10, so S = {2, 3, . . . , 10}; and take R = {(4, 2), (6, 2), (8, 2), (8, 4), (9, 3), (10, 2), (6, 3), (10, 5)}.Then R captures the "is divided evenly by but is not equal to" relation on S, since a pair (a, b) will belong to R only if a "is divided evenly by but is not equal to" b. By looking at R, we can determine if aRb or [image] for any a and b in S.

Of particular interest are the kinds of binary relations called partial orders. A binary relation R on S is called a partial order on S if it has the following properties:

Irreflexivity For any a in S,

Asymmetry For any two distinct a and b in S, aRb implies

Transitivity For any three distinct a, b, and c in S, aRb and bRc implies aRc

These properties specify that whenever certain pairs are in R, then other pairs must necessarily be included in R or excluded from R. The "is divided evenly by but is not equal to" relation on S in the preceding paragraph fulfills all three properties of a partial order.

It is irreflexive, since a is divided evenly by but is equal to a for all a in S.

It is asymmetric, since if a "is divided evenly by" b, then b cannot be divided evenly by a when a is not equal to b.

It is transitive, since whenever a is divided evenly by b and b is divided evenly by c, then a must also be divided evenly by c.

To show that a relation does not have a property requires only one counter-example.

Two extreme binary relations on a set S are the null relation, which contains no pairs at all, and the universal relation, which contains all possible pairs. The null relation is a partial order. This is because no counterexamples can be found. The universal relation will never be irreflexive (unless S has no objects in it) or asymmetric (unless S has only one object), but it will always be transitive.

11.1.2 Graphic Representation of Partial Orders

One can represent any binary relation R on a set of n objects graphically, as in Figure 11.1. Let S consist of ten objects, the integers 1, 2, 3, . . . , 10. Take R to contain the pairs (1, 2), (8, 3), (7, 6), (2, 7), (5, 4), (7, 8), (2, 4), (4, 7), (2, 8), (8, 5). Create a node for each object in S and draw arrows between two nodes for each pair in R. For example, an arrow will appear from 7 to 8 for the pair (7, 8).

Any such diagram can be interpreted as representing a binary relation on a set S. For such a diagram to represent a partial order requires that

No arrows appear from any node to itself (irreflexivity)

No arrows appear from any node i to another node j and back from j to i (asymmetry)

If arrows appear from i to j and from j to k, then an arrow must appear from i to k (transitivity)

In depicting partial orders, it is unnecessary to draw all the arrows needed to represent transitivity, because they are always assumed. With this understanding, it is not difficult to see that a diagram represents a partial order if and only if it contains no loops. This means that no node can be found such that one can follow a series of arrows starting from that node and ending at that node. In Figure 11.1, there is a loop involving nodes 5, 4, 7, and 8. Hence the diagram does not represent a partial order. Note that if one starts with a diagram representing a partial order (no loops), erasing a node and all arrows emanating from it results in a diagram that represents a new partial order on the set of remaining nodes.

Figure 11.1 Binary Relation R on a Set of n = 10 Objects

11.1.3 Topological Sorts: Consistent Rankings

Partial orders are important. They are meant to capture abstractly any concrete example in which the pairs of R (or arrows of the diagram) mean "is greater than," "precedes," "is ranked higher than," or "comes before." A special case of a partial order is one whose diagram looks like the following:

3    5    1    2    4    6    8    7

For such a diagram, it seems clear what would be meant by a listing or ranking of the eight objects that is consistent with the given partial order. Such a ranking would be 3, 5, 1, 2, 4, 6, 8, 7. No other ranking would be consistent with the given partial order. This can be generalized to any partial order by interpreting the arrows as constraints on the ordering of objects in a ranking. For example, an arrow from 3 to 5 would mean that 3 must be ranked higher than 5. Then, a ranking consistent with a partial order would mean a ranking of the n objects that violates no constraints (that is, a ranking that violates no arrows or pairs).

Example 11.5

Consider the partial order on the set of nine integers represented by Figure 11.2.

The ranking 2, 3, 4, 5, 6, 7, 8, 9, 10 would not be consistent with this partial order, because it violates at least one of the constraints--namely, that 6 be ranked higher than 3, or that 10 be ranked higher than 2. This diagram actually represents the "is divided evenly by but is not equal to" relation introduced earlier.

Figure 11.2 A Partial Order on the Integers 2, . . . , 9

The definition of topological sorting can now be stated more formally than at the outset of the chapter. A topological sort is a ranking of the n objects of S that is consistent with the given partial order. Let us try to solve the following topological sorting problem.

Example 11.6

Given a partial order on a set S of n objects, produce a topological sort of the n objects, if one exists. If no such ranking exists, then print out a message saying that none exists.*

* If a relation is asymmetric and transitive but not irreflexive, then one can eliminate all pairs of the form (x, x) from the relation or, equivalently, erase all self-loops from its diagram. The resulting relation is a partial order, and it still makes sense to find a topological sort with respect to it. The topological sort then satisfies the constraints of the original relation involving distinct objects.

For simplicity assume that S is always the set of the first n integers 1, 2, 3, . . . , n, and that the partial order is given as m pairs of integers representing arrows in the corresponding diagram. Assume no pairs are replicated. For concreteness, take the input to have the following form:

10       value of n

7 3

5 3

2 10

6 8

5 6      m = 9 pairs

4 8

7 1

4 2

7 6

This particular instance of the topological sorting problem will be referred to repeatedly throughout the chapter. Is there an obvious algorithm for this problem? Try to think of one before continuing.

11.2: A Searching Solution

Consider the specific ranking 4, 8, 1, 2, 7, l0, 9, 5, 3, 6 of the 10 objects of the example. It is easy to determine whether or not this ranking violates any constraints of the partial order by checking each one. Since this ranking violates two of the nine constraints, 6 [image] 8 and 7 [image] 1, it is not a topological sort. Any ranking may be checked in this way. This leads to an obvious algorithm based on the generation of all rankings of the n objects.

A search-based algorithm:

If a ranking is found which violates no constraints, then the ranking is a topological sort, print it and stop. If all possible rankings have been tried and no topological sort found, then print out the message that none exists, and stop.

To implement this algorithm in a programming language requires finding a way to generate the rankings so that all the distinct rankings are eventually generated. The procedure permutations of Chapter 4 does this. Any implementation of the algorithm could take time proportional at least to the number of distinct rankings of n objects. A particular ranking can be constructed by selecting any one of the n objects to be first in the ranking, then selecting any one of the remaining n - 1 objects to be second in the ranking, and so on. Thus there are n X (n - 1) X (n - 2) X [image] [image] [image] X 2 X 1 distinct rankings of n objects. This product is n factorial. Consequently this solution can deal with values of n no greater than 16.* For larger n, an algorithm must be found that need not consider all rankings.

* See Section 1.8.2.

11.3: A Constructed Solution

The first solution to Example 1l.6 searched through all possible rankings. Another approach is to attempt the construction of the required ranking. To this end consider a ranking, obj1, obj2, . . . , objn, of the n objects that is a topological sort with respect to a given diagram. What can be said about obj1, the highest-ranked object? Clearly there must be no objects that are constrained by the diagram to appear higher in the ranking than obj1. Otherwise, those constraints would be violated, and the ranking would not be a topological sort. In terms of the diagram, this means that no arrows must point to obj1. We describe this by saying that obj1 must have no predecessors.

Now suppose the node corresponding to obj1 is erased, as well as all the arrows emanating from obj1. As noted earlier, this results in a new diagram that represents a new partial order on the remaining n - 1 objects. Consider the ranking of these n - 1 objects obtained by taking the original topological sort of the n objects and removing obj1. That is, consider obj2, obj3, . . . , objn. It must be a topological sort of the remaining n - 1 objects in the new diagram with respect to the new partial order. This is because any constraint in the new diagram that is violated by this ranking would have been violated in the original ranking of the n objects. Note that if any topological sort of the remaining n - 1 objects were taken with respect to the new partial order, and obj1 were placed in front of it, the ranking of n objects thus obtained would be a topological sort with respect to the original partial order. Placing obj1 first in this ranking cannot violate any of the original constraints because they did not require that any object precede obj1.

We have just discovered how to construct a topological sort:

A construction-based algorithm:

While S is not empty

1. Select any object with no predecessors.

2. Place it in the next position of the output ranking.

3. Remove that object from S, and

4. Remove its arrows from the partial order.

The output ranking produced will always be a topological sort.

11.3.1 Correctness

One difficulty remains. How can the programmer be certain the algorithm can actually be carried out? It may be impossible to carry out because at some point there may be no objects with zero predecessors. That this is indeed an algorithm requires showing that there always is at least one object with no predecessors.

One method of proof for any proposition is to assume it is not true and then show that this assumption leads to a contradiction. This method of proof is applied here. The goal is to prove that every time the loop task is to be carried out, at least one object with no predecessors can be found among those remaining. To assume that this is not true means that at some point the task cannot be carried out because every object left has at least one predecessor. Recall that the original diagram, representing a partial order, had no loops. It will be shown that the assumption implies a loop in the diagram. This will be the contradiction.

Suppose i1 is one of the remaining objects. Under the assumption, every object has at least one predecessor, so i1 has a predecessor (say, i2). Note that i2 cannot be the same object as i1, or the irreflexivity property would not hold. Again, i2 must have a predecessor (say, i3). The object i3 cannot be i1 or i2. If it were i1 or i2, then a loop involving i2 and i1 or a loop with just i2 would exist. Continuing with this argument, i1, i2, i3, i4, . . . must all be distinct objects. But since there are only n objects in total, eventually an object must repeat. This implies that the repeated object is the starting and ending object of a loop. The conclusion is that the assumption was false. Therefore what we wanted to prove must be true, and we never get bogged down--there will always be an object with no predecessors.

This algorithm yields additional insight into the topological sorting problem. Since it is clear that the loop can always be carried out to completion, it will always produce a topological sort of the n objects. In other words, a topological sort always exists. This was certainly not obvious before.

To be sure the algorithm is understood, it is applied next to Example 11.6, depicted again in Figure 11.3. Initially (see Figure 11.3(a)), 4, 7, 5, and 9 have zero predecessors.

The first step is to select and remove 5 (either 4, 7, or 9 could also have been removed) and its successor arrows, to obtain Figure 11.3(b). S is not empty, so the loop must be repeated. Now, 4, 7, and 9 have no predecessors. Select and remove 7 and its successor arrows, and obtain Figure 11.3(c). S is still not empty, and now 4, 9, 1, 3, and 6 have no predecessors. Select 4. Updating produces Figure 11.3(d).

Figure 11.3 Graphic Depiction of Example 11.6

S is still not empty, but 9, 1, 3, 6, and 2 now have zero predecessors. Continuing in this way, removing 1, 3, 6, 8, 9, 2, and finally 10 in succeeding iterations yields the final ranking: 5, 7, 4, 1, 3, 6, 8, 9, 2, 10.

Notice that every time an object is selected and removed, the diagram is updated to reflect the new S and the new partial order. As a result, some additional objects may end up with zero predecessors. Once an object has zero predecessors, it will remain in each succeeding diagram, always with zero predecessors, until it is removed.

The first refinement describes an implementation of the algorithm.

First refinement:

1. Read n.

2. Initialize for phase I.

3. While there is another input pair,

a. record the information in the pair i, j phase I

4. Initialize for phase II. phase II

5. While the set S is not empty,

a. select an object with no predecessors, and output it as the next object in the output ranking;

b. update the records to reflect the removal of the output objects.

Phase I reads into memory the information about the number of objects and the partial order. Every time an i, j pair is read in, the new information that this represents must be incorporated into a representation in memory. The "record the information" of task 3a refers to the modification of this representation to reflect the new arrow, i [image] j.

Phase II describes the processing required by our algorithm. Task 5a involves the selection, removal, and outputting of an object. Task 5b involves modifying the partial order representation and S, to reflect the removal of an object and its arrows, and outputting it in the ranking. After task 5b is carried out, that representation should reflect the new set S and the new partial order resulting from the object's removal.

11.3.2 An Initial Implementation

A straightforward representation of the partial order would be to keep an image in memory of all input arrows. We might use two arrays, first and second, for this purpose. First would contain the first integer, and second would contain the second integer of an input pair. The "record the information" task would then involve putting i and j into the next available locations of first and second, respectively. For Example 11.6, after phase I was executed, the result would be the arrays shown in Figure 11.4.

Suppose the while loop of phase II is being executed, and task 5a is to be carried out. An object with no predecessors must be selected. How is such an object found? Any object that has no predecessors will not appear in the second array. Somehow the second array must be searched to select an object with no predecessors. This can be accomplished by using an array S, traversing second, and marking location i of S when object i appears in second. Any location in S left unmarked corresponds to an object with no predecessors.

Suppose this is done, and object 5 is selected. Object 5 can then be output to the next place in the ranking. To accomplish task 5b, the "updating of the records," then requires that the array first be traversed. As it is traversed, each time a 5 is encountered, the program must mark the location of first in which it appears, and the corresponding location of second. The mark signifies that the corresponding arrow emanating from 5 has been removed from the representation.

Figure 11.4 Result of Phase I of the First Refinement

This completes one execution of the loop. This same processing must be repeated for each repetition of that loop. The loop will be repeated n times, once for each of the n objects. The first and second arrays must be traversed each time. This will take time proportional to their length, which, in general, will be m. Hence this implementation of the algorithm will take time proportional to n X m. The storage required is roughly 2m locations for first and second, and n locations for S.

11.3.3 A Better Implementation

It is apparent that a lot of time is spent finding the information needed to carry out the processing in phase II. Efficiency can be improved by focusing on the following important question. Precisely what information is needed to carry out the processing?

Notice that each time through the loop, the program must select an object with no predecessors. To do this requires kdnowing which of the objects left in S have zero predecessors. Suppose that it is known, before and after each execution of the loop body, exactly how many predecessors each remaining object has. With this knowledge, an object with zero predecessor count can be selected. Once an object becomes eligible to be selected (has zero predecessors), it remains eligible until it is actually selected. Why not collect all those initially eligible in the initialization for phase II? The program could then select among them arbitrarily. Think of this collection as kept in a bag from which the object selected for output is pulled.

As the rest of the loop body is executed, the records must be updated to reflect the removal of this object and its arrows. Removing its arrows amounts to reducing the predecessor count of each successor of the object removed.Whenever the predecessor count of one of these successors becomes zero, that successor becomes eligible for selection on the next repetition of the loop. That successor is immediately added to the bag. Consequently, at the start of each loop repetition, the bag will contain exactly those objects eligible for selection, thus eliminating the need to search for them. In fact, the "S not empty" test is now equivalent to asking if the "bag is not empty." Keeping track of the successors of each object eliminates the need to search for them when that object is output. Incidently, we now know two operations that will be needed on the count collection, decrease(i,count) and iszero(i,count). The first reduces the count of the ith object by one, and the second returns true only if the count of the ith object is zero. The precise information needed for processing has now been isolated. What must be known are the predecessor counts and successors for each object, and what's in the bag.

The loop body of phase I must update predecessor counts and successors.The input pair i, j contains the information that j has one more predecessor than currently reflected in j's predecessor count, and that i has one more successor, j, than currently reflected by i's successors. The loop body of phase II must update the predecessor counts of all successors of the removed object. It must also update the bag. We have developed the following better refinement of the algorithm.

A better refinement:

1. Read n.

2. Initialize

a. Initialize the predecessor counts for each of the n objects to zero.

b. Initialize current successors for each of the n objects to zero.

3. While there is another input pair,

a. increase the predecessor count of j by l;

b. add j to the current successors of i.

4. Place all objects with zero predecessor count into the bag.

5. While the bag is not empty,

a. remove an object from the bag and output it in the output ranking;

b. for each successor of the output object, decrease its predecessor count by l; if the predecessor count becomes zero, add the successor to the bag.

The implementation treats the predecessor counts, the collections of successor counts, the bag, and the output ranking as data abstractions. In order to refine the solution further, we must decide how to store the predecessor counts, the successors of each object, the bag, and the output ranking. These are data structure decisions. They should be based on the kinds of operations to be performed in tasks 1 through 5. First we functionally modularize the solution.

1. read(n);

2. a. predinitialization(n,count);

b. succinitialization(n,successor);

3. while nextpair(i,j)

a. increase(j,count);

b. insert(i,j,successor)

4. baginitialization(bag,n,count);

5. while not emptybag(bag)

a. remove(bag,obj);

b. update(successor,obj,bag,count)

Every operation performed in tasks 1 through 5 that involves the bag, any of the n predecessor counts or collections of successors, or the output ranking has been implemented by a corresponding function. Note the use of a prefix to designate the structure involved in some of these operations. When written in this form, the program is independent of the specific details of implementation of each of these structures. This means that even if changes in the data structure implementations are ultimately adopted, this version of the solution will never need to be modified.

The next step is to decide on specific data structures for implementation of the data abstractions. This choice must be based on the functions that operate on the data structures. These functions include predinitialization,succinitialization, increase, decrease, iszero, insert, baginitialization, emptybag, and remove.

At some future time a decision may be made to change the implementation of any data abstraction. This requires finding all declarations and definitions of the data structures and functions involved and replacing them with the new versions.This modification may be made more easily and more reliably when the originals have been localized in the program. Such localization is called encapsulation. Not all high-level languages provide appropriate tools for achieving the same degree of encapsulation. In the ideal final program the data abstractions are defined in precisely one place; even more important, no operations are allowed on the corresponding data structures except those specified by the functions of the data abstraction. In the implementation above, this means, for example, that for the bag only the baginitialization, emptybag, remove, plus any other operations we take as basic to the bag will appear together. This makes it easier to determine which operations may be affected by a new implementation of the bag.The greater the encapsulation and modularization, the more expeditious the process of achieving correct modifications and adaptations. How to do this in C was discussed in Chapter 5.

We are actually in a position now to make a decision on the implementations for count, successor, and bag. The programmer does not know, in advance, the order in which objects will be output. In task 5b of the better refinement, after an object is output, its successors must be accessed and their predecessor counts updated. This means that the predecessor counts and the collections of successors must be accessed in arbitrary order. Because arrays support random access to their entries, we choose to store the predecessor counts in an array count and pointers to each collection of successors in an array succ. Count[i] and succ [i] will then contain, respectively, the predecessor count and a pointer to the successors of object i. This allows the predecessor count and collection of successors of any object to be selected, taking constant time. Since we shall need to access the collection of successors of objects, we define access_succ(i,succ) as a basic operation that returns a pointer to the collection of successors of object i.

There are n collections of successors, one for each object. These may be stored separately or may share storage. This choice has important ramifications, which will be discussed later. For now, it is best to share storage and to implement each collection of successors as a list of records. The records consist of two integer fields, succobj, containing a successor object, and link, containing a pointer to the next list record. Either dynamic memory or an array may be used to store the records. Selecting the array for storage means the programmer must manage its allocation himself or herself. Dynamic memory is probably a more natural choice, but an array of records is used to illustrate its simple management in this case and to make the later discussion comparing shared or separate storage for the list records more concrete. Lists will be the array for record storage, and a variable t will keep track of the next available record in lists for task 3b. Since only insertions will be made into the lists, the entries of lists will be allocated one after another as needed, starting with the first. Thus t must be initialized correctly.

The apparent choice for the output ranking is an integer array rank, with a variable next specifying where the next output object is to be placed in the rank array. For convenience, we use the record ranking with the two fields rank and next.

The bag could be implemented similarly, in an array with bag pointing to the next available element for a new bag entry. When an object is to be removed from the bag, we can store its value in obj and then decrease bag by 1. Thus an object can be added or removed from the bag in constant time. This implementation is permissible because objects may be selected from the bag in any order. However, implementing the bag as in Figure 11.5 affords an advantage. By allowing the bag and the output ranking to share storage, it saves time. However, this selection is made primarily to emphasize a point to be made later about the independence of modules. In order to simplify the example, only the array entries starting with 1 are used. As you know, in C they start with zero; 0th array positions will not be used in this example so that ranking and array positions will be identical.

Figure 11.5 Data Structure Implementation of Bag and Output-Ranking Data Abstractions

The X's represent nine objects placed in the rank array. Next, pointing to position 6, indicates that the next object to be selected from the bag will be the one in rank[next],rank[6] in this case. Bag, pointing to 10, indicates that when a successor is to be added to the bag, it should be placed in rank[bag],rank[10] in this case. In other words, the bag contains all objects in rank between next and bag-1.

The bag is initialized by traversing the count array and placing any object with zero count into the location of the rank array to which bag points. Bag, of course, must then be updated by 1 to move its pointer down. Bag must initially be 1, and next must initially be 1. Objects are then output in the order in which they are placed into the bag, easing processing and keeping the time constant for addition or removal of an object. Each time an object is selected, next must be increased by 1 to move it down. The bag is empty when next equals bag.

Before proceeding, phase I is illustrated by applying it to Example 11.6. For this example, count, succ, t, and list would appear as in Figure 11.6, after the first eight pairs have been input. Figure 11.7 graphically depicts the successor lists.

At the start of phase I, no information was processed yet on predecessors or successors for any object. The first pair, 7 3, contains the information that 3 has one more predecessor than was known, and that 7 has one more successor than was known, namely 3. One must be added to the count for 3, and 3 to the list of successors of 7, and so on for the other seven pairs.

Figure 11.6 Data Structures Involved in Phase I after Input of First Eight Pairs

The detailed processing of an input pair is illustrated for the last pair of this example. It is the ninth pair, 7 6. To update the count for 6, add 1 to count[6]. To create a record to contain the successor 6, place 6 into lists[9].succobj and add this new record to the current successor list for 7 by changing two pointers. First, copy the pointer from succ[7], the head of the list to which the new record is to be added, into the link field of the new record lists[9].link. Then place a pointer into succ[7] so that it points to the new record. In this case, 9 is placed into succ[7]. Since successors need not be kept in any special order, always adding records at the front of successor lists saves processing time. Adding the record anywhere else on the list would require traversal time from the head of the list to the insertion place. T must be incremented by 1 so that it points to the next available record in lists. The final situation after all nine input pairs have been read and processed is shown in Figures 11.8 and 11.9. An X indicates a null pointer.

Figure 11.7 Graphic Depiction of Successor Lists

Figure 11.8 Data Structures of Figure11.7 after Processing Last Input Pair of Phase I

We are now ready to consider the actual implementations for the functions of the data abstractions. Following is the code that might be used for those functions whose implementation may not be apparent.

Figure 11.9 Graphic Depiction of Successor Lists of Figure 11.8

increase

count[j] = count[j] + 1;

decrease

count[i] = count[i] - 1;

iszero

return(count[i] == 0);

avail

>Comment

static int t=0;

t++;

return(t);

insert

    >Comment

pointer = avail();

    >Comment

setinfo(pointer,j);

    >Comment

setlink(pointer,succ[i]);

succ[i] = pointer;

succinitialization

    >Comment

for (i=1;i<=n;i++)

succ[i] = setnull();

emptybag

    >Comment

emptybag =(bag == ranking.next);

remove

    >Comment

*pobj = ranking.rank[ranking.next];

ranking.next = ranking.next + 1;

Note that avail must correctly initialize and update t.

Task 5b of the better (or "bag") refinement of the algorithm could be done by writing the procedure update from scratch. Instead, it is done here using a tool that is available, the traverse procedure of Chapter 3. First, traverse is modified by adding bag and count as additional parameters of both traverse and process, and changing its name to update.

Process, called by update, must now be implemented so that update does its job. Its code is

    >Comment

currentsucc = info(recordpointer);

decrease(currentsucc,count);

if(iszero(currentsucc,count))

{

ranking.rank[*pbag] = currentsucc;

*pbag = *pbag + 1;

}

A complete program using topsort follows. Notice that topsort has two parameters. It is the same as our version, except it copies the rank field of ranking into the output array as a last step.

Because of the implementation, some functions must have access to specific variables. C requires these variables to be either passed as pointers or declared as global. For example, process must have access to rank, and emptybag to next. Incidentally, as written, topsort does not depend on the medium assumed for the input pairs. Normally, we want programs to be independent of the form of the input. This has been accomplished here for the pairs by using the function nextpair(i,j), which inputs the next pair and returns true if it is a nonsentinel pair. The program uses 0 0 as the sentinel pair.

    >Comment

#defineLIMIT 21

    >Comment

typedef int outputarray[LIMIT];

main()

/* Reads the number of objects n and the input pairs

specifying a partial order on the objects, and prints

a topological sort of the n objects. Each pair should

not occur more than once in the input and n should

not exceed twenty. The sentinel pair is 0 0.

*/

{

int n,i;

    >Comment

outputarray topologicalsort;

    >Comment

topsort(&n,topologicalsort);

    >Comment

printf("\n A TOPOLOGICAL SORT IS");

for (i=1;i>=n;1++)

printf("\n %d",topologicalsort[i]);

}

    >Comment

typedef int countcollection[LIMIT];

predinitialization(n,count)

/* Initializes all n counts to zero. */

int n;

countcollection count;

{

int i;

for (i=1;i<=n;i++)

count[i] = 0;

}

increase(j,count)

/* Increases the jth count by one. */

int j;

countcollection count;

{

count[j] = count[j] + 1;

}

decrease(j,count)

/* Decreases the jth count by one. */

int j;

countcollection count;

{

count[j] = count[j] - 1;

}

iszero(i,count)

/* Returns true only if the ith count is zero. */

int i;

countcollection count;

{

return(count[i] == 0);

}

    >Comment

#define RECORDLIMIT 191

/* RECORDLIMIT should be at least

(((LIMIT-1)*(LIMIT-2))/2)+1

*/

#define NULL - 1

typedef int listpointer;

typedef struct

{

int succobj;

listpointer link;

}listrecords,recordsarray[RECORDLIMIT];

    >Comment

recordsarray lists;

listpointer setnull()

/* Returns a null pointer. */

{

return(NULL);

}

anotherrecord(recordpointer)

/* Returns true only if recordpointer

points to a record.

*/

listpointer recordpointer;

{

return(recordpointer != NULL);

}

info(pointer)

/* Returns the contents of the succobj field

of the record pointed to by pointer.

*/

listpointer pointer;

{

return(lists[pointer].succobj);

}

listpointer next(pointer)

/* Returns the link field value of

the record pointed to by pointer.

*/

listpointer recordpointer;

{

return(lists[recordpointer].link);

}

setinfo(pointer,value)

/* Copies value into the succobj field

of the record pointed to by pointer.

*/

listpointer pointer;

int value;

{

lists[pointer].succobj = value;

}

setlink(pointer1,pointer2)

/* Copies pointer 2 into the link field

of the record pointed to by pointer 1.

*/

listpointer pointer1,pointer2;

{

lists[pointer 1].link = pointer2;

}

listpointer avail()

/* Returns a pointer to storage allocated

for a list record.

*/

{

static int t=0;

t++;

return(t);

}

    >Comment

typedef listpointer succ_collection[LIMIT];

insert(i,j,succ)

/* Add j into the ith collection in succ. */

int i,j;

succ_collection succ;

{

listpointer pointer,avail();

pointer = avail();

setinfo(pointer,j);

setlink(pointer,succ[i]);

succ[i] = pointer;

}

succinitialization(n,succ)

/* Initializes all n collections of succ to empty. */

int n;

succ_collection succ;

{

int i;

listpointer setnull();

for (i=1;i<=n;i++)

succ[i] = setnull();

}

listpointer access_succ(i,succ)

/* Returns a pointer to the i collection in succ. */

int i;

succ_collection succ;

{

return(succ[i]);

}

    >Comment

typedef struct

{

outputarray rank;

int next;

}rankingrecord;

rankingrecord ranking;

1 allocates storage for ranking

copy(n,topologicalsort,ranking)

/* Copies the rank field of ranking

into topologicalsort.

*/

int n;

outputarray topologicalsort;

rankingrecord ranking;

{

int i;

for (i=1;i<=n;i++)

topologicalsort[i] = ranking.rank[i];

}

    >Comment

typedef int bagcollection;

baginitialization(pbag,n,count)

/* Initializes the bag so it contains

only objects whose counts are zero.

*/

bagcollection *pbag;

int n;

countcollection count;

{

int i;

ranking.next = 1;

*pbag = 1;

for(i=1;i<=n;i++)

if(iszero(i,count))

{

ranking.rank[*pbag] = i;

*pbag = *pbag + 1;

}

}

emptybag(bag)

/* Returns true only if bag is empty. */

bagcollection bag;

{

return(bag == ranking.next);

}

remove(pbag,pobj)

/* Sets obj to an object to be removed

from bag, and removes it.

*/

bagcollection *pbag;

int *pobj;

{

*pobj = ranking.rank[ranking.next];

ranking.next = ranking.next + 1;

}

process(listname,recordpointer,pbag,count)

/* Decreases the count of the successor pointed

to by recordpointer and adds it to the bag if

its count has become zero.

*/

listpointer listname,recordpointer;

bagcollection *pbag;

countcollection count;

{

int currentsucc;

currentsucc = info(recordpointer);

decrease(currentsucc,count);

if (iszero(currentsucc,count))

{

ranking.rank[*pbag] = currentsucc;

*pbag = *pbag + 1;

}

}

    >Comment

topsort(pn,topologicalsort)

/* Inputs the number of objects n and the

partial order and generates a solution in

topologicalsort.

*/

int *pn;

outputarray topologicalsort;

{

    >Comment

countcollection count;

succ_collection succ;

bagcollection bag;

int i,j,obj;

    >Comment

printf("\n enter n between 1 and %d\n",LIMIT-1);

scanf("%d",pn);

    >Comment

predinitialization(*pn,count);

    >Comment

succinitialization(*pn,succ);

    >Comment

while (nextpair(&i,&j))

{

    >Comment

increase(j,count);

    >Comment

insert(i,j,succ);

{

    >Comment

baginitialization(&bag,*pn,count);

    >Comment

while(!emptybag(bag))

{

    >Comment

remove(&bag,&obj);

    >Comment

update(succ,obj,&bag,count);

}

    >Comment

copy(*pn,topologicalsort,ranking);

}

nextpair(pi,pj)

int*pi,*pj;

/* Input the next pair and return true

only if it was not the sentinel pair.

*/

{

print("\n enter a pair \n");

scanf("%d %d",pi,pj);

return(!((*pi == 0) && (*pj == 0)));

}

    >Comment

update(succ,obj,pbag,count)

/* Update the counts of all successors of obj

and place any whose counts become zero

into bag.

*/

succ_collection succ;

int obj;

bagcollection *pbag;

countcollection count;

{

listpointer listname,recordpointer,access_succ();

listname = access_succ(obj,succ);

recordpointer = listname;

while(anotherrecord(recordpointer))

{

process(listname,recordpointer,pbag,count);

recordpointer = next(recordpointer);

}

}

Note that the ranking and bag implementations are not independent. Thus, if it is desirable to change the implementation of one, the other is also affected. As written, the bag acts as a stack, but it may be important to pick the entry to be removed from the bag more selectively. One way to do this is to implement the bag as a priority queue. In any case, to make ranking and bag independent requires changing their declarations, definitions, and basic operations.

11.4: Analysis of Topsort

We will now analyze the time requirements of this implementation. Refer to the refinement shown in the implementation that used data abstractions on page 503. Reading n takes constant time. The count and succ array initializations take time proportional to n. The while loop read and test takes constant time. Reading the next input pair and tasks 3a and 3b, which involve executing five instructions, take constant time. Each repetition of the loop thus takes constant time. Since the loop is executed once for each of the m input pairs, the total loop time will be proportional to m. Phase I thus takes some constant time, plus time proportional to n, plus time proportional to m.

Initializing the bag at the start of phase II takes time proportional to n, since it involves traversing the count array and placing each object whose count is zero into the bag. The while loop body of phase II, unlike that of phase I, does not take the same amount of time on every repetition. Task 5a takes constant time, but task 5b depends on the number of successors of the object that was removed by task 5a. The loop itself is executed n times, once for each object output.

How can we determine the total loop time required? Notice that each of the n objects will eventually be output in some loop execution. We do not know in what order the objects will be output. However, the time taken in its particular loop execution for each object output is the same. This time is at most a constant plus time proportional to the number of successors of the object. Thus, the total time for the while loop in phase II is the sum of the time required to output each object. The total time is then a constant times n plus time proportional to the sum

[(number of successors of object 1) plus (number of successors of object 2) . . . plus the (number of successors of object n)]

This sum is just the total number of successors, m. Hence the total time for phase II is made up of the same kinds of components as the total time for phase I. We conclude that the total time for this implementation has the same form.

Note that the search-based algorithm could take time O(n!). The straight-forward implementation of the construction-based algorithm could take time O(n   [image]  m). Topsort takes time of the form c1 + c2n + c3m and represents a very substantial improvement. It was made possible because we were able to determine precisely, and to obtain efficiently, the information required to do the processing steps of the algorithm. The topological sort problem is more complex than earlier ones in this book, but it illustrates the same theme. The processing to be done determines the data structures that are most appropriate. Clearly, any implementation of an algorithm that does topological sorting must take time of the form an + bm, since it is necessary to read in all m input pairs and output all n objects.

How much storage does the implementation require? Suppose the program is to run correctly for values of n between 1 and 20. The only difficulty in determining the actual amount of storage required is in deciding what length to declare for lists. This depends on how large m may be. In general, if there are n objects, each object can have no more than n - 1 successors--that is, an arrow to every other object. There are then at most n(n - 1) possible successors. Each successor requires one record of the lists array. Actually, because of the asymmetry property, at most one-half of the n(n - 1) possibilities can appear. Hence we need a total of at most 1/2   [image]  n(n - 1) records for lists. If n is 20, then (20   [image]  19)/2 records will do. The storage required is proportional to n2.

Knuth [1973a] and Wirth [1976] give different implementations of the algorithm for the topological sorting problem. See Aho, Hopcroft, and Ullman [1983] for a somewhat different point of view on its solution.

11.5: Behavior for Replicated Pairs or Loops

In the solution to the topological sorting problem, it is assumed that each distinct input pair appears only once and also that the input contains no loops. Either assumption might be violated. Of course, more obvious errors could occur. For example, n or one of the input pair members could be negative or out of range. It is not difficult to add a validation routine to check for these kinds of input errors. Subtler errors, such as replication of pairs or the occurrence of loops, are not as obviously remedied.

What does happen if pairs are replicated or loops occur in the input ? How would you recognize loops? Replication of an input pair i, j causes the count of j to be 1 larger than it should be. It also causes the successor j to appear on the list of successors of i one more time. When object i is eventually removed, its list of successors is traversed. This results in the count of j being reduced one additional time, so that the extra increase due to the i, j replication is cancelled. The implementation will thus work correctly as long as there is room for the extra successor records to appear in lists. If there is no room to accommodate these extra records, it is not possible to predict what will occur when the implementation is executed.

Loops, on the other hand, result in some objects never being output. Objects that are involved in a loop, or that are the successors of an object in a loop, will never have their counts go to zero, and will never appear in the bag of objects. In fact, had the test for completion been "S not empty" or "n objects output," loops in the input would cause an infinite loop in the program. It should now be clear that next-1, when the loop in phase II is exited, will be the actual number of objects that were output by the loop. If this is less than n, then loops occurred in the input. This can provide a simple test for loops.

11.6: Final Comments on Topsort

As already noted, the implementation of the output ranking and the bag are coupled because they share storage. As a result they are not independent. This means that a change in the choice of implementation for one will affect the other. In general, such a situation is to be avoided--and, as pointed out earlier, could have been avoided. Certainly the small saving in time and storage did not warrant the added complexity.

There are, however, some important storage considerations that were alluded to earlier. Topsort will run out of storage before it runs out of time, since storage requirements grow as n2. The bulk of the storage needed is taken by the lists array. Suppose a maximum size, based on the available storage of the computer system, is declared for the lists array. Say its length is 50,000. Assuming each of its records takes two entries, this provides enough storage to guarantee the solution of any problem with no more than 25,000 successors (m   [image]  25,000). It makes no difference how the successors are distributed among the objects; only the total number is relevant.

Suppose, instead, that the decision had been to represent each collection of successors by dedicating an array to the collection. Since the programmer does not know in advance how the successors will be distributed, each of the required n arrays should have the same length. Lists are no longer necessary, since the successors can be placed sequentially in the proper array. Of course, a variable will be needed to keep track of the last successor in each array. An additional array may be used for these pointers. If n is to be no greater than (say) 500, then each of the arrays containing the successors can be declared of length 50,000/500, or 100. The upshot is that the solution may now be guaranteed for a different class of problems. These problems must have n   [image]  500, and each object may have no more than 100 successors. Even though the total number of successors may now be 50,000 (twice as many as before), their distribution is critical. When storage is at a premium, such considerations are important.

Had dynamic memory been used instead of the lists array for storage of the successor records, the program would contain no explicit storage limitation (other than the declarations for the length of count, succ, and rank). Still, the maximum dynamic memory size would have imposed a limit on the value of m for which the program would execute.

11.6.1 An Input Validation

One final point involving time and storage trade-offs. Suppose an input validation must be added to topsort just prior to the processing of the current i, j pair read in phase I. The function is to check whether i, j is a duplicate of an earlier input pair. If so, it is to be ignored; if not, it is to be processed. One way to accomplish this, which requires no additional storage, is to traverse the current list of successors of object i. If j appears on the list as a successor, then i, j is a duplicate; otherwise it is not. This takes time proportional to the current number of successors of i and thus adds a total time to phase I that is proportional to m2. Instead, the function can work in constant time if n(n - 1) additional storage is available. Use the storage for an n(n - 1) array that is initialized to all zeros. When i, j is read, simply test the (i, j)th array entry. If it is zero, the pair is not a duplicate. Then set the entry to 1 to indicate that this i, j pair has been processed. In this way, an entry of 1 will mean that the pair is a duplicate from now on. This adds total time to phase I of O(m) but requires n(n - 1) additional storage.

11.7: Reviewing Methodology

A topological sort produces a ranking of objects that satisfies constraints on their allowable positions within the ranking. The obvious algorithm for finding a topological sort, searching through all rankings until one satisfying the constraints is found, is not feasible. A feasible algorithm was developed by constructing a ranking that satisfied the constraints. The initial implementation merely produced an image of the input data in memory. It was improved significantly by using an array to contain predecessor counts, creating lists of successors, and introducing a bag to hold objects whose predecessor counts became zero. This solution was achieved by a process of stepwise refinement, stressing data abstraction, encapsulation, modularity, and the proper choice of data structures. The functional modularity of the final solution allowed immediate application of a routine developed earlier--the list traversal routine. Again, because of functional modularity, it was not difficult to determine where and how to build in appropriate checks of the input data.

More generally, the analysis of the storage and time requirements of a fairly complex program was demonstrated. Even though this cannot always be accomplished, or may be difficult for complex programs, it serves as an example of what we hope to be able to do.

Exercises

1.     Within a textbook, chapters cover specific topics. The information required to understand each topic is expected to have appeared before the topic appears in the book. Suppose you are given a list of topics and the other topics on which each depends. How might you select an ordering of the topics for their appearance in the book?

2.     Which of these binary relations is a partial order on S?

a.     "Square root of" on the set S of all real numbers greater than 1

b.     "Is older than" on the set S of all your relatives

c.     "Sits in front of" on the set S of all students in a class

d.     "Lives diagonally across from" on the set S of all people

3.     Write down the graphic representation for the binary relation R on the first 10 integers. R = (3, 2), (4, 6), (7, 6), (8, 1), (9, 7), (9, 8).

4.     Write a detailed modification of the algorithm of Section 11.2 that will check if a given ranking is consistent with the given partial order.

5.     Write a program to implement the refinement of Exercise 4 and determine its worst-case time and storage requirements.

6.     Apply the construction-based algorithm of Section 11.3 to the following data to obtain a consistent ranking. Always select the smallest integer for output.

S = 1, 2, 3, . . . , 12

R = (3, 2), (4, 6), (7, 9), (12, 10), (5, 6), (2, 4), (8, 9), (2, 6), (3, 6), (7, 12), (3, 4), (7, 10)

7.     If the second implementation of the construction-based algorithm is applied to the input of Exercise 6, what integers are in the bag after 3 is removed?

8.     State clearly and concisely what purpose the bag serves in the better "bag" refinement and why it was introduced.

9.     Write an expanded version of the better refinement that will output integers from the bag by selecting the smallest possible integer first.

10.     What will the succ, lists and count arrays have in them after phase I, when the input pairs of the running example appear in the order 7 6, 5 3, 2 10, 6 8, 5 6, 7 3, 4 8, 7 1, 4 2?

11.     What will the count, succ, and lists arrays look like after phase II as compared to after phase I?

12.     Suppose the better implementation were modified so that no count array were stored in memory, and only the succ and lists arrays were available after phase I. Write a new phase II refinement under this constraint.

13.     Suppose a solution to the topological sort problem had been constructed by determining what object should be at the bottom of the ranking. How would the better refinement be changed to reflect this new solution?

14.     a.     Suppose the bag is implemented in a separate record bag instead of using the ranking record. How must the data abstraction implementations change?

b.     Suppose the output ranking is not kept in the ranking record but, instead, is simply printed. How must the program and data abstraction implementations change?

15.     The function process is itself dependent on the ranking and bag implementations. Modify it so that process becomes independent of these implementations.

16.     Suppose task 5b of the better refinement is further expanded to:

5b.i.  For each successor of the output object, decrease its predecessor count by 1.

5b.ii. For each successor of the output object, if its predecessor count is zero, add the     successor to the bag.

a.     ow should the implementation of the data abstraction algorithm (on p. 503) be modified to reflect this refinement?

b.     rite implementations for tasks 5b.i and 5b.ii that use traverse as a tool. This should include writing the corresponding two process functions.

c.     ow will this expansion of task 5b affect the time required for the algorithm?

17.     Suppose the input for Exercise 10 had the pairs below appended after 4 2. What would the graphic depictions of the successor lists be after phase I of topsort?

7 3, 5 6, 4 2, 7 3, 6 8

18.     For the input of Exercise 17, what would be in the count array after phase I?

19.     For the input of Exercise 17, show what will be in the count array, the successor lists, and the bag after 7 is output.

20.     Take as input the Example 11.6 with pairs 10 4 and 10 1 appended after 4 2. What will the second implementation of topsort output in the rank array? What will be the value of next after phase II?

Suggested Assignments

1.     Write and run a program whose input will be a series of topological sorting problems. For each problem the program should execute a slight modification of the topsort function called newtopsort. Newtopsort has parameters n and topologicalsort. Newtopsort is to read in and echo print n and all input pairs for the example it has been called to work on. It should print out the relevant portions of the count, succ and lists arrays as they appear after phase I. When newtopsort returns, the program prints the topological sort contained in the topologicalsort array. Except for the modifications needed to do the printing, newtopsort corresponds exactly to the sample implementation of topsort, with one exception: Implement the bag as an array separate from rank. Newtopsort should work correctly for valid input and for any number of objects between 1 and 20. You should make up four input sample problems to use. Two should be valid. One should include replications, and one should contain loops.

2.     Suppose, instead of using the sample implementation, you do the following to keep successor information: Declare lists to be an n(n - 1) two-dimensional array and keep the successors of object i in the ith row of lists. For the example, after phase I, lists would be This two-dimensional lists array takes up the same total amount of storage as the one-dimensional lists array of the better implementation. For both implementations, the limiting factor on whether or not the program will run might then be the value of n beyond which available storage is exhausted. Suppose you are willing to give up the guarantee that topsort will always run correctly. This means you may attempt to run problems whose n is large enough that you are not sure whether the implementation of lists can hold all successors. Discuss the relative advantages of the implementations.

3.     This is the same as Assignment 1 except, instead of the lists array, use dynamic memory to store the successor lists.

4.     Assume your program for Assignment 1 treated the bag with the operations empty-bag, remove, and baginitialization and count with the operations predinitialization and increase as data abstractions.

a.     What does this mean?

b.     Write a function, number, that also treats them as data abstractions and returns the number of objects in the bag when it is invoked. When it returns, the bag must contain exactly the same objects as before number was invoked. Assume number does not have access to information about the output ranking.

Go to Chapter 12      Return to Table of Contents


Data Structures Using C: CHAPTER 12: HUFFMAN CODING AND OPTIMAL AND NE

 Return to Table of Contents

CHAPTER 12: HUFFMAN CODING AND OPTIMAL AND NEARLY OPTIMAL BINARY SEARCH TREES

Three illustrations of the critical effect on program behavior of data structure selection are presented

Huffman coding, a text compression and coding technique, is used to show

good algorithm development

the benefits of good data structure selection

Optimal binary search trees, a related topic, are useful for storing static collections of information to be randomly accessed and traversed in sorted order; they demonstrate

the use of recursion

how to develop efficient programs based on recursion

Nearly optimal binary search trees are considered to emphasize program development

12.1: Techniques for Compressing Text or Storing Records

Many important techniques are available for the compression of text. Text compression is important when dealing with large collections of information, since internal memory is scarce and access to external memory is slow. Huffman coding is an elegant method that can be used profitably when characters do not appear with equal frequency in a text. This is the case with the letters of the alphabet in English text.

Binary trees can be used as the basis for encoding text and for the creation of Huffman codes. Binary search trees can also be used to generate more quickly codes of comparable efficiency.

The application of "simple" and balanced binary search trees for record storage and retrieval by key was demonstrated in Chapter 9. As shown there, "simple" binary search trees provide excellent average search times with random input when keys are accessed equally often. Also, AVL trees guarantee excellent worst-case search time. We shall see in this chapter that optimal binary search trees are useful when the keys have known, even unequal, frequencies, and when the trees, once grown, remain unchanged. Words in English text occur with known and unequal frequencies. An optimal binary search tree could be applied, for example, as a dictionary storing the 500 most frequently used English words, along with their definitions and their common misspellings. A search of the tree would then yield the definitions and misspellings. Once built, this tree would remain unchanged.

This chapter shows that the problems of finding Huffman codes and of finding optimal or nearly optimal binary search trees are related. The solutions illustrate the use of trees, heaps, and list-structures in the construction of good algorithms. The chapter also introduces the "greedy" heuristic. This method leads to an optimal solution for finding Huffman codes and to a nearly optimal solution for finding binary search trees. In both cases, the proper selection of data structures plays a significant role.

12.2: Weighted Path Length

An important measure associated with binary trees is the weighted path length. This measure underlies the entire development in this chapter. Suppose a numeric weight is assigned to each node of a binary tree T. The meaning of these weights is dependent on the intended application. When the application of the binary tree is to text compression or text encoding, only the weights assigned to terminal nodes have meaning. Each terminal node corresponds to a different character that can appear in the text. The weight assigned to the node represents the relative number of occurrences of the character in the text. When the binary tree is used for record storage, each node stores a record with a particular key. The weight assigned to a node represents the frequency of search requests for the key corresponding to the node. These weights must be estimated or determined empirically for each specific application.

The weighted path length of a node in T is the product of its depth and its assigned weight. The weighted path length of T, W(T), is the sum of the weighted path lengths of all nodes of T. As an example, consider the binary tree in Figure 12.1(a), with assigned weights indicated for each node. Its weighted path length is

(1    3) + (2    17) + (2    10) + (3    15) + (3    0)

 + (3    20) + (4    31) + (4    4)

or 302. This was calculated directly from the definition of the weighted path length.

Another way to determine the weighted path length is to determine first the value of each node of the tree. The value of a node is the sum of the assigned weights of all nodes in the subtree with that node as root. Thus the root of a tree is assigned a value that is the sum of all the weights of the tree. The example would have values assigned to each node as indicated in Figure 12.1(b). The weighted path length of the tree is then calculated by adding up the values of the nodes.Thus, from Figure 12.1(b), the weighted path length is

100 + 67 + 30 + 15 + 35 + 20 + 31 + 4 = 302

It is easy to see that adding the node values must always yield the weighted path length, since the weight of any node is counted once for every subtree in which it appears. For example, the weight 31 is counted in the value attached to each node from the node to which it is assigned to the root. This is a total of four nodes, exactly the number of times that the node's weight should count in the weighted path length.

(a) Nodes Labeled with Assigned Weights

(b) Nodes Labeled with Values Derived by Adding Assigned Weights

Figure 12.1 Binary Tree T

The weighted path length can be viewed in still another way. It is given by the value of the root plus the weighted path length of the two subtrees of the root node. For the example shown in Figure 12.1(b), this is given by 100 + 152 + 50, where 100 is the root's value and 152 and 50 are the respective weighted path lengths of the left and right subtrees. Thus there are three ways in which the weighted path length of a binary tree can be calculated.

12.3: Huffman Coding

With this background, let us turn to Huffman coding itself. First we will look at how information may be coded based on a binary tree and how it is coded in computers. Consider a binary tree with n terminal nodes. Assign a zero to every left successor branch and a one to every right successor branch of the tree. Encode each terminal node with the sequence of 0's and l's encountered when following the path from the root to the terminal node. Since this path is unique, the sequence or code assigned to each node must be unique. For example, in binary tree T1 (shown in Figure 12.2) the twenty-six letters of the alphabet and the space character designated by [image] were assigned to the terminal nodes. Thus the code for the leftmost terminal node (N) in Figure 12.2 is the sequence 00000. Any character, word, or information associated with a terminal node is encoded with the unique code for that node.

The code assigned to the letter S in T1 is the sequence 00011 of length 5, one less than the depth of the encoded node to which S is assigned. In this way, a code is automatically generated for the twenty-seven characters associated with the terminal nodes of T1. Any string of these characters can then be encoded into a string of 0's and 1's. This string is simply the sequence of the combined sequences of 0's and l's that constitute the codes for each of the characters in the string. For instance, the string of six characters "A [image] TREE" will be encoded

100101001100011111

of length 18. Again, because each path is unique, given any two code words, one can never be an extension of another. Consequently, no other sequence of code words will yield exactly this sequence of eighteen 0's and 1's, and any sequence starting with these eighteen 0's and 1's must start with A [image] TREE.

The tree T1 generates a variable-length code, since it does not assign the same length sequences of 0's and 1's to each character. A fixed-length code assigns equal-length strings of 0's and 1's to each character. A fixed-length code would be generated by a tree if each terminal node of the tree had the same depth.

Figure 12.2 Binary Tree T1

Since elements of computer memory actually store only sequences of 0's and 1's (by design), character information must be encoded. One way to encode the character set of a computer is by means of a standard fixed-length code, which uses six 0's and 1's per character.

The character set of a computer is the alphabet of characters that it can accept. For example, not all computers accept braces (that is, {}). Two important standard character sets are ASCII (American Standard Code for Information Interchange) and EBCDIC (Extended Binary Coded Decimal Interchange Codes). Both use fixed-length codes. ASCII uses a seven-bit code and EBCDIC uses eight bits. An n-bit fixed-length code can distinguish 2n characters, and a sequence of l characters would then be represented by a sequence of l X n 0's and 1's. Variable-length codes are not constrained to using the same number of 0's and l's for each character.

Using a variable-length code, a text in which the characters occur with different frequencies can be compressed. Shorter-length sequences would be assigned to characters used more frequently, and longer-length sequences to characters used less frequently. This is the idea behind the Morse code. Morse, in fact, estimated the relative frequencies of characters by looking at the frequency with which printer's boxes containing the different characters of the alphabet needed to be refilled.

The use of fixed-length codes makes it easy to decode a sequence of 0's and 1's to recover the original sequence of characters or text. It is not so easy for variable-length codes. A variable-length code must be chosen with care, if it is to allow proper decoding and avoid ambiguity. One way to achieve such a code is to use the scheme outlined for Figure 12.2, which associates a code to each binary tree. Decoding the code generated by a given binary tree would then proceed as follows:

1. Start at the root of a binary tree that is identical to the one used for encoding the original text.

2. Follow the branches according to the dictates of the sequence of 0's and l's to be decoded--go left for a 0 and right for a 1.

3. When a terminal node is reached, the sequence of 0's and 1's that led to it is decoded as the character associated with that terminal node.

4. Start again at the root to decode the remainder of the sequence in the same way.

The relative frequency with which each of twenty-six characters of the alphabet and the blank space between words occur in English text is as follows:

      A   B   C   D    E   F   G   H   I   J  K  L

186  64  13  22  32  103  21  15  47  57  1  5  32

M   N   O   P   Q  R   S   T   U   V  W   X  Y   Z

20  57  63  15  1  48  51  80  23  8  18  1  16  1

These relative frequencies are used to assign weights to the nodes of a tree such as T1. Each internal node is assigned weight 0, and each terminal node is assigned the relative frequency of the character associated with it. Thus, node S of T1 is assigned weight 51. The average amount of compression is measured by the weighted path length of the tree. In this way, the amount of compression achieved with different codes can be compared, since each tree has associated with it a unique code. The average length of the assigned codes is obtained by subtracting 1 from the weighted path length when the weights are normalized to sum to 1. Weights are normalized by dividing each weight by the sum of all the weights. The designer of a compression scheme can make this calculation directly or write a program for this purpose.

The weighted path length of T1 can be improved. Take the subtree with root at depth 5, and I and S as its successors, and interchange it with T to obtain binary tree T2 (shown in Figure 12.3). The weights of I and S, 57 and 51, will now each contribute one less time to the weighted path length of T2 than to T1, while T, with weight 80, will contribute one more time to the weighted path length of T2 than to T1. The result is that the weighted path length of T2 will be (57 + 51) - 80 less than that of T1.

Figure 12.3 Binary Tree T2, Obtained by Interchanging Two Subtrees

The programmer's goal should be to find a binary tree, called the optimal binary tree, which has minimum weighted path length when the characters of the alphabet are stored at its n terminal nodes. Incidentally, the references here are to characters associated with the terminal nodes, but more generally they can be words or messages. A straightforward algorithm for finding this optimal binary tree is to generate each binary tree with n terminal nodes, associate the alphabetic characters with these n nodes, and calculate the weighted path length of the tree. A tree that yields the minimal weighted path length is optimal. Unfortunately, this algorithm is feasible only for small n. If n = 13, there are more than 1012 such trees, and the number of trees increases by a factor of almost 4 each time n increases by 1. We have to find a better way.

12.3.1 The Huffman Algorithm

An optimal binary tree generates what is called a Huffman code. Fortunately, the searching just described is not needed to find it; it can be constructed as follows.

1.     Start with a collection of n trees, each with just a root. Assign an alphabetic character to each of these nodes. The weight of each node is the weight associated with its alphabetic character. The value of each of these root nodes is equivalent to its weight. (Value in this case equals the weight of the root itself, since these trees have no other nodes.)

2.     Find the two trees with smallest values. Combine them as the successors of a tree whose root has a value that is the sum of their values. Remove the two combined trees and add the new tree obtained to the current collection of trees. Repeat the second step n - 2 times to obtain exactly one tree. This will be the optimal tree and will give the assignments of characters to terminal nodes.

When this algorithm is applied to the twenty-seven characters with their weights reflecting their frequency of occurrence in English, the first combination made is J with Q, since each has weight 1. The resultant tree has value 2. Next X and Z, also each of weight 1, are combined, yielding another tree with value 2. These two trees are combined, yielding a tree with value 4. The current collection now consists of twenty-four trees. The next combination involves weights 4 and 5 and yields value 9. Following this procedure, a final optimal tree that might result from the construction is shown in Figure 12.4. Other optimal trees could have been constructed, but all would have the same weighted path length of 5,124. This means that the average length (number) of 0's and 1's needed per character will be 4.124. The value 4.124 is obtained by normalizing the weighted path length (dividing it by the sum of the weights) and subtracting one. Thus 5124/1000 - 1 =  4.124. This saves almost 20 percent over a fixed-length code of length 5 (the shortest fixed-length code that can distinguish twenty-seven characters).

This construction procedure is the Huffman algorithm. A proof is given later showing that it does, in fact, produce an optimal binary tree. Notice that a cumulative total weight can be kept as the algorithm is carried out. To do this, start with the variable weight set at the sum of all the weights of the alphabetic characters. Every time two trees are combined, add the value of the resultant tree to weight. When the algorithm terminates, after making (n - 1) combinations, weight will contain the weighted path length of the optimal binary tree constructed.

The algorithm chooses the combination of current trees that yields a minimal value for the combined trees. It is in this sense that it is a "greedy" algorithm. It doesn't consider future ramifications of the current choice of subtrees to combine but takes the current two "best" subtrees. In this instance, best means "of smallest value." The algorithm makes a "local" best or greedy decision. It is an example of the greedy method of algorithm design, which involves making choices on the basis of the immediate best alternative. This may lead to later choices being constrained to such poor alternatives as to lead to an overall suboptimal result. Being less greedy now may lead to better choices later, which in turn may yield better overall results. Hence it is not obvious that taking the best possible next combination, as done here, must lead to an optimal tree.

Figure 12.4 A Final Optimal Tree

The greedy method does not yield optimal solutions in all situations. A traveler who applied the greedy method to selecting a route might always choose the next road because it is the shortest of all the current possibilities. This method will clearly not always yield the shortest overall route. When making change in U.S. currency using our system of 1-, 5-, 10-, 25-, and 50-cent pieces and dollar bills, we normally use a greedy algorithm. For instance, to give $3.70 change most people would give three $1 bills, one 50-cent piece, and two dimes (not seven 50-cent pieces, one dime, and two nickels). The algorithm is greedy because it always takes as many pieces of currency of the next largest value as possible. This always minimizes the total number of pieces of currency used to make change in our system. If a system had 1-, 5-, and 11-cent pieces and dollar bills, then making change "greedily" to give $3.70 would require three $1 bills, six 11-cent pieces, and four pennies. Fewer pieces of currency result from using three dollar bills, five 11-cent pieces, and three nickels, Fortunately, the greedy method does yield optimal binary trees.

12.3.2 Representation of Huffman Trees

Let us now consider how the binary tree constructed by the Huffman algorithm should be stored in memory. To this end we consider an example that shows that even though the weighted path length is as small as possible for a given set of weights, the depth of the tree can still be great.

Example 12.1

Suppose that the first n Fibonacci* numbers are assigned as weights to characters of an alphabet represented as terminal nodes in a tree. The task is to generate a Huffman code.

*The Fibonacci sequence is the sequence of integers F 0, F 1, F 2, . . . that starts with F 0 = 0 and F 1 = 1, with each succeeding integer found by adding its two predecessors. Thus the sequence is 0, 1, 1, 2, 3, 5, 8, 13, 21, 34, 55, . . . , Fn-1, . . . , where F 0 = 0, F 1 = 1, Fn+2 = Fn+1 + F n, n   [image]  0.

The Huffman algorithm would construct the tree shown in Figure 12.5. This tree has depth n. Hence the Huffman algorithm may generate trees whose depth is O(n). If a sequential representation were used for such trees, considerable storage would be wasted. For large n, the storage would not even be available. A good alternative is a linked representation of the tree.

Figure 12.5 Tree Constructed by the Huffman Algorithm for Fibonacci Weights

12.3.3 Implementation

In this section the Huffman algorithm is implemented so that the Huffman tree and its weighted path length are generated. For a list of n records, a linked representation of the tree is illustrated in Figure 12.6(a) for n = 5 and weights 4, 3, 10, 2, 6.

The X's in the figure represent null pointers. The information field of each list record contains a pointer to a record representing the root of a binary tree. The records of the tree might be stored using pointer variables or in arrays. The records have leftptr, info, and rightptr fields. The info field contains the value assigned to the record's subtree by the Huffman algorithm. Thus each list record represents a tree. Weight is a variable initially set to the sum of the n weights and incremented by the value attached to each combined tree.

The implementation could proceed by traversing the list to find the trees with the two smallest values. These trees must be combined as dictated by step 2 of the algorithm into a new tree whose value is the sum of their values. The two trees that were combined must be deleted from the list, and the new tree added to the list. The result of these operations is shown in Figure 12.6(b) after one combination has taken place.

Every time step 2 is repeated, two trees are combined and the length of the list decreases by 1. The final list would contain one record; its information field would point to the root of the generated Huffman tree. Weight would contain its weighted path length. For this example, the resultant list would be as shown in Figure 12.6(c).

The list traversal required to find the two smallest value trees results in a worst-case time proportional to the length of the list for each traversal. Since the tree must be traversed n - 1 times, the worst-case time is O(n2). We can do better.

To improve the implementation, the two smallest values whose trees are to be combined must be found quickly without traversing the entire list. Recall that a heap, in this case a min heap, always has its smallest record at its root. Suppose the records of the binary tree are kept in a heap, or better, suppose the pointers are kept in the heap. Each pointer points to a corresponding record of the binary tree. The initial heap is shown in Figure 12.7.

(a) List of Five Records Pointing to Five Initial Trees of the Huffman Algorithm

(b) List after Combining Two Trees with Smallest Values

(c) Final List and Huffman Tree after Repeated Combinings of Values

Figure 12.6 A Linked List Representing the Trees Constructed by the Huffman Algorithm

To find the two smallest values, simply remove the top heap pointer T4, reheap, and again remove the top pointer, T2. The records corresponding to the two smallest values can be combined. The new pointer to the combined record can then be placed at the top of the heap (which was left vacant), and the heap reheaped. Figure 12.7(b) shows the heap obtained by removing T4, reheaping, and removing T2. The two trees removed are combined, and reheaping is done to obtain the heap shown in Figure 12.7(c).

(a) Initial Heap

(b) Incomplete Heap

(c)Heap after Two Trees Are Combined and Reheaping Done

Figure 12.7 Heap of Pointers of the Huffman Tree

With this implementation the computer time consumed in finding the two smallest values, combining their corresponding records, and reheaping is O(lg n). Since the entire process is repeated (n - 1) times, the total time will be O(n lg n). It was the proper choice of data structure that made this improvement possible!

12.3.4 A Proof

For those readers who would like a proof of the Huffman construction, the following is an interesting example of the use of mathematical induction applied to trees. It can easily be adopted to prove that a recursive program for the Huffman algorithm is correct.

Assume the weights are w1, w2, . . . , wn and are indexed so that w1   [image]  w2   [image]  . . .    wn. You should convince yourself that an optimal binary tree must be a full binary tree.* An optimal tree can always be found, since the number of full binary trees with n terminal nodes is finite. It is clear that the Huffman construction yields optimal trees for n = 1 and 2. This is the basis for the induction, which will be on the number of weights or terminal nodes, n. The induction hypothesis is that the Huffman construction yields optimal trees for any n - 1 weights. We must show that as a result, it must generate optimal trees for n weights.

*See Exercise 5.

Given any tree with n terminal nodes, Tn, in which Figure 12.8(a) appears as a subtree, let T'n - 1 be the tree with n - 1 terminal nodes obtained from Tn by replacing this subtree by the terminal node shown in Figure 12.8(b). Then the weighted path length of Tn equals the weighted path length of T'n - 1 + (w1 + w2).

Given T'n-1, inversely, Tn can be obtained from it. Therefore, Tn must be optimal with respect to weights w1, w2 . . . , wn, if and only if T'n-1 is optimal with respect to weights (w1 + w2), w3, . . . , wn. This is because if only one tree were not optimal, it could be replaced by an optimal tree, thus improving the weighted path length of the corresponding version of the other.

By the induction hypothesis, the Huffman construction yields a tree, T'n-1, with n - 1 terminal nodes, that is optimal with respect to weights (w1 + w2), w3, . . . , wn. The corresponding tree, Tn, with n terminal nodes, is just the tree given by the Huffman construction for weights w1, w2 , . . . , wn . Since T'n-1 is optimal, so is Tn. This completes the proof.

(a) Subtree

(b)Terminal Node after Combining Weights

Figure 12.8 Subtree Replacement

12.4: Optimal Binary Search Trees

So far we have been considering binary trees with nonzero weights assigned only to terminal nodes. Information (a character or message) has been associated only with their terminal nodes. We now consider extended binary search trees, which have keys stored at their internal nodes.

Suppose n keys, k1, k2, . . . , kn, are stored at the internal nodes of a binary search tree. It is assumed that the keys are given in sorted order, so that k1 < k2 < . . . < kn. An extended binary search tree is obtained from the binary search tree by adding successor nodes to each of its terminal nodes as indicated in Figure 12.9 by  's.

Although the programming goal in this chapter is to find optimal binary search trees, extended binary search trees are used along the way. In the extended tree, the  's represent terminal nodes, while the other nodes are internal nodes. These terminal nodes represent unsuccessful searches of the tree for key values. Such searches do not end successfully because the search key is not actually stored in the tree. They actually terminate when a null subtree pointer is encountered along the search path.

In general, the terminal node in the extended tree that is the left successor of k1 can be interpreted as representing all key values that are not stored and are less than k1. Similarly, the terminal node in the extended tree that is the right successor of kn represents all key values not stored in the tree that are greater than kn. The terminal node that is accessed between ki and ki+1 in an inorder traversal represents all key values not stored that lie between ki and ki+1. For example, in the extended tree in Figure 12.9(b), if the possible key values are 0,1, 2, 3, . . . ,100, then the terminal node labeled 0 represents the missing key values 0,1, and 2 if k1 is 3. The terminal node labeled 3 represents the key values between k3 and k4. If k3 is 17 and k4 is 21, then the terminal node labeled 3 represents the missing key values 18,19, and 20. If k6 is 90, then terminal node 6 represents the missing key values 91 through 100.

Assuming that the relative frequency with which each key value is accessed is known, weights can be assigned to each node of the extended tree. The weights represent the relative frequencies of searches terminating at each node. The weighted path length of the extended tree is then a natural measure for the average time to search the binary search tree for a key.

(a) Binary Search Tree

(b) Extended Binary Search Tree

Figure 12.9 Extension of a Binary Search Tree

Example 12.2

Find the extended binary search tree that has the minimal weighted path length. The optimal binary search tree is obtained from this tree simply by omitting the extended nodes.

Before dealing with this problem, we compare it to the Huffman coding problem that has already been solved. When the weights attached to the internal nodes of the extended binary search tree are zero, this problem is similar to the Huffman coding problem. The difference is that the task now is to find the binary search tree with minimal weighted path length, whereas the Huffman algorithm finds the binary tree with minimal weighted path length. Since the binary search trees are only a subset of all binary trees, the Huffman tree will yield a value of the minimal weighted path length that is never larger than that of the optimal binary search tree. The Huffman algorithm will not solve the current problem unless, by chance, the tree that it constructs is a binary search tree or can be converted to one with no increase in the weighted path length. An algorithm, the Hu-Tucker algorithm, has been developed for this special case, and requires, as does the Huffman algorithm, time O(n lg n) and O(n) storage. Initially, its time was thought to be O(n2), but Knuth showed how to reduce this time by selecting the appropriate data structure. We now proceed to the general case in which the weights of the internal nodes need not be zero.

12.4.1 Finding Optimal Binary Search Trees

An obvious way to find an optimal binary search tree is to generate each possible binary search tree for the keys, calculate its weighted path length, and keep that tree with the smallest weighted path length. This search through all possible solutions is not feasible except for small n, since the number of such trees grows exponentially with n. A feasible alternative would be a recursive algorithm based on the structure inherent in optimal binary search trees. Such an algorithm can be developed in a way similar to that used for the construction of Fibonacci trees in Chapter 7.

How do we decompose the optimal search tree problem in Example 12.2 into components with the same structure? Consider the characteristics of any optimal tree. Of course, it has a root and two subtrees. A moment's reflection should convince you that both subtrees must themselves be optimal binary search trees with respect to their keys and weights. First, any subtree of any binary search tree must be a binary search tree. Second, the subtrees must also be optimal. Otherwise, they could be replaced by optimal subtrees with smaller weighted path lengths, which would imply that the original tree could not have been optimal.

Since there are n possible keys as candidates for the root of the optimal tree, the recursive solution must try them all. For each candidate key as root, all keys less than that key must appear in its left subtree, and all keys greater than it must appear in its right subtree. To state the recursive algorithm based on these observations requires some notation.

Denote the weights assigned to the n stored keys by   [image] 's and the weights assigned to the terminal nodes by   [image] 's. The weight assigned to ki is   [image] i for i = 1, 2, . . . , n, and that assigned to the external node labeled i is   [image] i for i = 0, 1, 2, . . . , n. Let obst(i,j) denote the optimal binary search tree containing keys ki, . . . , kj, and let w(i,j) denote its weighted path length. Obst(i,j) will involve weights   [image] i-1,   [image] i, . . . ,   [image] i,   [image] j.

The optimal tree with root constrained to be kk and containing keys ki, . . . , kk, . . . , kj must then have obst(i,k-1) as its left subtree and obst(k+1,...,j) as its right subtree. Its weighted path length, w(i, j), is given by sw(i, j) + w(i, k - 1) + w(k + 1, j), where sw(i, j) is the sum of the weights   [image] i-1,   [image] i, . . . ,   [image] j,   [image] j. Sw(i, j) is the value assigned to the root of obst(i,j), the sum of its weights.

Finally the algorithm can be stated.

For each k(k) as root, k = 1, 2, . . . , n

find obst(1,k-1) and obst(k+1,n).

Find a k that minimizes [w(1, k - 1) + w(k + 1, n)] over k = 1, 2, . . . , n.

Obst(1,n) is given by the tree with root kk, obst(1,k-1) as its left subtree, and

obst(k+1,n) as its right subtree.

The weighted path length of obst(1,n) is w(1, n).

w(1, n) = sw(1, n) + w(1, k-1) + w(k+1, n).

The execution tree for obst(1,4) is given in Figure 12.10. Apparently this algorithm exhibits the same proliferation of recursive calls for identical component problems as did the algorithm for the Fibonacci tree construction in Chapter 7. These are summarized in Table 12.1. This inefficiency can be avoided by using a bottom-up construction, just as was done for Fibonacci trees. To do this requires a clear understanding of exactly which optimal subtrees must be retained for later reference.

All possible optimal subtrees are not required. Those that are needed consist of sequences of keys that are immediate successors of the smallest key in the

Table 12.1 Components Replicated by Recursive Calls

Replicated Component  Occurrences

----------------------------------

 (  2,   3,   3,   4,   4)       2

 (  0,   1,   1,   2,   2)       2

 (  1,   2,   2,   3,   3)       2

 (  0,   1,   1)               4

 (  1,   2,   2)               5

 (  2,   3,   3)               5

 (  3,   4,   4)               4

 (  0)                       4

 (  1)                       3

 (  2)                       4

 (  3)                       3

 (  4)                       4

Figure 12.10 The Execution Tree for n = 4

subtree, successors in the sorted order for the keys. The final solution tree contains all n keys. There will be two with n - 1 keys, three with n - 2 keys, four with n - 3 keys, and in general k + 1 with n - k keys, for k = 0, 1, 2, . . . , n. Consequently there are a total of (n + 1) X (n + 2)/2 optimal subtrees to be retained. Notice that to determine obst(i,j) requires knowing the optimal smaller subtrees involving weights:

(  i-1) and (  i,   i+1, . . . ,   j,   j)

i.e., obst(i,i-1) and obst(i+1,j)

(  i-1,   i,   i) and (  i+1,   i+2, . . . , Bj,   j)

i.e., obst(i,i) and obst(i+2,j)

(  i-1,   i, . . . ,   j-1,   j-1) and (  j)

i.e., obst(i,j-1) and obst(j+1,j)

The bottom-up approach generates all smallest required optimal subtrees first, then all next smallest, and so on until the final solution involving all the weights is found. Since the algorithm requires access to each subtree's weighted path length, these weighted path lengths must also be retained to avoid their recalculation. Finally, the root of each subtree must also be stored for reference.

This algorithm requires O(n3) execution time and O(n2) storage. Knuth, who developed this solution, noticed an important fact that reduces the execution time to O(n2). He was able to show that the minimization need not be over all k from 1 to n. Instead, the minimization may be taken only over all k between r(i,j - 1) and r(i + 1,j), where r(i,j) denotes the root of obst(i,j). See Knuth [1973b].

Intuitively, this is plausible because obst(i,j-1) is the optimal binary search tree containing keys ki, . . . , kr(i,j-1), . . . , kj-1. Thus taking kr(i,j-1) as its root gives the tree just the "balance" it needs to be optimal, distributing keys to its left and right subtrees in the optimal way. Obst(i,j) includes kj as well. If its root is also kr(i,j-1), then kj will appear in the right subtree of this optimal tree adding more weight to its right subtree. Think of the optimal tree as "balanced." Its root should not be a predecessor of kr(i,j-1), since this would cause even more weight to appear in its right subtree. Reasoning this way, it is not unreasonable to expect the root of obst(i,j) to be kr(i,j-1) or one of its successors. Thus looking only at k where r(i,  j - 1)   [image]  k should suffice. A symmetrical argument should make it equally reasonable that looking only at k, k   [image]  r(i + 1, j) should suffice. The actual proof is complex and adds little insight, so it is omitted here.

The bottom-up algorithm, incorporating the new minimization limits, may now be written.

1.     For i from 0 to n

a.     Set w(i + 1, i) to 0 and sw(i + 1, i) to   [image] i

b.     For j from i + 1 to n

Set sw(i + 1, j) to sw(i + 1, j - 1) +   [image] j +   [image] j

2.     For j from 1 to n

a.      Set w(j, j) to sw(j, j) and r(j, j) to j.

(This initializes all obst's containing 0 keys and 1 key.)

3.     For l from 2 to n

a.     For j from l to n,

i.     set i to j - l + 1;

ii     set w(i, j) to sw(i, j) + minimum[w(i, k - 1) + w(k + 1,j)], the minimum to be over k satisfying r(i, j - 1)   [image]  k   [image]  r(i + 1, j) and set r(i, j) to a value of k corresponding to the minimum.

Actually, the w(i,j)'s will differ from the weighted path length of obst(i,j) by the sum of   [image] i-1,   [image] i, . . . ,   [image] j. This does not affect the final tree but means that to obtain the true weighted path length,   [image] 0 +   [image] 1 + . . . +   [image] n must be added to w(1, n).

Arrays for w, ws, and r seem to be a natural choice for implementing this algorithm and provide rapid access to the information required for the processing involved.

Example 12.3

Find the optimal binary search tree for n = 6 and weights   [image] 1= 10,   [image] 2= 3,   [image] 3= 9,   [image] 4= 2,   [image] 5= 0,   [image] 6= 10,   [image] 0= 5,   [image] 1= 6,   [image] 2= 4,   [image] 3= 4,   [image] 4= 3,   [image] 5= 8,   [image] 6= 0.

Figure 12.11(a) shows the arrays as they would appear after the initialization, and 12.11(b) gives their final disposition. The actual optimal tree is shown in Figure 12.12. It has a weighted path length of 188.

Since this algorithm requires O(n2 ) time and O(n2) storage, as n increases, it will run out of storage even before it runs out of time. The storage needed can be reduced by almost half, at the expense of a slight increase in time, by implementing the two-dimensional arrays as one-dimensional arrays using the technique of Chapter 2. Using one-dimensional arrays may enable problems to be solved that otherwise will abort because of insufficient storage. In any case, the algorithm is feasible only for moderate values of n.

(a) Initial Array Values

(b) Final Array Values

Figure 12.11 Arrays Used to Implement the Bottom-up Recursive Algorithm for Finding Optimal Binary Search Trees

Figure 12.12 The Optimal Binary Search Tree Found by Means of the Bottom-up Recursive Algorithm

In order to understand exactly how the algorithm creates the optimal tree, you should simulate the algorithm to reproduce the arrays of Figure 12.11.

12.5: Nearly Optimal Binary Search Trees

To date, no one knows how to construct the optimal binary search tree using less time or storage in the general case of nonzero   [image] 's and   [image] 's. For this reason it is important to develop algorithms that produce good, but not necessarily optimal, solutions with less time and storage being required. Algorithms have been found that construct nearly optimal binary search trees. The weighted path lengths of such trees cannot differ by more than a constant factor from the optimal value.

12.5.1 Greedy Binary Search Trees

One such algorithm produces greedy binary search trees. These trees are constructed by an algorithm using the greedy method. Unlike the Huffman algorithm, which is also based on the greedy method, they are not guaranteed to be optimal, but they are guaranteed to be nearly optimal. Other distinct, nearly optimal, tree constructions are found in Mehlhorn [1975, 1977] and in Horibe and Nemetz [1979]. A comparison of all these and other algorithms for the generation of nearly optimal binary search trees appears in Korsh [1982]. The proof that the following greedy method yields nearly optimal trees is in Korsh [1981].

In constructing a nearly optimal binary search tree by the greedy method, the programmer must ensure that the solution is a binary search tree. Looking at the optimal tree (Figure 12.12) found by the preceding section's algorithm will provide the idea behind the greedy tree construction. Think of the tree being generated by creating the subtrees shown in Figure 12.13, one after another, starting with the subtree with k4 at its root and ending with the final optimal tree. Each subtree has its value (the sum of its weights) associated with it.

Figure 12.13 Trees Created from Subtrees (Circled Numbers Are the Trees' Values)

Each subtree is formed by taking for its root a key that has not yet appeared and adjoining a left and right subtree. These subtrees may correspond to an extended node (with weight some   [image] i) or to an already produced subtree. The value of the subtree formed is given by the   [image] i of its key plus the value of its left and right subtrees. These subtrees must be selected in a constrained fashion, or the resultant tree will not be a binary search tree. These constraints will be delineated shortly, but for now, the main point is that the weighted path of the optimal tree is the sum of the values of each of the subtrees. The greedy trees are built by always selecting the next subtree to generate as the one with smallest value from among all those satisfying the constraints. This is exactly how the Huffman construction works, except there are no constraints on the choice of subtrees since the resultant tree need not be a binary search tree.

12.5.2 Greedy Construction

To start the development, we use the   [image] 's and   [image] 's of Example 12.3 and array them as shown in Figure 12.14(a). In an inorder traversal of any binary search tree with the six keys, the weights assigned to the accessed nodes would occur in the order   [image] 0   [image] 1   [image] 1   [image] 2   [image] 2   [image] 3, and so on. This sequence looks like triples, as shown in Figure12.14(b). A value can be associated with each triple by summing up its weights. Thus the triples just shown have values 21, 13, 17, 9, 11, and 18 respectively. A greedy tree is constructed by always finding an allowed triple with minimal value and generating its corresponding subtree next. The weighted path length of the resultant tree is the sum of the weights of the (n - 1) triples thus created. The greedy construction procedure is as follows. The allowable triples shown in the array have values 21, 13, 17, 9, 11, and 18. The triple with the lowest value (9) is

        2

     4    3

Combining the corresponding key and terminal nodes yields the subtree shown in Figure 12.14(c). Its assigned value is the triple value 9. Delete that triple from the array, and replace it by a terminal node value of 9, as shown in Figure 12.14(d).

Now repeat this process. The next minimal value is 13, corresponding to the triple This gives the subtree shown in Figure 12.14(e). The condensed array is shown in Figure 12.14(f). Repeating the process finds the minimal triple whose value is 17. The corresponding subtree is Figure 12.14(g), and the new array is shown in Figure 12.14(h). Continuing in this fashion produces the arrays shown in Figure 12.14(i), (j), and (k).

        3

     6    4

               1       2         3       4       5        6

             10     3        9     2      0       10

        5        6       4      4       3      8      0

          0         1        2       3        4       5        6

(a) Original Array of   's and   's from Example 12.3

(b) A Sequence of Triples

(c) Subtree with Value 9

(d) First Condensed Array

(e) Subtree with Value 13

(f) Second Condensed Array

Figure 12.14 Construction of Greedy Binary Search Tree

(g) Subtree with Value 17

(h) Third Condensed Array

(i) Fourth Condensed Array

(j) Fifth Condensed Array

(k) Final Array

Figure 12.14

The greedy tree thus constructed by always taking a minimal triple is shown in Figure 12.15. Its weighted path length can be found by keeping an accumulated sum, as in the Huffman algorithm. In this case the greedy algorithm actually yields the optimal tree.

12.5.3 Implementation

Just as in the Huffman tree construction, if we are not careful we will end up with an O(n2) time implementation. Instead, we can use a heap similar to that for the Huffman construction to obtain an O(n 1g n) time requirement and O(n) storage. However, we can do even better.

Figure 12.15 Greedy Binary Search Tree Constructed as Illustrated in Figure 12.14

The preceding method leads to a sequence of n - 1 combined triples with values v1, v2, . . . , vn-1. In this sequence v1   [image]  v2   [image]  . . .    vn-1. That is, the triples are generated in order, from the smallest to the highest value. The improved algorithm, introduced now, also generates a sequence of n - 1 combined triples, but their values will not necessarily be the same as v1, . . . , v2, vn-1, nor will they necessarily be generated in order by size. The improved algorithm, however, will produce the greedy tree that would have been obtained had they been combined in order by size when it is unique. A unique tree exists when there is exactly one smallest triple at each step. In general, the resultant tree will be greedy, but it will not necessarily be the same greedy tree generated by v1, . . . , v2, vn-1, nor will it necessarily have the same weighted path length.

We start with a list-structure illustrated in Figure 12.16. Each record represents a potential triple to be combined in the construction of a greedy tree. Traverse the list starting at the left and search for a locally minimal triple -- that is, a triple whose value does not exceed the value of its successor triple. No triple to the left of this locally minimal triple can be combined before it in any greedy tree construction. In addition, any future combining of locally minimal triples to its right cannot cause the resultant tree to fail to be greedy. In other words, combining the minimal triple always allows a greedy tree to be constructed with the combined minimal triple as a subtree. Combine the locally minimal triple (say it is the ith), and obtain the new list-structure shown in Figure 12.16(b).

Here   [image] 'i-1 and   [image] 'i were both set to the value of the combined triple   [image] 'i-1 +   [image] i +   [image] 'i, the ith record was deleted, and the pointers changed as shown. The left-to-right traversal is continued, starting now at the (i - 2)th record, combining as before for each locally minimal triple encountered. Details involving the ends of the list-structure can be handled by special checking or by introducing artificial records with large weights at the ends. This results in the construction of a greedy tree. This procedure is reminiscent of the evaluation of an infix expression of Chapter 4. That algorithm is greedy in the sense that we look for the highest local priority operator to evaluate first in a left-to-right scan, evaluate it, and continue from that point.

(a) Starting List Structure

(b) List Structure after Combining the ith Locally Minimal Triple

(c) Starting List Structure for the Weights of Example 12.3

(d) Result of Combining the Triple 6 3 4

(e) Result of Combining 4 2 3

(f) Result of Combining 5 10 13

(g) Result of Combining 9 0 8

(h) Result of Combining 17 10 0

(i) Result of Combining 28 9 27

Figure 12.16 Implementation of the Greedy Algorithm

Applying this improved algorithm to Example 12.3, the starting list-structure is as shown in Figure 12.16(c). The first locally minimal triple found is 6 3 4. The list-structure resulting from combining it is shown in Figure 12.16(d). Starting again at the first record, 4 2 3 is the next minimal triple, yielding the list shown in Figure 12.16(e). Starting again, the first record here is found to be the locally minimal triple to obtain the list shown in Figure 12.16(f).

Starting again at the next first record, 9 0 8, the next locally minimal triple, gives the list shown in Figure 12.16(g). Once again, we start at the first record. The next locally minimal triple is 17 10 0. Combining it results in Figure 12.16(h). Finally the triple 28 9 27 is combined to obtain the result shown in Figure 12.16(i).

We have constructed the same tree as before (in Figures 12.14 and 12.15). Again, this need not always occur. However, even when it does, as here, the order in which the subtrees were formed need not be the same. We generated subtrees with values 13, 9, 28, 17, 27, and finally 64, rather than in order from the smallest to largest as in the other method. Both trees in this example have the same weighted path length of 188.

This implementation takes time and storage only O(n)! This was made possible by the new method, coupled with the proper choice of data structure.

12.5.4 Alphabetic Codes

Consider the twenty-seven characters with relative frequencies given in Section 12.3 as weights. Order them as   [image]  < A < B < C <   [image]    [image]    [image]  < Z. Suppose we want to construct an extended binary search tree with these characters associated with the terminal nodes. We take the weights given there as the   [image] 's of this problem and take the   [image] 's to be zero. The extended binary tree will then generate what is called an alphabetic code. The optimal binary search tree will have weighted path length 5,200, and the greedy binary search tree will have weighted path length 5,307. Recall that the Huffman tree had weighted path length 5,124. Requiring use of a binary search tree rather than a binary tree results in an increase relative to 5,124 of only (5,200 - 5,124)/5,124 or 1.5 percent. The greedy construction yields an increase relative to 5,200 of (5,307 - 5,200)/5,200 or 2 percent.

12.6: Conclusion

Binary trees are useful data structures for encoding and decoding information. The Huffman algorithm generates optimal binary trees for this purpose and can do so efficiently. Binary search trees are useful data structures for storing information that is to be searched; they also produce good codes.

Greedy binary search trees, which are nearly optimal, can be generated very efficiently. The greedy method of algorithm design does not yield optimal solutions but may still yield very good solutions. Again it was demonstrated that careful selection of data structures can significantly change the time requirements of an algorithm.

Exercises

1.     a.     Using its definition, calculate the weighted path length of the following tree. The weight of a node appears at the node.

b.     Calculate the weighted path length of the tree using the other method of assigning a "value" to each node and adding up all the "values."

c.     Calculate the weighted path length of the same tree by adding the sum of all its weights to the weighted path length of its left subtree plus the weighted path of its right subtree.

2.     Write a process function to turn a postorder traversal into a function for determining the weighted path length of a binary tree T based on the method of Exercise 1(b).

3.     Determine the Huffman code (tree) for weights 1, 2, 3, 4, 5.

4.     Determine the weighted path length for your solution to Exercise 3.

5.     The Huffman tree for n weights will always have (n - 1) internal nodes and n terminal nodes. This is because it is a full binary tree. That is, every internal node has exactly 0 or exactly two successors. Prove, by induction on the total number of nodes of a full binary tree, that this is true.

6.     Write a nonrecursive function, huffman, with parameters n, weights, tree, and code. Weights is an array containing n weights, code is an array returned containing the n codes (sequences of 0's and 1's) assigned to each weight by the Huffman code, and tree points to the root of the corresponding Huffman tree upon return. Your function should take O(n 1g n) time.

7.     Write a recursive version of the function of Exercise 6.

8.     Simulate the proof (by induction on n) that the Huffman tree yields a binary tree with minimal weighted path length for n weights for Exercise 3.

9.     Write a function decode with parameters n, alpha, and tree. Alpha is a sequence of n 0's and 1's, and tree points to a Huffman tree for the twenty-seven characters of Section 12.3 (determined by their weights). Decode is to decode the sequence of n 0's and 1's into the corresponding characters. The sequence represents a coded version of a message composed of the characters.

10.     Suppose you know the Huffman tree for the twenty-seven characters of Section 12.3 (determined by their weights). Write a function encode to encode a message composed of characters into the Huffman code. To do this you might consider using the following data structures:

a.     The Huffman tree.

b.     An array with twenty-seven records kept in sorted order by character, each record containing a character and its corresponding Huffman code.

c.     A hash table of records, each record consisting of a character and its Huffman code; the key is the character.

Discuss the relative advantages and disadvantages for each of these in the function encode.

11.     Write a function encode to encode a message in a Huffman code.

12.     Let   [image] 0 = 10,   [image] 1 = 6,   [image] 2 = 8,   [image] 3 = 4,   [image] 4 = 2,   [image] 1 = 2,   [image] 2 = 5,   [image] 3 = 7,   [image] 4 = 6. Determine the optimal binary search tree by generating all possible binary search trees storing the four keys with weights 2, 5, 7, and 6, and calculating their weighted path lengths.

13.     The recursive approach of Section 12.4.1 is based on the idea that an optimal binary search tree must have an optimal left and an optimal right subtree. Can you use this idea to show that some of the trees you considered in Exercise 12 could have been ignored in searching for the optimal tree?

14.     Simulate the bottom-up algorithm of Section 12.4.1 to reproduce the arrays of Figure12.11.

15.     Why do we want to find nearly optimal binary search trees rather than optimal binary search trees for large numbers of keys?

16.     Find the greedy binary search tree for the weights of Exercise 12 using the definition of a greedy tree. Determine the relative difference between its weighted path length and the optimal value.

17.     Find the greedy binary search tree for the weights of Exercise 12 using the implementation of the greedy tree construction. Is its weighted path length the same as the greedy tree of Exercise 16?

18.     Write a nonrecursive function greedy to implement the greedy tree construction of Section 12.5.3.

19.     Give a recursive definition of a greedy binary search tree.

20.     Write a recursive function greedy to implement the greedy tree construction of Section 12.5.3.

21.     Another method of constructing a nearly optimal binary search tree is to select the key to be placed at the root to be a key that minimizes the difference between the weights of its left and right subtrees. Having obtained the root, apply this procedure recursively to construct its left and right subtrees. This is a top-down construction, as opposed to the greedy tree construction, which is bottom-up. What tree does it generate for the weights of Exercise 12 and for the example of Section 12.5.2?

22.     Write a recursive function for the construction method of Exercise 21. How much time does your function require?

23.     Can you find a way to implement the procedure of Exercise 21 in O(n) time, where n is the number of internal keys to be stored?

24.     What will a greedy tree look like if the n   [image] 's are zero and the n + 1   [image] 's are given by the first n + 1 Fibonacci numbers? Give two answers--one when the greedy tree definition is used, and one when the O(n) implementation is used.

Suggested Assignments

1.     a.     Write programs to find and output optimal greedy trees and their weighted path lengths.

b.     Explain why the greedy tree implementation with the list structure is linear in time. (Hint: When the next locally minimal triple is to be found, only list-structure records not previously accessed will be considered, as well as two records that were previously accessed.)

2.     a.     Write a function to read a file composed of the twenty-six letters of the alphabet, plus the blank character, the comma, and the period; produce an output file that is the encoded version of the input file. The Huffman code is to be used, with weights based on the number of times each character appears in the input text.

b.     This is the same as (a) except that the "words" of the input text are taken as the objects to be encoded, and their number of occurrences is the basis for the code. Use any reasonable definition of a word. For example, take commas, periods, blanks, and any sequence of other characters to be words.

Note: For both parts, use a function getnexttoken. In part (a) it must return the next character in the input or EOF, while in part (b) it must return the next word or EOF. Also, there must be a table in which the tokens (characters or words) are stored. This table must be searched for a token, inserted into, and its entries updated. Your function must treat the table with these operations as a data abstraction, so the solutions to both parts should look the same except that the definitions of certain functions should change from one solution to the other.

c.     Run your program for part (b) with the table implemented as

i.     a binary search tree

ii.     a hash table using double hashing

iii.     a table composed of two parts--an optimal binary search tree storing the 25 most frequently used English words (take the   [image] 's to be zero) and a binary search storing any other words that occur in the input. The optimal binary search tree is to be built in advance of the input.

d.     Write a function to read a text file consisting of the twenty-six letters of the alphabet, plus the blank and period characters, and produce an output file consisting of the text in Huffman code. Use the heap implementation in generating the Huffman tree.

Go to Chapter 13      Return to Table of Contents


Data Structures Using C: CHAPTER 13: SOME POINTERS ON STORAGE MANAGEME

 Return to Table of Contents

CHAPTER 13: SOME POINTERS ON STORAGE MANAGEMENT

Applies the basic data structures to the management of dynamic memory

Presents alternative ways to structure the collection of unused memory locations

two-way circular lists

the buddy system

indexing methods

Discusses problems that arise when structures share storage

Explains fundamental techniques for reclaiming unused storage

Illustrates data abstraction and functional modularization in the development of three stackless binary tree traversals

Calls attention to the problems of

garbage generation

dangling references

13.1: The Need for Storage Management

In languages such as C, arrays are static data structures. They do not change size or shape during the execution of a program. Chains, however, do change size, though they don't change shape. They are dynamic. List-structures, more complex lists, and binary and general trees can change both size and shape and thus are also dynamic data structures. The basic distinction between static and dynamic data structures relates to whether a fixed amount of memory is allocated in advance of program execution. Static structures can have a fixed amount of memory allocated for their sole use prior to program execution. For dynamic structures, the amount of memory required is not known in advance and thus cannot be allocated in this way.

In FORTRAN and COBOL, the total amount of storage needed by the variables of a program is known by the compiler and can be set aside before program execution. This means that a fixed amount of storage is allocated to variables for the entire execution of the program. This is not done in C, which is more versatile. It uses both static and dynamic storage allocation. Some storage is allocated for the entire execution of the program, while other storage is allocated when needed and is reused when no longer needed. This requires storage management during program execution. Dynamic structures such as lists and trees need even more complex storage management during program execution.

Consider a program that maintains a data base of stock portfolios for the clients of a brokerage house. Suppose each client's stocks are kept in a binary search tree ordered by stock name, with each node of the tree represented as a record containing information about the stock. It is not possible to predict in advance how much storage is needed, since how many stocks each client will buy is not known. Experience may show that all clients' stocks will never number more than 2,000, but the distribution of these stocks among the clients varies throughout the year. Thus it will be necessary to have storage available for 2,000 records. At a given moment this storage will be distributed among the clients' binary search trees. Whenever a stock is sold for one client, its storage must be deleted from that client's tree. It may be needed later to store the record of a stock just bought for another client. This allocation and reclaiming of record storage during program execution requires dynamic storage management. Some languages, such as C, provide needed storage management. Others do not, so the programmer must provide it.

This chapter discusses the fundamentals of storage management techniques. It offers insight into how programming languages manage storage and provides a better appreciation of the pitfalls that lie below the surface features of a language. Such insight is especially important when the programmer employs complex data structures that share storage.

You have already seen in Chapter 3 how the simple case of fixed-length records with no sharing might be handled. Problems arise when records have variable lengths and when data structures share memory. The remedies for these problems involve the creation and processing of data structures tailored to these needs. Sophisticated traversal algorithms have been developed to conserve storage. They are introduced here for three reasons. First, they are essential for the implementation of those data structures. Second, they are interesting in themselves. Finally, they serve as a good example of the use of data abstraction.

Although storage management is viewed here from the perspective of a single program's needs, these same techniques are employed by operating systems to deal with the allocation and reclamation of storage for a group of active programs.

In some problems, an upper bound on the problem size is not known in advance. One way to deal with this situation is to pick a reasonably large size and declare static data structures of this size. If the problem's memory requirement exceeds this size, an error message may be printed. When the number of available memory elements is fixed, there will generally be some problem size for which programs fail. However, it is possible that the program uses several data structures. During execution, when one data structure is large the others may be small, so that the total amount of storage required for all the structures may actually be available at all times. With static data structures, where memory is allocated and remains dedicated to a particular structure, storage must be allocated to accommodate the worst case. The worst case calls for the sum of the maximum storages needed for each data structure of the program. If, instead, the available memory can be used for the data structure that currently requires it and reclaimed for later use when that structure no longer needs it, then larger problems can be solved. This point is illustrated in the following examples.

Example 13.1

The lists array used to store successor records for the topological sort of Chapter 11 allows all successor lists to share storage. No storage is dedicated to any individual list. As long as the total number of successors on all lists does not exceed the length of lists, the program will execute correctly. This contrasts with the alternate collection of n arrays discussed in Section 11.6. With the latter approach, storage is dedicated to each list; although twice as many total successors may be accommodated, no list can contain more than its dedicated amount.

Example 13.2

Suppose five stacks are each implemented, as in Chapter 4, using static array structures, each with a maximum size of 5,000 elements. This requires 25,000 total memory locations. However, if the number of entries in all the stacks at any one time never exceeds 10,000, proper sharing of storage would allow a reduction of up to 15,000 in the amount of storage needed.

Example 13.3

A program can store, in linked representation, a number of binary trees that must be traversed using stacks. If the trees must be stored and traversed simultaneously, it is possible that available storage for the records of the tree nodes and for the entries of the stacks will not be sufficient for the worst case. However, if the available storage can be shared, so that it is used for tree records or for stack entries when needed, then the program might execute correctly.

In Example 13.1, the records sharing storage all have the same size. In Example 13.2, the five stacks can all store records of the same size, or they can store records of different lengths. Example 13.3 needs shared storage for records, which may or may not be of the same length for each binary tree, and for stack entries. Normally the record lengths would be significantly greater than that of the stack entries. One fundamental distinction between storage management techniques involves whether or not the storage allocated will all be the same size. When dealing with the fixed-size case, it is convenient to use the techniques for storage allocation and storage reclamation, discussed in Chapter 3, which involve a list of available space. When dealing with more general data structures or with variable-size records, additional complications arise.

13.2: The Heap or Dynamic Memory

As one or more programs execute, they may require storage for the creation of new records, which typically are to be inserted into dynamic data structures. How can a fixed amount of storage that is made available for the creation of these records be managed efficiently?

The records, or more generally, the components of dynamic data structures, are referred to as nodes. The region of memory consisting of contiguous elements that is dedicated to the storage of these nodes is called the heap or dynamic memory. We will use the term heap in the further discussion. In the heap all nodes need not be the same length, and nodes may appear as part of more than one data structure. Associated with each node is a length field, indicating the size of the node (that is, the number of memory elements it uses). If all nodes were the same size, there would be no need to store size information within each node. All dynamic data structures reside in the heap. At all times, the heap may contain three kinds of nodes:

Nodes that are in use because they appear in dynamic data structures

Nodes that are not in use and appear in no dynamic data structure but are on the current list of available space

Nodes that are not in use because they are in no dynamic data structure but are not on the list of available space

Nodes on the list of available space are said to be known. Nodes that are available but not on the list of available space are called garbage. All heads (names) of dynamic data structures, which point to their first records, are outside the heap and are kept track of in some way. This means that at all times the program knows what dynamic data structures are in existence, since their names are known.

Example 13.4

Suppose av, 1, and t are, respectively, the heads of the available space list, a list, and a binary tree. A heap containing these dynamic data structures may be visualized as in Figure 13.1. Each node has been labeled to indicate its current status:

1--in use since the node appears on l or t

2--not in use but on the available space list

3--not in use and not on the available space list, hence garbage

Storage allocation now involves three tasks:

1. Decide which node on the list of available space will be made available when a request for creation of a node is made.

2. Delete that node from the list of available space.

3. Set a pointer value to point to the node.

An avail-like function, discussed in Chapter 3, will perform these three tasks.

13.3: The List of Available Space for Variable-Length Records

When dealing with variable-size records, which is the usual situation, storage management is complex because of fragmentation. The fragmentation problem is as follows. When all records being allocated dynamically are the same size, then as long as unused storage finds its way to the list of available space, it can be reused. However, it is possible with variable-size records that storage for a large record is needed but is not immediately in existence. Suppose each available block of consecutive elements of storage is kept on the list of available space as an individual record, with a field of each record used to store its length. It is possible for records of length 10, 15, 3, 20, and 45 to appear on the list when a record of length 65 must be created, but no individual record is large enough to be allocated for it. However, a total of 93 elements is unused. This is the external fragmentation problem. Enough storage is unused to satisfy the need, but no record currently on the list of available space is, by itself, large enough to be used. The problem may be resolved by invoking a compaction procedure, which moves the field values in used nodes to contiguous elements of memory at one end of the heap. All unused elements then also become contiguous and can represent a node large enough that the needed storage can be taken.

Figure 13.1 A Heap Containing the Available Space List (AV), a List (L), and a Binary Tree (T) and Nodes Labeled 1, 2, or 3 to Show Status

Figure 13.2 The Heap of Example 13.4 (Figure13.1) after Compaction

Example 13.5

Suppose the compaction procedure is applied to the heap of Example 13.4, which was illustrated in Figure 13.1. The result may be visualized as in Figure 13.2.

Compaction, like garbage collection, is a time-consuming process that the programmer or computer system should only invoke infrequently. To overcome this, it is possible to organize the list of available space in an attempt to minimize fragmentation and thus the need for compaction. However, in limiting fragmentation the program should not spend too much computer time determining which node to allocate each time a request is made. Organizing the list of available space and using a reasonable policy to select the node to be allocated helps to reduce fragmentation. However, additional time is required to search the list to select the node to be allocated and to place a reclaimed node in proper position on the list to maintain its organization.

Fragmentation can also be reduced by creating larger nodes, by adding the freed storage to storage adjacent to it that is already free. This process is called coalescing. These policies and ways of organizing must be addressed by designers of compilers or operating systems and by programmers when they create storage management schemes for their own use.

Recall that the more structured or organized the data, the faster the access to required information. At the same time a price must be paid to maintain the organization. The primary considerations for a storage management scheme are the time to satisfy a request for a new node, the time to insert a node on the list of available space, the storage utilization, and the likelihood of satisfying a request. How well these characteristics turn out for a particular storage management scheme depends on the sequence of requests and returns. Mathematical analysis is generally not possible, and simulation must be used to evaluate different schemes. To provide some idea of the trade-offs and possibilities, three fundamental organizations for the available space list and appropriate allocation and reclamation policies are presented next.

13.3.1 Two-Way Circular Lists

The simplest organization for the list of available space is a two-way circular list, with nodes kept in arbitrary order, in order by size, or in order by actual memory address. Two-way lists are convenient for making insertions and deletions. They allow the list to be traversed starting from any list record, and in either direction. A request for a node of size l then requires a traversal of the list, starting from some initial node.

Two simple policies for selection of the list node to be deleted to satisfy a request are first-fit and best-fit. First-fit selects the first node encountered whose size is sufficient. Best-fit does a complete traversal and selects that node of smallest size that is sufficient to satisfy the request. Of course, if an exact fit is found, the traversal need not continue. For both policies, any leftover storage is retained as a node on the list.

When a node is returned to the list, its neighbors (in terms of storage addresses) are coalesced with it if free, and the resultant node properly inserted on the list. All nodes require tag and size fields, with the list nodes also needing backward and forward link fields. The tag field distinguishes nodes in use from those not in use, depending on which of two values it contains.

Allowing the starting node to circulate around the list, always being set to the successor of the deleted (or truncated) node, typically improves performance.This circulation provides time for the smaller nodes left behind it to coalesce so that, by the next time around, larger nodes have been formed.

Intuitively, spending more time fitting the size of the selected node to the size of the requested node should cut down on fragmentation effects. Surprisingly, this need not always happen, but it is the normal situation.

13.3.2 The Buddy System

Another organization, called the buddy system, provides faster request and return time responses than two-way circular lists. The buddy system requires the heap to be of length 2m for some integer m, occupying addresses 0 to 2m - 1. Over time, the heap is split into nodes of varying lengths. These lengths are constrained to be powers of 2, and nodes are kept on separate lists, each list containing nodes of a specific length. An array a of length m contains the heads of these lists, so a[k] points to the first node on the list corresponding to length 2k. Initially, a[m] points to the first address of the heap; all other array entries are null, since all other lists are empty.

To satisfy a request for a node of length n, k is set to [image] lg n [image] . If a[k] is not null, a node is deleted from its list to satisfy the request. Otherwise, the next nonnull list is found. Its nodes will be of length 2r, where r > k. A node is deleted from that list and split into two nodes, each of length 2r-1. That node created by the split with the greater address is placed on the appropriate list, while the other is used to satisfy the request if r - 1 = k. Otherwise it is split and processed in the same way. Eventually a node of length k will be generated to satisfy the request.

During this iterated splitting process, each time a node is split, it is split into two special nodes called buddies. Buddies bear a unique relation to each other. Only buddies are allowed to coalesce. Although this can result in unused nodes being contiguous but not coalescing because they are not buddies, it is a rare occurrence. This restriction allows very fast coalescing of buddies, as will be shown.

When a node is to be returned to the list of available space (a misnomer now), its buddy is first checked to see if it is not in use and, if it is not, the buddies are coalesced. This node is, in turn, coalesced with its buddy when possible, and so on, until no further coalescing is possible. This coalescing process is the reverse of the splitting process that created the buddies in the first place.

At this point you must be wondering how a node's buddy is found. The scheme ensures that it may be done quickly because a node of size 2k with starting address

has as its buddy a node of size 2k with starting address

In general, given the address of a node of size 2k to be returned, its (k + 1)th bit (addresses are expressed in binary notation) is changed from 1 to 0 or 0 to 1, and the result is the address of its buddy. A tag field is used to indicate whether or not a node is in use.

Example 13.6

How does the buddy system respond when m is 6 and a sequence of requests is made for nodes of size 8, 16, 4, and 6?

(a) After a Sequence of Requests Is Made for Nodes of Size 8, 16, 4, 6

(b) After the Node of Length 16 Is Returned

Figure 13.3 Sequence of Memory Configurations for Example 13.6

The sequence of memory configurations is as shown in Figure 13.3(a). lf the node of length 16 is now returned, the configuration becomes as shown in Figure13.3(b)

A[4] is now the head of a list with two nodes, each of length 16. The released node of length 16 starting at address 16 (10000 in binary) has the node of length 16 starting at address 0 (00000 in binary) as its buddy. Since it is not free, it cannot coalesce with it. Neither can it coalesce with the node of length 4 starting at address 12, since that node is not its buddy.

The distribution of nodes can be mirrored by a binary tree. This tree is a kind of derivation tree for nodes currently in existence and expands and contracts as nodes are split and coalesced. The tree shown in Figure 13.4 reflects the configuration for Example 13.6 before the node of length 16 is returned.

Buddies appear as successors of the same parent. Thus [12, 15] and [16, 31], although contiguous in memory, are not buddies and therefore may not be coalesced.

With the buddy system, lists are never actually traversed to find storage to allocate, although time is spent finding the proper list from which to take the storage. The result is quick response to requests and returns. However, in contrast to two-way lists, this scheme produces significant internal fragmentation. Internal fragmentation occurs when allocated nodes are larger than requested, resulting in wasted storage residing within a node. This is one of the drawbacks of the buddy system. Studies have been done on similar schemes that allow a greater variety of node sizes to appear. They also tend to be fast and to reduce storage loss due to internal fragmentation.

Figure 13.4 Binary Tree Reflecting the Memory Configuration for Example 13.6 Just before Return of Node of Length 16

13.3.3 Indexing Methods

At this point each of the two approaches, circular lists and buddy systems, seems to solve half the problem. Circular lists using the best-fit method utilize storage very efficiently, causing little internal fragmentation. Buddy systems produce significant internal fragmentation but offer fast allocation by narrowly focusing the search for a proper size node. The question is whether it is possible to achieve good storage utilization and allocation time by combining their features. As you would guess, it is. It can be done by using the appropriate structures. The means to do it is called the indexing approach. The indexing approach breaks up the list of available space into collections of two-way circular lists. Each of these circular lists contains nodes of just one size. Each node on the circular list represents a block of consecutive memory locations available for the storage of a record of the size of the node. Assuming that requests for nodes of sizes (say, 1 to 4) are very frequent, an array a contains pointers to each of the four lists corresponding to these sizes. The last array entry contains a pointer to an AVL tree. The nodes of this tree are circular lists, each list corresponding to nodes of specific size that are requested much less frequently. Thus a[2] contains a pointer to a circular list containing four blocks or nodes, each of size 2. In the AVL tree pointed to by a[5], the root node is a circular list containing three blocks of size 10. The tree is ordered on the basis of node sizes. A typical configuration might be as shown in Figure 13.5.

In this way, a specific request for a node of length less than 5 can be quickly satisfied if the exact matching list is not empty. If it is empty, the next nonempty greater-size list can be used, with one of its nodes truncated to satisfy the request. Any leftover is inserted on the appropriate list. For instance, with the situation of Figure 13.5, a request for a node of size 3 will be satisfied by deleting a node of size 4 from the list pointed to by a[4]. This node will be split into a node of size 3 to satisfy the request, and the leftover node of size 1 inserted on the list pointed to by a[1]. In the infrequent case when the requested node size is 5 or greater, the AVL tree must be searched similarly. It is clear that this scheme can cause internal fragmentation, but the search times should be fast on the average. In fact the indexing approach is a compromise. It is not as fast as the buddy system but faster than simple circular lists. Likewise, the fragmentation is less than with the buddy system but more than with simple circular lists.

Figure 13.5 An Indexing Array Containing Pointers to Circular Lists and an AVL Tree Storing Circular Lists

Thus choosing the appropriate scheme of storage management is a difficult task. It requires insight into the kind of request and return distribution involved in a given application. In the worst case all schemes may fail miserably. Intuitively, good performance should result when the distribution of node sizes on the list of available space reflects the distribution of requests and returns. In turn, this distribution should be reflected in the distribution of node sizes actually in use. This is analogous to the situation mentioned in Chapter 12 for the distribution of letters in printer's boxes. The distribution of letter requests is reflected in actual text, and the printer's ordering policy for letters should have caused the contents of the letter boxes to reflect this distribution.

The three techniques considered in this section, and other techniques, are compared extensively in Knuth [1973a] and Standish [1980]. See also Tremblay and Sorenson [1984].

13.4: Shared Storage

When components of data structures are identical and require significant amounts of storage, it is only natural to attempt to avoid component replication. This means that identical components will all share the same storage, so references to each will always be to the same place. As an example, consider Figure 13.6. The two list-structures ls1 and ls2 share a common sublist. As a result, p1, p3, and p4 point to the shared sublist.

Figure 13.6 A Shared Sublist

As usual, what we gain in one area we pay for in another. Here, storage is saved, but the price is paid in more complex processing. In particular, suppose the sublist pointed to by p3 is to be deleted from ls2 by setting p3 to null. Since p1 and p4 still point to it, it is not valid to return its records' storage to the list of available space, as would be the case without sharing. Deciding when and how to reclaim deleted records or sublists by insertion on the list of available space requires more sophisticated reclamation algorithms than those of Chapter 3.

13.5: Storage Reclamation Techniques

Saying that a node or component of a data structure is to be reclaimed for use means that the storage used for its records is to become known to the avail function. That is, the storage no longer needed by its records is to be inserted on the list of available space. There are two basic methods of reclaiming released nodes, garbage collection and reference counters. Other methods are essentially combinations of these. These methods are the two extremes with respect to the time at which storage is reclaimed. Garbage collection does not make storage available for reuse until it is needed, putting off reclamation as long as possible. Reference counters make storage available for reuse as soon as it actually becomes reclaimable.

13.5.1 Garbage Collection

The garbage collection method of reclamation is used when the available space list is empty or when the amount of storage that is known to avail becomes less than some given amount. Until that time, whenever a sublist is to be deleted from a data structure, the deletion is accomplished by updating appropriate pointers, but the storage vacated by the sublist's deletion is not made known. Initially, all records are assumed to be unmarked. The garbage collection routine works by traversing each data structure that is in use and "marking" each one of its records as being in use.

To accomplish this properly, the garbage collector program must know the names of all data structures in use. We have assumed that these are accessible through variables outside the heap. After marking, every record of the heap that is not marked will be made known (that is, placed on the available space list). Every record that is marked will be reset to unmarked so that the garbage collector will work properly on its next call. The marking phase of the garbage collector works by traversing all data structures in use. Thus marking routines generally take execution time proportional to the number of records in use and are therefore slow. Similarly, the collection routine, which inserts the unmarked records on the list of available space, takes time proportional to the heap size.

Example 13.7

Apply garbage collection to the heap of Example 13.4.

In the marking phase, the garbage collection would traverse l and t, marking all their nodes. The collection phase would then traverse the heap and collect all unmarked nodes on the available space list. Assuming the garbage collector recognizes when two unmarked nodes are adjacent and combines them into a larger node, the resultant heap would appear as shown in Figure 13.7. A better approach might be to garbage collect and compact together, thus producing an available list consisting of one large block, reducing fragmentation.

Figure 13.7 Heap from Example 13.4 after Garbage Collection

13.5.2 The Reference Counter Method

The reference counter method assumes that the first record of every sublist in use has information giving the number of pointers that point to it. This number is called the reference count of the sublist. The pointers can be stored in the name variables or in sublist fields of complex records. A "smart" header record may be used for this purpose. It is assumed here that only sublists may be shared. Each time a new pointer refers to a sublist, the reference count of the sublist must be increased by 1. Each time a pointer that refers to a sublist is changed so that it no longer points to that sublist, the reference count of the sublist must be decreased by 1. After a decrease that leaves the reference count at zero, the storage for records of the sublist may be made known. Again, a data structure traversal may be used to determine which storage elements are to be made known. No traversal algorithm has been given in this text for data structures that allow sublist sharing. Such algorithms differ from the list-structure traversal algorithm because of the possibility of looping or revisiting records. As an exercise you should modify the list-structure traversals of Chapter 6 to work for this more general case. The idea is to mark each accessed record and make sure that marked records are not revisited.

Deletion of a sublist whose reference count has become zero is similar to the traversal of successor lists for an object that has been removed from the bag in the topological sort algorithm. The sublist must be traversed and, in addition to the storage for all records being inserted on the list of available space, each sublist pointed to by a record must have its reference count decremented by 1.

Example 13.8

Associate reference counts with the first record of each of the sublists of Figure 13.6. The list-structures with associated reference counts would appear as shown in Figure 13.8.

Suppose p1 is set to null. The reference count of its sublist would be reduced to 2. Since it has not gone to zero, no storage is available for reclamation. If p6 is then set to null, the reference count of its sublist would be reduced to 0. Its storage could be reclaimed. This would require that p3's sublist have its reference count reduced by 1. The result would be as shown in Figure 13.8(b).

A problem can occur with the reference counter method whenever loops appear, as demonstrated in the following example.

Example 13.9

Consider the loop in ls1 of Figure 13.9. What happens when p1, p2, and p3 of Figure 13.6 no longer reference the sublist? Then this sublist has no variables referencing it. It cannot be accessed, since we do not know where it is. It is serving no useful purpose, yet its storage was not reclaimed, since the reference count of records 1 and 2 never went to zero. This problem is analogous to the existence of loops in the input for a topological sort. Such unused storage, or garbage, can accumulate and clog the memory.

It is possible to share individual records as well as sublists. In this case, garbage collection will still work, as will the reference counter method. Now, however, the individual records (not just the sublists) must have reference counters associated with them, requiring additional storage and updating overhead. Loops may cause the same problems for shared records as for shared sublists .

(a) Initial List Structures with Reference Counts

(b) Final List Structures with Reference Counts

Figure 13.8 List-structures of Figure 13.6 with Associated Reference Counts

Figure 13.9 A Lost Loop

13.5.3 Overview of Storage Reclamation

There is a spectrum of reclamation techniques, but the two basic underlying approaches are garbage collection and reference counters. Garbage collection represents one extreme. Nodes that are no longer needed in a dynamic data structure are deleted from the structure, thus becoming garbage. They are not returned at that time to the list of available space, even though they are then reusable. Reference counters, at the other extreme, return such unused nodes to the list of available space as soon as possible, when their reference counters become zero. These methods of reclamation are done implicitly by the programming language in conjunction with the operating system.

In addition, programmer control allows the programmer to use commands signaling that nodes are no longer needed by a dynamic data structure. The nodes may then be returned to the list of available space directly, left for garbage collection, or, if reference counters are used, their reference counts will be decremented and tested for possible return to the list of available space. In C the standard library function free can be used for such signaling.

In the design of a specific reclamation system, a number of trade-offs are possible. For example, reference counters require additional storage and additional time to increment, decrement, or test when insertions and deletions occur. However, they do not always work. The reason is that garbage can accumulate when loops are present in the dynamic data structures (as in Example 13.9). If the amount of garbage generated in this way is significant, it can limit the size of the problem that can be solved. Still, reference counters allow the reclamation process to be spread over time. Garbage collection, in contrast, occasionally requires a relatively large block of time. Garbage collection is a more complex technique, involving more sophisticated algorithms. Recall that when garbage collection is invoked, even though many available nodes may exist, they are not known. It is not possible simply to look at a node of the heap to determine if it is in use.

To summarize, garbage collection works in two phases. The first phase, or marking phase, requires the traversal of all dynamic data structures in existence. Recall that the heads of such structures are assumed to be known. Initially, whether in use or not, every node has a field that is marked "unused." As each node of a dynamic data structure is accessed in the traversal, it is marked "used." When the marking phase is completed, the heap then consists of nodes that are marked "unused" and nodes that are marked "used." Consequently, it is now possible to look at a node and determine its status. The second phase, or collection phase, involves a traversal through the nodes of the heap. Each node marked "unused" is inserted on the list of available space. Each node marked "used" is merely marked "unused" so that the garbage collector will work correctly the next time it is invoked.

The marking algorithms known are quite elegant. The traversal of the dynamic data structures cannot simply use a stack, since no nodes are known in which to implement the stack. Instead, these algorithms allow the traversals to be accomplished by using tag fields, usually one bit, associated with each node. It is even possible to do the traversals with no tag fields, with a fixed amount of additional storage required. Of course, if we had an infinite amount of storage available, no reclamation would be necessary. Storage management would be trivial! The next section features three traversal algorithms for use in phase I of the garbage collector, or in other applications where storage is at a premium. See Pratt [1975] for more on storage management. The algorithms to be discussed next and generalizations are also given in Standish [1980].

13.6: Stackless Traversals

Garbage collection routines require the traversal of dynamic data structures that are more general than binary trees, such as list-structures and more complex lists. Algorithms for their traversal are generalizations of those we consider. For increased clarity, we will restrict our discussion to binary tree traversals and present three techniques that do not use extra storage for a stack.

These algorithms may be written assuming a particular binary tree implementation, such as arrays or pointer variables. Instead, to emphasize data abstraction and functional modularity, the following function, which is independent of the tree implementation, has been chosen. The function is a preorder traversal saving the path from the root on the stack.

preorder(pt)

/* Preorder traverses the binary tree. */

binarytreepointer *pt;

{

binarytreepointer null,p,lptr,rptr;

binarytreepointer setnull(),left(),right(),nextsubtree();

stack s;

null = setnull();

>Comment

p = *pt;

setstack(&s);

>Comment

while(p != null)

{

>Comment

process(pt,p);

>Comment

lptr = left(p);

>Comment

if(lptr != null)

{

>Comment

push(p,&s);

>Comment

p = lptr;

}

>Comment

else

{

>Comment

rptr = right(p);

>Comment

if(rptr != null)

{

>Comment

push(p,&s);

>Comment

p = rptr;

}

>Comment

else

>Comment

p = nextsubtree(&s,&p);

}

}

}

For each traversal, the basic concept is described. Then the appropriate implementation for the routines of the procedure is described.

Consider the binary tree of Figure 13.10, and suppose node.p has just been processed during the traversal using the preorder procedure. The stack will contain pointers p1, p2, p3, p4, p5, p6, and p7. At this point, the traversal of the left subtree of node.p4 has been completed. The purpose of the nextsubtree(&s,&p) function in the preorder traversal algorithms is to find node.p4, to allow the traversal to pick up at its right subtree. In general, whenever the stack is not empty, a null p value signifies the completion of the traversal of the left subtree of some node. The traversal must then pick up at that node's right subtree. There may be more than one such node; it is the most recent that must be found. The stack allows this node, and its right subtree, to be found conveniently. Nextsubtree(&s,&p) is called after the terminal node pointed to by p has been processed. It finds the nonnull right subtree that must be traversed next, returns a pointer to it, and updates the stack to reflect the path from that subtree's root to the root of the tree. If no such nonnull right subtree exists, then it returns a null value. Each of the tree traversals we now consider actually uses stack information, but it is, in effect, stored in the tree, eliminating the requirement for additional stack storage. The preorder procedure treats the tree as a data abstraction, so it is independent of the tree implementation.

Figure 13.10 A Binary Tree T

13.6.1 Threaded Trees

A threaded tree implementation of the preorder traversal procedure is shown in Figure 13.11. It resulted from the replacement of the null left and right pointer field values of each node t by a pointer to the predecessor and to the successor, respectively, of that node in an inorder traversal of t. These pointers are called threads. The "leftmost" node of t does not have a predecessor, and the "rightmost" node of t does not have a successor. Their left and right null pointers, respectively, are left unchanged and are not considered threads. We assume that a field is associated with each node to indicate whether the node contains threads in its left link field, in its right link field, in both, or in neither.

To preorder traverse threaded trees, proceed as before, but whenever a node with threads is reached, follow its right pointer. This pointer leads directly to the node whose right subtree must then be preorder traversed. Notice, for example, that the right thread of node.p points to node.p4. The traversal is complete when a null right thread is encountered. Threaded trees may also be easily inorder and postorder traversed.

Procedure preorder may be used for the traversal of a threaded tree, with stack declared to be of type binarytreepointer, by implementing its routines as follows:

      Routine                            Task

--------------------------------------------------------------------

setnull( )          determined by the implementation (array or pointer

                  variables)

setstack(&s)       sets s to null

push(p,&s)        sets s to p

left(p)            returns a copy of leftptr.p if it is not a thread, and a

                  null value otherwise

right(p)            similar to left(p)

nextsubtree(&s,&p)  set s to p

                while (rightptr.s is a thread),

                 set s to rightptr.s

                return a copy of rightptr.s

13.6.2 Link Inversion

Now consider the tree of Figure 13.12. It represents the binary tree t of Figure 13.10 during a link-inversion traversal after node.p has been processed. Notice that a path exists in the tree from predp back to the root. Each node has a tag field associated with it, and all tag values are initially zero. Preadp always points to the predecessor of p in t, and every node on the path from predp to the root has its right pointer field pointing to its predecessor if its tag field is 1, and its left pointer field pointing to its predecessor if its tag field is 0. The root of the subtree whose left subtree traversal is complete after the processing of p may be found by following the backward links from predp until a tag value of 0 is encountered. As the path back is followed, a tag of 1 at a predecessor node means we are coming from the node's right subtree. A tag of 0 means we are coming from the left subtree. Again, for node.p this leads to node.p4.

Figure 13.11 A Threaded Implementation of the Binary Tree T

Using the backward links with appropriate tag values to preserve the path to the root is the essential concept of a link-inversion traversal. During the traversal, the creation of appropriate links must be handled. For example, as the path to node.p4 is traversed, after p has been processed, the nodes with tag values of 1 must have their correct right pointer values restored. Then when node.p4 is found (tag = 0), its correct left pointer value must be restored, its tag field set to 1, its right pointer field set to its predecessor, and predp and p correctly updated before node.p4's right subtree is traversed. The traversal is complete when the backtracking leads to a null predp value. Figure 13.13 shows the state of the tree just before node.p4's right subtree is traversed. Node.p must now be processed, then its left pointer field must be set to its predecessor, predp updated to p, and p updated to the left pointer value of node.p. The traversal then continues with the left subtree of p. Initially, p must be set to t and predp to null.

Figure 13.12 A Link-inverted Binary Tree T

Procedure preorder may be used for the link-inversion traversal if stack is declared to be of type binarytreepointer, and its routines are implemented as follows.

    Routine                            Task

-----------------------------------------------------------------------------

setnull( )          determined by the implementation

setstack(&s)       set s to null

push(p,&s)          if leftptr.p is not null, then

                      set leftptr.p to s and s to p

                  else

                      set tag(p) to 1, set rightptr.p to s and s to p

left(p)            returns a copy of leftptr.p

right(p)           returns a copy of rightptr.p

nextsubtree(&s,&p)  set found to false

                while((not found) and (s not null))

                    if tag(s) = 1, then

                        set pred to s

                        set tag(s) to 0, s to rightptr.s,

                        rightptr.pred to p, and p to pred

                    else

                        set pred to leftptr.s and leftptr.s to p

                        if rightptr.s null, then

                            set found to true, tag(s) to 1, return

                            rightptr.s, and set rightptr.s to

                            pred

                        else

                            set p to s, set s to pred

                if s = null, then

                     return null.

In this implementation s plays the role of predp.

In a stack traversal, the storage required for the stack is proportional to the depth of the traversed tree. In the threaded-tree and link-inversion traversals, storage is not required for the stack but is required for the tag fields. This storage may actually be available, but unused, because of the implementation of the tree records. In this case, using it for the tag fields costs nothing. Otherwise, the additional storage is proportional to the number of nodes in the traversed tree. For the stack traversal, the constant of proportionality is given by the length of the pointer fields, while in the threaded-tree and link-inversion traversals it is given by the length of the tag fields.

The threaded-tree traversal is simpler than the link-inversion traversal and does not disturb the tree itself during the traversal. This allows more than one program to access the tree concurrently. However, the insertion and deletion of nodes in a threaded tree is more complex and time-consuming. Inorder and postorder traversals may also be done using link inversion.

Figure 13.13 New State before Subtree Pointed to by P Is Traversed in Binary Tree T

13.6.3 Robson Traversal

It is remarkable that an algorithm has been found which does not require a stack or even tag fields. This is the Robson traversal, the final stackless traversal we consider. Again, we give a preorder version, although it may be used for inorder and postorder traversals as well.

Consider the tree of Figure 13.14. In the link-inversion traversal, for every descent to the left from node.p (that is, every traversal of the left subtree of node.p), the left pointer field of node.p was set to point to node.p's predecessor. For every descent to the right (to traverse the right subtree of node.p), the leftpointer of node.p was restored to its original value, the tag of node.p was set to 1, and the right pointer field of node.p was set to point to node.p's predecessor. Consequently, having completed the traversal of the left subtree of some node q, it was possible to follow the backward pointing pointers to get to q. Q was the first node encountered with a tag value of 0, on the path back.

The Robson traversal does not have tag fields available for this purpose. Instead, a stack is kept in the tree itself, which will allow node q to be found. Q is the root of the most recent subtree, whose nonnull left subtree has already been traversed and whose nonnull right subtree is in the process of being traversed. A variable top is used, and kept updated, to point to the current node q. A variable stack is used, and kept updated, to point to the next most recent subtree whose nonnull left subtree has already been traversed and whose nonnull right subtree is in the process of being traversed. Variables p and predp are used, as in link inversion, to point to the node currently being processed and to its predecessor, respectively. Each stack entry, except the current top entry, is kept in the "rightmost" terminal node of one of the subtrees whose nonnull left subtree has been completely traversed, and whose nonnull right subtree is in the process of being traversed. In Figure 13.14, when node.p is being processed, there are four subtrees whose left subtrees have already been traversed and whose right subtrees are currently being traversed. Their roots are labeled 1 to 4 in Figure 13.14. Nodes 1, 3, and 4 have nonnull left subtrees. Each of these nodes has its left pointer field pointing to its predecessor and its right pointer pointing to the root of its left subtree. Node 2, which has a null left subtree, has its original null value, and its right pointer field is pointing to its predecessor. Nodes 1, 3, and 4 are thus the roots that must have stack entries. Top points to the most recent, node 1. Stack points to the stack entry corresponding to the next most recent root, node 3. This stack entry is kept in the rightmost terminal node of node 1's left subtree. This rightmost node has its left pointer field pointing to the node that contains the next stack entry, and its right pointer field pointing to its corresponding root (node 3 in this case). All stack entries (except the first, pointed to by top) have this same format. In Figure 13.14, the last stack entry points to node 4.

Figure 13.14 Robson Traversal Configuration of the Binary Tree T

Notice that a path back to the root always exists from node.predp. In Figure 13.14, after node.p is processed, the traversal of node 1's right subtree has been completed. This also completes the traversal of the right subtrees of nodes 2 and 3 and the left subtree of node 3's predecessor. The Robson traversal now proceeds by following the backward path, restoring appropriate left or right subtrees along the way, until the predecessor of node 3 is encountered. At this point, its right subtree must be traversed. Figure 13.15 shows the situation just before that traversal . During the traversal of the backward path, if node.predp has a null right subtree, or top=predp, then we are coming from the left subtree of node.predp, otherwise from the right subtree. Thus the stack entries allow node q to be found even though no tag fields are used.

Figure 13.15 Robson Traversal before P Is Processed in the Binary Tree T

During the backtracking, the stack is popped, and top is updated as necesssary. Top and stack are initialized to null, predp to null, and p to t. An additional variable avail is used and updated to point to a terminal node every time one is encountered, so that its pointer fields are available if a new stack entry must be created. In this case, the available node must retain a pointer to node.top in its right pointer field and a pointer to the current entry to which stack points in its left pointer field; stack must be updated to point to the new entry; and top must be updated to point to the new top root. At all times, the path from predp back to the root is stored in the tree, as well as the stack of entries pointing to roots of subtrees (with nonnull left subtrees), whose left subtree traversals have completed, and whose right subtrees are being traversed. Each node along the path from predp to the root is in one of the following states:

If the node's left subtree is being traversed, then

its left pointer is pointing to its predecessor, and its right pointer is undisturbed.

If the node's right subtree is being traversed, then

if the node has a null left subtree, then

its right pointer is pointing to its predecessor

else

its left pointer is pointing to its predecessor, and its right pointer is pointing to

the root of the node's left subtree.

The following routines are to be used with the preorder traversal so that it implements the Robson traversal. Stack is declared to be of the same type as binarytreepointer, and s plays the role of predp.

   Routine                                   Task

-----------------------------------------------------------------------------

setnull( )           determined by the implementation

setstack(&s)        sets top, stack, and s to null

push(p,&s)         if leftptr.p is not null then

                     set leftptr.p to s and s to p

                 else

                     set rightptr.p to s and s to p

left(p)             returns a copy of leftptr.p

right(p)            returns a copy of rightptr.p

nextsubtree(&s, &p)  set avail to p

                     set found to false

                     while((not found) and (s not null))

                         if top = s, then

                             save stack in hold, pop the stack by setting

                             top to rightptr.stack, stack to leftptr.

                             stack, and leftptr.hold and

                             rightptr.hold to null. Save leftptr.s in

                             pred, restore leftptr.s by setting it to

                             rightptr.s, restore rightptr.s by setting it

                             to p, set p to s, and s to pred

                         else

                             if leftptr.s = null, then

                                 save rightptr.s in pred, restore

                                 rightptr.s by setting it to p, set p to s,

                                 and s to pred

                             else

                                 if rightptr.s null, then

                                     push an entry for s onto the stack by

                                     setting leftptr.avail to stack,

                                     rightptr.avail to top, stack to

                                     avail, and top to s. Return the value

                                     of rightptr.s, set rightptr.s to p,

                                     and set found to true

                                 else

                                     save leftptr.s in pred, restore

                                     leftptr.s by setting it to p, set p to s,

                                     and s to pred.

                     if s = null, then

                          return null.

13.7: Pitfalls: Garbage Generation and Dangling References

This chapter has shown how a list of available space can be maintained to keep track of storage known to be currently unused. Storage may be allocated dynamically, during program execution, by deleting appropriate elements from this list when needed.

The need and means for storage reclamation have also been discussed. Garbage collection and reference counter techniques (or some combination) may be used for this purpose. When a node of a binary tree, a list, or other dynamic data structure is not needed, the programmer frequently knows at what point this occurs. Some high-level languages allow commands to be used by the programmer to signal this occurrence. For example, C provides the function free. Like malloc (and other allocation functions), its parameter will be a pointer type variable. When p points to a node that the programmer knows is no longer needed, free(p) may be invoked. This signals that the storage pointed to by p may be relinquished and reclaimed for use for another dynamically allocated data structure. Exactly how free uses this information is determined by the compiler.

High-level languages are usually designed with particular storage management schemes in mind. However, the language specifications, or standards, do not normally specify that these schemes must be used. It is possible for a compiler to use garbage collection or reference counters, to leave the reclamation entirely to the programmer, or to use some combination. If necessary programmers may even write their own programs to manage storage. We will not discuss these problems further except to illustrate two potential dangers.

Consider the following disaster segment as an illustration of these dangers.

(1) p = q;

(2) r = malloc ( sizeof(binarytreerecord));

(3) r -> leftptr = tree;

(4) free(tree);

(5) process(&s);

(6) s -> leftptr = lefttree;

Suppose p, q, r, s, lefttree, and tree are all pointer variables pointing to nodes of the same type. Before the segment is executed, the situation might be as shown in Figure 13.16(a). Statement (1), when executed, results in the situation sketched in Figure 13.16(b).

The tree, pointed to by p originally, now has no pointer pointing to it. Hence there is no way the program can access any of its nodes. Also, the storage allocated to this tree, while no longer in use, has not been returned to the list of available space. Unless the storage management technique uses garbage collection or reference counters, that storage is lost. It cannot be accessed and will never be returned to the list of available space. It truly has become garbage. If free(p) had been invoked before statement (1), this might have been avoided, depending on whether free recognizes that the other nodes of the tree may also be returned. If free merely returns the storage for the node to which p points, this could have been avoided only by the programmer traversing the tree pointed to by p and "freeing" each of its nodes.

Suppose, instead, that free uses reference counters and, recognizing that the tree pointed to by p has no other references, returns its storage to the list of available space. Now statement (2) is executed. The result is shown in Figure 13.16(c).

After statement (4) is executed, assuming that free simply returns the storage allocated to the node pointed to by tree to the list of available space, we get Figure 13.16(e). However, the node pointed to by r [image] leftptr is now on the list of available space. R [image] leftptr is called a dangling reference because it points to a node that is on the list of available space. When process is invoked in statement (5), it is possible that it invokes malloc, so storage is allocated to s. Then we have the situation shown in Figure 13.16(f).

(a) Initial Situation

(b) Result of Statement (1)

(c) Result of Statement (2)

(d) Result of Statement (3)

(e) Result of Statement (4)

(f) Result of Statement (5)

(g) Result of Statement (6)

Figure 13.16 Simulation of the Disaster Segment

Statement (6) produces Figure 13.16(g). The binary tree that r now points to has been changed, because of the dangling reference. The programmer, unaware that this has occurred, may find the program's output incorrect and be unable to determine what went wrong. This is especially difficult to discover, since the next time the program executes, process may not need to invoke malloc. Even if it does, the storage allocated by malloc to s might not cause this problem. Still, the programmer may not realize that an error has occurred.

Of the two problems, garbage generation and dangling reference generation, the latter is more serious. Garbage generation may cause longer execution time, or, at worst, program failure. Dangling references, however, may cause serious errors undetected by the programmer.

13.8: Conclusion

Dynamic data structures require dynamic storage allocation and reclamation. This may be accomplished by the programmer or may be done implicitly by a high-level language. It is important to understand the fundamentals of storage management because these techniques have significant impact on the behavior of programs.The basic idea is to keep a pool of memory elements that may be used to store components of dynamic data structures when needed.

Allocated storage may be returned to the pool when no longer needed. In this way, it may be used and reused. This contrasts sharply with static allocation, in which storage is dedicated for the use of static data structures. It cannot then be reclaimed for other uses, even when not needed for the static data structure. As a result, dynamic allocation makes it possible to solve larger problems that might otherwise be storage-limited.

Garbage collection and reference counters are two basic techniques for implementing storage management. Combinations of these techniques may also be designed. Explicit programmer control is also possible. Potential pitfalls of these techniques are garbage generation, dangling references, and fragmentation.

High-level language may take most of the burden for storage management from the programmer. The concept of pointers or pointer variables underlies the use of these facilities, and complex algorithms are required for their implementation.

Exercises

1. a. Consider the Suggested Assignments of Chapter 11. Suppose only 200 memory elements are available for lists to store successors. Given an example of a problem with fewer than twenty objects for which the implementation of Assignment 1 will work but the implementation of Assignment 2 will fail. Assume count and succ have length 20, the lists array of (1) has length 200, and the lists array of (2) is 20 [image] 10.

b. Can you find such an example for which Assignment 2 works but Assignment 1 fails?

2. a. Consider two implementations. First, a total of 50 memory elements in an array are to be dynamically allocated and reclaimed so that a binary tree (in linked representation) as well as a stack may be stored in the array. Second, an array of length 35 is to hold the binary tree, and another array of length 15 is to implement the stack. Each array is dynamically managed. The binary tree is to be preorder traversed using the stack. Give an example of a binary tree, with fewer than 35 nodes, that may be successfully traversed with the first implementation but not the second.

b. Suppose that after a node is accessed and processed, its storage may be reclaimed. Give an example of a binary tree with fewer than 35 nodes that may be successfully traversed with the first implementation but not the second.

3. In Exercise 1, the lists array of (1) functioned as a heap. Why was its storage management so simple?

4. In Exercise 2, the arrays functioned as heaps. Why was the first implementation able to do traversals that the second could not do?

5. In the first implementation of Exercise 2(b), suppose that, after a node is accessed and processed, it is simply put onto the available space list.

a. Is the head of the tree a dangling reference?

b. Are fixed-size or variable-size nodes being dealt with?

c. After your answer to Exercise 2(b) is traversed, what will be garbage, and what will be on the list of available space?

6. This is the same as Exercise 5, except that, after a node is accessed and processed, it is not put onto the available space list. After the binary tree has been traversed, the head of the tree is set to null.

7. Why is compaction not necessary with fixed-size nodes?

8. Suppose Example 13.3 deals with three binary trees. Each node has an integer valued info, leftptr, and rightptr field. The stack needs elements that are also integer valued. An integer array of length l is used as a heap in which nodes of the binary trees and the stack are stored. Assume one additional field is used for each node to indicate its length. Design a storage allocation scheme and write an appropriate avail function.

9. a. Suppose in Exercise 8 that when a node of a binary tree is deleted, its predecessor node will be changed to point to the left successor of the deleted node. Write a function delete that carries out this deletion and also returns the storage used by the deleted node to a list of available space.

b. How would the delete function change if reference counters were used as a reclamation scheme?

10. Write avail for the two-way circular list implementation for

a. Randomly ordered list entries.

b. List entries in order by size

c. List entries in order by address

11. Write avail for the buddy system implementation.

12. Write a reclaim function for the buddy system.

13. Why will garbage collection take a longer time when many nodes are in use and a shorter time when few nodes are in use?

14. What are the disadvantages of reference counters compared to garbage collection?

15. Write a function to traverse a list-structure with shared sublists to reclaim storage of a sublist whose reference count has gone to zero.

16. Write a function to insert a node in a threaded binary tree t. The node is to be inserted as the node accessed next in an inorder traversal of t after node.pred is accessed.

17. Write a function to delete a node in a threaded binary tree t. The deleted node is to be node.p.

18. Write a function to return a pointer to the predecessor of node.p in a threaded binary tree t.

19. Write a function to read information into the info fields of a binary tree t, implemented using pointer variables. The info fields will contain character strings of up to twenty characters. The input is given in the order that the info fields would be accessed in a preorder traversal of t.

20. How will your solution to Exercise 19 change if the binary tree t is implemented as a threaded tree?

21. Implement the transformation of Section 7.6.1 using a list implementation for the stack. The binary tree is implemented using pointer variables as a threaded tree. Assume the general tree is given by a two-dimensional array of size 50 [image] 50. If the [i,j] entry of the array is 1, then node j is a successor of node i; if the [i,j] entry is 0, then node j is not a successor of node i.

22. This is the same as Exercise 21 except that the general tree will be given by a list of successors for each node. These lists should be pointed to by pointer variables, one for each node of the general tree. The binary tree should be a link-inversion tree.

23. Suppose input, representing a general tree, is given as a sequence of pairs. A pair i, j implies that node j is a successor of node i. A sentinel pair has its i, j values equal. For example, the general tree of Figure 7.13 might be input as B,F A,C D,J I,K I,L C,H B,E I,M A,D D,I A,B B,G X,X. Write a function that will create the successor lists of Exercise 22.

24. Simulate the preorder traversal implementation for the link-inversion traversal of the binary tree of Figure 13.10.

25. Simulate the preorder traversal implementation for the Robson traversal of the binary tree of Figure 13.10.

26. Why will there always be enough storage in the binary tree, during a Robson traversal, for the stack of root nodes whose left subtrees are nonnull, whose traversal has been completed, and whose right subtrees are being traversed?

27. What purpose does the stack of Exercise 26 serve in the Robson traversal?

28. Write a function to do an inorder traversal of a threaded binary tree t.

29. Write a function to do a link-inversion traversal of a binary tree t assuming the tree is implemented using pointer variables.

30. Write a function to do a Robson traversal of a binary tree t assuming the tree is implemented using pointer variables.

Suggested Assignments

1. Write and run a function to copy a list-structure with shared sublists. Use pointer variables.

2. Explain how storage may be reclaimed if all nodes of the copied list-structure that have nonnull sublists are deleted.

3. Why would we not write recursives versions of the three stackless traversals?

4. Simulate the different implementations for the list of available space of Section 13.3, choosing a number of request and return distributions.


Data Structures Using C: PREFACE

 Return to Table of Contents

PREFACE

One must learn by doing the thing, for though you can think you know it you have no certainty until you try it.

Sophocles

Data Structures, Algorithms, and Program Style Using C is intended for use in a second programming course, such as CS2 of the ACM curriculum, as well as in a Data Structures course. It will also be invaluable for computer professionals who wish to develop applications using the C programming language. All programs appear in C, although the concepts and methods presented are independent of the language used. The book emphasizes current techniques for the creation and testing of programs and stresses the importance of writing programs that are general enough to be readily adapted for use in the solution of complex problems. Data abstraction and modularity are stressed throughout. Complex concepts are presented in a concise, informal way; theory is coupled with concrete examples.

The book is written to provide a significant amount of flexibility in the order in which the material is covered, as illustrated in the accompanying chart. For a second programming/introductory data structures class, the fundamental concepts of structured programming and data structures are presented in Chapters 1 through 7. These chapters, combined with selected topics from Chapters 8 and 9, provide a challenging course. Chapters 8 through 13 deal with the application of the concepts to more complex problems and also introduce some advanced data structures. Where data structures are taught after the second programming course, selected material from Chapters 1 through 8 can serve as a quick review of the basics, followed by an in-depth study of Chapters 9 through 13. Care has been taken throughout to include sets of exercises that are graduated by degree of difficulty and complexity. Also included are suggested assignments that incorporate the essential concepts of each chapter.

Just as knowing the grammar of a natural language does not ensure quality writing, knowing the grammar of a programming language does not ensure quality programs. Learning to program well entails utilizing principles of good style and requires skill in the use of a wide variety of tools. It is an advanced task, beyond simply learning the grammar of a language. A master mechanic, upon opening the hood of an automobile, does not see a jumble of wires and metal, but rather essential components linked together and aligned in a systematic fashion. A knowledgeable programmer must learn not only to write programs but to read them in the same fashion as a master mechanic looks at an automobile engine. He or she has to be able to see the major components and linkages of the program. Program structure can enhance or obscure a program's intelligibility.

This book explains and demonstrates the tools needed to craft good programs and to solve substantial problems. These tools, although few in number, must be used with care and forethought. The tools include the basic paradigms for problem solving, for structuring programs, and for structuring data. Their proper use involves data abstraction, modularization, and the appropriate choice of data structures. Because the book emphasizes the entire program development process, readers will learn how to write well-designed programs and how to recognize them. They will, at the same time, develop insight into program analysis and learn how to analyze programs in order to determine their correctness and efficiency.

* Heapsort requires Chapter 7.

Many individuals contributed to this book. Special thanks are due to Sharon Doherty, Mary Helen Fein, Giorgio Ingargiola, and Paul LaFollette for their help, as well as to the students who labored through the early versions of the manuscript. The staff of PWS-KENT Publishing Company, including Bob Prior and Elise Kaiser, all provided valuable support and guidance. We would also like to thank David Hoyt for help during the production process. It is also a delight to acknowledge the painstaking and helpful work of the reviewers: James Calhoun,Western Illinois University; Michael Driscoll, DePaul University; Terry Hamberger, York College of Pennsylvania; Marek Holynski, Boston University; Sam C. Hsu, Florida Atlantic University; Paul W. Ross, Millersville University of Pennsylvania; Shai Simonson, University of Illinois at Chicago; John B. Tappen, University of Southern Colorado; and Therril Valentine, McNeese State University. Finally, and certainly most important, we deeply appreciate and thank our wives Nina and Judy and our children, who suffered from us and with us, avoided us and supported us, during the time this book was being written.

James F. Korsh

Leonard J. Garrett

Go to Chapter 1 Return to Table of Contents


Data Structures Using C: Table of Contents

Data Structures, Algorithms and Program Style Using C

by James F. Korsh and Leonard J. Garrett

PREFACE

CHAPTER 1: PROGRAMMING STRUCTURE

CHAPTER 2: RECORDS, ARRAYS, AND POINTERS

CHAPTER 3: LISTS

CHAPTER 4: INTRODUCTION TO RECURSION, STACKS, AND QUEUES

CHAPTER 5: PACKAGING DATA ABSTRACTIONS

CHAPTER 6: MORE COMPLEX LISTS

CHAPTER 7: TREES

CHAPTER 8: INTRODUCTION TO SEARCHING AND SORTING

CHAPTER 9: MORE SEARCHING: INSERTION AND DELETION

CHAPTER 10: FILES

CHAPTER 11: TOPOLOGICAL SORTING: AN ARCHETYPAL SOLUTION

CHAPTER 12: HUFFMAN CODING AND OPTIMAL AND NEARLY OPTIMAL BINARY SEARCH TREES

CHAPTER 13: SOME POINTERS ON STORAGE MANAGEMENT

BIBLIOGRAPHY


Structures, Algorithm Analysis: CHAPTER 1: INTRODUCTION

 Return to Table of Contents

CHAPTER 1: INTRODUCTION

In this chapter, we discuss the aims and goals of this text and briefly review programming concepts and discrete mathematics. We will

See that how a program performs for reasonably large input is just as important as its performance on moderate amounts of input.

Review good programming style.

Summarize the basic mathematical background needed for the rest of the book.

Briefly review recursion.

1.1. What's the Book About?

Suppose you have a group of n numbers and would like to determine the kth largest. This is known as the selection problem. Most students who have had a programming course or two would have no difficulty writing a program to solve this problem. There are quite a few "obvious" solutions.

One way to solve this problem would be to read the n numbers into an array, sort the array in decreasing order by some simple algorithm such as bubblesort, and then return the element in position k.

A somewhat better algorithm might be to read the first k elements into an array and sort them (in decreasing order). Next, each remaining element is read one by one. As a new element arrives, it is ignored if it is smaller than the kth element in the array. Otherwise, it is placed in its correct spot in the array, bumping one element out of the array. When the algorithm ends, the element in the kth position is returned as the answer.

Both algorithms are simple to code, and you are encouraged to do so. The natural questions, then, are which algorithm is better and, more importantly, is either algorithm good enough? A simulation using a random file of 1 million elements and k = 500,000 will show that neither algorithm finishes in a reasonable amount of time--each requires several days of computer processing to terminate (albeit eventually with a correct answer). An alternative method, discussed in Chapter 7, gives a solution in about a second. Thus, although our proposed algorithms work, they cannot be considered good algorithms, because they are entirely impractical for input sizes that a third algorithm can handle in a reasonable amount of time.

A second problem is to solve a popular word puzzle. The input consists of a two-dimensional array of letters and a list of words. The object is to find the words in the puzzle. These words may be horizontal, vertical, or diagonal in any direction. As an example, the puzzle shown in Figure 1.1 contains the words this, two, fat, and that. The word this begins at row 1, column 1 (1,1) and extends to (1, 4); two goes from (1, 1) to (3, 1); fat goes from (4, 1) to (2, 3); and that goes from (4, 4) to (1, 1).

Again, there are at least two straightforward algorithms that solve the problem. For each word in the word list, we check each ordered triple (row, column, orientation) for the presence of the word. This amounts to lots of nested for loops but is basically straightforward.

Alternatively, for each ordered quadruple (row, column, orientation, number of characters) that doesn't run off an end of the puzzle, we can test whether the word indicated is in the word list. Again, this amounts to lots of nested for loops. It is possible to save some time if the maximum number of characters in any word is known.

It is relatively easy to code up either solution and solve many of the real-life puzzles commonly published in magazines. These typically have 16 rows, 16 columns, and 40 or so words. Suppose, however, we consider the variation where only the puzzle board is given and the word list is essentially an English dictionary. Both of the solutions proposed require considerable time to solve this problem and therefore are not acceptable. However, it is possible, even with a large word list, to solve the problem in a matter of seconds.

An important concept is that, in many problems, writing a working program is not good enough. If the program is to be run on a large data set, then the running time becomes an issue. Throughout this book we will see how to estimate the running time of a program for large inputs and, more importantly, how to compare the running times of two programs without actually coding them. We will see techniques for drastically improving the speed of a program and for determining program bottlenecks. These techniques will enable us to find the section of the code on which to concentrate our optimization efforts.

   1  2  3  4

-------------

1  t  h  i  s

2  w  a  t  s

3  o  a  h  g

4  f  g  d  t

Figure 1.1 Sample word puzzle

1.2. Mathematics Review

This section lists some of the basic formulas you need to memorize or be able to derive and reviews basic proof techniques.

1.2.1. Exponents

  xa xb = xa+b

    xa

    --  = xa-b

    xb

  (xa)b = xab

xn + xn = 2xn    x2n

2n + 2n = 2n+1

1.2.2. Logarithms

In computer science, all logarithms are to base 2 unless specified otherwise.

DEFINITION:     xa = b if and only if logx b = a

Several convenient equalities follow from this definition.

THEOREM 1.1.

PROOF:

Let x = logc b, y = logc a, and z = loga b. Then, by the definition of logarithms, cx = b, cy = a, and az = b. Combining these three equalities yields (cy)z = cx = b. Therefore, x = yz, which implies z = x/y, proving the theorem.

THEOREM 1.2.

log ab = log a + log b

PROOF:

Let x = log a, y = log b, z = log ab. Then, assuming the default base of 2, 2x= a, 2y = b, 2z = ab. Combining the last three equalities yields 2x2y = 2z = ab. Therefore, x + y = z, which proves the theorem.

Some other useful formulas, which can all be derived in a similar manner, follow.

log a/b = log a - log b

log(ab) = b log a

log x < x for all x > 0

log 1 = 0,  log 2 = 1,  log 1,024 = 10,  log 1,048,576 = 20

1.2.3. Series

The easiest formulas to remember are

and the companion,

In the latter formula, if 0 < a < 1, then

and as n tends to   [image], the sum approaches 1/(1 -a). These are the "geometric series" formulas.

We can derive the last formula for   [image]  in the following manner. Let S be the sum. Then

S = 1 + a + a2 + a3 + a4 + a5 + . . .

Then

aS = a + a2 + a3 + a4 + a5 + . . .

If we subtract these two equations (which is permissible only for a convergent series), virtually all the terms on the right side cancel, leaving

S - aS = 1

which implies that

We can use this same technique to compute   [image], a sum that occurs frequently. We write

and multiply by 2, obtaining

Subtracting these two equations yields

Thus, S = 2.

Another type of common series in analysis is the arithmetic series. Any such series can be evaluated from the basic formula.

For instance, to find the sum 2 + 5 + 8 +. . . + (3k - 1), rewrite it as 3(1 + 2+ 3 +. . . + k) - (1 + 1 + 1 +. . . + 1), which is clearly 3k(k + 1)/2 - k. Another way to remember this is to add the first and last terms (total 3k + 1), the second and next to last terms (total 3k + 1), and so on. Since there are k/2 of these pairs, the total sum is k(3k + 1)/2, which is the same answer as before.

The next two formulas pop up now and then but are fairly infrequent.

When k = -1, the latter formula is not valid. We then need the following formula, which is used far more in computer science than in other mathematical disciplines. The numbers, HN, are known as the harmonic numbers, and the sum is known as a harmonic sum. The error in the following approximation tends to y   [image]  0.57721566, which is known as Euler's constant.

These two formulas are just general algebraic manipulations.

1.2.4. Modular Arithmetic

We say that a is congruent to b modulo n, written a   [image]  b(mod n), if n divides a - b. Intuitively, this means that the remainder is the same when either a or b is divided by n. Thus, 81   [image]  61   [image]  1(mod 10). As with equality, if a   [image]  b (mod n), then a + c   [image]  b + c(mod n) and a d   [image]  b d (mod n).

There are a lot of theorems that apply to modular arithmetic, some of which require extraordinary proofs in number theory. We will use modular arithmetic sparingly, and the preceding theorems will suffice.

1.2.5. The P Word

The two most common ways of proving statements in data structure analysis are proof by induction and proof by contradiction (and occasionally a proof by intimidation, by professors only). The best way of proving that a theorem is false is by exhibiting a counterexample.

Proof by Induction

A proof by induction has two standard parts. The first step is proving a base case, that is, establishing that a theorem is true for some small (usually degenerate) value(s); this step is almost always trivial. Next, an inductive hypothesis is assumed. Generally this means that the theorem is assumed to be true for all cases up to some limit k. Using this assumption, the theorem is then shown to be true for the next value, which is typically k + 1. This proves the theorem (as long as k is finite).

As an example, we prove that the Fibonacci numbers, F0 = 1, F1 = 1, F2 = 2, F3 = 3, F4 = 5, . . . , Fi = Fi-1 + Fi-2, satisfy Fi < (5/3)i, for i   [image]  1. (Some definitions have F0 = 0, which shifts the series.) To do this, we first verify that the theorem is true for the trivial cases. It is easy to verify that F1 = 1 < 5/3 and F2 = 2 <25/9; this proves the basis. We assume that the theorem is true for i = 1, 2, . . . , k; this is the inductive hypothesis. To prove the theorem, we need to show that Fk+1 < (5/3)k+1. We have

Fk + 1= Fk + Fk-1

by the definition, and we can use the inductive hypothesis on the right-hand side, obtaining

Fk+1 < (5/3)k + (5/3)k-1

< (3/5)(5/3)k+1 + (3/5)2(5/3)k+1

< (3/5)(5/3)k+1 + (9/25)(5/3)k+1

which simplifies to

Fk+1 < (3/5 + 9/25)(5/3)k+1

< (24/25)(5/3)k+1

< (5/3)k+1

proving the theorem.

As a second example, we establish the following theorem.

THEOREM 1.3.

PROOF:

The proof is by induction. For the basis, it is readily seen that the theorem is true when n = 1. For the inductive hypothesis, assume that the theorem is true for 1   [image]  k   [image]  n. We will establish that, under this assumption, the theorem is true for n + 1. We have

Applying the inductive hypothesis, we obtain

Thus,

proving the theorem.

Proof by Counterexample

The statement Fk   [image]  k2 is false. The easiest way to prove this is to compute F11 = 144 > 112.

Proof by Contradiction

Proof by contradiction proceeds by assuming that the theorem is false and showing that this assumption implies that some known property is false, and hence the original assumption was erroneous. A classic example is the proof that there is an infinite number of primes. To prove this, we assume that the theorem is false, so that there is some largest prime pk. Let p1, p2, . . . , pk be all the primes in order and consider

N = p1p2p3. . . pk + 1

Clearly, N is larger than pk, so by assumption N is not prime. However, none of p1, p2, . . . , pk divide N exactly, because there will always be a remainder of 1. This is a contradiction, because every number is either prime or a product of primes. Hence, the original assumption, that pk is the largest prime, is false, which implies that the theorem is true.

int

f( int x )

{

/*1*/       if ( x  = 0 )

/*2*/              return 0;

else

/*3*/              return( 2*f(x-1) + x*x );

}

Figure 1.2 A recursive function

1.3. A Brief Introduction to Recursion

Most mathematical functions that we are familiar with are described by a simple formula. For instance, we can convert temperatures from Fahrenheit to Celsius by applying the formula

C = 5(F - 32)/9

Given this formula, it is trivial to write a C function; with declarations and braces removed, the one-line formula translates to one line of C.

Mathematical functions are sometimes defined in a less standard form. As an example, we can define a function f, valid on nonnegative integers, that satisfies f(0) = 0 and f(x) = 2f(x - 1) + x2. From this definition we see that f(1) = 1, f(2) = 6, f(3) = 21, and f(4) = 58. A function that is defined in terms of itself is called recursive. C allows functions to be recursive.* It is important to remember that what C provides is merely an attempt to follow the recursive spirit. Not all mathematically recursive functions are efficiently (or correctly) implemented by C's simulation of recursion. The idea is that the recursive function f ought to be expressible in only a few lines, just like a non-recursive function. Figure 1.2 shows the recursive implementation of f.

*Using recursion for numerical calculations is usually a bad idea. We have done so to illustrate the basic points.

Lines 1 and 2 handle what is known as the base case, that is, the value for which the function is directly known without resorting to recursion. Just as declaring f(x) = 2 f(x - 1) + x2 is meaningless, mathematically, without including the fact that f (0) = 0, the recursive C function doesn't make sense without a base case. Line 3 makes the recursive call.

There are several important and possibly confusing points about recursion. A common question is: Isn't this just circular logic? The answer is that although we are defining a function in terms of itself, we are not defining a particular instance of the function in terms of itself. In other words, evaluating f(5) by computing f(5) would be circular. Evaluating f(5) by computing f(4) is not circular--unless, of course f(4) is evaluated by eventually computing f(5). The two most important issues are probably the how and why questions. In Chapter 3, the how and why issues are formally resolved. We will give an incomplete description here.

It turns out that recursive calls are handled no differently from any others. If f is called with the value of 4, then line 3 requires the computation of 2 * f(3) + 4 * 4. Thus, a call is made to compute f(3). This requires the computation of 2 * f(2) + 3 * 3. Therefore, another call is made to compute f(2). This means that 2 * f(1) + 2 * 2 must be evaluated. To do so, f(1) is computed as 2 * f(0) + 1 * 1. Now, f(0) must be evaluated. Since this is a base case, we know a priori that f(0) = 0. This enables the completion of the calculation for f(1), which is now seen to be 1. Then f(2), f(3), and finally f(4) can be determined. All the bookkeeping needed to keep track of pending function calls (those started but waiting for a recursive call to complete), along with their variables, is done by the computer automatically. An important point, however, is that recursive calls will keep on being made until a base case is reached. For instance, an attempt to evaluate f(-1) will result in calls to f(-2), f(-3), and so on. Since this will never get to a base case, the program won't be able to compute the answer (which is undefined anyway). Occasionally, a much more subtle error is made, which is exhibited in Figure 1.3. The error in the program in Figure 1.3 is that bad(1) is defined, by line 3, to be bad(1). Obviously, this doesn't give any clue as to what bad(1) actually is. The computer will thus repeatedly make calls to bad(1) in an attempt to resolve its values. Eventually, its bookkeeping system will run out of space, and the program will crash. Generally, we would say that this function doesn't work for one special case but is correct otherwise. This isn't true here, since bad(2) calls bad(1). Thus, bad(2) cannot be evaluated either. Furthermore, bad(3), bad(4), and bad(5) all make calls to bad(2). Since bad(2) is unevaluable, none of these values are either. In fact, this program doesn't work for any value of n, except 0. With recursive programs, there is no such thing as a "special case."

These considerations lead to the first two fundamental rules of recursion:

1.     Base cases. You must always have some base cases, which can be solved without recursion.

2.     Making progress. For the cases that are to be solved recursively, the recursive call must always be to a case that makes progress toward a base case.

Throughout this book, we will use recursion to solve problems. As an example of a nonmathematical use, consider a large dictionary. Words in dictionaries are defined in terms of other words. When we look up a word, we might not always understand the definition, so we might have to look up words in the definition. Likewise, we might not understand some of those, so we might have to continue this search for a while. As the dictionary is finite, eventually either we will come to a point where we understand all of the words in some definition (and thus understand that definition and retrace our path through the other definitions), or we will find that the definitions are circular and we are stuck, or that some word we need to understand a definition is not in the dictionary.

int

bad( unsigned int n )

{

/*2*/            return 0;

else

/*3*/            return( bad (n/3 + 1) + n - 1 );

}

Figure 1.3 A nonterminating recursive program

Our recursive strategy to understand words is as follows: If we know the meaning of a word, then we are done; otherwise, we look the word up in the dictionary. If we understand all the words in the definition, we are done; otherwise, we figure out what the definition means by recursively looking up the words we don't know. This procedure will terminate if the dictionary is well defined but can loop indefinitely if a word is either not defined or circularly defined.

Printing Out Numbers

Suppose we have a positive integer, n, that we wish to print out. Our routine will have the heading print_out(n). Assume that the only I/O routines available will take a single-digit number and output it to the terminal. We will call this routine print_digit; for example, print_digit(4) will output a 4 to the terminal.

Recursion provides a very clean solution to this problem. To print out 76234, we need to first print out 7623 and then print out 4. The second step is easily accomplished with the statement print_digit(n%10), but the first doesn't seem any simpler than the original problem. Indeed it is virtually the same problem, so we can solve it recursively with the statement print_out(n/10).

This tells us how to solve the general problem, but we still need to make sure that the program doesn't loop indefinitely. Since we haven't defined a base case yet, it is clear that we still have something to do. Our base case will be print_digit(n) if 0   [image]  n < 10. Now print_out(n) is defined for every positive number from 0 to 9, and larger numbers are defined in terms of a smaller positive number. Thus, there is no cycle. The entire procedure* is shown Figure 1.4.

*The term procedure refers to a function that returns void.

We have made no effort to do this efficiently. We could have avoided using the mod routine (which is very expensive) because n%10 = n -   [image] n/10  [image]  * 10.

Recursion and Induction

Let us prove (somewhat) rigorously that the recursive number-printing program works. To do so, we'll use a proof by induction.

THEOREM 1.4

The recursive number-printing algorithm is correct for n   [image]  0.

PROOF:

First, if n has one digit, then the program is trivially correct, since it merely makes a call to print_digit. Assume then that print_out works for all numbers of k or fewer digits. A number of k + 1 digits is expressed by its first k digits followed by its least significant digit. But the number formed by the first k digits is exactly   [image] n/10  [image] , which, by the indicated hypothesis is correctly printed, and the last digit is n mod10, so the program prints out any (k + 1)-digit number correctly. Thus, by induction, all numbers are correctly printed.

void

print_out( unsigned int n ) /* print nonnegative n */

{

if( n<10 )

print_digit( n );

else

{

print_out( n/10 );

print_digit( n%10 );

}

}

Figure 1.4 Recursive routine to print an integer

This proof probably seems a little strange in that it is virtually identical to the algorithm description. It illustrates that in designing a recursive program, all smaller instances of the same problem (which are on the path to a base case) may be assumed to work correctly. The recursive program needs only to combine solutions to smaller problems, which are "magically" obtained by recursion, into a solution for the current problem. The mathematical justification for this is proof by induction. This gives the third rule of recursion:

3.     Design rule. Assume that all the recursive calls work.

This rule is important because it means that when designing recursive programs, you generally don't need to know the details of the bookkeeping arrangements, and you don't have to try to trace through the myriad of recursive calls. Frequently, it is extremely difficult to track down the actual sequence of recursive calls. Of course, in many cases this is an indication of a good use of recursion, since the computer is being allowed to work out the complicated details.

The main problem with recursion is the hidden bookkeeping costs. Although these costs are almost always justifiable, because recursive programs not only simplify the algorithm design but also tend to give cleaner code, recursion should never be used as a substitute for a simple for loop. We'll discuss the overhead involved in recursion in more detail in Section 3.3.

When writing recursive routines, it is crucial to keep in mind the four basic rules of recursion:

1.     Base cases. You must always have some base cases, which can be solved without recursion.

2.     Making progress. For the cases that are to be solved recursively, the recursive call must always be to a case that makes progress toward a base case.

3.     Design rule. Assume that all the recursive calls work.

4.     Compound interest rule. Never duplicate work by solving the same instance of a problem in separate recursive calls.

The fourth rule, which will be justified (along with its nickname) in later sections, is the reason that it is generally a bad idea to use recursion to evaluate simple mathematical functions, such as the Fibonacci numbers. As long as you keep these rules in mind, recursive programming should be straightforward.

Summary

This chapter sets the stage for the rest of the book. The time taken by an algorithm confronted with large amounts of input will be an important criterion for deciding if it is a good algorithm. (Of course, correctness is most important.) Speed is relative. What is fast for one problem on one machine might be slow for another problem or a different machine. We will begin to address these issues in the next chapter and will use the mathematics discussed here to establish a formal model.

Exercises

1.1     Write a program to solve the selection problem. Let k = n/2. Draw a table showing the running time of your program for various values of n.

1.2     Write a program to solve the word puzzle problem.

1.3     Write a procedure to output an arbitrary real number (which might be negative) using only print_digit for I/O.

1.4     C allows statements of the form

#include filename

which reads filename and inserts its contents in place of the include statement. Include statements may be nested; in other words, the file filename may itself contain an include statement, but, obviously, a file can't include itself in any chain. Write a program that reads in a file and outputs the file as modified by the include statements.

1.5     Prove the following formulas:

a.     log x < x for all x > 0

b.     log(ab) = b log a

1.6     Evaluate the following sums:

1.7     Estimate

*1.8     What is 2100 (mod 5)?

1.9     Let Fi be the Fibonacci numbers as defined in Section 1.2. Prove the following:

**c.     Give a precise closed-form expression for Fn.

1.10     Prove the following formulas:

References

There are many good textbooks covering the mathematics reviewed in this chapter. A small subset is [1], [2], [3], [11], [13], and [14]. Reference [11] is specifically geared toward the analysis of algorithms. It is the first volume of a three-volume series that will be cited throughout this text. More advanced material is covered in [6].

Throughout this book we will assume a knowledge of C [10]. Occasionally, we add a feature where necessary for clarity. We also assume familiarity with pointers and recursion (the recursion summary in this chapter is meant to be a quick review). We will attempt to provide hints on their use where appropriate throughout the textbook. Readers not familiar with these should consult [4], [8], [12], or any good intermediate programming textbook.

General programming style is discussed in several books. Some of the classics are [5], [7], and [9].

1.     M. O. Albertson and J. P. Hutchinson, Discrete Mathematics with Algorithms, John Wiley & Sons, New York, 1988.

2.     Z. Bavel, Math Companion for Computer Science, Reston Publishing Company, Reston, Va., 1982.

3.     R. A. Brualdi, Introductory Combinatorics, North-Holland, New York, 1977.

4.     W. H. Burge, Recursive Programming Techniques, Addison-Wesley, Reading, Mass., 1975.

5.     E. W. Dijkstra, A Discipline of Programming, Prentice Hall, Englewood Cliffs, N.J., 1976.

6.     R. L. Graham, D. E. Knuth, and O. Patashnik, Concrete Mathematics, Addison-Wesley, Reading, Mass., 1989.

7.     D. Gries, The Science of Programming, Springer-Verlag, New York, 1981.

8.     P. Helman and R. Veroff, Walls and Mirrors: Intermediate Problem Solving and Data Structures, 2d ed., Benjamin Cummings Publishing, Menlo Park, Calif., 1988.

9.     B. W. Kernighan and P. J. Plauger, The Elements of Programming Style, 2d ed., McGraw- Hill, New York, 1978.

10.     B. W. Kernighan and D. M. Ritchie, The C Programming Language, 2d ed., Prentice Hall, Englewood Cliffs, N.J., 1988.

11.     D. E. Knuth, The Art of Computer Programming, Vol. 1: Fundamental Algorithms, 2d ed., Addison-Wesley, Reading, Mass., 1973.

12.     E. Roberts, Thinking Recursively, John Wiley & Sons, New York, 1986.

13.     F. S. Roberts, Applied Combinatorics, Prentice Hall, Englewood Cliffs, N.J., 1984.

14.     A. Tucker, Applied Combinatorics, 2d ed., John Wiley & Sons, New York, 1984.

Go to Chapter 2      Return to Table of Contents


Structures, Algorithm Analysis: CHAPTER 10: ALGORITHM DESIGN TECHNIQUE

 Return to Table of Contents

CHAPTER 10: ALGORITHM DESIGN TECHNIQUES

So far, we have been concerned with the efficient implementation of algorithms. We have seen that when an algorithm is given, the actual data structures need not be specified. It is up to the programmer to choose the approriate data structure in order to make the running time as small as possible.

In this chapter, we switch our attention from the implementation of algorithms to the design of algorithms. Most of the algorithms that we have seen so far are straightforward and simple. Chapter 9 contains some algorithms that are much more subtle, and some require an argument (in some cases lengthy) to show that they are indeed correct. In this chapter, we will focus on five of the common types of algorithms used to solve problems. For many problems, it is quite likely that at least one of these methods will work. Specifically, for each type of algorithm we will

See the general approach.

Look at several examples (the exercises at the end of the chapter provide many more examples).

Discuss, in general terms, the time and space complexity, where appropriate.

10.1. Greedy Algorithms

The first type of algorithm we will examine is the greedy algorithm. We have already seen three greedy algorithms in Chapter 9: Dijkstra's, Prim's, and Kruskal's algorithms. Greedy algorithms work in phases. In each phase, a decision is made that appears to be good, without regard for future consequences. Generally, this means that some local optimum is chosen. This "take what you can get now" strategy is the source of the name for this class of algorithms. When the algorithm terminates, we hope that the local optimum is equal to the global optimum. If this is the case, then the algorithm is correct; otherwise, the algorithm has produced a suboptimal solution. If the absolute best answer is not required, then simple greedy algorithms are sometimes used to generate approximate answers, rather than using the more complicated algorithms generally required to generate an exact answer.

There are several real-life examples of greedy algorithms. The most obvious is the coin-changing problem. To make change in U.S. currency, we repeatedly dispense the largest denomination. Thus, to give out seventeen dollars and sixty-one cents in change, we give out a ten-dollar bill, a five-dollar bill, two one-dollar bills, two quarters, one dime, and one penny. By doing this, we are guaranteed to minimize the number of bills and coins. This algorithm does not work in all monetary systems, but fortunately, we can prove that it does work in the American monetary system. Indeed, it works even if two-dollar bills and fifty-cent pieces are allowed.

Traffic problems provide an example where making locally optimal choices does not always work. For example, during certain rush hour times in Miami, it is best to stay off the prime streets even if they look empty, because traffic will come to a standstill a mile down the road, and you will be stuck. Even more shocking, it is better in some cases to make a temporary detour in the direction opposite your destination in order to avoid all traffic bottlenecks.

In the remainder of this section, we will look at several applications that use greedy algorithms. The first application is a simple scheduling problem. Virtually all scheduling problems are either NP-complete (or of similar difficult complexity) or are solvable by a greedy algorithm. The second application deals with file compression and is one of the earliest results in computer science. Finally, we will look at an example of a greedy approximation algorithm.

10.1.1. A Simple Scheduling Problem

We are given jobs j1, j2, . . . , jn, all with known running times t1, t2, . . . , tn, respectively. We have a single processor. What is the best way to schedule these jobs in order to minimize the average completion time? In this entire section, we will assume nonpreemptive scheduling: Once a job is started, it must run to completion.

As an example, suppose we have the four jobs and associated running times shown in Figure 10.1. One possible schedule is shown in Figure 10.2. Because j1 finishes in 15 (time units), j2 in 23, j3 in 26, and j4 in 36, the average completion time is 25. A better schedule, which yields a mean completion time of 17.75, is shown in Figure 10.3.

The schedule given in Figure 10.3 is arranged by shortest job first. We can show that this will always yield an optimal schedule. Let the jobs in the schedule be ji1, ji2, . . . , jin. The first job finishes in time ti1. The second job finishes after ti1 + ti2, and the third job finishes after ti1 + ti2 + ti3. From this, we see that the total cost, C, of the schedule is

(10.1)

(10.2)

Job  Time

---------

 j1    15

 j2     8

 j3     3

 j4    10

Figure 10.1 Jobs and times

Figure 10.2 Schedule #1

Figure 10.3 Schedule #2 (optimal)

Notice that in Equation (10.2), the first sum is independent of the job ordering, so only the second sum affects the total cost. Suppose that in an ordering there exists some x > y such that tix < tiy. Then a calculation shows that by swapping jix and jiy, the second sum increases, decreasing the total cost. Thus, any schedule of jobs in which the times are not monotonically nonincreasing must be suboptimal. The only schedules left are those in which the jobs are arranged by smallest running time first, breaking ties arbitrarily.

This result indicates the reason the operating system scheduler generally gives precedence to shorter jobs.

The Multiprocessor Case

We can extend this problem to the case of several processors. Again we have jobs j1, j2, . . . , jn, with associated running times t1, t2, . . . , tn, and a number P of processors. We will assume without loss of generality that the jobs are ordered, shortest running time first. As an example, suppose P = 3, and the jobs are as shown in Figure 10.4.

Figure 10.5 shows an optimal arrangement to minimize mean completion time. Jobs j1, j4, and j7 are run on Processor 1. Processor 2 handles j 2, j5, and j8, and Processor 3 runs the remaining jobs. The total time to completion is 165, for an average of [image].

The algorithm to solve the multiprocessor case is to start jobs in order, cycling through processors. It is not hard to show that no other ordering can do better, although if the number of processors P evenly divides the number of jobs n, there are many optimal orderings. This is obtained by, for each 0 [image] i <n/P, placing each of the jobs jiP+1 through j(i+1)P on a different processor. In our case, Figure 10.6 shows a second optimal solution.

Job  Time

---------

 j1     3

 j2     5

 j3     6

 j4    10

 j5    11

 j6    14

 j7    15

 j8    18

 j9    20

Figure 10.4 Jobs and times

Figure 10.5 An optimal solution for the multiprocessor case

Even if P does not divide n exactly, there can still be many optimal solutions, even if all the job times are distinct. We leave further investigation of this as an exercise.

Minimizing the Final Completion Time

We close this section by considering a very similar problem. Suppose we are only concerned with when the last job finishes. In our two examples above, these completion times are 40 and 38, respectively. Figure 10.7 shows that the minimum final completion time is 34, and this clearly cannot be improved, because every processor is always busy.

Although this schedule does not have minimum mean completion time, it has merit in that the completion time of the entire sequence is earlier. If the same user owns all these jobs, then this is the preferable method of scheduling. Although these problems are very similar, this new problem turns out to be NP-complete; it is just another way of phrasing the knapsack or bin-packing problems, which we will encounter later in this section. Thus, minimizing the final completion time is apparently much harder than minimizing the mean completion time.

Figure 10.6 A second optimal solution for the multiprocessor case

Figure 10.7 Minimizing the final completion time

10.1.2. Huffman Codes

In this section, we consider a second application of greedy algorithms, known as file compression.

The normal ASCII character set consists of roughly 100 "printable" characters. In order to distinguish these characters, [image] log 100 [image] = 7 bits are required. Seven bits allow the representation of 128 characters, so the ASCII character set adds some other "nonprintable" characters. An eighth bit is added as a parity check. The important point, however, is that if the size of the character set is C, then [image] log C [image] bits are needed in a standard encoding.

Suppose we have a file that contains only the characters a, e, i, s, t, plus blank spaces and newlines. Suppose further, that the file has ten a's, fifteen e's, twelve i's, three s's, four t's, thirteen blanks, and one newline. As the table in Figure 10.8 shows, this file requires 174 bits to represent, since there are 58 characters and each character requires three bits.

Character  Code  Frequency  Total Bits

--------------------------------------

   a        000     10          30

   e        001     15          45

   i        010     12          36

   s        011      3           9

   t        100      4          12

 space      101      3          39

newline     110      1           3

--------------------------------------

 Total                         174

Figure 10.8 Using a standard coding scheme

In real life, files can be quite large. Many of the very large files are output of some program and there is usually a big disparity between the most frequent and least frequent characters. For instance, many large data files have an inordinately large amount of digits, blanks, and newlines, but few q's and x's. We might be interested in reducing the file size in the case where we are transmitting it over a slow phone line. Also, since on virtually every machine disk space is precious, one might wonder if it would be possible to provide a better code and reduce the total number of bits required.

The answer is that this is possible, and a simple strategy achieves 25 percent savings on typical large files and as much as 50 to 60 percent savings on many large data files. The general strategy is to allow the code length to vary from character to character and to ensure that the frequently occurring characters have short codes. Notice that if all the characters occur with the same frequency, then there are not likely to be any savings.

The binary code that represents the alphabet can be represented by the binary tree shown in Figure 10.9.

The tree in Figure 10.9 has data only at the leaves. The representation of each character can be found by starting at the root and recording the path, using a 0 to indicate the left branch and a 1 to indicate the right branch. For instance, s is reached by going left, then right, and finally right. This is encoded as 011. This data structure is sometimes referred to as a trie. If character ci is at depth di and occurs fi times, then the cost of the code is equal to [image] di fi.

Figure 10.9 Representation of the original code in a tree

Figure 10.10 A slightly better tree

A better code than the one given in Figure 10.9 can be obtained by noticing that the newline is an only child. By placing the newline symbol one level higher at its parent, we obtain the new tree in Figure 10.9. This new tree has cost of 173, but is still far from optimal.

Notice that the tree in Figure 10.10 is a full tree: All nodes either are leaves or have two children. An optimal code will always have this property, since otherwise, as we have already seen, nodes with only one child could move up a level.

If the characters are placed only at the leaves, any sequence of bits can always be decoded unambiguously. For instance, suppose the encoded string is 0100111100010110001000111. 0 is not a character code, 01 is not a character code, but 010 represents i, so the first character is i. Then 011 follows, giving a t. Then 11 follows, which is a newline. The remainder of the code is a, space, t, i, e, and newline. Thus, it does not matter if the character codes are different lengths, as long as no character code is a prefix of another character code. Such an encoding is known as a prefix code. Conversely, if a character is contained in a nonleaf node, it is no longer possible to guarantee that the decoding will be unambiguous.

Putting these facts together, we see that our basic problem is to find the full binary tree of minimum total cost (as defined above), where all characters are contained in the leaves. The tree in Figure 10.11 shows the optimal tree for our sample alphabet. As can be seen in Figure 10.12, this code uses only 146 bits.

Figure 10.11 Optimal prefix code

Character   Code  Frequency  Total Bits

------------------------=--------------

   a         001     10          30

   e          01     15          30

   i          10     12          24

   s       00000      3          15

   t        0001      4          16

 space        11     13          26

newline    00001      1           5

---------------------------------------

 Total                          146

Figure 10.12 Optimal prefix code

Notice that there are many optimal codes. These can be obtained by swapping children in the encoding tree. The main unresolved question, then, is how the coding tree is constructed. The algorithm to do this was given by Huffman in 1952. Thus, this coding system is commonly referred to as a Huffman code.

Huffman's Algorithm

Huffman's Algorithm

Throughout this section we will assume that the number of characters is C. Huffman's algorithm can be described as follows: We maintain a forest of trees. The weight of a tree is equal to the sum of the frequencies of its leaves. C - 1 times, select the two trees, T1 and T2, of smallest weight, breaking ties arbitrarily, and form a new tree with subtrees Tl and T2. At the beginning of the algorithm, there are C single-node trees-one for each character. At the end of the algorithm there is one tree, and this is the optimal Huffman coding tree.

A worked example will make the operation of the algorithm clear. Figure 10.13 shows the initial forest; the weight of each tree is shown in small type at the root. The two trees of lowest weight are merged together, creating the forest shown in Figure 10.14. We will name the new root T1, so that future merges can be stated unambiguously. We have made s the left child arbitrarily; any tiebreaking procedure can be used. The total weight of the new tree is just the sum of the weights of the old trees, and can thus be easily computed. It is also a simple matter to create the new tree, since we merely need to get a new node, set the left and right pointers, and record the weight.

Figure 10.13 Initial stage of Huffman's algorithm

Figure 10.14 Huffman's algorithm after the first merge

Figure 10.15 Huffman's algorithm after the second merge

Figure 10.16 Huffman's algorithm after the third merge

Now there are six trees, and we again select the two trees of smallest weight. These happen to be T1 and t, which are then merged into a new tree with root T2 and weight 8. This is shown in Figure 10.15. The third step merges T2 and a, creating T3, with weight 10 + 8 = 18. Figure 10.16 shows the result of this operation.

After the third merge is completed, the two trees of lowest weight are the single-node trees representing i and the blank space. Figure 10.17 shows how these trees are merged into the new tree with root T4. The fifth step is to merge the trees with roots e and T3, since these trees have the two smallest weights. The result of this step is shown in Figure 10.18.

Finally, the optimal tree, which was shown in Figure 10.11, is obtained by merging the two remaining trees. Figure 10.19 shows this optimal tree, with root T6.

Figure 10.17 Huffman's algorithm after the fourth merge

Figure 10.18 Huffman's algorithm after the fifth merge

Figure 10.19 Huffman's algorithm after the final merge

We will sketch the ideas involved in proving that Huffman's algorithm yields an optimal code; we will leave the details as an exercise. First, it is not hard to show by contradiction that the tree must be full, since we have already seen how a tree that is not full is improved.

Next, we must show that the two least frequent characters [image] and [image] must be the two deepest nodes (although other nodes may be as deep). Again, this is easy to show by contradiction, since if either [image] or [image] is not a deepest node, then there must be some [image] that is (recall that the tree is full). If [image] is less frequent than [image], then we can improve the cost by swapping them in the tree.

We can then argue that the characters in any two nodes at the same depth can be swapped without affecting optimality. This shows that an optimal tree can always be found that contains the two least frequent symbols as siblings; thus the first step is not a mistake.

The proof can be completed by using an induction argument. As trees are merged, we consider the new character set to be the characters in the roots. Thus, in our example, after four merges, we can view the character set as consisting of e and the metacharacters T3 and T4. This is probably the trickiest part of the proof; you are urged to fill in all of the details.

The reason that this is a greedy algorithm is that at each stage we perform a merge without regard to global considerations. We merely select the two smallest trees.

If we maintain the trees in a priority queue, ordered by weight, then the running time is O(C log C), since there will be one build_heap, 2C - 2 delete_mins, and C - 2 inserts, on a priority queue that never has more than C elements. A simple implementation of the priority queue, using a linked list, would give an O (C2) algorithm. The choice of priority queue implementation depends on how large C is. In the typical case of an ASCII character set, C is small enough that the quadratic running time is acceptable. In such an application, virtually all the running time will be spent on the disk I/O required to read the input file and write out the compressed version.

There are two details that must be considered. First, the encoding information must be transmitted at the start of the compressed file, since otherwise it will be impossible to decode. There are several ways of doing this; see Exercise 10.4. For small files, the cost of transmitting this table will override any possible savings in compression, and the result will probably be file expansion. Of course, this can be detected and the original left intact. For large files, the size of the table is not significant.

The second problem is that as described, this is a two-pass algorithm. The first pass collects the frequency data and the second pass does the encoding. This is obviously not a desirable property for a program dealing with large files. Some alternatives are described in the references.

10.1.3. Approximate Bin Packing

In this section, we will consider some algorithms to solve the bin packing problem. These algorithms will run quickly but will not necessarily produce optimal solutions. We will prove, however, that the solutions that are produced are not too far from optimal.

We are given n items of sizes s1, s2, . . . , sn. All sizes satisfy 0 < si [image] 1. The problem is to pack these items in the fewest number of bins, given that each bin has unit capacity. As an example, Figure 10.20 shows an optimal packing for an item list with sizes 0.2, 0.5, 0.4, 0.7, 0.1, 0.3, 0.8.

Figure 10.20 Optimal packing for 0.2, 0.5, 0.4, 0.7, 0.1, 0.3, 0.8

There are two versions of the bin packing problem. The first version is on-line bin packing. In this version, each item must be placed in a bin before the next item can be processed. The second version is the off-line bin packing problem. In an off-line algorithm, we do not need to do anything until all the input has been read. The distinction between on-line and off-line algorithms was discussed in Section 8.2.

On-line Algorithms

The first issue to consider is whether or not an on-line algorithm can actually always give an optimal answer, even if it is allowed unlimited computation. Remember that even though unlimited computation is allowed, an on-line algorithm must place an item before processing the next item and cannot change its decision.

To show that an on-line algorithm cannot always give an optimal solution, we will give it particularly difficult data to work on. Consider an input sequence I1 of m small items of weight [image] followed by m large items of weight [image], 0 < [image] < 0.01. It is clear that these items can be packed in m bins if we place one small item and one large item in each bin. Suppose there were an optimal on-line algorithm A that could perform this packing. Consider the operation of algorithm A on the sequence I2, consisting of only m small items of weight [image]. I2 can be packed in [m/2] bins. However, A will place each item in a separate bin, since A must yield the same results on I2 as it does for the first half of I1, since the first half of I1 is exactly the same input as I2. This means that A will use twice as many bins as is optimal for I2. What we have proven is that there is no optimal algorithm for on-line bin packing.

What the argument above shows is that an on-line algorithm never knows when the input might end, so any performance guarantees it provides must hold at every instant throughout the algorithm. If we follow the foregoing strategy, we can prove the following.

THEOREM 10.1.

There are inputs that force any on-line bin-packing algorithm to use at least [image] the optimal number of bins.

PROOF:

Suppose otherwise, and suppose for simplicity that m is even. Consider any on-line algorithm A running on the input sequence I1, above. Recall that this sequence consists of m small items followed by m large items. Let us consider what the algorithm A has done after processing the mth item. Suppose A has already used b bins. At this point in the algorithm, the optimal number of bins is m/2, because we can place two elements in each bin. Thus we know that [image], by our assumption of a [image] performance guarantee.

Now consider the performance of algorithm A after all items have been packed. All bins created after the bth bin must contain exactly one item, since all small items are placed in the first b bins, and two large items will not fit in a bin. Since the first b bins can have at most two items each, and the remaining bins have one item each, we see that packing 2m items will require at least 2m - b bins. Since the 2m items can be optimally packed using m bins, our performance guarantee assures us that [image].

Figure 10.21 Next fit for 0.2, 0.5, 0.4, 0.7, 0.1, 0.3, 0.8

The first inequality implies that [image], and the second inequality implies that [image], which is a contradiction. Thus, no on-line algorithm can guarantee that it will produce a packing with less than [image] the optimal number of bins.

There are three simple algorithms that guarantee that the number of bins used is no more than twice optimal. There are also quite a few more complicated algorithms with better guarantees.

Next Fit

Probably the simplest algorithm is next fit. When processing any item, we check to see whether it fits in the same bin as the last item. If it does, it is placed there; otherwise, a new bin is created. This algorithm is incredibly simple to implement and runs in linear time. Figure 10.21 shows the packing produced for the same input as Figure 10.20.

Not only is next fit simple to program, its worst-case behavior is also easy to analyze.

THEOREM 10.2.

Let m be the optimal number of bins required to pack a list I of items. Then next fit never uses more than 2m bins. There exist sequences such that next fit uses 2m - 2 bins.

PROOF:

Consider any adjacent bins Bj and Bj + 1. The sum of the sizes of all items in Bj and Bj + 1 must be larger than 1, since otherwise all of these items would have been placed in Bj. If we apply this result to all pairs of adjacent bins, we see that at most half of the space is wasted. Thus next fit uses at most twice the number of bins.

To see that this bound is tight, suppose that the n items have size si = 0.5 if i is odd and si = 2/n if i is even. Assume n is divisible by 4. The optimal packing, shown in Figure 10.22, consists of n/4 bins, each containing 2 elements of size 0.5, and one bin containing the n/2 elements of size 2/n, for a total of (n/4) + 1. Figure 10.23 shows that next fit uses n/2 bins. Thus, next fit can be forced to use almost twice as many bins as optimal.

Figure 10.22 Optimal packing for 0.5, 2/n, 0.5, 2/n, 0.5, 2/n, . . .

Figure 10.23 Next fit packing for 0.5, 2/n, 0.5, 2/n, 0.5, 2/n, . . .

First Fit

Although next fit has a reasonable performance guarantee, it performs poorly in practice, because it creates new bins when it does not need to. In the sample run, it could have placed the item of size 0.3 in either B1 or B2, rather than create a new bin.

The first fit strategy is to scan the bins in order and place the new item in the first bin that is large enough to hold it. Thus, a new bin is created only when the results of previous placements have left no other alternative. Figure 10.24 shows the packing that results from first fit on our standard input.

A simple method of implementing first fit would process each item by scanning down the list of bins sequentially. This would take O(n2). It is possible to implement first fit to run in O(n log n); we leave this as an exercise.

A moment's thought will convince you that at any point, at most one bin can be more than half empty, since if a second bin were also half empty, its contents would fit into the first bin. Thus, we can immediately conclude that first fit guarantees a solution with at most twice the optimal number of bins.

Figure 10.24 First fit for 0.2, 0.5, 0.4, 0.7, 0.1, 0.3, 0.8

On the other hand, the bad case that we used in the proof of next fit's performance bound does not apply for first fit. Thus, one might wonder if a better bound can be proven. The answer is yes, but the proof is complicated.

THEOREM 10.3.

Let m be the optimal number of bins required to pack a list I of items. Then first fit never uses more than [image] bins. There exist sequences such that first fit uses [image] bins.

PROOF:

See the references at the end of the chapter.

An example where first fit does almost as poorly as the previous theorem would indicate is shown in Figure 10.25. The input consists of 6m items of size [image], followed by 6m items of size [image] , followed by 6m items of size [image]. One simple packing places one item of each size in a bin and requires 6m bins. First fit requires 10m bins.

When first fit is run on a large number of items with sizes uniformly distributed between 0 and 1, empirical results show that first fit uses roughly 2 percent more bins than optimal. In many cases, this is quite acceptable.

Figure 10.25 A case where first fit uses 10m bins instead of 6m

Figure 10.26 Best fit for 0.2, 0.5, 0.4, 0.7, 0.1, 0.3, 0.8

First Fit

Although next fit has a reasonable performance guarantee, it performs poorly in practice, because it creates new bins when it does not need to. In the sample run, it could have placed the item of size 0.3 in either B1 or B2, rather than create a new bin.

The first fit strategy is to scan the bins in order and place the new item in the first bin that is large enough to hold it. Thus, a new bin is created only when the results of previous placements have left no other alternative. Figure 10.24 shows the packing that results from first fit on our standard input.

A simple method of implementing first fit would process each item by scanning down the list of bins sequentially. This would take O(n2). It is possible to implement first fit to run in O(n log n); we leave this as an exercise.

A moment's thought will convince you that at any point, at most one bin can be more than half empty, since if a second bin were also half empty, its contents would fit into the first bin. Thus, we can immediately conclude that first fit guarantees a solution with at most twice the optimal number of bins.

Figure 10.24 First fit for 0.2, 0.5, 0.4, 0.7, 0.1, 0.3, 0.8

On the other hand, the bad case that we used in the proof of next fit's performance bound does not apply for first fit. Thus, one might wonder if a better bound can be proven. The answer is yes, but the proof is complicated.

THEOREM 10.3.

Let m be the optimal number of bins required to pack a list I of items. Then first fit never uses more than [image] bins. There exist sequences such that first fit uses [image] bins.

PROOF:

See the references at the end of the chapter.

An example where first fit does almost as poorly as the previous theorem would indicate is shown in Figure 10.25. The input consists of 6m items of size [image], followed by 6m items of size [image] , followed by 6m items of size [image]. One simple packing places one item of each size in a bin and requires 6m bins. First fit requires 10m bins.

When first fit is run on a large number of items with sizes uniformly distributed between 0 and 1, empirical results show that first fit uses roughly 2 percent more bins than optimal. In many cases, this is quite acceptable.

Figure 10.25 A case where first fit uses 10m bins instead of 6m

Figure 10.26 Best fit for 0.2, 0.5, 0.4, 0.7, 0.1, 0.3, 0.8

10.2. Divide and Conquer

Another common technique used to design algorithms is divide and conquer. Divide and conquer algorithms consist of two parts:

Divide: Smaller problems are solved recursively (except, of course, base cases).

Conquer: The solution to the original problem is then formed from the solutions to the subproblems.

Traditionally, routines in which the text contains at least two recursive calls are called divide and conquer algorithms, while routines whose text contains only one recursive call are not. We generally insist that the subproblems be disjoint (that is, essentially nonoverlapping). Let us review some of the recursive algorithms that have been covered in this text.

We have already seen several divide and conquer algorithms. In Section 2.4.3, we saw an O (n log n) solution to the maximum subsequence sum problem. In Chapter 4, we saw linear-time tree traversal strategies. In Chapter 7, we saw the classic examples of divide and conquer, namely mergesort and quicksort, which have O (n log n) worst-case and average-case bounds, respectively.

We have also seen several examples of recursive algorithms that probably do not classify as divide and conquer, but merely reduce to a single simpler case. In Section 1.3, we saw a simple routine to print a number. In Chapter 2, we used recursion to perform efficient exponentiation. In Chapter 4, we examined simple search routines for binary search trees. In Section 6.6, we saw simple recursion used to merge leftist heaps. In Section 7.7, an algorithm was given for selection that takes linear average time. The disjoint set find operation was written recursively in Chapter 8. Chapter 9 showed routines to recover the shortest path in Dijkstra's algorithm and other procedures to perform depth-first search in graphs. None of these algorithms are really divide and conquer algorithms, because only one recursive call is performed.

We have also seen, in Section 2.4, a very bad recursive routine to compute the Fibonacci numbers. This could be called a divide and conquer algorithm, but it is terribly inefficient, because the problem really is not divided at all.

In this section, we will see more examples of the divide and conquer paradigm. Our first application is a problem in computational geometry. Given n points in a plane, we will show that the closest pair of points can be found in O(n log n) time. The exercises describe some other problems in computational geometry which can be solved by divide and conquer. The remainder of the section shows some extremely interesting, but mostly theoretical, results. We provide an algorithm which solves the selection problem in O(n) worst-case time. We also show that 2 n-bit numbers can be multiplied in o(n2) operations and that two n x n matrices can be multiplied in o(n3) operations. Unfortunately, even though these algorithms have better worst-case bounds than the conventional algorithms, none are practical except for very large inputs.

10.2.1. Running Time of Divide and Conquer Algorithms

10.2.1. Running Time of Divide and Conquer Algorithms

All the efficient divide and conquer algorithms we will see divide the problems into subproblems, each of which is some fraction of the original problem, and then perform some additional work to compute the final answer. As an example, we have seen that mergesort operates on two problems, each of which is half the size of the original, and then uses O(n) additional work. This yields the running time equation (with appropriate initial conditions)

T(n) = 2T(n/2) + O(n)

We saw in Chapter 7 that the solution to this equation is O(n log n). The following theorem can be used to determine the running time of most divide and conquer algorithms.

THEOREM 10.6.

The solution to the equation T(n) = aT(n/b) + [image] (nk), where a [image] 1 and b > 1, is

PROOF:

Following the analysis of mergesort in Chapter 7, we will assume that n is a power of b; thus, let n = bm. Then n/b = bm-l and nk = (bm)k = bmk = bkm = (bk)m. Let us assume T(1) = 1, and ignore the constant factor in [image] (nk). Then we have

T(bm) = aT(bm-l)+(bk)m

If we divide through by am, we obtain the equation

(10.3)

We can apply this equation for other values of m, obtaining

(10.4)

(10.5)

(10.6)

We use our standard trick of adding up the telescoping equations (10.3) through (10.6). Virtually all the terms on the left cancel the leading terms on the right, yielding

(10.7)

(10.8)

Thus

(10.9)

If a > bk, then the sum is a geometric series with ratio smaller than 1. Since the sum of infinite series would converge to a constant, this finite sum is also bounded by a constant, and thus Equation (10.10) applies:

T(n) = O(am) = O(alogb n) O = O(nlogb a)

(10.10)

If a = bk, then each term in the sum is 1. Since the sum contains 1 + logb n terms and a = bk implies that logb a = k,

T(n) = O(am logb n) = O(nlogba logb n) = O(nk logb n)

= O (nk log n)

(10.11)

Finally, if a < bk, then the terms in the geometric series are larger than 1, and the second formula in Section 1.2.3 applies. We obtain

(10.12)

proving the last case of the theorem.

As an example, mergesort has a = b = 2 and k = 1. The second case applies, giving the answer O(n log n). If we solve three problems, each of which is half the original size, and combine the solutions with O(n) additional work, then a = 3, b = 2 and k = 1. Case 1 applies here, giving a bound of O(nlog23) = O(n1.59). An algorithm that solved three half-sized problems, but required O(n2) work to merge the solution, would have an O(n2) running time, since the third case would apply.

There are two important cases that are not covered by Theorem 10.6. We state two more theorems, leaving the proofs as exercises. Theorem 10.7 generalizes the previous theorem.

THEOREM 10.7.

The solution to the equation T(n) = aT(n/b) + [image] (nk logp n), where a [image] 1, b > 1, and p [image] 0 is

THEOREM 10.8.

, then the solution to the equation [image] is T(n) = O(n).

10.2.2. Closest-Points Problem

The input to our first problem is a list P of points in a plane. If pl = (x1, y1) and p2 = (x2, y2), then the Euclidean distance between p l and p 2 is [(x1 - x2)2 + (yl - y2)2]l/2. We are required to find the closest pair of points. It is possible that two points have the same position; in that case that pair is the closest, with distance zero.

If there are n points, then there are n (n - 1)/2 pairs of distances. We can check all of these, obtaining a very short program, but at the expense of an O(n2) algorithm. Since this approach is just an exhaustive search, we should expect to do better.

Let us assume that the points have been sorted by x coordinate. At worst, this adds O(n log n) to the final time bound. Since we will show an O(n log n) bound for the entire algorithm, this sort is essentially free, from a complexity standpoint.

Figure 10.29 shows a small sample point set P. Since the points are sorted by x coordinate, we can draw an imaginary vertical line that partitions the points set into two halves, Pl and Pr. This is certainly simple to do. Now we have almost exactly the same situation as we saw in the maximum subsequence sum problem in Section 2.4.3. Either the closest points are both in Pl, or they are both in Pr, or one is in Pl and the other is in Pr. Let us call these distances dl, dr, and dc. Figure 10.30 shows the partition of the point set and these three distances.

We can compute dl and dr recursively. The problem, then, is to compute dc. Since we would like an O(n log n) solution, we must be able to compute dc with only O(n) additional work. We have already seen that if a procedure consists of two half-sized recursive calls and O(n) additional work, then the total time will be O(n log n).

Let [image] = min(dl, dr). The first observation is that we only need to compute dc if dc improves on [image] . If dc is such a distance, then the two points that define dc must be within [image] of the dividing line; we will refer to this area as a strip. As shown in Figure 10.31, this observation limits the number of points that need to be considered (in our case, [image] = dr).

There are two strategies that can be tried to compute dc. For large point sets that are uniformly distributed, the number of points that are expected to be in the strip is very small. Indeed, it is easy to argue that only [image] points are in the strip on average. Thus, we could perform a brute force calculation on these points in O(n) time. The pseudocode in Figure 10.32 implements this strategy, assuming the C convention that the points are indexed starting at 0.

Figure 10.29 A small point set

Figure 10.30 P partitioned into P1 and P2; shortest distances are shown

Figure 10.31 Two-lane strip, containing all points considered for dc strip

/* Points are all in the strip */

for( i=0; i<NUM_POINTS_IN_STRIP; i++ )

for( j=i+1; j<NUM_POINTS_IN_STRIP; j++ )

if( dist( pi,pj ) <   )

   = dist( pi,pj );

Figure 10.32 Brute force calculation of min(  , dc)

/* Points are all in the strip and sorted by y coordinate */

for( i=0; i<NUM_POINTS_IN_STRIP; i++ )

for( j=i+1; j<NUM_POINTS_IN_STRIP; j++ )

if ( pi and pj 's coordinates differ by more than    )

break;     /* goto next pi */

else

if( dist( pi, pj) <   )

   = dist( pi, pj);

Figure 10.33 Refined calculation of min(  , dc)

In the worst case, all the points could be in the strip, so this strategy does not always work in linear time. We can improve this algorithm with the following observation: The y coordinates of the two points that define dc can differ by at most   [image] . Otherwise, dc >   [image] . Suppose that the points in the strip are sorted by their y coordinates. Therefore, if pi and pj's y coordinates differ by more than   [image] , then we can proceed to pi + l. This simple modification is implemented in Figure 10.33.

This extra test has a significant effect on the running time, because for each pi only a few points pj are examined before pi's and pj's y coordinates differ by more than   [image]  and force an exit from the inner for loop. Figure 10.34 shows, for instance, that for point p3, only the two points p4 and p5 lie in the strip within   [image]  vertical distance.

Figure 10.34 Only p4 and p5 are considered in the second for loop

Figure 10.35 At most eight points fit in the rectangle; there are two coordinates shared by two points each

In the worst case, for any point pi, at most 7 points pj are considered. This is because these points must lie either in the   [image]  by   [image]  square in the left half of the strip or in the   [image]  by   [image]  square in the right half of the strip. On the other hand, all the points in each   [image]  by   [image]  square are separated by at least   [image] . In the worst case, each square contains four points, one at each corner. One of these points is pi, leaving at most seven points to be considered. This worst-case situation is shown in Figure 10.35. Notice that even though pl2 and pr1 have the same coordinates, they could be different points. For the actual analysis, it is only important that the number of points in the   [image]  by 2  [image]  rectangle be O(1), and this much is certainly clear.

Because at most seven points are considered for each pi, the time to compute a dc that is better than   [image]  is O(n). Thus, we appear to have an O(n log n) solution to the closest-points problem, based on the two half-sized recursive calls plus the linear extra work to combine the two results. However, we do not quite have an O (n log n) solution yet.

The problem is that we have assumed that a list of points sorted by y coordinate is available. If we perform this sort for each recursive call, then we have O(n log n) extra work: this gives an O(n log2 n) algorithm. This is not all that bad, especially when compared to the brute force O(n2). However, it is not hard to reduce the work for each recursive call to O(n), thus ensuring an O(n log n) algorithm.

We will maintain two lists. One is the point list sorted by x coordinate, and the other is the point list sorted by y coordinate. We will call these lists P and Q, respectively. These can be obtained by a preprocessing sorting step at cost O(n log n) and thus does not affect the time bound. Pl and Ql are the lists passed to the left-half recursive call, and Pr and Qr are the lists passed to the right-half recursive call. We have already seen that P is easily split in the middle. Once the dividing line is known, we step through Q sequentially, placing each element in Ql or Qr, as appropriate. It is easy to see that Ql and Qr will be automatically sorted by y coordinate. When the recursive calls return, we scan through the Q list and discard all the points whose x coordinates are not within the strip. Then Q contains only points in the strip, and these points are guaranteed to be sorted by their y coordinates.

This strategy ensures that the entire algorithm is O (n log n), because only O (n) extra work is performed.

10.2.3. The Selection Problem

The selection problem requires us to find the kth smallest element in a list S of n elements. Of particular interest is the special case of finding the median. This occurs when k =   [image] n/2  [image] .

In Chapters 1, 6, 7 we have seen several solutions to the selection problem. The solution in Chapter 7 uses a variation of quicksort and runs in O(n) average time. Indeed, it is described in Hoare's original paper on quicksort.

Although this algorithm runs in linear average time, it has a worst case of O (n2). Selection can easily be solved in O(n log n) worst-case time by sorting the elements, but for a long time it was unknown whether or not selection could be accomplished in O(n) worst-case time. The quickselect algorithm outlined in Section 7.7.6 is quite efficient in practice, so this was mostly a question of theoretical interest.

Recall that the basic algorithm is a simple recursive strategy. Assuming that n is larger than the cutoff point where elements are simply sorted, an element v, known as the pivot, is chosen. The remaining elements are placed into two sets, S1 and S2. S1 contains elements that are guaranteed to be no larger than v, and S2 contains elements that are no smaller than v. Finally, if k   [image]  |S1|, then the kth smallest element in S can be found by recursively computing the kth smallest element in S1. If k = |S1| + 1, then the pivot is the kth smallest element. Otherwise, the kth smallest element in S is the (k - |S1| -1 )st smallest element in S2. The main difference between this algorithm and quicksort is that there is only one subproblem to solve instead of two.

In order to obtain a linear algorithm, we must ensure that the subproblem is only a fraction of the original and not merely only a few elements smaller than the original. Of course, we can always find such an element if we are willing to spend some time to do so. The difficult problem is that we cannot spend too much time finding the pivot.

For quicksort, we saw that a good choice for pivot was to pick three elements and use their median. This gives some expectation that the pivot is not too bad, but does not provide a guarantee. We could choose 21 elements at random, sort them in constant time, use the 11th largest as pivot, and get a pivot that is even more likely to be good. However, if these 21 elements were the 21 largest, then the pivot would still be poor. Extending this, we could use up to O (n / log n) elements, sort them using heapsort in O(n) total time, and be almost certain, from a statistical point of view, of obtaining a good pivot. In the worst case, however, this does not work because we might select the O (n / log n) largest elements, and then the pivot would be the [n - O(n / log n)]th largest element, which is not a constant fraction of n.

The basic idea is still useful. Indeed, we will see that we can use it to improve the expected number of comparisons that quickselect makes. To get a good worst case, however, the key idea is to use one more level of indirection. Instead of finding the median from a sample of random elements, we will find the median from a sample of medians.

The basic pivot selection algorithm is as follows:

1.     Arrange the n elements into   [image] n/5  [image]  groups of 5 elements, ignoring the (at most four) extra elements.

2.     Find the median of each group. This gives a list M of   [image] n/5  [image]  medians.

3.     Find the median of M. Return this as the pivot, v.

We will use the term median-of-median-of-five partitioning to describe the quickselect algorithm that uses the pivot selection rule given above. We will now show that median-of-median-of-five partitioning guarantees that each recursive subproblem is at most roughly 70 percent as large as the original. We will also show that the pivot can be computed quickly enough to guarantee an O (n) running time for the entire selection algorithm.

Let us assume for the moment that n is divisible by 5, so there are no extra elements. Suppose also that n/5 is odd, so that the set M contains an odd number of elements. This provides some symmetry, as we shall see. We are thus assuming, for convenience, that n is of the form 10k + 5. We will also assume that all the elements are distinct. The actual algorithm must make sure to handle the case where this is not true. Figure 10.36 shows how the pivot might be chosen when n = 45.

In Figure 10.36, v represents the element which is selected by the algorithm as pivot. Since v is the median of nine elements, and we are assuming that all elements are distinct, there must be four medians that are larger than v and four that are smaller. We denote these by L and S, respectively. Consider a group of five elements with a large median (type L). The median of the group is smaller than two elements in the group and larger than two elements in the group. We will let H represent the huge elements. These are elements that are known to be larger than a large median. Similarly, T represents the tiny elements, which are smaller than a small median. There are 10 elements of type H: Two are in each of the groups with an L type median, and two elements are in the same group as v. Similarly, there are 10 elements of type T.

Figure 10.36 How the pivot is chosen

Elements of type L or H are guaranteed to be larger than v, and elements of type S or T are guaranteed to be smaller than v. There are thus guaranteed to be 14 large and 14 small elements in our problem. Therefore, a recursive call could be on at most 45 - 14 - 1 = 30 elements.

Let us extend this analysis to general n of the form 10k + 5. In this case, there are k elements of type L and k elements of type S . There are 2k + 2 elements of type H, and also 2k + 2 elements of type T. Thus, there are 3k + 2 elements that are guaranteed to be larger than v and 3k + 2 elements that are guaranteed to be smaller. Thus, in this case, the recursive call can contain at most 7k + 2 < 0.7n elements. If n is not of the form 10k + 5, similar arguments can be made without affecting the basic result.

It remains to bound the running time to obtain the pivot element. There are two basic steps. We can find the median of five elements in constant time. For instance, it is not hard to sort five elements in eight comparisons. We must do this   [image] n/5  [image]  times, so this step takes O(n) time. We must then compute the median of a group of   [image] n/5  [image]  elements. The obvious way to do this is to sort the group and return the element in the middle. But this takes O(  [image] n/5  [image]  log   [image] n/5  [image] ) = O(n log n) time, so this does not work. The solution is to call the selection algorithm recursively on the   [image] n/5  [image] elements.

This completes the description of the basic algorithm. There are still some details that need to be filled in if an actual implementation is desired. For instance, duplicates must be handled correctly, and the algorithm needs a cutoff large enough to ensure that the recursive calls make progress. There is quite a large amount of overhead involved, and this algorithm is not practical at all, so we will not describe any more of the details that need to be considered. Even so, from a theoretical standpoint, the algorithm is a major breakthrough, because, as the following theorem shows, the running time is linear in the worst case.

THEOREM 10.9.

The running time of quickselect using median-of-median-of-five partitioning is O(n).

PROOF:

The algorithm consists of two recursive calls of size 0.7n and 0.2n, plus linear extra work. By Theorem 10.8, the running time is linear.

Reducing the Average Number of Comparisons

Reducing the Average Number of Comparisons

Divide and conquer can also be used to reduce the expected number of comparisons required by the selection algorithm. Let us look at a concrete example. Suppose we have a group S of 1,000 numbers and are looking for the 100th smallest number, which we will call x. We choose a subset S' of S consisting of 100 numbers. We would expect that the value of x is similar in size to the 10th smallest number in S'. More specifically, the fifth smallest number in S' is almost certainly less than x, and the 15th smallest number in S' is almost certainly greater than x.

More generally, a sample S' of s elements is chosen from the n elements. Let   [image]  be some number, which we will choose later so as to minimize the average number of comparisons used by the procedure. We find the (v1 = ks/n -   [image])th and (v2 = ks/n +   [image])th smallest elements in S'. Almost certainly, the kth smallest element in S will fall between v1 and v2, so we are left with a selection problem on 2  [image]  elements. With low probability, the kth smallest element does not fall in this range, and we have considerable work to do. However, with a good choice of s and   [image], we can ensure, by the laws of probability, that the second case does not adversely affect the total work.

If an analysis is performed, we find that if s = n2/3 log1/3 n and   [image]  = n1/3 log2/3 n, then the expected number of comparisons is n + k + O(n2/3 log1/3 n), which is optimal except for the low-order term. (If k > n/2, we can consider the symmetric problem of finding the (n - k)th largest element.)

Most of the analysis is easy to do. The last term represents the cost of performing the two selections to determine v1 and v2. The average cost of the partitioning, assuming a reasonably clever strategy, is equal to n plus the expected rank of v2 in S, which is n + k + O(n  [image] /s). If the kth element winds up in S', the cost of finishing the algorithm is equal to the cost of selection on S', namely O(s). If the kth smallest element doesn't wind up in S', the cost is O(n). However, s and   [image]  have been chosen to guarantee that this happens with very low probability o(1/n), so the expected cost of this possibility is o(1), which is a term that goes to zero as n gets large. An exact calculation is left as Exercise 10.21.

This analysis shows that finding the median requires about 1.5n comparisons on average. Of course, this algorithm requires some floating-point arithmetic to compute s, which can slow down the algorithm on some machines. Even so, experiments have shown that if correctly implemented, this algorithm compares favorably with the quickselect implementation in Chapter 7.

10.2.4. Theoretical Improvements for Arithmetic Problems

In this section we describe a divide and conquer algorithm that multiplies two n-digit numbers. Our previous model of computation assumed that multiplication was done in constant time, because the numbers were small. For large numbers, this assumption is no longer valid. If we measure multiplication in terms of the size of numbers being multiplied, then the natural multiplication algorithm takes quadratic time. The divide and conquer algorithm runs in subquadratic time. We also present the classic divide and conquer algorithm that multiplies two n by n matrices in subcubic time.

Multiplying Integers

Matrix Multiplication

Multiplying Integers

Suppose we want to multiply two n-digit numbers x and y. If exactly one of x and y is negative, then the answer is negative; otherwise it is positive. Thus, we can perform this check and then assume that x, y   [image]  0. The algorithm that almost everyone uses when multiplying by hand requires   [image] (n2) operations, because each digit in x is multiplied by each digit in y.

If x = 61,438,521 and y = 94,736,407, xy = 5,820,464,730,934,047. Let us break x and y into two halves, consisting of the most significant and least significant digits, respectively. Then xl = 6,143, xr = 8,521, yl = 9,473, and yr = 6,407. We also have x = xl104 + xr and y = yl104 + yr. It follows that

xy = xlyl108 + (xlyr + xryl)104 + xryr

Notice that this equation consists of four multiplications, xlyl, xlyr, xryl, and xryr, which are each half the size of the original problem (n/2 digits). The multiplications by 108 and 104 amount to the placing of zeros. This and the subsequent additions add only O(n) additional work. If we perform these four multiplications recursively using this algorithm, stopping at an appropriate base case, then we obtain the recurrence

T(n) = 4T(n/2) + O(n)

From Theorem 10.6, we see that T(n) = O(n2), so, unfortunately, we have not improved the algorithm. To achieve a subquadratic algorithm, we must use less than four recursive calls. The key observation is that

xlyr + xryl = (xl - xr)(yr - yl) + xlyl + xryr

Thus, instead of using two multiplications to compute the coefficient of 104, we can use one multiplication, plus the result of two multiplications that have already been performed. Figure 10.37 shows how only three recursive subproblems need to be solved.

It is easy to see that now the recurrence equation satisfies

T(n) = 3T(n/2) + O(n),

and so we obtain T(n) = O(nlog23) = O(n1.59). To complete the algorithm, we must have a base case, which can be solved without recursion.

Figure 10.37 The divide and conquer algorithm in action

When both numbers are one-digit, we can do the multiplication by table lookup. If one number has zero digits, then we return zero. In practice, if we were to use this algorithm, we would choose the base case to be that which is most convenient for the machine.

Although this algorithm has better asymptotic performance than the standard quadratic algorithm, it is rarely used, because for small n the overhead is significant, and for larger n there are even better algorithms. These algorithms also make extensive use of divide and conquer.

Matrix Multiplication

A fundamental numerical problem is the multiplication of two matrices. Figure 10.38 gives a simple O(n3) algorithm to compute C = AB, where A, B, and C are n   [image]  n matrices. The algorithm follows directly from the definition of matrix multiplication. To compute Ci,j, we compute the dot product of the ith row in A with the jth column in B. As usual, arrays begin at index 0.

For a long time it was assumed that   [image] (n3) was required for matrix multiplication. However, in the late sixties Strassen showed how to break the   [image] (n3) barrier. The basic idea of Strassen's algorithm is to divide each matrix into four quadrants, as shown in Figure 10.39. Then it is easy to show that

C1,1 = A1,1B1,1 + A1,2B2,1

C1,2 = A1,1B1,2 + A1,2B2,2

C2,1 = A2,1B1,1 + A2,2B2,1

C2,2 = A2,1B1,2 + A2,2B2,2

/* Standard matrix multiplication. Arrays start at 0 */

void

matrix_multiply( matrix A, matrix B, matrix C, unsigned int n )

{

int i, j, k;

for( i=0; i<n; i++ )     /* Initialization */

for( j=O; j<n; j++ )

C[i][j] = 0.0;

for( i=0; i<n; i++ )

for( j=0; j<n; j++ )

for( k=0; k<n; k++ )

C[i][j] += A[i][k] * B[k][j];

}

Figure 10.38 Simple O(n3) matrix multiplication

Figure 10.39 Decomposing AB = C into four quadrants

As an example, to perform the multiplication AB

we define the following eight n/2 by n/2 matrices:

We could then perform eight n/2 by n/2 matrix multiplications and four n/2 by n/2 matrix additions. The matrix additions take O(n2) time. If the matrix multiplications are done recursively, then the running time satisfies

T(n) = 8T(n/2) + O(n2).

From Theorem 10.6, we see that T(n) = O(n3), so we do not have an improvement. As we saw with integer multiplication, we must reduce the number of subproblems below 8. Strassen used a strategy similar to the integer multiplication divide and conquer algorithm and showed how to use only seven recursive calls by carefully arranging the computations. The seven multiplications are

M1 = (A1,2 - A2,2)(B2,1 + B2,2)

M2 = (A1,1 + A2,2)(B1,1 + B2,2)

M3 = (A1,1 - A2,1)(B1,1 + B1,2)

M4 = (A1,1 + A1,2)B2,2

M5 = A1,1(B1,2 - B2,2)

M6 = A2,2(B2,1 - B1,1)

M7 = (A2,1 + A2,2)B1,1

Once the multiplications are performed, the final answer can be obtained with eight more additions.

C1,1 = M1 + M2 - M4 + M6

C1,2 = M4 + M5

C1,3 = M6 + M7

C1,4 = M2 - M3 + M5 - M7

It is straightforward to verify that this tricky ordering produces the desired values. The running time now satisfies the recurrence

T(n) = 7T(n/2) + O(n2).

The solution of this recurrence is T(n) = O(nlog27) = O(n2.81).

As usual, there are details to consider, such as the case when n is not a power of two, but these are basically minor nuisances. Strassen's algorithm is worse than the straightforward algorithm until n is fairly large. It does not generalize for the case where the matrices are sparse (contain many zero entries), and it does not easily parallelize. When run with floating-point entries, it is less stable numerically than the classic algorithm. Thus, it is has only limited applicability. Nevertheless, it represents an important theoretical milestone and certainly shows that in computer science, as in many other fields, even though a problem seems to have an intrinsic complexity, nothing is certain until proven.

10.3. Dynamic Programming

In the previous section, we have seen that a problem that can be mathematically expressed recursively can also be expressed as a recursive algorithm, in many cases yielding a significant performance improvement over a more naïve exhaustive search.

Any recursive mathematical formula could be directly translated to a recursive algorithm, but the underlying reality is that often the compiler will not do justice to the recursive algorithm, and an inefficient program results. When we suspect that this is likely to be the case, we must provide a little more help to the compiler, by rewriting the recursive algorithm as a nonrecursive algorithm that systematically records the answers to the subproblems in a table. One technique that makes use of this approach is known as dynamic programming.

10.3.1. Using a Table Instead of Recursion

In Chapter 2, we saw that the natural recursive program to compute the Fibonacci numbers is very inefficient. Recall that the program shown in Figure 10.40 has a running time T(n) that satisfies T(n)   [image]  T(n - 1) + T(n - 2). Since T(n) satisfies the same recurrence relation as the Fibonacci numbers and has the same initial conditions, T(n) in fact grows at the same rate as the Fibonacci numbers, and is thus exponential.

On the other hand, since to compute Fn, all that is needed is Fn-1 and Fn-2, we only need to record the two most recently computed Fibonacci numbers. This yields the O(n) algorithm in Figure 10.41

The reason that the recursive algorithm is so slow is because of the algorithm used to simulate recursion. To compute Fn, there is one call to Fn-1 and Fn-2. However, since Fn-1 recursively makes a call to Fn-2 and Fn-3, there are actually two separate calls to compute Fn-2. If one traces out the entire algorithm, then we can see that Fn-3 is computed three times, Fn-4 is computed five times, Fn-5 is computed eight times, and so on. As Figure 10.42 shows, the growth of redundant calculations is explosive. If the compiler's recursion simulation algorithm were able to keep a list of all precomputed values and not make a recursive call for an already solved subproblem, then this exponential explosion would be avoided. This is why the program in Figure 10.41 is so much more efficient. calculations is explosive. If the compiler's recursion simulation algorithm were able to keep a list of all precomputed values and not make a recursive call for an already solved subproblem, then this exponential explosion would be avoided. This is why the program in Figure 10.41 is so much more efficient.

/* Compute Fibonacci numbers as described in Chapter 1 */

unsigned int

fib( unsigned int n )

{

if( n <= 1 )

return 1;

else

return( fib( n-1 ) + fib( n-2 ) );

}

Figure 10.40 Inefficient algorithm to compute Fibonacci numbers

unsigned int

fibonacci( unsigned int n )

{

unsigned int i, last, next_to_last, answer;

if( n <= 1 )

return 1;

last = next_to_last = 1;

for( i = 2; i <= n; i++ )

{

answer = last + next_to_last;

next_to_last = last;

last = answer;

}

return answer;

}

Figure 10.41 Linear algorithm to compute Fibonacci numbers

Figure 10.42 Trace of the recursive calculation of Fibonacci numbers

double

eval( unsigned int n )

{

int i;

double sum;

if( n == 0 )

return 1.0;

else

{

sum = 0.0;

for( i=0; i<n; i++ )

sum += eval(i);

return( 2.0 * sum / n + n );

}

}

Figure 10.43 Recursive program to evaluate

As a second example, we saw in Chapter 7 how to solve the recurrence   [image]  with C(0) = 1. Suppose that we want to check, numerically, whether the solution we obtained is correct. We could then write the simple program in Figure 10.43 to evaluate the recursion.

Once again, the recursive calls duplicate work. In this case, the running time T(n) satisfies   [image]  because, as shown in Figure 10.44, there is one (direct) recursive call of each size from 0 to n -1, plus O(n) additional work (where else have we seen the tree shown in Figure 10.44?). Solving for T(n), we find that it grows exponentially. By using a table, we obtain the program in Figure 10.45. This program avoids the redundant recursive calls and runs in O(n2). It is not a perfect program; as an exercise, you should make the simple change that reduces its running time to O(n).

Figure 10.44 Trace of the recursive calculation in eval

double

eval( unsigned int n )

{

int i,j;

double sum, answer;

double *c;

c = (double*) malloc( sizeof (double)*(n+1) );

if( c == NULL )

fatal_error("Out of space!!!");

c[0] = 1.0;

for( i=1; i<=n; i++ )    /* Evaluate Ci, 1    i    n */

{

sum = 0.0;

                       /*          i-1     */

for( j=0; j<i; j++ )   /* Evaluate      Cj  */

                       /*          j=0     */

sum += c[j];

c[i] = 2.0 * sum/i + i;

}

answer = c[n];

free( c );

return answer;

}

Figure 10.45 Evaluating    with a table

10.3.2. Ordering Matrix Multiplications

Suppose we are given four matrices, A, B, C, and D, of dimensions A = 50 X 10, B = 10 X 40, C = 40 X 30, and D = 30 X 5. Although matrix multiplication is not commutative, it is associative, which means that the matrix product ABCD can be parenthesized, and thus evaluated, in any order. The obvious way to multiply two matrices of dimensions p X q and q X r, respectively, uses pqr scalar multiplications. (Using a theoretically superior algorithm such as Strassen''s algorithm does not significantly alter the problem we will consider, so we will assume this performance bound.) What is the best way to perform the three matrix multiplications required to compute ABCD?

In the case of four matrices, it is simple to solve the problem by exhaustive search, since there are only five ways to order the multiplications. We evaluate each case below:

       (A((BC)D)): Evaluating BC requires 10 X 40 X 30 = 12,000 multiplications. Evaluating (BC)D requires the 12,000 multiplications to compute BC, plus an additional 10 X 30 X 5 = 1,500 multiplications, for a total of 13,500. Evaluating (A((BC)D) requires 13,500 multiplications for (BC)D, plus an additional 50 X 10 X 5 = 2,500 multiplications, for a grand total of 16,000 multiplications.

       (A(B(CD))): Evaluating CD requires 40 X 30 X 5 = 6,000 multiplications. Evaluating B(CD) requires 6,000 multiplications to compute CD, plus an additional 10 X 40 X 5 = 2,000 multiplications, for a total of 8,000. Evaluating (A(B(CD)) requires 8,000 multiplications for B(CD), plus an additional 50 X 10 X 5 = 2,500 multiplications, for a grand total of 10,500 multiplications.

       ((AB)(CD)): Evaluating CD requires 40 X 30 X 5 = 6,000 multiplications. Evaluating AB requires 50 X 10 X 40 = 20,000 multiplications. Evaluating ((AB)(CD)) requires 6,000 multiplications for CD, 20,000 multiplications for AB, plus an additional 50 X 40 X 5 = 10,000 multiplications for a grand total of 36,000 multiplications.

       (((AB)C)D): Evaluating AB requires 50 X 10 X 40 = 20,000 multiplications. Evaluating (AB)C requires the 20,000 multiplications to compute AB, plus an additional 50 X 40 X 30 = 60,000 multiplications, for a total of 80,000. Evaluating (((AB)C)D) requires 80,000 multiplications for (AB)C, plus an additional 50 X 30 X 5 = 7,500 multiplications, for a grand total of 87,500 multiplications.

       ((A(BC))D): Evaluating BC requires 10 X 40 X 30 = 12,000 multiplications. Evaluating A(BC) requires the 12,000 multiplications to compute BC, plus an additional 50 X 10 X 30 = 15,000 multiplications, for a total of 27,000. Evaluating ((A(BC))D) requires 27,000 multiplications for A(BC), plus an additional 50 X 30 X 5 = 7,500 multiplications, for a grand total of 34,500 multiplications.

The calculations show that the best ordering uses roughly one-ninth the number of multiplications as the worst ordering. Thus, it might be worthwhile to perform a few calculations to determine the optimal ordering. Unfortunately, none of the  obvious greedy strategies seems to work. Moreover, the number of possible orderings grows quickly. Suppose we define T(n) to be this number. Then T(1) = T(2) = 1, T(3) = 2, and T(4) = 5, as we have seen. In general,

To see this, suppose that the matrices are A1, A2, . . . , A n, and the last multiplication performed is (A1A2. . . A i)(A i+1A i+2 . . . A n). Then there are T(i) ways to compute (A1A2   [image]    [image]    [image]  A i) and T(n - i) ways to compute (A i+1A i+2   [image]    [image]    [image]  A n). Thus, there are T(i)T(n - i) ways to compute (A1A2   [image]    [image]    [image]  A i) (A i+1A i+2   [image]    [image]    [image]  A n) for each possible i.

The solution of this recurrence is the well-known Catalan numbers, which grow exponentially. Thus, for large n, an exhaustive search through all possible orderings is useless. Nevertheless, this counting argument provides a basis for a solution that is substantially better than exponential. Let ci be the number of columns in matrix A i for 1   [image]  i   [image]  n. Then A i has ci-1 rows, since otherwise the multiplications are not valid. We will define c0 to be the number of rows in the first matrix, A1.

Suppose mLeft,Right is the number of multiplications required to multiply A Left A Left+1   [image]    [image]    [image]  A Right-1A Right. For consistency, mLeft,Left = 0. Suppose the last multiplication is (A Left...A i)(A i+1   [image]    [image]    [image]  A Right), where Left   [image]  i   [image]  Right. Then the number of multiplications used is mLeft,i + mi+1,Right + cLeft-1cicRight. These three terms represent the multiplications required to compute (A Left   [image]    [image]    [image]  A i),(A i+1   [image]    [image]    [image] A Right), and their product, respectively.

If we define MLeft,Right to be the number of multiplications required in an optimal ordering, then, if Left < Right,

This equation implies that if we have an optimal multiplication arrangement of A Left   [image]    [image]    [image]  A Right, the subproblems A Left   [image]    [image]    [image]  A i and A i+1  [image]    [image]    [image]  A Right cannot be performed suboptimally. This should be clear, since otherwise we could improve the entire result by replacing the suboptimal computation by an optimal computation.

The formula translates directly to a recursive program, but, as we have seen in the last section, such a program would be blatantly inefficient. However, since there are only approximately n2/2 values of MLeft,Right that ever need to be computed, it is clear that a table can be used to store these values. Further examination shows that if Right - Left = k, then the only values Mx,y that are needed in the computation of MLeft,Right satisfy y - x < k. This tells us the order in which we need to compute the table.

If we want to print out the actual ordering of the multiplications in addition to the final answer M1,n, then we can use the ideas from the shortest-path algorithms in Chapter 9. Whenever we make a change to MLeft,Right, we record the value of i that is responsible. This gives the simple program shown in Figure 10.46.

Although the emphasis of this chapter is not coding, it is worth noting that many programmers tend to shorten variable names to a single letter. c, i, and k are used as single-letter variables because this agrees with the names we have used in the description of the algorithm, which is very mathematical. However, it is generally best to avoid l as a variable name, because "l" looks too much like 1 and can make for very difficult debugging if you make a transcription error.

Returning to the algorithmic issues, this program contains a triply nested loop and is easily seen to run in O(n3) time. The references describe a faster algorithm, but since the time to perform the actual matrix multiplication is still likely to be much larger than the time to compute the optimal ordering, this algorithm is still quite practical.

/* Compute optimal ordering of matrix multiplication */

/* c contains number of columns for each of the n matrices */

/* c[0] is the number of rows in matrix 1 */

/* Minimum number of multiplications is left in M[1][n] */

/* Actual ordering can be computed via */

/* another procedure using last_change */

/* M and last_change are indexed starting at 1, instead of zero */

void

opt_matrix( int c[], unsigned int n, two_d_array M,

two_d_array last_change)

{

int i, k, Left, Right, this_M;

for( Left = 1; Left <= n; Left++ )

M[Left][Left] = 0;

for( k = 1; k < n; k++) /* k is Right-Left */

for( Left = 1; Left <= n-k; Left++ )

{   /* for each position */

Right = Left + k;

M[Left][Right] = INT_MAX;

for( i = Left; i < Right; i++ )

{

this_M = M[Left][i] + M[i+1][Right]

+ c[Left-1] * c[i] * c[Right];

if( this_M < M[Left][Right] ) /* Update min */

{

M[Left][Right] = this_M;

last_change[Left][Right] = i;

}

}

}

}

Figure 10.46 Program to find optimal ordering of Matrix Multiplications

10.3.3. Optimal Binary Search Tree

Our second dynamic programming example considers the following input: We are given a list of words, w1, w2,..., wn, and fixed probabilities p1, p2, . . . , pn of their occurrence. The problem is to arrange these words in a binary search tree in a way that minimizes the expected total access time. In a binary search tree, the number of comparisons needed to access an element at depth d is d + 1, so if wi is placed at depth di, then we want to minimize   [image].

As an example, Figure 10.47 shows seven words along with their probability of occurrence in some context. Figure 10.48 shows three possible binary search trees. Their searching costs are shown in Figure 10.49.

The first tree was formed using a greedy strategy. The word with the highest probability of being accessed was placed at the root. The left and right subtrees were then formed recursively. The second tree is the perfectly balanced search tree. Neither of these trees is optimal, as demonstrated by the existence of the third tree. From this we can see that neither of the obvious solutions works.

This is initially surprising, since the problem appears to be very similar to the construction of a Huffman encoding tree, which, as we have already seen, can be solved by a greedy algorithm. Construction of an optimal binary search tree is harder, because the data is not constrained to appear only at the leaves, and also because the tree must satisfy the binary search tree property.

A dynamic programming solution follows from two observations. Once again, suppose we are trying to place the (sorted) words wLeft, wLeft+1, . . . , wRight-1, wRight into a binary search tree. Suppose the optimal binary search tree has wi as the root, where Left   [image]  i   [image]  Right. Then the left subtree must contain wLeft, . . . ,wi-1, and the right subtree must contain wi+1 . . . ,wRight (by the binary search tree property). Further, both of these subtrees must also be optimal, since otherwise they could be replaced by optimal subtrees, which would give a better solution for wLeft . . . , wRight. Thus, we can write a formula for the cost CLeft,Right of an optimal binary search tree. Figure 10.50 may be helpful.

If Left > Right, then the cost of the tree is 0; this is the NULL case, which we always have for binary search trees. Otherwise, the root costs pi. The left subtree has a cost of CLeft,i-1, relative to its root, and the right subtree has a cost of Ci+l,Right relative to its root. As Figure 10.50 shows, each node in these subtrees is one level deeper from wi than from their respective roots, so we must add   [image]. This gives the formula

From this equation, it is straightforward to write a program to compute the cost of the optimal binary search tree. As usual, the actual search tree can be maintained by saving the value of i that minimizes CLeft,Right. The standard recursive routine can be used to print the actual tree.

Word  Probability

-----------------

  a      0.22

 am      0.18

and      0.20

egg      0.05

 if      0.25

the      0.02

two      0.08

Figure 10.47 Sample input for optimal binary search tree problem

Figure 10.48 Three possible binary search trees for data in previous table

    Input              Tree #1         Tree #2          Tree #3

-----------------------------------------------------------------

Word  Probability    Access Cost     Access Cost      Access Cost

 wi       pi       Once  Sequence  Once  Sequence  Once  Sequence

-----------------------------------------------------------------

  a      0.22       2      0.44      3     0.66     2      0.44

 am      0.18       4      0.72      2     0.36     3      0.54

and      0.20       3      0.60      3     0.60     1      0.20

egg      0.05       4      0.20      1     0.05     3      0.15

 if      0.25       1      0.25      3     0.75     2      0.50

the      0.02       3      0.06      2     0.04     4      0.08

two      0.08       2      0.16      3     0.24     3      0.24

-----------------------------------------------------------------

Totals   1.00              2.43            2.70            2.15

Figure 10.49 Comparison of the three binary search trees

Figure 10.50 Structure of an optimal binary search tree

Figure 10.51 shows the table that will be produced by the algorithm. For each subrange of words, the cost and root of the optimal binary search tree are maintained. The bottommost entry, of course, computes the optimal binary search tree for the entire set of words in the input. The optimal tree is the third tree shown in Fig. 10.48.

The precise computation for the optimal binary search tree for a particular subrange, namely am..if, is shown in Figure 10.52. It is obtained by computing the minimum-cost tree obtained by placing am, and, egg, and if at the root. For instance, when and is placed at the root, the left subtree contains am..am (of cost 0.18, via previous calculation), the right subtree contains egg..if (of cost 0.35), and   [image], for a total cost of 1.21.

Figure 10.51 Computation of the optimal binary search tree for sample input

Figure 10.52 Computation of table entry (1.21, and) for am..if

The running time of this algorithm is O(n3), because when it is implemented, we obtain a triple loop. An O(n2) algorithm for the problem is sketched in the exercises.

10.3.4. All-Pairs Shortest Path

Our third and final dynamic programming application is an algorithm to compute shortest weighted paths between every pair of points in a directed graph G = (V, E). In Chapter 9, we saw an algorithm for the single-source shortest-path problem, which finds the shortest path from some arbitrary vertex s to all others. That algorithm (Dijkstra's) runs in O(  [image] V  [image] 2) time on dense graphs, but substantially faster on sparse graphs. We will give a short algorithm to solve the all-pairs problem for dense graphs. The running time of the algorithm is O(  [image] V  [image] 3), which is not an asymptotic improvement over   [image] V  [image]  iterations of Dijkstra's algorithm but could be faster on a very dense graph, because its loops are tighter. The algorithm also performs correctly if there are negative edge costs, but no negative-cost cycles; Dijkstra's algorithm fails in this case.

Let us recall the important details of Dijkstra's algorithm (the reader may wish to review Section 9.3). Dijkstra's algorithm starts at a vertex s and works in stages. Each vertex in the graph is eventually selected as an intermediate vertex. If the current selected vertex is v, then for each w   [image]  V, we set dw = min(dw, dv + cv,w). This formula says that the best distance to w (from s) is either the previously known distance to w from s, or the result of going from s to v (optimally) and then directly from v to w.

Dijkstra's algorithm provides the idea for the dynamic programming algorithm: we select the vertices in sequential order. We will define Dk,i,j to be the weight of the shortest path from vi to vj that uses only v1, v2, . . . ,vk as intermediates. By this definition, D0,i,j = ci,j, where ci,j is   [image]  if (vi, vj) is not an edge in the graph. Also, by definition, D|V|,i,j is the shortest path from vi to vj in the graph.

/* Compute All-Shortest Paths */

/* A[] contains the adjacency matrix */

/* with A[i][i] presumed to be zero */

/* D[] contains the values of shortest path */

/* |V | is the number of vertices */

/* A negative cycle exists iff */

/* d[i][j] is set to a negative value at line 9 */

/* Actual Path can be computed via another procedure using path */

/* All arrays are indexed starting at 0 */

void

all_pairs( two_d_array A, two_d_array D, two_d_array path )

{

int i, j, k;

/*1*/        for( i = 0; i < |V |; i++ ) /* Initialize D and path */

/*2*/               for( j = 0; j < |V |; j++ )

{

/*3*/                  D[i][j] = A[i][j];

/*4*/                  path[i][j] = NOT_A_VERTEX;

}

/*5*/        for( k = 0; k < |v |; k++ )

/* Consider each vertex as an intermediate */

/*6*/        for( i = 0; i < |V |; i++ )

/*7*/                  for( j = 0; j < |V |; j++ )

/*8*/                       if( d[i][k] + d[k][j] < d[i][j] )

/*update min */

{

/*9*/                            d[i][j] = d[i][k] + d[k][j];

/*10*/                           path[i][j] = k;

}

}

Figure 10.53 All-pairs shortest path

As Figure 10.53 shows, when k > 0 we can write a simple formula for Dk,i,j. The shortest path from vi to vj that uses only v1, v2, . . . ,vk as intermediates is the shortest path that either does not use vk as an intermediate at all, or consists of the merging of the two paths vi   [image] vk and vk   [image]  vj, each of which uses only the first k - 1 vertices as intermediates. This leads to the formula

Dk,i,j = min{Dk - 1,i,j, Dk - 1,i,k + Dk - 1,k,j}

The time requirement is once again O(|V|3). Unlike the two previous dynamic programming examples, this time bound has not been substantially lowered by another approach. Because the kth stage depends only on the (k - 1)st stage, it appears that only two |V| X |V| matrices need to be maintained.

However, using k as an intermediate vertex on a path that starts or finishes with k does not improve the result unless there is a negative cycle. Thus, only one matrix is necessary, because Dk-1,i,k = Dk,i,k and Dk-1,k,j = Dk,k,j, which implies that none of the terms on the right change values and need to be saved. This observation leads to the simple program in Figure 10.53, which numbers vertices starting at zero to conform with C's conventions.

On a complete graph, where every pair of vertices is connected (in both directions), this algorithm is almost certain to be faster than |V| iterations of Dijkstra's algorithm, because the loops are so tight. Lines 1 through 4 can be executed in parallel, as can lines 6 through 10. Thus, this algorithm seems to be well-suited for parallel computation.

Dynamic programming is a powerful algorithm design technique, which provides a starting point for a solution. It is essentially the divide and conquer paradigm of solving simpler problems first, with the important difference being that the simpler problems are not a clear division of the original. Because subproblems are repeatedly solved, it is important to record their solutions in a table rather than recompute them. In some cases, the solution can be improved (although it is certainly not always obvious and frequently difficult), and in other cases, the dynamic programming technique is the best approach known.

In some sense, if you have seen one dynamic programming problem, you have seen them all. More examples of dynamic programming can be found in the exercises and references.

10.4. Randomized Algorithms

Suppose you are a professor who is giving weekly programming assignments. You want to make sure that the students are doing their own programs or, at the very least, understand the code they are submitting. One solution is to give a quiz on the day that each program is due. On the other hand, these quizzes take time out of class, so it might only be practical to do this for roughly half of the programs. Your problem is to decide when to give the quizzes.

Of course, if the quizzes are announced in advance, that could be interpreted as an implicit license to cheat for the 50 percent of the programs that will not get a quiz. One could adopt the unannounced strategy of giving quizzes on alternate programs, but students would figure out the strategy before too long. Another possibility is to give quizzes on what seems like the important programs, but this would likely lead to similar quiz patterns from semester to semester. Student grapevines being what they are, this strategy would probably be worthless after a semester.

One method that seems to eliminate these problems is to use a coin. A quiz is made for every program (making quizzes is not nearly as time-consuming as grading them), and at the start of class, the professor will flip a coin to decide whether the quiz is to be given. This way, it is impossible to know before class whether or not the quiz will occur, and these patterns do not repeat from semester to semester. Thus, the students will have to expect that a quiz will occur with 50 percent probability, regardless of previous quiz patterns. The disadvantage is that it is possible that there is no quiz for an entire semester. This is not a likely occurrence, unless the coin is suspect. Each semester, the expected number of quizzes is half the number of programs, and with high probability, the number of quizzes will not deviate much from this.

This example illustrates what we call randomized algorithms. At least once during the algorithm, a random number is used to make a decision. The running time of the algorithm depends not only on the particular input, but also on the random numbers that occur.

The worst-case running time of a randomized algorithm is almost always the same as the worst-case running time of the nonrandomized algorithm. The important difference is that a good randomized algorithm has no bad inputs, but only bad random numbers (relative to the particular input). This may seem like only a philosophical difference, but actually it is quite important, as the following example shows.

Consider two variants of quicksort. Variant A uses the first element as pivot, while variant B uses a randomly chosen element as pivot. In both cases, the worst-case running time is   [image] (n2), because it is possible at each step that the largest element is chosen as pivot. The difference between these worst cases is that there is a particular input that can always be presented to variant A to cause the bad running time. Variant A will run in   [image] (n2) time every single time it is given an already sorted list. If variant B is presented with the same input twice, it will have two different running times, depending on what random numbers occur.

Throughout the text, in our calculations of running times, we have assumed that all inputs are equally likely. This is not true, because nearly sorted input, for instance, occurs much more often than is statistically expected, and this causes problems, particularly for quicksort and binary search trees. By using a randomized algorithm, the particular input is no longer important. The random numbers are important, and we can get an expected running time, where we now average over all possible random numbers instead of over all possible inputs. Using quicksort with a random pivot gives an O(n log n)-expected-time algorithm. This means that for any input, including already-sorted input, the running time is expected to be O(n log n), based on the statistics of random numbers. An expected running time bound is somewhat stronger than an average-case bound but, of course, is weaker than the corresponding worst-case bound. On the other hand, as we saw in the selection problem, solutions that obtain the worst-case bound are frequently not as practical as their average-case counterparts. Randomized algorithms usually are.

In this section we will examine two uses of randomization. First, we will see a novel scheme for supporting the binary search tree operations in O(log n) expected time. Once again, this means that there are no bad inputs, just bad random numbers. From a theoretical point of view, this is not terribly exciting, since balanced search trees achieve this bound in the worst case. Nevertheless, the use of randomization leads to relatively simple algorithms for searching, inserting, and especially deleting.

Our second application is a randomized algorithm to test the primality of large numbers. No efficient polynomial-time nonrandomized algorithms are known for this problem. The algorithm we present runs quickly but occasionally makes an error. The probability of error can, however, be made negligibly small.

10.4.1. Random Number Generators

Since our algorithms require random numbers, we must have a method to generate them. Actually, true randomness is virtually impossible to do on a computer, since these numbers will depend on the algorithm, and thus cannot possibly be random. Generally, it suffices to produce pseudorandom numbers, which are numbers that appear to be random. Random numbers have many known statistical properties; pseudorandom numbers satisfy most of these properties. Surprisingly, this too is much easier said than done.

Suppose we only need to flip a coin; thus, we must generate a 0 or 1 randomly. One way to do this is to examine the system clock. The clock might record time as an integer that counts the number of seconds since January 1, 1970.* We could then use the lowest bit. The problem is that this does not work well if a sequence of random numbers is needed. One second is a long time, and the clock might not change at all while the program is running. Even if the time were recorded in units of microseconds, if the program were running by itself the sequence of numbers that would be generated would be far from random, since the time between calls to the generator would be essentially identical on every program invocation. We see, then, that what is really needed is a sequence of random numbers.ç These numbers should appear independent. If a coin is flipped and heads appears, the next coin flip should still be equally likely to come up heads or tails.

*UNIX does this.

çWe will use random in place of pseudorandom in the rest of this section.

The standard method to generate random numbers is the linear congruential generator, which was first described by Lehmer in 1951. Numbers x1, x2, . . . are generated satisfying

xi + 1 = axi mod m.

To start the sequence, some value of x0 must be given. This value is known as the seed. If x0 = 0, then the sequence is far from random, but if a and m are correctly chosen, then any other 1   [image]  x0 < m is equally valid. If m is prime, then xi is never 0. As an example, if m = 11, a = 7, and x0 = 1, then the numbers generated are

7, 5, 2, 3, 10, 4, 6, 9, 8, 1, 7, 5, 2, . . .

Notice that after m - 1 = 10 numbers, the sequence repeats. Thus, this sequence has a period of m -1, which is as large as possible (by the pigeonhole principle). If m is prime, there are always choices of a that give a full period of m - 1. Some choices of a do not; if a = 5 and x0 = 1, the sequence has a short period of 5.

5, 3, 4, 9, 1, 5, 3, 4, . . .

Obviously, if m is chosen to be a large, 31-bit prime, the period should be significantly large for most applications. Lehmer suggested the use of the 31-bit prime m = 231 - 1 = 2,147,483,647. For this prime, a = 75 = 16,807 is one of the many values that gives a full-period generator. Its use has been well studied and is recommended by experts in the field. We will see later that with random number generators, tinkering usually means breaking, so one is well advised to stick with this formula until told otherwise.

This seems like a simple routine to implement. Generally, a global variable is used to hold the current value in the sequence of x's. This is the rare case where a global variable is useful. This global variable is initialized by some routine. When debugging a program that uses random numbers, it is probably best to set x0 = 1, so that the same random sequence occurs all the time. When the program seems to work, either the system clock can be used or the user can be asked to input a value for the seed.

It is also common to return a random real number in the open interval (0, 1) (0 and 1 are not possible values); this can be done by dividing by m. From this, a random number in any closed interval [a, b] can be computed by normalizing. This yields the "obvious" routine in Figure 10.54 which, unfortunately, works on few machines.

The problem with this routine is that the multiplication could overflow; although this is not an error, it affects the result and thus the pseudo-randomness. Schrage gave a procedure in which all of the calculations can be done on a 32-bit machine without overflow. We compute the quotient and remainder of m/a and define these as q and r, respectively. In our case, q = 127,773, r = 2,836, and r < q. We have

unsigned int seed;        /* global variable */

#define a     16807            /* 7^5 */

#define m     2147483647       /* 2^31 - 1 */

double

random( void )

{

seed = ( a * seed ) % m;

return( ( (double) seed ) / m );

}

Figure 10.54 Random number generator that does not work

Since   [image], we can replace the leading axi and obtain

Since m = aq + r, it follows that aq - m = -r. Thus, we obtain

The term   [image]  is either 0 or 1, because both terms are integers and their difference lies between 0 and 1. Thus, we have

A quick check shows that because r < q, all the remaining terms can be calculated without overflow (this is one of the reasons for chosing a = 75). Furthermore,   [image] (xi) = 1 only if the remaining terms evaluate to less than zero. Thus   [image] (xi) does not need to be explicitly computed but can be determined by a simple test. This leads to the program in Figure 10.55.

This program works as long as INT_MAX   [image]  231 - 1. One might be tempted to assume that all machines have a random number generator at least as good as the one in Figure 10.55 in their standard library. Sadly, this is not true. Many libraries have generators based on the function

xi+1 = (axi + c) mod 2b

where b is chosen to match the number of bits in the machine's integer, and c is odd. These libraries also return xi, instead of a value between 0 and 1. Unfortunately, these generators always produce values of xi that alternate between even and odd--hardly a desirable property. Indeed, the lower k bits cycle with period 2k (at best). Many other random number generators have much smaller cycles than the one provided in Figure 10.55. These are not suitable for the case where long sequences of random numbers are needed. Finally, it may seem that we can get a better random number generator by adding a constant to the equation. For instance, it seems that

xi+1 = (16807xi + 1) mod (231 - 1)

would somehow be even more random. This illustrates how fragile these generators are.

[16807(1319592028) + 1] mod (231-1) = 1319592028,

so if the seed is 1,319,592,028, the generator gets stuck in a cycle of period 1.

unsigned int seed;      /* global variable */

#define a      16807          /* 7^5 */

#define m      2147483647     /* 2^31 - 1*/

#define q      127773         /* m/a */

#define r      2836           /* m%a */

double

random( void )

{

int tmp_seed;

tmp_seed = a * ( seed % q ) - r * (seed / q );

if( tmp_seed >= 0)

seed = tmp_seed;

else

seed = tmp_seed + m;

return( ( (double) seed ) / m );

}

Figure 10.55 Random number generator that works on 32 bit machines

10.4.2. Skip Lists

Our first use of randomization is a data structure that supports both searching and insertion in O(log n) expected time. As mentioned in the introduction to this section, this means that the running time for each operation on any input sequence has expected value O(log n), where the expectation is based on the random number generator. It is possible to add deletion and all the operations that involve ordering and obtain expected time bounds that match the average time bounds of binary search trees.

The simplest possible data structure to support searching is the linked list. Figure 10.56 shows a simple linked list. The time to perform a search is proportional to the number of nodes that have to be examined, which is at most n.

Figure 10.57 shows a linked list in which every other node has an additional pointer to the node two ahead of it in the list. Because of this, at most   [image] n/2  [image]  + 1 nodes are examined in the worst case.

We can extend this idea and obtain Figure 10.58. Here, every fourth node has a pointer to the node four ahead. Only   [image] n/4  [image]  + 2 nodes are examined.

The limiting case of this argument is shown in Figure 10.59. Every 2ith node has a pointer to the node 2i ahead of it. The total number of pointers has only doubled, but now at most   [image] log n  [image]  nodes are examined during a search. It is not hard to see that the total time spent for a search is O(log n), because the search consists of either advancing to a new node or dropping to a lower pointer in the same node. Each of these steps consumes at most O(log n) total time during a search. Notice that the search in this data structure is essentially a binary search.

Figure 10.56 Simple linked list

Figure 10.57 Linked list with pointers to two cells ahead

Figure 10.58 Linked list with pointers to four cells ahead

Figure 10.59 Linked list with pointers to 2i cells ahead

Figure 10.60 A skip list

The problem with this data structure is that it is much too rigid to allow efficient insertion. The key to making this data structure usable is to relax the structure conditions slightly. We define a level k node to be a node that has k pointers. As Figure 10.59 shows, the ith pointer in any level k node (k   [image]  i) points to the next node with at least i levels. This is an easy property to maintain; however, Figure 10.59 shows a more restrictive property than this. We thus drop the restriction that the ith pointer points to the node 2i ahead, and we replace it with the less restrictive condition above.

When it comes time to insert a new element, we allocate a new node for it. We must at this point decide what level the node should be. Examining Figure 10.59, we find that roughly half the nodes are level 1 nodes, roughly a quarter are level 2, and, in general, approximately 1/2i nodes are level i. We choose the level of the node randomly, in accordance with this probability distribution. The easiest way to do this is to flip a coin until a head occurs and use the total number of flips as the node level. Figure 10.60 shows a typical skip list.

Given this, the skip list algorithms are simple to describe. To perform a find, we start at the highest pointer at the header. We traverse along this level until we find that the next node is larger than the one we are looking for (or ). When this occurs, we go to the next lower level and continue the strategy. When progress is stopped at level 1, either we are in front of the node we are looking for, or it is not in the list. To perform an insert, we proceed as in a find, and keep track of each point where we switch to a lower level. The new node, whose level is determined randomly, is then spliced into the list. This operation is shown in Figure 10.61.

A cursory analysis shows that since the expected number of nodes at each level is unchanged from the original (nonrandomized) algorithm, the total amount of work that is expected to be performed traversing to nodes on the same level is unchanged. This tells us that these operations have O(log n) expected costs. Of course, a more formal proof is required, but it is not much different from this.

Skip lists are similar to hash tables, in that they require an estimate of the number of elements that will be in the list (so that the number of levels can be determined). If an estimate is not available, we can assume a large number or use a technique similar to rehashing. Experiments have shown that skip lists are as efficient as many balanced search tree implementations and are certainly much simpler to implement in many languages.

10.4.3. Primality Testing

In this section we examine the problem of determining whether or not a large number is prime. As was mentioned at the end of Chapter 2, some cryptography schemes depend on the difficulty of factoring a large, 200-digit number into two 100-digit primes. In order to implement this scheme, we need a method of generating these two primes. The problem is of major theoretical interest, because nobody now knows how to test whether a d-digit number n is prime in time polynomial in d. For instance, the obvious method of testing for the divisibility by odd numbers from 3 to   [image]  requires roughly   [image]  divisions, which is about 2d/2. On the other hand, this problem is not thought to be NP-complete; thus, it is one of the few problems on the fringe--its complexity is unknown at the time of this writing.

Figure 10.61 Before and after an insertion

In this chapter, we will give a polynomial-time algorithm that can test for primality. If the algorithm declares that the number is not prime, we can be certain that the number is not prime. If the algorithm declares that the number is prime, then, with high probability but not 100 percent certainty, the number is prime. The error probability does not depend on the particular number that is being tested but instead depends on random choices made by the algorithm. Thus, this algorithm occasionally makes a mistake, but we will see that the error ratio can be made arbitrarily negligible.

The key to the algorithm is a well-known theorem due to Fermat.

THEOREM 10.10.

Fermat's Lesser Theorem: If p is prime, and 0 < a < p, then ap-1   [image]  1(mod p).

PROOF:

A proof of this theorem can be found in any textbook on number theory.

For instance, since 67 is prime, 266   [image]  1(mod 67). This suggests an algorithm to test whether a number n is prime. Merely check whether 2n-1   [image]  1(mod n). If   [image]  (mod n), then we can be certain that n is not prime. On the other hand, if the equality holds, then n is probably prime. For instance, the smallest n that satisfies 2n-1   [image]  1(mod n) but is not prime is n = 341.

This algorithm will occasionally make errors, but the problem is that it will always make the same errors. Put another way, there is a fixed set of n for which it does not work. We can attempt to randomize the algorithm as follows: Pick 1 < a < n - 1 at random. If an-1   [image]  1(mod n), declare that n is probably prime, otherwise declare that n is definitely not prime. If n = 341, and a = 3, we find that 3340   [image]  56(mod 341). Thus, if the algorithm happens to choose a = 3, it will get the correct answer for n = 341.

Although this seems to work, there are numbers that fool even this algorithm for most choices of a. One such set of numbers is known as the Carmichael numbers. These are not prime but satisfy an-1   [image]  1(mod n) for all 0 < a < n that are relatively prime to n. The smallest such number is 561. Thus, we need an additional test to improve the chances of not making an error.

In Chapter 7, we proved a theorem related to quadratic probing. A special case of this theorem is the following:

THEOREM 10.11.

If p is prime and 0 < x < p, the only solutions to x2  [image]  1(mod p) are x = 1, p - 1.

PROOF:

x2   [image]  1(mod p) implies that x2 -1   [image]  0(mod p). This implies (x - 1)(x + 1)   [image]  0(mod p). Since p is prime, 0   [image]  x < p, and p must divide either (x - 1) or (x + 1), the theorem follows.

Therefore, if at any point in the computation of an-1mod n we discover a violation of this theorem, we can conclude that n is definitely not prime. If we use power, from Section 2.4.4, we see that there will be several opportunities to apply this test. We modify this routine to perform operations mod n, and apply the test of Theorem 10.11. This strategy is implemented in Figure 10.62. Because power needs to return two pieces of information, we pass the address of these items ( result and what_n_is ) by pointers.

Recall that if test_prime returns DEFINITELY_COMPOSITE, it has proven that n cannot be prime. The proof is nonconstructive, because it gives no method of actually finding the factors. It has been shown that for any (sufficiently large) n, at most (n - 9)/4 values of a fool this algorithm. Thus, if a is chosen at random, and the algorithm answers PROBABLY_PRIME, then the algorithm is correct at least 75 percent of the time. Suppose test_prime is run 50 times. The probability that the algorithm is fooled once is at most 1/4. Thus, the probability that 50 independent random trials fool the algorithm is never more than 1/450 = 2-100. This is actually a very conservative estimate, which holds for only a few choices of n. Even so, one is more likely to see a hardware error than an incorrect claim of primality.

10.5. Backtracking Algorithms

The last algorithm design technique we will examine is backtracking. In many cases, a backtracking algorithm amounts to a clever implementation of exhaustive search, with generally unfavorable performance. This is not always the case, however, and even so, in some cases, the savings over a brute force exhaustive search can be significant. Performance is, of course, relative: An O(n2) algorithm for sorting is pretty bad, but an O(n5) algorithm for the traveling salesman (or any NP-complete) problem would be a landmark result.

A practical example of a backtracking algorithm is the problem of arranging furniture in a new house. There are many possibilities to try, but typically only a few are actually considered. Starting with no arrangement, each piece of furniture is placed in some part of the room. If all the furniture is placed and the owner is happy, then the algorithm terminates. If we reach a point where all subsequent placement of furniture is undesirable, we have to undo the last step and try an alternative. Of course, this might force another undo, and so forth. If we find that we undo all possible first steps, then there is no placement of furniture that is satisfactory. Otherwise, we eventually terminate with a satisfactory arrangement. Notice that although this algorithm is essentially brute force, it does not try all possibilities directly. For instance, arrangements that consider placing the sofa in the kitchen are never tried. Many other bad arrangements are discarded early, because an undesirable subset of the arrangement is detected. The elimination of a large group of possibilities in one step is known as pruning.

We will see two examples of backtracking algorithms. The first is a problem in computational geometry. Our second example shows how computers select moves in games, such as chess and checkers.

10.5.1. The Turnpike Reconstruction Problem

Suppose we are given n points, p1, p2, . . . , pn, located on the x-axis. xi is the x coordinate of pi. Let us further assume that x1 = 0 and the points are given from left to right. These n points determine n(n - 1)/2 (not necessarily unique) distances d1, d2, . . . , dn between every pair of points of the form | xi - xj | (i   [image]  j ). It is clear that if we are given the set of points, it is easy to construct the set of distances in O(n2) time. This set will not be sorted, but if we are willing to settle for an O(n2 log n) time bound, the distances can be sorted, too. The turnpike reconstruction problem is to reconstruct a point set from the distances. This finds applications in physics and molecular biology (see the references for pointers to more specific information). The name derives from the analogy of points to turnpike exits on East Coast highways. Just as factoring seems harder than multiplication, the reconstruction problem seems harder than the construction problem. Nobody has been able to give an algorithm that is guaranteed to work in polynomial time. The algorithm that we will present seems to run in O(n2log n); no counterexample to this conjecture is known, but it is still just that - a conjecture.

enum test_result { PROBABLY_PRIME, DEFINITELY_COMPOSITE };

typedef enum test_result test_result;

/* Compute result = ap mod n. */

/* If at any point x2    1(mod n) is detected with x    1, x    n - 1, */

/* then set what_n_is to DEFINITELY_COMPOSITE */

/* We are assuming very large integers, so this is pseudocode. */

void

power( unsigned int a, unsigned int p, unsigned int n,

unsigned int *result, test_result *what_n_is )

{

unsigned int x;

/*1*/       if( p = 0 )         /* Base case */

/*2*/            *result = 1;

else

{

/*3*/            power( a, p/2, n, &x, what_n_is );

/*4*/            *result = (x * x) % n;

/* Check whether x2    1(mod n), x    1, x    n - 1 */

/*5*/            if( (*result = 1) && (x != 1) && (x != n-1) )

/*6*/                 *what_n_is = DEFINITELY_COMPOSITE;

/* If p is odd, we need one more a */

/*7*/            if( (p % 2) = 1 )

/*8*/                 *result = (*result * a) % n;

}

}

/* test_prime: Test whether n    3 is prime using one value of a */

/* repeat this procedure as many times as needed */

/* for desired error rate */

test_result

test_prime( unsigned int n )

{

unsigned int a, result;

test_result what_n_is;

/*9*/       a = rand_int( 2, n-2 ); /* choose a randomly from 2..n-2 */

/*10*/      what_n_is = PROBABLY_PRIME;

/* Compute an-1 mod n */

/*11*/      power( a, n-1, n, &result, &what_n_is );

/*12*/      if( ( result != 1) | | (what_n_is = DEFINITELY_COMPOSITE) )

/*13*/           return DEFINITELY_COMPOSITE;

else

/*14*/           return PROBABLY_PRIME;

}

Figure 10.62 A probabilistic primality testing algorithm

Of course, given one solution to the problem, an infinite number of others can be constructed by adding an offset to all the points. This is why we insist that the first point is anchored at 0 and that the point set that constitutes a solution is output in nondecreasing order.

Let D be the set of distances, and assume that | D | = m = n(n - 1)/2. As an example, suppose that

D = {1, 2, 2, 2, 3, 3, 3, 4, 5, 5, 5, 6, 7, 8, 10}

Since | D | = 15, we know that n = 6. We start the algorithm by setting x1 = 0. Clearly, x6 = 10, since 10 is the largest element in D. We remove 10 from D. The points that we have placed and the remaining distances are as shown in the following figure.

The largest remaining distance is 8, which means that either x2 = 2 or x5 = 8. By symmetry, we can conclude that the choice is unimportant, since either both choices lead to a solution (which are mirror images of each other), or neither do, so we can set x5 = 8 without affecting the solution. We then remove the distances x6 - x5 = 2 and x5 - x1 = 8 from D, obtaining

The next step is not obvious. Since 7 is the largest value in D, either x4 = 7 or x2 = 3. If x4 = 7, then the distances x6 - 7 = 3 and x5 - 7 = 1 must also be present in D. A quick check shows that indeed they are. On the other hand, if we set x2 = 3, then 3 - x1 = 3 and x5 - 3 = 5 must be present in D. These distances are also in D, so we have no guidance on which choice to make. Thus, we try one and see if it leads to a solution. If it turns out that it does not, we can come back and try the other. Trying the first choice, we set x4 = 7, which leaves

At this point, we have x1 = 0, x4 = 7, x5 = 8, and x6 = 10. Now the largest distance is 6, so either x3 = 6 or x2 = 4. But if x3 = 6, then x4 - x3 = 1, which is impossible, since 1 is no longer in D. On the other hand, if x2 = 4 then x2 - x0 = 4, and x5 - x2 = 4. This is also impossible, since 4 only appears once in D. Thus, this line of reasoning leaves no solution, so we backtrack.

Since x4 = 7 failed to produce a solution, we try x2 = 3. If this also fails, we give up and report no solution. We now have

Once again, we have to choose between x4 = 6 and x3 = 4. x3 = 4 is impossible, because D only has one occurrence of 4, and two would be implied by this choice. x4 = 6 is possible, so we obtaining

The only remaining choice is to assign x3 = 5; this works because it leaves D empty, and so we have a solution.

Figure 10.63 shows a decision tree representing the actions taken to arrive at the solution. Instead of labeling the branches, we have placed the labels in the branches' destination nodes. A node with an asterisk indicates that the points chosen are inconsistent with the given distances; nodes with two asterisks have only impossible nodes as children, and thus represent an incorrect path.

Figure 10.63 Decision tree for the worked turnpike reconstruction example

int

turnpike(int x [], dist_set D , unsigned int n)

{

/*1*/       x[1] = 0;

/*2*/       x[n ] = delete_max(D );

/*3*/       x[n - 1] = delete_max(D );

/*4*/       if(x[n ]-x[n - 1]    D )

{

/*5*/             delete( x[n ]-x[n - 1],D );

/*6*/             return place( x, D, n, 2,n - 2); }

else

/*7*/             return FALSE;

}

Figure 10.64 Turnpike reconstruction algorithm: driver routine (pseudocode)

The pseudocode to implement this algorithm is mostly straightforward. The driving routine, turnpike, is shown in Figure 10.64. It receives the point array x (which need not be initialized), the distance array D, and n.* If a solution is discovered, then TRUE will be returned, the answer will be placed in x, and D will be empty. Otherwise, FALSE will be returned, x will be undefined, and the distance array D will be untouched. The routine sets x1, xn-1, and xn, as described above, alters D, and calls the backtracking algorithm place to place the other points. We presume that a check has already been made to ensure that | D | = n(n -1)/2.

*We have used one-letter variable names, which is generally poor style, for consistency with the worked example. We also, for simplicity, do not give the type of variables.

The more difficult part is the backtracking algorithm, which is shown in Figure 10.65. Like most backtracking algorithms, the most convenient implementation is recursive. We pass the same arguments plus the boundaries Left and Right; xLeft, . . . , xRight are the x coordinates of points that we are trying to place. If D is empty (or Left > Right ), then a solution has been found, and we can return. Otherwise, we first try to place xRight = Dmax. If all the appropriate distances are present (in the correct quantity), then we tentatively place this point, remove these distances, and try to fill from Left to Right- 1. If the distances are not present, or the attempt to fill Left to Right- 1 fails, then we try setting xLeft = xn - dmax, using a similar strategy. If this does not work, then there is no solution; otherwise a solution has been found, and this information is eventually passed back to turnpike by the return statement and x array.

The analysis of the algorithm involves two factors. Suppose lines 9 through 11 and 18 through 20 are never executed. We can maintain D as a balanced binary search (or splay) tree (this would require a code modification, of course). If we never backtrack, there are at most O(n2) operations involving D, such as deletion and the finds implied at lines 4 and 12 to 13. This claim is obvious for deletions, since D has O(n2) elements and no element is ever reinserted. Each call to place uses at most 2n finds, and since place never backtracks in this analysis, there can be at most 2n2 finds. Thus, if there is no backtracking, the running time is O(n2 log n).

Of course, backtracking happens, and if it happens repeatedly, then the performance of the algorithm is affected. No polynomial bound on the amount of backtracking is known, but on the other hand, there are no pathological examples that show that backtracking must occur more than O(1) times. Thus, it is entirely possible that this algorithm is O(n2 log n). Experiments have shown that if the points have integer coordinates distributed uniformly and randomly from [0, Dmax], where Dmax =   [image] (n2), then, almost certainly, at most one backtrack is performed during the entire algorithm.

10.5.2. Games

As our last application, we will consider the strategy that a computer might use to play a strategic game, such as checkers or chess. We will use, as an example, the much simpler game of tic-tac-toe, because it makes the points easier to illustrate.

Tic-tac-toe is, of course, a draw if both sides play optimally. By performing a careful case-by-case analysis, it is not a difficult matter to construct an algorithm that never loses and always wins when presented the opportunity. This can be done, because certain positions are known traps and can be handled by a lookup table. Other strategies, such as taking the center square when it is available, make the analysis simpler. If this is done, then by using a table we can always choose a move based only on the current position. Of course, this strategy requires the programmer, and not the computer, to do most of the thinking.

/* Backtracking algorithm to place the points */

/* x[left]...x[right]. */

/* x[1]...[left-1] and x[right+1]...x[n]

/* are already tentatively placed * /

/* If place returns true,

/* then x[left]...x[right] will have value. */

int

place( int x[ ], dist_set D , unsigned int n, int Left, int Right )

{

int d_max, found = FALSE;

/*1*/       if D is empty then

/*2*/            return TRUE;

/*3*/       d_max = find_max( D );

/* Check if setting x[Right] = d_max is feasible. */

/*4*/       if( |x[ j ]-d_max|    D for all 1    j < Left and Right < j    n )

{

/*5*/            x[Right] = d_max; /* Try x[Right] = d_max */

/*6*/            for( 1    j < Left, Right < j    n )

/*7*/                 delete( |x[j ]-d_max|, D );

/*8*/            found = place( x, D, n, Left, Right-1 );

/*9*/            if( !found ) /* Backtrack */

/*10*/                for( 1    j < Left, Right < j    n) / Undo the deletion */

/*11*/                     insert( |x[j ]-d_max:| D );

}

/* If first attempt failed, try to see if setting */

/* x[Left]=x[n]-d_max is feasible */

/*12*/      if( !found && (|x[n]-d_max-x[j ]|    D

/*13*/           for all 1    j < Left and Right < j      n) )

{

/*14*/           x[Left] = x [n] -d_max;      / * Same logic as before */

/*15*/           for( 1    j < Left, Right < j    n )

/*16*/                delete( |x[n]-d_max -x [j ] |, D );

/*17*/           found = place( x, D, n, Left + 1, Right );

/*18*/      if( !found )     /* Backtrack; undo the deletion */

/*19*/           for( 1    j < Left, Right < j    n )

/*20*/                insert( |x[n]-d_max-x[j ]|, D );

}

/*21*/      return found;

}

Figure 10.65 Turnpike reconstruction algorithm: backtracking steps (pseudocode)

Minimax Strategy

The general strategy is to use an evaluation function to quantify the "goodness" of a position. A position that is a win for a computer might get the value of +1; a draw could get 0; and a position that the computer has lost would get a - 1. A position for which this assignment can be determined by examining the board is known as a terminal position.

If a position is not terminal, the value of the position is determined by  recursively assuming optimal play by both sides. This is known as a minimax strategy, because one player (the human) is trying to minimize the value of the position, while the other player (the computer) is trying to maximize it.

A successor position of P is any position Ps that is reachable from P by playing one move. If the computer is to move when in some position P, it recursively evaluates the value of all the successor positions. The computer chooses the move with the largest value; this is the value of P. To evaluate any successor position Ps, all of Ps's successors are recursively evaluated, and the smallest value is chosen. This smallest value represents the most favorable reply for the human player.

The code in Figure 10.66 makes the computer's strategy more clear. Lines 1 through 4 evaluate immediate wins or draws. If neither of these cases apply, then the position is nonterminal. Recalling that value should contain the maximum of all possible successor positions, line 5 initializes it to the smallest possible value, and the loop in lines 6 through 13 searches for improvements. Each successor position is recursively evaluated in turn by lines 8 through 10. This is recursive, because, as we will see, the procedure find_human_move calls find_comp_move. If the human's response to a move leaves the computer with a more favorable position than that obtained with the previously best computer move, then the value and best_move are updated. Figure 10.67 shows the procedure for the human's move selection. The logic is virtually identical, except that the human player chooses the move that leads to the lowest-valued position. Indeed, it is not difficult to combine these two procedures into one by passing an extra variable, which indicates whose turn it is to move. This does make the code somewhat less readable, so we have stayed with separate routines.

Since these routines must pass back both the value of the position and the best move, we pass the address of two variables that will get this information, by using pointers. The last two parameters now answer the question "WHERE?" instead of "WHAT? "

/* Recursive procedure to find best move for computer */

/* best_move points to a number from 1-9 indicating square. */

/* Possible evaluations satisfy COMP_LOSS < DRAW < COMP_WIN */

/* Complementary procedure find_human_move is below */

/* board_type is an array; thus board can be changed by place ( ) */

void

find_comp_move( board_type board, int *best_move, int *value )

{

int dc, i, response; /* dc means don't care */

/*1*/       if( full_board( board ) )

/*2*/            *value = DRAW;

else

/*3*/       if( immediate_comp_win( board, best_move ) )

/*4*/       *value = COMP_WIN;

else

{

/*5*/            *value = COMP_LOSS;

/*6*/            for( i=1; i<=9; i++ )    /* try each square */

{

/*7*/                 if( is_empty( board, i ) )

{

/*8*/                     place( board, i, COMP );

/*9*/                     find_human_move( board, &dc, &response );

/*10*/                    unplace( board, i ); /* Restore board */

/*11*/                    if( response >* value ) /* Update best move */

{

/*12*/                         *value = response;

/*13*/                         *best_move = i;

}

}

}

}

}

Figure 10.66 Minimax tic-tac-toe algorithm: computer selection

void

find_human_move( board_type board, int *best_move, int *value )

{

int dc, i, response; /* dc means don't care */

/*1*/       if( full_board( board ) )

/*2*/            *value = DRAW;

else

/*3*/       if( immediate_human_win( board, best_move ) )

/*4*/            *value = COMP_LOSS;

else

{

/*5*/            *value = COMP_WIN;

/*6*/            for( i=1; i<=9; i++ ) /* try each square */

{

/*7*/                 if( is_empty( board, i ) )

{

/*8*/                      place( board, i, HUMAN );

/*9*/                      find_comp_move( board, &dc, &response );

/*10*/                     unplace( board, i ); /* Restore board */

/*11*/                     if( response < * value ) /* Update best move */

{

/*12*/                          *value = response;

/*13*/                          *best_move = i;

}

}

}

}

}

Figure 10.67 Min-max tic-tac-toe algorithm: human selection

As an example, in Figure 10.66, best_move contains the address where the best move can be placed. find_comp_move can examine or alter the data at that address by accessing *best_move. Line 9 shows how the calling routine should behave. Since the caller has two integers prepared to store the data, and find_human_move only wants the addresses of these two integers, the address operator (&) is used.

If the & operator is not used at line 9, and both dc and response are zero (which would be typical of uninitialized data), the find_human_move will try to place its best move and position value in memory location zero. Of course, this is not what was intended, and will almost certainly result in a program crash (try it!). This is the most common error when using the scanf family of library routines.

We leave supporting routines as an exercise. The most costly computation is the case where the computer is asked to pick the opening move. Since at this stage the game is a forced draw, the computer selects square 1.* A total of 97,162 positions were examined, and the calculation took 2.5 seconds on a VAX 8800. No attempt was made to optimize the code. When the computer moves second, the number of positions examined is 5,185 if the human selects the center square, 9,761 when a corner square is selected, and 13,233 when a noncorner edge square is selected.

*We numbered the squares starting from the top left and moving right. However, this is only important for the supporting routines.

For more complex games, such as checkers and chess, it is obviously  infeasible to search all the way to the terminal nodes.ç In this case, we have to stop the search after a certain depth of recursion is reached. The nodes where the  recursion is stopped become terminal nodes. These terminal nodes are evaluated with a function that estimates the value of the position. For instance, in a chess program, the evaluation function measures such variables as the relative amount and strength of pieces and positional factors. The evaluation function is crucial for success, because the computer's move selection is based on maximizing this function. The best computer chess programs have surprisingly sophisticated evaluation functions.

çIt is estimated that if this search were conducted for chess, at least 10100 positions would be examined for the first move. Even if the improvements described later in this section were incorporated, this number could not be reduced to a practical level.

Nevertheless, for computer chess, the single most important factor seems to be number of moves of look-ahead the program is capable of. This is sometimes known as ply; it is equal to the depth of the recursion. To implement this, an extra parameter is given to the search routines.

The basic method to increase the look-ahead factor in game programs is to come up with methods that evaluate fewer nodes without losing any information. One method which we have already seen is to use a table to keep track of all positions that have been evaluated. For instance, in the course of searching for the first move, the program will examine the positions in Figure 10.68. If the values of the positions are saved, the second occurrence of a position need not be recomputed; it essentially becomes a terminal position. The data structure that records this is known as a transposition table; it is almost always implemented by hashing. In many cases, this can save considerable computation. For instance, in a chess endgame, where there are relatively few pieces, the time savings can allow a search to go several levels deeper.

  -   Pruning

Probably the most significant improvement one can obtain in general is known as   [image] -  [image]  pruning. Figure 10.69 shows the trace of the recursive calls used to evaluate some hypothetical position in a hypothetical game. This is commonly referred to as a game tree. (We have avoided the use of this term until now, because it is somewhat misleading: no tree is actually constructed by the algorithm. The game tree is just an abstract concept.) The value of the game tree is 44.

Figure 10.68 Two searches that arrive at identical position

Figure 10.70 shows the evaluation of the same game tree, with several  unevaluated nodes. Almost half of the terminal nodes have not been checked. We show that evaluating them would not change the value at the root.

First, consider node D. Figure 10.71 shows the information that has been gathered when it is time to evaluate D. At this point, we are still in find_human_move and are contemplating a call to find_comp_move on D. However, we already know that find_human_move will return at most 40, since it is a min node. On the other hand, its max node parent has already found a sequence that guarantees 44. Nothing that D does can possibly increase this value. Therefore, D does not need to be evaluated. This pruning of the tree is known as   [image]  pruning. An identical situation occurs at node B. To implement   [image]  pruning, get_comp_move passes its tentative maximum (  [image] ) to get_human_move. If the tentative minimum of get_human_move falls below this value, then get_human_move returns immediately.

A similar thing happens at nodes A and C. This time, we are in the middle of a find_comp_move and are about to make a call to find_human_move to evaluate C. Figure 10.72 shows the situation that is encountered at node C. However, the sfind_human_move, at the min level, which has called find_comp_move, has already determined that it can force a value of at most 44 (recall that low values are good for the human side). Since find_comp_move has a tentative maximum of 68, nothing that C does will affect the result at the min level. Therefore, C should not be evaluated. This type of pruning is known as   [image]  pruning; it is the symmetric version of   [image]  pruning. When both techniques are combined, we have   [image] -  [image]  pruning.

Figure 10.69 A hypothetical game tree

Figure 10.70 A pruned game tree

Figure 10.71 The node marked ? is unimportant

Figure 10.72 The node marked ? is unimportant

Implementing   [image] -  [image]  pruning requires surprisingly little code. It is not as difficult as one might think, although many programmers have a very hard time doing it without looking at a reference book. Figure 10.73 shows half of the   [image] -  [image]  pruning scheme (minus type declarations); you should have no trouble coding the other half.

/* Same as before, but perform    -   pruning.  */

/* The main routine should make the call with    = COMP_LOSS,

        = COMP_WIN. */

void

find_comp_move( board_type board, int *best_move, int *value,

int   , int    )

{

int dc, i, response; /* dc means don't care */

/*1*/       if( full_board( board ) )

/*2*/            *value = DRAW;

else

/*3*/       if( immediate-comp_win( board, best_move ) )

/*4*/            *value = COMP_WIN;

else

{

/*5*/            *value =    ;

/*6*/            for( i=1; (i<=9) && (*value<  ); i++)     /* try each square */

{

/*7*/                 if( is_empty( board, i ) )

{

/*8*/                      place( board, i, COMP );

/*9*/                      find_human_move( board, &dc, &response, *value,    );

/*10*/                     unplace( board, i ); /* Restore board */

/*11*/                     if( response >* value ) /* Update best move */

{

/*12*/                          *value = response;

/*13*/                          *best_move = i;

}

}

}

}

}

Figure 10.73 Min-max tic-tac-toe algorithm with   -   pruning: Computer selection.

To take full advantage of   [image] -  [image]  pruning, game programs usually try to apply the evaluation function to nonterminal nodes in an attempt to place the best moves early in the search. The result is even more pruning than one would expect from a random ordering of the nodes. Other techniques, such as searching deeper in more active lines of play, are also employed.

In practice,   [image] -  [image]  pruning limits the searching to only   [image]  nodes, where n is the size of the full game tree. This a huge saving and means that searches using   [image] -  [image]  pruning can go twice as deep as compared to an unpruned tree. Our tic-tac-toe example is not ideal, because there are so many identical values, but even so, the initial search of 97,162 nodes is reduced to 4,493 nodes. (These counts include nonterminal nodes).

In many games, computers are among the best players in the world. The techniques used are very interesting, and can be applied to more serious problems. More details can be found in the references.

Summary

This chapter illustrates five of the most common techniques found in algorithm design. When confronted with a problem, it is worthwhile to see if any of these methods apply. A proper choice of algorithm, combined with judicious use of data structures, can often lead quickly to efficient solutions.

Exercises

10.1     Show that the greedy algorithm to minimize the mean completion time for multiprocessor job scheduling works.

10.2     The input is a set of jobs j1, j2, . . . , jn, each of which takes one time unit to complete. Each job ji erans di dollars if it is completed by the time limit ti, but no money if completed after the time limit.

(a)     Give an O(n2) greedy algorithm to solve the problem.

**(b)     Modify your algorithm to obtain an O(n log n) time bound. Hint: The time bound is due entirely to sorting the jobs by money. The rest of the algorithm can be implemented, using the disjoint set data structure, in o(n log n).

10.3     A file contains only colons, spaces, newline, commas, and digits in the following frequency:  colon (100), space (605), newline (100), commas (705), 0 (431), 1 (242), 2 (176), 3 (59), 4 (185), 5 (250), 6 (174), 7 (199), 8 (205), 9 (217). Construct the Huffman code.

10.4     Part of the encoded file must be a header indicating the Huffman code. Give a method for constructing the header of size at most O(n) (in addition to the symbols), where n is the number of symbols.

10.5     Complete the proof that Huffman's algorithm generates an optimal prefix code.

10.6     Show that if the symbols are sorted by frequency, Huffman's algorithm can be implemented in linear time.

10.7     Write a program to implement file compression (and uncompression) using Huffman's algorithm.

*10.8     Show that any on-line bin-packing algorithm can be forced to use at least    the optimal number of bins, by considering the following sequence of items: n items of size   , n items of size   , n items of size   .

10.9     Explain how to implement first fit and best fit in O(n log n) time.

10.10     Show the operation of all of the bin-packing strategies discussed in Section 10.1.3 on the input 0.42, 0.25, 0.27, 0.07, 0.72, 0.86, 0.09, 0.44, 0.50, 0.68, 0.73, 0.31, 0.78, 0.17, 0.79, 0.37, 0.73, 0.23, 0.30.

10.11     Write a program that compares the performance (both in time and number of bins used) of the various bin packing heuristics.

10.12     Prove Theorem 10.7.

10.13     Prove Theorem 10.8.

*10.14     n points are placed in a unit square. Show that the distance between the closest pair is O(n-1/2).

*10.15     Argue that for the closest-points algorithm, the average number of points in the strip is (  . Hint: Use the result of the previous exercise.

10.16     Write a program to implement the closest-pair algorithm.

10.17     What is the asymptotic running time of quickselect, using a median-of-median-of-three partitioning strategy?

10.18     Show that quickselect with median-of-median-of-seven partitioning is linear. Why is median-of-median-of-seven partitioning not used in the proof?

10.19      Implement the quickselect algorithm in Chapter 7, quickselect using median-of-median-of-five patitioning, and the sampling algorithm at the end of Section 10.2.3. Compare the running times.

10.20     Much of the information used to compute the median-of-median-of-five is thrown away. Show how the  number of comparisons can be reduced by more careful use of the information.

*10.21     Complete the analysis of the sampling algorithm described at the end of Section 10.2.3, and explain how the values of    and s are chosen.

10.22     Show how the recursive multiplication algorithm computes xy, where x = 1234 and y = 4321. Include all recursive computations.

10.23     Show how to multiply two complex numbers x = a + bi and y = c + di using only three multiplications.

10.24      (a)     Show that

xlyr + xryl = (xl + xr)(yl + yr) - xlyl - xryr

(b)     This gives an O(n1.59) algorithm to multiply n-bit numbers. Compare this method to the solution in the text.

10.25     * (a)     Show how to multiply two numbers by solving five problems that are roughly one-third of the original size.

**(b)     Generalize this problem to obtain an O(n1+  ) algorithm for any constant    > 0.

(c)     Is the algorithm in part (b) better than O(n log n)?

10.26     Why is it important that Strassen's algorithm does not use commutativity in the multiplication of 2 X 2 matrices?

10.27     Two 70 X 70 matrices can be multiplied using 143,640 multiplications. Show how this can be used to improve the bound given by Strassen's algorithm.

10.28     What is the optimal way to compute A1A2A3A4A5A6, where the dimensions of the matrices are: Al: 10 X 20, A2: 20 X 1, A3: 1 X 40, A4: 40 X 5, A5: 5 X 30, A6: 30 X 15?

10.29     Show that none of the following greedy algorithms for chained matrix multiplication work. At each step

(a)     Compute the cheapest multiplication.

(b)     Compute the most expensive multiplication.

(c)     Compute the multiplication between the two matrices Mi and Mi+1, such that the number of columns in Mi is minimized (breaking ties by one of the rules above).

10.30     Write a program to compute the best ordering of matrix multiplication. Include the routine to print out the actual ordering.

10.31     Show the optimal binary search tree for the following words, where the frequency of occurrence is in parentheses: a (0.18), and (0.19), I (0.23), it (0.21) , or (0.19).

*10.32     Extend the optimal binary search tree algorithm to allow for unsuccessful searches. In this case, qj, for 1    j < n, is the probability that a search is performed for any word W satisfying wj < W < wj+1. q0 is the probability of performing a search for W < w1, and qn is the probability of performing a search for W > wn. Notice that   .

*10.33     Suppose Ci,i = 0 and that otherwise

Suppose that W satisfies the quadrangle inequality, namely, for all i    i'    j    j',

Wi, j + Wi',j'    Wi',j + Wi, j'

Suppose further, that W is monotone: If i    i' and j'    j', then Wi,j    Wi',,j'.

(a)     Prove that C satisfies the quadrangle inequality.

(b)     Let Ri, j be the largest k that achieves the minimum Ci,k-1 + Ck,j. (That is, in case of ties, choose the largest k). Prove that

Ri, j    Ri, j+1    Ri+1,j+1

(c)     Show that R is nondecreasing along each row and column.

(d)     Use this to show that all entries in C can be computed in O(n2) time.

(e)     Which of the dynamic programming algorithms can be solved in O(n2) using these techniques?

10.34     Write a routine to reconstruct the shortest paths from the algorithm in Section 10.3.4.

10.35     Examine the random number generator on your system. How random is it?

10.36     Write the routines to perform insertion, deletion, and searching in skip lists.

10.37     Give a formal proof that the expected time for the skip list operations is O(log n).

10.38     Figure 10.74 shows a routine to flip a coin, assuming that random returns an integer (which is prevalent in many systems). What is the expected performance of the skip list algorithms if the random number generator uses a modulus of the form m = 2b (which is unfortunately prevalent on many systems)?

10.39      (a)     Use the exponentiation algorithm to prove that 2340    1(mod 341).

(b)     Show how the randomized primality test works for n = 561 with several choices of a.

10.40     Implement the turnpike reconstruction algorithm.

10.41     Two point sets are homometric if they yield the same distance set and are not rotations of each other. The following distance set gives two distinct point sets: 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 13, 16, 17 . Find the two point sets.

enum coin_side { heads, tails };

typedef enum coin_side coin_side;

coin_side

flip( void )

{

if( ( rand() % 2 ) == 0 )

return heads;

else

return tails;

}

Figure 10.74 Questionable coin flipper

Figure 10.75 Game tree, which can be pruned

10.42     Extend the reconstruction algorithm to find all homometric point sets given a distance set.

10.43     Show the result of   -   pruning the tree in Figure 10.75.

10.44      (a)     Does the code in Figure 10.73 implement    pruning or    pruning?

(b)     Implement the complementary routine.

10.45     Write the remaining procedures for tic-tac-toe.

10.46     The one-dimensional circle packing problem is as follows: You have n circles of radii r1, r2, . . . , rn. These circles are packed in a box such that each circle is tangent to the bottom of the box, and are arranged in the original order. The problem is to find the width of the minimum-sized box.

Figure 10.76 shows an example with circles of radii 2, 1, 2 respectively. The minimum-sized box has width

*10.47     Suppose that the edges in an undirected graph G satisfy the triangle inequality: cu,v + cv,w    cu,w. Show how to compute a traveling salesman tour of cost at most twice optimal. Hint: Construct a minimum spanning tree.

*10.48     You are a tournament director and need to arrange a round robin tournament among n = 2k players. In this tournament, everyone plays exactly one game each day; after n - 1 days, a match has occurred between every pair of players. Give an algorithm to do this.

Figure 10.76 Sample for circle packing problem

10.49 (a)     Prove that in a round robin tournament it is always possible to arrange the players in an order pi1, p2, pin such that for all 1    j < n, pij has won the match against pij + 1.

(b)     Gve an O(n log n) algorithm to find one such arrangement. Your algorithm may serve as a proof for part (a).

*10.50     We are given a set P = p1, p2, . . . , pn of n points in a plane. A Voronoi diagram is a partition of the plane into n regions Ri such that all points in Ri are closer to pi than any other point in P. Figure 10.77 shows a sample Voronoi diagram for seven (nicely arranged) points. Give an O(n log n) algorithm to construct the Voronoi diagram.

*10.51     A convex polygon is a polygon with the property that any line segment whose endpoints are on the polygon lies entirely within the polygon. The convex hull problem consists of finding the smallest (area) convex polygon which encloses a set of points in the plane. Figure 10.78 shows the convex hull for a set of 40 points. Give an O(n log n) algorithm to find the convex hull.

Figure 10.77 Voronoi diagram

Figure 10.78 Example of a convex hull

*10.52     Consider the problem of right-justifying a paragraph. The paragraph contains a sequence of words w1, w2, . . . ,wn of length a1, a2, . . . , an, which we wish to break into lines of length L. Words are separated by blanks whose ideal length is b (millimeters), but blanks can stretch or shrink as necessary (but must be >0), so that a line wi wi+1. . . wj has length exactly L. However, for each blank b' we charge |b' - b ugliness points. The exception to this is the last line, for which we charge only if b' < b (in other words, we charge only for shrinking), since the last line does not need to be justified. Thus, if bi is the length of the blank between ai and ai+1, then the ugliness of setting any line (but the last) wiwi+1 . . . wj for j > i is   , where b' is the average size of a blank on this line. This is true of the last line only if b' < b, otherwise the last line is not ugly at all.

(a)     Give a dynamic programming algorithm to find the least ugly setting of w1, w2, . . ., wn into lines of length L. Hint: For i = n, n - 1, . . . , 1, compute the best way to set wj, wi+1, . . . , wn

(b)     Give the time and space complexities for your algorithm (as a function of the number of words, n).

(c)     Consider the special case where we are using a line printer instead of a laser printer, and assume the optimal value of b is 1 (space). In this case, no shrinking of blanks is allowed, since the next smallest blank space would be 0. Give a linear-time algorithm to generate the least ugly setting on a line printer.

*10.53     The longest increasing subsequence problem is as follows: Given numbers a1, a2, . . ., an, find the maximum value of k such that ai1 < ai2 <          < aik, and i1 < i2 <          < ik. As an example, if the input is 3, 1, 4, 1, 5, 9, 2, 6, 5, the maximum increasing subsequence has length four ( 1, 4, 5, 9 among others ). Give an O(n2) algorithm to solve the longest increasing subsequence problem.

*10.54     The longest common subsequence problem is as follows: Given two sequences A = a1, a2, . . . , am, and B = b1, b2, . . . , bn, find the length, k, of the longest sequence C = c1, c2, . . . , ck such that C is a subsequence of both A and B. As an example, if

A = d, y, n, a, m, i, c

and

B = p, r, o, g, r, a, m, m, i, n, g,

then the longest common subsequence is a,m and has length 2. Give an  algorithm to solve the longest common subsequence problem. Your algorithm should run in O(mn) time.

*10.55     The pattern matching problem is as follows: Given a string S of text, and a pattern P, find the first occurrence of P in S. Approximate pattern matching allows k mismatches of three types:

1.     A character can be in S that is not in P.

2.     A character can be in P that is not in S.

3.     P and S can differ in a position.

As an example, if we are searching for the pattern "textbook" with at most three mismatches in the string "data structures txtborkk", we find a match (insert an e, change an r to an o, delete a k). Give an O(mn) algorithm to solve the approximate string matching problem, where m = |P| and n = |S|.

*10.56     One form of the knapsack problem is as follows: We are given a set of integers A = a1, a2, . . . , an and an integer K. Is there a subset of A whose sum is exactly K?

(a)     Give an algorithm that solves the knapsack problem in O(nK) time.

(b)     Why does this not show that P = NP?

*10.57     You are given a currency system with coins of (decreasing) value c1, c2, . . . , cn cents.

(a)     Give an algorithm that computes the minimum number of coins required to give K cents in change.

(b)     Give an algorithm that computes the number of different ways to give K cents in change.

*10.58     Consider the problem of placing eight queens on an (eight by eight) chess board. Two queens are said to attack each other if they are on the same row, column, or (not necessarily main) diagonal.

(a)     Give a randomized algorithm to place eight nonattacking queens on the board.

(b)     Give a backtracking algorithm to solve the same problem.

(c)     Implement both algorithms and compare the running time.

distance

shortest( s, t, G )

{

distance dt,tmp;

if( s == t )

return 0;

dt =   ;

for each vertex v adjacent to s

{

tmp = shortest( v, t, G );

if( cs,v + tmp < dt )

dt = cs,v + tmp;

}

return dt

}

Figure 10.79 Recursive shortest path algorithm

*10.59     In the game of chess, a knight in row r and column c may move to row 1    r'    B and column 1    c'    B (where B is the size of the board) provided that either

|r - r'| = 2 and |c - c'| = 1

or

|r - r'| = 1 and |c - c'| = 2

A knight's tour is a sequence of moves that visits all squares exactly once before returning to the starting point.

(a)     If B is odd, show that a knight's tour cannot exist.

(b)     Give a backtracking algorithm to find a knight's tour.

10.60     Consider the recursive algorithm in Figure 10.79 for finding the shortest weighted path in an acyclic graph, from s to t.

(a)     Why does this algorithm not work for general graphs?

(b)     Prove that this algorithm terminates for acyclic graphs.

(c)     What is the worst-case running time of the algorithm?

References

The original paper on Huffman codes is [21]. Variations on the algorithm are discussed in [29], [31], and [32]. Another popular compression scheme is Ziv-Lempel encoding [52], [53]. Here the codes have a fixed length but represent strings instead of characters. [3] and [34] are good surveys of the common compression schemes.

The analysis of bin-packing heuristics first appeared in Johnson's Ph.D. thesis  and was published in [22]. The improved lower bound for on-line bin packing given in Exercise 10.8 is from [50]; this result has been improved further in [35]. [44] describes another approach to on-line bin packing.

Theorem 10.7 is from [6]. The closest points algorithm appeared in [45]. [47] describes the turnpike reconstruction problem and its applications. Two books on  the relatively new field of computational geometry are [14] and [40]. [2] contains the lecture notes for a computational geometry course taught at MIT; it includes  an extensive bibliography.

The linear-time selection algorithm appeared in [8]. [17] discusses the  sampling approach that finds the median in 1.5n expected comparisons. The O(n1.59)  multiplication is from [23]. Generalizations are discussed in [9] and [24]. Strassen's algorithm appears in the short paper [48]. The paper states the results and not  much else. Pan [38] gives several divide and conquer algorithms, including the one in Exercise 10.27. The best known bound is O(n2.376), which is due to Coppersmith  and Winograd [13].

The classic references on dynamic programming are the books [4] and [5]. The  matrix ordering problem was first studied in [19]. It was shown in [20] that the problem can be solved in O(n log n) time.

An O(n2) algorithm was provided for the construction of optimal binary search  trees by Knuth [25]. The all-pairs shortest-path algorithm is from Floyd [16]. A theoretically better O(n3(log logn/logn)l/3) algorithm is given by Fredman [18], but not surprisingly, it is not practical. Under certain conditions, the running time of dynamic programs can automatically be improved by a factor of n or more. This is discussed in Exercise 10.33, [15], and [51].

The discussion of random number generators is based on [39]. Park and Miller  attribute the portable implementation to Schrage [46]. Skip lists are discussed  by Pugh in [41]. The randomized primality-testing algorithm is due to Miller [36] and Rabin [43]. The theorem that at most (n - 9)/4 values of a fool  the algorithm is from Monier [37]. Other randomized algorithms are discussed in [42].

More information on   -   pruning can be found in [1], [26], and [27]. The top programs that play chess, checkers, Othello, and backgammon have all achieved  world class status. [33] describes an Othello program. The paper appears in a special issue on computer games (mostly chess); this issue is a gold mine of ideas. One of the papers describes the use of dynamic programming to solve chess endgames completely when only a few pieces are left on the board. Related research has  resulted in the change of the 50-move rule in certain cases.

Exercise 10.41 is solved in [7]. It is the only known case of a  homometric point set with no duplicate distances. Determining whether any others exist  for n > 6 is open. Christofides [12] gives a solution to Exercise 10.47, and  also an algorithm which generates a tour at most    optimal. Exercise 10.52 is  discussed in [28]. Exercise 10.55 is solved in [49]. An O(kn) algorithm is given in [30]. Exercise 10.57 is discussed in [10], but do not be misled by the title of  the paper.

1.     B. Abramson, "Control Strategies for Two-Player Games," ACM Computing Surveys, 21 (1989), 137-161.

2.     A. Aggarwal and J. Wein, Computational Geometry: Lecture Notes for 18.409, MIT Laboratory for Computer Science, 1988.

3.     T. Bell, I. H. Witten, and J. G. Cleary, "Modeling for Text Compression," ACM Computing Surveys, 21 (1989), 557-591.

4.     R. E. Bellman, Dynamic Programming, Princeton University Press, Princeton, NJ, 1957.

5.     R. E. Bellman and S. E. Dreyfus, Applied Dynamic Programming, Princeton University Press, Princeton, NJ, 1962.

6.     J. L. Bentley, D. Haken, and J. B. Saxe, "A General Method for Solving Divide-and-Conquer Recurrences," SIGACT News, 12 (1980), 36-44.

7.     G. S. Bloom, "A Counterexample to the Theorem of Piccard," Journal of Combinatorial Theory A (1977), 378-379.

8.     M. Blum, R. W. Floyd, V. R. Pratt, R. L. Rivest, and R. E. Tarjan, "Time Bounds for Selection," Journal of Computer and System Sciences 7 (1973), 448-461.

9.     A. Borodin and J. I. Munro, The Computational Complexity of Algebraic and Numerical Problems, American Elsevier, New York, 1975.

10.     L. Chang and J. Korsh, "Canonical Coin Changing and Greedy Solutions," Journal of the ACM 23 (1976), 418-422.

12.     N. Christofides, "Worst-case Analysis of a New Heuristic for the Traveling Salesman Problem," Management Science Research Report #388, Carnegie-Mellon University, Pittsburgh, PA, 1976.

13.     D. Coppersmith and S. Winograd, "Matrix Multiplication via Arithmetic Progressions," Proceedings of the Nineteenth Annual ACM Symposium of the Theory of Computing (1987), 1-6.

14.     H. Edelsbrunner, Algorithms in Combinatorial Geometry, Springer-Verlag, Berlin, 1987.

15.     D. Eppstein, Z. Galil, R. Giancarlo, "Speeding up Dynamic Programming," Proceedings of the Twenty-ninth Annual IEEE Symposium on the Foundations of Computer Science, (1988), 488-495.

16.     R. W. Floyd, "Algorithm 97: Shortest Path," Communications of the ACM 5 (1962), 345.

17.     R. W. Floyd and R. L. Rivest, "Expected Time Bounds for Selection," Communications of the ACM 18 (1975), 165-172.

18.     M. L. Fredman, "New Bounds on the Complexity of the Shortest Path Problem," SIAM Journal on Computing 5 (1976), 83-89.

19.     S. Godbole, "On Efficient Computation of Matrix Chain Products," IEEE Transactions on Computers 9 (1973), 864-866.

20.     T. C. Hu and M. R. Shing, "Computations of Matrix Chain Products, Part I," SIAM Journal on Computing 11 (1982), 362-373.

21.     D. A. Huffman, "A Method for the Construction of Minimum Redundancy Codes," Proceedings of the IRE 40 (1952), 1098-1101.

22.     D. S. Johnson, A. Demers, J. D. Ullman, M. R. Garey, and R. L. Graham, "Worst-case Performance Bounds for Simple One-Dimensional Packing Algorithms," SIAM Journal on Computing, 3 (1974), 299-325.

23.     A. Karatsuba and Y. Ofman, "Multiplication of Multi-digit Numbers on Automata," Doklady Akademii Nauk SSSR 145 (1962), 293-294.

24.     D. E. Knuth, The Art of Computer Programming, Vol 2: Seminumerical Algorithms, second edition, Addison-Wesley, Reading, MA, 1981.

25.     D. E. Knuth, "Optimum Binary Search Trees," Acta Informatica 1 (1971), 14-25.

26.     D. E. Knuth and R. W. Moore, "Estimating the Efficiency of Backtrack Programs," Mathematics of Computation 29, (1975) 121-136.

27.     D. E. Knuth, "An Analysis of Alpha-Beta Cutoffs," Artificial Intelligence 6 (1975), 293-326.

28.     D. E. Knuth, TEX and Metafont, New Directions in Typesetting, Digital Press, Bedford, MA, 1981.

29.     D. E. Knuth, "Dynamic Huffman Coding,"Journal of Algorithms 6 (1985), 163-180.

30.     G. M. Landau and U. Vishkin, "Introducing Efficient Parallelism into Approximate String Matching and a New Serial Algorithm," Proceedings of the Eighteenth Annual ACM Symposium on Theory of Computing (1986), 220-230.

31.     L. L. Larmore, "Height-Restricted Optimal Binary Trees," SlAM Journal on Computing 16 (1987), 1115-1123.

32.     L. L. Larmore and D. S. Hirschberg, "A Fast Algorithm for Optimal Length-Limited Huffman Codes," Journal of the ACM 37 (1990), 464-473.

33.     K. Lee and S. Mahajan, "The Development of a World Class Othello Program," Artificial Intelligence 43 (1990), 21-36.

34.     D. A. Lelewer and D. S. Hirschberg, "Data Compression," ACM Computing Surveys 19 (1987), 261-296.

35.     F. M. Liang, "A Lower Bound for On-line Bin Packing," Information Processing Letters 10 (1980), 76-79.

36.     G. L. Miller, "Riemann's Hypothesis and Tests for Primality," Journal of Computer and System Sciences 13 (1976), 300-317.

37.     L. Monier, "Evaluation and Comparison of Two Efficient Probabilistic Primality Testing Algorithms," Theoretical Computer Science 12 (1980), 97-108.

38.     V. Pan, "Strassen's Algorithm is Not Optimal," Proceedings of the Nineteenth Annual IEEE Symposium on the Foundations of Computer Science (1978), 166-176.

39.     S. K. Park and K. W. Miller, "Random Number Generators: Good Ones are Hard To Find," Communications of the ACM 31 (1988), 1192-1201.

40.     F. P. Preparata and M. I. Shamos, Computational Geometry: An Introduction, Springer-Verlag, New York, NY, 1985.

41.     W. Pugh, "Skip Lists: A Probabilistic Alternative to Balanced Trees," Communications of the ACM 33 (1990), 668-676.

42.     M. O. Rabin, "Probabilistic Algorithms," in Algorithms and Complexity, Recent Results and New Directions (J. F. Traub, ed.), Academic Press, New York, 1976, 21-39.

43.     M. O. Rabin, "Probabilistic Algorithms for Testing Primality," Journal of Number Theory, 12 (1980), 128-138.

44.     P. Ramanan, D. J. Brown, C. C. Lee, and D. T. Lee, "On-line Bin Packing in Linear Time," Journal of Algorithms 10 (1989), 305-326.

45.     M. I. Shamos and D. Hoey, "Closest-Point Problems," Proceedings of the Sixteenth Annual IEEE Symposium on the Foundations of Computer Science (1975), 151-162.

46.     L. Schrage, "A More Portable FORTRAN Random Number Generator," ACM Transactions on Mathematics Software 5 (1979), 132-138.

47.     S. S. Skiena, W. D. Smith, and P. Lemke, "Reconstructing Sets From Interpoint Distances," Proceedings of the Sixth Annual ACM Symposium on Computational Geometry (1990), 332-339.

48.     V. Strassen, "Gaussian Elimination is Not Optimal," Numerische Mathematik 13 (1969), 354-356.

49.     R. A. Wagner and M. J. Fischer, "The String-to-String Correction Problem," Journal of the ACM 21 (1974), 168-173.

50.     A. C. Yao, "New Algorithms for Bin Packing," Journal of the ACM 27 (1980), 207-227.

51.     F. F. Yao, "Efficient Dynamic Programming Using Quadrangle Inequalities," Proceedings of the Twelfth Annual ACM Symposium on the Theory of Computing (1980), 429-435.

52.     J. Ziv and A. Lempel, "A Universal Algorithm for Sequential Data Compression," IEEE Transactions on Information Theory IT23 (1977), 337-343.

53.     J. Ziv and A. Lempel, "Compression of Individual Sequences via Variable-rate Coding," IEEE Transactions on Information Theory IT24 (1978), 530-536.

Go to Chapter 11              Return to Table of Contents


Structures, Algorithm Analysis: CHAPTER 11: AMORTIZED ANALYSIS

 Return to Table of Contents

CHAPTER 11: AMORTIZED ANALYSIS

In this chapter, we will analyze the running time for several of the advanced data structures that have been presented in Chapters 4 and 6. In particular, we will consider the worst-case running time for any sequence of m operations. This contrasts with the more typical analysis, in which a worst-case bound is given for any single operation.

As an example, we have seen that AVL trees support the standard tree operations in O(log n) worst-case time per operation. AVL trees are somewhat complicated to implement, not only because there are a host of cases, but also because height balance information must be maintained and updated correctly. The reason that AVL trees are used is that a sequence of [image] (n) operations on an unbalanced search tree could require [image] (n2) time, which would be expensive. For search trees, the O(n) worst-case running time of an operation is not the real problem. The major problem is that this could happen repeatedly. Splay trees offer a pleasant alternative. Although any operation can still require [image] (n) time, this degenerate behavior cannot occur repeatedly, and we can prove that any sequence of m operations takes O(m log n) worst-case time (total). Thus, in the long run this data structure behaves as though each operation takes O(log n). We call this an amortized time bound.

Amortized bounds are weaker than the corresponding worst-case bounds, because there is no guarantee for any single operation. Since this is generally not important, we are willing to sacrifice the bound on a single operation, if we can retain the same bound for the sequence of operations and at the same time simplify the data structure. Amortized bounds are stronger than the equivalent average-case bound. For instance, binary search trees have O (log n) average time per operation, but it is still possible for a sequence of m operations to take O (mn) time.

Because deriving an amortized bound requires us to look at an entire sequence of operations instead of just one, we expect that the analysis will be more tricky. We will see that this expectation is generally realized.

In this chapter we shall

Analyze the binomial queue operations.

Analyze skew heaps.

Introduce and analyze the Fibonacci heap.

Analyze splay trees.

11.1. An Unrelated Puzzle

Consider the following puzzle: Two kittens are placed on opposite ends of a football field, 100 yards apart. They walk towards each other at the speed of ten yards per minute. At the same time, their mother is at one end of the field. She can run at 100 yards per minute. The mother runs from one kitten to the other, making turns with no loss of speed, until the kittens (and thus the mother) meet at midfield. How far does the mother run?

It is not hard to solve this puzzle with a brute force calculation. We leave the details to you, but one expects that this calculation will involve computing the sum of an infinite geometric series. Although this straightforward calculation will lead to an answer, it turns out that a much simpler solution can be arrived at by introducing an extra variable, namely, time.

Because the kittens are 100 yards apart and approach each other at a combined velocity of 20 yards per minute, it takes them five minutes to get to midfield. Since the mother runs 100 yards per minute, her total is 500 yards.

This puzzle illustrates the point that sometimes it is easier to solve a problem indirectly than directly. The amortized analyses that we will perform will use this idea. We will introduce an extra variable, known as the potential, to allow us to prove results that seem very difficult to establish otherwise.

11.2. Binomial Queues

The first data structure we will look at is the binomial queue of Chapter 6, which we now review briefly. Recall that a binomial tree B0 is a one-node tree, and for k > 0, the binomial tree Bk is built by melding two binomial trees Bk-1 together. Binomial trees B0 through B4 are shown in Figure 11.1.

Figure 11.1 Binomial trees B0, B1, B2, B3, and B4

Figure 11.2 Two binomial queues H1 and H2

The rank of a node in a binomial tree is equal to the number of children; in particular, the rank of the root of Bk is k. A binomial queue is a collection of heap-ordered binomial trees, in which there can be at most one binomial tree Bk for any k. Two binomial queues, H1 and H2, are shown in Figure 11.2.

The most important operation is merge. To merge two binomial queues, an operation similar to addition of binary integers is performed: At any stage we may have zero, one, two, or possibly three Bk trees, depending on whether or not the two priority queues contain a Bk tree and whether or not a Bk tree is carried over from the previous step. If there is zero or one Bk tree, it is placed as a tree in the resultant binomial queue. If there are two Bk trees, they are melded into a Bk+1 tree and carried over; if there are three Bk trees, one is placed as a tree in the binomial queue and the other two are melded and carried over. The result of merging H1 and H2 is shown in Figure 11.3.

Insertion is performed by creating a one-node binomial queue and performing a merge. The time to do this is m + 1, where m represents the smallest type of binomial tree Bm not present in the binomial queue. Thus, insertion into a binomial queue that has a B0 tree but no B1 tree requires two steps. Deletion of the minimum is accomplished by removing the minimum and splitting the original binomial queue into two binomial queues, which are then merged. A less terse explanation of these operations is given in Chapter 6.

We consider a very simple problem first. Suppose we want to build a binomial queue of n elements. We know that building a binary heap of n elements can be done in O (n), so we expect a similar bound for binomial queues.

Figure 11.3 Binomial queue H3: the result of merging H1 and H2

CLAIM:

A binomial queue of n elements can be built by n successive insertions in O(n) time.

The claim, if true, would give an extremely simple algorithm. Since the worst-case time for each insertion is O (log n), it is not obvious that the claim is true. Recall that if this algorithm were applied to binary heaps, the running time would be O(n log n).

To prove the claim, we could do a direct calculation. To measure the running time, we define the cost of each insertion to be one time unit plus an extra unit for each linking step. Summing this cost over all insertions gives the total running time. This total is n units plus the total number of linking steps. The 1st, 3rd, 5th, and all odd-numbered steps require no linking steps, since there is no B0 present at the time of insertion. Thus, half of the insertions require no linking steps. A quarter of the insertions require only one linking step (2nd, 6th, 10th, and so on). An eighth require two, and so on. We could add this all up and bound the number of linking steps by n, proving the claim. This brute force calculation will not help when we try to analyze a sequence of operations that include more than just insertions, so we will use another approach to prove this result.

Consider the result of an insertion. If there is no B0 tree present at the time of the insertion, then the insertion costs a total of one unit, by using the same accounting as above. The result of the insertion is that there is now a B0 tree, and thus we have added one tree to the forest of binomial trees. If there is a B0 tree but no B1 tree, then the insertion costs two units. The new forest will have a B1 tree but will no longer have a B0 tree, so the number of trees in the forest is unchanged. An insertion that costs three units will create a B2 tree but destroy a B0 and B1 tree, yielding a net loss of one tree in the forest. In fact, it is easy to see that, in general, an insertion that costs c units results in a net increase of 2 - c trees in the forest, because a Bc-1 tree is created but all Bi trees 0 [image] i < c - 1 are removed. Thus, expensive insertions remove trees, while cheap insertions create trees.

Let Ci be the cost of the ith insertion. Let Ti be the number of trees after the ith insertion. T0 = 0 is the number of trees initially. Then we have the invariant

Ci + (Ti - Ti-1) = 2

(11.1)

We then have

     C1 + (T1 - T0) = 2

     C2 + (T2 - T1) = 2

Cn-1 + (Tn-1 - Tn-2) = 2

   Cn + (Tn - Tn-1) = 2

If we add all these equations, most of the Ti terms cancel, leaving

or equivalently,

Recall that T0 = 0 and Tn, the number of trees after the n insertions, is certainly not negative, so (Tn - T0) is not negative. Thus

which proves the claim.

During the build_binomial_queue routine, each insertion had a worst-case time of O(log n), but since the entire routine used at most 2n units of time, the insertions behaved as though each used no more than 2 units each.

This example illustrates the general technique we will use. The state of the data structure at any time is given by a function known as the potential. The potential function is not maintained by the program, but rather is an accounting device that will help with the analysis. When operations take less time than we have allocated for them, the unused time is "saved" in the form of a higher potential. In our example, the potential of the data structure is simply the number of trees. In the analysis above, when we have insertions that use only one unit instead of the two units that are allocated, the extra unit is saved for later by an increase in potential. When operations occur that exceed the allotted time, then the excess time is accounted for by a decrease in potential. One may view the potential as representing a savings account. If an operation uses less than its allotted time, the difference is saved for use later on by more expensive operations. Figure 11.4 shows the cumulative running time used by build_binomial_queue over a sequence of insertions. Observe that the running time never exceeds 2n and that the potential in the binomial queue after any insertion measures the amount of savings.

Figure 11.4 A sequence of n inserts

Once a potential function is chosen, we write the main equation:

Tactual + [image] Potential = Tamortized

(11.2)

Tactual, the actual time of an operation, represents the exact (observed) amount of time required to execute a particular operation. In a binary search tree, for example, the actual time to perform a find(x) is 1 plus the depth of the node containing x. If we sum the basic equation over the entire sequence, and if the final potential is at least as large as the initial potential, then the amortized time is an upper bound on the actual time used during the execution of the sequence. Notice that while Tactual varies from operation to operation, Tamortized is stable.

Picking a potential function that proves a meaningful bound is a very tricky task; there is no one method that is used. Generally, many potential functions are tried before the one that works is found. Nevertheless, the discussion above suggests a few rules, which tell us the properties that good potential functions have. The potential function should

Always assume its minimum at the start of the sequence. A popular method of choosing potential functions is to ensure that the potential function is initially 0, and always nonnegative. All of the examples that we will encounter use this strategy.

Cancel a term in the actual time. In our case, if the actual cost was c, then the potential change was 2 - c. When these are added, an amortized cost of 2 is obtained. This is shown in Figure 11.5.

We can now perform a complete analysis of binomial queue operations.

Figure 11.5 The insertion cost and potential change for each operation in a sequence

THEOREM 11.1.

The amortized running times of insert, delete_min, and merge are O(1), O(log n), and O(log n), respectively, for binomial queues.

PROOF:

The potential function is the number of trees. The initial potential is 0, and the potential is always nonnegative, so the amortized time is an upper bound on the actual time. The analysis for insert follows from the argument above. For merge, assume the two trees have n1 and n2 nodes with T1 and T2 trees, respectively. Let n = n1 + n2. The actual time to perform the merge is O(log(n1) + log(n2)) = O(log n). After the merge, there can be at most log n trees, so the potential can increase by at most O(log n). This gives an amortized bound of O(log n). The delete_min bound follows in a similar manner.

11.3. Skew Heaps

The analysis of binomial queues is a fairly easy example of an amortized analysis. We now look at skew heaps. As is common with many of our examples, once the right potential function is found, the analysis is easy. The difficult part is choosing a meaningful potential function.

Recall that for skew heaps, the key operation is merging. To merge two skew heaps, we merge their right paths and make this the new left path. For each node on the new path, except the last, the old left subtree is attached as the right subtree. The last node on the new left path is known to not have a right subtree, so it is silly to give it one. The bound does not depend on this exception, and if the routine is coded recursively, this is what will happen naturally. Figure 11.6 shows the result of merging two skew heaps.

Suppose we have two heaps, H1 and H2, and there are r1 and r2 nodes on their respective right paths. Then the actual time to perform the merge is proportional to r1 + r2, so we will drop the Big-Oh notation and charge one unit of time for each node on the paths. Since the heaps have no structure, it is possible that all the nodes in both heaps lie on the right path, and this would give a [image] (n) worst-case bound to merge the heaps (Exercise 11.3 asks you to construct an example). We will show that the amortized time to merge two skew heaps is O(log n).

Figure 11.6 Merging of two skew heaps

What is needed is some sort of a potential function that captures the effect of skew heap operations. Recall that the effect of a merge is that every node on the right path is moved to the left path, and its old left child becomes the new right child. One idea might be to classify each node as a right node or left node, depending on whether or not it is a right child, and use the number of right nodes as a potential function. Although the potential is initially 0 and always nonnegative, the problem is that the potential does not decrease after a merge and thus does not adequately reflect the savings in the data structure. The result is that this potential function cannot be used to prove the desired bound.

A similar idea is to classify nodes as either heavy or light, depending on whether or not the right subtree of any node has more nodes than the left subtree.

DEFINITION: A node p is heavy if the number of descendants of p's right subtree is at least half of the number of descendants of p, and light otherwise. Note that the number of descendants of a node includes the node itself.

As an example, Figure 11.7 shows a skew heap. The nodes with keys 15, 3, 6, 12, and 7 are heavy, and all other nodes are light.

The potential function we will use is the number of heavy nodes in the (collection) of heaps. This seems like a good choice, because a long right path will contain an inordinate number of heavy nodes. Because nodes on this path have their children swapped, these nodes will be converted to light nodes as a result of the merge.

THEOREM 11.2.

The amortized time to merge two skew heaps is O(log n).

PROOF:

Let H1 and H2 be the two heaps, with n1 and n2 nodes respectively. Suppose the right path of H1 has l1 light nodes and h1 heavy nodes, for a total of l1 + h1. Likewise, H2 has l2 light and h2 heavy nodes on its right path, for a total of l2 + h2 nodes.

Figure 11.7 Skew heap -- heavy nodes are 3, 6, 7, 12, and 15

Figure 11.8 Change in heavy/light status after a merge

If we adopt the convention that the cost of merging two skew heaps is the total number of nodes on their right paths, then the actual time to perform the merge is l1 + l2 + h1 + h2. Now the only nodes whose heavy/light status can change are nodes that are initially on the right path (and wind up on the left path), since no other nodes have their subtrees altered. This is shown by the example in Figure 11.8.

If a heavy node is initially on the right path, then after the merge it must become a light node. The other nodes that were on the right path were light and may or may not become heavy, but since we are proving an upper bound, we will have to assume the worst, which is that they become heavy and increase the potential. Then the net change in the number of heavy nodes is at most l1 + l2 - h1 - h2. Adding the actual time and the potential change (Equation 11.2) gives an amortized bound of 2(l1 + l2).

Now we must show that l1 + l2 = O(log n). Since l1 and l2 are the number of light nodes on the original right paths, and the right subtree of a light node is less than half the size of the tree rooted at the light node, it follows directly that the number of light nodes on the right path is at most log n1 + log n2, which is O(log n).

The proof is completed by noting that the initial potential is 0 and that the potential is always nonnegative. It is important to verify this, since otherwise the amortized time does not bound the actual time and is meaningless.

Since the insert and delete_min operations are basically just merges, they also have O(log n) amortized bounds.

11.4. Fibonacci Heaps

In Section 9.3.2, we showed how to use priority queues to improve on the naïve O(|V|2) running time of Dijkstra's shortest-path algorithm. The important observation was that the running time was dominated by|E|decrease_key operations and |V| insert and delete_min operations. These operations take place on a set of size at most |V|. By using a binary heap, all these operations take O(log |V|) time, so the resulting bound for Dijkstra's algorithm can be reduced to O(|E| log |V|).

In order to lower this time bound, the time required to perform the decrease_key operation must be improved. d-heaps, which were described in Section 6.5, give an O(logd |V|) time bound for the delete_min operation as well as for insert, but an O(d logd |V|) bound for delete_min. By choosing d to balance the costs of |E| decrease_key operations with |V| delete_min operations, and remembering that d must always be at least 2, we see that a good choice for d is

d = max(2,    E  /  V    ).

This improves the time bound for Dijkstra's algorithm to

O(  E  log(2+    E  /  V    )  V  ).

The Fibonacci heap is a data structure that supports all the basic heap operations in O(1) amortized time, with the exception of delete_min and delete, which take O (log n) amortized time. It immediately follows that the heap operations in Dijkstra's algorithm will require a total of O(|E| + |V| log |V|) time.

Fibonacci heaps* generalize binomial queues by adding two new concepts:

*The name comes from a property of this data structure, which we will prove later in the section.

A different implementation of decrease_key: The method we have seen before is to percolate the element up toward the root. It does not seem reasonable to expect an O(1) amortized bound for this strategy, so a new method is needed.

Lazy merging: Two heaps are merged only when it is required to do so. This is similar to lazy deletion. For lazy merging, merges are cheap, but because lazy merging does not actually combine trees, the delete_min operation could encounter lots of trees, making that operation expensive. Any one delete_min could take linear time, but it is always possible to charge the time to previous merge operations. In particular, an expensive delete_min must have been preceded by a large number of unduly cheap merges, which have been able to store up extra potential.

11.4.1. Cutting Nodes in Leftist Heaps

In binary heaps, the decrease_key operation is implemented by lowering the value at a node and then percolating it up toward the root until heap order is established. In the worst case, this can take O(log n) time, which is the length of the longest path toward the root in a balanced tree.

This strategy does not work if the tree that represents the priority queue does not have O(log n) depth. As an example, if this strategy is applied to leftist heaps, then the decrease_key operation could take [image] (n) time, as the example in Figure 11.9 shows.

We see that for leftist heaps, another strategy is needed for the decrease_key operation. Our example will be the leftist heap in Figure 11.10. Suppose we want to decrease the key with value 9 down to 0. If we make the change, we find that we have created a violation of heap order, which is indicated by a dashed line in Figure 11.11.

Figure 11.9 Decreasing n - 1 to 0 via percolate up would take (n) time

Figure 11.10 Sample leftist heap H

Figure 11.11 Decreasing 9 to 0 creates a heap order violation

Figure 11.12 The two trees after the cut

We do not want to percolate the 0 to the root, because, as we have seen, there are cases where this could be expensive. The solution is to cut the heap along the dashed line, thus creating two trees, and then merge the two trees back into one. Let x be the node to which the decrease_key operation is being applied, and let p be its parent. After the cut, we have two trees, namely, H1 with root x, and T2, which is the original tree with H1 removed. The situation is shown in Figure 11.12.

If these two trees were both leftist heaps, then they could be merged in O (log n) time, and we would be done. It is easy to see that H1 is a leftist heap, since none of its nodes have had any changes in their descendants. Thus, since all of its nodes originally satisfied the leftist property, they still must.

Nevertheless, it seems that this scheme will not work, because T2 is not necessarily leftist. However, it is easy to reinstate the leftist heap property by using two observations:

Only nodes on the path from p to the root of T2 can be in violation of the leftist heap property; these can be fixed by swapping children.

Since the maximum right path length has at most [image] log(n + 1) [image] nodes, we only need to check the first [image] log(n + 1) [image] nodes on the path from p to the root of T2. Figure 11.13 shows H1 and T2 after T2 is converted to a leftist heap.

Figure 11.13 T2 converted to the leftist heap H2

Figure 11.14 decrease_key (H,x,9) completed by merging Hl and H2

Because we can convert T2 to the leftist heap H2 in O (log n) steps, and then merge H1 and H2, we have an O (log n) algorithm for performing the decrease_key operation in leftist heaps. The heap that results in our example is shown in Figure 11.14.

11.4.2. Lazy Merging for Binomial Queues

The second idea that is used by Fibonacci heaps is lazy merging. We will apply this idea to binomial queues and show that the amortized time to perform a merge operation (as well as insertion, which is a special case) is O(1). The amortized time for delete_min will still be O(log n).

The idea is as follows: To merge two binomial queues, merely concatenate the two lists of binomial trees, creating a new binomial queue. This new queue may have several trees of the same size, so it violates the binomial queue property. We will call this a lazy binomial queue in order to maintain consistency. This is a fast operation, which always takes constant (worst-case) time. As before, an insertion is done by creating a one-node binomial queue and merging. The difference is that the merge is lazy.

The delete_min operation is much more painful, because it is where we finally convert the lazy binomial queue back into a standard binomial queue, but, as we will show, it is still O (log n) amortized time-but not O(log n) worst-case time, as before. To perform a delete_min, we find (and eventually return) the minimum element. As before, we delete it from the queue, making each of its children new trees. We then merge all the trees into a binomial queue by merging two equal-sized trees until it is no longer possible.

As an example, Figure 11.15 shows a lazy binomial queue. In a lazy binomial queue, there can be more than one tree of the same size. We can tell the size of a tree by examining the root's rank field, which gives the number of children (and thus implicitly the type of tree). To perform the delete_min, we remove the smallest element, as before, and obtain the tree in Figure 11.16.

Figure 11.15 Lazy binomial queue

Figure 11.16 Lazy binomial queue after removing the smallest element (3)

We now have to merge all the trees and obtain a standard binomial queue. A standard binomial queue has at most one tree of each rank. In order to do this efficiently, we must be able to perform the merge in time proportional to the number of trees present (T) (or log n, whichever is larger). To do this, we form an array of lists, L0, L1, . . . , LR max+ 1, where Rmax is the rank of the largest tree. Each list Lr contains all of the trees of rank r. The procedure in Figure 11.17 is then applied.

Each time through the loop, at lines 3 through 5, the total number of trees is reduced by 1. This means that this part of the code, which takes constant time per execution, can only be performed T - 1 times, where T is the number of trees. The for loop counters, and tests at the end of the while loop take O (log n) time, so the running time is O (T + log n), as required. Figure 11.18 shows the execution of this algorithm on the previous collection of binomial trees.

Amortized Analysis of Lazy Binomial Queues

Amortized Analysis of Lazy Binomial Queues

To carry out the amortized analysis of lazy binomial queues, we will use the same potential function that was used for standard binomial queues. Thus, the potential of a lazy binomial queue is the number of trees.

THEOREM 11.3.

The amortized running times of merge and insert are both O(1) for lazy binomial queues. The amortized running time of delete_min is O(log n).

PROOF:

The potential function is the number of trees in the collection of binomial queues. The initial potential is 0, and the potential is always nonnegative. Thus, over a sequence of operations, the total amortized time is an upper bound on the total actual time.

/*1*/  for( r = 0; r <=   log n   ; r++ )

/*2*/  while ( |Lr|    2 )

{

/*3*/        remove two trees from Lr;

/*4*/        merge the two trees into a new tree;

/*5*/        add the new tree to Lr+1 ;

}

Figure 11.17 Procedure to reinstate a binomial queue

Figure 11.18 Combining the binomial trees into a binomial queue

For the merge operation, the actual time is constant, and the number of trees in the collection of binomial queues is unchanged, so, by Equation (11.2), the amortized time is O(1).

For the insert operation, the actual time is constant, and the number of trees can increase by at most 1, so the amortized time is O(1).

The delete_min operation is more complicated. Let r be the rank of the tree that contains the minimum element, and let T be the number of trees. Thus, the potential at the start of the delete_min operation is T. To perform a delete_min, the children of the smallest node are split off into separate trees. This creates T + r trees, which must be merged into a standard binomial queue. The actual time to perform this is T + r + log n, if we ignore the constant in the Big-Oh notation, by the argument above.* On the other hand, once this is done, there can be at most log n trees remaining, so the potential function can increase by at most (log n) - T. Adding the actual time and the change in potential gives an amortized bound of 2 log n + r. Since all the trees are binomial trees, we know that r [image] log n. Thus we arrive at an O(log n) amortized time bound for the delete_min operation.

*We can do this because we can place the constant implied by the Big-Oh notation in the potential function and still get the cancellation of terms, which is needed in the proof.

11.4.3. The Fibonacci Heap Operations

As we mentioned before, the Fibonacci heap combines the leftist heap decrease_key operation with the lazy binomial queue merge operation. Unfortunately, we cannot use both operations without a slight modification. The problem is that if arbitrary cuts are made in the binomial trees, the resulting forest will no longer be a collection of binomial trees. Because of this, it will no longer be true that the rank of every tree is at most [image] log n [image] . Since the amortized bound for delete_min in lazy binomial queues was shown to be 2 log n + r, we need r = O(log n) for the delete_min bound to hold.

In order to ensure that r = O(log n), we apply the following rules to all non-root nodes:

Mark a (nonroot) node the first time that it loses a child (because of a cut).

If a marked node loses another child, then cut it from its parent. This node now becomes the root of a separate tree and is no longer marked. This is called a cascading cut, because several of these could occur in one decrease_key operation.

Figure 11.19 shows one tree in a Fibonacci heap prior to a decrease_key operation.

When the node with key 39 is changed to 12, the heap order is violated. Therefore, the node is cut from its parent, becoming the root of a new tree. Since the node containing 33 is marked, this is its second lost child, and thus it is cut from its parent (10). Now 10 has lost its second child, so it is cut from 5. The process stops here, since 5 was unmarked. 5 is now marked. The result is shown in Figure 11.20.

Notice that 10 and 33, which used to be marked nodes, are no longer marked, because they are now root nodes. This will be a crucial observation in our proof of the time bound.

Figure 11.19 A tree in the Fibonacci heap prior to decreasing 39 to 12

Figure 11.20 The resulting segment of the Fibonacci heap after the decrease_key operation

11.4.4. Proof of the Time Bound

Recall that the reason for marking nodes is that we needed to bound the rank (number of children) r of any node. We will now show that any node with n descendants has rank O(log n).

LEMMA 11.1.

Let x be any node in a Fibonacci heap. Let ci be the ith youngest child of x. Then the rank of ci is at least i - 2.

PROOF:

At the time when ci was linked to x, x already had (older) children c1, c2, . . . , ci-1. Thus, x had at least i - 1 children when it linked to ci. Since nodes are only linked if they have the same rank, it follows that at the time that ci was linked to x, ci had at least i - 1 children. Since that time, it could have lost at most one child, or else it would have been cut from x. Thus, ci has at least i - 2 children.

From Lemma 11.1, it is easy to show that any node of rank r must have a lot of descendants.

LEMMA 11.2.

Let Fk be the Fibonacci numbers defined (in Section 1.2) by F0 = 1, F1 = 1, and Fk = Fk-1 + Fk-2. Any node of rank r [image] 1 has at least Fr+1 descendants (including itself).

PROOF:

Let Sr be the smallest tree of rank r. Clearly, S0 = 1 and S1 = 2. By Lemma 11.1, a tree of rank r must have subtrees of rank at least r - 2, r - 3, . . . , 1, and 0, plus another subtree, which has at least one node. Along with the root of Sr itself, this gives a minimum value for Sr>1 of [image]. It is easy to show that Sr = Fr+1 (Exercise 1.9a).

Because it is well known that the Fibonacci numbers grow exponentially, it immediately follows that any node with s descendants has rank at most O(log s). Thus, we have

LEMMA 11.3.

The rank of any node in a Fibonacci heap is O(log n).

PROOF:

Immediate from the discussion above.

If all we were concerned about was the time bounds for the merge, insert, and delete_min operations, then we could stop here and prove the desired amortized time bounds. Of course, the whole point of Fibonacci heaps is to obtain an O(1) time bound for decrease_key as well.

The actual time required for a decrease_key operation is 1 plus the number of cascading cuts that are performed during the operation. Since the number of cascading cuts could be much more than O(1), we will need to pay for this with a loss in potential. If we look at Figure 11.20, we see that the number of trees actually increases with each cascading cut, so we will have to enhance the potential function to include something that decreases during cascading cuts. Notice that we cannot just throw out the number of trees from the potential function, since then we will not be able to prove the time bound for the merge operation. Looking at Figure 11.20 again, we see that a cascading cut causes a decrease in the number of marked nodes, because each node that is the victim of a cascading cut becomes an unmarked root. Since each cascading cut costs 1 unit of actual time and increases the tree potential by 1, we will count each marked node as two units of potential. This way, we have a chance of canceling out the number of cascading cuts.

THEOREM 11.4.

The amortized time bounds for Fibonacci heaps are O(1) for insert, merge, and decrease_key and O(log n) for delete_min.

PROOF:

The potential is the number of trees in the collection of Fibonacci heaps plus twice the number of marked nodes. As usual, the initial potential is 0 and is always nonnegative. Thus, over a sequence of operations, the total amortized time is an upper bound on the total actual time.

For the merge operation, the actual time is constant, and the number of trees and marked nodes is unchanged, so, by Equation (11.2), the amortized time is O(1).

For the insert operation, the actual time is constant, the number of trees increases by 1, and the number of marked nodes is unchanged. Thus, the potential increases by at most 1, so the amortized time is O(1).

For the delete_min operation, let r be the rank of the tree that contains the minimum element, and let T be the number of trees before the operation. To perform a delete_min, we once again split the children of a tree, creating an additional r new trees. Notice that, although this can remove marked nodes (by making them unmarked roots), this cannot create any additional marked nodes. These r new trees, along with the other T trees, must now be merged, at a cost of T + r + log n = T + O(log n), by Lemma 11.3. Since there can be at most O(log n) trees, and the number of marked nodes cannot increase, the potential change is at most O(log n) - T. Adding the actual time and potential change gives the O(log n) amortized bound for delete_min.

Finally, for the decrease_key operation, let C be the number of cascading cuts. The actual cost of a decrease_key is C + 1, which is the total number of cuts performed. The first (noncascading) cut creates a new tree and thus increases the potential by 1. Each cascading cut creates a new tree, but converts a marked node to an unmarked (root) node, for a net loss of one unit per cascading cut. The last cut also can convert an unmarked node (in Figure 11.20 it is node 5) into a marked node, thus increasing the potential by 2. The total change in potential is thus 3 - C. Adding the actual time and the potential change gives a total of 4, which is O (1).

11.5. Splay Trees

As a final example, we analyze the running time of splay trees. Recall, from Chapter 4, that after an access of some item x is performed, a splaying step moves x to the root by a series of three operations: zig, zig-zag, and zig-zig. These tree rotations are shown in Figure 11.21. We adopt the convention that if a tree rotation is being performed at node x, then prior to the rotation p is its parent and g is its grandparent (if x is not the child of the root).

Recall that the time required for any tree operation on node x is proportional to the number of nodes on the path from the root to x. If we count each zig operation as one rotation and each zig-zig or zig-zag as two rotations, then the cost of any access is equal to 1 plus the number of rotations.

In order to show an O(log n) amortized bound for the splaying step, we need a potential function which can increase by at most O(log n) over the entire splaying step, but which will also cancel out the number of rotations performed during the step. It is not at all easy to find a potential function that satisfies these criteria. A simple first guess at a potential function might be the sum of the depths of all the nodes in the tree. This does not work, because the potential can increase by [image] (n) during an access. A canonical example of this occurs when elements are inserted in sequential order.

A potential function [image], which does work, is defined as

S(i) represents the number of descendants of i (including i itself). The potential function is the sum, over all nodes i in the tree T, of the logarithm of S(i).

Figure 11.21 zig, zig-zag, and zig-zig operations; each has a symmetric case (not shown)

To simplify the notation, we will define

R(i) = logS(i).

This makes

R(i) represents the rank of node i. The terminology is similar to what we used in the analysis of the disjoint set algorithm, binomial queues, and Fibonacci heaps. In all these data structures, the meaning of rank is somewhat different, but is generally meant to be on the order (magnitude) of the logarithm of the size of the tree. For a tree T with n nodes, the rank of the root is simply R(T) = log n. Using the sum of ranks as a potential function is similar to using the sum of heights as a potential function. The important difference is that while a rotation can change the heights of many nodes in the tree, only x, p, and g can have their ranks changed.

Before proving the main theorem, we need the following lemma.

LEMMA 11.4.

If a + b [image] c, and a and b are both positive integers, then

log a + log b [image] 2 log c - 2.

PROOF:

By the arithmetic-geometric mean inequality,

Thus

Squaring both sides gives

ab    c2/4

Taking logarithms of both sides proves the lemma.

With the preliminaries taken care of, we are ready to prove the main theorem.

THEOREM 11.5.

The amortized time to splay a tree with root T at node x is at most 3(R(T) - R(x)) + 1 = O(log n).

PROOF:

The potential function is the sum of the ranks of the nodes in T.

If x is the root of T, then there are no rotations, so there is no potential change. The actual time is 1 to access the node, thus the amortized time is 1 and the theorem is true. Thus, we may assume that there is at least one rotation.

For any splaying step, let Ri(x) and Si(x) be the rank and size of x before the step, and let Rf(x) and Sf(x) be the rank and size of x immediately after the splaying step. We will show that the amortized time required for a zig is at most 3(Rf(x) - Ri(x)) + 1 and that the amortized time for either a zig-zag or zig-zig is at most 3(Rf(x) - Ri(x)). We will show that when we add over all steps, the sum telescopes to the desired time bound.

Zig step: For the zig step, the actual time is 1 (for the single rotation), and the potential change is Rf(x) + Rf(p) - Ri(x) - Ri(p). Notice that the potential change is easy to compute, because only x and p's trees change size. Thus

ATzig = 1 + Rf(x) + Rf(p) - Ri(x) - Ri(p)

From Figure 11.21 we see that Si(p) [image] Sf(p); thus, it follows that Ri(p) [image] Rf(p). Thus,

ATzig    1 + Rf(x) - Ri(x).

Since Sf(x) [image] Si(x), it follows that Rf(x) - Ri(x) [image] 0, so we may increase the right side, obtaining

ATzig    1 + 3(Rf(x) - Ri(x)).

Zig-zag step: For the zig-zag case, the actual cost is 2, and the potential change is Rf(x) + Rf(p) + Rf(g) - Ri(x) - Ri(p) - Ri(g). This gives an amortized time bound of

ATzig-zag = 2 + Rf(x) + Rf(p) + Rf(g) - Ri(x) - Ri(p) - Ri(g).

From Figure 11.21 we see that Sf(x) = Si(g), so their ranks must be equal. Thus, we obtain

ATzig-zag = 2 + Rf(p) + Rf(g) - Ri(x) - Ri(p).

We also see that Si(p) [image] Si(x). Consequently, Ri(x) [image] Ri(p). Making this substitution gives

ATzig-zag    2 + Rf(p) + Rf(g) - 2Ri(x).

From Figure 11.21 we see that Sf(p) + Sf(g) [image] Sf(x). If we apply Lemma 11.4, we obtain

log Sf(p) + log Sf(g)    2 log Sf(x) - 2.

By definition of rank, this becomes

Rf(p) + Rf(g)    2Rf(x) - 2.

Substituting this we obtain

ATzig-zag    2Rf(x) - 2Ri(x)

   2(Rf(x) - Ri(x))

Since Rf(x) [image] Ri(x), we obtain

ATzig-zag    3(Rf(x) - Ri(x)).

Zig-zig step: The third case is the zig-zig. The proof of this case is very similar to the zig-zag case. The important inequalities are Rf(x) = Ri(g), Rf(x) [image] Rf(p), Ri(x) [image] Ri(p), and Si(x) + Sf(g) [image] Sf(x). We leave the details as Exercise 11.8.

Figure 11.22 The splaying steps involved in splaying at node 2

The amortized cost of an entire splay is the sum of the amortized costs of each splay step. Figure 11.22 shows the steps which are performed in a splay at node 2. Let R1 (2), R2(2), R3(2), and R4(2) be the rank of node 2 in each of the four trees. The cost of the first step, which is a zig-zag, is at most 3(R2(2) - R1 (2)). The cost of the second step, which is a zig-zig, is 3(R3(2) - R2(2)). The last step is a zig and has a cost no larger than 3(R4(2) - R3(2)) + 1. The total cost thus telescopes to 3(R4(2) - R1(2)) + 1.

In general, by adding up the amortized costs of all the rotations, of which at most one can be a zig, we see that the total amortized cost to splay at node x is at most 3(Rf(x) - Ri(x)) + 1, where Ri(x) is the rank of x before the first splaying step and Rf(x) is the rank of x after the last splaying step. Since the last splaying step leaves x at the root, we obtain an amortized bound of 3(Rf(T) - Ri(x)) + 1, which is O(log n).

Because every operation on a splay tree requires a splay, the amortized cost of any operation is within a constant factor of the amortized cost of a splay. Thus, all splay tree operations take O(log n) amortized time. By using a more general potential function, it is possible to show that splay trees have several remarkable properties. This is discussed in more detail in the exercises.

Summary

In this chapter, we have seen how an amortized analysis can be used to apportion charges among operations. To perform the analysis, we invent a fictitious potential function. The potential function measures the state of the system. A high-potential data structure is volatile, having been built on relatively cheap operations. When the expensive bill comes for an operation, it is paid for by the savings of previous operations. One can view potential as standing for potential for disaster, in that very expensive operations can only occur when the data structure has a high potential and has used considerably less time than has been allocated.

Low potential in a data structure means that the cost of each operation has been roughly equal to the amount allocated for it. Negative potential means debt-- more time has been spent than has been allocated, so the allocated (or amortized) time is not a meaningful bound.

As expressed by Equation (11.2), the amortized time for an operation is equal to the sum of the actual time and potential change. Taken over an entire sequence of operations, the amortized time for the sequence is equal to the total sequence time plus the net change in potential. As long as this net change is positive, then the amortized bound provides an upper bound for the actual time spent and is meaningful.

The keys to choosing a potential function are to guarantee that the minimum potential occurs at the beginning of the algorithm, and to have the potential increase for cheap operations and decrease for expensive operations. It is important that the excess or saved time be measured by an opposite change in potential. Unfortunately, this is sometimes easier said than done.

Exercises

11.1 When do m consecutive insertions into a binomial queue take less than 2m time units?

11.2 Suppose a binomial queue of n = 2k - 1 elements is built. Alternately perform m insert and delete_min pairs. Clearly, each operation takes O(log n) time. Why does this not contradict the amortized bound of O(1) for insertion?

*11.3 Show that the amortized bound of O(log n) for the skew heap operations described in the text cannot be converted to a worst-case bound, by giving a sequence of operations that lead to a merge requiring [image] (n) time.

*11.4 Show how to merge two skew heaps with one top-down pass and reduce the merge cost to O(1) amortized time.

11.5 Extend skew heaps to support the decrease_key operation in O(log n) amortized time.

11.6 Implement Fibonacci heaps and compare their performance with binary heaps when used in Dijkstra's algorithm.

11.7 A standard implementation of Fibonacci heaps requires four pointers per node (parent, child, and two siblings). Show how to reduce the number of pointers, at the cost of at most a constant factor in the running time.

11.8 Show that the amortized time of a zig-zig splay is at most 3(Rf(x) - Ri(x)).

11.9 By changing the potential function, it is possible to prove different bounds for splaying. Let the weight function W(i) be some function assigned to each node in the tree, and let S(i) be the sum of the weights of all the nodes in the subtree rooted at i, including i itself. The special case W(i) = 1 for all nodes corresponds to the function used in the proof of the splaying bound. Let n be the number of nodes in the tree, and let m be the number of accesses. Prove the following two theorems:

a. The total access time is O(m + (m + n)log n).

*b. If qi is the number of times that item i is accessed, and qi > 0 for all i, then the total access time is

11.10 a. Show how to implement the merge operation on splay trees so that a sequence of n -1 merges starting from n single-element trees takes O(n log2 n) time.

*b. Improve the bound to O(n log n).

11.11 In Chapter 5, we described rehasing: When a table becomes more than half full, a new table twice as large is constructed, and the entire old table is rehashed. Give a formal amortized analysis, with potential function, to show that the amortized cost of an insertion is still O(1).

11.12 Show that if deletions are not allowed, then any sequence of m insertions into an n node 2-3 tree produces O(m + n) node splits.

11.13 A deque with heap order is a data structure consisting of a list of items, on which the following operations are possible:

push(x,d): Insert item x on the front end of deque d.

pop(d): Remove the front item from deque d and return it.

inject(x,d): Insert item x on the rear end of deque d.

eject(d): Remove the rear item from deque d and return it.

find _min(d): Return the smallest item from deque d (breaking ties arbitrarily).

a. Describe how to support these operations in constant amortized time per operation.

**b. Describe how to support these operations in constant worst-case time per operation.

References

An excellent survey of amortized analysis is provided in [9].

Most of the references below duplicate citations in earlier chapters. We cite them again for convenience and completeness. Binomial queues were first described in [10] and analyzed in [1]. Solutions to 11.3 and 11.4 appear in [8]. Fibonacci heaps are described in [3]. Exercise 11.9a shows that splay trees are optimal, to within a constant factor of the the best static search trees. 11.9b shows that splay trees are optimal, to within a constant factor of the best optimal search trees. These, as well as two other strong results, are proved in the original splay tree paper [6].

The merge operation for splay trees is described in [5]. Exercise 11.12 is solved, with an implicit use of amortization, in [2]. The paper also shows how to merge 2-3 trees efficiently. A solution to 11.13 can be found in [4].

Amortized analysis is used in [7] to design an on-line algorithm that processes a series of queries in time only a constant factor larger than any off-line algorithm in its class.

1. M. R. Brown, "Implementation and Analysis of Binomial Queue Algorithms," SIAM Journal on Computing 7 (1978), 298-319.

2. M. R. Brown and R. E. Tarjan, "Design and Analysis of a Data Structure for Representing Sorted Lists," SIAM Journal on Computing 9 (1980), 594-614.

3. M. L. Fredman and R. E. Tarjan, "Fibonacci Heaps and Their Uses in Improved Network Optimization Algorithms," Journal of the ACM 34 (1987), 596-615.

4. H. Gajewska and R. E. Tarjan, "Deques with Heap Order," Information Processing Letters 22 (1986), 197-200.

5. G. Port and A. Moffat, "A Fast Algorithm for Melding Splay Trees," Proceedings of the First Workshop on Algorithms and Data Structures, 1989, 450-459.

6. D. D. Sleator and R. E. Tarjan, "Self-adjusting Binary Search Trees," Journal of the ACM 32 (1985), 652-686.

7. D. D. Sleator and R. E. Tarjan, "Amortized Efficiency of List Update and Paging Rules," Communications of the ACM 28 (1985), 202-208.

8. D. D. Sleator and R. E. Tarjan, "Self-adjusting Heaps," SIAM Journal on Computing 15 (1986), 52-69.

9. R. E. Tarjan, "Amortized Computational Complexity," SIAM Journal on Algebraic and Discrete Methods 6 (1985), 306-318.

10. J. Vuillemin, "A Data Structure for Manipulating Priority Queues," Communications of the ACM 21 (1978), 309-314.

Return to Table of Contents


Structures, Algorithm Analysis: CHAPTER 2: ALGORITHM ANALYSIS

 Return to Table of Contents

CHAPTER 2: ALGORITHM ANALYSIS

An algorithm is a clearly specified set of simple instructions to be followed to solve a problem. Once an algorithm is given for a problem and decided (somehow) to be correct, an important step is to determine how much in the way of resources, such as time or space, the algorithm will require. An algorithm that solves a problem but requires a year is hardly of any use. Likewise, an algorithm that requires a gigabyte of main memory is not (currently) useful.

In this chapter, we shall discuss

How to estimate the time required for a program.

How to reduce the running time of a program from days or years to fractions of a second.

The results of careless use of recursion.

Very efficient algorithms to raise a number to a power and to compute the greatest common divisor of two numbers.

2.1. Mathematical Background

The analysis required to estimate the resource use of an algorithm is generally a theoretical issue, and therefore a formal framework is required. We begin with some mathematical definitions.

Throughout the book we will use the following four definitions:

DEFINITION: T(n) = O(f(n)) if there are constants c and n0 such that T(n) [image] cf (n) when n [image] n0.

DEFINITION: T(n) = [image] (g(n)) if there are constants c and n0 such that T(n) [image] cg(n) when n [image] n0.

DEFINITION: T(n) = [image] (h(n)) if and only if T(n) = O(h(n)) and T(n) = [image] (h(n)).

DEFINITION: T(n) = o(p(n)) if T(n) = O(p(n)) and T(n) [image] [image] (p(n)).

The idea of these definitions is to establish a relative order among functions. Given two functions, there are usually points where one function is smaller than the other function, so it does not make sense to claim, for instance, f(n) < g(n). Thus, we compare their relative rates of growth. When we apply this to the analysis of algorithms, we shall see why this is the important measure.

Although 1,000n is larger than n2 for small values of n, n2 grows at a faster rate, and thus n2 will eventually be the larger function. The turning point is n = 1,000 in this case. The first definition says that eventually there is some point n0 past which c [image] f (n) is always at least as large as T(n), so that if constant factors are ignored, f(n) is at least as big as T(n). In our case, we have T(n) = 1,000n, f(n) = n2, n0 = 1,000, and c = 1. We could also use n0 = 10 and c = 100. Thus, we can say that 1,000n = O(n2) (order n-squared). This notation is known as Big-Oh notation. Frequently, instead of saying "order . . . ," one says "Big-Oh . . . ."

If we use the traditional inequality operators to compare growth rates, then the first definition says that the growth rate of T(n) is less than or equal to ( [image]) that of f(n). The second definition, T(n) = [image] (g(n)) (pronounced "omega"), says that the growth rate of T(n) is greater than or equal to ( [image]) that of g(n). The third definition, T(n) = [image] (h(n)) (pronounced "theta"), says that the growth rate of T(n) equals ( = ) the growth rate of h(n). The last definition, T(n) = o(p(n)) (pronounced "little-oh"), says that the growth rate of T(n) is less than (<) the growth rate of p(n). This is different from Big-Oh, because Big-Oh allows the possibility that the growth rates are the same.

To prove that some function T(n) = O(f(n)), we usually do not apply these definitions formally but instead use a repertoire of known results. In general, this means that a proof (or determination that the assumption is incorrect) is a very simple calculation and should not involve calculus, except in extraordinary circumstances (not likely to occur in an algorithm analysis).

When we say that T(n) = O(f(n)), we are guaranteeing that the function T(n) grows at a rate no faster than f(n); thus f(n) is an upper bound on T(n). Since this implies that f(n) = [image] (T(n)), we say that T(n) is a lower bound on f(n).

As an example, n3 grows faster than n2, so we can say that n2 = O(n3) or n3 = [image] (n2). f(n) = n2 and g(n) = 2n2 grow at the same rate, so both f(n) = O(g(n)) and f(n) = [image] (g(n)) are true. When two functions grow at the same rate, then the decision whether or not to signify this with [image] () can depend on the particular context. Intuitively, if g(n) = 2n2, then g(n) = O(n4), g(n) = O(n3), and g(n) = O(n2) are all technically correct, but obviously the last option is the best answer. Writing g(n) = [image] (n2) says not only that g(n) = O(n2), but also that the result is as good (tight) as possible.

The important things to know are

RULE 1:

If T1(n) = O(f(n)) and T2(n) = O(g(n)), then

(a) T1(n) + T2(n) = max (O(f(n)), O(g(n))),

(b) T1(n) * T2(n) = O(f(n) * g(n)),

Function     Name

--------------------

  c        Constant

  logn     Logarithmic

  log2n    Log-squared

  n        Linear

  n log n

  n2       Quadratic

  n3       Cubic

  2n       Exponential

Figure 2.1 Typical growth rates

RULE 2:

If T(x) is a polynomial of degree n, then T(x) =   [image] (xn).

RULE 3:

logk n = O(n) for any constant k. This tells us that logarithms grow very slowly.

To see that rule 1(a) is correct, note that by definition there exist four constants c1, c2, n1, and n2 such that T1(n)   [image]  c1 f(n) for n   [image]  n1 and T2(n)   [image]  c2g(n) for n   [image]  n2. Let n0 = max(n1, n2). Then, for n   [image]  n0, T1(n)   [image]  c1f(n) and T2(n)   [image]  c2g(n), so that T1(n) + T2(n)   [image]  c1f(n) + c2g(n). Let c3 = max(c1, c2). Then,

           T1(n) + T2(n)    c3f(n) + c3g(n)

                            c3(f(n) + g(n))

                            2c3 max(f(n), g(n))

                            c max(f(n), g(n))

for c = 2c3 and n   [image]  n0.

We leave proofs of the other relations given above as exercises for the reader. This information is sufficient to arrange most of the common functions by growth rate (see Fig. 2.1).

Several points are in order. First, it is very bad style to include constants or low-order terms inside a Big-Oh. Do not say T(n) = O(2n2) or T(n) = O(n2 + n). In both cases, the correct form is T(n) = O(n2). This means that in any analysis that will require a Big-Oh answer, all sorts of shortcuts are possible. Lower-order terms can generally be ignored, and constants can be thrown away. Considerably less precision is required in these cases.

Secondly, we can always determine the relative growth rates of two functions f(n) and g(n) by computing limn  [image]   [image]  f(n)/g(n), using L'Hôpital's rule if necessary.*

*L'Hôpital's rule states that if limn  [image]   [image]  f(n) =   [image]  and limn  [image]   [image]  g(n) =   [image], then limn  [image]   [image]  f(n)/g(n) = limn  [image]   [image]   f'(n) / g'(n), where f'(n) and g'(n) are the derivatives of f(n) and g(n), respectively.

The limit can have four possible values:

       The limit is 0: This means that f(n) = o(g(n)).

       The limit is c   [image]  0: This means that f(n) =   [image] (g(n)).

       The limit is   [image]: This means that g(n) = o(f(n)).

       The limit oscillates: There is no relation (this will not happen in our context).

Using this method almost always amounts to overkill. Usually the relation between f(n) and g(n) can be derived by simple algebra. For instance, if f(n) = n log n and g(n) = n1.5, then to decide which of f(n) and g(n) grows faster, one really needs to determine which of log n and n0.5 grows faster. This is like determining which of log2 n or n grows faster. This is a simple problem, because it is already known that n grows faster than any power of a log. Thus, g(n) grows faster than f(n).

One stylistic note: It is bad to say f(n)   [image]  O(g(n)), because the inequality is implied by the definition. It is wrong to write f(n)   [image]  O(g(n)), which does not make sense.

2.2. Model

In order to analyze algorithms in a formal framework, we need a model of computation. Our model is basically a normal computer, in which instructions are executed sequentially. Our model has the standard repertoire of simple instructions, such as addition, multiplication, comparison, and assignment, but, unlike real computers, it takes exactly one time unit to do anything (simple). To be reasonable, we will assume that, like a modern computer, our model has fixed size (say 32-bit) integers and that there are no fancy operations, such as matrix inversion or sorting, that clearly cannot be done in one time unit. We also assume infinite memory.

This model clearly has some weaknesses. Obviously, in real life, not all operations take exactly the same time. In particular, in our model one disk read counts the same as an addition, even though the addition is typically several orders of magnitude faster. Also, by assuming infinite memory, we never worry about page faulting, which can be a real problem, especially for efficient algorithms. This can be a major problem in many applications.

2.3. What to Analyze

The most important resource to analyze is generally the running time. Several factors affect the running time of a program. Some, such as the compiler and computer used, are obviously beyond the scope of any theoretical model, so, although they are important, we cannot deal with them here. The other main factors are the algorithm used and the input to the algorithm.

Typically, the size of the input is the main consideration. We define two functions, Tavg(n) and Tworst(n), as the average and worst-case running time, respectively, used by an algorithm on input of size n. Clearly, Tavg(n)   [image]  Tworst(n). If there is more than one input, these functions may have more than one argument.

We remark that generally the quantity required is the worst-case time, unless otherwise specified. One reason for this is that it provides a bound for all input, including particularly bad input, that an average-case analysis does not provide. The other reason is that average-case bounds are usually much more difficult to compute. In some instances, the definition of "average" can affect the result. (For instance, what is average input for the following problem?)

As an example, in the next section, we shall consider the following problem:

MAXIMUM SUBSEQUENCE SUM PROBLEM:

Given (possibly negative) integers a1, a2, . . . , an, find the maximum value of   [image]. (For convenience, the maximum subsequence sum is 0 if all the integers are negative.)

Example:

For input -2, 11, -4, 13, -5, -2, the answer is 20 (a2 through a4).

This problem is interesting mainly because there are so many algorithms to solve it, and the performance of these algorithms varies drastically. We will discuss four algorithms to solve this problem. The running time on some computer (the exact computer is unimportant) for these algorithms is given in Figure 2.2.

There are several important things worth noting in this table. For a small amount of input, the algorithms all run in a blink of the eye, so if only a small amount of input is expected, it might be silly to expend a great deal of effort to design a clever algorithm. On the other hand, there is a large market these days for rewriting programs that were written five years ago based on a no-longer-valid assumption of small input size. These programs are now too slow, because they used poor algorithms. For large amounts of input, Algorithm 4 is clearly the best choice (although Algorithm 3 is still usable).

Second, the times given do not include the time required to read the input. For Algorithm 4, the time merely to read in the input from a disk is likely to be an order of magnitude larger than the time required to solve the problem. This is typical of many efficient algorithms. Reading the data is generally the bottleneck; once the data are read, the problem can be solved quickly. For inefficient algorithms this is not true, and significant computer resources must be used. Thus, it is important that, where possible, algorithms be efficient enough not to be the bottleneck of a problem.

      Algorithm           1          2         3        4

--------------------    -----------------------------------

         Time           O(n3)      O(n2)  O(n log n)  (n)

------------------------------------------------------------

  Input  n = 10         0.00103    0.00045  0.00066  0.00034

  Size   n = 100        0.47015    0.01112  0.00486  0.00063

         n = 1,000    448.77       1.1233   0.05843  0.00333

         n = 10,000      NA      111.13     0.68631  0.03042

         n = 100,000     NA         NA      8.0113   0.29832

Figure 2.2 Running time of several algorithms for maximum subsequence sum (in seconds)

 Figure 2.3 shows the growth rates of the running times of the four algorithms. Even though this graph encompasses only values of n ranging from 10 to 100, the relative growth rates are still evident. Although the graph for Algorithm 3 seems linear, it is easy to verify that it is not, by using a straightedge (or piece of paper).  Figure 2.4 shows the performance for larger values. It dramatically illustrates how useless inefficient algorithms are for even moderately large amounts of input.

Figure 2.3 Plot (n vs. milliseconds) of various maximum subsequence sum algorithms

Figure 2.4 Plot (n vs. seconds) of various maximum subsequence sum algorithms

2.4. Running Time Calculations

There are several ways to estimate the running time of a program. The previous table was obtained empirically. If two programs are expected to take similar times, probably the best way to decide which is faster is to code them both up and run them!

Generally, there are several algorithmic ideas, and we would like to eliminate the bad ones early, so an analysis is usually required. Furthermore, the ability to do an analysis usually provides insight into designing efficient algorithms. The analysis also generally pinpoints the bottlenecks, which are worth coding carefully.

To simplify the analysis, we will adopt the convention that there are no particular units of time. Thus, we throw away leading constants. We will also throw away low-order terms, so what we are essentially doing is computing a Big-Oh running time. Since Big-Oh is an upper bound, we must be careful to never underestimate the running time of the program. In effect, the answer provided is a guarantee that the program will terminate within a certain time period. The program may stop earlier than this, but never later.

2.4.1. A Simple Example

Here is a simple program fragment to calculate

unsigned int

sum( int n )

{

unsigned int i, partial_sum;

/*1*/       partial_sum = 0;

/*2*/       for( i=1; i<=n; i++ )

/*3*/            partial_sum += i*i*i;

/*4*/       return( partial_sum );

}

The analysis of this program is simple. The declarations count for no time. Lines 1 and 4 count for one unit each. Line 3 counts for three units per time executed (two multiplications and one addition) and is executed n times, for a total of 3n units. Line 2 has the hidden costs of initializing i, testing i   [image]  n, and incrementing i. The total cost of all these is 1 to initialize, n + 1 for all the tests, and n for all the increments, which is 2n + 2. We ignore the costs of calling the function and returning, for a total of 5n + 4. Thus, we say that this function is O (n).

If we had to perform all this work every time we needed to analyze a program, the task would quickly become infeasible. Fortunately, since we are giving the answer in terms of Big-Oh, there are lots of shortcuts that can be taken without affecting the final answer. For instance, line 3 is obviously an O (1) statement (per execution), so it is silly to count precisely whether it is two, three, or four units -- it does not matter. Line 1 is obviously insignificant compared to the for loop, so it is silly to waste time here. This leads to several obvious general rules.

2.4.2. General Rules

RULE 1-FOR LOOPS:

The running time of a for loop is at most the running time of the statements inside the for loop (including tests) times the number of iterations.

RULE 2-NESTED FOR LOOPS:

Analyze these inside out. The total running time of a statement inside a group of nested for loops is the running time of the statement multiplied by the product of the sizes of all the for loops.

As an example, the following program fragment is O(n2):

for( i=0; i<n; i++ )

for( j=0; j<n; j++ )

k++;

RULE 3-CONSECUTIVE STATEMENTS:

These just add (which means that the maximum is the one that counts -- see 1(a) on page 16).

As an example, the following program fragment, which has O(n) work followed by O (n2) work, is also O (n2):

for( i=0; i<n; i++)

a[i] = 0;

for( i=0; i<n; i++ )

for( j=0; j<n; j++ )

a[i] += a[j] + i + j;

RULE 4-lF/ELSE:

For the fragment

if( cond )

S1

else

S2

the running time of an if/else statement is never more than the running time of the test plus the larger of the running times of S1 and S2.

Clearly, this can be an over-estimate in some cases, but it is never an under-estimate.

Other rules are obvious, but a basic strategy of analyzing from the inside (or deepest part) out works. If there are function calls, obviously these must be analyzed first. If there are recursive procedures, there are several options. If the recursion is really just a thinly veiled for loop, the analysis is usually trivial. For instance, the following function is really just a simple loop and is obviously O (n):

unsigned int

factorial( unsigned int n )

{

if( n <= 1 )

return 1;

else

return( n * factorial(n-1) );

}

This example is really a poor use of recursion. When recursion is properly used, it is difficult to convert the recursion into a simple loop structure. In this case, the analysis will involve a recurrence relation that needs to be solved. To see what  might happen, consider the following program, which turns out to be a horrible use of recursion:

/* Compute Fibonacci numbers as described Chapter 1 */

unsigned int

fib( unsigned int n )

{

/*1*/       if( n <= 1 )

/*2*/            return 1;

else

/*3*/            return( fib(n-1) + fib(n-2) );

}

At first glance, this seems like a very clever use of recursion. However, if the program is coded up and run for values of n around 30, it becomes apparent that this program is terribly inefficient. The analysis is fairly simple. Let T(n) be the running time for the function fib(n). If n = 0 or n = 1, then the running time is some constant value, which is the time to do the test at line 1 and return. We can say that T(0) = T(1) = 1, since constants do not matter. The running time for other values of n is then measured relative to the running time of the base case. For n > 2, the time to execute the function is the constant work at line 1 plus the work at line 3. Line 3 consists of an addition and two function calls. Since the function calls are not simple operations, they must be analyzed by themselves. The first function call is fib(n - 1) and hence, by the definition of T, requires T(n - 1) units of time. A similar argument shows that the second function call requires T(n - 2) units of time. The total time required is then T(n - 1) + T(n - 2) + 2, where the 2 accounts for the work at line 1 plus the addition at line 3. Thus, for n   [image]  2, we have the following formula for the running time of fib(n):

T(n) = T(n - 1) + T(n - 2) + 2

Since fib(n) = fib(n - 1) + fib(n - 2), it is easy to show by induction that T(n)   [image]  fib(n). In Section 1.2.5, we showed that fib(n) < (5/3)  . A similar calculation shows that fib(n)   [image]  (3/2)  , and so the running time of this program grows exponentially. This is about as bad as possible. By keeping a simple array and using a for loop, the running time can be reduced substantially.

This program is slow because there is a huge amount of redundant work being performed, violating the fourth major rule of recursion (the compound interest rule), which was discussed in Section 1.3. Notice that the first call on line 3, fib(n - 1), actually computes fib(n - 2) at some point. This information is thrown away and recomputed by the second call on line 3. The amount of information thrown away compounds recursively and results in the huge running time. This is perhaps the finest example of the maxim "Don't compute anything more than once" and should not scare you away from using recursion. Throughout this book, we shall see outstanding uses of recursion.

2.4.3 Solutions for the Maximum Subsequence Sum Problem

We will now present four algorithms to solve the maximum subsequence sum problem posed earlier. The first algorithm is depicted in Figure 2.5. The indices in the for loops reflect the fact that, in C, arrays begin at 0, instead of 1. Also, the algorithm computes the actual subsequences (not just the sum); additional code is required to transmit this information to the calling routine.

Convince yourself that this algorithm works (this should not take much). The running time is O(n  ) and is entirely due to lines 5 and 6, which consist of an O(1) statement buried inside three nested for loops. The loop at line 2 is of size n.

int

max_subsequence_sum( int a[], unsigned int n )

{

int this_sum, max_sum, best_i, best_j, i, j, k;

/*1*/       max_sum = 0; best_i = best_j = -1;

/*2*/       for( i=0; i<n; i++ )

/*3*/           for( j=i; j<n; j++ )

{

/*4*/                this_sum=0;

/*5*/                for( k = i; k<=j; k++ )

/*6*/                     this_sum += a[k];

/*7*/                if( this_sum > max_sum )

{    /* update max_sum, best_i, best_j */

/*8*/                      max_sum = this_sum;

/*9*/                      best_i = i;

/*10*/                     best_j = j;

}

}

/*11*/      return( max_sum );

}

Figure 2.5 Algorithm 1

The second loop has size n - i + 1, which could be small, but could also be of size n. We must assume the worst, with the knowledge that this could make the final bound a bit high. The third loop has size j - i + 1, which, again, we must assume is of size n. The total is O(1   [image]  n   [image]  n   [image]  n) = O(n  ). Statement 1 takes only O(1) total, and statements 7 to 10 take only O(n  ) total, since they are easy statements inside only two loops.

It turns out that a more precise analysis, taking into account the actual size of these loops, shows that the answer is   [image] (n  ), and that our estimate above was a factor of 6 too high (which is all right, because constants do not matter). This is generally true in these kinds of problems. The precise analysis is obtained from the sum   [image]  1, which tells how many times line 6 is executed. The sum can be evaluated inside out, using formulas from Section 1.2.3. In particular, we will use the formulas for the sum of the first n integers and first n squares. First we have

Next we evaluate

This sum is computed by observing that it is just the sum of the first n - i + 1 integers. To complete the calculation, we evaluate

We can avoid the cubic running time by removing a for loop. Obviously, this is not always possible, but in this case there are an awful lot of unnecessary computations present in the algorithm. The inefficiency that the improved algorithm corrects can be seen by noticing that   [image]  so the computation at lines 5 and 6 in Algorithm 1 is unduly expensive. Figure 2.6 shows an improved algorithm. Algorithm 2 is clearly O(n  ); the analysis is even simpler than before.

There is a recursive and relatively complicated O(n log n) solution to this problem, which we now describe. If there didn't happen to be an O(n) (linear) solution, this would be an excellent example of the power of recursion. The algorithm uses a "divide-and-conquer" strategy. The idea is to split the problem into two roughly equal subproblems, each of which is half the size of the original. The subproblems are then solved recursively. This is the "divide" part. The "conquer" stage consists of patching together the two solutions of the subproblems, and possibly doing a small amount of additional work, to arrive at a solution for the whole problem.

int

max_subsequence_sum( int a[], unsigned int n )

{

int this_sum, max_sum, best_i, best_j, i, j, k;

/*1*/       max_sum = 0; best_i = best_j = -1;

/*2*/       for( i=0; i<n; i++ )

{

/*3*/            this_sum = 0;

/*4*/            for( j=i; j<n; j++ )

{

/*5*/                 this_sum += a[j];

/*6*/                 if( this_sum > max_sum )

/* update max_sum, best_i, best_j */;

}

}

/*7*/       return( max_sum );

}

Figure 2.6 Algorithm 2

In our case, the maximum subsequence sum can be in one of three places. Either it occurs entirely in the left half of the input, or entirely in the right half, or it crosses the middle and is in both halves. The first two cases can be solved recursively. The last case can be obtained by finding the largest sum in the first half that includes the last element in the first half and the largest sum in the second half that includes the first element in the second half. These two sums can then be added together. As an example, consider the following input:

     First Half            Second Half

----------------------------------------

  4   -3   5   -2        -1   2   6   -2

The maximum subsequence sum for the first half is 6 (elements a1 through a3), and for the second half is 8 (elements a6 through a7).

The maximum sum in the first half that includes the last element in the first half is 4 (elements a1 through a4), and the maximum sum in the second half that includes the first element in the second half is 7 (elements a5 though a7). Thus, the maximum sum that spans both halves and goes through the middle is 4 + 7 = 11 (elements a1 through a7).

We see, then, that among the three ways to form a large maximum subsequence, for our example, the best way is to include elements from both halves. Thus, the answer is 11. Figure 2.7 shows an implementation of this strategy.

int

max_sub_sequence_sum( int a[], unsigned int n )

{

return max_sub_sum( a, 0, n-1 );

}

int

max_sub_sum( int a[], int left, int right )

{

int max_left_sum, max_right_sum;

int max_left_border_sum, max_right_border_sum;

int left_border_sum, right_border_sum;

int center, i;

/*1*/       if ( left == right )      /* Base Case */

/*2*/            if( a[left] > 0 )

/*3*/                 return a[left];

else

/*4*/                return 0;

/*5*/       center = (left + right )/2;

/*6*/       max_left_sum = max_sub_sum( a, left, center );

/*7*/       max_right_sum = max_sub_sum( a, center+1, right );

/*8*/       max_left_border_sum = 0; left_border_sum = 0;

/*9*/       for( i=center; i>=left; i-- )

{

/*10*/           left_border_sum += a[i];

/*11*/           if( left_border_sum > max_left_border_sum )

/*12*/                max_left_border_sum = left_border_sum;

}

/*13*/      max_right_border_sum = 0; right_border_sum = 0;

/*14*/      for( i=center+1; i<=right; i++ )

{

/*15*/           right_border_sum += a[i];

/*16*/           if( right_border_sum > max_right_border_sum )

/*17*/                max_right_border_sum = right_border_sum;

}

/*18*/      return max3( max_left_sum, max_right_sum,

max_left_border_sum + max_right_border_sum );

}

Figure 2.7 Algorithm 3

The code for Algorithm 3 deserves some comment. The general form of the call for the recursive procedure is to pass the input array along with the left and right borders, which delimit the portion of the array that is operated upon. A one-line driver program sets this up by passing the borders 0 and n -1 along with the array.

Lines 1 to 4 handle the base case. If left == right, then there is one element, and this is the maximum subsequence if the element is nonnegative. The case left > right is not possible unless n is negative (although minor perturbations in the code could mess this up). Lines 6 and 7 perform the two recursive calls. We can see that the recursive calls are always on a smaller problem than the original, although, once again, minor perturbations in the code could destroy this property. Lines 8 to 12 and then 13 to 17 calculate the two maximum sums that touch the center divider. The sum of these two values is the maximum sum that spans both halves. The pseudoroutine max3 returns the largest of the three possibilities.

Algorithm 3 clearly requires more effort to code than either of the two previous algorithms. However, shorter code does not always mean better code. As we have seen in the earlier table showing the running times of the algorithms, this algorithm is considerably faster than the other two for all but the smallest of input sizes.

The running time is analyzed in much the same way as for the program that computes the Fibonacci numbers. Let T(n) be the time it takes to solve a maximum subsequence sum problem of size n. If n = 1, then the program takes some constant amount of time to execute lines 1 to 4, which we shall call one unit. Thus, T(1) = 1. Otherwise, the program must perform two recursive calls, the two for loops between lines 9 and 17, and some small amount of bookkeeping, such as lines 5 and 18. The two for loops combine to touch every element from a0 to an_1, and there is constant work inside the loops, so the time expended in lines 9 to 17 is O(n). The code in lines 1 to 5, 8, and 18 is all a constant amount of work and can thus be ignored when compared to O(n). The remainder of the work is performed in lines 6 and 7. These lines solve two subsequence problems of size n/2 (assuming n is even). Thus, these lines take T(n/2) units of time each, for a total of 2T(n/2). The total time for the algorithm then is 2T(n/2) + O(n). This gives the equations

T(1) = 1

T(n) = 2T(n/2) + O(n)

To simplify the calculations, we can replace the O(n) term in the equation above with n; since T(n) will be expressed in Big-Oh notation anyway, this will not affect the answer. In Chapter 7, we shall see how to solve this equation rigorously. For now, if T(n) = 2T(n/2) + n, and T(1) = 1, then T(2) = 4 = 2 * 2, T(4) = 12 = 4 * 3, T(8) = 32 = 8 * 4, T(16) = 80 = 16 * 5. The pattern that is evident, and can be derived, is that if n = 2  , then T(n) = n * (k + 1) = n log n + n = O(n log n).

This analysis assumes n is even, since otherwise n/2 is not defined. By the recursive nature of the analysis, it is really valid only when n is a power of 2, since otherwise we eventually get a subproblem that is not an even size, and the equation is invalid. When n is not a power of 2, a somewhat more complicated analysis is required, but the Big-Oh result remains unchanged.

In future chapters, we will see several clever applications of recursion. Here, we present a fourth algorithm to find the maximum subsequence sum. This algorithm is simpler to implement than the recursive algorithm and also is more efficient. It is shown in Figure 2.8.

int

max_subsequence_sum( int a[], unsigned int n )

{

int this_sum, max_sum, best_i, best_j, i, j;

/*1*/       i = this_sum = max_sum = 0; best_i = best_j = -1;

/*2*/       for( j=0; j<n; j++ )

{

/*3*/            this_sum += a[j];

/*4*/            if( this_sum > max_sum )

{       /* update max_sum, best_i, best_j */

/*5*/                 max_sum = this_sum;

/*6*/                 best_i = i;

/*7*/                 best_j = j;

}

else

/*8*/            if( this_sum < 0 )

{

/*9*/                i = j + 1;

/*10*/               this_sum = 0;

}

}

/*11*/      return( max_sum );

}

Figure 2.8 Algorithm 4

It should be clear why the time bound is correct, but it takes a little thought to see why the algorithm actually works. This is left to the reader. An extra advantage of this algorithm is that it makes only one pass through the data, and once a[i] is read and processed, it does not need to be remembered. Thus, if the array is on a disk or tape, it can be read sequentially, and there is no need to store any part of it in main memory. Furthermore, at any point in time, the algorithm can correctly give an answer to the subsequence problem for the data it has already read (the other algorithms do not share this property). Algorithms that can do this are called on-line algorithms. An on-line algorithm that requires only constant space and runs in linear time is just about as good as possible.

2.4.4 Logarithms in the Running Time

The most confusing aspect of analyzing algorithms probably centers around the logarithm. We have already seen that some divide-and-conquer algorithms will run in O(n log n) time. Besides divide-and-conquer algorithms, the most frequent appearance of logarithms centers around the following general rule: An algorithm is O(log n) if it takes constant (O(1)) time to cut the problem size by a fraction (which is usually   [image]). On the other hand, if constant time is required to merely reduce the problem by a constant amount (such as to make the problem smaller by 1), then the algorithm is O(n).

Something that should be obvious is that only special kinds of problems can be O(log n). For instance, if the input is a list of n numbers, an algorithm must take   [image] (n) merely to read the input in. Thus when we talk about O(log n) algorithms for these kinds of problems, we usually presume that the input is preread. We provide three examples of logarithmic behavior.

Binary Search

The first example is usually referred to as binary search:

BINARY SEARCH:

Given an integer x and integers a1, a2, . . . , an, which are presorted and already in memory, find i such that ai = x, or return i = 0 if x is not in the input.

The obvious solution consists of scanning through the list from left to right and runs in linear time. However, this algorithm does not take advantage of the fact that the list is sorted and is thus not likely to be best. The best strategy is to check if x is the middle element. If so, the answer is at hand. If x is smaller than the middle element, we can apply the same strategy to the sorted subarray to the left of the middle element; likewise, if x is larger than the middle element, we look to the right half. (There is also the case of when to stop.) Figure 2.9 shows the code for binary search (the answer is mid). As usual, the code reflects C's convention that arrays begin with index 0. Notice that the variables cannot be declared unsigned (why?). In cases where the unsigned qualifier is questionable, we will not use it. As an example, if the unsigned qualifier is dependent on an array not beginning at zero, we will discard it.

We will also avoid using the unsigned type for variables that are counters in a for loop, because it is common to change the direction of a loop counter from increasing to decreasing and the unsigned qualifier is typically appropriate for the former case only. For example, the code in Exercise 2.10 does not work if i is declared unsigned.

Clearly all the work done inside the loop is O(1) per iteration, so the analysis requires determining the number of times around the loop. The loop starts with high - low = n - 1 and finishes with high - low   [image]  -1. Every time through the loop the value high - low must be at least halved from its previous value; thus, the number of times around the loop is at most   [image] log(n - 1)  [image]  + 2. (As an example, if high - low = 128, then the maximum values of high - low after each iteration are 64, 32, 16, 8, 4, 2, 1, 0, -1.) Thus, the running time is O(log n). Equivalently, we could write a recursive formula for the running time, but this kind of brute-force approach is usually unnecessary when you understand what is really going on and why.

Binary search can be viewed as our first data structure. It supports the find operation in O(log n) time, but all other operations (in particular insert) require O(n) time. In applications where the data are static (that is, insertions and deletions are not allowed), this could be a very useful data structure. The input would then need to be sorted once, but afterward accesses would be fast. An example could be a program that needs to maintain information about the periodic table of elements (which arises in chemistry and physics). This table is relatively stable, as new elements are added infrequently. The element names could be kept sorted. Since there are only about 110 elements, at most eight accesses would be required to find an element. Performing a sequential search would require many more accesses.

int

binary_search( input_type a[ ], input_type x, unsigned int n )

{

int low, mid, high;       /* Can't be unsigned; why? */

/*1*/       low = 0; high = n - 1;

/*2*/       while( low <= high )

{

/*3*/            mid = (low + high)/2;

/*4*/            if( a[mid] < x )

/*5*/                 low = mid + 1;

else

/*6*/           if ( a[mid] < x )

/*7*/                 high = mid - 1;

else

/*8*/                return( mid );  /* found */

}

/*9*/       return( NOT_FOUND );

}

Figure 2.9 Binary search.

Euclid's Algorithm

A second example is Euclid's algorithm for computing the greatest common divisor. The greatest common divisor (gcd) of two integers is the largest integer that divides both. Thus, gcd (50, 15) = 5. The algorithm in Figure 2.10 computes gcd(m, n), assuming m   [image]  n. (If n > m, the first iteration of the loop swaps them).

The algorithm works by continually computing remainders until 0 is reached. The last nonzero remainder is the answer. Thus, if m = 1,989 and n = 1,590, then the sequence of remainders is 399, 393, 6, 3, 0. Therefore, gcd (1989, 1590) = 3. As the example shows, this is a fast algorithm.

As before, the entire running time of the algorithm depends on determining how long the sequence of remainders is. Although log n seems like a good answer, it is not at all obvious that the value of the remainder has to decrease by a constant factor, since we see that the remainder went from 399 to only 393 in the example. Indeed, the remainder does not decrease by a constant factor in one iteration. However, we can prove that after two iterations, the remainder is at most half of its original value. This would show that the number of iterations is at most 2 log n = O(log n) and establish the running time. This proof is easy, so we include it here. It follows directly from the following theorem.

unsigned int

gcd( unsigned int m, unsigned int n )

{

unsigned int rem;

/*1*/       while( n > 0 )

{

/*2*/            rem = m % n;

/*3*/            m = n;

/*4*/            n = rem;

}

/*5*/       return( m );

}

Figure 2.10 Euclid's algorithm.

THEOREM 2.1.

If m > n, then mmod n < m/2.

PROOF:

There are two cases. If n   [image]  m/2, then obviously, since the remainder is smaller than n, the theorem is true for this case. The other case is n > m/2. But then n goes into m once with a remainder m - n < m/2, proving the theorem.

One might wonder if this is the best bound possible, since 2 log n is about 20 for our example, and only seven operations were performed. It turns out that the constant can be improved slightly, to roughly 1.44 log n, in the worst case (which is achievable if m and n are consecutive Fibonacci numbers). The average-case performance of Euclid's algorithm requires pages and pages of highly sophisticated mathematical analysis, and it turns out that the average number of iterations is about   [image].

Exponentiation

Our last example in this section deals with raising an integer to a power (which is also an integer). Numbers that result from exponentiation are generally quite large, so an analysis only works if we can assume that we have a machine that can store such large integers (or a compiler that can simulate this). We will count the number of multiplications as the measurement of running time.

int

pow( int x, unsigned int n)

{

/*1*/       if( n == 0 )

/*2*/           return 1;

/*1*/       if( n == 1 )

/*4*/           return x;

/*5*/       if( even( n ) )

/*6*/           return( pow( x*x, n/2 ) );

else

/*7*/           return( pow( x*x, n/2 ) * x );

}

Figure 2.11 Efficient exponentiation

The obvious algorithm to compute xn uses n - 1 multiples. The recursive algorithm in Figure 2.11 does better. Lines 1 to 4 handle the base case of the recursion. Otherwise, if n is even, we have xn = xn/2 . xn/2, and if n is odd, x   = x(n-1)/2   [image]  x(n-1)/2   [image]  x.

For instance, to compute x62, the algorithm does the following calculations, which involves only nine multiplications:

x3 = (x2)x,  x7 = (x3)2x,  x15 = (x7)2x,  x31 = (x15)2x,  x62 = (x31)2

The number of multiplications required is clearly at most 2 log n, because at most two multiplications (if n is odd) are required to halve the problem. Again, a recurrence formula can be written and solved. Simple intuition obviates the need for a brute-force approach.

It is sometimes interesting to see how much the code can be tweaked without affecting correctness. In Figure 2.11, lines 3 to 4 are actually unnecessary, because if n is 1, then line 7 does the right thing. Line 7 can also be rewritten as

/*7*/   return( pow( x, n-1 ) * x );

without affecting the correctness of the program. Indeed, the program will still run in O(log n), because the sequence of multiplications is the same as before. However, all of the following alternatives for line 6 are bad, even though they look correct:

/*6a*/   return( pow( pow( x, 2 ), n/2 ) );

/*6b*/   return( pow( pow( x, n/2 ), 2 ) );

/*6c*/   return( pow( x, n/2 ) * pow( x, n/2 ) );

Both lines 6a and 6b are incorrect because when n is 2, one of the recursive calls to pow has 2 as the second argument. Thus, no progress is made, and an infinite loop results (in an eventual crash).

Using line 6c affects the efficiency, because there are now two recursive calls of size n/2 instead of only one. An analysis will show that the running time is no longer O(log n). We leave it as an exercise to the reader to determine the new running time.

2.4.5 Checking Your Analysis

Once an analysis has been performed, it is desirable to see if the answer is correct and as good as possible. One way to do this is to code up the program and see if the empirically observed running time matches the running time predicted by the analysis. When n doubles, the running time goes up by a factor of 2 for linear programs, 4 for quadratic programs, and 8 for cubic programs. Programs that run in logarithmic time take only an additive constant longer when n doubles, and programs that run in O(n log n) take slightly more than twice as long to run under the same circumstances. These increases can be hard to spot if the lower-order terms have relatively large coefficients and n is not large enough. An example is the jump from n = 10 to n = 100 in the running time for the various implementations of the maximum subsequence sum problem. It also can be very difficult to differentiate linear programs from O(n log n) programs purely on empirical evidence.

Another commonly used trick to verify that some program is O(f(n)) is to compute the values T(n)/ f(n) for a range of n (usually spaced out by factors of 2), where T(n) is the empirically observed running time. If f(n) is a tight answer for the running time, then the computed values converge to a positive constant. If f(n) is an over-estimate, the values converge to zero. If f(n) is an under-estimate and hence wrong, the values diverge.

As an example, the program fragment in Figure 2.12 computes the probability that two distinct positive integers, less than or equal to n and chosen randomly, are relatively prime. (As n gets large, the answer approaches 6/  [image] 2.)

You should be able to do the analysis for this program instantaneously. Figure 2.13 shows the actual observed running time for this routine on a real computer. The table shows that the last column is most likely, and thus the analysis that you should have gotten is probably correct. Notice that there is not a great deal of difference between O(n2) and O(n2 log n), since logarithms grow so slowly.

2.4.6. A Grain of Salt

Sometimes the analysis is shown empirically to be an over-estimate. If this is the case, then either the analysis needs to be tightened (usually by a clever observation), or it may be the case that the average running time is significantly less than the worst-case running time and no improvement in the bound is possible. There are many complicated algorithms for which the worst-case bound is achievable by some bad input but is usually an over-estimate in practice. Unfortunately, for most of these problems, an average-case analysis is extremely complex (in many cases  still unsolved), and a worst-case bound, even though overly pessimistic, is the best analytical result known.

rel = 0; tot = 0;

for( i=1; i<=n; i++ )

for( j=i+1; j<=n; j++ )

{

tot++;

if( gcd( i, j) = 1 )

rel++;

}

printf( "Percentage of relatively prime pairs is %lf\n",

( (double) rel )/tot );

Figure 2.12 Estimate the probability that two random numbers are relatively prime

    n     CPU time (T)     T/n2        T/n3     T/n2log n

---------------------------------------------------------

    100       022        .002200   .000022000   .0004777

    200       056        .001400   .000007000   .0002642

    300       118        .001311   .000004370   .0002299

    400       207        .001294   .000003234   .0002159

    500       318        .001272   .000002544   .0002047

---------------------------------------------------------

    600       466        .001294   .000002157   .0002024

    700       644        .001314   .000001877   .0002006

    800       846        .001322   .000001652   .0001977

    900     1,086        .001341   .000001490   .0001971

  1,000     1,362        .001362   .000001362   .0001972

---------------------------------------------------------

  1,500     3,240        .001440   .000000960   .0001969

  2,000     5,949        .001482   .000000740   .0001947

  4,000    25,720        .001608   .000000402   .0001938

Figure 2.13 Empirical running times for previous routine

Summary

This chapter gives some hints on how to analyze the complexity of programs. Unfortunately, it is not a complete guide. Simple programs usually have simple analyses, but this is not always the case. As an example, we shall see, later in the text, a sorting algorithm (Shellsort, Chapter 7) and an algorithm for maintaining disjoint sets (Chapter 8) each of which requires about 20 lines of code. The analysis of Shellsort is still not complete, and the disjoint set algorithm has an analysis that is extremely difficult and requires pages and pages of intricate calculations. Most of the analysis that we will encounter here will be simple and involve counting through loops.

An interesting kind of analysis, which we have not touched upon, is lowerbound analysis. We will see an example of this in Chapter 7, where it is proved that any algorithm that sorts by using only comparisons requires   [image] (n log n) comparisons in the worst case. Lower-bound proofs are generally the most difficult, because they apply not to an algorithm but to a class of algorithms that solve a problem.

We close by mentioning that some of the algorithms described here have real-life application. The gcd algorithm and the exponentiation algorithm are both used in cryptography. Specifically, a 200-digit number is raised to a large power (usually another 200-digit number), with only the low 200 or so digits retained after each multiplication. Since the calculations require dealing with 200-digit numbers, efficiency is obviously important. The straightforward algorithm for exponentiation would require about 10200 multiplications, whereas the algorithm presented requires only about 1,200.

Exercises

2.1     Order the following functions by growth rate: n,   [image], n1.5, n2, n log n, n log log n, n log2 n, n log(n2), 2/n, 2  , 2n/2, 37, n2 log n, n3. Indicate which functions grow at the same rate.

2.2     Suppose Tl(n) = O(f(n)) and T2(n) = O(f(n)). Which of the following are true?

a.     T1(n) + T2(n) = O(f(n))

b.     T1(n) - T2(n) = o(f(n))

d.     T1(n) = O(T2(n))

2.3     Which function grows faster: n log n or n1+  [image] /    [image]    [image]  > 0 ?

2.4     Prove that for any constant, k, logkn = o(n).

2.5     Find two functions f(n) and g(n) such that neither   [image] (n) = O(g(n)) nor g(n) = O(f(n)).

2.6     For each of the following six program fragments:

a.     Give an analysis of the running time (Big-Oh will do).

b.     Implement the code in the language of your choice, and give the running time for several values of n.

c.     Compare your analysis with the actual running times.

(1)   sum = 0;

for( i=0; i<n; i++ )

sum++;

(2)   sum = 0;

for( i=0; i<n; i++ )

for( j=0; j<n; j++ )

sum++;

(3)   sum = 0;

for( i=0; i<n; i++ )

for( j=0; j<n*n; j++ )

sum++;

(4)   sum = 0;

for( i=0; i<n; i++ )

for( j=0; j<i; j++ )

sum++;

(5)   sum = 0;

for( i=0; i<n; i++ )

for( j=0; j<i*i; j++ )

for( k=0; k<j; k++)

sum++;

(6)   sum = 0;

for( i=1; i<n; i++ )

for( j=1; j<i*i; j++ )

if( j%1 == 0 )

for( k=0; k<j; k++ )

sum++;

2.7     Suppose you need to generate a random permutation of the first n integers. For example, {4, 3, 1, 5, 2} and {3, 1, 4, 2, 5} are legal permutations, but {5, 4, 1, 2, 1} is not, because one number (1) is duplicated and another (3) is missing. This routine is often used in simulation of algorithms. We assume the existence of a random number generator, rand_int(i, j), which generates integers between i and j with equal probability. Here are three algorithms:

1.     Fill the array a from a[0] to a[n - 1] as follows: To fill a[i], generate random numbers until you get one that is not already in a[0], a[1], a[2], . . . , a[i-1].

2.     Same as algorithm (1), but keep an extra array called the used array. When a random number, ran, is first put in the array a, set used[ran] = 1. This means that when filling a[i] with a random number, you can test in one step to see whether the random number has been used, instead of the (possibly) i steps in the first algorithm.

3.     Fill the array such that a[i] = i + 1. Then

for( i=1; i<n; i++ )

swap( &a[i], &a[ rand_int( 0, i ) ] );

a.     Prove that all three algorithms generate only legal permutations and that all permutations are equally likely.

b.     Give as accurate (Big-Oh) an analysis as you can of the expected running time of each algorithm.

c.     Write (separate) programs to execute each algorithm 10 times, to get a good average. Run program (1) for n = 250, 500, 1,000, 2,000; program (2) for n = 2,500, 5,000, 10,000, 20,000, 40,000, 80,000, and program (3) for n = 10,000, 20,000, 40,000, 80,000, 160,000, 320,000, 640,000.

d.     Compare your analysis with the actual running times.

e.     What is the worst-case running time of each algorithm?

2.8     Complete the table in Figure 2.2 with estimates for the running times that were too long to simulate. Interpolate the running times for these algorithms and estimate the time required to compute the maximum subsequence sum of one million numbers. What assumptions have you made?

2.9     How much time is required to compute

a.     using a simple routine to perform exponentiation?

b.     using the routine in Section 2.4.4?

2.10     Consider the following algorithm (known as Horner's rule) to evaluate

poly = 0;

for( i=n; i>=0; i-- )

poly = x * poly + ai

a.     Show how the steps are performed by this algorithm for x = 3, f(x) = 4x   + 8x   + x + 2.

b.     Explain why this algorithm works.

c.     What is the running time of this algorithm?

2.11     Give an efficient algorithm to determine if there exists an integer i such that ai = i in an array of integers a1 < a2 < a3 < . . . < an. What is the running time of your algorithm?

2.12     Give efficient algorithms (along with running time analyses) to

a.     find the minimum subsequence sum

*b.     find the minimum positive subsequence sum

*c.     find the maximum subsequence product

2.13     a.     Write a program to determine if a positive integer, n, is prime.

b.     In terms of n, what is the worst-case running time of your program? (You should be able to do this in   [image].)

c.     Let B equal the number of bits in the binary representation of n. What is the value of B?

d.     In terms of B, what is the worst-case running time of your program?

e.     Compare the running times to determine if a 20-bit and a 40-bit number are prime.

f.     Is it more reasonable to give the running time in terms of n or B? Why?

*2.14 The Sieve of Erastothenes is a method used to compute all primes less than n. We begin by making a table of integers 2 to n. We find the smallest integer, i, that is not crossed out, print i, and cross out i, 2i, 3i, . . . . When   [image]  the algorithm terminates. What is the running time of this algorithm?

2.15     Show that x62 can be computed with only eight multiplications.

2.16     Write the fast exponentiation routine without recursion.

2.17     Give a precise count on the number of multiplication used by the fast exponentiation routine. (Hint: Consider the binary representation of n.)

2.18     Programs A and B are analyzed and found to have worst-case running times no greater than 150n log2 n and n2, respectively. Answer the following questions if possible:

a.     Which program has the better guarantee on the running time, for large values of n (n > 10,000)?

b.     Which program has the better guarantee on the running time, for small values of n (n < 100)?

c.     Which program will run faster on average for n = 1,000?

d.     Is it possible that program B will run faster than program A on all possible inputs ?

2.19     A majority element in an array, A, of size n is an element that appears more than n/2 times (thus, there is at most one). For example, the array

3, 3, 4, 2, 4, 4, 2, 4, 4

has a majority element (4), whereas the array

3, 3, 4, 2, 4, 4, 2, 4

does not. If there is no majority element, your program should indicate this. Here is a sketch of an algorithm to solve the problem:

First, a candidate majority element is found (this is the harder part). This candidate is the only element that could possibly be the majority element. The second step determines if this candidate is actually the majority. This is just a sequential search through the array. To find a candidate in the array, A, form a second array, B. Then compare A1 and A2. If they are equal, add one of these to B; otherwise do nothing. Then compare A3 and A4. Again if they are equal, add one of these to B; otherwise do nothing. Continue in this fashion until the entire array is read. Then recursively find a candidate for B; this is the candidate for A (why?.

a.     How does the recursion terminate?

*b.     How is the case where n is odd handled?

*c.     What is the running time of the algorithm?

d.     How can we avoid using an extra array B?

*e.     Write a program to compute the majority element.

*2.20     Why is it important to assume that integers in our computer model have a fixed size?

2.21     Consider the word puzzle problem described in Chapter 1. Suppose we fix the size of the longest word to be 10 characters.

a.     In terms of r and c, which are the number of rows and columns in the puzzle, and W, which is the number of words, what is the running time of the algorithms described in Chapter 1?

b.     Suppose the word list is presorted. Show how to use binary search to obtain an algorithm with significantly better running time.

2.22     Suppose that line 5 in the binary search routine had the expression low = mid instead of low = mid + 1. Would the routine still work?

2.23     Suppose that lines 6 and 7 in Algorithm 3 (Fig. 2.7) are replaced by

/*6*/    max_left_sum = max_sub_sum( a, left, center-1);

/*7*/    max_right_sum = max_sub_sum( a, center, right);

Would the routine still work?

*2.24 The inner loop of the cubic maximum subsequence sum algorithm performs n(n + 1)(n + 2)/6 iterations of the innermost code. The quadratic version performs n(n + 1)/2 iterations. The linear version performs n iterations. What pattern is evident? Can you give a combinatoric explanation of this phenomenon?

References

Analysis of the running time of algorithms was first made popular by Knuth in the three-part series [5], [6], and [7]. The analysis of the gcd algorithm appears in [6]. Another early text on the subject is [1].

Big-Oh, big-omega, big-theta, and little-oh notation were advocated by Knuth in [8]. There is still not a uniform agreement on the matter, especially when it comes to using   [image] ( ). Many people prefer to use O( ), even though it is less expressive. Additionally, O( ) is still used in some corners to express a lower bound, when   [image] ( ) is called for.

The maximum subsequence sum problem is from [3]. The series of books [2], [3], and [4] show how to optimize programs for speed.

1.     A. V. Aho, J. E. Hopcroft, and J. D. Ullman, The Design and Analysis of Computer Algorithms, Addison-Wesley, Reading, Mass., 1974.

2.     J. L. Bentley, Writing Efficient Programs, Prentice Hall, Englewood Cliffs, N.J., 1982.

3.     J. L. Bentley, Programming Pearls, Addison-Wesley, Reading, Mass., 1986.

4.     J. L. Bentley, More Programming Pearls, Addison-Wesley, Reading, Mass., 1988.

5.     D. E. Knuth, The Art of Computer Programming, Vol 1: Fundamental Algorithms, 2d ed., Addison-Wesley, Reading, Mass., 1973.

6.     D. E. Knuth, The Art of Computer Programming, Vol 2: Seminumerical Algorithms, 2d ed., Addison-Wesley, Reading, Mass., 1981.

7.     D. E. Knuth, The Art of Computer Programming, Vol 3: Sorting and Searching, Addison-Wesley, Reading, Mass., 1975.

8.     D. E. Knuth, "Big Omicron and Big Omega and Big Theta," ACM SIGACT News, 8 (1976), 18-23.

Go to Chapter 3    Return to Table of Contents


Structures, Algorithm Analysis: CHAPTER 3: LISTS, STACKS, AND QUEUES

 Return to Table of Contents

CHAPTER 3: LISTS, STACKS, AND QUEUES

This chapter discusses three of the most simple and basic data structures. Virtually every significant program will use at least one of these structures explicitly, and a stack is always implicitly used in your program, whether or not you declare one. Among the highlights of this chapter, we will

Introduce the concept of Abstract Data Types (ADTs).

Show how to efficiently perform operations on lists.

Introduce the stack ADT and its use in implementing recursion.

Introduce the queue ADT and its use in operating systems and algorithm design.

Because these data structures are so important, one might expect that they are hard to implement. In fact, they are extremely easy to code up; the main difficulty is keeping enough discipline to write good general-purpose code for routines that are generally only a few lines long.

3.1. Abstract Data Types (ADTs)

One of the basic rules concerning programming is that no routine should ever exceed a page. This is accomplished by breaking the program down into modules. Each module is a logical unit and does a specific job. Its size is kept small by calling other modules. Modularity has several advantages. First, it is much easier to debug small routines than large routines. Second, it is easier for several people to work on a modular program simultaneously. Third, a well-written modular program places certain dependencies in only one routine, making changes easier. For instance, if output needs to be written in a certain format, it is certainly important to have one routine to do this. If printing statements are scattered throughout the program, it will take considerably longer to make modifications. The idea that global variables and side effects are bad is directly attributable to the idea that modularity is good.

An abstract data type (ADT) is a set of operations. Abstract data types are mathematical abstractions; nowhere in an ADT's definition is there any mention of how the set of operations is implemented. This can be viewed as an extension of modular design.

Objects such as lists, sets, and graphs, along with their operations, can be viewed as abstract data types, just as integers, reals, and booleans are data types. Integers, reals, and booleans have operations associated with them, and so do abstract data types. For the set ADT, we might have such operations as union, intersection, size, and complement. Alternately, we might only want the two operations union and find, which would define a different ADT on the set.

The basic idea is that the implementation of these operations is written once in the program, and any other part of the program that needs to perform an operation on the ADT can do so by calling the appropriate function. If for some reason implementation details need to change, it should be easy to do so by merely changing the routines that perform the ADT operations. This change, in a perfect world, would be completely transparent to the rest of the program.

There is no rule telling us which operations must be supported for each ADT; this is a design decision. Error handling and tie breaking (where appropriate) are also generally up to the program designer. The three data structures that we will study in this chapter are primary examples of ADTs. We will see how each can be implemented in several ways, but if they are done correctly, the programs that use them will not need to know which implementation was used.

3.2. The List ADT

We will deal with a general list of the form a1, a2, a3, . . . , an. We say that the size of this list is n. We will call the special list of size 0 a null list.

For any list except the null list, we say that ai+l follows (or succeeds) ai (i < n) and that ai-1 precedes ai (i > 1). The first element of the list is a1, and the last element is an. We will not define the predecessor of a1 or the successor of an. The position of element ai in a list is i. Throughout this discussion, we will assume, to simplify matters, that the elements in the list are integers, but in general, arbitrarily complex elements are allowed.

Associated with these "definitions" is a set of operations that we would like to perform on the list ADT. Some popular operations are print_list and make_null, which do the obvious things; find, which returns the position of the first occurrence of a key; insert and delete, which generally insert and delete some key from some position in the list; and find_kth, which returns the element in some position (specified as an argument). If the list is 34, 12, 52, 16, 12, then find(52) might return 3; insert(x,3) might make the list into 34, 12, 52, x, 16, 12 (if we insert after the position given); and delete(3) might turn that list into 34, 12, x, 16, 12.

Of course, the interpretation of what is appropriate for a function is entirely up to the programmer, as is the handling of special cases (for example, what does find(1) return above?). We could also add operations such as next and previous, which would take a position as argument and return the position of the successor and predecessor, respectively.

3.2.1. Simple Array Implementation of Lists

Obviously all of these instructions can be implemented just by using an array. Even if the array is dynamically allocated, an estimate of the maximum size of the list is required. Usually this requires a high over-estimate, which wastes considerable space. This could be a serious limitation, especially if there are many lists of unknown size.

An array implementation allows print_list and find to be carried out in linear time, which is as good as can be expected, and the find_kth operation takes constant time. However, insertion and deletion are expensive. For example, inserting at position 0 (which amounts to making a new first element) requires first pushing the entire array down one spot to make room, whereas deleting the first element requires shifting all the elements in the list up one, so the worst case of these operations is O(n). On average, half the list needs to be moved for either operation, so linear time is still required. Merely building a list by n successive inserts would require quadratic time.

Because the running time for insertions and deletions is so slow and the list size must be known in advance, simple arrays are generally not used to implement lists.

3.2.2. Linked Lists

In order to avoid the linear cost of insertion and deletion, we need to ensure that the list is not stored contiguously, since otherwise entire parts of the list will need to be moved. Figure 3.1 shows the general idea of a linked list.

The linked list consists of a series of structures, which are not necessarily adjacent in memory. Each structure contains the element and a pointer to a structure containing its successor. We call this the next pointer. The last cell's next pointer points to ; this value is defined by C and cannot be confused with another pointer. ANSI C specifies that is zero.

Recall that a pointer variable is just a variable that contains the address where some other data is stored. Thus, if p is declared to be a pointer to a structure, then the value stored in p is interpreted as the location, in main memory, where a structure can be found. A field of that structure can be accessed by p [image] field_name, where field_name is the name of the field we wish to examine. Figure 3.2 shows the actual representation of the list in Figure 3.1. The list contains five structures, which happen to reside in memory locations 1000, 800, 712, 992, and 692 respectively. The next pointer in the first structure has the value 800, which provides the indication of where the second structure is. The other structures each have a pointer that serves a similar purpose. Of course, in order to access this list, we need to know where the first cell can be found. A pointer variable can be used for this purpose. It is important to remember that a pointer is just a number. For the rest of this chapter, we will draw pointers with arrows, because they are more illustrative.

Figure 3.1 A linked list

Figure 3.2 Linked list with actual pointer values

Figure 3.3 Deletion from a linked list

Figure 3.4 Insertion into a linked list

To execute print_list(L) or find(L,key), we merely pass a pointer to the first element in the list and then traverse the list by following the next pointers. This operation is clearly linear-time, although the constant is likely to be larger than if an array implementation were used. The find_kth operation is no longer quite as efficient as an array implementation; find_kth(L,i) takes O(i) time and works by traversing down the list in the obvious manner. In practice, this bound is pessimistic, because frequently the calls to find_kth are in sorted order (by i). As an example, find_kth(L,2), find_kth(L,3), find_kth(L,4), find_kth(L,6) can all be executed in one scan down the list.

The delete command can be executed in one pointer change. Figure 3.3 shows the result of deleting the third element in the original list.

The insert command requires obtaining a new cell from the system by using an malloc call (more on this later) and then executing two pointer maneuvers. The general idea is shown in Figure 3.4. The dashed line represents the old pointer.

3.2.3. Programming Details

The description above is actually enough to get everything working, but there are several places where you are likely to go wrong. First of all, there is no really obvious way to insert at the front of the list from the definitions given. Second, deleting from the front of the list is a special case, because it changes the start of the list; careless coding will lose the list. A third problem concerns deletion in general. Although the pointer moves above are simple, the deletion algorithm requires us to keep track of the cell before the one that we want to delete.

Figure 3.5 Linked list with a header

It turns out that one simple change solves all three problems. We will keep a sentinel node, which is sometimes referred to as a header or dummy node. This is a common practice, which we will see several times in the future. Our convention will be that the header is in position 0. Figure 3.5 shows a linked list with a header representing the list a1, a2, . . . , a5.

To avoid the problems associated with deletions, we need to write a routine find_previous, which will return the position of the predecessor of the cell we wish to delete. If we use a header, then if we wish to delete the first element in the list, find_previous will return the position of the header. The use of a header node is somewhat controversial. Some people argue that avoiding special cases is not sufficient justification for adding fictitious cells; they view the use of header nodes as little more than old-style hacking. Even so, we will use them here, precisely because they allow us to show the basic pointer manipulations without obscuring the code with special cases. Otherwise, whether or not a header should be used is a matter of personal preference.

As examples, we will write about half of the list ADT routines. First, we need our declarations, which are given in Figure 3.6.

The first function that we will write tests for an empty list. When we write code for any data structure that involves pointers, it is always best to draw a picture first. Figure 3.7 shows an empty list; from the figure it is easy to write the function in Figure 3.8.

The next function, which is shown in Figure 3.9, tests whether the current element, which by assumption exists, is the last of the list.

typedef struct node *node_ptr;

struct node

{

element_type element;

node_ptr next;

};

typedef node_ptr LIST;

typedef node_ptr position;

Figure 3.6 Type declarations for linked lists

Figure 3.7 Empty list with header

int

is_empty( LIST L )

{

return( L->next == NULL );

}

Figure 3.8 Function to test whether a linked list is empty

int

is_last( position p, LIST L )

{

return( p->next == NULL );

}

Figure 3.9 Function to test whether current position is the last in a linked list

The next routine we will write is find. Find, shown in Figure 3.10, returns the position in the list of some element. Line 2 takes advantage of the fact that the and (&&) operation is short-circuited: if the first half of the and is false, the result is automatically false and the second half is not executed.

/* Return position of x in L; NULL if not found */

position

find ( element_type x, LIST L )

{

position p;

/*1*/        p = L->next;

/*2*/        while( (p != NULL) && (p->element != x) )

/*3*/        p = p->next;

/*4*/        return p;

}

Figure 3.10 Find routine

Some programmers find it tempting to code the find routine recursively, possibly because it avoids the sloppy termination condition. We shall see later that this is a very bad idea and should be avoided at all costs.

Our fourth routine will delete some element x in list L. We need to decide what to do if x occurs more than once or not at all. Our routine deletes the first occurrence of x and does nothing if x is not in the list. To do this, we find p, which is the cell prior to the one containing x, via a call to find_previous. The code to implement this is shown in Figure 3.11. The find_previous routine is similar to find and is shown in Figure 3.12.

The last routine we will write is an insertion routine. We will pass an element to be inserted along with the list L and a position p. Our particular insertion routine will insert an element after the position implied by p. This decision is arbitrary and meant to show that there are no set rules for what insertion does. It is quite possible to insert the new element into position p (which means before the element currently in position p), but doing this requires knowledge of the element before position p. This could be obtained by a call to find_previous. It is thus important to comment what you are doing. This has been done in Figure 3.13.

Notice that we have passed the list to the insert and is_last routines, even though it was never used. We did this because another implementation might need this information, and so not passing the list would defeat the idea of using ADTs.*

* This is legal, but some compilers will issue a warning.

/* Delete from a list. Cell pointed */

/* to by p->next is wiped out. */

/* Assume that the position is legal. */

/* Assume use of a header node. */

void

delete( element_type x, LIST L )

{

position p, tmp_cell;

p = find_previous( x, L );

if( p->next != NULL )  /* Implicit assumption of header use */

{                      /* x is found: delete it */

tmp_cell = p->next;

p->next = tmp_cell->next; /* bypass the cell to be deleted */

free( tmp_cell );

}

}

Figure 3.11 Deletion routine for linked lists

/* Uses a header. If element is not found, then next field */

/* of returned value is NULL */

position

find_previous( element_type x, LIST L )

{

position p;

/*1*/  p = L;

/*2*/  while( (p->next != NULL) && (p->next->element != x) )

/*3*/       p = p->next;

/*4*/  return p;

}

Figure 3.12 Find_previous--the find routine for use with delete

/* Insert (after legal position p).*/

/* Header implementation assumed. */

void

insert( element_type x, LIST L, position p )

{

position tmp_cell;

/*1*/        tmp_cell = (position) malloc( sizeof (struct node) );

/*2*/        if( tmp_cell == NULL )

/*3*/              fatal_error("Out of space!!!");

else

{

/*4*/              tmp_cell->element = x;

/*5*/              tmp_cell->next = p->next;

/*6*/              p->next = tmp_cell;

}

}

Figure 3.13 Insertion routine for linked lists

With the exception of the find and find_previous routines, all of the operations we have coded take O(1) time. This is because in all cases only a fixed number of instructions are performed, no matter how large the list is. For the find and find_previous routines, the running time is O(n) in the worst case, because the entire list might need to be traversed if the element is either not found or is last in the list. On average, the running time is O(n), because on average, half the list must be traversed.

We could write additional routines to print a list and to perform the next function. These are fairly straightforward. We could also write a routine to implement previous. We leave these as exercises.

3.2.4. Common Errors

The most common error that you will get is that your program will crash with a nasty error message from the system, such as "memory access violation" or "segmentation violation." This message usually means that a pointer variable contained a bogus address. One common reason is failure to initialize the variable. For instance, if line 1 in Figure 3.14 is omitted, then p is undefined and is not likely to be pointing at a valid part of memory. Another typical error would be line 6 in Figure 3.13. If p is , then the indirection is illegal. This function knows that p is not , so the routine is OK. Of course, you should comment this so that the routine that calls insert will insure this. Whenever you do an indirection, you must make sure that the pointer is not NULL. Some C compliers will implicity do this check for you, but this is not part of the C standard. When you port a program from one compiler to another, you may find that it no longer works. This is one of the common reasons why.

The second common mistake concerns when and when not to use malloc to get a new cell. You must remember that declaring a pointer to a structure does not create the structure but only gives enough space to hold the address where some structure might be. The only way to create a record that is not already declared is to use the malloc command. The command malloc(size_p) has the system create, magically, a new structure and return a pointer to it. If, on the other hand, you want to use a pointer variable to run down a list, there is no need to declare a new structure; in that case the malloc command is inappropriate. A type cast is used to make both sides of the assignment operator compatible. The C library provides other variations of malloc such as calloc.

void

delete_list( LIST L )

{

position p;

/*1*/        p = L->next;        /* header assumed */

/*2*/        L->next = NULL;

/*3*/        while( p != NULL )

{

/*4*/             free( p );

/*5*/             p = p->next;

}

}

Figure 3.14 Incorrect way to delete a list

When things are no longer needed, you can issue a free command to inform the system that it may reclaim the space. A consequence of the free(p) command is that the address that p is pointing to is unchanged, but the data that resides at that address is now undefined.

If you never delete from a linked list, the number of calls to malloc should equal the size of the list, plus 1 if a header is used. Any less, and you cannot possibly have a working program. Any more, and you are wasting space and probably time. Occasionally, if your program uses a lot of space, the system may be unable to satisfy your request for a new cell. In this case a  pointer is returned.

After a deletion in a linked list, it is usually a good idea to free the cell, especially if there are lots of insertions and deletions intermingled and memory might become a problem. You need to keep a temporary variable set to the cell to be disposed of, because after the pointer moves are finished, you will not have a reference to it. As an example, the code in Figure 3.14 is not the correct way to delete an entire list (although it may work on some systems).

Figure 3.15 shows the correct way to do this. Disposal is not necessarily a fast thing, so you might want to check to see if the disposal routine is causing any slow performance and comment it out if this is the case. This author has written a program (see the exercises) that was made 25 times faster by commenting out the disposal (of 10,000 nodes). It turned out that the cells were freed in a rather peculiar order and apparently caused an otherwise linear program to spend O(n log n) time to dispose of n cells.

One last warning: malloc(sizeof node_ptr) is legal, but it doesn't allocate enough space for a structure. It allocates space only for a pointer.

void

delete_list( LIST L )

{

position p, tmp;

/*1*/        p = L->next;   /* header assumed */

/*2*/        L->next = NULL;

/*3*/        while( p != NULL )

{

/*4*/             tmp = p->next;

/*5*/             free( p );

/*6*/             p = tmp;

}

}

Figure 3.15 Correct way to delete a list

Figure 3.16 A doubly linked list

3.2.5. Doubly Linked Lists

Sometimes it is convenient to traverse lists backwards. The standard implementation does not help here, but the solution is simple. Merely add an extra field to the data structure, containing a pointer to the previous cell. The cost of this is an extra link, which adds to the space requirement and also doubles the cost of insertions and deletions because there are more pointers to fix. On the other hand, it simplifies deletion, because you no longer have to refer to a key by using a pointer to the previous cell; this information is now at hand. Figure 3.16 shows a doubly linked list.

3.2.6. Circularly Linked Lists

A popular convention is to have the last cell keep a pointer back to the first. This can be done with or without a header (if the header is present, the last cell points to it), and can also be done with doubly linked lists (the first cell's previous pointer points to the last cell). This clearly affects some of the tests, but the structure is popular in some applications. Figure 3.17 shows a double circularly linked list with no header.

3.2.7. Examples

We provide three examples that use linked lists. The first is a simple way to represent single-variable polynomials. The second is a method to sort in linear time, for some special cases. Finally, we show a complicated example of how linked lists might be used to keep track of course registration at a university.

The Polynomial ADT

We can define an abstract data type for single-variable polynomials (with nonnegative exponents) by using a list. Let   [image]. If most of the coefficients ai are nonzero, we can use a simple array to store the coefficients. We could then write routines to perform addition, subtraction, multiplication, differentiation, and other operations on these polynomials. In this case, we might use the type declarations given in Figure 3.18. We could then write routines to perform various operations. Two possibilities are addition and multiplication. These are shown in Figures 3.19 to 3.21. Ignoring the time to initialize the output polynomials to zero, the running time of the multiplication routine is proportional to the product of the degree of the two input polynomials. This is adequate for dense polynomials, where most of the terms are present, but if p1(x) = 10x1000 + 5x14 + 1 and p2(x) = 3x1990 - 2x1492 + 11x + 5, then the running time is likely to be unacceptable. One can see that most of the time is spent multiplying zeros and stepping through what amounts to nonexistent parts of the input polynomials. This is always undesirable.

Figure 3.17 A double circularly linked list

typedef struct

{

int coeff_array[ MAX_DEGREE+1 ];

unsigned int high_power;

} *POLYNOMIAL;

Figure 3.18 Type declarations for array implementation of the polynomial ADT

An alternative is to use a singly linked list. Each term in the polynomial is contained in one cell, and the cells are sorted in decreasing order of exponents. For instance, the linked lists in Figure 3.22 represent p1(x) and p2(x). We could then use the declarations in Figure 3.23.

void

zero_polynomial( POLYNOMIAL poly )

{

unsigned int i;

for( i=0; i<=MAX_DEGREE; i++ )

poly->coeff_array[i] = 0;

poly->high_power = 0;

}

Figure 3.19 Procedure to initialize a polynomial to zero

void

add_polynomial( POLYNOMIAL poly1, POLYNOMIAL poly2,

POLYNOMIAL poly_sum )

{

int i;

zero_polynomial( poly_sum );

poly_sum->high_power = max( poly1->high_power,

poly2->high_power);

for( i=poly_sum->high_power; i>=0; i-- )

poly_sum->coeff_array[i] = poly1->coeff_array[i]

+ poly2->coeff_array[i];

}

Figure 3.20 Procedure to add two polynomials

void

mult_polynomial( POLYNOMIAL poly1, POLYNOMIAL poly2,

POLYNOMIAL poly_prod )

{

unsigned int i, j;

zero_polynomial( poly_prod );

poly_prod->high_power = poly1->high_power

+ poly2->high_power;

if( poly_prod->high_power > MAX_DEGREE )

error("Exceeded array size");

else

for( i=0; i<=poly->high_power; i++ )

for( j=0; j<=poly2->high_power; j++ )

poly_prod->coeff_array[i+j] +=

poly1->coeff_array[i] * poly2->coeff_array[j];

}

Figure 3.21 Procedure to multiply two polynomials

Figure 3.22 Linked list representations of two polynomials

typedef struct node *node_ptr;

struct node

{

int coefficient;

int exponent;

node_ptr next;

} ;

typedef node_ptr POLYNOMIAL; /* keep nodes sorted by exponent */

Figure 3.23 Type declaration for linked list implementation of the Polynomial ADT

The operations would then be straightforward to implement. The only potential difficulty is that when two polynomials are multiplied, the resultant polynomial will have to have like terms combined. There are several ways to do this, but we will leave this as an exercise.

Radix Sort

A second example where linked lists are used is called radix sort. Radix sort is sometimes known as card sort, because it was used, until the advent of modern computers, to sort old-style punch cards.

If we have n integers in the range 1 to m (or 0 to m - 1) 9, we can use this information to obtain a fast sort known as bucket sort. We keep an array called count, of size m, which is initialized to zero. Thus, count has m cells (or buckets), which are initially empty. When ai is read, increment (by one) count[ai]. After all the input is read, scan the count array, printing out a representation of the sorted list. This algorithm takes O(m + n); the proof is left as an exercise. If m =   [image] (n), then bucket sort is O(n).

Radix sort is a generalization of this. The easiest way to see what happens is by example. Suppose we have 10 numbers, in the range 0 to 999, that we would like to sort. In general, this is n numbers in the range 0 to np - 1 for some constant p. Obviously, we cannot use bucket sort; there would be too many buckets. The trick is to use several passes of bucket sort. The natural algorithm would be to bucket-sort by the most significant "digit" (digit is taken to base n), then next most significant, and so on. That algorithm does not work, but if we perform bucket sorts by least significant "digit" first, then the algorithm works. Of course, more than one number could fall into the same bucket, and, unlike the original bucket sort, these numbers could be different, so we keep them in a list. Notice that all the numbers could have some digit in common, so if a simple array were used for the lists, then each array would have to be of size n, for a total space requirement of   [image] (n2).

The following example shows the action of radix sort on 10 numbers. The input is 64, 8, 216, 512, 27, 729, 0, 1, 343, 125 (the first ten cubes arranged randomly). The first step bucket sorts by the least significant digit. In this case the math is in base 10 (to make things simple), but do not assume this in general. The buckets are as shown in Figure 3.24, so the list, sorted by least significant digit, is 0, 1, 512, 343, 64, 125, 216, 27, 8, 729. These are now sorted by the next least significant digit (the tens digit here) (see Fig. 3.25). Pass 2 gives output 0, 1, 8, 512, 216, 125, 27, 729, 343, 64. This list is now sorted with respect to the two least significant digits. The final pass, shown in Figure 3.26, bucket-sorts by most significant digit. The final list is 0, 1, 8, 27, 64, 125, 216, 343, 512, 729.

To see that the algorithm works, notice that the only possible failure would occur if two numbers came out of the same bucket in the wrong order. But the previous passes ensure that when several numbers enter a bucket, they enter in sorted order. The running time is O(p(n + b)) where p is the number of passes, n is the number of elements to sort, and b is the number of buckets. In our case, b = n.

  0  1  512  343  64  125  216  27  8  729

-------------------------------------------

  0  1    2    3   4    5    6   7  8    9

Figure 3.24 Buckets after first step of radix sort

  8       729

  1  216   27

  0  512  125     343     64

--------------------------------------

  0    1    2  3    4  5   6  7  8  9

Figure 3.25 Buckets after the second pass of radix sort

  64

  27

   8

   1

   0  125  216  343     512     729

------------------------------------------

   0    1    2    3  4    5  6    7  8  9

Figure 3.26 Buckets after the last pass of radix sort

As an example, we could sort all integers that are representable on a computer (32 bits) by radix sort, if we did three passes over a bucket size of 211. This algorithm would always be O(n) on this computer, but probably still not as efficient as some of the algorithms we shall see in Chapter 7, because of the high constant involved (remember that a factor of log n is not all that high, and this algorithm would have the overhead of maintaining linked lists).

Multilists

Our last example shows a more complicated use of linked lists. A university with 40,000 students and 2,500 courses needs to be able to generate two types of reports. The first report lists the class registration for each class, and the second report lists, by student, the classes that each student is registered for.

The obvious implementation might be to use a two-dimensional array. Such an array would have 100 million entries. The average student registers for about three courses, so only 120,000 of these entries, or roughly 0.1 percent, would actually have meaningful data.

What is needed is a list for each class, which contains the students in the class. We also need a list for each student, which contains the classes the student is registered for. Figure 3.27 shows our implementation.

As the figure shows, we have combined two lists into one. All lists use a header and are circular. To list all of the students in class C3, we start at C3 and traverse its list (by going right). The first cell belongs to student S1. Although there is no explicit information to this effect, this can be determined by following the student's linked list until the header is reached. Once this is done, we return to C3's list (we stored the position we were at in the course list before we traversed the student's list) and find another cell, which can be determined to belong to S3. We can continue and find that S4 and S5 are also in this class. In a similar manner, we can determine, for any student, all of the classes in which the student is registered.

Figure 3.27 Multilist implementation for registration problem

Using a circular list saves space but does so at the expense of time. In the worst case, if the first student was registered for every course, then every entry would need to be examined in order to determine all the course names for that student. Because in this application there are relatively few courses per student and few students per course, this is not likely to happen. If it were suspected that this could cause a problem, then each of the (nonheader) cells could have pointers directly back to the student and class header. This would double the space requirement, but simplify and speed up the implementation.

3.2.8. Cursor Implementation of Linked Lists

Many languages, such as BASIC and FORTRAN, do not support pointers. If linked lists are required and pointers are not available, then an alternate implementation must be used. The alternate method we will describe is known as a cursor implementation.

The two important items present in a pointer implementation of linked lists are

1.     The data is stored in a collection of structures. Each structure contains the data and a pointer to the next structure.

2.     A new structure can be obtained from the system's global memory by a call to malloc and released by a call to free.

Our cursor implementation must be able to simulate this. The logical way to satisfy condition 1 is to have a global array of structures. For any cell in the array, its array index can be used in place of an address. Figure 3.28 gives the type declarations for a cursor implementation of linked lists.

We must now simulate condition 2 by allowing the equivalent of malloc and free for cells in the CURSOR_SPACE array. To do this, we will keep a list (the freelist) of cells that are not in any list. The list will use cell 0 as a header. The initial configuration is shown in Figure 3.29.

A value of 0 for next is the equivalent of a  pointer. The initialization of CURSOR_SPACE is a straightforward loop, which we leave as an exercise. To perform an malloc, the first element (after the header) is removed from the freelist.

typedef unsigned int node_ptr;

struct node

{

element_type element;

node_ptr next;

};

typedef node_ptr LIST;

typedef node_ptr position;

struct node CURSOR_SPACE[ SPACE_SIZE ];

Figure 3.28 Declarations for cursor implementation of linked lists

  Slot  Element  Next

----------------------

    0              1

    1              2

    2              3

    3              4

    4              5

    5              6

    6              7

    7              8

    8              9

    9             10

   10              0

Figure 3.29 An initialized CURSOR_SPACE

To perform a free, we place the cell at the front of the freelist. Figure 3.30 shows the cursor implementation of malloc and free. Notice that if there is no space available, our routine does the correct thing by setting p = 0. This indicates that there are no more cells left, and also makes the second line of cursor_new a nonoperation (no-op).

Given this, the cursor implementation of linked lists is straightforward. For consistency, we will implement our lists with a header node. As an example, in Figure 3.31, if the value of L is 5 and the value of M is 3, then L represents the list a, b, e, and M represents the list c, d, f.

position

cursor_alloc( void )

{

position p;

p = CURSOR_SPACE[O].next;

CURSOR_SPACE[0].next = CURSOR_SPACE[p].next;

return p;

}

void

cursor_free( position p)

{

CURSOR_SPACE[p].next = CURSOR_SPACE[O].next;

CURSOR_SPACE[O].next = p;

}

Figure 3.30 Routines: cursor-alloc and cursor-free

  Slot  Element  Next

----------------------

    0      -       6

    1      b       9

    2      f       0

    3    header    7

    4      -       0

    5    header   10

    6      -       4

    7      c       8

    8      d       2

    9      e       0

   10      a       1

Figure 3.31 Example of a cursor implementation of linked lists

To write the functions for a cursor implementation of linked lists, we must pass and return the identical parameters as the pointer implementation. The routines are straightforward. Figure 3.32 implements a function to test whether a list is empty. Figure 3.33 implements the test of whether the current position is the last in a linked list.

The function find in Figure 3.34 returns the position of x in list L.

The code to implement deletion is shown in Figure 3.35. Again, the interface for the cursor implementation is identical to the pointer implementation. Finally, Figure 3.36 shows a cursor implementation of insert.

The rest of the routines are similarly coded. The crucial point is that these routines follow the ADT specification. They take specific arguments and perform specific operations. The implementation is transparent to the user. The cursor implementation could be used instead of the linked list implementation, with virtually no change required in the rest of the code.

int

is_empty( LIST L )  /* using a header node */

{

return( CURSOR_SPACE[L].next == 0

}

Figure 3.32 Function to test whether a linked list is empty--cursor implementation

int

is_last( position p, LIST L)  /* using a header node */

{

return( CURSOR_SPACE[p].next == 0

}

Figure 3.33 Function to test whether p is last in a linked list--cursor implementation

position

find( element_type x, LIST L) /* using a header node */

{

position p;

/*1*/       p = CURSOR_SPACE[L].next;

/*2*/       while( p && CURSOR_SPACE[p].element != x )

/*3*/             p = CURSOR_SPACE[p].next;

/*4*/       return p;

}

Figure 3.34 Find routine--cursor implementation

void

delete( element_type x, LIST L )

{

position p, tmp_cell;

p = find_previous( x, L );

if( !is_last( p, L) )

{

tmp_cell = CURSOR_SPACE[p].next;

CURSOR_SPACE[p].next = CURSOR_SPACE[tmp_cell].next;

cursor_free( tmp_cell );

}

}

Figure 3.35 Deletion routine for linked lists--cursor implementation

/* Insert (after legal position p); */

/* header implementation assumed */

void

insert( element_type x, LIST L, position p )

{

position tmp_cell;

/*1*/       tmp_cell = cursor_alloc( )

/*2*/       if( tmp_cell ==0 )

/*3*/       fatal_error("Out of space!!!");

else

{

/*4*/            CURSOR_SPACE[tmp_cell].element = x;

/*5*/            CURSOR_SPACE[tmp_cell].next = CURSOR_SPACE[p].next;

/*6*/            CURSOR_SPACE[p].next = tmp_cell;

}

}

Figure 3.36 Insertion routine for linked lists--cursor implementation

The freelist represents an interesting data structure in its own right. The cell that is removed from the freelist is the one that was most recently placed there by virtue of free. Thus, the last cell placed on the freelist is the first cell taken off. The data structure that also has this property is known as a stack, and is the topic of the next section.

3.3. The Stack ADT

3.3.1. Stack Model

A stack is a list with the restriction that inserts and deletes can be performed in only one position, namely the end of the list called the top. The fundamental operations on a stack are push, which is equivalent to an insert, and pop, which deletes the most recently inserted element. The most recently inserted element can be examined prior to performing a pop by use of the top routine. A pop or top on an empty stack is generally considered an error in the stack ADT. On the other hand, running out of space when performing a push is an implementation error but not an ADT error.

Stacks are sometimes known as LIFO (last in, first out) lists. The model depicted in Figure 3.37 signifies only that pushes are input operations and pops and tops are output. The usual operations to make empty stacks and test for emptiness are part of the repertoire, but essentially all that you can do to a stack is push and pop.

Figure 3.38 shows an abstract stack after several operations. The general model is that there is some element that is at the top of the stack, and it is the only element that is visible.

Figure 3.37 Stack model: input to a stack is by push, output is by pop

Figure 3.38 Stack model: only the top element is accessible

3.3.2. Implementation of Stacks

Of course, since a stack is a list, any list implementation will do. We will give two popular implementations. One uses pointers and the other uses an array, but, as we saw in the previous section, if we use good programming principles the calling routines do not need to know which method is being used.

Linked List Implementation of Stacks

The first implementation of a stack uses a singly linked list. We perform a push by inserting at the front of the list. We perform a pop by deleting the element at the front of the list. A top operation merely examines the element at the front of the list, returning its value. Sometimes the pop and top operations are combined into one. We could use calls to the linked list routines of the previous section, but we will rewrite the stack routines from scratch for the sake of clarity.

First, we give the definitions in Figure 3.39. We implement the stack using a header. Then Figure 3.40 shows that an empty stack is tested for in the same manner as an empty list.

Creating an empty stack is also simple. We merely create a header node; make_null sets the next pointer to NULL (see Fig. 3.41). The push is implemented as an insertion into the front of a linked list, where the front of the list serves as the top of the stack (see Fig. 3.42). The top is performed by examining the element in the first position of the list (see Fig. 3.43). Finally, we implement pop as a delete from the front of the list (see Fig. 3.44).

It should be clear that all the operations take constant time, because nowhere in any of the routines is there even a reference to the size of the stack (except for emptiness), much less a loop that depends on this size. The drawback of this implementation is that the calls to malloc and free are expensive, especially in comparison to the pointer manipulation routines. Some of this can be avoided by using a second stack, which is initially empty. When a cell is to be disposed from the first stack, it is merely placed on the second stack. Then, when new cells are needed for the first stack, the second stack is checked first.

Array Implementation of Stacks

An alternative implementation avoids pointers and is probably the more popular solution. The only potential hazard with this strategy is that we need to declare an array size ahead of time. Generally this is not a problem, because in typical applications, even if there are quite a few stack operations, the actual number of elements in the stack at any time never gets too large. It is usually easy to declare the array to be large enough without wasting too much space. If this is not possible, then a safe course would be to use a linked list implementation.

If we use an array implementation, the implementation is trivial. Associated with each stack is the top of stack, tos, which is -1 for an empty stack (this is how an empty stack is initialized). To push some element x onto the stack, we increment tos and then set STACK[tos] = x, where STACK is the array representing the actual stack. To pop, we set the return value to STACK[tos] and then decrement tos. Of course, since there are potentially several stacks, the STACK array and tos are part of one structure representing a stack. It is almost always a bad idea to use global variables and fixed names to represent this (or any) data structure, because in most real-life situations there will be more than one stack. When writing your actual code, you should attempt to follow the model as closely as possible, so that no part of your code, except for the stack routines, can attempt to access the array or top-of-stack variable implied by each stack. This is true for all ADT operations. Modern languages such as Ada and C++ can actually enforce this rule.

typedef struct node *node_ptr;

struct node

{

element_type element;

node_ptr next;

};

typedef node_ptr STACK;

/* Stack implementation will use a header. */

Figure 3.39 Type declaration for linked list implementation of the stack ADT

int

is_empty( STACK S )

{

return( S->next == NULL );

}

Figure 3.40 Routine to test whether a stack is empty-linked list implementation

STACK

create_stack( void )

{

STACK S;

S = (STACK) malloc( sizeof( struct node ) );

if( S == NULL )

fatal_error("Out of space!!!");

return S;

}

void

make_null( STACK S )

{

if( S != NULL )

S->next = NULL;

else

error("Must use create_stack first");

}

Figure 3.41 Routine to create an empty stack-linked list implementation

void

push( element_type x, STACK S )

{

node_ptr tmp_cell;

tmp_cell = (node_ptr) malloc( sizeof ( struct node ) );

if( tmp_cell == NULL )

fatal_error("Out of space!!!");

else

{

tmp_cell->element = x;

tmp_cell->next = S->next;

S->next = tmp_cell;

}

}

Figure 3.42 Routine to push onto a stack-linked list implementation

element_type

top( STACK S )

{

if( is_empty( S ) )

error("Empty stack");

else

return S->next->element;

}

Figure 3.43 Routine to return top element in a stack--linked list implementation

void

pop( STACK S )

{

node_ptr first_cell;

if( is_empty( S ) )

error("Empty stack");

else

{

first_cell = S->next;

S->next = S->next->next;

free( first_cell );

}

}

Figure 3.44 Routine to pop from a stack--linked list implementation

Notice that these operations are performed in not only constant time, but very fast constant time. On some machines, pushes and pops (of integers) can be written in one machine instruction, operating on a register with auto-increment and auto-decrement addressing. The fact that most modern machines have stack operations as part of the instruction set enforces the idea that the stack is probably the most fundamental data structure in computer science, after the array.

One problem that affects the efficiency of implementing stacks is error testing. Our linked list implementation carefully checked for errors. As described above, a pop on an empty stack or a push on a full stack will overflow the array bounds and cause a crash. This is obviously undesirable, but if checks for these conditions were put in the array implementation, they would likely take as much time as the actual stack manipulation. For this reason, it has become a common practice to skimp on error checking in the stack routines, except where error handling is crucial (as in operating systems). Although you can probably get away with this in most cases by declaring the stack to be large enough not to overflow and ensuring that routines that use pop never attempt to pop an empty stack, this can lead to code that barely works at best, especially when programs get large and are written by more than one person or at more than one time. Because stack operations take such fast constant time, it is rare that a significant part of the running time of a program is spent in these routines. This means that it is generally not justifiable to omit error checks. You should always write the error checks; if they are redundant, you can always comment them out if they really cost too much time. Having said all this, we can now write routines to implement a general stack using arrays.

A STACK is defined in Figure 3.45 as a pointer to a structure. The structure contains the top_of_stack and stack_size fields. Once the maximum size is known, the stack array can be dynamically allocated. Figure 3.46 creates a stack of a given maximum size. Lines 3-5 allocate the stack structure, and lines 6-8 allocate the stack array. Lines 9 and 10 initialize the top_of_stack and stack_size fields. The stack array does not need to be initialized. The stack is returned at line 11.

The routine dispose_stack should be written to free the stack structure. This routine first frees the stack array and then the stack structure (See Figure 3.47). Since create_stack requires an argument in the array implementation, but not in the linked list implementation, the routine that uses a stack will need to know which implementation is being used unless a dummy parameter is added for the later implementation. Unfortunately, efficiency and software idealism often create conflicts.

struct stack_record

{

unsigned int stack_size;

int top_of_stack;

element_type *stack_array;

};

typedef struct stack_record *STACK;

#define EMPTY_TOS (-1) /* Signifies an empty stack */

Figure 3.45 STACK definition--array implementaion

STACK

create_stack( unsigned int max_elements )

{

STACK S;

/*1*/       if( max_elements < MIN_STACK_SIZE )

/*2*/            error("Stack size is too small");

/*3*/       S = (STACK) malloc( sizeof( struct stack_record ) );

/*4*/       if( S == NULL )

/*5*/            fatal_error("Out of space!!!");

/*6*/       S->stack_array = (element_type *)

malloc( sizeof( element_type ) * max_elements );

/*7*/       if( S->stack_array == NULL )

/*8*/            fatal_error("Out of space!!!");

/*9*/       S->top_of_stack = EMPTY_TOS;

/*10*/      S->stack_size = max_elements;

/*11*/      return( S );

}

Figure 3.46 Stack creation--array implementaion

void

dispose_stack( STACK S )

{

if( S != NULL )

{

free( S->stack_array );

free( S );

}

}

Figure 3.47 Routine for freeing stack--array implementation

We have assumed that all stacks deal with the same type of element. In many languages, if there are different types of stacks, then we need to rewrite a new version of the stack routines for each different type, giving each version a different name. A cleaner alternative is provided in C++, which allows one to write a set of generic stack routines and essentially pass the type as an argument.* C++ also allows stacks of several different types to retain the same procedure and function names (such as push and pop): The compiler decides which routines are implied by checking the type of the calling routine.

*This is somewhat of an oversimplification.

Having said all this, we will now rewrite the four stack routines. In true ADT spirit, we will make the function and procedure heading look identical to the linked list implementation. The routines themselves are very simple and follow the written description exactly (see Figs. 3.48 to 3.52).

Pop is occasionally written as a function that returns the popped element (and alters the stack). Although current thinking suggests that functions should not change their input variables, Figure 3.53 illustrates that this is the most convenient method in C.

int

is_empty( STACK S )

{

return( S->top_of_stack == EMPTY_TOS );

}

Figure 3.48 Routine to test whether a stack is empty--array implementation

void

make_null( STACK S )

{

S->top_of_stack = EMPTY_TOS;

}

Figure 3.49 Routine to create an empty stack--array implementation

void

push( element_type x, STACK S )

{

if( is_full( S ) )

error("Full stack");

else

S->stack_array[ ++S->top_of_stack ] = x;

}

Figure 3.50 Routine to push onto a stack--array implementation

element_type

top( STACK S )

{

if( is_empty( S ) )

error("Empty stack");

else

return S->stack_array[ S->top_of_stack ];

}

Figure 3.51 Routine to return top of stack--array implementation

void

pop( STACK S )

{

if( is_empty( S ) )

error("Empty stack");

else

S->top_of_stack--;

}

Figure 3.52 Routine to pop from a stack--array implementation

element_type

pop( STACK S )

{

if( is_empty( S ) )

error("Empty stack");

else

return S->stack_array[ S->top_of_stack-- ];

}

Figure 3.53 Routine to give top element and pop a stack--array implementation

3.3.3. Applications

It should come as no surprise that if we restrict the operations allowed on a list, those operations can be performed very quickly. The big surprise, however, is that the small number of operations left are so powerful and important. We give three of the many applications of stacks. The third application gives a deep insight into how programs are organized.

Balancing Symbols

Compilers check your programs for syntax errors, but frequently a lack of one symbol (such as a missing brace or comment starter) will cause the compiler to spill out a hundred lines of diagnostics without identifying the real error.

A useful tool in this situation is a program that checks whether everything is balanced. Thus, every right brace, bracket, and parenthesis must correspond to their left counterparts. The sequence [()] is legal, but [(]) is wrong. Obviously, it is not worthwhile writing a huge program for this, but it turns out that it is easy to check these things. For simplicity, we will just check for balancing of parentheses, brackets, and braces and ignore any other character that appears.

The simple algorithm uses a stack and is as follows:

Make an empty stack. Read characters until end of file. If the character is an open anything, push it onto the stack. If it is a close anything, then if the stack is empty report an error. Otherwise, pop the stack. If the symbol popped is not the corresponding opening symbol, then report an error. At end of file, if the stack is not empty report an error.

You should be able to convince yourself that this algorithm works. It is clearly linear and actually makes only one pass through the input. It is thus on-line and quite fast. Extra work can be done to attempt to decide what to do when an error is reported--such as identifying the likely cause.

Postfix Expressions

Suppose we have a pocket calculator and would like to compute the cost of a shopping trip. To do so, we add a list of numbers and multiply the result by 1.06; this computes the purchase price of some items with local sales tax added. If the items are 4.99, 5.99, and 6.99, then a natural way to enter this would be the sequence

4.99 + 5.99 + 6.99 * 1.06 =

Depending on the calculator, this produces either the intended answer, 19.05, or the scientific answer, 18.39. Most simple four-function calculators will give the first answer, but better calculators know that multiplication has higher precedence than addition.

On the other hand, some items are taxable and some are not, so if only the first and last items were actually taxable, then the sequence

4.99 * 1.06 + 5.99 + 6.99 * 1.06 =

would give the correct answer (18.69) on a scientific calculator and the wrong answer (19.37) on a simple calculator. A scientific calculator generally comes with parentheses, so we can always get the right answer by parenthesizing, but with a simple calculator we need to remember intermediate results.

A typical evaluation sequence for this example might be to multiply 4.99 and 1.06, saving this answer as a1. We then add 5.99 and a1, saving the result in a1. We multiply 6.99 and 1.06, saving the answer in a2, and finish by adding al and a2, leaving the final answer in al. We can write this sequence of operations as follows:

4.99 1.06 * 5.99 + 6.99 1.06 * +

This notation is known as postfix or reverse Polish notation and is evaluated exactly as we have described above. The easiest way to do this is to use a stack. When a number is seen, it is pushed onto the stack; when an operator is seen, the operator is applied to the two numbers (symbols) that are popped from the stack and the result is pushed onto the stack. For instance, the postfix expression

6 5 2 3 + 8 * + 3 + *

is evaluated as follows: The first four symbols are placed on the stack. The resulting stack is

Next a '+' is read, so 3 and 2 are popped from the stack and their sum, 5, is pushed.

Next 8 is pushed.

Now a '*' is seen, so 8 and 5 are popped as 8 * 5 = 40 is pushed.

Next a '+' is seen, so 40 and 5 are popped and 40 + 5 = 45 is pushed.

Now, 3 is pushed.

Next '+' pops 3 and 45 and pushes 45 + 3 = 48.

Finally, a '*' is seen and 48 and 6 are popped, the result 6 * 48 = 288 is pushed.

The time to evaluate a postfix expression is O(n), because processing each element in the input consists of stack operations and thus takes constant time. The algorithm to do so is very simple. Notice that when an expression is given in postfix notation, there is no need to know any precedence rules; this is an obvious advantage.

Infix to Postfix Conversion

Not only can a stack be used to evaluate a postfix expression, but we can also use a stack to convert an expression in standard form (otherwise known as infix) into postfix. We will concentrate on a small version of the general problem by allowing only the operators +, *, and (, ), and insisting on the usual precedence rules. We will further assume that the expression is legal. Suppose we want to convert the infix expression

a + b * c + ( d * e + f ) * g

into postfix. A correct answer is a b c * + d e * f + g * +.

When an operand is read, it is immediately placed onto the output. Operators are not immediately output, so they must be saved somewhere. The correct thing to do is to place operators that have been seen, but not placed on the output, onto the stack. We will also stack left parentheses when they are encountered. We start with an initially empty stack.

If we see a right parenthesis, then we pop the stack, writing symbols until we encounter a (corresponding) left parenthesis, which is popped but not output.

If we see any other symbol ('+','*', '(' ), then we pop entries from the stack until we find an entry of lower priority. One exception is that we never remove a '(' from the stack except when processing a ')'. For the purposes of this operation, '+' has lowest priority and '(' highest. When the popping is done, we push the operand onto the stack.

Finally, if we read the end of input, we pop the stack until it is empty, writing symbols onto the output.

To see how this algorithm performs, we will convert the infix expression above into its postfix form. First, the symbol a is read, so it is passed through to the output. Then '+' is read and pushed onto the stack. Next b is read and passed through to the output. The state of affairs at this juncture is as follows:

Next a '*' is read. The top entry on the operator stack has lower precedence than '*', so nothing is output and '*' is put on the stack. Next, c is read and output. Thus far, we have

The next symbol is a '+'. Checking the stack, we find that we will pop a '*' and place it on the output, pop the other '+', which is not of lower but equal priority, on the stack, and then push the '+'.

The next symbol read is an '(', which, being of highest precedence, is placed on the stack. Then d is read and output.

We continue by reading a '*'. Since open parentheses do not get removed except when a closed parenthesis is being processed, there is no output. Next, e is read and output.

The next symbol read is a '+'. We pop and output '*' and then push '+'. Then we read and output   [image].

Now we read a ')', so the stack is emptied back to the '('. We output a '+'.

We read a '*' next; it is pushed onto the stack. Then g is read and output.

The input is now empty, so we pop and output symbols from the stack until it is empty.

As before, this conversion requires only O(n) time and works in one pass through the input. We can add subtraction and division to this repertoire by assigning subtraction and addition equal priority and multiplication and division equal priority. A subtle point is that the expression a - b - c will be converted to ab - c- and not abc - -. Our algorithm does the right thing, because these operators associate from left to right. This is not necessarily the case in general, since exponentiation associates right to left: 223 = 28 = 256 not 43 = 64. We leave as an exercise the problem of adding exponentiation to the repertoire of assignments.

Function Calls

The algorithm to check balanced symbols suggests a way to implement function calls. The problem here is that when a call is made to a new function, all the variables local to the calling routine need to be saved by the system, since otherwise the new function will overwrite the calling routine's variables. Furthermore, the current location in the routine must be saved so that the new function knows where to go after it is done. The variables have generally been assigned by the compiler to machine registers, and there are certain to be conflicts (usually all procedures get some variables assigned to register #1), especially if recursion is involved. The reason that this problem is similar to balancing symbols is that a function call and function return are essentially the same as an open parenthesis and closed parenthesis, so the same ideas should work.

When there is a function call, all the important information that needs to be saved, such as register values (corresponding to variable names) and the return address (which can be obtained from the program counter, which is typically in a register), is saved "on a piece of paper" in an abstract way and put at the top of a pile. Then the control is transferred to the new function, which is free to replace the registers with its values. If it makes other function calls, it follows the same procedure. When the function wants to return, it looks at the "paper" at the top of the pile and restores all the registers. It then makes the return jump.

Clearly, all of this work can be done using a stack, and that is exactly what happens in virtually every programming language that implements recursion. The information saved is called either an activation record or stack frame. The stack in a real computer frequently grows from the high end of your memory partition downwards, and on many systems there is no checking for overflow. There is always the possibility that you will run out of stack space by having too many simultaneously active functions. Needless to say, running out of stack space is always a fatal error.

In languages and systems that do not check for stack overflow, your program will crash without an explicit explanation. On these systems, strange things may happen when your stack gets too big, because your stack will run into part of your program. It could be the main program, or it could be part of your data, especially if you have a big array. If it runs into your program, your program will be corrupted; you will have nonsense instructions and will crash as soon as they are executed. If the stack runs into your data, what is likely to happen is that when you write something into your data, it will destroy stack information -- probably the return address -- and your program will attempt to return to some weird address and crash.

In normal events, you should not run out of stack space; doing so is usually an indication of runaway recursion (forgetting a base case). On the other hand, some perfectly legal and seemingly innocuous program can cause you to run out of stack space. The routine in Figure 3.54, which prints out a linked list, is perfectly legal and actually correct. It properly handles the base case of an empty list, and the recursion is fine. This program can be proven correct. Unfortunately, if the list contains 20,000 elements, there will be a stack of 20,000 activation records representing the nested calls of line 3. Activation records are typically large because of all the information they contain, so this program is likely to run out of stack space. (If 20,000 elements are not enough to make the program crash, replace the number with a larger one.)

This program is an example of an extremely bad use of recursion known as tail recursion. Tail recursion refers to a recursive call at the last line. Tail recursion can be mechanically eliminated by changing the recursive call to a goto preceded by one assignment per function argument. This simulates the recursive call because nothing needs to be saved -- after the recursive call finishes, there is really no need to know the saved values. Because of this, we can just go to the top of the function with the values that would have been used in a recursive call. The program in Figure 3.55 shows the improved version. Keep in mind that you should use the more natural while loop construction. The goto is used here to show how a compiler might automatically remove the recursion.

Removal of tail recursion is so simple that some compilers do it automatically. Even so, it is best not to find out that yours does not.

void             /* Not using a header */

print_list( LIST L )

{

/*1*/       if( L != NULL )

{

/*2*/            print_element( L->element );

/*3*/            print_list( L->next );

}

}

Figure 3.54 A bad use of recursion: printing a linked list

void

print_list( LIST L ) /* No header */

{

top:

if( L != NULL )

{

print_element( L->element );

L = L->next;

goto top;

}

}

Figure 3.55 Printing a list without recursion; a compiler might do this (you should not)

Recursion can always be completely removed (obviously, the compiler does so in converting to assembly language), but doing so can be quite tedious. The general strategy requires using a stack and is obviously worthwhile only if you can manage to put only the bare minimum on the stack. We will not dwell on this further, except to point out that although nonrecursive programs are certainly generally faster than recursive programs, the speed advantage rarely justifies the lack of clarity that results from removing the recursion.

3.4. The Queue ADT

Like stacks, queues are lists. With a queue, however, insertion is done at one end, whereas deletion is performed at the other end.

3.4.1. Queue Model

The basic operations on a queue are enqueue, which inserts an element at the end of the list (called the rear), and dequeue, which deletes (and returns) the element at the start of the list (known as the front). Figure 3.56 shows the abstract model of a queue.

Figure 3.56 Model of a queue

3.4.2. Array Implementation of Queues

As with stacks, any list implementation is legal for queues. Like stacks, both the linked list and array implementations give fast O(1) running times for every operation. The linked list implementation is straightforward and left as an exercise. We will now discuss an array implementation of queues.

For each queue data structure, we keep an array, QUEUE[], and the positions q_front and q_rear, which represent the ends of the queue. We also keep track of the number of elements that are actually in the queue, q_size. All this information is part of one structure, and as usual, except for the queue routines themselves, no routine should ever access these directly. The following figure shows a queue in some intermediate state. By the way, the cells that are blanks have undefined values in them. In particular, the first two cells have elements that used to be in the queue.

The operations should be clear. To enqueue an element x, we increment q_size and q_rear, then set QUEUE[q_rear] = x. To dequeue an element, we set the return value to QUEUE[q_front], decrement q_size, and then increment q_front. Other strategies are possible (this is discussed later). We will comment on checking for errors presently.

There is one potential problem with this implementation. After 10 enqueues, the queue appears to be full, since q_front is now 10, and the next enqueue would be in a nonexistent position. However, there might only be a few elements in the queue, because several elements may have already been dequeued. Queues, like stacks, frequently stay small even in the presence of a lot of operations.

The simple solution is that whenever q_front or q_rear gets to the end of the array, it is wrapped around to the beginning. The following figure shows the queue during some operations. This is known as a circular array implementation.

The extra code required to implement the wraparound is minimal (although it probably doubles the running time). If incrementing either q_rear or q_front causes it to go past the array, the value is reset to the first position in the array.

There are two warnings about the circular array implementation of queues. First, it is important to check the queue for emptiness, because a dequeue when the queue is empty will return an undefined value, silently.

Secondly, some programmers use different ways of representing the front and rear of a queue. For instance, some do not use an entry to keep track of the size, because they rely on the base case that when the queue is empty, q_rear = q_front - 1. The size is computed implicitly by comparing q_rear and q_front. This is a very tricky way to go, because there are some special cases, so be very careful if you need to modify code written this way. If the size is not part of the structure, then if the array size is A_SIZE, the queue is full when there are A_SIZE -1 elements, since only A_SIZE different sizes can be differentiated, and one of these is 0. Pick any style you like and make sure that all your routines are consistent. Since there are a few options for implementation, it is probably worth a comment or two in the code, if you don't use the size field.

In applications where you are sure that the number of enqueues is not larger than the size of the queue, obviously the wraparound is not necessary. As with stacks, dequeues are rarely performed unless the calling routines are certain that the queue is not empty. Thus error calls are frequently skipped for this operation, except in critical code. This is generally not justifiable, because the time savings that you are likely to achieve are too minimal.

We finish this section by writing some of the queue routines. We leave the others as an exercise to the reader. First, we give the type definitions in Figure 3.57. We add a maximum size field, as was done for the array implementation of the stack; queue_create and queue_dispose routines also need to be provided. We also provide routines to test whether a queue is empty and to make an empty queue (Figs. 3.58 and 3.59). The reader can write the function is_full, which performs the test implied by its name. Notice that q_rear is preinitialized to 1 before q_front. The final operation we will write is the enqueue routine. Following the exact description above, we arrive at the implementation in Figure 3.60.

3.4.3. Applications of Queues

There are several algorithms that use queues to give efficient running times. Several of these are found in graph theory, and we will discuss them later in Chapter 9. For now, we will give some simple examples of queue usage.

struct queue_record

{

unsigned int q_max_size;  /* Maximum # of elements */

/* until Q is full */

unsigned int q_front;

unsigned int q_rear;

unsigned int q_size;      /* Current # of elements in Q */

element_type *q_array;

};

typedef struct queue_record * QUEUE;

Figure 3.57 Type declarations for queue--array implementation

int

is_empty( QUEUE Q )

{

return( Q->q_size == 0 );

}

Figure 3.58 Routine to test whether a queue is empty-array implementation

void

make_null ( QUEUE Q )

{

Q->q_size = 0;

Q->q_front = 1;

Q->q_rear = 0;

}

Figure 3.59 Routine to make an empty queue-array implementation

unsigned int

succ( unsigned int value, QUEUE Q )

{

if( ++value == Q->q_max_size )

value = 0;

return value;

}

void

enqueue( element_type x, QUEUE Q )

{

if( is_full( Q ) )

error("Full queue");

else

{

Q->q_size++;

Q->q_rear = succ( Q->q_rear, Q );

Q->q_array[ Q->q_rear ] = x;

}

}

Figure 3.60 Routines to enqueue-array implementation

When jobs are submitted to a printer, they are arranged in order of arrival. Thus, essentially, jobs sent to a line printer are placed on a queue.*

*We say essentially a queue, because jobs can be killed. This amounts to a deletion from the middle of the queue, which is a violation of the strict definition.

Virtually every real-life line is (supposed to be) a queue. For instance, lines at ticket counters are queues, because service is first-come first-served.

Another example concerns computer networks. There are many network setups of personal computers in which the disk is attached to one machine, known as the file server. Users on other machines are given access to files on a first-come first-served basis, so the data structure is a queue.

Further examples include the following:

       Calls to large companies are generally placed on a queue when all operators are busy.

       In large universities, where resources are limited, students must sign a waiting list if all terminals are occupied. The student who has been at a terminal the longest is forced off first, and the student who has been waiting the longest is the next user to be allowed on.

A whole branch of mathematics, known as queueing theory, deals with computing, probabilistically, how long users expect to wait on a line, how long the line gets, and other such questions. The answer depends on how frequently users arrive to the line and how long it takes to process a user once the user is served. Both of these parameters are given as probability distribution functions. In simple cases, an answer can be computed analytically. An example of an easy case would be a phone line with one operator. If the operator is busy, callers are placed on a waiting line (up to some maximum limit). This problem is important for businesses, because studies have shown that people are quick to hang up the phone.

If there are k operators, then this problem is much more difficult to solve. Problems that are difficult to solve analytically are often solved by a simulation. In our case, we would need to use a queue to perform the simulation. If k is large, we also need other data structures to do this efficiently. We shall see how to do this simulation in Chapter 6. We could then run the simulation for several values of k and choose the minimum k that gives a reasonable waiting time.

Additional uses for queues abound, and as with stacks, it is staggering that such a simple data structure can be so important.

Summary

This chapter describes the concept of ADTs and illustrates the concept with three of the most common abstract data types. The primary objective is to separate the implementation of the abstract data types from their function. The program must know what the operations do, but it is actually better off not knowing how it is done.

Lists, stacks, and queues are perhaps the three fundamental data structures in all of computer science, and their use is documented through a host of examples. In particular, we saw how stacks are used to keep track of procedure and function calls and how recursion is actually implemented. This is important to understand, not just because it makes procedural languages possible, but because knowing how recursion is implemented removes a good deal of the mystery that surrounds its use. Although recursion is very powerful, it is not an entirely free operation; misuse and abuse of recursion can result in programs crashing.

Exercises

3.1     Write a program to print out the elements of a singly linked list.

3.2     You are given a linked list, L, and another linked list, P, containing integers, sorted in ascending order. The operation print_lots(L,P) will print the elements in L that are in positions specified by P. For instance, if P = 1, 3, 4, 6, the first, third, fourth, and sixth elements in L are printed. Write the routine print_lots(L,P). You should use only the basic list operations. What is the running time of your routine?

3.3     Swap two adjacent elements by adjusting only the pointers (and not the data) using

a.     singly linked lists,

b.     doubly linked lists.

3.4     Given two sorted lists, L1 and L2, write a procedure to compute L1   [image]  L2 using only the basic list operations.

3.5     Given two sorted lists, L1 and L2, write a procedure to compute L1   [image]  L2 using only the basic list operations.

3.6     Write a function to add two polynomials. Do not destroy the input. Use a linked list implementation. If the polynomials have m and n terms respectively, what is the time complexity of your program?

3.7     Write a function to multiply two polynomials, using a linked list implementation. You must make sure that the output polynomial is sorted by exponent and has at most one term of any power.

a.     Give an algorithm to solve this problem in O(m2n2) time.

*b.     Write a program to perform the multiplication in O(m2n) time, where m is the number of terms in the polynomial of fewer terms.

*c.     Write a program to perform the multiplication in O(mn log(mn)) time.

d.     Which time bound above is the best?

3.8     Write a program that takes a polynomial,   [image] (x), and computes (  [image] (x))p. What is the complexity of your program? Propose at least one alternative solution that could be competitive for some plausible choices of   [image] (x) and p.

3.9     Write an arbitrary-precision integer arithmetic package. You should use a strategy similar to polynomial arithmetic. Compute the distribution of the digits 0 to 9 in 24000.

3.10     The Josephus problem is the following mass suicide "game": n people, numbered 1 to n, are sitting in a circle. Starting at person 1, a handgun is passed. After m passes, the person holding the gun commits suicide, the body is removed, the circle closes ranks, and the game continues with the person who was sitting after the corpse picking up the gun. The last survivor is tried for n - 1 counts of manslaughter. Thus, if m = 0 and n = 5, players are killed in order and player 5 stands trial. If m = 1 and n = 5, the order of death is 2, 4, 1, 5.

a.     Write a program to solve the Josephus problem for general values of m and n. Try to make your program as efficient as possible. Make sure you dispose of cells.

b.     What is the running time of your program?

c.     If m = 1, what is the running time of your program? How is the actual speed affected by the free routine for large values of n (n > 10000)?

3.11     Write a program to find a particular element in a singly linked list. Do this both recursively and nonrecursively, and compare the running times. How big does the list have to be before the recursive version crashes?

3.12     a.     Write a nonrecursive procedure to reverse a singly linked list in O(n) time.

*b.     Write a procedure to reverse a singly linked list in O(n) time using constant extra space.

3.13     You have to sort an array of student records by social security number. Write a program to do this, using radix sort with 1000 buckets and three passes.

3.14     Write a program to read a graph into adjacency lists using

a.     linked lists

b.     cursors

3.15 a.     Write an array implementation of self-adjusting lists. A self-adjusting list is like a regular list, except that all insertions are performed at the front, and when an element is accessed by a find, it is moved to the front of the list without changing the relative order of the other items.

b.     Write a linked list implementation of self-adjusting lists.

*c.     Suppose each element has a fixed probability, pi, of being accessed. Show that the elements with highest access probability are expected to be close to the front.

3.16     Suppose we have an array-based list a[0..n -1] and we want to delete all duplicates. last_position is initially n - 1, but gets smaller as elements are deleted. Consider the pseudocode program fragment in Figure 3.61. The procedure DELETE deletes the element in position j and collapses the list.

a.     Explain how this procedure works.

b.     Rewrite this procedure using general list operations.

/*1*/  for( i=0; i<last_position; i++ )

{

/*2*/       j = i + 1;

/*3*/       while( j<last_position )

/*4*/            if( a[i] == a[j]

/*5*/                 DELETE(j);

                 else

/*6*/                 j++;

}

Figure 3.61 Routine to remove duplicates from a lists--array implementation

*c.     Using a standard array implementation, what is the running time of this procedure?

d.     What is the running time using a linked list implementation?

*e.     Give an algorithm to solve this problem in O(n log n) time.

**f.     Prove that any algorithm to solve this problem requires   [image] (n log n) comparisons if only comparisons are used. Hint: Look to Chapter 7.

*g.     Prove that if we allow operations besides comparisons, and the keys are real numbers, then we can solve the problem without using comparisons between elements.

3.17     An alternative to the deletion strategy we have given is to use lazy deletion. To delete an element, we merely mark it deleted (using an extra bit field). The number of deleted and nondeleted elements in the list is kept as part of the data structure. If there are as many deleted elements as nondeleted elements, we traverse the entire list, performing the standard deletion algorithm on all marked nodes.

a.     List the advantages and disadvantages of lazy deletion.

b.     Write routines to implement the standard linked list operations using lazy deletion.

3.18     Write a program to check for balancing symbols in the following languages:

a.     Pascal (begin/end, ( ), [ ], { }).

b.     C (/* */, ( ), [ ], { }).

*c.     Explain how to print out an error message that is likely to reflect the probable cause.

3.19     Write a program to evaluate a postfix expression.

3.20      a.     Write a program to convert an infix expression which includes '(', ')', '+', '-', '*' and '/' to postfix.

b.     Add the exponentiation operator to your repertoire.

c.     Write a program to convert a postfix expression to infix.

3.21     Write routines to implement two stacks using only one array. Your stack routines should not declare an overflow unless every slot in the array is used.

3.22     *a.     Propose a data structure that supports the stack push and pop operations and a third operation find_min, which returns the smallest element in the data structure, all in O(1) worst case time.

*b.     Prove that if we add the fourth operation delete_min which finds and removes the smallest element, then at least one of the operations must take   [image] (logn) time. (This requires reading Chapter 7.)

3.23     *Show how to implement three stacks in one array.

3.24     If the recursive routine in Section 2.4 used to compute Fibonacci numbers is run for n = 50, is stack space likely to run out? Why or why not?

3.25     Write the routines to implement queues using

a.     linked lists

b.     arrays

3.26     A deque is a data structure consisting of a list of items, on which the following operations are possible:

push(x,d): Insert item x on the front end of deque d.

pop(d): Remove the front item from deque d and return it.

inject(x,d): Insert item x on the rear end of deque d.

eject(d): Remove the rear item from deque d and return it.

Write routines to support the deque that take O(1) time per operation.

Go to Chapter 4            Return to Table of Contents


Structures, Algorithm Analysis: CHAPTER 4: TREES

 Return to Table of Contents

CHAPTER 4: TREES

For large amounts of input, the linear access time of linked lists is prohibitive. In this chapter we look at a simple data structure for which the running time of most operations is O(log n) on average. We also sketch a conceptually simple modification to this data structure that guarantees the above time bound in the worst case and discuss a second modification that essentially gives an O(log n) running time per operation for a long sequence of instructions.

The data structure that we are referring to is known as a binary search tree. Trees in general are very useful abstractions in computer science, so we will discuss their use in other, more general applications. In this chapter, we will

See how trees are used to implement the file system of several popular operating systems.

See how trees can be used to evaluate arithmetic expressions.

Show how to use trees to support searching operations in O(log n) average time, and how to refine these ideas to obtain O(log n) worst-case bounds. We will also see how to implement these operations when the data is stored on a disk.

4.1. Preliminaries

A tree can be defined in several ways. One natural way to define a tree is recursively. A tree is a collection of nodes. The collection can be empty, which is sometimes denoted as A. Otherwise, a tree consists of a distinguished node r, called the root, and zero or more (sub)trees T1, T2, . . . , Tk, each of whose roots are connected by a directed edge to r.

The root of each subtree is said to be a child of r, and r is the parent of each subtree root. Figure 4.1 shows a typical tree using the recursive definition.

From the recursive definition, we find that a tree is a collection of n nodes, one of which is the root, and n - 1 edges. That there are n - 1 edges follows from the fact that each edge connects some node to its parent, and every node except the root has one parent (see Fig. 4.2).

Figure 4.1 Generic tree

Figure 4.2 A tree

In the tree of Figure 4.2, the root is A. Node F has A as a parent and K, L, and M as children. Each node may have an arbitrary number of children, possibly zero. Nodes with no children are known as leaves; the leaves in the tree above are B, C, H, I, P, Q, K, L, M, and N. Nodes with the same parent are siblings; thus K, L, and M are all siblings. Grandparent and grandchild relations can be defined in a similar manner.

A path from node n1 to nk is defined as a sequence of nodes n1, n2, . . . , nk such that ni is the parent of ni+1 for 1 [image] i < k. The length of this path is the number of edges on the path, namely k -1. There is a path of length zero from every node to itself. Notice that in a tree there is exactly one path from the root to each node.

For any node ni, the depth of ni is the length of the unique path from the root to ni. Thus, the root is at depth 0. The height of ni is the longest path from ni to a leaf. Thus all leaves are at height 0. The height of a tree is equal to the height of the root. For the tree in Figure 4.2, E is at depth 1 and height 2; F is at depth 1 and height 1; the height of the tree is 3. The depth of a tree is equal to the depth of the deepest leaf; this is always equal to the height of the tree.

If there is a path from n1 to n2, then n1 is an ancestor of n2 and n2 is a descendant of n1. If n1 [image] n2, then n1 is a proper ancestor of n2 and n2 is a proper descendant of n1.

4.1.1. Implementation of Trees

One way to implement a tree would be to have in each node, besides its data, a pointer to each child of the node. However, since the number of children per node can vary so greatly and is not known in advance, it might be infeasible to make the children direct links in the data structure, because there would be too much wasted space. The solution is simple: Keep the children of each node in a linked list of tree nodes. The declaration in Figure 4.3 is typical.

typedef struct tree_node *tree_ptr;

struct tree_node

{

element_type element;

tree_ptr first_child;

tree_ptr next_sibling;

};

Figure 4.3 Node declarations for trees

Figure 4.4 First child/next sibling representation of the tree shown in Figure 4.2

Figure 4.4 shows how a tree might be represented in this implementation. Arrows that point downward are first_child pointers. Arrows that go left to right are next_sibling pointers. Null pointers are not drawn, because there are too many.

In the tree of Figure 4.4, node E has both a pointer to a sibling (F) and a pointer to a child (I), while some nodes have neither.

4.1.2. Tree Traversals with an Application

There are many applications for trees. One of the popular uses is the directory structure in many common operating systems, including UNIX, VAX/VMS, and DOS. Figure 4.5 is a typical directory in the UNIX file system.

The root of this directory is /usr. (The asterisk next to the name indicates that /usr is itself a directory.) /usr has three children, mark, alex, and bill, which are themselves directories. Thus, /usr contains three directories and no regular files. The filename /usr/mark/book/ch1.r is obtained by following the leftmost child three times. Each / after the first indicates an edge; the result is the full pathname. This hierarchical file system is very popular, because it allows users to organize their data logically. Furthermore, two files in different directories can share the same name, because they must have different paths from the root and thus have different pathnames. A directory in the UNIX file system is just a file with a list of all its children, so the directories are structured almost exactly in accordance with the type declaration above.* Indeed, if the normal command to print a file is applied to a directory, then the names of the files in the directory can be seen in the output (along with other non-ASCII information).

*Each directory in the UNIX file system also has one entry that points to itself and another entry that points to the parent of the directory. Thus, technically, the UNIX file system is not a tree, but is treelike.

Figure 4.5 Unix directory

void

list_directory ( Directory_or_file D )

{

list_dir ( D, 0 );

}

void

list_dir ( Directory_or_file D, unsigned int depth )

{

/*1*/        if ( D is a legitimate entry)

{

/*2*/             print_name ( depth, D );

/*3*/             if( D is a directory )

/*4*/                  for each child, c, of D

/*5*/                       list_dir( c, depth+1 );

}

}

Figure 4.6 Routine to list a directory in a hierarchical file system

Suppose we would like to list the names of all of the files in the directory. Our output format will be that files that are depth d will have their names indented by d tabs. Our algorithm is given in Figure 4.6.

The heart of the algorithm is the recursive procedure list_dir. This routine needs to be started with the directory name and a depth of 0, to signify no indenting for the root. This depth is an internal bookkeeping variable, and is hardly a parameter that a calling routine should be expected to know about. Thus the driver routine list_directory is used to interface the recursive routine to the outside world.

The logic of the algorithm is simple to follow. The argument to list_dir is some sort of pointer into the tree. As long as the pointer is valid, the name implied by the pointer is printed out with the appropriate number of tabs. If the entry is a directory, then we process all children recursively, one by one. These children are one level deeper, and thus need to be indented an extra space. The output is in Figure 4.7.

This traversal strategy is known as a preorder traversal. In a preorder traversal, work at a node is performed before (pre) its children are processed. When this program is run, it is clear that line 2 is executed exactly once per node, since each name is output once. Since line 2 is executed at most once per node, line 3 must also be executed once per node. Furthermore, line 5 can be executed at most once for each child of each node. But the number of children is exactly one less than the number of nodes. Finally, the for loop iterates once per execution of line 5, plus once each time the loop ends. Each for loop terminates on a NULL pointer, but there is at most one of those per node. Thus, the total amount of work is constant per node. If there are n file names to be output, then the running time is O(n).

/usr

mark

book

chr1.c

chr2.c

chr3.c

course

cop3530

fall88

syl.r

spr89

syl.r

sum89

syl.r

junk.c

alex

junk.c

bill

work

course

cop3212

fall88

grades

prog1.r

prog2.r

fall89

prog1.r

prog2.r

grades

Figure 4.7 The (preorder) directory listing

Another common method of traversing a tree is the postorder traversal. In a postorder traversal, the work at a node is performed after (post) its children are evaluated. As an example, Figure 4.8 represents the same directory structure as before, with the numbers in parentheses representing the number of disk blocks taken up by each file.

Since the directories are themselves files, they have sizes too. Suppose we would like to calculate the total number of blocks used by all the files in the tree. The most natural way to do this would be to find the number of blocks contained in the subdirectories /usr/mark (30), /usr/alex (9), and /usr/bill (32). The total number of blocks is then the total in the subdirectories (71) plus the one block used by /usr, for a total of 72. The function size_directory in Figure 4.9 implements this strategy.

Figure 4.8 Unix directory with file sizes obtained via postorder traversal

unsigned int

size_directory( Directory_or_file D )

{

unsigned int total_size;

/*1*/         total_size = 0;

/*2*/         if( D is a legitimate entry)

{

/*3*/              total_size = file_size( D );

/*4*/              if( D is a directory )

/*5*/                   for each child, c, of D

/*6*/                        total_size += size_directory( c );

}

/*7*/         return( total_size );

}

Figure 4.9 Routine to calculate the size of a directory

                ch1.r                3

                ch2.r                2

                ch3.r                4

           book                     10

                syl.r                1

                     fall88          2

                syl.r                5

                     spr89           6

                syl.r                2

                     sum89           3

                cop3530             12

           course                   13

           junk.c                    6

      mark                          30

           junk.c                    8

      alex                           9

           work                      1

                         grades      3

                         prog1.r     4

                         prog2.r     1

                    fall88           9

                         prog2.r     2

                         prog1.r     7

                         grades      9

                    fall89          19

               cop3212              29

          course                    30

     bill                           32

/usr                                72

Figure 4.10 Trace of the size function

If D is not a directory, then size_directory merely returns the number of blocks used by D. Otherwise, the number of blocks used by D is added to the number of blocks (recursively) found in all of the children. To see the difference between the postorder traversal strategy and the preorder traversal strategy, Figure 4.10 shows how the size of each directory or file is produced by the algorithm.

4.2. Binary Trees

A binary tree is a tree in which no node can have more than two children.

Figure 4.11 shows that a binary tree consists of a root and two subtrees, Tl and Tr, both of which could possibly be empty.

A property of a binary tree that is sometimes important is that the depth of an average binary tree is considerably smaller than n. An analysis shows that the average depth is   [image], and that for a special type of binary tree, namely the binary search tree, the average value of the depth is O(log n). Unfortunately, the depth can be as large as n -1, as the example in Figure 4.12 shows.

Figure 4.11 Generic binary tree

Figure 4.12 Worst-case binary tree

4.2.1. Implementation

Because a binary tree has at most two children, we can keep direct pointers to them. The declaration of tree nodes is similar in structure to that for doubly linked lists, in that a node is a structure consisting of the key information plus two pointers (left and right) to other nodes (see

typedef struct tree_node *tree_ptr;

struct tree_node

{

element_type element;

tree_ptr left;

tree_ptr right;

};

typedef tree_ptr TREE;

Figure 4.13 Binary tree node declarations

Many of the rules that apply to linked lists will apply to trees as well. In particular, when an insertion is performed, a node will have to be created by a call to malloc. Nodes can be freed after deletion by calling free.

We could draw the binary trees using the rectangular boxes that are customary for linked lists, but trees are generally drawn as circles connected by lines, because they are actually graphs. We also do not explicitly draw NULL pointers when referring to trees, because every binary tree with n nodes would require n + 1 NULL pointers.

Binary trees have many important uses not associated with searching. One of the principal uses of binary trees is in the area of compiler design, which we will now explore.

4.2.2. Expression Trees

Figure 4.14 shows an example of an expression tree. The leaves of an expression tree are operands, such as constants or variable names, and the other nodes contain operators. This particular tree happens to be binary, because all of the operations are binary, and although this is the simplest case, it is possible for nodes to have more than two children. It is also possible for a node to have only one child, as is the case with the unary minus operator. We can evaluate an expression tree, T, by applying the operator at the root to the values obtained by recursively evaluating the left and right subtrees. In our example, the left subtree evaluates to a + (b * c) and the right subtree evaluates to ((d *e) + f )*g. The entire tree therefore represents (a + (b*c)) + (((d * e) + f)* g).

We can produce an (overly parenthesized) infix expression by recursively producing a parenthesized left expression, then printing out the operator at the root, and finally recursively producing a parenthesized right expression. This general strattegy ( left, node, right ) is known as an inorder traversal; it is easy to remember because of the type of expression it produces.

An alternate traversal strategy is to recursively print out the left subtree, the right subtree, and then the operator. If we apply this strategy to our tree above, the output is a b c * + d e * f + g * +, which is easily seen to be the postfix representation of Section 3.3.3. This traversal strategy is generally known as a postorder traversal. We have seen this traversal strategy earlier in Section 4.1.

Figure 4.14 Expression tree for (a + b * c) + ((d * e + f ) * g)

A third traversal strategy is to print out the operator first and then recursively print out the left and right subtrees. The resulting expression, + + a * b c * + * d e f g, is the less useful prefix notation and the traversal strategy is a preorder traversal, which we have also seen earlier in Section 4.1. We will return to these traversal strategies once again later in the chapter.

Constructing an Expression Tree

We now give an algorithm to convert a postfix expression into an expression tree. Since we already have an algorithm to convert infix to postfix, we can generate expression trees from the two common types of input. The method we describe strongly resembles the postfix evaluation algorithm of Section 3.2.3. We read our expression one symbol at a time. If the symbol is an operand, we create a one-node tree and push a pointer to it onto a stack. If the symbol is an operator, we pop pointers to two trees T1 and T2 from the stack (T1 is popped first) and form a new tree whose root is the operator and whose left and right children point to T2 and T1 respectively. A pointer to this new tree is then pushed onto the stack.

As an example, suppose the input is

a b + c d e + * *

The first two symbols are operands, so we create one-node trees and push pointers to them onto a stack.*

*For convenience, we will have the stack grow from left to right in the diagrams.

Next, a '+' is read, so two pointers to trees are popped, a new tree is formed, and a pointer to it is pushed onto the stack.*

Next, c, d, and e are read, and for each a one-node tree is created and a pointer to the corresponding tree is pushed onto the stack.

Now a '+' is read, so two trees are merged.

Continuing, a '*' is read, so we pop two tree pointers and form a new tree with a '*' as root.

Finally, the last symbol is read, two trees are merged, and a pointer to the final tree is left on the stack.

4.3. The Search Tree ADT-Binary Search Trees

An important application of binary trees is their use in searching. Let us assume that each node in the tree is assigned a key value. In our examples, we will assume for simplicity that these are integers, although arbitrarily complex keys are allowed. We will also assume that all the keys are distinct, and deal with duplicates later.

The property that makes a binary tree into a binary search tree is that for every node, X, in the tree, the values of all the keys in the left subtree are smaller than the key value in X, and the values of all the keys in the right subtree are larger than the key value in X. Notice that this implies that all the elements in the tree can be ordered in some consistent manner. In Figure 4.15, the tree on the left is a binary search tree, but the tree on the right is not. The tree on the right has a node with key 7 in the left subtree of a node with key 6 (which happens to be the root).

We now give brief descriptions of the operations that are usually performed on binary search trees. Note that because of the recursive definition of trees, it is common to write these routines recursively. Because the average depth of a binary search tree is O(log n), we generally do not need to worry about running out of stack space. We repeat our type definition in Figure 4.16. Since all the elements can be ordered, we will assume that the operators <, >, and = can be applied to them, even if this might be syntactically erroneous for some types.

Figure 4.15 Two binary trees (only the left tree is a search tree)

typedef struct tree_node *tree_ptr;

struct tree_node

{

element_type element;

tree_ptr left;

tree_ptr right;

};

typedef tree_ptr SEARCH_TREE;

Figure 4.16 Binary search tree declarations

4.3.1. Make_null

This operation is mainly for initialization. Some programmers prefer to initialize the first element as a one-node tree, but our implementation follows the recursive definition of trees more closely. It is also a simple routine, as evidenced by Figure 4.17.

4.3.2. Find

This operation generally requires returning a pointer to the node in tree T that has key x, or NULL if there is no such node. The structure of the tree makes this simple. If T is , then we can just return . Otherwise, if the key stored at T is x, we can return T. Otherwise, we make a recursive call on a subtree of T, either left or right, depending on the relationship of x to the key stored in T. The code in Figure 4.18 is an implementation of this strategy.

SEARCH_TREE

make_null ( void )

{

return NULL;

}

Figure 4.17 Routine to make an empty tree

tree_ptr

find( element_type x, SEARCH_TREE T )

{

if( T == NULL )

return NULL;

if( x < T->element )

return( find( x, T->left ) );

else

if( x > T->element )

return( find( x, T->right ) );

else

return T;

}

Figure 4.18 Find operation for binary search trees

Notice the order of the tests. It is crucial that the test for an empty tree be performed first, since otherwise the indirections would be on a NULL pointer. The remaining tests are arranged with the least likely case last. Also note that both recursive calls are actually tail recursions and can be easily removed with an assignment and a goto. The use of tail recursion is justifiable here because the simplicity of algorithmic expression compensates for the decrease in speed, and the amount of stack space used is expected to be only O(log n).

4.3.3. Find_min and find_max

These routines return the position of the smallest and largest elements in the tree, respectively. Although returning the exact values of these elements might seem more reasonable, this would be inconsistent with the find operation. It is important that similar-looking operations do similar things. To perform a find_min, start at the root and go left as long as there is a left child. The stopping point is the smallest element. The find_max routine is the same, except that branching is to the right child.

This is so easy that many programmers do not bother using recursion. We will code the routines both ways by doing find_min recursively and find_max nonrecursively (see Figs. 4.19 and 4.20).

Notice how we carefully handle the degenerate case of an empty tree. Although this is always important to do, it is especially crucial in recursive programs. Also notice that it is safe to change T in find_max, since we are only working with a copy. Always be extremely careful, however, because a statement such as T -> right : =T -> right -> right will make changes in most languages.

tree_ptr

find_min( SEARCH_TREE T )

{

if( T == NULL )

return NULL;

else

if( T->left == NULL )

return( T );

else

return( find_min ( T->left ) );

}

Figure 4.19 Recursive implementation of find_min for binary search trees

tree_ptr

find_max( SEARCH_TREE T )

{

if( T != NULL )

while( T->right != NULL )

T = T->right;

return T;

}

Figure 4.20 Nonrecursive implementation of find_max for binary search trees

4.3.4. Insert

The insertion routine is conceptually simple. To insert x into tree T, proceed down the tree as you would with a find. If x is found, do nothing (or "update" something). Otherwise, insert x at the last spot on the path traversed. Figure 4.21 shows what happens. To insert 5, we traverse the tree as though a find were occurring. At the node with key 4, we need to go right, but there is no subtree, so 5 is not in the tree, and this is the correct spot.

Duplicates can be handled by keeping an extra field in the node record indicating the frequency of occurrence. This adds some extra space to the entire tree, but is better than putting duplicates in the tree (which tends to make the tree very deep). Of course this strategy does not work if the key is only part of a larger record. If that is the case, then we can keep all of the records that have the same key in an auxiliary data structure, such as a list or another search tree.

Figure 4.21 Binary search trees before and after inserting 5

Figure 4.22 shows the code for the insertion routine. Since T points to the root of the tree, and the root changes on the first insertion, insert is written as a function that returns a pointer to the root of the new tree. Lines 8 and 10 recursively insert and attach x into the appropriate subtree.

tree_ptr

insert( element_type x, SEARCH_TREE T )

{

/*1*/       if( T == NULL )

{  /* Create and return a one-node tree */

/*2*/            T = (SEARCH_TREE) malloc ( sizeof (struct tree_node) );

/*3*/            if( T == NULL )

/*4*/                fatal_error("Out of space!!!");

else

{

/*5*/                T->element = x;

/*6*/                T->left = T->right = NULL;

}

}

else

/*7*/       if( x < T->element )

/*8*/            T->left = insert( x, T->left );

else

/*9*/       if( x > T->element )

/*10*/           T->right = insert( x, T->right );

/* else x is in the tree already. We'll do nothing */

/*11*/      return T; /* Don't forget this line!! */

}

Figure 4.22 Insertion into a binary search tree

4.3.5. Delete

As is common with many data structures, the hardest operation is deletion. Once we have found the node to be deleted, we need to consider several possibilities.

If the node is a leaf, it can be deleted immediately. If the node has one child, the node can be deleted after its parent adjusts a pointer to bypass the node (we will draw the pointer directions explicitly for clarity). See Figure 4.23. Notice that the deleted node is now unreferenced and can be disposed of only if a pointer to it has been saved.

The complicated case deals with a node with two children. The general strategy is to replace the key of this node with the smallest key of the right subtree (which is easily found) and recursively delete that node (which is now empty). Because the smallest node in the right subtree cannot have a left child, the second delete is an easy one. Figure 4.24 shows an initial tree and the result of a deletion. The node to be deleted is the left child of the root; the key value is 2. It is replaced with the smallest key in its right subtree (3), and then that node is deleted as before.

Figure 4.23 Deletion of a node (4) with one child, before and after

Figure 4.24 Deletion of a node (2) with two children, before and after

The code in Figure 4.25 performs deletion. It is inefficient, because it makes two passes down the tree to find and delete the smallest node in the right subtree when this is appropriate. It is easy to remove this inefficiency, by writing a special delete_min function, and we have left it in only for simplicity.

If the number of deletions is expected to be small, then a popular strategy to use is lazy deletion: When an element is to be deleted, it is left in the tree and merely marked as being deleted. This is especially popular if duplicate keys are present, because then the field that keeps count of the frequency of appearance can be decremented. If the number of real nodes in the tree is the same as the number of "deleted" nodes, then the depth of the tree is only expected to go up by a small constant (why?), so there is a very small time penalty associated with lazy deletion. Also, if a deleted key is reinserted, the overhead of allocating a new cell is avoided.

tree_ptr

delete( element_type x, SEARCH_TREE T )

{

tree_ptr tmp_cell, child;

if( T == NULL )

error("Element not found");

else

if( x < T->element )  /* Go left */

T->left = delete( x, T->left );

else

if( x > T->element )  /* Go right */

T->right = delete( x, T->right );

else      /* Found element to be deleted */

if( T->left && T->right )  /* Two children */

{      /* Replace with smallest in right subtree */

tmp_cell = find_min( T->right );

T->element = tmp_cell->element;

T->right = delete( T->element, T->right );

}

else      /* One child */

}

tmp_cell = T;

if( T->left == NULL )      /* Only a right child */

child = T->right;

if( T->right == NULL )     /* Only a left child */

child = T->left;

free( tmp_cell );

return child;

}

return T;

}

Figure 4.25 Deletion routine for binary search trees

4.3.6. Average-Case Analysis

Intuitively, we expect that all of the operations of the previous section, except make_null, should take O(log n) time, because in constant time we descend a level in the tree, thus operating on a tree that is now roughly half as large. Indeed, the running time of all the operations, except make_null, is O(d), where d is the depth of the node containing the accessed key.

We prove in this section that the average depth over all nodes in a tree is O(log n) on the assumption that all trees are equally likely.

The sum of the depths of all nodes in a tree is known as the internal path length. We will now calculate the average internal path length of a binary search tree, where the average is taken over all possible binary search trees.

Let D(n) be the internal path length for some tree T of n nodes. D(1) = 0. An n-node tree consists of an i-node left subtree and an (n - i - 1)-node right subtree, plus a root at depth zero for 0   [image]  i < n. D(i) is the internal path length of the left subtree with respect to its root. In the main tree, all these nodes are one level deeper. The same holds for the right subtree. Thus, we get the recurrence

D(n) = D(i) + D(n - i -1) + n -1

If all subtree sizes are equally likely, which is true for binary search trees (since the subtree size depends only on the relative rank of the first element inserted into the tree), but not binary trees, then the average value of both D(i) and D(n - i -1) is   [image]. This yields

This recurrence will be encountered and solved in Chapter 7, obtaining an average value of D(n) = O(n log n). Thus, the expected depth of any node is O(log n). As an example, the randomly generated 500-node tree shown in Figure 4.26 has nodes at expected depth 9.98.

It is tempting to say immediately that this result implies that the average running time of all the operations discussed in the previous section is O(log n), but this is not entirely true. The reason for this is that because of deletions, it is not clear that all binary search trees are equally likely. In particular, the deletion algorithm described above favors making the left subtrees deeper than the right, because we are always replacing a deleted node with a node from the right subtree. The exact effect of this strategy is still unknown, but it seems only to be a theoretical novelty. It has been shown that if we alternate insertions and deletions   [image] (n2) times, then the trees will have an expected depth of   [image]. After a quarter-million random insert/delete pairs, the tree that was somewhat right-heavy in Figure 4.26 looks decidedly unbalanced (average depth = 12.51). See Figure 4.27.

Figure 4.26 A randomly generated binary search tree

We could try to eliminate the problem by randomly choosing between the smallest element in the right subtree and the largest in the left when replacing the deleted element. This apparently eliminates the bias and should keep the trees balanced, but nobody has actually proved this. In any event, this phenomenon appears to be mostly a theoretical novelty, because the effect does not show up at all for small trees, and stranger still, if o(n2) insert/delete pairs are used, then the tree seems to gain balance!

Figure 4.27 Binary search tree after O(n2) insert/delete pairs

The main point of this discussion is that deciding what "average" means is generally extremely difficult and can require assumptions which may or may not be valid. In the absence of deletions, or when lazy deletion is used, it can be shown that all binary search trees are equally likely and we can conclude that the average running times of the operations above are O(log n). Except for strange cases like the one discussed above, this result is very consistent with observed behavior.

If the input comes into a tree presorted, then a series of inserts will take quadratic time and give a very expensive implementation of a linked list, since the tree will consist only of nodes with no left children. One solution to the problem is to insist on an extra structural condition called balance: no node is allowed to get too deep.

There are quite a few general algorithms to implement balanced trees. Most are quite a bit more complicated than a standard binary search tree, and all take longer on average. They do, however, provide protection against the embarrassingly simple cases. Below, we will sketch one of the oldest forms of balanced search trees, the AVL tree.

A second, newer, method is to forego the balance condition and allow the tree to be arbitrarily deep, but after every operation, a restructuring rule is applied that tends to make future operations efficient. These types of data structures are generally classified as self-adjusting. In the case of a binary search tree, we can no longer guarantee an O(log n) bound on any single operation, but can show that any sequence of m operations takes total time O(m log n) in the worst case. This is generally sufficient protection against a bad worst case. The data structure we will discuss is known as a splay tree; its analysis is fairly intricate and is discussed in Chapter 11.

4.4. AVL Trees

An AVL (Adelson-Velskii and Landis) tree is a binary search tree with a balance condition. The balance condition must be easy to maintain, and it ensures that the depth of the tree is O(log n). The simplest idea is to require that the left and right subtrees have the same height. As Figure 4.28 shows, this idea does not force the tree to be shallow.

Figure 4.28 A bad binary tree. Requiring balance at the root is not enough.

Another balance condition would insist that every node must have left and right subtrees of the same height. If the height of an empty subtree is defined to be -1 (as is usual), then only perfectly balanced trees of 2k - 1 nodes would satisfy this criterion. Thus, although this guarantees trees of small depth, the balance condition is too rigid to be useful and needs to be relaxed.

An AVL tree is identical to a binary search tree, except that for every node in the tree, the height of the left and right subtrees can differ by at most 1. (The height of an empty tree is defined to be -1.) In Figure 4.29 the tree on the left is an AVL tree, but the tree on the right is not. Height information is kept for each node (in the node structure). It is easy to show that the height of an AVL tree is at most roughly 1.44 log(n + 2) - .328, but in practice it is about log(n + 1) + 0.25 (although the latter claim has not been proven). As an example, the AVL tree of height 9 with the fewest nodes (143) is shown in Figure 4.30. This tree has as a left subtree an AVL tree of height 7 of minimum size. The right subtree is an AVL tree of height 8 of minimum size. This tells us that the minimum number of nodes, N(h), in an AVL tree of height h is given by N(h) = N(h -1) + N(h - 2) + 1. For h = 0, N(h) = 1. For h = 1, N(h) = 2. The function N(h) is closely related to the Fibonacci numbers, from which the bound claimed above on the height of an AVL tree follows.

Thus, all the tree operations can be performed in O(log n) time, except possibly insertion (we will assume lazy deletion). When we do an insertion, we need to update all the balancing information for the nodes on the path back to the root, but the reason that insertion is potentially difficult is that inserting a node could violate the AVL tree property. (For instance, inserting   [image]  into the AVL tree in Figure 4.29 would destroy the balance condition at the node with key 8.) If this is the case, then the property has to be restored before the insertion step is considered over. It turns out that this can always be done with a simple modification to the tree, known as a rotation. We describe rotations in the following section.

Figure 4.29 Two binary search trees. Only the left tree is AVL.

Figure 4.30 Smallest AVL tree of height 9

4.4.1. Single Rotation

4.4.2. Double Rotation

4.4.1. Single Rotation

The two trees in Figure 4.31 contain the same elements and are both binary search trees. First of all, in both trees k1 < k2. Second, all elements in the subtree X are smaller than k1 in both trees. Third, all elements in subtree Z are larger than k2. Finally, all elements in subtree Y are in between k1 and k2. The conversion of one of the above trees to the other is known as a rotation. A rotation involves only a few pointer changes (we shall see exactly how many later), and changes the structure of the tree while preserving the search tree property.

The rotation does not have to be done at the root of a tree; it can be done at any node in the tree, since that node is the root of some subtree. It can transform either tree into the other. This gives a simple method to fix up an AVL tree if an insertion causes some node in an AVL tree to lose the balance property: Do a rotation at that node. The basic algorithm is to start at the node inserted and travel up the tree, updating the balance information at every node on the path. If we get to the root without having found any badly balanced nodes, we are done. Otherwise, we do a rotation at the first bad node found, adjust its balance, and are done (we do not have to continue going to the root). In many cases, this is sufficient to rebalance the tree. For instance, in Figure 4.32, after the insertion of the   [image]  in the original AVL tree on the left, node 8 becomes unbalanced. Thus, we do a single rotation between 7 and 8, obtaining the tree on the right.

Figure 4.31 Single rotation

Figure 4.32 AVL property destroyed by insertion of   , then fixed by a rotation

Let us work through a rather long example. Suppose we start with an initially empty AVL tree and insert the keys 1 through 7 in sequential order. The first problem occurs when it is time to insert key 3, because the AVL property is violated at the root. We perform a single rotation between the root and its right child to fix the problem. The tree is shown in the following figure, before and after the rotation:

To make things clearer, a dashed line indicates the two nodes that are the subject of the rotation. Next, we insert the key 4, which causes no problems, but the insertion of 5 creates a violation at node 3, which is fixed by a single rotation. Besides the local change caused by the rotation, the programmer must remember that the rest of the tree must be informed of this change. Here, this means that 2's right child must be reset to point to 4 instead of 3. This is easy to forget to do and would destroy the tree (4 would be inaccessible).

Next, we insert 6. This causes a balance problem for the root, since its left subtree is of height 0, and its right subtree would be height 2. Therefore, we perform a single rotation at the root between 2 and 4.

The rotation is performed by making 2 a child of 4 and making 4's original left subtree the new right subtree of 2. Every key in this subtree must lie between 2 and 4, so this transformation makes sense. The next key we insert is 7, which causes another rotation.

4.4.2. Double Rotation

The algorithm described in the preceding paragraphs has one problem. There is a case where the rotation does not fix the tree. Continuing our example, suppose we insert keys 8 through 15 in reverse order. Inserting 15 is easy, since it does not destroy the balance property, but inserting 14 causes a height imbalance at node 7.

As the diagram shows, the single rotation has not fixed the height imbalance. The problem is that the height imbalance was caused by a node inserted into the tree containing the middle elements (tree Y in Fig. 4.31) at the same time as the other trees had identical heights. The case is easy to check for, and the solution is called a double rotation, which is similar to a single rotation but involves four subtrees instead of three. In Figure 4.33, the tree on the left is converted to the tree on the right. By the way, the effect is the same as rotating between k1 and k2 and then between k2 and k3. There is a symmetric case, which is also shown (see Fig. 4.34).

Figure 4.33 (Right-left) double rotation

Figure 4.34 (Left-right) double rotation

In our example, the double rotation is a right-left double rotation and involves 7, 15, and 14. Here, k3 is the node with key 7, k1 is the node with key 15, and k2 is the node with key 14. Subtrees A, B, C, and D are all empty.

Next we insert 13, which requires a double rotation. Here the double rotation is again a right-left double rotation that will involve 6, 14, and 7 and will restore the tree. In this case, k3 is the node with key 6, k1 is the node with key 14, and k2 is the node with key 7. Subtree A is the tree rooted at the node with key 5, subtree B is the empty subtree that was originally the left child of the node with key 7, subtree C is the tree rooted at the node with key 13, and finally, subtree D is the tree rooted at the node with key 15.

If 12 is now inserted, there is an imbalance at the root. Since 12 is not between 4 and 7, we know that the single rotation will work.

Insertion of 11 will require a single rotation:

To insert 10, a single rotation needs to be performed, and the same is true for the subsequent insertion of 9. We insert 8 without a rotation, creating the almost perfectly balanced tree that follows.

Finally, we insert   [image]  to show the symmetric case of the double rotation. Notice that    [image]  causes the node containing 9 to become unbalanced. Since   [image]  is between 9 and 8 (which is 9's child on the path to   [image], a double rotation needs to be performed, yielding the following tree.

The reader can verify that any imbalance caused by an insertion into an AVL tree can always be fixed by either a single or double rotation. The programming details are fairly straightforward, except that there are several cases. To insert a new node with key x into an AVL tree T, we recursively insert x into the appropriate subtree of T (let us call this Tlr). If the height of Tlr does not change, then we are done. Otherwise, if a height imbalance appears in T, we do the appropriate single or double rotation depending on x and the keys in T and Tlr, update the heights (making the connection from the rest of the tree above), and are done. Since one rotation always suffices, a carefully coded nonrecursive version generally turns out to be significantly faster than the recursive version. However, nonrecursive versions are quite difficult to code correctly, so many programmers implement AVL trees recursively.

Another efficiency issue concerns storage of the height information. Since all that is really required is the difference in height, which is guaranteed to be small, we could get by with two bits (to represent +1, 0, -1) if we really try. Doing so will avoid repetitive calculation of balance factors but results in some loss of clarity. The resulting code is somewhat more complicated than if the height were stored at each node. If a recursive routine is written, then speed is probably not the main consideration. In this case, the slight speed advantage obtained by storing balance factors hardly seems worth the loss of clarity and relative simplicity. Furthermore, since most machines will align this to at least an 8-bit boundary anyway, there is not likely to be any difference in the amount of space used. Eight bits will allow us to store absolute heights of up to 255. Since the tree is balanced, it is inconceivable that this would be insufficient (see the exercises).

With all this, we are ready to write the AVL routines. We will do only a partial job and leave the rest as an exercise. First, we need the declarations. These are given in Figure 4.35. We also need a quick function to return the height of a node. This function is necessary to handle the annoying case of a NULL pointer. This is shown in Figure 4.36. The basic insertion routine is easy to write, since it consists mostly of function calls (see Fig. 4.37).

typedef struct avl_node *avl_ptr;

struct avl_node

{

element_type element;

avl_ptr left;

avl_ptr right;

int height;

};

typedef avl_ptr SEARCH_TREE;

Figure 4.35 Node declaration for AVL trees

int

height( avl_ptr p )

{

if( p == NULL )

return -1;

else

return p->height;

}

Figure 4.36 Function to compute height of an AVL node

For the trees in Figure 4.38, s_rotate_left converts the tree on the left to the tree on the right, returning a pointer to the new root. s_rotate_right is symmetric. The code is shown in Figure 4.39.

The last function we will write will perform the double rotation pictured in Figure 4.40, for which the code is shown in Figure 4.41.

Deletion in AVL trees is somewhat more complicated than insertion. Lazy deletion is probably the best strategy if deletions are relatively infrequent.

4.5. Splay Trees

We now describe a relatively simple data structure, known as a splay tree, that guarantees that any m consecutive tree operations take at most O(m log n) time. Although this guarantee does not preclude the possibility that any single operation might take O(n) time, and thus the bound is not as strong as an O(log n) worst-case bound per operation, the net effect is the same: There are no bad input sequences. Generally, when a sequence of m operations has total worst-case running time of O(m f(n)), we say that the amortized running time is O(f(n)). Thus, a splay tree has O(log n) amortized cost per operation. Over a long sequence of operations, some may take more, some less.

Splay trees are based on the fact that the O(n) worst-case time per operation for binary search trees is not bad, as long at it occurs relatively infrequently. Any one access, even if it takes O(n), is still likely to be extremely fast. The problem with binary search trees is that it is possible, and not uncommon, for a whole sequence of bad accesses to take place. The cumulative running time then becomes noticeable. A search tree data structure with O(n) worst-case time, but a guarantee of at most O(m log n) for any m consecutive operations, is certainly satisfactory, because there are no bad sequences.

If any particular operation is allowed to have an O(n) worst-case time bound, and we still want an O(log n) amortized time bound, then it is clear that whenever a node is accessed, it must be moved. Otherwise, once we find a deep node, we could keep performing finds on it. If the node does not change location, and each access costs O(n), then a sequence of m accesses will cost O(m   [image]  n).

SEARCH_TREE

insert( element_type x, SEARCH_TREE T )

{

return insert1( x, T, NULL );

}

SEARCH_TREE

insert1( element_type x, SEARCH_TREE T, avl_ptr parent )

{

avl_ptr rotated_tree;

if( T == NULL )

{  /* Create and return a one-node tree */

T = (SEARCH_TREE) malloc ( sizeof (struct avl_node) );

if( T == NULL )

fatal_error("Out of space!!!");

else

{

T->element = x; T->height = 0;

T->left = T->right = NULL;

}

}

else

{

if( x < T->element )

{

T->left = insert1( x, T->left, T );

if( ( height( T->left ) - height( T->right ) ) == 2

{

if( x < T->left->element )

rotated_tree = s_rotate_left( T );

else

rotated_tree = d_rotate_left( T );

if( parent->left == T )

parent->left = rotated_tree;

else

parent->right = rotated_tree;

}

else

T->height = max( height(T->left), height(T->right) ) + 1;

}

else

/* Symmetric Case for right subtree */;

/* Else x is in the tree already. We'll do nothing */

}

return T;

}

Figure 4.37 Insertion into an AVL tree

Figure 4.38

/* This function can be called only if k2 has a left child. */

/* Perform a rotate between a node (k2) and its left child. */

/* Update heights. */

/* Then return new root. */

avl_ptr

s_rotate_left( avl_ptr k2 )

{

avl_ptr k1;

k1 = k2->left;

k2->left = k1->right;

k1->right = k2;

k2->height = max( height(k2->left), height(k2->right) ) + 1;

k1->height = max( height(k1->left), k2->height ) + 1;

return k1;  /* New root */

}

Figure 4.39 Routine to perform single rotation

Figure 4.40

/* This function can be called only if k3 has a left child */

/* and k3's left child has a right child */

/* Do the left-right double rotation. Update heights */

avl_ptr

d_rotate_left( avl_ptr k3 )

{

/* rotate between k1 and k2 */

k3->left = s_rotate_right( k3->left );

/* rotate between k3 and k2 */

return( s_rotate_left( k3 ) );

}

Figure 4.41 Routine to perform double rotation

The basic idea of the splay tree is that after a node is accessed, it is pushed to the root by a series of AVL tree rotations. Notice that if a node is deep, there are many nodes on the path that are also relatively deep, and by restructuring we can make future accesses cheaper on all these nodes. Thus, if the node is unduly deep, then we want this restructuring to have the side effect of balancing the tree (to some extent). Besides giving a good time bound in theory, this method is likely to have practical utility, because in many applications when a node is accessed, it is likely to be accessed again in the near future. Studies have shown that this happens much more often than one would expect. Splay trees also do not require the maintenance of height or balance information, thus saving space and simplifying the code to some extent (especially when careful implementations are written).

4.5.1. A Simple Idea (That Does Not Work)

4.5.2. Splaying

4.5.1. A Simple Idea (That Does Not Work)

One way of performing the restructuring described above is to perform single rotations, bottom up. This means that we rotate every node on the access path with its parent. As an example, consider what happens after an access (a find) on k1 in the following tree.

The access path is dashed. First, we would perform a single rotation between k1 and its parent, obtaining the following tree.

Then, we rotate between k1 and k3, obtaining the next tree.

Then two more rotations are performed until we reach the root.

These rotations have the effect of pushing k1 all the way to the root, so that future accesses on k1 are easy (for a while). Unfortunately, it has pushed another node (k3) almost as deep as k1 used to be. An access on that node will then push another node deep, and so on. Although this strategy makes future accesses of k1 cheaper, it has not significantly improved the situation for the other nodes on the (original) access path. It turns out that it is possible to prove that using this strategy, there is a sequence of m operations requiring   [image] (m   [image]  n) time, so this idea is not quite good enough. The simplest way to show this is to consider the tree formed by inserting keys 1, 2, 3, . . . , n into an initially empty tree (work this example out). This gives a tree consisting of only left children. This is not necessarily bad, though, since the time to build this tree is O(n) total. The bad part is that accessing the node with key 1 takes n -1 units of time. After the rotations are complete, an access of the node with key 2 takes n - 2 units of time. The total for accessing all the keys in order is   [image]. After they are accessed, the tree reverts to its original state, and we can repeat the sequence.

4.5.2. Splaying

The splaying strategy is similar to the rotation idea above, except that we are a little more selective about how rotations are performed. We will still rotate bottom up along the access path. Let x be a (nonroot) node on the access path at which we are rotating. If the parent of x is the root of the tree, we merely rotate x and the root. This is the last rotation along the access path. Otherwise, x has both a parent (p) and a grandparent (g), and there are two cases, plus symmetries, to consider. The first case is the zig-zag case (see Fig. 4.42). Here x is a right child and p is a left child (or vice versa). If this is the case, we perform a double rotation, exactly like an AVL double rotation. Otherwise, we have a zig-zig case: x and p are either both left children or both right children. In that case, we transform the tree on the left of Figure 4.43 to the tree on the right.

Figure 4.42 Zig-zag

Figure 4.43 Zig-zig

As an example, consider the tree from the last example, with a find on k1:

The first splay step is at k1, and is clearly a zig-zag, so we perform a standard AVL double rotation using k1, k2, and k3. The resulting tree follows.

The next splay step at k1 is a zig-zig, so we do the zig-zig rotation with k1, k4, and k5, obtaining the final tree.

Although it is hard to see from small examples, splaying not only moves the accessed node to the root, but also has the effect of roughly halving the depth of most nodes on the access path (some shallow nodes are pushed down at most two levels).

Figure 4.44 Result of splaying at node 1

To see the difference that splaying makes over simple rotation, consider again the effect of inserting keys 1, 2, 3, . . . , n into an initially empty tree. This takes a total of O(n), as before, and yields the same tree as simple rotations. Figure 4.44 shows the result of splaying at the node with key 1. The difference is that after an access of the node with key 1, which takes n -1 units, the access on the node with key 2 will only take about n/2 units instead of n - 2 units; there are no nodes quite as deep as before.

Figure 4.45 Result of splaying at node 1 a tree of all left children

Figure 4.46 Result of splaying previous tree at node 2

An access on the node with key 2 will bring nodes to within n/4 of the root, and this is repeated until the depth becomes roughly log n (an example with n = 7 is too small to see the effect well). Figures 4.45 to 4.53 show the result of accessing keys 1 through 9 in a 32-node tree that originally contains only left children. Thus we do not get the same bad behavior from splay trees that is prevalent in the simple rotation strategy. (Actually, this turns out to be a very good case. A rather complicated proof shows that for this example, the n accesses take a total of O(n) time).

These figures show off the fundamental and crucial property of splay trees. When access paths are long, thus leading to a longer-than-normal search time, the rotations tend to be good for future operations. When accesses are cheap, the rotations are not as good and can be bad. The extreme case is the initial tree formed by the insertions. All the insertions were constant-time operations leading to a bad initial tree. At that point in time, we had a very bad tree, but we were running ahead of schedule and had the compensation of less total running time. Then a couple of really horrible accesses left a nearly balanced tree, but the cost was that we had to give back some of the time that had been saved. The main theorem, which we will prove in Chapter 11, is that we never fall behind a pace of O(log n) per operation: We are always on schedule, even though there are occasionally bad operations.

Because the rotations for splay trees are performed in pairs from the bottom up, a recursive implementation does not work, (although modifications to the splaying steps can be made to allow a recursive implementation). The pairs of nodes to consider are not known until the length of the path is determined to be even or odd. Thus, splay trees are coded nonrecursively and work in two passes. The first pass goes down the tree and the second goes back up, performing rotations. This requires that the path be saved. This can be done by using a stack (which might need to store n pointers) or by adding an extra field to the node record that will point to the parent. Neither method is particularly difficult to implement. We will provide code for the splaying routine on the assumption that each node stores its parent.

Figure 4.47 Result of splaying previous tree at node 3

Figure 4.48 Result of splaying previous tree at node 4

Figure 4.49 Result of splaying previous tree at node 5

Figure 4.50 Result of splaying previous tree at node 6

Figure 4.51 Result of splaying previous tree at node 7

Figure 4.52 Result of splaying previous tree at node 8

Figure 4.53 Result of splaying previous tree at node 9

The type declarations (Fig. 4.54) are simple to understand. The splaying routine (Fig. 4.55) takes as argument the last node on the accessed path and makes it the new root. The routines single_rotate and double_rotate choose the correct type of rotation. We provide the code for single_rotate in Figure 4.56.

The rotation routines are similar to the AVL rotations, except that the parent pointers must be maintained. Some sample routines are in the figures that follow. Since zig rotations always make x the new root, we know that x will have no parent after the operation. The code for this is in Figure 4.57.

Zig-zigs and Zig-zags are similar. We will write the one routine to perform the zig-zig splay when both x and p are left children. One way to do this is to write a single_rotate routine that includes pointer changes for the parent, and then implement the complex rotations with two single rotations. This is the way we coded the AVL routines. We have taken a different approach in Figure 4.58 to show the diversity of styles available. See Figure 4.59. You should try to code the other cases yourself; it will be excellent pointer manipulation practice.

typedef struct splay_node *splay_ptr;

struct splay_node

{

element_type element;

splay_ptr left;

splay-ptr right;

splay-ptr parent;

};

typedef splay_ptr SEARCH_TREE;

Figure 4.54 Type declarations for splay trees

void

splay( splay_ptr current )

{

splay_ptr father;

father = current->parent;

while( father != NULL )

{

if( father->parent == NULL )

single_rotate (current );

else

double_rotate( current );

father = current->parent;

}

}

Figure 4.55 Basic splay routine

void

single_rotate( splay_ptr x )

{

if( x->parent->left == x)

zig_left( x );

else

zig_right( x );

}

Figure 4.56 Single rotation

void

zig_left( splay_ptr x )

{

splay ptr p, B;

p = x->parent;

B = x->right;

x->right = p;      /* x's new right child is p*/

x->parent = NULL;  /* x will now be a root */

if( B != NULL )

B->parent = p;

p->left = B;

p->parent = x;

}

Figure 4.57 Single rotation between root and its left child

We can perform deletion by accessing the node to be deleted. This puts the node at the root. If it is deleted, we get two subtrees TL and TR (left and right). If we find the largest element in TL (which is easy), then this element is rotated to the root of TL, and TL will now have a root with no right child. We can finish the deletion by making TR the right child.

The analysis of splay trees is difficult, because it must take into account the ever-changing structure of the tree. On the other hand, splay trees are much simpler to program than AVL trees, since there are fewer cases to consider and no balance information to maintain. Our splay tree code may look complicated, but as pointed out before, it can be simplified; it is probably much simpler than a nonrecursive AVL implementation. Some empirical evidence suggests that this translates into faster code in practice, although the case for this is far from complete. Finally, we point out that there are several variations of splay trees that can perform even better in practice.

Figure 4.58

void

zig_zig_left( splay_ptr x )

{

splay_ptr p, g, B, C, ggp;

p = x->parent;

g = p->parent;

B = x->right;

C = p->right;

ggp = g->parent;

x->right = p;           /* x's new right child is p*/

p->parent = x;

p->right = g;           /* p's new right child is g */

g->parent = p;

if( B != NULL )         /* p's new left child is subtree B */

B->parent = p;

p->left = B;

if( C != NULL )         /* g's new left child is subtree C */

C->parent = g;

g->left = C;

x->parent = ggp;        /* connect to rest of the tree */

if( ggp ! = NULL )

if( gpp->left == g )

ggp->left = x;

else

ggp->right = x;

}

Figure 4.59 Routine to perform a zig-zig when both children are initially left children

4.6. Tree Traversals (Revisited)

Because of the ordering information in a binary search tree, it is simple to list all the keys in sorted order. The recursive procedure in Figure 4.60 does this.

Convince yourself that this procedure works. As we have seen before, this kind of routine when applied to trees is known as an inorder traversal (which makes sense, since it lists the keys in order). The general strategy of an inorder traversal is to process the left subtree first, then perform processing at the current node, and finally process the right subtree. The interesting part about this algorithm, aside from its simplicity, is that the total running time is O(n). This is because there is constant work being performed at every node in the tree. Each node is visited once, and the work performed at each node is testing against NULL, setting up two procedure calls, and doing a print_element. Since there is constant work per node and n nodes, the running time is O(n).

void

print_tree( SEARCH_TREE T )

{

if( T != NULL )

{

print_tree( T->left );

print_element( T->element );

print_tree( T->right );

}

}

Figure 4.60 Routine to print a binary search tree in order

Sometimes we need to process both subtrees first before we can process a node. For instance, to compute the height of a node, we need to know the height of the subtrees first. The code in Figure 4.61 computes this. Since it is always a good idea to check the special cases - and crucial when recursion is involved - notice that the routine will declare the height of a leaf to be zero, which is correct. This general order of traversal, which we have also seen before, is known as a postorder traversal. Again, the total running time is O(n), because constant work is performed at each node.

The third popular traversal scheme that we have seen is preorder traversal. Here, the node is processed before the children. This could be useful, for example, if you wanted to label each node with its depth.

The common idea in all of these routines is that you handle the NULL case first, and then the rest. Notice the lack of extraneous variables. These routines pass only the tree, and do not declare or pass any extra variables. The more compact the code, the less likely that a silly bug will turn up. A fourth, less often used, traversal (which we have not seen yet) is level-order traversal. In a level-order traveresal, all nodes at depth d are processed before any node at depth d + 1. Level-order traversal differs from the other traversals in that it is not done recursively; a queue is used, instead of the implied stack of recursion.

int

height( TREE T )

{

if( T == NULL )

return -1;

else

return ( max( height(T->left), height(T->right) ) + 1 );

}

Figure 4.61 Routine to compute the height of a tree using a postorder traversal

4.7. B-Trees

Although all of the search trees we have seen so far are binary, there is a popular search tree that is not binary. This tree is known as a B-tree.

A B-tree of order m is a tree with the following structural properties:

       The root is either a leaf or has between 2 and m children.

       All nonleaf nodes (except the root) have between   [image] m/2  [image]  and m children.

       All leaves are at the same depth.

All data is stored at the leaves. Contained in each interior node are pointers p1, p2, . . . , pm to the children, and values k1, k2, . . . , km - 1, representing the smallest key found in the subtrees p2, p3, . . . , pm respectively. Of course, some of these pointers might be NULL, and the corresponding ki would then be undefined. For every node, all the keys in subtree p1 are smaller than the keys in subtree p2, and so on. The leaves contain all the actual data, which is either the keys themselves or pointers to records containing the keys. We will assume the former to keep our examples simple. There are various definitions of B-trees that change this structure in mostly minor ways, but this definition is one of the popular forms. We will also insist (for now) that the number of keys in a leaf is also between   [image] m/2  [image]  and m.

The tree in Figure 4.62 is an example of a B-tree of order 4.

Figure 4.62 B-tree of order 4

A B-tree of order 4 is more popularly known as a 2-3-4 tree, and a B-tree of order 3 is known as a 2-3 tree. We will describe the operation of B-trees by using the special case of 2-3 trees. Our starting point is the 2-3 tree that follows.

We have drawn interior nodes (nonleaves) in ellipses, which contain the two pieces of data for each node. A dash line as a second piece of information in an interior node indicates that the node has only two children. Leaves are drawn in boxes, which contain the keys. The keys in the leaves are ordered. To perform a find, we start at the root and branch in one of (at most) three directions, depending on the relation of the key we are looking for to the two (possibly one) values stored at the node.

To perform an insert on a previously unseen key, x, we follow the path as though we were performing a find. When we get to a leaf node, we have found the correct place to put x. Thus, to insert a node with key 18, we can just add it to a leaf without causing any violations of the 2-3 tree properties. The result is shown in the following figure.

Unfortunately, since a leaf can hold only two or three keys, this might not always be possible. If we now try to insert 1 into the tree, we find that the node where it belongs is already full. Placing our new key into this node would give it a fourth element which is not allowed. This can be solved by making two nodes of two keys each and adjusting the information in the parent.

Unfortunately, this idea does not always work, as can be seen by an attempt to insert 19 into the current tree. If we make two nodes of two keys each, we obtain the following tree.

This tree has an internal node with four children, but we only allow three per node. The solution is simple. We merely split this node into two nodes with two children. Of course, this node might be one of three children itself, and thus splitting it would create a problem for its parent (which would now have four children), but we can keep on splitting nodes on the way up to the root until we either get to the root or find a node with only two children. In our case, we can get by with splitting only the first internal node we see, obtaining the following tree.

If we now insert an element with key 28, we create a leaf with four children, which is split into two leaves of two children:

This creates an internal node with four children, which is then split into two children. What we have done here is split the root into two nodes. When we do this, we have a special case, which we finish by creating a new root. This is how (the only way) a 2-3 tree gains height.

Notice also that when a key is inserted, the only changes to internal nodes occur on the access path. These changes can be made in time proportional to the length of this path, but be forewarned that there are quite a few cases to handle, and it is easy to do this wrong.

There are other ways to handle the case where a node becomes overloaded with children, but the method we have described is probably the simplest. When attempting to add a fourth key to a leaf, instead of splitting the node into two we can first attempt to find a sibling with only two keys. For instance, to insert 70 into the tree above, we could move 58 to the leaf containing 41 and 52, place 70 with 59 and 61, and adjust the entries in the internal nodes. This strategy can also be applied to internal nodes and tends to keep more nodes full. The cost of this is slightly more complicated routines, but less space tends to be wasted.

We can perform deletion by finding the key to be deleted and removing it. If this key was one of only two keys in a node, then its removal leaves only one key. We can fix this by combining this node with a sibling. If the sibling has three keys, we can steal one and have both nodes with two keys. If the sibling has only two keys, we combine the two nodes into a single node with three keys. The parent of this node now loses a child, so we might have to percolate this strategy all the way to the top. If the root loses its second child, then the root is also deleted and the tree becomes one level shallower. As we combine nodes, we must remember to update the information kept at the internal nodes.

With general B-trees of order m, when a key is inserted, the only difficulty arises when the node that is to accept the key already has m keys. This key gives the node m + 1 keys, which we can split into two nodes with   [image]  (m + 1) / 2   [image]  and   [image]  (m + 1) / 2   [image]  keys respectively. As this gives the parent an extra node, we have to check whether this node can be accepted by the parent and split the parent if it already has m children. We repeat this until we find a parent with less than m children. If we split the root, we create a new root with two children.

The depth of a B-tree is at most   [image] log  [image] m/2  [image] n  [image]. At each node on the path, we perform O(log m) work to determine which branch to take (using a binary search), but an insert or delete could require O(m) work to fix up all the information at the node. The worst-case running time for each of the insert and delete operations is thus O(m logm n) = O( (m / log m ) log n), but a find takes only O(log n ). The best (legal) choice of m for running time considerations has been shown empirically to be either m = 3 or m = 4; this agrees with the bounds above, which show that as m gets larger, the insertion and deletion times increase. If we are only concerned with main memory speed, higher order B-trees, such as 5-9 trees, are not an advantage.

The real use of B-trees lies in database systems, where the tree is kept on a physical disk instead of main memory. Accessing a disk is typically several orders of magnitude slower than any main memory operation. If we use a B-tree of order m, then the number of disk accesses is O(logm n). Although each disk access carries the overhead of O(log m) to determine the direction to branch, the time to perform this computation is typically much smaller than the time to read a block of memory and can thus be considered inconsequential (as long as m is chosen reasonably). Even if updates are performed and O(m) computing time is required at each node, this too is generally not significant. The value of m is then chosen to be the largest value that still allows an interior node to fit into one disk block, and is typically in the range 32   [image]  m   [image]  256. The maximum number of elements that are stored in a leaf is chosen so that if the leaf is full, it fits in one block. This means that a record can always be found in very few disk accesses, since a typical B-tree will have a depth of only 2 or 3, and the root (and possibly the first level) can be kept in main memory.

Analysis suggests that a B-tree will be ln 2 = 69 percent full. Better space utilization can be obtained if, instead of always splitting a node when the tree obtains its (m + 1)th entry, the routine searches for a sibling that can take the extra child. The details can be found in the references.

Summary

We have seen uses of trees in operating systems, compiler design, and searching. Expression trees are a small example of a more general structure known as a parse tree, which is a central data structure in compiler design. Parse trees are not binary, but are relatively simple extensions of expression trees (although the algorithms to build them are not quite so simple).

Search trees are of great importance in algorithm design. They support almost all the useful operations, and the logarithmic average cost is very small. Nonrecursive implementations of search trees are somewhat faster, but the recursive versions are sleeker, more elegant, and easier to understand and debug. The problem with search trees is that their performance depends heavily on the input being random. If this is not the case, the running time increases significantly, to the point where search trees become expensive linked lists.

We saw several ways to deal with this problem. AVL trees work by insisting that all nodes' left and right subtrees differ in heights by at most one. This ensures that the tree cannot get too deep. The operations that do not change the tree, as insertion does, can all use the standard binary search tree code. Operations that change the tree must restore the tree. This can be somewhat complicated, especially in the case of deletion. We showed how to restore the tree after insertions in O(log n) time.

We also examined the splay tree. Nodes in splay trees can get arbitrarily deep, but after every access the tree is adjusted in a somewhat mysterious manner. The net effect is that any sequence of m operations takes O(m log n) time, which is the same as a balanced tree would take.

B-trees are balanced m-way (as opposed to 2-way or binary) trees, which are well suited for disks; a special case is the 2-3 tree, which is another common method of implementing balanced search trees.

In practice, the running time of all the balanced tree schemes is worse (by a constant factor) than the simple binary search tree, but this is generally acceptable in view of the protection being given against easily obtained worst-case input.

A final note: By inserting elements into a search tree and then performing an inorder traversal, we obtain the elements in sorted order. This gives an O(n log n) algorithm to sort, which is a worst-case bound if any sophisticated search tree is used. We shall see better ways in Chapter 7, but none that have a lower time bound.

Exercises

Questions 4.1 to 4.3 refer to the tree in Figure 4.63.

4.1     For the tree in Figure 4.63 :

a.     Which node is the root?

b.     Which nodes are leaves?

Figure 4.63

4.2     For each node in the tree of Figure 4.63 :

a.     Name the parent node.

b.     List the children.

c.     List the siblings.

d.     Compute the depth.

e.     Compute the height.

4.3     What is the depth of the tree in Figure 4.63?

4.4     Show that in a binary tree of n nodes, there are n + 1  pointers representing children.

4.5     Show that the maximum number of nodes in a binary tree of height h is 2h+1 - 1.

4.6     A full node is a node with two children. Prove that the number of full nodes plus one is equal to the number of leaves in a binary tree.

4.7     Suppose a binary tree has leaves l1, l2, . . . , lm at depth d1, d2, . . . , dm, respectively. Prove that   [image]  and determine when the equality is true.

4.8     Give the prefix, infix, and postfix expressions corresponding to the tree in Figure 4.64.

4.9     a.     Show the result of inserting 3, 1, 4, 6, 9, 2, 5, 7 into an initially empty binary search tree.

b.     Show the result of deleting the root.

4.10     Write routines to implement the basic binary search tree operations.

4.11     Binary search trees can be implemented with cursors, using a strategy similar to a cursor linked list implementation. Write the basic binary search tree routines using a cursor implementation.

4.12     Suppose you want to perform an experiment to verify the problems that can be caused by random insert/delete pairs. Here is a strategy that is not perfectlyrandom, but close enough. You build a tree with n elements by inserting n elements chosen at random from the range 1 to m =   [image] n. You then perform n2 pairs of insertions followed by deletions. Assume the existence of a routine, rand_int(a,b), which returns a uniform random integer between a and b inclusive.

Figure 4.64 Tree for Exercise 4.8

a.     Explain how to generate a random integer between 1 and m that is not already in the tree (so a random insert can be performed). In terms of n and   [image], what is the running time of this operation?

b.     Explain how to generate a random integer between 1 and m that is already  in the tree (so a random delete can be performed). What is the running time of this operation?

c.     What is a good choice of   [image]? Why?

4.13     Write a program to evaluate empirically the following strategies for deleting nodes with two children:

a.     Replace with the largest node, X, in TL and recursively delete X.

b.     Alternately replace with the largest node in TL and the smallest node in TR, and recursively delete appropriate node.

c.     Replace with either the largest node in TL or the smallest node in TR (recursively deleting the appropriate node), making the choice randomly. Which strategy seems to give the most balance? Which takes the least CPU time to process the entire sequence?

4.14     ** Prove that the depth of a random binary search tree (depth of the deepest node) is O(log n), on average.

4.15     *a.     Give a precise expression for the minimum number of nodes in an AVL tree of height h.

b.     What is the minimum number of nodes in an AVL tree of height 15?

4.16     Show the result of inserting 2, 1, 4, 5, 9, 3, 6, 7 into an initially empty AVL tree.

4.17     * Keys 1, 2, . . . , 2k -1 are inserted in order into an initially empty AVL tree. Prove that the resulting tree is perfectly balanced.

4.18     Write the remaining procedures to implement AVL single and double rotations.

4.19     Write a nonrecursive function to insert into an AVL tree.

4.20     * How can you implement (nonlazy) deletion in AVL trees?

4.21     a.     How many bits are required per node to store the height of a node in an n-node AVL tree?

b.     What is the smallest AVL tree that overflows an 8-bit height counter?

4.22     Write the functions to perform the double rotation without the inefficiency of doing two single rotations.

4.23     Show the result of accessing the keys 3, 9, 1, 5 in order in the splay tree in Figure 4.65.

Figure 4.65

4.24     Show the result of deleting the element with key 6 in the resulting splay tree for the previous exercise.

4.25     Nodes 1 through n = 1024 form a splay tree of left children.

a.     What is the internal path length of the tree (exactly)?

*b.     Calculate the internal path length after each of find(1), find(2), find(3), find(4), find(5), find(6).

*c.     If the sequence of successive finds is continued, when is the internal path length minimized?

4.26     a.     Show that if all nodes in a splay tree are accessed in sequential order, the resulting tree consists of a chain of left children.

**b. Show that if all nodes in a splay tree are accessed in sequential order, then the total access time is O(n), regardless of the initial tree.

4.27     Write a program to perform random operations on splay trees. Count the total number of rotations performed over the sequence. How does the running time compare to AVL trees and unbalanced binary search trees?

4.28     Write efficient functions that take only a pointer to a binary tree, T, and compute

a.     the number of nodes in T

b.     the number of leaves in T

c.     the number of full nodes in T

What is the running time of your routines?

4.29     Write a function to generate an n-node random binary search tree with distinct keys 1 through n. What is the running time of your routine?

4.30     Write a function to generate the AVL tree of height h with fewest nodes. What is the running time of your function?

4.31     Write a function to generate a perfectly balanced binary search tree of height h with keys 1 through 2h+1 - 1. What is the running time of your function?

4.32     Write a function that takes as input a binary search tree, T, and two keys k1 and k2, which are ordered so that k1   [image]  k2, and prints all elements x in the tree such that k1   [image]  key(x)   [image]  k2. Do not assume any information about the type of keys except that they can be ordered (consistently). Your program should run in O(K + log n) average time, where K is the number of keys printed. Bound the running time of your algorithm.

4.33     The larger binary trees in this chapter were generated automatically by a program. This was done by assigning an (x, y) coordinate to each tree node, drawing a circle around each coordinate (this is hard to see in some pictures), and connecting each node to its parent. Assume you have a binary search tree stored in memory (perhaps generated by one of the routines above) and that each node has two extra fields to store the coordinates.

a.     The x coordinate can be computed by assigning the inorder traversal number. Write a routine to do this for each node in the tree.

b.     The y coordinate can be computed by using the negative of the depth of the node. Write a routine to do this for each node in the tree.

c.     In terms of some imaginary unit, what will the dimensions of the picture be? How can you adjust the units so that the tree is always roughly two-thirds as high as it is wide?

d.     Prove that using this system no lines cross, and that for any node, X, all elements in X's left subtree appear to the left of X and all elements in X's right subtree appear to the right of X.

4.34     Write a general-purpose tree-drawing program that will convert a tree into the following graph-assembler instructions:

a.     circle(x, y)

b.     drawline(i, j)

The first instruction draws a circle at (x, y), and the second instruction connects the ith circle to the jth circle (circles are numbered in the order drawn). You should either make this a program and define some sort of input language or make this a function that can be called from any program. What is the running time of your routine?

4.35     Write a routine to list out the nodes of a binary tree in level-order. List the root, then nodes at depth 1, followed by nodes at depth 2, and so on. You must do this in linear time. Prove your time bound.

4.36     a.     Show the result of inserting the following keys into an initially empty 2-3 tree: 3, 1, 4, 5, 9, 2, 6, 8, 7, 0.

b.     Show the result of deleting 0 and then 9 from the 2-3 tree created in part (a).

4.37      *a.     Write a routine to perform insertion from a B-tree.

*b.     Write a routine to perform deletion from a B-tree. When a key is deleted, is it necessary to update   information in the internal nodes?

Figure 4.66 Tree for Exercise 4.39

*c.     Modify your insertion routine so that if an attempt is made to add into a node that already has m entries, a search is performed for a sibling with less than m children before the node is split.

4.38     A B*-tree of order m is a B-tree in which each each interior node has between 2m/3 and m children. Describe a method to perform insertion into a B*-tree.

4.39     Show how the tree in Figure 4.66 is represented using a child/sibling pointer implementation.

4.40     Write a procedure to traverse a tree stored with child/sibling links.

4.41     Two binary trees are similar if they are both empty or both nonempty and have similar left and right subtrees. Write a function to decide whether two binary trees are similar. What is the running time of your program?

4.42     Two trees, T1 and T2, are isomorphic if T1 can be transformed into T2 by swapping left and right children of (some of the) nodes in T1. For instance, the two trees in Figure 4.67 are isomorphic because they are the same if the children of A, B, and G, but not the other nodes, are swapped.

a.     Give a polynomial time algorithm to decide if two trees are isomorphic.

*b.     What is the running time of your program (there is a linear solution)?

4.43     *a.     Show that via AVL single rotations, any binary search tree T1 can be transformed into another search tree T2 (with the same keys).

*b.     Give an algorithm to perform this transformation using O(n log n) rotations on average.

**c. Show that this transformation can be done with O(n) rotations, worst-case.

Figure 4.67 Two isomorphic trees

4.44     Suppose we want to add the operation find_kth to our repertoire. The  operation find_kth(T,i) returns the element in tree T with ith smallest key. Assume  all elements have distinct keys. Explain how to modify the binary search tree to support this operation in O(log n) average time, without sacrificing the  time bounds of any other operation.

4.45     Since a binary search tree with n nodes has n + 1  pointers, half the  space allocated in a binary search tree for pointer information is wasted.  Suppose that if a node has a  left child, we make its left child point  to its inorder predecessor, and if a node has a  right child, we make its right child point to its inorder successor. This is known as a threaded tree and the extra pointers are called threads.

a.     How can we distinguish threads from real children pointers?

b.     Write routines to perform insertion and deletion into a tree threaded in the manner described above.

c.     What is the advantage of using threaded trees?

4.46     A binary search tree presupposes that searching is based on only one key per record. Suppose we would like to be able to perform searching based on either of two keys, key1 or key2.

a.     One method is to build two separate binary search trees. How many extra pointers does this require?

b.     An alternative method is a 2-d tree. A 2-d tree is similar to a binary search tree, except that branching at even levels is done with respect to key1, and branching at odd levels is done with key2. Figure 4.68 shows a 2-d tree, with the first and last names as keys, for post-WWII presidents. The presidents' names were inserted chronologically (Truman, Eisenhower, Kennedy, Johnson, Nixon, Ford, Carter, Reagan, Bush). Write a routine to perform insertion into a 2-d tree.

c.     Write an efficient procedure that prints all records in the tree that simultaneously satisfy the constraints low1   [image]  key1   [image]  high1 and low2   [image]  key2   [image]  high2.

d.     Show how to extend the 2-d tree to handle more than two search keys. The resulting strategy is known as a k-d tree.

Figure 4.68 A 2-d tree

References

More information on binary search trees, and in particular the mathematical properties of trees can be found in the two books by Knuth [23] and [24].

Several papers deal with the lack of balance caused by biased deletion algorithms in binary search trees. Hibbard's paper [20] proposed the original deletion algorithm and established that one deletion preserves the randomness of the trees. A complete analysis has been performed only for trees with three [21] and four nodes[5]. Eppinger's paper [15] provided early empirical evidence of nonrandomness, and the papers by Culberson and Munro, [11], [12], provide some analytical evidence (but not a complete proof for the general case of intermixed insertions and deletions).

AVL trees were proposed by Adelson-Velskii and Landis [1]. Simulation results for AVL trees, and variants in which the height imbalance is allowed to be at most k for various values of k, are presented in [22]. A deletion algorithm for AVL trees can be found in [24]. Analysis of the averaged depth of AVL trees is incomplete, but some results are contained in [25].

[3] and [9] considered self-adjusting trees like the type in Section 4.5.1. Splay trees are described in [29].

B-trees first appeared in [6]. The implementation described in the original paper allows data to be stored in internal nodes as well as leaves. The data structure we have described is sometimes known as a B+ tree. A survey of the different types of B-trees is presented in [10]. Empirical results of the various schemes is reported in [18]. Analysis of 2-3 trees and B-trees can be found in [4], [14], and [33].

Exercise 4.14 is deceptively difficult. A solution can be found in [16]. Exercise 4.26 is from [32]. Information on B*-trees, described in Exercise 4.38, can be found in [13]. Exercise 4.42 is from [2]. A solution to Exercise 4.43 using 2n -6 rotations is given in [30]. Using threads, a la Exercise 4.45, was first proposed in [28]. k-d trees were first proposed in [7]. Their major drawback is that both deletion and balancing are difficult. [8] discusses k-d trees and other methods used for multidimensional searching.

Other popular balanced search trees are red-black trees [19] and weight-balanced trees [27]. More balanced tree schemes can be found in the books [17], [26], and [31].

1.     G. M. Adelson-Velskii and E. M. Landis, "An Algorithm for the Organization of Information," Soviet Math. Doklady 3 (1962), 1259-1263.

2.     A. V. Aho, J. E. Hopcroft, and J. D. Ullman, The Design and Analysis of Computer Algorithms, Addison-Wesley, Reading, MA, 1974.

3.     B. Allen and J. I. Munro, "Self Organizing Search Trees," Journal of the ACM, 25 (1978), 526-535.

4.     R. A. Baeza-Yates, "Expected Behaviour of B+- trees under Random Insertions," Acta Informatica 26 (1989), 439-471.

5.     R. A. Baeza-Yates, "A Trivial Algorithm Whose Analysis Isn't: A Continuation," BIT 29 (1989), 88-113.

6.     R. Bayer and E. M. McGreight, "Organization and Maintenance of Large Ordered Indices," Acta Informatica 1 (1972), 173-189.

7.     J. L. Bentley, "Multidimensional Binary Search Trees Used for Associative Searching," Communications of the ACM 18 (1975), 509-517.

8.     J. L. Bentley and J. H. Friedman, "Data Structures for Range Searching," Computing Surveys 11 (1979), 397-409.

9.     J. R. Bitner, "Heuristics that Dynamically Organize Data Structures," SIAM Journal on Computing 8 (1979), 82-110.

10.     D. Comer, "The Ubiquitous B-tree," Computing Surveys 11 (1979), 121-137.

11.     J. Culberson and J. I. Munro, "Explaining the Behavior of Binary Search Trees under Prolonged Updates: A Model and Simulations," Computer Journal 32 (1989), 68-75.

12.     J. Culberson and J. I. Munro, "Analysis of the Standard Deletion Algorithms' in Exact Fit Domain Binary Search Trees," Algorithmica 5 (1990) 295-311.

13.     K. Culik, T. Ottman, and D. Wood, "Dense Multiway Trees," ACM Transactions on Database Systems 6 (1981), 486-512.

14.     B. Eisenbath, N. Ziviana, G. H. Gonnet, K. Melhorn, and D. Wood, "The Theory of Fringe Analysis and its Application to 2-3 Trees and B-trees," Information and Control 55 (1982), 125-174.

15.     J. L. Eppinger, "An Empirical Study of Insertion and Deletion in Binary Search Trees," Communications of the ACM 26 (1983), 663-669.

16.     P. Flajolet and A. Odlyzko, "The Average Height of Binary Trees and Other Simple Trees," Journal of Computer and System Sciences 25 (1982), 171-213.

17.     G. H. Gonnet and R. Baeza-Yates, Handbook of Algorithms and Data Structures, second edition, Addison-Wesley, Reading, MA, 1991.

18.     E. Gudes and S. Tsur, "Experiments with B-tree Reorganization," Proceedings of ACM SIGMOD Symposium on Management of Data (1980), 200-206.

19.     L. J. Guibas and R. Sedgewick, "A Dichromatic Framework for Balanced Trees," Proceedings of the Nineteenth Annual IEEE Symposium on Foundations of Computer Science (1978), 8-21.

20.     T. H. Hibbard, "Some Combinatorial Properties of Certain Trees with Applications to Searching and Sorting," Journal of the ACM 9 (1962), 13-28.

21.     A. T. Jonassen and D. E. Knuth, "A Trivial Algorithm Whose Analysis Isn't," Journal of Computer and System Sciences 16 (1978), 301-322.

22.     P. L. Karlton, S. H. Fuller, R. E. Scroggs, and E. B. Kaehler, "Performance of Height Balanced Trees," Communications of the ACM 19 (1976), 23-28.

23.     D. E. Knuth, The Art of Computer Programming: Volume 1: Fundamental Algorithms, second edition, Addison-Wesley, Reading, MA, 1973.

24.     D. E. Knuth, The Art of Computer Programming: Volume 3: Sorting and Searching, second printing, Addison-Wesley, Reading, MA, 1975.

25.     K. Melhorn, "A Partial Analysis of Height-Balanced Trees under Random Insertions and Deletions," SIAM Journal of Computing 11 (1982), 748-760.

26.     K. Melhorn, Data Structures and Algorithms 1: Sorting and Searching, Springer-Verlag, Berlin, 1984.

27.     J. Nievergelt and E. M. Reingold, "Binary Search Trees of Bounded Balance," SIAM Journal on Computing 2 (1973), 33-43.

28.     A. J. Perlis and C. Thornton, "Symbol Manipulation in Threaded Lists," Communications of the ACM 3 (1960), 195-204.

29.     D. D. Sleator and R. E. Tarjan, "Self-adjusting Binary Search Trees," Journal of ACM 32 (1985), 652-686.

30.     D. D. Sleator, R. E. Tarjan, and W. P. Thurston, "Rotation Distance, Triangulations, and Hyperbolic Geometry," Journal of AMS (1988), 647-682.

31.     H. F. Smith, Data Structures-Form and Function, Harcourt Brace Jovanovich, 1987.

32.     R. E. Tarjan, "Sequential Access in Splay Trees Takes Linear Time," Combinatorica 5 (1985), 367-378.

33.     A. C. Yao, "On Random 2-3 trees," Acta Informatica 9 (1978), 159-170.

Go to Chapter 5         Return to Table of Contents


Structures, Algorithm Analysis: CHAPTER 5: HASHING

 Return to Table of Contents

CHAPTER 5: HASHING

In Chapter 4, we discussed the search tree ADT, which allowed various operations on a set of elements. In this chapter, we discuss the hash table ADT, which supports only a subset of the operations allowed by binary search trees.

The implementation of hash tables is frequently called hashing. Hashing is a technique used for performing insertions, deletions and finds in constant average time. Tree operations that require any ordering information among the elements are not supported efficiently. Thus, operations such as find_min, find_max, and the printing of the entire table in sorted order in linear time are not supported.

The central data structure in this chapter is the hash table. We will

See several methods of implementing the hash table.

Compare these methods analytically.

Show numerous applications of hashing.

Compare hash tables with binary search trees.

5.1. General Idea

The ideal hash table data structure is merely an array of some fixed size, containing the keys. Typically, a key is a string with an associated value (for instance, salary information). We will refer to the table size as H_SIZE, with the understanding that this is part of a hash data structure and not merely some variable floating around globally. The common convention is to have the table run from 0 to H_SIZE-1; we will see why shortly.

Each key is mapped into some number in the range 0 to H_SIZE - 1 and placed in the appropriate cell. The mapping is called a hash function, which ideally should be simple to compute and should ensure that any two distinct keys get different cells. Since there are a finite number of cells and a virtually inexhaustible supply of keys, this is clearly impossible, and thus we seek a hash function that distributes the keys evenly among the cells. Figure 5.1 is typical of a perfect situation. In this example, john hashes to 3, phil hashes to 4, dave hashes to 6, and mary hashes to 7.

Figure 5.1 An ideal hash table

This is the basic idea of hashing. The only remaining problems deal with choosing a function, deciding what to do when two keys hash to the same value (this is known as a collision), and deciding on the table size.

5.2. Hash Function

If the input keys are integers, then simply returning key mod H_SIZE is generally a reasonable strategy, unless key happens to have some undesirable properties. In this case, the choice of hash function needs to be carefully considered. For instance, if the table size is 10 and the keys all end in zero, then the standard hash function is obviously a bad choice. For reasons we shall see later, and to avoid situations like the one above, it is usually a good idea to ensure that the table size is prime. When the input keys are random integers, then this function is not only very simple to compute but also distributes the keys evenly.

Usually, the keys are strings; in this case, the hash function needs to be chosen carefully.

One option is to add up the ASCII values of the characters in the string. In Figure 5.2 we declare the type INDEX, which is returned by the hash function. The routine in Figure 5.3 implements this strategy and uses the typical C method of stepping through a string.

The hash function depicted in Figure 5.3 is simple to implement and computes an answer quickly. However, if the table size is large, the function does not distribute the keys well. For instance, suppose that H_SIZE = 10,007 (10,007 is a prime number). Suppose all the keys are eight or fewer characters long. Since a char has an integer value that is always at most 127, the hash function can only assume values between 0 and 1016, which is 127 * 8. This is clearly not an equitable distribution!

typedef unsigned int INDEX;

Figure 5.2 Type returned by hash function

INDEX

hash( char *key, unsigned int H_SIZE )

{

unsigned int hash_val = 0;

/*1*/       while( *key != '\0' )

/*2*/            hash_val += *key++;

/*3*/       return( hash_val % H_SIZE );

}

Figure 5.3 A simple hash function

Another hash function is shown in Figure 5.4. This hash function assumes key has at least two characters plus the NULL terminator. 27 represents the number of letters in the English alphabet, plus the blank, and 729 is 272. This function only examines the first three characters, but if these are random, and the table size is 10,007, as before, then we would expect a reasonably equitable distribution. Unfortunately, English is not random. Although there are 263 = 17,576 possible combinations of three characters (ignoring blanks), a check of a reasonably large on-line dictionary reveals that the number of different combinations is actually only 2,851. Even if none of these combinations collide, only 28 percent of the table can actually be hashed to. Thus this function, although easily computable, is also not appropriate if the hash table is reasonably large.

Figure 5.5 shows a third attempt at a hash function. This hash function involves all characters in the key and can generally be expected to distribute well (it computes   [image]  key/key_size - i - 1] 32i, and brings the result into proper range). The code computes a polynomial function (of 32) by use of Horner's rule. For instance, another way of computing hk = k1 + 27k2 + 272k3 is by the formula hk = ((k3) * 27 + k2) * 27 + k1. Horner's rule extends this to an nth degree polynomial.

We have used 32 instead of 27, because multiplication by 32 is not really a multiplication, but amounts to bit-shifting by five. In line 2, the addition could be replaced with a bitwise exclusive or, for increased speed.

INDEX

hash( char *key, unsigned int H_SIZE )

{

return ( ( key[0] + 27*key[1] + 729*key[2] ) % H_SIZE );

}

Figure 5.4 Another possible hash function -- not too good

INDEX

hash( char *key, unsigned int H_SIZE )

{

unsigned int hash_val = O;

/*1*/       while( *key != '\0' )

/*2*/            hash_val = ( hash_val << 5 ) + *key++;

/*3*/       return( hash_val % H_SIZE );

}

Figure 5.5 A good hash function

The hash function described in Figure 5.5 is not necessarily the best with respect to table distribution, but does have the merit of extreme simplicity (and speed if overflows are allowed). If the keys are very long, the hash function will take too long to compute. Furthermore, the early characters will wind up being left-shifted out of the eventual answer. A common practice in this case is not to use all the characters. The length and properties of the keys would then influence the choice. For instance, the keys could be a complete street address. The hash function might include a couple of characters from the street address and perhaps a couple of characters from the city name and ZIP code. Some programmers implement their hash function by using only the characters in the odd spaces, with the idea that the time saved computing the hash function will make up for a slightly less evenly distributed function.

The main programming detail left is collision resolution. If, when inserting an element, it hashes to the same value as an already inserted element, then we have a collision and need to resolve it. There are several methods for dealing with this. We will discuss two of the simplest: open hashing and closed hashing.*

*These are also commonly known as separate chaining and open addressing, respectively.

5.3. Open Hashing (Separate Chaining)

The first strategy, commonly known as either open hashing, or separate chaining, is to keep a list of all elements that hash to the same value. For convenience, our lists have headers. This makes the list implementation the same as in Chapter 3. If space is tight, it might be preferable to avoid their use. We assume for this section that the keys are the first 10 perfect squares and that the hashing function is simply hash(x) = x mod 10. (The table size is not prime, but is used here for simplicity.) Figure 5.6 should make this clear.

To perform a find, we use the hash function to determine which list to traverse. We then traverse this list in the normal manner, returning the position where the item is found. To perform an insert, we traverse down the appropriate list to check whether the element is already in place (if duplicates are expected, an extra field is usually kept, and this field would be incremented in the event of a match). If the element turns out to be new, it is inserted either at the front of the list or at the end of the list, whichever is easiest. This is an issue most easily addressed while the code is being written. Sometimes new elements are inserted at the front of the list, since it is convenient and also because frequently it happens that recently inserted elements are the most likely to be accessed in the near future.

Figure 5.6 An open hash table

The type declarations required to implement open hashing are in Figure 5.7. The first few lines are the same as the linked list declarations of Chapter 3. The hash table structure contains the actual size and an array of linked lists, which are dynamically allocated when the table is initialized. The HASH_TABLE type is just a pointer to this structure.

typedef struct list_node *node_ptr;

struct list_node

{

element_type element;

node_ptr next;

};

typedef node_ptr LIST;

typedef node_ptr position;

/* LIST *the_list will be an array of lists, allocated later */

/* The lists will use headers, allocated later */

struct hash_tbl

{

unsigned int table_size;

LIST *the_lists;

};

typedef struct hash_tbl *HASH_TABLE;

Figure 5.7 Type declaration for open hash table

Notice that the the_lists field is actually a pointer to a pointer to a list_node structure. If typedefs and abstraction are not used, this can be quite confusing.

HASH_TABLE

initialize_table( unsigned int table_size )

{

HASH_TABLE H;

int i;

/*1*/       if( table size < MIN_TABLE_SIZE )

{

/*2*/            error("Table size too small");

/*3*/            return NULL;

}

/* Allocate table */

/*4*/       H = (HASH_TABLE) malloc ( sizeof (struct hash_tbl) );

/*5*/       if( H == NULL )

/*6*/            fatal_error("Out of space!!!");

/*7*/       H->table_size = next_prime( table_size );

/* Allocate list pointers */

/*8*/       H->the_lists = (position *)

malloc( sizeof (LIST) * H->table_size );

/*9*/       if( H->the_lists == NULL )

/*10*/           fatal_error("Out of space!!!");

/* Allocate list headers */

/*11*/      for(i=0; i<H->table_size; i++ )

{

/*12*/           H->the_lists[i] = (LIST) malloc

( sizeof (struct list_node) );

/*13*/           if( H->the_lists[i] == NULL )

/*14*/                fatal_error("Out of space!!!");

else

/*15*/                H->the_lists[i]->next = NULL;

}

/*16*/      return H;

}

Figure 5.8 Initialization routine for open hash table

Figure 5.8 shows the initialization function, which uses the same ideas that were seen in the array implementation of stacks. Lines 4 through 6 allocate a hash table structure. If space is available, then H will point to a structure containing an integer and a pointer to a list. Line 7 sets the table size to a prime number, and lines 8 through 10 attempt to allocate an array of lists. Since a LIST is defined to be a pointer, the result is an array of pointers.

If our LIST implementation was not using headers, we could stop here. Since our implementation uses headers, we must allocate one header per list and set its next field to NULL. This is done in lines 11 through 15. Of course, lines 12 through 15 could be replaced with the statement

H->the_lists[i] = make_null();

Although we have not used this option, because in this instance it is preferable to make the code as self-contained as possible, it is certainly worth considering. An inefficiency of our code is that the malloc on line 12 is performed H->table_size times. This can be avoided by replacing line 12 with one call to malloc before the loop occurs:

H->the lists = (LIST*) malloc

(H->table_size * sizeof (struct list_node));

Line 16 returns H.

The call find(key, H) will return a pointer to the cell containing key. The code to implement this is shown in Figure 5.9. Notice that lines 2 through 5 are identical to the code to perform a find that is given in Chapter 3. Thus, the list ADT implementation in Chapter 3 could be used here. Remember that if element_type is a string, comparison and assignment must be done with strcmp and strcpy, respectively.

Next comes the insertion routine. If the item to be inserted is already present, then we do nothing; otherwise we place it at the front of the list (see Fig. 5.10).*

*Since the table in Figure 5.6 was created by inserting at the end of the list, the code in Figure 5.10 will produce a table with the lists in Figure 5.6 reversed.

position

find( element_type key, HASH_TABLE H )

{

position p;

LIST L;

/*1*/       L = H->the_lists[ hash( key, H->table_size) ];

/*2*/       p = L->next;

/*3*/       while( (p != NULL) && (p->element != key) )

/* Probably need strcmp!! */

/*4*/            p = p->next;

/*5*/       return p;

}

Figure 5.9 Find routine for open hash table

void

insert( element_type key, HASH_TABLE H )

{

position pos, new_cell;

LIST L;

/*1*/      pos = find( key, H );

/*2*/      if( pos == NULL )

{

/*3*/          new_cell = (position) malloc(sizeof(struct list_node));

/*4*/          if( new_cell == NULL )

/*5*/              fatal_error("Out of space!!!");

else

{

/*6*/              L = H->the_lists[ hash( key, H->table size ) ];

/*7*/              new_cell->next = L->next;

/*8*/              new_cell->element = key; /* Probably need strcpy!! */

/*9*/              L->next = new_cell;

}

}

}

Figure 5.10 Insert routine for open hash table

The element can be placed anywhere in the list; this is most convenient in our case. Notice that the code to insert at the front of the list is essentially identical to the code in Chapter 3 that implements a push using linked lists. Again, if the ADTs in Chapter 3 have already been carefully implemented, they can be used here.

The insertion routine coded in Figure 5.10 is somewhat poorly coded, because it computes the hash function twice. Redundant calculations are always bad, so this code should be rewritten if it turns out that the hash routines account for a significant portion of a program's running time.

The deletion routine is a straightforward implementation of deletion in a linked list, so we will not bother with it here. If the repertoire of hash routines does not include deletions, it is probably best to not use headers, since their use would provide no simplification and would waste considerable space. We leave this as an exercise, too.

Any scheme could be used besides linked lists to resolve the collisions-a binary search tree or even another hash table would work, but we expect that if the table is large and the hash function is good, all the lists should be short, so it is not worthwhile to try anything complicated.

We define the load factor,   [image], of a hash table to be the ratio of the number of elements in the hash table to the table size. In the example above,   [image]  = 1.0. The average length of a list is   [image]. The effort required to perform a search is the constant time required to evaluate the hash function plus the time to traverse the list.

Figure 5.11 Closed hash table with linear probing, after each insertion

In an unsuccessful search, the number of links to traverse is   [image]  (excluding the final NULL link) on average. A successful search requires that about 1 + (  [image] /2) links be traversed, since there is a guarantee that one link must be traversed (since the search is successful), and we also expect to go halfway down a list to find our match. This analysis shows that the table size is not really important, but the load factor is. The general rule for open hashing is to make the table size about as large as the number of elements expected (in other words, let   [image]    [image]  1). It is also a good idea, as mentioned before, to keep the table size prime to ensure a good distribution.

5.4. Closed Hashing (Open Addressing)

Open hashing has the disadvantage of requiring pointers. This tends to slow the algorithm down a bit because of the time required to allocate new cells, and also essentially requires the implementation of a second data structure. Closed hashing, also known as open addressing, is an alternative to resolving collisions with linked lists. In a closed hashing system, if a collision occurs, alternate cells are tried until an empty cell is found. More formally, cells h0(x), h1(x), h2(x), . . . are tried in succession where hi(x) = (hash(x) +   [image] (i))mod H_SIZE, with   [image] (0) = 0. The function,   [image], is the collision resolution strategy. Because all the data goes inside the table, a bigger table is needed for closed hashing than for open hashing. Generally, the load factor should be below   [image]  = 0.5 for closed hashing. We now look at three common collision resolution strategies.

5.4.1. Linear Probing

5.4.2. Quadratic Probing

5.4.3. Double Hashing

5.4.1. Linear Probing

In linear probing,   [image]  is a linear function of i, typically   [image] (i) = i. This amounts to trying cells sequentially (with wraparound) in search of an empty cell. Figure 5.11 shows the result of inserting keys {89, 18, 49, 58, 69} into a closed table using the same hash function as before and the collision resolution strategy,   [image] (i) = i.

The first collision occurs when 49 is inserted; it is put in the next available spot, namely spot 0, which is open. 58 collides with 18, 89, and then 49 before an empty cell is found three away. The collision for 69 is handled in a similar manner. As long as the table is big enough, a free cell can always be found, but the time to do so can get quite large. Worse, even if the table is relatively empty, blocks of occupied cells start forming. This effect, known as primary clustering, means that any key that hashes into the cluster will require several attempts to resolve the collision, and then it will add to the cluster.

Although we will not perform the calculations here, it can be shown that the expected number of probes using linear probing is roughly 1/2(1 + 1/(1 -   [image])2) for insertions and unsuccessful searches and 1/2(1 + 1/ (1-   [image])) for successful searches. The calculations are somewhat involved. It is easy to see from the code that insertions and unsuccessful searches require the same number of probes. A moment's thought suggests that on average, successful searches should take less time than unsuccessful searches.

The corresponding formulas, if clustering were not a problem, are fairly easy to derive. We will assume a very large table and that each probe is independent of the previous probes. These assumptions are satisfied by a random collision resolution strategy and are reasonable unless   [image]  is very close to 1. First, we derive the expected number of probes in an unsuccessful search. This is just the expected number of probes until we find an empty cell. Since the fraction of empty cells is 1 -   [image], the number of cells we expect to probe is 1/(1 -   [image]). The number of probes for a successful search is equal to the number of probes required when the particular element was inserted. When an element is inserted, it is done as a result of an unsuccessful search. Thus we can use the cost of an unsuccessful search to compute the average cost of a successful search.

The caveat is that   [image]  changes from 0 to its current value, so that earlier insertions are cheaper and should bring the average down. For instance, in the table above,   [image]  = 0.5, but the cost of accessing 18 is determined when 18 is inserted. At that point,   [image]  = 0.2. Since 18 was inserted into a relatively empty table, accessing it should be easier than accessing a recently inserted element such as 69. We can estimate the average by using an integral to calculate the mean value of the insertion time, obtaining

These formulas are clearly better than the corresponding formulas for linear probing. Clustering is not only a theoretical problem but actually occurs in real implementations. Figure 5.12 compares the performance of linear probing (dashed curves) with what would be expected from more random collision resolution. Successful searches are indicated by an S, and unsuccessful searches and insertions are marked with U and I, respectively.

If   [image]  = 0.75, then the formula above indicates that 8.5 probes are expected for an insertion in linear probing. If   [image]  = 0.9, then 50 probes are expected, which is unreasonable. This compares with 4 and 10 probes for the respective load factors if clustering were not a problem. We see from these formulas that linear probing can be a bad idea if the table is expected to be more than half full. If   [image]  = 0.5, however, only 2.5 probes are required on average for insertion and only 1.5 probes are required, on average, for a successful search.

5.4.2. Quadratic Probing

Quadratic probing is a collision resolution method that eliminates the primary clustering problem of linear probing. Quadratic probing is what you would expect-the collision function is quadratic. The popular choice is   [image] (i) = i2. Figure 5.13 shows the resulting closed table with this collision function on the same input used in the linear probing example.

When 49 collides with 89, the next position attempted is one cell away. This cell is empty, so 49 is placed there. Next 58 collides at position 8. Then the cell one away is tried but another collision occurs. A vacant cell is found at the next cell tried, which is 22 = 4 away. 58 is thus placed in cell 2. The same thing happens for 69.

For linear probing it is a bad idea to let the hash table get nearly full, because performance degrades. For quadratic probing, the situation is even more drastic: There is no guarantee of finding an empty cell once the table gets more than half full, or even before the table gets half full if the table size is not prime. This is because at most half of the table can be used as alternate locations to resolve collisions.

Indeed, we prove now that if the table is half empty and the table size is prime, then we are always guaranteed to be able to insert a new element.

THEOREM 5.1.

If quadratic probing is used, and the table size is prime, then a new element can always be inserted if the table is at least half empty.

PROOF:

Let the table size, H_SIZE, be an (odd) prime greater than 3. We show that the first   [image] H_SIZE/2  [image]  alternate locations are all distinct. Two of these locations are h(x) + i2(mod H_SIZE) and h(x) + j2(mod H_SIZE), where 0 < i, j   [image]    [image] H_SIZE/2  [image] . Suppose, for the sake of contradiction, that these locations are the same, but i   [image]  j. Then

h(x) + i2 = h(x) + j2     (mod H_SIZE)

i2 = j2             (mod H_SIZE)

i2 - j2 = 0            (mod H_SIZE)

(i - j)(i + j) = 0            (mod H_SIZE)

Figure 5.12 Number of probes plotted against load factor for linear probing (dashed) and random strategy. S is successful search,U is unsuccessful search, I is insertion

Figure 5.13 Closed hash table with quadratic probing, after each insertion

Since H_SIZE is prime, it follows that either (i - j) or  (i + j) is equal to 0 (mod H_SIZE). Since i and j are distinct, the first option is not possible. Since 0 < i, j <   [image] H_SIZE/2  [image] , the second option is also impossible. Thus, the first   [image] H_SIZE/2  [image]  alternate locations are distinct. Since the element to be inserted can also be placed in the cell to which it hashes (if there are no collisions), any element has   [image] H_SIZE/2  [image]  locations into which it can go. If at most   [image] H_SIZE/2  [image]  positions are taken, then an empty spot can always be found.

If the table is even one more than half full, the insertion could fail (although this is extremely unlikely). Therefore, it is important to keep this in mind. It is also crucial that the table size be prime.* If the table size is not prime, the number of alternate locations can be severely reduced. As an example, if the table size were 16, then the only alternate locations would be at distances 1, 4, or 9 away.

*If the table size is a prime of the form 4k + 3, and the quadratic collision resolution strategy f(i) = + i2 is used, then the entire table can be probed. The cost is a slightly more complicated routine.

Standard deletion cannot be performed in a closed hash table, because the cell might have caused a collision to go past it. For instance, if we remove 89, then virtually all of the remaining finds will fail. Thus, closed hash tables require lazy deletion, although in this case there really is no laziness implied.

The type declarations required to implement closed hashing are in Figure 5.14. Instead of an array of lists, we have an array of hash table entry cells, which, as in open hashing, are allocated dynamically. Initializing the table (Figure 5.15) consists of allocating space (lines 1 through 10) and then setting the info field to empty for each cell.

enum kind_of_entry { legitimate, empty, deleted };

struct hash_entry

{

element_type element;

enum kind_of_entry info;

};

typedef INDEX position;

typedef struct hash_entry cell;

/* the_cells is an array of hash_entry cells, allocated later */

struct hash_tbl

{

unsigned int table_size;

cell *the_cells;

};

typedef struct hash_tbl *HASH_TABLE;

Figure 5.14 Type declaration for closed hash tables

HASH_TABLE

initialize_table( unsigned int table_size )

{

HASH_TABLE H;

int i;

/*1*/       if( table_size < MIN_TABLE_SIZE )

{

/*2*/            error("Table size too small");

/*3*/            return NULL;

}

/* Allocate table */

/*4*/       H = (HASH_TABLE) malloc( sizeof ( struct hash_tbl ) );

/*5*/       if( H == NULL )

/*6*/           fatal_error("Out of space!!!");

/*7*/       H->table_size = next_prime( table_size );

/* Allocate cells */

/*8*/       H->the cells = (cell *) malloc

( sizeof ( cell ) * H->table_size );

/*9*/       if( H->the_cells == NULL )

/*10*/           fatal_error("Out of space!!!");

/*11*/      for(i=0; i<H->table_size; i++ )

/*12*/           H->the_cells[i].info = empty;

/*13*/      return H;

}

Figure 5.15 Routine to initialize closed hash table

As with open hashing, find(key, H) will return the position of key in the hash table. If key is not present, then find will return the last cell. This cell is where key would be inserted if needed. Further, because it is marked empty, it is easy to tell that the find failed. We assume for convenience that the hash table is at least twice as large as the number of elements in the table, so quadratic resolution will always work. Otherwise, we would need to test i before line 4. In the implementation in Figure 5.16, elements that are marked as deleted count as being in the table. This can cause problems, because the table can get too full prematurely. We shall discuss this item presently.

Lines 4 through 6 represent the fast way of doing quadratic resolution. From the definition of the quadratic resolution function, f(i) = f(i - 1) + 2i -1, so the next cell to try can be determined with a multiplication by two (really a bit shift) and a decrement. If the new location is past the array, it can be put back in range by subtracting H_SIZE. This is faster than the obvious method, because it avoids the multiplication and division that seem to be required. The variable name i is not the best one to use; we only use it to be consistent with the text.

position

find( element_type key, HASH_TABLE H )

{

position i, current_pos;

/*1*/       i = 0;

/*2*/       current_pos = hash( key, H->table_size );

/* Probably need strcmp! */

/*3*/       while( (H->the_cells[current_pos].element != key ) &&

(H->the_cells[current_pos].info != empty ) )

{

/*4*/             current_pos += 2*(++i) - 1;

/*5*/             if( current_pos >= H->table_size )

/*6*/                  current_pos -= H->table_size;

}

/*7*/       return current_pos;

}

Figure 5.16 Find routine for closed hashing with quadratic probing

The final routine is insertion. As with open hashing, we do nothing if key is already present. It is a simple modification to do something else. Otherwise, we place it at the spot suggested by the find routine. The code is shown in Figure 5.17.

Although quadratic probing eliminates primary clustering, elements that hash to the same position will probe the same alternate cells. This is known as secondary clustering. Secondary clustering is a slight theoretical blemish. Simulation results suggest that it generally causes less than an extra   [image]  probe per search. The following technique eliminates this, but does so at the cost of extra multiplications and divisions.

5.4.3. Double Hashing

The last collision resolution method we will examine is double hashing. For double hashing, one popular choice is f(i) = i   [image]  h2(x). This formula says that we apply a second hash function to x and probe at a distance h2(x), 2h2(x), . . ., and so on. A poor choice of h2(x) would be disastrous. For instance, the obvious choice h2(x) = x mod 9 would not help if 99 were inserted into the input in the previous examples. Thus, the function must never evaluate to zero. It is also important to make sure all cells can be probed (this is not possible in the example below, because the table size is not prime). A function such as h2(x) = R - (x mod R), with R a prime smaller than H_SIZE, will work well. If we choose R = 7, then Figure 5.18 shows the results of inserting the same keys as before.

void

insert( element_type key, HASH_TABLE H )

{

position pos;

pos = find( key, H );

if( H->the_cells[pos].info != legitimate )

{    /* ok to insert here */

H->the_cells[pos].info = legitimate;

H->the_cells[pos].element = key;

/* Probably need strcpy!! */

}

}

Figure 5.17 Insert routine for closed hash tables with quadratic probing

Figure 5.18 Closed hash table with double hashing, after each insertion

The first collision occurs when 49 is inserted. h2(49) = 7 - 0 = 7, so 49 is inserted in position 6. h2(58) = 7 - 2 = 5, so 58 is inserted at location 3. Finally, 69 collides and is inserted at a distance h2(69) = 7 - 6 = 1 away. If we tried to insert 60 in position 0, we would have a collision. Since h2(60) = 7 - 4 = 3, we would then try positions 3, 6, 9, and then 2 until an empty spot is found. It is generally possible to find some bad case, but there are not too many here.

As we have said before, the size of our sample hash table is not prime. We have done this for convenience in computing the hash function, but it is worth seeing why it is important to make sure the table size is prime when double hashing is used. If we attempt to insert 23 into the table, it would collide with 58. Since h2(23) = 7 - 2 = 5, and the table size is 10, we essentially have only one alternate location, and it is already taken. Thus, if the table size is not prime, it is possible to run out of alternate locations prematurely. However, if double hashing is correctly implemented, simulations imply that the expected number of probes is almost the same as for a random collision resolution strategy. This makes double hashing theoretically interesting. Quadratic probing, however, does not require the use of a second hash function and is thus likely to be simpler and faster in practice.

5.5. Rehashing

If the table gets too full, the running time for the operations will start taking too long and inserts might fail for closed hashing with quadratic resolution. This can happen if there are too many deletions intermixed with insertions. A solution, then, is to build another table that is about twice as big (with associated new hash function) and scan down the entire original hash table, computing the new hash value for each (non-deleted) element and inserting it in the new table.

As an example, suppose the elements 13, 15, 24, and 6 are inserted into a closed hash table of size 7. The hash function is h(x) = x mod 7. Suppose linear probing is used to resolve collisions. The resulting hash table appears in Figure 5.19.

If 23 is inserted into the table, the resulting table in Figure 5.20 will be over 70 percent full. Because the table is so full, a new table is created. The size of this table is 17, because this is the first prime which is twice as large as the old table size. The new hash function is then h(x) = x mod 17. The old table is scanned, and elements 6, 15, 23, 24, and 13 are inserted into the new table. The resulting table appears in Figure 5.21.

This entire operation is called rehashing. This is obviously a very expensive operation -- the running time is O(n), since there are n elements to rehash and the table size is roughly 2n, but it is actually not all that bad, because it happens very infrequently. In particular, there must have been n/2 inserts prior to the last rehash, so it essentially adds a constant cost to each insertion.* If this data structure is part of the program, the effect is not noticeable. On the other hand, if the hashing is performed as part of an interactive system, then the unfortunate user whose insertion caused a rehash could see a slowdown.

*This is why the new table is made twice as large as the old table.

Figure 5.19 Closed hash table with linear probing with input 13,15, 6, 24

Figure 5.20 Closed hash table with linear probing after 23 is inserted

Figure 5.21 Closed hash table after rehashing

Rehashing can be implemented in several ways with quadratic probing. One alternative is to rehash as soon as the table is half full. The other extreme is to rehash only when an insertion fails. A third, middle of the road, strategy is to rehash when the table reaches a certain load factor. Since performance does degrade as the load factor increases, the third strategy, implemented with a good cutoff, could be best.

Rehashing frees the programmer from worrying about the table size and is important because hash tables cannot be made arbitrarily large in complex programs. The exercises ask you to investigate the use of rehashing in conjunction with lazy deletion. Rehashing can be used in other data structures as well. For instance, if the queue data structure of Chapter 3 became full, we could declare a double-sized array and copy everything over, freeing the original.

Figure 5.22 shows that rehashing is simple to implement.

HASH_TABLE

rehash( HASH_TABLE H )

{

unsigned int i, old_size;

cell *old_cells;

/*1*/        old_cells = H->the_cells;

/*2*/        old_size = H->table_size;

/* Get a new, empty table */

/*3*/       H = initialize_table( 2*old_size );

/* Scan through old table, reinserting into new */

/*4*/       for( i=0; i<old_size; i++ )

/*5*/            if( old_cells[i].info == legitimate )

/*6*/                 insert( old_cells[i].element, H );

/*7*/       free( old_cells );

/*8*/       return H;

}

Figure 5.22

5.6. Extendible Hashing

Our last topic in this chapter deals with the case where the amount of data is too large to fit in main memory. As we saw in Chapter 4, the main consideration then is the number of disk accesses required to retrieve data.

As before, we assume that at any point we have n records to store; the value of n changes over time. Furthermore, at most m records fit in one disk block. We will use m = 4 in this section.

If either open hashing or closed hashing is used, the major problem is that collisions could cause several blocks to be examined during a find, even for a well-distributed hash table. Furthermore, when the table gets too full, an extremely expensive rehashing step must be performed, which requires O(n) disk accesses.

A clever alternative, known as extendible hashing, allows a find to be performed in two disk accesses. Insertions also require few disk accesses.

We recall from Chapter 4 that a B-tree has depth O(logm/2n). As m increases, the depth of a B-tree decreases. We could in theory choose m to be so large that the depth of the B-tree would be 1. Then any find after the first would take one disk access, since, presumably, the root node could be stored in main memory. The problem with this strategy is that the branching factor is so high that it would take considerable processing to determine which leaf the data was in. If the time to perform this step could be reduced, then we would have a practical scheme. This is exactly the strategy used by extendible hashing.

Let us suppose, for the moment, that our data consists of several six-bit integers. Figure 5.23 shows an extendible hashing scheme for this data. The root of the "tree" contains four pointers determined by the leading two bits of the data. Each leaf has up to m = 4 elements. It happens that in each leaf the first two bits are identical; this is indicated by the number in parentheses. To be more formal, D will represent the number of bits used by the root, which is sometimes known as the directory. The number of entries in the directory is thus 2D. dl is the number of leading bits that all the elements of some leaf l have in common. dl will depend on the particular leaf, and dl   [image]  D.

Figure 5.23 Extendible hashing: original data

Suppose that we want to insert the key 100100. This would go into the third leaf, but as the third leaf is already full, there is no room. We thus split this leaf into two leaves, which are now determined by the first three bits. This requires increasing the directory size to 3. These changes are reflected in Figure 5.24.

Notice that all of the leaves not involved in the split are now pointed to by two adjacent directory entries. Thus, although an entire directory is rewritten, none of the other leaves are actually accessed.

If the key 000000 is now inserted, then the first leaf is split, generating two leaves with dl = 3. Since D = 3, the only change required in the directory is the updating of the 000 and 001 pointers. See Figure 5.25.

This very simple strategy provides quick access times for insert and find operations on large databases. There are a few important details we have not considered.

First, it is possible that several directory splits will be required if the elements in a leaf agree in more than D + 1 leading bits. For instance, starting at the original example, with D = 2, if 111010, 111011, and finally 111100 are inserted, the directory size must be increased to 4 to distinguish between the five keys. This is an easy detail to take care of, but must not be forgotten. Second, there is the possibility of duplicate keys; if there are more than m duplicates, then this algorithm does not work at all. In this case, some other arrangements need to be made.

These possibilities suggest that it is important for the bits to be fairly random. This can be accomplished by hashing the keys into a reasonably long integer; hence the reason for the name.

Figure 5.24 Extendible hashing: after insertion of 100100 and directory split

Figure 5.25 Extendible hashing: after insertion of 000000 and leaf split

We close by mentioning some of the performance properties of extendible hashing, which are derived after a very difficult analysis. These results are based on the reasonable assumption that the bit patterns are uniformly distributed.

The expected number of leaves is (n / m) log2 e. Thus the average leaf is ln 2 = 0.69 full. This is the same as B-trees, which is not entirely surprising, since for both data structures new nodes are created when the (m + 1)st entry is added.

The more surprising result is that the expected size of the directory (in other words, 2D) is O(n1+1 /m/ m). If m is very small, then the directory can get unduly large. In this case, we can have the leaves contain pointers to the records instead of the actual records, thus increasing the value of m. This adds a second disk access to each find operation in order to maintain a smaller directory. If the directory is too large to fit in main memory, the second disk access would be needed anyway.

Summary

Hash tables can be used to implement the insert and find operations in constant average time. It is especially important to pay attention to details such as load factor when using hash tables, since otherwise the time bounds are not valid. It is also important to choose the hash function carefully when the key is not a short string or integer.

For open hashing, the load factor should be close to 1, although performance does not significantly degrade unless the load factor becomes very large. For closed hashing, the load factor should not exceed 0.5, unless this is completely unavoidable. If linear probing is used, performance degenerates rapidly as the load factor approaches 1. Rehashing can be implemented to allow the table to grow (and shrink), thus maintaining a reasonable load factor. This is important if space is tight and it is not possible just to declare a huge hash table.

Binary search trees can also be used to implement insert and find operations. Although the resulting average time bounds are O(log n), binary search trees also support routines that require order and are thus more powerful. Using a hash table, it is not possible to find the minimum element. It is not possible to search efficiently for a string unless the exact string is known. A binary search tree could quickly find all items in a certain range; this is not supported by hash tables. Furthermore, the O(log n) bound is not necessarily that much more than O (1), especially since no multiplications or divisions are required by search trees.

On the other hand, the worst case for hashing generally results from an implementation error, whereas sorted input can make binary trees perform poorly. Balanced search trees are quite expensive to implement, so if no ordering information is required and there is any suspicion that the input might be sorted, then hashing is the data structure of choice.

Hashing applications are abundant. Compilers use hash tables to keep track of declared variables in source code. The data structure is known as a symbol table. Hash tables are the ideal application for this problem because only inserts and finds are performed. Identifiers are typically short, so the hash function can be computed quickly.

A hash table is useful for any graph theory problem where the nodes have real names instead of numbers. Here, as the input is read, vertices are assigned integers from 1 onwards by order of appearance. Again, the input is likely to have large groups of alphabetized entries. For example, the vertices could be computers. Then if one particular installation lists its computers as ibm1, ibm2, ibm3, . . . , there could be a dramatic effect on efficiency if a search tree is used.

A third common use of hash tables is in programs that play games. As the program searches through different lines of play, it keeps track of positions it has seen by computing a hash function based on the position (and storing its move for that position). If the same position reoccurs, usually by a simple transposition of moves, the program can avoid expensive recomputation. This general feature of all game-playing programs is known as the transposition table.

Yet another use of hashing is in online spelling checkers. If misspelling detection (as opposed to correction) is important, an entire dictionary can be prehashed and words can be checked in constant time. Hash tables are well suited for this, because it is not important to alphabetize words; printing out misspellings in the order they occurred in the document is certainly acceptable.

We close this chapter by returning to the word puzzle problem of Chapter 1. If the second algorithm described in Chapter 1 is used, and we assume that the maximum word size is some small constant, then the time to read in the dictionary containing W words and put it in a hash table is O(W). This time is likely to be dominated by the disk I/O and not the hashing routines. The rest of the algorithm would test for the presence of a word for each ordered quadruple (row, column, orientation, number of characters). As each lookup would be O(1), and there are only a constant number of orientations (8) and characters per word, the running time of this phase would be O(r   [image]  c). The total running time would be O (r   [image]  c + W), which is a distinct improvement over the original O (r   [image]  c   [image]  W). We could make further optimizations, which would decrease the running time in practice; these are described in the exercises.

Exercises

5.1     Given input {4371, 1323, 6173, 4199, 4344, 9679, 1989} and a hash function h(x) = x(mod 10), show the resulting

a.     open hash table

b.     closed hash table using linear probing

c.     closed hash table using quadratic probing

d.     closed hash table with second hash function h2(x) = 7 - (x mod 7)

5.2     Show the result of rehashing the hash tables in Exercise 5.1.

5.3          Write a program to compute the number of collisions required in a long random sequence of insertions using linear probing, quadratic probing, and double hashing.

5.4     A large number of deletions in an open hash table can cause the table to be fairly empty, which wastes space. In this case, we can rehash to a table half as large. Assume that we rehash to a larger table when there are twice as many elements as the table size. How empty should an open table be before we rehash to a smaller table?

5.5     An alternative collision resolution strategy is to define a sequence, f(i) = ri, where r0 = 0 and r1, r2, . . . , rn is a random permutation of the first n integers (each integer appears exactly once).

a.     Prove that under this strategy, if the table is not full, then the collision can always be resolved.

b.     Would this strategy be expected to eliminate clustering?

c.     If the load factor of the table is   [image], what is the expected time to perform an insert?

d.     If the load factor of the table is   [image]  , what is the expected time for a successful search?

e.     Give an efficient algorithm (theoretically as well as practically) to generate the random sequence. Explain why the rules for choosing P are important.

5.6     What are the advantages and disadvantages of the various collision resolution strategies?

5.7     Write a program to implement the following strategy for multiplying two sparse polynomials P1, P2 of size m and n respectively. Each polynomial is represented as a linked list with cells consisting of a coefficient, an exponent, and a next pointer (Exercise 3.7). We multiply each term in P1 by a term in P2 for a total of mn operations. One method is to sort these terms and combine like terms, but this requires sorting mn records, which could be expensive, especially in small-memory environments. Alternatively, we could merge terms as they are computed and then sort the result.

a.     Write a program to implement the alternate strategy.

b.     If the output polynomial has about O(m + n) terms, then what is the running time of both methods?

5.8     A spelling checker reads an input file and prints out all words not in some online dictionary. Suppose the dictionary contains 30,000 words and the file is one megabyte, so that the algorithm can make only one pass through the input file. A simple strategy is to read the dictionary into a hash table and look for each input word as it is read. Assuming that an average word is seven characters and that it is possible to store words of length l in l + 1 bytes (so space waste is not much of a consideration), and assuming a closed table, how much space does this require?

5.9     If memory is limited and the entire dictionary cannot be stored in a hash table, we can still get an efficient algorithm that almost always works. We declare an array H_TABLE of bits (initialized to zeros) from 0 to TABLE_SIZE - 1. As we read in a word, we set H_TABLE[hash(word)] = 1. Which of the following is true?

a.     If a word hashes to a location with value 0, the word is not in the dictionary.

b.     If a word hashes to a location with value 1, then the word is in the dictionary.

Suppose we choose TABLE_SIZE = 300,007.

c.     How much memory does this require?

d.     What is the probability of an error in this algorithm?

e.     A typical document might have about three actual misspellings per page of 500 words. Is this algorithm usable?

5.10     *Describe a procedure that avoids initializing a hash table (at the expense of memory).

5.11     Suppose we want to find the first occurrence of a string p1p2. . . pk in a long input string a1a2 . . . an. We can solve this problem by hashing the pattern string, obtaining a hash value hp, and comparing this value with the hash value formed from a1a2 . . . ak,a2a3 . . . ak+1,a3a4 . . . ak+2, and so on until an-k+1an-k+2 . . . an. If we have a match of hash values, we compare the strings character by character to verify the match. We return the position (in a) if the strings actually do match, and we continue in the unlikely event that the match is false.

*a.     Show that if the hash value of aiai+1 . . . ai + k - 1 is known, then the hash value of ai+1ai+2 . . . ai+k can be computed in constant time.

b.     Show that the running time is O(k + n) plus the time spent refuting false matches.

*c.     Show that the expected number of false matches is negligible.

d.     Write a program to implement this algorithm.

**e. Describe an algorithm that runs in O(k + n) worst case time.

**f.     Describe an algorithm that runs in O(n/k) average time.

5.12     A BASIC program consists of a series of statements, each of which is numbered in ascending order. Control is passed by use of a goto or gosub and a statement number. Write a program that reads in a legal BASIC program and renumbers the statements so that the first starts at number f and each statement has a number d higher than the previous statement. You may assume an upper limit of n statements, but the statement numbers in the input might be as large as a 32-bit integer. Your program must run in linear time.

5.13     a.     Implement the word puzzle program using the algorithm described at the end of the chapter.

b.     We can get a big speed increase by storing, in addition to each word w, all of w's prefixes. (If one of w's prefixes is another word in the dictionary, it is stored as a real word). Although this may seem to increase the size of the hash table drastically, it does not, because many words have the same prefixes. When a scan is performed in a particular direction, if the word that is looked up is not even in the hash table as a prefix, then the scan in that direction can be terminated early. Use this idea to write an improved program to solve the word puzzle.

c.     If we are willing to sacrifice the sanctity of the hash table ADT, we can speed up the program in part (b) by noting that if, for example, we have just computed the hash function for "excel," we do not need to compute the hash function for "excels" from scratch. Adjust your hash function so that it can take advantage of its previous calculation.

d.     In Chapter 2, we suggested using binary search. Incorporate the idea of using prefixes into your binary search algorithm. The modification should be simple. Which algorithm is faster?

5.14     Show the result of inserting the keys 10111101, 00000010, 10011011, 10111110, 01111111, 01010001, 10010110, 00001011, 11001111, 10011110, 11011011, 00101011, 01100001, 11110000, 01101111 into an initially empty extendible hashing data structure with m = 4.

5.15     Write a program to implement extendible hashing. If the table is small enough to fit in main memory, how does its performance compare with open and closed hashing?

References

Despite the apparent simplicity of hashing, much of the analysis is quite difficult and there are still many unresolved questions. There are also many interesting theoretical issues, which generally attempt to make it unlikely that the worst-case possibilities of hashing arise.

An early paper on hashing is [17]. A wealth of information on the subject, including an analysis of closed hashing with linear probing can be found in [11]. An excellent survey on the subject is [14]; [15] contains suggestions, and pitfalls, for choosing hash functions. Precise analytic and simulation results for all of the methods described in this chapter can be found in [8].

An analysis of double hashing can be found in [9] and [13]. Yet another collision resolution scheme is coalesced hashing, as described in [18]. Yao [20] has shown that uniform hashing, in which no clustering exists, is optimal with respect to cost of a successful search.

If the input keys are known in advance, then perfect hash functions, which do not allow collisions, exist [2], [7]. Some more complicated hashing schemes, for which the worst case depends not on the particular input but on random numbers chosen by the algorithm, appear in [3] and [4].

Extendible hashing appears in [5], with analysis in [6] and [19].

One method of implementing Exercise 5.5 is described in [16]. Exercise 5.11 (a-d) is from [10]. Part (e) is from [12], and part (f) is from [1].

1.     R. S. Boyer and J. S. Moore, "A Fast String Searching Algorithm," Communications of the ACM 20 (1977), 762-772.

2.     J. L. Carter and M. N. Wegman, "Universal Classes of Hash Functions," Journal of Computer and System Sciences 18 (1979), 143-154.

3.     M. Dietzfelbinger, A. R. Karlin, K. Melhorn, F. Meyer auf der Heide, H. Rohnert, and R. E. Tarjan, Dynamic Perfect Hashing: Upper and Lower Bounds," Proceedings of the Twenty-ninth IEEE Symposium on Foundations of Computer Science (1988), 524-531.

4.     R. J. Enbody and H. C. Du, "Dynamic Hashing Schemes," Computing Surveys 20 (1988), 85-113.

5.     R. Fagin, J. Nievergelt, N. Pippenger, and H. R. Strong, "Extendible Hashing-A Fast Access Method for Dynamic Files," ACM Transactions on Database Systems 4 (1979), 315-344.

6.     P. Flajolet, "On the Performance Evaluation of Extendible Hashing and Trie Searching," Acta Informatica 20 (1983), 345-369.

7.     M. L. Fredman, J. Komlos, and E. Szemeredi, "Storing a Sparse Table with O(1) Worst Case Access Time," Journal of the ACM 31 (1984), 538-544.

8.     G. H. Gonnet and R. Baeza-Yates, Handbook of Algorithms and Data Structures, second edition, Addison-Wesley, Reading, MA, 1991.

9.     L. J. Guibas and E. Szemeredi, "The Analysis of Double Hashing," Journal of Computer and System Sciences 16 (1978), 226-274.

10.     R. M. Karp and M. O. Rabin, "Efficient Randomized Pattern-Matching Algorithms," Aiken Computer Laboratory Report TR-31-81, Harvard University, Cambridge, MA, 1981.

11.     D. E. Knuth, The Art of Computer Programming, Vol 3: Sorting and Searching, second printing, Addison-Wesley, Reading, MA, 1975.

12.     D. E. Knuth, J. H. Morris, V. R. Pratt, "Fast Pattern Matching in Strings," SIAM Journal on Computing 6 (1977), 323-350.

13.     G. Lueker and M. Molodowitch, "More Analysis of Double Hashing," Proceedings of the Twentieth ACM Symposium on Theory of Computing (1988), 354-359.

14.     W. D. Maurer and T. G. Lewis, "Hash Table Methods," Computing Surveys 7 (1975), 5-20.

15.     B. J. McKenzie, R. Harries, and T. Bell, "Selecting a Hashing Algorithm," Software--Practice and Experience 20 (1990), 209-224.

16.     R. Morris, "Scatter Storage Techniques," Communications of the ACM 11 (1968), 38-44.

17.     W. W. Peterson, "Addressing for Random Access Storage," IBM Journal of Research and Development 1 (1957), 130-146.

18.     J. S. Vitter, "Implementations for Coalesced Hashing," Communications of the ACM 25 (1982), 911-926.

19.     A. C. Yao, "A Note on The Analysis of Extendible Hashing," Information Processing Letters 11 (1980), 84-86.

20.     A. C. Yao, "Uniform Hashing is Optimal," Journal of the ACM 32 (1985), 687-693.

Go to Chapter 6         Return to Table of Contents


Structures, Algorithm Analysis: CHAPTER 6: PRIORITY QUEUES (HEAPS)

 Return to Table of Contents

CHAPTER 6: PRIORITY QUEUES (HEAPS)

Although jobs sent to a line printer are generally placed on a queue, this might not always be the best thing to do. For instance, one job might be particularly important, so that it might be desirable to allow that job to be run as soon as the printer is available. Conversely, if, when the printer becomes available, there are several one-page jobs and one hundred-page job, it might be reasonable to make the long job go last, even if it is not the last job submitted. (Unfortunately, most systems do not do this, which can be particularly annoying at times.)

Similarly, in a multiuser environment, the operating system scheduler must decide which of several processes to run. Generally a process is only allowed to run for a fixed period of time. One algorithm uses a queue. Jobs are initially placed at the end of the queue. The scheduler will repeatedly take the first job on the queue, run it until either it finishes or its time limit is up, and place it at the end of the queue if it does not finish. This strategy is generally not appropriate, because very short jobs will seem to take a long time because of the wait involved to run. Generally, it is important that short jobs finish as fast as possible, so these jobs should have preference over jobs that have already been running. Furthermore, some jobs that are not short are still very important and should also have preference.

This particular application seems to require a special kind of queue, known as a priority queue. In this chapter, we will discuss

Efficient implementation of the priority queue ADT.

Uses of priority queues.

Advanced implementations of priority queues.

The data structures we will see are among the most elegant in computer science.

6.1. Model

A priority queue is a data structure that allows at least the following two operations: insert, which does the obvious thing, and delete_min, which finds, returns and removes the minimum element in the heap. The insert operation is the equivalent of enqueue, and delete_min is the priority queue equivalent of the queue's dequeue operation. The delete_min function also alters its input. Current thinking in the software engineering community suggests that this is no longer a good idea. However, we will continue to use this function because of historical reasons--many programmers expect delete_min to operate this way.

Figure 6.1 Basic model of a priority queue

As with most data structures, it is sometimes possible to add other operations, but these are extensions and not part of the basic model depicted in Figure 6.1.

Priority queues have many applications besides operating systems. In Chapter 7, we will see how priority queues are used for external sorting. Priority queues are also important in the implementation of greedy algorithms, which operate by repeatedly finding a minimum; we will see specific examples in Chapters 9 and 10. In this chapter we will see a use of priority queues in discrete event simulation.

6.2. Simple Implementations

There are several obvious ways to implement a priority queue. We could use a simple linked list, performing insertions at the front in O(1) and traversing the list, which requires O(n) time, to delete the minimum. Alternatively, we could insist that the list be always kept sorted; this makes insertions expensive (O(n)) and delete_mins cheap (O(1)). The former is probably the better idea of the two, based on the fact that there are never more delete_mins than insertions.

Another way of implementing priority queues would be to use a binary search tree. This gives an O(log n) average running time for both operations. This is true in spite of the fact that although the insertions are random, the deletions are not. Recall that the only element we ever delete is the minimum. Repeatedly removing a node that is in the left subtree would seem to hurt the balance of the tree by making the right subtree heavy. However, the right subtree is random. In the worst case, where the delete_mins have depleted the left subtree, the right subtree would have at most twice as many elements as it should. This adds only a small constant to its expected depth. Notice that the bound can be made into a worst-case bound by using a balanced tree; this protects one against bad insertion sequences.

Using a search tree could be overkill because it supports a host of operations that are not required. The basic data structure we will use will not require pointers and will support both operations in O(log n) worst-case time. Insertion will actually take constant time on average, and our implementation will allow building a heap of n items in linear time, if no deletions intervene. We will then discuss how to implement heaps to support efficient merging. This additional operation seems to complicate matters a bit and apparently requires the use of pointers.

6.3. Binary Heap

The implementation we will use is known as a binary heap. Its use is so common for priority queue implementations that when the word heap is used without a qualifier, it is generally assumed to be referring to this implementation of the data structure. In this section, we will refer to binary heaps as merely heaps. Like binary search trees, heaps have two properties, namely, a structure property and a heap order property. As with AVL trees, an operation on a heap can destroy one of the properties, so a heap operation must not terminate until all heap properties are in order. This turns out to be simple to do.

6.3.1. Structure Property

6.3.2. Heap Order Property

6.3.3. Basic Heap Operations

6.3.4. Other Heap Operations

6.3.1. Structure Property

A heap is a binary tree that is completely filled, with the possible exception of the bottom level, which is filled from left to right. Such a tree is known as a complete binary tree. Figure 6.2 shows an example.

It is easy to show that a complete binary tree of height h has between 2h and 2h+1 - 1 nodes. This implies that the height of a complete binary tree is [image] log n [image] , which is clearly O(log n).

An important observation is that because a complete binary tree is so regular, it can be represented in an array and no pointers are necessary. The array in Figure 6.3 corresponds to the heap in Figure 6.2.

Figure 6.2 A complete binary tree

Figure 6.3 Array implementation of complete binary tree

For any element in array position i, the left child is in position 2i, the right child is in the cell after the left child (2i + 1), and the parent is in position [image] i/2 [image] . Thus not only are pointers not required, but the operations required to traverse the tree are extremely simple and likely to be very fast on most computers. The only problem with this implementation is that an estimate of the maximum heap size is required in advance, but typically this is not a problem. In the figure above, the limit on the heap size is 13 elements. The array has a position 0; more on this later.

A heap data structure will, then, consist of an array (of whatever type the key is) and integers representing the maximum 2nd current heap size. Figure 6.4 shows a typical priority queue declaration. Notice the similarity to the stack declaration in Figure 3.47. Figure 6.4a creates an empty heap. Line 11 will be explained later.

Throughout this chapter, we shall draw the heaps as trees, with the implication that an actual implementation will use simple arrays.

6.3.2. Heap Order Property

The property that allows operations to be performed quickly is the heap order property. Since we want to be able to find the minimum quickly, it makes sense that the smallest element should be at the root. If we consider that any subtree should also be a heap, then any node should be smaller than all of its descendants.

Applying this logic, we arrive at the heap order property. In a heap, for every node X, the key in the parent of X is smaller than (or equal to) the key in X, with the obvious exception of the root (which has no parent).* In Figure 6.5 the tree on the left is a heap, but the tree on the right is not (the dashed line shows the violation of heap order). As usual, we will assume that the keys are integers, although they could be arbitrarily complex.

*Analogously, we can declare a (max) heap, which enables us to efficiently find and remove the maximum element, by changing the heap order property. Thus, a priority queue can be used to find either a minimum or a maximum, but this needs to be decided ahead of time.

By the heap order property, the minimum element can always be found at the root. Thus, we get the extra operation, find_min, in constant time.

struct heap_struct

{

/* Maximum # that can fit in the heap */

unsigned int max_heap_size;

/* Current # of elements in the heap */

unsigned int size;

element_type *elements;

};

typedef struct heap_struct *PRIORITY_QUEUE;

Figure 6.4 Declaration for priority queue

PRIORITY_QUEUE

create_pq( unsigned int max_elements )

{

PRIORITY_QUEUE H;

/*1*/       if( max_elements < MIN_PQ_SIZE )

/*2*/            error("Priority queue size is too small");

/*3*/       H = (PRIORITY_QUEUE) malloc ( sizeof (struct heap_struct) );

/*4*/       if( H == NULL )

/*5*/            fatal_error("Out of space!!!");

/* Allocate the array + one extra for sentinel */

/*6*/       H->elements = (element_type *) malloc

( ( max_elements+1) * sizeof (element_type) );

/*7*/       if( H->elements == NULL )

/*8*/           fatal_error("Out of space!!!");

/*9*/       H->max_heap_size = max_elements;

/*10*/      H->size = 0;

/*11*/      H->elements[0] = MIN_DATA;

/*12*/      return H;

}

Figure 6.4a

Figure 6.5 Two complete trees (only the left tree is a heap)

6.3.3. Basic Heap Operations

It is easy (both conceptually and practically) to perform the two required operations. All the work involves ensuring that the heap order property is maintained.

Insert

Delete_min

Insert

To insert an element x into the heap, we create a hole in the next available location, since otherwise the tree will not be complete. If x can be placed in the hole without violating heap order, then we do so and are done. Otherwise we slide the element that is in the hole's parent node into the hole, thus bubbling the hole up toward the root. We continue this process until x can be placed in the hole. Figure 6.6 shows that to insert 14, we create a hole in the next available heap location. Inserting 14 in the hole would violate the heap order property, so 31 is slid down into the hole. This strategy is continued in Figure 6.7 until the correct location for 14 is found.

This general strategy is known as a percolate up; the new element is percolated up the heap until the correct location is found. Insertion is easily implemented with the code shown in Figure 6.8.

We could have implemented the percolation in the insert routine by performing repeated swaps until the correct order was established, but a swap requires three assignment statements. If an element is percolated up d levels, the number of assignments performed by the swaps would be 3d. Our method uses d + 1 assignments.

Figure 6.6 Attempt to insert 14: creating the hole, and bubbling the hole up

Figure 6.7 The remaining two steps to insert 14 in previous heap

/* H->element[0] is a sentinel */

void

insert( element_type x, PRIORITY_QUEUE H )

{

unsigned int i;

/*1*/       if( is_full( H ) )

/*2*/            error("Priority queue is full");

else

{

/*3*/            i = ++H->size;

/*4*/            while( H->elements[i/2] > x )

{

/*5*/                 H->elements[i] = H->elements[i/2];

/*6*/                 i /= 2;

}

/*7*/            H->elements[i] = x;

}

}

Figure 6.8 Procedure to insert into a binary heap

If the element to be inserted is the new minimum, it will be pushed all the way to the top. At some point, i will be 1 and we will want to break out of the while loop. We could do this with an explicit test, but we have chosen to put a very small value in position 0 in order to make the while loop terminate. This value must be guaranteed to be smaller than (or equal to) any element in the heap; it is known as a sentinel. This idea is similar to the use of header nodes in linked lists. By adding a dummy piece of information, we avoid a test that is executed once per loop iteration, thus saving some time.

The time to do the insertion could be as much as O (log n), if the element to be inserted is the new minimum and is percolated all the way to the root. On average, the percolation terminates early; it has been shown that 2.607 comparisons are required on average to perform an insert, so the average insert moves an element up 1.607 levels.

Delete_min

Delete_mins are handled in a similar manner as insertions. Finding the minimum is easy; the hard part is removing it. When the minimum is removed, a hole is created at the root. Since the heap now becomes one smaller, it follows that the last element x in the heap must move somewhere in the heap. If x can be placed in the hole, then we are done. This is unlikely, so we slide the smaller of the hole's children into the hole, thus pushing the hole down one level. We repeat this step until x can be placed in the hole. Thus, our action is to place x in its correct spot along a path from the root containing minimum children.

In Figure 6.9 the left figure shows a heap prior to the delete_min. After 13 is removed, we must now try to place 31 in the heap. 31 cannot be placed in the hole, because this would violate heap order. Thus, we place the smaller child (14) in the hole, sliding the hole down one level (see Fig. 6.10). We repeat this again, placing 19 into the hole and creating a new hole one level deeper. We then place 26 in the hole and create a new hole on the bottom level. Finally, we are able to place 31 in the hole (Fig. 6.11). This general strategy is known as a percolate down. We use the same technique as in the insert routine to avoid the use of swaps in this routine.

Figure 6.9 Creation of the hole at the root

Figure 6.10 Next two steps in delete_min

Figure 6.11 Last two steps in delete_min

A frequent implementation error in heaps occurs when there are an even number of elements in the heap, and the one node that has only one child is encountered. You must make sure not to assume that there are always two children, so this usually involves an extra test. In the code, depicted in Figure 6.12, we've done this test at line 8. One extremely tricky solution is always to ensure that your algorithm thinks every node has two children. Do this by placing a sentinel, of value higher than any in the heap, at the spot after the heap ends, at the start of each percolate down when the heap size is even. You should think very carefully before attempting this, and you must put in a prominent comment if you do use this technique.

element_type

delete_min( PRIORITY_QUEUE H )

{

unsigned int i, child;

element_type min_element, last_element;

/*1*/       if( is_empty( H ) )

{

/*2*/            error("Priority queue is empty");

/*3*/            return H->elements[0];

}

/*4*/       min_element = H->elements[1];

/*5*/       last_element = H->elements[H->size--];

/*6*/       for( i=1; i*2 <= H->size; i=child )

{

/* find smaller child */

/*7*/            child = i*2;

/*8*/            if( ( child != H->size ) &&

( H->elements[child+1] < H->elements [child] ) )

/*9*/                 child++;

/* percolate one level */

/*10*/           if( last_element > H->elements[child] )

/*11*/                H->elements[i] = H->elements[child];

else

/*12*/                break;

}

/*13*/      H->elements[i] = last_element;

/*14*/      return min_element;

}

Figure 6.12 Function to perform delete_min in a binary heap

Although this eliminates the need to test for the presence of a right child, you cannot eliminate the requirement that you test when you reach the bottom because this would require a sentinel for every leaf.

The worst-case running time for this operation is O(log n). On average, the element that is placed at the root is percolated almost to the bottom of the heap (which is the level it came from), so the average running time is O (log n).

6.3.4. Other Heap Operations

Notice that although finding the minimum can be performed in constant time, a heap designed to find the minimum element (also known as a (min) heap) is of no help whatsoever in finding the maximum element. In fact, a heap has very little ordering information, so there is no way to find any particular key without a linear scan through the entire heap. To see this, consider the large heap structure (the elements are not shown) in Figure 6.13, where we see that the only information known about the maximum element is that it is at one of the leaves. Half the elements, though, are contained in leaves, so this is practically useless information. For this reason, if it is important to know where elements are, some other data structure, such as a hash table, must be used in addition to the heap. (Recall that the model does not allow looking inside the heap.)

If we assume that the position of every element is known by some other method, then several other operations become cheap. The three operations below all run in logarithmic worst-case time.

Decrease_key

Increase_key

Delete

Build_heap

Decrease_key

The decrease_key(x,  [image], H) operation lowers the value of the key at position x by a positive amount   [image]. Since this might violate the heap order, it must be fixed by a percolate up. This operation could be useful to system administrators: they can make their programs run with highest priority

Figure 6.13 A very large complete binary tree

Increase_key

The increase_key(x,   [image] , H) operation increases the value of the key at position x by a positive amount   [image]. This is done with a percolate down. Many schedulers automatically drop the priority of a process that is consuming excessive CPU time.

Delete

The delete(x, H) operation removes the node at position x from the heap. This is done by first performing decrease_key(x,   [image], H) and then performing delete_min (H). When a process is terminated by a user (instead of finishing normally), it must be removed from the priority queue.

Build_heap

The build_heap(H) operation takes as input n keys and places them into an empty heap. Obviously, this can be done with n successive inserts. Since each insert will take O(1) average and O(log n) worst-case time, the total running time of this algorithm would be O(n) average but O(n log n) worst-case. Since this is a special instruction and there are no other operations intervening, and we already know that the instruction can be performed in linear average time, it is reasonable to expect that with reasonable care a linear time bound can be guaranteed.

The general algorithm is to place the n keys into the tree in any order, maintaining the structure property. Then, if percolate_down(i) percolates down from node i, perform the algorithm in Figure 6.14 to create a heap-ordered tree.

The first tree in Figure 6.15 is the unordered tree. The seven remaining trees in Figures 6.15 through 6.18 show the result of each of the seven percolate downs. Each dashed line corresponds to two comparisons: one to find the smaller child and one to compare the smaller child with the node. Notice that there are only 10 dashed lines in the entire algorithm (there could have been an 11th -- where?) corresponding to 20 comparisons.

for(i=n/2; i>0; i-- )

percolate_down( i );

Figure 6.14 Sketch of build_heap

Figure 6.15 Left: initial heap; right: after percolate_down(7)

Figure 6.16 Left: after percolate_down(6); right: after percolate_down(5)

Figure 6.17 Left: after percolate_down(4); right: after percolate_down(3)

Figure 6.18 Left: after percolate_down(2); right: after percolate_down(1)

To bound the running time of build_heap, we must bound the number of dashed lines. This can be done by computing the sum of the heights of all the nodes in the heap, which is the maximum number of dashed lines. What we would like to show is that this sum is O(n).

THEOREM 6.1.

For the perfect binary tree of height h containing 2h+1 - 1 nodes, the sum of the heights of the nodes is 2h+1 - 1 - (h + 1).

PROOF:

It is easy to see that this tree consists of 1 node at height h, 2 nodes at height h - 1, 22 nodes at height h - 2, and in general 2i nodes at height h - i. The sum of the heights of all the nodes is then

= h +2(h - 1) + 4(h - 2) + 8(h - 3) + 16(h - 4) +. . .+ 2h-1(1)

(6.1)

Multiplying by 2 gives the equation

2S = 2h + 4(h - 1) + 8(h - 2) + 16(h - 3) + . . . + 2h(1)

(6.2)

We subtract these two equations and obtain Equation (6.3). We find that certain terms almost cancel. For instance, we have 2h - 2(h - 1) = 2, 4(h - 1) - 4(h - 2) = 4, and so on. The last term in Equation (6.2), 2h, does not appear in Equation (6.1); thus, it appears in Equation (6.3). The first term in Equation (6.1), h, does not appear in equation (6.2); thus, -h appears in Equation (6.3).

We obtain

S = - h + 2 + 4 + 8 + . . . + 2h-1 + 2h = (2h+1 - 1) - (h + 1)

(6.3)

which proves the theorem.

A complete tree is not a perfect binary tree, but the result we have obtained is an upper bound on the the sum of the heights of the nodes in a complete tree. Since a complete tree has between 2h and 2h+1 nodes, this theorem implies that this sum is O(n), where n is the number of nodes.

Although the result we have obtained is sufficient to show that build_heap is linear, the bound on the sum of the heights is not as strong as possible. For a complete tree with n = 2h nodes, the bound we have obtained is roughly 2n. The sum of the heights can be shown by induction to be n - b(n), where b(n) is the number of 1s in the binary representation of n.

6.4. Applications of Priority Queues

We have already mentioned how priority queues are used in operating systems design. In Chapter 9, we will see how priority queues are used to implement several graph algorithms efficiently. Here we will show how to use priority queues to obtain solutions to two problems.

6.4.1. The Selection Problem

6.4.2. Event Simulation

6.4.1. The Selection Problem

The first problem we will examine is the selection problem from Chapter 1. Recall that the input is a list of n elements, which can be totally ordered, and an integer k. The selection problem is to find the kth largest element.

Two algorithms were given in Chapter 1, but neither is very efficient. The first algorithm, which we shall call Algorithm 1A, is to read the elements into an array and sort them, returning the appropriate element. Assuming a simple sorting algorithm, the running time is O(n2). The alternative algorithm, 1B, is to read k elements into an array and sort them. The smallest of these is in the kth position. We process the remaining elements one by one. As an element arrives, it is compared with kth element in the array. If it is larger, then the kth element is removed, and the new element is placed in the correct place among the remaining k - 1 elements. When the algorithm ends, the element in the kth position is the answer. The running time is O(n   [image]  k) (why?). If k =   [image] n/2  [image] , then both algorithms are O(n2). Notice that for any k, we can solve the symmetric problem of finding the (n - k + 1)th smallest element, so k =   [image] n/2  [image]  is really the hardest case for these algorithms. This also happens to be the most interesting case, since this value of k is known as the median.

We give two algorithms here, both of which run in O(n log n) in the extreme case of k =   [image] n/2  [image] , which is a distinct improvement.

Algorithm 6A

Algorithm 6B

Algorithm 6A

For simplicity, we assume that we are interested in finding the kth smallest element. The algorithm is simple. We read the n elements into an array. We then apply the build_heap algorithm to this array. Finally, we'll perform k delete_min operations. The last element extracted from the heap is our answer. It should be clear that by changing the heap order property, we could solve the original problem of finding the kth largest element.

The correctness of the algorithm should be clear. The worst-case timing is O(n) to construct the heap, if build_heap is used, and O(log n) for each delete_min. Since there are k delete_mins, we obtain a total running time of O(n + k log n). If k = O(n/log n), then the running time is dominated by the build_heap operation and is O(n). For larger values of k, the running time is O(k log n). If k =   [image] n/2  [image] , then the running time is   [image] (n log n).

Notice that if we run this program for k = n and record the values as they leave the heap, we will have essentially sorted the input file in O(n log n) time. In Chapter 7, we will refine this idea to obtain a fast sorting algorithm known as heapsort.

Algorithm 6B

For the second algorithm, we return to the original problem and find the kth largest element. We use the idea from Algorithm 1B. At any point in time we will maintain a set S of the k largest elements. After the first k elements are read, when a new element is read, it is compared with the kth largest element, which we denote by Sk. Notice that Sk is the smallest element in S. If the new element is larger, then it replaces Sk in S. S will then have a new smallest element, which may or may not be the newly added element. At the end of the input, we find the smallest element in S and return it as the answer.

This is essentially the same algorithm described in Chapter 1. Here, however, we will use a heap to implement S. The first k elements are placed into the heap in total time O(k) with a call to build_heap. The time to process each of the remaining elements is O(1), to test if the element goes into S, plus O(log k), to delete Sk and insert the new element if this is necessary. Thus, the total time is O(k + (n - k ) log k ) = O (n log k ) . This algorithm also gives a bound of   [image] (n log n) for finding the median.

In Chapter 7, we will see how to solve this problem in O(n) average time. In Chapter 10, we will see an elegant, albeit impractical, algorithm to solve this problem in O(n) worst-case time.

6.4.2. Event Simulation

In Section 3.4.3, we described an important queuing problem. Recall that we have a system, such as a bank, where customers arrive and wait on a line until one of k tellers is available. Customer arrival is governed by a probability distribution function, as is the service time (the amount of time to be served once a teller is available). We are interested in statistics such as how long on average a customer has to wait or how long the line might be.

With certain probability distributions and values of k, these answers can be computed exactly. However, as k gets larger, the analysis becomes considerably more difficult, so it is appealing to use a computer to simulate the operation of the bank. In this way, the bank officers can determine how many tellers are needed to ensure reasonably smooth service.

A simulation consists of processing events. The two events here are (a) a customer arriving and (b) a customer departing, thus freeing up a teller.

We can use the probability functions to generate an input stream consisting of ordered pairs of arrival time and service time for each customer, sorted by arrival time. We do not need to use the exact time of day. Rather, we can use a quantum unit, which we will refer to as a tick.

One way to do this simulation is to start a simulation clock at zero ticks. We then advance the clock one tick at a time, checking to see if there is an event. If there is, then we process the event(s) and compile statistics. When there are no customers left in the input stream and all the tellers are free, then the simulation is over.

The problem with this simulation strategy is that its running time does not depend on the number of customers or events (there are two events per customer), but instead depends on the number of ticks, which is not really part of the input. To see why this is important, suppose we changed the clock units to milliticks and multiplied all the times in the input by 1,000. The result would be that the simulation would take 1,000 times longer!

The key to avoiding this problem is to advance the clock to the next event time at each stage. This is conceptually easy to do. At any point, the next event that can occur is either (a) the next customer in the input file arrives, or (b) one of the customers at a teller leaves. Since all the times when the events will happen are available, we just need to find the event that happens nearest in the future and process that event.

If the event is a departure, processing includes gathering statistics for the departing customer and checking the line (queue) to see whether there is another customer waiting. If so, we add that customer, process whatever statistics are required, compute the time when that customer will leave, and add that departure to the set of events waiting to happen.

If the event is an arrival, we check for an available teller. If there is none, we place the arrival on the line (queue); otherwise we give the customer a teller, compute the customer's departure time, and add the departure to the set of events waiting to happen.

The waiting line for customers can be implemented as a queue. Since we need to find the event nearest in the future, it is appropriate that the set of departures waiting to happen be organized in a priority queue. The next event is thus the next arrival or next departure (whichever is sooner); both are easily available.

It is then straightforward, although possibly time-consuming, to write the simulation routines. If there are C customers (and thus 2C events) and k tellers, then the running time of the simulation would be O(C log(k + 1))* because computing and processing each event takes O(logH), where H = k + 1 is the size of the heap.

* We use O(C log( k + 1)) instead of O(C log k) to avoid confusion for the k = 1 case.

6.5. d-Heaps

Binary heaps are so simple that they are almost always used when priority queues are needed. A simple generalization is a d-heap, which is exactly like a binary heap except that all nodes have d children (thus, a binary heap is a 2-heap). Figure 6.19 shows a 3-heap.

Notice that a d-heap is much more shallow than a binary heap, improving the running time of inserts to O(logdn). However, the delete_min operation is more expensive, because even though the tree is shallower, the minimum of d children must be found, which takes d - 1 comparisons using a standard algorithm. This raises the time for this operation to O(d logdn). If d is a constant, both running times are, of course, O(log n). Furthermore, although an array can still be used, the multiplications and divisions to find children and parents are now by d, which seriously increases the running time, because we can no longer implement division by a bit shift. d-heaps are interesting in theory, because there are many algorithms where the number of insertions is much greater than the number of delete_mins (and thus a theoretical speedup is possible). They are also of interest when the priority queue is too large to fit entirely in main memory. In this case, a d-heap can be advantageous in much the same way as B-trees.

The most glaring weakness of the heap implementation, aside from the inability to perform finds is that combining two heaps into one is a hard operation. This extra operation is known as a merge. There are quite a few ways of implementing heaps so that the running time of a merge is O(log n). We will now discuss three data structures, of various complexity, that support the merge operation efficiently. We will defer any complicated analysis until Chapter 11.

Figure 6.19 A d-heap

6.6. Leftist Heaps

It seems difficult to design a data structure that efficiently supports merging (that is, processes a merge in o(n) time) and uses only an array, as in a binary heap. The reason for this is that merging would seem to require copying one array into another which would take   [image] (n) time for equal-sized heaps. For this reason, all the advanced data structures that support efficient merging require the use of pointers. In practice, we can expect that this will make all the other operations slower; pointer manipulation is generally more time-consuming than multiplication and division by two.

Like a binary heap, a leftist heap has both a structural property and an ordering property. Indeed, a leftist heap, like virtually all heaps used, has the same heap order property we have already seen. Furthermore, a leftist heap is also a binary tree. The only difference between a leftist heap and a binary heap is that leftist heaps are not perfectly balanced, but actually attempt to be very unbalanced.

6.6.1. Leftist Heap Property

6.6.2. Leftist Heap Operations

6.6.1. Leftist Heap Property

We define the null path length, npl(X) of any node X to be the length of the shortest path from X to a node without two children. Thus, the npl of a node with zero or one child is 0, while npl(NULL) = -1. In the tree in Figure 6.20, the null path lengths are indicated inside the tree nodes.

Notice that the null path length of any node is 1 more than the minimum of the null path lengths of its children. This applies to nodes with less than two children because the null path length of  is -1.

The leftist heap property is that for every node X in the heap, the null path length of the left child is at least as large as that of the right child. This property is satisfied by only one of the trees in Figure 6.20, namely, the tree on the left. This property actually goes out of its way to ensure that the tree is unbalanced, because it clearly biases the tree to get deep towards the left. Indeed, a tree consisting of a long path of left nodes is possible (and actually preferable to facilitate merging); hence the name leftist heap.

Figure 6.20 Null path lengths for two trees; only the left tree is leftist

Because leftist heaps tend to have deep left paths, it follows that the right path ought to be short. Indeed, the right path down a leftist heap is as short as any in the heap. Otherwise, there would be a path that goes through some node X and takes the left child. Then X would violate the leftist property.

THEOREM 6.2.

A leftist tree with r nodes on the right path must have at least 2r - 1 nodes.

PROOF:

The proof is by induction. If r = 1, there must be at least one tree node. Otherwise, suppose that the theorem is true for 1, 2, . . ., r. Consider a leftist tree with r + 1 nodes on the right path. Then the root has a right subtree with r nodes on the right path, and a left subtree with at least r nodes on the right path (otherwise it would not be leftist). Applying the inductive hypothesis to these subtrees yields a minimum of 2r - 1 nodes in each subtree. This plus the root gives at least 2r+1 - 1 nodes in the tree, proving the theorem.

From this theorem, it follows immediately that a leftist tree of n nodes has a right path containing at most   [image] log(n + 1)  [image]  nodes. The general idea for the leftist heap operations is to perform all the work on the right path, which is guaranteed to be short. The only tricky part is that performing inserts and merges on the right path could destroy the leftist heap property. It turns out to be extremely easy to restore the property.

6.6.2. Leftist Heap Operations

The fundamental operation on leftist heaps is merging. Notice that insertion is merely a special case of merging, since we may view an insertion as a merge of a one-node heap with a larger heap. We will first give a simple recursive solution and then show how this might be done nonrecursively. Our input is the two leftist heaps, H1 and H2, in Figure 6.21. You should check that these heaps really are leftist. Notice that the smallest elements are at the roots. In addition to space for the data and left and right pointers, each cell will have an entry that indicates the null path length.

If either of the two heaps is empty, then we can return the other heap. Otherwise, to merge the two heaps, we compare their roots. First, we recursively merge the heap with the larger root with the right subheap of the heap with the smaller root. In our example, this means we recursively merge H2 with the subheap of H1 rooted at 8, obtaining the heap in Figure 6.22.

Since this tree is formed recursively, and we have not yet finished the description of the algorithm, we cannot at this point show how this heap was obtained. However, it is reasonable to assume that the resulting tree is a leftist heap, because it was obtained via a recursive step. This is much like the inductive hypothesis in a proof by induction. Since we can handle the base case (which occurs when one tree is empty), we can assume that the recursive step works as long as we can finish the merge; this is rule 3 of recursion, which we discussed in Chapter 1. We now make this new heap the right child of the root of H1 (see Fig. 6.23).

Figure 6.21 Two leftist heaps H1 and H2

Figure 6.22 Result of merging H2 with H1's right subheap

Figure 6.23 Result of attaching leftist heap of previous figure as H1's right child

Although the resulting heap satisfies the heap order property, it is not leftist because the left subtree of the root has a null path length of 1 while the right subtree has null path length of 2. Thus, the leftist property is violated at the root. However, it is easy to see that the remainder of the tree must be leftist. The right subtree of the root is leftist, because of the recursive step. The left subtree of the root has not been changed, so it too must still be leftist. Thus, we only need to fix the root. We can make the entire tree leftist by merely swapping the root's left and right children (Fig. 6.24) and updating the null path length -- the new null path length is 1 plus the null path length of the new right child -- completing the merge. Notice that if the null path length is not updated, then all null path lengths will be 0, and the heap will not be leftist but merely random. In this case, the algorithm will work, but the time bound we will claim will no longer be valid.

The description of the algorithm translates directly into code. The type definition (Fig. 6.25) is the same as the binary tree, except that it is augmented with the npl (null path length) field. We have seen in Chapter 4 that when an element is inserted into an empty binary tree, the pointer to the root will need to change. The easiest way to implement this is to have the insertion routine return a pointer to the new tree. Unfortunately, this will make the leftist heap insert incompatible with the binary heap insert (which does not return anything). The last line in Figure 6.25 represents one way out of this quandary. The leftist heap insertion routine which returns the new tree will be called insert1; the insert macro will make an insertion compatible with binary heaps. Using macros this way may not be the best or safest course, but the alternative, declaring a PRIORITY QUEUE as a pointer to a tree_ptr will flood the code with extra asterisks.

Because insert is a macro and is textually substituted by the preprocessor, any routine that calls insert must be able to see the macro definition. Figure 6.25 would typically be a header file, so placing the macro declaration there is the only reasonable course. As we will see later. delete_min also needs to be written as a macro.

Figure 6.24 Result of swapping children of H1's root

typedef struct tree_node *tree_ptr;

struct tree_node

{

element_type element;

tree_ptr left;

tree_ptr right;

unsigned int npl;

};

typedef tree_ptr PRIORITY_QUEUE;

#define insert( x, H ) ( H = insert1( (x), H ) )

Figure 6.25 Leftist heap type declarations

The routine to merge (Fig. 6.26) is a driver designed to remove special cases and ensure that H1 has the smaller root. The actual merging is performed in merge1 (Fig. 6.27).

The time to perform the merge is proportional to the sum of the length of the right paths, because constant work is performed at each node visited during the recursive calls. Thus we obtain an O(log n) time bound to merge two leftist heaps. We can also perform this operation nonrecursively by essentially performing two passes. In the first pass, we create a new tree by merging the right paths of both heaps. To do this, we arrange the nodes on the right paths of H1 and H2 in sorted order, keeping their respective left children. In our example, the new right path is 3, 6, 7, 8, 18 and the resulting tree is shown in Figure 6.28. A second pass is made up the heap, and child swaps are performed at nodes that violate the leftist heap property. In Figure 6.28, there is a swap at nodes 7 and 3, and the same tree as before is obtained. The nonrecursive version is simpler to visualize but harder to code. We leave it to the reader to show that the recursive and nonrecursive procedures do the same thing.

PRIORITY_QUEUE

merge( PRIORITY_QUEUE H1, PRIORITY_QUEUE H2 )

{

/*1*/       if( H1 == NULL )

/*2*/            return H2;

/*3*/       if( H2 == NULL )

/*4*/            return H1;

/*5*/       if( H1->element < H2->element )

/*6*/            return merge1( H1, H2 );

else

/*7*/            return merge1( H2, H1 );

}

Figure 6.26 Driving routine for merging leftist heaps

/* For merge1, H1 has smaller root, H1 and H2 are not NULL */

PRIORITY_QUEUE

merge1( PRIORITY_QUEUE H1, PRIORITY_QUEUE H2 )

{

/*1*/       if( H1->left == NULL )  /* single node */

/*2*/            H1->left = H2;     /* H1->right is already NULL,

H1->npl is already 0*/

else

{

/*3*/            H1->right = merge( H1->right, H2 );

/*4*/            if( H1->left->npl < H1->right->npl )

/*5*/                 swap_children( H1 );

/*6*/            H1->npl = H1->right->npl + 1;

}

/*7*/       return H1;

}

Figure 6.27 Actual routine to merge leftist heaps

Figure 6.28 Result of merging right paths of H1 and H2

As mentioned above, we can carry out insertions by making the item to be inserted a one-node heap and performing a merge. To perform a delete_min, we merely destroy the root, creating two heaps, which can then be merged. Thus, the time to perform a delete_min is O(logn). These two routines are coded in Figure 6.29 and Figure 6.30. Delete_min can be written as a macro that calls delete_min1 and find_min. This is left as an exercise to the reader.

The call to free on line 4 of Figure 6.30 might look chancy, but it is actually correct. The call does not destroy the variable H; rather, it indicates that the cell to which it points can be used. That cell is placed on the freelist. H, which is a pointer, is then set to point somewhere else by line 5. Also, notice how the headings for these routines can be made identical to those for the binary heap implementation. Either priority queue package could be used, and the implementation would be completely transparent to the calling routines.

Finally, we can build a leftist heap in O(n) time by building a binary heap (obviously using a pointer implementation). Although a binary heap is clearly leftist, this is not necessarily the best solution, because the heap we obtain is the worst possible leftist heap. Furthermore, traversing the tree in reverse-level order is not as easy with pointers. The build_heap effect can be obtained by recursively building the left and right subtrees and then percolating the root down. The exercises contain an alternative solution.

PRIORITY_QUEUE

insert1( element_type x, PRIORITY_QUEUE H )

{

tree_ptr single_node;

/*1*/       single_node = (tree_ptr) malloc( sizeof (struct tree_node) );

/*2*/       if( single_node == NULL )

/*3*/           fatal_error("Out of space!!!");

else

{

/*4*/           single_node->element = x; single_node->npl = 0;

/*5*/           single_node->left = single_node->right = NULL;

/*6*/           H = merge( single_node, H );

}

/*7*/       return H;

}

Figure 6.29 Insertion routine for leftist heaps

/* Delete_min1 returns the new tree; */

/* to get the minimum use find_min */

/* This is for convenience. */

PRIORITY_QUEUE

delete_min1( PRIORITY_QUEUE H )

{

PRIORITY_QUEUE left_heap, right_heap;

/*1*/       left_heap = H->left;

/*2*/       right_heap = H->right;

/*3*/       free( H );

/*4*/       return merge( left_heap, right_heap );

}

Figure 6.30 Delete_min routine for leftist heaps

6.7. Skew Heaps

A skew heap is a self-adjusting version of a leftist heap that is incredibly simple to implement. The relationship of skew heaps to leftist heaps is analogous to the relation between splay trees and AVL trees. Skew heaps are binary trees with heap order, but there is no structural constraint on these trees. Unlike leftist heaps, no information is maintained about the null path length of any node. The right path of a skew heap can be arbitrarily long at any time, so the worst-case running time of all operations is O(n). However, as with splay trees, it can be shown (see Chapter 11) that for any m consecutive operations, the total worst-case running time is O(m log n). Thus, skew heaps have O(log n) amortized cost per operation.

As with leftist heaps, the fundamental operation on skew heaps is merging. The merge routine is once again recursive, and we perform the exact same operations as before, with one exception. The difference is that for leftist heaps, we check to see whether the left and right children satisfy the leftist heap order property and swap them if they do not. For skew heaps, the swap is unconditional -- we always do it, with the one exception that the smallest of all the nodes on the right paths does not have its children swapped. This one exception is what happens in the natural recursive implementation, so it is not really a special case at all. Furthermore, it is not necessary to prove the bounds, but since this node is guaranteed not to have a right child, it would be silly to perform the swap and give it one. (In our example, there are no children of this node, so we do not worry about it.) Again, suppose our input is the same two heaps as before, Figure 6.31.

If we recursively merge H2 with the subheap of H1 rooted at 8, we will get the heap in Figure 6.32.

Again, this is done recursively, so by the third rule of recursion (Section 1.3) we need not worry about how it was obtained. This heap happens to be leftist, but there is no guarantee that this is always the case. We make this heap the new left child of H1 and the old left child of H1 becomes the new right child (see Fig. 6.33).

Figure 6.31 Two skew heaps H1 and H2

Figure 6.32 Result of merging H2 with H1's right subheap

Figure 6.33 Result of merging skew heaps H1 and H2

The entire tree is leftist, but it is easy to see that that is not always true: Inserting 15 into this new heap would destroy the leftist property.

We can perform all operations nonrecursively, as with leftist heaps, by merging the right paths and swapping left and right children for every node on the right path, with the exception of the last. After a few examples, it becomes clear that since all but the last node on the right path have their children swapped, the net effect is that this becomes the new left path (see the preceding example to convince yourself). This makes it very easy to merge two skew heaps visually.

The implementation of skew heaps is left as a (trivial) exercise. Skew heaps have the advantage that no extra space is required to maintain path lengths and no tests are required to determine when to swap children. It is an open problem to determine precisely the expected right path length of both leftist and skew heaps (the latter is undoubtedly more difficult). Such a comparison would make it easier to determine whether the slight loss of balance information is compensated by the lack of testing.

6.8. Binomial Queues

Although both leftist and skew heaps support merging, insertion, and delete_min all effectively in O(log n) time per operation, there is room for improvement because we know that binary heaps support insertion in constant average time per operation. Binomial queues support all three operations in O(log n) worst-case time per operation, but insertions take constant time on average.

< P>

6.8.1. Binomial Queue Structure

Binomial queues differ from all the priority queue implementations that we have seen in that a binomial queue is not a heap-ordered tree but rather a collection of heap-ordered trees, known as a forest. Each of the heap-ordered trees are of a constrained form known as a binomial tree (the name will be obvious later). There is at most one binomial tree of every height. A binomial tree of height 0 is a one-node tree; a binomial tree, Bk, of height k is formed by attaching a binomial tree, Bk-1, to the root of another binomial tree, Bk-1. Figure 6.34 shows binomial trees B0, B1, B2, B3, and B4.

It is probably obvious from the diagram that a binomial tree, Bk consists of a root with children B0, B1, . . ., Bk-1. Binomial trees of height k have exactly 2k nodes, and the number of nodes at depth d is the binomial coefficient   [image]. If we impose heap order on the binomial trees and allow at most one binomial tree of any height, we can uniquely represent a priority queue of any size by a collection of binomial trees. For instance, a priority queue of size 13 could be represented by the forest B3, B2, B0. We might write this representation as 1101, which not only represents 13 in binary but also represents the fact that B3, B2 and B0 are present in the representation and B1 is not.

As an example, a priority queue of six elements could be represented as in Figure 6.35.

Figure 6.34 Binomial trees B0, B1, B2, B3, and B4

Figure 6.35 Binomial queue H1 with six elements

6.8.2. Binomial Queue Operations

The minimum element can then be found by scanning the roots of all the trees. Since there are at most log n different trees, the minimum can be found in O(log n) time. Alternatively, we can maintain knowledge of the minimum and perform the operation in O(1) time, if we remember to update the minimum when it changes during other operations.

Merging two binomial queues is a conceptually easy operation, which we will describe by example. Consider the two binomial queues, H1 and H2 with six and seven elements, respectively, pictured in Figure 6.36.

The merge is performed by essentially adding the two queues together. Let H3 be the new binomial queue. Since H1 has no binomial tree of height 0 and H2 does, we can just use the binomial tree of height 0 in H2 as part of H3. Next, we add binomial trees of height 1. Since both H1 and H2 have binomial trees of height 1, we merge them by making the larger root a subtree of the smaller, creating a binomial tree of height 2, shown in Figure 6.37. Thus, H3 will not have a binomial tree of height 1. There are now three binomial trees of height 2, namely, the original trees of H1 and H2 plus the tree formed by the previous step. We keep one binomial tree of height 2 in H3 and merge the other two, creating a binomial tree of height 3. Since H1 and H2 have no trees of height 3, this tree becomes part of H3 and we are finished. The resulting binomial queue is shown in Figure 6.38.

Figure 6.36 Two binomial queues H1 and H2

Figure 6.37 Merge of the two B1 trees in H1 and H2

Figure 6.38 Binomial queue H3: the result of merging H1 and H2

Since merging two binomial trees takes constant time with almost any reasonable implementation, and there are O(log n) binomial trees, the merge takes O(log n) time in the worst case. To make this operation efficient, we need to keep the trees in the binomial queue sorted by height, which is certainly a simple thing to do.

Insertion is just a special case of merging, since we merely create a one-node tree and perform a merge. The worst-case time of this operation is likewise O(log n). More precisely, if the priority queue into which the element is being inserted has the property that the smallest nonexistent binomial tree is Bi, the running time is proportional to i + 1. For example, H3 (Fig. 6.38) is missing a binomial tree of height 1, so the insertion will terminate in two steps. Since each tree in a binomial queue is present with probability   [image], it follows that we expect an insertion to terminate in two steps, so the average time is constant. Furthermore, an easy analysis will show that performing n inserts on an initially empty binomial queue will take O(n) worst-case time. Indeed, it is possible to do this operation using only n - 1 comparisons; we leave this as an exercise.

As an example, we show in Figures 6.39 through 6.45 the binomial queues that are formed by inserting 1 through 7 in order. Inserting 4 shows off a bad case. We merge 4 with B0, obtaining a new tree of height 1. We then merge this tree with B1, obtaining a tree of height 2, which is the new priority queue. We count this as three steps (two tree merges plus the stopping case). The next insertion after 7 is inserted is another bad case and would require three tree merges.

A delete_min can be performed by first finding the binomial tree with the smallest root. Let this tree be Bk, and let the original priority queue be H. We remove the binomial tree Bk from the forest of trees in H, forming the new binomial queue H'. We also remove the root of Bk, creating binomial trees B0, B1, . . . , Bk - l, which collectively form priority queue H''. We finish the operation by merging H' and H''.

As an example, suppose we perform a delete_min on H3, which is shown again in Figure 6.46. The minimum root is 12, so we obtain the two priority queues H' and H'' in Figure 6.47 and Figure 6.48. The binomial queue that results from merging H' and H'' is the final answer and is shown in Figure 6.49.

For the analysis, note first that the delete_min operation breaks the original binomial queue into two. It takes O (log n) time to find the tree containing the minimum element and to create the queues H' and H''. Merging these two queues takes O (log n) time, so the entire delete_min operation takes O (log n) time.

6.8.3. Implementation of Binomial Queues

The delete_min operation requires the ability to find all the subtrees of the root quickly, so the standard representation of general trees is required: The children of each node are kept in a linked list, and each node has a pointer to its first child (if any). This operation also requires that the children be ordered by the size of their subtrees, in essentially the same way as we have been drawing them. The reason for this is that when a delete_min is performed, the children will form the binomial queue H''.

We also need to make sure that it is easy to merge two trees. Two binomial trees can be merged only if they have the same size, so if this is to be done efficiently, the size of the tree must be stored in the root. Also, when two trees are merged, one of the trees is added as a child to the other. Since this new tree will be the last child (as it will be the largest subtree), we must be able to keep track of the last child of each node efficiently. Only then will we be able to merge two binomial trees, and thus two binomial queues, efficiently. One way to do this is to use a circular doubly linked list. In this list, the left sibling of the first child will be the last child. The right sibling of the last child could be defined as the first child, but it might be easier just to define it as . This makes it easy to test whether the child we are pointing to is the last.

To summarize, then, each node in a binomial tree will contain the data, first child, left and right sibling, and the number of children (which we will call the rank). Since a binomial queue is just a list of trees, we can use a pointer to the smallest tree as the reference to the data structure.

Figure 6.51 shows how the binomial queue in Figure 6.50 is represented. Figure 6.52 shows the type declarations for a node in the binomial tree.

In order to merge two binomial queues, we need a routine to merge two binomial trees of the same size. Figure 6.53 shows how the pointers change when two binomial trees are merged. First, the root of the new tree gains a child, so we must update its rank. We then need to change several pointers in order to splice one tree into the list of children of the root of the other tree. The code to do this is simple and shown in Figure 6.54.

Figure 6.39 After 1 is inserted

Figure 6.40 After 2 is inserted

Figure 6.41 After 3 is inserted

Figure 6.42 After 4 is inserted

Figure 6.43 After 5 is inserted

Figure 6.44 After 6 is inserted

Figure 6.45 After 7 is inserted

Figure 6.46 Binomial queue H3

Figure 6.47 Binomial queue H', containing all the binomial trees in H3 except B3

Figure 6.48 Binomial queue H'': B3 with 12 removed

Figure 6.49 Result of delete_min(H3)

Figure 6.50 Binomial queue H3 drawn as a forest

Figure 6.51 Representation of binomial queue H3

typedef struct tree_node *tree_ptr;

struct tree_node

{

element_type element;

tree_ptr l_sib;

tree_ptr r_sib;

tree_ptr f_child;

unsigned int rank;

};

typedef tree_ptr PRIORITY_QUEUE;

Figure 6.52 Binomial queue type declarations

Figure 6.53 Merging two binomial trees

The routine to merge two binomial queues is relatively simple. We use recursion to keep the code size small; a nonrecursive procedure will give better performance, and is left as Exercise 6.32. We assume the macro extract(T, H), which removes the first tree from the priority queue H, placing the tree in T. Suppose the smallest binomial tree is contained in H1, but not in H2. Then, to merge H1, we remove the first tree in H1 and add to it the result of merging the rest of H1 with H2. If the smallest tree is contained in both Hl and H2, then we remove both trees and merge them, obtaining a one-tree binomial queue H'. We then merge the remainder of Hl and H2, and merge this result with H'. This strategy is implemented in Figure 6.55. The other routines are straightforward implementations, which we leave as exercises.

/* Merge two equal-sized binomial trees */

tree_ptr

merge_tree( tree_ptr T1, tree_ptr T2 )

{

if( T1->element > T2->element )

return merge_tree( T2, T1 );

if( T1->rank++ == 0 )

T1->f_child = T2;

else

{

T2->l_sib = T1->f_child->l_sib;

T2->l_sib->r_sib = T2;

T1->f_child->l_sib = T2;

}

return T1;

}

Figure 6.54 Routine to merge two equal-sized binomial trees

We can extend binomial queues to support some of the nonstandard operations that binary heaps allow, such as decrease_key and delete, when the position of the affected element is known. A decrease_key is a percolate up, which can be performed in O(log n) time if we add a field to each node pointing to its parent. An arbitrary delete can be performed by a combination of decrease_key and delete_min in O(log n) time.

Summary

In this chapter we have seen various implementations and uses of the priority queue ADT. The standard binary heap implementation is elegant because of its simplicity and speed. It requires no pointers and only a constant amount of extra space, yet supports the priority queue operations efficiently.

We considered the additional merge operation and developed three implementations, each of which is unique in its own way. The leftist heap is a wonderful example of the power of recursion. The skew heap represents a remarkable data structure because of the lack of balance criteria. Its analysis, which we will perform in Chapter 11, is interesting in its own right. The binomial queue shows how a simple idea can be used to achieve a good time bound.

We have also seen several uses of priority queues, ranging from operating systems scheduling to simulation. We will see their use again in Chapters 7, 9, 10.

PRIORITY_QUEUE

merge( PRIORITY_QUEUE H1, PRIORITY_QUEUE H2 )

{

PRIORITY_QUEUE H3;

tree_ptr T1, T2, T3;

/*1*/       if( H1 == NULL )

/*2*/            return H2;

/*3*/       if( H2 == NULL )

/*4*/            return H1;

/*5*/       if( H1->rank < H2->rank )

{

/*6*/            T1 = extract( H1 ); /* extract is a macro */

/*7*/            H3 = merge( H1, H2 );

/*8*/            T1->l_sib = H3->l_sib;

/*9*/            H3->l_sib->r_sib = NULL;

/*10*/           T1->r_sib = H3; H3->l_sib = T1;

/*11*/           return T1;

{

/*12*/      if( H2->rank < H1->rank )

/*13*/           return merge( H2, H1 );

/* Otherwise, first two trees have same rank */

/*14*/      T1 = extract( H1 ); T2 = extract( H2 );

/*15*/      H3 = merge( H1, H2 );

/*16*/      T3 = merge_tree( T1, T2 );

/*17*/      return merge( T3, H3 );

}

Figure 6.55 Routine to merge two priority queues

Exercises

6.1     Suppose that we replace the delete_min function with find_min. Can both insert and find_min be implemented in constant time?

6.2     a.     Show the result of inserting 10, 12, 1, 14, 6, 5, 8, 15, 3, 9, 7, 4, 11, 13, and 2, one at a time, into an initially empty binary heap.

b.     Show the result of using the linear-time algorithm to build a binary heap using the same input.

6.3     Show the result of performing three delete_min operations in the heap of the previous exercise.

6.4     Write the routines to do a percolate up and a percolate down in a binary heap.

6.5     Write and test a program that performs the operations insert, delete_min, build_heap, find_min, decrease_key, delete, and increase_key in a binary heap.

6.6     How many nodes are in the large heap in Figure 6.13?

6.7     a.     Prove that for binary heaps, build_heap does at most 2n - 2 comparisons between elements.

b.     Show that a heap of 8 elements can be constructed in 8 comparisons between heap elements.

**c. Give an algorithm to build a binary heap in 13/8n + O(log n) element comparisons.

** 6.8     Show that the expected depth of the kth smallest element in a large complete heap (you may assume n = 2k - 1) is bounded by log k.

6.9     *     a.     Give an algorithm to find all nodes less than some value, x, in a binary heap. Your algorithm should run in O(K), where K is the number of nodes output.

b.     Does your algorithm extend to any of the other heap structures discussed in this chapter?

**6.10     Propose an algorithm to insert m nodes into a binary heap on n elements in O(m + log n) time. Prove your time bound.

6.11     Write a program to take n elements and do the following:

a.     Insert them into a heap one by one,

b.     Build a heap in linear time.

Compare the running time of both algorithms for sorted, reverse-ordered, and random inputs.

6.12     Each delete_min operation uses 2 log n comparisons in the worst case.

*a.     Propose a scheme so that the delete_min operation uses only log n + log log n + O(1) comparisons between elements. This need not imply less data movement.

**b. Extend your scheme in part (a) so that only log n + log log log n + O(1) comparisons are performed.

**c. How far can you take this idea?

d.     Do the savings in comparisons compensate for the increased complexity of your algorithm?

6.13     If a d-heap is stored as an array, for an entry located in position i, where are the parents and children?

6.14     Suppose we need to perform m percolate_ups and n delete_mins on a d-heap that initially has n elements.

a.     What is the total running time of all operations in terms of m, n, and d?

b.     If d = 2, what is the running time of all heap operations?

c.     If d =   [image] (n), what is the total running time?

*d.     What choice of d minimizes the total running time?

6.15     A min-max heap is a data structure that supports both delete_min and delete_max in O(log n) per operation. The structure is identical to a binary heap, but the heap order property is that for any node, X, at even depth, the key stored at X is smaller than the parent but larger than the grandparent (where this makes sense), and for any node X at odd depth, the key stored at X is larger than the parent but smaller than the grandparent. See Figure 6.56.

a.     How do we find the minimum and maximum elements?

*b.     Give an algorithm to insert a new node into the min-max heap.

*c.     Give an algorithm to perform delete_min and delete_max.

*d.     Can you build a min-max heap in linear time?

*e.     Suppose we would like to support delete_min, delete_max, and merge. Propose a data structure to support all operations in O(log n) time.

6.16     Merge the two leftist heaps in Figure 6.57.

6.17     Show the result of inserting keys 1 to 15 in order into an initially empty leftist heap.

6.18     Prove or disprove: A perfectly balanced tree forms if keys 1 to 2k - 1 are inserted in order into an initially empty leftist heap.

6.19     Give an example of input which generates the best leftist heap.

Figure 6.56 Min-max heap

Figure 6.57

6.20     a.     Can leftist heaps efficiently support decrease_key?

b.     What changes, if any (if possible), are required to do this?

6.21     One way to delete nodes from a known position in a leftist heap is to use a lazy strategy. To delete a node, merely mark it deleted. When a find_min or delete_min is performed, there is a potential problem if the root is marked deleted, since then the node has to be actually deleted and the real minimum needs to be found, which may involve deleting other marked nodes. In this strategy, deletes cost one unit, but the cost of a delete_min or find_min depends on the number of nodes that are marked deleted. Suppose that after a delete_min or find_min there are k fewer marked nodes than before the operation.

*a.     Show how to perform the delete_min in O(k log n) time.

**b. Propose an implementation, with an analysis to show that the time to perform the delete_min is O(k log(2n/k)).

6.22     We can perform build_heap in linear time for leftist heaps by considering each element as a one-node leftist heap, placing all these heaps on a queue, and performing the following step: Until only one heap is on the queue, dequeue two heaps, merge them, and enqueue the result.

a.     Prove that this algorithm is O(n) in the worst case.

b.     Why might this algorithm be preferable to the algorithm described in the text?

6.23     Merge the two skew heaps in Figure 6.57.

6.24     Show the result of inserting keys 1 to 15 in order into a skew heap.

6.25     Prove or disprove: A perfectly balanced tree forms if the keys 1 to 2k - 1 are inserted in order into an initially empty skew heap.

6.26     A skew heap of n elements can be built using the standard binary heap algorithm. Can we use the same merging strategy described in Exercise 6.22 for skew heaps to get an O(n) running time?

6.27     Prove that a binomial tree Bk has binomial trees B0, B1, . . . , Bk-1 as children of the root.

6.28     Prove that a binomial tree of height k has   [image]  nodes at depth d.

6.29     Merge the two binomial queues in Figure 6.58.

6.30     a.     Show that n inserts into an initially empty binomial queue takes O(n) time in the worst case.

b.     Give an algorithm to build a binomial queue of n elements, using at most n - 1 comparisons between elements.

*6.31     Propose an algorithm to insert m nodes into a binomial queue of n elements in O(m + log n) worst-case time. Prove your bound.

6.32     Write nonrecursive routines to perform merge, insert, and delete_min using binomial queues.

**6.33     Suppose we extend binomial queues to allow at most two trees of the same height per structure. Can we obtain O(1) worst-case time for insertion while retaining O(log n) for the other operations?

Figure 6.58

6.34     Suppose you have a number of boxes, each of which can hold total weight C and items i1, i2, i3, . . . , in, which weigh w1, w2, w3, . . . , wn. The object is to pack all the items without placing more weight in any box than its capacity and using as few boxes as possible. For instance, if C = 5, and the items have weights 2, 2, 3, 3, then we can solve the problem with two boxes. In general, this problem is very hard and no efficient solution is known. Write programs to implement efficiently the following approximation strategies:

*a.     Place the weight in the first box for which it fits (creating a new box if there is no box with enough room). (This strategy and all that follow would give three boxes, which is suboptimal.)

b.     Place the weight in the box with the most room for it.

*c.     Place the weight in the most filled box that can accept it without overflowing.

**d. Are any of these strategies enhanced by presorting the items by weight?

6.35     Suppose we want to add the decrease_all_keys(  [image]) operation to the heap repertoire. The result of this operation is that all keys in the heap have their value decreased by an amount   [image]. For the heap implementation of your choice, explain the necessary modifications so that all other operations retain their running times and decrease_all_keys runs in O(1).

6.36     Which of the two selection algorithms has the better time bound?

References

The binary heap was first described in [21]. The linear-time algorithm for its construction is from [9].

The first description of d -heaps was in [14]. Leftist heaps were invented by Crane [7] and described in Knuth [15]. Skew heaps were developed by Sleator and Tarjan [17]. Binomial queues were invented by Vuillemin [20]; Brown provided a detailed analysis and empirical study showing that they perform well in practice [2], if carefully implemented.

Exercise 6.7 (b-c) is taken from [12]. A method for constructing binary heaps that uses about 1.52n comparisons on average is described in [16]. Lazy deletion in leftist heaps (Exercise 6.21) is from [6]. A solution to Exercise 6.33 can be found in [5].

Min-max heaps (Exercise 6.15) were originally described in [1]. More efficient implementation of the operations is given in [13] and [18]. An alternate representation for double ended priority queues is the deap. Details can be found in [3] and [4].

A theoretically interesting priority queue representation is the Fibonacci heap [11], which we will describe in Chapter 11. The Fibonacci heap allows all operations to be performed in O(1) amortized time, except for deletions, which are O(log n). Relaxed heaps [8] achieve identical bounds in the worst case. Another interesting implementation is the pairing heap [10]. Finally, a priority queue that works when the data consists of small integers is described in [19].

1.     M. D. Atkinson, J. R. Sack, N. Santoro, and T. Strothotte, "Min-Max Heaps and Generalized Priority Queues," Communications of the ACM 29 (1986), 996-1000.

2.     M. R. Brown, "Implementation and Analysis of Binomial Queue Algorithms," SIAM Journal on Computing 7 (1978), 298-319.

3.     S. Carlsson, "The Deap--A Double-ended Heap to Implement Double-ended Priority Queues," Information Processing Letters 26 (1987), 33-36.

4.     S. Carlsson, J. Chen, and T. Strothotte, "A Note on the Construction of the Data Structure 'Deap'," Information Processing Letters 31 (1989), 315-317.

5.     S. Carlsson, J. I. Munro, and P. V. Poblete, "An Implicit Binomial Queue with Constant Insertion Time," Proceedings of First Scandinavian Workshop on Algorithm Theory, 1988, 1-13.

6.     D. Cheriton and R. E. Tarjan, "Finding Minimum Spanning Trees," SIAM Journal on Computing 5 (1976), 724-742.

7.     C. A. Crane, "Linear Lists and Priority Queues as Balanced Binary Trees," Technical Report STAN-CS-72-259, Computer Science Department, Stanford University, Stanford, CA, 1972.

8.     J. R. Driscoll, H. N. Gabow, R. Shrairman, and R. E. Tarjan, "Relaxed Heaps: An Alternative to Fibonacci Heaps with Applications to Parallel Computation," Communications of the ACM 31 (1988), 1343-1354.

9.     R. W. Floyd, "Algorithm 245: Treesort 3:", Communications of the ACM 7 (1964), 701.

10.     M. L. Fredman, R. Sedgewick, D. D. Sleator, and R. E. Tarjan, "The Pairing Heap: A New Form of Self-adjusting Heap," Algorithmica 1 (1986), 111-129.

11.     M. L. Fredman and R. E. Tarjan, "Fibonacci Heaps and Their Uses in Improved Network Optimization Algorithms," Journal of the ACM 34 (1987), 596-615.

12.     G. H. Gonnet and J. I. Munro, "Heaps on Heaps," SIAM Journal on Computing 15 (1986), 964-971.

13.     A. Hasham and J. R. Sack, "Bounds for Min-max Heaps," BIT 27 (1987), 315-323.

14.     D. B. Johnson, "Priority Queues with Update and Finding Minimum Spanning Trees," Information Processing Letters 4 (1975), 53-57.

15.     D. E. Knuth, The Art of Computer Programming, Vol 3: Sorting and Searching, second printing, Addison-Wesley, Reading, MA, 1975.

16.     C. J. H. McDiarmid and B. A. Reed, "Building Heaps Fast," Journal of Algorithms 10 (1989), 352-365.

17.     D. D. Sleator and R. E. Tarjan, "Self-adjusting Heaps," SIAM Journal on Computing 15 (1986), 52-69.

18.     T. Strothotte, P. Eriksson, and S. Vallner, "A Note on Constructing Min-max Heaps," BIT 29 (1989), 251-256.

19.     P. van Emde Boas, R. Kaas, E. Zijlstra, "Design and Implementation of an Efficient Priority Queue," Mathematical Systems Theory 10 (1977), 99-127.

20.     J. Vuillemin, "A Data Structure for Manipulating Priority Queues," Communications of the ACM 21 (1978), 309-314.

21.     J. W. J. Williams, "Algorithm 232: Heapsort," Communications of the ACM 7 (1964), 347-348.

Go to Chapter 7       Return to Table of Contents


Structures, Algorithm Analysis: CHAPTER 7: SORTING

 Return to Table of Contents

CHAPTER 7: SORTING

In this chapter we discuss the problem of sorting an array of elements. To simplify matters, we will assume in our examples that the array contains only integers, although, obviously, more complicated structures are possible. For most of this chapter, we will also assume that the entire sort can be done in main memory, so that the number of elements is relatively small (less than a million). Sorts that cannot be performed in main memory and must be done on disk or tape are also quite important. This type of sorting, known as external sorting, will be discussed at the end of the chapter.

Our investigation of internal sorting will show that

There are several easy algorithms to sort in O(n2), such as insertion sort.

There is an algorithm, Shellsort, that is very simple to code, runs in o(n2), and is efficient in practice.

There are slightly more complicated O(n log n) sorting algorithms.

Any general-purpose sorting algorithm requires [image] (n log n) comparisons.

The rest of this chapter will describe and analyze the various sorting algorithms. These algorithms contain interesting and important ideas for code optimization as well as algorithm design. Sorting is also an example where the analysis can be precisely performed. Be forewarned that where appropriate, we will do as much analysis as possible.

7.1. Preliminaries

The algorithms we describe will all be exchangeable. Each will be passed an array containing the elements and an integer containing the number of elements.

We will assume that n, the number of elements passed to our sorting routines, has already been checked and is legal. For some of the sorting routines, it will be convenient to place a sentinel in position 0, so we will assume that the array ranges from 0 to n. The actual data will start at position 1 for all the sorts.

We will also assume the existence of the "<" and ">" operators, which can be used to place a consistent ordering on the input. Besides the assignment operator, these are the only operations allowed on the input data. Sorting under these conditions is known as comparison-based sorting.

7.2. Insertion Sort

7.2.1. The Algorithm

One of the simplest sorting algorithms is the insertion sort. Insertion sort consists of n - 1 passes. For pass p = 2 through n, insertion sort ensures that the elements in positions 1 through p are in sorted order. Insertion sort makes use of the fact that elements in positions 1 through p - 1 are already known to be in sorted order. Figure 7.1 shows a sample file after each pass of insertion sort.

Figure 7.1 shows the general strategy. In pass p, we move the pth element left until its correct place is found among the first p elements. The code in Figure 7.2 implements this strategy. The sentinel in a[0] terminates the while loop in the event that in some pass an element is moved all the way to the front. Lines 3 through 6 implement that data movement without the explicit use of swaps. The element in position p is saved in tmp, and all larger elements (prior to position p) are moved one spot to the right. Then tmp is placed in the correct spot. This is the same technique that was used in the implementation of binary heaps.

Original        34   8  64  51  32  21     Positions Moved

----------------------------------------------------------

After p = 2      8  34  64  51  32  21             1

After p = 3      8  34  64  51  32  21             0

After p = 4      8  34  51  64  32  21             1

After p = 5      8  32  34  51  64  21             3

After p = 6      8  21  32  34  51  64             4

Figure 7.1 Insertion sort after each pass

void

insertion_sort( input_type a[ ], unsigned int n )

{

unsigned int j, p;

input_type tmp;

/*1*/       a[0] = MIN_DATA;         /* sentinel */

/*2*/       for( p=2; p <= n; p++ )

{

/*3*/            tmp = a[p];

/*4*/            for( j = p; tmp < a[j-1]; j-- )

/*5*/                 a[j] = a[j-1];

/*6*/            a[j] = tmp;

}

}

Figure 7.2 Insertion sort routine.

7.2.2. Analysis of Insertion Sort

Because of the nested loops, each of which can take n iterations, insertion sort is O(n2). Furthermore, this bound is tight, because input in reverse order can actually achieve this bound. A precise calculation shows that the test at line 4 can be executed at most p times for each value of p. Summing over all p gives a total of

On the other hand, if the input is presorted, the running time is O(n), because the test in the inner for loop always fails immediately. Indeed, if the input is almost sorted (this term will be more rigorously defined in the next section), insertion sort will run quickly. Because of this wide variation, it is worth analyzing the average-case behavior of this algorithm. It turns out that the average case is   [image] (n2) for insertion sort, as well as for a variety of other sorting algorithms, as the next section shows.

7.3. A Lower Bound for Simple Sorting Algorithms

An inversion in an array of numbers is any ordered pair (i, j) having the property that i < j but a[i] > a[j]. In the example of the last section, the input list 34, 8, 64, 51, 32, 21 had nine inversions, namely (34,8), (34,32), (34,21), (64,51), (64,32), (64,21), (51,32), (51,21) and (32,21). Notice that this is exactly the number of swaps that needed to be (implicitly) performed by insertion sort. This is always the case, because swapping two adjacent elements that are out of place removes exactly one inversion, and a sorted file has no inversions. Since there is O(n) other work involved in the algorithm, the running time of insertion sort is O(I + n), where I is the number of inversions in the original file. Thus, insertion sort runs in linear time if the number of inversions is O(n).

We can compute precise bounds on the average running time of insertion sort by computing the average number of inversions in a permutation. As usual, defining average is a difficult proposition. We will assume that there are no duplicate elements (if we allow duplicates, it is not even clear what the average number of duplicates is). Using this assumption, we can assume that the input is some permutation of the first n integers (since only relative ordering is important) and that all are equally likely. Under these assumptions, we have the following theorem:

THEOREM 7.1.

The average number of inversions in an array of n distinct numbers is n(n - 1)/4.

PROOF:

For any list, L, of numbers, consider Lr, the list in reverse order. The reverse list of the example is 21, 32, 51, 64, 34, 8. Consider any pair of two numbers in the list (x, y), with y > x. Clearly, in exactly one of L and Lr this ordered pair represents an inversion. The total number of these pairs in a list L and its reverse Lr is n(n - 1)/2. Thus, an average list has half this amount, or n(n -1)/4 inversions.

This theorem implies that insertion sort is quadratic on average. It also provides a very strong lower bound about any algorithm that only exchanges adjacent elements.

THEOREM 7.2.

Any algorithm that sorts by exchanging adjacent elements requires   [image] (n2) time on average.

PROOF:

The average number of inversions is initially n(n - 1)/4 =   [image] (n2). Each swap removes only one inversion, so   [image] (n2) swaps are required.

This is an example of a lower-bound proof. It is valid not only for insertion sort, which performs adjacent exchanges implicitly, but also for other simple algorithms such as bubble sort and selection sort, which we will not describe here. In fact, it is valid over an entire class of sorting algorithms, including those undiscovered, that perform only adjacent exchanges. Because of this, this proof cannot be confirmed empirically. Although this lower-bound proof is rather simple, in general proving lower bounds is much more complicated than proving upper bounds and in some cases resembles voodoo.

This lower bound shows us that in order for a sorting algorithm to run in subquadratic, or o(n2), time, it must do comparisons and, in particular, exchanges between elements that are far apart. A sorting algorithm makes progress by eliminating inversions, and to run efficiently, it must eliminate more than just one inversion per exchange.

7.4. Shellsort

Shellsort, named after its inventor, Donald Shell, was one of the first algorithms to break the quadratic time barrier, although it was not until several years after its initial discovery that a subquadratic time bound was proven. As suggested in the previous section, it works by comparing elements that are distant; the distance between comparisons decreases as the algorithm runs until the last phase, in which adjacent elements are compared. For this reason, Shellsort is sometimes referred to as diminishing increment sort.

Shellsort uses a sequence, h1, h2, . . . , ht, called the increment sequence. Any increment sequence will do as long as h1 = 1, but obviously some choices are better than others (we will discuss that question later). After a phase, using some increment hk, for every i, we have a[i]   [image]  a[i+hk] (where this makes sense); all elements spaced hk apart are sorted. The file is then said to be hk-sorted. For example, Figure 7.3 shows an array after several phases of Shellsort. An important property of Shellsort (which we state without proof) is that an hk-sorted file that is then hk-1-sorted remains hk-sorted. If this were not the case, the algorithm would likely be of little value, since work done by early phases would be undone by later phases.

Original      81  94  11  93  12  35  17  95  28  58  41  75  15

----------------------------------------------------------------

After 5-sort  35  17  11  28  12  41  75  15  96  58  81  94  95

After 3-sort  28  12  11  35  15  41  58  17  94  75  81  96  95

After 1-sort  11  12  15  17  28  35  41  58  75  81  94  95  96

Figure 7.3 Shellsort after each pass

The general strategy to hk-sort is for each position, i, in hk + 1, hk + 2, . . . , n, place the element in the correct spot among i, i - hk, i - 2hk, etc. Although this does not affect the implementation, a careful examination shows that the action of an hk-sort is to perform an insertion sort on hk independent sub-arrays. This observation will be important when we analyze the running time of Shellsort.

A popular (but poor) choice for increment sequence is to use the sequence suggested by Shell: ht   [image] n/2  [image] , and hk =   [image] hk+1/2  [image] . Figure 7.4 contains a program that implements Shellsort using this sequence. We shall see later that there are increment sequences that give a significant improvement in the algorithm's running time.

The program in Figure 7.4 avoids the explicit use of swaps in the same manner as our implementation of insertion sort. Unfortunately, for Shellsort it is not possible to use a sentinel, and so the code in lines 3 through 7 is not quite as clean as the corresponding code in insertion sort (lines 3 through 5).

void

shellsort( input_type a[ ], unsigned int n )

{

unsigned int i, j, increment;

input_type tmp;

/*1*/       for( increment = n/2; increment > 0; increment /= 2 )

/*2*/            for( i = increment+1; i<=n; i++ )

{

/*3*/                 tmp = a[i];

/*4*/                 for( j = i; j > increment; j -= increment )

/*5*/                      if( tmp < a[j-increment] )

/*6*/                           a[j] = a[j-increment];

else

/*7*/                           break;

/*8*/                 a[j] = tmp;

}

}

Figure 7.4 Shellsort routine using Shell's increments (better increments are possible)

7.4.1. Worst-Case Analysis of Shellsort

Although Shellsort is simple to code, the analysis of its running time is quite another story. The running time of Shellsort depends on the choice of increment sequence, and the proofs can be rather involved. The average-case analysis of Shellsort is a long-standing open problem, except for the most trivial increment sequences. We will prove tight worst-case bounds for two particular increment sequences.

THEOREM 7.3.

The worst-case running time of Shellsort, using Shell's increments, is   [image] (n2).

PROOF:

The proof requires showing not only an upper bound on the worst-case running time but also showing that there exists some input that actually takes   [image] (n2) time to run. We prove the lower bound first, by constructing a bad case. First, we choose n to be a power of 2. This makes all the increments even, except for the last increment, which is 1. Now, we will give as input an array, input_data, with the n/2 largest numbers in the even positions and the n/2 smallest numbers in the odd positions. As all the increments except the last are even, when we come to the last pass, the n/2 largest numbers are still all in even positions and the n/2 smallest numbers are still all in odd positions. The ith smallest number (i   [image]  n/2) is thus in position 2i -1 before the beginning of the last pass. Restoring the ith element to its correct place requires moving it i -1 spaces in the array. Thus, to merely place the n/2 smallest elements in the correct place requires at least   [image]  work. As an example, Figure 7.5 shows a bad (but not the worst) input when n = 16. The number of inversions remaining after the 2-sort is exactly 1 + 2 + 3 + 4 + 5 + 6 + 7 = 28; thus, the last pass will take considerable time.

To finish the proof, we show the upper bound of O(n2). As we have observed before, a pass with increment hk consists of hk insertion sorts of about n/hk elements. Since insertion sort is quadratic, the total cost of a pass is O(hk(n/hk)2) = O(n2/hk). Summing over all passes gives a total bound of   [image]. Because the increments form a geometric series with common ratio 2, and the largest term in the series is   [image]. Thus we obtain a total bound of O(n2).

The problem with Shell's increments is that pairs of increments are not necessarily relatively prime, and thus the smaller increment can have little effect. Hibbard suggested a slightly different increment sequence, which gives better results in practice (and theoretically). His increments are of the form 1, 3, 7, . . . , 2k - 1. Although these increments are almost identical, the key difference is that consecutive increments have no common factors. We now analyze the worst-case running time of Shellsort for this increment sequence. The proof is rather complicated.

    Start     1  9  2  10  3  11  4  12  5  13  6  14  7  15  8  16

-------------------------------------------------------------------

After 8-sort  1  9  2  10  3  11  4  12  5  13  6  14  7  15  8  16

After 4-sort  1  9  2  10  3  11  4  12  5  13  6  14  7  15  8  16

After 2-sort  1  9  2  10  3  11  4  12  5  13  6  14  7  15  8  16

After 1-sort  1  2  3  4  5  6  7  8  9  10  11  12  13  14  15  16

Figure 7.5 Bad case for Shellsort with Shell's increments

THEOREM 7.4.

The worst-case running time of Shellsort using Hibbard's increments is   [image] (n3/2).

PROOF:

We will prove only the upper bound and leave the proof of the lower bound as an exercise. The proof requires some well-known results from additive number theory. References to these results are provided at the end of the chapter.

For the upper bound, as before, we bound the running time of each pass and sum over all passes. For increments hk > n1/2, we will use the bound O(n2/hk) from the previous theorem. Although this bound holds for the other increments, it is too large to be useful. Intuitively, we must take advantage of the fact that this increment sequence is special. What we need to show is that for any element ap in position p, when it is time to perform an hk-sort, there are only a few elements to the left of position p that are larger than ap.

When we come to hk-sort the input array, we know that it has already been hk+1- and hk+2-sorted. Prior to the hk-sort, consider elements in positions p and p - i, i < p. If i is a multiple of hk+1 or hk+2, then clearly a[p - i] < a[p]. We can say more, however. If i is expressible as a linear combination (in nonnegative integers) of hk+1 and hk+2, then a[p - i] < a[p]. As an example, when we come to 3-sort, the file is already 7- and 15-sorted. 52 is expressible as a linear combination of 7 and 15, because 52 = 1 * 7 + 3 * 15. Thus, a[100] cannot be larger than a[152] because a[100]   [image]  a[107]   [image]  a[122]   [image]  a[137]   [image]  a [152].

Now, hk+2 = 2hk +1 + 1, so hk +1 and hk +2 cannot share a common factor. In this case, it is possible to show that all integers that are at least as large as (hk+1 - 1)(hk+2 - 1) = 8h2k + 4hk can be expressed as a linear combination of hk+1 and hk+2 (see the reference at the end of the chapter).

This tells us that the body of the for loop at line 4 can be executed at most 8hk + 4 = O(hk) times for each of the n - hk positions. This gives a bound of O(nhk) per pass.

Using the fact that about half the increments satisfy   [image], and assuming that t is even, the total running time is then

Because both sums are geometric series, and since   [image],this simplifies to

The average-case running time of Shellsort, using Hibbard's increments, is thought to be O(n5/4), based on simulations, but nobody has been able to prove this. Pratt has shown that the   [image] (n3/2) bound applies to a wide range of increment sequences.

Sedgewick has proposed several increment sequences that give an O(n4/3) worst-case running time (also achievable). The average running time is conjectured to be O(n7/6) for these increment sequences. Empirical studies show that these sequences perform significantly better in practice than Hibbard's. The best of these is the sequence {1, 5, 19, 41, 109, . . .}, in which the terms are either of the form 9   [image]  4i - 9   [image]  2i + 1 or 4i - 3   [image]  2i + 1. This is most easily implemented by placing these values in an array. This increment sequence is the best known in practice, although there is a lingering possibility that some increment sequence might exist that could give a significant improvement in the running time of Shellsort.

There are several other results on Shellsort that (generally) require difficult theorems from number theory and combinatorics and are mainly of theoretical interest. Shellsort is a fine example of a very simple algorithm with an extremely complex analysis.

The performance of Shellsort is quite acceptable in practice, even for n in the tens of thousands. The simplicity of the code makes it the algorithm of choice for sorting up to moderately large input.

7.5. Heapsort

As mentioned in Chapter 6, priority queues can be used to sort in O(n log n) time. The algorithm based on this idea is known as heapsort and gives the best Big-Oh running time we have seen so far. In practice however, it is slower than a version of Shellsort that uses Sedgewick's increment sequence.

Recall, from Chapter 6, that the basic strategy is to build a binary heap of n elements. This stage takes O(n) time. We then perform n delete_min operations. The elements leave the heap smallest first, in sorted order. By recording these elements in a second array and then copying the array back, we sort n elements. Since each delete_min takes O(log n) time, the total running time is O(n log n).

The main problem with this algorithm is that it uses an extra array. Thus, the memory requirement is doubled. This could be a problem in some instances. Notice that the extra time spent copying the second array back to the first is only O(n), so that this is not likely to affect the running time significantly. The problem is space.

A clever way to avoid using a second array makes use of the fact that after each delete_min, the heap shrinks by 1. Thus the cell that was last in the heap can be used to store the element that was just deleted. As an example, suppose we have a heap with six elements. The first delete_min produces a1. Now the heap has only five elements, so we can place a1 in position 6. The next delete_min produces a2. Since the heap will now only have four elements, we can place a2 in position 5.

Using this strategy, after the last delete_min the array will contain the elements in decreasing sorted order. If we want the elements in the more typical increasing sorted order, we can change the ordering property so that the parent has a larger key than the child. Thus we have a (max)heap.

In our implementation, we will use a (max)heap, but avoid the actual ADT for the purposes of speed. As usual, everything is done in an array. The first step builds the heap in linear time. We then perform n - 1 delete_maxes by swapping the last element in the heap with the first, decrementing the heap size, and percolating down. When the algorithm terminates, the array contains the elements in sorted order. For instance, consider the input sequence 31, 41, 59, 26, 53, 58, 97. The resulting heap is shown in Figure 7.6.

Figure 7.7 shows the heap that results after the first delete_max. As the figures imply, the last element in the heap is 31; 97 has been placed in a part of the heap array that is technically no longer part of the heap. After 5 more delete_max operations, the heap will actually have only one element, but the elements left in the heap array will be in sorted order.

The code to perform heapsort is given in Figure 7.8.

Figure 7.6 (Max) heap after build_heap phase

Figure 7.7 Heap after first delete_max

void

heapsort( input_type a[], unsigned int n )

{

int i;

/*1*/      for( i=n/2; i>0; i-- )      /* build_heap */

/*2*/           perc_down (a, i, n );

/*3*/      for( i=n; i>=2; i-- )

{

/*4*/           swap( &a[1], &a[i] );    /* delete_max */

/*5*/           perc_down( a, 1, i-1 );

}

}

void

perc_down( input_type a[], unsigned int i, unsigned int n )

{

unsigned int child;

input_type tmp;

/*1*/      for( tmp=a[i]; i*2<=n; i=child )

{

/*2*/           child = i*2;

/*3*/           if( ( child != n ) && ( a[child+1] > a[child] ) )

/*4*/               child++;

/*5*/           if( tmp < a[child] )

/*6*/                a[i] = a[child];

else

/*7*/                break;

}

/*8*/      a[i] = tmp;

}

Figure 7.8 Heapsort

7.6. Mergesort

We now turn our attention to mergesort. Mergesort runs in O(n log n) worst-case running time, and the number of comparisons used is nearly optimal. It is a fine example of a recursive algorithm.

The fundamental operation in this algorithm is merging two sorted lists. Because the lists are sorted, this can be done in one pass through the input, if the output is put in a third list. The basic merging algorithm takes two input arrays a and b, an output array c, and three counters, aptr, bptr, and cptr, which are initially set to the beginning of their respective arrays. The smaller of a[aptr] and b[bptr] is copied to the next entry in c, and the appropriate counters are advanced. When either input list is exhausted, the remainder of the other list is copied to c. An example of how the merge routine works is provided for the following input.

If the array a contains 1, 13, 24, 26, and b contains 2, 15, 27, 38, then the algorithm proceeds as follows: First, a comparison is done between 1 and 2. 1 is added to c, and then 13 and 2 are compared.

2 is added to c, and then 13 and 15 are compared.

13 is added to c, and then 24 and 15 are compared. This proceeds until 26 and 27 are compared.

26 is added to c, and the a array is exhausted.

The remainder of the b array is then copied to c.

The time to merge two sorted lists is clearly linear, because at most n - 1 comparisons are made, where n is the total number of elements. To see this, note that every comparison adds an element to c, except the last comparison, which adds at least two.

The mergesort algorithm is therefore easy to describe. If n = 1, there is only one element to sort, and the answer is at hand. Otherwise, recursively mergesort the first half and the second half. This gives two sorted halves, which can then be merged together using the merging algorithm described above. For instance, to sort the eight-element array 24, 13, 26, 1, 2, 27, 38, 15, we recursively sort the first four and last four elements, obtaining 1, 13, 24, 26, 2, 15, 27, 38. Then we merge the two halves as above, obtaining the final list 1, 2, 13, 15, 24, 26, 27, 38. This algorithm is a classic divide-and-conquer strategy. The problem is divided into smaller problems and solved recursively. The conquering phase consists of patching together the answers. Divide-and-conquer is a very powerful use of recursion that we will see many times.

An implementation of mergesort is provided in Figure 7.9. The procedure called mergesort is just a driver for the recursive routine m_sort.

The merge routine is subtle. If a temporary array is declared locally for each recursive call of merge, then there could be log n temporary arrays active at any point. This could be fatal on a machine with small memory. On the other hand, if the merge routine dynamically allocates and frees the minimum amount of temporary memory, considerable time will be used by malloc. A close examination shows that since merge is the last line of m_sort, there only needs to be one temporary array active at any point. Further, we can use any part of the temporary array; we will use the same portion as the input array a. This allows the improvement described at the end of this section. Figure 7.10 implements the merge routine.

7.6.1. Analysis of Mergesort

7.6.1. Analysis of Mergesort

Mergesort is a classic example of the techniques used to analyze recursive routines. It is not obvious that mergesort can easily be rewritten without recursion (it can), so we have to write a recurrence relation for the running time. We will assume that n is a power of 2, so that we always split into even halves. For n = 1, the time to mergesort is constant, which we will denote by 1. Otherwise, the time to mergesort n numbers is equal to the time to do two recursive mergesorts of size n/2, plus the time to merge, which is linear. The equations below say this exactly:

T(1) = 1

T(n) = 2T(n/2) + n

void

mergesort( input_type a[], unsigned int n )

{

input_type *tmp_array;

tmp_array = (input_type *) malloc

( (n+1) * sizeof (input_type) );

if( tmp_array != NULL )

{

m_sort( a, tmp_array, 1, n );

free( tmp_array );

}

else

fatal_error("No space for tmp array!!!");

}

void

m_sort( input_type a[], input_type tmp_array[ ],

int left, int right )

{

int center;

if( left < right )

{

center = (left + right) / 2;

m_sort( a, tmp_array, left, center );

m_sort( a, tmp_array, center+1, right );

merge( a, tmp_array, left, center+1, right );

}

}

Figure 7.9 Mergesort routine

This is a standard recurrence relation, which can be solved several ways. We will show two methods. The first idea is to divide the recurrence relation through by n. The reason for doing this will become apparent soon. This yields

This equation is valid for any n that is a power of 2, so we may also write

/* 1_pos = start of left half, r_pos = start of right half */

void

merge( input_type a[ ], input_type tmp_array[ ],

int l_pos, int r_pos, int right_end )

{

int i, left_end, num_elements, tmp_pos;

left_end = r_pos - 1;

tmp_pos = l_pos;

num_elements = right_end - l_pos + 1;

/* main loop */

while( ( 1_pos <= left_end ) && ( r_pos <= right_end ) )

if( a[1_pos] <= a[r_pos] )

tmp_array[tmp_pos++] = a[l_pos++];

else

tmp_array[tmp_pos++] = a[r_pos++];

while( l_pos <= left_end )  /* copy rest of first half */

tmp_array[tmp_pos++] = a[l_pos++];

while( r_pos <= right_end ) /* copy rest of second half */

tmp_array[tmp_pos++] = a[r_pos++];

/* copy tmp_array back */

for(i=1; i <= num_elements; i++, right_end-- )

a[right_end] = tmp_array[right_end];

}

Figure 7.10 Merge routine

and

Now add up all the equations. This means that we add all of the terms on the left-hand side and set the result equal to the sum of all of the terms on the right-hand side. Observe that the term T(n/2)/(n/2) appears on both sides and thus cancels. In fact, virtually all the terms appear on both sides and cancel. This is called telescoping a sum. After everything is added, the final result is

because all of the other terms cancel and there are log n equations, and so all the 1s at the end of these equations add up to log n. Multiplying through by n gives the final answer.

T(n) = n log n + n = O(n log n)

Notice that if we did not divide through by n at the start of the solutions, the sum would not telescope. This is why it was necessary to divide through by n.

An alternative method is to substitute the recurrence relation continually on the right-hand side. We have

T(n) = 2T(n/2) + n

Since we can substitute n/2 into the main equation,

2T(n/2) = 2(2(T(n/4)) + n/2) = 4T(n/4) + n

we have

T(n) = 4T(n/4) + 2n

Again, by substituting n/4 into the main equation, we see that

4T(n/4) = 4(2T(n/8)) + (n/4) = 8T(n/8) + n

So we have

T(n) = 8T(n/8) + 3n

Continuing in this manner, we obtain

T(n) = 2kT(n/2k) + k    n

Using k = log n, we obtain

T(n) = nT(1) + n log n = n log n + n

The choice of which method to use is a matter of taste. The first method tends to produce scrap work that fits better on a standard   [image]  sheet of paper, leading to fewer mathematical errors, but it requires a certain amount of experience to apply. The second method is more of a brute force approach.

Recall that we have assumed n = 2k. The analysis can be refined to handle cases when n is not a power of 2. The answer turns out to be almost identical (this is usually the case).

Although mergesort's running time is O(n log n), it is hardly ever used for main memory sorts. The main problem is that merging two sorted lists requires linear extra memory, and the additional work spent copying to the temporary array and back, throughout the algorithm, has the effect of slowing down the sort considerably. This copying can be avoided by judiciously switching the roles of a and tmp_array at alternate levels of the recursion. A variant of mergesort can also be implemented nonrecursively (Exercise 7.13), but even so, for serious internal sorting applications, the algorithm of choice is quicksort, which is described in the next section. Nevertheless, as we will see later in this chapter, the merging routine is the cornerstone of most external sorting algorithms.

7.7. Quicksort

As its name implies, quicksort is the fastest known sorting algorithm in practice. Its average running time is O(n log n). It is very fast, mainly due to a very tight and highly optimized inner loop. It has O(n2) worst-case performance, but this can be made exponentially unlikely with a little effort. The quicksort algorithm is simple to understand and prove correct, although for many years it had the reputation of being an algorithm that could in theory be highly optimized but in practice was impossible to code correctly (no doubt because of FORTRAN). Like mergesort, quicksort is a divide-and-conquer recursive algorithm. The basic algorithm to sort an array S consists of the following four easy steps:

1.     If the number of elements in S is 0 or 1, then return.

2.     Pick any element v in S. This is called the pivot.

3.     Partition S - {v} (the remaining elements in S) into two disjoint groups: S1 = {x   [image]  S - {v}| x   [image]  v}, and S2 = {x   [image]  S -{v}| x   [image]  v}.

4.     Return { quicksort(S1) followed by v followed by quicksort(S2)}.

Since the partition step ambiguously describes what to do with elements equal to the pivot, this becomes a design decision. Part of a good implementation is handling this case as efficiently as possible. Intuitively, we would hope that about half the keys that are equal to the pivot go into S1 and the other half into S2, much as we like binary search trees to be balanced.

Figure 7.11 shows the action of quicksort on a set of numbers. The pivot is chosen (by chance) to be 65. The remaining elements in the set are partitioned into two smaller sets. Recursively sorting the set of smaller numbers yields 0, 13, 26, 31, 43, 57 (by rule 3 of recursion). The set of large numbers is similarly sorted. The sorted arrangement of the entire set is then trivially obtained.

It should be clear that this algorithm works, but it is not clear why it is any faster than mergesort. Like mergesort, it recursively solves two subproblems and requires linear additional work (step 3), but, unlike mergesort, the subproblems are not guaranteed to be of equal size, which is potentially bad. The reason that quicksort is faster is that the partitioning step can actually be performed in place and very efficiently. This efficiency more than makes up for the lack of equal-sized recursive calls.

The algorithm as described so far lacks quite a few details, which we now fill in. There are many ways to implement steps 2 and 3; the method presented here is the result of extensive analysis and empirical study and represents a very efficient way to implement quicksort. Even the slightest deviations from this method can cause surprisingly bad results.

7.7.1. Picking the Pivot

7.7.2. Partitioning Strategy

7.7.3. Small Files

7.7.4. Actual Quicksort Routines

7.7.5. Analysis of Quicksort

7.7.6. A Linear-Expected-Time Algorithm for Selection

7.7.1. Picking the Pivot

Although the algorithm as described works no matter which element is chosen as pivot, some choices are obviously better than others.

Figure 7.11 The steps of quicksort illustrated by example

A Wrong Way

A Safe Maneuver

Median-of-Three Partitioning

A Wrong Way

The popular, uninformed choice is to use the first element as the pivot. This is acceptable if the input is random, but if the input is presorted or in reverse order, then the pivot provides a poor partition, because virtually all the elements go into S1 or S2. Worse, this happens consistently throughout the recursive calls. The practical effect is that if the first element is used as the pivot and the input is presorted, then quicksort will take quadratic time to do essentially nothing at all, which is quite embarrassing. Moreover, presorted input (or input with a large presorted section) is quite frequent, so using the first element as pivot is an absolutely horrible idea and should be discarded immediately. An alternative is choosing the larger of the first two distinct keys as pivot, but this has the same bad properties as merely choosing the first key. Do not use that pivoting strategy either.

A Safe Maneuver

A safe course is merely to choose the pivot randomly. This strategy is generally perfectly safe, unless the random number generator has a flaw (which is not as uncommon as you might think), since it is very unlikely that a random pivot would consistently provide a poor partition. On the other hand, random number generation is generally an expensive commodity and does not reduce the average running time of the rest of the algorithm at all.

Median-of-Three Partitioning

The median of a group of n numbers is the   [image] n/2  [image]  th largest number. The best choice of pivot would be the median of the file. Unfortunately, this is hard to calculate and would slow down quicksort considerably. A good estimate can be obtained by picking three elements randomly and using the median of these three as pivot. The randomness turns out not to help much, so the common course is to use as pivot the median of the left, right and center elements. For instance, with input 8, 1, 4, 9, 6, 3, 5, 2, 7, 0 as before, the left element is 8, the right element is 0 and the center (in position   [image] (left + right)/2  [image] ) element is 6. Thus, the pivot would be v = 6. Using median-of-three partitioning clearly eliminates the bad case for sorted input (the partitions become equal in this case) and actually reduces the running time of quicksort by about 5 percent.

7.7.2. Partitioning Strategy

There are several partitioning strategies used in practice, but the one described here is known to give good results. It is very easy, as we shall see, to do this wrong or inefficiently, but it is safe to use a known method. The first step is to get the pivot element out of the way by swapping it with the last element. i starts at the first element and j starts at the next-to-last element. If the original input was the same as before, the following figure shows the current situation.

8  1  4  9  0  3  5  2  7  6

i                       j

For now we will assume that all the elements are distinct. Later on we will worry about what to do in the presence of duplicates. As a limiting case, our algorithm must do the proper thing if all of the elements are identical. It is surprising how easy it is to do the wrong thing.

What our partitioning stage wants to do is to move all the small elements to the left part of the array and all the large elements to the right part. "Small" and "large" are, of course, relative to the pivot.

While i is to the left of j, we move i right, skipping over elements that are smaller than the pivot. We move j left, skipping over elements that are larger than the pivot. When i and j have stopped, i is pointing at a large element and j is pointing at a small element. If i is to the left of j, those elements are swapped. The effect is to push a large element to the right and a small element to the left. In the example above, i would not move and j would slide over one place. The situation is as follows.

8  1  4  9  0  3  5  2  7  6

i                    j

We then swap the elements pointed to by i and j and repeat the process until i and j cross.

         After First Swap

----------------------------

2  1  4  9  0  3  5  8  7  6

i                    j

      Before Second Swap

----------------------------

2  1  4  9  0  3  5  8  7  6

         i        j

      After Second Swap

----------------------------

2  1  4  5  0  3  9  8  7  6

         i        j

     Before Third Swap

----------------------------

2  1  4  5  0  3  9  8  7  6

j  i

At this stage, i and j have crossed, so no swap is performed. The final part of the partitioning is to swap the pivot element with the element pointed to by i.

           After Swap with Pivot

---------------------------------------

2   1   4   5   0   3   6   8   7     9

                        i           pivot

When the pivot is swapped with i in the last step, we know that every element in a position p < i must be small. This is because either position p contained a small element to start with, or the large element originally in position p was replaced during a swap. A similar argument shows that elements in positions p > i must be large.

One important detail we must consider is how to handle keys that are equal to the pivot. The questions are whether or not i should stop when it sees a key equal to the pivot and whether or not j should stop when it sees a key equal to the pivot. Intuitively, i and j ought to do the same thing, since otherwise the partitioning step is biased. For instance, if i stops and j does not, then all keys that are equal to the pivot will wind up in S2.

To get an idea of what might be good, we consider the case where all the keys in the file are identical. If both i and j stop, there will be many swaps between identical elements. Although this seems useless, the positive effect is that i and j will cross in the middle, so when the pivot is replaced, the partition creates two nearly equal subfiles. The mergesort analysis tells us that the total running time would then be O(n log n).

If neither i nor j stop, and code is present to prevent them from running off the end of the array, no swaps will be performed. Although this seems good, a correct implementation would then swap the pivot into the last spot that i touched, which would be the next-to-last position (or last, depending on the exact implementation). This would create very uneven subfiles. If all the keys are identical, the running time is O(n2). The effect is the same as using the first element as a pivot for presorted input. It takes quadratic time to do nothing!

Thus, we find that it is better to do the unnecessary swaps and create even subfiles than to risk wildly uneven subfiles. Therefore, we will have both i and j stop if they encounter a key equal to the pivot. This turns out to be the only one of the four possibilities that does not take quadratic time for this input.

At first glance it may seem that worrying about a file of identical elements is silly. After all, why would anyone want to sort 5,000 identical elements? However, recall that quicksort is recursive. Suppose there are 100,000 elements, of which 5,000 are identical. Eventually, quicksort will make the recursive call on only these 5,000 elements. Then it really will be important to make sure that 5,000 identical elements can be sorted efficiently.

7.7.3. Small Files

For very small files (n   [image]  20), quicksort does not perform as well as insertion sort. Furthermore, because quicksort is recursive, these cases will occur frequently. A common solution is not to use quicksort recursively for small files, but instead use a sorting algorithm that is efficient for small files, such as insertion sort. An even better idea is to leave the file slightly unsorted and finish up with insertion sort. This works well, because insertion sort is efficient for nearly sorted files. Using this strategy can actually save about 15 percent in the running time (over doing no cutoff at all). A good cutoff range is n = 10, although any cutoff between 5 and 20 is likely to produce similar results. This also saves nasty degenerate cases, such as taking the median of three elements when there are only one or two. Of course, if there is a bug in the basic quicksort routine, then the insertion sort will be very, very slow.

7.7.4. Actual Quicksort Routines

The driver for quicksort is shown in Figure 7.12.

The general form of the routines will be to pass the array and the range of the array (left and right) to be sorted. The first routine to deal with is pivot selection. The easiest way to do this is to sort a[left], a[right], and a[center] in place. This has the extra advantage that the smallest of the three winds up in a[left], which is where the partitioning step would put it anyway. The largest winds up in a[right], which is also the correct place, since it is larger than the pivot. Therefore, we can place the pivot in a[right - 1] and initialize i and j to left + 1 and right - 2 in the partition phase. Yet another benefit is that because a[left] is smaller than the pivot, it will act as a sentinel for j. Thus, we do not need to worry about j running past the end. Since i will stop on keys equal to the pivot, storing the pivot in a[right - 1] provides a sentinel for i. The code in Figure 7.13 does the median-of-three partitioning with all the side effects described. It may seem that it is only slightly inefficient to compute the pivot by a method that does not actually sort a[left], a[center], and a[right], but, surprisingly, this produces bad results (see Exercise 7.37).

The real heart of the quicksort routine is in Figure 7.14. It includes the partitioning and recursive calls. There are several things worth noting in this implementation. Line 3 initializes i and j to 1 past their correct values, so that there are no special cases to consider. This initialization depends on the fact that median-of-three partitioning has some side effects; this program will not work if you try to use it without change with a simple pivoting strategy, because i and j start in the wrong place and there is no longer a sentinel for j.

void

quick_sort( input_type a[ ], unsigned int n )

{

q_sort( a, 1, n );

insertion_sort( a, n );

}

Figure 7.12 Driver for quicksort

/* Return median of left, center, and right. */

/* Order these and hide pivot */

input_type

median3( input_type a[], int left, int right )

{

int center;

center = (left + right) / 2;

if( a[left] > a[center] )

swap( &a[left], &a[center] );

if( a[left] > a[right] )

swap( &a[left], &a[right] );

if( a[center] > a[right] )

swap( &a[center], &a[right] );

/* invariant: a[left] <= a[center] <= a[right] */

swap( &a[center], &a[right-1] );     /* hide pivot */

return a[right-1];                   /* return pivot */

}

Figure 7.13 Code to perform median-of-three partitioning

The swap at line 8 is sometimes written explicitly, for speed purposes. For the algorithm to be fast, it is necessary to force the compiler to compile this code in-line. Many compilers will do this automatically, if asked to, but for those that do not the difference can be significant.

Finally, lines 5 and 6 show why quicksort is so fast. The inner loop of the algorithm consists of an increment/decrement (by 1, which is fast), a test, and a jump. There is no extra juggling as there is in mergesort. This code is still surprisingly tricky. It is tempting to replace lines 3 through 9 with the statements in Figure 7.15. This does not work, because there would be an infinite loop if a[i] = a[j] = pivot.

7.7.5. Analysis of Quicksort

Like mergesort, quicksort is recursive, and hence, its analysis requires solving a recurrence formula. We will do the analysis for a quicksort, assuming a random pivot (no median-of-three partitioning) and no cutoff for small files. We will take T(0) = T(1) = 1, as in mergesort. The running time of quicksort is equal to the running time of the two recursive calls plus the linear time spent in the partition (the pivot selection takes only constant time). This gives the basic quicksort relation

T(n) = T(i) + T(n - i - 1) + cn

(7.1)

where i = |S1| is the number of elements in S1. We will look at three cases.

void

q_sort( input_type a[], int left, int right )

{

int i, j;

input_type pivot;

/*1*/       if( left + CUTOFF <= right )

{

/*2*/           pivot = median3( a, left, right );

/*3*/           i=left; j=right-1;

/*4*/           for(;;)

{

/*5*/                while( a[++i] < pivot );

/*6*/                while( a[--j] > pivot );

/*7*/                if( i < j )

/*8*/                     swap( &a[i], &a[j] );

else

/*9*/                     break;

}

/*10*/          swap( &a[i], &a[right-1] );   /*restore pivot*/

/*11*/          q_sort( a, left, i-1 );

/*12*/          q_sort( a, i+1, right );

}

}

Figure 7.14 Main quicksort routine

/*3*/           i=left+1; j=right-2;

/*4*/           for(;;)

{

/*5*/                while( a[i] < pivot ) i++;

/*6*/                while( a[j] > pivot ) j--;

/*7*/                if( i < j )

/*8*/                     swap( &a[i], &a[j] );

else

/*9*/                     break;

}

Figure 7.15 A small change to quicksort, which breaks the algorithm

Worst-Case Analysis

Best-Case Analysis

Average-Case Analysis

Worst-Case Analysis

The pivot is the smallest element, all the time. Then i = 0 and if we ignore T(0) = 1, which is insignificant, the recurrence is

T(n) = T(n - 1) + cn, n > 1

(7.2)

We telescope, using Equation (7.2) repeatedly. Thus

T(n -1) = T(n - 2) + c(n - 1)

(7.3)

T(n - 2) = T(n - 3) + c(n - 2)

(7.4)

...

T(2) = T(1) + c(2)

(7.5)

Adding up all these equations yields

(7.6)

as claimed earlier.

Best-Case Analysis

In the best case, the pivot is in the middle. To simplify the math, we assume that the two subfiles are each exactly half the size of the original, and although this gives a slight overestimate, this is acceptable because we are only interested in a Big-Oh answer.

T(n) = 2T(n/2) + cn

(7.7)

Divide both sides of Equation (7.7) by n.

(7.8)

We will telescope using this equation.

(7.9)

(7.10)

(7.11)

We add all the equations from (7.7) to (7.11) and note that there are log n of them:

(7.12)

which yields

T(n) = cn log n + n = O(n log n)

(7.13)

Notice that this is the exact same analysis as mergesort, hence we get the same answer.

Average-Case Analysis

This is the most difficult part. For the average case, we assume that each of the file sizes for S1 is equally likely, and hence has probability 1/n. This assumption is actually valid for our pivoting and partitioning strategy, but it is not valid for some others. Partitioning strategies that do not preserve the randomness of the subfiles cannot use this analysis. Interestingly, these strategies seem to result in programs that take longer to run in practice.

With this assumption, the average value of T(i), and hence T(n - i -1), is   [image]. Equation (7.1) then becomes

(7.14)

If Equation (7.14) is multiplied by n, it becomes

(7.15)

We need to remove the summation sign to simplify matters. We note that we can telescope with one more equation.

(7.16)

If we subtract (7.16) from (7.15), we obtain

nT(n) - (n -1)T(n -1) = 2T(n -1) + 2cn -c

(7.17)

We rearrange terms and drop the insignificant -c on the right, obtaining

nT(n) = (n + 1)T(n -1) + 2cn

(7.18)

We now have a formula for T(n) in terms of T(n -1) only. Again the idea is to telescope, but Equation (7.18) is in the wrong form. Divide (7.18) by n(n + 1):

(7.19)

Now we can telescope.

(7.20)

(7.21)

(7.22)

Adding equations (7.19) through (7.22) yields

(7.23)

The sum is about loge  [image], where   [image]    [image]  0.577 is known as Euler's constant, so

(7.24)

(7.25)

Although this analysis seems complicated, it really is not--the steps are natural once you have seen some recurrence relations. The analysis can actually be taken further. The highly optimized version that was described above has also been analyzed, and this result gets extremely difficult, involving complicated recurrences and advanced mathematics. The effects of equal keys has also been analyzed in detail, and it turns out that the code presented does the right thing.

7.7.6. A Linear-Expected-Time Algorithm for Selection

Quicksort can be modified to solve the selection problem, which we have seen in chapters 1 and 6. Recall that by using a priority queue, we can find the kth largest (or smallest) element in O(n + k log n). For the special case of finding the median, this gives an O(n log n) algorithm.

Since we can sort the file in O(n log n) time, one might expect to obtain a better time bound for selection. The algorithm we present to find the kth smallest element in a set S is almost identical to quicksort. In fact, the first three steps are the same. We will call this algorithm quickselect. Let |Si| denote the number of elements in Si. The steps of quickselect are

1.     If |S| = 1, then k = 1 and return the elements in S as the answer. If a cutoff for small files is being used and |S|   [image]  CUTOFF, then sort S and return the kth smallest element.

2.     Pick a pivot element, v   [image]  S.

3.     Partition S - {v} into S1 and S2, as was done with quicksort.

4.     If k   [image]  |S1|, then the kth smallest element must be in S1. In this case, return quickselect (S1, k). If k = 1 + |S1|, then the pivot is the kth smallest element and we can return it as the answer. Otherwise, the kth smallest element lies in S2, and it is the (k - |S1| - 1)st smallest element in S2. We make a recursive call and return quickselect (S2, k - |S1| - 1).

In contrast to quicksort, quickselect makes only one recursive call instead of two. The worst case of quickselect is identical to that of quicksort and is O(n2). Intuitively, this is because quicksort's worst case is when one of S1 and S2 is empty; thus, quickselect is not really saving a recursive call. The average running time, however, is O(n). The analysis is similar to quicksort's and is left as an exercise.

The implementation of quickselect is even simpler than the abstract description might imply. The code to do this shown in Figure 7.16. When the algorithm terminates, the kth smallest element is in position k. This destroys the original ordering; if this is not desirable, then a copy must be made.

/* q_select places the kth smallest element in a[k]*/

void

q_select( input_type a[], int k, int left, int right )

{

int i, j;

input_type pivot;

/*1*/       if( left + CUTOFF <= right )

{

/*2*/            pivot = median3( a, left, right );

/*3*/            i=left; j=right-1;

/*4*/            for(;;)

{

/*5*/                 while( a[++i] < pivot );

/*6*/                 while( a[--j] > pivot );

/*7*/                 if (i < j )

/*8*/                      swap( &a[i], &a[j] );

else

/*9*/                     break;

}

/*10*/           swap( &a[i], &a[right-1] ); /* restore pivot */

/*11*/           if( k < i)

/*12*/                q_select( a, k, left, i-1 );

else

/*13*/            if( k > i )

/*14*/                 q-select( a, k, i+1, right );

}

else

/*15*/            insert_sort(a, left, right );

}

Figure 7.16 Main quickselect routine

Using a median-of-three pivoting strategy makes the chance of the worst case occuring almost negligible. By carefully choosing the pivot, however, we can eliminate the quadratic worst case and ensure an O(n) algorithm. The overhead involved in doing this is considerable, so the resulting algorithm is mostly of theoretical interest. In Chapter 10, we will examine the linear-time worst-case algorithm for selection, and we shall also see an interesting technique of choosing the pivot that results in a somewhat faster selection algorithm in practice.

7.8. Sorting Large Structures

Throughout our discussion of sorting, we have assumed that the elements to be sorted are simply integers. Frequently, we need to sort large structures by a certain key. For instance, we might have payroll records, with each record consisting of a name, address, phone number, financial information such as salary, and tax information. We might want to sort this information by one particular field, such as the name. For all of our algorithms, the fundamental operation is the swap, but here swapping two structures can be a very expensive operation, because the structures are potentially large. If this is the case, a practical solution is to have the input array contain pointers to the structures. We sort by comparing the keys the pointers point to, swapping pointers when necessary. This means that all the data movement is essentially the same as if we were sorting integers. This is known as indirect sorting; we can use this technique for most of the data structures we have described. This justifies our assumption that complex structures can be handled without tremendous loss efficiency.

7.9. A General Lower Bound for Sorting

Although we have O(n log n) algorithms for sorting, it is not clear that this is as good as we can do. In this section, we prove that any algorithm for sorting that uses only comparisons requires   [image] (n log n) comparisons (and hence time) in the worst case, so that mergesort and heapsort are optimal to within a constant factor. The proof can be extended to show that   [image] (n log n) comparisons are required, even on average, for any sorting algorithm that uses only comparisons, which means that quicksort is optimal on average to within a constant factor.

Specifically, we will prove the following result: Any sorting algorithm that uses only comparisons requires   [image] log n!   comparisons in the worst case and log n! comparisons on average. We will assume that all n elements are distinct, since any sorting algorithm must work for this case.

7.9.1 Decision Trees

7.9.1 Decision Trees

A decision tree is an abstraction used to prove lower bounds. In our context, a decision tree is a binary tree. Each node represents a set of possible orderings, consistent with comparisons that have been made, among the elements. The results of the comparisons are the tree edges.

Figure 7.17 A decision tree for three-element insertion sort

The decision tree in Figure 7.17 represents an algorithm that sorts the three elements a, b, and c. The initial state of the algorithm is at the root. (We will use the terms state and node interchangeably.) No comparisons have been done, so all orderings are legal. The first comparison that this particular algorithm performs compares a and b. The two results lead to two possible states. If a < b, then only three possibilities remain. If the algorithm reaches node 2, then it will compare a and c. Other algorithms might do different things; a different algorithm would have a different decision tree. If a > c, the algorithm enters state 5. Since there is only one ordering that is consistent, the algorithm can terminate and report that it has completed the sort. If a < c, the algorithm cannot do this, because there are two possible orderings and it cannot possibly be sure which is correct. In this case, the algorithm will require one more comparison.

Every algorithm that sorts by using only comparisons can be represented by a decision tree. Of course, it is only feasible to draw the tree for extremely small input sizes. The number of comparisons used by the sorting algorithm is equal to the depth of the deepest leaf. In our case, this algorithm uses three comparisons in the worst case. The average number of comparisons used is equal to the average depth of the leaves. Since a decision tree is large, it follows that there must be some long paths. To prove the lower bounds, all that needs to be shown are some basic tree properties.

LEMMA 7.1.

Let T be a binary tree of depth d. Then T has at most 2d leaves.

PROOF:

The proof is by induction. If d = 0, then there is at most one leaf, so the basis is true. Otherwise, we have a root, which cannot be a leaf, and a left and right subtree, each of depth at most d - 1. By the induction hypothesis, they can each have at most 2d-1 leaves, giving a total of at most 2d leaves. This proves the lemma.

LEMMA 7.2.

A binary tree with L leaves must have depth at least   [image] log L  [image].

PROOF:

Immediate from the preceding lemma.

THEOREM 7.5.

Any sorting algorithm that uses only comparisons between elements requires at least   [image] log n!   comparisons in the worst case.

PROOF:

A decision tree to sort n elements must have n! leaves. The theorem follows from the preceding lemma.

THEOREM 7.6.

Any sorting algorithm that uses only comparisons between elements requires   [image] (n log n) comparisons.

PROOF:

From the previous theorem, log n! comparisons are required.

This type of lower-bound argument, when used to prove a worst-case result, is sometimes known as an information-theoretic lower bound. The general theorem says that if there are P different possible cases to distinguish, and the questions are of the form YES/NO, then   [image] log P  [image]  questions are always required in some case by any algorithm to solve the problem. It is possible to prove a similar result for the average-case running time of any comparison-based sorting algorithm. This result is implied by the following lemma, which is left as an exercise: Any binary tree with L leaves has an average depth of at least log L.

7.10. Bucket Sort

Although we proved in the previous section that any general sorting algorithm that uses only comparisons requires   [image] (n log n) time in the worst case, recall that it is still possible to sort in linear time in some special cases.

A simple example is bucket sort. For bucket sort to work, extra information must be available. The input a1, a2, . . . , an must consist of only positive integers smaller than m. (Obviously extensions to this are possible.) If this is the case, then the algorithm is simple: Keep an array called count, of size m, which is initialized to all 0s. Thus, count has m cells, or buckets, which are initially empty. When ai is read, increment count[ai] by 1. After all the input is read, scan the count array, printing out a representation of the sorted list. This algorithm takes O(m + n); the proof is left as an exercise. If m is O(n), then the total is O(n).

Although this algorithm seems to violate the lower bound, it turns out that it does not because it uses a more powerful operation than simple comparisons. By incrementing the appropriate bucket, the algorithm essentially performs an m-way comparison in unit time. This is similar to the strategy used in extendible hashing (Section 5.6). This is clearly not in the model for which the lower bound was proven.

This algorithm does, however, question the validity of the model used in proving the lower bound. The model actually is a strong model, because a general-purpose sorting algorithm cannot make assumptions about the type of input it can expect to see, but must make decisions based on ordering information only. Naturally, if there is extra information available, we should expect to find a more efficient algorithm, since otherwise the extra information would be wasted.

Although bucket sort seems like much too trivial an algorithm to be useful, it turns out that there are many cases where the input is only small integers, so that using a method like quicksort is really overkill.

7.11. External Sorting

So far, all the algorithms we have examined require that the input fit into main memory. There are, however, applications where the input is much too large to fit into memory. This section will discuss external sorting algorithms, which are designed to handle very large inputs.

7.11.1. Why We Need New Algorithms

Most of the internal sorting algorithms take advantage of the fact that memory is directly addressable. Shellsort compares elements a[i] and a[i - hk] in one time unit. Heapsort compares elements a[i] and a[i * 2] in one time unit. Quicksort, with median-of-three partitioning, requires comparing a[left], a[center], and a[right] in a constant number of time units. If the input is on a tape, then all these operations lose their efficiency, since elements on a tape can only be accessed sequentially. Even if the data is on a disk, there is still a practical loss of efficiency because of the delay required to spin the disk and move the disk head.

To see how slow external accesses really are, create a random file that is large, but not too big to fit in main memory. Read the file in and sort it using an efficient algorithm. The time it takes to sort the input is certain to be insignificant compared to the time to read the input, even though sorting is an O(n log n) operation and reading the input is only O(n).

7.11.2. Model for External Sorting

The wide variety of mass storage devices makes external sorting much more device-dependent than internal sorting. The algorithms that we will consider work on tapes, which are probably the most restrictive storage medium. Since access to an element on tape is done by winding the tape to the correct location, tapes can be efficiently accessed only in sequential order (in either direction).

We will assume that we have at least three tape drives to perform the sorting. We need two drives to do an efficient sort; the third drive simplifies matters. If only one tape drive is present, then we are in trouble: any algorithm will require   [image] (n2) tape accesses.

7.11.3. The Simple Algorithm

The basic external sorting algorithm uses the merge routine from mergesort. Suppose we have four tapes, Ta1, Ta2, Tb1, Tb2, which are two input and two output tapes. Depending on the point in the algorithm, the a and b tapes are either input tapes or output tapes. Suppose the data is initially on Ta1. Suppose further that the internal memory can hold (and sort) m records at a time. A natural first step is to read m records at a time from the input tape, sort the records internally, and then write the sorted records alternately to Tb1 and Tb2. We will call each set of sorted records a run. When this is done, we rewind all the tapes. Suppose we have the same input as our example for Shellsort.

If m = 3, then after the runs are constructed, the tapes will contain the data indicated in the following figure.

Now Tb1 and Tb2 contain a group of runs. We take the first run from each tape and merge them, writing the result, which is a run twice as long, onto Ta1. Then we take the next run from each tape, merge these, and write the result to Ta2. We continue this process, alternating between Ta1 and Ta2, until either Tb1 or Tb2 is empty. At this point either both are empty or there is one run left. In the latter case, we copy this run to the appropriate tape. We rewind all four tapes, and repeat the same steps, this time using the a tapes as input and the b tapes as output. This will give runs of 4m. We continue the process until we get one run of length n.

This algorithm will require   [image] log(n/m)  [image]  passes, plus the initial run-constructing pass. For instance, if we have 10 million records of 128 bytes each, and four megabytes of internal memory, then the first pass will create 320 runs. We would then need nine more passes to complete the sort. Our example requires   [image] log 13/3  [image]  = 3 more passes, which are shown in the following figure.

7.11.4. Multiway Merge

If we have extra tapes, then we can expect to reduce the number of passes required to sort our input. We do this by extending the basic (two-way) merge to a k-way merge.

Merging two runs is done by winding each input tape to the beginning of each run. Then the smaller element is found, placed on an output tape, and the  appropriate input tape is advanced. If there are k input tapes, this strategy works the same way, the only difference being that it is slightly more complicated to find  the smallest of the k elements. We can find the smallest of these elements by using a priority queue. To obtain the next element to write on the output tape, we perform a delete_min operation. The appropriate input tape is advanced, and if the run on the input tape is not yet completed, we insert the new element into the priority queue. Using the same example as before, we distribute the input onto the three tapes.

We then need two more passes of three-way merging to complete the sort.

After the initial run construction phase, the number of passes required using k-way merging is   [image] logk(n/m)  [image], because the runs get k times as large in each pass. For the example above, the formula is verified, since   [image] log3 13/3  [image]  = 2. If we have 10 tapes, then k = 5, and our large example from the previous section would require   [image] log5 320  [image]  = 4 passes.

7.11.5. Polyphase Merge

The k-way merging strategy developed in the last section requires the use of 2k tapes. This could be prohibitive for some applications. It is possible to get by with only k + 1 tapes. As an example, we will show how to perform two-way merging using only three tapes.

Suppose we have three tapes, T1, T2, and T3, and an input file on T1 that will produce 34 runs. One option is to put 17 runs on each of T2 and T3. We could then merge this result onto T1, obtaining one tape with 17 runs. The problem is that since all the runs are on one tape, we must now put some of these runs on T2 to perform another merge. The logical way to do this is to copy the first eight runs from T1 onto T2 and then perform the merge. This has the effect of adding an extra half pass for every pass we do.

An alternative method is to split the original 34 runs unevenly. Suppose we put 21 runs on T2 and 13 runs on T3. We would then merge 13 runs onto T1 before T3 was empty. At this point, we could rewind T1 and T3, and merge T1, with 13 runs, and T2, which has 8 runs, onto T3. We could then merge 8 runs until T2 was empty, which would leave 5 runs left on T1 and 8 runs on T3. We could then merge T1 and T3, and so on. The following table below shows the number of runs on each tape after each pass.

     Run     After    After   After    After    After    After    After

    Const.  T3 + T2  T1 + T2  T1 + T3  T2 + T3  T1 + T2  T1 + T3  T2 + T3

-------------------------------------------------------------------------

T1    0       13        5        0       3        1        0        1

T2   21        8        0        5       2        0        1        0

T3   13        0        8        3       0        2        1        0

The original distribution of runs makes a great deal of difference. For instance, if 22 runs are placed on T2, with 12 on T3, then after the first merge, we obtain 12 runs on T1 and 10 runs on T2. Afte another merge, there are 10 runs on T1 and 2 runs on T3. At this point the going gets slow, because we can only merge two sets of runs before T3 is exhausted. Then T1 has 8 runs and T2 has 2 runs. Again, we can only merge two sets of runs, obtaining T1 with 6 runs and T3 with 2 runs. After three more passes, T2 has two runs and the other tapes are empty. We must copy one run to another tape, and then we can finish the merge.

It turns out that the first distribution we gave is optimal. If the number of runs is a Fibonacci number Fn, then the best way to distribute them is to split them into two Fibonacci numbers Fn-1 and Fn-2. Otherwise, it is necessary to pad the tape with dummy runs in order to get the number of runs up to a Fibonacci number. We leave the details of how to place the initial set of runs on the tapes as an exercise.

We can extend this to a k-way merge, in which case we need kth order Fibonacci numbers for the distribution, where the kth order Fibonacci number is defined as F(k)(n) = F(k)(n - 1) + F(k)(n - 2) +   [image]    [image]    [image]  + F(k)(n - k), with the appropriate initial conditions F(k)(n) = 0, 0   [image]  n   [image]  k - 2, F(k)(k - 1) =1.

7.11.6. Replacement Selection

The last item we will consider is construction of the runs. The strategy we have used so far is the simplest possible: We read as many records as possible and sort them, writing the result to some tape. This seems like the best approach possible, until one realizes that as soon as the first record is written to an output tape, the memory it used becomes available for another record. If the next record on the input tape is larger than the record we have just output, then it can be included in the run.

Using this observation, we can give an algorithm for producing runs. This technique is commonly referred to as replacement selection. Initially, m records are read into memory and placed in a priority queue. We perform a delete_min, writing the smallest record to the output tape. We read the next record from the input tape. If it is larger than the record we have just written, we can add it to the priority queue. Otherwise, it cannot go into the current run. Since the priority queue is smaller by one element, we can store this new element in the dead space of the priority queue until the run is completed and use the element for the next run. Storing an element in the dead space is similar to what is done in heapsort. We continue doing this until the size of the priority queue is zero, at which point the run is over. We start a new run by building a new priority queue, using all the elements in the dead space. Figure 7.18 shows the run construction for the small example we have been using, with m = 3. Dead elements are indicated by an asterisk.

In this example, replacement selection produces only three runs, compared with the five runs obtained by sorting. Because of this, a three-way merge finishes in one pass instead of two. If the input is randomly distributed, replacement selection can be shown to produce runs of average length 2m. For our large example, we would expect 160 runs instead of 320 runs, so a five-way merge would require four passes. In this case, we have not saved a pass, although we might if we get lucky and have 125 runs or less. Since external sorts take so long, every pass saved can make a significant difference in the running time.

       3 Elements In Heap Array    Output     Next Element Read

        H[1]   H[2]     H[3]

---------------------------------------------------------------

Run 1    11     94       81          11             96

         81     94       96          81             12*

         94     96       12*         94             35*

         96     35*      12*         96             17*

         17*    35*      12*       End of Run.  Rebuild Heap

---------------------------------------------------------------

Run 2    12     35       17          12             99

         17     35       99          17             28

         28     99       35          28             58

         35     99       58          35             41

         41     99       58          41             75*

         58     99       75*         58         end of tape

         99              75*         99

                         75*       End of Run.  Rebuild Heap

---------------------------------------------------------------

Run 3    75                          75

Figure 7.18 Example of run construction

As we have seen, it is possible for replacement selection to do no better than the standard algorithm. However, the input is frequently sorted or nearly sorted to start with, in which case replacement selection produces only a few very long runs. This kind of input is common for external sorts and makes replacement selection extremely valuable.

Summary

For most general internal sorting applications, either insertion sort, Shellsort, or quicksort will be the method of choice, and the decision of which to use will depend mostly on the size of the input. Figure 7.19 shows the running time obtained for each algorithm on various file sizes.

The data was chosen to be random permutations of n integers, and the times given include only the actual time to sort. The code given in Figure 7.2 was used for insertion sort. Shellsort used the code in Section 7.4 modified to run with Sedgewick's increments. Based on literally millions of sorts, ranging in size from 100 to 25 million, the expected running time of Shellsort with these increments is conjectured to be O(n7/6). The heapsort routine is the same as in Section 7.5. Two versions of quicksort are given. The first uses a simple pivoting strategy and does not do a cutoff. Fortunately, the input files were random. The second uses median-of-three partitioning and a cutoff of ten. Further optimizations were possible. We could have coded the median-of-three routine in-line instead of using a function, and we could have written quicksort nonrecursively. There are some other optimizations to the code that are fairly tricky to implement, and of course we could have used an assembly language. We have made an honest attempt to code all routines efficiently, but of course the performance can vary somewhat from machine to machine.

The highly optimized version of quicksort is as fast as Shellsort even for very small input sizes. The improved version of quicksort still has an O(n2) worst case (one exercise asks you to construct a small example), but the chances of this worst case appearing are so negligible as to not be a factor. If you need to sort large files, quicksort is the method of choice. But never, ever, take the easy way out and use the first element as pivot. It is just not safe to assume that the input will be random. If you do not want to worry about this, use Shellsort. Shellsort will give a small performance penalty but could also be acceptable, especially if simplicity is required. Its worst case is only O(n4/3); the chance of that worst case occuring is likewise negligible.

Heapsort, although an O (n log n) algorithm with an apparently tight inner loop, is slower than Shellsort. A close examination of the algorithm reveals that in order to move data, heapsort does two comparisons. Carlsson has analyzed an improvement suggested by Floyd that moves data with essentially only one comparison, but implementing this improvement makes the code somewhat longer. We leave it to the reader to decide whether the extra coding effort is worth the increased speed (Exercise 7.39).

         Insertion Sort  Shellsort  Heapsort    Quicksort   Quicksort(opt.)

    n        O(n2)        O(n7/6)    O(n log n)  O(n log n)   O(n log n)

---------------------------------------------------------------------------

     10    0.00044        0.00041     0.00057     0.00052      .00046

    100    0.00675        0.00171     0.00420     0.00284      .00244

   1000    0.59564        0.02927     0.05565     0.03153      .02587

  10000   58.864          0.42998     0.71650     0.36765      .31532

 100000      NA           5.7298      8.8591      4.2298      3.5882

1000000      NA          71.164     104.68       47.065      41.282

Figure 7.19 Comparison of different sorting algorithms (all times are in seconds)

Insertion sort is useful only for small files or very nearly sorted files. We have not included mergesort, because its performance is not as good as quicksort for main memory sorts and it is not any simpler to code. We have seen, however, that merging is the central idea of external sorts.

Exercises

7.1     Sort the sequence 3, 1, 4, 1, 5, 9, 2, 6, 5 using insertion sort.

7.2     What is the running time of insertion sort if all keys are equal?

7.3     Suppose we exchange elements a[i] and a[i + k], which were originally out of order. Prove that at least 1 and at most 2k - 1 inversions are removed.

7.4     Show the result of running Shellsort on the input 9, 8, 7, 6, 5, 4, 3, 2, 1 using the increments { 1, 3, 7 }.

7.5     What is the running time of Shellsort using the two-increment sequence 1, 2 }?

7.6     *a.     Prove that the running time of Shellsort is   [image] (n2) using increments of the form 1, c, c2, ...,  ci for any integer c.

**b. Prove that for these increments, the average running time is   [image] (n3/2).

*7.7     Prove that if a k-sorted file is then h-sorted, it remains k-sorted.

**7.8     Prove that the running time of Shellsort, using the increment sequence suggested by Hibbard, is   [image] (n3/2) in the worst case. Hint: You can prove the bound by considering the special case of what Shellsort does when all elements are either 0 or 1. Set input_data[i] = 1 if i is expressible as a linear combination of ht, ht-1, ..., h  [image] t/2  [image] +1 and 0 otherwise.

7.9     Determine the running time of Shellsort for

a.     sorted input

*b.     reverse-ordered input

7.10     Show how heapsort processes the input 142, 543, 123, 65, 453, 879, 572, 434, 111, 242, 811, 102.

7.11     a.     What is the running time of heapsort for presorted input?

**b. Is there any input for which heapsort runs in o(n log n) (in other words, are there any particularly good inputs for heapsort)?

7.12     Sort 3, 1, 4, 1, 5, 9, 2, 6 using mergesort.

7.13     How would you implement mergesort without using recursion?

7.14     Determine the running time of mergesort for

a.     sorted input

b.     reverse-ordered input

c.     random input

7.15     In the analysis of mergesort, constants have been disregarded. Prove that the number of comparisons used in the worst case by mergesort is n  [image] log n  [image]  - 2  [image] log n  [image]  + 1.

7.16     Sort 3, 1, 4, 1, 5, 9, 2, 6, 5, 3, 5 using quicksort with median-of-three partitioning and a cutoff of 3.

7.17     Using the quicksort implementation in this chapter, determine the running time of quicksort for

a.     sorted input

b.     reverse-ordered input

c.     random input

7.18     Repeat Exercise 7.17 when the pivot is chosen as

a.     the first element

b.     the largest of the first two nondistinct keys

c.     a random element

*d.     the average of all keys in the set

7.19     a.     for the quicksort implementation in this chapter, what is the running time when all keys are equal?

b.     Suppose we change the partitioning strategy so that neither i nor j stops when an element with the same key as the pivot is found. What fixes need to be made in the code to guarantee that quicksort works, and what is the running time, when all keys are equal?

c.     Suppose we change the partitioning strategy so that i stops at an element with the same key as the pivot, but j does not stop in a similar case. What fixes need to be made in the code to guarantee that quicksort works, and when all keys are equal, what is the running time of quicksort?

7.20     Suppose we choose the middle key as pivot. Does this make it unlikely that quicksort will require quadratic time?

7.21     Construct a permutation of 20 elements that is as bad as possible for quicksort using median-of-three partitioning and a cutoff of 3.

7.22     Write a program to implement the selection algorithm.

7.23     Solve the following recurrence:   .

7.24     A sorting algorithm is stable if elements with equal keys are left in the same order as they occur in the input. Which of the sorting algorithms in this chapter are stable and which are not? Why?

7.25     Suppose you are given a sorted list of n elements followed by â(n) randomly ordered elements. How would you sort the entire list if

a.     â(n) =O(1)?

b.     â(n) =O(log n)?

*d.     How large can â(n) be for the entire list still to be sortable in O(n) time?

7.26     Prove that any algorithm that finds an element x in a sorted list of n elements requires   [image] (log n ) comparisons.

7.27     Using Stirling's formula,   [image], give a precise estimate for log n !.

7.28     *a.     In how many ways can two sorted arrays of n elements be merged?

*b.     Give a nontrivial lower bound on the number of comparisons required to merge two sorted lists of n elements.

7.29     Prove that sorting n elements with integer keys in the range 1   [image]  key   [image]  m takes O(m + n) time using bucket sort.

7.30     Suppose you have an array of n elements containing only two distinct keys, true and false. Give an O(n) algorithm to rearrange the list so that all false elements precede the true elements. You may use only constant extra space.

7.31     Suppose you have an array of n elements, containing three distinct keys, true, false, and maybe. Give an O(n) algorithm to rearrange the list so that all false elements precede maybe elements, which in turn precede true elements. You may use only constant extra space.

7.32     a.     Prove that any comparison-based algorithm to sort 4 elements requires 5 comparisons.

b.     Give an algorithm to sort 4 elements in 5 comparisons.

7.33     a.     Prove that 7 comparisons are required to sort 5 elements using any comparison-based algorithm.

*b.     Give an algorithm to sort 5 elements with 7 comparisons.

7.34     Write an efficient version of Shellsort and compare performance when the following increment sequences are used:

a.     Shell's original sequence

b.     Hibbard's increments

e.     Sedgewick's increments.

7.35     Implement an optimized version of quicksort and experiment with combinations of the following:

a.     Pivot: first element, middle element, random element, median of three, median of five.

b.     Cutoff values from 0 to 20.

7.36     Write a routine that reads in two alphabetized files and merges them together, forming a third, alphabetized, file.

7.37     Suppose we implement the median of three routine as follows: Find the median of a[left], a[center], a[right], and swap it with a[right]. Proceed with the normal partitioning step starting i at left and j at right - 1 (instead of left + 1 and right - 2). Assume that a [0] = MIN_DATA, so that sentinels are present.

a.     Suppose the input is 2,3,4, ...,n -1, n, 1. What is the running time of this version of quicksort?

b.     Suppose the input is in reverse order. What is the running time of this version of quicksort?

7.38     Prove that any comparison-based sorting algorithm requires   [image] (n log n) comparisons on average.

7.39     Consider the following strategy for percolate_down. We have a hole at node X. The normal routine is to compare X's children and then move the child up to X if it is larger (in the case of a (max)heap) than the element we are trying to place, thereby pushing the hole down; we stop when it is safe to place the new element in the hole. The alternate strategy is to move elements up and the hole down as far as possible, without testing whether the new cell can be inserted. This would place the new cell in a leaf and probably violate the heap order; to fix the heap order, percolate the new cell up in the normal manner. Write a routine to include this idea, and compare the running time with a standard implementation of heapsort.

7.40     Propose an algorithm to sort a large file using only two tapes.

References

Knuth's book [10] is a comprehensive, though somewhat dated, reference for sorting. Gonnet and Baeza-Yates [4] has some more recent results, as well as a huge bibliography.

The original paper detailing Shellsort is [21]. The paper by Hibbard [5] suggested the use of the increments 2k - 1 and tightened the code by avoiding swaps. Theorem 7.4 is from [12]. Pratt's lower bound, which uses a more complex method than that suggested in the text, can be found in [14]. Improved increment sequences and upper bounds appear in [9], [20], and [23]; matching lower bounds have been shown in [24]. A recent unpublished result by Poonen shows that no increment sequence gives an O(n log n) worst-case running time. An identical result was obtained independently and appears in [13]. The average-case running time for Shellsort is still unresolved. Yao [26] has performed an extremely complex analysis for the three-increment case. The result has yet to be extended to more increments. Experiments with various increment sequences appear in [22].

Heapsort was invented by Williams [25]; Floyd [1] provided the linear-time algorithm for heap construction. The analysis of its average case has only recently been obtained [15].

An exact average-case analysis of mergesort has been claimed in [3]; the paper detailing the results is forthcoming. An algorithm to perform merging in linear time without extra space is described in [8].

Quicksort is from Hoare [6]. This paper analyzes the basic algorithm, describes most of the improvements, and includes the selection algorithm. A detailed analysis and empirical study was the subject of Sedgewick's dissertation [19]. Many of the important results appear in the three papers [16], [17], and [18].

Decision trees and sorting optimality are discussed in Ford and Johnson [2]. This paper also provides an algorithm that almost meets the lower bound in terms of number of comparisons (but not other operations). This algorithm was eventually shown to be slightly suboptimal by Manacher [11].

External sorting is covered in detail in [10]. Stable sorting, described in Exercise 7.24, has been addressed by Horvath [7].

1.     R. W. Floyd, "Algorithm 245: Treesort 3," Communications of the ACM 7 (1964), 701.

2.     L. R. Ford and S. M. Johnson, "A Tournament Problem," American Mathematics Monthly 66 (1959), 387-389.

3.     M. Golin and R. Sedgewick, "Exact Analysis of Mergesort," Fourth SIAM Conference on Discrete Mathematics, 1988.

4.     G. H. Gonnet and R. Baeza-Yates, Handbook of Algorithms and Data Structures, second edition, Addison-Wesley, Reading, MA, 1991.

5.     T. H. Hibbard, "An Empirical Study of Minimal Storage Sorting," Communications of the ACM 6 (1963), 206-213.

6.     C. A. R. Hoare, "Quicksort," Computer Journal 5 (1962), 10-15.

7.     E. C. Horvath, "Stable Sorting in Asymptotically Optimal Time and Extra Space," Journal of the ACM 25 (1978), 177-199.

8.     B. Huang and M. Langston, "Practical In-place Merging," Communications of the ACM 31 (1988), 348-352.

9.     J. Incerpi and R. Sedgewick, "Improved Upper Bounds on Shellsort," Journal of Computer and System Sciences 31 (1985), 210-224.

10.     D. E. Knuth, The Art of Computer Programming. Volume 3: Sorting and Searching, second printing, Addison-Wesley, Reading, MA, 1975.

11.     G. K. Manacher, "The Ford-Johnson Sorting Algorithm Is Not Optimal," Journal of the ACM 26 (1979), 441-456.

12.     A. A. Papernov and G. V. Stasevich, "A Method of Information Sorting in Computer Memories," Problems of Information Transmission 1 (1965), 63-75.

13.      C. G. Plaxton and T. Suel, "Improved Lower Bounds for Shellsort," Proceedings of the Thirty-third Annual IEEE Symposium on the Foundations of Computer Science, (1992).

14.     V. R. Pratt, Shellsort and Sorting Networks, Garland Publishing, New York, 1979. (Originally presented as the author's Ph.D. thesis, Stanford University, 1971.)

15.     R. Schaffer and R. Sedgewick, "The Analysis of Heapsort," Journal of Algorithms, to appear.

16.     R. Sedgewick, "Quicksort with Equal Keys," SIAM Journal on Computing 6 (1977), 240-267.

17.     R. Sedgewick, "The Analysis of Quicksort Programs," Acta Informatica 7 (1977), 327-355.

18.     R. Sedgewick, "Implementing Quicksort Programs," Communications of the ACM 21 (1978), 847-857.

19.     R. Sedgewick, Quicksort, Garland Publishing, New York, 1978. (Originally presented as the author's Ph.D. thesis, Stanford University, 1975.)

20.     R. Sedgewick, "A New Upper Bound for Shellsort," Journal of Algorithms 2 (1986), 159-173.

21.     D. L. Shell, "A High-Speed Sorting Procedure," Communications of the ACM 2 (1959), 30-32.

22.     M. A. Weiss, "Empirical Results on the Running Time of Shellsort," Computer Journal 34 (1991), 88-91.

23.     M. A. Weiss and R. Sedgewick, "More On Shellsort Increment Sequences," Information Processing Letters 34 (1990), 267-270.

24.     M. A. Weiss and R. Sedgewick, "Tight Lower Bounds for Shellsort," Journal of Algorithms 11 (1990), 242-251.

25.     J. W. J. Williams, "Algorithm 232: Heapsort," Communications of the ACM 7 (1964), 347-348.

26.     A. C. Yao, "An Analysis of (h, k, 1) Shellsort," Journal of Algorithms 1 (1980), 14-50.

Go to Chapter 8            Return to Table of Contents


Structures, Algorithm Analysis: CHAPTER 8: THE DISJOINT SET ADT

 Return to Table of Contents

CHAPTER 8: THE DISJOINT SET ADT

In this chapter, we describe an efficient data structure to solve the equivalence problem. The data structure is simple to implement. Each routine requires only a few lines of code, and a simple array can be used. The implementation is also extremely fast, requiring constant average time per operation. This data structure is also very interesting from a theoretical point of view, because its analysis is extremely difficult; the functional form of the worst case is unlike any we have yet seen. For the disjoint set ADT, we will

Show how it can be implemented with minimal coding effort.

Greatly increase its speed, using just two simple observations.

Analyze the running time of a fast implementation.

See a simple application.

8.1. Equivalence Relations

A relation R is defined on a set S if for every pair of elements (a, b), a, b [image] S, a R b is either true or false. If a R b is true, then we say that a is related to b.

An equivalence relation is a relation R that satisfies three properties:

1. (Reflexive) a R a, for all a [image] S.

2. (Symmetric) a R b if and only if b R a.

3. (Transitive) a R b and b R c implies that a R c.

We'll consider several examples.

The [image] relationship is not an equivalence relationship. Although it is reflexive, since a [image] a, and transitive, since a [image] b and b [image] c implies a [image] c, it is not symmetric, since a [image] b does not imply b [image] a.

Electrical connectivity, where all connections are by metal wires, is an equivalence relation. The relation is clearly reflexive, as any component is connected to itself. If a is electrically connected to b, then b must be electrically connected to a, so the relation is symmetric. Finally, if a is connected to b and b is connected to c, then a is connected to c. Thus electrical connectivity is an equivalence relation.

Two cities are related if they are in the same country. It is easily verified that this is an equivalence relation. Suppose town a is related to b if it is possible to travel from a to b by taking roads. This relation is an equivalence relation if all the roads are two-way.

8.2. The Dynamic Equivalence Problem

Given an equivalence relation ~, the natural problem is to decide, for any a and b, if a ~ b. If the relation is stored as a two-dimensional array of booleans, then, of course, this can be done in constant time. The problem is that the relation is usually not explicitly, but rather implicitly, defined.

As an example, suppose the equivalence relation is defined over the five-element set {a1, a2, a3, a4, a5}. Then there are 25 pairs of elements, each of which is either related or not. However, the information a1 ~ a2, a3 ~ a4, a5 ~ a1, a4 ~ a2 implies that all pairs are related. We would like to be able to infer this quickly.

The equivalence class of an element a [image] S is the subset of S that contains all the elements that are related to a. Notice that the equivalence classes form a partition of S: Every member of S appears in exactly one equivalence class. To decide if a ~ b, we need only to check whether a and b are in the same equivalence class. This provides our strategy to solve the equivalence problem.

The input is initially a collection of n sets, each with one element. This initial representation is that all relations (except reflexive relations) are false. Each set has a different element, so that Si [image] Sj = [image]; this makes the sets disjoint.

There are two permissible operations. The first is find, which returns the name of the set (that is, the equivalence class) containing a given element. The second operation adds relations. If we want to add the relation a ~ b, then we first see if a and b are already related. This is done by performing finds on both a and b and checking whether they are in the same equivalence class. If they are not, then we apply union. This operation merges the two equivalence classes containing a and b into a new equivalence class. From a set point of view, the result of [image] is to create a new set Sk = Si [image] Sj, destroying the originals and preserving the disjointness of all the sets. The algorithm to do this is frequently known as the disjoint set union/find algorithm for this reason.

This algorithm is dynamic because, during the course of the algorithm, the sets can change via the union operation. The algorithm must also operate on-line: When a find is performed, it must give an answer before continuing. Another possibility would be an off-line algorithm. Such an algorithm would be allowed to see the entire sequence of unions and finds. The answer it provides for each find must still be consistent with all the unions that were performed up until the find, but the algorithm can give all its answers after it has seen all the questions. The difference is similar to taking a written exam (which is generally off-line--you only have to give the answers before time expires), and an oral exam (which is on-line, because you must answer the current question before proceeding to the next question).

Notice that we do not perform any operations comparing the relative values of elements, but merely require knowledge of their location. For this reason, we can assume that all the elements have been numbered sequentially from 1 to n and that the numbering can be determined easily by some hashing scheme. Thus, initially we have Si = {i} for i = 1 through n.

Our second observation is that the name of the set returned by find is actually fairly abitrary. All that really matters is that find(x) = find( [image]) if and only if x and [image] are in the same set.

These operations are important in many graph theory problems and also in compilers which process equivalence (or type) declarations. We will see an application later.

There are two strategies to solve this problem. One ensures that the find instruction can be executed in constant worst-case time, and the other ensures that the union instruction can be executed in constant worst-case time. It has recently been shown that both cannot be done simultaneously in constant worst-case time.

We will now briefly discuss the first approach. For the find operation to be fast, we could maintain, in an array, the name of the equivalence class for each element. Then find is just a simple O(1) lookup. Suppose we want to perform union(a, b). Suppose that a is in equivalence class i and b is in equivalence class j. Then we scan down the array, changing all is to j. Unfortunately, this scan takes [image] (n). Thus, a sequence of n - 1 unions (the maximum, since then everything is in one set), would take [image] (n2) time. If there are [image] (n2) find operations, this performance is fine, since the total running time would then amount to O(1) for each union or find operation over the course of the algorithm. If there are fewer finds, this bound is not acceptable.

One idea is to keep all the elements that are in the same equivalence class in a linked list. This saves time when updating, because we do not have to search through the entire array. This by itself does not reduce the asymptotic running time, because it is still possible to perform [image] (n2) equivalence class updates over the course of the algorithm.

If we also keep track of the size of each equivalence class, and when performing unions we change the name of the smaller equivalence class to the larger, then the total time spent for n - 1 merges isO (n log n). The reason for this is that each element can have its equivalence class changed at most log n times, since every time its class is changed, its new equivalence class is at least twice as large as its old. Using this strategy, any sequence of m finds and up to n - 1 unions takes at most O(m + n log n) time.

In the remainder of this chapter, we will examine a solution to the union/find problem that makes unions easy but finds hard. Even so, the running time for any sequences of at most m finds and up to n - 1 unions will be only a little more than O(m + n).

8.3. Basic Data Structure

Recall that the problem does not require that a find operation return any specific name, just that finds on two elements return the same answer if and only if they are in the same set. One idea might be to use a tree to represent each set, since each element in a tree has the same root. Thus, the root can be used to name the set. We will represent each set by a tree. (Recall that a collection of trees is known as a forest.) Initially, each set contains one element. The trees we will use are not necessarily binary trees, but their representation is easy, because the only information we will need is a parent pointer. The name of a set is given by the node at the root. Since only the name of the parent is required, we can assume that this tree is stored implicitly in an array: each entry p[i] in the array represents the parent of element i. If i is a root, then p[i] = 0. In the forest in Figure 8.1, p[i] = 0 for 1 [image] i [image] 8. As with heaps, we will draw the trees explicitly, with the understanding that an array is being used. Figure 8.1 shows the explicit representation. We will draw the root's parent pointer vertically for convenience.

To perform a union of two sets, we merge the two trees by making the root of one tree point to the root of the other. It should be clear that this operation takes constant time. Figures 8.2, 8.3, and 8.4 represent the forest after each of union(5,6) union(7,8), union(5,7), where we have adopted the convention that the new root after the union(x,y) is x. The implicit representation of the last forest is shown in Figure 8.5.

A find(x) on element x is performed by returning the root of the tree containing x. The time to perform this operation is proportional to the depth of the node representing x, assuming, of course, that we can find the node representing x in constant time. Using the strategy above, it is possible to create a tree of depth n - 1, so the worst-case running time of a find is O(n). Typically, the running time is computed for a sequence of m intermixed instructions. In this case, m consecutive operations could take O(mn) time in the worst case.

The code in Figures 8.6 through 8.9 represents an implementation of the basic algorithm, assuming that error checks have already been performed. In our routine, unions are performed on the roots of the trees. Sometimes the operation is performed by passing any two elements, and having the union perform two finds to determine the roots.

The average-case analysis is quite hard to do. The least of the problems is that the answer depends on how to define average (with respect to the union operation). For instance, in the forest in Figure 8.4, we could say that since there are five trees, there are 5 [image] 4 = 20 equally likely results of the next union (as any two different trees can be unioned). Of course, the implication of this model is that there is only a [image] chance that the next union will involve the large tree. Another model might say that all unions between any two elements in different trees are equally likely, so a larger tree is more likely to be involved in the next union than a smaller tree. In the example above, there is an [image] chance that the large tree is involved in the next union, since (ignoring symmetries) there are 6 ways in which to merge two elements in {1, 2, 3, 4}, and 16 ways to merge an element in {5, 6, 7, 8} with an element in {1, 2, 3, 4}. There are still more models and no general agreement on which is the best. The average running time depends on the model; [image] (m), [image] (m log n), and [image] (mn) bounds have actually been shown for three different models, although the latter bound is thought to be more realistic.

Figure 8.1 Eight elements, initially in different sets

Figure 8.2 After union (5, 6)

Figure 8.3 After union (7, 8)

Figure 8.4 After union (5, 7)

Figure 8.5 Implicit representation of previous tree

typedef int DISJ_SET[ NUM_SETS+1 ];

typedef unsigned int set_type;

typedef unsigned int element_type;

Figure 8.6 Disjoint set type declaration

void

initialize( DISJ_SET S )

{

int i;

for( i = NUN_SETS; i > 0; i-- )

S[i] = 0;

}

Figure 8.7 Disjoint set initialization routine

/* Assumes root1 and root2 are roots. */

/* union is a C keyword, so this routine is named set_union. */

void

set_union( DISJ_SET S, set_type root1, set_type root2 )

{

S[root2] = root1;

}

Figure 8.8 Union (not the best way)

set_type

find( element_type x, DISJ_SET S )

{

if( S[x] <= 0 )

return x;

else

return( find( S[x], S ) );

}

Figure 8.9 A simple disjoint set find algorithm

Quadratic running time for a sequence of operations is generally unacceptable. Fortunately, there are several ways of easily ensuring that this running time does not occur.

8.4. Smart Union Algorithms

The unions above were performed rather arbitrarily, by making the second tree a subtree of the first. A simple improvement is always to make the smaller tree a subtree of the larger, breaking ties by any method; we call this approach union-by-size. The three unions in the preceding example were all ties, and so we can consider that they were performed by size. If the next operation were union (4, 5), then the forest in Figure 8.10 would form. Had the size heuristic not been used, a deeper forest would have been formed (Fig. 8.11).

Figure 8.10 Result of union-by-size

Figure 8.11 Result of an arbitrary union

Figure 8.12 Worst-case tree for n = 16

We can prove that if unions are done by size, the depth of any node is never more than log n. To see this, note that a node is initially at depth 0. When its depth increases as a result of a union, it is placed in a tree that is at least twice as large as before. Thus, its depth can be increased at most log n times. (We used this argument in the quick-find algorithm at the end of Section 8.2.) This implies that the running time for a find operation is O(log n), and a sequence of m operations takes O(m log n). The tree in Figure 8.12 shows the worst tree possible after 16 unions and is obtained if all unions are between equal-sized trees (the worst-case trees are binomial trees, discussed in Chapter 6).

To implement this strategy, we need to keep track of the size of each tree. Since we are really just using an array, we can have the array entry of each root contain the negative of the size of its tree. Thus, initially the array representation of the tree is all -1s (and Fig 8.7 needs to be changed accordingly). When a union is performed, check the sizes; the new size is the sum of the old. Thus, union-by-size is not at all difficult to implement and requires no extra space. It is also fast, on average. For virtually all reasonable models, it has been shown that a sequence of m operations requires O(m) average time if union-by-size is used. This is because when random unions are performed, generally very small (usually one-element) sets are merged with large sets throughout the algorithm.

An alternative implementation, which also guarantees that all the trees will have depth at most O(log n), is union-by-height. We keep track of the height, instead of the size, of each tree and perform unions by making the shallow tree a subtree of the deeper tree. This is an easy algorithm, since the height of a tree increases only when two equally deep trees are joined (and then the height goes up by one). Thus, union-by-height is a trivial modification of union-by-size.

The following figures show a tree and its implicit representation for both union-by-size and union-by-height. The code in Figure 8.13 implements union-by-height.

8.5. Path Compression

The union/find algorithm, as described so far, is quite acceptable for most cases. It is very simple and linear on average for a sequence of m instructions (under all models). However, the worst case of O(m log n ) can occur fairly easily and naturally.

/* assume root1 and root2 are roots */

/* union is a C keyword, so this routine is named set_union */

void

set_union (DISJ_SET S, set_type root1, set_type root2 )

{

if( S[root2] < S[root1] )  /* root2 is deeper set */

S[root1] = root2;     /* make root2 new root */

else

{

if( S[root2] == S[root1] ) /* same height, so update */

S[root1]--;

S[root2] = root1; /* make root1 new root */

}

}

Figure 8.13 Code for union-by-height (rank)

For instance, if we put all the sets on a queue and repeatedly dequeue the first two sets and enqueue the union, the worst case occurs. If there are many more finds than unions, this running time is worse than that of the quick-find algorithm. Moreover, it should be clear that there are probably no more improvements possible for the union algorithm. This is based on the observation that any method to perform the unions will yield the same worst-case trees, since it must break ties arbitrarily. Therefore, the only way to speed the algorithm up, without reworking the data structure entirely, is to do something clever on the find operation.

The clever operation is known as path compression. Path compression is performed during a find operation and is independent of the strategy used to perform unions. Suppose the operation is find(x). Then the effect of path compression is that every node on the path from x to the root has its parent changed to the root. Figure 8.14 shows the effect of path compression after find (15) on the generic worst tree of Figure 8.12.

The effect of path compression is that with an extra two pointer moves, nodes 13 and 14 are now one position closer to the root and nodes 15 and 16 are now two positions closer. Thus, the fast future accesses on these nodes will pay (we hope) for the extra work to do the path compression.

As the code in Figure 8.15 shows, path compression is a trivial change to the basic find algorithm. The only change to the find routine is that S[x] is made equal to the value returned by find; thus after the root of the set is found recursively, x is made to point directly to it. This occurs recursively to every node on the path to the root, so this implements path compression. As we stated when we implemented stacks and queues, modifying a parameter to a function called is not necessarily in line with current software engineering rules. Some languages will not allow this, so this code may well need changes.

Figure 8.14 An example of path compression

set_type

find( element_type x, DISJ_SET S )

{

if( S[x] <= 0 )

return x;

else

return( S[x] = find( S[x], S ) );

}

Figure 8.15 Code for disjoint set find with path compression

When unions are done arbitrarily, path compression is a good idea, because there is an abundance of deep nodes and these are brought near the root by path compression. It has been proven that when path compression is done in this case, a sequence of m operations requires at most O(m log n) time. It is still an open problem to determine what the average-case behavior is in this situation.

Path compression is perfectly compatible with union-by-size, and thus both routines can be implemented at the same time. Since doing union-by-size by itself is expected to execute a sequence of m operations in linear time, it is not clear that the extra pass involved in path compression is worthwhile on average. Indeed, this problem is still open. However, as we shall see later, the combination of path compression and a smart union rule guarantees a very efficient algorithm in all cases.

Path compression is not entirely compatible with union-by-height, because path compression can change the heights of the trees. It is not at all clear how to re-compute them efficiently. The answer is do not!! Then the heights stored for each tree become estimated heights (sometimes known as ranks), but it turns out that union-by-rank (which is what this has now become) is just as efficient in theory as union-by-size. Furthermore, heights are updated less often than sizes. As with union-by-size, it is not clear whether path compression is worthwhile on average. What we will show in the next section is that with either union heuristic, path compression significantly reduces the worst-case running time.

8.6. Worst Case for Union-by-Rank and Path Compression

When both heuristics are used, the algorithm is almost linear in the worst case. Specifically, the time required in the worst case is   [image] (m  [image] (m, n)) (provided m   [image]  n), where   [image] (m, n) is a functional inverse of Ackerman's function, which is defined below:*

A(1, j) = 2j for j    1

A(i, 1) = A(i - 1, 2) for i    2

A(i, j) = A(i - 1,A(i, j - 1)) for i, j    2

*Ackerman's function is frequently defined with A(1, j) = j + 1 for j   [image]  1. the form in this text grows faster; thus, the inverse grows more slowly.

From this, we define

  (m, n) = min{i    1|A(i,   m/ n  ) > log n}

You may want to compute some values, but for all practical purposes,   [image] (m, n)   [image]  4, which is all that is really important here. The single-variable inverse Ackerman function, sometimes written as log*n, is the number of times the logarithm of n needs to be applied until n   [image]  1. Thus, log* 65536 = 4, because log log log log 65536 = 1. log* 265536 = 5, but keep in mind that 265536 is a 20,000-digit number.   (m, n) actually grows even slower then log* n. However,   [image] (m, n) is not a constant, so the running time is not linear.

In the remainder of this section, we will prove a slightly weaker result. We will show that any sequence of m =   [image] (n) union/find operations takes a total of O(m log* n) running time. The same bound holds if union-by-rank is replaced with union-by-size. This analysis is probably the most complex in the book and one of the first truly complex worst-case analyses ever performed for an algorithm that is essentially trivial to implement.

8.6.1 Analysis of the Union/Find Algorithm

8.6.1 Analysis of the Union/Find Algorithm

In this section we establish a fairly tight bound on the running time of a sequence of m =   [image] (n) union/find operations. The unions and finds may occur in any order, but unions are done by rank and finds are done with path compression.

We begin by establishing some lemmas concerning the number of nodes of rank r. Intuitively, because of the union-by-rank rule, there are many more nodes of small rank than large rank. In particular, there can be at most one node of rank log n. What we would like to do is to produce as precise a bound as possible on the number of nodes of any particular rank r. Since ranks only change when unions are performed (and then only when the two trees have the same rank), we can prove this bound by ignoring the path compression.

LEMMA 8.1.

When executing a sequence of union instructions, a node of rank r must have 2r descendants (including itself).

PROOF:

By induction. the basis, r = 0, is clearly true. Let T be the tree of rank r with the fewest number of descendants and let x be T's root. Suppose the last union x was involved in was between T1 and T2. Suppose T1's root was x. If T1 had rank r, then T1 would be a tree of height r with fewer descendants than T, which contradicts the assumption that T is the tree with the smallest number of descendants. Hence the rank of T1   [image]  r - 1. The rank of T2   [image]  rank of T1. Since T has rank r and the rank could only increase because of T2, it follows that the rank of T2 = r - 1. Then the rank of T1 = r - 1. By the induction hypothesis, each tree has at least 2r-1 descendants, giving a total of 2r and establishing the lemma.

Lemma 8.1 tells us that if no path compression is performed, then any node of rank r must have at least 2r descendants. Path compression can change this, of course, since it can remove descendants from a node. However, when unions are performed, even with path compression, we are using the ranks, which are estimated heights. These ranks behave as though there is no path compression. Thus, when bounding the number of nodes of rank r, path compression can be ignored.

Thus, the next lemma is valid with or without path compression.

LEMMA 8.2.

The number of nodes of rank r is at most n/2r.

PROOF:

Without path compression, each node of rank r is the root of a subtree of at least 2r nodes. No node in the subtree can have rank r. Thus all subtrees of nodes of rank r are disjoint. Therefore, there are at most n/2r disjoint subtrees and hence n/2r nodes of rank r.

The next lemma seems somewhat obvious, but is crucial in the analysis.

LEMMA 8.3.

At any point in the union/find algorithm, the ranks of the nodes on a path from the left to a root increase monotonically.

PROOF:

The lemma is obvious if there is no path compression (see the example). If, after path compression, some node v is a descendant of w, then clearly v must have been a descendant of w when only unions were considered. Hence the rank of v is less than the rank of w.

Let us summarize the preliminary results. Lemma 8.2 tells us how many nodes can be assigned rank r. Because ranks are assigned only by unions, which have no idea of path compression, Lemma 8.2 is valid at any stage of the union/find algorithm--even in the midst of path compression. Figure 8.16 shows that while there are many nodes of ranks 0 and 1, there are fewer nodes of rank r as r gets larger.

Lemma 8.2 is tight, in the sense that it is possible for there to be n/2r nodes for any rank r. It is slightly loose, because it is not possible for the bound to hold for all ranks r simultaneously. While Lemma 8.2 describes the number of nodes in a rank r, Lemma 8.3 tells us their distribution. As one would expect, the rank of nodes is strictly increasing along the path from a leaf to the root.

We are now ready to prove the main theorem. Our basic idea is as follows: A find on any node v costs time proportional to the number of nodes on the path from v to the root. Let us, then, charge one unit of cost for every node on the path from v to the root for each find. To help us count the charges, we will deposit an imaginary penny into each node on the path. This is strictly an accounting gimmick, which is not part of the program. When the algorithm is over, we collect all the coins that have been deposited; this is the total cost.

As a further accounting gimmick, we deposit both American and Canadian pennies. We will show that during the execution of the algorithm, we can deposit only a certain number of American pennies during each find. We will also show that we can deposit only a certain number of Canadian pennies to each node. Adding these two totals gives us a bound on the total number of pennies that can be deposited.

We now sketch our accounting scheme in a little more detail. We will divide the nodes by their ranks. We then divide the ranks into rank groups. On each find, we will deposit some American coins into the general kitty and some Canadian coins into specific vertices. To compute the total number of Canadian coins deposited, we will compute the deposits per node. By adding up all the deposits for each node in rank r, we will get the total deposits per rank r. Then we will add up all the deposits for each rank r in group g and thereby obtain the total deposits for each rank group g. Finally, we add up all the deposits for each rank group g to obtain the total number of Canadian coins deposited in the forest. Adding this to the number of American coins in the kitty gives us the answer.

We will partition ranks into groups. Ranks r goes into group G(r), and G will be determined later. The largest rank in any rank group g is F(g), where F = G-1 is the inverse of G. The number of ranks in any rank group, g > 0, is thus F(g) - F(g - 1). Clearly G(n) is a very loose upper bound on the largest rank group. As an example, suppose that we partitioned the ranks as in Figure 8.7. In this case,   [image]. The largest rank in group g is F(g) = g2, and observe that group g > 0 contains ranks F(g - 1) + 1 through F(g) inclusive. This formula does not apply for rank group 0, so for convenience we will ensure that rank group 0 contains only elements of rank 0. Notice that the groups are made of consecutive ranks.

Figure 8.16 A large disjoint set tree (numbers below nodes are ranks)

As mentioned before, each union instruction takes constant time, as long as each root keeps track of how big its subtrees are. Thus, unions are essentially free, as far as this proof goes.

Each find(i) takes time proportional to the number of vertices on the path from the vertex representing i to the root. We will thus deposit one penny for each vertex on the path. If this is all we do, however, we cannot expect much of a bound, because we are not taking advantage of path compression. Thus, we need to take advantage of path compression in our analysis. We will use fancy accounting.

For each vertex, v, on the path from the vertex representing i to the root, we deposit one penny under one of two accounts:

1.     If v is the root, or if the parent of v is the root, or if the parent of v is in a different rank group from v, then charge one unit under this rule. This deposits an American penny into the kitty.

2.     Otherwise deposit a Canadian penny into the vertex.

LEMMA 8.4.

For any find (v), the total number of pennies deposited, either into the kitty or into a vertex, is exactly equal to the number of nodes on the path from v to the root.

PROOF:

Obvious.

Thus all we need to do is to sum all the American pennies deposited under rule 1 with all the Canadian pennies deposited under rule 2.

We are doing at most m finds. We need to bound the number of pennies that can be deposited into the kitty during a find.

LEMMA 8.5

Over the entire algorithm, the total deposits of American pennies under rule 1 amount to m(G(n) + 2).

Group           Rank

------------------------------

  0               0

  1               1

  2             2,3,4

  3          5 through 9

  4         10 through 16

  i      (i-1)2 + 1 through i2

Figure 8.17 Possible partitioning of ranks into groups

PROOF:

This is easy. For any find, two American pennies are deposited, because of the root and its child. By Lemma 8.3, the vertices going up the path are monotonically increasing in rank, and since there are at most G(n) rank groups, only G(n) other vertices on the path can qualify as a rule 1 deposit for any particular find. Thus, during any one find, at most G(n) + 2 American  pennies can be placed in the kitty. Thus, at most m(G(n) + 2) American pennies can be deposited under rule 1 for a sequence of m finds.

To get a good estimate for all the Canadian deposits under rule 2, we  will add up the deposits by vertices instead of by find instructions. If a coin is deposited into vertex v under rule 2, v will be moved by path compression and get a new parent of higher rank than its old parent. (This is where we are using the fact that path compression is being done.) Thus, a vertex v in rank group g > 0 can be moved at most F(g) - F(g - 1) times before its parent gets pushed out of rank group g, since that is the size of the rank group.* After this happens, all future charges to v will go under rule 1.

*This can be reduced by 1. We do not for the sake of clarity; the bound is not improved by being more careful here.

LEMMA 8.6.

The number of vertices, N(g), in rank group g > 0 is at most n/2F(g - 1).

PROOF:

By Lemma 8.2, there are at most n/2r vertices of rank r. Summing over  the ranks in group g, we obtain

LEMMA 8.7.

The number of Canadian pennies deposited into all vertices in rank group g is at most nF(g)/2F(g - 1).

PROOF:

Each vertex in the rank group can receive at most F(g) - F(g - 1)   [image]  F(g) Canadian pennies while its parent stays in its rank group, and Lemma 8.6 tells how many such vertices there are. The result is obtained by a simple multiplication.

LEMMA 8.8.

The total deposit under rule 2 is at most n   [image]  Canadian pennies.

PROOF:

Because rank group 0 contains only elements of rank 0, it cannot contribute to rule 2 charges (it cannot have a parent in the same rank group). The bound is obtained by summing the other rank groups.

Thus we have the deposits under rules 1 and 2. The total is

(8.1)

We still have not specified G(n) or its inverse F(n). Obviously, we are free to choose virtually anything we want, but it makes sense to choose G(n) to minimize the bound above. However, if G(n) is too small, then F(n) will be large, hurting the bound. An apparently good choice is to choose F(i) to be the function recursively defined by F(0) = 0 and F(i) = 2F(i - 1). This gives G(n) = 1 +[log* n]. Figure 8.18 shows how this partitions the ranks. Notice that group 0 contains only rank 0, which we required in the previous lemma. F is very similar to the single-variable Ackerman function, which differs only in the definition of the base case (F(0) = 1).

THEOREM 8.1.

The running time of m unions and finds is O(m log* n).

Group         Rank

---------------------------

  0               0

  1               1

  2               2

  3              3,4

  4         5 through 16

  5        17 through 216

  6      65537 through 265536

  7        truly huge ranks

Figure 8.18 Actual partitioning of ranks into groups used in the proof

PROOF:

Plug in the definitions of F and G into Equation (8.1). The total number of American pennies is O(mG(n)) = O(m log* n). The total number of Canadian pennies is   [image]. Since m =   [image] (n), the bound follows.

What the analysis shows is that there are few nodes that could be moved frequently by path compression, and thus the total time spent is relatively small.

8.7. An Application

As an example of how this data structure might be used, consider the following problem. We have a network of computers and a list of bidirectional connections; each of these connections allows a file transfer from one computer to another. Is it possible to send a file from any computer on the network to any other? An extra restriction is that the problem must be solved on-line. Thus, the list of connections is presented one at a time, and the algorithm must be prepared to give an answer at any point.

An algorithm to solve this problem can initially put every computer in its own set. Our invariant is that two computers can transfer files if and only if they are in the same set. We can see that the ability to transfer files forms an equivalence relation. We then read connections one at a time. When we read some connection, say (u, v), we test to see whether u and v are in the same set and do nothing if they are. If they are in different sets, we merge their sets. At the end of the algorithm, the graph is connected if and only if there is exactly one set. If there are m connections and n computers, the space requirement is O(n). Using union-by-size and path compression, we obtain a worst-case running time of O(m  [image] (m, n)), since there are 2m finds and at most n - 1 unions. This running time is linear for all practical purposes.

We will see a much better application in the next chapter.

Summary

We have seen a very simple data structure to maintain disjoint sets. When the union operation is performed, it does not matter, as far as correctness is concerned, which set retains its name. A valuable lesson that should be learned here is that it can be very important to consider the alternatives when a particular step is not totally specified. The union step is flexible; by taking advantage of this, we are able to get a much more efficient algorithm.

Path compression is one of the earliest forms of self-adjustment, which we have seen elsewhere (splay trees, skew heaps). Its use is extremely interesting, especially from a theoretical point of view, because it was one of the first examples of a simple algorithm with a not-so-simple worst-case analysis.

Exercises

8.1     Show the result of the following sequence of instructions: union(1, 2), union(3, 4), union(3, 5), union(1, 7), union(3, 6), union(8, 9), union(1, 8), union(3, 10), union(3, 11), union(3, 12), union(3, 13), union(14, 15), union(16, 17), union(14, 16), union(1, 3), union(1, 14), when the unions are

a.     performed arbitrarily

b.     performed by height

c.     performed by size

8.2     For each of the trees in the previous exercise, perform a find with path compression on the deepest node.

8.3     Write a program to determine the effects of path compression and the various unioning strategies. Your program should process a long sequence of equivalence operations using all six of the possible strategies.

8.4     Show that if unions are performed by height, then the depth of any tree is O(log n) .

8.5     a.     Show that if m = n2, then the running time of m union/find operations is O(m).

b.     Show that if m = n log n, then the running time of m union/find operations is O(m) .

*c.     Suppose m =   [image] (n log log n). What is the running time of m union/find operations?

*d.     Suppose m =   [image] (n log* n). What is the running time of m union/find operations?

8.6     Show the operation of the program in Section 8.7 on the following graph: (1,2), (3,4), (3,6), (5,7), (4,6), (2,4), (8,9), (5,8). What are the connected components?

8.7     Write a program to implement the algorithm in Section 8.7.

*8.8     Suppose we want to add an extra operation, deunion, which undoes the last union operation that has not been already undone.

a.     Show that if we do union-by-height and finds without path compression, then deunion is easy and a sequence of m union, find, and deunion operations take O(m log n) time.

b.     Why does path compression make deunion hard?

**c.     Show how to implement all three operations so that the sequence of m operations takes O(m log n/log log n) time.

* 8.9     Suppose we want to add an extra operation, remove(x), which removes x from its current set and places it in its own. Show how to modify the union/find algorithm so that the running time of a sequence of m union, find, and remove operations is O(m  [image] (m, n)).

**8.10     Give an algorithm that takes as input an n-vertex tree and a list of n pairs of vertices and determines for each pair (v, w) the closest common ancestor of v and w. Your algorithm should run in O(n log* n).

*8.11     Show that if all of the unions precede the finds, then the disjoint set algorithm with path compression requires linear time, even if the unions are done arbitrarily.

**8.12     Prove that if unions are done arbitrarily, but path compression is performed on the finds, then the worst-case running time is   [image] (m log n).

8.13     Prove that if unions are done by size and path compression is performed, the worst-case running time is O(m log* n).

8.14     Suppose we implement partial path compression on find(i) by making every other node on the path from i to the root point to its grandparent (where this makes sense). This is known as path halving.

a.     Write a procedure to do this.

b.     Prove that if path halving is performed on the finds and either union-by-height or union-by-size is used, the worst-case running time is O(m log* n).

References

Various solutions to the union/find problem can be found in [5], [8], and [10]. Hopcroft and Ullman showed the O(m log* n) bound of Section 8.6. Tarjan [14] obtained the bound O(m  [image] (m,n)). A more precise (but asymptotically identical) bound for m < n appears in [2] and [17]. Various other strategies for path compression and unions also achieve the same bound; see [17] for details.

A lower bound showing that under certain restrictions   [image] (m  [image] (m,n)) time is required to process m union/find operations was given by Tarjan [15]. Identical bounds under less restrictive conditions have been recently shown in [6] and [13].

Applications of the union/find data structure appear in [1] and [9]. Certain special cases of the union/find problem can be solved in O(m) time [7]. This reduces the running time of several algorithms, such as [1], graph dominance, and reducibility (see references in Chapter 9) by a factor of   [image] (m,n). Others, such as [9] and the graph connectivity problem in this chapter, are unaffected. The paper lists 10 examples. Tarjan has used path compression to obtain efficient algorithms for several graph problems [16].

Average-case results for the union/find problem appear in [4], [11], and [19]. Results bounding the running time of any single operation (as opposed to the entire sequence) appear in [3] and [12].

Exercise 8.8 is solved in [18].

1.     A. V. Aho, J. E. Hopcroft, J. D. Ullman, "On Finding Lowest Common Ancestors in Trees," SIAM Journal on Computing 5 (1976), 115-132.

2.     L. Banachowski, "A Complement to Tarjan's Result about the Lower Bound on the Complexity of the Set Union Problem," Information Processing Letters 11 (1980), 59-65.

3.     N. Blum, "On the Single-operation Worst-case Time Complexity of the Disjoint Set Union Problem," SIAM Journal on Computing 15 (1986), 1021-1024.

4.     J. Doyle and R. L. Rivest, "Linear Expected Time of a Simple Union Find Algorithm," Information Processing Letters 5 (1976), 146-148.

5.     M. J. Fischer, "Efficiency of Equivalence Algorithms," Complexity of Computer Computation (eds. R. E. Miller and J. W. Thatcher), Plenum Press, 1972, 153-168.

6.     M. L. Fredman and M. E. Saks, "The Cell Probe Complexity of Dynamic Data Structures," Proceedings of the Twenty-first Annual Symposium on Theory of Computing (1989), 345-354.

7.     H. N. Gabow and R. E. Tarjan, "A Linear-time Algorithm for a Special Case of Disjoint Set Union,"Journal of Computer and System Sciences 30 (1985), 209-221.

8.     B. A. Galler and M. J. Fischer, "An Improved Equivalence Algorithm," Communications of the ACM 7 (1964), 301-303.

9.     J. E. Hopcroft and R. M. Karp, "An Algorithm for Testing the Equivalence of Finite Automata," Technical Report TR-71-114, Department of Computer Science, Cornell University, Ithaca, NY, 1971.

10.     J. E. Hopcroft and J. D. Ullman, "Set Merging Algorithms," SIAM Journal on Computing 2 (1973), 294-303.

11.     D. E. Knuth and A. Schonhage, "The Expected Linearity of a Simple Equivalence Algorithm," Theoretical Computer Science 6 (1978), 281-315.

12.     J. A. LaPoutre, "New Techniques for the Union-Find Problem," Proceedings of the First Annual ACM-SIAM Symposium on Discrete Algorithms (1990), 54-63.

13.     J. A. LaPoutre, "Lower Bounds for the Union-Find and the Split-Find Problem on Pointer Machines," Proceedings of the Twenty Second Annual ACM Symposium on Theory of Computing (1990), 34-44.

14.     R. E. Tarjan, "Efficiency of a Good but Not Linear Set Union Algorithm," Journal of the ACM 22 (1975), 215-225.

15.     R. E. Tarjan, "A Class of Algorithms Which Require Nonlinear Time to Maintain Disjoint Sets," Journal of Computer and System Sciences 18 (1979), 110-127.

16.     R. E. Tarjan, "Applications of Path Compression on Balanced Trees," Journal of the ACM 26 (1979), 690-715.

17.     R. E. Tarjan and J. van Leeuwen, "Worst Case Analysis of Set Union Algorithms," Journal of the ACM 31 (1984), 245-281.

18.     J. Westbrook and R. E. Tarjan, "Amortized Analysis of Algorithms for Set Union with Backtracking," SIAM Journal on Computing 18 (1989), 1-11.

19.     A. C. Yao, "On the Average Behavior of Set Merging Algorithms," Proceedings of Eighth Annual ACM Symposium on the Theory of Computation (1976), 192-195.

Go to Chapter 9            Return to Table of Contents


Structures, Algorithm Analysis: CHAPTER 9: GRAPH ALGORITHMS

 Return to Table of Contents

CHAPTER 9: GRAPH ALGORITHMS

In this chapter we discuss several common problems in graph theory. Not only are these algorithms useful in practice, they are interesting because in many real-life applications they are too slow unless careful attention is paid to the choice of data structures. We will

Show several real-life problems, which can be converted to problems on graphs.

Give algorithms to solve several common graph problems.

Show how the proper choice of data structures can drastically reduce the running time of these algorithms.

See an important technique, known as depth-first search, and show how it can be used to solve several seemingly nontrivial problems in linear time.

9.1 Definitions

A graph G = (V, E) consists of a set of vertices, V, and a set of edges, E. Each edge is a pair (v,w), where v,w [image] V. Edges are sometimes referred to as arcs. If the pair is ordered, then the graph is directed. Directed graphs are sometimes referred to as digraphs. Vertex w is adjacent to v if and only if (v,w) [image] E. In an undirected graph with edge (v,w), and hence (w,v), w is adjacent to v and v is adjacent to w. Sometimes an edge has a third component, known as either a weight or a cost.

A path in a graph is a sequence of verices w1, w2, w3, . . . , wn such that (wi, wi+i) [image] E for 1 [image] i < n. The length of such a path is the number of edges on the path, which is equal to n - 1. We allow a path from a vertex to itself; if this path contains no edges, then the path lenght is 0. This is a convenient way to define an otherwise special case. If the graph contains an edge (v,v) from a vertex to itself, then the path v, v is sometimes referred to as a loop. The graphs we will consider will generally be loopless. A simple path is a path such that all vertices are distinct, except that the first and last could be the same.

A cycle in a directed graph is a path of length at least 1 such that w1 = wn; this cycle is simple if the path is simple. For undirected graphs, we require that the edges be distinct. The logic of these requirements is that the path u, v, u in an undirected graph should not be considered a cycle, because (u, v) and (v, u) are the same edge. In a directed graph, these are different edges, so it makes sense to call this a cycle. A directed graph is acyclic if it has no cycles. A directed acyclic graph is sometimes referred to by its abbreviation, DAG.

An undirected graph is connected if there is a path from every vertex to every other vertex. A directed graph with this property is called strongly connected. If a directed graph is not strongly connected, but the underlying graph (without direction to the arcs) is connected, then the graph is said to be weakly connected. A complete graph is a graph in which there is an edge between every pair of vertices.

An example of a real-life situation that can be modeled by a graph is the airport system. Each airport is a vertex, and two vertices are connected by an edge if there is a nonstop flight from the airports that are represented by the vertices. The edge could have a weight, representing the time, distance, or cost of the flight. It is reasonable to assume that such a graph is directed, since it might take longer or cost more (depending on local taxes, for example) to fly in different directions. We would probably like to make sure that the airport system is strongly connected, so that it is always possible to fly from any airport to any other airport. We might also like to quickly determine the best flight between any two airports. "Best" could mean the path with the fewest number of edges or could be taken with respect to one, or all, of the weight measures.

Traffic flow can be modeled by a graph. Each street intersection represents a vertex, and each street is an edge. The edge costs could represent, among other things, a speed limit or a capacity (number of lanes). We could then ask for the shortest route or use this information to find the most likely location for bottlenecks.

In the remainder of this chapter, we will see several more applications of graphs. Many of these graphs can be quite large, so it is important that the algorithms we use be efficient.

9.1.1. Representation of Graphs

We will consider directed graphs (undirected graphs are similarly represented).

Suppose, for now, that we can number the vertices, starting at 1. The graph shown in Figure 9.1 represents 7 vertices and 12 edges.

One simple way to represent a graph is to use a two-dimensional array. This is known as an adjacency matrix representation. For each edge (u, v), we set a[u][v] = 1; otherwise the entry in the array is 0. If the edge has a weight associated with it, then we can set a[u][v] equal to the weight and use either a very large or a very small weight as a sentinel to indicate nonexistent edges. For instance, if we were looking for the cheapest airplane route, we could represent nonexistent flights with a cost of [image] . If we were looking, for some strange reason, for the most expensive airplane route, we could use - [image] (or perhaps 0) to represent nonexistent edges.

Although this has the merit of extreme simplicity, the space requirement is [image] (|V|2), which can be prohibitive if the graph does not have very many edges. An adjacency matrix is an appropriate representation if the graph is dense: |E| = [image] (|V|2). In most of the applications that we shall see, this is not true. For instance, suppose the graph represents a street map. Assume a Manhattan-like orientation, where almost all the streets run either north-south or east-west. Therefore, any intersection is attached to roughly four streets, so if the graph is directed and all streets are two-way, then |E| [image] 4|V|. If there are 3,000 intersections, then we have a 3,000-vertex graph with 12,000 edge entries, which would require an array of size nine million. Most of these entries would contain zero. This is intuitively bad, because we want our data structures to represent the data that is actually there and not the data that is not present.

Figure 9.1 A directed graph

If the graph is not dense, in other words, if the graph is sparse, a better solution is an adjacency list representation. For each vertex, we keep a list of all adjacent vertices. The space requirement is then O|E| + |V|). The leftmost structure in Figure 9.2 is merely an array of header cells. The representation should be clear from Figure 9.2. If the edges have weights, then this additional information is also stored in the cells.

Adjacency lists are the standard way to represent graphs. Undirected graphs can be similarly represented; each edge (u, v) appears in two lists, so the space usage essentially doubles. A common requirement in graph algorithms is to find all vertices adjacent to some given vertex v, and this can be done, in time proportional to the number of such vertices found, by a simple scan down the appropriate adjacency list.

In most real-life applications, the vertices have names, which are unknown at compile time, instead of numbers. Since we cannot index an array by an unknown name, we must provide a mapping of names to numbers. The easiest way to do this is to use a hash table, in which we store a name and an internal number ranging from 1 to |V| for each vertex. The numbers are assigned as the graph is read. The first number assigned is 1. As each edge is input, we check whether each of the two vertices has been assigned a number, by seeing if it is in the hash table. If so, we use the internal number. Otherwise, we assign to the vertex the next available number and insert the vertex name and number into the hash table.

With this transformation, all the graph algorithms will use only the internal numbers. Since eventually we will need to output the real vertex names and not the internal numbers, we must also record, for each internal number, the corresponding vertex name. One way is to use an array of strings. If the vertex names are long, this can cost considerable space, because the vertex names are stored twice. An alternative is to keep an array of pointers into the hash table. The price of this alternative is a slight loss of the sanctity of the hash table ADT.

Figure 9.2 An adjacency list representation of a graph

The code that we present in this chapter will be pseudocode using ADTS as much as possible. We will do this to save space and, of course, to make the algorithmic presentation of the algorithms much clearer.

9.2. Topological Sort

A topological sort is an ordering of vertices in a directed acyclic graph, such that if there is a path from vi to vj, then vj appears after vi in the ordering. The graph in Figure 9.3 represents the course prerequisite structure at a state university in Miami. A directed edge (v,w) indicates that course v must be completed before course w may be attempted. A topological ordering of these courses is any course sequence that does not violate the prerequisite requirement.

It is clear that a topological ordering is not possible if the graph has a cycle, since for two vertices v and w on the cycle, v precedes w and w precedes v. Furthermore, the ordering is not necessarily unique; any legal ordering will do. In the graph in Figure 9.4, v1, v2, v5, v4, v3, v7, v6 and v1, v2, v5, v4, v7, v3, v6 are both topological orderings.

A simple algorithm to find a topological ordering is first to find any vertex with no incoming edges. We can then print this vertex, and remove it, along with its edges, from the graph. Then we apply this same strategy to the rest of the graph.

Figure 9.3 An acyclic graph representing course prerequisite structure

Figure 9.4 An acyclic graph

To formalize this, we define the indegree of a vertex v as the number of edges (u,v). We compute the indegrees of all vertices in the graph. Assuming that the indegree array is initialized and that the graph is read into an adjacency list, we can then apply the algorithm in Figure 9.5 to generate a topological ordering.

The function find_new_vertex_of_indegree_zero scans the indegree array looking for a vertex with indegree 0 that has not already been assigned a topological number. It returns NOT_A_VERTEX if no such vertex exists; this indicates that the graph has a cycle.

void

topsort( graph G )

{

unsigned int counter;

vertex v, w;

for( counter =  0; counter < NUM_VERTEX; counter++ )

{

v = find_new_vertex_of_indegree_zero( );

if( v = NOT_A_VERTEX )

{

error("Graph has a cycle");

break;

}

top_num[v] = counter;

for each w adjacent to v

indegree[w]--;

}

}

Figure 9.5 Simple topological sort pseudocode

Because find_new_vertex_of_indegree_zero is a simple sequential scan of the indegree array, each call to it takes O(|V|) time. Since there are |V| such calls, the running time of the algorithm is O(|V|2).

By paying more careful attention to the data structures, it is possible to do better. The cause of the poor running time is the sequential scan through the indegree array. If the graph is sparse, we would expect that only a few vertices have their indegrees updated during each iteration. However, in the search for a vertex of indegree 0, we look at (potentially) all the vertices, even though only a few have changed.

We can remove this inefficiency by keeping all the (unassigned) vertices of indegree 0 in a special box. The find_new_vertex_of_indegree_zero function then returns (and removes) any vertex in the box. When we decrement the indegrees of the adjacent vertices, we check each vertex and place it in the box if its indegree falls to 0.

To implement the box, we can use either a stack or a queue. First, the indegree is computed for every vertex. Then all vertices of indegree 0 are placed on an initially empty queue. While the queue is not empty, a vertex v is removed, and all edges adjacent to v have their indegrees decremented. A vertex is put on the queue as soon as its indegree falls to 0. The topological ordering then is the order in which the vertices dequeue. Figure 9.6 shows the status after each phase.

A pseudocode implementation of this algorithm is given in Figure 9.7. As before, we will assume that the graph is already read into an adjacency list and that the indegrees are computed and placed in an array. A convenient way of doing this in practice would be to place the indegree of each vertex in the header cell. We also assume an array top_num, in which to place the topological numbering.

       Indegree Before Dequeue #

  Vertex    1   2   3   4   5   6   7

--------------------------------------

    v1      0   0   0   0   0   0   0

    v2      1   0   0   0   0   0   0

    v3      2   1   1   1   0   0   0

    v4      3   2   1   0   0   0   0

    v5      1   1   0   0   0   0   0

    v6      3   3   3   3   2   1   0

    v7      2   2   2   1   0   0   0

--------------------------------------

  enqueue  v1  v2  v5  v4  v3  v7  v6

--------------------------------------

  dequeue  v1  v2  v5  v4  v3  v7  v6

Figure 9.6 Result of applying topological sort to the graph in Figure 9.4

void

topsort( graph G )

{

QUEUE Q;

unsigned int counter;

vertex v, w;

/*1*/        Q = create_queue( NUM_VERTEX ); make_null( Q ); counter =  0;

/*2*/        for each vertex v

/*3*/             if( indegree[v] =   0 )

/*4*/                  enqueue( v, Q );

/*5*/        while( !is_empty( Q ) )

{

/*6*/             v = dequeue( Q );

/*7*/             top_num[v] = ++counter; /* assign next number */

/*8*/             for each w adjacent to v

/*9*/                  if( --indegree[w] =  0 )

/*10*/                      enqueue( w, Q );

}

/*11*/       if( counter != NUM_VERTEX )

/*12*/            error("Graph has a cycle");

/*13*/       dispose_queue( Q ); /* free the memory */

}

Figure 9.7 Pseudocode to perform topological sort

The time to perform this algorithm is O(|E| + |V|) if adjacency lists are used. This is apparent when one realizes that the body of the for loop at line 8 is executed at most once per edge. The queue operations are done at most once per vertex, and the initialization steps also take time proportional to the size of the graph.

9.3. Shortest-Path Algorithms

In this section we examine various shortest-path problems. The input is a weighted graph: associated with each edge (vi, vj) is a cost ci,j to traverse the arc. The cost of a path v1v2 ... vn is   [image]  This is referred to as the weighted path length. The unweighted path length is merely the number of edges on the path, namely, n - 1.

SINGLE-SOURCE SHORTEST-PATH PROBLEM:

Given as input a weighted graph, G = (V, E), and a distinguished vertex, s , find the shortest weighted path from s to every other vertex in G.

For example, in the graph in Figure 9.8, the shortest weighted path from v1 to v6 has a cost of 6 and goes from v1 to v4 to v7 to v6. The shortest unweighted path between these vertices is 2. Generally, when it is not specified whether we are referring to a weighted or an unweighted path, the path is weighted if the graph is. Notice also that in this graph there is no path from v6 to v1.

The graph in the preceding example has no edges of negative cost. The graph in Figure 9.9 shows the problems that negative edges can cause. The path from v5 to v4 has cost 1, but a shorter path exists by following the loop v5, v4, v2, v5, v4, which has cost -5. This path is still not the shortest, because we could stay in the loop arbitrarily long. Thus, the shortest path between these two points is undefined. Similarly, the shortest path from v1 to v6 is undefined, because we can get into the same loop. This loop is known as a negative-cost cycle; when one is present in the graph, the shortest paths are not defined. Negative-cost edges are not necessarily bad, as the cycles are, but their presence seems to make the problem harder. For convenience, in the absence of a negative-cost cycle, the shortest path from s to s is zero.

Figure 9.8 A directed graph G

Figure 9.9 A graph with a negative-cost cycle

There are many examples where we might want to solve the shortest-path problem. If the vertices represent computers; the edges represent a link between computers; and the costs represent communication costs (phone bill per 1,000 bytes of data), delay costs (number of seconds required to transmit 1,000 bytes), or a combination of these and other factors, then we can use the shortest-path algorithm to find the cheapest way to send electronic news from one computer to a set of other computers.

We can model airplane or other mass transit routes by graphs and use a shortest-path algorithm to compute the best route between two points. In this and many practical applications, we might want to find the shortest path from one vertex, s, to only one other vertex, t. Currently there are no algorithms in which finding the path from s to one vertex is any faster (by more than a constant factor) than finding the path from s to all vertices.

We will examine algorithms to solve four versions of this problem. First, we will consider the unweighted shortest-path problem and show how to solve it in O(|E| + |V|). Next, we will show how to solve the weighted shortest-path problem if we assume that there are no negative edges. The running time for this algorithm is O (|E| log |V|) when implemented with reasonable data structures.

If the graph has negative edges, we will provide a simple solution, which unfortunately has a poor time bound of O (|E|   [image]  |V|). Finally, we will solve the weighted problem for the special case of acyclic graphs in linear time.

9.3.1. Unweighted Shortest Paths

Figure 9.10 shows an unweighted graph, G. Using some vertex, s, which is an input parameter, we would like to find the shortest path from s to all other vertices. We are only interested in the number of edges contained on the path, so there are no weights on the edges. This is clearly a special case of the weighted shortest-path problem, since we could assign all edges a weight of 1.

For now, suppose we are interested only in the length of the shortest paths, not in the actual paths themselves. Keeping track of the actual paths will turn out to be a matter of simple bookkeeping.

Figure 9.10 An unweighted directed graph G

Suppose we choose s to be v3. Immediately, we can tell that the shortest path  from s to v3 is then a path of length 0. We can mark this information, obtaining the graph in Figure 9.11.

Now we can start looking for all vertices that are a distance 1 away from s. These can be found by looking at the vertices that are adjacent to s. If we do this, we see that v1 and v6 are one edge from s. This is shown in Figure 9.12.

Figure 9.11 Graph after marking the start node as reachable in zero edges

Figure 9.12 Graph after finding all vertices whose path length from s is 1

Figure 9.13 Graph after finding all vertices whose shortest path is 2

We can now find vertices whose shortest path from s is exactly 2, by finding all the vertices adjacent to v1 and v6 (the vertices at distance 1), whose shortest paths are not already known. This search tells us that the shortest path to v2 and v4 is 2. Figure 9.13 shows the progress that has been made so far.

Finally we can find, by examining vertices adjacent to the recently evaluated v2 and v4, that v5 and v7 have a shortest path of three edges. All vertices have now been calculated, and so Figure 9.14 shows the final result of the algorithm.

This strategy for searching a graph is known as breadth-first search. It operates by processing vertices in layers: the vertices closest to the start are evaluated first, and the most distant vertices are evaluated last. This is much the same as a level-order traversal for trees.

Given this strategy, we must translate it into code. Figure 9.15 shows the initial configuration of the table that our algorithm will use to keep track of its progress.

For each vertex, we will keep track of three pieces of information. First, we will keep its distance from s in the entry dv. Initially all vertices are unreachable except for s, whose path length is 0. The entry in pv is the bookkeeping variable, which will allow us to print the actual paths. The entry known is set to 1 after a vertex is processed. Initially, all entries are unknown, including the start vertex.

Figure 9.14 Final shortest paths

  v   Known  dv  pv

------------------

  v1    0         0

  v2    0         0

  v3    0    0   0

  v4    0         0

  v5    0         0

  v6    0         0

  v7    0         0

Figure 9.15 Initial configuration of table used in unweighted shortest-path computation

When a vertex is known, we have a guarantee that no cheaper path will ever be found, and so processing for that vertex is essentially complete.

The basic algorithm can be described in Figure 9.16. The algorithm in Figure 9.16 mimics the diagrams by declaring as known the vertices at distance d = 0, then d = 1, then d = 2, and so on, and setting all the adjacent vertices w that still have dw =   [image]  to a distance dw = d + 1.

By tracing back through the pv variable, the actual path can be printed. We will see how when we discuss the weighted case.

The running time of the algorithm is O(|V|2), because of the doubly nested for loops. An obvious inefficiency is that the outside loop continues until NUM_VERTEX -1, even if all the vertices become known much earlier. Although an extra test could be made to avoid this, it does not affect the worst-case running time, as can be seen by generalizing what happens when the input is the graph in Figure 9.17 with start vertex v9.

void

unweighted( TABLE T )   /* assume T is initialized */

{

unsigned int curr_dist;

vertex v, w;

/*1*/        for( curr_dist = 0; curr_dist < NUM_VERTEX; curr_dist++)

/*2*/             for each vertex v

/*3*/                  if( ( !T[v].known ) && ( T[v].dist = curr_dist ) )

{

/*4*/                       T[v].known = TRUE;

/*5*/                       for each w adjacent to v

/*6*/                            if( T[w].dist = INT_MAX )

{

/*7*/                                 T[w].dist = curr_dist + 1;

/*8*/                                 T[w].path = v;

}

}

}

Figure 9.16 Pseudocode for unweighted shortest-path algorithm

Figure 9.17 A bad case for unweighted shortest-path algorithm without data structures

We can remove the inefficiency in much the same way as was done for topological sort. At any point in time, there are only two types of unknown vertices that have dv   [image]    [image]. Some have dv = curr_dist, and the rest have dv = curr_dist + 1. Because of this extra structure, it is very wasteful to search through the entire table to find a proper vertex at lines 2 and 3.

A very simple but abstract solution is to keep two boxes. Box #1 will have the unknown vertices with dv = curr_dist, and box #2 will have dv = curr_dist + 1. The test at lines 2 and 3 can be replaced by finding any vertex in box #1. After line 8 (inside the if block), we can add w to box #2. After the outside for loop terminates, box #1 is empty, and box #2 can be transferred to box #1 for the next pass of the for loop.

We can refine this idea even further by using just one queue. At the start of the pass, the queue contains only vertices of distance curr_dist. When we add adjacent vertices of distance curr_dist + 1, since they enqueue at the rear, we are guaranteed that they will not be processed until after all the vertices of distance curr_dist have been processed. After the last vertex at distance curr_dist dequeues and is processed, the queue only contains vertices of distance curr_dist + 1, so this process perpetuates. We merely need to begin the process by placing the start node on the queue by itself.

The refined algorithm is shown in Figure 9.18. In the pseudocode, we have assumed that the start vertex, s, is known somehow and T[s].dist is 0. A C routine might pass s as an argument. Also, it is possible that the queue might empty prematurely, if some vertices are unreachable from the start node. In this case, a distance of INT_MAX will be reported for these nodes, which is perfectly reasonable. Finally, the known field is not used; once a vertex is processed it can never enter the queue again, so the fact that it need not be reprocessed is implicitly marked. Thus, the known field can be discarded. Figure 9.19 shows how the values on the graph we have been using are changed during the algorithm. We keep the known field to make the table easier to follow, and for consistency with the rest of this section.

Using the same analysis as was performed for topological sort, we see that the running time is O(|E| + |V|), as long as adjacency lists are used.

9.3.2. Dijkstra's Algorithm

If the graph is weighted, the problem (apparently) becomes harder, but we can still use the ideas from the unweighted case.

We keep all of the same information as before. Thus, each vertex is marked as either known or unknown. A tentative distance dv is kept for each vertex, as before. This distance turns out to be the shortest path length from s to v using only known vertices as intermediates. As before, we record pv, which is the last vertex to cause a change to dv.

void

unweighted( TABLE T )   /* assume T is initialized (Fig 9.30) */

{

QUEUE Q;

vertex v, w;

/*1*/        Q = create_queue( NUM_VERTEX ); make_null( Q );

/* enqueue the start vertex s, determined elsewhere */

/*2*/        enqueue( s, Q );

/*3*/        while( !is empty( Q ) )

{

/*4*/             v = dequeue( Q );

/*5*/             T[v].known = TRUE; /* not really needed anymore */

/*6*/             for each w adjacent to v

/*7*/                  if( T[w].dist = INT_MAX )

{

/*8*/                       T[w].dist = T[v].dist + 1;

/*9*/                       T[w].path = v;

/*10*/                      enqueue( w, Q );

}

}

/*11*/       dispose_queue( Q ); /* free the memory */

}

Figure 9.18 Pseudocode for unweighted shortest-path algorithm

The general method to solve the single-source shortest-path problem is known as Dijkstra's algorithm. This thirty-year-old solution is a prime example of a greedy algorithm. Greedy algorithms generally solve a problem in stages by doing what appears to be the best thing at each stage. For example, to make change in U.S. currency, most people count out the quarters first, then the dimes, nickels, and pennies. This greedy algorithm gives change using the minimum number of coins. The main problem with greedy algorithms is that they do not always work. The addition of a 12-cent piece breaks the coin-changing algorithm, because the answer it gives (one 12-cent piece and three pennies) is not optimal (one dime and one nickel).

Dijkstra's algorithm proceeds in stages, just like the unweighted shortest-path algorithm. At each stage, Dijkstra's algorithm selects a vertex v, which has the smallest dv among all the unknown vertices, and declares that the shortest path from s to v is known. The remainder of a stage consists of updating the values of dw.

     Initial State    v3 Dequeued    v1 Dequeued    v6 Dequeued

     -------------  --------------  -------------  -------------

  v  Known  dv  pv   Known  dv  pv   Known  dv  pv  Known  dv  pv

----------------------------------------------------------------

  v1   0         0      0    1   v3    1    1   v3    1    1   v3

  v2   0         0      0         0     0    2   v1    0    2   v1

  v3   0    0   0      1    0   0     1    0   0     1    0   0

  v4   0         0      0         0     0    2   v1    0    2   v1

  v5   0         0      0         0     0         0     0         0

  v6   0         0      0    1   v3    0    1   v3    1    1   v3

  v7   0         0      0         0     0         0     0         0

----------------------------------------------------------------

  Q:      v3            v1,v6          v6,v2,v4         v2,v4

      v2 Dequeued    v4 Dequeued    v5 Dequeued    v7 Dequeued

     -------------  --------------  -------------  -------------

  v   Known  dv  pv  Known  dv  pv  Known  dv  pv  Known  dv  pv

----------------------------------------------------------------

  v1    1    1   v3    1    1   v3    1    1   v3    1    1  v3

  v2    1    2   v1    1    2   v1    1    2   v1    1    2  v1

  v3    1    0   0     1    0   0     1    0   0     1    0  0

  v4    0    2   v1    1    2   v1    1    2   v1    1    2  v1

  v5    0    3   v2    0    3   v2    1    3   v2    1    3  v2

  v6    1    1   v3    1    1   v3    1    1   v3    1    1  v3

  v7    0         0     0    3   v4    0    3   v4    1    3  v4

----------------------------------------------------------------

  Q:     v4,v5          v5,v7           v7              empty

Figure 9.19 How the data structure changes during the unweighted shortest-path algorithm

In the unweighted case, we set dw = dv + 1 if dw =   [image]. Thus, we essentially lowered the value of dw if vertex v offered a shorter path. If we apply the same logic to the weighted case, then we should set dw = dv + cv,w if this new value for dw would be an improvement. Put simply, the algorithm decides whether or not it is a good idea to use v on the path to w. The original cost, dw, is the cost without using v; the cost calculated above is the cheapest path using v (and only known vertices).

The graph in Figure 9.20 is our example. Figure 9.21 represents the initial configuration, assuming that the start node, s, is v1. The first vertex selected is v1, with path length 0. This vertex is marked known. Now that v1 is known, some entries need to be adjusted. The vertices adjacent to v1 are v2 and v4. Both these vertices get their entries adjusted, as indicated in Figure 9.22.

Next, v4 is selected and marked known. Vertices v3, v5, v6, and v7 are adjacent, and it turns out that all require adjusting, as shown in Figure 9.23.

Next, v2 is selected. v4 is adjacent but already known, so no work is performed on it. v5 is adjacent but not adjusted, because the cost of going through v2 is 2 + 10 = 12 and a path of length 3 is already known. Figure 9.24 shows the table after these vertices are selected.

Figure 9.20 The directed graph G(again)

  v   Known  dv  pv

-------------------

  v1     0    0   0

  v2     0         0

  v3     0         0

  v4     0         0

  v5     0         0

  v6     0         0

  v7     0         0

Figure 9.21 Initial configuration of table used in Dijkstra's algorithm

  v   Known  dv  pv

--------------------

  v1     1    0   0

  v2     0    2   v1

  v3     0         0

  v4     0    1   v1

  v5     0         0

  v6     0         0

  v7     0         0

Figure 9.22 After v1 is declared known

  v   Known  dv  pv

--------------------

  v1     1    0   0

  v2     0    2   v1

  v3     0    3   v4

  v4     1    1   v1

  v5     0    3   v4

  v6     0    9   v4

  v7     0    5   v4

Figure 9.23 After v4 is declared known

  v   Known  dv  pv

--------------------

  v1     1    0   0

  v2     1    2   v1

  v3     0    3   v4

  v4     1    1   v1

  v5     0    3   v4

  v6     0    9   v4

  v7     0    5   v4

Figure 9.24 After v2 is declared known

  v   Known  dv  pv

--------------------

  v1     1    0   0

  v2     1    2   v1

  v3     1    3   v4

  v4     1    1   v1

  v5     1    3   v4

  v6     0    8   v3

  v7     0    5   v4

Figure 9.25 After v5 and then v3 are declared known

  v  Known  dv  pv

-------------------

  v1    1    0   0

  v2    1    2   v1

  v3    1    3   v4

  v4    1    1   v1

  v5    1    3   v4

  v6    0    6   v7

  v7    1    5   v4

Figure 9.26 After v7 is declared known

The next vertex selected is v5 at cost 3. v7 is the only adjacent vertex, but it is not adjusted, because 3 + 6 > 5. Then v3 is selected, and the distance for v6 is adjusted down to 3 + 5 = 8. The resulting table is depicted in Figure 9.25.

Next v7 is selected; v6 gets updated down to 5 + 1 = 6. The resulting table is Figure 9.26.

Finally, v6 is selected. The final table is shown in Figure 9.27. Figure 9.28 graphically shows how edges are marked known and vertices updated during Dijkstra's algorithm.

  v  Known  dv  pv

-------------------

  v1    1    0   0

  v2    1    2   v1

  v3    1    3   v4

  v4    1    1   v1

  v5    1    3   v4

  v6    1    6   v7

  v7    1    5   v4

Figure 9.27 After v6 is declared known and algorithm terminates

Figure 9.28 Stages of Dijkstra's algorithm

To print out the actual path from a start vertex to some vertex v, we can write a recursive routine to follow the trail left in the p array.

We now give pseudocode to implement Dijkstra's algorithm. We will assume  that the vertices are numbered from 0 to NUM_VERTEX for convenience (see Fig. 9.29), and that the graph can be read into an adjacency list by the routine read_graph.

In the routine in Figure 9.30, the start vertex is passed to the initialization routine. This is the only place in the code where the start vertex needs to be known.

The path can be printed out using the recursive routine in Figure 9.31. The routine recursively prints the path all the way up to the vertex before v on the path, and then just prints v. This works because the path is simple.

typedef int vertex;

struct table_entry

{

LIST header;        /* Adjacency list header */

int known;

dist_type dist;

vertex path;

};

/* Vertices are numbered from 1 */

#define NOT_A_VERTEX 0

typedef struct table_entry TABLE[NUM_VERTEX+1];

Figure 9.29 Declarations for Dijkstra's algorithm

void

init_table( vertex start, graph G, TABLE T )

{

int i;

/*1*/        read graph( G, T ); /* read graph somehow */

/*2*/        for( i=NUM_VERTEX; i>0; i-- )

{

/*3*/             T[i].known = FALSE;

/*4*/             T[i].dist = INT_MAX;

/*5*/             T[i].path = NOT_A_VERTEX;

}

/*6*/        T[start].dist = 0;

}

Figure 9.30 Table initialization routine

/* print shortest path to v after dijkstra has run */

/* assume that the path exists */

void

print_path( vertex v, TABLE T )

{

if( T[v].path != NOT_A_VERTEX )

{

print_path( T[v].path, T );

printf(" to ");

}

printf("%v", v ); /* %v is a pseudocode option for printf */

}

Figure 9.31 Routine to print the actual shortest path

Figure 9.32 shows the main algorithm, which is just a for loop to fill up the table using the greedy selection rule.

A proof by contradiction will show that this algorithm always works as long  as no edge has a negative cost. If any edge has negative cost, the algorithm could  produce the wrong answer (see Exercise 9.7a). The running time depends on how the table is manipulated, which we have yet to consider. If we use the obvious  algorithm of scanning down the table to find the minimum dv, each phase will take O(|V|) time to find the minimum, and thus O(|V|2) time will be spent finding the  minimum over the course of the algorithm. The time for updating dw is constant per  update, and there is at most one update per edge for a total of O(|E|). Thus, the total running time is O(|E| + |V|2) = O(|V|2). If the graph is dense, with |E| =   [image] (|V|2), this algorithm is not only simple but essentially optimal, since it runs in time linear  in the number of edges.

If the graph is sparse, with |E| =   [image] (|V|), this algorithm is too slow. In this case, the distances would need to be kept in a priority queue. There are actually  two ways to do this; both are similar.

Lines 2 and 5 combine to form a delete_min operation, since once the unknown  minimum vertex is found, it is no longer unknown and must be removed from future  consideration. The update at line 9 can be implemented two ways.

One way treats the update as a decrease_key operation. The time to find  the minimum is then O(log |V|), as is the time to perform updates, which amount to decrease_key operations. This gives a running time of O(|E| log |V| + |V| log |V|) = O(|E| log |V|), an improvement over the previous bound for sparse graphs. Since priority queues do not efficiently support the find operation, the location in the priority queue of each value of di will need to be maintained and updated whenever di changes in the priority queue. This is not typical of the priority queue ADT and thus is considered ugly.

The alternate method is to insert w and the new value dw into the priority queue every time line 9 is executed. Thus, there may be more than one representative for each vertex in the priority queue. When the delete_min operation removes the  smallest vertex from the priority queue, it must be checked to make sure that it is not already known. Thus, line 2 becomes a loop performing delete_mins until an  unknown vertex emerges. Although this method is superior from a software point of view, and is certainly much easier to code, the size of the priority queue could  get to be as big as |E|. This does not affect the asymptotic time bounds, since |E|   [image]  |V|2 implies that log|E|   [image]  2 log |V|. Thus, we still get an O(|E| log |V|) algorithm.  However, the space requirement does increase, and this could be important in some applications. Moreover, because this method requires |E| delete_mins instead of only |V|, it is likely to be slower in practice.

void

dijkstra( TABLE T )

{

vertex v, w;

/*1*/        for( ; ; )

{

/*2*/        v = smallest unknown distance vertex;

/*4*/             break;

/*5*/        T[v].known = TRUE;

/*6*/        for each w adjacent to v

/*7*/             if( !T[w].known )

/*8*/                  if( T[v].dist + cv,w < T[w].dist )

{ /* update w */

/*9*/                       decrease( T[w].dist to

T[v].dist + cv,w );

/*10*/                      T[w].path = v;

}

}

}

Figure 9.32 Pseudocode for Dijkstra's algorithm

Notice that for the typical problems, such as computer mail and mass transit commutes, the graphs are typically very sparse because most vertices have only a couple of edges, so it is important in many applications to use a priority queue to solve this problem.

There are better time bounds possible using Dijkstra's algorithm if different data structures are used. In Chapter 11, we will see another priority queue  data structure called the Fibonacci heap. When this is used, the running time is O(|E| + |V| log |V|). Fibonacci heaps have good theoretical time bounds but a fair amount of overhead, so it is not dear whether using Fibonacci heaps is actually  better in practice than Dijkstra's algorithm with binary heaps. Needless to say, there are no average-case results for this problem, since it is not even obvious how to  model a random graph.

void                 /* assume T is initialized as in Fig 9.18 */

weighted_negative( TABLE T )

{

QUEUE Q;

vertex v, w;

/*1*/        Q = create_queue( NUM_VERTEX ); make_null( Q );

/*2*/        enqueue( s, Q ); /* enqueue the start vertex s */

/*3*/        while( !is_empty( Q ) )

{

/*4*/             v = dequeue( Q );

/*5*/             for each w adjacent to v

/*6*/                  if( T[v].dist + cv,w < T[w].dist )

{ /*update w */

/*7*/                       T[w].dist = T[v].dist + cv,w ;

/*8*/                       T[w].path = v;

/*9*/                       if( w is not already in Q )

/*10*/                           enqueue( w, Q );

}

}

/*11*/       dispose_queue( Q );

}

Figure 9.33 Pseudocode for weighted shortest-path algorithm with negative edge costs

9.3.3. Graphs with Negative Edge Costs

If the graph has negative edge costs, then Dijkstra's algorithm does not work. The  problem is that once a vertex u is declared known, it is possible that from some other,  unknown vertex v there is a path back to u that is very negative. In such a case, taking  a path from s to v back to u is better than going from s to u without using v.

A combination of the weighted and unweighted algorithms will solve the problem, but at the cost of a drastic increase in running time. We forget about the  concept of known vertices, since our algorithm needs to be able to change its mind.  We begin by placing s on a queue. Then, at each stage, we dequeue a vertex v. We  find all vertices w adjacent to v such that dw > dv + cv,w. We update dw and pw, and place w on a queue if it is not already there. A bit can be set for each vertex  to indicate presence in the queue. We repeat the process until the queue is empty. Figure 9.33 (almost) implements this algorithm.

Although the algorithm works if there are no negative-cost cycles, it is no  longer true that the code in lines 6 through 10 is executed once per edge. Each vertex can dequeue at most |V| times, so the running time is O(|E|   [image]  |V|) if adjacency lists are used (Exercise 9.7b). This is quite an increase from Dijkstra's algorithm, so it is fortunate that, in practice, edge costs are nonnegative. If negative-cost cycles are present, then the algorithm as written will loop indefinitely. By stopping the algorithm after any vertex has dequeued |V| + 1 times, we can guarantee termination.

9.3.4. Acyclic Graphs

If the graph is known to be acyclic, we can improve Dijkstra's algorithm by changing the order in which vertices are declared known, otherwise known as the vertex selection rule. The new rule is to select vertices in topological order. The algorithm can be done in one pass, since the selections and updates can take place as the topological sort is being performed.

This selection rule works because when a vertex v is selected, its distance, dv, can no longer be lowered, since by the topological ordering rule it has no incoming edges emanating from unknown nodes.

There is no need for a priority queue with this selection rule; the running time is O(|E| + |V|), since the selection takes constant time.

An acyclic graph could model some downhill skiing problem -- we want to get from point a to b, but can only go downhill, so clearly there are no cycles. Another possible application might be the modeling of (nonreversible) chemical reactions. We could have each vertex represent a particular state of an experiment. Edges would represent a transition from one state to another, and the edge weights might represent the energy released. If only transitions from a higher energy state to a lower are allowed, the graph is acyclic.

A more important use of acyclic graphs is critical path analysis. The graph in Figure 9.34 will serve as our example. Each node represents an activity that must be performed, along with the time it takes to complete the activity. This graph is thus known as an activity-node graph. The edges represent precedence relationships: An edge (v, w) means that activity v must be completed before activity w may begin. Of course, this implies that the graph must be acyclic. We assume that any activities that do not depend (either directly or indirectly) on each other can be performed in parallel by different servers.

Figure 9.34 Activity-node graph

This type of a graph could be (and frequently is) used to model construction projects. In this case, there are several important questions which would be of interest to answer. First, what is the earliest completion time for the project? We can see from the graph that 10 time units are required along the path A, C, F, H. Another important question is to determine which activities can be delayed, and by how long, without affecting the minimum completion time. For instance, delaying any of A, C, F, or H would push the completion time past 10 units. On the other hand, activity B is less critical and can be delayed up to two time units without affecting the final completion time.

To perform these calculations, we convert the activity-node graph to an event-node graph. Each event corresponds to the completion of an activity and all its dependent activities. Events reachable from a node v in the event-node graph may not commence until after the event v is completed. This graph can be constructed automatically or by hand. Dummy edges and nodes may need to be inserted in the case where an activity depends on several others. This is necessary in order to avoid introducing false dependencies (or false lack of dependencies). The event node graph corresponding to the graph in Figure 9.34 is shown in Figure 9.35.

To find the earliest completion time of the project, we merely need to find the length of the longest path from the first event to the last event. For general graphs, the longest-path problem generally does not make sense, because of the possibility of positive-cost cycles. These are the equivalent of negative-cost cycles in shortest-path problems. If positive-cost cycles are present, we could ask for the longest simple path, but no satisfactory solution is known for this problem. Since the event-node graph is acyclic, we need not worry about cycles. In this case, it is easy to adapt the shortest-path algorithm to compute the earliest completion time for all nodes in the graph. If E Ci is the earliest completion time for node i, then the applicable rules are

EC1 = 0

Figure 9.36 shows the earliest completion time for each event in our example event-node graph.

We can also compute the latest time, LCi, that each event can finish without affecting the final completion time. The formulas to do this are

LCn = ECn

These values can be computed in linear time by maintaining, for each vertex, a list of all adjacent and preceding vertices. The earliest completion times are computed for vertices by their topological order, and the latest completion times are computed by reverse topological order. The latest completion times are shown in Figure 9.37.

The slack time for each edge in the event-node graph represents the amount of time that the completion of the corresponding activity can be delayed without delaying the overall completion. It is easy to see that

Slack(v,w) = LCw - ECv - cv,w

Figure 9.35 Event-node graph

Figure 9.36 Earliest completion times

Figure 9.38 shows the slack (as the third entry) for each activity in the event-node graph. For each node, the top number is the earliest completion time and the bottom entry is the latest completion time.

Figure 9.37 Latest completion times

Figure 9.38 Earliest completion time, latest completion time, and slack

Some activities have zero slack. These are critical activities, which must finish on schedule. There is at least one path consisting entirely of zero-slack edges; such a path is a critical path.

9.3.5. All-Pairs Shortest Path

Sometimes it is important to find the shortest paths between all pairs of vertices in the graph. Although we could just run the appropriate single-source algorithm |V| times, we might expect a somewhat faster solution, especially on a dense graph, if we compute all the information at once.

In Chapter 10, we will see an O(|V|3) algorithm to solve this problem for weighted graphs. Although, for dense graphs, this is the same bound as running a simple (non-priority queue) Dijkstra's algorithm |V| times, the loops are so tight that the specialized all-pairs algorithm is likely to be faster in practice. On sparse graphs, of course, it is faster to run |V| Dijkstra's algorithms coded with priority queues.

9.4. Network Flow Problems

Suppose we are given a directed graph G = (V, E) with edge capacities cv,w. These capacities could represent the amount of water that could flow through a pipe or the amount of traffic that could flow on a street between two intersections. We have two vertices: s, which we call the source, and t, which is the sink. Through any edge, (v, w), at most cv,w units of "flow" may pass. At any vertex, v, that is not either s or t, the total flow coming in must equal the total flow going out. The maximum flow problem is to determine the maximum amount of flow that can pass from s to t. As an example, for the graph in Figure 9.39 on the left the maximum flow is 5, as indicated by the graph on the right.

Figure 9.39 A graph (left) and its maximum flow

As required by the problem statement, no edge carries more flow than its capacity. Vertex a has three units of flow coming in, which it distributes to c and d. Vertex d takes three units of flow from a and b and combines this, sending the result to t. A vertex can combine and distribute flow in any manner that it likes, as long as edge capacities are not violated and as long as flow conservation is maintained (what goes in must come out).

9.4.1. A Simple Maximum-Flow Algorithm

9.4.1. A Simple Maximum-Flow Algorithm

A first attempt to solve the problem proceeds in stages. We start with our graph, G, and construct a flow graph Gf. Gf tells the flow that has been attained at any stage in the algorithm. Initially all edges in Gf have no flow, and we hope that when the algorithm terminates, Gf contains a maximum flow. We also construct a graph, Gr, called the residual graph. Gr tells, for each edge, how much more flow can be added. We can calculate this by subtracting the current flow from the capacity for each edge. An edge in Gr is known as a residual edge.

At each stage, we find a path in Gr from s to t. This path is known as an augmenting path. The minimum edge on this path is the amount of flow that can be added to every edge on the path. We do this by adjusting Gf and recomputing Gr. When we find no path from s to t in Gr, we terminate. This algorithm is nondeterministic, in that we are free to choose any path from s to t; obviously some choices are better than others, and we will address this issue later. We will run this algorithm on our example. The graphs below are G, Gf, Gr respectively. Keep in mind that there is a slight flaw in this algorithm. The initial configuration is in Figure 9.40.

There are many paths from s to t in the residual graph. Suppose we select s, b, d, t. Then we can send two units of flow through every edge on this path. We will adopt the convention that once we have filled (saturated) an edge, it is removed from the residual graph. We then obtain Figure 9.41.

Next, we might select the path s, a, c, t, which also allows two units of flow. Making the required adjustments gives the graphs in Figure 9.42.

Figure 9.40 Initial stages of the graph, flow graph, and residual graph

Figure 9.41 G, Gf, Gr after two units of flow added along s, b, d, t

Figure 9.42 G, Gf, Gr after two units of flow added along s, a, c, t

The only path left to select is s, a, d, t, which allows one unit of flow. The resulting graphs are shown in Figure 9.43.

The algorithm terminates at this point, because t is unreachable from s. The resulting flow of 5 happens to be the maximum. To see what the problem is, suppose that with our initial graph, we chose the path s, a, d, t. This path allows 3 units of flow and thus seems to be a good choice. The result of this choice, however, is that there is now no longer any path from s to t in the residual graph, and thus, our algorithm has failed to find an optimal solution. This is an example of a greedy algorithm that does not work. Figure 9.44 shows why the algorithm fails.

In order to make this algorithm work, we need to allow the algorithm to change its mind. To do this, for every edge (v, w) with flow fv,w in the flow graph, we will add an edge in the residual graph (w, v) of capacity fv,w. In effect, we are allowing the algorithm to undo its decisions by sending flow back in the opposite direction. This is best seen by example. Starting from our original graph and selecting the augmenting path s, a, d, t, we obtain the graphs in Figure 9.45.

Notice that in the residual graph, there are edges in both directions between a and d. Either one more unit of flow can be pushed from a to d, or up to three units can be pushed back -- we can undo flow. Now the algorithm finds the augmenting path s, b, d, a, c, t, of flow 2. By pushing two units of flow from d to a, the algorithm takes two units of flow away from the edge (a, d) and is essentially changing its mind. Figure 9.46 shows the new graphs.

Figure 9.43 G, Gf, Gr after one unit of flow added along s, a, d, t -- algorithm terminates

Figure 9.44 G, Gf, Gr if initial action is to add three units of flow along s, a, d, t -- algorithm terminates with suboptimal solution

Figure 9.45 Graphs after three units of flow added along s, a, d, t using correct algorithm

There is no augmenting path in this graph, so the algorithm terminates. Surprisingly, it can be shown that if the edge capacities are rational numbers, this algorithm always terminates with a maximum flow. This proof is somewhat difficult and is beyond the scope of this text. Although the example happened to be acyclic, this is not a requirement for the algorithm to work. We have used acyclic graphs merely to keep things simple.

If the capacities are all integers and the maximum flow is f, then, since each augmenting path increases the flow value by at least 1, f stages suffice, and the total running time is O(f   [image]  |E|), since an augmenting path can be found in O(|E|) time by an unweighted shortest-path algorithm. The classic example of why this is a bad running time is shown by the graph in Figure 9.47.

Figure 9.46 Graphs after two units of flow added along s, b, d, a, c, t using correct algorithm

Figure 9.47 The classic bad case for augmenting

The maximum flow is seen by inspection to be 2,000,000 by sending 1,000,000 down each side. Random augmentations could continually augment along a path that includes the edge connected by a and b. If this were to occur repeatedly, 2,000,000 augmentations would be required, when we could get by with only 2.

A simple method to get around this problem is always to choose the augmenting path that allows the largest increase in flow. Finding such a path is similar to solving a weighted shortest-path problem and a single-line modification to Dijkstra's algorithm will do the trick. If capmax is the maximum edge capacity, then one can show that O(|E| log capmax) augmentations will suffice to find the maximum flow. In this case, since O(|E| log |V|) time is used for each calculation of an augmenting path, a total bound of O(|E|2 log |V| log capmax) is obtained. If the capacities are all small integers, this reduces to O(|E|2 log |V|).

Another way to choose augmenting paths is always to take the path with the least number of edges, with the plausible expectation that by choosing a path in this manner, it is less likely that a small, flow-restricting edge will turn up on the path. Using this rule, it can be shown that O(|E|  [image]  |V|) augmenting steps are required. Each step takes O(|E|), again using an unweighted shortest-path algorithm, yielding a O(|E|2|V|) bound on the running time.

Further data structure improvements are possible to this algorithm, and there are several, more complicated, algorithms. A long history of improved bounds has lowered the current best-known bound for this problem to O(|E||V| log(|V|2/ |E|)) (see the references). There are also a host of very good bounds for special cases. For instance, O(|E||V|1/2) time finds a maximum flow in a graph, having the property that all vertices except the source and sink have either a single incoming edge of capacity 1 or a single outgoing edge of capacity 1. These graphs occur in many applications.

The analyses required to produce these bounds are rather intricate, and it is not clear how the worst-case results relate to the running times encountered in practice. A related, even more difficult problem is the min-cost flow problem. Each edge has not only a capacity but a cost per unit of flow. The problem is to find, among all maximum flows, the one flow of minimum cost. Both of these problems are being actively researched.

9.5. Minimum Spanning Tree

The next problem we will consider is that of finding a minimum spanning tree in an undirected graph. The problem makes sense for directed graphs but appears to be more difficult. Informally, a minimum spanning tree of an undirected graph G is a tree formed from graph edges that connects all the vertices of G at lowest total cost. A minimum spanning tree exists if and only if G is connected. Although a robust algorithm should report the case that G is unconnected, we will assume that G is connected, and leave the issue of robustness as an exercise for the reader.

In Figure 9.48 the second graph is a minimum spanning tree of the first (it happens to be unique, but this is unusual). Notice that the number of edges in the minimum spanning tree is |V| - 1. The minimum spanning tree is a tree because it is acyclic, it is spanning because it covers every edge, and it is minimum for the obvious reason. If we need to wire a house with a minimum of cable, then a minimum spanning tree problem needs to be solved. There are two basic algorithms to solve this problem; both are greedy. We now describe them.

Figure 9.48 A graph G and its minimum spanning tree

9.5.1. Prim's Algorithm

9.5.2. Kruskal's Algorithm

9.5.1. Prim's Algorithm

One way to compute a minimum spanning tree is to grow the tree in successive stages. In each stage, one node is picked as the root, and we add an edge, and thus an associated vertex, to the tree.

At any point in the algorithm, we can see that we have a set of vertices that have already been included in the tree; the rest of the vertices have not. The algorithm then finds, at each stage, a new vertex to add to the tree by choosing the edge (u, v) such that the cost of (u, v) is the smallest among all edges where u is in the tree and v is not. Figure 9.49 shows how this algorithm would build the minimum spanning tree, starting from v1. Initially, v1 is in the tree as a root with no edges. Each step adds one edge and one vertex to the tree.

We can see that Prim's algorithm is essentially identical to Dijkstra's algorithm for shortest paths. As before, for each vertex we keep values dv and pv and an indication of whether it is known or unknown. dv is the weight of the shortest arc connecting v to a known vertex, and pv, as before, is the last vertex to cause a change in dv. The rest of the algorithm is exactly the same, with the exception that since the definition of dv is different, so is the update rule. For this problem, the update rule is even simpler than before: After a vertex v is selected, for each unknown w adjacent to v, dv = min(dw, cw,v).

Figure 9.49 Prim's algorithm after each stage

The initial configuration of the table is shown in Figure 9.50. v1 is selected, and v2, v3, and v4 are updated. The table resulting from this is shown in Figure 9.51. The next vertex selected is v4. Every vertex is adjacent to v4. v1 is not examined, because it is known. v2 is unchanged, because it has dv = 2 and the edge cost from v4 to v2 is 3; all the rest are updated. Figure 9.52 shows the resulting table. The next vertex chosen is v2 (arbitrarily breaking a tie). This does not affect any distances. Then v3 is chosen, which affects the distance in v6, producing Figure 9.53. Figure 9.54 results from the selection of v7, which forces v6 and v5 to be adjusted. v6 and then v5 are selected, completing the algorithm.

The final table is shown in Figure 9.55. The edges in the spanning tree can be read from the table: (v2, v1), (v3, v4), (v4, v1), (v5, v7), (v6, v7), (v7, v4). The total cost is 16.

The entire implementation of this algorithm is virtually identical to that of Dijkstra's algorithm, and everything that was said about the analysis of Dijkstra's algorithm applies here. Be aware that Prim's algorithm runs on undirected graphs, so when coding it, remember to put every edge in two adjacency lists. The running time is O (|V|2) without heaps, which is optimal for dense graphs, and O (|E| log |V|) using binary heaps, which is good for sparse graphs.

9.5.2. Kruskal's Algorithm

A second greedy strategy is continually to select the edges in order of smallest weight and accept an edge if it does not cause a cycle. The action of the algorithm on the graph in the preceding example is shown in Figure 9.56.

  v   Known  dv  pv

--------------------

  v1     0    0   0

  v2     0         0

  v3     0         0

  v4     0         0

  v5     0         0

  v6     0         0

  v7     0         0

Figure 9.50 Initial configuration of table used in Prim's algorithm

  v   Known  dv  pv

--------------------

  v1     1    0   0

  v2     0    2   v1

  v3     0    4   v1

  v4     0    1   v1

  v5     0         0

  v6     0         0

  v7     0         0

Figure 9.51 The table after v1 is declared known

  v   Known  dv  pv

--------------------

  v1     1    0   0

  v2     0    2   v1

  v3     0    2   v4

  v4     1    1   v1

  v5     0    7   v4

  v6     0    8   v4

  v7     0    4   v4

Figure 9.52 The table after v4 is declared known

Formally, Kruskal's algorithm maintains a forest -- a collection of trees. Initially, there are |V| single-node trees. Adding an edge merges two trees into one. When the algorithm terminates, there is only one tree, and this is the minimum spanning tree. Figure 9.57 shows the order in which edges are added to the forest.

The algorithm terminates when enough edges are accepted. It turns out to be simple to decide whether edge (u,v) should be accepted or rejected. The appropriate data structure is the union/find algorithm of the previous chapter.

The invariant we will use is that at any point in the process, two vertices belong to the same set if and only if they are connected in the current spanning forest. Thus, each vertex is initially in its own set. If u and v are in the same set, the edge is rejected, because since they are already connected, adding (u, v) would form a cycle. Otherwise, the edge is accepted, and a union is performed on the two sets containing u and v. It is easy to see that this maintains the set invariant, because once the edge (u, v) is added to the spanning forest, if w was connected to u and x was connected to v, then x and w must now be connected, and thus belong in the same set.

  v   Known  dv  pv

--------------------

  v1     1    0   0

  v2     1    2   v1

  v3     1    2   v4

  v4     1    1   v1

  v5     0    7   v4

  v6     0    5   v3

  v7     0    4   v4

Figure 9.53 The table after v2 and then v3 are declared known

  v   Known  dv  pv

--------------------

  v1     1    0   0

  v2     1    2   v1

  v3     1    2   v4

  v4     1    1   v1

  v5     0    6   v7

  v6     0    1   v7

  v7     1    4   v4

Figure 9.54 The table after v7 is declared known

  v   Known  dv  pv

--------------------

  v1     1    0   0

  v2     1    2   v1

  v3     1    2   v4

  v4     1    1   v1

  v5     1    6   v7

  v6     1    1   v7

  v7     1    4   v4

Figure 9.55 The table after v6 and v5 are selected (Prim's algorithm terminates)

The edges could be sorted to facilitate the selection, but building a heap in linear time is a much better idea. Then delete_mins give the edges to be tested in order. Typically, only a small fraction of the edges needs to be tested before the algorithm can terminate, although it is always possible that all the edges must be tried. For instance, if there was an extra vertex v8 and edge (v5, v8) of cost 100, all the edges would have to be examined. Procedure kruskal in Figure 9.58 finds a minimum spanning tree. Because an edge consists of three pieces of data, on some machines it is more efficient to implement the priority queue as an array of pointers to edges, rather than as an array of edges. The effect of this implementation is that, to rearrange the heap, only pointers, not large records, need to be moved.

  Edge     Weight    Action

----------------------------

  (v1,v4)    1     Accepted

  (v6,v7)    1     Accepted

  (v1,v2)    2     Accepted

  (v3,v4)    2     Accepted

  (v2,v4)    3     Rejected

  (v1,v3)    4     Rejected

  (v4,v7)    4     Accepted

  (v3,v6)    5     Rejected

  (v5,v7)    6     Accepted

Figure 9.56 Action of Kruskal's algorithm on G

Figure 9.57 Kruskal's algorithm after each stage

The worst-case running time of this algorithm is O(|E| log |E|), which is dominated by the heap operations. Notice that since |E| = O(|V|2), this running time is actually O(|E| log |V|). In practice, the algorithm is much faster than this time bound would indicate.

void

kruskal( graph G )

{

unsigned int edges_accepted;

DISJ_SET S;

PRIORITY_QUEUE H;

vertex u, v;

set_type u_set, v_set;

edge e;

/*1*/        initialize( S );

/*2*/        read_graph_into_heap_array( G, H );

/*3*/        build_heap( H );

/*4*/        edges_accepted = 0;

/*5*/        while( edges_accepted < NUM_VERTEX-1 )

{

/*6*/             e = delete_min( H );  /* e = (u, v) */

/*7*/             u_set = find( u, S );

/*8*/             v_set = find( v, S );

/*9*/             if( u_set != v_set )

{

/*10*/                 /* accept the edge */

/*11*/                 edges_accepted++;

/*12*/                 set_union( S, u_set, v_set );

}

}

}

Figure 9.58 Pseudocode for Kruskal's algorithm

9.6. Applications of Depth-First Search

Depth-first search is a generalization of preorder traversal. Starting at some vertex, v, we process v and then recursively traverse all vertices adjacent to v. If this process is performed on a tree, then all tree vertices are systematically visited in a total of O(|E|) time, since |E| =   [image] (|V|). If we perform this process on an arbitrary graph, we need to be careful to avoid cycles. To do this, when we visit a vertex v, we mark it visited, since now we have been there, and recursively call depth-first search on all adjacent vertices that are not already marked. We implicitly assume that for undirected graphs every edge (v, w) appears twice in the adjacency lists: once as (v, w) and once as (w, v). The procedure in Figure 9.59 performs a depth-first search (and does absolutely nothing else) and is a template for the general style.

void

dfs( vertex v )

{

visited[v] = TRUE;

for each w adjacent to v

if( !visited[w] )

dfs( w );

}

Figure 9.59 Template for depth-first search

The (global) boolean array visited[ ] is initialized to FALSE. By recursively calling the procedures only on nodes that have not been visited, we guarantee that we do not loop indefinitely. If the graph is undirected and not connected, or directed and not strongly connected, this strategy might fail to visit some nodes. We then search for an unmarked node, apply a depth-first traversal there, and continue this process until there are no unmarked nodes.* Because this strategy guarantees that each edge is encountered only once, the total time to perform the traversal is O(|E| + |V|), as long as adjacency lists are used.

* An efficient way of implementing this is to begin the depth-first search at v1. If we need to restart the depth-first search, we examine the sequence vk, vk + 1, . . . for an unmarked vertex, where vk - 1 is the vertex where the last depth-first search was started. This guarantees that throughout the algorithm, only O(|V|) is spent looking for vertices where new depth-first search trees can be started.

9.6.1 Undirected Graphs

An undirected graph is connected if and only if a depth-first search starting from any node visits every node. Because this test is so easy to apply, we will assume that the graphs we deal with are connected. If they are not, then we can find all the connected components and apply our algorithm on each of these in turn.

As an example of depth-first search, suppose in the graph of Figure 9.60 we start at vertex A. Then we mark A as visited and call dfs(B) recursively. dfs(B) marks B as visited and calls dfs(C) recursively. dfs(C) marks C as visited and calls dfs(D) recursively. dfs(D) sees both A and B, but both these are marked, so no recursive calls are made. dfs(D) also sees that C is adjacent but marked, so no recursive call is made there, and dfs(D) returns back to dfs(C). dfs(C) sees B adjacent, ignores it, finds a previously unseen vertex E adjacent, and thus calls dfs(E). dfs(E) marks E, ignores A and C, and returns to dfs(C). dfs(C) returns to dfs(B). dfs(B) ignores both A and D and returns. dfs(A) ignores both D and E and returns. (We have actually touched every edge twice, once as (v, w) and again as (w, v), but this is really once per adjacency list entry.)

Figure 9.60 An undirected graph

We graphically illustrate these steps with a depth-first spanning tree. The root of the tree is A, the first vertex visited. Each edge (v, w) in the graph is present in the tree. If, when we process (v, w), we find that w is unmarked, or if, when we process (w, v), we find that v is unmarked, we indicate this with a tree edge. If when we process (v, w), we find that w is already marked, and when processing (w, v), we find that v is already marked, we draw a dashed line, which we will call a back edge, to indicate that this "edge" is not really part of the tree. The depth-first search of the graph in Figure 9.60 is shown in Figure 9.61.

The tree will simulate the traversal we performed. A preorder numbering of the tree, using only tree edges, tells us the order in which the vertices were marked. If the graph is not connected, then processing all nodes (and edges) requires several calls to dfs, and each generates a tree. This entire collection is a depth-first spanning forest, which is so named for obvious reasons.

9.6.2. Biconnectivity

A connected undirected graph is biconnected if there are no vertices whose removal disconnects the rest of the graph. The graph in the example above is biconnected. If the nodes are computers and the edges are links, then if any computer goes down, network mail is unaffected, except, of course, at the down computer. Similarly, if a mass transit system is biconnected, users always have an alternate route should some terminal be disrupted.

Figure 9.61 Depth-first search of previous graph

If a graph is not biconnected, the vertices whose removal would disconnect the graph are known as articulation points. These nodes are critical in many applications. The graph in Figure 9.62 is not biconnected: C and D are articulation points. The removal of C would disconnect G, and the removal of D would disconnect E and F, from the rest of the graph.

Depth-first search provides a linear-time algorithm to find all articulation points in a connected graph. First, starting at any vertex, we perform a depth-first search and number the nodes as they are visited. For each vertex v, we call this preorder number num(v). Then, for every vertex v in the depth-first search spanning tree, we compute the lowest-numbered vertex, which we call low(v), that is reachable from v by taking zero or more tree edges and then possibly one back edge (in that order). The depth-first search tree in Figure 9.63 shows the preorder number first, and then the lowest-numbered vertex reachable under the rule described above.

The lowest-numbered vertex reachable by A, B, and C is vertex 1 (A), because they can all take tree edges to D and then one back edge back to A. We can efficiently compute low by performing a postorder traversal of the depth-first spanning tree. By the definition of low, low(v) is the minimum of

1.     num(v)

2.     the lowest num(w) among all back edges (v, w)

3.     the lowest low(w) among all tree edges (v, w)

The first condition is the option of taking no edges, the second way is to choose no tree edges and a back edge, and the third way is to choose some tree edges and possibly a back edge. This third method is succinctly described with a recursive call. Since we need to evaluate low for all the children of v before we can evaluate low(v), this is a postorder traversal. For any edge (v, w), we can tell whether it is a tree edge or a back edge merely by checking num(v) and num(w). Thus, it is easy to compute low(v): we merely scan down v's adjacency list, apply the proper rule, and keep track of the minimum. Doing all the computation takes O(|E| +|V|) time.

All that is left to do is to use this information to find articulation points. The root is an articulation point if and only if it has more than one child, because if it has two children, removing the root disconnects nodes in different subtrees, and if it has only one child, removing the root merely disconnects the root. Any other vertex v is an articulation point if and only if v has some child w such that low(w)   [image]  num(v). Notice that this condition is always satisfied at the root; hence the need for a special test.

The if part of the proof is clear when we examine the articulation points that the algorithm determines, namely C and D. D has a child E, and low(E)   [image]  num(D), since both are 4. Thus, there is only one way for E to get to any node above D, and that is by going through D. Similarly, C is an articulation point, because low (G)   [image]  num (C). To prove that this algorithm is correct, one must show that the only if part of the assertion is true (that is, this finds all articulation points). We leave this as an exercise. As a second example, we show (Fig. 9.64) the result of applying this algorithm on the same graph, starting the depth-first search at C.

Figure 9.62 A graph with articulation points C and D

Figure 9.63 Depth-first tree for previous graph, with num and low

We close by giving pseudocode to implement this algorithm. We will assume that the arrays visited[] (initialized to FALSE), num[], low[], and parent[] are global to keep the code simple. We will also keep a global variable called counter, which is initialized to 1 to assign the preorder traversal numbers, num[]. This is not normally good programming practice, but including all the declarations and passing the extra parameters would cloud the logic. We also leave out the easily implemented test for the root.

As we have already stated, this algorithm can be implemented by performing a preorder traversal to compute num and then a postorder traversal to compute low. A third traversal can be used to check which vertices satisfy the articulation point criteria. Performing three traversals, however, would be a waste. The first pass is shown in Figure 9.65.

The second and third passes, which are postorder traversals, can be implemented by the code in Figure 9.66. Line 8 handles a special case. If w is adjacent to v, then the recursive call to w will find v adjacent to w. This is not a back edge, only an edge that has already been considered and needs to be ignored. Otherwise, the procedure computes the minimum of the various low[] and num[] entries, as specified by the algorithm.

There is no rule that a traversal must be either preorder or postorder. It is possible to do processing both before and after the recursive calls. The procedure in Figure 9.67 combines the two routines assign_num and assign_low in a straightforward manner to produce the procedure find_art.

9.6.3. Euler Circuits

Consider the three figures in Figure 9.68. A popular puzzle is to reconstruct these figures using a pen, drawing each line exactly once. The pen may not be lifted from the paper while the drawing is being performed. As an extra challenge, make the pen finish at the same point at which it started. This puzzle has a surprisingly simple solution. Stop reading if you would like to try to solve it.

Figure 9.64 Depth-first tree that results if depth-first search starts at C

/* assign num and compute parents */

void

assign_num( vertex v )

{

vertex w;

/*1*/       num[v] = counter++;

/*2*/       visited[v] = TRUE;

/*3*/       for each w adjacent to v

/*4*/            if( !visited[w] )

{

/*5*/                 parent[w] = v;

/*6*/                 assign_num( w );

}

}

Figure 9.65 Routine to assign num to vertices (pseudocode)

/* assign low. Also check for articulation points */

void

assign_low( vertex v )

{

vertex w;

/*1*/       low[v] = num[v];             /* Rule 1 */

/*2*/       for each w adjacent to v

{

/*3*/            if( num[w] > num[v] )   /* forward edge */

{

/*4*/                  assign_low( w );

/*5*/                  if( low[w] >= num[v] )

/*6*/                       printf( "%v is an articulation point\n", v );

/*7*/                  low[v] = min( low[v], low[w] ); /* Rule 3 */

}

else

/*8*/            if( parent[v] != w )      /* back edge */

/*9*/                 low[v] = min( low[v], num[w] );  /* Rule 2 */

}

}

Figure 9.66 Pseudocode to compute low and to test for articulation points (test for the root is omitted)

void

find_art( vertex v )

{

vertex w;

/*1*/        visited[v] = TRUE;

/*2*/        low[v] = num[v] = counter++; /* Rule 1 */

/*3*/        for each w adjacent to v

{

/*4*/             if( !visited[w] ) /* forward edge */

{

/*5*/                  parent[w] = v;

/*6*/                  find_art( w );

/*7*/                  if( low[w] >= num[v] )

/*8*/                       printf ( "%v is an articulation point\n", v );

/*9*/                  low[v] = min( low[v], low[w] ); /* Rule    */

}

else

/*10*/            if( parent[v] != w ) /* back edge */

/*11*/                 low[v] = min( low[v], num[w] ); /* Rule 2 */

}

}

Figure 9.67 Testing for articulation points in one depth-first search (test for the root is omitted) (pseudocode)

Figure 9.68 Three drawings

The first figure can be drawn only if the starting point is the lower left- or right-hand corner, and it is not possible to finish at the starting point. The second figure is easily drawn with the finishing point the same as the starting point, but the third figure cannot be drawn at all within the parameters of the puzzle.

We can convert this problem to a graph theory problem by assigning a vertex to each intersection. Then the edges can be assigned in the natural manner, as in Figure 9.69.

After this conversion is performed, we must find a path in the graph that visits every edge exactly once. If we are to solve the "extra challenge," then we must find a cycle that visits every edge exactly once. This graph problem was solved in 1736 by Euler and marked the beginning of graph theory. The problem is thus commonly referred to as an Euler path (sometimes Euler tour) or Euler circuit problem, depending on the specific problem statement. The Euler tour and Euler circuit problems, though slightly different, have the same basic solution. Thus, we will consider the Euler circuit problem in this section.

The first observation that can be made is that an Euler circuit, which must end on its starting vertex, is possible only if the graph is connected and each vertex has an even degree (number of edges). This is because, on the Euler circuit, a vertex is entered and then left. If any vertex v has odd degree, then eventually we will reach the point where only one edge into v is unvisited, and taking it will strand us at v. If exactly two vertices have odd degree, an Euler tour, which must visit every edge but need not return to its starting vertex, is still possible if we start at one of the odd-degree vertices and finish at the other. If more than two vertices have odd degree, then an Euler tour is not possible.

The observations of the preceding paragraph provide us with a necessary condition for the existence of an Euler circuit. It does not, however, tell us that all connected graphs that satisfy this property must have an Euler circuit, nor does it give us guidance on how to find one. It turns out that the necessary condition is also sufficient. That is, any connected graph, all of whose vertices have even degree, must have an Euler circuit. Furthermore, a circuit can be found in linear time.

We can assume that we know that an Euler circuit exists, since we can test the necessary and sufficient condition in linear time. Then the basic algorithm is to perform a depth-first search. There is a surprisingly large number of "obvious" solutions that do not work. Some of these are presented in the exercises.

The main problem is that we might visit a portion of the graph and return to the starting point prematurely. If all the edges coming out of the start vertex have been used up, then part of the graph is untraversed. The easiest way to fix this is to find the first vertex on this path that has an untraversed edge, and perform another depth-first search. This will give another circuit, which can be spliced into the original. This is continued until all edges have been traversed.

Figure 9.69 Conversion of puzzle to graph

As an example, consider the graph in Figure 9.70. It is easily seen that this graph has an Euler circuit. Suppose we start at vertex 5, and traverse the circuit 5, 4, 10, 5. Then we are stuck, and most of the graph is still untraversed. The situation is shown in Figure 9.71.

We then continue from vertex 4, which still has unexplored edges. A depth-first search might come up with the path 4, 1, 3, 7, 4, 11, 10, 7, 9, 3, 4. If we splice this path into the previous path of 5, 4, 10, 5, then we get a new path of 5, 4, 1, 3, 7 ,4, 11, 10, 7, 9, 3, 4, 10, 5.

The graph that remains after this is shown in Figure 9.72. Notice that in this graph all the vertices must have even degree, so we are guaranteed to find a cycle to add. The remaining graph might not be connected, but this is not important. The next vertex on the path that has untraversed edges is vertex 3. A possible circuit would then be 3, 2, 8, 9, 6, 3. When spliced in, this gives the path 5, 4, 1, 3, 2, 8, 9, 6, 3, 7, 4, 11, 10, 7, 9, 3, 4, 10, 5.

Figure 9.70 Graph for Euler circuit problem

Figure 9.71 Graph remaining after 5, 4, 10, 5

Figure 9.72 Graph after the path 5, 4, 1, 3, 7, 4, 11, 10, 7, 9, 3, 4, 10, 5

The graph that remains is in Figure 9.73. On this path, the next vertex with an untraversed edge is 9, and the algorithm finds the circuit 9, 12, 10, 9. When this is added to the current path, a circuit of 5, 4, 1, 3, 2, 8, 9, 12, 10, 9, 6, 3, 7, 4, 11, 10, 7, 9, 3, 4, 10, 5 is obtained. As all the edges are traversed, the algorithm terminates with an Euler circuit.

To make this algorithm efficient, we must use appropriate data structures. We will sketch some of the ideas, leaving the implementation as an exercise. To make splicing simple, the path should be maintained as a linked list. To avoid repetitious scanning of adjacency lists, we must maintain, for each adjacency list, a pointer to the last edge scanned. When a path is spliced in, the search for a new vertex from which to perform the next dfs must begin at the start of the splice point. This guarantees that the total work performed on the vertex search phase is O(|E|) during the entire life of the algorithm. With the appropriate data structures, the running time of the algorithm is O(|E| + |V|).

A very similar problem is to find a simple cycle, in an undirected graph, that visits every vertex. This is known as the Hamiltonian cycle problem. Although it seems almost identical to the Euler circuit problem, no efficient algorithm for it is known. We shall see this problem again in Section 9.7.

9.6.4. Directed Graphs

Using the same strategy as with undirected graphs, directed graphs can be traversed in linear time, using depth-first search. If the graph is not strongly connected, a depth-first search starting at some node might not visit all nodes. In this case we repeatedly perform depth-first searches, starting at some unmarked node, until all vertices have been visited. As an example, consider the directed graph in Figure 9.74.

We arbitrarily start the depth-first search at vertex B. This visits vertices B, C, A, D, E, and F. We then restart at some unvisited vertex. Arbitrarily, we start at H, which visits I and J. Finally, we start at G, which is the last vertex that needs to be visited. The corresponding depth-first search tree is shown in Figure 9.75.

Figure 9.73 Graph remaining after the path 5, 4, 1, 3, 2, 8, 9, 6, 3, 7, 4, 11, 10, 7, 9, 3, 4, 10, 5

Figure 9.74 A directed graph

Figure 9.75 Depth-first search of previous graph

The dashed arrows in the depth-first spanning forest are edges (v, w) for which w was already marked at the time of consideration. In undirected graphs, these are always back edges, but, as we can see, there are three types of edges that do not lead to new vertices. First, there are back edges, such as (A, B) and (I, H). There are also forward edges, such as (C, D) and (C, E), that lead from a tree node to a descendant. Finally, there are cross edges, such as (F, C) and (G, F), which connect two tree nodes that are not directly related. Depth-first search forests are generally drawn with children and new trees added to the forest from left to right. In a depth- first search of a directed graph drawn in this manner, cross edges always go from right to left.

Some algorithms that use depth-first search need to distinguish between the three types of nontree edges. This is easy to check as the depth-first search is being performed, and it is left as an exercise.

One use of depth-first search is to test whether or not a directed graph is acyclic. The rule is that a directed graph is acyclic if and only if it has no back edges. (The graph above has back edges, and thus is not acyclic.) The alert reader may remember that a topological sort can also be used to determine whether a graph is acyclic. Another way to perform topological sorting is to assign the vertices topological numbers n, n - 1, . . . ,1 by postorder traversal of the depth-first spanning forest. As long as the graph is acyclic, this ordering will be consistent.

9.6.5. Finding Strong Components

By performing two depth-first searches, we can test whether a directed graph is strongly connected, and if it is not, we can actually produce the subsets of vertices that are strongly connected to themselves. This can also be done in only one depth-first search, but the method used here is much simpler to understand.

First, a depth-first search is performed on the input graph G. The vertices of G are numbered by a postorder traversal of the depth-first spanning forest, and then all edges in G are reversed, forming Gr. The graph in Figure 9.76 represents Gr for the graph G shown in Figure 9.74; the vertices are shown with their numbers.

The algorithm is completed by performing a depth-first search on Gr, always starting a new depth-first search at the highest-numbered vertex. Thus, we begin the depth-first search of Gr at vertex G, which is numbered 10. This leads nowhere, so the next search is started at H. This call visits I and J. The next call starts at B and visits A, C, and F. The next calls after this are dfs(D) and finally dfs(E). The resulting depth-first spanning forest is shown in Figure 9.77.

Each of the trees (this is easier to see if you completely ignore all nontree edges) in this depth-first spanning forest forms a strongly connected component. Thus, for our example, the strongly connected components are {G}, {H, I, J}, {B, A, C, F}, {D}, and {E}.

To see why this algorithm works, first note that if two vertices v and w are in the same strongly connected component, then there are paths from v to w and from w to v in the original graph G, and hence also in Gr. Now, if two vertices v and w are not in the same depth-first spanning tree of Gr, clearly they cannot be in the same strongly connected component.

To prove that this algorithm works, we must show that if two vertices v and w are in the same depth-first spanning tree of Gr, there must be paths from v to w and from w to v. Equivalently, we can show that if x is the root of the depth-first spanning tree of Gr containing v, then there is a path from x to v and from v to x. Applying the same logic to w would then give a path from x to w and from w to x. These paths would imply paths from v to w and w to v (going through x).

Since v is a descendant of x in Gr's depth-first spanning tree, there is a path from x to v in Gr and thus a path from v to x in G. Furthermore, since x is the root, x has the higher postorder number from the first depth-first search. Therefore, during the first depth-first search, all the work processing v was completed before the work at x was completed. Since there is a path from v to x, it follows that v must be a descendant of x in the spanning tree for G -- otherwise v would finish after x. This implies a path from x to v in G and completes the proof.

Figure 9.76 Gr numbered by postorder traversal of G

Figure 9.77 Depth-first search of Gr -- strong components are {G}, {H, I, J}, {B, A, C, F}, {D}, {E}

9.7. Introduction to NP-Completeness

In this chapter, we have seen solutions to a wide variety of graph theory problems. All these problems have polynomial running times, and with the exception of the network flow problem, the running time is either linear or only slightly more than linear (O(|E| log |E|)). We have also mentioned, in passing, that for some problems certain variations seem harder than the original.

Recall that the Euler circuit problem, which finds a path that touches every edge exactly once, is solvable in linear time. The Hamiltonian cycle problem asks for a simple cycle that contains every vertex. No linear algorithm is  known for this problem.

The single-source unweighted shortest-path problem for directed graphs is also solvable in linear time. No linear-time algorithm is known for the corresponding longest-simple-path problem.

The situation for these problem variations is actually much worse than we have described. Not only are no linear algorithms known for these variations, but there are no known algorithms that are guaranteed to run in polynomial time. The best known algorithms for these problems could take exponential time on some inputs.

In this section we will take a brief look at this problem. This topic is rather complex, so we will only take a quick and informal look at it. Because of this, the discussion may be (necessarily) somewhat imprecise in places.

We will see that there are a host of important problems that are roughly equivalent in complexity. These problems form a class called the NP-complete problems. The exact complexity of these NP-complete problems has yet to be determined and remains the foremost open problem in theoretical computer science. Either all these problems have polynomial-time solutions or none of them do.

9.7.1. Easy vs. Hard

9.7.2. The Class NP

9.7.3. NP-Complete Problems

9.7.1. Easy vs. Hard

When classifying problems, the first step is to examine the boundaries. We have already seen that many problems can be solved in linear time. We have  also seen some O(log n) running times, but these either assume some preprocessing (such as input already being read or a data structure already being built) or occur on arithmetic examples. For instance, the gcd algorithm, when applied on two numbers m and n, takes O(log n) time. Since the numbers consist of log m and log n bits respectively, the gcd algorithm is really taking time that is linear in the amount or size of input. Thus, when we measure running time, we will be concerned with the running time as a function of the amount of input. Generally, we cannot expect better than linear running time.

At the other end of the spectrum lie some truly hard problems. These problems are so hard that they are impossible. This does not mean the typical exasperated moan, which means that it would take a genius to solve the problem. Just as real numbers are not sufficient to express a solution to x2 < 0, one can prove that computers cannot solve every problem that happens to come along. These "impossible" problems are called undecidable problems.

One particular undecidable problem is the halting problem. Is it possible to have your C compiler have an extra feature that not only detects syntax errors but also infinite loops? This seems like a hard problem, but one might expect that if some very clever programmers spent enough time on it, they could produce this enhancement.

The intuitive reason that this problem is undecidable is that such a program might have a hard time checking itself. For this reason, these problems are sometimes called recursively undecidable.

If an infinite loop-checking program could be written, surely it could be used to check itself. We could then produce a program called LOOP. LOOP takes as input a program P and runs P on itself. It prints out the phrase YES if P loops when run on itself. If P terminates when run on itself, a natural thing to do would be to print out NO. Instead of doing that, we will have LOOP go into an infinite loop.

What happens when LOOP is given itself as input? Either LOOP halts, or it does not halt. The problem is that both these possibilities lead to contradictions, in much the same way as does the phrase "This sentence is a lie."

By our definition, LOOP(P) goes into an infinite loop if P(P) terminates. Suppose that when P = LOOP, P(P) terminates. Then, according to the LOOP program, LOOP(P) is obligated to go into an infinite loop. Thus, we must have LOOP(LOOP) terminating and entering an infinite loop, which is clearly not possible. On the other hand, suppose that when P = LOOP, P(P) enters an infinite loop. Then LOOP(P) must terminate, and we arrive at the same set of contradictions. Thus, we see that the program LOOP cannot possibly exist.

9.7.2. The Class NP

A few steps down from the horrors of undecidable problems lies the class NP. NP stands for nondeterministic polynomial-time. A deterministic machine, at each point in time, is executing an instruction. Depending on the instruction, it then goes to some next instruction, which is unique. A nondeterministic machine has a choice of next steps. It is free to choose any that it wishes, and if one of these steps leads to a solution, it will always choose the correct one. A nondeterministic machine thus has the power of extremely good (optimal) guessing. This probably seems like a ridiculous model, since nobody could possibly build a nondeterministic computer, and because it would seem to be an incredible upgrade to your standard computer (every problem might now seem trivial). We will see that nondeterminism is a very useful theoretical construct. Furthermore, nondeterminism is not as powerful as one might think. For instance, undecidable problems are still undecidable, even if nondeterminism is allowed.

A simple way to check if a problem is in NP is to phrase the problem as a yes/no question. The problem is in NP if, in polynomial time, we can prove that any "yes" instance is correct. We do not have to worry about "no" instances, since the program always makes the right choice. Thus, for the Hamiltonian cycle problem, a "yes" instance would be any simple circuit in the graph that includes all the vertices. This is in NP, since, given the path, it is a simple matter to check that it is really a Hamiltonian cycle. Appropriately phrased questions, such as "Is there a simple path of length > K?" can also easily be checked and are in NP. Any path that satisfies this property can be checked trivially.

The class NP includes all problems that have polynomial-time solutions, since obviously the solution provides a check. One would expect that since it is so much easier to check an answer than to come up with one from scratch, there would be problems in NP that do not have polynomial-time solutions. To date no such problem has been found, so it is entirely possible, though not considered likely by experts, that nondeterminism is not such an important improvement. The problem is that proving exponential lower bounds is an extremely difficult task. The information theory bound technique, which we used to show that sorting requires   [image] (n log n) comparisons, does not seem to be adequate for the task, because the decision trees are not nearly large enough.

Notice also that not all decidable problems are in NP. Consider the problem of determining whether a graph does not have a Hamiltonian cycle. To prove that a graph has a Hamiltonian cycle is a relatively simple matter-we just need to exhibit one. Nobody knows how to show, in polynomial time, that a graph does not have a Hamiltonian cycle. It seems that one must enumerate all the cycles and check them one by one. Thus the Non-Hamiltonian cycle problem is not known to be in NP.

9.7.3. NP-Complete Problems

Among all the problems known to be in NP, there is a subset, known as the NP-complete problems, which contains the hardest. An NP-complete problem has the property that any problem in NP can be polynomially reduced to it.

A problem P1 can be reduced to P2 as follows: Provide a mapping so that any instance of P1 can be transformed to an instance of P2. Solve P2, and then map the answer back to the original. As an example, numbers are entered into a pocket calculator in decimal. The decimal numbers are converted to binary, and all calculations are performed in binary. Then the final answer is converted back to decimal for display. For P1 to be polynomially reducible to P2, all the work associated with the transformations must be performed in polynomial time.

The reason that NP-complete problems are the hardest NP problems is that a problem that is NP-complete can essentially be used as a subroutine for any problem in NP, with only a polynomial amount of overhead. Thus, if any NP-complete problem has a polynomial-time solution, then every problem in NP must have a polynomial-time solution. This makes the NP-complete problems the hardest of all NP problems.

Suppose we have an NP-complete problem P1. Suppose P2 is known to be in NP. Suppose further that P1 polynomially reduces to P2, so that we can solve P1 by using P2 with only a polynomial time penalty. Since P1 is NP-complete, every problem in NP polynomially reduces to P1. By applying the closure property of polynomials, we see that every problem in NP is polynomially reducible to P2: We reduce the problem to P1 and then reduce P1 to P2. Thus, P2 is NP-complete.

As an example, suppose that we already know that the Hamiltonian cycle problem is NP-complete. The traveling salesman problem is as follows.

TRAVELING SALESMAN PROBLEM:

Given a complete graph G = (V, E), with edge costs, and an integer K, is there a simple cycle that visits all vertices and has total cost   [image]  K?

The problem is different from the Hamiltonian cycle problem, because all |V|(|V| - 1)/2 edges are present and the graph is weighted. This problem has many important applications. For instance, printed circuit boards need to have holes punched so that chips, resistors, and other electronic components can be placed. This is done mechanically. Punching the hole is a quick operation; the time-consuming step is positioning the hole puncher. The time required for positioning depends on the distance traveled from hole to hole. Since we would like to punch every hole (and then return to the start for the next board), and minimize the total amount of time spent traveling, what we have is a traveling salesman problem.

The traveling salesman problem is NP-complete. It is easy to see that a solution can be checked in polynomial time, so it is certainly in NP. To show that it is NP-complete, we polynomially reduce the Hamiltonian cycle problem to it. To do this we construct a new graph G'. G' has the same vertices as G. For G', each edge (v, w) has a weight of 1 if (v, w)   [image]  G, and 2 otherwise. We choose K = |V|. See Figure 9.78.

It is easy to verify that G has a Hamiltonian cycle problem if and only if G' has a Traveling Salesman tour of total weight |V|.

There is now a long list of problems known to be NP-complete. To prove that some new problem is NP-complete, it must be shown to be in NP, and then an appropriate NP-complete problem must be transformed into it. Although the transformation to a traveling salesman problem was rather straightforward, most transformations are actually quite involved and require some tricky constructions. Generally, several different NP-complete problems are considered before the problem that actually provides the reduction. As we are only interested in the general ideas, we will not show any more transformations; the interested reader can consult the references.

The alert reader may be wondering how the first NP-complete problem was actually proven to be NP-complete. Since proving that a problem is NP-complete requires transforming it from another NP-complete problem, there must be some NP-complete problem for which this strategy will not work. The first problem that was proven to be NP-complete was the satisfiability problem. The satisfiability problem takes as input a boolean expression and asks whether the expression has an assignment to the variables that gives a value of 1.

Figure 9.78 Hamiltonian cycle problem transformed to traveling salesman problem

Satisfiability is certainly in NP, since it is easy to evaluate a boolean expression and check whether the result is 1. In 1971, Cook showed that satisfiability was NP-complete by directly proving that all problems that are in NP could be transformed to satisfiability. To do this, he used the one known fact about every problem in NP: Every problem in NP can be solved in polynomial time by a nondeterministic computer. The formal model for a computer is known as a Turing machine. Cook showed how the actions of this machine could be simulated by an extremely complicated and long, but still polynomial, boolean formula. This boolean formula would be true if and only if the program which was being run by the Turing machine produced a "yes" answer for its input.

Once satisfiability was shown to be NP-complete, a host of new NP-complete problems, including some of the most classic problems, were also shown to be NP-complete.

In addition to the satisfiability, Hamiltonian circuit, traveling salesman, and longest-path problems, which we have already examined, some of the more well-known NP-complete problems which we have not discussed are bin packing, knapsack, graph coloring, and clique. The list is quite extensive and includes problems from operating systems (scheduling and security), database systems, operations research, logic, and especially graph theory.

Summary

In this chapter we have seen how graphs can be used to model many real-life problems. Many of the graphs that occur are typically very sparse, so it is important to pay attention to the data structures that are used to implement them.

We have also seen a class of problems that do not seem to have efficient solutions. In Chapter 10, some techniques for dealing with these problems will be discussed.

Exercises

9.1     Find a topological ordering for the graph in Figure 9.79.

9.2     If a stack is used instead of a queue for the topological sort algorithm in Section 9.1, does a different ordering result? Why might one data structure give a "better" answer?

9.3     Write a program to perform a topological sort on a graph.

9.4     An adjacency matrix requires O(  [image] V  [image] 2) merely to initialize using a standard double loop. Propose a method that stores a graph in an adjacency matrix (so that testing for the existence of an edge is O(1)) but avoids the quadratic running time.

9.5     a.     Find the shortest path from A to all other vertices for the graph in Figure 9.80.

b.     Find the shortest unweighed path from B to all other vertices for the graph in Figure 9.80.

Figure 9.79

9.6     What is the worst-case running time of Dijkstra's algorithm when implemented with d-heaps (Section 6.5)?

9.7     a.     Give an example where Dijkstra's algorithm gives the wrong answer in the presence of a negative edge but no negative-cost cycle.

**b. Show that the weighted shortest-path algorithm suggested in Section 9.3.3 works if there are negative-weight edges, but no negative-cost cycles, and that the running time of this algorithm is O(|E|  [image] |V|).

*9.8     Suppose all the edge weights in a graph are integers between 1 and   [image] E  [image]. How fast can Dijkstra's algorithm be implemented?

9.9     Write a program to solve the single-source shortest-path problem.

9.10     a.     Explain how to modify Dijkstra's algorithm to produce a count of the number of different minimum paths from v to w.

b.     Explain how to modify Dijkstra's algorithm so that if there is more than one minimum path from v to w, a path with the fewest number of edges is chosen.

Figure 9.80

9.11     Find the maximum flow in the network of Figure 9.79.

9.12     Suppose that G = (V, E) is a tree, s is the root, and we add a vertex t and edges of infinite capacity from all leaves in G to t. Give a linear-time algorithm to find a maximum flow from s to t.

9.13     A bipartite graph, G = (V, E), is a graph such that V can be partitioned into two subsets V1 and V2 and no edge has both its vertices in the same subset.

a.     Give a linear algorithm to determine whether a graph is bipartite.

b.     The bipartite matching problem is to find the largest subset E' of E such that no vertex is included in more than one edge. A matching of four edges (indicated by dashed edges) is shown in Figure 9.81. There is a matching of five edges, which is maximum.

Show how the bipartite matching problem can be used to solve the following problem: We have a set of instructors, a set of courses, and a list of courses that each instructor is qualified to teach. If no instructor is required to teach more than one course, and only one instructor may teach a given course, what is the maximum number of courses that can be offered?

c.     Show that the network flow problem can be used to solve the bipartite matching problem.

d.     What is the time complexity of your solution to part (b)?

9.14     Give an algorithm to find an augmenting path that permits the maximum flow.

9.15     a.     Find a minimum spanning tree for the graph in Figure 9.82 using both Prim's and Kruskal's algorithms.

b.     Is this minimum spanning tree unique? Why?

9.16     Does either Prim's or Kruskal's algorithm work if there are negative edge weights?

9.17      Show that a graph of V vertices can have VV_2 minimum spanning trees.

9.18     Write a program to implement Kruskal's algorithm.

9.19     If all of the edges in a graph have weights between 1 and   [image] E  [image], how fast can the minimum spanning tree be computed?

Figure 9.81 A bipartite graph

Figure 9.82

9.20     Give an algorithm to find a maximum spanning tree. Is this harder than finding a minimum spanning tree?

9.21     Find all the articulation points in the graph in Figure 9.83. Show the depth-first spanning tree and the values of num and low for each vertex.

9.22     Prove that the algorithm to find articulation points works.

9.23     a.     Give an algorithm to find the minimum number of edges that need to be removed from an undirected graph so that the resulting graph is acyclic.

*b.     Show that this problem is NP-complete for directed graphs.

9.24     Prove that in a depth-first spanning forest of a directed graph, all cross edges go from right to left.

9.25     Give an algorithm to decide whether an edge (v, w) in a depth-first spanning forest of a directed graph is a tree, back, cross, or forward edge.

9.26     Find the strongly connected components in the graph of Figure 9.84.

9.27     Write a program to find the strongly connected components in a digraph.

Figure 9.83

Figure 9.84

*9.28     Give an algorithm that finds the strongly connected components in only one depth-first search. Use an algorithm similar to the biconnectivity algorithm.

9.29     The biconnected components of a graph G is a partition of the edges into sets such that the graph formed by each set of edges is biconnected. Modify the algorithm in Figure 9.67 to find the biconnected components instead of the articulation points.

9.30     Suppose we perform a breadth-first search of an undirected graph and build a breadth-first spanning tree. Show that all edges in the tree are either tree edges or cross edges.

9.31     Give an algorithm to find in an undirected (connected) graph a path that goes through every edge exactly once in each direction.

9.32     a.     Write a program to find an Euler circuit in a graph if one exists.

b.     Write a program to find an Euler tour in a graph if one exists.

9.33     An Euler circuit in a directed graph is a cycle in which every edge is visited exactly once.

*a.     Prove that a directed graph has an Euler circuit if and only if it is strongly connected and every vertex has equal indegree and outdegree.

*b.     Give a linear-time algorithm to find an Euler circuit in a directed graph where one exists.

9.34     a.     Consider the following solution to the Euler circuit problem: Assume that the graph is biconnected. Perform a depth-first search, taking back edges only as a last resort. If the graph is not biconnected, apply the algorithm recursively on the biconnected components. Does this algorithm work?

b.     Suppose that when taking back edges, we take the back edge to the nearest ancestor. Does the algorithm work?

9.35     A planar graph is a graph that can be drawn in a plane without any two edges intersecting.

*a.     Show that neither of the graphs in Figure 9.85 is planar.

b.     Show that in a planar graph, there must exist some vertex which is connected to no more than five nodes.

**c. Show that in a planar graph,   [image] E  [image]    [image]  3  [image] V  [image]  - 6.

Figure 9.85

9.36     A multigraph is a graph in which multiple edges are allowed between pairs of vertices. Which of the algorithms in this chapter work without modification for multigraphs? What modifications need to be done for the others?

*9.37     Let G = (V, E) be an undirected graph. Use depth-first search to design a linear algorithm to convert each edge in G to a directed edge such that the resulting graph is strongly connected, or determine that this is not possible.

9.38     You are given a set of n sticks, which are laying on top of each other in some configuration. Each stick is specified by its two endpoints; each endpoint is an ordered triple giving its x, y, and z coordinates; no stick is vertical. A stick may be picked up only if there is no stick on top of it.

a.     Explain how to write a routine that takes two sticks a and b and reports whether a is above, below, or                                unrelated to b. (This has nothing to do with graph theory.)

b.     Give an algorithm that determines whether it is possible to pick up all the sticks, and if so, provides a sequence of stick pickups that accomplishes this.

9.39     The clique problem can be stated as follows: Given an undirected graph G = (V, E) and an integer K, does G contain a complete subgraph of at least K vertices?

The vertex cover problem can be stated as follows: Given an undirected graph G = (V, E) and an integer K, does G contain a subset V'   [image]  V such that   [image] V'  [image]    [image]  K and every edge in G has a vertex in V'? Show that the clique problem is polynomially reducible to vertex cover.

9.40     Assume that the Hamiltonian cycle problem is NP-complete for undirected graphs.

a.     Prove that the Hamiltonian cycle problem is NP-complete for directed graphs.

b.     Prove that the unweighted simple longest-path problem is NP-complete for directed graphs.

9.41     The baseball card collector problem is as follows: Given packets P1, P2, . . . , Pm, each of which contains a subset of the year's baseball cards, and an integer K, is it possible to collect all the baseball cards by choosing   [image]  K packets? Show that the baseball card collector problem is NP-complete.

References

Good graph theory textbooks include [7], [12], [21], and [34]. More advanced topics, including the more careful attention to running times, are covered in [36], [38], and [45].

Use of adjacency lists was advocated in [23]. The topological sort algorithm is from [28], as described in [31]. Dijkstra's algorithm appeared in [8]. The improvements using d-heaps and Fibonacci heaps are described in [27] and [14], respectively. The shortest-path algorithm with negative edge weights is due to Bellman [3]; Tarjan [45] describes a more efficient way to guarantee termination.

Ford and Fulkerson's seminal work on network flow is [13]. The idea of augmenting along shortest paths or on paths admitting the largest flow increase is from [11]. Other approaches to the problem can be found in [9], [30], and [20]. An algorithm for the min-cost flow problem can be found in [18].

An early minimum spanning tree algorithm can be found in [4]. Prim's algorithm is from [39]; Kruskal's algorithm appears in [32]. Two O(|E| log log |V|) algorithms are [5] and [46]. The theoretically best-known algorithms appear in [14] and [16]. An empirical study of these algorithms suggests that Prim's algorithm, implemented with decrease_key, is best in practice on most graphs [37].

The algorithm for biconnectivity is from [41]. The first linear-time strong components algorithm (Exercise 9.28) appears in the same paper. The algorithm presented in the text is due to Kosaraju (unpublished) and Sharir [40]. Other applications of depth-first search appear in [24], [25], [42], and [43] (as mentioned in Chapter 8, the results in [42] and [43] have been improved, but the basic algorithm is unchanged).

The classic reference work for the theory of NP-complete problems is [19]. Additional material can be found in [1]. The NP-completeness of satisfiability is shown in [6]. The other seminal paper is [29], which showed the NP-completeness of 21 problems. An excellent survey of complexity theory is [44]. An approximation algorithm for the traveling salesman problem, which generally gives nearly optimal results, can be found in [35].

A solution to Exercise 9.8 can be found in [2]. Solutions to the bipartite matching problem in Exercise 9.13 can be found in [22] and [33]. The problem can be generalized by adding weights to the edges and removing the restriction that the graph is bipartite. Efficient solutions for the unweighted matching problem for general graphs are quite complex. Details can be found in [10], [15], and [17].

Exercise 9.35 deals with planar graphs, which commonly arise in practice. Planar graphs are very sparse, and many difficult problems are easier on planar graphs. An example is the graph isomorphism problem, which is solvable in linear time for planar graphs [26]. No polynomial time algorithm is known for general graphs.

1.     A. V. Aho, J. E. Hopcroft, and J. D. Ullman, The Design and Analysis of Computer Algorithms, Addison-Wesley, Reading, MA, 1974.

2.     R. K. Ahuja, K. Melhorn, J. B. Orlin, and R. E. Tarjan, "Faster Algorithms for the Shortest Path Problem," Journal of the ACM 37 (1990), 213-223.

3.     R. E. Bellman, "On a Routing Problem," Quarterly of Applied Mathematics 16 (1958), 87-90.

4.     O. Boruvka, "Ojistém problému minimálním (On a Minimal Problem)," Práca Moravské   [image]  3 (1926), 37-58.

5.     D. Cheriton and R. E. Tarjan, "Finding Minimum Spanning Trees," SIAM Journal on Computing 5 (1976), 724-742.

6.     S. Cook, "The Complexity of Theorem Proving Procedures," Proceedings of the Third Annual ACM Symposium on Theory of Computing (1971), 151-158.

7.     N. Deo, Graph Theory wtth Applications to Engineering and Computer Science, Prentice Hall, Englewood Cliffs, NJ, 1974.

8.     E. W. Dijkstra, "A Note on Two Problems in Connexion with Graphs," Numerische Mathematik 1 (1959), 269-271.

9.     E. A. Dinic, "Algorithm for Solution of a Problem of Maximum Flow in Networks with Power Estimation," Soviet Mathematics Doklady 11 (1970), 1277-1280.

10.     J. Edmonds, "Paths, Trees, and Flowers," Canadian Journal of Mathematics 17 (1965) 449-467.

11.     J. Edmonds and R. M. Karp, "Theoretical Improvements in Algorithmic Efficiency for Network Flow Problems," Journal of the ACM 19 (1972), 248-264.

12.     S. Even, Graph Algorithms, Computer Science Press, Potomac, MD, 1979.

13.     L. R. Ford, Jr. and D. R. Fulkerson, Flows in Networks, Princeton University Press, Princeton, NJ, 1962.

14.     M. L. Fredman and R. E. Tarjan, "Fibonacci Heaps and Their Uses in Improved Network Optimization Algorithms," Journal of the ACM 34 (1987), 596-615.

15.     H. N. Gabow, "Data Structures for Weighted Matching and Nearest Common Ancestors with Linking," Proceedings of First Annual ACM-SIAM Symposium on Discrete Algorithms (1990), 434-443.

16.     H. N. Gabow, Z. Galil, T. H. Spencer, and R. E. Tarjan, "Efficient Algorithms for Finding Minimum Spanning Trees on Directed and Undirected Graphs," Combinatorica 6 (1986), 109-122.

17.      Z. Galil, "Efficient Algorithms for Finding Maximum Matchings in Graphs," ACM Computing Surveys 18 (1986), 23-38.

18.     Z. Galil and E. Tardos,"An O(n2(m + n log n)log n) Min-Cost Flow Algorithm," Journal of the ACM 35 (1988), 374-386.

19.      M. R. Garey and D. S. Johnson, Computers and Intractability: A Guide to the Theory of NP-Completeness, Freeman, San Francisco, 1979.

20.     A. V. Goldberg and R. E. Tarjan, "A New Approach to the Maximum-Flow Problem," Journal of the ACM 35 (1988), 921-940.

21.     F. Harary, Graph Theory, Addison-Wesley, Reading, MA, 1969.

22.     J. E. Hopcroft and R. M. Karp, "An n5/2 Algorithm for Maximum Matchings in Bipartite Graphs," SIAM Journal on Computing 2 (1973), 225-231.

23.     J. E. Hopcroft and R. E. Tarjan, "Algorithm 447: Efficient Algorithms for Graph Manipulation," Communications of the ACM 16 (1973), 372-378.

24.     J. E. Hopcroft and R. E. Tarjan, "Dividing a Graph into Triconnected Components," SIAM Journal on Computing 2 (1973), 135-158.

25.     J. E. Hopcroft and R. E. Tarjan, "Efficient Planarity Testing," Journal of the ACM 21 (1974), 549-568.

26.     J. E. Hopcroft and J. K. Wong, "Linear Time Algorithm for Isomorphism of Planar Graphs," Proceedings of the Sixth Annual ACM Symposium on Theory of Computing (1974), 172-184.

27.     D. B. Johnson, "Efficient Algorithms for Shortest Paths in Sparse Networks," Journal of the ACM 24 (1977), 1-13.

28.     A. B. Kahn, "Topological Sorting of Large Networks," Communications of the ACM 5 (1962), 558-562.

29.     R. M. Karp, "Reducibility among Combinatorial Problems," Complexity of Computer Computations (eds. R. E. Miller and J. W. Thatcher), Plenum Press, New York, 1972, 85-103.

30.     A. V. Karzanov, "Determining the Maximal Flow in a Network by the Method of Preflows," Soviet Mathematics Doklady 15 (1974), 434-437.

31.     D. E. Knuth, The Art of Computer Programming, Vol. 1: Fundamental Algorithms, second edition, Addison-Wesley, Reading, MA, 1973.

32.     J. B. Kruskal, Jr. "On the Shortest Spanning Subtree of a Graph and the Traveling Salesman Problem," Proceedings of the American Mathematical Society 7 (1956), 48-50.

33.     H. W. Kuhn, "The Hungarian Method for the Assignment Problem," Naval Research Logistics Quarterly 2 (1955), 83-97.

34.     E. L. Lawler, Combinatorial Optimization: Networks and Matroids, Holt, Reinhart, and Winston, New York, NY, 1976.

35.     S. Lin and B. W. Kernighan, "An Effective Heuristic Algorithm for the Traveling Salesman Problem," Operations Research 21 (1973), 498-516.

36.     K. Melhorn, Data Structures and Algorithms 2: Graph Algorithms and NP-completeness, Springer-Verlag, Berlin, 1984.

37.     B. M. E. Moret and H. D. Shapiro, "An Empirical Analysis of Algorithms for Constructing a Minimum Spanning Tree," Proceedings of the Second Workshop on Algorithms and Data Structures (1991), 400-411.

38.     C. H. Papadimitriou and K. Steiglitz, Combinatorial Optimization: Algorithms and Complexity, Prentice Hall, Englewood Cliffs, NJ, 1982.

39.     R. C. Prim, "Shortest Connection Networks and Some Generalizations," Bell System Technical Journal 36 (1957), 1389-1401.

40.     M. Sharir, "A Strong-Connectivity Algorithm and Its Application in Data Flow Analysis," Computers and Mathematics with Applications 7 (1981), 67-72.

41.     R. E. Tarjan, "Depth First Search and Linear Graph Algorithms," SIAM Journal on Computing 1 (1972), 146-160.

42.     R. E. Tarjan, "Testing Flow Graph Reducibility," Journal of Computer and System Sciences 9 (1974), 355-365.

43.     R. E. Tarjan, "Finding Dominators in Directed Graphs," SIAM Journal on Computing 3 (1974), 62-89.

44.     R. E. Tarjan, "Complexity of Combinatorial Algorithms," SIAM Review 20 (1978), 457-491.

45.     R. E. Tarjan, Data Structures and Network Algorithms, Society for Industrial and Applied Mathematics, Philadelphia, PA, 1983.

46.     A. C. Yao, "An O( |E | log log |V | ) Algorithm for Finding Minimum Spanning Trees," Information Processing Letters 4 (1975), 21-23.

Go to Chapter 10              Return to Table of Contents


Structures, Algorithm Analysis: PREFACE

 Return to Table of Contents

PREFACE

Purpose/Goals

This book describes data structures, methods of organizing large amounts of data, and algorithm analysis, the estimation of the running time of algorithms. As computers become faster and faster, the need for programs that can handle large amounts of input becomes more acute. Paradoxically, this requires more careful attention to efficiency, since inefficiencies in programs become most obvious when input sizes are large. By analyzing an algorithm before it is actually coded, students can decide if a particular solution will be feasible. For example, in this text students look at specific problems and see how careful implementations can reduce the time constraint for large amounts of data from 16 years to less than a second. Therefore, no algorithm or data structure is presented without an explanation of its running time. In some cases, minute details that affect the running time of the implementation are explored.

Once a solution method is determined, a program must still be written. As computers have become more powerful, the problems they solve have become larger and more complex, thus requiring development of more intricate programs to solve the problems. The goal of this text is to teach students good programming and algorithm analysis skills simultaneously so that they can develop such programs with the maximum amount of efficiency.

This book is suitable for either an advanced data structures (CS7) course or a first-year graduate course in algorithm analysis. Students should have some knowledge of intermediate programming, including such topics as pointers and recursion, and some background in discrete math.

Approach

I believe it is important for students to learn how to program for themselves, not how to copy programs from a book. On the other hand, it is virtually impossible to discuss realistic programming issues without including sample code. For this reason, the book usually provides about half to three-quarters of an implementation, and the student is encouraged to supply the rest.

The algorithms in this book are presented in ANSI C, which, despite some flaws, is arguably the most popular systems programming language. The use of C instead of Pascal allows the use of dynamically allocated arrays (see for instance rehashing in Ch. 5). It also produces simplified code in several places, usually because the and (&&) operation is short-circuited.

Most criticisms of C center on the fact that it is easy to write code that is barely readable. Some of the more standard tricks, such as the simultaneous assignment and testing against 0 via

if (x=y)

are generally not used in the text, since the loss of clarity is compensated by only a few keystrokes and no increased speed. I believe that this book demonstrates that unreadable code can be avoided by exercising reasonable care.

Overview

Chapter 1 contains review material on discrete math and recursion. I believe the only way to be comfortable with recursion is to see good uses over and over. Therefore, recursion is prevalent in this text, with examples in every chapter except Chapter 5.

Chapter 2 deals with algorithm analysis. This chapter explains asymptotic analysis and its major weaknesses. Many examples are provided, including an in-depth explanation of logarithmic running time. Simple recursive programs are analyzed by intuitively converting them into iterative programs. More complicated divide-and-conquer programs are introduced, but some of the analysis (solving recurrence relations) is implicitly delayed until Chapter 7, where it is performed in detail.

Chapter 3 covers lists, stacks, and queues. The emphasis here is on coding these data structures using ADTS, fast implementation of these data structures, and an exposition of some of their uses. There are almost no programs (just routines), but the exercises contain plenty of ideas for programming assignments.

Chapter 4 covers trees, with an emphasis on search trees, including external search trees (B-trees). The UNIX file system and expression trees are used as examples. AVL trees and splay trees are introduced but not analyzed. Seventy-five percent of the code is written, leaving similar cases to be completed by the student. Additional coverage of trees, such as file compression and game trees, is deferred until Chapter 10. Data structures for an external medium are considered as the final topic in several chapters.

Chapter 5 is a relatively short chapter concerning hash tables. Some analysis is performed and extendible hashing is covered at the end of the chapter.

Chapter 6 is about priority queues. Binary heaps are covered, and there is additional material on some of the theoretically interesting implementations of priority queues.

Chapter 7 covers sorting. It is very specific with respect to coding details and analysis. All the important general-purpose sorting algorithms are covered and compared. Three algorithms are analyzed in detail: insertion sort, Shellsort, and quicksort. External sorting is covered at the end of the chapter.

Chapter 8 discusses the disjoint set algorithm with proof of the running time. This is a short and specific chapter that can be skipped if Kruskal's algorithm is not discussed.

Chapter 9 covers graph algorithms. Algorithms on graphs are interesting not only because they frequently occur in practice but also because their running time is so heavily dependent on the proper use of data structures. Virtually all of the standard algorithms are presented along with appropriate data structures, pseudocode, and analysis of running time. To place these problems in a proper context, a short discussion on complexity theory (including NP-completeness and undecidability) is provided.

Chapter 10 covers algorithm design by examining common problem-solving techniques. This chapter is heavily fortified with examples. Pseudocode is used in these later chapters so that the student's appreciation of an example algorithm is not obscured by implementation details.

Chapter 11 deals with amortized analysis. Three data structures from Chapters 4 and 6 and the Fibonacci heap, introduced in this chapter, are analyzed.

Chapters 1-9 provide enough material for most one-semester data structures courses. If time permits, then Chapter 10 can be covered. A graduate course on algorithm analysis could cover Chapters 7-11. The advanced data structures analyzed in Chapter 11 can easily be referred to in the earlier chapters. The discussion of NP-completeness in Chapter 9 is far too brief to be used in such a course. Garey and Johnson's book on NP-completeness can be used to augment this text.

Exercises

Exercises, provided at the end of each chapter, match the order in which material is presented. The last exercises may address the chapter as a whole rather than a specific section. Difficult exercises are marked with an asterisk, and more challenging exercises have two asterisks.

A solutions manual containing solutions to almost all the exercises is available separately from The Benjamin/Cummings Publishing Company.

References

References are placed at the end of each chapter. Generally the references either are historical, representing the original source of the material, or they represent extensions and improvements to the results given in the text. Some references represent solutions to exercises.

Acknowledgments

I would like to thank the many people who helped me in the preparation of this and previous versions of the book. The professionals at Benjamin/Cummings made my book a considerably less harrowing experience than I had been led to expect. I'd like to thank my previous editors, Alan Apt and John Thompson, as well as Carter Shanklin, who has edited this version, and Carter's assistant, Vivian McDougal, for answering all my questions and putting up with my delays. Gail Carrigan at Benjamin/Cummings and Melissa G. Madsen and Laura Snyder at Publication Services did a wonderful job with production. The C version was handled by Joe Heathward and his outstanding staff, who were able to meet the production schedule despite the delays caused by Hurricane Andrew.

I would like to thank the reviewers, who provided valuable comments, many of which have been incorporated into the text. Alphabetically, they are Vicki Allan (Utah State University), Henry Bauer (University of Wyoming), Alex Biliris (Boston University), Jan Carroll (University of North Texas), Dan Hirschberg (University of California, Irvine), Julia Hodges (Mississippi State University), Bill Kraynek (Florida International University), Rayno D. Niemi (Rochester Institute of Technology), Robert O. Pettus (University of South Carolina), Robert Probasco (University of Idaho), Charles Williams (Georgia State University), and Chris Wilson (University of Oregon). I would particularly like to thank Vicki Allan, who carefully read every draft and provided very detailed suggestions for improvement.

At FIU, many people helped with this project. Xinwei Cui and John Tso provided me with their class notes. I'd like to thank Bill Kraynek, Wes Mackey, Jai Navlakha, and Wei Sun for using drafts in their courses, and the many students who suffered through the sketchy early drafts. Maria Fiorenza, Eduardo Gonzalez, Ancin Peter, Tim Riley, Jefre Riser, and Magaly Sotolongo reported several errors, and Mike Hall checked through an early draft for programming errors. A special thanks goes to Yuzheng Ding, who compiled and tested every program in the original book, including the conversion of pseudocode to Pascal. I'd be remiss to forget Carlos Ibarra and Steve Luis, who kept the printers and the computer system working and sent out tapes on a minute's notice.

This book is a product of a love for data structures and algorithms that can be obtained only from top educators. I'd like to take the time to thank Bob Hopkins, E. C. Horvath, and Rich Mendez, who taught me at Cooper Union, and Bob Sedgewick, Ken Steiglitz, and Bob Tarjan from Princeton.

Finally, I'd like to thank all my friends who provided encouragement during the project. In particular, I'd like to thank Michele Dorchak, Arvin Park, and Tim Snyder for listening to my stories; Bill Kraynek, Alex Pelin, and Norman Pestaina for being civil next-door (office) neighbors, even when I wasn't; Lynn and Toby Berk for shelter during Andrew, and the HTMC for work relief.

Any mistakes in this book are, of course, my own. I would appreciate reports of any errors you find; my e-mail address is weiss@fiu.edu.

M.A.W.

Miami, Florida

September 1992

Go to Chapter 1 Return to Table of Contents


Structures, Algorithm Analysis: Table of Contents

Data Structures and Algorithm Analysis in C

by Mark Allen Weiss

PREFACE

CHAPTER 1: INTRODUCTION

CHAPTER 2: ALGORITHM ANALYSIS

CHAPTER 3: LISTS, STACKS, AND QUEUES

CHAPTER 4: TREES

CHAPTER 5: HASHING

CHAPTER 6: PRIORITY QUEUES (HEAPS)

CHAPTER 7: SORTING

CHAPTER 8: THE DISJOINT SET ADT

CHAPTER 9: GRAPH ALGORITHMS

CHAPTER 10: ALGORITHM DESIGN TECHNIQUES

CHAPTER 11: AMORTIZED ANALYSIS


Data Structures: CHAPTER 1: ALGORITHMS AND TIMING ANALYSIS

 Return to Table of Contents

CHAPTER 1: ALGORITHMS AND TIMING ANALYSIS

The usual way of learning to program a computer is to write programs that solve direct, simple problems. A natural result is that programs written during the learning process are quite short; they focus on some new idea or language feature. By contrast, many interesting and useful programs require greater effort than do these beginning exercises.

Some program applications are interesting or useful to many people, and good programs that deal with them are important and sometimes commercially valuable. These programs include games and simulations of competitive quality, electronic spreadsheets, accounting packages, class rollbook managers, scheduling programs, inventory control packages, and systems software that makes the use of micro-processors easier. A commercial program that takes less than a person-year of full-time effort is considered "small."

Because commercially available software does not always match individual needs precisely, a great deal of programming is largely redundant. The solution to this waste is to develop high-quality software designed with concern for flexibility and generality. The variety of needs in the computing community is expanding exponentially because of the increase in the number and variety of users. There will be a large demand for nontrivial software for some time to come, and someone must fill it. (Some entrepreneurs have even created a demand and filled it at the same time by writing a program that proved to be a generally useful tool.) The programming that can satisfy these demands is eased by the knowledgeable use of computing resources, and that involves the structuring of data.

1.1 Sophistication in Programming

Interesting applications call for sophisticated programs. They seldom call for programs that are merely long or for the use of the minor details of a particular programming language. This textbook is designed to help students learn to write interesting, nontrivial programs. That requires sophistication, which is based on technique, analysis, and knowledge.

A crucial aspect of programming technique is developing a logical structure that makes each program easier to read, debug, alter, and document--in short, that makes it an effective tool for communication. The technique of structured programming is a major feature of introductory programming books; this book will reinforce the reader's practice of it, in part through examples and in part through writing programs that demand (1) clean structure and (2) modularization.

Analysis is needed to determine how effectively a proposed program solves a given problem. The effectiveness of a program is determined by its use of resources--memory, computing time, and programming time. Quite often, the comparison of programs is reduced to the comparison of the problem-solving processes--the algorithms--upon which the programs are based. The analysis of algorithms is a major topic in itself, but it is introduced very briefly in this chapter. The ideas that are introduced are used to make comparisons throughout the book, and analysis problems are included at the end of several chapters.

Comparisons cannot be made without knowledge of more than one approach to a problem. Effective ways to arrange and manage data are not always obvious to the beginning programmer. There are applications and relationships between applications that one does not realize when first learning to program.

The intent of this book, then, is to provide a solid basis of knowledge to underlie programming sophistication while reinforcing structured programming and promoting an awareness of analysis.

The Data Structures Connection

The Use of Pseudocode

The Data Structures Connection

The study of data structures is a natural link between learning a programming language and writing application programs. Data may be arranged and managed at many levels: within memory; at the programming-language level; in terms of the logical structure of an algorithm; and at the conceptual level, where a mental image of arrangements and relationships is developed. Programming tactics can differ at each of these levels, and the levels interact. For example, suppose that algorithms Alpha and Beta solve the same problem. Alpha is more wasteful of some resource than is Beta, and Beta is easy to implement in language A but awkward or costly in programming time to implement in language B. If programming is to be done in language A, Beta is the obvious choice, but if it is to be done in language B, a balance must be struck. The specific details of the application determine whether the awkwardness of Beta or the waste of the resource by Alpha is more significant.

A common example of the interaction among these levels is that the algorithms used for manipulating data in one structural arrangement are different from those used to manage data arranged in another structure. The overall plan of attack for a problem depends on a choice of data structures, which in turn may depend on features of the language to be used.

The Use of Pseudocode

The separation of a structure from any particular level of design means specifically that the structure must not be confused with a feature of some programming language, or even of such languages in general. Part of the separation is possible when algorithms are described in a form of pseudocode. Pseudocode is a carefully restricted set of statements that describe the logic of a program for people but not for computers. The restrictions promote good programming; pseudocode statements are chosen to allow easy translation into programming languages. This approach can cause problems when used in a cavalier fashion, because some "solutions" to a problem may prove difficult to implement in a working language. For example,

Unlock(door)

is a neat solution to a problem--if you have a key (that is, if the procedure Unlock can be rendered in a practical way). Pseudocode is used in the pages that follow, but we keep in mind that algorithms are to be translated into a working language before use, with a structured language such as Pascal, ADA, or Modula-2. Clean, logical structures are easier to translate into these languages than into others, but each language has its own foibles because each is itself the result of choices.

The conversion of a pseudocode description of an algorithm into a working language is a learning experience that focuses on the logic involved and distinguishes that logic from language features. A program or procedure given in executable form can be applied blindly and without understanding. Pseudocode requires some attention but no major effort to translate. At the same time, translating an algorithm into code is much quicker and easier than deriving the algorithm itself and thus increases the amount of material that can be applied in a limited time.

The meaning of each of the pseudocode structures used in this book is explained in the next section.

Note: Part II of this book, "Expansions and Applications," includes working programs realized in Pascal. They implement some of the procedures presented in pseudocode in the main part of the text.

1.2 The Pseudocode Language SUE

If it weren't for the powerful effects of data structures and of other ideas, commands given to computers could be comfortably limited to six basic operations:

1. Copy data from one memory cell to another.

2. Perform arithmetic and logical operations.

3. Input data: place data from the outside world into memory.

4. Output data: retrieve data from memory and send it to the outside world.

5. Branch unconditionally to some command.

6. Branch on the truth of a condition to some command.

These commands are communicated to a computer from a translator (usually a compiler or assembler). The translator derives the commands from statements in the language being translated. Because such statements need to be explicit and unambiguous, they involve rules of syntax and conventions that are not really required to describe the logic of a program. Algorithms, which communicate program logic to people, can be reduced to simpler and more universal statements than current working languages allow.

The logical structures that follow are universal in the sense that the ideas involved can be implemented in any language. The language SUE is designed to focus attention on these logical structures, to be flexible, and to be expandable as required. All of the algorithms in this book are described in SUE.

Note: The name SUE has been used for other languages, but it is difficult to avoid all short mnemonics; SUE is used here only as a useful abbreviation.

The essential constructs of SUE are

variable    expression    Assignment. The keystrokes that form the

                          assignment operator are a matter of syntax.

Write(v1,v2, . . . ,vn)     Output the values of a list of variables or

                          messages. (The distinctions between Write

                          and Writeln in Pascal, and between the

                          effects of commas and semicolons in BASIC,

                          are features of specific languages.)

Read(v1,v2, . . . ,vn)     Input the values of a list of variables. This

                          form, and Write, will also serve for files

                          when v1 is a file identifier.

Read and Write are intended to be more general than they are in most working languages. Variables may be of any structural type in SUE. The format of the output of an array, record, or other variable type is provided during the translation of SUE into a working language:

if condition             S1 and S2 may be any sequence of

   then S1               statements. The else phrase may be missing. In

   else S2                 a nested structure, an else phrase is taken to

   endif                 be part of the most recent if.

For example

if A < B

then if A < C

then Write(C)

else Write(A)

endif

endif

will, when A = 2, B = 3, and C = 1, display 2. By contrast, if the else were taken to be the alternate condition for the first if, there would be no output from this segment. Astute indention may resolve an else for the human reader, but it will not do so for a computer, and so endif terminates all if statements in the text.

Another kind of selection statement is case, which may appear in two forms:

case              The conditions condi must be TRUE or

   cond1: S1       FALSE. Si can be any sequence of

   cond2: S2    statements, and will be executed iff (if and only

   .           if) condi is TRUE. It is the programmer's

   .           responsibility to make sure that only one

   .           condition can be TRUE; there should be no

   condn: Sn       overlap. The else may be omitted, in which

     else: Se    case if no condi is TRUE, then no execution

   endcase     will take place in this statement.

case of V     Here the variable V must take on discrete

   Value1:     S1  values. It is sometimes convenient to use

   Value2:     S2  values such as Small, Medium,

  .           Large--instead of, for example, 1,2,3--although many

  .           languages do not support this feature. A

  .           language that restricts case values to be

   Valuen: Sn    ordinal (ordered discrete), may require a table

    else: Se    lookup or other map for implementation.

endcase

General case structures sometimes must be resolved in terms of nested if . . . then . . . else statements in a program.

Loop structures may be classified by the location of the test for exit from the loop: at the top, at the bottom, or in the middle. They all must determine the beginning and the end of the loop; that is, what is included and what is not, as well as the exit-test position. The three essential loop forms in SUE are:

repeat                     Here S1 and S2 may be any sequence of

  S1                       statements. The test for exit is in the middle,

  if cond then exit endif  not at the top or at the bottom. The

  S2                       condition must be TRUE or FALSE. The exit

  forever                  passes control to [next statement].

[next statement]

Note: Not all structured languages (notably standard Pascal) provide for exit. An exit can be made from any loop by declaring a label and using a GOTO (an unconditional branch).

An example of a GOTO in Pascal is

Pascal                   SUE

REPEAT                   repeat

  Read(x);                 Read(x)

  IF x < O THEN GOTO 1;    if x < 0 then exit endif

  Sum := Sum + x           Sum    Sum + x

  UNTIL Sum > 100;         until Sum > 100

1 : Write (Sum)          Write(Sum)

repeat                   The condition cond must be TRUE or

  S                      FALSE. S is any sequence of statements.

  until cond             The test for exit occurs explicitly at the

                         bottom, thus S will be executed at least once.

                         S may include an exit.

while cond do            The condition cond must be TRUE or

  S                      FALSE. S is any sequence of statements.

  endwhile               The test for exit occurs explicitly at the

                         top, thus S may not be executed at all. S may

                         include an exit.

A common mode of looping is the use of a control variable as a counter to keep track of the passes through a loop. This may be done in SUE with:

for v = Init to Final by Step do  S is any sequence of statements. The

   S                              next v may be replaced by endfor if

   next v                         convenient. The by Step part is

                                  omitted when the Step value is 1. The

                                  use of downto and a negative step

                                  value will run thae count from high

                                  to low.

The for loop is equivalent to the following, with one disclaimer stated below:

v    Init            Init, Final, and Step may be expressions,

while v    Final do   and the control values derived from them on

  S                  entry to the loop do not change during loop

  v    v + Step      execution. This is true even if they are

  endwhile           variables that have assignments made to them

                     within the loop. The expression Step may be

                     negative, in which case the test for exit

                     becomes while v >= Final do.

The disclaimer is that--in some languages and on some machines--compiler programs apply sophisticated (increment-and-branch) instructions in the translation of the for loop construct. The exact translation is determined by the writer(s) of the compiler and influenced by the properties of the hardware, not by definition of the language alone. Although the for loop is usually implemented in programming languages as a while, this is not always the case. For example, the FOR . . . NEXT loop in at least one heavily used BASIC translator is essentially a repeat . . . until structure, as is the analogous FORTRAN DO loop. In SUE, a for is a specific form of while, and the value of v immediately upon completion of the loop is unknown.

1.2.1 Subprograms

1.2.2 Records and Pointers

1.2.1 Subprograms

Computing languages differ in the way they pass information between programs or between parts of one program. SUE simplifies the description of information- sharing as much as possible, while remaining consistent with the modularization of programs in structured languages.

A procedure in SUE is a segment of code that is independent in the sense that it can be invoked conveniently, it does a specified task, and it then returns control to the point immediately following its invocation. A program is itself a procedure that may invoke functions or other procedures, called subprograms. A list of parameters that provide information to or return information from a procedure may be included in the invocation and the definition. Variables not listed are considered to be global (defined and available outside of the procedure). As an example:

procedure Main                procedure AddAbs(v)

   .                             if v < 0

   .                                then Sum    Sum - v

   .                                else Sum    Sum + v

   Sum    0                         endif

   AddAbs(x)                     end {AddAbs

   Read(y)

   AddAbs(y)

   Write(Sum)

   .

   .

   .

end {Main

The variable Sum is global to AddAbs, but v is local. This means that the identifier v is redefined within AddAbs, and its relationship to x and y is determined at the time of individual invocations. Identifier v is a parameter that is related to x the first time, and to y in the second call.

The formal parameter v can be related to actual parameters x and y in two major ways:

1.     Treat x (or y) as an expression. Its value is stored locally in v. No other information about the source is available to Abs. In this case, v is called a value parameter.

2.     Treat x (or y) as an address. The address is stored in v, which becomes a pointer. Whenever an assignment is made in Abs to v, the assignment is actually to the (actual) parameter x. In this case, v is called a variable parameter.

The passing of parameters to subprograms is a major topic in programming languages, and no attempt to resolve or restrict it is made in SUE. A parameter in SUE may be of either type, but it is generally left unspecified. The choice may be important during implementation of an algorithm in a specific language. The best guide to making this choice is an understanding of the algorithm.

A function in SUE is a procedure that returns a single value. A function, therefore, may be invoked by using it anywhere that a variable can be used. The result that it calculates becomes the value of the function name. For example:

procedure Main                  function Abs(v)

   .                               if v < 0

   .                                  then Abs    -v

   .                                  else Abs    v

   Sum    0                           endif

   Sum    Sum + Abs(x)             end {Abs

   Read(y)

   Sum    Sum + Abs(y)

   Write(Sum)

   .

   .

   .

end {Main

The essential distinction between Abs and AddAbs is that the identifier Abs is assigned a value in its definition and can be used to retrieve that value, but the identifier AddAbs has no assigned value--it is the name of a subprogram that performs a task.

A valuable tool for presenting the logic of some algorithms (and for implementing them when it is available) is return. A return may be used to exit immediately from a subprogram at any point. If the subprogram is a function, return v indicates that v is to be returned as the functional value. When it is not available as a language feature, return is implemented much like exit.

The correspondence between Pascal, for example, and SUE can be quite close:

Pascal                           SUE

FUNCTION Min(m,n : REAL): REAL;  function Min(m,n)

  LABEL   1;

  BEGIN

    IF (m < 0) AND (n < 0)         if (m < 0) AND (n < 0)

      THEN BEGIN                      then return 0

        Min := 0;

        GOTO 1                        endif

        END;                       if m > n

    IF m > n                          then return n

      THEN Min := n                   else return m

      ELSE Min := m;                  endif

1 : END;                           end {Min

Note: Simplifications applied in this book are the use of global variables and of multiple exits (from loops) and multiple returns (from subprograms). It is only their thoughtful use that is being endorsed here, and they may be routinely removed when algorithms are translated into a working language.

1.2.2 Records and Pointers

For the sake of completeness, records and pointers are reviewed in this section.

In a gradebook program, a name and a score may be kept in separate arrays (because they are of different types) and tied together logically by giving them the same index. An alternate storage structure for the names and scores is a record, perhaps declared by:

A record, as it is used here, is a data structure in its own right, available in many languages. As a structure, however, it differs from the array structure in only two ways:

1.     The cells of a record do not need to contain data of identical type.

2.     The identifier of a cell (called a field in a record) is not an index, but a name. For a variable Good of type Result, Good.Name identifies the field of type STRING and Good.Score identifies the field of type INTEGER.

Some data structures are formed from records called nodes that are linked together in the sense that a node contains the address of one or more other nodes. Such addresses are made available as variables so they can be altered. A variable that contains the address of data, rather than the value of data in question, is called a pointer variable.

In a simple structure, nodes may contain single values (of some type). Each node also contains an address value, called a link,and so it can be declared to be:

For a variable xNode of type Nodule, xNode.Value is the name of the field of type REAL, and xNode.Link is the name of the field of type   [image]  Nodule.

Suppose that xNode is a record of type Nodule. A pointer to a record like xNode assumes the role of a node identifier, simply because a pointer, like an identifier, is a locator. (Both technically determine more information than simply an address, but it is the same information as that provided by languages that allow pointer variables.) If x is a pointer to xNode, then x   [image]  is itself an identifier xNode, and there may not be a need for a variable xNode. The value of x is the address of xNode. This is pictorially represented by:

If xNode is of type Nodule, then x is of type   [image]  Nodule. When p is a pointer of type   [image]  Nodule, then NEW(p) will be assumed to have the effect

without an assumed initialization of p   [image].

1.3 The Timing of Algorithms

Comparisons of algorithms are made on the basis of the management of resources, and one crucial resource is execution time. It is generally not necessary to determine exact execution times, which may be data-dependent or machine-dependent anyway and which can be seriously misleading in interactive systems. There is an important difference, however, between an algorithm that takes a minute or so to run and one that takes an hour.

The difference in execution times between two approaches to solving a problem may be small for small data sets; for large data sets, by contrast, the execution times of one approach may grow very differently from those of the other. For algorithms that are to be used many times, any difference can be significant, but generally an analysis of execution time concentrates on how it grows with the increasing size of data sets.

The growth of execution time as a function of the size of the problem data provides a basis for comparing ways of solving the problem.

A simple approach that is frequently adequate for our purposes can be applied to the following program segment:

[statement 1]

for i = 1 to n do

[statement 2]

[statement 3]

next i

An analysis may begin by assuming that [statement k] takes one unit of time for k = 1,2,3. Then this segment requires one unit of time for [statement 1] followed by two units of time for each pass through the for loop. Therefore, the execution time is

T(n) = 1 + 2n

This can be easily generalized by assuming that [statement i] requires ti units of time:

T(n) = t1 + (t2 + t3)n

This equation remains true even if [statement 2] is itself a loop that always requires the same amount of time to execute. For example, if [statement 2] is

[statement 2]: for j = 1 to m do

[statement 4]

next j     {t2 = m    t4

Complications arise, however, if the execution of [statement 2] depends upon i:

[statement 2]: for j = 1 to i do

[statement 4]

next j

Now we have, since m = i:

At this point, we need to document some of the properties of summations, because loop execution times are a sum of the times required for each pass through them.

where k < n and f is any function of i

where a and b are not functions of i

From equation ii, then, the value of T(n) is:

If it may be assumed that t1 = t3 = t4 = 1, then:

a quadratic function on n.

As a more specific example, consider:

EP1.1     Input n numbers, and list those that follow the occurrence of the minimal entry value.

One solution is:

program TrailMin

Read(Entry[1])

Mindex    1

for i = 2 to n do

Read(Entry[i])

if Entry [i] < Entry[Mindex]

then Mindex    i

endif

next i

for i = Mindex + 1 to n do

Write(Entry[i])

next i

end {TrailMin

In order to make a detailed analysis of the execution time of TrailMin, it would be necessary to assume some relative execution times of the Read, Write, if, and assignment statements and to perhaps consider the test-for-exit overhead implicit in the for statement. A first approximation ignores these differences and concentrates on the number of times that statements are executed. This provides a basis for an overview of the behavior of an algorithm.

With the simplifying assumption that each statement requires one unit of time, the timing function for TrailMin is:

Clearly, though, Mindex may be a function of n itself. In section 1.3.2 it is shown that Mindex can be expected to have the middle value (n + 1)/2, and so T(n) may be taken to be (5n/2) - 1/2 as a reasonable approximation. If the point of the analysis is the growth of T(n) as n increases, then it is enough to have found that T(n) is of the form: a   [image]  n + b, a linear function of n.

The growth behavior of an algorithm is quite often taken to be the salient fact revealed by timing analysis, and it will be explored further in the next section. Growth behavior provides a way to choose between algorithms that are to be applied to large data sets.

Another way to characterize Mindex as a function of n arises from the recognition that the execution time of Mindex is partly determined by the distribution of values in the sequence of entries. Random variation plays a role in the behavior of algorithms, and thus statistics should play a role both in the comparison of algorithms and in the study of the data structures on which they are founded. (See section 1.3.2.)

The analysis of algorithms includes the study of the worst-case data set of a given size, the (statistical) average case, and the limits imposed by the problem itself. In some cases, results of importance are derived experimentally. In this text, the intent is in a rough but convenient measure that is fairly easy to recognize for common algorithms. That measure is the order of growth.

1.3.1 The Order of Growth

1.3.2 The Variability of Data     (optional)

1.3.1 The Order of Growth

Not all timing functions are linear in the size of the data set being processed. Common timing functions include quadratics and other polynomials, logarithms (a base of 2 is used in such studies), exponential functions, and others. These functions grow with n at dramatically different rates, as Table 1.1 reveals.

Table 1.1

Suppose that procedure A and procedure B are both solutions to the same problem, and the timing functions for A and B are:

Suppose further that a time unit is 1 microsecond (10-6 second). Then whenever n = 1024, TA(n) is about 0.01 second, and TB(n) is about 1 second. However, for n = 65536, TA(n) is about 1 second, and TB(n) is roughly 1 hour and 12 minutes. If these procedures need to be used several times to accomplish some task, then procedure B becomes impractical for large data sets. A third procedure with T(n) = 2n would require 2.81 X 1014 years to deal with just 48 data values, and that is about 20,000 times as long as the age of the universe! Clearly, algorithms with execution times that grow exponentially are to be avoided for large data sets. Unfortunately, for many very important problems, all known solutions grow exponentially. Research in the theory of algorithms is being actively pursued to study that kind of behavior and other problems.

When it is desirable to determine the growth rate of a function, it is useful to eliminate as many extraneous details as possible. As n grows large, any linear function TL(n) = a   [image]  n + b will be overtaken, passed, and left in the dust eventually, by any quadratic function TQ(n) = c   [image]  n2 + d   [image]  n + e. This happens no matter what the values of a, b, c, d, and e may be. The crucial factor, for a polynomial, is the highest power of n.

Comparison of growth rates is made explicit by the following:

DEFINITION: f(n) = O(g(n)) iff (if and only if) there exist constants c and m such that |f(n)|   [image]  c|g(n)| for all n > m.

The term O(g(n)) is read "big Oh of g of n." The statement f(n) = O(g(n)) says, in effect, that g(n) grows at least as fast as f(n). An example of its formal use is a   [image]  n + b = O(n). A proof of this is:

for all n > 1.

We need only to extract the kernel of this approach:

The most useful orders and their relative growth rates are:

Any polynomial has an order less than the order of an exponential function. The fastest growth rate dominates, and so, for example

for any polynomial P(n).

Fortunately, the order of an algorithm can often be found without going through a detailed analysis. Consider:

[statement 1]

for i = 1 to n do

[statement 2]

[statement 3]

next i

The order is clearly O(n) as long as statements 1, 2, and 3 may be assigned unit time. If they call other procedures, however, then it may be necessary to use the analysis techniques at the beginning of section 1.3.

There are some important points to note about the analysis of algorithms:

      If large data sets are to be handled by a procedure, then the order of its timing function is of great concern. It may be possible to recognize the order without a detailed analysis, as we will often do, and some approaches can be dismissed on that basis.

      It is possible to place too much emphasis on the order of T. Many procedures are applied exclusively to relatively small data sets. If so, concern with O(T(n)) may obscure an important difference in the execution times of two algorithms.

      The variability of data sets can profoundly affect the behavior of an algorithm. Much of statistics can be brought to bear on the study of computing processes in situations where it is useful to do so.

      Some well-known algorithms have defied analysis, and others arise in practice that are not cost-effective to analyze in detail, even if one is prepared and able to make the analysis. It is possible, and sometimes profitable, to study procedures experimentally by varying data sets and observing the reaction of the procedures to the changes.

Exercise E1.1 and problem P1.1 are appropriate at this time.

1.3.2 The Variability of Data     (optional)

When TrailMin (see section 1.3) is examined in terms of the order of growth, it is evident that T(n) = O(n), no matter what value Mindex has. For a more detailed analysis, it is necessary to study the variations of Mindex. The statistical tool that is required is a statistical average, the expected value. If a variable, x, can take on values with probability P[x], then the expected value is:

The possible values of Mindex are 1, 2, . . ., n. If they are all equally likely, then their probabilities are all equal to 1/n. Hence:

It may be known, however, that Mindex values tend to occur late in the sequence of entries. Suppose, for example, that the probability is q when Mindex = 1, 2q when Mindex = 2, and in general, kq when Mindex = k, for k = 1, 2, . . ., n. One of these has to be the value, so the probabilities sum to 1:

And so

whence:

Now the expected value of Mindex is:

With this result, using E[Mindex] to represent Mindex:

1.4 Comparison of Three Sorts     (optional)

There are many ways to sort data, and most of them have appropriate applications. The three ways that will be explored here are not suitable for large data sets, but they are relatively easy to understand and may be familiar already. The essential problem to be solved is:

EP1.2     Input n numbers, sort them, and then display them in nonincreasing order.

A whimsical solution, which is also the solemn solution of top-down programmers, is:

program InOut(n)

Ingress(n) {Input n entries into array e.

Sort(n)    {Arrange the entry values in nondecreasing order.

OutHaul(n) {Display e in (its new) order.

end {InOut

The procedures Ingress and OutHaul are quickly analyzed:

procedure Ingress(n)                                 {O(n)

for i = 1 to n do

Read(e[i])

next i

end {Ingress

procedure OutHaul(n)                                 {O(n)

for i = 1 to n do

Write(e[i])

next i

end {OutHaul

Systems analysts might leave the grubby details of Sort to a (lower-paid) junior programmer. We would be derelict in our duty if we did this because Sort is the heart of procedure InOut. Thus, the decision about which sorting technique should be used is a major one. The splitting of InOut into three procedures allows us to concentrate on the kernel of the problem without distractions. The difficulties are put off and isolated, in this case by dealing with input and output external to Sort.

1.4.1 SelectionSort     (optional)

One sorting technique begins by searching the entries for a maximal value and exchanging it with e[1]. The maximal value is then in e[1]. A repeat finds a maximal value of the other n - 1 entries and exchanges it with e[2]. Exchanges continue until e[n - 1] is in its proper place. When the first n - 1 values are arranged in nonincreasing order in this way, they are all in their proper places. Figure 1.1 is a series of snapshots of the process.

Figure 1.1

This process can be implemented by using the variable Target as the index of the position into which a maximal value is being exchanged. Then, for each value of Target from 1 to n - 1, a search is made for the maximal value of e[i], for all i   [image]  Target. This determines an index, Spot, of that maximal value. The exchange is made with e[Target] and e[Spot].

procedure SelectionSort(n)

for Target = 1 to n - 1 do

Large    e[Target]

Spot    Target

Hun                   {For the index, Spot, of a maximal

                      {value among the remaining entries.

Temp    e[Spot]

e[Spot]    e[Target]

e[Target]    Temp

next Target

end {SelectionSort

If the time required to carry out the hunt is t, and a unit time is required for assignment statements, then the timing function of SelectionSort is:

Hunt must be developed in order to resolve this further:

procedure Hunt

for i = Target + 1 to n do

if e[i] > Large

then Spot    i

Large    e[Spot]

endif

next i

end {Hunt

We see that Hunt, in fact, is a function of Target, and should properly be denoted by Hunt(Target). The timing function for Hunt, if the statistical behavior of the if statement is ignored, is:

If it is necessary to know the statistical behavior of the if, then the analysis becomes more complex. One must decide on probabilities for the condition of the if to be TRUE on any given pass through the for loop. Such a detailed analysis is generally uncalled for, because the intent is often to get a reasonable estimate of the behavior of SelectionSort to compare it with other sorting algorithms. Of course, if the distribution of data values is known, and the growth behaviors of the algorithms are similar, then it may be worth the effort.

With the t that was calculated above:

The O(n2) result is no surprise, since it is often produced by a loop within a loop. One would be suspicious of any other result.

Exercise E1.2 is appropriate at this point.

1.4.2 BubbleSort     (optional)

Procedure SelectionSort moves the maximal value to the top of e in a direct way; smaller values eventually move toward the bottom. Another way to produce the same effect is to slide the smallest value to the bottom of e, as it is encountered on the first pass, then the second-smallest to the next-to-last position on the second pass, and so on. Because the smallest item cannot be recognized until a pass is complete, the smallest item so far slides downward during each pass. (This general technique is called a bubble sort here and in other places, although some programmers reserve that term for particular variations.) Large values slowly rise like a bubble in liquid, one position per pass.

Snapshots of the process that follow assume that the data in an array are originally in this order:

-5 2 10 -7 0 10

BubbleSort proceeds by rearranging the data on the first pass as shown in Figure 1.2.

Figure 1.2

The final result of the second, third, fourth, and fifth passes is shown in Figure 1.3.

Figure 1.3

One version of the process is:

procedure BubbleSort(n)

for Target = n downto 2 do

Slide(Target)    {smallest-so-far into e[Target]

next Target

end {BubbleSort

Clearly,

for BubbleSort, where t is the timing function for Slide. "Sliding" really means "exchanging as we go," so Slide becomes:

procedure Slide(Target)

for i = 1 to Target - 1 do

if e[i] < e[i +1]

then Temp    e[i]

e[i]    e[i + 1]

e[i + 1]    Temp

endif

next i

end {Slide

With the assumption that the if statement requires one unit of time, the timing function for Slide is

whence For Slide:

The surprise here is how similar the timing is for SelectionSort and BubbleSort, rather than how different. There may be a considerable difference, of course, if they are used many times on small data sets in some process. Differences in speed between algorithms with similar timing functions can be very data dependent and even reversed by some sets of data. One effect of chance is the number of exchanges actually made in Slide. An experimental comparison of algorithms may be in order if a critical choice is to be made between them.

A number of modifications can be made to BubbleSort, such as:

      Clear a flag variable just before entering Slide and reset it if an exchange is made. If the flag remains cleared, then the entries are already in order, and so successive passes may be aborted.

      Make one pass down, sliding small values; and make the next one up, sliding large values. The rationale for this is that when sliding down, a small value may move a long way, but a large value can bubble only one step up. There is then a choice between splitting Slide into Slideup and Slidedown or generalizing it to handle both directions.

Some modifications of BubbleSort are more efficient for some data sets, but not necessarily for all.

Exercise E1.3 is appropriate at this point.

1.4.3 InsertionSort     (optional)

Neither SelectionSort nor BubbleSort seems to be natural to card players. Most people sort cards as they are received from the dealer, one at a time. The first card is simply picked up. The second card is placed on the correct side of the first. The third goes on the outside of the first two or between them, and so on. The crux of this process is that the items input so far are always in order; thus, it is easy to search through them for the proper position of the next one. Rather than combine Ingress with Sort in procedure InOut, we will use the same process in situ: e[1] is the first value, e[2] is either exchanged with it or not, etc. A picture of the process in snapshots is shown in Figure 1.4.

Figure 1.4

When e[i] is to be (re)located, it is necessary to find the index Target of the position within the set {e[j]: 1   [image]  j   [image]  i}, where e[i] should be placed. If j < i, the values e[j], e[j+1], . . . for k = Target to (i - 1) need to be shifted out of the way. Finally, the old value of e[i] is assigned to e[Target]. This becomes:

procedure InsertionSort(n)

for Spot = 2 to n do

Find(Target,Spot)

Temp    e[Spot]

Shift(Target, Spot)

e[Target]    Temp

next Spot

end {InsertionSort

The timing function of InsertionSort is:

where f and s are the timing functions for Find and Shift, respectively.

The procedure Find is not so obvious as it may seem. The target cell can be defined as the first cell with a value smaller than that of e[Spot], but what if there is no such cell? The placing of -7 in the snapshots illustrates this situation. When e[Spot] is not larger than e [j] for j < Spot, then e[Spot] should be exchanged with itself or not at all--it is already in place. With this problem recognized, then:

procedure Find(Target,Spot)    {f = 1 + (Spot - 1)

Target    Spot

for i = 1 to Spot - 1 do     { = Spot

if e[i] < e[Spot]

then Target   i

exit

endif

next i

end {Find

Procedure Shift is straightforward as long as two things are recognized. First, this process will wipe out e[Spot]. That, however, was taken care of in Insertion-Sort itself (so that it would be visible to the reader and debugger). Second, the copy operation must proceed from e[Spot] back to e[Target] and not the other way around. (Trace it!)

procedure Shift(Target,Spot)

for i = Spot downto Target + 1 do

e[i]    e[i - 1]

next i                 {s = Spot - Target

end {Shift

Now we have:

If it is assumed that Target is as likely to be one value (see section 1.3.1) in the range 1 . . Spot as another, then the expected value of Target is

whence:

It is possible to speed up the Find process with a search technique called binary search, which will be discussed in section 7.3. Binary search will not change the order of InsertionSort, however, because the repeated use of Shift forces InsertionSort to be an O(n2) algorithm anyway. The Shift operation can be avoided with a data structure, the linked list, which will be discussed in Chapter 3. The use of a linked list, however, precludes the use of a binary search, so the Find operation is relatively slow.

One might be tempted to conclude that sorting algorithms are all O(n2) or larger. The proof of statements of this type is part of the theory of algorithms. The correct statement is that sorting algorithms are at least of the order n   [image]  ln n. The sorting problem itself is intrinsically O(n . ln n), no matter what algorithms are dreamed up to solve it. Sorting algorithms that have O(n   [image]  ln n) behavior are discussed in Chapters 8 and 9.

Exercise E1.4 is appropriate at this point. Programs PGA.1 and PGA.2 and project PJA.1 (See Part II, section  A of this book.) are appropriate but depend on familiarity with the material in Part II, section  A.

1.5 Random Processes     (optional)

Computing systems, the programs to which they are applied, and the systems that motivate creation of the programs all tend to increase in complexity as computer science matures. They frequently cannot be characterized by analytical means. One consequence is the growing realization that the behavior of these systems and of even relatively simple programs should be studied experimentally. In practice, that involves choosing from distributions with a pseudorandom number generator and applying statistical measures to results. A very brief introduction to tools that can be used for this kind of study is to be found in Part II, section  A.

Summary

Programming beyond the elementary level requires knowledge, analysis, and technique. Data structures are an essential part of the knowledge, analysis at some level is needed to compare algorithms, and technique comes with practice and with the awareness of possible variations.

Pseudocode provides a language-independent medium for expressing algorithms. One form (SUE) closely tied to Pascal and similar languages is introduced here.

An introduction to the timing of algorithms leads to the determination of the order of growth by summation or inspection. The growth behavior of algorithms provides a measure of comparison for them throughout the book. The discussion in this chapter is background for understanding the assignment of orders of growth. The timing of three basic sorting algorithms provides an example of the top-down summation approach to deriving timing functions.

A brief discussion of random processes is provided in Part II, section  A, as an indication of how the average-case behavior of algorithms and the experimental approach to investigating programs can be carried out. This material is optional and left to program assignments.

Go to Chapter 2            Return to Table of Contents


Data Structures: CHAPTER 10: GRAPH ALGORITHMS

 Return to Table of Contents

CHAPTER 10: GRAPH ALGORITHMS

Note: The terminology applied to graphs and to digraphs is distinct in the mathematical literature. The sequence of terms (chain, simple chain, closed chain, circuit) applied to graphs corresponds directly to terms (path, simple path, closed path, cycle) applied to directed graphs. The distinction lies in whether or not edges are directional. In a program, links are directional. If all links in a structure are double links, it models a graph rather than a digraph. However, the algorithms that search for simple chains in a graph, for example, actually locate simple paths. Such algorithms take a simpler form when working within a graph than they do if starting with a digraph. Nevertheless, the terminology (path, simple path, closed path, and cycle) are firmly entrenched in the literature in descriptions of graph algorithms. The standard terminology will be followed here without further apology.

A graph is a collection of V vertices and E (directed or undirected) edges that join the vertices into pairs (see section 6.3). This abstraction reflects a very common pattern; the number of systems and relationships that can be modeled by a graph is huge. An example of a graph that will be called G1 in this chapter is shown in Figure 10.1.

Figure 10.1

G1 has two components: subgraphs G11 with vertices {S-P-R-E-A-D-I-N-G} and G12 with vertices {O-U-T}. A component is a maximal subgraph having a sequence of edges connecting any two vertices in it.

A sequence of edges connecting two vertices is a path, but the sequence of vertices encountered as endpoints of the edges defines the same path. A simple path is one that repeats no vertex. For example, (S,A,P) specifies a simple path from S to P but (S,A,I,G,D,R,A,P) is not simple.

A cycle is a path that is simple except that the first and last vertices are the same. For example, (R,D,G,R) is a cycle.

A graph with no cycles is a tree. A collection of trees is a forest. Every connected component of a graph contains spanning trees, trees that contain every vertex in the graph. One spanning forest of G1 is shown in Figure 10.2.

Figure 10.2

Edges may have a cost (distance, weight) associated with them, as in a model of an airline-route system. A minimal spanning tree is one with the minimal total cost of the included edges. With weighted edges, pairs of edges joined by paths have a shortest path--of minimal total weight. Neither minimal spanning trees nor shortest paths are unique, and the latter may not lie in the former.

Exercise E10.1 is appropriate at this point.

If two vertices are joined by an edge, they are adjacent. Graphs in which every pair of vertices is adjacent are complete. Graphs that approach completeness are dense. Those with few edges are sparse. As a working definition, graphs for which E is much closer to V than to V2 are sparse.

A subgraph of a graph is biconnected if the removal of any one vertex leaves it connected.

Exercise E10.2 is appropriate at this point.

Given a representation of a graph, some of the processes that are of common interest are:

Traverse it in one of a variety of ways, visiting every node.

Determine its components.

Determine whether there is a path between vertex v1 and vertex v2.

Find a spanning tree in it.

Find out if the graph contains a cycle.

Determine its biconnected components.

Also of interest for weighted graphs:

Find a minimal spanning tree in it.

Find the shortest path between two vertices.

The efficiency of these operations depends on the data structure used to represent a graph as well as the algorithm used to traverse it.

10.1 Graph Representation

10.2 The Priority - Queue Traverse of a Graph

10.3 Applications of Priority-Queue Traverses

10.4 Traverses of Weighted Graphs

Summary

Exercises

Problems

Programs

10.1 Graph Representation

A graph can be represented by itself in the sense that every vertex can be represented as a node linked to all adjacent vertices. For an arbitrary graph of N vertices, N -- 1 link-fields would be needed in the nodes. The vertices of a graph may have a limited degree (number of edges to adjacent vertices). A graph of degree 2 or 3 or 4 could reasonably be represented as itself and a general visitation algorithm such as those of section 7.7 or Part II, section M, applied to traverse its components. Determination of some of its attributes would be difficult, however.

As usual with data structures, a graph representation may be either endogenous or exogenous. Processing is simplified if it is assumed that the structure is exogenous, and access to the information that would appear in a vertex is reduced to an index, 1 . . . N. The index can be used as the argument of a function that maps it to the corresponding information, including a node identifier. In this chapter, Tag(k) is taken to be the identifier of vertex k. In the other direction, if id is the identifier of a vertex, Index(id) is the corresponding index. Hence Index(Tag(k)) = k and Tag(Index(id)) = id.

We assume here that the raw data concerning vertices and the edges connecting them are available from a file, perhaps the system input file, and that the maps Index and Tag can be created as the data is read. We assume that the number of vertices V and the number of edges E are also known. In practice, sentinel values or End-of-File flags may be used during input, and the entries counted. The function Index in practice may be a search for an index field in a node, just the application of a Pascal ORD function, a search through a sorted table of identifiers, or several other possibilities.

The timing estimates applied to the algorithms of this chapter assume that Index and Tag are O(1).

The efficiency of the representation of a graph depends on whether it is sparse or dense and which algorithms are applied to that representation.

One common representation (introduced in section 6.3) is the adjacency matrix. In simple form, it is a Boolean matrix with TRUE entries in row i and column j iff vertices i and j are adjacent. (Variations are a binary matrix using 1's and 0's or a REAL matrix with values representing weights.) For some applications, it is convenient to set the main diagonal entries (at (1,1),(2,2), . . . ) to TRUE, and that is assumed in this chapter for unweighted graphs. The results for G1 are shown in Table 10.1, where T is used for TRUE and FALSE entries are blank.

Table 10.1

If a graph is dense, an adjacency matrix representation is viable. Almost any algorithm applied to the structure will examine the V2 entries in the matrix, but a dense graph has that order of edges to examine anyway.

Given V, E, and Index, the Create procedure for an adjacency-matrix representation is:

procedure CreateAM                 {O(V2 + E)

for i = 1 to V do

for j = 1 to V do

a[i,j]    FALSE

next j

next i

for i = 1 to V do a[i,i]    TRUE next i

for e = 1 to E do

Read(id1, id2)

i    Index(id1)

j    Index(id2)

a[i,j]    TRUE

a[j,i]    TRUE

next e

end {CreateAM

For sparse graphs, a common representation of a graph is the adjacency-list, not to be confused with the circular-list structure applied to sparse matrices in section 6.3. In its abstract form, the adjacency listing for G1 is:

S : A

P : A G

R : A D G

E : N

A : S P R I

D : R N G

I : A G

N : E D G

G : P R D I N

O : U T

U : O T

T : U O

Exercise E10.3 is appropriate at this point.

The adjacency-list structure used here is an array of pointers to linear lists, one pointer per vertex. The list associated with a vertex pointer contains nodes that represent edges. The nodes of such a list contain the index of the vertex that the edge connects to its header vertex. For example, the (partial) list

S : A

P : A G

.

.

.

A : S P R I

.

.

.

is actually represented by Figure 10.3 if S,P,R, . . .,T are nodes 1,2, . . .,12.

Figure 10.3

Note: The link structure L can also be a linked list of header nodes, and the adjacency node values are then pointers to the headers.

Creation of the adjacency list with the simple structure illustrated above is done by:

procedure CreateAL                  {O(V + E)

for k = 1 to V do

L[k]    NIL

next k

for e = 1 to E do

Read(id1, id2)

i    Index(id1)

j    Index(id2)

NEW(p)

p    .Value    j

p    .Link    L[i]

L[i]    p

NEW(p)

p    .Value    i

p    .Link    L[j]

L[j]    p

next e

end {CreateAL

10.2 The Priority - Queue Traverse of a Graph

One way to traverse a graph is:

PQT: repeat

1. Process some vertex, k.

2. Put all vertices adjacent to k "on hold"; that is, place them in a holding structure and mark them to be in the state HELD, giving them a priority value. The other possible states for vertices are DONE (processed) and UNSEEN (not adjacent to any DONE vertex).

3. Choose a vertex from the holding structure as the new k.

until all vertices are in the stale DONE.

In step 2 of PQT, a vertex m adjacent to k may already be in state HELD, in which case an algorithm based on PQT may or may not change the priority of m.

A priority queue, section 8.6.2, is the natural holding structure for such a traverse. In some algorithms derived from PQT, the priority-queue acts as a stack or as a queue, and should be replaced by the simpler and more efficient structure. In some applications of PQT, an unordered list may be an efficient representation of the holding structure, but it is used as a priority queue.

When vertices are held in a stack, step 3 of PQT chooses the most recently held vertex. However, the priority of vertices adjacent to it are (re)determined, even if they are already in the priority queue. In effect, they are moved to the top of the stack when reencountered. (The priority of vertices in state HELD are updated during the traverse.) The result of using a stack in this sense is depth-first-search (dfs). Starting at S in G1:

   DONE       HELD       UNSEEN

1. S          -     PREADINGOUT

2. S          A     PRE DINGOUT

1. SA               PRE DINGOUT

2. SA         PRI     E D NGOUT

1. SAI        PR      E D NGOUT

2. SAI        PRG     E D N OUT

1. SAIG       PR      E D N OUT

2. SAIG       PRDN    E     OUT

1. SAIGN      PRD     E     OUT

2. SAIGN      PRDE          OUT

1. SAIGNE     PRD           OUT

2. SAIGNE     PRD           OUT

1. SAIGNED    PR            OUT

2. SAIGNED    PR            OUT

1. SAIGNEDR   P             OUT

2. SAIGNEDR   P             OUT

1. SAIGNEDRP                OUT

2. SAIGNEDRP                OUT

At this point, all vertices reachable from S have been processed -- a component has been discovered, and a new start must be made:

   DONE          HELD  UNSEEN

1. SAIGNEDRPO              UT

2. SAIGNEDRPO    UT

1. SAIGNEDRPOT   U

2. SAIGNEDRPOT   U

1. SAIGNEDRPOTU

2. SAIGNEDRPOTU

When vertices are held in a queue, step 3 of PQT chooses the longest-held vertex. The result is a breadth-first-search (bfs). Starting at S for G1:

   DONE       HELD       UNSEEN

1. S          -     PREADINGOUT

2. S          A     PRE DINGOUT

1. SA               PRE DINGOUT

2. SA         PRI     E D NGOUT

1. SAP        RI      E D NGOUT

2. SAP        RIG     E D N OUT

1. SAPR       IG      E D N OUT

2. SAPR       IGD     E   N OUT

1. SAPRI      IGD     E   N OUT

2. 2SAPRI     GD      E   N OUT

1. SAPRIG     D       E   N OUT

2. SAPRIG     DN      E     OUT

1. SAPRIGD    N       E     OUT

2. SAPRIGD    N       E     OUT

1. SAPRIGDN   E             OUT

2. SAPRIGDN   E             OUT

1. SAPRIGDNE                OUT

2. SAPRIGDNE                OUT

At this point, all vertices reachable from S have been processed -- a component has been discovered, and a new start must be made:

   DONE          HELD  UNSEEN

1. SAPRIGDNEO              UT

2. SAPRIGDNEO    UT

1. SAPRIGDNEOU   T

2. SAPRIGDNEOU   T

1. SAPRIGDNEOUT

2. SAPRIGDNEOUT

A quite different trse results when the graph edges are weighted, and the weights are used for priorities. A weighting of the edges of G1 is shown in Figure 10.4.

Figure 10.4

Since the weights are of edges, not vertices, the "weight" of a HELD vertex is continually updated to be the lowest weight of any edge connecting it to a vertex that is in state DONE. For W1, starting at S:

   DONE       HELD       UNSEEN

1. S          -     PREADINGOUT

2. S          A     PRE DINGOUT

1. SA               PRE DINGOUT

2. SA         RI      E D NGOUT

1. SAI        PR      E D NGOUT

2. SAI        PRG     E D N OUT

1. SAIP       RG      E D N OUT

2. SAIP       RG      E D N OUT

1. SAIPG      R       E D N OUT

2. SAIPG      RDN     E     OUT

1. SAIPGD     RN      E     OUT

2. SAIPGD     RN      E     OUT

1. SAIPGDR    N       E     OUT

2. SAIPGDR    N       E     OUT

1. SAIPGDRN           E     OUT

2. SAIPGDRN   E             OUT

1. SAIPGDRNE                OUT

2. SAIPGDRNE                OUT

At this point, all vertices reachable from S have been processed -- a component has been discovered, and a new start must be made:

   DONE          HELD  UNSEEN

1. SAIPGDRNEO              UT

2. SAIPGDRNEO    UT

1. SAIPGDRNEOT   U

2. SAIPGDRNEOT   U

1. SAIPGDRNEOTU

2. SAIPGDRNEOTU

Note: The order in which vertices of a graph are processed is affected by their entry order, the vertex at which the search begins, the process of insertion into an adjacency-list, and perhaps some details of the implementation of PQT. There is not one breadth-first or depth-first search, but there is a family of them.

A number of refinements may be made to PQT in implementation, and they involve several variables and lists. For sparse graphs supported on an adjacency-list with link-header list L, a traverse uses the following:

Unseen is the state value of a vertex not yet encountered in a traverse. It is treated here as an integer larger than any other priority.

Root is a state value attained only by the first vertex of a component--one that is processed before any vertex to which it is adjacent is processed. Zero is usually used for the value of Root.

T counts the number of vertices removed from state UNSEEN--hence, the number of vertices that have been encountered in the traverse at any point in the traverse.

PQ[1 . . V] is an indirect priority queue in which low values have a high priority. It contains indices of vertices. It does not appear explicitly in the algorithm but is used by the priority-queue operators. All vertices are initially placed in PQ, with priority Unseen. The value Unseen is chosen to be greater than the priority of any HELD or DONE vertex. Vertices are removed one at a time with function Remove, as they are processed.

Priority is a variable calculated to sequence the removals from PQ and hence its definition varies from one application to another. It may be set to T for breadth-first search, V -- T for depth-first search, or an edge-weight to find a minimum spanning tree.

OnPQ[1 . . V] is a Boolean list for which OnPQ[k] is TRUE if the kth vertex is still in PQ.

State[1 . . V] contains the priorities of the vertices in PQ. When changed, the values in State are set to a Priority value.

L[1. . V] is the list of adjacency-list headers.

Remove returns and deletes the front value of PQ.

With these tools an adjacency-list traverse is:

procedure SparsePQT

T    0

for k = 1 to V do                   {initialization

State[k]    Unseen

OnPQ[k]    TRUE

PROCESS0(k)

next k

CreatePQ                            {queue all vertices

repeat                              {retrieve them one by one

k    Remove

OnPQ[k]    FALSE

PROCESS1(k)

if State[k] = Unseen              {a new component starts

then State[k]    Root

T    T + 1

PROCESS2(k)

endif

p    L[k]                         {check the adjacency-list

while p    NIL do

m    p    .Value

if OnPQ[m]

then if State[m] = Unseen

then T    T + 1

PROCESS3(k)

endif

if State[m] > Priority

then State[m]    Priority

PROCESS4(k)

endif

endif

p    p    .Link

endwhile

until Empty(PQ)

end {SparsePQT

Processes 0,1,2,3, and 4 are null or simple or elaborate, depending on the application. If they are all null, then the traverse leaves no trace of its passage and builds no structures suitable for further processing in an application; hence, their use is precisely to build a legacy.

An adjacency matrix form of this procedure, DensePQT, can be constructed by simply replacing the while loop with:

for m = 1 to V do      {check the adjacency-list

if a[k,m]

then if OnPQ[m]

then T    T + 1

PROCESS3(k)

endif

if State[m] > Priority

then State[m]    Priority

PROCESS4(k)

endif

endif

next m

10.3 Applications of Priority-Queue Traverses

This section describes information that is easily extracted during a priority-queue traverse of a graph.

The Process Sequence

The Process Map

The Parent

The Ancestor

10.3.1 Tracking a Traverse

10.3.2 Path-Connections and Components

10.3.3 Spanning Forests

10.3.4 Cycle Testing

10.3.5 Biconnectivity

The Process Sequence

The list of vertices in the sequence in which they are processed (in PROCESS1) is the process sequence. The entry for the root of a component in the process sequence is negated to make component detection easier. The list is kept in Done[l . . V] and generated by the inclusions:

:c    0

PROCESS1:  c    c + 1

Done[c]    k

PROCESS2:  Done[c]    - Done[c]

The Process Map

The list that specifies at what position in the sequence a vertex is processed is the process map. In effect, it is an inverse of (the absolute value of) Done and kept in WhenDone[1 . . V]. WhenDone[k] = c iff |Done[c]| = k. WhenDone is generated simply by changing the PROCESS1 generation of Done to:

PROCESS1:  c    c + 1

Done[c]    k

WhenDone[k]    c

The Parent

A vertex k that determines the final priority of a vertex m is its Parent, kept in Parent[1 . . V]. Parenting data is gathered by:

PROCESS0 : Parent[k]    0

PROCESS4 : Parent[m]    k

The Ancestor

A vertex that pulls the vertex m into the HELD state from the UNSEEN state is the earliest-processed vertex adjacent to m and is called its ancestor. The list Ancestor[1. . V] can be generated with:

PROCESS0 : Ancestor[k]    Unseen

PROCESS2 : Ancestor[k]    O

PROCESS3 : Ancestor[m]    k

The data-collection lists Done, WhenDone, Parent, and Ancestor are all generated for essentially no execution-time cost with O(1) operations. However, they do not directly answer the questions about graphs mentioned at the beginning of this chapter and certainly they do not represent the most convenient form for working with components, paths, and spanning trees. Mining the ore they represent is the subject of the subsections that follow.

Note: Almost any application of the preceding data-acquisition lists can be done more succinctly and often more efficiently by incorporating their ultimate use into the traverse itself. The advantage of using them as a standard data-collection format is just that--a system one can come to know and understand and apply quickly.

A trace of the generation of the data-lists by a dfs on G1, starting at S is:

S = State    A = Ancestor  D = Done

Pa = Parent  P = Priority  W = WhenDone

Tag   k   m   T   P   S   W    D   A  Pa

----------------------------------------

S     1                   1   -1   0   0

          5   2  10  10            1   1

A     5                   2    5   1   1

          2   3   9   9            5   5

          3   4   8   8            5   5

          7   5   7   7            5   5

I     7                   3    7   5   5

          9   6   6   6

G     9                   4    9   7   7

          2   6   6   6            5   9

          3   6   6   6            5   9

          6   7   5   5            9   9

          8   8   4   4            9   9

N     8                   5    8   9   9

          6   8   4   4            9   8

          4   9   3   3            8   8

E     4                   6    4   8   8

D     6                   7    6   9   8

          3   9   3   3            5   6

R     3                   8    3   5   6

P     2                   9    2   5   6

0    10                  10  -10   0   0

         11  11   1   1           10  10

         12  12   0   0           10  10

T    12                  11   12  10  10

         11  12   0   0           10  12

U    11                  12   11  10  12

The resulting values are summarized in Table 10.2.

Table 10.2

Tag[k]  k  Ancestor[k]  Parent[k]  WhenDone[k]  Done[k]

-------------------------------------------------------

  S     1      0           0           1         -1

  P     2      5           8           9          5

  R     3      5           6           8          7

  E     4      8           8           6          9

  A     5      1           1           2          8

  D     6      9           8           7          4

  I     7      5           5           3          6

  N     8      9           9           5          3

  G     9      7           7           4          2

  O    10      0           0           0        -10

  U    11     10          12          12         12

  T    12     10          10          11         11

Program PG10.1 is appropriate at this point.

10.3.1 Tracking a Traverse

The progress of a traverse may be printed (or listed on a file) with:

PROCESS1: Write(Tag[k])

Components may be separated in the listing with a marker (such as an End-of-Line) by moving the Write below the test for Unseen and adding:

PROCESS2: Write(EOLN)

This separation is possible because, if the vertex pulled from PQ is in state UNSEEN, it is not reachable from any previously processed vertex and must be in a different component.

Because Remove is O(1n V) when PQ is implemented as a heap, and every edge and every vertex is examined, traversal time for SparsePQT is seen to be O((E + V) ln V). This can be improved to O(E + V) when PQ is replaced by a stack (for dfs) or a queue (for bfs). For DensePQT the corresponding times are O(V2 ln V) and O(V2).

10.3.2 Path-Connections and Components

A linear scan of Done yields sets of vertices in the same component, separated by root vertices with negative entries. In a computing system that supports set variables and manages them efficiently, the component data may be shifted into sets during a linear scan for further processing.

If Done itself is to be used to decide whether two vertices are path-connected (in the same component), it is necessary to scan the component that contains one of them. If WhenDone[k1] = c1 and WhenDone[k2] = c2, then k1 and k2 are in the same component iff there is no negative entry between c1 and c2 in Done. This is an O(V) operation for each pair of interest. If it is to be repeated, a scan can be used to build a list of component-completion times, a binary search used to place c1 between two of them, and a quick check used to see if c2 lies between the same pair of component-completion times.

The union-find operations of section 9.7 can be applied to Parent to build set structures within which the search for a common root node efficiently answers the question of whether two vertices are in the same component.

An approach more efficient than union-find structures uses Done as a guide to construct fully collapsed component sets. The procedure MakeSet that follows points all of the vertices to the root node of their component, which has an index that is the current value of SetLink.

procedure MakeSet                     {O(V)

for c = 1 to V do

if Done [c] > 0

then Set [c]    SetLink

else SetLink    - Done [c]

Set [c]    SetLink

endif

next c

end {MakeSet

The Boolean expression for path connection is then:

Set[k1] = Set[k2]

Note: MakeSet works for T and V -- T priorities because with their use, root nodes are processed before other nodes in any component. It will not suffice otherwise.

The root nodes in Set are self-referencing, as an alternative to root indices of value 0.

The logic of MakeSet can be incorporated into SparsePRT (or DensePQT) by:

: c    0

PROCESS1 : c    c + 1

Done[c]    k

PROCESS2 : SetLink    Done[c]

Set[c]    SetLink

Done[c]    -- Done[c]

else Set[c]    SetLink

If Done is not needed for any other purpose, it may be dispensed with:

: c    0

PROCESS1 : c    c + 1

PROCESS2 : SetLink    k

Set[c]    SetLink

else Set[c]    SetLink

Exercise E10.4 is appropriate at this point.

10.3.3 Spanning Forests

The data in Parent determines a spanning forest, but the links point from child to parent, in the wrong direction for most tree processing. A treeform that can be traversed from root to leaf is an adjacency-list that omits links extraneous to the forest. Figure 10.5 shows the dfs traverse of G1 beginning at S.

Figure 10.5

Such a forest can be formed from a linear scan of Parent:

procedure MakeForest               {O(V)

for k = 1 to V do

Forest[k]    NIL

next k

for k = 1 to V do

p    Parent[k]

if p    0

then NEW(q)

q    .Value    k

q    .Link    Forest [p]    .Link

Forest [p]    .Link    q

endif

next k

end {MakeForest

This process can be included in SparsePQT (or DensePQT) by:

PROCESS0: Parent[k]    0

Forest[k]    NIL

PROCESS2: else p    Parent[k]

NEW(q)

q    .Value    k

q   .Link    Forest[p]   .Link

Forest[p]    .Link    q

PROCESS4: Parent[m]    k

The data in Ancestor also determine a spanning forest. It may be generated directly in processes 0, 2, and 3 without the use of the list Ancestor.

Exercise E10.5, problem P10.1, and program PG10.2 are appropriate at this point.

10.3.4 Cycle Testing

A connected graph G has a cycle iff a spanning tree does not contain all of the edges of G. An extra edge (v1,v2) provides a path (not in the tree) between v1 and v2; and, together with the connecting path through the tree, it forms a cycle. A graph has no edges that connect trees in its spanning forest, and so any edge not in the forest forms a cycle within a component.

For any edge (k,m) one vertex say m, is processed later than the other and so is encountered in the adjacency loop of the other. An extra edge can be detected as an adjacency of m to a vertex n where n is processed prior to k. In simplest terms, when this occurs, OnPQ[m] is TRUE and State[m] [image] Unseen in the adjacency loop of k. The test for a cycle in dfs is then:

PROCESS0: Cycle    FALSE

PROCESS3: else Cycle    TRUE

All cycles will be detected (and so could be counted or otherwise noted) because all edges are examined during the traverse.

10.3.5 Biconnectivity

An articulation point of a graph is a vertex with the property that, if it is removed, the remaining subgraph has more components than the original. All connecting paths between some parts of the graph and others must pass through it. The articulation points in the graph G2 of Figure 10.6 are A, B, and C.

Figure 10.6

A biconnected graph is one where every pair of vertices is connected by at least two disjoint paths, so it has no articulation point through which all of the paths between some pair of vertices must pass. If the edge (E,F) is added to G2 to form G3, then G3 is biconnected.

Biconnected components (maximal biconnected subgraphs) are not disjoint, in contrast to (connected) components. For example, G4 in Figure 10.7 has two biconnected components joined by the articulation point K, which lies in both of them.

Figure 10.7

As a result, a reasonable goal in the exploration of the biconnectivity of a graph is the determination of its articulation points.

The information needed to detect articulation points is implicit in a search tree generated by dfs when used in conjunction with the edges not in the tree--ghost links.

The dfs search forest for G1 by dfs starting at S has four ghost links, dashed in Figure 10.8. The dfs search tree for G2 starting at C has one ghost link, as it must, since there are seven vertices and hence precisely six edges in the spanning tree itself. The seventh edge must be a ghost, as shown in Figure 10.9.

Figure 10.8

Figure 10.9

Here C is an articulation point because the entire graph cannot be searched from just one of its children--it is the only link between the subtrees rooted at its children.

Note: All ghost links (edges left out of the dfs spanning tree) connect vertices to their ancestors. If there had been an edge connecting sibling subtrees, it would have been followed during the search of the first subtree to be processed.

As one consequence of this note:

If the root of a dfs search tree for a component has more than one child, it is an articulation point.

This form of articulation point can be detected by a linear pass through either Done (for negative entries) or Parent (for zero entries) or Ancestor (for zero entries) to find component roots. An O( l ) check of Forest detects multiple children for such vertices.

A ghost link, such as (C,F) in T3 (Figure 10.9), forms a cycle. All of the vertices from the ancestor down through the search tree to the parent to the ghost link vertex and back through the ghost link to the ancestor lie in the cycle and hence form a biconnected subgraph. The possible articulation points in this cycle are vertices with children not on the cycle and the ancestor. (The ancestor may be the only bridge between the cycle to that part of the graph higher in the tree.)

If a vertex k has a child that roots a subtree with no ghost links to vertices higher in the tree than k, then removal of k disconnects this subtree from the parent of k (and higher nodes). In general:

A nonroot vertex k is an articulation point iff it has a child that roots a subtree that has no ghost links to vertices higher in the tree than k. (Vertices higher in the tree are necessarily processed earlier.)

We define the minimum completion time (mct) of a collection of vertices {w1, . . ., wm} to be the minimum completion time of their collective ancestors (to which they are connected by either tree links or ghost links).

Suppose that the completion time of vertex p is c. Then c is an upper bound on the mct of each of the subtrees rooted by its children because there is a link or ghost link from p to each child. If any such subtree has an mct of c, p is an articulation point. The union of p and its subtrees has an mct that is the minimum of the subtree mcts and the completion time of the ancestor of p. Children are necessarily completed later than their parents. This suggests a way to search for articulation points:

1. Scan through Done backwards, assigning values to lists UrMin[1 . . V] and UrMax[1 . . V ]. They contain, respectively, the earliest known ancestor completion time and the latest known ancestor completion time of the subtrees rooted by the children of the vertex.

2. Suppose that vertex k is not a component root and its completion time is c. When the scan is complete, if UrMax[k] = c, then k is an articulation point.

3. The entries of Children[0 . . V] are initialized to zero. During the scan Children[k] is incremented whenever one of the children of k is processed. Then a component-root r for which Children[r] > 1 is true is an articulation point.

The resulting procedure is:

procedure Articulation              {O(V)

Children[O]    0                  {component counter

for k = 1 to V do

Children[k]    0

p    Parent[k]

if p = 0 then p    k endif

UrMin[k]    WhenDone[p]

Urmax[k]    0

next k

for c = V downto 1 do

k    |Done[c]|

p    Parent[k]

Children[p]    Children[p] + 1

if p    0

then a    Ancestor[k]

t    WhenDone[a]

if UrMin[k] > t

then Min    t

else Min    UrMin[k]

endif

if Min < UrMin[p]

then UrMin[p]    Min

endif

if Min > UrMax[p]

then UrMax[p]    Min

endif

endif

next c

end {Articulation

Note that UrMax[k] remains 0 for a leaf vertex k. A partial trace of the variables in Articulation from the data-acquisition lists of section 10.3 generated by the dfs search of G1 is shown in Table 10.3.

Table 10.3

 c   k   p   a   t  Min

12  11  12  10  10  10

11  12  10  10  10  10

10  10   0   -   -   -

 9   2   8   5   2   2

 8   3   6   5   2   2

 7   6   8   9   4   2

 6   4   8   5   5   5

 .   .   .   .   .   .

 .   .   .   .   .   .

 .   .   .   .   .   .

Completion of this trace and determination of UrMin and UrMax are left for an exercise. However, the last entry shown is sufficient to determine that

UrMax[8] = 5

and hence vertex 8 is an articulation point.

Exercise E10.6, problem P10.2, and program PG10.3 are appropriate at this point.

10.4 Traverses of Weighted Graphs

The weighted graph W1 of section 10.2 is reproduced as Figure 10.10 for convenience.

Figure 10.10

Weights are an attribute of edges, but a spanning tree that is to be traversed from vertex to vertex needs to have them attached to the vertices. Vertex weights are determined during the construction of the tree using the following definitions:

DEFINITION: The weight of a vertex in state UNSEEN is Unseen. The weight of a vertex in state HELD is the minimum weight of the edges that connect it to vertices in state DONE. The weight of a vertex in state DONE is the weight with which it exits state HELD.

The fundamental source of a weight is an adjacency-matrix entry or a field p [image] .w in an adjacency-list edge node. During a traverse, weights of vertices are stored in State, since they determine priorities. The sum of the final State list is the total weight of a spanning forest.

When vertex m is processed in the adjacency loop of vertex k in SparsePQT, the value of the variable Priority is p [image] .w. (This is the weight of the edge connecting k and m.) It has already been traced for W1 in section 10.2, and construction of the tree itself is covered in section 10.3.3.

The spanning tree derived from dfs by using weights as described above is a minimal spanning tree, produced by what is known as a greedy method. The crux of a greedy method algorithm is that it makes the best available move based on the information available at any stage of the process. In some situations, and forming a minimal spanning tree is one, a global optimum can be derived by repeatedly using a local optimum--the greedy method.

When the greedy method is implemented with DensePQT, it follows the scheme known as "Prim's Algorithm." An alternate scheme, known as "Kruskal's Algorithm" is to choose edges one at a time, lowest-weight first, rejecting those that would close a cycle if added to the forest. When complete--all vertices are included--there is one spanning tree for each component.

The traditional approach to Kruskal's Algorithm retrieves edges from a sorted list, say Edges [1 . . E]. A heap of edges would do as well but would save no processing time: retrieval of all edges is a sorting process of order (E In E). An edge in the list is taken to be a record e that contains a weight e [image] .w, and vertices e [image] .a and e [image] .b.

When an edge e is used to attach vertex m to a vertex k already in the forest, the process of attachment involves lists already familiar from section 10.3:

procedure Attach(m,k)

c    c + 1

WhenDone[m]    c

Done    m

State[m]    e    .w

Parent[m]    k

Set[m]    Set[k]

end {Attach

The resulting lists can be applied as they were in section 10.3, with minor variations. Parent entries of component roots need to be changed from self-reference to zero for complete conformity. WhenDone[k] is the time at which k is added to the forest, but not necessarily when it is joined into its final component tree.

Figure 10.11 shows some snapshots of the process applied to W1.

Figure 10.11

At this point, all of the vertices have been chosen; but, as Figure 10.12 shows, the trees of the forest do not represent components of the graph.

Figure 10.12

Problem P10.3 is appropriate at this point.

The test for cycle generation is a test to see if the two vertices of an edge lie in the same tree. Set will not work as it did in section 10.3.3 because component roots are not identifiable until the process is complete and roots are not necessarily added before their children. Some variation of the Find operation of section 9.7, acting on Set, is needed to test for a common component.

The version of Kruskal's Algorithm presented here examines the vertices of the (greedily) chosen edge. Neither, one, or both are already in the forest: vertex m is in the forest if WhenDone[m] [image] 0. If precisely one vertex is in the forest, the other is attached to it. If neither is in the forest, one is attached to the other. If both are in the forest in distinct trees, one is attached to the other; but if they are in the same tree, the edge is rejected:

procedure Kruskal

for k = 1 to V do

Parent[k]    0

Set[k]    k

WhenDone[k]    0

next k

c    0

ec    0

while ec < E do

ec    ec + 1

e    Edges[ec]

if WhenDone[e    .a] = 0                           {e    .a is not in the forest

then if WhenDone[e    .b] = 0                    {and neither is e    .b

then Attach(e    .b,e    .b)

endif

Attach(e    .a,e    .b)

else if WhenDone[e    .b] = 0                    {only e    .a is in the forest

then Attach(e    .b,e    .a)

else if Find(e    .a)    Find(e    .b)     {both are in the forest

then Parent[e    .b]    e    .a   {and in distinct trees

Set[e    .b]    Set[e    .a]

State[e    .b]    e    .w

endif

endif

endif

endwhile

end {Kruskal

Program PG10.4 is appropriate at this point.

10.4.1 The Shortest Path Problem

10.4.1 The Shortest Path Problem

The shortest path problem is to find the path in a weighted graph between two given vertices that has the minimal weight sum of all such paths.

If the weights are all one, bfs will find the shortest path from the starting vertex to all vertices in its component. A bfs search attaches to a component tree all vertices reachable from the root with one edge, then all those reachable with two, and so on.

The minimum spanning tree determination with SparsePQT repeatedly adds the vertex closest to the tree. The shortest path variation is simply to add the vertex closest to the root vertex r from which paths are to be formed. (Its state may be initialized to zero instead of UNSEEN so that it is chosen first.)

Suppose that SparsePQT is used to derive the shortest path. During the traverse, the vertices in state DONE are connected to r by shortest paths. If m is moved into state HELD from state UNSEEN in the adjacency loop of k, the shortest known path from m to r is formed by attachment of m to the tree at k. The total path length to r is the path length from k to r plus the weight of the edge (k,m). If m is already in state HELD, the path from m to k to r may or may not be shorter than the shortest path known prior to the processing of k. The test is simply made by using:

Priority = State[k] + p    .w

The final path length from r to any k is in list State; the individual edges in the shortest path tree can be derived from list Parent.

A variation for dense graphs, using the adjacency matrix, is to use the list State simply as that: an unordered list. The State entries of vertices not yet in state DONE are negated to distinguish them. The Remove procedure is replaced by a linear scan for the largest negative vertex (closest to zero). When the state of a vertex is changed in the adjacency loop, it is changed to - priority. When a vertex is moved into state DONE, its State entry is complemented to make it positive.

The use of a[k,m] for Priority values yields Prim's Algorithm for the minimal spanning forest, and the use of State[k] + a[k,m] for Priority values yields Dijkstra's Algorithm for the shortest path problem.

Program PG10.5 is appropriate at this point.

Summary

Problems of interest in a wide variety of applications include the mathematical model, graph. Graph models have been used for a long time and have accumulated a great deal of nomenclature concerning their paths and cycles. Graphs can be modeled in a program in a number of ways. The two most common structures for representing a graph are the adjacency matrix, most suitable for dense graphs, and the adjaceny list for sparse graphs.

A basic requirement of most algorithms that deal with a graph is a traverse of it, generally to make a search for something. The approach taken here is to describe a very general traversal that can be specialized and perhaps made more efficient for particular uses. It is a priority-queue traverse, PQT. PQT is shown to be general enough to include dfs (depth-first search) and bfs (breadth-first search).

PQT is used to generate data-acquisition lists Parent, Ancestor, Done, and WhenDone. These in turn are used to answer questions about a graph--its cycles, components, paths, spanning forest, and biconnection properties.

The use of edge-weights for priorities in PQT allows it to be used to find minimal spanning trees and shortest paths.

This chapter is formed around a central algorithm. PQT can be specialized, tuned, and expanded as needed. An advantage to this approach is that customizing can be done on familiar grounds, using the programmer's time efficiently.

Further Reading

Further Reading

The reader seeking background material on graphs for this chapter can try any number of finite mathematics texts. They usually include a chapter on graphs and indicate some of the areas of application. A survey of applications at a higher level is [ROBERTS, 1978], which has been supplemented by [ROBERTS, 1984]. Both contain extensive bibliographies.

Numerous books on graph theory are available for the undergraduate level, and the list is growing. The choice is yours. The same is true of books titled Discrete Mathematics.

Graph algorithms are being researched extensively. Several books about graph algorithms exist, and again the list is growing. One such is [EVEN, 1979].

A nice treatment of priority-queue traverses, including recursive versions, is given in [SEDGEWICK, 1983].

A number of graph-algorithm topics not covered in this chapter are relatively accessible. They include those for directed graphs, networks, and matching problems. Both [SEDGEWICK, 1983] and [TARJAN, 1983] have good treatments of these topics.

Exercises

Exercises immediate applications of the text material

E10.1 Find the smallest graph with more than one minimal spanning tree (mint) and a mint-(shortest-path) pair that have no edges in common.

E10.2 In G1 of section 10.1, determine:

(a) the longest cycle

(b) a spanning forest different from the one illustrated

(c) the largest biconnected subgraph

E10.3 Form the adjacency listing and adjacency matrix for G1 if the nodes are in alphabetical order.

E10.4 Trace the operation of MakeSet on the summary data for G1 given in section 10.3.

E10.5 Draw the spanning tree for G1 that would be produced by MakeForest altered to use Ancestor rather than Parent.

E10.6 Complete the trace of Articulation variables started in section 10.3.5 and determine UrMin and UrMax.

Problems

Problems not immediate, but requiring no major extensions of the text material

P10.1 Incorporate the effect of MakeForest (section 10.3.3), using Ancestor rather than Parent, directly into SparsePQT (section 10.2).

P10.2 Design a procedure that uses UrMax and Children to determine and list of articulation points.

P10.3 Change all of the weights w in W1 (section 10.4) to 10 - w and trace the behavior of Kruskal's Algorithm with snapshots.

Programs

Programs for trying it yourself

PG10.1 Write a program that will read a file of vertices and edges and display a summary of data-acquisition lists like the one for G1 at the end of section 10.3.

PG10.2 Write a program that determines and displays the spanning forest of a graph. A variation is to use PG10.1 to write files used as input by this program.

PG10.3 Write a program that inputs vertex, edge, and identifier data for graphs and displays components and articulation points. A variation uses the program of PG10.1 to create files used as input for this program.

PG10.4 Write a program that sorts an edge list of a graph and applies Kruskal to it and displays the data-acquisition lists that result from it.

PG10.5 Write a program that uses SparsePQT to determine the shortest paths from a given vertex to all the vertices reachable from it.

Go to Section A Return to Table of Contents


Data Structures: CHAPTER 2: THE ARRAY STRUCTURE

 Return to Table of Contents

CHAPTER 2: THE ARRAY STRUCTURE

In most HLLs, arrays are automatically available to the programmer, but they are actually created from cells or simpler arrays. When a compiler, an operating system, or an assembler program is written, and when an HLL of limited power is used, arrays are created by the programmer. This chapter serves to isolate properties that characterize arrays as structures and hence organize the building of arrays, and it presents the array structure as a model data structure. A review of the role of variables and arrays in programming is included in section 2.1.

2.1 Static Cell Groups (optional)

The simplest way to manage data is to place it in a single memory cell and identify that cell with a name. To the machine, the cell identifier is a memory address or the information used to calculate one. In a high-level language, the identifier is the name of a variable. A compiler (or interpreter, or assembler) program creates the bridge from one form of identifier to the other. However the name of a cell is specified, many problems can be solved using only the simple variable as a data structure. Examples of problems natural to simple variable solutions abound in early chapters of elementary programming texts. Many of them are variations of problems like EP2.1 (Example Problem 2.1), EP2.2, EP2.3, EP2.4, and EP2.5, which follow:

EP2.1 Input 10 numbers and print the maximal value of the entries.

EP2.2 Input 10 numbers and print their mean. (The mean of n entries is defined in Part II, section A.4.)

EP2.3 Calculate the nth Fibonacci number, F(n), where F(n) = F(n - 1) + F(n - 2), and F(0) = 1 = F(1).

EP2.4 Given that the average topsoil in this country has a depth of six inches, that each year it is eroding at a rate of l percent of the depth it has that year, and that it will fail to support agriculture at depths below three inches, when does the topsoil depth fall below the critical level?

EP2.5 Input an integer, I, in the range - 32768 [image] I [image] 32767 and display its digits separately.

Consider a familiar problem like EP2. l . Someone who lacks programming experience might well recognize that ten inputs can form a sequence x1, x2, . . ., x10 and that they can be compared with each other. The direct implementation of these ideas is expressed in such statements as:

Read(x1)

Read(x2)

.

.

.

Read(x10)

if x1 > x2

then if x1 > x3

then ...

.

.

.

This process, of course, is terribly inefficient, particularly if there are 10,000 values instead of 10. Someone with even a little programming experience will recognize that this problem can be solved simply by retaining the largest value so far. (Nearly anyone can solve this problem over a phone, but only programmers seem to recognize how they do it. It is apparently not a trivial task to make the connection between the natural selection process and the use of a single memory location.) A programmer's solution is something like this:

program MaxTen                                  {O(1)

Read(Large)

for i = 2 to 10 do

Read(x)

if x > Large

then Large    x

endif

next i

Write(Large)

end {MaxTen

This solution for EP2.1 can be generalized easily to find the maximal value of a sequence of any fixed length. Furthermore, modifications to MaxTen will allow the sequence length to be an input variable, to find out (with a counter) which entry actually determines the final value of the variable Large, how many entries are equal to the final value of Large (with another counter), how many times Large changes value (with a third counter), and so on.

Note that, by isolating the core of the problem, which is finding the largest-so-far value, other questions can be answered as applications of the solution. Had the problem been stated initially as a request for all of the modifications mentioned above, it might have seemed overwhelming. Much of data structures as a subject is concerned with extracting and focusing upon central ideas that can be modified and combined in order to solve specific applications.

Some problems may seem to be simply variations of EP2.1-EP2.5 but actually call for a data structure that is more complex than the simple variable. For example:

EP2.6 Input 10 numbers and display the second-largest value in the set of entry values.

EP2.7 Input 10 numbers and display them in reverse order of entry.

EP2.8 Input 10 numbers and display the negative entries, followed by the others.

EP2.9 Input 10 numbers and display them in nonincreasing order of their magnitude.

EP2.10 Input 10 numbers and count how many of them are larger than their mean.

This list could be much longer, and EP2.6 could be removed from it by clever programming, but problems like EP2.9 permeate a large portion, perhaps most, applications of programming. Example problem EP2.9 is important because data frequently have to be sorted in convenient order. (Can you imagine using a telephone directory where names are arranged according to the date on which a phone was first connected to the system?) Most government policy decisions, such as When can a drug be safely released for sale? and Which vote will mollify the most constituents? are based at some level on statistical calculations like EP2.10 and the calculation of standard deviation (see Part II, section A.4).

Problems EP2.6-EP2. 10 can be solved if all of the data are retained in memory after input, so that more than one pass through the data is possible. The collection as a whole can be identified and retained as a group of storage cells, but it is helpful to give a unique name to each memory cell in the group. A specific cell name is necessary if individual access to the cells is to be provided, but the collection of names might need to be as large as the data set. The solution to this puzzle is built into most high-level languages and is presumed to be familiar to the reader--it is the subscripted variable, or array.

Note: An array with a single subscript is also called a list (usually indicating that it is used as one). It is not, however, a linked list in the sense defined in Chapter 3.

An array has an identifier for an entire group of cells, and (in the simple one-dimensional case) one other variable, the index, is used to select a particular cell within the group. With the usual notation for arrays, one solution to EP2.7 in SUE, with example input and output and a display of the final values of the array x, is:

2.2 The Array as a Structure

Among the reasons for studying arrays specifically as data structures, rather than just accepting them as gifts from a compiler, are these:

In some languages, only a limited number of indices is allowed. (Limits of 2 or 7 have been used.) To get around index limitations, a programmer may construct a high-dimensional array within an array of fewer indices. To do so requires a fundamental understanding of how arrays are structured.

It is fairly common to develop high-level, or very high-level, languages for special purposes, such as a "query language" for asking questions about the contents of a data base. Some of the structuring can be patterned after the details of array structure.

Controlling a microprocessor, designing a compiler for a high-level language, tuning the performance of a program, or controlling the interface between a computer and the outside world, often require an understanding of assembler (and machine) languages. Assembler languages leave the formation of arrays and any controls on their misuse to the programmer. Understanding how to use the array structure provided by high-level languages is often not sufficient for assembly language programming.

Data structures are often constructed by a programmer as superstructures imposed on arrays, although the same structures can also be implemented without arrays. Understanding these structures requires that they be distinguished from the array structure itself.

The array is a familiar programming tool. Abstraction of its properties and the operations needed to manage it provide a model that will be applied to unfamiliar structures in this text.

With these motivations in mind, we turn to study what makes an array with a single index the creature that it is.

There is usually an advantage in thinking of an array as a geometrical sequence of adjacent cells, as depicted in Figure 2.1. For ease of discussion, such an arrangement will be called an LSC, for Linear Sequence of Cells.

Figure 2.1

Actually, from the view of the array user, it does not matter where the memory cells of an array are located, given that the indices provide random access to them: the user must be able to access the cells in any order merely by specifying the name of the array and the correct index. That the individual cells may be jumbled or scattered in memory does not matter so long as the compiler is provided with a map of where the cells are and uses the index as a pointer to the correct one. Of course, cells are often arranged in storage as an LSC; but, on a magnetic drum or disk, for example, faster access may be provided by a different arrangement.

A map from an LSC to physical storage is usually provided by an operating system, which manages fundamental system resources. With such a map in hand, a compiler-writer can treat an array as an LSC, whether or not that is the geometry of the physical storage cells involved. The LSC becomes a logical abstraction for data-handling: a data structure.

When a multidimensional array is declared and used in an HLL, a compiler provides maps and operations that are invoked by statements in the HLL. The maps are needed to convert an array reference to a storage address. For example, suppose that an array is declared as a[0 . . 7, - 1 . . 2,2 . . 4] and managed by a compiler as an LSC occupying memory cells 27, 28, . . ., 122. Then a[5,0,3] is somewhere specific in the sequence of cells, and the collection of such somewheres is a required map. Such maps are also used when programming in assembly language and other situations, and they are the subject of section 2.3.

Maps from one structure to another are invoked by operations described by program statements. In the case of arrays and LSCs, the operations are relatively simple, but they are general enough to apply to the declaration and use of any array that is legitimate in the language being translated. They are even more general than that: the operations CREATE, STORE, and RETRIEVE are needed whenever arrays are used. Their implementation differs from one application to another and is intimately tied to storage maps. The operations essential for array management are derived below as the kernel of a set of general management features that apply to other data structures as well: activation, viability, assignment, retrieval, and location.

Activation

Arrays do not spring into existence simply because they are mentioned (although BASIC allows them to do so to a limited extent). Storage must be allocated for them, and in most languages the amount allocated never changes. (SNOBOL provides an exception.) Array allocation is static--once done, it cannot be changed by the program. Activation of those arrays that are supported by a language is done, for example, with a DIM statement in BASIC and a VAR declaration in Pascal. These creation statements also provide information needed to access individual cells of the array--given the location of a[0], where is a[3]? Sometimes, activation must be programmed because the required array features are missing (more indices in BASIC; any array in an assembler language).

The general activation operation is called CREATE, and so is the analogous operation for other data structures. (The first example in the book of a CREATE operation that is not simply a language declaration is based on arrays and is found in section 2.5.)

Viability

Activation of an array does not necessarily imply that it has been initialized to a standard value (although it may be in some languages). Strict control of operations can be provided by making retrieval from an uninitialized memory cell illegal. Usually, programs are allowed to retrieve unchecked garbage instead, because the overhead required to make such checks is considered to be too high.

Assignment

The ability to copy information into the cell of index i of array a is commonly provided in at least two forms:

a[i]    x

Read(a[i])

The general operation will be written STORE(a,i,x) or simply STORE(i,x). Hence STORE(Grade,3,85) has the effect of

Grade[3]    85.

Retrieval

The ability to copy information from the cell of index i is provided in at least two forms:

x    a[i]

Write(a[i])

The general operation will be written RETRIEVE(a,i) or simply RETRIEVE(i). Hence RETRIEVE(Grade,3) immediately after the STORE example above will retrieve the value 85.

Location

Retrieval and assignment operations depend upon the ability to locate a specified cell within the array structure. The array index provides direct access to the cell because it determines an absolute memory address. How it does so will be explained in section 2.3. The location of a cell within other structures is not always so direct.

In summary:

An instance of the structure ARRAY is operated on by CREATE, STORE, and RETRIEVE.

Any cell in an array can be accessed by STORE and RETRIEVE through the use of its index.

An array is static in the sense that no cells can be inserted or deleted from it by an operation.

There is usually no check on validity of the values in an array.

Finally, note that CREATE, STORE, and RETRIEVE are implicit rather than explicit in high-level languages, but their service is provided.

2.3 Array Addresses

STORE and RETRIEVE: it is the mapping of an index to a unique memory location for each possible in-bounds index. This map is needed to build multidimensional arrays from those with fewer dimensions. Even in the one-dimensional case a map is required, because the address of an array cell and its index are not identical--they are just in one-to-one correspondence. In many languages, array indices may lie in a range that does not begin with zero or one: a[L . . U] is to mean that storage is allocated for a[L],a[L + 1], . . .,a[U]. There are U - L + 1 legal (in-bounds) indices in this case.

Note: In some languages, such as Pascal, memory cells in an array may be specified by any values that form an ordinal sequence. For example, a[Red . . Violet] may mean that Red, Orange, Yellow, Green, Blue, and Violet follow in order, as indices. In effect, they represent indices 0,1,2,3,4, and 5. An auxiliary map is used by the compiler to preprocess the names of colors in order to bring about this effect, but the essential coorespondence is still the map of [0 . . 5] onto six memory addresses.

Suppose that the absolute address of a[L] is [image] , and in general, the address of a[L + i] is [image] + i. This linear storage scheme conforms to the standard geometrical picture of an array. With it, the map from index i to its corresponding address is simply:

address       + i - L

(The address [image] will be treated as a variable, Alpha, so that we may discuss its computation.)

However, this map itself provides no check for an out-of-bounds address. To make such a check requires a procedure or perhaps some variation of the following:

procedure Main

.

.

.

err    FALSE                    function Map(i)

address    Map(i)                 if i < L OR i > U

if err then ...                     then err    TRUE

.                                   endif

.                                 return Alpha + i + L

.                                 end {Map

end {Main

With the use of this function, if the declaration is a[1 . . 10] and the storage is linear, then a[7] has the address Alpha + 6, and a[- 1] is out of bounds. If the declaration is a[- 5 . . 4], then (the same) 10 memory cells are allocated, but a[7] is out of bounds, and a[- 1] has the address:

Alpha + (-1) - (-5) = Alpha + 4

If two indices are allowed, then the structure may be thought of as a table, arranged in rows and columns, such as in Figure 2.2.

Figure 2.2

Each pair of indices specifies a row and a column. Usually a[2,3] is thought of as the cross-hatched square in Figure 2.2--row 2, column 3. However, as long as a program is consistent about storing and retrieving from a table, the choice makes no difference at the program level. The picture is merely an aid to the programmer. In some languages, however, a table can be manipulated as a unit, even for input and output. Then it matters whether or not Read(a) is expecting the input in the order row1, row2, . . . or in the order column1, column2, . . . . That is still a programming issue. The structural issue is how to arrange the data in successive addresses Alpha, Alpha + 1, . . . so that the two indices specify a unique memory cell. The two common arrangements are shown in Figure 2.3.

Figure 2.3

Note: Most versions of BASIC use the row-major scheme, and FORTRAN uses colunm-major storage.

When a table is to be arranged in a linear pattern as is Figure 2.3 (or in a one-dimensional array), the map from an index-pair to an address (or to a single index) is designed to work for only one of these schemes. Figure 2.4 depicts the two common storage arrangements.

Figure 2.4

Consider the declaration a[1 . . 3,1 . . 4], and assume that a is stored in row-major order. To get to a[2,3], for example, it is necessary to count past row 1 and then choose the third item in row 2. Since there are four items in row 1, the address of a[2,3] is:

Alpha + 4 + 2 = Alpha + 4(i - 1) + (j - 1)

= Alpha + 6

Note that the factor of 4 comes from the number of columns in each row.

More generally, for a[1 . . U1,1 . . U2]. the row-major address of a[i,j] would be:

Alpha + U2(i - 1) +  (j - 1)

Generalizing further, for a[L1 . . U1,L2 . . U2], the address of a[i,j] would be:

Alpha + (U2 - L2 + 1)(i - L1) + (j - L2)

Similar calculations for the column-major scheme are left to the exercises.

The general row-major storage scheme tor a table is pictured in Figure 2.5.

Figure 2.5

In order to picture the storage layout for three indices i1, i2, and i3 in row-major (or in column-major order), it is necessary to generalize the meaning of "row-major" (or of "column-major"). Row-major storage of an array is normally taken to mean the following if the array is stored linearly:

As addresses increase from the initial value Alpha toward the last address, the first index changes most slowly, and the last index changes most rapidly.

For the declaration: a[1 . . U1,1 . . U2,1 . . U3], the indices change like this:

a[1,1,1],a[1,1,2], . . .,a[1,1,U3],a[1,2,1], . . .,a[1,U2U3],a[2,1,1], . . .

As a consequence, the last two indices define a table, one selected by the first index, as shown in Figure 2.6.

Figure 2.6

The front face F is simply the table defined by i2, and i3 when i1, is fixed at 1.

For the most general declaration of a three-dimensional array a, lower bounds Li replace 1, and the address calculation for a[i1,i2,i3] involves counting past (i1, - L1) tables and then locating a spot within table (i, - L1). There are (U2 - L2 + 1)(U3 - L3 + 1) cells in each table that is passed, hence:

address    Alpha + (U2 -L2 + 1)(U3 - L3 + 1)(i1 - L1) +

(U3 - L3 + 1)(i2 - L2) + (i3 - L3)

For the n-dimensional case, let sk = Uk - Lk + 1, the number of indices in the range of ik. Then:

address = Alpha + s2  s3          Sn(i1 - L1) + S3          Sn(i2 - L2) +          + (in - Ln)

Suppose, for example, that we wish to find the address of a[0,0,0,0] in an array declared as: a[ - 4 . . 3,0 . . 4, - 5 . . 5, - 2 . . 6]. Then S 1 = 8, S 2 = 5, S 3 = 11, and s4 = 9. The resulting address is:

Alpha + 5 X 11 X 9(4) + 11 X 9(0) + 9(5) + 2

= Alpha + 1980 + 0 + 45 + 2

= Alpha + 2027

Section 2.6 contains an application of the unfolding of many dimensions into one.

Exercises E2.1-E2.5 are appropriate at this point.

2.4 A Sparse and Static Table

Jon Bentley's column "Programming Pearls" in the May 1984 ACM Communications contains an array storage scheme that differs from the usual one for a good reason--it saves a valuable resource. Saved is the storage space that can be wasted when a table is sparse--meaning that most of the entries are the same, usually zero. From the standpoint of information retrieval, only two things are necessary: storing the table entries that are not zero (nonstandard) and detecting a zero entry in some way. The structure developed in this section that does these two things follows Bentley's solution to storing a table in less space than is required by the maps of section 2.3.

Suppose that a map is contained in a grid of NR by NC points that include some fixed number, NI, of items of interest--it is static. The items of interest might be the number of finches in the bird sanctuaries on a state grid, or nuclear reactors on a world grid, or sections containing wheat-rust on a satellite image. There are NR X NC grid points and NI may be much less than that. (In Bentley's problem, NI was 2000, NR was 150, NC was 200, and hence NR X NC was 30,000.) By comparison, there are about 200,000 controlled points on a television screen, and over half a million in the input-output table that describes the United States economy. The resolution provided by the human eye would allow even more points of interest on a large road map of Nevada--but they aren't there. Clearly, storage of all points in a table can be very wasteful of space, and the space may not even be available. In examples of this sort, the data that are likely to be provided for the location of a value are two coordinates--row and column.

The core of this problem is twofold:

1. Many fewer than NR X NC items occupy that much storage in the standard row-major (or column-major) map of an array, and the number of items does not change.

2. The row-and-column format is natural to the problem and provided as the input data for the search for a value in the structure.

The core of a solution is to provide maps from row-and-column data into a compact memory space so that it appears to a user to be a standard table. This approach saves the storage resource and preserves ease-of-use. The user's time, effort, and comfort are themselves important resources.

One scheme for packing the nonzero entries of a table T (illustrated in Figure 2.7) is to store only the nonzero entries and associate them with their row index. Figure 2.7 can be reduced to Figure 2.8. In Figure 2.9 the entries and their rows are stacked together in arrays Value and Row, and a map FirstInCol[1 . . NC + 1] provides the index in Row of the first nonzero row in each column of T. The extra column, 6, and its value NI + 1 = 9 is added to provide a sentinel for Retrieve. The resulting structure is SST (for Sparse and Static Table).

Figure 2.7

Figure 2.8

Figure 2.9

If it is assumed that the values and the indices require one word of storage each, conventional table storage requires NR X NC words of storage. The conventional RETRIEVE operation is O(1), and initialization of the structure requires NI STORE operations. A crucial point to be noticed is that NI can be changed with O(1) STORE operations in the conventional scheme.

Part of the price of the compactness of SST is that assignment of a zero to a cell in the structure or insertion of a nonzero value to a cell not represented in the structure would amount to deletion or insertion of a cell. SST just does not provide for this flexibility.

With the same assumptions, SST requires NI words for Value, NI words for Row, and NC + 1 words for FirstlnCol, a total of 2 X NI + NC + 1 words instead of NR X NC words. The RETRIEVE operation becomes:

function Retrieve(i,j)                             {O(NI)

for k    FirstInCol[j] to FirstInCol[j+1] - 1 do

if Row[k] = i

then return Value[k]

endif

next k

return 0

end {Retrieve

This is an O(NI) operation because most of the entries could be in a single column and the for-loop would then iterate up to NI times. If table entries are randomly located, the expected number of entries to be searched is NI/NC. (See section 1.3.2.) The search for the row index could be speeded up by using a binary search as discussed in section 8.3 in the sense of making the search for the row an O(In NI) operation. This apparent speedup is misleading, however, because the overhead required to set up and carry out the search is more than would be needed in the linear search above for the small number of row indices expected per column.

This back-of-the-envelope analysis shows that SST may save a great deal of storage space at the cost of some processing time. It is probably not suitable unless the table is sparse with the entries shared fairly evenly by the columns. The trade of processing time for storage is not uncommon, and sometimes the search for either compactness or efficiency will improve both because it promotes thoughtful analysis.

An HLL program that includes SST would create the arrays Value, Row, and FirstInCol simply by declaration. That does not perform CREATE for SST, however, because SST is a data structure that is shaped by its values and reasonably includes those pairs of indices [i,j] that denote nonzero entries. Hence CREATE should include initialization of the supporting arrays.

Suppose that the information for SST is available as a sequence of triplets (i,j,v) representing row i, column j, and value v, and is sorted by column and by row within each column. Then CREATE is both supporting-array declarations and the assignment of values to them.

The initialization procedure illustrates a theme that occurs for other data structures: the care needed to deal with special cases. In the general case, when a row i, column j, and value v are retrieved from an input file, a suitable procedure response is, for the right Dexi:

S1: Row[Dexi]    i

Value[Dexi]    v

Dexi    Dexi + 1

However, column j of the input triplet may be the one that follows the column of the previous entry triplet, in which case the above is preceded by (for the right Dexj):

S2: FirstInCol[Dexj]    Dexi

Dexj    Dexj + 1

What, however, must be done about missing columns? Examination of Retrieve reveals that if S2 is repeated until j = Dexj, the interplay between the two procedures will generally work. In particular, though, what if the first few or the last few columns are empty? Initialization of Dexj to 0 takes care of an empty leading column and a loop similar to S2 takes care of an empty trailing column.

Finally, the sentinel column NC + 1 can be initialized by simply treating it as the empty last column. The result is:

procedure CreateSST   {O(NI + NC)

Dexi    1

Dexj    0

for k    1 to NI do

Read(i,j,v)

while j > Dexj do  {skip empty columns

Dexj    Dexj + 1

FirstInCol[Dexj]    Dexi

endwhile

Row[Dexi]    i

Value[Dexi]    v

Dexi    Dexi + 1

next k

while Dexj    NC do    {skip empty columns

Dexj    Dexj + 1

FirstInCol[Dexj]    Dexi

endwhile

FirstInCol[NC + 1]    NI + 1

end {CreateSST

(The indication O(NI + NC) is one way to point out that this algorithm is either O(NI) or O(NC), whichever is larger.)

The interaction of a data structure and its implementation are illustrated with SST by the fact that if the entry triplets were available sorted by row, then column, the roles of rows and columns would switch: Row and FirstInCol would be replaced by Col and FirstInRow. The structure of SST would not change in any meaningful sense by storing the entries one row at a time--SST is an idea imposed on the underlying arrays by the programmer.

For an application where the number of entries in a table makes it sparse, but that number is not static, there are structures more suitable than SST. They include forms of the sparse matrix of sections 6.4 and 6.5 and Part II, section N, as well as the adjacency list applied in Chapter 10. (But they are less suitable than SST for a static table that is simply to be searched often once it is created.)

The discussion of SST given in this section assumes that the values in the structure are the values of interest. Such a structure is endogenous. It is also possible for the entries in Value to be pointers--indirect references to data external to the structure itself. Such a structure is exogenous. The data structure and the operations on it are little affected by such a change, but the leverage available is large. The data accessed through Value may be bulky and sorted on disk or some other external storage medium in any order. When used as an exogenous structure. SST becomes part of a map or directory to a file of records.

Finally, note that SST is an example of information hiding--the details of the implementation are not important at a user level. A user (program) that acquires a value in row i, column j with Retrieve(i,j) is indifferent to the choice between SST and the standard two-dimensional table format referenced by a particular form of Retrieve. The trade-off between compactness and efficiency and the details of implementation are isolated from functional behavior. At the user level, it is the functional behavior of a structure that needs to be considered in order to use the structure as a tool. The details of its design are distractions at that level. In telegraphic form: SST STILL AN ARRAY.

Exercise E2.6, problem P2.1, and program PG2.1 are appropriate at this point.

2.5 Hash Tables (optional)

It is not always the case that array indices are natural identifiers for items that are to be stored, accessed directly one at a time, and retrieved. Consider the following items:

beef   bellpepper  blackpepper  dillweed

onion  potato      olive        salt

cumin  carrot      mushroom     tomatopaste

These items could be maintained in lexicographic order, in which case potato would be item number 10 after all items were placed in the list. With standard algorithms, the RETRIEVE operation would be O(In n) and the STORE operation O(n) if order were maintained as the items were entered. While it is true that STORE and RETRIEVE are O(1) operations for arrays, that is only so if the indices are known and the value in the target of a STORE can be discarded. Without a complete set in hand it cannot be known that potato has index 10 in the sorted list of items.

Because an index integer is not known on the entry of one of the items, it would be helpful if the item itself could be used as a key to index the cell where it will be stored. If, for example, the data-space (the set of all possible data values) were the set of all possible recipe ingredients, the required array would be large, and most of it would be wasted as storage for any one recipe.

A solution would be to convert the keys (the list items, here) into unique integers and use them as array indices. A function that does so is called a hash function, the conversion process is called hashing, and the storage structure is called a hash table or scatter-storage. One such function (not commonly used) is to simply associate integers 1-26 with letters A-Z and add the integers corresponding to the letters of an item together. With this function, HF1, the hashed indices are:

Item        HF1(Item)  Item         HF1(Item)

beef            18     carrot           75

onion           67     salt             52

cumin           60     blackpepper     105

dillweed        74     olive            63

bellpepper     107     tomatopaste     145

potato          87     mushroom        122

Here is an apparent solution, although an array of 145 cells is needed to store these 12 items. For an arbitrary recipe, 20 or 25 items is quite a lot. With this function, there are 2710 possible keys of no more than 10 characters, and the maximum value of HF1 is only 260. The number of possible ingredients, named in English, is several thousand, many of which will have the same HF1 values.

Schemes to restrict the range of the function, like using HF1 (Item) MOD 25, creates the same problem that arises when just one more item is added to this list: broth. HF1(broth) = 63 = HF1(olive). This is called a collision, and collisions are inevitable if a hash table is to be reasonably space-efficient.

Computer scientists have spent much time exploring how to deal with collisions and juggle the space and time efficiency of hash tables because the utility of hashing is not confined to storing recipe ingredients. Hashing should be considered when any large collection of items is to be stored and retrieved efficiently from a small space.

A discussion of hash tables, confined to the use of array support for them, is to be found in Part II, section  B. A supplement to the discussion there is found in Part II, section  L, and may be read after other support structures have been introduced.

2.6 Unfolding Indices    (optional)

Many things have a natural dimensionality; they are most easily associated with one or two or some number of indices. However, they do not have to be modeled according to their natural dimensionality. Two-dimensional slices can be used to describe three-dimensional objects. Cross-sections of the atmosphere in a weather model or of a person in a CAT-scan can be used to store essentially three-dimensional information. Sometimes three dimensions can be usefully projected into a single pair of dimensions, as in a contour map.

In the other direction, an indexed set of essentially planar structures can be managed with three indices, forming a three-dimensional data structure. Examples include a sequence of board configurations tor almost any board game, an item-count table for each of a family of warehouses. or a set of crossword puzzles.

The success of these dimension changes, however, lies in the fact that they are reversible--it is possible to devise a picture of natural dimension from the data format. On balance, if a programmer thinks most easily and clearly about a problem when it is portrayed as three-dimensional (or two-dimensional or five-dimensional), then it should be programmed in its natural form. This is not simply information-hiding, but information-packaging.

In this section, a game is used as the model of a system that has a natural dimensionality. Games, it should be noted, serve as models for many things because they can be designed to eliminate extraneous concerns.

Suppose that a program is written to monitor (or to play) three-dimensional tic-tac-toe. The rules of the game and its interaction with users are important parts of the program design, but so is the management of a three-dimensional array. Assume that the language L in which the program is written does not allow three indices, and so the board array, B, needs to be unfolded into a one-dimensional array, Stretch. Suppose that the natural three-dimensional array would be declared as

B[1 . . 3,1 . . 3,1 . . 3],

in a language which allowed such a thing. The array B can be used to think about the design of the program, but the program will actually be implemented with Stretch[1 . . 27].

One way to organize the unfolding of B into Stretch is to develop procedures Create, Store, and Retrieve, which handle the index map from three indices to one, and then rely on the language L for the versions of the CREATE, STORE, and RETRIEVE operations that manage the array Stretch. Create, Store, and Retrieve can then be called upon by the rest of the program, much as though the language L did implement three-dimensional arrays. A clean treatment of that management allows the programmer to focus on aspects of the program other than index calculations.

In most languages, Create must be handled by the programmer, in the sense that Stretch must be declared with sufficient storage to hold B. The array Stretch probably should not be declared with any extra cells, simply as a debugging safeguard.

If Retrieve is implemented as a function with three parameters, then

x    y -- Retrieve[i,j,k]  or  Write(Retrieve[i,j,k])

may be used naturally in place of:

x    y B[i,j,k]  or  Write(B[i,j,k])

A simple version of Retrieve that does no out-of-bounds checks of i, j, and k is:

function Retrieve(i,j,k)                     {O(1)

Index    9 X (i -1) + 3 X (j-1) + (k-1) + 1

Retrieve    Stretch[Index]

end {Retrieve

This can be easily generalized with the help of section 2.5, but a caution is in order: there are illegal combinations of i, j, and k that are not out of bounds in Stretch itself. An example is [1,5,5], which this version of Retrieve maps blithely into Stretch[17], but which should not exist in the array B at all. This problem can be avoided if the bounds for B are checked individually as they are unfolded, in which case the bounds for Stretch will not be violated either. Furthermore, [1,1,1] corresponds to Index = 1 and not to the address Alpha of [1,3,2]. In terms of absolute addresses:

Index - 1    9(i - 1) + 3(j - 1) + (k - 1)

An equally stripped version of Store is:

procedure Store(Value,i,j,k)     {O(1)

Index    9 X (i - 1) + 3 X (j - 1) + (k - 1) + 1

Stretch[Index]    Value

end {Store

Store can be used in situations like this

Store(Value,i,j,k)  and  Read(Value)

Store(Value,i,j,k)

in place of:

B[i,j,k]    Value   and   Read(B[i,j,k])

Clearly, the bodies of the stripped versions of Store and Retrieve could be simply incorporated as needed in the program, but when index bounds on i, j, and k are included, separating them as procedures removes a lot of clutter and simplifies debugging. Note that the array B exists only in the programmer's mind and perhaps in the documentation.

Note: In this example, it was assumed that Store and Retrieve were to operate on only one array, Stretch. They can be modified by including an array name in the argument list in order to generalize them to deal with an arbitrary one. In that case, the bounds may also need to be generalized. They may be passed as an array of bounds or be stored in an array that is global and hence accessible to both procedures. Such generalization should be used with common sense, when it enhances more than it obscures.

Problems P2.2-P2.6, and programs PG2.2-PG2.5 are appropriate at this point.

Summary

Even a simple variable, supported by memory cells, is a structure. As a structure, the simple variable is not adequate to handle some common computing problems. For those, the array is needed.

The structure ARRAY provides some contrast with the structures discussed in later chapters. It is static in that it does not grow and shrink with the insertion or removal of information. An array is created (generally by including a variable declaration or dimensioning statement in a program). It can be accessed by two operations, STORE and RETRIEVE, which allow direct access to any cell of the array with the use of indices. These operations really characterize an array.

Array access via the indices requires a map that can be put in general form. This map is explored enough so that arrays can be constructed from arrays of fewer dimensions.

A specialized form of array, the SST (for Sparse and Static Table), illustrates four common features of data structures:

      It is explicitly a structure built out of supporting structures by programming.

      It is not a language feature.

      It is an example of information-packaging.

      It is used for a specific application as a conscious choice between competing structures.

A large collection of keys may be mapped into a few indices that then provide many of the benefits of array access. Such a map into a hash table is frequently used in applications. The hash tables introduced in section 2.5 are the subject of Part II, sections B and L .

In Chapters 6 and 10, arrays are supported with structures quite different from the linear string of memory cells assumed in this chapter, and the number of competing choices for such applications is increased.

Further Reading

Further Reading

The "Programming Pearls" column written by Jon Bentley for ACM Communications is a delight. It is nearly entirely accessible to anyone in the throes of a data structures course and surely totally accessible to anyone who has completed one. A common theme in "Pearls" is the wise choice of data structures, frequently based on problems that confronted professionals while they were practicing the craft of programming. (The first "Pearls" column appeared in the August 1983 issue.)

Exercises

Exercises immediate applications of the text material

E2.1     How many memory cells are required by A[0 . . 5]?

     by B[-1 . . 4, 5 . . 10]? by C[-5 . . -2, 2 . . 5, -2 . . 2]?

E2.2     Assume that the arrays A, B, and C of E2.1 are stored linearly in row-major order, beginning in addresses   [image],   [image], and   [image], respectively. What are the indices for the cells with addresses   [image]  + 2,   [image]  + 12,   [image]  + 17,   [image]  + 7,   [image]  + 16,   [image]  + 37?

E2.3     What is the address of A[0,0,0] in an array stored linearly in row-major order beginning in address   [image]  if it is declared as

     A[-5 . . 0,-2 . . 3,-2 . . 0]?

     as A[-1 . . 5,-2 . . 2,0 . . 3]?

E2.4     Suppose that the array B is stored linearly in column-major order, beginning in address   [image].

If B is declared as B[1 . . 3,1 . . 4], then what is the address of B[2,3]?

If B is declared as B[1 . .U1,1 . . U2], then what is the address of B[i,j]?

If B is declared as B[1 . . U1,L2 . . U2], then what is the address of B[i,j]?

If B is declared as B[L1 . . U1,L2 . . U2], then what is the address of B[i,j]?

If B is declared as B[L1 . . U1,L2 . . U2,L3 . . U3], then what is the address of B[i,j,k]?

If B is declared as B[-4 . . 3,0 . . 4,-5 . . 5,-2 . . 6], then what is the address of B[0,0,0,0]?

E2.5     Using sk = Uk - Lk + 1, what is the general form for the address of an n-dimensional array B, stored linearly in column-major order beginning in address   [image]?

E2.6     Cells of a 12-by-12 table contain either zero or the sum of their row and column indices. The cell value is zero except when both indices are prime: 2,3,5,7,11. Diagram the SST structure (of section 2.4) for this table.

Problems

Problems not immediate, but requiring no major extensions of the text material

P2.1     Redesign SST, Retrieve, and CreateSST with the roles of rows and columns interchanged.

P2.2     Most computing languages do not check to make sure that a memory cell has had a value assigned to it (that it is initialized), before retrieval from it is allowed. Simulate this type of checking for a one-dimensional array A, in the following way: develop (in SUE, perhaps) procedures CreateBF, StoreBF, and RetrieveBF. CreateBF(A,n) will initialize an array BF of n Boolean flags to FALSE, essentially to indicate that no cell in the array A has yet been initialized. StoreBF(A,i,x) will set the flag BF[i] to TRUE and then perform A[i]   [image]  x. RetrieveBF(A,j) will check BF[j], and if A[j] has not been initialized, then it will return an error signal, otherwise it will perform RetrieveBF   [image]  A[j].

     PG2.2 asks for a program that implements this map.

P2.3     In language L, arrays can be indexed only by integers. It is desirable to extend L within the confines of a large program to allow an array, Redletday, to have rows indexed by the letters 'A' through `D', and columns indexed by `V' through `Z'. After invoking CreateL to create an appropriate index map, a programmer should be able to use Store(`B', `X', Value) in place of Redletday[3,2] Value. Design the routines CreateL, StoreL, and RetrieveL in SUE.

     PG2.3 asks for a program that implements this map.

P2.4     Suppose that language L allows only one-dimensional arrays. Programmer Z is in the habit of declaring an array Dump[1 . . 64], and unfolding multidimensional arrays with index ranges of the form 1 . . 2, or 1 . . 4, or 1 . . 8, and so forth, into this array in every possible way. The number of possibilities when all 64 cells are used is surprisingly small. How many are there?

P2.5     Referring to P2.4, Programmer Z normally chooses arrays of dimension 1,2,3,4, or 5. Z uses a nondecreasing sequence of maximum indices for the range, but does not necessarily use the entire array Dump. For example, with 3 dimensions, the index ranges might be [1 . . 2,1 . . 2,1 . . 4]. Z has developed general procedures Cfold, Sfold, and Rfold that do the following:

     Cfold(n1,n2,n3,n4,n5) initializes an array, Unfold[1 . . 5], which can be used to map into Dump from the multidimensional array

Origami[1 . . n1,1 . . n2]

     or perhaps Origami[1 . .n1,1 .. n2, 1 . . n3]. The array Origami exists only in Z's mind, of course, since it cannot exist in language L. The number of dimensions in Origami are determined by the number of ni's greater than 1.

     Storage into Origami is carried out by Sfold, and retrieval from it by Rfold. Both use Unfold in order to manage their operations.

     Design Cfold, Sfold, and Rfold.

     PG2.4 explores this map, and PG2.5 asks for a program that implements the routines Cfold, Sfold, and Rfold.

P2.6     Design a procedure that copies array A[1 . . 5,0 . . 5], stored in row-major order, into array B[1 . . 10,1 . . 3], stored in column-major order. (Just such a transformation is required to transfer array A as created by a Pascal program into array B to be used by a FORTRAN program.)

Programs

Programs for trying it out yourself. A program, when written for an interactive system, should display instructions that explain to the user what it does and how to use it. It should provide a graceful way for the user to quit.

PG2.1     Write a program that accepts no more than 100 (i,j,v) triplets, counts them, creates an appropriate SST structure for storing them, and displays the resulting structure. The number of rows NR and the number of columns NC are input data. Test your program with the example data, that data less column 1, less column 2, and less column 5.

PG2.2     The solutions to problem P2.2 are procedures that check to make sure that the individual cells of an array A have been initialized before retrieval from them is allowed. Write a program to test procedures of this type. The program should allow a user to make entries into, and retrievals from, an array A[-5 . . 10] in any sequence. If retrieval is attempted from A[i], and there has been no store into A[i], then a message to the user should be generated.

     Comment: In a language such as Pascal, the Boolean arrays must be declared in advance of the call to CreateBF and may as well be tailored to their use. A declaration BF[-5 . . 10] would be appropriate here.

PG2.3     Write a program that emulates the addition of alphabetic indices to the language L of P2.3, with CreateL, StoreL, and RetrieveL. The user should be able to provide any sequence of letter-pairs and values in order to effect storage into Redletday, and to request the correct value of an entry in Redletday by specifying a pair of letters. Out-of-bounds pairs should prompt a message to that effect.

     Please note that CreateL, StoreL, and RetrieveL assume INTEGER indices only.

PG2.4     Write a program that will accept one to five integers n1, . . . ,n5 and display the map of Origami into the array Dump as it is developed in problem P2.5. For example, if 2, 4, 0, 0, 0 are entered, then one form of the map is:

Origami  Dump

indices  index

1   1      1

    2      2

    3      3

    4      4

2   1      5

    2      6

    3      7

    4      8

PG2.5     Emulate the implementation of the Cfold, Sfold, and Rfold discussed in P2.5 with a program that will accept n1, . . . ,n5, which in turn define the virtual array Origami, and then any sequence of indices (and values). The program should make out-of-bounds checks and allow apparent storage into and retrieval from Origami, while actually maintaining only Dump as an array.

Projects

Projects for fun; for serious students

PJ2.1     An attendant at a parking lot must keep track of the position of cars as they are parked in any available spot as well as their license numbers. When customers leave, they request their cars by license number, and one more parking slot becomes free. Help the attendant by writing a program that admits up to 16 cars into this (very small) lot and responds to an arbitrary sequence of requests for parking space, car retrieval, and display of the current status of the parking spaces. Arrays are to be used as data structures, and the design of the lot is your lot.

Go to Chapter 3            Return to Table of Contents


Data Structures: CHAPTER 3: LIST PROCESSING

 Return to Table of Contents

CHAPTER 3: LIST PROCESSING

One common way to preserve a sequence of items is to make a list of them. The shopping list in a kitchen, for example, grows as family members notice the need for additional items. Items may be added at the bottom, but such a list differs from a simple sequence because of the other operations applied to it. Occasionally an item will be scratched off the list when a supply of it is rediscovered, and certainly items will be removed at each stop of a shopping tour. A shopping tour may (or may not) reduce the list to nothing. A different list, perhaps developed as a party is planned, may be appended to the current list. Duplicate items may be noted and removed from one of the lists. When an item is to be added, the list will be searched to see if it is already there; but the list is not likely to be alphabetized in order to make the search easier. Occasionally a family member will update an entry by, for example, specifying a particular breakfast cereal in place of a general entry, "cereal." The list may be partitioned at shopping time into lists appropriate for different stops on a shopping tour. These list operations appear in a large variety of other applications as well.

The list is a commonly used data structure and can certainly be incorporated into programming. Lists are so general, common, and useful that even restricted forms of lists are major topics and are the basis for the two chapters that follow this one.

3.1 The List as a Structure

A data structure is many things, including an abstraction. In particular, it is a combination of allocated storage with a set of operations performed on it. The feature that best characterizes structure LIST is that its cells are linked together. Some idea of the nature of structure LIST can be gained by contrasting it with data structure ARRAY. The contrast is related to operations performed on instances of both structures.

The operations applied to lists differ significantly from those applied to arrays.

Activation

The list that forms a class roll usually exists, even if there are no names on it, which is true before registration or if everyone drops the course. It defines the roll, empty or not. A list is normally created empty, with no storage occupied by list entries. It may be necessary to allocate space for the list to grow into, just as a piece of paper is reserved for a shopping list; but it may also be possible to avoid the allocation of space before it is needed. In either case, the list itself is initially null (empty), as opposed to an array that is not yet initialized but consists of cells that contain some value.

Viability

A list is not only created null, it can become null through the removal of its entries. A list of chores, for example, is supposed to become null eventually. The attempt to remove an entry from a null list is usually treated as an error in a removal operation, and so the emptiness of a list is an often-checked property of it. (An alternate view is that removal of a specified item from a null list cannot fail--the item is no longer there after the attempt.)

Attachment

A list grows when an entry is attached to it. In many implementations, it is not simply a value that is attached, but rather a package containing at least one, but perhaps many, values. A common example is an entry in a list of personnel records. Placement of an entry into a sequence of entries in such a list is quite distinct from the assignment of values to the entry itself. No general restriction forces attachment to be made only to the "tail" of a list. Carefully chosen insertion positions can be used to maintain a list in some order.

Assignment

The assignment of values to entries, before or after the entries are attached to a list, is the only form of assignment for lists. Assignment does not really affect the list structure itself, which takes its form from its entries and their relationships, not from their values. For example, a revision of the job description in an entry of a list of available jobs does not change the number of jobs and probably will not change their order in the sequence.

Partitioning

It is convenient to be able to divide a list into sublists. For example, the list of tasks required to construct a building is often divided into separate lists for subcontractors. The essential partition divides a list into just two sublists. A list maintained in some order needs to be partitioned so that a new entry can be inserted at a specified position. An entry can be removed by partitioning a list into three pieces and rejoining a pair of them. (There are also analogs of a "scratch" operation in list processing, in which an entry is marked "inactive,'' but not removed.) Partitioning is usually a transparent--implicit rather than explicit--part of other operations.

Retrieval

The return of a value from a list requires the location of the entry that contains it and then retrieval of the value from the entry itself. The entries may be simply values (as in a shopping list), or the entry may instead be a structure (as in a list of personnel records). Retrieval must be supported by location of the correct entry--a search.

Lists are an extremely general data structure, so general that it seems wise to form a composite view of them by focusing on several particular restrictions and forms of lists. One form of list is the linear linked list of the next section, in which entries contain both sequencing information and values.

The (linear) linked list serves as the general model of a list. Recognize, however, that a singly subscripted variable can be a list of a particular and restricted sort. The cells a[1], a[2], . . ., a[n] of a variable a[1 . . Max] can form a linear list of array cells, or support some other data structure. In particular, the use of a subscripted variable for list support as it is discussed in section 3.5 is quite different from this simple meaning of "list."

A data structure such as a linked list may serve as a directory to data. If all of the data of interest is contained in its nodes, it is endogenous. If its nodes contain addresses of data stored elsewhere, it is exogenous. The auxiliary data may be on a disk or in some other form of secondary memory--or simply in another structure. This book generally deals with endogenous examples, and the reader should recognize that the node "value" may only be a key to more data.

Linked lists are:

1. Dynamic. They can grow and shrink to suit the need of the moment, and usually their occupancy of storage space does too.

2. Blind. It is not necessary to know in advance how large or small they will actually become as they are used.

3. Sparse. The potential data space may be much larger than the set of data in the list at any given time.

When a problem echoes these characteristics, linked lists are likely to be helpful in its solution.

3.2 Linked Lists

A linked list is a set of nodes. Each node contains a value and at least one link, which tells where to find another node. The format of a node may be a structure such as a record or an array. The crucial idea is that each node in a linked list contains information about how to find the next in a sequence of nodes. Immediate access to every node is not necessary, for one node will lead to another, just as stepping stones lead from one side of a creek to another.

In the simplest arrangement, shown in Figure 3.1, the links are linear: the address of Node 1 is stored in variable Head, a link to node 2 is stored in node 1, and so forth. The last link points to a special node, NIL, which is recognizably not the address of any node.

Figure 3.1

The links have a specialized role to play. They are pointers, which point to nodes of some type; and specialized notation is used for them. The notation p [image] (p-Target) is used as the name of the node pointed to by the pointer p. The value of p itself is the memory address of p [image].

Note: The node pointer NIL has no target. If p = NIL, p [image]. Value is an error that will crash a program in most systems.

It can be convenient in some applications for the last link in a list to point to a special node that is the same for all lists in a program--the universal sentinel node u. The choice is a matter of personal taste, but it can affect the simplicity of a procedure. In particular, u acts much like sentinel values commonly used to signal the end of an input stream. Sentinel lists will be used in some examples in the book, in the polynomial lists of section 3.8 for example, but NIL-end lists are the usual choice.

A sequence of nodes linked together in a (sentinel) list may be identified by their values or keys as is the list in Figure 3.2, but there is no intended implication that the nodes contain only the keys.

Figure 3.2

An operation must be provided to acquire storage space for a new node and to identify it. (In some implementations, this may be done by simply updating a variable that serves as an index to an array.) When p is a pointer of type [image] Nodule, then NEW (p) has the effect shown in Figure 3.3 without an assumed initialization of p [image].

Figure 3.3

As long as NEW can acquire storage space for a new node in some manner, nodes can be added to a list. They can also be deleted or updated, and the list can be searched for given values. Nodes can be inserted and deleted at the beginning, at the end, or at any point in the list.

When a node is deleted, the removed node still exists in memory, but not in the list structure. It generally cannot be relocated by a program--it is wasted space. In most languages which support pointer variables, there is an operation DISPOSE(p) that returns the memory occupied by p [image] to available space. In small programs for small applications, DISPOSE may not be needed. It will not be explicitly indicated in all the in-text algorithms where it might be useful for some applications.

Restrictions placed on where insertions or deletions are allowed to occur in the list are used to define the important data structures STACK and QUEUE, which are the subjects of Chapters 4 and 5. Some list operations are common to queues, stacks, and other applications of list processing. They receive a general definition in the sections that follow.

3.3 Common List Operations

Like all data structures, lists must be created; and so some operation CREATE is needed. This operation is dependent on the implementation and the support provided by the language involved. It can simply be the declaration of aList, for example, as a pointer to objects with the type that the aList nodes will have, and usually an initialization:

aList    NIL

However, it is very convenient for some applications to have two pointers to the nodes of the list, usually called Head and Tail, as depicted in Figure 3.4. (Much of what follows can be simplified if the tail pointer is not used.)

Figure 3.4

In this book, we will frequently take advantage of this convenience in the list processing by creating the identifying node of a linear linked list as a record containing a pair of pointers. Thus, the result of CREATE(aList) is:

A nonempty list then takes the form shown in Figure 3.5.

Figure 3.5

This form creates an entity accessed through a single identifier, aList, but it does make names longer. For convenience, we sometimes assume in this book that the body of a procedure operates under the blanket of

with aList do

.

.

.

endwith

within which aList.Head and Head are equivalent, as are aList.Tail and Tail. Alternate approaches are to assume that Head and Tail are global or to pass both variables to a procedure as arguments. The logic of an algorithm is not affected by the choice, although an implementing program can be affected. All three approaches are illustrated in this book.

Because lists are dynamic their management is not simply a matter of updating cells accessed directly. There may not even be cells in a list. The head node of a list is manipulated easily, interior nodes require a search to locate, and the tail node lies between these extremes. One result of the dynamic nature of lists is that they are extremely flexible choices for problem solving. As a consequence of flexibility, there are many variations of structure within lists and of operations to apply to them. Fortunately, there are commonalities (such as procedure LT of section 3.3.2) that reduce the apparent complexities to real convenience.

Some of the sections that follow present several versions of a process where one would do. The intent is to encourage the reader to develop a personal style, rather than to copy one.

3.3.1 The Empty List

As nodes are inserted and deleted, a list grows and shrinks. It is impossible to delete a node from an empty list, so a test for an empty list is a useful operation. Formally, the test is a function, EMPTY, of a list identifier and it returns either TRUE or FALSE. When applied to an empty list (one with no nodes), EMPTY returns the value TRUE. It is easily written as:

{NIL-end list}                       {sentinel list}

function Empty(aList)     {O(1)}     function Empty(aList)

if Head = NIL                        if Head = u

then return TRUE                     then return TRUE

else return FALSE                    else return FALSE

endif                                endif

end {Empty                           end {Empty

3.3.2 List Traversal

A great deal of list processing can be done with a traverse--a systematic visit to every node with one or more processes applied during the visitation.

As an example of a traverse, a list is easily displayed with the help of a routine NodeOut, which writes out the (possibly structured) value of a given node:

procedure DisplayList(aList)             {O(number of nodes)

p    Head

while p    NIL do

NodeOut(p)

p    p    .Link

endwhile

end {DisplayList

The general form of this procedure provides a framework on which a large part of list processing can be draped:

LT

procedure LT(aList)

PROCESS0

while CONDITION do

PROCESS1

p    p    .Link

endwhile

PROCESS2

end {LT

Note: Because lists are such a general structure, there are many variations of any list process and it is well to keep procedure LT in mind when designing algorithms. LT is the basis of most algorithms that make a list-walk (search a list).

DisplayList is an example of LT (List Traverse) described by:

CONDITION : p    NIL

PROCESS0 : p    Head

PROCESS1 : NodeOut(p)

PROCESS2 : NULL

There are useful variations of the while condition used to test for exit, such as:

CONDITION: p    .Link    NIL

CONDITION: p    u

CONDITION: p    .Link    u

CONDITION: p    Tail

CONDITION: p    .Link    Tail

Similarly, Head may be replaced by some other value in the initialization assignment.

PROCESS2 is particularly useful if LT is configured as a function: it then is used to return a value.

The time required to traverse a list of n nodes with LT is:

T(n) = t0 + n    t1 + t2,

where ti is the time required for PROCESSi. This may need to be modified if there is an exit or return in PROCESS1.

Suppose that p is the pointer to a node, created perhaps by NEW(p), and then assigned a value, by, for example: p [image] .Value [image] x . The two natural places to insert into a list like aList are at the head and at the tail as shown in Figure 3.6. These operations are created by changing links one at a time, in a sequence that will not destroy necessary information:

procedure InsertHead(aList,p)                   {O(1)

if Empty(aList) then Tail    p endif

p    .Link    Head

Head    p

end {InsertHead

3.3.3 Insertion

Figure 3.6

procedure InsertTail(aList,p)            {O(1)

if Empty(aList)

then Head    p

Tail    p

endif

Tail    .Link    p

p    .Link    NIL

Tail    p

end {InsertTail

Note that the last assignment cannot precede the first assignment (after the if) in either InsertHead or InsertTail.

If the tail pointer is not used, InsertTail involves an O(n) traverse to find the tail node.

Exercises E3.1 and E3.2 are appropriate at this point.

3.3.4 Deletion

Deletion is done most easily at the node pointed to by Head. Deletion at the tail position requires a traverse to locate the predecessor of the node pointed to by Tail. At both positions, care must be taken to avoid trying to delete from an empty list. The two options are shown schematically in Figures 3.7 and 3.8.

Deletion at the head position is simply:

Figure 3.7

Figure 3.8

procedure DeleteHead(aList)                {O(1)

if Empty(aList)

then {deal with this error

else p    Head

Head    Head    .Link

DISPOSE(p)

endif

end {DeleteHead

Without disposal of the head node, the else clause reduces to:

else Head    Head    .Link

Deletion at the tail position presents an apparent problem and a real problem. If the final node of a list is deleted, Tail (if used) still points to it. However, as it is implemented, Empty is indifferent to Tail, and no operation is affected, because operations are carefully designed with Empty in mind. Besides, the deleted node is disposed.

Deletion at the tail position requires a pointer to the next-to-last node in order to delete the last one. Hence a search must be made for it by starting at Head and working down to detect Tail one step ahead. A variable q, which trails one node behind p and "remembers" the previous node, is introduced. It eventually points to the new Tail. Figure 3.9 depicts the final configuration.

Figure 3.9

Once q is known, deletion is simply:

q    .Link    p    .Link    {since p    .Link may be u

Tail    q

DISPOSE(p)

The relinking works even if Tail is not known for the list. Last-node deletion can be done with the help of LT, using it to find the last node q and to reset the links. A version that assumes that Tail is available is

procedure DeleteTail(aList)                  {O(LT)

if Empty(Head)

then {deal with this error

else if Head = Tail

then Head    Tail    .Link

DISPOSE (Tail)

else LT

endif

endif

end {DeleteTail

where LT uses:

CONDITION : p    Tail

PROCESS0 : q    Head

p    q    .Link

PROCESS1 : q    p

PROCESS2 : q    .Link    Tail    .Link

DISPOSE(Tail)

Tail    q

When these processes are assembled into one structure, the result is:

procedure DeleteTail(Head,Tail)              {O(n)

if Empty(Head)

then {deal with this error

else if Head = Tail

then Head    Tail    .Link

DISPOSE(Tail)

else q    Head

p    q    .Link

while p    Tail do

q    p

p    p    .Link

endwhile

q    .Link    Tail    .Link

DISPOSE(Tail)

Tail    q

endif

endif

end {DeleteTail

Note: These and all list-processing routines should be checked for correctness with empty, singleton, and general lists. Treatment of the first and last node as well as a general node should be checked whenever pertinent.

The search for the pointer q to the next-to-last node provides some contrast in programming styles: using or not using Tail, using sentinel or NIL-end lists, and using nested ifs or returns. In the version given, for instance, Tail [image] .Link may be either NIL or u. Some possibilities for DeleteTail follow.

One variation of DeleteTail is to replace both else clauses with:

return

endif

Another is to pass only Head to DeleteTail. The search loop for a NlL-end list then becomes:

q    Head                           {PROCESS0

p    q    .Link                      {PROCESS0

if p = NIL                          {PROCESS0

then DISPOSE(q)

Head    NIL

return

endif

while p    .Link    NIL do            {CONDITION

q    p                            {PROCESS1

p    p    .Link

endwhile

q    .Link    p    .Link               {PROCESS2

DISPOSE(p)                          {PROCESS2

Tail    q                            {PROCESS2

There are two ways to look at this variation: Tail is not needed, or Tail is generated for free and may as well be retained.

It is possible to bury the test for a singleton, at the expense of making a subtle change: the roles of p and q are initially reversed in the algorithm below, in order to avoid the expression "p [image] .Link" when p = NIL.

procedure DeleteTail(aList)      {standard DeleteTail

if Empty(aList)                {O(n)

then {deal with this error

else p    Head

q    p    .Link         {q will be and stay NIL iff aList is a singleton

while p    .Link    NIL do

q    p

p    p    .Link

endwhile

if q = NIL

then Head    NIL

else q    .Link    p    .Link

Tail    q

endif

DISPOSE(p)

endif

end {DeleteTail

Formally, the if nested in the outer else is PROCESS2.

Finally, a sentinel version that simply ignores the attempt to delete from an empty list has a rather different flavor.

procedure DeleleTail(Head)                  {O(n)

q    Head                                 {PROCESS0

p    Head    .Link                         {PROCESS0

while p    .Link    u do                    {CONDITION

q    p                                  {PROCESS1

p    p    .Link

endwhile

if p    u                                  {PROCESS2

then DISPOSE(p)                            .

q    .Link    u                          .

else Head    u                             .

if q    u then DISPOSE (q) endif

endif                                   {PROCESS2

end {DeleteTail

The choice between these variations and others is largely a matter of programming style, tempered by the application and features of the language used to implement them.

The deletion of tail nodes has been used here to illustrate how much even a simple data structure operation may vary and how much care must be taken with linked structures. However, if deletion of tail nodes is common in an application, the doubly-linked list structure of section 6.1 may be preferred. (The additional complexity of that structure does exact a price that must be taken into account; it is an option, not a clear improvement.)

Exercises E3.3-E3.6 and project PJ3.1 are appropriate at this point.

With DeleteHead and InsertHead it is easy to insert into or delete from the head of a list. A structure formally restricted to these moves is called a STACK. It is also relatively easy to insert into the tail of a list (with InsertTail) and delete from the head of it. A structure formally restricted to this pair of moves is called a QUEUE.

3.3.5 Search and Copy

Procedure LT can be structured as a search operation that returns a pointer to that node in a linear linked list that contains a given value, or return NIL if the value is not in the list:

function Hunt(aList,x)              {O(n)

p    Head                         {PROCESS0

while p    NIL do                  {CONDITION

if p    .Value = x                {PROCESS1

then return p endif

p    p    .Link

endwhile

return NIL                        {PROCESS2

end {Hunt

For n nodes in aList and the assumption that PROCESS1 takes a unit of time, the timing function of Hunt is as given: T(n) = O(n). A closer estimate of the timing of Hunt is that if the expected value (section 1.3.2) of the index of the node returned is k, then T(n) is approximately k. Briefly, T(n) ~ [Hunt].

Problems P3.1 and P3.2 are appropriate at this point.

Of particular interest for some applications of linked lists is the operation that returns a copy of a list. It, too, is an adaptation of LT:

function Copy(aList)

NEW(bList)                   {O(n)

if Empty(aList)              {creates a list header, not a list value node.

then bList.Head    NIL    {this is not equivalent to: {return aList.

bList.Tail    NIL

return bList

endif

NEW(r)

bList.Head    r

p    aList.Head

while p    NIL do

q    r

q       p

NEW(r)                     {create one node ahead

q    .Link    r

p    p    .Link

endwhile

q    Link    NIL

bList.Tail    q

DISPOSE(r)

return bList

end {Copy

Copy can be written without the extra node r, as indicated in one of the problems.

Exercise E3.7 and problems P3.3 and P3.4 are appropriate at this point.

3.4 The Upper Crust at Ballyhoo U (optional)

Suppose that J. F. Joy of Ballyhoo U decides to teach a short course called "Voila!" toward the end of this semester. She enters a proclamation in her public account describing the course content and the time and place of course meetings. In this account is room for two lists of names: IN, limited to 25 entries, and WANTIN, of unlimited length. Entrance to the list IN is provided on a first-come-first-served basis for juniors and seniors. Other students and latecomers must sign the WANTIN list. Those who are on the IN list and find that they will not be able to attend are asked to remove their entry.

One week before the first course meeting, those on the IN list must personally confirm with Dr. Joy that they will be able to attend; otherwise, they are scratched, and their place taken by the leading entries of the WANTIN list. Let us see how the lists IN and WANTIN are managed.

A simple way to manage the lists is by writing a "driver" program that accepts a limited set of commands from a user and executes them. A suitable set is:

"IN"           : Add my entry to the IN list.

"WANTIN"       : Add my entry to the WANTIN list.

"SCRATCH"      : Remove my entry from the IN list, or if it

is not there, from the WANTIN list.

"INLIST"       : Display the entries of the IN list.

"WANTLIST"     : Display the entries of the WANTIN list.

"QUIT"         : Thanks, but I must go now.

The commands "IN" and "WANTIN" call on a common function, WhoAreYou, which prompts the user to enter his or her student number (for checking upper-class status). Function WhoAreYow packages the information it receives into a node suitable for entry into the lists IN and WANTIN. The resulting node is the functional value of WhoAreYou. This procedure maintains a counter, InCount, of the number of entries in IN. If InCount is 25, new entries are added to WANTIN.

Both "INLIST" and "WANTLIST" simply use the procedure LT of section 3.3.2, specialized for listing as ListOut.

The "SCRATCH" command involves a search through IN, and perhaps WANTIN, which can be done with Hunt. The search is followed, if successful, by the removal of a node from the appropriate list with a procedure Scratch.

A driver that behaves as a command-shell is outlined as:

program JoinUp

Write {instructions to the user.

repeat

Read(Command)

case of Command

"IN":    if InCount < 25

then me    WhoAreYou

InsertTail(IN,me)

InCount    InCount + 1

else Write {explanation

me    WhoAreYou

InsertTail(WANTIN,me)

endif

"WANTIN":       me    WhoAreYou

InsertTail(WANTIN,me)

"SCRATCH":       me    WhoAreYou

p    Hunt(IN,me)

if p    NIL

then Scratch(IN,p)

else p    Hunt(WANTIN,me)

if p    NIL

then Scratch(WANTIN,p)

else {report failure to find me

endif

endif

"INLIST"    :    ListOut(IN)

"WANTLIST"    :    ListOut(WANTIN)

"QUIT"   :    exit

else:    {Provide the user with a list

{of legitimate commands.

endcase

forever

end {JoinUp

Function WhoAreYou is not specifically list processing and is left to the reader.

Procedure Scratch does not need to be a general deletion routine. When it is invoked, the pointer p provided by Hunt is known to be non-NIL. The deletion operation, though, needs the predecessor of p, which is to become linked to the successor of p, p [image] .Link, instead of p:

procedure Scratch(aList,p)                     {O(n)

if p = Head

then DeleteHead(aList)

else q    Head

while q    .Link    p do

q    q    .Link

endwhile

q    .Link    p    .Link

DISPOSE(p)

endif

end {Scratch

InCount is not decremented by Scratch, and so an upper-class student who is forced to enter WANTIN is not kept there by a latecomer who follows a withdrawal.

Searching through the list twice is a waste, and so Hunt and Scratch may be combined.

Program PG3. 1 is appropriate at this point.

Professor Joy can use JoinUp to easily remove "no-shows" from IN; but, since JoinUp does not increment InCount, a routine is needed to do so. On most academic systems, this routine can be proprietary--unavailable to the student. The transfer of m nodes from WANTIN to IN with JoinUp will use WhoAreYou m times, even after adding a routine to readjust InCount. A short program can be written to do the same thing by either transferring nodes from WANTIN to IN or copying them. The copying procedure is:

program Floodgate                     {O(n)

m    25 - InCount

for i = 1 to m do

NEW(p)

p       WANTIN.Head

DeleteHead(WANTIN)

InsertTail(IN,p)

InCount    InCount + 1

next i

end {Floodgate

Note: The command-shell format of Joinup can be adapted to the management of a variety of resources. In this case the resource is a pair of lists, and the commands are the only contact the user needs with them. Details of the algorithms and implementation of the lists are deliberately transparent to the user (not visible).

3.5 Array Support for Linked Lists

A linked list may be implemented with values and links stored in arrays. One way to do this is to store single node values in array Value[1 . . Max], and their corresponding links in array Link[0 . . Max]. In effect, Value[i] and Link[i] together form the ith node of the list. Helpful conventions are: Link[0] points to the first node; Link[i] = 0 is equivalent to the NIL link used with pointer variables. Essentially, the correspondences between a pointer-linked list and an index-linked list are:

Head    Link[0]

p    i

p    .Value    Value[i]

p    .Link    Link[i]

Figure 3.10 depicts a specific example, with some of the links indicated.

Figure 3.10

A display routine for such a list, based on the general traverse of it, ALT (Array-List Traverse), by substituting a Write for PROCESS1, an initialization for PROCESS0, and leaving PROCESS2 null:

ALT

procedure ALT(Value,Link)   {0(n)

Next    Link[0]           {PROCESS0

while Next    0 do         {CONDITION

PROCESS1(Next)

Next    Link[Next]

endwhile

PROCESS2

end {ALT

The result is an orderly "SPITSHINE".

With the data in front of us, we easily see that this word can be hyphenated with the insertion of the new value "-" in the list:

Value[10]    "-"

Link[10]    Link[7]

Link[7]    10

The number of characters in the list, its Length, can be determined by the substitutions:

PROCESS0: Length    0

Next    Link[0]

PROCESS1: Length    Length + 1

If two pointers are used, which is a great convenience for some applications, then a declaration of Link[0 . . Max + 1] can be helpful, allowing Link[Max + 1] to serve as the tail pointer.

A feel for the effect of array support can be gained by rewriting some of the procedures discussed in conjunction with pointer variables in this new context. If it is assumed that Value and Link are global variables, deletion of the head node is accomplished as follows and in Figure 3.11.

procedure DeleteHead                               {O(1)

if Link[0] = 0

then {deal with this error

else Link[0]    Link[Link[0]]

endif

end {DeleteHead

Figure 3.11

If k is the index of a node to be added to the tail of the list, a version of InsertTail is needed:

procedure InsertTail(k)                  {O(1)

if Link[0] = 0

then Link[0]    k

endif

Link[Link[Max + 1]]    k

Link[Max + 1]    k

Link[k]    0

end {InsertTail

The result of adding "S" to the tail of the preceding list is depicted in Figure 3.12.

Figure 3.12

The rewriting of other list-processing procedures for array support is explored in the exercises.

Exercises E3.8-3.10 and project PJ3.1 are appropriate at this point.

If a value input or generated within the program is to be inserted, index k must be located. The program must keep track of storage available within the array Value, even after arbitrary deletions.

Note: The programmer must provide for the possibility that there will be no available storage within the value array.

To insert a value, say x, at the tail of an array-supported list generally requires two moves:

GetSpace(k)

InsertTail(k)

Unhappily, GetSpace may not be simple. It can be kept simple if insertions are made into Value[1 . . n] for n [image] Max, and the space occupied by deleted values is just ignored.

Memory is essentially an array of cells, so consideration of storage allocation will eventually come to the programmer's attention even if pointer-variables and built-in NEW operations are available in the programming language being used. These language features cause details of the allocation and retrieval of storage to be transparent (invisible) to the user. Transparency is not the same as nonexistence.

The problem of the use and release of storage during program execution is called dynamic memory allocation. Approaches commonly taken to solve this problem use material that is discussed in several later chapters. Chapter 6 deals with this topic again.

3.6 Ordered Lists

The list of people who have contributed to the Planetary Society or to a politician's campaign or who have had contact with a known disease carrier may grow to be large or remain small. Information about these people needs to be maintained in order, usually as determined by their full name used as a key to a record of pertinent data. Several factors form the core of the problem of maintaining the list and affect the choice of data structure used to solve it: the list is of indeterminate size, it changes size, it is sparse in the data-space of possible full names, and it must be ordered so that searching it is convenient. The core of a data structure that mirrors such problem domains is an ordered list, a list of nodes sorted by value (or key). Such a list may be either endogenous or exogenous, depending on the role played by node values in a particular application.

Figure 3.13 is an example of an ordered list where the node values are chosen for convenience to be simply integers serving as keys.

Figure 3.13

Nodes in nList have a natural type:

NumNode = RECORD

Value: INTEGER

Link :    NumNode

END {NumNode

Search-and-display routines for a list like nList are variations of procedure LT (section 3.3.2). Deletions at either end could be taken care of by DeleteHead and DeleteTail (both discussed in section 3.3.2). The management of nList as an ordered list requires that deletions take place from any position and that additions can be made so that the node values retain their numeric order.

The first requirement for insertion of a new value into nList is to encase it into a node:

{determine x

NEW(xNode)

xNode    .Value    x

The next requirement is to locate the position in which xNode should appear. It is convenient to make "position" mean a pointer to the proper predecessor of xNode: the node which should precede xNode when the new ordered list is formed by the incorporation of xNode.

A parameter Spot is added for convenience so that IntoList inserts xNode if it should precede the node Spot. When Spot [image] NIL, the entire list is available to define an insertion position.

The proper interior position can be found between Head and Spot (possibly NIL) with a version of LT, using:

CONDITION : p    .Link    Spot

PROCESS0 : p    Head

PROCESS1 : Next    p    .Link

if x    Next    .Value then exit endif

Insertion is then:

PROCESS2 : xNode    .Link    p    .Link

p    .Link    xNode

However, if x = 15 were to be included in the example list, then there would be no proper predecessor for it. A FIND operation would need to return information specifying that fact as well as to return the proper predecessor in the general case.

Here modularity can be overdone, and it is easier to find and insert in a single operation:

procedure IntoList(nList,xNode,Spot)           {O(n)

if Empty(nList)

then Head    xNode

Tail    xNode

xNode    .Link    NIL

return

endif

x    xNode    .Value

if (x    Head    .Value) OR (Spot = Head)

then xNode    .Link    Head               {new Head node

Head    xNode

else p    Head                           {LT

while p    .Link    Spot do

Next    p    .Link

if x    Next    .Value then exit endif

p    p    .Link

endwhile

xNode    .Link    p    .Link

p    .Link    xNode

endif

end {IntoList

Exercise E3.11 is appropriate at this point.

Procedure IntoList can be readily adapted to the sorting of a stream of input values into a linked list.

Problem P3.5 is appropriate at this point.

A more difficult task is an analog of InsertionSort (section 1.4.3) that sorts a linear linked list that already exists, rather than building it by steps so that it is sorted at each stage of its construction. The relinking involved can be studied by considering the stages of an example. In Figures 3.14 - 3.18, the links generated at each stage are dashed.

In Figure 3.14, the second node is to be inserted before the first. Figure 3.15 shows the third node inserted before the (new) first. The fourth node remains in place in Figure 3.16. In Figure 3.17, the fifth node is inserted between the second and third. Figure 3.18 shows that the sixth node is then inserted before the first.

Figure 3.14

Figure 3.15

Figure 3.16

Figure 3.17

Figure 3.18

An examination of the pointer moves required to insert the node designated by Spot shows that moves 2 and 3 (Figures 3.15 and 3.16) are essentially the action of IntoList(nList,Spot,Spot [image] .Link) after Spot is excised from the list. Hence the sorting procedure becomes:

procedure LinearSort(nList)                      {O(n2)

if Empty(nList)

then {deal with this error

else Spot    Head    .Link

if Spot = NIL then return endif

Prior    Head

while Spot    .Link    NIL do

SpotLink    Spot    .Link

Prior    .Link    SpotLink

IntoList(nList,Spot,SpotLink)

if Prior    .Link    SpotLink

then Prior    Spot

endif

Spot    SpotLink

endwhile

Prior    .Link    NIL

IntoList(nList,Spot,NIL)

if Tail    .Link    NIL

then Tail    Prior

endif

endif

end {LinearSort

Exercise E3.12, problems P3.6-P3.9, and programs PG3.2-PG3.4 are appropriate at this point.

3.7 Circular Lists

The tail node of a linear list can point back to the head node, forming a cyclic structure called a circular list, illustrated in Figure 3.19.

Figure 3.19

Circular lists find their place as a programming convenience in some applications because of two different effects of circularity on a sequence of searches:

1. An exhaustive search of a circular list returns to the first node searched. The list may be one of several in a combination of lists connecting common nodes, as is the case for the sparse matrix structure of section 6.3. Return to the initial node then brings a traverse to a point from which it may branch onto another substructure.

2. If a pointer to the last node of a circular list traverse is retained to become the first node of the next walk, a sequence of searches is distributed over the list. Even-handed attention can be useful in applications such as the dynamic storage allocation model of section 6.4.

Note: Insertion, deletion, and walking in circular lists differ little from the same operations applied to linear lists. The needed algorithms are to be found in Part II, section C, or derived by analogy.

3.8 Polynomial Arithmetic (optional)

Polynomials like those below are prime examples of objects that should be managed as linked lists:

P1:  31x17 - x11 + 5x4 + 10x3 - 3

P2:  7x17 + 4x6 - 2x2 + 8

P3:  x11- 5x4 + x2 + 3

P1 is a polynomial of degree 17, because 17 is the largest exponent of its nonzero terms. Only the coefficient-exponent (CE) pairs (31,17), (-1,11), (5,4), (10,3), and (-3,0) are needed to describe P1, since terms specified by (0,16), (0,15), . . . all have zero coefficients.

Given only that the largest exponent allowed for polynomials in a collection of them is n, a collection-manager is blind in the sense that the number of CE pairs needed to store one or the whole collection is not determined by n.

Polynomials such as P1, P2, and P3 are intrinsically sparse: a polynomial of degree 17 could take as many as 18 CE pairs to describe, but the examples clearly do not.

In a system where arithmetic is done on polynomials, they are dynamic. P1 + P3 = P4 is 31xI7 + 10x3 + x2, described by CE pairs (31,17), (10,3), and (1,2). The assignment P1 [image] P1 + P3 changes the number of CE pairs needed to describe P1.

An ordered linked sentinel list is a natural structure for polynomials. For P3 such a list has the form shown in Figure 3.20 where nodes are of the type:

PolyNode = RECORD

exp  : INTEGER

Coef : REAL

Link :    PolyNode

END {PolyNode

Figure 3.20

A system that deals with polynomial lists of this sort can create them with the help of IntoList (section 3.6), and a display routine is easily devised. Writing a program that performs polynomial arithmetic, which has a very algebraic flavor, then becomes a reasonable project (PJ3.2, in fact). Most of the effort required to devise polynomial arithmetic routines depends on understanding polynomials, rather than difficulties in procedure design, and that is typical of much programming. As an example of polynomial arithmetic on linked lists, here is a function that returns the sum of linked-list polynomials P and Q:

function PolyAdd(P,Q)

PAQ    u

while P    u AND Q    u do

NEW(R)

case

P    .exp > Q    .exp : R       P

P    P    .Link

InsertTail (PAQ,R)

P    .exp = Q    .exp : R    .exp    P    .exp

R    .Coef    P    .Coef + Q    .Coef

P    P    .Link

Q    Q    .Link

if R    .Coef    O

then InsertTail(PAQ,R)

endif

P    .exp < Q    .exp : R       Q

Q    Q    .Link

InsertTail(PAQ,R)

endcase

endwhile

if P = u

then Rest    Q

else Rest    P

endif

while Rest    u do

NEW(R)

R       Rest

InsertTail(PAQ ,R)

Rest    Rest   .Link

endwhile

return PAQ

end {PolyAdd

Exercise E3.13, problem P3.10, program PG3.5, and project PJ3.2 are appropriate at this point.

3.9 Arithmetic of Unbounded Precision (optional)

The federal deficit of the United States in 1983 was a number of approximately this size:

$197,800,400,650.23

Such a number is difficult to store in most programs. Only approximations to it can be stored as either an integer or a real number because it has too many significant digits. (Even when the number of digits of interest is small, the number base in most computers is 2, not 10, which causes some decimal values to be approximate.) The precision of a number is the number of significant digits in it, and a budget number surely needs to be precise to the penny.

The approximate forms available to the programmer as types INTEGER and REAL, double-precision or not, are a compromise between the time-and-space efficiency suitable for most programs and the need for precision in some. In most applications, 1.978 [image] 1011 would be an adequate representation of the number above.

Unbounded precision in arithmetic can be gained by retaining every digit generated in any arithmetic calculation. A suitable data structure for such arithmetic would need to grow and shrink during execution (be dynamic) and be open-ended (blind to the eventual size of a number). One solution is to use a linked list for the storage structure of an unbounded number, as was done for the polynomials of section 3.8. In fact, a value such as 743.1 is a polynomial in powers of the number base 10:

743.1 = 7    102 + 4    101 + 3    100 + 1    10-1

Figure 3.21 shows this value stored as a linked list.

Figure 3.21

Given that there is a data structure suitable for values of unbounded precision, it is possible to write a program (which could be used as a procedure within a larger program) that acts as a calculator for such numbers. In Part II, section D, a Pascal program acting as a calculator is given a command-shell form that supports commands to add, subtract, multiply, and divide integer values in a way that provides precision bounded only by the storage capacity of the Pascal system on which it is executed. The restriction to integer values simplifies both the supporting list structure and the arithmetic routines and allows the focus of the program to be a simulation of the response of a hand calculator to a sequence of keystrokes.

Summary

The LIST is such a general structure that lists with restricted access are the basis for later chapters and sections of the book. The linked list is the structure of choice for some applications because it is dynamic, sparse, and blind. It may serve as a directory for data and hence be exogenous, or it may contain the data of interest and be endogenous. (The data structures of following chapters are also used in these two ways.)

A list may be supported as a programming structure in several ways. The list model most used in this book is based on records linked by pointers. The final pointer may be NIL or a universal sentinel node, u. Array support is also possible and sometimes convenient.

The information in the nodes of a list is accessible via the head node and a sequence of links, but not directly, in contrast to the array structure. The head node is easily reached, the interior nodes require a list-walk to reach, and the reachability of the tail node depends on implementation.

Common list operations include a test for an empty list from which an attempt to delete usually represents an error. (Later in the book it is sometimes useful to assume that deletion from an empty list is an operation that cannot fail.)

Insertion at the head and tail of a list and deletion at the head are 0(1) operations. These operations provide one definition of structures STACK and QUEUE to follow in later chapters. Deletion at the tail involves a search for its predecessor and so is 0(n).

List operations (and operations on other structures) should be tested for their treatment of empty and singleton lists and behavior at the head, tail, and interior nodes.

Many list algorithms are modeled after one--the list-traverse (or list-walk) from head to tail. Procedures Search and Copy are examples of such algorithms. The general walk LT of section 3.3.2 has an array-support form ALT described in section 3.5.

Ordered lists and circular lists provide specialized variations that have a number of applications, some treated in later chapters.

Two list-management paradigms (that lead to exploratory assignments) are a sign-up list JoinUp (example in section 3.4) and polynomial arithmetic (introduced in section 3.8). Both JoinUp and the arithmetic of unbounded precision that was introduced in section 3.9 and is programmed in Pascal in Part II, section D, provide examples of command-shells that hide extraneous information from the user.

Exercises

Exercises immediate applications of the text material

E3.1 Show, with a diagram, the result of switching the two assignments after the if in InsertHead of section 3.3.3. What is the effect of moving the if statement below one or both assignments?

E3.2 In InsertTail of section 3.3.3, why aren't the two assignments of p to Tail redundant? Why must the if statement be executed before the assignments? Can the order of the assignments be altered in any way'?

E3.3 A deletion (or insertion) procedure should be checked for performance on these lists:

Trace the action of both DeleteHead of section 3.3.4 and DeleteTail of section 3.3.4 on lists A, B, and C.

E3.4 Trace the action of the standard DeleteTail of section 3.3.4 on lists A, B, and C of E3.3 when (q [image] p [image] .Link) is omitted. What is the result?

E3.5 Trace the action of the standard DeleteTail of section 3.3.4 on lists A, B, and C of E3.3 when (if q = NIL . . . ) is replaced by just the else phase. What is the result?

E3.6 Trace the action of the standard DeleteTail of section 3.3.4 on lists A, B, and C of E2.3 when the while condition is changed to: (p [image] NIL). What is the result?

E3.7 Trace the action of Copy of section 3.3.5 on lists A, B, and C of E3.3. Is it possible to change the sequence of assignments in the body of the while?

E3.8 Rewrite InsertHead of section 3.3.3 for explicit array support.

E3.9 Rewrite DeleteTail of section 3.3.4 for explicit array support.

E3.10 Rewrite Hunt of section 3.3.5 for explicit array support.

E3.11 Trace the action of IntoList of section 3.6 with insertion of values 10, 3, and - 10.

E3.12 Trace the action of LinearSort of section 3.6 on the example of section 3.6 by keeping track of all pointer variables during placement of the first and last node.

E3.13 For polynomials P2 and P3 of section 3.8, find: P2 + P3, P2 - P3, P2 [image] P3, and both the quotient Q and remainder R of P2 / P3.

Problems

Problems not immediate, but requiring no major extensions of the text material

P3.1 If the nodes of a list are numbered from 1 at Head to N at Tail, then there is an expected value, 1 [image] ex [image] N for the number of the node located by Hunt of section 3.3.5. Assume that every node is equally likely to contain the searched-for value x and that there is only a 50 percent chance that x is a node value at all. Determine T(N) for Hunt, as a function of ex.

P3.2 Continue the analysis of P3.1 with the assumption that x tends to be found near the head of the list. In particular, assume that finding x at nodek is twice as likely as finding it at nodek+1 for 1 [image] k [image] N - 1. In effect, find ex.

P3.3 Diagram the behavior of the alteration of Copy of section 3.3.5 shown below, for the lists A, B, and C of E3.3.

NEW(q)

bList. Head    q

p    aList.Head

while p    NIL do

q       p

NEW(r)

q    .Link    r

q    r

p    p    .Link

endwhile

q    .Link    NIL

return bList

.

.

.

P3.4 Diagram the behavior of the altered Copy of P3.3 for the lists A, B, and C of E3.3 if the while condition is changed to: (p [image] .Link [image] NIL).

P3.5 Write a procedure InputSort, which inputs a stream of values, encases them in nodes, and inserts them with IntoList of section 3.6 as they are entered into an ordered list. See PG3.2.

P3.6 Determine the timing function for LinearSort of section 3.6.

P3.7 Rewrite LinearSort of section 3.6 so that the while condition is: (Spot [image] NIL).

P3.8 Rewrite SelectionSort of section 1.4.1 to sort a linked list, using pointer variables, and determine its timing function. See PG3.3.

P3.9 Rewrite BubbleSort of section 1.4.2 to sort a linked list, using pointer variables, and determine its timing function. See PG3.4.

P3.10 Write a procedure that multiplies two linked-list polynomials.

Programs

Programs for trying it out yourself

PG3.1 Write and run program JoinUp of section 3.4.

PG3.2 Turn the procedure InputSort of P3.5 into a program that inputs a stream of values and sorts them by insertion into a linked list, then displays the values in order.

PG3.3 Turn the procedure of P3.8 into a program that inputs a stream of values, places them in a linked list, sorts them by exchange, and displays the values in order.

PG3.4 Turn P3.9 into a program that inputs a stream of values, places them into a linked list, sorts them as BubbleSort of section 1.4.2 does, and displays the values in order.

PG3.5 Write a program to accept two polynomials (supplied as coefficient-exponent pairs) and display their product.

Projects

Projects for fun; for serious students

PJ3.1 Rewrite PJ2. 1, using linked lists to track available space and in-lot cars. (The lot may be enlarged, although checking the "lot-full" test then becomes a bore.)

PJ3.2 Write a program that will manage a collection of linked-list polynomials and perform arithmetic on them as requested. The operations to be provided are addition, subtraction, multiplication, and division.

Go to Chapter 4 Return to Table of Contents


Data Structures: CHAPTER 4: STACKS

 Return to Table of Contents

CHAPTER 4: STACKS

A wide variety of processes exhibit a pattern similar to the movement of trays in a cafeteria, in and out of a spring-loaded pile. Trays are added to the top of the pile and also removed from the top of the pile. As a result, the last tray in is the first tray out. Common terminology for a list of items that grows and shrinks in this way is a LIFO (Last-In-First-Out) list or stack. The same pattern appears in discussions of the strategies available for inventory control of items on local grocery store shelves and in large warehouses. Stacks are used to unravel the flow of control when one computing procedure calls another, which calls another, etc. They are sometimes used as essential design features of computing hardware, of language translation, and of computing languages. The stack structure even imposes itself on the precious junk in a storeroom of a private dwelling, not always with happy effect.

Stacks can be implemented in arrays without the use of link indices, but linked lists are also a natural way to discuss stacks. Consider the linked list in Figure 4.1.

Figure 4.1.

If two deletions are made from the top (the head) of this list, and the value 2 is added at the top, the effect is to carry out the operations:

DELETE

DELETE

INSERT 2

The resulting snapshots of the list are shown in Figure 4.2.

Figure 4.2

In terms of the list-processing routines of Chapter 3, a linked-list structure is a stack iff insertion is restricted to InsertHead and deletion is restricted to DeleteHead.

4.1 The Stack Structure

Stacks need to be created, like any other structure, but insertion and deletion both occur at the top (the head), so no tail pointer or index is required. The formal operation that gives birth to a stack is CREATE.

Activation of a stack is usually accompanied by initializing the top pointer to NIL (or u). A stack may grow and then decrease until it is again empty, as though freshly activated. The formal (Boolean) operation that checks the viability of a stack is EMPTY.

A stack grows iff a single node is inserted into its top position. No way of assigning values directly to stack nodes is available in a pure stack. The formal operation that inserts a node into a stack is PUSH.

The only value of a stack accessible to its environment is the value of its top node.

The only retrievable value of a stack is that of its top node. The formal operation that returns the value of the top node is TOP.

The top node must be removed from a stack before the values of other nodes can be retrieved. The formal operation that removes the top node from a stack is DELETE.

The behavior of a stack is entirely described, however it may be implemented, by CREATE, EMPTY, INSERT, DELETE, and TOP, and their interactions. TOP followed by DELETE, implemented as one operation, is called POP.

However a stack is stored, it can be used to alter a sequence of values. For example, the input stream of characters S-T-A-C-K can be arranged in reverse order by stacking them and then unstacking them, as depicted in Figure 4.3.

Figure 4.3

The output produced by the pops is: K-C-A-T-S.

Figure 4.4 shows that the consonants can be moved after the vowel A by stacking them as encountered. The output produced by the pops is: A-K-C-T-S.

Figure 4.4

This process is easily generalized into the separation of two classes of items in a stream into leading and trailing segments in O(n) time.

Railroad cars are separated on siding tracks in just this way, although more than one siding (stack) may be needed for more than one category of car.

With the aid of more than one stack, shuffling can become more sophisticated. Suppose the letters l-N-P-U-T-S-T-R-E-A-M are placed in stack S1 in nondecreasing order, and the others shunted to stack S2 as in Figure 4.5.

Figure 4.5

Unstacking of S1 followed by unstacking of S2 forms a new stream: U-P-N-I-M-A- E-R-T-S-T. Now this stream may be reversed (by stacking it, for example) to form: T-S-T-R-E-A-M-I-N-P-U. Apply the process in Figure 4.5 to the new stream, depicted in Figure 4.6.

Figure 4.6

If S1 and S2 are unloaded with the help of a third stack as before, the resulting stream is: S-R-E-A-M-I-N-P-T-T-U.

Surprising as it may seem, six more repetitions of this process will produce the sorted stream: A-E-I-M-N-P-R-S-T-T-U. Try it!

Devising a sorting routine from this process is project PJ4.1.

Project PJ4.1 is appropriate at this point.

4.1.1 Stack Implementation and Management

Two natural implementations are a list structure, which uses pointer variables, and an array-management scheme. If the list is called liStack and the array is called aStack, then the same data set takes two schematic forms, shown in Figure 4.7.

Figure 4.7

There is also a hybrid form, in which a stack is implemented as a linked-list structure, which in turn is supported in arrays:

This form is seldom used and will not be pursued.

Any stack supported in an array must be programmed with some concern for the maximum possible index, Max. A variable, Top, is required to locate the top item in aStack. Table 4.1 allows comparison between the two major treatments by examining the stack operations in parallel. For comparison, the linked-list form of Push accepts values instead of nodes. Note that all of the stack management routines are fast: O(l). This intrinsic efficiency promotes the use of stacks in applications.

The programming that manages a stack with an input file (such as a sequence of keyboard entries) is straightforward. For example, a sequence of commands of the form: I (for INSERT), and D (for DELETE) can be processed by this loop:

repeat

Read(Command)

case of Command

"I" : Read(Item)

Push (Stack,Item)

"D" : Delete(Stack)

"E": exit      {E stands for End-Of-File

else: {deal with this error

endcase

forever

Commands to invoke TOP or the display of the complete stack are easily added to the preceding command loop. Difficulties (challenges) arise from exception handling (dealing with errors) and from the quirks of individual applications.

A command loop like the one above is thought of as a shell that surrounds the machinery of resource management. It forms an interface with the user, who sees it as a virtual machine. The user supplies commands that evoke a response. From the user's view, the loop is a machine that does whatever the commands say to do.

Table 4.1

            liStack                   CREATE              aStack

------------------------------------------------------------------------------

liStack    NIL                    {O(l)        Top    0                 {O(l)

------------------------------------------------------------------------------

                                      EMPTY

function Empty(liStack)          {O(l)         function Empty(aStack)   {O(l)

  if liStack = NIL                               if Top = 0

     then return TRUE                               then return TRUE

     else return FALSE                              else return FALSE

     endif                                          endif

end {Empty                                       end {Empty

------------------------------------------------------------------------------

                                      INSERT

procedure Push(liStack,i)        {O(l)         procedure Push(Stack,i)   {O(l)

  New(iNode)                                     if Top > Max

  iNode    .Value    i                               then {deal with this error

  iNode    .Link    liStack                          else Top    Top + 1

  liStack    iNode                                       aStack[Top]    i

  end {Push                                        endif

                                                 end {Push

------------------------------------------------------------------------------

                                      DELETE

procedure Delete(liStack)        {O(l)         procedure Delete(aStack)  {O(l)

  if Empty(liStack)                              if Empty(aStack)

     then {deal with this error                     then {deal with this error

     else liStack    liStack    .Link                 else Top    Top - 1

     endif                                          endif

  end {Delete                                  end {Delete

------------------------------------------------------------------------------

                                        TOP

function TopValue(liStack)       {O(l)         function TopValue(aStack) {O(l)

  if Empty(liStack)                              if Empty(aStack)

     then {deal with this error                     then {deal with this error

     else return liStack    .Value                   else return aStack[top]

     endif                                          endif

  end {TopValue                                  end {TopValue

The details of how the response gets done internally are of no concern at the command level. This kind of deliberate information-hiding is also present in the use of a data structure, and the concept of a virtual machine is a major guiding philosophy in the design of a modern operating system.

The shell format that professionals use to deal with complexity is well worth imitating.

Note: A realization in Pascal of the loop above as a "stack machine" is found in Part II, section E.

The display of a stack can be carried out by standard list procedures but only if the stack is treated as a list, not a stack. As an instance of STACK, display of the contents proceeds the same way as the display of the contents of a crowded closet or a full deep-freezer: items are popped off the stack, displayed, and then pushed onto an auxiliary stack. Restoration is a reverse of this process, without the display. It is more reasonable and common practice to forgo rigid formality and use a list-walk to display the contents of a stack.

Note: STACK is a structure that does not require a traverse in most applications. There is no direct access to interior nodes. For other data structures in the book, with the exeption of QUEUE, the traverse is a fundamental skeleton for constructing algorithms.

4.1.2 On the Use of Stacks

Stacks are sometimes chosen as a data structure simply because they model behavior that occurs in the system to be modeled (as would be the case in a cafeteria simulation). Actually, modeling natural stack behavior may not even require a stack, as indicated in section 4.5. Stacks may also be chosen to deliberately shape a computing scheme.

Stacks are dynamic--growing and shrinking at will--but that fact is hidden from the user who sees only the top item. Stacks are blind because it is not necessary to know their potential size when they are created. Stacks can be sparse because no interior item in a stack occupies a special place in it--only the sequence of PUSH and POP operations determines where in a stack an item is to be found. Furthermore, it does not matter except to determine when an item reaches the top. All of these features indicate that a linked list is a natural support for a stack.

Stacks provide O(l) direct access, although only to one item at a time, and the direct access moves from one item in the stack to another. That indicates the utility of an array as a natural support for a stack.

Neither support system captures the nature of the stack in and of itself, which arises from two major attributes:

1. A stack can resequence a stream of entries.

2. A stack is immediate. Entry and exit from a stack are O(l) operations. What you get is what you see on top; there is no apparent depth or size or structure to a stack. In some sense, a stack is a point-like structure.

Above all, a stack is so very easy to use, if what you want from it is the last thing put into it.

4.1.3 The Calculation Game (optional)

The game described below comes trom Nancy Griffith of Georgia Institute of Technology. It is introduced hereto serve as an example of the use of multiple stacks and as a source of assignments.

A deck of cards can be represented by 52 symbols, tour each of: A, 2, 3, . . . , 9, 10, J, Q, K. They may be regarded as the numbers 1 . . . 13. Suppose one card at a time is dealt from the deck and placed onto one of four foundations so that cards in the foundations F1. . . F4 are in the order:

F1: A 2 3 4 ...       F2: 2 4 6 8 ...

F3: 3 6 9 Q 2 5 8 ... F4: 4 8 Q 3 7 J ...

This is not possible for most shufflings of the deck, of course, but it can be done with the help of four waste piles, W1 . . . W4. The top of any waste pile can be used as a source for a card to be moved to a foundation at any time.

One of the possible strategies for the use of the waste piles is to deal from the deck onto them in the order W1, the W2, W3, W4, W1, . . . but to move a card from a waste pile to a foundation whenever possible. A program can be written to accomplish these moves and verify the result.

A shuffle can always be produced from a real deck of cards; the order produced in the real world can be used as an input stream. The professional approach is to use a random number generator (see Part II, section A). With each of the 52 card symbols S[i] arranged in some standard order, associate a pseudorandom number RN[i], as shown in Figure 4.8. Now sort S, using the values in RN as a key. The symbols of S will be distributed as randomly after the sort as the values in RN were before the sort.

Figure 4.8

This is a valuable exercise to do with all three variations of stack support in order to compare them and develop a feel for their commonalities and differences.

Program PG4.1 is appropriate at this point.

4.2.2 Conversion of Infix to Postfix

Conversion of infix expressions to postfix form depends upon the precedence of operations. To know that multiplication is done before addition in the expression A + B * C requires a "look-ahead" process, because the addition may or may not be applied immediately to the operands A and B that bracket it. It is true, in contrast, that A + B - C is equivalent to (A + B) - C in most languages.

In order to process a postfix expression left-to-right in the manner indicated, the tokens in the expression must be sequenced so that the operations occur in the order in which they are to be performed. With a little practice, one can spot the appropriate order. For example, the equivalent infix and postfix forms of several expressions are:

Infix Expression     Postfix Expression

a * b + c              a b * c +

a + b * c              a b c * +

a * (b + c)            a b c + *

(a + b) * c            a b + c *

A stack may be used to provide the same effect.

Parentheses complicate the conversion because they override the precedence and must be dealt with in some manner.

As a simple example, suppose that operators are restricted to:

~, *, /, +, - {and parentheses

A common assignment of precedence would be:

level 3: ~

level 2: *, /

level 1: +, -

A further assumption is that operators on the same level of precedence are to be evaluated left-to-right.

By examining the four variations of the example expression we can see that:

When an operator is encountered, it is not put in the postfix list PostX until its operands have been acquired. It may be retained on a stack, OpStack, until its proper postfix position is reached. This position is determined by the processing of a sentinel value, a right paren, or an operator with precedence at least as low.

If an operator that has been retained is of precedence higher than or equal to an incoming one, then it is unstacked and added to the list. The sentinel value has the lowest possible effective precedence.

A left paren--"("--is placed on OpStack. A right paren--")"--causes the transfer of OpStack items to PostX down to a left paren. Both parens are discarded.

The behavior of the stack and the list for a * b + c & is depicted in Figures 4.13 - 4.15. The resulting list is quickly retrieved.

Figure 4.13

Figure 4.14

Figure 4.15

With a function Level that returns the level of an operator, the infix-to-postfix conversion becomes:

procedure IntoPostfix

Next(Token)

while Token    "&" do

case of Token

{operand}: InsertTail(PostX,Token)

")": begin

while NOT Empty(OpStack)

Op    Top(OpStack)

Delete(OpStack)

if Op    "("

then InsertTail(PostX,Op)

else exit

endif

endwhile

end

"("  : Push(OpStack,Token)

else : begin {new operator

Op    Top(OpStack)

while Level(Op)    Level(Token) do

Delete(OpStack)

InsertTail(PostX, Op)

if Empty(OpStack)

then exit

else Op    Top(OpStack)

endif

endwhile

Push(OpStack, Token)

end

endcase

Next(Token)

endwhile

while NOT Empty(OpStack) do

Op    Top(OpStack)

Delete(OpStack)

InsertTail(PostX, Op)

endwhile

end {IntoPostfix

Exercise E4.2, problem P4.1, and program PG4.3 are appropriate at this point.

4.2.3 Postfix Lists in Translation (optional)

Conversion of infix expressions to postfix lists is not followed immediately by execution of the postfix list in most computing systems. The conversion normally occurs in a compiler, followed by translation of the list into instructions for a machine that may or may not provide stack operations directly. Nevertheless, the translation is driven by a process very similar to the execution of a postfix list. Schematically for A B C * + & in a postfix stack, the execution might proceed as in Figure 4.16.

Figure 4.16

Here "*" causes the generation of the machine instructions to multiply the values of the top two addresses on the stack, place the result in T1 and push the address T1 onto the stack. The translation algorithm then places T1 on its postfix stack, but the translated result of the encounter with "*" is simply the set of instructions equivalent to "multiply B and C and place the result in T1." The actual multiplication occurs at execution time, not compilation time, in a manner dependent on the machine architecture and details of translation. The postfix list and stack are used simply to organize the sequence of machine operations. A compiler is developed to translate a language into the instructions of a particular target machine. Several features of the design of the target machine might be used to make the resulting translation more efficient, but the essence of the stacking scheme is likely to remain in the design of the translation process.

Compilers are very complex programs, but much about them is general, if not universal; and the details of a specific target machine are only involved in parts of their design. Much of their generality and much of that machine isolation follow from the informed use of data structures.

4.3 Stacks in the Language FORTH (optional)

Charles H. Moore invented a language, FORTH, which is now in use in many varied applications. It is difficult to determine how much of the power of available versions of FORTH is due to the central role of stacks in the design of the language, but stack ideas certainly give it much of its character. FORTH has a unique combination of properties, as it is commonly implemented and used.

Two aspects of FORTH are of particular interest in the context of this chapter:

1. Statement syntax is intrinsically postfix in form.

2. Statement semantics are based on stack manipulation.

These are independent properties, although they form a natural partnership. Partly because of this combination, the modularity of FORTH programs has a different flavor and genesis than that of, say, Pascal programs.

A very brief introduction to elementary FORTH is found in Part II, section F. It is included as an interesting example of stack management made explicit in a computer language.

4.4 The p-Machine (optional)

When Pascal is translated into the intermediate code, p-code, the semantics of Pascal statements are translated into actions of a machine, the p-Machine. P-code statements are designed to invoke the possible actions of the p-Machine. For the translation of Pascal into p-code, it does not matter whether p-code will be executed directly in hardware. If p-code is further translated into machine instructions of a machine that is not a p-Machine, then the p-Machine that is the target of the translation is a virtual machine. The apparent operation of a virtual machine is the same as that of a hardware version, so an understanding of the p-Machine is still a guide for correct translation. To understand the p-Machine is to understand an elegant application of stack ideas.

A discussion of the p-Machine that is sufficient to program in p-code or to serve as the basis for a discussion of some of the aspects of HLL (High Level Language) translation is found in Part II, section G.

4.5 The Use of Stacks in Simulations (optional)

A stack may appear in a variety of guises--as a (physical or geometrical) model, as a formal data structure, as a programming technique, and in general as an idea. A program written to emulate the behavior of some process that apparently involves a stack may not actually use the data structure STACK.

Suppose a program models the conceptual behavior of the stock of canned flan on a grocery shelf. (Flan is custard in intimate contact with carmelized sugar.) A simple model is a stack of cans on the shelf. The public's desire for flan is sporadic, but it may be modeled either by a distribution derived algebraically from a uniform variate or by one derived from an experimentally measured PDF (Probability Distribution Function--see Part II, section A). In either case, a number of strategies for restocking the shelf are possible, each having costs associated with it.

If there is more than one stack of cans, the withdrawal, distribution, and restocking strategy between them determine such values as the mean number of cans on the shelf (in the stack or stacks). Pursuit of the many options is left to the exercises. Deeper questions of verification and the validity of a program model for the restocking problem are left to a course in simulations and models.

If the goal of the simulation is to determine the mean stack depth, the "stack" can be modeled simply with a counter. Each withdrawal decreases the count by one; restocking increases it by the number "added to the stack." Statistics may be gathered in several ways during the process to collect measurements of the stack depth.

Instead of mean depth, the desired information may be mean length of time that a can of flan spends on the shelf. (The carmel may crystallize after some length of time, causing rejection or dissatisfaction--flan has an estimated shelf life.) To determine mean shelf-time, a stack of can-records may be used, each marked with the (simulation) time at which they were stocked. Time passage statistics may be recorded when a can is withdrawn. A meaningful comparison study of "front-loading" and "rear-loading" restocking strategies may then be made from mean shelf-times. (Note that "rear-loading" defines a nonstack, a queue in fact--a structure studied in Chapter 5.)

Program PG4.5 is appropriate at this point.

4.6 Backtracking with Stacks (optional)

Inexperienced cave explorers sometimes play out string as they trace out the twists, turns, branches, and interconnections of a cave. The string allows them to backtrack--to return to a previous branch point by gathering in the string and hence retracing their path backwards. The call . . . return model of subprogram invocation (as well as its more sophisticated variates), is an example of backtracking that uses a stack rather than string for a temporary memory.

Stacks can be used to backtrack during traverses of some of the structures yet to be introduced, in particular the binary tree of Chapter 8. Carefully structured traverses of data structures play a role in artificial intelligence, and some of them use backtracking. A very simple model of an "intelligent" behavior requires no background except ARRAY and STACK: this model of how to explore a maze with the aid of a backtracking stack is found in Part II, section H.

Summary

Stack behavior is that of a LIFO (Last-In-First-Out) collection. Formally STACK is a structure in which only the top, last-entered item is accessible. Like a list, it is dynamic, growing with insertions and shrinking with deletions. Unlike a list, interior items can only be reached by repeated removal of the top item. A stack does not require a traverse.

In all its forms, a stack is managed by CREATE, PUSH (insert), DELETE, and TOP (value return). All of these are O(1) operations, and so wherever a stack is appropriate it is highly efficient.

STACK may be chosen as a data structure because it models common behavior. When chosen as a deliberate design basis, it is used to resequence a stream of items or because it is an immediate structure that is very easy and efficient to use.

Stacks can be supported as lists restricted to insertion and deletion at the head only. The nodes can be linked records, linked with pointer variables or indices. A direct form of array support that is attractive and common because of its simplicity does not involve links, only an index Top. It does, however, have the universal limitation of array-supported structures: it can be asked to grow out of its array.

The stack structure has had a profound influence on computing; in the theory of computing; and in the design of language translators, processors, and computing languages. Processors and languages not only incorporate the concept of a stack as a tool, but some have also made it a central feature of their design.

Applications of stacks that are treated in the chapter include a stack-directed sorting scheme, the game Calculations, reverse Polish notation, the language FORTH, the p-Machine used by some versions of Pascal, and the use of stacks in simulations. Extended discussions of FORTH and the p-Machine are found in Part II, sections F and G.

Stacks appear in most of the later chapters. In section 7.2 stacks play a central role in describing variations of the programming solutions of a problem. In a number of places they are used to help rearrange a sequence of items.

A realization of a stack manager (in Pascal) is found in Part II, section E.

Exercises

Exercises immediate applications of the text material

E4.1 Trace the growth and decay of the stack Post managed by function DoPostFix of section 4.2.1 for the following token streams:

3 5 * 6 + &

3 5 6 * + &

3 5 6 + * &

3 5 + 6 * &

E4.2 Trace the behavior of the stack OpStack and the list PostX as determined by procedure IntoPostFix of section 4.2.2 on the following:

a * b + c &      (a + b) * c &

a + b * c &      3 + ((a + b) * c) + d &

a * (b + c) &    ~(a + b) * (c + d) &

Problems

Problems not immediate, but requiring no major extensions of the text

P4.1 The precedence of operations is not the same for all languages. Find three infix expressions (possibly including Boolean operators) that are legal in both of two languages but have distinct postfix forms. (BASIC and Pascal are worth examining in this context.)

Programs

Programs for trying it yourself

PG4.1 Write a program to play the game Calculation of section 4.1.3, using stacks implemented in one of the following ways:

(a) pointer-linked records,

(b) an array, or

(c) an array-based linked list.

This is intended to be used for up to three program assignments.

PG4.2 Write a program that uses the function DoPostFix of section 4.2.1 to evaluate a sequence of postfix expressions.

PG4.3 Encase procedure IntoPostFix of section 4.2.2 in a program that accepts infix expressions and displays their postfix equivalent. (Note that this program may be combined with PG4.1 to evaluate infix expressions.)

PG4.5 Write a program that simulates a flan-can row of a grocery shelf that holds 15 cans. Withdrawals are to be generated at random times with any of the distributions discussed in section 1.5. Restock at the front (top) when there are fewer than three cans on the shelf, and determine the mean-age-since-stocking of the cans on the shelf after 100 cans have been withdrawn. Run again with restocking at regular intervals.

Projects

Projects for fun; for serious students

PJ4.1 Write a program that uses two or more stacks to sort an input stream of up to 50 integers.

Go to Chapter 5 Return to Table of Contents


Data Structures: CHAPTER 5: QUEUES

 Return to Table of Contents

CHAPTER 5: QUEUES

Waiting lines of people often form at banks and ticket booths, and lines of cars form at freeway exits. Waiting lines occur, in fact, whenever and wherever the number of demands on a resource in some time span exceed the capacity of that resource. A cashier serves only one customer at a time, and the number of lanes in an exit ramp limits the number of cars that can pass a given point at the same time. The pattern of entrance to and exit from a waiting line is that entities (such as people or cars) are served in the order in which they arrive. A waiting line with this pattern of entry and exit from it is said to be a queue, a FIFO list. FIFO, (First-In-First-Out) is in contrast to a LIFO, (Last-In-First-Out) list, a stack.

The effect of a waiting line at a resource is to smooth out an arrival sequence, the times of arrival of entities. Items remain in the queue until the server (the resource) can process them. Items arrive at the rear and leave at the front, rather than arriving and leaving at the top as they do in a stack.

Suppose that jobs tagged by letters W-A-I-T-S arrive at a milling machine from other stations in a machine shop at irregular intervals, but the milling process takes three minutes for each job. If the times of arrival from start-up are 5, 6, 8, 9, and 13, then a queue would form. Figure 5.1 shows snapshots of the queue contents at arrivals and departures.

Figure 5.1

Arrival times of 5, 8, 11, 12, and 14 produce output at the same times, but the arrival sequence 5, 8, 12, 13, and 14 does not do so because the server is idle part of the time.

Now suppose that the service times are 2, 1, 5, 5, and 2. Figure 5.2 shows the new snapshots. The distribution of arrival times and the distribution of service times both affect the queue length and the amount of time that an entity spends in a queue. This is queueing behavior, a basic model from which the idea of a queue is derived. Such behavior is central to simulations and other applications, and is treated in more detail in section 5.1. Waiting lines will occur, and some feel for where and when they do is an aid in the design of programs.

Figure 5.2

A queue can be stored and managed as a data structure QUEUE. Like ARRAY and STACK, QUEUE is not concerned with the complexities of the flow of a sequence of items through it, but only with access to cells. Queueing behavior is to be distinguished from the data structure QUEUE, which is simpler to deal with than the behavior it models.

5.1 Queueing Behavior

In a program, items seldom wait for a specific time to arrive, but they do wait for some process to complete (as do drivers at a traffic-light controlled by pedestrians). An input routine waits for a user to hit one or more keys, perhaps saving a string of keyed characters until they form a recognized token before allowing some other process to be initiated. On the other hand, processing of one input token may take longer than the entry of the next one. Systems that queue inputs and those that sometimes treat the keyboard as a nonentity have differing psychological impacts on the user.

Examples of queueing abound in the operating systems and in the hardware of computing systems. For instance, in a time-sharing system some jobs wait for service from a disk drive or a printer while another job is being processed by the CPU (Central Processing System). The common use, however, is simply to store items, in order, for later processing.

Queueing is fundamental to some of the applications of stacks in Chapter 4. The conversion of an infix expression to a postfix expression in section 4.2.2, for example, creates a queue called PostX--the postfix expression itself. The token stream PostX is a queue because tokens are both inserted to it and removed from it in the order in which they are encountered during execution of a program or during translation in a compiler.

From the viewpoint of processing tokens, stacks and queues are used in conjunction in a stack machine only to save a token and temporarily delay the processing of operands encountered in sequence in the token stream. In FORTH, the programmer creates the token queue. In Pascal, a compiler that translates to p-code creates the queue. In both these languages, the preparation of a queue is crucial to the successful operation of a stack. The stack eliminates the necessity to look ahead an arbitrary (and varying) number of predetermined tokens to decide when to insert the current token into the processing queue. Use of a stack allows the rest of the processing applied to the token stream to be done sequentially.

Queueing behavior is common in machine shops (and other manufacturing facilities), where jobs tend to back up somewhere: at a lathe, at a milling machine, at any work station where job requests arrive at a rate faster than the throughput determined by the mean service time of the work station for such jobs. Queueing behavior also occurs in the flow of goods from manufacturers to customers.

Machine shops, marketing systems, and computing systems are examples of networks where jobs (or other entities) flow from one server to another and where jobs may spend more time waiting than being served. Service is distributed in such systems, as it is in service networks of all kinds. Where service occurs in a network through which requests for service flow, so does queueing behavior. Analysis of queueing networks, done both analytically and operationally, is based on the behavior of an isolated queue, and that behavior is the pattern for the data structure QUEUE.

5.1.1 The Queue Structure

5.1.1 The Queue Structure

A data structure that models queueing behavior must do so through general data structure operations tailored to that behavior:

Queues need to be created with access (by pointer or index) to both a head (front) and a tail (rear), because insertions take place at the rear and deletions at the front. The formal operation which gives birth to a queue is CREATE.

A queue becomes empty at any time when there have been as many departures as arrivals. The formal (Boolean) operation that checks the viability of a queue is EMPTY.

A queue grows iff a single item is inserted into its rear position. No way of assigning values directly to queue items is available in a pure queue. The formal operation that inserts a value into a queue is INSERT.

The only retrievable value of a queue is the value of its front item. The formal operation that returns the value of the front item is FRONT.

The front item must be removed from a queue if the values of the other items are to be retrieved. The formal operation that removes the front item from a queue is DELETE.

The behavior of a queue (as a data structure) is entirely described, however it may be implemented, by CREATE, EMPTY, INSERT, DELETE, and FRONT.

A queue changes size with insertions and deletions and so is dynamic. It is created without knowledge of its future size and so is blind. It is sparse because the collection of potential entries is generally much larger than its actual entries at any given time. All of these thkngs make the pointer-linked list of records a natural support structure for the queue.

A queue provides direct access, although only at the front and rear, and that access should be efficient. Hence, if the maximum size of a queue is predictable, an array is a natural structure for queue support.

5.2 Queue Implementation and Management

A formal structure QUEUE defines a data management policy, independent of implementation. Implementation, however, affects program writing. To contrast the two natural implementations, suppose that two queues are to be managed by a program: Q1, implemented with a linked list, LinearQ, and Q2, implemented with an array, aQueue. Figure 5.3 depicts the two queues.

Figure 5.3

The scheme depicted in Figure 5.3 for the implementation of a queue in an array has a peculiarity: the index Front is one less than the actual index of the front value. The neatness of this scheme becomes apparent when queue operations are developed as procedures.

As two items are deleted from LinearQ and a node of value 2 is inserted, snapshots of Q1 would look like those in Figures 5.4 and 5.5. (The terms Head and Tail have been replaced by Front and Rear as a reminder of the waiting-line model of queueing behavior.) A queue is a list structure for which insertion is always done with InsertTail and deletion with DeleteHead. In contrast, when the list of Figure 5.3 is treated as a stack, insertions are made with InsertHead. For this example, the difference would be the placement of the value-2 node at the front of the list, rather than at the rear. This difference is significant; queueing behavior is quite distinct from stack behavior.

Exercise E5.1 is appropriate at this point.

Figure 5.4

Figure 5.5

When this same set of operations is applied to Q2, a quirk of the array implementation of a queue becomes apparent: deletions and insertions move a queue along in its substrate array: they have an inchworm effect. The effect on Q2 of DELETE, DELETE, INSERT 2 is shown in Figure 5.6. No new insertions can be made to Q2 (in this implementation, with the value of Max as shown) without shifting the entire queue Q2 within the substrate array aQueue. The array index Rear has reached its limit Max, but the array aQueue is not filled by Q2. It is convenient to develop a procedure, ShiftQ, to deal with the shift.

Figure 5.6

5.2.1 Queue Management Algorithms

5.2.2 On the Use of Queues

5.2.1 Queue Management Algorithms

A comparison between the management routines for Q1 and Q2 can now be made. (An assumption is made that with LinearQ do . . . is in effect where it is convenient for Q1. The identifiers InsertRear and DeleteFront replace InsertTail and DeleteHead to emphasize the specific application to queues.) For comparison, the list form of InsertRear in Table 5.1 accepts values instead of nodes.

Values rather than nodes are inserted into LinearQ to emphasize similarities with routines applied to aQueue. Sentinel lists, convenient for some applications, require little change in the procedures.

Table 5.1

                  Q1                                      Q2

----------------------------------------------------------------------------------------------

                                        CREATE

declarations                     {O(l)           declarations                    {O(l)

LinearQ.Front   NIL                              Front  0

                                                Rear  0

----------------------------------------------------------------------------------------------

                                        EMPTY

function Empty(LinearQ)          {O(l)          function Empty(aQueue)           {O(l)

  if Front = NIL                                  if Rear = 0

     then return TRUE                                then return TRUE

     else return FALSE                               else return FALSE

     endif                                           endif

  end {Empty                                      end {Empty

----------------------------------------------------------------------------------------------

                                        INSERT

                                                procedure InsertRear(aQueue, i)  {O(O(ShiftQ))

                                                  if Rear = Max

                                                     then ShiftQ(aQueue)

                                                     endif

                                                  if Rear = Max {still

procedure InsertRear(LinearQ,i)  {O(l)               then {deal with a full-array error

  NEW(iNode)                                         else Rear   Rear + 1

  iNode  .Value    i                                       aQueue[Rearl   i

  iNode    .Link   NIL                                endif

  if Empty(LinearQ)                               end {InsertRear

     then Front   iNode

     else Rear   .Link   iNode

     endif                                      procedure ShiftQ(aQueue)  {O(Max)

  Rear   iNode                                     if Front > 0

  end {InsertRear                                     then for k = 1 to Rear-Front do

                                                             aQueue[k]   aQueue[Front + k]

                                                             next k

                                                           Rear    Rear - Front

                                                           Front    0

                                                      endif

                                                   end {ShiftQ

-----------------------------------------------------------------------------------------

                                        DELETE

procedure DeleteFront(LinearQ)   {O(l)          procedure DeleteFront(aQueue)      {O(l)

  if Empty(LinearQ)                               if Empty(aQueue)

     then {deal with this error                      then {deal with this error

     else p    Front                                  else Front   Front + 1

          Front    Front    .Link                           if Front = Rear

     endif                                                    then Front   0

  DISPOSE(p)                                                       Rear   0

  end {DeleteFront                                            endif

                                                     endif

                                                  end {DeleteFront

-----------------------------------------------------------------------------------------

                                        FRONT

function FrontView(LinearQ)      {O(l)          function FrontView(aQueue)         {O(l)

  if Empty(LinearQ)                               if Empty(aQueue)

     then {deal with this error                      then {deal with this error

     else return Front   . Value                      else return aQueue[Front + 1]

     endif                                            endif

  end {FrontView                                  end {FrontView

Exercises E5.2 and E5.3, problem P5.1, and programs PG5.1 and PG5.2 are appropriate at this point.

A sequence of I (insert) and D (delete) commands that manage an input file as a queue are processed in essentially the same way as they were for a stack at the end of section 4.1.1.

Note: A realization in Pascal of the command loop as a "queue machine" is to be found in Part II, section I.

A queue can be displayed as a list by a simple variation of LT in section 3.3.2; but, as a queue, there is no access to internal nodes. One way to list a queue is to display the front node, delete it, and insert it back into the queue:

procedure ListQ(q)               {O(n)

  if Empty(q) then return endif

  Start    q

  repeat

    DISPLAY(Front(q))

    Node    q

    Delete(q)                    {Delete should not include DISPOSE

    Insert(Node)

    until Node = Start

  end {ListQ

5.2.2 On the Use of Queues

Queues are sometimes chosen to model a pervasive natural system, the waiting line. In many systems waiting lines are unavoidable. The effects of waiting on a time sequence of entities passed through a queue can be complex, but the data structure QUEUE is relatively simple. It does, however, take more care than does STACK. In pointer-linked record form, access to the tail node (the rear node) must be provided to make INSERT an O(1) operation. When supported by an array, not only is the maximum number of entries limited, but the inchworm effect can also occasionally force an O(n) shift operation of the queue within its array. (This problem can be alleviated with the use of circular lists as a support structure, as described in section 5.3.) The queue is more complex to manage, as a data structure, than is the stack.

The queue is a dynamic, immediate structure, and it can make efficient use of time and space. Above all, a queue is a retainer--it retains the order in which items enter it and produces them as needed.

Note that the word "queue" is commonly used to refer to structures more general than QUEUE. Generalizations of QUEUE are outlined in section 5.4, introduced in section 8.5.2, and applied in Chapter 10. One use of QUEUE is to provide a basis for its own generalizations.

5.3 Circular-Track Queues

Nodes linked into a cycle, like the list in Figure 5.7, were introduced in section 3.7:

Figure 5.7

A circular list may be maintained as a queue by the proper choice of INSERT and DELETE operations: InsertAfter and DeleteBack, both discussed in section 3.7. The result is a queue supported by a circular list, but it is commonly called a circular queue. A queue is said to be a linear structure, but it is actually a point structure, because only two points of a queue exist for the outside world, whatever its implementation. (If the queue is a two-point structure, then the stack is a one-point structure.)

In a queue maintained in an array, circularity can be transferred to index-management in the substrate array. An awkwardness inherent in the implementation of a queue with an array is that the queue array may not be full even though the cell of maximum index is in use--hence, the procedure ShiftQ in section 5.2. This inefficiency can be resolved in an array dimensioned a[0 . . Max - 1] with the help of modular arithmetic. In that case

(1) k    (k + 1) MOD Max

calculates a remainder upon division by Max, and so has the same effect as:

if k = Max  1

then k    0

else k    k + 1

endif

The effect is to bend the array itself into a (virtual) ring, shown in Figure 5.8.

Figure 5.8

The assignment (1) is a clockwise index shift, no matter what the value of k, 0 [image] k [image] Max - 1. This scheme forms a circular array without regard to the structure it supports, but it is normally used in support of a queue.

Now Front and Rear can both be moved clockwise when a deletion or addition is made, as shown in Figure 5.9.

Figure 5.9

If no more than Max - 1 items are allowed into the queue, then (Front = Rear) signals that the queue is empty; and, after (Rear [image] (Rear + 1) MOD Max), (Front = Rear) also signals that the array is full. If Max items are allowed into the queue, an additional flag needs to be carried in order to distinguish between the empty and full conditions. One approach (of several) is to use the flag Some, reset TRUE during insertion, and set FALSE when deletion produces an empty queue.

The queue operations become:

function Empty                          {O(1)

if (NOT Some) AND (Front = Rear)

then return TRUE

else return FALSE

endif

end {Empty

procedure InsertCQ(x)                  {O(1)

if (Front = Rear) AND Some

then {report full array

else Rear    (Rear + 1) MOD Max

a[Rear]    x

Some    TRUE

endif

end {InsertCQ

procedure DeleteCQ                     {O(1)

if Empty

then {deal with this error

else Front    (Front + 1) MOD Max

if Front = Rear

then Some    FALSE

endif

endif

end {DeleteCQ

Problem P5.2 is appropriate at this point.

5.4 Generalized Queues

Queueing behavior is exhibited in structures that are not queues in the strict sense. One of these is called DEQUE (double-ended-queue). This structure allows both entry and exit at either end, but nowhere else. Why it is not called a double stack is a mystery, since that is at least as accurate a description. If two nodes are inserted at one end and removed from the other, they will exit in queue order. If they are both removed from the entry end, then they exit in stack order. If they are removed from opposite ends, then their exit order is whimsical.

A queueing generalization of much greater utility than the deque is the priority queue. A priority queue is a structure in which nodes have a priority. This may be the key or other value, and it may even be possible to update the priority of a node. Retrieval from a priority queue is of the node with highest priority and no other. In that sense, there is one "front" node. At retrieval time, the entry order per se is of no significance.

Note: It is not uncommon for the term "queue" to refer to a structure that proves to be a priority queue on close examination. This meaning may even be more common in the general literature than that of pure queue.

A priority queue may be supported in many ways. One such is simply a sorted list, and the highest priority item in the queue is the head node. Another priority queue is an instance of QUEUE in which priority is the negative of the entry order (or the inverse of it). Another priority queue is an instance of STACK in which priority is given to the most-recently entered item (the entry order itself is the priority measure). There are many other ways to support priority queues.

One application of priority queues is as a retainer for events that have been scheduled to happen. Discrete-event simulations are those that move from action to action, and in such a simulation, this holding structure is called an event queue. An example of an event-queue application is the doctors'-waiting-room model treated in section 5.7.

Note: A Pascal program that simulates an airport customs process is to be found in Part II, section  K. The heart of the simulation is the management of an event queue.

A common way to support a priority queue so that insertion and retrieval are efficient when entry order does not determine the priority is a heap, discussed in Chapter 8. A major application of priority queues is made in the treatment of graphs in Chapter 10.

In some sense, this book is a bridge between the array of introductory programming and variations of priority queues used in practice.

5.5 Queues in Hardware     (optional)

A buffer containing data to be written on a printer or a disk is usually generated at a different rate than it is serviced. As a consequence, either these jobs are retained in a queue, or some process must become idle until another is complete. Such queues may be managed by either software or hardware, since both can perform the same essential logical operations.

Some queueing within computing systems is normally the domain of hardware, rather than software. Instructions in a machine are fetched, then executed, in an endless cycle. Both the execution phase and the fetch phase can exhibit overlap in which two or more instructions are being processed in some way at the same time. Specifically, these two overlap modes are pipelining and the use of cache memory. They are both associated with queueing behavior.

Note: Discussions of pipelining and cache memories are to be found in Part II, section  J.

5.6 On the Simulation of a Doctors' Waiting Room     (optional)

Many physical processes are expensive or impractical to observe directly, and certainly they can only be observed in operation during conditions that actually exist. Almost any industrial process, almost any natural biological, physical, or social environment, and almost any large computing system is quite complex. For example, building a large chemical plant based on a newly discovered sequence of reactions without knowing that it will work on a large scale much as it does in the laboratory would not be wise. In such a case it is worthwhile to model such processes and to study the model extensively, with the assumption that the behavior of the model is a valid surrogate for the behavior of the process itself.

If suitable approximations to a process behavior can be constructed mathematically, it is often cost-effective to use mathematical software to study that behavior. An alternative, sometimes the only one, is the development of a simulation of the process. A simulation is a program that imitates the behavior of a process, in contrast to an analytical model, which describes it.

When simulating a system involving queues it is not necessary to use queues as a data structure in the simulation program. It is natural, however, and may be efficient in several ways. Queueing behavior may also be effected with structures that are not strictly of type QUEUE, such as a generalized (priority) queue. On the other hand, queues are often involved in simulations that do not themselves emulate queues, as a scheduling structure, the event queue. In practice, the "event queue" is usually a priority queue.

The conjunction of these ideas can be illustrated with a model of a doctors' office that does not explicitly involve a queue, but does use queueing ideas, and that is programmed with an event queue. Some familiarity with statistical ideas is helpful (see Part II, section  A).

The office is serviced by five doctors and several support staff. Patients arrive in some distribution during an eight-hour day, are treated after a while, and leave. The treatment-with-wait generates a mean-service-time, mst, as a whole, shown schematically in Figure 5.10.

Figure 5.10

At random intervals one of the doctors is called away and returns after a random length of time. The number of services can vary from 0 to 5. One question that can be asked is: how many chairs are needed in the waiting room?

Four distributions are involved in this model:

      ia, the interarrival time, or time between patient arrivals. (This would generally be modeled as an exponential distribution: ia   [image]  - (1/  [image])   [image]  1n (Rand).

      fixup, the mean service time for one doctor to treat one patient (possibly exponential again)

      out, the time between doctor exits (possibly uniform, Rand)

      back, the length of time a doctor will be gone when called away (perhaps a normal distribution)

These distributions can be generated either as needed or as four sequential files, to be stored and retrieved as needed. They are, in effect, queues. At any given moment, the next event might be a patient arrival, a patient exit, a doctor exit, or a doctor arrival. Each event can be described by a procedure that is simple or elaborate, depending upon how much detail is defined by the model and how much information is to be gathered for statistical analysis.

The event procedures need to be scheduled. One reasonable approach is to take care of the next (in time) event next.

Simulation programs often deal with event scheduling by having each event-procedure determine when the next event of its type is to occur, by generating it or reading it from a file and then generating a record to be added to the event queue. This record is called an event. The event-record contains the time at which it is to take place and other attributes of interest. Above all, it can produce the next event--the one scheduled to happen soonest--considered to be the event with the highest priority. In terms of the structures studied so far, the event queue can be maintained as an ordered list. Event-procedures process event-records, selected for processing by their position on the event list and their attributes.

It is necessary to resolve collisions, times occupied by more than one event, as well as other aspects of the program that affect the validity of the model. (The validity of a model is how well its behavior conforms to that of the system being modeled.) For example, is it valid to allow a doctor to leave while treating a patient, or must the service be completed? (Are there any emergencies?) Should patient arrivals be handled before doctor arrivals in collision with them? (This affects the number waiting and hence the statistics generated by the program.) Since collisions can occur, the priority used to sort events to determine their order of processing cannot be simply their scheduled event time.

Suppose that in a program Infuse, which simulates the doctors' office, the event list is named eList and contains records of the form:

eListype = RECORD

Time : INTEGER

Tag  : INTEGER {patient ID

Type : ARRAY[1 .. 8] OF CHAR

Info : {a record of attribute information

Link :    Listype

END  {eListype

Types of events would reasonably include but not necessarily be confined to, SadSack (a patient arrival), CoolOff (a patient moves to an examination room), Aok (a patient departure), StatOut (a doctor departure now), AgainSam (a doctor arrival). The time might represent minutes from the start of the day. An integer Avail might be used to keep track of the number of doctors currently available, and Waiter the number being served.

Suppose that the current event list is of the form shown in Figure 5.11. When the next record of eList is retrieved, it is found to be at (simulation) time 106. When it is removed for examination the result is the list of Figure 5.12:

Figure 5.11

Figure 5.12

When the retrieved record is examined, it is found to be a patient arrival (SadSack). As part of the execution of the procedure that handles patient arrivals, at least one new record will be scheduled:

      The next SadSack record is scheduled at the time determined by the next ia value.

If patient arrivals have precedence over patient departures, but not doctor departures, then various values of ia that produce collisions would instead yield the snapshots shown in Figure 5.13.

Figure 5.13

The patient who arrived at time 106 does not disappear -- even though the arrival event disappears. The arrival procedure increments a counter representing the "queue" of patients in the waiting room. If individual patients are to be tracked through the process, then the waiting room may become a true queue structure, and a record is added to the rear of that queue when SadSack events are processed. Patient attributes such as illness type might be chosen at that point, if they are to affect the simulation. If so, service times must be split into categories.

A simulation program having a reasonable degree of validity involves many decisions, some hidden deep in the coding. Writing a simulation program is time-consuming and hence expensive; worse, it is very difficult to establish validity. Nevertheless, simulations are sometimes the most cost-effective approach to exploring a problem, but mathematical analysis should be tried first. For this reason, a great deal of effort is expended on approximate solutions to problems that are unsolved (or intractable). An alternate approach, available at major installations, is the use of simulation languages--languages written especially for this purpose. A discussion of the pros and cons of their use lies outside the bounds of this book.

Project PJ5.1 is appropriate at this point.

Note: A realization in Pascal of a simulation of the customs system at an airport is to be found in Part II, section  K. It is based on the use of a priority queue.

Simulation begins to be accessible with a knowledge of queues, but the crux of a simulation is the thoughtful analysis of the results. That requires a background in statistics. The simulation language SIMULA is perhaps the easiest for the user of languages related to pseudocode to learn. There are introductory texts for SIMULA, but compilers are not common.

Summary

Queueing behavior is that of a FIFO (First-In-First-Out) collection. It is to be found wherever there are waiting lines, and waiting lines occur wherever an arrival sequence and a servicing process are not synchronized, a common pattern. The time-sequencing effects of queueing can be complex, but the data structure used to model it is relatively simple. A queue has the effect of storing items until they are needed. It is a retainer.

Formally, QUEUE is a structure in which only the oldest (earliest-entry) item in the collection is accessible for viewing by FRONT or for removal by DELETE. An interior node can be accessed only by deleting the items in front of it. Like lists and stacks and unlike arrays, queues are dynamic, growing with insertions and shrinking with deletions, but they are point-like objects that hide their internal dynamism from the user.

Queues can be supported as lists to which insertion is applied at the rear (tail) and deletion at the front (head). A queue can also be supported in a direct way in a single array, using two indices Front and Rear.

The array-supported queue can reach the last cell in its array in two senses--by position and by total number of entries. The shifting required to resolve the position problem causes insertion to be O(n) in the worst case, although it is typically an O(1) operation like deletion.

The position problem for an array-supported queue can be eliminated by using circular queues, but then an extra flag is required to resolve empty and full conditions.

The queue has generalizations, one of which is the priority queue, to be explored in later chapters. Many applications of programmed queues involve the generalized forms, but pure queues have uses such as the pipelining of instructions and simulations of waiting lines.

A realization in Pascal of a "queue machine" is to be found in Part II, section  I, and a Pascal program that simulates an airport customs system is to be found in Part II, section  K.

Exercises

Exercises immediate applications of the text material

E5.1     What would the final list be if the sequence: 4 0 - 5 10 were treated as a stack with operations DELETE, DELETE, INSERT 2?

E5.2     Trace the behavior of the queue: 4 0 - 5 10, implemented with pointer variables, with the operations: DELETE, INSERT 2, DELETE, DELETE, INSERT 3, DELETE.

E5.3.     Trace the behavior of the queue: 4 0 - 5 10, implemented as an array, with the operations: DELETE, INSERT 2, DELETE, DELETE, INSERT 3, DELETE. Assume that the value 10 is initially in aQueue [Max - 1].

Problems

Problems not immediate, but requiring no major extensions of the text material

P5.1     Redesign the procedures for an array implementation of queues in section 5.2, using Front as the index of the front value itself.

P5.2     Modify the queue operations of section 5.3 so that up to (Max - 1) items may be kept in a circular queue implemented as an array, and Some is not required.

Programs

Programs for trying it yourself

PG5.1     At the beginning of this chapter, an input stream of characters was associated with both an entry time into a waiting line and a service time. Together these times determined the contents of a queue.

Write a program that accepts a sequence of triplets--for example, (W,5,2), (A,6, 1), (I,8,5), (T,9,5), and (S,13,2) --and produces a snapshot of the queue each time there is a change in it. The triples are to be (identifier, arrival time, service time) and the time at which a change in the queue occurs should be displayed with its contents.

It is not a bad idea to produce two versions of this program, one using pointer-linked records, and the other using array support for the queue. Obviously, only a few procedures should change from one version to the other.

PG5.2     A buffer B holds N blocks of data to be displayed by output device P. Programs generate blocks of output at random intervals with a mean of 11 seconds, and they are queued in B. P removes a block from B at regular intervals, every 10 seconds when there is one to remove.

Suppose that, because of overhead and storage limitations, each increment of N above 1 costs an average of $1 for each storage of a block into B. Each time a program finds the buffer full, it must be rerun, at a cost of $100. Clearly, if there is no variation in the production of blocks, N = 1 is optimal. If there is variation, blocks are sometimes generated close together, and N = 1 will exact a cost of $100.

The problem here is to write a program that experimentally determines the optimum value of N for clearly stated conditions of spread (dispersion) in the generation of blocks. A simple variation is to use a pseudorandom number generator to generate times between blocks that vary uniformly in the interval [i . . j] (for example, in [7 . . 15]). The material in Part II, section  A, will allow a much more sophisticated experiment to be designed with little additional programming.

Go to Chapter 6            Return to Table of Contents


Data Structures: CHAPTER 6: GENERAL LISTS: MULTIPLE ACCESS PATHS

 Return to Table of Contents

CHAPTER 6: GENERAL LISTS: MULTIPLE ACCESS PATHS

Structures STACK, QUEUE, and DEQUE are fundamentally simple in design. They provide immediate access to their nodes only at the ends of the list. Access to interior nodes is not part of their structure but can be gained by following a sequence of links from an end node--by making a list-walk.

For some problems it is natural to develop a solution that uses a more complex structure having multiple access paths to an interior node. The multiplicity of paths is created by a multiplicity of links leading from each node; any given node generally has more than one successor and more than one predecessor.

One class of multiply linked structures is the binary tree and its variations. The classes of tree-like structures are themes of Chapters 8 and 9. Trees are a special case of the graph, described and traversed in Chapter 10. However, some multiply linked structures can be managed with a combination of the structures discussed in previous chapters.

Several sections of Chapter 6 give different examples of multiply linked structures or applications of them. Doubly linked lists are introduced in section 6.1. Linkage patterns of interwoven lists called n-braids are introduced in section 6.2. The adjacency matrix and graph models that form much of the basis for design and analysis of general data structures are introduced in section 6.3. A survey of sparse matrix structures in section 6.4 provides the background for discussion of two structures for them: Main Diagonal Circulation in Part II, section N, and Orthogonal Lists in section 6.5. Both models use circular 2-braids. The doubly linked lists of section 6.1 are used in the sparse-matrix model and in modeling dynamic memory allocation in section 6.5.

Relatively simple combinations of list structures can be used to construct alternates to the hash table structure of Part II, section B. These combinations are discussed in Part II, section L.

Most of the algorithms that walk a general list structure use stacks, either implicitly (to support recursion--see Chapter 7) or explicitly, in analogy with the maze-walk of Part II, section H. An algorithm that walks a general list without using stacks is introduced in section 7.7 and discussed in Part II, section M.

6.1 Doubly Linked Lists

Linearly linked lists can be altered by fast operations when the INSERT and DELETE operations are proscribed, as they are in the STACK and QUEUE structures. In contrast, if additions and deletions are to be made at arbitrary positions within a list, then the fundamental operations become O(n) instead of O(1) because a search is required to locate an interior node.

When a node to be deleted from a linearly linked list has been located, its removal requires a pointer to both of its immediate neighbors in the list--its predecessor as well as its successor. The same requirement makes some operations, such as the DeleteTail procedure of section 3.3.4, awkward. In an operation on a linear list where the predecessor of a node is required, the predecessor must be located with a search. One way to enhance access to neighboring nodes within a list is to weave two linked lists together to form a doubly linked list, as shown in Figure 6.1 :

Figure 6.1

Procedures DeleteHead, InsertHead, and InsertTail are straightforward adaptations of single-linked list routines. With a simple return as a response to an empty list, DeleteTail is:

procedure DeleteTail(dList)         {O(1)

if Empty then return endif

e    Tail    .Forth                {NIL or u

Up    Tail    .Back

if Up = e

then Head    e                {singleton response

else Up    .Forth    e

endif

DISPOSE(Tail)

Tail    Up

end {DeleteTail

The deletion of a node from the middle of such a list is relatively straightforward. To remove an already located p, the deletion procedure might follow the schema in Figure 6.2.

Figure 6.2

The relinking can be done with:

procedure UnDouble(dList,p)   {O(1)

if Empty then return endif

Up    p    .Back

Down    p    .Forth

e    Tail    .Forth             {NIL or u

if Down = e

then DeleteTail(dList)

return

endif

if Up = e

then DeleteHead(dList)

return

endif

Up    .Forth    Down

Down    .Back    Up

DISPOSE( p)

end {UnDouble

An assumption made in UnDouble is a necessary condition for a list to be doubly linked; for any interior node (not Head or Tail):

p = p    .Forth    .Back = p    .Back    .Forth

Program PG6.1 is appropriate at this point.

Double links are especially efficient in circular lists, such as that in Figure 6.3. There is no distinguishable position in such a list, although Q is the distinguished node. It can be convenient to let Q float from node to node in a sequence of "search and process" operations, pointing to the last (distinguished) node associated with an operation. Because such a ring has no natural head or tail, treating it as a stack or queue requires care.

Figure 6.3

Exercises E6.1 and E6.2 are appropriate at this point.

6.2 n-braids

The doubly linked assumption can be relaxed for more general linked lists, as shown in Figure 6.4. The Forth list defines the sequence 1,2,3,4. The Back list defines the sequence 4,2,3,1. The independence of the Back and Forth linearly linked lists reintroduces the need to search for predecessors. The structure in Figure 6.4 is fundamentally two interwoven lists, a 2-braid, not a doubly linked list. A 2-braid can be used to simultaneously sequence one set of nodes in two ways. In general, an interweaving of n linear lists on one set of nodes (an n-braid) can be used to sequence the nodes in n ways without duplicating the nodes.

Figure 6.4

The nodes of an n-braid contain link fields associated with more than one list. Procedures to be applied to such nodes must either be written to process a specific braid or be generalized in some way. One way to generalize such procedures is to use an array of link-fields: Braid[1 . . n]. Then an index k is used to specify the braid link and hence to select braid k to be processed. It then requires very little alteration of standard list procedures to construct general braid procedures. In this chapter, we will assume that the following are available:

RingIn(b,p,q) inserts node p as the successor to node q into braid b, shown in Figure 6.5.

Figure 6.5

RingOut(b,q) deletes the successor of node q from braid b, shown in Figure 6.6.

Figure 6.6

The search for q in both RingIn and RingOut may depend on the structure within which a braid lies and the starting point of the search.

Note: Insertion into and deletion from an n-braid involves location of the node in each braid. Disposal of a node must be delayed until it is deleted from all its braids.

Problem P6.1 is appropriate at this point.

6.3 Matrices and Graph Adjacency

The systems to which combinations of data structures can be applied are sometimes complex. Two models that are powerful simplifiers and organizers are commonly applied to such systems to make them accessible to analysis and design. These two models are the matrix and the graph. They are introduced very briefly here as necessary background for discussions in later parts of the book.

When the arithmetic operations ADDITION and MULTIPLICATION and their interactions are defined for tables, the tables become matrices. There is a strong analogy between the definition of such mathematical structures and computing structures such as ARRAY, STACK, and QUEUE. The operations of the structure MATRIX can even be defined algorithmically, in which case the mathematical structure and the data structure correspond very closely to each other.

The algorithmic definitions of the matrix operations ADDITION and MULTIPLICATION are:

For two n X m tables, A[1 . . n,1 . . m] and B[1 . . n, 1 . . m], added to form C[1 . . n,1 . . m], ADDITION is defined by:

function AddMatrix(A,B)           {O(n2)

for i = 1 to n

for j = 1 to m

C[i,j]    A[i,j] + B[i,j]

next j

next i

return C

end {AddMatrix

For A[1 . . n, 1 . . p] and B [1 . . p, 1 . . m], multiplied to form the product C[1 . . n, 1 . . m], MULTIPLICATION is defined by:

function MultiMatrix(A,B)           {O(n3)

for i = 1 to n

for j = 1 to m                 {C[i,j] =    A[i,p] X B[p,j]

Sum    0

for k = 1 to p

Sum    Sum + A[i,k] * B[k,j]

next k

C[i,j]    Sum

next j

next i

return C

end {MultiMatrix

Tables can be used to describe many systems, physical and conceptual. It is uncanny and satisfying that matrix operations often model the behavior of such systems. As an example, many systems can be modeled geometrically as a graph: informally, a collection of nodes joined by edges, as in Figure 6.7.

Figure 6.7

Graphs participated in the genesis of topology, may be involved in the demise of some management schemes, and certainly picture all networks, including those used to approximate topographies in models of agricultural regions.

Graphs depict the connections called edges between entities called vertices; that information may also be tabulated. The table associated with a graph is called its adjacency matrix. To tabulate a graph:

Number the nodes (as done in Figure 6.7) and associate both a row and a column of table G with each node.

If there is an edge connecting node i with node j, then G[i,j] [image] 1, otherwise G[i,j] [image] 0.

Figure 6.8 shows the graphs from Figure 6.7 in tabulated form.

Figure 6.8

One way to interpret these tables is that they select the pairs of nodes connected by a one-step path. A chain in a graph is a sequence of edges that lead from one node to another. Technically, if an edge is defined by the pair of nodes it connects, like {1,2} for the edge e in (a), then a chain is a sequence {N1,N2}, {N2,N3}, . . . , {Nn-1,Nn} that joins node N1 with node Nn. The charming effectiveness of mathematics appears when the graph tables are treated as matrices and raised to powers as Gc2 is in Figure 6.9.

Figure 6.9

An entry Gc2[i,j] is the number of chains of precisely two steps that joins Node i to Node j. Similarly, Gc3 tabulates the number of paths of length 3 that join pairs of nodes in the graph of example (c). Hence the powers of a graph matrix provide valuable information about the system modeled by the graph. This is not the only connection between matrices and graphs, but merely an illustration of the ubiquity of matrix operations.

Chapter 10 deals with graph representations as data structures. In this chapter, it is the mathematical nature of a matrix that determines the structure used to represent it; and that structure differs from those used in Chapter 10.

Exercise E6.3, problem P6.2, and program PG6.2 are appropriate at this point.

6.3.1 Directed Graphs

6.3.1 Directed Graphs

There is no need to restrict the linkage patterns of data elements to linear structures or even to combinations of linear structures. For example, four nodes may be linked as in Figure 6.10.

Figure 6.10

A simpler geometrical model for this pattern is a digraph (directed graph), which differs from a graph precisely because the edges possess direction, as shown in Figure 6.11.

Figure 6.11

Like a graph, a digraph model has an adjacency matrix, although a digraph matrix is not symmetric about the main diagonal (M[i,i], 1 [image] i [image] Max) in contrast to the adjacency matrix of a graph. An entry of 1 in M[i,j] for a digraph signifies a directed arc (an edge) from node i to node j. An edge is thus an ordered pair, (i,j). For the example above, the adjacency matrix M is shown in Figure 6.12.

Figure 6.12

Since edges in a digraph are directional, so are sequences that follow them from one vertex to another. The analog of a graph chain is a digraph path. It is still true that Mk[i,j] is the number of k-step paths from [i] to [j], as shown for this example in Figure 6.13.

Figure 6.13

Digraphs arise as models of such diverse systems as dominance relations in sociology and management, communication networks, and relations between data elements in the flight and ticket information of an airline. Data can be structured as a collection of nodes linked in the same pattern as the natural digraph model. Such a node structure is called a digraph list, also a general list.

Note: It is the digraph rather than the graph that corresponds most closely to data structures formed with pointers, because they are directional. In data structures that model graphs, two links represent each edge, one for each direction.

The design of fundamental operations like INSERT and DELETE become very application-dependent and sometimes difficult in a general list. A search through a digraph structure appears to be forbidding; nevertheless it is required in some applications. Hence there are algorithms for visiting each node of a general list, and a node may be processed in any desired way during the visit. One such algorithm is to be found in Part II, section M. A different visitation algorithm for general lists is the subject of section 7.7, where it serves as an example of a recursive algorithm. The traverse of a (restructured) general list can also be patterned very closely after the treatment of graphs in Chapter 10.

6.4 Sparse Matrix Representation

In many applications of graphs modeled by matrices and in other applications involving matrices, most of the entries are zero. Such a matrix is called a sparse matrix. A workable pair of definitions is that an N by N matrix is sparse if it contains O(N) nonzero entries and dense if it contains O(N2) nonzero entries.

The value of the nonzero entries in a sparse matrix as well as their location by row and column can be important, and a listing of the nonzero entries defines the zero entries by default if it is known to be complete. Any structure to which matrix operations are applied needs a row and column organization. It is nevertheless possible to trade simplicity and the independent access to a large and sparsely used storage space for complexity of structure coupled with more efficient use of storage. The trade of fast access for efficient storage is frequently made for problems that involve sparse matrices.

A number of data structures can be used to represent a sparse matrix:

1. SST (Sparse and Static Table). This structure is intrinsically static; not only is its storage allocation fixed, but it is also designed for a specific subset of the cells of the large (virtual) matrix it represents. It is excellent for some sparse table storage, as indicated in section 2.4, but difficult at best to adapt to the general situation of a changing (but still sparse) matrix.

2. MDC (Main Diagonal Circulation). The backbone of this structure is a collection of nodes, each representing a header for two circular lists: a row list and a column list. Each nonzero cell is represented by a node braided into one row list and one c

olumn list. This structure is detailed in Part II, section N.

3. AM (Adjacency Matrix) and AL (Adjacency List). These are the structures of choice tor supporting a graph traverse, and they are explored in Chapter 10.

4. HT (Hash Table). The indices of a cell in a large table can be used to form a hashed address in a smaller table. This approach depends on the discussions of hashing in Part II, sections B and L . It is the basis of PJ6. 1.

5. OL (Orthogonal Lists). A circular list of row headers and a circular list of column headers form the framework of row and column (circular) lists. Each cell in the array is braided into one row and one column list. This approach is discussed in section 6.5.

For all of these structures, the user is generally a procedure that accesses the matrix as a table, as would those of section 6.3 when applied to sparse matrices. The desired service is provided by STORE(i,j,x) and RETRIEVE(i,j).

Note: It may be possible to rewrite procedures that access a sparse matrix to make them "more efficient," but don't. The result tends to be logical spaghetti, which is hard to debug. Leave if for another course.

In all of these structures, deletion is awkward, and deletion can occur as the result of STORE. When an assignment is made to sparse matrix M, as done by M[i,j] [image] x, several possibilities must be distinguished:

An existing node is to be updated:

M[i,j]    0 and x    0

A new node is to be inserted into M:

M[i,j]    0 and x    0

A node is to be deleted from M:

M[i,j]    0 and x = 0

In the case of MDC and OL, deletion and insertion are made into both row and column braids with the RingIn and RingOut procedures of section 6.2.

6.5 The Orthogonal List (optional)

An example of a very small but relatively sparse matrix is:

This matrix may be represented by an interwoven collection of circular lists in which there are (2n+1) header nodes and precisely as many cell nodes as needed for nonzero entries. Every node is braided into a row braid and a column braid, and the headers are distinguished by having either a row or a column index that is illegal, as in Figure 6.14, where 0 is the illegal index.

Figure 6.14

Exercise E6.4 is appropriate at this point.

General procedures such as RingIn and RingOut (section 6.2) are applicable to the OL structure if node records use indexed braid pointers and dimension identifiers (for row and column):

Both STORE and RETRIEVE involve finding the correct header, then the predecessor of the node to be accessed. Header location for row or column dex is an adaptation of LT (section 3.3.2):

function FindBraid(Other,dex)         {0(n)

b    (Other MOD 2) + 1

Head    OL    .Braid[Other]

while Head    OL do

if Head    .Dim[b] = dex

then exit endif

Head    Head    .Braid[Other]

endwhile

return Head

end {FindBraid

The appropriate row and column can then be located by:

Row    FindBraid(2,i)

Col    FindBraid(1,j)

A very similar routine locates the predecessor of the node with a given index in the other direction, within a given braid:

function FindPred(Head,b,dex)         {O(n)

Pred    Head

s    Head    .Braid[b]

Other    (b MOD 2) + 1

while s    Head do

if s    .Dim[Other]    dex

then exit endif

Pred    s

s    s    .Braid[b]

endwhile

return Pred

end {FindPred

With these tools available, RETRIEVE is straightforward: the node in the position where M[i,j] is expected is located as p. If M[i,j] is not present, p [image] .Braid[1] will not have the correct column index:

function Retrieve(i,j)                  {O(n)

Row    FindBraid(2,i)

RowPred    FindPred(Row,1,j)

p    RowPred    .Braid[1]

if p    .Dim[2]    j

then return 0

else return p    .Value

endif

end {Retrieve

STORE is more complex, since it may involve inserting or deleting a node. One version, tuned for clarity but not for efficiency, is:

procedure Store(i,j,x)                  {O(n)

Row    FindBraid(2,i)

Col    FindBraid(1,j)

RowPred    FindPred(Row,1,j)

ColPred    FindPred(Col,2,i)

p    RowPred    .Braid[1]

if x = 0

then if p    .Dim[2] = j             {M[i,j] is in the matrix

then RingOut(1,RowPred)

RingOut(2,ColPred)

DlSPOSE(p)

endif

else if p    .Dim[2] = j             {M[i,j] is in the matrix

then p    .Value    x

else NEW(n)

n    .Value    x

n    .Dim[1]    i

n    .Dim[2]    j

RingIn(1,RowPred,n)

RingIn(2,ColPred,n)

endif

endif

end {Store

Program PG6.3 is appropriate at this point.

6.6 A Dynamic Storage Allocation Model (optional)

At the lowest level, the main store of a computing system is a large array, part of it preempted by housekeeping procedures necessary for interface of the system to the outside world and to working programs. The operating system, called on by compiler-generated instructions, is asked to provide storage on demand from the main store each time a program is activated and each time a utility routine such as NEW is executed.

The main store array has a maximum address, like other arrays. If memory is partitioned among several users, each user has a segment available which--to that user--appears to be the entire store. No other user has access to this segment of memory at the same time.

If memory is shared, then its management becomes delicate, because a user may change values in memory cells accessed by other users. Memory is not actually increased in size by sharing, even though its utilization may be greater during a given span of time it would be for unshared memory. The issues involved with shared memory are dealt with in the study of operating systems.

In this section we assume that a single user--while active--has command of a store of fixed size. With that assumption, a program that manages a block of memory by allocating subblocks of fixed size behaves like the memory-management process of an operating system.

Consider a time-sharing system that has a finite number of active programs resident in memory, each waiting for or using a slice of processor time. A program is allocated storage space when it is activated. The amount to be allocated is specified when it is compiled, and no more space becomes available during execution. An apparent limitation of this scheme is that dynamic allocation of memory for pointer variables is not allowed. In some implementations the space allotted to a program may be large enough to allow some dynamic allocation during execution, but the allocation cannot exceed a set maximum.

A request for the allocation of memory for a new program comes from a scheduling process, the scheduler, to the memory management program, Mem. If memory is available, it is provided by Mem, and the program is activated. From time to time, a program completes its execution and becomes passive. When this happens Mem is informed and the storage occupied by the formerly active program is released; it is free and may be used for activation of new programs.

Note: The mangement scheme presented here is a model, based on data structures that have been described in previous sections. More sophisticated structures are applied in practice.

In practice, memory is soon broken into segments occupied by active programs, separated by segments of free storage. The blocks of free storage are linked together by Mem in a circular list called Avail, like the one in Figure 6.15. The release of each of the active blocks 1,3,5,6, and 7 differs somewhat from the release of the others.

The guiding principle for release is that when a block is freed, it is to be joined with any contiguous free blocks, enlarging them. The newly freed block is not to be joined to free blocks separated from it by active programs. This is one of several possible strategies, one that allows "quick release."

The catch to this strategy is that "contiguous" means adjoining, in absolute location, to blocks that are not on the Avail list. Blocks are adjoining on the availability list because of the sequence of entries into and exits from Avail, not because of their absolute location.

The property of being contiguous is bidirectional in memory, and so a natural way to link together all blocks, free and active, is as a doubly linked list, Store, illustrated in Figure 6.16. Mem is exogenous; it uses nodes that are records that point to memory blocks, although an operating system could store the linkage and other information in the blocks themselves (because it has the use of machine instructions that access any cell in any available way).

Figure 6.15

Figure 6.16

If block 3 is deactivated, then blocks 2,3, and 4 are collapsed into one block, say block 2, and the resulting structure is depicted in Figure 6.17.

Figure 6.17

The information required in a block node is an identifier (for active programs, three pointers (two for Store and one for Avail), the absolute starting address, and the size of the block:

BlockType = RECORD

id : ARRAY[1 .. 20] OF CHAR

Back :    BlockType   {Store

Forth :    BlockType   {Store

Next :    BlockType   {Avail

Start : INTEGER

Size : INTEGER

END {BlockType

The two major operations are the release of active blocks and the activation of a program. Both are nontrivial.

Both doubly linked lists and circular lists were used without introducing much complexity in previous chapters, and so it might seem that their action in concert would not be complicated. In fact, combined structures can present a much more complex management task.

6.6.1 The Release of an Active Block (optional)

6.6.2 The Activation of a Program (optional)

6.6.1 The Release of an Active Block (optional)

Given a node p that is to be deactivated, several possibilities must be distinguished in the procedure UnLeash that frees it. With the use of

Up    p    .Back

Down    p    .Forth

the possibilities are delineated in Table 6.1. A straightforward implementation of this organization will produce a nest of if . . . then . . . else constructs. An alternative way to orchestrate the release operation is to define simpler subtasks and then structure the way they work together to solve the overall task.

Table 6.1

Up = NIL  Down = NIL  Avail was empty and now is to contain a single node, p.

          Down    NIL  Down    .next = NIL  Down is active; it

                                          cannot be joined to p,

                                          and so p is added as a

                                          new node to Avail.

                      Down    .Next    NIL  Down is free; it is

                                          already in Avail, and so

                                          it can be updated to engulf p.

Up    NIL  Down = NIL  Up    .Next = NIL     Up is active; it cannot be

                                          joined to p, and so p is

                                          added as a new node to Avail.

                      Up    .Next    NIL    Up is free; it is already

                                          in Avail, and so it can be

                                          updated to engulf p.

          Down    NIL  Up    .Next = NIL    Up is active; it cannot be

                                          joined to p. However, Down

                                          may be either free or active,

                                          and two cases result.

                      Up    .Next    NIL    Up is free; it is already

                                          in Avail, and so it can be

                                          updated to engulf p.

                                          However, Down may be either

                                          free or active, and two cases

                                          result.

If Up is free, then p and Up may be joined to form a single free block. From the perspective of Up, it is simply extended to engulf p. Hence, a natural operation is the joining of a lower block to an upper block, say Join(dList,q), where q points to the upper block in a doubly linked list, dList. Then Join(dList,Up) can be followed by deletion of the lower block from dList with a standard DELETE operation, UnDouble(dlist,p). (In this example, the specific list involved will be called Store.) If p is not absorbed by Up it is inserted into Avail with Free( p). If Up engulfs p the combined block is renamed p.

If Down is free, then p and Down may be joined; their joining can be considered an engulfment of p by Down. When p and Down are combined with Bind(Down) block p must be deleted from Avail as well as from Store. This deletion from Avail is simply a DELETE operation of a node from a circular list.

When the special cases are taken care of, UnLeash becomes:

procedure UnLeash(p)

Up    p    .Back

Down    p    .Forth

if Up    NIL

then if Up    .Next    NIL

then Join(Store,p)            {join p to Up

UnDouble(Store,p)

p    Up

else Free(p)

endif

else Free(p)

endif

if Down    NIL

then if Down    .Next    NIL            {join (new) p to Down

then Bind(Down)

UnDouble(Store,p)

endif

endif

end {UnLeash

The details of Join, Free, and Bind are left to the problems.

Problems P6.3 and P6.4 are appropriate at this point.

6.6.2 The Activation of a Program (optional)

When a request (by one part of the operating system to the activation routine) is made to activate a program of size s, several possibilities must be distinguished:

1. There is a block p in Avail for which p [image] .Size = s.

2. There are blocks in Avail with size at least as large as s (but not necessarily an exact fit).

3. There is no single block in Avail large enough to contain the program, but if active programs are shifted in absolute location, several blocks in Avail can be joined to form a block larger than s.

4. The total of the sizes of the blocks in Avail is less than s, and so the request cannot be granted at all.

The shifting-and-collapsing operation implied in possibility 3 is called garbage collection. It is not always clear just when garbage collection should be done and how. Automatic garbage collection can be incorporated into the release operation so that, on release, all programs are shifted: only one free block ever exists. Analysis reveals that this is a wasteful overhead in most systems. The assumption here is that it is to be done if no single block is of size s or greater, but the sum of all free sizes is greater than s.

Possibility 2 includes possibility 1; several strategies have been devised to deal with it, and there are various implementations of them. In general, some piece of the free block is left over when there is no exact fit. A large number of activation-deactivation cycles can leave many blocks on the Avail list too small to be useful for active programs. One way to alleviate this problem is to allocate an entire block if it is a "close fit." The effectiveness of this technique is naturally dependent on the statistics of an application. Strategies for allocating blocks as large as s to a program include:

first-fit: Work around Avail until a free block is located as p, for which p [image] .Size [image] s, and then return p.

best-fit: Search all of Avail, and return p, where (p [image] .Size - s) [image] 0 is minimal.

worst-fit: Search all of Avail, and return p, where p [image] .Size - s is maximal and at least 0.

Allocation for any of these strategies involves transferring the program to p and unlinking p from Avail. They are not particularly difficult to program, and they are left to the problems.

It is convenient to move an active program into the bottom of a free block so that, if the excess forms a free block, it is already linked into Avail, although its size has changed.

The ideas of first-fit, best-fit, and worst-fit are used as strategies for solving other problems (such as bin-packing, the solution to which is related to the solution of a huge collection of important problems).

Problems P6.5-P6.7 are appropriate at this point.

Garbage collection in practice involves difficulties outside the scope of this book. For one thing, programs are moved, and they must execute after being moved. If the active blocks are treated as though they can be moved simply by updating the fields of their records, then garbage collection is a relatively straight-forward shifting operation. It is convenient to shift active blocks upward in memory, leaving a single free block at the bottom of storage. Hence Avail can be ignored during the shift and then reinitialized as a single node.

Writing a simulation of memory management that explores the effect of activation strategies and the effect of the distributions of program size and execution times is a substantial project as a class assignment. It is, however, a minor task compared to writing an operating system.

Projects PJ6.1-PJ6.3 are appropriate at this point.

Summary

Control traffic flows over links in one direction only, and so it is sometimes useful to trade space for the convenience of multiple-link paths. The most straightforward multiple-path structure is the doubly linked list, but lists may be interwoven in almost any pattern to form n-braids. Deletion and insertion in interwoven lists requires more care than in singly linked lists; the doubly linked list is intermediate in this respect.

The adjacency matrix, the graph, and the digraph are models with great power to simplify, synthesize, and organize. All three correspond to data structures of wide application and provide background for understanding them.

Matrices (and other tables) commonly waste space because many cells contain a standard value (such as zero). A number of structures are available for dealing with this situation and are treated in section 2.7; Part II, sections B, L, and N; section 6.4; Chapter 10; and PJ6.1. The Orthogonal List structure is detailed in section 6.4. This structure and the one detailed in Part II, section N, represent cells of the matrix as nodes braided into circular lists.

One (simplified) model of the dynamic allocation of memory in a computing system is a doubly linked list of blocks that contain either active programs or available storage. The available storage blocks are also on an interwoven circular list. Activating and deactivating programs presents a number of interesting problems not restricted to management of the structures themselves.

Exercises

Exercises immediate applications of the text material

E6.1 Write an insertion procedure for a doubly linked list, as discussed in section 6.1.

E6.2 Write the procedures for both insertion and deletion in a doubly linked circular list.

E6.3 Determine the number of two-step and the number of three-step paths for graphs (a) and (b) of section 6.3, and verify by calculating the products Ga2, Ga3, Gb2, and Gb3.

E6.4 Draw the graphs (a) and (b) of section 6.3 as sparse matrix lists.

Problems

Problems not immediate, but requiring no major extensions of the text material

P6.1 Write procedures Ringln and RingOut of section 6.2 for nodes in an n-braid.

P6.2 Deterrnine the timing functions of AddMatrix and MultiMatrix of section 6.3.

P6.3 Write the procedure Join of section 6.6.1.

P6.4 Write the procedure UnDouble of section 6.6.1.

P6.5 Write a first-fit strategy procedure Firstfit for locating storage in Avail as described in section 6.6.2. Return NIL if no free block is large enough.

P6.6 Write a best-fit strategy procedure Bestfit for locating storage in Avail as described in section 6.6.2. Return NIL if no free block is large enough.

P6.7 Write a worst-fit strategy procedure WorstFit for locating storage in Avail as described in section 6.6.2. Return NIL if no free block is large enough.

Programs

Programs for trying it yourself

PG6.1 Write a program that inputs a sequence of paired names and test scores and puts them together in a two-link record. The records are to be linked so that they are in order alphabetically by one sequence of links, and in order by score in another sequence of links. Display the records in alphabetic order, on a line with their score successor, and then vice versa.

PG6.2 Write a program that will accept a sequence of edges (pairs of nodes), and calculate the number of k-fold paths from node i to node j for any specified i, j, and k.

PG6.3 Write a program designed around an entry loop that allows a user to insert and retrieve values from an n x n Orthogonal List form of a sparse matrix (as described in section 6.5).

Projects

Projects for fun; for serious students

PJ6.1 (Assumes knowledge of Part II, sections B and L.) Investigate the hashing of a table T[1 . . 30,1 . . 30] into HT[1 . . 10,1 . . 10] . The values in T are all to be either 0 or 1. Start with a loading of HT of 20 percent and determine the mean number of collisions generated by random retrievals and by random assignments, half of which are zero. Determine the loading after 20, 40, . . ., 100 assignments. Repeat this experiment for greater initial loading of HT.

PJ6.2 Write an emulation of memory management as described in section 6.6. The user should be able to activate or deactivate programs by name on command.

PJ6.3 Write a simulation of memory management as described in section 6.6. The user should be able to specify several control parameters and the mean and standard deviations for program sizes. Control parameters should include the number of activations and deletions, fitting strategy, and the "closeness" factor.

Go to Chapter 7 Return to Table of Contents


Data Structures: CHAPTER 7: RECURSION

 Return to Table of Contents

CHAPTER 7: RECURSION

Recursion is an idea, and it is a programming technique. Calculations and definitions in and out of mathematics both involve recursion, and so it appears in computing as a natural expression of such definitions and calculations.

7.1 A Recursion Sampler

As others have pointed out [LEDGARD, 1981], a natural and effective definition of the word descendant is:

DEFINITION: A descendant of a person is

an offspring or

the descendant of an offspring of the person.

This definition is said to be recursive because it is in terms of itself. Two properties common to recursive definitions make it possible to search a statement to test whether or not a candidate satisfies the requirements for being recursive:

an escape clause ("an offspring") that allows a search to terminate, and

a refresh clause ("the descendant . . . ") that continues the search at a new point if escape is not invoked.

If Lottie has child Leonard has child David, then the check to test Leonard as a descendant of Lottie terminates at the first stage through the escape clause. The check for David as a descendant of Lottie must be refreshed once, continuing the search from Leonard.

Many natural-language words are defined by indirect recursion because they are defined in terms of each other. Some, like the word descendant, may be defined by direct recursion, in terms of themselves. Dictionaries are usually not so courageous.

Mathematical forms and functions can be recursive:

DEFINITION: N! = 1 for N = 0

= N X [(N - 1)!] for N > 0

DEFINITION: Xk = 1 for k = 0

= X X Xk-1 for k > 0

Both of these (and most similar examples) are closely tied to one of the pervading ideas of mathematical sciences -- Mathematical Induction, MI for short. MI is ubiquitous because it is one of the defining properties of the counting numbers 1,2,3 . . . . For the sake of completeness, here is a description of MI:

Suppose that S(n) is a statement that involves the positive integer n:

If 1. S(1) is true, and

2. S(k) is true whenever

S(1),S(2), . . . ,S(k-1) are true

then S(n) is true for all positive integers n.

The statement of MI leads naturally to iteration: prove S(1), then S(2) from it, then S(3) from S(1) and S(2) . . . .

The proof that S(n) is true for all n for some statement S has the flavor of recursion: to prove S(k), reduce it to statements about S(k - 1) and perhaps S(k - 2), . . . ,S(1) and by repetition of this, chase the truth of the supporting statements back to S(1). The chain of reasoning is then reversed for the flow of results, building back up to S(k).

A full discussion of MI is outside the scope of this book, but some of it would resemble the discussion of recursion to follow.

The simplest applications of recursion in programming are statements that act as alternates to simple iterations. For example, to find the pointer to the bottom node of a nonempty stack, pointed to by the pointer p, either an iterative or a recursive routine will do:

function ChaseTail(p)               {iterative

q    p

while q   .Link    NIL do

q    q   .Link

endwhile

return q

end {ChaseTail

function UnderDog(p)                {recursive

if p    .Link = NIL

then return p

else UnderDog( p    .Link)

endif

end {UnderDog

UnderDog is logically valid because it repeatedly calls itself at a new point until the escape clause is invoked, and the escape clause is bound to occur. UnderDog can be implemented because a compiler that allows recursion will translate it into an iterative procedure. Very briefly, for languages in which recursion is legal, the translator of UnderDog would build a stack of pointers p, p [image] .Link, p [image] .Link [image] .Link, . . . and then return the last one when the nest of invocations is unraveled. Underdog may be slower than ChaseTail, the iterated form of the operation, but both are O(n) for a stack of depth n.

Recursion can be applied at a more abstract level as it is in the English language to define data structures. For example:

DEFINITION: A linear linked list is NIL, or it is

a node linked to a linear linked list.

This definition describes a process by which a candidate may be searched for conformation, but it does not describe an algorithm. It does not force the process of refinement to terminate after a finite number of steps.

Figure 7.1 shows how the process described by the definition may be depicted.

Figure 7.1

A more specific form of the linear list definition is:

DEFINITION: A linear linked list, LinearList, is either

NIL or

a node, HeadNode, linked to a linear linked list, TaiList.

With this specification at hand, a formal structure LINLIST can be defined as any linked list that satisfies it. Given a formal data structure, there are natural operations on it, although they are not part of the definition. Examples that use a right arrow to denote "produces" are:

HEAD(LinearList) [image] HeadNode and TAlL(LinearList) [image] TaiList, or, more abstractly:

HEAD(LinearList) [image] node and TAIL(LinearList) [image] LinearList, where node is an item of class NODE, and LinearList is an item of class LINLIST.

These can be implemented as (rather abstract) functions:

function Head(LinearList)           function Tail(LinearList)

if LinearList = NIL                 if LinearList = NIL

then return NIL                     then return NIL

else return HeadNode                 else return TaiList

endif                                endif

end {Head                            end {Tail

Neither of these functions is recursive, although the structure on which they act was defined recursively.

No part of the topic data structures is exclusively the domain of either recursion or iteration--they are both universal tools.

Problems P7.1-P7.7 are appropriate at this point.

7.1.1 From Front to Rear (optional)

7.1.1 From Front to Rear (optional)

The structure STACK of Chapter 4 was defined nonrecursively in terms of operations, yet UnderDog is a recursive operation that may be carried out on it. In fact, finding the bottom of a stack, or the last node of an instance of LINLIST, is inherently an operation requiring a repeated search. (The search may be described iteratively or recursively, but it cannot be escaped.) Essentially the same operation appears recursively in one description of the interaction between FRONT and INSERT in the structure QUEUE, as:

FRONT(INSERT(Q,i))    if Empty(Q)

then i

else FRONT(Q)

endif

It is difficult to describe the relationship succinctly any other way, and the interaction between FRONT and INSERT is independent of implementation.

The trace of this interaction for a particular example demonstrates how recursion can be unraveled. After INSERT(Q, 10) is applied in Figure 7.2, then FRONT(INSERT(Q,10)) is also FRONT(Q). FRONT(Q) is FRONT (Q'), where Q' is the queue to which INSERT(Q',-5) is applied to construct Q, as in Figure 7.3.

Figure 7.2

Figure 7.3

Q', in turn, is not empty, hence FRONT(Q') = FRONT(Q\"), where, as in Figure 7.4, Q' = INSERT(Q\",0).

Figure 7.4

But Q\" is not empty, and FRONT(Q\")is FRONT(INSERT(Q\"', 4)), where Q\"' is empty. The value added to an empty queue is finally the front of the queue resulting from an INSERT operation, hence:

4 = FRONT(Q\") = FRONT(Q') = FRONT(Q)

In general, FRONT(Q) is the first value not yet deleted; it can be found by tracing back through all of the insertions made since it was inserted into Q. This process of determining FRONT(Q) by applying FRONT itself as part of the process is recursive. It is a natural basis of a formal, operational definition of FRONT.

An alternate way to define a structure such as STACK, QUEUE, and LIN-LIST is to make the definition functional--expressed in terms of the operators that act on it and the relationships between them. The description of the FRONT and INSERT interaction given above serves as part of such a definition of QUEUE.

Formal definitions of the kind indicated by this section are the subject of current development and research on computer science, under the title of abstract data structures.

7.2 The Fibonacci Connection

Any sequence may be described by a function of the positive integers. The ideal function of this kind describes a simple rule by which the nth value is to be calculated. For some sequences such as 1,2,4,9,16, . . ., a description as a function of the positive integers can be very direct: f (n) = n2.

The sequence of integers 1,1,2,3,5,8, . . . , the Fibonacci Sequence, can be taken to be f(1) = 1, f(2) = 1, f(3) = 2, f(4) = 3,f(5) = 5,f(6) = 8, . . . , but no simple description of this function is apparent. The advantage of modeling a sequence as a function is that the function may describe the sequence more succinctly than does a (partial) listing or it may provide insight into its behavior. There is such a function for the Fibonacci Sequence.

A pattern that describes the Fibonacci Sequence 1,1,2,3,5,8, . . . relates each value to the previous values: 1 + 1 = 2, 1 + 2 = 3, 2 + 3 = 5, 3 + 5 = 8, and so on. The rule that describes the nth value as a function of the previous values is:

f(1) = 1   f(2) = 1

f(n) = f(n - 2) + f(n - 1) for n = 3,4,5, . . .

This function may be used as a description of how to derive the nth value. For example:

f(5)  =    f(3)       +    f(4)

= [f(1) + f(2)] + [f(2) + f(3)]

= [ 1   +   1 ] + [ 1   + [f(1) + f(2)]]

=       2       + [ 1   + [ 1   +   1 ]]

=       2       + [ 1   +       2      ]

=       2       +       3

=               5

This search for the nth value involves only f itself--and f invokes itself.

Self-invocation is relatively common. It appears, for example, in solving algebraic equations such as

2h(n) = h(n) - 3n + 1

This definition of h is in terms of itself, and the solution is h(n) = 1 - 3n, so h(1),h(2),h(3), . . . is the sequence -2,-5,-8 . . . . A major difference between h and f is in the methods of solution that can be effectively applied to them.

The solution techniques for f that are relevant here are the algorithmic ones. For example:

function Fibo1(n)                  {0(2n)    TRANSPARENT STACK

if n = 1 then return 1 endif

if n = 2 then return 1 endif

return Fibo1(n - 2) + Fibo1(n - 1)

end {Fibo1

This is a recursive procedure because it invokes itself. It exhibits direct recursion; if, however, procedure A invokes procedure B, which itself invokes procedure A, indirect recursion is at work.

Procedure Fibol is a neat expression of the mathematical model of the Fibonacci function, but it will not work properly in a language that does not support recursion. Not all languages will support the recursive use of routines, since omitting recursion usually simplifies program compilation and speeds up the execution of programs.

If a recursive procedure is allowed in a language, then it must be possible to restructure it as an iterative procedure, or else it could not be executed on a machine that does one thing at a time. The use of recursion in programming languages can be allowed because the conversion from recursive algorithms to iterative executions is always possible and can be programmed in general into a compiler. As a direct consequence, problems having a natural expression in terms of recursion can be written that way by a programmer. The programmer can assume that the conversion to an iterative form is made correctly in translation.

The two main approaches to iterative calculations such as those needed for the Fibonacci sequence may be thought of as bottom-up iteration and top-down iteration.

Bottom-up Fibonacci Iteration

Top-down Fibonacci Iteration

Bottom-up Fibonacci Iteration

Bottom-up iteration starts with known or easily found values and builds up to the desired value. A bottom-up Fibonacci calculation is:

function UpFibo(n)               {O(n)NO STACK

Fibo    1

OneBack    1

for i = 3 to n do

TwoBack    OneBack

OneBack    Fibo

Fibo    TwoBack + OneBack

next i

return Fibo

end {UpFibo

A bottom-up Fibonacci calculation can also be done with the help of a stack in which the top values play the role of TwoBack, OneBack, and Fibo. For simplicity, assume that a particular stack is accessed by PUSH, POP, DELETE, and TOP without explicitly naming the stack. This implicit stack is similar to the stack of the FORTH machine of Part II, section F. The procedure to be developed has the flavor of the FORTH language itself, in which the standard set of stack operations is augmented with others (that can be defined in terms of the standard set). Stack operations DUP (duplicate the top value), ROT (rotate the top three values so the third comes to the top), and SUM (sum the top two values) operate as shown in Figure 7.5.

Figure 7.5

A procedure for finding f(n) with these operations is:

function UpStack(n)             {O(n)    IMPLICIT STACK

Push(1)

DUP

for i = 3 to n do

DUP

ROT

SUM

next i

return Top

end {UpStack

Top-down Fibonacci Iteration

In order to calculate f(5) in the top-down fashion it is first necessary to calculate f(3) and f(4) upon which it depends, neither of which can be directly evaluated. In order to calculate f(4), f(3) is needed again. In one case, f(3) is "requested" by f(4), and in the other by f(5). Evaluation records can be constructed as an integer k, a value f (set to zero if not yet known), and a link back to the requesting record, shown in Figure 7.6.

Figure 7.6

Figure 7.7 pictures the calculation of f(5).

Figure 7.7

With the requests reduced to this form, the values of higher nodes can be determined from lower nodes, as shown in Figure 7.8, and finally reaching the calculation in Figure 7.9.

Figure 7.8

Figure 7.9

A direct expression of this scheme as a data structure must be deferred to Chapter 8; but, with the help of a stack, the evaluation records can be saved as they are generated and discarded as they are used. Using levels in the stack as links, f(n) can be calculated by repeating:

1. If the f-value of the top record is not zero and the level is 1, then return that value.

2. If the f-value of the top record is zero, then generate two new records (for k - 1 and k - 2) and stack them, linking them back to the record that led to their generation.

3. If the f-value is not zero for the top value but is zero for the next-to-top value, then generate two new records from the latter.

4. If the f-value is not zero for the top two values, then sum them and add the sum to the value to which they are linked. (It happens that they will be linked to the same source of their generation because records are generated in pairs.)

A trace of this process for f(5) is shown in Figure 7.10.

Figure 7.10

This picture of the stack is reflected in an array implementation of the records where array indices act as the level for both the values and the links. A new record that is linked to the record with index Level and has a given kValue may be placed on the stack by:

procedure NewRecord (kValue,Level)

Top    Top + 1

if kValue < 3

then f[Top]    1

else f[Top]    0

endif

k[Top]    kValue

Link[Top]    Level

end {NewRecord

A record at index level can be split to produce two new records on top of the stack by:

procedure Split(Level)

kValue    k[Level]

NewRecord(kValue-1,Level)

NewRecord(kValue-2,Level)

end {Split

With these operations defined, a top-down iterative calculation of the nth Fibonacci number is:

function DownStack(n)                     {0(2n)    EXPLICIT STACK

if n    3 then return 1 endif

Top    0

NewRecord(n,0)

repeat

if f[Top] = 0

then Level    Top                 {split top record

Split(Level)

else if f[Top-1] = 0

then Level    Top-1           {split next-to-top record

Split(Level)

else Level    Link[Top]   {sum top values

Sum    f[Top] + f[Top-1]

f[Level]    Sum

Top    Top-2

endif

endif

until Top = 1

return f[1]

end {DownStack

A compiler that translates the recursive function subprogram Fibol would generate activation records for each call--activation records that would resemble the evaluation records of DownStack. Note that the "n" in the definition of Fibol keeps changing with each invocation. If the records of DownStack are taken to be the actual activation records, the second stack snapshot depicts three values of n: n = 3,4, and 5 stored in k[Level] for Level = 3,2, and 1.

The availability of some language features, however, may complicate the activation records (and their generation). Even the relatively simple PL/0 compiler that generates p-code (section 4.4) must take into account the block structure of the language. Specifically, an identifier may be redefined within a subprogram, where use of the identifier has a limited scope. It is not the same variable as one named identically elsewhere. The linkage patterns within the activation stack must trace variable references back to the location of their values within the stack. That location is determined by the sequence of procedure calls during execution, in conjunction with the effects of identifier scope.

The complex problems associated with the implementation of recursive routines like Fibol can be left to language design and compiler writing. What is not to be left to them is: How can recursion be used as a design tool? When should it be used as a program design?

There is no definite answer to these questions, but--as a rule of thumb--if recursion is a natural expression of a problem solution and execution-time is not crucial, then use it.

7.3 Search in an Ordered List

The search through an ordered list for an arbitrary element is a common operation in computing. If output is ordered, then a search through it is efficient--the justification for sorting in the first place. The output of a program ultimately is ordered for the convenience of a search by a human or another program that accepts it as input. Sorting is justified by searching, and searching is made cost-effective by sorting.

The collection of objects to be searched in this section is essentially a file, a collection of records. The records may simply be numeric or character or string data stored in an array or list structure, or they may be more complex records. Records are assumed to contain a key, an item by which the looked-for record can be uniquely identified, and by which the file has been sorted. The sorting can be indirect through a map, but the crucial factor is whether there is independent (random) access to the key.

The problem addressed in a search, as the term is used here, is to locate an arbitrary key, Lost, somewhere in an ordered file of keys or determine that it is not there.

Two natural assumptions that can be made in casting about for efficient search techniques are (1) the file to be searched is ordered and (2) it allows independent access to keys. The simplest model of such a structure is an array containing key values that may be compared to Lost. The essential logic of the search is unchanged if the keys are not the only record values of interest or if they must be accessed through a memory map. It will be assumed here that the keys are the entries in array Key, and Lost is to be found in Key[1 . . Max].

7.3.1 Sequential Searches

7.3.2 Binary Searches

7.3.1 Sequential Searches

A general (unordered) linked list, aList, with only two accessible entries (aList.Head and aList.Tail), supports only an exhaustive, linear search. (To determine that Lost is not present, every node must be checked.) The ordering of such a list structure provides for ordered output but still supports only sequential searching. The only advantage offered by ordering such a list for a linear search is that the search may be terminated as soon as the values become too large (or too small) for an absent search value to be found in the remainder of the list.

One not very efficient sequential search is LSD (for Linear Search--Dumb):

function LSD(Key,Max,Lost)              {O(n)

Here    0

for k    1 to Max do

If Key[k] = Lost

then Here    k

endif

next k

return Here

end {LSD

LSD has the fault that it just keeps floating along, even after finding Lost. It could be fixed in appearance by replacing Here [image] k with return k, but in some HLL languages it cannot be implemented that way.

The for loop contains a test for the end of the search, which is what the body of the loop also contains. Only one test would suffice if it were always the case that Lost would be found. If Key is large enough, this becomes LSE (Linear Search--Extended):

function LSE(Key,Max,Lost)                {O(n)

Key[Max + 1]    Lost

k    1

while Key[k]    Lost

k    k + 1

endwhile

return k

end {LSE

This is an example of code-tuning. It can save 20-30 percent of the execution time in a long search. That is still slow, compared to the search discussed in the next section, but LSE is advantageous because it can be applied to linked lists and even unsorted lists.

7.3.2 Binary Searches

A natural and recursive way to look for Lost among the values in Key[1 . . Max] stored in nondecreasing order is:

If Lost is the middle value, then quit looking. If it is smaller, then search the lower half of Key, otherwise the upper half of Key, this same way.

Each iteration of this search has roughly half as many objects to search through. If Max = 2k, then the first iteration is searching within 2k objects, the second 2k-1, and so on until there is only one value, Key[i], to compare with Lost. No more than (k + 1) = (ln Max) + 1 comparisons need to be made. In general, Max may not be a power of 2, and the search may terminate at any stage, and so a careful analysis must consider these factors. The result is of the same order, however, and the algorithmic expression of a binary search proves to be O(ln n). For large data sets this is a major difference from the O(n) needed for a linear search. (For n = 1,048,576, ln n is 20.)

Given that the essential idea of a binary search is recursive, there is still a choice between an iterative and a recursive algorithm for carrying it out. The iterative version that returns the index of the found value, or zero if it is not found, is:

function IBS(Key,Max,Lost)          {O(ln n)    ITERATIVE BINARY

Lower    1                                    SEARCH

Upper    Max

while Lower    Upper do

Mid    (Lower + Upper) DIV 2

case

Lost > Key[Mid]: Lower    Mid + 1

Lost = Key[Mid]: return Mid

Lost < Key[Mid]: Upper    Mid - 1

endcase

endwhile

return 0

end {IBS

It is worthwhile to trace this algorithm on a simple example.

Exercise E7.1 is appropriate at this point.

The recursive version of binary search needs to have Lower and Upper passed to it, since it operates on a segment of an array, not on an entire array. This is a fairly common effect of the change from an iterative to a recursive routine. Some local variables become parameters because they must change with each invocation of the routine. One version, initially invoked as RBS(Key,1,Max,Lost), is:

function RBS(Key,Lower,Upper,Lost)   {O(ln n)    RECURSIVE BINARY

if Lower    Upper                               SEARCH

then Mid    (Lower + Upper) DIV 2

case

Lost > Key[Mid]: RBS(Key,Mid + 1, Upper,Lost)

Lost = Key[Mid]: return Mid

Lost < Key[Mid]: RBS(Key,Lower,Mid - 1,Lost)

endcase

else return 0

endif

end {RBS

In one sense, RBS is more restrictive than IBS because it involves more parameters in the initial call. In another sense, RBS is less restrictive because it can be used to search any segment of an array, whereas IBS must search an initial segment. The latter can be easily generalized to do the same thing and both provide the same service.

More run-time overhead can be expected for RBS than for IBS because of the stacking of activation records, but both are still O(ln n) algorithms. Otherwise, for large n, RBS may use a great deal of space on the stack.

The idea of binary search is recursive, but the implementation does not have to be. The idea of dividing a data set into segments and dealing with each of them recursively has been generalized to an entire class of algorithms--the general technique is called Divide-and-Conquer (D-and-C). Among the D-and-C algorithms are sorting algorithms, one of which is the subject of section 7.5.

Exercise E7.2 is appropriate at this point.

7.4 The Towers of Hanoi (optional)

A rich variety of stories have been formed about an apparently ancient puzzle:

There are three pegs, a, b, and c, and a stack of disks on peg a. The disks are all of different sizes and stacked so that this condition holds: no disk rests on a smaller disk. How can they be shifted to peg b (using all three pegs) so that the condition never fails on any peg during the process?

It is told that the Tibetan lamas know the solution, have been shifting 64 golden disks on three golden pegs by hand for centuries (with centuries more to go); if they succeed in some future millenium--the legend goes--the universe will end.

From the common name of the puzzle it must be known in Indochina. From its occurrence in this text, the puzzle has a role in computer science. That role is to be an example of the power of recursion.

To (legally) shift n disks from a to b, simply shift the top n - 1 disks to c, move the bottom disk from a to b, and then shift the disks on c to b. The second shift requires a change of role for the pegs, and so the information passed to the solution procedure is the number of disks to be moved, the source peg, the target peg, and the helper peg. The recursive solution is then:

procedure Towers(n,Source,Target,Helper)

if n = 0 then return endif

Towers(n - 1,Source,Helper,Target)

MoveOne(Source, Target)

Towers(n - 1,Helper,Target,Source)

end {Towers

If the pegs are represented by stacks, MoveOne simply pops the top value of Source, and pushes it onto Target.

Exercise E7.3, problems P7.1-P7.7, and program PG7.1 are appropriate at this point.

7.5 QuickSort (optional)

A number of sorting techniques involve exchanging one value (or node) for another. An obvious source of inefficiency with exchanges is that elements of a file to be sorted are exchanged more than once during the sorting process. If it were possible to exchange an element directly into its proper position (ignoring the chaos in the rest of the file at that point), then n such operations would sort the file. It would be an O(n) process.

It is not possible to make a direct exchange of an element to its correct (sorted) position because that position is not known. The correct position of an element is not even detectable without a check of the entire file for the property of being sorted. The sorted position of any element is a global property of the file.

One way to develop sorting techniques is to examine weaker properties-- those that are possessed by a sorted file but do not imply that it is necessarily sorted. A combination of such properties or the repeated establishment of one can lead to a sorted file.

Consider an array Key[1 . . Max]. If Key is sorted then for any index k, 1 [image] k [image] Max, the following property holds:

Property P(k):

Key[i]    Key[k] for 1    i    k

Key[k]    Key[j] for k    j    Max

Property P(k) for some k can be true with otherwise arbitrary order, and so provide a subgoal on the way to the sorting of Key. For example, Figure 7.11 has P(4), but not P(k) for other values of k, and this sequence is not sorted.

Figure 7.11

However:

If P(k) is true for every k, 1 [image] k [image] Max, then Key is necessarily sorted.

A recursive sort called QuickSort can be developed by repeatedly establishing P(k) within the subrange Key[i] . . Key[j]. The details of QuickSort are to be found in Part II, Section O, where it is developed and implemented in Pascal.

7.6 The Eight-Queens Problem (optional)

Simply because of aesthetics some problems have become part of the repertoire of nearly everyone who is acquainted with computer science beyond an elementary programming course. Either the problem itself or its solution is elegant. One such problem is:

Place eight chess queens on a chess board so that they do not threaten each other.

A chess queen can capture another piece by moving any number of squares along one of the eight rows or eight columns or any 45-degree diagonal. Eight queens is an upper bound because no two queens can occupy the same column (or row). A solution places the nth queen in column n of the board and determines the row, Queen[n], of the nth queen so that it controls a row and diagonals distinct from those on which the other queens lie. It is possible to do this iteratively [WIRTH, 1971]. The solution presented here is recursive.

Suppose that procedure Remainder is designed so that Remainder(n) places the queens from the nth one through the eighth one to solve the problem, if possible, and also sets TruePath to TRUE if it succeeds and to FALSE otherwise. In particular, if TruePath is set TRUE by Remainder(1), the problem is solved. A solution is:

program EightQueens

SetUp

Remainder(1)

if TruePath

then DisplayBoard

else "No solution was found."

endif

end {EightQueens

If the procedure call Remainder(n) is made with n > 8, then eight queens have been successfully placed, and it returns with TruePath = TRUE. When entered with n [image] 8, Remainder makes a procedure call NextRow(i,j) that searches for an available row beginning at row i. Initially NextRow is called with i = 0 and j = n and looks for the available row of lowest index, then documents the placing of queen n on that row and column n. To check the ultimate suitability of that choice, Remainder(n + 1) is called, which of course makes Remainder recursive. If it fails (TruePath = FALSE), then the row chosen by the call NextRow(i,n) is canceled (with procedure Cancel) and a row of greater index is attempted:

procedure Remainder(n)

TruePath    FALSE

if n > 8

then TruePath    TRUE

return

endif

Row    NextRow(0,n)

while (Row    8) AND (NOT TruePath) do

Remainder(n + 1)

if NOT TruePath

then Cancel(Row,n)

Row    NextRow(Row,n)

endif

endwhile

end {Remainder

The function called NextRow(i,j) simply steps through the rows i, i + 1, . . . , j and checks to see if they are safe, meaning unoccupied, with a function Safe:

function NextRow(Row,Column)

repeat

Row    Row + 1

until Safe

if Row    8 then Document endif

return Row

end {NextRow

Note: Cancellation is a form of backtracking--a retraction of a partially developed solution in order to try another direction. Backtracking is a general approach to solving problems often applied to binary trees and to trees, discussed in Chapters 8 and 9.

At this point in the development of EightQueens, it is no longer possible to avoid the details of how to mark and detect the presence of a queen in the rows and diagonals controlled by it.

A queen in row i, column j, also controls two diagonals, one tilting 45 degrees to the right from vertical, and one tilting 45 degrees to the left. From the view of placing another queen, it does not matter which position in a row, column, or diagonal is occupied by another queen. Rows and columns are characterized by their index. The fifteen distinct diagonals that tilt to the right are characterized by positions for which i + j is constant and has one of the values: 2,3, . . . , 16. (A move of one step up and to the right on a right-tilted diagonal decreases the row by one and increases the column by one.) Similarly, the fifteen diagonals that tilt left are characterized by i - j (in the set - 7, . . . , 7).

Figure 7.12 shows the row, column, and diagonals controlled by a queen at row 5, column 6. No information about the occupation of columns is needed since columns are possessed in sequence. Controlled rows and columns are marked in Boolean variables RowMark, TiltRight, and TiltLeft, all initialized to FALSE to indicate that they are not controlled by any queen. Documentation of an occupation is performed by setting the appropriate cells to TRUE, cancellation by resetting them to FALSE, and checking for safety by insuring that the appropriate row and diagonals are all FALSE:

Figure 7.12

function Safe

if Row > 8

then return TRUE

else return NOT (     RowMark[Row]

OR TiltRight[Column + Row]

OR TiltLeft[Column - Row])

endif

end {Safe

procedure Document

Queen[Column]    Row

RowMark[Row]    TRUE

TiltRight[Column + Row]    TRUE

TiltLeft[Column - Row]    TRUE

end {Document

procedure Cancel(Row,Column)

RowMark[Row]    FALSE

TiltRight[Column + Row]    FALSE

TiltLeft[Column - Row]    FALSE

end {Cancel

Program PG7.2 and project PJ7.1 are appropriate at this point.

7.7 A General-List Visitation Algorithm (optional)

The records of a general list (graph) structure may contain any number of link-fields, and the links may form a tangle of great complexity. With just two link-fields a small number of nodes may form a pattern that is not any structure studied in the previous chapters--for example the pattern in Figure 7.13.

Figure 7.13

It is necessary to visit every node in such a structure to search within it, or copy it, or process the data field(s) in any way. Ways to traverse graphs based on the data structures of Chapters 8 and 9 are treated in Chapter 10. In this section, the structure is modeled by its own form.

A general record format that will serve for such a structure is:

aNode = RECORD

Data: DataType

Link: ARRAY[1 .. Max] OF    aNode

END {aNode

A recursive process that visits every node of a structure and processes it is:

procedure Rvisit0(p)                {Recursive Visit 0

if p    NIL

then Process(p)

for i = 1 to Max do

Rvisit0(Link[i])

next i

endif

end {Rvisit0

When Rvisit0 is applied to this example with p initially pointing to node 1, it will process nodes 1, 2, 3, 5, 6, 4, 5, 6. The multiple processing of nodes (such as 5 and 6 in this example) can cause severe problems for a user as well as inefficiency. A more serious problem is that if a link-field of node 6 pointed to node 1, it would put Rvisit0 into an infinite loop. One remedy for both problems is to introduce a Mark field set to TRUE for a node not yet processed and FALSE otherwise:

Node = RECORD

Data: datatype

Mark: BOOLEAN

Link: ARRAY[1 .. Max] OF    Node

END {Node

With this change, a recursive visit algorithm that processes each node only once is:

procedure Rvisit(p)                {Recursive Visit

if (p    NIL) AND (p    .Mark)

then Process(p)

p    .Mark    FALSE

for i = 1 to Max do

Rvisit(Link[i])

next i

endif

end {Rvisit

The advantage of recursive algorithms such as Rvisit is that they tend to be much simpler and more easily developed than equivalent iterative algorithms. Nevertheless, they do have shortcomings.

One problem with Rvisit itself is that if the mark-fields are all initially TRUE, then after one execution of Rvisit, they are all FALSE; and it cannot be reapplied. A separate (and similar) routine is required to complement the values of the Mark field, or Rvisit must be changed to allow p [image] .mark to be compared to a Boolean argument that can then be alternated between TRUE and FALSE.

A more severe problem with Rvisit and other recursive routines is that if a large number of nodes are involved, the run-time stack may become quite large, for it contains a copy of the information needed to continue the process for each currently unresolved link.

It is possible to derive an equivalent iterative algorithm from any recursive algorithm, but it is an awkward process and the result often benefits from rewriting. More importantly, the translation is essentially the building of explicit stacks to replace the implicit stacks that allow recursion to work.

An alternative to Rvisit that is not recursive or the explicit-stack translation of a recursive routine was discovered by Deutsch, Schorr, and Waite. A structured version, Visit, is presented in Part II, section M.

Project PJ7.2 is appropriate at this point.

7.8 LISP: Lists and Recursion (optional)

A language used extensively in the field of artificial intelligence, and hence of increasing importance, is LISP. It is a working language becoming more widely available; a number of introductory books treat LISP programming. (One such is [TOURETSKY, 1983].) LISP combines a number of views about computation that give it a quality drastically different from Algol-derivations (Pascal-like languages). These include the central importance of applying functions and the use of predicates--statements with Boolean values. This section is only intended to point out how LISP is related to two themes of importance in this book: linked lists and recursion.

The central objects in a LISP program are lists, including functions that operate on them and the definitions of the functions. In the usual syntax, lists are enclosed in parentheses, and each pair of parentheses forms a list:

ITEM is not a list, but an atom.

(ITEM) is a list and has the effective structure shown in Figure 7.14

Figure 7.14

Similarly, (TWO ITEMS) has the structure shown in Figure 7.15.

Figure 7.15

The structure of ((TWO ITEMS)) is depicted in Figure 7.16.

Figure 7.16

And (((TWO ITEMS))) has the structure shown in Figure 7.17.

Figure 7.17

Storage efficiency is gained by saving atoms only once, as illustrated in Figure 7.18.

Figure 7.18

One list may be the atom of another, shown in Figure 7.19.

Figure 7.19

The recursive nature of LISP is evident in the four primitive functions below from which other functions are built. They are all designed to be used most naturally in recursive ways:

CAR returns the first atom of a list.

CDR returns the tail of a list--all but the first atom.

CONS returns a list constructed from an atom and a list.

LIST returns a list constructed from atoms.

The action of these are illustrated in Figure 7.20.

Figure 7.20

One way to retrieve an interior atom from a list is to recursively apply CDR until it is the first atom of the list, then apply CAR. That process is typical of the way LISP is written.

Note: CAR is derived from: "Contents of Address portion of Register," CDR from "Contents of Decrement portion of Register." The terms have apparently been retained for the love of the arcane found in cults.

Further pursuit of LISP itself requires a more extensive treatment than belongs in this book, but the effect of CDR, CAR, CONS, and LIST can be programmed in any language.

Project PJ7.3 is appropriate at this point.

Summary

Recursion is a fundamental way to express some relationships. When available, it can be a very convenient and delightfully succinct form for the expression of procedures.

The foundation on which language translators build recursion is a stack of activation records. Activation records are linked together on the "stack" in (possibly) interwoven lists that trace the value determinations of variables to their appropriate sources. (The p-Machine of section 4.4 and Part II, section G, is a relatively simple example of how this can be done.)

Recursion can be a marvelously easy form within which to structure an algorithm. It can also be wasteful of time and space, as illustrated by various ways to calculate the Fibonacci sequence.

The values of recursion are convenience, saving of programming time, and the development of an understanding of a problem. These may be traded for execution efficiency at the cost of the time and care required to develop an iterative solution.

Some examples in which recursion is generally deemed to be worth the runtime cost are: binary search, the Towers of Hanoi, QuickSort, and the Eight-Queens problem. (Other ways to traverse a general list are to be found in Chapter 10 and Part II, section M.)

QuickSort is frequently the sorting method of choice in applications because its average-case behavior is O(n ln n). QuickSort is an example of a recursive algorithm that tends to execute more efficiently than most iterative rivals.

The Eight-Queens solution is an example of backtracking, a programming technique of general utility.

LISP is a language that incorporates recursion and list processing into the mold of thought that it promotes in programmers.

Above all, recursion is a choice for problem solving that should be available to any programmer.

Exercises

Exercises immediate applications of the text material

E7.1 Trace the procedure calls IBS(Key,6,8) and IBS(Key,6,5) when Key contains these values: Key[1] = 1, Key[2] = 3, Key[3] = 4, Key[4] = 6, Key[5] = 8, Key[6] = 10.

E7.2 Trace the calls of procedure RBS(Key, 1,6,8), RBS(Key, 1,6,5), and RBS(Key,3,6,5) with the values of Key given in E7.1.

E7.3 Trace the stacks representing the pegs in procedure Towers of section 7.4 for n = 4. (PG7. 1 asks for a program that does this.)

Problems

Problems not immediate, but requiring no major extensions of the text material

Write recursive versions of problems P7.1--P7.4:

P7.1 Search a singly linked list for a given node value.

P7.2 Search a circularly linked list for a given node value.

P7.3 Calculate N-factorial.

P7.4 Calculate the sum of the first N integers.

P7.5 (Difficult) Develop an iterative version of the Towers of Hanoi solution given in section 7.4.

P7.6 Find the GCD (Greatest Common Denominator) of two values N and M as GCD(N,M), calculated as either of them if M = N, otherwise the GCD of the smallest of them and their difference. (Use a recursive procedure.)

P7.7 Find the determinant of an N by N matrix. The determinant of A can be calculated as A[1,1] for N = 1 and the sum from j = 1 to N of (( -- 1)i+j times A[i,j] times the determinant of the ij-cofactor of A). The ij-cofactor of A is the matrix formed by deleting the ith row and the jth column of A. (Use a recursive procedure.)

Programs

Programs for trying it yourself

PG7.1 Implement procedure Towers of section 7.4 with a trace of the stacks. Run it for n = 4,5, and 6.

PG7.2 Implement the Eight-Queens solution of section 7.6 with a display routine that forms a board representation with queen markers in place.

Projects

Projects for fun and for serious students

PJ7.1 Write a program to find all solutions to the Eight-Queens puzzle.

PJ7.2 Implement Rvisit of section 7.7.

PJ7.3 Write a command-shell program to evoke functions CAR, CDR, CONS, LIST, DISPLAY, INSERT, and DELETE. The last two commands allow insertion and deletion at any depth in a list. The program is to implement DISPLAY, INSERT, and DELETE internally with the other command functions.

Comment: You may wish to implement the one-cell-per-token rule as a separate project.

Go to Chapter 8 Return to Table of Contents


Data Structures: CHAPTER 8: BINARY TREES

 Return to Table of Contents

CHAPTER 8: BINARY TREES

Program logic involves branches where there is more than one possible next action: two for an if statement, two or more for a case statement. People have zero or more children who may in turn have zero or more children who. . . . People have two immediate ancestors who in turn have ancestors. One-to-(some number) relationships are involved in these situations cnd in many others. A graphical structure that describes such relationships is a (directed) tree--mathematically, a (directed) graph without cycles.

As an example, if Faye has children Alia, March, and Ralph, two of whom have children, their family relationship may be described by Figure 8.1. The notation associated with such structures is fairly comfortable for anyone conversant witj both families and botany. Faye is the root node of a tree, Faye's Family. The tree rooted at Faye has three braches, and they are called subtrees. The subtree rooted at Alia has two branches, at Ralph one, and at March none. Wrye, Jan, March, and Will are leaf nodes of Faye's Family and of its substrees. All other nodes are interior nodes. Alia and Faye are both ancestors of Wrye and Jan.

Figure 8.1

In the literature, Faye is commonly called the father of March, Alia, and Ralph, and these three are the sons of Faye. (It is perfectly acceptable to use mother and daughter, as in done for cells in biology.) Wrye and Jan are siblings, often called brothers. Faye is the grandfather of Jan.

A single node with NIL (or u) children is a tree, and so is the empty tree. Faye's Family is specifically a tree within the more general category graph. The crucial feature making it a graph is that the subtrees (subgraphs) reachable from any pair of nodes are disjoint. To generalize, the ith child of a node is the root of a tree itself. It is the ith subtree of the tree rooted at n. This is abbreviated to "the ith subtree of n."

Faye's Family can be modeled by a data structure with links representing the directed access from a node to its children. The many applications for general tree structures are limited by their own generality: the nodes of a general tree might need to have any number of links. In some applications, such as representations of family trees, it may not always be possible to foretell how many links are enough. Even if a maximum is imposed, most of the links of most nodes may be unused.

Figure 8.2

A useful compromise that contains a one-to many relationship and yet controls the proliferation of children is the binary tree. A binary tree may have a left child and a right child, which is not the same as having two children. The trees in Figure 8.2 are equivalent as trees, but not equivalent as binary tree. A binary trees T1 [image] T2 because their left children (and redundantly their right children) differ. Will may be implemented as either the left child or the right child of Ralph, giving rise to two distinct binary trees, shown in Figure 8.3.

Figure 8.3

General trees with more than two children per node can be transformed into binary trees by the rule:

Order the children of a node 0 of tree T to obtain children c1,c2, . . .,ck. Then c1 becomes the left child of N. If N is the (i - 1)st child of a node P and there is an ith child of P, then that ith child of P becomes the right child of N.

The tree structure rooted at Faye becomes a binary tree in Figure 8.4.

Figure 8.4

Note: The distinction between binary trees and trees is important to recognize, but it is common to ignore the distinction in situations where it is not significant.

One advantage that tree-like structures have over linear linked lists is that, on the average, fewer links need to be processed to get from one node to another (assuming, of course, that the choice of which branch to take can be determined). The binary tree BT1 in Figure 8.4 has given up some of the advantage inherent in the original tree because there are more links between the root and some leaves than there were. That advantage can be regained by balancing the binary tree so that the distances from root to leaves tend to be as near the minimum as the application allows. In this particular case, rearranging the binary tree would lose the information modeled by it; for some applications, however, balancing is an advantage.

We will return to the issue of balancing in section 8.7--after examining general features of binary trees in section 8.1, their implementation in section 8.2, finding our way around them in sections 8.3 and 8.4, ordering them in section 8.5, and applying them in section 8.6. Section 8.8 exhibits one more way to walk around a tree.

Exercises E8.1 and E8.2 are appropriate at this point.

8.1 General Features of Binary Trees

Binary trees, like the data structures introduced in previous chapters, are defined by the operators applied to them. In contrast to the previous structures, binary trees have shape; and shape is determined by the number of nodes at various distances from each other. The efficiency of management algorithms is affected by the shape of binary trees. Insertion and deletion algorithms themselves change the shape of a tree. If no operations that change shape or size are applied to a binary tree, then it is static; otherwise it is dynamic.

The efficiency of algorithms that provide access to trees is amortized by averaging over a sequence of operations. For many applications, restrictions are applied in the form of structural parameters that control shape. For static trees, these restrictions only affect the cost of searching; but, for dynamic trees subject to many insertions and deletions, shape changes are controlled by constraints defined by parameters. Shape control is introduced in the interest of increasing the amortized efficiency at the cost of more overhead during individual uses of the algorithms.

8.1.1 Operations

The nature of the structure BINARY TREE is determined by the operations applied to it. Binary trees can be empty, and they remain empty until a node is added to them; and so their locator, root, has an appropriate initial value: NIL or u (or zero for array implementations). The genesis of a binary tree is an operation formally called CREATE.

Deletions from a binary tree can remove all of its nodes, and so root may be dynamically set to NIL or u (or zero for an array implementation). The formal name of the operation that checks the viability of a binary tree is EMPTY.

A node may be inserted into a binary tree at any link that is NIL. Such a node (any node!) is itself the root node of a binary tree. If the attached binary tree is a singleton, it becomes a leaf node. The formal operation involved is INSERT, but the question of where to insert depends heavily on the application. INSERT can be universal only if it inserts a subtree as a given child of a given node, replacing whatever was there.

The left child or right child of a given node may be set to NIL, which detaches the binary tree rooted at that child and hence partitions the binary tree into two binary trees. Note that--like attachment--partitioning deals with a subtree, not necessarily a single node. Deletion of a single node, like insertion of a single node, is application-dependent and of considerable interest in the management of binary trees.

In distinction from stacks and queues, all nodes of a binary tree are accessible by following links from the root node. The ways a binary tree can be systematically traversed to visit every node are treated in sections 8.3 and 8.4. In many applications information is available to make a choice at each node; there is a guide to following a link to the left or to the right. When this is possible, the process of locating a node can be much faster than for a general traverse. The efficiency of this guided branching is one major advantage of trees over both linear structures and more general structures (such as graphs).

Because every node in a binary tree can be located, values can be retrieved from and assigned to every node. These operations do not change the structure, they update it.

8.1.2 Node Levels and Counts

In practice, a procedure that manages a binary tree revolves about location, whether for attachment and detachment of nodes or for retrieval and assignment of values. Location, in turn, is tied to the concept of the level of the nodes in a binary tree. Briefly:

The root of a binary tree is at level zero and a child of a node is at the next higher level. The depth of a binary tree is the maximum level of any node.

For example, in BT1,

Faye is at level 0.

Alia is at level 1.

Wrye and March are at level 2.

Jan and Ralph are at level 3.

Will is at level 4.

The particular binary tree BT1 is structured to retain the family relationships of Faye's Family; but, if that is disregarded, the same names may be rearranged in a binary tree in a more compact shape, as shown in Figure 8.5.

Figure 8.5

The maximum number of nodes at level 0 in a binary tree is 1, at level 1 is 2, at level 2 is 2 x 2 = 4, and so on. In general:

There are no more than 2L nodes at level L in a binary tree.

There are no more than 2d+1 - 1 nodes in a binary tree of depth d.

(Both results can be proved by easy induction arguments.)

When the empty (NIL or u) nodes of a binary tree are included, it becomes an extended binary tree; the nonempty nodes are called internal nodes, and the empty nodes are called external nodes. The binary tree in Figure 8.6 uses u for empty, and hence external, nodes.

Figure 8.6

Note: N is an internal node BT3; but, if the external nodes are removed, N is not an interior node--it is a leaf node.

Every time a new internal node replaces an external node, it increases both the number of internal nodes and the number of external nodes by 1. Hence, by induction, a binary tree with n internal nodes has n + 1 external nodes.

The internal path length, IPL, of a binary tree is the sum of the levels of all of the internal nodes: IPL(BT3) = 7, and could be minimized to 6 by moving "N" to level 2.

The external path length (EPL) of a binary tree is the sum of the levels of all of the external nodes: EPL(BT3) = 17. If "N" were moved up to level 2, EPL(BT3) would decrease to 16.

The averages of the internal and external path lengths provide a measure of the average number of links needed to reach nodes during searches from the root. For BT3 these are:

A binary tree is full if, when it is extended, each internal node is either a leaf or has two internal descendents, not one. A binary tree is complete if all leaf nodes lie on the lowest levels and those on the bottom level are as far left as possible. Figure 8.7 shows an example of each. A complete binary tree has minimal depth for the number of nodes it contains.

Figure 8.7

Binary tree BT4 is full but not complete, and BT5 is complete.

If there are N nodes in a binary tree of minimal depth, then the path from the root to a leaf has no more than INT(ln N) links. The potential minimal cost of a search for a node is that of a binary search. In practice, the information required to know which way to branch at a node during a search is incorporated into the shape of the binary tree and tends to increase its depth.

Exercise E8.3 and problem P8.1 are appropriate at this point.

8.1.3 Shape

The two binary trees in Figure 8.8 are both constructed so that their node values can be retrieved in nondecreasing order with the same standard algorithm (InOrder; see section 8.3):

Figure 8.8

The location of the value 2 by a standard search algorithm (BSTT; see section 8.5.1) applied to BT7 takes a third as many link-hops as it does for BT6. If the values in these trees were arranged at random, the expected number of hops would be intermediate between these extremes. An insertion algorithm guaranteed to produce a more efficient binary tree form is itself less efficient than one that simply builds a tree suitable for searching. The compromise between forcing the shape of a binary tree and losing the speed of location within it is a central issue in section 8.7.

8.1.4 On the Use of Binary Trees

Trees are a naturally occurring pattern that can be modeled closely by data structures. Their fundamental feature is branching. Branching is an essential part of language and thought--of any conceptual framework that involves choices. The binary tree reduces branching to its most elementary form, two-way branching. Each branch that is taken in a binary tree also represents a branch that is not taken and sharpens the focus of attention. A sequence of branches sharpens exponentially, much like binary search.

The data structure BINARY TREE is a decision tool, a separator of subsets. Ways to exhaustively traverse a binary tree are useful and sometimes necessary, but in many major applications nodes are reached by relatively short sequences of branches. BINARY TREE provides guided access to an arbitrary node in the structure. It is the data structure of choice when such guidance can be built into the structure.

8.2 Implementations of Binary Trees

When binary trees are implemented with pointer-linked nodes, the nodes are constructed with three fields:

The Value field can, of course, be complex and include pointers to more data of interest. It is not uncommon for binary trees to be exogenous--to be used as directories to information stored elsewhere. This approach is particularly useful when the directory is small enough to place in the main memory of a computing system, but the data is not and resides on a disk or other peripheral storage. Sometimes the value of interior nodes are used simply to locate an appropriate leaf node that is a guide to the information of interest.

A binary tree rooted at node btLeft is linked as the left child of a node bt simply by bt [image] .Left [image] btLeft, shown in Figure 8.9. Linking to a right child is similar.

Figure 8.9

8.2.1 Array Support and the Heapform

8.2.1 Array Support and the Heapform

A binary tree can be supported with arrays in several ways. One way is to use two separate arrays for links and one for values. The 0th cell of one of the link-arrays can be used as a pointer to the root. The link array Left is used for this purpose in Figure 8.10.

Figure 8.10

A binary tree may be implemented in a single array A[0 . . Max] in such a way that no explicit links are needed:

The left child of node A[i] is A[2i].

The right child of node A[i] is A[2i + 1].

With this scheme, called heapform, the example above is stored as:

Note that Max must allow enough room for the rightmost leaf of the structure.

When a complete binary tree of n nodes is stored in an array in heapform, it occupies the span [1 . . n], a very compact, convenient, and efficient arrangement.

When an order condition is imposed on the placement of values within the binary tree, the result is a heap, a structure that merits a separate discussion in section 8.6.

Exercise E8.4 and problem P8.2 are appropriate at this point.

8.3 Recursive Binary Tree Traverses

Three kinds of operations are required to visit and process the nodes of a binary tree:

a move to the left child

a move to the right child

processing a node in some manner

It can be advantageous to process a node when:

It is first reached.

Its left child (subtree) has been processed.

Its right child (subtree) has been processed.

It is possible to mix these six things in a number of ways, but two conventions are used to reduce the number of possibilities:

The left subtree of a node is visited before its right subtree.

A node is processed only once during a traverse.

Three standard traverses remain when these conventions are applied. In recursive form, they are:

procedure PreOrder(bt)          OR    procedure PreOrder(bt)    {O(n)

if bt = NIL                           if bt    NIL

then return endif                      then PROCESS1(bt)

PROCESS1(bt)                                  PreOrder(bt    .Left)

PreOrder(bt    .Left)                          PreOrder(bt    .Right)

PreOrder(bt    .Right)                    endif

end {PreOrder                         end {PreOrder

procedure InOrder(bt)                                {O(n)

if bt = NIL then return endif

InOrder(bt    .Left)

PROCESS2(bt)

InOrder(bt    .Right)

end {InOrder

procedure PostOrder(bt)                              {O(n)

if bt = NIL then return endif

PostOrder(bt    .Left)

PostOrder(bt    .Right)

PROCESS3(bt)

end {PostOrder

These may be combined into one:

RBTT

procedure RBTT(bt)               {O(n)

if bt = NIL then return endif

PROCESS1(bt)

RBTT(bt    .Left)               {Recursive Binary Tree Traverse

PROCESS2(bt)

RBTT(bt    .Right)

PROCESS3(bt)

end {RBTT

Consider the binary tree shown in Figure 8.11. Any traverse visits A (and processes it or not), then B (and processes it or not), then C, then B again, and so on. Each node is visited three times and processed on (at least) one of its visits.

Figure 8.11

The visitation sequence of example BT8 is:

A  B  C  C  C  B  D  E  E  E  D  F  F  F  D  B  A  G  G  G  A

1  1  1  2  3  2  1  1  2  3  2  1  2  3  3  3  2  1  2  3  3

The three standard traverses select the first, second, or third visit to a node in the visitation sequence for processing. Suppose, for example, that PROCESSi is simply a display of the node value. Then traverses of BT8 produce:

Preorder: A  B  C  D  E  F  G

Inorder: C  B  E  D  F  A  G

Postorder: C  E  F  D  B  G  A

Program PG8.1 is appropriate at this point.

8.3.1 A Recursive Tree Display

The display of a binary tree presents a fundamental problem: one of any great size will not fit comfortably within a screen or print-page. That problem can be resolved by the recursive nature of trees: display a tree of modest size where the leaf nodes identify subtrees, then display the subtrees separately.

Whether to display levels on separate lines or turn the tree sideways so that levels run right to left is a design decision. For a binary tree formed from B-I-G-S-H-A-D-E, the two options, without frills, are shown in Figure 8.12.

Figure 8.12

Display 2 is an application of the "breadth-first" traverse of section 8.4.3. (The values are displaced from the center of the display-line by an amount determined by their depth. There must be enough spread to accommodate the bottom line, which may have 2d items in a tree of depth d.)

Display 1 can be done with a recursive traverse that violates the left-first rule. The version that follows prints a string that includes the node value and prints blank spaces or the node value at each level. It is initially called with d = 0:

procedure SideView(d,bt)                             {O(n)

if bt    NIL

then SideView (d + 1,bt    .Right)

Writeln({d blank strings},bt    .Value)

SideView(d + 1,bt    .Left)

endif

end {SideView

8.3.2 Traversals of Arithmetic-Expression Trees

As indicated by their names, the standard traverses are related to the common forms of display for arithmetic expressions.

If the interior nodes of a binary tree correspond to binary operations and the leaf nodes to their operands, it is an arithmetic-expression tree. For example, suppose that A, B, and D in BT8 (section 8.3) are replaced by /, *, and +, respectively, as shown in Figure 8.13.

Figure 8.13

The postorder traverse of BT9 produces

C E F + * G /

which, when the usual precedence is applied, represents the postfix form of the infix expression:

C * (E + F) / G

A compiler often reverses this process, although the binary tree may not be explicitly constructed [AHO, 1977].

The preorder traverse produces:

/ * C + E F G

the prefix form of the same expression. If we think of an operation as adding left and right parentheses around its results, the inorder traverse of BT9 produces:

((C * (E + F)) / G)

To be sure, producing an expression with correct precedence requires either additional processing or additional (parenthesis) nodes, but the order of the tokens is that of the infix expression. The parenthesized expression can be generated by processing an operation node three times: on contact, display "("; on return from the left subchild, display the operation; and on return from the right child, display ")".

Exercise E8.5 and problem P8.3 are appropriate at this point.

8.3.3. A Copy Procedure     (optional)

One fundamental tool that can be applied to a binary tree is a procedure to make a copy of it. The recursive version is particularly apt, because it only requires that a copy be made of both the left and right subtrees and that they be joined:

function TreeCopy(bt)

if bt = NIL then return NIL endif

NEW( p)

p       bt

p    .Left    TreeCopy(bt    .Left)

p    .Right    TreeCopy(bt    .Right)

return p

end {TreeCopy

8.4 Iterative Tree Walks

The recursive forms of tree walks (traverses) are quite elegant, but they can be costly to use because extensive stacks are formed when they are executed. Recursive traversal obscures a salient fact: the links of a binary tree are directional. When a traverse is at node p, the links of p lead to its children, but not to its parent. Some way is needed to backtrack to previously visited nodes while an iterative traverse is in progress.

Explicit backtracking is usually done by stacking node pointers (indices in an array implementation) and then unstacking them to make a return. Explicitly programmed stacks contain only the necessary pointers, whereas a recursive routine is compiled so that the entire routine can be reentered. Additional information is stacked to make that possible.

A binary tree can be traversed by stacking each node when it is first encountered, and then again just prior to traversing each of its subtrees. Each node is then stacked and unstacked three times. The order of stacking is arranged so that when a subtree traverse is complete, the parent node will be unstacked next. Some way is needed to distinguish between three encounters with a node during the traverse: the initial contact and two subtree returns. Processing at these three encounters corresponds precisely to preorder, inorder, and postorder processing. All three processing modes can be combined in one algorithm, or done with walks tuned for them individually.

8.4.1 A General Binary Tree Walk: BTT

For the algorithm that follows, nodes of a tree structure are assumed to contain an integer field, Visit, which counts the encounters of the walk with the node:

VisitNode = RECORD

Value  : {value type

Left   :    VisitNode

Right  :    VisitNode

Visit  : INTEGER

END

The Visit field of each node is assumed to contain 1 at the beginning of the walk and is left that way on exit. An explicit stack, NodeStack, replaces the implicit stack of the recursive procedures. NodeStack is a stack in which the Key field of a node contains a pointer to a tree node. NodeStack may be implemented with records, but it is less obtrusive to think of it as array-based so that Pop and Push take tree node pointers as arguments without the need to encase those pointers in stack nodes.

BTT

procedure BTT(bt)                                    {O(n)

if bt = NIL then return endif

Push(bt)

while NOT Empty(NodeStack) do                      {Binary Tree Traverse

Pop(p)

case of p    .Visit

1 : PROCESS1(p)                               {Prefix processing

p    .Visit      2

Push(p)

if p    .Left    NIL then Push (p    .Left)  endif

2 : PROCESS2(p)                               {Infix processing

p    .Visit    3

Push (p)

if p    .Right    NIL then Push (p    .Right)  endif

3 : PROCESS3(p)                               {Postfix processing

p    .Visit    1

endcase

endwhile

end {BTT

BTT can be used as a preorder listing procedure by setting processes 2 and 3 to  and using PROCESS1 as a display. The operation of the stack during such a walk can be traced by indicating a snapshot of it each time endcase is reached. For a preorder treewalk on BT8:

stack           output

A   B              A

A   B   C          B

A   B   C          C

A   B   C

A   B

A   B   D

A   B   D   E      D

A   B   D   E      E

A   B   D   E

A   B   D

A   B   D   F

A   B   D   F      F

A   B   D   F

A   B   D

A   B

A

A   G

A   G

A   G       G

A

This causes more stack activity than would a tree-walk algorithm with a planned restriction to preorder processing.

Problems P8.4 and P8.5 and program PG8.2 are appropriate at this point.

8.4.2 Individualized Iterative Tree Walks     (optional)

BTT works the stack heavily: every node is popped from it three times. A quite different approach is based on the statement:

      A tree walk is made by stepping left as far as possible, then taking a step to the right.

For the preorder display, the form is:

procedure PreStack(bt)                               {O(n)

if bt = NIL then return endif

p    bt

repeat

while p    NIL do

PROCESS1(p)

Push(p)

p    p    .Left

endwhile

if Empty(NodeStack)

then return

else Pop(p)

p    p    .Right

endif

forever

end {PreStack

The corresponding inorder walk is derived from PreStack simply by moving the processing position. The adaptation of this kind of walk to postorder processing involves reintroducing a Visit, although only two values are required. Because only two values are required for Visit, in an array implementation the (integer) pointers may simply be negated to indicate a previously visited node.

procedure PostStack(bt)                              {O(n)

if bt = NIL then return endif

p    bt

repeat

while p    NIL do

Push(p)

p    p    .Left

endwhile

Pop(p)

while p    .Visit = 3 do

PROCESS3(p)

p   . Visit    1

if Empty(NodeStack) then return endif

Pop(p)

endwhile

p    .Visit    3

Push(p)

p    p    .Right

forever

end {PostStack

Exercises E8.6 and E8.7 are appropriate at this point.

8.4.3 A Breadth-First Traverse     (optional)

The binary tree traverses of sections 8.3, 8.4.1, and 8.4.2 all move from the root of a tree first to the deepest level of the leftmost path or the rightmost path (in SideView), then move to siblings on the same level. This is called depth-first traversal. An alternate class of traverses that may be applied to trees in general as well as binary trees is a breadth-first traversal.

With the help of a queue, Q, and the queue operations applicable to it, InsertRear and DeleteFront, it is possible to process nodes in the order of their levels, left-to-right within each level.

A node p of Q consists of a pointer p   [image]  .ptr to a tree node, and a queue-link field. The insertion of a tree node t into Q in bare outline is:

procedure Insert(Q,t)                                {O(1)

NEW(p)

p    .ptr    t

InsertRear(Q,p)

end {Insert

The level order traverse is then:

procedure LevelOrder(BinaryTree,Q)                   {O(n)

Insert(Q,BinaryTree)

while NOT Empty(Q) do

p    Front(Q)

t    p    .ptr

DeleteFront(Q)

PROCESS(t)

if t    .Left    NIL

then Insert(Q,t    .Left)

endif

if t    .Right    NIL

then Insert(Q,t    .Right)

endif

endwhile

end {LevelOrder

When PROCESS(t) is a display routine, LevelOrder is the basis of a top-down display of a binary tree, in contrast to the sideways display of SideView (section 8.3.1). A display of this kind requires that the queue nodes contain information about both horizontal and vertical positioning. It is an interesting but reasonable challenge to adapt LevelOrder to that task.

Program PG8.3 is appropriate at this time.

The traversal of LevelOrder technique is generalized to use a priority queue and applied to graph structures in Chapter 10.

8.5 Ordered Binary Trees

A given subtree, st, may easily be inserted as the left or right subtree of a given node, p, if the old subtree can be dispensed with:

p    .Left    st     or     p   .Right    st

Applying these statements with st = NIL removes a subtree without regard to what it contains. Thus, the fundamental issue in most applications is what to do with the other nodes of a subtree when a single node p is added or deleted at its root. Both subtrees of p cannot be used as direct replacements for p when it is deleted, and so decisions about what to do with a subtree involve the relationship of the subtree nodes to the rest of the structure. Such relationships generally exhibit some structural parameter, a restriction on the placement of nodes in the binary tree. In particular, binary trees may be ordered.

A binary tree bt may be constructed so that it has property order:

      If p is any node of bt, no node in the left subtree (rooted at p   [image].Left) has a value that is as large as p   [image]  .Value, and no node in the right subtree (rooted at p   [image]  .Right) has a value that is as small as p   [image]  .Value. This imposes the restriction that no value be duplicated.

DEFINITION: A binary tree with property order is a binary search tree (BST).

8.5.1 A Binary Search Tree Traverse: BSTT

8.5.2 Insertion and Deletion in a BST

8.5.3 Static Tree Files     (optional)

8.5.1 A Binary Search Tree Traverse: BSTT

BT10 in Figure 8.14 is a BST but would not be one if the value 15 were replaced by 25 because the defining condition would be violated for the root node.

Figure 8.14

Note that an inorder traverse of BT10 returns the values in the defining order. (This statement can be used as a definition of a BST.) The creation of a BST is thus a sorting method. Without control of the depth of the tree, it is an O(n2) procedure.

The values of a BST do not, of course, need to be numeric. Any operator that has the mathematical property of transitivity (A.op.B and B.op.C implies A.op.C), together with the values on which it operates, will do for the formation of a BST.)

The traverse of a BST is greatly simplified because no backtracking is required--the order property guides a search directly to the appropriate node (which may be external). It is:

BSTT

procedure BSTT(bt,v)                   {O(n)

PROCESS0

p    bt                              {BST Traverse

while p    NIL do

case

p    .Value = v : PROCESSe

return

p    .Value < v : PROCESS1

p    p    .Left

p    .Value > v : PROCESSr

p    p    .Right

endcase

endwhile

PROCESSx

end {BSTT

8.5.2 Insertion and Deletion in a BST

In a BST, the insertion and deletion of nodes must be done in a way that maintains the structural parameter order. Insertion replaces an external node of the extended tree and only involves finding the appropriate parent and which child of the parent the inserted node is to become. Deletion may be of an interior node and involve shifting subtrees. The first task of either is to determine where in the binary tree a node belongs and if that spot is occupied.

Three possibilities need to be determined:

      The value is already in the BST.

      The value is (or would become) the left child of node Pred (if inserted).

      The value is (or would become) the right child of a node Pred (if inserted).

The procedure given here finds the value v in the binary search tree bt. It returns pointer p to the node in bt with value v, the predecessor Pred of p, and a signal Trio. Trio is zero or negative if v was found to be the value of an internal node, and positive if it was not found. Zero values of Trio can occur in two ways: when bt = NIL or when v was found to be the value of the root node. For a non-NIL bt,   [image] Trio  [image]  is 1 if p is the left child of Pred, and 2 if it is the right child.

This value is derived from BSTT by:

PROCESS0:  Trio    0

Pred    NIL

PROCESSe:  Trio    -Trio

PROCESS1:  Trio    1

Pred    p

PROCESSr:  Trio    2

Pred    p

PROCESSx:  NULL

The resulting procedure is:

procedure NodeSpot(bt,v,Trio,p,Pred)   {O(depth)

Pred    NIL

Trio    0

p    bt

while p    NIL do

if v = p    .Value                 {v is an internal value and

then Trio    -Trio              {Trio will not be positive

return

endif

Pred    p

if v < p    .Value

then Trio    1

p    p    .Left

else Trio    2

p    p    .Right

endif

endwhile

end {NodeSpot

NodeSpot is O(depth) for a binary tree of n nodes because it does not need to traverse the structure. The order property allows the algorithm to dispense with backtracking.

The insertion of a non-NIL node n with NIL child links into a BST bt is then done by first invoking NodeSpot(bt,n   [image]  .Value,Trio,p,Pred), followed by:

procedure InsertNode(bt,n,Pred,Trio)    {0(1)

if bt = NIL                           {create a new BST from n

then bt    n

return

endif

if Trio    0                           {v is already present. This

then return endif                   {could be treated as an error.

if Trio = 1

then Pred    .Left    n

else Pred    .Right    n             {Trio = 2

endif

end {InsertNode

For some applications, it is convenient to allow the insertion of duplicate values into an ordered tree. Insertion of duplicate values is not so straightforward as InsertNode, and deletion is much more complex. For example, equal values may be inserted either randomly or consistently to the left or right, but deleting all occurrences of a value presents a problem. In any case, the resulting tree is partially ordered, not ordered.

A deletion procedure must deal with a number of cases. Problem P8.6 asks the reader to try to display all of the cases graphically. It is worthwhile to do so before reading the algorithm given here.

Problem P8.6 is appropriate at this point.

The deletion procedure reads:

procedure CutValue(bt,v)               {O(depth)

if bt = NIL

then {deal with this error.

endif

NodeSpot(bt,v,Trio,p,Pred)

if Trio > 0                          {v was not found in bt.

then {deal with this error.

endif

TackOn    p   .Right                   {TackOn is the subtree that is

 {to replace p as a Pred child.

if p    .Left    NIL

then TackOn    p    .Left

if p    .Right    NIL

then NodeSpot(p    .Left,p    .Right    .Value,Top3,s,sp)

InsertNode(p    .Left,p    .Right,sp,Top3)

endif

endif

if Pred = NIL                        {the root node must go.

then bt    TackOn

else Trio    - Trio                {Specify which Pred child.

InsertNode(bt,TackOn,Pred,Trio)

endif

end {CutValue

Exercise E8.8 and program PG8.4 are appropriate at this point.

8.5.3 Static Tree Files     (optional)

In a complete binary tree of up to n = 1,048,575 nodes, BSTT examines no more than 20 nodes. In contrast, a tree of size n could have depth n and require n tests during a search. If a binary tree (or any tree) is a directory to a file that is to be searched or is the file itself, the shape may be either static or dynamic. The two corresponding classes of files are managed quite differently.

A dynamic file is one that grows and shrinks and changes shape. The shape is normally controlled by maintaining balance, the topic of section 8.7. The benefits of controlling the depth of a file directory can be great, when costs are amortized over a sequence of operations, but the price for maintaining balance lies in the complexity of the insertion and deletion routines. More complex management requires more programming and debugging time and generally more overhead: more run-time with each use of the routines.

A static file is created with a shape that remains fixed. The shape of a binary tree that is the directory to a static file can then be optimized for efficient searching at the time of its creation. The optimum shape is normally determined by the keys of the file--it is data-directed.

Since the inorder traverse of a BST visits the keys in order, the creation of a BST from a collection of keys may be considered a tree sort. However, the insertion sequence determines the shape of the tree and hence the efficiency of subsequent searches. In particular, the worst that can happen with procedure InsertNode is for the input file to be already ordered! (A linear list will result.) Insertion can be designed to produce a well-balanced static BST, and such a procedure is to be found in Part II, section  P.

There are static trees that cannot be balanced because their shape itself carries the information. One such static tree is a search tree for the international Morse code for telegraphy in which letters are represented by dots (short pulses) and dashes (long pulses). If dots correspond in interior nodes to "go left" and dashes to "go right" and letters are leaf nodes, Figure 8.15 shows a small portion of the search tree.

Figure 8.15

Program PG8.5 asks for the construction and use of the entire Morse code tree.

The Morse code is a cryptic form of trie (pronounced "try," not "tree"). Most tries use the sequence of letters in a character string to determine which way to branch at a node. Arrival at a leaf node completes the trace of the spelling of a key, and generally uncovers a pointer to an appropriate record. The nodes, however, necessarily have more than two children and so will be reintroduced in Chapter 9.

The Morse code trie has a property present in some search trees but not others: the links themselves implicitly encode information. In this book the term Code-Link Tree (CLT) is applied to trees with that property. When the links are associated with the digits determined by some number base, a CLT is called a radix search tree.

One way to optimize a static binary search tree involves constructing a CLT with these steps:

1.     Determine the probability (relative frequency) with which each key is searched.

2.     Encode the keys as a string of 0's and 1's so that the more frequently searched keys have shorter codes. Such codes are called Huffman codes.

3.     Construct a CLT from the codes.

A code string of 0's and 1's then leads to a message (a key) at a leaf node.

This approach is used for decoding information that must be transmitted efficiently and for compression of text files in which the keys are characters. It is most effective when the set of search probabilities has a wide range. A discussion of Huffman codes is to be found in Part II, section  Q, which uses material introduced in section 8.6.

8.6 Heaps

One balance condition that was introduced in section 8.1.2 can be rephrased as:

      A binary tree is complete if the nodes of every level in it except possibly the last one have two children, and the nodes of the last level are as far to the left as possible.

One specific structural parameter that determines a subset of the set of complete binary trees is stated as:

      A binary tree is a heap if it is complete and if the value of each node is at least as large as the value of its children.

This subset proves to be very efficient and easy to use in many applications.

The natural way to implement a heap is in an array in heapform (see section 8.2.1), where the child of a node in A[i] is either A[2i] or A[2i + 1]. Heaps are not often implemented in linked form because maintaining the balance condition requires additional information in each node.

If a heap is accessed in specified ways, it becomes one support structure for a priority queue, discussed in section 8.6.2 and applied in Chapter 10.

The heap formed from a set of values is not unique. Faye's Family can be configured as a heap in several ways; one, shown in Figure 8.16, is formed by adding values to an initially empty heap in the order: Faye, Alia, March, Ralph, Wrye, Jan, Will.

Figure 8.16

In contrast, Figure 8.17 shows the heap when it is constructed by adding names in alphabetical order.

Figure 8.17

Exercise E8.9 and problem P8.7 are appropriate at this point.

The heap structure as explored in the remainder of this section is an endogenous heap implemented in an array A[1 . . Max] in heapform, as shown in Figure 8.18 for Alia's heap.

Figure 8.18

Note: It is either the heapform-heap or the priority queue that is usually meant by "heap."

Migration

If a heap contains k - 1 items in A[1 . . k - 1], the value in A[k] may destroy the heap condition if it is included in the structure. If it is allowed to migrate upward, the heap condition can be restored:

procedure UpHeap(k)                                  {O(ln k)

while k    2 do

p    INT(k/2)

if A[k] > A[p]

then Temp    A[k]

A[k]    A[p]

A[p]    Temp

k    p

else exit

endif

endwhile

end {UpHeap

(This routine and those in the rest of this section are best understood by tracing an example.)

Exercise E8.10 is appropriate at this point.

Heap Insertion

A value can be inserted into the heap in A[1 . .n] (supported in A[1 . . Max]), by inserting it at A[n + 1] and reheaping with UpHeap:

procedure InsertHeap(v)                              {O(1n n)

if n    Max

then {deal with this error

endif

n    n + 1

A[n]    v

UpHeap(n)

end {InsertHeap

The insertion of a value, which may destroy the heap condition, followed by restitution of it, is a common feature of heap operations.

Heap Creation

The level-seeking of UpHeap can be used to heap an arbitrary array of values A[1 . . n].

procedure MakeHeap                                   {O(n ln n)

for k = 2 to n do

UpHeap(k)

next k

end {MakeHeap

The timing function of this procedure is T(n) = (n - 1) tk, where tk is the time required to execute UpHeap(k). The number of times the if-statement of UpHeap is executed can be no more than the maximum level in the heap, which is bounded by INT(1n n). Hence T(n) is 0(n 1n n).

One reason that heaps prove to be valuable in practice is the relative efficiency of heap operations.

Priority Retrieval

The largest value (or smallest, or whichever value is heaped to the top) is retrieved and deleted from the heap but not the array by: switch A[1] and A[n], decrement n, and reheap.

Establishment of the heapform after this swap, however, requires that A[1] trickle down to its proper level. This downshift can be done with:

procedure SwapDown(k)                                {O(1n n)

while k < n do

j    2 * k

if j > n then exit endif

if (j < n) AND (A[j] < A[j+1])

then j    j + 1

endif

if A[k] < A[j]

then Temp    A[k]

A[k]    A[j]

A[j]    Temp

endif

k    j

endwhile

end {SwapDown

This process can be made somewhat more efficient by shifting values instead of swapping them, as is done in procedure DownHeap that follows. The kth item is shifted no farther than n with:

procedure DownHeap(k,n)                              {O(ln n)

j    2 * k

Half    k

Temp    A[k]

while j    n do

if (j < n) AND (A[j] < A[j + 1])

then j    j + 1

endif

if Temp    A[j]

then A[Half]    Temp

return

endif

A[Half]    A[j]

Half    j

j    2 * j

endwhile

A[Half]    Temp

end {DownHeap

Now retrieval of the value with the highest priority (the top value) becomes:

function DeHeap                                      {O(ln n)

Temp    A[1]

A[I]    A[n]

A[n]    Temp

n    n - 1

DownHeap(1)

return A[n + 1]

end {DeHeap

A heap is created by repeated use of DownHeap, applied to A[i], for i = Half, Half - 1, . . ., 1, where Half = INT(n/2).

Exercise E8.11 is appropriate at this point.

Exogenous Heaps

The attempt to adapt the procedures of this section to pointer-variable form faces a fundamental difficulty: where is the attachment point for the node that is to be added to the heap next? A pointer to the current attachment point does not lead readily to the next one. A workable approach is to maintain a field in each node that contains a weight (the number of internal nodes) in the subtree rooted at the node. If the weights of the children are not equal and if the weights of the two left internal grandchildren are equal, the attachment point for insertion is to the right. Deletion involves location of the last-attached node with a test similar to the one used for insertion.

An alternate approach is to use an array-based heap of pointers to data records, with the heap nodes treated as discussed in this section. In effect, this approach forms an exogenous heap; the data records do not need to contain pointers and are not linked into a tree at all. The heap of pointers is a directory to an otherwise unstructured collection of records. It should be clear that any structure in this book can be used in a like manner.

8.6.1 HeapSort

Procedure DownHeap is easily turned into a sorting procedure of order O(n 1n n). The largest (highest-priority) item is moved to the top of the heap, because it is the child of some i and is swapped with A[i], then with A[i] for j = INT(i/2), and so on until it is exchanged with A[i]. It is removed (moved to the end of an active list) and the remainder reheaped to find the next-largest item. Repetitions locate the values in nonincreasing order:

procedure HeapSort(A,n)                              {O(n In n)

Half   INT(n/2)

for i = Half downto 1 by -1 do

DownHeap(i, n)

next i

for i = n - 1 downto 1 by -1 do

Temp    A[i+1]

A[i+1]    A[1]

A[1]    Temp

DownHeap(1, i)

next i

end {HeapSort

The largest item is placed in A[n], the next largest in A[n - 1], etc.

The pointer-variable form of HeapSort adds the root node to a linear list as it is removed. In any case, HeapSort has a destructive readout in the sense that the heap is destroyed in the process.

The importance of HeapSort is that few sorting procedures are O(n 1n n). Although QuickSort is O(n 1n n) on the average, it is O(n2) in the worst case. InsertionSort, BubbleSort, and SelectionSort are all O(n2). A sorting procedure safe and suitable for large sets of data is often not encountered in programming courses that do not have data structures as a prerequisite.

Exercise E8.12 and program PG8.6 are appropriate at this point.

8.6.2 Priority Queues

In a variety of applications it is desirable to rcpeatedly insert items into a set from which some optimal value (called the largest) can be extracted. (Here "largest" may refer to the item itself or a key field in a record.) An abstraction of such a structure is simply the set of operations that can be applied to it. In this sense, the structure priority queue is a set of data to which can be applied:

      Create a priority queue from n items.

      Insert an item.

      Remove the largest item.

      Replace the largest item big by v if v < big.

It can be useful to apply the following operations that are more difficult to implement efficiently:

      Change the priority (size) of an item.

      Delete a specified item.

      Join two priority queues into one.

There are implementations in which most of these operations are O(ln n), which is one reason for applying priority queues.

Three implementations of priority queues can be based on structures introduced in previous sections:

1.     An array of pointers to priority-class queues, each an instance of QUEUE--a pure queue. This implementation would likely be applied for a fixed number of classes, and most operation procedures would involve a binary search within the sorted array of k class headers. With natural algorithms, Create is O(n 1n k); Insert, Remove, and Replace are O(1n k). Change and Delete depend on a search of the individual class queues and so are O(m + 1n k), where m is the maximum size of a class queue. The operation JOIN is subject to interpretation in this context but is not likely to be as efficient as the other operations.

2.     A heapform-heap. Create is O(n 1n n); Insert, Remove, and Replace are all O(1n n). Change and Delete are simply linear searches, O(n). Join consists of adding the items in heap H1 to heap H2, where n1 < n2 are their sizes. It is then O(n1 1n (n1 + n2)). A heapform-heap is the most common implementation for a priority queue.

3.     A pointer-linked binary tree heap. Create is O(n 1n n). Insert and Replace are O(1n n). Change and Delete involve a binary tree search and so are O(n) at best. Delete and Remove would involve rebalancing of the binary tree if the balance condition is not relaxed. If balance is abandoned, the other operations approach O(n) instead of O(1n n). Join becomes O(m 1n n), where m = Min{n1, n2)}, n = n1 + n2, and n1 and n2 are the sizes of the heaps to be joined by insertion of the items of one heap into the other.

There are implementations other than the three above, and some can be useful:

4.     The degenerate priority-class queue. This implementation has only one item in each class; it is a sorted list of items in an array.

5.     A sorted linked list of items. Create is in effect InsertionSort, O(n2). Insert and Replace are O(n) but Remove is O(1). Change and Delete are essentially O(n) linear searches. Join is likely to be repeated insertion, O(m * n), where n = n1 + n2 and m = Min{n1, n2}.

6.     An unsorted linked list of items. Insert becomes O(1) and hence Create is O(n). If the position of a specified item is not known, Replace and Remove are O(n) linear searches, as are Change and Delete. Join becomes a trivial O(1) operation (as are Change and Delete if a pointer to the specified item is provided).

7.     An unsorted array of items. Create is not required, Insert is O(1). Replace, Remove, and Change are O(n) linear searches. Delete is an O(n) linear search followed by swapping the found item with A[n] and decrementing n. The JOIN operation is ambiguous or simply copies all of the items into some array.

Any priority queue can be used in a sorting algorithm by repeated application of Remove, but that does destroy the copy of the structure being used. This process is, of course, HeapSort for the heapform heap implementation.

8.6.3 On the Use of Priority Queues

A collection of items that differ in some quality used to select one of them for an application is a natural candidate for retention in a priority queue. The priority queue is the data structure that exemplifies the pair of demands: "Save this with the others . . . . Find the best one available."

For specific applications and restrictions, some of the data structures discussed in previous sections and in Chapters 9 and 10 are ideal. For the general situation where items are to be stored and retained in some order determined by their own qualities, the priority queue is the structure of choice. Above all, a priority queue is a structure of general utility and great flexibility.

Priority queues are applied to the construction of Huffman codes in Part II, section  Q, and used in the general traverse of a graph in Chapter 10.

8.7 Balanced Binary Trees

A general binary tree is of the form shown in Figure 8.19, where L and R are subtrees.

Figure 8.19

The root node may be considered as a pivot about which the binary tree it roots can be balanced or unbalanced by some criterion. Since this scheme applies at every node in a binary tree, balance is a local effect. It becomes a global effect with the restriction that a tree not be unacceptably out of balance at any node. Global balance is important for search trees because it determines the amortized cost of a sequence of searches.

Ways to measure subtrees L and R for comparison with each other that have been applied include:

      height (depth)

      weight (the number of external nodes)

      internal path length (IPL)

Perfect balance is too much to ask because it generally is present only for complete trees, and so balance criteria are designed to optimize the balance for arbitrary trees. Insertions and deletions can change the balance at arbitrary points within a binary tree, for better or worse. When an operation changes the balance to the point that it needs to be restored, rebalance is done by rotation about a pivot node.

Suppose, for example, that the binary tree shown schematically in Figure 8.20 is balanced at n and at p, but an insertion is made into subtree R2 that causes an inbalance (by the criterion in use).

Figure 8.20

Then a single left rotation (SLR) can be used to restore balance and maintain the order condition, as shown in Figure 8.21.

Figure 8.21

As a more specific example, the effect of an SLR on the tree in Figure 8.22 helps its balance (by almost any criterion).

Figure 8.22

Sometimes a single rotation is not sufficient, and a double rotation is required, as shown in Figure 8.23.

Figure 8.23

For a specific example, a double rotation improves the balance of the tree in Figure 8.24 (by almost any criterion).

Figure 8.24

The right-rotation forms are symmetric to the left-rotations illustrated. Both single and double rotations are designed to maintain the order of the keys.

Since rotations may take place around any node in the tree, the parent of the pivot node is involved: the pivot node is either its left or its right child, and that pointer is changed. For the sake of clarity, it will be assumed that the pivot node is the root in the discussion that follows, and all the subtrees shown are non-NIL.

With these simplifying assumptions, SLR becomes:

SLR:  p   .Right    n   .Left

n   .Left    p

{switch parent-of-p pointer to n

Similarly, DLR is:

DLR:     p    .Left    n    .Right

gp    .Right    n   .Left

n    .Right    p

n    .Left    gp

{switch parent-of-gp pointer to n

In practice, a number of special cases must be dealt with.

Several general features of binary tree balancing are of interest. Insertion is always made at a leaf node. For height- and weight-balance criteria, insertion requires no more than one rotation (if any) in order to restore balance. With IPL balance, rotations can propagate down the tree.

Deletion can always be of a node with no more than one internal child. If the search for a deletion key locates it at node p, then in the sequence of values that determines the tree, the values adjacent to p   [image]  .Key are

Pred   .Key < p   .Key < Succ   .Key

where Pred is the node in the left subtree of p to which p would be added by an insertion. Similarly, Succ is the node in the right subtree of p to which p would be added by an insertion. Insertion always replaces an external node, which has a parent with no more than one internal child. Consequently, the p   [image] .Key can be swapped with Pred   [image] .Key (or with Succ   [image] .Key if p   [image]  Left is external), and the latter deleted.

Rotations caused by deletion can propagate up along the sequence of nodes in the search path to the deleted node. Reversing this path involves stacking the search path as it is generated, or retracing it, or building the tree with parent links in the nodes, or reversing links as the path is traced (a technique used in Part II, section  M) and then restoring them.

The nodes of a balanced tree generally contain information fields that indicate the state of balance at a node. Rotation, insertion, and deletion procedures maintain the fields to reflect altered conditions.

Perhaps the easiest dynamically balanced tree structure to maintain, and one of the most often applied, is the B-tree discussed in section 9.5.

A tree structure that is reformulated as a binary tree and then balanced is the Red-Black Tree of section 9.6 and Part II, section  U.

Further details about height-balanced trees, commonly called AVL trees, are to be found in section 8.7.1. Details of the management routines for them are to be found in Part II, section  R. Weight-balanced trees, commonly called BB(  [image]) trees, are discussed in section 8.7.2.

IPL-balanced trees are treated in [GONNET, 1983]. Their advantage is in management routines that are simpler than for AVL trees, and their disadvantage is in rotations that can propagate down the tree.

The analysis of the cost of maintenance procedures for various balance criteria, their amortization, and their relative merits lie outside the aims of this book. As pointed out elsewhere [GONNET, 1983], research is yet to be completed on some questions concerning:

      the (worst case, average, or amortized worst case)

number of (rotations, node inspections)

to (search, insert, or delete)

a key in (AVL, BB(  [image]), or IPL)

trees.

8.7.1 AVL Trees     (optional)

8.7.2 BB(a)Trees     (optional)

8.7.1 AVL Trees     (optional)

Height-balancing was achieved by G. M. Adel'son-Vel'skii and E. M. Landis in 1962, and hence such trees are called either AVL trees or height-balanced trees.

If bt is a binary tree with subtrees L and R, then bt is height-balanced (has the AVL property) iff:

1.       h(L) -  h(R)|   [image]  1, where h is the height function.

2.     L and R are themselves height-balanced.

The tree in Figure 8.25 fails to be height-balanced at E, and hence G, although it would be 1-balanced at G on the basis of the relative heights of its subtrees alone.

Figure 8.25

A detailed treatment of the management routines for AVL trees is to be found in Part II, section  R.

8.7.2 BB(  )Trees     (optional)

Suppose that bt is an extended binary tree with left and right subtrees L and R. With the notation:   T  [image]  for the number of external nodes in tree T, the weight-balance of bt is:

  (bt) = 1/2 if bt is empty, and

otherwise

Clearly,   [image] (bt) = 1/2 implies perfect balance and it must be true that

0 <   (bt) < 1

The parameter that measures how close to 1/2 the balance is to be maintained is   [image]. A tree is of weight-balance   [image]  or in the set BB(  [image]) if:

1.             (bt)   [image]  1 -   [image]  and   [image]    [image]  1/2

2.     if bt is not empty, L and R are in BB(  [image]).

Clearly,   [image]  = 0 is no restriction and   [image]  = 1/2 asks for perfect balance. BB(x) is a subset of BB(y) if x   [image]  y. It happens that if 1/3   [image]    [image]    [image]  1/2 , then BB(  [image])1/3 = BB (1/2).

The   [image]  values of the nodes of a tree determine its   [image] -class (which it may be desirable to adjust after insertions and deletions). The nodes in Figure 8.26 have their   [image]  value as identifier:

Figure 8.26

The details of managing BB(  [image]) trees are similar to those for AVL trees described in Part II, section  R.

Exercise E8.13 and problem P8.8 are appropriate at this point.

8.8 Threaded Binary Trees     (optional)

A binary tree can be traversed without the use of (implicit or explicit) stacks in several ways. One way is with the use of threads.

A thread is created by replacing NIL (external) links. A NIL right link is replaced with a pointer to the node that would be processed next in an inorder traverse. A NIL left link is replaced by a pointer to the node that was processed last. The leftmost left link and the rightmost right link are set to NIL.

A threaded version of BT8 is shown in Figure 8.27.

Figure 8.27

The nodes of a threaded binary tree must be designed to implement threads. The links and the threads need to be distinguished. They are distinguished in an array implementation by using negative indices tor links and in a pointer implementation by using flags:

ThreadNode = RECORD

Value  : {ValueType

Left   :    ThreadNode

Lflag  : BOOLEAN

Right  :   ThreadNode

Rflag  : BOOLEAN

END    {ThreadNode

It is convenient to initialize the flag fields to TRUE, implying that the associated fields are pointers, not threads.

An unthreaded binary tree where the nodes are of type ThreadNode can be threaded by calling Baste(bt,NIL,NIL):

procedure Baste( p,Pred,Next)                        {O(n)

if p = NIL then return endif

if p    .Left = NIL

then p    .Left    Pred

p    .Lflag    FALSE

else Baste( p    .Left,Pred,p)

endif

if p    .Right = NIL

then p    .Right    Next

p    .Rflag    FALSE

else Baste( p    .Right,p,Next)

endif

end {Baste

Exercise E8.14 is appropriate at this point.

A traverse of a threaded tree follows the standard pattern of: "step left as far as possible, step right one link, move to the appropriate successor." When a node is reached by following a normal link, its left subtree is to be explored. If it is reached by a thread, its left subtree has already been explored. Processing of nodes can be in postorder, preorder, or in the inorder shown on the next page.

procedure InThread(bt)                               {O(n)

GoLeft    TRUE

p    bt

while p    NIL do

if GoLeft

then while p    .Lflag do

p    p    .Left

endwhile

endif

PROCESS2(p)

GoLeft    p    .Rflag

p    p    .Right

endwhile

end {InThread

Insertion of a node into a threaded binary tree raises the question of placement--of where the insertion is to take place. Balancing and order are not related to the choice between external nodes and threads and must depend on applications. Threads have only a minor affect on the emplacement of a new leaf node. To illustrate, consider the insertion of a node q as the right child of a leaf node p in a threaded binary tree:

procedure ThreadRight(p,q)

q    .Lflag    FALSE

q    .Left    p

q    .Rflag    p    .Rflag

q    .Right    p    .Right

p    .Right    q

end {ThreadRight

Adding a node as the left child of a leaf node is similar and is left as a problem.

Problem P8.9 is appropriate af this point.

Deletion of nodes from a threaded binary tree (any binary tree) depends upon shape parameters that are heavily application dependent. The crucial issue is: what should be done with the subtrees of the excised node? The possible responses to this question differ only in detail from those applied in unthreaded trees.

Program PG8.7 is appropriate at this point.

Summary

Trees in the mathematical sense of a graph without cycles are a common form, and they can be represented in a program with the structure BINARY TREE. The nodes of a binary tree have two distinguished (left and right) child nodes. Like lists, stacks, and queues, the binary tree is dynamic and may be either endogenous or exogenous.

An important feature of binary trees is that every node is accessible from the root node by following links. There are several ways to traverse a binary tree, either recursively or iteratively. One is the universal iterative binary tree traverse BTT, which allows processing a node at each of three contacts with it during the traverse. Many application algorithms can be based on BTT or on its recursive counterpart RBTT.

Traverses more specialized than BTT process a node on only one of the traverse contacts with it, the standards being: preorder, inorder, and postorder traverses. The recursive forms of these traverses are especially handy for quick application. The iterative forms of the standard traverses require explicit stacks that are less extensive than those in the corresponding recursive forms.

A breadth-first traverse uses a queue to traverse a binary tree level-by-level. When generalized, it becomes the priority-queue traverse of a graph in Chapter 10.

Reaching a sought-for node in the minimum number of steps requires that a decision to traverse either a left subtree or a right subtree be made at each node in the walk. A binary tree with nodes ordered so that node keys direct the walk efficiently is a BST (Binary Search Tree).

An array support system for binary trees called heapform is efficient and of great utility. When order and completeness are included in a binary tree supported in heapform, it is a heap. (A heap may also be supported with pointers, but not conveniently.) Heaps are a preferred support structure for priority queues, which in practice replace several of the simplified application models presented in this book. Note the layers of structure: (array and binary tree)   [image]  heapform   [image]  heap   [image]  priority queue.

Heap algorithms can be assembled into HeapSort, a sorting method guaranteed to be O(n In n) in the worst case. This is the theoretical optimum.

Binary trees are heavily used as directories to (endogenous or exogenous) files and hence repeatedly searched. The cost of searches must be amortized over a sequence of operations.

For static files, the shape is determined at creation and must be optimized at that point. A general balancing insertion algorithm is to be found in Part II, section  P. A highly tuned static file technique, the use of Huffman codes, is applied to file compression and decoding of messages. The algorithms are to be found in Part II, section  Q.

Dynamic files change shape with the operations of insertion and deletion, and so those operations are often sophisticated in order to maintain balance. Height, weight, and internal path length are all applied to devise balance criteria. Height-balanced trees (AVL trees) are detailed in Part II, section  R. A more general structure, the B-tree, is described in Chapter 9.

Binary tree traverses can be directed without stacks by replacing NIL pointers with threads that point back to appropriate ancestors, at the cost of the initial threading.

Exercises

Exercises immediate applications of the text material

E8.1     Suppose that the vowels form a tree with "O" as the root, and its children are "U", "I", "A", left-to-right and "E" is the only child of "I". Restructure this tree as a binary tree by the transformation rule given in the introduction.

E8.2     How many ways are there to arrange the vowels "A", "E", "I", "O", "U" in a binary tree so that if C1 is in the left subtree of C2, then C1 < C2, and if C3 is in the right subtree of C2, C3 > C2.

E8.3     Derive the level of each node for the tree in Figure 8.28, the depth of the tree, the IPL, and the EPL. Modify the tree to find variations with the same set of keys that are full, or complete.

Figure 8.28

E8.4     Show how the vowel binary tree of E8.1 would appear in single-array and three-array form. (In the three-array form, the vowels are to be stored in alphabetical order.)

E8.5     Add parenthesis nodes to BT9 (section 8.3.2) so that the inorder trace produces the correct infix expression.

E8.6     Trace the action of procedure PreStack on BT8 by indicating the stack contents and p each time they change.

E8.7     There is a return on entrance to PostStack, but it is not required in InStack and PreStack. Why is this so?

E8.8     Trace the insertion of T-I-P into the BST of Figure 8.29, followed by deletion of E and O:

Figure 8.29

E8.9     Construct the heap H2 of section 8.6 in stages, as was done for heap H1.

E8.10     Trace procedure UpHeap of section 8.5 on the letter sequence T-R-A-S-H-D-U-M-P, applied for k = 2, 3 . . . 9.

E8.11     Trace the action of procedure DownHeap (k,n) for k = 3,2,1 on Faye's Family, initially stored in alphabetical order.

E8.12     Trace the action of HeapSort on an array containing the letter sequence of E8.10.

E8.13     Calculate the weight-balance factors for all nodes of the binary trees BT1-BT6.

E8.14     Trace the values of the parameters p, Pred, and Next in procedure Baste of section 8.8 when it is applied to BT8. Show them as they would appear on top of the stack created by recursive calls and indicate thread assignments.

Problems

Problems not immediate, but requiring no major extensions of the text material

P8.1     Develop the induction proofs of the two level-count results of section 8.1.2.

P8.2     Suppose that pointers require two bytes of storage and values require four. How much total storage is required for a binary tree of N items? Derive answers for pointers, three-array, and single-array storage.

P8.3     Derive a scheme for adding parentheses to the display of a binary tree in which interior nodes represent binary operators so that the display traverse produces the correct infix expression.

P8.4     Design a procedure that displays node values on all three contacts with a node, then prints an encounter index under the display. (See the discussion of visitation order in section 8.3 for example output.)

P8.5     Design an iterative version of function TreeCopy of section 8.3.3.

P8.6     Redesign procedures NodeSpot and InsertNode of section 8.5.2 so that values may be duplicated in a (partially) ordered binary tree.

P8.7     How many distinct heaps of Faye's Family are there?

P8.8     Design a procedure to calculate weight-balance factors for an arbitrary binary tree.

P8.9     Write the procedure that inserts node q as the left child of a leaf node p in a threaded binary tree.

Programs

Programs for trying it yourself

PG8.1     Write a program that inputs a binary tree (assumed to be correctly linked by the user) and displays the PreOrder, InOrder, and PostOrder traverses of it. (In a language that fails to support recursion, the results of section 8.4.1 are required.)

PG8.2     Repeat PG8. I with the iterative procedure BTT and display the stack at an appropriate point if requested by the user.

PG8.3     Write a program that will accept a list of values, treat them as a heapform binary tree, and display them both top-down and sideways.

PG8.4     Write a command-shell program that manages a binary search tree, allowing display, insertion, deletion, and location of values.

PG8.5     Write a program that constructs the International Morse code tree introduced in section 8.5.3 from the table below and allows users to enter a code message of dots, dashes, and separators. The output of the program is the decoded message.

A   -     B -           C -  -      D -         E

F     -      G --       H              I           J   ---

K -  -    L   -         M --     N -       O ---

P   --      Q --  -    R   -       S            T -

U     -     V       -     W   --     X -    -   Y -  --

Z --

PG8.6     Write a program that inputs 10 integers and sorts them with HeapSort.

PG8.7     Write a program that creates a threaded tree by accretion (but does not support deletion) and displays the new tree.

Go to Chapter 9            Return to Table of Contents


Data Structures: CHAPTER 9: MULTIPLE ACCESS PATHS TO DATA

 Return to Table of Contents

CHAPTER 9: MULTIPLE ACCESS PATHS TO DATA

ARRAY supplies an individual access path for each of its data cells. STACK and QUEUE supply only one access point for data retrieval and hide data not at that point from the user. A linked list provides one general access point (the head) and one limited one (the tail), but it does provide a traverse path to any cell. The traverse path is unique for singly linked lists, and varied for doubly linked and circular lists. BINARY TREE provides only one entry point, but any cell can be reached from it. Without the help of a stack or other external structure, the access path to any cell in a binary tree is unique. This chapter explores some of the ways to enhance the access to structured data.

Multiple access to the cells of a binary tree for backtracking to the parent of a node can be provided with two-way links, in analogy with doubly linked lists. One effect is the removal of stack action in the general traverse, which decreases execution overhead at the expense of extra space in the nodes. Two-way binary trees are discussed in section 9.3, and parent-links are applied in several later sections. Parent links may even replace the child links, as they do in the representation of sets in section 9.7.

Two forms of generalized access should not be overlooked: indirection and external storage.

External storage has been a background theme for all of the structures studied in this book because access to data stored on secondary storage is not done on the same time scale as it is for main storage. Just as important is the massive size of the data files that may be stored externally. The choice of data structure to be used for external storage can be profoundly affected by the need to deal with data sets much too large for main memory. An approach to sorting too much data to retain in main memory is outlined in section 9.2, where it leads to merging and merge- sorting in Part II, sections S and T. Some of the structures that provide access to external data are studied in sections 9.4, 9.5, and 9.6 .

BINARY TREE provides only two children at each node. An obvious generalization is to structure nodes with more than two children. Having more than two children provides quicker access to the deeper levels of the tree. External files are commonly accessed through a directory with such multichild nodes in order to decrease the number of levels used to reach a given number of nodes.

Applications of multiway tree structures include m-way structures in section 9.4, B-trees in section 9.5, 2-3-4 trees in section 9.6 and Part II, section U, and the decision tree of section 9.8 and Part II, section V.

Finally, if a data structure is large, it may be desirable to limit access to part of it--to prune it. That is the theme of (multiway) game trees, section 9.9 and Part II, section W.

9.1 Access by Indirection

One major activity applied to data is sorting, which generally reduces to the sorting of records by key. It is not uncommon to need access to the data in complex records, such as personnel records, by way of more than one key. Sophisticated ways to access data interrelated in complex ways have been developed under the general heading of data bases. A much simpler approach can be of value: indirect sorting.

As a focus, consider an array of student records, a[l . . n], each containing a name and an identification number. If the records are already sorted physically in ID-number order, the names are surely out of alphabetical order, and locating a name becomes an O(n) operation rather than a binary search of O(ln n) cost. It is desirable to provide an ordered appearance for a name search without physically moving the records out of ID-number order.

Any sorting algorithm can be implemented indirectly; and, for contrast, both a direct and an indirect sort with the same algorithm are developed here.

A sorting procedure that compares favorably in practice with those of Chapter 1 is the ShellSort, developed by D. H. Shell. It is O(n2) but may be ten or more times as fast as the other methods are for a few hundred items [KERNIGHAN, 1974]. For large data sets, it is less efficient than HeapSort, but easier to get right. ShellSort acts like BubbleSort except that, as it makes a sweep, it compares a[i] with a[i + Gap], where Gap is initially INT(n/2) instead of 1. If switches occur on a sweep, it resweeps with the same value of Gap; otherwise, it halves Gap and repeats.

procedure ShellSort(a,n)                     {O(n2)

Gap    n

while Gap > 1 do

Gap    INT(Gap/2)

Bound    n - Gap

repeat

Exchanges    0

for Lower = 1 to Bound do

Upper    Lower + Gap

if a[Lower].Name > a[Upper].Name

then Temp    a[Lower]

a[Lower]    a[Upper]

a[Upper]    Temp

Exchanges    Exchanges + 1

endif

next Lower

until Exchanges    0

endwhile

end {ShellSort

To do this procedure by indirection, an array of indices to records in the array must be first created. (Pointers to scattered records would also serve.) The indices are initialized to the initial position of the records:

procedure MakeMap

for i = 1 to n do

Map[i]    i

next i

end {MakeMap

Immediately after execution of MakeMap, a[i] and a[Map[i]] are identical. However, the indirect sort compares a[Map[i]] to a[Map[j]]; and, if a switch is indicated, it is Map[i] and Map[i] that are switched, not the records themselves. (This technique works for the sorts in Chapter 1 as well.)

procedure VirtualShell(a,Map,n)                 {O(n2)

Gap    n

while Gap > 1 do

Gap    INT(Gap/2)

Bound    n - Gap

repeat

Exchanges    0

for Lower = 1 to Bound do

Upper    Lower + Gap

if a[Map[Lower]].Name > a[Map[Upper]].Name

then TempDex    Map[Lower]

Map[Lower]    Map[Upper]

Map[Upper]    TempDex

Exchanges    Exchanges + 1

endif

next Lower

until Exchanges    0

endwhile

end {VirtualShell

After execution of VirtualShell, the records may be displayed in the order determined by their Name fields by:

procedure MapOut

for i = 1 to n do

NodeOut(Map[i])  {display routine for a node

next i

end {MapOut

Exercise E9.1 and problem P9.1 are appropriate at this point.

9.1.1 The Indirect Heap

HEAP is a structure that involves a great deal of data movement when it is managed directly. Actually, it is only the address of data that needs to be provided in heap form, just as a data-set is defined to be fully sorted for a user by an appropriate sequence of addresses. The creation of a heap from the data used for ShellSort follows the procedures in section 8.6 rather closely:

procedure UpHeap(k)

while k    2 do

p = INT(k/2)

if a[Map[k]].Name > a[Map[p]].Name

then TempDex    Map[k]

Map[k]    Map[p]

Map[p]    TempDex

k    p

else exit

endif

endwhile

end {UpHeap

procedure MakeHeap

for k = 2 to n do

UpHeap(k)

next k

end {MakeHeap

The Name indexed by Map[1] is the top name, whatever its index in a[l . . n].

Finally, it is possible to keep an inverse map. For example, if Track[i] is initialized to i for i = 1,2, . . .,n, then the location of a[j] within the indirect heap, Map, can be followed by adding a small amount of overhead to the switch:

if a[Map[k]].Name > a[Map[p]].Name

then Track[Map[k]]    p

Track[Map[p]]    k

TempDex    Map[k]

Map[k]    Map[p]

Map[p]    TempDex

k    p

else exit

endif

After execution of MakeHeap, the location of a[i] in the (indirect) heap Map is Track[i]. The parent and child nodes and the level of a[i] can all be derived from Track[i] .

Indirection can be a powerful tool for preserving both storage and time, simply because pointers are relatively small.

Exercise E9.2 is appropriate at this point.

9.2 External Files

Data is stored on disks or tapes in order to preserve it beyond the life of a program execution, but these storage media have been developed with the secondary goal of providing cheap bulk storage. The $2 disk on which this sentence was stored in a portable personal computer holds 195,000 bytes on a side. That is 195,000 characters, or 3,250 sentences like this one. That is enough for a file containing this chapter. The main memory in which such a file would be edited holds an operating system, an editing program, and only a portion of a chapter file. A search through such a file (for misspelled words, perhaps) would require a number of transfers of data between main and secondary storage. Direct access to data in secondary storage may be slower than access to data in the main store of a computing system by a factor of 100 or 1,000 or 10,000. Both their bulk and the time required to access external storage have an impact on the choice of data structures applied to it.

External files with exogenous and/or balanced search-tree directories can combine the speed of main memory with the bulk storage capability of external storage. A common form for such files is that of the m-way trees of section 9.4 and the B-trees of section 9.5, and this is supplemented by material on balanced binary trees in section 8.7; Part II, sections P and R, and Red-Black (2-3-4) trees in section 9.7 and Part II, section  U.

One approach to organizing large sorted sequential files that has been applied frequently is similar to the use of index tabs in dictionaries and physical files: quick access is provided to a directory of segments of a file. A key can be efficiently placed within a particular segment, and the segment accessed for further search. This organization is called an indexed-sequential file.

An indexed-sequential file is sorted as a whole. Sorting such a file, or any large file, may need to be done in manageable pieces much smaller than the file itself. The pieces are then merged to form a single file sorted as a whole.

9.2.1 The Merge Operation

Suppose that two arrays, sorted in nondecreasing order, a[1 . . m] and b[1 . . n], are to be written as a single sorted list on a file. (The output file is taken here to be the standard output file.) This merging operation can be done simply by examining the next item in both arrays and writing the smallest of them:

procedure StraightMerge                       {O(n + m)

if a[m]    b[n]

then Sentinel    a[m] + 1

else Sentinel    b[n] + 1

endif

a[m + 1]    Sentinel

b[n + 1]    Sentinel

i    1

j    1

for k = 1 to n + m do

if a[i] < b[j]

then Write(a[i])

i    i + 1

else Write(b[j])

j    j + 1

next k

end {StraightMerge

Variations of StraightMerge can be designed for linked lists, for more efficient operation when one run is larger than another, and as a sorting procedure. Further discussion is to be found in Part II, section  S.

When a large file is to be sorted by merging techniques, the efficiency of the complete operation is affected by the size of the sorted segments, called runs. Runs are merged to form larger runs, until there is only one. As the runs increase in size, they cannot be in main memory, and so data is read from two or more files and written onto another in sorted order. The operation of creating runs from sequential files is enhanced by the application of a priority queue. Details are to be found in Part II, section  T.

Exercise E9.3 is appropriate at this point.

9.3 Doubly Linked Two-Way Trees

As noted in Chapter 8, trees and binary trees are not the same structure, but the relationships between the nodes of a tree can be retained in a binary tree. Clearly, any binary tree becomes a tree simply by ignoring the distinction between left child and right child. In that sense, a binary tree is a restricted form of tree. The three common restrictions are:

      A link is directional from parent to child.

      There are no more than two children per node.

      Children of a node are distinguished.

The middle restriction can be restated as:

      The degree of each node of a binary tree is no more than 2.

(The degree of a node is the number of links it has to other nodes.) Trees of degree m are used in m-way search trees, in direct analogy with the binary search trees of section 8.5.

Removal of the first two restrictions together with the assumption that a child has but one parent provides a structure that is commonly used to manage sets (see section 9.7). Removal of the last two restrictions provides a family of structures used in decision making and in the analysis of games (see sections 9.8 and 9.9). Removal of all three restrictions creates a graph structure, the subject of Chapter 10.

An increase in the number of children of each node tends to decrease the depth of a tree with a given number of nodes. The depth of a tree determines the maximum number of links that must be traversed in search, insert, or delete operations. Tree depth can be changed by either an insertion or a deletion, which makes it desirable to formulate these operations so that paths from the root to all leaves tend to be the same length. It is possible to balance the root-leaf paths of general tree structures, much like binary trees are balanced in the AVL structure of section 8.7.1. The structures introduced in this chapter provide alternates to AVL trees and randomly formed binary trees, and their balance is a major theme.

Exercise E9.4 and program P9.2 are appropriate at this point.

The mathematical structure of a tree with no more than two children differs in two ways from the computing structure BINARY TREE. One difference is the distinction between left and right subtrees in the binary tree structure. The other difference is that the links in binary trees are directional, and the edges connecting nodes in a mathematical tree are two-way connections. When restriction is made to directional links in a mathematical structure it becomes a directed graph or digraph. A linked list necessarily forms a digraph because links are directional, but when each link is paired with a link in the reverse direction, the pair models an undirected, two-way link.

Figure 9.1 is a valid directed graph. Directed graphs are very general structures, and so general algorithms to traverse them tend to be either more complex or slower than algorithms dealing with structures that have a higher degree of internal structure.

Figure 9.1

Removal of the one-way linkage restriction on binary trees is well worth considering for any application, particularly if the tree is exogenous. In that case, the Value field of a node record is a pointer to a storage record, and the storage required for the link does not increase the overall storage requirements a great deal. One additional link in each node can be a great help in the management of a structure:

TwoWayType = RECORD

Parent :    TwoWayNode

Value  : {value type or pointer type

Child  : ARRAY[1 . . 2] OF    TwoWayNode

Visit  : INTEGER

END

A Visit field is included because it is useful for indicating which subtrees of a node have been explored. A recursive traverse of a tree twt built of TwoWayType nodes does not differ from the recursive traverse of a binary tree, but the extra link simplifies an iterative traverse. (We assume in the following that the parent field of a root node (only) is NIL.) Only minor modification of BTT of section 8.4.1 is needed to walk a tree formed from such nodes.

In the traverse TwoWayVisit that follows, p   [image]. Visit is set to zero in each node when the traverse is complete. If the tree was initiated that way, TwoWayVisit leaves it unchanged. Clearly the Child field can be expanded to include more children.

TwoWayVisit includes the cases of preorder, inorder, and postorder processing by applying distinct processes at each of the three encounters with a node.

procedure TwoWayVisit(twt)                {O(n)

p    twt

while p    NIL do

case of p    .Visit

0 : PROCESS1(p)

p    .Visit    1

q    p    .Child[1]

if q    NIL

then p    q

endif

1 : PROCESS2(p)

p    .Visit    2

q    p    .Child[2]

if q    NIL

then p    q

endif

2 : PROCESS3(p)

p    .Visit    0

p    p    .Parent

endcase

endwhile

end {TwoWayVisit

To make a search of a two-way tree for a given value Wanted, it is only necessary to let TwoWayVisit be a function, let processes 2 and 3 be null processes, and let PROCESS1 be:

if p    .Value = Wanted then return p endif

Exercises E9.5 and E9.6 and problem P9.3 are appropriate at this point.

An immediate generalization in to assume x children and use Child[0] as the parent link:

procedure xWayVisit(xwt)                  {O(n)

p    xwt

while p       NIL do

with p do

Visit    (Visit + 1) MOD (x + 1)

PROCESS(Visit,p)

q    Child[Visit]

if (q    NIL) OR (Visit = 0)

then p    q

endif

endwith

endwhile

end {xWayVisit

9.4 m-Way Search Trees

A very general tree structure, as shown in Figure 9.2, can be built with nodes of the form:

mWayNode = RECORD

Value : ARRAY[1 . . n] OF {value or pointer type

Key : ARRAY[1 . . n] OF {key type

ptr : ARRAY[0 . . n] OF    mWayNode

END {mWayNode

Figure 9.2

A tree with nodes of this kind is an m-way search tree (MWT) if no more than m of the pointers in ptr may be non-NIL, and suitable restrictions are placed on the keys.

The nodes of an MWT may be generalized by using Value[i] as a pointer to a value record, thereby creating an exogenous search tree. The search tree may then serve as a directory to a file too large to be contained in main  (quick-access) memory. Generally, the records themselves reside in auxiliary memory devices, such as disks, as an external file. "Disk" is often used as a synonym for all such devices because it is by far the most common one in use. Any detailed timing analysis of a program that deals with access to disk storage must take into account the difference between fetching and storing on disk and in main memory. Generally, reading from and writing on a file that must be opened and closed by the operating system is many times slower than assignment from or to other variables and needs a separate accounting.

In some implementations, n may be contained as a field of the record, so that the record size may vary. Essentially, the "key" of a node of this type is multivalued; its value is the set of values in the entire array Key, and the search value is found in a node when any of Key[1] . . Key[n] match it. If n is large enough, the search through the n keys of a node may be a binary search, but the dominant factor in the time required for a search often is the number of disk accesses it uses. The number of disk accesses in a search is affected by several factors, such as the shape of the tree and where the key happens to be found in the tree if it is there. One factor that can be controlled by design is whether the key nodes are stored with the records or in a separate directory. An important parameter that can be controlled simply by programming is m, since it determines the minimal depth of the tree.

The formal shape restrictions for the m-way search tree structure follow:

      An empty tree is an m-way search tree.

      No more than m pointers of the array ptr point to internal nodes.

      Key[i] < Key[i + 1] for 1   [image]  i < n.

      All key values in ptr[i] are less than Key[i+1] for 0   [image]  i < n.

      All key values in ptr[i] are greater than Key[i] for 1   [image]  i   [image]  n. (There is no Key[0].)

      All subtrees ptr[i] for 0   [image]  i   [image]  n are m-way search trees.

A 3-way search tree file with integer keys 1-9, 12, 15, 18, 21, and 24 with n = 2, m = 3 has the form shown in Figure 9.3. This file can readily be extended to a 3-way file with keys 1-26. Note that it is not particularly well balanced; with no restrictions on balance the extension can be done in a number of ways.

Figure 9.3

The graph of Faye's Family in Chapter 8 may be taken to indicate family relationships. Additional data, such as the ages of family members would be carried in the value field of records or in records that are the target of a pointer used as a value field. An alternate structure is a 4-way search tree where ages are used as keys (but then descendant information may be carried in some other way), as in Figure 9.4.

Exercise E9.7 is appropriate at this point.

Choices need to be made about including information in the nodes themselves to simplify the algorithms that deal with MWTs. Inclusion of less information in the nodes may complicate the algorithms but save node storage. The choice made here is to include quite a bit of information in the nodes, assuming them to be used as a directory to an external file (external to the MWT-the choice of storage location is an independent decision).

Since m < n is allowed, some keys are inactive and may be so marked. The management of structures involving m-way search trees is eased if the active keys are to the left of the inactive keys. If the nodes are maintained in that form, only m - 1 key slots (m pointers) are needed in the nodes, but one more key slot is usually included because it is very convenient for maintenance of the file. Simplified management routines can be used if three fields are added to the record structure mWayNode:

Figure 9.4

      Active: INTEGER (the number of active keys) If this field is present, there is no need to mark active keys individually.

      Up:    mWayNode (a pointer to the parent of a node, Node) If this field is used, Node itself is sufficient to locate its parent node.

      UpKey: INTEGER (the key in the parent node that points to the node)

As for all tree structures, random insertion and deletion in m-way search trees tends to randomize their shape and increase the search time in the worst case. To prevent this, MWTs can be balanced, like the AVL trees of section 8.7, and the Red-Black Trees of section 9.6. One form of relatively well-balanced MWT is the B-tree, treated in section 9.5. Management of balanced structures requires sophisticated algorithms and an increase in overhead and should be avoided for relatively small files of perhaps 100 nodes or less.

Search, insertion, and deletion in m-way search trees generally can be readily extracted from the procedures developed in section 9.5 tor B-trees.

Problems P9.4 and P9.5 and program PG9.1 are apropriate at this point.

9.5 B-trees

An m-way search tree (MWT) can be balanced by maintaining the number of children of each node to be between limits. Limits of INT(m/2) and m determine a B-tree. It is convenient to think of external nodes of an MWT as failure nodes.

With this terminology, an m-way search tree is a B-tree of order m if it is empty or:

      The root node has at least two children.

      All nodes except the root and failure nodes have at least INT(m/2) and no more than m children.

      All failure nodes are on the same level. (Each path from root to leaf is the same length.)

Both 3-way1 and 4-way2 fail to qualify as B-trees, but a directory to Faye's Family may be stored in a B-tree as shown in Figure 9.5.

Figure 9.5

An MWT grows by adding to a node at the deepest level. A node to which an addition is made may split because the number of keys it is to contain has reached m + 1.

Note: It is convenient to allow a B-tree of order m to have m + 1 children temporarily, so that splits can be delayed. If this is not done, a split can propagate up the tree. Delayed splits can be done on the way down the tree on any subsequent operation. Over a sequence of operations the effect is much the same, although in a strict sense, B-trees of order m can have nodes that are (temporarily) of order m + 1.

A node of order 5 in abbreviated form is shown in Figure 9.6. Here a-e may be external, but in general they are subtrees. With the addition of one more key, this node becomes unstable (ripe for splitting) if it is in a B-tree of order 5, as shown in Figure 9.7.

Figure 9.6

Figure 9.7

(Here f may be a subtree other than an external node if G is lifted from below by a splitting operation of the kind about to be described.)

When this node is encountered on the way down in a subsequent operation, it will be split as shown in Figure 9.8. Details of the management procedures are to be found in the sections that follow.

Figure 9.8

Problem P9.6 is appropriate at this point.

9.5.1 Creation and Splitting of B-tree Nodes

9.5.2 Insertion into a B-tree

9.5.3 Deletion from a B-tree

9.5.1 Creation and Splitting of B-tree Nodes

When a node splits, a middle key is moved up to the parent node. The growth and splitting can reach the root level of the tree, in which case a new root node may be created. The left child of the new root is the left half of the old root, and its right child the newly split off right half of the old root, as illustrated in Figure 9.9.

Figure 9.9

Because of this process, it is convenient to have a Create operation that attaches a given left child LefTree to an (empty) new root node mwt:

procedure Create(mwt,LefTree)                   {O(m)

NEW(mwt)

with mwt do

ptr[[0]    LefTree

Active    0

Up    NIL

UpKey    0

for i = 1 to m do

ptr[i]    NIL

next i

endwith

end {Create

The right child may then be included by an operation NewKeyIn, which inserts a subtree, along with the key that locates it, into a specified position in a given node:

procedure NewKeyIn(Node,Ndex,NewKey,NewValue,SubTree)  {O(m)

with Node    do

k    Ndex

for i = Active downto k do

Value[i + 1]    Value[i]

Key[i+1]    Key[i]

ptr[i + 1]    ptr[i]

if ptr [i]    NIL

then ptr[i]    .UpKey    ptr[i]    .UpKey + 1

endif

next i

Value[k]    NewValue

Key[k]    NewKey

Active    Active + 1

endwith

end {NewKeyIn

With these operations, an MWT may be started with a single record by:

Create(mwt,NIL)

NewKeyIn(mwt,1,NewKey,NewValue,NIL)

A new root node may be constructed with the old tree as its left child, and SubTree as its right child by:

old    mwt

Create(mwt, old)

NewKeyIn(mwt,1,NewKey,NewValue,SubTree)

Addition and deletion of keys in an MWT depend upon a search for a key or for its appropriate position if it is absent. The core of the tree search is the scan across the keys of a single node to locate either the search key itself or the subtree in which further searching should take place:

procedure mWayScan(Node,ScanKey,Ndex,Found)     {O(m)

Found    FALSE                   {ScanKey may be larger

Ndex    Node   .Active            {than the keys in Node.

for i = 1 to Node    .Active do

if ScanKey = Node    .Key[i]   {The search is complete.

then Ndex    i

Found    TRUE

return

endif

if ScanKey < Node   .Key[i]    {The subtree to be searched

                              {next has been located.

then Ndex    i - 1

exit

endif

next i

end {mWayScan

The search through the MWT as a whole accepts the location of the search key and follows the lead provided by mWayScan. Nodes with m active keys are split as encountered with procedure mWaySplit, which provides a suitable node, index, key, and subtree for insertion of one key from the split node into the next level up. The search then proceeds from the (altered) parent node:

procedure mWaySearch(mwt,SearchKey,Node,Ndex,Found)

Node    mwt

Parent    mwt                    {O(m    (mwt depth))

while Node    NIL do

if Node   .Active = m

then mWaySplit(Node,Ndex,NewKey,NewValue,SubTree)

NewKeyIn(Node,Ndex,NewKey,NewValue,SubTree)

endif

mWayScan(Node,SearchKey,Ndex,Found)

if Found then return endif    {SearchKey is found in Node.

Parent    Node

Node    Node   .ptr[Ndex]

endwhile

Node    Parent                   {SearchKey is not in mwt.

end {mWaySearch

The procedure that remains to be specified is mWaySplit. An example will illustrate how a node q is split. Suppose that m is 5, the parent of q is p, and x is the ancestor key of q: x = p  [image].Key[q  [image].UpKey], as shown in Figure 9.10.

Figure 9.10

Since q has six children, it must be split. The "middle" key, k3, is moved up to p, inserted into p between keys x and y, and then keys k4 and k5 moved to a new subtree, depicted in Figure 9.11.

Figure 9.11

The special case that must be considered is when q is the root of the MWT, and so p must be created empty with q as its left child and become the root of the tree. The mechanics are:

procedure mWaySplit(Node,Ndex,NewKey,NewValue,SubTree)           {O(m)

k    INT(Node    .Active/2 + 1)       {The index of the key to

NEW(r)                               {be moved up one level.

j    1

for i = k + 1 to Node   .Active do    {Shift the right

r    .Value[j]    Node    .Value[i]  {half of Node to r

r    .Key[j]    Node  .Kev[i]

r    .ptr[j]    Node  .ptr[i]

j    j + 1

next i

r    .ptr[0]    Node    .ptr[k]

j    0

for i = k to Node    .Active do

if Node   . ptr[i]    NIL

then Node    .ptr[i]    .Up    r

Node    .ptr[i]    .UpKey    j

endif

j    j + 1

next i

NewValue    Node    .Value[k]

NewKey    Node   . Key[k]

SubTree    r

Node    .Active    k - 1

r    .Active    m - k

if Node    .Up    NIL                  {Insert the middle key at the next

then Ndex    Node    .UpKey + 1     {higher level, at the right of

Node    Node    .Up            {UpKey, locating the new node.

else Create(mwt,mwt)               {A new node is needed. The old

Node    .Up    mwt             {mwt is the left child of the new.

Node    .UpKey    0

Node    mwt

Ndex    1

endif

r    .Up    Node

r    .UpKey    Ndex

end {mWaySplit

Problem P9.7 is appropriate at this point.

9.5.2 Insertion into a B-tree

Insertions of (key) values into an MWT include two special cases: the MWT may be empty, or the value may be already present. In the absence of either special case, a search provides the location for the insertion. After the initial insertion, it may be necessary to repeatedly split a node into a pair of nodes, which calls for the insertion of a key one level higher to distinguish between them. The process stops when an expanded node has no more than m - 1 keys. (In particular, a newly created root node has only one.) With the assumption that the splitting process mWaySplit provides a suitable node, index, key, and subtree for insertion at the next level up by NewKeyIn, the insertion procedure is:

procedure mWayIn(mwt,SearchKey,NewValue)     {O(m    depth)

if mwt = NIL

then Create(mwt,NIL)

NewKeyIn(mwt,1,SearchKey,NewValue,NIL)

return

endif

mWaySearch(mwt,SearchKey,Node,Ndex,Found )

if Found                                  {SearchKey is not really new.

then return

else NewKeyIn(Node,Ndex +1,SearchKey,NewValue,NIL)

endif

end {mWayIn

The depth of an MWT containing n keys is restricted by the need for each node to have at least b = INT(m/2) children and the restriction of failure nodes to the deepest level. If the effect of deletions upon the root node is ignored, there will be at least (b + b2) keys in an MWT of depth 2, (b + b2 + b3) for one of depth 3, and so on. The depth is no more than INT(logb n) + 1, where logb is the logarithm of base b.

Exercises E9.8 and E9.9 and problem P9.8 are appropriate at this point.

9.5.3 Deletion from a B-tree

Deletion from an interior node of a B-tree can be transformed to deletion that begins at a leaf node by simply exchanging the key to be deleted with the leftmost key in the subtree it locates. Location of the starting point is done by:

function LeftMost(p)                        {O(depth)

while p    .ptr[0]    NIL do

p    p    .ptr[0]

endwhile

return p

end {LeftMost

Reduction of a general deletion to a process CutLeaf that begins at a leaf node then becomes:

procedure mWayOut(mwt, OutKey)

if mwt = NIL then return endif

mWaySearch(mwt,OutKey,Node,Ndex,Found)

if NOT Found then return endif           {since OutKey is already gone

SubTree    Node    .ptr[Ndex]

if SubTree    NIL

then SmallChild    LeftMost(Subtree)

Node    .Value[Ndex]    SmallChild    .Value[1]

Node    .Key[Ndex]    SmallChild    .Key[1]

Node    SmallChild

Ndex    1

endif

CutLeaf(Node,Ndex)            {Now the key to be deleted is in a leaf node.

end {mWayOut

The order of mWayOut is at least O(depth) because of mWaySearch, but then depends on CutLeaf. Hence it is O(Max {depth,O(CutLeaf)}).

Deletion of a key from a node can reduce the number of keys in the node below the allowed limit of INT((m + 1)/2) - 1. The "hole" that is created may be repairable by shifting a key from the nearest right sibling or, if that is not possible, from the nearest left sibling. If neither of these shifts is possible, a key is moved down from the parent level, and the shifting process may need to be repeated at that level. The shifting of keys from node to node may propagate to the root and possibly reduce the depth of the tree.

The actual compaction required to remove a key no longer needed in a specific node is:

procedure Compact(Node,Ndex)                          {O(m)

with Node    do

for i    Ndex to Active - 1 do

Value[i]    Valuel[i + 1]

Key[i]    Key[i + 1]

if prt[i + 1]    NIL

then prt[i + 1]    .UpKey    1

endif

ptr[Active]    NIL

Active    Active - 1

if Active = 0

then if ptr[0]    NIL

then Up    .ptr[UpKey]    ptr[0]

endif

if Up    NIL

then Up    .ptr[UpKey]    ptr[0]

else mwt    ptr[0]

endif

DISPOSE(Node)

endif

endwith

end {Compact

The shift of a key to the left (or right) actually is a rotation involving the key of the parent node of the pair of nodes involved. The depleted example shown in Figure 9.12 is repaired with the rotation shown in Figure 9.13, followed by compaction.

Figure 9.12

Figure 9.13

The details of the process are:

procedure ShiftLeft

p    parent

NU    Node    .UpKey + 1

NA    Node    .Active + 1

RS    RightSib

NewKeyIn(Node,NA,p    .Key[NU],p    .Value[NU],RS    .ptr[0])

p    .Value[NU]    RS    .Value[1]

p    .Key[NU]    RS    .Key[1]

RS    .ptr[0]    RS    .ptr[1]

Compact(RS,1)

end {ShiftLeft

The shift of a key to the right is very similar, as diagrammed in Figure 9.14 and Figure 9.15, and is left to the problems for detail. It is followed by the removal of the (redundant) rightmost key in LeftSib by compaction.

Figure 9.14

Figure 9.15

The shift of a key down from the parent level only takes place if both the right sibling and the left sibling of a node fail to have enough keys to shift. In that case, the node involved can be combined with one of them, since they cannot both be NIL. (If both were NIL, the parent node would contain no key, which is never allowed to happen.) The keys of the right node of a pair to be combined (Figure 9.16) can be incorporated into the left node of the pair, as shown in Figure 9.17. And this incorporation is followed by the deletion of the (redundant) rightmost key from Parent.

Figure 9.16

Figure 9.17

The procedure is:

procedure ShiftDown(Node,RightSib)              {O(m)

p    Parent

NU    Node    .UpKey + 1

NA    Node    .Active + 1

RS    RightSib

NewKeyIn(Node,p    .Key[NU],NA,p    .Value[NU],RS    .ptr[0])

for i = 1 to RS    .Active do

NewKeyIn(Node,NA,RS    .Key[i],RS    .Value[i],RS    .ptr[i])

next i

Ndex    NU

Node    Parent

DISPOSE(RS)

Compact(p,NU)

end {ShiftDown

Finally, with these tools, we can fonn CutLeaf:

procedure CutLeaf (Node,Ndex)

Limit    INT((m + 1)/2 - 1)                {O(m    depth)

while Node    mwt do

Compact(Node,Ndex)                       {Remove key from Node

Size    Node    .Active

if Size    Limit then return endif        {since no reshaping is required.

Parent    Node    .Up

NU    Node    .UpKey + 1

RightSib    Parent    .ptr[NU]

LeftSib    NIL

if NU > 1 then LeftSib    Parent    .ptr[NU - 2] endif

if RightSib    NIL                        {Try to fill the hole at

                                          this level.

then if RightSib    .Active > Limit

then ShiftLeft

return

endif

else if LeftSib    .Active > Limit   {LeftSib cannot be NIL.

then ShiftRight

return

endif

endif

if RightSib    NIL                       {A key is moved down.

then ShiftDown(Node,RightSib)

else ShiftDown(LeftSib,Node)

endif

endwhile                                {The root node has been reached,

Compact(Node,Ndex)                        {and a key shifted down from it.

end {CutLeaf

Problem P9.9 and program PG9.2 are appropriate at this point, and problems P9.4 and P9.5 should be reconsidered.

9.6 2-3-4 Trees          (optional)

The depth of binary search trees can be reduced by using nodes with more than two children and by maintaining balance defined by criteria that tend to keep all paths from root to leaf close to the same length, hence proportional to the logarithm of the number of nodes in the tree. The AVL trees of section 8.7 use balancing to reduce the tree depth, and the m-way search trees of section 9.3 use multichild nodes as a reduction technique. A third approach is to use nodes with 2,3, or 4 children only and keep all but the deepest level full. Such trees are sometimes called 2-3 trees since the 4-child nodes are considered to be temporary, like the (m + 1)-child nodes of the m-way search trees in section 9.4. Like other search trees, 2-3 trees are often exogenous but are illustrated with endogenous examples for the sake of clarity.

Consider a search tree of 2-, 3-, and 4-child nodes formed from the sequence of keys L-O-O-K-I-N-G-W-I-T-H-O-U-T. Figure 9.18 shows what the first three stages might be.

Figure 9.18

To add K it is necessary to split the 4-node before K is added, and it may be done as in Figure 9.19.

Figure 9.19

Then, in sequence, I and N are added in Figure 9.20. Then, G-W-I-T-H are added in Figure 9.21.

Figure 9.20

Figure 9.21

The insertion of the O of O-U-T would cause the 4-node parent of a 4-node to be split. This entails traversing back up the tree, but it can be avoided if 4-nodes are split on the way down the tree, during insertion operations. This modification is shown in Figure 9.22.

Figure 9.22

If the parent of a 4-node is split before the node itself, there are only two cases to consider, 2-node and 3-node parents, shown in Figures 9.23 and 9.24.

Figure 9.23

Figure 9.24

(The variations of each case provide no surprises.) Note that the depth of the tree is unaltered by these operations. A tree managed in this way begins with a single node; the tree depth is increased only when the root is a 4-node and is then split by an insertion operation, and the terminal nodes all occur at the same level, the deepest one. Such trees are nicely balanced.

The examples given here do not address deletion and they assume that keys may be duplicated. A complete management system needs to allow deletion and deal with duplicate keys, perhaps by representing them as lists or counts, or extending their keys to distinguish them, or simply disallowing them.

It is possible to manage 2-3-4-trees very much like m-way search trees, but there is an alternate way: they may be restructured as binary trees, called Red-Black Trees. This is the approach explored in Part II, section  U.

Exercise E9.10 is appropriate at this point.

9.7 Sets and Their Trees     (optional)

A set is a collection of items without order or duplication or internal structure. Sets are useful because they are simple in principle--they represent categories to which items belong or not. Direct management of sets is available in some languages, but not all. For example, in Pascal

IF Token IN CodeSymbols THEN Accept (Token)

tests to see if the value of variable Token is an element of the set CodeSymbols.

As a structure, sets must be implemented with other structures at some level. One common support for sets is a tree-like structure that links the elements of a set to the set identifier. For example, if the set Even is {16,2,4} and the set Odd is {3,1} then they may be indicated as shown in Figure 9.25.

Figure 9.25

Because of the directions of the links, an item needs to be located before its relationship to a set can be determined. In practice the "items" are often indices to an array of pointers that determine a "set" (or several sets) by the relationships between them. The structure of the indices is commonly exogenous and associated with values that may be pointers to records. For convenience, we assume that the locating structure is an array A of records, each of which contains a pointer field p used to link the records together into a tree-like structure. The abbreviation x   [image]  .p is used for A[x]   [image].p.

The links of a set tree point from child to parent, and one element of a set, sometimes called the canonical element, is used as the root of the tree representing the set. The canonical element is simply called the root (of the set) in what follows. Figure 9.26 shows the even-odd example of Figure 9.25 when these design decisions are made.

Figure 9.26

Finally, a collection of sets is often mananged so that the sets remain disjoint: they have no element in common. The entire collection of sets is then implemented with a forest--a collection of trees.

A minimal set of management operations is CREATE, UNION, and FIND. The operation UNION(x,y) forms and returns one set, called the union of the sets x and y, containing all of the elements of both. The operation FIND(x) returns the root and hence the set membership of element x. It is convenient to write both of these as functions with side effects. In the most straightforward form they are:

procedure NewSet(x)           function QuickUnion(x,y)  {0(1)

x    .p    x                   x    .p    y

end {NewSet                   return y

end {QuickUnion

function QuickFind(x)         {O(depth)

while x    .p    x do

x    x    .p

endwhile

return x

end {QuickFind

Figure 9.27 shows the value of QuickUnion (2,1) applied to the even-odd example.

Figure 9.27

A set formed by an arbitrary sequence of UNIONS can be a tree of depth up to d = n, where n is the number of nodes in the set. QuickFind is O(d), which is O(n). Several ways that are used to reduce the average depth of set trees by modifying both the UNION and FIND operations will be treated here.

One way to reduce the depth of the tree returned by the UNION operation involves storing the count of the number of elements of a set in its root node. The UNION operation then chooses a root with maximal count as the root of the new tree. If the entire forest is stored in arrays, so that pointers are integer indices, the parent field and count field can be combined: the "parent" pointer is set to the negative value of the count for a root node. In a more general setting, separate fields are needed, and the UNION operation becomes:

procedure CountUnion(x,y)            {0(1)

Total    x    .c + y    .c

if x    .c > y    .c

then y    .p    x

x    .c    Total

else x    .p    y

y    .c    Total

endif

end {CountUnion

CountUnion reduces set tree depths to some extent in a statistical sense but still allows a depth to reach n in the worst case. The cost of using UNION-FIND operations can be averaged over a sequence of operations, rather than simply noting the worst-case or average-case cost of individual operations. Such a cost is said to be amortized over the sequence, in either  a worst-case or an average-case way. Accounting by amortization encourages path compression. Path compression uses information gathered during FIND operations to shorten the paths from leaf to root so that subsequent FIND operations will be faster. The price paid tor this extended operation is regained later and lowers the amortized average for a sequence of operations.

The most drastic form of path compression is:

      The Collapsing Rule. If y is a node on the path from x to its root node Root, then set y   [image]  .p to Root.

With this modification:

function Find(x)       {O(depth)

Root    x

while Root    .p    Root do

Root    Root    .p

endwhile

while x    Root do       {collapsing

y    x    .p

x    .p    Root

x    y

endwhile

return Root

end {Find

When applied to the even-odd example union, Find(16) returns 1 and has the side effect shown in Figure 9.28.

Figure 9.28

Various ways to apply path compression without making two passes over the path have been devised. One of them is path halving--a node on the path from leaf to root is repointed to its grandparent:

function HalFind(x)                       {O(depth)

while x    x    .p do

y    x    .p

x    .p    y    .p

x    y

endwhile

return x

end {HalFind

Finally, we may combine path compression with a more sophisticated UNION operation that uses a rank to decide which root shall be chosen as the root of the union. Ranks of singleton sets are initialized to zero and changed only during UNION operations. If x   [image]  .Rank > y   [image]  .Rank, then x becomes the new root. If x   [image]  .Rank = y   [image]  .Rank, then y is the new root, and its rank is incremented (by one). The operation is:

function Union(x,y)       {O(1)

if x    .Rank > y    .Rank

then y    .p    x

return x

else if x    .Rank = y    .Rank

then y    .Rank    y    .Rank + 1

endif

endif

x    .p    y

return y

end {Union

The amortized behavior of sequences of path-compressed FIND and ranked UNION operations have been studied by Tarjan, and found to be very effective. (See [TARJAN, 1983].)

Problem P9.10 and program PG9.3 are appropriate at this point.

9.8 Trees as Decision Tools     (optional)

A node of a tree can contain (or point to) information to be used to make a decision: a decision to carry out a process, to discontinue a traverse of the tree, or to describe the result of processing a subtree rooted at the node. Examination of the node value by a decision-making process results in an action. A tree built and used to aid in the making of decisions is a decision tree. A binary search tree is a simple example of a decision tree.

A decision tree can be specifically constructed to develop the structure of a program that solves a problem, as illustrated with the Eight Coins problem in Part II, section  V.

One common application of trees as decision tools is their use in the calculation of probabilities: value fields at a node contain a set of probabilities that events will occur, and each event is associated with a child of the node. A node roots a (sub)tree if the event it represents is itself divided into events, in which case those events are associated with probabilities that sum to the node probability.

To illustrate, Irish folklore includes an interesting probability problem with possible historical significance. The story is that in olden times, on the twelfth birthday of an Irish maiden she was to choose six long grass stems and hold them in one hand. A friend tied the tops in three pairs, and the bottoms in three pairs. If, when the maiden opened her hand the stems formed a single loop, she would be married within the year. The probability question is: what is the chance of a single loop resulting from random pair selection? A probability tree can help to answer that question.

The three initial top ties form three linear pairs of stems (2-strands). Each bottom tie after that selects an event that either allows the possibility of a single ring or does not. The (probability) decision tree that supplies the answer is shown in Figure 9.29. This tree has three levels below the root, one for the result of each tie, and it has been pruned. The nodes 2, 3, and 5 are associated with short loops, and so the subtrees they head are of no interest in the search for the single loop.

Probabilities for the nodes are determined as follows:

      The first (bottom) tie chooses two stems from six, and there are 15 equally likely ways to do that. Of these 15 ways, 3 will form a 2-loop, and 12 a 4-strand. Hence the probability of node 2 is 1/5 and that of node 3 is 4/5 = p1.

      For tie 2, there are six equally likely pairs to be chosen from the four loose ends represented by node 3. One of these pairs forms a 2-loop, another forms a 4-loop, and the other four form 4-strands. Hence the probability of node 6 is 4/6 = p2.

      Tie 3 will form a 6-loop from node 6 with probability 1 = p3.

Figure 9.29

The probability that a single loop will result is p1 x p2 x p3 = 8/15, intuitively quite high, and possibly a bit of a culture shock.

Clearly, it is better to approach such a problem with pencil and paper than to design a program to build and traverse the tree. With more complex problems, however, the tree becomes too large to deal with except by programming. An intermediate situation arises when a tree is used as a design guide for a program that does not actually form the tree involved in its conception. The Eight-Coins problem of Part II, section  V, is such a problem.

9.9 Game Trees     (optional)

Many games are played by making decisions based on the current state of the game, described perhaps by a board configuration or by several sets of cards. A game often has the Markov property--the effect of moves in the game depends only on its current state and not in any way on how that state was reached. A data structure can be formed with nodes that represent game positions (states), linked to nodes reachable in one move by a player. Such a structure is a tree, specifically a game tree.

The term game tree is most often applied to trees that represent two-person games where the players alternate. One player, player Odd, may be associated with levels 1,3,5, . . ., and the other, player Even, with levels 2,4,6, . . . of the game tree.

If moves are not made at random (as they are in the probability decision tree of section 9.8), a player needs to assign values to subtrees rooted at the current node in order to evaluate the choices of moves.

There is an astronomical number of nodes in the tree of most games complex enough to be interesting. The restriction to games with finite trees does not resolve this problem (there are believed to be about 10100 nodes in a game tree for chess), and so a variety of ways to limit a search for the "best next move" are needed. Limiting is a matter of pruning the tree--removing subtrees or at least removing them from consideration. Some pruning is based on hard (explicit) information and some is heuristic--based on the application of a "rule of thumb that usually works."

Discussions of node evaluation and pruning are to be found in Part II, section  W.

Summary

Access paths to data are restricted by most of the structures studied in previous chapters. Such paths can be enhanced by general techniques of exogeny and indirection. Indirection is applied to ShellSort and to heaps in section 9.1, but it can be applied to all data structures. The major benefit of indirection is that pointers (or indices) are moved, rather than bulky records. Exogeny gains its greatest power from application to external files because of the difference in time scale between access to main memory and to secondary memory.

Even though any tree can be modeled with a binary tree, in some applications a natural organization uses nodes with more than two children. Given the same number of nodes, the minimal depth of a tree decreases with the number of allowed children, an advantage for directories of external files.

To take advantage of the minimal possible depth requires approaching it, which implies that balance will be maintained over a sequence of deletion and insertion operations. The cost of maintenance for a single operation is increased in order to decrease the cost amortized over the long term. One common technique applied to multiway search trees is to allow nodes to grow a bit larger than desired and split them on the way down in a later operation. Another is to rotate nodes around a pivot to move a structure in the direction of balance. Rotations propagate through the tree during deletions to restore balance.

Two tree structures in common use for (endogenous or exogenous) directories for large files are introduced here, (joining the AVL trees of section 8.7 in that role). One is m-way search trees (section 9.4) and their balanced form, B-trees (section 9.5). The other is the binary tree emulation of 2-3-4 trees (section 9.6), Red-Black trees (treated in Part II, section  U). Maintenance of B-trees is probably easier than for any other form, and they are popular in practice.

Trees are used as a crutch to the intelligent sifting of information in a number of ways, as indicated in section 9.7. One use of trees is simply as an aid in the design of a program, and the Eight-Coins problem of Part II, Section  V, serves as an example. Another is the use of trees within the large, complex, and increasingly important area of artificial intelligence. A brief touch of that role is exhibited in section 9.9 and Part II, section  W, in the form of game trees and their pruning.

Exercises

Exercises immediate applications of the text material

E9.1     Trace the action of ShellSort on the letter sequence: S-H-E-L-L-E-Y-B-E-A-N-S .

E9.2     Trace the action of MakeHeap as an inverse-enhanced indirect heap on the letter sequence: T-R-A-S-H-B-I-N-J-O-E.

E9.3     Separately sort the sequences S-H-U-F-F-L-E and B-U-C-K-A-N-D-W-I-N-G and merge them by tracing the action of StraightMerge on them.

E9.4     What is the maximum number of nodes in a tree with six levels if the interior nodes have (a) 2 children, (b) 5 children, or (c) 10 children?

E9.5     Rewrite TwoWayVisit assuming that the leaf nodes are their own children.

E9.6     Rewrite TwoWayVisit assuming that the parent of the root node and the children of leaf nodes are the universal sentinel node u.

E9.7     Redraw Faye's Chronograph of section 9.4 as a 3-way search tree.

E9.8     Draw the sequence of 4-way B-trees created from keys that are names of months, entered in calendar order with mWayln and kept in alphabetical order.

E9.9     Draw the sequence of 4-way B-trees created from keys that are the names of months, entered in alphabetical order with mWayIn and kept in calendar order.

E9.10     Trace the construction of a 2-3-4 tree from the letter sequence: S-T-R-A-N-G-E-D-U-M-B-O-W-L .

Problems

Problems not immediate, but requiring no major extensions of the text material

P9.1     Rewrite BubbleSort (Chapter 1 ) as an indirect procedure.

P9.2     What is the maximum number of nodes in a tree with L levels if the interior nodes have m children?

P9.3     Rewrite TwoWayVisit of section 9.3 specifically to be a traverse of each of the types inorder, preorder, postorder, displaying the node values in the appropriate order.

P9.4     (Difficult.) Write a procedure that inserts nodes into m-way search trees, without regard for balance.

P9.5     (Difficult.) Write a procedure that deletes nodes from m-way search trees, without regard for balance.

P9.6     Trace the creation of a 3-way B-tree by adding the letters: O-P-T-I-M-A-L-S-C-U-D-E-R-Y one at a time.

P9.7     Rewrite procedure mWayScan of section 9.5 so that it uses a binary search.

P9.8     Trace the behavior of mWaySearch, looking for your birthday month in the final structures created in E9.8 and E9.9.

P9.9     Show the result of applying mWayOut to the removal of the summer months June, July, and August from the structures created in E9.8 and E9.9.

P9.10     Calendar months are to be placed in the same set if they are in the same pair of seasons: Spring-Summer and Fall-Winter. For each of the set-management techniques from section 9.7, what is the final configuration (set-trees) of the months entered in alphabetical order? Assume that a month record has a tag field that contains one of the values SS or FW. A month is attached to the most recently entered node if it is in the same season pair; otherwise the new month starts a new tree. The months are to be entered in the order: April, Dec, Aug, July, June, Feb, May, Sept, Jan, March, Nov, Oct. After entry of Oct, union each tree through the nearest node of the same season pair on its left.

(a)     No compaction (QuickUnion, QuickFind)

(b)     Root-count (CountUnion)

(c)     Collapsing rule (Find)

(d)     Path halving (HalFind)

(e)     Ranking (Union)

Programs

Programs for trying it yourself

PG9.1     Write a program to manage a 3-way search-tree for alphabetic keys with numerical values. Use it to build a search tree of the calendar months 1-12 with keys January-December, inserted in that order. Display the resulting tree, delete the summer months June-August, and display again. (Balance is not required.)

PG9.2     Write a program that manages a B-tree as described for the m-way search tree in PG9. 1.

PG9.3     Write a program that manages ranked sets of calendar months as described in P9.10, option (e). The forest created by each addition of a month should be displayed. If the program is written in a language with built-in set manipulation, it is not to be used.

Go to Chapter 10            Return to Table of Contents


Data Structures: PREFACE

 Return to Table of Contents

PREFACE

Many application programs and discussions of topics in computer science are awkward without a relatively extensive acquaintance with data structures. Hence, the data structures course is moving rapidly downward in the curriculum. In the last few years it has changed from being a graduate-level course in analytical techniques to a foundation course for undergraduates and nonmajors. If it isn't already, data structures soon will be taught at four-year colleges as a prerequisite for most upper-level computer science courses and as a terminal computer science course in two-year programs and for many nonmajors. Consequently, the course is taught on a wide range of levels. The broadening of the audience requires some adjustment in the way a data structures course is taught.

Although its wide range of content and academic rigor make it suitable for use in higher-level courses, Data Structures: From Arrays to Priority Queues is intended to serve a broader audience. The book can function as an introduction to data structures quite early in the development of programming mastery. It is aimed at the student who has had one or more semesters of programming in a structured language and no specific college mathematics. At lower levels, a data structures course needs to develop algorithmic sophistication rather than exercise mathematical skills. Thus, this book keeps the comparison of data structures and algorithms in constant view, but it does not require a great deal of mathematical sophistication. Data Structures brings the reader to an awareness of the timing of algorithms, particularly at the level of the order of growth, but extensive analysis of algorithms applied to data structures is left to later courses. The intent is to foster an appreciation of the effectiveness of data structures and their associated algorithms, but in an introduction it is more important to construct a workable set of tools than to hone a smaller set. Data structures themselves are the primary focus, not their detailed analysis.

The topic of data structures plays much the same essential role in computer science that calculus plays in mathematics. The apparent agenda in a data structures course is the comparative study of alternate ways to solve and implement program solutions to problems. The hidden agenda is the development of an understanding of the use of abstraction in problem solving. This book focuses on the hidden agenda as a foundation for study of the apparent agenda. Skill in abstraction is developed gradually, as a natural part of problem solving with programs. Three features that explicitly address the hidden agenda are:

1. Use of Pascal-like pseudocode

2. Comparison of algorithmic and implementation variations

3. Generalized algorithms for data structure traverses

A form of pseudocode is used to present algorithms for several reasons. One reason is simply that no one language is ideal for all purposes, is used universally, or is dominant forever. A number of languages, including Algol, Pascal, ADA, and Modula-2, have common features extracted here. Another reason for presenting algorithms in pseudocode for the reader to translate into a working language is that the task requires some effort, but not too much. Translation requires a choice of parameter-passing classes for procedure variables and attention to some other details. Translating pseudocode does not, however, require the time and effort needed to derive an algorithm. An extensive amount of conceptual material can be presented succinctly in pseudocode. Use of pseudocode thus provides an appropriate balance between covering the maximum amount of ground and providing in-depth understanding of the details of the geography that is traversed. Use of pseudocode, plus examples in the text, also helps readers to develop skill in abstraction and understanding of shell-structuring and modularity.

The management of data structures involves pitfalls that lead to subtle bugs in programs, and such potential problems are best exposed by being viewed from several perspectives. Therefore, Data Structures emphasizes the variety of solutions to problems. For example, general traverse algorithms are developed for lists, binary trees, and priority queues. Such general algorithms provide a framework within which variations can be created and problems can be solved. The algorithms can be tailored to specific applications, and the exercise of doing so has merit in itself. The general algorithms are better guides to the variations needed for applications than is one specific variation that often fails to fit. General algorithms also promote the development of skill in abstraction in a relatively gentle way.

A separate collection of expansion and enrichment sections is included in Part II. The sections introduce or explore in depth topics that are related to the mainstream of the text. Some of these sections contain substantial working Pascal programs, initially developed in pseudocode, which provide examples of program development. The instructor thus has options for major examples, enrichment, and independent reading assignments, yet because the additional sections are separate, they do not distract from the main content.

To help the instructor tailor the course to students with varied backgrounds, most topics in the book can be covered at a variety of levels:

1. Pictorial discussion and general comprehension

2. Tracing of the crucial process(es) acting on a specific example

3. Pseudocode detailing of an algorithm with discussion of problems and variations

4. Program assignments based on class discussion

5. Program assignments based on reading

6. Experimentation with programs and analysis of the results

Potential assignments are separated into exercises, problems, programs, and projects, and the amount of effort required increases in that order. Possible assignments are mentioned in the text where they are appropriate, but they are grouped together at the end of the chapter so they can be found easily at any time. Summaries offer the reader an overview of material in each chapter.

The instructor's manual includes answers to the exercises and problems and a Pascal solution to a program assignment from each chapter.

The flexibility designed into this text has allowed it to be used at NWMSU for classes with students of mixed backgrounds, many of whom are second-semester freshmen. The central thrust of the course is coverage of the sections in Part One of this text that are not marked optional. In that course, Chapter 7 is considered remedial but is available for self-study, the timing-function portion of Chapter 1 is the only part of that chapter requiring class time, and Chapter 10 is seldom reached. Some optional sections in Part I and various sections of Part II are selected by instructors for supplementary readings, assignments, and sometimes ciass discussion.

Thorough coverage of the entire text requires a second semester or a very fast pace for even well-prepared students. The intent of the rich offering of Part II and the optional sections in Part I is to support flexibility through the selection of sections and levels of coverage. (Besides, students will explore accessible material on their own.)

The reviewers listed below made many suggestions that were incorporated into this book and improved it immeasurably. They are not, however, responsible for the remaining errors, whether inadvertant or stubborn: Cathy Dickerson, Black Hawk College; William H. Ford, University of the Pacific; Ken Friedenbach, University of Santa Clara; Judith L. Gersting, Indiana University-Purdue University; Henry Gordon, Kutztown State College; Nancy Griffeth, Georgia Institute of Technology; Greg Jones, Providence, Utah; Leonard Larsen, University of Wisconsin, Eau Claire; Louise Moyer, California State University, Hayward; Ron Peterson, Weber State College; Douglas Re, San Francisco Community College; Dean Sanders, Illinois State University.

Two groups of people aided and abetted this work in ways that must remain untold in detail but are appreciated in depth. One group is a team assembled by Wadsworth that includes: Myron Flemming, Frank Ruggirello, Serina Beauparlant, Mary Forkner, and Joan Pendleton. The other group consists of colleagues at NWMSU: Merry McDonald, Gary McDonald, Robert Franks, Hong-Shi Yuan, Phil Heeler, and Margaret Adams.

My brother, David, put considerable effort into improving my technical writing in the early stages of this book, and I can only hope that the effect carried through the project.

Finally, this book could not have been produced if my wife, Carlene, had not taken over virtually all of the management of a busy household while working full time.

Wayne Amsbury

Go to Chapter 1 Return to Table of Contents


Data Structures: REFERENCES

 Return to Table of Contents

REFERENCES

Adelson-Velskii, G. M. and E. M. Landis. Soviet Math (Dokl.) 3 (1962), 1259-1263.

Aho, Alfred V. and Jeffrey D. Ullman, Principles of Compiler Design, Addison-Wesley, 1977.

Bentley, Jon L., "Programming Pearls," ACM Communications, August, 1983 and thereafter.

Brodie, Leo, Starting FORTH, Prentice-Hall, 1981.

BYTE Magazine, August, 1980 (Issue on the FORTH language).

Even, Shimon, Graph Algorithms, Computer Science Press, 1979.

Intel, The 8086 Family User's Manual, Intel Corp., 1979.

Kernighan, B. W. and P. J. Plauger, The Elements of Programming Style, McGraw-Hill, 1974.

Knuth, Donald E., The Art of Programming, Vol. 1, Addison-Wesley, 1973.

Ledgard, Henry and Michael Marcotty, The Programming Language Landscape, SRA, 1981.

Roberts, Fred S., Discrete Mathematical Models, Prentice-Hall, 1976.

Roberts, Fred S., Applied Combinatorics, Prentice-Hall, 1984.

Sedgewick, Robert, Algorithms, Addison-Wesley, 1983.

STACK, IEEE Computer Society Journal, Computer, May 1977 (devoted to stack machines).

Tarjan, Robert E., Data Structures and Network Algorithms, SIAM, 1983.

Touretsky, David S., LISP, A Gentle Introduction to Symbolic Computation, Harper & Row, 1984.

Wirth, Nicklaus, Algorithms + Data Structures = Programs, Prentice-Hall, 1976.


Data Structures: SECTION A: RANDOM PROCESSES

 Return to Table of Contents

SECTION A: RANDOM PROCESSES

It is not possible to know the future -- of a process, of a program, of the behavior of nearly any interesting system. The universe itself, in fact, seems to be indeterminate. To demonstrate the failure of prophecy, give a carefully designed program to ten people to use. They will find errors and want changes--that much is predictable. They will enter, or wish to enter, data sets that the programmer would consider extremely unlikely. One way to prepare for this kind of variation is to generate data sets randomly. The ability to do so is also useful for simulating a large variety of other natural processes.

The simplest way to imitate unpredictable variation is the generation of pseudorandom numbers. A random variable x, 0 < x < 1, has the property that any real number in this range is equally likely. A pseudorandom variable r, 0 < r < 1, has values that approximate true randomness in two ways:

The set of values of r is not complete, but there are many of them and no obvious gaps.

The values in the range of r are of approximately equal likelihood.

A great deal of work has been done on developing programs to produce pseudorandom numbers [KNUTH, 1973], and no programmer should be without access to such programs. In some of the exercises a function Rand, available to provide a pseudorandom number on demand, is assumed. Also assumed is a mechanism for reusing the same value or sequence of values produced by it. This assumption is convenient for debugging purposes to isolate the effect of a change in a program, and it provides a way to compare the effect of more than one procedure on the same data. If no other way is available, successive values of Rand may be stored in an array or in a file.

In some languages, notably standard Pascal, the programmer might need to provide a pseudorandom number generator. The subject of generating pseudorandom numbers is too large to pursue here, but one popular algorithm is illustrated in Pascal:

FUNCTION Random (VAR Seed : INTEGER) : REAL;

CONST  Modulus = {65536};

Multiplier = {25173};

Increment = {13849};

BEGIN

Seed  : = ((Multiplier * Seed) + Increment) MOD Modulus;

Random : = Seed/Modulus

END {Random};

The values of the constants given here work well on a DEC VAX and other machines, but they will not work well in all cases. For example, for a PDP 11/70, Seed is declared externally by

VAR Seed : 0 . . 65535;

and the constants become:

Modulus = 32768;

Multiplier = 13077;

Increment = 6925;

The function Rand only produces values between 0 and 1. That does not explicitly allow, for example, the random choice of an index for an array declared as a[ - 2 . . 3]. The requisite mapping can be derived with the following steps:

The expression x = D [image] Rand has values in the range 0 < x < D when 0 < Rand < 1, and D > 0.

The expression x =L + D [image] Rand has values in the range L < x < L + D.

The expression x = L + (T - L) [image] Rand has values in the range L < x < T.

The expression INT(L + (T - L) [image] Rand) takes on integer values L, L + 1, . . . , T - 1 when INT is a function that returns the largest integer less than or equal to its argument.

Consequently, with T = U + 1, INT(L + (U - L + 1) [image] Rand) has possible values L, L + 1, . . . , U. In particular, INT(- 2 + 6 [image] Rand will randomly provide in-bounds indices for the array a[ - 2 . . 3].

A set of 10 values, randomly chosen in the range [- 30 . . 30], may be provided in the array e used in the sorting routines in section 1.5 by:

procedure ThirtyThirty                       {O(1)

for i = 1 to 10 do

e[i]    - 30 + 61 * Rand

next i

end {ThirtyThirty

It is not feasible to explore hypothesis testing, confidence intervals, analysis of variance, and so on in this text. That material belongs in a course in statistics. However, the applicability of Rand can be extended enormously by a few additional tools. Some acquaintance with the ideas of probability and statistics is helpful in reading the subsections that follow.

Program PGA.1 is appropriate at this point.

A.1 Mean and Standard Deviation

A sample of values from a population determines a (sample) mean that provides an estimate of the mean value of the population, and a (sample)standard deviation that is an estimate of the dispersion of the population. For a sample of size n they can be calculated by:

These measures provide a convenient and significant way to reduce a collection of measurements to two values, but intelligent interpretation then requires knowledge of statistics.

A.2 Frequency Distributions

A plot of how many minutes 20 people had to wait in line at the check-out counter of a grocery store might look like the histogram of Figure A.1.

Figure A.1

If this sample of size 20 is the only information about waiting time available, then it must provide the model for the probability, P[w], of waiting w minutes. That model is the relative frequency, which is scaled to the right of the histogram. Given that this model is a valid estimate, the relative frequency as a function of time can be accepted as a pmf (probability mass function). The accumulated (summed) values of a pmf are the cumulative probability. The distribution of cumulative probability is called the PDF (Probability Distribution Function). For this example, the PDF is plotted in Figure A.2.

Now suppose that we wish to write a program to process data that have the distribution model derived from the sample. We may provide a random variate, Line, for test data by the following:

Choose a value with Rand, uniformly distributed between 0 and 1. If PDF[w - 1] < Rand [image] PDF[w], then Line [image] w.

Geometrically, this amounts to drawing the horizontal line from a point on the vertical PDF axis to where it first strikes the stairstep graph and projecting down to a value w. The distribution of Line is that of the pmf (from the histogram), and points may be drawn from this distribution as required.

Figure A.2

Values that have the pmf of any discrete PDF may be generated from a table like Figure A.2 in this way, which is called discrete inversion.

Exercise EA.1 and Problem PA.1 are appropriate at this point.

A.3 Normal Distributions

The most-used probability distribution is not discrete as are those discussed in section A.1 but is the normal distribution:

Here, [image] is the mean value of x, and [image] is the standard deviation of its values. The standard normal distribution is defined as the one for which [image] = 0 and [image] = 1. The relationship given above between z and x defines a map for any normally distributed variable to z-scores, and back again. It is possible to approximate the distribution of z-scores, (the standard normal distribution), by using n values of Rand:

{Rand is called n times.

In particular, this equation is simplified if n is taken to be 12, in which case:

The z-scores generated in this way can be mapped to a value from a normal distribution by:

x =    +   z

A function that returns a value from a normal distribution is:

function Norm(Mu,Sigma)             {O(Rand)

z    0

for i = 1 to 12 do

z    z + Rand

next i

z    z - 6

Norm    Mu + Sigma * z

end {Norm

Note: Some caution must be applied to normal distributions because extreme values can occur, whatever [image] and [image] may be. Sometimes these values are not physically meaningful: the distribution being modeled differs from a normal distribution by not having them. They are "out of bounds" and must be dealt with by checks within the program.

Program PGA.2 is appropriate at this point.

A.4 Exponential Distributions

Arrival times for customers in line at a bank teller's window and for many other arrival and departure processes have neither a uniform distribution like Rand nor a normal distribution like Norm. The arrival times are more likely to form a Poisson Distribution for which the probability that n arrivals have occurred by time t is:

In practice, the time interval between arrivals is more useful. This is called the interarrival time, and it has an exponential distribution. A variate with an exponential distribution may be derived from Rand by

where LOG is the logarithm to the base e (2.718 . . . ). The distribution of values of ia has both mean and standard deviation equal to 1/ [image].

Exercises

Exercises immediate applications of the text material

EA.1 Develop the function Line that will return a value, with the set of such values distributed according to the histogram given in section A.2.

Problems

Problems not immediate, but requiring no major extensions of the text material

PA.1 The solutions in a high-school programming contest are timed to the nearest minute. Data from a preliminary contest for the first, "courage-builder," problem came out like this:

time       1  2  3  4  5  6  7  8  9  10

----------------------------------------

number

of         0  0  0  2  1  5  8  9  7   4

solutions

Plot the (relative) frequency distribution of this data. Assume that this represents a pmf, and plot the corresponding PDF. What times correspond to the following possible values of Rand?

(a) 0.007134 (c) 0.567321 (e) 0.410022

(b) 0.916528 (d) 0.757505

Programs

Programs for trying it out yourself. A program, when written for an interactive system, should display instructions that explain to the user what it does and how to use it. It should provide a graceful way for the user to quit.

PGA.1 Write a program that compares the execution times of SelectionSort, BubbleSort, and InsertionSort for 20 sets of 20 values, generated by the uniform variate, Rand. Calculate the mean and standard deviation for the three execution times.

In a time-sharing system it may not be possible to determine execution times themselves, but it is possible to count statement executions, including those that result from data-determined, conditional branches.

PGA.2 Do PGA.1, except that a normal distribution is to be used for data values, with mean 10 and standard deviation 10. For contrast, also use data already in order and in reverse order.

Projects

Projects for fun; for serious students

PJA.1 Extend PGA.1 and PGA.2 to a serious study of SelectionSort, BubbleSort, and InsertionSort for values of n = 8, 16, 32, 64, 128, 512, and 2048. Use uniform, normal, and exponential variates, in-order and reverse-order data sets, and data sets that are somewhat disarranged from in-order in a variety of ways. For example, choose two indices at random from an in-order sequence and switch the values they point to prior to invoking the sort procedure being studied. Calculate means and standard deviations of execution time. If you have statistics in your background, test the hypothesis that the three sorts have equal execution times (under various circumstances).

Go to Section B Return to Table of Contents


Data Structures: SECTION B: HASH TABLES

 Return to Table of Contents

SECTION B: HASH TABLES

A hash table differs from the array that provides storage for it because store and retrieve operations are specified by a key value rather than by an index and the number of legal keys is larger than the number of cells in the array. The operations of interest are STORE(Key) and RETRIEVE(Key). It is possible to simply store the key and retrieve a Boolean function that indicates the presence or lack of presence of the key in the table. This is the simple way to discuss hash tables, but it is more likely that the key is indeed just a key in a record containing a Value field or a pointer (or index) that addresses some bulk of information. In a word, hash tables are apt to be exogenous--directories to data stored external to them.

Note: For the sake of simpler exposition in this section, STORE(Key) assigns only a key value and RETRIEVE(Key) returns only the index of the location of Key in a table of keys. These are to be understood as surrogates for more realistic forms of the operations.

The function HF1 of section 2.5 simply converts letters to integers and sums them. It exhibits two of the properties needed to form a hash table:

HF1 transforms keys into integers.

HF1 possesses compression: the domain space defined by the set of all possible keys is much larger than the range space, the target table.

HF1 also exhibits an unhappy consequence of compression by hash functions:

HF1(BROTH) = HF1(OLIVE)

In short:

Compression causes collisions.

The address functions of section 2.3 transform a domain value that is an n-tuple of the form

[i1, i2, . . ., in]

into a range value that is a single index in a one-dimensional table. The correspondence is one-to-one, and so there is no compression; but compression can be introduced much as it was for HF1. For example, a function HF2 can be defined by

HF2([i1, i2, . . ., in]) = (i1 +          + in) MOD 25

If a key of any kind can be located in an n-dimensional table at

t = [i1, i2, . . ., in]

then HF2(t) provides the same service as for any n-dimensional table of keys as HF1 does for its specific data space.

Both HF1 and HF2 fail to define their range as a hash table structure by themselves because compression allows collisions: two elements of the domain can have identical range values. In any table structure, RETRIEVE must be the inverse of STORE, and so collisions must be resolved. With this in mind, the desired HASH TABLE structure consists of:

a domain

a range

a hash function h that exhibits compression

a resolution of the collisions caused by h

operations CREATE, STORE, and RETRIEVE

The domain is normally a given in a general sense and may be enormous. For example, it might be identifiers of up to 30 characters in length. The subset of the domain encountered in a specific application tends to be smaller but not well defined, and that makes a compressive system very attractive. CREATE requires no special attention. The range is a design decision that takes into account much of this section.

Suppose that a hash function is available that distributes the domain keys over the indices 0,1, . . . ,24 in a table HT with uniform probability. The first entry will not collide with any other. The probability that the second entry will not collide with a previous entry is 24/25. The probability that the third, fourth, . . . entries will not collide when entered is 23/25, 22/25, . . . (directly proportional to the number of free spaces). The probability that there will be no collisions for seven entries in a table of 25 cells is:

Hence the probability that there will be a collision is near 60 percent even though the table is less than one-third full. The cause is not really the small size of the table. (Figure out how likely a birthday collision is in a room containing 25 people.)

The ratio of n/m for n items in a table of size m is called the load factor, [image]. In this example, [image] = 7/25 = 0.28. Clearly, the probability of a collision increases with [image].

The likelihood of collisions is increased if regularities in the domain keys affect the hash value. For example, a hash function based simply on the last letter of an identifier will map a lot of keys into wherever e, s, and n happen to go. (Try it on the words of the first two sentences of this paragraph.) Every part of a key should be involved--else, for example, the set {THEMES, THEME, THYME, TIME, CHIME, CHINA} will tend to collide. A hash function should tend to scatter natural clusters of the domain over the range.

The probability that a table location will be accessed is affected by two things:

the selection of elements from the potential set (the domain)

the transformation of the selected elements by the hash function into range indices

If this probability is not uniform, then there is said to be primary clustering. HF1 and HF2 are not particularly good hash functions. For one thing, all permutations of the same letters (or of the same indices) collide.

B.1 A Choice of Hash Functions

Although no hash function serves well in all circumstances, there are theoretical results that provide guidelines. One approach that can be recommended for integer values of z is

h(z) = INT(m(z   MOD 1))

which satisfies 0 [image] h(z) < m when 0 < [image] < 1. The table indices addressed by h are 0 . . . m - 1. Suggested values of [image] are:

The size of the range table is chosen to be a prime number that is not close to a power of 2 (admittedly difficult for small tables).

To use the function h as a hash function first requires that the keys be turned into an integer in some efficient way. Keys are stored as binary strings in memory, and a binary string can be decoded as an integer (or sequence of integers) or converted with a utility function (such as ORD in Pascal). For the sake of illustration, consider the following simple approach applied to the ingredients of the recipe in section 2.5:

1. Ingredients are considered to be 12 characters in length, left-adjusted and padded with blank spaces.

2. Letters have as their index a two-digit value: A = 01, B = 02, . . ., Z = 26. Blanks have value 00. These values are found by a table lookup, which is much faster than calculation of them.

3. An ingredient is six four-digit integers. For example, BELLPEPPER is 0205, 1212, 1605, 1616, 0518, 0000.

4. The six integers for an ingredient are summed to form z. In functional terms, q(BELLPEPPER) = 5156 = z.

The calculation of q is much faster than it might appear to be. Let s[1 . . 12] be the string of characters of an ingredient. Then a calculation of z requires only one multiplication:

z    s[1] + s[3] + s[5] + s[7] + s[9] + s[11]

z    z X 100 + s[2] + s[4] + s[6] + s[8] + s[10] + s[12]

It is possible for z to be as large as 6 X 2626 = 15,756, but taking the sum MOD 10,000 restricts it to four digits (and more than that are unlikely anyway).

Now h(z) for BELLPEPPER, with m = 17 is:

h(z) = INT(m(z   MOD 1))

= INT(17(5156     MOD 1))

= INT(17(3186.5827 MOD 1))

= INT(17(0.5827))

= INT(9.9059)

= 9

For contrast, a second stage of multiplication by the prime base m can be introduced:

h2(z) = INT(m(m(z   MOD 1) MOD 1))

= INT(17(9.9059 MOD 1))

= 15

Both h and h2 can be easily determined with a hand calculator or a small program as a check on how well they work on a particular data set. Multiplication by 100 can be replaced with multiplication by a power of 2 and hence--in assembly languages--by shifts. There are a large number of possible variations of this scheme.

The results, for the 13 ingredients (including BROTH) and prime bases 17, 31, and 47, are shown in Table B.1 .

Table B.1

                    m = 17  m = 31   m = 47

               z    h   h2   h   h2   h   h2

-------------------------------------------

BEEF         0711   7    8  13    2  19  39

BELLPEPPER   5156   9   15  18    1  27  18

BLACKPEPPER  5342   9    2  16   20  25  11

BROTH        2538   9   11  17   20  26  37

CARROT       3639   0    7   0   24   1   8

CUMIN        3030  10   15  19   28  30   9

DILLWEED     2330   0    5   0   18   0  41

MUSHROOM     6557   7   10  13   27  21   3

OLIVE        2934   5    4   9   20  14  29

ONION        2443  14    9  26   17  40  12

POTATO       5631   2    9   4   18   6  46

SALT         3121  15    0  27   12  41  25

TOMATOPASTE  9352  14    8  26   13  40   4

Collisions are produced by all of these functions, except h2 for the base of 47. In fact, that is a lucky accident. The loading factor for 13 items in a table of 47 items is 13/47 = 0.28. More importantly, the probability of no collisions with 13 items, no matter how randomly h2 scatters its key, is:

There is less than a 50/50 chance that ten pigeons can be placed at random in 47 pigeon holes without having two pigeons in one hole.

As the size of the table increases, the minimal loading factor with a chance of collision less than 50 percent shrinks. For m = 997 it is less than 4 percent.

Problem PB.1 is appropriate at this point.

The unavoidable conclusion is that any practical use of hash tables produces collisions. One way to implement collision resolution is a technique called linear probing.

B.2 Linear Probing

Suppose a procedure Hash calculates an index k in a hash table HT[0 . . m - 1] for Key. If the table is empty, STORE(Key) would simply be HT[k] [image] Key. If it is not, HT[k] may already contain a key, and a collision occurs. The first task of a STORE operation is thus to determine if HT[k] is empty. For that purpose HT might be initialized with a value that cannot be a key or associated with a flag array. For the recipe example and most identifiers, HT may be initialized to null, blank, or empty strings.

If HT[k] is not empty, then another location must be chosen for Key, and it must be found again by RETRIEVE(Key). One simple strategy is to start looking at index k + 1, then k + 2, and so on until a free cell is located, or the table is exhausted without finding one. This is called linear probing.

A general probe function for a hash table begins with a starting index and the key value and returns an index. The return value is the index of the actual location of the key if it is found in HT, or the negative of that index if the table was searched to exhaustion. An appropriate empty slot for a STORE operation can then be located by probing with the null string Blank.

function Probe(Start,s)                  {O(m)

i    Start

repeat

if HT[i] = s

then return i endif

i    (i + l) MOD m

until i = Start

return - i

end {Probe

procedure StoreHT(Key)                   {O(m)

k    Hash(Key)

dex    Probe(k,Blank)

if dex < 0

then {deal with a full table

else HT[dex]    Key

endif

end {StoreHT

function RetrieveHT(Key)                 {O(m)

k    Hash(Key)

dex    Probe(k,Key)

if dex < 0

then {report not there

else return dex

endif

end {RetrieveHT

If the user of RetrieveHT(Key) is designed to interpret negative returns as "not there," then RetrieveHT may be replaced by Probe(Hash(Key), Key).

The effect of StoreHT can be illustrated with the use of m = 17, h, and the 13 ingredients in the example, entered in alphabetical order:

                               m = 17

HT[7]      BEEF          0  CARROT

HT[9]      BELLPEPPER    1  DILLWEED

HT[9]   collision        2  POTATO

HT[10]     BLACKPEPPER

HT[9]   collision

HT[10]  collision        5  OLIVE

HT[11]     BROTH

HT[0]      CARROT        7  BEEF

HT[10]  collision        8  MUSHROOM

HT[11]  collision        9  BELLPEPPER

HT[12]  CUMIN           10  BLACKPEPPER

HT[0]   collision       11  BROTH

HT[1]      DILLWEED     12  CUMIN

HT[7]   collision

HT[8]      MUSHROOM     14  ONION

HT[5]      OLIVE        15  SALT

HT[14]     ONION        16  TOMATOPASTE

HT[2]      POTATO

HT[15]     SALT

HT[14]  collision

HT[15]  collision

HT[16]     TOMATOPASTE

Problem PB.2 is appropriate at this point.

B.3 Secondary Clustering and Double Hashing

Two things about the sequence of HT entries are worth noting:

      A different entry order stores some of the keys in different places. For the example, entry in reverse alphabetic order would place CUMIN in HT[10] and BELLPEPPER in HT[12].

      Entries into HT[k] can increase the number of comparisons needed to store (or retrieve) items that hash to nearby indices, both smaller and larger. This is called secondary clustering as opposed to primary clustering in which distinct keys hash to the same index.

Secondary clustering can be alleviated to a certain extent (in tables with more room to maneuver than this example) by double hashing. This consists of skipping through the table from a collision in steps of   [image], where   [image]  depends on the key. In Probe,   [image]  was 1 for all keys, and a workable change is:      HashAgain(Key) followed within the search loop by: i   [image]  (i +   [image]) MOD m.

It is necessary for   [image]  to be relatively prime to m (else the search will be compromised). One way to guarantee that is to use Hash and HashAgain that satisfy:

0    Hash(z) < m

0    HashAgain(z) < m

Hash(z)    HashAgain(z)

A slight variation applicable to the example is to shift the four-digit values over one, and derive a new z, say z2. For example, z2 for BELLPEPPER would be

2051 + 2121 + 6051 + 6160 + 5180 + 0000 = 21,563

Or simply use 1 -   [image]  in place of   [image], or use h2.

With q, Hash = h, HashAgain = h2, m = 31, and alphabetical entry order:

HT[13]     BEEF                        m = 31

HT[18]     BELLPEPPER            0  CARROT

HT[16]     BLACKPEPPER

HT[17]     BROTH

HT[0]      CARROT

HT[19]     CUMIN                 4  POTAT0

HT[0]     collision              5  DILLWEED

HT[18]    collision

HT[5]      DILLWEED (   = 18)

HT[13]    collision              8  TOMATOPASTE

HT[9]      MUSHROOM (   = 27)      9  MUSHROOM

HT[9]     collision

HT[29]     OLIVE (   = 20)

HT[26]     ONION

HT[4]      POTAT0                13  BEEF

HT[27]     SALT

HT[26]  collision

HT[8]      TOMATOPASTE (   = 13)  16  BLACKPEPPER

                                 17  BROTH

                                 18  BELLPEPPER

                                 19  CUMIN

                                 26  ONION

                                 27  SALT

                                 29  OLIVE

Actually there is one more collision than for   [image]  = 1, but additional entries mapped into the range 16 - 19 by Hash are less likely to interfere with each other when their   [image] s differ, because there is a good chance of skipping over the cluster.

The number of collisions increases rapidly after the loading factor n/m reaches about 3/4, as can be seen by applying double hashing with h2 and m = 17. There is nowhere to go in a small table to escape collisions.

Problem PB.3 is appropriate at this point.

B.4 Deletion and Rehashing

Deletion from a hash table is not merely a matter of replacing a key with the empty key, because a subsequent search may need to keep stepping past its cell into the sequence of stored entries with the same hash value. A deletion cell can be marked as available with some special value and used on a subsequent store that reaches it. (Since retrieval must continue past this point, separate Probe procedures are needed for store and retrieve operations.)

It is possible to devise an algorithm that rehashes a table. The number of deletions and the loading factor probably both affect the optimal point to do rehashing, which is not particularly efficient. It is probably just as well to make a linear sweep through the table and hash active keys into a new copy of the table, as long as there is room for two copies of the table in memory.

An alternate implementation of hash tables that permits deletion is based on the linked lists described in Chapter 3, and it is to be found in section L.

Problems

Problems not immediate, but requiring no major extensions of the text material

PB.1     Construct a table of hashed indices like the one of section B. 1 for the calendar months.

PB.2     Construct the hash table for the calendar months entered in calendar order, into a hash table of      prime base m = 17.

PB.3     Construct the double-hashing table for the calendar months in calendar order with Hash = h, HashAgain = h2, and m = 31.

Programs

Programs for trying it yourself

PGB.1     Write a program that accepts identifiers of up to 12 characters in length, converts them to an      integer with q, and stores them in a hash table HT with the use of Hash = h, HashAgain = h2, and      m = 47. The program should report the number of collisions, the average number of tests by      Probe, and the loading factor.

Go to Section C            Return to Table of Contents


Data Structures: SECTION C: CIRCULAR LIST OPERATIONS

 Return to Table of Contents

SECTION C: CIRCULAR LIST OPERATIONS

A display routine for a circular list differs from that of a linear list because the end of an unsuccessful search is signaled by a return to the starting point instead of by a NIL pointer:

procedure RoundOut(cList)              {O(n)

if Empty(cList)

then {deal with this error

else Display(cList)               {PROCESS0

p    cList    .Link            {PROCESS0

while p    cList do

Display(p)                 {PROCESS2

p    p    .Link

endwhile

endif

end {RoundOut

Additions to a circular list, cList, can take place most easily if the new node is to follow immediately after the node distinguished by the pointer cList, as shown in Figure C.1.

Figure C.1

The insertion routine is straightforward:

procedure InsertAfter(cList,xNode)          {O(1)

if Empty(cList)

then xNode    .Link    xNode

cList    xNode

else xNode    .Link    cList    .Link

cList    .Link    xNode

endif

end {InsertAfter

When a single list identifier, cList, is used, and a node is to be added prior to it, then it is necessary to locate the node that points to cList, say Prior, and then apply InsertAfter(Prior,xNode), as shown in Figure C.2.

Figure C.2

The location logic is a simple version of LT (section 3.3.2):

Prior    cList

if NOT Empty(cList)

then p    cList    .Link

while p    cList do

Prior    p

p    p    .Link

endwhile

endif

Deletion in the vicinity of cList can take several forms. In particular, the node cList [image] .Link (if it exists) can be readily deleted. So can cList; in Figure C.3 the cList pointer value can move forward or backward relative to the link directions.

Figure C.3

Some combinations of insertion and deletion operations define formal structures in common use. Of particular interest is the circular queue, discussed in Chapter 5 :

A circular queue: InsertAfter(cList,xNode)

DeleteBack(cList)

The form taken by DeleteNext is simply:

procedure DeleteNext(cList)                  {O(1)

if Empty(cList)

then {deal with this error

return

endif

if cList    .Link = cList

then cList    NIL

return

else Next    cList    .Link

cList    .Link    Next    .Link

endif

end {DeleteNext

A deletion procedure such as DeleteBack is awkward unless the pointer Prior is available. Such a pointer can be made available by a change to a two-window system, as shown in Figure C.4.

Figure C.4

With the window-pair system, circularity appears to be specious. Nevertheless, circular lists defined with a pair of pointers turn out to be useful when they are maintained in a single array. Routines designed to maintain the single array, specialized to the structure QUEUE, are treated in section 5.3.

Details of the other deletion procedures are left to the exercises.

Exercises

Problems

Exercises

Exercises immediate applications of the text material

EC.1 When insertion prior to cList is made to an empty list something must be done after the search logic and InsertAfter(cList,xNode) of section 3.7 to properly prepare for later insertions. What is it?

Problems

Problems not immediate, but requiring no major extensions of the text material

PC.1 Develop the procedure DeleteOn.

PC.2 Develop the procedure DeleteBack.

PC.3 Develop the procedure DeleteLast.

Go to Section D Return to Table of Contents


Data Structures: SECTION D: INTEGER ARITHMETIC OF UNBOUNDED PRECISION

 Return to Table of Contents

SECTION D: INTEGER ARITHMETIC OF UNBOUNDED PRECISION

The phrase "unbounded precision" means that the program providing the calculation itself places no limitation on the number of digits in a value. There are limitations to the storage capacity of any computing system; but they are generally very large relative to values that are likely to be used, and the program simply ignores that bound.

The program Unbound presented in this section is a simulation of a calculator used in infix mode: a value, an operator, and then a value are entered and the result display is prompted by the entry of a character " = '' or one that represents another operator. The functions provided are addition ("+"), subtraction ("-"), multiplication ("*"), division ("/"), display ("="), clear the accumulator ("C''), and quit ("Q"). The inclusion of M + (add to memory), M - (subtract from memory), MRC (recover from memory), and MCL (memory clear) are left as a challenge.

The program Unbound is developed in this section, rather than simply described or listed, in order to document the series of design decisions implemented by the program. The logic of Unbound is clarified by developing it in pseudocode and then translating that logic into Pascal.

D.1 The Entry Sequence

A crucial feature of a calculator is that it is fairly robust: mistaken entries do little harm, and that feature should be retained in a simulating program. The possibilities for ambiguous entries are much increased with the use of a terminal keyboard in place of the more restricted calculator keypad. In that spirit, extraneous characters are ignored. The accepted characters are digits 0 -9 and operators ''+'', "-", ''*'', ''/'', '' = '', ''C'', ''Q''. All others are ignored. For example, A17B3 is accepted as 173. For the sake of robustness, division by 0 simply results in 0.

The basic response of the program is designed to deal with entry sequences such as: 24 + 9 =, which results in a display of 33 and the retention of that value in an accumulator (represented by the variable AC). A string of digits that designates a number is terminated by an operator, and so it is (value,operator)-pairs that are accepted by the program. Although an operator terminates the entry of a number, it is the previous operator that usually determines the action. (Operators such as ''C" and ''Q'' are exceptions.) In more detail, if the entering value is X and the result of arithmetic is held in AC, the action of this sequence can be traced as shown in Table D.1.

Table D.1

Entries   X  NewOp  Op  AC

--------------------------

                     C   0

  24+    24    +     C  24

  9=      9    =     +  33

The operation of most calculators is more complex than this example seems to imply. What, for example, is the result of the following entry sequences?

- 24 + 9 =

- 24 + 9 +

24 + +

24 + + =

24 + + + =

The results generated by the program developed here are -15, -15, 24, 48, and 72, respectively. The last of these is not standard, but it is convenient to have each new operator invoke the response of the previous one. If no new X is entered, the AC value is used as X.

D.2 Utility Routines

A procedure Preface is used to initialize variables and provide instructions to the user; Clear is used to set registers such as AC to zero; and Display writes out the value in the accumulator AC. These utility routines are supplemented by arithmetic procedures that act on AC.

The procedure ValueIn that extracts the tokens X and NextOp also returns a Boolean flag NoX to indicate the presence of a numeric value between operators. The command-shell executes the previous operator, Op, in response to the next operator, NextOp. If no X-value is entered withNextOp, then the current value of AC is copied into X. Operators "Q'' and ''C'' require immediate response. When "Q'' is NextOp, it causes immediate termination of the program. When NextOp is ''C'' both AC and X are cleared, and Op is set to "C'' also. When Op is either "C'' or ''='', and an X-value is entered with NextOp, the X-value becomes the new AC value.

The command-shell program that results is:

program UnBound

Preface

Done    FALSE

Op    'C'

repeat

ValueIn(X,NoX,NextOp)

if NextOp = 'Q'

then Done    TRUE

else if NextOp = 'C'

then Clear(AC)

Clear(X)

Op    'C'

endif

if NoX then X    AC endif

case of Op

'=', 'C': Swap                     {AC with X

'+'     :Add(AC,X)

{'-', '*', '/' are similar to '+'}

endcase

endif

Op    NextOp

Display

until Done

end {UnBound

The next level of detail is dependent on the implementation of values, and that implementation must allow assignments X [image] AC and AC [image] X for unbounded strings of digits. Values need to carry a sign, have leading zeroes suppressed for the sake of efficiency and display, be compared in magnitude (left-to-right for corresponding digits), and be used right-to-left in arithmetic operations. The chosen implementation for AC and X is a pointer to a header of a doubly linked list. Such a list plays the role of a register--a storage location directly associated with computation. For reasons that only become apparent when operations are examined in detail, it is convenient to carry one leading zero and link it to itself. When AC [image] 743, the resulting register structure is shown in Figure D.1.

Figure D.1

The Sign field is Boolean, with TRUE corresponding to positive values--making tests involving signs more convenient.

A header such as AC or X can then be initialized by:

procedure NewRegister(R)

NEW(d)

d    .Left    d

d    .Value    0

d    .Right    NIL

NEW(R)

R    .First    d

R    .Last    d

R    .Sign    TRUE

R    .Count    0

end  {NewRegister

Setting an initialized register back to zero can be even simpler:

procedure Clear(R)

R    .Last    R    .First

R    .Count    0

R    .Sign    TRUE

end  {Clear

This procedure, however, does not recover the (possibly many) digit nodes that were in the register R. A more practical version of Clear is:

procedure Clear(R)

p    R    .Last

q    p    .Left

while  p    R    .First do

DISPOSE(p)

p    q

q    p    .Left

endwhile

p    .Right    NIL

R    .Last    p

R    .Count    0

R    .Sign    TRUE

end  {Clear

The speed of arithmetic operations is directly dependent on the number of digits involved, and so removal of duplicate leading zeroes is an essential part of housekeeping:

procedure Compact(R)

p    R    .First

q    p    .Right

while q    NIL do

if q    .Value    0

then exit

else t    q    .Right

DISPOSE(q)

R    .Count    R    .Count - 1

P    .Right    t

q    t

if t    NIL

then t    .Left    p endif

endif

endwhile

if R   .Count = 0 then R   .Last    R   .First endif

end {Compact

Note: The pair of tests at the top of this loop require a fair amount of modification when implemented in Pascal because the condition:

(q    NIL) AND (q    .Value    0)

is risky. A more direct translation results from the use of GOTO to implement exit.

On the principle of "never trust the user," compaction is included as the last action in ValueIn.

The input-token generator ValueIn is:

procedure ValueIn(X,NoX,NextOp)

NoX    TRUE

Clear(X)

Read(ch)

while NOT (ch IN OperatorSet) do

if ch IN [0..9]

then NoX    FALSE

NEW(d)

d    .Right    NIL

d    .Value    ORD(ch) - ORD('0')

d    .Left    X    .Last

X    .Last    .Right    d

X    .Last    d

X    .Count    X    .Count + 1

endif

Read(ch)

endwhile

NextOp    ch

Compact(X)

end {ValueIn

D.3 Arithmetic Operations

The fundamental operation of arithmetic is addition, but it becomes subtraction when the signs of the operands differ. When done by hand, the value of smaller magnitude is subtracted from the other. One simple approach to addition is based on two steps:

1. Compare AC and X and switch them if necessary to insure that [image] AC [image] [image] [image] X [image].

2. If the signs agree, then add magnitudes with procedure Carry; otherwise, subtract the magnitudes with procedure Borrow.

The resulting logic is:

procedure Add(X)

R    LargerOf(AC,X)

if R = X

then X    AC

AC    R

endif

if X    .Count = 0 then return endif

if AC    .Sign = X    .Sign

then Carry(AC,X)

else Borrow(AC,X)

endif

end  {Add

function LargerOf(AC,X)        {O(n) Assumes compacted operands

if AC    .Count > X    .Count  then return AC endif

if X    .Count > AC    .Count  then return X endif

k    AC    .Count

pAC    AC    .First    .Right

pX    X    .First    .Right

while k > 0 do

if pAC    .Value > pX    .Value  then return AC endif

if pAC    .Value < pX    .Value  then return X endif

pAC    pAC    .Right

pX    pX    .Right

k    k - 1

endwhile

return AC                    {X = AC

end  {LargerOf

Note: The returns are replaced by a nest of IF-statements in Pascal. Add, Subtract, Multiply, and Divide all may switch X and AC to avoid copying a long list of digits, and so X is a VAR parameter in the Pascal implementation.

The last carry operation can create a new digit (as in 997 + 18), which is signaled by a carry after the last X-digit has been added to AC. The last borrow can propagate to the left (as in 10003 - 4), but that can be handled by continuing to subtract the leading zero node, X   [image]  .First. With these problems recognized:

procedure Carry(X)

pAC    AC   .Last

pX    X    .Last

Over    0

for k = 1 to AC    .Count do

Sum    pAC    .Value + pX    .Value + Over

if Sum > 9

then Over    1

pAC    .Value    Sum - 10

else Over    0

pAC    .Value    Sum

endif

pAC    pAC    .Left

pX    pX    .Left

next k

if Over > 0

then NEW(d)

d    .Right    pAC    .Right

d    .Left    pAC

pAC    .Right    .Left    d

pAC    .Right    d

d    .Value    Over

AC    .Count    AC    .Count + 1

endif

end {Carry

procedure Borrow

pAC    AC    .Last

pX    X    .Last

Loan    0

for k = 1 to AC    .Count do

Difference    pAC   .Value - pX    .Value - Loan

if Difference < 0

then Loan    1

pAC    .Value    Difference + 10

else Loan    0

pAC    .Value    Difference

endif

pAC    pAC    .Left

pX    pX    .Left

next k

Compact(AC)

end {Borrow

With Add complete, subtraction becomes very simple:

procedure Subtract(X)

X    .Sign    NOT X    .Sign

Add(X)

end {Subtract

Multiplication can be done by repeated addition in two ways: 47   [image]  31 = (47 + 47 + . . . + 47, 31 times) and 47   [image]  31 = (47 + 3   [image]  470). The latter approach is used here. It will square 2 raised to the 96th power in a few seconds on an inexpensive microcomputer (after compilation by a $30 Pascal compiler). This operation would take years by repeated addition.

Several values are involved in multiplication: the original operands X and AC, the shifting values Y (such as 47, 470, . . . ), the product of a digit of X with a Y, say M, and the running sum of these products. AC supplies the original Y and then may contain the running sum. The product sign must be determined but not affect the use of Add to multiply magnitudes, and a value with a minimal number of digits should be used to control the multiplication loop. This housekeeping is done by:

procedure SetUp

Y    AC

NewRegister(AC)

New Register(M)

R    LargerOf(Y,X)

if R = X

then X    Y

Y    R

endif

NewSign    (Y    .Sign = X    .Sign)

X    .Sign    TRUE

Y    .Sign    TRUE

Clear(AC)

end {SetUp

A routine is needed to shift successive products one digit to the left. One approach is to shift Y:

procedure ShiftLeft(Y)

NEW(d)

d    .Value    O

d    .Right    NIL

d    .Left    Y   .Last

Y    .Count    Y    .Count + 1

Y    .Last    .Right    d

Y    .Last    d

end {ShiftLeft

Multiplication then can be done with:

procedure Multiply(X)

SetUp

if X    .Count = 0 then return endif

pX    X    .Last

for j = 1 to X    .Count do

Over    0

MakeCopy(M,Y)

pM    M    .Last

pX    X    .Last

v    pX    .Value

for k = 1 to M    .Count do

Product    pM    .Value * v + Over

pM    .Value    Product MOD 10

Over    Product DIV 10

pM    pM    .Left

next k

if Over > 0

then NEW(d)

d    .Right    pM    .Right

d    .Left    pM

pM    .Right    .Left    d

pM    .Right    d

M    .Count    M    .Count + 1

d    .Value    Over

endif

Add(AC,M)

ShiftLeft(Y)

pX    pX    .Left

next j

AC    .Sign    NewSign

Clear(Y)

Clear(M)

end {Multiply

Division by repeated subtraction is as slow as multiplication by repeated addition, but the alternative algorithm used by fifth graders is much more complex than repeated subtraction. For contrast with multiplication, a version of repeated subtraction is applied here: the divisor is added to itself until it is larger than the dividend. Each such addition is paired with an increment of the result.

Auxiliary registers are used as they are for multiplication and destroyed on exit from Divide. The initialization is:

procedure Prolog

NewSign    (AC    .Sign = X    .Sign)

AC    .Sign    TRUE

X    .Sign    TRUE

D    AC

NewRegister(AC)

NewRegister(R)

NewRegister(S)

end {Prolog

The resulting divide logic is:

procedure Divide(X)

if X    .Count = 0

then Clear(AC)

return

endif

Prolog

MakeCopy(S,X)

MakeCopy(AC,One)

R    LargerOf(D,S)

while R = D do

Carry(AC,One)

Carry(S,X)

R    LargerOf(D,S)

endwhile

Borrow(AC,One)

AC    .Sign    NewSign

Clear(D)

Clear(S)

end {Divide

A listing of a Pascal program that results from this design follows:

PROGRAM UnBound (InPut,OutPut);

TYPE  Digit =    DigitNode;

Register =    RegisterNode;

DigitNode = RECORD

Value : 0 . . 9;

Left : Digit;

Right : Digit

END;

RegisterNode = RECORD

Sign : BOOLEAN;

Count : INTEGER;

First : Digit;

Last : Digit

END;

VAR  Op ,NextOP : CHAR;

NoX,Done  : BOOLEAN;

One,X,AC  : Register;

OperatorSet : SET OF CHAR;

Numeric  : SET OF CHAR;

{  }

PROCEDURE Display (AC : Register);

VAR pAC : Digit;

k : INTEGER;

BEGIN

Write1n;

pAC := AC    .First;

IF  NOT (AC    . Sign)  THEN Write('-');

FOR  k := 1 TO (AC    . Count + 1) DO BEGIN

Write(pAC    .Value);

pAC := pAC    . Right

END;

Write1n

END {Display};

{  }

PROCEDURE NewRegister(VAR R : Register);

VAR d : Digit;

BEGIN

NEW(d);

d    .Left := d;

d    .Value := 0;

d    .Right := NIL;

NEW(R);

R    .First := d;

R    .Last := d;

R    .Sign := TRUE;

R    .Count := 0

END   {NewRegister};

{  }

PROCEDURE Clear (VAR R : Register);

VAR p , q : Digit;

BEGIN

p := R    .Last;

q := P    .Left;

WHILE (p <> R    .First) DO BEGIN

DISPOSE(p);

p := q;

q := p    .Left

END;

R    .Last := p;

R    .Count := 0;

P    .Right := NIL;

R    .Sign := TRUE

END {Clear};

{  }

PROCEDURE MakeOne(VAR One : Register);

VAR d : Digit;

BEGIN

NewRegister(One);

NEW(d);

d    .Value := 1;

d    .Left := One    .First;

d    .Right := NIL;

One    .First    .Right := d;

One    .Last := d;

One    .Count := 1

END {MakeOne};

{  }

PROCEDURE Preface;

BEGIN

Write1n('This program simulates a calculator that');

Write1n('allows unbounded integers. Enter values');

Write1n('followed by operators. The accepted operators');

Write1n('are: +,-,*,/,=,C (for clear),Q (for quit).');

NewRegister(AC);

NewRegister(X);

MakeOne(One);

Numeric := ['0'..'9'];

OperatorSet := ['+','-','*','/','=','C','Q']

END  {Preface};

{  }

PROCEDURE Compact(VAR R : Register);

VAR p, q, t : Digit;

BEGIN

p := R    .First;

q := p    .Right;

IF  (q  <>  NIL) THEN IF (q    .Value <> 0) THEN q:= NIL;

WHILE   (q <> NIL)  DO BEGIN

t := q    .Right;

DISPOSE(q);

R    .Count := R    .Count - 1;

P    .Right := t;

q := t;

IF (t <> NIL)

THEN BEGIN

t    .Left := p;

IF (t    .Value <> 0) THEN q := NIL

END

END;

IF  (R    .Count = 0) THEN R    .Last := R    .First

END  {Compact};

{  }

PROCEDURE ValueIn(VAR X : Register; VAR NoX : BOOLEAN;

VAR NextOP : CHAR);

VAR  ch : CHAR;

d : Digit;

BEGIN

NoX := TRUE;

Clear(X);

Read(ch);

WHILE NOT (ch IN OperatorSet)

DO BEGIN

IF  (ch IN Numeric)

THEN BEGIN

NoX := FALSE;

NEW(d);

d    .Right := NIL;

d    .Value := ORD(ch) - ORD('0');

d    .Left := X    .Last;

X    .Last    .Right := d;

X    .Last := d;

X    .Count := X    .Count + 1

END;

Read(ch)

END;

NextOp := ch;

Compact(X)

END  {ValueIn};

{  }

FUNCTION LargerOf(AC,X : Register) : Register;

VAR  K : INTEGER;

pAC,pX : Digit;

BEGIN

LargerOf := AC;

IF   (AC    .Count <= X    .Count) THEN

IF     (AC    .Count < X    .Count)

THEN LargerOf := X

ELSE BEGIN

k := AC    .Count;

pAC := AC    .First    .Right;

pX := X    .First    .Right;

WHILE (k > 0) DO

IF (pAC    .Value < = pX    .Value)

THEN IF  (pAC    .Value < pX    .Value)

THEN BEGIN

LargerOf := X;

K := O

END

ELSE BEGIN

pAC := pAC    .Right;

pX := pX    .Right;

k := k - 1

END

ELSE k := O

END;

END  {LargerOf};

{  }

PROCEDURE MakeCopy(VAR M : Register; Y : Register);

VAR  d,pY : Digit;

k : INTEGER;

BEGIN

Clear(M);

IF  (Y    .Count > 0) THEN BEGIN

pY := Y    .First    .Right;

FOR k := 1 TO Y    .Count DO BEGIN

NEW(d);

d    .Right := NIL;

d    .Value := PY    .Value;

pY := pY    .Right;

d    .Left := M    .Last;

M    .Last    .Right := d;

M    .Last := d

END;

M    .Count := Y    .Count;

END

END {MakeCopy};

{  }

PROCEDURE Carry(VAR AC : Register; X : Register);

VAR  pAC, pX, d : Digit;

k,Sum, Over : INTEGER;

BEGIN

pAC := AC    .Last;

pX := X    .Last;

Over := 0;

FOR K := 1 TO AC    .Count DO BEGIN

Sum := pAC    .Value + pX    .Value + Over;

IF  (Sum > 9)

THEN BEGIN

Over := 1;

pAC    .Value := Sum - 10

END

ELSE BEGIN

Over := 0;

pAC    .Value := Sum

END;

pAC := pAC    .Left;

pX := pX    .Left

END;

IF  (Over > 0)

THEN BEGIN

NEW(d);

d    .Right := pAC    .Right;

d    .Left := pAC;

pAC    .Right    .Left := d;

pAC    .Right := d;

d    .Value := Over;

AC    .Count := AC    .Count + 1

END

END {Carry};

{  }

PROCEDURE Borrow(VAR AC: Register; X: Register);

VAR  pAC,pX: Digit;

k,Loan,Difference: INTEGER;

BEGIN

pAC := AC   .Last;

pX := X   .Last;

Loan := 0;

FOR k := 1 TO AC   .Count DO BEGIN

Difference := pAC   .Value - pX   .Value - Loan;

IF  (Difference < 0)

THEN BEGIN

Loan := 1;

pAC   .Value := Difference + 10

END

ELSE BEGIN

Loan := 0;

pAC   .Value := Difference

END;

pAC := pAC   .Left;

pX := pX   .Left

END;

Compact(AC)

END {Borrow};

{  }

PROCEDURE Add(VAR AC: Register; VAR X : Register);

VAR R: Register;

BEGIN                   {X is VAR as AC may swap with it.}

R := LargerOf(AC,X);

IF  (R = X)

THEN BEGIN

X := AC;

AC := R

END;

IF  (X   .Count <>0)

THEN IF  (AC   .Sign = X   .Sign)

THEN Carry(AC,X)

ELSE Borrow(AC,X)

END  {Add};

{  }

PROCEDURE Subtract(VAR AC: Register; VAR X: Register);

BEGIN

X   .Sign := NOT X   .Sign;

Add(AC,X)

END {Subtract};

{  }

PROCEDURE Multiply (VAR AC: Register; VAR X: Register);

VAR  pM,pX,d: Digit;

M,Y : Register;

v,k,j,Over,Product: INTEGER;

NewSign: BOOLEAN;

{  }

PROCEDURE SetUp;

VAR  R: Register;

BEGIN

Y := AC;

NewRegister(AC);

NewRegister(M);

R := LargerOf( Y,X);

IF  (R = X)

THEN BEGIN

X := Y;

Y := R

END;

NewSign := (Y   .Sign = X   .Sign);

X   .Sign := TRUE;

Y   .Sign := TRUE;

Clear(AC)

END {SetUp};

{  }

PROCEDURE ShiftLeft(VAR Y: Register);

VAR  d: Digit;

BEGIN

NEW (d);

d   .Value := 0;

d   .Right := NIL;

d   .Left := Y   .Last;

Y   .Count := Y   .Count + 1;

Y   .Last   .Right := d;

Y   .Last := d

END {ShiftLeft};

{  }

BEGIN {Multiply}

SetUp;

IF  (X   .Count <> 0)

THEN BEGIN

pX := X   .Last;

FOR  j := 1 TO X   .Count DO BEGIN

Over := 0;

MakeCopy(M,Y);

pM := M   .Last;

v := pX   .Value;

FOR k := 1 TO M   .Count DO BEGIN

Product:=pM   .Value* v + Over;

pM   .Value := Product MOD 10;

Over := Product DIV 10;

pM := pM   .Left

END;

IF  (Over > 0)

THEN BEGIN

NEW (d);

d   .Right := pM   .Right;

d   .Left := pM;

pM   .Right   .Left := d;

pM   .Right := d;

M   .Count := M   .Count + 1;

d   .Value := Over

END;

Add(AC,M);

ShiftLeft(Y);

pX := pX   .Left

END;

AC   .Sign := NewSign;

Clear(Y);

Clear(M)

END

END  {Multiply};

{  }

PROCEDURE Divide(VAR AC: Register; VAR X: Register);

VAR  NewSign: BOOLEAN;

D,R,S : Register;

{  }

PROCEDURE Prolog;

BEGIN

NewSign := (AC   .Sign = X   .Sign);

AC   .Sign := TRUE;

X   .Sign := TRUE;

D := AC;

NewRegister(AC);

NewRegister(R);

NewRegister(S)

END {Prolog};

{  }

BEGIN                    {Divide}

IF  (X   .Count = 0)

THEN Clear(AC)

ELSE BEGIN

Prolog;

MakeCopy(S,X);

MakeCopy(AC,One);

R := LargerOf(D,S);

WHILE (R = D) DO BEGIN

Carry(AC,One);

Carry(S,X);

R := LargerOf(D,S)

END;

Borrow(AC,One);

AC   .Sign := NewSign;

Clear(D); Clear(S)

END

END  {Divide};

{  }

PROCEDURE Swap;

VAR  R: Register;

BEGIN

R := AC;

AC := X;

X := R

END  {Swap};

{**

**}

BEGIN  {UnBound}

Preface;

Done := FALSE;

Op := 'C';

REPEAT

ValueIn(X,NoX,NextOp);

IF  (NextOp = 'Q')

THEN Done = TRUE

ELSE BEGIN

IF  (NextOp = 'C')

THEN BEGIN

Clear(X); Clear(AC);

Op :='C'

END;

IF  NoX  THEN MakeCopy(X,AC);

CASE  Op  OF

'C' : Swap;    {AC with X}

'=' : Swap;

'+' : Add(AC,X);

'-' : Subtract(AC,X);

'*' : Multiply(AC,X);

'/' : Divide(AC,X)

END

END;

Op := NextOp;

Display(AC)

UNTIL Done

END  {UnBound}.

Go to Section E            Return to Table of Contents


Data Structures: SECTION E: THE STACK MACHINE

 Return to Table of Contents

SECTION E: THE STACK MACHINE

A Pascal program that provides command-shell access to a stack is a straightforward application of the procedures of Chapter 4, as long as some of the realities of Pascal as a working language are respected. In particular, care must be taken with the distinction between value and variable parameters in procedures.

Input and output from a terminal must deal with end-of-line codes, and the dual use of the screen (or print paper) can affect program details. Standard programming techniques suffice, applied within a token-entry procedure that can acquire values for the insert and bound commands.

The command repertoire of section 4.1.1 can be extended somewhat, but a complete set seems to be:

I insert

D delete the top value

T display the top value

B set a bound for the number of items on the stack

C clear (empty) the stack

E exit (quit)

For the sake of robustness, miskeying of input is ignored if possible. For the sake of simplicity, errors of deletion from an empty stack and insertion into a full one only invoke mild reproof. Simplicity also dictates the replacement of function TopValue with procedure Display that writes an "empty stack" message when necessary:

PROGRAM StackMachine(Input,Output);

TYPE  ValueType = 0. .99;

StackType = ARRAY [1. .99] OF INTEGER;

Var  Stack : StackType ;

Value : ValueType ;

Bound : ValueType ;

Top : ValueType ;

Op : CHAR;

{   }

PROCEDURE Preface;

BEGIN

Write('This program operates a stack with the');

Writeln(' single-letter commands:');

Writeln('I - insert a value into the stack');

Writeln('D - delete the top value from the stack');

Writeln('T - display the top value of the stack');

Writeln('B - set the stack bound between 1 and 99');

Writeln('C - clear (empty) the stack');

Writeln('E - exit (quit)'); Writeln;

Writeln('Commands should be separated by spaces.');

Writeln('B and I are to be followed by an integer');

Writeln(' in the range 0 to 99.') Writeln;

END {Preface};

{   }

PROCEDURE WhatNow(VAR Op : CHAR; VAR Value : ValueType);

BEGIN

Value := 0;

Read(Op);

WHILE NOT (Op IN ['I','D','T','B','C','E']) DO Read(Op);

IF (Op = 'I') THEN Readln(Value);

IF (Op = 'B') THEN Readln(Value)

END {WhatNow};

{   }

PROCEDURE Push(Value : ValueType);

BEGIN

IF  (Top >= Bound)

THEN Writeln('Stack full. Try another command.')

ELSE BEGIN

Top := Top + 1;

Stack[Top] := Value

END  {ELSE}

END  {Push};

{   }

FUNCTION Empty(Stack : StackType) : BOOLEAN;

BEGIN

IF  (Top = 0)

THEN Empty := TRUE

ELSE Empty := FALSE

END  {Empty};

{   }

PROCEDURE DeleteTop(Stack : StackType);

BEGIN

IF  Empty(Stack)

THEN Writeln('Stack empty. Try another command.')

ELSE Top := Top - 1

END  {DeleteTop};

{   }

PROCEDURE Display(Top : ValueType);

VAR k :  ValueType;

BEGIN

Writeln;

IF  Empty(Stack)

THEN Writeln('The stack is Empty.')

ELSE Writeln('The top value is' ,Stack[Top] : 3);

Writeln

END  {Display};

{**

**}

BEGIN  {StackMachine}

Preface;

Top := 0 ;

Bound := 10;

WhatNow(Op,Value);

WHILE  (Op < > 'E')  DO BEGIN

CASE  Op  OF

'I' : Push(Value);

'D' : DeleteTop(Stack);

'T' : Display(Top);

'B' : Bound := Value;

'C' : Top : = 0

END { CASE };

WhatNow( Op,Value);

END {WHILE}

END  {StackMachine}.

Go to Section F Return to Table of Contents


Data Structures: SECTION F: STACKS IN THE LANGUAGE FORTH

 Return to Table of Contents

SECTION F: STACKS IN THE LANGUAGE FORTH

FORTH modules are normally built "bottom up" by defining an operation, called a word, that acts on the system stack, then extending that word to a more general or more powerful one, and so on, until the entire program is, in a word, a word. FORTH is extendable, because the language can be enlarged by the use of defining statements that enter words into a list called the dictionary. The words entered into the dictionary are then recognized as operations from which more words may be built.

FORTH is a threaded language as it is usually implemented, meaning that execution proceeds from the call of a routine (the execution of a word), which consists of calls to other routines, which in turn call others, and so on. At the lowest level the calls request use of the primitive operations known as machine instructions.

FORTH is an interpretative language as it is usually implemented, translating and then executing each word as it is encountered. Interpretation is perhaps a more natural partner of stack-oriented languages than of others, because each word performs some operation on the stack, immediately. The interpretative scheme in turn combines readily with an interactive system.

A FORTH statement, and the resulting interaction, is:

3 4 +  7 OK

The user types 3 4 + . @ (@ means ''hit the RETURN key''), and FORTH responds 7 OK. (The underlining of the FORTH response is included for visual clarity in the text; it does not appear in actual output.) Table F. I shows the execution of this statement in terms of formal stack operations and SUE.

Table F.1

Token     Response                       Stack S

  3       INSERT(S,3)                V1 V2          Vt 3

  4       INSERT(S,4)                V1 V2          Vt 3 4

  +       begin

            Sum    TOP(S)

            DELETE(S)                V1 V2          Vt 3

            Sum    Sum + TOP(S)

            DELETE(S)                V1 V2          Vt

            INSERT(S,sum)            V1 V2          Vt 7

            end

          Write(TOP(S))

Similarly

2 3 4 + *  14 OK

has the effect on an initially empty stack that is shown in Figure F.1.

Figure F.1

In general, FORTH reads tokens (separated by blank spaces) left to right, stacks operands, and executes words. In this context, arithmetic operators like "*" and "+" are built-in dictionary words. The definition of a word, say neword, is added to the dictionary by simply executing a statement of the form:

: neword {details of what to do} ; OK

Here the OK response indicates that a new entry has been put into the dictionary.

Conversion of a length measured in inches to the corresponding length in centimeters can be defined by:

: intocm 2.54 * ; OK

This defines intocm to be the sequence of operations:

Push 2.54 onto S.

Multiply the top two values, delete them, and push the product onto S; that is, execute the word "*".

The word intocm may then be used as any other word:

8 intocm  2032 OK

If the top value of the stack is 5 (however it got there):

intocm  127 OK

Values may be stored in variables as well as on the stack. A variable identifier is bound to a memory cell (declared) when the connection between the two is established. For example

VARIABLE x OK

declares x as a variable, and x can then be assigned the value 10 by:

10 x ! OK

(The command "!'' is read ''store".) Retrieval of a variable value (to the stack) is done with "@", read "fetch". With the assignment above:

x @  10 OK

x @ intocm  254 OK

A redefinition of neword does not delete the old definition. When the word neword is encountered in the scan of a statement being interpreted, the latest definition is retrieved. It is possible to back up one definition of neword by execution of:

FORGE T neword OK

All dictionary entries made since neword are lost. In effect, the dictionary itself is treated as a stack with respect to the occurrences of a given word within it, as shown in Figure F.2. It is convenient to assume that a word, ''.S'', is available to display the stack contents, nondestructively.

Figure F.2

If the stack contains 5 7 2, bottom to top, then:

S 5 7 2 OK

A quirk (or astute design choice, if you like) is that access to the top of the FORTH stack is not just a TOP(S) operation but is essentially the operation pair [TOP(S), DELETE(S)], a pop. The use of the top value deletes it. Hence, to square the top value, for example, it is necessary first to duplicate it. A built-in word "DUP" is provided for this purpose. For example, if the stack contains: 5 7 - 2 then:

S     5  7  -2  OK

DUP  S 5  7    -2  -2  OK

DUP  *  S  5  7   -2  4

In terms of formal stack operations, DUP is INSERT(S,TOP(S)).

A number of stack manipulation words are contained in the basic dictionary of a FORTH compiler. The action of some of them is displayed here; their expression in terms of formal stack operations is left to the exercises and problems:

S 5 7 2 OK

/MOD S 5 1 3 OK                       {remainder then quotient, of top

{value division of next-to-top

S  5 7 2 OK

MOD S 5 1 OK                          {remainder of top value division

{of next-to-top

S   5 7 2 OK

SWAP S 5 2 7 OK

S  5 7 2 OK

DUP  S 5 7 2 2

S 5  7 2 OK

OVER S 5 7 2 7 OK                     {copy next-to-top and push it

S 5 7 2 3 OK

ROT S 5 2 3 7 OK                      {cyclic rotation of the

{top 3 values

S 5  7 2 OK

DROP S 5 7 OK

S  1  5 7 2 OK

2SWAP S 7 2 1 5 OK                    {swaps top pairs

S 5  7 2 OK

2DUP S 5 7 2 7 2 OK                   {duplicates top pair

S  5  7 2 3 4 OK

20VER S 5 7 2 3 4 7 2 OK              {copies second pair

{and pushes it

S  5  7 2 OK

2DROP S 5 OK                          {discards the top pair

Exercise EF.1 and problem PF.1 are appropriate at this point.

F.1 FORTH Control Structures

The discussion of FORTH control structures in this section is very brief, but excellent treatments of FORTH are available for the reader interested in pursuing this topic further, including [BRODIE, 1981] and [BYTE, 1980].

The heart of any procedural language is the set of control structures it provides. It is necessary to have conditional branch statements in a language, and it is very awkward to do without explicit loops. Some peculiarities of control statements in FORTH are created by the way in which words are dealt with sequentially, in conjunction with an operation on the stack.

Everything in FORTH is postfix and stack-related, but the necessary structures are provided. For example

: Checkten 10 = IF Powwow THEN; OK

adds a word to the dictionary, which--when executed--plays the same role in program logic as the SUE statement

if TOP(S) = 10

then Powwow

endif

but it differs because it also manipulates the stack. A partial trace of its action is shown in Table F.2.

Table F.2

token        operation

 10          INSERT(S, 10)

  =          begin

               t1    TOP(S)

               DELETE(S)

               t2    TOP(S)

               DELETE(S)

               if t1 = t2

                   then INSERT(S,1)     {any value but 0 is

                   else INSERT(S,0)     {considered to be TRUE

                   endif

                 end

  IF           bool    TOP(S)

               DELETE(S)

               {The decision to execute or not to execute Powwow

               {is then, in effect,

               {if bool then Powwow endif

Similarly

: Spliten 10 = IF Powwow ELSE Wait THEN ; OK

adds a word to the dictionary, which--when executed--plays the role of the SUE statement:

if TOP(S) = 10

then Powwow

else Wait

endif

Problem PF.2 and program PGF.1 are appropriate at this point.

Loop structures in FORTH are only slightly more complex.

A DO-loop is specified in FORTH by:

: DEF {limit} {index} DO {what you will} LOOP;

One such loop definition is:

:echo 5 to 1 DO . "shout " LOOP ; OK

When executed:

echo shout shout shout shout shout OK

The index of the DO-loop is accessed within the loop by the FORTH word "I". With the help of the FORTH word "CR" that prints a carriage return:

: echo2 3 to 1 DO CR . "shout " I LOOP ; OK

echo2

shout 1

shout 2

shout 3 OK

Exercise EF.2 and Problem PF.3 are appropriate at this point.

Additional features of FORTH are described in [BRODIE, 1981] and [BYTE, 1980],

Exercises

Exercises immediate applications of the text material

EF.1     Write procedures for stack operations to emulate the following FORTH words:

MOD, /MOD, SWAP, DUP, DROP, OVER, 2DROP

(PF. 1 asks for others.)

EF.2     What is the output of the following loop:

:mystery 10 1 DO I * LOOP;

Problems

Problems not immediate, but requiring no major extensions of the text

PF.1     Write procedures, in terms of stack operations, that emulate the following FORTH words:

ROT , 2SWAP , 2DUP , 2OVER

PF.2     Write a procedure for stack operations, to emulate the if . . . then . . . else construct in FORTH.

PF.3     Choose one of the example problems EP2. 1-EP2.5 of Chapter 2.1. Write a sequence of variable definitions and FORTH dictionary words with the property that the final word in the sequence solves the problem.

Programs

Programs for trying it yourself

PGF.1     Write a program that manages a stack on demand from an input file with FORTH commands. ".", ".S", and the commands of EF. 1 are a minimum viable set. If convenient, a special symbol may be used in place of linefeed. The "FORTH" response should be identical to that of the FORTH interpreter discussed in this section.

Go to Section G            Return to Table of Contents


Data Structures: SECTION G: THE P-MACHINE

 Return to Table of Contents

SECTION G: THE P-MACHINE

The information required to execute p-code instructions is contained in Figure G.1.

Figure G.1

A p-code program is created and loaded into the program store PS by a p-code compiler (and loader, if that operation is separate). The PS is treated as a ROM (Read Only Memory) by the p-Machine. That is, no program ever alters it, but its contents are accessible. At the initiation of program execution, the program address register P is set to the address of the first (p-code) instruction in PS; and the machine automatically cycles through the following loop until the execution of an instruction (HALT or STOP, in effect) shuts off execution:

repeat {Execution cycle

1. Fetch the instruction at the address in P, and place it in I. (It will have a PS address.)

2. Increment P to point to the presumed next instruction.

3. Execute the instruction, i, in I.

if i is HALT then exit.

if i is an unconditional branch, or a conditional branch with a TRUE condition, then update P to the branch address.

forever

The instruction set is chosen with regard to the properties of the hardware that can be built to carry out the cycle, but there are many options. Because the p-Machine is a general-purpose computer, the chosen instruction-set embodies several standard functions:

1. I/0 (Input/Output)

2. Copy, to transfer data from one location to another

3. Arithmetic and logic

4. Branch

5. Conditional Branch

These functions help to define the instruction set, and so does the way instructions interact with the stack S. The stack interactions fundamentally determine the ability of p-code to support language features that supplement the actions listed above. The most crucial of such features are those associated with procedures. Procedures affect the p-Machine instructions to such an extent that they need to be discussed before the instruction set itself.

A data set is pushed onto S and popped from S in two ways--in many-valued segments and as individual values--in order to manage the data relevant to the execution of procedures. Such data can vary from one invocation to another of the same procedure. Data, and information about the location of data, can be passed to invoked procedures in a number of ways.

From the view of p-code, it is only necessary to know that a data set local to a particular invocation of a procedure is to be found in the data segment that represents that invocation. A value in a procedure invocation M is located by using the base address (which is in the base register B when M is being executed) and an offset, A, within the segment, that is specific to the variable. This process is illustrated in Figure G.2.

Figure G.2

All data are local to one, and only one, procedure invocation. Consider, for example, the nested procedures below:

procedure Alpha

declare aVar

procedure Beta

declare b

procedure Gamma

declare c

.

.

.

aVar    3 * b

.

.

.

end {Gamma

end {Beta

end {Alpha

During execution of procedure Gamma, the stack S might appear as shown in Figure G.3.

Figure G.3

The dashed arrows imply that the segment to which the variable aVar belongs can be traced back through a sequence of base addresses--a linked list, in effect. Then aVar is located within its procedure invocation (of Alpha in this case). Clearly, some linking mechanism needs to be provided within the instruction format and the segments as they are stacked. The mechanism provided will even allow procedures to call themselves, directly or indirectly. For example, Gamma may refer to Gamma (directly) or to Beta (which may invoke Gamma again).

Because of the need for nested invocations of procedures, two links are provided in each segment. The segment scheme is shown in Figure G.4.

Figure G.4

The static link points to the data segment of the immediately containing procedure (in the source code block structure). It is used to resolve variable references. In the example, the static link in an invocation of Gamma points to the base of an invocation of Beta.

The dynamic link is the address of the data segment of the calling (invoking) procedure. It is used to pop the segment off the stack at return time. In the most straightforward use of procedures, the dynamic link has the same value as the static link.

To determine the static link, the compiler keeps track of a level of declarations in the source code. Essentially this is the nesting level of the defining procedure. In the example, variables aVar, b, and c might be at levels 1, 2, and 3, respectively. The difference in levels between a variable and a segment that refers to it is precisely the number of times that the static link must be used to reach the segment in which the variable is declared. For example, to find the variable aVar at level 1 from procedure Gamma at level 3, the static links must be followed twice: to Beta, then to Alpha. Consequently, the address of a value in S consists of a level difference and an offset within a segment.

The behavior of the p-Machine does not entirely determine the syntax of p-code statements, but it certainly limits the options. In a simple, suitable format instructions contain three fields:

INST    instruction code        {opcode

L    level difference

A    displacement        {offset

An instruction is written: INST L,A

The implication is that the pair (L,A) determines an address; but, for some instructions, the pair of fields has other uses. The syntax remains the same; the semantics varies.

A workable instruction set is the following:

LIT  0,A  {Load the constant (literal) A onto the

{top of the stack S: S[T]    A

OPR  0,A  {Execute operation A. The possible

{operations include I/O and arithmetic.

LOD  L,A  {Push the value at address (L,A) onto

{S. The address (L,A) is level difference

{L, offset A.

STO  L,A  {Pop the value of S and store it in (L,A).

CAL  L,A  {Invoke a procedure, with first executable

{statement at A in PS; start a new segment

{with dynamic link set to the old B value,

{and static link determined from L.

INT  0,A  {Increment T by A: T    T + A

JMP  0,A  {Jump (transfer control) unconditionally

{to the instruction at A in PS.

JPC  0,A  {If the top stack value, S[T], is zero,

{then jump to A in PS; otherwise, don't.

This instruction set uses the stack S for transfers of control and data in sophisticated ways, but computation is confined to rather simple stack manipulations. For example, for operation code A = 2, execution of OPR is:

begin {addition

T    T - 1

S[T]    S[T] + S[T+1]

end

For A = 16, the execution is:

begin {input

T    T + 1

Readln(S[T])

end

The OPR statement has seventeen variations:

OPR  0,0   {the last line of a program

OPR  0,1   {unary minus

OPR  0,2   {addition

OPR  0,3   {subtraction

OPR  0,4   {multiplication

OPR  0,5   {integer division

OPR  0,6   {logical odd function--if S[T] is

{odd, then replace it with TRUE(1),

{else with FALSE(0)

OPR  0,7   {no operator--used to debug programs

OPR  0,8   {test for S[T--1] = S[T] and replace the

{pair with TRUE(1) or FALSE(0)

OPR  0,9   {test for

OPR  0,10  {test for < i.e. S[T-1] < S[T]

OPR  0,11  {test for

OPR  0,12  {testfor >

OPR  0,13  {test for

OPR  0,14  {print S[T]

OPR  0,15  {carriage return

OPR  0,16  {read into S[T]

Exercise EG.1 and problems PG.1 and PG.2 are appropriate at this point.

It may surprise you to discover how little of the complexity of this computing machine is generated by reading, writing, and arithmetic and how much by the flow of information and the flow of control. That is, however, true to the nature of computing.

The definitive exposition of p-code and the p-Machine is by the inventor [WIRTH, 1974], who includes a compiler of (limited) Pascal, PL/0, into p-code. The p-code concept has been extended to form the basis of compilers of full Pascal, notably at the University of California at San Diego. A further extension has brought forth machines that execute p-code directly in hardware.

Both FORTH (see section F) and the p-Machine are founded on stack management, but they are quite distinct. For example, only a few of the FORTH words come into play in p-code, and they are the familiar operations of arithmetic and logic. FORTH does not need a program store. Commands are interpreted and executed immediately. FORTH makes its stack the center of attention: the p-Machine uses its stack as a scaffold hidden behind the scenes of the high-level languages it supports.

Exercises

Problems

Exercises

Exercises immediate applications of the text material

EG.1     Convert the SUE segments that follow into p-code:

i    1;                     Read(a,b)

while i    10 do             if a > b

j    1                      then a    a + b

while j    3 do                   Write(a,b)

Write(i,j)               else a    a - b

j    j + 1                     Write(a,b)

endwhile                 endif

i    i + 1

endwhile

i    1                      i    1

while i    10 do             repeat

Write(i)                   Write(i)

i    i + 1                 i    1

endwhile                   until i > 10

i    1                      Read(a)

j    2                      if a = 1

k    4                        then b    10

while i    k do                     Write(a,b)

if j < k                   endif

then j    j + 1        if a = 2

Write(i,j,k)        then b    20

else k    k - 1               Write(a,b)

Write(i,j,k)        endif

endif                  if a = 3

i    i - 1                  then b    20

endwhile                            Write(a,b)

Write(i,j,k)                  endif

Problems

Problems not immediate, but requiring no major extensions of the text

PG.1     Write a p-code program that will input 10 numbers and print their sum.

PG.2     Choose one of the example problems EP2.1-EP2.5 of section 2.1 and solve it with a p-code program.

Go to Section H            Return to Table of Contents


Data Structures: SECTION H: FOLLOWING A MAZE

 Return to Table of Contents

SECTION H: FOLLOWING A MAZE

A maze of passages is (topologically) equivalent to a rectangular array of block positions, some occupied and some forming passages because they are not occupied. A very simple example is shown in Figure H.1 .

Figure H.1

Such an array of blocks corresponds to an array of values. For example,

M[I,J] = 1

may correspond to an open (unoccupied) block and M[I,J] = 0 to a closed (occupied) block. The maze of Figure H.1 is then represented by:

M  1  2  3  4  5  6  7  8

1  0  0  0  1  0  0  0  0

2  0  1  1  1  0  0  1  0

3  0  1  0  1  0  0  1  0

4  0  1  0  1  1  1  1  0

5  0  1  0  0  0  0  1  1

6  0  1  1  0  1  1  1  0

7  0  0  0  0  1  0  0  0

8  0  0  0  0  1  0  0  0

While searching such a table for a passageway from an entrance position (R,C) to an exit position, it is convenient to mark positions (I,J) already visited by the assignment M[I,J] [image] 2. An algorithm that searches the table form of the maze for passageways from an entrance to another exit can be designed around a stack of positions:

Push (R,C) and set M[R,C] to 2. The pair (R,C) is the current position.

repeat

1. Test the four positions adjacent to (R,C) (on the North, East, South, and West). For a position (I,J) for which M[I,J] = 1, push (I,J) and set M[I,J] to 2. If no adjacent position is open, then delete (I,J) and set M[R,C] to 0 to indicate that the current position leads nowhere.

2. (R,C) [image] TopValue

until (R,C) is a border position.

If the final (R,C) is the initial entrance pair, then the maze has no other exit because all branch-paths from all blocks accessible from the start block have been tried. The contents of the stack define a path from entrance to exit. If adjacent positions are tested in clockwise order, a trace of the first few loop iterations for the example maze are:

(R,C)  STACK

(1,4)  (1,4)

(1,4)  (1,4) (2,4)

(2,4)  (1,4) (2,4) (3,4) (2,3)

(2,3)  (1,4) (2,4) (3,4) (2,3) (2,2)

(2,2)  (1,4) (2.4) (3,4) (2,3) (2,2) (3,2)

(3,2)  (l,4) (2,4) (3,4) (2,3) (2,2) (3,2) (4,2)

(4,2)  (l,4) (2,4) (3,4) (2,3) (2,2) (3,2) (4,2) (5,2)

(5,2)  (1,4) (2,4) (3,4) (2,3) (2,2) (3,2) (4,2) (5,2) (6,2)

(4,2)  (1,4) (2,4) (3,4) (2,3) (2,2) (3,2) (4,2) (5,2) (6,2) (6,3)

(4,2)  (1,4) (2,4) (3,4) (2,3) (2,2) (3,2) (4,2) (5,2) (6,2)

  .

  .

  .

Exercise EH.1 is appropriate at this point.

From this point, the algorithm will backtrack, erasing false passages as it goes.

An appropriate data structure for the stack is an array of records, each containing a pair of indices. The entrance pair, for instance, may be entered as integers I0 and J0 and placed in a position record Start by: Start.Row [image] I0, Start.Column [image] J0.

The remaining technical problem is the search of positions adjacent to the current one. The positions adjacent to pair (R,C) are:

(I - 1,J), (I,J + 1), (I + 1,J), and (I,J - 1)

These can be derived by adding pairs (-1,0), (0,1), (1,0), and (0,-1) to (R,C). For convenience in looping through the four possibilities, they can be stored in an array of pairs, shown in Figure H.2.

Figure H.2

An implementation of this scheme requires that (I,J) + Delta(k) fall within the table for any (I,J) in the maze. Consequently, extra rows of zeros should be provided at each edge of the maze table (in rows 0 and RowMax + 1, columns 0 and ColumnMax + 1).

The algorithm, configured as a procedure to which the maze-table and entrance data are passed is:

procedure Mouse(Start,M,RowMax,ColumnMax)

Push(Start)

R    Start.Row

C    Start.Column

M[R,C]    2

repeat

DeadEnd    TRUE

for k    1 to 4 do

I    R + Delta[k].Row

J    C + Delta[k].Column

if M[I,J] = 1

then Test.Row    I

Test.Column    J

Push(Test)

M[I,J]    2

DeadEnd    FALSE

endif

next k

if DeadEnd

then Delete

M[I,J]    0

endif

Position    TopValue

R    Position.Row

C    Position.Column

if       R = 0 OR R = RowMax

OR   C = 0 OR C = ColumnMax

then Done    TRUE

else Done    FALSE

endif

until Done

end {Mouse

Exercises

Exercises immediate applications of the text material

EH.1 Complete the trace of the example maze.

Problems

Problems not immediate, but requiring no major extensions of the text

PH.1 Mouse can maintain a second stack of branch points--blocks with two or more adjacent open blocks. This stack can be used to find all the paths from entrance to other exits. How?

Programs

Programs for trying it yourself

PGH.1 Implement Mouse within a program.

Go to Section I Return to Table of Contents


Data Structures: SECTION I: THE QUEUE MACHINE

 Return to Table of Contents

SECTION I: THE QUEUE MACHINE

A Pascal program that provides command-shell access to a queue is a straightforward application of the procedures of Chapter 5, as long as some of the realities of Pascal as a working language are respected. In particular, care must be taken with the distinction between value and variable parameters in procedures.

Input and output from a terminal must deal with end-of-line codes, and the dual use of the screen (or print paper) can affect program details. Standard programming techniques suffice, applied with a token-entry procedure that can acquire values for the insert and bound commands.

The command repertoire for a queue machine is:

I insert

D delete the front value

F display the front value

B set a bound for the number of items on the queue

C clear (empty) the queue

E exit (quit)

For the sake of robustness, miskeying of input is ignored if possible. For the sake of simplicity, errors of deletion from an empty queue and insertion into a full one only invoke mild reproof. Simplicity also dictates the replacement of function Front with procedure Display that writes an "empty queue" message when necessary.

PROGRAM QueueMachine(Input,Output);

TYPE ValueType = 0..99;

QueueType = ARRAY [1..99] OF INTEGER;

VAR ValueIn  :  ValueType;

Queue  :  QueueType;

Bound  :  ValueType;

Front  :  ValueType;

Rear  :  ValueType;

Op :  CHAR;

{  }

PROCEDURE Preface;

BEGIN

Write('This program operates a queue with the');

Writeln('single-letter commands:');

Writeln('I - insert a value into the queue');

Writeln('D - delete the front value from the queue');

Writeln('F - display the front value of the queue');

Writeln('B - set the queue bound between 1 and 99');

Writeln('C - clear (empty) the queue');

Writeln('E - exit (quit)'); Writeln;

Writeln('Commands should be separated by spaces. B and I');

Write('are to be followed by an integer in the range 0');

Writeln(' to 99.'); Writeln;

END {Preface};

{  }

PROCEDURE WhatNow(VAR Op : CHAR; VAR ValueIn : ValueType);

BEGIN

ValueIn := 0;

Read(Op);

WHILE NOT (Op IN ['I','D','F','B','C','E']) DO

Read (Op);

IF  (Op = 'I') THEN Readln(ValueIn);

IF  (Op = 'B') THEN Readln(ValueIn)

END {WhatNow};

{  }

PROCEDURE InsertRear(ValueIn : ValueType);

{  }

PROCEDURE ShiftQ(Queue : QueueType);

VAR k : INTEGER;

BEGIN

IF  (Front > 0)

THEN BEGIN

FOR k := 1 TO (Rear - Front) DO

Queue[k] := Queue[Front+k];

Rear := Rear - Front;

Front := 0

END

END   {ShiftQ};

{  }

BEGIN

IF  (Rear >= Bound)

THEN ShiftQ(Queue)

END;

IF  (Rear >= Bound)

THEN Writeln('The Queue is full. Try another command.')

ELSE BEGIN

Rear := Rear + 1;

Queue[Rear] := ValueIn

END

END  {InsertRear};

{  }

FUNCTION Empty(Queue : QueueType) : BOOLEAN;

BEGIN

IF (Rear = 0)

THEN Empty := TRUE

ELSE Empty := FALSE

END {Empty};

{ }

PROCEDURE DeleteFront(Queue : QueueType);

BEGIN

IF Empty(Queue)

THEN Writeln('The queue is empty. Try another command.')

ELSE BEGIN

Front := Front + 1;

IF (Front = Rear)

THEN BEGIN

Front := 0;

Rear := 0

END

END

END {DeleteFront};

{ }

PROCEDURE Display(Front : ValueType);

VAR k : ValueType;

BEGIN

Writeln;

IF  Empty(Queue)

THEN Writeln('The queue is empty.')

ELSE Writeln('The front value is',Queue[Front+1] : 3);

Writeln

END  {Display};

{**

**}

BEGIN {QueueMachine}

Preface;

Front : = 0;

Rear  : = 0;

Bound : = 10;

WhatNow(Op,ValueIn);

WHILE (Op <> 'E') DO BEGIN

CASE Op OF

'I' : InsertRear(ValueIn);

'D' : DeleteFront(Queue);

'F' : Display(Front);

'B' : Bound := ValueIn;

'C' : BEGIN

Front := 0;

Rear := 0

END

END {CASE};

WhatNow(Op,ValueIn)

END {WHILE}

END {StackMachine}.

Go to Section J            Return to Table of Contents


Data Structures: SECTION J: QUEUES IN HARDWARE

 Return to Table of Contents

SECTION J: QUEUES IN HARDWARE

Some microprocessors, such as the Intel 8086, have an instruction queue built into the CPU [INTEL, 1979]. The motivation is that the CPU execution cycle contains essentially four phases: execution, fetch, read, and write. Here read, write, and fetch refer to sending and receiving information from the CPU itself. In particular, write refers to the sending of information to some address that may be associated with a display, but may just be an update of a memory location. The effect of the write depends on the instruction that commands it, and the address. The read phase involves acquiring data from some address, which may be in memory or may be associated with a terminal. Instructions may or may not read and write in this general sense when they are executed, depending upon their function. The term fetch specifically refers to acquiring an instruction from memory. The crucial fact is that the CPU is otherwise idle during the fetch operation, unless it is possible to overlap the phases. With overlap, some of the phases can be done simultaneously by the hardware. The instruction queue that makes this possible is a set of 6 registers, which are storage locations, like memory cells, except that they are in the CPU itself, not in memory. (Register is the more general term; memory cells are registers used in a particular way.) The scheme is a queue, as illustrated in Figure J.1.

The IQ (Instruction Queue) is used to store the next few instructions that are expected to be executed in order. Execution of a branch operation (conditional or not) will cause the next instruction to be taken out of sequence, and so the next instruction may need to be fetched immediately and the instruction queue reloaded from the new point. Without concern for branches, however, fetches into the IQ may take place during execution of the current instruction.

A schematic trace of the overlap can be given using exec, wr, f, and rd as abbreviations for execute, write, fetch, and read, respectively. A simplified example of instructions 1,2,3,4, and 5 during eight units of time might look like Figure J.2.

Figure J.1

Figure J.2

The amount of time saved by the IQ mechanism depends upon a variety of factors:

The speed at which instructions can be moved from the IQ to the decoder, relative to the time required for a fetch from memory. It might be faster by a factor of 1000; and, if so, the IQ-to-decoder move is negligible.

The amount of time spent in the fetch phase, compared to time spent during execute, read, and write phases. A rough estimate is that fetches can use 10 - 40 percent of the CPU time if there is no overlap.

The frequency of branch operations. A rough estimate is that 1 instruction in 16 is a branch.

The actual time saved in execution depends so heavily on the particulars of the instruction stream that it needs to be determined or at least verified experimentally. A speedup of 15 percent or so due to the IQ would be a modest expectation.

Program PGJ.1 is appropriate at this point.

The use of an instruction queue is a simple form of a more extensive overlap of instruction execution called pipelining. Execution of an instruction in a processor is actually the execution of a sequence of more primitive operations, called microcode. Microcode is designed to be neatly done with electronic circuitry. The instruction decoder, in effect, invokes a procedure in which the statements are microcode operations. Such procedures may include subprocedures and calls on standard (microcode) utility subprocedures, and so on, just like any computer coding. (Some people program in microcode for a living. It may be a relief to know that the buck stops there--the substrate that supports microcode is engineering, not programming.) The meaning of an instruction is determined by a microcode program. Some machines are even designed to allow users to redesign the instruction set by re(micro)programming them.

The circuits controlled by microcode instructions do not have to be idle while other circuits are at work. Pipelining allows parts of several instructions to be moving through the execution system at the same time. The ultimate set of divisions of an instruction that permits pipelining is not (execute, read, write, and fetch), but the set of available microcode operations.

The fetch of instructions from the program store is an execution bottleneck, and so is the fetch of data, which can be relieved with the use of a cache memory. (Data, from the viewpoint of the CPU, include instructions--whatever comes from memory.) A cache memory is memory that can be accessed much more rapidly than the main store because of the way it is constructed. It costs more, else it would simply replace the main store, and so it is relatively small.

Effective use of a buffer, such as a cache memory, is made when it is full of precisely those values that will be needed next. This can be done with high success in several ways, some involving queueing.

Typically, a cache memory ranges from 1K (1024 bytes) to 64K and is 5 to 10 times as fast as main memory, having perhaps 100 nsec (nanoseconds, 10-9 seconds) for an access time. Circuitry is designed into the machine to keep track of what is in the cache; it is a "cache directory," such as Figure J.3 shows.

Figure J.3

When a memory-reference is made, if the information is in the cache, the cache is used for the read or store. An 85-95 percent "hit rate" is typical, but it depends on some strategy for keeping those data most likely to be used soon in the cache. The hit rate depends upon how local the memory references tend to be.

One way to enhance the hit rate involves dividing the cache into areas that will be called segments here. Data are fetched from the main store in chunks of segment size, and the collection of segments is treated as a queue (or perhaps a priority queue). A directory of segments that are currently in the cache is maintained. Whenever a segment is accessed, whether or not it is already in the cache, it can be moved to the tail of the queue. The head of the queue is then a segment that has been idle longer than any other segment and may be bumped by addition of a new segment.

A generalization of a cache queue is a cache priority queue, which allows the use of a variety of strategies for determining which segments to retain in the cache and which to swap for a currently needed value.

Programs

Programs

Programs for trying it yourself

PGJ.1 Write a program that explores the effect of an IQ. Instructions are to be chosen at random, with:

100 percent requiring an execute phase and a fetch phase, of 6 and 10 units of time, respectively

25 percent requiring an additional read phase of 8 units of time

50 percent requiring an additional write phase of 8 units of time

20 percent are branch operations that empty an IQ if it is being used

Determine the time required for each of five runs of 40 instructions each, both with and without an IQ of six registers. Run again with the times for execute and fetch switched to 10 and 6, respectively.

PGJ.2 Write a program that explores the effect of a cache memory. A sequence of memory references are chosen to randomly call on segments in the cache memory 85 percent of the time. Two units of time are required for retrieval from the cache and 10 for direct retrieval from memory (which brings a new segment to the cache). The tail segment is twice as likely as the next to be hit; it in turn is twice as likely as the next, and so on. The overhead cost of moving a segment to the tail is one unit of time. There are six segments in the cache. Choose five sequences of 40 memory requests each, and determine the total retrieval time needed for them.

PGJ.3 Combine PGJ.1 and PGJ.2 to determine the effect of an IQ and a cache acting in concert.

Go to Section K Return to Table of Contents


Data Structures: SECTION K: A SIMULATION OF AN AIRPORT CUSTOMS STATION

 Return to Table of Contents

SECTION K: A SIMULATION OF AN AIRPORT CUSTOMS STATION

A simulation, as the term is used here, is a program that imitates the behavior of a system well enough to be used as an experimental surrogate for the system itself.

Pursuit of the terms "well enough" and "experimental" in the definition given lead to a vast but amorphous literature and practice. The practice is an art of importance that increases with the power of computing, yet the literature contains no decisive entry point. The intent of this section is to provide a program that is a framework for the study of a simple system and hence a window onto one part of the practice of simulation.

A professionally developed simulation program is designed for a real system on the basis of carefully collected and analyzed data. The worth of a thoughtfully determined goal is balanced against the (sometimes considerable) cost of writing and using it. The assumptions built into the system as a program model are checked for validity against the real world, and the behavior of the program is verified to be that of the assumed system. Many systems that are simulated involve random behavior, imitated by a program model, and so experiments are run on the simulation program. These experiments and the analysis of their results are both developed with statistical care.

This section is only intended to provide an example of the possibilities and a minimal framework for experiment. It is only the program portion of a simulation process that appears here.

K.1 The Customs Station

An airport customs station receives arriving passengers at a check-in counter. Passengers wait until one of several clerks is available to serve them. They are served in a time that has an (essentially) normal distribution about a mean. (The distribution cannot actually be normal, since negative service times are not possible, and they are discarded when drawn from it. Such a distribution is called truncated normal.) If the passengers have prepaid their customs tax, they depart the counter. If they have not, they are sent to a tax station, where they wait in a tax queue until served by one of several tax agents. They are served at the tax counter in time that has an exponential distribution. On exit from the tax counter, a passenger rejoins the customs-clerk queue but, when reaching a clerk a second time, is serviced in a constant time. A schematic of this system is shown in Figure K.1.

Figure K.1

This system looks deceptively simple, but the events that occur in it are several:

Arrival: A passenger arrives at the station from outside the station "universe." If there is a free clerk, the passenger is scheduled for check-in immediately; otherwise, the passenger waits in the clerk queue.

Check-in: A passenger is simply scheduled for checkout at a time that is "now" plus a check-out service time. This becomes the DoTime of the passenger--the time at which his or her event is to be done.

Check-out: A passenger completes service by a clerk and either departs the station or is sent to the tax counter. If there is a free tax agent, the passenger is scheduled for Tax-in immediately and otherwise waits in the tax queue.

Tax-in: A passenger is scheduled for Tax-out, at a DoTime value determined by "now" plus the tax service-time distribution.

Tax-out: A passenger completes service by a tax agent and, if there is an available clerk, is sent to the clerk counter again, to Check-in, in effect. If no clerk is available, the passenger waits in the clerk queue.

These are processes, represented by procedures in a simulation program written in a language such as Pascal. These processes occur at times determined by random distributions and so occur in a random sequence. At any time, the next process to be executed may be any one of them; and several can occur at the same time. A procedural simulation program moves from event to event, each a process acting on a passenger. The program must be able to schedule the next event and handle events one at a time. Simulation time is tracked by a "clock" that is advanced to the scheduled time of the current event, the DoTime of the customer being processed. The clock is not advanced when a completed event is followed by a "simultaneous" event, scheduled at the current clock time.

A common simulation technique is to have one event in a distribution stream (such as the arrival stream) schedule the next one. The process that manages the arrival of passenger I fetches the next random interarrival time from the arrival stream, say ia. If the system clock is Clock, then the arrival of the next passenger, say J, occurs at time: Clock + ia. The arrival of the passenger after J is determined at the time of arrival of J, when process Arrival acts on J. Scheduling of processes can be done by a generalization of this kind of scheduling of events in a distribution stream, as indicated by the process descriptions.

Passengers spend most of their time outside of processes. A passenger who has just been checked in at one time will be processed again at his (or her) scheduled check-out time. A number of processes may be active between these times, and several events may occur at the check-out time. The program representation of a passenger (normally a record) waits somewhere when not being processed. This somewhere is an event queue--a priority queue of passenger records not currently in process. The priority involved is the time of the next event that is to happen to the passenger. Hence a passenger (record) is scheduled for one event after another until it departs the system, and it waits its turn on the event queue. The priorities may be determined by simply keeping the records of the event queue sorted by the DoTime (next-event time) of the passenger records in it.

The scheme outlined above for managing processes (events) and entities (passengers) advances the simulation clock from one scheduled event to the next in an unpredictable but very organized manner. (The next event being the DoTime of the record at the front of the priority queue.) This is called discrete-event simulation, although it applies strictly only to the program portion of a simulation. It is the technique chosen for the simulation program Airport developed for the airport customs station in the remainder of this section.

K.2 The Event-Cycle Level of Airport

The program level that corresponds to a command-shell in the management of a simple discrete-event simulation is an event-cycle level. At this level, control passes from one process to another, driven by the scheduling of events for entities (passenger records in this case).

The number of passenger entities in the station at any (simulation) time is ultimately determined by the interaction of four distributions: arrival, check-out service, tax service, and the uniform distribution used to decide if a passenger has prepaid the customs tax. The interaction of these is complex for the system as a whole, but for an individual passenger it is relatively simple: a passenger progresses in stages. The stages are each associated with a particular procedure:

Stage 1. Birth: The passenger arrives, an event managed by procedure Arrival. Arrival marks the record of passenger P as stage 2 so that its next stage will be recognized by any process that handles it. Arrival assigns P to an idle clerk if there is one, and schedules CheckIn at the current time. "Scheduled" means placement in the sorted list EventQueue. If no clerk is available, P is placed in a waiting line, ClerkQueue, and no event is scheduled by Arrival for P. Arrival also schedules the next arrival, drawing from the interarrival distribution to do so.

Stage 2. Initial check-in: An idle clerk is put to work on passenger P by procedure Check-in. In effect, P is scheduled for check-out at the current time Clock plus an increment drawn from a truncated normal distribution that represents clerk service times.

Stage 3. Initial check-out: The initial check-out service by a clerk is complete. The managing procedure, CheckOut, draws a uniform variate to decide whether to send P to Depart (where the passenger record will die--leave the system) or to the tax counter. If P is to go to the tax counter, the record is marked stage 4 and scheduled for processing by TaxIn at the current time Clock if there is a tax agent available. If no tax agent is available, P is placed in the waiting line TaxQueue. Since check-out frees a clerk, if ClerkQueue is not empty a passenger is removed from it with the queueing discipline FIFO (first-in-first-out): it is a pure queue. CheckOut does not make the decision to process this newly freed passenger, however, but simply schedules the processing.

Stage 4. Tax-counter check-in: Procedure TaxIn draws a service time for P from a (truncated) exponential distribution and schedules completion of tax service for P at that increment plus Clock. The correct routing of P is insured by marking the record stage 5.

Stage 5. Tax-counter check-out: Procedure TaxOut schedules P for CheckIn at the current Clock if there are idle clerks; otherwise, P is placed in ClerkQueue. Routing is insured by marking P as stage 6. Since TaxOut frees a tax agent, if TaxQueue is not empty, a passenger is retrieved from it--with the FIFO queueing discipline--and scheduled.

Stage 6. After-tax check-in: This is managed by CheckIn and differs from stage 2 in that a constant increment PaidTime is added to Clock to form the DoTime of the check-out event for P. Stage 7 marking routes P back to CheckOut.

Stage 7. After-tax check-out: This is managed by CheckOut and leads automatically to death--departure from the system.

Passengers are placed in EventQueue by procedure Schedule and removed from it by procedure NextEvent. Procedure NextEvent deals with a subtlety that is typical of simulation programming: more than one passenger can be scheduled for check-in with the same idle clerk or tax agent. This is a (deliberate) artifact of the program design; but alternate designs tend to lead to other timing problems, and some can create bugs that are not easy to detect. As given here, NextEvent simply reschedules a passenger for whom there is no idle server and chooses another passenger.

EventQueue can never be empty because the last arrival always schedules the next, and that (unborn) passenger cannot be scheduled for any process except Arrival, which then schedules another arrival. . . .

The division of labor among the procedures is somewhat arbitrary. In particular, the decision to schedule P or place the record in a waiting line ClerkQueue (and TaxQueue) can be moved to CheckIn (and TaxIn). With that change, CheckIn, CheckOut, and ClerkQueue form a unit that communicates with the other processes principally by way of the action of Schedule and NextEvent on EventQueue. Procedures TaxIn, TaxOut, and waiting line TaxQueue form a similar tax unit. The program presented in this section leaves them as described, and the alternative design is suggested as a programming project in PJK.5. (The change simplifies NextEvent.)

In various forms, the ability to package procedures, data structures, and persistent variables confined within a unit, and to spawn copies of a unit as a type has been incorporated into languages such as Simula, Ada, and specifically simulation languages. Such languages would be an improvement on Pascal for writing simulation programs if they were not so complex themselves. A principle of the conservation of complexity seems to be at work for simulations.

With the procedures roughed out, the event-cycle level of Airport becomes:

program AirPort

Clock    0

Preface

RunAgain(HowLong)

while HowLong do

OverTime    Now + HowLong

while Now < OverTime do

p    NextEvent(EventQueue)

case of p    .WhatStage

1 : Arrival(p)

2,6 : CheckIn(p)

3,7 : CheckOut(p)

4 : TaxIn(p)

5 : TaxOut(p)

endcase

endwhile

RunAgain(HowLong)

endwhile

end {Airport

K.3 The Gathering of Data

The last and most-abused stage of a simulation is the gathering and interpretation of data generated by it. Statistical techniques of all levels may be brought to bear on simulation data, as long as it is recognized that the data derives from a model and not the actual system.

The generation of pseudorandom numbers and choice of distributions has attracted a great deal of study but perhaps not always enough in practice. It is common to make repeated runs of a simulation program and determine the mean and standard deviation of both single-run results and results from a sequence of runs. Those results are the measurement of the interaction of a model with approximations of random number sequences.

Even elementary statistics lies outside the scope of this book, and so the goal of Airport is simply to provide data on the mean response time of the model. The amount of (simulation) time spent in the system by a departing passenger is the difference between Clock values at arrival and departure. These times, and the times of occurrence for other stages, are kept in an array Stage[1 . . 7] in the passenger record. They are updated by the procedures that process a passenger at each stage. The only additional information that is gathered by Airport is the state of the system: the queue lengths and the number of idle servers. The intent here is to provide a framework that may be extended in programming projects as desired.

The data provided by Airport is enough to experimentally explore the relationship between the mean times of the distributions, the numbers of clerks and tax agents, and the response time. The experimental space for exploring response time has eight dimensions if the standard deviation of the normal distribution for clerk service is taken into account.

When making short runs, the initial conditions of the queues and the initially all-idle servers bias the results. Suggested solutions are long runs, program modification to allow a "warm start," or a limiting process. This is a universal problem in the interpretation of simulation results, with no single general solution.

Projects

Projects for fun; for serious students

PJK.1 With the choices MeanA = 1.0, Mu = 3.0, Sigma = 1.0, MeanT = 2.0, PaidTime = 1.0, and NotPaid = 0.5, determine an optimal balance of clerks and tax agents. The trade-off is to be that two minutes of additional mean response time is as costly in customer-relations as a clerk or tax agent is in money.

PJK.2 Modify Airport so that it allows an initial run to reach equilibrium (perhaps) before data-gathering begins. The warm start is to be followed by a run in segments, as before.

PJK.3 Modify Airport so that it reports the mean queue-lengths of EventQueue, ClerkQueue, and TaxQueue.

PJK.4 Modify Airport so that it reports the utilization of its resources: what fraction of the time are how many clerks and tax agents busy? The output should be a plot of time against number busy (a histogram) for each type of server.

PJK.5 Modify Airport so that the decision to schedule a passenger or to place the record in a waiting line is included in CheckIn and TaxIn.

K.4 The Program Airport

PROGRAM Airport (Input, Output);

TYPE  Person =    Entity;

Entity = RECORD

What Stage : 0 . .7;

Stage : ARRAY[1 . . 7] OF INTEGER;

DoTime : INTEGER;

Link : Person;

id : INTEGER

END;

WaitingLine = RECORD

Front : Person;

Rear : Person

END;

VAR  Passenger : Person;

EventQueue : Person;

p : Person;

ClerkQueue : WaitingLine;

TaxQueue : WaitingLine;

EQL,CQL,TQL : INTEGER;

ResponseSum : INTEGER;

IdleTaxMen : INTEGER;

IdleClerks : INTEGER;

Departures : INTEGER;

Arrivals : INTEGER;

OverTime : INTEGER;

PaidTime : INTEGER;

HowLong : INTEGER;

Seed : INTEGER;

Clock : INTEGER;

Sigma,Mu : REAL;

NotPaid : REAL;

MeanA : REAL;

MeanT : REAL;

{  }

FUNCTION Random(VAR Seed : INTEGER) : REAL;

CONST Multiplier = 13077;

Increment = 6925;

Modulus = 32767;

BEGIN  {A version that works well on a KayPro when

compiled with JRT Pascal.}

Seed := (Multiplier * Seed + Increment) MOD Modulus;

Random := (Seed/Modulus + 1.0) /2.0

END {Random};

{  }

FUNCTION Normal : INTEGER;

VAR  z : REAL;

k,N : INTEGER;

BEGIN

REPEAT

z := 0;

FOR k := 1 TO 12 DO z := z + Random(Seed);

z := z - 6.0;

N := TRUNC(0.5 + Sigma * z + Mu)

UNTIL (N > 0);

Normal := N

END {Normal};

{  }

FUNCTION LN(X : REAL) : REAL; EXTERN;

{  }

FUNCTION Expon(OneOver : REAL) : INTEGER;

VAR  k,ia : INTEGER;

BEGIN

REPEAT

ia := TRUNC(0 . 5 - LN(Random(Seed)) * OneOver)

UNTIL (ia > 0);

Expon : = ia

END {Expon};

{  }

PROCEDURE Report(p : Person);

VAR  k : INTEGER;

BEGIN                     {Used for debugging}

Writeln;                {of modifications}

Writeln (EQL,CQL,TQL);

Writeln ('Passenger' ,p    . id);

Writeln ('Stage ',p    .WhatStage);

Write ('Stage history : ');

FOR k := 1 TO 7 DO Write(p    .Stage[k]/60.0 : 7 : 2);

Writeln; Writeln

END {Report};

{  }

FUNCTION NewPassenger : Person;

VAR  p : Person;

k : INTEGER;

BEGIN

NEW(p);

p    .WhatStage := 1;

p    .DoTime := Clock + Expon(MeanA);

FOR k := 1 TO 7 DO p    .Stage[k] := 0;

p    .id := Arrivals + 1;

NewPassenger := p

END {NewPassenger};

{  }

PROCEDURE Schedule(VAR p : Person);

VAR  q,r : Person;

BEGIN

r := EventQueue;

IF (r = NIL)

THEN BEGIN

p    .Link := NIL;

EventQueue := p

END

ELSE IF (p    .DoTime < r    .DoTime)

THEN BEGIN

p    .Link := r;

EventQueue := p

END

ELSE BEGIN

q := r    .Link;

WHILE (q <> NIL) DO

IF (p    .DoTime <> q    .DoTime)

THEN BEGIN

r := q;

q := q    .Link

END

ELSE q := NIL;

p    .Link := r    .Link;

r    .Link := p

END;

EQL := EQL + 1

END {Schedule};

{  }

PROCEDURE Preface;

VAR  p : Person;

PT : REAL;

PROCEDURE Setup;

BEGIN

EventQueue := NIL;

ClerKQueue.Front := NIL;

ClerkQueue.Rear := NIL;

TaxQueue.Front := NIL;

TaxQueue.Rear := NIL;

EQL := 0; CQL := 0; TQL := 0;

Seed := 1;

Arrivals := 0;

Departures := 0;

ResponseSum := 0;

MeanA := MeanA * 60;

MeanT := MeanT * 60;

Sigma := Sigma * 60;

Mu := Mu * 60;

p := NewPassenger;

PaidTime := TRUNC(0.5 + PT * 60)

END {Setup};

{  }

BEGIN

Writeln('This is a simulation of an airport customs');

Writeln('station, Passengers enter the station with');

Writeln('exponentially distributed interarrival times.');

Writeln('What mean time (in minutes) would you prefer?');

Write('MeanA = '); READLN (MeanA);

Writeln('Clerks deal with a new passenger in a normally');

Writeln('distributed time. What mean service time would');

Writeln('you prefer?');

Write('Mu = '); READLN(Mu);

Writeln('What standard deviation for this distribution');

Writeln('would you prefer?');

Write('Sigma = '); READLN(Sigma);

Writeln('How many clerks do you prefer at the counter?');

Write('IdleClerks = '); READLN(IdleClerks);

Writeln('A fraction of the passengers have not paid entry')

Writeln('tax and are shuttled to a tax counter. What');

Writeln('fraction would you prefer?');

Write('NotPaid = '); READLN(NotPaid);

Writeln('Tax service requires an exponentially dis-');

Writeln('tributed time. What mean time would you prefer?');

Write('MeanT = '); READLN(MeanT);

Writeln('How many tax men do you prefer at the counter?');

Write('IdleTaxMen = '); READLN(IdleTaxMen);

Writeln('Passengers re-enter the clerk queue after');

Writeln('taxes and are re-serviced in constant time.');

Writeln('What time would you prefer?');

Write('PaidTime = '); READLN(PT);

Writeln('You may run the simulation in segments: and the');

Write('state of the system passes on to the next seg');

Writeln('ment.');

Setup;

Schedule(p)

END {Preface};

{  }

FUNCTION Empty(VAR Q : WaitingLine) : BOOLEAN;

BEGIN

IF (Q.Front = NIL)

THEN Empty : = TRUE

ELSE Empty : = FALSE

END {Empty};

{  }

PROCEDURE InsertRear(VAR Q : WaitingLine; VAR p : Person);

BEGIN

p   .Link := NIL:

IF Empty(Q)

THEN Q.Front := p

ELSE Q.Rear    .LinK := p;

Q. Rear := p

END {InsertRear};

{  }

PROCEDURE RemoveFront(VAR Q : WaitingLine; VAR p : Person);

BEGIN

IF Not Empty(Q)

THEN BEGIN

p := Q.Front;

Q.Front := Q.Front   .Link;

p   .DoTime := Clock

END

END {RemoveFront};

{  }

PROCEDURE RunAgain(VAR HowLong : INTEGER);

BEGIN

Writeln('The current time is ',Clock);

Writeln('How long do you want this run to be?');

Write('HowLong = '); READLN(HowLong);

HowLong := HowLong * 60

END {RunAgain};

{  }

FUNCTION NextEvent (VAR EventQueue : Person) : Person;

VAR  NoRoom : BOOLEAN;

p : Person ;

s : INTEGER;

BEGIN

NoRoom: = TRUE;

WHILE NoRoom DO BEGIN

NoRoom := FALSE;

p : = EventQueue;

EventQueue : = EventQueue    .Link;

EQL : = EQL - 1;

s : p   .WhatStage;

IF ((s=2) OR (s=6)) AND (IdleClerks <> 0)

THEN BEGIN

InsertRear(ClerkQueue,p);

CQL : = CQL + 1;

NoRoom : = TRUE

END;

IF ((s=4) AND (IdleTaxMen < = 0))

THEN BEGIN

InsertRear (TaxQueue,p);

TQL := TQL + 1;

NoRoom := TRUE

END

END;

NextEvent : p;

END  {NextEvent};

{  }

PROCEDURE Arrival(VAR p : Person);

VAR q : Person;

BEGIN

Arrivals : = Arrivals + 1;

Clock : =p    .DoTime;

p    .Stage [1] : = Clock;

q := NewPassenger;

Schedule(q);

p    .WhatStage:= 2;

IF  (IdleClerks > 0)

THEN Schedule(p)

ELSE BEGIN

InsertRear (ClerkQueue,p);

CQL := CQL + 1

END

END {Arrival}

{  }

PROCEDURE Depart(p : Person);

VAR  s : INTEGER;

BEGIN

Departures := Departures + 1;

s : = p    .WhatStage ;

ResponseSum := ResponseSum + p    .Stage[s] - p    .Stage [i];

DISPOSE(p)

END {Depart};

{  }

PROCEDURE CheckIn(VAR p : Person);

VAR  s,DeltaC : INTEGER;

BEGIN

IdleClerks : = IdleClerks - 1;

Clock := p    .DoTime;

s := p    .WhatStage ;

p    .Stage [s] := Clock;

IF  (s = 6)

THEN DeltaC := PaidTime

ELSE DeltaC := Normal;

p    .DoTime := Clock + DeltaC;

p    .WhatStage := s + 1;

Schedule(p)

END  {CheckIn};

{  }

PROCEDURE CheckOut(VAR p : Person);

VAR   s : INTEGER;

TaxPaid : BOOLEAN;

BEGIN

IdleClerks := IdleClerks + 1;

Clock := p    .DoTime;

s := p    .WhatStage;

   .Stage[s] := Clock;

WRITE('CKO ');REPORT(p);        {MODEL TRACE}

TaxPaid := (s = 7)  OR  (Random(Seed) > NotPaid);

IF  TaxPaid

THEN Depart(p)

ELSE BEGIN

p    .WhatStage := 4;

IF  IdleTaxMen > 0

THEN Schedule(p)

ELSE BEGIN

InsertRear(TaxQueue,p);

TQL := TQL + 1

END

END;

IF   NOT Empty(ClerkQueue)

THEN BEGIN

RemoveFront (ClerkQueue ,p);

Schedule(p);

CQL := CQL - 1

END

END   {CheckOut};

{  }

PROCEDURE TaxIn(VAR p : Person);

VAR s : INTEGER;

BEGIN

IdleTaxMen := IdleTaxMen - 1;

Clock : = p    .DoTime;

s := p    .WhatStage;

p    .Stage[s] := Clock;

p    .DoTime := Clock + Expon (MeanT);

p    .WhatStage : = s + 1;

Schedule(p)

END  {TaxIn};

{  }

PROCEDURE TaxOut(VAR p : Person);

VAR s : INTEGER;

BEGIN

IdleTaxMen := IdleTaxMen + 1;

Clock := p    .DoTime;

s := p    .WhatStage;

p    .Stage[s] := Clock;

p    .WhatStage := 6;

IF   (IdleClerks > 0)

THEN Schedule(p)

ELSE BEGIN

InsertRear(ClerkQueue,p);

CQL := CQL + 1

END;

IF   NOT Empty(TaxQueue)

THEN BEGIN

RemoveFront(TaxQueue,p);

Schedule(p);

TQL := TQL - 1

END

END   {TaxOut};

{  }

PROCEDURE Summary;

VAR ResponseTime : REAL;

BEGIN

Write('There were ' ,Arrivals,' arrivals, and ');

Writeln(Departures,' departures.');

ResponseTime := ResponseSum/Departures;

Writeln('The mean amount of time spent in the system');

Write('by departing passengers was:', ResponseTime/60);

Writeln(' minutes.');

Writeln('Number of idle clerks: ', IdleClerks);

Writeln('Clerk queue length: ',CQL);

Writeln('Number of idle tax men: ',IdleTaxMen);

Writeln('Tax queue length: ',TQL);

Writeln('Event queue length: ',EQL)

END  {Summary};

{  }

BEGIN                          {AIRPORT}

Clock := 0;

Preface;

RunAgain(HowLong);

WHILE  (HowLong > 0)  DO BEGIN

OverTime := Clock + HowLong;

WHILE (Clock < OverTime) DO BEGIN

p := NextEvent(EventQueue);

CASE  p    .WhatStage OF

1 : Arrival(p);

2,6 : CheckIn(p);

3,7 : CheckOut(p);

4 : TaxIn(p);

5 : TaxOut(p)

END

END;

RunAgain(HowLong)

END;

Summary

END  {Airport}.

Go to Section L Return to Table of Contents


Data Structures: SECTION L: CHAINING IN SCATTER STORAGE

 Return to Table of Contents

SECTION L: CHAINING IN SCATTER STORAGE

In section B, the collision of keys mapped to the the same range index in a hash table was resolved by hunting through the table for an empty cell during the STORE operation and for the key during a RETRIEVE operation. The hash table was a static structure, and the actual location of a hashed key in the table was not always the direct result of the hashing function.

An alternate method is to use a table cell as the header of a linked list that contains all of the keys that hash to the index of that cell. Reaching the head of the list is an immediate result of hashing the key, although the list must be searched for retrieval. (The STORE operation may probe the list to detect duplicate entries.)

With the definitions of section B and the use of q, m = 17, Hash = h, and alphabetical entry order, a list-header hash table for the list of ingredients of section 2.5 is shown in Figure L.1.

Figure L.1

The major change introduced by chaining lies in the search for a key in the linear list of a header. The function Probe used by both StoreHT and RetrieveHT is a list-walk:

function Probe(Start,s)                   {O(m)

p    H[Start]

while p    NIL do

if p    .Key = s then exit endif

p    p    .Link

endwhile

return p

end  {Probe

procedure StoreHT(Key)                    {O(Probe)

k    Hash(Key)

p    Probe(k,Key)

if p = NIL

then NEW(q)

q    .Value    Key

q    .Link    HT[k]    .Link

HT[k]    .Link    q

else {deal with a duplicate entry

endif

end  {StoreHT

function RetrieveHT(Key)                  {O(Probe)

k    Hash(Key)

p    Probe(k,Key)

if p = NIL

then {report not there

else return p

endif

end  {RetrieveHT

This logic retrieves the location or presence of a key in the table, enough for many applications. In other applications it must be expanded somewhat to allow deletion of keys from the table--something difficult to manage with a static hash table. A chained table can be an excellent choice for a hash table from which deletion is to occur.

The chained hash table has a slight advantage over the static table in the easy detection or inclusion of duplicate entries. If the keys are 20-character (20-byte) strings and if pointers require 2 bytes, the static table occupies m X 20 bytes. For n entries, the chained table requires 2 X m + n X (20 + 1) bytes. As n/m   [image]  0, this approaches 2 X m. For keys that are themselves pointers, chaining uses more space. A deeper analysis is beyond the scope of this book, but double-hashing of unchained tables is a common professional choice.

Problem

Problem not immediate, but requiring no major extensions of the text

PL.1     Design the procedures and functions that would allow deletion from a chained hash table.

Program

Program for trying it yourself

PGL.1     Write a program that accepts identifiers of up to 12 characters in length, converts them to an integer with q, and stores them in a chained hash table HT with the use of Hash = h and m = 47. The program should report the number of collisions, the average number of links followed by Probe, and the loading factor.

Go to Section M            Return to Table of Contents


Data Structures: SECTION M: WALKING A GENERAL LIST

 Return to Table of Contents

SECTION M: WALKING A GENERAL LIST

One technique that is helpful for working through the maze of links of a general list is to mark processed nodes, as done in Rvisit (section 7.7). It is convenient to cycle through the paths generated by Link[1] ... Link[n] by making Mark an integer: Mark = 0 for an unprocessed node and Mark = i when the Link[i] path is being traced.

Marking the nodes prevents redundant wandering through the structure but does not provide the retracing of a path needed to return to a branchpoint in the traverse. The retracing can be done by reversing the links and using two pointers, Son and Father to bridge positions that temporarily lack a link between them.

The operation of the pointers is most easily seen when it is applied to a linear list, as shown in Figure M.1, (where NIL links have been left blank for clarity). This scheme can be carried out with the help of a temporary pointer Base:

Figure M.1

If Son [image] .Mark = 0 then process Son and increment Son [image] .Mark to indicate that the link is being followed.

If Son [image] .Mark = 1 then examine Son [image] .Link. If it is NIL then this route cannot be processed further; increment Son [image] .Mark to indicate that Son is completely processed. If Son [image] .Link is not NIL then shift pointers to follow Son [image] .Link, as shown in Figure M.2 .

Figure M.2

If Son [image] .Mark = 2 then the path leaving from Son is completely processed. The pointers are then shifted to retrace the path that led to Son, as illustrated in Figure M.3 .

Figure M.3

This schematic process can be generalized to nodes with n link fields by incrementing the Mark field each time one of the links leading from a node is followed. The retracing path is followed when Mark = n + 1. An abbreviated trace applied to the example of a general list given at the beginning of section 7.7 is shown in Figure M.4 .

Figure M.4

The SUE version of this scheme is:

procedure Visit(p)

Father    NIL

Son    p

repeat

k    Son    .Mark

case of k

0 : PROCESS(Son)

Son    .Mark    Son    .Mark + 1

1 .. n : if Son    .Mark[k] = NIL

then Son    .Mark    Son    .Mark + 1

else Base    Father

Father    Son

Son    Father    .Link[k]

Father    .Link[k]    Base

endif

n + 1 : k    Father    .Mark

Base    Father    .Link[k]

Father    .Link[k]    Son

Son    Father

Father    Base

Son    .Mark    Son    .Mark + 1

endcase

until Father = NIL

end  {Visit

Projects

Projects

Projects for fun; for serious students

PJM.1 Implement and test the general visitation algorithm of this section. One way to create a general list for testing is to first place a sparse number of 1's in an N by N table T, with none on the main diagonal T[k,k]. Create an array of N nodes, each of which has N Link fields. If T[i,j] = 1, then for pointer p to node i, set p [image] .Link[i] to node [j].

PJM.2 A fundamental limitation of this algorithm is that the final value of p [image] .Mark is n + 1, not 0. A change to modular arithmetic for the Mark field causes infinite loops. (Why?) One solution is to add a switch that causes the counting process to work with an initial mark of either 0 or n + 2. If the switch is global, it can be reset on exit from Visit so that it alternates between two values. Incorporate the necessary changes into Visit and run it. Hint: Let k [image] | p [image] .Mark - Switch | and update p [image] .Mark with a displacement that may be [image] 1 depending on Switch.

Go to Section N Return to Table of Contents


Data Structures: SECTION N: THE MAIN DIAGONAL CIRCULATION SPARSE MATRI

 Return to Table of Contents

SECTION N: THE MAIN DIAGONAL CIRCULATION SPARSE MATRIX

An example of a very small but relatively sparse matrix is:

M  1  2  3  4

1  1  0  0  0

2  1  1  0  1        n by n for n = 4

3  0  1  0  0

4  1  0  0  0

This matrix is represented as an OL (Orthogonal List) structure in section 6.5. Its cells can also be represented by an interwoven collection of circular lists where there is one value node for each nonempty cell, one braid for each row, one braid for each column, and a header node for each possible index. The value nodes can be represented in the form:

CellType = RECORD

Braid[1]:    .CellType  {row

Braid[2]:    .CellType  {column

Dim[1]: INTEGER      {row index

Dim[2]: INTEGER      {column index

END

The header node for index k contains a subnode of CellType that is on both the kth row braid and the kth column braid. As shown in Figure N.1, the header nodes themselves are linked into a double linear list:

HeadType = RECORD

Junction : Cell Type

Forth :     HeadType  {row

Back  :     HeadType  {column

END {HeadType

When the links for the first column and the third row are added to this picture, the result is as shown in Figure N.2. Figure N.3 shows the result when all of the links for the example are in place. (See pages 446 and 447.) The value nodes are linked into the header structure, as shown in Figure N.4 on page 448. This structure is an MDC, for Main Diagonal Circulation Sparse Matrix.

In an MDC, header nodes can account for roughly half of the nodes in the structure. For example, a 100 X 100 matrix with 117 value nodes out of 10,000 possible nodes will also use 100 header nodes. For comparison, if the data field is a pointer, OL of section 6.5 would require space for 317 nodes of CeIlType and MDC would require the equivalent of 257.

The list of headers can also be treated as sparse, with headers only for nonempty rows and columns; but deletion and insertion become somewhat more complex. The basic treatment given here can be modified as needed.

Both STORE and RETRIEVE involve finding the current header, then the predecessor of the node to be accessed. Header location for row or column dex is an adaptation of LT (section 3.3.2):

function FindIndex(dex)

p    M.Head

while p    .Junction.Dim[1] > dex do

p    p    .Forth

endwhile

return p

end {FindIndex

The appropriate row and column can then be located by:

Row    FindIndex(dex)

Col    Row

With this minor change for determining Row and Col, the procedures FindPred, Retrieve, and Store of section 6.5 for structure OL apply to MDC as well.

Figure N.1

Figure N.2

Figure N.3

Figure N.4

Go to Section O Return to Table of Contents


Data Structures: SECTION O: A QUICKSORT PROGRAM

 Return to Table of Contents

SECTION O: A QUICKSORT PROGRAM

The procedure QuickSort acts on a sequence of keys, assumed to be in array Key[Low . . High]. It repeatedly establishes Property P(k)

Key[i] [image] Key[k] for Low [image] i [image] k

Key[k] [image] Key[j] for k [image] j [image] High

for some k, until the set of keys Key[Low] [image] Key[k] [image] Key[High] is sorted.

The value of Key[k] is called the pivot, and its initial position within the subrange Key [Low] . . . Key [High] is called the PivotIndex. When P(k) has been established, k is the index of the (new) position of the pivot in the range Low [image] k [image] High. QuickSort then calls upon itself to further arrange Key[Low] . . . Key[k - 1] and also Key[k + 1] . . . Key[High].

Two auxiliary routines are needed by QuickSort. Function FindPivot determines the pivot index (and returns 0 if all of the keys in the index subrange [Low . . High] are identical--hence, if Low = High). Procedure Partition establishes P(Last) in the index subrange [Low . . High]. (Last is the last candidate for k in the action of Partition.) With these routines:

procedure QuickSort(Low,High)

PivotIndex    FindPivot(Low,High)

if PivotIndex    0

then Pivot    Key[PivotIndex]

Partition(Last,Low,High,Pivot)

QuickSort(Low,Last - 1)

QuickSort(Last + 1,High)

endif

end  {QuickSort

It is tempting to look for clever pivot-selection strategies, but a simple one is fairly serviceable: choose the larger of the first two values within the subrange [Low . . High] that differ. The logic of this strategy is:

function FindPivot(Low,High)

Value    Key[Low]

for m = Low + 1 to High do

case

Key[m] > Value : return m

Key[m] < Value : return Low

endcase

next m

return 0

end {FindPivot

Another is: take the first one, Key[Low].

These strategies are helpful in a set with duplicate values, but are inefficient in a set that is already sorted. A better strategy is to choose a pivot at random from Key[Low . . High], but the machinery for choosing a random index is almost as complex as QuickSort itself unless it is available as a utility routine. The interested reader may consult section A and the program in section K.

When the pivot value has been determined, P(Last) may be developed within the subrange [Low . . High] by swapping values that are larger than pivot and have a smaller index, with values that are no larger than pivot and have a larger index. There are a number of ways to do the testing and swapping operations, but all involve a fundamental asymmetry because one test of values relative to Pivot is a strict inequality and the other is not.

The strategy used here is relatively simple to debug. The strategy starts with Last = Low + 1, and sweeps from Low + 1 to High, testing Key[dex] against Pivot. When Key[dex] < Pivot, it is swapped with Key[Last], and Last is incremented. Now Key[Last] (the new Last) may not be less than Pivot, but it will be swapped with the next such value encountered. This process creates a segment of values from Key[Low + 1] to Key[Last - 1] that are all smaller than Pivot. Finally, Key[Low] and Key[Last] (the very last) are swapped. In short:

procedure Partition(Last,Low,High,Pivot)

Last    Low

for  dex = Low + 1 to High  do

if  Key[dex] < Pivot

then Last    Last + 1

Swap(Last,dex)

endif

next dex

if Low    Last  then Swap(Low,Last) endif

end {Partition

QuickSort is faster on the average than the sorting techniques SelectionSort, BubbleSort, and InsertionSort discussed in Chapter 1, but not for very small or nearly sorted sets. A rough analysis is:

The operation likely to consume the most time is the exchange of records in Swap.

No element in Key[Low] . . . Key[High] is exchanged more than once during a call Partition(Last,Low,High,Pivot). Hence in terms of Swap units of time, Partition is O(High - Low + 1).

It is possible that Last = Low or Last = High, in which case one of the subranges at the next call contains High - Low elements. Hence in the worst case, Partition is called n times for a file of size n. Quicksort is thus 0(n2).

Partition tends to produce two new subranges, each with roughly half as many elements as are in [Low . . High]. Hence QuickSort tends to execute ln n levels, each acting on subranges of no more than n elements. Hence the average growth behavior of QuickSort is O(n ln n).

It is common to supplement QuickSort with a switch to InsertionSort for small subranges, a practice derived from experience.

Sorting algorithms with worst-case behavior O(n 1n n) are possible, and one such algorithm is discussed in section 8.6.1.

0.1 The QuickSort Demonstration Program

0.1 The QuickSort Demonstration Program

PROGRAM  QuickSortDemo(Input,Output);

CONST   Max = 50;

TYPE   KeyType = ARRAY [1   Max] OF INTEGER;

VAR    Key : KeyType;

Swaps : INTEGER;

Tests : INTEGER;

Tail : INTEGER;

Done : BOOLEAN;

{  }

PROCEDURE Preface;

BEGIN

Writeln('This program demonstrates the action of a');

Writeln('QuickSort procedure. It accepts a sequence');

Writeln('of up to ' ,Max,' non-negative integers, sorts');

Writeln('them, and reports the number of comparisons');

Writeln('and swaps needed to sort them.')

END  {Preface};

{  }

PROCEDURE Display(Low,High : INTEGER);

VAR m : INTEGER;

BEGIN

Writeln;

FOR  m := Low TO High DO Write(Key[m] : 4);

Writeln;

Writeln(Swaps,' Swaps, ', Tests,' Comparisons')

END {Display};

{  }

PROCEDURE Setup(VAR Key : KeyType; VAR Tail : INTEGER);

VAR NonNeg : INTEGER;

BEGIN

Writeln('Enter values separated by blank spaces,');

Writeln('terminating the sequence with a negative');

Writeln('value.');

Tail := 0 ;

Read(NonNeg);

WHILE  (NonNeg  >= 0)  DO BEGIN

Tail := Tail + 1;

Key[Tail] := NonNeg;

Read(NonNeg)

END;  Readln;

Swaps := 0;  Tests := 0

END  {Setup};

{  }

PROCEDURE RunAgain(VAR Done : BOOLEAN);

VAR ch : CHAR;

BEGIN

Write('If you wish to run again then enter Y, else N.');

Readln(ch);

IF  (ch = 'Y')

THEN Done := FALSE

ELSE Done := TRUE

END  {RunAgain};

{  }

PROCEDURE  QuicKSort(Low,High : INTEGER);

VAR  PivotIndex  : INTEGER;

Pivot  : INTEGER;

Last  : INTEGER;

{  }

FUNCTION FindPivot(Low,High : INTEGER)  : INTEGER;

BEGIN

IF  (Low < High)             {subject to modification}

THEN FindPivot := Low

ELSE FindPivot := 0

END   {FindPivot};

{  }

PROCEDURE Partition(VAR Last : INTEGER;

Low,High,Pivot : INTEGER);

VAR  dex : INTEGER;

{  }

PROCEDURE Swap(One,Two : INTEGER);

VAR Temp : INTEGER;

BEGIN

Temp := Key[One];

Key[One] := Key[Two];

Key[Two] := Temp;

Swaps := Swaps + 1

END {Swap};

{  }

BEGIN {Partition}

Last := Low;

FOR  dex := Low + 1 TO High   DO

IF  Key[dex] < Pivot

THEN BEGIN

Last := Last + 1;

Swap(Last,dex)

END;

IF   (Low <> Last) THEN Swap(Low,Last);

IF   (High > Low) THEN Tests := Tests + High - Low

END   {Partition};

{  }

BEGIN            {QuickSort}

PivotIndex :=  FindPivot(Low,High);

IF  PivotIndex  <> 0

THEN BEGIN

Pivot := Key[PivotIndex];

Partition(Last,Low,High,Pivot);

QuicKSort(Low,Last - 1);

QuicKSort(Last + 1,High)

END

END   {QuicKSort};

{  }

BEGIN               {QuicKSortDemo}

Preface;

REPEAT

Setup(Key,Tail);

Display(1,Tail);

QuickSort(1,Tail);

Display(1,Tail);

RunAgain(Done)

UNTIL  Done

END  {QuickSortDemo}.

Go to Section P Return to Table of Contents


Data Structures: SECTION P: A BALANCED STATIC BST

 Return to Table of Contents

SECTION P: A BALANCED STATIC BST

A static BST that is quite well balanced can be formed from a set of nodes by first sorting them and then arranging them in a tree according to their sorted order. The first 11 nodes can be placed in trees like those of Figure P.1 as they arrive.

Figure P.1

If node 10 is made the right child of node 8, the result is a BST. If node 12 had arrived before the final connection, node 10 would be its left child and node 12 would be the right child of node 8. After arrival of node 13 and after final connection, the tree would look like the one in Figure P.2.

Figure P.2

The general idea is to place each node at the position where it would be in a complete binary tree. This creates a forest of binary trees, and the trees of the forest are connected together into one tree after the last node has been acquired. (The dashed links in the example are created at that time.) Note that the twelfth node cannot arrive until after the eighth. In general there is only one node on each level that is not the child of a higher node. There is never more than one unattached subtree per level.

An incoming kth node is a leaf if k is odd, on the next level up if it is twice an odd number, another level up if it is divisible by 4 but not 8, and so on. (Note that 4, 12, and 20 all contain factors of 4, multiplied by an odd number.) In general, the height of the kth node, counting from the leaves, is determined by the power of the multiples of 2 that k contains. The height determination is:

function FinalHeight(k)

FH    0

while NOT ODD(k) do

k    k DIV 2

FH    FH + 1

endwhile

return FH

end {FinalHeight

Nodes are accessed by function NextNode, which is assumed to take them from a sorted file of nodes or create them from a sorted file of keys (of records perhaps). The node acquired by NextNode is inserted into a subtree. For example, the eleventh node is inserted into the subtree rooted at the tenth node and already containing the ninth node, but not yet connected as the right child of the eighth node (if the final node count is less than 12) or the left child of the twelfth node. After the last node is received, the root of the final tree is located as OneRoot, and all the subtrees are connected together by OneTree. The logic of this scheme is:

procedure StaticBalance(OneRoot)

Setup

InCount    0

p    NextNode

while p    NIL do

InCount    InCount + 1

Insert(p)

p    NextNode

endwhile

OneTree(OneRoot)

end  {StaticBalance

The crucial piece of the puzzle is the linking of p into some subtree. Building a complete tree one node at a time quickly shows that the kth node is not only a child of a node at the next higher level: it is a child of the last node entered at that next level that is not yet linked to both of its children. If there is no such node, the entry node becomes the root of a new tree of the forest; otherwise, it becomes the root of a subtree of a tree that is already in the forest. A structure is needed to keep track of the one subtree root per level; and an array of pointers, Root, is chosen to provide fast access to them. A maximum height of 20 allows up to 1,048,576 nodes in the forest. The initialization of Setup is then:

for k = -1 to MaxHeight do

Root[k]    NIL

next k

The insertion procedure is:

procedure Insert(p)

Height    FinalHeight(InCount)

p    .Right    NIL

p    .Left    Root[Height - 1]

Root[Height]    p

r    Root[Height + 1]

if r    NIL

then if r    .Right = NIL

then r    .Right    p

endif

endif

end {Insert

To find the final root of the combined tree, it suffices to search through Root for the node of greatest height that is not NIL:

OneRoot    MaxHeight

while Root[OneRoot] = NIL AND OneRoot > 1 do

OneRoot    OneRoot -1

endwhile

From the height of OneRoot, the subtrees can be systematically connected together:

procedure OneTree(OneRoot)

OneRoot    MaxHeight

while Root[OneRoot] = NIL AND OneRoot > 1 do

OneRoot    OneRoot - 1

endwhile

Height    OneRoot

while Height > 0 do

r    Root[Height]

if r    .Right    NIL

then Height    Height - 1

else p    r   .Left

Hunt    Height - 1

repeat

p    p    .Right

Hunt    Hunt = 1

until p = NIL OR p    Root[Hunt]

r    .Right    Root[Hunt]

Height    Hunt

endif

endwhile

end {0neTree

Problem

Program

Problem

Problem not immediate, but requiring no major extensions of the text

PP.1 Trace the action on OneTree when a total of k nodes are entered into StaticBalance for k values 11, 12, and 13.

Program

Program for trying it yourself

PGP.1 Incorporate the algorithm StaticBalance into a program (with a tree display) and experiment with it.

Go to Section Q Return to Table of Contents


Data Structures: SECTION Q: HUFFMAN CODE TREES

 Return to Table of Contents

SECTION Q: HUFFMAN CODE TREES

Messages are frequently decoded one character at a time or one bit (binary digit) at a time. A variety of coding and decoding structures is available for this task, including a search tree (or a search trie, described in section 8.5.3). The decoding can be done efficiently if the encoding is done so that the most frequently used messages have the shortest paths in the decoding tree. Since files are messages, application of this approach provides file compression.

One extremely short file is:

ABBREVIATE_THIS_SHORT_MESSAGE_AS_MUCH_AS_POSSIBLE_

When sorted and counted, the character frequency of it is:

A 5   F 0   K 0   P 1   U 1   Z 0

B 3   G 1   L 1   Q 0   V 1   _ 8

C 1   H 3   M 2   R 2   W 0

D 0   I 3   N 0   S 8   X 0

E 5   J 0   O 2   T 3   Y 0

The relative frequencies of the letters that actually occur are:

0.16  S  _

0.10  A  E

0.06  B  H  I  T

0.04  M  O  R

0.02  C  G  L  P  U  V

0.00  D  F  J  K  N  Q  W  X  Y  Z

The letters with zero frequency may be ignored, and the symbol SP used for the underscore_. The characters in the file can then be used to form a tree in stages. The characters are first considered to be one-node trees with a value that is their frequency of occurrence. At each stage, the lowest available pair of frequencies is taken as the children of a new node. The new node has an effective frequency that is the sum of the frequencies of its children, since it represents their combination. The process is iterated on the roots of the forest of trees that results from the previous stage. The results of the first three stages of this process are shown in Figure Q.1 and the next three in Figure Q.2. Figure Q.3 illustrates the next two combinations.

Figure Q.1

Figure Q.2

Figure Q.3

The next two stages join trees of value (0.08 and 0.08) and (0.08 and 0.10). The final result is shown in Figure Q.4.

Figure Q.4

Now assign 0 to left links and 1 to right links and let the sequence of links used to reach a leaf directly from the root be its binary code.

The resulting codes are:

A  011     M   0011     C  00000

S  101     I   0100     G  00001

           T   0101     L  00010

           B   1000     P  00011

           O   1100     U  00100

           R   1101     V  00101

           H   1001

           E   1110

           SP  1111

These codes are unique, since the path to any leaf is unique. They also can be separated from each other during decoding, even though they vary in length. Suppose, for example, the following message, encoded by HT, is received:

1010101110011011001

It can be uniquely decoded by simply following the tree links from root to leaves:

The first digit, 1 , means: take the right link.

The second digit, 0, means: take the left link.

The third digit, 1, means: take the right link. This, however, reaches a leaf node, "S". Hence this character code is complete and the next digit begins a new search from the root.

Exercises EQ.1 and EQ.2 are appropriate at this point.

Even for so short a file, there is some compression. If characters are encoded in a standard 8-bit code, this file is 50 X 8 = 400 bits long. Encoded with the tree above, it is 5 X 3 + 3 X 4 + [image] [image] [image] + 8 X 4 = 193 bits in length. In statistical terms, the length of a file encoded with this tree tends to be the expected code length times the number of characters in the file. The expected code length of this tree is [image] bits.

The general algorithm that joins subtrees of highest priority (frequency, probability, and so on) to form a final tree is called the Huffman Algorithm. It remains to describe it in detail.

Q.1 Building a Huffman Tree

The entire coding and decoding system must include a way to encode as well as decode. A reasonable structural system is an array Symbol[1 .. N] of symbol nodes, arranged for easy access for encoding--sorted by field Key perhaps. Each node needs to contain the encoding string, Code, of binary digits, to be derived from the Huffman tree after it is constructed.

Each node has a left and right link, which are indices of other nodes in the tree to be constructed. Since the tree contains internal nodes, Symbol actually contains Max = N + (N - 1) nodes. Each node has a field Measure that contains a positive fraction, with the property that the sum of these values from 1 to N is 1.00. The nodes thus have the form:

SymbolType = RECORD

Measure : REAL

Right : INTEGER

Left : INTEGER

Key : {some ordered type

Code : {digit string

END

The heap initially occupies an integer array Map[1 . . N], the values of its cells being pointers to (indices of) nodes in Symbol. (With little change in what follows, the cells of Map could contain pointers into a collection of independent records.) The nodes of Symbol can be heaped indirectly by using Measure values for comparisons, but shifting pointers within Map. For example, UpHeap (section 8.6) becomes:

procedure UpHeap(k)

while  k    2 do

p    INT(k/2)

mk    Symbol[Map[k]].Measure

mp    Symbol[Map[p]].Measure

if  mk < mp

then TempDex    Map[k]

Map[k]    Map[p]

Map[p]    TempDex

k    p

else exit

endif

endwhile

end  {UpHeap

Note that the inequality is reversed, because small measures have high priority.

The required alteration to DownHeap (section 8.6) is similar. The next level of heap management is unaffected:

procedure MakeHeap

for  k = 2 to N  do

UpHeap(k)

next k

end  {MakeHeap

The task of tree-building is accomplished by using the top two items of the heap with an initial directory Map[1 .. N] and adding one combined item derived from them by summing their Measure fields. Eventually there are 2N - 1 items in the heap; the original N values and their combinations pointed to by Map[1 .. 2N - 1].

A modification of DeHeap (section 8.6) is needed. It assumes the symbols are sorted so that the measure of Map[k] is no larger than that of Map[k + 1]:

procedure MakePair(SN)                {O(ln SN)

TopLeft    Map[1]

Lm    Symbol[TopLeft].Measure

m2    Symbol[Map[2]].Measure

m3    Symbol[Map[3]].Measure

if m2 < m3

then Rm    m2

TopRight    Map[2]

else Rm    m3

TopRight    Map[3]

endif

SN    SN + 1

Symbol[SN].Left    TopLeft

Symbol[SN].Right    TopRight

Symbol[SN].Measure    Lm + Rm

UpHeap(SN)

end {MakePair

Once the arrays and Symbol have been initialized for N nodes, the tree is constructed and located by:

function Huff Tree                           {O(N    ln N)

SN    N

for  k = 1 to N - 1  do MakePair(SN) next k

return Map[1]

end  {Huff Tree

Finally, the codes are determined for encoding by a traverse of the tree:

If d1d2 . . . dk is the code string for a parent, then d1d2 . . . dk 0 is the code for the left child, and d1 d2 . . . dk 1 is the code for the right child. Every node is reached from its parent.

The position of a given symbol can be tracked by initializing Track[k] to k and incorporating two statements into the then condition of UpHeap just before the switch of Map values:

Track[Map[k]]    p

Track[Map[p]]    k

The final value of Track[k] is the index m such that Map[m] = k--the heap position of Symbol[k].

Exercises

Exercises immediate application of text materials

EQ.1 Decode the rest of the message used as an example in this section.

EQ.2 Form a Huffman tree from the file:

THIS_STUFF_IS_OLD_HAT

Problem

Problem not immediate, but requiring no major extensions of the text

PQ.1 Put together the entire Huffman code construction for a set of N frequency counts.

Program

Program for trying it yourself

PGQ.1 Write a program that reads a sequence of characters and displays the corresponding Huffman tree.

Project

Project for fun; for serious students

PJQ.1 Write a program that uses one text file to generate a Huffman tree, then will accept other files and encode a text file or decode a binary string that has been encoded with the same tree.

Go to Section R Return to Table of Contents


Data Structures: SECTION R: AVL TREES

 Return to Table of Contents

SECTION R: AVL TREES

The time required to locate a value or discover that it is not present in a binary search tree is dependent on the depth of the tree. It is customary to refer to the depth as height and talk about height-balanced binary trees, a denotation due to the developers of efficient management techniques that control the shape of ordered binary trees. Such trees are called AVL trees after Adelson-Velskii and Landis [ADELSON, 1962.]

The height of tree T is h(T). For node p, h(p) is the height of the subtree rooted at p.

The balance factor of binary tree T with subtrees L rooted at its left child and R rooted at its right child is Factor(T) = h(L) - h(R). This definition extends to any node of a binary tree. If T is an empty tree, Factor(T) = 0.

If [image] Factor(T) [image] [image] 1, then T is height-balanced.

The nodes of BT8, in Figure R.1 are tagged with their balance factors:

Figure R.1

Exercise ER.1 and Problem PR.1 are appropriate at this point.

Given a binary tree bt and a value v, one position at which v may be inserted to retain the order property is determined by NodeSpot and inserted by InsertNode (section 8.5.2). The result is not the only BST with the values it contains. The goal of AVL management is to rearrange bt after the addition of each value so that it is both height-balanced and a BST. There are a number of cases to consider.

A general binary tree T is of the form shown in Figure R.2 .

Figure R.2

If h(L) = h(R), then Factor(T) = 0 and the addition of one node to form a new binary tree T' may (or may not) change the balance factor of the root node p to [image] 1, whence T' is still balanced at the node p. Balancing within L or R may be necessary. Note, for example, that a child added to the leftmost leaf node of BT8, would not change the balance of the root node, but an addition to other leaves would do so. Two rightmost additions will leave the root node balanced but the balance factor of the rightmost node of BT8, would change to + 2.

Exercise ER.2 is appropriate at this point.

If h(L) - h(R) = 1, then the addition of a node to R will leave the resulting Factor(p) either 0 or 1 for the root node p, still balanced. If, however, the new tree T' is formed by adding a node to L, the result can be that Factor(p) = 2 unless an adjustment is made. A suitable adjustment switches some of the nodes of L to the right subtree without violating the order property. In that case, p is said to act as a pivot.

Suppose an addition is made to the subtree L, rooted at node Lroot. If the left and right subtrees of Lroot are LL and LR, respectively, then no unbalance is created by an addition unless the height of one of these subtrees is h(R) and the addition is to that subtree and increases its height. Figure R.3 is a model of T.

Figure R.3

If the addition increases h(LL) to h(R) + 1, T can be restructured without violating the BST property by using p as a pivot, as shown in Figure R.4. If h(L) - h(R) = - 1 and the addition is at the right in the subtree RR (the analog of LL), the pivot in the opposite direction that may be necessary is symmetric.

Figure R.4

Figure R.5

A second case occurs when h(LR) = h(R) and an addition is made to LR that causes h(LR) to become h(R) + 1, whence h(Lroot) becomes h(R) + 2. The pivot above will not balance the tree and so one level deeper is involved, as Figure R.5 illustrates. This can be rearranged without violating the BST property by using p as a pivot, as in Figure R.6. The new T' will be balanced if either h(LRR) or h(LRL) is increased by the added node. The symmetric right subtree case for which Factor(T) = - 1 may also occur and be dealt with by a symmetric pivot.

Figure R.6

A special case is required for the addition of a node nn to the non-null child of a two-node binary tree. Figure R.7 shows how the left version of this pivot is modeled.

Figure R.7

It is important to realize that "root node" generally refers to the root of a subtree, and that pivoting is carried out as deep in the tree as possible.

Exercises ER.3 and ER.4 are appropriate at this point.

R.1 The Insertion Algorithm for AVL Trees

It is convenient to include a field for the balance factor in the nodes of an AVL tree:

AVLnode = RECORD

Value  : {value type

Left :    AVLnode

Right :    AVLnode

Factor : INTEGER

END

As a node is inserted, the location of the parent of the inserted node and the closest ancestor with Factor > 0, are retained in Parent and Pivot, respectively.

An AVL tree is a BST, and so InsertNode (section 8.5.2) can be used to insert a node p in the tree T. However, it is necessary to modify NodeSpot (section 8.5.2) in order to retain a pointer, Pivot, to the closest ancestor of the new node with nonzero balance factor and its immediate ancestor Parent. This can be done by initializing Pivot to T and Parent to NIL and adding the statement

if p    .Factor    0

then Pivot    p

Parent    Pred

endif

as the first statement within the while loop in NodeSpot.

The insertion algorithm becomes:

procedure AVLinsert(nn,T)

if Empty(T)

then T    nn

nn   .Factor    0

return

endif

NodeSpot(T,nn    .Value,Trio,p,Pred,Pivot,Parent)

InsertNode(T,nn,Pred,Trio)

{small-tree cases are

if T    .Right = NIL                      {treated separately

then PairLeft

return

endif

if  T    .Left = NIL

then PairRight

return

endif

{change factors between Pivot and

NewFactors                               {nn, and determine the apparent

{change to Pivot    .Factor, PF

if  Pivot    .Factor = 0

then Pivot    .Factor    PF

return

endif

if  Pivot    .Factor + PF = 0

then Pivot    .Factor = 0

return

endif

if  PF < 0

then LeftPivot

else RightPivot

endif

end  {AVLinsert

procedure NewFactors

p    Pivot

v    nn    .Value

if p    .Value < v

then p    p    .Right

PF     -1

else p    p    .Left

PF    +1

endif

Hub    p                               {the first node between Pivot and nn

while p    nn do

if p    .Factor < v

then p    .Factor    -1

p    p    .Right

else p    .Factor    +1

p    p    .Left

endif

endwhile

end  {NewFactors

procedure LeftPivotd

if Hub    .Factor = 1                  {do a simple rotation of Hub

then Pivot    .Left    Hub    .Right

Hub     .Right    Pivot

Pivot    .Factor    Hub    .Factor

Hub     .Factor    0

if  nn    .Value < Parent    .Value

then Parent    .Left    Hub

else Parent    .Right    Hub

endif

return

endif

{do a deeper rotation

Cog    Hub    .Left

CogLeft    Cog    .Left

CogRight    Cog    .Right

Hub    .Right    CogLeft

Pivot    .Left    CogRight

Cog    .Left    Hub

Cog    .Right    Pivot

case of  Cog    .Factor

+1  : Pivot    .Factor    -1

Hub    .Factor    0

-1  : Pivot    .Factor    0

Hub    .Factor    +1

else: Pivot    .Factor    0

Hub    .Factor    0

endcase

Cog    .Factor    0

if nn    .Value < Parent    .Value

then Parent    .Left    Cog

else Parent    .Right    Cog

endif

end  {LeftPivot

RightPivot is similar.

A PairLeft procedure can be implemented like this:

procedure PairLeft

T    .Left    NIL

if Trio = 1

then Pred    .Right    T

T    Pred

else nn    .Right    T

nn    .Left    Pred

Pred    .Right    NIL

T    nn

endif

T    .Factor    0

end {PairLeft

PairRight is similar.

Deletion from an AVL tree requires modification of CutValue (section 8.5.2) to determine Parent and Pivot, and the rebalance operation must proceed up the tree. It is common to forego deletion entirely; inactive nodes may be so marked or simply ignored.

Exercises

Exercises immediate applications of the text material

ER.1     Calculate the balance factors for all of the binary trees in Chapter 8.

ER.2     Trace the effects of adding two nodes to BT8 at every possible combination of positions.

ER.3     Suppose that the days of the week are used to form an AVL tree by adding them one at a time to an empty tree. Diagram the stages of the tree when they are added in the reverse order: Saturday, Friday, . . ., Sunday.

ER.4     Form an AVL tree from the sequence: 1 3 5 7 9 11 8 6 14 by adding them one at a time. They are to be filed in nondecreasing order. Then repeat for the AVL tree that keeps them in nonincreasing order.

Problem

Problem not immediate, but not a major extension of the text

PR.1     Design a procedure to calculate balance factors for an arbitrary binary tree.

Project

Project for fun; for serious students

PJR.1     Write a program that creates an AVL tree by insertion and displays it.

Go to Section S            Return to Table of Contents


Data Structures: SECTION S: FILE MERGING AND MERGESORT

 Return to Table of Contents

SECTION S: FILE MERGING AND MERGESORT

The StraightMerge procedure of section 9.2.1 has two variations of interest, the merging of linked lists and binary merging. The discussion of these forms of merging in this section is based on the assumption that the number of items to be merged is large, but will fit into memory. (Lists too large to fit into memory are discussed in section T.) For large lists, it is worthwhile to spend some effort on preparation before plunging into the heart of an algorithm. In both cases, it is assumed for the sake of simplicity that key values are numerical and that the lists to be merged are sorted in nondecreasing order.

S.1 A Linked List Merge

The two lists a and b to be merged are accessed by records containing Head and Tail pointers. The merging operation is a function that produces a similar list c.

The role played by the sentinel value in StraightMerge is that of simplifier: it simplifies the test for exit in a much-repeated loop and also precludes the need to run out the tail of one of the lists when the other is exhausted. A similar and extended role can be played by a sentinel value in a distinguished node Pivot when linked lists are merged.

Pivot is first created with a sentinel value larger than either of the tail node values of a and b and then used as the tail node of both, as shown in Figure S.1.

Figure S.1

The merging operation selectively removes nodes from a and b and adds them to the new list c, with head node Pivot [image] .Link, as Figure S.2 illustrates.

Figure S.2

Preparation for the merge is done by:

procedure Setup                                  {O(1)

if a.Tail    .Key    b.Tail    .Key

then Sentinel    a.Tail    .Key + 1

else Sentinel    b.Tail    .Key + 1

endif

NEW(Pivot)

Pivot    .Link    NIL

Pivot    .Key    Sentinel

c.Tail    Pivot

a.Tail    .Link    Pivot

a.Tail    Pivot

b.Tail    .Link    Pivot

b.Tail    Pivot

end {Setup

After Setup has been executed, the leading node of smallest value in either list can be shifted by:

procedure GetLeader                              {O(1)

if a.Head    .Key    b.Head   .Key

then Temp    a.Head

a.Head   a.Head   .Link

else Temp    b.Head

b.Head    b.Head   .Link

endif

Temp   .Link    NIL

c.Tail   .Link    Temp

c.Tail   Temp

end {GetLeader

With these tools, the merge is:

function ListMerge(a,b)                          {O(n)

Setup

while a.Head    b.Head do

GetLeader

endwhile

c.Head    Pivot    .Link

return c

end {ListMerge

In practice, GetLeader would not be a called procedure; it is a shorthand reference to code that is incorporated at the point mentioned.

S.2 Binary Merge

Sometimes one list (say b) is much smaller than the other (a) with which it is to be merged. Suppose the lists are stored in arrays a[1 . . N] and b[1 . . M], sorted in nondecreasing order, and they are to be merged into an array c[1 . . NM], where NM = N + M.

The general idea of binary merge is that since N is much larger than M, entire segments of a tend to fall between values of b. The search for the possible position of a b-value is localized to a small segment of a that is searched by the method of binary search to determine an interval of a that can be moved to c. It is convenient to extend a with a sentinel value in a[0] and then to load c from right to left.

A value b[i] is compared with a segment of a of length Size: a[Left .. Right]. If b[i] < a[Left], then the entire segment of a can be moved to c. If not, the segment is searched with a tailored version of binary search for the largest index s such that: a[s] < b[i]. The subsequence b[i],a[s + 1] ,a[s + 2] ,...,a[Right] can then be moved into c. In either case, a new segment of a is chosen for testing against the (possibly) new value b[i].

The move operation is:

procedure Move(x,y)                              {O(y - x)

for z = y downto x do

c[k]    a[z]

k    k - 1

next z

end {Move

An effective search segment size is one less than the largest power of 2 less than (N/M). It can be determined by Size [image] Power(N/M) - 1, using:

function Power(x)                                {O(ln x)

PX    1

while PX    x do

PX    PX X 2

endwhile

return (PX DIV 2)

end {Power

The initialization procedure is:

procedure Setup                                  {O(1)

if a[1] < b[1]

then Sentinel    a[1] - 1

else Sentinel    b[1] - 1

endif

a[0]    Sentinel

Size    Power(N / M) - 1

i    M

k    N X M

end {Setup

The merge operation is then:

procedure BinaryMerge                            {O(n 1n n)

Setup

while i    1 do

if b[i] < a[Left]

then Move(Left,Right)

Right    Left - 1

else s    BinSearch(b[i],Left,Right)

Move(s + 1,Right)

Right    s

c[k]    b[i]

k    k - 1

i    i - 1

endif

Left    Right - Size

if Left < 0 then Left    0 endif

endwhile

end {BinaryMerge

S.3 Merge Sorting

The simplest form of sorting routine based on merging sorts an array, a, with the help of an array, b, of the same size. Segments in one array are merged into the other. The segments are initially of length one and are increased in size from one pass of the routine to another until one of them contains the entire file. It is convenient to make passes back and forth between the arrays. The merging procedures of the previous sections do not work unmodified, partly because sentinel records would need to be placed in the middle of a file.

A utility routine used in the merge that copies a segment of one array into another is:

procedure Copy, (a,x,y,b,z)                      {O(y - x)

for dex = x to y do

b[z]    a[dex]

z    z +1

next dex

end {Copy

With the help of Copy, the basic merge operation is:

procedure TailMerge(a,L,M,R,b)                   {O(R - L)

i    L

k    L

j    M + 1

while i    M AND j    R do

if a[i]    a[j]

then b[k]    a[i]

i    i + 1

else b[k]    a[j]

j    j + 1

endif

k    k + 1

endwhile

if i > M

then Copy(a,j,R,b,k)

else Copy(a,i,M,b,k)

endif

end {TailMerge

Now one phase (pass) is of the form:

procedure Phase(a,b,Size)                        {O(N)

s    1

while s    (N - 2 * Size - 1) do

TailMerge(a,s,s + Size - 1,s + 2 * Size - 1,b)

s    s + 2 * Size

endwhile

if (s + Size - 1) < N

then TailMerge(a,s,s + Size - 1,N,b)

else Copy(a,s,N,b,s)

endif

end {Phase

The sort then becomes a repetition of phases:

procedure MergeSort                              {O(N 1n N)

Size    1

while Size < N do

Phase(a,b,Size)

Size    2 * Size

Phase(b,a,Size)

Size    2 * Size

endwhile

end {MergeSort

Programs

Programs for trying it yourself

PGS.1 Write a program that creates two linked lists, sorts them, displays them, merges them with ListMerge, and displays the resulting list.

PGS.2 Write a program that initializes two arrays to be sorted--one eight times larger than the other--sorts them, merges them with BinaryMerge, and displays the result.

PGS.3 Implement MergeSort.

Go to Section T Return to Table of Contents


Data Structures: SECTION T: PRIORITY QUEUE EXTENSION OF MERGE RUNS

 Return to Table of Contents

SECTION T: PRIORITY QUEUE EXTENSION OF MERGE RUNS

The merge operations of section S work within memory, which may not be large enough to hold a file of interest. A common way to handle large files is to write them in segments of sorted values, called runs. Pairs of runs are merged to form larger runs--the essential idea of MergeSort. The access to external storage is expensive, however, and so larger runs and fewer phases increase efficiency.

One way to form the initial runs for a file is to simply make one pass through the file, writing values found to be in order onto a run file and the others onto a reject file for a second pass. Consider the integer file:

2 1 6 3 8 9 4 4 10 5 2 8

The first pass would produce the following run and reject:

run     2 6 8 9 10

reject  1 3 4 4 5 2 8

The straightforward version of this scheme, acting on a source file, whose termination can be recognized by the flag EOF(Source), is:

Read(Source, Cutoff)

while NOT EOF(Source) do

Read(Source,x)

if x    Cutoff

then Write(RunFile,x)

Cutoff    x

else Write(Reject,x)

endif

endwhile

Unfortunately, if the largest item is first, the run will be only one item long. A second approach is to input M items, sort them internally, and write them out as a run. It is this value M that can be stretched with the help of a priority queue, specifically a heap, although other forms of priority queues would serve.

Suppose the items from Source are tagged with a run number R, and placed in a heap, PQ, with room for M elements. PQ is taken to be an array of records of the form:

RECORD

Run : INTEGER

Key : {comparable type

END

The heap is maintained in lexicographic order:

PQ[i] < PQ[j] iff:

PQ[i].Run < PQ[j].Run, or

PQ[i].Run = PQ[j].Run and PQ[i].Key < PQ[j].Key

For consistency with section S, the highest priority in the heap is the smallest value. The heap is formed with:

procedure BuildPQ

for i = 1 to M do

Read(Source,x)

PQ[i].Run    1

PQ[i].Key    x

UpHeap(i)

next i

HN    m

end {BuildPQ

The stretched run is then created with:

procedure PQruns

BuildPQ

R    1

while NOT Empty(PQ) do

Top    PQ[1]

if Top.Run = R

then Write(RunFile(R),Top.Key)

Swap(PQ[1],PQ[HN])

HN    HN - 1

DownHeap(1)

else R    R + 1

endif

if NOT EOF(Source)

then Read(Source,x)

if x    Top.Key

then xRun    R

else xRun    R + 1

endif

HN    HN + 1

PQ[HN].Run    xRun

PQ[HN].Key    x

UpHeap(HN)

endif

endwhile

end {PQruns

It is possible to derive additional enhancement by replacing R [image] R + 1 by a procedure DumpPQ that stores the queue on external storage so that the pass through the source file can continue. The dumps are retrieved for the next run.

Program

Program

Program for trying it yourself

PGT.1 Implement PQruns.

Go to Section U Return to Table of Contents


Data Structures: SECTION U: RED-BLACK TREES

 Return to Table of Contents

SECTION U: RED-BLACK TREES

A 3-ode or a 4-ode can be modeled with a binary subtree, bound together with links that are called red links to distinguish them from the links of the 2-3-4 tree in which they occur, called black links. With the use of dashed lines to represent red links, the tree is shown in Figure U.1 . In a tree represented this way, 4-nodes are characterized by having a subtree root with red links to both children.

The depth of an RBT (Red-Black Tree) binary tree may be twice that of the corresponding 2-3-4 tree, but it is still proportional to the log of the number of keys in it. The links of an RBT need to be flagged with their color; and, if the nodes are entirely supported in arrays, the sign bit of the pointer index may be used. An alternate scheme, used here, is to flag a node with the color of the link that leads to it from its parent. It is convenient to include a field in the node that contains a pointer Up to the parent of the node.

The creation operation becomes:

procedure Create(rbt,LeftChild,RightChild)

NEW(rbt)

rbt    .Up    rbt

rbt    .Left    LeftChild

rbt    .Right    RightChild

rbt    .Red    FALSE

end {Create

Figure U.1

The search operation, likely to be used more than any other, is then:

procedure RBSearch(rbt,SearchKey,Node,Found)

Node    rbt

Parent    rbt

while Node    u do                 {u is the universal tail.

if SearchKey = Node    .Key

then Found    TRUE

return

endif

Parent    Node

if  SearchKey < Node    .Key

then Node    Node    .Left

else Node    Node    .Right

endif

endwhile

Found    FALSE

Node    Parent

end {RBSearch

Insertion includes a similar traverse that splits 4-nodes as they are encountered. The splitting operation is applied to the root node, or to a 4-node reached from a 2-node (a 2-4 subtree) or a 4-node reached from a 3-node (a 3-4 subtree).

The root-node split is shown schematically in Figure U.2 . The split simply changes two red links to black links. No key value is actually shifted.

Figure U.2

The 2-4 subtree split schematically changes the (2-node to 4-node subtree) to a (3-node to two 2-nodes) subtree, shown in Figure U.3.

Figure U.3

Once again, no information is altered except the color of the links.

In the case of a 3-4 subtree, the 3-node may have either binary node as the subtree root. For example, Figure U.4 may be represented in either of the ways shown in Figure U.5 .

Figure U.4

Figure U.5

Similarly, the 3-4 subtree of Figure U.6 has two representations, shown in Figure U.7, both having a red grandparent-parent link relative to the 4-node.

Figure U.6

Figure U.7

When the grandparent-parent link of a 4-node is red, a simple color-shift applied during splitting of the node produces two red links in sequence, as Figure U.8 illustrates for case (2) (from Figure U.5).

Figure U.8

This is prevented for the case of Figure U.4 by a rotation, in which the lexicographic order of the keys is unchanged, but the relative levels of two nodes are switched. A rotation switches the binary-tree levels of the two keys in a 3-node and shifts links to their parent and children to match. A rotation of keys x and y is considered to be the rotation of the child and grandchild of the node n, as depicted in Figure U.9. After this rotation, splitting the right child of gc does not produce two red links in succession.

Figure U.9

If a 4-node is the middle child of a 3-node, it remains a middle child after rotation and retains a red grandparent-parent link, as shown in Figure U.10. One approach to resolving this problem is a prior rotation involving the root of the 4-node itself, shown in Figure U.11. Figure U.12 shows the result when this rotation is followed by a rotation at the (original) great-grandparent of node.

Figure U.10

Figure U.11

Figure U.12

With color changes at z and f, this double rotation has the effect in the RBT shown in Figure U.13. None of these rotations change the number of levels in the 2-3-4 tree.

Figure U.13

It is convenient to design a generic rotation that internally traces the path from node to grandchild in order to detect left and right branches:

procedure Rotate(SKey,n)

if SKey > n    .Key

then c    n    .Right

else c    n    .Left

endif

if SKey > c    .Key

then gc    c    .Right

ggc    gc    .Left

c    .Right    ggc

gc    .Left    c

else gc    c    .Left

ggc    gc    .Right

c    .Left    ggc

gc    .Right    c

endif

if SKey > n    .Key

then n    .Right    gc

else n    .Left    gc

endif

gc    .Up    n

c    .Up    gc

ggc    .Up    c

Color    gc    .Red

gc    .Red    c    .Red

c    .Red    Color

end {Rotate

The middle-child case is characterized by the alteration in direction of grandparent-parent and parent-node links and requires a rotation to change it into the other case. The splitting operation thus becomes:

procedure RBSplit(SKey,Node)

p    Node    .Up

if p    .Red

then gp    p    .Up

ggp    gp    .Up

if SKey < gp    .Key    SKey < p    .Key

then Rotate(SKey,gp)

if SKey < p    .Key

then p    .Right    .Red    TRUE

Node    .Left    .Red    FALSE

else p    .Left    .Red    TRUE

Node    .Left    .Red    FALSE

endif

endif

Rotate(SKey,ggp)

endif

if Node    rbt then Node    .Red    TRUE endif

Node    .Left    .Red    FALSE

Node    .Right    .Red    FALSE

end {RBSplit

The insertion procedure repeats the search process explicitly, except that it also splits 4-nodes during the traverse. The use of a universal tail node u allows the new node to be treated as the root of a 4-node subtree and immediately split upon entry to properly adjust its relationship to its parent:

procedure RedBlackIn(rbt,NewKey,NewValue)

if rbt = u

then Create(rbt,u,u)

rbt    .Value    NewValue

rbt    .Key    NewKey

return

endif

Node    rbt

while Node    u do

if NewKey = Node    .Key

then return               {or otherwise deal with

endif                     {a duplicate key.

Parent    Node

if Node    .Left    .Red AND Node    .Right    .Red

then RBSplit(NewKey,Node)

else if NewKey < Node    .Key

then Node    Node    .Left

else Node    Node    .Right

endif

endif

endwhile

Create(Node,u,u)

with Node do

Up    Parent

Value    NewValue

Key    NewKey

Red    FALSE

endwith

if NewKey < Parent    .Key

then Parent    .Left    Node

else Parent    .Right    Node

endif

RBSplit(NewKey,Node)

end {RedBlackIn

Problem

Program

Problem

Problem not immediate, but requiring no major extensions of text

PU.1     Draw the sequence of Red-Black trees that would be created by entering the months in calendar order if they are to be kept in alphabetical order.

Program

Program for trying it yourself

PGU.1     Write a program that will accept strings and place them in a Red-Black tree ordered lexicographically. The program should display the tree after entries. Test it with the months of the year, entered in reverse calendar order.

Go to Section V            Return to Table of Contents


Data Structures: SECTION V: THE EIGHT COINS PROBLEM

 Return to Table of Contents

SECTION V: THE EIGHT COINS PROBLEM

Some problems are both well enough known and representative enough that every computer science major should work through them at some time. One of these is the Eight Coins Problem, a representative of a class of common weighing puzzles. It is this:

One of eight apparently identical coins is known to be counterfeit and hence of different weight from the other seven. An equal-arm balance is to be used to discover in three weighings which coin is different and whether it is heavier or lighter.

There are sixteen possible results: 1-Light, 1-Heavy, 2-light, . . ., 8-Heavy, and a structured approach is likely to be worth developing. A systematic approach to the determination is to form a decision tree where each node describes a weighing to be made, determined by the results discovered so far. An interior node also determines which child to use for the next decision, and a leaf node is the correct answer if reached by proper use of the tree.

Let the coins have weights w[i] for 1 [image] i [image] 8. A verbal description of the decisions to be made is:

1. Compare the sum of weights 1-3, s1 , and weights 4-6, s2. There are three possibilities: s1 = s2,s1 < s2, and s1 > s2. If s 1 = s2, then the counterfeit coin is isolated as the pair of coins 7 and 8, and the other six are all of standard weight. When coins 7 and 8 are compared with each other, one of them will be heavier. The other two cases use the second comparison to isolate a similar pair of coins and provide relative weight information. At the third weighing, one of the isolated pair is compared to a standard coin, and the answer becomes apparent.

1.1 s1 = s2,: The bad coin is isolated in the pair w[7] and w[8], and so they are compared to each other.

1.1.1 w[7] > w[8]: One of them, say w[7], is compared with a standard coin, say w[1].

1.1.1.1 w[7] > standard: Then w[7] is the bad coin, and it is heavy. Otherwise, w[8] is the bad coin, and it is light.

1.1.2 w[7] < w[8]: This is symmetric to 1.1.1 above.

1.2 s1 < s2: Now w[7] and w[8] are known to be standard. Two of the coins in the triplets forming s1 and s2 need to be isolated, and two are switched. For example, the comparison might be between s3 = w[1] + w[4] and s4 = w[2] + w[5], isolating w[3] and w[6] from the other four.

1.2.1 s3: < s4: Clearly coins 3 and 6 were standard, and either coin 1 is light or coin 5 is heavy because switching coins 2 and 4 left the balance tipped in the same direction. Either coin 1 or coin 5 is to be tested against a standard, because these two are now isolated as the bad pair.

1.2.2 s3 = s4: Then coin 3 or coin 6 is counterfeit, and one should be tested against one of the six known standards.

1.2.3 s1 > s4: The switching of coins 2 and 4 caused the balance to shift, and so one of them is counterfeit.

1.3 s1 > s2: This is entirely symmetric to 1.2 above.

Even with the abbreviation of the verbal description, the decisions to be made do not jump out at the reader. A tree diagram is helpful.

In Figure V.1, the notation indicates a comparison of i, . . ., j with k, . . ., l.

Figure V.1

The tree is illustrated in Figure V.2. The design of the tree calls for a comparison of a final pair with a standard weight:

Figure V.2

procedure Compare(Coin1,Coin2,Standard,BadCoin,Light)

if Coin1 > Standard

then BadCoin    Coin1

Light    FALSE

else BadCoin    Coin2

Light    TRUE

endif

end {Compare

The procedure then becomes:

procedure EightCoins(w,BadCoin,Light)

S1  0

s2  0

for i = 1 to 3 do

s1    s1 + w[i]

s2    s2 + w[i + 3]

next i

if s1 = s2

then if w[7] > s[8]

then Compare(w[7],w[8],w[1],BadCoin,Light)

else Compare(w[8],w[7],w[1],BadCoin,Light)

endif

return

endif

if s1 > s2

then case

s3 = s4 : Compare(w[3],w[6],w[8],BadCoin,Light)

s3 > s4 : Compare(w[1],w[5],w[8],BadCoin,Light)

s3 > s4 : Compare(w[2],w[4],w[8],BadCoin,Light)

endcase

return

endif

{s1 < s2

case

s3 = s4 : Compare(w[6],w[3],w[8],BadCoin,Light)

s3 > s4 : Compare(w[5],w[1],w[8],BadCoin,Light)

s3 < s4 : Compare(w[5],w[1],w[8],BadCoin,Light)

endcase

end {EightCoins

Program

Program

Program for trying it yourself

PGV.1 Write the Eight Coins program and run it with several of the possible input sets.

Go to Section W Return to Table of Contents


Data Structures: SECTION W: NODE EVALUATION AND PRUNING OF GAME TREES

 Return to Table of Contents

SECTION W: NODE EVALUATION AND PRUNING OF GAME TREES

A game progresses as the odd player makes a move to level 2, the even player moves to level 3, and so on. The active player is player Odd on odd levels and player Even on the others. The value of a node is the value of it to the player who moves from that node. A leaf node represents a final configuration of the game and hence a win or loss for some player. The value of a leaf node is the payoff for some player, and the convention is:

A positive value for a leaf node represents a winning value for the player who moves from it. A negative value represents a winning amount for the other player. Zero represents a tie.

In very few games can the board configuration be described with great brevity and the game tree drawn on a sheet of notebook paper (but see problem PW. 1). The game tree in Figure W. I is arbitrary and greatly pruned. In GT1 there are wins for Odd at nodes 5, 11, and 14 and wins for Even at nodes 6, 10, and 12.

Figure W.1

The tension in a competitive game comes from the opposition in viewpoint of the value of a configuration. Players can evaluate the node they occupy by examining the values of the children of that node. Odd wishes to maximize the payoff.

The value of node p for Odd is the maximum from the view of Odd of the values of p [image]. Child[i] [image] for all i.

Similarly, Even wishes to minimize the payoff (to Odd), but minimizing values is equivalent to maximizing their negatives. We can combine these disparate views into a single rule:

The value of node p for the player moving from it is:

v = Maximum{ - p    .Child[i]    .Value for all i}

With this convention, the values of the nodes of GT1 can be derived bottom-up from an assignment of values to the terminal positions represented by the leaf nodes. The value of node 2 for example is:

Maximum( - ( + 5), - ( - 10)) = + 10

The value of node 2 for Odd is - ( + 10) = -10, and so a move to node 2 would represent a loss of 10 units for Odd.

Exercise EW. 1 and problem PW. 1 are appropriate at this point.

Node evaluation as the maximum of the negative of the values (as determined by the opposition) of the children, is based on a crucial assumption:

The player in opposition always makes the move that is optimal for him or her.

The bottom-up determination of node values can be done with either recursion or with a post-order traverse--once the leaf-node values are in place, either calculated or estimated. With the assumption that leaf values are determined by a procedure Leaf Value, a recursive procedure NodeValue can be designed to assign a value v to the given node, choose the next node Choice, and return v.

function NodeValue(gt, Choice)     {O(n)

Choice    NIL

NoChild    TRUE

MaxValue    TooSmall             {less than any possible node value

for  i = 1 to MaxChild  do

p    gt    .Child[i]

if  p    NIL

then NoChild    FALSE

v    - NodeValue(p,Choice)

if  v > MaxValue

then MaxValue    v

Choice    p    .Child[i]

endif

endif

next i

if  NoChild

then v    Leaf Value(gt)

else v    MaxValue

endif

gt    .Value    v

return v

end  {NodeValue

When a game tree is large, a search that reaches to the leaves of the tree is impractical, since NodeValue or its equivalent will not execute in a reasonable length of time. Most human players of games "look ahead" only a few moves to make a decision, rather than making an exhaustive search. A procedure can do the same, effectively pruning the tree.

In a look-ahead procedure, some way is needed to assign values to nodes that are not terminal and whose subtree is not traversed. Evaluation of such nodes may be done with a procedure, say BaseValue, that plays a role similar to that of Leaf Value for terminal nodes. Both procedures are highly dependent upon the game being played.

Given the existence of BaseValue, the number of nodes to be searched can be limited by restricting the depth of the search. A node being processed is some number of levels down from a root node, and an index LevelsDown can be used to track it. LevelsDown is initialized to the maximum depth to be allowed for the search and decremented at each new level. If LevelsDown = 0 at some node, then BaseValue is applied in lieu of searching the children of the node. When modified to use these tools, NodeValue can be used for a limited look-ahead determination of node values.

Problems PW.2 and PW.3 are appropriate at this point.

P>

W.1 Alpha-Beta Pruning of a Game Tree

A procedure like NodeValue examines every subtree of a child, but that may not be necessary. Consider the tree fragment in Figure W.2.

Figure W.2

After (the even) node N21 has been processed, its value is known to be - 5; hence the value of N1 is at least + 5. The known minimum for a node at any point in a search of its subtrees is called its alpha value. The adversary Even controls the moves from N2j for j = 1,2, . . .; and so, as soon as N34 is found to be 4, the value of N22 is known to be at least - 4. Since the value of N22 to Even is at least - 4, it is worth no more than 4 to Odd, whence Odd will prefer N21 to N22. There is no need to traverse the other children of N22.

In general, an even node chooses its child with the smallest (odd) value and passes that value up to its (odd) parent. If an even node e has a child with a value smaller than the alpha value of its parent, then the search of the tree rooted at e may be abandoned. It is said to be pruned from the search. The children of N22 are pruned and not generated in the sense that they will not be used even if they physically exist in the structure.

The equivalent pruning on information given by the grandchildren of an even node is called beta pruning. The combination of both kinds of pruning is called alpha-beta pruning. Because of the symmetry introduced by the alternation of values in NodeValue, alpha-beta pruning is identical for odd nodes.

An additional phase of pruning can be initiated by passing the alpha value of a node p to a grandchild, gcp, as a lower bound, Bound. As soon as it is known that a grandchild of gcp has a value lower than Bound, the child cgcp in which it is rooted will not allow the value it passes up to gcp to be as large as Bound, and so the remaining children of cgcp can be pruned. Figure W.3 illustrates this phase.

Figure W.3

After N21 is valued, alpha(N1) is 5. This puts a lower bound of 5 on N31: as soon as N51 has a value of 4 determined for it, the children of N41 will only lower its value, and so the other children of N41 can be ignored. This form of pruning is called bounding.

Alpha-beta pruning, bounding, and level-limiting can all be incorporated into NodeValue to form:

function PruneValue(gt,Choice,Bound,Level)

if  Level = 0  then return endif

Choice    NIL

NoChild    TRUE

MaxValue    Bound

for  i = 1  to MaxChild  do

p    gt    .Child[i]

if  p    NIL

then NoChild    FALSE

v    - PruneChild(p, Choice, - Bound,Level - 1)

if  v > MaxValue

then MaxValue    v

Choice    p    .Child[i]

endif

endif

next i

if  NoChild

then v    Bound

else v    MaxValue

endif

p    .Value    v

return v

end  {PruneValue

Exercise

Exercise immediate application of text materials

EW.1 Show how the values of the interior nodes of GT1 are derived from the values of terminal nodes.

Problems

Problems

PW.1 Draw the game tree for the game of NIM when n = 6. NIM is played by two alternating players, each of whom may pick up 1, 2, or 3 toothpicks from a pile that initially holds n. The player who empties the pile loses.

PW.2 Redesign NodeValue to be depth-limited.

PW.3 Write a search procedure that calls NodeValue.

Program

Program for trying it yourself

PGW.1 Write a program that creates 32 random leaf-node values in the range [ - 10 . . 10], builds a game tree from them, prunes it with PruneValue, and displays the pruned tree.

Project

Project for fun; for serious students

PJW.1 Make PGW.1 a procedure, n leaf nodes a variable, and study the mean and standard deviation of the nodes in the pruned trees as a function of n.

Go to References Return to Table of Contents


Data Structures: Table of Contents

Data Structures: From Arrays to Priority Queues

by Wayne Amsbury

PREFACE

PART ONE: The Mainstream

CHAPTER 1: ALGORITHMS AND TIMING ANALYSIS

CHAPTER 2: THE ARRAY STRUCTURE

CHAPTER 3: LIST PROCESSING

CHAPTER 4: STACKS

CHAPTER 5: QUEUES

CHAPTER 6: GENERAL LISTS: MULTIPLE ACCESS PATHS

CHAPTER 7: RECURSION

CHAPTER 8: BINARY TREES

CHAPTER 9: MULTIPLE ACCESS PATHS TO DATA

CHAPTER 10: GRAPH ALGORITHMS

PART TWO: Expansions and Applications

SECTION A: RANDOM PROCESSES

SECTION B: HASH TABLES

SECTION C: CIRCULAR LIST OPERATIONS

SECTION D: INTEGER ARITHMETIC OF UNBOUNDED PRECISION

SECTION E: THE STACK MACHINE

SECTION F: STACKS IN THE LANGUAGE FORTH

SECTION G: THE P-MACHINE

SECTION H: FOLLOWING A MAZE

SECTION I: THE QUEUE MACHINE

SECTION J: QUEUES IN HARDWARE

SECTION K: A SIMULATION OF AN AIRPORT CUSTOMS STATION

SECTION L: CHAINING IN SCATTER STORAGE

SECTION M: WALKING A GENERAL LIST

SECTION N: THE MAIN DIAGONAL CIRCULATION SPARSE MATRIX

SECTION O: A QUICKSORT PROGRAM

SECTION P: A BALANCED STATIC BST

SECTION Q: HUFFMAN CODE TREES

SECTION R: AVL TREES

SECTION S: FILE MERGING AND MERGESORT

SECTION T: PRIORITY QUEUE EXTENSION OF MERGE RUNS

SECTION U: RED-BLACK TREES

SECTION V: THE EIGHT COINS PROBLEM

SECTION W: NODE EVALUATION AND PRUNING OF GAME TREES

REFERENCES


Information Retrieval: CHAPTER 1: INTRODUCTION TO INFORMATION STORAGE

 Return to Table of Contents

CHAPTER 1: INTRODUCTION TO INFORMATION STORAGE AND RETRIEVAL SYSTEMS

W. B. Frakes

Software Engineering Guild, Sterling, VA 22170

Abstract

This chapter introduces and defines basic IR concepts, and presents a domain model of IR systems that describes their similarities and differences. The domain model is used to introduce and relate the chapters that follow. The relationship of IR systems to other information systems is dicussed, as is the evaluation of IR systems.

1.1 INTRODUCTION

Automated information retrieval (IR) systems were originally developed to help manage the huge scientific literature that has developed since the 1940s. Many university, corporate, and public libraries now use IR systems to provide access to books, journals, and other documents. Commercial IR systems offer databases containing millions of documents in myriad subject areas. Dictionary and encyclopedia databases are now widely available for PCs. IR has been found useful in such disparate areas as office automation and software engineering. Indeed, any discipline that relies on documents to do its work could potentially use and benefit from IR.

This book is about the data structures and algorithms needed to build IR systems. An IR system matches user queries--formal statements of information needs--to documents stored in a database. A document is a data object, usually textual, though it may also contain other types of data such as photographs, graphs, and so on. Often, the documents themselves are not stored directly in the IR system, but are represented in the system by document surrogates. This chapter, for example, is a document and could be stored in its entirety in an IR database. One might instead, however, choose to create a document surrogate for it consisting of the title, author, and abstract. This is typically done for efficiency, that is, to reduce the size of the database and searching time. Document surrogates are also called documents, and in the rest of the book we will use document to denote both documents and document surrogates.

An IR system must support certain basic operations. There must be a way to enter documents into a database, change the documents, and delete them. There must also be some way to search for documents, and present them to a user. As the following chapters illustrate, IR systems vary greatly in the ways they accomplish these tasks. In the next section, the similarities and differences among IR systems are discussed.

1.2 A DOMAIN ANALYSIS OF IR SYSTEMS

This book contains many data structures, algorithms, and techniques. In order to find, understand, and use them effectively, it is necessary to have a conceptual framework for them. Domain analysis--systems analysis for multiple related systems--described in Prieto-Diaz and Arrango (1991), is a method for developing such a framework. Via domain analysis, one attempts to discover and record the similarities and differences among related systems.

The first steps in domain analysis are to identify important concepts and vocabulary in the domain, define them, and organize them with a faceted classification. Table 1.1 is a faceted classification for IR systems, containing important IR concepts and vocabulary. The first row of the table specifies the facets--that is, the attributes that IR systems share. Facets represent the parts of IR systems that will tend to be constant from system to system. For example, all IR systems must have a database structure--they vary in the database structures they have; some have inverted file structures, some have flat file structures, and so on.

A given IR system can be classified by the facets and facet values, called terms, that it has. For example, the CATALOG system (Frakes 1984) discussed in Chapter 8 can be classified as shown in Table 1.2.

Terms within a facet are not mutually exclusive, and more than one term from a facet can be used for a given system. Some decisions constrain others. If one chooses a Boolean conceptual model, for example, then one must choose a parse method for queries.

Table 1.1: Faceted Classification of IR Systems (numbers in parentheses indicate chapters)

Conceptual  File           Query         Term        Document    Hardware

Model       Structure      Operations    Operations  Operations

------------------------------------------------------------------------------

Boolean(1)  Flat File(10)  Feedback(11)  Stem(8)     Parse(3,7)  vonNeumann(1)

Extended    Inverted       Parse(3,7)    Weight(14)  Display     Parallel(18)

 Boolean(15) File(3)

Probabil-   Signature(4)   Boolean(12)   Thesaurus   Cluster(16) IR

 istic(14)                                (9)                     Specific(17)

String      Pat Trees(5)   Cluster(16)   Stoplist(7) Rank(14)    Optical

 Search(10)                                                       Disk(6)

Vector      Graphs(1)                    Truncation  Sort(1)     Mag. Disk(1)

 Space(14)                                (10)

            Hashing(13)                              Field Mask(1)

                                                     Assign IDs(3)

Table 1.2: Facets and Terms for CATALOG IR System

     Facets                         Terms

-----------------------------------------------------------------

File Structure       Inverted file

Query Operations     Parse, Boolean

Term Operations      Stem, Stoplist, Truncation

Hardware             von Neumann, Mag. Disk

Document Operations  parse, display, sort, field mask, assign IDs

Conceptual Model     Boolean

Viewed another way, each facet is a design decision point in developing the architecture for an IR system. The system designer must choose, for each facet, from the alternative terms for that facet. We will now discuss the facets and their terms in greater detail.

1.2.1 Conceptual Models of IR

The most general facet in the previous classification scheme is conceptual model. An IR conceptual model is a general approach to IR systems. Several taxonomies for IR conceptual models have been proposed. Faloutsos (1985) gives three basic approaches: text pattern search, inverted file search, and signature search. Belkin and Croft (1987) categorize IR conceptual models differently. They divide retrieval techniques first into exact match and inexact match. The exact match category contains text pattern search and Boolean search techniques. The inexact match category contains such techniques as probabilistic, vector space, and clustering, among others. The problem with these taxonomies is that the categories are not mutually exclusive, and a single system may contain aspects of many of them.

Almost all of the IR systems fielded today are either Boolean IR systems or text pattern search systems. Text pattern search queries are strings or regular expressions. Text pattern systems are more common for searching small collections, such as personal collections of files. The grep family of tools, described in Earhart (1986), in the UNIX environment is a well-known example of text pattern searchers. Data structures and algorithms for text pattern searching are discussed in Chapter 10.

Almost all of the IR systems for searching large document collections are Boolean systems. In a Boolean IR system, documents are represented by sets of keywords, usually stored in an inverted file. An inverted file is a list of keywords and identifiers of the documents in which they occur. Boolean list operations are discussed in Chapter 12. Boolean queries are keywords connected with Boolean logical operators (AND, OR, NOT). While Boolean systems have been criticized (see Belkin and Croft [1987] for a summary), improving their retrieval effectiveness has been difficult. Some extensions to the Boolean model that may improve IR performance are discussed in Chapter 15.

Researchers have also tried to improve IR performance by using information about the statistical distribution of terms, that is the frequencies with which terms occur in documents, document collections, or subsets of document collections such as documents considered relevant to a query. Term distributions are exploited within the context of some statistical model such as the vector space model, the probabilistic model, or the clustering model. These are discussed in Belkin and Croft (1987). Using these probabilistic models and information about term distributions, it is possible to assign a probability of relevance to each document in a retrieved set allowing retrieved documents to be ranked in order of probable relevance. Ranking is useful because of the large document sets that are often retrieved. Ranking algorithms using the vector space model and the probabilistic model are discussed in Chapter 14. Ranking algorithms that use information about previous searches to modify queries are discussed in Chapter 11 on relevance feedback.

In addition to the ranking algorithms discussed in Chapter 14, it is possible to group (cluster) documents based on the terms that they contain and to retrieve from these groups using a ranking methodology. Methods for clustering documents and retrieving from these clusters are discussed in Chapter 16.

1.2.2 File Structures

A fundamental decision in the design of IR systems is which type of file structure to use for the underlying document database. As can be seen in Table 1.1, the file structures used in IR systems are flat files, inverted files, signature files, PAT trees, and graphs. Though it is possible to keep file structures in main memory, in practice IR databases are usually stored on disk because of their size.

Using a flat file approach, one or more documents are stored in a file, usually as ASCII or EBCDIC text. Flat file searching (Chapter 10) is usually done via pattern matching. On UNIX, for example, one can store a document collection one per file in a UNIX directory, and search it using pattern searching tools such as grep (Earhart 1986) or awk (Aho, Kernighan, and Weinberger 1988).

An inverted file (Chapter 3) is a kind of indexed file. The structure of an inverted file entry is usually keyword, document-ID, field-ID. A keyword is an indexing term that describes the document, document-ID is a unique identifier for a document, and field-ID is a unique name that indicates from which field in the document the keyword came. Some systems also include information about the paragraph and sentence location where the term occurs. Searching is done by looking up query terms in the inverted file.

Signature files (Chapter 4) contain signatures--it patterns--that represent documents. There are various ways of constructing signatures. Using one common signature method, for example, documents are split into logical blocks each containing a fixed number of distinct significant, that is, non-stoplist (see below), words. Each word in the block is hashed to give a signature--a bit pattern with some of the bits set to 1. The signatures of each word in a block are OR'ed together to create a block signature. The block signatures are then concatenated to produce the document signature. Searching is done by comparing the signatures of queries with document signatures.

PAT trees (Chapter 5) are Patricia trees constructed over all sistrings in a text. If a document collection is viewed as a sequentially numbered array of characters, a sistring is a subsequence of characters from the array starting at a given point and extending an arbitrary distance to the right. A Patricia tree is a digital tree where the individual bits of the keys are used to decide branching.

Graphs, or networks, are ordered collections of nodes connected by arcs. They can be used to represent documents in various ways. For example, a kind of graph called a semantic net can be used to represent the semantic relationships in text often lost in the indexing systems above. Although interesting, graph-based techniques for IR are impractical now because of the amount of manual effort that would be needed to represent a large document collection in this form. Since graph-based approaches are currently impractical, we have not covered them in detail in this book.

1.2.3 Query Operations

Queries are formal statements of information needs put to the IR system by users. The operations on queries are obviously a function of the type of query, and the capabilities of the IR system. One common query operation is parsing (Chapters 3 and 7), that is breaking the query into its constituent elements. Boolean queries, for example, must be parsed into their constituent terms and operators. The set of document identifiers associated with each query term is retrieved, and the sets are then combined according to the Boolean operators (Chapter 12).

In feedback (Chapter 11), information from previous searches is used to modify queries. For example, terms from relevant documents found by a query may be added to the query, and terms from nonrelevant documents deleted. There is some evidence that feedback can significantly improve IR performance.

1.2.4 Term Operations

Operations on terms in an IR system include stemming (Chapter 8), truncation (Chapter 10), weighting (Chapter 14), and stoplist (Chapter 7) and thesaurus (Chapter 9) operations. Stemming is the automated conflation (fusing or combining) of related words, usually by reducing the words to a common root form. Truncation is manual conflation of terms by using wildcard characters in the word, so that the truncated term will match multiple words. For example, a searcher interested in finding documents about truncation might enter the term "truncat?" which would match terms such as truncate, truncated, and truncation. Another way of conflating related terms is with a thesaurus which lists synonymous terms, and sometimes the relationships among them. A stoplist is a list of words considered to have no indexing value, used to eliminate potential indexing terms. Each potential indexing term is checked against the stoplist and eliminated if found there.

In term weighting, indexing or query terms are assigned numerical values usually based on information about the statistical distribution of terms, that is, the frequencies with which terms occur in documents, document collections, or subsets of document collections such as documents considered relevant to a query.

1.2.5 Document Operations

Documents are the primary objects in IR systems and there are many operations for them. In many types of IR systems, documents added to a database must be given unique identifiers, parsed into their constituent fields, and those fields broken into field identifiers and terms. Once in the database, one sometimes wishes to mask off certain fields for searching and display. For example, the searcher may wish to search only the title and abstract fields of documents for a given query, or may wish to see only the title and author of retrieved documents. One may also wish to sort retrieved documents by some field, for example by author. There are many sorting algorithms and because of the generality of the subject we have not covered it in this book. A good description of sorting algorithms in C can be found in Sedgewick (1990). Display operations include printing the documents, and displaying them on a CRT.

Using information about term distributions, it is possible to assign a probability of relevance to each document in a retrieved set, allowing retrieved documents to be ranked in order of probable relevance (Chapter 14). Term distribution information can also be used to cluster similar documents in a document space (Chapter 16).

Another important document operation is display. The user interface of an IR system, as with any other type of information system, is critical to its successful usage. Since user interface algorithms and data structures are not IR specific, we have not covered them in detail here.

1.2.6 Hardware for IR

Hardware affects the design of IR systems because it determines, in part, the operating speed of an IR system--a crucial factor in interactive information systems--and the amounts and types of information that can be stored practically in an IR system. Most IR systems in use today are implemented on von Neumann machines--general purpose computers with a single processor. Most of the discussion of IR techniques in this book assumes a von Neumann machine as an implementation platform. The computing speeds of these machines have improved enormously over the years, yet there are still IR applications for which they may be too slow. In response to this problem, some researchers have examined alternative hardware for implementing IR systems. There are two approaches--parallel computers and IR specific hardware.

Chapter 18 discusses implementation of an IR system on the Connection machine--a massively parallel computer with 64,000 processors. Chapter 17 discusses IR specific hardware--machines designed specifically to handle IR operations. IR specific hardware has been developed both for text scanning and for common operations like Boolean set combination.

Along with the need for greater speed has come the need for storage media capable of compactly holding the huge document databases that have proliferated. Optical storage technology, capable of holding gigabytes of information on a single disk, has met this need. Chapter 6 discusses data structures and algorithms that allow optical disk technology to be successfully exploited for IR.

1.2.7 Functional View of Paradigm IR System

Figure 1.1 shows the activities associated with a common type of Boolean IR system, chosen because it represents the operational standard for IR systems.

Figure 1.1: Example of Boolean IR system

When building the database, documents are taken one by one, and their text is broken into words. The words from the documents are compared against a stoplist--a list of words thought to have no indexing value. Words from the document not found in the stoplist may next be stemmed. Words may then also be counted, since the frequency of words in documents and in the database as a whole are often used for ranking retrieved documents. Finally, the words and associated information such as the documents, fields within the documents, and counts are put into the database. The database then might consist of pairs of document identifiers and keywords as follows.

keyword1 - document1-Field_2

keyword2 - document1-Field_2, 5

keyword2 - document3-Field_1, 2

keyword3 - document3-Field_3, 4

keyword-n - document-n-Field_i, j

Such a structure is called an inverted file. In an IR system, each document must have a unique identifier, and its fields, if field operations are supported, must have unique field names.

To search the database, a user enters a query consisting of a set of keywords connected by Boolean operators (AND, OR, NOT). The query is parsed into its constituent terms and Boolean operators. These terms are then looked up in the inverted file and the list of document identifiers corresponding to them are combined according to the specified Boolean operators. If frequency information has been kept, the retrieved set may be ranked in order of probable relevance. The result of the search is then presented to the user. In some systems, the user makes judgments about the relevance of the retrieved documents, and this information is used to modify the query automatically by adding terms from relevant documents and deleting terms from nonrelevant documents. Systems such as this give remarkably good retrieval performance given their simplicity, but their performance is far from perfect. Many techniques to improve them have been proposed.

One such technique aims to establish a connection between morphologically related terms. Stemming (Chapter 8) is a technique for conflating term variants so that the semantic closeness of words like "engineer," "engineered," and "engineering" will be recognized in searching. Another way to relate terms is via thesauri, or synonym lists, as discussed in Chapter 9.

1.3 IR AND OTHER TYPES OF INFORMATION SYSTEMS

How do IR systems relate to different types of information systems such as database management systems (DBMS), and artificial intelligence (AI) systems? Table 1.3 summarizes some of the similarities and differences.

Table 1.3: IR, DBMS, Al Comparison

                 Data Object      Primary Operation     Database Size

------------------------------------------------------------------------

IR            document              retrieval        small to very large

                                    (probabilistic)

DBMS          table                 retrieval        small to very large

(relational)                        (deterministic)

AI            logical statements    inference        usually small

One difference between IR, DBMS, and AI systems is the amount of usable structure in their data objects. Documents, being primarily text, in general have less usable structure than the tables of data used by relational DBMS, and structures such as frames and semantic nets used by AI systems. It is possible, of course, to analyze a document manually and store information about its syntax and semantics in a DBMS or an AI system. The barriers for doing this to a large collection of documents are practical rather than theoretical. The work involved in doing knowledge engineering on a set of say 50,000 documents would be enormous. Researchers have devoted much effort to constructing hybrid systems using IR, DBMS, AI, and other techniques; see, for example, Tong (1989). The hope is to eventually develop practical systems that combine IR, DBMS, and AI.

Another distinguishing feature of IR systems is that retrieval is probabilistic. That is, one cannot be certain that a retrieved document will meet the information need of the user. In a typical search in an IR system, some relevant documents will be missed and some nonrelevant documents will be retrieved. This may be contrasted with retrieval from, for example, a DBMS where retrieval is deterministic. In a DBMS, queries consist of attribute-value pairs that either match, or do not match, records in the database.

One feature of IR systems shared with many DBMS is that their databases are often very large--sometimes in the gigabyte range. Book library systems, for example, may contain several million records. Commercial on-line retrieval services such as Dialog and BRS provide databases of many gigabytes. The need to search such large collections in real time places severe demands on the systems used to search them. Selection of the best data structures and algorithms to build such systems is often critical.

Another feature that IR systems share with DBMS is database volatility. A typical large IR application, such as a book library system or commercial document retrieval service, will change constantly as documents are added, changed, and deleted. This constrains the kinds of data structures and algorithms that can be used for IR.

In summary, a typical IR system must meet the following functional and nonfunctional requirements. It must allow a user to add, delete, and change documents in the database. It must provide a way for users to search for documents by entering queries, and examine the retrieved documents. It must accommodate databases in the megabyte to gigabyte range, and retrieve relevant documents in response to queries interactively--often within 1 to 10 seconds.

1.4 IR SYSTEM EVALUATION

IR systems can be evaluated in terms of many criteria including execution efficiency, storage efficiency, retrieval effectiveness, and the features they offer a user. The relative importance of these factors must be decided by the designers of the system, and the selection of appropriate data structures and algorithms for implementation will depend on these decisions.

Execution efficiency is measured by the time it takes a system, or part of a system, to perform a computation. This can be measured in C based systems by using profiling tools such as prof (Earhart 1986) on UNIX. Execution efficiency has always been a major concern of IR systems since most of them are interactive, and a long retrieval time will interfere with the usefulness of the system. The nonfunctional requirements of IR systems usually specify maximum acceptable times for searching, and for database maintenance operations such as adding and deleting documents.

Storage efficiency is measured by the number of bytes needed to store data. Space overhead, a common measure of storage efficiency, is the ratio of the size of the index files plus the size of the document files over the size of the document files. Space overhead ratios of from 1.5 to 3 are typical for IR systems based on inverted files.

Most IR experimentation has focused on retrieval effectiveness--usually based on document relevance judgments. This has been a problem since relevance judgments are subjective and unreliable. That is, different judges will assign different relevance values to a document retrieved in response to a given query. The seriousness of the problem is the subject of debate, with many IR researchers arguing that the relevance judgment reliability problem is not sufficient to invalidate the experiments that use relevance judgments. A detailed discussion of the issues involved in IR experimentation can be found in Salton and McGill (1983) and Sparck-Jones (1981).

Many measures of retrieval effectiveness have been proposed. The most commonly used are recall and precision. Recall is the ratio of relevant documents retrieved for a given query over the number of relevant documents for that query in the database. Except for small test collections, this denominator is generally unknown and must be estimated by sampling or some other method. Precision is the ratio of the number of relevant documents retrieved over the total number of documents retrieved. Both recall and precision take on values between 0 and 1.

Since one often wishes to compare IR performance in terms of both recall and precision, methods for evaluating them simultaneously have been developed. One method involves the use of recall-precision graphs--bivariate plots where one axis is recall and the other precision. Figure 1.2 shows an example of such a plot. Recall-precision plots show that recall and precision are inversely related. That is, when precision goes up, recall typically goes down and vice-versa. Such plots can be done for individual queries, or averaged over queries as described in Salton and McGill (1983), and van Rijsbergen (1979).

Figure 1.2: Recall-precision graph

A combined measure of recall and precision, E, has been developed by van Rijsbergen (1979). The evaluation measure E is defined as:

where P = precision, R = recall, and b is a measure of the relative importance, to a user, of recall and precision. Experimenters choose values of E that they hope will reflect the recall and precision interests of the typical user. For example, b levels of .5, indicating that a user was twice as interested in precision as recall, and 2, indicating that a user was twice as interested in recall as precision, might be used.

IR experiments often use test collections which consist of a document database and a set of queries for the data base for which relevance judgments are available. The number of documents in test collections has tended to be small, typically a few hundred to a few thousand documents. Test collections are available on an optical disk (Fox 1990). Table 1.4 summarizes the test collections on this disk.

Table 1.4: IR Test Collections

Collection        Subject           Documents  Queries

------------------------------------------------------

  ADI       Information Science         82        35

  CACM      Computer Science          3200        64

  CISI      Library Science           1460        76

  CRAN      Aeronautics               1400       225

  LISA      Library Science           6004        35

  MED       Medicine                  1033        30

  NLM       Medicine                  3078       155

  NPL       Electrical Engineering   11429       100

  TIME      General Articles           423        83

IR experiments using such small collections have been criticized as not being realistic. Since real IR databases typically contain much larger collections of documents, the generalizability of experiments using small test collections has been questioned.

1.5 SUMMARY

This chapter introduced and defined basic IR concepts, and presented a domain model of IR systems that describes their similarities and differences. A typical IR system must meet the following functional and nonfunctional requirements. It must allow a user to add, delete, and change documents in the database. It must provide a way for users to search for documents by entering queries, and examine the retrieved documents. An IR system will typically need to support large databases, some in the megabyte to gigabyte range, and retrieve relevant documents in response to queries interactively--often within 1 to 10 seconds. We have summarized the various approaches, elaborated in subsequent chapters, taken by IR systems in providing these services. Evaluation techniques for IR systems were also briefly surveyed. The next chapter is an introduction to data structures and algorithms.

REFERENCES

AHO, A., B. KERNIGHAN, and P. WEINBERGER. 1988. The AWK Programming Language. Reading, Mass.: Addison-Wesley.

BELKIN N. J., and W. B. CROFT. 1987. "Retrieval Techniques," in Annual Review of Information Science and Technology, ed. M. Williams. New York: Elsevier Science Publishers, 109-145.

EARHART, S. 1986. The UNIX Programming Language, vol. 1. New York: Holt, Rinehart, and Winston.

FALOUTSOS, C. 1985. "Access Methods for Text," Computing Surveys, 17(1), 49-74.

FOX, E., ed. 1990. Virginia Disk One, Blacksburg: Virginia Polytechnic Institute and State University.

FRAKES, W. B. 1984. "Term Conflation for Information Retrieval," in Research and Development in Information Retrieval, ed. C. S. van Rijsbergen. Cambridge: Cambridge University Press.

PRIETO-DIAZ, R., and G. ARANGO. 1991. Domain Analysis: Acquisition of Reusable Information for Software Construction. New York: IEEE Press.

SALTON, G., and M. MCGILL 1983. An Introduction to Modern Information Retrieval. New York: McGraw-Hill.

SEDGEWICK, R. 1990. Algorithms in C. Reading, Mass.: Addison-Wesley.

SPARCK-JONES, K. 1981. Information Retrieval Experiment. London: Butterworths.

TONG, R, ed. 1989. Special Issue on Knowledge Based Techniques for Information Retrieval, International Journal of Intelligent Systems, 4(3).

VAN RIJSBERGEN, C. J. 1979. Information Retrieval. London: Butterworths.

Go to Chapter 2 Back to Table of Contents


Information Retrieval: CHAPTER 2: INTRODUCTION TO DATA STRUCTURES AND

 Return to Table of Contents

CHAPTER 2: INTRODUCTION TO DATA STRUCTURES AND ALGORITHMS RELATED TO INFORMATION RETRIEVAL

Ricardo A. Baeza-Yates

Depto. de Ciencias de la Computación, Universidad de Chile, Casilla 2777, Santiago, Chile

Abstract

In this chapter we review the main concepts and data structures used in information retrieval, and we classify information retrieval related algorithms.

2.1 INTRODUCTION

Infomation retrieval (IR) is a multidisciplinary field. In this chapter we study data structures and algorithms used in the implementation of IR systems. In this sense, many contributions from theoretical computer science have practical and regular use in IR systems.

The first section covers some basic concepts: strings, regular expressions, and finite automata. In section 2.3 we have a look at the three classical foundations of structuring data in IR: search trees, hashing, and digital trees. We give the main performance measures of each structure and the associated trade-offs. In section 2.4 we attempt to classify IR algorithms based on their actions. We distinguish three main classes of algorithms and give examples of their use. These are retrieval, indexing, and filtering algorithms.

The presentation level is introductory, and assumes some programming knowledge as well as some theoretical computer science background. We do not include code bccause it is given in most standard textbooks. For good C or Pascal code we suggest the Handbook of Algorithms and Data Structures of Gonnet and Baeza-Yates (1991).

2.2 BASIC CONCEPTS

We start by reviewing basic concepts related with text: strings, regular expressions (as a general query language), and finite automata (as the basic text processing machine). Strings appear everywhere, and the simplest model of text is a single long string. Regular expressions provide a powerful query language, such that word searching or Boolean expressions are particular cases of it. Finite automata are used for string searching (either by software or hardware), and in different ways of text filtering and processing.

2.2.1 Strings

We use [image] to denote the alphabet (a set of symbols). We say that the alphabet is finite if there exists a bound in the size of the alphabet, denoted by [image] [image] [image]. Otherwise, if we do not know a priori a bound in the alphabet size, we say that the alphabet is arbitrary. A string over an alphabet [image] is a finite length sequence of symbols from [image]. The empty string ( [image] ) is the string with no symbols. If x and y are strings, xy denotes the concatenation of x and y. If [image] = xyz is a string, then x is a prefix, and z a suffix of [image]. The length of a string x ( [image] x [image]) is the number of symbols of x. Any contiguous sequence of letters y from a string [image] is called a substring. If the letters do not have to be contiguous, we say that y is a subsequence.

2.2.2 Similarity between Strings

When manipulating strings, we need to know how similar are a pair of strings. For this purpose, several similarity measures have been defined. Each similarity model is defined by a distance function d, such that for any strings s1, s2, and s3, satisfies the following properties:

d(s1, s1) = 0,    d(s1, s2)    0,    d(s1, s3)    d(s1, s2) + d(s2, s3)

The two main distance functions are as follows:

The Hamming distance is defined over strings of the same length. The function d is defined as the number of symbols in the same position that are different (number of mismatches). For example, d(text,that) = 2.

The edit distance is defined as the minimal number of symbols that is necessary to insert, delete, or substitute to transform a string s1 to s2. Clearly, d(s1, s2) [image] [image] length(s1) - length(s2) [image]. For example, d(text, tax) = 2.

2.2.3 Regular Expressions

We use the usual definition of regular expressions (RE for short) defined by the operations of concatenation, union (+) and star or Kleene closure (*) (Hopcroft and Ullman (1979). A language over an alphabet [image] is a set of strings over [image]. Let L1 and L2 be two languages. The language {xy [image] x [image] L1 and y [image] L2} is called the concatenation of L1 and L2 and is denoted by L1 L2. If L is a language, we define L0 = { [image] } and Li = LLi-1 for i [image] 1. The star or Kleene closure of L, L*, is the language [image]. The plus or positive closure is defined by L+ = LL*.

We use L(r) to represent the set of strings in the language denoted by the regular expression r. The regular expressions over [image] and the languages that they denote (regular sets or regular languages) are defined recursively as follows:

is a regular expression and denotes the empty set.

(empty string) is a regular expression and denotes the set { [image] }.

For each symbol a in [image], a is a regular expression and denotes the set {a}.

If p and q are regular expressions, then p + q (union), pq (concatenation), and p* (star) are regular expressions that denote L(p) [image] L(q), L(p)L(q), and L(p)*, respectively.

To avoid unnecessary parentheses we adopt the convention that the star operator has the highest precedence, then concatenation, then union. All operators are left associative.

We also use:

to denote any symbol from [image] (when the ambiguity is clearly resolvable by context).

r? to denote zero or one occurrence of r (that is, r? = [image] + r).

[a1 . . am] to denote a range of symbols from [image]. For this we need an order in [image].

r [image] k to denote [image] (finite closure).

Examples:

All the examples given here arise from the Oxford English Dictionary:

1. All citations to an author with prefix Scot followed by at most 80 arbitrary characters then by works beginning with the prefix Kenilw or Discov:

<A>Scot     80 <W>(Kenilw + Discov)

where< > are characters in the OED text that denote tags (A for author, W for work).

2. All "bl" tags (lemma in bold) containing a single word consisting of lowercase alphabetical only:

<bl>[a..z]*</bl>

3. All first citations accredited to Shakespeare between 1610-11:

<EQ>(<LQ>)?<Q><D> 161(0+1)</D>  <A>Shak

where EQ stands for the earliest quotation tag, LQ for quotation label, Q for the quotation itself, and D for date.

4. All references to author W. Scott:

<A>((Sirb)? W)?bScott b?</A>

where b denotes a literal space.

We use regular languages as our query domain, and regular languages can be represented by regular expressions. Sometimes, we restrict the query to a subset of regular languages. For example, when searching in plain text, we have the exact string matching problem, where we only allow single strings as valid queries.

2.2.4 Finite Automata

A finite automaton is a mathematical model of a system. The automaton can be in any one of a finite number of states and is driven from state to state by a sequence of discrete inputs. Figure 2.1 depicts an automaton reading its input from a tape.

Figure 2.1: A finite automaton

Formally, a finite automaton (FA) is defined by a 5-tuple (Q, [image], [image] , q0, F) (see Hopcroft and Ullman [1979]), where

Q is a finite set of states,

is a finite input alphabet,

q0 [image] Q is the initial state,

F [image] Q is the set of final states, and

is the (partial) transition function mapping Q X ( [image] + { [image] }) to zero or more elements of Q. That is, [image] (q, a) describes the next state(s), for each state q and input symbol a; or is undefined.

A finite automaton starts in state q0 reading the input symbols from a tape. In one move, the FA in state q and reading symbol a enters state(s) [image] (q, a), and moves the reading head one position to the right. If [image] (q, a) [image] F, we say that the FA has accepted the string written on its input tape up to the last symbol read. If [image] (q, a) has an unique value for every q and a, we say that the FA is deterministic (DFA); otherwise we say that it is nondeterministic (NFA).

The languages accepted by finite automata (either DFAs or NFAs) are regular languages. In other words, there exists a FA that accepts L(r) for any regular expression r; and given a DFA or NFA, we can express the language that it recognizes as RE. There is a simple algorithm that, given a regular expression r, constructs a NFA that accepts L (r) in O (|r|) time and space. There are also algorithms to convert a NFA to a NFA without [image] transitions (O(|r|2) states) and to a DFA (0(2|r|) states in the worst case).

Figure 2.2 shows the DFA that searches an occurrence of the fourth query of the previous section in a text. The double circled state is the final state of the DFA. All the transitions are shown with the exception of

the transition from every state (with the exception of states 2 and 3) to state 1 upon reading a <, and

the default transition from all the states to state 0 when there is no transition defined for the read symbol.

Figure 2.2: DFA example for <A>((Sir b)? W)?bScott b? < / A>.

A DFA is called minimal if it has the minimum possible number of states. There exists an O(| [image] |n log n) algorithm to minimize a DFA with n states.

A finite automaton is called partial if the [image] function is not defined for all possible symbols of [image] for each state. In that case, there is an implicit error state belonging to F for every undefined transition.

DFA will be used in this book as searching machines. Usually, the searching time depends on how the transitions are implemented. If the alphabet is known and finite, using a table we have constant time per transition and thus O (n) searching time. If the alphabet is not known in advance, we can use an ordered table in each state. In this case, the searching time is O (n log m). Another possibility would be to use a hashing table in each state, achieving constant time per transition on average.

2.3 DATA STRUCTURES

In this section we cover three basic data structures used to organize data: search trees, digital trees, and hashing. They are used not only for storing text in secondary memory, but also as components in searching algorithms (especially digital trees). We do not describe arrays, because they are a well-known structure that can be used to implement static search tables, bit vectors for set manipulation, suffix arrays (Chapter 5), and so on.

These three data structures differ on how a search is performed. Trees define a lexicographical order over the data. However, in search trees, we use the complete value of a key to direct the search, while in digital trees, the digital (symbol) decomposition is used to direct the search. On the other hand, hashing "randomizes" the data order, being able to search faster on average, with the disadvantage that scanning in sequential order is not possible (for example, range searches are expensive).

Some examples of their use in subsequent chapters of this book are:

Search trees: for optical disk files (Chapter 6), prefix B-trees (Chapter 3), stoplists (Chapter 7).

Hashing: hashing itself (Chapter 13), string searching (Chapter 10), associated retrieval, Boolean operations (Chapters 12 and 15), optical disk file structures (Chapter 6), signature files (Chapter 4), stoplists (Chapter 7).

Digital trees: string searching (Chapter 10), suffix trees (Chapter 5).

We refer the reader to Gonnet and Baeza-Yates (1991) for search and update algorithms related to the data structures of this section.

2.3.1 Search Trees

The most well-known search tree is the binary search tree. Each internal node contains a key, and the left subtree stores all keys smaller that the parent key, while the right subtree stores all keys larger than the parent key. Binary search trees are adequate for main memory. However, for secondary memory, multiway search trees are better, because internal nodes are bigger. In particular, we describe a special class of balanced multiway search trees called B-tree.

A B-tree of order m is defined as follows:

The root has between 2 and 2m keys, while all other internal nodes have between m and 2m keys.

If ki is the i-th key of a given internal node, then all keys in the i - 1 - th child are smaller than ki, while all the keys in the i-th child are bigger.

All leaves are at the same depth.

Usually, a B-tree is used as an index, and all the associated data are stored in the leaves or buckets. This structure is called B+-tree. An example of a B+-tree of order 2 is shown in Figure 2.3, using bucket size 4.

Figure 2.3: A B+ -tree example (Di denotes the primary key i, plus its associated data).

B-trees are mainly used as a primary key access method for large databases in secondary memory. To search a given key, we go down the tree choosing the appropriate branch at each step. The number of disk accesses is equal to the height of the tree.

Updates are done bottom-up. To insert a new record, we search the insertion point. If there is not enough space in the corresponding leaf, we split it, and we promote a key to the previous level. The algorithm is applied recursively, up to the root, if necessary. In that case, the height of the tree increases by one. Splits provides a minimal storage utilization of 50 percent. Therefore, the height of the tree is at most logm+1(n/b) + 2 where n is the number of keys, and b is the number of records that can be stored in a leaf. Deletions are handled in a similar fashion, by merging nodes. On average, the expected storage utilization is ln 2 [image] .69 (Yao 1979; Baeza-Yates 1989).

To improve storage utilization, several overflow techniques exist. Some of them are:

B*-trees: in case of overflow, we first see if neighboring nodes have space. In that case, a subset of the keys is shifted, avoiding a split. With this technique, 66 percent minimal storage utilization is provided. The main disadvantage is that updates are more expensive (Bayer and McCreight 1972; Knuth 1973).

Partial expansions: buckets of different sizes are used. If an overflow occurs, a bucket is expanded (if possible), or split. Using two bucket sizes of relative ratio 2/3, 66 percent minimal and 81 percent average storage utilization is achieved (Lomet 1987; Baeza-Yates and Larson 1989). This technique does not deteriorate update time.

Adaptive splits: two bucket sizes of relative ratios 1/2 are used. However, splits are not symmetric (balanced), and they depend on the insertion point. This technique achieves 77 percent average storage utilization and is robust against nonuniform distributions (low variance) (Baeza-Yates 1990).

A special kind of B-trees, Prefix B-trees (Bayer and Unterauer 1977), supports efficiently variable length keys, as is the case with words. This kind of B-tree is discussed in detail in Chapter 3.

2.3.2 Hashing

A hashing function h (x) maps a key x to an integer in a given range (for example, 0 to m - 1). Hashing functions are designed to produce values uniformly distributed in the given range. For a good discussion about choosing hashing functions, see Ullman (1972), Knuth (1973), and Knott (1975). The hashing value is also called a signature.

A hashing function is used to map a set of keys to slots in a hashing table. If the hashing function gives the same slot for two different keys, we say that we have a collision. Hashing techniques mainly differ in how collisions are handled. There are two classes of collision resolution schemas: open addressing and overflow addressing.

In open addressing (Peterson 1957), the collided key is "rehashed" into the table, by computing a new index value. The most used technique in this class is double hashing, which uses a second hashing function (Bell and Kaman 1970; Guibas and Szemeredi 1978). The main limitation of this technique is that when the table becomes full, some kind of reorganization must be done. Figure 2.4 shows a hashing table of size 13, and the insertion of a key using the hashing function h (x) = x mod 13 (this is only an example, and we do not recommend using this hashing function!).

Figure 2.4: Insertion of a new key using double hashing.

In overflow addressing (Williams 1959; Knuth 1973), the collided key is stored in an overflow area, such that all key values with the same hashing value are linked together. The main problem of this schema is that a search may degenerate to a linear search.

Searches follow the insertion path until the given key is found, or not (unsuccessful case). The average search time is constant, for nonfull tables.

Because hashing "randomizes" the location of keys, a sequential scan in lexicographical order is not possible. Thus, ordered scanning or range searches are very expensive. More details on hashing can be found in Chapter 13.

Hashing schemes have also been used for secondary memory. The main difference is that tables have to grow dynamically as the number of keys increases. The main techniques are extendible hashing which uses hashing on two levels: a directory and a bucket level (Fagin et al. 1979), and linear hashing which uses an overflow area, and grows in a predetermined way (Litwin 1980; Larson 1980; Larson and Kajla 1984). For the case of textual databases, a special technique called signature files (Faloutsos 1987) is used most frequently. This technique is covered in detail in Chapter 4 of this book.

To improve search time on B-trees, and to allow range searches in hashing schemes, several hybrid methods have been devised. Between them, we have to mention the bounded disorder method (Litwin and Lomet 1987), where B+-tree buckets are organized as hashing tables.

2.3.3 Digital Trees

Efficient prefix searching can be done using indices. One of the best indices for prefix searching is a binary digital tree or binary trie constructed from a set of substrings of the text. This data structure is used in several algorithms.

Tries are recursive tree structures that use the digital decomposition of strings to represent a set of strings and to direct the searching. Tries were invented by de la Briandais (1959) and the name was suggested by Fredkin (1960), from information retrie val. If the alphabet is ordered, we have a lexicographically ordered tree. The root of the trie uses the first character, the children of the root use the second character, and so on. If the remaining subtrie contains only one string, that string's identity is stored in an external node.

Figure 2.5 shows a binary trie (binary alphabet) for the string "01100100010111 . . . " after inserting all the substrings that start from positions 1 through 8. (In this case, the substring's identity is represented by its starting position in the text.)

The height of a trie is the number of nodes in the longest path from the root to an external node. The length of any path from the root to an external node is bounded by the height of the trie. On average, the height of a trie is logarithmic for any square-integrable probability distribution (Devroye 1982). For a random uniform distribution (Regnier 1981), we have

for a binary trie containing n strings.

The average number of internal nodes inspected during a (un)successful search in a binary trie with n strings is log2n + O(1). The average number of internal nodes is [image] + O (n) (Knuth 1973).

A Patricia tree (Morrison 1968) is a trie with the additional constraint that single-descendant nodes are eliminated. This name is an acronym for "Practical Algorithm To Retrieve Information Coded In Alphanumerical." A counter is kept in each node to indicate which is the next bit to inspect. Figure 2.6 shows the Patricia tree corresponding to the binary trie in Figure 2.5.

Figure 2.5: Binary trie (external node label indicates position in the text) for the first eight suffixes in "01100100010111 . . .".

Figure 2.6: Patricia tree (internal node label indicates bit number).

For n strings, such an index has n external nodes (the n positions of the text) and n -1 internal nodes. Each internal node consists of a pair of pointers plus some counters. Thus, the space required is O(n).

It is possible to build the index in [image] time, where [image] denotes the height of the tree. As for tries, the expected height of a Patricia tree is logarithmic (and at most the height of the binary trie). The expected height of a Patricia tree is log2 n + o(log2 n) (Pittel 1986).

A trie built using the substrings (suffixes) of a string is also called suffix tree (McCreight [1976] or Aho et al. [1974]). A variation of these are called position trees (Weiner 1973). Similarly, a Patricia tree is called a compact suffix tree.

2.4 ALGORITHMS

It is hard to classify IR algorithms, and to draw a line between each type of application. However, we can identify three main types of algorithms, which are described below.

There are other algorithms used in IR that do not fall within our description, for example, user interface algorithms. The reason that they cannot be considered as IR algorithms is because they are inherent to any computer application.

2.4.1 Retrieval Algorithms

The main class of algorithms in IR is retrieval algorithms, that is, to extract information from a textual database. We can distinguish two types of retrieval algorithms, according to how much extra memory we need:

Sequential scanning of the text: extra memory is in the worst case a function of the query size, and not of the database size. On the other hand, the running time is at least proportional to the size of the text, for example, string searching (Chapter 10).

Indexed text: an "index" of the text is available, and can be used to speed up the search. The index size is usually proportional to the database size, and the search time is sublinear on the size of the text, for example, inverted files (Chapter 3) and signature files (Chapter 4).

Formally, we can describe a generic searching problem as follows: Given a string t (the text), a regular expression q (the query), and information (optionally) obtained by preprocessing the pattern and/or the text, the problem consists of finding whether t [image] [image] *q [image] * (q for short) and obtaining some or all of the following information:

1. The location where an occurrence (or specifically the first, the longest, etc.) of q exists. Formally, if t [image] [image] *q [image] *, find a position m [image] 0 such that t [image] [image] m q [image] *. For example, the first occurrence is defined as the least m that fulfills this condition.

2. The number of occurrences of the pattern in the text. Formally, the number of all possible values of m in the previous category.

3. All the locations where the pattern occurs (the set of all possible values of m).

In general, the complexities of these problems are different.

We assume that [image] is not a member of L(q). If it is, the answer is trivial. Note that string matching is a particular case where q is a string. Algorithms to solve this problem are discussed in Chapter 10.

The efficiency of retrieval algorithms is very important, because we expect them to solve on-line queries with a short answer time. This need has triggered the implementation of retrieval algorithms in many different ways: by hardware, by parallel machines, and so on. These cases are explained in detail in Chapter 17 (algorithms by hardware) and Chapter 18 (parallel algorithms).

2.4.2 Filtering Algorithms

This class of algorithms is such that the text is the input and a processed or filtered version of the text is the output. This is a typical transformation in IR, for example to reduce the size of a text, and/or standardize it to simplify searching.

The most common filtering/processing operations are:

Common words removed using a list of stopwords. This operation is discussed in Chapter 7.

Uppercase letters transformed to lowercase letters.

Special symbols removed and sequences of multiple spaces reduced to one space.

Numbers and dates transformed to a standard format (Gonnet 1987).

Spelling variants transformed using Soundex-like methods (Knuth 1973).

Word stemming (removing suffixes and/or prefixes). This is the topic of Chapter 8.

Automatic keyword extraction.

Word ranking.

Unfortunately, these filtering operations may also have some disadvantages. Any query, before consulting the database, must be filtered as is the text; and, it is not possible to search for common words, special symbols, or uppercase letters, nor to distinguish text fragments that have been mapped to the same internal form.

2.4.3 Indexing Algorithms

The usual meaning of indexing is to build a data structure that will allow quick searching of the text, as we mentioned previously. There are many classes of indices, based on different retrieval approaches. For example, we have inverted files (Chapter 3), signature files (Chapter 4), tries (Chapter 5), and so on, as we have seen in the previous section. Almost all type of indices are based on some kind of tree or hashing. Perhaps the main exceptions are clustered data structures (this kind of indexing is called clustering), which is covered in Chapter 16, and the Direct Acyclic Word Graph (DAWG) of the text, which represents all possible subwords of the text using a linear amount of space (Blumer et al. 1985), and is based on finite automata theory.

Usually, before indexing, the text is filtered. Figure 2.7 shows the complete process for the text.

Figure 2.7: Text preprocessing

The preprocessing time needed to build the index is amortized by using it in searches. For example, if building the index requires O (n log n) time, we would expect to query the database at least O (n) times to amortize the preprocessing cost. In that case, we add O (log n) preprocessing time to the total query time (that may also be logarithmic).

Many special indices, and their building algorithms (some of them in parallel machines), are covered in this book.

REFERENCES

AHO, A., J. HOPCROFT, and J. ULLMAN. 1974. The Design and Analysis of Computer Algorithms. Reading, Mass.: Addison-Wesley.

BAEZA-YATES, R. 1989. "Expected Behaviour of B+-Trees under Random Insertions." Acta Informatica, 26(5), 439-72. Also as Research Report CS-86-67, University of Waterloo, 1986.

BAEZA-YATES, R. 1990. "An Adaptive Overflow Technique for the B-tree," in Extending Data Base Technology Conference (EDBT 90), eds. F. Bancilhon, C. Thanos and D. Tsichritzis, pp. 16-28, Venice. Springer Verlag Lecture Notes in Computer Science 416.

BAEZA-YATES, R., and P.-A. LARSON. 1989. "Performance of B+-trees with Partial Expansions." IEEE Trans. on Knowledge and Data Engineering, 1, 248-57. Also as Research Report CS-87-04, Dept. of Computer Science, University of Waterloo, 1987.

BAYER, R., and E. MCCREIGHT. 1972. "Organization and Maintenance of Large Ordered Indexes."Acta Informatica, 1(3), 173-89.

BAYER, R., and K. UNTERAUER. 1977. "Prefix B-trees." ACM TODS, 2(1), 11-26.

BELL, J., and C. KAMAN. 1970. "The Linear Quotient Hash Code." CACM, 13(11), 675-77.

BLUMER, A., J. BLUMER, D. HAUSSLER, A. EHRENFEUCHT, M. CHEN, and J. SEIFERAS. 1985. "The Smallest Automaton Recognizing the Subwords of a Text." Theoretical Computer Science, 40, 31-55.

DE LA BRIANDAIS, R. 1959. "File Searching Using Variable Length Keys, in AFIPS Western JCC, pp. 295-98, San Francisco, Calif.

DEVROYE, L. 1982. "A Note on the Average Depth of Tries." Computing, 28, 367-71.

FAGIN, R., J. NIEVERGELT, N. PIPPENGER, and H. STRONG. 1979. "Extendible Hashing--a Fast Access Method for Dynamic Files. ACM TODS, 4(3), 315-44.

FALOUTSOS, C. 1987. Signature Files: An Integrated Access Method for Text and Attributes, Suitable for Optical Disk Storage." Technical Report CS-TR-1867, University of Maryland.

FREDKIN, E. 1960. "Trie Memory." CACM, 3, 490-99.

GONNET, G. 1987. "Extracting Information from a Text Database: An Example with Dates and Numerical Data," in Third Annual Conference of the UW Centre for the New Oxford English Dictionary, pp. 85-89, Waterloo, Canada.

GONNET, G. and R. BAEZA-YATES. 1991. Handbook of Algorithms and Data Structures--In Pascal and C. (2nd ed.). Wokingham, U.K.: Addison-Wesley.

GUIBAS, L., and E. SZEMEREDI. 1978. "The Analysis of Double Hashing." JCSS, 16(2), 226-74.

HOPCROFT, J., and J. ULLMAN. 1979. Introduction to Automata Theory. Reading, Mass.: Addison-Wesley.

KNOTT, G. D. 1975. "Hashing Functions." Computer Journal, 18(3), 265-78.

KNUTH, D. 1973. The Art of Computer Programming: Sorting and Searching, vol. 3. Reading, Mass.: Addison-Wesley.

LARSON, P.-A. 1980. "Linear Hashing with Partial Expansions," in VLDB, vol. 6, pp. 224-32, Montreal.

LARSON, P.-A., and A. KAJLA. 1984. "File Organization: Implementation of a Method Guaranteeing Retrieval in One Access." CACM, 27(7), 670-77.

LITWlN, W. 1980. "Linear Hashing: A New Tool for File and Table Addressing," in VLDB, vol. 6, pp. 212-23, Montreal.

LITWIN, W., and LOMET, D. 1987. "A New Method for Fast Data Searches with Keys. IEEE Software, 4(2), 16-24.

LOMET, D. 1987. "Partial Expansions for File Organizations with an Index. ACM TODS, 12: 65-84. Also as tech report, Wang Institute, TR-86-06, 1986.

MCCREIGHT, E. 1976. "A Space-Economical Suffix Tree Construction Algorithm." JACM, 23, 262-72.

MORRlSON, D. 1968. "PATRlClA-Practical Algorithm to Retrieve Information Coded in Alphanumeric." JACM, 15, 514-34.

PETERSON, W. 1957. "Addressing for Random-Access Storage. IBM J Res. Development, 1(4), 130-46.

PITTEL, B. 1986. "Paths in a Random Digital Tree: Limiting Distributions." Adv. Appl. Prob., 18, 139-55.

REGNIER, M. 1981. "On the Average Height of Trees in Digital Search and Dynamic Hashing." Inf. Proc. Letters, 13, 64-66.

ULLMAN, J. 1972. "A Note on the Efficiency of Hashing Functions." JACM, 19(3), 569-75.

WEINER, P. 1973. "Linear Pattern Matching Algorithm," in FOCS, vol. 14, pp. 1-11.

WILLIAMS, F. 1959. "Handling Identifiers as Internal Symbols in Language Processors." CACM, 2(6), 21-24.

YAO, A. 1979. "The Complexity of Pattern Matching for a Random String." SIAM J. Computing, 8, 368-87.

Go to Chapter 3 Back to Table of Contents


Information Retrieval: CHAPTER 3: INVERTED FILES

 Return to Table of Contents

CHAPTER 3: INVERTED FILES

Donna Harman

National Institute of Standards and Technology, Gaithersburg, MD 20899

Edward Fox

Virginia Polytechnic Institute and State University, Department of Computer Science, Blacksburg, VA 24061-0106

R. Baeza-Yates

Depto. de Ciencias de la Computación, Universidad de Chile, Casilla 2777, Santiago, Chile

W. Lee

Virginia Polytechnic Institute and State University, Department of Computer Science, Blacksburg, VA 24061-0106

Abstract

This chapter presents a survey of the various structures (techniques) that can be used in building inverted files, and gives the details for producing an inverted file using sorted arrays. The chapter ends with two modifications to this basic method that are effective for large data collections.

3.1 INTRODUCTION

Three of the most commonly used file structures for information retrieval can be classified as lexicographical indices (indices that are sorted), clustered file structures, and indices based on hashing. One type of lexicographical index, the inverted file, is presented in this chapter, with a second type of lexicographical index, the Patricia (PAT) tree, discussed in Chapter 5. Clustered file structures are covered in Chapter 16, and indices based on hashing are covered in Chapter 13 and Chapter 4 (signature files).

The concept of the inverted file type of index is as follows. Assume a set of documents. Each document is assigned a list of keywords or attributes, with optional relevance weights associated with each keyword (attribute). An inverted file is then the sorted list (or index) of keywords (attributes), with each keyword having links to the documents containing that keyword (see Figure 3.1) . This is the kind of index found in most commercial library systems. The use of an inverted file improves search efficiency by several orders of magnitude, a necessity for very large text files. The penalty paid for this efficiency is the need to store a data structure that ranges from 10 percent to 100 percent or more of the size of the text itself, and a need to update that index as the data set changes.

Figure 3.1: An inverted file implemented using a sorted array

Usually there are some restrictions imposed on these indices and consequently on later searches. Examples of these restrictions are:

a controlled vocabulary which is the collection of keywords that will be indexed. Words in the text that are not in the vocabulary will not be indexed, and hence are not searchable.

a list of stopwords (articles, prepositions, etc.) that for reasons of volume or precision and recall will not be included in the index, and hence are not searchable.

a set of rules that decide the beginning of a word or a piece of text that is indexable. These rules deal with the treatment of spaces, punctuation marks, or some standard prefixes, and may have signficant impact on what terms are indexed.

a list of character sequences to be indexed (or not indexed). In large text databases, not all character sequences are indexed; for example, character sequences consisting of all numerics are often not indexed.

It should be noted that the restrictions that determine what is to be indexed are critical to later search effectiveness and therefore these rules should be carefully constructed and evaluated. This problem is further discussed in Chapter 7.

A search in an inverted file is the composition of two searching algorithms; a search for a keyword (attribute), which returns an index, and then a possible search on that index for a particular attribute value. The result of a search on an inverted file is a set of records (or pointers to records).

This Chapter is organized as follows. The next section presents a survey of the various implementation structures for inverted files. The third section covers the complete implementation of an algorithm for building an inverted file that is stored as a sorted array, and the fourth section shows two variations on this implementation, one that uses no sorting (and hence needs little working storage) and one that increases efficiency by making extensive use of primary memory. The final section summarizes the chapter.

3.2 STRUCTURES USED IN INVERTED FILES

There are several structures that can be used in implementing inverted files: sorted arrays, B-trees, tries, and various hashing structures, or combinations of these structures. The first three of these structures are sorted (lexicographically) indices, and can efficiently support range queries, such as all documents having keywords that start with "comput." Only these three structures will be further discussed in this chapter. (For more on hashing methods, see Chapters 4 and 13.)

3.2.1 The Sorted Array

An inverted file implemented as a sorted array structure stores the list of keywords in a sorted array, including the number of documents associated with each keyword and a link to the documents containing that keyword. This array is commonly searched using a standard binary search, although large secondary-storage-based systems will often adapt the array (and its search) to the characteristics of their secondary storage.

The main disadvantage of this approach is that updating the index (for example appending a new keyword) is expensive. On the other hand, sorted arrays are easy to implement and are reasonably fast. (For this reason, the details of creating a sorted array inverted file are given in section 3.3.)

3.2.2 B-trees

Another implementation structure for an inverted file is a B-tree. More details of B-trees can be found in Chapter 2, and also in a recent paper (Cutting and Pedersen 1990) on efficient inverted files for dynamic data (data that is heavily updated). A special case of the B-tree, the prefix B-tree, uses prefixes of words as primary keys in a B-tree index (Bayer and Unterauer 1977) and is particularly suitable for storage of textual indices. Each internal node has a variable number of keys. Each key is the shortest word (in length) that distinguishes the keys stored in the next level. The key does not need to be a prefix of an actual term in the index. The last level or leaf level stores the keywords themselves, along with their associated data (see Figure 3.2). Because the internal node keys and their lengths depend on the set of keywords, the order (size) of each node of the prefix B-tree is variable. Updates are done similarly to those for a B-tree to maintain a balanced tree. The prefix B-tree method breaks down if there are many words with the same (long) prefix. In this case, common prefixes should be further divided to avoid wasting space.

Figure 3.2: A prefix B-tree

Compared with sorted arrays, B-trees use more space. However, updates are much easier and the search time is generally faster, especially if secondary storage is used for the inverted file (instead of memory). The implementation of inverted files using B-trees is more complex than using sorted arrays, and therefore readers are referred to Knuth (1973) and Cutting and Pedersen (1990) for details of implementation of B-trees, and to Bayer and Unterauer (1977) for details of implementation of prefix B-trees.

3.2.3 Tries

Inverted files can also be implemented using a trie structure (see Chapter 2 for more on tries). This structure uses the digital decomposition of the set of keywords to represent those keywords. A special trie structure, the Patricia (PAT) tree, is especially useful in information retrieval and is described in detail in Chapter 5. An additional source for tested and optimized code for B-trees and tries is Gonnet and Baeza-Yates (1991).

3.3 BUILDING AN INVERTED FILE USING A SORTED ARRAY

The production of sorted array inverted files can be divided into two or three sequential steps as shown in Figure 3.3. First, the input text must be parsed into a list of words along with their location in the text. This is usually the most time consuming and storage consuming operation in indexing. Second, this list must then be inverted, from a list of terms in location order to a list of terms ordered for use in searching (sorted into alphabetical order, with a list of all locations attached to each term). An optional third step is the postprocessing of these inverted files, such as for adding term weights, or for reorganizing or compressing the files.

Figure 3.3: Overall schematic of sorted array inverted file creation

Creating the initial word list requires several different operations. First, the individual words must be recognized from the text. Each word is then checked against a stoplist of common words, and if it can be considered a noncommon word, may be passed through a stemming algorithm. The resultant stem is then recorded in the word-within-location list. The parsing operation and the use of a stoplist are described in Chapter 7, and the stemming operation is described in Chapter 8.

The word list resulting from the parsing operation (typically stored as a disk file) is then inverted. This is usually done by sorting on the word (or stem), with duplicates retained (see Figure 3.4). Even with the use of high-speed sorting utilities, however, this sort can be time consuming for large data sets (on the order of n log n). One way to handle this problem is to break the data sets into smaller pieces, process each piece, and then correctly merge the results. Methods that do not use sorting are given in section 3.4.1. After sorting, the duplicates are merged to produce within-document frequency statistics. (A system not using within-document frequencies can just sort with duplicates removed.) Note that although only record numbers are shown as locations in Figure 3.4, typically inverted files store field locations and possibly even word location. These additional locations are needed for field and proximity searching in Boolean operations and cause higher inverted file storage overhead than if only record location was needed. Inverted files for ranking retrieval systems (see Chapter 14) usually store only record locations and term weights or frequencies.

Figure 3.4: Inversion of word list

Although an inverted file could be used directly by the search routine, it is usually processed into an improved final format. This format is based on the search methods and the (optional) weighting methods used. A common search technique is to use a binary search routine on the file to locate the query words. This implies that the file to be searched should be as short as possible, and for this reason the single file shown containing the terms, locations, and (possibly) frequencies is usually split into two pieces. The first piece is the dictionary containing the term, statistics about that term such as number of postings, and a pointer to the location of the postings file for that term. The second piece is the postings file itself, which contains the record numbers (plus other necessary location information) and the (optional) weights for all occurrences of the term. In this manner, the dictionary used in the binary search has only one "line" per unique term. Figure 3.5 illustrates the conceptual form of the necessary files; the actual form depends on the details of the search routine and on the hardware being used. Work using large data sets (Harman and Candela 1990) showed that for a file of 2,653 records, there were 5,123 unique terms with an average of 14 postings/term and a maximum of over 2,000 postings for a term. A larger data set of 38,304 records had dictionaries on the order of 250,000 lines (250,000 unique terms, including some numbers) and an average of 88 postings per record. From these numbers it is clear that efficient storage structures for both the binary search and the reading of the postings are critical.

Figure 3.5: Dictionary and postings file from the last example

3.4 MODIFICATIONS TO THE BASIC TECHNIQUE

Two different techniques are presented as improvements on the basic inverted file creation discussed in section 3.3. The first technique is for working with very large data sets using secondary storage. The second technique uses multiple memory loads for inverting files.

3.4.1 Producing an Inverted File for Large Data Sets without Sorting

Indexing large data sets using the basic inverted file method presents several problems. Most computers cannot sort the very large disk files needed to hold the initial word list within a reasonable time frame, and do not have the amount of storage necessary to hold a sorted and unsorted version of that word list, plus the intermediate files involved in the internal sort. Whereas the data set could be broken into smaller pieces for processing, and the resulting files properly merged, the following technique may be considerably faster. For small data sets, this technique carries a significant overhead and therefore should not be used. (For another approach to sorting large amounts of data, see Chapter 5.)

The new indexing method (Harman and Candela 1990) is a two-step process that does not need the middle sorting step. The first step produces the initial inverted file, and the second step adds the term weights to that file and reorganizes the file for maximum efficiency (see Figure 3.6).

Figure 3.6: Flowchart of new indexing method

The creation of the initial inverted file avoids the use of an explicit sort by using a right-threaded binary tree (Knuth 1973). The data contained in each binary tree node is the current number of term postings and the storage location of the postings list for that term. As each term is identified by the text parsing program, it is looked up in the binary tree, and either is added to the tree, along with related data, or causes tree data to be updated. The postings are stored as multiple linked lists, one variable length linked list for each term, with the lists stored in one large file. Each element in the linked postings file consists of a record number (the location of a given term), the term frequency in that record, and a pointer to the next element in the linked list for that given term. By storing the postings in a single file, no storage is wasted, and the files are easily accessed by following the links. As the location of both the head and tail of each linked list is stored in the binary tree, the entire list does not need to be read for each addition, but only once for use in creating the final postings file (step two).

Note that both the binary tree and the linked postings list are capable of further growth. This is important in indexing large data sets where data is usually processed from multiple separate files over a short period of time. The use of the binary tree and linked postings list could be considered as an updatable inverted file. Although these structures are not as efficient to search, this method could be used for creating and storing supplemental indices for use between updates to the primary index. However, see the earlier discussion of B-trees for better ways of producing updatable inverted files.

The binary tree and linked postings lists are saved for use by the term weighting routine (step two). This routine walks the binary tree and the linked postings list to create an alphabetical term list (dictionary) and a sequentially stored postings file. To do this, each term is consecutively read from the binary tree (this automatically puts the list in alphabetical order), along with its related data. A new sequentially stored postings file is allocated, with two elements per posting. The linked postings list is then traversed, with the frequencies being used to calculate the term weights (if desired). The last step writes the record numbers and corresponding term weights to the newly created sequential postings file. These sequentially stored postings files could not be created in step one because the number of postings is unknown at that point in processing, and input order is text order, not inverted file order. The final index files therefore consist of the same dictionary and sequential postings file as for the basic inverted file described in section 3.3.

Table 3.1 gives some statistics showing the differences between an older indexing scheme and the new indexing schemes. The old indexing scheme refers to the indexing method discussed in section 3.3 in which records are parsed into a list of words within record locations, the list is inverted by sorting, and finally the term weights are added.

Table 3.1: Indexing Statistics

 Text Size   Indexing Time  Working Storage  Index Storage

(megabytes)     (hours)       (megabytes)     (megabytes)

----------------------------------------------------------

               old   new      old   new       old  new

   1.6        0.25  0.50      4.0  0.70       0.4  0.4

   50           8   10.5      132    6         4     4

   359         --    137       --     70        52   52

   806         --    313       --    163       112  112

Note that the size of the final index is small, only 8 percent of the input text size for the 50 megabyte database, and around 14 percent of the input text size for the larger databases. The small size of the final index is caused by storing only the record identification number as location. As this index was built for a ranking retrieval system (see Chapter 14), each posting contains both a record id number and the term's weight in that record. This size remains constant when using the new indexing method as the format of the final indexing files is unchanged. The working storage (the storage needed to build the index) for the new indexing method is not much larger than the size of the final index itself, and substantially smaller than the size of the input text. However, the amount of working storage needed by the older indexing method would have been approximately 933 megabytes for the 359 megabyte database, and over 2 gigabytes for the 806 megabyte database, an amount of storage beyond the capacity of many environments. The new method takes more time for the very small (1.6 megabyte) database because of its additional processing overhead. As the size of the database increases, however, the processing time has an n log n relationship to the size of the database. The older method contains a sort (not optimal) which is n log n (best case) to n squared (worst case), making processing of the very large databases likely to have taken longer using this method.

3.4.2 A Fast Inversion Algorithm

The second technique to produce a sorted array inverted file is a fast inversion algorithm called FAST-INV (Copyright © Edward A. Fox, Whay C. Lee, Virginia Tech). This technique takes advantage of two principles: the large primary memories available on today's computers and the inherent order of the input data. The following summary of this technique is adapted from a technical report by Fox and Lee (1991).

The first principle is important since personal computers with more than 1 megabyte of primary memory are common, and mainframes may have more than 100 megabytes of memory. Even if databases are on the order of 1 gigabyte, if they can be split into memory loads that can be rapidly processed and then combined, the overall cost will be minimized.

The second principle is crucial since with large files it is very expensive to use polynomial or even n log n sorting algorithms. These costs are further compounded if memory is not used, since then the cost is for disk operations.

The FAST-INV algorithm follows these two principles, using primary memory in a close to optimal fashion, and processing the data in three passes. The overall scheme can be seen in Figure 3.7.

The input to FAST-INV is a document vector file containing the concept vectors for each document in the collection to be indexed. A sample document vector file can be seen in Figure 3.8. The document numbers appear in the left-hand column and the concept numbers of the words in each document appear in the right- hand column. This is similar to the initial word list shown in Figure 3.4 for the basic method, except that the words are represented by concept numbers, one concept number for each unique word in the collection (i.e., 250,000 unique words implies 250,000 unique concept numbers). Note however that the document vector file is in sorted order, so that concept numbers are sorted within document numbers, and document numbers are sorted within collection. This is necessary for FAST-INV to work correctly.

Figure 3.7: Overall scheme of FAST-INV

Figure 3.8: Sample document vector

Preparation

In order to better explain the FAST-INV algorithm, some definitions are needed.

HCN = highest concept number in dictionary

L = number of document/concept (or concept/document) pairs in the collection

M = available primary memory size, in bytes

In the first pass, the entire document vector file can be read and two new files produced: a file of concept postings/pointers (CONPTR) and a load table. It is assumed that M >> HCN, so these two files can be built in primary memory. However, it is assumed that M < L, so several primary memory loads will be needed to process the document data. Because entries in the document vector file will already be grouped by concept number, with those concepts in ascending order, it is appropriate to see if this data can be somehow transformed beforehand into j parts such that:

L / j < M, so that each part will fit into primary memory

HCN / j concepts, approximately, are associated with each part

This allows each of the j parts to be read into primary memory, inverted there, and the output to be simply appended to the (partially built) final inverted file.

Specifically, preparation involves the following:

1. Allocate an array, con_entries_cnt, of size HCN, initialized to zero.

2. For each <doc#,con#> entry in the document vector file: increment con_entries_cnt[con#] .

3. Use the just constructed con_entries_cnt to create a disk version of CONPTR.

4. Initialize the load table.

5. For each <con#,count> pair obtained from con_entries_cnt: if there is no room for documents with this concept to fit in the current load, then create an entry in the load table and initialize the next load entry; otherwise update information for the current load table entry.

After one pass through the input, the CONPTR file has been built and the load table needed in later steps of the algorithm has been constructed. Note that the test for room in a given load enforces the constraint that data for a load will fit into available memory. Specifically:

Let LL = length of current load (i.e., number of concept/weight pairs)

S = spread of concept numbers in the current load (i.e., end concept - start concept + 1 )

8 bytes = space needed for each concept/weight pair

4 bytes = space needed for each concept to store count of postings for it

Then the constraint that must be met for another concept to be added to the current load is

8 * LL + 4 * S < M

Splitting document vector file

The load table indicates the range of concepts that should be processed for each primary memory load. There are two approaches to handling the multiplicity of loads. One approach, which is currently used, is to make a pass through the document vector file to obtain the input for each load. This has the advantage of not requiring additional storage space (though that can be obviated through the use of magnetic tapes), but has the disadvantage of requiring expensive disk I/O.

The second approach is to build a new copy of the document vector collection, with the desired separation into loads. This can easily be done using the load table, since sizes of each load are known, in one pass through the input. As each document vector is read, it is separated into parts for each range of concepts in the load table, and those parts are appended to the end of the corresponding section of the output document collection file. With I/O buffering, the expense of this operation is proportional to the size of the files, and essentially costs the same as copying the file.

Inverting each load

When a load is to be processed, the appropriate section of the CONPTR file is needed. An output array of size equal to the input document vector file subset is needed. As each document vector is processed, the offset (previously recorded in CONPTR) for a given concept is used to place the corresponding document/weight entry, and then that offset is incremented. Thus, the CONPTR data allows the input to be directly mapped to the output, without any sorting. At the end of the input load the newly constructed output is appended to the inverted file.

An example

Figure 3.9 illustrates the FAST-INV processing using sample data. The document vector input files are read through once to produce the concept postings/pointers file (stored on disk as CONPTR) and the load table. Three loads will be needed, for concepts in ranges 1-4, 5-11, and 12-14. There are 10 distinct concepts, and HCN is 14.

Figure 3.9: A FAST-INV example

The second phase of processing uses the load table to split the input document vector files and create the document vector loads files. There are three parts, corresponding to the three loads. It can be seen that the document vectors in each load are shortened since only those concepts in the allowable range for the load are included.

The final phase of processing involves inverting each part of the document vectors loads file, using primary memory, and appending the result to the inverted file. The appropriate section of the CONPTR file is used so that inversion is simply a copying of data to the correct place, rather than a sort.

Performance results

Based on the discussion above, it can be seen that FAST-INV is a linear algorithm in the size of the input file, L. The input disk files must be read three times, and written twice (using the second splitting scheme). Processing in primary memory is limited to scans through the input, with appropriate (inexpensive) computation required on each entry.

Thus, it should be obvious that FAST-INV should perform well in comparison to other inversion methods, such as those used in other systems such as SIRE and SMART. To demonstrate this fact, a variety of tests were made. Table 3.2 summarizes the results for indexing the 12,684 document/8.68 megabyte INSPEC collection.

Table 3.2: Real Time Requirements (min.)[:sec.] for SIRE, SMART

Method    Comments                Indexing  Inversion

--------------------------------------------------------------

SIRE      Dictionary built during inversion        35     72

SMART     Dictionary built during indexing         49     11

FAST-INV  Dictionary built during indexing         49     1:14

More details of these results, including the various loading experiments made during testing, can be found in the full technical report (Fox and Lee 1991).

3.5 SUMMARY

This Chapter has presented a survey of inverted file structures and detailed the actual building of a basic sorted array type of inverted file. Two modifications that are effective for indexing large data collections were also presented.

REFERENCES

BAYER, R., and K. UNTERAUER. 1977. "Prefix B-Trees." ACM Transactions on Database Systems, 2(1), 11-26.

CUTTING, D., and J. PEDERSEN. 1990. "Optimizations for Dynamic Inverted Index Maintenance." Paper presented at the 13th International Conference on Research and Development in Information Retrieval. Brussels, Belgium.

FOX, E. A., and W. C. LEE. 1991. "FAST-INV: A Fast Algorithm for Building Large Inverted Files." Technical Report TR-91-10, VPI&SU Department of Computer Science, March, 1991. Blacksburg, Va. 24061-0106.

GONNET, G. H., and R. BAEZA-YATES. 1991. Handbook of Algorithms and Data Structures (2nd ed.). Reading, Mass.: Addison-Wesley.

HARMAN, D., and G. CANDELA. 1990. "Retrieving Records from a Gigabyte of Text on a Minicomputer using Statistical Ranking." Journal of the American Society for Information Science, 41(8), 581-89.

KNUTH, DONALD E. 1973. The Art of Computer Programming. Reading, Mass.: Addison-Wesley.

Go to Chapter 4 Back to Table of Contents


Information Retrieval: CHAPTER 4: SIGNATURE FILES

 Return to Table of Contents

CHAPTER 4: SIGNATURE FILES

Christos Faloutsos

University of Maryland, College Park and UMIACS

Abstract

This chapter presents a survey and discussion on signature-based text retrieval methods. It describes the main idea behind the signature approach and its advantages over other text retrieval methods, it provides a classification of the signature methods that have appeared in the literature, it describes the main representatives of each class, together with the relative advantages and drawbacks, and it gives a list of applications as well as commercial or university prototypes that use the signature approach.

4.1 INTRODUCTION

Text retrieval methods have attracted much interest recently (Christodoulakis et al. 1986; Gonnet 1982; Gonnet and Tompa 1987; Stanfill and Kahle 1986; Tsichritzis and Christodoulakis 1983). There are numerous applications involving storage and retrieval of textual data:

Electronic office filing (Tsichritzis and Christodoulakis 1983; Christodoulakis et al. 1986).

Computerized libraries. For example, the U.S. Library of Congress has been pursuing the "Optical Disk Pilot Program" (Price 1984; Nofel 1986), where the goal is to digitize the documents and store them on an optical disk. A similar project is carried out at the National Library of Medicine (Thoma et al. 1985).

Automated law (Hollaar 1979) and patent offices (Hollaar et al. 1983). The U.S. Patent and Trademark Office has been examining electronic storage and retrieval of the recent patents on a system of 200 optical disks.

Electronic storage and retrieval of articles from newspapers and magazines.

Consumers' databases, which contain descriptions of products in natural language.

Electronic encyclopedias (Ewing et al. 1986; Gonnet and Tompa 1987).

Indexing of software components to enhance reusabililty (Standish 1984).

Searching databases with descriptions of DNA molecules (Lipman and Pearson 1985).

Searching image databases, where the images are manually annotated (Christodoulakis et al. 1986). A similar approach could be used to search a database with animations, if scripted animation is used (Lewis 1989).

The main operational characteristics of all the above applications are the following two:

Text databases are traditionally large.

Text databases have archival nature: there are insertions in them, but almost never deletions and updates.

Test retrieval methods form the following large classes (Faloutsos 1985): Full text scanning, inversion, and signature files, on which we shall focus next.

Signature files are based on the idea of the inexact filter: They provide a quick test, which discards many of the nonqualifying items. The qualifying items definitely pass the test; some additional items ("false hits" or "false drops") may also pass it accidentally. The signature file approach works as follows: The documents are stored sequentially in the "text file." Their "signatures" (hash-coded bit patterns) are stored in the "signature file." When a query arrives, the signature file is scanned and many nonqualifying documents are discarded. The rest are either checked (so that the "false drops" are discarded) or they are returned to the user as they are.

A brief, qualitative comparison of the signature-based methods versus their competitors is as follows: The signature-based methods are much faster than full text scanning (1 or 2 orders of magnitude faster, depending on the individual method). Compared to inversion, they require a modest space overhead (typically [image] 10-15% [Christodoulakis and Faloutsos 1984], as opposed to 50-300% that inversion requires [Haskin 1981]); moreover, they can handle insertions more easily than inversion, because they need "append-only" operations--no reorganization or rewriting of any portion of the signatures. Methods requiring "append-only" insertions have the following advantages: (a) increased concurrency during insertions (the readers may continue consulting the old portion of index structure, while an insertion takes place) (b) these methods work well on Write-Once-Read-Many (WORM) optical disks, which constitute an excellent archival medium (Fujitani 1984; Christodoulakis 1987).

On the other hand, signature files may be slow for large databases, precisely because their response time is linear on the number of items N in the database. Thus, signature files have been used in the following environments:

1. PC-based, medium size db

2. WORMs

3. parallel machines

4. distributed text db

This chapter is organized as follows: In section 4.2 we present the basic concepts in signature files and superimposed coding. In section 4.3 we discuss methods based on compression. In section 4.4 we discuss methods based on vertical partitioning of the signature file. In section 4.5 we discuss methods that use both vertical partitioning and compression. In section 4.6 we present methods that are based on horizontal partitioning of the signature file. In section 4.7 we give the conclusions.

4.2 BASIC CONCEPTS

Signature files typically use superimposed coding (Moders 1949) to create the signature of a document. A brief description of the method follows; more details are in Faloutsos (1985). For performance reasons, which will be explained later, each document is divided into "logical blocks," that is, pieces of text that contain a constant number D of distinct, noncommon words. (To improve the space overhead, a stoplist of common words is maintainted.) Each such word yields a "word signature," which is a bit pattern of size F, with m bits set to "1", while the rest are "0" (see Figure 4.1). F and m are design parameters. The word signatures are OR'ed together to form the block signature. Block signatures are concatenated, to form the document signature. The m bit positions to be set to "1" by each word are decided by hash functions. Searching for a word is handled by creating the signature of the word and by examining each block signature for "1" 's in those bit positions that the signature of the search word has a "1".

Word             Signature

-----------------------------------

free             001  000  110  010

text             000  010  101  001

-----------------------------------

block signature  001  010  111  011

Figure 4.1: Illustration of the superimposed coding method. It is assumed that each logical block consists of D=2 words only. The signature size F is 12 bits, m=4 bits per word.

In order to allow searching for parts of words, the following method has been suggested (Faloutsos and Christodoulakis 1984): Each word is divided into successive, overlapping triplets (e.g., "fr", "fre", "ree", "ee" for the word "free"). Each such triplet is hashed to a bit position by applying a hashing function on a numerical encoding of the triplet, for example, considering the triplet as a base-26 number. In the case of a word that has l triplets, with l > m, the word is allowed to set l (nondistinct) bits. If l < m, the additional bits are set using a random number generator, initialized with a numerical encoding of the word.

An important concept in signature files is the false drop probability Fd. Intuitively, it gives the probability that the signature test will fail, creating a "false alarm" (or "false hit" or "false drop"). Notice that the signature test never gives a false dismissal.

DEFINITION: False drop probability, Fd, is the probability that a block signature seems to qualify, given that the block does not actually qualify. Expressed mathematically:

Fd = Prob{signature qualifies/block does not}

The signature file is an F [image] N binary matrix. Previous analysis showed that, for a given value of F, the value of m that minimizes the false drop probability is such that each row of the matrix contains "1" 's with probability 50 percent. Under such an optimal design, we have (Stiassny 1960):

Fd = 2-m

F1n2 = mD

This is the reason that documents have to be divided into logical blocks: Without logical blocks, a long document would have a signature full of "l" 's, and it would always create a false drop. To avoid unnecessary complications, for the rest of the discussion we assume that all the documents span exactly one logical block.

Table 4.1:. Symbols and definitions

Symbol                     Definition

--------------------------------------------------------

   F     signature size in bits

   m     number of bits per word

   D     number of distinct noncommon words per document

   Fd    false drop probability

The most straightforward way to store the signature matrix is to store the rows sequentially. For the rest of this work, the above method will be called SSF, for Sequential Signature File. Figure 4.2 illustrates the file structure used: In addition to the text file and the signature file, we need the so-called "pointer file," with pointers to the beginnings of the logical blocks (or, alternatively, to the beginning of the documents).

Figure 4.2: File structure for SSF

Although SSF has been used as is, it may be slow for large databases. Many methods have been suggested, trying to improve the response time of SSF, trading off space or insertion simplicity for speed. The main ideas behind all these methods are the following:

1. Compression: If the signature matrix is deliberately sparse, it can be compressed.

2. Vertical partitioning: Storing the signature matrix columnwise improves the response time on the expense of insertion time.

3. Horizontal partitioning: Grouping similar signatures together and/or providing an index on the signature matrix may result in better-than-linear search.

The methods we shall examine form the classes shown in Figure 4.3. For each of these classes we shall describe the main idea, the main representatives, and the available performance results, discussing mainly

the storage overhead,

the response time on single word queries,

the performance on insertion, as well as whether the insertion maintains the "append-only" property.

     Sequential storage of the signature matrix

          without compression

               sequential signature files (SSF)

          with compression

               bit-block compression (BC)

               variable bit-block compression (VBC)

     Vertical partitioning

          without compression

               bit-sliced signature files (BSSF, B'SSF))

               frame sliced (FSSF)

               generalized frame-sliced (GFSSF)

          with compression

               compressed bit slices (CBS)

               doubly compressed bit slices (DCBS)

               no-false-drop method (NFD)

     Horizontal partitioning

          data independent partitioning

               Gustafson's method

               partitioned signature files

          data dependent partitioning

               2-level signature files

               S-trees

Figure 4.3: Classification of the signature-based methods

4.3 COMPRESSION

In this section we examine a family of methods suggested in Faloutsos and Christodoulakis (1987). These methods create sparse document signatures on purpose, and then compress them before storing them sequentially. Analysis in that paper showed that, whenever compression is applied, the best value for m is 1. Also, it was shown that the resulting methods achieve better false drop probability than SSF for the same space overhead.

The idea in these methods is that we use a (large) bit vector of B bits and we hash each word into one (or perhaps more, say n) bit position(s), which are set to "1" (see Figure 4.4). The resulting bit vector will be sparse and therefore it can be compressed.

data        0000  0000  0000  0010  0000

base        0000  0001  0000  0000  0000

management  0000  1000  0000  0000  0000

system      0000  0000  0000  0000  1000

----------------------------------------

block

signature   0000  1001  0000  0010  1000

Figure 4.4: Illustration of the compression-based methods. With B = 20 and n = 1 bit per word, the resulting bit vector is sparse and can be compressed.

The spacewise best compression method is based on run-length encoding (McIlroy 1982), using the approach of "infinite Huffman codes" (Golomb 1966; Gallager and van Voorhis 1975). However, searching becomes slow. To determine whether a bit is "1" in the sparse vector, the encoded lengths of all the preceding intervals (runs) have to be decoded and summed (see Figure 4.5).

Figure 4.5: Compression using run-length encoding. The notation [x] stands for the encoded value of number x

4.3.1 Bit-block Compression (BC)

This method accelerates the search by sacrificing some space, compared to the run-length encoding technique. The compression method is based on bit-blocks, and was called BC (for bit-Block Compression). To speed up the searching, the sparse vector is divided into groups of consecutive bits (bit-blocks); each bit-block is encoded individually. The size of the bit-blocks is chosen according to Eq. (A2.2) in Faloutsos and Christodoulakis (1987) to achieve good compression. For each bit-block we create a signature, which is of variable length and consists of at most three parts (see Figure 4.6):

Part I. It is one bit long and it indicates whether there are any "l"s in the bit-block (1) or the bit-block is empty (0). In the latter case, the bit-block signature stops here.

Part II. It indicates the number s of "1"s in the bit-block. It consists of s - 1 "1"s and a terminating zero. This is not the optimal way to record the number of "1"s. However this representation is simple and it seems to give results close to the optimal. [See Eq. (4.1.4, 4.1.5) in Faloutsos and Christodoulakis (1987)].

Part III. It contains the offsets of the "1"s from the beginning of the bit-block (1 gb bits for each "1", where b is the bit-block size).

                b

               <-->

sparse vector  0000  1001  0000  0010  1000

-------------------------------------------

Part I          0      1     0     1     1

Part II               10           0     0

Part III             0011         10    00

Figure 4.6: Illustration of the BC method with bit-block size b = 4.

Figure 4.6 illustrates how the BC method compresses the sparse vector of Figure 4.4. Figure 4.7 illustrates the way to store parts of a document signature: the first parts of all the bit-block signatures are stored consecutively, then the second parts, and so on.

                    0 1 0 1 1 | 10 0 0 | 00 11 10 00

Figure 4.7: BC method--Storing the signature by concatenating the parts. Vertical lines indicate the part boundaries.

4.3.2 Variable Bit-block Compression (VBC)

The BC method was slightly modified to become insensitive to changes in the number of words D per block. This is desirable because the need to split messages in logical blocks is eliminated, thus simplifying the resolution of complex queries: There is no need to "remember" whether some of the terms of the query have appeared in one of the previous logical blocks of the message under inspection.

The idea is to use a different value for the bit-block size bopt for each message, according to the number W of bits set to "1" in the sparse vector. The size of the sparse vector B is the same for all messages. Analysis in Faloutsos and Christodoulakis (1987) shows how to choose the optimal size b of the bit-blocks for a document with W (distinct) words; arithmetic examples in the same paper indicate the advantages of the modified method.

This method was called VBC (for Variable bit-Block Compression). Figure 4.8 illustrates an example layout of the signatures in the VBC method. The upper row corresponds to a small message with small W, while the lower row to a message with large W. Thus, the upper row has a larger value of bopt, fewer bit-blocks, shorter Part I (the size of Part I is the number of bit-blocks), shorter Part II (its size is W) and fewer but larger offsets in Part III (the size of each offset is log bopt bits).

Figure 4.8: An example layout of the message signatures in the VBC method.

4.3.3 Performance

With respect to space overhead, the two methods (BC and VBC) require less space than SSF for the same false drop probability. Their response time is slightly less than SSF, due to the decreased I/0 requirements. The required main-memory operations are more complicated (decompression, etc.), but they are probably not the bottleneck. VBC achieves significant savings even on main-memory operations. With respect to insertions, the two methods are almost as easy as the SSF; they require a few additional CPU cycles to do the compression.

Figure 4.9: Comparison of Fd of BC (dotted line) against SSF (solid line), as a function of the space overhead Ov. Analytical results, from Faloutsos and Christodoulakis (1987).

As shown in Figure 4.9, the BC method achieves [image] 50 percent savings in false drops for documents with D = 40 vocabulary words each.

4.4 VERTICAL PARTITIONING

The idea behind the vertical partitioning is to avoid bringing useless portions of the document signature in main memory; this can be achieved by storing the signature file in a bit-sliced form (Roberts 1979; Faloutsos and Chan 1988) or in a "frame-sliced" form (Lin and Faloutsos 1988).

4.4.1 Bit-Sliced Signature Files (BSSF)

The bit-sliced design is illustrated in Figure 4.10.

Figure 4.10: Transposed bit matrix

To allow insertions, we propose using F different files, one per each bit position, which will be referred to as "bit-files." The method will be called BSSF, for "Bit-Sliced Signature Files." Figure 4.11 illustrates the proposed file structure.

Searching for a single word requires the retrieval of m bit vectors (instead of all of the F bit vectors) which are subsequently ANDed together. The resulting bit vector has N bits, with "1" 's at the positions of the qualifying logical blocks.

An insertion of a new logical block requires F disk accesses, one for each bit-file, but no rewriting! Thus, the proposed method is applicable on WORM optical disks. As mentioned in the introduction, commercial optical disks do not allow a single bit to be written; thus, we have to use a magnetic disk, that will hold the last page of each file; when they become full, we will dump them on the optical disk. The size of each bit file can be predicted (Faloutsos and Chan 1988) and the appropriate space can be preallocated on the disk.

The following data are taken from product specification:

Micropolis, [image] Winchester, 1350 Series Seek time (avg) = 28 ms; Latency time (avg) = 8.33 ms; Transfer rate = 10 Mbits/sec; Sector size = 512 bytes.

Alcatel Thompson Gigadisk GM 1001 (WORM) Seek time within +/- 100 tracks = 35 ms; Seek time beyond current band = 200 ms; Latency (avg) = 27 ms; Transfer rate = 3.83 Mbits/sec; Sector size = 1K bytes.

Figure 4.11: File structure for Bit-Sliced Signature Files. The text file is omitted.

4.4.2 B'SSF, a Faster Version of BSSF

The traditional design of BSSF suggests chosing the optimal value for m to be such that the document signatures contain "l"s by 50 percent. A typical value of m is of the order of 10 (Christodoulakis and Faloutsos 1984). This implies 10 random disk accesses on a single word query. In Lin and Faloutsos (1988) it is suggested to use a smaller than optimal value of m; thus, the number of random disk accesses decreases. The drawback is that, in order to maintain the same false drop probability, the document signatures have to be longer.

4.4.3 Frame-Sliced Signature File

The idea behind this method (Lin and Faloutsos 1988) is to force each word to hash into bit positions that are close to each other in the document signature.Then, these bit files are stored together and can be retrieved with few random disk accesses. The main motivation for this organization is that random disk accesses are more expensive than sequential ones, since they involve movement of the disk arm.

More specifically, the method works as follows: The document signature (F bits long) is divided into k frames of s consecutive bits each. For each word in the document, one of the k frames will be chosen by a hash function; using another hash function, the word sets m bits (not necessarily distinct) in that frame. F, k, s, m are design parameters. Figure 4.12 gives an example for this method.

      Word      Signature

      free      000000  110010

      text      010110  000000

doc. signature  010110  110010

Figure 4.12: D = 2 words. F = 12, s = 6, k = 2, m = 3. The word free is hashed into the second frame and sets 3 bits there. The word text is hashed into the first frame and also sets 3 bits there.

The signature matrix is stored framewise. Each frame will be kept in consecutive disk blocks. Only one frame has to be retrieved for a single word query, that is, only one random disk access is required. At most, n frames have to be scanned for an n word query. Insertion will be much faster than BSSF since we need only access k frames instead of F bit-slices. This method will be referred to as Frame-Sliced Signature file (FSSF).

4.4.4 The Generalized Frame-Sliced Signature File (GFSSF)

In FSSF, each word selects only one frame and sets m bit positions in that frame. A more general approach is to select n distinct frames and set m bits (not necessarily distinct) in each frame to generate the word signature. The document signature is the OR-ing of all the word signatures of all the words in that document. This method is called Generalized Frame-Sliced Signature File (GFSSF; Lin and Faloutsos 1988).

Notice that BSSF, B'SSF, FSSF, and SSF are actually special cases of GFSSF:

When k = F, n = m, it reduces to the BSSF or B'SSF method.

When n = 1, it reduces to the FSSF method.

When k = 1, n = 1, it becomes the SSF method (the document signature is broken down to one frame only).

4.4.5 Performance

Since GFSSF is a generalized model, we expect that a careful choice of the parameters will give a method that is better (whatever the criterion is) than any of its special cases. Analysis in the above paper gives formulas for the false drop probability and the expected response time for GFSSF and the rest of the methods. Figure 4.13 plots the theoretically expected performance of GFSSF, BSSF, B'SSF, and FSSF. Notice that GFSSF is faster than BSSF, B'SSF, and FSSF, which are all its special cases. It is assumed that the transfer time for a page

Ttrans = 1 msec

and the combined seek and latency time Tseek is

Tseek = 40 msec

Figure 4.13: Response time vs. space overhead: a comparison between BSSF, B'SSF, FSSF and GFSSF. Analytical results on a 2.8Mb database.

The text file was 2.8Mb long, with average document size [image] 1 Kb. Each document contained D = 58 distinct noncommon words.

The experiments were conducted on a SUN 3/50 with a local disk, when the load was light; the system was implemented in C, under UNIX. The experiments on the 2.8Mb database showed good agreement between the theoretical and experimental values for the false drop probability. The maximum relative error was 16 percent. The average response time ("real time") was 420 ms for FSSF with s = 63, m = 8, Ov = 18 percent, and 480 ms for GFSSF with s = 15, n = 3, m = 3, and Ov = 18 percent.

4.5 VERTICAL PARTITIONING AND COMPRESSION

The idea in all the methods in this class (Faloutsos and Chan 1988) is to create a very sparse signature matrix, to store it in a bit-sliced form, and compress each bit slice by storing the position of the "1"s in the slice. The methods in this class are closely related to inversion with a hash table.

4.5.1 Compressed Bit Slices (CBS)

Although the bit-sliced method is much faster than SSF on retrieval, there may be room for two improvements:

1. On searching, each search word requires the retrieval of m bit files, exactly because each word signature has m bits set to "1". The search time could be improved if m was forced to be "1".

2. The insertion of a logical block requires too many disk accesses (namely, F, which is typically 600-1,000).

If we force m = l, then F has to be increased, in order to maintain the same false drop probability (see the formulas in Faloutsos and Chan [1988]. For the next three methods, we shall use S to denote the size of a signature, to highlight the similarity of these methods to inversion using hash tables. The corresponding bit matrix and bit files will be sparse and they can be compressed. Figure 4.14 illustrate a sparse bit matrix. The easiest way to compress each bit file is to store the positions of the "1" 's. However, the size of each bit file is unpredictable now, subject to statistical variations. Therefore, we store them in buckets of size Bp, which is a design parameter. As a bit file grows, more buckets are allocated to it on demand. These buckets are linked together with pointers. Obviously, we also need a directory (hash table) with S pointers, one for each bit slice. Notice the following:

1. There is no need to split documents into logical blocks any more.

2. The pointer file can be eliminated. Instead of storing the position of each "1" in a (compressed) bit file, we can store a pointer to the document in the text file.

Figure 4.14: Sparse bit matrix

Thus, the compressed bit files will contain pointers to the appropriate documents (or logical blocks). The set of all the compressed bit files will be called "leve1 1" or "postings file," to agree with the terminology of inverted files (Salton and McGill 1983).

The postings file consists of postings buckets, of size Bp bytes (Bp is a design parameter). Each such bucket contains pointers to the documents in the text file, as well as an extra pointer, to point to an overflow postings bucket, if necessary.

Figure 4.15 illustrates the proposed file structure, and gives an example, assuming that the word "base" hashes to the 30-th position (h("base") = 30), and that it appears in the document starting at the 1145-th byte of the text file.

Figure 4.15: Illustration of CBS

Searching is done by hashing a word to obtain the postings bucket address. This bucket, as well as its overflow buckets, will be retrieved, to obtain the pointers to the relevant documents. To reduce the false drops, the hash table should be sparse. The method is similar to hashing. The differences are the following:

a. The directory (hash table) is sparse; Traditional hashing schemes require loads of 80-90 percent.

b. The actual word is stored nowhere. Since the hash table is sparse, there will be few collisions. Thus, we save space and maintain a simple file structure.

4.5.2 Doubly Compressed Bit Slices (DCBS)

The motivation behind this method is to try to compress the sparse directory of CBS. The file structure we propose consists of a hash table, an intermediate file, a postings file and the text file as in Figure 4.16. The method is similar to CBS. It uses a hashing function h1(), which returns values in the range (O,(S-1)) and determines the slot in the directory. The difference is that DCBS makes an effort to distinguish among synonyms, by using a second hashing function h2(), which returns bit strings that are h bits long. These hash codes are stored in the "intermediate file," which consists of buckets of Bi bytes (design parameter). Each such bucket contains records of the form (hashcode, ptr). The pointer ptr is the head of a linked list of postings buckets.

Figure 4.16: Illustration of DCBS

Figure 4.16 illustrates an example, where the word "base" appears in the document that starts at the 1145-th byte of the text file. The example also assumes that h = 3 bits, hl("base") = 30, and h2("base") = (011)2.

Searching for the word "base" is handled as follows:

Step 1 h1("base") = 30: The 30-th pointer of the directory will be followed. The corresponding chain of intermediate buckets will be examined.

Step 2 h2("base") = (011)2: The records in the above intermediate buckets will be examined. If a matching hash code is found (at most one will exist!), the corresponding pointer is followed, to retrieve the chain of postings buckets.

Step 3 The pointers of the above postings buckets will be followed, to retrieve the qualifying (actually or falsely) documents.

Insertion is omitted for brevity.

4.5.3 No False Drops Method (NFD)

This method avoids false drops completely, without storing the actual words in the index. The idea is to modify the intermediate file of the DCBS, and store a pointer to the word in the text file. Specifically, each record of the intermediate file will have the format (hashcode, ptr, ptr-to-word), where ptr-to-word is a pointer to the word in the text file. See Figure 4.17 for an illustration.

Figure 4.17: Illustration of NFD

This way each word can be completely distinguished from its synonyms, using only h bits for the hash code and p (=4 bytes, usually) for the ptr-to-word. The advantages of storing ptr-to-word instead of storing the actual word are two: (1) space is saved (a word from the dictionary is [image] 8 characters long (Peterson 1980), and (2) the records of the intermediate file have fixed length. Thus, there is no need for a word delimiter and there is no danger for a word to cross bucket boundaries.

Searching is done in a similar way with DCBS. The only difference is that, whenever a matching hash code is found in Step 2, the corresponding ptr-to-word is followed, to avoid synonyms completely.

4.5.4 Performance

In Faloutsos and Chan (1988), an analytical model is developed for the performance of each of the above methods. Experiments on the 2.8Mb database showed that the model is accurate. Figure 4.18 plots the theoretical performance of the methods (search time as a function of the overhead).

The final conclusion is that these methods are fast, requiring few disk accesses, they introduce 20-25 percent space overhead, and they still require append-only operations on insertion.

Figure 4.18: Total disk accesses on successful search versus space overhead. Analytical results for the 2.8 Mb database, with p = 3 bytes per pointer. Squares correspond to the CBS method, circles to DCBS, and triangles to NFD.

4.6 HORIZONTAL PARTITIONING

The motivation behind all these methods is to avoid the sequential scanning of the signature file (or its bit-slices), in order to achieve better than O(N) search time. Thus, they group the signatures into sets, partitioning the signature matrix horizontally. The grouping criterion can be decided beforehand, in the form of a hashing function h(S), where S is a document signature (data independent case). Alternatively, the groups can be determined on the fly, using a hierarchical structure (e.g. a B-tree--data dependent case).

4.6.1 Data Independent Case

Gustafson's method

The earliest approach was proposed by Gustafson (1971). Suppose that we have records with, say six attributes each. For example, records can be documents and attributes can be keywords describing the document. Consider a hashing function h that hashes a keyword w to a number h(w) in the range 0-15. The signature of a keyword is a string of 16 bits, all of which are zero except for the bit at position h(w). The record signature is created by superimposing the corresponding keyword signatures. If k < 6 bits are set in a record signature, additional 6 - k bits are set by some random method. Thus, there are comb (16,6) = 8,008 possible distinct record signatures (where C(m,n) denotes the combinations of m choose n items). Using a hash table with 8,008 slots, we can map each record signature to one such slot as follows: Let p1 < p2 < . . . < p6 the positions where the "1"s occur in the record signature. Then the function

C(p1, 1) + C(p2, 2) + . . . + C(p6, 6)

maps each distinct record signature to a number in the range 0-8,007. The interesting point of the method is that the extent of the search decreases quickly (almost exponentially) with the number of terms in the (conjunctive) query. Single word queries touch C(15,5) = 3,003 slots of the hash table, two-word queries touch C(14, 4) = 1,001 slots, and so on.

Although elegant, Gustafson's method suffers from some practical problems:

1. Its performance deteriorates as the file grows.

2. If the number of keywords per document is large, then either we must have a huge hash table or usual queries (involving 3-4 keywords) will touch a large portion of the database.

3. Queries other than conjunctive ones are handled with difficulty.

Partitioned signature files

Lee and Leng (1989) proposed a family of methods that can be applied for longer documents. They suggested using a portion of a document signature as a signature key to partition the signature file. For example, we can choose the first 20 bits of a signature as its key and all signatures with the same key will be grouped into a so-called "module." When a query signature arrives, we will first examine its signature key and look for the corresponding modules, then scan all the signatures within those modules that have been selected. They report 15 to 85 percent speedups over SSF, depending on the number of bits being specified in the query signature.

4.6.2 Data Dependent Case

Two-level signature files

Sacks-Davis and his colleagues (1983, 1987) suggested using two levels of signatures. Their documents are bibliographic records of variable length. The first level of signatures consists of document signatures that are stored sequentially, as in the SSF method. The second level consists of "block signatures"; each such signature corresponds to one block (group) of bibliographic records, and is created by superimposing the signatures of all the words in this block, ignoring the record boundaries. The second level is stored in a bit-sliced form. Each level has its own hashing functions that map words to bit positions.

Searching is performed by scanning the block signatures first, and then concentrating on these portions of the first-level signature file that seem promising.

Analysis on a database with N [image] 106 records (with 128 bytes per record on the average) reported response times as low as 0.3 seconds for single word queries, when 1 record matched the query. The BSSF method required [image] 1-5 seconds for the same situation.

A subtle problem arises when multiterm conjunctive queries are asked, (e.g., "data and retrieval"): A block may result in an unsuccessful block match, because it may contain the desired terms, but not within the same record. The authors propose a variety of clever ways to minimize these block matches.

S-tree

Deppisch (1986) proposed a B-tree like structure to facilitate fast access to the records (which are signatures) in a signature file. The leaf of an S-tree consists of k "similar" (i.e., with small Hamming distance) document signatures along with the document identifiers. The OR-ing or these k document signatures forms the "key" of an entry in an upper level node, which serves as a directory for the leaves. Recursively we construct directories on lower level directories until we reach the root. The S-tree is kept balanced in a similar manner as a B-trees: when a leaf node overflows it is split in two groups of "similar" signatures; the father node is changed appropriately to reflect the new situation. Splits may propagate upward until reaching the root.

The method requires small space overhead; the response time on queries is difficult to estimate analytically. The insertion requires a few disk accesses (proportional to the height of the tree at worst), but the append-only property is lost. Another problem is that higher level nodes may contain keys that have many 1's and thus become useless.

4.7 DISCUSSION

Signature files provide a space-time trade-off between full text scanning, which is slow but requires no overheads, and inversion, which is fast but requires expensive insertions and needs significant space overhead. Thus, signature-based methods have been applied in the following environments:

1. Medium-size databases. "Hyperties" (Lee et al. 1988), a hypertext package that is commercially available on IBM PCs, uses the SSF method, due to its simplicity, low space overhead, and satisfactory search time (3-10 seconds, for queries with low selectivities) on databases of [image] 200 Kb (after compression). Thanks to the low overhead, the whole package (software, data, and indices) fits in floppy diskettes. Another commercial product (Kimbrell 1988) is also reported to use signature files for text retrieval.

2. Databases with low query frequencies. In this case, it is not worth building and maintaining a B-tree inverted index; signature files provide a low-cost alternative, that avoids scanning the full text. This situation is probably true for message files in Office Automation; several prototype systems use the signature approach, like the Office Filing Project (Tsichritzis et al. 1983) at the University of Toronto, MINOS (Christodoulakis et al. 1986) at the University of Waterloo, the MULTOS project (Croft and Savino 1988) funded by the ESPRIT project of the European Community.

3. Due to the append-only insertion, signature-based methods can be used on WORM optical disks.

4. Due to the linear searching, signature files can easily benefit from parallelism. Stanfill and Kahle (1986) used signature files on the Connection Machine (Hillis 1985). Keeping the whole signature file in core, they can provide fast responses against databases that are [image] 10 times larger than the available main memory storage.

5. In distributed environments. In this case, keeping copies of signature files of all the sites can save remote-logins, for a modest space overhead.

6. Chang and Schek (1989) recommend signature files as a simple and flexible access method for records and "long fields" (=text and complex objects) in IBM's STAR-BUST extensible DBMS.

REFERENCES

CHANG, W. W. and H. J. SCHEK. 1989. "A Signature Access Method for the Starbust Database System." Proc. VLDB Conference, pp. 145-53. Amsterdam, Netherlands, Aug. 22-25.

CHRISTODOULAKIS, S. 1987. "Analysis of Retrieval Performance for Records and Objects Using Optical Disk Technology," ACM TODS, 12(2), 137-69.

CHRISTODOULAKIS, S., and C. FALOUTSOS. 1984. "Design Considerations for a Message File Server." IEEE Trans. on Software Engineering, SE-10 (2), 201-10.

CHRISTODOULAKIS, S., F. Ho, and M. THEODORIDOU. 1986. "The Multimedia Object Presentation Manager in MINOS: A Symmetric Approach." Proc. ACM SIGMOD.

CHRISTODOULAKIS, S., M. THEODORIDOU, F. Ho, M. PAPA, and A. PATHRIA. 1986. "Multimedia Document Presentation, Information Extraction and Document Formation in MINOS: A Model and a System." ACM TOOIS, 4 (4).

CROFT, W. B, and P. SAVINO. 1988. "Implementing Ranking Strategies Using Text Signatures." ACM Trans. on Office Informations Systems (TOOIS), 6 (1), 42-62.

DEPPISCH, U. 1986. "S-tree: A Dynamic Balanced Signature Index for Office Retrieval." Proc. of ACM "Research and Development in Information Retrieval," pp. 77-87, Pisa, Italy, Sept. 8-10.

EWING, J., S. MEHRABANZAD, S. SHECK, D. OSTROFF, and B. SHNEIDERMAN. 1986. "An Experimental Comparison of a Mouse and Arrow-jump Keys for an Interactive Encyclopedia." Int. Journal of Man-Machine Studies, 24, (1) 29-45.

FALOUTSOS, C. 1985. "Access Methods for Text." ACM Computing Surveys, 17 (1), 49-74.

FALOUTSOS, C., and R. CHAN. 1988. "Fast Text Access Methods for Optical and Large Magnetic Disks: Designs and Performance Comparison." Proc. 14th International Conf. on VLDB, pp. 280-93, Long Beach, Calif. August.

FALOUTSOS, C., and S. CHRISTODOULAKIS. 1984. "Signature Files: An Access Method for Documents and its Analytical Performance Evaluation." ACM Trans. on Office Information Systems, 2 (4), 267-88.

FALOUTSOS, C., and S. CHRISTODOULAKIS. 1987. "Description and Performance Analysis of Signature File Methods." ACM TOOIS, 5 (3), 237-57.

FUJITANI, L. 1984. "Laser Optical Disk: The Coming Revolution in On-Line Storage." CACM, 27 (6), 546-54.

GALLAGER, R. G., and D. C. VAN VOORHIS. 1975. "Optimal Source Codes for Geometrically Distributed Integer Alphabets." IEEE Trans. on Information Theory, IT-21, 228-30.

GOLOMB, S. W. 1966. "Run Length Encodings." IEEE Trans. on Information Theory, IT-12, 399-401.

GONNET, G. H. 1982. "Unstructured Data Bases," Tech Report CS-82-09, University of Waterloo.

GONNET, G. H., and F. W. TOMPA. 1987. "Mind Your Grammar: A New Approach to Modelling Text." Proc. of the Thirteenth Int. Conf. on Very Large Data Bases, pp. 339-346, Brighton, England, September 1-4.

GUSTAFSON, R. A. 1971. "Elements of the Randomized Combinatorial File Structure." ACM SIGIR, Proc. of the Symposium on Information Storage and Retrieval, pp. 163-74, University of Maryland. April.

HASKIN, R. L. 1981. "Special-Purpose Processors for Text Retrieval." Database Engineering, 4 (1), 16-29.

HILLIS, D. 1985. The Connection Machine. Cambridge, Mass.: MIT Press.

HOLLAAR, L. A. 1979. "Text Retrieval Computers." IEEE Computer Magazine, 12 (3), 40-50.

HOLLAAR, L. A., K. F. SMITH, W. H. CHOW, P. A. EMRATH, and R. L. HASKIN. 1983. "Architecture and Operation of a Large, Full-Text Information-Retrieval System," in Advanced Database Machine Architecture, D. K. Hsiao, ed. pp. 256-99. Englewood Cliffs, N.J.: Prentice Hall.

KIMBRELL, R. E. 1988. "Searching for Text? Send an N-gram!" Byte, 13 (5) 297-312.

LEE, D. L., and C.-W. LENG. 1989. "Partitioned Signature File: Designs and Performance Evaluation." ACM Trans. on Information Systems (TOIS), 7 (2), 158-80.

LEE, R., C. FALOUTSOS, C. PLAISANT, and B. SHNEIDERMAN. 1988. "Incorporating String Search in a Hypertext System: User Interface and Physical Design Issues." Dept. of Computer Science, University of Maryland, College Park. Working paper.

LEWIS, E. P. 1989. "Animated Images for a Multimedia Database." Master's Thesis, Dept. of Computer Science, University of Maryland.

LIN, Z., and C. FALOUTSOS. 1988. "Frame Sliced Signature Files," CS-TR-2146 and UMI-ACS-TR-88-88, Dept. of Computer Science, University of Maryland.

LIPMAN, D. J., and W. R. PEARSON. 1985. "Rapid and Sensitive Protein Similarity Searches." Science, 227, 1435-41, American Association for the Advancement of Science, March 22.

MCILROY, M. D. 1982. "Development of a Spelling List." IEEE Trans. on Communications, COM-30, (1), 91-99.

MOOERS, C. 1949. "Application of Random Codes to the Gathering of Statistical Information." Bulletin 31, Zator Co., Cambridge, Mass. Based on M.S. thesis, MIT, January 1948.

NOFEL, P. J. 1986. "40 Million Hits on Optical Disk." Modern Office Technology, 84-88. March.

PETERSON, J. L. 1980. "Computer Programs for Detecting and Correcting Spelling Errors." CACM, 23 (12), 676-87.

PRICE, J. 1984. "The Optical Disk Pilot Project At the Library of Congress." Videodisc and Optical Disk, 4 (6), 424-32.

ROBERTS, C. S. 1979. "Partial-Match Retrieval via the Method of Superimposed Codes." Proc. IEEE, 67 (12), 1624-42.

SACKS-DAVIS, R., A. KENT, and K. RAMAMOHANARAO. 1987. "Multikey Access Methods Based on Superimposed Coding Techniques" ACM Trans. on Database Systems (TODS), 12 (4), 655-96.

SACKS-DAVIS, R., and K. RAMAMOHANARAO. 1983. "A Two Level Superimposed Coding Scheme for Partial Match Retrieval." Information Systems, 8 (4), 273-80.

SALTON, G., and M. J. MCGILL. 1983. Introduction to Modern Information Retrieval. New York: McGraw-Hill.

STANDISH, T. A. 1984. "An Essay on Software Reuse." IEEE Trans. on Software Engineering, SE-10 (5), 494-97.

STANFILL, C., and B. KAHLE. 1986. "Parallel Free-Text Search on the Connection Machine System." CACM, 29 (12), 1229-39.

STIASSNY, S., 1960. "Mathematical Analysis of Various Superimposed Coding Methods." American Documentation, 11 (2), 155-69.

THOMA, G. R., S. SUTHASINEKUL, F. A. WALKER, J. COOKSON, and M. RASHIDIAN. 1985. "A Prototype System for the Electronic Storage and Retrieval of Document Images." ACM TOOIS, 3 (3).

TSICHRITZIS, D., and S. CHRISTODOULAKIS. 1983. "Message Files." ACM Trans. on Office Information Systems, 1 (1), 88-98.

TSICHRITZIS, D., S. CHRISTODOULAKIS, P. ECONOMOPOULOS, C. FALOUTSOS, A. LEE, D. LEE, J. VANDENBROEK, and C. WOO. 1983. "A Multimedia Office Filing System." Proc. 9th International Conference on VLDB, Florence, Italy, October-November.

Go to Chapter 5 Back to Table of Contents


Information Retrieval: CHAPTER 5: NEW INDICES FOR TEXT: PAT TREES AND

 Return to Table of Contents

CHAPTER 5: NEW INDICES FOR TEXT: PAT TREES AND PAT ARRAYS

Gaston H. Gonnet

Dept. of Computer Science, ETH, Zurich, Switzerland

Ricardo A. Baeza-Yates

Depto. de Ciencias de la Computación, Universidad de Chile, Casilla 2777, Santiago, Chile

Tim Snider

Centre for the New OED and Text Research, University of Waterloo, Waterloo, Ontario, Canada N2L 3G1

Abstract

We survey new indices for text, with emphasis on PAT arrays (also called suffix arrays). A PAT array is an index based on a new model of text that does not use the concept of word and does not need to know the structure of the text.

5.1 INTRODUCTION

Text searching methods may be classified as lexicographical indices (indices that are sorted), clustering techniques, and indices based on hashing. In this chapter we discuss two new lexicographical indices for text, called PAT trees and PAT arrays. Our aim is to build an index for the text of size similar to or smaller than the text.

Briefly, the traditional model of text used in information retrieval is that of a set of documents. Each document is assigned a list of keywords (attributes), with optional relevance weights associated to each keyword. This model is oriented to library applications, which it serves quite well. For more general applications, it has some problems, namely:

A basic structure is assumed (documents and words). This may be reasonable for many applications, but not for others.

Keywords must be extracted from the text (this is called "indexing"). This task is not trivial and error prone, whether it is done by a person, or automatically by a computer.

Queries are restricted to keywords.

For some indices, instead of indexing a set of keywords, all words except for those deemed to be too common (called stopwords) are indexed.

We prefer a different model. We see the text as one long string. Each position in the text corresponds to a semi-infinite string (sistring), the string that starts at that position and extends arbitrarily far to the right, or to the end of the text. It is not difficult to see that any two strings not at the same position are different. The main advantages of this model are:

No structure of the text is needed, although if there is one, it can be used.

No keywords are used. The queries are based on prefixes of sistrings, that is, on any substring of the text.

This model is simpler and does not restrict the query domain. Furthermore, almost any searching structure can be used to support this view of text.

In the traditional text model, each document is considered a database record, and each keyword a value or a secondary key. Because the number of keywords is variable, common database techniques are not useful in this context. Typical data-base queries are on equality or on ranges. They seldom consider "approximate text searching."

This paper describes PAT trees and PAT arrays. PAT arrays are an efficient implementation of PAT trees, and support a query language more powerful than do traditional structures based on keywords and Boolean operations. PAT arrays were independently discovered by Gonnet (1987) and Manber and Myers (1990). Gonnet used them for the implementation of a fast text searching system, PATTM (Gonnet 1987; Fawcett 1989), used with the Oxford English Dictionary (OED). Manber and Myers' motivation was searching in large genetic databases. We will explain how to build and how to search PAT arrays.

5.2 THE PAT TREE STRUCTURE

The PAT tree is a data structure that allows very efficient searching with preprocessing. This section describes the PAT data structure, how to do some text searches and algorithms to build two of its possible implementations. This structure was originally described by Gonnet in the paper "Unstructured Data Bases" by Gonnet (1983). In 1985 it was implemented and later used in conjunction with the computerization of the Oxford English Dictionary. The name of the implementation, the PATTM system, has become well known in its own right, as a software package for very fast string searching.

5.2.1 Semi-infinite Strings

In what follows, we will use a very simple model of text. Our text, or text data-base, will consist of a single (possibly very long) array of characters, numbered sequentially from one onward. Whether the text is already presented as such, or whether it can be viewed as such is not relevant. To apply our algorithms it is sufficient to be able to view the entire text as an array of characters.

A semi-infinite string is a subsequence of characters from this array, taken from a given starting point but going on as necessary to the right. In case the semi-infinite string (sistring) is used beyond the end of the actual text, special null characters will be considered to be added at its end, these characters being different than any other in the text. The name semi-infinite is taken from the analogy with geometry where we have semi-infinite lines, lines with one origin, but infinite in one direction. Sistrings are uniquely identified by the position where they start, and for a given, fixed text, this is simply given by an integer.

Example:

      Text           Once upon a time, in a far away land . . .

      sistring 1     Once upon a time . . .

      sistring 2     nce upon a time . . .

      sistring 8     on a time, in a . . .

      sistring 11    a time, in a far . . .

      sistring 22    a far away land . . .

Sistrings can be defined formally as an abstract data type and as such present a very useful and important model of text. For the purpose of this section, the most important operation on sistrings is the lexicographical comparison of sistrings and will be the only one defined. This comparison is the one resulting from comparing two sistrings' contents (not their positions). Note that unless we are comparing a sistring to itself, the comparison of two sistrings cannot yield equal. (If the sistrings are not the same, sooner or later, by inspecting enough characters, we will have to find a character where they differ, even if we have to start comparing the fictitious null characters at the end of the text).

For example, the above sistrings will compare as follows:

22 < 11 < 2 < 8 < 1

Of the first 22 sistrings (using ASCII ordering) the lowest sistring is "a far away. . ." and the highest is "upon a time. . . ."

5.2.2 PAT Tree

A PAT tree is a Patricia tree (Morrison 1968; Knuth 1973; Flajolet and Sedgewick 1986; and Gonnet 1988) constructed over all the possible sistrings of a text. A Patricia tree is a digital tree where the individual bits of the keys are used to decide on the branching. A zero bit will cause a branch to the left subtree, a one bit will cause a branch to the right subtree. Hence Patricia trees are binary digital trees. In addition, Patricia trees have in each internal node an indication of which bit of the query is to be used for branching. This may be given by an absolute bit position, or by a count of the number of bits to skip. This allows internal nodes with single descendants to be eliminated, and thus all internal nodes of the tree produce a useful branching, that is, both subtrees are non-null. Patricia trees are very similar to compact suffix trees or compact position trees (Aho et al. 1974).

Patricia trees store key values at external nodes; the internal nodes have no key information, just the skip counter and the pointers to the subtrees. The external nodes in a PAT tree are sistrings, that is, integer displacements. For a text of size n, there are n external nodes in the PAT tree and n - 1 internal nodes. This makes the tree O(n) in size, with a relatively small asymptotic constant. Later we will want to store some additional information (the size of the subtree and which is the taller subtree) with each internal node, but this information will always be of a constant size.

Figure 5.1 shows an example of a PAT tree over a sequence of bits (normally it would be over a sequence of characters), just for the purpose of making the example easier to understand. In this example, we show the Patricia tree for the text "01100100010111..." after the first 8 sistrings have been inserted. External nodes are indicated by squares, and they contain a reference to a sistring, and internal nodes are indicated by a circle and contain a displacement. In this case we have used, in each internal node, the total displacement of the bit to be inspected, rather than the skip value.

Notice that to reach the external node for the query 00101 we first inspect bit 1 (it is a zero, we go left) then bit 2 (it is zero, we go left), then bit 3 (it is a one, we go right), and then bit 5 (it is a one, we go right). Because we may skip the inspection of some bits (in this case bit 4), once we reach our desired node we have to make one final comparison with one of the sistrings stored in an external node of the current subtree, to ensure that all the skipped bits coincide. If they do not coincide, then the key is not in the tree.

Figure 5.1: PAT tree when the sistrings 1 through 8 have been inserted

5.2.3 Indexing Points

So far we have assumed that every position in the text is indexed, that is, the Patricia tree has n external nodes, one for each position in the text. For some types of search this is desirable, but in some other cases not all points are necessary or even desirable to index. For example, if we are interested in word and phrase searches, then only those sistrings that are at the beginning of words (about 20% of the total for common English text) are necessary. The decision of how many sistrings to include in the tree is application dependent, and will be a trade-off between size of the index and search requirements.

5.3 ALGORITHMS ON THE PAT TREE

In this section we will describe some of the algorithms for text searching when we have a PAT tree of our text.

5.3.1 Prefix Searching

Notice that every subtree of the PAT tree has all the sistrings with a given prefix, by construction. Then prefix searching in a PAT tree consists of searching the prefix in the tree up to the point where we exhaust the prefix or up to the point where we reach an external node. At this point we need to verify whether we could have skipped bits. This is done with a single comparison of any of the sistrings in the subtree (considering an external node as a subtree of size one). If this comparison is successful, then all the sistrings in the subtree (which share the common prefix) are the answer; otherwise there are no sistrings in the answer.

It is important to notice that the search ends when the prefix is exhausted or when we reach an external node and at that point all the answer is available (regardless of its size) in a single subtree. For random Patricia trees, the height is O(log n) (Pittel 1985; Apostolico and Szpankowski 1987) and consequently with PAT trees we can do arbitrary prefix searching in O(log n) time, independent of the size of the answer. In practice, the length of the query is less than O(log n), thus the searching time is proportional to the query length.

By keeping the size of each subtree in each internal node, we can trivially find the size of any matched subtree. (Knowing the size of the answer is very appealing for information retrieval purposes.) Figure 5.2 shows the search for the prefix "10100" and its answer.

Figure 5.2: Prefix searching.

5.3.2 Proximity Searching

We define a proximity search as finding all places where a string S 1 is at most a fixed (given by the user) number of characters away from another string S 2. The simplest algorithm for this type of query is to search for sl and s2. Then, sort by position the smaller of the two answers. Finally, we traverse the unsorted answer set, searching every position in the sorted set, and checking if the distance between positions (and order, if we always want s1 before s2) satisfies the proximity condition. If m1 and m2 (m1 < m2) are the respective answer set sizes, this algorithm requires (m1 + m2) log m1 time. This is not very appealing if m1 or m2 are O(n). For the latter case (when one of the strings S 1 or S 2 is small), better solutions based on PAT arrays have been devised by Manber and Baeza-Yates (1991).

5.3.3 Range Searching

Searching for all the strings within a certain range of values (lexicographical range) can be done equally efficiently. More precisely, range searching is defined as searching for all strings that lexicographically compare between two given strings. For example, the range "abc" . . "acc" will contain strings like "abracadabra," "acacia," "aboriginal," but not "abacus" or "acrimonious."

To do range searching on a PAT tree, we search each end of the defining intervals and then collect all the subtrees between (and including) them. It should be noticed that only O(height) subtrees will be in the answer even in the worst case (the worst case is 2 height - 1) and hence only O(log n) time is necessary in total. As before, we can return the answer and the size of the answer in time O(log n) independent of the actual size of the answer.

5.3.4 Longest Repetition Searching

The longest repetition of a text is defined as the match between two different positions of a text where this match is the longest (the most number of characters) in the entire text. For a given text, the longest repetition will be given by the tallest internal node in the PAT tree, that is, the tallest internal node gives a pair of sistrings that match for the greatest number of characters. In this case, tallest means not only the shape of the tree but has to consider the skipped bits also. For a given text, the longest repetition can be found while building the tree and it is a constant, that is, will not change unless we change the tree (that is, the text).

It is also interesting and possible to search for the longest repetition not just for the entire tree/text, but for a subtree. This means searching for the longest repetition among all the strings that share a common prefix. This can be done in O(height) time by keeping one bit of information at each internal node, which will indicate on which side we have the tallest subtree. By keeping such a bit, we can find one of the longest repetitions starting with an arbitrary prefix in O(log n) time. If we want to search for all of the longest repetitions, we need two bits per internal node (to indicate equal heights as well) and the search becomes logarithmic in height and linear in the number of matches.

5.3.5 "Most Significant" or "Most Frequent" Searching

This type of search has great practical interest, but is slightly difficult to describe. By "most significant" or "most frequent" we mean the most frequently occurring strings within the text database. For example, finding the "most frequent" trigram is finding a sequence of three letters that appears most often within our text.

In terms of the PAT tree, and for the example of the trigrams, the number of occurrences of a trigram is given by the size of the subtree at a distance 3 characters from the root. So finding the most frequent trigram is equivalent to finding the largest subtree at a distance 3 characters from the root. This can be achieved by a simple traversal of the PAT tree which is at most O(n/average size of answer) but is usually much faster.

Searching for "most common" word is slightly more difficult but uses a similar algorithm. A word could be defined as any sequence of characters delimited by a blank space. This type of search will also require a traversal, but in this case the traversal is only done in a subtree (the subtree of all sistrings starting with a space) and does not have a constant depth; it traverses the tree to the places where each second blank appears.

We may also apply this algorithm within any arbitrary subtree. This is equivalent to finding the most frequently occurring trigram, word, and the like, that follows some given prefix.

In all cases, finding the most frequent string with a certain property requires a subtree selection and then a tree traversal which is at most O(n/k) but typically much smaller. Here, k is the average size of each group of strings of the given property. Techniques similar to alpha-beta pruning can be used to improve this search.

5.3.6 Regular Expression Searching

The main steps of the algorithm due by Baeza-Yates and Gonnet (1989) are:

a.     Convert the regular expression passed as a query into a minimized deterministic finite automation (DFA), which may take exponential space/time with respect to the query size but is independent of the size of the text (Hopcroft and Ullman 1979).

b.     Next eliminate outgoing transitions from final states (see justification in step (e). This may induce further minimization.

c.     Convert character DFAs into binary DFAs using any suitable binary encoding of the input alphabet; each state will then have at most two outgoing transitions, one labeled 0 and one labeled 1.

d.     Simulate the binary DFA on the binary digital trie from all sistrings of text using the same binary encoding as in step b. That is, associate the root of the tree with the initial state, and, for any internal node associated with state i, associate its left descendant with state j if i   [image]  j for a bit 0, and associate its right descendant with state k if i   [image]  k for a 1 (see Figure 5.3).

e.     For every node of the index associated with a final state, accept the whole subtree and halt the search in that subtree. (For this reason, we do not need outgoing transitions in final states).

f.     On reaching an external node, run the remainder of the automaton on the single string determined by this external node.

Figure 5.3: Simulating the automaton on a binary digital tree

For random text, it is possible to prove that for any regular expression, the average searching time is sublinear (because of step e), and of the form:

O(logm(n)n  [image])

where   [image]  < 1, and m  [image]  0 (integer) depend on the incidence matrix of the simulated DFA (that is, they depend on the regular expression). For details, see Baeza-Yates and Gonnet (1989).

5.4 BUILDING PAT TREES AS PATRICIA TREES

The implementation of PAT trees using conventional Patricia trees is the obvious one, except for some implementation details which cannot be overlooked as they would increase the size of the index or its accessing time quite dramatically. Since this type of index is typically built over very large text databases, and the size of the index tree is linear in the size of the text, an economic implementation is mandatory.

It is easy to see that the internal nodes will be between 3 and 4 words in size. Each external node could be one word and consequently we are taking between 4n and 5n words for the index, or about 18n chars tor indexing n characters, most likely unacceptable from the practical point of view.

Second, the organization for the tree should be such that we can gain from the reading of large records external files will use physical records which are certainly larger than one internal node).

The main ideas to solve (or alleviate) both problems are

a.     bucketing of external nodes, and

b.     mapping the tree onto the disk using supernodes.

Collecting more than one external node is called bucketing and is an old idea in data processing. A bucket replaces any subtree with size less than a certain constant (say b) and hence saves up to b - l internal nodes. The external nodes inside a bucket do not have any structure associated with them, and any search that has to be done in the bucket has to be done on all the members of the bucket. This increases the number of comparisons for each search, in the worst case by b. On the other hand, and assuming a random distribution of keys, buckets save a significant number of internal nodes. In general, it is not possible to have all buckets full, and on average the number of keys per bucket is b In 2. Hence, instead of n - 1 internal nodes, we have about (n/bln2) internal nodes for random strings.

This means that the overhead of the internal nodes, which are the largest part of the index, can be cut down by a factor of b1n2. We have then a very simple trade-off between time (a factor of b) and space (a factor of b1n2).

Organizing the tree in super-nodes has advantages from the point of view of the number of accesses as well as in space. The main idea is simple: we allocate as much as possible of the tree in a disk page as long as we preserve a unique entry point for every page. De facto, every disk page has a single entry point, contains as much of the tree as possible, and terminates either in external nodes or in pointers to other disk pages (notice that we need to access disk pages only, as each disk page has a single entry point). The pointers in internal nodes will address either a disk page or another node inside the same page, and consequently can be substantially smaller (typically about half a word is enough). This further reduces the storage cost of internal nodes.

Unfortunately, not all disk pages will be 100 percent full. A bottom-up greedy construction guarantees at least 50 percent occupation. Actual experiments indicate an occupation close to 80 percent.

With these constraints, disk pages will contain on the order of 1,000 internal/external nodes. This means that on the average, each disk page will contain about 10 steps of a root-to-leaf path, or in other words that the total number of accesses is a tenth of the height of the tree. Since it is very easy to keep the root page of the tree in memory, this means that a typical prefix search can be accomplished with 2-3 disk accesses to read the index (about 30 to 40 tree levels) and one additional final access to verify the skipped bits (buckets may require additional reading of strings). This implementation is the most efficient in terms of disk accesses for this type of search.

5.5 PAT TREES REPRESENTED AS ARRAYS

The previous implementation has a parameter, the external node bucket size, b. When a search reaches a bucket, we have to scan all the external nodes in the bucket to determine which if any satisfy the search. If the bucket is too large, these costs become prohibitive.

However, if the external nodes in the bucket are kept in the same relative order as they would be in the tree, then we do not need to do a sequential search, we could do an indirect binary search (i.e., compare the sistrings referred to by the external node) to find the nodes that satisfy the search. Consequently, the cost of searching a bucket becomes 2 log b - 1 instead of b.

Although this is not significant for small buckets, it is a crucial observation that allows us to develop another implementation of PAT trees. This is simply to let the bucket grow well beyond normal bucket sizes, including the option of letting it be equal to n, that is, the whole index degenerates into a single array of external nodes ordered lexicographically by sistrings. With some additions, this idea was independently discovered by Manber and Myers (1990), who called the structures suffix arrays.

There is a straightforward argument that shows that these arrays contain most of the information we had in the Patricia tree at the cost of a factor of log2 n. The argument simply says that for any interval in the array which contains all the external nodes that would be in a subtree, in log2 n comparisons in the worst case we can divide the interval according to the next bit which is different. The sizes of the subtrees are trivially obtained from the limits of any portion of the array, so the only information that is missing is the longest-repetition bit, which is not possible to represent without an additional structure. Any operation on a Patricia tree can be simulated in O(log n) accesses. Figure 5.4 shows this structure.

Figure 5.4: A PAT array.

5.5.1 Searching PAT Trees as Arrays

It turns out that it is not neccessary to simulate the Patricia tree for prefix and range searching, and we obtain an algorithm which is O(log n) instead of O(log2 n) for these operations. Actually, prefix searching and range searching become more uniform. Both can be implemented by doing an indirect binary search over the array with the results of the comparisons being less than, equal (or included in the case of range searching), and greater than. In this way, the searching takes at most 2 log2 n - 1 comparisons and 4log2 n disk accesses (in the worst case).

The "most frequent" searching can also be improved, but its discussion becomes too technical and falls outside the scope of this section. The same can be said about "longest repetition" which requires additional supporting data structures. For regular expression searching, the time increases by a O(log n) factor. Details about proximity searching are given by Manber and Baeza-Yates (1991).

In summary, prefix searching and range searching can be done in time O(log2 n) with a storage cost that is exactly one word (a sistring pointer) per index point (sistring). This represents a significant economy in space at the cost of a modest deterioration in access time.

5.5.2 Building PAT Trees as Arrays

A historical note is worth entering at this point. Most of the research on this structure was done within the Centre for the New Oxford English Dictionary at the University of Waterloo, which was in charge of researching and implementing different aspects of the computerization of the OED from 1985. Hence, there was considerable interest in indexing the OED to provide fast searching of its 600Mb of text. A standard building of a Patricia tree in any of its forms would have required about n   [image]  log2(n)   [image]  t hours, where n is the number of index points and t is the time for a random access to disk. As it turned out, the dictionary had about n = 119,000,000 and our computer systems would give us about 30 random accesses per second. That is, 119,000,000   [image]  27/30   [image]  60   [image]  60 = 29,750 hours or about 3.4 years. That is, we would still be building the index for the OED, 2nd ed.

Even if we were using an algorithm that used a single random access per entry point (a very minimal requirement!), the total disk time would be 119,000,000/30   [image]  60   [image]  60   [image]  24 = 45.9 days. Still not acceptable in practical terms. It is clear from these numbers that we have to investigate algorithms that do not do random access to the text, but work based on different principles.

Clearly we need better algorithms than the "optimal' algorithm here. We would like to acknowledge this indirect contribution by the OED, as without this real test case, we would never have realized how difficult the problem was and how much more work had to be done to solve it in a reasonable time. To conclude this note, we would say that we continue to research for better building algorithms although we can presently build the index for the OED during a weekend.

This subsection will be divided in two. First we will present the building operations, which can be done efficiently, and second, two of the most prominent algorithms for large index building.

Building PAT arrays in memory

If a portion of the text is small enough to fit in main memory together with its PAT array, then this process can be done very efficiently as it is equivalent to string sorting. Note that here we are talking about main memory; if paging is used to simulate a larger memory, the random access patterns over the text will certainly cause severe memory thrashing.

Quicksort is an appropriate algorithm for this building phase since it has an almost sequential pattern of access over the sorted file. For maximal results we can put all of the index array on external storage and apply external quicksort (see Gonnet and Baeza-Yates, section 4.4.5 [1984]) indirectly over the text. In this case it is possible to build an index for any text which together with the program can fit in main memory. With today's memory sizes, this is not a case to ignore. This is the algorithm of choice for small files and also as a building block for other algorithms.

Merging small against large PAT arrays

A second case that can be solved efficiently is the case of merging two indices (to produce a single one) when the text plus twice the index of one of them fits in main memory. This algorithm is not trivial and deserves a short explanation. The text of the small file together with a PAT array for the small file (of size n1) plus an integer array of size n1 + 1 are kept in main memory. The integer array is used to count how many sistrings of the big file fall between each pair of index points in the small file (see Figure 5.5). To do this counting, the large file is read sequentially and each sistring is searched in the PAT array of the small file until it is located between a pair of points in the index. The corresponding counter is incremented. This step will require O(n2 log n1) comparisons and O(n2) characters to be read sequentially. Once the counting is finished, the merging takes place by reading the PAT array of the large file and inserting the PAT array of the small file guided by the counts (see Figure 5.6). This will require a sequential reading of n1+ n2 words. In total, this algorithm performs a linear amount of sequential input and output and O(n2 log n1) internal work, and its behavior is not only acceptable but exceptionally good.

Figure 5.5: Small index in main memory

Figure 5.6: Merging the small and the large index

Given these simple and efficient building blocks we can design a general index building algorithm. First we split the text file into pieces, the first piece being as large as possible to build an index in main memory. The remaining pieces are as large as possible to allow merging via the previous algorithm (small against large). Then we build indices for all these parts and merge each part.

An improvement may be made by noticing that index points at the front of the text will be merged many times into new points being added at the end. We take advantage of this by constructing partial indices on blocks of text one half the size of memory. These indices may be merged with each other, the entire merge taking place in memory. The merged index is not created at this point, since it would fill memory and could not be merged with any further index. As before, a vector of counters is kept, indicating how many entries of the first index fall between each pair of adjacent entries in the second. When the nth block of text is being indexed, the n - 1 previous indices are merged with it. The counters are accumulated with each merge. When all of the merges have been done, the counters are written to a file. When all of the blocks have been indexed and merged, the files of counters are used as instructions to merge all the partial indices.

The number of parts is O(n / m) where m is the total amount of available memory, and overall the algorithm requires O(n2 / m) sequential access and O(n2 log n / m) time. Given the sizes of present day memories and the differences of accessing time for random versus sequential access, the above quadratic algorithm would beat a linear algorithm even for databases like the Oxford English Dictionary!

An interesting property of the last version of the algorithm is that each of the merges of partial indices is independent. Therefore, all of the O((n / m)2) merges may be run in parallel on separate processors. After all merges are complete, the counters for each partial index must be accumulated and the final index merge may be done.

Another practical advantage of this algorithm is that the final index merge is done without reference to the text. Thus, at the time that the final merge is being done, the text may be removed from the system and reloaded after removal of the partial indices. In situations where the text, the final index, and the sum of partial indices may each be one or more gigabytes, this is an important consideration.

5.5.3 Delayed Reading Paradigm

The fact that comparing sistrings through random access to disk is so expensive is a consequence of two phenomena. First, the reading itself requires some relatively slow) physical movement. Second, the amount of text actually used is a small fraction of the text made available by an I/O operation.

Here we will describe a programming technique that tries to alleviate the above problem without altering the underlying algorithms. The technique is simply to suspend execution of the program every time a random input is required, store these requests in a request pool, and when convenient/necessary satisfy all requests in the best ordering possible.

This technique works on algorithms that do not block on a particular I/O request but could continue with other branches of execution with a sufficiently high degree of parallelism. More explicitly, we have the following modules:

a.     the index building program

b.     a list of blocked requests

c.     a list of satisfied requests

The c list is processed in a certain order. This ordering is given inversely by the likelihood of a request to generate more requests: the requests likely to generate fewer requests are processed first. Whenever the c list is exhausted or the available memory for requests is exhausted, then the b list is sorted for the best I/O performance and all the I/O is performed.

5.5.4 Merging Large Files

We have seen an algorithm for merging a large file against a small file, small meaning that it fits, with its index, in memory. We may also need to merge two or more large indices. We can do this by reading the text associated with each key and comparing. If we have n keys and m files, and use a heap to organize the current keys for each file being merged, this gives us O(n log m) comparisons. More importantly, it gives us n random disk accesses to fetch key values from the text. If n is 100,000,000 and we can do 50 random disk accesses per second, then this approach will take 2,000,000 seconds or about 23 days to merge the files. A second problem inherent with sistrings is that we do not know how many characters will be necessary to compare the keys. This will be addressed later.

An improvement can be made by reducing the number of random disk accesses and increasing the amount of sequential disk I/O. The sistring pointers in the PAT array are ordered lexicographically according to the text that they reference and so will be more or less randomly ordered by location in the text. Thus, processing them sequentially one at a time necessitates random access to the text. However, if we can read a sufficiently large block of pointers at one time and indirectly sort them by location in the text, then we can read a large number of keys from the text in a sequential pass. The larger the number of keys read in a sequential pass, the greater the improvement in performance. The greatest improvement is achieved when the entire memory is allocated to this key reading for each merge file in turn. The keys are then written out to temporary disk space. These keys are then merged by making a sequential pass over all the temporary files.

Thus, there are two constraints affecting this algorithm: the size of memory and the amount of temporary disk space available. They must be balanced by the relationship

memory   [image]  number of files = temporary disk space

We must also consider how many characters are necessary for each comparison. This will be dependent on the text being indexed. In the case of the OED, the longest comparison necessary was 600 characters. Clearly it is wasteful to read (and write) 600 characters for each key being put in a temporary file. In fact, we used 48 characters as a number that allowed about 97 percent of all comparisons to be resolved, and others were flagged as unresolved in the final index and fixed in a later pass. With 48 characters per key it was possible to read 600,000 keys into a 32Mb memory. For a 600Mb text, this means that on average we are reading 1 key per kilobyte, so we can use sequential I/O. Each reading of the file needs between 30 and 45 minutes and for 120,000,000 index points it takes 200 passes, or approximately 150 hours. Thus, for building an index for the OED, this algorithm is not as effective as the large against small indexing described earlier.

However, this algorithm will work for the general case of merging indices, whether they are existing large indices for separate text files, being merged to allow simultaneous searching, or whether they are partial indices for a large text file, each constructed in memory and being merged to produce a final index. In the latter case, there are some further improvements that can be made.

Since partial indices were constructed in memory, we know that the entire text fits in memory. When we are reading keys, the entire text may be loaded into memory. Keys may then be written out to fill all available temporary disk space, without the constraint that the keys to be written must fit in memory.

Another improvement in this situation can be made by noticing that lexicographically adjacent keys will have common prefixes for some length. We can use "stemming" to reduce the data written out for each key, increasing the number of keys that can be merged in each pass.

5.6 SUMMARY

We finish by comparing PAT arrays with two other kind of indices: signature files and inverted files.

Signature files use hashing techniques to produce an index being between 10 percent and 20 percent of the text size. The storage overhead is small; however, there are two problems. First, the search time on the index is linear, which is a drawback for large texts. Second, we may find some answers that do not match the query, and some kind of filtering must be done (time proportional to the size of the enlarged answer). Typically, false matches are not frequent, but they do occur.

On the other hand, inverted files need a storage overhead varying from 30 percent to 100 percent (depending on the data structure and the use of stopwords). The search time for word searches is logarithmic. Similar performance can be achieved by PAT arrays. The big plus of PAT arrays is their potential use in other kind of searches, that are either difficult or very inefficient over inverted files. That is the case with searching for phrases (especially those containing frequently occurring words), regular expression searching, approximate string searching, longest repetitions, most frequent searching, and so on. Moreover, the full use of this kind of index is still an open problem.

REFERENCES

AHO, A., HOPCROFT, J., and J. ULLMAN. 1974. The Design and Analysis of Computer Algorithms. Reading, Mass.: Addison-Wesley.

APOSTOLICO, A., and W. SZPANKOWSKI. 1987. "Self-alignments in Words and their Applicatons." Technical Report CSD-TR-732, Department of Computer Science, Purdue University, West Lafayette, Ind. 47907.

BAEZA-YATES, R., and G. GONNET. 1989. "Efficient Text Searching of Regular Expressions," in ICALP'89, Lecture Notes in Computer Science 372, eds. G. Ausiello, M. DezaniCiancaglini, and S. Ronchi Della Rocca. pp. 46-62, Stresa, Italy: Springer-Verlag. Also as UW Centre for the New OED Report, OED-89-01, University of Waterloo, April, 1989.

FAWCETT, H. 1989. A Text Searching System: PAT 3.3, User's Guide. Centre for the New Oxford English Dictionary, University of Waterloo.

FLAJOLET, P. and R. SEDGEWICK. 1986. "Digital Search Trees Revisited." SIAM J Computing, 15; 748-67.

GONNET, G. 1983. "Unstructured Data Bases or Very Efficient Text Searching," in ACM PODS, vol. 2, pp. 117-24, Atlanta, Ga.

GONNET, G. 1984. Handbook of Algorithms and Data Structures. London: Addison-Wesley.

GONNET, G. 1987. "PAT 3.1: An Efficient Text Searching System. User's Manual." UW Centre for the New OED, University of Waterloo.

GONNET, G. 1988. "Efficient Searching of Text and Pictures (extended abstract)." Technical Report OED-88-02, Centre for the New OED., University of Waterloo.

HOPCROFT, J., and J. ULLMAN. 1979. Introduction to Automata Theory. Reading, Mass.: Addison-Wesley.

KNUTH, D. 1973. The Art of Computer Programming: Sorting and Searching, vol. 3. Reading, Mass.: Addison-Wesley.

MANBER, U., and R. BAEZA-YATES. 1991. "An Algorithm for String Matching with a Sequence of Don't Cares." Information Processing Letters, 37; 133-36.

MANBER, U., and G. MYERS. 1990. "Suffix Arrays: A New Method for On-line String Searches," in 1st ACM-SIAM Symposium on Discrete Algorithms, pp. 319-27, San Francisco.

MORRISON, D. 1968. "PATRICIA-Practical Algorithm to Retrieve Information Coded in Alphanumeric." JACM, 15; 514-34.

PITTEL, B. 1985. "Asymptotical Growth of a Class of Random Trees." The Annals of Probability, 13; 414-27.

Go to Chapter 6     Back to Table of Contents


Information Retrieval: CHAPTER 6: FILE ORGANIZATIONS FOR OPTICAL DISKS

 Return to Table of Contents

CHAPTER 6: FILE ORGANIZATIONS FOR OPTICAL DISKS

Daniel Alexander Ford and Stavros Christodoulakis

Department of Computer Science, University of Waterloo, Waterloo, Ontario, Canada

Abstract

Optical disk technology is a new and promising secondary storage technology. Optical disks have immense capacities and very fast retrieval performance; they are also rugged and have very long storage lifetimes. These characteristics are making them a serious threat to the traditional dominance of magnetic disks. It is important to understand and study this young and significant technology and to design retrieval structures that best utilize its characteristics.

This chapter first presents a tutorial on optical disk technology and a discussion of important technical issues of file system design for optical disks, many of which are quite subtle. It then proceeds to discuss six file systems that have been developed for optical disks, including the Write-Once B-Tree of Easton (1986), the Time Split B-Tree of Lomet and Salzberg (1989), the Compact Disk File System of Garfinkel (1986), the Optical File Cabinet of Gait (1988), Buffered Hashing of Christodoulakis and Ford (1989a), and BIM trees of Christodoulakis and Ford (1989b).

6.1 INTRODUCTION

In this section we discuss file structures for optical disks. We first present an overview of optical disk technology, explaining where it came from and how it works. We then discuss some technical issues affecting the implementation of file structures on some common forms of optical disk technology. Later, we discuss in detail a variety of different structures that have been developed.

6.2 OVERVIEW OF OPTICAL DISK TECHNOLOGY

The foundation for all current optical disk technology was formed in the late 1960s and early 1970s by early video disk research. In mid-1971, N. V. Philips began conducting experiments in recording video signals on a flat glass plate using a spiral track of optically detectable depressions. This system was refined until it could store 30 minutes of color video and sound, and was called Video Long Play (VLP); it was first demonstrated on September 5, 1972. At about this time, other companies also began research efforts, including M.C.A., 3-M, Thomson-CSF, KODAK and Harris, among others.

The development of small inexpensive semiconductor laser diodes in 1975 stimulated development further, and the use of plastic disk platters which were cheaper and more easily replicated than glass platters was pioneered. Eventually, the technological base for the development of the optical disks in use today was emerged from the research efforts in the fields of optics, disk material technology, tracking and focus control servo system and lasers.

Features and benefits

Optical disks have many features and benefits that make them a useful storage medium. In general, their immense capacity, fast random access, and long storage life are major advantages. Other advantages are their portability and durability. Disks usually come encased in a protective cassette that can be easily carried. They are not subject to wear or head crashes as there is no physical contact between the disk and the access mechanism. The recording surface itself is not exposed, but rather, is protected by the plastic substrate that forms the disk platter. Optical disks, including erasable optical disks which employ a magneto-optic process for data registration, are also completely unaffected by magnetic fields.

The integrity of data stored on optical disks is also impressive. The process by which data is recorded on a Write-Once Read Many (WORM) disk surface causes an irreversible physical change. There is no chance of accidental erasure and overwrites are usually prevented by the drive or by driver software. Erasable optical disks cannot prevent accidental or malicious data destruction, but are still more durable then magnetic media. CD-ROM (Compact Disk Read Only Memory) disks are physically pressed out of plastic and so cannot be written to, or erased. Further, since there is no physical contact between an optical disk platter and the access mechanism, the disk is not subject to wear with use (the head of a Winchester type magnetic disk actually rides on the lubricated disk surface before it reaches flying speed). Small scratches and dust on the disk surface do not affect stored data as they are far enough away from the recording surface to be out of the optical system's focal plane. Data is also not subject to destruction from external magnetic fields and does not require periodic rewriting as is the case for magnetic media. The expected lifetime of an optical disk is not really known for certain (they have not been around long enough), but accelerated aging tests place it at least ten years and possibly as high as thirty. In contrast, the lifetime of magnetic tape is between two years to ten years, depending how it is handled and stored.

Optical disks also have the advantage of being removable. WORM and erasable disks are encased in a sturdy protective cassette when not in the drive and can be easily and safely transported, as can CD-ROM disks, which are designed to endure unrestrained consumer use.

There are few disadvantages inherent to optical disk technology. In fact, some characteristics that might be considered undesirable are actually advantages in certain applications. The unerasability of WORM type optical disks, for instance, make them the storage medium of choice for archival type applications. Some records such as transaction logs, banking records, and school transcripts are never legitimately altered, so some systems that maintain these types of records use WORM disks to ensure this.

Technical description

Optical disks are typically available in two different access configurations, either mounted in a single disk drive or as a set of disks in a "jukebox." In the single drive arrangement, disks are mounted manually be inserting the disk (and its protective cassette for WORM and erasable) into a slot on the front of the drive. For WORM disks, the platter is actually released and the cassette withdrawn; to remove the disk, the process is reversed. In the jukebox arrangement, the disks are stored in groups of several tens or even hundreds, representing many gigabytes of storage. They are selected and mounted mechanically in a manner similar to that used in audio disk jukeboxes. The time needed to switch a disk is about 5 to 20 seconds. For both types of configurations, the drive, disks, associated access mechanisms, and optics are the same.

When mounted, the disk is clamped to a spindle that protrudes through a large hole in its center. The access mechanism is usually a sled that slides along a guide path beneath the rotating disk platter. The sled contains the laser and associated optics. Note that only one side of the disk surface can be accessed when the disk is mounted. To access the other side, the disk must be physically removed, turned over, and remounted. CD-ROM disks are single sided and cannot be accessed if improperly mounted.

Electrical interfacing to the disk drives is usually done via standard interfaces such as the Small Computer Serial Interface (SCSI), the Enhanced Small Device Interface (ESDI), or more commonly for small personal computers through a direct bus interface.

There are three common sizes and storage capacities of optical disks available. The 305.5 millimeter (12 inch) platter is the most widely used for WORM disks and has a capacity of roughly 1 gigabyte per disk side. WORM disks are also available in 203 millimeter (8 inch, 750 megabytes) and 130 millimeter (5.25 inch, 200 megabytes) sizes. Erasable disks are available in 130 millimeter (5.25 inch, 600 megabytes) and, depending on the manufacturer, 86, 87, 88, or 89 millimeter (3.5 inch, 80 megabytes) sizes. The CD-ROM disk platters are standardized at 130 millimeters. The exact capacity of a disk will depend on the recording format employed (discussed later); some formats will increase the values stated above.

The way in which a disk platter is constructed varies with the type of optical disk. WORM disks usually use a construction know as the "air sandwich." This fabrication technique joins two transparent 1.5 millimeter thick platter halves together leaving a small "clean room" air gap sandwiched between them where the recording surfaces reside. The two halves both support and protect the recording surfaces while allowing light from the laser to reach them. The final disk platter is about 3 to 4 millimeters thick. Erasable disks are fabricated in a similar manner.

CD-ROM disks are single sided and are essentially a smaller version of one of the platter halves used for WORM disks. Data is permanently registered on a CD-ROM disk when it is fabricated by a pressing process. After pressing, the disk is given a thin coating of aluminum to make it reflective and is then sealed.

Materials and registration techniques for WORM disks

A single-sided optical disk platter, which is one side of an "air sandwich," consists of a thin film of Tellurium alloy (10-50 nanometers) that forms the active recording surface, and a supporting plastic substrate, usually poly(vinyl chloride) or poly(methyl methacrylate).

The disk is formed using an injection molding process. This is done under clean room conditions as the high storage densities of optical disks makes them particularly sensitive to contaminants like dust. It is during the molding process that the disk is pregrooved and sometimes also preformatted with address and synchronization information. These steps are important as they define the positions of tracks and identify individual sectors on what would otherwise be a featureless surface.

Data is recorded on the disk by a series of optically detectable changes to its surface. Small changes in reflectivity called "pits" are thermally induced in the active layer through the application of energy from a laser. The unaltered spaces between pits are called "lands." It takes 50-100 nanoseconds of 10 milliwatts of incident power focused on a 1 micron diameter spot to form the pit. Spacing between adjacent tracks of pits and lands is 1.6 microns.

There are several methods by which the applied energy can form pits in the active layer. Of these, the techniques called Ablation and Vesicular are favored.

With the Ablation technique, surface tension causes the thin film to pull away from the spot heated by the laser, leaving a hole in the surface. Sometimes, a layer of aluminum is deposited beneath the tellurium film to act as a mirror to be exposed when the hole is formed. With the Vesicular technique, a bubble is formed in the recording surface when vapor resulting from the heating process is trapped under the film. The bubble forms a bump that tends to disperse the light hitting it (making the pit darker).

Materials and registration techniques for erasable disks

Erasable optical disks are physically much like WORM disks. The main difference between the two is in the coating used on the recording surface. There are two types of erasable optical disk: Phase-Change and Magneto-Optic.

Erasable optical disks that employ phase-change technology rely on coatings for the recording surface consisting of thin films of tellurium or selenium. These coatings have the ability to exist in two different optically detectable states, amorphous and crystalline, and will switch between the two when heated to two different temperatures. If a spot on the recording surface is heated to a low temperature with a laser with 8 milliwatts of power, the spot will crystallize. If it is heated to a higher temperature with a laser with 18 milliwatts of power, the spot will melt and when it cools will revitrify to the amorphous state.

To register data on the recording surface, the power of the laser scanning the disk is simply modulated between 8 and 18 milliwatts. If a crystallized spot is scanned with 8 milliwatts of power, it will remain crystallized, if it is scanned with 18 milliwatts of power it will switch to the amorphous state. Similarly, if an amorphous spot is scanned with l8 milliwatts of power, it will remain in the amorphous state, and if it is scanned with 8 milliwatts of power it will switch to the crystallized state.

Reading data from the disk is simply a matter of scanning the surface with a low power laser (1 milliwatt) and detecting the changes in reflectivity that exist on the recording surface.

Erasable optical disks that employ magneto-optic technology store data on the disk magnetically, but read and write it optically. The coating used on the recording surface is a rare-earth transition-metal alloy such as terbium iron cobalt, terbium iron, or gadolinium terbium iron. It has the property of allowing the polarity of its magnetization to be changed when it is heated to a certain temperature (150 [image] C).

Recording data is a two-stage process requiring two passes of the disk head over a sector. The first pass serves to erase the contents of the sector. This is done by first placing a magnetic field with north-pole down in the vicinity of the spot upon which the laser focuses. As the sector is scanned, this spot will quickly heat to 150 [image] C and then immediately cool. As it does, domains of north-pole down are recorded throughout the sector; a north-pole down domain represents a 0 bit. Once the sector has been erased by the first pass, data can be written by the second.

When the sector is scanned a second time, the applied magnetic field is reversed. By modulating the power of the laser, selected portions of the sector can be heated to the required temperature and have their magnetizations reversed to north-pole up; a north-pole up domain represents a 1 bit. The remaining unheated portions of the sector retain their north-pole down magnetization.

Recorded data is read from the disk in a single pass. Reading relies on a physical effect known as the Kerr magneto-optic effect, which was discovered by Kerr (1877) and which causes light passing through a magentic field to become elliptically polarized. This effect allows the magnetization of a spot or domain of the disk surface to be detected optically. To read a disk sector, it is scanned with the laser in a lower power mode (1 milliwatt) than used for writing (8 milliwatts). If the magnetization of a domain being scanned is north-pole-up, the polarization of the light reflected from the surface will be rotated clockwise, if north-pole-down, counter-clockwise. The sequence of polarity changes is detected and interpreted to produce a bit stream.

The main stumbling block in the development of erasable magneto-optic disk technology was the chemical instability of the coating caused by its repeated heating and cooling during the write-erase cycle. This instability limited the number of cycles that could occur and was caused by the high temperature (600 [image] C) required to change the magnetization of a domain. The development of newer coatings that require lower temperatures (150 [image] C) solved this problem. Current erasable magneto-optic disks now allow some ten million or more write-erase cycles.

There are performance differences between the two types of erasable technologies. The Kerr effect only causes about a 1 percent change in the polarization of the reflected light. This requires more optics and associated hardware in the disk head of a magneto-optic disk drive to detect such a small change. The reflectivity difference between the two states of phase-change type erasable optical disks is relatively large and much easier to detect so the disk head can be much simpler. The net result is that the seek performance of a magneto-optic disk drive will generally be poorer than that of a phase-change drive because of its more massive disk head.

Optics

The optical assemblies found in all types of optical disk drives are similar in nature and resemble those of a medium power microscope. The task of the assembly is to focus a beam of coherent light from a semiconductor laser on to a 1 micron size spot and provide a return path for reflected light to reach a photodetector. This is a difficult feat to accomplish as the requirements for economical mass production of disks and drives imply a certain degree of flexibility in their precision. As such, it cannot be guaranteed that the disk platter will be perfectly flat or round, or that the hole for the drive's spindle will be perfectly centered. These imperfections can cause the outer edge of the disk to move up and down as much as 1 millimeter and the disk itself side-to-side as much as 60 micrometers (37 tracks) as it rotates several times a second. These motions cause the position of a track to vary with time in three dimensions and require the optical assemblies to be much more than a simple arrangement of lenses.

To follow the moving track, the objective lens is encased in a voice-coil like arrangement that allows it to be moved up and down to adjust its focus and from side to side to allow it to follow the wandering track. This radial flexibility gives most optical disk drives the ability to quickly access more than one track on the disk from a single position of the access mechanism, simply by adjusting the position of the objective lens. This viewing capability is usually limited to a window of some 10 to 20 tracks on either side of the current position of the access mechanism.

Lasers and light paths

The light source in an optical disk drive is a Gallium Arsenide (GaAlAs) laser diode with a wavelength of about 800 nanometers (.8 microns). The first experimental drives employed gas lasers (HeNe) which have a shorter wavelength and allowed higher storage densities and data transfer rates, but they were later abandoned in favor of the longer wavelength semiconductor laser diodes, which are cheaper and small enough to be mounted in the optical assembly. Laser diodes also have the advantage of being modulated electrically, eliminating the need for the expensive external acoustooptic or electrooptic modulator required by the gas lasers.

The recording density is limited by the wavelength of the laser because the wavelength determines the size of the smallest feature that can be resolved on the disk surface. The shorter the wavelength, the smaller the feature and hence, the higher the possible recording density.

The light path in the optical assembly consists of a collimating lens, a polarizing beam splitter, a quarter-wave plate, and an objective lens. The collimating lens takes the highly divergent light from the diode and forms a straight directed beam. This beam passes unchanged through the beam splitter and on to the quarter-wave plate that rotates its polarization by 90 degrees (a quarter of a wave). The altered beam then enters the objective lens that focuses it on the disk surface to the required 1 micron size spot.

On the return path, light reflected from the disk surface passes back through the objective lens and on again through the quarter-wave plate where its polarization is given another 90 degree twist. Now 180 degrees out of phase, the returning beam is not passed by the beam splitter but is instead reflected perpendicularly toward the photodetector.

Recording formats

While optical disks as a family share similar optical assemblies, disk fabrication materials and techniques, they can differ considerably in their recording formats. There are four different formats now in current use. The most common are the Constant Angular Velocity (CAV) and the Constant Linear Velocity (CLV) formats. The other two are modified versions of the above called, appropriately enough, Modified Constant Angular Velocity (MCAV) and Modified Constant Linear Velocity (MCLV; also called Quantized Linear Velocity--QLV).

In the CAV format, the sequences of pits and lands are usually arranged into equal capacity concentric tracks and the disk drive rotates the disk at a constant rate (angular velocity). This causes the length of both the pits and the lands to become elongated as their position moves away from the center of the disk because the outer surface of the disk platter passes beneath the optical assembly at a faster linear rate than does the inner surface. This elongation causes the storage density on the surface of a CAV format WORM disk to be less at the outer edge of the disk than at the center.

In the CLV format, the sequence of pits and lands forms a single spiral track and the rate at which the disk platter rotates is adjusted by the drive to match the position of the optical assembly. The disk rotates faster when accessing the inner surface and slower when accessing the outer surface. This adjustment ensures that the disk surface passes beneath the assembly and its optics at a constant rate (linear velocity). Adjusting the rotation rate prevents the pits and lands from becoming elongated and results in a constant storage density across the surface of the disk.

The two formats, CAV and CLV, have complimentary advantages and disadvantages. As stated above, CLV disks have a greater storage capacity than CAV disks, but they also tend to have slower seek times. This is a consequence of the extra time required to accelerate the disk to the rotation rate that matches the new position of the access mechanism. So while a CAV disk may store less data than a CLV disk, accesses to that data are slightly faster.

The Modified CAV and Modified CLV formats each combine features of the CAV and CLV formats in an attempt to obtain greater storage capacities and seek times. A disk employing the MCAV format rotates at a constant rate and also has a nearly constant storage density across its surface. Its concentric tracks are divided into equal track capacity bands. Each band has one more sector in each of its tracks than the next innermost band that it surrounds. A clocking scheme adjusts to the varying linear rates at which pits and lands pass beneath the optical assembly. The MCLV scheme is similar except that the disk platter rotates at a different rate for each band.

All the formats are available for WORM disks. Erasable disks are available in the CAV and MCAV formats. CD-ROM disks are standardized and use only the CLV format.

6.3 FILE SYSTEMS

Virtually all of the research into file systems for optical disks has concentrated on WORM and CD-ROM optical disks. These are the oldest forms of optical disk technology and the types with the most differences from magnetic disks. Commercially available erasable optical disks are a relatively recent phenomenon and are close enough in capabilities to magnetic disks that conventional file systems can usually be adapted. Thus, we concentrate our discussion on file systems developed for WORM and CD-ROM optical disks. But, before we proceed we first present a short discussion on some technical issues affecting the implementation of file structures on optical disks.

6.3.1 Technical Issues for File Systems

Optical disk technology is similar enough to magnetic disk technology that the same types of file structures that are used on magnetic disks can usually be used in some form on optical disks. There are, however, some differences that can make some choices better than others.

For example, conventional pointer linked file structures (e.g., B-trees) are a poor choice for WORM optical disks. The reason for this is that each modification of the file usually requires some of the pointers linking the file structure together to change; rebalancing a B-tree after an insert or delete is a good example. Since storage space cannot be reclaimed on a WORM optical disk, changing the value of a pointer requires the new value to be stored in a new disk sector, consuming space. Thus, in the course of normal file maintenance operations, extra storage space is consumed.

This type of problem is present when linked lists or trees are used (as is true for B-trees). If an element or node is modified, then it must be stored on the disk in a new position and all pointers to the position of the old version of the node must be updated to point to the position of the new version. Changing those pointers in turn changes their positions on the disk requiring any pointers to their old positions to be changed as well. This means that if an element of a list is modified, all elements between it and the head of the list must be duplicated. The same is true for trees: if a node is modified, all nodes on the path up to, and including, the root, must be duplicated.

This is not the only problem with using pointer linked structures on WORM optical disks; the direction that the pointers point is also of considerable consequence. A forward pointer is one that points from an older disk sector (written before) to a younger disk sector (written after), and a backward pointer points from a younger sector to an older sector. It is a characteristic of WORM optical disks that it is not possible to detect a bad disk sector before the sector has been written; thus, if a forward pointer is stored on the disk, there is a chance that the sector is points to may subsequently turn out to be unusable, making the forward pointer invalid. Sector substitution schemes that might deal with this problem can be envisioned, but a better solution is to simply avoid the problem where possible by using backward pointers. Backward pointers do not have this problem as they always point to valid sectors.

Preallocation of disk space on a WORM disk can also lead to problems. Most disk drives are unable to differentiate between reading a blank sector and reading a defective sector (i.e., they both are unreadable). Thus, if space has been reserved on a disk, it will be impossible to detect the difference between the beginning of preallocated (and blank) space on the disk and a previously written sector that is now unreadable (possibly because of a media defect, dirt, or scratches). This inability will render unreliable any organization that depends on preallocated space on a WORM disk.

A further aspect of WORM optical disk technology to consider when designing file structures is the granularity of data registration. On a WORM disk the smallest unit of data storage is the disk sector which has a typical capacity of 1 kilobyte. It is not possible to write a portion of a disk sector at one time and then another portion of the same sector later. A sector may be written once, and only once, and can never be updated. This restriction comes from the error correction code added to the data when the disk sector is written. If a sector is modified after its initial write, its contents would become inconsistent with the error correction code and the drive would either correct the "error" without comment or report a bad sector. With the inability to update the contents of a disk sector, there is a danger of wasting the storage capacity of the disk through sector fragmentation.

Not all of the characteristics of optical disks lead to problems for the implementations of file structures. On CD-ROM disks, for example, since the data never changes some optimizations are possible. Also, the spiral track found on CLV format disks lends itself nicely to hash file organizations by allowing hash buckets of arbitrary size to be created, allowing bucket overflow to be eliminated.

6.3.2 Write-Once B-Tree

The Write-Once B-Tree (WOBT) of Easton (1986) is a variation of a B-tree organization developed for WORM disks. The difference between the two structures is the manner in which the contents of the tree's nodes are maintained.

In a conventional B-tree, when a node is modified, its previous value or state is discarded in favor of the new value. This practice recycles storage space, but is not desirable in all situations. For some applications the ability to access any previous state of the tree is a distinct advantage; the WOBT allows this.

The WOBT manages the contents of the tree nodes in a manner that preserves all versions of a node throughout the life of the tree. This is accomplished by not overwriting nodes when they change, but instead appending new time-stamped entries to the node. The most current state of the tree is represented by the latest version of each entry (including pointers) in a node.

When a node is split in a WOBT, only those entries that are valid at the current time are copied into the new version of the node. The old version is retained intact. Deletions are handled by inserting a deletion marking record that explicitly identifies the record to be deleted and its deletion time.

The diagram in Figure 6.1 illustrates a WOBT with three nodes containing the records C, D, F, G, and H. The diagram shows the root node labeled 1 and two children, nodes 2 and 3. The root has an extra entry used to link different versions of the root node together. Being the first root of the tree, node 1 has a NULL (0) pointer for this entry. The rest of the entries in the root point to the children of the root. The first data entry indicates that C is the highest record in node 2, and F the highest in node 3.

Figure 6.1: Write-once B-tree

When a record A is added to the tree, it is simply appended to node 2 in the space available for it, and an entry is propagated up to the parent of node 2, the root. This is illustrated in Figure 6.2.

Figure 6.2: Write-once B-tree after insertion of "A"

The root now contains two entries which point to node 2. When we access the tree with respect to the current time, we use only the most recent values of the record/pointer pairs, in this case A/2 and F/3, which are found by a sequential search of the node. If we access the tree with respect to a time before the insertion of A, we find the record/pointer pairs C/2 and F/3. The record A in node 2 would not be found in that search because its later time-stamp would disqualify it.

When we insert a further record "J" into the tree, we must split both node 3 and the root node 1. The result is illustrated in Figure 6.3. When we split node 3 we end up with two new nodes 4 and 5. When node 1 is "split" in this case, we can make room in the new node by not including the data/pointer pair C/2 which is now obsolete. This results in one new node, node 6. The extra entry in the new root node now points back to the old root, node 1, so that accesses with respect to a previous time are still possible. Node 3 is no longer in the current version of the tree, but can still be accessed as part of the old version of the tree (prior to the insertion of "J") through the old root.

Figure 6.3: Write-once B-tree after insertion of "J"

Features to note about the WOBT are that search times will be slower than for a conventional B-tree as extra time will be required for a sequential search of each node to determine the current values of the data/pointer pairs. The WOBT also has problems with sector fragmentation since all modifications of the tree are stored as they are made directly on the WORM disk. It is also not particularly efficient with frequent updates to a single record, since each update must be stored on the disk in at least one disk sector.

However, despite these drawbacks, the WOBT is a robust (using backward pointers only) and elegant solution to the problem of efficiently implementing a multiway tree on an indelible storage device.

6.3.3 Time-Split B-Tree

The Time-Split B-tree (TSBT) of Lomet and Salzberg (1989) is an enhancement of the Write-once B-tree that eliminates some of its problems while adding to its utility. The basic structure and operation of the TSBT are identical to that of the WOBT. The difference between the two is that the TSBT distributes the contents of the tree over both a magentic disk and a WORM optical disk, and employs a slightly different approach to node splitting that reduces redundancy.

Combining magnetic and optical storage technologies allows their individual strengths to complement each other. This is an idea also employed in the buffered hashing organization of Christodoulakis and Ford (1989a). Write-once optical disks have a very low cost per stored bit and, as well, ensure that data cannot be accidently or maliciously deleted. Magnetic disks offer faster access and allow modifications to stored data without consuming storage space. A TSBT employs a magnetic disk to store the current changeable contents of the tree and the write-once disk to store the unchangeable historical contents.

The migration of the historical contents of the TSBT to the write-once disk is a consequence of node splitting. The TSBT divides node splits into two types: Time Splits and Key Splits. A time split occurs when a node is full of many historical (i.e., not in the current version of the tree) entries. A time split makes a new version of the node that omits the historical versions of records. In the TSBT, the old version of the node is removed from the magnetic disk and stored on the WORM optical disks.

A key split is the normal type of split associated with conventional B-trees and occurs when a node is full and most of the records are current (in a conventional B-tree, the records are always current). The result is two new nodes stored on the magnetic disk, each with roughly half of the records (historical and current) of the original node.

The advantages of the Time-split B-tree over the Write-once B-tree are many.The magnetic disk allows faster access to the current data than is possible if it were stored on an optical disk alone. It also allows transactions to make nonpermanent entries before they commit; these can be deleted if the transaction aborts. Records can also be updated without consuming disk space. Lower sector fragmentation is also a result, since buffering on the magnetic disk tends to reduce fragmentation by buffering many small changes into one larger change.

The splitting policy of a TSBT can be skewed to favor different objectives. If the total amount of storage space consumed is a concern, then key space splits should be favored as time splits tend to increase redundancy in the database. If the size of the current version of the B-tree (i.e., the part of the tree being stored on the magnetic disk) is a concern, then time splits should be favored as they free occupied space on the magnetic disk.

When implementing the TSBT, care must be taken to recognize that the size of the tree is limited not by the capacity of the optical disk, but by the capacity of the magnetic disk, as it stores all of the current contents of the tree. If the magnetic disk is full, the amount of remaining storage space on the optical disk will be irrelevant.

6.3.4 Compact Disk File System

The Compact Disk File System (CDFS) of Garfinkel (1986) is a system independent hierarchical file system for WORM optical disks. The goals of the CDFS are to be completely transportable across a variety of modern operating systems, to make efficient use of storage space, and to have a relatively high retrieval performance.

Unlike the write-once and time-split B-trees, the CDFS does not provide a structure for the organization of records, but rather a structure for the organization of groups of complete files. The application for which it is primarily intended is to organize files that experience few modifications, such as those belonging to a source code archive. The smallest unit of registration in the CDFS organization is the file.

The basic unit of organization in the CDFS is called a "transaction." A transaction results from the process of writing a complete group of files on the optical disk. All the files in a transaction group are placed on the disk immediately adjacent to the position of the previous transaction. Each individual file is stored contiguously. At the end of a transaction, an updated directory list for the entire file system is stored along with an "End of Transaction" (EOT) record. The EOT record contains a link to the EOT record of the previous transaction allowing access to historical versions of the organization (a dummy EOT record is stored at the start of an empty disk). The last transaction on the disk is the starting point for all accesses and the directory list it contains represents the current version of the file hierarchy.

The CDFS contains three types of "files": regular files, directories, and links. Each file is given a unique sequence number for the file system and a version number. If a file is updated by writing a new copy of it at some later time, the new copy retains the sequence number, but receives a new version number.

Each stored file entry consists of two parts, a file header and a file body. The header, which is invisible to a user, stores a large amount of explicit information about the file. This is an attempt by the CDFS to span the entire space of file charecteristics that any given operating system might record or require. For example, the file header contains the name of the owner of the file; on some systems (e.g., UNIX) this information must be derived by consulting a system database. This explicit information allows the contents of a single disk employing a CDFS to appear to be a native file system on more than one operating system (with appropriate drivers for each system).

A directory is a special type of file that contains entries identifying other files known as members. These entries include pointers to the disk positions of the members and other information such as file sizes and modification times. This extra information is redundant since it is also stored in the file header, but it serves to improve the performance of directory list operations by eliminating the seeks required to access each member.

A link entry is simply a pointer to a file or a directory and allows a file to be a member of more than one directory.

The directory list stored at the end of the files in the transaction is an optimization to reduce seeks. It is a list of the positions of all current directories and subdirectories in the hierarchical file system. Using the directory list improves performance by reducing the seeks needed to traverse the file directory tree.

The diagram in Figure 6.4 illustrates how an instance of a CDFS is stored in a series of disk sectors. The example is for two transactions for a CDFS consisting of three files. The second transaction is used to store a second expanded version of the second file in the file system. The arrows in the diagram represent the pointers which link the various constituents of the CDFS together. For example, (backward) pointers exist between the EOT records, between each EOT record and its directory list, between the directory list and the directories (in this case just one, the root), and between the root and the three files.

Figure 6.4: State of compact disk file system after two transactions

The CDFS is an efficient means of organizing an archive of a hierarchical file system. Its main drawback is that it does not allow efficient updates to files. Any change to a single file requires the entire file to be rewritten along with a new copy of the directory list. Extra information could be added to the file header to allow files to be stored noncontiguously. This would allow portions of the file to be changed while other parts remained intact.

The robustness of the CDFS inherent in the degree of redundancy found in the organization, coupled with the access it allows to all previous versions of a file, makes it ideal for use in storing file archives.

Being relatively system independent, the CDFS is also an excellent organization for data interchange. It would be possible, for example, to copy a complete UNIX file system to an optical disk employing the CDFS and then transport it to an appropriate VMS installation and access the archived UNIX file system as if it were a native VMS file system.

6.3.5 The Optical File Cabinet

The Optical File Cabinet (OFC) of Gait (1988) is another file system for WORM optical disks. Its goals are quite different from those of the CDFS described previously. Its main objective is to use a WORM disk to simulate an erasable file system such as that found on a magnetic disk, and appear to an operating system just as if it was any other magnetic disk file system. It does this by creating a logical disk block space which can be accessed and modified at random on a block-by-block basis through a conventional file system interface.

The mapping between the logical and physical blocks is provided by a structure called the File System Tree (FST) which resides on the WORM optical disk, (see Figure 6.5). To translate between logical and physical blocks, the logical block number is used to find a path through the FST to a leaf; the leaves of the tree are the physical disk blocks of the WORM optical disk.

Figure 6.5: File system tree (FST) for optical file cabinet

Both the interior nodes and the leaves of the tree are buffered in main memory. The buffers are periodically flushed (e.g., every 5 minutes) and written on the optical disk in a process called checkpointing. Each flush results in a new version of the FST residing on the WORM disk. The roots of each of the different versions of the FST are pointed to by members of a list also residing on the optical disk called the time-stamp list. The current version of the FST is found by traversing the time-stamp list to find its last element.

The time-stamp list is implemented by a "continuation list." A continuation list consists of a series of disk blocks linked together via forward pointers. The pointer in the last element of a continuation list contains a pointer to a preallocated but empty disk block. The end of the list is detected when an attempt to read the next disk block in the list fails. In the OFC, it is assumed that the reason for the failure is because the preallocated (and empty) sector was read.

The use of forward pointers by the Optical File Cabinet file system seems to be a major flaw. As discussed previously, forward pointers are unreliable on WORM optical disks. It would also appear that if any of the disk sectors in which the time-stamp list is stored were ever damaged (e.g., by a permanent scratch or temporary dirt), the entire organization would be crippled. The system would interpret the bad sector read as the end of the time-stamp list and access historical contents of the file system rather than the current contents. At the next checkpoint time it would attempt to write to the damaged sector and find itself in a dead end. Even if none of the current sectors of the time-stamp list are damaged, the list could still encounter a preallocated sector that is damaged (e.g., media defect). An error such as this would require the entire time-stamp list to be copied and a scheme for identifying and locating alternate time stamp lists on the disk.

The utility of the Optical File Cabinet file system is also difficult to establish. As a replacement for magnetic disks it is an expensive substitute, as the write-once disks eventually fill up and must be replaced.

The most appropriate application seems to be in fault-tolerant systems that require their entire file systems to be permanently checkpointed at frequent intervals.

For systems with less demanding requirements the Write-once B-tree or the Time-split B-tree implemented for specific files might be more appropriate. The same is true for the archiving of copies of specific files (e.g., source code), where an organization such as the Compact Disk File System would be a better choice.

6.3.6 Buffered Hashing (Bhash)

Buffered Hashing (Bhash) of Christodoulakis and Ford (1989a) is a Hash file organization for WORM optical disks which employs a rewritable buffer to obtain performance improvements and reductions in storage space consumption. The buffer helps to reduce sector fragmentation, but its main purpose is to group many small changes to the file (record insertions) into one larger insertion. This is important because preallocating space for a hash bucket, as is done on magnetic disks, is not really a viable option for a WORM disk. This is because it is difficult to distinguish between a preallocated empty disk sector and an occupied but damaged sector; both are read as bad sectors.

Since successive insertions are unlikely to be to the same hash bucket, it is likely that the contents of a hash bucket will be stored in different (unpredictable) spots spread around the disk. By having a larger insertion, the degree to which the contents of a bucket are spread around the disk as a function of the number of record insertions is reduced. This, in turn, will reduce the number of seeks required to retrieve the contents of a hash bucket.

The rewritable buffer employed can be either main memory or a magnetic disk. As records are added to the file, they are first stored in the buffer and linked into a list of other records that belong to their hash bucket. When the buffer is full, the largest group of records belonging to a single hash bucket are removed (flushed) from the buffer and stored as a contiguous unit on the WORM optical disk. This group is linked to a list on the optical disk of other such groups that were flushed previously and that belong to the same bucket. The space in the buffer freed by the flush is then reused to store more records belonging to any hash bucket.

The length of the list (number of different groups) on the WORM disk will determine the number of seeks required to complete the retrieval of a bucket. If adding a new group to the list would make the length exceed some retrieval performance limit on the number of seeks required to access a bucket, the groups of the list are merged together to form a single larger contiguous group. This group is stored on the WORM disk as well, and is used in subsequent retrievals of the bucket. The new bucket group also becomes the first member of a new group list, to which subsequent bucket flushes will be appended. When the length of that list exceeds the limit, it too will be merged.

Deleted records are either removed from the buffer if they have not yet been flushed at the time of the deletion, or are marked as deleted by storing a deletion record in the bucket.

Because pointers to all previous groups on the disk are available and can be stored on a magnetic disk (such as the one being used for the buffer), access to all previous versions of the database is possible.

The parameters of the organization can be adjusted to obtain either fast retrieval performance at the expense of high storage space consumption or low disk space consumption at the expense of slower retrieval performance. Retrieval performance is affected primarily by the length limits placed on the group lists. If high performance is desired, the allowed length of the lists will be very short, resulting in many merges and increased storage consumption. If low storage consumption is required, the allowed length of the lists will be quite long, resulting in very infrequent merges but more seeks.

The size of the buffer and the number of buckets also plays a role in determining the performance of the organization. If the buffer is large, it will obviously require less flushing and hence fewer merges will occur and less space will be consumed. If the number of buckets is very large, then we can expect that the size of a group flushed from the buffer will be small, so flushes will occur more often, this will consume more space since it will increase the number of merges and in turn the number of redundantly stored records.

The Bhash file organization is a good method for implementing a hash file organization on a WORM optical disk. It can be tuned by an implementor to make efficient use of storage space or to meet strict retrieval performance demands. It also can function as a roll-back database, giving access to all previous versions of each record ever stored in the hash file.

6.3.7 Balanced Implicit Multiway Trees (BIM Trees)

The static nature of data stored on a CD-ROM optical disk allows conventional B-tree structures to be fine tuned to produce a completely balanced multiway tree that does not require storage space for pointers. Such a tree is called a Balanced Implicit Multiway Tree or a BIM tree, as presented in Christodoulakis and Ford (1989b). Because all the data to be stored on a CD-ROM disk is available at the time the disk is created, it is possible to preconstruct a perfectly balanced multitree for the data. And given that the size of all nodes of the tree and their layout is known, no pointers need to be stored since it is easy to compute the position on the disk for a given node in the tree.

When constructing a BIM tree, it is possible to choose node sizes and layouts that will improve the expected retrieval performance for accesses from the tree. If the node size is chosen such that a parent node and all of its children can fill within the viewing window of the objective lens of the disk drive, it will be possible to eliminate the seek required to traverse the link between the parent and one of its children. For example, with the root buffered in main memory and each of the second level nodes stored with all of their children within the viewing window of the objective lens, only a single seek, to the second level, would be required to retrieve any record within a three-level BIM tree.

6.3.8 Hashing for CD-ROM and CLV Format Optical Disks

The spiral track that is characteristic of Constant Linear Velocity (CLV) format optical disks such as CD-ROM is ideal for implementing hashing file organizations and can guarantee single seek access as discussed in Christodoulakis and Ford (1989b).

The biggest complication in a hashing organization, and the biggest source of performance degradation, is the resolution of hash bucket overflows. Buckets overflow because they are usually associated with a physical division on the storage device that has a finite capacity, either a track or a cylinder. The spiral track, which can be read continuously for the entire capacity of CLV format disks, allows hash buckets to be as large or as small as is necessary to store their contents. With a spiral track there is no need to impose an arbitrary physical limit (up to the capacity of the disk) on a bucket. This is particularly true for CD-ROM disks on which the data never changes.

On such disks, hash buckets can be laid out along the spiral track one after another. To determine the position of each bucket, a small bucket position translation table recording the beginning position of each bucket can be used. With the translation table in main memory, the contents of a bucket can be accessed with a single seek. This last feature is particularly attractive for optical disks, as they have relatively slow seek times.

REFERENCES

CHRISTODOULAKIS, S., and D. A. FORD. 1989a, June. Retrieval Performance Versus Disc Space Utilization on WORM Optical Discs. Paper presented at the annual meeting of the Special Interest Group for the Management of Data of the Association of Computing Machinery (ACM SIGMOD'89), Portland, Oregon, 306-14.

CHRISTODOULAKIS, S., and D. A. FORD. 1989b, June. File Organizations and Access Methods for CLV Optical Disks. Paper presented at the annual meeting of the Special Interest Group for Information Retrieval of the Association of Computing Machinery (ACM SIGIR'89), Cambridge, Massachusetts.

EASTON, M. C. 1986. "Key-Sequence Data Sets on Indelible Storage." IBM Journal of Research and Development, 30(3), 230-41.

GAIT, J. 1988. "The Optical File Cabinet: A Random-Access File System for Write-Once Optical Disks." Computer, 21 (6), 11-22.

GARFINKEL, S. L. 1986. "A File System For Write-Once Media." Technical Report MIT Media Lab, September.

KERR, J. 1877. "On the Rotation of the Plane of Polarization by Reflection from the Pole of a Magnet. Philosophical Magazine, 3, 321-43.

LOMET, D., and B. SALZBERG 1989, June. Access Method for Multiversion Data. Paper presented at the annual meeting of the Special Interest Group for the Management of Data of the Association of Computing Machinery (ACM SIGMOD'89), Portland, Oregon, 315-24.

Go to Chapter 7 Back to Table of Contents


Information Retrieval: CHAPTER 7: LEXICAL ANALYSIS AND STOPLISTS

 Return to Table of Contents

CHAPTER 7: LEXICAL ANALYSIS AND STOPLISTS

Christopher Fox

AT&T Bell Laboratories, Holmdel, NJ 07733

Abstract

Lexical analysis is a fundamental operation in both query processing and automatic indexing, and filtering stoplist words is an important step in the automatic indexing process. This chapter presents basic algorithms and data structures for lexical analysis, and shows how stoplist word removal can be efficiently incorporated into lexical analysis.

7.1 INTRODUCTION

Lexical analysis is the process of converting an input stream of characters into a stream of words or tokens. Tokens are groups of characters with collective significance. Lexical analysis is the first stage of automatic indexing, and of query processing. Automatic indexing is the process of algorithmically examining information items to generate lists of index terms. The lexical analysis phase produces candidate index terms that may be further processed, and eventually added to indexes (see Chapter 1 for an outline of this process). Query processing is the activity of analyzing a query and comparing it to indexes to find relevant items. Lexical analysis of a query produces tokens that are parsed and turned into an internal representation suitable for comparison with indexes.

In automatic indexing, candidate index terms are often checked to see whether they are in a stoplist, or negative dictionary. Stoplist words are known to make poor index terms, and they are immediately removed from further consideration as index terms when they are identified.

This chapter discusses the design and implementation of lexical analyzers and stoplists for information retrieval. These topics go well together because, as we will see, one of the most efficient ways to implement stoplists is to incorporate them into a lexical analyzer.

7.2 LEXICAL ANALYSIS

7.2.1 Lexical Analysis for Automatic Indexing

The first decision that must be made in designing a lexical analyzer for an automatic indexing system is: What counts as a word or token in the indexing scheme? At first, this may seem an easy question, and there are some easy answers to it--for example, terms consisting entirely of letters should be tokens. Problems soon arise, however. Consider the following:

Digits--Most numbers do not make good index terms, so often digits are not included as tokens. However, certain numbers in some kinds of databases may be important (for example, case numbers in a legal database). Also, digits are often included in words that should be index terms, especially in databases containing technical documents. For example, a database about vitamins would contain important tokens like "B6" and "B12." One partial (and easy) solution to the last problem is to allow tokens to include digits, but not to begin with a digit.

Hyphens--Another difficult decision is whether to break hyphenated words into their constituents, or to keep them as a single token. Breaking hyphenated terms apart helps with inconsistent usage (e.g., "state-of-the-art" and "state of the art" are treated identically), but loses the specificity of a hyphenated phrase. Also, dashes are often used in place of ems, and to mark a single word broken into syllables at the end of a line. Treating dashes used in these ways as hyphens does not work. On the other hand, hyphens are often part of a name, such as "Jean-Claude," "F-16," or "MS-DOS."

Other Punctuation--Like the dash, other punctuation marks are often used as parts of terms. For example, periods are commonly used as parts of file names in computer systems (e.g., "COMMAND.COM" in DOS), or as parts of section numbers; slashes may appear as part of a name (e.g., "OS/2"). If numbers are regarded as legitimate index terms, then numbers containing commas and decimal points may need to be recognized. The underscore character is often used in terms in programming languages (e.g., "max_size" is an identifier in Ada, C, Prolog, and other languages).

Case--The case of letters is usually not significant in index terms, and typically lexical analyzers for information retrieval systems convert all characters to either upper or lower case. Again, however, case may be important in some situations. For example, case distinctions are important in some programming languages, so an information retrieval system for source code may need to preserve case distinctions in generating index terms.

There is no technical difficulty in solving any of these problems, but information system designers must think about them carefully when setting lexical analysis policy. Recognizing numbers as tokens adds many terms with poor discrimination value to an index, but may be a good policy if exhaustive searching is important. Breaking up hyphenated terms increases recall but decreases precision, and may be inappropriate in some fields (like an author field). Preserving case distinctions enhances precision but decreases recall.

Commercial information systems differ somewhat in their lexical analysis policies, but are alike in usually taking a conservative (recall enhancing) approach. For example, Chemical Abstracts Service, ORBIT Search Service, and Mead Data Central's LEXIS/NEXIS all recognize numbers and words containing digits as index terms, and all are case insensitive. None has special provisions for most punctuation marks in most indexed fields. However, Chemical Abstracts Service keeps hyphenated words as single tokens, while the ORBIT Search Service and LEXIS/NEXIS break them apart (if they occur in title or abstract fields).

The example we use to illustrate our discussion is simple so it can be explained easily, and because the simplest solution often turns out to be best. Modifications to it based on the considerations discussed above are easy to make. In the example, any nonempty string of letters and digits, not beginning with a digit, is regarded as a token. All letters are converted to lower case. All punctuation, spacing, and control characters are treated as token delimiters.

7.2.2 Lexical Analysis for Query Processing

Designing a lexical analyzer for query processing is like designing one for automatic indexing. It also depends on the design of the lexical analyzer for automatic indexing: since query search terms must match index terms, the same tokens must be distinguished by the query lexical analyzer as by the indexing lexical analyzer. In addition, however, the query lexical analyzer must usually distinguish operators (like the Boolean operators, stemming or truncating operators, and weighting function operators), and grouping indicators (like parentheses and brackets). A lexical analyzer for queries should also process certain characters, like control characters and disallowed punctuation characters, differently from one for automatic indexing. Such characters are best treated as delimiters in automatic indexing, but in query processing, they indicate an error. Hence, a query lexical analyzer should flag illegal characters as unrecognized tokens.

The example query lexical analyzer presented below recognizes left and right parentheses (as grouping indicators), ampersand, bar, and caret (as Boolean operators), and any alphanumeric string beginning with a letter (as search terms). Spacing characters are treated as delimiters, and other characters are returned as unrecognized tokens. All uppercase characters are converted to lowercase.

7.2.3 The Cost of Lexical Analysis

Lexical analysis is expensive because it requires examination of every input character, while later stages of automatic indexing and query processing do not. Although no studies of the cost of lexical analysis in information retrieval systems have been done, lexical analysis has been shown to account for as much as 50 percent of the computational expense of compilation (Wait 1986). Thus, it is important for lexical analyzers, particularly for automatic indexing, to be as efficient as possible.

7.2.4 Implementing a Lexical Analyzer

Lexical analysis for information retrieval systems is the same as lexical analysis for other text processing systems; in particular, it is the same as lexical analysis for program translators. This problem has been studied thoroughly, so we ought to adopt the solutions in the program translation literature (Aho, Sethi, and Ullman 1986). There are three ways to implement a lexical analyzer:

Use a lexical analyzer generator, like the UNIX tool lex (Lesk 1975), to generate a lexical analyzer automatically;

Write a lexical analyzer by hand ad hoc; or

Write a lexical analyzer by hand as a finite state machine.

The first approach, using a lexical analyzer generator, is best when the lexical analyzer is complicated; if the lexical analyzer is simple, it is usually easier to implement it by hand. In our discussion of stoplists below, we present a special purpose lexical analyzer generator for automatic indexing that produces efficient lexical analyzers that filter stoplist words. Consequently, we defer further discussion of this alternative.

The second alternative is the worst. An ad hoc algorithm, written just for the problem at hand in whatever way the programmer can think to do it, is likely to contain subtle errors. Furthermore, finite state machine algorithms are extremely fast, so ad hoc algorithms are likely to be less efficient.

The third approach is the one we present in this section. We assume some knowledge of finite state machines (also called finite automata), and their use in program translation systems. Readers unfamiliar with these topics can consult Hopcroft and Ullman (1979), and Aho, Sethi, and Ullman (1986). Our example is an implementation of a query lexical analyzer as described above.

The easiest way to begin a finite state machine implementation is to draw a transition diagram for the target machine. A transition diagram for a machine recognizing tokens for our example query lexical analyzer is pictured in Figure 7.1.

In this diagram, characters fall into ten classes: space characters, letters, digits, the left and right parentheses, ampersand, bar, caret, the end of string character, and all other characters. The first step in implementing this finite state machine is to build a mechanism for classifying characters. The easiest and fastest way to do this is to preload an array with the character classes for the character set. Assuming the ASCII character set, such an array would contain 128 elements with the character classes for the corresponding ASCII characters. If such an array is called char_class, for example, then the character class for character 'c' is simply char_class [c]. The character classes themselves form a distinct data type best declared as an enumeration in C. Figure 7.2 contains C declarations for a character class type and array. (Note that the end of file character requires special treatment in C because it is not part of ASCII).

Figure 7.1: Transition diagram for a query lexical analyzer

The same technique is used for fast case conversion. In Figure 7.2, an array of 128 characters called convert_case is preloaded with the printing characters, with lowercase characters substituted for uppercase characters. Nonprinting character positions will not be used, and are set to 0.

/**************  Character Classification  *****************/

/* Tokenizing requires that ASCII be broken into character */

/* classes distinguished for tokenizing. White space       */

/* characters separate tokens. Digits and letters make up  */

/* the body of search terms. Parentheses group sub-        */

/* expressions. The ampersand, bar, and caret are          */

/* operator symbols.                                       */

typedef enum {

WHITE_CH,             /* whitespace characters */

DIGIT_CH,             /* the digits */

LETTER_CH,            /* upper and lower case */

LFT_PAREN_CH,         /* the "(" character */

RGT_PAREN_CH,         /* the ")" character */

AMPERSAND_CH,         /* the "&" character */

BAR_CH,               /* the "|" character */

CARET_CH,             /* the "^" character */

EOS_CH,               /* the end of string character */

OTHER_CH,             /* catch-all for everything else */

} CharClassType;

static CharClassType char_class[128] = {

/* ^@ */  EOS_CH,      /* ^A */  OTHER_CH,    /* ^B */  OTHER_CH,

/* ^C */  OTHER_CH,    /* ^D */  OTHER_CH,    /* ^E */  OTHER_CH,

/* ^F */  OTHER_CH,    /* ^G */  OTHER_CH,    /* ^H */  WHITE_CH,

/* ^I */  WHITE_CH,    /* ^J */  WHITE_CH,    /* ^K */  WHITE_CH,

/* ^L */  WHITE_CH,    /* ^M */  WHITE_CH,    /* ^N */  OTHER_CH,

/* ^O */  OTHER_CH,    /* ^P */  OTHER_CH,    /* ^Q */  OTHER_CH,

/* ^R */  OTHER_CH,    /* ^S */  OTHER_CH,    /* ^T */  OTHER_CH,

/* ^U */  OTHER_CH,    /* ^V */  OTHER_CH,    /* ^W */  OTHER_CH,

/* ^X */  OTHER_CH,    /* ^Y */  OTHER_CH,    /* ^Z */  OTHER_CH,

/* ^[ */  OTHER_CH,    /* ^\ */  OTHER_CH,    /* ^] */  OTHER_CH,

/* ^^ */  OTHER_CH,    /* ^_ */  OTHER_CH,    /*    */  WHITE_CH,

/*  ! */  OTHER_CH,    /*  " */  OTHER_CH,    /*  # */  OTHER_CH,

/*  $ */  OTHER_CH,    /*  % */  OTHER_CH,    /*  & */  AMPERSAND_CH,

/*  ' */  OTHER_CH,    /*  ( */  LFT_PAREN_C, /*  ) */  RGT_PAREN_CH,

/*  * */  OTHER_CH,    /*  + */  OTHER_CH,    /*  , */  OTHER_CH,

/*  - */  OTHER_CH,    /*  . */  OTHER_CH,    /*  / */  OTHER_CH,

/*  0 */  DIGIT_CH,    /*  1 */  DIGIT_CH,    /*  2 */  DIGIT_CH,

/*  3 */  DIGIT_CH,    /*  4 */  DIGIT_CH,    /*  5 */  DIGIT_CH,

/*  6 */  DIGIT_CH,    /*  7 */  DIGIT_CH,    /*  8 */  DIGIT_CH,

/*  9 */  DIGIT_CH,    /*  : */  OTHER_CH,    /*  ; */  OTHER_CH,

/*  < */  OTHER_CH,    /*  = */  OTHER_CH,    /*  > */  OTHER_CH,

/*  ? */  OTHER_CH,    /*  @ */  OTHER_CH,    /*  A */  LETTER_CH,

/*  B */  LETTER_CH,   /*  C */  LETTER_CH,   /*  D */  LETTER_CH,

/*  E */  LETTER_CH,   /*  F */  LETTER_CH,   /*  G */  LETTER_CH,

/*  H */  LETTER_CH,   /*  I */  LETTER_CH,   /*  J */  LETTER_CH,

/*  K */  LETTER_CH,   /*  L */  LETTER_CH,   /*  M */  LETTER_CH,

/*  N */  LETTER_CH,   /*  O */  LETTER_CH,   /*  P */  LETTER_CH,

/*  Q */  LETTER_CH,   /*  R */  LETTER_CH,   /*  S */  LETTER_CH,

/*  T */  LETTER_CH,   /*  U */  LETTER_CH,   /*  V */  LETTER_CH,

/*  W */  LETTER_CH,   /*  X */  LETTER_CH,   /*  Y */  LETTER_CH,

/*  Z */  LETTER_CH,   /*  [ */  OTHER_CH,    /*  \ */  OTHER_CH,

/*  ] */  OTHER_CH,    /*  ^ */  CARET_CH,    /*  _ */  OTHER_CH,

/*  ` */  OTHER_CH,    /*  a */  LETTER_CH,   /*  b */  LETTER_CH,

/*  c */  LETTER_CH,   /*  d */  LETTER_CH,   /*  e */  LETTER_CH,

/*  f */  LETTER_CH,   /*  g */  LETTER_CH,   /*  h */  LETTER_CH,

/*  i */  LETTER_CH,   /*  j */  LETTER_CH,   /*  k */  LETTER_CH,

/*  l */  LETTER_CH,   /*  m */  LETTER_CH,   /*  n */  LETTER_CH,

/*  o */  LETTER_CH,   /*  p */  LETTER_CH,   /*  q */  LETTER_CH,

/*  r */  LETTER_CH,   /*  s */  LETTER_CH,   /*  t */  LETTER_CH,

/*  u */  LETTER_CH,   /*  v */  LETTER_CH,   /*  w */  LETTER_CH,

/*  x */  LETTER_CH,   /*  y */  LETTER_CH,   /*  z */  LETTER_CH,

/*  { */  OTHER_CH,    /*  | */  BAR_CH,      /*  } */  OTHER_CH,

/*   */  OTHER_CH,    /* ^? */  OTHER_CH,                           };

/**************   Character Case Conversion   *************/

/* Term text must be accumulated in a single case. This   */

/* array is used to convert letter case but otherwise     */

/* preserve characters.                                   */

static char convert_case[128] = {

/* ^@ */    0,    /* ^A */    0,    /* ^B */    0,    /* ^C */     0,

/* ^D */    0,    /* ^E */    0,    /* ^F */    0,    /* ^G */     0,

/* ^H */    0,    /* ^I */    0,    /* ^J */    0,    /* ^K */     0,

/* ^L */    0,    /* ^M */    0,    /* ^N */    0,    /* ^O */     0,

/* ^P */    0,    /* ^Q */    0,    /* ^R */    0,    /* ^S */     0,

/* ^T */    0,    /* ^U */    0,    /* ^V */    0,    /* ^W */     0,

/* ^X */    0,    /* ^Y */    0,    /* ^Z */    0,    /* ^[ */     0,

/* ^\ */    0,    /* ^] */    0,    /* ^^ */    0,    /* ^_ */     0,

/*    */  ' ',    /*  ! */  '!',    /*  " */  '"',    /*  # */   '#',

/*  $ */  '$',    /*  % */  '%',    /*  & */  '&',    /*  ' */  ''',

/*  ( */  '(',    /*  ) */  ')',    /*  * */  '*',    /*  + */   '+',

/*  , */  ',',    /*  - */  '-',    /*  . */  '.',    /*  / */   '/',

/*  0 */  '0',    /*  1 */  '1',    /*  2 */  '2',    /*  3 */   '3',

/*  4 */  '4',    /*  5 */  '5',    /*  6 */  '6',    /*  7 */   '7',

/*  8 */  '8',    /*  9 */  '9',    /*  : */  ':',    /*  ; */   ';',

/*  < */  '<',    /*  = */  '=',    /*  > */  '>',    /*  ? */   '?',

/*  @  */ '@',    /*  A */  'a',    /*  B */  'b',    /*  C */   'c',

/*  D */  'd',    /*  E */  'e',    /*  F */  'f',    /*  G */   'g',

/*  H */   'h',   /*  I */  'i',    /*  J */  'j',    /*  K */   'k',

/*  L */   'l',   /*  M */  'm',    /*  N */  'n',    /*  O */    o',

/*  P */   'p',   /*  Q */  'q',    /*  R */  'r',    /*  S */   's',

/*  T */   't',   /*  U */  'u',    /*  V */  'v',    /*  W */   'w',

/*  X */   'x',   /*  Y */  'y',    /*  Z */  'z',    /*  [ */   '[',

/*  \ */  '\',   /*  ] */  ']',    /*  ^ */  '^',    /*  _ */   '_',

/*  ` */   '`',   /*  a */  'a',    /*  b */  'b',    /*  c */   'c',

/*  d */   'd',   /*  e */  'e',    /*  f */  'f',    /*  g */   'g',

/*  h */   'h',   /*  i */  'i',    /*  j */  'j',    /*  k */   'k',

/*  l */   'l',   /*  m */  'm',    /*  n */  'n',    /*  o */   'o',

/*  p */   'p',   /*  q */  'q',    /*  r */  'r',    /*  s */   's',

/*  t */   't',   /*  u */  'u',    /*  v */  'v',    /*  w */   'w',

/*  x */   'x',   /*  y */  'y',    /*  z */  'z',    /*  { */   '{' ,

/*  | */   '|',   /*  } */  '}',    /*  ~ */  '~',    /* ^? */     0, };

/********************   Tokenizing   ***********************/

/* The lexer distinguishes terms, parentheses, the and, or */

/* and not operators, the unrecognized token, and the end  */

/* of the input.                                           */

typedef enum {

TERM_TOKEN      = 1,    /* a search term */

LFT_PAREN_TOKEN = 2,    /* left parenthesis */

RGT_PAREN_TOKEN = 3,    /* right parenthesis */

AND_TOKEN       = 4,    /* set intersection connective */

OR_TOKEN        = 5,    /* set union connective */

NOT_TOKEN       = 6,    /* set difference connective */

END_TOKEN       = 7,    /* end of the query */

NO_TOKEN        = 8,    /* the token is not recognized */

} TokenType;

Figure 7.2: Declarations for a simple query lexical analyzer

There also needs to be a type for tokens. An enumeration type is best for this as well. This type will have an element for each of the tokens: term, left parenthesis, right parenthesis, ampersand, bar, caret, end of string, and the unrecognized token. Processing is simplified by matching the values of the enumeration type to the final states of the finite state machine. The declaration of the token type also appears in Figure 7.2.

The code for the finite state machine must keep track of the current state, and have a way of changing from state to state on input. A state change is called a transition. Transition information can be encoded in tables, or in flow of control. When there are many states and transitions, a tabular encoding is preferable; in our example, a flow of control encoding is probably clearest. Our example implementation reads characters from an input stream supplied as a parameter. The routine returns the next token from the input each time it is called. If the token is a term, the text of the term (in lowercase) is written to a term buffer supplied as a parameter. Our example code appears in Figure 7.3.

/*FN************************************************************************

GetToken( stream )

Returns: void

Purpose: Get the next token from an input stream

Plan:    Part 1: Run a state machine on the input

Part 2: Coerce the final state to return the token type

Notes:   Run a finite state machine on an input stream, collecting

the text of the token if it is a term. The transition table

for this DFA is the following (negative states are final):

             State  |  White  Letter  (    )  &   |   ^  EOS  Digit  Other

             -------------------------------------------------------------

               0    |   0      1     -2  -3  -4  -5  -6  -7    -8     -8

               1    |  -1      1     -1  -1  -1  -1  -1  -1     1     -1

See the token type above to see what is recognized in the

various final states.

**/

static TokenType

GetToken( stream, term )

FILE *stream;  /* in: where to grab input characters */

char *term;    /* out: the token text if the token is a term */

{

int next_ch;  /* from the input stream */

int state;    /* of the tokenizer DFA */

int i;        /* for scanning through the term buffer */

/* Part 1: Run a state machine on the input */

state = 0;

i = 0;

while ( 0  < = state )

{

if ( EOF == (next_ch = getc(stream)) ) next_ch = '\0';

term[i++] = convert_case[next_ch];

switch( state )

{

case 0 :

switch( char_class[next_ch] )

{

case WHITE_CH :     i = 0; break;

case LETTER_CH :    state =  1; break;

case LFT_PAREN_CH : state = -2; break;

case RGT_PAREN_CH : state = -3; break;

case AMPERSAND_CH : state = -4; break;

case BAR_CH :       state = -5; break;

case CARET_CH :     state = -6; break;

case EOS_CH :       state = -7; break;

case DIGIT_CH :     state = -8; break;

case OTHER_CH :     state = -8; break;

default :           state =-8; break;

}

break;

case 1 :

if ( (DIGIT_CH != char_class[next_ch])

&& (LETTER_CH != char_class[next_ch]) )

{

ungetc( next_ch, stream );

term[i-1] = '\0';

state = -1;

}

break;

default : state = -8; break;

}

}

/* Part 2: Coerce the final state to return the type token */

return( (TokenType) (-state) );

} /* GetToken */

Figure 7.3: Code for a simple query lexical analyzer

The algorithm begins in state 0. As each input character is consumed, a switch on the state determines the transition. Input is consumed until a final state (indicated by a negative state number) is reached. When recognizing a term, the algorithm keeps reading until some character other than a letter or a digit is found. Since this character may be part of another token, it must be pushed back on the input stream. The final state is translated to a token type value by changing its sign and coercing it to the correct type (this was the point of matching the token type values to the final machine states).

/*FN***********************************************************************

main( argc, argv )

Returns: int -- 0 on success, 1 on failure

Purpose: Program main function

Plan:    Part 1: Open a file named on the command line

Part 2: List all the tokens found in the file

Part 3: Close the file and return

Notes:   This program simply lists the tokens found in a single file

named on the command line.

**/

int

main(argc, argv)

int argc;       /* in: how many arguments */

char *argv[ ]   /* in: text of the arguments */

{

TokenType token;   /* next token in the input stream */

char term[128];    /* the term recognized */

FILE *stream;      /* where to read the data from */

if ( (2 != argc) || !(stream = fopen(argv[l],"r")) ) exit(l);

do

switch( token = GetToken(stream,term) )

{

case TERM_TOKEN :      (void)printf ( "term: %s\n", term ); break;

case LFT_PAREN_TOKEN : (void)printf ( "left parenthesis\n" ); break;

case RGT PAREN-TOKEN : (void)printf ( "right parenthesis\n" ); break;

case AND_TOKEN :       (void)printf ( "and operator\n" ); break;

case OR_TOKEN :        (void)printf ( "or operator\n" ); break;

case NOT_TOKEN :       (void)printf ( "not operator\n" ); break;

case END_TOKEN :       (void)printf ( "end of string\n" ); break;

case NO_TOKEN :        (void)printf ( "no token\n" ); break;

default :              (void)printf ( "bad data\n" ); break;

}

while ( END_TOKEN != token );

fclose ( stream );

} /* main */

Figure 7.4: Test program for a query lexical analyzer

Figure 7.4 contains a small main program to demonstrate the use of this lexical analyzer. The program reads characters from a file named on the command line, and writes out a description of the token stream that it finds. In real use, the tokens returned by the lexical analyzer would be processed by a query parser, which would also probably call retrieval and display routines.

The code above, augmented with the appropriate include files, is a complete and efficient implementation of our simple lexical analyzer for queries. When tested, this code tokenized at about a third the speed that the computer could read characters--about as fast as can be expected. An even simpler lexical analyzer for automatic indexing can be constructed in the same way, and it will be just as fast.

7.3 STOPLISTS

It has been recognized since the earliest days of information retrieval (Luhn 1957) that many of the most frequently occurring words in English (like "the," "of," "and," "to," etc.) are worthless as index terms. A search using one of these terms is likely to retrieve almost every item in a database regardless of its relevance, so their discrimination value is low (Salton and McGill 1983; van Rijsbergen 1975). Furthermore, these words make up a large fraction of the text of most documents: the ten most frequently occurring words in English typically account for 20 to 30 percent of the tokens in a document (Francis and Kucera 1982). Eliminating such words from consideration early in automatic indexing speeds processing, saves huge amounts of space in indexes, and does not damage retrieval effectiveness. A list of words filtered out during automatic indexing because they make poor index terms is called a stoplist or a negative dictionary.

One way to improve information retrieval system performance, then, is to eliminate stopwords during automatic indexing. As with lexical analysis, however, it is not clear which words should be included in a stoplist. Traditionally, stoplists are supposed to have included the most frequently occurring words. However, some frequently occurring words are too important as index terms. For example, included among the 200 most frequently occurring words in general literature in English are "time," "war," "home," "life," "water," and "world." On the other hand, specialized databases will contain many words useless as index terms that are not frequent in general English. For example, a computer literature database probably need not use index terms like "computer," "program," "source," "machine," and "language."

As with lexical analysis in general, stoplist policy will depend on the database and features of the users and the indexing process. Commercial information systems tend to take a very conservative approach, with few stopwords. For example, the ORBIT Search Service has only eight stopwords: "and," "an," "by," "from," "of," "the," and "with." Larger stoplists are usually advisable. An oft-cited example of a stoplist of 250 words appears in van Rijsbergen (1975). Figure 7.5 contains a stoplist of 425 words derived from the Brown corpus (Francis and Kucera 1982) of 1,014,000 words drawn from a broad range of literature in English. Fox (1990) discusses the derivation of (a slightly shorter version of) this list, which is specially constructed to be used with the lexical analysis generator described below.

a          about       above      across      after

again      against     all        almost      alone

along      already     also       although    always

among      an          and        another     any

anybody    anyone      anything   anywhere    are

area       areas       around     as          ask

asked      asking      asks       at          away

b          back        backed     backing     backs

be         because     became     become      becomes

been       before      began      behind      being

beings     best        better     between     big

both       but         by         c           came

can        cannot      case       cases       certain

certainly  clear       clearly    come        could

d          did         differ     different   differently

do         does        done       down        downed

downing    downs       during     e           each

early      either      end        ended       ending

ends       enough      even       evenly      ever

every      everybody   everyone   everything  everywhere

f          face        faces      fact        facts

far        felt        few        find        finds

first      for         four       from        full

fully      further     furthered  furthering  furthers

g          gave        general    generally   get

gets       give        given      gives       go

going      good        goods      got         great

greater    greatest    group      grouped     grouping

groups     h           had        has         have

having     he          her        herself     here

high       higher      highest    him         himself

his        how         however    i           if

important  in          interest   interested  interesting

interests  into        is         it          its

itself     j           just       k           keep

keeps      kind        knew       know        known

knows      l           large      largely     last

later      latest      least      less        let

lets       like        likely     long        longer

longest    m           made       make        making

man        many        may        me          member

members    men         might      more        most

mostly     mr          mrs        much        must

my         myself      n          necessary   need

needed     needing     needs      never       new

newer      newest      next       no          non

not        nobody      noone      nothing     now

nowhere    number      numbered   numbering   numbers

o          of          off        often       old

older      oldest      on         once        one

only       open        opened     opening     opens

or         order       ordered    ordering    orders

other      others      our        out         over

p          part        parted     parting     parts

per        perhaps     place      places      point

pointed    pointing    points     possible    present

presented  presenting  presents   problem     problems

put        puts        q          quite       r

rather     really      right      room        rooms

s          said        same       saw         say

says       second      seconds    see         seem

seemed     seeming     seems      sees        several

shall      she         should     show        showed

showing    shows       side       sides       since

small      smaller     smallest   so          some

somebody   someone     something  somewhere   state

states     still       such       sure        t

take       taken       than       that        the

their      them        then       there       therefore

these      they        thing      things      think

thinks     this        those      though      thought

thoughts   three       through    thus        to

today      together    too        took        toward

turn       turned      turning    turns       two

u          under       until      up          upon

us         use         uses       used        v

very       w           want       wanted      wanting

wants      was         way        ways        we

well       wells       went       were        what

when       where       whether    which       while

who        whole       whose      why         will

with       within      without    work        worked

working    works       would      x           y

year       years       yet        you         young

younger    youngest    your       yours       z

Figure 7.5: A stoplist for general text

7.3.1 Implementing Stoplists

There are two ways to filter stoplist words from an input token stream: (a) examine lexical analyzer output and remove any stopwords, or (b) remove stopwords as part of lexical analysis.

The first approach, filtering stopwords from lexical analyzer output, makes the stoplist problem into a standard list searching problem: every token must be looked up in the stoplist, and removed from further analysis if found. The usual solutions to this problem are adequate, including binary search trees, binary search of an array, and hashing (Tremblay and Sorenson, 1984, Chapter 13). Undoubtedly the fastest solution is hashing.

When hashing is used to search a stoplist, the list must first be inserted into a hash table. Each token is then hashed into the table. If the resulting location is empty, the token is not a stopword, and is passed on; otherwise, comparisons must be made to determine whether the hashed value really matches the entries at that hash table location. If not, then the token is passed on; if so, the token is a word, and is eliminated from the token stream. This strategy is fast, but is slowed by the need to re-examine each character in a token to generate its hash value, and by the need to resolve collisions.

The hashing strategy can be improved by incorporating computation of hash values into the character-by-character processing of lexical analysis. The output of the lexical analysis phase is then a hash value as well as a token, with a small increase in the cost of lexical analysis. Some improvement can also be realized by generating a perfect hashing function for the stoplist (a perfect hashing function for a set of keys hashes the keys with no collisions-see Chapter 13). This minimizes the overhead of collision resolution, but has no effect on the number of collisions, which is sure to be large unless the hash table is enormous.

Although hashing is an excellent approach, probably the best implementation of stoplists is the second strategy: remove stoplist words as part of the lexical analysis process. Since lexical analysis must be done anyway, and recognizing even a large stoplist can be done at almost no extra cost during lexical analysis, this approach is extremely efficient. Furthermore, lexical analyzers that filter stoplists can be generated automatically, which is easier and less error-prone than writing stopword filters by hand.

The rest of this chapter presents a lexical analyzer generator for automatic indexing. The lexical analyzer generator accepts an arbitrary list of stopwords. It should be clear from the code presented here how to elaborate the generator, or the driver program, to fit other needs.

7.3.2 A Lexical Analyzer Generator

The heart of the lexical analyzer generator is its algorithm for producing a finite state machine. The algorithm presented here is based on methods of generating minimum state deterministic finite automata (DFAs) using derivatives of regular expressions (Aho and Ullman 1975) adapted for lists of strings. (A DFA is minimum state if it has a few states as possible.) This algorithm is similar to one described by Aho and Corasick (1975) for string searching.

During machine generation, the algorithm labels each state with the set of strings the machine would accept if that state were the initial state. It is easy to examine these state labels to determine: (a) the transition out of each state, (b) the target state for each transition, and (c) the states that are final states. For example, suppose a state is labeled with the set of strings {a, an, and, in, into, to}. This state must have transitions on a, i, and t. The transition on a must go to a state labeled with the set {n, nd,   [image]}, the transition on i to a state labeled {n, nto}, and the transition on t to a state labeled {o}. A state label L labeling a target state for a transition on symbol a is called a derivative label L with transition a. A state is made a final state if and only if its label contains the empty string.

An algorithm for generating a minimum state DFA using this labeling mechanism is presented in Figure 7.6. An example of a fully constructed machine appears in Figure 7.7.

create an initial state qo and label it with the input set Lo;

place qo in a state queue Q;

while Q is not empty do:

{

remove state qi from Q;

generate the derivative state labels from the label Li for qi;

for each derivative state label Lj with transition a:

{

if no state qj labelled Lj exists, create qj and put it in Q;

create an arc labelled a from qi to qj;

}

}

make all states whose label contains    final states.

Figure 7.6: Algorithm for generating a finite state machine

A C language implementation for this algorithm for generating a finite state machine appears in Figure 7.8. This algorithm relies on simple routines for allocating memory and manipulating lists of strings not listed to save space.

Several techniques are used to speed up the algorithm and to save space. State labels are hashed to produce a signature for faster label searching. Labels are also kept in a binary search tree by hash signature. To save space in the transition table, which is very sparse, transitions are kept in a short list that is searched whenever a transition for a given state is needed. These lists usually have no more than three or four items, so searching them is still reasonably fast. Once the machine is constructed, all the space used tor the state labels is deallocated, so the only data structures of any size that remain are the state table and the compressed transition table. When a transition for a symbol is not found in the transition table, a default  transition to a special dead state is used. When a machine blocks in the dead state, it does not recognize its input.

Figure 7.7: An example of a generated finite state machine

#define DEAD_STATE            -1   /* used to block a DFA */

#define TABLE_INCREMENT       256  /* used to grow tables */

/*************************  Hashing  ****************************/

/* Sets of suffixes labeling states during the DFA construction */

/* are hashed to speed searching. The hashing function uses an  */

/* entire integer variable range as its hash table size; in an  */

/* effort to get a good spread through this range, hash values  */

/* start big, and are incremented by a lot with every new word  */

/* in the list. The collision rate is low using this method.    */

#define HASH_START                  5775863

#define HASH_INCREMENT              38873647

/**************  State Label Binary Search Tree  ****************/

/* During DFA construction, all states must be searched by      */

/* their labels to make sure that the minimum number of states  */

/* are used. This operation is sped up by hashing the labels    */

/* to a signature value, then storing the signatures and labels */

/* in a binary search tree. The tree is destroyed once the DFA  */

/* is fully constructed.                                        */

typedef struct TreeNode {

StrList label;           /* state label used as search key      */

unsigned signature;      /* hashed label to speed searching     */

int state;               /* whose label is representd by node   */

struct TreeNode *left;   /* left binary search subtree          */

struct TreeNode *right;  /* right binary search subtree         */

} SearchTreeNode, *SearchTree;

/*********************  DFA State Table  ************************/

/* The state table is an array of structures holding a state    */

/* label, a count of the arcs out of the state, a pointer into  */

/* the arc table for these arcs, and a final state flag. The    */

/* label field is used only during machine construction.        */

typedef struct {

StrList label;           /* for this state - used during build  */

int num_arcs;            /* for this state in the arc table     */

int arc_offset;          /* for finding arcs in the arc table   */

short is_final;          /* TRUE iff this is a final state      */

} StateTableEntry, *StateTable;

/**********************  DFA Arc Table  *************************/

/* The arc table lists all transitions for all states in a DFA  */

/* in compacted form. Each state's transitions are offset from  */

/* the start of the table, then listed in arc label order.      */

/* Transitions are found by a linear search of the sub-section  */

/* of the table for a given state.                              */

typedef struct {

char label;              /* character label on an out-arrow     */

int target;              /* the target state for the out-arrow  */

} ArcTableEntry, *ArcTable;

**********************  DFA Structure  ***********************   /

/* A DFA is represented as a pointer to a structure holding the */

/* machine's state and transition tables, and bookkeepping      */

/* counters. The tables are arrays whose space is malloc'd,     */

/* then realloc'd if more space is required. Once a machine is  */

/* constructed, the table space is realloc'd one last time to   */

/* fit the needs of the machine exactly.                        */

typedef struct {

int num_states;          /* in the DFA (and state table)        */

int max_states;          /* now allocated in the state table    */

int num_arcs;            /* in the arc table for this machine   */

int max_arcs;            /* now allocated in the arc table      */

StateTable state_table;  /* the compacted DFA state table       */

ArcTable arc_table;      /* the compacted DFA transition table  */

SearchTree tree;         /* storing state labels used in build  */

} DFAStruct, *DFA;

/*FN**************************************************************************

DestroyTree( tree )

Returns: void

Purpose: Destroy a binary search tree created during machine construction

Plan:    Part 1: Return right away of there is no tree

Part 2: Deallocate the subtrees

Part 3: Deallocate the root

Notes:   None.

**/

static void

DestroyTree( tree )

SearchTree tree;   /* in: search tree destroyed */

{

/* Part 1: Return right away of there is no tree */

if ( NULL == tree ) return;

/* Part 2: Deallocate the subtrees */

if ( NULL != tree->left )  DestroyTree ( tree->left );

if ( NULL != tree->right ) DestroyTree ( tree->right );

/* Part 3: Deallocate the root */

tree->left = tree->right = NULL;

(void)free( (char *)tree );

} /* DestroyTree */

/*FN************************************************************************

GetState( machine, label, signature )

Returns: int -- state with the given label

Purpose: Search a machine and return the state with a given state label

Plan:    Part 1: Search the tree for the requested state

Part 2: If not found, add the label to the tree

Part 3: Return the state number

Notes:   This machine always returns a state with the given label

because if the machine does not have a state with the given

label, then one is created.

**/

static int

GetState(machine, label, signature)

DFA machine;        /* in: DFA whose state labels are searched;

StrList label;      /* in: state label searched for */

unsigned signature; /* in: signature of the label requested */

{

SearchTree *ptr;       /* pointer to a search tree link field */

SearchTree new_node;   /* for a newly added search tree node */

/* Part 1: Search the tree for the requested state */

ptr = &(machine->tree);

while ( (NULL != *ptr) && (   (signature != (*ptr)->signature)

|| !StrListEqual(label,(*ptr)->label)) )

ptr = (signature <= (*ptr)->signature) ? &(*ptr)->left : &(*ptr)->right;

/* Part 2: If not found, add the label to the tree */

if ( NULL == *ptr )

{

/* create a new node and fill in its fields */

new_node = (SearchTree)GetMemory( NULL, sizeof(SearchTreeNode) );

new-node->signature = signature;

new_node->label = (StrList)label;

new_node->state = machine->num_states;

new_node->left = new_node->right = NULL;

/* allocate more states if needed, set up the new state */

if ( machine->num_states == machine->max_states )

{

machine->max_states += TABLE_INCREMENT;

machine->state_table =

(StateTable)GetMemory(machine-state_table,

machine-max_states*sizeof(StateTableEntry));

}

machine->state_table[machine-num_states].label = (StrList)label;

machine->num_states++;

/* hook the new node into the binary search tree */

*ptr = new_node;

}

else

StrListDestroy( label );

/* Part 3: Return the state number */

return( (*ptr)->state );

} /* GetState */

/*FN*********************************************************************************

AddArc( machine, state, arc_label, state_label, state_signature )

Returns: void

Purpose: Add an arc between two states in a DFA

Plan:  Part 1: Search for the target state among existing states

Part 2: Make sure the arc table is big enough

Part 3: Add the new arc

Notes: None.

**/

static void

AddArc( machine, state, arc_label, state_label, state_signature )

DFA machine;               /* in/out: machine with an arc added */

int state;                 /* in: with an out arc added */

char arc_label;            /* in: label on the new arc */

StrList state_label;       /* in: label on the target state */

unsigned state_signature;  /* in: label hash signature to speed searching */

{

register int target;       /* destination state for the new arc */

/* Part 1: Search for the target state among existing states */

StrListSort( 0, state_label );

target = GetState( machine, state_label, state_signature );

/* Part 2: Make sure the arc table is big enough */

if ( machine->num_arcs == machine->max_arcs )

{

machine->max_arcs += TABLE_INCREMENT;

machine->arc_table =

(ArcTable)GetMemory( machine->arc_table,

machine->max_arcs * sizeof(ArcTableEntry) );

}

/* Part 3: Add the new arc */

machine->arc_table[machine->num_arcs].label = arc_label;

machine->arc_tablel[machine->num_arcs].target = target;

machine->num_arcs++;

machine->state_table[state].num_arcs++;

} /* AddArc */

/*FN**********************************************************************************

BuildDFA( words )

Returns: DFA -- newly created finite state machine

Purpose: Build a DFA to recognize a list of words

Plan:    Part 1: Allocate space and initialize variables

Part 2: Make and label the DFA start state

Part 3: Main loop - build the state and arc tables

Part 4: Deallocate the binary search tree and the state labels

Part 5: Reallocate the tables to squish them down

Part 6: Return the newly constructed DFA

Notes: None.

**/

DFA

BuildDFA( words )

StrList words; /* in: that the machine is built to recognize */

{

DFA machine;                /* local for easier access to machine */

register int state;         /* current state's state number */

char arc_label;             /* for the current arc when adding arcs */

char *string;               /* element in a set of state labels */

char ch;                    /* the first character in a new string */

StrList current_label;      /* set of strings labeling a state */

StrList target_label;       /* labeling the arc target state */

unsigned target_signature;  /* hashed label for binary search tree */

register int i;             /* for looping through strings */

/* Part 1: Allocate space and initialize variables */

machine = (DFA)GetMemory( NULL, sizeof(DFAStruct) );

machine->max_states = TABLE_INCREMENT;

machine->state_table =

(StateTable)GetMemory(NULL, machine->max_states*sizeof(StateTableEntry));

machine->num_states = 0;

machine->max_arcs = TABLE_INCREMENT;

machine->arc_table =

(ArcTable)GetMemory ( NULL, machine->max_arcs * sizeof(ArcTableEntry) );

machine->num_arcs = 0;

machine->tree = NULL;

/* Part 2: Make and label the DFA start state */

StrListUnique( 0, words );  /* sort and unique the list */

machine->state_table[0].label = words;

machine->num_states = 1;

/* Part 3: Main loop - build the state and arc tables */

for ( state = 0; state < machine->num_states; state++ )

{

/* The current state has nothing but a label, so  */

/* the first order of business is to set up some  */

/* of its other major fields                      */

machine->state_table[state].is_final = FALSE;

machine->state_table[state].arc_offset = machine->num_arcs;

machine->state_table[state].num_arcs = 0;

/* Add arcs to the arc table for the current state  */

/* based on the state's derived set.  Also set the  */

/* state's final flag if the empty string is found  */

/* in the suffix list                               */

current_label = machine->state_table[state].label;

target_label = StrListCreate();

target_signature = HASH_START;

arc_label = EOS;

for ( i = 0; i < StrListSize(current_label); i++ )

{

/* get the next string in the label and lop it */

string = StrListPeek( current_label, i );

ch = *string++;

/* the empty string means mark this state as final */

if ( EOS == ch )

{ machine->state_table[state].is_final = TRUE; continue; }

/* make sure we have a legitimate arc_label */

if ( EOS == arc_label ) arc_label = ch;

/* if the first character is new, then we must */

/* add an arc for the previous first character */

if ( ch != arc_label )

{

AddArc (machine, state, arc_label, target_label, target_signature);

target_label = StrListCreate();

target_signature = HASH_START;

arc_label = ch;

}

/* add the current suffix to the target state label */

StrListAppend( target_label, string );

target_signature += (*string + l) * HASH_INCREMENT;

while ( *string ) target_signature += *string++;

}

/* On loop exit we have not added an arc for the  */

/* last bunch of suffixes, so we must do so, as   */

/* long as the last set of suffixes is not empty  */

/* (which happens when the current state label    */

/* is the singleton set of the empty string).     */

if ( 0 < StrListSize(target_label) )

AddArc( machine, state, arc_label, target_label, target_signature );

}

/* Part 4: Deallocate the binary search tree and the state labels */

DestroyTree( machine->tree ); machine->tree = NULL;

for ( i = 0; i < machine->num_states; i++ )

{

StrListDestroy( machine->state_table[i].label );

machine->state_table[i]. label = NULL;

}

/* Part 5: Reallocate the tables to squish them down */

machine->state_table = (StateTable)GetMemory( machine->state_table,

machine->num_states * sizeof(StateTableEntry) );

machine->arc_table = (ArcTable)GetMemory( machine->arc_table,

machine->num_arcs * sizeof(ArcTableEntry) );

/* Part 6: Return the newly constructed DFA */

return( machine );

} /* BuildDFA */

Figure 7.8: Code for DFA generation

After a finite state machine is constructed, it is easy to generate a simple driver program that uses it to process input. Figure 7.9 contains an example of such a driver program. This program assumes the existence of a character class array like the one in Figure 7.2, except that it has only three character classes: one for digits, one for letters, and one for everything else (called delimiters). It also assumes there is a character case conversion array like the one in Figure 7.2. The driver takes a machine constructed using the code in Figure 7.7, and uses it to filter all stopwords, returning only legitimate index terms. It filters numbers, but accepts terms containing digits. It also converts the case of index terms.

/*FN*************************************************************************

GetTerm( stream, machine, size, output )

Returns:  char * - - NULL if stream is exhausted, otherwise output buffer

Purpose:  Get the next token from an input stream, filtering stop words

Plan:     Part 1: Return NULL immediately if there is no input

Part 2: Initialize the local variables

Part 3: Main Loop: Put an unfiltered word into the output buffer

Part 4: Return the output buffer

Notes:   This routine runs the DFA provided as the machine parameter,

and collects the text of any term in the output buffer. If

a stop word is recognized in this process, it is skipped.

Care is also taken to be sure not to overrun the output buffer.

**/

char *

GetTerm( stream, machine, size, output )

FILE *stream;  /* in: source of input characters */

DFA machine;   /* in: finite state machine driving process */

int size;      /* in: bytes in the output buffer */

char *output;  /* in/out: where the next token in placed */

{

char *outptr;        /* for scanning through the output buffer */

int ch;              /* current character during input scan */

register int state;  /* current state during DFA execution */

/* Part 1: Return NULL immediately if there is no input */

if ( EOF = = (ch = getc(stream)) ) return( NULL );

/* Part 2: Initialize the local variables */

outptr = output;

/* Part 3: Main Loop: Put an unfiltered word into the output buffer */

do

{

/* scan past any leading delimiters */

while ( (EOF != ch ) &&

((DELIM_CH == char_class[ch]) | |

(DIGIT_CH == char_class[ch])) ) ch = getc( stream );

/* start the machine in its start state */

state = 0;

/* copy input to output until reaching a delimiter, and also */

/* run the DFA on the input to watch for filtered words */

while ( (EOF != ch) && (DELIM_CH != char_class[ch]) )

{

if ( outptr == (output+size-1) ) { outptr = output; state = 0; }

*outptr++ = convert_case[ch];

if ( DEAD_STATE != state )

{

register int i;  /* for scanning through arc labels */

int arc_start;   /* where the arc label list starts */

int arc_end;     /* where the arc label list ends */

arc_start = machine->state_table[state].arc_offset;

arc_end = arc_start + machine->state_table[state].num_arcs;

for ( i = arc_start; i < arc_end; i++ )

if ( convert_case[ch] == machine->arc_table[i].label )

{ state = machine->arc_table[i].target; break;  }

if ( i == arc_end ) state = DEAD_STATE;

}

ch = getc( stream );

}

/* start from scratch if a stop word is recognized */

if ( (DEAD_STATE != state) && machine->state_table[state].is_final )

outptr = output;

/* terminate the output buffer */

*outptr = EOS;

}

while ( (EOF != ch) && !*output );

/* Part 4: Return the output buffer */

return( output );

} /* GetTerm */

Figure 7.9: An example of DFA driver program

Once the finite state machine blocks in the dead state, the string is not recognized. The driver program takes advantage of this fact by not running the finite state machine once it enters the dead state.

A lexical analyzer generator program can use these components in several ways. A lexical analyzer can be generated at indexing time, or ahead of time and stored in a file. The input can be read from a stream, as in the example driver  program, or from another input source. A lexical analyzer data structure can be defined, and different stoplists and lexical analysis rules used in each one, then different analyzers can be run on different sorts of data. All these alternatives are easy to implement once the basic finite state machine generator and driver generator programs are in place.

As an illustration, Figure 7.10 contains the main function for a program that reads a stoplist from a file, builds a finite state machine using the function from Figure 7.8, then uses the driver function from Figure 7.9 to generate and print all the terms in an input file, filtering out the words in the stoplist.

/*FN**************************************************************************

main( argc, argv )

Returns:  int -- 0 on success, 1 on failure

Purpose:  Program main function

Plan:     Part 1: Read the stop list from the stop words file

Part 2: Create a DFA from a stop list

Part 3: Open the input file and list its terms

Part 4: Close the input file and return

Notes:   This program reads a stop list from a file called "words.std,"

and uses it in generating the terms in a file named on the

command line.

**/

int

main( argc, argv )

int argc;      /* in: how many arguments */

char *argv[];  /* in: text of the arguments */

{

char term[128];  /* for the next term found */

FILE *stream;    /* where to read characters from */

StrList words;   /* the stop list filtered */

DFA machine;     /* build from the stop list */

/* Part 1: Read the stop list from the stop words file */

words = StrListCreate( );

StrListAppendFile( words, "words.std" );

/* Part 2: Create a DFA from a stop list */

machine = BuildDFA( words );

/* Part 3: Open the input file and list its terms */

if ( !(stream = fopen(argv[1], "r")) ) exit(1);

while ( NULL != GetTerm(stream, machine, 128, term) )

(void)printf( "%s/n", term );

/* Part 4: Close the input file and return */

(void)fclose( stream );

return(0);

} /* main */

Figure 7.10: Main function for a term generator program

Lexical analyzers built using the method outlined here can be constructed quickly, and are small and fast. For example, the finite state machine generated by the code in Figure 7.8 for the stoplist of 425 words in Figure 7.5 has only 318 states and 555 arcs. The program in Figure 7.10 built this finite state machine from scratch, then used it to lexically analyze the text from this chapter in under 1 second on a Sun SparcStation 1.

7.4 SUMMARY

Lexical analysis must be done in automatic indexing and in query processing. Important decisions about the lexical form of indexing and query terms, and of query operators and grouping indicators, must be made in the design phase based on characteristics of databases and uses of the target system. Once this is done, it is a simple matter to apply techniques from program translation to the problem of lexical analysis. Finite state machine based lexical analyzers can be built quickly and reliably by hand or with a lexical analyzer generator.

Problems in the selection of stoplist words for automatic indexing are similar to those encountered in designing lexical analyzers, and likewise depend on the characteristics of the database and the system. Removing stoplist words during automatic indexing can be treated like a search problem, or can be incorporated into the lexical analysis process. Although hashing can be used to solve the searching problem very efficiently, it is probably best to incorporate stoplist processing into lexical analysis. Since stoplists may be large, automatic generation of lexical analyzers is the preferred approach in this case.

REFERENCES

AHO, A., and M. CORASICK. 1975. "Efficient String Matching: An Aid to Bibliographic Search." Communications of the ACM, 18(6), 333-40.

AHO, A., R. SETHI, and J. ULLMAN. 1986. Compilers: Principles, Techniques, and Tools. New York: Addison-Wesley.

FOX, C. 1990. "A Stop List for General Text." SIGIR Forum, 24(1-2), 19-35.

FRANCIS, W., and H. KUCERA. 1982. Frequency Analysis of English Usage. New York: Houghton Mifflin.

HOPCROFT, J., and J. ULLMAN. 1979. Introduction to Automata Theory, Languages, and Computation. New York: Addison-Wesley.

LESK, M. 1975. Lex-A Lexical Analyzer Generator. Murray Hill, N.J.: AT&T Bell Laboratories.

LUHN, H. P. 1957. "A Statistical Approach to Mechanized Encoding and Searching of Literary Information. IBM Journal of Research and Development, 1(4).

SALTON, G., and M. MCGILL. 1983. Modern Information Retrieval. New York: McGraw-Hill.

TREMBLAY, J. P., and P. SORENSON. 1984. An Introduction to Data Structures with  Applications, Second Edition. New York: McGraw-Hill.

VAN RIJSBERGEN, C. J. 1975. Information Retrieval. London: Butterworths.

WAIT, W. 1986. "The Cost of Lexical Analysis." Software Practice and Experience, 16(5), 473-88.

Go to Chapter 8     Back to Table of Contents


Information Retrieval: CHAPTER 8: STEMMING ALGORITHMS

 Return to Table of Contents

CHAPTER 8: STEMMING ALGORITHMS

W. B. Frakes

Software Engineering Guild, Sterling, VA 22170

Abstract

This chapter describes stemming algorithms--programs that relate morphologically similar indexing and search terms. Stemming is used to improve retrieval effectiveness and to reduce the size of indexing files. Several approaches to stemming are described--table lookup, affix removal, successor variety, and n-gram. Empirical studies of stemming are summarized. The Porter stemmer is described in detail, and a full implementation in C is presented.

8.1 INTRODUCTION

One technique for improving IR performance is to provide searchers with ways of finding morphological variants of search terms. If, for example, a searcher enters the term stemming as part of a query, it is likely that he or she will also be interested in such variants as stemmed and stem. We use the term conflation, meaning the act of fusing or combining, as the general term for the process of matching morphological term variants. Conflation can be either manual--using some kind of regular expressions--or automatic, via programs called stemmers. Stemming is also used in IR to reduce the size of index files. Since a single stem typically corresponds to several full terms, by storing stems instead of terms, compression factors of over 50 percent can be achieved.

As can be seen in Figure 1.2 in Chapter 1, terms can be stemmed at indexing time or at search time. The advantage of stemming at indexing time is efficiency and index file compression--since index terms are already stemmed, this operation requires no resources at search time, and the index file will be compressed as described above. The disadvantage of indexing time stemming is that information about the full terms will be lost, or additional storage will be required to store both the stemmed and unstemmed forms.

Figure 8.1 shows a taxonomy for stemming algorithms. There are four automatic approaches. Affix removal algorithms remove suffixes and/or prefixes from terms leaving a stem. These algorithms sometimes also transform the resultant stem. The name stemmer derives from this method, which is the most common. Successor variety stemmers use the frequencies of letter sequences in a body of text as the basis of stemming. The n-gram method conflates terms based on the number of digrams or n-grams they share. Terms and their corresponding stems can also be stored in a table. Stemming is then done via lookups in the table. These methods are described below.

Figure 8.1: Conflation methods

There are several criteria for judging stemmers: correctness, retrieval effectiveness, and compression performance. There are two ways stemming can be incorrect--overstemming and understemming. When a term is overstemmed, too much of it is removed. Overstemming can cause unrelated terms to be conflated. The effect on IR performance is retrieval of nonrelevant documents. Understemming is the removal of too little of a term. Understemming will prevent related terms from being conflated. The effect of understemming on IR performance is that relevant documents will not be retrieved. Stemmers can also be judged on their retrieval effectiveness--usually measured with recall and precision as defined in Chapter 1, and on their speed, size, and so on. Finally, they can be rated on their compression performance. Stemmers for IR are not usually judged on the basis of linguistic correctness, though the stems they produce are usually very similar to root morphemes, as described below.

8.1.1 Example of Stemmer Use in Searching

To illustrate how a stemmer is used in searching, consider the following example from the CATALOG system (Frakes 1984, 1986). In CATALOG, terms are stemmed at search time rather than at indexing time. CATALOG prompts for queries with the string "Look for:". At the prompt, the user types in one or more terms of interest. For example:

Look for: system users

will cause CATALOG to attempt to find documents about system users. CATALOG takes each term in the query, and tries to determine which other terms in the database might have the same stem. If any possibly related terms are found, CATALOG presents them to the user for selection. In the case of the query term "users," for example, CATALOG might respond as follows:

   Search Term:  users

   Term             Occurrences

1. user              15

2. users              1

3. used               3

4. using              2

Which terms (0 = none, CR = all):

The user selects the terms he or she wants by entering their numbers.

This method of using a stemmer in a search session provides a naive system user with the advantages of term conflation while requiring little knowledge of the system or of searching techniques. It also allows experienced searchers to focus their attention on other search problems. Since stemming may not always be appropriate, the stemmer can be turned off by the user. Having a user select the terms from the set found by the stemmer also reduces the likelihood of false matches.

8.2 TYPES OF STEMMING ALGORITHMS

There are several approaches to stemming. One way to do stemming is to store a table of all index terms and their stems. For example:

   Term       Stem

---------------------

engineering  engineer

engineered   engineer

engineer     engineer

Terms from queries and indexes could then be stemmed via table lookup. Using a B-tree or hash table, such lookups would be very fast.

There are problems with this approach. The first is that there is no such data for English. Even if there were, many terms found in databases would not be represented, since they are domain dependent that is, not standard English. For these terms, some other stemming method would be required. Another problem is the storage overhead for such a table, though trading size for time is sometimes warranted. Storing precomputed data, as opposed to computing the data values on the fly, is useful when the computations are frequent and/or expensive. Bentley (1982), for example, reports cases such as chess computations where storing precomputed results gives significant performance improvements.

8.2.1 Successor Variety

Successor variety stemmers (Hafer and Weiss 1974) are based on work in structural linguistics which attempted to determine word and morpheme boundaries based on the distribution of phonemes in a large body of utterances. The stemming method based on this work uses letters in place of phonemes, and a body of text in place of phonemically transcribed utterances.

Hafer and Weiss formally defined the technique as follows:

Let   [image]  be a word of length n;   [image] i, is a length i prefix of   [image] . Let D be the corpus of words. D  [image] i is defined as the subset of D containing those terms whose first i letters match   [image] i exactly. The successor variety of   [image] i, denoted S  [image] i, is then defined as the number of distinct letters that occupy the i + 1st position of words in D  [image] i. A test word of length n has n successor varieties S  [image] i, S  [image] 2, . . . , S  [image] n.

In less formal terms, the successor variety of a string is the number of different characters that follow it in words in some body of text. Consider a body of text consisting of the following words, for example.

able, axle, accident, ape, about.

To determine the successor varieties for "apple," for example, the following process would be used. The first letter of apple is "a." "a" is followed in the text body by four characters: "b," "x," "c," and "p." Thus, the successor variety of  "a" is four. The next successor variety for apple would be one, since only "e" follows "ap" in the text body, and so on.

When this process is carried out using a large body of text (Hafer and Weiss report 2,000 terms to be a stable number), the successor variety of substrings of a term will decrease as more characters are added until a segment boundary is reached. At this point, the successor variety will sharply increase. This information is used to identify stems.

Once the successor varieties for a given word have been derived, this information must be used to segment the word. Hafer and Weiss discuss four ways of doing this.

1.     Using the cutoff method, some cutoff value is selected for successor varieties and a boundary is identified whenever the cutoff value is reached. The problem with this method is how to select the cutoff value--if it is too small, incorrect cuts will be made; if too large, correct cuts will be missed.

2.     With the peak and plateau method, a segment break is made after a character whose successor variety exceeds that of the character immediately preceding it and the character immediately following it. This method removes the need for the cutoff value to be selected.

3.     In the complete word method method, a break is made after a segment if the segment is a complete word in the corpus.

4.     The entropy method takes advantage of the distribution of successor variety letters. The method works as follows. Let |D  [image] i| be the number of words in a text body beginning with the i length sequence of letters   [image]. Let |D  [image] ij| be the number of words in D  [image] i with the successor j. The probability that a member of D  [image] i has the successor j is given by   [image]. The entropy of |D  [image] i| is

Using this equation, a set of entropy measures can be determined for a word. A set of entropy measures for predecessors can also be defined similarly. A cutoff value is selected, and a boundary is identified whenever the cutoff value is reached.

Hafer and Weiss experimentally evaluated the various segmentation methods using two criteria: (1) the number of correct segment cuts divided by the total number of cuts, and (2) the number of correct segment cuts divided by the total number of true boundaries. They found that none of the methods performed perfectly, but that techniques that combined certain of the methods did best.

To illustrate the use of successor variety stemming, consider the example below where the task is to determine the stem of the word READABLE.

        Test Word: READABLE

        Corpus:    ABLE, APE, BEATABLE, FIXABLE, READ, READABLE

                   READING, READS, RED, ROPE, RIPE.

  Prefix    Successor Variety     Letters

-----------------------------------------

R                  3              E,I,O

RE                 2              A,D

REA                1              D

READ               3              A,I,S

READA              1              B

READAB             1              L

READABL            1              E

READABLE           1              BLANK

Using the complete word segmentation method, the test word "READABLE" will be segmented into "READ" and "ABLE," since READ appears as a word in the corpus. The peak and plateau method would give the same result.

After a word has been segmented, the segment to be used as the stem must be selected. Hafer and Weiss used the following rule:

if (first segment occurs in <= 12 words in corpus)

first segment is stem

else (second segment is stem)

The check on the number of occurrences is based on the observation that if a segment occurs in more than 12 words in the corpus, it is probably a prefix. The authors report that because of the infrequency of multiple prefixes in English, no segment beyond the second is ever selected as the stem. Using this rule in the example above, READ would be selected as the stem of READABLE.

In summary, the successor variety stemming process has three parts: (1) determine the successor varieties for a word, (2) use this information to segment the word using one of the methods above, and (3) select one of the segments as the stem. The aim of Hafer and Weiss was to develop a stemmer that required little or no human processing. They point out that while affix removal stemmers work well, they require human preparation of suffix lists and removal rules. Their stemmer requires no such preparation. The retrieval effectiveness of the Hafer and Weiss stemmer is discussed below.

8.2.2 n-gram stemmers

Adamson and Boreham (1974) reported a method of conflating terms called the shared digram method. A digram is a pair of consecutive letters. Since trigrams, or n-grams could be used, we have called it the n-gram method. Though we call this a "stemming method," this is a bit confusing since no stem is produced.

In this approach, association measures are calculated between pairs of terms based on shared unique digrams. For example, the terms statistics and statistical can be broken into digrams as follows.

           statistics => st ta at ti is st ti ic cs

           unique digrams = at cs ic is st ta ti

           statistical => st ta at ti is st ti ic ca al

           unique digrams = al at ca ic is st ta ti

Thus, "statistics" has nine digrams, seven of which are unique, and "statistical" has ten digrams, eight of which are unique. The two words share six unique digrams: at, ic, is, st, ta, ti.

Once the unique digrams for the word pair have been identified and counted, a similarity measure based on them is computed. The similarity measure used was Dice's coefficient, which is defined as

where A is the number of unique digrams in the first word, B the number of unique digrams in the second, and C the number of unique digrams shared by A and B. For the example above, Dice's coefficient would equal (2 x 6)/(7 + 8) = .80. Such similarity measures are determined for all pairs of terms in the database, forming a similarity matrix. Since Dice's coefficient is symmetric (Sij = Sji), a lower triangular similarity matrix can be used as in the example below.

       word1  word2  word3. . .wordn-1

word1

word2  S21

word3  S31     S32

wordn  Sn1     Sn2    Sn3       Sn(n-1)

Once such a similarity matrix is available, terms are clustered using a single link clustering method as described in Chapter 16. The algorithm for calculating a digram similarity matrix follows.

/* calculate similarity matrix for words based on digrams */

#define MAXGRAMS 50   /* maximum n-grams for a word */

#define GRAMSIZE 2    /* size of the n-gram */

void

digram_smatrix (wordlist, word_list_length, smatrix)

char *wordlist[];  /* list of sorted unique words */

int word_list_length;      /* length of wordlist */

double *smatrix[];

{

int i, j;    /* loop counters */

int uniq_in_wordl; /* number of unique digrams in word 1 */

int uniq_in_word2; /* number of unique digrams in word 2 */

int common_uniq;   /* number of unique digrams shared by words 1 and 2 */

char uniq_digrams_1 [MAXGRAMS] [GRAMSIZE]; /* array of digrams */

char uniq_digrams_2 [MAXGRAMS] [GRAMSIZE]; /* array of digrams */

int unique_digrams();  /* function to calculate # of unique digrams in word */

int common_digrams();  /* function to calculate # of shared unique digrams */

for ( i=0; i< word_list_length; ++i)

for (j=i+1; j <word_list_length ;++j) {

/* find unique digrams for first word in pair */

uniq_in_word1 = unique_digrams(wordlist [i], uniq_digrams 1);

/* find unique digrams for second word in pair */

uniq_in_word2 = unique_digrams(wordlist [j], uniq_digrams_2);

/* find number of common unique digrams */

common_uniq = common_digrams(uniq_digrams_1, uniq_digrams_2);

/* calculate similarity value and store in similarity matrix */

smatrix[i][j] = (2*common_uniq)/(uniq_in_word1+uniq_in_word2);

} /* end for */

} /* end digram_smatrix */

When they used their method on a sample of words from Chemical Titles, Adamson and Boreham found that most pairwise similarity measures were 0. Thus, the similarity matrix will be sparse, and techniques for handling sparse matrices will be appropriate. Using a cutoff similarity value of .6, they found ten of the eleven clusters formed were correct. More significantly, in hardly any cases did the method form false associations. The authors also report using this method successfully to cluster document titles.

8.2.3 Affix Removal Stemmers

Affix removal algorithms remove suffixes and/or prefixes from terms leaving a stem. These algorithms sometimes also transform the resultant stem. A simple example of an affix removal stemmer is one that removes the plurals from terms. A set of rules for such a stemmer is as follows (Harman 1991).

If a word ends in "ies" but not "eies" or "aies"

Then "ies" -> "y"

If a word ends in "es" but not "aes", "ees", or "oes"

then "es" -> "e"

If a word ends in "s", but not "us" or "ss"

then "s" -> NULL

In this algorithm only the first applicable rule is used.

Most stemmers currently in use are iterative longest match stemmers, a kind of affix removal stemmer first developed by Lovins (1968). An iterative longest match stemmer removes the longest possible string of characters from a word according to a set of rules. This process is repeated until no more characters can be removed. Even after all characters have been removed, stems may not be correctly conflated. The word "skies," for example, may have been reduced to the stem "ski" which will not match "sky." There are two techniques to handle this--recoding or partial matching.

Recoding is a context sensitive transformation of the form AxC -> AyC where A and C specify the context of the transformation, x is the input string, and y is the transformed string. We might, for example, specify that if a stem ends in an "i" following a "k," then i -> y. In partial matching, only the n initial characters of stems are used in comparing them. Using this approach, we might say that two stems are equivalent if they agree in all but their last characters.

In addition to Lovins, iterative longest match stemmers have been reported by Salton (1968), Dawson (1974), Porter (1980), and Paice (1990). As discussed below, the Porter algorithm is more compact than Lovins, Salton, and Dawson, and seems, on the basis of experimentation, to give retrieval performance comparable to the larger algorithms. The Paice stemmer is also compact, but since experimental data was not available for the Paice algorithm, we chose the Porter algorithm as the example of this type of stemmer.

The Porter algorithm consists of a set of condition/action rules. The conditions fall into three classes: conditions on the stem, conditions on the suffix, and conditions on the rules.

There are several types of stem conditions.

1.     The measure, denoted m, of a stem is based on its alternate vowel-consonant sequences. Vowels are a, e, i, o, u, and y if preceded by a consonant. Consonants are all letters that are not vowels. Let C stand for a sequence of consonants, and V for a sequence of vowels. The measure m, then, is defined as

[C](VC)m[V]

The superscript m in the equation, which is the measure, indicates the number of VC sequences. Square brackets indicate an optional occurrence. Some examples of measures for terms follows.

          Measure             Examples

          ---------------------------------------

           m=0      TR, EE, TREE, Y, BY

           m=1      TROUBLE, OATS, TREES, IVY

           m=2      TROUBLES, PRIVATE, OATEN

2.     * < X >--the stem ends with a given letter X

3.     *v*--the stem contains a vowel

4.     *d--the stem ends in a double consonant

5.     *o--the stem ends with a consonant-vowel-consonant, sequence, where the final consonant is not w, x, or y.

Suffix conditions take the form: (current_suffix == pattern). Rule conditions take the form: (rule was used).

Actions are rewrite rules of the form:

old_suffix -> new_suffix

The rules are divided into steps. The rules in a step are examined in sequence, and only one rule from a step can apply. The longest possible suffix is always removed because of the ordering of the rules within a step. The algorithm is as follows.

{

step1a(word);

step1b(stem);

if (the second or third rule of step 1b was used)

step1b1(stem);

step1c(stem);

step2(stem);

step3(stem);

step4(stem);

step5a(stem);

step5b(stem);

}

The rules for the steps of the stemmer are as follows.

Step 1a Rules

Conditions  Suffix  Replacement      Examples

---------------------------------------------------

 NULL       sses      ss         caresses -> caress

 NULL       ies       i          ponies -> poni

                                 ties -> tie

 NULL       ss        ss         carress -> carress

 NULL       s         NULL       cats -> cat

Step 1b Rules

Conditions  Suffix  Replacement      Examples

----------------------------------------------------

  (m>0)      eed       ee        feed-> feed

                                 agreed -> agree

  (*v*)      ed        NULL      plastered-> plaster

                                 bled -> bled

  (*v*)      ing       NULL      motoring-> motor

                                 sing-> sing

Step 1b1 Rules

        Conditions        Suffix  Replacement        Examples

------------------------------------------------------------------------

NULL                       at     ate            conflat(ed) -> conflate

NULL                       bl     ble            troubl(ing) -> trouble

NULL                       iz     ize            siz(ed) -> size

(*d and not

 (*<L> or *<S> or *<Z>))   NULL   single letter  hopp(ing) -> hop

                                                 tann(ed) -> tan

                                                 fall(ing) -> fall

                                                 hiss(ing) -> hiss

                                                 fizz(ed) -> fizz

(m=1 and *o)               NULL   e              fail(ing) -> fail

                                                 fil(ing) -> file

Step 1c Rules

Conditions    Suffix    Replacement    Examples

-----------------------------------------------------------

  (*v*)         y             i      happy - > happi

                                     sky -> sky

Step 2 Rules

Conditions  Suffix   Replacement            Examples

--------------------------------------------------------------

  (m>0)     ational      ate      relational -> relate

  (m>0)     tional       tion     conditional -> condition

                                  rational -> rational

  (m>0)     enci         ence     valenci -> valence

  (m>0)     anci         ance     hesitanci -> hesitance

  (m>0)     izer         ize      digitizer -> digitize

  (m>0)     abli         able     conformabli -> conformable

  (m>0)     alli         al       radicalli -> radical

  (m>0)     entli        ent      differentli -> different

  (m>0)     eli          e        vileli -> vile

  (m>0)     ousli        ous      analogousli -> analogous

  (m>0)     ization      ize      vietnamization -> vietnamize

  (m>0)     ation        ate      predication -> predicate

  (m>0)     ator         ate      operator -> operate

  (m>0)     alism        al       feudalism -> feudal

  (m>0)     iveness      ive      decisiveness -> decisive

  (m>0)     fulness      ful      hopefulness -> hopeful

  (m>0)     ousness      ous      callousness -> callous

  (m>0)     aliti        al       formaliti -> formal

  (m>0)     iviti        ive      sensitiviti -> sensitive

  (m>0)     biliti       ble      sensibiliti -> sensible

Step 3 Rules

Conditions  Suffix  Replacement        Examples

--------------------------------------------------------

  (m>0)     icate     ic         triplicate -> triplic

  (m>0)     ative     NULL       formative -> form

  (m>0)     alize     al         formalize -> formal

  (m>0)     iciti     ic         electriciti -> electric

  (m>0)     ical      ic         electrical -> electric

  (m>0)     ful       NULL       hopeful -> hope

  (m>0)     ness      NULL       goodness -> good

Step 4 Rules

       Conditions         Suffix   Replacement         Examples

---------------------------------------------------------------------

(m>1)                     al         NULL      revival -> reviv

(m>1)                     ance       NULL      allowance -> allow

(m>1)                     ence       NULL      inference -> infer

(m>1)                     er         NULL      airliner-> airlin

(m>1)                     ic         NULL      gyroscopic -> gyroscop

(m>1)                     able       NULL      adjustable -> adjust

(m>1)                     ible       NULL      defensible -> defens

(m>1)                     ant        NULL      irritant -> irrit

(m>1)                     ement      NULL      replacement -> replac

(m>1)                     ment       NULL      adjustment -> adjust

(m>1)                     ent        NULL      dependent -> depend

(m>1 and (*<S> or *<T>))  ion        NULL      adoption -> adopt

(m>1)                     ou         NULL      homologou-> homolog

(m>1)                     ism        NULL      communism-> commun

(m>1)                     ate        NULL      activate -> activ

(m>1)                     iti        NULL      angulariti -> angular

(m>1)                     ous        NULL      homologous -> homolog

(m>1)                     ive        NULL      effective -> effect

(m>1)                     ize        NULL      bowdlerize -> bowdler

Step 5a Rules

  Conditions        Suffix  Replacement     Examples

---------------------------------------------------------

(m>1)               e         NULL      probate -> probat

                                        rate - > rate

(m=1 and not *o)    e         NULL      cease - > ceas

Step 5b Rules

      Conditions        Suffix  Replacement         Examples

-------------------------------------------------------------------

(m> 1 and *d and *<L>)  NULL     single letter  controll -> control

                                                roll -> roll

A full implementation of this stemmer, in C, is in the appendix to this chapter.

8.3 EXPERIMENTAL EVALUATIONS OF STEMMING

There have been many experimental evaluations of stemmers. Salton (1968) examined the relative retrieval performance of fully stemmed terms against terms with only the suffix "s" removed. The stemmer used was an iterative longest match stemmer of the type described above, employing about 200 endings. Three document collections were used in these studies: the IRE-3 collection consisting of 780 computer science abstracts and 34 search requests, the ADI collection consisting of 82 papers and 35 search requests, and the Cranfield-1 collection consisting of 200 aerodynamics abstracts and 42 search requests.

Differences between the two conflation methods, fully stemmed and "s" stemmed, were calculated on 14 dependent variables for each query: rank recall, log precision, normalized recall, normalized precision, and precision for ten recall levels. As Salton points out, these measures are probably intercorrelated, and since the inferential tests used require independence of the dependent variables, the results reported must be viewed with caution. This data was analyzed using both related group t tests and sign tests. Both of these statistical methods yielded significant results at the .05 probability level, but since none of the t values are reported, precluding their use in an estimate of effect size, and since sufficient data is provided for the estimate of effect size from the sign tests, the latter will be discussed here. (The effect size is the percentage of the variance of the independent variable accounted for by the dependent variables.) The effect size for a sign test is the difference between the obtained proportion (that is, the number of cases favoring a given method over the number of all cases where a difference was detected), and the expected proportion if there is no difference between means (that is, .50*).

*The effect size for a sign test can only take on values in the range 0 to .5.

For the IRE-3 collection, 476 differences were used for the sign test (the 14 dependent variables measures   [image]  34 queries). Of these, 272 pairs favored full stemming, 132 pairs favored suffix "s" stemming, and in 72 cases neither method was favored. Thus, the effect size for the IRE-3 collection is 272/ (272+132) - .50 = .175. In the test of ADI collection, 254 cases were found to favor full stemming, 107 cases favored suffix "s" stemming, and 129 cases favored neither. The effect size for this collection is .20. The results for the Cranfield-1 collection are opposite those for IRE-3 and ADI. For Cranfield-1, 134 cases favored full stemming, 371 favored the suffix "s" stemming, and 183 favored neither. The effect size for this collection is .235.

The most striking feature of these experiments is the discrepancy between the results for the IRE-3 and ADI collections, and the results for the Cranfield-1 collection. Salton offers the plausible explanation that the discrepancy is due to the highly technical and homogenous nature of the Cranfield vocabulary. He states:

To be able to differentiate between the various document abstracts, it is . . . important to maintain finer distinctions for the Cranfield case than for ADI and IRE, and these finer distinctions are lost when several words are combined into a unique class through the suffix cutoff process.

One of Salton's assertions about these experiments seems ill founded in light of the effect sizes calculated above. He states that, "For none of the collections (ADI, IRE-3, Cranfield-1) is the improvement of one method over the other really dramatic, so that in practice, either procedure might reasonably be used" . It seems much more reasonable, given the differences observed between the methods, to conclude that conflation may have a significant effect on retrieval performance, but that the effect may depend on the vocabulary involved. It may be, however, that Salton's remarks are based on unreported t-test data which may have indicated small effect sizes. The sign test is insensitive to the magnitude of differences, and thus a large effect might be found using a sign test, while a small effect was found using a t-test.

Hafer and Weiss (1974) tested their stemmer against other stemming methods using the ADI collection, and the Carolina Population Center (CPC) Collection consisting of 75 documents and five queries. Comparisons were made on the basis of recall-precision plots. No statistical testing was reported. For the ADI collection, the Hafer and Weiss stemmer outperformed the SMART stemmer. The authors also determined "each family derived from a common stem . . . by hand." . The method and criteria used are not reported. In a test using the CRC collection, the Hafer and Weiss stemmer performed equally to the manually derived stems. All methods of stemming outperformed full terms for both test collections.

Van Rijsbergen et al. (1980) tested their stemmer (Porter 1980) against the stemmer described by Dawson (1974) using the Cranfield-1 test collection. Both of these stemmers are of the longest match type, though the Dawson stemmer is more complex. They report that the performance of their stemmer was slightly better across ten paired recall-precision levels, but that the observed differences are probably not meaningful. No statistical results of any kind are reported.

Katzer et al. (1982) examined the performance of stemmed title-abstract terms against six other document representations: unstemmed title-abstract terms, unstemmed abstract terms, unstemmed title terms, descriptors, identifiers, and descriptors and identifiers combined. The stemmer used was a simple suffix removal stemmer employing 20 endings.

The database consisted of approximately 12,000 documents from Computer and Control Abstracts. Seven experienced searchers were used in the study, with each searcher searching different representations for each query. Relevance judgments on a scale of 1-4 (highly relevant to nonrelevant) were obtained from the users who requested the searches.

The dependent variable measures used were recall and precision for highly relevant documents, and recall and precision for all documents. On no dependent variable did stemmed terms perform less well than other representations, and in some cases they performed significantly better than descriptors, and on all document recall, stemming did significantly better than both descriptors and title terms. The effect sizes, calculated here on the basis of reported data, were .26 for highly relevant, and .22 and .20 for all relevant. The results of this study must be viewed with some caution because for a given query, different searchers searched different representations. This means that observed differences between representations may to some degree have been caused by searcher-representation interactions.

Lennon et al. (1981) examined the performance of various stemmers both in terms of retrieval effectiveness and inverted file compression. For the retrieval comparisons, the Cranfield-1400 collection was used. This collection contains 1,396 documents, and also 225 queries. The titles of documents and the queries were matched, and the documents ranked in order of decreasing match value. A cutoff was then applied to specify a given number of the highest ranked documents. This procedure was done for each of the methods used, that is, eight stemming methods, and nonstemmed terms.

The evaluation measure was E, as defined in Chapter 1. The various methods were evaluated at b levels of .5 (indicating that a user was twice as interested in precision as recall), and 2 (indicating that a user was twice as interested in recall as precision). These levels of b were not chosen by users, but were set by the experimenters.

The eight stemmers used in this study were: the Lovins stemmer, the Porter stemmer, the RADCOL stemmer, a suffix frequency stemmer based on the RADCOL project, a stemmer developed by INSPEC, the Hafer and Weiss stemmer, a trigram stemmer, and a stemmer based on the frequency of word endings in a corpus. As stated above, unstemmed terms were also used.

Thus, the two questions that this study addressed concerned (1) the relative effectiveness of stemming versus nonstemming, and (2) the relative performance of the various stemmers examined. In terms of the first question, only the Hafer and Weiss stemmer did worse than unstemmed terms at the two levels of b. At the b = .5 level, the INSPEC, Lovins, Porter, trigram, and both frequency stemmers all did better than unstemmed terms, and at the b = 2 level, all but Hafer and Weiss did better than unstemmed terms.

This information was derived from a table of means reported for the two b levels. As for the second question, though Lennon et al. (1981) report that, "a few of the differences between algorithms were statistically significant at the .05 probability level of significance in the precision oriented searches" , they fail to report any test statistics or effect sizes, or even identify which pairs of methods were significantly different. They conclude that

Despite the very different means by which the various algorithms have been developed, there is relatively little difference in terms of the dictionary compression and retrieval performance obtained from their use; in particular, simple, fully automated methods perform as well as procedures which involve a large degree of manual involvement in their development.

This assertion seems to be contradicted by the significant differences found between algorithms. However, the paucity of reported data in this study makes an independent evaluation of the results extremely difficult.

Frakes (1982) did two studies of stemming. The first tested the hypothesis that the closeness of stems to root morphemes will predict improved retrieval effectiveness. Fifty-three search queries were solicited from users and searched by four professional searchers under four representations: title, abstract, descriptors, and identifiers. The experiments were done using a database of approximately 12,000 document representations from six months of psychological abstracts. The retrieval system used was DIATOM (Waldstein 1981), a DIALOG simulator. In addition to most of the features of DIALOG, DIATOM allows stemmed searching using Lovins' (1968) algorithm, and permits search histories to be trapped and saved into files. As is the case with almost all IR systems, DIATOM allows only right and internal truncation; thus, the experiments did not address prefix removal.

User need statements were solicited from members of the Syracuse University academic community and other institutions. Each user need statement was searched by four experienced searchers under four different representations (abstract, title, identifiers, and descriptors). Each searcher did a different search for each representation. In addition, some user statements were searched using a stemmed title-abstract representation. This study made use of only title and abstract searches, since searches of these fields were most likely to need conflation, together with stemmed searches. However, relevant documents retrieved under all representations were used in the calculation of recall measures.

The hypothesis for the first experiment, based on linguistic theory, was that terms truncated on the right at the root morpheme boundary will perform better than terms truncated on the right at other points. To test this hypothesis, searches based on 34 user need statements were analyzed using multiple regression, and the non-parametric Spearman's test for correlation. The independent variables were positive and negative deviations of truncation points from root boundaries, where deviation was measured in characters. Root morphemes were determined by applying a set of generation rules to searcher supplied full-term equivalents of truncated terms, and checking the results against the linguistic intuitions of two judges.

The dependent variables were E measures (see Chapter 1) at three trade-off values: b = .5 ( precision twice as important as recall), b = 1 (precision equals recall in importance), b = 2 ( recall twice as important as precision). The data analysis showed that searchers truncated at or near the root morpheme boundaries. The mean number of characters that terms truncated by searchers varied from root boundaries in a positive direction was .825 with a standard deviation of .429. Deviations in a negative direction were even smaller with a mean of .035 and a standard deviation of .089. The tests of correlation revealed no significant relationship between the small deviations of stems from root boundaries, and retrieval effectiveness.

The purpose of the second experiment was to determine if conflation can be automated with no loss of retrieval effectiveness. Smith (1979), in an extensive survey of artificial intelligence techniques for information retrieval, stated that "the application of truncation to content terms cannot be done automatically to duplicate the use of truncation by intermediaries because any single rule [used by the conflation algorithm] has numerous exceptions" ( p. 223). However, no empirical test of the relative effectiveness of automatic and manual conflation had been made. The hypothesis adopted for the second experiment was that stemming will perform as well or better than manual conflation.

To test this hypothesis, searches based on 25 user need statements were analyzed. The primary independent variable was conflation method--with the levels automatic (stemming) and manual. The blocking variable was queries, and searchers was included as an additional independent variable to increase statistical power. Data was derived by re-executing each search with stemmed terms in place of the manually conflated terms in the original searches. The dependent variables were E measures at the three trade-off levels as described above. No significant difference was found between manual and automatic conflation, indicating that conflation can be automated with no significant loss in retrieval performance.

Walker and Jones (1987) did a thorough review and study of stemming algorithms. They used Porter's stemming algorithm in the study. The database used was an on-line book catalog (called RCL) in a library. One of their findings was that since weak stemming, defined as step 1 of the Porter algorithm, gave less compression, stemming weakness could be defined by the amount of compression. They also found that stemming significantly increases recall, and that weak stemming does not significantly decrease precision, but strong stemming does. They recommend that weak stemming be used automatically, with strong stemming reserved for those cases where weak stemming finds no documents. Another finding was that the number of searches per session was about the same for weak and strong stemming.

Harman (1991) used three stemmers--Porter, Lovins, and S removal--on three databases--Cranfield 1400, Medlars, and CACM--and found that none of them significantly improved retrieval effectiveness in a ranking IR system called IRX. The dependent variable measure was again E, at b levels of .5, 1.0, and 2.0. Attempts to improve stemming performance by reweighting stemmed terms, and by stemming only shorter queries, failed in this study.

8.3.1 Stemming Studies: Conclusion

Table 8.1 summarizes the various studies of stemming for improving retrieval effectiveness.

These studies must be viewed with caution. The failure of some of the authors to report test statistics, especially effect sizes, make interpretation difficult. Since some of the studies used sample sizes as small as 5 (Hafer and Weiss), their validity is questionable. Given these cautions, we offer the following conclusions.

       Stemming can affect retrieval performance, but the studies are equivocal. Where effects have been found, the majority have been positive, with the Hafer and Weiss stemmer in the study by Lennon et al., and the effect of the strong stemmer in the Walker and Jones study, being the exceptions. Otherwise there is no evidence that stemming will degrade retrieval effectiveness.

       Stemming is as effective as manual conflation.

       Salton's results indicate that the effect of stemming is dependent on the nature of the vocabulary used. A specific and homogeneous vocabulary may exhibit different conflation properties than will other types of vocabularies.

       There also appears to be little difference between the retrieval effectiveness of different full stemmers, with the exception of the Hafer and Weiss stemmer, which gave poorer performance in the Lennon et al. study.

Table 8.1: Stemming Retrieval Effectiveness Studies Summary

Study      Question       Test Collection  Dependent Vars  Results

-----------------------------------------------------------------------

Salton     Full stemmer   IRE-3            14 DV's         Full > s

           vs.

           suffix s       ADI                              Full > s

           stemmer

                          Cranfield-1                      s > Full

-----------------------------------------------------------------------

Hafer and  HW stemmer     ADI              recall(R),      HW > SMART

Weiss      vs.

           SMART                           precision (P)

           stemmer

           HW stemmer     CRC                              HW=Manual

           vs.

           manual

           stemming

           stemmed vs.    ADI                              Stemmed >

           unstemmed      CRC                              Unstemmed

-----------------------------------------------------------------------

VanRijs-   Porter vs.     Cranfield-1      R,P             Porter =

bergen     Dawson                                          Dawson

et al.     stemmer

-----------------------------------------------------------------------

Katzer     stemmed vs.    CCA              R,P(highest)    stemmed =

et al.     unstemmed      12,000           R,P(all)        unstemmed

           stemmed vs.                                     stemmed >

           other reps.                                     title,

                                                           descriptors

-----------------------------------------------------------------------

Lennon     stemmed vs.    Cranfield-1400   E(.5,2)         stemmed >

et al.     unstemmed                                       unstemmed >

                                                           HW stemmer

           stemmer                                         stemmers =

           comparison

-----------------------------------------------------------------------

Frakes     closeness      Psychabs         E(.5,1,2)       No

           of stem to                                      improvement

           root morpheme

           improves IR

           performance

           truncation                                      truncation=

           vs. stemming                                    stemming

-----------------------------------------------------------------------

Walker     stemmed vs.    PCL              R               stemmed >

and Jones  unstemmed                                       unstemmed

           weak stemmed   PCL              P               weak stemmed=

           vs. unstemmed                                   unstemmed

           strong stemmed PCL              P               strong

           vs. unstemmed                                   stemmed <

                                                           unstemmed

-----------------------------------------------------------------------

Harman     stemmed vs.    Cranfield-1400   E(.5,1,2)       stemmed =

           unstemmed                                       unstemmed

                          Medlars

                          CACM                             " "

                                                           " "

8.4 STEMMING TO COMPRESS INVERTED FILES

Since a stem is usually shorter than the words to which it corresponds, storing stems instead of full words can decrease the size of index files. Lennon et al. (1981), for example, report the following compression percentages for various stemmers and databases. For example, the indexing file for the Cranfield collection was 32.1 percent smaller after it was stemmed using the INSPEC stemmer.

Index Compression Percentages from Stemming

Stemmer  Cranfield  National Physical Laboratory  INSPEC  Brown Corpus

----------------------------------------------------------------------

INSPEC     32.1                 40.7                40.5      47.5

Lovins     30.9                 39.2                39.5      45.8

RADCOL     32.1                 41.8                41.8      49.1

Porter     26.2                 34.6                33.8      38.8

Frequency  30.7                 39.9                40.1      50.5

It is obvious from this table that the savings in storage using stemming can be substantial.

Harman (1991) reports that for a database of 1.6 megabytes of source text, the index was reduced 20 percent, from .47 megabytes to .38 megabytes, using the Lovins stemmer. For larger databases, however, reductions are smaller. For a database of 50 megabytes, the index was reduced from 6.7 to 5.8 megabytes--a savings of 13.5 percent. She points out, however, that for real world databases that contain numbers, misspellings, proper names, and the like, the compression factors are not nearly so large.

Compression rates increase for affix removal stemmers as the number of suffixes increases, as the following table, also from the Lennon et al. study, shows.

         Compression vs. Suffix Set Size: NPL Database

------------------------------------------------------------------

Number of Suffixes  100   200   300   400   500   600   700   800

Compression         28.2  33.7  35.9  37.3  38.9  40.1  41.0  41.6

8.5 SUMMARY

Stemmers are used to conflate terms to improve retrieval effectiveness and/or to reduce the size of indexing files. Stemming will, in general, increase recall at the cost of decreased precision. Studies of the effects of stemming on retrieval effectiveness are equivocal, but in general stemming has either no effect, or a positive effect, on retrieval performance where the measures used include both recall and precision. Stemming can have a marked effect on the size of indexing files, sometimes decreasing the size of the files as much as 50 percent.

Several approaches to stemming were described--table lookup, affix removal, successor variety, and n-gram. The Porter stemmer was described in detail, and a full implementation of it, in C, is presented in the appendix to this chapter. In the next chapter, the thesaurus approach to conflating semantically related words is discussed.

REFERENCES

ADAMSON, G., and J. BOREHAM. 1974. "The Use of an Association Measure Based on Character Structure to Identify Semantically Related Pairs of Words and Document Titles," Information Storage and Retrieval, 10, 253-60.

BENTLEY, J. 1982. Writing Efficient Programs. Englewood Cliffs, N.J.: Prentice Hall.

DAWSON, J. 1974. "Suffix Removal and Word Conflation." ALLC Bulletin, Michelmas, 33-46.

FRAKES, W. B. 1982. Term Conflation for Information Retrieval, Doctoral Dissertation, Syracuse University.

FRAKES, W. B. 1984. "Term Conflation for Information Retrieval" in Research and Development in Information Retrieval, ed. C. van Rijsbergen. New York: Cambridge University Press.

FRAKES, W. B. 1986. "LATTIS: A Corporate Library and Information System for the UNIX Environment," Proceedings of the National Online Meeting, Medford, N.J.: Learned Information Inc., 137-42.

HAFER, M., and S. WEISS. 1974. "Word Segmentation by Letter Successor Varieties," Information Storage and Retrieval, 10, 371-85.

HARMAN, D. 1991. "How Effective is Suffixing?" Journal of the American Society for Information Science, 42(1), 7-15.

KATZER, J., M. MCGILL, J. TESSIER, W. FRAKES, and P. DAS GUPTA. 1982. "A Study of the Overlaps among Document Representations," Information Technology: Research and Development, 1, 261-73.

LENNON, M., D. PIERCE, B. TARRY, and P. WILLETT. 1981. "An Evaluation of Some Conflation Algorithms for Information Retrieval." Journal of Information Science 3, 177-83.

LOVINS, J. B. 1968. "Development of a Stemming Algorithm." Mechanical Translation and Computational Linguistics, 11(1-2), 22-31.

PAICE, C. 1990. "Another Stemmer," ACM SIGIR Forum, 24(3), 56-61.

PORTER, M. F. 1980. "An Algorithm for Suffix Stripping." Program, 14(3), 130-37.

SALTON, G. 1968. Automatic Information Organization and Retrieval. New York: McGraw-Hill.

SMITH, L. C. 1979. Selected Artificial Intelligence Techniques in Information Retrieval. Doctoral Dissertation, Syracuse University.

VAN RIJSBERGEN, C. J., S. E. ROBERTSON, and M. F. PORTER. 1980. New Models in Probabilistic Information Retrieval. Cambridge: British Library Research and Development Report no. 5587.

WALKER, S., and R. JONES. 1987. "Improving Subject Retrieval in Online Catalogues." British Library Research Paper no. 24, vol. 1.

WALDSTEIN, R. 1981. "Diatom: A Dialog Simulator." Online, 5, 68-72.

APPENDIX

/******************************* stem.c **********************************

Version:     1.1

Purpose:     Implementation of the Porter stemming algorithm.

Provenance:  Written by C. Fox, 1990.

Notes:       This program is a rewrite of earlier versions by B. Frakes and S. Cox

**/

/************************ Standard Include Files *************************/

#include <stdio.h>

#include <string.h>

#include <ctype.h>

/******************** Private Function Declarations **********************/

static int WordSize( /* word */ );

static int ContainsVowel( /* word */ );

static int EndsWithCVC( /* word */ );

static int AddAnE( /* word */ );

static int RemoveAnE( /* word */ );

static int ReplaceEnd( /* word, rule */ );

/***************** Private Defines and Data Structures *******************/

#define FALSE                      0

#define TRUE                       1

#define EOS                        '\0'

#define IsVowel(c)       ('a' == (c) | | 'e' == (c) | | 'i' == (c) | | 'o' == (c) | | 'u' == (c))

typedef struct {

int id;               /* returned if rule fired */

char *old_end;        /* suffix replaced */

char *new_end;        /* suffix replacement */

int old_offset;       /* from end of word to start of suffix */

int new_offset;       /* from beginning to end of new suffix */

int min_root_size;    /* min root word size for replacement */

int (*condition)();   /* the replacement test function */

} RuleList;

static char LAMBDA[1] = " ";     /* the constant empty string */

static char *end;                /* pointer to the end of the word */

static RuleList step1a_rules[] =

{

101,  "sses",     "ss",     3,   1,  -1,  NULL,

102,  "ies",       "i",     2,   0,  -1,  NULL,

103,  "ss",       "ss",     1,   1,  -1,  NULL,

104,  "s",        LAMBDA,   0,  -1,  -1,  NULL,

000,  NULL,       NULL,     0,   0,   0,  NULL,

};

static RuleList step1b_rules[] =

{

105,  "eed",      "ee",     2,   1,   0,  NULL,

106,  "ed",       LAMBDA,   1,  -1,  -1,  ContainsVowel,

107,  "ing",      LAMBDA,   2,  -1,  -1,  ContainsVowel,

000,  NULL,       NULL,     0,   0,   0,  NULL,

};

static RuleList step1b1_rules[ ] =

{

108,  "at",       "ate",    1,   2,  -1,  NULL,

109,  "bl",       "ble",    1,   2,  -1,  NULL,

110,  "iz",       "ize",    1,   2,  -1,  NULL,

111,  "bb"        "b",      1,   0,  -1,  NULL,

112,  "dd"        "d",      1,   0,  -1,  NULL,

113,  "ff",       "f",      1,   0,  -1,  NULL,

114,  "gg",       "g",      1,   0,  -1,  NULL,

115,  "mm,        "m",      1,   0,  -1,  NULL,

116,  "nn",       "n",      1,   0,  -1,  NULL,

117,  "pp",       "p",      1,   0,  -1,  NULL,

118,  "rr",       "r",      1,   0,  -1,  NULL,

119,  "tt",       "t",      1,   0,  -1,  NULL,

120,  "ww",       "w",      1,   0,  -1,  NULL,

121,  "xx",       "x",      1,   0,  -1,  NULL,

122,  LAMBDA      "e",     -1,   0,  -1,  AddAnE,

000,  NULL,       NULL,     0,   0,   0,  NULL,

} ;

static RuleList step1c_rules[ ] =

{

123,  "y",        "i",      0,   0,  -1,  ContainsVowel,

000,  NULL,       NULL,    0,   0,   0,  NULL,

};

static RuleList step2_rules[ ] =

{

203,  "ational",  "ate",    6,   2,   0,  NULL,

204,  "tional",   "tion",   5,   3,   0,  NULL,

205,  "enci",     "ence",   3,   3,   0,  NULL,

206,  "anci",     "ance",   3,   3,   0,  NULL,

207,  "izer",     "ize",    3,   2,   0,  NULL,

208,  "abli",     "able",   3,   3,   0,  NULL,

209,  "alli",     "al",     3,   1,   0,  NULL,

210,  "entli",    "ent",    4,   2,   0,  NULL,

211,  "eli",      "e",      2,   0,   0,  NULL,

213,  "ousli",    "ous",    4,   2,   0,  NULL,

214,  "ization",  "ize",    6,   2,   0,  NULL,

215,  "ation",    "ate",    4,   2,   0,  NULL,

216,  "ator",     "ate",    3,   2,   0,  NULL,

217,  "alism",    "al",     4,   1,   0,  NULL,

218,  "iveness",  "ive",    6,   2,   0,  NULL,

219,  "fulnes",   "ful",    5,   2,   0,  NULL,

220,  "ousness",  "ous",    6,   2,   0,  NULL,

221,  "aliti",    "al",     4,   1,   0,  NULL,

222,  "iviti",    "ive",    4,   2,   0,  NULL,

223,  "biliti",   "ble",    5,   2,   0,  NULL,

000,  NULL,       NULL,     0,   0,   0,  NULL,

};

static RuleList step3_rules[ ] =

{

301,  "icate",    "ic",     4,   1,   0,  NULL,

302,  "ative",    LAMBDA,   4,  -1,   0,  NULL,

303,  alize",     "al",     4,   1,   0,  NULL,

304,  "iciti",    "ic",     4,   1,   0,  NULL,

305,  "ical",     "ic",     3,   1,   0,  NULL,

308,  "ful",      LAMBDA,   2,  -1,   0,  NULL,

309,  "ness",     LAMBDA,   3,  -1,   0,  NULL,

000,  NULL,       NULL,     0,   0,   0,  NULL,

};

static RuleList step4_rules[ ] =

{

401,  "al",       LAMBDA,   1,  -1,   1,  NULL,

402,  "ance",     LAMBDA,   3,  -1,   1,  NULL,

403,  "ence",     LAMBDA,   3,  -1,   1,  NULL,

405,  "er",       LAMBDA,   1,  -1,   1,  NULL,

406,  "ic",       LAMBDA,   1,  -1,   1,  NULL,

407,  "able",     LAMBDA,   3,  -1,   1,  NULL,

408,  "ible",     LAMBDA,   3,  -1,   1,  NULL,

409,  "ant",      LAMBDA,   2,  -1,   1,  NULL,

410,  "ement",    LAMBDA,   4,  -1,   1,  NULL,

411,  "ment",     LAMBDA,   3,  -1,   1,  NULL,

412,  "ent",      LAMBDA,   2,  -1,   1,  NULL,

423,  "sion",     "s",      3,   0,   1,  NULL,

424,  "tion",     "t",      3,   0,   1,  NULL,

415,  "ou",       LAMBDA,   1,  -1,   1,  NULL,

416,  "ism",      LAMBDA,   2,  -1,   1,  NULL,

417,  "ate",      LAMBDA,   2,  -1,   1,  NULL,

418,  "iti",      LAMBDA,   2,  -1,   1,  NULL,

419,  "ous",      LAMBDA,   2,  -1,   1,  NULL,

420,  "ive",      LAMBDA,   2,  -1,   1,  NULL,

421,  "ize",      LAMBDA,   2,  -1,   1,  NULL,

000,  NULL,       NULL,     0,   0,   0,  NULL,

};

static RuleList step5a_rules[ ] =

{

501,  "e",        LAMBDA,   0,  -1,   1,  NULL,

502,  "e",        LAMBDA,   0,  -1,  -1,  RemoveAnE,

000,  NULL,       NULL,     0,   0,   0,  NULL,

};

static RuleList step5b_rules[ ] =

{

503,  "ll",       "l",      1,   0,   1,  NULL,

000,  NULL,       NULL,     0,   0,   0,  NULL,

};

/*******************************************************************/

/********************  Private Function Declarations  **************/

/*FN****************************************************************

WordSize( word )

Returns:  int -- a weird WordSize of word size in adjusted syllables

Purpose:  Count syllables in a special way: count the number

vowel-consonant pairs in a word, disregarding initial

consonants and final vowels. The letter "y" counts as a

consonant at the beginning of a word and when it has a vowel

in front of it; otherwise (when it follows a consonant) it

is treated as a vowel. For example, the WordSize of "cat"

is 1, of "any" is 1, of "amount" is 2, of "anything" is 3.

Plan:     Run a DFA to compute the word size

Notes:    The easiest and fastest way to compute this funny measure is

with a finite state machine. The initial state 0 checks

the first letter. If it is a vowel, then the machine changes

to state 1, which is the "last letter was a vowel" state.

If the first letter is a consonant or y, then it changes

to state 2, the "last letter was a consonant state". In

state 1, a y is treated as a consonant (since it follows

a vowel), but in state 2, y is treated as a vowel (since

it follows a consonant. The result counter is incremented

on the transition from state 1 to state 2, since this

transition only occurs after a vowel-consonant pair, which

is what we are counting.

**/

static int

WordSize( word )

register char *word;  /*  in: word having its WordSize taken  */

{

register int result;  /*  WordSize of the word  */

register int state;   /*  current state in machine  */

result = 0;

state = 0;

/* Run a DFA to computer the word size */

while ( EOS != *word )

{

switch ( state )

{

case 0:  state = (IsVowel(*word)) ? 1: 2;

break;

case 1:  state = (IsVowel(*word)) ? 1 : 2;

if ( 2 == state ) result++;

break;

case 2:  state = (IsVowel(*word) | | ('y' == *word)) ? 1 : 2;

break;

}

word++;

}

return( result );

} /*  WordSize  */

/*FN************************************************************************

ContainsVowel( word )

Returns:  int -- TRUE (1) if the word parameter contains a vowel,

FALSE (0) otherwise.

Purpose:  Some of the rewrite rules apply only to a root containing

a vowel, where a vowel is one of "aeiou" or y with a

consonant in front of it.

Plan:     Obviously, under the definition of a vowel, a word contains

a vowel iff either its first letter is one of "aeiou", or

any of its other letters are "aeiouy". The plan is to

test this condition.

Notes:    None

**/

static int

ContainsVowel( word )

register char *word; /*  in: buffer with word checked */

{

if  (  EOS == *word  )

return(  FALSE  );

else

return ( IsVowel(*word)  | |  (NULL != strpbrk(word+1,"aeiouy"))  );

}  /*  ContainsVowel */

/*FN**********************************************************************

EndsWithCVC( word )

Returns:  int -- TRUE (1) if the current word ends with a

consonant-vowel-consonant combination, and the second

consonant is not w, x, or y, FALSE (0) otherwise.

Purpose:  Some of the rewrite rules apply only to a root with

this characteristic.

Plan:     Look at the last three characters.

Notes:    None

**/

static int

EndsWithCVC( word )

register char *word;  /* in: buffer with the word checked */

{

int length;        /* for finding the last three characters */

if  ( (length = strlen(word)) < 2 )

return( FALSE );

else

{

end = word + length - 1;

return(    (NULL == strchr("aeiouwxy", *end--))     /* consonant */

&& (NULL != strchr("aeiouy",   *end--))        /* vowel */

&& (NULL == strchr("aeiou",    *end  )) )   /* consonant */

}

} /* EndsWithCVC */

/*FN***********************************************************************

AddAnE( word )

Returns:  int -- TRUE (1) if the current word meets special conditions

for adding an e.

Purpose:  Rule 122 applies only to a root with this characteristic.

Plan:     Check for size of 1 and a consonant-vowel-consonant ending.

Notes:    None

**/

static int

AddAnE( word )

register char *word;

{

return( (1 == WordSize(word)) && EndsWithCVC(word) );

} /* AddAnE */

/*FN************************************************************************

RemoveAnE( word )

Returns:  int -- TRUE (1) if the current word meets special conditions

for removing an e.

Purpose:  Rule 502 applies only to a root with this characteristic.

Plan:     Check for size of 1 and no consonant-vowel-consonant ending.

Notes:    None

**/

static int

RemoveAnE( word )

register char *word;

{

return( (1 == WordSize(word)) && !EndsWithCVC(word) );

} /* RemoveAnE */

/*FN************************************************************************

ReplaceEnd( word, rule )

Returns:  int -- the id for the rule fired, 0 is none is fired

Purpose:  Apply a set of rules to replace the suffix of a word

Plan:     Loop through the rule set until a match meeting all conditions

is found. If a rule fires, return its id, otherwise return 0.

Connditions on the length of the root are checked as part of this

function's processing because this check is so often made.

Notes:    This is the main routine driving the stemmer. It goes through

a set of suffix replacement rules looking for a match on the

current suffix. When it finds one, if the root of the word

is long enough, and it meets whatever other conditions are

required, then the suffix is replaced, and the function returns.

**/

static int

ReplaceEnd( word, rule )

register char *word;  /* in/out: buffer with the stemmed word */

RuleList *rule;       /* in: data structure with replacement rules */

{

register char *ending;  /* set to start of possible stemmed suffix */

char tmp_ch;            /* save replaced character when testing */

while ( 0 != rule->id )

{

ending = end - rule->old_offset;

if ( word = ending )

if ( 0 == strcmp(ending,rule->old_end) )

{

tmp_ch = *ending;

*ending = EOS;

if ( rule->min_root_size < WordSize(word) )

if ( !rule->condition | | (*rule->condition)(word) )

{

(void)strcat( word, rule->new_end );

end = ending + rule->new_offset;

break;

}

*ending = tmp_ch;

}

rule++;

}

return( rule->id );

} /* ReplaceEnd */

/************************************************************************/

/*********************  Public Function Declarations  ********************/

/*FN************************************************************************

Stem( word )

Returns:  int -- FALSE (0) if the word contains non-alphabetic characters

and hence is not stemmed, TRUE (1) otherwise

Purpose:  Stem a word

Plan:     Part 1: Check to ensure the word is all alphabetic

Part 2: Run through the Porter algorithm

Part 3: Return an indication of successful stemming

Notes:    This function implements the Porter stemming algorithm, with

a few additions here and there. See:

Porter, M.F., "An Algorithm For Suffix Stripping,"

Program 14 (3), July 1980, pp. 130-137.

Porter's algorithm is an ad hoc set of rewrite rules with

various conditions on rule firing. The terminology of

"step 1a" and so on, is taken directly from Porter's

article, which unfortunately gives almost no justification

for the various steps. Thus this function more or less

faithfully refects the opaque presentation in the article.

Changes from the article amount to a few additions to the

rewrite rules; these are marked in the RuleList data

structures with comments.

**/

int

Stem( word )

register char *word;  /* in/out: the word stemmed  */

{

int rule;    /* which rule is fired in replacing an end  */

/* Part 1: Check to ensure the word is all alphabetic */

for ( end = word; *end != EOS; end++ )

if ( !isalpha(*end) ) return( FALSE );

end--;

/* Part 2: Run through the Porter algorithm */

ReplaceEnd( word, step1a_rules );

rule = ReplaceEnd( word, step1b_rules );

if (  (106 == rule) | | (107 == rule) )

ReplaceEnd( word, step1b1_rules );

ReplaceEnd( word, step1c_rules );

ReplaceEnd( word, step2_rules );

ReplaceEnd( word, step3_rules );

ReplaceEnd( word, step4_rules );

ReplaceEnd( word, step5a_rules );

ReplaceEnd( word, step5b_rules );

/* Part 3: Return an indication of successful stemming */

return( TRUE );

} /* Stem */

Go to Chapter 9     Back to Table of Contents


Information Retrieval: CHAPTER 9: THESAURUS CONSTRUCTION

 Return to Table of Contents

CHAPTER 9: THESAURUS CONSTRUCTION

Padmini Srinivasan

University of Iowa

Abstract

Thesauri are valuable structures for Information Retrieval systems. A thesaurus provides a precise and controlled vocabulary which serves to coordinate document indexing and document retrieval. In both indexing and retrieval, a thesaurus may be used to select the most appropriate terms. Additionally, the thesaurus can assist the searcher in reformulating search strategies if required. This chapter first examines the important features of thesauri. This should allow the reader to differentiate between thesauri. Next, a brief overview of the manual thesaurus construction process is given. Two major approaches for automatic thesaurus construction have been selected for detailed examination. The first is on thesaurus construction from collections of documents, and the second, on thesaurus construction by merging existing thesauri. These two methods were selected since they rely on statistical techniques alone and are also significantly different from each other. Programs written in C language accompany the discussion of these approaches.

9.1 INTRODUCTION

Roget's thesaurus is very different from the average thesaurus associated with Information Retrieval (IR) systems accessing machine readable document databases. The former is designed to assist the writer in creatively selecting vocabulary. In IR systems, used to retrieve potentially relevant documents from large collections, the thesaurus serves to coordinate the basic processes of indexing and document retrieval. (The term document is used here generically and may refer to books, articles, magazines, letters, memoranda, and also software.) In indexing, a succinct representation of the document is derived, while retrieval refers to the search process by which relevant items are identified. The IR thesaurus typically contains a list of terms, where a term is either a single word or a phrase, along with the relationships between them. It provides a common, precise, and controlled vocabulary which assists in coordinating indexing and retrieval. Given this objective, it is clear that thesauri are designed for specific subject areas and are therefore domain dependent.

Figure 9.1 displays a short extract from an alphabetical listing of thesaurus terms and their relationships in the INSPEC thesaurus. This thesaurus is designed for the INSPEC domain, which covers physics, electrical engineering, electronics, as well as computers and control. The thesaurus is logically organized as a set of hierarchies. In addition to this hierarchical arrangement, the printed INSPEC thesaurus also includes an alphabetical listing of thesaural terms. Each hierarchy is built from a root term representing a high-level concept in the domain. In Figure 9.1, "computer-aided instruction" is a part of the hierarchy whose root note or top term (TT) is "computer applications." The "see also" link leads to cross-referenced thesaural terms. NT suggests a more specific thesaural term; similarly, BT provides a more general thesaural term. RT signifies a related term and this relationship includes a variety of links such as part-whole and object-property. UF is utilized to indicate the chosen form from a set of alternatives. In the above example, "computer-aided instruction" is used for the alternative "teaching machines." The converse of UF is USE, which takes the user from a form that is not part of the vocabulary to a valid alternative. The example shows that "caesium" is the preferred form over "cesium." CC and FC are from the INSPEC classification scheme and indicate the subject area in which the term is used.

          cesium

                    USE caesium

          computer-aided instruction

                    see also education

                    UF  teaching machinse

                    BT  educational computing

                    TT  computer applications

                    RT  education

                        teaching

                    CC  C7810C

                    FC  7810Cf

Figure 9.1: A short extract from the 1979 INSPEC thesaurus

A carefully designed thesaurus can be of great value. The indexer is typically instructed to select the most appropriate thesaural entries for representing the document. In searching, the user can employ the thesaurus to design the most appropriate search strategy. If the search does not retrieve enough documents, the thesaurus can be used to expand the query by following the various links between terms. Similarly, if the search retrieves too many items, the thesaurus can suggest more specific search vocabulary. In this way the thesaurus can be valuable for reformulating search strategies. It is quite common to provide on-line thesauri, which simplifies the query reformulation process. In addition, such query modifications can also be more system initiated than user initiated. However, this obviously requires rather complex algorithms since the system has to know not only how to reformulate the query but also when to do so.

There exists a vast literature on the principles, methodologies, and problems involved in thesaurus construction. However, only a small portion is devoted to the "automatic" construction of thesauri. This mirrors the current state of art, marked by an abundance of manually generated thesauri. In fact, there is even much skepticism over the possibility of fully automating this procedure. This is because manual thesauri are highly complex structures which exhibit a variety of relationships  including hierarchical, nonhierarchical, equivalent, and associative ones. The automatic detection of such relationships continues to be challenging. But, the reader can be assured that some automatic methodologies have emerged over the last few decades. As in most other subfields of IR, these methods are strongly motivated by statistics. In contrast, manual thesaurus construction is a highly conceptual and knowledge-intensive task and therefore also extremely labor intensive. Consequently, the search for alternative automatic methods is definitely of value.

The rest of this chapter is organized into six sections. The next, section 9.2, describes features of thesauri. Section 9.3 introduces the different approaches for automatic thesaurus construction following a brief description of manual thesaurus construction. Sections 9.4 and 9.5 focus on two major approaches for automatic thesaurus construction. The three C programs included with this chapter are described briefly in section 9.6. The last section presents the conclusions.

9.2 FEATURES OF THESAURI

Some important features of thesauri will be highlighted here. For a more detailed discussion, please consult Aitchison and Gilchrist (1972). The objective is for the reader to be able to compare thesauri. In general, the discussion applies to both manually and automatically generated thesauri. However, differences between the two are also identified where appropriate.

9.2.1 Coordination Level

Coordination refers to the construction of phrases from individual terms. Two distinct coordination options are recognized in thesauri: precoordination and post-coordination. A precoordinated thesaurus is one that can contain phrases. Consequently, phrases are available for indexing and retrieval. A postcoordinated thesaurus does not allow phrases. Instead, phrases are constructed while searching. The choice between the two options is difficult. The advantage in precoordination is that the vocabulary is very precise, thus reducing ambiguity in indexing and in searching. Also, commonly accepted phrases become part of the vocabulary. However, the disadvantage is that the searcher has to be aware of the phrase construction rules employed. Thesauri can adopt an intermediate level of coordination by allowing both phrases and single words. This is typical of manually constructed thesauri. However, even within this group there is significant variability in terms of coordination level. Some thesauri may emphasize two or three word phrases, while others may emphasize even larger sized phrases. Therefore, it is insufficient to state that two thesauri are similar simply because they follow precoordination. The level of coordination is important as well. It should be recognized that the higher the level of coordination, the greater the precision of the vocabulary but the larger the vocabulary size. It also implies an increase in the number of relationships to be encoded. Therefore, the thesaurus becomes more complex. The advantage in postcoordination is that the user need not worry about the exact ordering of the words in a phrase. Phrase combinations can be created as and when appropriate during searching. The disadvantage is that search precision may fall, as illustrated by the following well known example, from Salton and McGill (1983): the distinction between phrases such as "Venetian blind" and "blind Venetian" may be lost. A more likely example is "library school" and "school library." The problem is that unless search strategies are designed carefully, irrelevant items may also be retrieved. Precoordination is more common in manually constructed thesauri. Automatic phrase construction is still quite difficult and therefore automatic thesaurus construction usually implies post-coordination. Section 9.4 includes a procedure for automatic phrase construction.

9.2.2 Term Relationships

Term relationships are the most important aspect of thesauri since the vocabulary connections they provide are most valuable for retrieval. Many kinds of relationships are expressed in a manual thesaurus. These are semantic in nature and reflect the underlying conceptual interactions between terms. We do not provide an exhaustive discussion or listing of term relationships here. Instead, we only try to illustrate the variety of relationships that exist. Aitchison and Gilchrist (1972) specify three categories of term relationships: (1) equivalence relationships, (2) hierarchical relationships, and (3) nonhierarchical relationships. The example in Figure 9.1 illustrates all three categories. Equivalence relations include both synonymy and quasi-synonymy. Synonyms can arise because of trade names, popular and local usage, superseded terms, and the like. Quasi-synonyms are terms which for the purpose of retrieval can be regarded as synonymous, for example, "genetics" and "heredity," which have significant overlap in meaning. Also, the terms "harshness" and "tenderness," which represent different viewpoints of the same property continuum. A typical example of a hierarchical relation is genus-species, such as "dog" and "german shepherd." Nonhierarchical relationships also identify conceptually related terms. There are many examples including: thing--part such as "bus" and "seat"; thing--attribute such as "rose" and "fragance."

Wang, Vandendorpe, and Evens (1985) provide an alternative classification of term relationships consisting of: (1) parts--wholes, (2) collocation relations, (3) paradigmatic relations, (4) taxonomy and synonymy, and (5) antonymy relations. Parts and wholes include examples such as set--element; count--mass. Collocation relates words that frequently co-occur in the same phrase or sentence. Paradigmatic relations relate words that have the same semantic core like "moon" and "lunar" and are somewhat similar to Aitchison and Gilchrist's quasi-synonymy relationship. Taxonomy and synonymy are self-explanatory and refer to the classical relations between terms. It should be noted that the relative value of these relationships for retrieval is not clear. However, some work has been done in this direction as in Fox (1981) and Fox et al. (1988). Moreover, at least some of these semantic relationships are commonly included in manual thesauri. Identifying these relationships requires knowledge of the domain for which the thesaurus is being designed. Most if not all of these semantic relationships are difficult to identify by automatic methods, especially by algorithms that exploit only the statistical relationships between terms as exhibited in document collections. However, it should be clear that these statistical associations are only as good as their ability to reflect the more interesting and important semantic associations between terms.

9.2.3 Number of Entries for Each Term

It is in general preferable to have a single entry for each thesaurus term. However, this is seldom achieved due to the presence of homographs--words with  multiple meanings. Also, the semantics of each instance of a homograph can only be contextually deciphered. Therefore, it is more realistic to have a unique representation or entry for each meaning of a homograph. This also allows each homograph entry to be associated with its own set of relations. The problem is that multiple term entries add a degree of complexity in using the thesaurus--especially if it is to be used automatically. Typically the user has to select between alternative meanings. In a manually constructed thesaurus such as INSPEC, this problem is resolved by the use of parenthetical qualifiers, as in the pair of homographs, bonds (chemical) and bonds (adhesive). However, this is hard to achieve automatically.

9.2.4 Specificity of Vocabulary

Specificity of the thesaurus vocabulary is a function of the precision associated with the component terms. A highly specific vocabulary is able to express the subject in great depth and detail. This promotes precision in retrieval. The concomitant disadvantage is that the size of the vocabulary grows since a large number of terms are required to cover the concepts in the domain. Also, specific terms tend to change (i.e., evolve) more rapidly than general terms. Therefore, such vocabularies tend to require more regular maintenance. Further, as discussed previously, high specificity implies a high coordination level which in turn implies that the user has to be more concerned with the rules for phrase construction.

9.2.5 Control on Term Frequency of Class Members

This has relevance mainly for statistical thesaurus construction methods which work by partitioning the vocabulary into a set of classes where each class contains a collection of equivalent terms. Salton and McGill (1983, 77-78) have stated that in order to maintain a good match between documents and queries, it is necessary to ensure that terms included in the same thesaurus class have roughly equal frequencies. Further, the total frequency in each class should also be roughly similar. (Appropriate frequency counts for this include: the number of term occurrences in the document collection; the number of documents in the collection in which the term appears at least once). These constraints are imposed to ensure that the probability of a match between a query and a document is the same across classes. In other words, terms within the same class should be equally specific, and the specificity across classes should also be the same.

9.2.6 Normalization of Vocabulary

Normalization of vocabulary terms is given considerable emphasis in manual thesauri. There are extensive rules which guide the form of the thesaural entries. A simple rule is that terms should be in noun form. A second rule is that noun phrases should avoid prepositions unless they are commonly known. Also, a limited number of adjectives should be used. There are other rules to direct issues such as the singularity of terms, the ordering of terms within phrases, spelling, capitalization, transliteration, abbreviations, initials, acronyms, and punctuation. In other words, manual thesauri are designed using a significant set of constraints and rules regarding the structure of individual terms. The advantage in normalizing the vocabulary is that variant forms are mapped into base expressions, thereby bringing consistency to the vocabulary. As a result, the user does not have to worry about variant forms of a term. The obvious disadvantage is that, in order to be used effectively, the user has to be well aware of the normalization rules used. This is certainly nontrivial and often viewed as a major hurdle during searching (Frost 1987). In contrast, normalization rules in automatic thesaurus construction are simpler, seldom involving more than stoplist filters and stemming. (These are discussed in more detail later.) However, this feature can be regarded both as a weakness and a strength.

These different features of thesauri have been presented so that the reader is aware of some of the current differences between manual and automatic thesaurus construction methods. All features are not equally important and they should be weighed according to the application for which the thesaurus is being designed. This section also gives an idea of where further research is required. Given the growing abundance of large-sized document databases, it is indeed important to be challenged by the gaps between manual and automatic thesauri.

9.3 THESAURUS CONSTRUCTION

9.3.1 Manual Thesaurus Construction

The process of manually constructing a thesaurus is both an art and a science. We present here only a brief overview of this complex process. First, one has to define the boundaries of the subject area. (In automatic construction, this step is simple, since the boundaries are taken to be those defined by the area covered by the document database.) Boundary definition includes identifying central subject areas and peripheral ones since it is unlikely that all topics included are of equal importance. Once this is completed, the domain is generally partitioned into divisions or subareas. Once the domain, with its subareas, has been sufficiently defined, the desired characteristics of the thesaurus have to be identified. Since manual thesauri are more complex structurally than automatic ones, as the previous section has shown, there are more decisions to be made.

Now, the collection of terms for each subarea may begin. A variety of sources may be used for this including indexes, encyclopedias, handbooks, textbooks, journal titles and abstracts, catalogues, as well as any existing and relevant thesauri or vocabulary systems. Subject experts and potential users of the thesaurus should also be included in this step. After the initial vocabulary has been identified, each term is analyzed for its related vocabulary including synonyms, broader and narrower terms, and sometimes also definitions and scope notes. These terms and their relationships are then organized into structures such as hierarchies, possibly within each subarea. The process of organizing the vocabulary may reveal gaps which can lead to the addition of terms; identify the need for new levels in the hierarchies; bring together synonyms that were not previously recognized; suggest new relationships between terms; and reduce the vocabulary size. Once the initial organization has been completed, the entire thesaurus will have to be reviewed (and refined) to check for consistency such as in phrase form and word form. Special problems arise in incorporating terms from existing thesauri which may for instance have different formats and construction rules. At this stage the hierarchically structured thesaurus has to be "inverted" to produce an alphabetical arrangement of entries--a more effective arrangement for use. Typically both the alphabetical and hierarchical arrangements are provided in a thesaurus. Following this, the manually generated thesaurus is ready to be tested by subject experts and edited to incorporate their suggestions.

The above informal description is very sketchy. It is a long process that involves a group of individuals and a variety of resources. Once the thesaurus has been designed and implemented for use within a retrieval system, the next problem is that it needs to be maintained in order to ensure its continued viability and effectiveness. That is, the thesaurus should reflect any changes in the terminology of the area. The problem is that since the older documents are still within the system, the updated thesaurus must also retain the older information. Updates are typically slow and again involve several individuals who regularly review and suggest new and modified vocabulary terms as well as relationships. Therefore, typically a thesaurus evolves with time and slowly responds to changes in the terminology of the subject.

9.3.2 Automatic Thesaurus Construction

In selecting automatic thesaurus construction approaches for discussion here, the criteria used are that they should be quite different from each other in addition to being interesting. Also, they should use purely statistical techniques. (The alternative is to use linguistic methods.) Consequently, the two major approaches selected here have not necessarily received equal attention in the literature. The first approach, on designing thesauri from document collections, is a standard one. The second, on merging existing thesauri, is better known using manual methods. Programs included with this chapter are based on these two major approaches. We also discuss a third automatic approach which is quite novel and interesting, although it is based on tools from expert systems and does not use statistical methods. In this approach, thesauri are built using information obtained from users. We first present a brief overview of each approach and then provide detailed descriptions.

9.3.3 From a Collection of Document Items

Here the idea is to use a collection of documents as the source for thesaurus construction. This assumes that a representative body of text is available. The idea is to apply statistical procedures to identify important terms as well as their significant relationships. It is reiterated here that the central thesis in applying statistical methods is to use computationally simpler methods to identify the more important semantic knowledge for thesauri. It is semantic knowledge that is used by both indexer and searcher. Until more direct methods are known, statistical methods will continue to be used. Work by Soergel (1974) is relevant to this point since it includes an interesting discussion on the various semantic interpretations of significant statistical associations between words. The first two programs, select.c and hierarchy.c, included with this chapter are based on this approach of designing a thesaurus from a collection of documents.

9.3.4 By Merging Existing Thesauri

This second approach is appropriate when two or more thesauri for a given subject exist that need to be merged into a single unit. If a new database can indeed be served by merging two or more existing thesauri, then a merger perhaps is likely to be more efficient than producing the thesaurus from scratch. This approach has been discussed at some length in Forsyth and Rada (1986). The challenge is that the merger should not violate the integrity of any component thesaurus. Rada has experimented with augmenting the MeSH thesaurus with selected terms from SNOMED (Forsyth and Rada 1986, 216). MeSH stands for Medical Subject Headings and is the thesaurus used in MEDLINE, a medical document retrieval system, constructed and maintained by the National Library of Medicine. It provides a sophisticated controlled vocabulary for indexing and accessing medical documents. SNOMED, which stands for Systematized Nomenclature of Medicine, is a detailed thesaurus developed by the College of American Pathologists for use in hospital records. MeSH terms are used to describe documents, while SNOMED terms are for describing patients. Ideally, a patient can be completely described by choosing one or more terms from each of several categories in SNOMED. Both MeSH and SNOMED follow a heirarchical structure. Rada's focus in his experiments has been on developing suitable algorithms for merging related but separate thesauri such as MeSH and SNOMED and also in evaluating the end products. The third program (merge.c) included here  implements two different merging algorithms adapted from Rada's work.

9.3.5 User Generated Thesaurus

In this third alternative, the idea is that users of IR systems are aware of and use many term relationships in their search strategies long before these find their way into thesauri. The objective is to capture this knowledge from the user's search. This is the basis of TEGEN--the thesaurus generating system designed by Guntzer et al. (1988). They propose TEGEN as a viable alternative technique for automatic thesaurus construction. The procedure involves examining the types of Boolean operators used between search terms, the type of query modification performed by the user, and so on. User feedback is included to resolve any ambiguities and uncertainties. Feedback is also required to select the specific type of relationship between two terms once it has been decided that the terms are indeed related. Therefore, their approach requires considerable interaction with the user population. TEGEN is designed using production rules which perform a detailed analysis of the user's search pattern. Given that this approach utilizes mainly expert system methodologies, no representative program is included here.

9.4 THESAURUS CONSTRUCTION FROM TEXTS

The overall process may be divided into three stages: (1) Construction of vocabulary: This involves normalization and selection of terms. It also includes phrase construction depending on the coordination level desired. (2) Similarity computations between terms: This step identifies the significant statistical associations between terms. (3) Organization of vocabulary: Here the selected vocabulary is organized, generally into a hierarchy, on the basis of the associations computed in step 2. The first program select.c implements procedures for stages 1 and 2. Program hierarchy.c implements one method for organizing the vocabulary in the third stage.

9.4.1 Construction of Vocabulary

The objective here is to identify the most informative terms (words and phrases) for the thesaurus vocabulary from document collections. The first step is to identify an appropriate document collection. The only loosely stated criteria is that the collection should be sizable and representative of the subject area. The next step is to determine the required specificity for the thesaurus. If high specificity is needed, then the emphasis will be on identifying precise phrases; otherwise, more general terms can be sought. Terms can be selected from titles, abstracts, or even the full text of the documents if available. This initial set of vocabulary terms is now ready for normalization. The simplest and most common normalization procedure is to eliminate very trivial words such as prepositions and conjunctions. For this, an appropriate stoplist of trivial words needs to be constructed. The article by Fox (1989-1990) on the construction of stoplists may be useful here. The next standard normalization procedure is to stem the vocabulary. Stemming reduces each word into its root form. For example, the terms "information," "informing," and "informed" could all be stemmed into the same root "inform." The chapter in this book on stemming algorithms may be consulted for this. The resulting pool of stems is now ready to be analyzed statistically with two objectives: first, to select the most interesting stems as discussed in the following section, and second, to create interesting phrases for a higher coordination level as discussed in the section on phrase construction.

Stem evaluation and selection

There are a number of methods for statistically evaluating the worth of a term. The ones we discuss here are: (1) selection of terms based on frequency of occurrence, (2) selection of terms based on Discrimination Value, (3) selection of terms based on the Poisson model. Program select.c, includes routines for all three methods, which are described briefly below.

Selection by Frequency of Occurence: This is one of the oldest methods and is based on Luhn's work, which has been extensively discussed in the literature; see for example, Salton and McGill (1983). The basic idea is that each term may be placed in one of three different frequency categories with respect to a collection of documents: high, medium, and low frequency. Terms in the mid-frequency range are the best for indexing and searching. Terms in the low-frequency range have minimal impact on retrieval, while high-frequency terms are too general and negatively impact search precision. Salton recommends creating term classes for the low-frequency terms; however, it is not evident how to do this automatically. High-frequency terms are generally coordinated into phrases to make them more specific (see the later section on phrase construction for this). Threshold frequencies are generally not fixed and therefore user specified. Program select.c includes a routine for this selection method. Threshold frequencies are specified by the user via the global variables LOW_THRESHOLD and HIGH_THRESHOLD.

Selection by Discrimination Value (DV): DV measures the degree to which a term is able to discriminate or distinguish between the documents of the collection as described by Salton and Yang (1973). The more discriminating a term, the higher its value as an index term. The overall procedure is to compute the average interdocument similarity in the collection, using some appropriate similarity function. Next, the term k being evaluated is removed from the indexing vocabulary and the same average similarity is recomputed. The discrimination value (DV) for the term is then computed as:

DV(k) = (Average similarity without k) - (Average similarity with k)

Good discriminators are those that decrease the average similarity by their presence, that is, those for which DV is positive. Poor discriminators have negative DV, while neutral discriminators have no effect on average similarity. Terms that are positive discriminators can be included in the vocabulary and the rest rejected.

Program select.c includes a routine called dv-all which computes DV for all terms with respect to a collection. In the program, average similarity with all terms intact is referred to as the baseline. The algorithm used is a straightforward one using the method of centroids. In this method the average interdocument similarity is computed by first calculating the average document vector or centroid vector for the collection. This is performed by a routine called centroid. Next, the similarity between every document and this centroid is calculated. This generates the total similarity in the collection, which can then be used to calculate the average similarity. For large collections of documents, a more efficient algorithm such as that based on the cover coefficient concept may be tried as suggested by Can and Ozkarahan (1987). However, this is not done here.

Selection by the Poisson Method: This is based on the work by Bookstein and Swanson (1974), Harter (1975), and Srinivasan (1990) on the family of Poisson models. The Poisson distribution is a discrete random distribution that can be used to model a variety of random phenomena including the number of typographical errors in a page of writing and the number of red cars on a highway per hour. In all the research that has been performed on the family of Poisson models, the one significant result is that trivial words have a single Poisson distribution, while the distribution of nontrivial words deviates significantly from that of a Poisson distribution. This result is used here to select nontrivial words as thesaurus terms.

The get-Poisson-dist routine in the select.c program prints out the distributions in a collection for all terms in the inverted file. Two distributions are produced for each term: the actual distribution and the one expected under the assumption of a single Poisson model. These distributions will have to be compared using the chi-square test to identify any significant differences. The reader is referred to Harter (1975) for information on these chi-square comparisons. Terms whose distributions deviate significantly are selected to be in the vocabulary. The rest may be discarded.

Phrase construction

This step may be used to build phrases if desired. As mentioned before, this decision is influenced by the coordination level selected. Also, phrase construction can be performed to decrease the frequency of high-frequency terms and thereby increase their value for retrieval. Two methods are described below. Program select.c includes routines for the first method. Given insufficient details for the second method, it is not implemented here. However, since it is an interesting approach, we include a sketchy algorithm.

Salton and McGill Procedure: This is the standard one proposed in Salton and McGill (1983, 133-34) and adapted by Rada in Forsyth and Rada (1986, 200). Their procedure is a statistical alternative to syntactic and/or semantic methods for identifying and constructing phrases. Basically, a couple of general criteria are used. First, the component words of a phrase should occur frequently in a common context, such as the same sentence. More stringent contextual criteria are possible, such as the words should also appear within some specified distance. The second general requirement is that the component words should represent broad concepts, and their frequency of occurrence should be sufficiently high. These criteria motivate their algorithm, which is described below.

1.     Compute pairwise co-occurrence for high-frequency words. (Any suitable contextual constraint such as the ones above may be applied in selecting pairs of terms.)

2.     If this co-occurrence is lower than a threshold, then do not consider the pair any further.

3.     For pairs that qualify, compute the cohesion value. Two formulas for computing cohesion are given below. Both ti and tj represent terms, and size-factor is related to the size of the thesaurus vocabulary.

COHESION (ti, tj) =

co-occurrence-frequency/sqrt(frequency(ti) * frequency(tj))

(Rada, page 200)

COHESION (ti, tj) =

size-factor * (co-occurrence-frequency/(total-frequency(ti) *

total-frequency(tj))) (Salton and McGill, 85)

4.     If cohesion is above a second threshold, retain the phrase as a valid vocabulary phrase.

We do not include a program for this complete algorithm. However, the routine cohesion in select.c is an implementation of the Rada formula.

Choueka Procedure: The second phrase construction method is based on the work by Choueka (1988). He proposes a rather interesting and novel approach for identifying collocational expressions by which he refers to phrases whose meaning cannot be derived in a simple way from that of the component words, for example, "artificial intelligence." The algorithm proposed is statistical and combinatorial and requires a large collection (at least a million items) of documents to be effective. The author has been quite successful in identifying meaningful phrases and is apparently extending the algorithm. Unfortunately, some critical details are missing from their paper. Therefore, we do not include an implementation of their approach. However, a sketchy procedure is included since the overall process looks rather interesting.

1.     Select the range of length allowed for each collocational expression. Example: two to six words.

2.     Build a list of all potential expressions from the collection with the prescribed length that have a minimum frequency (again, a preset value).

3.     Delete sequences that begin or end with a trivial word. The trivial words include prepositions, pronouns, articles, conjunctions, and so on.

4.     Delete expressions that contain high-frequency nontrivial words.

5.     Given an expression such as a b c d, evaluate any potential subexpressions such as a b c and b c d for relevance. Discard any that are not sufficiently relevant. (It is not clear from the paper how relevance is decided, but perhaps it is also based on frequency.)

6.     Try to merge smaller expressions into larger and more meaningful ones. For example, a b c d and b c d may merge to form a b c d. (Again, the exact criteria for allowing a merger are not given.)

The main difference between this procedure and the previous one is that this one considers phrases that have more than two words. It is of course possible to extend the previous procedure to include phrases with more than two words. However, computing cohesion is likely to be more challenging than simply applying the formula recursively over larger sized phrases. Choueka's procedure also allows phrases to be substituted by longer and more general phrases as in step 6. However, the criteria used for allowing this should be carefully formulated.

9.4.2 Similarity Computation

Once the appropriate thesaurus vocabulary has been identified, and phrases have been designed if necessary, the next step is to determine the statistical similarity between pairs of terms. There are a number of similarity measures available in the literature. An extensive study has been done on the comparison of different similarity measures (McGill et al. 1979). Select.c includes two similarity routines:

1.     Cosine: which computes the number of documents associated with both terms divided by the square root of the product of the number of documents associated with the first term and the number of documents associated with the second.

2.     Dice: which computes the number of documents associated with both terms divided by the sum of the number of documents associated with one term and the number associated with the other.

Either measure can be used to assess the similarity or association between terms in a document collection. The algorithms implemented in the program select.c can be made more accurate by including only those instances in each numerator wherein the two terms co-occur in the same sentence and within some specified distance (that is, in a unit smaller than the entire document), as suggested in Salton and McGill (1983) and Soergel (1974). This makes the criteria for similarity more stringent.

9.4.3 Vocabulary Organization

Once the statistical term similarities have been computed, the last step is to impose some structure on the vocabulary which usually means a hierarchical arrangement of term classes. For this, any appropriate clustering program can be used. A standard clustering algorithm generally accepts all pairwise similarity values corresponding to a collection of objects and uses these similarity values to partition the objects into clusters or classes such that objects within a cluster are more similar than objects in different clusters. Some clustering algorithms can also generate hierarchies. It should be noted there are major differences between available clustering algorithms, and the selection should be made after carefully studying their characteristics.

Given the chapter in this book on clustering, such programs are not included here. Instead, we include the implementation of an alternative simple procedure to organize the vocabulary which is described in Forsyth and Rada (1986, 200-01). This algorithm implemented in hierarchy.c is quite different from the standard clustering algorithms and is based on the following assumptions: (1) high-frequency words have broad meaning, while low-frequency words have narrow meaning; and (2) if the density functions of two terms, p and q (of varying frequencies) have the same shape, then the two words have similar meaning. As a consequence of these two assumptions, if p is the term with the higher frequency, then q becomes a child of p. These two assumptions motivate their entire procedure as outlined below. Besides illustrating the procedure, hierarchy.c also includes appropriate data structures for storing thesauri organized as hierarchies.

1.     Identify a set of frequency ranges.

2.     Group the vocabulary terms into different classes based on their frequencies and the ranges selected in step 1. There will be one term class for each frequency range.

3.     The highest frequency class is assigned level 0, the next, level 1 and so on.

4.     Parent-child links are determined between adjacent levels as follows. For each term t in level i, compute similarity between t and every term in level i-1. Term t becomes the child of the most similar term in level i-1 . If more than one term in level i-1 qualifies for this, then each becomes a parent of t. In other words, a term is allowed to have multiple parents.

5.     After all terms in level i have been linked to level i-1 terms, check level i-1 terms and identify those that have no children. Propagate such terms to level i by creating an identical "dummy" term as its child.

6.     Perform steps 4 and 5 for each level starting with level 1.

9.5 MERGING EXISTING THESAURI

The third program, merge.c, is designed to merge different hierarchies. This is useful when different thesauri (perhaps with different perspectives) are available for the same subject, or when a new subject is being synthesized from existing ones. The algorithm for the program has been adapted from Chapter 14 of Forsyth and Rada (1986) in which experiments in augmenting MeSH and SNOMED have been described. Two different merging algorithms have been implemented. The first, called simple-merge, links hierarchies wherever they have terms in common. The second, called complex-merge, adopts a more interesting criteria. It links terms from different hierarchies if they are similar enough. Here, similarity is computed as a function of the number of parent and child terms in common. Also, sufficiency is decided based on a preset user specified threshold.

9.6 BRIEF DESCRIPTION OF C PROGRAMS INCLUDED

Three programs are included at the end of this chapter. The first can be used to select terms and to construct phrases; the second generates (or reads) and stores hierarchies. The third program merges different hierarchies.

9.6.1 Program Select.c

This program contains a variety of routines for the various selection criteria used in designing the thesaurus vocabulary (see Appendix 9.A). It requires two input files: a direct file and an inverted file. Both contain the same information, but arranged differently. The direct file is a listing of document numbers corresponding to the database of interest. Each document number is associated with a set of term and term-weight pairs. A document may be associated with all its component terms or perhaps only a select few. The term-weight represents the strength of association between the document and the term. This term-weight may be assigned manually, or for example be some function of the term's frequency of occurrence in the document. The file is arranged such that the rows pertaining to the same document are grouped together. The inverted file is a listing of terms. Here each term is linked to its associated document numbers and the term-weights. The interpretation of term-weights should be the same in both input files. In fact, the two files should contain identical information. The inverted index is arranged such that rows corresponding to a term are grouped together. In both files, document numbers are represented by integers; the terms are character strings and the weights are decimal numbers. One or more spaces may be used to distinguish between the three. Figure 9.2 below shows a brief extract of both files.

2 math 2.0                mellitus 1 1.0

2 logic 1.0               logic 2 1.0

1 diabetes 2.0            diabetes 1 2.0

1 mellitus 1.0            math 2 2.0

3 math 1.0                math 3 1.0

   Direct file extract    Inverted file extract

Figure 9.2: Short extracts from both input files

Besides these two files, an output file must also be specified. The user will have to specify four global variables. The first two are MAXWORD: specifying the maximum number of characters in a term; and MAXWDF: specifying the expected maximum frequency of occurrence for a term within a document. The other two parameters are: LOW-THRESHOLD AND HIGH-THRESHOLD, which are used when partitioning the terms by frequency of occurrence in the collection into HIGH, MEDIUM, and LOW frequency classes.

9.6.2 Program Hierarchy.c

This program can perform two major and separate functions (see Appendix 9.B). First, if given the hierarchical relationships between a set of terms, it records these relationships in its internal inverted file structure. This can then be used for other purposes. Second, it is also capable of generating the hierarchical structure automatically using Rada's algorithm. For this, the input required is the inverted file, which has the same structure as in Figure 9.2. The second input file is a link file, which is a sequence of rows representing link information. A row consists of a parent term followed by any number of spaces and then a child term. This file is used if the link information is simply provided to the program. For this the user will have to set two parameters: MAXWORD, which specifies the maximum size for a term, and NUMBER-OF-LEVELS, which constrains the size of the generated hierarchy.

9.6.3 Program merge.c

This program contains routines to perform the two types of merging functions described (see Appendix 9.C). Four input files are required here: an inverted file and a link file for each thesaurus hierarchy. Their formats are as described before. Two parameters will have to be set: MAXWORD described before and THRESHOLD which specifies the minimum similarity for use in the complex merge routine.

9.7 CONCLUSION

This chapter began with an introduction to thesauri and a general description of thesaural features. Two major automatic thesaurus construction methods have been detailed. A few related issues pertinent to thesauri have not been considered here: evaluation of thesauri, maintenance of thesauri, and how to automate the usage of thesauri. The focus has been on the central issue, which is the construction of thesauri. However, these secondary issues will certainly be important in any realistic situation.

REFERENCES

AITCHISON, J., and A. GILCHRIST. 1972. Thesaurus Construction -- A Practical Manual. London: ASLIB.

BOOKSTEIN, A., and D. R. SWANSON. 1974. "Probabilistic Models for Automatic Indexing." J. American Society for Information Science, 25(5), 312-18.

CAN, F., and E. OZKARAHAN. 1985. Concepts of the Cover-Coefficient-Based Clustering Methodology. Paper presented at the Eighth International Conference on Research and Development in Information Retrieval. Association for Computing Machinery, 204-11.

CHOUEKA, Y. 1988. Looking for Needles in a Haystack OR Locating Interesting Collocational Expressions in Large Textual Databases. Paper presented at the Conference on User-Oriented Content-Based Text and Image Handling, MIT, Cambridge, Mass. 609-23.

FORSYTH, R., and R. RADA. 1986. Machine Learning -- Applications in Expert Systems and Information Retrieval. West Sussex, England: Ellis Horwood Series in Artificial Intelligence.

FOX, E. A. 1981. "Lexical Relations: Enhancing Effectiveness of Information Retrieval Systems." SIGIR Newsletter, 15(3).

FOX, E. A., J. T. NUTTER, T. AHLSWERE, M. EVENS, and J. MARKOWITZ. 1988. Building A Large Thesaurus for Information Retrieval. Paper presented at the Second Conference on Applied Natural Language Processing. Association for Computational Linguistics, 101-08.

FOX, C. FALL 1989/Winter 1990. "A Stop List for General Text." SIGIR Forum, 21(1-2), 19-35.

FROST, C. O. 1987. "Subject Searching in an Online Catalog." Information Technology and Libraries, 6, 60-63.

GUNTZER, U., G. JUTTNER, G. SEEGMULLER, and F. SARRE. 1988. Automatic Thesaurus Construction by Machine Learning from Retrieval Sessions. Paper presented at the Conference on User-Oriented Content-Based Text and Image Handling, MIT, Cambridge, Mass., 588-96.

HARTER, S. P. 1975. "A Probabilistic Approach to Automatic Keyword Indexing. Parts I and II." J. American Society for Information Science, 26, 197-206 and 280-89.

MCGILL, M. et al. 1979. An Evaluation of Factors Affecting Document Ranking by  Information Retrieval Systems. Project report. Syracuse, New York: Syracuse University School of Information Studies.

SALTON, G., and M. MCGILL. 1983. Introduction to Modern Information Retrieval. NewYork: McGraw-Hill.

SALTON, G., and C. S. YANG. 1973. "On the Specification of Term Values in Automatic Indexing." Journal of Documentation, 29(4), 351-72.

SOERGEL, D. 1974. "Automatic and Semi-Automatic Methods as an Aid in the Construction of Indexing Languages and Thesauri." Intern. Classif. 1(1), 34-39.

SRINIVASAN, P. 1990. "A Comparison of Two-Poisson, Inverse Document Frequency and Discrimination Value Models of Document Representation." Information Processing and Management, 26(2) 269-78.

WANG, Y-C., J. VANDENDORPE, and M. EVENS. 1985. "Relationship Thesauri in Information Retrieval." J. American Society of Information Science, 15-27.

APPENDIX

/*

PROGRAM NAME: hierarky.c

PURPOSE:  This program will generate a hierarchy in two ways.

1)  It can simply read the parent-child links from an input fifile and

store the links in the inverted file structure,

OR

2)  It can use the Rada algorithm which splits up words

into different frequency groups and then builds links

between them.

INPUT FILES REQUIRED:  (Depends on the option selected).

Option 1:  requires inverted file and link file.

Option 2:  requires inverted file.

1)  inverted file: sequences of

term  document number  weight.

(multiple entries for any term should be grouped together)

2)  links file: sequences of

parent term  child term

NOTES:  Filters such as stop lists and stemmers should be used

before running this program.

PARAMETERS TO BE SET BY USER:

1)  MAXWORD: identifies the maximum size of a term

2)  NUMBER_OF_LEVELS:  specifies the desired number of levels

in the thesaurus hierarchy to be generated, used by the

Rada algorithm

COMMAND LINE:  (INPUT & OUTPUT FILES ARE SPECIFIED INTERACTIVELY)

hierarky

***********************************************************************/

#include <stdio.h>

#include <string.h>

#include <math.h>

#define MAXWORD 20                /* maximum size of a term           */

#define NUMBER_OF_LEVELS 10       /* # of levels desired in the thesaurus  */

struct doclist {                  /* sequences of document # and weight pairs  */

int doc;                        /* document number                 */

float weight                    /* term weight in document         */

struct doclist *nextdoc;        /* ptr. to next doclist record     */

} doclistfile;

struct parentlist {

char term[MAXWORD];             /* parent term                      */

struct invert *parent;          /* ptr. to parent term in inverted file  */

struct parentlist *nextparent;  /* ptr. to next parentlist record  */

} parentfile;

struct childlist {

char term[MAXWORD];             /* child term                     */

struct invert *child;           /* ptr. to child term in inverted file  */

struct childlist *nextchild;  /* ptr. to next childlist record  */

} childfile;

struct invert {                 /* inverted file                  */

char term[MAXWORD];           /* term                           */

struct doclist *doc           /* sequences of document # and weigh  */

struct parentlist *parents;   /* ptr. to parent terms          */

struct childlist *children;   /* ptr. to child terms           */

int level;                    /* thesaurus level based on term frequency  */

struct invert *nextterm;      /* ptr. to next invert record    */

} invfile;

struct invert *startinv         /* ptr. to first record in inverted file  */

struct invert *lastinv          /* ptr. to last record in inverted file  */

struct doclist *lastdoc;        /* ptr. to last document in doclist  */

static char currentterm[MAXWORD];  /* tracks current term in inverted file  */

static int Number_of_docs;  /* total # of documents which is computed */

static struct invert *get_mem_invert ( );          /* these 4 functions will obtain  */

static struct doclist *get_mem_doclist ( );        /* memory for records. The type of  */

static struct parentlist *get_mem_parentlist ( );  /* is indicated by the name of  */

static struct childlist *get_mem_childlist ( );   /* the function  */

static FILE *input;              /* inverted file                     */

static FILE *input1;             /* link file                         */

static FILE *output;             /* holds any output                  */

static

float cohesion ( ),              /* compute cohesion between two terms */

total_wdf ( ),             /* compute total frequency of term in dbse.  */

get_freq_range ( );

static

void  read_invfile ( ),            /* read in the inverted file         */

read_links ( ),              /* read in the links file            */

add_link ( ),                /* called within read_links ( )      */

pr_invert ( ),               /* print the inverted file           */

add_invert ( ),              /* called within read_invfile ( )    */

write_levels ( ),            /* initialize the levels information */

generate_Rada_hierarchy ( ), /* generate the Rada hierarchy     */

get_term_data ( );           /* get basic information about terms */

struct invert *find_term ( );      /* searches for term in inverted file and  */

/* returns its address.            */

int main (argc)

int argc;

{

char ch, fname[128];

startinv = NULL; lastinv = NULL; lastdoc = NULL;

currentterm [ 0 ] = '\0'; Number_of_docs = 0;

if (argc > 1)

{

(void) printf ("There is an error in the command line\n");

(void) printf ("Correct usage is:\n");

(void) printf ("hierarchy\n");

exit (1);

}

(void)  printf ("\nMake a selection\n");

(void)  printf ("To simply read links from a link file enter 1\n");

(void)  printf ("To use Rada's algorithm to generate links enter 2\n");

(void)  printf ("To quit enter 3\n");

(void)  printf ("Enter selection: ");

ch=getchar ( );

switch (ch)

{

case '1':

(void)  printf ("\nEnter name of inverted file: ");

(void)  scanf ("%s", fname);

if ( (input=fopen (fname, "r") ) ==NULL) {

(void) printf ("cannot open file %s\n", fname);

exit (1);

}

(void)  printf ("Enter name of link file: ");

(void)  scanf ("%s", fname);

if ( (input1=fopen (fname,"r") ) == NULL)  {

(void) printf ("cannot open file %s\n",fname);

exit (1);

}

(void)  printf ("Enter name of output file: ");

(void)  scanf ("%s", fname);

if ( (output=fopen (fname,"w") ) ==NULL)  {

(void)  printf ("cannot open file %s\n", fname);

exit (1);

}

read_invfile ( );

(void)  fprintf (output,"\nINVERTED FILE\n\n");

pr_invert ( );

read_links ( );

(void)  fprintf (output,"\nINVERTED FILE WITH LINK INFORMATION\n\n");

pr_invert ( );

(void)  fclose (input); (void) fclose (input1); (void) fclose (output);

break;

case '2':

(void)  printf ("\nEnter name of inverted file: ");

(void)  scanf ("%s", fname);

if ( (input=fopen (fname,"r") ) ==NULL) {

(void) printf ("cannot open file %s\n", fname);

exit (1);

}

(void)  printf ("Enter name of output file: ");

(void)  scanf ("%s", fname);

if ( (output=fopen (fname,"w") ) ==NULL) {

(void)  printf ("cannot open file %s\n", fname);

exit (1);

}

read_invfile ( );

(void)  fprintf (output,"\nINVERTED FILE\n\n");

pr_invert ( );

generate_Rada_hierarchy ( );

(void)  fprintf (output,"\nINVERTED FILE AFTER GENERATING RADA HIERARCHY\n\n");

pr_invert ( );

(void)  fclose (input); (void) fclose (output);

break;

case '3':

exit (0);

}

return (0);

}

/***********************************************************************

read_invfile ( )

Returns:  void

Purpose:  Read in the inverted file entries from the disk file

**/

static void read_invfile ( )

{

int docid;            /* holds current document number            */

char temp [MAXWORD];  /* holds current term                       */

float weight;         /* holds current term weight                */

struct doclist *p;    /* structure to store doc#-weight pair      */

(void)  fscanf (input,"%s%d%f", temp, &docid, &weight);  /*   r ad next line  */

while  (strlen (temp)  > 0)

/* while its found a legitimate term     */

{

if (!strncmp (currentterm, temp, MAXWORD) ) {

/* if this term has previously been entered in inverted file then */

/* only need to attach a doclist record to the same entry         */

p = get_mem_doclist ( );  /* get memory for doclist record        */

p->doc = docid;           /* assign document number               */

p->weight = weight;       /* assign term weight                   */

p->nextdoc = NULL;

if (lastdoc) lastdoc-nextdoc = p;  /* connect p to the doclist chain

for this term if it already exists                */

lastdoc = p;    /*  set this global variable                      */

}

else add_invert (docid, temp, weight);

/* else term is a brand new term & need to make a new inverted file entry  */

temp [0] = ' \0 ';

(void)  fscanf (input,"%s%d%f", temp, &docid, &weight);  /* read next line  */

}

}

/***********************************************************************

add_invert(docid,temp,weight)

Returns:  void

Purpose:  Start a new entry in the inverted file. It is called in the

read_invfile function when a new term is read from the input

file.

**/

static void add_invert (docid, temp, weight)

int docid;                               /* in: document number     */

char temp [MAXWORD];                     /* in: new index term      */

float weight;                            /* in: index term weight   */

{

struct invert *p;                        /* p will get attached to inverted file  */

p = get_mem_invert ( );                  /* get memory for p        */

(void) strncpy (p-term, temp, MAXWORD);  /* copy over the term      */

p->parents = NULL;                       /* to begin this term has no parent terms   */

p->children = NULL;                      /* also no child terms     */

p->doc = get_mem-doclist ( );            /* start a doclist structure */

p->doc->doc = docid;                     /* assign document number  */

p->doc->weight = weight;                 /* assign term weight      */

p->doc->nextdoc = NULL;

p->nextterm = NULL;

if (startinv = NULL) startinv = p;

/* if this is the first entry in inverted file, then update global variable  */

if (lastinv) lastinv-nextterm = p;      /* update ptr. to last inverted file record  */

lastinv = p;

lastdoc = p->doc;                       /* update ptr. to last document  */

(void) strncpy(current term, temp, MAXWORD);  /* update global variable current term  */

}                                     /* to the new term just added   */

/**********************************************************************

read_links( )

Returns:  void

Purpose:  Read-parent child link information from a file and record

links in the inverted record structure

**/

static void read_links( )

{

char parent[MAXWORD],    /* tracks parent term          */

child[MAXWORD];     /* tracks child term           */

(void) fscanf(input1,"%s%s",parent,child);  /* read input line    */

while (strlen(parent) > 0)

/* while a legitimate parent has been found   */

{

add_link(parent,child);  /* this function will add the appropriate links in  */

/* the inverted file                    */

child[0] = '\0'; parent[0] = '\0';  /*  now throw out the old parent & child  */

(void) fscanf(input1,"%s%s",parent,child);   /* read next input line  */

}

}

/***********************************************************************

add_link(parent,child)

Returns:  void.

Purpose:  Used within read_links. Basically, for each parent-child

link specified, it adds the appropriate link information into

the inverted file.

Notes:    If a term in the link file is not in the inverted file then

the program will give a suitable message and exit.

**/

static void add_link(parent,child)

char parent[MAXWORD],                 /* in: holds the parent term    */

child[MAXWORD];                  /* in: holds the child term     */

{

struct invert *p,               /* holds add. of parent term in inv. file  */

*q;  /* holds add. of child term in inv. file           */

struct parentlist *new_parent;  /* structure used to store parent info.  */

struct childlist *new_child;    /* structure used to store child info.  */

p = find_term(parent);          /* find address of parent term       */

if  (!p) { printf("\nPlease check the input files.\n");

printf("\nParent term %s is not in the inverted file\n",parent);

exit(0);}

q = find_term(child);           /* find address of child term        */

if  (!q) { printf("\nPlease check the input files. Output may be incorrect.\n");

printf("\nChild term %s is not in the inverted file\n",child);

exit(0);}

/* first add parent links for given child */

new_parent = get_mem_parentlist( );  /* get memory for parentlist record */

(void) strncpy(new_parent->term, parent, MAXWORD);  /* copy over parent term */

new_parent->parent = p;  /* store address of parent term in inverted file */

if (q->parents == NULL) {  /* i. e. no parents listed for given child yet */

q->parents = new_parent;             /* first parent link made   */

new_parent->nextparent = NULL;

}

else {            /* at least 1 parent already listed for given child *

new_parent->nextparent = q->parents; /* attach newparent to front of list */

q->parents = new_parent;

}

/* next add child links for given parent */

new_child = get_mem_childlist( );  /* get memory for childlist record */

(void) strncpy(new_child->term,child,MAXWORD);  /* copy over child term */

new_child->child = q;  /* store address of child term in inverted file*/

if (p->children == NULL) {  /* no children listed for given parent yet*/

p->children = new_child;  /* first child link made                */

new_child->nextchild = NULL;

}

else {       /* at least 1 child already listed for given parent      */

new_child->nextchild = p->children;  /* attach newchild to front of list */

p->children = new_child;

}

}

/***********************************************************************

pr_invert( )

Returns:  void

Purpose:  Print the inverted file. It prints each term, its

associated document numbers, term-weights and parent child terms.

**/

static void pr_invert( )

{

struct invert *inv_addr;         /* tracks address of current inv. file record */

struct doclist *doc_addr;        /* tracks address of current doclist record */

struct parentlist *parent_addr;  /* tracks address of current parentlist record */

struct childlist *child_addr;    /* tracks address of current childlist record */

inv_addr = startinv;  /* begin at top of inverted file   */

while (inv_addr) {    /* while a legitimate term. . . .  */

(void) fprintf(output, "TERM: %s\nPARENT TERMS: ", inv_addr->term);

parent_addr = inv_addr->parents;  /* find addr. of first parent  */

while (parent_addr) {             /* printing all parents */

(void) fprintf(output, "%s ", parent_addr->term);

parent_addr = parent_addr->nextparent;  /*loop through remaining parents */

}

(void) fprintf(output,"\nCHILD TERMS: ");

child_addr = inv_addr->children;  /* find addr. of first child */

while (child_addr) {              /* printing all children     */

(void) fprintf(output,"%s ",child_addr->term);

child_addr = child_addr->nextchild;  /* loop through remaining children  */

}

(void) fprintf(output,"\n\n");

(void) fprintf(output,"  DOCUMENT NUMBER  TERM

doc_addr = inv_addr->doc;  /* find addr. of first associated doc. */

while (doc_addr) {            /* print all docs. and term weights */

(void) fprintf(output,"      %-30d ",doc_addr->doc);

(void) fprintf(output,"%-10.5f\n", doc_addr->weight);

doc_addr = doc_addr->nextdoc; /* loop through remaining documents */

}

(void) fprintf(output,"\n");

inv_addr = inv_addr->nextterm; /* go to next inverted file entry */

}

}

/***********************************************************************

total_wdf(term)

Returns:  float

Purpose:  Compute total within document frequency for specified term

in the database

**/

static float total_wdf(term)

char term[MAXWORD];  /* in: term for which total_wdf is required */

{

struct invert *term_addr;    /* add. of above term in inverted file */

struct doclist *doc_ad;      /* tracks add. of associated doclist record */

float totalwdf;              /* tracks total wdf */

totalwdf = 0.0;

term_addr = find_term(term); /* obtain address of the term in inv. file */

if (term_addr) {             /* if term was found  */

doc_ad = term_addr->doc;   /* find address of associated doclist record */

while (doc_ad) {

totalwdf = totalwdf + doc_ad-weight;  /* loop through doclist reeords to */

doc_ad = doc_ad-nextdoc;              /* compute the total weight */

}}

else (void) fprintf(output, "Term %s is not in the inverted file. Could lead to

problems\n",term);

return(totalwdf);

}

/***********************************************************************

get_freq_range(minimum,maximum)

Returns:  float

Purpose:  Compute the difference between the maximum total term frequency

and the minimum total term frequency observed in the inverted file.

**/

static float get_freq_range(minimum, maximum)

float *minimum,            /* out: returns minimum totalwdf       */

*maximum;            /* out: returns maximum totalwdf       */

{

struct invert *inv_addr;

float freq, max, min;

inv_addr = startinv;     /* begin at top of inverted file         */

/* initialize min and max to equal frequency of 1st term in file  */

if  (inv_addr)  {

freq = total_wdf(inv_addr->term);

min = freq; max = freq;

inv_addr = inv_addr->nextterm;  /* go to next term in inv. file */

}

while (inv_addr {

/* while a legitimate term compare with max and min.              */

freq = total_wdf(inv_addr->term);

if  (freq < max) max = freq;

if  (freq > min) min = freq;

inv_addr = inv_addr->nextterm;  /* go to next term in inv. file */

}

*minimum = min;  *maximum = max;

return(max - min);  /*  returning the difference  */

}

/***********************************************************************

write_levels( )

Returns:  void

Purpose:  Write the level numbers for each term into the inverted file

depending on the total wdf of the term in the database and

the user selected parameter NUMBER_OF_LEVELS.

The level numbers are marked 0, 1, 2, ... etc. Level 0

refers to the highest frequency class, level 1 the next

frequency class etc.

**/

static void write_levels( )

{      i,                   /* counter through the different levels   */

number;              /* holds NUMBER_OF_LEVELS                */

float  freq,                /* holds frequency of term in database    */

range,               /* holds diff. between highest & lowest freqs. */

current_low,         /* tracks lower frequency of current level */

current_high;        /* tracks higher frequency of current level */

float  high,                /* highest term frequency in database     */

low;                 /* lowest term frequency in database     */

struct invert *inv_addr;    /* tracks current inverted file record    */

/* range holds the difference between highest & lowest totalwdf in dbse. */

range = get_freq_range(&low,&high);

number = NUMBER_OF_LEVELS;  /* user specified global parameter        */

inv_addr = startinv;        /* start with the first term in inverted file */

while(inv_addr)  {          /* while a legitimate term was found     */

freq = total_wdf(inv_addr->term);

current_low = low;

for (i=(number-1); i>=0; i- -) {

if (i == 0) current_high = high; else current_high = current_low + (range/number);

/* if the term's frequency is within this narrow range, then level = i */

if ((freq >= current_low) && (freq <= current_high)) {

inv_addr->level = i;

break;

}

current_low = current_high;  /* loop through the frequency levels */

}  /* ending for loop */

inv_addr = inv_addr->nextterm;     /* loop through other inv. file terms */

}

}

/***********************************************************************

generate_Rada_hierarchy ( )

Returns:  void

Purpose:  Create the levelptrs data structure and generate the hierarchy

according to Rada's algorithm

**/

static void generate_Rada_hierarchy ( )

{

struct termlist  {

struct invert *term;        /* pointer to term in inverted file    */

int mark;                   /* equals 1 if term is propagated else 0 */

struct termlist *nextterm;  /* pointer to next termlist record     */

} termlistfile, *p, *q, *r;

/* levelptrs is an array of pointers. Each slot points to the start of

the chain of termlist records for that level  */

struct termlist *levelptrs[NUMBER_OF_LEVELS];

int i;

struct invert *inv_addr;      /* tracks current term in inverted file */

float coh, max_cohesion;

write_levels ( );  /* this routine computes and writes the level number for each */

/* term in the inverted file                       *

for (i=0;i < NUMBER_OF_LEVELS;i++) levelptrs[i] = NULL;  /* intializing the array */

/* now create the termlist chain for each level  */

inv_addr = startinv;             /* start with first term in inverted file */

while (inv_addr) {               /* while there is a term there       */

p = (struct termlist *)malloc(sizeof(termlistfile));  /* get memory for termlist */

if  (!p)  {

(void) fprintf(output,"\nout of memory\n");

exit(1);

}

p-term = inv_addr;            /* assign the address of term in inverted file*/

p-mark = 0;                   /* initially term not linked */

/* Note: this term has been assigned to a frequency level already. Now, if this */

/* is the first term read for this level then set the appropriate levelptrs entry*/

/* to point to this term  */

if  (levelptrs[inv_addr  level] == NULL)  {

levelptrs[inv_addr  level] = p;

p->nextterm = NULL;

}

else  {   /* now this is not the first term encountered for this level, so simply */

/* attach it to the front of the chain                    */

p->nextterm = levelptrs[inv_addr->level];

levelptrs[inv_addr->level] = p;

}

inv_addr = inv_addr->nextterm;                 /* process next inverted file term */

}  /* end while */

/* start with each level and compute max-cohesion with previous level */

for (i=1;i < NUMBER_OF_LEVELS;i++) {

p = levelptrs[i];

while (p)  {

max_cohesion = 0.0;

q = levelptrs[i-1];                /* q set to the previous level's first term */

while (q)  {          /* as long as there are terms in this previous level . . . */

coh = cohesion(p->term->term,q->term->term);

if (coh > max_cohesion) max_cohesion = coh;

q = q->nextterm;

}

/* max_cohesion for terms in p has been computed */

/*  adding parent-child links and marking parents as propagated */

q = levelptrs[i-1];

while (q && max_cohesion 0.0)  {

coh = cohesion(p->term->term,q->term->term);

if  (coh == max_cohesion)  {       /* this ensures multiple links possible */

add_link(q->term->term,p->term->term);  /* routine adds the actual link */

q-mark = 1;                /* to show that parent term has been linked */

}

q = q  nextterm;

}  /* end while(q) */

p = p  nextterm;                                    /* go to next term in level i */

max_cohesion = 0.0;

}  /* end while(p) */

/* checking all terms in level[i-1] to make sure they have propagated */

q = levelptrs[i-1];

while (q)  {

if  (q  mark == 0) {                     /* i.e. term has no child in next level */

q  mark = 1;

r = (struct termlist *)malloc(sizeof(termlistfile));

if  (!r)  {  (void) fprintf(output,"\nout of memory\n");

exit(2);

}

r->term = q->term; r  mark = 0;  /* making a copy of term as its dummy child */

r->nextterm = levelptrs[i];  /* inserting r at beginning of level i chain */

levelptrs[i] = r;

}

q = q  nextterm;

}

}  /* for */

}

/***********************************************************************

cohesion(term1, term2)

Returns:  void

Purpose:  Compute the cohesion between two terms

**/

static float cohesion(term1, term2)

char   term1[MAXWORD],   /* in: the two terms which are being        */

term2[MAXWORD];   /* in: compared to determine cohesion       */

{

float  11,               /* holds # of documents associated with term 1 */

12,               /* holds # of documents associated with term 2 */

common;           /* holds # of documents in common           */

get_term_data(term1, term2, &/11,2, &common);

return(common/(sqrt(/11 * 12)));

}

/***********************************************************************

get_term_data(term1, term2, l1, l2, common);

Returns:  void

Purpose:  Given two terms, it determines the number of documents in

each and in common between them.

**/

static void get_term_data(term1, term2, l1, l2, common)

char   term1[MAXWORD],        /* in: term 1                           */

term2[MAXWORD];        /* in: term 2                          */

float  *11,                   /* out: # of documents associated with term 1 */

*12,                   /* out: # of documents associated with term 2 */

*common;               /* out: # of documents associated with both */

{

struct  invert *p, *q;        /* holds addresses of both terms in inv. file */

struct doclist *doc_ad1, *doc_ad2;  /*tracks addresses of doclists records */

int     count1,               /* # of documents associated with term 1 */

count2,                /* # of documents associated with term 2 */

com;                   /* # of documents in common            */

p = find_term(term1);  q = find_term(term2);     /* find addresses of terms */

doc_ad1 = p-doc;                      /* start with doclist record for term1 */

count1 = 0;  count2  = 0;  com = 0;   /* initialize                   */

/* first get length for document 1 and number of common terms */

while  (doc_ad1)  {

count1 = count1 +1;

doc_ad2 = q  doc;

/* for each doc. of term1 loop through all docs. of term2  */

while  (doc_ad2)  {

if  (doc_ad1  doc = doc_ad2  doc)  {    /* if they are the same doc. # */

com = com +1;

break;

}

doc_ad2 = doc_ad2-nextdoc;

}

doc_ad1 = doc_ad1-nextdoc;

}

/* now get length of document 2 */

doc_ad2 = q  doc;

while (doc_ad2)  {

count2 = count2 + 1;

doc_ad2 = doc_ad2  nextdoc;

}

*l1 = count1;  *l2 = count2;  *common = com;

}

/***********************************************************************

*find_term(term)

Returns:  address of a struct invert record

Purpose:  Search for a specified term in the inverted file and

return address of the corresponding inverted file record.

**/

static struct invert *find_term(term)

char term[MAXWORD];         /*  in: term to be located in inverted file */

{

struct invert *inv_addr;    /*  tracks addr. of current rec. in inv, file */

inv_addr = startinv;                         /*  begin at top on inv. file */

while(inv_addr) {

if  (!strcmp(term, inv_addr-term)) return(inv_addr);

inv_addr = inv_addr-nextterm;

}

(void)  fprintf(output,"Term %s not found\n",term);

return (NULL);

}

/***********************************************************************

*get_mem_invert( )

Returns:  address of a struct invert record

Purpose:  dynamically obtain enough memory to store 1 invert record.

**/

static struct invert *get_mem_invert( )

{

struct_invert *record;

record = (struct invert *)malloc(sizeof(invfile));

if  (!record)  {

(void) fprintf(output,"\nout of memory\n");

return (NULL);

}

return(record);

}

/***********************************************************************

*get_mem_doclist( )

Returns:  address of a struct doclist record

Purpose:  dynamically obtain enough memory to store 1 doclist record.

**/

static struct doclist *get_mem_doclist( )

{

struct doclist *record;

record = (struct doclist *)malloc(sizeof(doclistfile));

if  (!record)  {

(void) fprintf(output,"\nout of memory\n");

return (NULL);

}

return(record);

}

/***********************************************************************

*get_mem_parentlist()

Returns:  address of a struct parentlist record

Purpose:  dynamically obtain enough memory to store i parentlist recorded

**/

static struct parentlist *get_mem_parentlist ( )

{

struct parentlist *record;

record = (struct parentlist *)malloc(sizeof (parentfile));

if  (!record)  {

(void) fprintf (output,"\nout of memory\n");

return (NULL);

}

return(record);

}

/***********************************************************************

*get_mem_childlist()

Returns:  address of a struct childlist record

Purpose:  dynamically obtain enough memory to store 1 childlist record

**/

static struct childlist *get_mem_childlist( )

{

struct childlist *record;

record = (struct childlist *)malloc(sizeof (childfile));

if  (!record)  {

(void) fprintf(output,"\nout of memory\n");

return(NULL);

}

return(record);

}

/*

PROGRAM NAME:  select. c

PURPOSE:  1)  Compute Discrimination Value of terms,

2)  Compute Poisson Distributions for terms,

3)  Parition terms by their total within document frequencies,

4)  Compute cohesion between pairs of terms,

5)  Compute Dice's coefficient of similarity between two terms.

INPUT FILES REQUIRED:

1)  a direct file, sequences of:

document#  term  weight

(multiple entries for any document should be grouped together )

2)  an inverted file, sequences of

term  document#  weight

(multiple entries for any term should be grouped together)

NOTES:  Filters such as stop lists and stemmers should be used before

before running this program.

PARAMETERS TO BE SET BY USER:

1)  MAXWORD - maximum size of a term

2)  MAXWDF  - maximum value expected for the within document

frequency for a term in the collecgtion.

3)  LOW_THRESHOLD - threshold for LOW and MID frequency ranges

4)  HIGH_THRESHOLD - threshold for MID and HIGH frequency ranges

COMMAND LINE:

select direct_file inverted_file output_file

/***********************************************************************

#include <stdio.h>

#include <string.h>

#include <math.h>

#define MAXWORD 20           /* maximum size of a term               */

#define MAXWDF 30            /* maximum WDF for a word in a database */

#define LOW_THRESHOLD 2.0

#define HIGH_THRESHOLD 4.0

struct termlist  {           /* sequences of term and weight pairs   */

char term[MAXWORD];        /* term                                 */

float weight;              /* term weight in document              */

struct termlist *nextterm; /* ptr. to next termlist record         */

}  termlistfile;

struct doclist  {            /* sequences of document # and weight pairs */

int doc;                   /* document number                      */

float weight;              /* term weight in document              */

struct doclist *nextdoc;   /* ptr. to next doclist record          */

} doclistfile;

struct direct  {             /* direct file: document to list of terms*/

int docnum;                /* document #                           */

struct termlist *terms;    /* sequences of term and weight pairs   */

struct direct *next;       /* ptr. to next direct record           */

} directfile;

struct invert  {             /* inverted file: term to list of documnts */

char term[MAXWORD];        /* term                                */

struct doclist *doc;       /* sequences of document # and weight pairs */

struct invert *next;       /* ptr. to next invert record          */

} invfile;

static struct invert *startinv;    /* ptr. to first record in inverted file */

static struct invert *lastinv;     /* ptr. to last record in inverted file */

static struct doclist *lastdoc;    /* ptr. to last document in doclist */

static struct direct *startdir;    /* ptr. to first record in direct file */

static struct direct *lastdir;     /* ptr. to last record in direct file */

static struct termlist *lastterm;  /* ptr. to last term in termlist   */

static struct direct *start_centroid; /* ptr. to centroid record      */

static FILE *input;                /* direct file                     */

static FILE *input1;               /* inverted file                   */

static FILE *output;               /* file to hold all output         */

static int currentdoc;             /* tracks current document in direct file */

static char currenterm[MAXWORD];   /* tracks current term in inverted file */

static int Number_of_docs;         /* total # of document which is computed */

static

float av_doc_similarity( ),    /* compute average doc. similarity in dbse. */

factorial( ),            /* compute factorial of a number   */

cosine( ),               /* compute cosine between two terms*/

dice( ),                 /* compute dice beteen two terms   */

total_wdf( ),            /* compute total frequency of term in dbse. */

cohesion( );             /* compute cohesion between two terms */

static

void  initialize( ),           /* initialize files and global variables */

read_directfile( ),      /* read in the direct file          */

add_direct( ),           /* called within read_directfile( ) */

pr_direct( ),            /* print the direct file            */

read_invfile( ),         /* read in the inverted file        */

add_invert( ),           /* called within read_invfile( )    */

pr_invert( ),            /* print the inverted file          */

centroid( ),             /* compute the document centroid for dbse. */

pr_centroid( ),          /* print the document centroid     */

get_Poisson_dist( ),     /* compute Poisson distributions for terms */

Partition_terms( ),      /* partition terms by frequency    */

dv_all( ),               /* compute discrimination value of terms */

get_doc_data( ),         /* get basic info. about documents */

get_term_data( );        /* get basic info. about terms     */

static struct direct *get_mem_direct( );     /* these 4 get_mem functions are */

static struct invert *get_mem_invert( );     /* used to obtain memory for a */

static struct doclist *get_mem_doclist( );   /* record. The record type is */

static struct termlist *get_mem termlist( ); /* obvious from the name */

struct invert *find_term( );   /* searches for term in inverted file  */

/* and returns its address            */

int main(argc,argv)

int argc;

char *argv[ ];

{

char ch, word1[MAXWORD], word2[MAXWORD];

if (argc!=4)

{

(void) printf ("There is an error in the command line\n");

(void) printf("Correct usage is\n");

(void) printf ("select direct_file inverted_file output_file\n");

exit(1);

}

initialize(argv);

(void) fprintf(output,"\nREADING IN DIRECT FILE\n");

read_directfile( );

(void) fprintf(output,"\nPRINTING DIRECT FILE\n\n");

pr_direct( );

(void) fprintf(output,"\nNUMBER OF DOCUMENTS IS: %d\n\n",Number_of_docs);

(void) fprintf(output,"\nREADING IN INVERTED FILE\n");

read_invfile( );

(void) fprintf(output,"\nPRINTING INVERTED FILE\n");

pr_invert( );

(void) printf("\nPlease make a selection\n\n");

(void) printf ("To compute DV for all terms enter 1\n");

(void) printf ("TO compute Poisson distributions enter 2\n");

(void) printf("To partition terms by frequency enter 3\n");

(void) printf ("To compute cohesion between two terms (for phrase construction) enter 4\n");

(void) printf("To compute Dice's coefficient between two terms enter 5\n");

(void) printf ("To quit enter 6\n\n");

(void) printf("Enter your choice:  ");

ch = getchar( );

switch(ch)

{

case '1':

centroid( );

(void) fprintf(output, "\nCENTROID\n\n");

pr_centroid( );

(void) fprintf(output,"\nDISCRIMINATION VALUES FOR ALL TERMS\n\n");

dv_all( );

break;

case '2':

(void) fprintf(output,"\nACTUAL AND POISSON DISTRIBUTIONS OF WITHIN DOCUMENT

FREQUENCIES FOR ALL TERMS\n\n");

(void) fprintf(output,"WDF = Within Document Frequency & #docs = Number of

documents\n\n");

get_Poisson_dist( );

break;

case '3':

(void) printf("Make sure that the threshold parameters are set correctly in

the programm\n");

(void) fprintf(output,"\nPARTITIONING THE TERMS INTO LOW, MEDIUM, HIGH

FREQUENCY CLASSES\n\n");

Partition_terms( );

break;

case '4':

(void) printf("enter first word: ");

(void) scanf("%s", word1);

if (find_term(word1) == NULL)  {

printf("sorry, %s is not in the inverted file\n", word1);

break;

}

(void) printf("enter second word:  ");

(void) scanf("%s", word2);

if (find_term(word2) == NULL)  {

printf("sorry, %s is not in the inverted file\n", word2);

break;

}

(void) fprintf(output, "Cohesion between %s and %s is %f\n", word1, word2,

cohesion(word1, word2));

break;

case '5':

(void) printf("enter first word:  ");

(void) scanf ("%s", word1);

if (find_term(word1) == NULL)  {

printf("sorry, %s is not in the inverted file\n",word1);

break;

}

(void) printf ("enter second word: ");

(void) scanf("%s", word2);

if (find_term(word2) == NULL)  {

printf("sorry, %s is not in the inverted file\n", word2);

break;

}

(void) fprintf(output,"Dice's coefficient between %s and %s is %f\n", word1,

word2,dice(word1,word2));

break;

case '6':

exit(0);

default:

(void) printf("no selection made\n");

}

(void) fclose(input1); (void) fclose(input); (void) fclose(output);

return(0);

}

/***********************************************************************

initialize(argv)

Returns:  void

Purpose:  Open all required files and initialize global variables

**/

static void initialize(argv)

char *argv[ ];   /* in: holds the three parameters input at the command line */

{

if  (( input = fopen(argv[1],"r")) = NULL )  {

(void) printf("couldn't open file %s\n",argv[1]);

exit(1); /* input direct fil */}

if  (( input1 = fopen(argv[2],"r")) == NULL )  {

(void) printf("couldn't open file %s\n",argv[2]);

exit(1); /* input inverted file */}

if  (( output = fopen(argv[3],"w")) == NULL)  {

(void) printf("couldn't open file %s for output\n",argv[3]);

exit(1); /* output file */

}

/* set initial values of global variables */

startinv = NULL;  lastinv = NULL;  lastdoc = NULL;

startdir = NULL;  lastdir = NULL;   lastterm = NULL;

start_centroid = NULL;

currentdoc = 0;   currentterm[0] = '\0';   Number_of_docs = 0;

}

/***********************************************************************

/*

read_directfile( )

Returns:  void

Purpose:  Read in the direct file entries from the 1st input file

**/

static void read_directfile( )

{

int docid;            /* holds the current document number        */

char temp[MAXWORD];   /* holds the current term                   */

float weight;         /* holds the current term weight            */

struct termlist *p;   /* structure to store the term-weight pair  */

(void) fscanf(input,"%d%s%f",&docid,temp,&weight);  /* read the next line */

while (docid > 0)

/* while its found a legitimate document number             */

{

if (docid == currentdoc) {

/* if this document number has previously been entered in direct file */

/* then only need to attach a termlist record to the same entry */

p = get_mem_termlist( );   /* get memory for a termlist record */

(void) strncpy(p-term,temp,MAXWORD);  /* copy the new word over */

p->weight = weight;             /* assign the new weight over */

p->nextterm = NULL;

if (lastterm) lastterm->nextterm = p;  /* connect p to the termlist

chain for this document */

lastterm = p; /* set this global variable */

}

else { /* else docid represents a new document */

Number_of_docs = Number_of_docs +1;  /* increment global variable */

add_direct(docid,temp,weight);  /* starts a brand new entry in */

/* the direct file               */

}

docid = 0;

(void) fscanf(input,"%d%s%f",&docid,temp,&weight);

}

}

/***********************************************************************

add_direct(docid,temp,weight)

Returns:  void

Purpose:  Start a new entry in the direct file. It is called in

the read_directfile function when a new document number is

read from the input file.

**/

static void add_direct(docid,temp,weight)

int docid;             /* in: new document number                */

char temp[MAXWORD];    /* in: index term                         */

float weight;          /* in: index term weight                  */

{

struct direct *p;      /* structure p will be attached to direct file */

p = get_mem_direct( );  /* get memory for p  */

p->docnum = docid;     /* assign the document number  */

p->terms = get_mem_termlist( );  /* get memory for termlist structure */

(void) strncpy(p->terms->term,temp,MAXWORD);  /* assign index term to it */

p->terms->weight = weight;         /* assign term weight to it  */

p->terms->nextterm = NULL;         /* current end of termlist  */

p->next = NULL;                    /* current end of direct file  */

if (startdir == NULL) startdir = p  /* if this is the very first document

then global variable pointing to start of direct file should be updated */

if (lastdir) lastdir->next = p;  /* update pointer to last direct file rec. */

lastdir = p;

lastterm = p->terms;  /* update pointer to last term */

currentdoc = docid;   /* update the global variable currentdoc to the */

/* document number just added                   */

}

/***********************************************************************

pr_direct()

Returns:  void

Purpose:  Print the direct file. It prints sequences of document# term weight.

**/

static void pr_direct ( )

{

struct direct *dir_addr;     /* tracks address of current direct file record */

struct termlist *term_addr;  /* tracks address of current termlist record */

dir_addr = startdir;   /* start with beginning of direct file     */

while (dir_addr {  /* check for legitimate direct file record     */

(void) fprintf(output,"DOCUMENT NUMBER: %d \n",dir_addr-docnum);

(void) fprintf(output,"              TERM            TERM WEIGHT\n");

term_addr = dir_addr-terms; /* get addr. of first term         */

while (term_addr) {  /* loop through all the terms             */

(void) fprintf(output,"                 %-30s ",term_addr-term);

(void) fprintf(output,"%-10.3f\n",term_addr-weight);

term_addr = term_addr-nextterm;   /* go to next term for the doc. */

}

(void) fprintf(output,"\n");

dir_addr = dir_addr-next; /* go to next direct file record  */

}

}

/***********************************************************************

read_invfile( )

Returns:  void

Purpose:  Read in the inverted file entries from 2nd input file

**/

static void read_invfile( )

{

int docid;           /* holds currenct document number            */

char temp[MAXWORD];  /* holds current term                        */

float weight;        /* holds current term weight                 */

struct doclist *p;   /* structure to store doc#-weight pair       */

(void) fscanf(input1,"%s%d%f",temp,&docid,&weight); /* read next line */

while (strlen(temp) > 0)

/* while its found a legitimate term  */

{

if (!strncmp(currentterm,temp,MAXWORD))  {

/* if this term has previously been entered in inverted file     */

/* then only need to attach a doclist record to same term entry  */

p = get_mem_doclist( );  /* get memory for doclist record         */

p->doc = docid;      /* assign document number */

p->weight = weight;  /* assign weight          */

p->nextdoc = NULL;

if (lastdoc) lastdoc->nextdoc = p;  /* connect p to the doclist

chain for this ter  */

lastdoc = p; /* set this global variable   */

}

else add_invert(docid,temp,weight);

/* else term is a brand new term & need to make a new inverted file entry */

temp[0] = '\0';

(void) fscanf(input1,"%s%d%f",temp,&docid,&weight); /* read next line */

}

}

/***********************************************************************

add_invert(docid,temp,weight);

Returns:  void

Purpose:  Start a new entry in the inverted file. It is called in the

read_invfile function when a new term is read from the

input file

**/

static void add_invert(docid,temp,weight)

int docid;             /* in: document number       */

char temp[MAXWORD];    /* in: new index term        */

float weight;          /* in: index term weight     */

{

struct invert *p;      /* structure p will be attached to inverted file */

p = get_mem_invert( );  /* get memory for p */

(void) strncpy(p->term,temp,MAXWORD);  /* copy over the term */

p->doc = get_mem_doclist( ); /* start a doclist structure */

p->doc->doc = docid;        /* assign document number */

p->doc->weight = weight;    /* assign term weight     */

p->doc->nextdoc = NULL;

p->next = NULL;

if (startinv == NULL) startinv = p;

/* if this is the first entry in inverted file, then update global var. */

if (lastinv) lastinv->next = p; /* update ptr. to last inverted file record */

lastinv = p;

lastdoc = p->doc; /* update ptr. to last document */

(void) strncpy(currentterm,temp,MAXWORD); /* update global var. currentterm to the */

/* new term just entered */

}

/***********************************************************************

pr_invert( )

Returns:  void

Purpose:  Print the inverted file. It prints sequences of

term document# weight.

**/

static void pr_invert( )

{

struct invert *inv_addr;  /* tracks address of current inverted file record */

struct doclist *doc_addr; /* tracks address of current doclist record */

inv_addr = startinv;  /* start with beginning of inverted file */

while (inv_addr) {    /* check for legitimate inverted file record */

(void) fprintf(output,"TERM: %s\n",inv_addr-term);

(void) fprintf(output,"         DOCUMENT NUMBER     TERM WEIGHT\n");

doc_addr = inv_addr-doc;  /* get addr. of first document */

while (doc_addr) { /*loop through all the associated doc.#s and weights*/

(void) fprintf(output,"                %-30d ",doc_addr-doc);

(void) fprintf(output,"%-10.5f\n",doc_addr-weight);

doc_addr = doc_addr->nextdoc; /* get addr. of next document */

}

(void) fprintf(output,"\n");

inv_addr = inv_addr->next; /* go to next inverted file record */

}

}

/***********************************************************************

centroid( )

Returns:  void

Purpose:  Compute and return the centroid for the documents of

the database.

Notes:    Centroid is stored as a direct file record. Document

number given to it is Number_of_docs +1. The centroid is

computed by determining for each index term in the inverted

file, its average weight in the database. These average

weights are then stored in the direct file entry for the

centroid. (Note that these average weights are not used

anywhere, but could be used in computing DV for terms).

**/

static void centroid( )

{

struct invert *inv_addr;   /* tracks address of current inverted file record */

struct doclist *doc_addr;  /* tracks address of current doclist record*/

float  total_weight,       /* tracks total weight for each term       */

av_term weight;     /* holds average term weight for each term*/

struct termlist *q,        /* structure used to create centroid entry in

the direct file                         */

*lastterm;           /* tracks the last term in the centroid    */

start_centroid = get_mem_direct( ); /* centroid stored as direct file record */

start_centroid->docnum = Number_of_docs +1; /* assign its pseudo doc.# */

start_centroid->next = NULL;  /* end of direct file chain             */

lastterm = NULL;

inv_addr = startinv;  /* begin at top of inverted file  */

while (inv_addr) {  /* while there is a legitimate inv. file record...*/

doc_addr = inv_addr->doc;  /* get address of first document  */

total_weight = 0.0;  /* start with a 0 total weight for this term */

while (doc_addr) {  /* if this is a legitimate doc. addr. */

total_weight = total_weight + doc_addr->weight;

/* update total weight for term */

doc_addr = doc_addr->nextdoc; /* loop through all docs. for the term */

}

av_term_weight = total_weight/Number_of_docs;

/* calculating average term wt. */

q = get_mem_termlist( );

(void) strncpy(q->term,inv-addr->term,MAXWORD);

q->weight = av_term_weight;

q->nextterm = NULL;

if (lastterm == NULL) start_centroid->terms = q;

/* if this is the first term entry for the centroid */

else lastterm->extterm = q;

/* else connect this term to the centroid's termlist chain */

lastterm = q;

inv_addr = inv_addr->next; /* go on to the next inverted file entry */

}

}

/***********************************************************************

pr_centroid( )

Returns:  void

Purpose:  Print the centroid from the direct file

**/

static void pr_centroid( )

{

struct termlist *term_addr;  /* tracks address of current termlist record */

/* note the centroid is given a document number = Number_of_docs + 1 */

/* therefore it may be treated as a special kind of document vector  */

if (start_centroid) {

/* if there is a centroid */

(void) fprintf (output,"----------------------------------------\n");

(void) fprintf(output,"TERM                       WEIGHT \n");

(void) fprintf(output,"-----------------------------------------\n");

term_addr = start_centroid->terms; /* get first term address */

while (term_addr) { /* printing out term and weight pairs */

(void) fprintf(output,"%-30s ",term_addr->term);

(void) fprintf(output,"%-10.5f\n",term_addr->weight);

term_addr = term_addr->nextterm; /* loop through all terms */

}

(void) fprintf(output,"\n");

}

}

/***********************************************************************

get_Poisson_dist( )

Returns:  void

Purpose:  Get the Poisson distribution data for any term

Notes:    This function has two parts:

PART I:  Determine the actual within doc. freq. distribution

PART II: Determine the distribution anticipated under the

single Poisson model.

It is assumed that the within document frequency of a term

is stored as the term weight in the inverted file.

**/

static void get_Poisson_dist( )

{

struct invert *inv_addr;  /* tracks address of current inverted file record */

struct doclist *doc_ad;   /* tracks address of current doclist record */

float dist[MAXWDF][2];    /* store for each term                      */

/* column 1 = within document frequency (wdf) */

/* column 2 = document frequency           */

int i,                    /* counter to add information to the dist array */

j,                   /* counter to loop through dist array      */

found,               /* flag used to match wdf in dist array    */

numdocs,             /* counter to track # of docs. with the same wdf */

docs_with_term;      /* tracks the number of documents having the term */

float first,              /* these five local variables are          */

second,             /* used to determine expected distribution */

result,

exponent,

lambda;             /* single Poisson parameter                */

inv_addr = startinv;      /* start at the beginning of the inverted file */

/* PART I:  For each term determine the number of documents in the

collection that have a particular wdf */

while (inv_addr)  {  /* check for legitimate inv. file record */

docs_with_term = 0;

doc_ad = inv_addr->doc;  /* get the first doc. address  */

i = 0;  /* used to check if this is the very first entry in dist  */

while (doc_ad) {

if (i == 0) { /* if first entry in dist */

dist[i][0] = doc_ad->weight;  dist[i][1] = 1;

/* assign wdf and doc. frequency = 1 to first row in dist */

i++;

docs_with_term++;

}

else  {  /* dist already has other entries, hence look for

any previous entries for the same wdf value */

found = 0;

for (j=0;j < i;j++) { /* loop through dist  */

if (dist[j][0] == doc_ad->weight) { /* if found the same wdf */

dist[j][1] = dist[j][1] + 1;  /* add 1 to the doc. frequency */

found = 1;

docs_with_term++;

}

}  /* ending for */

if (found == 0)  { /* if not found the same wdf in dist */

/* start new row in dist */

dist[i][0] = doc_ad->weight;  dist[i][1] = 1;

i++;

docs_with_term++;

}

} /* ending else */

doc_ad = doc_ad->nextdoc;/* loop through other documents for same term */

}  /* ending while */

/* ending if */

/* now print out actual distribution information for this term  */

(void) fprintf(output,"\nTerm = %s\n",inv_addr->term);

(void) fprintf(output,"\nActual Distribution:  ");

(void) fprintf(output,"          WDF        #docs\n");

(void) fprintf(output,"                          0         %d\n",Number_

of_docs->docs_with_term);

for  (j=0;j < i;j++)

(void) fprintf(output,"                %-16. 0f %-6. 0f\n",dist[j][0]

,dist[j][1]);

/* PART II:  */

/* computing lambda - the only single Poisson parameter  */

(void) fprintf(output,"\nExpected Distribution:  ");

(void) fprintf(output,"           WDF       #docs\n");

/* call the function total_wdf to compute the total frequency of the term */

lambda = (total_wdf(inv_addr->term))/Number_of_docs;

first = exp(-lambda);

numdocs = -1; j = -1;

/* computing document frequency for each within document frequency value */

while (numdocs != 0)  {

j = j + 1;

exponent = j;  /* type conversion necessary for pow function */

if  (j == 0) result = first * Number_of_docs;

else  {

second = pow(lambda,exponent);

result = ( ( (first * second)/factorial(j)) * Number_of_docs);

}

if ( (result - floor(result) )  0.5) numdocs = floor(result);

else numdocs = ceil(result);

(void) fprintf (output,"  %-16d %-6d\n",j,numdocs);

}

inv_addr = inv_addr->next; /* continue with the next inverted file term */

} /* end while */

}

/***********************************************************************

factorial(n)

Returns:  float

Purpose:  Return the factorial of a number. Used in get_poisson_dist

**/

static float factorial(n)

int n;  /* in: compute factorial for this parameter */

{

float answer;  /* holds the result */

if (n==1) return(1.0);

answer = factorial(n-1)*n;

return(answer);

}

/***********************************************************************

total_wdf(term)

Returns:  float

Purpose:  Compute total frequency in the database for a

specified term using the inverted file.

Notes:     It is assumed that the appropriate term frequency is stored

as the term weight in the inverted file. This routine can

also be used to filter out the low and high frequency terms.

The resulting mid frequency terms can be used as input to

the program which generates hierarchies.

**/

static float total_wdf(term)

char term[MAXWORD];   /* in: term for which total frequency is to be found */

{

struct invert *inv_addr;   /* tracks current inverted file record */

struct doclist *doc_addr;  /* tracks current doclist record       */

float total;               /* tracks the total frequency          */

total = 0.0; /* initial value */

/* function find_term will find out where the term is in the inverted file */

inv_addr = find_term(term);

if  (inv_addr) { /* if this term was found in the inverted file */

doc_addr = inv_addr->doc; /* get first associated document address */

while (doc_addr) { /* update the total frequency */

total = total + doc_addr-weight;

doc_addr = doc_addr->nextdoc; /* loop through other associated docs. */

}

}

return(total);

}

/***********************************************************************

Partition_terms( )

Returns:  void

Purpose:  Assign each term in the inverted file to one class:

HIGH, MEDIUM or LOW frequency depending upon its total

frequency in the collection. This function utilizes

two parameters defined at the top of the program:

LOW_THRESHOLD and HIGH_THRESHOLD, which should be set

by the user.

**/

static void Partition_terms( )

{

struct invert *inv_addr;  /* tracks address of current inverted file record */

float total;              /* holds total frequency of each term       */

inv_addr = startinv; /* start at the beginning of the inverted file */

(void) fprintf(output,"\nTerm - Total Frequency - Frequency Class\n\n");

while  (inv_addr) {  /* if a legitimate address */

/* compute total frequency for term in collection */

total = total_wdf(inv_addr->term);

(void) fprintf(output,"\n%s - %f -",inv_addr->term, total);

if (total < LOW_THRESHOLD) (void) fprintf(output," LOW\n");

else if (total > HIGH_THRESHOLD) (void) fprintf(output," HIGH\n");

else (void) fprintf(output," MEDIUM\n");

inv_addr = inv_addr->next; /* continue with next inverted file entry */

}

}

/***********************************************************************

cohesion(term1, term2)

Returns:  float

Purpose:  Compute the cohesion between two terms

**/

static float cohesion(term1, term2)

char term1 [MAXWORD],  /* in: the two terms which are being  */

term2 [MAXWORD];  /* in: compared to determine cohesion */

{

float  l1,             /* holds # of documents associated with term1 */

l2,             /* holds # of documents associated with term2 */

common;         /* holds # of documents in common             */

get_term_data(term1, term2, &l1, &l2, &common);

return(common/(sqrt(l1 * l2) ) );

}

/***********************************************************************

dv_all( )

Returns:  void

Purpose:  Compute Discrimination Value (DV) for all terms in the

database

Notes:     Similarity between two documents as calculated here is a

function of the number of terms in common and the number

of terms in each. Term weights are not involved

**/

static void dv_all( )

{

struct invert *inv_addr;  /* tracks address of current inv. file record */

float DV,                 /* holds computed DV                        */

baseline;           /* holds baseline similarity                */

/* first compute baseline similarity */'

baseline = av_doc_similarity("-");  /* the dummy term '-' is used for this */

(void) fprintf(output,"--------------------------------------------/n");

(void) fprintf(output, "TERM  DV\n");

(void) fprintf(output, "-------------------------------------------/n");

inv_addr = startinv;  /* begin at top of inverted file */

while (inv_addr) {  /* if legitimate inverted file record */

DV = av_doc_similarity(inv_addr ->term - baseline;

(void) fprintf(output," %-30s %-10.5f  \n", inv_addr->term, DV);

inv_addr = inv_addr->next;  /* go to next inverted file record */

}

}

/***********************************************************************

av_doc_similarity(term)

Returns:  float

Purpose:  Compute average similarity between each document

and the centroid of the database. The word specified in

term is ignored during computations.

**/

static float av_doc_similarity (term)

char term[MAXWORD];       /* in: term is ignored during computations  */

{

struct direct *dir_addr;  /* tracks current direct file record        */

float dl1,                /* holds # of terms in document             */

dl2,                /* holds # of terms in centroid             */

common,             /* holds # of terms in common between them  */

total_sim;          /* holds total similarity between them      */

total_sim = 0.0;

dir_addr = startdir; /* begin with first direct file record */

while (dir_addr) {

/* get_doc_data returns #of terms in each document and #of terms in common */

get_doc_data(dir_addr, start_centroid, &dl1, &dl2, &common, term);

total_sim = total_sim + cosine (dl1, dl2, common);

dir_addr = dir_addr->next; /* go to next direct file record */

}

return (total_sim/Number_of_docs);

}

/***********************************************************************

dice (term1, term2)

Returns:  float

Purpose:  Returns Dice's coefficient of similarity between

any two documents

**/

static float dice (term1, term2)

char term1[MAXWORD],     /* in: the two terms that are being compared */

term2[MAXWORD];     /* in:                                       */

{

float l1, l2, common;

get_term data(term1, term2, &l1, &l2, &common);

if (l1 == 0 || l2 == 0) return(0.0);

return(common/(l1 + l2));

}

/***********************************************************************

cosine (l1, l2, common)

Returns:  float

Purpose:  Returns cosine similarity between two documents

**/

static float cosine (l1, l2, common)

float l1,                /* in: # of terms associated with document  1*/

l2,                /* in: # of terms associated with document 2 */

common;            /* in: # of terms in commn between then      */

{

float temp;

if (11 == 0 || 12 == 0) return (0.0);

temp = sqrt (11 * 12);

return(common/temp);

}

/***********************************************************************

get_doc_data(p, q, l1, l2, common, index)

Returns:  void

Purpose:  Given two document numbers, it determines the number of

of index terms in each and in common. It will exclude the index

term (specified as the last parameter) from consideration. Used

in av_doc_similarity for DV calculations.

**/

static void get_doc_data (p,q, l1,l2, common, index)

char index[MAXWORD];   /* in: term to be excluded from computations   */

struct direct *p, *q;  /* in: addresses of two documents numbers      */

float *l1, *l2;        /* out: number of terms in each document       */

float *common;         /* out: number of terms in common              */

{

struct termlist *term_addr1,  /* holds address of first docs. termlist*/

*term_addr2; /* holds address of second docs. termlist */

int count1,                   /* number of terms in first doc.        */

count2,                  /* number of terms in second doc.       */

com;                     /* number of terms in common            */

term_addr1 = p->terms;

count1 = 0; count2 = 0; com = 0;

/* first find out number of terms in document 1 & # of common terms */

while (term_addr1) {

if (strncmp(term_addr1->term, index, MAXWORD)) count1 = count1 + 1;

/* if its not the term to exclude */

term_addr2 = q->terms;

while (term_addr2) {

if (!strncmp(term_addr1-term, term_addr2-term, MAXWORD)) {

/* if the two terms are the same */

if (! strncmp(term_addr1-term, index, MAXWORD)) com = com + 1;

/* if they do not match the term to exclude */

break;

}

term_addr2 = term_addr2-nextterm;

}

term_addr1 = term_addr1-nextterm;

}

/* now find out number of terms in document 2 */

term_addr2 = q ->terms;

while (term_addr2) {

if (strncmp(term_addr2->term, index, MAXWORD)) count2 = count2 + 1;

term_addr2 = term_addr2-nextterm;

}

*l1 = count1; *12 = count2; *common = com;

}

/***********************************************************************

get_term_data(term1, term2, l1, l2, common)

Returns:  void

Purpose:  Get info regarding number of documents in common

between any two terms.

**/

static void get_term_data(term1, term2, l1, l2, common)

char  term1[MAXWORD],   /* in: term 1 to be compared with             */

term2[MAXWORD];   /* in: term 2                                 */

float *l1,              /* out: # of documents associated with 1st term */

*l2,              /* out: # of documents associated with 2nd term */

*common;          /* out: # of documents in common between them */

{

struct invert *p,       /* holds address of first term in inverted file */

*q;       /* holds address of second term in inv. file */

struct doclist *doc_ad1, /* tracks doclist for first term            */

*doc_ad2; /* tracks doclist for second term            *

int count1,             /* holds # of documents associated with 1st term */

count2,            /* holds # of documents associated with 2nd term */

com;               /* holds # of documents common between them   */

/* find addresses of both terms in the inverted file */

p = find_term(terml); q = find_term(term2);

doc_ad1 = p-doc;  /* obtain 1st terms doclist address */

count1 = 0; count2 = 0; com = 0;

/* first get # of documents indexed by term 1 & # of docs. in common */

while (doc_ad1) {

count1 = count1 +1;

doc_ad2 = q->doc;

while (doc_ad2) {

if (doc_ad1->doc == doc_ad2->doc) {

/* if the document numbers are the same */

com = com +1;

break;

}

doc_ad2 = doc_ad2->nextdoc;

}

doc_ad1 = doc_ad1->nextdoc;

}

/* now get # of documents indexed by term 2 */

doc_ad2 = q->doc;

while (doc_ad2) {

count2 = count2 + 1;

doc_ad2 = doc_ad2->nextdoc;

}

*l1 = count1; *l2 - count2; *common = com;

}

/***********************************************************************

*find_term(term)

Returns:  address of a struct invert record

Purpose:  search for a specified term in the inverted file &

return address of the record

**/

struct invert *find_term(term)

char term[MAXWORD];       /* in: term to be located in inverted file  */

{

struct invert *inv_addr;  /* tracks addr. of current rec. in inv. file*/

inv_addr = startinv;      /* begin at top of inv. file                */

while(inv_addr) {

if (!strcmp(term,inv_addr->term)) {return(inv_addr);}

inv_addr = inv_addr->next;

}

(void) fprintf(output,"Findterm routine: Term %s not found\n",term);

return(NULL);

}

/***********************************************************************

*get_mem_direct( )

Returns:  address of a struct direct record

Purpose:  dynamically obtain enough memory to store 1 direct record

**/

static struct direct *get_mem_direct( )

{

struct direct *record;

record = (struct direct *)malloc(sizeof(directfile));

if (!record)  {

(void) fprintf(output,"\nout of memory\n");

exit(0);

}

return(record);

}

/***********************************************************************

*get_mem_termlist( )

Returns:  address of a struct termlist record

Purpose:  dynamically obtain enough memory to store one termlist record

**/

static struct termlist *get_mem_termlist( )

{

struct termlist *record;

record = (struct termlist *)malloc(sizeof (termlistfile));

if (!record) {

(void) fprintf(output,"\nout of memory\n");

exit(0);

}

return(record);

}

/***********************************************************************

*get_mem_invert( )

Returns:  address of a struct invert record

Purpose:  dynamically obtain enough memory to store one inverted

file record

**/

static struct invert *get_mem_invert( )

{

struct invert *record;

record = (struct invert *)malloc(sizeof(invfile));

if (!record) {

(void) fprintf(output,"\nout of memory\n");

exit(0);

}

return(record);

}*

/***********************************************************************

*get_mem_doclist( )

Returns:  address of a struct doclist record

Purpose:  dynamically obtain enough memory to store one doclist

record

**/

static struct doclist *get_mem_doclist( )

{

struct doclist *record;

record = (struct doclist *)malloc(sizeof(doclistfile));

if (!record) {

(void) (void) fprintf(output,"\nout of memory\n");

exit(0);

}

return(record);

/*

PROGRAM NAME:   merge.c

PURPOSE:   This program is used to merge two separate hierarchies.

The program first reads each inverted file. It then reads in the

corresponding link file which gives parent-child links to build

the hierarchy. It can then perform two different types of mergers:

1)  Simple merge in which a point of connection between the two

thesauri is made wherever they have terms in common.

2)  Complex merge in which any two terms are connected if they

have sufficiently (above a specified threshold) similar sets

of parent and child terms.

INPUT FILES REQUIRED:

1)  inverted file for 1st hierarchy

2)  links file for 1st hierarchy

3)  inverted file for 2nd hierarchy

4)  links file for 2nd hierarchy

An inverted file consists of sequences of

term   document number   weight

(multiple entries for any term should be grouped together)

A link file consists of sequences of

parent term  child term

PARAMETERS TO BE SET BY USER:

1)  MAXWORD:  which specifies the maximum size of a term

2)  THRESHOLD:  which specifies the minimum similarity level

for use in complex merge.

COMMAND LINE:

merge inverted_file_1 link_file_1 inverted_file_2 link_file_2 output_file

/***********************************************************************

#include <stdio.h>

#include <string.h>

#include <math.h>

#define MAXWORD 20             /* maximum size of a term             */

#define THRESHOLD 0.6          /* similarity threshold for complex_merge */

struct doclist {               /* sequences of document # and weight */

int doc;                     /* document number                    */

float weight;                /* term weight in document            */

struct doclist *nextdoc;     /* ptr. to next doclist record        */

} doclistfile;

struct parentlist {             /* sequences of parent terms         */

char term[MAXWORD];           /* parent term                       */

struct invert *parent;        /* ptr. to parent term in inverted file */

struct parentlist *nextparent;/* ptr. to next parentlist record   */

} parentfile;

struct connections {            /* holds information about connected terms */

struct invert *termadd;      /* address of connected term in inverted file */

struct connections *next_connection  /* ptr. to next connections record */

} connectlist;

struct childlist {             /* sequences of child terms           */

char term[MAXWORD];          /* child term                         */

struct invert *child;        /* ptr. to child term in inverted file*/

struct childlist *nextchild; /* ptr. to next childlist record      */

} childfile;

struct invert {                /* inverted file                      */

char term[MAXWORD];          /* term                               */

struct doclist *doc;         /* sequences of document # and weight */

struct parentlist *parents;  /* pointer to list of parent terms    */

struct childlist *children;  /* pointer to list of children terms  */

struct connections *connect; /* pointer to connection in other hierarchy */

struct invert *nextterm;     /* ptr. to next invert record         */

} invfile;

static struct invert *startinv;    /* ptr. to first record in inverted file */

static struct invert *lastinv;     /* ptr. to last record in inverted file */

static struct doclist *lastdoc;    /* ptr. to last document in doclist */

static struct invert *start_inv1;  /* ptr. to the start of 1st inverted file */

static struct invert *start_inv2;  /* ptr. to the start of 2nd inverted file */

static FILE *input1;               /* first inverted file             */

static FILE *input2;               /* first link file                 */

static FILE *input3;               /* second inverted file            */

static FILE *input4;               /* second link file                */

static FILE *output;               /* holds any outputs               */

static char currentterm[MAXWORD];  /* tracks current term in inverted file */

static struct invert *get_mem_invert( ); /* these four get_mem functions * /

static struct doclist *get_mem_doclist( );       /* obtain memory dynamically for */

static struct parentlist *get_mem_parentlist( ); /* storing different types of */

static struct childlist *get_mem_childlist( );   /* records.         */

static struct connections *get_mem_connections( );

static struct invert *find_term( );   /* searches for term in inverted file and */

/* returns address of the term */

static int compare( );

static

void initialize( ),              /* initialize global variables      */

open_files( ),             /* open files                       */

read_invfile( ),           /* read in the inverted file        */

add_invert( ),             /* called within read_invfile( )    */

read_links( ),             /* read in the links information    */

add_link( ),               /* called within read_links( )      */

pr_invert( ),              /* print the inverted file          */

simple_merge( ),           /* simple merge between both hierarchies */

complex_merge( );          /* complex merge between hierarchies */

int main(argc,argv)

int argc;

char *argv[ ];

{

char ch;

if (argc!=6)

{

(void) printf( "There is an error in the command line\n");

(void) printf("Correct usage is:\n");

(void) printf("merge inverted_file_1 link_file_1 inverted_file_2 link_file_2 output_file\n");

exit(1);

}

start_inv1 = NULL; start_inv2 = NULL; /* initialize start of both inverted files */

initialize( );

open_files(argv);

(void) fprintf(output, "\nREADING FIRST INVERTED FILE\n");

read_invfile(input1);

start_inv1 = startinv;

(void) fprintf(output, "\nREADING FIRST LINK FILE\n");

read_links(input2, start_inv1);

(void) fprintf(output, "\nPRINTING FIRST INVERTED FILE\n\n");

pr_invert(start_inv1);

/*  re-initialize  */

initialize( );

(void) fprintf(output, "\nREADING SECOND INVERTED FILE\n");

read_invfile(input3);

start_inv2 = startinv;

(void) fprintf(output, "\nREADING SECOND LINK FILE\n");

read_links(input4, start_inv2);

(void) fprintf(output, "\nPRINTING SECOND INVERTED FILE\n\n");

pr_invert(start_inv2);

(void) printf("Make a selection\n");

(void) printf("To use the simple_merge algorithm, enter 1\n");

(void) printf("To use the complex_merge algorithm, enter 2\n");

(void) printf("\nEnter selection: ");

ch = getchar( );

switch(ch)

{

case  '1':

(void) fprintf(output,"\nPERFORMING A SIMPLE MERGE OF THE TWO INVERTED FILES\n");

simple_merge(start_inv1,start_inv2);

(void) fprintf(output,"\nPRINTING FIRST INVERTED FILE AFTER SIMPLE MERGE\n\n");

pr_invert(start_inv1);

(void) fprintf(output,"\nPRINTING SECOND INVERTED FILE AFTER SIMPLE MERGE\n\n");

pr_invert(start_inv2);

break;

case  '2':

(void) fprintf(output,"\nPERFORMING A COMPLEX MERGE OF THE TWO INVERTED FILES\n");

complex_merge(start_inv1,start_inv2);

(void) fprintf(output,"\nPRINTING FIRST INVERTED FILE AFTER COMPLEX MERGE\n\n");

pr_invert(start_inv1);

(void) fprintf(output,"\nPRINTING SECOND INVERTED FILE AFTER COMPLEX MERGE\n\n");

pr_invert(start_inv2);

}

(void) fclose(input1);(void) fclose(input2);

(void) fclose(input3);(void) fclose(input4);(void) fclose(output);

return(0);

}

/***********************************************************************

open_files(argv)

Returns:  void

Purpose:  Open all input & output files

**/

static void open_files(argv)

char *argv[ ];

{

if (( input1 = fopen(argv[1],"r")) == NULL )  {

(void) printf("couldn't open file %s\n",argv[1]);

exit(1); /* inverted file for first thesaurus hierarchy */

}

if (( input2 = fopen(argv[2],"r")) == NULL )  {

(void) printf("couldn't open file %s\n",argv[2]);

exit(1); /* link file for first thesaurus hierarchy */

}

if (( input3 = fopen(argv[3],"r")) == NULL )  {

(void) printf("couldn't open file %s\n",argv[3]);

exit(1); /* inverted file for second thesaurus hierarchy */

}

if (( input4 = fopen(argv[4],"r")) == NULL )    {

(void)  printf("couldn't open file %s\n",argv[4]);

exit(1);  /*  link file for second thesaurus hierarchy */

}

if (( output = fopen(argv[5],"w")) == NULL)  {

(void) printf("couldn't open file %s for output \n",argv[5]);

exit(1);  /*  output file */

}

}

/***********************************************************************

initialize( )

Returns:  void

Purpose:  Initialize global variables

**/

static void initialize( )

{

startinv = NULL;            /* start of inverted file        */

lastinv = NULL;             /* end of inverted file          */

lastdoc = NULL;             /* last document considered      */

currentterm[0] = '\0';      /* current term being considered */

}

/***********************************************************************

read_invfile(input)

Returns:  void

Purpose:  Read the inverted file from a disk file

**/

static void read_invfile(input)

FILE *input;

{

int docid;              /* holds current document number              */

char temp[MAXWORD];     /* holds current term                         */

float weight;           /* holds current term weight                  */

struct doclist *p;      /* structure to hold document numner-weight pair */

(void) fscanf(input, "%s%d%f", temp, &docid, &weight);  /* read next line */

while (strlen(temp)  >  0)             /*  while a legitimate line   */

{

if (!strncmp(currentterm,temp,MAXWORD)) {

/* if temp is the same as current term then simply add next document-weight info */

p = get_mem_doclist( );

p->doc = docid;         /* assign doc. number  */

p->weight = weight;     /* assign doc. weight  */

p->nextdoc = NULL;

if  (lastdoc) lastdoc->nextdoc = p;  /* connect p to doclist chain for this term */

lastdoc = p;                         /* set this global variable */

}

else add_invert(docid,temp,weight);  /* temp not the same as current term, hence */

temp[0] = '\0';                      /* start a new entry in the inverted file */

(void) fscanf(input, "%s%d%f", temp, &docid, &weight); /* read next input line */

}

}

/***********************************************************************

add_invert(docid,temp,weight)

Returns:  void

Purpose:  Called in read_invfile when a new term is being read from the file.

Starts a new entry in the inverted file.

**/

static void add_invert(docid, temp, weight)

int docid;                      /* in: document number                */

char temp[MAXWORD];             /* in: new index term                 */

float weight;                   /* in: index term weight              */

{

struct invert *p;               /* structure p will be attached to inv. file */

p = get_mem_invert( );          /* get memory for p                   */

(void) strncpy(p->term,temp,MAXWORD);  /* copy over the term          */

p->parents = NULL;              /* initially term has no parents      */

p->children = NULL;             /* also no children terms             */

p->doc = get_mem_doclist( );    /* get memory for a doclist structure */

p->doc->doc = docid;            /* assign the document number         */

p->doc->weight = weight;        /* assign term weight                 */

p->doc->nextdoc = NULL;

p->connect = NULL;              /* initially this term not connected to any */

/* other in any other hierarchy     */

p->nextterm = NULL;

if (startinv == NULL) startinv = p;  /* if this is the 1st term in inverted file */

if (lastinv) lastinv->nextterm = p;  /* always update lastinv pointer */

lastinv = p;

lastdoc = p-doc;

(void) strncpy(currentterm,temp,MAXWORD); /* update the value of currentterm to the */

/* new term that has just been read  */

}

/***********************************************************************

read_links(input,startinv)

Returns:  void

Purpose:  Add the link information to the inverted file

**/

static void read_links(input,startinv)

FILE *input;                    /* in: input file                    */

struct invert *startinv;        /* in: start of this inverted file   */

{

char parent[MAXWORD],           /* holds parent term                 */

child[MAXWORD];            /* holds child term                  */

(void) fscanf(input, "%s%s", parent,child);  /* read first input line */

while (strlen(parent) > 0 && strlen(child) > 0)  /* while non-trivial input */

{

if (!find_term(parent,startinv) | | !find_term(child,startinv))

{

(void) printf("Please check your input files\n");

(void) printf("Term %s or term %s is not in inverted file\n",parent,child);

exit(0);

}

add_link(parent,child,startinv);    /* this function makes links      */

child[0] = '\0'; parent[0] = '\0';  /* throw out old parent & child info */

(void) fscanf(input, "%s%s",parent,child);  /* read next line         */

}

}

/***********************************************************************

add_link(parent,chile,startinv)

Returns:  void

Purpose:  Function is used within read_links

**/

static void add_link(parent, child, startinv)

char  parent[MAXWORD],          /* in:   specify parent term          */

child[MAXWORD];           /* in:   specify child term           */

struct invert *startinv;        /* in:   specify start of this inv. file */

{

struct invert *p, *q;           /* holds adds. of both terms in inv. file */

struct parentlist *new_parent;  /* structure to hold new parent info. */

struct childlist *new_child;    /* structure to hold new child info.  */

p = find_term(parent, startinv);    /* find address of parent & child */

q = find_term(child, startinv);     /* terms in the inverted file     */

/* first add parent links for the given child */

new_parent = get_mem_parentlist( ); /* get memory for parent record   */

(void) strncpy(new_parent->term, parent, MAXWORD); /* copy over parent term */

new_parent->parent = p;    /* store addr.(in inverted file) of parent term */

if (q->parents == (NULL)  {   /* i. e. no parents listed for this child yet*/

q->parents = new_parent;         /* first parent link made        */

new_parent->nextparent = NULL;

}

else  {       /* at least 1 parent already listed for given child    */

new_parent->nextparent = q->parents; /* attach new parent in front of list */

q->parents = new_parent;

}

/* next add child links for given parent */

new_child = get_mem_childlist( );    /* get memory for child record   */

(void) strncpy(new_child->term, child, MAXWORD);    /* copy over child term */

new_child->child = q;      /* store addr. (in inverted file) of child term */

if (p->children == NULL) {   /* i. e. no child terms listed for this parent */

p->children = new_child;         /* first child link made         */

new_child->nextchild = NULL;

}

else {        /* at least 1 child already exists for given parent    */

new_child->nextchild = p->children;/* attach new child to front of list */

p->children = new_child;

}

}

/***********************************************************************

pr_invert(startinv)

Returns:  void

Purpose:  Print either inverted file. Prints each term, its

associated document numbers, term-weights and parent

and child terms.

**/

static void pr_invert(startinv)

struct invert *startinv;          /* in: specifies start of inverted file */

{

struct invert *inv_addr;          /* tracks add. of current inv. file record */

struct doclist *doc_addr;         /* tracks add. of current doclist record */

struct parentlist *parent_addr;   /* tracks add. of current parentlist record */

struct childlist *child_addr;     /* tracks add. of current childlist record */

struct connections *connect_term_add; /* tracks connected terms      */

inv_addr = startinv;                  /* begin at top of inv. file   */

while (inv_addr) {                    /* while a legitimate term. . . .*/

(void) fprintf(output,"TERM: %s\nPARENT TERMS:, ", inv_addr>term);

parent_addr = inv_addr->parents;    /* find addr. of first parent */

while (parent_addr) {               /* printing all parents       */

(void) fprintf(output,"%s",parent_addr->term);

parent_addr = parent_addr->nextparent;  /* loop through remaining parents */

if(parent_addr) (void) fprintf(output,", ");

}

(void) fprintf(output,"\nCHILD TERMS: ");

child_addr = inv_addr->children;       /* find addr. of first child   */

while (child_addr) {                   /* printing all children       */

(void) fprintf(output,"%s",child_addr->term);

child_addr = child_addr->nextchild;  /* loop through remaining childrend */

if(child_addr) (void) fprintf(output,", ");

}

(void) fprintf(output,"\nDOCUMENT NUMBER            TERM WEIGHT\n");

doc_addr = inv_addr->doc;   /* find addr. of first associated doc.   */

while (doc_addr) {             /* printing all documents             */

(void) fprintf(output,"%-30d",doc_addr->doc);

(void) fprintf(output,"%-10.5f\n",doc_addr->weight);

doc_addr = doc_addr->nextdoc;   /* loop through remaining docs.    */

}

/* if the terms is connected then print the term from the other hierarchy */

(void) fprintf(output, "CONNECTIONS IN OTHER THESAURUS HIERARCHY:\n");

connect_term_add = inv_addr->connect;

while (connect_term_add)

{

(void) fprintf(output,"   %s", connect_term_add-termadd-term);

connect_term_add = connect_term_add-next_connection;

if(connect_term_add) (void) fprintf(output,", ");

}

(void) fprintf(output,"\n\n");

inv_addr = inv_addr->nextterm;       /* loop to next term in inverted file */

}

}

/***********************************************************************

simple_merge(startinv1, startinv2)

Returns:  void

Purpose:  In this function, two terms in different hierarchies are merged

if they are identical.

**/

static void simple_merge(startinv1, startinv2)

struct invert *startinv1,       /* in: specifies start of 1st inv. file */

*startinv2;       /* in: specifies start of 2nd inv. file */

{

struct invert *inv_addr1, *inv_addr2;

struct connections *r1, *r2  /* storage to hold info. about connected terms */

inv_addr1 = startinv1;          /* start with top of 1st inv. file   */

while (inv_addr1) {        /* looking for this term in the other inv. file */

inv_addr2 = find_term(inv_addr1->term, startinv2);

if  (inv_addr2) {   /* if term was found then update connect  */

r1 = get_mem_connections( );

r1->termadd = inv_addr2;

r1->next_connection = inv_addr1->connect;

inv_addr1->connect = r1;

r2 = get_mem_connections( );

r2->termadd = inv_addr1;

r2-=>next_connection = inv_addr2->connect;

inv_addr2->connect = r2;

}

inv_addr1 = inv_addr1->nextterm;

}

}

/***********************************************************************

complex_merge(startinv1, startinv2)

Returns:  void

Purpose:  In this routine any two terms in different hierarchies are merged

if they have 'similar' parents and children. Similarity is

computed and compared to a pre-fixed user specified THRESHOLD

**/

static void complex_merge(startinv1, startinv2)

struct invert *startinv1,   /* in: specifies start of 1st inv. file */

*startinv2;   /* in: specifies start of 2nd inv. file */

{

struct invert *inv1_addr,   /* tracks current term in 1st inv. file */

*inv2_addr;   /* tracks current term in 2nd inv. file */

struct connections *r1, *r2; /* tracks connected terms              */

int compare ( );

inv1_addr = startinv1;       /* begin at top of 1st inv. file       */

while(inv1_addr) {           /* while addr. legitimate . . . .      */

inv2_addr = startinv2;   /* now begin at top of 2nd inv. file   */

while(inv2_addr)  {

if (compare(inv1_addr, inv2_addr)) {  /* this returns 1 of 2 terms are */

/* similar enough, then connect the two terms */

r1 = get_mem_connections();

r1->termadd = inv2_addr;

r1->next_connection = inv1_addr->connect;

inv1_addr->connect = r1;

r2 = get_mem_connections();

r2->termadd = inv1_addr;

r2->next_connection = inv2_addr->connect;

inv2_addr->connect = r2;

}

inv2_addr = inv2_addr->nextterm;  /* loop through 2nd inv. file */

}

inv1_addr = inv1_addr->nextterm;        /* loop through 1st inv. file */

}

}

/***********************************************************************

compare(p,q)

Returns:  int

Purpose:  Used to compare two terms for more than just equality.

A similarity value is computed and if it is greater than a THRESHOLD then 1 is returned, else 0

**/

static int compare(p,q)

struct invert *p, *q;      /* addresses of two terms to be compared   */

{

struct parentlist *parent1,  /* tracks parentlist of 1st term         */

*parent2;  /* tracks parentlist of 2nd term         */

struct childlist  *child1,   /* tracks childlist of 1st term          */

*child2;   /* trakcs childlist of 2nd term          */

float count1,                /* tracks # of parents + children of 1st term */

count2,                /* tracks # of parents + children of 2nd term  */

count;                 /* tracks # of common parents + children */

count = 0.0; count1 = 0.0; count2 = 0.0;  /* initialize all counts    */

/* first check # of parents for p-term & the # of common parents      */

parent1 = p->parents;

while(parent1) {  /* parent of 1st term */

count1 = count1 + 1.0;

parent2 = q->parents;

while(parent2) {  /* loop through parents of second term  */

if(!strncmp(parent1->term,parent2->term,MAXWORD)) count = count + 1.0;

parent2 = parent2->nextparent;

}

parent1 = parent1->nextparent;

}

/* next compute # of parents for q->term  */

parent2 = q->parents;

while (parent2) { /* loop through parents of 2nd term again       */

count2 = count2 + 1.0;

parent2 = parent2->nextparent;

}

/* now check # of children for p-term & the # of common children  */

child1 = p->children;

while(child1) {  /* loop through children of 1st term             */

count1 = count1 + 1.0;

child2 = q->children;

while(child2) {  /* loop through children of 2nd term          */

if (!strncmp(child1->term,child2->term,MAXWORD)) count = count + 1.0;

child2 = child2->nextchild;

}

child1 = child1->nextchild;

}

/* next compute # of children for q->term  */

child2 = q->children;

while (child2)  {  /* loop through children of 2nd term          */

count2 = count2 + 1.0;

child2 = child2->nextchild;

}

if  (count ! = 0.0) { /* if there is anything in common at all   */

if  ((count/(sqrt(count1 * count2))) = THRESHOLD) {

/* printf("value is %f\n",(count/(sqrt(count1*count2))));  */

return(1);

}

else return(0);

}

return(0);

}

/**********************************************************************

find_term(term,startinv)

Returns: address of a struct invert record

Purpose: Search for a specified term in the specified inverted file

and return address of the corresponding record. If not

found then returns NULL.

**/

static struct invert *find_term(term, startinv)

char term[MAXWORD];       /* in: term to be searched         */

struct invert *startinv;  /* in: inverted file to search in  */

{

struct invert *inv_addr;

inv_addr = startinv;      /* begin at top of inverted file   */

while(inv_addr) {

if (!strcmp(term,inv_addr->term)) return(inv_addr);

inv_addr = inv_addr-nextterm;

}

return(NULL);

}

/**********************************************************************

*get_mem_invert( )

Returns:  address of a struct invert record

Purpose:  dynamically obtain enough memory to store 1 invert record

**/

static struct invert *get_mem_invert ()

{

struct invert *record;

record = (struct invert *)malloc(sizeof(invfile));

if (!record) {

(void) fprintf(output,"\nout of memory\n");

return(NULL);

}

return(record);

}

/***********************************************************************

*get_mem_doclist ()

Returns:  address of a struct doclist record

Purpose:  dynamically obtain enough memory to store 1 doclist record

**/

static struct doclist *get_mem_doclist()

{

struct doclist *record;

record = (struct doclist *)malloc(sizeof(doclistfile));

if (!record) {

(void) fprintf(output,"\nout of memory\n");

return(NULL);

}

return(record);

}

/***********************************************************************

*get_mem_parentlist()

Returns:  address of a struct parentlist record

Purpose:  dynamically obtain enough memory to store 1 parentlist record.

**/

static struct parentlist *get_mem_parentlist()

{

struct parentlist *record;

record = (struct parentlist *)malloc(sizeof(parentfile));

if  (!record) {

(void) fprintf(output,"\nout of memory\n");

return(NULL);

}

return(record);

}

/****************************************************************************

*get_mem_childlst()

Returns:  address of a struct childlist record

Purpose:  dynamically obtain enough memory to store 1 childlist record.

**/

static struct childlist *get_mem_childlist ()

{

struct childlist *record;

record = (struct childlist *)malloc(sizeof(childfile));

if (!record) {

(void) fprintf(output,"\nout of memory\n");

return(NULL);

}

return(record);

}

/***********************************************************************

*get_mem_connections()

Returns:  address of a struct connections record

Purpose:  dynamically obtain enough memory to store 1 connections record.

**/

static struct connections *get_mem_connections( )

{

struct connections *record;

record = (struct connections *)malloc(sizeof(connectlist));

if (!record) {

(void) fprintf(output,"\nout of memory\n");

return(NULL);

}

return(record);

}

Go to Chapter 10     Back to Table of Contents


Information Retrieval: CHAPTER 10: STRING SEARCHING ALGORITHMS

 Return to Table of Contents

CHAPTER 10: STRING SEARCHING ALGORITHMS

Ricardo A. Baeza-Yates

Department of Computer Science, University of Chile, Casilla 2777, Santiago, Chile

Abstract

We survey several algorithms for searching a string in a text. We include theoretical and empirical results, as well as the actual code of each algorithm. An extensive bibliography is also included.

10.1 INTRODUCTION

String searching is an important component of many problems, including text editing, data retrieval, and symbol manipulation. Despite the use of indices for searching large amounts of text, string searching may help in an information retrieval system. For example, it may be used for filtering of potential matches or for searching retrieval terms that will be highlighted in the output.

The string searching or string matching problem consists of finding all occurrences (or the first occurrence) of a pattern in a text, where the pattern and the text are strings over some alphabet. We are interested in reporting all the occurrences. It is well known that to search for a pattern of length m in a text of length n (where n > m) the search time is 0(n) in the worst case (for fixed m). Moreover, in the worst case, at least n - m + 1 characters must be inspected. This result is due to Rivest (1977). However, for different algorithms the constant in the linear term can be very different. For example, in the worst case, the constant multiple in the naive algorithm is m, whereas for the Knuth-Morris-Pratt (1977) algorithm it is two.

We present the most important algorithms for string matching: the naive or brute force algorithm, the Knuth-Morris-Pratt (1977) algorithm, different variants of the Boyer-Moore (1977) algorithm, the Shift-or algorithm from Baeza-Yates and Gonnet (1989), and the Karp-Rabin (1987) algorithm, which is probabilistic. Experimental results for random text and one sample of English text are included. We also survey the main theoretical results for each algorithm.

Although we only cover string searching, references for related problems are given. We use the C programming language described by Kernighan and Ritchie (1978) to present our algorithms.

10.2 PRELIMINARIES

We use the following notation:

n: the length of the text

m: the length of the pattern (string)

c: the size of the alphabet *

the expected number of comparisons performed by an algorithm while searching the pattern in a text of length n

Theoretical results are given for the worst case number of comparisons, and the average number of comparisons between a character in the text and a character in the pattern (text pattern comparisons) when finding all occurrences of the pattern in the text, where the average is taken uniformly with respect to strings of length n over a given finite alphabet.

Quoting Knuth et al. (1977). "It might be argued that the average case taken over random strings is of little interest, since a user rarely searches for a random string. However, this model is a reasonable approximation when we consider those pieces of text that do not contain the pattern, and the algorithm obviously must compare every character of the text in those places where the pattern does occur." Our experimental results show that this is the case.

The emperical data, for almost all algorithms, consists of results for two types of text: random text and English text. The two cost functions we measured were the number of comparisons performed between a character in the text and a character in the pattern, and the execution time. To determine the number of comparisons, 100 runs were performed. The execution time was measured while searching l,000 patterns. In each case, patterns of lengths 2 to 20 were considered.

In the case of random text, the text was of length 40,000, and both the text and the pattern were chosen uniformly and randomly from an alphabet of size c. Alphabets of size c = 4 (DNA bases) and c = 30 (approximately the number of lowercase English letters) were considered.

For the case of English text we used a document of approximately 48,000 characters, with the patterns chosen at random from words inside the text in such a way that a pattern was always a prefix of a word (typical searches). The alphabet used was the set of lowercase letters, some digits, and punctuation symbols, giving 32 characters. Unsuccessful searches were not considered, because we expect unsuccessful searches to be faster than successfu1 searches (fewer comparisons on average). The results for English text are not statistically significant because only one text sample was used. However, they show the correlation of searching patterns extracted from the same text, and we expect that other English text samples will give similar results. Our experimental results agree with those presented by Davies and Bowsher (1986) and Smit (1982).

We define a random string of length [image] as a string built as the concatenation of [image] characters chosen independently and uniformly from [image]. That is, the probability of two characters being equal is 1/c. Our random text model is similar to the one used in Knuth et al. (1977) and Schaback (1988).

For example, the probability of finding a match between a random text of length m and a random pattern of length m is

The expected number of matches of a random pattern of length m in a random text of length n is

10.3 THE NAIVE ALGORITHM

The naive, or brute force, algorithm is the simplest string matching method. The idea consists of trying to match any substring of length m in the text with the pattern (see Figure 10.1).

naivesearch( text, n, pat, m ) /* Search pat[1..m] in text[1..n] */

char text[], pat[];

int n, m;

{

int i, j, k, lim;

lim = n-m+1;

for( i = 1; i<= lim; i++ ) /* Search */

{

k = i;

for( j=1; j<=m && text[k] == pat[j]; j++ ) k++;

if( j > m ) Report_match_at_position( i-j+1 );

}

}

Figure 10.1: The naive or brute force string matching algorithm

The expected number of text pattern comparisons performed by the naive or brute force algorithm when searching with a pattern of length m in a text of length n (n   [image]  m) is given by Baeza-Yates (1989c) as

This is drastically different from the worst case mn.

10.4 THE KNUTH-MORRIS-PRATT ALGORITHM

The classic Knuth, Morris, and Pratt (1977) algorithm, discovered in 1970, is the first algorithm for which the constant factor in the linear term, in the worst case, does not depend on the length of the pattern. It is based on preprocessing the pattern in time O(m). In fact, the expected number of comparisons performed by this algorithm (search time only) is bounded by

The basic idea behind this algorithm is that each time a mismatch is detected, the "false start" consists of characters that we have already examined. We can take advantage of this information instead of repeating comparisons with the known characters. Moreover, it is always possible to arrange the algorithm so that the pointer in the text is never decremented. To accomplish this, the pattern is preprocessed to obtain a table that gives the next position in the pattern to be processed after a mismatch. The exact definition of this table (called next in Knuth et al. [1977]) is

next[j] = max{i|(pattern[k] = pattern [j - i + k] for k = 1, . . . , i - 1)

and pattern[i]   [image]  pattern[j]}

for j = 1, . . . , m. In other words, we consider the maximal matching prefix of the pattern such that the next character in the pattern is different from the character of the pattern that caused the mismatch. This algorithm is presented in Figure 10.2.

Example 1 The next table for the pattern abracadabra is

                      a  b  r  a  c  a  d  a  b  r  a

            next[j]   0  1  1  0  2  0  2  0  1  1  0  5

When the value in the next table is zero, we have to advance one position in the text and start comparing again from the beginning of the pattern. The last value of the next table (five) is used to continue the search after a match has been found.

In the worst case, the number of comparisons is 2n + O(m). Further explanation of how to preprocess the pattern in time O(m) to obtain this table can be found in the original paper or in Sedgewick (1983; see Figure 10.3).

kmpsearch( text, n, pat, m ) /* Search pat[1..m] in text[1..n] */

char text[], pat[];

int n, m;

{

int j, k, resume;

int next[MAX_PATTERN_SIZE];

pat[m+1] = CHARACTER_NOT_IN_THE_TEXT;

initnext( pat, m+1, next ); /* Preprocess pattern */

resume = next[m+1];

j = k = 1;

do {                        /* Search */

if( j==0 || text[k]==pat[j] )

{

k++; j++;

}

else j = next[j];

if( j > m )

{

Report_match_at_position( k-m );

J = resume;

}

} while( k <= n );

pat[m+1] = END_OF_STRING;

}

Figure 10.2: The Knuth-Morris-Pratt algorithm

initnext( pat, m, next ) /* Preprocess pattern of length m */

char pat[];

int m, next[];

{

int i, j;

i = 1; j = next[1] = 0;

do

{

if( j == 0 || pat[i] == pat[j] )

{

i++; j++;

if( pat[i] != pat[j] )  next[i] = j;

else                    next[i] = next[j];

}

else j = next[j];

}

while( i <= m ) ;

}

Figure 10.3: Pattern preprocessing in the Knuth-Morris-Pratt algorithm

An algorithm for searching a set of strings, similar to the KMP algorithm, was developed by Aho and Corasick (1975) . However the space used and the preprocessing time to search for one string is improved in the KMP algorithm. Variations that compute the next table "on the fly" are presented by Barth (1981) and Takaoka (1986). Variations for the Aho and Corasick algorithm are presented by Bailey and Dromey (1980) and Meyer (1985).

10.5 THE BOYER-MOORE ALGORITHM

Also in 1977, the other classic algorithm was published by Boyer and Moore (1977). Their main idea is to search from right to left in the pattern. With this scheme, searching is faster than average.

The Boyer-Moore (BM) algorithm positions the pattern over the leftmost characters in the text and attempts to match it from right to left. If no mismatch occurs, then the pattern has been found. Otherwise, the algorithm computes a shift; that is, an amount by which the pattern is moved to the right before a new matching attempt is undertaken.

The shift can be computed using two heuristics: the match heuristic and the occurrence heuristic. The match heuristic is obtained by noting that when the pattern is moved to the right, it must

1.     match all the characters previously matched, and

2.     bring a different character to the position in the text that caused the mismatch.

The last condition is mentioned in the Boyer-Moore paper (1977), but was introduced into the algorithm by Knuth et al. (1977). Following the later reference, we call the original shift table dd, and the improved version   [image]. The formal definitions are

Example 2 The   [image]  table for the pattern abracadabra is

                        a   b   r   a   c   a   d   a   b  r  a

            ddhat [j]  17  16  15  14  13  12  11  13  12  4  1

The occurrence hueristic is obtained by noting that we must align the position in the text that caused the mismatch with the first character of the pattern that matches it. Formally calling this table d, we have

d[x] = min{s|s = m or (0    s < m and pattern [m - s] = x)}

for every symbol x in the alphabet. See Figure 10.4 for the code to compute both tables (i.e.,   [image]  and d) from the pattern.

bmsearch( text, n, pat, m )  /* Search pat[1..m] in text[1..n]  */

char text[], pat[];

int n, m;

{

int k, j, skip;

int dd[MAX_PATTERN_SIZE], d[MAX_ALPHABET_SIZE];

initd( pat, m, d );  /* Preprocess the pattern */

initdd( pat, m, dd );

k = m; skip = dd[1] + 1;

while( k <= n )  /* Search */

{

j = m;

while( j>0 && text[k] == pat[j] )

{

j--; k--;

}

if( j == 0 )

{

Report_match_at_position( k+1 );

k += skip;

}

else k += max( d[text[k]], dd[j] );

}

}

Figure 10.4: The Boyer-Moore algorithm

Example 3 The d table for the pattern abracadabra is

d['a'] = 0   d['b'] = 2   d['c'] = 6   d['d'] = 4   d['r'] = 1

and the value for any other character is 11.

Both shifts can be precomputed based solely on the pattern and the alphabet. Hence, the space needed is m + c + O(1). Given these two shift functions, the algorithm chooses the larger one. The same shift strategy can be applied after a match. In Knuth et al. (1977) the preprocessing of the pattern is shown to be linear in the size of the pattern, as it is for the KMP algorithm. However, their algorithm is incorrect. The corrected version can be found in Rytter's paper (1980; see Figure 10.5).

initd( pat, m, d ) /* Preprocess pattern of length m : d table */

char pat[];

int m, d[];

{

int k;

for( k=0; k = MAX_ALPHABET_SIZE; k++ ) d[k] = m;

for( k=1; k<=m; k++ ) d[pat[k]] = m-k;

}

initdd( pat, m, dd ) /* Preprocess pattern of length m : dd hat table */

char pat[];

int m, dd[];

{

int j, k, t, t1, q, q1;

int f[MAX_PATTERN_SIZE+1];

for( k=1; k<=m; k++ ) dd[k] = 2*m-k;

for( j=m, t=m+1; j > 0; j--, t-- ) /* setup the dd hat table */

{

f[j] = t;

while( t <= m && pat[j] != pat[t] )

{

dd[t] = min( dd[t], m-j );

t = f[t];

}

}

q = t; t = m + 1 - q; q1 = 1; /* Rytter's correction */

for( j=1, t1=0; j <= t; t1++, j++ )

{

f[j] = t1;

while( t1 >= 1 && pat[j] != pat[t1] ) t1 = f[t1];

}

while( q < m )

{

for( k=q1; k<=q; k++ ) dd[k] = min( dd[k], m+q-k );

q1 = q + 1; q = q + t - f[t]; t = f[t];

}

}

Figure 10.5: Preprocessing of the pattern in the Boyer-Moore algorithm

Knuth et al. (1977) have shown that, in the worst case, the number of comparisons is O(n + rm), where r is the total number of matches. Hence, this algorithm can be as bad as the naive algorithm when we have many matches, namely,   [image] (n) matches. A simpler alternative proof can be found in a paper by Guibas and Odlyzko (1980). In the best case Cn = n/m. Our simulation results agree well with the emprical and theoretical results in the original Boyer-Moore paper (1977). Some experiments in a distributed environment are presented by Moller-Nielsen and Straunstrup (1984). A variant of the BM algorithm when m is similar to n is given by Iyengar and Alia (1980). Boyer-Moore type algorithms to search a set of strings are presented by Commentz-Walter (1979) and Baeza-Yates and Regnier (1990).

To improve the worst case, Galil (1979) modifies the algorithm so that it remembers how many overlapping characters it can have between two successive matches. That is, we compute the length,   [image], of the longest proper prefix that is also a suffix of the pattern. Then, instead of going from m to 1 in the comparison loop, the algorithm goes from m to k, where   [image]  if the last event was a match, or k = 1 otherwise. For example,   [image]  for the pattern ababa. This algorithm is truly linear, with a worst case of O(n + m) comparisons. Recently, Cole (1991) proved that the exact worst case is 3n + O(m) comparisons. However, according to empirical results, as expected, it only improves the average case for small alphabets, at the cost of using more instructions. Recently, Apostolico and Giancarco (1986) improved this algorithm to a worst case of 2n - m + 1 comparisons.

10.5.1 The Simplified Boyer-Moore Algorithm

A simplified version of the Boyer-Moore algorithm (simplified-Boyer-Moore, or SBM, algorithm) is obtained by using only the occurrence heuristic. The main reason behind this simplification is that, in practice, patterns are not periodic. Also, the extra space needed decreases from O(m + c) to O(c). That is, the space depends only on the size of the alphabet (almost always fixed) and not on the length of the pattern (variable). For the same reason, it does not make sense to write a simplified version that uses Galil's improvement because we need O(m) space to compute the length of the overlapping characters. Of course, the worst case is now O(mn), but it will be faster on the average.

10.5.2 The Boyer-Moore-Horspool Algorithm

Horspool (1980) presented a simplification of the Boyer-Moore algorithm, and based on empirical results showed that this simpler version is as good as the original Boyer-Moore algorithm. Moreover, the same results show that for almost all pattern lengths this algorithm is better than algorithms that use a hardware instruction to find the occurrence of a designated character.

Horspool noted that when we know that the pattern either matches or does not, any of the characters from the text can be used to address the heuristic table. Based on this, Horspool (1980) improved the SBM algorithm by addressing the occurrence table with the character in the text corresponding to the last character of the pattern. We call this algorithm the Boyer-Moore-Horspool, or BMH, algorithm.

To avoid a comparison when the value in the table is zero (the last character of the pattern), we define the initial value of the entry in the occurrence table, corresponding to the last character in the pattern, as m, and then we compute the occurence heuristic table for only the first m - 1 characters of the pattern. Formally

d[x] = min{s|s = m or (1    s < m and pattern[m - s] = x)}

Example 4 The d table for the pattern abracadabra is

d['a'] = 3   d['b'] = 2   d['c'] = 6   d['d'] = 4   d['r'] = 1

and the value for any other character is 11.

The code for an efficient version of the Boyer-Moore-Horspool algorithm is extremely simple and is presented in Figure 10.6 where MAX_ALPHABET_SIZE is the size of the alphabet. In this algorithm, the order of the comparisons is not relevant, as noted by Baeza-Yates (1989c) and Sunday (1990). Thus, the algorithm compares the pattern from left to right. This algorithm also includes the idea of using the character of the text that corresponds to position m + 1 of the pattern, a modification due to Sunday (1990). Further improvements are due to Hume and Sunday (1990).

bmhsearch( text, n, pat, m ) /* Search pat[1 . . m] in text[1..n] */

char text[], pat[];

int n, m;

{

int d[MAX_ALPHABET_SIZE], i, j, k, lim;

for( k=0; k<MAX_ALPHABET_SIZE; k++ ) d[k] = m+1; /* Preprocessing  */

for( k=1; k<=m; k++ ) d[pat[k]] = m+1-k;

pat[m+1] = CHARACTER_NOT_IN_THE_TEXT; /* To avoid having code      */

/* for special case n-k+1=m  */

lim = n-m+1;

for( k=1; k <= lim; k += d[text[k+m]] ) /* Searching */

{

i=k;

for( j=1; text[i] == pat[j]; j++ ) i++;

if( j == m+1 ) Report_match_at_position( k );

}

/* restore pat[m+1] if necessary */

}

Figure 10.6: The Boyer-Moore-Horspool-Sunday algorithm

Based on empirical and theoretical analysis, the BMH algorithm is simpler and faster than the SBM algorithm, and is as good as the BM algorithm for alphabets of size at least 10. Also, it is not difficult to prove that the expected shift is larger for the BMH algorithm. Improvements to the BMH algorithm for searching in English text are discussed by Baeza-Yates (1989b, 1989a) and Sunday (1990). A hybrid algorithm that combines the BMH and KMP algorithms is proposed by Baeza-Yates (1989c).

Figure 10.7 shows, for the algorithms studied up to this point, the expected number of comparisons per character for random text with c = 4. The codes used are the ones given in this chapter, except that the Knuth-Morris-Pratt algorithm was implemented as suggested by their authors. (The version given here is slower but simpler.)

Figure 10.7: Expected number of comparisons for random text (c = 4)

10.6 THE SHIFT-OR ALGORITHM

The main idea is to represent the state of the search as a number, and is due to Baeza-Yates and Gonnet (1989). Each search step costs a small number of arithmetic and logical operations, provided that the numbers are large enough to represent all possible states of the search. Hence, for small patterns we have an O(n) time algorithm using O(  [image]    [image]    [image]) extra space and O(m +   [image]    [image]    [image]) preprocessing time, where   [image]  denotes the alphabet.

The main properties of the shift-or algorithm are:

       Simplicity: the preprocessing and the search are very simple, and only bitwise logical operations, shifts, and additions are used.

       Real time: the time delay to process one text character is bounded by a constant.

       No buffering: the text does not need to be stored.

It is worth noting that the KMP algorithm is not a real time algorithm, and the BM algorithm needs to buffer the text. All these properties indicate that this algorithm is suitable for hardware implementation.

This algorithm is based on finite automata theory, as the KMP algorithm, and also exploits the finiteness of the alphabet, as in the BM algorithm.

Instead of trying to represent the global state of the search as previous algorithms do, we use a vector of m different states, where state i tells us the state of the search between the positions 1, . . . , i of the pattern and the positions ( j - i + 1), . . . , j of the text, where j is the current position in the text.

We use one bit to represent each individual state, where the state is 0 if the last i characters have matched, or 1 if not. We can then represent the vector state efficiently as a number in base 2 by

where the si are the individual states. We report a match if sm is 0, or equivalently, if state < 2m-1. That match ends at the current position.

To update the state after reading a new character in the text, we must:

       shift the vector state 1 bit to the left to reflect that we have advanced one position in the text. In practice, this sets the initial state of s1 to be 0 by default.

       update the individual states according to the new character. For this we use a table T that is defined by preprocessing the pattern with one entry per alphabet symbol, and the bitwise operator or that, given the old vector state and the table value, gives the new state.

Each search step is then:

state = (state << 1) or T [curr char]

where << denotes the shift left operation.

The definition of the table T is

for every symbol x of the alphabet, where   [image] (C) is 0 if the condition C is true, and 1 otherwise. Therefore, we need m   [image]    [image]   [image]   [image]  bits of extra memory, but if the word size is at least m, only   [image]   [image]   [image]  words are needed. We set up the table by preprocessing the pattern before the search. This can be done in   [image]  time.

Example 5 Let {a, b, c, d} be the alphabet, and ababc the pattern. The entries for table T (one digit per position in the pattern) are then:

T[a] = 11010   T[b] = 10101   T[c] = 01111   T[d] = 11111

We finish the example by searching for the first occurrence of ababc in the text abdabababc. The initial state is 11111.

text :   a     b     d     a     b     a     b     a     b      c

T[x] : 11010 10101 11111 11010 10101 11010 10101 11010 10101 01111

state: 11110 11101 11111 11110 11101 11010 10101 11010 10101 01111

For example, the state 10101 means that in the current position we have two partial matches to the left, of lengths two and four, respectively. The match at the end of the text is indicated by the value 0 in the leftmost bit of the state of the search.

The complexity of the search time in both the worst and average case is   [image], where   [image]  is the time to compute a shift or other simple operation on numbers of m bits using a word size of w bits. In practice, for small patterns (word size 32 or 64 bits), we have O(n) time for the worst and the average case.

Figure 10.8 shows an efficient implementation of this algorithm. The programming is independent of the word size insofar as possible. We use the following symbolic constants:

       MAXSYM: size of the alphabet. For example, 128 for ASCII code.

       WORD: word size in bits (32 in our case).

       B: number of bits per individual state; in this case, one.

sosearch( text, n, pat, m ) /* Search pat[1..m] in text[1..n] */

register char *text;

char pat[];

int n, m;

{

register char *end;

register unsigned int state, lim;

unsigned int T[MAXSYM], i, j;

char *start;

if( m > WORD )

Abort( "Use pat size <= word size" );

for( i=0; i<MAXSYM; i++ ) T[i] = ~0;        /* Preprocessing */

for( lim=0, j=1, i=1; i<=m; lim |= j, j <<= B, i++ )

T[pat[i]] &= ~j;

lim = ~(lim >>B);

text++; end = text+n+1;                     /* Search */

state = ~0;                                 /* Initial state */

for( start=text; text end; text++ )

{

state = (state <<B) | T[*text];  /* Next state */

if( state < lim ) Report_match_at_position( text-start-m+2 );

}

}

Figure 10.8: Shift-Or algorithm for string matching (simpler version)

The changes needed for a more efficient implementation of the algorithm (that is, scan the text until we see the first character of the pattern) are shown in Figure 10.9. The speed of this version depends on the frequency of the first letter of the pattern in the text. The empirical results for this code are shown in Figures 10.11 and 10.12. Another implementation is possible using the bitwise operator and instead of the or operation, and complementing the value of Tx for all x   [image]    [image].

initial = ~0; first = pat[1]; start = text; /* Search */

do {

state = initial;

do {

state = (state << B) | T[*text]; /* Next state */

if( state < lim ) Report_match_at_position( text-start-m+2 );

text++;

} while( state != initial );

while( text < end && *text != first ) /* Scan */

text++;

} while( text < end );

Figure 10.9: Shift-Or algorithm for string matching

*Based on implementation of Knuth, Morris, and Pratt (1977).

By just changing the definition of table T we can search for patterns such that every pattern position is:

       a set of characters (for example, match a vowel),

       a "don't care" symbol (match any character), or

       the complement of a set of characters.

Furthermore, we can have "don't care" symbols in the text. This idea has been recently extended to string searching with errors and other variants by Wu and Manber (l991).

10.7 THE KARP-RABIN ALGORITHM

A different approach to string searching is to use hashing techniques, as suggested by Harrison (1971). All that is necessary is to compute the signature function of each possible m-character substring in the text and check if it is equal to the signature function of the pattern.

Karp and Rabin (1987) found an easy way to compute these signature functions efficiently for the signature function h(k) = k mod q, where q is a large prime. Their method is based on computing the signature function for position i given the value for position i - 1. The algorithm requires time proportional to n + m in almost all cases, without using extra space. Note that this algorithm finds positions in the text that have the same signature value as the pattern. To ensure that there is a match, we must make a direct comparison of the substring with the pattern. This algorithm is probabilistic, but using a large value for q makes collisions unlikely [the probability of a random collision is O(1 /q)].

Theoretically, this algorithm may still require mn steps in the worst case, if we check each potential match and have too many matches or collisions. In our empirical results we observed only 3 collisions in 107 computations of the signature function, using large alphabets.

The signature function represents a string as a base-d number, where d = c is the number of possible characters. To obtain the signature value of the next position, only a constant number of operations is needed. The code for the case d = 128 (ASCII) and q = 16647133 for a word size of 32 bits, based in Sedgewick's exposition (1983), is given in Figure 10.10 (D = log2 d and Q = q). By using a power of 2 for d (d   [image]  c ), the multiplications by d can be computed as shifts. The prime q is chosen as large as possible, such that (d + 1 )q does not cause overflow. We also impose the condition that d is a primitive root mod q. This implies that the signature function has maximal cycle; that is

Thus, the period of the signature function is much larger than m for any practical case, as shown in Gonnet and Baeza-Yates (1990).

rksearch( text, n, pat, m )  /* Search pat[1..m] in text[1..n] */

char text[], pat[];          /* (0 m = n)                      */

int n, m;

{

int h1, h2, dM, i, j;

dM = 1;

for( i=1; i<m; i++ ) dM = (dM << D) % Q;  /* Compute the signature  */

h1 = h2 = O;                              /* of the pattern and of  */

for( i=1; i<=m; i++ )                     /* the beginning of the   */

{                                         /* text                   */

h1 = ((h1 << D) + pat[i] ) % Q;

h2 = ((h2 << D) + text[i] ) % Q;

}

for( i = 1; i <= n-m+1; i++ )  /* Search */

{

if( h1 == h2 )  /* Potential match */

{

for(j=1; j<=m && text[i-1+j] == pat[j]; j++ );  /* check */

if( j > m )                                /* true match */

Report_match_at_position( i );

}

h2 = (h2 + (Q << D) - text[i]*dM ) % Q;  /* update the signature */

h2 = ((h2 << D) + text[i+m] ) % Q;        /* of the text */

}

}

Figure 10.10: The Karp-Rabin algorithm

In practice, this algorithm is slow due to the multiplications and the modulus operations. However, it becomes competitive for long patterns. We can avoid the computation of the modulus function at every step by using implicit modular arithmetic given by the hardware. In other words, we use the maximum value of an integer (determined by the word size) for q. The value of d is selected such that dk mod 2r has maximal cycle length (cycle of length 2r-2), for r from 8 to 64, where r is the size, in bits, of a word. For example, an adequate value for d is 31.

With these changes, the evaluation of the signature at every step (see Figure 10.10) is

h2 = h2*D - text[j-m]*dM + text[i+m]; /* update the signature value */

and overflow is ignored. In this way, we use two multiplications instead of one multiplication and two modulus operations.

10.8 CONCLUSIONS

We have presented the most important string searching algorithms. Figure 10.11 shows the execution time of searching 1000 random patterns in random text for all the algorithms considered, with c = 30. Based on the empirical results, it is clear that Horspool's variant is the best known algorithm for almost all pattern lengths and alphabet sizes. Figure 10.12 shows the same empirical results as Figure 10.11, but for English text instead of random text. The results are similar. For the shift-or algorithm, the given results are for the efficient version. The results for the Karp-Rabin algorithm are not included because in all cases the time exceeds 300 seconds.

The main drawback of the Boyer-Moore type algorithms is the preprocessing time and the space required, which depends on the alphabet size and/or the pattern size. For this reason, if the pattern is small (1 to 3 characters long) it is better to use the naive algorithm. If the alphabet size is large, then the Knuth-Morris-Pratt algorithm is a good choice. In all the other cases, in particular for long texts, the Boyer-Moore algorithm is better. Finally, the Horspool version of the Boyer-Moore algorithm is the best algorithm, according to execution time, for almost all pattern lengths.

The shift-or algorithm has a running time similar to the KMP algorithm. However, the main advantage of this algorithm is that we can search for more general patterns ("don't care" symbols, complement of a character, etc.) using exactly the same searching time (see Baeza-Yates and Gonnet [1989]); only the preprocessing is different.

Figure 10.11: Simulation results for all the algorithms in random text (c = 30)

The linear time worst case algorithms presented in previous sections are optimal in the worst case with respect to the number of comparisons (see Rivest [1977]). However, they are not space optimal in the worst case because they use space that depends on the size of the pattern, the size of the alphabet, or both. Galil and Seiferas (1980, 1983) show that it is possible to have linear time worst case algorithms using constant space. (See also Slisenko [1980, 1983].) They also show that the delay between reading two characters of the text is bounded by a constant, which is interesting for any real time searching algorithms (Galil 1981). Practical algorithms that achieve optimal worst case time and space are presented by Crochemore and Perrin (1988, 1989). Optimal parallel algorithms for string matching are presented by Galil (1985) and by Vishkin (1985). (See also Berkman et al. [1989] and Kedem et al. [1989].)

Figure 10.12: Simulation results for all the algorithms in English text

Many of the algorithms presented may be implemented with hardware (Haskin 1980; Hollaar 1979). For example, Aho and Corasick machines (see Aoe et al. (1985), Cheng and Fu (1987), and Wakabayashi et al. (1985).

If we allow preprocessing of the text, we can search a string in worst case time proportional to its length. This is achieved by using a Patricia tree (Morrison 1968) as an index. This solution needs O(n) extra space and O(n) preprocessing time, where n is the size of the text. See also Weiner (1973), McCreight (1976), Majster and Reiser (1980), and Kemp et al. (1987). For other kinds of indices for text, see Faloutsos (1985), Gonnet (1983), Blumer et al. (1985; 1987). For further references and problems, see Gonnet and Baeza-Yates (1991).

REFERENCES

AHO, A. 1980. "Pattern Matching in Strings," in Formal Language Theory: Perspectives and Open Problems, ed. R. Book, pp. 325-47. London: Academic Press.

AHO, A., and M. CORASICK. 1975. "Efficient String Matching: An Aid to Bibliographic Search," Communications of the ACM, 18, pp. 333-40.

AOE, J., Y. YAMAMOTO, and R. SHIMADA. 1985. "An Efficient Implementation of Static String Pattern Matching Machines," in IEEE Int. Conf. on Supercomputing Systems, vol. 1, pp. 491-98, St. Petersburg, Fla.

APOSTOLICO, A., and R. GIANCARLO. 1986. "The Boyer-Moore-Galil String Searching Strategies Revisited." SIAM J on Computing, 15, 98-105.

BAEZA-YATES, R. 1989a. Efficient Text Searching. Ph.D. thesis, Dept. of Computer Science, University of Waterloo. Also as Research Report CS-89-17.

BAEZA-YATES, R. 1989b. "Improved String Searching." Software-Practice and Experience, 19(3), 257-71.

BAEZA-YATES, R. 1989c. "String Searching Algorithms Revisited," in Workshop in Algorithms and Data Structures, F. Dehne, J.-R. Sack, and N. Santoro, eds. pp. 75-96, Ottawa, Canada. Springer Verlag Lecture Notes on Computer Science 382.

BAEZA-YATES, R., and G. GONNET. 1989. "A New Approach to Text Searching," in Proc. of 12th ACM SIGIR, pp. 168-75, Cambridge, Mass. (To appear in Communications of ACM).

BAEZA-YATES, R., and M. REGNIER. 1990. "Fast Algorithms for Two Dimensional and Multiple Pattern Matching," in 2nd Scandinavian Workshop in Algorithmic Theory, SWAT'90, R. Karlsson and J. Gilbert. eds. Lecture Notes in Computer Science 447, pp. 332-47, Bergen, Norway: Springer-Verlag.

BAILEY, T., and R. DROMEY. 1980. "Fast String Searching by Finding Subkeys in Subtext." Inf. Proc. Letters, 11, 130-33.

BARTH, G. 1981. "An Alternative for the Implementation of Knuth-Morris-Pratt Algorithm." Inf. Proc. Letters, 13, 134-37.

BERKMAN, O., D. BRESLAUER, Z. GALIL, B. SCHIEBER, and U. VISHKIN. 1989. "Highly Parellelizable Problems," in Proc. 20th ACM Symp. on Theory of Computing, pp. 309-19, Seattle, Washington.

BLUMER, A., J. BLUMER, A. EHRENFEUCHT, D. HAUSSLER, and R. MCCONNELL. 1987. "Completed Inverted Files for Efficient Text Retrieval and Analysis". JACM, 34, 578-95.

BLUMER, A., J. BLUMER D. HAUSSLER, A. EHRENFEUCHT, M. CHEN, and J. SEIFERAS. 1985. "The Smallest Automaton Recognizing the Subwords of a Text. Theorefical Computer Science, 40, 31-55.

BOYER, R., and S. MOORE. 1977. "A Fast String Searching Algorithm." CACM, 20, 762-72.

CHENG, H., and K. FU. 1987. "Vlsi Architectures for String Matching and Pattern Matching." Pattern Recognition, 20, 125-41.

COLE, R. 1991. "Tight Bounds on the Complexity of the Boyer-Moore String Matching Algorithm, 2nd Symp. on Discrete Algorithms," pp. 224-33, SIAM, San Francisco, Cal.

COMMENTZ-WALTER, B. 1979. "A String Matching Algorithm Fast on the Average," in ICALP, Lecture Notes in Computer Science 71, pp. 118-32. Springer-Verlag.

CROCHEMORE, M. 1988. "String Matching with Constraints," in Mathematical Foundations of Computer Science, Carlesbad, Czechoslovakia. Lecture Notes in Computer Science 324. Springer-Verlag.

CROCHEMORE, M., and D. PERRIN. 1988. "Pattern Matching in Strings," in 4th Conference on Image Analysis and Processing, ed. D. Gesu. pp. 67-79. Springer-Verlag.

CROCHEMORE, M., and D. PERRIN. 1989. "Two Way Pattern Matching." Technical Report 98-8, L.I.T.P., University. Paris 7 (submitted for publication).

DAVIES, G., and S. BOWSHER. 1986. "Algorithms for Pattern Matching." Software-Practice and Experience, 16, 575-601.

FALOUTSOS, C. 1985. "Access Methods for Text." ACM C. Surveys, 17, 49-74.

GALIL, Z. 1979. "On Improving the Worst Case Running Time of the Boyer-Moore String Matching Algorithm. CACM, 22, 505-08.

GALIL, Z. 1981. "String Matching in Real Time." JACM, 28, 134-49.

GALIL, Z. 1985. "Optimal Parallel Algorithms for String Matching." Information and Control, 67, 144-57.

GALIL, Z., and J. SEIFERAS. 1980. "Saving Space in Fast String-Matching." SIAM J on Computing, 9, 417 - 38.

GALIL, Z., and J. SEIFERAS. 1983. "Time-Space-Optimal String Matching." JCSS, 26, 280-94.

GONNET, G. 1983. "Unstructured Data Bases or Very Efficient Text Searching," in ACM PODS, vol. 2, pp. 117-24, Atlanta, Ga.

GONNET, G., and R. BAEZA-YATES. 1990. "An Analysis of the Karp-Rabin String Matching Algorithm. Information Processing Letters, pp. 271-74.

GONNET, G. H., and BAEZA-YATES, R. 1991. "Handbook of Algorithms and Data Structures," 2nd Edition, Addison-Wesley.

GUIBAS, L., and A. ODLYZKO. 1980. "A New Proof of the Linearity of the Boyer-Moore String Searching Algorithm. SIAM J on Computing, 9, 672-82.

HARRISON, M. 1971. "Implementation of the Substring Test by Hashing." CACM, 14, 777-79.

HASKIN, R. 1980. "Hardware for Searching Very Large Text Databases," in Workshop Computer Architecture for Non-Numeric Processing, vol. 5, pp. 49-56, California.

HOLLAAR, L. 1979. "Text Retrieval Computers." IEEE Computer, 12, 40-50.

HORSPOOL, R. N. 1980. "Practical Fast Searching in Strings." Software - Practice and Experience, 10, 501-06.

HUME, A., and D. M. SUNDAY. 1991. "Fast String Searching." AT&T Bell Labs Computing Science Technical Report No. 156. To appear in Software-Practice and Experience.

IYENGAR, S., and V. ALIA. 1980. "A String Search Algorithm." Appl. Math. Comput., 6, 123-31.

KARP, R., and M. RABIN. 1987. "Efficient Randomized Pattern-Matching Algorithms. IBM J Res. Development, 31, 249-60.

KEDEM, Z., G. LANDAU, and K. PALEM. 1989. "Optimal Parallel Suffix-Prefix Matching Algorithm and Applications," in SPAA'89, Santa Fe, New Mexico.

KEMP, M., R. BAYER, and U. GUNTZER. 1987. "Time Optimal Left to Right Construction of Position Trees." Acta Informatica, 24, 461-74.

KERNIGHAN, B., and D. RITCHIE. 1978. The C Programming Language. Englewood Cliffs, N.J.: Prentice Hall.

KNUTH, D., J. MORRIS, and V. PRATT. 1977. "Fast Pattern Matching in Strings." SIAM J on Computing, 6, 323-50.

MAJSTER, M., and A. REISER. 1980. "Efficient On-Line Construction and Correction of Position Trees." SIAM J on Computing, 9, 785-807.

MCCREIGHT, E. 1976. "A Space-Economical Suffix Tree Construction Algorithm." JACM, 23, 262-72.

MEYER, B. 1985. "Incremental String Matching." Inf. Proc. Letters, 21, 219-27.

MOLLER-NIELSEN, P., and J. STAUNSTRUP. 1984. "Experiments with a Fast String Searching Algorithm." Inf . Proc . Letters, 18, 129-35.

MORRIS, J., and V. PRATT. 1970. "A Linear Pattern Matching Algorithm." Technical Report 40, Computing Center, University of California, Berkeley.

MORRISON, D. 1968. "PATRICIA-Practical Algorithm to Retrieve Information Coded in Alphanumeric." JACM, 15, 514-34.

RIVEST, R. 1977. "On the Worst-Case Behavior of String-Searching Algorithms." SIAM J on Computing, 6, 669-74.

RYTTER, W. 1980. "A Correct Preprocessing Algorithm for Boyer-Moore String-Searching." SIAM J on Computing, 9, 509-12.

SCHABACK, R. 1988. "On the Expected Sublinearity of the Boyer-Moore Algorithm." SIAM J on Computing, 17, 548-658.

SEDGEWICK, R. 1983. Algorithms. Reading, Mass.: Addison-Wesley.

SLISENKO, A. 1980. "Determination in Real Time of all the Periodicities in a Word." Sov. Math. Dokl., 21, 392-95.

SLISENKO, A. 1983. "Detection of Periodicities and String-Matching in Real Time." J. Sov. Math., 22, 1316-86.

SMIT, G. 1982. "A Comparison of Three String Matching Algorithms." Software-Practice and Experience, 12, 57-66.

SUNDAY, D. 1990. "A Very Fast Substring Search Algorithm." Communications of the ACM, 33(8), 132-42.

TAKAOKA, T. 1986. "An On-Line Pattern Matching Algorithm." Inf. Proc. Letters, 22, 329-30.

VISHKIN, U. 1985. "Optimal Parallel Pattern Matching in Strings." Information and Control, 67, 91-113.

WAKABAYASHI, S., T. KIKUNO, and N. YOSHIDA. 1985. "Design of Hardware Algorithms by Recurrence Relations." Systems and Computers in Japan, 8, 10-17.

WEINER, P. 1973. "Linear Pattern Matching Algorithm," in FOCS, vol. 14, pp. 1-11.

WU, S., and U. MANBER. 1991. "Fast Text Searching With Errors." Technical Report TR-91-11, Department of Computer Science, University of Arizona. To appear in Proceedings of USENIX Winter 1992 Conference, San Francisco, January, 1992.

Go to Chapter 11     Back to Table of Contents


Information Retrieval: CHAPTER 11: RELEVANCE FEEDBACK AND OTHER QUERY

 Return to Table of Contents

CHAPTER 11: RELEVANCE FEEDBACK AND OTHER QUERY MODIFICATION TECHNIQUES

Donna Harman

National Institute of Standards and Technology

Abstract

This chapter presents a survey of relevance feedback techniques that have been used in past research, recommends various query modification approaches for use in different retrieval systems, and gives some guidelines for the efficient design of the relevance feedback component of a retrieval system.

11.1 INTRODUCTION

Even the best of the information retrieval systems have a limited recall; users may retrieve a few relevant documents in response to their queries, but almost never all the relevant documents. In many cases this is not important to users, but in those cases where high recall is critical, users seldom have many ways to retrieve more relevant documents. As a first choice they can "expand" their search by broadening a narrow Boolean query or by looking further down a ranked list of retrieved documents. Often this is wasted effort: a broad Boolean search pulls in too many unrelated documents and the tail of a ranked list of documents contains documents matching mostly less discriminating query terms. The second choice for these users is to modify the original query. Very often, however, this becomes a random operation in that users have already made their best effort at a statement of the problem in the original query and are uncertain as to what modification(s) may be useful.

Users often input queries containing terms that do not match the terms used to index the majority of the relevant documents (either controlled or full text indexing) and almost always some of the unretrieved relevant documents are indexed by a different set of terms than those in the query or in most of the other relevant documents. This problem has long been recognized as a major difficulty in information retrieval systems (Lancaster 1969). More recently, van Rijsbergen (1986) spoke of the limits of providing increasingly better ranked results based solely on the initial query, and indicated a need to modify that query to further increase performance.

The vast majority of research in the area of relevance feedback and automatic query modification has dealt with a ranking model of retrieval systems, although some of this work has been adapted for the Boolean model. This chapter will deal almost exclusively with feedback and query modification based on the ranking model, and readers are referred to a special issue of Information Processing and Management (Radecki 1988) for discussion of relevance feedback and query modification in Boolean models. Because relevance feedback is strongly related to ranking, readers unfamiliar with basic ranking models should read Chapter 14 before this chapter.

Two components of relevance feedback have evolved in research. First, extensive work has been done in the reweighting of query terms based on the distribution of these terms in the relevant and nonrelevant documents retrieved in response to those queries. This work forms the basis of the probabilistic model for ranking (see Chapter 14 on Ranking Algorithms for a description of this model). Specifically it has been shown that query terms appearing in relevant documents should have increased term weights for later searching (and conversely terms in nonrelevant documents should have decreased term weights). The details of this reweighting appear later in this chapter. Note that this is a very similar idea to traditional feedback in cybernetics or in biologically based systems such as neural networks.

A second component of relevance feedback or query modification is based on changing the actual terms in the query. Whereas query reweighting will increase the ranks of unretrieved documents that contain the reweighted terms, it provides no help for unretrieved relevant documents containing terms that do not match the original query terms. Various methods of query expansion have been tried and these methods are discussed later in the chapter.

This chapter on relevance feedback is organized as follows. Section 11.2 is a discussion of the research in the area of relevance feedback and query modification, starting with early research in the SMART environment, discussing problems in evaluation, and then reviewing research using the two different components of relevance feedback: the reweighting of query terms and the addition of new query terms. The final part of section 11.2 describes research experiments involving other types of relevance feedback. Section 11.3 reviews the use of relevance feedback and query modification in operational retrieval systems, mostly in experimental prototypes. Section 11.4 contains recommendations for the use of relevance feedback in Boolean systems, in systems based on vector space search models, and in systems based on probabilistic indexing or ad hoc combinations of term-weighting schemes. Section 11.5 presents some thoughts on constructing efficient relevance feedback operations, an area that has received little actual experimental work, and section 11.6 summarizes the chapter.

11.2 RESEARCH IN RELEVANCE FEEDBACK AND QUERY MODIFICATION

11.2.1 Early Research

In their forward-looking paper published in 1960, Maron and Kuhns mentioned query modification, suggesting that terms closely related to the original query terms can be added to the query and thus retrieve more relevant documents. Relevance feedback was the subject of much experimentation in the early SMART system (Salton 1971). In 1965 Rocchio published (republished as Rocchio 1971) some experiments in query modification that combined term reweighting and query expansion. Based on what is now known as the vector space model (for more on the vector space model, see Chapter 14 on Ranking Algorithms), he defined the modified query to be

where

     Q0 = the vector for the initial query

     Ri = the vector for relevant document i

     Si = the vector for nonrelevant document i

     n1 = the number of relevant documents

     n2 = the number of nonrelevant documents

Q1 therefore is the vector sum of the original query plus the vectors of the relevant and nonrelevant documents. Using this as the basic relation, he suggested possible additional constraints such as extra weighting for the original query, or limits to the amount of feedback from nonrelevant documents. He ran a variation of the basic formula on a very small test collection, constraining the feedback by only allowing terms to be in Q1 if they either were in Q0 or occurred in at least half the relevant documents and in more relevant than nonrelevant documents. The results of these experiments were very positive.

In 1969 Ide published a thesis based on a series of experiments extending Rocchio's work, again using the SMART system (Ide 1971). She not only verified the positive results of Rocchio, but developed three particular strategies that seemed to be the most successful. The first was the basic Rocchio formula, minus the normalization for the number of relevant and nonrelevant documents. The second strategy was similar, but allowed only feedback from relevant documents, and the third strategy (Ide dec hi) allowed limited negative feedback from only the highest-ranked nonrelevant document. She found no major significant difference between performance of the three strategies on average, but found that the relevant only strategies worked best for some queries, and other queries did better using negative feedback in addition.

Although the early SMART work was done with small collections, it provided clear evidence of the potential power of this technique. Additionally it defined some of the major problems in the area of evaluation (Hall and Weiderman 1967; Ide 1969). These are discussed in the next section because the evaluation problem has continued to plague feedback research, with the use of inappropriate evaluation procedures often indicating performance gains using feedback that are unrealistic.

11.2.2 Evaluation of Relevance Feedback

Standard evaluation in information retrieval (Salton and McGill 1983) compares recall-precision figures generated from averaging the performance in individual queries and comparing this averaged performance across different retrieval techniques. If this evaluation method is used in a simplistic manner in comparing the results after one iteration of feedback against those using no feedback, the results generally show spectacular improvement. Unfortunately, a significant part of this improvement results from the relevant documents used to reweight the query terms moving to higher ranks (e.g., documents initially ranked at 1, 4, and 8 moving to ranks 1, 2, and 4). Not only is this an unrealistic evaluation because presumably the user has already seen these documents, but it also masks any real improvement in relevant document ranks below those initially shown the user.

There were several more realistic methods of evaluation tried (Salton 1970) but a de facto standard (the residual collection method) has since been used for most research. In this method the initial run is made and the user is shown the top x documents. These documents are then used for relevance feedback purposes. The evaluation of the results compares only the residual collections, that is, the initial run is remade minus the documents previously shown the user and this is compared with the feedback run minus the same documents. This method provides an unbiased and more realistic evaluation of feedback. However, because highly ranked relevant documents have been removed from the residual collection, the recall-precision figures are generally lower than those for standard evaluation methods and cannot be directly compared with performance as measured by the standard evaluation method.

11.2.3 Research in Term Reweighting without Query Expansion

The probabilistic model proposed by Robertson and Sparck Jones (1976) is based on the distribution of query terms in relevant and nonrelevant documents (for more on the probabilistic model, see Chapter 14). They developed a formula for query term-weighting that relies entirely on this distribution.

where

     Wij = the term weight for term i in query j

       r = the number of relevant documents for query j having term i

       R = the total number of relevant documents for query j

       n = the number of documents in the collection having term i

       N = the number of documents in the collection

Note that this weighting method assumes that all relevant documents are known before a query is submitted, a situation not in itself realistic, but suggestive of a method of relevance feedback after some knowledge is gained about relevance (especially for use in SDI services). Robertson and Sparck Jones used this formula to get significantly better retrieval performance for the manually indexed Cranfield 1400 collection, employing an acceptable alternative method of evaluation (the test and control method).

Sparck Jones (1979a) extended this experiment to a larger collection and again showed significant improvements. She also performed an experiment (1979b) to simulate the use of this relevance weighting formula in an operational relevance feedback situation in which a user sees only a few relevant documents in the initial set of retrieved documents, and those few documents are the only ones available to the weighting scheme. These experiments required adding a constant to the above formula to handle situations in which query terms appeared in none of the retrieved relevant documents. The results from this reweighting with only a few relevant documents still showed significant performance improvements over weighting using only the IDF measure (Sparck Jones 1972; see Chapter 14 on Ranking Algorithms for a definition of IDF), indicating that the probabilistic weighting schemes provide a useful method for relevance feedback, especially in the area of term reweighting.

Croft (1983) extended this weighting scheme both by suggesting effective initial search methods (Croft and Harper 1979) using probabilistic indexing and by adapting the weighting using the probabilistic model to handle within-document frequency weights.

where

       Wijk = the term weight for term i in query j and document k

       IDF1 = the IDF weight for term i in the entire collection

        pij = the probability that term i is assigned within the set of

              relevant documents for query j

       qij = the probability that term i is assigned within the set of

             nonrelevant documents for query j

        r = the number of relevant documents for query j having term i

        R = the total number of relevant documents for query j

        n = the number of documents in the collection having term i

        N = the number of documents in the collection

where

   freqik = the frequency of term i in document k

 maxfreqk = the maximum frequency of any term in document k

Two constants, C and K, allow this scheme to be adjusted to handle different types of data. Croft and Harper (1979) found that for the manually indexed Cranfield collection, C needed to dominate the initial search because the mere assignment of an index term to a document implied the importance of that term to that document. For automatically indexed collections, the importance of a term is better measured by the IDF alone, with C set to 0 for the initial search. Croft (1983) found that the best value for K was 0.3 for the automatically indexed Cranfield collection, and 0.5 for the NPL collection, confirming that within-document term frequency plays a much smaller role in the NPL collection with its short documents having few repeating terms. When K is used in the subsequent feedback runs, the optimum value increases to 0.5 for Cranfield and 0.7 for NPL. This could reflect the improved weighting for term importance within an entire collection using the relevance weighting instead of the IDF, requiring a lesser role for within-document frequency weighting.

The results of this new weighting scheme with optimal C and K settings showed improvements in the automatically indexed Cranfield collection of up to 35 percent over weighting using only the term distribution in the initial set of relevant and nonrelevant documents, although this improvement was somewhat less using the NPL collection where documents are shorter.

It is interesting to note the parallel between the best methods of initial searching and the results of adding within-document frequency weighting to the probabilistic weighting. In the initial searching it was shown (see Chapter 14) that it is important in most situations to weight the terms both by a measure of their importance in a collection (such as using the IDF measure) and by a measure of their importance within a given document. Harman (1986) suggested that these two measures are complementary and that combining them provided improvements roughly analogous to adding the improvements found by using each measure separately. Probabilistic weighting affects only the importance of a term within a collection for a given query; the importance of that term within a given document can only be measured by some function of its frequency within that document, and therefore it should be expected that including both weighting components produces the best results.

Other models of probabilistic indexing have been proposed (or re-evaluated) recently; in particular see Fuhr (1989) and Wong and Yao (1990).

11.2.4 Query Expansion without Term Reweighting

If a query has retrieved no relevant documents or if the terms in the query do not match the terms in the relevant documents not yet retrieved, it becomes critical to expand the query. The early SMART experiments both expanded the query and reweighted the query terms by adding the vectors of the relevant and nonrelevant documents. However, it is possible to expand a query without reweighting terms.

Ideally, query expansion should be done using a thesaurus that adds synonyms, broader terms, and other appropriate words. The manually constructed thesaurus needed for this is seldom available, however, and many attempts have been made to automatically create one. Most of these involve term-term associations or clustering techniques. As these techniques are discussed in Chapter 16 on clustering, only the results will be briefly described here.

Lesk (1969) tried several variations of term-term clustering using the SMART system and had little success. The fact that term co-occurrence is used to generate these thesauri implies that most terms entering the thesaurus occur in the same documents as the seed or initial term (especially in small collections), and this causes the same set of documents to be retrieved using these related terms, only with a higher rank. Not only does this cause little overall improvement, but that improvement is only in the precision of the search rather than in increasing recall.

Sparck Jones and Barber (1971) also tried term-term clustering. Again they did not get much improvement and found that it was critical to limit the query expansion to only those terms strongly connected to initial query terms, with no expansion of high-frequency terms, to avoid any degradation in performance. These results were confirmed by Minker, Wilson, and Zimmerman (1972) who found no significant improvement in relevant document retrieval, with degradations in performance often occurring.

Harman (1988) used a variation of term-term association (the nearest neighbor technique) and found similar results, with no improvement and with less degradation when adding only the top connected terms and no high-frequency terms. She suggested that users could "filter" the new query terms, that is, show users a list of suggested additional query terms and ask them to select appropriate ones. In a simulation assuming a perfect user selection, however, the improvements were only 8.7 percent using the automatically indexed Cranfield collection. By comparison, applying a similar user filtering process with a selection of terms from relevant documents provided an improvement of over 16 percent, almost twice that for term-term clustering techniques.

Because the area of automatic thesaurus building is so intuitively appealing, work still continues. Crouch (1988) presented preliminary results of another term-term correlation study, and Guntzer et al. (1989) is currently working on a semi-automatic method to construct a thesaurus based on actual term usage in queries. Two additional promising methods involve the use of highly structured networks. Belew (1989) described a rich connectionist representation which was modified based on retrieval experience, and Dumais (1990) used an elaborate factor analysis method called latent semantic indexing to expand queries.

It is also possible to provide query expansion using terms from the relevant documents without term reweighting. As mentioned earlier, Harman (1988) produced lists of terms from relevant documents using several methods of ranking those terms. These terms were then automatically added to the query. She found significant differences between the ranking methods, with techniques involving a combination of the normalized noise measure (similar to the IDF; see Chapter 14 on Ranking Algorithms) with the total frequency of the term within the set of relevant documents outperforming simple sorts based only on the number of relevant documents containing the term. She also found that adding too many terms from a sorted list decreased performance, with the best performance for the Cranfield collection occurring after adding about 20 terms, and slightly worse performance both for higher than 20 terms and for only 10 terms. As a third part of the experiment, it was shown that using a simulated perfect user selection from the top 20 terms produced a 31 percent improvement over methods with no user selection. Several on-line retrieval systems have also used this approach of showing users sorted term lists (see section 11.3).

11.2.5 Query Expansion with Term Reweighting

The vast amount of relevance feedback and query expansion research has been done using both query expansion and term-reweighting. The early SMART experiments (see section 11.2.1) added term vectors to effectively reweight the query terms and to expand the query. Researchers using probabilistic indexing techniques for reweighting also tried some specific experiments in query expansion.

Harper and van Rijsbergen (1978) used the probabilistic model to reweight query terms, using all relevant document information for weighting, and expanding the query by adding all terms directly connected to a query term by a maximum spanning tree (MST) technique of term-term clustering. Additionally, they developed a new relevance weighting scheme, the EMIM weighting measure (see Harper 1980 for implementation details of EMIM). This weighting scheme better handles the parameter estimation problem seen in the Robertson-Sparck Jones measure (1976) when a query term appears in no retrieved relevant documents, although it is not as theoretically optimal as the Robertson-Sparck Jones measure. The EMIM weighting scheme with query expansion by MST showed significant improvements (using all relevant documents) over the formula used by Robertson and Sparck Jones. Using only 10 or 20 documents for feedback, the EMIM reweighting with expansion still showed significant improvements over the old formula. It should be noted, however, that two effects are compounded in this experiment: reweighting using an improved weighting scheme (EMIM) and query expansion using a MST technique.

Harper's thesis (1980) elaborated on the area of query expansion and reweighting. Using the EMIM relevance weighting scheme, he tried the same MST expansion on the manually indexed Cranfield 1400 collection, a Cranfield 1400 collection with only titles for documents, and the larger UKCIS2 collection. The improvements for query expansion were much less with the Cranfield titles and the UKCIS2 collection than for the manually indexed Cranfield collection, largely because the MST had much shorter documents to use for expansion. In addition to the expansion using the MST, Harper tried expanding queries using a selection of terms from retrieved relevant documents. He selected these terms by ranking a union of all terms in the retrieved relevant documents using the EMIM measure, and then selecting a given number from the top of this list. He found significant performance improvements for the manually indexed Cranfield collection over that using the MST expansion, but no improvement for the UKCIS2 collection.

Performance improvements for query expansion seem to be heavily dependent on the test collection being used. The Cranfield collection, either as manually indexed, or with automatically indexed abstracts, consistently shows performance improvements for query expansion, either using the MST method or expanding by terms from relevant documents. Collections with short documents such as the Cranfield title collection or the UKCIS collection probably do not provide enough terms to make expansion effective. A further example of this effect is shown in Smeaton and van Rijsbergen (1983), where a very extensive number of different query expansion methods showed no improvements when run using the NPL test collection, a collection having documents indexed by only 20 terms apiece on the average.

Wu and Salton (1981) experimented with term relevance weighting (also known as term precision weighting), a method for reweighting using relevance feedback that is very similar to the Robertson-Sparck Jones formula. They tried reweighting and expanding the query by all the terms from relevant documents. They found a 27 percent improvement in average precision for the small (424 document) Cranfield collection using reweighting alone, with an increase up to 32.7 percent when query expansion was added to reweighting. There was more improvement with the Medlars 450 collection, with 35.6 percent improvement for reweighting alone and 61.4 percent for both reweighting and expansion.

A recent paper by Salton and Buckley (1990) compared twelve different feedback procedures, involving two different levels of query expansion, across six different test collections. The standard three-vector feedback methods were used, in addition to other probabilistic feedback methods. Concentrating only on the vector feedback methods, the three they used were as follows.

where

        Q0 = the vector for the initial query

        Ri = the vector for relevant document i

        Si = the vector for nonrelevant document i

        n1 = the number of relevant documents

        n2 = the number of nonrelevant documents

The basic operational procedure in these three methods is the merging of document vectors and original query vectors. This automatically reweights query terms by adding the weights from the actual occurrence of those query terms in the relevant documents, and subtracting the weights of those terms occurring in the nonrelevant documents. Queries are automatically expanded by adding all the terms not in the original query that are in the relevant documents and nonrelevant documents, using both positive and negative weights based on whether the terms are coming from relevant or nonrelevant documents (no new terms are actually added with negative weights; the contribution of nonrelevant document terms is to modify the weighting of new terms coming from relevant documents). The Ide dec-hi method only uses the top nonrelevant document for feedback, instead of all nonrelevant documents retrieved within the first set shown the user. The Rocchio method both allows adjustment for the relative input of relevant and nonrelevant documents and uses a weighting scheme based on a normalized version of the document weights rather than the actual document weights themselves.

The results favored the Ide dec-hi method for all six collections, although the other methods follow closely behind. Setting [image] = 0.25 and [image] = 0.75 for the Rocchio method created the best results, limiting the effects of negative feedback. These results confirm earlier work on the smaller collections by the SMART project in which Ide found no major significant difference between performance of the three strategies on average, but found that the "relevant only" strategies worked best for some queries, and other queries did better using negative feedback in addition.

Two types of document weighting were used in these experiments: a binary weighting and weighting done using the best SMART weighting schemes involving a combination of within-document frequencies of terms and the IDF measure for these terms. It was concluded that using appropriate term weighting for the documents (as opposed to binary weighting schemes) is important for most collections. This agrees with Croft's results (1983) using within-document frequencies in addition to reweighting by term distribution in relevant and nonrelevant documents. They also found differences in performance across test collections and suggested that the types of data to be used in retrieval should be examined in selecting feedback and query expansion mechanisms (see Section 4 on Recommendations for Use of Relevance Feedback).

11.2.6 Other Research in Relevance Feedback

Noreault (1979) tried an entirely different approach to relevance feedback. After users had selected a relevant document(s), he used this (these) document(s) as a "new query," effectively ranking (using the cosine correlation as implemented in the SIRE system) all the documents in the collection against this document(s). The top 30 of these retrieved documents were then added to those initially selected by the original query. He found on average that this added 5.1 new relevant documents per query. This feedback method would work with any type of retrieval system such as Boolean systems, provided some method existed for selecting related documents, as the query itself is not modified during the retrieval process.

Attar and Fraenkel (1981) used an approach based on local feedback only. They produced an ordered list of the terms in all the documents retrieved in a first iteration search, with the list ordered by the frequency of a term in that set and by its "distance" from the initial query. These terms were shown to a user for selection as new query terms or sifted by an automatic procedure for addition to the query. They found that both expert and nonexpert users could correctly select terms from the suggested list, but that their automatic sifting mechanism could not do this well. Note that terms are selected from all retrieved documents, not just relevant ones, so that this technique could be used as a feedback technique for queries retrieving no relevant documents on the first iteration.

Dillon et al. (1983) used a hybrid approach to provide relevance feedback to a Boolean environment. They devised a new query based only on terms from previously retrieved documents, with terms weighted by a formula similar to the term precision formula used by Salton (but with significant differences). The weighted terms were not used, however, to produce a ranked list of retrieved documents, but were used to automatically construct a revised Boolean query. Results suggest that this method can be effective if very careful attention is paid to the construction of the revised Boolean query.

11.3 USE OF RELEVANCE FEEDBACK IN ON-LINE RETRIEVAL SYSTEMS

One of the earliest (and few) applications of relevance feedback in an operational system was by Vernimb (1977) in the European Nuclear Documentation System (ENDS). This system has an automatic procedure based on a user/system dialogue, removing the need for a user to deal with a Boolean retrieval system. The starting position for this approach is at least two relevant documents, either obtained by a manual search, or by any type of search using a Boolean or ranking retrieval system. The ENDS system then automatically constructs a Boolean query using an elaborate procedure based on the co-occurrence of terms in these known relevant documents, and the user is shown a new set of documents for another iteration of relevance judgments and automatic query adjustments. This procedure is further modified by a term-weighting scheme based on the SMART model that allows the list of documents shown the user to be ranked based on occurrences of terms in the set of relevant documents versus the set of nonrelevant documents. The users of this system found it very satisfactory, and some limited experimental results showed that its performance was considerably better than traditional procedures (such as traditional Boolean retrieval).

Another type of man-machine dialogue using relevance feedback was developed by Oddy (1977) in his THOMAS system. Although this system was purely experimental, it was designed to be a prototype system for the end-user and therefore resembles an on-line retrieval system more than a system designed for retrieval experiments. In this system the user inputs a natural language phrase or sentence and the system returns a single reference based on a search in the concept network (a network built by the MEDUSA system at the University of Newcastle for access to MEDLARS records). This reference contains both a list of authors of the reference and a list of index terms associated with that reference. The user is asked to not only judge the relevance of that reference, but to select index terms that are relevant (or optionally insert new index terms). The query is appropriately modified by the system and a new reference is sought. This type of query modification allows users to browse the references and, although it is not as automatic as the ENDS system, it gives a user more explicit control over the browsing process.

The CUPID system (Porter 1982) implemented the probabilistic retrieval model at the University of Cambridge. This system takes a natural language query, returns a ranked list of titles, and asks the user for relevance judgments. These judgments are used for two distinct purposes: (1) to reweight the original query terms using the Robertson-Sparck Jones reweighting formula, and (2) to select a ranked set of terms from the relevant documents to submit to the user for possible addition to the query. This process can be iterated repeatedly until the user has sufficient information. An improved version of CUPID, the MUSCAT system, is currently running at the Scott Polar Research Institute, Cambridge (Porter and Galpin 1988).

Doszkocs's CITE system (1982) was designed as an interface to MEDLINE and was used as an on-line catalog system (Doszkocs 1983). It combined a ranking retrieval method (see Chapter 14 for more on the CITE system) with an effective search strategy for the huge document files and the controlled and full text vocabulary used in MEDLINE. CITE returned a list of ranked references based on a user's natural language query, and asked for relevance judgments from the user. The system used these judgments to display a ranked list of medical subject headings (ranked by a frequency analysis of their use in the retrieved documents). The user then selected terms for adding to the query and the search was repeated.

Cirt, a front end to a standard Boolean retrieval system, uses term-weighting, ranking, and relevance feedback (Robertson et al. 1986). The system accepts lists of terms without Boolean syntax and converts these terms into alternative Boolean searches for searching on the Boolean system. The user is then shown sets of documents, defined by Boolean combinations of terms, in rank order as specified by a special weighting and ranking algorithm (Bovey and Robertson 1984). If relevance judgments are made, feedback takes the form of appropriately reweighting terms to produce different ranked orders of documents.

Relevance feedback plays a crucial role in the Connection Machine retrieval system (Stanfill and Kahle 1986). Here a query is automatically expanded by adding all the terms in the relevant documents, causing very large queries that can take advantage of the speed offered by the massively parallel architecture. This system relies heavily on co-occurrence of terms and therefore needs large numbers of query terms to work effectively.

Two approaches for query expansion produce lists of related terms in ranked order for user selection and addition to the query. The first approach, the Associative Interactive Dictionary on the TOXLINE system (Doszkocs 1978), does a statistical analysis of the terms appearing in relevant documents in comparison to their appearance in all documents, and displays a list of terms ranked on these statistics. The second, the ZOOM facility of ESA-IRS Online search (Ingwersen 1984), generalizes the first approach to deal with a complete set of retrieved documents. It analyzes the frequency of single words, phrases, or codes appearing in the set of documents, and displays these in order of highest frequency of occurrence for user selection.

11.4 RECOMMENDATIONS FOR USE OF RELEVANCE FEEDBACK

11.4.1 Relevance Feedback Methodologies

It should be stated again that relevance feedback and/or query modification are not necessary for many queries, are of marginal use to many users, and may possibly be of marginal use in some operational systems. Often users are only interested in an "answer" to their question (such as a paragraph in an on-line manual), or in a single good bibliographic reference for introduction to a new area of interest. Some users may never be interested in a high-recall search; others may not be willing to spend any extra effort in retrieving more relevant documents or may be unaware of the low recall of their initial search. An experiment by Fox (1987) showed little use of relevance feedback in a student experiment, probably because it was "not explained adequately" and because the nine test queries did not need further modification.

Some situations call for high recall, however. In a test of a large operational full-text retrieval system (Blair and Maron 1985), only about 20 percent of the relevant documents were retrieved, even though the users thought they had retrieved most of the relevant documents (and felt they needed all the relevant documents).

If a decision is made to offer relevance feedback or some type of query expansion technique, then the appropriate method depends on two issues: the type of retrieval system being used for initial searching, and the type of data that is being used. Three basic retrieval systems are being addressed here: Boolean-based systems, systems based on ranking using a vector space model, and systems based on ranking using either an ad hoc combination of term-weighting schemes or using the probabilistic indexing methods. In terms of data characteristics, two characteristics have been found to be important experimentally: the length of the documents (short or not short), and the type of indexing (controlled or full text). Possibly other factors exist, but there is a lack of direct experimentation about these issues.

Feedback in Boolean retrieval systems

If a Boolean retrieval system is being used, two options seem feasible. The most difficult option would be to modify the system to specially handle relevance feedback either by designing frontends to automatically construct modified Boolean queries (such as was done by Dillon, Vernimb, and the Cirt system), or to graft a ranking system onto the Boolean system to handle all but the initial query (such as done by Noreault). Front-end construction or specially modified Boolean systems (see again the special issue of Information Processing and Management, [Radecki 1988]) require major modifications to the entire system and can be difficult to tune. However, they offer the greatest improvement in performance.

The second option would be to produce ranked lists of terms for user selection (such as the ZOOM feature on ERA-IRS [Ingwersen 1984]). This option leaves the Boolean mechanism intact, but improves the query by suggesting alternative terms and by showing the user the term distribution in the retrieved set of documents. The ranking of the list could be done by term frequency within the entire retrieved set (the simplest method) or by more complex methods taking account of term distribution within only the relevant documents (such as Doszkocs's AID system [Doszkocs 1978]). The first method would be useful in helping a user construct a better query by pointing out the term distribution in the entire retrieved document set (relevant and nonrelevant), and the second method would help a user zero in on new terms by narrowing the term list to only those terms in the relevant documents. Note that the effects of the type of data being used for retrieval are minimal for this type of feedback as user selection is providing the term filter rather than automatic term distribution statistics.

Feedback in retrieval systems based on the vector space model

If the retrieval system is based on the vector space model using a cosine correlation as a ranking measure, then the feedback methods developed in the SMART project are appropriate. As discussed earlier, the vector space model automatically combines the two components of feedback, term reweighting and query expansion, making implementation of relevance feedback more straightforward using this method but somewhat less flexible. Experiments showed that the Ide dec-hi method is the best general purpose feedback technique for the vector space model. It uses minimal negative feedback and can be implemented without concern for deciding parameter settings.

where

        Q0 = the vector for the initial query

        Ri = the vector for relevant document i

         S = the vector for top nonrelevant document

The use of normalized vector space document weighting is highly recommended for most data rather than binary weighting schemes (see section 14.5 of Chapter 14 for details on these weighting schemes). The only decision to be made using this type of relevance feedback is how much query expansion to allow. Since results suggest that query expansion using all terms from the retrieved relevant documents may be only slightly better than a selection of those terms (Salton and Buckley 1990), it is recommended that only a limited number of terms be added (mainly to improve response time), and that those be chosen based on the criteria presented in that paper. This means that whereas all query terms will be reweighted during query modification, the terms to be added to the query are automatically pulled from a sorted list of new terms taken from relevant documents. These terms are sorted by their total frequency within all retrieved relevant documents, and the number of terms added is the average number of terms in the retrieved relevant documents. A possible alternative to this selection method would be to substitute user selection from the top of the list rather than adding a fixed number of terms.

Feedback in retrieval systems based on other types of statistical ranking

If the retrieval system is based on either the probabilistic indexing model or an ad hoc combination of term-weighting schemes, then the term-weighting and query expansion can be viewed as two separate components of feedback, with no specific relationship.

The term-weighting schemes should be carefully examined to determine the correct method for implementing reweighting based on relevance feedback. As noted earlier, the reweighting schemes only affect the weighting based on the importance of a term within an entire collection (global importance). Term-weighting based on term importance within a given document (local importance, such as those based on normalized within-document frequencies) should not be changed; nor should term-weightings based on document structure or other such considerations. The approach taken by Croft (1983) and described earlier in section 11.2.3 is the recommended model to follow in reweighting methodology, in which he simply replaced the portion of the term-weighting based on global term importance (the IDF in this case) with a revised version of the Robertson-Jones weighting formula.

where

Wijk = the term weight for term i in query j and document k

IDFi = the IDF weight for term i in the entire collection

 pij = the probability that term i is assigned within the set of relevant

       documents for query j

 qij = the probability that term i is assigned within the set of nonrelevant

       documents for query j

   r = the number of relevant documents for query j having term i

   R = the total number of relevant documents for query j

   n = the number of documents in the collection having term i

   N = the number of documents in the collection

where

  freqik = the frequency of term i in document k

maxfreqk = the maximum frequency of any term in document k

Two adjustable constants, C and K, provide methods for adjusting the weighting scheme for different types of data. Based on the experiments previously reported, a reasonable estimate for C would be 0 for automatically indexed collections or for feedback searching. This allows the IDF or the relevance weighting to be the dominant factor. Only for manually indexed collections should C be set higher to allow the mere existence of a term within a document to carry more weight. The setting of the constant K should be around 0.3 for the initial search of regular length documents (i.e., documents having many multiple occurrences of a term), rising to around 0.5 for feedback searches. This allows the within-document frequency to play a large role in the initial search, and a somewhat diminished role in subsequent searches, where it is assumed that the relevance weighting becomes more and more important. For short documents, the within-document frequency plays a minimal role and should be either removed from the scheme (such as suggested by Salton and Buckley [1990]) or downweighted by setting K from 0.5 to 0.7.

If other weighting schemes based on criteria such as document structure have been used, then it is likely that the overall importance of these schemes in the total weighting needs to be modified in a manner similar to the downweighting of document frequency weights using the constant K. This will depend on the relative importance of these additional weighting schemes with respect to the importance of matching a given keyword, and will require some experimentation in order to properly introduce relevance feedback to these schemes.

The weighting scheme above only affects the ranks of documents containing terms originally in the query. To be truly effective, some type of query expansion needs to be done. Two alternative sources exist for new query terms. First, the query could be expanded by maximal spanning tree or nearest neighbor techniques (or other automatic thesaurus methods), offering users a selection of terms that are the terms most closely related to the individual query terms. These terms should be in a single list, grouped by expanded terms for each query term, with users selecting appropriate terms for addition to the query.

Although this expansion method works even in situations with no relevant documents retrieved, it has been shown that query expansion by related terms is not as effective as expansion by terms from relevant documents. The second alternative method is therefore to present users with a sorted list of terms from the relevant documents, with the list sorted on the total frequency of the given term within the entire set of retrieved relevant documents (possibly modified by the IDF or normalized noise measure of the term), with users selecting appropriate terms for addition to the query.

Whereas user selection is strongly recommended, it would be possible to use either of these query expansion techniques automatically, adding either the top 20 or so terms from a sorted list of terms from the relevant documents, or adding the closest related term using the automatic thesaurus method for each input query term (see Harman 1992).

A somewhat different approach to user term selection is to get more initial input from the user, such as possible related terms, relative importance of query terms, and important grouping of those terms (phrases). Croft and Das (1990) showed that using this technique in combination with relevance feedback with user selected terms improved results even more than feedback alone, with only a 35 percent overlap in terms found in relevant documents with those suggested by the users initially.

Care should be taken when using either of the two query expansion methods on collections of short documents such as ones with less than 25 index terms. User selection becomes even more critical in this situation, as it has been shown in the past that these expansion methods are unreliable for short documents, such as titles or very short abstracts.

11.5 SOME THOUGHTS ON THE EFFICIENT IMPLEMENTATION OF RELEVANCE FEEDBACK OR QUERY MODIFICATION

Because of the lack of use of relevance feedback in operational systems in general, little work has been done in achieving efficient implementations for large data sets of the recommended feedback algorithms discussed in section 11.4. This section therefore contains only guidelines for efficient implementations, rather than descriptions of actual implementations that have been fully tested. The first part of the section lists the data and data structure needs for relevance feedback, with discussions of alternative methods of meeting these needs, and the second part of the section contains a proposal expanding the basic ranking system described in section 14.6 of Chapter 14 to include relevance feedback and other query modification.

11.5.1 Data and Structure Requirements for Relevance Feedback and Query Modification

The major data needed by relevance feedback and other query modification techniques is a list of the terms contained in each retrieved document. These terms become input to structures designed for reweighting of query terms and for collecting lists of terms for query expansion. Unfortunately, most of the retrieval systems described in this chapter and Chapter 14 involve the use of inverted files for fast response time. This means that the index lists the documents containing each term, but not the terms contained in each document. For large data sets, this inverted file is the only supplemental file stored because of space issues. The input text is kept for display purposes only, with no lists being kept of all the significant words in each document.

For small collections, lists of the terms within each document can be kept (see Porter [1988] for a description of the necessary structures used in the MUSCAT system). For larger data sets, an alternative method would be to parse the retrieved documents in the background while users are looking at the document titles. Although this may be time consuming, the prototype experience of Harman and Candela (1990) using a background secondary string search on retrieved records indicates that this additional work can be accomplished with minimal effect on response time. Even a slight response time delay for feedback may be preferable to the high overhead of storage needed for the lists of terms in a per document order.

Assuming that these term lists are available, either through past storage or through on-line parsing, a structure needs to be built to merge a list of all terms in relevant documents, keeping track of the total number of postings (documents containing one or more occurrences of the term), and the total frequency of terms within all relevant documents. If negative feedback is used such as in Rocchio's algorithm, then an additional list needs to be kept for nonrelevant documents. These lists could be implemented as sets of sorted lists using some type of binary search and insert algorithm, or using an efficient binary tree method such as the right-threaded tree used in the indexing method described in Chapter 3 on Inverted Files. This structure is the major data structure needed for query term reweighting and for production of lists of sorted terms for query expansion. Suggestions for the use of this structure are given in section 11.5.2.

If some type of automatic thesaurus method is to be used in query expansion, then additional storage may be needed. This storage could be minimized by attaching the closest related term to each term in the dictionary portion of the inverted file (or the closest two or three terms). Alternatively, there could be extensive structures needed for more sophisticated methods using complex interrelationships.

11.5.2 A Proposal for an Efficient Implementation of Relevance Feedback

This proposal is based on the implementation of the ranking system as described in section 14.6 of Chapter 14. It should be made clear that the modifications for feedback suggested in this section have not been implemented and therefore should be used only as an illustration of implementations rather than as thoroughly tested techniques. Nevertheless, the issues discussed in this section apply to any efficient implementation of relevance feedback.

The inverted file as shown in Figure 11.1 is the same as that shown in Figure 14.3 in Chapter 14, with two modifications: the terms are complete words rather than stems, and there are two additional fields holding closely related terms. These modifications allow query expansion using some type of automatic thesaurus such as a nearest neighbor algorithm.

Figure 11.1: Dictionary and Postings File with Related Terms

The use of the full terms (as compared to stems only) is necessary because of the addition of the related terms. Stems could be used if some extension to the basic retrieval system to permit the use of both stems and full terms were in effect. See section 14.7.1 of Chapter 14 for an example of such a system and form a natural extension of Figure 14.5 of that chapter to include related terms. The inverted file would be constructed exactly as described in section 14.6 of that chapter, with the two related terms being added using some type of term-term relating algorithm (see Chapter 16 on clustering for details of these algorithms).

The searching itself would proceed as described in the basic retrieval system. After a ranked list of documents are retrieved, users would have an option of doing relevance feedback rather than submitting a new query. If the user selects to start relevance feedback, he or she would be asked to mark retrieved documents that are relevant and to indicate to the system the last document scanned during retrieval. The system would start building the merged list of terms previously described in the background as soon as the marking starts. After the user indicates that all relevant documents have been marked, the system recomputes the weights of the original query terms based on the reweighting schemes recommended in section 11.4. Additionally, the query is expanded by this merged term list, either by automatically adding the terms with the highest "rank" in the list (again see section 11.4 for recommended methods of sorting the list) or preferably by showing the user the sorted list and adding only those terms selected by the user. If no relevant documents are indicated, no reweighting is done, and the term list shown to the user is the list of related terms based on those previously stored in the inverted file. The expanded and reweighted query is run as if it were an initial query, but all documents marked as scanned by the user do not appear in the revised retrieved list.

Two further modifications are necessary to the basic searching routine to allow feedback. First, the weights stored in the postings must be only the normalized document frequency weights (option 2 in section 14.6.1 of Chapter 14) rather than the combined weighting using the normalized document frequency weights and the IDF. This permits the IDF to be pulled from the inverted file on the initial search and combined with the normalized document frequency weight, but allows the new relevance weights to be combined with the normalized document frequency weights on subsequent searches.

The second modification is not strictly necessary but probably would ensure adequate response time. The basic retrieval algorithm is time-dependent on the number of query terms. As the number of query terms grows large, the number of retrieved documents also grows large and the final sort for the list of documents becomes very time-consuming. The pruning modifications described in section 14.7.5 of Chapter 14 become critical to good performance for long queries. Some of the more elaborate pruning methods described in that section may be even more appropriate as they eliminate even the need to retrieve some documents.

11.6 SUMMARY

This chapter has presented a survey of relevance feedback techniques both for reweighting query terms and for adding new query terms. Recommendations and guidelines for building relevance feedback components have been given. More experimentation in the use of relevance feedback in operational systems is needed to verify the guidelines in this proposal. Relevance feedback has clearly proven itself in the laboratory and now needs only efficient implementations to hasten its use in operational systems.

REFERENCES

ATTAR, R., and A. S. FRAENKEL. 1981. "Experiments in Local Metrical Feedback in Full-Text Retrieval Systems." Information Processing and Management, 17(3), 115-26.

BLAIR, D. C., and M. E. MARON. 1985. "An Evaluation of the Retrieval Effectiveness of a Full-Text Document-Retrieval System." Communications of the ACM, 28(3), 289-311.

BELEW, R. K. 1989. "Adaptive Information Retrieval: Using a Connectionist Representation to Retrieve and Learn About Documents." Paper presented at ACM Conference on Research and Development in Information Retrieval, Boston, Mass.

BOVEY, J. D., and S. E. ROBERTSON. 1984. "An Algorithm for Weighted Searching on a Boolean System." Information Technology: Research and Development, 3(1), 84-87.

CROFT, W. B. 1983. "Experiments with Representation in a Document Retrieval System." Information Technology: Research and Development, 2(1), 1-21.

CROFT, W. B., and R. DAS. 1990. "Experiments in Query Acquistion and Use in Document Retrieval Systems." Paper presented at 13th International Conference on Research and Development in Information Retrieval, Brussels, Belgium.

CROFT, W. B., and D. J. HARPER. 1979. Using Probabilistic Models of Document Retrieval Without Relevance Information. Journal of Documentation, 35(4), 285-95.

CROUCH, C. J. 1988. "A Cluster-Based Approach to Thesaurus Construction." Paper presented at ACM Conference on Research and Development in Information Retrieval, Grenoble, France.

DILLON, M., J. ULMSCHNEIDER, and J. DESPER. 1983. "A Prevalence Formula for Automatic Relevance Feedback in Boolean Systems." Information Processing and Management, 19(1), 27-36.

DOSZKOCS, T. E. 1978. "AID, an Associative Interactive Dictionary for Online Searching." Online Review 2(2), 163-74.

DOSZKOCS, T. E. 1982. "From Research to Application: The CITE Natural Language Information Retrieval System," in Research and Development in Information Retrieval, eds. G. Salton and H. J. Schneider, pp. 251-62. Berlin: Springer.

DUMAIS, S. T. 1990. "Enhancing Performance in Latent Semantic Indexing (LSI) Retrieval." Unpublished manuscript.

FOX, E. A. 1987. "Testing the Applicability of Intelligent Methods for Information Retrieval." Information Services & Use 7, 119-38.

FUHR, N. 1989. "Models for Retrieval with Probabilistic Indexing." Information Processing and Management, 25(1), 55-72.

GUNTZER, U., G. JUTTNER, G. SEEGMULLER, and F. SARRE. 1989. "Automatic Thesaurus Construction by Machine Learning from Retrieval Sessions." Information Processing and Management, 25(3), 265-73.

HALL, H., and N. WEIDERMAN. 1967. "The Evaluation Problem in Relevance Feedback." Report No. ISR-12 to the National Science Foundation from Department of Computer Science, Cornell University, June.

HARMAN, D. 1986. "An Experimental Study of Factors Important in Document Ranking." Paper presented at ACM Conference on Research and Development in Information Retrieval, Pisa, Italy.

HARMAN, D. 1988. "Towards Interactive Query Expansion." Paper presented at ACM Conference on Research and Development in Information Retrieval, Grenoble, France.

HARMAN, D., and G. CANDELA. 1990. "Retrieving Records from a Gigabyte of Text on a Minicomputer Using Statistical Ranking." Journal of the American Society for Information Science, 41(8), 581-89.

HARMAN, D. 1992. "Relevance Feedback Revisited." Paper presented at ACM Conference on Research and Development in Information Retrieval, Copenhagen, Denmark.

HARPER, D. J. 1980. Relevance Feedback in Document Retrieval Systems: An Evaluation of Probabilistic Strategies. Doctoral dissertation, Jesus College, Cambridge, England.

HARPER, D. J., and C. J. VAN RIJSBERGEN. 1978. "An Evaluation of Feedback in Document Retrieval Using Co-Occurrence Data." Journal of Documentation, 34(3), 189-216.

IDE, E. 1969. "Relevance Feedback in an Automatic Document Retrieval System." Report No. ISR-15 to National Science Foundation from Department of Computer Science, Cornell University, January.

IDE, E. 1971. "New Experiments in Relevance Feedback," in The SMART Retrieval System, ed. G. Salton, pp. 337-54. Englewood Cliffs, N.J.: Prentice Hall.

INGWERSEN, P. 1984. "A Cognitive View of Three Selected Online Search Facilities." Online Review 8(5), 465-92.

LANCASTER, F. W. 1969. "MEDLARS: Report on the Evaluation of Its Operating Efficiency." American Documentation, 20(2) 119-48.

LESK, M. E. 1969. "Word-Word Associations in Document Retrieval Systems." American Documentation, 20(1), 8-36.

MARON, M. E., and J. L. KUHNS. 1960. "On Relevance, Probabilistic Indexing and Information Retrieval." Association for Computing Machinery, 7(3), 216-44.

MINKER, J., G. A. WILSON, and B. H. ZIMMERMAN. 1972. "An Evaluation of Query Expansion by the Addition of Clustered Terms for a Document Retrieval System." Information Storage and Retrieval, 8(6), 329-48.

NOREAULT, T. 1979. User Directed Relevance Feedback. Doctoral dissertation, School of Information Studies, Syracuse University.

ODDY, R. N. 1977. "Information Retrieval Through Man-Machine Dialogue." Documentation, 33(1), 1-14.

PORTER, M. F. 1982. "Implementing a Probabilistic Information Retrieval System." Information Technology: Research and Development, 1(2), 131-156.

PORTER, M. F., and V. GALPIN. 1988. "Relevance Feedback in a Public Access Catalogue for a Research Library: Muscat at the Scott Polar Research Institute." Program, 22(1), 1-20.

RADECKI, T. 1988. "Improvements to Conventional Boolean Retrieval Systems." Information Processing and Management, 24(3), 513-23.

ROCCHIO, J. J. 1971. "Relevance Feedback in Information Retrieval." In Salton G. (Ed.), The SMART Retrieval System (pp. 313-23). Englewood Cliffs, N.J.: Prentice Hall, Inc.

ROBERTSON, S. E., and K. SPARCK JONES 1976. "Relevance Weighting of Search Terms." J. of the American Society for Information Science, 27(3), 129-46.

ROBERTSON, S. E., C. L. THOMPSON, M. J. MACASKILL, and J. D. BOVEY, 1986. "Weighting, Ranking and Relevance Feedback in a Front-end System." Journal of Information Science, 12, 71-5.

SALTON, G. 1970. "Evaluation Problems in Interactive Information Retrieval." Information Storage and Retrieval, 6(1), 29-44.

SALTON, G. 1971. The SMART Retrieval System. Englewood Cliffs, N.J.: Prentice Hall, Inc.

SALTON, G., and C. BUCKLEY 1990. "Improving Retrieval Performance by Relevance Feedback." Journal of the American Society for Information Science, 41(4), 288-97.

SALTON, G., and M. MCGILL 1983. Introduction to Modern Information Retrieval. New York: McGraw-Hill.

SMEATON, A. F., and C. J. VAN RIJSBERGEN 1983. "The Retrieval Effects of Query Expansion on a Feedback Document Retrieval System." The Computer Journal, 26(3), 239-46.

SPARCK JONES, K. 1972. "A Statistical Interpretation of Term Specificity and Its Application in Retrieval." Documentation, 28(1), 11 -21.

SPARCK JONES, K. 1979a. "Experiments in Relevance Weighting of Search Terms." Information Processing and Management, 15(3), 133-44.

SPARCK JONES, K. 1979b. "Search Term Relevance Weighting Given Little Relevance Information." Documentation, 35(1), 30-48.

SPARCK JONES, K., and E. O. BARBER. 1971. "What Makes an Automatic Keyword Classification Effective." J. American Society for Information Science, 22(3), 166-75.

STANFILL, C., and B. KAHLE. 1986. "Parallel Free-Text Search on the Connection Machine System." Communications of the ACM, 29(12), 1229-39.

VAN RIJSBERGEN, C. J. 1986. "A New Theoretical Framework For Information Retrieval." Paper presented at ACM Conference on Research and Development in Information Retrieval, Pisa, Italy.

VERNIMB, C. 1977. "Automatic Query Adjustment in Document Retrieval." Information Processing and Management, 13(6), 339-53.

WONG, S. K. M., and Y. Y. YAO. 1990. "Query Formulation in Linear Retrieval Models." J. American Society for Information Science, 41(5), 324-29.

WU, H., and G. SALTON. 1981. "The Estimation of Term Relevance Weights using Relevance Feedback." Documentation, 37(4), 194-214.

Go to Chapter 12 Back to Table of Contents


Information Retrieval: CHAPTER 12: BOOLEAN OPERATIONS

 Return to Table of Contents

CHAPTER 12: BOOLEAN OPERATIONS

Steven Wartik

Software Productivity Consortium

Abstract

This chapter presents an overview of Boolean operations, which are one means of expressing queries in information retrieval systems. The concepts of Boolean operations are introduced, and two implementations based on sets are given. One implementation uses bit vectors; the other, hashing. The relative performance characteristics of the approaches are shown.

12.1 INTRODUCTION

This chapter explores one aspect of how information retrieval systems efficiently process user requests. Most systems rely heavily on the ability to perform Boolean operations. User requests are typically phrased in terms of Boolean operations. Hence, the ability to translate them into a form that may be speedily evaluated is of paramount importance. Techniques for doing so form the subject of this chapter.

Information retrieval systems manage documents. These differ from the data stored in traditional database management systems in that they have much less structure. The data are not organized into neat little units with uniform content, where an integer may be expected at a particular location or a string at another. Little may be reliably determined beyond the existence of a document and its boundaries. Of course, information may be derived from a document (such as inverted indices--see Chapter 3). However, such information is invisible to the user of an information retrieval system. He or she sees only a document, with no particular structure save units derived from natural languages, such as words, sentences, or paragraphs.

This creates a problem for information retrieval systems: queries tend to be vague, especially when compared to database management systems. The latter, with their precise characterization of data, permit a broad range of queries that can exploit known properties of the data. Thus, while a database management system user might issue a query such as:

select Name, Age from Person where Age > 25

the user of an information retrieval system has far less flexibility. The information retrieval system cannot hope to determine what sequence of characters in the documents under consideration represent age. It has no knowledge of document structure and therefore cannot make inferences about portions of a document.

Queries on documents in information retrieval systems, then, are limited to operations involving character strings, at best with some knowledge of sentence structure. A typical query might be, "Give me the names of all documents containing sentences with the words 'computer' and 'science'." Despite its seeming simplicity, such a query can be very costly. In a database management system, one might optimize this query by working from sorted data, and using search algorithms that capitalize on such order. An information retrieval system, however, cannot reorder a document's text. A search for a word within a document may therefore take time linearly proportional to the size of the document. As mentioned, the situation can be improved through the use of indices, but the use of a partial index can limit the flexibility of an information retrieval system, and a full index is often prohibitively expensive.

In any case, the size of the data is often enormous. Information retrieval systems manipulate huge amounts of information, often orders of magnitude more than is found in database management systems. Even if indexing is used, the data sets will be prohibitively large. Simple-minded algorithms to process queries will take unacceptable amounts of time and space.

Boolean operations are the key to describing queries, and their efficient implementation is essential. The typical technique is to use sets. This chapter covers how to implement Boolean operations as sets, and also how to efficiently implement set operations. It emhasizes sets that are expected to contain large numbers of elements. Section 12.2 discussess Boolean operations in more detail, describing precisely their relationship to sets. Section 12.3 presents an abstraction of sets. Sections 12.5 and 12.6 present various implementations of this abstraction. The final section analyzes the run-time properties of the implementation approaches.

12.2 BOOLEAN EXPRESSIONS AND THEIR REPRESENTATION AS SETS

Boolean expressions are formed from user queries. In some information retrieval systems, the user enters them directly. In more sophisticated systems, the user enters natural language, which the system converts into Boolean expressions. In any event, they are then evaluated to determine the results of the query. This section will show how Boolean expressions are built from queries, and how those expressions are translated into sets.

An information retrieval system accesses some set of documents. The system must be able to uniquely identify each document. In this chapter, it will suffice to have documents named doc1, doc2, . . . , docn. It is assumed that the name "doci" provides enough information to locate the information associated with document i. In reality, an addressing table would probably be associated with the names of documents, linking each name to some disk location, but this detail can be safely ignored for now.

Boolean expressions are formed from queries. They represent a request to determine what documents contain (or do not contain) a given set of keywords. For example

Find all documents containing "information".

is a query that, when evaluated, should yield a (possibly empty) set of documents, each of which contains the word "information" somewhere within its body. Most of the words in the sentence are noise, however. By definition, a query searches a set of documents to determine their content. The above is therefore usually represented as the Boolean expression:

information

which means, "A set whose elements are the names of all documents containing the pattern "information"."

Some queries attempt to find documents that do not contain a particular pattern, a fact that the corresponding Boolean expression must indicate. This is done using a "not" operator, prefixed to the expression. For example, the query:

Find all documents that do not contain "information".

is represented as the Boolean expression:

not information

Most queries involve searching more than one term. For example, a user might say any of the following:

Find all documents containing "information" and "retrieval".

Find all documents containing "information"

or "retrieval" (or both).

Find all documents containing "information"

or "retrieval", but not both.

Only documents containing both "information" and "retrieval" will satisfy the first query, whereas the second will be satisfied by a document that contains either of the two words. Documents satisfying the third query are a subset of those that satisfy the second. In fact, those documents in the second query are the union of the documents in the first and third.

Each of the above queries illustrates a particular concept that may form a Boolean expression. These concepts are, respectively, conjunction, disjunction, and exclusive disjunction. They are represented in Boolean expressions using the operators and, or, and xor. The queries above would therefore be:

information and retrieval

information or retrieval

information xor retrieval

Boolean expressions may be formed from other Boolean expressions to yield rather complex structures. Consider the following query:

Find all documents containing "information", "retrieval",

or not containing both "retrieval" and "science".

This translates into the following Boolean expression:

(information and retrieval) or not (retrieval and science)

Parentheses are often helpful to avoid ambiguity.

Each portion of a Boolean expression yields a set of documents. These portions are evaluated separately, then simplified using techniques that will be discussed in this chapter. That is, "all documents containing 'information' " yields a set of documents D1, and "all documents containing 'retrieval' " yields some set D2. The information retrieval system must combine these two sets to yield the set D3 that contains only documents with both "information" and "retrieval."

The reader may be wondering how certain complex queries are handled. For instance, how is "all documents containing 'information' and 'retrieval' in the same sentence" represented as a Boolean expression? This question will be answered in other chapters. The concern of this chapter is not to study techniques for determining if a document contains a pattern. Rather, given a set of documents known to contain a pattern, the issue at hand is how to combine those patterns efficiently to answer a larger query. In other words, other chapters will discuss how to determine which documents contain "information," and which contain "retrieval." Here, the assumption is that the respective sets are known; the issue is how to combine those sets to determine what documents might contain both, either, or neither.

Combining the terms of Boolean expressions is conceptually quite simple. It involves sequences of familiar set operations. Let U represent the names of all documents stored. Let D1 and D2 represent the names of those documents that contain patterns P1 and P2, respectively. The following list defines how to evaluate Boolean expression operators in terms of the sets:

1.     U - D1 is the set of all documents not containing P1 (not).

2.     D1   [image]  D2 is the set of all documents containing both P1 and P2 (and).

3.     D1   [image]  D2 is the set of all documents containing either P1 or P2 (or).

4.     D1   [image]  D2 - D1   [image]  D2 is the set of all documents containing either P1 or P2, but not both (xor).

The following example illustrates these expressions. Consider a set of five documents, and suppose that they contain the terms shown in Figure 12.1. The expressions given above will now be evaluated using the data in Figure 12.1. For instance, the Boolean expression "information" is the names of all documents containing the term "information":

{doc1, doc3}

The expression "information and retrieval" is:

{doc1, doc3}   [image]  {doc1, doc2, doc4} = {doc1}

whereas "information or retrieval" is:

{doc1, doc3}   [image]  {doc1, doc2, doc4} = {doc1, doc2, doc3, doc4}

As a more complex example:

{information and retrieval} or not {retrieval and science}

is:

({doc1, doc3}   [image]  {doc1, doc2, doc4})   [image]  {doc1, doc2, doc3, doc4, doc5}

- ({doc1, doc2}  [image]  {doc2, doc4, doc5})

=    {doc2}   [image]  {doc1, doc3, doc4, doc5}

=    {doc1, doc2, doc3, doc4, doc5}

Document                 Terms

---------------------------------------------------

  doc1      algorithm, information, retrieval

  doc2      retrieval, science

  doc3      algorithm, information, science

  doc4      pattern, retrieval, science

  doc5      science, algorithm

Figure 12.1: Example documents and their terms

12.3 OPERATIONS ON SETS

Before discussing how to implement sets, it is necessary to define them more precisely. Sets are a familiar concept, and implementations for them may be found in many data structures textbooks. However, no two textbooks seem to have precisely the same operations, and the semantics of the operations vary from book to book (Does an operation modify its inputs? Is it a procedure or a function?). A little time spent discussing the meaning of a set as used in this chapter will avoid confusion.

12.3.1 Set Interface

A set is a homogeneous, unordered collection of elements. In programming language terms, a set has an associated "element data type," and all elements of the set must be of this type. Each element data type has a key. For a given set, no two elements ever simultaneously possess the same key. An element data type may also specify other data that will be included along with the element. These data values need not be unique among all elements of a set. If the key constitutes the entire value of a set element, however, then by definition all values in a set differ.

From the discussion in the previous section, it should be no surprise that the familiar set operations of union, intersection, and difference will be required. However, certain supporting operations will be necessary to provide a satsifactory algorithmic implementation of sets. The presentation will be done using the information hiding principle advanced by Parnas (1972), wherein a set is presented in two parts--a collection of access programs that define the operations a program may legally perform on a set, and an implementation for those programs.

To specify a set, one must provide an element data type E. This specifies the types of elements that may populate a given instance of a set. Some programming languages, such as Ada, permit programmers to enforce the constraint that a set contain homogeneous data. In C, the programming language used here, this constraint is unenforceable. However, only a little care is required to ensure that sets are in fact homogeneous.

The first order of business is to declare a data type for a set, and for elements of the set:

typedef <...> set;

typedef <...> elementType;

The structure of a set will be defined later, when the implementation is given. For now, it suffices to know that such a type is available. The definition for element types will depend on the type of data to be stored in the set. In C, it is generally possible to use the type "char*" to store data of any type. However, creating typespecific operations is usually better practice. Moreover, one implementation for sets given here cannot easily represent pointer data; in fact, short would be the best choice. This point will be discussed in more detail in each implementation.

Figure 12.2 shows the operations that constitute the interface to a set. A few conventions will be used that should be noted now:

1.     All operations use pointers to values of type set. In C, pointer-valued parameters are generally used for parameters that a routine is to modify, or for efficiency. Here, the reason is usually efficiency. An operation does not modify its parameter unless so noted.

2.     The existence of a Boolean data type is assumed. This type may be easily declared using the facilities of the C preprocessor:

#define TRUE 1

#define FALSE 0

typedef int boolean;

3.     The system using the sets will not modify element values once they have been inserted into a set. This problem might arise in C if a set contained strings. Consider the following:

v = "abc";

Insert(s, v);

v[1] = 'x';

Unless Insert has stored a copy of the string, the value of v will change inside s, since s will contain a pointer to the string rather than a unique value. This problem can be solved easily in two ways:

a.     The application will agree not to modify values that are used in sets.

b.     The specification of a set can include a function that creates a copy of a datum.

Case a turns out to suffice for information retrieval. Since it is simpler, it will be used in this chapter.

4.     Unless explicitly stated otherwise, a set variable may not appear twice in a parameter list. That is, neither of the following are legal:

Copy (s, s);

Unite(a, b, b);

This restriction is actually not necessary for all routines, but occurs frequently enough to warrant its inclusion.

void Clear(set *s);               /* Make s contain zero elements. */

void Insert(set *s,               /* Modify s to contain e, if  */

elementType e);       /* it does not already.       */

void Delete(set *s,               /* Remove e from s.           */

elementType e);

void Unite(set *s1, set *s2,      /* Make s3 the union of sets   */

set *s3);              /* s1 and s2.                  */

void Intersect(set *s1, set *s2,   /* Make s3 the intersection   */

set *s3);          /* of sets s1 and s2.         */

void Subtract(set *s1, set *s2,    /* Make s3 the difference     */

set *s3);            /* of sets s1 and s2.         */

boolean Empty(set *s);             /* Return TRUE iff s has      */

/* zero elements.             */

boolean Member(set *s,             /* Return TRUE iff e is a     */

elementType e);    /* member of s.               */

void Copy(set *source,             /* Make "destination" the     */

set *destination);       /* same set as "source".      */

void Iterate(set *s,               /* Call f(e) once for every   */

boolean (*f)());      /* element e in set s.        */

boolean Error_Occurred();          /* Return TRUE iff the last   */

/* operation caused an error. */

Figure 12.2: Set interface operations

Most of the operations are self-explanatory, as they implement familiar set functions. The Iterate operation may require some explanation. Once a Boolean expression has been evaluated, the information retrieval system will need to determine all elements in the set. The system should not depend on a particular implementation of sets, so it must have some other means to access all elements. This is exactly what Iterate provides. The system will call it, supplying as a parameter a C routine that accomplishes some desired function. Iterate will execute this function once for every element in the set, supplying the element to the routine as a parameter. For example, suppose a developer writes the following C procedure:

boolean PrintElement(i)

elementType  i;

{

printf("The value is %d\n", i);

return TRUE;

}

The following statement would then print all elements of a set S of integers:

Iterate(S, PrintElement);

No qualifications are made about the order of the elements in the iteration; it is completely random so far as the routine calling Iterate is concerned. The only guarantee is that each element in the set is passed to f () exactly once.

It will be possible to pass parameters to the operations that would, if accepted, result in an erroneous set. Some error-handling mechanism is therefore needed. This mechanism must allow an information retrieval system to detect when an error has occurred. The Error_Occurred () routine, which can be used to check the status of an operation, achieves this. Its value is TRUE if an error occurred during the last executed set operation, and FALSE if no error occurred. A system that wished to check the validity of each operation might contain code such as the following:

Intersect (s1, s2, s3);

if ( Error_Occurred () ) {

...

}

The error handling will be accomplished such that a set will always be valid according to the rules of its implementation, even if an error occurred. Applications may therefore attempt corrective action and try again.

12.3.2 Packaging

The interface given in Figure 12.2 can provide an information retrieval system with operations on sets of various types. These operations may be used by many parts of the system. Their packaging must make this expedient. Information retrieval systems are large and consist of many subsystems. The code for a typical information retrieval system will probably be spread throughout many files. The set operations must be packaged such that each subsystem--that is, set of files--needing access to sets may have them in the simplest possible way, involving no duplication of code.

The usual C technique for doing this is to package the interface in an "include file" (see Kernighan and Ritchie [1988]). This file contains specifications of the types, constants, and procedures that make up the set package, but omits any executable code. (The one exception is routines implemented as macros, rather than procedures. Their expansion, which includes executable code, appears in the include file.)

An implementation of a set would therefore be packaged as follows:

1.     An include file, which might be named "set. h". It would contain all of Figure 12.2, plus the type definition and accompanying constants for the set data type. (Ideally the definition of the type would not be in the interface, but C discourages this.)

2.     A file containing implementations of the routines, which might be named "set. c". Since set. h contains certain type definitions needed in the implementation, it begins with the line:

#include "set. h"

12.3.3 Example

The routines' use will be illustrated by using them to solve the query given at the end of section 12.2. Doing so requires the use of six sets--three for each of the patterns, plus three to hold intermediate values and the results. Figure 12.3 gives the code. The issue of how the sets are initialized is ignored here; it is the subject of Chapter 3. That would involve calls to the Insert routine, one for each document to be inserted in a set.

Note that the code has been given independently from the implementation. While this is possible, subsequent analysis will demonstrate that complete independence is not desirable. The implementations include routines that improve run-time performance by supplying implementation-specific details.

#include "set.h"

SolveQuery ()

{

set info_docs, retr_docs, sci_docs;

set t1, t2, t3;

Clear (&info_docs);

Clear (&retr_docs);

Clear (&sci_docs);

Clear (&t1);

Clear (&t2);

Clear (&t3);

... Find the documents containing "information",

"retrieval" and "science", and put their names

into info_docs, retr_docs, and sci_docs,

respectively ...

Intersect(&info_docs, &retr_docs, &t1);

Intersect(&retr_docs, &sci_docs, &t2);

Unite(&t1, &retr_docs, &t3);

Subtract(&t3, &t2, &t1);

/* Set t1 contains the result of the query. */

}

Figure 12.3: Solving a simple set query

12.4 GENERAL CONSIDERATIONS FOR IMPLEMENTING SETS

Before considering specific techniques for implementing sets, it is worth mentioning the trade-offs one may encounter. Space and time considerations are as applicable in set algorithms as in any other area. However, the sets used in information retrieval can vary greatly in size, indeed by many orders of magnitude. Small documents, or ones being queried in the context of a particular domain, may have only a few terms. Then again, some documents will have tens of thousands of unique index terms on which users may wish to search. Sets for the former may easily be kept in memory and manipulated using fixed-space algorithms. Sets for the latter may require secondary memory implementation strategies. The same information retrieval system may well access both types of documents. One implementation approach to sets might not be enough.

In any case, large rather than small sets are likely to be the exception rather than the rule. Furthermore, the amount of information on one's system seldom shrinks. The ever-increasing amount of knowledge has made gigabyte information bases commonplace. Terabyte systems are not uncommon. Indeed, they will probably be the rule as global networks proliferate. Planning for a small, stable information base is unwise, then, and will lead to unacceptable performance. Every ounce of computing power must therefore be squeezed out of the algorithms that implement Boolean operations. The ones given here attempt to do so. They sometimes sacrifice clarity for speed, but a reader familiar with the C programming language should be able to figure them out without undue effort.

Two implementation strategies are given. This only scratches the surface of possibilities. The two approaches are sufficiently diverse as to be representative of most major trade-offs, however, and therefore will help the reader understand the benefits of other approaches he or she may come across. The technique of balanced trees is worthy of consideration; its performance is, in most respects, intermediate with respect to the approaches in this chapter. See Sedgewick (1990) for an excellent treatment of the topic.

The implementor of an information retrieval system should carefully consider the relationship of the two approaches. Certain external factors may drive which ones are practical. For instance, one technique (bit vectors) is very fast but relies on representing document names as unique integers. Doing so is certainly feasible, but--unless one's documents really are named doc1, doc2, . . .--potentially time-consuming. Whether performing the mapping for each set will result in time savings depends on the document names and the hardware available.

The desire to use this approach may also drive other design decisions in an information retrieval system. Document names are maintained in inverted indices; if the indices are sorted, the names can be mapped much more rapidly than if they are unsorted. Of course, sorting the names is an expensive operation, and practical only for precomputed indices.

This discussion illustrates the complexity that arises from a seemingly simple choice. Such decisions may influence the architecture of the entire system.

12.5 BIT VECTORS

Bit vectors are a simple approach to implementing sets. They provide an extremely fast implementation that is particularly suited to sets where the element type has a small domain of possible values. The representation is not compact, as will be seen, but may be appropriate if the number of documents is small.

The approach using bit vectors is to represent a set as a sequence of bits, with one bit for each possible element in the domain of the element type. It is assumed that each value in the element's domain maps to a unique bit. If a bit's value is 1, then the element represented by that bit is in the set; if 0, the element is not in the set. These values map to the TRUE and FALSE constants of the Boolean data type, which provides a clearer understanding of the bit's purpose (TRUE means an element is in a set, FALSE means it is not) than do the digits.

Most computers do not understand bit strings of arbitrary lengths. A program must reference a minimum or maximum number of bits in a given operation--usually corresponding to a byte, word, or the like. It is therefore necessary to arrange the bit string such that it can be treated as a whole, or referenced bit-by-bit, as appropriate. The usual approach is to declare a bit string as a sequence of contiguous storage. Access to individual bits is achieved by first accessing the byte containing the desired bit(s), and then applying a "mask-and-shift" sequence (illustrated below) to recover specific bits.

The technique takes full advantage of computer hardware to achieve efficiency of operations. Most computers have instructions to perform and, or, xor, and not operations on bit strings. The C programming language, which provides access to bit-level operations, is well suited to bit vector implementations.

Consider an example. Suppose there are 20 documents, and suppose further that they may be uniquely identified by numbering them from 1 to 20, inclusive. A string of 20 bits will be used to represent sets of these documents. The first task is to determine the mapping between the numbers and bits. A logical approach is to have document i map to bit i. However, in C the first bit has index 0, not 1. Therefore, document i actually maps to bit i - 1.

Suppose a search reveals that, of the 20 documents, only documents 3, 8, 11, and 12 contain patterns of interest. The bit vector set to represent this consists of 0's at all elements except those four positions:

001000010011000000000000

The alert reader will notice that this string actually consists of 24 bits, not 20. The reason is that computers are not usually capable of storing exactly 20 bits. Most store information in multiples of 8 bits, an assumption made here. Padding the string with zeros is therefore necessary.

Suppose the system now needs to know whether document i is in the set. There is no C operation that allows bit i - 1 to be retrieved directly. However, it can be done by extracting the byte containing bit i - 1 and then setting all other bits in that byte to zero. If the resulting byte is nonzero, then bit i - 1 must have been 1; if it is zero, bit i - 1 must have been 0. The first step, then, is to compute the byte containing bit i - 1. Since there are 8 bits in a byte, each byte stores information on up to 8 elements of a set. Therefore, bit i - 1 is in byte (i - 1)/8 (as with bits, the first byte is indexed by 0, not 1).

The next step is to isolate the individual bit. Modulo arithmetic is used to do so: the bit is at location (i - 1 ) mod 8 within the byte, since the mapping scheme sequences bits in order. Shifting the bit to the first position of the byte, and then and'ing the byte with 1, yields a byte that contains 1 if the bit is 1, and 0 if the bit is 0. The complete C expression to access the bit is therefore:

( vector [ (i-1) /8] >> (i-1)%8 ) & 01

The inverse operation--setting a particular bit, that is, adding an element to a set--is performed in essentially the reverse manner.

An understanding of bit-manipulation concepts makes the implementation of the remaining operations straightforward. For instance, the intersection of two sets as bit vectors is just a loop that uses C's & operation:

for ( i = 0; i < length(s1); i++ )

s3[i] = s1[i]&s2[i];

assuming that length (s) returns the number of bytes needed to represent set s.

The drawback of using bit vectors arises when the domain of the element type is large. Representing a set of integers on a 32-bit computer requires bit vectors of length 231/8 = 268,435,456 bytes, an amount beyond most computers' capacities. Knowing that the set will only contain a few of these elements at any time does not help. A set implemented using the above mapping still requires the full amount of space to store the two-element set containing -231 and 231 - 1. Indeed, many compilers that use bit vectors to implement sets restrict the domain to that of the maximum number of bits in a single word--typically 60 or 64. This usually limits their utility to sets of single characters.

The situation is even worse if the element type is character strings, such as might be used to hold document names. A 10-element character string restricted to upper- and lowercase letters has 5210 possible values. Clearly, bit strings will be impractical for such situations. However, there are enough small domains to warrant the consideration of bit vectors as a set implementation technique.

If large bit vectors are impractical, there must be a way to specify the domain of elements of a set. This will require adding a new function to the interface given in Figure 12.2. The function will create a new set capable of storing elements from a given range of values.

void Create(int lower, int upper, set *s);

In other words, all sets will be treated as sets of integers between lower and upper, inclusive. The set operations will return sets of the same domain as their parameters, and will refuse to operate on sets of two different domains.

Since set operations need to know the domain of elements, a set will need to include information on elements. The implementation for sets, then, will contain three pieces of information: the lower and upper bounds of the set, and the bit string defining the set's elements at any time. The following structure presents this information:

#define MAX_ELEMENTS 256

#define WORDSIZE 8

typedef struct  {

int     lower;

int     upper;

char    bits[MAX_ELEMENTS\WORDSIZE];

} set;

This implementation rather arbitrarily restricts the maximum number of elements that a set may contain to 256. It is possible to provide more flexibility, by making bits a pointer and allocating space for it at run time, but doing so defeats the point of using bit strings for fast access--the system must worry about garbage collection issues. Such flexibility is better achieved using other representations. The implementation also defines the word size--that is, the number of bits per word in a value of type char--at 8. It is worth noting that using an int array might prove faster. Most C implementations have 8 bits in a datum of type char, and 16 or 32 in an int; using an int might make the operations that manipulate entire bytes more efficient. The decision will depend on the underlying hardware, and the types of accesses to sets that occur most frequently. Empirical analysis is the best way to resolve the question.

The implementation for Create is:

void Create(lower, upper, s)

int     lower, upper;

set     *s;

{

if ( lower > upper

|| (upper - lower) >= MAX_ELEMENTS ) {

error_occurred = TRUE;

return;

}

s->lower = lower;

s->upper = upper;

error_occurred = FALSE;

}

The routine simply initializes the bounds of the set. The first few lines, which deal with errors, bear explanation. As mentioned above, errors will be flagged through the use of a function Error_Occurred. This routine will be implemented as a reference to a "hidden" variable:

static boolean error_occurred;

which would appear in the set's implementation file. Each set operation has the responsibility to set this variable to either TRUE or FALSE. The implementation of Error_Occurred is then:

boolean Error_Occurred()

{

return error_occurred;

}

Note that create does not initialize the set's contents; that is, the set is not guaranteed empty.

void Clear(s)

set     *s;

{

register int    i, Number_Of_Bytes;

Number_Of_Bytes = (s->upper - s-lower)/WORDSIZE + 1;

for ( i = 0; i < Number_Of_Bytes; i++ )

s->bits[i] = 0;

error_occurred = FALSE;

}

Since a zero indicates that an element is not in a set, and since s's bits are all zeros, set s contains no elements.

Insertion and deletion of elements are handled by setting the appropriate bit to 1 and 0, respectively. Figure 12.4 shows the code to do so. The routines mask the byte containing the appropriate bit with a bit string that turns the bit on or off, respectively. The bit string for Insert consists of a string of 0's, with a 1 at the appropriate bit; or'ing this string with a byte with result in a byte unchanged except for (possibly) the bit at the location of the 1 in the bit string. The bit string for Delete consists of a string of 1'S, with a 0 at the appropriate bit; and'ing this string with a byte will result in a byte unchanged except for (possibly) the bit at the location of the 0. C's bit-oriented operators make this simple.

The code for routines that test properties of a set was given above, in simplified form. Figure 12.5 shows the complete C routines. Note that Member is defined to be erroneous if e is not of the correct domain, that is, if it is outside the lower and upper bounds. It still returns FALSE since, as a function, it must return some value. An alternate approach would be to indicate that it is not in the set, without also flagging an error. However, this weakens the abstraction.

void Insert(s, e)

set            *s;

elementType     e;

{

if ( e < s->lower || e > s->upper ) {

error_occurred = TRUE;

return;

}

s->bits[(e-s->lower)/WORDSIZE] |= 01 << ( (e-s->lower)%WORDSIZE);

error_occurred = FALSE;

}

void Delete(s, e)

set             *s;

elementType     e;

{

if ( e < s->lower || e > s->upper ) {

error_occurred = TRUE;

return;

}

s->bits[(e-s->lower)/WORDSIZE] &= ~(01 << ((e-s->lower)%WORDSIZE));

error_occurred = FALSE;

}

Figure 12.4: Inserting and deleting elements in a bit-vector set

boolean Empty(s)

set     *s;

{

register int  i, Number_Of_Bytes;

error_occurred = FALSE;

Number_Of_Bytes = (s->upper - s->lower)/WORDSIZE + 1;

for ( i = 0; i < Number_Of_Bytes; i++ )

if ( s->bits[i] )

return FALSE;

return TRUE;

}

boolean Member(s, e)

set   *s;

elementType   e;

{

if ( error_occurred = (e < s->lower || e < s->upper) )

return FALSE;

return (s->bits[(e - s->lower)/WORDSIZE] >>

(e - s->lower)%WORDSIZE) & 01;

}

Figure 12.5: Empty and Member bit-vector routines

The Intersect, Unite and Subtract routines are next. The operations will be restricted such that all three parameters must have the same bounds. This makes the bit-vector approach simple. (Extending the operations to accommodate sets with other bounds is not difficult, but is somewhat harder for systems to use. It is left as an exercise.) Two sets thus constrained will have the same mapping function, so bit i of one set will represent the same element as bit i of another. This means that entire bytes can be combined using C's bit operators. The equivalence of two sets is tested using the macro:

#define equivalent(s1, s2) \

((s1)->lower==(s2)->lower && (s1)->upper==(s2)->upper)

Figures 12.6-12.8 show the code.

void Unite(s1, s2, s3)

set    *s1, *s2;

set    *s3;

{

register int i, Number_Of_Bytes;

if ( ! (equivalent(s1, s2) && equivalent(s2, s3)) ) {

error_occurred = TRUE;

return;

}

Number_Of_Bytes = (s1->upper - s1->lower)/WORDSIZE + 1;

for ( i = 0 ; i Number_Of_Bytes; i++ )

s3->bits[i] = (s1->bits[i] | s2->bits[i]);

error_occurred = FALSE;

}

Figure 12.6: Unite bit-vector routine

Applications sometimes need to copy sets. Using bit vectors, this is straightforward. The bits array must be copied byte by byte, since C does not permit assignment of entire arrays:

void Copy(source, destination)

set *source,*destination;

{

register int      i;

if ( ! equivalent(source, destination) ) {

error_occurred = TRUE;

return;

}

for ( i = 0; i < MAX_ELEMENTS/WORDSIZE; i++ )

destination-bits[i] = source->bits[i];

error_occurred = FALSE;

}

void Intersect(s1, s2, s3)

set     *s1, *s2;

set     *s3;

{

register int i, Number_Of_Bytes;

if ( ! (equivalent(s1, s2) && equivalent(s2, s3)) ) {

error_occurred = TRUE;

return;

}

Number_Of_Bytes = (s1->upper - s1->lower)/WORDSIZE + 1;

for ( i = 0 ; i < Number_Of_Bytes; i++ )

s3->bits[i] = (s1->bits[i] & s2->bits[i]);

error_occurred = FALSE;

}

Figure 12.7: Intersect bit-vector routine

void Subtract(s1, s2, s3)

set     *s1,  *s2;

set     *s3;

{

register int   i, Number_Of_Bytes;

if ( ! (equivalent(s1, s2) && equivalent(s2, s3)) ) {

error_occurred = TRUE;

return;

}

Number_Of_Bytes = (s1->upper - s1->lower)/WORDSIZE + 1;

for ( i = 0 ; i < Number_Of_Bytes; i++ )

s3->bits[i] = s1->bits[i] & ~ (s2->bits[i]);

error_occurred = FALSE;

}

Figure 12.8: Subtract bit-vector routine

The final operation is iteration. This operation uses C's dereferencing features to execute an application-specified function. Figure 12.9 shows the code. The function f receives each element--not the bit, but the actual value. This requires converting the bit position into the value, the inverse mapping from that required for other operations. The function is expected to return a Boolean value. If this value is TRUE, another element will be passed to f; if FALSE, the iteration will cease. This lets applications exert some control over the iteration.

void Iterate(s, f)

set     *s;

boolean (*f) ();

{

register int  i, j, Number_Of_Bytes;

Number_Of_Bytes = (s->upper - s->lower)/WORDSIZE + 1;

error_occurred = FALSE;

for ( i = 0; i < Number_Of_Bytes; i++ )

for ( j = 0; j < WORDSIZE; j++ )

if ( (s->bits[i] >> j) % 2 )

if ( ! (*f)(j + i*WORDSIZE + s->lower) )

return;

}

Figure 12.9: Bit-vector iterator routine

12.6 HASHING

If a large number of documents must be searched, or if representing their names  using integers is not convenient, then bit vectors are unacceptable. This section will show how to use hashing, another common set implementation technique. The approach is not as fast as bit vectors, but it makes far more efficient use of space. In fact, whereas bit vectors explicitly restrict the maximum number of elements, hashing permits sets of arbitrary size. The trade-off is in speed: performance is inversely proportional to a set's cardinality. However, good performance characteristics can be achieved if one has some idea in advance of the average size of the sets that will be manipulated, a quantity that is usually available.

Hashing is actually useful in implementing many important IR operations. Chapter 13 gives an in-depth presentation of hashing. Readers not familiar with hashing may wish to study that chapter before reading any more of this section.

The implementation will use chaining, not open addressing. It is possible, but risky, to use open addressing. A hash table must be resized when full, which is a difficult operation, as explained later.

The concepts underlying the hashing-based implementation are straightforward. Each set is implemented using a single hash table. An element is inserted in a set by inserting it into the set's hash table. Its presence is verified by examining the appropriate bucket.

This implementation will be less restrictive about typing than the implementation given for bit vectors. No assumptions will be made about the type of data; the application using the hash table will be responsible for assuring that the elements are all homogeneously typed. The results will be unpredictable if this requirement is violated. In return, it will be possible to store any type of data--not just integers. The application must provide some hints to the set operation routines about the nature of the data, however. It will supply the number of buckets and a hashing function â(v), mapping from the domain of the set elements to integers. This lets applications achieve much better performance. The information will be passed as part of the process of creating a set. There will be some enforcement of set types, however, and it will be based on this information.

The implementation of the set, then, will need to maintain this information. A plausible implementation of the set type is:

typedef char        element_Type;

typedef struct be_str {

elementType   datum;

struct be_str  *next_datum;

} bucket_element;

typedef struct {

int            Number_Of_Buckets;

bucket_element **buckets;

int            (*hashing_function)();

boolean        (*comparator)();

} set;

The first type definition gives the data type for set elements. Each datum is of type "char *", a C convention for a variable to contain information of any type. The second definition is used to construct the linked lists that form the buckets. The third definition contains the information needed for a set. Note that the buckets are not preallocated; only a pointer is included in the definition. The array of buckets is allocated when the set is created.

The restrictions on set equivalence can be greatly relaxed for hash sets. There are no bounds that need to be identical, as in bit vectors. The appropriate hashing function must be applied--that is, the hashing function from one table should not be used on another--but adhering to this rule is simple, since the function is included in the structure.

The creation routine is given in Figure 12.10. Note that its interface is different from the bit-vector approach, reflecting the different information needed to maintain and manipulate the sets. A "Comparator" routine must also be provided. This routine, given two values of type elementType, should return TRUE if they are equal, and FALSE if they are not. Such a routine is needed for hash tables containing strings or dynamically allocated data, since C's "= =" operator does not understand such structures.

void Create(Number_Of_Buckets,  Hashing_Function, Comparator, s)

int     Number_Of_Buckets;

int     (*Hashing_Function)();

boolean (*Comparator)();

set     *s;

{

register unsigned int  Bucket_Array_Size;

register int           i;

if ( Number_Of_Buckets <= 0 ) {

error_occurred = TRUE;

return;

}

s->Number_Of_Buckets = Number_Of_Buckets;

s->hashing_function = Hashing_Function;

s->comparator = Comparator;

Bucket_Array_Size = sizeof(bucket_element) * Number_Of_Buckets;

s->buckets = (bucket_element **)malloc(Bucket_Array_Size);

if ( error_occurred = (s->buckets == NULL) )

return;

for ( i = 0; i < Number_Of_Buckets; i++ )

s->buckets[i] = (bucket_element *)NULL;

}

Figure 12.10: Hashing version of Create

C's global memory allocation routine, malloc, is used to provide space for the buckets. The use of dynamic memory requires some care. It must be freed when no longer needed, lest the application eventually run out of space. This will require the introduction of a "Destroy" operation, whose purpose is opposite of Create. Its implementation is left as an exercise to the reader.

In the bit-vector implementation, Create leaves the set in an uninitialized state, needing to be cleared before use. This is not feasible in hashing. The reason is that clearing the set involves freeing dynamically allocated space. In C, the presence of such storage is indicated by a non-null pointer. Therefore, Clear might become confused if Create did not set all buckets to . That, however, leaves the set "cleared." This does leave Clear able to de-allocate space:

void Clear(s)

set *s;

{

register int      i;

register bucket_element *b, *next_b;

for ( i = 0; i < s->Number_Of_Buckets; i++ )

if ( s->buckets[i] != NULL ) {

b = s->buckets[i];

while ( b != NULL ) {

next_b = b->next_datum;

free( (char *)b );

b = next_b;

}

s->buckets[i] = NULL;

}

error_occurred = FALSE;

}

Almost all the operations on sets will use the hashing function. Rather than repeat it in each routine, the following C macro will be used:

#define hash(s,e) (abs ((* ((s)->hashing_function)) (e)) %

(s)->Number_Of_Buckets)

Insertion and deletion into a hash table involve linked-list operations. Figures 12.11 and 12.12 contain one implementation. This version of Insert ( ) inserts an element by traversing the list associated with the bucket to which it maps. If the element is not already in the list, it is inserted at the list's head. Insertion at the head is common because, in practice, elements added recently tend to be accessed most often. Data sets for which this does not hold could use an implementation where the element is added at the end of the list. Note that application-supplied hashing  function is equivalent to â (v), not h (v), that is, its range is all integers rather than all integers between a certain bucket size. This simplifies dynamic resizing of hash tables (an improvement discussed in Chapter 13).

Deleting an element is done using a standard linked-list technique: the list is searched for the element while a pointer is kept to the previous node. If the element is found, the pointer to the previous node is used to make that node point to the one following the element to be deleted. The storage for the deleted node is then freed. Note that if the datum component of the node points to dynamically allocated memory, that memory will be lost unless some other node points to it. This situation may be rectified by adding a Free_datum function parameter to Create, an extension left to the reader.

Testing whether the set is empty involves testing if there are any pointers to  elements. Figure 12.13 contains the code for this. It also contains the implementation of Member, done using linked-list traversal.

Constructing the union of two sets implemented using hashing is somewhat more involved than is the operation with bit vectors. No one-to-one correspondence can be established between elements. All elements that hash to the same bucket will be in the same list, but the respective lists may be ordered differently--a list's order depends on the order in which elements were inserted. The approach used is to add each element of set s1 to s3, then scan through each element of s2, adding the element to s3 if it is not already there. Figure 12.14 shows the implementation.

void Insert(s, e)

set             *s;

elementType     e;

{

register bucket_element  *b, *New_Element;

register int             bucket;

error_occurred = FALSE;

bucket = hash(s,e);

for ( b = s->buckets[bucket]; b != NULL ; b = b->next_datum )

if ( (*(s->comparator))(b->datum, e) )

return;

New_Element = (bucket_element *)malloc(sizeof (bucket_element));

if ( New_Element == NULL ) {

error_occurred = TRUE;

return;

}

New_Element->datum = e;

New_Element->next_datum = s-->buckets[bucket];

s->buckets[bucket] = New_Element;

}

Figure 12.11: Inserting elements in a hash table set

void Delete(s, e)

set            *s;

elementType    e;

{

register bucket_element *b, *previous;

register int            bucket;

error_occurred = FALSE;

bucket = hash(s, e);

if ( (b = s->buckets[bucket]) == NULL )

return;

if ( (*(s->comparator))(b->datum, e) )

s->buckets[bucket] = b->next_datum;

else {

while ( b->next_datum != NULL ) {

if ( (*(s->comparator))(b->datum, e) )

break;

previous = b;

b = b->next_datum;

}

if ( b == NULL )

return;

previous->next_datum = b->next_datum;

}

free ( (char *)b );

}

Figure 12.12: Deleting elements from a hash table set

boolean Empty(s)

set   *s;

{

register int i;

error_occurred = FALSE;

for ( i = 0; i < s->Number_Of_Buckets; i++ )

if ( s->buckets[i] != NULL )

return FALSE;

return TRUE;

}

boolean Member (s, e)

set            *s;

elementType    e;

{

register bucket_element *b;

register int            bucket;

error_occurred = FALSE;

bucket = hash(s, e);

for ( b = s->buckets[bucket]; b != NULL ; b = b->next_datum )

if ( (*(s->comparator))(b->datum, e) )

return TRUE;

return FALSE;

}

Figure 12.13: Empty and Member hashing routines

The concepts behind the implementations of intersection and subtraction are similar. They are shown in Figures 12.15 and 12.16. Intersecting involves traversing through s1 and adding to s3 those elements that are also in s2; subtracting works by adding to s3 those elements that are in s1 but not s2.

void Unite(s1, s2, s3)

set     *s1, *s2;

set     *s3;

{

register int              i;

register bucket_element   *b;

Copy(s1, s3);

if ( Error_Occurred() )

return;

for ( i = 0; i < s2->Number_Of_Buckets; i++ ) {

if ( s2->buckets[i] == NULL )

continue;

for ( b = s2->buckets[i]; b != NULL; b = b->next_datum )

if ( ! Member(s3, b->datum) ) {

Insert(s3, b->datum);

if ( Error_Occurred() )

return;

}

}

error_occurred = FALSE;

}

Figure 12.14: Unite hashing routine

void Intersect(s1, s2, s3)

set     *s1, *s2;

set     *s3;

{

register int            i;

register bucket_element *b;

Clear(s3);

for ( i = 0; i < s1->Number_Of_Buckets; i++ ) {

if ( s1->buckets[i] == NULL )

continue;

for ( b = s1->buckets[i]; b != NULL; b = b->next_datum )

if ( Member(s2, b->datum) ) {

Insert(s3, b->datum);

if ( Error_Occurred() )

return;

}

}

error_occurred = FALSE;

}

Figure 12.15: Intersect hashing routine

void Subtract (s1, s2, s3)

set     *s1, *s2;

set     *s3;

{

register int             i;

register bucket_element  *b;

Clear(s3);

for ( i = 0; i < s1->Number_Of_Buckets; i++ ) {

if ( s1->buckets[i] == NULL )

continue;

for ( b = s1->buckets[i]; b != NULL; b = b->next_datum )

if ( ! Member(s2, b->datum) ) {

Insert(s3, b->datum);

if ( Error_Occurred() )

return;

}

}

error_occurred = FALSE;

}

Figure 12.16: Subtract hashing routine

Copying a hash table involves copying the bucket array, plus all lists in the buckets. The code to do so is in Figure 12.17. The routine does not simply copy the lists; the data from one must be rehashed into the other, since the two may not have identical bucket array lengths or hash functions. Even if the tables are equivalent in these respects, they will not be identical after the copying operation: the order of elements within the buckets is reversed. This will only show up during iteration. Since the order of elements during iteration is not defined, the difference is irrelevant.

The final operation is iteration. It is shown in Figure 12.18. The technique to be used is similar to that of Copy: traverse through each list in each bucket, passing each datum to the function provided to Iterate. Unlike bit vectors, this traversal will almost certainly not yield the elements in any particular oder. The order will depend on both the hashing function and the order of insertion.

12.7 ANALYSIS

To aid the reader in understanding the relative merits of the two implementations, this section presents an analysis of the time and space required for the two set implementation techniques. The analysis for each is straightforward. The operations of most concern are Insert, Unite, Intersect, and Subtract, since it is presumed that most applications will spend the majority of their time manipulating sets rather than creating them. For the sake of completeness, however, all routines will be analyzed.

void Copy(source, destination)

set *source, *destination;

{

register int                i  h;

register bucket_element     *e, *b;

Clear (destination);

for ( i = o; i < source->Number_Of_Buckets; i++ ) {

if ( source->buckets[i] == NULL )

continue;

for ( b = source->buckets[i]; b != NULL; b = b->next_datum ) {

h = hash(destination, b->datum);

e = (bucket_element *)malloc(sizeof (bucket_element));

if ( e == NULL ) {

error_occurred = TRUE;

return;

}

e->datum = b->datum;

e->next_datum = destination->buckets[h];

destination->buckets[h] = e;

}

}

error_occurred = FALSE;

}

Figure 12.17: Copying a hash table set

void Iterate(s, f)

set      *s;

boolean  (*f) ();

{

register int              i;

register bucket_element   *b;

error_occurred = FALSE;

for ( i = 0; i < s->Number_Of_Buckets; i++ )

for ( b = s->buckets[i]; b != NULL; b = b->next_datum )

if ( ! (*f) (b->datum) )

return;

}

Figure 12.18: Hashing iterator routine

For bit vectors, the insertion and deletion operations are O(C), that is, their running time is constant. The exact time will depend on the speed of the division operation on the underlying hardware, the fundamental bottleneck in the implementation (bit operations such as &= and ~  are usually much quicker). The other operations, which iterate across the set, have running time that depends on the set domain. Assuming this quantity to be N, they require O(N) steps. The actual value will be closer to N/WORDSIZE, since the operations are able to access entire words rather than individual bits. The exception is Iterate( ), which must scan every bit, making it closer to O(N).

The space requirements for bit vectors are somewhat worse. They are also constant, but potentially quite large. In the implementation given here, MAX_ELEMENTS/WORDSIZE+2   [image]  sizeof (int) bytes will be needed to store any bit-vector set. As noted previously, this can easily be rather large.

The behavior of hashing algorithms is not quite so predictable. It depends on the randomness of the hashing function on the data being stored. Assuming that the hashing function is "reasonably" random, worst-case and expected behavior are not difficult to derive. Worst-case will be presented first. The hashing function is usually chosen to be O(C) (nonlinear functions defeat the advantages of hashing). The time required once the hashing function is computed will be proportional to the number of elements in the bucket. At worst, all elements will be in a single bucket. If so, then insertion, deletion, and membership tests all require O(N), where N is the number of elements in the set. The union, intersection, and difference routines have significantly poorer worst-case behavior. Consider the Unite operation on two sets that each have all elements in a single bucket. The Member() test requires O(N) steps for a single invocation. Since the inner for-loop iterates through N elements, the complexity of the loop is O(N2). This logic also applies to Intersect and Subtract.

The expected running time is significantly better. If the elements are distributed randomly, and if the number of elements N is less than the number of buckets B, then the expected number of elements in any bucket is less than one. Thus, the expected time for insertion, deletion, and membership testing would be constant, and the expected time for uniting, intersecting, or subtracting two sets would be O(N). This is likely to be a great improvement over bit vectors, since the number of elements in a set is usually very small compared to the number of elements in a domain. Of course, N cannot easily be predicted in advance, so N may very well exceed B. Even so, N must reach B2 before the performance degrades to O(N2) for union, intersection, and difference. Moreover, the number of buckets can also be increased at any time to exceed N, lowering the expected complexity to O(C). Table 12.1 summarizes this data for both approaches.

Table 12.1: Relative Algorithmic Complexity for Different Set Implementations

                                  Hashing           Bit Vectors

                          Worst-Case  Average  Worst-Case  Average

-------------------------------------------------------------------

Insert/Delete/Member        O(N)       O(C)       0(C)       O(C)

Unite/Intersect/Subtract    O(N2)      O(N)       O(W)       0(W)

Create/Empty                O(B)       O(B)       O(W)       0(W)

Copy                        O(N)       O(N)       0(W)       0(W)

N = number of elements

C = constant

B = number of buckets in hash table

W = number of words in bit string

Hash tables require considerably less space than bit vectors. The size is determined more by the number of elements than the size of the domain. The exact formula for the space depends on the number of buckets as well. Assuming that P is the space required to store a pointer, and that each element requires E units of space, a hash table of N element requires

BP + NP + NE

units. Table 12.2 contrasts the space requirements for bit vectors and hash tables. Bytes are assumed to contain 8 bits, pointers 16. The sets are representing integers; it is assumed that the minimum possible size will be used (i.e., short, int, or long). The hash table has 503 buckets, implying that any hash table requires at least 1,006 bytes. The tables show how the advantages of bit vectors rapidly diminish in proportion to the size of the number of elements in the domain.

Table 12.2: Relative Space Requirements for Different Set Implementations

        Bit Vectors                      Hash Tables

Domain   Number of Elements      Domain   Number of Elements

 Size     64    1024   215        Size     64   1024   215

---------------------------     ----------------------------

  256     32     --     --           256    1198  4078  99,310

  215     4096  4096  4096         215    1262  5102  132,078

  231     228   228   228           231    1390  7100  197,614

REFERENCES

KERNIGHAN, B. W., and D. M. RITCHIE. 1988. The C Programming Language, 2nd ed. Englewood Cliffs, N.J.: Prentice Hall.

PARNAS, D. L. 1972. On the Criteria to be Used in Decomposing Systems into Modules. Communications of the ACM, 15(12), 1053-58.

SEDGEWICK, R. 1990. Algorithms in C. Reading, Mass: Addison-Wesley.

Go to Chapter 13     Back to Table of Contents


Information Retrieval: CHAPTER 13: HASHING ALGORITHMS

 Return to Table of Contents

CHAPTER 13: HASHING ALGORITHMS

Steven Wartik, Edward Fox, Lenwood Heath and Qi-Fan Chen

Software Productivity Consortium, Virginia Polytechnic Institute and State University

Abstract

This chapter discusses hashing, an information storage and retrieval technique useful for implementing many of the other structures in this book. The concepts underlying hashing are presented, along with two implementation strategies. The chapter also contains an extensive discussion of perfect hashing, an important optimization in information retrieval, and an O (n) algorithm to find minimal perfect hash functions for a set of keys.

13.1 INTRODUCTION

Accessing information based on a key is a central operation in information retrieval. An information retrieval system must determine, given a particular value, the location (or locations) of information that have been decided to be relevant to that value. Other chapters in this book have addressed the issue; in particular, Chapters 3-6 have dealt with useful techniques for organizing files for efficient access. However, these chapters have been concerned with file-level concerns. They have not covered the fundamental underlying algorithms for organizing the indices to these files.

This chapter discusses hashing, a ubiquitous information retrieval strategy for providing efficient access to information based on a key. Under many conditions, hashing is effective both in time and space. Information can usually be accessed in constant time (although the worst-case performance can be quite poor). Space use is not optimal but is at least acceptable for most circumstances.

The material in this chapter stresses the most important concepts of hashing, and how it can be implemented. Some important theory for hashing is presented, but without derivation. Knuth (1973) provides a fuller treatment, and also gives a history and bibliography of many of the concepts in this chapter.

Hashing does have some drawbacks. As this chapter will demonstrate, they can be summarized by mentioning that it pays to know something about the data being stored; lack of such knowledge can lead to large performance fluctuations. Relevant factors include some knowledge of the domain (English prose vs. technical text, for instance), information regarding the number of keys that will be stored, and stability of data. If these factors can be predicted with some reliability--and in an information retrieval system, they usually can--then hashing can be used to advantage over most other types of retrieval algorithms.

13.2 CONCEPTS OF HASHING

The problem at hand is to define and implement a mapping from a domain of keys to a domain of locations. The domain of keys can be any data type--strings, integers, and so on. The domain of locations is usually the m integers between 0 and m - 1, inclusive. The mapping between these two domains should be both quick to compute and compact to represent.

Consider the problem first from the performance standpoint. The goal is to avoid collisions. A collision occurs when two or more keys map to the same location. If no keys collide, then locating the information associated with a key is simply the process of determining the key's location. Whenever a collision occurs, some extra computation is necessary to further determine a unique location for a key. Collisions therefore degrade performance.

Assume the domain of keys has N possible values. Collisions are always possible whenever N > m, that is, when the number of values exceeds the number of locations in which they can be stored. The best performance is therefore achieved by having N = m, and using a 1: 1 mapping between keys and locations. Defining such a mapping is easy. It requires only a little knowledge of the representation of the key domain. For example, if keys are consecutive integers in the range (N1, N2), then m = N2 - N1 + 1 and the mapping on a key k is k - N1. If keys are two-character strings of lowercase letters, then m = 26 [image] 26, and the mapping (using C character manipulation for ASCII) is (k [ 0 ] - ' a ' ) * (k [ 1 ] - ' a ' ) . These two mappings can be computed in constant time, and are therefore ideal from a performance standpoint.

Now consider compactness. If keys are drawn from the set of strings of letters up to 10 characters long--too short to be of use in information retrieval systems-- then [image]. An implementation of a mapping with m = N is impossible: no existing computer contains enough memory. The mapping that is best suited to performance is wasteful of space.

The problem is that the mapping is defined over all N values, that is, over all possible keys in a very large domain. In practice, no application ever stores all keys in a domain simultaneously unless the size of the domain is small. Let n be the number of keys actually stored. In most applications, [image], so only a few of the many possible keys are in use at any time. From the standpoint of compactness, the ideal representation would have m = 1; that is, all locations would be in use, no matter how many keys are in the table. All keys collide, and a strategy must exist to resolve collisions. As discussed above, any such strategy will require extra computation.

The mapping involved in hashing thus has two facets of performance: number of collisions and amount of unused storage. Optimization of one occurs at the expense of the other. The approach behind hashing is to optimize both; that is, to tune both simultaneously so as to achieve a reasonably low number of collisions together with a reasonably small amount of unused space.

The concept underlying hashing is to choose m to be approximately the maximum expected value of n. (The exact value depends on the implementation technique.) Since [image], in theory many collisions are possible, and indeed are expected in practice. However, n is usually sufficiently small that memory consumption is not excessive; also, an attempt is made to choose a mapping that randomly spreads keys throughout the locations, lowering the probability of collision.

The information to be retrieved is stored in a hash table. A hash table is best thought of as an array of m locations, called buckets. Each bucket is indexed by an integer between 0 and m - 1. A bucket can be empty, meaning that it corresponds to no key in use, or not empty, meaning that it does correspond to a key in use, that information has been placed in it, and that subsequent attempts to place information in it based on another key will result in a collision. (Whether "not empty" is equivalent to "full" depends on the implementation, as sections 13.4.1 and 13.4.2 show.)

The mapping between a key and a bucket is called the hash function. This is a function whose domain is that of the keys, and whose range is between 0 and m - 1, inclusive. Storing a value in a hash table involves using the hash function to compute the bucket to which a key corresponds, and placing the information in that bucket. Retrieving information requires using the hash function to compute the bucket that contains it.

For example, suppose a hash table is to be implemented with 5 buckets, and a hash function h(k) = k mod 5, where k is an integer, is used to map keys into the table. Figure 13.1 shows the hash table after the keys 1, 2, and 64 have been inserted. 1 is stored in bucket 1 because 1 mod 5 = 1, 2 is stored in bucket 2 because 2 mod 5 = 2, and so on.

This scheme works well as long as there are no collisions. The time to store and retrieve data is proportional to the time to compute the hash function. Typically, this function is very simple, and can be calculated in constant time. The space required to store the elements is that required for an array of m elements. Unless m is very large, this should not be a problem.

In practice, however, collisions should be expected. Indeed, storing more than m distinct keys into a hash table with m will always cause collisions. Even with fewer than m keys, choosing a hash function that distributes keys uniformly is sufficiently difficult that collisions can be expected to start occurring long before the hash table fills up.

This suggests two areas for further study: first, how to choose a hash function, and second, what to do when a collision occurs. Section 13.3 discusses hash functions. Section 13.4 presents two implementation techniques and the collision resolution strategies they employ. Section 13.5 explains perfect hash functions, a technique for avoiding, rather than resolving, collisions.

Figure 13.1: Visual illustration of a hash table

13.3 HASHING FUNCTIONS

The choice of hash function is extremely important. The ideal function, termed a perfect hash function, would distribute all elements across the buckets such that no collisions ever occurred. As discussed in the previous section, this is desirable for performance reasons. Mapping a key to a bucket is quick, but all collision-resolution strategies take extra time--as much as [image] (m + n). Retrieval times become nonuniform, making performance hard to predict. A perfect hash function guarantees uniform performance.

Perfect hash functions are extremely hard to achieve, however, and are generally only possible in specialized circumstances, such as when the set of elements to hash is known in advance. Another such circumstance was illustrated in Chapter 12: the bit-vector implementation of a set illustrates a perfect hash function. Every possible value has a unique bucket. This illustrates an important point: the more buckets a hash table contains, the better the performance is likely to be. Bucket quantity must be considered carefully. Even with a large number of buckets, though, a hash table is still only as good as its hash function.

The most important consideration in choosing a hash function is the type of data to be hashed. Data sets are often biased in some way, and a failure to account for this bias can ruin the effectiveness of the hash table. For example, the hash table in Figure 13.1 is storing integers that are all powers of two. If all keys are to be powers of two, then h(k) = k mod 5 is a poor choice: no key will ever map to bucket 0. As another example, suppose a hash table is be used to store last names. A 26-bucket hash table using a hash function that maps the first letter of the last name to the buckets would be a poor choice: names are much more likely to begin with certain letters than others (very few last names begin with "X").

A hash function's range must be an integer between 0 and m - 1, where m is the number of buckets. This suggests that the hash function should be of the form:

h(v) = f(v) mod m

since modulo arithmetic yields a value in the desired range. Knuth (1973) suggests using as the value for m a prime number such that rk mod m = a mod m, where r is the radix of v, and a and k are small. Empirical evidence has found that this choice gives good performance in many situations. (However, Figure 13.1 shows that making m a prime number is not always desirable.)

The function f(v) should account for nuances in the data, if any are known. If v is not an integer, then the internal integer representation of its bits is used, or the first portion thereof. For example, â(v) for a nonempty charcter string might be the ASCII character-code value of the last character of the string. For a four-byte floating-point number, f(v) might be the integer value of the first byte.

Such simple-minded schemes are usually not acceptable. The problems of using the first character of a string have already been discussed; the last character is better, but not much so. In many computers, the first type of a floating-point word is the exponent, so all numbers of the same magnitude will hash to the same bucket. It is usually better to treat v as a sequence of bytes and do one of the following for f(v):

1. Sum or multiply all the bytes. Overflow can be ignored.

2. Use the last (or middle) byte instead of the first.

3. Use the square of a few of the middle bytes.

The last method, called the "mid-square" method, can be computed quickly and generally produces satisfactory results.

13.4 IMPLEMENTING HASHING

Once the hash function and table size have been determined, it remains to choose the scheme by which they shall be realized. All techniques use the same general approach: the hash table is implemented as an array of m buckets, numbered from 0 to m - 1. The following operations are usually provided by an implementation of hashing:

1. Initialization: indicate that the hash table contains no elements.

2. Insertion: insert information, indexed by a key k, into a hash table. If the hash table already contains k, then it cannot be inserted. (Some implementations do allow such insertion, to permit replacing existing information.)

3. Retrieval: given a key k, retrieve the information associated with it.

4. Deletion: remove the information associated with key k from a hash table, if any exists. New information indexed by k may subsequently be placed in the table.

Compare this definition with the one given in Chapter 12. There, the "information" associated with a key is simply the presence or absence of that key. This chapter considers the more general case, where other information (e.g., document names) is associated with a key.

Figure 13.2 shows a C-language interface that realizes the above operations. The routines all operate on three data types: hashTable, key, and information. The definition of the hashTable type will be left to the following sections, since particular implementation techniques require different structures. The data types of the key and the information are application-dependent; of course, the hash function must be targeted toward the key's data type.

void Initialize(hashTab1e *h);  /* Make h empty.              */

void Insert(hashTable *h,       /* Insert i into h, keyed     */

key k,              /* by k.                      */

information i);

void Delete(hashTable *h,       /* Delete from h the          */

key k);             /* information associated     */

/* with k.                    */

information                     /* Retrieve from h the        */

Retrieve(hashTable *h,      /* information associated     */

key k);            /* with k.                    */

#define OKAY          0        /* These values are returned   */

#define DUPLICATE_KEY 1        /* by op_status(), indicating  */

#define NO_SUCH_KEY   2        /* the result of the operation */

#define TABLE_FULL    3        /* last executed.              */

int op_status();

Figure 13.2: Interface to a hashing implementation

13.4.1 Chained Hashing

The first implementation strategy is called chained hashing. It is so named because each bucket stores a linked list--that is, a chain--of key-information pairs, rather than a single one. The solution to a collision, then, is straightforward. If a key maps to a bucket containing no information, it is placed at the head of the list for that bucket. If a key maps to a bucket that already has information, it is placed in the list associated with that bucket. (Usually it is placed at the head of the list, on the assumption that information recently stored is likely to be accessed sooner than older information; this is also simplest to implement.) Figure 13.3 shows how the hash table from Figure 13.1 would appear if it was a chained table and all powers of two from 1 to 64 had been stored in it.

Figure 13.3: Visual illustration of a chained hash table

In this scheme, a bucket may have any number of elements. A hash table with m buckets may therefore store more than m keys. However, performance will degrade as the number of keys increases. Computing the bucket in which a key resides is still fast--a matter of evaluating the hash function--but locating it within that bucket (or simply determining its presence, which is necessary in all operations) requires traversing the linked list. In the worst case (where all n keys map to a single location), the average time to locate an element will be proportional to n/2. In the best case (where all chains are of equal length), the time will be proportional to n/m. It is best not to use hashing when n is expected to greatly exceed m, so n/m can be expected to remain linearly proportional to m. Usually, most chains' lengths will be close to this ratio, so the expected time is still constant.

The hashing-based implementation of Boolean operations in Chapter 12 uses chains. Most of the code from the routines Insert, Delete, Clear (for Initialize), and Member (for Retrieve) can be used directly. Chapter 12 also discussed the need for a routine to create the table. This routine had as  parameters the hash function and an element-comparison routine, which shows how this information may be associated with the table.

Since the code is so similar, it will not be repeated here. Instead, a sketch is given of the differences. This is easily illustrated by considering the new data types needed. The hash table is declared as follows:

typedef ... information;

typedef ... key;

typedef struct be_str {

key           k;

information   i;

struct be_str *next_datum;

};

typedef struct {

int             Number_Of_Buckets;

bucket_element  *buckets;

int             (*hashing_function)();

boolean         (*comparator)();

} hashTable;

As in Chapter 12, the specifics of information and key are application-dependent, and so are not given here. The main difference from the routines in Chapter 12 lies in the need to associate the information with its key in a bucket. Hence, each bucket has a field for information. The line in Insert:

New_Element->datum = e;

is replaced with:

New_Element->k = k;

New_Element->i = i;

The other routines differ analogously.

13.4.2 Open Addressing

Sometimes, the likelihood that the number of keys will exceed the number of buckets is quite low. This might happen, for instance, when a collection of data is expected to remain static for a long period of time (as is often true in information retrieval). In such a circumstance, the keys associated with the data can be determined at the time the data is stored. As another example, sometimes the maximum number of keys is known, and the average number is expected to hover around that value. In these cases, the flexibility offered by chained hashing is wasted. A technique tailored to the amount of storage expected to be used is preferable.

This is the philosophy behind open addressing. Each bucket in the hash table can contain a single key-information pair. (That is, a bucket that is not empty is "full.") Information is stored in the table by computing the hash function value of its key. If the bucket indexed by that value is empty, the key and information are stored there. If the bucket is not empty, another bucket is (deterministically) selected. This process, called probing, repeats until an empty bucket is found.

Many strategies to select another bucket exist. The simplest is linear probing. The hash table is implemented as an array of buckets, each being a structure holding a key, associated information, and an indication of whether the bucket is empty. When a collision is detected at bucket b, bucket b + 1 is "probed" to see if it is empty. If b + 1 = m, then bucket 0 is tried. In other words, buckets are sequentially searched until an empty bucket is found.

For example, consider Figure 13.4 (a), which shows a 10-bucket hashing table containing powers of two between 1 and 16, where h(v) = v mod 10. Suppose the keys 32, 64, 128, 256, and 512 are now inserted. Key 32 collides with key 2, so it is inserted in bucket 3. Key 64 collides with key 4, so it is inserted in bucket 5. By the time key 512 is inserted, the table is as in Figure 13.4 (b), with all buckets except 0 full. Inserting key 512 necessitates the sequence of probes shown in Figure 13.4 (c). Worse, this sequence is duplicated each time 512 is retrieved. Retrieving all other elements will take little time requiring at most one extra probe, but retrieving 512 will be expensive.

This illustrates a characteristic of linear probing known as clustering: the tendency of keys to cluster around a single location. Clustering greatly increases the average time to locate keys in a hash table. It has been shown (see Sedgewick [1990], for instance) that, for a table less than two-thirds full, linear probing uses an average of less than five probes, but for tables that are nearly full, the average number of probes becomes proportional to m.

Figure 13.4: Clustering in a hash table

The problem is that hashing is based on the assumption that most keys won't map to the same address. Linear probing, however, avoids collisions by clustering keys around the address to which they map, ruining the suitability of nearby buckets. What is desirable is a more random redistribution. A better scheme than linear probing is to redistribute colliding keys randomly. This will not prevent collisions, but it will lessen the number of collisions due to clustering.

The redistribution can be done using some function h2(b) that, when applied to a bucket index b, yields another bucket index. In other words, the original hash function having failed, a second one is applied to the bucket. As with linear probing, another probe will be tried if the new bucket is also full. This introduces some constraints on h2. h2 = b + m would be a poor choice: it would always try the same bucket. h2 = b + m/i where m is a multiple of i would also be bad: it can only try m/i of the buckets in the table. Ideally, successive applications of h2 should examine every bucket. (Note that linear probing does this: it is simply h2(b) = (b + 1) mod m). This will happen if h2(b) = (b + i) mod m, where i is not a divisor of m. A commonly used scheme employs a formula called quadratic probing. This uses a sequence of probes of the form h + i, where i = 1, 4, 9, . . . . This is not guaranteed to probe every bucket, but the results are usually quite satisfactory.

In open addressing, it is possible to fill a hash table--that is, to place it in a state where no more information can be stored in it. As mentioned above, open addressing is best when the maximum number of keys to be stored does not exceed m. There are, however, strategies for coping when this situation arises unexpectedly. The hash table may be redefined to contain more buckets, for instance. This extension is straightforward. It can degrade performance, however, since the hash function might have been tuned to the original bucket size.

The data structures for implementing open addressing are simpler than for chained hashing, because all that is needed is an array. A suitable definition for a hash table is:

typedef struct {

key         k;

information i;

bool    empty;

} bucket;

typedef struct {

int     number_of_buckets;

bucket *buckets;

int     (*hashing_function) ();

int     (*comparator) ();

} hashTable;

Each element of the table is a structure, not a pointer, so using a null pointer to indicate if a bucket is empty or full no longer works. Hence, a field empty has been added to each bucket. If the key is itself a pointer type, then the key field can be used to store this information. The implementation here considers the more general case.

Except for probing, the algorithms are also simpler, since no linked list manipulation is needed. Figures 13.5-13.8 show how it is done. As before, the macro:

#define hash(h, k) ((*((h)->hashing_function))(k) % (h)->number_of_buckets)

is defined to abstract the computation of a bucket for a key.

void Initialize(h)

hashTable *h;

{

register int i;

status = OKAY;

for ( i = 0; i < h->number_of_buckets; i++ )

h->buckets[i].empty = TRUE;

}

Figure 13.5: Initializing an open-addressed hash table

These routines all behave similarly. They attempt to locate either an empty bucket (for Insert) or a bucket with a key of interest. This is done through some probing sequence. The basic operations of probing are initialization (setting up to begin the probe), determining if all spots have been probed, and determining the next bucket to probe. Figure 13.9 shows how quadratic probing could be implemented. The probe is considered to have failed when (m + 1)/2 buckets have been tried. In practice, this value has been shown to give good results.

void Insert(h, k, i)

hashTable       *h;

key       k;

information i;

{

register int b;

b = hash(h, k);

status = OKAY;

Initialize_Probe(h);

while ( ! h->buckets[b].empty && ! Probe_Exhausted(h) ) {

if ( (*h->comparator)(k, h->buckets[b].k) ) {

status = DUPLICATE_KEY;

return;

}

b = probe(h);

}

if ( h->buckets[b].empty ) {

h->buckets[b].i = i;

h->buckets[b].k = k;

h->buckets[b].empty = FALSE;

}

else

status = TABLE_FULL;

}

Figure 13.6: Inserting into an open-addressed hash table

void Delete(h, k)

hashTable   *h;

key          k;

{

register int b;

status = OKAY;

b = hash(h, k);

Initialize_Probe(h);

while ( ! h->buckets[b].empty && ! Probe_Exhausted(h) ) {

if ( (*h->comparator)(k, h->buckets[b].k) ) {

h->buckets[b].empty = TRUE;

return;

}

b = probe(h);

}

status = NO_SUCH_KEY;

}

Figure 13.7: Deleting from an open-addressed hash table

information Retrieve(h, k)

hashTable    *h;

key           k;

{

register int b;

status = OKAY;

b = hash(h, k);

Initialize_Probe(h, b);

while ( ! h->buckets[b].empty && ! Probe_Exhausted(h) ) {

if ( (*h->comparator)(k, h->buckets[b].k) )

return h->buckets[b].i;

b = probe(h);

}

status = NO_SUCH_KEY;

return h->buckets[0].i;   /* Return a dummy value. */

}

Figure 13.8: Retrieving from an open-addressed hash table

static int      number_of_probes;

static int      last_bucket;

void Initialize_Probe(h, starting_bucket)

hashTable   *h;

int          starting_bucket;

{

number_of_probes = 1;

last_bucket = starting_bucket;

}

int probe(h)

hashTable   *h;

{

number_of_probes++;

last_bucket = (last_bucket + number_of_probes*number_of_probes)

% h->number_of_buckets;

return last_bucket;

}

bool Probe_Exhausted(h)

hashTable *h;

{

return (number_of_probes >= (h->number_of_buckets+1)/2);

}

Figure 13.9: Quadratic probing implementation

13.5 MINIMAL PERFECT HASH FUNCTIONS

Section 13.3 mentioned that the ideal hash function would avoid collisions by mapping all keys to distinct locations. This is termed a perfect hash function. A perfect hash function is ideal from a performance standpoint in that the time to locate the bucket corresponding to a particular key is always the time needed to compute the hash function. This predictability improves the ability to precisely infer performance characteristics.

Perfect hash functions are possible, but generally only when the set of keys to be hashed is known at the time the function is derived. Best of all under this  condition is a minimal perfect hash function, a perfect hash function with the property that it hashes m keys to m buckets with no collisions. Not only is performance optimized, but no space is wasted in the hash table.

In general, it is difficult to find a MPHF. Knuth (1973) observes that only one in 10 million functions is a perfect hash function for mapping the 31 most frequently used English words into 41 addresses. Minimal perfect hash functions are rarer still.

This section presents an algorithm for finding minimal perfect hash functions for a given set of keys. The algorithm is not guaranteed to work, but is almost always successful. Before explaining the algorithm, it will be helpful to give some background on the topic.

13.5.1 Existence Proof

One might ask whether a minimal perfect hash function (hereafter abbreviated MPHF) h exists for a set of keys. Jaeschke (1981) proves that the answer is yes. Consider the problem of mapping a set of m positive integers, bounded above by N without collisions into a hash table T with m buckets. The following algorithm defines a suitable MPHF:

Store the keys in an array k of length m.

Allocate an array A of length N, and initialize all values to ERROR.

for ( i = 1; i < m; i++ )

A[k[i]] = i

The array A defines h: allowable keys map into addresses {0, ..., m - 1} and other keys map into ERROR.

This defines a MPHF, but the array A is mostly empty; since usually m   [image]  N, the hash function occupies too much space to be useful. In other words, efficient use of storage in hashing encompasses the representation of the hash function as well as optimal use of buckets. Both must be acceptably small if minimal perfect hashing is to be practical.

13.5.2 An Algorithm to Compute a MPHF: Concepts

The scarcity of suitable functions suggests that it is best to search function spaces for them using computer programs. There are several strategies for doing so. The simplest is to select a class of functions that is likely to include a number of minimal perfect hash functions, and then search for a MPHF in that class by assigning different values to each of the parameters characterizing the class.

Carter and Wegman (1979) introduced the idea of a class H of functions that are universal2, that is, where no pair of distinct keys collides very often. By random selection from H, one can select candidate functions and expect that a hash function having a small number of collisions can be found quickly. This technique has been applied to dynamic hashing by Ramakrishna and Larson (1989).

Sprugnoli (1978) proposes two classes of functions, one with two and the other with four parameters, that each may yield a MPHF, but searching for usable parameter values is feasible only for very small key sets. Jaeschke (1981) suggests a reciprocal hashing scheme with three parameters, guaranteed to find a MPHF, but only practical when m   [image]  20. Chang (1986) proposes a method with only one parameter. Its value is likely to be very large, and a function is required that assigns a distinct prime to each key. However, he gives no algorithm for that function. A practical algorithm finding perfect hash functions for fairly large key sets is described by Cormack et al. (1985). They illustrate trade-offs between time and size of the hash function, but do not give tight bounds on total time to find PHFs or experimental details for very large key sets.

The above-mentioned "search-only" methods may (if general enough, and if enough time is allotted) directly yield a perfect hash function, with the right assignment of parameters. However, analysis of the lower bound on the size of a suitable MPHF suggests that if parameter values are not to be virtually unbounded, then there must be a moderate number of parameters to assign. In the algorithms of Cichelli (1980) and of Cercone et al. (1983) are two important concepts: using tables of values as the parameters, and using a mapping, ordering, and searching (MOS) approach (see Figure 13.10). While their tables seem too small to handle very large key sets, the MOS approach is an important contribution to the field of perfect hashing.

In the MOS approach, construction of a MPHF is accomplished in three steps. First, the mapping step transforms the key set from an original to a new universe. Second, the ordering step places the keys in a sequence that determines the order in which hash values are assigned to keys. The ordering step may partition the order into subsequences of consecutive keys. A subsequence may be thought of as a level, with the keys of each level assigned their hash values at the same time. Third, the searching step assigns hash values to the keys of each level. If the Searching step encounters a level it is unable to accommodate, it backtracks, sometimes to an earlier level, assigns new hash values to the keys of that level, and tries again to assign hash values to later levels.

Figure 13.10: MOS method to find perfect hash functions

13.5.3 Sager's Method and Improvement

Sager (1984, 1985) formalizes and extends Cichelli's approach. Like Cichelli, he assumes that a key is given as a character string. In the mapping step, three auxiliary (hash) functions are defined on the original universe of keys U:

h0: U    {0, . . . , m -- 1}

h1: U    {0, . . . , r -- 1}

h2: U    {r, . . . , 2r -- 1}

where r is a parameter (typically   [image]  m/2) that determines the space to store the perfect hash function (i.e., |h| = 2r). The auxiliary functions compress each key k into a unique identifier

(h0(k),h1(k),h2(k))

which is a triple of integers in a new universe of size mr2. The class of functions searched is

h(k) = (h0(k) + g(h1(k)) + g(h2(k)) (mod m)

(1)

where g is a function whose values are selected during the search.

Sager uses a graph that represents the constraints among keys. The mapping step goes from keys to triples to a special bipartite graph, the dependency graph, whose vertices are the h1(k) and h2(k) values and whose edges represent the words. The two vertex sets of the dependency graph are {0, . . . , r - 1} and {r, . . . , 2r - 1}. For each key k, there is an edge connecting h1(k) and h2(k), labeled by k. See Figure 13.11. Note that it is quite possible that some vertices will have no associated arcs (keys), and that some arcs may have the same pairs of vertices as their endpoints.

Figure 13.11: Dependency graph

In the ordering step, Sager employs a heuristic called mincycle that is based on finding short cycles in the graph. Each iteration of the ordering step identifies a set of unselected edges in the dependency graph in as many small cycles as possible. The set of keys corresponding to that set of edges constitutes the next level in the ordering.

There is no proof given that a minimum perfect hash function can be found, but mincycle is very successful on sets of a few hundred keys. Mincycle takes O(m4) time and O(m3) space, while the subsequent Searching step usually takes only O(m) time.

Sager chooses values for r that are proportional to m. A typical value is r = m/2. In the case of minimal perfect hashing (m = n), it requires 2r = n computer words of lg n bits each to represent g. Fox et al. (1989a) have shown, using an argument due to Mehlhorn (1982), that a lower bound on the number of bits per key needed to represent a MPHF is approximately 1.4427. Sager's value is therefore somewhat higher than the optimal. To save space, the ratio (2r/n) must be reduced as low as possible, certainly below 1. Early work to explore and improve Sager's technique led to an implementation, with some slight improvements and with extensive instrumentation added on, described by Datta (1988). Further investigation by Fox et al. (1989a) yielded a modified algorithm requiring O(m3) time. This algorithm has been used to find MPHFs for sets of over a thousand words.

One thousand word key sets is good but still impractical for many information retrieval applications. As described in Fox et al. (1989b), Heath subsequently devised an O(m log m) algorithm, which is practical for large sets of keys. It is based on three crucial observations of previous work:

1.     Randomness must be exploited whenever possible. The functions suggested by Sager do not yield distinct triples in the mapping stage with large key sets. Randomness can help improve this property.

2.     The vertex degree distribution in the dependency graph is highly skewed. This can be exploited to make the ordering step much more efficient. Previously, it required up to O(m3) time; this observation reduces it to O(m log m).

3.     Assigning g values to a set of related words can be viewed as trying to fit a pattern into a partially filled disk, where it is important to enter large patterns while the disk is only partially full.

Since the mapping and searching steps are O(m), the algorithm is O(m log m) with the improved ordering step.

13.5.4 The Algorithm

This section presents the algorithm. It is described in terms of its three steps, plus the main program that fits these steps together. A complete implementation is too large to fit comfortably into this chapter, but it is included as an appendix.

The main program

The main program takes four parameters: the name of a file containing a list of m keys, m, a ratio for determining the size of the hash table, and the name of a file in which the output is to be written. It executes each of the three steps and, if they all succeed, creates a file containing a MPHF for the keys. Figure 13.12 outlines the main routine.

The main program is responsible for allocating enough arcs and vertices to form the dependency graph for the algorithm. "arcs" is an array of the arcs, and "vertices" an array of the vertices. Each arc corresponds to exactly one key. The data structures associated with arcs are as follows:

typedef struct arc {                typedef struct {

int h0, h12[2];                     int no_arcs;

struct arc *next_edge[2];           arcType *arcArray;

} arcType;                          } arcsType;

main(argc, argv)

int    argc;

char   *argv[];

{

arcsType     arcs;

verticesType vertices;

int          seed;

allocate_arcs( &arcs, atoi(argv[2]) );

allocate_vertices( &vertices, atoi(argv[2]) * atof(argv[3]) );

if ( mapping( arcs, vertices, &seed, argv[1] ) == NORM ) {

ordering( arcs, vertices );

if ( searching ( arcs, vertices ) == NORM )

write_gfun ( vertices, seed, argv[4] );

}

}

Figure 13.12: Main program for MPHF algorithm

The arcType structure stores the ho, h1 and h2 values for an arc, plus two singly linked lists that store the vertices arcs incident to the vertices h1 and h2. The arcsType structure is an array of all arcs, with a record of how many exist.

The data structures for vertices have the form:

typedef struct {             typedef struct {

int g, pred, succ;          int no_vertices,

struct arc *first_edge;         maxDegree,

} vertexType;                       vsHead,

vsTail,

rlistHead;

vertexType* vertexArray;

} verticesType;

In vertexType, the first_edge field is the header of a linked list of arcs incident to the vertex (the next_edge values of arcType continue the list). The pred and succ fields are a doubly linked vertex list whose purpose is explained in the ordering stage. The g field ultimately stores the g function, computed during the searching stage. To save space, however, it is used for different purposes during each of the three stages.

The dependency graph created by the mapping step has 2r vertices. (The value of r is the product of m and the ratio supplied as the third parameter to the main routine. It is therefore possible to exert some control over the size of the hash function. A large value of r increases the probability of finding a MPHF at the expense of memory.) The variable vertices, of type verticesType, holds all vertices. The fields of vertices wil1 be explained as they are used in the various steps.

The mapping step

The code for the mapping step is shown in Figure 13.13. The step is responsible for constructing the dependency graph from the keys. This is done as follows. Three random number tables, one for each of ho, h1, h2, are initialized. The number of columns in the table determines the greatest possible key length. The number of rows is currently 128: one for each possible ASCII character. (This is not strictly necessary but helps exploit randomness.) Next, the routine map_to_triples maps each key k to a triple (ho, h1, h2) using the formulas:

int mapping( key_file, arcs, vertices, seed )

char         *key_file;      /* in: name of file containing keys.    */

arcsType     *arcs;          /* out: arcs in bipartite graph.        */

verticesType *vertices;      /* out: vertices in bipartite graph.    */

int          *seed;          /* out: seed selected to initialize the */

/* random tables.                       */

{

int mapping_tries = 0;

randomTablesType randomTables; /* Three random number tables.    */

while ( mapping_tries++ < MAPPINGS ) {

initialize_arcs( arcs );

initialize_vertices( vertices );

initialize_randomTable( randomTables, seed );

map_to_triples( key_file, arcs, vertices->no_vertices/2, randomTables );

if ( construct_graph(arcs, vertices) == NORM )

return(NORM);

}

return(ABNORM);

}

Figure 13.13: The mapping routine

int construct_graphs (arcs, vertices)

arcsType     *arcs;          /* in out: arcs.                       */

verticesType *vertices;      /* in out: vertices.                   */

{

int i,             /* Iterator over all arcs.                       */

j,             /* j = 0 and 1 for h1 and h2 side, respectively. */

status = NORM,

vertex;

for ( j = 0; j < 2; j++ ) {

for ( i = 0; i < arcs->no_arcs; i++ ) {

vertex = arcs->arcArray[i].h12[j];  /* Update vertex degree */

vertices->vertexArray[vertex].g++;  /* count and adjacency  */

arcs->arcArray[i].next_edge[j] =    /* list.                */

vertices->vertexArray[vertex].first_edge;

vertices->vertexArray[vertex].first_edge = &arcs ->arcArray[i];

if ( (j == 0) && check_dup( &arcs->arcArray[i] ) == ABNORM ) {

status = ABNORM;       /* Duplicate found. */

break;

}

/* Figure out the maximal degree of the graph. */

if ( vertices->vertexArray[vertex].g > vertices->maxDegree )

vertices->maxDegree = vertices->vertexArray[vertex].g;

}

}

return(status);

}

Figure 13.14: The construct_graph routine

(2)

(3)

(4)

That is, each function h1 is computed as the sum of the random numbers indexed by the ASCII values of the key's characters. The triple for the ith key in the input file is saved in the ith entry in the arcs array.

The mapping step then builds the dependency graph. Half of the vertices in the dependency graph correspond to the h1 values and are labeled 0, . . . , r - 1. The other half correspond to the h2 values and are labeled r, . . . , 2r - 1. There is one edge in the dependency graph for each key. A key k corresponds to an edge k, between the vertex labeled h1(k) and h2(k). There may be other edges between h1(k) and h2(k), but they are labeled with keys other than k.

Figure 13.14 shows the routine construct_graph(), which builds the dependency graph. The arcs have already been built (in map_to_triples()); what remains is to build the vertex array. Each arc is searched; the vertices associated with it are the values stored in h12[0] and h12[1]. The degree counts of these vertices are incremented, and the incidence lists associated with each vertex are updated.

Because the triples are generated using random numbers, it is possible for two triples to be identical. If this happens, a new set of random number tables must be  generated, and the mapping step repeated. There is never a guarantee that all triples will be unique, although duplicates are fairly rare in practice--it can be demonstrated to be approximately r2/2m3. This value rapidly approaches 1 as m grows. To prevent infinite loops, the mapping step is never attempted more than a fixed number of times.

The ordering step

The goal of the ordering step is to partition the keys into a sequence of levels. The step produces an ordering of the vertices of the dependency graph (excluding those of degree 0, which do not correspond to any key). From this ordering, the partition is easily derived. If the vertex ordering is v1, . . . , vt, then the level of keys K (vi) corresponding to a vertex vi, 1   [image]  i   [image]  t is the set of edges incident to both vi and a vertex earlier in the ordering. More formally, if 0   [image]  vi < r, then

K(vi) = {kj|h1(kj) = vi, h2(kj) = vs, s < i}

(5)

Similarly, if r   [image]  vi < 2r, then

K(vi) = {kj|h2(kj) = vi, h1(kj) = vs, s < i}

(6)

The rationale for this ordering comes from the observation that the vertex degree distribution is skewed toward vertices of low degree. For reasons discussed in the section on the searching step, it is desirable to have levels that are as small as possible, and to have any large levels early in the ordering.

The heuristic used to order the vertices is analogous to the algorithm by Prim (1957) for constructing a minimum spanning tree. At each iteration of Prim's algorithm, an arc is added to the minimum spanning tree that is lowest in cost such that one vertex of the arc is in the partially formed spanning tree and the other vertex is not. In the ordering step, the arc selected is the one that has maximal degree at the vertex not adjacent to any selected arcs. If several arcs are equivalent, one is chosen randomly. Since the dependency graph may consist of several disconnected graphs, the selection process will not stop after the vertices in the first component graph are ordered. Rather, it will continue to process components until all vertices of nonzero degree have been put into the ordering. Figure 13.15 shows the ordering step's implementation.

The vertex sequence is maintained in a list termed the "VS list," the head of which is the vsHead field of the vertices variable. Ordering the vertices efficiently requires quickly finding the next vertex to be added into the VS list. There are two types of vertices to consider. The first are those that start a new component graph to be explored. To handle these, the ordering step first builds (in the initialize_rList() routine) a doubly linked list of all vertices in descending  order of degree. This list, whose head is in the rlistHead field of the vertices variable, and whose elements are linked via the succ and prec fields of a vertexType, can be built in linear time because of the distribution of the degrees. Since the first vertex is most likely to be within the largest component of the graph, the routine almost always correctly orders the vertices in the largest component first, rather than in the smaller components (which would produce a degraded ordering). The starting vertices tor other components are easily decided from the rList.

void ordering( arcs, vertices )

arcsType            *arcs;     /* in out: the arcs.             */

verticesType        *vertices; /* in out: the vertices.         */

{

int         degree, side;

vertexType  *vertex;

arcType     *arc;

vertices->vsHead = vertices->vsTail = NP;   /* Initialize the VS list. */

initialize_rList( vertices );

allocate_vheap( arcs->no_arcs, vertices->no_vertices );

while ( vertices->rlistHead != -1 ) { /* Process each component graph. */

initialize_vheap();

vertex = &vertices->vertexArray[vertices->rlistHead];

do {

vertex->g = 0;                         /* Mark node "visited".   */

delete_from_rList( vertex, vertices );

append_to_VS( vertex, vertices );

if ( vertex->first_edge != 0 ) {

/* Add adjacent nodes that are not visited and  */

/* not in virtual heap to the virtual heap.     */

side = vertex - vertices->vertexArray >= vertices->no_vertices/2;

arc = vertex->first_edge;

while ( arc != 0 ) {

int     adj_node = arc->h12[(side+1)%2];

if ( (degree = vertices->vertexArray[adj_node].g) > 0 ) {

add_to_vheap( &vertices->vertexArray[adj_node], degree );

vertices->vertexArray[adj_node].g *= -1;

}

arc = arc->next_edge[side];

}

}

} while ( max_degree_vertex( &vertex ) == NORM );

}

free_vheap();

}

Figure 13.15: The ordering step

The second type of vertices to be added to the VS list are those adjacent to others already ordered. These are handled by keeping track of all vertices not yet ordered, but adjacent to vertices in the ordering. A heap is used to do this. The algorithm is as follows. Each component of the graph is considered in a separate iteration of the main loop of ordering: The heap is first emptied, and a vertex of maximal degree is extracted from the rList. It is marked visited (the g field is used to do so), and added to the heap.

All vertices adjacent to it are then added to the heap if they have not already been visited and are not already in the heap. Next, max_degree_vertex() removes from the heap a vertex of maximal degree with respect to others in the heap. This becomes the next "visited" vertex; it is deleted from the rList, added to the VS list, and its adjacent vertices added to the heap, as above. This process repeats until the heap is empty, at which time the next component in the graph is selected. When all components have been processed, and the heap is empty, the vertex sequence has been created.

The time to perform this step is bounded by the time to perform operations on the heap, normally O(log n). However, the skewed vertex distribution facilitates an optimization. A vertex of degree i, 1   [image]  i   [image]  5, is not stored on the heap. Instead, five stacks are used, stack i containing only vertices of degree i. The heap therefore contains only vertices of degree greater than 5, a small fraction of the total number. Vertices can be pushed onto and popped from a stack in constant time; the expected time of the ordering step becomes O(n) as a result.

This optimization detail is hidden by the vheap (virtual heap) module. It is not shown here but can be described quite simply. The add_to_vheap() routine takes both a vertex and its degree as parameters and can therefore determine whether to place the vertex on the heap or in a stack. The max_degree_vertex() routine obtains a vertex of maximum degree by first searching the heap; if the heap is empty, it searches stacks 5, 4, 3, 2, and 1, in that order. The implementations of these two routines are based on well-known algorithms for implementing stacks and heaps; see Sedgewick (1990), for instance.

The searching step

The search step scans the VS list produced by the ordering step and tries to  assign hash values to the keys, one level at a time. The hash function ultimately used has the form given in equation (1). Since h0, h1 and h2 have already been computed, what remains is to determine g.

The algorithm to compute g is again based on the insight into vertex degree distribution. The VS list contains vertices with higher degrees first; the partition into levels therefore contains larger groups of interrelated words early on. If these are processed first, the cases most likely to be troublesome are eliminated first. Note that the ordering step does not actually yield a data structure of physically distinct levels; this is simply a concept that is useful in understanding the algorithm. The levels are easily determined from the ordering in the VS list using equations (5) and (6).

int searching( arcs, vertices )

arcsType            *arcs;

verticesType        *vertices;

{

int         i,                     /* Each vertex in the VS list. */

searching_tries = 0,   /* Running count of searching tries.           */

status = ABNORM;       /* Condition variable.  */

char        *disk;                 /* Simulated hash table.               */

intSetType  primes,                /* Table of primes for pattern shifts. */

slotSet;               /* Set of hash addresses.              */

disk = (char*) owncalloc( arcs->no_arcs, sizeof(char) );

slotSet. intSetRep = (int*) owncalloc( vertices->maxDegree, sizeof(int) );

initialize_primes( arcs->no_arcs, &primes );

while ( (searching_tries++ < SEARCHINGS) && (status == ABNORM) ) {

status = NORM;

i = vertices->vsHead;             /* Get the highest-level vertex. */

initialize_search( arcs, vertices, disk );

while ( i != NP ) { /* Fit keys of level of vertex i onto the disk.  */

vertices->vertexArray[i].prec = VISIT;

if ( fit_pattern(arcs, vertices, i,

disk, &primes, &slotSet) == ABNORM ) {

status = ABNORM;        /* Search failed at vertex i. Try */

break;                  /* a new pattern.                */

}

else        /* Search succeeded. Proceed to next node. */

i = vertices->vertexArray[i].succ;

}

}

free( disk );

free( (char *)slotSet.intSetRep );

free( (char *)primes.intSetRep );

return(status);

}

Figure 13.16: The searching step

The simplest way to assure that the keys are correctly fit into a hash table is to actually build a hash table. The disk variable in searching () is used for this purpose. The initialize_search() routine initializes it to an array of m slots (since the hash function is to be minimal as well as perfect), each of which is set to EMPTY. As the g function is computed for a key, the hash address h for the key is determined, and slot h in disk is marked FULL.

The implementation of the searching step is shown in Figure 13.16. Because a searching step is not guaranteed to find a suitable set of values for g, the step is repeated up to a constant number of times. Each repetition works as follows. An empty disk is allocated, all vertices are marked as not visited (in this step, the prec field holds this information), and the g field of each vertex is set to a random value. Using a random value for g often contributes to a high probability of fitting all keys.

The next vertex i in the VS list is then selected. All keys at the level of i must be fit onto the disk. This is the task of fit_pattern(), which behaves as follows. Given vertex i and disk, it examines each arc in K(vi) to see if the arc's key is at the current level (this is indicated by whether or not the vertex adjacent to i had already been visited). If so, it determines if the current values of g for vertex i and the vertices adjacent to i make any keys hash to an already-filled spot. If all keys fit, then fit-pattern() has succeeded for this level. It marks each hash address for the examined keys as FULL in disk, and terminates.

The g value determines what might be considered a "pattern." Because all keys of a particular level are associated with some vertex vi, the hash function's value for all k   [image]  K(vi) is determined in part by the value of g for vi. Any change to g will shift the hash addresses of all keys associated with a given level.

This observation provides the motivation for the action to take when a collision is detected. If fit_pattern() detects a collision for some k   [image]  K(vi), it computes a new pattern--that is, a new value for g for vertex i. All keys on the level will therefore be shifted to new addresses. The formula used to compute the new value for g is:

g    ( g + s) mod m

where s is a prime number. This means that all possible values for g can be tested in at most m tries. The searching() routine computes a table of prime numbers before computing hash addresses. It passes this table to fit-pattern(), which  randomly chooses one value for the table to be used as the shift. Ideally, each vertex should have a different prime. Computing m primes is expensive, however, so a fixed number are generated. Twenty small primes have been shown, empirically, to give satisfactory results.

Using the MPHF

When the searching step is through, the main routine writes the MPHF to a file. All that is necessary is to write the size of the graph, the seed used to start random number generation, and the values of g for each vertex. Another program can then use the MPHF by:

1.     Regenerating the three random number tables, which are needed to recompute h0, h1, and h2.

2.     Rereading the values of g.

Recall that the mapping step included a call to the routine map_to_triples(), whose purpose was to compute the triples for all keys. This routine calls on compute_h012 () to actually compute the triples for a given key; compute_h012() is simply an implementation of equations (2)-(4). Given a key, then, all that is needed to compute its hash address is the code:

arcType arc;

compute_h012 ( no_arcs, r, tables, key, &arc );

hash = abs (arc.h0 +

mphf->gArray[arc.h12[0]] +

mphf->gArray[arc. h12[1]]

) % mphf->no_arcs;

where tables contains the three tables, mphf the g values, and r the number of vertices on one side of the dependency graph. This is the approach used by the retrieval routine in the appendix.

Discussion

The algorithm presented in this section provides a practical means to compute minimal perfect hash functions for small to extremely large sets of keys. Indeed, the algorithm has been verified on key set sizes of over one million; previous approaches could not have computed an MPHF for a set that large in reasonable time. One of its more interesting applications has been to compute the keys for an optical disk of information; see Fox (1990).

The resulting hash function is much more complex than the approaches suggested in section 13.3. Somewhat more time is necessary to use the MPHF, both to initialize it and to compute a key's value. For very small data sets, the overhead may not be justified, but for large data sets the algorithm in this section will be, on the average, much quicker. In any case, it is a constant-time algorithm, and there are always advantages to being able to predict the time needed to locate a key.

The MPHF uses a large amount of space. Ostensibly, its size is proportional to the number of vertices in the dependency graph, that is, O (r). Actually, except for very large key sets, most space is consumed by the three random number tables, which must be present at both the computation and regeneration of the MPHF. Each table has 128 rows and 150 columns, which requires over 1/4 megabyte of storage; this suggests that for small key sets, the approach suggested in section 13.5.1 might be more practical! However, an examination of the equations that access the tables (see the Mapping section step) shows that the actual number of columns used is no more than the length of the longest key. Moreover, while having 128 rows helps exploit randomness, a MPHF can be found with fewer. The implementation could therefore be rewritten to use much less space. This is left as an exercise to the reader.

REFERENCES

CARTER, J. L., and M. N. WEGMAN. 1979. "Universal Classes of Hash Functions." J. Computer and System Sciences, 18, 143-54.

CERCONE, N., M. KRAUSE, and J. BOATES. 1983. "Minimal and Almost Minimal Perfect Hash Function Search with Application to Natural Language Lexicon Design." Computers and Mathematics with Applications 9, 215 -31.

CHANG, C . C. 1986. "Letter Oriented Reciprocal Hashing Scheme." Information Sciences 38, 243-55.

CICHELLI, R. J. 1980. "Minimal Perfect Hash Functions Made Simple." Communications of the ACM, 23, 17-19.

CORMACK, G. V., R. N. S. HORSPOOL, and M. KAISERSWERTH. 1985. "Practical Perfect Hashing." The Computer Journal 28, 54-58.

DATTA, S. 1988. "Implementation of a Perfect Hash Function Scheme." Blacksburg, Va.: Technical Report TR-89-9 (Master's Report), Department of Computer Science, Virginia Polytechnic Institute and State University.

FOX, E. A., Q. CHEN, L. HEATH, and S. DATTA. 1989a. "A More Cost Effective Algorithm for Finding Perfect Hash Functions." Paper presented at the Seventeenth Annual ACM Computer Science Conference, Louisville, KY.

FOX, E. A., Q. CHEN, and L. HEATH. 1989b. "An O (n log n) Algorithm for Finding Minimal Perfect Hash Functions." Blacksburg, Va.: TR 89-10, Department of Computer Science, Virginia Polytechnic Institute and State University.

FOX, E. A. 1990. Virginia Disc One. CD-ROM published by Virginia Polytechnic Institute and State University. Ruckersville, Va.: Nimbus Records.

JAESCHKE, G. 1981. "Reciprocal Hashing--a Method for Generating Minimal Perfect Hash Functions." Communications of the ACM, 24, 829-33.

KNUTH, D. E. 1973. The Art of Computer Programming, Vol. 3: Sorting and Searching. Reading, Mass.: Addison-Wesley.

MEHLHORN, K. 1982. "On the Program Size of Perfect and Universal Hash Functions." Paper presented at the 23rd IEEE Symposium on Foundations of Computer Science.

PRIM, R. C. 1957. "Shortest Connection Networks and Some Generalizations." Bell System Technical Journal 36.

RAMAKRISHNA, M. V., and P. LARSON. 1989. "File Organization Using Composite Perfect Hashing." ACM Transactions on Database Systems, 14, 231-63.

SAGER, T. J. 1984. "A New Method for Generating Minimal Perfect Hashing Functions." Rolla, Mo.: Technical Report CSc-84-15, Department of Computer Science, University of Missouri-Rolla.

SAGER, T. J. 1985. "A Polynomial Time Generator for Minimal Perfect Hash Functions." Communications of the ACM, 28, 523-32.

SEDGEWICK, R. 1990. Algorithms in C. Reading, Mass.: Addison-Wesley.

SPRUGNOLI, R. 1978. "Perfect Hashing Functions: a Single Probe Retrieving Method for Static Sets." Communications of the ACM, 20, 841-50.

APPENDIX: MPHF IMPLEMENTATION

What follows is a complete implementation of the minimal perfect hashing function algorithm. It consists of nineteen files of source code, plus a makefile (for the Unix utility make) containing compilation instructions. The beginning of each file (except the makefile, which come first) is marked by a comment consisting of a line of asterisks, with the file's name embedded in it.

#

# Makefile for the minimal perfect hash algorithm.

#

# Directives:

#       phf                     Make phf, a program to generate a MPHF.

#

#       regen_mphf.a            Make an object code library capable of

#                               regenerating an MPHF from the specification

#                               file generated by phf.

#

#       regen_driver            Make regen_driver, a program to test

#                               the code in regen_mphf.a

#

#       all     (default)       Make the three above items.

#

#       regression             Execute a regression test. The phf program

#                              should terminate indicating success. The

#                              regen_driver program silently checks its

#                              results; no news is good news.

#

#       lint                   Various flavors of consistency checking.

#       lint_phf

#       lint_regen

#

#

COMMON_OBJS=    compute_hfns.o randomTables.o pmrandom.o support.o

COMMON_SRCS=    compute_hfns.c randomTables.c pmrandom.c support.c

COMMON_HDRS=    compute_hfns.h randomTables.h pmrandom.h support.h \

const.h types.h

MPHF_OBJS=      main.o mapping.o ordering.o searching.o vheap.o

MPHF_SRCS=      main.c mapping.c ordering.c searching.c vheap.c

MPHF_HDRS=      vheap.h

REGEN_OBJS=     regen_mphf.o

REGEN_SRCS=     regen_mphf.c

REGEN_HDRS=     regen_mphf.h

RD_OBJS=        regen_driver.o

RD_SRCS=        regen_driver.c

PHFLIB=         regen_mphf.a

CFLAGS= -O

LDFLAGS=

LIBS=   -lm

all: phf regen_driver

phf: $(PHFLIB) $(MPHF_OBJS)

$(CC) -o phf $ (LDFLAGS) $ (MPHF_OBJS) $ (PHFLIB) -lm

$(PHFLIB):       $(REGEN_OBJS) $ (COMMON_OBJS)

ar r $ (PHFLIB)  $?

ranlib $ (PHFLIB)

regen_driver:     $(RD_OBJS) $ (PHFLIB)

$(CC) $ (LDFLAGS) -o regen_driver $(RD_OBJS) $(PHFLIB) $(LIBS)

regression: phf regen_driver

. /phf keywords 'wc -l < keywords' 0.8 /tmp/hashing-output

. /regen_driver /tmp/hashing-output keywords > /tmp/hashed-words

rm /tmp/hashing-output /tmp/hashed-words

lint: lint_phf lint_regen

lint_phf:

lint $(MPHF_SRCS) $(COMMON_SRCS) $(LIBS)

lint_regen:

lint $(RD_SRCS) $(REGEN_SRCS) $(COMMON_SRCS) $(LIBS)

compute_hfns.o: const.h randomTables.h types.h

main.o:         const.h types.h support.h

mapping.o:      const.h pmrandom.h compute_hfns.h types.h support.h

ordering.o:     const.h types.h vheap.h support.h

randomTables.o: const.h pmrandom.h randomTables.h types.h

searching.o:    const.h pmrandom.h support.h types.h

support.o:      const.h types.h support.h

pmrandom.o:     pmrandom.h

vheap.o:        const.h support.h types.h vheap.h

/***************************  compute_hfns.h  ***************************

Purpose:      External declarations for computing the three h functions

associated with a key.

Provenance:   Written and tested by Q. Chen and E. Fox, March 1991.

Edited and tested by S. Wartik, April 1991.

Notes:        None.

**/

#ifdef __STDC__

extern void     compute_h012( int n, int r, randomTablesType tables,

char *key, arcType *arc );

#else

extern void     compute_h012();

#endif

/******************************  const.h  *******************************

Purpose:     Define globally-useful constant values.

Provenance:  Written and tested by Q. Chen and E. Fox, March 1991.

Edited and tested by S. Wartik, April 1991.

Notes:       None.

**/

#define MAX_INT ((unsigned)(~0)) >> 1   /* Maximum integer.    */

#define NP -1           /* Null pointer for array-based linked lists. */

#define NORM    0       /* Normal return.                      */

#define ABNORM -1       /* Abnormal return.                    */

#define MAPPINGS  4     /* Total number of mapping runs        */

#define SEARCHINGS 10   /* Total number of searching runs.     */

#define MAX_KEY_LENG COLUMNS   /* Maximum length of a key.     */

#define PRIMES 20              /* Number of primes, used in searching stage. */

#define NOTVISIT  0     /* Indication of an un-visited node.             */

#define VISIT     1     /* Indication of a visited node.                  */

#define EMPTY    '0'    /* Indication of an empty slot in the disk.       */

#define FULL     '1'    /* Indication of a filled slot in the disk.       */

/*****************************  pmrandom.h  *****************************

Purpose:      External declarations for random-number generator

package used by this program.

Provenance:   Written and tested by Q. Chen and E. Fox, March 1991.

Edited by S. Wartik, April 1991.

Notes:        The implementation is better than the random number

generator from the C library. It is taken from Park and

Miller's paper, "Random Number Generators: Good Ones are

Hard to Find," in CACM 31 (1988), pp. 1192-1201.

**/

#ifdef __STDC__

extern void      setseed(int);   /* Set the seed to a specified value.      */

extern int       pmrandom();     /* Get next random number in the sequence. */

extern int       getseed();      /* Get the current value of the seed.      */

#else

extern void      setseed();

extern int       pmrandom();

extern int       getseed();

#endif

#define DEFAULT_SEED     23

/**************************  randomTables.h  ****************************

Purpose:      External definitions for the three random number tables.

Provenance:   Written and tested by Q. Chen and E. Fox, March 1991.

Edited and tested by S. Wartik, April 1991.

**/

#define NO_TABLES 3      /* Number of random number tables.      */

#define ROWS 128         /* Rows of the random table (suitable for char). */

#define COLUMNS 150      /* Columns of the random table.         */

typedef int randomTablesType[NO_TABLES] [ROWS] [COLUMNS]; /* random number table */

#ifdef __STDC__

extern void     initialize_randomTable( randomTablesType tables, int *seed );

#else

extern void     initialize_randomTable();

#endif

/****************************  regen_mphf.h  *****************************

Purpose:     External declarations for regenerating and using

an already-computed minimal perfect hashing function.

Provenance:  Written and tested by Q. Chen and E. Fox, March 1991.

Edited and tested by S. Wartik, April 1991.

Notes:       None.

**/

typedef struct {

int         no_arcs;         /* Number of keys (arcs) in the key set.    */

int         no_vertices;     /* Number of vertices used to compute MPHF. */

int         seed;            /* The seed used for the random number tables. */

randomTablesType    tables;  /* The random number tables.                */

int         *gArray;         /* The array to hold g values.              */

} mphfType;

#ifdef __STDC__

extern int      regen_mphf ( mphfType *mphf, char *spec_file );

extern void     release_mphf ( mphfType *mphf );

extern int      retrieve ( mphfType *mphf, char *key );

#else

extern int      regen_mphf ();

extern void     release_mphf ();

extern int      retrieve ();

#endif

/*****************************  support.h  ******************************

Purpose:     External interface for support routines.

Provenance:  Written and tested by Q. Chen and E. Fox, March 1991.

Edited and tested by S. Wartik, April 1991.

Notes:       None.

**/

#ifdef __STDC__

extern char    *owncalloc(int n, int size);

extern char    *ownrealloc(char *area, int new_size);

extern void    write_gfun(arcsType *arcs, verticesType *vertices,

int tbl_seed, char *spec_file);

extern int     verify_mphf(arcsType *arcs, verticesType *vertices);

#else

extern char     *owncalloc();

extern char     *ownrealloc();

extern void     write_gfun();

extern int      verify_mphf();

#endif

/******************************  types.h  *******************************

Purpose:     Define globally-useful data types.

Provenance:  Written and tested by Q. Chen and E. Fox, March 1991.

Edited and tested by S. Wartik, April 1991.

Notes:       None.

**/

#include "const.h"

typedef struct arc {          /* arc data structure               */

int h0,                   /* h0 value                          */

h12[2];               /* h1 and h2 values                  */

struct arc *next_edge[2]; /* pointer to arc sharing same h1 or */

}  arcType;                   /* h2 values                         */

typedef struct {              /* vertex data structure               */

struct arc *first_edge;   /* pointer to the first adjacent edge  */

int  g,                   /* g value.                            */

prec,                /* backward pointer of the vertex-list */

succ;                /* forward pointer of the vertex-list  */

}  vertexType;

typedef struct {             /* arcs data structure         */

int no_arcs;             /* number of arcs in the graph */

arcType*arcArray;        /* arc array                   */

} arcsType;

typedef struct {             /* vertices data structure         */

int no_vertices,          /* number of vertices in the graph */

maxDegree,            /* max degree of the graph         */

vsHead,               /* VS list head                    */

vsTail,               /* VS list tail                    */

rlistHead;            /* remaining vertex list head      */

vertexType* vertexArray;  /* vertex array                    */

} verticesType;

typedef struct {             /* integer set data structure       */

int count,               /* number of elements in the set    */

*intSetRep;          /* set representation               */

} intSetType;

/******************************  vheap.h  *******************************

Purpose:     Define a "virtual heap" module.

Provenance:  Written and tested by Q. Chen and E. Fox, March l99l.

Edited and tested by S. Wartik, April l99l.

Notes:       This isn't intended as a general-purpose stack/heap

implementation. It's tailored toward stacks and heaps

of vertices and their degrees, using a representation suitable

for accessing them (in this case, an integer index into

the vertices->verex array identifies the vertex).

**/

#ifdef _STDC_

extern void      allocate_vheap( int no_arcs, int no_vertices );

extern void      initialize_vheap();

extern void      add_to_vheap ( vertexType *vertex, int degree );

extern int       max_degree_vertex ( vertexType **vertex );

extern void      free_vheap();

#else

extern void      allocate_vheap();

extern void      initialize_vheap();

extern void      add_to_vheap ();

extern int       max_degree_vertex ();

extern void      free_vheap();

#endif

/***************************** compute_hfns.c *****************************

Purpose:      Computation of the three h functions associated with

a key.

Provenance:   Written and tested by Q. Chen and E. Fox, March l99l.

Edited and tested by S. Wartik, April l99l.

Notes:        None.

**/

#include <stdio.h>

#include <string.h>

#include "types.h"

#include "randomTables.h"

/*************************************************************************

compute_h012( int, int, randomTablesType, char*, arcType* )

Return:       void

Purpose:      Compute the triple for a key. On return, the h0 and h12

fields of "arc" have the triple's values.

**/

void compute_h012(n, r, tables, key, arc)

int              n,           /* in: number of arcs.                     */

r;           /* in: size of h1 or h2 side of the graph. */

randomTablesType tables;      /* in: pointer to the random tables.       */

char             *key;        /* in: key string.                         */

arcType          *arc;        /* out: the key's arc entry.               */

{

int i,              /* Iterator over each table.                  */

j,              /* Iterator over each character in "key".     */

sum[NO_TABLES], /* Running sum of h0, h1 and h2 values.       */

length;         /* The length of "key".                       */

length = strlen(key) ;

sum[0] = sum[1] = sum[2] = 0 ;

for ( i = 0; i < NO_TABLES; i++ )         /* Sum over all the characters   */

for ( j = 0; j < length; j++ )        /* in the key.  */

sum[i] += table[i][(key[j]%ROWS)][j];

arc->h0     = abs( sum[0] ) % n;         /* Assign mappings for each    */

arc->h12[0] = abs( sum[1] ) % r;         /* of h0, h1, and h2 according */

arc->hl2[1] = abs( sum[2] ) % r + r;     /* to the sums computed.       */

}

/******************************** main.c *********************************

Purpose:     Main routine, driving the MPHF creation.

Provenance:  Written and tested by Q. Chen and E. Fox, March 1991.

Edited and tested by S. Wartik, April 1991.

Notes:       When compiled, the resulting program is used as follows:

phf I L R O

where:

I  Name of the file to be used as input. It should contain

one or more newline-terminated strings.

L  The number of lines in I.

R  A real number, giving a ratio between L and the size of

the hashing function generated. 1.0 is usually a viable

value. In general, L*R should be an integer.

O  Name of a file to be used as output. It will contain

the MPHF if one is found.

**/

#include <stdio.h>

#include <math.h>

#include <string.h>

#include "types.h"

#include "support.h"

#ifdef __STDC__

extern void   ordering( arcsType *arcs, verticesType *vertices );

extern void   allocate_arcs( arcsType* arcs, int n );

extern void   allocate_vertices( verticesType* vertices, int n );

extern void   free_arcs( arcsType* arcs );

extern void   free_vertices( verticesType* vertices );

extern void   exit();

#else

extern void   ordering();

extern void   allocate_arcs();

extern void   allocate_vertices();

extern void   free_arcs();

extern void   free_vertices();

extern void   exit();

#endif

/*************************************************************************

main(argc, argv)

Returns:     int -- zero on success, non-zero on failure.

Purpose:     Take the inputs and call three routines to carry out mapping,

ordering and searching three tasks. If they all succeed,

write the MPHF to the spec file.

**/

main( argc, argv )

int    argc;

char   *argv[ ];   /* arg1: key file;   arg2: key set size;  */

/* arg3: ratio;      arg4: spec file      */

{

int      status,     /* Return status variable.                  */

seed;       /* Seed used to initialize the three random */

/* tables.                                  */

char    *key_file_name,

*specification_file_name;

int     lines_in_keyword_file;

double  ratio;

arcsType       arcs;     /* These variables hold all the arcs  */

verticesType  vertices;  /* and vertices generated.            */

if ( argc != 5 ) {

fprintf(stderr, "Usage: %s keywords kw-lines ratio output-file\n",

argv[0]);

exit(1);

}

key_file_name = argv[1];

if ( (lines_in_keyword_file = atoi(argv[2])) <= 0 ) {

fputs("The 2nd parameter must be a positive integer. \n", stderr);

exit(1);

}

else if ( (ratio = atof(argv[3]) ) <= 0.0 ){

fputs ("The 3rd parameter must be a positive floating-point value.\n",

stderr);

exit(1);

}

allocate_arcs ( &arcs, lines_in_keyword_file );

allocate_vertices( &vertices, (int)(lines_in_ keyword_file * ratio) );

specification_file_name = argv[4];

if ( (status = mapping( key_file_name, &arcs, &vertices, &seed )) == NORM ) {

ordering( &arcs, &vertices );

if ( (status = searching( &arcs, &vertices )) == NORM &&

(status = verify_mphf( &arcs, &vertices )) == NORM )

write_gfun ( &arcs, &vertices, seed, specification_file_name );

}

free_arcs( &arcs );

free_vertices( &vertices );

fprintf(stderr, "MPHF creation %s.\n",

(status == NORM ? "succeeded" : "failed"));

return(status);

}

/*************************************************************************

allocate_arcs( arcsType*, int )

Returns:     void

Purpose:     Given an expected number of arcs, allocate space for an arc data

structure containing that many arcs, and place the number of

arcs in the "no_arcs" field of the arc data structure.

**/

void allocate_arcs( arcs, n )

arcsType   *arcs;          /* out: Receives allocated storage.  */

int        n;              /* in: Expected number of arcs.      */

{

arcs->no_arcs = n;

arcs->arcArray = (arcType*) owncalloc( sizeof(arcType), n );

}

/*************************************************************************

allocate_vertices( verticesType* , int )

Purpose:     Given an expected number of vertices, allocate space for a vertex

data structure containing that many vertices, and place the

number of vertices in the "no_vertices" field of the vertex

data structure.

**/

void allocate_vertices( vertices, n )

verticesType        *vertices;     /* out: Receives allocated storage. */

int                 n;             /* in: Expected number of vertices. */

{

if (n % 2 != 0) n++;

vertices->no_vertices = n;

vertices->vertexArray = (vertexType*) owncalloc(sizeof(vertexType), n );

}

/*************************************************************************

free_arcs( arcsType* )

Purpose:   Deallocate space for an arc data structure.

**/

void free_arcs( arcs )

arcsType   *arcs;              /* in out: Space to de-allocate.  */

{

free( (char *)arcs->arcArray );

}

/*************************************************************************

free_vertices( verticesType* )

Purpose:   Deallocate space for a vertex data structure.

**/

void free_vertices( vertices )

verticesType            *vertices;     /* in out: Space to de-allocate.    */

{

free( (char *)vertices->vertexArray );

}

/****************************** mapping.c *******************************

Purpose:     Implement the mapping stage of the MPHF algorithm.

Provenance:  Written and tested by Q. Chen and E. Fox, March 1991.

Edited and tested by S. Wartik, April 1991.

Notes:       To save space, the "g" field of the vertex structure is used

here to hold a vertex's degree.

**/

#include <stdio.h>

#include "types.h"

#include "pmrandom.h"

#include "randomTables.h"

#include "compute_hfns.h"

#ifdef__STDC__

extern void   initialize_arcs( arcsType *arcs );

extern void   initialize_vertices( verticesType *vertices );

extern int    check_dup( arcType *firstArc );

extern int    construct_graph( arcsType *arcs, verticesType *vertices );

extern void   map_to_triples( char *key_file, arcsType *arcs,

int r, randomTablesType tables );

extern void   exit( int status );

#else

extern void   initialize_arcs();

extern void   initialize_vertices();

extern int    check_dup();

extern int    construct_graph();

extern void   map_to_triples();

extern void   exit();

#endif

/*************************************************************************

mapping( char*, arcsType*, verticesType*, int )

Return:      int -- NORM if a mapping can be found, ABNORM if not.

Purpose:     Perform the mapping stage: Map all keys to triples and construct

the bipartite graph. This involves:

-- Allocating the arcs and vertices structures.

-- Generating the h0, hl, and h2 functions.

-- Building the lists of edges, ordered by degree.

**/

int mapping( key_file, arcs, vertices, seed )

char         *key_file;     /* in: name of file containing keys.   */

arcsType     *arcs;         /* out: arcs in bipartite graph.       */

verticesType *vertices;     /* out: vertices in bipartite graph.   */

int          *seed;         /* out: seed selected to initialize the */

/* random tables.   */

{

int mapping_tries = 0;

randomTablesType randomTables;      /* Three random number tables.      */

while ( mapping_tries++< MAPPINGS ) {

initialize_arcs( arcs );

initialize_vertices( vertices );

initialize_randomTable( randomTables, seed );

map_to_triples( key_file, arcs, vertices->no_vertices/2, randomTables );

if ( construct_graph(arcs, vertices) == NORM )

return(NORM);

fputs((mapping_tries < MAPPINGS ? "Trying again.\n" : "Giving up.\n"),

stderr);

}

return(ABNORM);

}

/*************************************************************************

map_to_triples( char*, arcsType*, int, randomTablesType )

Return:      void

Purpose:     Compute triples of (h0, hl, h2) for all keys and store

them in the arc data structure.

**/

void map_to_triples(key_file, arcs, r, tables)

char         *key_file;     /* in: key file name             */

arcsType     *arcs;         /* out: the arcs data structure  */

int          r;             /* in: size of hl or h2 side     */

randomTablesType tables;    /* in: random number tables      */

{

FILE         *fp;                  /* Input file pointer.    */

int          i = 0;                /* Iterator over arcs.    */

char         string[MAX_KEY_LENG]; /* Key string holder.     */

if ( (fp = fopen(key_file, "r")) == NULL ) {

fprintf(stderr, "Can't read \"%s\".\n", key_file);

exit(1);

}

while ( fgets( string, MAX_KEY_LENG, fp ) != NULL &&

i <arcs->no_arcs)  {

string[strlen(string)-1 = '\0';        /* Exclude the '\n'. */

compute_h012 ( arcs->no_arcs, r, tables, string, &arcs->arcArray[i++] );

}

if ( i != arcs->no_arcs) {

fprintf(stderr, "File \"%s\" contains %d keys, not %d. ",

key_file, i, arcs->no_arcs);

fputs("Re-execute with correct value.\n", stderr);

exit(1);

}

else if ( ! feof(fp ) ) {

fprintf(stderr, "File \"%s\" contains more than %d keys. ",

key_file, arcs->no_arcs);

fputs("Re-execute with correct value.\n", stderr);

exit(1);

}

fclose(fp);

}

/********************************8****************************************

construct_graph( arcsType*, verticesType* )

Return:      int -- NORM if a graph can be built without duplicate arcs,

ABNORM if it can't.

Purpose:     Construct the bipartite graph out of triples. On successful

return,

-- Each vertex's degree has been determined, and placed

in its "g" field.

-- The maximal degree of the graph has been determined.

-- The "first_edge" field of vertices is a linked list

of adjacent edges.

**/

int construct_graph( arcs, vertices )

arcsType     *arcs;         /* in out: arcs.                      */

verticesType *vertices;     /* in out: vertices.                  */

{

int i,            /* Iterator over all arcs.                      */

j,            /* j = 0 and 1 for h1 and h2 side, respectively */

status = NORM,

vertex;

for ( j = 0; j < 2; j++ ) {

for ( i = 0; i < arcs->no_arcs; i++ ) {

vertex = arcs->arcArray[i].h12[j];  /* Update vertex degree */

vertices->vertexArray[vertex].g++;  /* count and vertex     */

arcs->arcArray[i].next_edge[j] =    /* adjacency list.      */

vertices->vertexArray[vertex].first_edge;

vertices->vertexArray[vertex].first_edge = &arcs ->arcArray[i];

if ( (j == 0) && check_dup( &arcs->arcArray[i] ) == ABNORM )  {

fputs("Duplicate found.\n", stderr);

status = ABNORM;

break;

}

/* Figure out the maximal degree of the graph. */

if ( vertices->vertexArray[vertex].g > vertices->maxDegree )

vertices->maxDegree = vertices->vertexArray[vertex].g;

}

}

return(status);

}

/************************************8************************************

check_dup( arcType )

Return:      int -- NORM if no duplicate triple exists, ABNORM if one does.

Purpose:     Test if some arc on the arc list has an identical triple to

the first arc on the list.

**/

int check_dup(firstArc)

arcType *firstArc;  /* in: arc at the head of a list.      */

{

arcType *arc = firstArc->next_edge[0];

while ( arc != 0 ) {

if ( ( firstArc->h0 == arc->h0 ) &&

( firstArc->h12[1] == arc->h12[1] ) )

return(ABNORM);                         /* Duplication found. */

arc = arc->next_edge[0];

}

return(NORM);  /* No duplication.              */

}

/*************************************************************************

initialize_arcs( arcsType* )

Return:      void

Purpose:     Make the edge pointers of each arc nil.

**/

void initialize_arcs( arcs )

arcsType *arcs;          /* out: arcs structure.    */

{

int i;

for ( i = 0; i < arcs->no_arcs; i++ ) {

arcs->arcArray[i].next_edge[0] = 0;

arcs->arcArray[i].next_edge[1] = 0;

}

}

/*************************************************************************

initialize_vertices( verticesType* )

Return:       void

Purpose:     For each vertex, set the degree to 0 and make the

edge list empty.

**/

void initialize_vertices( vertices )

verticesType *vertices;     /* out: vertex structure.     */

{

int i;

vertices->maxDegree = 0;

for ( i = 0; i < vertices->no_vertices; i++ ) {

vertices->vertexArray[i].first_edge = 0;

vertices->vertexArray[i].g = 0;

}

}

/*********************************** ordering.c *************************

Purpose:     Implement the ordering stage of the MPHF algorithm.

Provenance:  Written and tested by Q. Chen and E. Fox, March 1991.

Edited and tested by S. Wartik, April 1991.

Notes:       None.

**/

#include <stdio.h>

#include "types.h"

#include "support.h"

#include "vheap.h"

#ifdef __STDC__

extern void   delete_from_rList( vertexType *vertex, verticesType *vertices );

extern void   append_to_VS( vertexType *vertex, verticesType *vertices );

extern void   initialize_rList( verticesType *vertices );

#else

extern void   delete_from_rList();

extern void   append_to_VS();

extern void   initialize_rList();

#endif

/*************************************************************************

ordering( arcs, vertices )

Return:      void

Purpose:     Generate an ordering of the vertices.

Notes:       The ordering of the vertices is a linked list, the head

of which is in vertices->vsList. The "next element"

pointer for each node is in the "succ" field of each

vertex component. Note that the "succ" field has two

purposes in this step. One is that just mentioned. The

other is to be part of the rList used in this step.

**/

void ordering( arcs, vertices )

arcsType            *arcs;     /* in out: the arcs data structure.  */

verticesType        *vertices; /* in out: the vertices data structure.  */

{

int         degree,

side;                /* Indicates side of graph.   */

vertexType  *vertex;

arcType     *arc;

vertices->vsHead = vertices->vsTail = NP;    /* Initialize the VS list.  */

initialize_rList( vertices );

allocate_vheap( arcs->no_arcs, vertices->no_vertices );

while ( vertices->rlistHead != -l ) {       /* Process each component graph. */

initialize_vheap();

vertex = &vertices->vertexArray[vertices->rlistHead];

do {

vertex->g = 0;   /* Mark node "visited". */

delete_from_rList( vertex, vertices );

append_to_VS( vertex, vertices );

if ( vertex->first_edge != 0 ) {

/* Add adjacent nodes that are not visited and   */

/* not in virtual heap to the virtual heap.      */

side = vertex - vertices->vertexArray >= vertices->no_vertices/2;

arc = vertex->first_edge;

while ( arc != 0 ) {

int adj_node; /* Node adjacent to vertex.     */

adj_node = arc->h12[(side+1)%2];

degree = vertices->vertexArray[adj_node].g;

if ( degree > 0 ) {   /* One such node is found. */

add_to_vheap( &vertices->vertexArray[adj_node], degree );

vertices->vertexArray[adj_node].g *= -1;

}

arc = arc->next_edge[side];

}

}

} while ( max_degree_vertex( &vertex ) == NORM );

}

free_vheap();

}

delete_from_rList( vertex, vertices )

Return:      void

Purpose:     Delete a vertex pointing at by vertex from the rList stored

in the vertices data structure.

**/

void delete_from_rList( vertex, vertices )

vertexType     *vertex;    /* in: vertex to delete.    */

verticesType   *vertices;   /* out: vertices data structure.  */

{

if ( vertex->prec != NP )

vertices->vertexArray[vertex-prec]. succ = vertex ->succ;

else

vertices->rlistHead = vertex->succ;

if ( vertex->succ != NP )

vertices->vertexArray[vertex->succ].prec = vertex->pre>c;

}

/*********************************************************************

append_to_VS( vertex, vertices )

Return:      void

Purpose:     Append the vertex to the vertex ordering VS.

**/

void append_to_VS( vertex, vertices )

vertexType *vertex;         /* in: the vertex to be added.       */

verticesType *vertices;     /* out: the vertices data structure. */

{

int newTail = vertex - vertices->vertexArray;

vertex->succ = vertex->prec = NP;

if ( vertices->vsHead == NP )

vertices->vsHead = newTail;

else

vertices->vertexArray[vertices->vsTail].succ = newTail;

vertices->vsTail = newTail;

}

/*************************************************************************

initialize_rList( vertices )

Return:      void

Purpose:     Set up an rList from the vertices. An rList is a

doubly-linked list of vertices in decending order of

degree.

Notes:        pred and succ are used to store the list.

**/

void initialize_rList( vertices )

verticesType *vertices;      /* in out: vertices to be ordered.  */

{

int        i, j, previous;

intSetType heads,  /* Two sets of pointers. Element i of "heads" points at */

tails;  /* the head of a list about degree i, 0<=i<=maxDegree.  */

/* The elements of "tails" are the corresponding tails. */

heads.count = vertices->maxDegree + 1;

heads.intSetRep = (int*)owncalloc( heads.count, sizeof(int) );

for ( i = 0; i < heads.count; i++ )

heads.intSetRep[i] = NP;

tails.count = vertices->maxDegree + 1;

tails.intSetRep = (int*)owncalloc( tails.count, sizeof(int) );

for ( i = 0; i < tails.count; i++ )

tails.intSetRep[i] = NP;

/* Construct lists for vertices being of */

/* degree 0, 1, ... maxDegree.        */

for ( i = 0; i < vertices->no_vertices; i++ ) {

previous = heads.intSetRep[vertices->vertexArray[i].g];

vertices->vertexArray[i].succ = previous;

if ( previous != NP )

vertices->vertexArray[previous].prec = i;

else

tails.intSetRep[vertices->vertexArray[i].g] = i;

heads.intSetRep[vertices->vertexArray[i].g] = i;

vertices->vertexArray[i].prec = NP;

}

/* Construct the rList by linking lists for vertices being of  */

/* degree 0, 1, ... maxDegree.                                 */

for ( i = heads.count - 1; i > 1; i -- )

if ( tails.intSetRep[i] != NP ) {

for ( j = i - 1; j >= 1; j-- )

if ( heads.intSetRep[j] ! = NP )

break;

if ( j >= 1 ) {

vertices->vertexArray[tails.intSetRep[i]].succ =

heads.intSetRep[j];

vertices->vertexArray[heads.intSetRep[j]].prec =

tails.intSetRep[i];

}

}

vertices->rlistHead = heads.intSetRep[vertices-> maxDegree];

free( (char *)heads.intSetRep );

free( (char *)tails.intSetRep );

}

/********************************* pmrandom.c ***************************

Purpose:     Implement a random-number generator package for this program.

Provenance:  Written and tested by Q. Chen and E. Fox, March 1991.

Edited by S. Wartik, April 1991.

Notes:       It is assumed that the C data type "int" can store

32-bit quantities.

**/

#include "pmrandom.h"

static int seed = DEFAULT_SEED;  /* The seed of the random number generator. */

/*************************************************************************

setseed(int)

Returns:     int

Purpose:     Set the seed for the random number generator.

Plan:        Uses a formula suggested by Park and Miller. See above.

Notes:       None.

**/

void setseed( new_seed )

int new_seed;

{

int low, high, test;

if ( (new_seed < 1) || (new_seed > 2147483646) )

new_seed = DEFAULT_SEED;

high = new_seed / 127773;        /* 127773 = 2147483647 div 16807 */

low  = new_seed % 127773;

test = 16807 * low - 2836 * high;  /* 2836 = 2147483647 mod 16807 */

seed = ( test > 0 ) ? test : test + 2147483647;

}

/*************************************************************************

pmrandom()

Returns:     void

Purpose:     Return the next random number in the sequence.

Plan:        Uses the formula:

f() = ( 16807 * seed ) mod 2147483647.

The value of "seed" must be within [1, ..., 2147483646].

Notes:       None.

**/

int pmrandom()

{

int tmp = seed;

setseed(seed);

return(tmp);

}

/*************************************************************************

getseed()

Returns:     int

Purpose:     Get the current value of the seed.

Notes:       None.

**/

int getseed()

{

return (seed);

}

/************************** randomTables.c ******************************

Purpose:     Routines for handling the random number tables.

Provenance:  Written and tested by Q. Chen and E. Fox, March 1991.

Edited and tested by S. Wartik, April 1991.

Notes:       None.

**/

#include "types.h"

#include "pmrandom.h"

#include "randomTables.h"

/**************************************************************************

initialize_randomTable(randomTablesType,int)

Return:      void

Purpose:     Initialize the three random number tables and return the

seed used.

**/

void initialize_randomTable(tables,seed)

randomTablesType tables;    /*out: Tables of random numbers.       */

int              *seed;     /*out: seed used to initialize tables. */

{

int i, j, k;             /*Iterators over the tables.          */

*seed = getseed();

setseed(*seed);

for (i=0; i < NO_TABLES; i++)    /*Initialize the tables.      */

for (j=0; j < ROWS; j++)

for (k = 0; k < COLUMNS; k++)

tables[i][j][k] = pmrandom();

}

/***************************** regen_driver.c ***************************

Purpose:     A program to test regenerating and using a precomputed hashing

function.

Provenance:  Written and tested by Q. Chen and E. Fox, April 1991.

Edited and tested by S. Wartik, April 1991.

Notes:       The program is used as follows:

regen_driver mphf-file keyword-file

The result is a set of lines, written to stdout, indicating

the bucket of each keyword in the keyword file.

**/

#include < stdio.h >

#include < string.h >

#include < math.h >

#include "types.h"

#include "randomTables.h"

#include "regen_mphf.h"

#ifdef _STDC_

extern void   retrieveAll ( mphfType *mphf, char *key_file );

extern void   exit( int status );

#else

extern void   retrieveAll ();

extern void   exit();

#endif

/*************************************************************************

main( int, char** )

Return:      Nothing.

Purpose:     See the header for this file.

**/

main( argc, argv )

int argc;

char *argv[ ];   /* arg1: mphf file; arg2: key file */

mphfType mphf;

if ( argc != 3 ) {

fprintf(stderr, "Usage: %s mphf-file key-file\n", argv[0]);

exit(1);

}

if ( regen_mphf ( &mphf, argv[1] ) == NORM )

retrieveAll ( &mphf, argv[2] );

else {

fprintf(stderr, "Can't regenerate hashing function from \"%s\".\n",

argv[1]);

exit(1);

}

release_mphf ( &mphf );

exit(0);

}

/*************************************************************************

retrieveAll( mphfType*, char* )

Return:      void

Purpose:     Given a file of keys and a structure describing a

MPHF previously computed for those keys, print

each key's location on the standard output stream.

**/

void retrieveAll( mphf, key_file )

mphfType     *mphf;                  /* in: mphf specification.    */

char         *key_file;              /* in: the key file.          */

{

FILE         *fp;                    /* Handle for specification file.  */

char         string [MAX_KEY_LENG];  /* Key string.                */

int         hash;                    /* Computed hash value.        */

int         max_bucket_length;       /* The maximum number of chars */

/* needed to represent a bucket */

/* index as a string.          */

if ( (fp = fopen(key_file, "r")) == 0 ) {

fprintf (stderr, "Can't read file \"%s\ ". \n", key_file);

exit(1);

}

max_bucket_length = (int)log10((double)mphf->no_arcs) + 1;

while ( fgets( string, MAX_KEY_LENG, fp ) != 0 ) {

string[strlen(string)-1] = '\0';

hash = retrieve( mphf, string );

printf("Bucket %*d: %s\n", max_bucket_length, hash, string);

}

fclose(fp);

}

/****************************** regen_mphf.c ****************************

Purpose:     Routines to regenerate and use a previously-computed

minimal perfect hashing function.

Provenance:  Written and tested by Q. Chen and E. Fox, March 1991.

Edited and tested by S. Wartik, April 1991.

Notes:       None.

**/

#include <stdio.h>

#include "types.h"

#include "randomTables.h"

#include "regen_mphf.h"

#include "compute_hfns.h"

/*************************************************************************

regen_mphf( mphfType*, char* )

Return:      int -- NORM if the MPHF could be reconstructed,

ABNORM if it couldn't.

Purpose:     Regenerate a MPHF from a specification file.

Notes:       What is regenerated is the table of random numbers. The

retrieve() procedure can use these numbers to re-create

the h0, h1 and h2 values, and from that, the hash value.

If the specification file doesn't seem to correspond

to the expected format, ABNORM is returned. However,

there is no way to tell what caused the error.

**/

int regen_mphf( mphf, spec_file_name )

mphfType    *mphf;               /* out: the regenerated MPHF structure. */

char        *spec_file_name;     /* in: MPHF specification file.  */

{

int         i;                   /* Iterator through vertices.    */

FILE        *spec_file;

if ( (spec_file = fopen(spec_file_name, "r")) == NULL ) return ABNORM;

if ( fscanf(spec_file, "%d\n%d\n%d\n",

&mphf->no_arcs, &mphf->no_vertices, &mphf->seed) != 3 ) {

fclose(spec_file);

return ABNORM;             /* File is improperly formatted.  */

}

mphf->gArray = (int*) owncalloc( mphf->no_vertices, sizeof(int) );

for ( i = 0; i < mphf->no_vertices; i++)

if ( fscanf(spec_file, "%d\n", &mphf->gArray[i] ) != 1 {

fclose(spec_file);

return ABNORM;    /* File is improperly formatted.  */

}

if ( ! feof(spec_file) ) {

fclose(spec_file);

return ABNORM;       /* File is improperly formatted.  */

}

initialize_randomTable( mphf->tables, &mphf->seed );

fclose(spec_file);

return NORM;

}

/*************************************************************************

release_mphf( mphfType*, char* )

Return:      void

Purpose:     Release the dynamically-allocated storage associated with

an MPHF.

**/

void release_mphf( mphf )

mphfType *mphf;        /* in out: pointer to the MPHF structure. */

{

free( (char *)mphf->gArray );

}

/*************************************************************************

retrieve( mphfType*, char* )

Return:      int -- a value in the range 0..mphf-no_arcs-1.

Purpose:     Given an MPHF and a key, return the key's hash value.

**/

int retrieve( mphf, key )

mphfType     *mphf;     /* in: the mphf specification.               */

char         *key;      /* in: the key, terminated by a null character. */

{

int          hash;      /* The computed hash value.                 */

arcType      arc;       /* Storage used to hold the h0, h1 and h2 values. */

compute_h012(mphf->no_arcs, (mphf->no_vertices) / 2,

mphf->tables, key, &arc);

hash = abs(arc.h0 +

mphf->gArray[arc.h12[0]] +

mphf->gArray[arc.h12[1]]

) % mphf->no_arcs;

return hash;

}

/*************************** searching.c ***********************************

Purpose:     Implement the searching stage of the MPHF algorithm.

Provenance:  Written and tested by Q. Chen and E. Fox, March 1991.

Edited and tested by S. Wartik, April 1991.

Notes:       The other two stages must have been performed already.

**/

#include <stdio.h>

#include "types.h"

#include "pmrandom.h"

#include "support.h"

#ifdef __STDC__

extern int    fit_pattern( arcsType* arcs, verticesType* vertices, int i,

char* disk, intSetType *primes, intSetType* slotSet );

extern void   initialize_search( arcsType* arcs, verticesType* vertices, char* disk );

extern void   initialize_primes( int n, intSetType* primes );

#else

extern int    fit_pattern();

extern void   initialize_search();

extern void   initialize_primes();

#endif

/*************************************************************************

searching( arcsType*, verticesType* )

Return:      int -- NORM on success, ABNORM on failure.

Purpose:     Search a MPHF for the key set.

Notes:       The "prec" field is used as the "vertex visited" marker.

The slotSet variable actually is only used in fit_pattern().

However, since storage for it must be dynamically allocated,

and since this routine calls fit_pattern() repeatedly,

it's declared here, where storage can be allocated just once.

**/

int searching( arcs, vertices )

arcsType            *arcs;

verticesType        *vertices;

{

int         i,                    /* Each vertex in the VS list.     */

searching_tries = 0,  /* Running count of searching tries.*/

status = ABNORM;      /* Condition variable.             */

char        *disk;                /* Simulated hash table.           */

intSetType  primes,               /* Table of primes for pattern shifts. */

slotSet;              /* Set of hash addresses.           */

disk = (char*) owncalloc( arcs->no_arcs, sizeof(char) );

slotSet.intSetRep = (int*) owncalloc( vertices->maxDegree, sizeof(int) );

initialize_primes( arcs->no_arcs, &primes );

while ( (searching_tries++ < SEARCHINGS) && (status == ABNORM) ) {

status = NORM;

i = vertices->vsHead;           /* Get the highest-level vertex.  */

initialize_search( arcs, vertices, disk );

while ( i != NP ) {  /* Fit keys of level of vertex i onto the disk.   */

vertices->vertexArray[i].prec = VISIT;

if ( fit_pattern(arcs, vertices, i, disk, &primes, &slotSet)== ABNORM )

status = ABNORM;        /* Search failed at vertex i. Try  */

break;                  /* a new pattern.                  */

}

else                  /* Search succeeded. Proceed to next node.  */

i = vertices->vertexArray[i].succ;

}

}

free( disk );

free( (char *)slotSet.intSetRep );

free( (char *)primes.intSetRep );

return(status);

}

/*************************************************************************

fit_pattern( arcsType*, verticesType*, int, char*,

intSetType*, intSetType* )

Return:      int -- NORM if a fit is found, ABNORM if not.

Purpose:     Compute a pattern for a level and fit it onto the hash table.

If a pattern is found, then the g values for vertices on that

level are set appropriately, and the slots on the disk for

the vertices are filled.

**/

int fit_pattern( arcs, vertices, i, disk, primes, slotSet )

arcsType         *arcs;     /* in: The arcs in the graph.             */

verticesType     *vertices; /* in out: The vertices in the graph.     */

int              i;         /* in: Vertex's location in vertex-selected list. */

char             *disk;     /* in out: The hash table (disk).         */

intSetType       *primes,   /* in: Prime number table                 */

*slotSet;  /* Set of slots taken by keys in this pattern.   */

{

arcType  *arc;              /* Current arc.                      */

int      shift,             /* Shift value for the pattern.      */

side,             /* Side indicator (0 or 1).          */

hashAddress       /* Hash address being tried.         */

fitOK = ABNORM,   /* Fit condition variable.           */

no_fits = 0;      /* Running count of attempts to fit. */

side = (i >= vertices->no_vertices/2);

shift = primes->intSetRep[pmrandom () % primes->count];

while ((no_fits++ < arcs->no_arcs) && (fitOK == ABNORM) ) {

fitOK = NORM;

slotSet->count = 0;          /* Initialize slot set to empty.     */

arc = vertices->vertexArray[i].first_edge;

while ( arc != 0 ) { /* Iterate over all arcs in this level. */

/* If the key for arc is at this level,    */

/* get its hash address.   */

if ( vertices->vertexArray[arc->hl2[(side+l)%2]].prec == VISIT ) {

hashAddress = abs(arc-h0 +

vertices->vertexArray[arc->h12[0]].g +

vertices->vertexArray[arc->h12[1]].g

) % arcs->no_arcs;

/* See if this key can be put at hashAddress. */

if ( disk[hashAddress] != EMPTY ) {     /* Collision. Clear   */

int k;                              /* marked slots in disk. */

for ( k = 0; k < slotSet->count; k++ )

disk[slotSet->intSetRep[k]] = EMPTY;

/* Try a new shift. */

vertices->vertexArray[i].g =

( vertices->vertexArray[i].g + shift ) % arcs->no_arcs;

fitOK = ABNORM;

break;

}

else {  /* Success. Remember the address, */

/* and mark the table.            */

slotSet->intSetRep[slotSet->count++] = hashAddress;

disk[hashAddress] = FULL;

}

} /* end of if */

arc = arc->next_edge[side];    /* Hash next arc.  */

} /* end of inner while */

} /* end of outer while */

return(fitOK);

}

/*************************************************************************

initialize_search( arcsType*, verticesType*, char* )

Return:      void

Purpose:     Prepare for the search stage: Put random values in all

the g fields, mark all vertices un-visited, and empty the disk.

**/

void

initialize_search( arcs, vertices, disk )

arcsType     *arcs;          /* in: arcs.             */

verticesType *vertices;      /* out: vertices.        */

char         *disk;          /* out: the hash table.  */

{

int i;

setseed ( pmrandom () );                             /* Set the seed.   */

for ( i = 0; i < vertices->no_vertices; i++ ) {

vertices->vertexArray[i].g = pmrandom() % arcs->no_arcs;

vertices->vertexArray[i].prec = NOTVISIT;

}

         /* Reset the hash table. */

for ( i = 0; i < arcs->no_arcs; disk[i++] = EMPTY );

}

/*************************************************************************

initialize_primes( int, intSetType* )

Return:      void

Purpose:     Set up the prime number table.

**/

void

initialize_primes( n, primes )

int n;                /* in: the size of the hash table.  */

intSetType *primes;   /* out: the prime number table.     */

{

int i,

testingNumber = 2;    /* Testing number for possible prime numbers. */

primes->intSetRep = (int*) owncalloc( PRIMES, sizeof(int) );

primes->intSetRep[0] = 1;        /* 1 is added to the table, although it */

primes->count = l;               /* is not a prime.                */

while ( (testingNumber++ < n) && (primes->count < PRIMES) ) {

if ( n % testingNumber != 0 ) {              /* Get first PRIMES-l*/

for ( i = testingNumber - l; i> 0; i--)          /* prime numbers. */

if ( testingNumber % i == 0 )

break;

if ( i == 1 ) primes->intSetRep[primes->count++] = testingNumber;

} /* end of if */

} /* end of while */

}

/*************************** support.c ***********************************

Purpose:     Provide some useful support routines:

-- Storage allocators that exit on error (since this

isn't a subroutine library, there's no need for

fancy error-handling).

-- A routine to write the MPHF to a file.

-- A routine to verify the correctness of a MPHF.

Provenance:  Written and tested by Q. Chen and E. Fox, March 1991.

Edited and tested by S. Wartik, April 1991.

Notes:       None.

**/

#include < stdio.h >

#include "types.h"

#ifdef __STDC__

extern char   *malloc( unsigned int size );

extern char   *realloc ( char *area, unsigned int size );

extern void   exit();

#else

extern char   *malloc(),

*realloc();

extern void   exit();

#endif

/*************************************************************************

owncalloc( n, size )

Return:      char * -- Pointer to a chunk of memory.

Purpose:     Allocate a chunk of memory of 'n' elements each of size 'size'.

Return the pointer to the chunk. Abort if no space is available.

**/

char  *owncalloc( n, size )

int          n,      /* in: number of elements.   */

size;   /* in: size of each element. */

{

char         *temp;

if (  (temp = malloc( (unsigned int)(n*size) )) == 0 ) {

fputs("Panic: cannot allocate memory.\n", stderr);

exit(1);

}

return(temp);

}

/*************************************************************************

ownrealloc( n, size )

Return:      char * -- Pointer to a chunk of memory.

Purpose:     Re-allocate a chunk of memory pointed to by area -- make it

new_size bytes. Abort if no space is available.

**/

char  *ownrealloc( area, new_size )

char         *area;            /* in: area to re-allocate.  */

int          new_size;         /* in: new_size.             */

{

char         *temp;

if ( (temp = realloc( area, (unsigned)new_size )) == 0 ) {

fputs("Panic: cannot reallocate memory.\n", stderr);

exit(1);

}

return(temp);

}

/*************************************************************************

write_gfun( arcs, vertices, tbl_seed, spec_file )

Return:      void

Purpose:     Write the MPHF specification to a file

**/

void

write_gfun( arcs, vertices, tbl_seed, spec_file )

arcsType        *arcs;       /* in: the arcs.                      */

verticesType    *vertices;   /* in: the vertices.                  */

int             tbl_seed;    /* in: seed used to set up random number tables. */

char            *spec_file;  /* in: name of the specification file. */

{

int        i;                /* Iterator through vertices.         */

FILE       *fp;              /* Handle for specification file.     */

if ( (fp = fopen(spec_file, "w")) == NULL ) {

fprintf(stderr, "Can't create hashing specification file \"%s\".\n",

spec_file);

exit(1);

}

fprintf(fp, "%d\n%d\n%d\n",

arcs->no_arcs, vertices->no_vertices, tbl_seed);

for ( i = 0; i < vertices->no_vertices; i++ )

fprintf(fp,"%d\n", vertices->vertexArray[i].g);

fclose(fp);

}

/*************************************************************************

verify_mphf( arcs, vertices )

Return:      int -- NORM if MPHF is correct, ABNORM if not.

Purpose:     Verify the computed MPHF is indeed minimal and perfect

**/

int verify_mphf( arcs, vertices )

arcsType     *arcs;              /* in: the arcs.         */

verticesType *vertices;          /* in: the vertices.     */

{

int        i,

status = NORM,

hash;                 /* Hash value of a key.  */

char       *disk;                /* Hash table.           */

disk = owncalloc( arcs->no_arcs, sizeof(char) );

for ( i = 0; i < arcs->no_arcs; disk[i++] = EMPTY );

for ( i = 0; i < arcs->no_arcs; i++ ) {

hash = abs ( arcs->arcArray[i].h0 +

vertices->vertexArray[arcs->arcArray[i].h12[0]].g +

vertices->vertexArray[arcs->arcArray[i].h12[1]].g

)% arcs->no_arcs ;

if ( hash < 0 ) {

fprintf(stderr, "Panic: negative hash value.\n");

status = ABNORM;

break;

}

if ( disk[hash] == FULL ) {

fprintf(stderr, "Panic: hash entry collided at");

fprintf(stderr, " position %d by the %dth word!\n", hash, i);

status = ABNORM;

break;

}

else

disk[hash] = FULL;

}

free( (char *)disk );

return(status);

}

/******************************* vheap.c ********************************

Purpose:     Implement a "virtual heap": a combination of stacks and

a heap.

Provenance:  Written and tested by Q. Chen and E. Fox, March 1991.

Edited and tested by S. Wartik, April 1991.

Notes:       The point of the combination is that a stack is a more

efficient data structure. Vertices of low degree

(specifically, those <= NO_STACKS) are stored in stacks,

since they are more common. Vertices of high degree are

stored in the heap.

**/

#include <math.h>

#include <stdio.h>

#include "types.h"

#include "support.h"

#include "vheap.h"

#define NO_STACKS 6     /* The number of stacks in use.            */

#define DEF_SIZE 10     /* The default size of a heap or a stack.  */

typedef struct {           /* Stack data structure.                   */

int stackTop,          /* Stack top.                              */

stackSize;          /* Allocated stack area size.             */

vertexType **stackRep;  /* Stack area.                            */

} stackType;

typedef struct {           /* Heap cell data structure.               */

int degree;            /* Key field, containing vertex's degree.  */

vertexType *vertex:    /* Info field, holding vertex's address.   */

} heapCell;

typedef struct {           /* Heap data structure.                    */

int heapTop,           /* Heap top.                               */

heapSize;          /* Allocated heap area size.              */

heapCell *heapRep;    /* Heap area.                                 */

} heapType;

stackType     stacks[NO STACKS]    /* The stacks of the virtual heap. */

heapType      heap;                /* The heap portion.               */

#ifdef __STDC__

extern void   push( stackType *stack, vertexType *vertex );

extern int    pop( stackType *stack, vertexType **vertex );

extern void   enter heap( int degree, vertexType *vertex );

extern int    remove_from_heap( vertexType **vertex );

#else

extern void   push();

extern int    pop();

extern void   enter_heap();

extern int    remove_from_heap();

#endif

/*************************************************************************

add_to_vheap( vertex, degree )

Return:      void

Purpose:     Add a vertex of a specified degree to the virtual heap.

**/

void add_to_vheap( vertex, degree )

vertexType       *vertex;        /* in: a vertex to be added.     */

int              degree;         /* in: the vertex's degree.      */

{

if ( degree > NO_STACKS )

enter_heap( degree, vertex );

else

push( &stacks[degree-1], vertex );

}

/*************************************************************************

max_degree_vertex( vertex )

Return:      int -- NORM if a vertex could be found, ABNORM if the

virtual heap (stacks and heap) is empty.

Purpose:     Find the unvisited vertex with maximal degree from the

virtual heap. Place it in "vertex".

Plan:        First check the heap; remove_from_heap() automatically

removes a vertex of maximal degree. If the heap is

empty, try the stacks, one at a time.

**/

int max_degree_vertex( vertex )

vertexType  **vertex;           /* out: the vertex found. */

{

int i;

if ( remove_from_heap( vertex ) == NORM ) /* heap empty? */

return(NORM);

for( i = NO_STACKS - 1; i >= 0; i-- )         /* stacks empty? */

if ( pop( &stacks[i], vertex ) == NORM )

return (NORM);

return(ABNORM);   /* No node at all. The component has been processed. */

}

/*************************************************************************

push(stack, vertex )

Return:      void

Purpose:     Push a vertex pointer onto a stack.

**/

static void push(stack, vertex)

stackType        *stack;     /* in out: the stack.    */

vertexType       *vertex;    /* in: the vertex.      */

{

stack->stackTop++;

/* Expand stack if it doesn't have enough space. */

if ( stack->stackTop >= stack->stackSize ) {

fprintf(stderr, "Warning: stack overflow. Re-allocating.\n");

stack->stackSize *= 2;

stack->stackRep =

(vertexType**)ownrealloc( (char *)stack->stackRep,

sizeof(vertexType*) * stack-stackSize );

}

stack->stackRep[stack->stackTop] = vertex;

}

/*************************************************************************

pop( stack, vertex )

Return:      int -- Index of a vertex.

Purpose:     Pop up a vertex pointer from the stack. Return -1 if the stack

was empty, 0 if it wasn't.

**/

static int pop( stack, vertex )

stackType   *stack;

vertexType  **vertex;

{

if ( stack->stackTop == -1 )

return(-1);       /* stack empty */

*vertex = stack->stackRep[stack->stackTop--];

return(0);    /* stack not empty */

}

/*************************************************************************

enter_heap( degree, vertex )

Return:      void

Purpose:     Insert a vertex pointer and its degree into the heap.

**/

static void enter_heap( degree, vertex )

int         degree;      /* in: the degree of the node.    */

vertexType  *vertex;     /* in: the vertex pointer.        */

{

int k = heap.heapTop++ ;

if ( k >= heap.heapSize ) {

heap.heapSize = 2 * heap.heapSize;

heap.heapRep =

(heapCell*)ownrealloc( (char *)heap.heapRep,

sizeof(heapCell) * heap.heapSize );

}

heap.heapRep[k].degree = degree;

heap.heapRep[k].vertex = vertex;

while ( heap.heapRep[k/2].degree <= degree ) {

heap.heapRep[k].degree = heap.heapRep[k/2].degree;

heap.heapRep[k].vertex = heap.heapRep[k/2].vertex;

k /= 2;

}

heap.heapRep[k].degree = degree;

heap.heapRep[k].vertex = vertex;

}

/*************************************************************************

remove_from_heap( vertex )

Return:      int -- -1 if the heap is empty when the routine is called,

0 if it isn't.

Purpose:     Remove a vertex of maximal degree from the heap, and

return it.

**/

static int remove_from_heap( vertex )

vertexType       **vertex;    /* out: the vertex selected. */

{

int         k, j;         /* Iterators through the heap.            */

heapCell    tempCell;     /* Heap element currently being examined. */

if ( heap.heapTop == 1 ) return(-1);

*vertex = heap.heapRep[1].vertex;

heap.heapTop--

tempCell.degree =

heap.heapRep[1].degree= heap.heapRep[heap.heapTop].degree;

tempCell.vertex = heap.heapRep[1].vertex =

heap.heapRep[heap.heapTop].vertex;

k = 1;                                     /* Go down the heap. */

while ( k <= heap.heapTop / 2 ) {

j = 2 * k;

if ( (j < heap.heapTop ) &&

(heap.heapRep[j].degree< heap.heapRep[j+1].degree) )

j++;

if ( tempCell.degree > heap.heapRep[j].degree )

break;

heap.heapRep[k].degree = heap.heapRep[j].degree;

heap.heapRep[k].vertex = heap.heapRep[j].vertex;

k = j;

} /* end of while */

heap.heapRep[k].degree = tempCell.degree;

heap.heapRep[k].vertex = tempCell.vertex;

return(0);

}

/*************************************************************************

initialize_vheap()

Return:      void

Purpose:     Set the heap and stacks to their empty states.

**/

void initialize_vheap()

{

int i;

heap.heapRep[0].degree = MAX_INT;

heap.heapTop = 1;

for ( i = 1; i < heap.heapSize; i++ ) {

heap.heapRep[i].degree = 0;

heap.heapRep[i].vertex = 0;

}

for ( i = 0; i NO_STACKS; stacks[i++].stackTop = -1 );

}

/*************************************************************************

free_vheap()

Return:      void

Purpose:     Deallocate space for stacks and heap.

**/

void free_vheap()

{

int i;

for ( i = 0; i NO_STACKS; free((char *)stacks[i++].stackRep) );

free( (char *)heap.heapRep );

}

/*************************************************************************

allocate_vheap( no_arcs, no_vertices )

Return:      void

Purpose:     Estimate and allocate space for the heap and the stacks.

**/

void allocate_vheap( no_arcs, no_vertices )

int    no_arcs,         /* in: number of arcs.   */

no_vertices;       /* in: number of vertices.  */

{

int   i,          /* iteration variable.     */

sum = 0;       /* partial sum of degree.   */

double lambda,       /* lambda =   E   / (   V   / 2 )  */

Pr0,         /* Pr0 = Pr(X = 0)         */

Pri;         /* Pri = Pr(X = i)         */

lambda = (double)(2*no_arcs) / (double)no_vertices;

Pr0 = Pri = exp(-lambda);       /* Compute Pr(x = 0).  */

for ( i = 1; i = NO_STACKS; i++ ) {  /* Compute the expected number */

Pri *= lambda/(double)(i);     /* of nodes of degree 1, 2, . . . , */

/* NO_STACKS.             */

stacks[i-1].stackSize = (int) 2 * no_vertices * Pri;

sum += stacks[i-1].stackSize ;

}

for ( i = 0; i NO_STACKS; i++ ) { /* Allocate stack space. */

if ( stacks[i].stackSize = 0 ) stacks[i].stackSize = DEF_SIZE;

stacks[i].stackRep =

(vertexType**) owncalloc( stacks[i].stackSize, sizeof(vertexType*)

);

}

heap.heapSize = no_vertices - sum - (int) 2 * no_vertices * Pr0;

if ( heap.heapSize = 0 ) heap.heapSize = DEF_SIZE;

heap.heapRep =             /* Allocate heap space. */

(heapCell*) owncalloc( heap.heapSize, sizeof(heapCell) );

}

Go to Chapter 14     Back to Table of Contents


Information Retrieval: CHAPTER 14: RANKING ALGORITHMS

 Return to Table of Contents

CHAPTER 14: RANKING ALGORITHMS

Donna Harman

National Institute of Standards and Technology

Abstract

This chapter presents both a summary of past research done in the development of ranking algorithms and detailed instructions on implementing a ranking type of retrieval system. This type of retrieval system takes as input a natural language query without Boolean syntax and produces a list of records that "answer" the query, with the records ranked in order of likely relevance. Ranking retrieval systems are particularly appropriate for end-users.

14.1 INTRODUCTION

Boolean systems were first developed and marketed over 30 years ago at a time when computing power was minimal compared with today. Because of this, these systems require the user to provide sufficient syntactical restrictions in their query to limit the number of documents retrieved, and those retrieved documents are not ranked in order of any relationship to the user's query. Although the Boolean systems offer very powerful on-line search capabilities to librarians and other trained intermediaries, they tend to provide very poor service to end-users, particularly those who use the system on an infrequent basis (Cleverdon 1983). These end-users are likely to be familiar with the terminology of the data set they are searching, but lack the training and practice necessary to get consistently good results from a Boolean system because of the complex query syntax required by these systems. The ranking approach to retrieval seems to be more oriented toward these end-users. This approach allows the user to input a simple query such as a sentence or a phrase (no Boolean connectors) and retrieve a list of documents ranked in order of likely relevance.

The main reason the natural language/ranking approach is more effective for end-users is that all the terms in the query are used for retrieval, with the results being ranked based on co-occurrence of query terms, as modified by statistical term-weighting (to be explained later in the chapter). This method eliminates the often-wrong Boolean syntax used by end-users, and provides some results even if a query term is incorrect, that is, it is not the term used in the data, it is misspelled, and so on. The ranking methodology also works well for the complex queries that may be difficult for end-users to express in Boolean logic. For example, "human factors and/or system performance in medical databases" is difficult for end-users to express in Boolean logic because it contains many high- or medium-frequency words without any clear necessary Boolean syntax. The ranking method would do well with this query.

This chapter describes the implementation of a ranking system and is organized in the following manner. Section 14.2 shows a conceptual illustration of how ranking is done. Section 14.3 presents various theoretical models used in ranking and reviews past experiments using these models. Section 14.4 describes results from several experiments directly comparing various ranking schemes, along with brief discussions of the type of ranking methods found in the few operational systems that have used this retrieval technique. Section 14.5 summarizes the results from sections 14.3 and 14.4, presenting a series of recommended ranking schemes for various situations. Section 14.6 describes the implementation of a basic ranking retrieval system, with section 14.7 showing possible variations to this scheme based on retrieval environments. Section 14.8 discusses some topics closely related to ranking and provides some suggestions for further reading in these areas, and section 14.9 summarizes the chapter.

The terms "record" and "document" are used interchangeably throughout this chapter, as are the terms "data set," "database," and "collection," depending on the retrieval environment being discussed.

14.2 HOW RANKING IS DONE

Assume that a given textual data set uses i unique terms. A document can then be represented by a vector (t1, t2, t3, . . . , tn), where ti has a value of 1 if term i is present, and 0 if term i is absent in the document. A query can be represented in the same manner. Figure 14.1 shows this representation for a data set with seven unique terms.

The top section of Figure 14.1 shows the seven terms in this data set. The second section shows a natural language query and its translation into a conceptual vector, with 1's in vector positions of words included in the query, and 0's to indicate a lack of those words. For example, the first "1" indicates the presence of the word "factor," the second "1" indicates the presence of the word "information," the first "0" indicates the absence of the word "help." The third section of Figure 14.1 shows a similar conceptual representation of three documents in this data set. To determine which document best matches the query, a simple dot product of the query vector and each document vector is made (left side of the fourth section) and the results used to rank the documents.

It is possible to perform the same operation using weighted vectors as shown in the right side of the bottom section of Figure 14.1. In the example, each term is weighted by the total number of times it appears in the record, for example, "factors" appears twice in document 1, "information" appears three times in document 1, and so on. These term-weights could reflect different measures, such as the scarcity of a term in the data set (i.e., "human" probably occurs less frequently than "systems" in a computer science data set), the frequency of a term in the given document (as shown in the example), or some user-specified term-weight. This term-weighting usually provides substantial improvement in the ranking.

Figure 14.1: A simple illustration of statistical ranking

14.3 RANKING MODELS AND EXPERIMENTS WITH THESE MODELS

In 1957 Luhn published a paper proposing a statistical approach to searching literary information. He suggested that "the more two representations agreed in given elements and their distribution, the higher would be the probability of their representing similar information." Maron and Kuhns (1960) went much further by suggesting how to actually weight terms, including some small-scale experiments in term-weighting. The information retrieval research community has continued to develop many models for the ranking technique over the last 30 years (for an overview, see Belkin and Croft [1987]). Although it is not necessary to understand the theoretical models involved in ranking in detail in order to implement a ranking retrieval system, it is helpful to know about them as they have provided a framework for the vast majority of retrieval experiments that contributed to the development of the ranking techniques used today.

All the experimental results presented in the models are based on using standard test collections and using standard recall and precision measures for evaluation. Results are presented in a roughly chronological order to provide some sense of the development of knowledge about ranking through these experiments.

Ranking models can be divided into two types: those that rank the query against individual documents and those that rank the query against entire sets of related documents. The first type of ranking model, those that rank individual documents against the query, covers several theoretical models, but the principle ones are the vector space model and the probabilistic model.

14.3.1 The Vector Space Model

The sample document and query vectors described in section 14.2 can be envisioned as an n-dimensional vector space, where n corresponds to the number of unique terms in the data set. A vector matching operation, based on the cosine correlation used to measure the cosine of the angle between vectors, can then be used to compute the similarity between a document and a query, and documents can be ranked based on that similarity.

where

     tdij = the ith term in the vector for document j

     tqik = the ith term in the vector for query k

        n = the number of unique terms in the data set

This model has been used as the basis for many ranking retrieval experiments, in particular the SMART system experiments under Salton and his associates (1968, 1971, 1973, 1981, 1983, 1988). These ranking experiments started in 1964 at Harvard University, moved to Cornell University in 1968, and form a large part of the research in information retrieval. The SMART experiments cover many areas of information retrieval such as relevance feedback, clustering, and experiments with suffixing, synonyms, and phrases. Only those experiments dealing directly with term-weighting and ranking will be discussed here.

Early experiments (Salton and Lesk 1968; Salton 1971, p. 143) using the SMART system tested an overlap similarity function against the cosine correlation measure and tried simple term-weighting using the frequency of terms within the documents. These experiments showed that within-document frequency weighting improved performance over no term-weighting (in varying amounts depending on the test collection used). Further, they showed that use of the cosine correlation with frequency term-weighting provided better performance than the overlap similarity because of the automatic inclusion of document length normalization by the cosine similarity function (again, the results varied somewhat depending on the test collection used).

A second major set of experiments was done by Salton and Yang (1973) to further develop the term-weighting schemes. Of particular interest in these experiments were the term-weighting schemes relying on term importance within an entire collection rather than only within a given document. Clearly more weight should be given to query terms matching document terms that are rare within a collection. Salton and Yang were able to show significant performance improvement using a term-weighting scheme that combined the within-document frequency weighting with a new term-weighting scheme, the inverted document frequency (IDF; Sparck Jones 1972) that is based on the Zipf distribution of a term within the entire collection (see page 373 for the definition of IDF).

A recent paper by Salton and Buckley (1988) summarizes 20 years of SMART experiments in automatic term-weighting by trying 287 distinct combinations of term-weighting assignments, with or without cosine normalization, on six standard collections. Besides confirming that the best document term-weighting is provided by a product of the within-document term frequency and the IDF, normalized by the cosine measure, they show performance improvements using enhanced query term-weighting measures for queries with term frequencies greater than one. These weighting schemes are further discussed in section 14.5.

14.3.2 Probabilistic Models

Although a model of probabilistic indexing was proposed and tested by Maron and Kuhns (1960), the major probabilistic model in use today was developed by Robertson and Sparck Jones (1976). This model is based on the premise that terms that appear in previously retrieved relevant documents for a given query should be given a higher weight than if they had not appeared in those relevant documents. In particular, they presented the following table showing the distribution of term t in relevant and nonrelevant documents for query q.

     N = the number of documents in the collection

     R = the number of relevant documents for query q

     n = the number of documents having term t

     r = the number of relevant documents having term t

They then use this table to derive four formulas that reflect the relative distribution of terms in the relevant and nonrelevant documents, and propose that these formulas be used for term-weighting (the logs are related to actual use of the formulas in term-weighting).

Robertson and Sparck Jones also formally derive these formulas, and show that theoretical preference is for F4.

Formula F1 had been used by Barkla (1969) for relevance feedback in a SDI service and by Miller (1971) in devising a probabilistic search strategy for Medlars. It was also used by Sparck Jones (1975) in devising optimal performance yardsticks for test collections. Formula F4 (minus the log) is the term precision weighting measure proposed by Yu and Salton (1976).

Robertson and Sparck Jones used these four formulas in a series of experiments with the manually indexed Cranfield collection. They did experiments using all the relevance judgments to weight the terms to see what the optimal performance would be, and also used relevance judgments from half the collection to weight the terms for retrieval from the second half of the collection. In both cases, formula F4 was superior (closely followed by F3), with a large drop in performance between the optimal performance and the "predictive" performance, as would be expected. The experimental verification of the theoretical superiority of F4 provided additional weight to the importance of this new model.

The use of this theory as a predictive device was further investigated by Sparck Jones (1979a) who used a slightly modified version of F1 and F4 and again got much better results for F4 than for F1, even on a large test collection. Sparck Jones (1979b) tried using this measure (F4 only) in a manner that would mimic a typical on-line session using relevance feedback and found that adding the relevance weighting from only the first couple of relevant documents retrieved by a ranking system still produced performance improvements.

Work up to this point using probabilistic indexing required the use of at least a few relevant documents, making this model more closely related to relevance feed-back than to term-weighting schemes of other models. In 1979 Croft and Harper published a paper detailing a series of experiments using probabilistic indexing without any relevance information. Starting with a probabilistic restatement of F4, they assume that all query terms have equal probability of occurring in relevant documents and derive a term-weighting formula that combines a weight based on the number of matching terms and on a term-weighting similar to the IDF measure.

where

     Q = the number of matching terms between document j and query k

     C = a constant for tuning the similarity function

    ni = the number of documents having term i in the data set

     N = the number of documents in the data set

Experimental results showed that this term-weighting produced somewhat better results than the use of the IDF measure alone. Being able to provide different values to C allows this weighting measure to be tailored to various collections. Setting C to 1 ranks the documents by IDF weighting within number of matches, a method that was suitable for the manually indexed Cranfield collection used in this study (because it can be assumed that each matching query term was very significant). C was set much lower in tests with the UKCIS2 collection (Harper 1980) because the terms were assumed to be less accurate, and the documents were very short (consisting of titles only).

Croft (1983) expanded his combination weighting scheme to incorporate within-document frequency weights, again using a tuning factor K on these weights to allow tailoring to particular collections. The results show significant improvement over both the IDF weighting alone and the combination weighting, with the scaling factor K playing a large part in tuning the weighting to different collections.

where

   Q = the number of matching terms between document j and query k

IDFi = the IDF weight for term i in the entire collection (see page 373 for

       the definition of IDF)

where

  freqij = the frequency of term i in document j

       K = a constant for adjusting the relative importance of the two

           weighting schemes

maxfreqj = the maximum frequency of any term in document j

The best value for K proved to be 0.3 for the automatically indexed Cranfield collection, and 0.5 for the NPL collection, confirming that within-document term frequency plays a much smaller role in the NPL collection with its short documents having few repeating terms.

14.3.3 Other Models for Ranking Individual Documents

Several other models have been used in developing term-weighting measures. The inverted document frequency measure heavily used in implementing both the vector space model and the probabilistic model was derived by Sparck Jones (1972) from observing the Zipf distribution curve for collection vocabulary. Others have tried more complex term distributions, most notably the 2-Poisson model proposed by Bookstein and Swanson (1974) and implemented and tested by Harter (1975) and Raghavan et al. (1983). This distribution model proved much less successful because of the difficulty in estimating the many parameters needed for implementation.

14.3.4 Set-Oriented Ranking Models

The most well known of the set-oriented models are the clustering models where a query is ranked against a hierarchically grouped set of related documents. This model is the subject of Chapter 16 and will not be further discussed here.

Models based on fuzzy set theory have been proposed (for a summary, see Bookstein [1985]) but have not received enough experimental implementations to be used in practice (except when combined with Boolean queries such as in the P-Norm discussed in Chapter 15).

The theory of rough sets has been applied to information retrieval (Srinivasan 1989) but similarly has not been developed far enough to be used in practice.

14.4 OTHER EXPERIMENTS INVOLVING RANKING

14.4.1 Direct Comparison of Similarity Measures and Term-Weighting Schemes

There have been several studies examining the various factors involved in ranking that have not been based on any particular model but have instead used some method of comparing directly various similarity measures and term-weighting schemes.

Sparck Jones (1973) explored different types of term frequency weightings involving term frequency within a document, term frequency within a collection, term postings within a document (a binary measure), and term postings within a collection, along with normalizing these measures for document length. She used four collections, with indexing generally taken from manually extracted keywords instead of using full-text indexing, and with all queries based on manual keywords. Her results showed that using the term frequency (or postings) within a collection always improved performance, but that using term frequency (or postings) within a document improved performance only for some collections. The various term-weighting schemes were not combined in this experiment.

McGill et al. (1979) examined the literature from different fields to select 67 similarity measures and 39 term-weighting schemes. He used these to rank results from Boolean retrievals using both controlled (manually indexed) and uncontrolled (full-text) indexing. For both controlled and uncontrolled vocabulary he found a significant difference in the performance of similarity measures, with a group of about 15 different similarity measures all performing significantly better than the rest. This group included both the cosine correlation and the inner product function used in the probabilistic models. The term-weighting results were more mixed, with no significant difference found when using controlled vocabulary (i.e., term-weighting made no difference) and an overall significant difference found for uncontrolled vocabulary. There was a lack of significant difference between pairs of term-weighting measures for uncontrolled vocabulary, however, which could indicate that the difference between linear combinations of term-weighting schemes is significant but that individual pairs of term-weighting schemes are not significantly different.

A different approach was taken by Harman (1986). She selected four term-weighting factors proven important in past research and tried different combinations in order to arrive at an "optimum" term-weighting scheme. Full-text indexing was used on various standard test collections, with full-text indexing also done on the queries. The four factors investigated were: the number of matches between a document and a query, the distribution of a term within a document collection, the frequency of a term within a document, and the length of the document. Two different measures for the distribution of a term within a document collection were used, the IDF measure by Sparck Jones and a revised implementation of the "noise" measure (Dennis 1964; Salton and McGill 1983). Note that the use of noise here refers to how much a term can be considered useful for retrieval versus being simply a "noisy" term, and examines the concentration of terms within documents rather than just the number of postings or occurrences. She found that when using the single measures alone, the distribution of the term within the collection improved performance almost twice as much for the Cranfield collection as using only within-document frequency. Combining the within-document frequency with either the IDF or noise measure, and normalizing for document length improved results more than twice as much as using the IDF or noise alone in the Cranfield collection. Other collections showed less improvement, but the same relative merit of the term-weighting schemes was found. The noise measure consistently slightly outperformed the IDF (however with no significant difference).

14.4.2 Ranking Based on Document Structure

Some ranking experiments have relied more on document or intradocument structure than on the term-weighting described earlier. Bernstein and Williamson (1984) built a ranking retrieval system for a highly structured knowledge base, the Hepatitis Knowledge Base. Their ranking algorithms used not only weights based on term importance both within an entire collection and within a given document, but also on the structural position of the term, such as within summary paragraphs versus within text paragraphs. A very elaborate weighting scheme was devised for this experiment, tailored to the particular structure of the knowledge base. In SIBRIS, an operational information retrieval system (Wade et al. 1989), document and query structures are also used to influence the ranking, increasing term-weights for terms in titles of documents and decreasing term weights for terms added to a query from a thesaurus.

A very different approach based on complex intradocument structure was used in the experiments involving latent semantic indexing (Lochbaum and Streeter 1989). The indexing and retrieval were based on the singular value decomposition (related to factor analysis) of a term-document matrix from the entire document collection. This was combined with weighting using both a function of term frequency within a document (the root mean square normalization), and a function of term frequency within the entire collection (the noise or entropy measure, or alternatively the IDF measure). The results were generally superior to those using term-weighting alone, although further development is necessary before this approach is fast enough for use in large retrieval systems.

14.4.3 Ranking Techniques Used in Operational Systems

Several operational retrieval systems have implemented ranking algorithms as central to their search mechanism. The SIRE system, as implemented at Syracuse University (Noreault et al. 1977) built a hybrid system using Boolean searching and a vector-model-based ranking scheme, weighting by the use of raw term frequency within documents (for more on the hybrid aspects of this system, see section 14.7.3). A commercial outgrowth of this system, marketed as Personal Librarian, uses ranking based on different factors, including the IDF and the frequency of a term within a document. This system assigns higher ranks to documents matching greater numbers of query terms than would normally be done in the ranking schemes discussed experimentally.

The CITE system, designed as an interface to MEDLINE (Doszkocs 1982), ranked documents based solely on the IDF weighting, as no within-document frequencies were available from the MEDLINE files. For details on the search system associated with CITE, see section 14.7.2. The OPAKI project (Walker and Jones 1987) worked with on-line catalogs and also used the IDF measure alone. Although other small-scale operational systems using ranking exist, often their ranking algorithms are not clear from publications, and so these are not listed here.

14.5 A GUIDE TO SELECTING RANKING TECHNIQUES

In looking at results from all the experiments, some trends clearly emerge.

1. The use of term-weighting based on the distribution of a term within a collection always improves performance (or at minimum does not hurt performance).The IDF measure has been commonly used, either in its form as originally used, or in a form somewhat normalized.

where

   N = the number of documents in the collection

  ni = the total number of occurrences of term i in the collection

maxn = the maximum frequency of any term in the collection

A possible alternative is the noise or entropy measure tried in several experiments .

where

       N = the number of documents in the collection

maxnoise = the highest noise of any term in the collection

  Freqik = the frequency of term i in document k

  TFreqi = the total frequency of term i in the collection

2. The combination of the within-document frequency with the IDF weight often provides even more improvement. There are several reasons why this improvement is inconsistent across collections. First, it is very important to normalize the within-document frequency in some manner, both to moderate the effect of high-frequency terms in a document (i.e., a term appearing 20 times is not 20 times as important as one appearing only once) and to compensate for document length. This normalization has taken various forms in different experiments, but the lack of proper normalization techniques in some experiments has likely hidden possible improvements. A second reason for the inconsistent improvements found for within-document frequencies is the fact that some collections have very short documents (such as titles only) and therefore within-document frequencies play no role in these collections. Finally, the effects of within-document frequency may need to be tailored to collections, such as was done by Croft (1983) in using a sliding importance factor K, and by Salton and Buckley (1988) in providing different combination schemes for term-weighting. This tailoring seems to be particularly critical for manually indexed or controlled vocabulary data where use of within-document frequencies may even hurt performance. Either of the following normalized within-document frequency measures can be safely used.

where

  freqij = the frequency of term i in document j

maxfreqj = the maximum frequency of any term in document j

 lengthj = the number of unique terms in document j

3. Assuming within-document term frequencies are to be used, several methods can be used for combining these with the IDF measure. The combination recommended for most situations by Salton and Buckley is given below (a complete set of weighting schemes is presented in their 1988 paper).

where

and

     wij = freqij X IDFi

  freqiq = the frequency of term i in query q

maxfreqj = the maximum frequency of any term in query maxfreqq

    IDFi = the IDF of term i in the entire collection

  freqij = the frequency of term i in document j

Salton and Buckley suggest reducing the query weighting wiq to only the within-document frequency (freqiq) for long queries containing multiple occurrences of terms, and to use only binary weighting of documents (Wij = 1 or 0) for collections with short documents or collections using controlled vocabulary.

Many combinations of term-weighting can be done using the inner product. Whereas there is more flexibility available here than in the cosine measure, the need for providing normalization of within-document frequencies is more critical. Two possible combinations are given below that calculate the matching strength of a query to document j, with symbol definitions the same as those previously given.

where

C should be set to low values (near 0) for automatically indexed collections, and to higher values such as 1 for manually indexed collections. K should be set to low values (0.3 was used by Croft) for collections with long (35 or more terms) documents, and to higher values (0.5 or higher) for collections with short documents, reducing the role of within-document frequency.

One alternative ranking using the inner product (but without adjustable constants) is given below.

4. It can be very useful to add additional weight for document structure, such as higher weightings for terms appearing in the title or abstract versus those appearing only in the text. This additional weighting needs to be considered with respect to the particular data set being used for searching. User weighting can also be considered as additional weighting, although this type of weighting has generally proven unsatisfactory in the past.

5. The use of relevance weighting after some initial retrieval is very effective. Relevance weighting is discussed further in Chapter 11 on relevance feedback.

14.6 DATA STRUCTURES AND ALGORITHMS FOR RANKING

This section will describe a simple but complete implementation of the ranking part of a retrieval system. Modifications of this implementation that enhance its efficiency or are necessary for other retrieval environments are given in section 14.7, with cross-references made to these enhancements throughout this section.

The implementation will be described as two interlocking pieces: the indexing of the text and the using (searching) of that index to return a ranked list of record identification numbers (ids). The index shown is a straightforward inverted file, created once per major update (thus only once for a static data set), and is used to provide the necessary speed for searching. Although it is possible to build a ranking retrieval system without some type of index (either by storing and searching all terms in a document or by using signature files in ranking such as described in section 14.8.5), the use of these indices improves efficiency by several orders of magnitude. The penalty paid for this efficiency is the need to update the index as the data set changes. Except for data sets with critical hourly updates (such as stock quotes), this is generally not a problem. An enhancement to the indexing program to allow easier updating is given in section 14.7.4.

The description of the search process does not include the interface issues or the actual data retrieval issues. It is assumed that a natural language query is passed to the search process in some manner, and that the list of ranked record id numbers that is returned by the search process is used as input to some routine which maps these ids onto data locations and displays a list of titles or short data descriptors for user selection.

14.6.1 The Creation of an Inverted File

The inverted file described here is a modification to the inverted files described in Chapter 3 on that subject. It would be feasible to use structures other than simple inverted files, such as the more complex structures mentioned in that chapter, as long as the elements needed for ranking are provided. The use of a ranking system instead of a Boolean retrieval system has several important implications for supporting inverted file structures.

1. The use of ranking means that there is little need for the adjacency operations or field restrictions necessary in Boolean. Therefore, only the record id has to be stored as the location for each word, creating a much smaller index than for Boolean systems (in the order of 10% to 15% of the text size). If it is determined that the ranking system must also handle adjacency or field restrictions, then either the index must record the additional location information (field location, word position within record, and so on) as described for Boolean inverted files, or an alternative method (see section 14.8.4) can be used that does not increase storage but increases response time when using these particular operations. The inverted file presented here will assume that only record location is necessary.

2. The use of ranking means that strategies needed in Boolean systems to increase precision are not only unnecessary but should be discarded in favor of strategies that increase recall at the expense of precision. In the area of parsing, this may mean relaxing the rules about hyphenation to create indexing both in hyphenated and nonhyphenated form. In the area of stoplists, it may mean a less restrictive stoplist. For example, in a data set about computers, the ultra-high frequency term "computer" may be in a stoplist for Boolean systems but would not need to be considered a common word for ranking systems. In the area of stemming, a ranking system seems to work better by automatically expanding the query using stemming (Frakes 1984; Harman and Candela 1990) rather than by forcing the user to ask for expansion by wild-cards. A more appropriate stemming strategy for ranking therefore is to use stemming in creation of the inverted file. An enhancement of this stemming option would be to allow the user to specify a "don't stem" character, and the modifications necessary to handle this are given in section 14.7.1.

Although an inverted file with frequency information (Figure 14.2) could be used directly by the search routine, it is usually processed into an improved final format.

Figure 14.2: Inverted file with frequency information

This format is based on the search methods and the weighting methods used, but a common search technique (and the one discussed here) is to use a binary search routine on the file to locate the query words. This implies that the file to be searched should be as short as possible, and for this reason the single file shown containing the terms, record ids, and frequencies is usually split into two pieces for searching: the dictionary containing the term, along with statistics about that term such as number of postings and IDF, and then a pointer to the location of the postings file for that term. The postings file contains the record ids and the weights for all occurrences of the term. In this manner the dictionary used in the binary search has only one "line" per unique term.

The above illustration is a conceptual form of the necessary files; the actual form depends on the details of the search routine and on the hardware being used. Work using large data sets (Harman and Candela 1990) showed that for a file of 2,653 records, there were 5,123 unique terms with an average of 14 postings/term and a maximum of over 2,000 postings for a term. A larger data set of 38,304 records had dictionaries on the order of 250,000 lines (250,000 unique terms, including some numerals) and an average of 88 postings per record. From these statistics it is clear that efficient storage structures for both the binary search and the reading of the postings are critical. Ideally, both files could be read into memory when a data set is opened. Somewhat less ideally, only the dictionary could be stored in memory, with disk access for the postings file. Usually, however, both parts of the index must be processed from disk. More details of the storage and use of these files is given in the description of the search process. Recent work on the effective use of inverted files suggests better ways of storing and searching these files (Burkowski 1990; Cutting and Pedersen 1990).

The dictionary and postings file shown (Figure 14.3) stores a term-weight of simply the raw frequency of a term in a record. If this is the actual weight stored, then all the calculations of term-weights must be done in the search routine itself, providing a heavy overhead per posting. There are four major options for storing weights in the postings file, each having advantages and disadvantages.

Figure 14.3: A dictionary and postings file

1. Store the raw frequency. This produces the slowest search (likely much too slow for large data sets), but the most flexible system in that term-weighting algorithms can be changed without changing the index.

2. Store a normalized frequency. Any of the normalized frequencies shown in section 14.5 can be used to translate the raw frequency to a normalized frequency. Using Harman's normalized frequency as an example, the raw frequency for each term from the final table of the inversion process would be transformed into a log function and then divided by the log of the length of the corresponding record (the lengths of the records were collected and saved in the parsing step). This operation would be done during the creation of the final dictionary and postings file, and this normalized frequency would be inserted in the postings file in place of the raw frequency shown. The same procedure could be done for Croft's normalized frequency or any other normalized frequency used in an inner product similarity function, assuming appropriate record statistics have been stored during parsing. However, this option is not suitable for use with the cosine similarity function using Salton's method as the normalization for length includes the IDF factor. The advantage of this term-weighting option is that updating (assuming only the addition of new records and not modification of old ones) would not require the postings to be changed. This option would improve response time considerably over option 1, although option 3 may be somewhat faster (depending on search hardware).

3. Store the completely weighted term. Again, any of the combination weighting schemes shown in section 14.5 are suitable, including those using the cosine similarity function. This option allows a simple addition of each weight during the search process, rather than first multiplying by the IDF of the term, and provides very fast response time. This was the option taken by Harman and Candela (1990) in searching on 806 megabytes of data. The disadvantage of this option is that updating requires changing all postings because the IDF is an integral part of the posting (and the IDF measure changes as any additions are made to the data set). Additionally, relevance feedback reweighting is difficult using this option.

4. If no within-record weighting is used, then the postings records do not have to store weights. All processing would be done in the search routines.

14.6.2 Searching the Inverted File

One way of using an inverted file to produce statistically ranked output is to first retrieve all records containing the search terms, then use the weighting information for each term in those records to compute the total weight for each of those retrieved records, and finally sort those records. The search time for this method is heavily dependent on the number of retrieved records and becomes prohibitive when used on large data sets.

This process can be made much less dependent on the number of records retrieved by using a method developed by Doszkocs for CITE (Doszkocs 1982). In this method, a block of storage was used as a hash table to accumulate the total record weights by hashing on the record id into unique "accumulator" addresses (for more details, see Doszkocs [1982]). This makes the searching process relatively independent of the number of retrieved records--only the sort for the final set of ranks is affected by the number of records being sorted.

This was the method chosen for the basic search process (see Figure 14.4). A block of storage containing an "accumulator" for every unique record id is reserved, usually on the order of 300 Kbytes for large data sets. The query is parsed using the same parser that was used for the index creation, with each term then checked against the stoplist for removal of common terms. If the query term is not common, it is then passed through the stemming routine and a binary search for that stem is executed against the dictionary. (For algorithms to do efficient binary searches, see Knuth [1973], and for an alternative to binary searching see section 14.7.4.) If the stem is found in the dictionary, the address of the postings list for that stem is returned, along with the corresponding IDF and the number of postings.

Figure 14.4: Flowchart of search engine

The next step is to use the address of the postings and the number of postings to read the record ids, with the total term-weight for each record id being added to the contents of its unique accumulator. If option 3 was used for weighting, then this total is immediately available and only a simple addition is needed. If option 2 was used for weighting, then the weight stored in the postings is the normalized frequency of the stem in that record, and this needs to be multiplied by the IDF of that stem before the addition. If option 1 was used for weighting, then the full term-weight must be calculated, as the weight stored in the posting is the raw frequency of the stem in that record. Loading the necessary record statistics, such as record length, into memory before searching is essential to maintain any reasonable response time for this weighting option. As each query term is processed, its postings cause further additions to the accumulators. When all the query terms have been handled, accumulators with nonzero weights are sorted to produce the final ranked record list.

There are several major inefficiencies of this technique. First, the I/O needs to be minimized. A read of one byte essentially takes the same time as a read of many bytes (a buffer full) and this factor can be utilized by doing a single read for all the postings of a given term, and then separating the buffer into record ids and weights. This requires a sequential storage of the postings in the index, with the postings pointer in the dictionary being used to control the location of the read operation, and the number of postings (also stored in the dictionary) being used to control the length of the read (and the separation of the butfer). As some terms have thousands of postings for large data sets, doing a separate read for each posting can be very time-consuming. This same logic could be applied to the binary search of the dictionary, which takes about 14 reads per search for the larger data sets. The time saved may be considerably less, however. A final time savings on I/O could be done by loading the dictionary into memory when opening a data set.

A second time savings can be gained at the expense of some memory space. Whereas the storage for the "accumulators" can be hashed to avoid having to hold one storage area for each data set record, this is definitely not necessary for smaller data sets, and may not be useful except for extremely large data sets such as those used in CITE (which need even more modification; see section 14.7.2). Some time is saved by direct access to memory rather than through hashing, and as many unique postings are involved in most queries, the total time savings may be considerable. Harman and Candela (1990) found that almost every user query had at least one term that had postings in half the data set, and usually at least three quarters of the data set was involved in most queries.

A final major bottleneck can be the sort step of the "accumulators" for large data sets. Even a fast sort of thousands of records is very time consuming. An enhancement can be made to reduce the number of records sorted (see section 14.7.5).

14.7 MODIFICATIONS AND ENHANCEMENTS TO THE BASIC INDEXING AND SEARCH PROCESSES

There are many possible modifications and enhancements to the basic indexing and search processes, some of which are necessary because of special retrieval environments (those involving large and very large data sets are discussed), and some of which are techniques for enhancing response time or improving ease of updating. Each of the following topics deals with a specific set of changes that need to be made in the basic indexing and/or search routines to allow the particular enhancement being discussed. The level of detail is somewhat less than in section 14.6, either because less detail is available or because the implementation of the technique is complex and details are left out in the interest of space. It should be noted that, unlike section 14.6, some of the implementations discussed here should be used with caution as they are usually more experimental, and may have unknown problems or side effects.

14.7.1 Handling Both Stemmed and Unstemmed Query Terms

It was observed by Frakes (1984) and confirmed by Harman and Candela (1990) that if query terms were automatically stemmed in a ranking system, users generally got better results. In some cases, however, a stem is produced that leads to improper results, causing query failure. The basic indexing and search processes described in section 14.6 suggest no manner of coping with this problem, as the original record terms are not stored in the inverted file; only their stems are used. The following technique was developed for the prototype retrieval system described in Harman and Candela (1990) to handle this problem, but it is not thought to be an optimal method.

Clearly two separate inverted files could be created and stored, one for stems and one for the unstemmed terms. Query terms would normally use the stemmed version, but query terms marked with a "don't stem" character would be routed to the unstemmed version. Whereas this would solve the problem for smaller data sets, it creates a storage problem for the large data sets. A hybrid inverted file was devised to merge these files, saving no space in the dictionary part, but saving considerable storage over that needed to store two versions of the postings. This storage savings is at the expense of some additional search time and therefore may not be the optimal solution. An example of the merged inverted file is shown in Figure 14.5.

This hybrid dictionary is in alphabetic stem order, with the terms sorted within the stem, and contains the stem, the number of postings and IDF of the stem, the term, the number of postings and IDF of the term, a bit to indicate if the term is stemmed or not stemmed, and the offset of the postings for this stem/term combination. Note that the merged dictionary takes one line per unstemmed term, making it considerably larger than the stemmed dictionary, and resulting in longer binary searches for most terms (which will be stemmed). The hybrid postings list saves the storage necessary for one copy of the record id by merging the stemmed and unstemmed weight (creating a postings element of 3 positions for stemmed terms). Terms that have no stem for a given data set only have the basic 2-element postings record.

Figure 14.5: Merged dictionary and postings file

As can be expected, the search process needs major modifications to handle these hybrid inverted files. Each query term that is stemmed must now map to multiple dictionary entries, and postings lists must be handled more carefully as some terms have three elements in their postings list and some have only two.

Clearly, for data sets that are relatively small it is best to use the two separate inverted files because the storage savings are not large enough to justify the additional complexity in indexing and searching. Possibly the use of two separate dictionaries, both mapping to the same hybrid posting file, would improve search time without the loss of storage efficiency, but this has not been tried.

14.7.2 Searching in Very Large Data Sets

In 1982, MEDLINE had approximately 600,000 on-line records, with records being added at a rate of approximately 21,000 per month (Doszkocs 1982). The use of the fixed block of storage to accumulate record weights that is described in the basic search process (section 14.6) becomes impossible for this huge data set. Doszkocs solved the problem in his experimental front-end to MEDLINE (the CITE system) by segmenting the inverted file into 8K segments, each holding about 48,000 records, and then hashing these record addresses into the fixed block of accumulators. The basic search process is therefore unchanged except that instead of each record of the data set having a unique accumulator, the accumulators hold only a subset of the records and each subset is processed as if it were the entire data set, with each set of results shown to the user. The subsetting or segmenting is done in reverse chronological order. Because users are often most concerned with recent records, they seldom request to search many segments. For more details see Doszkocs (1982).

14.7.3 A Boolean System with Ranking

There are many ways to combine Boolean searches and ranking. A simple extension of the basic search process in section 14.6 can be made that allows noncomplex Boolean statements to be handled (see section 14.8.4). This extension, however, limits the Boolean capability and increases response time when using Boolean operators. Very elaborate schemes have been devised that combine Boolean with ranking, and references are made to these in section 14.8.3. However, none of these schemes involve extensions to the basic search process in section 14.6.

The SIRE system (Noreault, Koll, and McGill 1977) incorporates a full Boolean capability with a variation of the basic search process. The system accepts queries that are either Boolean logic strings (similar to many commercial on-line systems) or natural language queries (processed as Boolean queries with implicit OR connectors between all query terms). The user may request ranked output.

A Boolean query is processed in two steps. After stemming, each term in the query is checked against the inverted file (this could be done by using the binary search described in section 14.6). The record ids and raw frequencies for the term being processed are combined with those of the previous set of terms according to the appropriate Boolean logic. A running sum containing the numerator of the cosine similarity is updated by adding the new record frequencies, and this is continued until the entire Boolean query is processed. Note that this combining of sets for complex Boolean queries can be a complicated operation. If ranked output is wanted, the denominator of the cosine is computed from previously stored document lengths and the query length, and the records are sorted based on their similarity to the query.

Whereas the cosine similarity is used here with raw frequency term-weighting only (at least in the experiment described in Noreault, Koll and McGill [1977]), any of the term-weighting functions described in section 14.5 could be used. There are no modifications to the basic inverted file needed unless adjacency, field restrictions, and other such types of Boolean operations are desired. The major modification to the basic search process is to correctly merge postings from the query terms based on the Boolean logic in the query before ranking is done. As the final computations of the similarity measure and the sorting of the ranks are done only for those records that are selected by the Boolean logic, this enhancement probably has a faster response time for Boolean queries, and no increase in response time for natural language queries compared to the basic search process described in section 14.6.

14.7.4 Hashing into the Dictionary and Other Enhancements for Ease of Updating

The basic inverted file creation and search process described in section 14.6 assumes a fairly static data set or a willingness to do frequent updates to the entire inverted file. For smaller data sets, or for environments where ease of update and flexibility are more important than query response time, the inverted file could have a structure more conducive to updating. This was done in Croft's experimental re trieval system (Croft and Ruggles 1984). Their inverted file consists of the dictionary containing the terms and pointers to the postings file, but the dictionary is not alphabetically sorted. Instead it is a bucketed (10 slots/bucket) hash table that is accessed by hashing the query terms to find matching entries. Not only is this likely to be a faster access method than the binary search, but it also creates an extendable dictionary, with no reordering for updates. This necessity for ease of update also changes the postings structure, which becomes a series of linked variable length lists capable of infinite update expansion. The term-weighting is done in the search process using the raw frequencies stored in the postings lists.

This system therefore is much more flexible and much easier to update than the basic inverted file and search process described in section 14.6. Although the hash access method is likely faster than a binary search, the processing of the linked postings records and the search-time term-weighting will hurt response time considerably. This is not a major factor for small data sets and for some retrieval environments, especially those involved in research into new retrieval mechanisms.

Note that the binary search described in the basic search process could be replaced with the hashing method to further decrease response time for searching using the basic search process. This would require a different organization of the final inverted index file that contains the dictionary, but would not affect the postings lists (which would be sequentially stored for search time improvements).

14.7.5 Pruning

A major time bottleneck in the basic search process is the sort of the accumulators for large data sets. Various methods have been developed for dealing with this problem. Buckley and Lewit (1985) presented an elaborate "stopping condition" for reducing the number of accumulators to be sorted without significantly affecting performance. Perry and Willett (1983) and Lucarella (1983) also described methods of reducing the number of cells involved in this final sort. The following method serves only as an illustration of a very simple pruning procedure, with an example of the time savings that can be expected using a pruning technique on a large data set. This method is based on the fact that most records for queries are retrieved based on matching only query terms of high data set frequency. These records are still sorted, but serve only to increase sort time, as they are seldom, if ever, useful. These records can be retrieved in the normal manner, but pruned before addition to the retrieved record list (and therefore not sorted). Harman and Candela (1990) experimented with various pruning algorithms using this method, looking for an algorithm that not only improved response time, but did not significantly hurt retrieval results. Their changed search algorithm with pruning is as follows:

1. Sort all query terms (stems) by decreasing IDF value.

2. Do a binary search for the first term (i.e., the highest IDF) and get the address of the postings list for that term.

3. Read the entire postings file for that term into a buffer and add the term weights for each record id into the contents of the unique accumulator for the record id.

4. Check the IDF of the next query term. If the IDF is greater than or equal to one third the maximum IDF of any term in the data set, then repeat steps 2, 3, and 4. Otherwise repeat steps 2, 3, and 4, but do not add weights to zero weight accumulators, that is, high-frequency (low IDF) terms are allowed to only increment the weights of already selected record ids, not select a new record.

5. Sort the accumulators with nonzero weights to produce the final ranked record list. Note that records containing only high-frequency terms will not have any weight added to their accumulator and therefore are not sorted.

6. If a query has only high-frequency terms (several user queries had this problem), then pruning cannot be done (or a fancier algorithm needs to be created). A check needs to be made after step 1 for this.

Table 14.1 shows some timing results of this pruning algorithm. The test queries are those brought in by users during testing of a prototype ranking retrieval system.

Table 14.1:: Response Time

Size of Data Set       1.6 Meg  50 Meg  268 Meg  806 Meg

--------------------------------------------------------

Number of queries         13      38       17       17

--------------------------------------------------------

Average number of          4.1     3.5      3.5      3.5

terms per query

--------------------------------------------------------

Average number of        797    2843     5869    22654

records retrieved

--------------------------------------------------------

Average response time      0.38    1.2      2.6      4.1

per query (no pruning)

--------------------------------------------------------

Average response time      0.28   0.58      1.1      1.6

per query

(pruning)

The response time for the 806 megabyte data set assumes parallel processing of the three parts of the data set, and would be longer if the data set could not be processed in parallel.

As can be seen, the response times are greatly affected by pruning. The other pruning techniques mentioned earlier should result in the same magnitude of time savings, making pruning techniques an important issue for ranking retrieval systems needing fast response times.

14.8 TOPICS RELATED TO RANKING

14.8.1 Ranking and Relevance Feedback

Ranking retrieval systems and relevance feedback have been closely connected throughout the past 25 years of research. Relevance feedback was one of the first features to be added to the basic SMART system (Salton 1971), and is the foundation for the probabilistic indexing model (Robertson and Sparck Jones 1976). Whereas ranking can be done without the use of relevance feedback, retrieval will be further improved by the addition of this query modification technique. For further details, see Chapter 11.

14.8.2 Ranking and Clustering

Ranking retrieval systems have also been closely associated with clustering. Early efforts to improve the efficiency of ranking systems for use in large data sets proposed the use of clustering techniques to avoid dealing with ranking the entire collection (Salton 1971). It was also suggested that clustering could improve the performance of retrieval by pregrouping like documents (Jardine and van Rijsbergen 1971). For further details on clustering and its use in ranking systems, see Chapter 16.

14.8.3 Ranking and Boolean Systems

Because of the predominance of Boolean retrieval systems, several attempts have been made to integrate the ranking model and the Boolean model (for a summary, see Bookstein [1985]). The only methodology for this that has received widespread testing using the standard collections is the P-Norm method allowing the use of soft Boolean operators. This method is well described in Salton and Voorhees (1985) and in Chapter 15.

14.8.4 Use of Ranking in Two-level Search Schemes

The basic ranking search methodology described in the chapter is so fast that it is effective to use in situations requiring simple restrictions on natural language queries. Examples of these types of restrictions would be requirements involving Boolean operators, proximity operators, special publication dates, specific authors, or the use of phrases instead of simple terms. These situations can be accommodated by the basic ranking search system using a two-level search. The input query is processed similarly to a natural language query, except that the system notes the presence of special syntax denoting phrase limits or other field or proximity limitations. Using the following examples

clustering using "nearest neighbor" techniques

efficient clustering techniques [Author Willett]

the queries would be parsed into single terms and the documents ranked as if there were no special syntax. An efficient file structure is used to record which query term appears in which given retrieved document. The list of ranked documents is returned as before, but only documents passing the added restriction are given to the user. This usually requires a second pass over the actual document, that is each document marked as containing "nearest" and "neighbor" is passed through a fast string search algorithm looking for the phrase "nearest neighbor," or all documents containing "Willett" have their author field checked for "Willett." Although this seems a tedious method of handling phrases or field restrictions, it can be done in parallel with user browsing operations so that users are often unaware that a second processing step is occurring. This method was used in the prototype built by Harman and Candela (1990) and provided a very effective way of handling phrases and other limitations without increasing indexing overhead.

14.8.5 Ranking and Signature Files

It is possible to provide ranking using signature files (for details on signature files, see Chapter 4 on that subject). Croft and Savino (1988) provide a ranking technique that combines the IDF measure with an estimated normalized within-document frequency, using simple modifications of the standard signature file technique (see the chapter on signature files). That study also suggests that the ability of a ranking system to use the smaller inverted files discussed in this chapter makes storage and efficiency of ranking techniques competitive with that of signature files.

Signature files have also been used in SIBRIS, an operational information retrieval system (Wade et al. 1989), which is based on a two-stage search using signature files for a first cut and then ranking retrieved documents by term-weighting.

14.9 SUMMARY

This chapter has presented a survey of statistical ranking models and experiments, and detailed the actual implementation of a basic ranking retrieval system. Extensions to this basic system have been shown that modify the basic system to efficiently handle different retrieval environments.

REFERENCES

BELKIN, N. J. and W. B. CROFT. 1987. "Retrieval Techniques," in Williams, M. (Ed.), Annual Review of Information Science and Technology, ed. M. Williams, pp. 109-45. New York: Elsevier Science Publishers.

BARKLA, J. K. 1969. "Construction of Weighted Term Profiles by Measuring Frequency and Specificity in Relevant Items." Paper presented at the Second International Cranfield Conference on Mechanized Information Storage and Retrieval Systems, Cranfield, Bedford, England.

BERNSTEIN, L. M., and R. E. WILLIAMSON. 1984. "Testing of a Natural Language Retrieval System for a Full Text Knowledge Base." J. American Society for Information Science, 35(4), 235-47.

BOOKSTEIN, A. 1985. "Probability and Fuzzy-Set Applications to Information Retrieval," in Annual Review of Information Science and Technology, ed. M. Williams, pp. 117-51. New York: Knowledge Industry Publications, Inc.

BOOKSTEIN, A., and D. KRAFT. 1977. "Operations Research Applied to Document Indexing and Retrieval Decisions." J. Association for Computing Machinery, 24(3), 418-27.

BOOKSTEIN, A., and D. R. SWANSON. 1974. "Probabilistic Models for Automatic Indexing." J. American Society for Information Science, 25, 312-19.

BUCKLEY, C., and A. LEWIT. 1985. "Optimization of Inverted Vector Searches." Paper presented at the Eighth International Conference on Research and Development in Information Retrieval, Montreal, Canada.

BURKOWSKI, F. J. 1990. "Surrogate Subsets: A Free Space Management Strategy for the Index of a Text Retrieval System." Paper presented at ACM Conference on Research and Development in Information Retrieval, Brussels, Belgium.

CLEVERDON, C. 1983. "Optimizing Convenient Online Access to Bibliographic Databases." Information Services and Use, 4(1/2), 37-47.

COOPER, W. S., and M. E. MARON. 1978. "Foundations of Probabilistic and Utility-Theoretic Indexing." J. Association for Computing Machinery, 25(1), 67-80.

CROFT, W. B. 1983. "Experiments with Representation in a Document Retrieval System." Information Technology: Research and Development, 2(1), 1-21.

CROFT, W. B., and D. J. HARPER. 1979. "Using Probabilistic Models of Document Retrieval Without Relevance Information." Documentation, 35(4), 285-95.

CROFT, W. B., and L. RUGGLES. 1984. "The Implementation of a Document Retrieval System," in Research and Development in Information Retrieval, eds. G. Salton and H. J. Schneider, pp. 28-37. Berlin: Springer-Verlag.

CROFT, W. B., and P. SAVINO. 1988. "Implementing Ranking Strategies Using Text Signatures." ACM Transactions on Office Information Systems, 6(1), 42-62.

CUTTING, D., and J. PEDERSEN. 1990. "Optimizations for Dynamic Inverted Index Maintenance." Paper presented at ACM Conference on Research and Development in Information Retrieval, Brussels, Belgium.

DENNIS, S. F. 1964. "The Construction of a Thesaurus Automatically from a Sample of Text." Paper presented at the Statistical Association Methods for Mechanized Documentation. (National Bureau of Standards Miscellaneous Publication 269).

DOSZKOCS, T. E. 1982. "From Research to Application: The CITE Natural Language Information Retrieval System," in Research and Development in Information Retrieval, eds. G. Salton and H. J. Schneider, pp. 251-62. Berlin: Springer-Verlag.

FRAKES, W. B. 1984. "Term Conflation for Information Retrieval." Paper presented at the Third Joint BCS and ACM symposium on Research and Development in Information Retrieval, Cambridge, England.

HARMAN, D. 1986. "An Experimental Study of Factors Important in Document Ranking." Paper presented at ACM Conference on Research and Development in Information Retrieval, Pisa, Italy.

HARMAN, D., and G. CANDELA. 1990. "Retrieving Records from a Gigabyte of Text on a Minicomputer using Statistical Ranking." J. American Society for Information Science, in press.

HARPER, D. J. 1980. Relevance Feedback in Document Retrieval Systems: An Evaluation of Probabilistic Strategies. Doctoral dissertation, Jesus College, Cambridge, England.

HARTER, S. P. 1975. "A Probabilistic Approach to Automatic Keyword Indexing." J. American Society for Information Science, 26(5), 280-89.

JARDINE, N., and C. J. VAN RIJSBERGEN. 1971. "The Use of Hierarchic Clustering in Information Retrieval." Information Storage and Retrieval, 7(5), 217-40.

KNUTH, D. E. 1973. The Art of Computer Programming, Reading, Mass.: Addison-Wesley.

LOCHBAUM, K. E., and L. A. STREETER. 1989. "Comparing and Combining the Effectiveness of Latent Semantic Indexing and the Ordinary Vector Space Model for Information Retrieval." Information Processing and Management, 25(6), 665-76.

LUCARELLA, D. 1983. "A Document Retrieval System Based on Nearest Neighbor Searching." J. of Information Science, 6, 25-33.

LUHN, H. P. 1957. "A Statistical Approach to Mechanized Encoding and Searching of Literary Information." IBM J. Research and Development, 1(4), 309-17.

MARON, M. E., and J. L. KUHNS. 1960. "On Relevance, Probabilistic Indexing and Information Retrieval." J. Association for Computing Machinery, 7(3), 216-44.

MCGILL, M., M. KOLL, and T. NOREAULT. 1979. An Evaluation of Factors Affecting Document Ranking by Information Retrieval Systems. Report from the School of Information Studies, Syracuse University, Syracuse, New York.

MILLER, W. L. 1971. "A Probabilistic Search Strategy for Medlars." J. Documentation, 27(4), 254-66.

NOREAULT, T., M. KOLL, and M. MCGILL. 1977. "Automatic Ranked Output from Boolean Searches in SIRE." J. American Society for Information Science, 28(6), 333-39.

PERRY, S. A., and P. WILLETT. 1983. "A Review of the Use of Inverted Files for Best Match Searching in Information Retrieval Systems." J. Information Science, 6, 59-66.

RAGHAVAN, V. V., H. P. SHI, and C. T. YU. 1983. "Evaluation of the 2-Poisson Model as a Basis for Using Term Frequency Data in Searching." Paper presented at the Sixth International Conference on Research and Development in Information Retrieval, Bethesda, Maryland.

ROBERTSON, S. E., and K. SPARCK JONES. 1976. "Relevance Weighting of Search Terms." J. American Society for Information Science, 27(3), 129-46.

SALTON, G. 1971. The SMART Retrieval System -- Experiments in Automatic Document Processing. Englewood Cliffs, N.J.: Prentice Hall.

SALTON, G., and C. BUCKLEY. 1988. "Term-Weighting Approaches in Automatic Text Retrieval," Information Processing and Management, 24(5), 513-23.

SALTON, G., and M. E. LESK. 1968. "Computer Evaluation of Indexing and Text Processing." J. Association for Computing Machinery, 15(1), 8-36.

SALTON, G., and M. MCGILL. 1983. Introduction to Modern Information Retrieval. New York: McGraw-Hill.

SALTON, G., H. WU, and C. T. YU. 1981. "The Measurement of Term Importance in Automatic Indexing." J. American Society for Information Science, 32(3), 175-86.

SALTON, G., and C. S. YANG. 1973. "On the Specification of Term Values in Automatic Indexing." J. Documentation, 29(4), 351-72.

SPARCK JONES, K. 1972. "A Statistical Interpretation of Term Specificity and Its Application in Retrieval." J. Documentation, 28(1), 11-20.

SPARCK JONES, K. 1973. "Index Term Weighting." Information Storage and Retrieval, 9(11), 619-33.

SPARCK JONES, K. 1975. "A Performance Yardstick for Test Collections." J. Documentation, 31(4), 266-72.

SPARCK JONES, K. 1979a. "Experiments in Relevance Weighting of Search Terms." Information Processing and Management, 15(3), 133-44.

SPARCK JONES, K. 1979b. "Search Term Relevance Weighting Given Little Relevance Information." J. Documentation, 35(1), 30-48.

SPARCK JONES, K. 1981. Information Retrieval Experiment. London: Butterworths.

SRINIVASAN, P. 1989. "Intelligent Information Retrieval Using Rough Set Approximations." Information Processing and Management, 25(4), 347-61.

VAN RIJSBERGEN, C. J. 1976. "File Organization in Library Automation and Information Retrieval." J. Documentation, 32(4), 294-317.

WADE, S. J., P. WILLETT, and D. BAWDEN. 1989. "SIBRIS: the Sandwich Interactive Browsing and Ranking Information System." J. Information Science, 15, 249-60.

WALKER, S., and R. M. JONES. 1987. Improving Subject Retrieval in Online Catalogues, British Library Research Paper 24.

YU, C. T., and G. SALTON. 1976. "Precision Weighting -- An Effective Automatic Indexing Method." J. Association for Computing Machinery, 23(1), 76-88.

Go to Chapter 15 Back to Table of Contents


Information Retrieval: CHAPTER 15: EXTENDED BOOLEAN MODELS

 Return to Table of Contents

CHAPTER 15: EXTENDED BOOLEAN MODELS

E. Fox, S. Betrabet, M. Koushik

Department of Computer Science, Virginia Tech (Virginia Polytechnic Institute & State University), Blacksburg, VA 24061-0106

W. Lee

Systems Analyst, Acumenics Research and Technology, 9990 Lee Highway, Suite 580, Fairfax, VA 22030

Abstract

The classical interpretation of the Boolean operators in an information retrieval system is in general too strict. A standard Boolean query rarely comes close to retrieving all and only those documents which are relevant to a query. Many models have been proposed with the aim of softening the interpretation of the Boolean operators in order to improve the precision and recall of the search results. This chapter discusses three such models: the Mixed Min and Max (MMM), the Paice, and the P-norm models. The MMM and Paice models are essentially variations of the classical fuzzy-set model, while the P-norm scheme is a distance-based approach. Our experimental results indicate that each of the above models provides better performance than the classical Boolean model in terms of retrieval effectiveness.

15.1 INTRODUCTION

In the standard Boolean retrieval model, each document is associated with a set of keywords or index terms, and each query is in the form of a Boolean expression. The Boolean expression consists of a set of index terms connected by the Boolean operators and, or, and not. The documents retrieved for a given query are those that contain index terms in the combination specified by the query. As can be seen from the discussion in Chapter 10, the Boolean model is popular in operational environments because it is easy to implement and is very efficient in terms of the time required to process a query. Boolean retrieval systems are also capable of giving high performance in terms of recall and precision if the query is well formulated. However, the model has the following limitations, as discussed by a number of authors: Bookstein (1985), Cater & Kraft (1987), Cooper (1988), Wong et al. (1988), Paice (1984), Salton, Fox & Wu (1983).

The Boolean model gives counterintuitive results for certain types of queries. For example, consider a query of the form A and B and C and D and E. A document indexed by all but one of the above terms will not be retrieved in response to this query. Intuitively, it appears that the user would be interested in such a document, and that it should be retrieved. Similarly, for a query of the form A or B or C or D or E, a document indexed by any of these terms is considered just as important as a document indexed by some or all of them. This limitation of the Boolean model is due to its strict interpretation of the Boolean operators. Our experimental studies have shown that such a strict interpretation is not compatible with the user's interpretation; for example, results of the retrieval based on a strict interpretation have lower precision than results from a P-norm interpretation as shown by Salton, Fox & Wu (1983). Hence, we need to soften the Boolean operators, and account for the uncertainties that are present in choosing them. We can do this by making the and query behave a bit like the or query and the or query behave somewhat like the and query.

The standard Boolean model has no provision for ranking documents. However, systems combining features of both the Boolean and the vector models have been built to allow for ranking the result of a Boolean query, for example, the SIRE system (1977) Noreault, Koll & McGill. Ranking the documents in the order of decreasing relevance allows the user to see the most relevant document first. Also, the user would be able to sequentially scan the documents, and stop at a certain point if he/she finds that many of the documents are no longer relevant to the query.

During the indexing process for the Boolean model, it is necessary to decide whether a particular document is either relevant or nonrelevant with respect to a given index term. In the Boolean model, there is no provision for capturing the uncertainty that is present in making indexing decisions. Assigning weights to index terms adds information during the indexing process.

The Boolean model has no provision for assigning importance factors or weights to query terms. Yet, searchers often can rate or rank index terms in queries based on how indicative they are of desired content. It would be useful to allow weights to be assigned to (some) query terms, to indicate that the presence or absence of a particular query term is more important than that of another term.

Many retrieval models have been proposed as alternatives to the Boolean model. This chapter discusses three such models:

MMM

Paice

P-norm

These models avoid the strict interpretation of the Boolean operators, and attempt to provide a ranking of the retrieved documents in order of decreasing relevance to the query. They attempt to take into account the uncertainty involved in the indexing process. In addition, the P-norm model has the ability to consider weighted query terms. Experimental results have shown that these models can given an improvement of more than 100 percent in precision, at fixed recall levels, over the standard Boolean model as reported in Lee (1988).

We begin with a description of each model, and then discuss the data structures and algorithms that implement them. A few examples of the similarity computation in each model are also provided. We conclude with a short discussion of the relative merits of these models.

15.2 EXTENDED BOOLEAN MODELS

In the three models discussed below, a document has a weight associated with each index term. This document weight is a measure of the degree to which the document is characterized by that term. Without loss of generality, we assume that document weights for all index terms lie in the range [0, 1]. This is less restrictive than in the standard Boolean model, which limits the values to the extremes of the range, namely 0 and 1.

To retrieve documents relevant to a given query, we need to calculate the query-document similarity for documents in the collection. The query-document similarity is an attempt to predict the relevance of a document to the query. In the following subsections, we consider each model and its method for calculating similarity.

15.2.1 The MMM model

This model is based on the concept of fuzzy sets proposed by Zadeh (1965). In fuzzy-set theory, an element has a varying degree of membership, say dA, to a given set A instead of the traditional membership choice (is an element/is not an element).

In the Mixed Min and Max (MMM) model developed by Fox and Sharat (1986), each index term has a fuzzy set associated with it. The document weight of a document with respect to an index term A is considered to be the degree of membership of the document in the fuzzy set associated with A. The degree of membership for union and intersection are defined as follows in Fuzzy set theory:

dA  B = min(dA, db)

dA  B = max(dA, db)

According to fuzzy set theory, documents that should be retrieved for a query of the form A or B, should be in the fuzzy set associated with the union of the two sets A and B. Similarly, the documents that should be retrieved for a query of the form A and B, should be in the fuzzy set associated with the intersection of the two sets. Hence, it is possible to define the similarity of a document to the or query to be max(dA, dB), and the similarity of the document to the and query to be min(dA, dB). The MMM model attempts to soften the Boolean operators by considering the query-document similarity to be a linear combination of the min and max document weights.

Thus, given a document D with index-term weights dA1, dA2, . . . , dAn for terms A1, A2, . . . , An, and the queries

Qor = (A1 or A2 or . . . or An) and

Qand = (A1 and A2 and . . . and An),

the query-document similarity in the MMM model is computed as follows:

SlM(Qor, D) = Cor1 * max(dA1, dA2, . . . , dAn) + Cor2 * min(dA1, dA2, . . . , dAn)

SlM(Qand, D) = Cand1 * min(dA1, dA2, . . . , dAn) + Cand2 * max(dA1, dA2 . . . , dAn)

where Cor1, Cor2 are "softness" coefficients for the or operator, and Cand1, Cand2 are softness coefficients for the and operator. Since we would like to give the maximum of the document weights more importance while considering an or query and the minimum more importance while considering an and query, generally we have Cor1 > Cor2 and Cand1 > Cand2. For simplicity it is generally assumed that Cor1 = 1 - Cor2 and Cand1 = 1 - Cand2.

Our experiments (Lee & Fox 1988) indicate that the best performance usually occurs with Cand1 in the range [0.5, 0.8] and with Cor1 > 0.2. In general, the computational cost is low, and retrieval effectiveness is good, much better than with the standard Boolean model, as can be seen from Table 15.1 in section 15.4 below.

15.2.2 The Paice Model

The model proposed by Paice (1984) is also based on fuzzy-set theory. It is similar to the MMM model in that it assumes that there is a fuzzy set associated with each index term, and the document weight of a document with respect to an index term represents the degree of membership of the document with respect to the fuzzy set associated with that index term. However, while the MMM model considers only the maximum and the minimum document weights for the index terms while calculating the similarity, the Paice model takes into account all of the document weights.

Thus, given a document D with index-term weights dA1, dA2 ,. . . , dAn for terms A1, A2, . . . , An, and the queries

Qor = (A1 or A2 or ... An) and

Qand = (A1 and A2 and ... and An)

the query-document similarity in the Paice model is computed as follows:

where the di's are considered in descending order for or queries and in ascending order for and queries. Note that when n = 2, the Paice model behaves like the MMM model.

The experiments conducted by Lee & Fox (1988) have shown that setting the value of r to 1.0 for and queries, and to 0.7 for or queries, gives good retrieval effectiveness. The computational cost for this model is higher than that for the MMM model. This is because the MMM model only requires the determination of min or max of a set of term weights each time an and or or clause is considered, which can be done with an O(n) algorithm as described in Aho and Ullman (1974). The Paice model requires the term weights to be sorted in ascending or descending order, depending on whether an and clause or an or clause is being considered. This requires at least an 0(n log n) sorting algorithm. A good deal of floating point calculation is needed also.

15.2.3 The P-norm Model

Besides allowing document weights for index terms, the P-norm model also allows query terms to have weights. In the P-norm model, a document D with weights dA1, dA2, . . . , dAn with respect to index terms Al, A2, . . . , An is considered to be a point with coordinates (dA1, dA2, . . . , dAn) in an n-dimensional space. These document weights are generally obtained using term frequency and inverse document frequency statistics, with proper normalization as described by Fox (1983).

Consider an or query of the form dA1 or dA2 . . . or dAn. It is clear that the point having all the n coordinates equal to 0 (indicating that all the index terms are absent) is to be avoided for this query, whereas for an and query of the form dA1 and dA2. . . and dAn, the point with all the n coordinates equal to 1 (indicating that all the index terms have weight 1) is the most desirable point. Thus, it is possible to rank the documents (i.e., from those with highest similarity to those with lowest similarity) in order of decreasing distance from the point (0, 0, . . . , 0) for an or query, and in order of increasing distance from the point (1, 1, . . . , 1) for an and query.

In the P-norm model, the generalized queries are of the form:

Q or p  = (A1, a1) orP (A2, a2) orP ... orP (An, an), and

Q and p = (A1, a1) andP (A2, a2) andP ... andP (An, an).

We note that the operators have coefficients P which indicate the degree of strictness (from 1 for least strict to [image] for most strict, i.e., the Boolean case) of the operator. Also, the query terms have weights (i.e., ai) which indicate their relative importance. Sophisticated users may wish to assign the P-values and query weights. Extensive experimentation demonstrated that system-assigned p-values (e.g., p = 2 throughout a query) can give good results, and that uniform query weighting of 1 or based on inverse document frequency will lead to effective retrieval as described by Fox (1983).

The query-document similarity for the P-norm model is calculated using the following formulae:

Numerous experiments have shown the P-norm model to be very effective as reported in Salton, Fox & Wu (1983), Fox & Sharat (1986), Lee & Fox (1988), Fox (1983). However, when the P-value is greater than one, the computational expense can be high. This is because of the need for expensive exponentiation computations.

15.3 IMPLEMENTATION

As in any other information retrieval system, we need to convert the given set of documents and queries into a standard format. We first discuss the data structures used in our implementation, and then follow up with a discussion of the actual algorithms.

15.3.1 Data Structures

There are several important data structures for both the document and query representations.

Document structure

Given a set of documents, the goal is to index them, that is, find the index terms in the collection, and assign document weights for each document with respect to each index term. To calculate the document weight for an index term, a weighting technique such as normalized (tf * idf)ik can be used. A more detailed discussion of weighting techniques is provided in Chapter 10. The data structure for a document consists of an array doc _wts that stores the document weights as shown below. The element doc_wt [i] is the document weight for the ith index term.

typedef struct

{

float doc_wts[NUM_ITERM];  /*document weight vector*/

} DOC_STRUCT

Query Structure

We assume that each query has an id associated with it, and is in the form of a tree with the operators as internal nodes and index terms as the leaves (see Figure 15.1). The data structure used to store a query is as follows:

typedef struct

{  long         query_id;           /* query id */

TREE_NODE   *beg_node  _array;  /* array of query nodes */

ITERM_TUPLE *beg_iterm_array;   /* array of index terms */

OP_TUPLE    *beg_op_array;      /* array of operators */

} QUERY_STRUCT

We see from the query structure that there are three arrays associated with each query (see Figure 15.2):

       an array of the nodes in the query tree

       an array of index terms

       an array of operators

Figure 15.1: Structure of query (A or B or C) and (D or E)

Figure 15.2: Structure of QUERY-STRUCT

In the query structure we store the address of the first element of each array. In our implementation we assume that the first element of the node array is the root of the query tree.

The tree-node structure is as follows:

typedef struct

{

int         iterm_op_index;   /* index of index-term/operator for

leaf/non-leaf node resp. */

short       child_index;      /* index of the left child of the

current node */

short       sibling_index;    /* index of right sibling of the

current node */

} TREE_NODE     /* structure of a node in the query tree */

In this structure, the child_index is the index of the leftmost child of the current node within the array of nodes. Similarly, sibling_index is the index of the next node to the right of the current node within the array of nodes. If a node does not have any children / siblings to its right, the value of its child_index / sibling_index will be equal to UNDEF (-1).

It is possible to determine whether the current node is an operator node or an index-term node by examining the value of child_index (see Figure 15.3). Since index terms can only be leaves of the tree, they cannot have any children, whereas operators will have at least one child. Hence, if child_index has value UNDEF, then it implies that the current node is an index-term node; otherwise it implies that the node is an operator node. The iterm_op_index in the TREE_NODE structure is the index number of the index-term/operator associated with the current node within the array of index-term/operators.

The data structure that stores information about an index term is as follows:

typedef struct

{

long        iterm_num;   /* unique ID given to index-term */

float       iterm_wt;    /* weight of the index-term */

} ITERM_TUPLE   /* structure of an index term */

Figure 15.3: Data structures generated for query ((A or B or C) and (D or E))

Here, iterm_num is the unique number given to an index term. Note that iterm_wt, which is the weight of the index term in the query, is used only for P-norm queries.

The data structure for an operator is as follows:

typedef struct

{

int           op_type;   /* type ot operator -

AND_OP / OR_OP / NOT_OP */

long          op_coeff;  /* coefficient of the operator -

= 0.0 for NOT_OP

= C_or1 / C _and1 for MMM model,

= r value for Paice model, and

= P value for P-norm model */

float         op_wt;     /* wt. of the operator */

} OP_TUPLE        /* structure of an operator */

If the MMM model is being used, then op_coeff is the Cor1 / Cand1 value. (Note: We assume that Cor2 = 1 - Cor1 and Cand2 = 1 - Cand1.) If on the other hand the Paice model is being used, then the op_coeff is the r value. If the P-norm model is being used, then op_coeff is the P-value.

15.3.2 Algorithms and Examples

Given a query and a set of documents in the standard form, we use the following algorithm to retrieve and rank documents:

For each document

Find similarity;

Sort similarities;

Display document / similarity data in descending order.

In order to calculate the query-document similarity, we start from the root of the query tree, and find the similarity for each child. (Refer to the appendix for the C program.) If the child is an index term, then the similarity is the document weight for that index term. If it is an operator, then we recursively calculate the similarity for each child of that node. Once the similarity value is determined for a child, further processing is different for the various models and is described below:

       For the MMM model:

We update the values of two variables, maximum and minimum (both of type double), where

maximum = max(dA1 dA2,...,dAn), and

minimum = min(dA1 dA2,...,dAn).

       For the Paice model:

We store the similarity in an array (of type double), child_sims, which will contain the similarities of the children of the current node.

       For the P-norm model:

We update the values of two variables, numerator and denominator (both of type double), where

Once all the children of the node have been considered, we use the appropriate formula to find the actual similarity.

       For the MMM model, we can calculate the similarity for the node by using the following formula:

SIM(Q,D) = Cop * maximum + (1 - Cop) * minimum,

For example, if the document weights for the index terms A, B, and C in a query Q of the form A or B or C and for a particular document D are 0.5, 0.8, and 0.6, respectively, then the maximum of the three is 0.8, and the minimum 0.5. If the coefficient of the or operator is 0.7, then, using the above formula, the MMM similarity between the given document and query is

SIM(Q,D) = 0.7 * 0.8 + (1 - 0.7) * 0.5 = 0.71

If the Paice similarity is being calculated, we begin by sorting the array child_sims in descending order for an OR node, and in ascending order for an AND node. We then use the following formula to calculate the Paice query-document similarity.

For the query Q given above, since the operator is OR_OP we sort the similarities in descending order. We will therefore have the similarities in the order: 0.8, 0.6, 0.5. Hence, using the formula given above, with the assumption that r = 0.7, the Paice similarity between the given document and query will then be

SIM(Q,D) = 0.70*0.8 + 0.71*0.6 + 0.72*0.5/0.70 + 0.71 + 0.72

 = 1.465/2.19 = 0.6689

       If the P-norm similarity is being calculated, then we use the values of the numerator and denominator in the following formula to calculate the P-norm query-document similarity:

SIM(Qor,D) = (numerator/denominator)1/p_value

SIM(Qand,D) = 1 - (numerator/denominator)1/p_value

If for the query Q given above, the query term weights are 0.5 and the operator coefficient or the P or operator has the coefficient value is 2, then the value of the numerator and denominator after considering the three index terms will be

Index  Term       Numerator        Denominator

  1      A   0 + 0.52 * 0.52       0 + 0.52

             = 0.0625              = 0.25

  2      B   0.0625 + 0.52 * 0.82  0.25 + 0.52

             = 0.2225              = 0.50

  3      C   0.225 + 0.52 * 0.62   0.50 + 0.52

             = 0.3125              = 0.75.

Using the final values of the numerator and denominator and the formula given above, the query-document similarity SIM (Qor, D) will be

SIM(Qor,D) = (0.3125/0.75)1/2 = 0.6455

The C programs that implement these algorithms are given in the appendix. The routine cal_sim() is called with the array containing the document weights which are of type float as the first parameter, the query vector which is of type QUERY_STRUCT as the second parameter, and the third argument sim_type which determines the type of similarity being computed, that is, MMM, Paice, or P-norm. Based on the value of the third parameter, one of the following sets of routines is called for further computations:

       mmm_sim( ): MMM model

       paice_sim( ): Paice model

       update_numer_denom( ) & p_norm_sim( ): P-norm model

15.4 CONCLUSION

Numerous experiments performed with the extended Boolean models have shown that they give much better performance than the standard Boolean model as reported by Lee & Fox (1988). Table 15.1 gives the percentage improvement in the average precision that these models showed over the standard Boolean model for three test collections: the CISI, CACM, and the INSPEC collections. As can be seen in the table, all three models show substantial improvement over the standard Boolean model. In fact, for the CACM and the INSPEC collections, all three models show more than 100 percent improvement.

Table 15.1: Percentage Improvements Over Standard Boolean Model Test Collection

Scheme           CISI                CACM              INSPEC

        -------------------  ------------------  ------------------

        Best Average         Best Average        Best Average

        Precision      Rank   Precision    Rank   Precision    Rank

-------------------------------------------------------------------

P-norm      79          1        106         2       210         1

Paice       77          2        104         3       206         2

MMM         68          3        109         1       195         3

The P-norm model is computationally expensive because of the number of exponentiation operations that it requires. The Paice model requires one to sort the similarities of the children. This adds to its computational overhead. The MMM model clearly has the least computational overhead of the three models.

The standard Boolean model is still the most efficient model. But by using any one of the extended Boolean models, it is possible to get much better performance. Based on experimental studies, the MMM model appears to be the least expensive, while the P-norm model gives the best results in terms of average precision.

REFERENCES

AHO, A. J., and J. ULLMAN. 1974. "The Design and Analysis of Computer Algorithms." Reading, Mass.: Addison-Wesley.

BOOKSTEIN, A. 1985. "Probability and Fuzzy-Set Applications to Information Retrieval," in M. E. Williams, ed., Annual Review of Information Science and Technology (ARIST), 20, 117-51.

CATER, S. C., and D. H. KRAFT. 1987, June. "TIRS: A Topological Information Retrieval System Satisfying the Requirements of the Waller-Kraft Wish List." Presented at the 10th Annual Int'l ACM-SIGIR Conference on R&D in Information Retrieval, 171-80.

COOPER, W. S. 1988. "Getting Beyond Boole." Information Processing & Management 24(3), 243-48.

FOX, E. A. 1983. "Extending the Boolean and Vector Space Models of Information Retrieval with P-Norm Queries and Multiple Concept Types." Cornell University, August.

FOX, E. A., and S. SHARAT. 1986. "A Comparison of Two Methods for Soft Boolean Interpretation in Information Retrieval." Technical Report TR-86-1, Virginia Tech, Department of Computer Science.

LEE, W. C., and E. A. FOX. 1988. "Experimental Comparison of Schemes for Interpreting Boolean Queries." Virginia Tech M.S. Thesis, Technical Report TR-88-27 Department of Computer Science.

NOREAULT, T., M. KOLL, and M. J. MCGILL. 1977. "Automatic Ranked Output from Boolean Searches in SIRE." J. American Society for Information Science, 28(6), 333-39.

PAICE, C. P. 1984. "Soft Evaluation of Boolean Search Queries in Information Retrieval Systems." Information Technology, Res. Dev. Applications, 3(1), 33-42.

SALTON, G., E. A. FOX, and H. WU. 1983. "Extended Boolean Information Retrieval." Communications of the ACM, 26(12), 1022-36.

WONG, S. K. M., W. ZIARKO, V. V. RAGHAVAN, and P. C. N. WONG. 1988. "Extended Boolean Query Processing in the Generalized Vector Space Model." Information Systems 14(1), 47-63.

ZADEH, L. A. 1965. "Fuzzy Sets," Information and Control, 8, 338-53.

APPENDIX

/**

** Compute query-document similarity using MMM, Paice and P-norm

** formulas

**/

#include <stdio.h>

#include <math.h>

#define UNDEF-l

#define NOT_OP -2

#define AND_OP -3

#define OR_OP  -4

#define MMM_SIM    1

#define PAICE_SIM  2

#define P_NORM_SIM 3

#define P_INFINITY 0.0      /* Used only for P-norm model */

typedef struct

{

long iterm_num;           /* Unique ID given to the index-term */

float iterm_wt;           /* Weight of index-term (P-norm queries only) */

} ITERM_TUPLE;            /* Structure of an index-term in a query */

typedef struct

{

int op_type;              /* operator type - AND_OP, OP_OP, NOT_OP */

float op_coeff;           /* Coefficient of the operator

= 0.0 for NOT_OP,

= C_or1/C_and1 for MMM model,

= r value for Paice model, and

= P value for P-norm model */

float op_wt;              /* Wt. of operator - (P-norm queries only) */

} OP_TUPLE;               /* Structure of an operator in a query */

typedef struct

{

long iterm_op_index;            /* If child_index = UNDEF then concept

index, else operator index */

short child_index;              /* Current node's left child index */

short sibling_index;            /* Current node's right sibling index */

} TREE_NODE;                    /* Structure of a node in the query tree */

typedef struct

{

long     query_id;                     /* Query id */

TREE_NODE   *beg_node_array;           /* Array of query nodes */

ITERM_TUPLE  *beg_iterm_array;         /* Array of index terms */

OP-TUPLE     *beg_op_array;            /* Array of operators */

} QUERY-STRUCT;                        /* Structure of a query */

#define NUM_ITERMS 1000                   /* Max. no. of index-terms

allowed in a document */

/***************************************************************************

double

calc_sim ( doc_wts, query_vec, sim_type )

Returns: The MMM / Paice / P-norm similarity between the given

document and query.

Purpose: To compute:

MMM_SIM (Doc, Query ) if sim_type = MMM_SIM, or

PAICE_SIM (Doc, Query ) if sim_type = PAICE_SIM, or

P_NORM_SIM (Doc, Query ) if sim_type = P_NORM_SIM,

Plan:  Call the routine to calculate the similarity by passing

to it the index of the root of the query tree (= zero).

***************************************************************************/

double

calc_sim ( doc_wts, query_vec, sim_type )

float  doc_wts[NUM_ITERMS];                  /* In: doc. weights vector  */

register QUERY_STRUCT *query_vec;            /* In: Given Query vector   */

short  sim_type;                             /* In: Similarity type flag */

{

double calc_tree_sim ();

if ( ( sim_type != MMM_SIM ) && ( sim_type != PAICE-SIM ) &&

( sim_type != P_NORM_SIM ) )

fprintf ( stderr, "calc_sim: illegal similarity\

type %d, query %d\n", sim_type,

query_vec-> query_id );

return ( calc_tree_sim (doc_wts, query_vec, 0, sim_type ) );

}

/***************************************************************************

static double

calc_tree_sim ( doc_wts, query_vec, root_index, sim_type )

Returns: The similarity between the given document and the

subtree of the query tree with its root at ROOT_INDEX

Purpose: To compute: SIM (Doc, Query-subtree)

Plan:   If the root of the subtree is an index-term then

similarity is the document wt. for that index-term

else, if it is an operator then,

if the operator == NOT_OP then

(1) Calculate the similarity of the child of

the current node (CHILD_VALUE)

(2) Return (1.0 - CHILD_VALUE)

else if the operator == AND_OP or OR_OP then

(1) Calculate the similarity considering

each child of the current node

(2) If (SIM_TYPE == MMM_SIM) then

Find the maximum and the minimum of

these similarities,

else if (SIM_TYPE == PAICE_SIM) then

Store this similarity in the array

child_sims,

else if (SIM_TYPE == P_NORM_SIM) then

Find the numerator and denominator

to be used in the formula

else

Print "invalid sim_type".

(3) Use the appropriate similarity computation

formula to compute the similarity for

the current node.

***************************************************************************/

static double

calc_tree_sim ( doc_wts, query_vec, root_index, sim_type)

float   doc_wts [NUM_ITERMS];             /* In: Document weights */

register QUERY_STRUCT *query_vec;         /* In: Query vector */

short root_index;                         /* In: Index of the root of the

subtree to be evaluated */

short sim_type;                           /* In: Similarity type flag */

{

register TREE_NODE *root_ptr;             /* Addr. of the root of the

subtree being considered */

register TREE_NODE *child_ptr;            /* Addr. of a child of the root

of subtree being considered */

register OP_TUPLE *op_ptr;                /* Addr. of current operator */

register ITERM_TUPLE *iterm_ptr;          /* Addr. of current index-term */

long           doc_index;                 /* Index of the weight of the

current index-term within

DOC-WTS array */

short          child_index;               /* Index of the child of the root

within the set of tree nodes */

double         child_value;               /* Sim. of child's subtree */

/* Declarations for MMM model */

double         maximum;                   /* Maximum (children's sim.) */

double         minimum;                   /* Minimum (children's sim.) */

double         mmm_sim ();                /* Func. computing MMM sim. */

/* Declarations for Paice model */

double   child_sims[NUM_ITERMS];           /* Array to store sim. of all

the children of the root */

int     curr_i;                            /* Number of sim. stored in

child_sims */

double   paice_sim ();              /* Func. computing Paice sim. */

/* Declarations for P-norm model */

double         child_wt;               /* Wt. of the child */

double         numerator;              /* Numerator in sim. formula */

double         denominator;            /* Denominator in sim. formula */

void           update_numer_denom();   /* Func. updating value of */

/* numerator & denominator */

double         p_norm_sim ();          /* Func. computing P-norm sim. */

double calc_tree_sim();                /* Recursive call for each subtree */

/* Addr. of first node of the tree + index of the current node

= the addr. of the root of the subtree under consideration */

root_ptr = query_vec->beg_node_array + root_index;

if  UNDEF == root_ptr->child_index)

{

/* If node is an index-term, then return its doc-wt. Index in */

/* the doc_wts array is the ITERM_NUM of current index-term */

iterm_ptr = query_vec->beg_iterm_array + root_ptr->iterm_op_index;

doc-index = iterm_ptr->iterm_num;

return ( (double) doc_wts [doc_index] );

}

else

{ /* If current node is an operator, then compute its sim. */

op_ptr = query_vec->beg_op_array + root_ptr->iterm_op_index;

if (  ( op_ptr->op_type != NOT_OP ) &&

( op_ptr->op_type != OR_OP ) &&

( op_ptr->op_type != AND_OP ) )

{             /* if neither NOT, OR or AND */

fprintf ( stderr, "calc_tree sim:\

illegal operator type %d, query %d.\n",

op_ptr->op_type, query_vec->query_id);

return ( (double) UNDEF );

}

switch (op_ptr->op_type)

{

case (NOT_OP):  /* If the operator is NOT_OP */

if (UNDEF != (query_vec->beg_node_array +

root_ptr->child_index)->sibling_index)

{   /* NOT_OP operator can have only one child */

fprintf(stderr,

"calc_tree_sim: NOT operator has more\

than one child.\n');

return ( (double) UNDEF);

}

/* if only child, return (1.0 - similarity of child) */

if ((double) UNDEF == (child_value =

calc_tree_sim ( doc_wts, query_vec,

root_ptr->child_index, sim_type)))

return ( (double) UNDEF );

return ( 1.0-child_value );

break;

case (OR_OP):

case (AND_OP):  /* If the operator is OR_OP or AND_OP */

maximum = -99999.0;

minimum = 99999.0;        /* Init. for MMM model */

curr_i = -1;              /* Init. for Paice model */

numerator = 0.0;

denominator = 0.0;        /* Init. for P-norm model */

/* Start with the first child of the current node, *

* consider each of its siblings, until none left */

for (child_index = root_ptr->child_index;

UNDEF != child_index;

child_index = (query_vec->beg_node_array +

child_index)->sibling_index )

{

if ( (double)UNDEF == (child_value =

calc_tree_sim ( doc_wts, query_vec, child_index,

sim_type)))

return ((double UNDEF);

switch (sim_type)

{

case (MMM_SIM): /* update max and mim */

maximum = (child_value > maximum) ? child_value

: maximum;

minimum = (child_value < minimum) ? child_value

: minimum;

break;

case (PAICE_SIM):

curr_i = curr_i + 1;

child_sims [curr_i] = child_value;

break;

case (P_NORM_SIM): /* Find the wt. of the child */

child_ptr = query_vec->beg_node_array +

root_ptr->child_index;

if (UNDEF == child_ptr->child_index)

child_wt = (query_vec->beg_iterm_array +

child_ptr->iterm_op_index)->iterm_wt;

else

child_wt = (query_vec->beg_op_array +

child_ptr->iterm_op_index)->op_wt;

update_numer_denom ( child_value, child_wt,

op_ptr->op_type, op_ptr->op_coeff,

&numerator, &denominator );

break;

}   /* switch - sim_type */

}     /* for */

/* After considering all the children, compute the *

* sim. of current subtree with appropriate formula */

if ( sim_type == MMM_SIM )

return ( mmm_sim ( op_ptr->op_coeff,

op_ptr->op_type, maximum, minimum));

else

if ( sim_type == PAICE_SIM )

return (paice_sim (op_ptr->op_coeff,

op_ptr->op_type, child_sims, curr_i ));

else

if ( sim_type == P_NORM_SIM )

return ( p_norm_sim (op_ptr->op_coeff,

op_ptr->op_type, numerator, denominator ) );

}     /* switch - op_ptr->op_type */

}     /* else */

}

/***************************************************************************

double

mmm_sim (coeff, type, maximum, minimum)

Returns: The MMM similarity

Purpose: To calculate the MMM similarity using

MAXIMUM and MINIMUM.

SIM(Qor, D) = coeff * MAXIMUM + (1-coeff) * MINIMUM

SIM(Qand, D) = coeff * MINIMUM + (1-coeff) * MAXIMUM

Plan:   Depending on the type of the operator use the

appropriate formula

***************************************************************************/

double

mmm_sim ( op_coeff, op_type, maximum, minimum )

register float op_coeff;  /* In: Value of the coefficient */

int op_type;                 /* In: Type of operator */

double maximum;                   /* In: Maximum of the similarities */

double minimum;                   /* In: Minimum of the similarities */

{

if ( op_type == OR_OP )

return ( (op_coeff * maximum + (1 - op_coeff) * minimum));

else

if ( op_type == AND_OP )

return ( (op_coeff * minimum + (1 - op_coeff) * maximum));

}

/***************************************************************************

double

paice_sim (op_coeff, op_type, child_sims, num_i)

Returns: The Paice similarity

Purpose: To calculate the Paice similarity using the

Paice similarity computation formula

Plan:   Sort the array in ascending order

If the operator is OR_OP then:

numerator = SUM (child_sims[i] * op_coeff/\(num_i - i) ) ;

for i = 0 to num_i

denominator = SUM (op_coeff/\(num_i-i));

for i = 0 to num_i

else if the operator is AND_OP then:

numerator = SUM (child_sims[i] *op_coeff/\(i) );

for i = 0 to num_i

denominator = SUM (op_coeff/\(i) );

for i = 0 to num_i

***************************************************************************/

double

paice_sim ( op_coeff, op_type, child_sims, num_i )

register float op_coeff;                 /* In: Coefficient r */

int    op_type;                          /* In: Type of operator */

double  child_sims [NUM_ITERMS];     /* In: Array containing sim. */

int    num_i;                            /* In: No. of elements in

child_sim */

{

int    i;

double  numerator;

double  denominator;

double  power;

void   qsort ();               /* Quick Sort Func. in C lib. */

int    comp_double ();             /* Func. used by qsort */

qsort ( (char *) child_sims, num_i,

sizeof (double), comp_double );

numerator = 0;

denominator = 0;

if ( op_type == OR_OP )

{

for (i = 0; i <=num_i; i++)

{

power = pow((double) op_coeff, (double)num_i-i);

numerator = numerator + power*child_sims[i];

denominator = denominator + power;

}

return (numerator/denominator);

}

else

if(op_type == AND_OP)

{

for (i = 0; i <= num_i; i++)

{

power = pow((double) op_coeff, (double)i);

numerator = numerator + power * child_sims[i];

denominator = denominator + power;

}

return ( numerator / denominator );

}

}

/***************************************************************************

int

comp_double ( d1, d2 );

Returns: int

Purpose: Compares d1 and d2 and returns -1, 0, or 1.

***************************************************************************/

int

comp_double (d1, d2)

double *d1, *d2;

{

if (*d1 < *d2)

return (-1);

else

if (*d1 == *d2)

return (0);

else

return (1);

}

/***************************************************************************

void

update_numer_denom (value, weight, op_type, p_value,

numerator, denominator)

Returns: Void

Purpose: Update the values of NUMERATOR and DENOMINATOR

Plan:  If P_value == P_INFINITY then

if VALUE > NUMERATOR then

NUMERATOR = VALUE

DENOMINATOR = 1

else

if OP_TYPE == OR_OP then

NUMERATOR = NUMERATOR + weight/\P_value * value/\P_value

DENOMINATOR = DENOMINATOR + weight/\P_value

if OP_TYPE == AND_OP then

NUMERATOR = NUMERATOR + weight/\P_value *

(1 - value) /\P_value

DENOMINATOR = DENOMINATOR + weight/\P_value

***************************************************************************/

void

update_numer_denom ( value, weight, op_type, p_value,

numerator, denominator )

double value;                    /* In: The sim. of the child */

double weight;                   /* In: The query weight */

register float p_value;          /* In: The p_value */

int op_type;                     /* In: The type of the operator */

register double *numerator;   /* Out: Numerator in p-norm sim.

calculation formula */

register double *denominator;  /* Out: Denominator in p-norm sim.

calculation formula */

{

double power

double pow ()

if (p_value == P_INFINITY)

{

*denominator = 1;

if (value > *numerator)

*numerator = value;

}

else

switch (op_type)

{

case (OR_OP):

power = pow (weight, p_value);

*numerator = *numerator + power * pow(value,p_value);

*denominator = *denominator + power;

break;

case (AND_OP):

power = pow (weight, p_value );

*numerator = *numerator + power * pow(1-value,p_value);

*denominator = *denominator + power;

break;

}

}

/***************************************************************************

double

p_norm_sim (p_value,op_type,numerator,denominator)

Returns: The P-norm similarity

Purpose: To calculate the P-norm similarity using the

P-norm similarity computation formula

Plan:   Depending on the type of the operator use the

appropriate formula

SIM(Q (P_INFINITY) , D) = NUMERATOR

SIM (Qor_P, D) = (NUMERATOR / DENOMINATOR) /\ (1/P)

SIM (Qand_P, D) = 1 -  (NUMERATOR / DENOMINATOR) /\ (1/P)

***************************************************************************/

double

p_norm_sim ( p_value, op_type numerator, denominator)

register float p_value;     /* In: P value */

int op_type;                /* In: Type of operator */

double numerator;           /* In: Numerator */

double denominator;         /* In: Denominator */

{

double pow ();

if (p_value == P_INFINITY)

return (numerator);

else

if (op_type == OR_OP)

return (pow (numerator / denominator, 1/p_value) );

else

if (op_type == AND_OP)

return (1 - pow (numerator / denominator, 1/p_value) );

}

Go to Chapter 16     Back to Table of Contents


Information Retrieval: CHAPTER 16: CLUSTERING ALGORITHMS

 Return to Table of Contents

CHAPTER 16: CLUSTERING ALGORITHMS

Edie Rasmussen

University of Pittsburgh

Abstract

Cluster analysis is a technique for multivariate analysis that assigns items to automatically created groups based on a calculation of the degree of association between items and groups. In the information retrieval (IR) field, cluster analysis has been used to create groups of documents with the goal of improving the efficiency and effectiveness of retrieval, or to determine the structure of the literature of a field. The terms in a document collection can also be clustered to show their relationships. The two main types of cluster analysis methods are the nonhierarchical, which divide a data set of N items into M clusters, and the hierarchical, which produce a nested data set in which pairs of items or clusters are successively linked. The nonhierarchical methods such as the single pass and reallocation methods are heuristic in nature and require less computation than the hierarchical methods. However, the hierarchical methods have usually been preferred for cluster-based document retrieval. The commonly used hierarchical methods, such as single link, complete link, group average link, and Ward's method, have high space and time requirements. In order to cluster the large data sets with high dimensionality that are typically found in IR applications, good algorithms (ideally O(N2) time, O(N) space) must be found. Examples are the SLINK and minimal spanning tree algorithms for the single link method, the Voorhees algorithm for group average link, and the reciprocal nearest neighbor algorithm for Ward's method.

16.1 CLUSTER ANALYSIS

16.1.1 Introduction

Cluster analysis is a statistical technique used to generate a category structure which fits a set of observations. The groups which are formed should have a high degree of association between members of the same group and a low degree between members of different groups (Anderberg 1973). While cluster analysis is sometimes referred to as automatic classification, this is not strictly accurate since the classes formed are not known prior to processing, as classification implies, but are defined by the items assigned to them.

Because there is no need for the classes to be identified prior to processing, cluster analysis is useful to provide structure in large multivariate data sets. It has been described as a tool of discovery because it has the potential to reveal previously undetected relationships based on complex data (Anderberg 1973). An early application of cluster analysis was to determine taxonomic relationships among species. Psychiatric profiles, medical and clinical data, census and survey data, images, and chemical structures and properties have all been studied using cluster analytic methods, and there is an extensive and widely scattered journal literature on the subject.

Basic texts include those by Anderberg (1973), Hartigan (1975), Everitt (1980), Aldenderfer and Blashfield (1984), Romesburg (1984), Spath (1985), Jain and Dubes (1988) and Kaufman (1990). Taxonomic applications have been described by Sneath and Sokal (1973), and social science applications by Lorr (1983) and Hudson and Associates (1982). Comprehensive reviews by Lee (1981), Dubes and Jain (1980) and Gordon (1987) are also recommended.

Because cluster analysis is a technique for multivariate analysis that has application in many fields, it is supported by a number of software packages which are often available in academic and other computing environments. Most of the methods and some of the algorithms described in this chapter are found in statistical analysis packages such as SAS, SPSSX, and BMDP and cluster analysis packages such as CLUSTAN and CLUSTAR/CLUSTID. Brief descriptions and sources for these and other packages are provided by Romesburg (1984).

16.1.2 Applications in Information Retrieval

The ability of cluster analysis to categorize by assigning items to automatically created groups gives it a natural affinity with the aims of information storage and retrieval. Cluster analysis can be performed on documents in several ways:

Documents may be clustered on the basis of the terms that they contain. The aim of this approach has usually been to provide more efficient or more effective retrieval, though it has also been used after retrieval to provide structure to large sets of retrieved documents. In distributed systems, clustering can be used to allocate documents for storage. A recent review (Willett 1988) provides a comprehensive summary of research on term-based document clustering.

Documents may be clustered based on co-occurring citations in order to provide insights into the nature of the literature of a field (e.g., Small and Sweeney [1985]).

Terms may be clustered on the basis of the documents in which they co-occur, in order to aid in the construction of a thesaurus or in the enhancement of queries (e.g., Crouch [1988]).

Although cluster analysis can be easily implemented with available software packages, it is not without problems. These include:

Selecting the attributes on which items are to be clustered and their representation.

Selecting an appropriate clustering method and similarity measure from those available .

Creating the clusters or cluster hierarchies, which can be expensive in terms of computational resources.

Asessing the validity of the result obtained.

If the collection to be clustered is a dynamic one, the requirements for update must be considered.

If the aim is to use the clustered collection as the basis for information retrieval, a method for searching the clusters or cluster hierarchy must be selected.

The emphasis in this chapter will be on the range of clustering methods available and algorithms for their implementation, with discussion of the applications and evaluation that have been carried out in an information retrieval environment. The following notation will be used: N for the numberof items Di in a data set, L for its dimensionality, and M for the number of clusters Ci created. It will be assumed that the items to be clustered are documents, and each document Di (or cluster representative Ci) is represented by (weighti1, . . ., weightiL), where weightik is the weight assigned to termk in Di or Ci. The choice of an appropriate document representation is discussed elsewhere in this text; a summary of research on term-weighting approaches is provided by Salton and Buckley (1988).

16.2 MEASURES OF ASSOCIATION

16.2.1 Introduction

In order to cluster the items in a data set, some means of quantifying the degree of association between them is required. This may be a distance measure, or a measure of similarity or dissimilarity. Some clustering methods have a theoretical requirement for use of a specific measure (Euclidean distance for Ward's method, for example), but more commonly the choice of measure is at the discretion of the researcher.

While there are a number of similarity measures available, and the choice of similarity measure can have an effect on the clustering results obtained, there have been only a few comparative studies (summarized by Willett [1988]). In cluster-based retrieval, the determination of interdocument similarity depends on both the document representation, in terms of the weights assigned to the indexing terms characterizing each document, and the similarity coefficient that is chosen. The results of tests by Willett (1983) of similarity coefficients in cluster-based retrieval suggest that it is important to use a measure that is normalized by the length of the document vectors. The results of tests on weighting schemes were less definitive but suggested that weighting of document terms is not as significant in improving performance in cluster-based retrieval as it is in other types of retrieval. Sneath and Sokal (1973) point out that simple similarity coefficients are often monotonic with more complex ones, and argue against the use of weighting schemes. The measures described below are commonly used in information retrieval applications. They are appropriate for binary or real-valued weighting scheme.

16.2.2 Similarity Measures

A variety of distance and similarity measures is given by Anderberg (1973), while those most suitable for comparing document vectors are discussed by Salton (1989). The Dice, Jaccard and cosine coefficients have the attractions of simplicity and normalization and have often been used for document clustering.

Dice coefficient:

If binary term weights are used, the Dice Coefficient reduces to:

where C is the number of terms that Di and Dj have in common, and A and B are the number of terms in Di and Dj.

Jaccard coefficient:

Cosine coefficient:

16.2.3 The Similarity Matrix

Many clustering methods are based on a pairwise coupling of the most similar documents or clusters, so that the similarity between every pair of points must be known. This necessitates the calculation of the similarity matrix; when the similarity measure is symmetric (Sij = Sji), the lower triangular matrix is sufficient (Figure 16.1).

The inverted file algorithm is particularly useful in limiting the amount of computation required to calculate a similarity matrix, if the similarity measure used is one that results in a 0 value whenever a document-document or document-cluster pair have no terms in common (Willett 1980; Perry and Willett 1983). The document term list is used as an index to the inverted index lists that are needed for the similarity calculation. Only those document/cluster pairs that share at least one common term will have their similarity calculated; the remaining values in the similarity matrix are set to 0. The inverted file algorithm is as follows:

Figure 16.1: Similarity matrix

for ( docno = 0; docno < n; docno++ )

{

for ( i = 0; i < doclength; i++ )

{

retrieve_inverted_list ( term[i] );

for ( j = 0; j < invlength; j++ ) counter[doc[j]]++;

}

for ( doc2 = 0; doc2 < n; doc2++ )

{

if (counter [doc2]) calc_similarity( docno, doc2 );

}

}

The inverted file algorithm can be effectively incorporated in the clustering algorithms described in this chapter when the calculation of the similarity matrix, or a single row of it, is required for a document collection.

It should be noted that the similarity matrix can be the basis for identifying a nearest neighbor (NN), that is, finding the closest vector to a given vector from a set of N multidimensional vectors. The identification of an NN arises in many clustering algorithms, and for large data sets makes a significant contribution to the computational requirement. Calculating and storing the similarity matrix, or recalculating it when needed, provides a brute force approach to nearest neighbor identification. Therefore, if an efficient NN-finding algorithm can be incorporated into the clustering algorithm, considerable savings of processing time may be achieved. However, although there are a number of techniques available for introducing efficiency into the NN-finding process (Bentley et al. 1980; Murtagh 1985), these techniques are generally inappropriate for data sets with the high dimensionality typical of information retrieval applications. Use of the inverted file algorithm to calculate the similarity matrix or a row of it seems to be the best optimization technique available in these circumstances.

16.3 CLUSTERING METHOS

16.3.1 Methods and Associated Algorithms

There are a very large number of ways of sorting N objects into M groups, a problem compounded by the fact that M is usually unknown. Most of the possible arrangements are of no interest; it is the role of a clustering method to identify a set of groups or cluster that reflects some underlying structure in the data. Moreover, there are many clustering methods available, which have differing theoretical or empirical bases and therefore produce different cluster structures. For a given clustering method, there may be a choice of clustering algorithm or means to implement the method. The choice of clustering method will determine the outcome, the choice of algorithm will determine the efficiency with which it is achieved. In this section, an overview of the clustering methods most used in information retrieval will be provided. The associated algorithms that are best suited to the processing of the large data sets found in information retrieval applications are discussed in sections 16.4 and 16.5.

16.3.2 Computational and Storage Requirements

In cases where the data set to be processed is very large, the resources required for cluster analysis may be considerable. A major component of the computation required is the calculation of the document-document or document-cluster similarity. The time requirement will be minimally O(NM), where M is the number of clusters, for the simpler reallocation methods; where the similarity matrix must be constructed, the proportionality is at least N2. Most of the preferred clustering methods have time requirements of at least O(N2). The storage requirement will be O(N) if the data set is stored, or O(N2) if the similarity matrix is stored. For large N this may be unacceptable, and disk accesses may make processing time too large if the similarity matrix is stored on disk. An alternative is to recalculate the similarity matrix from the stored data whenever it is needed to identify the current most similar pair, but this increases the time requirement by a factor of N2.

Because of the heavy demands of processing and storage requirements, much of the early work on cluster analysis for information retrieval was limited to small data sets, often only a few hundred items. However, improvements in processing and storage capacity and the introduction of efficient algorithms for implementing some clustering methods and finding nearest neighbors have made it feasible to cluster increasingly large data sets. Salton and Bergmark (1981) have pointed out that there is a high degree of parallelism in the calculation of a set of similarity values, and parallel hardware also offers the potential for increased processing efficiency (Willett and Rasmussen 1990).

16.3.3 Survey of Clustering Methods

Clustering methods are usually categorized according to the type of cluster structure they produce. The simple nonhierarchical methods divide the data set of N objects into M clusters; where no overlap is allowed, these are known as partitioning methods. Each item has membership in the cluster with which it is most similar, and the cluster may be represented by a centroid or cluster representative that is indicative of the characteristics of the items it contains. The more complex hierarchical methods produce a nested data set in which pairs of items or clusters are successively linked until every item in the data set is connected. The hierarchical methods can be either agglomerative, with N - 1 pairwise joins beginning from an unclustered data set, or divisive, beginning with all objects in a single cluster and progressing through N - 1 divisions of some cluster into a smaller cluster. The divisive methods are less commonly used and few algorithms are available; only agglomerative methods will be discussed in this chapter.

Nonhierarchical methods

The nonhierarchical methods are heuristic in nature, since a priori decisions about the number of clusters, cluster size, criterion for cluster membership, and form of cluster representation are required. Since the large number of possible divisions of N items into M clusters make an optimal solution impossible, the nonhierarchical methods attempt to find an approximation, usually by partitioning the data set in some way and then reallocating items until some criterion is optimized. The computational requirement O(NM) is much lower than for the hierarchical methods if M << N, so that large data sets can be partitioned. The nonhierarchical methods were used for most of the early work in document clustering when computational resources were limited; see for example work on the SMART project, described by Salton (1971).

Hierarchical methods

Most of the early published work on cluster analysis employed hierarchical methods (Blashfield and Aldenderfer 1978), though this was not so in the IR field. With improvements in computer resources, the easy availability of software packages for cluster analysis, and improved algorithms, the last decade of work on clustering in IR retrieval has concentrated on the hierarchical agglomerative clustering methods (HACM, Willett [1988]).

The cluster structure resulting from a hierarchical agglomerative clustering method is often displayed as a dendrogram like that shown in Figure 16.2. The order of pairwise coupling of the objects in the data set is shown, and the value of the similarity function (level) at which each fusion occurred. The dendrogram is a useful representation when considering retrieval from a clustered set of documents, since it indicates the paths that the retrieval process may follow.

Figure 16.2: Dendrogram of a hierarchical classification

The most commonly used hierarchical agglomerative clustering methods and their characteristics are:

single link: The single link method joins, at each step, the most similar pair of objects that are not yet in the same cluster. It has some attractive theoretical properties (Jardine and Sibson 1971) and can be implemented relatively efficiently, so it has been widely used. However, it has a tendency toward formation of long straggly clusters, or chaining, which makes it suitable for delineating ellipsoidal clusters but unsuitable for isolating spherical or poorly separated clusters.

complete link: The complete link method uses the least similar pair between each of two clusters to determine the intercluster similarity; it is called complete link because all entities in a cluster are linked to one another within some minimum similarity. Small, tightly bound clusters are characteristic of this method.

group average link: As the name implies, the group average link method uses the average values of the pairwise links within a cluster to determine similarity. All objects contribute to intercluster similarity, resulting in a structure intermediate between the loosely bound single link clusters and tightly bound complete link clusters. The group average method has ranked well in evaluative studies of clustering methods (Lorr 1983).

Ward's method: Ward's method is also known as the minimum variance method because it joins at each stage the cluster pair whose merger minimizes the increase in the total within-group error sum of squares, based on the Euclidean distance between centroids. It tends to produce homogeneous clusters and a symmetric hierarchy, and its definition of a cluster center of gravity provides a useful way of representing a cluster. Tests have shown it to be good at recovering cluster structure, though it is sensitive to outliers and poor at recovering elongated clusters (Lorr 1983).

Two other HACM are sometimes used, the centroid and median methods. In the centroid method, each cluster as it is formed is represented by the coordinates of a group centroid, and at each stage in the clustering the pair of clusters with the most similar mean centroid is merged. The median method is similar but the centroids of the two merging clusters are not weighted proportionally to the size of the clusters. A disadvantage of these two methods is that a newly formed cluster may be more like some point than were its constituent points, resulting in reversals or inversions in the cluster hierarchy.

16.4 ALGORITHMS FOR NONHIERARCHICAL METHODS

16.4.1 Single Pass Methods

The single pass method is particularly simple since it requires that the data set be processed only once. The general algorithm is as follows:

1. Assign the first document D1 as the representative for C1.

2. For Di, calculate the similarity S with the representative for each existing cluster.

3. If Smax is greater than a threshold value ST, add the item to the corresponding cluster and recalculate the cluster representative; otherwise, use Di to initiate a new cluster.

4. If an item Di remains to be clustered, return to step 2.

Though the single pass method has the advantage of simplicity, it is often criticized for its tendency to produce large clusters early in the clustering pass, and because the clusters formed are not independent of the order in which the data set is processed. It is sometimes used to form the groups that are used to initiate reallocation clustering. An example of a single pass algorithm developed for document clustering is the cover coefficient algorithm (Can and Ozkarahan 1984). In this algorithm, a set of documents is selected as cluster seeds, and then each document is assigned to the cluster seed that maximally covers it. For Di, the cover coefficient is a measure that incorporates the extent to which it is covered by Dj and the uniqueness of Di, that is, the extent to which it is covered by itself.

16.4.2 Reallocation Methods

The reallocation methods operate by selecting some initial partition of the data set and then moving items from cluster to cluster to obtain an improved partition. Anderberg (1973) discusses some of the criteria that have been suggested to establish an initial partition and to monitor the improvement achieved by reallocation. A general algorithm is:

1. Select M cluster representatives or centroids.

2. For i = 1 to N, assign Di to the most similar centroid.

3. For j = 1 to M, recalculate the cluster centroid Cj.

4. Repeat steps 2 and 3 until there is little or no change in cluster membership during a pass through the file.

The single pass and reallocation methods were used in early work in cluster analysis in IR, such as the clustering experiments carried out in the SMART project (Salton 1971). Their time and storage requirements are much lower than those of the HACM and much larger data sets could be processed. With improved processing capability and more efficient hierarchical algorithms, the HACMs are now usually preferred in practice, and the nonhierarchical methods will not be considered further in this chapter.

16.5 ALGORITHMS FOR HIERARCHICAL METHODS

16.5.1 General Algorithm for the HACM

All of the hierarchical agglomerative clustering methods can be described by a general algorithm:

1. Identify the two closest points and combine them in a cluster.

2. Identify and combine the next two closest points (treating existing clusters as points).

3. If more than one cluster remains, return to step 1.

Individual HACM differ in the way in which the most similar pair is defined, and in the means used to represent a cluster. Lance and Williams (1966) proposed a general combinatorial formula, the Lance-Williams dissimilarity update formula, for calculating dissimilarities between new clusters and existing points, based on the dissimilarities prior to formation of the new cluster. If objects Ci and Cj have just been merged to form cluster Cij, the dissimilarity d between the new cluster and any existing cluster Ck is given by:

dCi,jck = [image] idcick + ajdcjck + [image] dcicj + [image] [image] dcick - dcjck

This formula can be modified to accomodate a variety of HACM by choice of the values of [image], [image], and [image]. The hierarchical clustering methods previously discussed are presented in Table 16.1 in the context of their Lance-Williams parameters and cluster centers.

Table 16.1: Characteristics of HACM

Notes: mi is the number of items in Ci; the dissimilarity measure used for Ward's method must be the increase in variance (section 16.5.5).

There are three approaches to implementation of the general HACM (Anderberg 1973), each of which has implications for the time and storage requirements for processing. In the stored matrix approach, an N [image] N matrix containing all pairwise dissimilarity values is stored, and the Lance-Williams update formula makes it possible to recalculate the dissimilarity between cluster centers using only the stored values. The time requirement is O (N2), rising to O (N3) if a simple serial scan of the similarity matrix is used; the storage requirement is O (N2). A stored data approach has only an O (N) storage requirement but the need to recalculate the pairwise dissimilarity values for each fusion leads to an O (N3) time requirement. In the sorted matrix approach, the dissimilarity matrix is calculated (O (N2)) and then sorted (O (N2 log N2)) prior to the construction of the hierarchy (O (N2)). The data set need not be stored and the similarity matrix is processed serially, which minimizes disk accesses. However, this approach is suitable only for the single link and complete link methods, which do not require the recalculation of similarities during clustering.

In addition to the general algorithm, there are also algorithms specific to individual HACM. For some methods, algorithms have been developed that are the optimal O (N2) in time and O (N) in space (Murtagh 1984). These include several algorithms for the single link method, Defay's algorithm for complete link, and a nearest-neighbor algorithm for Ward's method, all of which are discussed below.

16.5.2 Single Link Method

The single link method merges at each stage the closest previously unlinked pair of points in the data set. Since the distance between two clusters is defined as the distance between the closest pair of points each of which is in one of the two clusters, no cluster centroid or representative is required, and there is no need to recalculate the similarity matrix during processing. This makes the method attractive both from a computational and a storage perspective, and it also has desirable mathematical properties (Jardine and Sibson 1971), so that it is one of the most widely used of the HACM.

A number of algorithms for the single link method have been reviewed by Rohlf (1982), including related minimal spanning tree algorithms. The computational requirements range from O (NlogN) to O (N5). Many of these algorithms are not suitable for information retrieval applications where the data sets have large N and high dimensionality. The single link algorithms discussed below are those that have been found most useful for information retrieval.

Van Rijsbergen algorithm

Van Rijsbergen (1971) developed an algorithm to generate the single link hierarchy that allowed the similarity values to be presented in any order and therefore did not require the storage of the similarity matrix. It is O (N2) in time and O (N) in storage requirements. It generates the hierarchy in the form of a data structure that both facilitates searching and is easily updated, and was the first to be applied to a relatively large collection of 11,613 documents (Croft 1977). However, most later work with large collections has used either the SLINK or Prim-Dijkstra algorithm, which are quite simple to implement.

SLINK algorithm

The SLINK algorithm (Sibson 1973) is optimally efficient, O (N2) for computation and O (N) for time, and therefore suitable for large data sets. It is simply a sequence of operations by which a representation of the single link hierarchy can be recursively updated; the dendrogram is built by inserting one point at a time into the representation.

The hierarchy is generated in a form known as the pointer representation, which consists of two functions [image] and [image] for a data set numbered 1..N, with the following conditions:

(N) = N

(i) > i

(N) =

( [image] (i)) > [image] (i) for i < N

In simple terms, [image] (i) is the lowest level (distance or dissimilarity) at which i is no longer the last (i.e., the highest numbered) object in its cluster, and [image] (i) is the last object in the cluster it joins at this level; a mathematical definition for these parameters is provided by Sibson (1973).

Fortran code for SLINK is provided in the original paper (Sibson 1973). In the pseudocode below, three arrays of dimension N are used: pi (to hold the pointer representation), lambda (to hold the distance value associated with each pointer), and distance (to process the current row of the distance matrix); next indicates the current pointer for a point being examined.

/* initialize pi and lambda for a single point representation */

pi [0] = 0;

lambda[0] = MAXINT;

/*iteratively add the remaining N-1 points to the hierarchy */

for (i = 1; i < N; i++)

{

pi [i] = i;

lambda[i] = MAXINT;

/* calculate and store a row of the distance matrix for i */

for (j = 0; j < i-1; j++) distance[j] =calc_distance(i,j);

for (j = 0; j < i-1; j++)

{

next = pi[j];

if (lambda[j] < distance[j])

distance[next] = min(distance[next],distance[j]);

else

{

distance[next] = min(lambda[j],distance[next]);

pi[j] = i;

lambda[j] = distance[j];

}

}

/* relabel clusters if necessary */

for (j = 0; j <i-1; j++)

{

next = pi  [j];

if (lambda[next] < lambda [j])

pi[j] = i;

}

}

For output in the form of a dendrogram, the pointer representation can be converted into the packed representation. This can be accomplished in O (N2) time (with a small coefficient for N2) and O (N) space. A FORTRAN subroutine to effect the transformation is provided in Sibson's original paper.

Minimal spanning tree algorithms

A minimal spanning tree (MST) is a tree linking N objects with N - 1 connections so that there are no loops and the sum of the N - 1 dissimilarities is minimized. It can be shown that all the information required to generate a single link hierarchy for a set of points is contained in their MST (Gower and Ross 1969). Once an MST has been constructed, the corresponding single link hierarchy can be generated in O (N2) operations; or the data structures for the MST can be modified so that the hierarchy can be built simultaneously (Rohlf 1982).

Two fundamental construction principles for MSTs are:

1. Any isolated point can be connected to a nearest neighbor.

2. Any isolated fragment (subset of an MST) can be connected to a nearest neighbor by a shortest available link.

The Prim-Dijkstra algorithm (Dijkstra 1976) consists of a single application of principle 1, followed by N - 1 iterations of principle 2, so that the MST is grown by enlarging a single fragment:

1. Place an arbitrary point in the MST and connect its nearest neighbor to it.

2. Find the point not in the MST closest to any point in the MST and add it to the fragment.

3. If a point remains that is not in the fragment, return to step 2.

The algorithm requires O (N2) time if, for each point not in the fragment, the identity and distance to its nearest neighbor in the fragment is stored. As each new point is added to the fragment, the distance from that point to each point not in the fragment is calculated, and the NN information is updated if necessary. Since the dissimilarity matrix need not be stored, the storage requirement is O (N).

A FORTRAN version of the Prim-Dijkstra algorithm is provided by Whitney (1972). The algorithm here uses arrays npoint and ndistance to hold information on the nearest in-tree neighbor for each point, and notintree is a list of the nt unconnected points. Lastpoint is the latest point added to the tree.

/*  initialize lists */

for (i = 0; i < n; i++)

{

ndistance[i] = MAXINT;

notintree[i] = i;

}

/* arbitrarily place the Nth point in the MST */

lastpoint = n;

nt = n-1;

/*  grow the tree an object at a time */

for ( i = 0; i < n-1;i++)

{

/*consider the lastpoint in the tree for the NN list */

for (j = 0; j < nt; j++)

{

D = calculate_distance(lastpoint, notintree[j]);

if  (D < ndistance[j]

{

npoint[j] =lastpoint;

ndistance[j] = D;

}

}

/* find the unconnected point closest to a point in the     */

/* tree                                                     */

nj = index_of_min(ndistance);

/* add this point to the MST; store this point and their    */

/* clustering level                                         */

lastpoint = notintree[nj];

store_in_MST ( lastpoint, npoint[nj], ndistance[nj]);

/* remove lastpoint from notintree list;  */

/* close up npoint and ndistance lists    */

notintree[nj] = nt;

npoint[nj] = npoint[nt];

ndistance[nj] =ndistance[nt];

nt = nt - 1;

}

}

16.5.3 Complete Link Method

The small, tightly bound clusters typical of the complete link method have performed well in comparative studies of document retrieval (Voorhees 1986a). Unfortunately, it is difficult to apply to large data sets since there does not seem to be an algorithm more effective than the stored data or stored matrix approach to the general HACM algorithm.

Defays' CLINK algorithm

The best-known algorithm for implementing the complete link method is the CLINK algorithm developed by Defays (1977). It is presented in a form analogous to the SLINK algorithm, uses the same three arrays (pi, lambda, and distance), and like SLINK, produces output in the form of the pointer representation. Defays presents a CLINK subroutine which allows his algorithm to be incorporated into Sibson's original FORTRAN program for SLINK. CLINK is efficient, requiring O (N2) time, O (N) space, but it does not seem to generate an exact hierarchy and has given unsatisfactory results in some information retrieval experiments (El-Hamdouchi and Willett 1989).

Voorhees algorithm

The Voorhees algorithm (1986b) for the complete link method has been used to cluster relatively large document collections with better retrieval results than the CLINK algorithm (El-Hamdouchi and Willett 1989). It is a variation on the sorted matrix approach, and is based on the fact that if the similarities between all pairs of documents are processed in descending order, two clusters of size mi and mj can be merged as soon as the mi   [image]  mith similarity of documents in the respective clusters is reached. This requires a sorted list of document-document similarities, and a means of counting the number of similarities seen between any two active clusters. The large number of zero-valued similarities in a typical document collection make it more efficient than its worst case O (N3) time, O (N2) storage requirement would suggest; however it is still very demanding of resources, and El-Hamdouchi and Willett found it impractical to apply it to the largest of the collections they studied.

16.5.4 Group Average Link Method

Because the similarity between two clusters is determined by the average value of all the pairwise links between points for which each is in one of the two clusters, no general O (N2) time, O (N) space algorithm is known. The general HACM algorithm can be used, but with O (N3) time for the stored data approach and O (N2) storage for the stored matrix approach, implementation may be impractical for a large collection. However, a more efficient special case algorithm is available.

Voorhees algorithm

Voorhees (1986b) has pointed out that the group average link hierarchy can be constructed in O (N2) time, O (N) space if the similarity between documents chosen is the inner product of two vectors using appropriately weighted vectors. In this case, the similarity between a cluster centroid and any document is equal to the mean similarity between the document and all the documents in the cluster. Since the centroid of the cluster is the mean of all the document vectors, the centroids can be used to compute the similarities between the clusters while requiring only O(N) space. Voorhees was able to cluster a document collection of 12,684 items using this algorithm, for which she provides pseudocode.

Using Voorhees' weighting scheme and intercentroid similarity, El-Hamdouchi (1987) was able to implement the group average link method using the reciprocal nearest neighbor algorithm described below for Ward's method.

16.5.5 Ward's Method

Ward's method (Ward 1963; Ward and Hook 1963) follows the general algorithm for the HACM, where the object/cluster pair joined at each stage is the one whose merger minimizes the increase in the total within-group squared deviation about the means, or variance. When two points Di and Dj are clustered, the increase in variance Iij is given by:

where mi is the number of objects in Di and d2ij is the squared Euclidean distance, given by:

where Di is represented by a vector (xi1, xi2, . . . , xiL) in L-dimensional space. The cluster center for a pair of points Di and Dj is given by:

Reciprocal nearest neighbor algorithm

The mathematical properties of Ward's method make it a suitable candidate for a reciprocal nearest neighbor (RNN) algorithm (Murtaugh 1983, 1985). For any point or cluster, there exists a chain of nearest neighbors (NNs) so that

NN(i) = j; NN(j) = k; ...; NN(p) = q; NN(q) = p

The chain must end in some pair of objects that are RNNs, since the interobject distances are monotonically decreasing along the chain (ties must be arbitrarily resolved).

An efficient clustering algorithm can be based on the process of following a chain of nearest neighbors:

1.     Select an arbitrary point.

2.     Follow the NN chain from this point till an RNN pair is found.

3.     Merge these two points and replace them with a single point.

4.     If there is a point in the NN chain preceding the merged points, return to step 2; otherwise return to step 1. Stop when only one point remains.

This algorithm requires O(N2) computation but only O(N) storage. It carries out agglomerations in restricted spatial regions, rather than in strict order of increasing dissimilarity, but still results (for Ward's method) in a hierarchy that is unique and exact. This is designated the Single Cluster Algorithm since it carries out one agglomeration per iteration; a Multiple Cluster Algorithm, suitable for parallel processing, has also been proposed (Murtagh 1985).

16.6 EVALUATION AND VALIDATION

As Dubes and Jain (1980, p. 179) point out:

The thoughtful user of a clustering method or algorithm must answer two questions: (i) Which clustering method is appropriate for a particular data set? (ii) How does one determine whether the results of a clustering method truly characterize the data?

These are important questions because any clustering method will produce a set of clusters, and results are sought which reflect some "natural" grouping rather than one that arises as an artifact of the method. The answer to the first question can be found in evaluative studies of clustering methods, and to the second question, in validation techniques for clustering solutions.

16.6.1 Evaluation

Many evaluative studies have attempted to determine the "best" clustering method (Lorr 1983) by applying a range of clustering methods to test data sets and comparing the quality of the results, for example by using artificially created data structures, or comparing the cluster results to a classification established by experts in the field. Even under laboratory conditions it is difficult to evaluate clustering methods, since each method has different properties and strengths. The results of these studies do not suggest a single best method, though Ward's method, and in more recent studies, the group average method, have performed well. It is usually advisable to apply more than one clustering method and use some validation method to check the reliability of the resulting cluster structure.

For retrieval purposes, the "best" method for clustering a document collection is that which provides the most effective retrieval in response to a query. Several evaluative studies have taken this approach, using standard test collections of documents and queries. Most of the early work used approximate clustering methods, or the least demanding of the HACM, the single link method, a restriction imposed by limited processing resources. However, two recent studies are noteworthy for their use of relatively large document collections for the evaluation of a variety of HACM (El-Hamdouchi and Willett 1989; Voorhees 1986a). Voorhees compared single link, complete link, and group average methods, using document collections of up to 12,684 items, while El-Hamdouchi and Willett compared these three methods plus Ward's method on document collections of up to 27,361 items. Voorhees found complete link most effective for larger collections, with complete and group average link comparable for smaller collections; single link hierarchies provided the worst retrieval performance. El-Hamdouchi and Willett found group average most suitable for document clustering. Complete link was not as effective as in the Voorhees study, though this may be attributed to use of Defays' CLINK algorithm. As noted in section 16.8.1, there are several ways in which retrieval from a clustered document collection can be performed, making comparisons difficult when using retrieval as an evaluative tool for clustering methods.

16.6.2 Validation

Cluster validity procedures are used to verify whether the data structure produced by the clustering method can be used to provide statistical evidence of the phenomenon under study. Dubes and Jain (1980) survey the approaches that have been used, categorizing them on their ability to answer four questions: is the data matrix random? how well does a hierarchy fit a proximity matrix? is a partition valid? and which individual clusters appearing in a hierarchy are valid? Willett (1988) has reviewed the application of validation methods to clustering of document collections, primarily the application of the random graph hypothesis and the use of distortion measures.

An approach that is carried out prior to clustering is also potentially useful. Tests for clustering tendency attempt to determine whether worthwhile retrieval performance would be achieved by clustering a data set, before investing the computational resources which clustering the data set would entail. El-Hamdouchi and Willett (1987) describe three such tests. The overlap test is applied to a set of documents for which query-relevance judgments are available. All the relevant-relevant (RR) and relevant-nonrelevant (RNR) interdocument similarities are calculated for a given query, and the overlap (the fraction of the RR and RNR distributions that is common to both) is calculated. Collections with a low overlap value are expected to be better suited to clustering than those with high overlap values. Voorhees' nearest neighbor test considers, for each relevant document for a query, how many of its n nearest neighbors are also relevant; by averaging over all relevant documents for all queries in a test colleciton, a single indicator for a collection can be obtained. The density test is defined as the total number of postings in the document collection divided by the product of the number of documents and the number of terms that have been used for the indexing of those documents. It is particularly useful because it does not require any predetermined query-relevance data or any calculation of interdocument similarities. Of the three tests, the density test provided the best indication of actual retrieval performance from a clustered data set.

The goal of a hierarchical clustering process may be to partition the data set into some unknown number of clusters M (which may be visualized as drawing a horizontal line across the dendrogram at some clustering level). This requires the application of a stopping rule, a statistical test to predict the clustering level that will determine M. Milligan and Cooper (1985) evaluated and ranked 30 such rules, one or more of which is usually present in a software package for cluster analysis (though not necessarily those ranked highest by Milligan and Cooper).

16.7 UPDATING THE CLUSTER STRUCTURE

In many information retrieval environments, the collection is dynamic, with new items being added on a regular basis, and, less frequently, old items withdrawn. Since clustering a large data set is resource-intensive, some mechanism for updating the cluster structure without the need to recluster the entire collection is desirable. Relatively little work has been done on methods for cluster maintenance (Can and Ozkarahan 1989), particularly for the hierarchical methods. In certain cases, update of the cluster structure is implicit in the clustering algorithm. This is true of both the van Rijsbergen and SLINK algorithms for the single link method, and the CLINK algorithm for the complete link method, all of which operate by iteratively inserting a document into an existing hierarchy.

Where the application uses a partitioned data set (from a nonhierarchical or hierarchical method), new items may simply be added to the most similar partition until the cluster structure becomes distorted and it is necessary to regenerate it. For a few methods, cluster update has been specifically incorporated. Crouch's reallocation algorithm includes a mechanism for cluster maintenance (Crouch 1975). Can and Ozkarahan (1989) review the approaches that have been taken for cluster maintenance and propose a strategy for dynamic cluster maintenance based on their cover coefficient concept.

16.8 DOCUMENT RETRIEVAL FROM A CLUSTERED DATA SET

Document clustering has been studied because of its potential for improving the efficiency of retrieval, for improving the effectiveness of retrieval, and because it provides an alternative to Boolean or best match retrieval. Initially the emphasis was on efficiency: document collections were partitioned, using nonhierarchical methods, and queries were matched against cluster centroids, which reduced the number of query-document comparisons that were necessary in a serial search. Studies of retrieval from partitioned document collections showed that though retrieval efficiency was achieved, there was a decrease in retrieval effectiveness (Salton 1971). Subsequent study has concentrated on the effectiveness of retrieval from hierarchically clustered document collections, based on the cluster hypothesis, which states that associations between documents convey information about the relevance of documents to requests (van Rijsbergen 1979).

16.8.1 Approaches to Retrieval

There are several ways in which a query can be matched against the documents in a hierarchy (Willett 1988). A top-down search involves entering the tree at the root and matching the query against the cluster at each node, moving down the tree following the path of greater similarity. The search is terminated according to some criterion, for instance when the cluster size drops below the number of documents desired, or when the query-cluster similarity begins to decrease. A single cluster is retrieved when the search is terminated. Since it is difficult to adequately represent the clusters in the very large top-level clusters, a useful modification is to eliminate the top-level clusters by applying a threshold clustering level to the hierarchy to obtain a partition, and using the best of these mid-level clusters as the starting point for the top-down search. The top-down strategy has been shown to work well with the complete link method (Voorhees 1986a).

A bottom-up search begins with some document or cluster at the base of the tree and moves up until the retrieval criterion is satisfied; the beginning document may be an item known to be relevant prior to the search, or it can be obtained by a best match search of documents or lowest-level clusters. Comparative studies suggest that the bottom-up search gives the best results (apart from the complete link method), particularly when the search is limited to the bottom-level clusters (Willett 1988). Output may be based on retrieval of a single cluster, or the top-ranking clusters may be retrieved to produce a predetermined number of either documents or clusters; in the latter case, the documents retrieved may themselves be ranked against the query.

A simple retrieval mechanism is based on nearest neighbor clusters, that is, retrieving a document and that document most similar to it. Griffiths et al. (1984) determined that for a variety of test collections, search performance comparable to or better than that obtainable from nonclustered collections could be obtained using this method.

16.8.2 Cluster Representatives

A centroid or cluster representative is a record that is used to represent the characteristics of the documents in a cluster. It is required in retrieval so that the degree of similarity between a query and a cluster can be determined; it is also needed in the nonhierarchical methods where document-cluster similarity must be determined in order to add documents to the most similar cluster. Ranks or frequencies have been used to weight terms in the representative; usually a threshold is applied to eliminate less significant terms and shorten the cluster representative. A binary representative may also be used, for example including a term if it occurs in more than log2m documents in the cluster (Jardine and van Rijsbergen 1971).

16.9 CONCLUSION

Cluster analysis is an effective technique for structuring a data set, and a wide choice of methods and algorithms exists. It is important to consider the questions raised in section 16.1.2 regarding the potential application, selection of methods and algorithm and the parameters associated with it, and evaluation and validation of the results. Much of the work to date in cluster analysis has been limited by the considerable resources required by clustering, and only recently have results from relatively large-scale clustering become available.

REFERENCES

ALDENDERFER, M. S., and R. K. BLASHFIELD. 1984. Cluster Analysis. Beverly Hills: Sage.

ANDERBERG, M. R., 1973. Cluster Analysis for Applications. New York: Academic.

BENTLEY, J. L., B. W. WEIDE, and A. C. YAO. 1980. "Optimal Expected-Time Algorithms for Closest Point Problems." ACM Transactions on Mathematical Software, 6, 563-80.

BLASHFIELD, R. K., and M. S. ALDENDERFER. 1978. "The Literature on Cluster Analysis." Multivariate Behavioral Research, 13, 271-95.

CAN, F., and E. A. OZKARAHAN. 1984. "Two Partitioning Type Clustering Algorithms." J. American Society for Information Science, 35, 268-76.

CAN, F., and E. A. OZKARAHAN. 1989. "Dynamic Cluster Maintenance." Information Processing & Management, 25, 275-91.

CROFT, W. B. 1977. "Clustering Large Files of Documents Using the Single-Link Method." J. American Society for Information Science, 28, 341-44.

CROUCH, C. J. 1988. "A Cluster-Based Approach to Thesaurus Construction" in 11th International Conference on Research and Development in Information Retrieval, New York: ACM, 309-20.

CROUCH, D. B. 1975. "A File Organization and Maintenance Procedure for Dynamic Document Collections." Information Processing & Management, 11, 11-21.

DEFAYS, D. 1977. "AEfficient n Algorithm for a Complete Link Method." Computer Journal, 20, 364-66.

DIJKSTRA, E. W. 1976. "The Problem of the Shortest Subspanning Tree." A Discipline of Programming. Englewood Cliffs, N.J.: Prentice Hall, 154-60.

DUBES, R., and A. K. JAIN. 1980. "Clustering Methodologies in Exploratory Data Analysis." Advances in Computers, 19, 113-227.

EL-HAMDOUCHI, A. 1987. The Use of Inter-Document Relationships in Information Retrieval. Ph.D. thesis, University of Sherfield.

EL-HAMDOUCHI, A., and P. WILLETT. 1987. "Techniques for the Measurement of Clustering Tendency in Document Retrieval Systems." J. Information Science, 13, 361-65.

EL-HAMDOUCHI, A., and P. WILLETT. 1989. "Comparison of hierarchic agglomerative clustering methods for document retrieval." Computer Journal, 32, 220-227.

EVERITT, B. 1980. Cluster Analysis, 2nd ed. New York: Halsted.

GORDON, A. D. 1987. "A Review of Hierarchical Classification." J. Royal Statistical Society, Series A, 150(2), 119-37.

GOWER, J. C., and G. J. S. ROSS. 1969. "Minimum Spanning Trees and Single Linkage Cluster Analysis." Applied Statistics, 18, 54-64.

GRIFFITHS, A., L . A. ROBINSON, and P. WILLETT. 1984. "Hierarchic Agglomerative Clustering Methods for Automatic Document Classification." J. Documentation, 40, 175-205.

HARTIGAN, J. A. 1975. Clustering Algorithms. New York: Wiley.

HUDSON, H. C., and ASSOCIATES. 1983. Classifying Social Data: New Applications of Analytical Methods for Social Science Research. San Francisco: Jossey-Bass.

JAIN, A. K., and R. C. DUBES. 1988. Algorithms for Clustering Data. Englewood Cliffs, N.J.: Prentice Hall.

JARDINE, N., and R. SIBSON. 1971. Mathematical Taxonomy. London: Wiley.

JARDINE, N., and C. J. VAN RIJSBERGEN. 1971. "The Use of Hierarchic Clustering in Information Retrieval." Information Storage and Retrieval, 7, 217-40.

KAUFMAN, L. 1990. Finding Groups in Data: An Introduction to Cluster Analysis. New York: Wiley.

LANCE, G. N., and W. T. WILLIAMS. 1966. "A General Theory of Classificatory Sorting Strategies. 1 . Hierarchical systems." Computer Journal, 9, 373-80.

LEE, R. C. T. 1981. "Clustering Analysis and Its Applications." Advances in Information Systems Science, 8, 169-292.

LORR, M. 1983. Cluster Analysis for Social Scientists: Techniques for Analyzing and Simplifying Complex Blocks of Data. San Francisco: Jossey-Bass.

MILLIGAN, G. W., and M. C. COOPER. 1985. "An Examination of Procedures for Determining the Number of Clusters in a Data Set." Psychometrika, 50, 159-79.

MURTAGH, F. 1983. "A Survey of Recent Advances in Hierarchical Clustering Algorithms." Computer Journal, 26, 354-59.

MURTAGH, F. 1984. "Complexities of Hierarchic Clustering Algorithms: State of the Art." Computational Statistics Quarterly, 1, 101-13.

MURTAGH, F. 1985. Multidimensional Clustering Algorithms. Vienna: Physica-Verlag (COMP-STAT Lectures 4).

PERRY, S. A., and P. WILLETT. 1983. "A Review of the Use of Inverted Files for Best Match Searching in Information Retrieval Systems." J. Information Science, 6, 59-66.

ROHLF, F. J. 1982. "Single-Link Clustering Algorithms," Classification, Pattern Recognition, and Reduction of Dimensionality, eds. P. R. Krishnaiah and J. N. Kanal, pp. 267-84. Amsterdam: North-Holland (Handbook of Statistics, Vol. 2).

ROMESBURG, H. C. 1984. Cluster Analysis for Researchers. Belmont, Calif.: Lifetime Learning.

SALTON, G., ed. 1971. The SMART Retrieval System. Englewood Cliffs, N.J.: Prentice Hall.

SALTON, G. 1989. Automatic Text Processing. Reading, Mass.: Addison-Wesley.

SALTON, G., and D. BERGMARK. 1981. "Parallel Computations in Information Retrieval." Lecture Notes in Computer Science, 111, 328-42.

SALTON, G., and C. BUCKLEY. 1988. "Term-Weighting Approaches in Automatic Text Retrieval." Information Processing & Management, 24, 513-23.

SIBSON, R. 1973. "SLINK: an Optimally Efficient Algorithm for the Single-Link Cluster Method." Computer Journal, 16, 30-34.

SMALL, H., and E. SWEENEY. 1985. "Clustering the Science Citation Index Using Co-citations. I. A Comparison of Methods." Scientometrics, 7, 391-409.

SNEATH, P. H. A., and R. R. Sokal. 1973. Numerical Taxonomy; the Principles and Practice of Numerical Classification. San Francisco: W. H. Freeman.

SPATH, H. 1985. Cluster Dissection and Analysis. Chichester: Ellis Horwood.

VAN RIJSBERGEN, C. J. 1971. "An Algorithm for Information Structuring and Retrieval." Computer Journal, 14, 407-12.

VAN RIJSBERGEN, C. J. 1979. Information Retrieval. London: Butterworths.

VOORHEES, E. M. 1986a. The Effectiveness and Efficiency of Agglomerative Hierarchic Clustering in Document Retrieval. Ph.D. thesis, Cornell University.

VOORHEES, E. M. 1986b. "Implementing Agglomerative Hierarchic Clustering Algorithms for Use in Document Retrieval. Information Processing & Management, 22, 465-76.

WARD, J. H., JR. 1963. "Hierarchical Grouping to Optimize an Objective Function. J. American Statistical Association, 58(301), 235-44.

WARD, J. H., JR. and M. E. HOOK. 1963. "Application of an Hierarchical Grouping Procedure to a Problem of Grouping Profiles." Educational and Psychological Measurement, 23, 69-81.

WHITNEY, V. K. M. 1972. "Algorithm 422: Minimal Spanning Tree." Communications of the ACM, 15, 273-74.

WILLETT, P. 1980. "Document Clustering Using an Inverted File Approach." J. Information Science, 2, 223-31.

WILLETT, P. 1983. "Similarity Coefficients and Weighting Functions for Automatic Document Classification: an Empirical Comparison." International Classification, 10, 138-42.

WILLETT, P. 1988. "Recent Trends in Hierarchic Document Clustering: A Critical Review." Information Processing & Management, 24(5), 577-97.

WILLETT, P., and E. M. RASMUSSEN. 1990. Parallel Database Processing. London: Pitman (Research Monographs in Parallel and Distributed Computing).

Go to Chapter 17     Back to Table of Contents


Information Retrieval: CHAPTER 17: SPECIAL-PURPOSE HARDWARE FOR INFORM

 Return to Table of Contents

CHAPTER 17: SPECIAL-PURPOSE HARDWARE FOR INFORMATION RETRIEVAL

Lee A. Hollaar

Department of Computer Science, University of Utah

Abstract

Many approaches for using special purpose hardware to enhance the performance of an information retrieval have been proposed or implemented. Rather than using conventional digital computers or parallel processors developed for general applications, these special purpose processors have been designed specifically for particular facets of information retrieval, such as the rapid searching of unstructured data or manipulating surrogates or indices for the information in the database. After discussing the reasons why a special purpose processor may be desirable, three different techniques for implementing a text scanner are presented--associative memories, finite state machines, and cellular arrays. The problems of using a conventional microprocessor are discussed, along with possible hardware enhancements to improve its performance. System performance considerations, especially when the use of an index or other surrogate shift the performance from text scanning to a "seek and search" mode, are covered. Finally, special purpose processors for handling surrogates and the use of optical processing are presented.

17.1 INTRODUCTION

The previous chapters have described techniques for implementing information retrieval systems on general purpose computing systems, including parallel machines designed to handle a variety of applications. However, there are times when an information retrieval system implemented using general purpose hardware may not be cost-effective or provide adequate performance. For example, to achieve satisfactory performance most information retrieval systems use some form of index. However, for a database consisting of messages that must be available as soon as they are received, there is no time available to construct, update, and maintain an index. Instead, special purpose searching hardware can provide the desired solution.

Special purpose hardware development for information retrieval has generally fallen into two areas--systems for rapidly searching unstructured data and systems for manipulating surrogates or indices for the information in the database. While most special purpose systems have been implemented using conventional digital logic, recently the use of optical processing has been suggested. The availability of fast microprocessors may also lead to special purpose systems that resemble a conventional workstation, but have additional logic to enhance their processing of information, more efficient connection and use of peripheral devices such as disk drives, or operating system software optimized for the particular application.

17.1.1 Search Patterns

The design of a special purpose processor is heavily influenced by the types of operations that will be performed. In the case of text searching, these are the patterns and operators that are allowed when specifying a query. The most basic operation is performing an exact match search for a specified string of characters, which may represent either a single word or phrase. Proper handling of phrases may be more complex than simply searching for the words of the phrase each separated by a blank. Unless the text in the database has been carefully edited so that it fits a standard form, the words of the desired phrase may not be separated only by a single blank, but could be separated by a newline sequence (if the phrase started on one line and ended on another), more than one blank or some other space character (like a horizontal tab), typographic markup (such as changing from roman to italic type), a footnote reference, or other complications. Most special purpose text searchers can only handle the most basic phrase structures.

In addition to exactly matching a string of characters, it is often desirable to specify a more complex match pattern. A regular expression can specify not only a specific character to match at a pattern location, but can indicate a set of characters (such as an upper- or lowercase A, any vowel, the start of a line, or a word delimiter). It also can indicate that a pattern element can optionally occur or can occur more than once, or indicate two or more alternative patterns that are acceptable in a location.

While regular expressions could be used to specify the search pattern, often an information retrieval system will allow only a specific set of operations based on the nature of searching text. The most common of these operations is the don't care or wildcard, which specifies that any character is acceptable in a particular location in the search pattern (single-character or fixed-length don't care, FLDC) or an arbitrary number of characters are acceptable at the location (variable-length don't care, VLDC). The VLDC can either come at the start of a word (an initial VLDC, matching a pattern with any prefix), end of a word (a terminal VLDC, matching a pattern with any suffix), at both the start or end of a word (matching a specified stem), embedded within a word (an embedded VLDC), or any combination of these. Some text searchers cannot handle VLDCs, or can handle only initial or terminal VLDCs.

One other special pattern may specify that a numeric string should occur at the pattern location, and its value should be within a given range. Another pattern may specify an error tolerance, where the pattern is matched if there are fewer than n characters that don't match. These options are not commonly available on hardware-based text searchers (Yu et al. 1986).

Search expression often contain two or more patterns combined with Boolean operators, the most common being and, or, and and not. Word proximity, an extended form of and, indicates that two search terms need to be located within a specified number of words in the database for a match to occur. Proximity can be either directed (X must be followed within n words by Y) or undirected (R and S must be within n words of each other, but in any order). Like phrases, word proximity is complicated by markup or other information embedded in the text.

A search expression also can specify that a term, phrase, or Boolean combination must occur in particular contexts for a match to occur. For example, a search may look for "hardware and text retrieval in abstract" or "patent and software within 2 sentences." Other operators may expand the number of terms in a query. A macro facility allows the user to include a predefined subexpression into a query, while a thesaurus can automatically form the or of all synonyms of a word. For example, "airplane" might expand to "airplane or plane or 747 or DC-10 . . . ."

17.1.2 Hardware Implementations

Many people have proposed implementations of information retrieval functions using custom hardware. Many of these consider only one aspect of information retrieval, without considering its effect on the overall retrieval system and its performance. Simple searchers, capable of exact matching against a small number of patterns (possibly containing don't cares) have been particularly popular as a VLSI design project. However, only a small number of these have been incorporated into complete information retrieval systems.

The algorithms implemented in hardware differ from the algorithms described in the previous chapters in two fundamental ways. They are much simpler but do many things in parallel or inherently in the logic. The reason for the simplicity is that each alternative in an algorithm often requires its own logic or increases the complexity of the other hardware. This not only complicates the design, but can result in slower performance because of longer data paths and increased loading of output drivers. It is also easier to design a simple function that is then replicated many times than a large single unit incorporating a variety of functions.

Many operations that would be cumbersome on a conventional processor are very simple to perform in custom hardware. As an example, the program to reverse the order of the bits of a word takes a number of steps of isolating, aligning, and replacing each bit of the word. In hardware, it simply takes the changing of the connections from straight-through to reversed. No active logic is required and the operation is done in zero time. If operations do not have to share hardware, they can be done simultaneously (for example, the shift-and-add being done in parallel with the decrementing of a counter during a multiplication). Because of this, most hardware solutions are not simply faster implementations of standard information retrieval algorithms.

17.2 TEXT SEARCHERS

The most commonly proposed special purpose hardware system for text information retrieval is a searcher. The rationale for needing a special search engine is that a conventional processor cannot search rapidly enough to give reasonable response time. For example, egrep on a 12 MIPS RISC processor (Sun SPARCstation-l) can search at about 250 KBytes per second. The searching of a 10 GByte database would take 40,000 seconds (over 11 hours). If a large database resides on a number of disk drives, it will be faster and more efficient not to bring the raw information from the disk into a central processor, but instead search it with a processor closer to the disk. Because each search processor is operating only on the data on its disk, the search time will remain the same as the database size increases and the additional disks (each with a searcher) necessary to hold the database increases the parallelism of the search.

However, because of the slow searching speeds for a central processor, most commercial information retrieval systems use some sort of inverted file index rather than actually searching the documents. The argument in the past against inverted files was that they substantially increase the storage requirements of a database. The index may take more disks than the database itself (Bird et al. 1978), but this may be less of a concern with lower-cost disk drives. This means that to be competitive, the cost of searching must be comparable to the cost of a disk drive.

However, there are a number of problems associated with inverted files or other document surrogates besides their storage requirements. In many cases, information is discarded to reduce the size of the surrogate. For inverted files, commonly occurring words are generally not included in the index. This means that a search for a phrase like "changing of the guard" really finds anything that has "changing," any two words, and "guard." The phrase "To be or not to be, that is the question" matches any document containing the word "question," since all other words in the phrase are discarded. A similar problem exists for superimposed code words, where the combining of the codes for two words makes it seem like another word is present when it is not.

These artifacts caused by the use of a surrogate tend to confuse a user because it is not clear why a document that doesn't match the query was retrieved. Time is spent trying to understand an irrelevant document, perhaps more time than would be spent reviewing a relevant one. But even if this were not the case, there are other problems associated with the use of a surrogate for locating documents. Time must be spent building and maintaining the surrogate, a nontrivial amount of time for very large databases. Also, a document is not retrievable until it has been entered into the surrogate data structure. This may not be acceptable for a message retrieval system, where incoming documents must be available as soon as they have arrived.

17.2.1 Searcher Structure

Most proposed text searchers divide the work into three activities: search control, term matching, and query resolution. Search control includes taking a request from the processor running the retrieval system, configuring the search hardware (typically by loading appropriate tables and registers), managing the reading of the data from the disks, and reporting back the results to the retrieval system. This search control is best performed by a conventional microprocessor (or, in the past, a minicomputer).

Searching for particular terms or patterns is performed by the special hardware, with the results passed to a query resolver that handles operations such as Booleans, contexts, and proximities. This considerably simplifies the search patterns. A good example of this is a query that specifies that five different terms must occur in the same paragraph, in any order. The regular expression for handling this is very complex, consisting of looking for the start of paragraph pattern, then all possible orderings of the five terms (120 sequences), then the end of the paragraph pattern. A better way is for the term matcher to look for the individual terms of the pattern. The query resolver then combines the term matcher results, which indicate that particular terms have been detected to determine if the pattern has been matched. Because this latter processing is required only when a term has been found, its realtime requirements are considerably less than that of comparing characters from the disk, and it can be handled by a conventional processor.

In effect, the term matcher converts a document into a list of entries that are a shorthand description of the document as it pertains to the query. The query resolution program can then process this greatly reduced information according to whatever scheme is desired. This could be a Boolean query language enhanced with contexts and word location proximity, weighted term document scoring, or any other technique.

Three different techniques have been proposed for implementing the term matcher: associative memories, finite state machines, and cellular arrays (Hollaar 1979). As their performance improves, standard microprocessors, perhaps augmented with simple hardware assists, can also be used for the term matcher.

If commodity disk drives are used to keep costs low, rather than special drives like those with parallel reads, the disk transfer and seek rates will determine the speed of the searcher. There are two possible options: match the term comparator to the nominal transfer rate of the disk (raw read bit rate), or match it to the effective transfer rate (average bit rate when seeks and rotational latency are considered). In either case, the same performance will result, since it is based on how many characters come from the disk in a unit of time. Matching to the effective transfer rate permits a lower comparator speed at the expense of buffer memory. How much lower depends on the seek characteristics for the typical or worst-case queries. Matching to the nominal transfer rate was used for many searcher implementations, because of the high cost of memory for buffers at the time they were proposed.

Associative Memories

In an associative memory, each memory word has a comparator. The bit pattern in the comparand register is sent to each of these comparators, which indicate whether it matches their memory word. The comparison can be done in one cycle regardless of the size of the memory, but the cost per bit is considerably higher than for conventional memory because of the logic required. (A dynamic RAM memory cell may need only a single transistor, while an associative memory cell has tens or hundreds of transistors, depending on whether features such as masking to selectively compare bits in a word are implemented.)

If the cost were ignored, an associative memory would seem ideal for finding the locations of a pattern within a database. It is not. First of all, the data must be broken down into fixed-length groups to match the width of the associative memory. This could be done on word delimiters, in which case the width of the associative memory would have to be the length of the longest word and much space would be wasted, or it would have to be every n characters, packing the associative memory. In the latter case, however, words start in arbitrary positions within the width of the associative memory and will not always line up with the pattern. This can be solved by cycling the pattern through the width of the associative memory, but this is complicated by words that start in one memory location and end in another. Handling of VLDCs, especially embedded ones, is also difficult.

A more cost-effective approach is to store the search terms in the associative memory and shift the data through the comparand register. Essentially, this uses the associative memory as a set of comparators, each programmed with one of the search terms. This approach works well for fully specified terms, ones with fixed-length don't cares, or for initial or terminal VLDCs. Embedded VLDCs cannot be easily handled.

Besides the obvious parallel-comparator implementation for an associative memory, two schemes based on hardware hashing of the comparand have been suggested. Bird et al. (1979) proposed a hashing based on the length of the comparand word and its initial characters, while Burkowski (1982) used a small mapping RAM based on subsets of the search terms selected so that each has a unique bit pattern. While these approaches substantially reduce the cost of implementing the associative memory, they both suffer from the inability to handle embedded VLDCs or classes of characters (any delimiter, a number, any letter) in a pattern.

Finite State Machines

Finite state automata are capable of recognizing any regular expression. An FSA can be described as a 5-tuple {A, S, M, B, F} where A is the input alphabet, S is a set of elements called states, M is a mapping from AxS into S, B is a member of S called the beginning state, and F is a nonempty subset of S called the final states. The operation of the FSA is simple. At the start, it is in state B. Whenever a character arrives, the mapping M is used to determine the next state of the FSA. This process continues until a final state is reached, indicating the term that has been matched. Implementation is equally simple. A memory holds the mapping function M as an array of states. Each state contains an array with one entry for each character in the input alphabet, indicating the next state. A register holds the memory base address for the current state, and is concatenated with the input character to give the memory address to read to find the next state, which is then loaded into the register.

There is a problem with the straightforward FSA implementation--the size of memory required for the mapping function. For a query with 250 terms (as might occur when a thesaurus is used), about 2000 states would be necessary. For seven-bit input characters, this would require over 3 million bits of memory. Although that does not seem high today, 15 years ago that was the same order of magnitude for the memory found on a mainframe computer, and would cost hundreds of thousands of dollars.

Two ways of reducing the memory requirements of the FSA, at the expense of additional logic, have been proposed. Bird observed that there are two different state transition types in the mapping memory: sequential and index (Mayper et al. 1980). Sequential states are used for matching single characters, and are stored sequentially in memory. Matching the character causes the state register to be advanced by one and a mismatch causes it to go to a specified failure address. For a seven-bit input character, only eight bits are necessary for each sequential state--seven for the match character and one to indicate it is a sequential state.

In those places where there is more than one character of interest, the index state is used. It consists of a code indicating it is an index state and a bit vector indicating the interesting characters in the mapping. A seven-bit input character requires a 128-bit vector. When an index state is processed, the first action is to determine whether the bit corresponding to the input character is set. If not, a transition is made to a default state. If the bit is set, the number of bits set for characters with a lower value is determined and used as an index added to the current state address to give the location of a word that contains the address of the next state. Since there are far more sequential states than index states, the bit requirements are reduced. Extrapolating from the figures in Haskin et al. (1983), approximately 250 Kbits would be necessary for the 250-term query.

There are a number of difficulties with the Bird approach. The first is the dramatic size difference between the sequential and index states (8 bits vs. over 128 bits). This is mitigated by having more than one sequential state in each memory word, although some parts of the word might end up unused, and by operating on nibbles (half bytes) rather than the full character for index states. This reduces the vector to 16 bits and simplifies the hardware necessary to compute the index, at the expense of two index operations per character and a more complex state table.

Another difficulty is that when any search term has an initial VLDC, all states must be index states, substantially increasing the memory requirements. To solve this problem, Bird suggested having two matchers, one for terms that don't have initial VLDCs and a smaller one for those that have them. He also suggested a third FSA to handle phrases, rather than as part of the query resolver.

The other way of reducing the FSA memory requirements was proposed by Haskin (1980) (further described in Haskin et al. [1983] and Hollaar et al. [1984]). It consists of dividing the FSA state table into a number of partitions that can then be processed in parallel. The partitions are made so that each state must check only a single character, like the Bird sequential state, to determine whether to go to a specified address or a mismatch state. If there is more than one character that could occur simultaneously, each must be in a different partition. This can be easily determined by an examination of all the terms when the state tables are being constructed. A successful match not only specifies the next state in a partition, but can force another partition into a specified state.

For example, if the terms DOG and DOT were specified, the first state would look for a D and transition to state 2 when it is found. That state would look for an O. However, since the next state needs to look for both a G and a T, a successful match of the O would cause a transition to state 3, which would look for a G, and would force another partition into a state looking for a T. This fork operation consists of forcing the address of a neighbor matcher, since the partitioning technique assures that the neighbor is in an idle state.

A state can be programmed for an exact match, a match ignoring the bit that differentiates between upper- and lowercase characters, or a match if the input character is of a particular class (numeric, alphabetic, delimiter). A startup mechanism is used to simplify the partitioning of the state table, and to reduce the matcher requirements eliminating starts that would terminate after the initial character of the word. All forms of fixed- and variable-length don't cares can be handled. For the set of 250 terms, approximately 20 character matchers of less than 5 Kbits each are necessary, for a total memory requirement of under 100 Kbits, less than half the bits of the Bird FSA and a thirtieth of the conventional FSA requirements. Moreover, the matchers are cycled only once for each input character, allowing the use of slower memories than would be possible with the Bird FSA.

Cellular Arrays

A cellular array uses a large number of very simple matchers connected in a string or as a more complex structure (Hollaar 1979). In its most basic form, the cell is programmed with a single match character. The cell has an input enable line and an output match line, as well as an input that is connected to a bus containing the current input character. Whenever the enable line is true, the cell compares the character on the bus to its match character, and if they are equal, sets its output match line true. FLDCs can be handled by a dummy cell that repeats its enable line delayed by one character time, and VLDCs by a cell that sets its match output whenever its enable line is set and clears it when a word delimiter is found (assuming that VLDCs should not extend across a word). The cells can also be extended to handle matching of classes of characters, like delimiters. Of course, any addition to a cell beyond an exact character match complicates its design and means fewer cells can be placed on an integrated circuit.

One of the first cellular matchers was General Electric's GESCAN, developed in the mid-1960s (Crew et al. 1967). It was capable of searching data stored on special magnetic tape drives at 120,000 characters per second, based on a query programmed into 80 cells. Copeland (1978) proposed a similar cellular system as an extension to a database machine; it was capable of handling a single bounded-length pattern string. Mukhopadhyay (1979) proposed a complex cell and a variety of ways to interconnect the cells to handle regular expressions and much of the pattern matching capabilities of SNOBOL4. However, the number of gates needed for a general interconnection network for n cells is O(n2), so for a reasonable number of cells the cost of the interconnections will exceed the cost of the cells. Lee (1985) (also Lee and Lochovsky [1990]) proposed a cellular system as part of a larger hardware-implemented information retrieval system that placed blocks of the data into the cellular array and broadcast programming instructions based on the search pattern to all the cells. Foster and Kung (1980) proposed a systolic pattern matcher, where both the pattern and the data being searched are moved through the cellular array. This eliminates the need to broadcast the data to every cell, eliminating a heavily loaded data bus.

Cellular arrays are used in two currently marketed search machines. A spin-off of General Electric, GESCAN International, has developed a new implementation of GESCAN that can be attached to a minicomputer or workstation. Rather than use a special tape drive, data is read from the host disk system and transferred to the search logic. The host also acts as the query resolver. Each integrated circuit contains 16 cells, and can operate at 5 MBytes per second. The Fast Data Finder from TRW (Yu et al. 1986) uses a cell that can perform not only single character comparisons, but also includes support for ignoring misspellings and numeric comparisons. Each integrated circuit holds 8 cells, and up to 9,216 cells can be included in a system. It can search at 12.5 MBytes per second, although the disk storing the data often limits its speed.

Standard Microprocessors

It was clear a decade ago that available microprocessors could not keep up with the disk for a multiterm match, but the latest RISC-based processors are substantially faster than mainframe computers were only a few years ago, at a fraction of the cost. While their performance depends on the particular algorithm used for searching, looking at the inner loop for the various grep programs, egrep appears to have the tightest loop for complex multiterm searches. On a Sun SPARC, an optimized version takes 10 instructions and 13 cycles. This means that the inner loop searching rate for a 25 MHz SPARCstation-1 is about 1.92 MBytes per second, approximately the nominal disk rate of 1.875 MBytes per second for a 15 MHz drive. A 40 MHz SPARC will run at over 3 MBytes per second.

But even as microprocessors become faster, they still may not be a good choice for handling the basic term matching operation. To achieve a high MIPS rate, the program for the microprocessor must be stored in a high-speed memory. For a 25 MHz machine, it must have a cycle time of less than 40 ns. This means the use of a cache memory to achieve better performance than would be available with dynamic RAM, increasing the cost and complexity of the microprocessor system.

If we look at the inner loop for egrep, we find that it implements a finite state recognizer, using two arrays. The first one is indexed by the current state and the input character and gives the next state. The second indicates whether a match has been found. For a system handling 250 terms, to accommodate large scarches generated by a thesaurus, about 2,000 states would be necessary. An FSA for handling 2,000 states and 7-bit characters would take 12 256K RAMs and an 11-bit holding register, costing well under $50.

Of course, it is not necessary for the line between the microprocessor and the special search hardware to be drawn between term matching and query resolution as an alternative to performing the entire search in the microprocessor. Hardware augmentation can be used to improve the search performance of the microprocessor. For example, a simple startup mechanism such as was used in the PFSA could be used to initially examine the input characters, with the processor only having to work when the start of a possible match has been found (Hollaar 1991). As microprocessor cost performance continues to improve, they will play a larger role in the implementation of special purpose text searchers. These systems will differ from general purpose systems using the same microprocessor by having low-overhead control programs rather than general operating systems like UNIX, simplified inter-faces connecting their peripheral devices, file systems optimized for text searching, and some custom logic to assist in the matching process.

17.2.2 Search Performance

Text searchers were proposed as a means of improving the response time over a conventional computer doing the search, without the problems associated with using a surrogate such as an inverted file. Using a special purpose searcher that operates at disk speed, the time necessary to complete a user's query is the time it takes to read all the data from a disk drive. While this obviously varies with the type of disk, for most low-cost, high-capacity disks it is about five minutes. This is far too long for an interactive system, and complicates the search control by requiring that a number of different user queries must be combined in a batch to give reasonable performance.

One solution to this problem is to make the search go faster by using higher performance disks, but this substantially increases the system cost because of the low production volumes and resulting higher costs for such disks. Furthermore, the speedup is on the order of 10, while a factor of 100 or more is desirable for an interactive system.

A more reasonable solution is to combine the attributes of searching and using a surrogate to overcome the difficulties with each approach. An inexact surrogate can be used to eliminate documents that have no hope of matching a query. Superimposed codewords will provide a list of documents that is a superset of the documents containing the search terms; if the term doesn't really occur in the document, but is an artifact of the superimposed codeword scheme, it will be eliminated by the search. A fully inverted file, where every term is indexed with its location within a document is not necessary; phrases, contexts, and word location proximities can be handled by the following search.

Rather than the 50 percent to 300 percent overhead for a fully inverted file (Bird et al. 1978), a partially inverted file could have an overhead of less that 5% (Hollaar 1991). This is because only one entry is necessary in the list of documents containing a word no matter how many times a word occurs in the document, and information regarding the location of the word within the document is not stored. The partially inverted surrogate also provides a quick feedback to the user on whether the follow-up search should even be started. It may indicate that too few documents (and possibly no documents, if a term that is not in the database is used) or too many documents would be searched, based on the user's intuition of what is in the database. The search could then be canceled and the query refined, based on information from the index.

It is not necessary to index a document before it is available. As long as the number of unindexed documents remains low relative to an average search, they can simply be added to the list of documents for every search, making them available as soon as the text is loaded.

Seek and search mode

When a surrogate is used to reduce the number of documents to be searched, the search goes from a scan to a seek and search mode. This changes the critical disk parameter from its transfer rate to its seek time relative to document transfer time. To see why this is so, consider an ESDI drive like the Maxtor XT-8760, where the seek time and rotational latency to position to the start of a document is about 20 milliseconds. It takes another 25 msec to read 35,000 characters (the size of the average U. S. Patent, for example) off two tracks, for a total of 45 msec. The effective transfer rate is about 750 KBytes per second, 40 percent of the nominal transfer rate of 1,875 KBytes per second. The effective transfer rate will be less for smaller documents, since the seek time will remain the same but the amount of data read will be less. A disk drive with the same seek characteristics, but which could read the data in zero time, would be only 2.25 times faster.

Just as parallel transfer drives are not particularly effective in a seek and search mode, optical drives with their high seek times are even more devastating to search performance. If the seek time in the small document example above were changed to a 150 ms positioning time (typical of today's optical disks), the effective transfer rate is only 148 KBytes per second, less than 8 percent of the nominal transfer rate.

Substantial improvements can be made by using an appropriate file system. A randomly organized file system, where blocks are placed in any convenient free location on the disk and a pointer (either in the previous block or in a master block) is used to locate the next data block(such as used in most file systems) is convenient for a time sharing system, where files come and go. However, using our example ESDI disk drive, if we have to do an average seek before reading a 512 byte block, we will have an effective transfer rate of under 50 KBytes per second. The use of a large block size improves this, at the expense of higher unusable disk capacity due to internal fragmentation.

The use of a contiguous file system, where each document is stored in consecutive disk blocks, substantially improves disk performance. In one disk revolution after positioning, almost 50 blocks can be read, rather than just one. Since the documents are seldom removed (or expanded) after they are loaded into a archival text database, the problems of file reorganization and lost disk space are minimal.

17.3 SURROGATE PROCESSORS

While text searching has been the most commonly proposed information retrieval operating to be implemented using special purpose hardware, a number of people have examined manipulating document surrogates using custom processors. These have included processors for the merging of the sorted lists of documents in an inverted file index and for searching the document signatures to find specified patterns.

17.3.1 List Merging

The index for an inverted file contains pointers to items within the database. These pointers are generally stored in sorted order to ease their combining to determine those documents that match a particular query. The basic merging operation produces a list of pointers that is the union (for an OR operation), intersection (for an AND), or Boolean difference (for an AND NOT). While this is a simple operation to program on a general purpose computer, the overhead associated with aligning data, processing auxiliary fields, and flow of control such as loop counters mean that the mainframe computers of the 1970s were saturated keeping up with disk reads.

Stellhorn (1977) proposed a system that took blocks of the sorted lists and merged them in a block-parallel, bit-serial fashion. Logic following the merge operation removed either duplicate (for an OR) or duplicate and single entries (for an AND). Hollaar (1978) developed a system based on a simple merge element (both serial and parallel versions were designed) connected by a network or arranged as a binary tree, which was programmed to match the merge operation specified by the query. Both of these systems were 10 to 100 times faster than a conventional computer of the time. However, as processors have become less expensive and much faster, special purpose hardware for list merging is does not seem cost-effective today.

17.3.2 Signature Processors

A signature file, as discussed in a previous chapter, provides an alternative to storing a list of document pointers for each word in the database. A signature is developed for each document that attempts to capture the contents of the document. These signatures, which are considerably smaller than the actual documents, are then searched for potential matches. These matches can be directly presented to the user, or can be searched to eliminate false matches caused by information dropped when the signatures were created.

The most common signature is based on superimposed codewords. Keywords or word stems are selected from the document and hashed to produce an n-bit vector that has m bits set. These vectors are then ORed together to form the signature. Determining whether a document contains a particular word simply requires checking to see if the bits corresponding to the hash of that word are set in the signature. Note that superimposed codewords do not readily allow don't cares or context and proximity operations, and may produce "false drops" when the bit vectors representing two words are ORed, and the result has all the bits of another word, not present in the document, set.

Ahuja and Roberts (1980) proposed storing the superimposed codeword signatures in an associative memory. To reduce the cost, rather than have comparators for every bit in the associative memory, they divided the signatures into blocks and stored each block in a memory module. During the associative search, a codeword is read from the memory, compared against the desired pattern, whether there was a match is saved, and the next codeword from the memory is examined. This word-serial, bit-parallel approach reduces the hardware requirements to one comparator (for as many bits as are in the codeword) per memory module, plus the associated control logic.

Lee (1985) (also Lee and Lochovsky [1990]) observed that it is not necessary to check every bit position of the signatures, but only those that are set in the hash of the word of interest. Since the number of bits set in a hash is small compared to the number of bits in the codeword, this can substantially improve the search performance. Since the same bits in each codeword are being skipped, a word-parallel, bit-serial architecture is best.

17.4 OPTICAL PROCESSORS

The previously discussed special purpose systems were all based on conventional digital electronics. Spurred by work on laser-based compact disk technology and the use of fiber optics for high-speed data transmission, there has been renewed interest in employing optically based approaches in information retrieval. This has included improving the bandwidth of optical mass storage and optical searching of information.

17.4.1 Improved Optical Storage

As we previously mentioned, most optical storage systems now in use do not have either the data rate or access speed of magnetic disk memories. This is particularly true for compact disks, which were originally designed to handle the relatively low data rates needed for reproducing audio information. However, there are ways of substantially increasing the data rates of optical disks beyond the standard techniques of increasing the bit density or rotational speed of the disk.

It is difficult to build a magnetic read head that can read data from many adjacent, closely spaced tracks at the same time. However, it is possible to have a laser beam that illuminates many adjacent tracks, with the information from each track falling on a different position of a photodetector array. This allows a byte or word of data to be read out with only minimal problems of alignment or skewed data. In the case of a parallel-head magnetic drive, the relative timing of the bits of the word depends on the mechanical alignment of the different heads, which can change with temperature or movement during seeks.

Even more interesting is the use of a holographic memory, rather than a rotating optical disk. This offers the potential of not only improved data rates, but also better storage densities and access times. Berra et al. (1989) discusses a page hologram memory capable of holding 725 MBytes on a 6-inch square card, and volume holograms capable of holding up to 1012 bits. Since access to information in the holographic memory does not require moving a read head assembly and waiting for a disk to rotate to the start of the data, access times are reduced and rotational latency is eliminated. Instead, the laser beam can be deflected to the proper position in under 100 msec and all the information on the page will be immediately available.

17.4.2 Optical Data Processing

Although it is possible to convert the optical signals from a laser disk or holographic memory to electrical signals and process them using conventional digital electronics, one also can use optical means for comparing the information against a pattern without the need for conversion. Berra et al. (1989) discuss a number of these techniques, and present an optical full text search design (Mitkas et al. 1989).

Optical comparisons are performed by using an exclusive-OR operation on dual rail logic (both true and complement of each bit is available). A 1 bit is stored as a TF pattern, and a 0 bit as FT. A spatial light modulator (SLM), which is a one-or two-dimensional array that can alter some parameter of the optical signals passing through the array, such as intensity or phase, is placed in the light path. In its simplest form, the SLM is programmed to form a mask in which individual bits are either passed (P) or blocked (X). XP is used for comparing against a 1, PX for comparing against a 0, and XX can be used for a don't care in a particular bit position. If all pattern bits match the data bits, no light will be passed. It only takes a lens and a single detector to determine if there is a match.

For example, if the pattern is 10d1, where d is a don't care, the mask will be XPPXXXXP. For data of 1011 (dual rail form of TFFTTFTF) or 1001 (TFFTFTTF), no light will be passed, while for data 0011 (FTFTTFTF), the resulting light pattern will be FTFFFFFF, so light is passed and a mismatch is indicated.

The optical text search machine contains a page composer, which generates optical patterns to be fed to the optical comparator. Its main element is a fast optical scanner that converts the sequential (and possibly byte-parallel) input from the optical disk to two-dimensional patterns consisting of shifting sequences of the bytes of text. The optical comparator masks these patterns with an array programmed with all the search terms. Photodetectors sense whether no light passes, indicating a match of a particular term. It is possible to do 100,000 parallel comparisons using an SLM in a single step. Other optical techniques can be employed to allow the multiple input streams to be handled through the same SLM, further increasing its performance.

17.5 SUMMARY

A number of special-purpose hardware systems have been proposed to provide most cost-effective or faster performance for information retrieval systems. Most of these have been text scanners, and several of these have been used in special applications.

While high-speed microprocessors and general purpose parallel machines can now provide much of the performance that previously required special purpose logic (such as for index merging), it still appears that hardware augmentation to those processors can improve the performance of a large system at a small cost. Special purpose hardware for information retrieval will continue to be an interesting research and development area.

REFERENCES

AHUJA, S. R., & C. S. ROBERTS. 1980, "An Associative/Parallel Processor for Partial Match Retrieval Using Superimposed Codes." Proceedings of the 7th Annual Symposium on Computer Architecture, May 6-8, 1980 (published as SIGARCH Newsletter, vol. 8, no. 3) pp. 218-27.

BERRA, P. B., A. GHAFOOR, P. A. MITKAS, S. J. MARCINKOWSKI, & M. GUIZANI. 1989. IEEE Transactions on Knowledge and Data Engineering, (1), 111-32.

BIRD, R. M., J. B. NEWSBAUM, & J. L. TREFFTZS. 1978, "Text File Inversion: An Evaluation." Fourth Workshop on Computer Architecture for Non-Numeric Processing, Syracuse University, August 1978 (published as SIGIR vol. 13, no. 2; SIGARCH vol. 7, no. 2; and SIGMOD vol. 10, no. 1), pp. 42-50.

BIRD, R. M., & J. C. TU. 1979, "Associative Crosspoint Processor System." U. S. Patent 4,152, 762, May 1, 1979.

BURKOWSKI, F. J. 1982, "A Hardware Hashing Scheme in the Design of a Multiterm String Comparator." IEEE Transactions on Computers, C-31, (9), 825-34.

COPELAND, G. P. 1978, "String Storage and Searching for Data Base Applications: Implementation of the INDY Backend Kernel." Fourth Workshop on Computer Architecture for Non-Numeric Processing, Syracuse University, August 1978 (published as SIGIR vol. 13, no. 2; SIGARCH vol 7, no. 2; and SIGMOD vol. 10, no. 1), pp. 8-17.

CREW, B. L., & M. N. GUNZBURG. 1967, "Information Storage and Retrieval System." U. S. Patent 3,358,270, December 12, 1967.

FOSTER, M. J., & H. T. KUNG. 1980. "Design of Special-Purpose VLSI Chips: Examples and Opinions." Proceedings of the 7th Annual Symposium on Computer Architecture, May 6-8, 1980 (published as SIGARCH Newsletter, vol. 8, no. 3) pp. 300-07.

HASKIN, R. L. 1980. "Hardware for Searching Very Large Text Databases." Ph.D. Thesis, University of Illinois at Urbana-Champaign, August.

HASKIN, R. L., & L. A. HOLLAAR. 1983, "Operational Characteristics of a Hardware-based Pattern Matcher." ACM Transactions on Database Systems, 8(1).

HOLLAAR, L. A. 1978. "Specialized Merge Processor Networks for Combining Sorted Lists." ACM Transactions on Database Systems, 3(3), 272-84.

HOLLAAR, L. A. 1979. "Text Retrieval Computers." Computer, 12(3), 40-52.

HOLLAAR, L. A., & R. L. HASKIN. 1984, "Method and System for Matching Encoded Characters." U. S. Patent 4,450,520, May 22, 1984.

HOLLAAR, L. A. 1991. "Special-Purpose Hardware For Text Searching: Past Experience, Future Potential." Information Processing & Management, Vol. 27, No. 4, pp. 371-78.

LEE, D. L. 1985. "The Design and Evaluation of a Text-Retrieval Machine for Large Databases." Ph.D. Thesis, University of Toronto, September.

LEE, D. L., & F. H. Lochovsky. 1990, "HYTREM--A Hybrid Text-Retrieval Machine for Large Databases." IEEE Transactions on Computers, 39(1), 111-23.

MAYPER, V. Jr., A. L. NAGY, R. M. BIRD, J. C. TU, & L. S. MICHAELS. 1980. "Finite State Automaton with Multiple State Types." U. S. Patent 4,241,402, December 23, 1980.

MITKAS, P. A., P. B. BERRA, & P. S. GUILFOYLE. 1989, "An Optical System for Full Text Search." Proceedings of SIGIR 89.

MUKHOPADHYAY, A . 1979, "Hardware Algorithms for Nonnumeric Computation." IEEE Transactions on Computers, C-28(6), 384-89.

ROBERTS, D. C. 1978. "A Specialized Computer Architecture for Text Retrieval." Fourth Workshop on Computer Architecture for Non-Numeric Processing, Syracuse University, August 1978 (published as SIGIR vol. 13, no. 2; SIGARCH vol. 7, no. 2; and SIGMOD vol . 10, no . 1), pp . 51-59.

STELLHORN, W. H. 1977. "An Inverted File Processor for Information Retrieval." IEEE Transactions on Computers, C-26(12), 1258-67.

YU, K.-I, S.-P. HSU, R. E. HEISS, Jr., & L. Z. HASIUK. 1986. "Pipelined for Speed: The Fast Data Finder System." Quest, Technology at TRW, 9(2), Winter 1986/1987, 4-19.

Go to Chapter 18 Back to Table of Contents


Information Retrieval: CHAPTER 18: PARALLEL INFORMATION RETRIEVAL ALGO

 Return to Table of Contents

CHAPTER 18: PARALLEL INFORMATION RETRIEVAL ALGORITHMS

Craig Stanfill

Thinking Machines Corporation, 245 First Street, Cambridge, Massachusetts

Abstract

Data Parallel computers, such as the Connection Machine CM-2, can provide interactive access to text databases containing tens, hundreds, or even thousands of Gigabytes of data. This chapter starts by presenting a brief overview of data parallel computing, a performance model of the CM-2, and a model of the workload involved in searching text databases. The remainder of the chapter discusses various algorithms used in information retrieval and gives performance estimates based on the data and processing models just presented. First, three algorithms are introduced for determining the N highest scores in a list of M scored documents. Next, the parallel signature file representation is described. Two document scoring algorithms are fully described; a sketch of a boolean query algorithm are also presented. The discussion of signatures concludes with consideration of false hit rates, data compression, secondary/tertiary storage, and the circumstances under which signatures should be considered. The final major section discusses inverted file methods. Two methods, parallel inverted files and partitioned posting files, are considered in detail. Finally, issues relating to secondary storage are briefly considered.

18.1 INTRODUCTION

The time required to search a text database is, in the limit, proportional to the size of the database. It stands to reason, then, that as databases grow they will eventually become so large that interactive response is no longer possible using conventional (serial) machines. At this point, we must either accept longer response times or employ a faster form of computer. Parallel computers are attractive in this respect: parallel machines exist which are up to four orders of magnitude faster than typical serial machines. In this chapter we will examine the data structure and algorithmic issues involved in harnessing this power.

In this chapter we are concerned with vector-model document ranking systems. In such systems, both documents and queries are modeled as vectors. At a superficial level, retrieval consists of (1) computing the dot-product of the query vector with every document vector; and (2) determining which documents have the highest scores. In practice, of course, both document and query vectors are extremely sparse; coping with this sparseness lies at the heart of the design of practical scoring algorithms.

The organization of the chapter is as follows. First, we will describe the notation used to describe parallel algorithms, and present a timing model for one parallel computer, the Connection Machinel\®System model CM-2 [image]. 1 Second, we will define a model of the retrieval task that will allow us to derive performance estimates. Third, we will look at algorithms for ranking documents once they have been scored.2 Fourth, we will consider one database representation, called parallel signature files, which represents the database as a set of signatures. Fifth, we will consider two different file structures based on inverted indexes. Sixth, we will briefly look at issues relating to secondary storage and I/O. Finally, we will summarize the results and delineate areas for continued research.

1Connection Machine\® is a registered trademark of Thinking Machines Corporation. C*\® is a registered trademark of Thinking Machines Corporation. CM-2 [image], CM [image], and DataVault [image] are trademarks of Thinking Machines Corporation.

2This topic is presented first because certain aspects of document scoring will be difficult to understand if the data arrangements convenient to document ranking have not yet been presented.

18.2 DATA PARALLEL COMPUTING

The algorithms fpresented in this chapter utilize the data parallel computing model proposed by Hillis and Steele (1986). In this model, there is a single program that controls a large number of processing elements, each of which will be performing the same operation at any moment. Parallelism is expressed by the creation of parallel data structures and the invocation of parallel operators. The model is made manifest in data parallel programming languages, such as the C* language developed by Thinking Machines Corporation (1990). The body of this section presents some basic data structures, operators, and notations which will be required to understand the algorithms presented in the remainder of the chapter. The section will conclude with a concise performance model for one implementation of the data parallel model (the Connection Machine model CM-2). More details on the architecture of this machine are provided by Hillis (1985) and Thinking Machines Corporation (1987).

Data types are implicit in the algorithmic notation presented below. In general, scalar variables are lowercase, for example, i. Scalar constants are all uppercase, for example, N_PROCS. Parallel integer-valued variables are prefixed with P_, e.g. P_x. Parallel Boolean-valued variables are prefixed with B_. Other aspects of data type, such as underlying structure and array declarations, will be left implicit and may be deduced by reading the accompanying text.

18.2.1 Shapes and Local Computation

C* includes all the usual C data structures and operations. These are called scalar variables and scalar operations.

A shape may be thought of as an array of processors. Each element of a shape is referred to as a position. A parallel variable is defined by a base type and a shape. It can be thought of as a vector having one element per position in the shape. A parallel variable can store one value at each of its positions. For example, if P is defined in a shape with 8 positions, it will have storage for 8 different values. When we display the contents of memory in the course of describing data structures and algorithms, variables will run from top to bottom and positions will run from left to right. Thus, if we have two variables, P_1 and P_2, we might display them as follows:

P_1  8   6  3   4   9   1  2  0

-------------------------------

P_2  7  14  8  29  17  34  1  9

Individual values of a parallel variable are obtained by left indexing: element 4 of P_1 is referenced as [4] P_1, and has a value of 9. All indexing is zero-based.

There is a globally defined parallel variable, P_position, which contains 0 in position 0, l in position 1, and so on.

It is also possible to have parallel arrays. For example, P_array might be a parallel variable of length 3:

P_array [0]   4  38  17  87  30  38  90  81

-------------------------------------------

P_array [1]  37   3  56  39  89  10  10  38

-------------------------------------------

P_array [2]  01  83  79  85  13  87  38  61

Array subscripting (right indexing) is done as usual; element 1 of P_array would be referred to as P_array [1], and would be a parallel integer having 8 positions. Left and right indexing may be combined, so that the 4'th position of the 0'th element of P_array would be referred to as [4] P_array[0], and have the value of 30.

Each scalar arithmetic operator (+, *, etc.) has a vector counterpart that is applied elementwise to its operands. For example, the following line of code multiplies each element of P_x by the corresponding element of P_y, then the stores the result in P_z:

P_z = P_x * P_y;

This might result in the following data store:

P_x  1  1  1  2  2  2  3  3

P_y  1  2  3  1  2  3  1  2

---------------------------

P_z  1  2  3  2  4  6  3  6

At any moment, a given shape has a set of active positions. All positions are initially active. Parallel operations, such as arithmetic and assignment, take effect only at active positions. The set of active positions may be altered by using the where statement, which is a parallel analogue to the scalar if statement. The where statement first evaluates a test. The body will then be executed with the active set restricted to those positions where the test returned nonzero results. The else clause, if present, will then be executed wherever the test returned 0. For example, the following computes the smaller of two numbers:

where (P_1 P_2)

P_min = P_1;

else

P_min = P_2;

18.2.2 Nonlocal Operations

Everything mentioned up to this point involves the simple extension of scalar data structures and operations to vector data structures and element-wise vector operations. We will now consider some basic operations that involve operating on data spread across multiple positions; these are collectively referred to as nonlocal operations.

The simplest of these operations are the global reduction operations. These operations compute cumulative sums, cumulative minima/maxima, and cumulative bitwise AND/ORs across all active positions in a shape. The following unary operators are used to stand for the global reduction operators:

          +=  Cumulative sum

          &=  Cumulative bitwise AND

          |=  Cumulative bitwise OR

          >?=  Cumulative maximum

          <?=  Cumulative minimum

Suppose, for example, we wish to compute the arithmetic mean of P_x. This may be done by computing the cumulative sum of P_x and dividing it by the number of active positions. This second quantity can be computed by finding the cumulative sum (over all active positions) of 1:

mean = (+= P_x) / (+= 1);

Parallel left-indexing may be used to send data from one position to another. In this operation, it is useful to think of each position in a shape as corresponding to a processor. When it sees an expression such as [P_i] P_y = P_x, it will send its value of P_x to position P_i, and store it in P_y. For example, one might see the following:

P_x  5  0  6  4  1  7  3  2

P_i  7  4  1  2  5  0  6  3

---------------------------

P_y  7  6  4  2  0  1  3  5

In the event that multiple positions are sending data to the same destination, conflicts may be resolved by arbitrarily choosing one value, by choosing the largest/ smallest value, by adding the values, or by taking the bitwise AND/OR of the values. These different methods of resolving collisions are specified by using one of the following binary operatores:3

          =  Send with overwrite (arbitrary choice)

          +=  Send with add

          &=  Send with bitwise AND

            =  Send with bitwise OR

          <?=  Send with minimum

          >?=  Send with maximum

These are binary forms of the global reduce operations introduced above.

3These are referred to as the send-reduce operators.

The final group of nonlocal operations to be considered here are called scan operations, and are used to compute running sums, running maxima/minima, and running bitwise AND/OR's. In it simplest form, scan_with_add will take a parallel variable and return a value which, at a given position, is the cumulative sum of all positions to its left, including itself. For example:

P_x                  2  0  1  2  4   3   2   1

scan_with_add (P_x)  2  2  3  5  9  12  14  15

Optionally, a Boolean flag (called a segment flag) may be supplied. Wherever this flag is equal to 1, it causes the running total to be reset to 0. For example:

P_x                  2  0  1  2  4  3  2  1

B_s                  1  0  0  0  1  0  1  0

add_scan (P_x, B_s)  2  2  3  5  4  7  2  3

18.2.3 Performance Model

We will now consider the performance of one parallel computer, the Connection Machine model CM-2. In scalar C, the various primitive operators such as + and = have fairly uniform time requirements. On parallel computers, however, different operators may have vastly differing time requirements. For example, adding two parallel variables is a purely local operation, and is very fast. Parallel left-indexing, on the other hand, involves moving data from one processor to another, and is two orders of magnitude slower.

In addition, any realization of this model must take into account the fact that a given machine has a finite number of processing elements and, if a shape becomes large enough, several positions will map to the same physical processor. We call the ratio of the number of positions in a shape to the number of physical processors the virtual processing ratio (VP ratio). As the VP ratio increases, each processor must do the work of several and, as a first approximation, a linear increase in running time will be observed.

The following symbols will be used:

          Nprocs   The number of physical processors

                   The VP ratio

We express the time required for each operator by an equation of the form c1 + c2r. For example, if an operator takes time 2 + 10r, then it will take 12 microseconds at a VP ratio of 1, 22 microseconds at a VP ratio of 2, and so forth. For convenience, we will also include the time required for an operator at a VP ratio of 1. On the CM-2, the time required for scalar operations is generally insignificant and will be ignored.

To arrive at a time estimate for an algorithm, we first create an algorithm skeleton in which all purely scalar operations except for looping are eliminated, and all identifiers are replaced with P for parallel integers, B for parallel Booleans, and S for scalars. Loop constructs will be replaced by a simple notation of the form loop (count) . All return statements will be deleted. Assignment statements which might reasonably be eliminated by a compiler will be suppressed. Finally, because the cost of scalar right-indexing is zero, all instances of P [S] will be replaced with P. From this skeleton, the number of times each parallel operator is called may be determined. The time requirements are then looked up in a table provided at the end of this section, and an estimate constructed.

For example, suppose we have a parallel array of length N and wish to find the sum of its elements across all positions. The algorithm for this is as follows:

sum_array(P_array)

{

P_result = 0;

for (i = 0; i< N; i++)

P_result += P_array;

return (+= P_result);

}

This has the skeleton:

P = S

loop (N)

P += P

(+= P)

It requires time:

Operation  Calls         Time per Call

------------------------------------------

P = S        1             3 + 15r

P += P       n             3 + 28r

(+= P)       1           137 + 70r

------------------------------------------

Total             (140 + 3n) + (85 + 28n)r

The following timing equations characterize the performance of the CM-2.4

4Throughout this chapter, all times are in microseconds unless noted otherwise.

Operator          Time     r = 1  Comments

B = S            3 + 3r        6

B $= B           3 + 3r        6

B $$ B           3 + 3r        6

where(B)         8 + 2r       10

S = [S]P           16         16

[S]P = S           16         16

P = S            3 + 15r      18

P += S           3 + 28r      31

P += P           3 + 28r      31

P[P] = P        11 + 60r      71

P = P[P]        11 + 60r      71

P == S          18 + 67r      85  Same time for <= etc.

(>?= P)        137 + 70r     207  Same time for += etc.

scan_with_or   632 + 56r     688

scan_with_add  740 + 170r    910

[P]P = P          2159r      2159  Same time for += etc.

[P]P[P] = P       2159r      2159  Same time for += etc.

18.3 A MODEL OF THE RETRIEVAL TASK

18.4 RANKING

Retrieval consists of (1) scoring documents, and (2) determining which documents received the highest scores. This second step--ranking--will be considered first. Any of the ranking algorithms discussed below may be used in combination with any of the scoring algorithms which will be discussed later. It should be noted that, while scoring is probably the more interesting part of the retrieval process, ranking may be a large portion of the overall compute cost, and ranking algorithms are as deserving of careful design as are scoring algorithms.

The problem may be stated as follows: given a set of Ndocs integers (scores), identify the Nret highest-ranking examples. The scores may be stored in one of two formats: with either one score per position, or with an array of Nrows scores per position. The former case involves a VP ratio

The second case, assuming a VP ratio of one is used, requires an array of size

We will assume there is a fast method for converting a parallel variable at a VP ratio of r to a parallel array having Nrows cells per processor. Such a function is, in fact, provided on the Connection Machine; it requires essentially zero time.

18.4.1 System-Supplied Rank Functions

Many parallel computing systems provide a parallel ranking routine which, given a parallel integer, returns 0 for the largest integer, 1 for the next largest, and so forth.5 This may be used to solve the problem quite directly: one finds the rank of every score, then sends the score and document identifier to the position indexed by its rank. The first Nret values are then read out.

5On the CM-2 this routine takes time 30004r.

For example:

 P_score    83  98   1  38  78  37  17  55

------------------------------------------

rank         1   0   7   4   2   5   6   3

After send  98  83  78  55  38  37  17   1

The algorithm is as follows:

rank_system(dest, P_doc_score, P_doc_id)

{

P_rank = rank(P_doc_score);

[P_rank]P_doc_score = P_doc_score;

[P_rank] P_doc_id   = P_doc_id;

for (i = 0; i < N_RET; i++)

{

dest[i].score = [i] P_doc_score;

dest[i].id = [i]P_doc_id;

}

}

This has the skeleton:

P = rank ()

[P]P = P

[P]P = P

loop (N_RET)

{

S = [S]P

S = [S]P

}

Its timing characteristics are:

Operation  Calls   Time per Call

-----------------------------------

rank ( )     1         30004r

P]P = P      2          2159r

S = [S]P    2Nret        16

-----------------------------------

Total              32Nret + 34322r

Substituting the VP ratio r =   [image]  gives a time of:

18.4.2 Iterative Extraction

The system-supplied ranking function does much more work than is really required: it ranks all Ndocs scores, rather than the Nret which are ultimately used. Since we usually have Nret << Ndocs, we may look for an algorithm that avoids this unnecessary work. The algorithm that follows, called iterative extraction, accomplishes this by use of the global-maximum (>?=P) operation.

The insight is as follows: if we were only interested in the higest-ranking document, we could determine it by direct application of the global maximum operation. Having done this, we could remove that document from further consideration and repeat the operation. For example, we might start with:

P_score  83  98  1  38  78  37  17  55

We find that the largest score is 98, located at position 1. That score can be eliminated from further consideration by setting it to - 1:

P_score  83  -1  1  38  78  37  17  55

On the next iteration, 83 will be the highest-ranking score. The algorithm is as follows:

rank_iterative(dest, P_doc_score, P_doc_id)

{

for  (i = 0; i < N_RET; i++)

{

best_score = ( > ?= P_doc_score);

where (P_doc_score == best_score)

{

position = ( <?= P_position);

dest[i].score = [position]P_doc_score;

dest[i].id    = [position]P_doc_id;

[position]P_doc_score = -1;

}

}

}

This has the skeleton:

loop (N_RET)

{

S = ( >?= P)

where (P == S)

{

S = ( <?= P)

S = [S]P

S = [S]P

[S]P = S

}

}

And the timing:

Operation  Calls    Time per Call

------------------------------------

(>? = P)   2Nret      137 + 70r

P == S      Nret       18 + 67r

where       Nret        8 + 2r

S = [S]P   2Nret          16

[S]P = S    Nret          16

------------------------------------

Total             348Nret + 209Nretr

Substituting the VP ratio   [image]  gives a time of:

18.4.3 Hutchinson's Algorithm

The following algorithm, due to Jim Hutchinson (1988), improves on iterative extraction. Hutchinson's algorithm starts with an array of

scores stored in each position, at a VP ratio of 1. For example, with 32 documents, 8 processors, and 4 rows we might have the following data:

P_scores [0]  88  16  87  10  94  04  21  11

P_scores [1]  90  17  83  30  37  39  42  17

P_scores [2]  48  43  10  62   4  12  10   9

P_scores [3]  83  98   1  38  78  37  17  55

We start by extracting the largest score in each row, placing the results in a parallel variable called P_best:

P_scores [0]  88  16  87  10  -1   4  21  11

P_scores [1]  -1  17  83  30  37  39  42  17

P_scores [2]  48  43  10  -1   4  12  10   9

P_scores [3]  83  -1   1  38  78  37  17  55

--------------------------------------------

P_best        94  90  62  98  -1  -1  -1  -1

We then extract the best of the best (in this case 98):

P_scores [0]  88  16  87  10  94   4  21  11

P_scores [1]  -1  17  83  30  37  39  42  17

P_scores [2]  48  43  10  -1   4  12  10   9

P_scores [3]  83  -1   1  38  78  37  17  55

--------------------------------------------

P_best        94  90  62  -1  -1  -1  -1  -1

and replenish it from the appropriate row (3 in this case).

P_scores [0]  88  16  87  10  94   4  21  11

P_scores [1]  -1  17  83  30  37  39  42  17

P_scores [2]  48  43  10  -1   4  12  10   9

P_scores [3]  -1  -1   1  38  78  37  17  55

--------------------------------------------

P_best        94  90  62  83  -1  -1  -1  -1

This is repeated Nret times. The algorithm involves two pieces. First is the basic extraction step:

extract_step(P_best_score, P_best_id, P_scores, P_ids, row)

{

max_ score = (>?= P_ scores[row]);

where (P_ scores[row] == max_score)

{

position = (<?= P_position);

[row]P_best_score = [position]P_scores[row];

[row]P_best_id    = [position]P_scores[row];

[position]P_scores[row] = -1;

}

}

This has the skeleton:

S = (>?= P)

where(P == S)

{

S = (<?= P)

[S] P = [S] P

[S] P = [S] P

[S] P = S

}

We do not have a separate timing figure for [S] P = [S] P, but we note that this could be rewritten as S = [S]P; [S]P = S. The timing is thus as follows:

Operation  Calls  Time per Call

-------------------------------

(>?= P)      2        207

where        l         10

S == P       1         85

[S]P = S     3         16

S = [S] P    2         16

-------------------------------

Total                 589

Given this extraction step subroutine, one can easily implement Hutchinson's algorithm:

rank_hutchinson(dest, P_scores, P_ids)

{

P_best_score = -1;

P_best_id    = 0;

for (row = 0; row < N_ROWS; row++)

extract_step(P_best_score, P_best_id,

P_scores, P_ids, row);

for (i = 0; i < N_RET; i++)

{

best_of_best = (>?= P_best_score);

where (P_best_score == best_of_best)

{

position        =  (<?= P_position);

dest [i].score  =  [position]P_scores;

dest[i].id      =  [position]P_ids;

[positions]P_best_score = -1;

}

extract_step(P_best_score, P_best_id,

P_scores, P_ids, row);

}

}

This has the skeleton:

P = S

P = S

loop(N_ROWS)

extract_step ()

loop(N_RET)

{

S = (>?= P)

where(P == S)

{

S = (<?= P)

S = [S]P

S = [S]P

[S]P = S

}

extract_step()

}

Its timing is as follows:

Operation         Calls          Time per Call

-------------------------------------------------

( > ? = P )       2Nret              207

S == P             Nret               85

whereV             Nret               10

S= [S]P           2Nret               16

[S]P = S           Nret               16

extract_step  (Nrows + Nret)         589

-------------------------------------------------

Total                         1149Nret + 589Nrows

Substituting in the value of Nrows we arrive at:

18.4.4 Summary

We have examined three ranking algorithms in this section: the system-defined ranking algorithm, iterative extraction, and Hutchinson's algorithm. Their times are as follows:

For very small versions of Nret, iterative extraction is prefered; for very large values of Nret, the system ranking function is preferred, but in most cases Hutchinson's algorithm will be preferred. Considering various sizes of database, with a 65,536 processor Connection Machine, and our standard database parameters, the following rank times should be observed:

     |D|      Ndocs     System  Iterative  Hutchinson

   1 GB  200 X 103     138 ms      24 ms        25ms

  10 GB    2 X 106    1064 ms     137 ms       41 ms

 100 GB   20 X 106   10503 ms    1286 ms      203 ms

1000 GB  200 X 106  104751 ms   12764 ms     1821 ms

The time required to rank documents is clearly not an obstacle to the implementation of very large IR systems. The remainder of the chapter will be concerned with several methods for representing and scoring documents; these methods will then use one of the algorithms described above (presumably Hutchinson's) for ranking.

18.5 PARALLEL SIGNATURE FILES

The first scoring method to be considered here is based on parallel signature files. This file structure has been described by Stanfill and Kahle (1986, 1988, 1990a) and by Pogue and Willet (1987). This method is an adaptation of the overlap encoding techniques discussed in this book.

Overlap encoded signatures are a data structure that may be quickly probed for the presence of a word. A difficulty associated with this data structure is that the probe will sometimes return present when it should not. This is variously referred to as a false hit or a false drop. Adjusting the encoding parameters can reduce, but never eliminate, this possibility. Depending on the probability of such a false hit, signatures may be used in two manners. First, it is possible to use signatures as a filtering mechanism, requiring a two-phase search in which phase 1 probes a signature file for possible matches and phase 2 re-evaluates the query against the full text of documents accepted by phase 1. Second, if the false hit rate is sufficiently low, it is possible to use signatures in a single phase system. We will choose our signature parameters in anticipation of the second case but, if the former is desired, the results shown below may still be applied.

18.5.1 Overlap Encoding

An overlap encoding scheme is defined by the following parameters:

     Sbits    Size of signature in bits

     Sweight  Weight of word signatures

     Swords   Number of words to be inserted in each signature

     Hj(Ti)   A set of Sweight hash functions

Unless otherwise specified, the following values will be used:

          Sbits   4096

          Sweight    10

          Swords    120

A signature is created by allocating Sbits bits of memory and initializing them to 0. To insert a word into a signature, each of the hash functions is applied to the word and the corresponding bits set in the signature. The algorithm for doing this is as follows:

create_signature (B_signature, words)

{

for (i = 0; i < S_BITS; i++)

B_signature[i] = 0;

for (i = 0; i < S_WORDS; i++)

for (j = 0; j < S_WEIGHT; j++)

B_signature[hash(j, words[i])] = 1;

}

The timing characteristics of this algorithm will not be presented.

18.5.2 Probing a Signature

To test a signature for the presence of a word, all Sweight hash functions are applied to it and the corresponding bits of the signature are ANDed together. A result of 0 is interpreted as absent and a result of 1 is interpreted as present.

probe_signature(B_signature, word)

{

B_result = 1;

for (i = 0; i < S_WEIGHT; i++)

B_result &= B_signature[i];

return B-result;

}

This has the skeleton:

B = S

loop(S_WEIGHT)

B &= B

Its timing is:

Operation               Calls       Time per Call

-----------------------------------------------------

B = S                     1            3 + 3r

B &= B                  Sweight        3 + 3r

-----------------------------------------------------

Total                            3(1 + Sweight)(1 + r)

Total for Sweight = 10                 33 + 33r

The VP ratio will be determined by total number of signatures in the database. This, in turn, depends on the number of signatures per document. A randomly chosen document will have the length L, and require

signatures. If the distribution of L is reasonably smooth, then a good approximation for the average number of signatures per document is:

The number of signatures in a database is then

and the VP ratio is

We can now compute the average time per query term:

18.5.3 Scoring Algorithms

Documents having more than Swords must be split into multiple signatures. These signatures can then be placed in consecutive positions, and flag bits used to indicate the first and last positions for each document. For example, given the following set of documents:

 ______________________________________________________________

|                     |                     | Still another    |

| This is the initial | This is yet another | document taking  |

| document            | document            | yet more space   |

|                     |                     | than the others  |

|_____________________|_____________________|__________________|

we might arrive at signatures divided as follows:

                                               Still

             This    initial  This    another  another  taking  space

             is the  docu-    is yet  docu-    docu-    yet     than   others

B_signature          ment             ment     ment     more    the

-----------------------------------------------------------------------------

B_first         1      0         1       0        1       0      0       0

B_last          0      1         0       1        0       0      0       1

We can then determine which documents contain a given word by (1) probing the signatures for that word, and (2) using a scan_with_or opcration to combine the results across multiple positions. For example, probing for "yet" we obtain the following results:

                                       Still

              This    initial  This    another  another  taking  space

              is the  docu-    is yet  docu-    docu-    yet     than   others

B_signature           ment             ment     ment     more    the

------------------------------------------------------------------------------

B_first          1      0         1      0        1        0       0      0

B_last           0      1         0      1        0        0       0      1

probe

 ("yet")         0      0         1      0        0        1       0      0

scan_with_or     0      0         1      1        0        1       1      1

This routine returns either 1 or 0 in the last position of each document, according to whether any of its signatures contained the word. The value at other positions is not meaningful.

The algorithm for this is as follows:

probe_document(B_signature, B_first, word)

{

B_local = probe_signature(B_signature, word);

B_result = scan_with_or(B_local, B_first);

}

This is the skeleton:

B = probe_signature()

B = scan_with_or()

Its timing is:

Operation        Calls  Time per Call

-------------------------------------

probe_signature    1      33 + 33r

scan_with_or       1     632 + 56r

-------------------------------------

Total                    665 + 89r

Using the above building blocks, it is fairly simple to construct a scoring alogrithm. In this algorithm a query consists of an array of terms. Each term consists of a word and a weight. The score for a document is the sum of the weights of the words it contains. It may be implemented thus:

score_document(B_signature, B_first, terms)

{

P_score = 0;

for (i = 0; i < N_TERMS; i++)

{

B_probe = probe_document(B_signature, terms[i].word, B_first)

where (B_probe)

P_score += terms[i].weight;

}

return P_score;

}

This has the skeleton:

P = S

loop (N_TERMS)

{

B = probe_document()

where(B)

P += S

}

Its timing characteristics are as follows:

Operation       Calls          Time per Call

-------------------------------------------------------

P = S             1               3 + 15r

probe_document  Nterms           665 + 89r

where           Nterms             8 + 2r

P += S          Nterms             3 + 28r

-------------------------------------------------------

Total                  3 + 676Nterms + (15 + 119Nterms)r

It is straightforward to implement Boolean queries with the probe_document operation outlined above. Times for Boolean queries will be slightly less than times for document scoring. As a simplification, each query term may be: (1) a binary AND operation, (2) a binary OR operation, (3) a NOT operation, or (4) a word. Here is a complete Boolean query engine:

query(B_signature, B_first, term)

{

arg0 = term-args[0];

arg1 = term-args[1];

switch (term-connective)

{

case AND: return   query(B_signature, B_first, arg0) &&

query(B_signature, B_first, arg1);

case OR: return    query(B_signature, B_first, arg0)

query(B_signature, B_first, arg1);

case NOT: return ! query(B_signature, B_first, arg0);

case WORD: return probe_document(B_signature, B_first, arg0);

}

}

The timing characteristics of this routine will not be considered in detail; it should suffice to state that one call to probe_document is required for each word in the query, and that probe_document accounts for essentially all the time consumed by this routine.

18.5.4 An Optimization

The bulk of the time in the signature scoring algorithm is taken up by the probe_document operation. The bulk of the time for that operation, in turn, is taken up by the scan_with_or operation. This operation is performed once per query term. We should then seek to pull the operation outside the query-term loop. This may be done by (1) computing the score for each signature independently, then (2) summing the scores at the end. This is accomlished by the following routine:

score_document(B_signature, B_first, terms)

{

P_score = 0;

for (i = 0; i < N_TERMS; i++)

{

B_probe = probe_signature(B_signature, term[i].word);

where (B_probe)

P_score += term[i].weight;

}

P_score = scan_with_add(P_score, B_first);

return P_score;

}

This has the skeleton:

P = S;

loop (N_TERMS)

{

B = probe_signature ();

where (B)

P += S;

}

P = scan_with_add ();

The timing characteristics are as follows:

Operation        Calls              Time per Call

-----------------------------------------------------------

P = S              1                  3 + 15r

probe_signature  Nterms               33 + 33r

where            Nterms                8 + 2r

P += S           Nterms                3 + 28r

scan_with_add      1                 740 + 170r

-----------------------------------------------------------

Total                   (743 + 44Nterms) + (185 + 63Nterms)r

Comparing the two scoring algorithms, we see:

Basic Algorithm      (3 + 676Nterms) + (15 + 119Nterms)r

Improved Algorithm  (743 + 44Nterms) + (185 + 63Nterms)r

The dominant term in the timing formula, Ntermsr, has been reduced from 119 to 63 so, in the limit, the new algorithm is 1.9 times faster. The question arises, however, as to what this second algorithm is computing. If each query term occurs no more than once per document, then the two algorithms compute the same result. If, however, a query term occurs in more than one signature per document, it will be counted double or even treble, and the score of that document will, as a consequence, be elevated. This might, in fact, be beneficial in that it yields an approximation to document-term weighting. Properly controlled, then, this feature of the algorithm might be beneficial. In any event, it is a simple matter to delete duplicate word occurances before creating the signatures.

18.5.5 Combining Scoring and Ranking

The final step in executing a query is to rank the documents using one of the algorithms noted in the previous section. Those algorithms assumed, however, that every position contained a document score. The signature algorithm leaves us with only the last position of each document containing a score. Use of the previously explained algorithms thus requires some slight adaptation. The simplest such adaptation is to pad the scores out with - 1. In addition, if Hutchinson's ranking algorithm is to be used, it will be necessary to force the system to view a parallel score variable at a high VP ratio as an array of scores at a VP ratio of 1; the details are beyond the scope of this discussion.

Taking into account the VP ratio used in signature scoring, the ranking time will be:

Substituting the standard values for Nterms, Nret,   [image], and Swords gives us a scoring time of:

and a ranking time of:

The times for various sizes of database, on a machine with 65,536 processors, are as follows:

      D         Ndocs    Score     Rank     Total

   1 GB  200 X 103     9 ms    28 ms     37 ms

  10 GB    2 X 106    74 ms    75 ms    149 ms

 100 GB   20 X 106   723 ms   545 ms   1268 ms

1000 GB  200 X 106  7215 ms  5236 ms  12451 ms

18.5.6 Extension to Secondary/Tertiary Storage

It is possible that a signature file will not fit in primary storage, either because it is not possible to configure a machine with sufficient memory or because the  expense of doing so is unjustified. In such cases it is necessary that the signature file reside on either secondary or tertiary storage. Such a file can then be searched by repetitively (1) transferring signatures from secondary storage to memory, (2) using the above signature-based algorithms to score the documents, and (3) storing the scores in a parallel array. When the full database has been passed through memory, any of the above ranking algorithms may be invoked to find the best matches. The algorithms described above need to be modified, but the compute time should be  unchanged. There will, however, be the added expense of reading the signature file into primary memory. If RIO is the I/O rate in megabytes per second, and c is the signature file compression factor (q.v. below), then the time to read a signature file through memory will be:    For a fully configured CM-2, RIO = 200. The signature parameters we have assumed yield a compression factor c = 30 percent (q.v. below). This leads to the following I/O times:

      D    I/O Time

   1 GB     2 sec

  10 GB    15 sec

 100 GB   150 sec

1000 GB  1500 sec

Comparing the I/O time with the compute time, it is clear that this method is I/O bound. As a result, it is necessary to execute multiple queries in one batch in order ot make good use of the compute hardware. This is done by repeatedly (1) transferring signatures from secondary storage to memory: (2) calling the signature- based scoring routine once for each query; and (3) saving the scores produced for each query in a separate array. When all signatures have been read, the ranking algorithm is called once for each query. Again, the algorithms described above need modification, but the basic principles remain unchanged. Given the above parameters, executing batches of 100 queries seems reasonable, yielding the following times:

                   Search Time

     D    I/O Time  (100 queries)     Total

  1 GB     2 sec          4 sec     6 sec

 10 GB    15 sec         15 sec    30 sec

100 GB   150 sec        127 sec   277 sec

100 GB  1500 sec       1245 sec  2745 sec

This has not, in practice, proved an attractive search method.

18.5.7 Effects of Signature Parameters

It is guaranteed that, if a word is inserted into a signature, probing for it will return present. It is possible, however, for a probe to return present for a word that was never inserted. This is referred to variously as a false drop or a false hit. The probability of a false hit depends on the size of the signature, the number of hash codes, and the number of bits set in the table. The number of bits actually set depends, in turn, on the number of words inserted into the table. The following approximation has proved useful:

There is a trade-off between the false hit probability and the amount of space required for the signatures. As more words are put into each signature (i.e., as Swords increases), the total number of signatures decreases while the probability of a false hit increases. We will now evaluate the effects of signature parameters on storage requirements and the number of false hits.

A megabyte of text contains, on the average, Rdocs documents, each of which requires an average of   [image]  signatures. Each signature, in turn, requires   [image]  bytes of storage. Multiplying the two quantities yields the number of bytes of signature space required to represent 1 megabyte of input text. This gives us a compression factor6 of:

6The compression factor is defined as the ratio of the signature file size to the full text.

If we multiply the number of signatures per megabyte by Pfalse, we get the expected number of false hits per megabyte:

We can now examine how varying Swords alters the false hit probability and the compression factor:

Swords  Signatures/MB  Compression        Pfalse  False hits/GB

   40           1540          77%  4.87 X 10-11    7.50 X 10-5

   80            820          42%  3.09 X 10-8     2.50 X 10-2

  120            580          30%  1.12 X 10-6     6.48 X 10-1

  160            460          24%  1.25 X 10-5     5.75 X 100

  200            388          20%  7.41 X 10-5     2.88 X 101

  240            340          17%  2.94 X 10-4     1.00 X 102

  280            306          16%  8.88 X 10-4     2.72 X 102

  320            280          14%  2.20 X 10-3     6.15 X 102

Signature representations may also be tuned by varying Sbits and Swords in concert. As long as Sbits = kSwords for some constant k, the false hit rate will remain approximately constant. For example, assuming Sweight = 10 and Sbits = 34.133Swords, we get the following values for Pfalse:

Swords  Sbits      Pfalse

   80   2731  1.1163 X 10-6

  120   4096  1.1169 X 10-6

  160   5461  1.1172 X 10-6

Since the computation required to probe a signature is constant regardless of the size of the signature, doubling the signature size will (ideally) halve the number of signatures and consequently halve the amount of computation. The degree to which computational load may be reduced by increasing signature size is limited by its effect on storage requirements. Keeping Sbits = 34.133Swords, Sweight = 10 and varying Swords, we get the following compression rates:

Swords  Sbits    c

   60   2048  27%

  120   4096  30%

  240   8192  35%

Clearly, for a fixed k (hence, as described above, a fixed false hit rate), storage costs increase as Sbits increases, and it is not feasible to increase Sbits indefinitely.

For the database parameters assumed above, it appears that a signature size of 4096 bits is reasonable.

18.5.8 Discussion

The signature-based algorithms described above have a number of advantages and disadvantages. There are two main disadvantages. First, as noted by Salton and Buckley (1988) and by Croft (1988), signatures do not support general document-term weighting, a problem that may produce results inferior to those available with full document-term weighting and normalization. Second, as pointed out by Stone (1987), the I/O time will, for single queries, overwhelm the query time. This limits the practical use of parallel signature files to relatively small databases which fit in memory. Parallel signature files do, however, have several strengths that make them worthy of consideration for some applications. First, constructing and updating a signature file is both fast and simple: to add a document, we simply generate new signatures and append them to the file. This makes them attractive for databases which are frequently modified. Second, the signature algorithms described above make very simple demands on the hardware; all local operations can be easily and efficiently implemented using bit-serial SIMD hardware, and the only nonlocal operation scan_with_add can be efficiently implemented with very simple interprocessor communication methods which scale to very large numbers of processors. Third, signature representations work well with serial storage media such as tape. Given recent progress in the development of high-capacity, high-transfer rate, low-cost tape media, this ability to efficiently utilize serial media may become quite important. In any event, as the cost of random access memory continues to fall, the restriction that the database fit in primary memory may become less important.

18.6 PARALLEL INVERTED FILES

An inverted file is a data structure that, for every word in the source file, contains a list of the documents in which it occurs. For example, the following source file:

 ______________________________________________________________

|                     |                     | Still another    |

| This is the initial | This is yet another | document taking  |

| document            | document            | yet more space   |

|                     |                     | than the others  |

|_____________________|_____________________|__________________|

has the following inverted index:

another   1  2

document  0  1  2

initial   0

is        0  1

more      2

others    2

space     2

still     2

taking    2

than      2

the       0  2

this      0  1

yet       1  2

Each element of an inverted index is called a posting, and minimally consists of a document identifier. Postings may contain additional information needed to support the search method being implemented. For example, if document-term weighting is used, each posting must contain a weight. In the event that a term occurs multiple times in a document, the implementer must decided whether to generate a single posting or multiple postings. For IR schemes based on document-term weighting, the former is preferred; for schemes based on proximity operations, the latter is most useful.

The two inverted file algorithms described in this chapter differ in (1) how they store and represent postings, and (2) how they process postings.

18.6.1 Data Structure

The parallel inverted file structure proposed by Stanfill, Thau, and Waltz (1989) is a straightforward adaptation of the conventional serial inverted file structure. A parallel inverted file is a parallel array of postings such that the postings for a given word occupy contiguous positions within a contiguous series of rows, plus an index structure indicating the start row, end row, start position, and end position of the block of postings for each word. For example, given the database and inverted file shown above, the following parallel inverted file would result:

 Postings

----------

1  2  0  1

2  0  0  1

2  2  2  2

2  2  0  2

0  1  1  2

                   Index

-----------------------------------------

Word      First    First   Last    Last

          Row    Position  Row   Position

-----------------------------------------

another    0        0       0       1

document   0        2       1       0

initial    1        1       1       1

is         1        2       1       3

more       2        0       2       0

others     2        1       2       1

space      2        2       2       2

still      2        3       2       3

taking     3        0       3       0

than       3        1       3       1

the        3        2       3       3

this       4        0       4       1

yet        4        2       4       3

In order to estimate the performance of algorithms using this representation, it is necessary to know how many rows of postings need to be processed. The following discussion uses these symbols:

Pi                 The number of postings for term Ti

Ri                 The number of rows in which postings for Ti occur

    The average number of rows per query term

(r, p)             A row-position pair

Assume the first posting for term Ti is stored starting at (r, p). The last posting for Ti will then be stored at

and the number of rows occupied by Ti will be

Assuming p is uniformly distributed between 0 and Nprocs-1, the expected value of this expression is

From our frequency distribution model we know Ti occurs f (Ti) times per megabyte, so Pi = |D| f (Ti). This gives us:

Taking into account the random selection of query terms (the random variable Q), we get a formula for the average number of rows per query-term:

Also from the distribution model, f(Q) = Z, and   [image]. This gives us:

18.6.2 The Scoring Algorithm

The scoring algorithm for parallel inverted files involves using both left- and right-indexing to increment a score accumulator. We start by creating an array of score registers, such as is used by Hutchinson's ranking algorithm. Each document is assigned a row and a position within that row. For example, document i might be mapped to row i mod Nprocs, position   [image]. Each posting is then modified so that, rather than containing a document identifier, it contains the row and position to which it will be sent. The Send with add operation is then used to add a weight to the score accumulator. The algorithm is as follows:

score_term (P_scores, P_postings, term)

{

for (row = term.start_row; row <= term.end_row; row++)

{

if (row == term.start_row)

start_position = term.start_position;

else

start_position = 0;

if (row == term.end_row)

end_position = term.end_position;

else

end_position = N_PROCS-1;

where ((start_position <= P_position) &&

(P_position <= end_position))

{

P_dest_pos = P_postings[row].dest_pos;

P_dest_row = P_postings[row].dest_row;

[P_dest_pos] P_scores [P_dest_row] += term.weight;

}

}

}

The inner loop of this algorithm will be executed, on the average,   [image]  times. This yields the following skeleton:

loop(R_BAR)

where ((S = P) && (P = S)) /* Also 1 B && B operation */

[P] P [P] += S

This has the following timing characteristics:

Taking into account the value of   [image]  yields the following time per query term:

Substituting our standard value for   [image], we get times for a 65,536 processor CM-2. Times for scoring 10 terms and for ranking are also included. Finally, the total retrieval time for a 10-term query is shown

     |D|    Time  10 Terms     Rank    Total

   1 GB    2 ms     25 ms    25 ms    50 ms

  10 GB    3 ms     34 ms    41 ms    75 ms

 100 GB   13 ms    131 ms   203 ms   334 ms

1000 GB  110 ms   1097 ms  1821 ms  2918 ms

18.6.3 Document Term Weighting

Up to now, the algorithms we have discussed support only binary document models, in which the only information encoded in the database is whether a given term appears in a document or not. The parallel inverted file structure can also support document term weighting, in which each posting incorporates a weighting factor; this weighting factor measures the strength of usage of a term within the document. The following variant on the query execution algorithm is then used:

score_weighted_term (P_scores, P_postings, term)

{

for (row = term.start_row; row = term.end_row; row++)

{

if (row == term.start_row)

start_position = term.start_position;

else

start_position = 0;

if (row == term.end_row)

end_position = term.end_position;

else

end_position = N_PROCS-1;

where ((start_position = P_position) &&

(P_position = end_position))

{

P_dest_pos = P_postings[row].dest_pos;

P_dest_row = P_postings[row].dest_row;

P_weight = term.weight * P_postings[row].weight;

[P_dest_pos] P_scores [P_dest_row] += P_weight;

}

}

}

This algorithm requires only slightly more time than the unweighted version.

18.7 PARTITIONED POSTING FILES

18.8 SECONDARY STORAGE

One major advantage of inverted files is that it is possible to query them without loading the entire file into memory. The algorithms shown above have assumed that the section of the file required to process a given query are already in memory. While a full discussion of the evolving field of I/O systems for parallel computing is beyond the scope of this paper, a brief presentation is in order. In the final analysis, most of what is known about I/O systems with large numbers of disks (e.g. mainframe computers) will probably hold true for parallel systems.

This discussion is oriented towards the partitioned posting file representation. For this algorithm, the disk system is called on to simply read partitions into memory.

18.8.1 Single-Transfer I/O on Disk Arrays

I/O systems for parallel computers are typically built from large arrays of simple disks. For example, the CM-2 supports a disk array called the Data VaultTM which contains 32 data disk plus 8 ECC disks.8 It may be thought of as a single disk drive with an average latency of 200 milliseconds and a transfer rate of 25 MB/sec- ond. Up to 8 Data Vaults may be simultaneously active, yielding a transfer rate of up to 200 MB/second. This access method achieves very high transfer rates, but does not yield many I/O's per second; this can be crippling for all but the very largest databases. Consider, for example, a 64K processor Connection Machine with 8 disk arrays operating in single transfer mode. Assume we are using the partitioned posting file representation, and that each posting requires 4 bytes of storage. The storage required by each partition is then 4FNprocs.

8Data Vault is a trademark of Thinking Machines Corporation

The average query-term requires NP partitions to be loaded. These partitions may be contiguously stored on disk, so the entire group of partitions may be transferred in a single operation. The time is then:

Given a seek time of 200 milliseconds and a transfer rate of 200 M-B/second, plus our other standard assumptions, the following per-term times will result:

      D      Seek  Transfer  Score

   1 GB  200 ms      5 ms   1 ms

  10 GB  200 ms      5 ms   1 ms

 100 GB  200 ms     11 ms   2 ms

1000 GB  200 ms     65 ms   9 ms

Under these circumstances, the system is severely seek-bound for all but the very largest databases.

18.8.2 Independent Disk Access

Fortunately, the disk arrays contain buried in them the possibility of solving the problem. Each Data Vault has 32 disks embedded in it; a system with 8 disk arrays thus has a total of 256 disks. It is possible to access these disks independently. Under this I/O model, each disk transfers a block of data into the memories of all processors. The latency is stil 200 milliseconds, but 256 blocks of data will be transferred rather than l. This has the capability of greatly reducing the impact of seek times on system performance.

At this point in time, independent disk access methods for parallel computers are still in development; considerable work is required to determine their likely performance in the context of information retrieval. Stanfill and Thau (1991) have arrived at some preliminary results.

18.9 SUMMARY

The basic algorithmic issues associated with implementing information retrieval systems on databases of up to 1000 GB may be considered solved at this point in time.

The largest difficulty remaining in the implementation of parallel inverted file algorithms remains the I/O system. Disk arrays, operating in single-transfer mode, do not provide a sufficiently large number of I/O's per second to match available processing speeds until database sizes approach a thousand Gigabytes or more. Multi-transfer I/O systems have the potential to solve this problem, but are not yet available for data parallel computers. At this point, parallel inverted file algorithms are restricted to databases which either fit in primary memory or are large enough that the high latency time is less of an issue. However, these problems are very likely to find solution in the next few years.

It should be clear at this point that the engineering and algorithmic issues involved in building large-scale Information Retrieval systems are well on their way to solution and, over the next decade, we can reasonably look forward to interactive access to text databases, no matter how large they may be.

REFERENCES

CROFT, B. (1988). Implementing Ranking Strategies Using Text Signatures. ACM Transactions on Office Information Systems, 6(1), 42-62.

HILLIS, D. (1985) The Connection Machine, Cambridge, MA: MIT Press.

HILLIS, D. & and STEELE, G. (1986). Data Parallel Algorithms. Communications of the ACM, 29(12), 1170-1183.

HUTCHINSON, J. (1988). Personal Communications.

POGUE, C. & WILLETT, P. (1987). Use of Text Signatures for Document Retrieval in a Highly Parallel Environment. Parallel Computing, 4, 259-268.

SALTON, G. & BUCKLEY, C. (1988). Parallel Text Search Methods. Communications of the ACM, 31(2), 202-215.

STANFILL, C. & KAHLE, B. (1986). Parallel Free-Text Search on the Connection Machine System. Communications of the ACM, 29 (12), 1229-1239.

STANFILL, C. (1988a). Parallel Computing for Information Retrieval: Recent Developments. Technical Report DR88-1. Cambridge, MA: Thinking Machines Corporationl

STANFILL, C., THAU, R. & WALTZ, D. (1989). A Parallel Indexed Algorithm for Information Retrieval. Paper Presented at the International Conference on Research and Development in Information Retrieval. Cambridge, MA.

STANFILL, C. (1990a). Information Retrieval Using Parallel Signature Files. IEEE Data Engineering Bulletin, 13 (1), 33-40.

STANFILL, C. (1990b). Partitioned Posting Files: a Parallel Inverted File Structure for Information Retrieval. Paper presented at the International Conference on Research and Development in Information Retrieval. Brussels, Belgium.

STANFILL, C. & THAU, R. (1991). Information Retrieval on the Connection Machine: 1 to 8192 Gigabytes. Information Processing and Management, 27(4), 285-310.

STONE, H. (1987). Parallel Querying of Large Databases: a Case Study. Computer, 20(10), 11-21.

Thinking Machines Corporation. (1987). Connection Machine model CM-2 technical specifications. Cambridge, MA: Thinking Machines Corporation.

Thinking Machines Corporation. (1990). C* Programming Guide. Cambridge, MA: Thinking Machines Corporation.

Back to Table of Contents


Information Retrieval: FOREWORD

 Return to Table of Contents

FOREWORD

Udi Manber

Department of Computer Science, University of Arizona

In the not-so-long ago past, information retrieval meant going to the town's library and asking the librarian for help. The librarian usually knew all the books in his possession, and could give one a definite, although often negative, answer. As the number of books grew--and with them the number of libraries and librarians--it became impossible for one person or any group of persons to possess so much information. Tools for information retrieval had to be devised. The most important of these tools is the index--a collection of terms with pointers to places where information about them can be found. The terms can be subject matters, author names, call numbers, etc., but the structure of the index is essentially the same. Indexes are usually placed at the end of a book, or in another form, implemented as card catalogs in a library. The Sumerian literary catalogue, of c. 2000 B.C., is probably the first list of books ever written. Book indexes had appeared in a primitive form in the 16th century, and by the 18th century some were similar to today's indexes. Given the incredible technology advances in the last 200 years, it is quite surprising that today, for the vast majority of people, an index, or a hierarchy of indexes, is still the only available tool for information retrieval! Furthermore, at least from my experience, many book indexes are not of high quality. Writing a good index is still more a matter of experience and art than a precise science.

Why do most people still use 18th century technology today? It is not because there are no other methods or no new technology. I believe that the main reason is simple: Indexes work. They are extremely simple and effective to use for small to medium-size data. As President Reagan was fond of saying "if it ain't broke, don't fix it." We read books in essentially the same way we did in the 18th century, we walk the same way (most people don't use small wheels, for example, for walking, although it is technologically feasible), and some people argue that we teach our students in the same way. There is a great comfort in not having to learn something new to perform an old task. However, with the information explosion just upon us, "it" is about to be broken. We not only have an immensely greater amount of information from which to retrieve, we also have much more complicated needs. Faster computers, larger capacity high-speed data storage devices, and higher bandwidth networks will all come along, but they will not be enough. We will need better techniques for storing, accessing, querying, and manipulating information.

It is doubtful that in our lifetime most people will read books, say, from a notebook computer, that people will have rockets attached to their backs, or that teaching will take a radical new form (I dare not even venture what form), but it is likely that information will be retrieved in many new ways, but many more people, and on a grander scale.

I exaggerated, of course, when I said that we are still using ancient technology for information retrieval. The basic concept of indexes--searching by keywords--may be the same, but the implementation is a world apart from the Sumerian clay tablets. And information retrieval of today, aided by computers, is not limited to search by keywords. Numerous techniques have been developed in the last 30 years, many of which are described in this book. There are efficient data structures to store indexes, sophisticated query algorithms to search quickly, data compression methods, and special hardware, to name just a few areas of extraordinary advances. Considerable progress has been made for even seemingly elementary problems, such as how to find a given pattern in a large text with or without preprocessing the text. Although most people do not yet enjoy the power of computerized search, and those who do cry for better and more powerful methods, we expect major changes in the next 10 years or even sooner. The wonderful mix of issues presented in this collection, from theory to practice, from software to hardware, is sure to be of great help to anyone with interest in information retrieval.

An editorial in the Australian Library Journal in 1974 states that "the history of cataloging is exceptional in that it is endlessly repetitive. Each generation rethinks and reformulates the same basic problems, reframing them in new contexts and restating them in new terminology." The history of computerized cataloging is still too young to be in a cycle, and the problems it faces may be old in origin but new in scale and complexity. Information retrieval, as is evident from this book, has grown into a broad area of study. I dare to predict that it will prosper. Oliver Wendell Holmes wrote in 1872 that "It is the province of knowledge to speak and it is the privilege of wisdom to listen." Maybe, just maybe, we will also be able to say in the future that it is the province of knowledge to write and it is the privilege of wisdom to query.

Go to Preface Back to Table of Contents


Information Retrieval: PREFACE

 Return to Table of Contents

PREFACE

Text is the primary way that human knowledge is stored, and after speech, the primary way it is transmitted. Techniques for storing and searching for textual documents are nearly as old as written language itself. Computing, however, has changed the ways text is stored, searched, and retrieved. In traditional library indexing, for example, documents could only be accessed by a small number of index terms such as title, author, and a few subject headings. With automated systems, the number of indexing terms that can be used for an item is virtually limitless.

The subfield of computer science that deals with the automated storage and retrieval of documents is called information retrieval (IR). Automated IR systems were originally developed to help manage the huge scientific literature that has developed since the 1940s, and this is still the most common use of IR systems. IR systems are in widespread use in university, corporate, and public libraries. IR techniques have also been found useful, however, in such disparate areas as office automation and software engineering. Indeed, any field that relies on documents to do its work could potentially benefit from IR techniques.

IR shares concerns with many other computer subdisciplines, such as artificial intelligence, multimedia systems, parallel computing, and human factors. Yet, in our observation, IR is not widely known in the computer science community. It is often confused with DBMS--a field with which it shares concerns and yet from which it is distinct. We hope that this book will make IR techniques more widely known and used.

Data structures and algorithms are fundamental to computer science. Yet, despite a large IR literature, the basic data structures and algorithms of IR have never been collected in a book. This is the need that we are attempting to fill. In discussing IR data structures and algorithms, we attempt to be evaluative as well as descriptive. We discuss relevant empirical studies that have compared the algorithms and data structures, and some of the most important algorithms are presented in detail, including implementations in C.

Our primary audience is software engineers building systems with text processing components. Students of computer science, information science, library science, and other disciplines who are interested in text retrieval technology should also find the book useful. Finally, we hope that information retrieval researchers will use the book as a basis for future research.

Bill Frakes

Ricardo Baeza-Yates

ACKNOWLEDGEMENTS

Many people improved this book with their reviews. The authors of the chapters did considerable reviewing of each others' work. Other reviewers include Jim Kirby, Jim O'Connor, Fred Hills, Gloria Hasslacher, and Ruben Prieto-Diaz. All of them have our thanks. Special thanks to Chris Fox, who tested The Code on the disk that accompanies the book; to Steve Wartik for his patient unravelling of many Latex puzzles; and to Donna Harman for her helpful suggestions.

Go to Chapter 1 Back to Table of Contents


Information Retrieval: Table of Contents

Information Retrieval: Data Structures & Algorithms

edited by William B. Frakes and Ricardo Baeza-Yates

FOREWORD

PREFACE

CHAPTER 1: INTRODUCTION TO INFORMATION STORAGE AND RETRIEVAL SYSTEMS

CHAPTER 2: INTRODUCTION TO DATA STRUCTURES AND ALGORITHMS RELATED TO INFORMATION RETRIEVAL

CHAPTER 3: INVERTED FILES

CHAPTER 4: SIGNATURE FILES

CHAPTER 5: NEW INDICES FOR TEXT: PAT TREES AND PAT ARRAYS

CHAPTER 6: FILE ORGANIZATIONS FOR OPTICAL DISKS

CHAPTER 7: LEXICAL ANALYSIS AND STOPLISTS

CHAPTER 8: STEMMING ALGORITHMS

CHAPTER 9: THESAURUS CONSTRUCTION

CHAPTER 10: STRING SEARCHING ALGORITHMS

CHAPTER 11: RELEVANCE FEEDBACK AND OTHER QUERY MODIFICATION TECHNIQUES

CHAPTER 12: BOOLEAN OPERATIONS

CHAPTER 13: HASHING ALGORITHMS

CHAPTER 14: RANKING ALGORITHMS

CHAPTER 15: EXTENDED BOOLEAN MODELS

CHAPTER 16: CLUSTERING ALGORITHMS

CHAPTER 17: SPECIAL-PURPOSE HARDWARE FOR INFORMATION RETRIEVAL

CHAPTER 18: PARALLEL INFORMATION RETRIEVAL ALGORITHMS


Intro to Algorithms: BIBLIOGRAPHY

 Return to Table of Contents

BIBLIOGRAPHY

[1] Milton Abramowitz and Irene A. Stegun, editors. Handbook of Mathematical Functions. Dover, 1965.

[2] G. M. and E. M. Landis. An algorithm for the organization of information. Soviet Mathematics Doklady, 3: 1259-1263, 1962.

[3] Leonard M. Adleman, Carl Pomerance, and Robert S. Rumely. On distinguishing prime numbers from composite numbers. Annals of Mathematics, 117: 173-206, 1983.

[4] Alfred V. Aho, John E. Hopcroft, and Jeffrey D. Ullman. The Design and Analysis of Computer Algorithms. Addison-Wesley, 1974.

[5] Alfred V. Aho, John E. Hopcroft, and Jeffrey D. Ullman. Data Structures and Algorithms. Addison-Wesley, 1983.

[6] Ravindra K. Ahuja, Kurt Mehlhorn, James B. Orlin, and Robert E. Tarjan. Faster algorithms for the shortest path problem. Technical Report 193, MIT Operations Research Center, 1988.

[7] Howard H. Aiken and Grace M. Hopper. The automatic sequence controlled calculator. In Brian Randell, editor, The Origins of Digital Computers, pages 203-222. Springer-Verlag, third edition, 1982.

[8] M. Ajtai, J. Komlós, and E. Szemerédi. An O(n log n) sorting network. InProceedings of the Fifteenth Annual ACM Symposium on Theory of Computing, pages 1-9, 1983.

[9] Selim G. Akl. The Design and Analysis of Parallel Algorithms. Prentice-Hall, 1989.

[10] Richard J. Anderson and Gary L. Miller. Deterministic parallel list ranking. In John H. Reif, editor, 1988 Aegean Workshop on Computing, volume 319 of Lecture Notes in Computer Science, pages 81-90. Springer-Verlag, 1988.

[11] Richard J. Anderson and Gary L. Miller. A simple randomized parallel algorithm for list-ranking. Unpublished manuscript, 1988.

[12] Tom M. Apostol. Calculus, volume 1. Blaisdell Publishing Company, second edition, 1967.

[13] A. J. Atrubin. A one-dimensional real-time iterative multiplier. IEEE Transactions on Electronic Computers, EC-14(1):394-399, 1965.

[14] Sara Baase. Computer Algorithms: Introduction to Design and Analysis. Addison-Wesley, second edition, 1988.

[15] Eric Bach. Private communication, 1989.

[16] Eric Bach. Number-theoretic algorithms. In Annual Review of Computer Science, volume 4, pages 119- 172. Annual Reviews, Inc., 1990.

[17] R. Bayer. Symmetric binary B-trees: Data structure and maintenance algorithms. Acta Informatica, 1:290-306, 1972.

[18] R. Bayer and E. M. McCreight. Organization and maintenance of large ordered indexes. Acta Informatica, 1(3):173-189, 1972.

[19] Paul W. Beame, Stephen A. Cook, and H. James Hoover. Log depth circuits for division and related problems. SIAM Journal on Computing, 15(4):994-1003, 1986.

[20] Pierre Beauchemin, Gilles Brassard, Claude Crépeau, Claude Goutier, and Carl Pomerance. The generation of random numbers that are probably prime. Journal of Cryptology, 1:53-64, 1988.

[21] Richard Bellman. Dynamic Programming. Princeton University Press, 1957.

[22] Richard Bellman. On a routing problem. Quarterly of Applied Mathematics, 16(1):87-90, 1958.

[23] Michael Ben-Or. Lower bounds for algebraic computation trees. In Proceedings of the Fifteenth Annual ACM Symposium on Theory of Computing, pages 80-86, 1983.

[24] Jon L. Bentley. Writing Efficient Programs. Prentice-Hall, 1982.

[25] Jon L. Bentley. Programming Pearls. Addison-Wesley, 1986.

[26] Jon L. Bentley, Dorothea Haken, and James B. Saxe. A general method for solving divide-and-conquer recurrences. SIGACT News, 12(3):36-44, 1980.

[27] William H. Beyer, editor. CRC Standard Mathematical Tables. The Chemical Rubber Company, 1984.

[28] Patrick Billingsley. Probability and Measure. John Wiley & Sons, second edition, 1986.

[29] Manuel Blum, Robert W. Floyd, Vaughan Pratt, Ronald L. Rivest, and Robert E. Tarjan. Time bounds for selection. Journal of Computer and System Sciences, 7(4):448-461, 1973.

[30] Béla Bollobás. Random Graphs. Academic Press, 1985.

[31] J. A. Bondy and U. S. R. Murty. Graph Theory with Applications. American Elsevier, 1976.

[32] Robert S. Boyer and J. Strother Moore. A fast string-searching algorithm. Communications of the ACM, 20(10):762-772, 1977.

[33] Gilles Brassard and Paul Bratley. Algorithmics: Theory and Practice. Prentice-Hall, 1988.

[34] Richard P. Brent. The parallel evaluation of general arithmetic expressions. Journal of the ACM, 21(2):201-206, 1974.

[35] Richard P. Brent. An improved Monte Carlo factorization algorithm. BIT, 20(2):176-184, 1980.

[36] Mark R. Brown. The Analysis of a Practical and Nearly Optimal Priority Queue. PhD thesis, Computer Science Department, Stanford University, 1977. Technical Report STAN-CS-77-600.

[37] Mark R. Brown. Implementation and analysis of binomial queue algorithms. SIAM Journal on Computing, 7(3):298-319, 1978.

[38] Arthur W. Burks, editor. Theory of Self-Reproducing Automata. University of Illinois Press, 1966.

[39] Joseph J. F. Cavanagh. Digital Computer Arithmetic. McGraw-Hill, 1984.

[40] H. Chernoff. A measure of asymptotic efficiency for tests of a hypothesis based on the sum of observations. Annals of Mathematical Statistics, 23:493-507, 1952.

[41] Kai Lai Chung. Elementary Probability Theory with Stochastic Processes. Springer-Verlag, 1974.

[42] V. Chvátal. A greedy heuristic for the set-covering problem. Mathematics of Operations Research, 4(3):233-235, 1979.

[43] V. Chvátal, D. A. Klarner, and D. E. Knuth. Selected combinatorial research problems. Technical Report STAN-CS-72-292, Computer Science Department, Stanford University, 1972.

[44] Alan Cobham. The intrinsic computational difficulty of functions. In Proceedings of the 1964 Congress for Logic, Methodology, and the Philosophy of Science, pages 24-30. North-Holland, 1964.

[45] H. Cohen and H. W. Lenstra, Jr. Primality testing and jacobi sums. Mathematics of Computation, 42(165):297-330, 1984.

[46] Richard Cole. Parallel merge sort. In Proceedings of the 27th Annual Symposium on Foundations of Computer Science, pages 511-516. IEEE Computer Society, 1986.

[47] Richard Cole and Uzi Vishkin. Deterministic coin tossing with applications to optimal parallel list ranking. Information and Control, 70(1):32-53, 1986.

[48] D. Comer. The ubiquitous B-tree. ACM Computing Surveys, 11(2):121-137, 1979.

[49] Stephen Cook. The complexity of theorem proving procedures. In Proceedings of the Third Annual ACM Symposium on Theory of Computing, pages 151-158, 1971.

[50] Stephen Cook, Cynthia Dwork, and Rüdiger Reischuk. Upper and lower time bounds for parallel random access machines without simultaneous writes. SIAM Journal on Computing, 15(1):87-97, 1986.

[51] James W. Cooley and John W. Tukey. An algorithm for the machine calculation of complex Fourier series. Mathematics of Computation, 19(90): 297-301, April 1965.

[52] Don Coppersmith and Shmuel Winograd. Matrix multiplication via arithmetic progressions. In Proceedings of the Nineteenth Annual ACM Symposium on Theory of Computing, pages 1-6, 1987.

[53] George B. Dantzig. Linear Programming and Extensions. Princeton University Press, 1963.

[54] Whitfield Diffie and Martin E. Hellman. New directions in cryptography. IEEE Transactions on Information Theory, IT-22(6):644-654, 1976.

[55] E. W. Dijkstra. A note on two problems in connexion with graphs. Numerische Mathematik, 1:269-271, 1959.

[56] John D. Dixon. Factorization and primality tests. The American Mathematical Monthly, 91(6):333-352, 1984.

[57] Alvin W. Drake. Fundamentals of Applied Probability Theory. McGrawHill, 1967.

[58] James R. Driscoll, Harold N. Gabow, Ruth Shrairman, and Robert E. Tarjan. Relaxed heaps: An alternative to Fibonacci heaps with applications to parallel computation. Communications of the ACM, 31(11):1343-1354, 1988.

[59] James R. Driscoll, Neil Sarnak, Daniel D. Sleator, and Robert E. Tarjan. Making data structures persistent. In Proceedings of the Eighteenth Annual ACM Symposium on Theory of Computing, pages 109-121, 1986.

[60] Herbert Edelsbrunner. Algorithms in Combinatorial Geometry, volume 10 of EATCS Monographs on Theoretical Computer Science. Springer-Verlag, 1987.

[61] Jack Edmonds. Paths, trees, and flowers. Canadian Journal of Mathematics, 17:449-467, 1965.

[62] Jack Edmonds. Matroids and the greedy algorithm. Mathematical Programming, 1:126-136, 1971.

[63] Jack Edmonds and Richard M. Karp. Theoretical improvements in the algorithmic efficiency for network flow problems. Journal of the ACM, 19:248-264, 1972.

[64] G. Estrin, B. Gilchrist, and J. H. Pomerene. A note on high-speed digital multiplication. IRE Transactions on Electronic Computers, 5(3):140, 1956.

[65] Shimon Even. Graph Algorithms. Computer Science Press, 1979.

[66] William Feller. An Introduction to Probability Theory and Its Applications. John Wiley & Sons, third edition, 1968.

[67] M. J. Fischer and A. R. Meyer. Boolean matrix multiplication and transitive closure. In Proceedings of the Twelfth Annual Symposium on Switching and Automata Theory, pages 129-131. IEEE Computer Society, 1971.

[68] Robert W. Floyd. Algorithm 97 (SHORTEST PATH). Communications of the ACM, 5(6):345, 1962.

[69] Robert W. Floyd. Algorithm 245 (TREESORT). Communications of the ACM, 7:701, 1964.

[70] Robert W. Floyd and Ronald L. Rivest. Expected time bounds for selection. Communications of the ACM, 18(3):165-172, 1975.

[71] Lestor R. Ford, Jr., and D. R. Fulkerson. Flows in Networks. Princeton University Press, 1962.

[72] Lestor R. Ford, Jr., and Selmer M. Johnson. A tournament problem. The American Mathematical Monthly, 66:387-389, 1959.

[73] Steven Fortune and James Wyllie. Parallelism in random access machines. In Proceedings of the Tenth Annual ACM Symposium on Theory of Computing, pages 114-118, 1978.

[74] Michael L. Fredman and Michael E. Saks. The cell probe complexity of dynamic data structures. In Proceedings of the Twenty First Annual ACM Symposium on Theory of Computing, 1989.

[75] Michael L. Fredman and Robert E. Tarjan. Fibonacci heaps and their uses in improved network optimization algorithms. Journal of the ACM, 34(3):596-615, 1987.

[76] Harold N. Gabow and Robert E. Tarjan. A linear-time algorithm for a special case of disjoint set union. Journal of Computer and System Sciences, 30(2):209-221, 1985.

[77] Harold N. Gabow and Robert E. Tarjan. Faster scaling algorithms for network problems. SIAM Journal on Computing, 18(5):1013-1036, 1989.

[78] Zvi Galil and Joel Seiferas. Time-space-optimal string matching. Journal of Computer and System Sciences, 26(3):280-294, 1983.

[79] Michael R. Garey and David S. Johnson. Computers and Intractability: A Guide to the Theory of NP-Completeness. W. H. Freeman, 1979.

[80] [image] Gavril. Algorithms for minimum coloring, maximum clique, minimum covering by cliques, and maximum independent set of a chordal graph. SIAM Journal on Computing, 1(2):180-187, 1972.

[81] Alan George and Joseph W-H Liu. Computer Solution of Large Sparse Positive Definite Systems. Prentice-Hall, 1981.

[82] Andrew V. Goldberg. Efficient Graph Algorithms for Sequential and Parallel Computers. PhD thesis, Department of Electrical Engineering and Computer Science, MIT, 1987.

[83] Andrew V. Goldberg, ...va Tardos, and Robert E. Tarjan. Network flow algorithms. Technical Report STAN-CS-89-1252, Computer Science Department, Stanford University, 1989.

[84] Andrew V. Goldberg and Serge A. Plotkin. Parallel ( [image] + 1) coloring of constant-degree graphs. Information Processing Letters, 25(4):241-245, 1987.

[85] Andrew V. Goldberg and Robert E. Tarjan. A new approach to the maximum flow problem. In Proceedings of the Eighteenth Annual ACM Symposium on Theory of Computing, pages 136-146, 1986.

[86] Shafi Goldwasser and Silvio Micali. Probabilistic encryption. Journal of Computer and System Sciences, 28(2):270-299, 1984.

[87] Shafi Goldwasser, Silvio Micali, and Charles Rackoff. The knowledge complexity of interactive proof systems. SIAM Journal on Computing, 18(1):186-208, 1989.

[88] Shafi Goldwasser, Silvio Micali, and Ronald L. Rivest. A digital signature scheme secure against adaptive chosen-message attacks. SIAM Journal on Computing, 17(2):281-308, 1988.

[89] Gene H. Golub and Charles F. Van Loan. Matrix Computations. The Johns Hopkins University Press, 1983.

[90] G. H. Gonnet. Handbook of Algorithms and Data Structures. Addison-Wesley, 1984.

[91] R. L. Graham. An efficient algorithm for determining the convex hull of a finite planar set. Information Processing Letters, 1:132-133, 1972.

[92] R. L. Graham and Pavol Hell. On the history of the minimum spanning tree problem. Annals of the History of Computing, 7(1):43-57, 1985.

[93] Leo J. Guibas and Robert Sedgewick. A diochromatic framework for balanced trees. In Proceedings of the 19th Annual Symposium on Foundations of Computer Science, pages 8-21. IEEE Computer Society, 1978.

[94] Frank Harary. Graph Theory. Addison-Wesley, 1969.

[95] J. Hartmanis and R. E. Stearns. On the computational complexity of algorithms. Transactions of the American Mathematical Society, 117:285-306, 1965.

[96] Frederick J. Hill and Gerald R. Peterson. Introduction to Switching Theory and Logical Design. John Wiley & Sons, second edition, 1974.

[97] C. A. R. Hoare. Algorithm 63 (partition) and algorithm 65 (find). Communications of the ACM, 4(7):321-322, 1961.

[98] C. A. R. Hoare. Quicksort. Computer Journal, 5(1):10-15, 1962.

[99] W. Hoeffding. On the distribution of the number of successes in independent trials. Annals of Mathematical Statistics, 27:713-721, 1956.

[100] Micha Hofri. Probabilistic Analysis of Algorithms. Springer-Verlag, 1987.

[101] John E. Hopcroft and Richard M. Karp. An n5/2 algorithm for maximum matchings in bipartite graphs. SIAM Journal on Computing, 2(4):225-231, 1973.

[102] John E. Hopcroft and Robert E. Tarjan. Efficient algorithms for graph manipulation. Communications of the ACM, 16(6):372-378, 1973.

[103] John E. Hopcroft and Jeffrey D. Ullman. Set merging algorithms. SIAM Journal on Computing, 2(4):294-303, 1973.

[104] John E. Hopcroft and Jeffrey D. Ullman. Introduction to Automata Theory, Languages, and Computation. Addison-Wesley, 1979.

[105] Ellis Horowitz and Sartaj Sahni. Fundamentals of Computer Algorithms. Computer Science Press, 1978.

[106] T. C. Hu and M. T. Shing. Some theorems about matrix multiplication. In Proceedings of the 21st Annual Symposium on Foundations of Computer Science, pages 28-35. IEEE Computer Society, 1980.

[107] David A. Huffman. A method for the construction of minimum-redundancy codes. Proceedings of the IRE, 40(9):1098-1101, 1952.

[108] Kai Hwang. Computer Arithmetic: Principles, Architecture, and Design. John Wiley & Sons, 1979.

[109] Kai Hwang and Fayé A. Briggs. Computer Architecture and Parallel Processing. McGraw-Hill, 1984.

[110] Kai Hwang and Doug DeGroot. Parallel Processing for Supercomputers and Artificial Intelligence. McGraw-Hill, 1989.

[111] Oscar H. Ibarra and Chul E. Kim. Fast approximation algorithms for the knapsack and sum of subset problems. Journal of the ACM, 22(4):463-468, 1975.

[112] R. A. Jarvis. On the identification of the convex hull of a finite set of points in the plane. Information Processing Letters, 2:18-21, 1973.

[113] D. S. Johnson. Approximation algorithms for combinatorial problems. Journal of Computer and System Sciences, 9:256-278, 1974.

[114] Donald B. Johnson. Efficient algorithms for shortest paths in sparse networks. Journal of the ACM, 24(1):1-13, 1977.

[115] N. Karmarkar. A new polynomial-time algorithm for linear programming. Combinatorica, 4(4):373-395, 1984.

[116] Richard M. Karp. Reducibility among combinatorial problems. In Raymond E. Miller and James W. Thatcher, editors, Complexity of Computer Computations, pages 85-103. Plenum Press, 1972.

[117] Richard M. Karp and Michael O. Rabin. Efficient randomized pattern-matching algorithms. Technical Report TR-31-81, Aiken Computation Laboratory, Harvard University, 1981.

[118] Richard M. Karp and Vijaya Ramachandran. A survey of parallel algorithms for shared-memory machines. Technical Report UCB/CSD 88/408, Computer Science Division (EECS), University of California, Berkeley, 1988.

[119] A. V. Karzanov. Determining the maximal flow in a network by the method of preflows. Soviet Mathematics Doklady, 15:434-437, 1974.

[120] D. G. Kirkpatrick and R. Seidel. The ultimate planar convex hull algorithm? SIAM Journal on Computing, 15(2):287-299, 1986.

[121] Donald E. Knuth. Fundamental Algorithms, volume 1 of The Art of Computer Programming. Addison-Wesley, 1968. Second edition, 1973.

[122] Donald E. Knuth. Seminumerical Algorithms, volume 2 of The Art of Computer Programming. Addison-Wesley, 1969. Second edition, 1981.

[123] Donald E. Knuth. Sorting and Searching, volume 3 of The Art of Computer Programming. Addison-Wesley, 1973.

[124] Donald E. Knuth. Big omicron and big omega and big theta. ACM SIGACT News, 8(2):18-23, 1976.

[125] Donald E. Knuth, James H. Morris, Jr., and Vaughan R. Pratt. Fast pattern matching in strings. SIAM Journal on Computing, 6(2):323-350, 1977.

[126] Zvi Kohavi. Switching and Finite Automata Theory. McGraw-Hill, 1970.

[127] Bernhard Korte and Lászlo Lovász. Mathematical structures underlying greedy algorithms. In F. Gecseg, editor, Fundamentals of Computation Theory, number 117 in Lecture Notes in Computer Science, pages 205-209. Springer-Verlag, 1981.

[128] Bernhard Korte and Lászlo Lovász. Structural properties of greedoids. Combinatorica, 3:359-374, 1983.

[129] Bernhard Korte and Lászlo Lovász. Greedoids--a structural framework for the greedy algorithm. In W. Pulleybank, editor, Progress in Combinatorial Optimization, pages 221-243. Academic Press, 1984.

[130] Bernhard Korte and Lászlo Lovász. Greedoids and linear objective functions. SIAM Journal on Algebraic and Discrete Methods, 5(2):229-238, 1984.

[131] J. B. Kruskal. On the shortest spanning subtree of a graph and the traveling salesman problem. Proceedings of the American Mathematical Society, 7:48-50, 1956.

[132] Eugene L. Lawler. Combinatorial Optimization: Networks and Matroids. Holt, Rinehart, and Winston, 1976.

[133] Eugene L. Lawler, J. K. Lenstra, A. H. G. Rinnooy Kan, and D. B. Shmoys, editors. The Traveling Salesman Problem. John Wiley & Sons, 1985.

[134] C. Y. Lee. An algorithm for path connection and its applications. IRE Transactions on Electronic Computers, EC-10(3):346-365, 1961.

[135] F. Thomson Leighton. Introduction to Parallel Algorithms and Architectures: Networks and Algorithms. Morgan-Kaufmann, in preparation.

[136] Debra A. Lelewer and Daniel S. Hirschberg. Data compression. ACM Computing Surveys, 19(3):261-296, 1987.

[137] H. W. Lenstra, Jr. Factoring integers with elliptic curves. Annals of Mathematics, 126:649-673, 1987.

[138] L. A. Levin. Universal sorting problems. Problemy Peredachi Informatsii, 9(3):265-266, 1973. In Russian.

[139] Harry R. Lewis and Christos H. Papadimitriou. Elements of the Theory of Computation. Prentice-Hall, 1981.

[140] C. L. Liu. Introduction to Combinatorial Mathematics. McGraw-Hill, 1968.

[141] Lászlo Lovász. On the ratio of optimal integral and fractional covers. Discrete Mathematics, 13:383-390, 1975.

[142] Udi Manber. Introduction to Algorithms: A Creative Approach. Addison-Wesley, 1989.

[143] William J. Masek and Michael S. Paterson. A faster algorithm computing string edit distances. Journal of Computer and System Sciences, 20(1):18-31, 1980.

[144] Kurt Mehlhorn. Sorting and Searching, volume 1 of Data Structures and Algorithms. Springer-Verlag, 1984.

[145] Kurt Mehlhorn. Graph Algorithms and NP-Completeness, volume 2 of Data Structures and Algorithms. Springer-Verlag, 1984.

[146] Kurt Mehlhorn. Multidimensional Searching and Computational Geometry, volume 3 of Data Structures and Algorithms. Springer-Verlag, 1984.

[147] Gary L. Miller. Riemann's hypothesis and tests for primality. Journal of Computer and System Sciences, 13(3):300-317, 1976.

[148] Louis Monier. Algorithmes de Factorisation D'Entiers. PhD thesis, L'Université Paris-Sud, Centre D'Orsay, 1980.

[149] Louis Monier. Evaluation and comparison of two efficient probabilistic primality testing algorithms. Theoretical Computer Science, 12(1): 97-108, 1980.

[150] Edward F. Moore. The shortest path through a maze. In Proceedings of the International Symposium on the Theory of Switching, pages 285-292. Harvard University Press, 1959.

[151] Ivan Niven and Herbert S. Zuckerman. An Introduction to the Theory of Numbers. John Wiley & Sons, fourth edition, 1980.

[152] Yu. Ofman. On the algorithmic complexity of discrete functions. Soviet Physics Doklady, 7(7):589-591, 1963. English translation.

[153] Alan V. Oppenheim and Alan S. Willsky, with Ian T. Young. Signals and Systems. Prentice-Hall, 1983.

[154] Christos H. Papadimitriou and Kenneth Steiglitz. Combinatorial Optimization: Algorithms and Complexity. Prentice-Hall, 1982.

[155] Michael S. Paterson, 1974. Unpublished lecture, Ile de Berder, France.

[156] J. M. Pollard. A Monte Carlo method for factorization. BIT, 15:331-334, 1975.

[157] Carl Pomerance. On the distribution of pseudoprimes. Mathematics of Computation, 37(156):587-593, 1981.

[158] Carl Pomerance. The quadratic sieve factoring algorithm. In T. Beth, N. Cot, and I. Ingemarrson, editors, Advances in Cryptology, volume 209 of Lecture Notes in Computer Science, pages 169-182. Springer-Verlag, 1984.

[159] Carl Pomerance, editor. Proceedings of the AMS Symposia in Applied Mathematics: Computational Number Theory and Cryptography. American Mathematical Society, to appear.

[160] Franco P. Preparata and Micheal Ian Shamos. Computational Geometry: An Introduction. Springer-Verlag, 1985.

[161] William H. Press, Brian P. Flannery, Saul A. Teukolsky, and William T. Vetterling. Numerical Recipes: The Art of Scientific Computing. Cambridge University Press, 1986.

[162] William H. Press, Brian P. Flannery, Saul A. Teukolsky, and William T. Vetterling. Numerical Recipes in C. Cambridge University Press, 1988.

[163] R. C. Prim. Shortest connection networks and some generalizations. Bell System Technical Journal, 36:1389-1401, 1957.

[164] Paul W. Purdom, Jr., and Cynthia A. Brown. The Analysis of Algorithms. Holt, Rinehart, and Winston, 1985.

[165] Michael O. Rabin. Probabilistic algorithms. In J. F. Traub, editor, Algorithms and Complexity: New Directions and Recent Results, pages 21-39. Academic Press, 1976.

[166] Michael O. Rabin. Probabilistic algorithm for testing primality. Journal of Number Theory, 12:128-138, 1980.

[167] Edward M. Reingold, Jürg Nievergelt, and Narsingh Deo. Combinatorial Algorithms: Theory and Practice. Prentice-Hall, 1977.

[168] Hans Riesel. Prime Numbers and Computer Methods for Factorization. Progress in Mathematics. Birkhäuser, 1985.

[169] Ronald L. Rivest, Adi Shamir, and Leonard M. Adleman. A method for obtaining digital signatures and public-key cryptosystems. Communications of the ACM, 21(2):120-126, 1978. See also U.S. Patent 4,405,829.

[170] D. J. Rosenkrantz, R. E. Stearns, and P. M. Lewis. An analysis of several heuristics for the traveling salesman problem. SIAM Journal on Computing, 6:563-581, 1977.

[171] Y. A. Rozanov. Probability Theory: A Concise Course. Dover, 1969.

[172] S. Sahni and T. Gonzalez. P-complete approximation problems. Journal of the ACM, 23:555-565, 1976.

[173] John E. Savage. The Complexity of Computing. John Wiley & Sons, 1976.

[174] Robert Sedgewick. Implementing quicksort programs. Communications of the ACM, 21(10):847-857, 1978.

[175] Robert Sedgewick. Algorithms. Addison-Wesley, second edition, 1988.

[176] Michael I. Shamos and Dan Hoey. Geometric intersection problems. In Proceedings of the 16th Annual Symposium on Foundations of Computer Science, pages 208-215. IEEE Computer Society, 1975.

[177] Daniel D. Sleator and Robert E. Tarjan. A data structure for dynamic trees. Journal of Computer and System Sciences, 26(3):362-391, 1983.

[178] Daniel D. Sleator and Robert E. Tarjan. Self-adjusting binary search trees. Journal of the ACM, 32(3):652-686, 1985.

[179] Joel Spencer. Ten Lectures on the Probabilistic Method. Regional Conference Series on Applied Mathematics (No. 52). SIAM, 1987.

[180] Staff of the Computation Laboratory. Description of a Relay Calculator, volume XXIV of The Annals of the Computation Laboratory of Harvard University. Harvard University Press, 1949.

[181] Gilbert Strang. Introduction to Applied Mathematics. Wellesley-Cambridge Press, 1986.

[182] Gilbert Strang. Linear Algebra and Its Applications. Harcourt Brace Jovanovich, third edition, 1988.

[183] Volker Strassen. Gaussian elimination is not optimal. Numerische Mathematik, 14(3):354-356, 1969.

[184] T. G. Szymanski. A special case of the maximal common subsequence problem. Technical Report TR-170, Computer Science Laboratory, Princeton University, 1975.

[185] Robert E. Tarjan. Depth first search and linear graph algorithms. SIAM Journal on Computing, 1(2):146-160, 1972.

[186] Robert E. Tarjan. Efficiency of a good but not linear set union algorithm. Journal of the ACM, 22(2):215-225, 1975.

[187] Robert E. Tarjan. A class of algorithms which require nonlinear time to maintain disjoint sets. Journal of Computer and System Sciences, 18(2):110-127, 1979.

[188] Robert E. Tarjan. Data Structures and Network Algorithms. Society for Industrial and Applied Mathematics, 1983.

[189] Robert E. Tarjan. Amortized computational complexity. SIAM Journal on Algebraic and Discrete Methods, 6(2):306-318, 1985.

[190] Robert E. Tarjan and Jan van Leeuwen. Worst-case analysis of set union algorithms. Journal of the ACM, 31(2):245-281, 1984.

[191] Robert E. Tarjan and Uzi Vishkin. An efficient parallel biconnectivity algorithm. SIAM Journal on Computing, 14(4):862-874, 1985.

[192] George B. Thomas, Jr., and Ross L. Finney. Calculus and Analytic Geometry. Addison-Wesley, seventh edition, 1988.

[193] Leslie G. Valiant. Parallelism in comparison problems. SIAM Journal on Computing, 4(3):348-355, 1975.

[194] P. van Emde Boas. Preserving order in a forest in less than logarithmic time. In Proceedings of the 16th Annual Symposium on Foundations of Computer Science pages 75-84. IEEE Computer Society, 1975.

[195] Uzi Vishkin. Implementation of simultaneous memory address access in models that forbid it. Journal of Algorithms, 4(1):45-50, 1983.

[196] Jean Vuillemin. A data structure for manipulating priority queues. Communications of the ACM, 21(4):309-315, 1978.

[197] C. S. Wallace. A suggestion for a fast multiplier. IEEE Transactions on Electronic Computers, EC-13(1):14-17, 1964.

[198] Stephen Warshall. A theorem on boolean matrices. Journal of the ACM, 9(1):11-12, 1962.

[199] A. Weinberger and J. L. Smith. A one-microsecond adder using one-megacycle circuitry. IRE Transactions on Electronic Computers, EC-5(2), 1956.

[200] Hassler Whitney. On the abstract properties of linear dependence. American Journal of Mathematics, 57:509-533, 1935.

[201] Herbert S. Wilf. Algorithms and Complexity. Prentice-Hall, 1986.

[202] J. W. J. Williams. Algorithm 232 (heapsort). Communications of the ACM, 7:347-348, 1964.

[203] S. Winograd. On the algebraic complexity of functions. In Actes du Congrès International des Mathèmaticiens, volume 3, pages 283-288, 1970.

[204] James C. Wyllie. The Complexity of Parallel Computations. PhD thesis, Department of Computer Science, Cornell University, 1979.

[205] Andrew C.-C. Yao. A lower bound to finding convex hulls. Journal of the ACM, 28(4):780-787, 1981.

Back to Table of Contents


Intro to Algorithms: CHAPTER 1: INTRODUCTION

 Return to Table of Contents

CHAPTER 1: INTRODUCTION

This chapter will familiarize you with the framework we shall use throughout the book to think about the design and analysis of algorithms. It is self-contained, but it does include several references to material that will be introduced in Part I.

We begin with a discussion of computational problems in general and of the algorithms needed to solve them, with the problem of sorting as our running example. We introduce a "pseudocode" that should be familiar to readers who have done computer programming to show how we shall specify our algorithms. Insertion sort, a simple sorting algorithm, serves as an initial example. We analyze the running time of insertion sort, introducing a notation that focuses on how that time increases with the number of items to be sorted. We also introduce the divide-and-conquer approach to the design of algorithms and use it to develop an algorithm called merge sort. We end with a comparison of the two sorting algorithms.

1.1 Algorithms

Informally, an algorithm is any well-defined computational procedure that takes some value, or set of values, as input and produces some value, or set of values, as output. An algorithm is thus a sequence of computational steps that transform the input into the output.

We can also view an algorithm as a tool for solving a well-specified computational problem. The statement of the problem specifies in general terms the desired input/output relationship. The algorithm describes a specific computational procedure for achieving that input/output relationship.

We begin our study of algorithms with the problem of sorting a sequence of numbers into nondecreasing order. This problem arises frequently in practice and provides fertile ground for introducing many standard design techniques and analysis tools. Here is how we formally define the sorting problem:

Input: A sequence of n numbers [image] a1, a2, . . . , an [image] .

Output: A permutation (reordering) [image] of the input sequence such that [image].

Given an input sequence such as [image] 31, 41, 59, 26, 41, 58 [image], a sorting algorithm returns as output the sequence [image] 26, 31, 41, 41, 58, 59 [image]. Such an input sequence is called an instance of the sorting problem. In general, an instance of a problem consists of all the inputs (satisfying whatever constraints are imposed in the problem statement) needed to compute a solution to the problem.

Sorting is a fundamental operation in computer science (many programs use it as an intermediate step), and as a result a large number of good sorting algorithms have been developed. Which algorithm is best for a given application depends on the number of items to be sorted, the extent to which the items are already somewhat sorted, and the kind of storage device to be used: main memory, disks, or tapes.

An algorithm is said to be correct if, for every input instance, it halts with the correct output. We say that a correct algorithm solves the given computational problem. An incorrect algorithm might not halt at all on some input instances, or it might halt with other than the desired answer. Contrary to what one might expect, incorrect algorithms can sometimes be useful, if their error rate can be controlled. We shall see an example of this in Chapter 33 when we study algorithms for finding large prime numbers. Ordinarily, however, we shall be concerned only with correct algorithms.

An algorithm can be specified in English, as a computer program, or even as a hardware design. The only requirement is that the specification must provide a precise description of the computational procedure to be followed.

In this book, we shall typically describe algorithms as programs written in a pseudocode that is very much like C, Pascal, or Algol. If you have been introduced to any of these languages, you should have little trouble reading our algorithms. What separates pseudocode from "real" code is that in pseudocode, we employ whatever expressive method is most clear and concise to specify a given algorithm. Sometimes, the clearest method is English, so do not be surprised if you come across an English phrase or sentence embedded within a section of "real" code. Another difference between pseudocode and real code is that pseudocode is not typically concerned with issues of software engineering. Issues of data abstraction, modularity, and error handling are often ignored in order to convey the essence of the algorithm more concisely.

Insertion sort

We start with insertion sort, which is an efficient algorithm for sorting a small number of elements. Insertion sort works the way many people sort a bridge or gin rummy hand. We start with an empty left hand and the cards face down on the table. We then remove one card at a time from the table and insert it into the correct position in the left hand. To find the correct position for a card, we compare it with each of the cards already in the hand, from right to left, as illustrated in Figure 1.1.

Figure 1.1 Sorting a hand of cards using insertion sort.

Our pseudocode for insertion sort is presented as a procedure called INSERTION-SORT, which takes as a parameter an array A[1 . . n] containing a sequence of length n that is to be sorted. (In the code, the number n of elements in A is denoted by length[A].) The input numbers are sorted in place: the numbers are rearranged within the array A, with at most a constant number of them stored outside the array at any time. The input array A contains the sorted output sequence when INSERTION-SORT is finished.

INSERTION-SORT (A)

1  for j    2 to length[A]

2       do key    A[j]

3           Insert A[j] into the sorted sequence A[1 . . j - 1].

4        i    j - 1

5        while i > 0 and A[i] > key

6           do A[i + 1]    A[i]

7              i    i - 1

8        A[i + 1]    key

Figure 1.2 shows how this algorithm works for A = [image] 5, 2, 4, 6, 1, 3 [image]. The index j indicates the "current card" being inserted into the hand. Array elements A[1.. j - 1] constitute the currently sorted hand, and elements A[j + 1 . . n] correspond to the pile of cards still on the table. The index j moves left to right through the array. At each iteration of the "outer" for loop, the element A[j] is picked out of the array (line 2). Then, starting in position j - 1, elements are successively moved one position to the right until the proper position for A[j] is found (lines 4-7), at which point it is inserted (line 8).

Figure 1.2 The operation of INSERTION-SORT on the array A = 5, 2, 4, 6, 1, 3 . The position of index j is indicated by a circle.

Pseudocode conventions

We use the following conventions in our pseudocode.

1. Indentation indicates block structure. For example, the body of the for loop that begins on line 1 consists of lines 2-8, and the body of the while loop that begins on line 5 contains lines 6-7 but not line 8. Our indentation style applies to if-then-else statements as well. Using indentation instead of conventional indicators of block structure, such as begin and end statements, greatly reduces clutter while preserving, or even enhancing, clarity.1

1 In real programming languages, it is generally not advisable to use indentation alone to indicate block structure, since levels of indentation are hard to determine when code is split across pages.

2. The looping constructs while, for, and repeat and the conditional constructs if, then, and else have the the same interpretation as in Pascal.

3. The symbol [image] indicates that the remainder of the line is a comment.

4. A multiple assignment of the form i [image] j [image] e assigns to both variables i and j the value of expression e; it should be treated as equivalent to the assignment j [image] e followed by the assignment i [image] j.

5. Variables (such as i, j, and key) are local to the given procedure. We shall not use global variables without explicit indication.

6. Array elements are accessed by specifying the array name followed by the index in square brackets. For example, A[i] indicates the ith element of the array A. The notation ". ." is used to indicate a range of values within an array. Thus, A[1. . j] indicates the subarray of A consisting of elements A[1], A[2], . . . , A[j].

7. Compound data are typically organized into objects, which are comprised of attributes or fields. A particular field is accessed using the field name followed by the name of its object in square brackets. For example, we treat an array as an object with the attribute length indicating how many elements it contains. To specify the number of elements in an array A, we write length[A]. Although we use square brackets for both array indexing and object attributes, it will usually be clear from the context which interpretation is intended.

A variable representing an array or object is treated as a pointer to the data representing the array or object. For all fields f of an object x, setting y [image] x causes f[y] = f[x]. Moreover, if we now set f[x] [image] 3, then afterward not only is f[x] = 3, but f[y] = 3 as well. In other words, x and y point to ("are") the same object after the assignment y [image] x.

Sometimes, a pointer will refer to no object at all. In this case, we give it the special value NIL.

8. Parameters are passed to a procedure by value: the called procedure receives its own copy of the parameters, and if it assigns a value to a parameter, the change is not seen by the calling routine. When objects are passed, the pointer to the data representing the object is copied, but the object's fields are not. For example, if x is a parameter of a called procedure, the assignment x [image] y within the called procedure is not visible to the calling procedure. The assignment f[x] [image] 3, however, is visible.

Exercises

1.1-1

Using Figure 1.2 as a model, illustrate the operation of INSERTION-SORT on the array A = [image] 31, 41, 59, 26, 41, 58 [image].

1.1-2

Rewrite the INSERTION-SORT procedure to sort into nonincreasing instead of nondecreasing order.

1.1-3

Consider the searching problem:

Input: A sequence of n numbers A = [image] a1, a2, . . . ,an [image] and a value v.

Output: An index i such that v = A[i] or the special value NIL if v does not appear in A.

Write pseudocode for linear search, which scans through the sequence, looking for v.

1.1-4

Consider the problem of adding two n-bit binary integers, stored in two n-element arrays A and B. The sum of the two integers should be stored in binary form in an (n + 1)-element array C. State the problem formally and write pseudocode for adding the two integers.

1.2 Analyzing algorithms

Analyzing an algorithm has come to mean predicting the resources that the algorithm requires. Occasionally, resources such as memory, communication bandwidth, or logic gates are of primary concern, but most often it is computational time that we want to measure. Generally, by analyzing several candidate algorithms for a problem, a most efficient one can be easily identified. Such analysis may indicate more than one viable candidate, but several inferior algorithms are usually discarded in the process.

Before we can analyze an algorithm, we must have a model of the implementation technology that will be used, including a model for the resources of that technology and their costs. For most of this book, we shall assume a generic one-processor, random-access machine (RAM) model of computation as our implementation technology and understand that our algorithms will be implemented as computer programs. In the RAM model, instructions are executed one after another, with no concurrent operations. In later chapters, however, we shall have occasion to investigate models for parallel computers and digital hardware.

Analyzing even a simple algorithm can be a challenge. The mathematical tools required may include discrete combinatorics, elementary probability theory, algebraic dexterity, and the ability to identify the most significant terms in a formula. Because the behavior of an algorithm may be different for each possible input, we need a means for summarizing that behavior in simple, easily understood formulas.

Even though we typically select only one machine model to analyze a given algorithm, we still face many choices in deciding how to express our analysis. One immediate goal is to find a means of expression that is simple to write and manipulate, shows the important characteristics of an algorithm's resource requirements, and suppresses tedious details.

Analysis of insertion sort

The time taken by the INSERTION-SORT procedure depends on the input: sorting a thousand numbers takes longer than sorting three numbers. Moreover, INSERTION-SORT can take different amounts of time to sort two input sequences of the same size depending on how nearly sorted they already are. In general, the time taken by an algorithm grows with the size of the input, so it is traditional to describe the running time of a program as a function of the size of its input. To do so, we need to define the terms "running time" and "size of input" more carefully.

The best notion for input size depends on the problem being studied. For many problems, such as sorting or computing discrete Fourier transforms, the most natural measure is the number of items in the input--for example, the array size n for sorting. For many other problems, such as multiplying two integers, the best measure of input size is the total number of bits needed to represent the input in ordinary binary notation. Sometimes, it is more appropriate to describe the size of the input with two numbers rather than one. For instance, if the input to an algorithm is a graph, the input size can be described by the numbers of vertices and edges in the graph. We shall indicate which input size measure is being used with each problem we study.

The running time of an algorithm on a particular input is the number of primitive operations or "steps" executed. It is convenient to define the notion of step so that it is as machine-independent as possible. For the moment, let us adopt the following view. A constant amount of time is required to execute each line of our pseudocode. One line may take a different amount of time than another line, but we shall assume that each execution of the ith line takes time ci, where ci is a constant. This viewpoint is in keeping with the RAM model, and it also reflects how the pseudocode would be implemented on most actual computers.2

2There are some subtleties here. Computational steps that we specify in English are often variants of a procedure that requires more than just a constant amount of time. For example, later in this book we might say "sort the points by x-coordinate," which, as we shall see, takes more than a constant amount of time. Also, note that a statement that calls a subroutine takes constant time, though the subroutine, once invoked, may take more. That is, we separate the process of calling the subroutine--passing parameters to it, etc.--from the process of executing the subroutine.

In the following discussion, our expression for the running time of INSERTION-SORT will evolve from a messy formula that uses all the statement costs ci to a much simpler notation that is more concise and more easily manipulated. This simpler notation will also make it easy to determine whether one algorithm is more efficient than another.

We start by presenting the INSERTION-SORT procedure with the time "cost" of each statement and the number of times each statement is executed. For each j = 2, 3, . . . , n, where n = length[A], we let tj be the number of times the while loop test in line 5 is executed for that value of j. We assume that comments are not executable statements, and so they take no time.

The running time of the algorithm is the sum of running times for each statement executed; a statement that takes ci steps to execute and is executed n times will contribute ci n to the total running time.3 To compute T(n), the running time of INSERTION-SORT, we sum the products of the cost and times columns, obtaining

3This characteristic does not necessarily hold for a resource such as memory. A statement that references m words of memory and is executed n times does not necessarily consume mn words of memory in total.

Even for inputs of a given size, an algorithm's running time may depend on which input of that size is given. For example, in INSERTION-SORT, the best case occurs if the array is already sorted. For each j = 2, 3, . . . , n, we then find that A[i] [image] key in line 5 when i has its initial value of j - 1. Thus tj = 1 for j = 2,3, . . ., n, and the best-case running time is

T(n) = c1n + c2 (n - 1) + c4 (n - 1) + c5 (n - 1) + c8 (n - 1)

= (c1 + c2 + c4 + c8)n - (c2 + c4 + c5 + c8).

This running time can be expressed as an + b for constants a and b that depend on the statement costs ci; it is thus a linear function of n.

If the array is in reverse sorted order--that is, in decreasing order--the worst case results. We must compare each element A[j] with each element in the entire sorted subarray A[1. . j - 1], and so tj = j for j = 2,3, . . . , n. Noting that

and

(we shall review these summations in Chapter 3), we find that in the worst case, the running time of INSERTION-SORT is

This worst-case running time can be expressed as an2 + bn+ c for constants a, b, and c that again depend on the statement costs ci; it is thus a quadratic function of n.

Typically, as in insertion sort, the running time of an algorithm is fixed for a given input, although in later chapters we shall see some interesting "randomized" algorithms whose behavior can vary even for a fixed input.

Worst-case and average-case analysis

In our analysis of insertion sort, we looked at both the best case, in which the input array was already sorted, and the worst case, in which the input array was reverse sorted. For the remainder of this book, though, we shall usually concentrate on finding only the worst-case running time, that is, the longest running time for any input of size n. We give three reasons for this orientation.

The worst-case running time of an algorithm is an upper bound on the running time for any input. Knowing it gives us a guarantee that the algorithm will never take any longer. We need not make some educated guess about the running time and hope that it never gets much worse.

For some algorithms, the worst case occurs fairly often. For example, in searching a database for a particular piece of information, the searching algorithm's worst case will often occur when the information is not present in the database. In some searching applications, searches for absent information may be frequent.

The "average case" is often roughly as bad as the worst case. Suppose that we randomly choose n numbers and apply insertion sort. How long does it take to determine where in subarray A[1. . j - 1] to insert element A[j]? On average, half the elements in A[1. . j - 1] are less than A[j], and half the elements are greater. On average, therefore, we check half of the subarray A[1. . j - 1], so tj = j/2. If we work out the resulting average-case running time, it turns out to be a quadratic function of the input size, just like the worst-case running time.

In some particular cases, we shall be interested in the average-case or expected running time of an algorithm. One problem with performing an average-case analysis, however, is that it may not be apparent what constitutes an "average" input for a particular problem. Often, we shall assume that all inputs of a given size are equally likely. In practice, this assumption may be violated, but a randomized algorithm can sometimes force it to hold.

Order of growth

We have used some simplifying abstractions to ease our analysis of the INSERTION-SORT procedure. First, we ignored the actual cost of each statement, using the constants ci to represent these costs. Then, we observed that even these constants give us more detail than we really need: the worst-case running time is an2 + bn + c for some constants a, b, and c that depend on the statement costs ci. We thus ignored not only the actual statement costs, but also the abstract costs ci.

We shall now make one more simplifying abstraction. It is the rate of growth, or order of growth, of the running time that really interests us. We therefore consider only the leading term of a formula (e.g., an2), since the lower-order terms are relatively insignificant for large n. We also ignore the leading term's constant coefficient, since constant factors are less significant than the rate of growth in determining computational efficiency for large inputs. Thus, we write that insertion sort, for example, has a worst-case running time of [image] (n2) (pronounced "theta of n-squared"). We shall use [image] -notation informally in this chapter; it will be defined precisely in Chapter 2.

We usually consider one algorithm to be more efficient than another if its worst-case running time has a lower order of growth. This evaluation may be in error for small inputs, but for large enough inputs a [image] (n2) algorithm, for example, will run more quickly in the worst case than a [image] (n3) algorithm.

Exercises

1.2-1

Consider sorting n numbers stored in array A by first finding the smallest element of A and putting it in the first entry of another array B. Then find the second smallest element of A and put it in the second entry of B. Continue in this manner for the n elements of A. Write pseudocode for this algorithm, which is known as selection sort. Give the best-case and worst-case running times of selection sort in [image] -notation.

1.2-2

Consider linear search again (see Exercise 1.1-3). How many elements of the input sequence need to be checked on the average, assuming that the element being searched for is equally likely to be any element in the array? How about in the worst case? What are the average-case and worst-case running times of linear search in [image] -notation? Justify your answers.

1.2-3

Consider the problem of determining whether an arbitrary sequence [image] x1, x2, . . . , xn [image] of n numbers contains repeated occurrences of some number. Show that this can be done in [image] (n lg n) time, where lg n stands for log2 n.

1.2-4

Consider the problem of evaluating a polynomial at a point. Given n coefficients a0, a1, . . . , an - 1 and a real number x, we wish to compute [image]. Describe a straightforward [image] (n2)-time algorithm for this problem. Describe a [image] (n)-time algorithm that uses the following method (called Horner's rule) for rewriting the polynomial:

1.2-5

Express the function n3/1000 - 100n2 - 100n + 3 in terms of [image] -notation.

1.2-6

How can we modify almost any algorithm to have a good best-case running time?

1.3 Designing algorithms

There are many ways to design algorithms. Insertion sort uses an incremental approach: having sorted the subarray A[1 . . j - 1], we insert the single element A[j] into its proper place, yielding the sorted subarray A[1 . . j].

In this section, we examine an alternative design approach, known as "divide-and-conquer." We shall use divide-and-conquer to design a sorting algorithm whose worst-case running time is much less than that of insertion sort. One advantage of divide-and-conquer algorithms is that their running times are often easily determined using techniques that will be introduced in Chapter 4.

1.3.1 The divide-and-conquer approach

Many useful algorithms are recursive in structure: to solve a given problem, they call themselves recursively one or more times to deal with closely related subproblems. These algorithms typically follow a divide-and-conquer approach: they break the problem into several subproblems that are similar to the original problem but smaller in size, solve the subproblems recursively, and then combine these solutions to create a solution to the original problem.

The divide-and-conquer paradigm involves three steps at each level of the recursion:

Divide the problem into a number of subproblems.

Conquer the subproblems by solving them recursively. If the subproblem sizes are small enough, however, just solve the subproblems in a straightforward manner.

Combine the solutions to the subproblems into the solution for the original problem.

The merge sort algorithm closely follows the divide-and-conquer paradigm. Intuitively, it operates as follows.

Divide: Divide the n-element sequence to be sorted into two subsequences of n/2 elements each.

Conquer: Sort the two subsequences recursively using merge sort.

Combine: Merge the two sorted subsequences to produce the sorted answer.

We note that the recursion "bottoms out" when the sequence to be sorted has length 1, in which case there is no work to be done, since every sequence of length 1 is already in sorted order.

The key operation of the merge sort algorithm is the merging of two sorted sequences in the "combine" step. To perform the merging, we use an auxiliary procedure MERGE(A,p,q,r), where A is an array and p, q, and r are indices numbering elements of the array such that p [image] q < r. The procedure assumes that the subarrays A[p. .q] and A[q + 1. .r] are in sorted order. It merges them to form a single sorted subarray that replaces the current subarray A[p. .r].

Although we leave the pseudocode as an exercise (see Exercise 1.3-2), it is easy to imagine a MERGE procedure that takes time [image] (n), where n = r - p + 1 is the number of elements being merged. Returning to our card-playing motif, suppose we have two piles of cards face up on a table. Each pile is sorted, with the smallest cards on top. We wish to merge the two piles into a single sorted output pile, which is to be face down on the table. Our basic step consists of choosing the smaller of the two cards on top of the face-up piles, removing it from its pile (which exposes a new top card), and placing this card face down onto the output pile. We repeat this step until one input pile is empty, at which time we just take the remaining input pile and place it face down onto the output pile. Computationally, each basic step takes constant time, since we are checking just two top cards. Since we perform at most n basic steps, merging takes [image] (n) time.

We can now use the MERGE procedure as a subroutine in the merge sort algorithm. The procedure MERGE-SORT(A,p,r) sorts the elements in the subarray A[p. .r]. If p [image] r, the subarray has at most one element and is therefore already sorted. Otherwise, the divide step simply computes an index q that partitions A[p. .r] into two subarrays: A[p. .q], containing n/2] elements, and A[q + 1. .r], containing [image] n/2 [image] elements.4

4The expression [image] x [image] denotes the least integer greater than or equal to x, and [image] x [image] denotes the greatest integer less than or equal to x. These notations are defined in Chapter 2.

MERGE-SORT(A,p,r)

1 if p < r

2     then q      (p + r)/2

3          MERGE-SORT(A,p,q)

4          MERGE-SORT(A, q + 1, r)

5          MERGE(A,p,q,r)

To sort the entire sequence A = [image] A[1],A[2], . . . ,A[n] [image], we call MERGE-SORT(A, 1, length[A]), where once again length[A] = n. If we look at the operation of the procedure bottom-up when n is a power of two, the algorithm consists of merging pairs of 1-item sequences to form sorted sequences of length 2, merging pairs of sequences of length 2 to form sorted sequences of length 4, and so on, until two sequences of length n/2 are merged to form the final sorted sequence of length n. Figure 1.3 illustrates this process.

1.3.2 Analyzing divide-and-conquer algorithms

When an algorithm contains a recursive call to itself, its running time can often be described by a recurrence equation or recurrence, which describes the overall running time on a problem of size n in terms of the running time on smaller inputs. We can then use mathematical tools to solve the recurrence and provide bounds on the performance of the algorithm.

A recurrence for the running time of a divide-and-conquer algorithm is based on the three steps of the basic paradigm. As before, we let T(n) be the running time on a problem of size n. If the problem size is small enough, say n [image] c for some constant c, the straightforward solution takes constant time, which we write as [image] (1). Suppose we divide the problem into a subproblems, each of which is 1/b the size of the original. If we take D(n) time to divide the problem into subproblems and C(n) time to combine the solutions to the subproblems into the solution to the original problem, we get the recurrence

Figure 1.3 The operation of merge sort on the array A = 5, 2, 4, 6, 1, 3, 2, 6 . The lengths of the sorted sequences being merged increase as the algorithm progresses from bottom to top.

In Chapter 4, we shall see how to solve common recurrences of this form.

Analysis of merge sort

Although the pseudocode for MERGE-SORT works correctly when the number of elements is not even, our recurrence-based analysis is simplified if we assume that the original problem size is a power of two. Each divide step then yields two subsequences of size exactly n/2. In Chapter 4, we shall see that this assumption does not affect the order of growth of the solution to the recurrence.

We reason as follows to set up the recurrence for T(n), the worst-case running time of merge sort on n numbers. Merge sort on just one element takes constant time. When we have n > 1 elements, we break down the running time as follows.

Divide: The divide step just computes the middle of the subarray, which takes constant time. Thus, D(n) = [image] (1).

Conquer: We recursively solve two subproblems, each of size n/2, which contributes 2T(n/2) to the running time.

Combine: We have already noted that the MERGE procedure on an n-element subarray takes time [image] (n), so C(n) = [image] (n).

When we add the functions D(n) and C(n) for the merge sort analysis, we are adding a function that is [image] (n) and a function that is [image] (1). This sum is a linear function of n, that is, [image] (n). Adding it to the 2T(n/2) term from the "conquer" step gives the recurrence for the worst-case running time T(n) of merge sort:

In Chapter 4, we shall show that T(n) is [image] (n 1g n), where 1g n stands for log2 n. For large enough inputs, merge sort, with its [image] (n lg n) running time, outperforms insertion sort, whose running time is [image] (n2), in the worst case.

Exercises

1.3-1

Using Figure 1.3 as a model, illustrate the operation of merge sort on the array A = [image] 3, 41, 52, 26, 38, 57, 9, 49 [image].

1.3-2

Write pseudocode for MERGE(A,p,q,r).

1.3-3

Use mathematical induction to show that the solution of the recurrence

1.3-4

Insertion sort can be expressed as a recursive procedure as follows. In order to sort A[1 . . n], we recursively sort A[1 . . n - 1] and then insert A[n] into the sorted array A[1 . . n - 1]. Write a recurrence for the running time of this recursive version of insertion sort.

1.3-5

Referring back to the searching problem (see Exercise 1.1-3), observe that if the sequence A is sorted, we can check the midpoint of the sequence against v and eliminate half of the sequence from further consideration. Binary search is an algorithm that repeats this procedure, halving the size of the remaining portion of the sequence each time. Write pseudocode, either iterative or recursive, for binary search. Argue that the worst-case running time of binary search is [image] (lg n).

1.3-6

Observe that the while loop of lines 5-7 of the INSERTION-SORT procedure in Section 1.1 uses a linear search to scan (backward) through the sorted subarray A[1 . . j - 1]. Can we use a binary search (see Exercise 1.3-5) instead to improve the overall worst-case running time of insertion sort to [image] (n lg n)?

1.3-7

Describe a [image] (n lg n)-time algorithm that, given a set S of n real numbers and another real number x, determines whether or not there exist two elements in S whose sum is exactly x.

1.4 Summary

A good algorithm is like a sharp knife--it does exactly what it is supposed to do with a minimum amount of applied effort. Using the wrong algorithm to solve a problem is like trying to cut a steak with a screwdriver: you may eventually get a digestible result, but you will expend considerably more effort than necessary, and the result is unlikely to be aesthetically pleasing.

Algorithms devised to solve the same problem often differ dramatically in their efficiency. These differences can be much more significant than the difference between a personal computer and a supercomputer. As an example, let us pit a supercomputer running insertion sort against a small personal computer running merge sort. They each must sort an array of one million numbers. Suppose the supercomputer executes 100 million instructions per second, while the personal computer executes only one million instructions per second. To make the difference even more dramatic, suppose that the world's craftiest programmer codes insertion sort in machine language for the supercomputer, and the resulting code requires 2n2 supercomputer instructions to sort n numbers. Merge sort, on the other hand, is programmed for the personal computer by an average programmer using a high-level language with an inefficient compiler, with the resulting code taking 50n 1g n personal computer instructions. To sort a million numbers, the supercomputer takes

By using an algorithm whose running time has a lower order of growth, even with a poor compiler, the personal computer runs 20 times faster than the supercomputer!

This example shows that algorithms, like computer hardware, are a technology. Total system performance depends on choosing efficient algorithms as much as on choosing fast hardware. Just as rapid advances are being made in other computer technologies, they are being made in algorithms as well.

Exercises

1.4-1

Suppose we are comparing implementations of insertion sort and merge sort on the same machine. For inputs of size n, insertion sort runs in 8n2 steps, while merge sort runs in 64n 1g n steps. For which values of n does insertion sort beat merge sort? How might one rewrite the merge sort pseudocode to make it even faster on small inputs?

1.4-2

What is the smallest value of n such that an algorithm whose running time is 100n2 runs faster than an algorithm whose running time is 2n on the same machine?

Problems

1-1 Comparison of running times

For each function f(n) and time t in the following table, determine the largest size n of a problem that can be solved in time t, assuming that the algorithm to solve the problem takes f(n) microseconds.

1-2 Insertion sort on small arrays in merge sort

Although merge sort runs in [image] (n lg n) worst-case time and insertion sort runs in [image] (n2) worst-case time, the constant factors in insertion sort make it faster for small n. Thus, it makes sense to use insertion sort within merge sort when subproblems become sufficiently small. Consider a modification to merge sort in which n/k sublists of length k are sorted using insertion sort and then merged using the standard merging mechanism, where k is a value to be determined.

a. Show that the n/k sublists, each of length k, can be sorted by insertion sort in [image] (nk) worst-case time.

b. Show that the sublists can be merged in [image] (n lg(n/k)) worst-case time.

c. Given that the modified algorithm runs in [image] (nk + n 1g(n/k)) worst-case time, what is the largest asymptotic ( [image] -notation) value of k as a function of n for which the modified algorithm has the same asymptotic running time as standard merge sort?

d. How should k be chosen in practice?

1-3 Inversions

Let A[1 . . n] be an array of n distinct numbers. If i < j and A[i] > A[j], then the pair (i, j) is called an inversion of A.

a. List the five inversions of the array [image] 2, 3, 8, 6, 1 [image].

b. What array with elements from the set { 1,2, . . . , n} has the most inversions? How many does it have?

c. What is the relationship between the running time of insertion sort and the number of inversions in the input array? Justify your answer.

d. Give an algorithm that determines the number of inversions in any permutation on n elements in [image] (n 1g n) worst-case time. (Hint: Modify merge sort.)

Chapter notes

There are many excellent texts on the general topic of algorithms, including those by Aho, Hopcroft, and Ullman[4, 5], Baase [14], Brassard and Bratley [33], Horowitz and Sahni [105], Knuth[121, 122, 123], Manber [142], Mehlhorn[144, 145, 146], Purdom and Brown [164], Reingold, Nievergelt, and Deo [167], Sedgewick [175], and Wilf [201]. Some of the more practical aspects of algorithm design are discussed by Bentley[24, 25] and Gonnet [90].

In 1968, Knuth published the first of three volumes with the general title The Art of Computer Programming[121, 122, 123]. The first volume ushered in the modern study of computer algorithms with a focus on the analysis of running time, and the full series remains an engaging and worthwhile reference for many of the topics presented here. According to Knuth, the word "algorithm" is derived from the name "al-Khowârizmî," a ninth-century Persian mathematician.

Aho, Hopcroft, and Ullman [4] advocated the asymptotic analysis of algorithms as a means of comparing relative performance. They also popularized the use of recurrence relations to describe the running times of recursive algorithms.

Knuth [123] provides an encyclopedic treatment of many sorting algorithms. His comparison of sorting algorithms (page 381) includes exact step-counting analyses, like the one we performed here for insertion sort. Knuth's discussion of insertion sort encompasses several variations of the algorithm. The most important of these is Shell's sort, introduced by D. L. Shell, which uses insertion sort on periodic subsequences of the input to produce a faster sorting algorithm.

Merge sort is also described by Knuth. He mentions that a mechanical collator capable of merging two decks of punched cards in a single pass was invented in 1938. J. von Neumann, one of the pioneers of computer science, apparently wrote a program for merge sort on the EDVAC computer in 1945.

Go to Part I Back to Table of Contents


Intro to Algorithms: CHAPTER 2: GROWTH OF FUNCTIONS

 Return to Table of Contents

CHAPTER 2: GROWTH OF FUNCTIONS

The order of growth of the running time of an algorithm, defined in Chapter 1, gives a simple characterization of the algorithm's efficiency and also allows us to compare the relative performance of alternative algorithms. Once the input size n becomes large enough, merge sort, with its [image] (n lg n) worst-case running time, beats insertion sort, whose worst-case running time is [image] (n2). Although we can sometimes determine the exact running time of an algorithm, as we did for insertion sort in Chapter 1, the extra precision is not usually worth the effort of computing it. For large enough inputs, the multiplicative constants and lower-order terms of an exact running time are dominated by the effects of the input size itself.

When we look at input sizes large enough to make only the order of growth of the running time relevant, we are studying the asymptotic efficiency of algorithms. That is, we are concerned with how the running time of an algorithm increases with the size of the input in the limit, as the size of the input increases without bound. Usually, an algorithm that is asymptotically more efficient will be the best choice for all but very small inputs.

This chapter gives several standard methods for simplifying the asymptotic analysis of algorithms. The next section begins by defining several types of "asymptotic notation," of which we have already seen an example in [image] -notation. Several notational conventions used throughout this book are then presented, and finally we review the behavior of functions that commonly arise in the analysis of algorithms.

2.1 Asymptotic notation

The notations we use to describe the asymptotic running time of an algorithm are defined in terms of functions whose domains are the set of natural numbers N = {0, 1, 2, . . .}. Such notations are convenient for describing the worst-case running-time function T(n), which is usually defined only on integer input sizes. It is sometimes convenient, however, to abuse asymptotic notation in a variety of ways. For example, the notation is easily extended to the domain of real numbers or, alternatively, restricted to a subset of the natural numbers. It is important, however, to understand the precise meaning of the notation so that when it is abused, it is not misused. This section defines the basic asymptotic notations and also introduces some common abuses.

-notation

In Chapter 1, we found that the worst-case running time of insertion sort is T(n) = [image] (n2). Let us define what this notation means. For a given function g(n), we denote by [image] (g(n)) the set of functions

(g(n)) = {â(n) : there exist positive constants c1, c2, and n0 such that

0 [image] c1g(n) [image] â(n) [image] c2g(n) for all n [image] n0}.

A function â(n) belongs to the set [image] (g(n)) if there exist positive constants c1 and c2 such that it can be "sandwiched" between c1g(n) and +c2g(n), for sufficiently large n. Although [image] (g(n)) is a set, we write "â(n) = [image] (g(n))" to indicate that â(n) is a member of [image] (g(n)), or "â(n) [image] [image] (g(n))." This abuse of equality to denote set membership may at first appear confusing, but we shall see later in this section that it has advantages.

Figure 2.1 (a) gives an intuitive picture of functions â(n) and g(n), where â(n) = [image] (g(n)). For all values of n to the right of n0, the value of â(n) lies at or above c1g(n) and at or below c2g(n). In other words, for all n [image] n0, the function â(n) is equal to g(n) to within a constant factor. We say that g(n) is an asymptotically tight bound for â(n).

The definition of [image] (g(n)) requires that every member of [image] (g(n)) be asymptotically nonnegative, that is, that â(n) be nonnegative whenever n is sufficiently large. Consequently, the function g(n) itself must be asymptotically nonnegative, or else the set [image] (g(n)) is empty. We shall therefore assume that every function used within [image] -notation is asymptotically non-negative. This assumption holds for the other asymptotic notations defined in this chapter as well.

In Chapter 1, we introduced an informal notion of [image] -notation that amounted to throwing away lower-order terms and ignoring the leading coefficient of the highest-order term. Let us briefly justify this intuition by using the formal definition to show that 1/2n2 - 3n = [image] (n2). To do so, we must determine positive constants c1, c2, and n0 such that

for all n [image] n0. Dividing by n2 yields

Figure 2.1 Graphic examples of the , O, and notations. In each part, the value of n0 shown is the minimum possible value; any greater value would also work. (a) -notation bounds a function to within constant factors. We write (n) = (g(n)) if there exist positive constants n0, c1, and c2 such that to the right of n0, the value of â(n) always lies between c1g(n) and c2g(n) inclusive. (b) O-notation gives an upper bound for a function to within a constant factor. We write â(n) = O(g(n)) if there are positive constants n0 and c such that to the right of n0, the value of â(n) always lies on or below cg(n). (c) -notation gives a lower bound for a function to within a constant factor. We write â(n) = (g(n)) if there are positive constants n0 and c such that to the right of n0, the value of â(n) always lies on or above cg(n).

The right-hand inequality can be made to hold for any value of n [image] 1 by choosing c2 [image] 1/2. Likewise, the left-hand inequality can be made to hold for any value of n [image] 7 by choosing c1 [image] 1/14. Thus, by choosing c1 = 1/14, c2 = 1/2, and n0 = 7, we can verify that [image]. Certainly, other choices for the constants exist, but the important thing is that some choice exists. Note that these constants depend on the function [image] a different function belonging to [image] (n2) would usually require different constants.

We can also use the formal definition to verify that 6n3 [image] [image] (n2). Suppose for the purpose of contradiction that c2 and n0 exist such that 6n3 [image] c2n2 for all n [image] n0. But then n [image] c2/6, which cannot possibly hold for arbitrarily large n, since c2 is constant.

Intuitively, the lower-order terms of an asymptotically positive function can be ignored in determining asymptotically tight bounds because they are insignificant for large n. A tiny fraction of the highest-order term is enough to dominate the lower-order terms. Thus, setting c1 to a value that is slightly smaller than the coefficient of the highest-order term and setting c2 to a value that is slightly larger permits the inequalities in the definition of [image] -notation to be satisfied. The coefficient of the highest-order term can likewise be ignored, since it only changes c1 and c2 by a constant factor equal to the coefficient.

As an example, consider any quadratic function â(n) = an2 + bn + c, where a, b, and c are constants and a > 0. Throwing away the lower-order terms and ignoring the constant yields â(n) = [image] (n2). Formally, to show the same thing, we take the constants c1 = a/4, c2 = 7a/4, and n0 = 2 . max [image]. The reader may verify that 0 [image] c1n2 [image] an2 + bn + c [image] c2n2 for all n [image] n0. In general, for any polynomial p(n) = [image] where the ai are constants and ad > 0, we have p(n) = [image] (nd) (see Problem 2-1).

Since any constant is a degree-0 polynomial, we can express any constant function as [image] (n0), or [image] (1). This latter notation is a minor abuse, however, because it is not clear what variable is tending to infinity.1 We shall often use the notation [image] (1) to mean either a constant or a constant function with respect to some variable.

1The real problem is that our ordinary notation for functions does not distinguish functions from values. In [image] -calculus, the parameters to a function are clearly specified: the function n2 could be written as [image] n.n2, or even [image] r.r2. Adopting a more rigorous notation, however, would complicate algebraic manipulations, and so we choose to tolerate the abuse.

O-notation

The [image] -notation asymptotically bounds a function from above and below. When we have only an asymptotic upper bound, we use O-notation. For a given function g(n), we denote by O(g(n)) the set of functions

O(g(n)) = {â(n) : there exist positive constants c and n0 such that

0 [image] â(n) [image] cg(n) for all n [image] n0}.

We use O-notation to give an upper bound on a function, to within a constant factor. Figure 2.1 (b) shows the intuition behind O-notation. For all values n to the right of n0, the value of the function â(n) is on or below g(n).

To indicate that a function â(n) is a member of O(g(n)), we writeâ(n) = O(g(n)). Note that â(n) = [image] (g(n)) implies â(n) = O(g(n)), since [image] -notation is a stronger notion than O-notation. Written set-theoretically, we have [image] (g(n)) [image] O(g(n)). Thus, our proof that any quadratic function an2 + bn + c, where a > 0, is in [image] (n2) also shows that any quadratic function is in O(n2). What may be more surprising is that any linear function an + b is in O(n2), which is easily verified by taking c = a + |b| and n0 = 1.

Some readers who have seen O-notation before may find it strange that we should write, for example, n = O(n2). In the literature, O-notation is sometimes used informally to describe asymptotically tight bounds, that is, what we have defined using [image] -notation. In this book, however, when we write â(n) = O(g(n)), we are merely claiming that some constant multiple of g(n) is an asymptotic upper bound on â(n), with no claim about how tight an upper bound it is. Distinguishing asymptotic upper bounds from asymptotically tight bounds has now become standard in the algorithms literature.

Using O-notation, we can often describe the running time of an algorithm merely by inspecting the algorithm's overall structure. For example, the doubly nested loop structure of the insertion sort algorithm from Chapter 1 immediately yields an O(n2) upper bound on the worst-case running time: the cost of the inner loop is bounded from above by O(1) (constant), the indices i and j are both at most n, and the inner loop is executed at most once for each of the n2 pairs of values for i and j.

Since O-notation describes an upper bound, when we use it to bound the worst-case running time of an algorithm, by implication we also bound the running time of the algorithm on arbitrary inputs as well. Thus, the O(n2) bound on worst-case running time of insertion sort also applies to its running time on every input. The [image] (n2) bound on the worst-case running time of insertion sort, however, does not imply a [image] (n2) bound on the running time of insertion sort on every input. For example, we saw in Chapter 1 that when the input is already sorted, insertion sort runs in [image] (n) time.

Technically, it is an abuse to say that the running time of insertion sort is O(n2), since for a given n, the actual running time depends on the particular input of size n. That is, the running time is not really a function of n. What we mean when we say "the running time is O(n2)" is that the worst-case running time (which is a function of n) is O(n2), or equivalently, no matter what particular input of size n is chosen for each value of n, the running time on that set of inputs is O(n2).

-notation

Just as O-notation provides an asymptotic upper bound on a function, [image] -notation provides an asymptotic lower bound. For a given function g(n), we denote by [image] (g(n)) the set of functions

(g(n)) = {f(n): there exist positive constants c and n0 such that

0 [image] cg(n) [image] f (n) for all n [image] n0} .

The intuition behind [image] -notation is shown in Figure 2.1(c). For all values n to the right of n0, the value of f(n) is on or above g(n).

From the definitions of the asymptotic notations we have seen thus far, it is easy to prove the following important theorem (see Exercise 2.1-5).

Theorem 2.1

For any two functions f(n) and g(n), f(n) = [image] (g(n)) if and only if f(n) = O(g(n)) and f(n) = [image] (g(n)).

As an example of the application of this theorem, our proof that an2 + bn + c = [image] (n2) for any constants a, b, and c, where a > 0, immediately implies that an2 + bn + c = [image] (n2) and an2 + bn + c = O(n2). In practice, rather than using Theorem 2.1 to obtain asymptotic upper and lower bounds from asymptotically tight bounds, as we did for this example, we usually use it to prove asymptotically tight bounds from asymptotic upper and lower bounds.

Since [image] -notation describes a lower bound, when we use it to bound the best-case running time of an algorithm, by implication we also bound the running time of the algorithm on arbitrary inputs as well. For example, the best-case running time of insertion sort is [image] (n), which implies that the running time of insertion sort is [image] (n).

The running time of insertion sort therefore falls between [image] (n) and O(n2), since it falls anywhere between a linear function of n and a quadratic function of n. Moreover, these bounds are asymptotically as tight as possible: for instance, the running time of insertion sort is not [image] (n2), since insertion sort runs in [image] (n) time when the input is already sorted. It is not contradictory, however, to say that the worst-case running time of insertion sort is [image] (n2), since there exists an input that causes the algorithm to take [image] (n2) time. When we say that the running time (no modifier) of an algorithm is [image] (g(n)), we mean that no matter what particular input of size n is chosen for each value of n, the running time on that set of inputs is at least a constant times g(n), for sufficiently large n.

Asymptotic notation in equations

We have already seen how asymptotic notation can be used within mathematical formulas. For example, in introducing O-notation, we wrote "n = O(n2)." We might also write 2n2 + 3n + 1 = 2n2 + [image] (n). How do we interpret such formulas?

When the asymptotic notation stands alone on the right-hand side of an equation, as in n = O(n2), we have already defined the equal sign to mean set membership: n [image] O(n2). In general, however, when asymptotic notation appears in a formula, we interpret it as standing for some anonymous function that we do not care to name. For example, the formula 2n2 + 3n + 1 = 2n2 + [image] (n) means that 2n2 + 3n + 1 = 2n2 + f(n), where [image] (n) is some function in the set [image] (n). In this case, [image] (n) = 3n + 1, which indeed is in [image] (n).

Using asymptotic notation in this manner can help eliminate inessential detail and clutter in an equation. For example, in Chapter I we expressed the worst-case running time of merge sort as the recurrence

T(n) = 2T(n/2) +   (n).

If we are interested only in the asymptotic behavior of T(n), there is no point in specifying all the lower-order terms exactly; they are all understood to be included in the anonymous function denoted by the term [image] (n).

The number of anonymous functions in an expression is understood to be equal to the number of times the asymptotic notation appears. For example, in the expression

there is only a single anonymous function (a function of i). This expression is thus not the same as O(1) + O(2) + [image] [image] [image] + O(n), which doesn't really have a clean interpretation.

In some cases, asymptotic notation appears on the left-hand side of an equation, as in

2n2 +   (n) =   (n2).

We interpret such equations using the following rule: No matter how the anonymous functions are chosen on the left of the equal sign, there is a way to choose the anonymous functions on the right of the equal sign to make the equation valid. Thus, the meaning of our example is that for any function [image] (n) [image] [image] (n), there is some function g(n) [image] [image] (n2) such that 2n2 + [image] (n) = g(n) for all n. In other words, the right-hand side of an equation provides coarser level of detail than the left-hand side.

A number of such relationships can be chained together, as in

2n2 + 3n + 1 = 2n2 +   (n)

=   (n2).

We can interpret each equation separately by the rule above. The first equation says that there is some function [image] (n) [image] [image] (n) such that 2n2 + 3n + 1 = 2n2 + [image] (n) for all n. The second equation says that for any function g(n) [image] [image] (n) (such as the [image] (n) just mentioned), there is some function h(n) [image] [image] (n2) such that 2n2 + g(n) = h(n) for all n. Note that this interpretation implies that 2n2 + 3n + 1 = [image] (n2), which is what the chaining of equations intuitively gives us.

o-notation

The asymptotic upper bound provided by O-notation may or may not be asymptotically tight. The bound 2n2 = O(n2) is asymptotically tight, but the bound 2n = O(n2) is not. We use o-notation to denote an upper bound that is not asymptotically tight. We formally define o(g(n)) ("little-oh of g of n") as the set

o(g(n)) = { [image] (n): for any positive constant c > 0, there exists a constant

n0 > 0 such that 0 [image] f(n) < cg(n) for all n [image] n0}.

For example, 2n = o(n2), but 2n2 [image] o(n2).

The definitions of O-notation and o-notation are similar. The main difference is that in [image] (n) = O(g(n)), the bound 0 [image] [image] (n) [image] cg(n) holds for some constant c > 0, but in [image] (n) = o(g(n)), the bound 0 [image] [image] (n) < cg(n) holds for all constants c > 0. Intuitively, in the o-notation, the function f(n) becomes insignificant relative to g(n) as n approaches infinity; that is,

(2.1)

Some authors use this limit as a definition of the o-notation; the definition in this book also restricts the anonymous functions to be asymptotically nonnegative.

-notation

By analogy, [image] -notation is to [image] -notation as o-notation is to O-notation. We use [image] -notation to denote a lower bound that is not asymptotically tight. One way to define it is by

  (n)      (g(n)) if and only if g(n)    o(  (n)).

Formally, however, we define [image] (g(n)) ("little-omega of g of n") as the set

(g(n)) = { [image] (n): for any positive constant c > 0, there exists a constant

n0 > 0 such that 0 [image] cg(n) < [image] (n) for all n [image] n0}.

For example, n2/2 = [image] (n), but n2/2 [image] [image] (n2). The relation [image] (n) = [image] (g(n)) implies that

if the limit exists. That is, [image] (n) becomes arbitrarily large relative to g(n) as n approaches infinity.

Comparison of functions

Many of the relational properties of real numbers apply to asymptotic comparisons as well. For the following, assume that [image] (n) and g(n) are asymptotically positive.

Transitivity:

â(n) =   (g(n)) and g(n) =   (h(n)) imply â(n) =   (h(n)),

â(n) = O(g(n)) and g(n) = O(h(n)) imply â(n) = O(h(n)),

â(n) =   (g(n)) and g(n) =   (h(n)) imply â(n) =   (h(n)),

â(n) = o(g(n)) and g(n) = o(h(n)) imply â(n) = o(h(n)),

â(n) =   (g(n)) and g(n) =   (h(n)) imply â(n) =   (h(n)).

Reflexivity:

â(n) =   (â(n)),

â(n) = O(â(n)),

â(n) =   (â(n)),

Symmetry:

â(n) =   (g(n)) if and only if g(n) =   (â(n)).

Transpose symmetry:

  (n) = O(g(n)) if and only if g(n) =   (f(n)),

  (n) = o(g(n)) if and only if g(n) =   (  (n)).

Because these properties hold for asymptotic notations, one can draw an analogy between the asymptotic comparison of two functions f and g and the comparison of two real numbers a and b:

  (n) = O(g(n))      a    b,

  (n) =   (g(n))      a    b,

  (n) =   (g(n))      a = b,

  (n) = o(g(n))      a < b,

  (n) =   (g(n))      a > b.

One property of real numbers, however, does not carry over to asymptotic notation:

Trichotomy: For any two real numbers a and b, exactly one of the following must hold: a < b, a = b, or a > b.

Although any two real numbers can be compared, not all functions are asymptotically comparable. That is, for two functions [image] (n) and g(n), it may be the case that neither [image] (n) = O(g(n)) nor [image] (n) = [image] (g(n)) holds. For example, the functions n and n 1+sin n cannot be compared using asymptotic notation, since the value of the exponent in n1 +sin n oscillates between 0 and 2, taking on all values in between.

Exercises

2.1-1

Let f(n) and g(n) be asymptotically nonnegative functions. Using the basic definition of [image] -notation, prove that max( [image] (n),g(n)) = [image] (f(n) + g(n)).

2.1-2

Show that for any real constants a and b, where b > 0,

(n + a)b =   (nb) .

(2.2)

2.1-3

Explain why the statement, "The running time of algorithm A is at least O(n2)," is content-free.

2.1-4

Is 2n+1 = O(2n)? Is 22n = O(2n)?

2.1-5

Prove Theorem 2.1.

2.1-6

Prove that the running time of an algorithm is [image] (g(n)) if and only if its worst-case running time is O(g(n)) and its best-case running time is [image] (g(n)).

2.1-7

Prove that o(g(n)) [image] [image] (g(n)) is the empty set.

2.1-8

We can extend our notation to the case of two parameters n and m that can go to infinity independently at different rates. For a given function g(n, m), we denote by O(g(n, m)) the set of functions

O(g(n, m)) = { [image] (n,m): there exist positive constants c, n0, and m0

such that 0 [image] [image] (n, m) [image] cg(n, m)

for all n [image] n0 and m [image] m0}.

Give corresponding definitions for [image] (g(n, m)) and [image] (g(n, m)).

2.2 Standard notations and common functions

This section reviews some standard mathematical functions and notations and explores the relationships among them. It also illustrates the use of the asymptotic notations.

Monotonicity

A function [image] (n) is monotonically increasing if m [image] n implies [image] (m) [image] [image] (n). Similarly, it is monotonically decreasing if m [image] n implies [image] (m) [image] [image] (n). A function [image] (n) is strictly increasing if m < n implies [image] (m) < [image] (n) and strictly decreasing if m < n implies [image] (m) > [image] (n).

Floors and ceilings

For any real number x, we denote the greatest integer less than or equal to x by [image] x [image] (read "the floor of x") and the least integer greater than or equal to x by [image] x [image] (read "the ceiling of x"). For all real x,

x - 1 <   x      x      x   < x +1.

For any integer n,

  n/2   +   n/2  = n,

and for any integer n and integers a [image] 0 and b [image] 0,

    n/a  /b   =   n/ab

(2.3)

and

    n/a  /b   =   n/ab   .

(2.4)

The floor and ceiling functions are monotonically increasing.

Polynomials

Given a positive integer d, a polynomial in n of degree d is a function p(n) of the form

where the constants a0, a1, . . ., ad are the coefficients of the polynomial and ad [image] 0. A polynomial is asymptotically positive if and only if ad > 0. For an asymptotically positive polynomial p(n) of degree d, we have p(n) = [image] (nd). For any real constant a [image] 0, the function na is monotonically increasing, and for any real constant a [image] 0, the function na is monotonically decreasing. We say that a function [image] (n) is polynomially bounded if [image] (n) = n0(1), which is equivalent to saying that [image] (n) = O(nk) for some constant k (see Exercise 2.2-2).

Exponentials

For all real a [image] 0, m, and n, we have the following identities:

a0  =  1,

a1  =  a,

a-1  =   1/a,

(am)n  =   amn,

(am)n  =   (an)m,

aman   =   am+n.

For all n and a [image] 1, the function an is monotonically increasing in n. When convenient, we shall assume 00 = 1.

The rates of growth of polynomials and exponentials can be related by the following fact. For all real constants a and b such that a > 1,

(2.5)

from which we can conclude that

nb = 0(an).

Thus, any positive exponential function grows faster than any polynomial.

Using e to denote 2.71828 . . ., the base of the natural logarithm function, we have for all real x,

(2.6)

where "!" denotes the factorial function defined later in this section. For all real x, we have the inequality

ex    1 + x,

(2.7)

where equality holds only when x = 0. When |x| [image] 1, we have the approximation

1 + x    ex    l + x +x2 .

(2.8)

When x [image] 0, the approximation of ex by 1 + x is quite good:

ex = 1 + x +   (x2).

(In this equation, the asymptotic notation is used to describe the limiting behavior as x [image] 0 rather than as x [image] [image].) We have for all x,

Logarithms

We shall use the following notations:

lg n   =   log2 n     (binary logarithm),

ln n   =   loge n     (natural logarithm),

lgk n   =   (lg n)k    (exponentiation),

lg lg n  =  lg(lg n)    (composition).

An important notational convention we shall adopt is that logarithm functions will apply only to the next term in the formula, so that lg n + k will mean (lg n) + k and not lg(n + k). For n > 0 and b > 1, the function logb n is strictly increasing.

For all real a > 0, b > 0, c > 0, and n,

(2.9)

Since changing the base of a logarithm from one constant to another only changes the value of the logarithm by a constant factor, we shall often use the notation "lg n" when we don't care about constant factors, such as in O-notation. Computer scientists find 2 to be the most natural base for logarithms because so many algorithms and data structures involve splitting a problem into two parts.

There is a simple series expansion for ln(1 + x) when [image] x [image] < 1:

We also have the following inequalities for x > -1:

(2.10)

where equality holds only for x = 0.

We say that a function f(n) is polylogarithmically bounded if f(n) = lgO(1) n. We can relate the growth of polynomials and polylogarithms by substituting lg n for n and 2a for a in equation (2.5), yielding

From this limit, we can conclude that

lgb n = o(na)

for any constant a > 0. Thus, any positive polynomial function grows faster than any polylogarithmic function.

Factorials

The notation n! (read "n factorial") is defined for integers n [image] 0 as

Thus, n! = 1.2.3...n.

A weak upper bound on the factorial function is n! nn, since each of the n terms in the factorial product is at most n. Stirling's approximation,

(2.11)

where e is the base of the natural logarithm, gives us a tighter upper bound, and a lower bound as well. Using Stirling's approximation, one can prove

n!  =  o(nn),

n!  =    (2n),

lg(n!)  =    (n lg n).

The following bounds also hold for all n:

(2.12)

The iterated logarithm function

We use the notation lg* n (read "log star of n") to denote the iterated logarithm, which is defined as follows. Let the function lg(i) n be defined recursively for nonnegative integers i as

Be sure to distinguish lg(i) n (the logarithm function applied i times in succession, starting with argument n) from lgi n (the logarithm of n raised to the ith power). The iterated logarithm function is defined as

The iterated logarithm is a very slowly growing function:

lg* 2  =  1,

lg* 4  =  2,

lg* 16   =  3,

lg* 65536     =  4,

lg*(265536)     =  5.

Since the number of atoms in the observable universe is estimated to be about 1080, which is much less than 265536, we rarely encounter a value of n such that lg* n > 5.

Fibonacci numbers

The Fibonacci numbers are defined by the following recurrence:

F0  =  0,

F1  =  1,

Fi  =  Fi-1+Fi-2  for i    2.

(2.13)

Thus, each Fibonacci number is the sum of the two previous ones, yielding the sequence

0, 1, 1, 2, 3, 5, 8, 13, 21, 34, 55, ... .

Fibonacci numbers are related to the golden ratio [image] and to its conjugate [image], which are given by the following formulas:

(2.14)

Specifically, we have

(2.15)

which can be proved by induction (Exercise 2.2-7). Since [image] < 1, we have [image], so that the ith Fibonacci number Fi is equal to [image] rounded to the nearest integer. Thus, Fibonacci numbers grow exponentially.

Exercises

2.2-1

Show that if f(n) and g(n) are monotonically increasing functions, then so are the functions f(n) + g(n) and f(g(n)), and if f(n) and g(n) are in addition nonnegative, then f(n) . g(n)is monotonically increasing.

2.2-2

Use the definition of O-notation to show that T(n) = no(1) if and only if there exists a constant k > 0 such that T(n) = O(nk).

2.2-3

Prove equation (2.9).

2.2-4

Prove that lg(n!) = [image] (n lg n) and that n! = o(nn).

2.2-5

Is the function [lg n]! polynomially bounded? Is the function [lg lg n]! polynomially bounded?

2.2-6 *

Which is asymptotically larger: lg(lg* n) or lg*(lg n)?

2.2-7

Prove by induction that the ith Fibonacci number satisfies the equality [image] where [image] is the golden ratio and is its conjugate.

2.2-8

Prove that for i 0, the (i + 2)nd Fibonacci number satisfies Fi+2 i.

Problems

2-1 Asymptotic behavior of polynomials

Let

where ad > 0, be a degree-d polynomial in n, and let k be a constant. Use the definitions of the asymptotic notations to prove the following properties.

a. If k d, then p(n) = O(nk).

b. If k d, then p(n) = (nk).

c. If k = d, then p(n) = (nk).

d. If k > d, then p(n) = o(nk).

e. If k < d, then p(n) = (nk).

2-2 Relative asymptotic growths

Indicate, for each pair of expressions (A, B) in the table below, whether A is O, o, , , or of B. Assume that k 1, > 0, and c > 1 are constants. Your answer should be in the form of the table with "yes" or "no" written in each box.

2-3 Ordering by asymptotic growth rates

a. Rank the following functions by order of growth; that is, find an arrangement g1, g2, . . . ,g30 of the functions satisfying g1 = (g2), g2 = (g3), ..., g29 = (g30). Partition your list into equivalence classes such that f(n) and g(n) are in the same class if and only if f(n) = (g(n)).

b. Give an example of a single nonnegative function f(n) such that for all functions gi(n) in part (a), f(n)is neither O(gi(n)) nor (gi(n)).

2-4 Asymptotic notation properties

Let f(n) and g(n) be asymptotically positive functions. Prove or disprove each of the following conjectures.

a. f(n) = O(g(n)) implies g(n) = O(f(n)).

b. f(n) + g(n) = (min(â(n), g(n))).

c. f(n) = O(g(n)) implies lg(f(n)) = O(lg(g(n))), where lg(g(n)) > 0 and f(n) 1 for all sufficiently large n.

d. f(n) = O(g(n)) implies 2f(n) = O(2g(n)).

e. f(n) = O ((f(n))2).

f. f(n) = O(g(n)) implies g(n) = (f(n)).

g. f(n) = (f(n/2)).

h. f(n) + o(f(n)) = (f (n)).

2-5 Variations on O and

Some authors define in a slightly different way than we do; let's use (read "omega infinity") for this alternative definition. We say that if there exists a positive constant c such that f(n) cg(n) 0 for infinitely many integers n.

a. Show that for any two functions f(n) and g(n) that are asymptotically nonnegative, either f(n) = O(g(n)) or f(n) = (g(n)) or both, whereas this is not true if we use in place of

b. Describe the potential advantages and disadvantages of using instead of to characterize the running times of programs.

Some authors also define O in a slightly different manner; let's use O' for the alternative definition. We say that f(n) = O'(g(n)) if and only if f(n) = O'(g(n)).

c. What happens to each direction of the "if and only if" in Theorem 2.1 under this new definition?

Some authors define (read "soft-oh") to mean O with logarithmic factors ignored:

(g(n)) = {f(n): there exist positive constants c, k, and n0 such that

0 f(n) cg(n)1gk(n) for all n n0}.

d. Define in a similar manner. Prove the corresponding analog to Theorem 2.1.

2-6 Iterated functions

The iteration operator "*" used in the 1g* function can be applied to monotonically increasing functions over the reals. For a function â satisfying f(n) < n, we define the function f(i) recursively for nonnegative integers i by

For a given constant c R, we define the iterated function by

which need not be well-defined in all cases. In other words, the quantity (n) is the number of iterated applications of the function â required to reduce its argument down to c or less.

For each of the following functions f (n) and constants c, give as tight a bound as possible on (n).

Chapter notes

Knuth [121] traces the origin of the O-notation to a number-theory text by P. Bachmann in 1892. The o-notation was invented by E. Landau in 1909 for his discussion of the distribution of prime numbers. The and notations were advocated by Knuth [124] to correct the popular, but technically sloppy, practice in the literature of using O-notation for both upper and lower bounds. Many people continue to use the O-notation where the -notation is more technically precise. Further discussion of the history and development of asymptotic notations can be found in Knuth[121, 124] and Brassard and Bratley [33].

Not all authors define the asymptotic notations in the same way, although the various definitions agree in most common situations. Some of the alternative definitions encompass functions that are not asymptotically nonnegative, as long as their absolute values are appropriately bounded.

Other properties of elementary mathematical functions can be found in any good mathematical reference, such as Abramowitz and Stegun [1] or Beyer [27], or in a calculus book, such as Apostol [12] or Thomas and Finney [192]. Knuth [121] contains a wealth of material on discrete mathematics as used in computer science.

Go to Chapter 3 Back to Table of Contents


Intro to Algorithms: CHAPTER 3: SUMMATIONS

 Return to Table of Contents

CHAPTER 3: SUMMATIONS

When an algorithm contains an iterative control construct such as a while or for loop, its running time can be expressed as the sum of the times spent on each execution of the body of the loop. For example, we found in Section 1.2 that the jth iteration of insertion sort took time proportional to j in the worst case. By adding up the time spent on each iteration, we obtained the summation (or series)

Evaluating this summation yielded a bound of [image] (n2) on the worst-case running time of the algorithm. This example indicates the general importance of understanding how to manipulate and bound summations. (As we shall see in Chapter 4, summations also arise when we use certain methods to solve recurrences.)

Section 3.1 lists several basic formulas involving summations. Section 3.2 offers useful techniques for bounding summations. The formulas in Section 3.1 are given without proof, though proofs for some of them are presented in Section 3.2 to illustrate the methods of that section. Most of the other proofs can be found in any calculus text.

3.1 Summation formulas and properties

Given a sequence a1, a2, . . . of numbers, the finite sum a1 + a2 + . . . +an can be written

If n = 0, the value of the summation is defined to be 0. If n is not an integer, we assume that the upper limit is [image] n [image] . Similarly, if the sum begins with k = x, where x is not an integer, we assume that the initial value for the summation is [image] x [image] . (Generally, we shall put in floors and ceilings explicitly.) The value of a finite series is always well defined, and its terms can be added in any order.

Given a sequence a1, a2, . . . of numbers, the infinite sum a1 + a2 + [image] [image] [image] can be written

which is interpreted to mean

If the limit does not exist, the series diverges; otherwise, it converges. The terms of a convergent series cannot always be added in any order. We can, however, rearrange the terms of an absolutely convergent series, that is, a series [image] for which the series [image] also converges.

Linearity

For any real number c and any finite sequences a1, a2, . . . , an and b1, b2, . . . , bn,

The linearity property is also obeyed by infinite convergent series.

The linearity property can be exploited to manipulate summations incorporating asymptotic notation. For example,

In this equation, the [image] -notation on the left-hand side applies to the variable k, but on the right-hand side, it applies to n. Such manipulations can also be applied to infinite convergent series.

Arithmetic series

The summation

which came up when we analyzed insertion sort, is an arithmetic series and has the value

(3.1)

(3.2)

Geometric series

For real x [image] 1, the summation

is a geometric or exponential series and has the value

(3.3)

When the summation is infinite and |x| < 1, we have the infinite decreasing geometric series

(3.4)

Harmonic series

For positive integers n, the nth harmonic number is

(3.5)

Integrating and differentiating series

Additional formulas can be obtained by integrating or differentiating the formulas above. For example, by differentiating both sides of the infinite geometric series (3.4) and multiplying by x, we get

(3.6)

Telescoping series

For any sequence a0, a1, . . . , an,

(3.7)

since each of the terms a1, a2, . . . , an-1 is added in exactly once and subtracted out exactly once. We say that the sum telescopes. Similarly,

As an example of a telescoping sum, consider the series

Since we can rewrite each term as

we get

Products

The finite product a1 a2 [image] [image] [image] an can be written

If n = 0, the value of the product is defined to be 1. We can convert a formula with a product to a formula with a summation by using the identity

Exercises

3.1-1

Find a simple formula for

3.1-2

Show that [image] by manipulating the harmonic series.

3.1-3

Show that

3.1-4

Evaluate the sum

3.1-5

Use the linearity property of summations to prove that

3.1-6

Prove that

3.1-7

Evaluate the product

3.1-8

Evaluate the product

3.2 Bounding summations

There are many techniques available for bounding the summations that describe the running times of algorithms. Here are some of the most frequently used methods.

Mathematical induction

The most basic way to evaluate a series is to use mathematical induction. As an example, let us prove that the arithmetic series [image] evaluates to 1/2n(n + 1). We can easily verify this for n = 1, so we make the inductive assumption that it holds for n and prove that it holds for n + 1. We have

One need not guess the exact value of a summation in order to use mathematical induction. Induction can be used to show a bound as well. As an example, let us prove that the geometric series [image] is 0(3n). More specifically, let us prove that [image] for some constant c. For the initial condition n = 0, we have [image] as long as c [image] 1. Assuming that the bound holds for n, let us prove that it holds for n + 1. We have

as long as (1/3 + 1/c) [image] 1 or, equivalently, c [image] 3/2. Thus, [image], as we wished to show.

We have to be careful when we use asymptotic notation to prove bounds by induction. Consider the following fallacious proof that [image]. Certainly, [image]. Assuming the bound for n, we now prove it for n+1:

The bug in the argument is that the "constant" hidden by the "big-oh" grows with n and thus is not constant. We have not shown that the same constant works for all n.

Bounding the terms

Sometimes, a good upper bound on a series can be obtained by bounding each term of the series, and it often suffices to use the largest term to bound the others. For example, a quick upper bound on the arithmetic series (3.1) is

In general, for a series [image], then

The technique of bounding each term in a series by the largest term is a weak method when the series can in fact be bounded by a geometric series. Given the series [image] suppose that ak+1/ak [image] r for all k [image] 0, where r < 1 is a constant. The sum can be bounded by an infinite decreasing geometric series, since ak [image] a0rk, and thus

We can apply this method to bound the summation [image]. The first term is 1/3, and the ratio of consecutive terms is

for all k [image] 1. Thus, each term is bounded above by (1/3)(2/3)k, so that

A common bug in applying this method is to show that the ratio of consecutive terms is less than 1 and then to assume that the summation is bounded by a geometric series. An example is the infinite harmonic series, which diverges since

The ratio of the (k + 1)st and kth terms in this series is k/(k + 1) < 1, but the series is not bounded by a decreasing geometric series. To bound a series by a geometric series, one must show that the ratio is bounded away from 1; that is, there must be an r < 1, which is a constant, such that the ratio of all pairs of consecutive terms never exceeds r. In the harmonic series, no such r exists because the ratio becomes arbitrarily close to 1.

Splitting summations

One way to obtain bounds on a difficult summation is to express the series as the sum of two or more series by partitioning the range of the index and then to bound each of the resulting series. For example, suppose we try to find a lower bound on the arithmetic series [image], which has already been shown to have an upper bound of n2. We might attempt to bound each term in the summation by the smallest term, but since that term is 1, we get a lower bound of n for the summation--far off from our upper bound of n2.

We can obtain a better lower bound by first splitting the summation. Assume for convenience that n is even. We have

which is an asymptotically tight bound, since [image].

For a summation arising from the analysis of an algorithm, we can often split the summation and ignore a constant number of the initial terms. Generally, this technique applies when each term ak in a summation [image] is independent of n. Then for any constant k0 > 0, we can write

since the initial terms of the summation are all constant and there is a constant number of them. We can then use other methods to bound [image]. For example, to find an asymptotic upper bound on

we observe that the ratio of consecutive terms is

if k [image] 3. Thus, the summation can be split into

since the second summation is a decreasing geometric series.

The technique of splitting summations can be used to determine asymptotic bounds in much more difficult situations. For example, we can obtain a bound of 0(lg n) on the harmonic series (3.5):

The idea is to split the range 1 to n into [image] 1g n [image] pieces and upper bound the contribution of each piece by 1. Thus,

(3.8)

Approximation by integrals

When a summation can be expressed as [image], where â(k) is a monotonically increasing function, we can approximate it by integrals:

(3.9)

The justification for this approximation is shown in Figure 3.1. The summation is represented as the area of the rectangles in the figure, and the integral is the shaded region under the curve. When â(k) is a monotonically decreasing function, we can use a similar method to provide the bounds

(3.10)

The integral approximation (3.10) gives a tight estimate for the nth harmonic number. For a lower bound, we obtain

(3.11)

For the upper bound, we derive the inequality

which yields the bound

(3.12)

Figure 3.1 Approximation of by integrals. The area of each rectangle is shown within the rectangle, and the total rectangle area represents the value of the summation. The integral is represented by the shaded area under the curve. By comparing areas in (a), we get , and then by shifting the rectangles one unit to the right, we get .in (b).

Exercises

3.2-1

Show that [image] is bounded above by a constant.

3.2-2

Find an asymptotic upper bound on the summation

3.2-3

Show that the nth harmonic number is [image] (1g n) by splitting the summation.

3.2-4

Approximate [image] with an integral.

3.2-5

Why didn't we use the integral approximation (3.10) directly on [image] to obtain an upper bound on the nth harmonic number?

Problems

3-1 Bounding summations

Give asymptotically tight bounds on the following summations. Assume that r [image] 0 and s [image] 0 are constants.

Chapter notes

Knuth [121] is an excellent reference for the material presented in this chapter. Basic properties of series can be found in any good calculus book, such as Apostol [12] or Thomas and Finney [192].

Go to Chapter 4 Back to Table of Contents


Intro to Algorithms: CHAPTER 4: RECURRENCES

 Return to Table of Contents

CHAPTER 4: RECURRENCES

As noted in Chapter 1, when an algorithm contains a recursive call to itself, its running time can often be described by a recurrence. A recurrence is an equation or inequality that describes a function in terms of its value on smaller inputs. For example, we saw in Chapter 1 that the worst-case running time T(n) of the MERGE-SORT procedure could be described by the recurrence

(4.1)

whose solution was claimed to be T(n) = [image] (n lg n).

This chapter offers three methods for solving recurrences--that is, for obtaining asymptotic " [image] " or "O" bounds on the solution. In the substitution method, we guess a bound and then use mathematical induction to prove our guess correct. The iteration method converts the recurrence into a summation and then relies on techniques for bounding summations to solve the recurrence. The master method provides bounds for recurrences of the form

T(n) = aT(n/b) +   (n),

where a [image] 1, b > 1, and [image] (n) is a given function; it requires memorization of three cases, but once you do that, determining asymptotic bounds for many simple recurrences is easy.

Technicalities

In practice, we neglect certain technical details when we state and solve recurrences. A good example of a detail that is often glossed over is the assumption of integer arguments to functions. Normally, the running time T(n) of an algorithm is only defined when n is an integer, since for most algorithms, the size of the input is always an integer. For example, the recurrence describing the worst-case running time of MERGE-SORT is really

(4.2)

Boundary conditions represent another class of details that we typically ignore. Since the running time of an algorithm on a constant-sized input is a constant, the recurrences that arise from the running times of algorithms generally have T(n) = [image] (1) for sufficiently small n. Consequently, for convenience, we shall generally omit statements of the boundary conditions of recurrences and assume that T(n) is constant for small n. For example, we normally state recurrence (4.1) as

T(n) = 2T(n/2) +   (n),

(4.3)

without explicitly giving values for small n. The reason is that although changing the value of T(1) changes the solution to the recurrence, the solution typically doesn't change by more than a constant factor, so the order of growth is unchanged.

When we state and solve recurrences, we often omit floors, ceilings, and boundary conditions. We forge ahead without these details and later determine whether or not they matter. They usually don't, but it is important to know when they do. Experience helps, and so do some theorems stating that these details don't affect the asymptotic bounds of many recurrences encountered in the analysis of algorithms (see Theorem 4.1 and Problem 4-5). In this chapter, however, we shall address some of these details to show the fine points of recurrence solution methods.

4.1 The substitution method

The substitution method for solving recurrences involves guessing the form of the solution and then using mathematical induction to find the constants and show that the solution works. The name comes from the substitution of the guessed answer for the function when the inductive hypothesis is applied to smaller values. This method is powerful, but it obviously can be applied only in cases when it is easy to guess the form of the answer.

The substitution method can be used to establish either upper or lower bounds on a recurrence. As an example, let us determine an upper bound on the recurrence

T(n) = 2T(  n/2  ) + n,

(4.4)

which is similar to recurrences (4.2) and (4.3). We guess that the solution is T(n) = O(n lg n). Our method is to prove that T(n) [image] cn lg n for an appropriate choice of the constant c > 0. We start by assuming that this bound holds for [image] n/2 [image] , that is, that T( [image] n/2 [image] ) [image] c [image] n/2 [image] 1g ( [image] n/2 [image] ). Substituting into the recurrence yields

T(n)    2(c   n/2   1g (  n/2  )) + n

   cn lg(n/2) + n

= cn lg n - cn lg 2 + n

= cn lg n - cn + n

   cn lg n,

where the last step holds as long as c [image] 1.

Mathematical induction now requires us to show that our solution holds for the boundary conditions. That is, we must show that we can choose the constant c large enough so that the bound T(n) [image] cn lg n works for the boundary conditions as well. This requirement can sometimes lead to problems. Let us assume, for the sake of argument, that T(1) = 1 is the sole boundary condition of the recurrence. Then, unfortunately, we can't choose c large enough, since T(1) [image] c1 lg 1 = 0.

This difficulty in proving an inductive hypothesis for a specific boundary condition can be easily overcome. We take advantage of the fact that asymptotic notation only requires us to prove T(n) [image] cn lg n for n [image] n0, where n0 is a constant. The idea is to remove the difficult boundary condition T(1) = 1 from consideration in the inductive proof and to include n = 2 and n = 3 as part of the boundary conditions for the proof. We can impose T(2) and T(3) as boundary conditions for the inductive proof because for n > 3, the recurrence does not depend directly on T(1). From the recurrence, we derive T(2) = 4 and T(3) = 5. The inductive proof that T(n) [image] cn lg n for some constant c [image] 2 can now be completed by choosing c large enough so that T(2) [image] c2 lg 2 and T(3) [image] c3 lg 3. As it turns out, any choice of c [image] 2 suffices. For most of the recurrences we shall examine, it is straightforward to extend boundary conditions to make the inductive assumption work for small n.

Making a good guess

Unfortunately, there is no general way to guess the correct solutions to recurrences. Guessing a solution takes experience and, occasionally, creativity. Fortunately, though, there are some heuristics that can help you become a good guesser.

If a recurrence is similar to one you have seen before, then guessing a similar solution is reasonable. As an example, consider the recurrence

T(n) = 2T(  n/2   + 17) + n,

which looks difficult because of the added "17" in the argument to T on the right-hand side. Intuitively, however, this additional term cannot substantially affect the solution to the recurrence. When n is large, the difference between T( [image] n/2 [image] ) and T( [image] n/2 [image] + 17) is not that large: both cut n nearly evenly in half. Consequently, we make the guess that T(n) = O(n lg n), which you can verify as correct by using the substitution method (see Exercise 4.1-5).

Another way to make a good guess is to prove loose upper and lower bounds on the recurrence and then reduce the range of uncertainty. For example, we might start with a lower bound of T(n) = [image] (n) for the recurrence (4.4), since we have the term n in the recurrence, and we can prove an initial upper bound of T(n) = O(n2). Then, we can gradually lower the upper bound and raise the lower bound until we converge on the correct, asymptotically tight solution of T(n) = [image] (n lg n).

Subtleties

There are times when you can correctly guess at an asymptotic bound on the solution of a recurrence, but somehow the math doesn't seem to work out in the induction. Usually, the problem is that the inductive assumption isn't strong enough to prove the detailed bound. When you hit such a snag, revising the guess by subtracting a lower-order term often permits the math to go through.

Consider the recurrence

T(n) = T(  n/2  ) + T (  n/2  ) + 1.

We guess that the solution is O(n), and we try to show that T(n) [image] cn for an appropriate choice of the constant c. Substituting our guess in the recurrence, we obtain

T(n)    c   n/2   + c   n/2   + 1

= cn + 1,

which does not imply T(n) [image] cn for any choice of c. It's tempting to try a larger guess, say T(n) = O(n2), which can be made to work, but in fact, our guess that the solution is T(n) = O(n) is correct. In order to show this, however, we must make a stronger inductive hypothesis.

Intuitively, our guess is nearly right: we're only off by the constant 1, a lower-order term. Nevertheless, mathematical induction doesn't work unless we prove the exact form of the inductive hypothesis. We overcome our difficulty by subtracting a lower-order term from our previous guess. Our new guess is T(n) [image] cn - b, where b [image] 0 is constant. We now have

T(n)    (c  n/2   - b) + (c  n/2   - b) + 1

= cn - 2b + 1

   cn - b ,

as long as b [image] 1. As before, the constant c must be chosen large enough to handle the boundary conditions.

Most people find the idea of subtracting a lower-order term counterintuitive. After all, if the math doesn't work out, shouldn't we be increasing our guess? The key to understanding this step is to remember that we are using mathematical induction: we can prove something stronger for a given value by assuming something stronger for smaller values.

Avoiding pitfalls

It is easy to err in the use of asymptotic notation. For example, in the recurrence (4.4) we can falsely prove T(n) = O(n) by guessing T(n) [image] cn and then arguing

T(n)    2(c  n/2  ) + n

   cn + n

= O(n),         wrong!!

since c is a constant. The error is that we haven't proved the exact form of the inductive hypothesis, that is, that T(n) [image] cn.

Changing variables

Sometimes, a little algebraic manipulation can make an unknown recurrence similar to one you have seen before. As an example, consider the recurrence

which looks difficult. We can simplify this recurrence, though, with a change of variables. For convenience, we shall not worry about rounding off values, such as [image], to be integers. Renaming m = 1g n yields

T(2m) = 2T(2m/2) + m.

We can now rename S(m) = T(2m) to produce the new recurrence

S(m) = 2S(m/2) + m,

which is very much like recurrence (4.4) and has the same solution: S(m) = O(m lg m). Changing back from S(m) to T(n), we obtain T(n) = T(2m) = S(m) = O(m lg m) = O(lg n lg lg n).

Exercises

4.1-1

Show that the solution of T(n) = T( [image] n/2 [image] ) + 1 is O(lg n).

4.1-2

Show that the solution of T(n) = 2T( [image] n/2 [image] ) + n is [image] (n lg n). Conclude that the solution is [image] (n lg n).

4.1-3

Show that by making a different inductive hypothesis, we can overcome the difficulty with the boundary condition T(1) = 1 for the recurrence (4.4) without adjusting the boundary conditions for the inductive proof.

4.1-4

Show that [image] (n lg n) is the solution to the "exact" recurrence (4.2) for merge sort.

4.1-5

Show that the solution to T(n) = 2T( [image] n/2 [image] + 17) + n is O(n lg n).

4.1-6

Solve the recurrence [image] by making a change of variables. Do not worry about whether values are integral.

4.2 The iteration method

The method of iterating a recurrence doesn't require us to guess the answer, but it may require more algebra than the substitution method. The idea is to expand (iterate) the recurrence and express it as a summation of terms dependent only on n and the initial conditions. Techniques for evaluating summations can then be used to provide bounds on the solution.

As an example, consider the recurrence

T(n) = 3T(  n/4  ) + n.

We iterate it as follows:

T(n) = n + 3T(  n/4  )

= n + 3 (  n/4   + 3T(  n/16  ))

= n + 3(  n/4   + 3(  n/16   + 3T(  n/64  )))

= n + 3   n/4   + 9   n/16   + 27T(  n/64  ),

where [image] [image] n/4 [image] /4 [image] = [image] n/16 [image] and [image] [image] n/16 [image] /4 [image] = [image] n/64 [image] follow from the identity (2.4).

How far must we iterate the recurrence before we reach a boundary condition? The ith term in the series is 3i [image] n/4i [image]. The iteration hits n = 1 when [image] n/4i [image] = 1 or, equivalently, when i exceeds log4 n. By continuing the iteration until this point and using the bound [image] n/4i [image] [image] n/4i, we discover that the summation contains a decreasing geometric series:

Here, we have used the identity (2.9) to conclude that 3log4 n = nlog43, and we have used the fact that log4 3 < 1 to conclude that [image] (nlog43) = o(n).

The iteration method usually leads to lots of algebra, and keeping everything straight can be a challenge. The key is to focus on two parameters: the number of times the recurrence needs to be iterated to reach the boundary condition, and the sum of the terms arising from each level of the iteration process. Sometimes, in the process of iterating a recurrence, you can guess the solution without working out all the math. Then, the iteration can be abandoned in favor of the substitution method, which usually requires less algebra.

When a recurrence contains floor and ceiling functions, the math can become especially complicated. Often, it helps to assume that the recurrence is defined only on exact powers of a number. In our example, if we had assumed that n = 4k for some integer k, the floor functions could have been conveniently omitted. Unfortunately, proving the bound T(n) = O(n) solely for exact powers of 4 is technically an abuse of the O-notation. The definitions of asymptotic notation require that bounds be proved for all sufficiently large integers, not just those that are powers of 4. We shall see in Section 4.3 that for a large class of recurrences, this technicality can be overcome. Problem 4-5 also gives conditions under which an analysis for exact powers of an integer can be extended to all integers.

Recursion trees

Exercises

4.3 The master method

The master method provides a "cookbook" method for solving recurrences of the form

T(n) = aT(n/b)+â(n),

(4.5)

where a [image] 1 and b > 1 are constants and â(n) is an asymptotically positive function. The master method requires memorization of three cases, but then the solution of many recurrences can be determined quite easily, often without pencil and paper.

The recurrence (4.5) describes the running time of an algorithm that divides a problem of size n into a subproblems, each of size n/b, where a and b are positive constants. The a subproblems are solved recursively, each in time T(n/b). The cost of dividing the problem and combining the results of the subproblems is described by the function â(n). (That is, using the notation from Section 1.3.2, â(n) = D(n) + C(n).) For example, the recurrence arising from the MERGE-SORT procedure has a = 2, b = 2, and â(n) = [image] (n).

As a matter of technical correctness, the recurrence isn't actually well defined because n/b might not be an integer. Replacing each of the a terms T(n/b) with either T( [image] n/b [image] ) or T( [image] n/b [image] ) doesn't affect the asymptotic behavior of the recurrence, however. (We'll prove this in the next section.) We normally find it convenient, therefore, to omit the floor and ceiling functions when writing divide-and-conquer recurrences of this form.

The master theorem

The master method depends on the following theorem.

Theorem 4.1

Let a [image] 1 and b > 1 be constants, let â(n) be a function, and let T(n) be defined on the nonnegative integers by the recurrence

T(n) = aT(n/b) + â(n),

where we interpret n/b to mean either [image] n/b [image] or [image] n/b [image] . Then T(n) can be bounded asymptotically as follows.

1. If â(n) = O(nlogba- [image] ) for some constant [image] > 0, then T(n) = [image] (nlogba).

2. If â(n) = [image] (nlogba), then T(n) = [image] (nlogbalg n).

3. If â(n) = [image] (nlogba+ [image] ) for some constant [image] > 0, and if aâ(n/b) [image] câ(n) for some constant c < 1 and all sufficiently large n, then T(n) = [image] (â(n)).

Before applying the master theorem to some examples, let's spend a moment trying to understand what it says. In each of the three cases, we are comparing the function â(n) with the function nlogba. Intuitively, the solution to the recurrence is determined by the larger of the two functions. If, as in case 1, the function nlogba is the larger, then the solution is T(n) = [image] (â(nlogba). If, as in case 3, the function â(n) is the larger, then the solution is T(n) = [image] (â(n)). If, as in case 2, the two functions are the same size, we multiply by a logarithmic factor, and the solution is T(n) = [image] (nlogba lg n) = [image] (â(n)lg n).

Beyond this intuition, there are some technicalities that must be understood. In the first case, not only must â(n) be smaller than nlogba, it must be polynomially smaller. That is, â(n) must be asymptotically smaller than nlogba by a factor of n [image] for some constant [image] > 0. In the third case, not only must â(n) be larger than nlogba, it must be polynomially larger and in addition satisfy the "regularity" condition that aâ(n/b) [image] câ(n). This condition is satisfied by most of the polynomially bounded functions that we shall encounter.

It is important to realize that the three cases do not cover all the possibilities for â(n). There is a gap between cases 1 and 2 when â(n) is smaller than nlogba but not polynomially smaller. Similarly, there is a gap between cases 2 and 3 when â(n) is larger than nlogba but not polynomially larger. If the function â(n) falls into one of these gaps, or if the regularity condition in case 3 fails to hold, the master method cannot be used to solve the recurrence.

Using the master method

To use the master method, we simply determine which case (if any) of the master theorem applies and write down the answer.

As a first example, consider

T(n) = 9T(n/3) + n.

For this recurrence, we have a = 9, b = 3, â(n) = n, and thus nlogba = nlog39 = [image] (n2). Since â(n) = 0(nlog39- [image] ), where [image] = 1, we can apply case 1 of the master theorem and conclude that the solution is T(n) = [image] (n2).

Now consider

T(n) = T(2n/3) + 1,

in which a = 1, b = 3/2, â(n) = 1, and nlogba = nlog3/21 = n0 = 1. Case 2 applies, since â(n) = [image] (nlogba) = [image] (1), and thus the solution to the recurrence is T(n) = [image] (lg n).

For the recurrence

T(n) = 3T(n/4) + n lg n,

we have a = 3, b = 4, â(n) = n lg n, and nlogba = nlog43 = 0(n0.793). Since â(n) = [image] (nlog43+ [image] ), where [image] [image] 0.2, case 3 applies if we show that the regularity condition holds for â(n). For sufficiently large n, aâ(n/b) = 3(n/4) lg(n/4) [image] (3/4)n lg n = câ(n) for c = 3/4. Consequently, by case 3, the solution to the recurrence is T(n) = [image] (n lg n).

The master method does not apply to the recurrence

T(n) = 2T(n/2) = + n 1g n,

even though it has the proper form: a = 2, b = 2, â(n) = n1g n, and nlogba = n. It seems that case 3 should apply, since â(n) = n1g n is asymptotically larger than nlogba = n but not polynomially larger. The ratio â(n)/nlogba = (n 1g n)/n = 1g n is asymptotically less than n [image] for any positive constant [image]. Consequently, the recurrence falls into the gap between case 2 and case 3. (See Exercise 4.4-2 for a solution.)

Exercises

4.3-1

Use the master method to give tight asymptotic bounds for the following recurrences.

a. T(n) = 4T(n/2) + n.

b. T(n) = 4T(n/2) + n2.

c. T(n) = 4T(n/2) + n3.

4.3-2

The running time of an algorithm A is described by the recurrence T(n) = 7T(n/2) + n2. A competing algorithm A' has a running time of T'(n) = aT'(n/4) + n2. What is the largest integer value for a such that A' is asymptotically faster than A?

4.3-3

Use the master method to show that the solution to the recurrence T(n) = T(n/2) + [image] (1) of binary search (see Exercise 1.3-5) is T(n) = [image] (1g n).

4.3-4

Consider the regularity condition aâ(n/b) [image] câ(n) for some constant c < 1, which is part of case 3 of the master theorem. Give an example of a simple function â(n) that satisfies all the conditions in case 3 of the master theorem except the regularity condition.

* 4.4 Proof of the master theorem

This section contains a proof of the master theorem (Theorem 4.1) for more advanced readers. The proof need not be understood in order to apply the theorem.

The proof is in two parts. The first part analyzes the "master" recurrence (4.5), under the simplifying assumption that T(n) is defined only on exact powers of b > 1, that is, for n = 1, b, b2, . . .. This part gives all the intuition needed to understand why the master theorem is true. The second part shows how the analysis can be extended to all positive integers n and is merely mathematical technique applied to the problem of handling floors and ceilings.

In this section, we shall sometimes abuse our asymptotic notation slightly by using it to describe the behavior of functions that are only defined over exact powers of b. Recall that the definitions of asymptotic notations require that bounds be proved for all sufficiently large numbers, not just those that are powers of b. Since we could make new asymptotic notations that apply to the set {bi : i = 0,1, . . .}, instead of the nonnegative integers, this abuse is minor.

Nevertheless, we must always be on guard when we are using asymptotic notation over a limited domain so that we do not draw improper conclusions. For example, proving that T(n) = O(n) when n is an exact power of 2 does not guarantee that T(n) = O(n). The function T(n) could be defined as

in which case the best upper bound that can be proved is T(n) = O(n2). Because of this sort of drastic consequence, we shall never use asymptotic notation over a limited domain without making it absolutely clear from the context that we are doing so.

4.4.1 The proof for exact powers

The first part of the proof of the master theorem analyzes the master recurrence (4.5),

T(n) = aT(n/b) + â(n),

under the assumption that n is an exact power of b > 1, where b need not be an integer. The analysis is broken into three lemmas. The first reduces the problem of solving the master recurrence to the problem of evaluating an expression that contains a summation. The second determines bounds on this summation. The third lemma puts the first two together to prove a version of the master theorem for the case in which n is an exact power of b.

Lemma 4.2

Let a [image] 1 and b > 1 be constants, and let â(n) be a nonnegative function defined on exact powers of b. Define T(n) on exact powers of b by the recurrence

where i is a positive integer. Then

(4.6)

Proof Iterating the recurrence yields

T(n) = â(n) + aT(n/b)

= â(n) + aâ(n/b) + a2T(n/b2)

= â(n) + aâ(n/b) + a2â(n/b2) + . . .

+ alogbn-1 f(n/blogbn-1) + alogbnT(1) .

Since alogb n = nlogba, the last term of this expression becomes

alogbn T(1) =   (nlogba);

using the boundary condition T(1) = [image] (1). The remaining terms can be expressed as the sum

thus,

which completes the proof.

The recursion tree

Before proceeding, let's try to develop some intuition by using a recursion tree. Figure 4.3 shows the tree corresponding to the iteration of the recurrence in Lemma 4.2. The root of the tree has cost â(n), and it has a children, each with cost â(n/b). (It is convenient to think of a as being an integer, especially when visualizing the recursion tree, but the mathematics does not require it.) Each of these children has a children with cost â(n/b2), and thus there are a2 nodes that are distance 2 from the root. In general, there are a2 nodes that are distance j from the root, and each has cost â(n/bj). The cost of each leaf is T(1) = [image] (1), and each leaf is distance logbn from the root, since n/blogbn = 1. There are alogbn = nlogba leaves in the tree.

We can obtain equation (4.6) by summing the costs of each level of the tree, as shown in the figure. The cost for a level j of internal nodes is ajâ(n/bj), and so the total of all internal node levels is

In the underlying divide-and-conquer algorithm, this sum represents the costs of dividing problems into subproblems and then recombining the subproblems. The cost of all the leaves, which is the cost of doing all nlogba subproblems of size 1, is [image] (nlogba).

In terms of the recursion tree, the three cases of the master theorem correspond to cases in which the total cost of the tree is (1) dominated by the costs in the leaves, (2) evenly distributed across the levels of the tree, or (3) dominated by the cost of the root.

The summation in equation (4.6) describes the cost of the dividing and combining steps in the underlying divide-and-conquer algorithm. The next lemma provides asymptotic bounds on the summation's growth.

Figure 4.3 The recursion tree generated by T(n) = aT(n/b)+ f(n). The tree is a complete a-ary tree with nlogba leaves and height logb a. The cost of each level is shown at the right, and their sum is given in equation (4.6).

Lemma 4.3

Let a [image] 1 and b > 1 be constants, and let â(n) be a nonnegative function defined on exact powers of b. A function g(n) defined over exact powers of b by

(4.7)

can then be bounded asymptotically for exact powers of b as follows.

1. If â(n) = O(nlogba- [image] ) for some constant [image] > 0, then g(n) = O(nlogba).

2. If â(n) = [image] (nlogba), then g(n) = [image] (nlogba 1g n).

3. If aâ(n/b) [image] câ(n) for some constant c < 1 and all n [image] b, then g(n) = [image] (â(n)).

Proof For case 1, we have â(n) = O(nlogba- [image]), implying that â(n/bj) = O((n/bj)logba- [image] ). Substituting into equation (4.7) yields

(4.8)

We bound the summation within the O-notation by factoring out terms and simplifying, which leaves an increasing geometric series:

Since b and [image] are constants, the last expression reduces to nlogba- [image] O(n [image] ) = O(nlogba). Substituting this expression for the summation in equation (4.8) yields

g(n) = O(nlogba),

and case 1 is proved.

Under the assumption that â(n) = [image] (nlogba) for case 2, we have that â(n/bj) = [image] ((n/bj)logba). Substituting into equation (4.7) yields

(4.9)

We bound the summation within the [image] as in case 1, but this time we do not obtain a geometric series. Instead, we discover that every term of the summation is the same:

Substituting this expression for the summation in equation (4.9) yields

g(n) =   (nlogbalogbn)

=   (nlogba 1g n),

and case 2 is proved.

Case 3 is proved similarly. Since [image] (n) appears in the definition (4.7) of g(n) and all terms of g(n) are nonnegative, we can conclude that g(n) = [image] ( [image] (n)) for exact powers of b. Under the assumption that af (n/b) [image] cf(n) for some constant c < 1 and all n [image] b, we have aj [image] (n/bj) [image] cj [image] (n). Substituting into equation (4.7) and simplifying yields a geometric series, but unlike the series in case 1, this one has decreasing terms:

since c is constant. Thus, we can conclude that g(n) = [image] ( [image] (n)) for exact powers of b. Case 3 is proved, which completes the proof of the lemma.

We can now prove a version of the master theorem for the case in which n is an exact power of b.

Lemma 4.4

Let a [image] 1 and b > 1 be constants, and let [image] (n) be a nonnegative function defined on exact powers of b. Define T(n) on exact powers of b by the recurrence

where i is a positive integer. Then T(n) can be bounded asymptotically for exact powers of b as follows.

1. If [image] (n) = O(nlogba- [image]) for some constant [image] > 0, then T(n) = [image] (nlogba).

2. If [image] (n) = [image] (nlogba), then T(n) = [image] (nlogba lg n).

3. If [image] (n) = [image] (nlogba+ [image]) for some constant [image] > 0, and if a [image] (n/b) [image] c [image] (n) for some constant c < 1 and all sufficiently large n, then T(n) = [image] ( [image] (n)).

Proof We use the bounds in Lemma 4.3 to evaluate the summation (4.6) from Lemma 4.2. For case 1, we have

T(n) =   (nlogba) + O(nlogba)

=   (nlogba),

and for case 2,

T(n) =   (nlogba) +   (nlogba 1g n)

=   (nlogba lg n).

For case 3, the condition a [image] (n/b) [image] c [image] (n) implies [image] (n) = [image] (nlogba+ [image]) (see Exercise 4.4-3). Consequently,

T(n) =   (nlogba) +   (  (n))

=   (  (n)).

4.4.2 Floors and ceilings

To complete the proof of the master theorem, we must now extend our analysis to the situation in which floors and ceilings are used in the master recurrence, so that the recurrence is defined for all integers, not just exact powers of b. Obtaining a lower bound on

T(n) = aT(  n/b  ) +   (n)

(4.10)

and an upper bound on

T(n) = aT(  n/b  ) +   (n)

(4.11)

is routine, since the bound [image] n/b [image] [image] n/b can be pushed through in the first case to yield the desired result, and the bound [image] n/b [image] [image] n/b can be pushed through in the second case. Lower bounding the recurrence (4.11) requires much the same technique as upper bounding the recurrence (4.10), so we shall only present this latter bound.

We wish to iterate the recurrence (4.10), as was done in Lemma 4.2. As we iterate the recurrence, we obtain a sequence of recursive invocations on the arguments

n,

  n/b   ,

    n/b  /b   ,

      n/b  /b  /b   ,

Let us denote the ith element in the sequence by ni, where

(4.12)

Our first goal is to determine the number of iterations k such that nk is a constant. Using the inequality [image] x [image] [image] x + 1, we obtain

In general,

and thus, when i = [image] logb n [image] , we obtain ni [image] b + b / (b - 1) = O(1).

We can now iterate recurrence (4.10), obtaining

(4.13)

which is much the same as equation (4.6), except that n is an arbitrary integer and not restricted to be an exact power of b.

We can now evaluate the summation

(4.14)

from (4.13) in a manner analogous to the proof of Lemma 4.3. Beginning with case 3, if af( [image] n/b [image] ) [image] cf(n) for n > b + b/(b - 1), where c < 1 is a constant, then it follows that aj f(nj) [image] cjf(n). Therefore, the sum in equation (4.14) can be evaluated just as in Lemma 4.3. For case 2, we have f(n) = [image] (nlogba). If we can show that f(nj) = O(nlogba/aj) = O((n/bj)logba), then the proof for case 2 of Lemma 4.3 will go through. Observe that j [image] [image] logbn [image] implies bj/n [image] 1. The bound f(n) = O(nlogba) implies that there exists a constant c > 0 such that for sufficiently large nj,

since c(1 + b/(b - 1))logba is a constant. Thus, case 2 is proved. The proof of case 1 is almost identical. The key is to prove the bound f(nj) = O(nlogba- [image] ), which is similar to the corresponding proof of case 2, though the algebra is more intricate.

We have now proved the upper bounds in the master theorem for all integers n. The proof of the lower bounds is similar.

Exercises

4.4-1

Give a simple and exact expression for ni in equation (4.12) for the case in which b is a positive integer instead of an arbitrary real number.

4.4-2

Show that if f(n) = [image] (nlogba 1gk n), where k [image] 0, then the master recurrence has solution T(n) = [image] (nlogba 1gk+1 n). For simplicity, confine your analysis to exact powers of b.

4.4-3

Show that case 3 of the master theorem is overstated, in the sense that the regularity condition af(n/b) [image] cf(n) for some constant c < 1 implies that there exists a constant [image] > 0 such that f(n) = [image] (nlogba+ [image] ).

Problems

4-1 Recurrence examples

Give asymptotic upper and lower bounds for T(n) in each of the following recurrences. Assume that T(n) is constant for n [image] 2. Make your bounds as tight as possible, and justify your answers.

a. T(n) = 2T [image] (n/2) + n3.

b. T(n) = T(9n/10) + n.

c. T(n) = 16T(n/4) + n2.

d. T(n) = 7T(n/3) + n2.

e. T(n) = 7T(n/2) + n2.

g. T(n) = T(n - 1) + n.

4-2 Finding the missing integer

An array A[1 . . n] contains all the integers from 0 to n except one. It would be easy to determine the missing integer in O(n) time by using an auxiliary array B[0 . . n] to record which numbers appear in A. In this problem, however, we cannot access an entire integer in A with a single operation. The elements of A are represented in binary, and the only operation we can use to access them is "fetch the jth bit of A[i]," which takes constant time.

Show that if we use only this operation, we can still determine the missing integer in O(n) time.

4-3 Parameter-passing costs

Throughout this book, we assume that parameter passing during procedure calls takes constant time, even if an N-element array is being passed. This assumption is valid in most systems because a pointer to the array is passed, not the array itself. This problem examines the implications of three parameter-passing strategies:

1. An array is passed by pointer. Time = (1).

2. An array is passed by copying. Time = (N), where N is the size of the array.

3. An array is passed by copying only the subrange that might be accessed by the called procedure. Time = (p - q + 1 ) if the subarray A[p . . q] is passed.

a. Consider the recursive binary search algorithm for finding a number in a sorted array (see Exercise 1.3-5). Give recurrences for the worst-case running times of binary search when arrays are passed using each of the three methods above, and give good upper bounds on the solutions of the recurrences. Let N be the size of the original problem and n be the size of a subproblem.

b. Redo part (a) for the MERGE-SORT algorithm from Section 1.3.1.

4-4 More recurrence examples

Give asymptotic upper and lower bounds for T(n) in each of the following recurrences. Assume that T(n) is constant for n 2. Make your bounds as tight as possible, and justify your answers.

a. T(n) = 3T(n/2) + n 1g n.

b. T(n) = 3T(n/3 + 5) + n / 2.

c. T(n) = 2T(n/2) + n / lg n.

d. T(n) = T(n - 1) + 1 / n.

e. T(n) = T(n - 1) + 1g n.

4-5 Sloppiness conditions

Often, we are able to bound a recurrence T(n) at exact powers of an integral constant b. This problem gives sufficient conditions for us to extend the bound to all real n > 0.

a. Let T(n) and h(n) be monotonically increasing functions, and suppose that T(n) < h(n) when n is an exact power of a constant b > 1. Moreover, suppose that h(n) is "slowly growing" in the sense that h(n) = O(h(n/b)). Prove that T(n) = O(h(n)).

b. Suppose that we have the recurrence T(n) = aT(n/b) + f(n), where a 1, b > 1, and f(n) is monotonically increasing. Suppose further that the initial conditions for the recurrence are given by T(n) = g(n) for n n0, where g(n) is monotonically increasing and g(n0) < aT(n0/b)+ f(n0). Prove that T(n) is monotonically increasing.

c. Simplify the proof of the master theorem for the case in which f(n) is monotonically increasing and slowly growing. Use Lemma 4.4.

4-6 Fibonacci numbers

This problem develops properties of the Fibonacci numbers, which are defined by recurrence (2.13). We shall use the technique of generating functions to solve the Fibonacci recurrence. Define the generating function (or formal power series) as

a. Show that

b. Show that

where

and

c. Show that

d. Prove that , rounded to the nearest integer. Hint: )

e. Prove that Fi + 2 i for i 0.

4-7 VLSI chip testing

Professor Diogenes has n supposedly identical VLSI1 chips that in principle are capable of testing each other. The professor's test jig accommodates two chips at a time. When the jig is loaded, each chip tests the other and reports whether it is good or bad. A good chip always reports accurately whether the other chip is good or bad, but the answer of a bad chip cannot be trusted. Thus, the four possible outcomes of a test are as follows:

Chip A says  Chip B says  Conclusion

--------------------------------------------------------

B is good    A is good    both are good, or both are bad

B is good    A is bad     at least one is bad

B is bad     A is good    at least one is bad

B is bad     A is bad     at least one is bad

1 VLSI stands for "very-large-scale integration," which is the integrated-circuit chip technology used to fabricate most microprocessors today.

a. Show that if more than n/2 chips are bad, the professor cannot necessarily determine which chips are good using any strategy based on this kind of pairwise test. Assume that the bad chips can conspire to fool the professor.

b. Consider the problem of finding a single good chip from among n chips, assuming that more than n/2 of the chips are good. Show that n/2 pairwise tests are sufficient to reduce the problem to one of nearly half the size.

c. Show that the good chips can be identified with (n) pairwise tests, assuming that more than n/2 of the chips are good. Give and solve the recurrence that describes the number of tests.

Chapter notes

Recurrences were studied as early as 1202 by L. Fibonacci, for whom the Fibonacci numbers are named. A. De Moivre introduced the method of generating functions (see Problem 4-6) for solving recurrences. The master method is adapted from Bentley, Haken, and Saxe [26], which provides the extended method justified by Exercise 4.4-2. Knuth [121] and Liu [140] show how to solve linear recurrences using the method of generating functions. Purdom and Brown [164] contains an extended discussion of recurrence solving.

Go to Chapter 5 Back to Table of Contents


Intro to Algorithms: CHAPTER 5: SETS, ETC.

 Return to Table of Contents

CHAPTER 5: SETS, ETC.

In earlier chapters, we touched on the elements of discrete mathematics. This chapter reviews more completely the notations, definitions, and elementary properties of sets, relations, functions, graphs, and trees. Readers already well versed in this material need only skim this chapter.

5.1 Sets

A set is a collection of distinguishable objects, called its members or elements. If an object x is a member of a set S, we write x [image] S (read "x is a member of S" or, more briefly, "x is in S"). If x is not a member of S, we write x [image] S. We can describe a set by explicitly listing its members as a list inside braces. For example, we can define a set S to contain precisely the numbers 1, 2, and 3 by writing S = {1, 2, 3}. Since 2 is a member of the set S, we can write 2 [image] S, and since 4 is not a member, we have 4 [image] S. A set cannot contain the same object more than once, and its elements are not ordered. Two sets A and B are equal, written A = B, if they contain the same elements. For example, {1, 2, 3, 1} = {1, 2, 3} = {3, 2, 1}.

We adopt special notations for frequently encountered sets.

denotes the empty set, that is, the set containing no members.

Z denotes the set of integers, that is, the set {. . . , 2, -1, 0, 1, 2, . . .}.

R denotes the set of real numbers.

N denotes the set of natural numbers, that is, the set {0, 1, 2, . . .}.1

1Some authors start the natural numbers with 1 instead of 0. The modern trend seems to be to start with 0.

If all the elements of a set A are contained in a set B, that is, if x [image] A implies x [image] B, then we write A [image] B and say that A is a subset of B. A set A is a proper subset of B, written A [image] B, if A [image] B but A [image] B. (Some authors use the symbol " [image] to denote the ordinary subset relation, rather than the proper-subset relation.) For any set A, we have A [image] A. For two sets A and B, we have A = B if and only if A [image] B and B [image] A. For any three sets A, B, and C, if A [image] B and B [image] C, then A [image] C. For any set A, we have [image] [image] A.

We sometimes define sets in terms of other sets. Given a set A, we can define a set B [image] A by stating a property that distinguishes the elements of B. For example, we can define the set of even integers by {x : x [image] Z and x/2 is an integer}. The colon in this notation is read "such that." (Some authors use a vertical bar in place of the colon.)

Given two sets A and B, we can also define new sets by applying set operations:

The intersection of sets A and B is the set

A    B = {x : x    A and x    B} .

The union of sets A and B is the set

A    B = {x : x    A and x    B} .

The difference between two sets A and B is the set

A - B = {x : x    A and x    B} .

Set operations obey the following laws.

Empty set laws:

Idempotency laws:

A    A = A,

A    A = A.

Commutative laws:

A    B = B    A,

A    B = B    A.

Associative laws:

A    (B    C) = (A    B)    C,

A    (B    C) = (A    B)    C.

Distributive laws:

A    (B    C) = (A    B)    (A    C),

A    (B    C) = (A    B)    (A    C).

(5.1)

Absorption laws:

A    (A    B) = A,

A    (A    B) = A.

DeMorgan's laws:

A - (B    C) = (A - B)    (A - C),

A - (B    C) = (A - B)    (A - C).

(5.2)

Figure 5.1 A Venn diagram illustrating the first of DeMorgan's laws (5.2). Each of the sets A, B, and C is represented as a circle in the plane.

The first of DeMorgan's laws is illustrated in Figure 5.1, using a Venn diagram, a graphical picture in which sets are represented as regions of the plane.

Often, all the sets under consideration are subsets of some larger set U called the universe. For example, if we are considering various sets made up only of integers, the set Z of integers is an appropriate universe. Given a universe U, we define the complement of a set A as [image]. For any set A [image] U, we have the following laws:

DeMorgan's laws (5.2) can be rewritten with complements. For any two sets A, B [image] U, we have

Two sets A and B are disjoint if they have no elements in common, that is, if [image]. A collection S = {Si} of nonempty sets forms a partition of a set S if

the sets are pairwise disjoint, that is, Si, Sj [image] S and i [image] j imply [image], and

their union is S, that is,

In other words, S forms a partition of S if each element of S appears in exactly one Si [image] S.

The number of elements in a set is called the cardinality (or size) of the set, denoted |S|. Two sets have the same cardinality if their elements can be put into a one-to-one correspondence. The cardinality of the empty set is [image]. If the cardinality of a set is a natural number, we say the set is finite; otherwise, it is infinite. An infinite set that can be put into a one-to-one correspondence with the natural numbers N is countably infinite; otherwise, it is uncountable. The integers Z are countable, but the reals R are uncountable.

For any two finite sets A and B, we have the identity

|A    B| = |A| + |B| - |A    B| ,

(5.3)

from which we can conclude that

|A    B|    |A| + |B| .

If A and B are disjoint, then |A [image] B| = 0 and thus |A [image] B| = |A| + |B|. If A [image] B, then |A| [image] |B|.

A finite set of n elements is sometimes called an n-set. A 1-set is called a singleton. A subset of k elements of a set is sometimes called a k-subset.

The set of all subsets of a set S, including the empty set and the set S itself, is denoted 2S and is called the power set of S. For example, [image]. The power set of a finite set S has cardinality 2|S|.

We sometimes care about setlike structures in which the elements are ordered. An ordered pair of two elements a and b is denoted (a, b) and can be defined formally as the set (a, b) = {a, {a, b}}. Thus, the ordered pair (a, b) is not the same as the ordered pair (b, a).

The Cartesian product of two sets A and B, denoted A [image] B, is the set of all ordered pairs such that the first element of the pair is an element of A and the second is an element of B. More formally,

A    B = {(a,b) : a    A and b    B}.

For example, {a, b} [image] {a, b, c} = {(a, a), (a, b), (a, c), (b, a), (b, b), (b, c)}. When A and B are finite sets, the cardinality of their Cartesian product is

|A    B| = |A|    |B| .

(5.4)

The Cartesian product of n sets Al, A2,. . . , An is the set of n-tuples

A1    A2              An = {(a1,a2,...,an):ai    Ai,i = 1, 2,...,n} ,

whose cardinality is

|A1    A2              An| = |A1|    |A2|        |An|

if all sets are finite. We denote an n-fold Cartesian product over a single set A by the set

An = A    A              A ,

whose cardinality is |An| = |A|n if A is finite. An n-tuple can also be viewed as a finite sequence of length n (see page 84).

Exercises

5.1-1

Draw Venn diagrams that illustrate the first of the distributive laws (5.1).

5.1-2

Prove the generalization of DeMorgan's laws to any finite collection of sets:

5.1-3

Prove the generalization of equation (5.3), which is called the principle of inclusion and exclusion:

|A1    A2              An| =

|A1| + |A2| +        + |An|

-|A1    A2| - |A1    A3| -            (all pairs)

+|A1    A2    A3   +                   (all triples)

+(-1)n-1 |A1    A2              An| .

5.1-4

Show that the set of odd natural numbers is countable.

5.1-5

Show that for any finite set S, the power set 2S has 2|S| elements (that is, there are 2|S| distinct subsets of S).

5.1-6

Give an inductive definition for an n-tuple by extending the set-theoretic definition for an ordered pair.

5.2 Relations

A binary relation R on two sets A and B is a subset of the Cartesian product A [image] B. If (a, b) [image] R, we sometimes write a R b. When we say that R is a binary relation on a set A, we mean that R is a subset of A [image] A. For example, the "less than" relation on the natural numbers is the set {(a,b): a,b [image] N and a < b}. An n-ary relation on sets A1, A2, . . . , An is a subset of A1 [image] A2 [image] [image] [image] [image] [image] An.

A binary relation R [image] A [image] A is reflexive if

a R a

for all a [image] A. For example, "=" and " [image] " are reflexive relations on N, but "<" is not. The relation R is symmetric if

a R b implies b R a

for all a, b [image] A. For example, "=" is symmetric, but "<" and " [image] " are not. The relation R is transitive if

a R b and b R c imply a R c

for all a,b,c [image] A. For example, the relations "<," " [image]," and "=" are transitive, but the relation R = {(a, b): a, b [image] N and a = b - 1 } is not, since 3 R 4 and 4 R 5 do not imply 3 R 5.

A relation that is reflexive, symmetric, and transitive is an equivalence relation. For example, "=" is an equivalence relation on the natural numbers, but "<" is not. If R is an equivalence relation on a set A, then for a [image] A, the equivalence class of a is the set [a] = {b [image] A: a R b}, that is, the set of all elements equivalent to A. For example, if we define R = {(a, b): a, b [image] N and a + b is an even number}, then R is an equivalence relation, since a + a is even (reflexive), a + b is even implies b + a is even (symmetric), and a + b is even and b + c is even imply a + c is even (transitive). The equivalence class of 4 is [4] = {0, 2, 4, 6, . . .}, and the equivalence class of 3 is [3] = {1, 3, 5, 7, . . .}. A basic theorem of equivalence classes is the following.

Theorem 5.1

The equivalence classes of any equivalence relation R on a set A form a partition of A, and any partition of A determines an equivalence relation on A for which the sets in the partition are the equivalence classes.

Proof For the first part of the proof, we must show that the equivalence classes of R are nonempty, pairwise-disjoint sets whose union is A. Because R is reflexive, a [image] [a], and so the equivalence classes are nonempty; moreover, since every element a [image] A belongs to the equivalence class [a], the union of the equivalence classes is A. It remains to show that the equivalence classes are pairwise disjoint, that is, if two equivalence classes [a] and [b] have an element c in common, then they are in fact the same set. Now a R c and b R c, which by symmetry and transitivity imply a R b. Thus, for any arbitrary element x [image] [a], we have x R a implies x R b, and thus [a] [image] [b]. Similarly, [b] [image] [a], and thus [a] = [b].

For the second part of the proof, let A = {Ai} be a partition of A, and define R = {(a,b): there exists i such that a [image] Ai and b [image] Ai}. We claim that R is an equivalence relation on A. Reflexivity holds, since a [image] Ai implies a R a. Symmetry holds, because if a R b, then a and b are in the same set Ai, and hence b R a. If a R b and b R c, then all three elements are in the same set, and thus a R c and transitivity holds. To see that the sets in the partition are the equivalence classes of R, observe that if a [image] Ai, then x [image] [a] implies x [image] Ai, and x [image] Ai implies x [image] [a].

A binary relation R on a set A is antisymmetric if

a R b and b R a imply a = b .

For example, the " [image] " relation on the natural numbers is antisymmetric, since a [image] b and b [image] a imply a = b. A relation that is reflexive, antisymmetric, and transitive is a partial order, and we call a set on which a partial order is defined a partially ordered set. For example, the relation "is a descendant of" is a partial order on the set of all people (if we view individuals as being their own descendants).

In a partially ordered set A, there may be no single "maximum" element x such that y R x for all y [image] A. Instead, there may several maximal elements x such that for no y [image] A is it the case that x R y. For example, in a collection of different-sized boxes there may be several maximal boxes that don't fit inside of any other box, yet no single "maximum" box into which any other box will fit.

A partial order R on a set A is a total or linear order if for all a, b [image] A, we have a R b or b R a, that is, if every pairing of elements of A can be related by R. For example, the relation " [image] " is a total order on the natural numbers, but the "is a descendant of" relation is not a total order on the set of all people, since there are individuals neither of whom is descended from the other.

Exercises

5.2-1

Prove that the subset relation " [image] on all subsets of Z is a partial order but not a total order.

5.2-2

Show that for any positive integer n, the relation "equivalent modulo n" is an equivalence relation on the integers. (We say that a [image] b (mod n) if there exists an integer q such that a - b =qn.) Into what equivalence classes does this relation partition the integers?

5.2-3

Give examples of relations that are

a. reflexive and symmetric but not transitive,

b. reflexive and transitive but not symmetric,

c. symmetric and transitive but not reflexive.

5.2-4

Let S be a finite set, and let R be an equivalence relation on S [image] S. Show that if in addition R is antisymmetric, then the equivalence classes of S with respect to R are singletons.

5.2-5

Professor Narcissus claims that if a relation R is symmetric and transitive, then it is also reflexive. He offers the following proof. By symmetry, a R b implies b R a. Transitivity, therefore, implies a R a. Is the professor correct?

5.3 Functions

Given two sets A and B, a function f is a binary relation on A [image] B such that for all a [image] A, there exists precisely one b [image] B such that (a,b) [image] [image]. The set A is called the domain of f, and the set B is called the codomain of f. We sometimes write f: A [image] B; and if (a, b) [image] f, we write b = f (a), since b is uniquely determined by the choice of a.

Intuitively, the function f assigns an element of B to each element of A. No element of A is assigned two different elements of B, but the same element of B can be assigned to two different elements of A. For example, the binary relation

f = {(a,b) : a    N and b = a mod 2 }

is a function f : N [image] {0,1}, since for each natural number a, there is exactly one value b in {0, 1} such that b = a mod 2. For this example 0 = f (0), 1 = f (1), 0 = f (2), etc. In contrast, the binary relation

g = {(a,b) : a    N and a + b is even}

is not a function, since (1, 3) and (1, 5) are both in g, and thus for the choice a = 1, there is not precisely one b such that (a, b) [image] g.

Given a function f : A [image] B, if b = f (a), we say that a is the argument of f and that b is the value of f at a. We can define a function by stating its value for every element of its domain. For example, we might define f (n) = 2n for n [image] N, which means f = {(n,2n): n [image] N}. Two functions f and g are equal if they have the same domain and codomain and if, for all a in the domain, f (a) = g(a).

A finite sequence of length n is a function f whose domain is the set {0, 1, . . . , n - 1}. We often denote a finite sequence by listing its values: f (0), f (1), . . . , f (n - 1) [image]. An infinite sequence is a function whose domain is the set N of natural numbers. For example, the Fibonacci sequence, defined by (2.13), is the infinite sequence [image] 0, 1, 1, 2, 3, 5, 8, 13, 21, . . . .

When the domain of a function f is a Cartesian product, we often omit the extra parentheses surrounding the argument of f. For example, if f : A1 [image] A2 [image] [image] [image] [image] [image] An [image] B, we would write b = f (a1, a2, . . . , an) instead of b = f ((a1, a2, . . . , an)). We also call each ai, an argument to the function f , though technically the (single) argument to f is the n-tuple (a1, a2, . . . ,an).

If f : A [image] B is a function and b = f (a), then we sometimes say that b is the image of a under f . The image of a set A' [image] A under f is defined by

f(A' = {b    B : b = f(a) for some a    A'} .

The range of f is the image of its domain, that is, f (A). For example, the range of the function f : N [image] N defined by f (n) = 2n is f (N) = {m : m = 2n for some n [image] N}.

A function is a surjection if its range is its codomain. For example, the function f(n) = [image] n/2 [image] is a surjective function from N to N, since every element in N appears as the value of f for some argument. In contrast, the function f(n) = 2n is not a surjective function from N to N, since no argument to f can produce 3 as a value. The function f (n) = 2n is, however, a surjective function from the natural numbers to the even numbers. A surjection f : A [image] B is sometimes described as mapping A onto B. When we say that f is onto, we mean that it is surjective.

A function f : A [image] B is an injection if distinct arguments to â produce distinct values, that is, if a [image] a' implies â(a) = â(a'). For example, the function f (n) = 2n is an injective function from N to N, since each even number b is the image under â of at most one element of the domain, namely b/2. The function f (n) = [image] n/2 [image] is not injective, since the value 1 is produced by two arguments: 2 and 3. An injection is sometimes called a one-to-one function.

A function f : A [image] B is a bijection if it is injective and surjective. For example, the function f (n) = (-1)n [image] n/2 [image] is a bijection from N to Z:

0        0 ,

1      -1 ,

2      1 ,

3      -2 ,

4      2 ,

The function is injective, since no element of Z is the image of more than one element of N. It is surjective, since every element of Z appears as the image of some element of N. Hence, the function is bijective. A bijection is sometimes called a one-to-one correspondence, since it pairs elements in the domain and codomain. A bijection from a set A to itself is sometimes called a permutation.

When a function â is bijective, its inverse â-1 is defined as

f-1(b) = a if and only if f(a) = b.

For example, the inverse of the function â(n) = (-1)n [image] n/2 [image] is

Exercises

5.3-1

Let A and B be finite sets, and let f : A [image] B be a function. Show that

a. if f is injective, then |A| [image] |B|;

b. f f is surjective, then |A| [image] |B|.

5.3-2

Is the function f (x) = x + 1 bijective when the domain and the codomain are N? Is it bijective when the domain and the codomain are Z?

5.3-3

Give a natural definition for the inverse of a binary relation such that if a relation is in fact a bijective function, its relational inverse is its functional inverse.

5.3-4

Give a bijection from Z to Z [image] Z.

5.4 Graphs

This section presents two kinds of graphs: directed and undirected. The reader should be aware that certain definitions in the literature differ from those given here, but for the most part, the differences are slight. Section 23.1 shows how graphs can be represented in computer memory.

A directed graph (or digraph) G is a pair (V, E), where V is a finite set and E is a binary relation on V. The set V is called the vertex set of G, and its elements are called vertices (singular: vertex). The set E is called the edge set of G, and its elements are called edges. Figure 5.2(a) is a pictorial representation of a directed graph on the vertex set {1, 2, 3, 4, 5, 6}. Vertices are represented by circles in the figure, and edges are represented by arrows. Note that self-loops--edges from a vertex to itself--are possible.

In an undirected graph G = (V, E), the edge set E consists of unordered pairs of vertices, rather than ordered pairs. That is, an edge is a set {u, v}, where u,v [image] V and u [image] v. By convention, we use the notation (u, v) for an edge, rather than the set notation {u,v}, and (u,v) and (v, u) are considered to be the same edge. In an undirected graph, self-loops are forbidden, and so every edge consists of exactly two distinct vertices. Figure 5.2(b) is a pictorial representation of an undirected graph on the vertex set {1, 2, 3, 4, 5, 6}.

Many definitions for directed and undirected graphs are the same, although certain terms have slightly different meanings in the two contexts. If (u, v) is an edge in a directed graph G = (V, E), we say that (u, v) is incident from or leaves vertex u and is incident to or enters vertex v. For example, the edges leaving vertex 2 in Figure 5.2(a) are (2, 2), (2, 4), and (2, 5). The edges entering vertex 2 are (1, 2) and (2, 2). If (u, v) is an edge in an undirected graph G = (V, E), we say that (u, v) is incident on vertices u and v. In Figure 5.2(b), the edges incident on vertex 2 are (1, 2) and (2, 5).

Figure 5.2 Directed and undirected graphs. (a) A directed graph G = (V, E), where V = {1, 2, 3, 4, 5, 6} and E = {(1, 2), (2, 2), (2, 4), (2, 5), (4, 1), (4, 5), (5, 4), (6, 3)}. The edge (2, 2) is a self-loop. (b) An undirected graph G = (V, E), where V = {1, 2, 3, 4, 5, 6} and E = {(1, 2), (1, 5), (2, 5), (3, 6)}. The vertex 4 is isolated. (c) The subgraph of the graph in part (a) induced by the vertex set {1, 2, 3, 6}.

If (u, v) is an edge in a graph G = (V, E), we say that vertex v is adjacent to vertex u. When the graph is undirected, the adjacency relation is symmetric. When the graph is directed, the adjacency relation is not necessarily symmetric. If v is adjacent to u in a directed graph, we sometimes write u [image] v. In parts (a) and (b) of Figure 5.2, vertex 2 is adjacent to vertex 1, since the edge (1, 2) belongs to both graphs. Vertex 1 is not adjacent to vertex 2 in Figure 5.2(a), since the edge (2, 1) does not belong to the graph.

The degree of a vertex in an undirected graph is the number of edges incident on it. For example, vertex 2 in Figure 5.2(b) has degree 2. In a directed graph, the out-degree of a vertex is the number of edges leaving it, and the in-degree of a vertex is the number of edges entering it. The degree of a vertex in a directed graph is its in-degree plus its out-degree. Vertex 2 in Figure 5.2(a) has in-degree 2, out-degree 3, and degree 5.

A path of length k from a vertex u to a vertex u' in a graph G = (V, E) is a sequence [image] v0, v1, v2, . . . , vk [image] of vertices such that u = v0, u' = vk, and (vi - 1, vi) [image] E for i = 1, 2, . . . , k. The length of the path is the number of edges in the path. The path contains the vertices v0, v1 , . . . , vk and the edges (v0, v1), (v1, v2), . . . , (vk - 1, vk). If there is a path p from u to u', we say that u' is reachable from u via p, which we sometimes write as [image] if G is directed. A path is simple if all vertices in the path are distinct. In Figure 5.2(a), the path [image] 1, 2, 5, 4 [image] is a simple path of length 3. The path [image] 2, 5, 4, 5 [image] is not simple.

A subpath of path p = [image] v0, v1, . . . , vk [image] is a contiguous subsequence of its vertices. That is, for any 0 [image] i [image] j [image] k, the subsequence of vertices [image] vi, vi + 1, . . . ,vj [image] is a subpath of p.

In a directed graph, a path [image] v0, v1, . . . , vk [image] forms a cycle if v0 = vk and the path contains at least one edge. The cycle is simple if, in addition, v1, v2, . . . , vk are distinct. A self-loop is a cycle of length 1. Two paths [image] v0, v1, v2, . . . , vk - 1, v0 [image] and [image] v'0, v'1, v'2, . . . , v'k - 1,v'0 [image] form the same cycle if there exists an integer j such that v'i = v(i + j) modk for i = 0, 1, . . . , k - 1. In Figure 5.2(a), the path [image] 1, 2, 4, 1 [image] forms the same cycle as the paths [image] 2, 4, 1, 2 [image] and [image] 4, 1, 2, 4 [image]. This cycle is simple, but the cycle [image] 1, 2, 4, 5, 4, 1 [image] is not. The cycle [image] 2, 2 [image] formed by the edge (2, 2) is a self-loop. A directed graph with no self-loops is simple. In an undirected graph, a path [image] v0, vl, . . . , vk [image] forms a cycle if v0 = vk and v1, v2, . . . , vk are distinct. For example, in Figure 5.2(b), the path [image] 1, 2, 5, 1 [image] is a cycle. A graph with no cycles is acyclic.

An undirected graph is connected if every pair of vertices is connected by a path. The connected components of a graph are the equivalence classes of vertices under the "is reachable from" relation. The graph in Figure 5.2(b) has three connected components: {1, 2, 5}, {3, 6}, and {4}. Every vertex in {1,2,5} is reachable from every other vertex in {1, 2, 5}. An undirected graph is connected if it has exactly one connected component, that is, if every vertex is reachable from every other vertex.

A directed graph is strongly connected if every two vertices are reachable from each other. The strongly connected components of a graph are the equivalence classes of vertices under the "are mutually reachable" relation. A directed graph is strongly connected if it has only one strongly connected component. The graph in Figure 5.2(a) has three strongly connected components: {1, 2, 4, 5}, {3}, and {6}. All pairs of vertices in {1, 2, 4, 5} are mutually reachable. The vertices {3, 6} do not form a strongly connected component, since vertex 6 cannot be reached from vertex 3.

Two graphs G = (V, E) and G' = (V', E') are isomorphic if there exists a bijection f : V [image] V' such that (u, v) [image] E if and only if (f(u), f(v)) [image] E'. In other words, we can relabel the vertices of G to be vertices of G', maintaining the corresponding edges in G and G'. Figure 5.3(a) shows a pair of isomorphic graphs G and G' with respective vertex sets V = {1, 2, 3, 4, 5, 6} and V' = {u, v, w, x, y, z}. The mapping from V to V' given by f(1) = u, f(2) = v, f(3) = w, f(4) = x, f(5) = y, f(6) = z is the required bijective function. The graphs in Figure 5.3(b) are not isomorphic. Although both graphs have 5 vertices and 7 edges, the top graph has a vertex of degree 4 and the bottom graph does not.

We say that a graph G' = (V', E') is a subgraph of G = (V,E) if V' [image] V and E' [image] E. Given a set V' [image] V, the subgraph of G induced by V' is the graph G' = (V', E'), where

E' = {(u, v)    E: u, v    V'} .

Figure 5.3 (a) A pair of isomorphic graphs. The vertices of the top graph are mapped to the vertices of the bottom graph by f(1) = u, f(2) = v, f(3) = w, f(4) = x, f(5) = y, f(6) = z. (b) Two graphs that are not isomorphic, since the top graph has a vertex of degree 4 and the bottom graph does not.

The subgraph induced by the vertex set {1, 2, 3, 6} in Figure 5.2(a) appears in Figure 5.2(c) and has the edge set {(1, 2), (2, 2), (6, 3)}.

Given an undirected graph G = (V, E), the directed version of G is the directed graph G' = (V, E'), where (u, v) [image] E' if and only if (u, v) [image] E. That is, each undirected edge (u, v) in G is replaced in the directed version by the two directed edges (u, v) and (v, u). Given a directed graph G =(V, E), the undirected version of G is the undirected graph G' = (V, E'), where (u, v) [image] E' if and only if u [image] v and (u, v) [image] E. That is, the undirected version contains the edges of G "with their directions removed" and with self-loops eliminated. (Since (u, v) and (v, u) are the same edge in an undirected graph, the undirected version of a directed graph contains it only once, even if the directed graph contains both edges (u, v) and (v, u). In a directed graph G = (V, E), a neighbor of a vertex u is any vertex that is adjacent to u in the undirected version of G. That is, v is a neighbor of u if either (u, v) [image] E or (v, u) [image] E. In an undirected graph, u and v are neighbors if they are adjacent.

Several kinds of graphs are given special names. A complete graph is an undirected graph in which every pair of vertices is adjacent. A bipartite graph is an undirected graph G = (V, E) in which V can be partitioned into two sets V1 and V2 such that (u, v) [image] E implies either u [image] V1 and v [image] V2 or u [image] V2 and v [image] V1. That is, all edges go between the two sets V1and V2. An acyclic, undirected graph is a forest, and a connected, acyclic, undirected graph is a (free) tree (see Section 5.5). We often take the first letters of "directed acyclic graph" and call such a graph a dag.

There are two variants of graphs that you may occasionally encounter. A multigraph is like an undirected graph, but it can have both multiple edges between vertices and self-loops. A hypergraph is like an undirected graph, but each hyperedge, rather than connecting two vertices, connects an arbitrary subset of vertices. Many algorithms written for ordinary directed and undirected graphs can be adapted to run on these graphlike structures.

Exercises

5.4-1

Attendees of a faculty party shake hands to greet each other, and each professor remembers how many times he or she shook hands. At the end of the party, the department head sums up the number of times that each professor shook hands. Show that the result is even by proving the handshaking lemma: if G = (V, E) is an undirected graph, then

5.4-2

Show that in an undirected graph, the length of a cycle must be at least 3.

5.4-3

Show that if a directed or undirected graph contains a path between two vertices u and v, then it contains a simple path between u and v. Show that if a directed graph contains a cycle, then it contains a simple cycle.

5.4-4

Show that any connected, undirected graph G = (V, E) satisfies |E| [image] |V| - 1.

5.4-5

Verify that in an undirected graph, the "is reachable from" relation is an equivalence relation on the vertices of the graph. Which of the three properties of an equivalence relation hold in general for the "is reachable from" relation on the vertices of a directed graph?

5.4-6

What is the undirected version of the directed graph in Figure 5.2(a)? What is the directed version of the undirected graph in Figure 5.2(b)?

5.4-7

Show that a hypergraph can be represented by a bipartite graph if we let incidence in the hypergraph correspond to adjacency in the bipartite graph. (Hint: Let one set of vertices in the bipartite graph correspond to vertices of the hypergraph, and let the other set of vertices of the bipartite graph correspond to hyperedges.)

5.5 Trees

As with graphs, there are many related, but slightly different, notions of trees. This section presents definitions and mathematical properties of several kinds of trees. Sections 11.4 and 23.1 describe how trees can be represented in a computer memory.

5.5.1 Free trees

As defined in Section 5.4, a free tree is a connected, acyclic, undirected graph. We often omit the adjective "free" when we say that a graph is a tree. If an undirected graph is acyclic but possibly disconnected, it is a forest. Many algorithms that work for trees also work for forests. Figure 5.4(a) shows a free tree, and Figure 5.4(b) shows a forest. The forest in Figure 5.4(b) is not a tree because it is not connected. The graph in Figure 5.4(c) is neither a tree nor a forest, because it contains a cycle.

The following theorem captures many important facts about free trees.

Theorem 5.2

Let G = (V, E) be an undirected graph. The following statements are equivalent.

1. G is a free tree.

2. Any two vertices in G are connected by a unique simple path.

3. G is connected, but if any edge is removed from E, the resulting graph is disconnected.

4. G is connected, and |E| = |V| - 1.

5. G is acyclic, and |E| = |V| - 1.

6. G is acyclic, but if any edge is added to E, the resulting graph contains a cycle.

Proof (1) [image] (2): Since a tree is connected, any two vertices in G are connected by at least one simple path. Let u and v be vertices that are connected by two distinct simple paths p1 and p2, as shown in Figure 5.5. Let w be the vertex at which the paths first diverge; that is, w is the first vertex on both p1 and p2 whose successor on p1 is x and whose successor on p2 is y, where x [image] y. Let z be the first vertex at which the paths reconverge; that is, z is the first vertex following w on p1 that is also on p2. Let p' be the subpath of p1 from w through x to z, and let p\" be the subpath of p2 from w through y to z. Paths p' and p\" share no vertices except their endpoints. Thus, the path obtained by concatenating p' and the reverse of p\"is a cycle. This is a contradiction. Thus, if G is a tree, there can be at most one path between two vertices.

Figure 5.4 (a) A free tree. (b) A forest. (c) A graph that contains a cycle and is therefore neither a tree nor a forest.

Figure 5.5 A step in the proof of Theorem 5.2: if (1) G is a free tree, then (2) any two vertices in G are connected by a unique simple path. Assume for the sake of contradiction that vertices u and v are connected by two distinct simple paths p1 and p2. These paths first diverge at vertex w, and they first reconverge at vertex z. The path p' concatenated with the reverse of the path p\" forms a cycle, which yields the contradiction.

(2) [image] (3): If any two vertices in G are connected by a unique simple path, then G is connected. Let (u, v) be any edge in E. This edge is a path from u to v, and so it must be the unique path from u to v. If we remove (u, v) from G, there is no path from u to v, and hence its removal disconnects G.

(3) [image] (4): By assumption, the graph G is connected, and by Exercise 5.4-4, we have |E| [image] |V| - 1. We shall prove |E| [image] |V| - 1 by induction. A connected graph with n = 1 or n = 2 vertices has n - 1 edges. Suppose that G has n [image] 3 vertices and that all graphs satisfying (3) with fewer than n vertices also satisfy |E| [image] |V| - 1. Removing an arbitrary edge from G separates the graph into k [image] 2 connected components (actually k = 2). Each component satisfies (3), or else G would not satisfy (3). Thus, by induction, the number of edges in all components combined is at most |V| - k [image] |V| - 2. Adding in the removed edge yields |E| [image] |V| - 1.

(4) [image] (5): Suppose that G is connected and that |E| = |V| - 1. We must show that G is acyclic. Suppose that G has a cycle containing k vertices v1, v2, . . . , vk. Let Gk = (Vk, Ek) be the subgraph of G consisting of the cycle. Note that |Vk| = |Ek| = k. If k < |V|, there must be a vertex vk+1 [image] V - Vk that is adjacent to some vertex vi [image] Vk, since G is connected. Define Gk+1 = (Vk+1, Ek+1) to be the subgraph of G with Vk+1 = Vk [image] {vk+1} and Ek+1 = Ek [image] {(v1, vk+1)}. Note that |Vk+1| = |Ek+1| = k+1. If k+1 < n, we can continue, defining Gk+2 in the same manner, and so forth, until we obtain Gn = (Vn, En), where n = |V|, Vn = V, and |En| = |Vn| = |V|. Since Gn is a subgraph of G, we have En [image] E, and hence |E| [image] |V|, which contradicts the assumption that |E| = |V| - 1. Thus, G is acyclic.

(5) [image] (6): Suppose that G is acyclic and that |E | = |V | - 1. Let k be the number of connected components of G. Each connected component is a free tree by definition, and since (1) implies (5), the sum of all edges in all connected components of G is |V | - k. Consequently, we must have k = 1, and G is in fact a tree. Since (1) implies (2), any two vertices in G are connected by a unique simple path. Thus, adding any edge to G creates a cycle.

(6) [image] (1): Suppose that G is acyclic but that if any edge is added to E, a cycle is created. We must show that G is connected. Let u and v be arbitrary vertices in G. If u and v are not already adjacent, adding the edge (u, v) creates a cycle in which all edges but (u, v) belong to G. Thus, there is a path from u to v, and since u and v were chosen arbitrarily, G is connected.

5.5.2 Rooted and ordered trees

A rooted tree is a free tree in which one of the vertices is distinguished from the others. The distinguished vertex is called the root of the tree. We often refer to a vertex of a rooted tree as a node2 of the tree. Figure 5.6(a) shows a rooted tree on a set of 12 nodes with root 7.

2The term "node" is often used in the graph theory literature as a synonym for "vertex." We shall reserve the term "node" to mean a vertex of a rooted tree.

Consider a node x in a rooted tree T with root r. Any node y on the unique path from r to x is called an ancestor of x. If y is an ancestor of x, then x is a descendant of y. (Every node is both an ancestor and a descendant of itself.) If y is an ancestor of x and x y, then y is a proper ancestor of x and x is a proper descendant of y. The subtree rooted at x is the tree induced by descendants of x, rooted at x. For example, the subtree rooted at node 8 in Figure 5.6(a) contains nodes 8, 6, 5, and 9.

If the last edge on the path from the root r of a tree T to a node x is (y, x), then y is the parent of x, and x is a child of y. The root is the only node in T with no parent. If two nodes have the same parent, they are siblings. A node with no children is an external node or leaf. A nonleaf node is an internal node.

Figure 5.6 Rooted and ordered trees. (a) A rooted tree with height 4. The tree is drawn in a standard way: the root (node 7) is at the top, its children (nodes with depth 1 ) are beneath it, their children (nodes with depth 2) are beneath them, and so forth. If the tree is ordered, the relative left-to-right order of the children of a node matters; otherwise it doesn't. (b) Another rooted tree. As a rooted tree, it is identical to the tree in (a), but as an ordered tree it is different, since the children of node 3 appear in a different order.

The number of children of a node x in a rooted tree T is called the degree of x.3 The length of the path from the root r to a node x is the depth of x in T. The largest depth of any node in T is the height of T.

3Notice that the degree of a node depends on whether T is considered to be a rooted tree or a free tree. The degree of a vertex in a free tree is, as in any undirected graph, the number of adjacent vertices. In a rooted tree, however, the degree is the number of children--the parent of a node does not count toward its degree.

An ordered tree is a rooted tree in which the children of each node are ordered. That is, if a node has k children, then there is a first child, a second child, . . . , and a kth child. The two trees in Figure 5.6 are different when considered to be ordered trees, but the same when considered to be just rooted trees.

5.5.3 Binary and positional trees

Binary trees are best described recursively. A binary tree T is a structure defined on a finite set of nodes that either

contains no nodes, or

is comprised of three disjoint sets of nodes: a root node, a binary tree called its left subtree, and a binary tree called its right subtree.

The binary tree that contains no nodes is called the empty tree or null tree, sometimes denoted NIL. If the left subtree is nonempty, its root is called the left child of the root of the entire tree. Likewise, the root of a nonnull right subtree is the right child of the root of the entire tree. If a subtree is the null tree NIL, we say that the child is absent or missing. Figure 5.7(a) shows a binary tree.

Figure 5.7 Binary trees. (a) A binary tree drawn in a standard way. The left child of a node is drawn beneath the node and to the left. The right child is drawn beneath and to the right. (b) A binary tree different from the one in (a). In (a), the left child of node 7 is 5 and the right child is absent. In (b), the left child of node 7 is absent and the right child is 5. As ordered trees, these trees are the same, but as binary trees, they are distinct. (c) The binary tree in (a) represented by the internal nodes of a full binary tree: an ordered tree in which each internal node has degree 2. The leaves in the tree are shown as squares.

A binary tree is not simply an ordered tree in which each node has degree at most 2. For example, in a binary tree, if a node has just one child, the position of the child--whether it is the left child or the right child--matters. In an ordered tree, there is no distinguishing a sole child as being either left or right. Figure 5.7(b) shows a binary tree that differs from the tree in Figure 5.7(a) because of the position of one node. Considered as ordered trees, however, the two trees are identical.

The positioning information in a binary tree can be represented by the internal nodes of an ordered tree, as shown in Figure 5.7(c). The idea is to replace each missing child in the binary tree with a node having no children. These leaf nodes are drawn as squares in the figure. The tree that results is a full binary tree: each node is either a leaf or has degree exactly 2. There are no degree-1 nodes. Consequently, the order of the children of a node preserves the position information.

The positioning information that distinguishes binary trees from ordered trees can be extended to trees with more than 2 children per node. In a positional tree, the children of a node are labeled with distinct positive integers. The ith child of a node is absent if no child is labeled with integer i. A k-ary tree is a positional tree in which for every node, all children with labels greater than k are missing. Thus, a binary tree is a k-ary tree with k = 2.

A complete k-ary tree is a k-ary tree in which all leaves have the same depth and all internal nodes have degree k. Figure 5.8 shows a complete binary tree of height 3. How many leaves does a complete k-ary tree of height h have? The root has k children at depth 1, each of which has k children at depth 2, etc. Thus, the number of leaves at depth h is kh. Consequently, the height of a complete k-ary tree with n leaves is logk n. The number of internal nodes of a complete k-ary tree of height h is

by equation (3.3). Thus, a complete binary tree has 2h - 1 internal nodes.

Figure 5.8 A complete binary tree of height 3 with 8 leaves and 7 internal nodes.

Exercises

5.5-1

Draw all the free trees composed of the 3 vertices A, B, and C. Draw all the rooted trees with nodes A, B, and C with A as the root. Draw all the ordered trees with nodes A, B, and C with A as the root. Draw all the binary trees with nodes A, B, and C with A as the root.

5.5-2

Show that for n 7, there exists a free tree on n nodes such that picking each of the n nodes as a root results in a different rooted tree.

5.5-3

Let G = (V, E) be a directed acyclic graph in which there is a vertex v0 V such that there exists a unique path from v0 to every vertex v V. Prove that the undirected version of G forms a tree.

5.5-4

Show by induction that the number of degree-2 nodes in any binary tree is 1 less than the number of leaves.

5.5-5

Show by induction that a binary tree with n nodes has height at least lg n .

5.5-6

The internal path length of a full binary tree is the sum, taken over all internal nodes of the tree, of the depth of each node. Likewise, the exernal path length is the sum, taken over all leaves of the tree, of the depth of each leaf. Consider a full binary tree with n internal nodes, internal path length i, and external path length e. Prove that e = i + 2n.

5.5-7

Let us associate a "weight" w(x) = 2-d with each leaf x of depth d in a binary tree T. Prove that x w(x) 1, where the sum is taken over all leaves x in T. (This is known as the Kraft inequality.)

5.5-8

Show that every binary tree with L leaves contains a subtree having between L/3 and 2L/3 leaves, inclusive.

Problems

5-1 Graph coloring

A k-coloring of an undirected graph G = (V, E) is a function c: V {0, l, . . . , k -1} such that c(u) c(v) for every edge (u, v) E. In other words, the numbers 0, 1, . . . , k - 1 represent the k colors, and adjacent vertices must have different colors.

a. Show that any tree is 2-colorable.

b. Show that the following are equivalent:

1. G is bipartite.

2. G is 2-colorable.

3. G has no cycles of odd length.

c. Let d be the maximum degree of any vertex in a graph G. Prove that G can be colored with d + 1 colors.

d. Show that if G has O(|V|) edges, then G can be colored with O colors.

5-2 Friendly graphs

Reword each of the following statements as a theorem about undirected graphs, and then prove it. Assume that friendship is symmetric but not reflexive.

a. In any group of n 2 people, there are two people with the same number of friends in the group.

b. Every group of six people contains either three mutual friends or three mutual strangers.

c. Any group of people can be partitioned into two subgroups such that at least half the friends of each person belong to the subgroup of which that person is not a member.

d. If everyone in a group is the friend of at least half the people in the group, then the group can be seated around a table in such a way that everyone is seated between two friends.

5-3 Bisecting trees

Many divide-and-conquer algorithms that operate on graphs require that the graph be bisected into two nearly equal-sized subgraphs by removing a small number of edges. This problem investigates bisections of trees.

a. Show that by removing a single edge, we can partition the vertices of any n-vertex binary tree into two sets A and B such that |A| 3n/4 and |B| 3n/4.

b. Show that the constant 3/4 in part (a) is optimal in the worst case by giving an example of a simple tree whose most evenly balanced partition upon removal of a single edge has |A| = 3n/4.

c. Show that by removing at most O(lg n) edges, we can partition the vertices of any n-vertex tree into two sets A and B such that |A| = n/2 and |B| = n/2 .

Chapter notes

G. Boole pioneered the development of symbolic logic, and he introduced many of the basic set notations in a book published in 1854. Modern set theory was created by G. Cantor during the period 1874-1895. Cantor focused primarily on sets of infinite cardinality. The term "function" is attributed to G. W. Leibnitz, who used it to refer to several kinds of mathematical formulas. His limited definition has been generalized many times. Graph theory originated in 1736, when L. Euler proved that it was impossible to cross each of the seven bridges in the city of Königsberg exactly once and return to the starting point.

A useful compendium of many definitions and results from graph theory is the book by Harary [94].

Go to Chapter 6 Back to Table of Contents


Intro to Algorithms: CHAPTER 6: COUNTING AND PROBABILITY

 Return to Table of Contents

CHAPTER 6: COUNTING AND PROBABILITY

This chapter reviews elementary combinatorics and probability theory. If you have a good background in these areas, you may want to skim the beginning of the chapter lightly and concentrate on the later sections. Most of the chapters do not require probability, but for some chapters it is essential.

Section 6.1 reviews elementary results in counting theory, including standard formulas for counting permutations and combinations. The axioms of probability and basic facts concerning probability distributions are presented in Section 6.2. Random variables are introduced in Section 6.3, along with the properties of expectation and variance. Section 6.4 investigates the geometric and binomial distributions that arise from studying Bernoulli trials. The study of the binomial distribution is continued in Section 6.5, an advanced discussion of the "tails" of the distribution. Finally, Section 6.6 illustrates probabilistic analysis via three examples: the birthday paradox, tossing balls randomly into bins, and winning streaks.

6.1 Counting

Counting theory tries to answer the question "How many?" without actually enumerating how many. For example, we might ask, "How many different n-bit numbers are there?" or "How many orderings of n distinct elements are there?" In this section, we review the elements of counting theory. Since some of the material assumes a basic understanding of sets, the reader is advised to start by reviewing the material in Section 5.1.

Rules of sum and product

A set of items that we wish to count can sometimes be expressed as a union of disjoint sets or as a Cartesian product of sets.

The rule of sum says that the number of ways to choose an element from one of two disjoint sets is the sum of the cardinalities of the sets.That is, if A and B are two finite sets with no members in common, then |A [image] B| = |A| + |B|, which follows from equation (5.3). For example, each position on a car's license plate is a letter or a digit. The number of possibilities for each position is therefore 26 + 10 = 36, since there are 26 choices if it is a letter and 10 choices if it is a digit.

The rule of product says that the number of ways to choose an ordered pair is the number of ways to choose the first element times the number of ways to choose the second element. That is, if A and B are two finite sets, then |A [image] B| = |A| [image] |B|, which is simply equation (5.4). For example, if an ice-cream parlor offers 28 flavors of ice cream and 4 toppings, the number of possible sundaes with one scoop of ice cream and one topping is 28 [image] 4 = 112.

Strings

A string over a finite set S is a sequence of elements of S. For example, there are 8 binary strings of length 3:

000, 001, 010, 011, 100, 101, 110, 111 .

We sometimes call a string of length k a k-string. A substring s' of a string s is an ordered sequence of consecutive elements of s. A k-substring of a string is a substring of length k. For example, 010 is a 3-substring of 01101001 (the 3-substring that begins in position 4), but 111 is not a substring of 01101001.

A k-string over a set S can be viewed as an element of the Cartesian product Sk of k-tuples; thus, there are |S|k strings of length k. For example, the number of binary k-strings is 2k. Intuitively, to construct a k-string over an n-set, we have n ways to pick the first element; for each of these choices, we have n ways to pick the second element; and so forth k times. This construction leads to the k-fold product n [image] n [image] [image] [image] n = nk as the number of k-strings.

Permutations

A permutation of a finite set S is an ordered sequence of all the elements of S, with each element appearing exactly once. For example, if S = {a, b, c}, there are 6 permutations of S:

abc, acb, bac, bca, cab, cba .

There are n! permutations of a set of n elements, since the first element of the sequence can be chosen in n ways, the second in n - 1 ways, the third in n - 2 ways, and so on.

A k-permutation of S is an ordered sequence of k elements of S, with no element appearing more than once in the sequence. (Thus, an ordinary permutation is just an n-permutation of an n-set.) The twelve 2-permutations of the set {a, b, c, d} are

ab, ac, ad, ba, bc, bd, ca, cb, cd, da, db, dc.

The number of k-permutations of an n-set is

(6.1)

since there are n ways of choosing the first element, n - 1 ways of choosing the second element, and so on until k elements are selected, the last being a selection from n - k + 1 elements.

Combinations

A k-combination of an n-set S is simply a k-subset of S. There are six 2-combinations of the 4-set {a, b, c, d}:

ab, ac, ad, bc, bd, cd .

(Here we use the shorthand of denoting the 2-set {a,b} by ab, and so on.) We can construct a k-combination of an n-set by choosing k distinct (different) elements from the n-set.

The number of k-combinations of an n-set can be expressed in terms of the number of k-permutations of an n-set. For every k-combination, there are exactly k! permutations of its elements, each of which is a distinct k-permutation of the n-set. Thus, the number of k-combinations of an n-set is the number of k-permutations divided by k!; from equation (6.1), this quantity is

(6.2)

For k = 0, this formula tells us that the number of ways to choose 0 elements from an n-set is 1 (not 0), since 0! = 1.

Binomial coefficients

We use the notation [image] (read "n choose k") to denote the number of k-combinations of an n-set. From equation (6.2), we have

(6.3)

This formula is symmetric in k and n - k:

(6.4)

These numbers are also known as binomial coefficients, due to their appearance in the binomial expansion:

(6.5)

A special case of the binomial expansion occurs when x = y = 1:

(6.6)

This formula corresponds to counting the 2n binary n-strings by the number of 1's they contain: there are [image] binary n-strings containing exactly k 1's, since there are [image] ways to choose k out of the n positions in which to place the 1's.

There are many identities involving binomial coefficients. The exercises at the end of this section give you the opportunity to prove a few.

Binomial bounds

We sometimes need to bound the size of a binomial coefficient. For 1 [image] k [image] n, we have the lower bound

(6.7)

Taking advantage of the inequality k! (k/e)k derived from Stirling's formula (2.12), we obtain the upper bounds

(6.8)

(6.9)

For all 0 [image] k [image] n, we can use induction (see Exercise 6.1-12) to prove the bound

(6.10)

where for convenience we assume that 00 = 1. For k = [image] n, where 0 [image] [image] [image] 1, this bound can be rewritten as

(6.11)

(6.12)

where

H(  ) = -  lg   - (1 -   ) lg (1 -   )

(6.13)

is the (binary) entropy function and where, for convenience, we assume that 0 lg 0 = 0, so that H (0) = H (1) = 0.

Exercises

6.1-1

How many k-substrings does an n-string have? (Consider identical k-substrings at different positions as different.) How many substrings does an n-string have in total?

6.1-2

An n-input, m-output boolean function is a function from {TRUE, FALSE}n to {TRUE, FALSE}m. How many n-input, 1-output boolean functions are there? How many n-input, m-output boolean functions are there?

6.1-3

In how many ways can n professors sit around a circular conference table? Consider two seatings to be the same if one can be rotated to form the other.

6.1-4

In how many ways can three distinct numbers be chosen from the set {1, 2, . . . , 100} so that their sum is even?

6.1-5

Prove the identity

(6.14)

for 0 < k [image] n.

6.1-6

Prove the identity

for 0 [image] k < n.

6.1-7

To choose k objects from n, you can make one of the objects distinguished and consider whether the distinguished object is chosen. Use this approach to prove that

6.1-8

Using the result of Exercise 6.1-7, make a table for n = 0, 1, . . . , 6 and 0 [image] k [image] n of the binomial coefficients [image] with [image] at the top, [image] and [image] on the next line, and so forth. Such a table of binomial coefficients is called Pascal's triangle.

6.1-9

Prove that

6.1-10

Show that for any n [image] 0 and 0 [image] k [image] n, the maximum value of [image] is achieved when k = [image] n/2 [image] or k = [image] n/2 [image] .

6.1-11

Argue that for any n [image] 0, j [image] 0, k [image] 0, and j + k [image] n,

**image 104_g.gif not available**

(6.15)

Provide both an algebraic proof and an argument based on a method for choosing j + k items out of n. Give an example in which equality does not hold.

6.1-12

Use induction on k [image] n/2 to prove inequality (6.10), and use equation (6.4) to extend it to all k [image] n.

6.1-13

Use Stirling's approximation to prove that

**image 104_h.gif not available**

(6.16)

6.1-14

By differentiating the entropy function H ( [image]), show that it achieves its maximum value at [image] = 1/2. What is H (1/2)?

6.2 Probability

Probability is an essential tool for the design and analysis of probabilistic and randomized algorithms. This section reviews basic probability theory.

We define probability in terms of a sample space S, which is a set whose elements are called elementary events. Each elementary event can be viewed as a possible outcome of an experiment. For the experiment of flipping two distinguishable coins, we can view the sample space as consisting of the set of all possible 2-strings over {H, T}:

S = {HH, HT, TH, TT} .

An event is a subset1 of the sample space S. For example, in the experiment of flipping two coins, the event of obtaining one head and one tail is {HT, TH}. The event S is called the certain event, and the event [image] is called the null event. We say that two events A and B are mutually exclusive if [image]. We sometimes treat an elementary event s [image] S as the event {s}. By definition, all elementary events are mutually exclusive.

1 For a general probability distribution, there may be some subsets of the sample space S that are not considered to be events. This situation usually arises when the sample space is uncountably infinite. The main requirement is that the set of events of a sample space be closed under the operations of taking the complement of an event, forming the union of a finite or countable number of events, and taking the intersection of a finite or countable number of events. Most of the probability distributions we shall see are over finite or countable sample spaces, and we shall generally consider all subsets of a sample space to be events. A notable exception is the continuous uniform probability distribution, which will be presented shortly.

Axioms of probability

A probability distribution Pr{} on a sample space S is a mapping from events of S to real numbers such that the following probability axioms are satisfied:

1. Pr {A} [image] 0 for any event A.

2. Pr {S} = 1.

3. Pr {A [image] B} = Pr {A} + Pr {B} for any two mutually exclusive events A and B. More generally, for any (finite or countably infinite) sequence of events A1, A2, . . . that are pairwise mutually exclusive,

We call Pr {A} the probability of the event A. We note here that axiom 2 is a normalization requirement: there is really nothing fundamental about choosing 1 as the probability of the certain event, except that it is natural and convenient.

Several results follow immediately from these axioms and basic set theory (see Section 5.1). The null event [image] has probability [image]. If A [image] B, then Pr{A} [image] Pr{B}. Using [image] to denote the event S - A (the complement of A), we have [image]. For any two events A and B,

PR{A   B} = Pr{A} + Pr{B} - Pr{A    B}

(6.17)

   Pr{A} + Pr{B} .

(6.18)

In our coin-flipping example, suppose that each of the four elementary events has probability 1/4. Then the probability of getting at least one head is

Pr{HH, HT, TH} = Pr{HH} + Pr{HT} + Pr{TH}

= 3/4.

Alternatively, since the probability of getting strictly less than one head is Pr{TT} = 1/4, the probability of getting at least one head is 1 - 1/4 = 3/4.

Discrete probability distributions

A probability distribution is discrete if it is defined over a finite or countably infinite sample space. Let S be the sample space. Then for any event A,

since elementary events, specifically those in A, are mutually exclusive. If S is finite and every elementary event s [image] S has probability

Pr{s} = 1/  S   ,

then we have the uniform probability distribution on S. In such a case the experiment is often described as "picking an element of S at random."

As an example, consider the process of flipping a fair coin, one for which the probability of obtaining a head is the same as the probability of obtaining a tail, that is, 1/2. If we flip the coin n times,we have the uniform probability distribution defined on the sample space S = {H, T}n, a set of size 2n. Each elementary event in S can be represented as a string of length n over {H, T}, and each occurs with probability 1/2n. The event

A = {exactly k heads and exactly n - k tails occur}

is a subset of S of size [image], since there are [image] strings of length n over {H, T} that contain exactly k H'S. The probability of event A is thus [image].

Continuous uniform probability distribution

The continuous uniform probability distribution is an example of a probability distribution in which all subsets of the sample space are not considered to be events. The continuous uniform probability distribution is defined over a closed interval [a, b] of the reals, where a < b. Intuitively, we want each point in the interval [a, b] to be "equally likely." There is an uncountable number of points, however, so if we give all points the same finite, positive probability, we cannot simultaneously satisfy axioms 2 and 3. For this reason, we would like to associate a probability only with some of the subsets of S in such a way that the axioms are satisfied for these events.

For any closed interval [c,d], where a [image] c [image] d [image] b, the continuous uniform probability distribution defines the probability of the event [c, d] to be

Note that for any point x = [x, x], the probability of x is 0. If we remove the endpoints of an interval [c, d], we obtain the open interval (c, d). Since [c, d] = [c, c] [image] (c, d) [image] [d, d], axiom 3 gives us Pr{[c, d]} = Pr{(c, d)}. Generally, the set of events for the continuous uniform probability distribution is any subset of [a, b] that can be obtained by a finite or countable union of open and closed intervals.

Conditional probability and independence

Sometimes we have some prior partial knowledge about the outcome of an experiment. For example, suppose that a friend has flipped two fair coins and has told you that at least one of the coins showed a head. What is the probability that both coins are heads? The information given eliminates the possibility of two tails. The three remaining elementary events are equally likely, so we infer that each occurs with probability 1/3. Since only one of these elementary events shows two heads, the answer to our question is 1/3.

Conditional probability formalizes the notion of having prior partial knowledge of the outcome of an experiment. The conditional probability of an event A given that another event B occurs is defined to be

(6.19)

whenever Pr{B} [image] 0. (We read "Pr{A [image] B}" as "the probability of A given B.") Intuitively, since we are given that event B occurs, the event that A also occurs is A [image] B. That is, A [image] B is the set of outcomes in which both A and B occur. Since the outcome is one of the elementary events in B, we normalize the probabilities of all the elementary events in B by dividing them by Pr{B}, so that they sum to 1. The conditional probability of A given B is, therefore, the ratio of the probability of event A [image] B to the probability of event B. In the example above, A is the event that both coins are heads, and B is the event that at least one coin is a head. Thus, Pr{A [image] B} = (1/4)/(3/4) = 1/3.

Two events are independent if

Pr{A    B} = Pr{A}Pr{B},

which is equivalent, if Pr{B} [image] 0, to the condition

Pr{A  B} = Pr{A}.

For example, suppose that two fair coins are flipped and that the outcomes are independent. Then the probability of two heads is (1/2)(1/2) = 1/4. Now suppose that one event is that the first coin comes up heads and the other event is that the coins come up differently. Each of these events occurs with probability 1/2, and the probability that both events occur is 1/4; thus, according to the definition of independence, the events are independent--even though one might think that both events depend on the first coin. Finally, suppose that the coins are welded together so that they both fall heads or both fall tails and that the two possibilities are equally likely. Then the probability that each coin comes up heads is 1/2, but the probability that they both come up heads is 1/2 [image] (1/2)(1/2). Consequently, the event that one comes up heads and the event that the other comes up heads are not independent.

A collection A1, A2, . . . , An of events is said to be pairwise independent if

Pr{Ai    Aj} = Pr{Ai} Pr{Aj}

for all 1 [image] i < j [image] n. We say that they are (mutually) independent if every k-subset Ai1,Ai2,...,Aik of the collection, where 2 [image] k [image] n and 1 [image] i1 < i2 < [image] [image] [image] < ik [image] n, satisfies

Pr{Ai1    Ai2              Aik} =  Pr{Ai1}Pr{Ai2}        Pr{Aik}.

For example, suppose we flip two fair coins. Let A1 be the event that the first coin is heads, let A2 be the event that the second coin is heads, and let A3 be the event that the two coins are different. We have

          Pr{A1}  =  1/2 ,

          Pr{A2}  =  1/2 ,

          Pr{A3}  =  1/2 ,

     Pr{A1    A2}  =  1/4 ,

     Pr{A1    A3}  =  1/4 ,

     Pr{A2    A3}  =  1/4 ,

Pr{A1    A2    A3}  =  0.

Since for 1 [image] i < j [image] 3, we have Pr{Ai [image] Aj} = Pr{Ai}Pr{Aj} = 1/4, the events A1, A2, and A3 are pairwise independent. The events are not mutually independent, however, because Pr{A1 [image] A2 [image] A3} = 0 and Pr{A1} Pr{A2} Pr{A3} = 1/8 [image] 0.

Bayes's theorem

From the definition of conditional probability (6.19), it follows that for two events A and B, each with nonzero probability,

Pr{A    B} = Pr{B}Pr{A|B}

(6.20)

= Pr{A}Pr{B|A}.

Solving for Pr{A |B}, we obtain

(6.21)

which is known as Bayes's theorem. The denominator Pr {B} is a normalizing constant that we can reexpress as follows. Since [image] and B [image] A and [image] are mutually exclusive events,

Substituting into equation (6.21), we obtain an equivalent form of Bayes's theorem:

Bayes's theorem can simplify the computing of conditional probabilities. For example, suppose that we have a fair coin and a biased coin that always comes up heads. We run an experiment consisting of three independent events: one of the two coins is chosen at random, the coin is flipped once, and then it is flipped again. Suppose that the chosen coin comes up heads both times. What is the probability that it is biased?

We solve this problem using Bayes's theorem. Let A be the event that the biased coin is chosen, and let B be the event that the coin comes up heads both times. We wish to determine Pr{A|B}. We have Pr{A} = 1/2, [image] hence,

Exercises

6.2-1

Prove Boole's inequality: For any finite or countably infinite sequence of events A1, A2, . . . ,

Pr{A1    A2            }    Pr{A1} + Pr{A2}+          .

(6.22)

6.2-2

Professor Rosencrantz flips one fair coin. Professor Guildenstern flips two fair coins. What is the probability that Professor Rosencrantz obtains more heads than Professor Guildenstern?

6.2-3

A deck of 10 cards, each bearing a distinct number from 1 to 10, is shuffled to mix the cards thoroughly. Three cards are removed one at a time from the deck. What is the probability that the three cards are selected in sorted (increasing) order?

6.2-4

You are given a biased coin, that when flipped, produces a head with (unknown) probability p, where 0 < p < 1. Show how a fair "coin flip" can be simulated by looking at multiple flips. (Hint: Flip the coin twice and then either output the result of the simulated fair flip or repeat the experiment.) Prove that your answer is correct.

6.2-5

Describe a procedure that takes as input two integers a and b such that 0 < a < b and, using fair coin flips, produces as output heads with probability a/b and tails with probability (b - a)/b. Give a bound on the expected number of coin flips, which should be polynomial in lg b.

6.2-6

Prove that

6.2-7

Prove that for any collection of events A1, A2, . . . , An,

Pr{A1    A2                An} = Pr{A1}    Pr{A2 | A1}    Pr{A3 | A1    A2}

Pr{An | A1    A2                An-1}.

6.2-8

Show how to construct a set of n events that are pairwise independent but such that any subset of k > 2 of them are not mutually independent.

6.2-9

Two events A and B are conditionally independent, given C, if

Pr{A    B | C} = Pr{A | C}    Pr{B | C} .

Give a simple but nontrivial example of two events that are not independent but are conditionally independent given a third event.

6.2-10

You are a contestant in a game show in which a prize is hidden behind one of three curtains. You will win the prize if you select the correct curtain. After you have picked one curtain but before the curtain is lifted, the emcee lifts one of the other curtains, revealing an empty stage, and asks if you would like to switch from your current selection to the remaining curtain. How will your chances change if you switch?

6.2-11

A prison warden has randomly picked one prisoner among three to go free. The other two will be executed. The guard knows which one will go free but is forbidden to give any prisoner information regarding his status. Let us call the prisoners X, Y, and Z. Prisoner X asks the guard privately which of Y or Z will be executed, arguing that since he already knows that at least one of them must die, the guard won't be revealing any information about his own status. The guard tells X that Y is to be executed. Prisoner X feels happier now, since he figures that either he or prisoner Z will go free, which means that his probability of going free is now 1/2. Is he right, or are his chances still 1/3? Explain.

6.3 Discrete random variables

A (discrete) random variable X is a function from a finite or countably infinite sample space S to the real numbers. It associates a real number with each possible outcome of an experiment, which allows us to work with the probability distribution induced on the resulting set of numbers. Random variables can also be defined for uncountably infinite sample spaces, but they raise technical issues that are unnecessary to address for our purposes. Henceforth, we shall assume that random variables are discrete.

For a random variable X and a real number x, we define the event X = x to be {s [image] S : X (s) = x}; thus,

The function

f(x) = Pr{X = x}

is the probability density function of the random variable X. From the probability axioms, Pr{X = x} [image] 0 and [image] x Pr{X = x} = 1.

As an example, consider the experiment of rolling a pair of ordinary, 6-sided dice. There are 36 possible elementary events in the sample space. We assume that the probability distribution is uniform, so that each elementary event s [image] S is equally likely: Pr {s} = 1/36. Define the random variable X to be the maximum of the two values showing on the dice. We have Pr {X = 3} = 5/36, since X assigns a value of 3 to 5 of the 36 possible elementary events, namely, (1, 3), (2, 3), (3, 3), (3, 2), and (3, 1).

It is common for several random variables to be defined on the same sample space. If X and Y are random variables, the function

f(x,y) = Pr{X = x and Y = y}

is the joint probability density function of X and Y. For a fixed value y,

and similarly, for a fixed value x,

Using the definition (6.19) of conditional probability, we have

We define two random variables X and Y to be independent if for all x and y, the events X = x and Y = y are independent or, equivalently, if for all x and y, we have Pr{X = x and Y = y} = Pr{X = x} Pr{Y = y}.

Given a set of random variables defined over the same sample space, one can define new random variables as sums, products, or other functions of the original variables.

Expected value of a random variable

The simplest and most useful summary of the distribution of a random variable is the "average" of the values it takes on. The expected value (or, synonymously, expectation or mean) of a discrete random variable X is

(6.23)

which is well defined if the sum is finite or converges absolutely. Sometimes the expectation of X is denoted by [image] x or, when the random variable is apparent from context, simply by [image] .

Consider a game in which you flip two fair coins. You earn $3 for each head but lose $2 for each tail. The expected value of the random variable X representing your earnings is

E[X] = 6    Pr{2 H'S} + 1    Pr{1 H, 1 T} - 4    Pr{2 T'S}

= 6(1/4)+ 1(1/2) - 4(1/4)

= 1 .

The expectation of the sum of two random variables is the sum of their expectations, that is,

E[X + Y] = E[X] + E[Y] ,

(6.24)

whenever E[X] and E[Y] are defined. This property extends to finite and absolutely convergent summations of expectations.

If X is any random variable, any function g(x) defines a new random variable g(X). If the expectation of g(X) is defined, then

Letting g(x) = ax, we have for any constant a,

E[aX] = aE[X] .

(6.25)

Consequently, expectations are linear: for any two random variables X and Y and any constant a,

E[aX + Y] = aE[X] + E[Y] .

(6.26)

When two random variables X and Y are independent and each has a defined expectation,

In general, when n random variables X1, X2, . . . , Xn are mutually independent,

E[X1X2          Xn] = E[X1]E[X2]          E[Xn] .

(6.27)

When a random variable X takes on values from the natural numbers N = {0,1, 2, . . .}, there is a nice formula for its expectation:

(6.28)

since each term Pr{X [image] i} is added in i times and subtracted out i - 1 times (except Pr{X [image] 0}, which is added in 0 times and not subtracted out at all).

Variance and standard deviation

The variance of a random variable X with mean E [X] is

Var[X] = E[(X  - E[X])2]

= E[X2  - 2XE[X] + E2[X]]

= E[X2] - 2E[XE[X]] + E2[X]

= E[X2] - 2E2[X] + E2[X]

= E[X2] - E2[X].

(6.29)

The justification for the equalities E [E2 [X]] = E2 [X] and E [XE [X]] = E2 [X] is that E [X] is not a random variable but simply a real number, which means that equation (6.25) applies (with a = E[X]). Equation (6.29) can be rewritten to obtain an expression for the expectation of the square of a random variable:

E[X2] = Var[X] + E2[X].

(6.30)

The variance of a random variable X and the variance of aX are related:

Var[aX] = a2Var[X].

When X and Y are independent random variables,

Var[X + Y] = Var[X] + Var[Y].

In general, if n random variables X1, X2, . . . , Xn are pairwise independent, then

(6.31)

The standard deviation of a random variable X is the positive square root of the variance of X. The standard deviation of a random variable X is sometimes denoted [image] x or simply [image] when the random variable X is understood from context. With this notation, the variance of X is denoted [image] 2.

Exercises

6.3-1

Two ordinary, 6-sided dice are rolled. What is the expectation of the sum of the two values showing? What is the expectation of the maximum of the two values showing?

6.3-2

An array A[1 . . n] contains n distinct numbers that are randomly ordered, with each permutation of the n numbers being equally likely. What is the expectation of the index of the maximum element in the array? What is the expectation of the index of the minimum element in the array?

6.3-3

A carnival game consists of three dice in a cage. A player can bet a dollar on any of the numbers 1 through 6. The cage is shaken, and the payoff is as follows. If the player's number doesn't appear on any of the dice, he loses his dollar. Otherwise, if his number appears on exactly k of the three dice, for k = 1, 2, 3, he keeps his dollar and wins k more dollars. What is his expected gain from playing the carnival game once?

6.3-4

Let X and Y be independent random variables. Prove that f(X) and g(Y) are independent for any choice of functions f and g.

6.3-5

Let X be a nonnegative random variable, and suppose that E (X) is well defined. Prove Markov's inequality:

Pr{X    t}    E[X]/t

(6.32)

for all t > 0.

6.3-6

Let S be a sample space, and let X and X' be random variables such that X(s) [image] X'(s) for all s [image] S. Prove that for any real constant t,

Pr{X    t}    Pr{X'    t} .

6.3-7

Which is larger: the expectation of the square of a random variable, or the square of its expectation?

6.3-8

Show that for any random variable X that takes on only the values 0 and 1, we have Var [X] = E [X] E [1 - X].

6.3-9

Prove that Var[aX] = a2Var[x] from the definition (6.29) of variance.

6.4 The geometric and binomial distributions

A coin flip is an instance of a Bernoulli trial, which is defined as an experiment with only two possible outcomes: success, which occurs with probability p, and failure, which occurs with probability q = 1 - p. When we speak of Bernoulli trials collectively, we mean that the trials are mutually independent and, unless we specifically say otherwise, that each has the same probability p for success. Two important distributions arise from Bernoulli trials: the geometric distribution and the binomial distribution.

The geometric distribution

Suppose we have a sequence of Bernoulli trials, each with a probability p of success and a probability q = 1 - p of failure. How many trials occur before we obtain a success? Let the random variable X be the number of trials needed to obtain a success. Then X has values in the range {1, 2, . . .}, and for k [image] 1,

Pr{X = k} = qk-1p ,

(6.33)

since we have k - 1 failures before the one success. A probability distribution satisfying equation (6.33) is said to be a geometric distribution. Figure 6.1 illustrates such a distribution.

Figure 6.1 A geometric distribution with probability p = 1/3 of success and a probability q = 1 - p of failure. The expectation of the distribution is 1/p = 3.

Assuming p < 1, the expectation of a geometric distribution can be calculated using identity (3.6):

(6.34.)

Thus, on average, it takes 1/p trials before we obtain a success, an intuitive result. The variance, which can be calculated similarly, is

Var[X] = q/p2 .

(6.35)

As an example, suppose we repeatedly roll two dice until we obtain either a seven or an eleven. Of the 36 possible outcomes, 6 yield a seven and 2 yield an eleven. Thus, the probability of success is p = 8/36 = 2/9, and we must roll 1/p = 9/2 = 4.5 times on average to obtain a seven or eleven.

Figure 6.2 The binomial distribution b(k; 15, 1/3) resulting from n = 15 Bernoulli trials, each with probability p = 1/3 of success. The expectation of the distribution is np = 5.

The binomial distribution

How many successes occur during n Bernoulli trials, where a success occurs with probability p and a failure with probability q = 1 - p? Define the random variable X to be the number of successes in n trials. Then X has values in the range {0, 1, . . . , n}, and for k = 0, . . . , n,

(6.36)

since there are [image] ways to pick which k of the n trials are successes, and the probability that each occurs is pkqn-k. A probability distribution satisfying equation (6.36) is said to be a binomial distribution. For convenience, we define the family of binomial distributions using the notation

(6.37)

Figure 6.2 illustrates a binomial distribution. The name "binomial" comes from the fact that (6.37) is the kth term of the expansion of (p + q)n. Consequently, since p + q = 1,

(6.38)

as is required by axiom 2 of the probability axioms.

We can compute the expectation of a random variable having a binomial distribution from equations (6.14) and (6.38). Let X be a random variable that follows the binomial distribution b(k; n, p), and let q = 1 - p. By the definition of expectation, we have

(6.39)

By using the linearity of expectation, we can obtain the same result with substantially less algebra. Let Xi be the random variable describing the number of successes in the ith trial. Then E[Xi] = p [image] 1+ q [image] 0 = p, and by linearity of expectation (6.26), the expected number of successes for n trials is

The same approach can be used to calculate the variance of the distribution. Using equation (6.29), we have [image]. Since Xi only takes on the values 0 and 1, we have [image], and hence

Var[Xi] = p - p2 = pq .

(6.40)

To compute the variance of X, we take advantage of the independence of the n trials; thus, by equation (6.31),

(6.41)

As can be seen from Figure 6.2, the binomial distribution b(k;n,p) increases as k runs from 0 to n until it reaches the mean np, and then it decreases. We can prove that the distribution always behaves in this manner by looking at the ratio of successive terms:

(6.42)

This ratio is greater than 1 precisely when (n + 1)p - k is positive. Consequently, b(k;n,p) > b(k - 1;n,p) for k < (n + 1)p (the distribution increases), and b(k;n,p) < b(k - 1;n,p) for k > (n + l)p (the distribution decreases). If k = (n + 1) p is an integer, then b(k;n,p) = b(k - 1;n,p), so the distribution has two maxima: at k = (n + l) p and at k - 1 = (n + 1)p - 1 = np - q. Otherwise, it attains a maximum at the unique integer k that lies in the range np - q < k < (n + 1)p.

The following lemma provides an upper bound on the binomial distribution.

Lemma 6.1

Let n [image] 0, let 0 < p < 1, let q = 1 - p, and let 0 [image] k [image] n. Then

Proof Using equation (6.10), we have

Exercises

6.4-1

Verify axiom 2 of the probability axioms for the geometric distribution.

6.4-2

How many times on average must we flip 6 fair coins before we obtain 3 heads and 3 tails?

6.4-3

Show that b(k; n, p) = b(n - k; n, q), where q = 1 - p.

6.4-4

Show that value of the maximum of the binomial distribution b(k; n, p) is approximately [image] where q = 1 - p.

6.4-5

Show that the probability of no successes in n Bernoulli trials, each with probability p = 1/n, is approximately 1/e. Show that the probability of exactly one success is also approximately 1/e.

6.4-6

Professor Rosencrantz flips a fair coin n times, and so does Professor Guildenstern. Show that the probability that they get the same number of heads is [image]. (Hint: For Professor Rosencrantz, call a head a success; for Professor Guildenstern, call a tail a success.) Use your argument to verify the identity

6.4-7

Show that for 0 [image] k [image] n,

b(k;n, 1/2)    2nH(k/n)-n,

where H(x) is the entropy function (6.13).

6.4-8

Consider n Bernoulli trials, where for i = 1,2, . . . ,n, the ith trial has probability pi of success, and let X be the random variable denoting the total number of successes. Let p [image] pi for all i = 1, 2, . . . , n. Prove that for 1 [image] k [image] n,

6.4-9

Let X be the random variable for the total number of successes in a set A of n Bernoulli trials, where the ith trial has a probability pi of success, and let X' be the random variable for the total number of successes in a second set A' of n Bernoulli trials, where the ith trial has a probability p'i [image] pi of success. Prove that for 0 [image] k [image] n,

Pr{X'    k}    Pr{X    k} .

(Hint: Show how to obtain the Bernoulli trials in A' by an experiment involving the trials of A, and use the result of Exercise 6.3-6.)

* 6.5 The tails of the binomial distribution

The probability of having at least, or at most, k successes in n Bernoulli trials, each with probability p of success, is often of more interest than the probability of having exactly k successes. In this section, we investigate the tails of the binomial distribution: the two regions of the distribution b(k; n, p) that are far from the mean np. We shall prove several important bounds on (the sum of all terms in) a tail.

We first provide a bound on the right tail of the distribution b(k; n, p). Bounds on the left tail can be determined by inverting the roles of successes and failures.

Theorem 6.2

Consider a sequence of n Bernoulli trials, where success occurs with probability p. Let X be the random variable denoting the total number of successes. Then for 0 [image] k [image] n, the probability of at least k successes is

Proof We make use of the inequality (6.15)

We have

since [image] by equation (6.38).

The following corollary restates the theorem for the left tail of the binomial distribution. In general, we shall leave it to the reader to adapt the bounds from one tail to the other.

Corollary 6.3

Consider a sequence of n Bernoulli trials, where success occurs with probability p. If X is the random variable denoting the total number of successes, then for 0 [image] k [image] n, the probability of at most k successes is

Our next bound focuses on the left tail of the binomial distribution. Far from the mean, the number of successes in the left tail diminishes exponentially, as the following theorem shows.

Theorem 6.4

Consider a sequence of n Bernoulli trials, where success occurs with probability p and failure with probability q = 1 - p. Let X be the random variable denoting the total number of successes. Then for 0 < k < np, the probability of fewer than k successes is

Proof We bound the series [image] by a geometric series using the technique from Section 3.2, page 47. For i = 1, 2, . . . , k, we have from equation (6.42),

If we let

it follows that

b(i - 1;n,p) < x b(i;n,p)

for 0 < i [image] k. Iterating, we obtain

b(i;n,p) < xk-i b(k;n,p)

for 0 [image] i < k, and hence

When k [image] np/2, we have kq/(np - k) [image] 1, which means that b(k; n, p) bounds the sum of all terms smaller than k. As an example, suppose we flip n fair coins. Using p = 1/2 and k = n/4, Theorem 6.4 tells us that the probability of obtaining fewer than n/4 heads is less than the probability of obtaining exactly n/4 heads. Furthermore, for any r [image] 4, the probability of obtaining fewer than n/r heads is less than that of obtaining exactly n/r heads. Theorem 6.4 can also be quite useful in conjunction with upper bounds on the binomial distribution, such as Lemma 6.1.

A bound on the right tail can be determined similarly.

Corollary 6.5

Consider a sequence of n Bernoulli trials, where success occurs with probability p. Let X be the random variable denoting the total number of successes. Then for np < k < n, the probability of more than k successes is

The next theorem considers n Bernoulli trials, each with a probability pi of success, for i = 1, 2, . . . , n. As the subsequent corollary shows, we can use the theorem to provide a bound on the right tail of the binomial distribution by setting pi = p for each trial.

Theorem 6.6

Consider a sequence of n Bernoulli trials, where in the ith trial, for i = 1, 2 , . . . , n, success occurs with probability pi and failure occurs with probability qi = 1 - pi. Let X be the random variable describing the total number of successes, and let [image] = E[X]. Then for r > [image],

Proof Since for any [image] > 0, the function e [image] x is strictly increasing in x,

Pr{X -       r} = Pr{e  (X-  )    e  r} ,

where [image] will be determined later. Using Markov's inequality (6.32), we obtain

Pr{X -       r}    E[e  (X-  )]e-  r .

(6.43)

The bulk of the proof consists of bounding E[e [image] (X- [image])] and substituting a suitable value for [image] in inequality (6.43). First, we evaluate E[e [image] (X- [image])]. For i = 1, 2, . . . , n, let Xi be the random variable that is 1 if the ith Bernoulli trial is a success and 0 if it is a failure. Thus,

and

Substituting for X - [image], we obtain

which follows from (6.27), since the mutual independence of the random variables Xi implies the mutual independence of the random variables e [image] (Xi-pi) (see Exercise 6.3-4). By the definition of expectation,

E [e   (Xi-pi)] = e  (1-pi) pi + e  (0-pi) qi

= pie  qi + qie-  pi

   pie   + 1

   ex(pie  ) ,

(6.44)

where exp(x) denotes the exponential function: exp(x) = ex. (Inequality (6.44) follows from the inequalities [image] > 0, q [image] 1, e [image] q [image] e [image] , and e- [image] p [image] 1, and the last line follows from inequality (2.7)). Consequently,

since [image]. Hence, from inequality (6.43), it follows that

Pr{X -       r}    exp(  e   -   r) .

(6.45)

Choosing [image] = 1n(r/ [image] ) (see Exercise 6.5-6), we obtain

When applied to Bernoulli trials in which each trial has the same probability of success, Theorem 6.6 yields the following corollary bounding the right tail of a binomial distribution.

Corollary 6.7

Consider a sequence of n Bernoulli trials, where in each trial success occurs with probability p and failure occurs with probability q = 1 - p. Then for

Proof For a binomial distribution, equation (6.39) implies that [image] = E [X] = np.

Exercises

6.5-1

Which is less likely: obtaining no heads when you flip a fair coin n times, or obtaining fewer than n heads when you flip the coin 4n times?

6.5-2

Show that

for all a > 0 and all k such that 0 < k < n.

6.5-3

Prove that if 0 < k < np, where 0 < p < 1 and q = 1 - p, then

6.5-4

Show that the conditions of Theorem 6.6 imply that

Similarly, show that the conditions of Corollary 6.7 imply that

6.5-5

Consider a sequence of n Bernoulli trials, where in the ith trial, for i = 1, 2, . . . , n, success occurs with probability pi and failure occurs with probability qi = 1 - pi. Let X be the random variable describing the total number of successes, and let [image] = E [X]. Show that for r [image] 0,

Pr{X -   ]    r}    e-r2/2n.

(Hint: Prove that pie [image] qi + qie- [image] pi [image] e- [image] 2/2. Then follow the outline of the proof of Theorem 6.6, using this inequality in place of inequality (6.44).)

6.5-6

Show that choosing [image] = 1n(r/ [image]) minimizes the right-hand side of inequality (6.45).

6.6 Probabilistic analysis

This section uses three examples to illustrate probabilistic analysis. The first determines the probability that in a room of k people, some pair shares the same birthday. The second example examines the random tossing of balls into bins. The third investigates "streaks" of consecutive heads in coin flipping.

6.6.1 The birthday paradox

A good example to illustrate probabilistic reasoning is the classical birthday paradox. How many people must there be in a room before there is a good chance two of them were born on the same day of the year? The answer is surprisingly few. The paradox is that it is in fact far fewer than the number of days in the year, as we shall see.

To answer the question, we index the people in the room with the integers 1, 2, . . . , k, where k is the number of people in the room. We ignore the issue of leap years and assume that all years have n = 365 days. For i = 1, 2, . . . , k, let bi be the day of the year on which i's birthday falls, where 1 [image] bi [image] n. We also assume that birthdays are uniformly distributed across the n days of the year, so that Pr{bi = r} = 1/n for i = 1, 2, . . . , k and r = 1, 2, . . . , n.

The probability that two people i and j have matching birthdays depends on whether the random selection of birthdays is independent. If birthdays are independent, then the probability that i's birthday and j's birthday both fall on day r is

Pr{bi = r and bj = r} = Pr{bi = r}Pr {bj = r}

= 1/n2 .

Thus, the probability that they both fall on the same day is

More intuitively, once bi is chosen, the probability that bj is chosen the same is 1/n. Thus, the probability that i and j have the same birthday is the same as the probability that the birthday of one of them falls on a given day. Notice, however, that this coincidence depends on the assumption that the birthdays are independent.

We can analyze the probability of at least 2 out of k people having matching birthdays by looking at the complementary event. The probability that at least two of the birthdays match is 1 minus the probability that all the birthdays are different. The event that k people have distinct birthdays is

where Ai is the event that person (i + 1)'s birthday is different from person j's for all j [image] i, that is,

Ai = {bi +1    bj: j = 1, 2 . . . , i}.

Since we can write Bk = Ak - 1 [image] Bk - 1, we obtain from equation (6.20) the recurrence

Pr{Bk} = Pr{Bk - 1}Pr{Ak - 1    Bk - 1} ,

(6.46)

where we take Pr{B1} = 1 as an initial condition. In other words, the probability that b1, b2,...,bk are distinct birthdays is the probability that b1, b2, . . . , bk - 1, are distinct birthdays times the probability that bk [image] bi for i = 1, 2, . . . , k - 1, given that b1,b2,...,bk - 1 are distinct.

If b1, b2, . . . , bk - 1 are distinct, the conditional probability that bk [image] bi for i = 1, 2, . . . , k - 1 is (n - k + 1)/n, since out of the n days, there are n - (k - 1) that are not taken. By iterating the recurrence (6.46), we obtain

The inequality (2.7), 1 + x [image] ex, gives us

when -k(k - l)/2n [image] ln(1/2). The probability that all k birthdays are distinct is at most 1/2 when k(k - 1) [image] 2n ln 2 or, solving the quadratic equation, when [image]. For n = 365, we must have k [image] 23. Thus, if at least 23 people are in a room, the probability is at least 1/2 that at least two people have the same birthday. On Mars, a year is 669 Martian days long; it therefore takes 31 Martians to get the same effect.

Another method of analysis

We can use the linearity of expectation (equation (6.26)) to provide a simpler but approximate analysis of the birthday paradox. For each pair (i, j) of the k people in the room, let us define the random variable Xij, for 1 [image] i [image] j [image] k, by

The probability that two people have matching birthdays is 1/n, and thus by the definition of expectation (6.23),

E[Xij] = 1    (1/n) + 0    (1 - 1/n)

= 1/n.

The expected number of pairs of individuals having the same birthday is, by equation (6.24), just the sum of the individual expectations of the pairs, which is

When k(k -1) [image] 2n, therefore, the expected number of pairs of birthdays is at least 1. Thus, if we have at least [image] individuals in a room, we can expect at least two to have the same birthday. For n = 365, if k = 28, the expected number of pairs with the same birthday is (28 [image] 27)/(2 [image] 365) [image] 1.0356. Thus, with at least 28 people, we expect to find at least one matching pair of birthdays. On Mars, where a year is 669 Martian days long, we need at least 38 Martians.

The first analysis determined the number of people required for the probability to exceed 1/2 that a matching pair of birthdays exists, and the second analysis determined the number such that the expected number of matching birthdays is 1. Although the numbers of people differ for the two situations, they are the same asymptotically: .

6.6.2 Balls and bins

Consider the process of randomly tossing identical balls into b bins, numbered 1, 2, . . . , b. The tosses are independent, and on each toss the ball is equally likely to end up in any bin. The probability that a tossed ball lands in any given bin is 1/b. Thus, the ball-tossing process is a sequence of Bernoulli trials with a probability 1/b of success, where success means that the ball falls in the given bin. A variety of interesting questions can be asked about the ball-tossing process.

How many balls fall in a given bin? The number of balls that fall in a given bin follows the binomial distribution b(k; n, 1/b). If n balls are tossed, the expected number of balls that fall in the given bin is n/b.

How many balls must one toss, on the average, until a given bin contains a ball? The number of tosses until the given bin receives a ball follows the geometric distribution with probability 1/b, and thus the expected number of tosses until success is 1/(1/b) = b.

How many balls must one toss until every bin contains at least one ball? Let us call a toss in which a ball falls into an empty bin a "hit." We want to know the average number n of tosses required to get b hits.

The hits can be used to partition the n tosses into stages. The ith stage consists of the tosses after the (i - 1)st hit until the ith hit. The first stage consists of the first toss, since we are guaranteed to have a hit when all bins are empty. For each toss during the ith stage, there are i -1 bins that contain balls and b - i + 1 empty bins. Thus, for all tosses in the ith stage, the probability of obtaining a hit is (b - i + 1)/b.

Let ni denote the number of tosses in the ith stage. Thus, the number of tosses required to get b hits is n = [image]. Each random variable ni has a geometric distribution with probability of success (b - i + 1)/b, and therefore

By linearity of expectation,

The last line follows from the bound (3.5) on the harmonic series. It therefore takes approximately b ln b tosses before we can expect that every bin has a ball.

6.6.3 Streaks

Suppose you flip a fair coin n times. What is the longest streak of consecutive heads that you expect to see? The answer is [image] (lg n), as the following analysis shows.

We first prove that the expected length of the longest streak of heads is O(lg n). Let Aik be the event that a streak of heads of length at least k begins with the ith coin flip or, more precisely, the event that the k consecutive coin flips i, i+ 1, . . ., i+k - 1 yield only heads, where 1 [image] k [image] n and 1 [image] i [image] n - k + 1. For any given event Aik, the probability that all k flips are heads has a geometric distribution with p = q = 1/2:

Pr {Aik} = 1/2k .

(6.47)

For k = 2   lg n  ,

Pr{Ai,2  1g n  } = 1/2 2  1g n

   1/2 2 1g n

= 1/n2,

and thus the probability that a streak of heads of length at least [image] 2 1g n [image] begins in position i is quite small, especially considering that there are at most n positions (actually n - 2 [image] 1g n [image] + 1 ) where the streak can begin. The probability that a streak of heads of length at least [image] 2 1g n [image] begins anywhere is therefore

since by Boole's inequality (6.22), the probability of a union of events is at most the sum of the probabilities of the individual events. (Note that Boole's inequality holds even for events such as these that are not independent.)

The probability is therefore at most 1/n that any streak has length at least 2 [image] 1g n [image] ; hence, the probability is at least 1 - 1/n that the longest streak has length less than 2 [image] 1g n [image] . Since every streak has length at most n, the expected length of the longest streak is bounded above by

(2   lg n   )(1 - 1/n) + n(1/n) = O(lg n).

The chances that a streak of heads exceeds r [image] 1g n flips diminish quickly with r. For r 1, the probability that a streak of r 1g n heads starts in position i is

Pr{Ai,r  1g n  } = 1/2r  lg n

   1/nr.

Thus, the probability is at most n/nr = 1/nr-1 that the longest streak is at least r 1g n , or equivalently, the probability is at least 1- 1/nr-1 that the longest streak has length less than r 1g n .

As an example, for n = 1000 coin flips, the probability of having a streak of at least 2 1g n = 20 heads is at most 1/n = 1/1000. The chances of having a streak longer than 3 1g n = 30 heads is at most 1/n2 = 1/1,000,000.

We now prove a complementary lower bound: the expected length of the longest streak of heads in n coin flips is (1g n). To prove this bound, we look for streaks of length 1g n /2. From equation (6.47), we have

The probability that a streak of heads of length at least 1g n /2 does not begin in position i is therefore at most . We can partition the n coin flips into at least 2n/ 1g n groups of 1g n /2 consecutive coin flips. Since these groups are formed from mutually exclusive, independent coin flips, the probability that every one of these groups fails to be a streak of length 1g n /2 is

For this argument, we used inequality (2.7), 1+ x ex, and the fact, which you might want to verify, that (2n/1g n - 1) / 1g n for n 2. (For n = 1, the probability that every group fails to be a streak is trivially at most 1/n = 1.)

Thus, the probability that the longest streak exceeds 1g n /2 is at least 1- 1/n. Since the longest streak has length at least 0, the expected length of the longest streak is at least

(  1g n   /2)(1 - 1/n) + 0  (1/n) =   (1g n).

Exercises

6.6-1

Suppose that balls are tossed into b bins. Each toss is independent, and each ball is equally likely to end up in any bin. What is the expected number of ball tosses before at least one of the bins contains two balls?

6.6-2

For the analysis of the birthday paradox, is it important that the birthdays be mutually independent, or is pairwise independence sufficient? Justify your answer.

6.6-3

How many people should be invited to a party in order to make it likely that there are three people with the same birthday?

6.6-4

What is the probability that a k-string over a set of size n is actually a k-permutation? How does this question relate to the birthday paradox?

6.6-5

Suppose that n balls are tossed into n bins, where each toss is independent and the ball is equally likely to end up in any bin. What is the expected number of empty bins? What is the expected number of bins with exactly one ball?

6.6-6

Sharpen the lower bound on streak length by showing that in n flips of a fair coin, the probability is less than 1/n that no streak longer than 1g n - 2 1g 1g n consecutive heads occurs.

Problems

6-1 Balls and bins

In this problem, we investigate the effect of various assumptions on the number of ways of placing n balls into b distinct bins.

a. Suppose that the n balls are distinct and that their order within a bin does not matter. Argue that the number of ways of placing the balls in the bins is bn.

b. Suppose that the balls are distinct and that the balls in each bin are ordered. Prove that the number of ways of placing the balls in the bins is (b + n - 1 )!/(b - 1)!. (Hint: Consider the number of ways of arranging n distinct balls and b - 1 indistinguishable sticks in a row.)

c. Suppose that the balls are identical, and hence their order within a bin does not matter. Show that the number of ways of placing the balls in the bins is . (Hint: Of the arrangements in part (b), how many are repeated if the balls are made identical?)

d. Suppose that the balls are identical and that no bin may contain more than one ball. Show that the number of ways of placing the balls is .

e. Suppose that the balls are identical and that no bin may be left empty. Show that the number of ways of placing the balls is .

6-2 Analysis of max program

The following program determines the maximum value in an unordered array A[1 . . n].

1 max   -

2 for i    1 to n

3       do    Compare A[i] to max.

4          if A[i] > max

5             then max    A[i]

We want to determine the average number of times the assignment in line 5 is executed. Assume that all numbers in A are randomly drawn from the interval [0, 1].

a. If a number x is randomly chosen from a set of n distinct numbers, what is the probability that x is the largest number in the set?

b. When line 5 of the program is executed, what is the relationship between A[i] and A[j] for 1 j i?

c. For each i in the range 1 i n, what is the probability that line 5 is executed?

d. Let s1, s2, . . ., sn be n random variables, where si represents the number of times (0 or 1) that line 5 is executed during the ith iteration of the for loop. What is E[si]?

e. Let s = s1 + s2 +... + sn be the total number of times that line 5 is executed during some run of the program. Show that E[s] = (lg n).

6-3 Hiring problem

Professor Dixon needs to hire a new research assistant. She has arranged interviews with n applicants and would like to base her decision solely on their qualifications. Unfortunately, university regulations require that after each interview she immediately reject or offer the position to the applicant.

Professor Dixon decides to adopt the strategy of selecting a positive integer k < n, interviewing and then rejecting the first k applicants, and hiring the first applicant thereafter who is better qualified than all preceding applicants. If the best-qualified applicant is among the first k interviewed, then she will hire the nth applicant. Show that Professor Dixon maximizes her chances of hiring the best-qualified applicant by choosing k approximately equal to n/e and that her chances of hiring the best-qualified applicant are then approximately 1/e.

6-4 Probabilistic counting

With a t-bit counter, we can ordinarily only count up to 2t - 1. With R. Morris's probabilistic counting, we can count up to a much larger value at the expense of some loss of precision.

We let a counter value of i represent a count of ni for i = 0, 1, ..., 2t - 1, where the ni form an increasing sequence of nonnegative values. We assume that the initial value of the counter is 0, representing a count of n0 = 0. The INCREMENT operation works on a counter containing the value i in a probabilistic manner. If i = 2t - 1, then an overflow error is reported. Otherwise, the counter is increased by 1 with probability 1/(ni+l - ni), and it remains unchanged with probability 1 - l/(ni+1 - ni).

If we select ni = i for all i 0, then the counter is an ordinary one. More interesting situations arise if we select, say, ni = 2i-1 for i > 0 or ni = Fi (the ith Fibonacci number--see Section 2.2).

For this problem, assume that n2t-1 is large enough that the probability of an overflow error is negligible.

a. Show that the expected value represented by the counter after n INCREMENT operations have been performed is exactly n.

b. The analysis of the variance of the count represented by the counter depends on the sequence of the ni. Let us consider a simple case: ni = 100i for all i 0. Estimate the variance in the value represented by the register after n INCREMENT operations have been performed.

Chapter notes

The first general methods for solving probability problems were discussed in a famous correspondence between B. Pascal and P. de Fermat, which began in 1654, and in a book by C. Huygens in 1657. Rigorous probability theory began with the work of J. Bernoulli in 1713 and A. De Moivre in 1730. Further developments of the theory were provided by P. S. de Laplace, S.-D. Poisson, and C. F. Gauss.

Sums of random variables were originally studied by P. L. Chebyshev and A. A. Markov. Probability theory was axiomatized by A. N. Kolmogorov in 1933. Bounds on the tails of distributions were provided by Chernoff [40] and Hoeffding [99]. Seminal work in random combinatorial structures was done by P. Erdös.

Knuth [121] and Liu [140] are good references for elementary combinatorics and counting. Standard textbooks such as Billingsley [28], Chung [41], Drake [57], Feller [66], and Rozanov [171] offer comprehensive introductions to probability. Bollobás [30], Hofri [100], and Spencer [179] contain a wealth of advanced probabilistic techniques.

Go to Part II Back to Table of Contents


Intro to Algorithms: CHAPTER 7: HEAPSORT

 Return to Table of Contents

CHAPTER 7: HEAPSORT

In this chapter, we introduce another sorting algorithm. Like merge sort, but unlike insertion sort, heapsort's running time is O(n lg n). Like insertion sort, but unlike merge sort, heapsort sorts in place: only a constant number of array elements are stored outside the input array at any time. Thus, heapsort combines the better attributes of the two sorting algorithms we have already discussed.

Heapsort also introduces another algorithm design technique: the use of a data structure, in this case one we call a "heap," to manage information during the execution of the algorithm. Not only is the heap data structure useful for heapsort, it also makes an efficient priority queue. The heap data structure will reappear in algorithms in later chapters.

We note that the term "heap" was originally coined in the context of heapsort, but it has since come to refer to "garbage-collected storage," such as the programming language Lisp provides. Our heap data structure is not garbage-collected storage, and whenever we refer to heaps in this book, we shall mean the structure defined in this chapter.

7.1 Heaps

The (binary) heap data structure is an array object that can be viewed as a complete binary tree (see Section 5.5.3), as shown in Figure 7.1. Each node of the tree corresponds to an element of the array that stores the value in the node. The tree is completely filled on all levels except possibly the lowest, which is filled from the left up to a point. An array A that represents a heap is an object with two attributes: length[A], which is the number of elements in the array, and heap-size[A], the number of elements in the heap stored within array A. That is, although A[1 . . length[A]] may contain valid numbers, no element past A[heap-size[A]], where heap-size[A] [image] [length[A], is an element of the heap. The root of the tree is A[1], and given the index i of a node, the indices of its parent PARENT(i), left child LEFT(i), and right child RIGHT(i) can be computed simply:

Figure 7.1 A heap viewed as (a) a binary tree and (b) an array. The number within the circle at each node in the tree is the value stored at that node. The number next to a node is the corresponding index in the array.

PARENT(i)

return   i/2

LEFT(i)

return 2i

RIGHT(i)

return 2i + 1

On most computers, the LEFT procedure can compute 2i in one instruction by simply shifting the binary representation of i left one bit position. Similarly, the RIGHT procedure can quickly compute 2i + 1 by shifting the binary representation of i left one bit position and shifting in a 1 as the low-order bit. The PARENT procedure can compute [image] i/2 [image] by shifting i right one bit position. In a good implementation of heapsort, these three procedures are often implemented as "macros" or "in-line" procedures.

Heaps also satisfy the heap property: for every node i other than the root,

A[PARENT(i)]    A[i],

(7.1)

that is, the value of a node is at most the value of its parent. Thus, the largest element in a heap is stored at the root, and the subtrees rooted at a node contain smaller values than does the node itself.

We define the height of a node in a tree to be the number of edges on the longest simple downward path from the node to a leaf, and we define the height of the tree to be the height of its root. Since a heap of n elements is based on a complete binary tree, its height is [image] (lg n) (see Exercise 7.1-2). We shall see that the basic operations on heaps run in time at most proportional to the height of the tree and thus take O(lg n) time. The remainder of this chapter presents five basic procedures and shows how they are used in a sorting algorithm and a priority-queue data structure.

The HEAPIFY procedure, which runs in O(lg n) time, is the key to maintaining the heap property (7.1).

The BUILD-HEAP procedure, which runs in linear time, produces a heap from an unordered input array.

The HEAPSORT procedure, which runs in O(n lg n) time, sorts an array in place.

The EXTRACT-MAX and INSERT procedures, which run in O(1g n) time, allow the heap data structure to be used as a priority queue.

Exercises

7.1-1

What are the minimum and maximum numbers of elements in a heap of height h?

7.1-2

Show that an n-element heap has height [image] lg n [image].

7.1-3

Show that the largest element in a subtree of a heap is at the root of the subtree.

7.1-4

Where in a heap might the smallest element reside?

7.1-5

Is an array that is in reverse sorted order a heap?

7.1-6

Is the sequence [image] 23, 17, 14, 6, 13, 10, 1, 5, 7, 12 [image] a heap?

7.2 Maintaining the heap property

HEAPIFY is an important subroutine for manipulating heaps. Its inputs are an array A and an index i into the array. When HEAPIFY is called, it is assumed that the binary trees rooted at LEFT(i) and RIGHT(i) are heaps, but that A[i] may be smaller than its children, thus violating the heap property (7.1). The function of HEAPIFY is to let the value at A[i] "float down" in the heap so that the subtree rooted at index i becomes a heap.

Figure 7.2 The action of HEAPIFY(A, 2), where heap-size[A] = 10. (a) The initial configuration of the heap, with A[2] at node i = 2 violating the heap property since it is not larger than both children. The heap property is restored for node 2 in (b) by exchanging A[2] with A[4], which destroys the heap property for node 4. The recursive call HEAPIFY(A, 4) now sets i = 4. After swapping A[4] with A[9], as shown in (c), node 4 is fixed up, and the recursive call HEAPIFY(A, 9) yields no further change to the data structure.

HEAPIFY(A, i)

1 l    LEFT(i)

2 r    RIGHT(i)

3 if l    heap-size[A] and A[l] > A[i]

4     then largest    l

5     else largest    i

6 if r    heap-size[A] and A[r] > A[largest]

7     then largest    r

8 if largest    i

9     then exchange A[i]    A[largest]

10          HEAPIFY(A,largest)

Figure 7.2 illustrates the action of HEAPIFY. At each step, the largest of the elements A[i], A[LEFT(i)], and A[RIGHT(i)] is determined, and its index is stored in largest. If A[i] is largest, then the subtree rooted at node i is a heap and the procedure terminates. Otherwise, one of the two children has the largest element, and A[i] is swapped with A[largest], which causes node i and its children to satisfy the heap property. The node largest, however, now has the original value A[i], and thus the subtree rooted at largest may violate the heap property. Consequently, HEAPIFY must be called recursively on that subtree.

The running time of HEAPIFY on a subtree of size n rooted at given node i is the [image] (1) time to fix up the relationships among the elements A[i], A[LEFT (i)], and A[RIGHT (i)], plus the time to run HEAPIFY on a subtree rooted at one of the children of node i. The children's subtrees each have size at most 2n/3--the worst case occurs when the last row of the tree is exactly half full--and the running time of HEAPIFY can therefore be described by the recurrence

T(n)    T(2n/3) +   (1).

The solution to this recurrence, by case 2 of the master theorem (Theorem 4.1), is T(n) = O(lg n). Alternatively, we can characterize the running time of HEAPIFY on a node of height h as O(h).

Exercises

7.2-1

Using Figure 7.2 as a model, illustrate the operation of HEAPIFY (A, 3) on the array A = [image] 27, 17, 3, 16, 13, 10, 1, 5, 7, 12, 4, 8, 9, 0 [image].

7.2-2

What is the effect of calling HEAPIFY(A, i) when the element A[i] is larger than its children?

7.2-3

What is the effect of calling HEAPIFY (A, i) for i > heap-size [A]/2?

7.2-4

The code for HEAPIFY is quite efficient in terms of constant factors, except possibly for the recursive call in line 10, which might cause some compilers to produce inefficient code. Write an efficient HEAPIFY that uses an iterative control construct (a loop) instead of recursion.

7.2-5

Show that the worst-case running time of HEAPIFY on a heap of size n is [image] (lg n). (Hint: For a heap with n nodes, give node values that cause HEAPIFY to be called recursively at every node on a path from the root down to a leaf.)

7.3 Building a heap

We can use the procedure HEAPIFY in a bottom-up manner to convert an array A[1 . . n], where n = length[A], into a heap. Since the elements in the subarray A[( [image] n/2 [image] + l) . . n] are all leaves of the tree, each is a 1-element heap to begin with. The procedure BUILD-HEAP goes through the remaining nodes of the tree and runs HEAPIFY on each one. The order in which the nodes are processed guarantees that the subtrees rooted at children of a node i are heaps before HEAPIFY is run at that node.

BUILD-HEAP(A)

1  heap-size[A]    length[A]

2  for i      length[A]/2  downto 1

3        do HEAPIFY(A, i)

Figure 7.3 shows an example of the action of BUILD-HEAP.

We can compute a simple upper bound on the running time of BUILD-HEAP as follows. Each call to HEAPIFY costs O(lg n) time, and there are O(n) such calls. Thus, the running time is at most O(n lg n). This upper bound, though correct, is not asymptotically tight.

We can derive a tighter bound by observing that the time for HEAPIFY to run at a node varies with the height of the node in the tree, and the heights of most nodes are small. Our tighter analysis relies on the property that in an n-element heap there are at most [image] n/2h+1 [image] nodes of height h (see Exercise 7.3-3).

The time required by HEAPIFY when called on a node of height h is O(h), so we can express the total cost of BUILD-HEAP as

(7.2)

The last summation can be evaluated by substituting x = 1/2 in the formula (3.6), which yields

Thus, the running time of BUILD-HEAP can be bounded as

Hence, we can build a heap from an unordered array in linear time.

Figure 7.3 The operation of BUILD-HEAP, showing the data structure before the call to HEAPIFY in line 3 of BUILD-HEAP. (a) A 10-element input array A and the binary tree it represents. The figure shows that the loop index i points to node 5 before the call HEAPIFY(A, i). (b) The data structure that results. The loop index i for the next iteration points to node 4. (c)-(e) Subsequent iterations of the for loop in BUILD-HEAP. Observe that whenever HEAPIFY is called on a node, the two subtrees of that node are both heaps. (f) The heap after BUILD-HEAP finishes.

Exercises

7.3-1

Using Figure 7.3 as a model, illustrate the operation of BUILD-HEAP on the array A = [image] 5, 3, 17, 10, 84, 19, 6, 22, 9 [image].

7.3-2

Why do we want the loop index i in line 2 of BUILD-HEAP to decrease from [image] length[A]/2 [image] to 1 rather than increase from 1 to [image] length[A]/2 [image]?

7.3-3

Show that there are at most [image] n/2h+1 [image] nodes of height h in any n-element heap.

7.4 The heapsort algorithm

The heapsort algorithm starts by using BUILD-HEAP to build a heap on the input array A[1 . . n], where n = length[A]. Since the maximum element of the array is stored at the root A[1], it can be put into its correct final position by exchanging it with A[n]. If we now "discard" node n from the heap (by decrementing heap-size[A]), we observe that A[1 . . (n - 1)] can easily be made into a heap. The children of the root remain heaps, but the new root element may violate the heap property (7.1). All that is needed to restore the heap property, however, is one call to HEAPIFY(A, 1), which leaves a heap in A[1 . . (n - 1)]. The heapsort algorithm then repeats this process for the heap of size n - 1 down to a heap of size 2.

HEAPSORT(A)

1  BUILD-HEAP(A)

2  for i    length[A] downto 2

3        do exchange A[1]    A[i]

4           heap-size[A]    heap-size[A] -1

5           HEAPIFY(A, 1)

Figure 7.4 shows an example of the operation of heapsort after the heap is initially built. Each heap is shown at the beginning of an iteration of the for loop in line 2.

The HEAPSORT procedure takes time O(n lg n), since the call to BUILD-HEAP takes time O(n) and each of the n - 1 calls to HEAPIFY takes time O(lg n).

Figure 7.4 The operation of HEAPSORT. (a) The heap data structure just after it has been built by BUILD-HEAP. (b)-(j) The heap just after each call of HEAPIFY in line 5. The value of i at that time is shown. Only lightly shaded nodes remain in the heap. (k) The resulting sorted array A.

Exercises

7.4-1

Using Figure 7.4 as a model, illustrate the operation of HEAPSORT on the array A = [image] 5, 13, 2, 25, 7, 17, 20, 8, 4 [image].

7.4-2

What is the running time of heapsort on an array A of length n that is already sorted in increasing order? What about decreasing order?

7.4-3

Show that the running time of heapsort is [image] (n lg n).

7.5 Priority queues

Heapsort is an excellent algorithm, but a good implementation of quicksort, presented in Chapter 8, usually beats it in practice. Nevertheless, the heap data structure itself has enormous utility. In this section, we present one of the most popular applications of a heap: its use as an efficient priority queue.

A priority queue is a data structure for maintaining a set S of elements, each with an associated value called a key. A priority queue supports the following operations.

INSERT(S, x) inserts the element x into the set S. This operation could be written as S [image] S [image] {x}.

MAXIMUM(S) returns the element of S with the largest key.

EXTRACT-MAX(S) removes and returns the element of S with the largest key.

One application of priority queues is to schedule jobs on a shared computer. The priority queue keeps track of the jobs to be performed and their relative priorities. When a job is finished or interrupted, the highest-priority job is selected from those pending using EXTRACT-MAX. A new job can be added to the queue at any time using INSERT.

A priority queue can also be used in an event-driven simulator. The items in the queue are events to be simulated, each with an associated time of occurrence that serves as its key. The events must be simulated in order of their time of occurrence, because the simulation of an event can cause other events to be simulated in the future. For this application, it is natural to reverse the linear order of the priority queue and support the operations MINIMUM and EXTRACT-MIN instead of MAXIMUM and EXTRACT-MAX. The simulation program uses EXTRACT-MIN at each step to choose the next event to simulate. As new events are produced, they are inserted into the priority queue using INSERT.

Not surprisingly, we can use a heap to implement a priority queue. The operation HEAP-MAXIMUM returns the maximum heap element in [image] (1) time by simply returning the value A[1] in the heap. The HEAP-EXTRACT-MAX procedure is similar to the for loop body (lines 3-5) of the HEAPSORT procedure:

HEAP-EXTRACT-MAX(A)

1  if heap-size[A] < 1

2     then error "heap underflow"

3  max    A[1]

4  A[1]    A[heap-size[A]]

5  heap-size[A]    heap-size[A] - 1

6  HEAPIFY(A, 1)

7  return max

The running time of HEAP-EXTRACT-MAX is O(lg n), since it performs only a constant amount of work on top of the O(lg n) time for HEAPIFY.

The HEAP-INSERT procedure inserts a node into heap A. To do so, it first expands the heap by adding a new leaf to the tree. Then, in a manner reminiscent of the insertion loop (lines 5-7) of INSERTION-SORT from Section 1.1, it traverses a path from this leaf toward the root to find a proper place for the new element.

HEAP-INSERT(A,key)

1  heap-size[A]    heap-size[A] + 1

2  i    heap-size[A]

3  while i > 1 and A[PARENT(i)] < key

4      do A[i]    A[PARENT(i)]

5         i    PARENT(i)

6  A[i]    key

Figure 7.5 shows an example of a HEAP-INSERT operation. The running time of HEAP-INSERT on an n-element heap is O(lg n), since the path traced from the new leaf to the root has length O(lg n).

In summary, a heap can support any priority-queue operation on a set of size n in O(lg n) time.

Exercises

7.5-1

Using Figure 7.5 as a model, illustrate the operation of HEAP-INSERT(A, 3) on the heap A = [image] 15,13, 9, 5, 12, 8, 7, 4, 0, 6, 2, 1 [image].

7.5-2

Illustrate the operation of HEAP-EXTRACT-MAX on the heap A = [image] 15, 13, 9, 5, 12, 8, 7, 4, 0, 6, 2, 1 [image].

Figure 7.5 The operation of HEAP-INSERT. (a) The heap of Figure 7.4(a) before we insert a node with key 15. (b) A new leaf is added to the tree. (c) Values on the path from the new leaf to the root are copied down until a place for the key 15 is found. (d) The key 15 is inserted.

7.5-3

Show how to implement a first-in, first-out queue with a priority queue. Show how to implement a stack with a priority queue. (FIFO's and stacks are defined in Section 11.1.)

7.5-4

Give an O(lg n)-time implementation of the procedure HEAP-INCREASE-KEY(A, i, k), which sets A[i] [image] max(A[i],k) and updates the heap structure appropriately.

7.5-5

The operation HEAP-DELETE(A, i) deletes the item in node i from heap A. Give an implementation of HEAP-DELETE that runs in O(lg n) time for an n-element heap.

7.5-6

Give an O(n lg k)-time algorithm to merge k sorted lists into one sorted list, where n is the total number of elements in all the input lists. (Hint: Use a heap for k-way merging.)

Problems

7-1 Building a heap using insertion

The procedure BUILD-HEAP in Section 7.3 can be implemented by repeatedly using HEAP-INSERT to insert the elements into the heap. Consider the following implementation:

BUILD-HEAP'(A )

1  heap-size[A]    1

2  for i    2 to length[A]

3        do HEAP-INSERT(A, A[i])

a. Do the procedures BUILD-HEAP and BUILD-HEAP' always create the same heap when run on the same input array? Prove that they do, or provide a counterexample.

b. Show that in the worst case, BUILD-HEAP' requires [image] (n lg n) time to build an n-element heap.

7-2 Analysis of d-ary heaps

A d-ary heap is like a binary heap, but instead of 2 children, nodes have d children.

a. How would you represent a d-ary heap in an array?

b. What is the height of a d-ary heap of n elements in terms of n and d?

c. Give an efficient implementation of EXTRACT-MAX. Analyze its running time in terms of d and n.

d. Give an efficient implementation of INSERT. Analyze its running time in terms of d and n.

e. Give an efficient implementation of HEAP-INCREASE-KEY(A, i, k), which sets A[i] [image] max(A[i], k) and updates the heap structure appropriately. Analyze its running time in terms of d and n.

Chapter notes

The heapsort algorithm was invented by Williams [202], who also described how to implement a priority queue with a heap. The BUILD-HEAP procedure was suggested by Floyd [69].

Go to Chapter 8 Back to Table of Contents


Intro to Algorithms: CHAPTER 8: QUICKSORT

 Return to Table of Contents

CHAPTER 8: QUICKSORT

Quicksort is a sorting algorithm whose worst-case running time is [image] (n2) on an input array of n numbers. In spite of this slow worst-case running time, quicksort is often the best practical choice for sorting because it is remarkably efficient on the average: its expected running time is [image] (n lg n), and the constant factors hidden in the [image] (n lg n) notation are quite small. It also has the advantage of sorting in place (see page 3), and it works well even in virtual memory environments.

Section 8.1 describes the algorithm and an important subroutine used by quicksort for partitioning. Because the behavior of quicksort is complex, we start with an intuitive discussion of its performance in Section 8.2 and postpone its precise analysis to the end of the chapter. Section 8.3 presents two versions of quicksort that use a random-number generator. These "randomized" algorithms have many desirable properties. Their average-case running time is good, and no particular input elicits their worst-case behavior. One of the randomized versions of quicksort is analyzed in Section 8.4, where it is shown to run in O(n2) time in the worst case and in O(n lg n) time on average.

8.1 Description of quicksort

Quicksort, like merge sort, is based on the divide-and-conquer paradigm introduced in Section 1.3.1. Here is the three-step divide-and-conquer process for sorting a typical subarray A[p . . r].

Divide: The array A[p . . r] is partitioned (rearranged) into two nonempty subarrays A[p . . q] and A[q + 1 . . r] such that each element of A[p . . q] is less than or equal to each element of A[q + 1 . . r]. The index q is computed as part of this partitioning procedure.

Conquer: The two subarrays A[p . . q] and A[q + 1 . . r] are sorted by recursive calls to quicksort.

Combine: Since the subarrays are sorted in place, no work is needed to combine them: the entire array A[p . . r] is now sorted.

The following procedure implements quicksort.

QUICKSORT(A,p,r)

1  if p < r

2      then q    PARTITION(A,p,r)

3           QUICKSORT(A,p,q)

4           QUICKSORT(A,q + 1,r)

To sort an entire array A, the initial call is QUICKSORT(A, 1, length[A]).

Partitioning the array

The key to the algorithm is the PARTITION procedure, which rearranges the subarray A[p . . r] in place.

PARTITION(A,p,r)

1  x    A[p]

2  i    p - 1

3  j    r + 1

4  while TRUE

5      do repeat j    j - 1

6           until A[j]    x

7         repeat i    i + 1

8           until A[i]    x

9         if i < j

10            then exchange A[i]    A[j]

11            else return j

Figure 8.1 shows how PARTITION works. It first selects an element x = A[p] from A[p . . r] as a "pivot" element around which to partition A[p . . r]. It then grows two regions A[p . . i] and A[j . . r] from the top and bottom of A[p . . r], respectively, such that every element in A[p . . i] is less than or equal to x and every element in A[j . . r] is greater than or equal to x. Initially, i = p - 1 and j = r + 1, so the two regions are empty.

Within the body of the while loop, the index j is decremented and the index i is incremented, in lines 5-8, until A[i] [image] x [image] A[j]. Assuming that these inequalities are strict, A[i] is too large to belong to the bottom region and A[j] is too small to belong to the top region. Thus, by exchanging A[i] and A[j] as is done in line 10, we can extend the two regions. (If the inequalities are not strict, the exchange can be performed anyway.)

The body of the while loop repeats until i [image] j, at which point the entire array A[p . . r] has been partitioned into two subarrays A[p . . q] and A[q + 1 . . r], where p [image] q < r, such that no element of A[p . . q] is larger than any element of A[q + 1. . r]. The value q = j is returned at the end of the procedure.

Conceptually, the partitioning procedure performs a simple function: it puts elements smaller than x into the bottom region of the array and elements larger than x into the top region. There are technicalities that make the pseudocode of PARTITION a little tricky, however. For example, the indices i and j never index the subarray A[p . . r] out of bounds, but this isn't entirely apparent from the code. As another example, it is important that A[p] be used as the pivot element x. If A[r] is used instead and it happens that A[r] is also the largest element in the subarray A[p . . r], then PARTITION returns to QUICKSORT the value q = r, and QUICKSORT loops forever. Problem 8-1 asks you to prove PARTITION correct.

Figure 8.1 The operation of PARTITION on a sample array. Lightly shaded array elements have been placed into the correct partitions, and heavily shaded elements are not yet in their partitions. (a) The input array, with the initial values of i and j just off the left and right ends of the array. We partition around x = A[p] = 5. (b) The positions of i and j at line 9 of the first iteration of the while loop. (c) The result of exchanging the elements pointed to by i and j in line 10. (d) The positions of i and j at line 9 of the second iteration of the while loop. (e) The positions of i and j at line 9 of the third and last iteration of the while loop. The procedure terminates because i j, and the value q = j is returned. Array elements up to and including A[j] are less than or equal to x = 5, and array elements after A[j] are greater than or equal to x = 5.

The running time of PARTITION on an array A[p . . r] is [image] (n), where n = r - p + 1 (see Exercise 8.1-3).

Exercises

8.1-1

Using Figure 8.1 as a model, illustrate the operation of PARTITION on the array A = [image] 13, 19, 9, 5, 12, 8, 7, 4, 11, 2, 6, 21 [image].

8.1-2

What value of q does PARTITION return when all elements in the array A[p . . r] have the same value?

8.1-3

Give a brief argument that the running time of PARTITION on a subarray of size n is [image] (n).

8.1-4

How would you modify QUICKSORT to sort in nonincreasing order?

8.2 Performance of quicksort

The running time of quicksort depends on whether the partitioning is balanced or unbalanced, and this in turn depends on which elements are used for partitioning. If the partitioning is balanced, the algorithm runs asymptotically as fast as merge sort. If the partitioning is unbalanced, however, it can run asymptotically as slow as insertion sort. In this section, we shall informally investigate how quicksort performs under the assumptions of balanced versus unbalanced partitioning.

Worst-case partitioning

The worst-case behavior for quicksort occurs when the partitioning routine produces one region with n - 1 elements and one with only l element. (This claim is proved in Section 8.4.1.) Let us assume that this unbalanced partitioning arises at every step of the algorithm. Since partitioning costs [image] (n) time and T(1) = [image] (1), the recurrence for the running time is

T(n) = T(n - 1) +   (n).

To evaluate this recurrence, we observe that T(1) = [image] (1) and then iterate:

We obtain the last line by observing that [image] is the arithmetic series (3.2). Figure 8.2 shows a recursion tree for this worst-case execution of quicksort. (See Section 4.2 for a discussion of recursion trees.)

Thus, if the partitioning is maximally unbalanced at every recursive step of the algorithm, the running time is [image] (n2). Therefore the worstcase running time of quicksort is no better than that of insertion sort. Moreover, the [image] (n2) running time occurs when the input array is already completely sorted--a common situation in which insertion sort runs in O(n) time.

Figure 8.2 A recursion tree for QUICKSORT in which the PARTITION procedure always puts only a single element on one side of the partition (the worst case). The resulting running time is (n2).

Best-case partitioning

If the partitioning procedure produces two regions of size n/2, quicksort runs much faster. The recurrence is then

T(n) = 2T(n/2) +   (n),

which by case 2 of the master theorem (Theorem 4.1) has solution T(n) = [image] (n lg n). Thus, this best-case partitioning produces a much faster algorithm. Figure 8.3 shows the recursion tree for this best-case execution of quicksort.

Balanced partitioning

The average-case running time of quicksort is much closer to the best case than to the worst case, as the analyses in Section 8.4 will show. The key to understanding why this might be true is to understand how the balance of the partitioning is reflected in the recurrence that describes the running time.

Suppose, for example, that the partitioning algorithm always produces a 9-to-1 proportional split, which at first blush seems quite unbalanced. We then obtain the recurrence

T(n) = T(9n/10) + T(n/10) + n

on the running time of quicksort, where we have replaced [image] (n) by n for convenience. Figure 8.4 shows the recursion tree for this recurrence. Notice that every level of the tree has cost n, until a boundary condition is reached at depth log10 n = [image] (lg n), and then the levels have cost at most n. The recursion terminates at depth log10/9 n = [image] (lg n). The total cost of quicksort is therefore [image] (n lg n). Thus, with a 9-to-1 proportional split at every level of recursion, which intuitively seems quite unbalanced, quicksort runs in [image] (n lg n) time--asymptotically the same as if the split were right down the middle. In fact, even a 99-to-1 split yields an O(n lg n) running time. The reason is that any split of constant proportionality yields a recursion tree of depth [image] (lg n), where the cost at each level is O(n). The running time is therefore [image] (n lg n) whenever the split has constant proportionality.

Figure 8.3 A recursion tree for QUICKSORT in which PARTITION always balances the two sides of the partition equally (the best case). The resulting running time is (n lg n).

Figure 8.4 A recursion tree for QUICKSORT in which PARTITION always produces a 9-to-1 split, yielding a running time of (n lg n).

Intuition for the average case

To develop a clear notion of the average case for quicksort, we must make an assumption about how frequently we expect to encounter the various inputs. A common assumption is that all permutations of the input numbers are equally likely. We shall discuss this assumption in the next section, but first let's explore its ramifications.

When we run quicksort on a random input array, it is unlikely that the partitioning always happens in the same way at every level, as our informal analysis has assumed. We expect that some of the splits will be reasonably well balanced and that some will be fairly unbalanced. For example, Exercise 8.2-5 asks to you show that about 80 percent of the time PARTITION produces a split that is more balanced than 9 to 1, and about 20 percent of the time it produces a split that is less balanced than 9 to 1.

In the average case, PARTITION produces a mix of "good" and "bad" splits. In a recursion tree for an average-case execution of PARTITION, the good and bad splits are distributed randomly throughout the tree. Suppose for the sake of intuition, however, that the good and bad splits alternate levels in the tree, and that the good splits are best-case splits and the bad splits are worst-case splits. Figure 8.5(a) shows the splits at two consecutive levels in the recursion tree. At the root of the tree, the cost is n for partitioning and the subarrays produced have sizes n - 1 and 1: the worst case. At the next level, the subarray of size n - 1 is best-case partitioned into two subarrays of size (n - 1)/2. Let's assume that the boundary-condition cost is 1 for the subarray of size 1.

The combination of the bad split followed by the good split produces three subarrays of sizes 1, (n -1)/2, and (n - 1)/2 at a combined cost of 2n - 1 = [image] (n). Certainly, this situation is no worse than that in Figure 8. 5 (b), namely a single level of partitioning that produces two subarrays of sizes (n - 1)/2 + 1 and (n - 1)/2 at a cost of n = [image] (n). Yet this latter situation is very nearly balanced, certainly better than 9 to 1. Intuitively, the [image] (n) cost of the bad split can be absorbed into the [image] (n) cost of the good split, and the resulting split is good. Thus, the running time of quicksort, when levels alternate between good and bad splits, is like the running time for good splits alone: still O(n lg n), but with a slightly larger constant hidden by the O-notation. We shall give a rigorous analysis of the average case in Section 8.4.2.

Figure 8.5 (a) Two levels of a recursion tree for quicksort. The partitioning at the root costs n and produces a "bad" split: two subarrays of sizes 1 and n - 1. The partitioning of the subarray of size n - 1 costs n - 1 and produces a "good" split: two subarrays of size (n - 1)/2. (b) A single level of a recursion tree that is worse than the combined levels in (a), yet very well balanced.

Exercises

8.2-1

Show that the running time of QUICKSORT is [image] (n lg n) when all elements of array A have the same value.

8.2-2

Show that the running time of QUICKSORT is [image] (n2) when the array A is sorted in nonincreasing order.

8.2-3

Banks often record transactions on an account in order of the times of the transactions, but many people like to receive their bank statements with checks listed in order by check number. People usually write checks in order by check number, and merchants usually cash them with reasonable dispatch. The problem of converting time-of-transaction ordering to check-number ordering is therefore the problem of sorting almost-sorted input. Argue that the procedure INSERTION-SORT would tend to beat the procedure QUICKSORT on this problem.

8.2-4

Suppose that the splits at every level of quicksort are in the proportion 1 - [image] to [image], where 0 < [image] [image] 1/2 is a constant. Show that the minimum depth of a leaf in the recursion tree is approximately ep1g n/lg [image] and the maximum depth is approximately - lg n/lg(1 - [image]). (Don't worry about integer round-off.)

8.2-5

Argue that for any constant 0 < [image] [image] 1/2, the probability is approximately 1 - 2 [image] that on a random input array, PARTITION produces a split more balanced than 1 - [image] to [image]. For what value of [image] are the odds even that the split is more balanced than less balanced?

8.3 Randomized versions of quicksort

In exploring the average-case behavior of quicksort, we have made an assumption that all permutations of the input numbers are equally likely. When this assumption on the distribution of the inputs is valid, many people regard quicksort as the algorithm of choice for large enough inputs. In an engineering situation, however, we cannot always expect it to hold. (See Exercise 8.2-3.) This section introduces the notion of a randomized algorithm and presents two randomized versions of quicksort that overcome the assumption that all permutations of the input numbers are equally likely.

An alternative to assuming a distribution of inputs is to impose a distribution. For example, suppose that before sorting the input array, quicksort randomly permutes the elements to enforce the property that every permutation is equally likely. (Exercise 8.3-4 asks for an algorithm that randomly permutes the elements of an array of size n in time O(n).) This modification does not improve the worst-case running time of the algorithm, but it does make the running time independent of the input ordering.

We call an algorithm randomized if its behavior is determined not only by the input but also by values produced by a random-number generator. We shall assume that we have at our disposal a random-number generator RANDOM. A call to RANDOM(a,b) returns an integer between a and b, inclusive, with each such integer being equally likely. For example, RANDOM(0, 1) produces a 0 with probability 1/2 and a 1 with probability 1/2. Each integer returned by RANDOM is independent of the integers returned on previous calls. You may imagine RANDOM as rolling a (b - a + 1 )-sided die to obtain its output. (In practice, most programming environments offer a pseudorandom-number generator: a deterministic algorithm that returns numbers that "look" statistically random.)

This randomized version of quicksort has an interesting property that is also possessed by many other randomized algorithms: no particular input elicits its worst-case behavior. Instead, its worst case depends on the random-number generator. Even intentionally, you cannot produce a bad input array for quicksort, since the random permutation makes the input order irrelevant. The randomized algorithm performs badly only if the random-number generator produces an unlucky permutation to be sorted. Exercise 13.4-4 shows that almost all permutations cause quicksort to perform nearly as well as the average case: there are very few permutations that cause near-worst-case behavior.

A randomized strategy is typically useful when there are many ways in which an algorithm can proceed but it is difficult to determine a way that is guaranteed to be good. If many of the alternatives are good, simply choosing one randomly can yield a good strategy. Often, an algorithm must make many choices during its execution. If the benefits of good choices outweigh the costs of bad choices, a random selection of good and bad choices can yield an efficient algorithm. We noted in Section 8.2 that a mixture of good and bad splits yields a good running time for quicksort, and thus it makes sense that randomized versions of the algorithm should perform well.

By modifying the PARTITION procedure, we can design another randomized version of quicksort that uses this random-choice strategy. At each step of the quicksort algorithm, before the array is partitioned, we exchange element A[p] with an element chosen at random from A[p . . r]. This modification ensures that the pivot element x = A[p] is equally likely to be any of the r - p + 1 elements in the subarray. Thus, we expect the split of the input array to be reasonably well balanced on average. The randomized algorithm based on randomly permuting the input array also works well on average, but it is somewhat more difficult to analyze than this version.

The changes to PARTITION and QUICKSORT are small. In the new partition procedure, we simply implement the swap before actually partitioning:

RANDOMIZED-PARTITION(A,p,r)

1  i    RANDOM(p,r)

2  exchange A[p]    A[i]

3  return PARTITION(A,p,r)

We now make the new quicksort call RANDOMIZED-PARTITION in place of PARTITION:

RANDOMIZED-QUICKSORT(A,p,r)

1  if p < r

2      then q    RANDOMIZED-PARTITION(A,p,r)

3           RANDOMIZED-QUICKSORT(A,p,q)

4           RANDOMIZED-QUICKSORT(A,q + 1,r)

We analyze this algorithm in the next section.

Exercises

8.3-1

Why do we analyze the average-case performance of a randomized algorithm and not its worst-case performance?

8.3-2

During the running of the procedure RANDOMIZED-QUICKSORT, how many calls are made to the random-number generator RANDOM in the worst case? How does the answer change in the best case?

8.3-3

Describe an implementation of the procedure RANDOM(a, b) that uses only fair coin flips. What is the expected running time of your procedure?

8.3-4

Give a [image] (n)-time, randomized procedure that takes as input an array A[1 . . n] and performs a random permutation on the array elements.

8.4 Analysis of quicksort

Section 8.2 gave some intuition for the worst-case behavior of quicksort and for why we expect it to run quickly. In this section, we analyze the behavior of quicksort more rigorously. We begin with a worst-case analysis, which applies to either QUICKSORT or RANDOMIZED-QUICKSORT, and conclude with an average-case analysis of RANDOMIZED-QUICKSORT.

8.4.1 Worst-case analysis

We saw in Section 8.2 that a worst-case split at every level of recursion in quicksort produces a [image] (n2) running time, which, intuitively, is the worst-case running time of the algorithm. We now prove this assertion.

Using the substitution method (see Section 4.1), we can show that the running time of quicksort is O(n2). Let T(n) be the worst-case time for the procedure QUICKSORT on an input of size n. We have the recurrence

(8.1)

where the parameter q ranges from 1 to n - 1 because the procedure PARTITION produces two regions, each having size at least 1. We guess that T(n) [image] cn2 for some constant c. Substituting this guess into (8.1), we obtain

The expression q2 + (n - q)2 achieves a maximum over the range 1 [image] q [image] n - 1 at one of the endpoints, as can be seen since the second derivative of the expression with respect to q is positive (see Exercise 8.4-2). This gives us the bound max1 [image] q [image] n - 1(q2 + (n - q)2) [image] 12 + (n - 1)2 = n2 - 2(n - 1).

Continuing with our bounding of T(n), we obtain

T(n)    cn2 - 2c(n - 1) +   (n)

   cn2 ,

since we can pick the constant c large enough so that the 2c(n - 1) term dominates the [image] (n) term. Thus, the (worst-case) running time of quicksort is [image] (n2).

8.4.2 Average-case analysis

We have already given an intuitive argument why the average-case running time of RANDOMIZED-QUICKSORT is [image] (n 1g n): if the split induced by RANDOMIZED-PARTITION puts any constant fraction of the elements on one side of the partition, then the recursion tree has depth [image] (1g n) and [image] (n) work is performed at [image] (1g n) of these levels. We can analyze the expected running time of RANDOMIZED-QUICKSORT precisely by first understanding how the partitioning procedure operates. We can then develop a recurrence for the average time required to sort an n-element array and solve this recurrence to determine bounds on the expected running time. As part of the process of solving the recurrence, we shall develop tight bounds on an interesting summation.

Analysis of partitioning

We first make some observations about the operation of PARTITION. When PARTITION is called in line 3 of the procedure RANDOMIZED-PARTITION, the element A[p] has already been exchanged with a random element in A[p . . r]. To simplify the analysis, we assume that all input numbers are distinct. If all input numbers are not distinct, it is still true that quick-sort's average-case running time is O(n lg n), but a somewhat more intricate analysis than we present here is required.

Our first observation is that the value of q returned by PARTITION depends only on the rank of x = A[p] among the elements in A[p . . r]. (The rank of a number in a set is the number of elements less than or equal to it.) If we let n = r - p + 1 be the number of elements in A[p . . r], swapping A[p] with a random element from A[p . . r] yields a probability 1/n that rank(x) = i for i = 1,2, . . . , n.

We next compute the likelihoods of the various outcomes of the partitioning. If rank(x) = 1, then the first time through the while loop in lines 4-11 of PARTITION, index i stops at i = p and index j stops at j = p. Thus, when q = j is returned, the "low" side of the partition contains the sole element A[p]. This event occurs with probability 1/n since that is the probability that rank(x) = 1.

If rank(x) [image] 2, then there is at least one element smaller than x = A[p]. Consequently, the first time through the while loop, index i stops at i = p but j stops before reaching p. An exchange with A[p] is then made to put A[p] in the high side of the partition. When PARTITION terminates, each of the rank(x) - 1 elements in the low side of the partition is strictly less than x. Thus, for each i = 1,2, . . . , n - l, when rank(x) [image] 2, the probability is 1/n that the low side of the partition has i elements.

Combining these two cases, we conclude that the size q - p + 1 of the low side of the partition is 1 with probability 2/n and that the size is i with probability 1 /n for i = 2,3, . . . , n - 1.

A recurence for the average case

We now establish a recurrence for the expected running time of RANDOMIZED-QUICKSORT. Let T(n) denote the average time required to sort an n-element input array. A call to RANDOMIZED-QUICKSORT with a 1-element array takes constant time, so we have T(1) = [image] (1). A call to RANDOMIZED-QUICKSORT with an array A[l . . n] of length n uses time [image] (n) to partition the array. The PARTITION procedure returns an index q, and then RANDOMIZED-QUICKSORT is called recursively with subarrays of length q and n - q. Consequently, the average time to sort an array of length n can be expressed as

(8.2)

The value of q has an almost uniform distribution, except that the value q = 1 is twice as likely as the others, as was noted above. Using the facts that T(1) = [image] (1) and T(n - 1) = O(n2) from our worst-case analysis, we have

and the term [image] (n) in equation (8.2) can therefore absorb the expression [image]. We can thus restate recurrence (8.2) as

(8.3)

Observe that for k = 1,2, . . . , n - 1, each term T(k) of the sum occurs once as T(q) and once as T(n - q). Collapsing the two terms of the sum yields

(8.4)

Solving the recurrence

We can solve the recurrence (8.4) using the substitution method. Assume inductively that T(n) [image] an 1g n + b for some constants a > 0 and b > 0 to be determined. We can pick a and b sufficiently large so that an 1g n + b is greater than T(1). Then for n > 1, we have by substitution

We show below that the summation in the last line can be bounded by

(8.5)

Using this bound, we obtain

since we can choose a large enough so that [image] dominates [image] (n) + b. We conclude that quicksort's average running time is O(n lg n).

Tight bounds on the key summation

It remains to prove the bound (8.5) on the summation

Since each term is at most n lg n, we have the bound

which is tight to within a constant factor. This bound is not strong enough to solve the recurrence as T(n) = O(n lg n), however. Specifically, we need a bound of [image] for the solution of the recurrence to work out.

We can get this bound on the summation by splitting it into two parts, as discussed in Section 3.2 on page 48. We obtain

The lg k in the first summation on the right is bounded above by 1g(n/2) = 1g n - 1. The lg k in the second summation is bounded above by lg n. Thus,

if n [image] 2. This is the bound (8.5).

Exercises

8.4-1

Show that quicksort's best-case running time is [image] (n1gn).

8.4-2

Show that q2 + (n - q)2 achieves a maximum over q = 1, 2, . . . , n - 1 when q = 1 or q = n - 1.

8.4-3

Show that RANDOMIZED-QUICKSORT's expected running time is [image] (n 1g n).

8.4-4

The running time of quicksort can be improved in practice by taking advantage of the fast running time of insertion sort when its input is "nearly" sorted. When quicksort is called on a subarray with fewer than k elements, let it simply return without sorting the subarray. After the top-level call to quicksort returns, run insertion sort on the entire array to finish the sorting process. Argue that this sorting algorithm runs in O(nk + n 1g(n/k)) expected time. How should k be picked, both in theory and in practice?

8.4-5

Prove the identity

and then use the integral approximation method to give a tighter upper bound than (8.5) on the summation [image].

8.4-6

Consider modifying the PARTITION procedure by randomly picking three elements from array A and partitioning about their median. Approximate the probability of getting at worst an [image] -to-(1 - [image] ) split, as a function of [image] in the range 0 < [image] < 1.

Problems

8-1 Partition correctness

Give a careful argument that the procedure PARTITION in Section 8.1 is correct. Prove the following:

a. The indices i and j never reference an element of A outside the interval [p . . r].

b. The index j is not equal to r when PARTITION terminates (so that the split is always nontrivial).

c. Every element of A[p . . j] is less than or equal to every element of A[j+ 1 . . r] when PARTITION terminates.

8-2 Lomuto's partitioning algorithm

Consider the following variation of PARTITION, due to N. Lomuto. To partition A[p . . r], this version grows two regions, A[p . . i] and A[i + 1 . . j], such that every element in the first region is less than or equal to x = A [r] and every element in the second region is greater than x.

LOMUTO-PARTITION(A, p, r)

1  x    A[r]

2  i    p - 1

3  for j    p to r

4       do if A[j]    x

5             then i    i + 1

6                  exchange A[i]    A[j]

7  if i < r

8      then return i

9      else return i - 1

a. Argue that LOMUTO-PARTITION is correct.

b. What are the maximum numbers of times that an element can be moved by PARTITION and by LOMUTO-PARTITION?

c. Argue that LOMUTO-PARTITION, like PARTITION, runs in [image] (n) time on an n-element subarray.

d. How does replacing PARTITION by LOMUTO-PARTITION affect the running time of QUICKSORT when all input values are equal?

e. Define a procedure RANDOMIZED-LOMUTO-PARTITION that exchanges A[r] with a randomly chosen element in A[p . . r] and then calls LOMUTO-PARTITION. Show that the probability that a given value q is returned by RANDOMIZED-LOMUTO-PARTITION is equal to the probability that p + r - q is returned by RANDOMIZED-PARTITION.

8-3 Stooge sort

Professors Howard, Fine, and Howard have proposed the following "elegan" sorting algorithm:

a. Argue that STOOGE-SORT(A, 1, length[A]) correctly sorts the input array A[1 . . n], where n = length[A].

b. Give a recurrence for the worst-case running time of STOOGE-SORT and a tight asymptotic ( [image] -notation) bound on the worst-case running time.

c. Compare the worst-case running time of STOOGE-SORT with that of insertion sort, merge sort, heapsort, and quicksort. Do the professors deserve tenure?

8-4 Stack depth for quicksort

The QUICKSORT algorithm of Section 8.1 contains two recursive calls to itself. After the call to PARTITION, the left subarray is recursively sorted and then the right subarray is recursively sorted. The second recursive call in QUICKSORT is not really necessary; it can be avoided by using an iterative control structure. This technique, called tail recursion, is provided automatically by good compilers. Consider the following version of quicksort, which simulates tail recursion.

QUICKSORT'(A,p,r)

1  while p < r

2      do    Partition and sort left subarray

3         q    PARTITION(A,p,r)

4         QUICKSORT'(A,p,q)

5         p    q + 1

a. Argue that QUICKSORT'(A, 1, length[A]) correctly sorts the array A.

Compilers usually execute recursive procedures by using a stack that contains pertinent information, including the parameter values, for each recursive call. The information for the most recent call is at the top of the stack, and the information for the initial call is at the bottom. When a procedure is invoked, its information is pushed onto the stack; when it terminates, its information is popped. Since we assume that array parameters are actually represented by pointers, the information for each procedure call on the stack requires O(1) stack space. The stack depth is the maximum amount of stack space used at any time during a computation.

b. Describe a scenario in which the stack depth of QUICKSORT' is [image] (n) on an n-element input array.

c. Modify the code for QUICKSORT' so that the worst-case stack depth is [image] (1g n).

8-5 Median-of-3 partition

One way to improve the RANDOMIZED-QUICKSORT procedure is to partition around an element x that is chosen more carefully than by picking a random element from the subarray. One common approach is the median-of-3 method: choose x as the median (middle element) of a set of 3 elements randomly selected from the subarray. For this problem, let us assume that the elements in the input array A[1 . . n] are distinct and that n [image] 3. We denote the sorted output array by A'[1 . . n]. Using the median-of-3 method to choose the pivot element x, define pi = Pr{x = A'[i]}.

a. Give an exact formula for pi as a function of n and i for i = 2, 3, . . . , n - 1 . (Note that p1 = pn = 0.)

b. By what amount have we increased the likelihood of choosing x = A'[ [image] (n + 1)/2 [image] ], the median of A[1 . . n], compared to the ordinary implementation? Assume that n [image] [image], and give the limiting ratio of these probabilities.

c. If we define a "good" split to mean choosing x = A'[i], where n/3 [image] i [image] 2n/3, by what amount have we increased the likelihood of getting a good split compared to the ordinary implementation? (Hint: Approximate the sum by an integral.)

d. Argue that the median-of-3 method affects only the constant factor in the [image] (n 1g n) running time of quicksort.

Chapter notes

The quicksort procedure was invented by Hoare [98]. Sedgewick [174] provides a good reference on the details of implementation and how they matter. The advantages of randomized algorithms were articulated by Rabin [165].

Go to Chapter 9 Back to Table of Contents


Intro to Algorithms: CHAPTER 9: SORTING IN LINEAR TIME

 Return to Table of Contents

CHAPTER 9: SORTING IN LINEAR TIME

We have now introduced several algorithms that can sort n numbers in O(n lg n) time. Merge sort and heapsort achieve this upper bound in the worst case; quicksort achieves it on average. Moreover, for each of these algorithms, we can produce a sequence of n input numbers that causes the algorithm to run in [image] (n lg n) time.

These algorithms share an interesting property: the sorted order they determine is based only on comparisons between the input elements. We call such sorting algorithms comparison sorts. All the sorting algorithms introduced thus far are comparison sorts.

In Section 9.1, we shall prove that any comparison sort must make [image] (n lg n) comparisons in the worst case to sort a sequence of n elements. Thus, merge sort and heapsort are asymptotically optimal, and no comparison sort exists that is faster by more than a constant factor.

Sections 9.2 and 9.3 examine three sorting algorithms--counting sort, radix sort, and bucket sort--that run in linear time. Needless to say, these algorithms use operations other than comparisons to determine the sorted order. Consequently, the [image] (n lg n) lower bound does not apply to them.

9.1 Lower bounds for sorting

In a comparison sort, we use only comparisons between elements to gain order information about an input sequence [image] a1, a2, . . . ,an) That is, given two elements ai and aj, we perform one of the tests ai < aj, ai [image] aj, ai = aj, ai [image] aj, or ai > aj to determine their relative order. We may not inspect the values of the elements or gain order information about them in any other way.

In this section, we assume without loss of generality that all of the input elements are distinct. Given this assumption, comparisons of the form ai = aj are useless, so we can assume that no comparisons of this form are made. We also note that the comparisons ai [image] aj, ai [image] aj, ai > aj, and ai < aj are all equivalent in that they yield identical information about the relative order of ai and aj. We therefore assume that all comparisons have the form ai [image] aj.

Figure 9.1 The decision tree for insertion sort operating on three elements. There are 3! = 6 possible permutations of the input elements, so the decision tree must have at least 6 leaves.

The decision-tree model

Comparison sorts can be viewed abstractly in terms of decision trees. A decision tree represents the comparisons performed by a sorting algorithm when it operates on an input of a given size. Control, data movement, and all other aspects of the algorithm are ignored. Figure 9.1 shows the decision tree corresponding to the insertion sort algorithm from Section 1.1 operating on an input sequence of three elements.

In a decision tree, each internal node is annotated by ai : aj for some i and j in the range 1 [image] i, j [image] n, where n is the number of elements in the input sequence. Each leaf is annotated by a permutation [image] [image] (1), [image] (2), . . . , [image] (n) [image]. (See Section 6.1 for background on permutations.) The execution of the sorting algorithm corresponds to tracing a path from the root of the decision tree to a leaf. At each internal node, a comparison ai [image] aj is made. The left subtree then dictates subsequent comparisons for ai [image] aj, and the right subtree dictates subsequent comparisons for ai > aj. When we come to a leaf, the sorting algorithm has established the ordering a [image] (1) [image] a [image] (2) [image] . . . a [image] (n). Each of the n! permutations on n elements must appear as one of the leaves of the decision tree for the sorting algorithm to sort properly.

A lower bound for the worst case

The length of the longest path from the root of a decision tree to any of its leaves represents the worst-case number of comparisons the sorting algorithm performs. Consequently, the worst-case number of comparisons for a comparison sort corresponds to the height of its decision tree. A lower bound on the heights of decision trees is therefore a lower bound on the running time of any comparison sort algorithm. The following theorem establishes such a lower bound.

Theorem 9.1

Any decision tree that sorts n elements has height [image] (n lg n).

Proof Consider a decision tree of height h that sorts n elements. Since there are n! permutations of n elements, each permutation representing a distinct sorted order, the tree must have at least n! leaves. Since a binary tree of height h has no more than 2h leaves, we have

n!    2h ,

which, by taking logarithms, implies

h    lg(n!) ,

since the lg function is monotonically increasing. From Stirling's approximation (2.11), we have

where e = 2.71828 . . . is the base of natural logarithms; thus

Corollary 9.2

Heapsort and merge sort are asymptotically optimal comparison sorts.

Proof The O(n lg n) upper bounds on the running times for heapsort and merge sort match the [image] (n lg n) worst-case lower bound from Theorem 9.1.

Exercises

9.1-1

What is the smallest possible depth of a leaf in a decision tree for a sorting algorithm?

9.1-2

Obtain asymptotically tight bounds on lg(n!) without using Stirling's approximation. Instead, evaluate the summation [image] using techniques from Section 3.2.

9.1-3

Show that there is no comparison sort whose running time is linear for at least half of the n! inputs of length n. What about a fraction of 1/n of the inputs of length n? What about a fraction 1/2n?

9.1-4

Professor Solomon claims that the [image] (n lg n) lower bound for sorting n numbers does not apply to his computer environment, in which the control flow of a program can split three ways after a single comparison ai : aj, according to whether ai < aj, ai = aj, or ai > aj. Show that the professor is wrong by proving that the number of three-way comparisons required to sort n elements is still [image] (n lg n).

9.1-5

Prove that 2n - 1 comparisons are necessary in the worst case to merge two sorted lists containing n elements each.

9.1-6

You are given a sequence of n elements to sort. The input sequence consists of n/k subsequences, each containing k elements. The elements in a given subsequence are all smaller than the elements in the succeeding subsequence and larger than the elements in the preceding subsequence. Thus, all that is needed to sort the whole sequence of length n is to sort the k elements in each of the n/k subsequences. Show an [image] (n lg k) lower bound on the number of comparisons needed to solve this variant of the sorting problem. (Hint: It is not rigorous to simply combine the lower bounds for the individual subsequences.)

9.2 Counting sort

Counting sort assumes that each of the n input elements is an integer in the range 1 to k, for some integer k. When k = O(n), the sort runs in O(n) time.

The basic idea of counting sort is to determine, for each input element x, the number of elements less than x. This information can be used to place element x directly into its position in the output array. For example, if there are 17 elements less than x, then x belongs in output position 18. This scheme must be modified slightly to handle the situation in which several elements have the same value, since we don't want to put them all in the same position.

In the code for counting sort, we assume that the input is an array A[1 . . n], and thus length[A] = n. We require two other arrays: the array B[1 . . n] holds the sorted output, and the array C[1 . . k] provides temporary working storage.

Figure 9.2 The operation of COUNTING-SORT on an input array A[1 . . 8], where each element ofA is a positive integer no larger than k = 6. (a) The array A and the auxiliary array C after line 4. (b) The array C after line 7. (c)-(e) The output array B and the auxiliary array C after one, two, and three iterations of the loop in lines 9-11, respectively. Only the lightly shaded elements of array B have been filled in. (f) The final sorted output array B.

Counting sort is illustrated in Figure 9.2. After the initialization in lines 1-2, we inspect each input element in lines 3-4. If the value of an input element is i, we increment C[i]. Thus, after lines 3-4, C[i] holds the number of input elements equal to i for each integer i = 1, 2, . . . , k. In lines 6-7, we determine for each i = 1, 2, . . . , k, how many input elements are less than or equal to i; this is done by keeping a running sum of the array C.

Finally, in lines 9-11, we place each element A[j] in its correct sorted position in the output array B. If all n elements are distinct, then when we first enter line 9, for each A[j], the value C[A[j]] is the correct final position of A[j] in the output array, since there are C[A[j]] elements less than or equal to A[j]. Because the elements might not be distinct, we decrement C[A[j]] each time we place a value A[j] into the B array; this causes the next input element with a value equal to A[j], if one exists, to go to the position immediately before A[j] in the output array.

How much time does counting sort require? The for loop of lines 1-2 takes time O(k), the for loop of lines 3-4 takes time O(n), the for loop of lines 6-7 takes time O(k), and the for loop of lines 9-11 takes time O(n). Thus, the overall time is O(k + n). In practice, we usually use counting sort when we have k = O(n), in which case the running time is O(n).

Counting sort beats the lower bound of [image] (n 1g n) proved in Section 9.1 because it is not a comparison sort. In fact, no comparisons between input elements occur anywhere in the code. Instead, counting sort uses the actual values of the elements to index into an array. The [image] (n lg n) lower bound for sorting does not apply when we depart from the comparison-sort model.

An important property of counting sort is that it is stable: numbers with the same value appear in the output array in the same order as they do in the input array. That is, ties between two numbers are broken by the rule that whichever number appears first in the input array appears first in the output array. Of course, the property of stability is important only when satellite data are carried around with the element being sorted. We shall see why stability is important in the next section.

Exercises

9.2-1

Using Figure 9.2 as a model, illustrate the operation of COUNTING-SORT on the array A = [image] 7, 1, 3, 1, 2, 4, 5, 7, 2, 4, 3 [image].

9.2-2

Prove that COUNTING-SORT is stable.

9.2-3

Suppose that the for loop in line 9 of the COUNTING-SORT procedure is rewritten:

9 for j    1 to length[A]

Show that the algorithm still works properly. Is the modified algorithm stable?

9.2-4

Suppose that the output of the sorting algorithm is a data stream such as a graphics display. Modify COUNTING-SORT to produce the output in sorted order without using any substantial additional storage besides that in A and C. (Hint: Link elements of A that have the same key into lists. Where is a "free" place to keep the pointers for the linked list?)

9.2-5

Describe an algorithm that, given n integers in the range 1 to k, preprocesses its input and then answers any query about how many of the n integers fall into a range [a . . b] in O(1) time. Your algorithm should use O(n + k) preprocessing time.

9.3 Radix sort

Radix sort is the algorithm used by the card-sorting machines you now find only in computer museums. The cards are organized into 80 columns, and in each column a hole can be punched in one of 12 places. The sorter can be mechanically "programmed" to examine a given column of each card in a deck and distribute the card into one of 12 bins depending on which place has been punched. An operator can then gather the cards bin by bin, so that cards with the first place punched are on top of cards with the second place punched, and so on.

For decimal digits, only 10 places are used in each column. (The other two places are used for encoding nonnumeric characters.) A d-digit number would then occupy a field of d columns. Since the card sorter can look at only one column at a time, the problem of sorting n cards on a d-digit number requires a sorting algorithm.

Intuitively, one might want to sort numbers on their most significant digit, sort each of the resulting bins recursively, and then combine the decks in order. Unfortunately, since the cards in 9 of the 10 bins must be put aside to sort each of the bins, this procedure generates many intermediate piles of cards that must be kept track of. (See Exercise 9.3-5.)

Radix sort solves the problem of card sorting counterintuitively by sorting on the least significant digit first. The cards are then combined into a single deck, with the cards in the 0 bin preceding the cards in the 1 bin preceding the cards in the 2 bin, and so on. Then the entire deck is sorted again on the second least-significant digit and recombined in a like manner. The process continues until the cards have been sorted on all d digits. Remarkably, at that point the cards are fully sorted on the d-digit number. Thus, only d passes through the deck are required to sort. Figure 9.3 shows how radix sort operates on a "deck" of seven 3-digit numbers.

It is essential that the digit sorts in this algorithm be stable. The sort performed by a card sorter is stable, but the operator has to be wary about not changing the order of the cards as they come out of a bin, even though all the cards in a bin have the same digit in the chosen column.

In a typical computer, which is a sequential random-access machine, radix sort is sometimes used to sort records of information that are keyed by multiple fields. For example, we might wish to sort dates by three keys: year, month, and day. We could run a sorting algorithm with a comparison function that, given two dates, compares years, and if there is a tie, compares months, and if another tie occurs, compares days. Alternatively, we could sort the information three times with a stable sort: first on day, next on month, and finally on year.

329      720     720      329

457      355     329      355

657      436     436      436

839      457      839      457

436      657     355      657

720      329     457      720

355      839     657      839

Figure 9.3 The operation of radix sort on a list of seven 3-digit numbers. The first column is the input. The remaining columns show the list after successive sorts on increasingly significant digit positions. The vertical arrows indicate the digit position sorted on to produce each list from the previous one.

The code for radix sort is straightforward. The following procedure assumes that each element in the n-element array A has d digits, where digit 1 is the lowest-order digit and digit d is the highest-order digit.

RADIX-SORT(A, d)

1  for i    1 to d

2       do use a stable sort to sort array A on digit i

The correctness of radix sort follows by induction on the column being sorted (see Exercise 9.3-3). The analysis of the running time depends on the stable sort used as the intermediate sorting algorithm. When each digit is in the range 1 to k, and k is not too large, counting sort is the obvious choice. Each pass over n d-digit numbers then takes time [image] (n + k). There are d passes, so the total time for radix sort is [image] (dn + kd). When d is constant and k = O(n), radix sort runs in linear time.

Some computer scientists like to think of the number of bits in a computer word as being [image] (lg n). For concreteness, let's say that d lg n is the number of bits, where d is a positive constant. Then, if each number to be sorted fits in one computer word, we can treat it as a d-digit number in radix-n notation. As a concrete example, consider sorting 1 million 64-bit numbers. By treating these numbers as four-digit, radix-2l6 numbers, we can sort them in just four passes using radix sort. This compares favorably with a typical [image] (n 1g n) comparison sort, which requires approximately 1g n = 20 operations per number to be sorted. Unfortunately, the version of radix sort that uses counting sort as the intermediate stable sort does not sort in place, which many of the [image] (n lg n) comparison sorts do. Thus, when primary memory storage is at a premium, an algorithm such as quicksort may be preferable.

Exercises

9.3-1

Using Figure 9.3 as a model, illustrate the operation of RADIX-SORT on the following list of English words: COW, DOG, SEA, RUG, ROW, MOB, BOX, TAB, BAR, EAR, TAR, DIG, BIG, TEA, NOW, FOX.

9.3-2

Which of the following sorting algorithms are stable: insertion sort, merge sort, heapsort, and quicksort? Give a simple scheme that makes any sorting algorithm stable. How much additional time and space does your scheme entail?

9.3-3

Use induction to prove that radix sort works. Where does your proof need the assumption that the intermediate sort is stable?

9.3-4

Show how to sort n integers in the range 1 to n2 in O(n) time.

9.3-5

In the first card-sorting algorithm in this section, exactly how many sorting passes are needed to sort d-digit decimal numbers in the worst case? How many piles of cards would an operator need to keep track of in the worst case?

9.4 Bucket sort

Bucket sort runs in linear time on the average. Like counting sort, bucket sort is fast because it assumes something about the input. Whereas counting sort assumes that the input consists of integers in a small range, bucket sort assumes that the input is generated by a random process that distributes elements uniformly over the interval [0,1). (See Section 6.2 for a definition of uniform distribution.)

The idea of bucket sort is to divide the interval [0, 1) into n equal-sized subintervals, or buckets, and then distribute the n input numbers into the buckets. Since the inputs are uniformly distributed over [0, 1), we don't expect many numbers to fall into each bucket. To produce the output, we simply sort the numbers in each bucket and then go through the buckets in order, listing the elements in each.

Our code for bucket sort assumes that the input is an n-element array A and that each element A[i] in the array satisfies 0 [image] A[i] < 1. The code requires an auxiliary array B[0..n - 1] of linked lists (buckets) and assumes that there is a mechanism for maintaining such lists. (Section 11.2 describes how to implement basic operations on linked lists.)

Figure 9.4 The operation of BUCKET-SORT. (a) The input array A[1 . .10]. (b) The array B[0 . . 9] of sorted lists (buckets) after line 5 of the algorithm. Bucket i holds values in the interval [i/10,(i + 1)/10). The sorted output consists of a concatenation in order of the lists B[0], B[1], . . . , B[9].

BUCKET-SORT(A)

1  n    length [A]

2  for i    1 to n

3       do insert A[ i] into list B [  nA[i]  ]

4  for i    0 to n -1

5       do sort list B[i] with insertion sort

6  concatenate the lists B[0], B[1], . . . , B[n - 1] together in order

Figure 9.4 shows the operation of bucket sort on an input array of 10 numbers.

To see that this algorithm works, consider two elements A[i] and A[j]. If these elements fall in the same bucket, they appear in the proper relative order in the output sequence because their bucket is sorted by insertion sort. Suppose they fall into different buckets, however. Let these buckets be B[i'] and B[j'], respectively, and assume without loss of generality that i' < j'. When the lists of B are concatenated in line 6, elements of bucket B[i'] come before elements of B[j'], and thus A[i] precedes A[j] in the output sequence. Hence, we must show that A[i] [image] A[j]. Assuming the contrary, we have

i'  =    nA[i]

      nA[j]

=  j',

which is a contradiction, since i' < j'. Thus, bucket sort works.

To analyze the running time, observe that all lines except line 5 take O(n) time in the worst case. The total time to examine all buckets in line 5 is O(n), and so the only interesting part of the analysis is the time taken by the insertion sorts in line 5.

To analyze the cost of the insertion sorts, let ni be the random variable denoting the number of elements placed in bucket B[i]. Since insertion sort runs in quadratic time (see Section 1.2), the expected time to sort the elements in bucket [image]. The total expected time to sort all the elements in all the buckets is therefore

(9.1)

In order to evaluate this summation, we must determine the distribution of each random variable ni. We have n elements and n buckets. The probability that a given element falls into bucket B[i] is 1/n, since each bucket is responsible for 1/n of the interval [0,1). Thus, the situation is analogous to the ball-tossing example of Section 6.6.2: we have n balls (elements) and n bins (buckets), and each ball is thrown independently with probability p = 1 /n of falling into any particular bucket. Thus, the probability that ni = k follows the binomial distribution b(k; n, p), which has mean E[ni] = np = 1 and variance Var[ni] = np(1 - p) = 1- 1/n. For any random variable X, equation (6.30) gives

Using this bound in equation (9.1), we conclude that the expected time for insertion sorting is O(n). Thus, the entire bucket sort algorithm runs in linear expected time.

Exercises

9.4-1

Using Figure 9.4 as a model, illustrate the operation of BUCKET-SORT on the array A = [image].79, .13, .16, .64, .39, .20, .89, .53, .71, .42 [image].

9.4-2

What is the worst-case running time for the bucket-sort algorithm? What simple change to the algorithm preserves its linear expected running time and makes its worst-case running time O(n lg n)?

9.4-3

We are given n points in the unit circle, pi = (xi, yi), such that [image]. Suppose that the points are uniformly distributed; that is, the probability of finding a point in any region of the circle is proportional to the area of that region. Design a [image] (n) expected-time algorithm to sort the n points by their distances [image] from the origin. (Hint: Design the bucket sizes in BUCKET-SORT to reflect the uniform distribution of the points in the unit circle.)

9.4-4

A probability distribution function P(x) for a random variable X is defined by P(x) = Pr{X [image] x}. Suppose a list of n numbers has a continuous probability distribution function P that is computable in O(1) time. Show how to sort the numbers in linear expected time.

Problems

9-1 Average-case lower bounds on comparison sorting

In this problem, we prove an [image] (n lg n) lower bound on the expected running time of any deterministic or randomized comparison sort on n inputs. We begin by examining a deterministic comparison sort A with decision tree TA. We assume that every permutation of A's inputs is equally likely.

a. Suppose that each leaf of TA is labeled with the probability that it is reached given a random input. Prove that exactly n! leaves are labeled 1/n! and that the rest are labeled 0.

b. Let D(T) denote the external path length of a tree T; that is, D(T) is the sum of the depths of all the leaves of T. Let T be a tree with k > 1 leaves, and let RT and LT be the right and left subtrées of T. Show that D(T) = D(RT) + D(LT) + k.

c. Let d(m) be the minimum value of D(T) over all trees T with m leaves. Show that d(k) = min1 [image] i [image] k {d (i)+d(k - i)+k}. (Hint: Consider a tree T with k leaves that achieves the minimum. Let i be the number of leaves in RT and k - i the number of leaves in LT.)

d. Prove that for a given value of k, the function i lg i + (k - i) lg(k - i) is minimized at i = k/2. Conclude that d(k) = [image] (k lg k).

e. Prove that D(TA) = [image] (n! lg(n!)) for TA, and conclude that the expected time to sort n elements is [image] (n lg n).

Now, consider a randomized comparison sort B. We can extend the decision-tree model to handle randomization by incorporating two kinds of nodes: ordinary comparison nodes and "randomization" nodes. A randomization node models a random choice of the form RANDOM( 1, r) made by algorithm B; the node has r children, each of which is equally likely to be chosen during an execution of the algorithm.

f. Show that for any randomized comparison sort B, there exists a deterministic comparison sort A that makes no more comparisons on the average than B does.

9-2 Sorting in place in linear time

a. Suppose that we have an array of n data records to sort and that the key of each record has the value 0 or 1. Give a simple, linear-time algorithm for sorting the n data records in place. Use no storage of more than constant size in addition to the storage provided by the array.

b. Can your sort from part (a) be used to radix sort n records with b-bit keys in O(bn) time? Explain how or why not.

c. Suppose that the n records have keys in the range from 1 to k. Show how to modify counting sort so that the records can be sorted in place in O(n + k) time. You may use O(k) storage outside the input array. (Hint: How would you do it for k = 3?)

Chapter notes

The decision-tree model for studying comparison sorts was introduced by Ford and Johnson [72]. Knuth's comprehensive treatise on sorting [123] covers many variations on the sorting problem, including the information- theoretic lower bound on the complexity of sorting given here. Lower bounds for sorting using generalizations of the decision-tree model were studied comprehensively by Ben-Or [23].

Knuth credits H. H. Seward with inventing counting sort in 1954, and also with the idea of combining counting sort with radix sort. Radix sorting by the least-significant digit first appears to be a folk algorithm widely used by operators of mechanical card-sorting machines. According to Knuth, the first published reference to the method is a 1929 document by L. J. Comrie describing punched-card equipment. Bucket sorting has been in use since 1956, when the basic idea was proposed by E. J. Isaac and R. C. Singleton.

Go to Chapter 10 Back to Table of Contents


Intro to Algorithms: CHAPTER 10: MEDIANS AND ORDER STATISTICS

 Return to Table of Contents

CHAPTER 10: MEDIANS AND ORDER STATISTICS

The ith order statistic of a set of n elements is the ith smallest element. For example, the minimum of a set of elements is the first order statistic (i = 1), and the maximum is the nth order statistic (i = n). A median, informally, is the "halfway point" of the set. When n is odd, the median is unique, occurring at i = (n + 1)/2. When n is even, there are two medians, occurring at i = n/2 and i = n/2 + 1. Thus, regardless of the parity of n, medians occur at i = [image] (n + 1)/2 [image] and i = [image] (n + 1)/2 [image] .

This chapter addresses the problem of selecting the ith order statistic from a set of n distinct numbers. We assume for convenience that the set contains distinct numbers, although virtually everything that we do extends to the situation in which a set contains repeated values. The selection problem can be specified formally as follows:

Input: A set A of n (distinct) numbers and a number i, with 1 [image] i [image] n.

Output: The element x [image] A that is larger than exactly i -1 other elements of A.

The selection problem can be solved in O(n lg n) time, since we can sort the numbers using heapsort or merge sort and then simply index the ith element in the output array. There are faster algorithms, however.

In Section 10.1, we examine the problem of selecting the minimum and maximum of a set of elements. More interesting is the general selection problem, which is investigated in the subsequent two sections. Section 10.2 analyzes a practical algorithm that achieves an O(n) bound on the running time in the average case. Section 10.3 contains an algorithm of more theoretical interest that achieves the O(n) running time in the worst case.

10.1 Minimum and maximum

How many comparisons are necessary to determine the minimum of a set of n elements? We can easily obtain an upper bound of n - 1 comparisons: examine each element of the set in turn and keep track of the smallest element seen so far. In the following procedure, we assume that the set resides in array A, where length[A] = n.

MINIMUM (A)

1  min    A[1]

2  for i    2 to length[A]

3        do if min > A[i]

4              then min    A[i]

5  return min

Finding the maximum can, of course, be accomplished with n - 1 comparisons as well.

Is this the best we can do? Yes, since we can obtain a lower bound of n - 1 comparisons for the problem of determining the minimum. Think of any algorithm that determines the minimum as a tournament among the elements. Each comparison is a match in the tournament in which the smaller of the two elements wins. The key observation is that every element except the winner must lose at least one match. Hence, n - 1 comparisons are necessary to determine the minimum, and the algorithm MINIMUM is optimal with respect to the number of comparisons performed.

An interesting fine point of the analysis is the determination of the expected number of times that line 4 is executed. Problem 6-2 asks you to show that this expectation is [image] (lg n).

Simultaneous minimum and maximum

In some applications, we must find both the minimum and the maximum of a set of n elements. For example, a graphics program may need to scale a set of (x, y) data to fit onto a rectangular display screen or other graphical output device. To do so, the program must first determine the minimum and maximum of each coordinate.

It is not too difficult to devise an algorithm that can find both the minimum and the maximum of n elements using the asymptotically optimal [image] (n) number of comparisons. Simply find the minimum and maximum independently, using n - 1 comparisons for each, for a total of 2n - 2 comparisons.

In fact, only 3 [image] n/2 [image] comparisons are necessary to find both the minimum and the maximum. To do this, we maintain the minimum and maximum elements seen thus far. Rather than processing each element of the input by comparing it against the current minimum and maximum, however, at a cost of two comparisons per element, we process elements in pairs. We compare pairs of elements from the input first with each other, and then compare the smaller to the current minimum and the larger to the current maximum, at a cost of three comparisons for every two elements.

Exercises

10.1-1

Show that the second smallest of n elements can be found with n+ [image] lg n [image] - 2 comparisons in the worst case. (Hint: Also find the smallest element.)

10.1-2

Show that [image] 3n/2 [image] - 2 comparisons are necessary in the worst case to find both the maximum and minimum of n numbers. (Hint: Consider how many numbers are potentially either the maximum or minimum, and investigate how a comparison affects these counts.)

10.2 Selection in expected linear time

The general selection problem appears more difficult than the simple problem of finding a minimum, yet, surprisingly, the asymptotic running time for both problems is the same: (n). In this section, we present a divide-and-conquer algorithm for the selection problem. The algorithm RANDOMIZED-SELECT is modeled after the quicksort algorithm of Chapter 8. As in quicksort, the idea is to partition the input array recursively. But unlike quicksort, which recursively processes both sides of the partition, RANDOMIZED-SELECT only works on one side of the partition. This difference shows up in the analysis: whereas quicksort has an expected running time of [image] (n lg n), the expected time of RANDOMIZED-SELECT is [image] (n).

RANDOMIZED-SELECT uses the procedure RANDOMIZED-PARTITION introduced in Section 8.3. Thus, like RANDOMIZED-QUICKSORT, it is a randomized algorithm, since its behavior is determined in part by the output of a random-number generator. The following code for RANDOMIZED-SELECT returns the ith smallest element of the array A[p . . r].

RANDOMIZED-SELECT(A, p, r, i)

1  if p = r

2      then return A[p]

3  q    RANDOMIZED-PARTITION(A, p, r)

4  k    q - p + 1

5  if i    k

6      then return RANDOMIZED-SELECT(A, p, q, i)

7      else return RANDOMIZED-SELECT(A, q + 1, r, i - k)

After RANDOMIZED-PARTITION is executed in line 3 of the algorithm, the array A[p .. r] is partitioned into two nonempty subarrays A[p .. q] and A[q + 1 .. r] such that each element of A[p .. q] is less than each element of A[q + 1 .. r]. Line 4 of the algorithm computes the number k of elements in the subarray A[p .. q]. The algorithm now determines in which of the two subarrays A[p .. q] and A[q + 1 .. r] the ith smallest element lies. If i [image] k, then the desired element lies on the low side of the partition, and it is recursively selected from the subarray in line 6. If i > k, however, then the desired element lies on the high side of the partition. Since we already know k values that are smaller than the ith smallest element of A[p .. r]--namely, the elements of A[p .. q]--the desired element is the (i - k)th smallest element of A[q + 1 .. r], which is found recursively in line 7.

The worst-case running time for RANDOMIZED-SELECT is [image] (n2), even to find the minimum, because we could be extremely unlucky and always partition around the largest remaining element. The algorithm works well in the average case, though, and because it is randomized, no particular input elicits the worst-case behavior.

We can obtain an upper bound T(n) on the expected time required by RANDOMIZED-SELECT on an input array of n elements as follows. We observed in Section 8.4 that the algorithm RANDOMIZED-PARTITION produces a partition whose low side has 1 element with probability 2/n and i elements with probability 1/n for i = 2, 3, . . . , n - 1. Assuming that T(n) is monotonically increasing, in the worst case RANDOMIZED-SELECT is always unlucky in that the ith element is determined to be on the larger side of the partition. Thus, we get the recurrence

The second line follows from the first since max(1, n - 1 ) = n - 1 and

If n is odd, each term T( [image] n/2 [image] ), T( [image] n/2 [image] + 1), . . . ,T(n - 1) appears twice in the summation, and if n is even, each term T( [image] n/2 [image] + 1), T( [image] n/2 [image] + 2), . . . , T(n - 1) appears twice and the term T( [image] n/2 [image] ) appears once. In either case, the summation of the first line is bounded from above by the summation of the second line. The third line follows from the second since in the worst case T(n - 1) = O(n2), and thus the term [image] can be absorbed by the term O(n).

We solve the recurrence by substitution. Assume that T(n) [image] cn for some constant c that satisfies the initial conditions of the recurrence. Using this inductive hypothesis, we have

since we can pick c large enough so that c(n/4 + 1/2) dominates the O(n) term.

Thus, any order statistic, and in particular the median, can be determined on average in linear time.

Exercises

10.2-1

Write an iterative version of RANDOMIZED-SELECT.

10.2-2

Suppose we use RANDOMIZED-SELECT to select the minimum element of the array A = [image] 3, 2, 9, 0, 7, 5, 4, 8, 6, 1 [image]. Describe a sequence of partitions that results in a worst-case performance of RANDOMIZED-SELECT.

10.2-3

Recall that in the presence of equal elements, the RANDOMIZED-PARTITION procedure partitions the subarray A[p . . r] into two nonempty subarrays A[p . . q] and A[q + 1 . . r] such that each element in A[p . . q] is less than or equal to every element in A[q + 1 . . r]. If equal elements are present, does the RANDOMIZED-SELECT procedure work correctly?

10.3 Selection in worst-case linear time

We now examine a selection algorithm whose running time is O(n) in the worst case. Like RANDOMIZED-SELECT, the algorithm SELECT finds the desired element by recursively partitioning the input array. The idea behind the algorithm, however, is to guarantee a good split when the array is partitioned. SELECT uses the deterministic partitioning algorithm PARTITION from quicksort (see Section 8.1), modified to take the element to partition around as an input parameter.

Figure 10.1 Analysis of the algorithm SELECT. The n elements are represented by small circles, and each group occupies a column. The medians of the groups are whitened, and the median-of-medians x is labeled. Arrows are drawn from larger elements to smaller, from which it can be seen that 3 out of every group of 5 elements to the right of x are greater than x, and 3 out of every group of 5 elements to the left of x are less than x. The elements greater than x are shown on a shaded background.

The SELECT algorithm determines the ith smallest of an input array of n elements by executing the following steps.

1. Divide the n elements of the input array into [image] n/5 [image] groups of 5 elements each and at most one group made up of the remaining n mod 5 elements.

2. Find the median of each of the [image] n/5 [image] groups by insertion sorting the elements of each group (of which there are 5 at most) and taking its middle element. (If the group has an even number of elements, take the larger of the two medians.)

3. Use SELECT recursively to find the median x of the [image] n/5 [image] medians found in step 2.

4. Partition the input array around the median-of-medians x using a modified version of PARTITION. Let k be the number of elements on the low side of the partition, so that n - k is the number of elements on the high side.

5. Use SELECT recursively to find the ith smallest element on the low side if i [image] k, or the (i - k)th smallest element on the high side if i > k.

To analyze the running time of SELECT, we first determine a lower bound on the number of elements that are greater than the partitioning element x. Figure 10.1 is helpful in visualizing this bookkeeping. At least half of the medians found in step 2 are greater than or equal to the median-of-medians x. Thus, at least half of the [image] n/5 [image] groups contribute 3 elements that are greater than x, except for the one group that has fewer than 5 elements if 5 does not divide n exactly, and the one group containing x itself. Discounting these two groups, it follows that the number of elements greater than x is at least

Similarly, the number of elements that are less than x is at least 3n/10 - 6. Thus, in the worst case, SELECT is called recursively on at most 7n/10 + 6 elements in step 5.

We can now develop a recurrence for the worst-case running time T(n) of the algorithm SELECT. Steps 1, 2, and 4 take O(n) time. (Step 2 consists of O(n) calls of insertion sort on sets of size O(1).) Step 3 takes time T( [image] n/5 [image] ), and step 5 takes time at most T(7n / 10 + 6), assuming that T is monotonically increasing. Note that 7n / 10 + 6 < n for n > 20 and that any input of 80 or fewer elements requires O(1) time. We can therefore obtain the recurrence

We show that the running time is linear by substitution. Assume that T(n) [image] cn for some constant c and all n [image] 80. Substituting this inductive hypothesis into the right-hand side of the recurrence yields

T(n)    c   n/5   + c(7n/10 + 6) + O(n)

   cn/5 + c + 7cn/10 + 6c + O(n)

   9cn/10 + 7c + O(n)

   cn,

since we can pick c large enough so that c(n / 10 - 7) is larger than the function described by the O(n) term for all n > 80. The worst-case running time of SELECT is therefore linear.

As in a comparison sort (see Section 9.1), SELECT and RANDOMIZED-SELECT determine information about the relative order of elements only by comparing elements. Thus, the linear-time behavior is not a result of assumptions about the input, as was the case for the sorting algorithms in Chapter 9. Sorting requires [image] (n 1g n) time in the comparison model, even on average (see Problem 9-1), and thus the method of sorting and indexing presented in the introduction to this chapter is asymptotically inefficient.

Exercises

10.3-1

In the algorithm SELECT, the input elements are divided into groups of 5. Will the algorithm work in linear time if they are divided into groups of 7? How about groups of 3?

10.3-2

Analyze SELECT to show that the number of elements greater than the median-of-medians x and the number of elements less than x is at least [image] n/4 [image] if n [image] 38.

10.3-3

Show how quicksort can be made to run in O(n 1g n) time in the worst case.

10.3-4

Suppose that an algorithm uses only comparisons to find the ith smallest element in a set of n elements. Show that it can also find the i - 1 smaller elements and the n - i larger elements without performing any additional comparisons.

10.3-5

Given a "black-box" worst-case linear-time median subroutine, give a simple, linear-time algorithm that solves the selection problem for an arbitrary order statistic.

10.3-6

The kth quantiles of an n-element set are the k - 1 order statistics that divide the sorted set into k equal-sized sets (to within 1). Give an O(n 1g k)-time algorithm to list the kth quantiles of a set.

10.3-7

Describe an O(n)-time algorithm that, given a set S of n distinct numbers and a positive integer k [image] n, determines the k numbers in S that are closest to the median of S.

10.3-8

Let X[1 . . n] and Y[1 . . n] be two arrays, each containing n numbers already in sorted order. Give an O(1g n)-time algorithm to find the median of all 2n elements in arrays X and Y.

10.3-9

Professor Olay is consulting for an oil company, which is planning a large pipeline running east to west through an oil field of n wells. From each well, a spur pipeline is to be connected directly to the main pipeline along a shortest path (either north or south), as shown in Figure 10.2. Given x- and y-coordinates of the wells, how should the professor pick the optimal location of the main pipeline (the one that minimizes the total length of the spurs)? Show that the optimal location can be determined in linear time.

Figure 10.2 We want to determine the position of the east-west oil pipeline that minimizes the total length of the north-south spurs.

Problems

10-1 Largest i numbers in sorted order

Given a set of n numbers, we wish to find the i largest in sorted order using a comparison-based algorithm. Find the algorithm that implements each of the following methods with the best asymptotic worst-case running time, and analyze the running times of the methods in terms of n and i.

a. Sort the numbers and list the i largest.

b. Build a priority queue from the numbers and call EXTRACT-MAX i times.

c. Use an order-statistic algorithm to find the ith largest number, partition, and sort the i largest numbers.

10-2 Weighted median

For n distinct elements x1, x2, . . . , xn with positive weights w1, w2, . . . , wn such that [image] the weighted median is the element xk satisfying

a. Argue that the median of x1, x2, . . . , xn is the weighted median of the xi with weights wi = 1 / n for i = 1, 2, . . . , n.

b. Show how to compute the weighted median of n elements in O(n 1g n) worst-case time using sorting.

c. Show how to compute the weighted median in [image] (n) worst-case time using a linear-time median algorithm such as SELECT from Section 10.3.

The post-office location problem is defined as follows. We are given n points p1, p2, . . . , pn with associated weights w1, w2, . . . , wn. We wish to find a point p (not necessarily one of the input points) that minimizes the sum [image], where d(a, b) is the distance between points a and b.

d. Argue that the weighted median is a best solution for the 1-dimensional post-office location problem, in which points are simply real numbers and the distance between points a and b is d(a, b) = |a - b|.

e. Find the best solution for the 2-dimensional post-office location problem, in which the points are (x, y) coordinate pairs and the distance between points a = (x1, y1) and b = (x2, y2) is the Manhattan distance: d(a, b) = |x1 - x2| + |y1 - y2|.

10-3 Small order statistics

The worst-case number T(n) of comparisons used by SELECT to select the ith order statistic from n numbers was shown to satisfy T(n) = [image] (n), but the constant hidden by the [image] -notation is rather large. When i is small relative to n, we can implement a different procedure that uses SELECT as a subroutine but makes fewer comparisons in the worst case.

a. Describe an algorithm that uses Ui(n) comparisons to find the ith smallest of n elements, where i [image] n / 2 and

(Hint: Begin with [image] n / 2 [image] disjoint pairwise comparisons, and recurse on the set containing the smaller element from each pair.)

b. Show that Ui(n) = n + O(T(2i)1g(n / i)).

c. Show that if i is a constant, then Ui(n) = n + O(1g n).

d. Show that if i = n / k for k [image] 2, then Ui(n) = n + O(T(2n / k) 1g k).

Chapter notes

The worst-case median-finding algorithm was invented by Blum, Floyd, Pratt, Rivest, and Tarjan [29]. The fast average-time version is due to Hoare [97]. Floyd and Rivest [70] have developed an improved average-time version that partitions around an element recursively selected from a small sample of the elements.

Go to Part III Back to Table of Contents


Intro to Algorithms: CHAPTER 11: ELEMENTARY DATA STRUCTURES

 Return to Table of Contents

CHAPTER 11: ELEMENTARY DATA STRUCTURES

In this chapter, we examine the representation of dynamic sets by simple data structures that use pointers. Although many complex data structures can be fashioned using pointers, we present only the rudimentary ones: stacks, queues, linked lists, and rooted trees. We also discuss a method by which objects and pointers can be synthesized from arrays.

11.1 Stacks and queues

Stacks and queues are dynamic sets in which the element removed from the set by the DELETE operation is prespecified. In a stack, the element deleted from the set is the one most recently inserted: the stack implements a last-in, first-out, or LIFO, policy. Similarly, in a queue , the element deleted is always the one that has been in the set for the longest time: the queue implements a first-in, first-out, or FIFO, policy. There are several efficient ways to implement stacks and queues on a computer. In this section we show how to use a simple array to implement each.

Stacks

The INSERT operation on a stack is often called PUSH, and the DELETE operation, which does not take an element argument, is often called POP. These names are allusions to physical stacks, such as the spring-loaded stacks of plates used in cafeterias. The order in which plates are popped from the stack is the reverse of the order in which they were pushed onto the stack, since only the top plate is accessible.

As shown in Figure 11.1, we can implement a stack of at most n elements with an array S [1..n]. The array has an attribute top[S] that indexes the most recently inserted element. The stack consists of elements S[1..top[S]], where S[1] is the element at the bottom of the stack and S[top[S]] is the element at the top.

When top [S] = 0, the stack contains no elements and is empty. The stack can be tested for emptiness by the query operation STACK-EMPTY. If an empty stack is popped, we say the stack underflows, which is normally an error. If top[S] exceeds n, the stack overflows. (In our pseudocode implementation, we don't worry about stack overflow.)

Figure 11.1 An array implementation of a stack S. Stack elements appear only in the lightly shaded positions. (a) Stack S has 4 elements. The top element is 9. (b) Stack S after the calls PUSH(S, 17) and PUSH(S, 3). (c) Stack S after the call POP(S) has returned the element 3, which is the one most recently pushed. Although element 3 still appears in the array, it is no longer in the stack; the top is element 17.

The stack operations can each be implemented with a few lines of code.

STACK-EMPTY(S)

1  if top [S] = 0

2      then return TRUE

3      else return FALSE

PUSH(S, x)

1  top[S]    top [S] + 1

2  S [top[S]]    x

POP(S)

1  if STACK-EMPTY(S)

2     then error "underflow"

3     else top [S]    top [S] - 1

4          return S [top [S] + 1]

Figure 11.1 shows the effects of the modifying operations PUSH and POP. Each of the three stack operations takes O (1) time.

Queues

We call the INSERT operation on a queue ENQUEUE, and we call the DELETE operation DEQUEUE; like the stack operation POP, DEQUEUE takes no element argument. The FIFO property of a queue causes it to operate like a line of people in the registrar's office. The queue has a head and a tail. When an element is enqueued, it takes its place at the tail of the queue, 1ike a newly arriving student takes a place at the end of the line. The element dequeued is always the one at the head of the queue, like the student at the head of the line who has waited the longest. (Fortunately, we don't have to worry about computational elements cutting into line.)

Figure 11.2 A queue implemented using an array Q[1 . . 12]. Queue elements appear only in the lightly shaded positions. (a) The queue has 5 elements, in locations Q [7..11]. (b) The configuration of the queue after the calls ENQUEUE(Q, 17), ENQUEUE(Q, 3), and ENQUEUE(Q, 5). (c) The configuration of the queue after the call DEQUEUE(Q) returns the key value 15 formerly at the head of the queue. The new head has key 6.

Figure 11.2 shows one way to implement a queue of at most n - 1 elements using an array Q [1..n]. The queue has an attribute head [Q] that indexes, or points to, its head. The attribute tail [Q] indexes the next location at which a newly arriving element will be inserted into the queue. The elements in the queue are in locations head[Q], head [Q] + 1, . . . , tail [Q] - 1, where we "wrap around" in the sense that location 1 immediately follows location n in a circular order. When head [Q] = tail [Q], the queue is empty. Initially, we have head [Q] = tail [Q] = 1. When the queue is empty, an attempt to dequeue an element causes the queue to underflow. When head [Q] = tail [Q] + 1, the queue is full, and an attempt to enqueue an element causes the queue to overflow.

In our procedures ENQUEUE and DEQUEUE, the error checking for underflow and overflow has been omitted. (Exercise 11.1-4 asks you to supply code that checks for these two error conditions.)

ENQUEUE(Q, x)

1  Q [tail [Q]]    x

2  if tail [Q] = length [Q]

3      then tail [Q]    1

4      else tail [Q]    tail [Q] + 1

DEQUEUE(Q)

1  x    Q [head [Q]]

2  if head [Q] = length [Q]

3      then head [Q]    1

4      else head [Q]    head [Q] + 1

5  return x

Figure 11.2 shows the effects of the ENQUEUE and DEQUEUE operations. Each operation takes O(1) time.

Exercises

11.1-1

Using Figure 11.1 as a model, illustrate the result of each of the operations PUSH(S, 4), PUSH(S, 1), PUSH(S, 3), POP(S), PUSH(S, 8), and POP(S) on an initially empty stack S stored in array S [1 . . 6].

11.1-2

Explain how to implement two stacks in one array A [1 . . n] in such a way that neither stack overflows unless the total number of elements in both stacks together is n. The PUSH and POP operations should run in O(1) time.

11.1-3

Using Figure 11.2 as a model, illustrate the result of each of the operations ENQUEUE(Q, 4), ENQUEUE(Q, 1), ENQUEUE(Q, 3), DEQUEUE(Q), ENQUEUE(Q, 8), and DEQUEUE(Q) on an initially empty queue Q stored in array Q [1 . . 6].

11.1-4

Rewrite ENQUEUE and DEQUEUE to detect underflow and overflow of a queue.

11.1-5

Whereas a stack allows insertion and deletion of elements at only one end, and a queue allows insertion at one end and deletion at the other end, a deque (double-ended queue) allows insertion and deletion at both ends. Write four O(1)-time procedures to insert elements into and delete elements from both ends of a deque constructed from an array.

11.1-6

Show how to implement a queue using two stacks. Analyze the running time of the queue operations.

11.1-7

Show how to implement a stack using two queues. Analyze the running time of the stack operations.

11.2 Linked lists

A linked list is a data structure in which the objects are arranged in a linear order. Unlike an array, though, in which the linear order is determined by the array indices, the order in a linked list is determined by a pointer in each object. Linked lists provide a simple, flexible representation for dynamic sets, supporting (though not necessarily efficiently) all the operations listed on page 198.

As shown in Figure 11.3, each element of a doubly linked list L is an object with a key field and two other pointer fields: next and prev. The object may also contain other satellite data. Given an element x in the list, next [x] points to its successor in the linked list, and prev [x] points to its predecessor. If prev [x] = NIL, the element x has no predecessor and is therefore the first element, or head, of the list. If next [x] = NIL, the element x has no successor and is therefore the last element, or tail, of the list. An attribute head [L] points to the first element of the list. If head [L] = NIL, the list is empty.

A list may have one of several forms. It may be either singly linked or doubly linked, it may be sorted or not, and it may be circular or not. If a list is singly linked, we omit the prev pointer in each element. If a list is sorted, the linear order of the list corresponds to the linear order of keys stored in elements of the list; the minimum element is the head of the list, and the maximum element is the tail. If the list is unsorted, the elements can appear in any order. In a circular list, the prev pointer of the head of the list points to the tail, and the next pointer of the tail of the list points to the head. The list may thus be viewed as a ring of elements. In the remainder of this section, we assume that the lists with which we are working are unsorted and doubly linked.

Searching a linked list

The procedure LIST-SEARCH(L, k) finds the first element with key k in list L by a simple linear search, returning a pointer to this element. If no object with key k appears in the list, then NIL is returned. For the linked list in Figure 11.3(a), the call LIST-SEARCH(L, 4) returns a pointer to the third element, and the call LIST-SEARCH(L, 7) returns NIL.

Figure 11.3 (a) A doubly linked list L representing the dynamic set {1, 4, 9, 16}. Each element in the list is an object with fields for the key and pointers (shown by arrows) to the next and previous objects. The next field of the tail and the prev field of the head are NIL, indicated by a diagonal slash. The attribute head[L] points to the head. (b) Following the execution of LIST-INSERT(L, x), where key[x] = 25, the linked list has a new object with key 25 as the new head. This new object points to the old head with key 9. (c) The result of the subsequent call LIST-DELETE(L, x), where x points to the object with key 4.

LIST-SEARCH(L, k)

1  x    head[L]

2  while x    NIL and key[x]    k

3      do x    next[x]

4  return x

To search a list of n objects, the LIST-SEARCH procedure takes   [image] (n) time in the worst case, since it may have to search the entire list.

Inserting into a linked list

Given an element x whose key field has already been set, the LIST-INSERT procedure "splices" x onto the front of the linked list, as shown in Figure 11.3(b).

LIST-INSERT(L, x)

1  next[x]    head[L]

2  if head[L]    NIL

3      then prev[head[L]]    x

4  head[L]    x

5  prev[x]    NIL

The running time for LIST-INSERT on a list of n elements is O(1).

Deleting from a linked list

The procedure LIST-DELETE removes an element x from a linked list L. It must be given a pointer to x, and it then "splices" x out of the list by updating pointers. If we wish to delete an element with a given key, we must first call LIST-SEARCH to retrieve a pointer to the element.

LIST-DELETE(L, x)

1  if prev[x]    NIL

2     then next[prev[x]]    next[x]

3     else head[L]    next[x]

4  if next[x]    NIL

5     then prev[next[x]]    prev[x]

Figure 11.3(c) shows how an element is deleted from a linked list. LIST-DELETE runs in O(1) time, but if we wish to delete an element with a given key,   [image] (n) time is required in the worst case because we must first call LIST-SEARCH.

Sentinels

The code for LIST-DELETE would be simpler if we could ignore the boundary conditions at the head and tail of the list.

LIST-DELETE'(L, x)

1  next[prev[x]]    next[x]

2  prev[next[x]]    prev[x]

A sentinel is a dummy object that allows us to simplify boundary conditions. For example, suppose that we provide with list L an object nil[L] that represents NIL but has all the fields of the other list elements. Wherever we have a reference to NIL in list code, we replace it by a reference to the sentinel nil[L]. As shown in Figure 11.4, this turns a regular doubly linked list into a circular list, with the sentinel nil[L] placed between the head and tail; the field next[nil[L]] points to the head of the list, and prev[nil[L]] points to the tail. Similarly, both the next field of the tail and the prev field of the head point to nil[L]. Since next[nil[L]] points to the head, we can eliminate the attribute head[L] altogether, replacing references to it by references to next[nil[L]]. An empty list consists of just the sentinel, since both next[nil[L]] and prev[nil[L]] can be set to nil[L].

The code for LIST-SEARCH remains the same as before, but with the references to NIL and head[L] changed as specified above.

Figure 11.4 A linked list L that uses a sentinel nil[L] (heavily shaded) is the regular doubly linked list turned into a circular list with nil[L] appearing between the head and tail. The attribute head[L] is no longer needed, since we can access the head of the list by next[nil[L]]. (a) An empty list. (b) The linked list from Figure 11.3(a), with key 9 at the head and key 1 at the tail. (c) The list after executing LIST-INSERT(L, x), where, key[x] = 25. The new object becomes the head of the list. (d) The list after deleting the object with key 1. The new tail is the object with key 4.

LIST-SEARCH'(L, k)

1  x    next[nil[L]]

2  while x    nil[L] and key[x]    k

3      do x    next[x]

4  return x

We use the two-line procedure LIST-DELETE' to delete an element from the list. We use the following procedure to insert an element into the list.

LIST-INSERT'(L, x)

1  next[x]    next[nil[L]]

2  prev[next[nil[L]]]    x

3  next[nil[L]]    x

4  prev[x]    nil[L]

Figure 11.4 shows the effects of LIST-INSERT' and LIST-DELETE' on a sample list.

Sentinels rarely reduce the asymptotic time bounds of data structure operations, but they can reduce constant factors. The gain from using sentinels within loops is usually a matter of clarity of code rather than speed; the linked list code, for example, is simplified by the use of sentinels, but we save only O(1) time in the LIST-INSERT' and LIST-DELETE' procedures. In other situations, however, the use of sentinels helps to tighten the code in a loop, thus reducing the coefficient of, say, n or n2 in the running time.

Sentinels should not be used indiscriminately. If there are many small lists, the extra storage used by their sentinels can represent significant wasted memory. In this book, we only use sentinels when they truly simplify the code.

Exercises

11.2-1

Can the dynamic-set operation INSERT be implemented on a singly linked list in O(1) time? How about DELETE?

11.2-2

Implement a stack using a singly linked list L. The operations PUSH and POP should still take O(1) time.

11.2-3

Implement a queue by a singly linked list L. The operations ENQUEUE and DEQUEUE should still take O(1) time.

11.2-4

Implement the dictionary operations INSERT, DELETE, and SEARCH using singly linked, circular lists. What are the running times of your procedures?

11.2-5

The dynamic-set operation UNION takes two disjoint sets Sl and S2 as input, and it returns a set S = S1   [image]  S2 consisting of all the elements of S1 and S2. The sets S1 and S2 are usually destroyed by the operation. Show how to support UNION in O(1) time using a suitable list data structure.

11.2-6

Write a procedure that merges two singly linked, sorted lists into one singly linked, sorted list without using sentinels. Then, write a similar procedure using a sentinel with key   [image]  to mark the end of each list. Compare the simplicity of code for the two procedures.

11.2-7

Give a   [image] (n)-time nonrecursive procedure that reverses a singly linked list of n elements. The procedure should use no more than constant storage beyond that needed for the list itself.

11.2-8

Explain how to implement doubly linked lists using only one pointer value np[x] per item instead of the usual two (next and prev). Assume that all index values can be interpreted as k-bit integers, and define np[x] to be np[x] = next[x] XOR prev[x], the k-bit "exclusive-or" of next[x] and prev[x]. (The value NIL is represented by 0.) Be sure to describe what information is needed to access the head of the list. Show how to implement the SEARCH, INSERT, and DELETE operations on such a list. Also show how to reverse such a list in O(1) time.

11.3 Implementing pointers and objects

How do we implement pointers and objects in languages, such as Fortran, that do not provide them? In this section, we shall see two ways of implementing linked data structures without an explicit pointer data type. We shall synthesize objects and pointers from arrays and array indices.

A multiple-array representation of objects

We can represent a collection of objects that have the same fields by using an array for each field. As an example, Figure 11.5 shows how we can implement the linked list of Figure 11.3(a) with three arrays. The array key holds the values of the keys currently in the dynamic set, and the pointers are stored in the arrays next and prev. For a given array index x, key[x], next[x], and prev[x] represent an object in the linked list. Under this interpretation, a pointer x is simply a common index into the key, next, and prev arrays.

In Figure 11.3(a), the object with key 4 follows the object with key 16 in the linked list. In Figure 11.5, key 4 appears in key[2], and key 16 appears in key[5], so we have next[5] = 2 and prev[2] = 5. Although the constant NIL appears in the next field of the tail and the prev field of the head, we usually use an integer (such as 0 or -1) that cannot possibly represent an actual index into the arrays. A variable L holds the index of the head of the list.

In our pseudocode, we have been using square brackets to denote both the indexing of an array and the selection of a field (attribute) of an object. Either way, the meanings of key[x], next[x], and prev[x] are consistent with implementation practice.

Figure 11.5 The linked list of Figure 11.3(a) represented by the arrays key, next, and prev. Each vertical slice of the arrays represents a single object. Stored pointers correspond to the array indices shown at the top; the arrows show how to interpret them. Lightly shaded object positions contain list elements. The variable L keeps the index of the head.

Figure 11.6 The linked list of Figures 11.3(a) and 11.5 represented in a single array A. Each list element is an object that occupies a contiguous subarray of length 3 within the array. The three fields key, next, and prev correspond to the offsets 0, 1, and 2, respectively. A pointer to an object is an index of the first element of the object. Objects containing list elements are lightly shaded, and arrows show the list ordering.

A single-array representation of objects

The words in a computer memory are typically addressed by integers from 0 to M - 1, where M is a suitably large integer. In many programming languages, an object occupies a contiguous set of locations in the computer memory. A pointer is simply the address of the first memory location of the object, and other memory locations within the object can be indexed by adding an offset to the pointer.

We can use the same strategy for implementing objects in programming environments that do not provide explicit pointer data types. For example, Figure 11.6 shows how a single array A can be used to store the linked list from Figures 11.3(a) and 11.5. An object occupies a contiguous subarray A[j . . k]. Each field of the object corresponds to an offset in the range from 0 to k - j, and a pointer to the object is the index j. In Figure 11.6, the offsets corresponding to key, next, and prev are 0, 1, and 2, respectively. To read the value of prev[i], given a pointer i, we add the value i of the pointer to the offset 2, thus reading A[i + 2].

The single-array representation is flexible in that it permits objects of different lengths to be stored in the same array. The problem of managing such a heterogeneous collection of objects is more difficult than the problem of managing a homogeneous collection, where all objects have the same fields. Since most of the data structures we shall consider are composed of homogeneous elements, it will be sufficient for our purposes to use the multiple-array representation of objects.

Allocating and freeing objects

To insert a key into a dynamic set represented by a doubly linked list, we must allocate a pointer to a currently unused object in the linked-list representation. Thus, it is useful to manage the storage of objects not currently used in the linked-list representation so that one can be allocated. In some systems, a garbage collector is responsible for determining which objects are unused. Many applications, however, are simple enough that they can bear responsibility for returning an unused object to a storage manager. We shall now explore the problem of allocating and freeing (or deallocating) homogeneous objects using the example of a doubly linked list represented by multiple arrays.

Figure 11.7 The effect of the ALLOCATE-OBJECT and FREE-OBJECT procedures. (a) The list of Figure 11.5 (lightly shaded) and a free list (heavily shaded). Arrows show the free-list structure. (b) The result of calling ALLOCATE-OBJECT() (which returns index 4), setting key[4] to 25, and calling LIST-INSERT(L, 4). The new free-list head is object 8, which had been next[4] on the free list. (c) After executing LIST-DELETE(L, 5), we call FREE-OBJECT(5). Object 5 becomes the new free-list head, with object 8 following it on the free list.

Suppose that the arrays in the multiple-array representation have length m and that at some moment the dynamic set contains n   [image]  m elements. Then n objects represent elements currently in the dynamic set, and the remaining m - n objects are free; the free objects can be used to represent elements inserted into the dynamic set in the future.

We keep the free objects in a singly linked list, which we call the free list. The free list uses only the next array, which stores the next pointers within the list. The head of the free list is held in the global variable free. When the dynamic set represented by linked list L is nonempty, the free list may be intertwined with list L, as shown in Figure 11.7. Note that each object in the representation is either in list L or in the free list, but not in both.

The free list is a stack: the next object allocated is the last one freed. We can use a list implementation of the stack operations PUSH and POP to implement the procedures for allocating and freeing objects, respectively.

Figure 11.8 Two linked lists, L1 (lightly shaded) and L2 (heavily shaded), and a free list (darkened) intertwined.

We assume that the global variable free used in the following procedures points to the first element of the free list.

ALLOCATE-OBJECT( )

1  if free = NIL

2      then error "out of space"

3      else x    free

4           free    next[x]

5           return x

FREE-OBJECT(x)

1  next[x]    free

2  free    x

The free list initially contains all n unallocated objects. When the free list has been exhausted, the ALLOCATE-OBJECT procedure signals an error. It is common to use a single free list to service several linked lists. Figure 11.8 shows three linked lists and a free list intertwined through key, next, and prev arrays.

The two procedures run in O(1) time, which makes them quite practical. They can be modified to work for any homogeneous collection of objects by letting any one of the fields in the object act like a next field in the free list.

Exercises

11.3-1

Draw a picture of the sequence   [image] 13, 4, 8, 19, 5, 11  [image]  stored as a doubly linked list using the multiple-array representation. Do the same for the single-array representation.

11.3-2

Write the procedures ALLOCATE-OBJECT and FREE-OBJECT for a homogeneous collection of objects implemented by the single-array representation.

11.3-3

Why don't we need to set or reset the prev fields of objects in the implementation of the ALLOCATE-OBJECT and FREE-OBJECT procedures?

11.3-4

It is often desirable to keep all elements of a doubly linked list compact in storage, using, for example, the first m index locations in the multiple-array representation. (This is the case in a paged, virtual-memory computing environment.) Explain how the procedures ALLOCATE-OBJECT and FREE-OBJECT can be implemented so that the representation is compact. Assume that there are no pointers to elements of the linked list outside the list itself. (Hint: Use the array implementation of a stack.)

11.3-5

Let L be a doubly linked list of length m stored in arrays key, prev, and next of length n. Suppose that these arrays are managed by ALLOCATE-OBJECT and FREE-OBJECT procedures that keep a doubly linked free list F. Suppose further that of the n items, exactly m are on list L and n - m are on the free list. Write a procedure COMPACTIFY-LIST (L, F) that, given the list L and the free list F, moves the items in L so that they occupy array positions 1, 2, . . . , m and adjusts the free list F so that it remains correct, occupying array positions m + 1, m + 2, . . . , n. The running time of your procedure should be   [image] (m), and it should use only a constant amount of extra space. Give a careful argument for the correctness of your procedure.

11.4 Representing rooted trees

The methods for representing lists given in the previous section extend to any homogeneous data structure. In this section, we look specifically at the problem of representing rooted trees by linked data structures. We first look at binary trees, and then we present a method for rooted trees in which nodes can have an arbitrary number of children.

We represent each node of a tree by an object. As with linked lists, we assume that each node contains a key field. The remaining fields of interest are pointers to other nodes, and they vary according to the type of tree.

Binary trees

As shown in Figure 11.9, we use the fields p, left, and right to store pointers to the parent, left child, and right child of each node in a binary tree T. If p[x] = NIL, then x is the root. If node x has no left child, then left[x] = NIL, and similarly for the right child. The root of the entire tree T is pointed to by the attribute root[T]. If root [T] = NIL, then the tree is empty.

Figure 11.9 The representation of a binary tree T. Each node x has the fields p[x] (top), left[x] (lower left), and right[x] (lower right). The key fields are not shown.

Rooted trees with unbounded branching

The scheme for representing a binary tree can be extended to any class of trees in which the number of children of each node is at most some constant k: we replace the left and right fields by child1, child2, . . . , childk. This scheme no longer works when the number of children of a node is unbounded, since we do not know how many fields (arrays in the multiple-array representation) to allocate in advance. Moreover, even if the number of children k is bounded by a large constant but most nodes have a small number of children, we may waste a lot of memory.

Fortunately, there is a clever scheme for using binary trees to represent trees with arbitrary numbers of children. It has the advantage of using only O(n) space for any n-node rooted tree. The left-child, right-sibling representation is shown in Figure 11.10. As before, each node contains a parent pointer p, and root[T] points to the root of tree T. Instead of having a pointer to each of its children, however, each node x has only two pointers:

1.     left-child[x] points to the leftmost child of node x, and

2.     right-sibling[x] points to the sibling of x immediately to the right.

If node x has no children, then left-child[x] = NIL, and if node x is the rightmost child of its parent, then right-sibling[x] = NIL.

Other tree representations

We sometimes represent rooted trees in other ways. In Chapter 7, for example, we represented a heap, which is based on a complete binary tree, by a single array plus an index. The trees that appear in Chapter 22 are only traversed toward the root, so only the parent pointers are present; there are no pointers to children. Many other schemes are possible. Which scheme is best depends on the application.

Figure 11.10 The left-child, right-sibling representation of a tree T. Each node x has fields p[x] (top), left-child[x] (lower left), and right-sibling[x] (lower right). Keys are not shown.

Exercises

11.4-1

Draw the binary tree rooted at index 6 that is represented by the following fields.

index  key  left  right

----------------------

   1    12   7     3

   2    15   8    NIL

   3     4  10    NIL

   4    10   5     9

   5     2  NIL   NIL

   6    18   1    4

   7     7  NIL   NIL

   8    14   6    2

   9    21  NIL   NIL

  10     5  NIL   NIL

11.4-2

Write an O(n)-time recursive procedure that, given an n-node binary tree, prints out the key of each node in the tree.

11.4-3

Write an O(n)-time nonrecursive procedure that, given an n-node binary tree, prints out the key of each node in the tree. Use a stack as an auxiliary data structure.

11.4-4

Write an O(n)-time procedure that prints all the keys of an arbitrary rooted tree with n nodes, where the tree is stored using the left-child, right-sibling representation.

11.4-5

Write an O(n)-time nonrecursive procedure that, given an n-node binary tree, prints out the key of each node. Use no more than constant extra space outside of the tree itself and do not modify the tree, even temporarily, during the procedure.

11.4-6

The left-child, right-sibling representation of an arbitrary rooted tree uses three pointers in each node: left-child, right-sibling, and parent. From any node, the parent and all the children of the node can be reached and identified. Show how to achieve the same effect using only two pointers and one boolean value in each node.

Problems

11-1     Comparisons among lists

For each of the four types of lists in the following table, what is the asymptotic worst-case running time for each dynamic-set operation listed?

                   unsorted,  sorted,  unsorted,  sorted,

                    singly    singly    doubly    doubly

                    linked    linked    linked    linked

---------------------------------------------------------

SEARCH(L,k)

INSERT(L,x)

DELETE(L,x)

SUCCESSOR(L,x)

PREDECESSOR(L,x)

MINIMUM(L)

MAXIMUM(L)

11-2     Mergeable heaps using linked lists

A mergeable heap supports the following operations: MAKE-HEAP (which creates an empty mergeable heap), INSERT, MINIMUM, EXTRACT-MIN, and UNION. Show how to implement mergeable heaps using linked lists in each of the following cases. Try to make each operation as efficient as possible. Analyze the running time of each operation in terms of the size of the dynamic set(s) being operated on.

a.     Lists are sorted.

b.     Lists are unsorted.

c.     Lists are unsorted, and dynamic sets to be merged are disjoint.

11-3     Searching a sorted compact list

Exercise 11.3-4 asked how we might maintain an n-element list compactly in the first n positions of an array. We shall assume that all keys are distinct and that the compact list is also sorted, that is, key[i] < key[next[i]] for all i = 1, 2, . . . , n such that next[i]   [image]  NIL. Under these assumptions, we expect that the following randomized algorithm can be used to search the list much faster than linear time.

COMPACT-LIST-SEARCH(L, k)

1  i    head[L]

2  n    length[L]

3  while i    NIL and key[i] < k

4      do j    RANDOM(l, n)

5         if key[i] < key[j] and key[j] < k

6            then i    j

7         i    next[i]

8         if key[i] = k

9            then return i

10  return NIL

If we ignore lines 4-6 of the procedure, we have the usual algorithm for searching a sorted linked list, in which index i points to each position of the list in turn. Lines 4-6 attempt to skip ahead to a randomly chosen position j. Such a skip is beneficial if key[j] is larger than key[i] and smaller than k; in such a case, j marks a position in the list that i would have to pass by during an ordinary list search. Because the list is compact, we know that any choice of j between 1 and n indexes some object in the list rather than a slot on the free list.

a.     Why do we assume that all keys are distinct in COMPACT-LIST-SEARCH? Argue that random skips do not necessarily help asymptotically when the list contains repeated key values.

We can analyze the performance of COMPACT-LIST-SEARCH by breaking its execution into two phases. During the first phase, we discount any progress toward finding k that is accomplished by lines 7-9. That is, phase 1 consists of moving ahead in the list by random skips only. Likewise, phase 2 discounts progress accomplished by lines 4-6, and thus it operates like ordinary linear search.

Let Xt be the random variable that describes the distance in the linked list (that is, through the chain of next pointers) from position i to the desired key k after t iterations of phase l.

b.     Argue that the expected running time of COMPACT-LIST-SEARCH is O(t+E [Xt]) for all t   [image]  0.

c.     Show that   [image]. (Hint. Use equation (6.28).)

d.     Show that

e.     Prove that E[Xt]   [image]  n/(t + 1), and explain why this formula makes intuitive sense.

f.     Show that COMPACT-LIST-SEARCH runs in   [image]  expected time.

Chapter notes

Aho, Hopcroft, and Ullman [5] and Knuth [121] are excellent references for elementary data structures. Gonnet [90] provides experimental data on the performance of many data structure operations.

The origin of stacks and queues as data structures in computer science is unclear, since corresponding notions already existed in mathematics and paper-based business practices before the introduction of digital computers. Knuth [121] cites A. M. Turing for the development of stacks for subroutine linkage in 1947.

Pointer-based data structures also seem to be a folk invention. According to Knuth, pointers were apparently used in early computers with drum memories. The A-l language developed by G. M. Hopper in 1951 represented algebraic formulas as binary trees. Knuth credits the IPL-II language, developed in 1956 by A. Newell, J. C. Shaw, and H. A. Simon, for recognizing the importance and promoting the use of pointers. Their IPL-III language, developed in 1957, included explicit stack operations.

Go to Chapter 12     Back to Table of Contents


Intro to Algorithms: CHAPTER 12: HASH TABLES

 Return to Table of Contents

CHAPTER 12: HASH TABLES

Many applications require a dynamic set that supports only the dictionary operations INSERT, SEARCH, and DELETE. For example, a compiler for a computer language maintains a symbol table, in which the keys of elements are arbitrary character strings that correspond to identifiers in the language. A hash table is an effective data structure for implementing dictionaries. Although searching for an element in a hash table can take as long as searching for an element in a linked list-- [image] (n) time in the worst case--in practice, hashing performs extremely well. Under reasonable assumptions, the expected time to search for an element in a hash table is O(1).

A hash table is a generalization of the simpler notion of an ordinary array. Directly addressing into an ordinary array makes effective use of our ability to examine an arbitrary position in an array in O (1) time. Section 12.1 discusses direct addressing in more detail. Direct addressing is applicable when we can afford to allocate an array that has one position for every possible key.

When the number of keys actually stored is small relative to the total number of possible keys, hash tables become an effective alternative to directly addressing an array, since a hash table typically uses an array of size proportional to the number of keys actually stored. Instead of using the key as an array index directly, the array index is computed from the key. Section 12.2 presents the main ideas, and Section 12.3 describes how array indices can be computed from keys using hash functions. Several variations on the basic theme are presented and analyzed; the "bottom line" is that hashing is an extremely effective and practical technique: the basic dictionary operations require only O (1) time on the average.

12.1 Direct-address tables

Direct addressing is a simple technique that works well when the universe U of keys is reasonably small. Suppose that an application needs a dynamic set in which each element has a key drawn from the universe U = {0,1, . . . , m - 1}, where m is not too large. We shall assume that no two elements have the same key.

Figure 12.1 Implementing a dynamic set by a direct-address table T. Each key in the universe U = {0,1, . . . , 9} corresponds to an index in the table. The set K = {2, 3, 5, 8} of actual keys determines the slots in the table that contain pointers to elements. The other slots, heavily shaded, contain NIL.

To represent the dynamic set, we use an array, or direct-address table, T [0 . . m - 1], in which each position, or slot, corresponds to a key in the universe U. Figure 12.1 illustrates the approach; slot k points to an element in the set with key k. If the set contains no element with key k, then T[k] = NIL.

The dictionary operations are trivial to implement.

DIRECT-ADDRESS-SEARCH(T,k)

return T[k]

DIRECT-ADDRESS-INSERT(T,x)

T[key[x]]    x

DIRECT-ADDRESS-DELETE(T,x)

T[key[x]]    NIL

Each of these operations is fast: only O(1) time is required.

For some applications, the elements in the dynamic set can be stored in the direct-address table itself. That is, rather than storing an element's key and satellite data in an object external to the direct-address table, with a pointer from a slot in the table to the object, we can store the object in the slot itself, thus saving space. Moreover, it is often unnecessary to store the key field of the object, since if we have the index of an object in the table, we have its key. If keys are not stored, however, we must have some way to tell if the slot is empty.

Exercises

12.1-1

Consider a dynamic set S that is represented by a direct-address table T of length m. Describe a procedure that finds the maximum element of S. What is the worst-case performance of your procedure?

12.1-2

A bit vector is simply an array of bits (0's and 1's). A bit vector of length m takes much less space than an array of m pointers. Describe how to use a bit vector to represent a dynamic set of distinct elements with no satellite data. Dictionary operations should run in O(1) time.

12.1-3

Suggest how to implement a direct-address table in which the keys of stored elements do not need to be distinct and the elements can have satellite data. All three dictionary operations (INSERT, DELETE, and SEARCH) should run in O(1) time. (Don't forget that DELETE takes as an argument a pointer to an object to be deleted, not a key.)

12.1-4

We wish to implement a dictionary by using direct addressing on a huge array. At the start, the array entries may contain garbage, and initializing the entire array is impractical because of its size. Describe a scheme for implementing a direct-address dictionary on a huge array. Each stored object should use O(1) space; the operations SEARCH, INSERT, and DELETE should take O(1) time each; and the initialization of the data structure should take O(1) time. (Hint: Use an additional stack, whose size is the number of keys actually stored in the dictionary, to help determine whether a given entry in the huge array is valid or not.)

12.2 Hash tables

The difficulty with direct addressing is obvious: if the universe U is large, storing a table T of size |U| may be impractical, or even impossible, given the memory available on a typical computer. Furthermore, the set K of keys actually stored may be so small relative to U that most of the space allocated for T would be wasted.

When the set K of keys stored in a dictionary is much smaller than the universe U of all possible keys, a hash table requires much less storage than a direct-address table. Specifically, the storage requirements can be reduced to   [image] (|K|), even though searching for an element in the hash table still requires only O(1) time. (The only catch is that this bound is for the average time, whereas for direct addressing it holds for the worst-case time.)

Figure 12.2 Using a hash function h to map keys to hash-table slots. Keys k2 and k5 map to the same slot, so they collide.

With direct addressing, an element with key k is stored in slot k. With hashing, this element is stored in slot h(k); that is, a hash function h is used to compute the slot from the key k. Here h maps the universe U of keys into the slots of a hash table T[0 . . m - 1]:

h: U   {0,1, . . . , m - 1} .

We say that an element with key k hashes to slot h(k); we also say that h(k) is the hash value of key k. Figure 12.2 illustrates the basic idea. The point of the hash function is to reduce the range of array indices that need to be handled. Instead of |U| values, we need to handle only m values. Storage requirements are correspondingly reduced.

The fly in the ointment of this beautiful idea is that two keys may hash to the same slot--a collision. Fortunately, there are effective techniques for resolving the conflict created by collisions.

Of course, the ideal solution would be to avoid collisions altogether. We might try to achieve this goal by choosing a suitable hash function h. One idea is to make h appear to be "random," thus avoiding collisions or at least minimizing their number. The very term "to hash," evoking images of random mixing and chopping, captures the spirit of this approach. (Of course, a hash function h must be deterministic in that a given input k should always produce the same output h(k).) Since |U| > m, however, there must be two keys that have the same hash value; avoiding collisions altogether is therefore impossible. Thus, while a well-designed, "random"- looking hash function can minimize the number of collisions, we still need a method for resolving the collisions that do occur.

The remainder of this section presents the simplest collision resolution technique, called chaining. Section 12.4 introduces an alternative method for resolving collisions, called open addressing.

Figure 12.3 Collision resolution by chaining. Each hash-table slot T[j] contains a linked list of all the keys whose hash value is j. For example, h(k1) = h(k4) and h(k5) = h(k2) = h(k7).

Collision resolution by chaining

In chaining, we put all the elements that hash to the same slot in a linked list, as shown in Figure 12.3. Slot j contains a pointer to the head of the list of all stored elements that hash to j; if there are no such elements, slot j contains NIL.

The dictionary operations on a hash table T are easy to implement when collisions are resolved by chaining.

CHAINED-HASH-INSERT(T,x)

insert x at the head of list T[h(key[x])]

CHAINED-HASH-SEARCH(T,k)

search for an element with key k in list T[h(k)]

CHAINED-HASH-DELETE(T,x)

delete x from the list T[h(key[x])]

The worst-case running time for insertion is O(1). For searching, the worst-case running time is proportional to the length of the list; we shall analyze this more closely below. Deletion of an element x can be accomplished in O(1) time if the lists are doubly linked. (If the lists are singly linked, we must first find x in the list T[h(key[x])], so that the next link of x's predecessor can be properly set to splice x out; in this case, deletion and searching have essentially the same running time.)

Analysis of hashing with chaining

How well does hashing with chaining perform? In particular, how long does it take to search for an element with a given key?

Given a hash table T with m slots that stores n elements, we define the load factor   [image]  for T as n/m, that is, the average number of elements stored in a chain. Our analysis will be in terms of   [image]; that is, we imagine   [image]  staying fixed as n and m go to infinity. (Note that   [image]  can be less than, equal to, or greater than l .)

The worst-case behavior of hashing with chaining is terrible: all n keys hash to the same slot, creating a list of length n. The worst-case time for searching is thus   [image] (n) plus the time to compute the hash function--no better than if we used one linked list for all the elements. Clearly, hash tables are not used for their worst-case performance.

The average performance of hashing depends on how well the hash function h distributes the set of keys to be stored among the m slots, on the average. Section 12.3 discusses these issues, but for now we shall assume that any given element is equally likely to hash into any of the m slots, independently of where any other element has hashed to. We call this the assumption of simple uniform hashing.

We assume that the hash value h(k) can be computed in O(1) time, so that the time required to search for an element with key k depends linearly on the length of the list T[h(k)]. Setting aside the O(1) time required to compute the hash function and access slot h(k), let us consider the expected number of elements examined by the search algorithm, that is, the number of elements in the list T[h(k)] that are checked to see if their keys are equal to k. We shall consider two cases. In the first, the search is unsuccessful: no element in the table has key k. In the second, the search successfully finds an element with key k.

Theorem 12.1

In a hash table in which collisions are resolved by chaining, an unsuccessful search takes time   [image] (1 +   [image]  ), on the average, under the assumption of simple uniform hashing.

Proof     Under the assumption of simple uniform hashing, any key k is equally likely to hash to any of the m slots. The average time to search unsuccessfully for a key k is thus the average time to search to the end of one of the m lists. The average length of such a list is the load factor   [image]  = n/m. Thus, the expected number of elements examined in an unsuccessful search is   [image], and the total time required (including the time for computing h(k)) is   [image] (1 +   [image]).

Theorem 12.2

In a hash table in which collisions are resolved by chaining, a successful search takes time   [image] (1 +  [image]), on the average, under the assumption of simple uniform hashing.

Proof     We assume that the key being searched for is equally likely to be any of the n keys stored in the table. We also assume that the CHAINED-HASH-INSERT procedure inserts a new element at the end of the list instead of the front. (By Exercise 12.2-3, the average successful search time is the same whether new elements are inserted at the front of the list or at the end.) The expected number of elements examined during a successful search is 1 more than the number of elements examined when the sought-for element was inserted (since every new element goes at the end of the list). To find the expected number of elements examined, we therefore take the average, over the n items in the table, of 1 plus the expected length of the list to which the ith element is added. The expected length of that list is (i- 1)/m, and so the expected number of elements examined in a successful search is

Thus, the total time required for a successful search (including the time for computing the hash function) is   [image] (2 +   [image] /2 - 1/2m) =   [image] (1 +   [image]).

What does this analysis mean? If the number of hash-table slots is at least proportional to the number of elements in the table, we have n = O(m) and, consequently,   [image]  = n/m = O(m)/m = O(1). Thus, searching takes constant time on average. Since insertion takes O( l ) worst-case time (see Exercise 12.2-3), and deletion takes O(l) worst-case time when the lists are doubly linked, all dictionary operations can be supported in O( l ) time on average.

Exercises

12.2-1

Suppose we use a random hash function h to hash n distinct keys into an array T of length m. What is the expected number of collisions? More precisely, what is the expected cardinality of {(x,y): h(x) = h(y)}?

12.2-2

Demonstrate the insertion of the keys 5, 28, 19, 15, 20, 33, 12, 17, 10 into a hash table with collisions resolved by chaining. Let the table have 9 slots, and let the hash function be h(k) = k mod 9.

12.2-3

Argue that the expected time for a successful search with chaining is the same whether new elements are inserted at the front or at the end of a list. (Hint: Show that the expected successful search time is the same for any two orderings of any list.)

12.2-4

Professor Marley hypothesizes that substantial performance gains can be obtained if we modify the chaining scheme so that each list is kept in sorted order. How does the professor's modification affect the running time for successful searches, unsuccessful searches, insertions, and deletions?

12.2-5

Suggest how storage for elements can be allocated and deallocated within the hash table itself by linking all unused slots into a free list. Assume that one slot can store a flag and either one element plus a pointer or two pointers. All dictionary and free-list operations should run in O(l) expected time. Does the free list need to be doubly linked, or does a singly linked free list suffice?

12.2-6

Show that if |U| > nm, there is a subset of U of size n consisting of keys that all hash to the same slot, so that the worst-case searching time for hashing with chaining is   [image] (n).

12.3 Hash functions

In this section, we discuss some issues regarding the design of good hash functions and then present three schemes for their creation: hashing by division, hashing by multiplication, and universal hashing.

What makes a good hash function?

A good hash function satisfies (approximately) the assumption of simple uniform hashing: each key is equally likely to hash to any of the m slots. More formally, let us assume that each key is drawn independently from U according to a probability distribution P; that is, P(k) is the probability that k is drawn. Then the assumption of simple uniform hashing is that

(12.1)

Unfortunately, it is generally not possible to check this condition, since P is usually unknown.

Sometimes (rarely) we do know the distribution P. For example, suppose the keys are known to be random real numbers k independently and uniformly distributed in the range 0   [image]  k < 1. In this case, the hash function

h(k) =   km

can be shown to satisfy equation (12.1).

In practice, heuristic techniques can be used to create a hash function that is likely to perform well. Qualitative information about P is sometimes useful in this design process. For example, consider a compiler's symbol table, in which the keys are arbitrary character strings representing identifiers in a program. It is common for closely related symbols, such as pt and pts, to occur in the same program. A good hash function would minimize the chance that such variants hash to the same slot.

A common approach is to derive the hash value in a way that is expected to be independent of any patterns that might exist in the data. For example, the "division method" (discussed further below) computes the hash value as the remainder when the key is divided by a specified prime number. Unless that prime is somehow related to patterns in the probability distribution P, this method gives good results.

Finally, we note that some applications of hash functions might require stronger properties than are provided by simple uniform hashing. For example, we might want keys that are "close" in some sense to yield hash values that are far apart. (This property is especially desirable when we are using linear probing, defined in Section 12.4.)

Interpreting keys as natural numbers

Most hash functions assume that the universe of keys is the set N = {0,1,2, . . .} of natural numbers. Thus, if the keys are not natural numbers, a way must be found to interpret them as natural numbers. For example, a key that is a character string can be interpreted as an integer expressed in suitable radix notation. Thus, the identifier pt might be interpreted as the pair of decimal integers (112,116), since p = 112 and t = 116 in the ASCII character set; then, expressed as a radix-128 integer, pt becomes (112   [image]  128) + 116 = 14452. It is usually straightforward in any given application to devise some such simple method for interpreting each key as a (possibly large) natural number. In what follows, we shall assume that the keys are natural numbers.

12.3.1 The division method

In the division method for creating hash functions, we map a key k into one of m slots by taking the remainder of k divided by m. That is, the hash function is

h(k) = k mod m .

For example, if the hash table has size m = 12 and the key is k = 100, then h(k) = 4. Since it requires only a single division operation, hashing by division is quite fast.

When using the division method, we usually avoid certain values of m. For example, m should not be a power of 2, since if m = 2p, then h(k) is just the p lowest-order bits of k. Unless it is known a priori that the probability distribution on keys makes all low-order p-bit patterns equally likely, it is better to make the hash function depend on all the bits of the key. Powers of 10 should be avoided if the application deals with decimal numbers as keys, since then the hash function does not depend on all the decimal digits of k. Finally, it can be shown that when m = 2p - 1 and k is a character string interpreted in radix 2p, two strings that are identical except for a transposition of two adjacent characters will hash to the same value.

Good values for m are primes not too close to exact powers of 2. For example, suppose we wish to allocate a hash table, with collisions resolved by chaining, to hold roughly n = 2000 character strings, where a character has 8 bits. We don't mind examining an average of 3 elements in an unsuccessful search, so we allocate a hash table of size m = 701. The number 701 is chosen because it is a prime near   [image]  = 2000/3 but not near any power of 2. Treating each key k as an integer, our hash function would be

h(k) = k mod 701 .

As a precautionary measure, we could check how evenly this hash function distributes sets of keys among the slots, where the keys are chosen from "real" data.

12.3.2 The multiplication method

The multiplication method for creating hash functions operates in two steps. First, we multiply the key k by a constant A in the range 0 < A < 1 and extract the fractional part of kA. Then, we multiply this value by m and take the floor of the result. In short, the hash function is

h(k) =   m (k A mod 1)   ,

where "k A mod 1" means the fractional part of kA, that is, kA -   [image] kA  [image] .

An advantage of the multiplication method is that the value of m is not critical. We typically choose it to be a power of 2--m = 2p for someinteger p--since we can then easily implement the function on most computers as follows. Suppose that the word size of the machine is w bits and that k fits into a single word. Referring to Figure 12.4, we first multiply k by the w-bit integer   [image] A   [image]  2w  [image]. The result is a 2w-bit value r1 2w + r0, where r1 is the high-order word of the product and r0 is the low-order word of the product. The desired p-bit hash value consists of the p most significant bits of r0.

Figure 12.4 The multiplication method of hashing. The w-bit representation of the key k is multiplied by the w-bit value   A.2w  , where 0 < A < 1 is a suitable constant. The p highest-order bits of the lower w-bit half of the product form the desired hash value h(k).

Although this method works with any value of the constant A, it works better with some values than with others. The optimal choice depends on the characteristics of the data being hashed. Knuth [123] discusses the choice of A in some detail and suggests that

(12.2)

is likely to work reasonably well.

As an example, if we have k = 123456, m = 10000, and A as in equation (12.2), then

h(k)  =    10000    (123456    0.61803 . . . mod 1)

=    10000    (76300.0041151. . . mod 1)

=    10000    0.0041151 . . .

=    41.151 . . .

=  41 .

12.3.3 Universal hashing

If a malicious adversary chooses the keys to be hashed, then he can choose n keys that all hash to the same slot, yielding an average retrieval time of   [image]  (n). Any fixed hash function is vulnerable to this sort of worst-case behavior; the only effective way to improve the situation is to choose the hash function randomly in a way that is independent of the keys that are actually going to be stored. This approach, called universal hashing, yields good performance on the average, no matter what keys are chosen by the adversary.

The main idea behind universal hashing is to select the hash function at random at run time from a carefully designed class of functions. As in the case of quicksort, randomization guarantees that no single input will always evoke worst-case behavior. Because of the randomization, the algorithm can behave differently on each execution, even for the same input. This approach guarantees good average-case performance, no matter what keys are provided as input. Returning to the example of a compiler's symbol table, we find that the programmer's choice of identifiers cannot now cause consistently poor hashing performance. Poor performance occurs only if the compiler chooses a random hash function that causes the set of identifiers to hash poorly, but the probability of this occurring is small and is the same for any set of identifiers of the same size.

Let   [image]  be a finite collection of hash functions that map a given universe U of keys into the range {0,1, . . . , m - 1}. Such a collection is said to be universal if for each pair of distinct keys x,y   [image]  U, the number of hash functions   [image]  for which h(x) = h(y) is precisely   [image]. In other words, with a hash function randomly chosen from   [image], the chance of a collision between x and y when x   [image]  y is exactly 1/m, which is exactly the chance of a collision if h(x) and h(y) are randomly chosen from the set {0,1, . . . , m - 1}.

The following theorem shows that a universal class of hash functions gives good average-case behavior.

Theorem 12.3

If h is chosen from a universal collection of hash functions and is used to hash n keys into a table of size m, where n   [image]  m, the expected number of collisions involving a particular key x is less than 1.

Proof     For each pair y, z of distinct keys, let cyz be a random variable that is 1 if h(y) = h(z) (i.e., if y and z collide using h) and 0 otherwise. Since, by definition, a single pair of keys collides with probability 1/m, we have

E[cyz] = 1/m .

Let Cx be the total number of collisions involving key x in a hash table T of size m containing n keys. Equation (6.24) gives

Since n   [image]  m, we have E [Cx] < 1.

But how easy is it to design a universal class of hash functions? It is quite easy, as a little number theory will help us prove. Let us choose our table size m to be prime (as in the division method). We decompose a key x into r+ 1 bytes (i.e., characters, or fixed-width binary substrings), so that x =   [image] x0, x1,. . . , xr  [image]; the only requirement is that the maximum value of a byte should be less than m. Let a =   [image] a0, a1, . . . , ar  [image]  denote a sequence of r + 1 elements chosen randomly from the set {0,1, . . . , m - 1}. We define a corresponding hash function   [image]:

(12.3)

With this definition,

(12.4)

has mr+1 members.

Theorem 12.4

The class   [image]  defined by equations (12.3) and (12.4) is a universal class of hash functions.

Proof     Consider any pair of distinct keys x, y. Assume that x0   [image]  y0. (A similar argument can be made for a difference in any other byte position.) For any fixed values of a1, a2, . . . , ar, there is exactly one value of a0 that satisfies the equation h(x) = h(y); this a0 is the solution to

To see this property, note that since m is prime, the nonzero quantity x0 - y0 has a multiplicative inverse modulo m, and thus there is a unique solution for a0 modulo m. (See Section 33.4.) Therefore, each pair of keys x and y collides for exactly mr values of a, since they collide exactly once for each possible value of   [image] al, a2, . . ., ar  [image]  (i.e., for the unique value of a0 noted above). Since there are mr+l possible values for the sequence a, keys x and y collide with probability exactly mr/mr+1 = 1/m. Therefore,   [image]  is universal.

Exercises

12.3-1

Suppose we wish to search a linked list of length n, where each element contains a key k along with a hash value h(k). Each key is a long character string. How might we take advantage of the hash values when searching the list for an element with a given key?

12.3-2

Suppose a string of r characters is hashed into m slots by treating it as a radix-128 number and then using the division method. The number m is easily represented as a 32-bit computer word, but the string of r characters, treated as a radix-128 number, takes many words. How can we apply the division method to compute the hash value of the character string without using more than a constant number of words of storage outside the string itself?

12.3-3

Consider a version of the division method in which h(k) = k mod m, where m = 2p - I and k is a character string interpreted in radix 2p. Show that if string x can be derived from string y by permuting its characters, then x and y hash to the same value. Give an example of an application in which this property would be undesirable in a hash function.

12.3-4

Consider a hash table of size m = 1000 and the hash function h(k) =   [image] m (k A mod 1)  [image]  for A =   [image]. Compute the locations to which the keys 61, 62, 63, 64, and 65 are mapped.

12.3-5

Show that if we restrict each component ai of a in equation (12.3) to be nonzero, then the set   [image]  as defined in equation (12.4) is not universal. (Hint: Consider the keys x = 0 and y = 1.)

12.4 Open addressing

In open addressing, all elements are stored in the hash table itself. That is, each table entry contains either an element of the dynamic set or NIL. When searching for an element, we systematically examine table slots until the desired element is found or it is clear that the element is not in the table. There are no lists and no elements stored outside the table, as there are in chaining. Thus, in open addressing, the hash table can "fill up" so that no further insertions can be made; the load factor   [image]  can never exceed 1.

Of course, we could store the linked lists for chaining inside the hash table, in the otherwise unused hash-table slots (see Exercise 12.2-5), but the advantage of open addressing is that it avoids pointers altogether. Instead of following pointers, we compute the sequence of slots to be examined. The extra memory freed by not storing pointers provides the hash table with a larger number of slots for the same amount of memory, potentially yielding fewer collisions and faster retrieval.

To perform insertion using open addressing, we successively examine, or probe, the hash table until we find an empty slot in which to put the key. Instead of being fixed in the order 0, 1, . . . , m - 1 (which requires   [image] (n) search time), the sequence of positions probed depends upon the key being inserted. To determine which slots to probe, we extend the hash function to include the probe number (starting from 0) as a second input. Thus, the hash function becomes

h:U X {0, 1, . . . , m -1}    {0, 1, . . . , m -1} .

With open addressing, we require that for every key k, the probe sequence

  h(k, 0), h(k, 1), . . . , h(k, m - 1)

be a permutation of   [image] 0, 1, . . . , m - 1  [image], so that every hash-table position is eventually considered as a slot for a new key as the table fills up. In the following pseudocode, we assume that the elements in the hash table T are keys with no satellite information; the key k is identical to the element containing key k. Each slot contains either a key or NIL (if the slot is empty).

HASH-INSERT(T,k)

1  i    0

2   repeat j    h(k,i)

3          if T[j] = NIL

4             then T[j]    k

5                  return j

6             else i    i + 1

7  until i = m

8 error "hash table overflow"

The algorithm for searching for key k probes the same sequence of slots that the insertion algorithm examined when key k was inserted. Therefore, the search can terminate (unsuccessfully) when it finds an empty slot, since k would have been inserted there and not later in its probe sequence. (Note that this argument assumes that keys are not deleted from the hash table.) The procedure HASH-SEARCH takes as input a hash table T and a key k, returning j if slot j is found to contain key k, or NIL if key k is not present in table T.

HASH-SEARCH(T, k)

1  i    0

2   repeat j    h(k, i)

3          if T[j]= j

4             then return j

5           i    i + 1

6    until T[j] = NIL or i = m

7  return NIL

Deletion from an open-address hash table is difficult. When we delete a key from slot i, we cannot simply mark that slot as empty by storing NIL in it. Doing so might make it impossible to retrieve any key k during whose insertion we had probed slot i and found it occupied. One solution is to mark the slot by storing in it the special value DELETED instead of NIL. We would then modify the procedure HASH-SEARCH so that it keeps on looking when it sees the value DELETED, while HASH-INSERT would treat such a slot as if it were empty so that a new key can be inserted. When we do this, though, the search times are no longer dependent on the load factor   [image], and for this reason chaining is more commonly selected as a collision resolution technique when keys must be deleted.

In our analysis, we make the assumption of uniform hashing: we assume that each key considered is equally likely to have any of the m! permutations of {0, 1, . . . , m - 1} as its probe sequence. Uniform hashing generalizes the notion of simple uniform hashing defined earlier to the situation in which the hash function produces not just a single number, but a whole probe sequence. True uniform hashing is difficult to implement, however, and in practice suitable approximations (such as double hashing, defined below) are used.

Three techniques are commonly used to compute the probe sequences required for open addressing: linear probing, quadratic probing, and double hashing. These techniques all guarantee that   [image] h(k, 1), h(k, 2), . . . , h(k, m)  [image]  is a permutation of   [image] 0, 1, . . . , m - 1  [image]  for each key k. None of these techniques fulfills the assumption of uniform hashing, however, since none of them is capable of generating more than m2 different probe sequences (instead of the m! that uniform hashing requires). Double hashing has the greatest number of probe sequences and, as one might expect, seems to give the best results.

Linear probing

Given an ordinary hash function h': U   [image]  {0, 1, . . . , m - 1}, the method of linear probing uses the hash function

h(k,i) = (h'(k) + i) mod m

for i = 0,1,...,m - 1. Given key k, the first slot probed is T[h'(k)]. We next probe slot T[h'(k) + 1], and so on up to slot T[m - 1]. Then we wrap around to slots T[0], T[1], . . . , until we finally probe slot T[h'(k) - 1]. Since the initial probe position determines the entire probe sequence, only m distinct probe sequences are used with linear probing.

Linear probing is easy to implement, but it suffers from a problem known as primary clustering. Long runs of occupied slots build up, increasing the average search time. For example, if we have n = m/2 keys in the table, where every even-indexed slot is occupied and every odd-indexed slot is empty, then the average unsuccessful search takes 1.5 probes. If the first n = m/2 locations are the ones occupied, however, the average number of probes increases to about n/4 = m/8. Clusters are likely to arise, since if an empty slot is preceded by i full slots, then the probability that the empty slot is the next one filled is (i + 1)/m, compared with a probability of 1/m if the preceding slot was empty. Thus, runs of occupied slots tend to get longer, and linear probing is not a very good approximation to uniform hashing.

Quadratic probing

Quadratic probing uses a hash function of the form

h(k,i) = (h'(k) + c1i + c2i2) mod m,

(12.5)

where (as in linear probing) h' is an auxiliary hash function, c1 and c2   [image]  0 are auxiliary constants, and i = 0, 1, . . . , m - 1. The initial position probed is T[h'(k)]; later positions probed are offset by amounts that depend in a quadratic manner on the probe number i. This method works much better than linear probing, but to make full use of the hash table, the values of c1, c2, and m are constrained. Problem 12-4 shows one way to select these parameters. Also, if two keys have the same initial probe position, then their probe sequences are the same, since h(k1, 0) = h(k2, 0) implies h(k1, i) = h(k2, i). This leads to a milder form of clustering, called secondary clustering. As in linear probing, the initial probe determines the entire sequence, so only m distinct probe sequences are used.

Double hashing

Double hashing is one of the best methods available for open addressing because the permutations produced have many of the characteristics of randomly chosen permutations. Double hashing uses a hash function of the form

h(k, i) = (h1(k) + ih2(k)) mod m,

where h1 and h2 are auxiliary hash functions. The initial position probed is T[h1 (k)]; successive probe positions are offset from previous positions by the amount h2(k), modulo m. Thus, unlike the case of linear or quadratic probing, the probe sequence here depends in two ways upon the key k, since the initial probe position, the offset, or both, may vary. Figure 12.5 gives an example of insertion by double hashing.

Figure 12.5 Insertion by double hashing. Here we have a hash table of size 13 with h1(k) = k mod 13 and h2(k) = 1 + (k mod 11). Since 14    1 mod 13 and 14    3 mod 11, the key 14 will be inserted into empty slot 9, after slots 1 and 5 have been examined and found to be already occupied.

The value h2(k) must be relatively prime to the hash-table size m for the entire hash table to be searched. Otherwise, if m and h2(k) have greatest common divisor d > 1 for some key k, then a search for key k would examine only (1/d)th of the hash table. (See Chapter 33.) A convenient way to ensure this condition is to let m be a power of 2 and to design h2 so that it always produces an odd number. Another way is to let m be prime and to design h2 so that it always returns a positive integer less than m. For example, we could choose m prime and let

h1(k) = k mod m ,

h2(k) = 1 + (k mod m'),

where m' is chosen to be slightly less than m (say, m - 1 or m - 2). For example, if k = 123456 and m = 701, we have h1(k) = 80 and h2(k) = 257, so the first probe is to position 80, and then every 257th slot (modulo m) is examined until the key is found or every slot is examined.

Double hashing represents an improvement over linear or quadratic probing in that   [image] (m2) probe sequences are used, rather than   [image] (m), since each possible (h1 (k), h2(k)) pair yields a distinct probe sequence, and as we vary the key, the initial probe position h1(k) and the offset h2(k) may vary independently. As a result, the performance of double hashing appears to be very close to the performance of the "ideal" scheme of uniform hashing.

Analysis of open-address hashing

Our analysis of open addressing, like our analysis of chaining, is expressed in terms of the load factor   [image]  of the hash table, as n and m go to infinity. Recall that if n elements are stored in a table with m slots, the average number of elements per slot is   [image]  = n/m. Of course, with open addressing, we have at most one element per slot, and thus n   [image]  m, which implies   [image]    [image]  1.

We assume that uniform hashing is used. In this idealized scheme, the probe sequence   [image] h(k, 0), h(k, 1), . . . , h(k, m - 1)  [image]  for each key k is equally likely to be any permutation on   [image] 0, 1, . . . , m - 1  [image]. That is, each possible probe sequence is equally likely to be used as the probe sequence for an insertion or a search. Of course, a given key has a unique fixed probe sequence associated with it; what is meant here is that, considering the probability distribution on the space of keys and the operation of the hash function on the keys, each possible probe sequence is equally likely.

We now analyze the expected number of probes for hashing with open addressing under the assumption of uniform hashing, beginning with an analysis of the number of probes made in an unsuccessful search.

Theorem 12.5

Given an open-address hash table with load factor   [image]  = n/m < 1, the expected number of probes in an unsuccessful search is at most 1/(1 -   [image] ), assuming uniform hashing.

Proof     In an unsuccessful search, every probe but the last accesses an occupied slot that does not contain the desired key, and the last slot probed is empty. Let us define

pi = Pr {exactly i probes access occupied slots}

for i = 0, 1, 2, . . . . For i > n, we have pi = 0, since we can find at most n slots already occupied. Thus, the expected number of probes is

(12.6)

To evaluate equation (12.6), we define

qi = Pr {at least i probes access occupied slots}

for i = 0, 1, 2, . . . . We can then use identity (6.28):

What is the value of qi for i   [image]  1? The probability that the first probe accesses an occupied slot is n/m; thus,

With uniform hashing, a second probe, if necessary, is to one of the remaining m - 1 unprobed slots, n - 1 of which are occupied. We make a second probe only if the first probe accesses an occupied slot; thus,

In general, the ith probe is made only if the first i - 1 probes access occupied slots, and the slot probed is equally likely to be any of the remaining m - i + 1 slots, n - i + 1 of which are occupied. Thus,

for i = 1, 2, . . . , n, since (n - j) / (m - j)   [image]  n / m if n   [image]  m and j   [image]  0. After n probes, all n occupied slots have been seen and will not be probed again, and thus qi = 0 for i   [image]  n.

We are now ready to evaluate equation (12.6). Given the assumption that   [image]  < 1, the average number of probes in an unsuccessful search is

(12.7)

Equation (12.7) has an intuitive interpretation: one probe is always made, with probability approximately   [image]  a second probe is needed, with probability approximately   [image] 2 a third probe is needed, and so on.

If   [image]  is a constant, Theorem 12.5 predicts that an unsuccessful search runs in O(1) time. For example, if the hash table is half full, the average number of probes in an unsuccessful search is 1/(1 - .5) = 2. If it is 90 percent full, the average number of probes is 1/(1 - .9) = 10.

Theorem 12.5 gives us the performance of the HASH-INSERT procedure almost immediately.

Corollary 12.6

Inserting an element into an open-address hash table with load factor   [image]  requires at most 1/(1 -   [image] ) probes on average, assuming uniform hashing.

Proof     An element is inserted only if there is room in the table, and thus   [image]  < 1. Inserting a key requires an unsuccessful search followed by placement of the key in the first empty slot found. Thus, the expected number of probes is 1/(1 -   [image] ).

Computing the expected number of probes for a successful search requires a little more work.

Theorem 12.7

Given an open-address hash table with load factor   [image]  < 1, the expected number of probes in a successful search is at most

assuming uniform hashing and assuming that each key in the table is equally likely to be searched for.

Proof     A search for a key k follows the same probe sequence as was followed when the element with key k was inserted. By Corollary 12.6, if k was the (i + 1)st key inserted into the hash table, the expected number of probes made in a search for k is at most 1 / (1 - i/m) = m/(m - i). Averaging over all n keys in the hash table gives us the average number of probes in a successful search:

where Hi =   [image]  is the ith harmonic number (as defined in equation (3.5)). Using the bounds ln i   [image]  Hi   [image]  ln i + 1 from equations (3.11)and (3.12), we obtain

for a bound on the expected number of probes in a successful search.

If the hash table is half full, the expected number of probes is less than 3.387. If the hash table is 90 percent full, the expected number of probes is less than 3.670.

Exercises

12.4-1

Consider inserting the keys 10, 22, 31, 4, 15, 28, 17, 88, 59 into a hash table of length m = 11 using open addressing with the primary hash function h'(k) = k mod m. Illustrate the result of inserting these keys using linear probing, using quadratic probing with c1 = 1 and c2 = 3, and using double hashing with h2(k) = 1 + (k mod (m - 1)).

12.4-2

Write pseudocode for HASH-DELETE as outlined in the text, and modify HASH-INSERT and HASH-SEARCH to incorporate the special value DELETED.

12.4-3

Suppose that we use double hashing to resolve collisions; that is, we use the hash function h(k, i) = (h1(k) + ih2(k)) mod m. Show that the probe sequence   [image] h(k, 0), h(k, 1), . . . , h(k, m - 1)  [image]  is a permutation of the slot sequence   [image] 0, 1, . . . , m - 1  [image]  if and only if h2(k) is relatively prime to m. (Hint: See Chapter 33.)

12.4-4

Consider an open-address hash table with uniform hashing and a load factor   [image]  = 1/2. What is the expected number of probes in an unsuccessful search? What is the expected number of probes in a successful search? Repeat these calculations for the load factors 3/4 and 7/8.

12.4-5

Suppose that we insert n keys into a hash table of size m using open addressing and uniform hashing. Let p(n, m) be the probability that no collisions occur. Show that p(n, m)   [image]  e-n(n - 1)/2m. (Hint: See equation (2.7).) Argue that when n exceeds   [image], the probability of avoiding collisions goes rapidly to zero.

12.4-6

The bound on the harmonic series can be improved to

(12.8)

where   [image]  = 0.5772156649 . . . is known as Euler's constant and   [image]  satisfies 0 <   [image]  < 1. (See Knuth [121] for a derivation.) How does this improved approximation for the harmonic series affect the statement and proof of Theorem 12.7?

12.4-7

Consider an open-address hash table with a load factor   [image] . Find the nonzero value   [image]  for which the expected number of probes in an unsuccessful search equals twice the expected number of probes in a successful search. Use the estimate (1/  [image] ) ln(1/(1 -   [image] )) for the number of probes required for a successful search.

Problems

12-1     Longest-probe bound for hashing

A hash table of size m is used to store n items, with n   [image]  m/2. Open addressing is used for collision resolution.

a.     Assuming uniform hashing, show that for i = 1, 2, . . . , n, the probability that the ith insertion requires strictly more than k probes is at most 2-k.

b.     Show that for i = 1, 2, . . ., n, the probability that the ith insertion requires more than 2 lg n probes is at most 1/n2.

Let the random variable Xi denote the number of probes required by the ith insertion. You have shown in part (b) that Pr{Xi >2 1g n}   [image]  1/n2. Let the random variable X = max1  [image] i  [image] n Xi denote the maximum number of probes required by any of the n insertions.

c.     Show that Pr{X > 2 1g n}   [image]  1/n.

d.     Show that the expected length of the longest probe sequence is E[X] = O(lg n)

12-2     Searching a static set

You are asked to implement a dynamic set of n elements in which the keys are numbers. The set is static (no INSERT or DELETE operations), and the only operation required is SEARCH. You are given an arbitrary amount of time to preprocess the n elements so that SEARCH operations run quickly.

a.     Show that SEARCH can be implemented in O(1g n) worst-case time using no extra storage beyond what is needed to store the elements of the set themselves.

b.     Consider implementing the set by open-address hashing on m slots, and assume uniform hashing. What is the minimum amount of extra storage m - n required to make the average performance of an unsuccessful SEARCH operation be at least as good as the bound in part (a)? Your answer should be an asymptotic bound on m - n in terms of n.

12-3     Slot-size bound for chaining

Suppose that we have a hash table with n slots, with collisions resolved by chaining, and suppose that n keys are inserted into the table. Each key is equally likely to be hashed to each slot. Let M be the maximum number of keys in any slot after all the keys have been inserted. Your mission is to prove an O(1g n/1g 1g n) upper bound on E[M], the expected value of M.

a.     Argue that the probability Qk that k keys hash to a particular slot is given by

b.     Let Pk be the probability that M = k, that is, the probability that the slot containing the most keys contains k keys. Show that Pk   [image]  nQk.

c.     Use Stirling's approximation, equation (2.1l), to show that Qk < ek/kk.

d.     Show that there exists a constant c > 1 such that Qk 0 < 1/n3 for k0 = c lg n/lg lg n. Conclude that Pk 0 < 1/n2 for k0 = c lg n/lg lg n.

e.     Argue that

Conclude that E [M] = O(lg n/1g 1g n).

12-4     Quadratic probing

Suppose that we are given a key k to search for in a hash table with positions 0, 1, . . . , m - 1, and suppose that we have a hash function h mapping the key space into the set {0, 1, . . . , m - 1}. The search scheme is as follows.

1.     Compute the value i   [image]  h(k), and set j   [image]  0.

2.     Probe in position i for the desired key k. If you find it, or if this position is empty, terminate the search.

3.     Set j   [image]  (j + l) mod m and i   [image]  (i + j) mod m, and return to step 2.

Assume that m is a power of 2.

a.     Show that this scheme is an instance of the general "quadratic probing" scheme by exhibiting the appropriate constants c1 and c2 for equation (12.5).

b.     Prove that this algorithm examines every table position in the worst case.

12-5     k-universal hashing

Let   [image]  be a class of hash functions in which each h maps the universe U of keys to {0, 1, . . . , m - 1}. We say that   [image] is k-universal if, for every fixed sequence of k distinct keys <x1, x2, . . . , xk> and for any h chosen at random from   [image], the sequence <h(xl), h(x2), . . . , h(xk)> is equally likely to be any of the mk sequences of length k with elements drawn from {0, 1, . . . , m -1}.

a.     Show that if   [image]  is 2-universal, then it is universal.

b.     Show that the class   [image]  defined in Section 12.3.3 is not 2-universal.

c.     Show that if we modify the definition of   [image]  in Section 12.3.3 so that each function also contains a constant term b, that is, if we replace h(x) with

ha, b(x)=a    x + b,

then   [image]  is 2-universal.

Chapter notes

Knuth [123] and Gonnet [90] are excellent references for the analysis of hashing algorithms. Knuth credits H. P. Luhn (1953) for inventing hash tables, along with the chaining method for resolving collisions. At about the same time, G. M. Amdahl originated the idea of open addressing.

Go to Chapter 13     Back to Table of Contents


Intro to Algorithms: CHAPTER 13: BINARY SEARCH TREES

 Return to Table of Contents

CHAPTER 13: BINARY SEARCH TREES

Search trees are data structures that support many dynamic-set operations, including SEARCH, MINIMUM, MAXIMUM, PREDECESSOR, SUCCESSOR, INSERT, and DELETE. Thus, a search tree can be used both as a dictionary and as a priority queue.

Basic operations on a binary search tree take time proportional to the height of the tree. For a complete binary tree with n nodes, such operations run in [image] (lg n) worst-case time. If the tree is a linear chain of n nodes, however, the same operations take [image] (n) worst-case time. We shall see in Section 13.4 that the height of a randomly built binary search tree is O(lg n), so that basic dynamic-set operations take [image] (lg n) time.

In practice, we can't always guarantee that binary search trees are built randomly, but there are variations of binary search trees whose worst-case performance on basic operations can be guaranteed to be good. Chapter 14 presents one such variation, red-black trees, which have height O(lg n). Chapter 19 introduces B-trees, which are particularly good for maintaining data bases on random-access, secondary (disk) storage.

After presenting the basic properties of binary search trees, the following sections show how to walk a binary search tree to print its values in sorted order, how to search for a value in a binary search tree, how to find the minimum or maximum element, how to find the predecessor or successor of an element, and how to insert into or delete from a binary search tree. The basic mathematical properties of trees were introduced in Chapter 5.

13.1 What is a binary search tree?

A binary search tree is organized, as the name suggests, in a binary tree, as shown in Figure 13.1. Such a tree can be represented by a linked data structure in which each node is an object. In addition to a key field, each node contains fields left, right, and p that point to the nodes corresponding to its left child, its right child, and its parent, respectively. If a child or the parent is missing, the appropriate field contains the value NIL. The root node is the only node in the tree whose parent field is NIL.

Figure 13.1 Binary search trees. For any node x, the keys in the left subtree of x are at most key[x], and the keys in the right subtree of x are at least key[x]. Different binary search trees can represent the same set of values. The worst-case running time for most search-tree operations is proportional to the height of the tree. (a) A binary search tree on 6 nodes with height 2. (b) A less efficient binary search tree with height 4 that contains the same keys.

The keys in a binary search tree are always stored in such a way as to satisfy the binary-search-tree property:

Let x be a node in a binary search tree. If y is a node in the left subtree of x, then key[y] [image] key[x]. If y is a node in the right subtree of x, then key[x] [image] key[y].

Thus, in Figure 13.1(a), the key of the root is 5, the keys 2, 3, and 5 in its left subtree are no larger than 5, and the keys 7 and 8 in its right subtree are no smaller than 5. The same property holds for every node in the tree. For example, the key 3 in Figure 13.1(a) is no smaller than the key 2 in its left subtree and no larger than the key 5 in its right subtree.

The binary-search-tree property allows us to print out all the keys in a binary search tree in sorted order by a simple recursive algorithm, called an inorder tree walk. This algorithm derives its name from the fact that the key of the root of a subtree is printed between the values in its left subtree and those in its right subtree. (Similarly, a preorder tree walk prints the root before the values in either subtree, and a postorder tree walk prints the root after the values in its subtrees.) To use the following procedure to print all the elements in a binary search tree T, we call INORDER-TREE-WALK( root[T]).

INORDER-TREE-WALK(x)

1  if x    NIL

2      then INORDER-TREE-WALK (left[x])

3           print key[x]

4           INORDER-TREE-WALK (right[x])

As an example, the inorder tree walk prints the keys in each of the two binary search trees from Figure 13.1 in the order 2, 3, 5, 5, 7, 8. The correctness of the algorithm follows by induction directly from the binary-search-tree property. It takes [image] (n) time to walk an n-node binary search tree, since after the initial call, the procedure is called recursively exactly twice for each node in the tree--once for its left child and once for its right child.

Exercises

13.1-1

Draw binary search trees of height 2, 3, 4, 5, and 6 on the set of keys {1, 4, 5, 10, 16, 17, 21}.

13.1-2

What is the difference between the binary-search-tree property and the heap property (7.1)? Can the heap property be used to print out the keys of an n-node tree in sorted order in O(n) time? Explain how or why not.

13.1-3

Give a nonrecursive algorithm that performs an inorder tree walk. (Hint: There is an easy solution that uses a stack as an auxiliary data structure and a more complicated but elegant solution that uses no stack but assumes that two pointers can be tested for equality.)

13.1-4

Give recursive algorithms that perform preorder and postorder tree walks in [image] (n) time on a tree of n nodes.

13.1-5

Argue that since sorting n elements takes [image] (n lg n) time in the worst case in the comparison model, any comparison-based algorithm for constructing a binary search tree from an arbitrary list of n elements takes [image] (n lg n) time in the worst case.

13.2 Querying a binary search tree

The most common operation performed on a binary search tree is searching for a key stored in the tree. Besides the SEARCH operation, binary search trees can support such queries as MINIMUM, MAXIMUM, SUCCESSOR, and PREDECESSOR. In this section, we shall examine these operations and show that each can be supported in time O(h) on a binary search tree of height h.

Figure 13.2 Queries on a binary search tree. To search for the key 13 in the tree, the path 15 6 7 13 is followed from the root. The minimum key in the tree is 2, which can be found by following left pointers from the root. The maximum key 20 is found by following right pointers from the root. The successor of the node with key 15 is the node with key 17, since it is the minimum key in the right subtree of 15. The node with key 13 has no right subtree, and thus its successor is its lowest ancestor whose left child is also an ancestor. In this case, the node with key 15 is its successor.

Searching

We use the following procedure to search for a node with a given key in a binary search tree. Given a pointer to the root of the tree and a key k, TREE-SEARCH returns a pointer to a node with key k if one exists; otherwise, it returns NIL.

TREE-SEARCH (x, k)

1 if x = NIL or k = key[x]

2     then return x

3  if k < key[x]

4     then return TREE-SEARCH (left[x], k)

5     else return TREE-SEARCH (right[x], k)

The procedure begins its search at the root and traces a path downward in the tree, as shown in Figure 13.2. For each node x it encounters, it compares the key k with key[x]. If the two keys are equal, the search terminates. If k is smaller than key[x], the search continues in the left subtree of x, since the binary-search-tree property implies that k could not be stored in the right subtree. Symmetrically, if k is larger than key[k], the search continues in the right subtree. The nodes encountered during the recursion form a path downward from the root of the tree, and thus the running time of TREE-SEARCH is O(h), where h is the height of the tree.

The same procedure can be written iteratively by "unrolling" the recursion into a while loop. On most computers, this version is more efficient.

ITERATIVE-TREE-SEARCH (x,k)

1  while x    NIL and k    key[x]

2      do if k < key[x]

3            then x    left[x]

4            else x    right[x]

5  return x

Minimum and maximum

An element in a binary search tree whose key is a minimum can always be found by following left child pointers from the root until a NIL is encountered, as shown in Figure 13.2. The following procedure returns a pointer to the minimum element in the subtree rooted at a given node x.

TREE-MINIMUM (x)

1  while left[x]    NIL

2      do x    left[x]

3  return x

The binary-search-tree property guarantees that TREE-MINIMUM is correct. If a node x has no left subtree, then since every key in the right subtree of x is at least as large as key[x], the minimum key in the subtree rooted at x is key[x]. If node x has a left subtree, then since no key in the right subtree is smaller than key[x] and every key in the left subtree is not larger than key[x], the minimum key in the subtree rooted at x can be found in the subtree rooted at left[x].

The pseudocode for TREE-MAXIMUM is symmetric.

TREE-MAXIMUM (x)

1  while right[x]    NIL

2      do x    right[x]

3  return x

Both of these procedures run in O(h) time on a tree of height h, since they trace paths downward in the tree.

Successor and predecessor

Given a node in a binary search tree, it is sometimes important to be able to find its successor in the sorted order determined by an inorder tree walk. If all keys are distinct, the successor of a node x is the node with the smallest key greater than key[x]. The structure of a binary search tree allows us to determine the successor of a node without ever comparing keys. The following procedure returns the successor of a node x in a binary search tree if it exists, and NIL if x has the largest key in the tree.

TREE SUCCESSOR(x)

1  if right[x]    NIL

2      then return TREE-MINIMUM(right[x])

3  y    p[x]

4  while y    NIL and x = right[y]

5       do x    y

6          y    p[y]

7  return y

The code for TREE-SUCCESSOR is broken into two cases. If the right subtree of node x is nonempty, then the successor of x is just the left-most node in the right subtree, which is found in line 2 by calling TREE-MINIMUM(right[x]). For example, the successor of the node with key 15 in Figure 13.2 is the node with key 17.

On the other hand, if the right subtree of node x is empty and x has a successor y, then y is the lowest ancestor of x whose left child is also an ancestor of x. In Figure 13.2, the successor of the node with key 13 is the node with key 15. To find y, we simply go up the tree from x until we encounter a node that is the left child of its parent; this is accomplished by lines 3-7 of TREE-SUCCESSOR.

The running time of TREE-SUCCESSOR on a tree of height his O(h), since we either follow a path up the tree or follow a path down the tree. The procedure TREE-PREDECESSOR, which is symmetric to TREE-SUCCESSOR, also runs in time O(h).

In summary, we have proved the following theorem.

Theorem 13.1

The dynamic-set operations SEARCH, MINIMUM, MAXIMUM, SUCCESSOR, and PREDECESSOR can be made to run in O(h) time on a binary search tree of height h.

Exercises

13.2-1

Suppose that we have numbers between 1 and 1000 in a binary search tree and want to search for the number 363. Which of the following sequences could not be the sequence of nodes examined?

a. 2, 252, 401, 398, 330, 344, 397, 363.

b. 924, 220, 911, 244, 898, 258, 362, 363.

c. 925, 202, 911, 240, 912, 245, 363.

d. 2, 399, 387, 219, 266, 382, 381, 278, 363.

e. 935, 278, 347, 621, 299, 392, 358, 363.

13.2-2

Professor Bunyan thinks he has discovered a remarkable property of binary search trees. Suppose that the search for key k in a binary search tree ends up in a leaf. Consider three sets: A, the keys to the left of the search path; B, the keys on the search path; and C, the keys to the right of the search path. Professor Bunyan claims that any three keys a [image] A, b [image] B, and c [image] C must satisfy a [image] b [image] c. Give a smallest possible counterexample to the professor's claim.

13.2-3

Use the binary-search-tree property to prove rigorously that the code for TREE-SUCCESSOR is correct.

13.2-4

An inorder tree walk of an n-node binary search tree can be implemented by finding the minimum element in the tree with TREE-MINIMUM and then making n - 1 calls to TREE-SUCCESSOR. Prove that this algorithm runs in [image] (n) time.

13.2-5

Prove that no matter what node we start at in a height-h binary search tree, k successive calls to TREE-SUCCESSOR take O(k + h) time.

13.2-6

Let T be a binary search tree, let x be a leaf node, and let y be its parent. Show that key[y] is either the smallest key in T larger than key[x] or the largest key in the tree smaller than key[x].

13.3 Insertion and deletion

The operations of insertion and deletion cause the dynamic set represented by a binary search tree to change. The data structure must be modified to reflect this change, but in such a way that the binary-search-tree property continues to hold. As we shall see, modifying the tree to insert a new element is relatively straightforward, but handling deletion is somewhat more intricate.

Insertion

To insert a new value v into a binary search tree T, we use the procedure TREE-INSERT. The procedure is passed a node z for which key[z] = v, left[z] = NIL, and right[z] = NIL. It modifies T and some of the fields of z in such a way that z is inserted into an appropriate position in the tree.

TREE-INSERT(T,z)

1  y    NIL

2  x    root[T]

3  while x    NIL

4       do y    x

5          if key[z] < key[x]

6             then x    left[x]

7             else x    right[x]

8  p[z]    y

9  if y = NIL

10      then root[T]    z

11      else if key[z] < key[y]

12              then left[y]    z

13              else right[y]    z

Figure 13.3 shows how TREE-INSERT works. Like the procedures TREE-SEARCH and ITERATIVE-TREE-SEARCH, TREE-INSERT begins at the root of the tree and traces a path downward. The pointer x traces the path, and the pointer y is maintained as the parent of x. After initialization, the while loop in lines 3-7 causes these two pointers to move down the tree, going left or right depending on the comparison of key[z] with key[x], until x is set to NIL. This NIL occupies the position where we wish to place the input item z. Lines 8-13 set the pointers that cause z to be inserted.

Like the other primitive operations on search trees, the procedure TREE-INSERT runs in O(h) time on a tree of height h.

Deletion

The procedure for deleting a given node z from a binary search tree takes as an argument a pointer to z. The procedure considers the three cases shown in Figure 13.4. If z has no children, we modify its parent p[z] to replace z with NIL as its child. If the node has only a single child, we "splice out" z by making a new link between its child and its parent. Finally, if the node has two children, we splice out z's successor y, which has no left child (see Exercise 13.3-4) and replace the contents of z with the contents of y.

The code for TREE-DELETE organizes these three cases a little differently.

Figure 13.3 Inserting an item with key 13 into a binary search tree. Lightly shaded nodes indicate the path from the root down to the position where the item is inserted. The dashed line indicates the link in the tree that is added to insert the item.

Figure 13.4 Deleting a node z from a binary search tree. In each case, the node actually removed is lightly shaded. (a) If z has no children, we just remove it. (b) If z has only one child, we splice out z. (c) If z has two children, we splice out its successor y, which has at most one child, and then replace the contents of z with the contents of y.

TREE-DELETE(T, z)

1  if left[z] = NIL or right[z] = NIL

2      then y    z

3      else y    TREE-SUCCESSOR(z)

4  if left[y]    NIL

5      then x    left[y]

6      else x    right[y]

7  if x    NIL

8      then p[x]    p[y]

9  if p[y] = NIL

10      then root[T]    x

11      else if y = left[p[y]]

12              then left[p[y]]    x

13              else right[p[y]]    x

14  if y    z

15      then key[z]    key[y]

16              If y has other fields, copy them, too.

17  return y

In lines 1-3, the algorithm determines a node y to splice out. The node y is either the input node z (if z has at most 1 child) or the successor of z (if z has two children). Then, in lines 4-6, x is set to the non-NIL child of y, or to NIL if y has no children. The node y is spliced out in lines 7-13 by modifying pointers in p[y] and x. Splicing out y is somewhat complicated by the need for proper handling of the boundary conditions, which occur when x = NIL or when y is the root. Finally, in lines 14-16, if the successor of z was the node spliced out, the contents of z are moved from y to z, overwriting the previous contents. The node y is returned in line 17 so that the calling procedure can recycle it via the free list. The procedure runs in O(h) time on a tree of height h.

In summary, we have proved the following theorem.

Theorem 13.2

The dynamic-set operations INSERT and DELETE can be made to run in O(h) time on a binary search tree of height h.

Exercises

13.3-1

Give a recursive version of the TREE-INSERT procedure.

13.3-2

Suppose that a binary search tree is constructed by repeatedly inserting distinct values into the tree. Argue that the number of nodes examined in searching for a value in the tree is one plus the number of nodes examined when the value was first inserted into the tree.

13.3-3

We can sort a given set of n numbers by first building a binary search tree containing these numbers (using TREE-INSERT repeatedly to insert the numbers one by one) and then printing the numbers by an inorder tree walk. What are the worst-case and best-case running times for this sorting algorithm?

13.3-4

Show that if a node in a binary search tree has two children, then its successor has no left child and its predecessor has no right child.

13.3-5

Suppose that another data structure contains a pointer to a node y in a binary search tree, and suppose that y's predecessor z is deleted from the tree by the procedure TREE-DELETE. What problem can arise? How can TREE-DELETE be rewritten to solve this problem?

13.3-6

Is the operation of deletion "commutative" in the sense that deleting x and then y from a binary search tree leaves the same tree as deleting y and then x? Argue why it is or give a counterexample.

13.3-7

When node z in TREE-DELETE has two children, we could splice out its predecessor rather than its successor. Some have argued that a fair strategy, giving equal priority to predecessor and successor, yields better empirical performance. How might TREE-DELETE be changed to implement such a fair strategy?

* 13.4 Randomly built binary search trees

We have shown that all the basic operations on a binary search tree run in O(h) time, where h is the height of the tree. The height of a binary search tree varies, however, as items are inserted and deleted. In order to analyze the behavior of binary search trees in practice, it is reasonable to make statistical assumptions about the distribution of keys and the sequence of insertions and deletions.

Unfortunately, little is known about the average height of a binary search tree when both insertion and deletion are used to create it. When the tree is created by insertion alone, the analysis becomes more tractable. Let us therefore define a randomly built binary search tree on n distinct keys as one that arises from inserting the keys in random order into an initially empty tree, where each of the n! permutations of the input keys is equally likely. (Exercise 13.4-2 asks you to show that this notion is different from assuming that every binary search tree on n keys is equally likely.) The goal of this section is to show that the expected height of a randomly built binary search tree on n keys is O(lg n).

We begin by investigating the structure of binary search trees that are built by insertion alone.

Lemma 13.3

Let T be the tree that results from inserting n distinct keys k1, k2, . . . , kn (in order) into an initially empty binary search tree. Then ki is an ancestor of kj in T, for l [image] i < j [image] n, if and only if

ki = min {kl : 1    l    i and kl > kj}

or

ki = max {kl: 1    l    i and kl < kj} .

Proof [image]: Suppose that ki is an ancestor of kj. Consider the tree Ti that results after the keys k1, k2, . . . , ki have been inserted. The path in Ti from the root to ki is the same as the path in T from the root to ki. Thus, if kj were inserted into Ti, it would become either the left or the right child of ki. Consequently (see Exercise 13.2-6), ki is either the smallest key among k1, k2, . . . , ki that is larger than kj or the largest key among k1, k2, . . . , ki that is smaller than kj.

: Suppose that ki is the smallest key among k1, k2, . . . , ki that is larger than kj. (The case in which ki is the largest key among k1, k2, . . . , ki that is smaller than kj is handled symmetrically.) Comparing kj to any of the keys on the path in T from the root to ki yields the same results as comparing ki to the keys. Hence, when kj is inserted, it follows a path through ki and is inserted as a descendant of ki.

As a corollary of Lemma 13.3, we can precisely characterize the depth of a key based on the input permutation.

Corollary 13.4

Let T be the tree that results from inserting n distinct keys k1, k2, . . . , kn (in order) into an initially empty binary search tree. For a given key kj, where 1 [image] j [image] n, define

Gj = {ki : 1    i < j and kl > ki > kj for all l < i such that kl > kj}

and

Lj = {ki : 1    i < j and kl < ki < kj for all l < i such that kl < kj} .

Then the keys on the path from the root to kj are exactly the keys in Gj [image] Lj, and the depth in T of any key kj is

d(kj, T) = |Gj| + |Lj| .

Figure 13.5 illustrates the two sets Gj and Lj. The set Gj contains any key ki inserted before kj such that ki is the smallest key among k1, k2, . . . , ki that is larger than kj. (The structure of Lj is symmetric.) To better understand the set Gj, let us explore a method by which we can enumerate its elements. Among the keys k1, k2, . . . , kj-1, consider in order those that are larger than kj. These keys are shown as G'j, in the figure. As each key is considered in turn, keep a running account of the minimum. The set Gj consists of those elements that update the running minimum.

Let us simplify this scenario somewhat for the purpose of analysis. Suppose that n distinct numbers are inserted one at a time into a dynamic set. If all permutations of the numbers are equally likely, how many times on average does the minimum of the set change? To answer this question, suppose that the ith number inserted is ki, for i = 1, 2, . . . , n. The probability is l/i that ki is the minimum of the first i numbers, since the rank of ki among the first i numbers is equally likely to be any of the i possible ranks. Consequently, the expected number of changes to the minimum of the set is

where Hn = ln n + O(1) is the nth harmonic number (see equation (3.5) and Problem 6-2).

We therefore expect the number of changes to the minimum to be approximately ln n, and the following lemma shows that the probability that it is much greater is very small.

Lemma 13.5

Let k1, k2, . . . , kn be a random permutation of n distinct numbers, and let |S| be the random variable that is the cardinality of the set

S = {ki : 1    i    n and kl > ki for all l < i} .

(13.1)

Then Pr{|S| [image] ( [image] + 1)Hn} [image] 1/n2, where Hn is the nth harmonic number and [image] [image] 4.32 satisfies the equation (ln [image] - 1) [image] = 2.

Proof We can view the cardinality of the set S as being determined by n Bernoulli trials, where a success occurs in the ith trial when ki is smaller than the elements k1, k2, . . . , ki-l. Success in the ith trial occurs with probability 1/i. The trials are independent, since the probability that ki is the minimum of k1, k2, . . . , ki is independent of the relative ordering of k1, k2, . . . , ki-1.

We can use Theorem 6.6 to bound the probability that |S| [image] ( [image] + 1 )Hn . The expectation of |S| is [image] = Hn [image] ln n. Since [image] > 1, Theorem 6.6 yields

Figure 13.5 Illustrating the two sets Gj and Lj that comprise the keys on a path from the root of a binary search tree to a key kj = 17. (a) The nodes with keys in Gj are black, and the nodes with keys in Lj are white. All other nodes are shaded. The path from the root down to the node with key kj is shaded. Keys to the left of the dashed line are less than kj, and keys to the right are greater. The tree is constructed by inserting the keys shown in the topmost list in (b). The set G'j = {21, 25, 19, 29}consists of those elements that are inserted before 17 and are greater than 17. The set Gj = {21, 19} consists of those elements that update a running minimum of the elements in G'j. Thus, the key 21 is in Gj, since it is the first element. The key 25 is not in Gj, since it is larger than the running minimum 21. The key 19 is in Gj, since it is smaller than the running minimum 21. The key 29 is not in Gj, since it is larger than the running minimum 19. The structures of L'j and Lj are symmetric.

which follows from the definition of [image] .

We now have the tools to bound the height of a randomly built binary search tree.

Theorem 13.6

The average height of a randomly built binary search tree on n distinct keys is O(lg n).

Proof Let k1, k2, . . . , kn be a random permutation on the n keys, and let T be the binary search tree that results from inserting the keys in order into an initially empty tree. We first consider the probability that the depth d(kj, T) of a given key kj is at least t, for an arbitrary value t. By the characterization of d (kj, T) in Corollary 13.4, if the depth of kj is at least t, then the cardinality of one of the two sets Gj and Lj must be at least t/2. Thus,

Pr{d(kj, T)    t}    Pr{|Gj|    t/2} + Pr{|Lj|    t/2} .

(13.2)

Let us examine Pr{|Gj| [image] t/2} first. We have

Pr{|Gj|    t/2}

= Pr{|{ki: 1    i < j and kl > ki > kj for all l < i}|    t/2}

   Pr{|{ki: i    n and ki > ki for all l > i}|    t/2}

= Pr{|S|    t/2} ,

where S is defined as in equation (13.1). To justify this argument, note that the probability does not decrease if we extend the range of i from i < j to i [image] n, since more elements are added to the set. Likewise, the probability does not decrease if we remove the condition that ki > kj, since we are substituting a random permutation on possibly fewer than n elements (those ki that are greater than kj) for a random permutation on n elements.

Using a symmetric argument, we can prove that

Pr{|Lj|    t/2}    Pr{|S|    t/2},

and thus, by inequality (13.2), we obtain

Pr {d (kj, T)    t}    2 Pr{|S|    t/2} .

If we choose t = 2( [image] + 1)Hn, where Hn is the nth harmonic number and [image] [image] 4.32 satisfies (1n [image] - l) [image] = 2, we can apply Lemma 13.5 to conclude that

Pr{d(kj, T)    2(   + 1)Hn}      2Pr{|S|    (   + 1)Hn}

    2/n2 .

Since there are at most n nodes in a randomly built binary search tree, the probability that any node's depth is at least 2( [image] + 1 )Hn is therefore, by Boole's inequality (6.22), at most n(2/n2) = 2/n. Thus, at least 1 - 2/n of the time, the height of a randomly built binary search tree is less than 2( [image] + 1)Hn, and at most 2/n of the time, it is at most n. The expected height is therefore at most (2( [image] + 1)Hn)(l - 2/n) + n(2/n) = O(lg n).

Exercises

13.4-1

Describe a binary search tree on n nodes such that the average depth of a node in the tree is [image] (lg n) but the height of the tree is w(lg n). How large can the height of an n-node binary search tree be if the average depth of a node is [image] (lg n)?

13.4-2

Show that the notion of a randomly chosen binary search tree on n keys, where each binary search tree of n keys is equally likely to be chosen, is different from the notion of a randomly built binary search tree given in this section. (Hint: List the possibilities when n = 3.)

13.4-3

Given a constant r [image] 1, determine t such that the probability is less than 1 /nr that the height of a randomly built binary search tree is at least tHn.

13.4-4

Consider RANDOMIZED-QUICKSORT operating on a sequence of n input numbers. Prove that for any constant k > 0, all but O(l/nk) of the n! input permutations yield an O(n1g n) running time.

Problems

13-1 Binary search trees with equal keys

Equal keys pose a problem for the implementation of binary search trees.

a. What is the asymptotic performance of TREE-INSERT when used to insert n items with identical keys into an initially empty binary search tree?

We propose to improve TREE-INSERT by testing before line 5 whether or not key[z] = key[x] and by testing before line 11 whether or not key[z] = key[y]. If equality holds, we implement one of the following strategies. For each strategy, find the asymptotic performance of inserting n items with identical keys into an initially empty binary search tree. (The strategies are described for line 5, in which we compare the keys of z and x. Substitute y for x to arrive at the strategies for line 11.)

b. Keep a Boolean flag b[x] at node x, and set x to either left[x] or right[x]based on the value of b[x], which alternates between FALSE and TRUE each time the node is visited during TREE-INSERT.

c. Keep a list of nodes with equal keys at x, and insert z into the list.

d. Randomly set x to either left[x] or right[x]. (Give the worst-case performance and informally derive the average-case performance.)

13-2 Radix trees

Given two strings a = a0a1. . .ap and b = b0b1. . .bq, where each ai and each bj is in some ordered set of characters, we say that string a is lexicographically less than string b if either

1. there exists an integer j, 0 [image] j [image] min(p, q), such that ai = bi for all i = 0, 1, . . . , j - 1 and aj < bj, or

2. p<q and ai = bi for all i = 0, l, . . . , p.

For example, if a and b are bit strings, then 10100 < 10110 by rule 1(letting j = 3) and 10100 < 101000 by rule 2. This is similar to the ordering used in English-language dictionaries.

The radix tree data structure shown in Figure 13.6 stores the bit strings 1011, 10, 011, 100, and 0. When searching for a key a = a0a1 . . . ap, we go left at a node of depth i if ai = 0 and right if ai = 1. Let S be a set of distinct binary strings whose lengths sum to n. Show how to use a radix tree to sort S lexicographically in [image] (n) time. For the example in Figure 13.6, the output of the sort should be the sequence 0, 011, 10, 100, 1011.

13-3 Average node depth in a randomly built binary search tree

In this problem, we prove that the average depth of a node in a randomly built binary search tree with n nodes is O(lg n). Although this result is weaker than that of Theorem 13.6, the technique we shall use reveals a surprising similarity between the building of a binary search tree and the running of RANDOMIZED-QUICKSORT from Section 8.3.

We start by recalling from Chapter 5 that the internal path length P(T)of a binary tree T is the sum, over all nodes x in T, of the depth of node x, which we denote by d (x, T).

Figure 13.6 A radix tree storing the bit strings l011, 10, 011, 100, and 0. Each node's key can be determined by traversing the path from the root to that node. There is no need, therefore, to store the keys in the nodes; the keys are shown here for illustrative purposes only. Nodes are heavily shaded if the keys corresponding to them are not in the tree; such nodes are present only to establish a path to other nodes.

a. Argue that the average depth of a node in T is

Thus, we wish to show that the expected value of P(T) is O(n 1g n).

b. Let TL and TR denote the left and right subtrees of tree T, respectively. Argue that if T has n nodes, then

P(T) = P(TL) + P(TR) + n - 1 .

c. Let P(n) denote the average internal path length of a randomly built binary search tree with n nodes. Show that

d. Show that P(n) can be rewritten as

e. Recalling the analysis of the randomized version of quicksort, conclude that P(n) = O(n lg n).

At each recursive invocation of quicksort, we choose a random pivot element to partition the set of elements being sorted. Each node of a binary search tree partitions the set of elements that fall into the subtree rooted at that node.

f. Describe an implementation of quicksort in which the comparisons to sort a set of elements are exactly the same as the comparisons to insert the elements into a binary search tree. (The order in which comparisons are made may differ, but the same comparisons must be made.)

13-4 Number of different binary trees

Let bn denote the number of different binary trees with n nodes. In this problem, you will find a formula for bn as well as an asymptotic estimate.

a. Show that b0 = 1 and that, for n [image] 1,

b Let B(x) be the generating function

(see Problem 4-6 for the definition of generating functions). Show that B(x) = xB(x)2 + 1 and hence

The Taylor expansion of f(x) around the point x = a is given by

where f(k) (x) is the kth derivative of f evaluated at x.

c. Show that

(the nth Catalan number) by using the Taylor expansion of [image] around x = 0. (If you wish, instead of using the Taylor expansion, you may use the generalization of the binomial expansion (6.5) to non-integral exponents n, where for any real number n and integer k, we interpret [image] to be n(n - 1) . . . (n - k + 1)/k! if k [image] 0, and 0 otherwise.)

d. Show that

Chapter notes

Knuth [123] contains a good discussion of simple binary search trees as well as many variations. Binary search trees seem to have been independently discovered by a number of people in the late 1950's.

Go to Chapter 14 Back to Table of Contents


Intro to Algorithms: CHAPTER 14: RED-BLACK TREES

 Return to Table of Contents

CHAPTER 14: RED-BLACK TREES

Chapter 13 showed that a binary search tree of height h can implement any of the basic dynamic-set operations--such as SEARCH, PREDECESSOR, SUCCESSOR, MINIMUM, MAXIMUM, INSERT, and DELETE--in O(h) time. Thus, the set operations are fast if the height of the search tree is small; but if its height is large, their performance may be no better than with a linked list. Red-black trees are one of many search-tree schemes that are "balance" in order to guarantee that basic dynamic-set operations take O(lg n) time in the worst case.

14.1 Properties of red-black trees

A red-black tree is a binary search tree with one extra bit of storage per node: its color, which can be either RED or BLACK. By constraining the way nodes can be colored on any path from the root to a leaf, red-black trees ensure that no such path is more than twice as long as any other, so that the tree is approximately balanced.

Each node of the tree now contains the fields color, key, left, right, and p. If a child or the parent of a node does not exist, the corresponding pointer field of the node contains the value NIL. We shall regard these NIL'S as being pointers to external nodes (leaves) of the binary search tree and the normal, key-bearing nodes as being internal nodes of the tree.

A binary search tree is a red-black tree if it satisfies the following red-black properties:

1. Every node is either red or black.

2. Every leaf (NIL) is black.

3. If a node is red, then both its children are black.

4. Every simple path from a node to a descendant leaf contains the same number of black nodes.

An example of a red-black tree is shown in Figure 14.1.

We call the number of black nodes on any path from, but not including, a node x to a leaf the black-height of the node, denoted bh(x). By property 4, the notion of black-height is well defined, since all descending paths from the node have the same number of black nodes. We define the black-height of a red-black tree to be the black-height of its root.

Figure 14.1 A red-black tree with black nodes darkened and red nodes shaded. Every node in a red-black tree is either red or black, every leaf (NIL) is black, the children of a red node are both black, and every simple path from a node to a descendant leaf contains the same number of black nodes. Each non-NIL node is marked with its black-height; NIL'S have black-height 0.

The following lemma shows why red-black trees make good search trees.

Lemma 14.1

A red-black tree with n internal nodes has height at most 21g(n + 1).

Proof We first show that the subtree rooted at any node x contains at least 2bh(x) -1 internal nodes. We prove this claim by induction on the height of x. If the height of x is 0, then x must be a leaf (NIL), and the subtree rooted at x indeed contains at least 2bh(x)-1 = 20 - 1 = 0 internal nodes. For the inductive step, consider a node x that has positive height and is an internal node with two children. Each child has a black-height of either bh(x) or bh(x) - 1, depending on whether its color is red or black, respectively. Since the height of a child of x is less than the height of x itself, we can apply the inductive hypothesis to conclude that each child has at least 2bh(x) -1 -1 internal nodes. Thus, the subtree rooted at x contains at least (2bh(x)-1 -1) + (2bh(x)-1 -1) + 1 = 2bh(x)- 1 internal nodes, which proves the claim.

To complete the proof of the lemma, let h be the height of the tree. According to property 3, at least half the nodes on any simple path from the root to a leaf, not including the root, must be black. Consequently, the black-height of the root must be at least h/2; thus,

n    2h/2 - 1.

Moving the 1 to the left-hand side and taking logarithms on both sides yields lg (n + 1) [image] h/2, or h [image] 21g (n + 1).

An immediate consequence of this lemma is that the dynamic-set operations SEARCH, MINIMUM, MAXIMUM, SUCCESSOR, and PREDECESSOR can be implemented in O(lg n) time on red-black trees, since they can be made to run in O(h) time on a search tree of height h (as shown in Chapter 13) and any red-black tree on n nodes is a search tree with height O(lg n). Although the algorithms TREE-INSERT and TREE-DELETE from Chapter 13 run in O(lg n) time when given a red-black tree as input, they do not directly support the dynamic-set operations INSERT and DELETE, since they do not guarantee that the modified binary search tree will be a red-black tree. We shall see in Sections 14.3 and 14.4, however, that these two operations can indeed be supported in O(lg n) time.

Exercises

14.1-1

Draw the complete binary search tree of height 3 on the keys {1, 2, . . . , 15}. Add the NIL leaves and color the nodes in three different ways such that the black-heights of the resulting red-black trees are 2, 3, and 4.

14.1-2

Suppose that the root of a red-black tree is red. If we make it black, does the tree remain a red-black tree?

14.1-3

Show that the longest simple path from a node x in a red-black tree to a descendant leaf has length at most twice that of the shortest simple path from node x to a descendant leaf.

14.1-4

What is the largest possible number of internal nodes in a red-black tree with black-height k? What is the smallest possible number?

14.1-5

Describe a red-black tree on n keys that realizes the largest possible ratio of red internal nodes to black internal nodes. What is this ratio? What tree has the smallest possible ratio, and what is the ratio?

14.2 Rotations

The search-tree operations TREE-INSERT and TREE-DELETE, when run on a red-black tree with n keys, take O(lg n) time. Because they modify the tree, the result may violate the red-black properties enumerated in Section 14.1. To restore these properties, we must change the colors of some of the nodes in the tree and also change the pointer structure.

Figure 14.2 The rotation operations on a binary search tree. The operation RIGHT-ROTATE(T,x) transforms the configuration of the two nodes on the left into the configuration on the right by changing a constant number of pointers. The configuration on the right can be transformed into the configuration on the left by the inverse operation LEFT-ROTATE(T,y). The two nodes might occur anywhere in a binary search tree. The letters , , and represent arbitrary subtrees. A rotation operation preserves the inorder ordering of keys: the keys in precede key[x], which precedes the keys in , which precede key[y], which precedes the keys in .

We change the pointer structure through rotation, which is a local operation in a search tree that preserves the inorder key ordering. Figure 14.2 shows the two kinds of rotations: left rotations and right rotations. When we do a left rotation on a node x, we assume that its right child y is non-NIL. The left rotation "pivots" around the link from x to y. It makes y the new root of the subtree, with x as y's left child and y's left child as x's right child.

The pseudocode for LEFT-ROTATE assumes that right[x] [image] NIL.

Figure 14.3 shows how LEFT-ROTATE operates. The code for RIGHT-ROTATE is similar. Both LEFT-ROTATE and RIGHT-ROTATE run in O(l) time. Only pointers are changed by a rotation; all other fields in a node remain the same.

Figure 14.3 An example of how the procedure LEFT-ROTATE(T,x) modifies a binary search tree. The NIL leaves are omitted. Inorder tree walks of the input tree and the modified tree produce the same listing of key values.

Exercises

14.2-1

Draw the red-black tree that results after TREE-INSERT is called on the tree in Figure 14.1 with key 36. If the inserted node is colored red, is the resulting tree a red-black tree? What if it is colored black?

14.2-2

Write pseudocode for RIGHT-ROTATE.

14.2-3

Argue that rotation preserves the inorder key ordering of a binary tree.

14.2-4

Let a, b, and c be arbitrary nodes in subtrees [image] , [image] , and [image] , respectively, in the left tree of Figure 14.2. How do the depths of a, b, and c change when a left rotation is performed on node x in the figure?

14.2-5

Show that any arbitrary n-node tree can be transformed into any other arbitrary n-node tree using O(n) rotations. (Hint: First show that at most n -1 right rotations suffice to transform any tree into a right-going chain.)

14.3 Insertion

Insertion of a node into an n-node red-black tree can be accomplished in O(lg n) time. We use the TREE-INSERT procedure (Section 13.3) to insert node x into the tree T as if it were an ordinary binary search tree, and then we color x red. To guarantee that the red-black properties are preserved, we then fix up the modified tree by recoloring nodes and performing rotations. Most of the code for RB-INSERT handles the various cases that can arise as we fix up the modified tree.

The code for RB-INSERT is less imposing than it looks. We shall break our examination of the code into three major steps. First, we shall determine what violations of the red-black properties are introduced in lines 1-2 when the node x is inserted and colored red. Second, we shall examine the overall goal of the while loop in lines 3-17. Finally, we shall explore each of the three cases into which the while loop is broken and see how they accomplish the goal. Figure 14.4 shows how RB-INSERT operates on a sample red-black tree.

Which of the red-black properties can be violated after lines 1-2? Property 1 certainly continues to hold, as does property 2, since the newly inserted red node has NIL'S for children. Property 4, which says that the number of blacks is the same on every path from a given node, is satisfied as well, because node x replaces a (black) NIL, and node x is red with NIL children. Thus, the only property that might be violated is property 3 which says that a red node cannot have a red child. Specifically, property 3 is violated if x's parent is red, since x is itself colored red in line 2. Figure 14.4(a) shows such a violation after the node x has been inserted.

Figure 14.4 The operation of RB-INSERT. (a) A node x after insertion. Since x and its parent p[x] are both red, a violation of property 3 occurs. Since x's uncle y is red, case 1 in the code can be applied. Nodes are recolored and the pointer x is moved up the tree, resulting in the tree shown in (b). Once again, x and its parent are both red, but x's uncle y is black. Since x is the right child of p[x], case 2 can be applied. A left rotation is performed, and the tree that results is shown in (c). Now x is the left child of its parent, and case 3 can be applied. A right rotation yields the tree in (d), which is a legal red-black tree.

The goal of the while loop in lines 3-17 is to move the one violation of property 3 up the tree while maintaining property 4 as an invariant. At the beginning of each iteration of the loop, x points to a red node with a red parent--the only violation in the tree. There are two possible outcomes of each iteration of the loop: the pointer x moves up the tree, or some rotations are performed and the loop terminates.

There are actually six cases to consider in the while loop, but three of them are symmetric to the other three, depending on whether x's parent p[x] is a left child or a right child of x's grandparent p[p[x]], which is determined in line 4. We have given the code only for the situation in which p[x] is a left child. We have made the important assumption that the root of the tree is black--a property we guarantee in line 18 each time we terminate--so that p[x] is not the root and p[p[x]] exists.

Case 1 is distinguished from cases 2 and 3 by the color of x's parent's sibling, or "uncle." Line 5 makes y point to x's uncle right [p[p[x]]], and a test is made in line 6. If y is red, then case 1 is executed. Otherwise, control passes to cases 2 and 3. In all three cases, x's grandparent p[p[x]] is black, since its parent p[x] is red, and property 3 is violated only between x and p[x].

The situation for case 1 (lines 7-10) is shown in Figure 14.5. Case 1 is executed when both p[x] and y are red. Since p[p[x]] is black, we can color both p[x] and y black, thereby fixing the problem of x and p[x] both being red, and color p[p[x]] red, thereby maintaining property 4. The only problem that might arise is that p[p[x]] might have a red parent; hence, we must repeat the while loop with p[p[x]] as the new node x.

In cases 2 and 3, the color of x's uncle y is black. The two cases are distinguished by whether x is a right or left child of p[x]. Lines 12-13 constitute case 2, which is shown in Figure 14.6 together with case 3. In case 2, node x is a right child of its parent. We immediately use a left rotation to transform the situation into case 3 (lines 14-16), in which node x is a left child. Because both x and p[x] are red, the rotation affects neither the black-height of nodes nor property 4. Whether we enter case 3 directly or through case 2, x's uncle y is black, since otherwise we would have executed case 1. We execute some color changes and a right rotation, which preserve property 4, and then, since we no longer have two red nodes in a row, we are done. The body of the while loop is not executed another time, since p[x] is now black.

What is the running time of RB-INSERT? Since the height of a red-black tree on n nodes is O(lg n), the call to TREE-INSERT takes O(lg n) time. The while loop only repeats if case 1 is executed, and then the pointer x moves up the tree. The total number of times the while loop can be executed is therefore O(lg n). Thus, RB-INSERT takes a total of O(lg n) time. Interestingly, it never performs more than two rotations, since the while loop terminates if case 2 or case 3 is executed.

Figure 14.5 Case 1 of the procedure RB-INSERT. Property 3 is violated, since x and its parent p[x] are both red. The same action is taken whether (a) x is a right child or (b) x is a left child. Each of the subtrees , , , and has a black root, and each has the same black-height. The code for case 1 changes the colors of some nodes, preserving property 4: all downward paths from a node to a leaf have the same number of blacks. The while loop continues with node x's grandparent p[p[x]] as the new x. Any violation of property 3 can now occur only between the new x, which is red, and its parent, if it is red as well.

Figure 14.6 Cases 2 and 3 of the procedure RB-INSERT. As in case 1, property 3 is violated in either case 2 or case 3 because x and its parent p[x] are both red. Each of the subtrees , , , and has a black root, and each has the same black-height. Case 2 is transformed into case 3 by a left rotation, which preserves property 4: all downward paths from a node to a leaf have the same number of blacks. Case 3 causes some color changes and a right rotation, which also preserve property 4. The while loop then terminates, because property 3 is satisfied: there are no longer two red nodes in a row.

Exercises

14.3-1

In line 2 of RB-INSERT, we set the color of the newly inserted node x to red. Notice that if we had chosen to set x's color to black, then property 3 of a red-black tree would not be violated. Why didn't we choose to set x's color to black?

14.3-2

In line 18 of RB-INSERT, we set the root's color to black. What is the advantage of doing so?

14.3-3

Show the red-black trees that result after successively inserting the keys 41, 38, 31, 12, 19, 8 into an initially empty red-black tree.

14.3-4

Suppose that the black-height of each of the subtrees [image], [image], [image], [image], [image] in Figures 14.5 and 14.6 is k. Label each node in each figure with its black-height to verify that property 4 is preserved by the indicated transformation.

14.3-5

Consider a red-black tree formed by inserting n nodes with RB-INSERT. Argue that if n > 1, the tree has at least one red node.

14.3-6

Suggest how to implement RB-INSERT efficiently if the representation for red-black trees includes no storage for parent pointers.

14.4 Deletion

Like the other basic operations on an n-node red-black tree, deletion of a node takes time O(lg n). Deleting a node from a red-black tree is only slightly more complicated than inserting a node.

In order to simplify boundary conditions in the code, we use a sentinel to represent NIL (see page 206). For a red-black tree T, the sentinel nil[T] is an object with the same fields as an ordinary node in the tree. Its color field is BLACK, and its other fields--p, left, right, and key--can be set to arbitrary values. In the red-black tree, all pointers to NIL are replaced by pointers to the sentinel nil[T].

We use sentinels so that we can treat a NIL child of a node x as an ordinary node whose parent is x. We could add a distinct sentinel node for each NIL in the tree, so that the parent of each NIL is well defined, but that would waste space. Instead, we use the one sentinel nil[T] to represent all the NIL'S. When we wish to manipulate a child of a node x, however, we must be careful to set p[nil[T]] to x first.

The procedure RB-DELETE is a minor modification of the TREE-DELETE procedure (Section 13.3). After splicing out a node, it calls an auxiliary procedure RB-DELETE-FIXUP that changes colors and performs rotations to restore the red-black properties.

RB-DELETE (T, z)

1 if left[z] = nil[T] or right[z] = nil[T]

2      then y    z

3      else y    TREE-SUCCESSOR(z)

4 if left[y]    nil[T]

5     then x    left[y]

6     else x    right[y]

7 p[x]    p[y]

8 if p[y] = nil[T]

9    then root[T]    x

10    else if y = left[p[y]]

11            then left[p[y]]    x

12            else right[p[y]]    x

13 if y    z

14     then key[z]    key[y]

15             If y has other fields, copy them, too.

16 if color[y] = BLACK

17     then RB-DELETE-FIXUP (T,x)

18 return y

There are three differences between the procedures TREE-DELETE and RB-DELETE. First, all references to NIL in TREE-DELETE have been replaced by references to the sentinel nil[T] in RB-DELETE. Second, the test for whether x is NIL in line 7 of TREE-DELETE has been removed, and the assignment p[x] [image] p[y] is performed unconditionally in line 7 of RB-DELETE. Thus, if x is the sentinel nil[T], its parent pointer points to the parent of the spliced-out node y. Third, a call to RB-DELETE-FIXUP is made in lines 16-17 if y is black. If y is red, the red-black properties still hold when y is spliced out, since no black-heights in the tree have changed and no red nodes have been made adjacent. The node x passed to RB-DELETE-FIXUP is the node that was y's sole child before y was spliced out if y had a non-NIL child, or the sentinel nil[T] if y had no children. In the latter case, the unconditional assignment in line 7 guarantees that x's parent is now the node that was previously y's parent, whether x is a key-bearing internal node or the sentinel nil[T].

We can now examine how the procedure RB-DELETE-FIXUP restores the red-black properties to the search tree.

If the spliced-out node y in RB-DELETE is black, its removal causes any path that previously contained node y to have one fewer black node. Thus, property 4 is now violated by any ancestor of y in the tree. We can correct this problem by thinking of node x as having an "extra" black. That is, if we add 1 to the count of black nodes on any path that contains x, then under this interpretation, property 4 holds. When we splice out the black node y, we "push" its blackness onto its child. The only problem is that now node x may be "doubly black," thereby violating property 1.

The procedure RB-DELETE-FIXUP attempts to restore property 1. The goal of the while loop in lines 1-22 is to move the extra black up the tree until (1) x points to a red node, in which case we color the node black in line 23, (2) x points to the root, in which case the extra black can be simply "removed," or (3) suitable rotations and recolorings can be performed.

Within the while loop, x always points to a nonroot black node that has the extra black. We determine in line 2 whether x is a left child or a right child of its parent p[x]. (We have given the code for the situation in which x is a left child; the situation in which x is a right child--line 22--is symmetric.) We maintain a pointer w to the sibling of x. Since node x is doubly black, node w cannot be nil[T]; otherwise, the number of blacks on the path from p[x] to the NIL leaf w would be smaller than the number on the path from p[x] to x.

The four cases in the code are illustrated in Figure 14.7. Before examining each case in detail, let's look more generally at how we can verify that the transformation in each of the cases preserves property 4. The key idea is that in each case the number of black nodes from (and including) the root of the subtree shown to each of the subtrees [image], [image], . . ., [image] is preserved by the transformation. For example, in Figure 14.7(a), which illustrates case 1, the number of black nodes from the root to either subtree [image] or [image] is 3, both before and after the transformation. (Remember, the pointer x adds an extra black.) Similarly, the number of black nodes from the root to any of [image], [image], [image], and [image] is 2, both before and after the transformation. In Figure 14.7(b), the counting must involve the color c, which can be either red or black. If we define count(RED) = 0 and count(BLACK) = 1, then the number of black nodes from the root to [image] is 2 + count(c), both before and after the transformation. The other cases can be verified similarly (Exercise 14.4-5).

Case 1 (lines 5-8 of RB-DELETE-FIXUP and Figure 14.7(a)) occurs when node w, the sibling of node x, is red. Since w must have black children, we can switch the colors of w and p[x] and then perform a left-rotation on p[x] without violating any of the red-black properties. The new sibling of x, one of w's children, is now black, and thus we have converted case 1 into case 2, 3, or 4.

Cases 2, 3, and 4 occur when node w is black; they are distinguished by the colors of w's children. In case 2 (lines 10-11 of RB-DELETE-FIXUP and Figure 14.7(b)), both of w's children are black. Since w is also black, we take one black off both x and w, leaving x with only one black and leaving w red, and add an extra black to p[x]. We then repeat the while loop with p[x] as the new node x. Observe that if we enter case 2 through case 1, the color c of the new node x is red, since the original p[x] was red, and thus the loop terminates when it tests the loop condition.

Case 3 (lines 13-16 and Figure 14.7(c)) occurs when w is black, its left child is red, and its right child is black. We can switch the colors of w and its left child left[w] and then perform a right rotation on w without violating any of the red-black properties. The new sibling w of x is now a black node with a red right child, and thus we have transformed case 3 into case 4.

Case 4 (lines 17-21 and Figure 14.7(d)) occurs when node x's sibling w is black and w's right child is red. By making some color changes and performing a left rotation on p[x], we can remove the extra black on x without violating any of the red-black properties. Setting x to be the root causes the while loop to terminate when it tests the loop condition.

Figure 14.7 The cases in the while loop of the procedure RB-DELETE. Darkened nodes are black, heavily shaded nodes are red, and lightly shaded nodes, which may be either red or black, are represented by c and c'. The letters , , . . ., represent arbitrary subtrees. In each case, the configuration on the left is transformed into the configuration on the right by changing some colors and/or performing a rotation. A node pointed to by x has an extra black. The only case that causes the loop to repeat is case 2. (a) Case I is transformed to case 2, 3, or 4 by exchanging the colors of nodes B and D and performing a left rotation. (b) In case 2, the extra black represented by the pointer x is moved up the tree by coloring node D red and setting x to point to node B. If we enter case 2 through case 1, the while loop terminates, since the color c is red. (c) Case 3 is transformed to case 4 by exchanging the colors of nodes C and D and performing a right rotation. (d) In case 4, the extra black represented by x can be removed by changing some colors and performing a left rotation (without violating the red-black properties), and the loop terminates.

What is the running time of RB-DELETE? Since the height of a red-black tree of n nodes is O(lg n), the total cost of the procedure without the call to RB-DELETE-FIXUP takes O(lg n) time. Within RB-DELETE-FIXUP, cases 1, 3, and 4 each terminate after performing a constant number of color changes and at most three rotations. Case 2 is the only case in which the while loop can be repeated, and then the pointer x moves up the tree at most O(lg n) times and no rotations are performed. Thus, the procedure RB-DELETE-FIXUP takes O(lg n) time and performs at most three rotations, and the overall time for RB-DELETE is therefore also O(lg n).

Exercises

14.4-1

Argue that the root of the red-black tree is always black after RB-DELETE executes.

14.4-2

In Exercise 14.3-3, you found the red-black tree that results from successively inserting the keys 41, 38, 31, 12,19, 8 into an initially empty tree. Now show the red-black trees that result from the successive deletion of the keys in the order 8, 12, 19, 31, 38, 41.

14.4-3

In which lines of the code for RB-DELETE-FIXUP might we examine or modify the sentinel nil[T]?

14.4-4

Simplify the code for LEFT-ROTATE by using a sentinel for NIL and another sentinel to hold the pointer to the root.

14.4-5

In each of the cases of Figure 14.7, give the count of black nodes from the root of the subtree shown to each of the subtrees [image], [image], . . ., [image] , and verify that each count remains the same after the transformation. When a node has a color c or c', use the notation count(c) or count(c') symbolically in your count.

14.4-6

Suppose that a node x is inserted into a red-black tree with RB-INSERT and then immediately deleted with RB-DELETE. Is the resulting red-black tree the same as the initial red-black tree? Justify your answer.

Problems

14-1 Persistent dynamic sets

During the course of an algorithm, we sometimes find that we need to maintain past versions of a dynamic set as it is updated. Such a set is called persistent. One way to implement a persistent set is to copy the entire set whenever it is modified, but this approach can slow down a program and also consume much space. Sometimes, we can do much better.

Consider a persistent set S with the operations INSERT, DELETE, and SEARCH, which we implement using binary search trees as shown in Figure 14.8(a). We maintain a separate root for every version of the set. In order to insert the key 5 into the set, we create a new node with key 5. This node becomes the left child of a new node with key 7, since we cannot modify the existing node with key 7. Similarly, the new node with key 7 becomes the left child of a new node with key 8 whose right child is the existing node with key 10. The new node with key 8 becomes, in turn, the right child of a new root r' with key 4 whose left child is the existing node with key 3. We thus copy only part of the tree and share some of the nodes with the original tree, as shown in Figure 14.8(b).

a. For a general persistent binary search tree, identify the nodes that need to be changed to insert a key k or delete a node y.

b. Write a procedure PERSISTENT-TREE-INSERT that, given a persistent tree T and a key k to insert, returns a new persistent tree T' that is the result of inserting k into T. Assume that each tree node has the fields key, left, and right but no parent field. (See also Exercise 14.3-6.)

c. If the height of the persistent binary search tree T is h, what are the time and space requirements of your implementation of PERSISTENT-TREE-INSERT? (The space requirement is proportional to the number of new nodes allocated.)

d. Suppose that we had included the parent field in each node. In this case, PERSISTENT-TREE-INSERT would need to perform additional copying. Prove that PERSISTENT-TREE-INSERT would then require [image] (n) time and space, where n is the number of nodes in the tree.

e. Show how to use red-black trees to guarantee that the worst-case running time and space is O(lg n) per insertion or deletion.

14-2 Join operation on red-black trees

The join operation takes two dynamic sets S1 and S2 and an element x such that for any x1 [image] S1 and x2 [image] S2, we have key[x1] [image] key[x] [image] key[x2]. It returns a set S = S1 [image] {x} [image] S2. In this problem, we investigate how to implement the join operation on red-black trees.

Figure 14.8 (a) A binary search tree with keys 2, 3, 4, 7, 8, 10. (b) The persistent binary search tree that results from the insertion of key 5. The most recent version of the set consists of the nodes reachable from the root r', and the previous version consists of the nodes reachable from r. Heavily shaded nodes are added when key 5 is inserted.

a. Given a red-black tree T, we store its black-height as the field bh[T]. Argue that this field can be maintained by RB-INSERT and RB-DELETE without requiring extra storage in the tree and without increasing the asymptotic running times. Show that while descending through T, we can determine the black-height of each node we visit in O(1) time per node visited.

We wish to implement the operation RB-JOIN(T1,x,T2), which destroys T1 and T2 and returns a red-black tree T = T1 [image] {x} [image] T2. Let n be the total number of nodes in T1 and T2.

b. Assume without loss of generality that bh[T1] [image] bh[T2]. Describe an O(lg n)-time algorithm that finds a black node y in T1 with the largest key from among those nodes whose black-height is bh[T2].

c. Let Ty be the subtree rooted at y. Describe how Ty can be replaced by Ty [image] {x} [image] T2 in O(1) time without destroying the binary-search-tree property.

d. What color should we make x so that red-black properties 1, 2, and 4 are maintained? Describe how property 3 can be enforced in O(lg n) time.

e. Argue that the running time of RB-JOIN is O(lg n).

Chapter notes

The idea of balancing a search tree is due to [image] and Landis [2], who introduced a class of balanced search trees called "AVL trees" in 1962. Balance is maintained in AVL trees by rotations, but as many as [image] (lg n) rotations may be required after an insertion to maintain balance in an n-node tree. Another class of search trees, called "2-3 trees," was introduced by J. E. Hopcroft (unpublished) in 1970. Balance is maintained in a 2-3 tree by manipulating the degrees of nodes in the tree. A generalization of 2-3 trees introduced by Bayer and McCreight [18], called B-trees, is the topic of Chapter 19.

Red-black trees were invented by Bayer [17] under the name "symmetric binary B-trees." Guibas and Sedgewick [93] studied their properties at length and introduced the red/black color convention.

Of the many other variations on balanced binary trees, perhaps the most intriguing are the "splay trees" introduced by Sleator and Tarjan [177], which are "self-adjusting." (A good description of splay trees is given by Tarjan [188].) Splay trees maintain balance without any explicit balance condition such as color. Instead, "splay operations" (which involve rotations) are performed within the tree every time an access is made. The amortized cost (see Chapter 18) of each operation on an n-node tree is O(lg n).

Go to Chapter 15 Back to Table of Contents


Intro to Algorithms: CHAPTER 15: AUGMENTING DATA STRUCTURES

 Return to Table of Contents

CHAPTER 15: AUGMENTING DATA STRUCTURES

There are some engineering situations that require no more than a "textbook" data structure--such as a doubly linked list, a hash table, or a binary search tree--but many others require a dash of creativity. Only in rare situations will you need to create an entirely new type of data structure, though. More often, it will suffice to augment a textbook data structure by storing additional information in it. You can then program new operations for the data structure to support the desired application. Augmenting a data structure is not always straightforward, however, since the added information must be updated and maintained by the ordinary operations on the data structure.

This chapter discusses two data structures that are constructed by augmenting red-black trees. Section 15.1 describes a data structure that supports general order-statistic operations on a dynamic set. We can then quickly find the ith smallest number in a set or the rank of a given element in the total ordering of the set. Section 15.2 abstracts the process of augmenting a data structure and provides a theorem that can simplify the augmentation of red-lack trees. Section 15.3 uses this theorem to help design a data structure for maintaining a dynamic set of intervals, such as time intervals. Given a query interval, we can then quickly find an interval in the set that overlaps it.

15.1 Dynamic order statistics

Chapter 10 introduced the notion of an order statistic. Specifically, the ith order statistic of a set of n elements, where i [image] { 1, 2, . . ., n}, is simply the element in the set with the ith smallest key. We saw that any order statistic could be retrieved in O(n) time from an unordered set. In this section, we shall see how red-black trees can be modified so that any order statistic can be determined in O(lg n) time. We shall also see how the rank of an element--its position in the linear order of the set--can likewise be determined in O(lg n) time.

Figure 15.1 An order-statistic tree, which is an augmented red-black tree. Shaded nodes are red, and darkened nodes are black. In addition to its usual fields, each node x has a field size[x], which is the number of nodes in the subtree rooted at x.

A data structure that can support fast order-statistic operations is shown in Figure 15.1. An order-statistic tree T is simply a red-black tree with additional information stored in each node. Besides the usual red-black tree fields key[x], color[x], p[x], left[x], and right[x] in a node x, we have another field size[x]. This field contains the number of (internal) nodes in the subtree rooted at x (including x itself), that is, the size of the subtree. If we define size[NIL] to be 0, then we have the identity

size[x] = size[left[x]] + size[right[x]] + 1 .

(To handle the boundary condition for NIL properly, an actual implementation might test explicitly for NIL each time the size field is accessed or, more simply, as in Section 14.4, use a sentinel nil[T] to represent NIL, where size[nil[T]] = 0.)

Retrieving an element with a given rank

Before we show how to maintain this size information during insertion and deletion, let us examine the implementation of two order-statistic queries that use this additional information. We begin with an operation that retrieves an element with a given rank. The procedure OS-SELECT(x, i) returns a pointer to the node containing the ith smallest key in the subtree rooted at x. To find the ith smallest key in an order-statistic tree T, we call OS-SELECT(root[T], i).

OS-SELECT(x,i)

1  r    size[left[x]] + 1

2  if i = r

3     then return x

4  elseif i < r

5     then return OS-SELECT(left[x],i)

6  else return OS-SELECT(right[x],i - r)

The idea behind OS-SELECT is similar to that of the selection algorithms in Chapter 10. The value of size[left[x]] is the number of nodes that come before x in an inorder tree walk of the subtree rooted at x. Thus, size[left[x]] + 1 is the rank of x within the subtree rooted at x.

In line 1 of OS-SELECT, we compute r, the rank of node x within the subtree rooted at x. If i = r, then node x is the ith smallest element, so we return x in line 3. If i < r, then the ith smallest element is in x's left subtree, so we recurse on left[x] in line 5. If i > r, then the ith smallest element is in x's right subtree. Since there are r elements in the subtree rooted at x that come before x's right subtree in an inorder tree walk, the ith smallest element in the subtree rooted at x is the (i - r)th smallest element in the subtree rooted at right[x]. This element is determined recursively in line 6.

To see how OS-SELECT operates, consider a search for the 17th smallest element in the order-statistic tree of Figure 15.1. We begin with x as the root, whose key is 26, and with i = 17. Since the size of 26's left subtree is 12, its rank is 13. Thus, we know that the node with rank 17 is the 17 - 13 = 4th smallest element in 26's right subtree. After the recursive call, x is the node with key 41, and i = 4. Since the size of 41's left subtree is 5, its rank within its subtree is 6. Thus, we know that the node with rank 4 is in the 4th smallest element in 41's left subtree. After the recursive call, x is the node with key 30, and its rank within its subtree is 2. Thus, we recurse once again to find the 4 - 2 = 2nd smallest element in the subtree rooted at the node with key 38. We now find that its left subtree has size 1, which means it is the second smallest element. Thus, a pointer to the node with key 38 is returned by the procedure.

Because each recursive call goes down one level in the order-statistic tree, the total time for OS-SELECT is at worst proportional to the height of the tree. Since the tree is a red-black tree, its height is O(lg n), where n is the number of nodes. Thus, the running time of OS-SELECT is O(lg n) for a dynamic set of n elements.

Determining the rank of an element

Given a pointer to a node x in an order-statistic tree T, the procedure OS-RANK returns the position of x in the linear order determined by an inorder tree walk of T.

OS-RANK(T,x)

1  r    size[left[x]] + 1

2  y    x

3  while y    root[T]

4      do if y = right[p[y]]

5            then r    r + size[left[p[y]]] + 1

6         y    p[y]

7  return r

The procedure works as follows. The rank of x can be viewed as the number of nodes preceding x in an inorder tree walk, plus 1 for x itself. The following invariant is maintained: at the top of the while loop of lines 3-6, r is the rank of key[x] in the subtree rooted at node y. We maintain this invariant as follows. In line 1, we set r to be the rank of key[x] within the subtree rooted at x. Setting y [image] x in line 2 makes the invariant true the first time the test in line 3 executes. In each iteration of the while loop, we consider the subtree rooted at p[y]. We have already counted the number of nodes in the subtree rooted at node y that precede x in an inorder walk, so we must add the nodes in the subtree rooted at y's sibling that precede x in an inorder walk, plus 1 for p[y] if it, too, precedes x. If y is a left child, then neither p[y] nor any node in p[y]'s right subtree precedes x, so we leave r alone. Otherwise, y is a right child and all the nodes in p[y]'s left subtree precede x, as does p[y] itself. Thus, in line 5, we add size[left[y]] + 1 to the current value of r Setting y [image] p[y] makes the invariant true for the next iteration. When y = root[T], the procedure returns the value of r, which is now the rank of key[x].

As an example, when we run OS-RANK on the order-statistic tree of Figure 15.1 to find the rank of the node with key 38, we get the following sequence of values of key[y] and r at the top of the while loop:

iteration  key[y]  r

--------------------

    1        38    2

    2        30    4

    3        41    4

    4        26   17

The rank 17 is returned.

Since each iteration of the while loop takes O(1) time, and y goes up one level in the tree with each iteration, the running time of OS-RANK is at worst proportional to the height of the tree: O(lg n) on an n-node order-statistic tree.

Maintaining subtree sizes

Given the size field in each node, OS-SELECT and OS-RANK can quickly compute order-statistic information. But unless these fields can be efficiently maintained by the basic modifying operations on red-black trees, our work will have been for naught. We shall now show that subtree sizes can be maintained for both insertion and deletion without affecting the asymptotic running times of either operation.

We noted in Section 14.3 that insertion into a red-black tree consists of two phases. The first phase goes down the tree from the root, inserting the new node as a child of an existing node. The second phase goes up the tree, changing colors and ultimately performing rotations to maintain the red-black properties.

Figure 15.2 Updating subtree sizes during rotations. The two size fields that need to be updated are the ones incident on the link around which the rotation is performed. The updates are local, requiring only the size information stored in x, y, and the roots of the subtrees shown as triangles.

To maintain the subtree sizes in the first phase, we simply increment size[x] for each node x on the path traversed from the root down toward the leaves. The new node added gets a size of 1. Since there are O(lg n) nodes on the traversed path, the additional cost of maintaining the size fields is O(lg n).

In the second phase, the only structural changes to the underlying red-black tree are caused by rotations, of which there are at most two. Moreover, a rotation is a local operation: it invalidates only the two size fields in the nodes incident on the link around which the rotation is performed. Referring to the code for LEFT-ROTATE(T,x) in Section 14.2, we add the following lines:

13  size[y]    size[x]

14  size[x]    size[left[x]] + size[right[x]] + 1

Figure 15.2 illustrates how the fields are updated. The change to RIGHT-ROTATE is symmetric.

Since at most two rotations are performed during insertion into a red-black tree, only O(1) additional time is spent updating size fields in the second phase. Thus, the total time for insertion into an n-node order-statistic tree is O(lg n)--asymptotically the same as for an ordinary red-black tree.

Deletion from a red-black tree also consists of two phases: the first operates on the underlying search tree, and the second causes at most three rotations and otherwise performs no structural changes. (See Section 14.4.) The first phase splices out one node y. To update the subtree sizes, we simply traverse a path from node y up to the root, decrementing the size field of each node on the path. Since this path has length O(lg n) in an n-node red-black tree, the additional time spent maintaining size fields in the first phase is O(lg n). The O(1) rotations in the second phase of deletion can be handled in the same manner as for insertion. Thus, both insertion and deletion, including the maintenance of the size fields, take O(lg n) time for an n-node order-statistic tree.

Exercises

15.1-1

Show how OS-SELECT(T, 10) operates on the red-black tree T of Figure 15.2.

15.1-2

Show how OS-RANK(T,x) operates on the red-black tree T of Figure 15.2 and the node x with key[x] = 35.

15.1-3

Write a nonrecursive version of OS-SELECT.

15.1-4

Write a recursive procedure OS-KEY-RANK(T,k) that takes as input an order-statistic tree T and a key k and returns the rank of k in the dynamic set represented by T. Assume that the keys of T are distinct.

15.1-5

Given an element x in an n-node order-statistic tree and a natural number i, how can the ith successor of x in the linear order of the tree be determined in O(lg n) time?

15.1-6

Observe that whenever the size field is referenced in either OS-SELECT or OS-RANK, it is only used to compute the rank of x in the subtree rooted at x. Accordingly, suppose we store in each node its rank in the subtree of which it is the root. Show how this information can be maintained during insertion and deletion. (Remember that these two operations can cause rotations.)

15.1-7

Show how to use an order-statistic tree to to count the number of inversions (see Problem 1-3) in an array of size n in time O(n lg n).

15.1-8

Consider n chords on a circle, each defined by its endpoints. Describe an O(n lg n)-time algorithm for determining the number of pairs of chords that intersect inside the circle. (For example, if the n chords are all diameters that meet at the center, then the correct answer is [image].) Assume that no two chords share an endpoint.

15.2 How to augment a data structure

The process of augmenting a basic data structure to support additional functionality occurs quite frequently in algorithm design. It will be used again in the next section to design a data structure that supports operations on intervals. In this section, we shall examine the steps involved in such augmentation. We shall also prove a theorem that allows us to augment red-black trees easily in many cases.

Augmenting a data structure can be broken into four steps:

1. choosing an underlying data structure,

2. determining additional information to be maintained in the underlying data structure,

3. verifying that the additional information can be maintained for the basic modifying operations on the underlying data structure, and

4. developing new operations.

As with any prescriptive design method, you should not blindly follow the steps in the order given. Most design work contains an element of trial and error, and progress on all steps usually proceeds in parallel. There is no point, for example, in determining additional information and developing new operations (steps 2 and 4) if we will not be able to maintain the additional information efficiently. Nevertheless, this four-step method provides a good focus for your efforts in augmenting a data structure, and it is also a good way to organize the documentation of an augmented data structure.

We followed these steps in Section 15.1 to design our order-statistic trees. For step 1, we chose red-black trees as the underlying data structure. A clue to the suitability of red-black trees comes from their efficient support of other dynamic-set operations on a total order, such as MINIMUM, MAXIMUM, SUCCESSOR, and PREDECESSOR.

For step 2, we provided the size fields, which in each node x stores the size of the subtree rooted at x. Generally, the additional information makes operations more efficient. For example, we could have implemented OS-SELECT and OS-RANK using just the keys stored in the tree, but they would not have run in O(lg n) time. Sometimes, the additional information is pointer information rather than data, as in Exercise 15.2-1.

For step 3, we ensured that insertion and deletion could maintain the size fields while still running in O(lg n) time. Ideally, a small number of changes to the data structure should suffice to maintain the additional information. For example, if we simply stored in each node its rank in the tree, the OS-SELECT and OS-RANK procedures would run quickly, but inserting a new minimum element would cause a change to this information in every node of the tree. When we store subtree sizes instead, inserting a new element causes information to change in only O(lg n) nodes.

For step 4, we developed the operations OS-SELECT and OS-RANK. After all, the need for new operations is why we bother to augment a data structure in the first place. Occasionally, rather than developing new operations, we use the additional information to expedite existing ones, as in Exercise 15.2-1.

Augmenting red-black trees

When red-black trees underlie an augmented data structure, we can prove that certain kinds of additional information can always be efficiently maintained by insertion and deletion, thereby making step 3 very easy. The proof of the following theorem is similar to the argument from Section 15.1 that the size field can be maintained for order-statistic trees.

Theorem 15.1

Let â be a field that augments a red-black tree T of n nodes, and suppose that the contents of â for a node x can be computed using only the information in nodes x, left[x], and right[x], including â[left[x]] and â[right[x]]. Then, we can maintain the values of f in all nodes of T during insertion and deletion without asymptotically affecting the O(lg n) performance of these operations.

Proof The main idea of the proof is that a change to an â field in a node x propagates only to ancestors of x in the tree. That is, changing â[x] may require â[p[x]] to be updated, but nothing else; updating â[p[x]] may require â[p[p[x]]] to be updated, but nothing else; and so on up the tree. When â[root[T]] is updated, no other node depends on the new value, so the process terminates. Since the height of a red-black tree is O(lg n), changing an f field in a node costs O(lg n) time in updating nodes dependent on the change.

Insertion of a node x into T consists of two phases. (See Section 14.3.) During the first phase, x is inserted as a child of an existing node p[x]. The value for f[x] can be computed in O(1) time since, by supposition, it depends only on information in the other fields of x itself and the information in x's children, but x's children are both NIL. Once â[x] is computed, the change propagates up the tree. Thus, the total time for the first phase of insertion is O(lg n). During the second phase, the only structural changes to the tree come from rotations. Since only two nodes change in a rotation, the total time for updating theâ fields is O(lg n) per rotation. Since the number of rotations during insertion is at most two, the total time for insertion is O(lg n).

Like insertion, deletion has two phases. (See Section 14.4.) In the first phase, changes to the tree occur if the deleted node is replaced by its successor, and then again when either the deleted node or its successor is spliced out. Propagating the updates to f caused by these changes costs at most O(lg n) since the changes modify the tree locally. Fixing up the red-black tree during the second phase requires at most three rotations, and each rotation requires at most O(lg n) time to propagate the updates to f. Thus, like insertion, the total time for deletion is O(lg n).

In many cases, such as maintenance of the size fields in order-statistic trees, the cost of updating after a rotation is O(1), rather than the O(lg n) derived in the proof of Theorem 15.1. Exercise 15.2-4 gives an example.

Exercises

15.2-1

Show how the dynamic-set queries MINIMUM, MAXIMUM, SUCCESSOR, and PREDECESSOR can each be supported in O(1) worst-case time on an augmented order-statistic tree. The asymptotic performance of other operations on order-statistic trees should not be affected.

15.2-2

Can the black-heights of nodes in a red-black tree be maintained as fields in the nodes of the tree without affecting the asymptotic performance of any of the red-black tree operations? Show how, or argue why not.

15.2-3

Can the depths of nodes in a red-black tree be efficiently maintained as fields in the nodes of the tree? Show how, or argue why not.

15.2-4

Let [image] be an associative binary operator, and let a be a field maintained in each node of a red-black tree. Suppose that we want to include in each node x an additional field [image] such that [image] [x] = a[x1] [image] a[x2], [image] [image] [image] [image] [image] a[xm], where x1, x2, . . . , xm is the inorder listing of nodes in the subtree rooted at x. Show that the [image] fields can be properly updated in O(1) time after a rotation. Modify your argument slightly to show that the size fields in order-statistic trees can be maintained in O(1) time per rotation.

15.2-5

We wish to augment red-black trees with an operation RB-ENUMERATE(x, a,b) that outputs all the keys k such that a< k [image] b in a red-black tree rooted at x. Describe how RB-ENUMERATE can be implemented in [image] (m + lg n) time, where m is the number of keys that are output and n is the number of internal nodes in the tree. (Hint: There is no need to add new fields to the red-black tree.)

15.3 Interval trees

In this section, we shall augment red-black trees to support operations on dynamic sets of intervals. A closed interval is an ordered pair of real numbers [t1, t2], with t1 [image] t2. The interval [t1, t2] represents the set {t [image] R: t1 [image] t [image] t2}. Open and half-open intervals omit both or one of the endpoints from the set, respectively. In this section, we shall assume that intervals are closed; extending the results to open and half-open intervals is conceptually straightforward.

Intervals are convenient for representing events that each occupy a continuous period of time. We might, for example, wish to query a database of time intervals to find out what events occurred during a given interval. The data structure in this section provides an efficient means for maintaining such an interval database.

We can represent an interval [t1, t2] as an object i, with fields low[i] = t1 (the low endpoint) and high[i] = t2 (the high endpoint). We say that intervals i and i' overlap if [image], that is, if low[i] [image] high[i'] and low[i'] [image] high[i]. Any two intervals i and i' satisfy the interval trichotomy; that is, exactly one of the following three properties holds:

a. i and i' overlap,

b. high[i] < low[i'],

c. high[i'] < low[i].

Figure 15.3 shows the three possibilities.

An interval tree is a red-black tree that maintains a dynamic set of elements, with each element x containing an interval int[x]. Interval trees support the following operations.

INTERVAL-INSERT(T,x) adds the element x, whose int field is assumed to contain an interval, to the interval tree T.

Figure 15.3 The interval trichotomy for two closed intervals i and i'. (a) If i and i'overlap, there are four situations; in each, low[i] high[i'] and low[i'] high[i]. (b) high[i] < low[i']. (c) high[i'] < low[i].

Figure 15.4 An interval tree. (a) A set of 10 intervals, shown sorted bottom to top by left endpoint. (b) The interval tree that represents them. An inorder tree walk of the tree lists the nodes in sorted order by left endpoint.

INTERVAL-DELETE(T,x) removes the element x from the interval tree T.

INTERVAL-SEARCH(T,i) returns a pointer to an element x in the interval tree T such that int[x] overlaps interval i, or NIL if no such element is in the set.

Figure 15.4 shows how an interval tree represents a set of intervals. We shall track the four-step method from Section 15.2 as we review the design of an interval tree and the operations that run on it.

Step 1: Underlying data structure

We choose a red-black tree in which each node x contains an interval int[x] and the key of x is the low endpoint, low[int[x]], of the interval. Thus, an inorder tree walk of the data structure lists the intervals in sorted order by low endpoint.

Step 2: Additional information

In addition to the intervals themselves, each node x contains a value max[x], which is the maximum value of any interval endpoint stored in the subtree rooted at x. Since any interval's high endpoint is at least as large as its low endpoint, max[x] is the maximum value of all right endpoints in the subtree rooted at x.

Step 3: Maintaining the information

We must verify that insertion and deletion can be performed in O(lg n) time on an interval tree of n nodes. We can determine max[x] given interval int[x] and the max values of node x's children:

max[x] = max(high[int[x]], max[left[x]], max[right[x]]).

Thus, by Theorem 15.1, insertion and deletion run in O(lg n) time. In fact, updating the max fields after a rotation can be accomplished in O(1) time, as is shown in Exercises 15.2-4 and 15.3-1.

Step 4: Developing new operations

The only new operation we need is INTERVAL-SEARCH(T, i), which finds an interval in tree T that overlaps interval i. If there is no interval that overlaps i in the tree, NIL is returned.

INTERVAL-SEARCH(T,i)

1  x    root[T]

2  while x    NIL and i does not overlap int[x]

3      do if left[x]   NIL and max[left[x]]    low[i]

4            then x    left[x]

5            else x    right[x]

6  return x

The search for an interval that overlaps i starts with x at the root of the tree and proceeds downward. It terminates when either an overlapping interval is found or x becomes NIL. Since each iteration of the basic loop takes O(1) time, and since the height of an n-node red-black tree is O(lg n), the INTERVAL-SEARCH procedure takes O(lg n) time.

Before we see why INTERVAL-SEARCH is correct, let's examine how it works on the interval tree in Figure 15.4. Suppose we wish to find an interval that overlaps the interval i = [22, 25]. We begin with x as the root, which contains [16,21] and does not overlap i. Since max[left[x]] = 23 is greater than low[i] = 22, the loop continues with x as the left child of the root--the node containing [8, 9], which also does not overlap i. This time, max[left[x]] = 10 is less than low[i] = 22, so the loop continues with the right child of x as the new x. The interval [15, 23] stored in this node overlaps i, so the procedure returns this node.

As an example of an unsuccessful search, suppose we wish to find an interval that overlaps i = [11, 14] in the interval tree of Figure 15.4. We once again begin with x as the root. Since the root's interval [16, 21] does not overlap i, and since max[left[x]] = 23 is greater than low[i] = 11, we go left to the node containing [8, 9]. (Note that no interval in the right subtree overlaps i--we shall see why later.) Interval [8, 9] does not overlap i, and max[left[x]] = 10 is less than low[i] = 11, so we go right. (Note that no interval in the left subtree overlaps i.) Interval [15, 23] does not overlap i, and its left child is NIL, so we go right, the loop terminates, and NIL is returned.

To see why INTERVAL-SEARCH is correct, we must understand why it suffices to examine a single path from the root. The basic idea is that at any node x, if int[x] does not overlap i, the search always proceeds in a safe direction: an overlapping interval will definitely be found if there is one in the tree. The following theorem states this property more precisely.

Theorem 15.2

Consider any iteration of the while loop during the execution of INTERVAL-SEARCH(T, i).

1. f line 4 is executed (the search goes left), then x's left subtree contains an interval that overlaps i or no interval in x's right subtree overlaps i.

2. If line 5 is executed (the search goes right), then x's left subtree contains no interval that overlaps i.

Proof The proof of both cases depend on the interval trichotomy. We prove case 2 first, since it is simpler. Observe that if line 5 is executed, then because of the branch condition in line 3, we have left[x] = NIL, or max[left[x]] < low[i]x. If left[x] = NIL, the subtree rooted at left[x] clearly contains no interval that overlaps i, because it contains no intervals at all. Suppose, therefore, that left[x] [image] NIL and max[left]x]] < low[i]. Let i' be an interval in x's left subtree. (See Figure 15.5(a).) Since max[left[x]] is the largest endpoint in x's left subtree, we have

high[i']      max[left[x]]

<  low[i] ,

and thus, by the interval trichotomy, i' and i do not overlap, which completes the proof of case 2.

To prove case 1, we may assume that no intervals in x's left subtree overlap i (since if any do, we are done), and thus we need only prove that no intervals in x's right subtree overlap i under this assumption. Observe that if line 4 is executed, then because of the branch condition in line 3, we have max[left[x]] [image] low[i]. Moreover, by definition of the max field,

Figure 15.5 Intervals in the proof of Theorem 15.2. The value of max[left[x]] is shown in each case as a dashed line. (a) Case 2: the search goes right. No interval i' can overlap i. (b) Case 1: the search goes left. The left subtree of x contains an interval that overlaps i (situation not shown), or there is an interval i' in x's left subtree such that high[i'] = max[left[x]]. Since i does not overlap i', neither does it overlap any interval i\" in x' s right subtree, since low[i'] low[i\"].

there must be some interval i' in x's left subtree such that

high[i'] = max[left[x]]

   low[i].

(Figure 15.5(b) illustrates the situation.) Since i and i' do not overlap, and since it is not true that high[i'] < low[i], it follows by the interval trichotomy that high[i] < low[i']. Interval trees are keyed on the low endpoints of intervals, and thus the search-tree property implies that for any interval i" in x's right subtree,

high[i] < low[i']

   low[i"].

By the interval trichotomy, i and i" do not overlap.

Theorem 15.2 guarantees that if INTERVAL-SEARCH continues with one of x's children and no overlapping interval is found, a search starting with x's other child would have been equally fruitless.

Exercises

15.3-1

Write pseudocode for LEFT-ROTATE that operates on nodes in an interval tree and updates the max fields in O(1) time.

15.3-2

Rewrite the code for INTERVAL-SEARCH so that it works properly when all intervals are assumed to be open.

15.3-3

Describe an efficient algorithm that, given an interval i, returns an interval overlapping i that has the minimum low endpoint, or NIL if no such interval exists.

15.3-4

Given an interval tree T and an interval i, describe how all intervals in T that overlap i can be listed in O(min(n, k 1g n)) time, where k is the number of intervals in the output list. (Optional: Find a solution that does not modify the tree.)

15.3-5

Suggest modifications to the interval-tree procedures to support the operation INTERVAL-SEARCH-EXACTLY(T, i), which returns a pointer to a node x in interval tree T such that low[int]x]] = low[i] and high[int[x]] = high[i], or NIL if T contains no such node. All operations, including INTERVAL-SEARCH-EXACTLY, should run in O(1g n) time on an n-node tree.

15.3-6

Show how to maintain a dynamic set Q of numbers that supports the operation MIN-GAP, which gives the magnitude of the difference of the two closest numbers in Q. For example, if Q = {1, 5, 9, 15, 18, 22}, then MIN-GAP(Q) returns 18 - 15 = 3, since 15 and 18 are the two closest numbers in Q. Make the operations INSERT, DELETE, SEARCH, and MIN-GAP as efficient as possible, and analyze their running times.

15.3-7

VLSI databases commonly represent an integrated circuit as a list of rectangles. Assume that each rectangle is rectilinearly oriented (sides parallel to the x- and y-axis), so that a representation of a rectangle consists of its minimum and maximum x- and y-coordinates. Give an O(n 1g n)-time algorithm to decide whether or not a set of rectangles so represented contains two rectangles that overlap. Your algorithm need not report all intersecting pairs, but it must report that an overlap exists if one rectangle entirely covers another, even if the boundary lines do not intersect. (Hint: Move a "sweep" line across the set of rectangles.)

Problems

15-1 Point of maximum overlap

Suppose that we wish to keep track of a point of maximum overlap in a set of intervals--a point that has the largest number of intervals in the database overlapping it. Show how the point of maximum overlap can be maintained efficiently while intervals are inserted and deleted.

15-2 Josephus permutation

The Josephus problem is defined as follows. Suppose that n people are arranged in a circle and that we are given a positive integer m [image] n. Beginning with a designated first person, we proceed around the circle, removing every mth person. After each person is removed, counting continues around the circle that remains. This process continues until all n people have been removed. The order in which the people are removed from the circle defines the (n, m)-Josephus permutation of the integers 1, 2, . . . , n. For example, the (7, 3)-Josephus permutation is < 3, 6, 2, 7, 5, 1, 4 > .

a. Suppose that m is a constant. Describe an O(n)-time algorithm that, given an integer n, outputs the (n, m)-Josephus permutation.

b. Suppose that m is not a constant. Describe an O(n 1g n)-time algorithm that, given integers n and m, outputs the (n, m)-Josephus permutation.

Chapter notes

Preparata and Shamos [160] describe several of the interval trees that appear in the literature. Among the more important theoretically are those due independently to H. Edelsbrunner (1980) and E. M. McCreight (1981), which, in a database of n intervals, allow all k intervals that overlap a given query interval to be enumerated in O(k + 1g n) time.

Go to Part IV Back to Table of Contents


Intro to Algorithms: CHAPTER 16: DYNAMIC PROGRAMMING

 Return to Table of Contents

CHAPTER 16: DYNAMIC PROGRAMMING

Dynamic programming, like the divide-and-conquer method, solves problems by combining the solutions to subproblems. ("Programming" in this context refers to a tabular method, not to writing computer code.) As we saw in Chapter 1, divide-and-conquer algorithms partition the problem into independent subproblems, solve the subproblems recursively, and then combine their solutions to solve the original problem. In contrast, dynamic programming is applicable when the subproblems are not independent, that is, when subproblems share subsubproblems. In this context, a divide-and-conquer algorithm does more work than necessary, repeatedly solving the common subsubproblems. A dynamic-programming algorithm solves every subsubproblem just once and then saves its answer in a table, thereby avoiding the work of recomputing the answer every time the subsubproblem is encountered.

Dynamic programming is typically applied to optimization problems. In such problems there can be many possible solutions. Each solution has a value, and we wish to find a solution with the optimal (minimum or maximum) value. We call such a solution an optimal solution to the problem, as opposed to the optimal solution, since there may be several solutions that achieve the optimal value.

The development of a dynamic-programming algorithm can be broken into a sequence of four steps.

1. Characterize the structure of an optimal solution.

2. Recursively define the value of an optimal solution.

3. Compute the value of an optimal solution in a bottom-up fashion.

4. Construct an optimal solution from computed information.

Steps 1-3 form the basis of a dynamic-programming solution to a problem. Step 4 can be omitted if only the value of an optimal solution is required. When we do perform step 4, we sometimes maintain additional information during the computation in step 3 to ease the construction of an optimal solution.

The sections that follow use the dynamic-programming method to solve some optimization problems. Section 16.1 asks how we can multiply a chain of matrices so that the fewest total scalar multiplications are performed. Given this example of dynamic programming, Section 16.2 discusses two key characteristics that a problem must have for dynamic programming to be a viable solution technique. Section 16.3 then shows how to find the longest common subsequence of two sequences. Finally, Section 16.4 uses dynamic programming to find an optimal triangulation of a convex polygon, a problem that is surprisingly similar to matrix-chain multiplication.

16.1 Matrix-chain multiplication

Our first example of dynamic programming is an algorithm that solves the problem of matrix-chain multiplication. We are given a sequence (chain) [image] A1, A2, . . ., An [image] of n matrices to be multiplied, and we wish to compute the product

A1A2         An .

(16.1)

We can evaluate the expression (16.1) using the standard algorithm for multiplying pairs of matrices as a subroutine once we have parenthesized it to resolve all ambiguities in how the matrices are multiplied together. A product of matrices is fully parenthesized if it is either a single matrix or the product of two fully parenthesized matrix products, surrounded by parentheses. Matrix multiplication is associative, and so all parenthesizations yield the same product. For example, if the chain of matrices is [image] A1, A2, A3, A4 [image], the product A1A2A3A4 can be fully parenthesized in five distinct ways:

(A1(A2(A3A4))) ,

(A1((A2A3)A4)) ,

((A1A2)(A3A4)) ,

((A1(A2A3))A4) ,

(((A1A2)A3)A4) .

The way we parenthesize a chain of matrices can have a dramatic impact on the cost of evaluating the product. Consider first the cost of multiplying two matrices. The standard algorithm is given by the following pseudocode. The attributes rows and columns are the numbers of rows and columns in a matrix.

MATRIX-MULTIPLY(A,B)

1 if columns [A]    rows [B]

2    then error "incompatible dimensions"

3    else for i   1 to rows [A]

4             do for j   1 to columns[B]

5                    do C[i, j]    0

6                       for k    1 to columns [A]

7                           do C[i ,j]    C[i ,j] + A [i, k] B[k, j]

8         return C

We can multiply two matrices A and B only if the number of columns of A is equal to the number of rows of B. If A is a p X q matrix and B is a q X r matrix, the resulting matrix C is a p X r matrix. The time to compute C is dominated by the number of scalar multiplications in line 7, which is pqr. In what follows, we shall express running times in terms of the number of scalar multiplications.

To illustrate the different costs incurred by different parenthesizations of a matrix product, consider the problem of a chain [image] A1, A2, A3 [image] of three matrices. Suppose that the dimensions of the matrices are 10 X 100,100 [image] 5, and 5 [image] 50, respectively. If we multiply according to the parenthesization ((A1A2)A3), we perform 10 [image] 100 [image] 5 = 5000 scalar multiplications to compute the 10 X 5 matrix product A1A2, plus another 10 [image] 5 [image] 50 = 2500 scalar multiplications to multiply this matrix by A3, for a total of 7500 scalar multiplications. If instead we multiply according to the parenthesization (A1(A2A3)), we perform 100 [image] 5 [image] 50 = 25,000 scalar multiplications to compute the 100 X 50 matrix product A2A3, plus another 10 [image] 100 [image] 50 = 50,000 scalar multiplications to multiply A1 by this matrix, for a total of 75,000 scalar multiplications. Thus, computing the product according to the first parenthesization is 10 times faster.

The matrix-chain multiplication problem can be stated as follows: given a chain [image] A1, A2, . . . , An [image] of n matrices, where for i = 1, 2, . . . , n , matrix Ai has dimension pi - 1 X pi, fully parenthesize the product A1 A2...An in a way that minimizes the number of scalar multiplications.

Counting the number of parenthesizations

Before solving the matrix-chain multiplication problem by dynamic programming, we should convince ourselves that exhaustively checking all possible parenthesizations does not yield an efficient algorithm. Denote the number of alternative parenthesizations of a sequence of n matrices by P(n). Since we can split a sequence of n matrices between the kth and (k + 1)st matrices for any k = 1, 2, . . . , n -1 and then parenthesize the two resulting subsequences independently, we obtain the recurrence

Problem 13-4 asked you to show that the solution to this recurrence is the sequence of Catalan numbers:

P(n) = C (n - 1),

where

The number of solutions is thus exponential in n, and the brute-force method of exhaustive search is therefore a poor strategy for determining the optimal parenthesization of a matrix chain.

The structure of an optimal parenthesization

The first step of the dynamic-programming paradigm is to characterize the structure of an optimal solution. For the matrix-chain multiplication problem, we can perform this step as follows. For convenience, let us adopt the notation Ai..j for the matrix that results from evaluating the product AiAi + 1 [image] [image] [image] Aj. An optimal parenthesization of the product A1A2 [image] [image] [image] An splits the product between Ak and Ak + 1 for some integer k in the range 1 [image] k < n . That is, for some value of k, we first compute the matrices A1..k and Ak + 1..n and then multiply them together to produce the final product A1..n. The cost of this optimal parenthesization is thus the cost of computing the matrix A1..k, plus the cost of computing Ak + 1..n, plus the cost of multiplying them together.

The key observation is that the parenthesization of the "prefix" subchain A1A2 [image] [image] [image] Ak within this optimal parenthesization of A1A2 [image] [image] [image] An must be an optimal parenthesization of A1A2 [image] [image] [image] Ak. Why? If there were a less costly way to parenthesize A1A2 [image] [image] [image] Ak, substituting that parenthesization in the optimal parenthesization of A1A2 [image] [image] [image] An would produce another parenthesization of A1A2 [image] [image] [image] An whose cost was lower than the optimum: a contradiction. A similar observation holds for the the parenthesization of the subchain Ak+1Ak+2 [image] [image] [image] An in the optimal parenthesization of A1A2 [image] [image] [image] An: it must be an optimal parenthesization of Ak+1Ak+2 [image] [image] [image] An.

Thus, an optimal solution to an instance of the matrix-chain multiplication problem contains within it optimal solutions to subproblem instances. Optimal substructure within an optimal solution is one of the hallmarks of the applicability of dynamic programming, as we shall see in Section 16.2.

A recursive solution

The second step of the dynamic-programming paradigm is to define the value of an optimal solution recursively in terms of the optimal solutions to subproblems. For the matrix-chain multiplication problem, we pick as our subproblems the problems of determining the minimum cost of a parenthesization of AiAi+1. . . Aj for 1 [image] i [image] j [image] n. Let m[i, j] be the minimum number of scalar multiplications needed to compute the matrix Ai..j; the cost of a cheapest way to compute A1..n would thus be m[1, n].

We can define m[i, j] recursively as follows. If i = j, the chain consists of just one matrix Ai..i = Ai, so no scalar multiplications are necessary to compute the product. Thus, m[i,i] = 0 for i = 1, 2, . . . , n. To compute m[i, j] when i < j, we take advantage of the structure of an optimal solution from step 1. Let us assume that the optimal parenthesization splits the product AiAi+l . . . Aj between Ak and Ak+1, where i [image] k < j. Then, m[i, j] is equal to the minimum cost for computing the subproducts Ai..k and Ak+1..j, plus the cost of multiplying these two matrices together. Since computing the matrix product Ai..kAk+1..j takes pi-1pkpj scalar multiplications, we obtain

m[i, j] = m[i, k] + m[k + 1, j] + pi-1pkpj .

This recursive equation assumes that we know the value of k, which we don't. There are only j - i possible values for k, however, namely k = i, i + 1, . . . , j - 1. Since the optimal parenthesization must use one of these values for k, we need only check them all to find the best. Thus, our recursive definition for the minimum cost of parenthesizing the product AiAi+1 . . . Aj becomes

(16.2)

The m[i, j] values give the costs of optimal solutions to subproblems. To help us keep track of how to construct an optimal solution, let us define s[i, j] to be a value of k at which we can split the product AiAi+1 . . . Aj to obtain an optimal parenthesization. That is, s[i, j] equals a value k such that m[i, j] = m[i, k] + m[k + 1, j] + pi - 1pkpj .

Computing the optimal costs

At this point, it is a simple matter to write a recursive algorithm based on recurrence (16.2) to compute the minimum cost m[1, n] for multiplying A1A2 . . . An. As we shall see in Section 16.2, however, this algorithm takes exponential time--no better than the brute-force method of checking each way of parenthesizing the product.

The important observation that we can make at this point is that we have relatively few subproblems: one problem for each choice of i and j satisfying 1 [image] i [image] j [image] n, or [image] + n = [image] (n2) total. A recursive algorithm may encounter each subproblem many times in different branches of its recursion tree. This property of overlapping subproblems is the second hallmark of the applicability of dynamic programming.

Instead of computing the solution to recurrence (16.2) recursively, we perform the third step of the dynamic-programming paradigm and compute the optimal cost by using a bottom-up approach. The following pseudocode assumes that matrix Ai has dimensions pi - 1 X pi for i = 1, 2, . . . , n. The input is a sequence [image] p0, p1, . . . , pn [image], here length[p] = n + 1. The procedure uses an auxiliary table m[1 . . n,1 . . n] for storing the m[i, j] costs and an auxiliary table s[1 . . n, 1 . . n] that records which index of k achieved the optimal cost in computing m[i, j].

MATRIX-CHAIN-ORDER(p)

1 n    length[p] - 1

2 for i    1 to n

3      do m[i, i]    0

4 for l    2 to n

5      do for i    1 to n - l + 1

6             do j    i + l-1

7                m[i, j]

8                for k    i to j - 1

9                    do q    m[i, k] + m[k + 1, j] +pi - 1pkpj

10                       if q < m[i, j]

11                          then m[i, j]    q

12                               s[i, j]    k

13 return m and s

The algorithm fills in the table m in a manner that corresponds to solving the parenthesization problem on matrix chains of increasing length. Equation (16.2)shows that the cost m[i, j] of computing a matrix-chain product of j - i + 1 matrices depends only on the costs of computing matrix-chain products of fewer than j - i + 1 matrices. That is, for k = i, i + 1, . . . ,j - 1, the matrix Ai..k is a product of k - i + 1 < j - i + 1 matrices and the matrix Ak+1 . . j a product of j - k < j - i + 1 matrices.

The algorithm first computes m[i, i] [image] 0 for i = 1, 2, . . . , n (the minimum costs for chains of length 1) in lines 2-3. It then uses recurrence (16.2) to compute m[i, i+ 1] for i = 1, 2, . . . , n - 1 (the minimum costs for chains of length l = 2) during the first execution of the loop in lines 4-12. The second time through the loop, it computes m[i, i + 2] for i = 1, 2, . . . , n - 2 (the minimum costs for chains of length l = 3), and so forth. At each step, the m[i, j] cost computed in lines 9-12 depends only on table entries m[i, k] and m[k + 1 ,j] already computed.

Figure 16.1 illustrates this procedure on a chain of n = 6 matrices. Since we have defined m[i, j] only for i [image] j, only the portion of the table m strictly above the main diagonal is used. The figure shows the table rotated to make the main diagonal run horizontally. The matrix chain is listed along the bottom. Using this layout, the minimum cost m[i, j] for multiplying a subchain AiAi+1. . . Aj of matrices can be found at the intersection of lines running northeast from Ai and northwest from Aj. Each horizontal row in the table contains the entries for matrix chains of the same length. MATRIX-CHAIN-ORDER computes the rows from bottom to top and from left to right within each row. An entry m[i, j] is computed using the products pi - 1pkpj for k = i, i + 1, . . . , j - 1 and all entries southwest and southeast from m[i, j].

Figure 16.1 The m and s tables computed by MATRIX-CHAIN-ORDER for n = 6 and the following matrix dimensions:

matrix  dimension

-----------------

  A1     30 X 35

  A2     35 X 15

  A3     15 X 5

  A4     5 X 10

  A5     10 X 20

  A6     20 X 25

The tables are rotated so that the main diagonal runs horizontally. Only the main diagonal and upper triangle are used in the m table, and only the upper triangle is used in the s table. The minimum number of scalar multiplications to multiply the 6 matrices is m[1, 6] = 15,125. Of the lightly shaded entries, the pairs that have the same shading are taken together in line 9 when computing

A simple inspection of the nested loop structure of MATRIX-CHAIN-ORDER yields a running time of O(n3) for the algorithm. The loops are nested three deep, and each loop index (l, i, and k) takes on at most n values. Exercise 16.1-3 asks you to show that the running time of this algorithm is in fact also [image] (n3). The algorithm requires [image] (n2) space to store the m and s tables. Thus, MATRIX-CHAIN-ORDER is much more efficient than the exponential-time method of enumerating all possible parenthesizations and checking each one.

Constructing an optimal solution

Although MATRIX-CHAIN-ORDER determines the optimal number of scalar multiplications needed to compute a matrix-chain product, it does not directly show how to multiply the matrices. Step 4 of the dynamic-programming paradigm is to construct an optimal solution from computed information.

In our case, we use the table s[1 . . n, 1 . . n] to determine the best way to multiply the matrices. Each entry s[i, j] records the value of k such that the optimal parenthesization of AiAi + 1 [image] [image] [image] Aj splits the product between Ak and Ak + 1. Thus, we know that the final matrix multiplication in computing A1..n optimally is A1..s[1,n]As[1,n]+1..n. The earlier matrix multiplications can be computed recursively, since s[l,s[1,n]] determines the last matrix multiplication in computing A1..s[1,n], and s[s[1,n] + 1, n] determines the last matrix multiplication in computing As[1,n]+..n. The following recursive procedure computes the matrix-chain product Ai..j given the matrices A = [image] A1, A2, . . . , An [image], the s table computed by MATRIX-CHAIN-ORDER, and the indices i and j. The initial call is MATRIX-CHAIN-MULTIPLY(A, s, 1, n).

MATRIX-CHAIN-MULTIPLY(A, s, i, ,j)

1 if j >i

2     then X    MATRIX-CHAIN-MULTIPLY(A, s, i, s[i, j])

3          Y    MATRIX-CHAIN-MULTIPLY(A, s, s[i, j] + 1, j)

4          return MATRIX-MULTIPLY(X, Y)

5     else return Ai

In the example of Figure 16.1, the call MATRIX-CHAIN-MULTIPLY(A, s, 1, 6) computes the matrix-chain product according to the parenthesization

((A1(A2A3))((A4A5)A6)) .

(16.3)

Exercises

16.1-1

Find an optimal parenthesization of a matrix-chain product whose sequence of dimensions is [image] 5, 10, 3, 12, 5, 50, 6 [image].

16.1-2

Give an efficient algorithm PRINT-OPTIMAL-PARENS to print the optimal parenthesization of a matrix chain given the table s computed by MATRIX-CHAIN-ORDER. Analyze your algorithm.

16.1-3

Let R(i, j) be the number of times that table entry m[i, j] is referenced by MATRIX-CHAIN-ORDER in computing other table entries. Show that the total number of references for the entire table is

(Hint: You may find the identity

16.1-4

Show that a full parenthesization of an n-element expression has exactly n - 1 pairs of parentheses.

16.2 Elements of dynamic programming

Although we have just worked through an example of the dynamic-programming method, you might still be wondering just when the method applies. From an engineering perspective, when should we look for a dynamic-programming solution to a problem? In this section, we examine the two key ingredients that an optimization problem must have for dynamic programming to be applicable: optimal substructure and overlapping subproblems. We also look at a variant method, called memoization, for taking advantage of the overlapping-subproblems property.

Optimal substructure

The first step in solving an optimization problem by dynamic programming is to characterize the structure of an optimal solution. We say that a problem exhibits optimal substructure if an optimal solution to the problem contains within it optimal solutions to subproblems. Whenever a problem exhibits optimal substructure, it is a good clue that dynamic programming might apply. (It also might mean that a greedy strategy applies, however. See Chapter 17.)

In Section 16.1, we discovered that the problem of matrix-chain multiplication exhibits optimal substructure. We observed that an optimal parenthesization of A1 A2 [image] [image] [image] An that splits the product between Ak and Ak + 1 contains within it optimal solutions to the problems of parenthesizing A1A2 [image] [image] [image] A k and Ak + 1Ak + 2. . . An. The technique that we used to show that subproblems have optimal solutions is typical. We assume that there is a better solution to the subproblem and show how this assumption contradicts the optimality of the solution to the original problem.

The optimal substructure of a problem often suggests a suitable space of subproblems to which dynamic programming can be applied. Typically, there are several classes of subproblems that might be considered "natural" for a problem. For example, the space of subproblems that we considered for matrix-chain multiplication contained all subchains of the input chain. We could equally well have chosen as our space of subproblems arbitrary sequences of matrices from the input chain, but this space of subproblems is unnecessarily large. A dynamic-programming algorithm based on this space of subproblems solves many more problems than it has to.

Investigating the optimal substructure of a problem by iterating on subproblem instances is a good way to infer a suitable space of subproblems for dynamic programming. For example, after looking at the structure of an optimal solution to a matrix-chain problem, we might iterate and look at the structure of optimal solutions to subproblems, subsubproblems, and so forth. We discover that all subproblems consist of subchains of [image] Al, A2, . . . , An [image]. Thus, the set of chains of the form [image] Ai, Aj+l, . . . , Aj [image] for 1 [image] i [image] j [image] n makes a natural and reasonable space of subproblems to use.

Overlapping subproblems

The second ingredient that an optimization problem must have for dynamic programming to be applicable is that the space of subproblems must be "small" in the sense that a recursive algorithm for the problem solves the same subproblems over and over, rather than always generating new subproblems. Typically, the total number of distinct subproblems is a polynomial in the input size. When a recursive algorithm revisits the same problem over and over again, we say that the optimization problem has overlapping subproblems. In contrast, a problem for which a divide-and-conquer approach is suitable usually generates brand-new problems at each step of the recursion. Dynamic-programming algorithms typically take advantage of overlapping subproblems by solving each subproblem once and then storing the solution in a table where it can be looked up when needed, using constant time per lookup.

To illustrate the overlapping-subproblems property, let us reexamine the matrix-chain multiplication problem. Referring back to Figure 16.1, observe that MATRIX-CHAIN-ORDER repeatedly looks up the solution to subproblems in lower rows when solving subproblems in higher rows. For example, entry m[3, 4] is referenced 4 times: during the computations of m[2, 4], m[1, 4], m[3, 5], and m[3, 6]. If m[3, 4] were recomputed each time, rather than just being looked up, the increase in running time would be dramatic. To see this, consider the following (inefficient) recursive procedure that determines m[i, j], the minimum number of scalar multiplications needed to compute the matrix-chain product Ai..j = AiAi + 1, . . . Aj. The procedure is based directly on the recurrence (16.2).

Figure 16.2 The recursion tree for the computation of RECURSIVE-MATRIX-CHAIN(p, 1, 4). Each node contains the parameters i and j. The computations performed in a shaded subtree are replaced by a single table lookup in MEMOIZED-MATRIX-CHAIN(p, 1, 4).

RECURSIVE-MATRIX-CHAIN(p, i, j)

1 if i = j

2     then return 0

3 m[i, j]

4 for k    i to j - 1

5      do q    RECURSIVE-MATRIX-CHAIN(p, i, k)

+ RECURSIVE-MATRIX-CHAIN(p, k + 1, j) + pi-1pkpj

6         if q < m[i, j]

7            then m[i, j]    q

8 return m[i, j]

Figure 16.2 shows the recursion tree produced by the call RECURSIVE-MATRIX-CHAIN(p, 1, 4). Each node is labeled by the values of the parameters i and j. Observe that some pairs of values occur many times.

In fact, we can show that the running time T(n) to compute m[1, n] by this recursive procedure is at least exponential in n. Let us assume that the execution of lines 1-2 and of lines 6-7 each take at least unit time. Inspection of the procedure yields the recurrence

Noting that for i = 1, 2, . . . , n - 1, each term T(i) appears once as T(k) and once as T(n - k), and collecting the n - 1 1's in the summation together with the 1 out front, we can rewrite the recurrence as

(16.4)

We shall prove that T(n) = [image] (2n) using the substitution method. Specifically, we shall show that T(n) [image] 2n-1 for all n > 1. The basis is easy, since T(1) > 1 = 2 [image]. Inductively, for n [image] 2 we have

which completes the proof. Thus, the total amount of work performed by the call RECURSIVE-MATRIX-CHAIN(p, 1, n) is at least exponential in n.

Compare this top-down, recursive algorithm with the bottom-up, dynamic-programming algorithm. The latter is more efficient because it takes advantage of the overlapping-subproblems property. There are only [image] (n2) different subproblems, and the dynamic-programming algorithm solves each exactly once. The recursive algorithm, on the other hand, must repeatedly resolve each subproblem each time it reappears in the recursion tree. Whenever a recursion tree for the natural recursive solution to a problem contains the same subproblem repeatedly, and the total number of different subproblems is small, it is a good idea to see if dynamic programming can be made to work.

Memoization

There is a variation of dynamic programming that often offers the efficiency of the usual dynamic-programming approach while maintaining a top-down strategy. The idea is to memoize the natural, but inefficient, recursive algorithm. As in ordinary dynamic programming, we maintain a table with subproblem solutions, but the control structure for filling in the table is more like the recursive algorithm.

A memoized recursive algorithm maintains an entry in a table for the solution to each subproblem. Each table entry initially contains a special value to indicate that the entry has yet to be filled in. When the subproblem is first encountered during the execution of the recursive algorithm, its solution is computed and then stored in the table. Each subsequent time that the subproblem is encountered, the value stored in the table is simply looked up and returned.1

1This approach presupposes that the set of all possible subproblem parameters is known and that the relation between table positions and subproblems is established. Another approach is to memoize by using hashing with the subproblem parameters as keys.

The following procedure is a memoized version of RECURSIVE-MATRIX-CHAIN.

MEMOIZED-MATRIX-CHAIN(p)

1 n    length[p] - 1

2 for i    1 to n

3      do for j    i to n

4             do m[i, j]

5 return LOOKUP-CHAIN(p, 1, n)

LOOKUP-CHAIN(p, i, j)

1 if m[i,j] <

2    then return m[i, j]

3 if i = j

4    then m[i, j]    0

5    else for k    i to j - 1

6             do q    LOOKUP-CHAIN(p, i, k)

+ LOOKUP-CHAIN(p, k + 1, j) + pi - 1pkpj

7                if q < m[i, j]

8                   then m[i, j]    q

9 return m[i, j]

MEMOIZED-MATRIX-CHAIN, like MATRIX-CHAIN-ORDER, maintains a table m[1 . . n, 1 . . n] of computed values of m[i, j], the minimum number of scalar multiplications needed to compute the matrix Ai..j. Each table entry initially contains the value   [image]  to indicate that the entry has yet to be filled in. When the call LOOKUP-CHAIN(p, i, j) is executed, if m[i, j] <   [image]  in line 1, the procedure simply returns the previously computed cost m[i, j] (line 2). Otherwise, the cost is computed as in RECURSIVE-MATRIX-CHAIN, stored in m[i, j], and returned. (The value   [image]  is convenient to use for an unfilled table entry since it is the value used to initialize m[i, j] in line 3 of RECURSIVE-MATRIX-CHAIN.) Thus, LOOKUP-CHAIN(p, i, j) always returns the value of m[i, j], but it only computes it if this is the first time that LOOKUP-CHAIN has been called with the parameters i and j.

Figure 16.2 illustrates how MEMOIZED-MATRIX-CHAIN saves time over RECURSIVE-MATRIX-CHAIN. Shaded subtrees represent values that are looked up rather than computed.

Like the dynamic-programming algorithm MATRIX-CHAIN-ORDER, the procedure MEMOIZED-MATRIX-CHAIN runs in O(n3 time. Each of   [image] (n2) table entries is initialized once in line 4 of MEMOIZED-MATRIX-CHAIN and filled in for good by just one call of LOOKUP-CHAIN. Each of these   [image] (n2) calls to LOOKUP-CHAIN takes O(n) time, excluding the time spent in computing other table entries, so a total of O(n3) is spent altogether. Memoization thus turns an   [image] (2n) algorithm into an O(n3) algorithm.

In summary, the matrix-chain multiplication problem can be solved in O(n3) time by either a top-down, memoized algorithm or a bottom-up, dynamic-programming algorithm. Both methods take advantage of the overlapping-subproblems property. There are only   [image] (n2) different subproblems in total, and either of these methods computes the solution to each subproblem once. Without memoization, the natural recursive algorithm runs in exponential time, since solved subproblems are repeatedly solved.

In general practice, if all subproblems must be solved at least once, a bottom-up dynamic-programming algorithm usually outperforms a top-down memoized algorithm by a constant factor, because there is no overhead for recursion and less overhead for maintaining the table. Moreover, there are some problems for which the regular pattern of table accesses in the dynamic-programming algorithm can be exploited to reduce time or space requirements even further. Alternatively, if some subproblems in the subproblem space need not be solved at all, the memoized solution has the advantage of only solving those subproblems that are definitely required.

Exercises

16.2-1

Compare the recurrence (16.4) with the recurrence (8.4) that arose in the analysis of the average-case running time of quicksort. Explain intuitively why the solutions to the two recurrences should be so dramatically different.

16.2-2

Which is a more efficient way to determine the optimal number of multiplications in a chain-matrix multiplication problem: enumerating all the ways of parenthesizing the product and computing the number of multiplications for each, or running RECURSIVE-MATRIX-CHAIN? Justify your answer.

16.2-3

Draw the recursion tree for the MERGE-SORT procedure from Section 1.3.1 on an array of 16 elements. Explain why memoization is ineffective in speeding up a good divide-and-conquer algorithm such as MERGE-SORT.

16.3 Longest common subsequence

The next problem we shall consider is the longest-common-subsequence problem. A subsequence of a given sequence is just the given sequence with some elements (possibly none) left out. Formally, given a sequence X =   [image] x1, x2, . . . , xm  [image], another sequence Z =   [image] z1, z2, . . . , zk  [image]  is a subsequence of X if there exists a strictly increasing sequence   [image] i1, i2, . . . , ik  [image]  of indices of X such that for all j = 1, 2, . . . , k, we have xij = zj. For example, Z =   [image] B, C, D, B  [image]  is a subsequence of X =   [image] A, B, C, B, D, A, B  [image]  with corresponding index sequence   [image] 2, 3, 5, 7  [image].

Given two sequences X and Y, we say that a sequence Z is a common subsequence of X and Y if Z is a subsequence of both X and Y. For example, if X =   [image] A, B, C, B, D, A, B  [image]  and Y =   [image] B, D, C, A, B, A  [image], the sequence   [image] B, C, A  [image]  is a common subsequence of both X and Y. The sequence   [image] B, C, A  [image]  is not a longest common subsequence   [image] LCS  [image]  of X and Y, however, since it has length 3 and the sequence   [image] B, C, B, A  [image], which is also common to both X and Y, has length 4. The sequence   [image] B, C, B, A  [image]  is an LCS of X and Y, as is the sequence   [image] B, D, A, B  [image], since there is no common subsequence of length 5 or greater.

In the longest-common-subsequence problem, we are given two sequences X =   [image] x1, x2, . . . , xm  [image]  and Y =   [image] y1, y2, . . . , yn  [image]  and wish to find a maximum-length common subsequence of X and Y. This section shows that the LCS problem can be solved efficiently using dynamic programming.

Characterizing a longest common subsequence

A brute-force approach to solving the LCS problem is to enumerate all subsequences of X and check each subsequence to see if it is also a subsequence of Y, keeping track of the longest subsequence found. Each subsequence of X corresponds to a subset of the indices { 1, 2, . . . , m} of X. There are 2m subsequences of X, so this approach requires exponential time, making it impractical for long sequences.

The LCS problem has an optimal-substructure property, however, as the following theorem shows. As we shall see, the natural class of subproblems correspond to pairs of "prefixes" of the two input sequences. To be precise, given a sequence X =   [image] x1, x2, . . . ,xm  [image], we define the ith prefix of X, for i = 0, 1,...,m, as Xi =   [image] x1, x2, . . . , xi  [image] . For example, if X =   [image] A, B, C, B, D, A, B  [image], then X4 =   [image] A, B, C, B  [image]  and X0 is the empty sequence.

Theorem 16.1

Let X =   [image] x1, x2, . . . , xm  [image]  and Y =   [image] y1, y2, . . . , yn  [image]  be sequences, and let Z =   [image] z1, z2, . . . , zk  [image]  be any LCS of X and Y.

1.     If xm = yn, then zk = xm = yn and Zk-l is an LCS of Xm-l and Yn-l.

2.     If xm   [image]  yn, then zk   [image]  xm implies that Z is an LCS of Xm-1 and Y.

3.     If xm   [image]  yn, then zk   [image]  yn implies that Z is an LCS of X and Yn-l.

Proof     (1) If zk   [image]  xm, then we could append xm = yn to Z to obtain a common subsequence of X and Y of length k + 1, contradicting the supposition that Z is a longest common subsequence of X and Y. Thus, we must have zk = xm = yn. Now, the prefix Zk-l is a length-(k - 1)common subsequence of Xm-1 and Yn-l. We wish to show that it is an LCS. Suppose for the purpose of contradiction that there is a common subsequence W of Xm-l and Yn-l with length greater than k - 1. Then, appending xm = yn to W produces a common subsequence of X and Y whose length is greater than k, which is a contradiction.

(2) If zk   [image]  xm, then Z is a common subsequence of Xm-1 and Y. If there were a common subsequence W of Xm-l and Y with length greater than k, then W would also be a common subsequence of Xm and Y, contradicting the assumption that Z is an LCS of X and Y.

(3) The proof is symmetric to (2).

The characterization of Theorem 16.1 shows that an LCS of two sequences contains within it an LCS of prefixes of the two sequences. Thus, the LCS problem has an optimal-substructure property. A recursive solution also has the overlapping-subproblems property, as we shall see in a moment.

A recursive solution to subproblems

Theorem 16.1 implies that there are either one or two subproblems to examine when finding an LCS of X =   [image] x1, x2, . . . , xm  [image]  and Y =   [image] y1, y2, . . . , yn  [image]. If xm = yn we must find an LCS of Xm-l and Yn-l. Appending xm = yn, to this LCS yields an LCS of X and Y. If xm   [image]  yn, then we must solve two subproblems: finding an LCS of Xm-l and Y and finding an LCS of X and Yn-1. Whichever of these two LCS's is longer is an LCS of X and Y.

We can readily see the overlapping-subproblems property in the LCS problem. To find an LCS of X and Y, we may need to find the LCS's of X and Yn-l and of Xm-l and Y. But each of these subproblems has the subsubproblem of finding the LCS of Xm-l and Yn-1. Many other subproblems share subsubproblems.

Like the matrix-chain multiplication problem, our recursive solution to the LCS problem involves establishing a recurrence for the cost of an optimal solution. Let us define c[i, j] to be the length of an LCS of the sequences Xi and Yj. If either i = 0 or j = 0, one of the sequences has length 0, so the LCS has length 0. The optimal substructure of the LCS problem gives the recursive formula

(16.5)

Computing the length of an LCS

Based on equation (16.5), we could easily write an exponential-time recursive algorithm to compute the length of an LCS of two sequences. Since there are only   [image] (mn) distinct subproblems, however, we can use dynamic programming to compute the solutions bottom up.

Procedure LCS-LENGTH takes two sequences X =   [image] x1,x2,...,xm  [image]  and Y =   [image] y1,y2,...,yn  [image]  as inputs. It stores the c[i, j] values in a table c[0 . . m, 0 . . n] whose entries are computed in row-major order. (That is, the first row of c is filled in from left to right, then the second row, and so on.) It also maintains the table b[1 . . m, 1 . . n] to simplify construction of an optimal solution. Intuitively, b[i, j] points to the table entry corresponding to the optimal subproblem solution chosen when computing c[i, j]. The procedure returns the b and c tables; c[m, n] contains the length of an LCS of X and Y.

LCS-LENGTH(X,Y)

1 m    length[X]

2 n    length[Y]

3 for i    1 to m

4      do c[i,0]    0

5 for j    0 to n

6      do c[0, j]    0

7 for i    1 to m

8      do for j    1 to n

9             do if xi = yj

10                   then c[i, j]    c[i - 1, j -1] + 1

11                        b[i, j]    "  "

12                   else if c[i - 1, j]    c[i, j - 1]

13                           then c[i, j]    c[i - 1, j]

14                                b[i, j]    "  "

15                           else c[i, j]    c[i, j -1]

16                                b[i, j]    "  "

17 return c and b

Figure 16.3 shows the tables produced by LCS-LENGTH on the sequences X =   [image] A, B, C, B, D, A, B  [image]  and Y =   [image] B, D, C, A, B, A  [image]. The running time of the procedure is O(mn), since each table entry takes O(1) time to compute.

Constructing an LCS

The b table returned by LCS-LENGTH can be used to quickly construct an LCS of X =   [image] x1,x2,...,xm) and Y =   [image] y1,y2,...,yn). We simply begin at b[m, n] and trace through the table following the arrows. Whenever we encounter a "  [image]  in entry b[i,j], it implies that xi = yj is an element of the LCS. The elements of the LCS are encountered in reverse order by this method. The following recursive procedure prints out an LCS of X and Y in the proper, forward order. The initial invocation is PRINT-LCS(b, X, length[X], length[Y]).

Figure 16.3 The c and b tables computed by LCS-LENGTH on the sequences X =   A, B, C, B, D, A, B   and Y =   B, D, C, A, B, A  . The square in row i and column j contains the value of c[i, j] and the appropriate arrow for the value of b[i, j]. The entry 4 in c[7, 6]--the lower right-hand corner of the table--is the length of an LCS   B, C, B, A   of X and Y. For i, j > 0, entry c[i, j] depends only on whether xi = yj and the values in entries c[i -1, j], c[i, j - 1], and c[i - 1, j - 1], which are computed before c[i, j]. To reconstruct the elements of an LCS, follow the b[i, j] arrows from the lower right-hand corner; the path is shaded. Each "   on the path corresponds to an entry (highlighted) for which xi = yj is a member of an LCS.

PRINT-LCS(b,X,i,j)

1 if i = 0 or j = 0

2     then return

3 if b[i, j] = "  "

4     then PRINT-LCS(b,X,i - 1, j - 1)

5          print xi

6 elseif b[i,j] = "  "

7     then PRINT-LCS(b,X,i - 1,j)

8 else PRINT-LCS(b,X,i, j - 1)

For the b table in Figure 16.3, this procedure prints "BCBA" The procedure takes time O(m + n), since at least one of i and j is decremented in each stage of the recursion.

Improving the code

Once you have developed an algorithm, you will often find that you can improve on the time or space it uses. This is especially true of straightforward dynamic-programming algorithms. Some changes can simplify the code and improve constant factors but otherwise yield no asymptotic improvement in performance. Others can yield substantial asymptotic savings in time and space.

For example, we can eliminate the b table altogether. Each c[i, j] entry depends on only three other c table entries: c[i - 1,j - 1], c[i - 1,j], and c[i,j - 1]. Given the value of c[i,j], we can determine in O(1) time which of these three values was used to compute c[i,j], without inspecting table b. Thus, we can reconstruct an LCS in O(m + n) time using a procedure similar to PRINT-LCS. (Exercise 16.3-2 asks you to give the pseudocode.) Although we save   [image] (mn) space by this method, the auxiliary space requirement for computing an LCS does not asymptotically decrease, since we need   [image] (mn) space for the c table anyway.

We can, however, reduce the asymptotic space requirements for LCS-LENGTH, since it needs only two rows of table c at a time: the row being computed and the previous row. (In fact, we can use only slightly more than the space for one row of c to compute the length of an LCS. See Exercise 16.3-4.) This improvement works if we only need the length of an LCS; if we need to reconstruct the elements of an LCS, the smaller table does not keep enough information to retrace our steps in O(m + n) time.

Exercises

16.3-1

Determine an LCS of   [image] 1,0,0,1,0,1,0,1  [image]  and   [image] 0,1,0,1,1,0,1,1,0  [image].

16.3-2

Show how to reconstruct an LCS from the completed c table and the original sequences X =   [image] x1,x2,...,xm  [image]  and Y =   [image] y1,y2,...,yn  [image]  in O(m + n) time, without using the b table.

16.3-3

Give a memoized version of LCS-LENGTH that runs in O(mn) time.

16.3-4

Show how to compute the length of an LCS using only 2 min(m, n) entries in the c table plus O(1) additional space. Then, show how to do this using min(m,n) entries plus O(1) additional space.

16.3-5

Give an O(n2)-time algorithm to find the longest monotonically increasing subsequence of a sequence of n numbers.

16.3-6

Give an O(n 1g n)-time algorithm to find the longest monotonically increasing subsequence of a sequence of n numbers. (Hint: Observe that the last element of a candidate subsequence of length i is at least as large as the last element of a candidate subsequence of length i - 1. Maintain candidate subsequences by linking them through the input sequence.)

16.4 Optimal polygon triangulation

In this section, we investigate the problem of optimally triangulating a convex polygon. Despite its outward appearance, we shall see that this geometric problem has a strong similarity to matrix-chain multiplication.

A polygon is a piecewise-linear, closed curve in the plane. That is, it is a curve ending on itself that is formed by a sequence of straight-line segments, called the sides of the polygon. A point joining two consecutive sides is called a vertex of the polygon. If the polygon is simple, as we shall generally assume, it does not cross itself. The set of points in the plane enclosed by a simple polygon forms the interior of the polygon, the set of points on the polygon itself forms its boundary, and the set of points surrounding the polygon forms its exterior. A simple polygon is convex if, given any two points on its boundary or in its interior, all points on the line segment drawn between them are contained in the polygon's boundary or interior.

We can represent a convex polygon by listing its vertices in counterclockwise order. That is, if P =   [image] v0,v1,...,vn-1  [image]  is a convex polygon, it has n sides   [image], where we interpret vn as v0. (In general, we shall implicitly assume arithmetic on vertex indices is taken modulo the number of vertices.)

Given two nonadjacent vertices vi and vj, the segment   [image]  is a chord of the polygon. A chord   [image]  divides the polygon into two polygons:   vi,vi+1,...,vj  [image]  and   [image] vj,vj+1,...,vi  [image]. A triangulation of a polygon is a set T of chords of the polygon that divide the polygon into disjoint triangles (polygons with 3 sides). Figure 16.4 shows two ways of triangulating a 7-sided polygon. In a triangulation, no chords intersect (except at endpoints) and the set T of chords is maximal: every chord not in T intersects some chord in T. The sides of triangles produced by the triangulation are either chords in the triangulation or sides of the polygon. Every triangulation of an n-vertex convex polygon has n - 3 chords and divides the polygon into n - 2 triangles.

In the optimal (polygon) triangulation problem, we are given a convex polygon P =   [image] v0, v1,...,vn-1  [image]  and a weight function w defined on triangles formed by sides and chords of P. The problem is to find a triangulation that minimizes the sum of the weights of the triangles in the triangulation. One weight function on triangles that comes to mind naturally is

where   [image] vivj  [image]  is the euclidean distance from vi to vj. The algorithm we shall develop works for an arbitrary choice of weight function.

Figure 16.4 Two ways of triangulating a convex polygon. Every triangulation of this 7-sided polygon has 7 - 3 = 4 chords and divides the polygon into 7 - 2 = 5 triangles.

Correspondence to parenthesization

There is a surprising correspondence between the triangulation of a polygon and the parenthesization of an expression such as a matrix-chain product. This correspondence is best explained using trees.

A full parenthesization of an expression corresponds to a full binary tree, sometimes called the parse tree of the expression. Figure 16.5(a) shows a parse tree for the parenthesized matrix-chain product

((A1(A2A3))(A4(A5A6))) .

(16.6)

Each leaf of a parse tree is labeled by one of the atomic elements (matrices) in the expression. If the root of a subtree of the parse tree has a left subtree representing an expression El and a right subtree representing an expression Er, then the subtree itself represents the expression (ElEr). There is a one-to-one correspondence between parse trees and fully parenthesized expressions on n atomic elements.

A triangulation of a convex polygon   [image] v0, v1, . . . , vn - 1  [image]  can also be represented by a parse tree. Figure 16.5(b) shows the parse tree for the triangulation of the polygon from Figure 16.4(a). The internal nodes of the parse tree are the chords of the triangulation plus the side   [image], which is the root. The leaves are the other sides of the polygon. The root   [image]  is one side of the triangle   [image]. This triangle determines the children of the root: one is the chord   [image], and the other is the chord   [image]. Notice that this triangle divides the original polygon into three parts: the triangle   [image]  itself, the polygon   [image] v0, v1, . . . , v3  [image], and the polygon   v3, v4, . . . , v6  . Moreover, the two subpolygons are formed entirely by sides of the original polygon, except for their roots, which are the chords    and   .

Figure 16.5 Parse trees. (a) The parse tree for the parenthesized product ((A1(A2A3))(A4(A5A6))) and for the triangulation of the 7-sided polygon from Figure 16.4(a). (b) The triangulation of the polygon with the parse tree overlaid. Each matrix Ai corresponds to the side    for i = 1, 2, . . . , 6.

In recursive fashion, the polygon   v0, v1, . . . , v3   contains the left subtree of the root of the parse tree, and the polygon   v3, v4, . . . , v6   contains the right subtree.

In general, therefore, a triangulation of an n-sided polygon corresponds to a parse tree with n - 1 leaves. By an inverse process, one can produce a triangulation from a given parse tree. There is a one-to-one correspondence between parse trees and triangulations.

Since a fully parenthesized product of n matrices corresponds to a parse tree with n leaves, it therefore also corresponds to a triangulation of an (n + 1)-vertex polygon. Figures 16.5(a) and (b) illustrate this correspondence. Each matrix Ai in a product A1A2          An corresponds to a side    of an (n + 1)-vertex polygon. A chord   , where i < j, corresponds to a matrix Ai+1..j computed during the evaluation of the product.

In fact, the matrix-chain multiplication problem is a special case of the optimal triangulation problem. That is, every instance of matrix-chain multiplication can be cast as an optimal triangulation problem. Given a matrix-chain product A1A2          An, we define an (n + 1)-vertex convex polygon P =   v0, v1, . . . , vn  . If matrix Ai has dimensions pi-1    pi for i = 1, 2, . . . , n, we define the weight function for the triangulation as

An optimal triangulation of P with respect to this weight function gives the parse tree for an optimal parenthesization of A1A2          An.

Although the reverse is not true--the optimal triangulation problem is not a special case of the matrix-chain multiplication problem--it turns out that our code MATRIX-CHAIN-ORDER from Section 16.1, with minor modifications, solves the optimal triangulation problem on an (n + 1)- vertex polygon. We simply replace the sequence   p0, p1, . . . , pn   of matrix dimensions with the sequence   v0, v1, . . . , vn   of vertices, change references to p to references to v, and change line 9 to read:

After running the algorithm, the element m[1, n] contains the weight of an optimal triangulation. Let us see why this is so.

The substructure of an optimal triangulation

Consider an optimal triangulation T of an (n + 1)-vertex polygon P =   v0, v1, . . . , vn   that includes the triangle    for some k, where 1    k    n - 1. The weight of T is just the sum of the weights of    and triangles in the triangulation of the two subpolygons   v0, v1, . . . , vk   and   vk, vk+1, . . . , vn  . The triangulations of the subpolygons determined by T must therefore be optimal, since a lesser-weight triangulation of either subpolygon would contradict the minimality of the weight of T.

A recursive solution

Just as we defined m[i, j] to be the minimum cost of computing the matrix-chain subproduct AiAi+1          Aj, let us define t[i, j], for 1    i < j    n, to be the weight of an optimal triangulation of the polygon   vi - 1, v1 . . . , vj  . For convenience, we consider a degenerate polygon   vi - 1, vi   to have weight 0. The weight of an optimal triangulation of polygon P is given by t[1, n].

Our next step is to define t[i, j] recursively. The basis is the degenerate case of a 2-vertex polygon: t[i, i] = 0 for i = 1, 2, . . . , n. When j - i    1, we have a polygon   vi - 1, vi, . . . , vj   with at least 3 vertices. We wish to minimize over all vertices vk, for k = i, i + 1, . . . , j - 1, the weight of    plus the weights of the optimal triangulations of the polygons   vi - 1, vi, . . . , vk   and   vk, vk+1, . . . , vj  . The recursive formulation is thus

(16.7)

Compare this recurrence with the recurrence (16.2) that we developed for the minimum number m[i, j] of scalar multiplications needed to compute AiAi+1          Aj. Except for the weight function, the recurrences are identical, and thus, with the minor changes to the code mentioned above, the procedure MATRIX-CHAIN-ORDER can compute the weight of an optimal triangulation. Like MATRIX-CHAIN-ORDER, the optimal triangulation procedure runs in time   (n3) and uses   (n2) space.

Exercises

16.4-1

Prove that every triangulation of an n-vertex convex polygon has n - 3 chords and divides the polygon into n - 2 triangles.

16.4-2

Professor Guinevere suggests that a faster algorithm to solve the optimal triangulation problem might exist for the special case in which the weight of a triangle is its area. Is the professor's intuition accurate?

16.4-3

Suppose that a weight function w is defined on the chords of a triangulation instead of on the triangles. The weight of a triangulation with respect to w is then the sum of the weights of the chords in the triangulation. Show that the optimal triangulation problem with weighted chords is just a special case of the optimal triangulation problem with weighted triangles.

16.4-4

Find an optimal triangulation of a regular octagon with unit-length sides. Use the weight function

where |vivj| is the euclidean distance from vi to vj. (A regular polygon is one with equal sides and equal interior angles.)

Problems

16-1     Bitonic euclidean traveling-salesman problem

The euclidean traveling-salesman problem is the problem of determining the shortest closed tour that connects a given set of n points in the plane. Figure 16.6(a) shows the solution to a 7-point problem. The general problem is NP-complete, and its solution is therefore believed to require more than polynomial time (see Chapter 36).

J. L. Bentley has suggested that we simplify the problem by restricting our attention to bitonic tours, that is, tours that start at the leftmost point, go strictly left to right to the rightmost point, and then go strictly right to left back to the starting point. Figure 16.6(b) shows the shortest bitonic tour of the same 7 points. In this case, a polynomial-time algorithm is possible.

Describe an O(n2)-time algorithm for determining an optimal bitonic tour. You may assume that no two points have the same x-coordinate. (Hint: Scan left to right, maintaining optimal possibilities for the two parts of the tour.)

Figure 16.6 Seven points in the plane, shown on a unit grid. (a) The shortest closed tour, with length 24.88 . . .. This tour is not bitonic. (b) The shortest bitonic tour for the same set of points. Its length is 25.58 . . ..

16-2     Printing neatly

Consider the problem of neatly printing a paragraph on a printer. The input text is a sequence of n words of lengths l1,l2,...,ln, measured in characters. We want to print this paragraph neatly on a number of lines that hold a maximum of M characters each. Our criterion of "neatness" is as follows. If a given line contains words i through j and we leave exactly one space between words, the number of extra space characters at the end of the line is    We wish to minimize the sum, over all lines except the last, of the cubes of the numbers of extra space characters at the ends of lines. Give a dynamic-programming algorithm to print a paragraph of n words neatly on a printer. Analyze the running time and space requirements of your algorithm.

16-3     Edit distance

When a "smart" terminal updates a line of text, replacing an existing "source" string x[1..m] with a new "target" string y[1..n], there are several ways in which the changes can be made. A single character of the source string can be deleted, replaced by another character, or copied to the target string; characters can be inserted; or two adjacent characters of the source string can be interchanged ("twiddled") while being copied to the target string. After all the other operations have occurred, an entire suffix of the source string can be deleted, an operation known as "kill to end of line."

As an example, one way to transform the source string algorithm to the target string altruistic is to use the following sequence of operations.

Operation           Target string  Source string

------------------------------------------------

copy a              a                   lgorithm

copy l              al                   gorithm

replace g by t      alt                   orithm

delete o            alt                    rithm

copy r              altr                    ithm

insert u            altru                   ithm

insert i            altrui                  ithm

insert s            altruis                 ithm

twiddle it into ti  altruisti                 hm

insert c            altruistic                hm

kill hm             altruistic

There are many other sequences of operations that accomplish the same result.

Each of the operations delete, replace, copy, insert, twiddle, and kill has an associated cost. (Presumably, the cost of replacing a character is less than the combined costs of deletion and insertion; otherwise, the replace operation would not be used.) The cost of a given sequence of transformation operations is the sum of the costs of the individual operations in the sequence. For the sequence above, the cost of converting algorithm to altruistic is

(3 cost(copy)) + cost(replace) + cost(delete) + (3 cost(insert))

+ cost(twiddle) + cost(kill) .

Given two sequences x[1..m] and y[1..n] and a given set of operation costs, the edit distance from x to y is the cost of the least expensive transformation sequence that converts x to y. Describe a dynamic-programming algorithm to find the edit distance from x[l..m] to y[1..n] and print an optimal transformation sequence. Analyze the running time and space requirements of your algorithm.

16-4     Planning a company party

Professor McKenzie is consulting for the president of A.-B. Corporation, which is planning a company party. The company has a hierarchical structure; that is, the supervisor relation forms a tree rooted at the president. The personnel office has ranked each employee with a conviviality rating, which is a real number. In order to make the party fun for all attendees, the president does not want both an employee and his or her immediate supervisor to attend.

a.     Describe an algorithm to make up the guest list. The goal should be to maximize the sum of the conviviality ratings of the guests. Analyze the running time of your algorithm.

b.     How can the professor ensure that the president gets invited to his own party?

16-5     Viterbi algorithm

We can use dynamic programming on a directed graph G = (V,E) for speech recognition. Each edge (u,v)    E is labeled with a sound   (u,v) from a finite set    of sounds. The labeled graph is a formal model of a person speaking a restricted language. Each path in the graph starting from a distinguished vertex v0    V corresponds to a possible sequence of sounds produced by the model. The label of a directed path is defined to be the concatenation of the labels of the edges on that path.

a.     Describe an efficient algorithm that, given an edge-labeled graph G with distinguished vertex v0 and a sequence s =     1,  2,...,  k   of characters from   , returns a path in G that begins at v0 and has s as its label, if any such path exists. Otherwise, the algorithm should return NO-SUCH-PATH. Analyze the running time of your algorithm. (Hint: You may find concepts from Chapter 23 useful.)

Now, suppose that every edge (u,v)    E has also been given an associated nonnegative probability p(u, v) of traversing the edge (u, v) from vertex u and producing the corresponding sound. The sum of the probabilities of the edges leaving any vertex equals 1. The probability of a path is defined to be the product of the probabilities of its edges. We can view the probability of a path beginning at v0 as the probability that a "random walk" beginning at v0 will follow the specified path, where the choice of which edge to take at a vertex u is made probabilistically according to the probabilities of the available edges leaving u.

b.     Extend your answer to part (a) so that if a path is returned, it is a most probable path starting at v0 and having label s. Analyze the running time of your algorithm.

Chapter notes

R. Bellman began the systematic study of dynamic programming in 1955. The word "programming," both here and in linear programming, refers to the use of a tabular solution method. Although optimization techniques incorporating elements of dynamic programming were known earlier, Bellman provided the area with a solid mathematical basis [21].

Hu and Shing [106] give an O(n 1g n)-time algorithm for the matrix-chain multiplication problem. They also demonstrate the correspondence between the optimal polygon triangulation problem and the matrix-chain multiplication problem.

The O(mn)-time algorithm for the longest-common-subsequence problem seems to be a folk algorithm. Knuth [43] posed the question of whether subquadratic algorithms for the LCS problem exist. Masek and Paterson [143] answered this question in the affirmative by giving an algorithm that runs in O(mn/lg n) time, where n    m and the sequences are drawn from a set of bounded size. For the special case in which no element appears more than once in an input sequence, Szymanski [184] shows that the problem can be solved in O((n+m) lg(n+m)) time. Many of these results extend to the problem of computing string edit distances (Problem 16-3).

Go to Chapter 17     Back to Table of Contents


Intro to Algorithms: CHAPTER 17: GREEDY ALGORITHMS

 Return to Table of Contents

CHAPTER 17: GREEDY ALGORITHMS

Algorithms for optimization problems typically go through a sequence of steps, with a set of choices at each step. For many optimization problems, using dynamic programming to determine the best choices is overkill; simpler, more efficient algorithms will do. A greedy algorithm always makes the choice that looks best at the moment. That is, it makes a locally optimal choice in the hope that this choice will lead to a globally optimal solution. This chapter explores optimization problems that are solvable by greedy algorithms.

Greedy algorithms do not always yield optimal solutions, but for many problems they do. We shall first examine in Section 17.1 a simple but nontrivial problem, the activity-selection problem, for which a greedy algorithm efficiently computes a solution. Next, Section 17.2 reviews some of the basic elements of the greedy approach. Section 17.3 presents an important application of greedy techniques: the design of data-compression (Huffman) codes. In Section 17.4, we investigate some of the theory underlying combinatorial structures called "matroids" for which a greedy algorithm always produces an optimal solution. Finally, Section 17.5 illustrates the application of matroids using the problem of scheduling unit-time tasks with deadlines and penalties.

The greedy method is quite powerful and works well for a wide range of problems. Later chapters will present many algorithms that can be viewed as applications of the greedy method, including minimum-spanning-tree algorithms (Chapter 24), Dijkstra's algorithm for shortest paths form a single source (Chapter 25), and Chvátal's greedy set-covering heuristic (Chapter 37). Minimum spanning trees form a classic example of the greedy method. Although this chapter and Chapter 24 can be read independently of each other, you may find it useful to read them together.

17.1 An activity-selection problem

Our first example is the problem of scheduling a resource among several competing activities. We shall find that a greedy algorithm provides an elegant and simple method for selecting a maximum-size set of mutually compatible activities.

Suppose we have a set S = { 1, 2, . . . , n} of n proposed activities that wish to use a resource, such as a lecture hall, which can be used by only one activity at a time. Each activity i has a start time si and a finish time âi, where si [image] âi. If selected, activity i takes place during the half-open time interval [si,âi). Activities i and j are compatible if the intervals [si, âi) and [sj,âj) do not overlap (i.e., i and j are compatible if si [image] âj or sj [image] âi). The activity-selection problem is to select a maximum-size set of mutually compatible activities.

A greedy algorithm for the activity-selection problem is given in the following pseudocode. We assume that the input activities are in order by increasing finishing time:

â1    â2    . . .    ân .

(17.1)

If not, we can sort them into this order in time O(n 1g n), breaking ties arbitrarily. The pseudocode assumes that inputs s and â are represented as arrays.

GREEDY-ACTIVITY-SELECTOR(s, f)

1 n    length[s]

2 A    {1}

3 j    1

4 for i    2 to n

5      do if si    âj

6            then A    A    {i}

7                 j    i

8 return A

The operation of the algorithm is shown in Figure 17.1. The set A collects the selected activities. The variable j specifies the most recent addition to A. Since the activities are considered in order of nondecreasing finishing time, fj is always the maximum finishing time of any activity in A. That is,

âj = max{fk : K    A}.

(17.2)

Lines 2-3 select activity 1, initialize A to contain just this activity, and initialize j to this activity. Lines 4-7 consider each activity i in turn and add i to A if it is compatible with all previously selected activities. To see if activity i is compatible with every activity currently in A, it suffices by equation (17.2) to check (line 5) that its start time si is not earlier than the finish time fj of the activity most recently added to A. If activity i is compatible, then lines 6-7 add it to A and update j. The GREEDY-ACTIVITY-SELECTOR procedure is quite efficient. It can schedule a set S of n activities in [image] (n) time, assuming that the activities were already sorted initially by their finish times.

Figure 17.1 The operation of GREEDY-ACTIVITY-SELECTOR on 11 activities given at the left. Each row of the figure corresponds to an iteration of the for loop in lines 4-7. The activities that have been selected to be in set A are shaded, and activity i, shown in white, is being considered. If the starting time si of activity i occurs before the finishing time fj of the most recently selected activity j (the arrow between them points left), it is rejected. Otherwise (the arrow points directly up or to the right), it is accepted and put into set A.

The activity picked next by GREEDY-ACTIVITY-SELECTOR is always the one with the earliest finish time that can be legally seheduled. The activity picked is thus a "greedy" choice in the sense that, intuitively, it leaves as much opportunity as possible for the remaining activities to be scheduled. That is, the greedy choice is the one that maximizes the amount of unscheduled time remaining.

Proving the greedy algorithm correct

Greedy algorithms do not always produce optimal solutions. However, GREEDY-ACTIVITY-SELECTOR always finds an optimal solution to an instance of the activity-selection problem.

Theorem 17.1

Algorithm GREEDY-ACTIVITY-SELECTOR produces solutions of maximum size for the activity-selection problem.

Proof Let S = {1, 2, . . . , n} be the set of activities to schedule. Since we are assuming that the activities are in order by finish time, activity 1 has the earliest finish time. We wish to show that there is an optimal solution that begins with a greedy choice, that is, with activity 1.

Suppose that A [image] S is an optimal solution to the given instance of the activity-selection problem, and let us order the activities in A by finish time. Suppose further that the first activity in A is activity k. If k = 1, then schedule A begins with a greedy choice. If k [image] 1, we want to show that there is another optimal solution B to S that begins with the greedy choice, activity 1. Let B = A - {k} [image] {1}. Because fi [image] fk, the activities in B are disjoint, and since B has the same number of activities as A, it is also optimal. Thus, B is an optimal solution for S that contains the greedy choice of activity 1. Therefore, we have shown that there always exists an optimal schedule that begins with a greedy choice.

Moreover, once the greedy choice of activity 1 is made, the problem reduces to finding an optimal solution for the activity-selection problem over those activities in S that are compatible with activity 1. That is, if A is an optimal solution to the original problem S, then A' = A - {1} is an optimal solution to the activity-selection problem S' = {i [image] S: Si [image] f1}. Why? If we could find a solution B' to S' with more activities than A', adding activity 1 to B' would yield a solution B to S with more activities than A, thereby contradicting the optimality of A. Therefore, after each greedy choice is made, we are left with an optimization problem of the same form as the original problem. By induction on the number of choices made, making the greedy choice at every step produces an optimal solution.

Exercises

17.1-1

Give a dynamic-programming algorithm for the activity-selection problem, based on computing mi iteratively for i = 1, 2, . . . , n, where mi is the size of the largest set of mutually compatible activities among activities {1, 2, . . . , i}. Assume that the inputs have been sorted as in equation (17.1). Compare the running time of your solution to the running time of GREEDY-ACTIVITY-SELECTOR.

17.1-2

Suppose that we have a set of activities to schedule among a large number of lecture halls. We wish to schedule all the activities using as few lecture halls as possible. Give an efficient greedy algorithm to determine which activity should use which lecture hall.

(This is also known as the interval-graph coloring problem. We can create an interval graph whose vertices are the given activities and whose edges connect incompatible activities. The smallest number of colors required to color every vertex so that no two adjacent vertices are given the same color corresponds to finding the fewest lecture halls needed to schedule all of the given activities.)

17.1-3

Not just any greedy approach to the activity-selection problem produces a maximum-size set of mutually compatible activities. Give an example to show that the approach of selecting the activity of least duration from those that are compatible with previously selected activities does not work. Do the same for the approach of always selecting the activity that overlaps the fewest other remaining activities.

17.2 Elements of the greedy strategy

A greedy algorithm obtains an optimal solution to a problem by making a sequence of choices. For each decision point in the algorithm, the choice that seems best at the moment is chosen. This heuristic strategy does not always produce an optimal solution, but as we saw in the activity-selection problem, sometimes it does. This section discusses some of the general properties of greedy methods.

How can one tell if a greedy algorithm will solve a particular optimization problem? There is no way in general, but there are two ingredients that are exhibited by most problems that lend themselves to a greedy strategy: the greedy-choice property and optimal substructure.

Greedy-choice property

The first key ingredient is the greedy-choice property: a globally optimal solution can be arrived at by making a locally optimal (greedy) choice. Here is where greedy algorithms differ from dynamic programming. In dynamic programming, we make a choice at each step, but the choice may depend on the solutions to subproblems. In a greedy algorithm, we make whatever choice seems best at the moment and then solve the subproblems arising after the choice is made. The choice made by a greedy algorithm may depend on choices so far, but it cannot depend on any future choices or on the solutions to subproblems. Thus, unlike dynamic programming, which solves the subproblems bottom up, a greedy strategy usually progresses in a top-down fashion, making one greedy choice after another, iteratively reducing each given problem instance to a smaller one.

Of course, we must prove that a greedy choice at each step yields a globally optimal solution, and this is where cleverness may be required. Typically, as in the case of Theorem 17.1, the proof examines a globally optimal solution. It then shows that the solution can be modified so that a greedy choice is made as the first step, and that this choice reduces the problem to a similar but smaller problem. Then, induction is applied to show that a greedy choice can be used at every step. Showing that a greedy choice results in a similar but smaller problem reduces the proof of correctness to demonstrating that an optimal solution must exhibit optimal substructure.

Optimal substructure

A problem exhibits optimal substructure if an optimal solution to the problem contains within it optimal solutions to subproblems. This property is a key ingredient of assessing the applicability of dynamic programming as well as greedy algorithms. As an example of optimal substructure, recall that the proof of Theorem 17.1 demonstrated that if an optimal solution A to the activity selection problem begins with activity 1, then the set of activities A' = A - {1} is an optimal solution to the activity-selection problem S' = {i [image] S : si [image] â1}.

Greedy versus dynamic programming

Because the optimal-substructure property is exploited by both greedy and dynamic-programming strategies, one might be tempted to generate a dynamic-programming solution to a problem when a greedy solution suffices, or one might mistakenly think that a greedy solution works when in fact a dynamic-programming solution is required. To illustrate the subtleties between the two techniques, let us investigate two variants of a classical optimization problem.

The 0-1 knapsack problem is posed as follows. A thief robbing a store finds n items; the ith item is worth vi dollars and weighs wi pounds, where vi and wi are integers. He wants to take as valuable a load as possible, but he can carry at most W pounds in his knapsack for some integer W. What items should he take? (This is called the 0-1 knapsack problem because each item must either be taken or left behind; the thief cannot take a fractional amount of an item or take an item more than once.)

In the fractional knapsack problem, the setup is the same, but the thief can take fractions of items, rather than having to make a binary (0-1) choice for each item. You can think of an item in the 0-1 knapsack problem as being like a gold ingot, while an item in the fractional knapsack problem is more like gold dust.

Both knapsack problems exhibit the optimal-substructure property. For the 0-1 problem, consider the most valuable load that weighs at most W pounds. If we remove item j from this load, the remaining load must be the most valuable load weighing at most W - wj that the thief can take from the n - 1 original items excluding j. For the comparable fractional problem, consider that if we remove a weight w of one item j from the optimal load, the remaining load must be the most valuable load weighing at most W - w that the thief can take from the n - 1 original items plus wj - w pounds of item j.

Although the problems are similar, the fractional knapsack problem is solvable by a greedy strategy, whereas the 0-1 problem is not. To solve the fractional problem, we first compute the value per pound vi/wi for each item. Obeying a greedy strategy, the thief begins by taking as much as possible of the item with the greatest value per pound. If the supply of that item is exhausted and he can still carry more, he takes as much as possible of the item with the next greatest value per pound, and so forth until he can't carry any more. Thus, by sorting the items by value per pound, the greedy algorithm runs in O(n1gn) time. The proof that the fractional knapsack problem has the greedy-choice property is left as Exercise 17.2-1.

To see that this greedy strategy does not work for the 0-1 knapsack problem, consider the problem instance illustrated in Figure 17.2(a). There are 3 items, and the knapsack can hold 50 pounds. Item 1 weighs 10 pounds and is worth 60 dollars. Item 2 weighs 20 pounds and is worth 100 dollars. Item 3 weighs 30 pounds and is worth 120 dollars. Thus, the value per pound of item 1 is 6 dollars per pound, which is greater than the value per pound of either item 2 (5 dollars per pound) or item 3 (4 dollars per pound). The greedy strategy, therefore, would take item 1 first. As can be seen from the case analysis in Figure 17.2(b), however, the optimal solution takes items 2 and 3, leaving 1 behind. The two possible solutions that involve item 1 are both suboptimal.

For the comparable fractional problem, however, the greedy strategy, which takes item 1 first, does yield an optimal solution, as shown in Figure 17.2 (c). Taking item 1 doesn't work in the 0-1 problem because the thief is unable to fill his knapsack to capacity, and the empty space lowers the effective value per pound of his load. In the 0-1 problem, when we consider an item for inclusion in the knapsack, we must compare the solution to the subproblem in which the item is included with the solution to the subproblem in which the item is excluded before we can make the choice. The problem formulated in this way gives rise to many overlapping subproblems--a hallmark of dynamic programming, and indeed, dynamic programming can be used to solve the 0-1 problem. (See Exercise 17.2-2.)

Figure 17.2 The greedy strategy does not work for the 0-1 knapsack problem. (a) The thief must select a subset of the three items shown whose weight must not exceed 50 pounds. (b) The optimal subset includes items 2 and 3. Any solution with item 1 is suboptimal, even though item 1 has the greatest value per pound. (c) For the fractional knapsack problem, taking the items in order of greatest value per pound yields an optimal solution.

Exercises

17.2-1

Prove that the fractional knapsack problem has the greedy-choice property.

17.2-2

Give a dynamic-programming solution to the 0-1 knapsack problem that runs in O(n W) time, where n is number of items and W is the maximum weight of items that the thief can put in his knapsack.

17.2-3

Suppose that in a 0-1 knapsack problem, the order of the items when sorted by increasing weight is the same as their order when sorted by decreasing value. Give an efficient algorithm to find an optimal solution to this variant of the knapsack problem, and argue that your algorithm is correct.

17.2-4

Professor Midas drives an automobile from Newark to Reno along Interstate 80. His car's gas tank, when full, holds enough gas to travel n miles, and his map gives the distances between gas stations on his route. The professor wishes to make as few gas stops as possible along the way. Give an efficient method by which Professor Midas can determine at which gas stations he should stop, and prove that your strategy yields an optimal solution.

17.2-5

Describe an efficient algorithm that, given a set {x1,x2, . . ., xn} of points on the real line, determines the smallest set of unit-length closed intervals that contains all of the given points. Argue that your algorithm is correct.

17.2-6

Show how to solve the fractional knapsack problem in O(n) time. Assume that you have a solution to Problem 10-2.

17.3 Huffman codes

Huffman codes are a widely used and very effective technique for compressing data; savings of 20% to 90% are typical, depending on the characteristics of the file being compressed. Huffman's greedy algorithm uses a table of the frequencies of occurrence of each character to build up an optimal way of representing each character as a binary string.

Suppose we have a 100,000-character data file that we wish to store compactly. We observe that the characters in the file occur with the frequencies given by Figure 17.3. That is, only six different characters appear, and the character a occurs 45,000 times.

There are many ways to represent such a file of information. We consider the problem of designing a binary character code (or code for short) wherein each character is represented by a unique binary string. If we use a fixed-length code, we need 3 bits to represent six characters: a = 000, b = 001, . . . , f = 101. This method requires 300,000 bits to code the entire file. Can we do better?

                             a     b     c     d      e     f

--------------------------------------------------------------

Frequency (in thousands)    45    13    12    16      9      5

Fixed-length codeword      000   001   010   011    100    101

Variable-length codeword    0    101   100   111   1101   1100

Figure 17.3 A character-coding problem. A data file of 100,000 characters contains only the characters a-f, with the frequencies indicated. If each character is assigned a 3-bit codeword, the file can be encoded in 300,000 bits. Using the variable-length code shown, the file can be encoded in 224,000 bits.

A variable-length code can do considerably better than a fixed-length code, by giving frequent characters short codewords and infrequent characters long codewords. Figure 17.3 shows such a code; here the 1-bit string 0 represents a, and the 4-bit string 1100 represents f. This code requires (45 [image] 1 + 13 [image] 3 + 12 [image] 3 + 16 [image] 3 + 9 [image] 4 + 5 [image] 4) [image] 1,000 = 224,000 bits to represent the file, a savings of approximately 25%. In fact, this is an optimal character code for this file, as we shall see.

Prefix codes

We consider here only codes in which no codeword is also a prefix of some other codeword. Such codes are called prefix codes.1 It is possible to show (although we won't do so here) that the optimal data compression achievable by a character code can always be achieved with a prefix code, so there is no loss of generality in restricting attention to prefix codes.

1 Perhaps "prefix-free codes" would be a better name, but the term "prefix codes" is standard in the literature.

Prefix codes are desirable because they simplify encoding (compression) and decoding. Encoding is always simple for any binary character code; we just concatenate the codewords representing each character of the file. For example, with the variable-length prefix code of Figure 17.3, we code the 3-character file abc as 0 [image] 101 [image] 100 = 0101100, where we use " [image] " to denote concatenation.

Decoding is also quite simple with a prefix code. Since no codeword is a prefix of any other, the codeword that begins an encoded file is unambiguous. We can simply identify the initial codeword, translate it back to the original character, remove it from the encoded file, and repeat the decoding process on the remainder of the encoded file. In our example, the string 001011101 parses uniquely as 0 [image] 0 [image] 101 [image] 1101, which decodes to aabe.

The decoding process needs a convenient representation for the prefix code so that the initial codeword can be easily picked off. A binary tree whose leaves are the given characters provides one such representation. We interpret the binary codeword for a character as the path from the root to that character, where 0 means "go to the left child" and 1 means "go to the right child." Figure 17.4 shows the trees for the two codes of our example. Note that these are not binary search trees, since the leaves need not appear in sorted order and internal nodes do not contain character keys.

An optimal code for a file is always represented by a full binary tree, in which every nonleaf node has two children (see Exercise 17.3-1). The fixed-length code in our example is not optimal since its tree, shown in Figure 17.4(a), is not a full binary tree: there are codewords beginning 10 . . . , but none beginning 11 . . . . Since we can now restrict our attention to full binary trees, we can say that if C is the alphabet from which the characters are drawn, then the tree for an optimal prefix code has exactly |C| leaves, one for each letter of the alphabet, and exactly |C| - 1 internal nodes.

Figure 17.4 Trees corresponding to the coding schemes in Figure 17.3. Each leaf is labeled with a character and its frequency of occurrence. Each internal node is labeled with the sum of the weights of the leaves in its subtree. (a) The tree corresponding to the fixed-length code a = 000, . . . , f = 100. (b) The tree corresponding to the optimal prefix code a = 0, b = 101, . . . , f = 1100

Given a tree T corresponding to a prefix code, it is a simple matter to compute the number of bits required to encode a file. For each character c in the alphabet C, let f(c) denote the frequency of c in the file and let dT(c) denote the depth of c's leaf in the tree. Note that dT(c) is also the length of the codeword for character c. The number of bits required to encode a file is thus

(17.3)

which we define as the cost of the tree T.

Constructing a Huffman code

Huffman invented a greedy algorithm that constructs an optimal prefix code called a Huffman code. The algorithm builds the tree T corresponding to the optimal code in a bottom-up manner. It begins with a set of |C| leaves and performs a sequence of |C| - 1 "merging" operations to create the final tree.

In the pseudocode that follows, we assume that C is a set of n characters and that each character c [image] C is an object with a defined frequency f[c]. A priority queue Q, keyed on f, is used to identify the two least-frequent objects to merge together. The result of the merger of two objects is a new object whose frequency is the sum of the frequencies of the two objects that were merged.

HUFFMAN(C)

1 n    |C|

2 Q    C

3 for i    1 to n - 1

4      do z    ALLOCATE-NODE()

5         x    left[z]    EXTRACT-MIN(Q)

6         y    right[z]    EXTRACT-MIN(Q)

7         f[z]    f[x] + f[y]

8         INSERT(Q, z)

9 return EXTRACT-MIN(Q)

For our example, Huffman's algorithm proceeds as shown in Figure 17.5. Since there are 6 letters in the alphabet, the initial queue size is n = 6, and 5 merge steps are required to build the tree. The final tree represents the optimal prefix code. The codeword for a letter is the sequence of edge labels on the path from the root to the letter.

Line 2 initializes the priority queue Q with the characters in C. The for loop in lines 3-8 repeatedly extracts the two nodes x and y of lowest frequency from the queue, and replaces them in the queue with a new node z representing their merger. The frequency of z is computed as the sum of the frequencies of x and y in line 7. The node z has x as its left child and y as its right child. (This order is arbitrary; switching the left and right child of any node yields a different code of the same cost.) After n - 1 mergers, the one node left in the queue--the root of the code tree--is returned in line 9.

The analysis of the running time of Huffman's algorithm assumes that Q is implemented as a binary heap (see Chapter 7). For a set C of n characters, the initialization of Q in line 2 can be performed in O(n) time using the BUILD-HEAP procedure in Section 7.3. The for loop in lines 3-8 is executed exactly |n| - 1 times, and since each heap operation requires time O(1g n), the loop contributes O(n 1g n) to the running time. Thus, the total running time of HUFFMAN on a set of n characters is O(n 1g n).

Correctness of Huffman's algorithm

To prove that the greedy algorithm HUFFMAN is correct, we show that the problem of determining an optimal prefix code exhibits the greedy-choice and optimal-substructure properties. The next lemma shows that the greedy-choice property holds.

Figure 17.5 The steps of Huffman's algorithm for the frequencies given in Figure 17.3. Each part shows the contents of the queue sorted into increasing order by frequency. At each step, the two trees with lowest frequencies are merged. Leaves are shown as rectangles containing a character and its frequency. Internal nodes are shown as circles containing the sum of the frequencies of its children. An edge connecting an internal node with its children is labeled 0 if it is an edge to a left child and 1 if it is an edge to a right child. The codeword for a letter is the sequence of labels on the edges connecting the root to the leaf for that letter. (a) The initial set of n = 6 nodes, one for each letter. (b)-(e) Intermediate stages.(f) The final tree.

Figure 17.6 An illustration of the key step in the proof of Lemma 17.2. In the optimal tree T, leaves b and c are two of the deepest leaves and are siblings. Leaves x and y are the two leaves that Huffman's algorithm merges together first; they appear in arbitrary positions in T. Leaves b and x are swapped to obtain tree T'. Then, leaves c and y are swapped to obtain tree T\". Since each swap does not increase the cost, the resulting tree T\" is also an optimal tree.

Lemma 17.2

Let C be an alphabet in which each character c [image] C has frequency f[c]. Let x and y be two characters in C having the lowest frequencies. Then there exists an optimal prefix code for C in which the codewords for x and y have the same length and differ only in the last bit.

Proof The idea of the proof is to take the tree T representing an arbitrary optimal prefix code and modify it to make a tree representing another optimal prefix code such that the characters x and y appear as sibling leaves of maximum depth in the new tree. If we can do this, then their codewords will have the same length and differ only in the last bit.

Let b and c be two characters that are sibling leaves of maximum depth in T. Without loss of generality, we assume that f[b] [image] f[c] and f[x] [image] f[y]. Since f[x] and f[y] are the two lowest leaf frequencies, in order, and f[b] and f[c] are two arbitrary frequencies, in order, we have f[x] [image] f[b] and f[y] [image] f[c]. As shown in Figure 17.6, we exchange the positions in T of b and x to produce a tree T', and then we exchange the positions in T' of c and y to produce a tree T\". By equation (17.3), the difference in cost between T and T' is

because both f[b] - f[x] and dT[b] - dT[x] are nonnegative. More specifically, f[b] - f[x] is nonnegative because x is a minimum-frequency leaf, and dT[b] - dT[x] is nonnegative because b is a leaf of maximum depth in T. Similarly, because exchanging y and c does not increase the cost, B(T') - B(T\") is nonnegative. Therefore, B(T\") [image] B(T), and since T is optimal, B(T) [image] B(T\"), which implies B(T\") = B(T). Thus, T\" is an optimal tree in which x and y appear as sibling leaves of maximum depth, from which the lemma follows.

Lemma 17.2 implies that the process of building up an optimal tree by mergers can, without loss of generality, begin with the greedy choice of merging together those two characters of lowest frequency. Why is this a greedy choice? We can view the cost of a single merger as being the sum of the frequencies of the two items being merged. Exercise 17.3-3 shows that the total cost of the tree constructed is the sum of the costs of its mergers. Of all possible mergers at each step, HUFFMAN chooses the one that incurs the least cost.

The next lemma shows that the problem of constructing optimal prefix codes has the optimal-substructure property.

Lemma 17.3

Let T be a full binary tree representing an optimal prefix code over an alphabet C, where frequency f [c] is defined for each character c [image] C. Consider any two characters x and y that appear as sibling leaves in T, and let z be their parent. Then, considering z as a character with frequency f[z] = f[x] + f[y], the tree T' = T - {x,y} represents an optimal prefix code for the alphabet C' = C - {x,y} [image] {z}.

Proof We first show that the cost B(T) of tree T can be expressed in terms of the cost B(T') of tree T' by considering the component costs in equation (17.3). For each c [image] C - {x,y}, we have dT(c) = dT'(c), and hence f[c]dT(c) = f[c]dT'(c). Since dT(x) = dT(y) = dT'(z) + 1, we have

f[x]dT(x) + f[y]dT(y) = (f[x]) + (f[y])dT'(z) + 1)

= f[z]dT'(z) + (f[x] + f[y]),

from which we conclude that

B(T) = B(T') + f[x] + f[y].

If T' represents a nonoptimal prefix code for the alphabet C', then there exists a tree T\" whose leaves are characters in C' such that B(T\") < B(T'). Since z is treated as a character in C', it appears as a leaf in T\". If we add x and y as children of z in T\", then we obtain a prefix code for C with cost B(T\") + f[x] + f[y] < B(T), contradicting the optimality of T. Thus, T' must be optimal for the alphabet C'.

Theorem 17.4

Procedure HUFFMAN produces an optimal prefix code.

Proof Immediate from Lemmas 17.2 and 17.3.

Exercises

17.3-1

Prove that a binary tree that is not full cannot correspond to an optimal prefix code.

17.3-2

What is an optimal Huffman code for the following set of frequencies, based on the first 8 Fibonacci numbers?

a:1 b:1 c:2 d:3 e:5 f:8 g:13 h:21

Can you generalize your answer to find the optimal code when the frequencies are the first n Fibonacci numbers?

17.3-3

Prove the total cost of a tree for a code can also be computed as the sum, over all internal nodes, of the combined frequencies of the two children of the node.

17.3-4

Prove that for an optimal code, if the characters are ordered so that their frequencies are nonincreasing, then their codeword lengths are nondecreasing.

17.3-5

Let C = {0, 1, . . . , n - 1} be a set of characters. Show that any optimal prefix code on C can be represented by a sequence of

2n - 1 + n   lg n

bits. (Hint: Use 2n- 1 bits to specify the structure of the tree, as discovered by a walk of the tree.)

17.3-6

Generalize Huffman's algorithm to ternary codewords (i.e., codewords using the symbols 0, 1, and 2), and prove that it yields optimal ternary codes.

17.3-7

Suppose a data file contains a sequence of 8-bit characters such that all 256 characters are about as common: the maximum character frequency is less than twice the minimum character frequency. Prove that Huffman coding in this case is no more efficient than using an ordinary 8-bit fixed-length code.

17.3-8

Show that no compression scheme can expect to compress a file of randomly chosen 8-bit characters by even a single bit. (Hint: Compare the number of files with the number of possible encoded files.)

* 17.4 Theoretical foundations for greedy methods

There is a beautiful theory about greedy algorithms, which we sketch in this section. This theory is useful in determining when the greedy method yields optimal solutions. It involves combinatorial structures known as "matroids." Although this theory does not cover all cases for which a greedy method applies (for example, it does not cover the activity-selection problem of Section 17.1 or the Huffman coding problem of Section 17.3), it does cover many cases of practical interest. Furthermore, this theory is being rapidly developed and extended to cover many more applications; see the notes at the end of this chapter for references.

17.4.1 Matroids

A matroid is an ordered pair [image] satisfying the following conditions.

1. S is a finite nonempty set.

2. is a nonempty family of subsets of S, called the independent subsets of S, such that if [image] and A [image] B, then [image]. We say that [image] is hereditary if it satisfies this property. Note that the empty set [image] is necessarily a member of [image].

3. If [image] , and |A| < |B|, then there is some element x [image] B - A such that [image]. We say that M satisfies the exchange property.

The word "matroid" is due to Hassler Whitney. He was studying matric matroids, in which the elements of S are the rows of a given matrix and a set of rows is independent if they are linearly independent in the usual sense. It is easy to show that this structure defines a matroid (see Exercise 17.4-2).

As another illustration of matroids, consider the graphic matroid [image] defined in terms of a given undirected graph G = ( V, E) as follows.

The set SG is defined to be E, the set of edges of G.

If A is a subset of E, then [image] if and only if A is acyclic. That is, a set of edges is independent if and only if it forms a forest.

The graphic matroid MG is closely related to the minimum-spanning-tree problem, which is covered in detail in Chapter 24.

Theorem 17.5

If G is an undirected graph, then [image] is a matroid.

Proof Clearly, SG = E is a finite set. Furthermore, [image] is hereditary, since a subset of a forest is a forest. Putting it another way, removing edges from an acyclic set of edges cannot create cycles.

Thus, it remains to show that MG satisfies the exchange property. Suppose that A and B are forests of G and that |B| > |A|. That is, A and B are acyclic sets of edges, and B contains more edges than A does.

It follows from Theorem 5.2 that a forest having k edges contains exactly |V|-k trees. (To prove this another way, begin with |V| trees and no edges. Then, each edge that is added to the forest reduces the number of trees by one.) Thus, forest A contains |V| - |A| trees, and forest B contains |V| - |B| trees.

Since forest B has fewer trees than forest A does, forest B must contain some tree T whose vertices are in two different trees in forest A. Moreover, since T is connected, it must contain an edge (u, v) such that vertices u and v are in different trees in forest A. Since the edge (u, v) connects vertices in two different trees in forest A, the edge (u, v) can be added to forest A without creating a cycle. Therefore, MG satisfies the exchange property, completing the proof that MG is a matroid.

Given a matroid [image], we call an element x [image] A an extension of [image] if x can be added to A while preserving independence; that is, x is an extension of A if [image]. As an example, consider a graphic matroid MG. If A is an independent set of edges, then edge e is an extension of A if and only if e is not in A and the addition of x to A does not create a cycle.

If A is an independent subset in a matroid M, we say that A is maximal if it has no extensions. That is, A is maximal if it is not contained in any larger independent subset of M. The following property is often useful.

Theorem 17.6

All maximal independent subsets in a matroid have the same size.

Proof Suppose to the contrary that A is a maximal independent subset of M and there exists another larger maximal independent subset B of M. Then, the exchange property implies that A is extendable to a larger independent set A [image] {x} for some x [image] B - A, contradicting the assumption that A is maximal.

As an illustration of this theorem, consider a graphic matroid MG for a connected, undirected graph G. Every maximal independent subset of MG must be a free tree with exactly |V| - 1 edges that connects all the vertices of G. Such a tree is called a spanning tree of G.

We say that a matroid [image] is weighted if there is an associated weight function w that assigns a strictly positive weight w(x) to each element x [image] S. The weight function w extends to subsets of S by summation:

for any A [image] S. For example, if we let w(e) denote the length of an edge e in a graphic matroid MG, then w(A) is the total length of the edges in edge set A.

17.4.2 Greedy algorithms on a weighted matroid

Many problems for which a greedy approach provides optimal solutions can be formulated in terms of finding a maximum-weight independent subset in a weighted matroid. That is, we are given a weighted matroid [image], and we wish to find an independent set [image] such that w(A) is maximized. We call such a subset that is independent and has maximum possible weight an optimal subset of the matroid. Because the weight w(x) of any element x [image] S is positive, an optimal subset is always a maximal independent subset--it always helps to make A as large as possible.

For example, in the minimum-spanning-tree problem, we are given a connected undirected graph G = (V, E) and a length function w such that w (e) is the (positive) length of edge e. (We use the term "length" here to refer to the original edge weights for the graph, reserving the term "weight" to refer to the weights in the associated matroid.) We are asked to find a subset of the edges that connects all of the vertices together and has minimum total length. To view this as a problem of finding an optimal subset of a matroid, consider the weighted matroid MG with weight function w', where w'(e) = w0 - w(e) and w0 is larger than the maximum length of any edge. In this weighted matroid, all weights are positive and an optimal subset is a spanning tree of minimum total length in the original graph. More specifically, each maximal independent subset A corresponds to a spanning tree, and since

w'(A) = (|V| - 1)w0 - w(A)

for any maximal independent subset A, the independent subset that maximizes w'(A) must minimize w(A). Thus, any algorithm that can find an optimal subset A in an arbitrary matroid can solve the minimum-spanning-tree problem.

Chapter 24 gives algorithms for the minimum-spanning-tree problem, but here we give a greedy algorithm that works for any weighted matroid. The algorithm takes as input a weighted matroid [image] with an an associated positive weight function w, and it returns an optimal subset A. In our pseudocode, we denote the components of M by S[M] and [image] and the weight function by w. The algorithm is greedy because it considers each element x [image] S in turn in order of nonincreasing weight and immediately adds it to the set A being accumulated if A [image] {x} is independent.

The elements of S are considered in turn, in order of nonincreasing weight. If the element x being considered can be added to A while maintaining A's independence, it is. Otherwise, x is discarded. Since the empty set is independent by the definition of a matroid, and since x is only added to A if A [image] {x} is independent, the subset A is always independent, by induction. Therefore, GREEDY always returns an independent subset A. We shall see in a moment that A is a subset of maximum possible weight, so that A is an optimal subset.

The running time of GREEDY is easy to analyze. Let n denote |S|. The sorting phase of GREEDY takes time O(n lg n). Line 4 is executed exactly n times, once for each element of S. Each execution of line 4 requires a check on whether or not the set A [image] {x} is independent. If each such check takes time O(f(n)), the entire algorithm runs in time O(n lg n + nf(n)).

We now prove that GREEDY returns an optima1 subset.

Lemma 17.7

Suppose that [image] is a weighted matroid with weight function w and that S is sorted into nonincreasing order by weight. Let x be the first element of S such that {x} is independent, if any such x exists. If x exists, then there exists an optimal subset A of S that contains x.

Proof If no such x exists, then the only independent subset is the empty set and we're done. Otherwise, let B be any nonempty optimal subset. Assume that x [image] B; otherwise, we let A = B and we're done.

No element of B has weight greater than w(x). To see this, observe that y [image] B implies that {y} is independent, since [image] and I is hereditary. Our choice of x therefore ensures that w(x) [image] w(y) for any y [image] B.

Construct the set A as follows. Begin with A = {x}. By the choice of x, A is independent. Using the exchange property, repeatedly find a new element of B that can be added to A until |A| = |B| while preserving the independence of A. Then, A = B - {y} [image] {x} for some y [image] B, and so

w(A) = w(B) - w(y) + w(x)

   w(B) .

Because B is optimal, A must also be optimal, and because x [image] A, the lemma is proven.

We next show that if an element is not an option initially, then it cannot be an option later.

Lemma 17.8

Let [image] be any matroid. If x is an element of S such that x is not an extension of [image], then x is not an extension of any independent subset A of S.

Proof The proof is by contradiction. Assume that x is an extension of A but not of [image]. Since x is an extension of A, we have that A [image] {x} is independent. Since [image] is hereditary, {x} must be independent, which contradicts the assumption that x is not an extension of [image].

Lemma 17.8 says that any element that cannot be used immediately can never be used. Therefore, GREEDY cannot make an error by passing over any initial elements in S that are not an extension of [image], since they can never be used.

Lemma 17.9

Let x be the first element of S chosen by GREEDY for the weighted matroid [image]. The remaining problem of finding a maximum-weight independent subset containing x reduces to finding a maximum-weight independent subset of the weighted matroid [image], where

the weight function for M' is the weight function for M, restricted to S'. (We call M' the contraction of M by the element x.)

Proof If A is any maximum-weight independent subset of M containing x, then A' = A - {x} is an independent subset of M'. Conversely, any independent subset A' of M' yields an independent subset A = A' [image] {x} of M. Since we have in both cases that w(A) = w(A') + w(x), a maximum-weight solution in M containing x yields a maximum-weight solution in M', and vice versa.

Theorem 17.10

If [image] is a weighted matroid with weight function w, then the call GREEDY(M, w) returns an optimal subset.

Proof By Lemma 17.8, any elements that are passed over initially because they are not extensions of [image] can be forgotten about, since they can never be useful. Once the first element x is selected, Lemma 17.7 implies that GREEDY does not err by adding x to A, since there exists an optimal subset containing x. Finally, Lemma 17.9 implies that the remaining problem is one of finding an optimal subset in the matroid M' that is the contraction of M by x. After the procedure GREEDY sets A to{x}, all of its remaining steps can be interpreted as acting in the matroid [image], because B is independent in M' if and only if B [image] {x} is independent in M, for all sets [image]. Thus, the subsequent operation of GREEDY will find a maximum-weight independent subset for M', and the overall operation of GREEDY will find a maximum-weight independent subset for M.

Exercises

17.4-1

Show that [image] is a matroid, where S is any finite set and [image] is the set of all subsets of S of size at most k, where k [image] |S|.

17.4-2

Given an n [image] n real-valued matrix T, show that [image] is a matroid, where S is the set of columns of T and [image] if and only if the columns in A are linearly independent.

17.4-3

Show that if [image] is a matroid, then [image] is a matroid, where [image] contains some maximal [image]. That is, the maximal independent sets of [image] are just the complements of the maximal independent sets of [image].

17.4-4

Let S be a finite set and let S1, S2, . . . , Sk be a partition of S into nonempty disjoint subsets. Define the structure [image] by the condition that [image]. Show that [image] is a matroid. That is, the set of all sets A that contain at most one member in each block of the partition determines the independent sets of a matroid.

17.4-5

Show how to transform the weight function of a weighted matroid problem, where the desired optimal solution is a minimum-weight maximal independent subset, to make it an standard weighted-matroid problem. Argue carefully that your transformation is correct.

* 17.5 A task-scheduling problem

An interesting problem that can be solved using matroids is the problem of optimally scheduling unit-time tasks on a single processor, where each task has a deadline and a penalty that must be paid if the deadline is missed. The problem looks complicated, but it can be solved in a surprisingly simple manner using a greedy algorithm.

A unit-time task is a job, such as a program to be run on a computer, that requires exactly one unit of time to complete. Given a finite set S of unit-time tasks, a schedule for S is a permutation of S specifying the order in which these tasks are to be performed. The first task in the schedule begins at time 0 and finishes at time 1, the second task begins at time 1 and finishes at time 2, and so on.

The problem of scheduling unit-time tasks with deadlines and penalties for a single processor has the following inputs:

a set S = {1, 2, . . . , n} of n unit-time tasks;

a set of n integer deadlines d1, d2, . . . , dn, such that each di satisfies 1 [image] di [image] n and task i is supposed to finish by time di; and

a set of n nonnegative weights or penalties w1,w2, . . . , wn, such that a penalty wi is incurred if task i is not finished by time di and no penalty is incurred if a task finishes by its deadline.

We are asked to find a schedule for S that minimizes the total penalty incurred for missed deadlines.

Consider a given schedule. We say that a task is late in this schedule if it finishes after its deadline. Otherwise, the task is early in the schedule. An arbitrary schedule can always be put into early-first form, in which the early tasks precede the late tasks. To see this, note that if some early task x follows some late task y, then we can switch the positions of x and y without affecting x being early or y being late.

We similarly claim that an arbitrary schedule can always be put into canonical form, in which the early tasks precede the late tasks and the early tasks are scheduled in order of nondecreasing deadlines. To do so, we put the schedule into early-first form. Then, as long as there are two early tasks i and j finishing at respective times k and k + 1 in the schedule such that dj < di, we swap the positions of i and j. Since task j is early before the swap, k + 1 [image] dj. Therefore, k + 1 < di, and so task i is still early after the swap. Task j is moved earlier in the schedule, so it also still early after the swap.

The search for an optimal schedule thus reduces to finding a set A of tasks that are to be early in the optimal schedule. Once A is determined, we can create the actual schedule by listing the elements of A in order of nondecreasing deadline, then listing the late tasks (i.e., S - A) in any order, producing a canonical ordering of the optimal schedule.

We say that a set A of tasks is independent if there exists a schedule for these tasks such that no tasks are late. Clearly, the set of early tasks for a schedule forms an independent set of tasks. Let [image] denote the set of all independent sets of tasks.

Consider the problem of determining whether a given set A of tasks is independent. For t = 1, 2, . . . , n, let Nt(A) denote the number of tasks in A whose deadline is t or earlier.

Lemma 17.11

For any set of tasks A, the following statements are equivalent.

1. The set A is independent.

2. For t = 1, 2, . . . , n, we have Nt(A) [image] t.

3. If the tasks in A are scheduled in order of nondecreasing deadlines, then no task is late.

Proof Clearly, if Nt(A) > t for some t, then there is no way to make a schedule with no late tasks for set A, because there are more than t tasks to finish before time t. Therefore, (1) implies (2). If (2) holds, then (3) must follow: there is no way to "get stuck" when scheduling the tasks in order of nondecreasing deadlines, since (2) implies that the ith largest deadline is at most i. Finally, (3) trivially implies (1).

Using property 2 of Lemma 17.11, we can easily compute whether or not a given set of tasks is independent (see Exercise 17.5-2).

The problem of minimizing the sum of the penalties of the late tasks is the same as the problem of maximizing the sum of the penalties of the early tasks. The following theorem thus ensures that we can use the greedy algorithm to find an independent set A of tasks with the maximum total penalty.

Theorem 17.12

If S is a set of unit-time tasks with deadlines, and [image] is the set of all independent sets of tasks, then the corresponding system [image] is a matroid.

Proof Every subset of an independent set of tasks is certainly independent. To prove the exchange property, suppose that B and A are independent sets of tasks and that |B| > |A|. Let k be the largest t such that Nt(B) [image] Nt(A). Since Nn(B) = |B| and Nn(A) = |A|, but |B| > |A|, we must have that k < n and that Nj(B) > Nj(A) for all j in the range k + 1 [image] j [image] n. Therefore, B contains more tasks with deadline k + 1 than A does. Let x be a task in B - A with deadline k + 1. Let A' = A = A [image] {x}.

We now show that A' must be independent by using property 2 of Lemma 17.11. For 1 [image] t [image] k, we have Nt(A') = Nt(A) [image] t,since A is independent. For k < t [image] n, we have Nt(A') [image] Nt(B) [image] t, since B is independent. Therefore, A' is independent, completing our proof that [image] is a matroid.

By Theorem 17.10, we can use a greedy algorithm to find a maximum- weight independent set of tasks A. We can then create an optimal schedule having the tasks in A as its early tasks. This method is an efficient algorithm for scheduling unit-time tasks with deadlines and penalties for a single processor. The running time is O(n2) using GREEDY, since each of the O(n) independence checks made by that algorithm takes time O(n) (see Exercise 17.5-2). A faster implementation is given in Problem 17-3.

                Task

     1   2   3   4    5    6  7

-------------------------------

di   4   2   4   3    1    4   6

wi  70  60  50  40   30   20  10

Figure 17.7 An instance of the problem of scheduling unit-time tasks with deadlines and penalties for a single processor.

Figure 17.7 gives an example of a problem of scheduling unit-time tasks with deadlines and penalties for a single processor. In this example, the greedy algorithm selects tasks 1, 2, 3, and 4, then rejects tasks 5 and 6, and finally accepts task 7. The final optimal schedule is

  2, 4, 1, 3, 7, 5, 6   ,

which has a total penalty incurred of w5 +w6 = 50.

Exercises

17.5-1

Solve the instance of the scheduling problem given in Figure 17.7, but with each penalty wi replaced by 80 - wi.

17.5-2

Show how to use property 2 of Lemma 17.11 to determine in time O(|A|) whether or not a given set A of tasks is independent.

Problems

17-1 Coin changing

Consider the problem of making change for n cents using the least number of coins.

a. Describe a greedy algorithm to make change consisting of quarters, dimes, nickels, and pennies. Prove that your algorithm yields an optimal solution.

b. Suppose that the available coins are in the denominations c0, c1, . . . , ck for some integers c >1 and k [image] 1. Show that the greedy algorithm always yields an optimal solution.

c. Give a set of coin denominations for which the greedy algorithm does not yield an optimal solution.

17-2 Acyclic subgraphs

a. Let G = (V, E) be an undirected graph. Using the definition of a matroid, show that [image] ) is a matroid, where [image] if and only if A is an acyclic subset of E.

b. The incidence matrix for an undirected graph G = (V, E) is a |V| X |E| matrix M such that Mve = 1 if edge e is incident on vertex v, and Mve = 0 otherwise. Argue that a set of columns of M is linearly independent if and only if the corresponding set of edges is acyclic. Then, use the result of Exercise 17.4-2 to provide an alternate proof that [image] of part (a) is matroid.

c. Suppose that a nonnegative weight w(e) is associated with each edge in an undirected graph G = ( V, E). Give an efficient algorithm to find an an acyclic subset of E of maximum total weight.

d. Let G(V, E) be an arbitrary directed graph, and let [image]) be defined so that [image] if and only if A does not contain any directed cycles. Give an example of a directed graph G such that the associated system [image]) is not a matroid. Specify which defining condition for a matroid fails to hold.

e. The incidence matrix for a directed graph G= (V, E) is a |V| X |E| matrix M such that Mve = -1 if edge e leaves vertex v, Mve = 1 if edge e enters vertex v, and and Mve = 0 otherwise. Argue that if a set of edges of G is linearly independent, then the corresponding set of edges does not contain a directed cycle.

f. Exercise 17.4-2 tells us that the set of linearly independent sets of columns of any matrix M forms a matroid. Explain carefully why the results of parts (d) and (e) are not contradictory. How can there fail to be a perfect correspondence between the notion of a set of edges being acyclic and the notion of the associated set of columns of the incidence matrix being linearly independent?

17-3 Scheduling variations

Consider the following algorithm for solving the problem in Section 17.5 of scheduling unit-time tasks with deadlines and penalties. Let all n time slots be initially empty, where time slot i is the unit-length slot of time that finishes at time i. We consider the jobs in order of monotonically decreasing penalty. When considering job j, if there exists a time slot at or before j's deadline dj that is still empty, assign job j to the latest such slot, filling it. If there is no such slot, assign job j to the latest of the as yet unfilled slots.

a. Argue that this algorithm always gives an optional answer.

b. Use the fast disjoint-set forest presented in Section 22.3 to implement the algorithm efficiently. Assume that the set of input jobs has already been sorted into monotonically decreasing order by penalty. Analyze the running time of your implementation.

Chapter notes

Much more material on greedy algorithms and matroids can be found in Lawler [132] and Papadimitriou and Steiglitz [154].

The greedy algorithm first appeared in the combinatorial optimization literature in a 1971 article by Edmonds [62], though the theory of matroids dates back to a 1935 article by Whitney [200].

Our proof of the correctness of the greedy algorithm for the activity-selection problem follows that of Gavril [80]. The task-scheduling problem is studied in Lawler [132], Horowitz and Sahni [105], and Brassard and Bratley [33].

Huffman codes were invented in 1952 [107]; Lelewer and Hirschberg [136] surveys data-compression techniques known as of 1987.

An extension of matroid theory to greedoid theory was pioneered by Korte and Lovász [127, 128, 129, 130], who greatly generalize the theory presented here.

Go to Chapter 18 Back to Table of Contents


Intro to Algorithms: CHAPTER 18: AMORTIZED ANALYSIS

 Return to Table of Contents

CHAPTER 18: AMORTIZED ANALYSIS

In an amortized analysis, the time required to perform a sequence of data-structure operations is averaged over all the operations performed. Amortized analysis can be used to show that the average cost of an operation is small, if one averages over a sequence of operations, even though a single operation might be expensive. Amortized analysis differs from average-case analysis in that probability is not involved; an amortized analysis guarantees the average performance of each operation in the worst case.

The first three sections of this chapter cover the three most common techniques used in amortized analysis. Section 18.1 starts with the aggregate method, in which we determine an upper bound T(n) on the total cost of a sequence of n operations. The amortized cost per operation is then T(n)/n.

Section 18.2 covers the accounting method, in which we determine an amortized cost of each operation. When there is more than one type of operation, each type of operation may have a different amortized cost. The accounting method overcharges some operations early in the sequence, storing the overcharge as "prepaid credit" on specific objects in the data structure. The credit is used later in the sequence to pay for operations that are charged less than they actually cost.

Section 18.3 discusses the potential method, which is like the accounting method in that we determine the amortized cost of each operation and may overcharge operations early on to compensate for undercharges later. The potential method maintains the credit as the "potential energy" of the data structure instead of associating the credit with individual objects within the data structure.

We shall use two examples to examine these three models. One is a stack with the additional operation MULTIPOP, which pops several objects at once. The other is a binary counter that counts up from 0 by means of the single operation INCREMENT.

While reading this chapter, bear in mind that the charges assigned during an amortized analysis are for analysis purposes only. They should not appear in the code. If, for example, a credit is assigned to an object x when using the accounting method, there is no need to assign an appropriate amount to some attribute credit[x] in the code.

The insight into a particular data structure gained by performing an amortized analysis can help in optimizing the design. In Section 18.4, for example, we shall use the potential method to analyze a dynamically expanding and contracting table.

18.1 The aggregate method

In the aggregate method of amortized analysis, we show that for all n, a sequence of n operations takes worst-case time T(n) in total. In the worst case, the average cost, or amortized cost, per operation is therefore T(n) / n. Note that this amortized cost applies to each operation, even when there are several types of operations in the sequence. The other two methods we shall study in this chapter, the accounting method and the potential method, may assign different amortized costs to different types of operations.

Stack operations

In our first example of the aggregate method, we analyze stacks that have been augmented with a new operation. Section 11.1 presented the two fundamental stack operations, each of which takes O(1) time:

PUSH(S, x) pushes object x onto stack S.

POP(S) pops the top of stack S and returns the popped object.

Since each of these operations runs in O(1) time, let us consider the cost of each to be 1. The total cost of a sequence of n PUSH and POP operations is therefore n, and the actual running time for n operations is therefore [image] (n).

The situation becomes more interesting if we add the stack operation MULTIPOP(S, k), which removes the k top objects of stack S, or pops the entire stack if it contains less than k objects. In the following pseudocode, the operation STACK-EMPTY returns TRUE if there are no objects currently on the stack, and FALSE otherwise.

MULTIPOP(S,k)

1  while not STACK-EMPTY(S) and k    0

2      do POP(S)

3         k    k - 1

Figure 18.1 shows an example of MULTIPOP.

What is the running time of MULTIPOP(S, k) on a stack of s objects? The actual running time is linear in the number of POP operations actually executed, and thus it suffices to analyze MULTIPOP in terms of the abstract costs of 1 each for PUSH and POP. The number of iterations of the while loop is the number min(s, k) of objects popped off the stack. For each iteration of the loop, one call is made to POP in line 2. Thus, the total cost of MULTIPOP is min(s, k), and the actual running time is a linear function of this cost.

Figure 18.1 The action of MULTIPOP on a stack S, shown initially in (a). The top 4 objects are popped by MULTIPOP(S, 4), whose result is shown in (b). The next operation is MULTIPOP(S, 7), which empties the stack shown in (c) since there were fewer than 7 objects remaining.

Let us analyze a sequence of n PUSH, POP, and MULTIPOP operations on an initially empty stack. The worst-case cost of a MULTIPOP operation in the sequence is O(n), since the stack size is at most n. The worst-case time of any stack operation is therefore O(n), and hence a sequence of n operations costs O(n2), since we may have O(n) MULTIPOP operations costing O(n) each. Although this analysis is correct, the O(n2) result, obtained by considering the worst-case cost of each operation individually, is not tight.

Using the aggregate method of amortized analysis, we can obtain a better upper bound that considers the entire sequence of n operations. In fact, although a single MULTIPOP operation can be expensive, any sequence of n PUSH, POP, and MULTIPOP operations on an initially empty stack can cost at most O(n). Why? Each object can be popped at most once for each time it is pushed. Therefore, the number of times that POP can be called on a nonempty stack, including calls within MULTIPOP, is at most the number of PUSH operations, which is at most n. For any value of n, any sequence of n PUSH, POP, and MULTIPOP operations takes a total of O(n) time. The amortized cost of an operation is the average: O(n)/n = O(1).

We emphasize again that although we have just shown that the average cost, and hence running time, of a stack operation is O(1), no probabilistic reasoning was involved. We actually showed a worst-case bound of O(n) on a sequence of n operations. Dividing this total cost by n yielded the average cost per operation, or the amortized cost.

Incrementing a binary counter

As another example of the aggregate method, consider the problem of implementing a k-bit binary counter that counts upward from 0. We use an array A[0 . . k - 1] of bits, where length[A] = k, as the counter. A binary number x that is stored in the counter has its lowest-order bit in A[0] and its highest-order bit in A[k - 1], so that [image]. Initially, x = 0, and thus A[i] = 0 for i = 0, 1, . . . , k - 1. To add 1 (modulo 2k) to the value in the counter, we use the following procedure.

Figure 18.2 An 8-bit binary counter as its value goes from 0 to 16 by a sequence of 16 INCREMENT operations. Bits that flip to achieve the next value are shaded. The running cost for flipping bits is shown at the right. Notice that the total cost is never more than twice the total number of INCREMENT operations.

INCREMENT(A)

1  i    0

2  while i< length[A] and A[i] = 1

3       do A[i]    0

4          i    i + 1

5  if i < length[A]

6      then A[i]    1

This algorithm is essentially the same one implemented in hardware by a ripple-carry counter (see Section 29.2.1). Figure 18.2 shows what happens to a binary counter as it is incremented 16 times, starting with the initial value 0 and ending with the value 16. At the start of each iteration of the while loop in lines 2-4, we wish to add a 1 into position i. If A[i] = 1, then adding 1 flips the bit to 0 in position i and yields a carry of 1, to be added into position i + 1 on the next iteration of the loop. Otherwise, the loop ends, and then, if i < k, we know that A[i] = 0, so that adding a 1 into position i, flipping the 0 to a 1, is taken care of in line 6. The cost of each INCREMENT operation is linear in the number of bits flipped.

As with the stack example, a cursory analysis yields a bound that is correct but not tight. A single execution of INCREMENT takes time [image] (k) in the worst case, in which array A contains all 1's. Thus, a sequence of n INCREMENT operations on an initially zero counter takes time O(nk) in the worst case.

We can tighten our analysis to yield a worst-case cost of O(n) for a sequence of n INCREMENT'S by observing that not all bits flip each time INCREMENT is called. As Figure 18.2 shows, A[0] does flip each time INCREMENT is called. The next-highest-order bit, A[1], flips only every other time: a sequence of n INCREMENT operations on an initially zero counter causes A[1] to flip [image] n/2 [image] times. Similarly, bit A[2] flips only every fourth time, or [image] n/4 [image] times in a sequence of n INCREMENT'S. In general, for i = 0, 1, . . . , [image] lg n [image] , bit A[i] flips [image] n/2i [image] times in a sequence of n INCREMENT operations on an initially zero counter. For i > [image] lg n [image] , bit A[i] never flips at all. The total number of flips in the sequence is thus

by equation (3.4). The worst-case time for a sequence of n INCREMENT operations on an initially zero counter is therefore O(n), so the amortized cost of each operation is O(n)/n = O(1).

Exercises

18.1-1

If a MULTIPUSH operation were included in the set of stack operations, would the O(1) bound on the amortized cost of stack operations continue to hold?

18.1-2

Show that if a DECREMENT operation were included in the k-bit counter example, n operations could cost as much as [image] (nk) time.

18.1-3

A sequence of n operations is performed on a data structure. The ith operation costs i if i is an exact power of 2, and 1 otherwise. Use an aggregate method of analysis to determine the amortized cost per operation.

18.2 The accounting method

In the accounting method of amortized analysis, we assign differing charges to different operations, with some operations charged more or less than they actually cost. The amount we charge an operation is called its amortized cost. When an operation's amortized cost exceeds its actual cost, the difference is assigned to specific objects in the data structure as credit. Credit can be used later on to help pay for operations whose amortized cost is less than their actual cost. Thus, one can view the amortized cost of an operation as being split between its actual cost and credit that is either deposited or used up. This is very different from the aggregate method, in which all operations have the same amortized cost.

One must choose the amortized costs of operations carefully. If we want analysis with amortized costs to show that in the worst case the average cost per operation is small, the total amortized cost of a sequence of operations must be an upper bound on the total actual cost of the sequence. Moreover, as in the aggregate method, this relationship must hold for all sequences of operations. Thus, the total credit associated with the data structure must be nonnegative at all times, since it represents the amount by which the total amortized costs incurred exceed the total actual costs incurred. If the total credit were ever allowed to become negative (the result of undercharging early operations with the promise of repaying the account later on), then the total amortized costs incurred at that time would be below the total actual costs incurred; for the sequence of operations up to that time, the total amortized cost would not be an upper bound on the total actual cost. Thus, we must take care that the total credit in the data structure never becomes negative.

Stack operations

To illustrate the accounting method of amortized analysis, let us return to the stack example. Recall that the actual costs of the operations were

PUSH     1 ,

POP      1 ,

MULTIPOP  min(k,s) ,

where k is the argument supplied to MULTIPOP and s is the stack size when it is called. Let us assign the following amortized costs:

PUSH     2 ,

POP      0 ,

MULTIPOP  0 .

Note that the amortized cost of MULTIPOP is a constant (0), whereas the actual cost is variable. Here, all three amortized costs are O(l), although in general the amortized costs of the operations under consideration may differ asymptotically.

We shall now show that we can pay for any sequence of stack operations by charging the amortized costs. Suppose we use a dollar bill to represent each unit of cost. We start with an empty stack. Recall the analogy of Section 11.1 between the stack data structure and a stack of plates in a cafeteria. When we push a plate on the stack, we use 1 dollar to pay the actual cost of the push and are left with a credit of 1 dollar (out of the 2 dollars charged), which we put on top of the plate. At any point in time, every plate on the stack has a dollar of credit on it.

The dollar stored on the plate is prepayment for the cost of popping it from the stack. When we execute a POP operation, we charge the operation nothing and pay its actual cost using the credit stored in the stack. To pop a plate, we take the dollar of credit off the plate and use it to pay the actual cost of the operation. Thus, by charging the PUSH operation a little bit more, we needn't charge the POP operation anything.

Moreover, we needn't charge MULTIPOP operations anything either. To pop the first plate, we take the dollar of credit off the plate and use it to pay the actual cost of a POP operation. To pop a second plate, we again have a dollar of credit on the plate to pay for the POP operation, and so on. Thus, we have always charged at least enough up front to pay for MULTIPOP operations. In other words, since each plate on the stack has 1 dollar of credit on it, and the stack always has a nonnegative number of plates, we have ensured that the amount of credit is always nonnegative. Thus, for any sequence of n PUSH, POP, and MULTIPOP operations, the total amortized cost is an upper bound on the total actual cost. Since the total amortized cost is O(n), so is the total actual cost.

Incrementing a binary counter

As another illustration of the accounting method, we analyze the INCREMENT operation on a binary counter that starts at zero. As we observed earlier, the running time of this operation is proportional to the number of bits flipped, which we shall use as our cost for this example. Let us once again use a dollar bill to represent each unit of cost (the flipping of a bit in this example).

For the amortized analysis, let us charge an amortized cost of 2 dollars to set a bit to 1. When a bit is set, we use 1 dollar (out of the 2 dollars charged) to pay for the actual setting of the bit, and we place the other dollar on the bit as credit. At any point in time, every 1 in the counter has a dollar of credit on it, and thus we needn't charge anything to reset a bit to 0; we just pay for the reset with the dollar bill on the bit.

The amortized cost of INCREMENT can now be determined. The cost of resetting the bits within the while loop is paid for by the dollars on the bits that are reset. At most one bit is set, in line 6 of INCREMENT, and therefore the amortized cost of an INCREMENT operation is at most 2 dollars. The number of l's in the counter is never negative, and thus the amount of credit is always nonnegative. Thus, for n INCREMENT operations, the total amortized cost is O(n), which bounds the total actual cost.

Exercises

18.2-1

A sequence of stack operations is performed on a stack whose size never exceeds k. After every k operations, a copy of the entire stack is made for backup purposes. Show that the cost of n stack operations, including copying the stack, is O(n) by assigning suitable amortized costs to the various stack operations.

18.2-2

Redo Exercise 18.1-3 using an accounting method of analysis.

18.2-3

Suppose we wish not only to increment a counter but also to reset it to zero (i.e., make all bits in it 0). Show how to implement a counter as a bit vector so that any sequence of n INCREMENT and RESET operations takes time O(n) on an initially zero counter. (Hint: Keep a pointer to the high-order 1.)

18.3 The potential method

Instead of representing prepaid work as credit stored with specific objects in the data structure, the potential method of amortized analysis represents the prepaid work as "potential energy,"or just "potential," that can be released to pay for future operations. The potential is associated with the data structure as a whole rather than with specific objects within the data structure.

The potential method works as follows. We start with an initial data structure D0 on which n operations are performed. For each i = 1, 2, . . . , n, we let ci be the actual cost of the ith operation and Di be the data structure that results after applying the ith operation to data structure Di - l. A potential function [image] maps each data structure Di to a real number [image] (Di), which is the potential associated with data structure Di. The amortized cost [image] of the ith operation with respect to potential function [image] is defined by

(18.1)

The amortized cost of each operation is therefore its actual cost plus the increase in potential due to the operation. By equation (18.1), the total amortized cost of the n operations is

(18.2)

The second equality follows from equation (3.7), since the [image] (Di) telescope.

If we can define a potential function [image] so that [image] (Dn) [image] [image] (D0), then the total amortized cost [image] is an upper bound on the total actual cost. In practice, we do not always know how many operations might be performed. Therefore, if we require that [image] (Di) [image] [image] (D0) for all i, then we guarantee, as in the accounting method, that we pay in advance. It is often convenient to define [image] (D0) to be 0 and then to show that [image] (Di) [image] 0 for all i. (See Exercise 18.3-1 for an easy way to handle cases in which [image] (D0) [image] 0.)

Intuitively, if the potential difference [image] (Di) - [image] (Di - 1) of the ith operation is positive, then the amortized cost [image] represents an overcharge to the ith operation, and the potential of the data structure increases. If the potential difference is negative, then the amortized cost represents an undercharge to the ith operation, and the actual cost of the operation is paid by the decrease in the potential.

The amortized costs defined by equations (18.1) and (18.2) depend on the choice of the potential function [image]. Different potential functions may yield different amortized costs yet still be upper bounds on the actual costs. There are often trade-offs that can be made in choosing a potential function; the best potential function to use depends on the desired time bounds.

Stack operations

To illustrate the potential method, we return once again to the example of the stack operations PUSH, POP, and MULTIPOP. We define the potential function [image] on a stack to be the number of objects in the stack. For the empty stack D0 with which we start, we have [image] (D0) = 0. Since the number of objects in the stack is never negative, the stack Di that results after the ith operation has nonnegative potential, and thus

  (Di)      0

=    (D0).

The total amortized cost of n operations with respect to [image] therefore represents an upper bound on the actual cost.

Let us now compute the amortized costs of the various stack operations. If the ith operation on a stack containing s objects is a PUSH operation, then the potential difference is

  (Di) -   (Di - 1)  =  (s + 1 ) - s

=  1 .

By equation (18.1), the amortized cost of this PUSH operation is

Suppose that the ith operation on the stack is MULTIPOP(S,k) and that k' = min(k,s) objects are popped off the stack. The actual cost of the operation is k', and the potential difference is

  (Di) -   (Di-1) = -k'.

Thus, the amortized cost of the MULTIPOP operation is

Similarly, the amortized cost of an ordinary POP operation is 0.

The amortized cost of each of the three operations is O(1), and thus the total amortized cost of a sequence of n operations is O(n). Since we have already argued that [image] (Di) [image] [image] (D0), the total amortized cost of n operations is an upper bound on the total actual cost. The worst-case cost of n operations is therefore O(n).

Incrementing a binary counter

As another example of the otential method, we again look at incrementing a binary counter. This time, we define the potential of the counter after the ith INCREMENT operation to be bi, the number of 1's in the counter after the ith operation.

Let us compute the amortized cost of an INCREMENT operation. Suppose that the ith INCREMENT operation resets ti bits. The actual cost of the operation is therefore at most ti +1, since in addition to resetting ti bits, it sets at most one bit to a 1. The number of 1's in the counter after the ith operation is therefore bi [image] bi-1 - ti +1, and the potential difference is

   (Di) -   (Di-1)      (bi-1 - ti + 1) - bi-1

=  1 - ti.

The amortized cost is therefore

If the counter starts at zero, then [image] (D0) = 0. Since [image] (Di) [image] 0 for all i, the total amortized cost of a sequence of n INCREMENT operations is an upper bound on the total actual cost, and so the worst-case cost of n INCREMENT operations is O(n).

The potential method gives us an easy way to analyze the counter even when it does not start at zero. There are initially b0 1's, and after n INCREMENT operations there are bn 1's, where 0 [image] b0, bn [image] k. We can rewrite equation (18.2) as

(18.3)

We have [image] for all 1 [image] i [image] n. Since [image] (D0) = b0 and [image] (Dn) = bn, the total actual cost of n INCREMENT operations is

Note in particular that since b0 [image] k, if we execute at least n = [image] (k) INCREMENT operations, the total actual cost is O(n), no matter what initial value the counter contains.

Exercises

18.3-1

Suppose we have a potential function [image] such that [image] (Di) [image] [image] (D0) for all i, but [image] (D0) [image] 0. Show that there exists a potential function [image] ' such that [image] '(D0) = 0, [image] '(Di) [image] 0 for all i [image] 1, and the amortized costs using [image] ' are the same as the amortized costs using [image].

18.3-2

Redo Exercise 18.1-3 using a potential method of analysis.

18.3-3

Consider an ordinary binary heap data structure with n elements that supports the instructions INSERT and EXTRACT-MIN in O(lg n) worst-case time. Give a potential function [image] such that the amortized cost of INSERT is O(lg n) and the amortized cost of EXTRACT-MIN is O(1), and show that it works.

18.3-4

What is the total cost of executing n of the stack operations PUSH, POP, and MULTIPOP, assuming that the stack begins with s0 objects and finishes with sn objects?

18.3-5

Suppose that a counter begins at a number with b 1's in its binary representation, rather than at 0. Show that the cost of performing n INCREMENT operations is O( n) if n = [image] (b). (Do not assume that b is constant.)

18.3-6

Show how to implement a queue with two ordinary stacks (Exercise 11.1-6) so that the amortized cost of each ENQUEUE and each DEQUEUE operation is O(1).

18.4 Dynamic tables

In some applications, we do not know in advance how many objects will be stored in a table. We might allocate space for a table, only to find out later that it is not enough. The table must then be reallocated with a larger size, and all objects stored in the original table must be copied over into the new, larger table. Similarly, if many objects have been deleted from the table, it may be worthwhile to reallocate the table with a smaller size. In this section, we study this problem of dynamically expanding and contracting a table. Using amortized analysis, we shall show that the amortized cost of insertion and deletion is only O(1), even though the actual cost of an operation is large when it triggers an expansion or a contraction. Moreover, we shall see how to guarantee that the unused space in a dynamic table never exceeds a constant fraction of the total space.

We assume that the dynamic table supports the operations TABLE-INSERT and TABLE-DELETE. TABLE-INSERT inserts into the table an item that occupies a single slot, that is, a space for one item. Likewise, TABLE-DELETE can be thought of as removing an item from the table, thereby freeing a slot. The details of the data-structuring method used to organize the table are unimportant; we might use a stack (Section 11.1), a heap (Section 7.1), or a hash table (Chapter 12). We might also use an array or collection of arrays to implement object storage, as we did in Section 11.3.

We shall find it convenient to use a concept introduced in our analysis of hashing (Chapter 12). We define the load factor [image] (T) of a nonempty table T to be the number of items stored in the table divided by the size (number of slots) of the table. We assign an empty table (one with no items) size 0, and we define its load factor to be 1. If the load factor of a dynamic table is bounded below by a constant, the unused space in the table is never more than a constant fraction of the total amount of space.

We start by analyzing a dynamic table in which only insertions are performed. We then consider the more general case in which both insertions and deletions are allowed.

18.4.1 Table expansion

Let us assume that storage for a table is allocated as an array of slots. A table fills up when all slots have been used or, equivalently, when its load factor is 1.1 In some software environments, if an attempt is made to insert an item into a full table, there is no alternative but to abort with an error. We shall assume, however, that our software environment, like many modern ones, provides a memory-management system that can allocate and free blocks of storage on request. Thus, when an item is inserted into a full table, we can expand the table by allocating a new table with more slots than the old table had and then copy items from the old table into the new one.

1In some situations, such as an open-address hash table, we may wish to consider a table to be full if its load factor equals some constant strictly less than 1. (See Exercise 18.4-2.)

A common heuristic is to allocate a new table that has twice as many slots as the old one. If only insertions are performed, the load factor of a table is always at least 1/2, and thus the amount of wasted space never exceeds half the total space in the table.

In the following pseudocode, we assume that T is an object representing the table. The field table[T] contains a pointer to the block of storage representing the table. The field num[T]contains the number of items in the table, and the field size[T] is the total number of slots in the table. Initially, the table is empty: num[T] = size[T] = 0.

TABLE-INSERT(T,x)

1  if size[T] = 0

2      then allocate table [T] with 1 slot

3           size[T]    1

4  if num[T] = size[T]

5      then allocate new-table with 2    size[T] slots

6           insert all items in table[T] into new-table

7           free table[T]

8           table[T]    new-table

9           size[T]    2    size[T]

10  insert x into table[T]

11  num[T]    num[T] + 1

Notice that we have two "insertion" procedures here: the TABLE-INSERT procedure itself and the elementary insertion into a table in lines 6 and 10. We can analyze the running time of TABLE-INSERT in terms of the number of elementary insertions by assigning a cost of 1 to each elementary insertion. We assume that the actual running time of TABLE-INSERT is linear in the time to insert individual items, so that the overhead for allocating an initial table in line 2 is constant and the overhead for allocating and freeing storage in lines 5 and 7 is dominated by the cost of transferring items in line 6. We call the event in which the then clause in lines 5-9 is executed an expansion.

Let us analyze a sequence of n TABLE-INSERT operations on an initially empty table. What is the cost ci of the ith operation? If there is room in the current table (or if this is the first operation), then ci = 1, since we need only perform the one elementary insertion in line 10. If the current table is full, however, and an expansion occurs, then ci = i: the cost is 1 for the elementary insertion in line 10 plus i - 1 for the items that must be copied from the old table to the new table in line 6. If n operations are performed, the worst-case cost of an operation is O(n), which leads to an upper bound of O(n2) on the total running time for n operations.

This bound is not tight, because the cost of expanding the table is not borne often in the course of n TABLE-INSERT operations. Specifically, the ith operation causes an expansion only when i - 1 is an exact power of 2. The amortized cost of an operation is in fact O(1), as we can show using the aggregate method. The cost of the ith operation is

The total cost of n TABLE-INSERT operations is therefore

since there are at most n operations that cost 1 and the costs of the remaining operations form a geometric series. Since the total cost of n TABLE-INSERT operations is 3n, the amortized cost of a single operation is 3.

By using the accounting method, we can gain some feeling for why the amortized cost of a TABLE-INSERT operation should be 3. Intuitively, each item pays for 3 elementary insertions: inserting itself in the current table, moving itself when the table is expanded, and moving another item that has already been moved once when the table is expanded. For example, suppose that the size of the table is m immediately after an expansion. Then, the number of items in the table is m/2, and the table contains no credit. We charge 3 dollars for each insertion. The elementary insertion that occurs immediately costs 1 dollar. Another dollar is placed as credit on the item inserted. The third dollar is placed as credit on one of the m/2 items already in the table. Filling the table requires m/2 additional insertions, and thus, by the time the table contains m items and is full, each item has a dollar to pay for its reinsertion during the expansion.

The potential method can also be used to analyze a sequence of n TABLE- INSERT operations, and we shall use it in Section 18.4.2 to design a TABLE-DELETE operation that has O(1) amortized cost as well. We start by defining a potential function [image] that is 0 immediately after an expansion but builds to the table size by the time the table is full, so that the next expansion can be paid for by the potential. The function

  (T) = 2  num[T] - size[T]

(18.4)

is one possibility. Immediately after an expansion, we have num[T] = size[T]/2, and thus [image] (T) = 0, as desired. Immediately before an expansion, we have num[T] = size[T], and thus [image] (T) = num[T], as desired. The initial value of the potential is 0, and since the table is always at least half full, num[T] [image] size[T]/2, which implies that [image] (T) is always nonnegative. Thus, the sum of the amortized costs of n TABLE-INSERT operations is an upper bound on the sum of the actual costs.

To analyze the amortized cost of the ith TABLE-INSERT operation, we let numi denote the number of items stored in the table after the ith operation, sizei denote the total size of the table after the ith operation, and [image] i denote the potential after the ith operation. Initially, we have num0 = 0, size0 =0, and [image] 0 = 0.

If the ith TABLE-INSERT operation does not trigger an expansion, then sizei = sizei - 1 and the amortized cost of the operation is

If the ith operation does trigger an expansion, then sizei/2 = sizei-1 = numi - 1, and the amortized cost of the operation is

Figure 18.3 plots the values of numi, sizei, and [image] i against i. Notice how the potential builds to pay for the expansion of the table.

18.4.2 Table expansion and contraction

To implement a TABLE-DELETE operation, it is simple enough to remove the specified item from the table. It is often desirable, however, to contract the table when the load factor of the table becomes too small, so that the wasted space is not exorbitant. Table contraction is analogous to table expansion: when the number of items in the table drops too low, we allocate a new, smaller table and then copy the items from the old table into the new one. The storage for the old table can then be freed by returning it to the memory-management system. Ideally, we would like to preserve two properties:

the load factor of the dynamic table is bounded below by a constant, and

the amortized cost of a table operation is bounded above by a constant.

Figure 18.3 The effect of a sequence of n TABLE-INSERT operations on the number numi of items in the table, the number sizei of slots in the table, and the potential i = 2 numi - sizei, each being measured after the ith operation. The thin line shows numi, the thick line shows sizei, and the dashed line shows i. Notice that immediately before an expansion, the potential has built up to the number of items in the table, and therefore it can pay for moving all the items to the new table. Afterwards, the potential drops to 0, but it is immediately increased by 2 when the item that caused the expansion is inserted.

We assume that cost can be measured in terms of elementary insertions and deletions.

A natural strategy for expansion and contraction is to double the table size when an item is inserted into a full table and halve the size when a deletion would cause the table to become less than half full. This strategy guarantees that the load factor of the table never drops below 1/2, but unfortunately, it can cause the amortized cost of an operation to be quite large. Consider the following scenario. We perform n operations on a table T, where n is an exact power of 2. The first n/2 operations are insertions, which by our previous analysis cost a total of [image] (n). At the end of this sequence of insertions, num[T] = size[T] = n/2. For the second n/2 operations, we perform the following sequence:

I, D, D, I, I, D, D, I, I,... ,

where I stands for an insertion and D stands for a deletion. The first insertion causes an expansion of the table to size n. The two following deletions cause a contraction of the table back to size n/2. Two further insertions cause another expansion, and so forth. The cost of each expansion and contraction is [image] (n), and there are [image] (n) of them. Thus, the total cost of the n operations is [image] (n2), and the amortized cost of an operation is [image] (n).

The difficulty with this strategy is obvious: after an expansion, we do not perform enough deletions to pay for a contraction. Likewise, after a contraction, we do not perform enough insertions to pay for an expansion.

We can improve upon this strategy by allowing the load factor of the table to drop below 1/2. Specifically, we continue to double the table size when an item is inserted into a full table, but we halve the table size when a deletion causes the table to become less than 1/4 full, rather than 1/2 full as before. The load factor of the table is therefore bounded below by the constant 1/4. The idea is that after an expansion, the load factor of the table is 1/2. Thus, half the items in the table must be deleted before a contraction can occur, since contraction does not occur unless the load factor would fall below 1/4. Likewise, after a contraction, the load factor of the table is also 1/2. Thus, the number of items in the table must be doubled by insertions before an expansion can occur, since expansion occurs only when the load factor would exceed 1.

We omit the code for TABLE-DELETE, since it is analogous to TABLE-INSERT. It is convenient to assume for analysis, however, that if the number of items in the table drops to 0, the storage for the table is freed. That is, if num[T] = 0, then size[T] = 0.

We can now use the potential method to analyze the cost of a sequence of n TABLE-INSERT and TABLE-DELETE operations. We start by defining a potential function [image] that is 0 immediately after an expansion or contraction and builds as the load factor increases to 1 or decreases to 1/4. Let us denote the load factor of a nonempty table T by [image] (T) = num[T]/size[T]. Since for an empty table, num[T] = size[T] = 0 and [image] [T] = 1, we always have num[T] = [image] (T) [image] size[T], whether the table is empty or not. We shall use as our potential function

(18.5)

Observe that the potential of an empty table is 0 and that the potential is never negative. Thus, the total amortized cost of a sequence of operations with respect to [image] is an upper bound on their actual cost.

Before proceeding with a precise analysis, we pause to observe some properties of the potential function. Notice that when the load factor is 1/2, the potential is 0. When it is 1, we have size[T] = num[T], which implies [image] (T) = num[T], and thus the potential can pay for an expansion if an item is inserted. When the load factor is 1/4, we have size[T] = 4 [image] num[T] , which implies [image] (T) = num[T], and thus the potential can pay for a contraction if an item is deleted. Figure 18.4 illustrates how the potential behaves for a sequence of operations.

To analyze a sequence of n TABLE-INSERT and TABLE-DELETE operations, we let ci denote the actual cost of the ith operation, [image] denote its amortized cost with respect to [image], numi denote the number of items stored in the table after the ith operation, sizei denote the total size of the table after the ith operation, [image] i denote the load factor of the table after the ith operation, and [image] i denote the potential after the ith operation. Initially, num0 = 0, size0 = 0, [image] 0 = 1, and [image] 0 = 0.

Figure 18.4 The effect of a sequence of n TABLE-INSERT and TABLE-DELETE operations on the number numi of items in the table, the number sizei of slots in the table, and the potential

each being measured after the ith operation. The thin line shows numi, the thick line shows sizei, and the dashed line shows i. Notice that immediately before an expansion, the potential has built up to the number of items in the table, and therefore it can pay for moving all the items to the new table. Likewise, immediately before a contraction, the potential has built up to the number of items in the table.

We start with the case in which the ith operation is TABLE-INSERT. If [image] i-1 [image] 1/2, the analysis is identical to that for table expansion in Section 18.4.1. Whether the table expands or not, the amortized cost [image] of the operation is at most 3. If [image] i-1 < 1/2, the table cannot expand as a result of the operation, since expansion occurs only when [image] i-1 = 1. If [image] i < 1/2 as well, then the amortized cost of the ith operation is

If   i-1 < 1/2 but   i    1/2, then

Thus, the amortized cost of a TABLE-INSERT operation is at most 3.

We now turn to the case in which the ith operation is TABLE-DELETE. In this case, numi = numi-1 - 1. If [image] i-1 < 1/2, then we must consider whether the operation causes a contraction. If it does not, then sizei = sizei-1 and the amortized cost of the operation is

If [image] i-1 < 1/2 and the ith operation does trigger a contraction, then the actual cost of the operation is ci = numi + 1, since we delete one item and move numi items. We have sizei/2 = sizei-1/4 = numi + 1, and the amortized cost of the operation is

When the ith operation is a TABLE-DELETE and [image] i-1 [image] 1/2, the amortized cost is also bounded above by a constant. The analysis is left as Exercise 18.4-3.

In summary, since the amortized cost of each operation is bounded above by a constant, the actual time for any sequence of n operations on a dynamic table is O(n).

Exercises

18.4-1

Argue intuitively that if [image] i-1 [image] 1/2 and [image] i [image] 1/2, then the amortized cost of a TABLE-INSERT operation is 0.

18.4-2

Suppose that we wish to implement a dynamic, open-address hash table. Why might we consider the table to be full when its load factor reaches some value [image] that is strictly less than 1? Describe briefly how to make insertion into a dynamic, open-address hash table run in such a way that the expected value of the amortized cost per insertion is O(1). Why is the expected value of the actual cost per insertion not necessarily O(1) for all insertions?

18.4-3

Show that if the ith operation on a dynamic table is TABLE-DELETE and [image] i-1 [image] 1/2, then the amortized cost of the operation with respect to the potential function (18.5) is bounded above by a constant.

18.4-4

Suppose that instead of contracting a table by halving its size when its load factor drops below 1/4, we contract it by multiplying its size by 2/3 when its load factor drops below 1/3. Using the potential function

  (T) = |2    num[T] - size[T]| ,

show that the amortized cost of a TABLE-DELETE that uses this strategy is bounded above by a constant.

Problems

18-1 Bit-reversed binary counter

Chapter 32 examines an important algorithm called the Fast Fourier Transform, or FFT. The first step of the FFT algorithm performs a bit-reversal permutation on an input array A[0 . . n - 1] whose length is n = 2k for some nonnegative integer k. This permutation swaps elements whose indices have binary representations that are the reverse of each other.

We can express each index a as a k-bit sequence [image] ak-1, ak-2, . . . , a0 [image], where [image]. We define

revk(  ak-1, ak-2,..., a0  ) = (  a0, a1,..., ak-1   ;

thus,

For example, if n = 16 (or, equivalently, k = 4), then revk(3) = 12, since the 4-bit representation of 3 is 0011, which when reversed gives 1100, the 4-bit representation of 12.

a. Given a function revk that runs in [image] (k) time, write an algorithm to perform the bit-reversal permutation on an array of length n = 2k in O(nk) time.

We can use an algorithm based on an amortized analysis to improve the running time of the bit-reversal permutation. We maintain a "bit-reversed counter " and a procedure BIT-REVERSED-INCREMENT that, when given a bit-reversed-counter value a, produces revk(revk(a) + 1). If k = 4, for example, and the bit-reversed counter starts at 0, then successive calls to BIT-REVERSED-INCREMENT produce the sequence

0000, 1000, 0100, 1100, 0010, 1010,... = 0, 8, 4, 12, 2, 10,... .

b. Assume that the words in your computer store k-bit values and that in unit time, your computer can manipulate the binary values with operations such as shifting left or right by arbitrary amounts, bitwise-AND, bitwise-OR, etc. Describe an implementation of the BIT-REVERSED-INCREMENT procedure that allows the bit-reversal permutation on an n-element array to be performed in a total of O(n) time.

c. Suppose that you can shift a word left or right by only one bit in unit time. Is it still possible to implement an O(n)-time bit-reversal permutation?

18-2 Making binary search dynamic

Binary search of a sorted array takes logarithmic search time, but the time to insert a new element is linear in the size of the array. We can improve the time for insertion by keeping several sorted arrays.

Specifically, suppose that we wish to support SEARCH and INSERT on a set of n elements. Let k = [image] lg(n + 1) [image] , and let the binary representation of n be [image] nk-1, nk-2, . . . , n0 [image]. We have k sorted arrays A0, A1, . . . , Ak-1, where for i = 0, 1, . . . , k - 1, the length of array Ai is 2i. Each array is either full or empty, depending on whether ni = 1 or ni = 0, respectively. The total number of elements held in all k arrays is therefore [image]. Although each individual array is sorted, there is no particular relationship between elements in different arrays.

a. Describe how to perform the SEARCH operation for this data structure. Analyze its worst-case running time.

b. Describe how to insert a new element into this data structure. Analyze its worst-case and amortized running times.

c. Discuss how to implement DELETE.

18-3 Amortized weight-balanced trees

Consider an ordinary binary search tree augmented by adding to each node x the field size[x] giving the number of keys stored in the subtree rooted at x. Let [image] be a constant in the range 1/2 [image] [image] < 1. We say that a given node x is [image] -balanced if

size[left[x]]       . size[x]

and

size[right[x]]       . size[x].

The tree as a whole is [image] -balanced if every node in the tree is [image] -balanced. The following amortized approach to maintaining weight-balanced trees was suggested by G. Varghese.

a. A 1/2-balanced tree is, in a sense, as balanced as it can be. Given a node x in an arbitrary binary search tree, show how to rebuild the subtree rooted at x so that it becomes 1/2-balanced. Your algorithm should run in time [image] (size[x], and it can use O(size[x]) auxiliary storage.

b. Show that performing a search in an n-node [image] -balanced binary search tree takes O(lg n) worst-case time.

For the remainder of this problem, assume that the constant [image] is strictly greater than 1/2. Suppose that INSERT and DELETE are implemented as usual for an n-node binary search tree, except that after every such operation, if any node in the tree is no longer [image] -balanced, then the subtree rooted at the highest such node in the tree is "rebuilt" so that it becomes 1/2-balanced.

We shall analyze this rebuilding scheme using the potential method. For a node x in a binary search tree T, we define

  (x) = |size[left[x]] - size[right[x]]| ,

and we define the potential of T as

where c is a sufficiently large constant that depends on [image].

c. Argue that any binary search tree has nonnegative potential and that a 1/2-balanced tree has potential 0.

d. Suppose that m units of potential can pay for rebuilding an m-node subtree. How large must c be in terms of [image] in order for it to take O(1) amortized time to rebuild a subtree that is not [image] -balanced?

e. Show that inserting a node into or deleting a node from an n-node [image] -balanced tree costs O(lg n) amortized time.

Chapter notes

The aggregate method of amortized' analysis was used by Aho, Hopcroft, and Ullman [4]. Tarjan [189] surveys the accounting and potential methods of amortized analysis and presents several applications. He attributes the accounting method to several authors, including M. R. Brown, R. E. Tarjan, S. Huddleston, and K. Mehlhorn. He attributes the potential method to D. D. Sleator. The term "amortized" is due to D. D. Sleator and R. E. Tarjan.

Go to Part V Back to Table of Contents


Intro to Algorithms: CHAPTER 19: B-TREES

 Return to Table of Contents

CHAPTER 19: B-TREES

B-trees are balanced search trees designed to work well on magnetic disks or other direct-access secondary storage devices. B-trees are similar to red-black trees (Chapter 14), but they are better at minimizing disk I/O operations.

B-trees differ significantly from red-black trees in that B-tree nodes may have many children, from a handful to thousands. That is, the "branching factor" of a B-tree can be quite large, although it is usually determined by characteristics of the disk unit used. B-trees are similar to red-black trees in that every n-node B-tree has height O(lg n), although the height of a B-tree can be considerably less than that of a red-black tree because its branching factor can be much larger. Therefore, B-trees can also be used to implement many dynamic-set operations in time O(lg n).

B-trees generalize binary search trees in a natural manner. Figure 19.1 shows a simple B-tree. If a B-tree node x contains n[x] keys, then x has n[x] + 1 children. The keys in node x are used as dividing points separating the range of keys handled by x into n[x] + 1 subranges, each handled by one child of x. When searching for a key in a B-tree, we make an (n[x] + 1)-way decision based on comparisons with the n[x] keys stored at node x.

Figure 19.1 A B-tree whose keys are the consonants of English. An internal node x containing n[x] keys has n[x] + 1 children. All leaves are at the same depth in the tree. The lightly shaded nodes are examined in a search for the letter R.

Figure 19.2 A typical disk drive.

Section 19.1 gives a precise definition of B-trees and proves that the height of a B-tree grows only logarithmically with the number of nodes it contains. Section 19.2 describes how to search for a key and insert a key into a B-tree, and Section 19.3 discusses deletion. Before proceeding, however, we need to ask why data structures designed to work on a magnetic disk are evaluated differently than data structures designed to work in main random-access memory.

Data structures on secondary storage

There are many different technologies available for providing memory capacity in a computer system. The primary memory (or main memory) of a computer system typically consists of silicon memory chips, each of which can hold 1 million bits of data. This technology is more expensive per bit stored than magnetic storage technology, such as tapes or disks. A typical computer system has secondary storage based on magnetic disks; the amount of such secondary storage often exceeds the amount of primary memory by several orders of magnitude.

Figure 19.2 shows a typical disk drive. The disk surface is covered with a magnetizable material. The read/write head can read or write data magnetically on the rotating disk surface. The read/write arm can position the head at different distances from the center of the disk. When the head is stationary, the surface that passes underneath it is called a track. Theinformation stored on each track is often divided into a fixed number of equal-sized pages; for a typical disk, a page might be 2048 bytes in length.The basic unit of information storage and retrieval is usually a page of information--that is, disk reads and writes are typically of entire pages.The access time--the time required to position the read/write head and to wait for a given page of information to pass underneath the head--may be large (e.g., 20 milliseconds), while the time to read or write a page, once accessed, is small. The price paid for the low cost of magnetic storage techniques is thus the relatively long time it takes to access the data. Since moving electrons is much easier than moving large (or even small) objects, storage devices that are entirely electronic, such as silicon memory chips, have a much smaller access time than storage devices that have moving parts, such as magnetic disk drives. However, once everything is positioned correctly, reading or writing a magnetic disk is entirely electronic (aside from the rotation of the disk), and large amounts of data can be read or written quickly.

Often, it takes more time to access a page of information and read it from a disk than it takes for the computer to examine all the information read. For this reason, in this chapter we shall look separately at the two principal components of the running time:

the number of disk accesses, and

the CPU (computing) time.

The number of disk accesses is measured in terms of the number of pages of information that need to be read from or written to the disk. We note that disk access time is not constant--it depends on the distance between the current track and the desired track and also on the initial rotational state of the disk. We shall nonetheless use the number of pages read or written as a crude first-order approximation of the total time spent accessing the disk.

In a typical B-tree application, the amount of data handled is so large that all the data do not fit into main memory at once. The B-tree algorithms copy selected pages from disk into main memory as needed and write back onto disk pages that have changed. Since the B-tree algorithms only need a constant number of pages in main memory at any time, the size of main memory does not limit the size of B-trees that can be handled.

We model disk operations in our pseudocode as follows. Let x be a pointer to an object. If the object is currently in the computer's main memory, then we can refer to the fields of the object as usual: key[x], for example. If the object referred to by x resides on disk, however, then we must perform the operation DISK-READ(x) to read object x into main memory before its fields can be referred to. (We assume that if x is already in main memory, then DISK-READ(x) requires no disk accesses; it is a "noop.") Similarly, the operation DISK-WRITE(x) is used to save any changes that have been made to the fields of object x. That is, the typical pattern for working with an object is as follows.

1  . . .

2  x    a pointer to some object

3  DISK-READ(x)

4  operations that access and/or modify the fields of x

5  DISK-WRITE(x)       Omitted if no fields of x were changed.

6  other operations that access but do not modify fields of x

7  ...

Figure 19.3 A B-tree of height 2 containing over one billion keys. Each internal node and leaf contains 1000 keys. There are 1001 nodes at depth 1 and over one million leaves at depth 2. Shown inside each node x is n[x], the number of keys in x.

The system can only keep a limited number of pages in main memory at any one time. We shall assume that pages no longer in use are flushed from main memory by the system; our B-tree algorithms will ignore this issue.

Since in most systems the running time of a B-tree algorithm is determined mainly by the number of DISK-READ and DISK-WRITE operations it performs, it is sensible to use these operations intensively by having them read or write as much information as possible. Thus, a B-tree node is usually as large as a whole disk page. The number of children a B-tree node can have is therefore limited by the size of a disk page.

For a large B-tree stored on a disk, branching factors between 50 and 2000 are often used, depending on the size of a key relative to the size of a page. A large branching factor dramatically reduces both the height of the tree and the number of disk accesses required to find any key. Figure 19.3 shows a B-tree with a branching factor of 1001 and height 2 that can store over one billion keys; nevertheless, since the root node can be kept permanently in main memory, only two disk accesses at most are required to find any key in this tree!

19.1 Definition of B-trees

To keep things simple, we assume, as we have for binary search trees and red-black trees, that any "satellite information" associated with a key is stored in the same node as the key. In practice, one might actually store with each key just a pointer to another disk page containing the satellite information for that key. The pseudocode in this chapter implicitly assumes that the satellite information associated with a key, or the pointer to such satellite information, travels with the key whenever the key is moved from node to node. Another commonly used B-tree organization stores all the satellite information in the leaves and only stores keys and child pointers in the internal nodes, thus maximizing the branching factor of the internal nodes.

A B-tree T is a rooted tree (with root root[T]) having the following properties.

1. Every node x has the following fields:

a. n[x], the number of keys currently stored in node x,

b. the n[x] keys themselves, stored in nondecreasing order: key1[x] [image] key2[x] [image] [image] [image] [image] [image] keyn[x][x], and

c. leaf [x], a boolean value that is TRUE if x is a leaf and FALSE if x is an internal node.

2. If x is an internal node, it also contains n[x] + 1 pointers c1 [x], c2[x], . . . , cn[x]+1[x] to its children. Leaf nodes have no children, so their ci fields are undefined.

3. The keys keyi[x] separate the ranges of keys stored in each subtree: if ki is any key stored in the subtree with root ci[x], then

k1    key1[x]    k2    key2[x]                keyn[x][x]    kn[x]+1 .

4. Every leaf has the same depth, which is the tree's height h.

5. There are lower and upper bounds on the number of keys a node can contain. These bounds can be expressed in terms of a fixed integer t [image] 2 called the minimum degree of the B-tree:

a. Every node other than the root must have at least t - 1 keys. Every internal node other than the root thus has at least t children. If the tree is nonempty, the root must have at least one key.

b. Every node can contain at most 2t - 1 keys. Therefore, an internal node can have at most 2t children. We say that a node is full if it contains exactly 2t - 1 keys.

The simplest B-tree occurs when t = 2. Every internal node then has either 2, 3, or 4 children, and we have a 2-3-4 tree. In practice, however, much larger values of t are typically used.

The height of a B-tree

The number of disk accesses required for most operations on a B-tree is proportional to the height of the B-tree. We now analyze the worst-case height of a B-tree.

Theorem 19.1

If n [image] 1, then for any n-key B-tree T of height h and minimum degree t [image] 2,

Figure 19.4 A B-tree of height 3 containing a minimum possible number of keys. Shown inside each node x is n[x].

Proof If a B-tree has height h, the number of its nodes is minimized when the root contains one key and all other nodes contain t - 1 keys. In this case, there are 2 nodes at depth 1, 2t nodes at depth 2, 2t2 nodes at depth 3, and so on, until at depth h there are 2th-1 nodes. Figure 19.4 illustrates such a tree for h = 3. Thus, the number n of keys satisfies the inequality

which implies the theorem.

Here we see the power of B-trees, as compared to red-black trees. Although the height of the tree grows as O(lg n) in both cases (recall that t is a constant), for B-trees the base of the logarithm can be many times larger. Thus, B-trees save a factor of about lg t over red-black trees in the number of nodes examined for most tree operations. Since examining an arbitrary node in a tree usually requires a disk access, the number of disk accesses is substantially reduced.

Exercises

19.1-1

Why don't we allow a minimum degree of t = 1?

19.1-2

For what values of t is the tree of Figure 19.1 a legal B-tree?

19.1-3

Show all legal B-trees of minimum degree 2 that represent {1, 2, 3, 4, 5}.

19.1-4

Derive a tight upper bound on the number of keys that can be stored in a B-tree of height h as a function of the minimum degree t.

19.1-5

Describe the data structure that would result if each black node in a red-black tree were to absorb its red children, incorporating their children with its own.

19.2 Basic operations on B-trees

In this section, we present the details of the operations B-TREE-SEARCH, B-TREE-CREATE, and B-TREE-INSERT. In these procedures, we adopt two conventions:

The root of the B-tree is always in main memory, so that a DISK-READ on the root is never required; a DISK-WRITE of the root is required, however, whenever the root node is changed.

Any nodes that are passed as parameters must already have had a DISK-READ operation performed on them.

The procedures we present are all "one-pass" algorithms that proceed downward from the root of the tree, without having to back up.

Searching a B-tree

Searching a B-tree is much like searching a binary search tree, except that instead of making a binary, or "two-way," branching decision at each node, we make a multiway branching decision according to the number of the node's children. More precisely, at each internal node x, we make an (n[x] + 1)-way branching decision.

B-TREE-SEARCH is a straightforward generalization of the TREE-SEARCH procedure defined for binary search trees. B-TREE-SEARCH takes as input a pointer to the root node x of a subtree and a key k to be searched for in that subtree. The top-level call is thus of the form B-TREE-SEARCH(root[T], k). If k is in the B-tree, B-TREE-SEARCH returns the ordered pair (y,i) consisting of a node y and an index i such that keyi[y] = k. Otherwise, the value NIL is returned.

B-TREE-SEARCH(x, k)

1  i    1

2  while i    n[x] and k    keyi[x]

3       do i    i + 1

4  if i    n[x] and k = keyi[x]

5      then return (x, i)

6  if leaf [x]

7      then return NIL

8      else DISK-READ(ci[x])

9           return B-TREE-SEARCH(ci[x], k)

Using a linear-search procedure, lines 1-3 find the smallest i such that k [image] keyi[x], or else they set i to n[x] + 1. Lines 4-5 check to see if we have now discovered the key, returning if we have. Lines 6-9 either terminate the search unsuccessfully (if x is a leaf) or recurse to search the appropriate subtree of x, after performing the necessary DISK-READ on that child.

Figure 19.1 illustrates the operation of B-TREE-SEARCH; the lightly shaded nodes are examined during a search for the key R.

As in the TREE-SEARCH procedure for binary search trees, the nodes encountered during the recursion form a path downward from the root of the tree. The number of disk pages accessed by B-TREE-SEARCH is therefore [image] (h) = [image] (logt n), where h is the height of the B-tree and n is the number of keys in the B-tree. Since n[x] < 2t, the time taken by the while loop of lines 2-3 within each node is O(t), and the total CPU time is O(th) = O(t logt n).

Creating an empty B-tree

To build a B-tree T, we first use B-TREE-CREATE to create an empty root node and then call B-TREE-INSERT to add new keys. Both of these procedures use an auxiliary procedure ALLOCATE-NODE, which allocates one disk page to be used as a new node in O(1) time. We can assume that a node created by ALLOCATE-NODE requires no DISK-READ, since there is as yet no useful information stored on the disk for that node.

B-TREE-CREATE(T)

1  x    ALLOCATE-NODE()

2  leaf[x]    TRUE

3  n[x]    0

4  DISK-WRITE(x)

5  root[T]    x

B-TREE-CREATE requires O(1) disk operations and O(1) CPU time.

Figure 19.5 Splitting a node with t = 4. Node y is split into two nodes, y and z, and the median key S of y is moved up into y's parent.

Splitting a node in a B-tree

Inserting a key into a B-tree is significantly more complicated than inserting a key into a binary search tree. A fundamental operation used during insertion is the splitting of a full node y (having 2t - 1 keys) around its median key keyt[y] into two nodes having t - 1 keys each. The median key moves up into y's parent--which must be nonfull prior to the splitting of y--to identify the dividing point between the two new trees; if y has no parent, then the tree grows in height by one. Splitting, then, is the means by which the tree grows.

The procedure B-TREE-SPLIT-CHILD takes as input a nonfull internal node x (assumed to be in main memory), an index i, and a node y such that y = ci[x] is a full child of x. The procedure then splits this child in two and adjusts x so that it now has an additional child.

Figure 19.5 illustrates this process. The full node y is split about its median key S, which is moved up into y's parent node x. Those keys in y that are greater than the median key are placed in a new node z, which is made a new child of x.

B-TREE-SPLIT-CHILD(x,i,y)

1  z    ALLOCATE-NODE()

2  leaf[z]    leaf[y]

3  n[z]    t - 1

4  for j    1 to t - 1

5        do keyj[z]    keyj+t[y]

6  if not leaf [y]

7      then for j    1 to t

8                do cj[z]    cj+t[y]

9  n[y]    t - 1

10  for j    n[x] + 1 downto i + 1

11       do cj+1[x]    cj[x]

12  ci+1[x]    z

13  for j    n[x] downto i

14        do keyj+1[x]    keyj[x]

15  keyi[x]    keyt[y]

16  n[x]    n[x] + 1

17  DISK-WRITE(y)

18  DISK-WRITE(z)

19  DISK-WRITE(x)

B-TREE-SPLIT-CHILD works by straightforward "cutting and pasting. " Here, y is the ith child of x and is the node being split. Node y originally has 2t - 1 children but is reduced to t - 1 children by this operation. Node z "adopts" the t - 1 largest children of y, and z becomes a new child of x, positioned just after y in x's table of children. The median key of y moves up to become the key in x that separates y and z.

Lines 1-8 create node z and give it the larger t - 1 keys and corresponding t children of y. Line 9 adjusts the key count for y. Finally, lines 10-16 insert z as a child of x, move the median key from y up to x in order to separate y from z, and adjust x's key count. Lines 17-19 write out all modified disk pages. The CPU time used by B-TREE-SPLIT-CHILD is [image] (t), due to the loops on lines 4-5 and 7-8. (The other loops run for at most t iterations.)

Inserting a key into a B-tree

Inserting a key k into a B-tree T of height h is done in a single pass down the tree, requiring O(h) disk accesses. The CPU time required is O(th) = O(t logt n). The B-TREE-INSERT procedure uses B-TREE-SPLIT-CHILD to guarantee that the recursion never descends to a full node.

Figure 19.6 Splitting the root with t = 4. Root node r is split in two, and a new root node s is created. The new root contains the median key of r and has the two halves of r as children. The B-tree grows in height by one when the root is split.

B-TREE-INSERT(T,k)

1  r    root[T]

2  if n[r] = 2t - 1

3      then s    ALLOCATE-NODE()

4           root[T]    s

5           leaf[s]    FALSE

6           n[s]    0

7           c1[s]    r

8           B-TREE-SPLIT-CHILD(s,1,r)

9           B-TREE-INSERT-NONFULL(s,k)

10  else B-TREE-INSERT-NONFULL(r,k)

Lines 3-9 handle the case in which the root node r is full: the root is split and a new node s (having two children) becomes the root. Splitting the root is the only way to increase the height of a B-tree. Figure 19.6 illustrates this case. Unlike a binary search tree, a B-tree increases in height at the top instead of at the bottom. The procedure finishes by calling B-TREE-INSERT-NONFULL to perform the insertion of key k in the tree rooted at the nonfull root node. B-TREE-INSERT-NONFULL recurses as necessary down the tree, at all times guaranteeing that the node to which it recurses is not full by calling B-TREE-SPLIT-CHILD as necessary.

The auxiliary recursive procedure B-TREE-INSERT-NONFULL inserts key k into node x, which is assumed to be nonfull when the procedure is called. The operation of B-TREE-INSERT and the recursive operation of B-TREE-INSERT-NONFULL guarantee that this assumption is true.

B-TREE-INSERT-NONFULL(x,k)

1  i    n[x]

2  if leaf[x]

3      then while i    1 and k < keyi[x]

4               do keyi+1 [x]    keyi[x]

5                  i    i - 1

6           keyi+1[x]    k

7           n[x]    n[x] + 1

8           DISK-WRITE(x)

9  else while i    1 and k < keyi[x]

10               do i    i - 1

11       i    i + 1

12       DISK-READ(ci[x])

13       if n[ci[x]] = 2t - 1

14          then B-TREE-SPLIT-CHILD(x,i,ci[x])

15               if k > keyi[x]

16                  then i    i + 1

17       B-TREE-INSERT-NONFULL(ci[x],k)

The B-TREE-INSERT-NONFULL procedure works as follows. Lines 3-8 handle the case in which x is a leaf node by inserting key k into x. If x is not a leaf node, then we must insert k into the appropriate leaf node in the subtree rooted at internal node x. In this case, lines 9-11 determine the child of x to which the recursion descends. Line 13 detects whether the recursion would descend to a full child, in which case line 14 uses B-TREE-SPLIT-CHILD to split that child into two nonfull children, and lines 15-16 determine which of the two children is now the correct one to descend to. (Note that there is no need for a DISK-READ(ci[x]) after line 16 increments i, since the recursion will descend in this case to a child that was just created by B-TREE-SPLIT-CHILD.) The net effect of lines 13-16 is thus to guarantee that the procedure never recurses to a full node. Line 17 then recurses to insert k into the appropriate subtree. Figure 19.7 illustrates the various cases of inserting into a B-tree.

The number of disk accesses performed by B-TREE-INSERT is O(h) for a B-tree of height h, since only O(1) DISK-READ and DISK-WRITE operations are performed between calls to B-TREE-INSERT-NONFULL. The total CPU time used is O(th) = O(t1ogt n). Since B-TREE-INSERT-NONFULL is tail-recursive, it can be alternatively implemented as a while loop, demonstrating that the number of pages that need to be in main memory at any time is O(1).

Figure 19.7 Inserting keys into a B-tree. The minimum degree t for this B-tree is 3, so a node can hold at most 5 keys. Nodes that are modified by the insertion process are lightly shaded. (a) The initial tree for this example. (b) The result of inserting B into the initial tree; this is a simple insertion into a leaf node. (c) The result of inserting Q into the previous tree. The node RSTUV is split into two nodes containing RS and UV, the key T is moved up to the root, and Q is inserted in the leftmost of the two halves (the RS node). (d) The result of inserting L into the previous tree. The root is split right away, since it is full, and the B-tree grows in height by one. Then L is inserted into the leaf containing JK. (e) The result of inserting F into the previous tree. The node ABCDE is split before F is inserted into the rightmost of the two halves (the DE node).

Exercises

19.2-1

Show the results of inserting the keys

F, S, Q, K, C, L, H, T, V, W, M, R, N, P, A, B, X, Y, D, Z, E

in order into an empty B-tree. Only draw the configurations of the tree just before some node must split, and also draw the final configuration.

19.2-2

Explain under what circumstances, if any, redundant DISK-READ or DISK-WRITE operations are performed during the course of executing a call to B-TREE-INSERT. (A redundant DISK-READ is a DISK-READ for a page that is already in memory. A redundant DISK-WRITE writes to disk a page of information that is identical to what is already stored there.)

19.2-3

Explain how to find the minimum key stored in a B-tree and how to find the predecessor of a given key stored in a B-tree.

19.2-4

Suppose that the keys {1, 2, . . . , n} are inserted into an empty B-tree with minimum degree 2. How many nodes does the final B-tree have?

19.2-5

Since leaf nodes require no pointers to children, they could conceivably use a different (larger) t value than internal nodes for the same disk page size. Show how to modify the procedures for creating and inserting into a B-tree to handle this variation.

19.2-6

Suppose that B-TREE-SEARCH is implemented to use binary search rather than linear search within each node. Show that this makes the CPU time required O(lg n), independently of how t might be chosen as a function of n.

19.2-7

Suppose that disk hardware allows us to choose the size of a disk page arbitrarily, but that the time it takes to read the disk page is a + bt, where a and b are specified constants and t is the minimum degree for a B-tree using pages of the selected size. Describe how to choose t so as to minimize (approximately) the B-tree search time. Suggest an optimal value of t for the case in which a = 30 milliseconds and b = 40 microseconds.

19.3 Deleting a key from a B-tree

Deletion from a B-tree is analogous to insertion but a little more complicated. We sketch how it works instead of presenting the complete pseudocode.

Assume that procedure B-TREE-DELETE is asked to delete the key k from the subtree rooted at x. This procedure is structured to guarantee that whenever B-TREE-DELETE is called recursively on a node x, the number of keys in x is at least the minimum degree t. Note that this condition requires one more key than the minimum required by the usual B-tree conditions, so that sometimes a key may have to be moved into a child node before recursion descends to that child. This strengthened condition allows us to delete a key from the tree in one downward pass without having to "back up" (with one exception, which we'll explain). The following specification for deletion from a B-tree should be interpreted with the understanding that if it ever happens that the root node x becomes an internal node having no keys, then x is deleted and x's only child c1[x] becomes the new root of the tree, decreasing the height of the tree by one and preserving the property that the root of the tree contains at least one key (unless the tree is empty).

Figure 19.8 illustrates the various cases of deleting keys from a B-tree.

1. If the key k is in node x and x is a leaf, delete the key k from x.

2. If the key k is in node x and x is an internal node, do the following.

a. If the child y that precedes k in node x has at least t keys, then find the predecessor k' of k in the subtree rooted at y. Recursively delete k', and replace k by k' in x. (Finding k' and deleting it can be performed in a single downward pass.)

b. Symmetrically, if the child z that follows k in node x has at least t keys, then find the successor k' of k in the subtree rooted at z. Recursively delete k', and replace k by k' in x. (Finding k' and deleting it can be performed in a single downward pass.)

c. Otherwise, if both y and z have only t- 1 keys, merge k and all of z into y, so that x loses both k and the pointer to z, and y now contains 2t - 1 keys. Then, free z and recursively delete k from y.

3. If the key k is not present in internal node x, determine the root ci[x] of the appropriate subtree that must contain k, if k is in the tree at all. If ci[x] has only t - 1 keys, execute step 3a or 3b as necessary to guarantee that we descend to a node containing at least t keys. Then, finish by recursing on the appropriate child of x.

a. If ci[x] has only t - 1 keys but has a sibling with t keys, give ci[x] an extra key by moving a key from x down into ci[x], moving a key from ci[x]'s immediate left or right sibling up into x, and moving the appropriate child from the sibling into ci[x].

Figure 19.8 Deleting keys from a B-tree. The minimum degree for this B-tree is t = 3, so a node (other than the root) cannot have less than 2 keys. Nodes that are modified are lightly shaded. (a) The B-tree of Figure 19.7(e). (b) Deletion of F. This is case 1: simple deletion from a leaf. (c) Deletion of M. This is case 2a: the predecessor L of M is moved up to take M's position. (d) Deletion of G. This is case 2c: G is pushed down to make node DEGJK, and then G is deleted from this leaf (case 1). (e) Deletion of D. This is case 3b: the recursion can't descend to node CL because it has only 2 keys, so P is pushed down and merged with CL and TX to form CLPTX; then, D is deleted from a leaf (case 1). (e') After (d), the root is deleted and the tree shrinks in height by one. (f) Deletion of B. This is case 3a: C is moved to fill B's position and E is moved to fill C's position.

b. If ci[x] and all of ci[x]'s siblings have t - 1 keys, merge ci with one sibling, which involves moving a key from x down into the new merged node to become the median key for that node.

Since most of the keys in a B-tree are in the leaves, we may expect that in practice, deletion operations are most often used to delete keys from leaves. The B-TREE-DELETE procedure then acts in one downward pass through the tree, without having to back up. When deleting a key in an internal node, however, the procedure makes a downward pass through the tree but may have to return to the node from which the key was deleted to replace the key with its predecessor or successor (cases 2a and 2b).

Although this procedure seems complicated, it involves only O(h) disk operations for a B-tree of height h, since only O(1) calls to DISK-READ and DISK-WRITE are made between recursive invocations of the procedure. The CPU time required is O(th) = O(t logt n).

Exercises

19.3-1

Show the results of deleting C, P, and V, in order, from the tree of Figure 19.8(f).

19.3-2

Write pseudocode for B-TREE-DELETE.

Problems

19-1 Stacks on secondary storage

Consider implementing a stack in a computer that has a relatively small amount of fast primary memory and a relatively large amount of slower disk storage. The operations PUSH and POP are supported on single-word values. The stack we wish to support can grow to be much larger than can fit in memory, and thus most of it must be stored on disk.

A simple, but inefficient, stack implementation keeps the entire stack on disk. We maintain in memory a stack pointer, which is the disk address of the top element on the stack. If the pointer has value p, the top element is the (p mod m)th word on page [image] p/m [image] of the disk, where m is the number of words per page.

To implement the PUSH operation, we increment the stack pointer, read the appropriate page into memory from disk, copy the element to be pushed to the appropriate word on the page, and write the page back to disk. A POP operation is similar. We decrement the stack pointer, read in the appropriate page from disk, and return the top of the stack. We need not write back the page, since it was not modified.

Because disk operations are relatively expensive, we use the total number of disk accesses as a figure of merit for any implementation. We also count CPU time, but we charge [image] (m) for any disk access to a page of m words.

a. Asymptotically, what is the worst-case number of disk accesses for n stack operations using this simple implementation? What is the CPU time for n stack operations? (Express your answer in terms of m and n for this and subsequent parts.)

Now, consider a stack implementation in which we keep one page of the stack in memory. (We also maintain a small amount of memory to keep track of which page is currently in memory.) We can perform a stack operation only if the relevant disk page resides in memory. If necessary, the page currently in memory can be written to the disk and the new page read in from the disk to memory. If the relevant disk page is already in memory, then no disk accesses are required.

b. What is the worst-case number of disk accesses required for n PUSH operations? What is the CPU time?

c. What is the worst-case number of disk accesses required for n stack operations? What is the CPU time?

Suppose that we now implement the stack by keeping two pages in memory (in addition to a small number of words for bookkeeping).

d. Describe how to manage the stack pages so that the amortized number of disk accesses for any stack operation is O(1/m) and the amortized CPU time for any stack operation is O(1).

19-2 Joining and splitting 2-3-4 trees

The join operation takes two dynamic sets S' and S\" and an element x such that for any x' [image] S' and x\" [image] S\", we have key[x'] < key[x] < key[x\"]. It returns a set S = S' [image] {x} [image] S\". The split operation is like an "inverse" join: given a dynamic set S and an element x [image] S, it creates a set S' consisting of all elements in S - {x} whose keys are less than key[x] and a set S\" consisting of all elements in S - {x} whose keys are greater than key[x]. In this problem, we investigate how to implement these operations on 2-3-4 trees. We assume for convenience that elements consist only of keys and that all key values are distinct.

a. Show how to maintain, for every node x of a 2-3-4 tree, the height of the subtree rooted at x as a field height[x]. Make sure that your implementation does not affect the asymptotic running times of searching, insertion, and deletion.

b. Show how to implement the join operation. Given two 2-3-4 trees T' and T\" and a key k, the join should run in O(|h'- h\"|) time, where h' and h\" are the heights of T' and T\", respectively.

c. Consider the path p from the root of a 2-3-4 tree T to a given key k, the set S' of keys in T that are less than k, and the set S\" of keys in T that are greater than k. Show that p breaks S' into a set of trees { T'0,T'1, . . . ,T'm } and a set of keys {k'1,k'2, . . . , k'm}, where, for i = 1, 2, . . . ,m, we have y < k'1 < z for any keys y [image] T'i-1 and z [image] T'i. What is the relationship between the heights of T'i-1 and T'i? Describe how p breaks S\" into sets of trees and keys.

d. Show how to implement the split operation on T. Use the join operation to assemble the keys in S' into a single 2-3-4 tree T' and the keys in S\" into a single 2-3-4 tree T\". The running time of the split operation should be O(lg n), where n is the number of keys in T. (Hint: The costs for joining should telescope.)

Chapter notes

Knuth [123], Aho, Hopcroft, and Ullman [4], and Sedgewick [175] give further discussions of balanced-tree schemes and B-trees. Comer [48] provides a comprehensive survey of B-trees. Guibas and Sedgewick [93] discuss the relationships among various kinds of balanced-tree schemes, including red-black trees and 2-3-4 trees.

In 1970, J. E. Hopcroft invented 2-3 trees, a precursor to B-trees and 2-3-4 trees, in which every internal node has either two or three children. B-trees were introduced by Bayer and McCreight in 1972 [18]; they did not explain their choice of name.

Go to Chapter 20 Back to Table of Contents


Intro to Algorithms: CHAPTER 20: BINOMIAL HEAPS

 Return to Table of Contents

CHAPTER 20: BINOMIAL HEAPS

This chapter and Chapter 21 present data structures known as mergeable heaps, which support the following five operations.

MAKE-HEAP() creates and returns a new heap containing no elements.

INSERT(H, x) inserts node x, whose key field has already been filled in, into heap H.

MINIMUM(H) returns a pointer to the node in heap H whose key is minimum.

EXTRACT-MIN(H) deletes the node from heap H whose key is minimum, returning a pointer to the node.

UNION(Hl, H2) creates and returns a new heap that contains all the nodes of heaps H1 and H2. Heaps H1 and H2 are "destroyed" by this operation.

In addition, the data structures in these chapters also support the following two operations.

DECREASE-KEY(H, x, k) assigns to node x within heap H the new key value k, which is assumed to be no greater than its current key value.

DELETE(H, x) deletes node x from heap H.

As the table in Figure 20.1 shows, if we don't need the UNION operation, ordinary binary heaps, as used in heapsort (Chapter 7), work well. Operations other than UNION run in worst-case time O(lg n) (or better) on a binary heap. If the UNION operation must be supported, however, binary heaps perform poorly. By concatenating the two arrays that hold the binary heaps to be merged and then running HEAPIFY, the UNION operation takes [image] (n) time in the worst case.

In this chapter, we shall examine "binomial heaps," whose worst-case time bounds are also shown in Figure 20.1. In particular, the UNION operation takes only O(lg n) time to merge two binomial heaps with a total of n elements.

In Chapter 21, we shall explore Fibonacci heaps, which have even better time bounds for some operations. Note, however, that the running times for Fibonacci heaps in Figure 20.1 are amortized time bounds, not worst-case per operation time bounds.

             Binary heap    Binomial heap   Fibonacci heap

Procedure    (worst-case)    (worst-case)    (amortized)

--------------------------------------------------------------

MAKE-HEAP           (1)              (1)              (1)

INSERT             (lg n)        O(lg n)             (1)

MINIMUM            (1)          O(lg n)             (l)

EXTRACT-MIN        (lg n)          (1g n)         O(lg n)

UNION               (n)          O(lg n)             (1)

DECREASE-KEY       (lg n)          (lg n)             (1)

DELETE              (1g n)          (lg n)         O(lg n)

Figure 20.1 Running times for operations on three implementations of mergeable heaps. The number of items in the heap(s) at the time of an operation is denoted by n.

This chapter ignores issues of allocating nodes prior to insertion and freeing nodes following deletion. We assume that the code that calls the heap procedures handles these details.

Binary heaps, binomial heaps, and Fibonacci heaps are all inefficient in their support of the operation SEARCH; it can take a while to find a node with a given key. For this reason, operations such as DECREASE-KEY and DELETE that refer to a given node require a pointer to that node as part of their input. This requirement poses no problem in many applications.

Section 20.1 defines binomial heaps after first defining their constituent binomial trees. It also introduces a particular representation of binomial heaps. Section 20.2 shows how we can implement operations on binomial heaps in the time bounds given in Figure 20.1.

20.1 Binomial trees and binomial heaps

A binomial heap is a collection of binomial trees, so this section starts by defining binomial trees and proving some key properties. We then define binomial heaps and show how they can be represented.

20.1.1 Binomial trees

The binomial tree Bk is an ordered tree (see Section 5.5.2) defined recursively. As shown in Figure 20.2(a), the binomial tree B0 consists of a single node. The binomial tree Bk consists of two binomial trees Bk-1 that are linked together: the root of one is the leftmost child of the root of the other. Figure 20.2(b) shows the binomial trees B0 through B4.

Some properties of binomial trees are given by the following lemma.

Figure 20.2 (a) The recursive definition of the binomial tree Bk. Triangles represent rooted subtrees. (b) The binomial trees Bo through B4. Node depths in B4 are shown. (c) Another way of looking at the binomial tree Bk.

Lemma 20.1

For the binomial tree Bk,

1. there are 2k nodes,

2. the height of the tree is k,

3. there are exactly [image] nodes at depth i for i = 0, 1, . . . , k, and

4. the root has degree k, which is greater than that of any other node; moreover if the children of the root are numbered from left to right by k - 1, k - 2, . . . , 0, child i is the root of a subtree Bi.

Proof The proof is by induction on k. For each property, the basis is the binomial tree B0. Verifying that each property holds for B0 is trivial.

For the inductive step, we assume that the lemma holds for Bk-1.

1. Binomial tree Bk consists of two copies of Bk-1, so Bk has 2k-1 + 2k-1 = 2k nodes.

2. Because of the way in which the two copies of Bk-1 are linked to form Bk, the maximum depth of a node in Bk is one greater than the maximum depth in Bk-1 . By the inductive hypothesis, this maximum depth is (k-1) + 1 = k.

3. Let D(k, i) be the number of nodes at depth i of binomial tree Bk. Since Bk is composed of two copies of Bk-1 linked together, a node at depth i in Bk-1 appears in Bk once at depth i and once at depth i + 1. In other words, the number of nodes at depth i in Bk is the number of nodes at depth i in Bk-1 plus the number of nodes at depth i - 1 in Bk-1 . Thus,

The second equality follows from the inductive hypothesis, and the third equality follows from Exercise 6.1-7.

4. The only node with greater degree in Bk than in Bk-1, is the root, which has one more child than in Bk-1. Since the root of Bk-1 has degree k-1, the root of Bk has degree k. Now by the inductive hypothesis, and as Figure 20.2(c) shows, from left to right, the children of the root of Bk-1 are roots of Bk-2, Bk-3, . . ., B0. When Bk-1 is linked to Bk-1, therefore, the children of the resulting root are roots of Bk-1, Bk-2, . . . , B0

Corollary 20.2

The maximum degree of any node in an n-node binomial tree is lg n.

Proof Immediate from properties l and 4 of Lemma 20.1.

The term "binomial tree" comes from property 3 of Lemma 20.1, since the terms [image] are the binomial coefficients. Exercise 20.1-3 gives further justification for the term.

20.1.2 Binomial heaps

A binomial heap H is a set of binomial trees that satisfies the following binomial-heap properties.

1. Each binomial tree in H is heap-ordered: the key of a node is greater than or equal to the key of its parent.

2. There is at most one binomial tree in H whose root has a given degree.

The first property tells us that the root of a heap-ordered tree contains the smallest key in the tree.

The second property implies that an n-node binomial heap H consists of at most [image] lg n [image] + 1 binomial trees. To see why, observe that the binary representation of n has [image] lg n [image] + 1 bits, say [image] b [image] lg n [image] ,b [image] lg n [image] - 1 , . . . , b0 [image] , so that [image]. By property 1 of Lemma 20.1, therefore, binomial tree Bi appears in H if and only if bit bi = 1. Thus, binomial heap H contains at most [image] lg n [image] + 1 binomial trees.

Figure 20.3 A binomial heap H with n = 13 nodes. (a) The heap consists of binomial trees B0, B2, and B3, which have 1, 4, and 8 nodes respectively, totaling n = 13 nodes. Since each binomial tree is heap-ordered, the key of any node is no less than the key of its parent. Also shown is the root list, which is a linked list of roots in order of increasing degree. (b) A more detailed representation of binomial heap H. Each binomial tree is stored in the left-child, right-sibling representation, and each node stores its degree.

Figure 20.3(a) shows a binomial heap H with 13 nodes. The binary representation of 13 is [image] 1101 [image], and H consists of heap-ordered binomial trees B3, B2, and B0, having 8, 4, and 1 nodes respectively, for a total of 13 nodes.

Representing binomial heaps

As shown in Figure 20.3(b), each binomial tree within a binomial heap is stored in the left-child, right-sibling representation of Section 11.4. Each node has a key field and any other satellite information required by the application. In addition, each node x contains pointers p[x] to its parent, child [x] to its leftmost child, and sibling[x] to the sibling of x immediately to its right. If node x is a root, then p[x] = NIL. If node x has no children, then child[x] = NIL, and if x is the rightmost child of its parent, then sibling[x] = NIL. Each node x also contains the field degree[x] , which is the number of children of x.

Figure 20.4 The binomial tree B4 with nodes labeled in binary by a postorder walk.

As Figure 20.3 also shows, the roots of the binomial trees within a binomial heap are organized in a linked list, which we refer to as the root list. The degrees of the roots strictly increase as we traverse the root list. By the second binomial-heap property, in an n-node binomial heap the degrees of the roots are a subset of {0, 1, . . . , [image] 1g n [image] }. The sibling field has a different meaning for roots than for nonroots. If x is a root, then sibling[x] points to the next root in the root list. (As usual, sibling[x] = NIL if x is the last root in the root list.)

A given binomial heap H is accessed by the field head[H], which is simply a pointer to the first root in the root list of H. If binomial heap H has no elements, then head[H] = NIL.

Exercises

20.1-1

Suppose that x is a node in a binomial tree within a binomial heap, and assume that sibling[x] [image] NIL. If x is not a root, how does degree[sibling[x]] compare to degree[x]? How about if x is a root?

20.1-2

If x is a nonroot node in a binomial tree within a binomial heap, how does degree[p[x]] compare to degree[x]?

20.1-3

Suppose we label the nodes of binomial tree Bk in binary by a postorder walk, as in Figure 20.4. Consider a node x labeled l at depth i, and let j = k - i. Show that x has j l's in its binary representation. How many binary k-strings are there that contain exactly j 1's? Show that the degree of x is equal to the number of 1's to the right of the rightmost 0 in the binary representation of l.

20.2 Operations on binomial heaps

In this section, we show how to perform operations on binomial heaps in the time bounds shown in Figure 20.1. We shall only show the upper bounds; the lower bounds are left as Exercise 20.2-10.

Creating a new binomial heap

To make an empty binomial heap, the MAKE-BINOMIAL-HEAP procedure simply allocates and returns an object H, where head[H] = NIL. The running time is [image] (1).

Finding the minimum key

The procedure BINOMIAL-HEAP-MINIMUM returns a pointer to the node with the minimum key in an n-node binomial heap H. This implementation assumes that there are no keys with value [image]. (See Exercise 20.2-5.)

BINOMIAL-HEAP-MINIMUM(H)

1  y    NIL

2  x    head[H]

3  min

4  while x    NIL

5      do if key[x] < min

6            then min    key[x]

7                 y    x

8         x    sibling[x]

9  return y

Since a binomial heap is heap-ordered, the minimum key must reside in a root node. The BINOMIAL-HEAP-MINIMUM procedure checks all roots, which number at most [image] lg n [image] + 1, saving the current minimum in min and a pointer to the current minimum in y. When called on the binomial heap of Figure 20.3, BINOMIAL-HEAP-MINIMUM returns a pointer to the node with key 1.

Because there are at most [image] lg n [image] + 1 roots to check, the running time of BINOMIAL-HEAP-MINIMUM is O(lg n).

Uniting two binomial heaps

The operation of uniting two binomial heaps is used as a subroutine by most of the remaining operations. The BINOMIAL-HEAP-UNION procedure repeatedly links binomial trees whose roots have the same degree. The following procedure links the Bk-1 tree rooted at node y to the Bk-1 tree rooted at node z; that is, it makes z the parent of y. Node z thus becomes the root of a Bk tree.

BINOMIAL-LINK(y,z)

1  p[y]    z

2  sibling[y]    child[z]

3  child[z]    y

4  degree[z]    degree[z]+1

The BINOMIAL-LINK procedure makes node y the new head of the linked list of node z's children in O(1) time. It works because the left-child, right-sibling representation of each binomial tree matches the ordering property of the tree: in a Bk tree, the leftmost child of the root is the root of a Bk-1 tree.

The following procedure unites binomial heaps H1 and H2, returning the resulting heap. It destroys the representations of H1 and H2 in the process. Besides BINOMIAL-LINK, the procedure uses an auxiliary procedure BINOMIAL-HEAP-MERGE that merges the root lists of H1 and H2 into a single linked list that is sorted by degree into monotonically increasing order. The BINOMIAL-HEAP-MERGE procedure, whose pseudocode we leave as Exercise 20.2-2, is similar to the MERGE procedure in Section 1.3.1.

Figure 20.5 shows an example of BINOMIAL-HEAP-UNION in which all four cases given in the pseudocode occur.

The BINOMIAL-HEAP-UNION procedure has two phases. The first phase, performed by the call of BINOMIAL-HEAP-MERGE, merges the root lists of binomial heaps H1 and H2 into a single linked list H that is sorted by degree into monotonically increasing order. There might be as many as two roots (but no more) of each degree, however, so the second phase links roots of equal degree until at most one root remains of each degree. Because the linked list H is sorted by degree, we can perform all the link operations quickly.

In detail, the procedure works as follows. Lines 1-3 start by merging the root lists of binomial heaps H1 and H2 into a single root list H. The root lists of H1 and H2 are sorted by strictly increasing degree, and BINOMIAL-HEAP-MERGE returns a root list H that is sorted by monotonically increasing degree. If the root lists of H1 and H2 have m roots altogether, BINOMIAL-HEAP-MERGE runs in O(m) time by repeatedly examining the roots at the heads of the two root lists and appending the root with the lower degree to the output root list, removing it from its input root list in the process.

The BINOMIAL-HEAP-UNION procedure next initializes some pointers into the root list of H. First, it simply returns in lines 4-5 if it happens to be uniting two empty binomial heaps. From line 6 on, therefore, we know that H has at least one root. Throughout the procedure, we maintain three pointers into the root list:

x points to the root currently being examined,

prev-x points to the root preceding x on the root list: sibling[prev-x] = x, and

next-x points to the root following x on the root list: sibling[x] = next-x.

Initially, there are at most two roots on the root list H of a given degree: because H1 and H2 were binomial heaps, they each had only one root of a given degree. Moreover, BINOMIAL-HEAP-MERGE guarantees us that if two roots in H have the same degree, they are adjacent in the root list.

In fact, during the execution of BINOMlAL-HEAP-UNION, there may be three roots of a given degree appearing on the root list H at some time. We shall see in a moment how this situation could occur. At each iteration of the while loop of lines 9-21, therefore, we decide whether to link x and next-x based on their degrees and possibly the degree of sibling[next-x]. An invariant of the loop is that each time we start the body of the loop, both x and next-x are non-NIL.

Case 1, shown in Figure 20.6(a), occurs when degree[x] [image] degree[next-x], that is, when x is the root of a Bk-tree and next-x is the root of a Bl-tree for some l > k. Lines 11-12 handle this case. We don't link x and next-x, so we simply march the pointers one position further down the list. Updating next-x to point to the node following the new node x is handled in line 21, which is common to every case.

Case 2, shown in Figure 20.6(b), occurs when x is the first of three roots of equal degree, that is, when

degree[x] = degree[next-x] = degree[sibling[next-x]].

We handle this case in the same manner as case 1: we just march the pointers one position further down the list. Line 10 tests for both cases 1 and 2, and lines 11-12 handle both cases.

Cases 3 and 4 occur when x is the first of two roots of equal degree, that is, when

degree[x] = degree[next-x]    degree[sibling[next-x]].

These cases may occur on the next iteration after any case, but one of them always occurs immediately following case 2. In cases 3 and 4, we link x and next-x. The two cases are distinguished by whether x or next-x has the smaller key, which determines the node that will be the root after the two are linked.

In case 3, shown in Figure 20.6(c), key[x] [image] key[next-x], so next-x is linked to x. Line 14 removes next-x from the root list, and line 15 makes next-x the leftmost child of x.

Figure 20.5 The execution of BINOMIAL-HEAP-UNION.(a) Binomial heaps H1 and H2. (b) Binomial heap H is the output of BINOMIAL-HEAP-MERGE(H1,H2). Initially, x is the first root on the root list of H. Because both x and next-x have degree 0 and key[x] < key[next-x], case 3 applies. (c) After the link occurs, x is the first of three roots with the same degree, so case 2 applies. (d) After all the pointers move down one position in the root list, case 4 applies, since x is the first of two roots of equal degree. (e) After the link occurs, case 3 applies. (f) After another link, case 1 applies, because x has degree 3 and next-x has degree 4. This iteration of the while loop is the last, because after the pointers move down one position in the root list, next-x = NIL.

In case 4, shown in Figure 20.6(d), next-x has the smaller key, so x is linked to next-x. Lines 16-18 remove x from the root list, which has two cases depending on whether x is the first root on the list (line 17) or is not (line 18). Line 19 then makes x the leftmost child of next-x, and line 20 updates x for the next iteration.

Following either case 3 or case 4, the setup for the next iteration of the while loop is the same. We have just linked two Bk-trees to form a Bk+l-tree, which x now points to. There were already zero, one, or two other Bk+1-trees on the root list from the output of BINOMIAL-HEAP-MERGE, so x is now the first of either one, two, or three Bk+l-trees on the root list. If x is the only one, then we enter case 1 in the next iteration: degree [x] [image] degree[next-x]. If x is the first of two, then we enter either case 3 or case 4 in the next iteration. It is when x is the first of three that we enter case 2 in the next iteration.

The running time of BINOMIAL-HEAP-UNION is O(1g n), where n is the total number of nodes in binomial heaps H1 and H2. We can see this as follows. Let H1 contain n1 nodes and H2 contain n2 nodes, so that n = n1 + n2. Then, H1 contains at most [image] 1g n1 [image] + 1 roots and H2 contains at most [image] 1g n2 [image] + 1 roots, so H contains at most [image] 1g n2 [image] + [image] 1g n1 [image] + 2 [image] 2 [image] 1g n [image] + 2 = O(1g n) roots immediately after the call of BINOMIAL-HEAP-MERGE. The time to perform BINOMIAL-HEAP-MERGE is thus O(lg n). Each iteration of the while loop takes O(1) time, and there are at most [image] 1g n1 [image] + [image] 1g n2 [image] + 2 iterations because each iteration either advances the pointers one position down the root list of H or removes a root from the root list. The total time is thus O(lg n).

Figure 20.6 The four cases that occur in BINOMIAL-HEAP-UNION. Labels a, b, c, and d serve only to identify the roots involved; they do not indicate the degrees or keys of these roots. In each case, x is the root of a Bk-tree and l > k. (a) Case 1: degree[x] degree[next-x]. The pointers move one position further down the root list. (b) Case 2: degree[x] = degree[next-x] = degree[sibling[next-x]]. Again, the pointers move one position further down the list, and the next iteration executes either case 3 or case 4. (c) Case 3: degree[x] = degree[next-x] degree[sibling[next-x] and key[x] key[next-x]. We remove next-x from the root list and link it to x, creating a Bk+1-tree. (d) Case 4: degree[x] = degree[next-x] degree[sibling[next-x]] and key[next-x] key[x]. We remove x from the root list and link it to next-x, again creating a B k+1-tree.

Inserting a node

The following procedure inserts node x into binomial heap H, assuming of course that node x has already been allocated and key[x] has already been filled in.

BINOMIAL-HEAP-INSERT(H,x)

1  H'    MAKE-BINOMIAL-HEAP()

2  p[x]    NIL

3  child[x]    NIL

4  sibling[x]    NIL

5  degree[x]    0

6  head[H']    x

7  H    BINOMIAL-HEAP-UNION(H,H')

The procedure simply makes a one-node binomial heap H' in O(1) time and unites it with the n-node binomial heap H in O(1g n) time. The call to BINOMIAL-HEAP-UNION takes care of freeing the temporary binomial heap H'. (A direct implementation that does not call BINOMIAL-HEAP-UNION is given as Exercise 20.2-8.)

Extracting the node with minimum key

The following procedure extracts the node with the minimum key from binomial heap H and returns a pointer to the extracted node.

BINOMIAL-HEAP-EXTRACT-MIN(H)

1  find the root x with the minimum key in the root list of H,

and remove x from the root list of H

2  H'    MAKE-BINOMIAL-HEAP()

3  reverse the order of the linked list of x's children,

and set head[H'] to point to the head of the resulting list

4  H    BINOMIAL-HEAP-UNION(H,H')

5  return x

This procedure works as shown in Figure 20.7. The input binomial heap H is shown in Figure 20.7(a). Figure 20.7(b) shows the situation after line 1: the root x with the minimum key has been removed from the root list of H. If x is the root of a Bk-tree, then by property 4 of Lemma 20.1, x's children, from left to right, are roots of Bk-1-, Bk-2-, . . . , B0-trees. Figure 20.7(c) shows that by reversing the list of x's children in line 3, we have a binomial heap H' that contains every node in x's tree except for x itself. Because x's tree is removed from H in line 1, the binomial heap that results from uniting H and H' in line 4, shown in Figure 20.7(d), contains all the nodes originally in H except for x. Finally, line 5 returns x.

Figure 20.7 The action of BINOMIAL-HEAP-EXTRACT-MIN. (a) A binomial heap H. (b) The root x with minimum key is removed from the root list of H. (c) The linked list of x's children is reversed, giving another binomial heap H'. (d) The result of uniting H and H'.

Since each of lines 1-4 takes O(lg n) time if H has n nodes, BINOMIAL-HEAP-EXTRACT-MIN runs in O(lg n) time.

Decreasing a key

The following procedure decreases the key of a node x in a binomial heap H to a new value k. It signals an error if k is greater than x's current key.

BINOMIAL-HEAP-DECREASE-KEY (H,x,k)

1  if k > key[x]

2      then error "new key is greater than current key"

3  key[x]    k

4  y    x

5  z    p[y]

6  while z    NIL and key[y] < key[z]

7      do exchange key[y]    key[z]

8            If y and z have satellite fields, exchange them, too.

9         y    z

10        z    p[y]

As shown in Figure 20.8, this procedure decreases a key in the same manner as in a binary heap: by "bubbling up" the key in the heap. After ensuring that the new key is in fact no greater than the current key and then assigning the new key to x, the procedure goes up the tree, with y initially pointing to node x. In each iteration of the while loop of lines 6-10, key[y] is checked against the key of y's parent z. If y is the root or key[y] [image] key[z], the binomial tree is now heap-ordered. Otherwise, node y violates heap ordering, so its key is exchanged with the key of its parent z, along with any other satellite information. The procedure then sets y to z, going up one level in the tree, and continues with the next iteration.

The BINOMIAL-HEAP-DECREASE-KEY procedure takes O(lg n) time. By property 2 of Lemma 20.1, the maximum depth of x is [image] lg n [image] , so the while loop of lines 6-10 iterates at most [image] lg n [image] times.

Deleting a key

It is easy to delete a node x's key and satellite information from binomial heap H in O(lg n) time. The following implementation assumes that no node currently in the binomial heap has a key of - [image] .

Figure 20.8 The action of BINOMIAL-HEAP-DECREASE-KEY. (a) The situation just before line 5 of the first iteration of the while loop. Node y has had its key decreased to 7, which is less than the key of y's parent z. (b) The keys of the two nodes are exchanged, and the situation just before line 5 of the second iteration is shown. Pointers y and z have moved up one level in the tree, but heap order is still violated. (c) After another exchange and moving pointers y and z up one more level, we finally find that heap order is satisfied, so the while loop terminates.

BINOMIAL-HEAP-DELETE(H,x)

1 BINOMIAL-HEAP-DECREASE-KEY(H,x,-  )

2 BINOMIAL-HEAP-EXTRACT-MIN(H)

The BINOMIAL-HEAP-DELETE procedure makes node x have the unique minimum key in the entire binomial heap by giving it a key of - [image]. (Exercise 20.2-6 deals with the situation in which - [image] cannot appear as a key, even temporarily.) It then bubbles this key and the associated satellite information up to a root by calling BINOMIAL-HEAP-DECREASE-KEY. This root is then removed from H by a call of BINOMIAL-HEAP-EXTRACT-MIN.

The BINOMIAL-HEAP-DELETE procedure takes O(lg n) time.

Exercises

20.2-1

Give an example of two binary heaps with n elements each such that BUILD-HEAP takes [image] (n) time on the concatenation of their arrays.

20.2-2

Write pseudocode for BINOMIAL-HEAP-MERGE.

20.2-3

Show the binomial heap that results when a node with key 24 is inserted into the binomial heap shown in Figure 20.7(d).

20.2-4

Show the binomial heap that results when the node with key 28 is deleted from the binomial heap shown in Figure 20.8(c).

20.2-5

Explain why the BINOMIAL-HEAP-MINIMUM procedure might not work correctly if keys can have the value [image] . Rewrite the pseudocode to make it work correctly in such cases.

20.2-6

Suppose there is no way to represent the key - [image]. Rewrite the BINOMIAL-HEAP-DELETE procedure to work correctly in this situation. It should still take O(lg n) time.

20.2-7

Discuss the relationship between inserting into a binomial heap and incrementing a binary number and the relationship between uniting two binomial heaps and adding two binary numbers.

20.2-8

In light of Exercise 20.2-7, rewrite BINOMIAL-HEAP-INSERT to insert a node directly into a binomial heap without calling BINOMIAL-HEAP-UNION.

20.2-9

Show that if root lists are kept in strictly decreasing order by degree (instead of strictly increasing order), each of the binomial heap operations can be implemented without changing its asymptotic running time.

20.2-10

Find inputs that cause BINOMIAL-HEAP-EXTRACT-MIN, BINOMIAL-HEAP-DECREASE-KEY, and BINOMIAL-HEAP-DELETE to run in [image] (lg n) time. Explain why the worst-case running times of BINOMIAL-HEAP-INSERT, BINOMIAL-HEAP-MINIMUM, and BINOMIAL-HEAP-UNION are [image] (lg n) but not [image] (lg n). (See Problem 2-5.)

Problems

20-1 2-3-4 heaps

Chapter 19 introduced the 2-3-4 tree, in which every internal node (other than possibly the root) has two, three, or four children and all leaves have the same depth. In this problem, we shall implement 2-3-4 heaps, which support the mergeable-heap operations.

The 2-3-4 heaps differ from 2-3-4 trees in the following ways. In 2-3-4 heaps, only leaves store keys, and each leaf x stores exactly one key in the field key[x]. There is no particular ordering of the keys in the leaves; that is, from left to right, the keys may be in any order. Each internal node x contains a value small[x] that is equal to the smallest key stored in any leaf in the subtree rooted at x. The root r contains a field height [r] that is the height of the tree. Finally, 2-3-4 heaps are intended to be kept in main memory, so that disk reads and writes are not needed.

Implement the following 2-3-4 heap operations. Each of the operations in parts (a)-(e) should run in O(lg n) time on a 2-3-4 heap with n elements. The UNION operation in part (f) should run in O(lg n) time, where n is the number of elements in the two input heaps.

a. MINIMUM, which returns a pointer to the leaf with the smallest key.

b. DECREASE-KEY, which decreases the key of a given leaf x to a given value k [image] key[x].

c. INSERT, which inserts leaf x with key k.

d. DELETE, which deletes a given leaf x.

e. EXTRACT-MIN, which extracts the leaf with the smallest key.

f. UNION, which unites two 2-3-4 heaps, returning a single 2-3-4 heap and destroying the input heaps.

20-2 Minimum-spanning-tree algorithm using mergeable heaps

Chapter 24 presents two algorithms to solve the problem of finding a minimum spanning tree of an undirected graph. Here, we shall see how mergeable heaps can be used to devise a different minimum-spanning-tree algorithm.

We are given a connected, undirected graph G = (V, E) with a weight function w: E [image] R. We call w(u,v) the weight of edge (u, v). We wish to find a minimum spanning tree for G: an acyclic subset T [image] E that connects all the vertices in V and whose total weight

is minimized.

The following pseudocode, which can be proven correct using techniques from Section 24.1, constructs a minimum spanning tree T. It maintains a partition {Vi} of the vertices of V and, with each set Vi, a set

Ei    {(u,v): u    Vi or v   Vi}

of edges incident on vertices in Vi.

Describe how to implement this algorithm using the mergeable-heap operations given in Figure 20.1. Give the running time of your implementation, assuming that the mergeable heaps are implemented by binomial heaps.

Chapter notes

Binomial heaps were introduced in 1978 by Vuillemin [196]. Brown[36, 37] studied their properties in detail.

Go to Chapter 21 Back to Table of Contents


Intro to Algorithms: CHAPTER 21: FIBONACCI HEAPS

 Return to Table of Contents

CHAPTER 21: FIBONACCI HEAPS

In Chapter 20, we saw how binomial heaps support in O(lg n) worst-case time the mergeable-heap operations INSERT, MINIMUM, EXTRACT-MIN, and UNION, plus the operations DECREASE-KEY and DELETE. In this chapter, we shall examine Fibonacci heaps, which support the same operations but have the advantage that operations that do not involve deleting an element run in O(1) amortized time.

From a theoretical standpoint, Fibonacci heaps are especially desirable when the number of EXTRACT-MIN and DELETE operations is small relative to the number of other operations performed. This situation arises in many applications. For example, some algorithms for graph problems may call DECREASE-KEY once per edge. For dense graphs, which have many edges, the O(1) amortized time of each call of DECREASE-KEY adds up to a big improvement over the [image] (lg n) worst-case time of binary or binomial heaps. The asymptotically fastest algorithms to date for problems such as computing minimum spanning trees (Chapter 24) and finding single-source shortest paths (Chapter 25) make essential use of Fibonacci heaps.

From a practical point of view, however, the constant factors and programming complexity of Fibonacci heaps make them less desirable than ordinary binary (or k-ary) heaps for most applications. Thus, Fibonacci heaps are predominantly of theoretical interest. If a much simpler data structure with the same amortized time bounds as Fibonacci heaps were developed, it would be of great practical use as well.

Like a binomial heap, a Fibonacci heap is a collection of trees. Fibonacci heaps, in fact, are loosely based on binomial heaps. If neither DECREASE-KEY nor DELETE is ever invoked on a Fibonacci heap, each tree in the heap is like a binomial tree. Fibonacci heaps differ from binomial heaps, however, in that they have a more relaxed structure, allowing for improved asymptotic time bounds. Work that maintains the structure can be delayed until it is convenient to perform.

Like the dynamic tables of Section 18.4, Fibonacci heaps offer a good example of a data structure designed with amortized analysis in mind. The intuition and analyses of Fibonacci heap operations in the remainder of this chapter rely heavily on the potential method of Section 18.3.

The exposition in this chapter assumes that you have read Chapter 20 on binomial heaps. The specifications for the operations appear in that chapter, as does the table in Figure 20.1, which summarizes the time bounds for operations on binary heaps, binomial heaps, and Fibonacci heaps. Our presentation of the structure of Fibonacci heaps relies on that of binomial heap structure. You will also find that some of the operations performed on Fibonacci heaps are similar to those performed on binomial heaps.

Like binomial heaps, Fibonacci heaps are not designed to give efficient support to the operation SEARCH; operations that refer to a given node therefore require a pointer to that node as part of their input.

Section 21.1 defines Fibonacci heaps, discusses their representation, and presents the potential function used for their amortized analysis. Section 21.2 shows how to implement the mergeable-heap operations and achieve the amortized time bounds shown in Figure 20.1. The remaining two operations, DECREASE-KEY and DELETE, are presented in Section 21.3. Finally, Section 21.4 finishes off a key part of the analysis.

21.1 Structure of Fibonacci heaps

Like a binomial heap, a Fibonacci heap is a collection of heap-ordered trees. The trees in a Fibonacci heap are not constrained to be binomial trees, however. Figure 21.1 (a) shows an example of a Fibonacci heap.

Unlike trees within binomial heaps, which are ordered, trees within Fibonacci heaps are rooted but unordered. As Figure 21.1(b) shows, each node x contains a pointer p[x] to its parent and a pointer child[x] to any one of its children. The children of x are linked together in a circular, doubly linked list , which we call the child list of x. Each child y in a child list has pointers left[y] and right[y] that point to y's left and right siblings, respectively. If node y is an only child, then left[y] = right[y] = y. The order in which siblings appear in a child list is arbitrary.

Circular, doubly linked lists (see Section 11.2) have two advantages for use in Fibonacci heaps. First, we can remove a node from a circular, doubly linked list in O(1) time. Second, given two such lists, we can concatenate them (or "splice" them together) into one circular, doubly linked list in O(1) time. In the descriptions of Fibonacci heap operations, we shall refer to these operations informally, letting the reader fill in the details of their implementations.

Two other fields in each node will be of use. The number of children in the child list of node x is stored in degree[x]. The boolean-valued field mark[x] indicates whether node x has lost a child since the last time x was made the child of another node. We won't worry about the details of marking nodes until Section 21.3. Newly created nodes are unmarked, and a node x becomes unmarked whenever it is made the child of another node.

Figure 21.1 (a) A Fibonacci heap consisting of five heap-ordered trees and 14 nodes. The dashed line indicates the root list. The minimum node of the heap is the node containing the key 3. The three marked nodes are blackened. The potential of this particular Fibonacci heap is 5 + 2 3 = 11. (b) A more complete representation showing pointers p (up arrows), child (down arrows), and left and right (sideways arrows). These details are omitted in the remaining figures in this chapter, since all the information shown here can be determined from what appears in part (a).

A given Fibonacci heap H is accessed by a pointer min[H] to the root of the tree containing a minimum key; this node is called the minimum node of the Fibonacci heap. If a Fibonacci heap H his empty, then min[H] = NIL.

The roots of all the trees in a Fibonacci heap are linked together using their left and right pointers into a circular, doubly linked list called the root list of the Fibonacci heap. The pointer min[H] thus points to the node in the root list whose key is minimum. The order of the trees within a root list is arbitrary.

We rely on one other attribute for a Fibonacci heap H: the number of nodes currently in H is kept in n[H].

Potential function

As mentioned, we shall use the potential method of Section 18.3 to analyze the performance of Fibonacci heap operations. For a given Fibonacci heap H, we indicate by t(H) the number of trees in the root list of H and by m (H) the number of marked nodes in H. The potential of Fibonacci heap H is then defined by

  (H) = t(H) + 2m(H) .

(21.1)

For example, the potential of the Fibonacci heap shown in Figure 21.1 is 5 + 2 [image] 3 = 11. The potential of a set of Fibonacci heaps is the sum of the potentials of its constituent Fibonacci heaps. We shall assume that a unit of potential can pay for a constant amount of work, where the constant is sufficiently large to cover the cost of any of the specific constant-time pieces of work that we might encounter.

We assume that a Fibonacci heap application begins with no heaps. The initial potential, therefore, is 0, and by equation (21.1), the potential is nonnegative at all subsequent times. From equation (18.2), an upper bound on the total amortized cost is thus an upper bound on the total actual cost for the sequence of operations.

Maximum degree

The amortized analyses we shall perform in the remaining sections of this chapter assume that there is a known upper bound D(n) on the maximum degree of any code in an n-node Fibonacci heap. Exercise 21.2-3 shows that when only the mergeable-heap operations are supported, D(n) = [image] lg n [image] . In Section 21.3, we shall show that when we support DECREASE-KEY and DELETE as well, D(n) = O(lg n).

21.2 Mergeable-heap operations

In this section, we describe and analyze the mergeable-heap operations as implemented for Fibonacci heaps. If only these operations--MAKE-HEAP, INSERT, MINIMUM, EXTRACT-MIN, and UNION--are to be supported, each Fibonacci heap is simply a collection of "unordered" binomial trees. An unordered binomial tree is like a binomial tree, and it, too, is defined recursively. The unordered binomial tree U0 consists of a single node, and an unordered binomial tree Uk consists of two unordered binomial trees Uk-1 for which the root of one is made into any child of the root of the other. Lemma 20.1, which gives properties of binomial trees, holds for unordered binomial trees as well, but with the following variation on property 4 (see Exercise 21.2-2):

4' For the unordered binomial tree Uk.The root has degree k, which is greater than that of any other node. The children of the root are roots of subtrees U0, U1, . . . , Uk-1 in some order.

Thus, if an n-node Fibonacci in heap is a collection of unordered binomial trees, then D(n) = lg n.

The key idea in the mergeable-heap operations on Fibonacci heaps is to delay work as long as possible. There is a performance trade-off among implementations of the various operations. If the number of trees in a Fibonacci heap is small, then we can quickly determine the new minimum node during an EXTRACT-MIN operation. However, as we saw with binomial heaps in Exercise 20.2-10, we pay a price for ensuring that the number of trees is small: it can take up to [image] (1g n) time to insert a node into a binomial heap or to unite two binomial heaps. As we shall see, we do not attempt to consolidate trees in a Fibonacci heap when we insert a new node or unite two heaps. We save the consolidation for the EXTRACT-MIN operation, which is when we really need to find the new minimum node.

Creating a new Fibonacci heap

To make an empty Fibonacci heap, the MAKE-FIB-HEAP procedure allocates and returns the Fibonacci heap object H, where n[H] = 0 and min[H] = NIL; there are no trees in H. Because t (H) = 0 and m(H) = 0, the potential of the empty Fibonacci heap is [image] (H) = 0. The amortized cost of MAKE-FIB-HEAP is thus equal to its O(1) actual cost.

Inserting a node

The following procedure inserts node x into Fibonacci heap H, assuming of course that the node has already been allocated and that key[x] has already been filled in.

FIB-HEAP-INSERT(H, x)

1  degree[x]    0

2  p[x]    NIL

3  child[x]    NIL

4  left[x]    x

5  right[x]    x

6  mark[x]    FALSE

7  concatenate the root list containing x with root list H

8  if min[H] = NIL or key[x] < key[min[H]]

9     then min[H]    x

10  n[H]    n[H] + 1

After lines 1-6 initialize the structural fields of node x, making it its own circular, doubly linked list , line 7 adds x to the root list of H in O(1) actual time. Thus, node x becomes a single-node heap-ordered tree, and thus an unordered binomial tree, in the Fibonacci heap. It has no children and is unmarked. Lines 8-9 then update the pointer to the minimum node of Fibonacci heap H if necessary. Finally, line 10 increments n[H] to reflect the addition of the new node. Figure 21.2 shows a node with key 21 inserted into the Fibonacci heap of Figure 21.1.

Unlike the BINOMIAL-HEAP-INSERT procedure, FIB-HEAP-INSERT makes no attempt to consolidate the trees within the Fibonacci heap. If k consecutive FIB-HEAP-INSERT operations occur, then k single-node trees are added to the root list.

Figure 21.2 Inserting a node into a Fibonacci heap. (a) A Fibonacci heap H. (b) Fibonacci heap H after the node with key 21 has been inserted. The node becomes its own heap-ordered tree and is then added to the root list, becoming the left sibling of the root.

To determine the amortized cost of FIB-HEAP-INSERT, let H be the input Fibonacci heap and H' be the resulting Fibonacci heap. Then, t(H') = t(H) + 1 and m(H') = m(H), and the increase in potential is

((t(H) + 1) + 2m(H)) - (t(H) + 2m(H)) = 1 .

Since the actual cost is O(1), the amortized cost is O(1) + 1 = O(1).

Finding the minimum node

The minimum node of a Fibonacci heap H is given by the pointer min[H], so we can find the minimum node in O(1) actual time. Because the potential of H does not change, the amortized cost of this operation is equal to its O(1) actual cost.

Uniting two Fibonacci heaps

The following procedure unites Fibonacci heaps H1 and H2, destroying H1 and H2 in the process.

FIB-HEAP-UNION(H1,H2)

1   H    MAKE-FIB-HEAP()

2   min[H]    min[H1]

3   concatenate the root list of H2 with the root list of H

4   if (min[H1] = NIL) or (min[H2]    NIL and min[H2] < min[H1])

5   then min[H]    min[H2]

6   n[H]    n[H1] + n[H2]

7   free the objects H1 and H2

8   return H

Lines 1-3 concatenate the root lists of H1 and H2 into a new root list H. Lines 2, 4, and 5 set the minimum node of H, and line 6 sets n[H] to the total number of nodes. The Fibonacci heap objects H1 and H2 are freed in line 7, and line 8 returns the resulting Fibonacci heap H. As in the FIB-HEAP-INSERT procedure, no consolidation of trees occurs.

The change in potential is

  (H) - (  (H1) +   (H2))

=  (t(H) + 2m(H)) - ((t(H1) + 2 m(H1)) + (t(H2) + 2m(H2)))

=  0,

because t(H) = t(H1) + t(H2) and m(H) = m(H1) + m(H2). The amortized cost of FIB-HEAP-UNION is therefore equal to its O(1) actual cost.

Extracting the minimum node

The process of extracting the minimum node is the most complicated of the operations presented in this section. It is also where the delayed work of consolidating trees in the root list finally occurs. The following pseudocode extracts the minimum node. The code assumes for convenience that when a node is removed from a linked list, pointers remaining in the list are updated, but pointers in the extracted node are left unchanged. It also uses the auxiliary procedure CONSOLIDATE, which will be presented shortly.

FIB-HEAP-EXTRACT-MIN(H)

1  z    min[H]

2  if z    NIL

3      then for each child x of z

4               do add x to the root list of H

5                  p[x]    NIL

6           remove z from the root list of H

7           if z = right[z]

8              then min[H]    NIL

9              else min[H]    right[z]

10                   CONSOLIDATE(H)

11           n[H]    n[H] - 1

12  return z

As shown in Figure 21.3, FIB-HEAP-EXTRACT-MIN works by first making a root out of each of the minimum node's children and removing the minimum node from the root list. It then consolidates the root list by linking roots of equal degree until at most one root remains of each degree.

We start in line 1 by saving a pointer z to the minimum node; this pointer is returned at the end. If z = NIL, then Fibonacci heap H is already empty and we are done. Otherwise, as in the BINOMIAL-HEAP-EXTRACT-MIN procedure, we delete node z from H by making all of z's children roots of H in lines 3-5 (putting them into the root list) and removing z from the root list in line 6. If z = right[z] after line 6, then z was the only node on the root list and it had no children, so all that remains is to make the Fibonacci heap empty in line 8 before returning z. Otherwise, we set the pointer min[H] into the root list to point to a node other than z (in this case, right[z]). Figure 21.3(b) shows the Fibonacci heap of Figure 21.3(a) after line 9 has been performed.

The next step, in which we reduce the number of trees in the Fibonacci heap, is consolidating the root list of H; this is performed by the call CONSOLIDATE(H). Consolidating the root list consists of repeatedly executing the following steps until every root in the root list has a distinct degree value.

1. Find two roots x and y in the root list with the same degree, where key[x] [image] key[y].

2. Link y to x: remove y from the root list, and make y a child of x. This operation is performed by the FIB-HEAP-LINK procedure. The field degree[x] is incremented, and the mark on y, if any, is cleared.

The procedure CONSOLIDATE uses an auxiliary array A[0 . . D(n[H])]; if A[i] = y, then y is currently a root with degree[y] = i.

CONSOLIDATE(H)

1 for i    0 to D(n[H])

2      do A[i]    NIL

3 for each node w in the root list of H

4      do x    w

5         d    degree[x]

6         while A[d]    NIL

7            do y    A[d]

8               if key[x] > key[y]

9                  then exchange x    y

10                FIB-HEAP-LINK(H,y,x)

11                A[d]    NIL

12                d    d + 1

13         A[d]    x

14 min[H]    NIL

15 for i    0 to D(n[H])

16      do if A[i]    NIL

17            then add A[i] to the root list of H

18                 if min[H] = NIL or key[A[i]] < key[min[H]]

19                    then min[H]    A[i]

FIB-HEAP-LINK(H, y, x)

1  remove y from the root list of H

2  make y a child of x, incrementing degree[x]

3  mark[y]    FALSE

In detail, the CONSOLIDATE procedure works as follows. In lines 1-2, we initialize A by making each entry NIL. When we are done processing each root w, it ends up in a tree rooted at some node x, which may or may not be identical to w. Array entry A[degree[x]] will then be set to point to x. In the for loop of lines 3-13, we examine each root w in the root list. The invariant maintained during each iteration of the for loop is that node x is the root of the tree containing node w. The while loop of lines 6-12 maintains the invariant that d = degree[x] (except in line 11, as we shall see in a moment). In each iteration of the while loop, A[d] points to some root y. Because d = degree[x] = degree[y], we want to link x and y. Whichever of x and y has the smaller key becomes the parent of the other as a result of the link operation, and so lines 8-9 exchange the pointers to x and y if necessary. Next, we link y to x by the call FIB-HEAP-LINK(H,y,x) in line 10. This call increments degree[x] but leaves degree[y] as d. Because node y is no longer a root, the pointer to it in array A is removed in line 11. Because the value of degree[x] is incremented by the call of FIB-HEAP-LINK, line 12 restores the invariant that d = degree[x]. We repeat the while loop until A[d] = NIL, in which case there is no other root with the same degree as x. We set A[d] to x in line 13 and perform the next iteration of the for loop. Figures 21.3(c)-(e) show the array A and the resulting trees after the first three iterations of the for loop of lines 3-13. In the next iteration of the for loop, three links occur; their results are shown in Figures 21.3(f)-(h). Figures 21.3(i)-(1) show the result of the next four iterations of the for loop.

Figure 21.3 The action of FIB-HEAP-EXTRACT-MIN. (a) A Fibonacci heap H. (b) The situation after the minimum node z is removed from the root list and its children are added to the root list. (c)-(e) The array A and the trees after each of the first three iterations of the for loop of lines 3-13 of the procedure CONSOLIDATE. The root list is processed by starting at the minimum node and following right pointers. Each part shows the values of w and x at the end of an iteration. (f)-(h) The next iteration of the for loop, with the values of w and x shown at the end of each iteration of the while loop of lines 6-12. Part (f) shows the situation after the first time through the while loop. The node with key 23 has been linked to the node with key 7, which is now pointed to by x. In part (g), the node with key 17 has been linked to the node with key 7, which is still pointed to by x. In part (h), the node with key 24 has been linked to the node with key 7. Since no node was previously pointed to by A[3], at the end of the for loop iteration, A[3] is set to point to the root of the resulting tree. (i)-(l) The situation after each of the next four iterations of the while loop. (m) Fibonacci heap H after reconstruction of the root list from the array A and determination of the new min[H] pointer.

All that remains is to clean up. Once the for loop of lines 3-13 completes, line 14 empties the root list, and lines 15-19 reconstruct it. The resulting Fibonacci heap is shown in Figure 21.3(m). After consolidating the root list, FIB-HEAP-EXTRACT-MIN finishes up by decrementing n[H] in line 11 and returning a pointer to the deleted node z in line 12.

Observe that if all trees in the Fibonacci heap are unordered binomial trees before FIB-HEAP-EXTRACT-MIN is executed, then they are all unordered binomial trees afterward. There are two ways in which trees are changed. First, in lines 3-5 of FIB-HEAP-EXTRACT-MIN, each child x of root z becomes a root. By Exercise 21.2-2, each new tree is itself an unordered binomial tree. Second, trees are linked by FIB-HEAP-LINK only if they have the same degree. Since all trees are unordered binomial trees before the link occurs, two trees whose roots each have k children must have the structure of Uk. The resulting tree therefore has the structure of Uk+1.

We are now ready to show that the amortized cost of extracting the minimum node of an n-node Fibonacci heap is O(D(n)). Let H denote the Fibonacci heap just prior to the FIB-HEAP-EXTRACT-MIN operation.

The actual cost of extracting the minimum node can be accounted for as follows. An O(D(n)) contribution comes from there being at most D(n) children of the minimum node that are processed in FIB-HEAP-EXTRACT-MIN and from the work in lines 1-2 and 14-19 of CONSOLIDATE. It remains to analyze the contribution from the for loop of lines 3-13. The size of the root list upon calling CONSOLIDATE is at most D(n) + t(H) - 1, since it consists of the original t(H) root-list nodes, minus the extracted root node, plus the children of the extracted node, which number at most D(n). Every time through the while loop of lines 6-12, one of the roots is linked to another, and thus the total amount of work performed in the for loop is at most proportional to D(n) + t(H). Thus, the total actual work is O(D(n) + t(H)).

The potential before extracting the minimum node is t(H) + 2m(H), and the potential afterward is at most (D(n) + 1) + 2m(H), since at most D(n) + 1 roots remain and no nodes become marked during the operation. The amortized cost is thus at most

O(D(n) + t(H)) + ((D(n) + 1) + 2m(H)) - (t(H) + 2m(H))

= O(D(n)) + O(t(H)) - t(H)

= O(D(n)),

since we can scale up the units of potential to dominate the constant hidden in O(t(H)). Intuitively, the cost of performing each link is paid for by the reduction in potential due to the link reducing the number of roots by one.

Exercises

21.2-1

Show the Fibonacci heap that results from calling FIB-HEAP-EXTRACT-MIN on the Fibonacci heap shown in Figure 21.3(m).

21.2-2

Prove that Lemma 20.1 holds for unordered binomial trees, but with property 4' in place of property 4.

21.2-3

Show that if only the mergeable-heap operations are supported, the maximum degree D(n) in an n-node Fibonacci heap is at most   [image] 1g n  [image] .

21.2-4

Professor McGee has devised a new data structure based on Fibonacci heaps. A McGee heap has the same structure as a Fibonacci heap and supports the mergeable-heap operations. The implementations of the operations are the same as for Fibonacci heaps, except that insertion and union perform consolidation as their last step. What are the worst-case running time of operations on McGee heaps? How novel is the professor's data structure?

21.2-5

Argue that when the only operations on keys are comparing two keys (as is the case for all the implementations in this chapter), not all of the mergeable-heap operations can run in O(1) amortized time.

21.3 Decreasing a key and deleting a node

In this section, we show how to decrease the key of a node in a Fibonacci heap in O(1) amortized time and how to delete any node from an n-node Fibonacci heap in O(D(n)) amortized time. These operations do not preserve the property that all trees in the Fibonacci heap are unordered binomial trees. They are close enough, however, that we can bound the maximum degree D(n) by O(1g n). Proving this bound will imply that FIB-HEAP-EXTRACT-MIN and FIB-HEAP-DELETE run in O(1g n) amortized time.

Decreasing a key

In the following pseudocode for the operation FIB-HEAP-DECREASE-KEY, we assume as before that removing a node from a linked list does not change any of the structural fields in the removed node.

FIB-HEAP-DECREASE-KEY(H,x,k)

1  if k > key[x]

2     then error "new key is greater than current key"

3  key[x]    k

4  y    p[x]

5  if y    NIL and key[x] < key[y]

6     then CUT(H,x,y)

7           CASCADING-CUT(H,y)

8  if key[x] < key[min[H]]

9     then min[H]    x

CUT(H,x,y)

1  remove x from the child list of y, decrementing degree[y]

2  add x to the root list of H

3  p[x]    NIL

4  mark[x]    FALSE

CASCADING-CUT(H,y)

1  z    p[y]

2  if z    NIL

3     then if mark[y] = FALSE

4             then mark[y]    TRUE

5             else CUT(H,y,z)

6                  CASCADING-CUT(H,z)

The FIB-HEAP-DECREASE-KEY procedure works as follows. Lines 1-3 ensure that the new key is no greater than the current key of x and then assign the new key to x. If x is a root or if key[x]   [image]  key[y], where y is x's parent, then no structural changes need occur, since heap order has not been violated. Lines 4-5 test for this condition.

If heap order has been violated, many changes may occur. We start by cutting x in line 6. The CUT procedure "cuts" the link between x and its parent y, making x a root.

We use the mark fields to obtain the desired time bounds. They help to produce the following effect. Suppose that x is a node that has undergone the following history:

1.     at some time, x was a root,

2.     then x was linked to another node,

3.     then two children of x were removed by cuts.

As soon as the second child has been lost, x is cut from its parent, making it a new root. The field mark[x] is TRUE if steps 1 and 2 have occurred and one child of x has been cut. The CUT procedure, therefore, clears mark[x] in line 4, since it performs step 1. (We can now see why line 3 of FIB-HEAP-LINK clears mark[y]: node y is being linked to another node, and so step 2 is being performed. The next time a child of y is cut, mark[y] will be set to TRUE.)

We are not yet done, because x might be the second child cut from its parent y since the time that y was linked to another node. Therefore, line 7 of FIB-HEAP-DECREASE-KEY performs a cascading-cut operation on y. If y is a root, then the test in line 2 of CASCADING-CUT causes the procedure to just return. If y is unmarked, the procedure marks it in line 4, since its first child has just been cut, and returns. If y is marked, however, it has just lost its second child; y is cut in line 5, and CASCADING-CUT calls itself recursively in line 6 on y's parent z. The CASCADING-CUT procedure recurses its way up the tree until either a root or an unmarked node is found.

Once all the cascading cuts have occurred, lines 8-9 of FIB-HEAP-DECREASE-KEY finish up by updating min[H] if necessary.

Figure 21.4 shows the execution of two calls of FIB-HEAP-DECREASE-KEY, starting with the Fibonacci heap shown in Figure 21.4(a). The first call, shown in Figure 21.4(b), involves no cascading cuts. The second call, shown in Figures 21.4(c)-(e), invokes two cascading cuts.

We shall now show that the amortized cost of FIB-HEAP-DECREASE-KEY is only O(1). We start by determining its actual cost. The FIB-HEAP-DECREASE-KEY procedure takes O(1) time, plus the time to perform the cascading cuts. Suppose that CASCADING-CUT is recursively called c times from a given invocation of FIB-HEAP-DECREASE-KEY. Each call of CASCADING-CUT takes O(1) time exclusive of recursive calls. Thus, the actual cost of FIB-HEAP-DECREASE-KEY, including all recursive calls, is O(c).

We next compute the change in potential. Let H denote the Fibonacci heap just prior to the FIB-HEAP-DECREASE-KEY operation. Each recursive call of CASCADING-CUT, except for the last one, cuts a marked node and clears the mark bit. Afterward, there are t(H) + c trees (the original t(H) trees, c- 1 trees produced by cascading cuts, and the tree rooted at x) and at most m(H) - c + 2 marked nodes (c - 1 were unmarked by cascading cuts and the last call of CASCADING-CUT may have marked a node). The change in potential is therefore at most

((t(H) + c) + 2(m(H) - c + 2)) - (t(H) + 2m(H)) = 4 - c .

Thus, the amortized cost of FIB-HEAP-DECREASE-KEY is at most

O(c) + 4 - c = O(1) ,

since we can scale up the units of potential to dominate the constant hidden in O(c).

You can now see why the potential function was defined to include a term that is twice the number of marked nodes. When a marked node y is cut by a cascading cut, its mark bit is cleared, so the potential is reduced by 2. One unit of potential pays for the cut and the clearing of the mark bit, and the other unit compensates for the unit increase in potential due to node y becoming a root.

Figure 21.4 Two calls of FIB-HEAP-DECREASE-KEY. (a) The initial Fibonacci heap. (b) The node with key 46 has its key decreased to 15. The node becomes a root, and its parent (with key 24), which had previously been unmarked, becomes marked. (c)-(e) The node with key 35 has its key decreased to 5. In part (c), the node, now with key 5, becomes a root. Its parent, with key 26, is marked, so a cascading cut occurs. The node with key 26 is cut from its parent and made an unmarked root in (d). Another cascading cut occurs, since the node with key 24 is marked as well. This node is cut from its parent and made an unmarked root in part (e). The cascading cuts stop at this point, since the node with key 7 is a root. (Even if this node were not a root, the cascading cuts would stop, since it is unmarked.) The result of the FIB-HEAP-DECREASE-KEY operation is shown in part (e), with min[H] pointing to the new minimum node.

Deleting a node

It is easy to delete a node from an n-node Fibonacci heap in O(D(n)) amortized time, as is done by the following pseudocode. We assume that there is no key value of -  [image]  currently in the Fibonacci heap.

FIB-HEAP-DELETE(H, x)

1  FIB-HEAP-DECREASE-KEY(H, x, -  )

2  FIB-HEAP-EXTRACT-MIN(H)

FIB-HEAP-DELETE is analogous to BINOMIAL-HEAP-DELETE. It makes x become the minimum node in the Fibonacci heap by giving it a uniquely small key of -  [image]. Node x is then removed from the Fibonacci heap by the FIB-HEAP-EXTRACT-MIN procedure. The amortized time of FIB-HEAP-DELETE is the sum of the O(1) amortized time of FIB-HEAP-DECREASE-KEY and the O(D(n)) amortized time of FIB-HEAP-EXTRACT-MIN.

Exercises

21.3-1

Suppose that a root x in a Fibonacci heap is marked. Explain how x came to be a marked root. Argue that it doesn't matter to the analysis that x is marked, even though it is not a root that was first linked to another node and then lost one child.

21.3-2

Justify the O(1) amortized time of FIB-HEAP-DECREASE-KEY using the aggregate method of Section 18.1.

21.4 Bounding the maximum degree

To prove that the amortized time of FIB-HEAP-EXTRACT-MIN and FIB-HEAP-DELETE is O(lg n), we must show that the upper bound D(n) on the degree of any node of an n-node Fibonacci heap is O(lg n). By Exercise 21.2-3, when all trees in the Fibonacci heap are unordered binomial trees, D(n) =   [image] lg n  [image]. The cuts that occur in FIB-HEAP-DECREASE-KEY, however, may cause trees within the Fibonacci heap to violate the unordered binomial tree properties. In this section, we shall show that because we cut a node from its parent as soon as it loses two children, D(n) is O(lg n). In particular, we shall show that D(n)   [image]    [image] log  [image]  n  [image], where   [image].

The key to the analysis is as follows. For each node x within a Fibonacci heap, define size(x) to be the number of nodes, including x itself, in the subtree rooted at x. (Note that x need not be in the root list--it can be any node at all.) We shall show that size(x) is exponential in degree[x]. Bear in mind that degree[x] is always maintained as an accurate count of the degree of x.

Lemma 21.1

Let x be any node in a Fibonacci heap, and suppose that degree[x] = k. Let y1, y2, ..., yk denote the children of x in the order in which they were linked to x, from the earliest to the latest. Then, degree [y1]   [image]  0 and degree[yi]   [image]  i - 2 for i = 2, 3, . . . , k.

Proof     Obviously, degree[y1]   [image]  0.

For i   [image]  2, we note that when yi was linked to x, all of y1, y2, . . . , yi-1 were children of x, so we must have had degree[x]   [image]  i - 1. Node yi is linked to x only if degree[x] = degree[yi], so we must have also had degree[yi]   [image]  i - 1 at that time. Since then, node yi has lost at most one child, since it would have been cut from x if it had lost two children. We conclude that degree [yi ]   [image]  i - 2.

We finally come to the part of the analysis that explains the name "Fibonacci heaps." Recall from Section 2.2 that for k = 0, 1, 2, . . . , the kth Fibonacci number is defined by the recurrence

The following lemma gives another way to express Fk.

Lemma 21.2

For all integers k   [image]  0,

Proof     The proof is by induction on k. When k = 0,

We now assume the inductive hypothesis that   [image], and we have

The following lemma and its corollary complete the analysis. It uses the inequality (proved in Exercise 2.2-8)

Fk+2      k ,

where   [image]  is the golden ratio defined in equation (2.14) as   [image].

Lemma 21.3

Let x be any node in a Fibonacci heap, and let k = degree[x]. Then, size (x)   [image]  Fk+2   [image]    [image] k, where   [image]  =   [image].

Proof     Let sk denote the minimum possible value of size(z) over all nodes z such that degree[z] = k. Trivially, s0 = 1, s1 = 2, and s2 = 3. The number sk is at most size(x). As in Lemma 21.1, let y1, y2, . . . , yk denote the children of x in the order in which they were linked to x. To compute a lower bound on size(x), we count one for x itself and one for the first child y1 (for which size(y1)   [image]  1 ) and then apply Lemma 21.1 for the other children. We thus have

We now show by induction on k that sk   [image]  Fk+2 for all nonnegative integer k. The bases, for k = 0 and k = 1, are trivial. For the inductive step, we assume that k   [image]  2 and that si   [image]  Fi + 2 for i = 0, 1, . . . , k - 1. We have

The last equality follows from Lemma 21.2.

Thus, we have shown that size(x)   [image]  sk + 2   [image]    [image] k.

Corollary 21.4

The maximum degree D(n) of any node in an n-node Fibonacci heap is O(lg n).

Proof     Let x be any node in an n-node Fibonacci heap, and let k = degree[x]. By Lemma 21.3, we have n   [image]  size(x)   [image]    [image] k. Taking base-  [image]  logarithms yields k   [image]  log  [image]  n. (In fact, because k is an integer, k   [image]    [image] log  [image]  n  [image] .) The maximum degree D(n) of any node is thus O(lg n).

Exercises

21.4-1

Professor Pinocchio claims that the height of an n-node Fibonacci heap is O(lg n). Show that the professor is mistaken by exhibiting, for any positive integer n, a sequence of Fibonacci-heap operations that creates a Fibonacci heap consisting of just one tree that is a linear chain of n nodes.

21.4-2

Suppose we generalize the cascading-cut rule to cut a node x from its parent as soon as it loses its kth child, for some integer constant k. (The rule in Section 21.3 uses k = 2.) For what values of k is D(n) = O(lg n)?

Problems

21-1 Alternative implementation of deletion

Professor Pisano has proposed the following variant of the FIB-HEAP-DELETE procedure, claiming that it runs faster when the node being deleted is not the node pointed to by min[H].

PISANO-DELETE(H, x)

1  if x = min[H]

2     then FIB-HEAP-EXTRACT-MIN(H)

3     else y    p[x]

4          if y    NIL

5             then CUT(H,x,y)

6                  CASCADING-CUT(H,y)

7          add x's child list to the root list of H

8          remove x from the root list of H

a.     The professor's claim that this procedure runs faster is based partly on the assumption that line 7 can be performed in O(1) actual time. What is wrong with this assumption?

b.     Give a good upper bound on the actual time of PISANO-DELETE when x   [image]  min[H]. Your bound should be in terms of degree[x] and the number c of calls to the CASCADING-CUT procedure.

c.     Let H' be the Fibonacci heap that results from an execution of PISANO-DELETE(H, x). Assuming that node x is not a root, bound the potential of H' in terms of degree[x], c, t(H), and m(H).

d.     Conclude that the amortized time for PISANO-DELETE is asymptotically no better than for FIB-HEAP-DELETE, even when x   [image]  min[H].

21-2 More Fibonacci-heap operations

We wish to augment a Fibonacci heap H to support two new operations without changing the amortized running time of any other Fibonacci-heap operations.

a.     Give an efficient implementation of the operation FIB-HEAP-CHANGE-KEY(H, x, k), which changes the key of node x to the value k. Analyze the amortized running time of your implementation for the cases in which k is greater than, less than, or equal to key[x].

b.     Give an efficient implementation of FIB-HEAP-PRUNE(H, r), which deletes min(r, n[H]) nodes from H. Which nodes are deleted should be arbitrary. Analyze the amortized running time of your implementation. (Hint: You may need to modify the data structure and potential function.)

Chapter notes

Fibonacci heaps were introduced by Fredman and Tarjan [75]. Their paper also describes the application of Fibonacci heaps to the problems of single-source shortest paths, all-pairs shortest pairs, weighted bipartite matching, and the minimum-spanning-tree problem.

Subsequently, Driscoll, Sarnak, Sleator, and Tarjan [58] developed "relaxed heaps" as an alternative to Fibonacci heaps. There are two varieties of relaxed heaps. One gives the same amortized time bounds as Fibonacci heaps. The other allows DECREASE-KEY to run in O(1) worst-case (not amortized) time and EXTRACT-MIN and DELETE to run in O(lg n) worst-case time. Relaxed heaps also have some advantages over Fibonacci heaps in parallel algorithms.

Go to Chapter 22     Back to Table of Contents


Intro to Algorithms: CHAPTER 22: DATA STRUCTURES FOR DISJOINT SETS

 Return to Table of Contents

CHAPTER 22: DATA STRUCTURES FOR DISJOINT SETS

Some applications involve grouping n distinct elements into a collection of disjoint sets. Two important operations are then finding which set a given element belongs to and uniting two sets. This chapter explores methods for maintaining a data structure that supports these operations.

Section 22.1 describes the operations supported by a disjoint-set data structure and presents a simple application. In Section 22.2, we look at a simple linked-list implementation for disjoint sets. A more efficient representation using rooted trees is given in Section 22.3. The running time using the tree representation is linear for all practical purposes but is theoretically superlinear. Section 22.4 defines and discusses Ackermann's function and its very slowly growing inverse, which appears in the running time of operations on the tree-based implementation, and then uses amortized analysis to prove a slightly weaker upper bound on the running time.

22.1 Disjoint-set operations

A disjoint-set data structure maintains a collection S = {S1, S2, . . . ,Sk} of disjoint dynamic sets. Each set is identified by a representative, which is some member of the set. In some applications, it doesn't matter which member is used as the representative; we only care that if we ask for the representative of a dynamic set twice without modifying the set between the requests, we get the same answer both times. In other applications, there may be a prespecified rule for choosing the representative, such as choosing the smallest member in the set (assuming, of course, that the elements can be ordered).

As in the other dynamic-set implementations we have studied, each element of a set is represented by an object. Letting x denote an object, we wish to support the following operations.

MAKE-SET(x) creates a new set whose only member (and thus representative) is pointed to by x. Since the sets are disjoint, we require that x not already be in a set.

UNION(x, y) unites the dynamic sets that contain x and y, say Sx and Sy, into a new set that is the union of these two sets. The two sets are assumed to be disjoint prior to the operation. The representative of the resulting set is some member of Sx [image] Sy, although many implementations of UNION choose the representative of either Sx or Sy, as the new representative. Since we require the sets in the collection to be disjoint, we "destroy" sets Sx and Sy, removing them from the collection S.

FIND-SET(x) returns a pointer to the representative of the (unique) set containing x.

Throughout this chapter, we shall analyze the running times of disjoint-set data structures in terms of two parameters: n, the number of MAKE-SET operations, and m, the total number of MAKE-SET, UNION, and FIND-SET operations. Since the sets are disjoint, each UNION operation reduces the number of sets by one. After n - 1 UNION operations, therefore, only one set remains. The number of UNION operations is thus at most n - 1. Note also that since the MAKE-SET operations are included in the total number of operations m, we have m [image] n.

An application of disjoint-set data structures

One of the many applications of disjoint-set data structures arises in determining the connected components of an undirected graph (see Section 5.4). Figure 22.1(a), for example, shows a graph with four connected components.

The procedure CONNECTED-COMPONENTS that follows uses the disjoint-set operations to compute the connected components of a graph. Once CONNECTED-COMPONENTS has been run as a preprocessing step, the procedure SAME-COMPONENT answers queries about whether two vertices are in the same connected component.1 (The set of vertices of a graph G is denoted by V[G], and the set of edges is denoted by E[G].)

1When the edges of the graph are "static"--not changing over time--the connected components can be computed faster by using depth-first search (Exercise 23.3-9). Sometimes, however, the edges are added "dynamically" and we need to maintain the connected components as each edge is added. In this case, the implementation given here can be more efficient than running a new depth-first search for each new edge.

CONNECTED-COMPONENTS(G)

1  for each vertex v    V[G]

2       do MAKE-SET(v)

3  for each edge (u,v)    E[G]

4       do if FIND-SET(u)    FIND-SET(v)

5             then UNION(u,v)

Edge processed                    Collection of disjoint sets

------------------------------------------------------------------------------

 initial sets   {a}        {b}    {c}  {d}  {e}      {f}  {g}  {h}    {i}  {j}

    (b,d)       {a}        {b,d}  {c}       {e}      {f}  {g}  {h}    {i}  {j}

    (e,g)       {a}        {b,d}  {c}       {e,g}    {f}       {h}    {i}  {j}

    (a,c)       {a,c}      {b,d}            {e,g}    {f}       {h}    {i}  {j}

    (h,i)       {a,c}      {b,d}            {e,g}    {f}       {h,i}       {j}

    (a,b)       {a,b,c,d}                   {e,g}    {f}       {h,i}       {j}

    (e,f)       {a,b,c,d}                   {e,f,g}            {h,i}       {j}

    (b,c)       {a,b,c,d}                   {e,f,g}            {h,i}       {j}

                                     (b)

Figure 22.1 (a) A graph with four connected components: {a, b, c, d}, {e, f, g}, {h, i}, and {j}. (b) The collection of disjoint sets after each edge is processed.

SAME-COMPONENT(u,v)

1  if FIND-SET(u) = FIND SET(v)

2     then return TRUE

3     else return FALSE

The procedure CONNECTED-COMPONENTS initially places each vertex v in its own set. Then, for each edge (u, v), it unites the sets containing u and v. By Exercise 22.1-2, after all the edges are processed, two vertices are in the same connected component if and only if the corresponding objects are in the same set. Thus, CONNECTED-COMPONENTS computes sets in such a way that the procedure SAME-COMPONENT can determine whether two vertices are in the same connected component. Figure 22.1 (b) illustrates how the disjoint sets are computed by CONNECTED-COMPONENTS.

Exercises

22.1-1

Suppose that CONNECTED-COMPONENTS is run on the undirected graph G = (V, E), where V = {a, b, c, d, e, f, g, h, i, j, k} and the edges of E are processed in the following order: (d, i), (f, k), (g, i), (b, g), (a, h), (i, j), (d, k), (b, j), (d, f), (g, j), (a, e), (i, d). List the vertices in each connected component after each iteration of lines 3-5.

22.1-2

Show that after all edges are processed by CONNECTED-COMPONENTS, two vertices are in the same connected component if and only if they are in the same set.

22.1-3

During the execution of CONNECTED-COMPONENTS on an undirected graph G = (V, E) with k connected components, how many times is FIND-SET called? How many times is UNION called? Express your answers in terms of |V|, |E|, and k.

22.2 Linked-list representation of disjoint sets

A simple way to implement a disjoint-set data structure is to represent each set by a linked list. The first object in each linked list serves as its set's representative. Each object in the linked list contains a set member, a pointer to the object containing the next set member, and a pointer back to the representative. Figure 22.2(a) shows two sets. Within each linked list, the objects may appear in any order (subject to our assumption that the first object in each list is the representative).

With this linked-list representation, both MAKE-SET and FIND-SET are easy, requiring O(1) time. To carry out MAKE-SET(x), we create a new linked list whose only object is x. For FIND-SET(x), we just return the pointer from x back to the representative.

A simple implementation of union

The simplest implementation of the UNION operation using the linked-list set representation takes significantly more time than MAKE-SET or FIND-SET. As Figure 22.2(b) shows, we perform UNION(x, y) by appending x's list onto the end of y's list. The representative of the new set is the element that was originally the representative of the set containing y. Unfortunately, we must update the pointer to the representative for each object originally on x's list, which takes time linear in the length of x's list.

In fact, it is not difficult to come up with a sequence of m operations that requires [image] (m2) time. We let n = [image] m/2 [image] + 1 and q = m - n = [image] m/2 [image] - 1 and suppose that we have objects x1, x2, . . . , xn. We then execute the sequence of m = n + q operations shown in Figure 22.3. We spend [image] (n) time performing the n MAKE-SET operations. Because the ith UNION operation updates i objects, the total number of objects updated by all the UNION operations is

Figure 22.2 (a) Linked-list representations of two sets. One contains objects b, c, e, and h, with c as the representative, and the other contains objects d, f, and g, with f as the representative. Each object on the list contains a set member, a pointer to the next object on the list, and a pointer back to the first object on the list, which is the representative. (b) The result of UNION(e, g). The representative of the resulting set is f.

Figure 22.3 A sequence of m operations that takes O(m2) time using the linked-list set representation and the simple implementation of UNION. For this example, n = m/2 +1 and q = m - n.

The total time spent is therefore [image] (n+q2), which is [image] (m2) since n = [image] (m) and q = [image] (m). Thus, on the average, each operation requires [image] (m) time. That is, the amortized time of an operation is [image] (m).

A weighted-union heuristic

The above implementation of the UNION procedure requires an average of [image] (m) time per call because we may be appending a longer list onto a shorter list; we must update the pointer to the representative for each member of the longer list. Suppose instead that each representative also includes the length of the list (which is easily maintained) and that we always append the smaller list onto the longer, with ties broken arbitrarily. With this simple weighted-union heuristic, a single UNION operation can still take [image] (m) time if both sets have [image] (m) members. As the following theorem shows, however, a sequence of m MAKE-SET, UNION, and FIND-SET operations, n of which are MAKE-SET operations, takes O(m + n 1g n) time.

Theorem 22.1

Using the linked-list representation of disjoint sets and the weighted-union heuristic, a sequence of m MAKE-SET, UNION, and FIND-SET operations, n of which are MAKE-SET operations, takes O(m + n 1g n) time.

Proof We start by computing, for each object in a set of size n, an upper bound on the number of times the object's pointer back to the representative has been updated. Consider a fixed object x. We know that each time x's representative pointer was updated, x must have started in the smaller set. The first time x's representative pointer was updated, therefore, the resulting set must have had at least 2 members. Similarly, the next time x's representative pointer was updated, the resulting set must have had at least 4 members. Continuing on, we observe that for any k [image] n, after x's representative pointer has been updated [image] 1g k [image] times, the resulting set must have at least k members. Since the largest set has at most n members, each object's representative pointer has been updated at most [image] 1g n [image] times over all the UNION operations. The total time used in updating the n objects is thus O(n 1g n).

The time for the entire sequence of m operations follows easily. Each MAKE-SET and FIND-SET operation takes O(1) time, and there are O(m) of them. The total time for the entire sequence is thus O(m + n 1g n).

Exercises

22.2-1

Write pseudocode for MAKE-SET, FIND-SET, and UNION using the linked-list representation and the weighted-union heuristic. Assume that each object x has attributes rep[x] pointing to the representative of the set containing x, last[x] pointing to the last object in the linked list containing x, and size[x] giving the size of the set containing x. Your pseudocode can assume that last[x] and size[x] are correct only if x is a representative.

22.2-2

Show the data structure that results and the answers returned by the FIND-SET operations in the following program. Use the linked-list representation with the weighted-union heuristic.

1  for i    1 to 16

2       do MAKE-SET(xi)

3  for i    1 to 15 by 2

4       do UNION(xi,xi+1)

5  for i    1 to 13 by 4

6       do UNION(xi,xi+2)

7  UNION(x1,x5)

8  UNION(x11,x13)

9  UNION (x1,x10)

10  FIND-SET(x2)

11  FIND-SET(x9)

22.2-3

Argue on the basis of Theorem 22.1 that we can obtain amortized time bounds of O(1) for MAKE-SET and FIND-SET and O(1g n) for UNION using the linked-list representation and the weighted-union heuristic.

22.2-4

Give a tight asymptotic bound on the running time of the sequence of operations in Figure 22.3 assuming the linked-list representation and the weighted-union heuristic.

22.3 Disjoint-set forests

In a faster implementation of disjoint sets, we represent sets by rooted trees, with each node containing one member and each tree representing one set. In a disjoint-set forest, illustrated in Figure 22.4(a), each member points only to its parent. The root of each tree contains the representative and is its own parent. As we shall see, although the straightforward algorithms that use this representation are no faster than ones that use the linked-list representation, by introducing two heuristics--"union by rank" and "path compression"--we can achieve the asymptotically fastest disjoint-set data structure known.

Figure 22.4 A disjoint-set forest. (a) Two trees representing the two sets of Figure 22.2. The tree on the left represents the set {b, c, e, h}, with c as the representative, and the tree on the right represents the set {d, f, g}, with f as the representative. (b) The result of UNION(e, g).

We perform the three disjoint-set operations as follows. A MAKE-SET operation simply creates a tree with just one node. We perform a FIND-SET operation by chasing parent pointers until we find the root of the tree. The nodes visited on this path toward the root constitute the find path. A UNION operation, shown in Figure 22.4(b), causes the root of one tree to point to the root of the other.

Heuristics to improve the running time

So far, we have not improved on the linked-list implementation. A sequence of n - 1 UNION operations may create a tree that is just a linear chain of n nodes. By using two heuristics, however, we can achieve a running time that is almost linear in the total number of operations m.

The first heuristic, union by rank, is similar to the weighted-union heuristic we used with the linked-list representation. The idea is to make the root of the tree with fewer nodes point to the root of the tree with more nodes. Rather than explicitly keeping track of the size of the subtree rooted at each node, we shall use an approach that eases the analysis. For each node, we maintain a rank that approximates the logarithm of the subtree size and is also an upper bound on the height of the node. In union by rank, the root with smaller rank is made to point to the root with larger rank during a UNION operation.

The second heuristic, path compression, is also quite simple and very effective. As shown in Figure 22.5, we use it during FIND-SET operations to make each node on the find path point directly to the root. Path compression does not change any ranks.

Figure 22.5 Path compression during the operation FIND-SET. Arrows and self-loops at roots are omitted. (a) A tree representing a set prior to executing FIND-SET(a). Triangles represent subtrees whose roots are the nodes shown. Each node has a pointer to its parent. (b) The same set after executing FIND-SET (a). Each node on the find path now points directly to the root.

Pseudocode for disjoint-set forests

To implement a disjoint-set forest with the union-by-rank heuristic, we must keep track of ranks. With each node x, we maintain the integer value rank[x], which is an upper bound on the height of x (the number of edges in the longest path between x and a descendant leaf). When a singleton set is created by MAKE-SET, the initial rank of the single node in the corresponding tree is 0. Each FIND-SET operation leaves all ranks unchanged. When applying UNION to two trees, we make the root of higher rank the parent of the root of lower rank. In case of a tie, we arbitrarily choose one of the roots as the parent and increment its rank.

Let us put this method into pseudocode. We designate the parent of node x by p[x]. The LINK procedure, a subroutine called by UNION, takes pointers to two roots as inputs.

MAKE-SET(x)

1  p[x]    x

2  rank[x]   0

UNION(x,y)

1  LINK(FIND-SET(x), FIND-SET(y))

LINK(x,y)

1  if rank[x] > rank[y]

2     then p[y]    x

3     else p[x]    y

4          if rank[x] = rank[y]

5             then rank[y]    rank[y] + 1

The FIND-SET procedure with path compression is quite simple.

FIND-SET(x)

1  if x    p[x]

2     then p[x]    FIND-SET(p[x])

3  return p[x]

The FIND-SET procedure is a two-pass method: it makes one pass up the find path to find the root, and it makes a second pass back down the find path to update each node so that it points directly to the root. Each call of FIND-SET(x) returns p[x] in line 3. If x is the root, then line 2 is not executed and p[x] = x is returned. This is the case in which the recursion bottoms out. Otherwise, line 2 is executed, and the recursive call with parameter p[x] returns a pointer to the root. Line 2 updates node x to point directly to the root, and this pointer is returned in line 3.

Effect of the heuristics on the running time

Separately, either union by rank or path compression improves the running time of the operations on disjoint-set forests, and the improvement is even greater when the two heuristics are used together. Alone, union by rank yields the same running time as we achieved with the weighted union heuristic for the list representation: the resulting implementation runs in time O(m lg n) (see Exercise 22. 4-3). This bound is tight (see Exercise 22.3-3). Although we shall not prove it here, if there are n MAKE-SET operations (and hence at most n - 1 UNION operations) and â FIND-SET operations, the path-compression heuristic alone gives a worst-case running time of   [image] (â log(1 + â/n)n) if â   [image]  n and   [image] (n + â lg n) if â < n.

When we use both union by rank and path compression, the worst-case running time is O(m   [image] (m, n)), where   [image] (m, n) is the very slowly growing inverse of Ackermann's function, which we define in Section 22.4. In any conceivable application of a disjoint-set data structure,   [image] (m, n)   [image]  4; thus, we can view the running time as linear in m in all practical situations. In Section 22.4, we prove the slightly weaker bound of O(m lg* n).

Exercises

22.3-1

Do Exercise 22.2-2 using a disjoint-set forest with union by rank and path compression.

22.3-2

Write a nonrecursive version of FIND-SET with path compression.

22.3-3

Give a sequence of m MAKE-SET, UNION, and FIND-SET operations, n of which are MAKE-SET operations, that takes   [image] (m lg n) time when we use union by rank only.

22.3-4

Show that any sequence of m MAKE-SET, FIND-SET, and UNION operations, where all the UNION operations appear before any of the FIND-SET operations, takes only O(m) time if both path compression and union by rank are used. What happens in the same situation if only the path-compression heuristic is used?

* 22.4 Analysis of union by rank with path compression

As noted in Section 22.3, the running time of the combined union-by-rank and path-compression heuristic is O(m   [image] (m, n)) for m disjoint-set operations on n elements. In this section, we shall examine the function   [image]  to see just how slowly it grows. Then, rather than presenting the very complex proof of the O(m   [image] (m, n)) running time, we shall offer a simpler proof of a slightly weaker upper bound on the running time: O(m lg* n).

Ackermann's function and its inverse

To understand Ackermann's function and its inverse   [image], it helps to have a notation for repeated exponentiation. For an integer i   [image]  0, the expression

stands for the function g(i), defined recursively by

Intuitively, the parameter i gives the "height of the stack of 2's" that make up the exponent. For example,

Figure 22.6 Values of A(i, j) for small values of i and j.

Recall the definition of the function 1g* in terms of the functions lg(i) defined for integer i   [image]  0:

The 1g* function is essentially the inverse of repeated exponentiation:

We are now ready to show Ackermann's function, which is defined for integers i, j   [image]  1 by

A(1,j)  =  2j                   for j    1 ,

A(i,1)  =  A(i - 1,2)           for i    2 ,

A(i,j)  =  A(i - 1,A(i,j - 1))  for i,j    2 .

Figure 22.6 shows the value of the function for small values of i and j.

Figure 22.7 shows schematically why Ackermann's function has such explosive growth. The first row, exponential in the column number j, is already rapidly growing. The second row consists of the widely spaced subset of columns   [image], . . . of the first row. Lines between adjacent rows indicate columns in the lower-numbered row that are in the subset included in the higher-numbered row. The third row consists of the even more widely spaced subset of columns   [image], . . . of the second row, which is an even sparser subset of columns of the first row. In general, the spacing between columns of row i - 1 appearing in row i increases dramatically with both the column number and the row number. Observe that   [image]  for all integers j   [image]  1. Thus, for i > 2, the function A(i, j) grows even more quickly than   [image].

Figure 22.7 The explosive growth of Ackermann's function. Lines between rows i - 1 and i indicate entries of row i - 1 appearing in row i. Due to the explosive growth, the horizontal spacing is not to scale. The horizontal spacing between entries of row i - 1 appearing in row i greatly increases with the column number and row number. If we trace the entries in row i to their original appearance in row 1, the explosive growth is even more evident.

We define the inverse of Ackermann's function by2

  (m,n) = min {i    1: A(i,   m/n  ) > lg n} .

2Although this function is not the inverse of Ackermann's function in the true mathematical sense, it captures the spirit of the inverse in its growth, which is as slow as Ackermann's function is fast. The reason we use the mysterious lg n threshold is revealed in the proof of the O(m   [image] (m, n)) running time, which is beyond the scope of this book.

If we fix a value of n, then as m increases, the function   [image] (m, n) is monotonically decreasing. To see this property, note that   [image] m/n  [image]  is monotonically increasing as m increases; therefore, since n is fixed, the smallest value of i needed to bring A(i,  [image] m/n  [image] ) above lg n is monotonically decreasing. This property corresponds to our intuition about disjoint-set forests with path compression: for a given number of distinct elements n, as the number of operations m increases, we would expect the average find-path length to decrease due to path compression. If we perform m operations in time O(m   [image] (m, n)), then the average time per operation is O(  [image] (m, n)), which is monotonically decreasing as m increases.

To back up our earlier claim that   [image] (m, n)   [image]  4 for all practical purposes, we first note that the quantity   [image] m/n  [image]  is at least 1, since m   [image]  n. Since Ackermann's function is strictly increasing with each argument,   [image] m/n  [image]    [image]  1 implies A(i,   [image] m/n  [image] )   [image]  A(i, 1) for all i   [image]  1. In particular, A(4,   [image] m/n  [image] )   [image]  A(4, 1). But we also have that

which is far greater than the estimated number of atoms in the observable universe (roughly 1080). It is only for impractically large values of n that A(4, 1)   [image]  1g n, and thus   [image] (m, n)   [image]  4 for all practical purposes. Note that the O(m 1g* n) bound is only slightly weaker than the O(m   [image] (m, n)) bound; 1g*65536 = 4 and 1g* 265536 = 5, so 1g* n   [image]  5 for all practical purposes.

Properties of ranks

In the remainder of this section, we prove an O(m 1g* n) bound on the running time of the disjoint-set operations with union by rank and path compression. In order to prove this bound, we first prove some simple properties of ranks.

Lemma 22.2

For all nodes x, we have rank[x]   [image]  rank[p[x]], with strict inequality if x   [image]  p[x]. The value of rank[x] is initially 0 and increases through time until x   [image]  p[x]; from then on, rank[x] does not change. The value of rank[p[x]] is a monotonically increasing function of time.

Proof     The proof is a straightforward induction on the number of operations, using the implementations of MAKE-SET, UNION, and FIND-SET that appear in Section 22.3. We leave it as Exercise 22.4-1.

We define size(x) to be the number of nodes in the tree rooted at node x, including node x itself.

Lemma 22.3

For all tree roots x, size(x)   [image]  2rank[x].

Proof     The proof is by induction on the number of LINK operations. Note that FIND-SET operations change neither the rank of a tree root nor the size of its tree.

Basis: The lemma is true before the first LINK, since ranks are initially 0 and each tree contains at least one node.

Inductive step: Assume that the lemma holds before performing the operation LINK(x, y). Let rank denote the rank just before the LINK, and let rank' denote the rank just after the LINK. Define size and size' similarly.

If rank[x]   [image]  rank[y], assume without loss of generality that rank[x] < rank[y]. Node y is the root of the tree formed by the LINK operation, and

size'(y)  =  size(x) + size(y)

    2rank[x] + 2rank[y]

    2rank[y]

=  2rank'[y].

No ranks or sizes change for any nodes other than y.

If rank[x] = rank[y], node y is again the root of the new tree, and

size'(y)  =  size(x) + size(y)

    2rank[x] + 2rank[y]

=  2rank[y]+ 1

=  2rank'[y] .

Lemma 22.4

For any integer r   [image]  0, there are at most n/2r nodes of rank r.

Proof     Fix a particular value of r. Suppose that when we assign a rank r to a node x (in line 2 of MAKE-SET or in line 5 of LINK), we attach a label x to each node in the tree rooted at x. By Lemma 22.3, at least 2r nodes are labeled each time. Suppose that the root of the tree containing node x changes. Lemma 22.2 assures us that the rank of the new root (or, in fact, of any proper ancestor of x) is at least r + 1. Since we assign labels only when a root is assigned a rank r, no node in this new tree will ever again be labeled. Thus, each node is labeled at most once, when its root is first assigned rank r. Since there are n nodes, there are at most n labeled nodes, with at least 2r labels assigned for each node of rank r. If there were more than n/2r nodes of rank r, then more than 2r. (n/2r) = n nodes would be labeled by a node of rank r, which is a contradiction. Therefore, at most n/2r nodes are ever assigned rank r.

Corollary 22.5

Every node has rank at most   [image] 1g n  [image] .

Proof     If we let r > 1g n, then there are at most n/2r < 1 nodes of rank r. Since ranks are natural numbers, the corollary follows.

Proving the time bound

We shall use the aggregate method of amortized analysis (see Section 18.1) to prove the O(m 1g* n) time bound. In performing the amortized analysis, it is convenient to assume that we invoke the LINK operation rather than the UNION operation. That is, since the parameters of the LINK procedure are pointers to two roots, we assume that the appropriate FIND-SET operations are performed if necessary. The following lemma shows that even if we count the extra FIND-SET operations, the asymptotic running time remains unchanged.

Lemma 22.6

Suppose we convert a sequence S' of m' MAKE-SET, UNION, and FIND-SET operations into a sequence S of m MAKE-SET, LINK, and FIND-SET operations by turning each UNION into two FIND-SET operations followed by a LINK. Then, if sequence S runs in O(m 1g* n) time, sequence S' runs in O(m' 1g* n) time.

Proof     Since each UNION operation in sequence S' is converted into three operations in S, we have m'   [image]  m   [image]  3m'. Since m = O(m'), an O(m 1g* n) time bound for the converted sequence S implies an O(m' 1g* n) time bound for the original sequence S'.

In the remainder of this section, we shall assume that the initial sequence of m' MAKE-SET, UNION, and FIND-SET operations has been converted to a sequence of m MAKE-SET, LINK, and FIND-SET operations. We now prove an O(m 1g* n) time bound for the converted sequence and appeal to Lemma 22.6 to prove the O(m' 1g* n) running time of the original sequence of m' operations.

Theorem 22.7

A sequence of m MAKE-SET, LINK, and FIND-SET operations, n of which are MAKE-SET operations, can be performed on a disjoint-set forest with union by rank and path compression in worst-case time O(m 1g* n).

Proof     We assess charges corresponding to the actual cost of each set operation and compute the total number of charges assessed once the entire sequence of set operations has been performed. This total then gives us the actual cost of all the set operations.

The charges assessed to the MAKE-SET and LINK operations are simple: one charge per operation. Since these operations each take O(1) actual time, the charges assessed equal the actual costs of the operations.

Before discussing charges assessed to the FIND-SET operations, we partition node ranks into blocks by putting rank r into block 1g* r for r = 0, 1, . . . ,   [image] 1g n  [image] . (Recall that   [image] 1g n  [image]  is the maximum rank.) The highest-numbered block is therefore block 1g*(1g n) = 1g* n - 1. For notational convenience, we define for integers j   [image]  -1,

Then, for j = 0, 1, . . . , lg* n - 1, the jth block consists of the set of ranks

{B(j - 1) + 1, B(j - 1) + 2, . . . , B(j)} .

We use two types of charges for a FIND-SET operation: block charges and path charges. Suppose that the FIND-SET starts at node x0 and that the find path consists of nodes x0, x1, . . . , xl, where for i = 1, 2, . . . , l, node xi is p[xi-1] and xl (a root) is p[xl]. For j = 0, 1, . . . , lg* n - 1, we assess one block charge to the last node with rank in block j on the path. (Note that Lemma 22.2 implies that on any find path, the nodes with ranks in a given block are consecutive.) We also assess one block charge to the child of the root, that is, to xl-1. Because ranks strictly increase along any find path, an equivalent formulation assesses one block charge to each node xi such that p[xi] = xl (xi is the root or its child) or 1g* rank[xi] < 1g* rank[xi+1](the block of xi's rank differs from that of its parent). At each node on the find path for which we do not assess a block charge, we assess one path charge.

Once a node other than the root or its child is assessed block charges, it will never again be assessed path charges. To see why, observe that each time path compression occurs, the rank of a node xi for which p[xi]   [image]  xl remains the same, but the new parent of xi has a rank strictly greater than that of xi's old parent. The difference between the ranks of xi and its parent is a monotonically increasing function of time. Thus, the difference between 1g* rank[p[xi]] and 1g* rank[xi] is also a monotonically increasing function of time. Once xi and its parent have ranks in different blocks, they will always have ranks in different blocks, and so xi will never again be assessed a path charge.

Since we have charged once for each node visited in each FIND-SET operation, the total number of charges assessed is the total number of nodes visited in all the FIND-SET operations; this total represents the actual cost of all the FIND-SET operations. We wish to show that this total is O(m 1g* n).

The number of block charges is easy to bound. There is at most one block charge assessed for each block number on the given find path, plus one block charge for the child of the root. Since block numbers range from 0 to 1g* n - 1, there are at most 1g* n + 1 block charges assessed for each FIND-SET operation. Thus, there are at most m(1g* n + 1) block charges assessed over all FIND-SET operations.

Bounding the path charges is a little trickier. We start by observing that if a node xi is assessed a path charge, then p[xi]   [image]  xl before path compression, so that xi will be assigned a new parent during path compression. Moreover, as we have observed, xi's new parent has a higher rank than its old parent. Suppose that node xi's rank is in block j. How many times can xi be assigned a new parent, and thus assessed a path charge, before xiis assigned a parent whose rank is in a different block (after which xi will never again be assessed a path charge)? This number of times is maximized if xi has the lowest rank in its block, namely B(j - 1) + 1, and its parents' ranks successively take on the values B(j - 1) + 2, B(j - 1) + 3, . . . , B(j). Since there are B(j) - B(j - 1) - 1 such ranks, we conclude that a vertex can be assessed at most B(j) - B(j - 1) - 1 path charges while its rank is in block j.

Our next step in bounding the path charges is to bound the number of nodes that have ranks in block j for integers j   [image]  0. (Recall that by Lemma 22.2, the rank of a node is fixed once it becomes a child of another node.) Let the number of nodes whose ranks are in block j be denoted by N(j). Then, by Lemma 22.4,

For j = 0, this sum evaluates to

N(0)  =  n/20 + n/21

=  3n/2

=  3n/2B(0).

For j   [image]  1, we have

Thus, N(j)   [image]  3n/2B(j) for all integers j   [image]  0.

We finish bounding the path charges by summing over all blocks the product of the maximum number of nodes with ranks in the block and the maximum number of path charges per node of that block. Denoting by P(n) the overall number of path charges, we have

Thus, the total number of charges incurred by FIND-SET operations is O(m(1g* n + 1) + n 1g* n), which is O(m 1g* n) since m   [image]  n. Since there are O(n) MAKE-SET and LINK operations, with one charge each, the total time is O(m 1g* n).

Corollary 22.8

A sequence of m MAKE-SET, UNION, and FIND-SET operations, n of which are MAKE-SET operations, can be performed on a disjoint-set forest with union by rank and path compression in worst-case time O(m 1g* n).

Proof     Immediate from Theorem 22.7 and Lemma 22.6.

Exercises

22.4-1

Prove Lemma 22.2.

22.4-2

For each node x, how many bits are necessary to store size(x)? How about rank[x]?

22.4-3

Using Lemma 22.2 and Corollary 22.5, give a simple proof that operations on a disjoint-set forest with union by rank but without path compression run in O(m 1g n) time.

22.4-4

Suppose we modify the rule about assessing charges so that we assess one block charge to the last node on the find path whose rank is in block j for j = 0, 1, . . . , 1g* n - 1. Otherwise, we assess one path charge to the node. Thus, if a node is a child of the root and is not the last node of a block, it is assessed a path charge, not a block charge. Show that   [image] (m) path charges could be assessed a given node while its rank is in a given block j.

Problems

22-1     Off-line minimum

The off-line minimum problem asks us to maintain a dynamic set T of elements from the domain {1, 2, . . . ,n} under the operations INSERT and EXTRACT-MIN. We are given a sequence S of n INSERT and m EXTRACT-MIN calls, where each key in {1, 2, . . . , n} is inserted exactly once. We wish to determine which key is returned by each EXTRACT-MIN call. Specifically, we wish to fill in an array extracted[1 . . m], where for i = 1, 2, . . . , m, extracted[i] is the key returned by the ith EXTRACT-MIN call. The problem is "off-line" in the sense that we are allowed to process the entire sequence S before determining any of the returned keys.

a.     In the following instance of the off-line minimum problem, each INSERT is represented by a number and each EXTRACT-MIN is represented by the letter E:

4, 8, E, 3, E, 9, 2, 6, E, E, E, 1, 7, E, 5.

Fill in the correct values in the extracted array.

To develop an algorithm for this problem, we break the sequence S into homogeneous subsequences. That is, we represent S by

I1,E,I2,E,I3,...,Im,E,Im+1 ,

where each E represents a single EXTRACT-MIN call and each Ij represents a (possibly empty) sequence of INSERT calls. For each subsequence Ij, we initially place the keys inserted by these operations into a set Kj, which is empty if Ij is empty. We then do the following.

OFF-LINE-MINIMUM(m,n)

1  for i    1 to n

2       do determine j such that i    Kj

3          if j    m + 1

4             then extracted[j]    i

5                 let l be the smallest value greater than j

for which set Kl exists

6                 Kl    Kj    Kl, destroying Kj

7  return extracted

b.     Argue that the array extracted returned by OFF-LINE-MINIMUM is correct.

c.     Describe how to use a disjoint-set data structure to implement OFF-LINE-MINIMUM efficiently. Give a tight bound on the worst-case running time of your implementation.

22-2     Depth determination

In the depth-determination problem, we maintain a forest F = {Ti} of rooted trees under three operations:

MAKE-TREE(v)     creates a tree whose only node is v.

FIND-DEPTH(v)     returns the depth of node v within its tree.

GRAFT(r, v)     makes node r, which is assumed to be the root of a tree, become the child of node v, which is assumed to be in a different tree than r but may or may not itself be a root.

a.     Suppose that we use a tree representation similar to a disjoint-set forest: p[v] is the parent of node v, except that p[v] = v if v is a root. If we implement GRAFT(r, v) by setting p[r]   [image]  v and FIND-DEPTH(v) by following the find path up to the root, returning a count of all nodes other than v encountered, show that the worst-case running time of a sequence of m MAKE-TREE, FIND-DEPTH, and GRAFT operations is   [image] (m2).

By using the union-by-rank and path-compression heuristics, we can reduce the worst-case running time. We use the disjoint-set forest S = {Si}, where each set Si (which is itself a tree) corresponds to a tree Ti in the forest F. The tree structure within a set Si, however, does not necessarily correspond to that of Ti. In fact, the implementation of Si does not record the exact parent-child relationships but nevertheless allows us to determine any node's depth in Ti.

The key idea is to maintain in each node v a "pseudodistance" d[v],which is defined so that the sum of the pseudodistances along the path from v to the root of its set Si equals the depth of v in Ti. That is, if the path from v to its root in Si is v0, v1, . . . ,vk, where v0 = v and vk is Si's root, then the depth of v in

b.     Give an implementation of MAKE-TREE.

c.     Show how to modify FIND-SET to implement FIND-DEPTH. Your implementation should perform path compression, and its running time should be linear in the length of the find path. Make sure that your implementation updates pseudodistances correctly.

d.     Show how to modify the UNION and LINK procedures to implement GRAFT (r, v), which combines the sets containing r and v. Make sure that your implementation updates pseudodistances correctly. Note that the root of a set Si is not necessarily the root of the corresponding tree Ti.

e.     Give a tight bound on the worst-case running time of a sequence of m MAKE-TREE, FIND-DEPTH, and GRAFT operations, n of which are MAKE-TREE operations.

22-3     Tarjan's off-line least-common-ancestors algorithm

The least common ancestor of two nodes u and v in a rooted tree T is the node w that is an ancestor of both u and v and that has the greatest depth in T. In the off-line least-common-ancestors problem, we are given a rooted tree T and an arbitrary set P = {{u, v}} of unordered pairs of nodes in T, and we wish to determine the least common ancestor of each pair in P.

To solve the off-line least-common-ancestors problem, the following procedure performs a tree walk of T with the initial call LCA(root[T]). Each node is assumed to be colored WHITE prior to the walk.

LCA(u)

1  MAKE-SET(u)

2  ancestor[FIND-SET(u)]    u

3  for each child v of u in T

4       do LCA(v)

5          UNION(u,v)

6          ancestor[FIND-SET(u)]    u

7  color[u]    BLACK

8  for each node v such that {u,v}    P

9       do if color[v] = BLACK

10             then print "The least common ancestor of"

u "and" v "is" ancestor [FIND-SET(v)]

a.     Argue that line 10 is executed exactly once for each pair {u, v}   [image]  P.

b.     Argue that at the time of the call LCA (u), the number of sets in the disjoint-set data structure is equal to the depth of u in T.

c.     Prove that LCA correctly prints the least common ancestor of u and v for each pair {u ,v}   [image]  P.

d.     Analyze the running time of LCA, assuming that we use the implementation of the disjoint-set data structure in Section 22.3.

Chapter notes

Many of the important results for disjoint-set data struckltures are due at least in part to R. E. Tarjan. The upper bound of O(m   [image] (m,n)) was first given by Tarjan[186, 188]. The O(m 1g* n) upper bound was proven earlier by Hopcroft and Ullman[4, 103]. Tarjan and van Leeuwen [190] discuss variants on the path-compression heuristic, including "one-pass methods," which sometimes offer better constant factors in their performance than do two-pass methods. Gabow and Tarjan [76] show that in certain applications, the disjoint-set operations can be made to run in O(m) time.

Tarjan [187] showed that a lower bound of   [image] (m   [image]  (m, n)) time is required for operations on any disjoint-set data structure satisfying certain technical conditions. This lower bound was later generalized by Fredman and Saks [74], who showed that in the worst case,   [image] (m   [image]  (m, n)) (1g n)-bit words of memory must be accessed.

Go to Part VI     Back to Table of Contents


Intro to Algorithms: CHAPTER 23: ELEMENTARY GRAPH ALGORITHMS

 Return to Table of Contents

CHAPTER 23: ELEMENTARY GRAPH ALGORITHMS

This chapter presents methods for representing a graph and for searching a graph. Searching a graph means systematically following the edges of the graph so as to visit the vertices of the graph. A graph-searching algorithm can discover much about the structure of a graph. Many algorithms begin by searching their input graph to obtain this structural information. Other graph algorithms are organized as simple elaborations of basic graph-searching algorithms. Techniques for searching a graph are at the heart of the field of graph algorithms.

Section 23.1 discusses the two most common computational representations of graphs: as adjacency lists and as adjacency matrices. Section 23.2 presents a simple graph-searching algorithm called breadth-first search and shows how to create a breadth-first tree. Section 23.3 presents depth-first search and proves some standard results about the order in which depth-first search visits vertices. Section 23.4 provides our first real application of depth-first search: topologically sorting a directed acyclic graph. A second application of depth-first search, finding the strongly connected components of a directed graph, is given in Section 23.5.

23.1 Representations of graphs

There are two standard ways to represent a graph G = (V, E): as a collection of adjacency lists or as an adjacency matrix. The adjacency-list representation is usually preferred, because it provides a compact way to represent sparse graphs--those for which |E| is much less than |V|2. Most of the graph algorithms presented in this book assume that an input graph is represented in adjacency-list form. An adjacency-matrix representation may be preferred, however, when the graph is dense--|E| is close to |V|2 -- or when we need to be able to tell quickly if there is an edge connecting two given vertices. For example, two of the all-pairs shortest-paths algorithms presented in Chapter 26 assume that their input graphs are represented by adjacency matrices.

The adjacency-list representation of a graph G = (V, E) consists of an array Adj of |V| lists, one for each vertex in V. For each u [image] V, the adjacency list Adj[u] contains (pointers to) all the vertices v such that there is an edge (u,v) [image] E. That is, Adj[u] consists of all the vertices adjacent to u in G. The vertices in each adjacency list are typically stored in an arbitrary order. Figure 23.1(b) is an adjacency-list representation of the undirected graph in Figure 23.1(a). Similarly, Figure 23.2(b) is an adjacency-list representation of the directed graph in Figure 23.2(a).

Figure 23.1 Two representations of an undirected graph. (a) An undirected graph G having five vertices and seven edges. (b) An adjacency-list representation of G. (c) The adjacency-matrix representation of G.

Figure 23.2 Two representations of a directed graph. (a) A directed graph G having six vertices and eight edges. (b) An adjacency-list representation of G. (c) The adjacency-matrix representation of G.

If G is a directed graph, the sum of the lengths of all the adjacency lists is |E|, since an edge of the form (u,v) is represented by having v appear in Adj[u]. If G is an undirected graph, the sum of the lengths of all the adjacency lists is 2|E|, since if (u,v) is an undirected edge, then u appears in v's adjacency list and vice versa. Whether a graph is directed or not, the adjacency-list representation has the desirable property that the amount of memory it requires is O(max(V, E)) = O(V + E).

Adjacency lists can readily be adapted to represent weighted graphs, that is, graphs for which each edge has an associated weight, typically given by a weight function w : E [image] R. For example, let G = (V, E) be a weighted graph with weight function w. The weight w(u,v) of the edge (u,v) [image] E is simply stored with vertex v in u's adjacency list. The adjacency-list representation is quite robust in that it can be modified to support many other graph variants.

A potential disadvantage of the adjacency-list representation is that there is no quicker way to determine if a given edge (u,v) is present in the graph than to search for v in the adjacency list Adj[u]. This disadvantage can be remedied by an adjacency-matrix re presentation of the graph, at the cost of using asymptotically more memory.

For the adjacency-matrix representation of a graph G = (V, E), we assume that the vertices are numbered 1, 2, . . . , |V| in some arbitrary manner. The adjacency-matrix representation of a graph G then consists of a |V| [image] |V| matrix A = (aij) such that

Figures 23.1(c) and 23.2(c) are the adjacency matrices of the undirected and directed graphs in Figures 23.1(a) and 23.2(a), respectively. The adjacency matrix of a graph requires [image] (V2) memory, independent of the number of edges in the graph.

Observe the symmetry along the main diagonal of the adjacency matrix in Figure 23.1(c). We define the the transpose of a matrix A = (aij) to be the matrix [image] given by [image]. Since in an undirected graph, (u,v) and (v,u) represent the same edge, the adjacency matrix A of an undirected graph is its own transpose: A = AT. In some applications, it pays to store only the entries on and above the diagonal of the adjacency matrix, thereby cutting the memory needed to store the graph almost in half.

Like the adjacency-list representation of a graph, the adjacency-matrix representation can be used for weighted graphs. For example, if G = (V, E) is a weighted graph with edge-weight function w, the weight w(u, v) of the edge (u,v) [image] E is simply stored as the entry in row u and column v of the adjacency matrix. If an edge does not exist, a NIL value can be stored as its corresponding matrix entry, though for many problems it is convenient to use a value such as 0 or [image].

Although the adjacency-list representation is asymptotically at least as efficient as the adjacency-matrix representation, the simplicity of an adjacency matrix may make it preferable when graphs are reasonably small. Moreover, if the graph is unweighted, there is an additional advantage in storage for the adjacency-matrix representation. Rather than using one word of computer memory for each matrix entry, the adjacency matrix uses only one bit per entry.

Exercises

23.1-1

Given an adjacency-list representation of a directed graph, how long does it take to compute the out-degree of every vertex? How long does it take to compute the in-degrees?

23.1-2

Give an adjacency-list representation for a complete binary tree on 7 vertices. Give an equivalent adjacency-matrix representation. Assume that vertices are numbered from 1 to 7 as in a binary heap.

23.1-3

The transpose of a directed graph G = (V, E) is the graph GT = (V, ET), where ET = {(v,u) [image] V [image] V : (u,v) [image] E}. Thus, GT is G with all its edges reversed. Describe efficient algorithms for computing GT from G, for both the adjacency-list and adjacency-matrix representations of G. Analyze the running times of your algorithms.

23.1-4

Given an adjacency-list representation of a multigraph G = (V, E), describe an O(V+E)-time algorithm to compute the adjacency-list representation of the "equivalent" undirected graph G' = (V, E'), where E' consists of the edges in E with all multiple edges between two vertices replaced by a single edge and with all self-loops removed.

23.1-5

The square of a directed graph G = (V, E) is the graph G2 = (V, E2) such that (u,w) [image] E2 if and only if for some v [image] V, both (u,v) [image] E and (v,w) [image] E. That is, G2 contains an edge between u and w whenever G contains a path with exactly two edges between u and w. Describe efficient algorithms for computing G2 from G for both the adjacency-list and adjacency-matrix representations of G. Analyze the running times of your algorithms.

23.1-6

When an adjacency-matrix representation is used, most graph algorithms require time [image] (V2), but there are some exceptions. Show that determining whether a directed graph contains a sink--a vertex with in-degree |V| - 1 and out-degree 0--can be determined in time O(V), even if an adjacency-matrix representation is used.

23.1-7

The incidence matrix of a directed graph G = (V, E) is a |V| [image] |E| matrix B = (bij) such that

Describe what the entries of the matrix product BBT represent, where BT is the transpose of B.

23.2 Breadth-first search

Breadth-first search is one of the simplest algorithms for searching a graph and the archetype for many important graph algorithms. Dijkstra's single-source shortest-paths algorithm (Chapter 25) and Prim's minimum-spanning-tree algorithm (Section 24.2) use ideas similar to those in breadth-first search.

Given a graph G = (V, E) and a distinguished source vertex s, breadth-first search systematically explores the edges of G to "discover" every vertex that is reachable from s. It computes the distance (fewest number of edges) from s to all such reachable vertices. It also produces a "breadth-first tree" with root s that contains all such reachable vertices. For any vertex v reachable from s, the path in the breadth-first tree from s to v corresponds to a "shortest path" from s to v in G, that is, a path containing the fewest number of edges. The algorithm works on both directed and undirected graphs.

Breadth-first search is so named because it expands the frontier between discovered and undiscovered vertices uniformly across the breadth of the frontier. That is, the algorithm discovers all vertices at distance k from s before discovering any vertices at distance k + 1.

To keep track of progress, breadth-first search colors each vertex white, gray, or black. All vertices start out white and may later become gray and then black. A vertex is discovered the first time it is encountered during the search, at which time it becomes nonwhite. Gray and black vertices, therefore, have been discovered, but breadth-first search distinguishes between them to ensure that the search proceeds in a breadth-first manner. If (u, v) [image] E and vertex u is black, then vertex v is either gray or black; that is, all vertices adjacent to black vertices have been discovered. Gray vertices may have some adjacent white vertices; they represent the frontier between discovered and undiscovered vertices.

Breadth-first search constructs a breadth-first tree, initially containing only its root, which is the source vertex s. Whenever a white vertex v is discovered in the course of scanning the adjacency list of an already discovered vertex u, the vertex v and the edge (u,v) are added to the tree. We say that u is the predecessor or parent of v in the breadth-first tree. Since a vertex is discovered at most once, it has at most one parent. Ancestor and descendant relationships in the breadth-first tree are defined relative to the root s as usual: if u is on a path in the tree from the root s to vertex v, then u is an ancestor of v and v is a descendant of u.

The breadth-first-search procedure BFS below assumes that the input graph G = (V, E) is represented using adjacency lists. It maintains several additional data structures with each vertex in the graph. The color of each vertex u [image] V is stored in the variable color[u], and the predecessor of u is stored in the variable [image] [u]. If u has no predecessor (for example, if u = s or u has not been discovered), then [image] [u] = NIL. The distance from the source s to vertex u computed by the algorithm is stored in d[u]. The algorithm also uses a first-in, first-out queue Q (see Section 11.1) to manage the set of gray vertices.

BFS(G,s)

1  for each vertex u    V[G] - {s}

2       do color[u]    WHITE

3          d[u]

4            [u]    NIL

5  color[s]    GRAY

6  d[s]    0

7    [s]    NIL

8  Q    {s}

9  while Q

10      do u    head[Q]

11         for each v    Adj[u]

12             do if color[v] = WHITE

13                   then color[v]    GRAY

14                        d[v]    d[u] + 1

15                          [v]    u

16                        ENQUEUE(Q,v)

17         DEQUEUE(Q)

18         color[u]    BLACK

Figure 23.3 illustrates the progress of BFS on a sample graph.

The procedure BFS works as follows. Lines 1-4 paint every vertex white, set d [u] to be infinity for every vertex u, and set the parent of every vertex to be NIL. Line 5 paints the source vertex s gray, since it is considered to be discovered when the procedure begins. Line 6 initializes d[s] to 0, and line 7 sets the predecessor of the source to be NIL. Line 8 initializes Q to the queue containing just the vertex s; thereafter, Q always contains the set of gray vertices.

The main loop of the program is contained in lines 9-18. The loop iterates as long as there remain gray vertices, which are discovered vertices that have not yet had their adjacency lists fully examined. Line 10 determines the gray vertex u at the head of the queue Q. The for loop of lines 11-16 considers each vertex v in the adjacency list of u. If v is white, then it has not yet been discovered, and the algorithm discovers it by executing lines 13-16. It is first grayed, and its distance d[v] is set to d[u] + 1. Then, u is recorded as its parent. Finally, it is placed at the tail of the queue Q. When all the vertices on u's adjacency list have been examined, u is removed from Q and blackened in lines 17-18.

Figure 23.3 The operation of BFS on an undirected graph. Tree edges are shown shaded as they are produced by BFS. Within each vertex u is shown d[u]. The queue Q is shown at the beginning of each iteration of the while loop of lines 9-18. Vertex distances are shown next to vertices in the queue.

Analysis

Before proving all the various properties of breadth-first search, we take on the somewhat easier job of analyzing its running time on an input graph G = (V,E). After initialization, no vertex is ever whitened, and thus the test in line 12 ensures that each vertex is enqueued at most once, and hence dequeued at most once. The operations of enqueuing and dequeuing take O(1) time, so the total time devoted to queue operations is O(V). Because the adjacency list of each vertex is scanned only when the vertex is dequeued, the adjacency list of each vertex is scanned at most once. Since the sum of the lengths of all the adjacency lists is [image] (E), at most O(E) time is spent in total scanning adjacency lists. The overhead for initialization is O(V), and thus the total running time of BFS is O(V + E). Thus, breadth-first search runs in time linear in the size of the adjacency- list representation of G.

Shortest paths

At the beginning of this section, we claimed that breadth-first search finds the distance to each reachable vertex in a graph G = (V,E) from a given source vertex s [image] V. Define the shortest-path distance [image] (s,v) from s to v as the minimum number of edges in any path from vertex s to vertex v, or else [image] if there is no path from s to v. A path of length [image] (s,v) from s to v is said to be a shortest path1 from s to v. Before showing that breadth-first search actually computes shortest-path distances, we investigate an important property of shortest-path distances.

1 In Chapters 25 and 26, we shall generalize our study of shortest paths to weighted graphs, in which every edge has a real-valued weight and the weight of a path is the sum of the weights of its constituent edges. The graphs considered in the present chapter are unweighted.

Lemma 23.1

Let G = (V,E) be a directed or undirected graph, and let s [image] V be an arbitrary vertex. Then, for any edge (u, v) [image] E,

  (s,v)      (s,u) + 1 .

Proof If u is reachable from s, then so is v. In this case, the shortest path from s to v cannot be longer than the shortest path from s to v followed by the edge (u, v), and thus the inequality holds. If u is not reachable from s, then [image] (s,u) = [image], and the inequality holds.

We want to show that BFS properly computes d[v] = [image] (s, v) for each vertex v [image] V. We first show that d [v] bounds [image] (s, v) from above.

Lemma 23.2

Let G = (V,E) be a directed or undirected graph, and suppose that BFS is run on G from a given source vertex s [image] V. Then upon termination, for each vertex v [image] V, the value d[v] computed by BFS satisfies d[v] [image] [image] (s, v).

Proof We use induction on the number of times a vertex is placed in the queue Q. Our inductive hypothesis is that d[v] [image] [image] (s,v) for all v [image] V.

The basis of the induction is the situation immediately after s is placed in Q in line 8 of BFS. The inductive hypothesis holds here, because d[s] = 0 = [image] (s, s) and d[v] = [image] [image] [image] (s, v) for all v [image] V - {s}.

For the inductive step, consider a white vertex v that is discovered during the search from a vertex u. The inductive hypothesis implies that d[u] [image] [image] (s, u). From the assignment performed by line 14 and from Lemma 23.1, we obtain

d[v]  =  d[u] + 1

      (s,u) + 1

      (s,v) .

Vertex v is then inserted into the queue Q, and it is never inserted again because it is also grayed and the then clause of lines 13-16 is executed only for white vertices. Thus, the value of d[v] never changes again, and the inductive hypothesis is maintained.

To prove that d[v] = [image] (s, v), we must first show more precisely how the queue Q operates during the course of BFS. The next lemma shows that at all times, there are at most two distinct d values in the queue.

Lemma 23.3

Suppose that during the execution of BFS on a graph G = (V, E), the queue Q contains the vertices [image] v1, v2, . . . , vr [image], where v1 is the head of Q and vr is the tail. Then, d[vr] [image] d[v1] + 1 and d[vi] [image] d[vi + 1] for i = 1, 2, . . . , r - 1.

Proof The proof is by induction on the number of queue operations. Initially, when the queue contains only s, the lemma certainly holds.

For the inductive step, we must prove the lemma holds after both dequeuing and enqueuing a vertex. If the head v1 of the queue is dequeued, the new head is v2. (If the queue becomes empty, then the lemma holds vacuously.) But then we have d[vr] [image] d[v1] + 1 [image] d[v2] + 1, and the remaining inequalities are unaffected. Thus, the lemma follows with v2 as the head. Enqueuing a vertex requires closer examination of the code. In line 16 of BFS, when the vertex v is enqueued, thus becoming vr+1, the head v1 of Q is in fact the vertex u whose adjacency list is currently being scanned. Thus, d[vr+1] = d[v] = d[u] + 1 = d[v1] + 1. We also have d[vr] [image] d[v1] + 1 = d[u] + 1 = d[v] = d[vr + 1], and the remaining inequalities are unaffected. Thus, the lemma follows when v is enqueued.

We can now prove that breadth-first search correctly finds shortest-path distances.

Theorem 23.4

Let G = (V, E) be a directed or undirected graph, and suppose that BFS is run on G from a given source vertex s [image] V. Then, during its execution, BFS discovers every vertex v [image] V that is reachable from the source s, and upon termination, d[v] = [image] (s, v) for all v [image] V. Moreover, for any vertex v [image] s that is reachable from s, one of the shortest paths from s to v is the shortest path from s to [image] [v] followed by the edge ( [image] [v], v).

Proof We start with the case in which v is unreachable from s. Since Lemma 23.2 gives d[v] [image] [image] (s, v) = [image], vertex v cannot have d[v] set to a finite value in line 14. By induction, there cannot be a first vertex whose d value is set to [image] by line 14. Line 14 is therefore only executed only for vertices with finite d values. Thus, if v is unreachable, it is never discovered.

The main part of the proof is for vertices reachable from s. Let Vk denote the set of vertices at distance k from s; that is, Vk = {v [image] V : (s, v) = k}. The proof proceeds by induction on k. As an inductive hypothesis, we assume that for each vertex v [image] Vk, there is exactly one point during the execution of BFS at which

v is grayed,

d[v] is set to k,

if v [image] s, then [image] [v] is set to u for some u [image] Vk - 1, and

v is inserted into the queue Q.

As we have noted before, there is certainly at most one such point.

The basis is for k = 0. We have V0 = {s}, since the source s is the only vertex at distance 0 from s. During the initialization, s is grayed, d[s] is set to 0, and s is placed into Q, so the inductive hypothesis holds.

For the inductive step, we start by noting that the queue Q is never empty until the algorithm terminates and that, once a vertex u is inserted into the queue, neither d[u] nor [image] [u] ever changes. By Lemma 23.3, therefore, if vertices are inserted into the queue over the course of the algorithm in the order v1, v2, . . . , vr, then the sequence of distances is monotonically increasing: d[vi] [image] d[vi + 1] for i = 1, 2, . . . , r - 1.

Now let us consider an arbitrary vertex v [image] Vk, where k [image] 1. The monotonicity property, combined with d[v] [image] k (by Lemma 23.2) and the inductive hypothesis, implies that v must be discovered after all vertices in Vk - 1 are enqueued, if it is discovered at all.

Since [image] (s, v) = k, there is a path of k edges from s to v, and thus there exists a vertex u [image] Vk-1 such that (u,v) [image] E. Without loss of generality, let u be the first such vertex grayed, which must happen since, by induction, all vertices in Vk - 1 are grayed. The code for BFS enqueues every grayed vertex, and hence u must ultimately appear as the head of the queue in line 10. When u appears as the head, its adjacency list is scanned and v is discovered. (The vertex v could not have been discovered earlier, since it is not adjacent to any vertex in Vj for j < k - 1--otherwise, v could not belong to Vk--and by assumption, u is the first vertex discovered in Vk - 1 to which v is adjacent.) Line 13 grays v, line 14 establishes d[v] = d[u] + 1 = k, line 15 sets [image] [v] to u, and line 16 inserts v into the queue. Since v is an arbitrary vertex in Vk, the inductive hypothesis is proved.

To conclude the proof of the lemma, observe that if v [image] Vk, then by what we have just seen, [image] [v] [image] Vk - 1. Thus, we can obtain a shortest path from s to v by taking a shortest path from s to [image] [v] and then traversing the edge ( [image] [v], v).

Breadth-first trees

The procedure BFS builds a breadth-first tree as it searches the graph, as illustrated in Figure 23.3. The tree is represented by the [image] field in each vertex. More formally, for a graph G = (V, E) with source s, we define the predecessor subgraph of G as G [image] = (V [image] , E [image] ), where

V   = {v    V :   [v]    NIL}    {s}

and

E   = {(  [v],v)    E : v    V   - {s}} .

The predecessor subgraph G [image] is a breadth-first tree if V [image] consists of the vertices reachable from s and, for all v [image] V [image] , there is a unique simple path from s to v in G [image] that is also a shortest path from s to v in G. A breadth-first tree is in fact a tree, since it is connected and |E [image] | = |V [image] | - 1 (see Theorem 5.2). The edges in E [image] are called tree edges.

After BFS has been run from a source s on a graph G, the following lemma shows that the predecessor subgraph is a breadth-first tree.

Lemma 23.5

When applied to a directed or undirected graph G = (V, E), procedure BFS constructs [image] so that the predecessor subgraph G [image] = (V [image] , E [image] ) is a breadth-first tree.

Proof Line 15 of BFS only sets [image] [v] = u if (u, v) [image] E and [image] (s, v) < [image] --that is, if v is reachable from s--and thus V [image] consists of the vertices in V reachable from v. Since G [image] forms a tree, it contains a unique path from s to each vertex in V [image] . By applying Theorem 23.4 inductively, we conclude that every such path is a shortest path.

The following procedure prints out the vertices on a shortest path from s to v, assuming that BFS has already been run to compute the shortest-path tree.

PRINT-PATH(G,s,v)

1  if v = s

2     then print s

3     else if   [v] = NIL

4             then print "no path from" s "to" v "exists"

5             else PRINT-PATH(G,s,  [v])

6                  print v

This procedure runs in time linear in the number of vertices in the path printed, since each recursive call is for a path one vertex shorter.

Exercises

23.2-1

Show the result of running breadth-first search on the directed graph of Figure 23.2(a), using vertex 3 as the source.

23.2-2

Show the result of running breadth-first search on the undirected graph of Figure 23.3, using vertex u as the source.

23.2-3

What is the running time of BFS if its input graph is represented by an adjacency matrix and the algorithm is modified to handle this form of input?

23.2-4

Argue that in a breadth-first search, the value d[u] assigned to a vertex u is independent of the order in which the vertices in each adjacency list are given.

23.2-5

Give an example of a directed graph G = (V, E), a source vertex s [image] V, and a set of tree edges E [image] [image] E such that for each vertex v [image] V, the unique path in E [image] from s to v is a shortest path in G, yet the set of edges E [image] cannot be produced by running BFS on G, no matter how the vertices are ordered in each adjacency list.

23.2-6

Give an efficient algorithm to determine if an undirected graph is bipartite.

23.2-7

The diameter of a tree T = (V, E) is given by

that is, the diameter is the largest of all shortest-path distances in the tree. Give an efficient algorithm to compute the diameter of a tree, and analyze the running time of your algorithm.

23.2-8

Let G = (V, E) be an undirected graph. Give an O(V + E)-time algorithm to compute a path in G that traverses each edge in E exactly once in each direction. Describe how you can find your way out of a maze if you are given a large supply of pennies.

23.3 Depth-first search

The strategy followed by depth-first search is, as its name implies, to search "deeper" in the graph whenever possible. In depth-first search, edges are explored out of the most recently discovered vertex v that still has unexplored edges leaving it. When all of v's edges have been explored, the search "backtracks" to explore edges leaving the vertex from which v was discovered. This process continues until we have discovered all the vertices that are reachable from the original source vertex. If any undiscovered vertices remain, then one of them is selected as a new source and the search is repeated from that source. This entire process is repeated until all vertices are discovered.

As in breadth-first search, whenever a vertex v is discovered during a scan of the adjacency list of an already discovered vertex u, depth-first search records this event by setting v's predecessor field [image] [v] to u. Unlike breadth-first search, whose predecessor subgraph forms a tree, the predecessor subgraph produced by a depth-first search may be composed of several trees, because the search may be repeated from multiple sources. The predecessor subgraph of a depth-first search is therefore defined slightly differently from that of a breadth-first search: we let G [image] = (V, E [image] ), where

E   = {(  [v], v) : v    V and   [v]    NIL} .

The predecessor subgraph of a depth-first search forms a depth-first forest composed of several depth-first trees. The edges in E [image] are called tree edges.

As in breadth-first search, vertices are colored during the search to indicate their state. Each vertex is initially white, is grayed when it is discovered in the search, and is blackened when it is finished, that is, when its adjacency list has been examined completely. This technique guarantees that each vertex ends up in exactly one depth-first tree, so that these trees are disjoint.

Besides creating a depth-first forest, depth-first search also timestamps each vertex. Each vertex v has two timestamps: the first timestamp d[v] records when v is first discovered (and grayed), and the second timestamp f[v] records when the search finishes examining v's adjacency list (and blackens v). These timestamps are used in many graph algorithms and are generally helpful in reasoning about the behavior of depth-first search.

The procedure DFS below records when it discovers vertex u in the variable d[u] and when it finishes vertex u in the variable f[u]. These timestamps are integers between 1 and 2 |V|, since there is one discovery event and one finishing event for each of the |V| vertices. For every vertex u,

d[u] < f[u] .

(23.1)

Vertex u is WHITE before time d[u], GRAY between time d[u] and time f[u], and BLACK thereafter.

The following pseudocode is the basic depth-first-search algorithm. The input graph G may be undirected or directed. The variable time is a global variable that we use for timestamping.

DFS(G)

1  for each vertex u    V[G]

2       do color[u]    WHITE

3            [u]    NIL

4  time    0

5  for each vertex u    V[G]

6       do if color[u] = WHITE

7             then DFS-VISIT(u)

Figure 23.4 illustrates the progress of DFS on the graph shown in Figure 23.2.

Procedure DFS works as follows. Lines 1-3 paint all vertices white and initialize their [image] fields to NIL. Line 4 resets the global time counter. Lines 5-7 check each vertex in V in turn and, when a white vertex is found, visit it using DFS-VISIT. Every time DFS-VISIT(u) is called in line 7, vertex u becomes the root of a new tree in the depth-first forest. When DFS returns, every vertex u has been assigned a discovery time d[u] and a finishing time â[u].

In each call DFS-VISIT(u), vertex u is initially white. Line 1 paints u gray, and line 2 records the discovery time d[u] by incrementing and saving the global variable time. Lines 3-6 examine each vertex v adjacent to u and recursively visit v if it is white. As each vertex v [image] Adj[u] is considered in line 3, we say that edge (u, v) is explored by the depth-first search. Finally, after every edge leaving u has been explored, lines 7-8 paint u black and record the finishing time in â[u].

Figure 23.4 The progress of the depth-first-search algorithm DFS on a directed graph. As edges are explored by the algorithm, they are shown as either shaded (if they are tree edges) or dashed (otherwise). Nontree edges are labeled B, C, or F according to whether they are back, cross, or forward edges. Vertices are timestamped by discovery time/finishing time.

What is the running time of DFS? The loops on lines 1-2 and lines 5-7 of DFS take time [image] (V), exclusive of the time to execute the calls to DFS-VISIT. The procedure DFS-VISIT is called exactly once for each vertex the v [image] V, since DFS-VISIT is invoked only on white vertices and the first thing it does is paint the vertex gray. During an execution of DFS-VISIT(v), the loop on lines 3-6 is executed |Adj[v]| times. Since

the total cost of executing lines 2-5 of DFS-VISIT is [image] (E). The running time of DFS is therefore [image] (V + E).

Properties of depth-first search

Depth-first search yields much information about the structure of a graph. Perhaps the most basic property of depth-first search is that the predecessor subgraph G [image] does indeed form a forest of trees, since the structure of the depth-first trees exactly mirrors the structure of recursive calls of DFS-VISIT. That is, u = [image] [v] if and only if DFS-VISIT (v) was called during a search of u's adjacency list.

Another important property of depth-first search is that discovery and finishing times have parenthesis structure. If we represent the discovery of vertex u with a left parenthesis "(u" and represent its finishing by a right parenthesis "u)," then the history of discoveries and finishings makes a well-formed expression in the sense that the parentheses are properly nested. For example, the depth-first search of Figure 23.5(a) corresponds to the parenthesization shown in Figure 23.5(b). Another way of stating the condition of parenthesis structure is given in the following theorem.

Theorem 23.6

In any depth-first search of a (directed or undirected) graph G = (V, E), for any two vertices u and v, exactly one of the following three conditions holds:

the intervals [d[u], â[u]] and [d[v], â[v]] are entirely disjoint,

the interval [d[u], â[u]] is contained entirely within the interval [d[v], â[u]], and u is a descendant of v in the depth-first tree, or

the interval [d[v], â[v]] is contained entirely within the interval [d[u], â[u]], and v is a descendant of u in the depth-first tree.

Proof We begin with the case in which d[u] < d[v]. There are two subcases to consider, according to whether d[v] < â[u] or not. In the first subcase, d[v] < â[u], so v was discovered while u was still gray. This implies that v is a descendant of u. Moreover, since v was discovered more recently than u, all of its outgoing edges are explored, and v is finished, before the search returns to and finishes u. In this case, therefore, the interval [d[v],â[v]] is entirely contained within the interval [d[u],â[u]]. In the other subcase, â[u] < d[v], and inequality (23.1) implies that the intervals [d[u],â[u]] and [d[v],â[v]] are disjoint.

The case in which d[v] < d[u] is similar, with the roles of u and v reversed in the above argument.

Corollary 23.7

Vertex v is a proper descendant of vertex u in the depth-first forest for a (directed or undirected) graph G if and only if d[u] < d[v] < â[v] < â[u].

Proof Immediate from Theorem 23.6.

Figure 23.5 Properties of depth-first search. (a) The result of a depth-first search of a directed graph. Vertices are timestamped and edge types are indicated as in Figure 23.4. (b) Intervals for the discovery time and finishing time of each vertex correspond to the parenthesization shown. Each rectangle spans the interval given by the discovery and finishing times of the corresponding vertex. Tree edges are shown. If two intervals overlap, then one is nested within the other, and the vertex corresponding to the smaller interval is a descendant of the vertex corresponding to the larger. (c) The graph of part (a) redrawn with all tree and forward edges going down within a depth-first tree and all back edges going up from a descendant to an ancestor.

The next theorem gives another important characterization of when one vertex is a descendant of another in the depth-first forest.

Theorem 23.8

In a depth-first forest of a (directed or undirected) graph G = (V, E), vertex v is a descendant of vertex u if and only if at the time d[u] that the search discovers u, vertex v can be reached from u along a path consisting entirely of white vertices.

Proof [image]: Assume that v is a descendant of u. Let w be any vertex on the path between u and v in the depth-first tree, so that w is a descendant of u. By Corollary 23.7, d[u] < d[w], and so w is white at time d[u].

: Suppose that vertex v is reachable from u along a path of white vertices at time d[u], but v does not become a descendant of u in the depth-first tree. Without loss of generality, assume that every other vertex along the path becomes a descendant of u. (Otherwise, let v be the closest vertex to u along the path that doesn't become a descendant of u.) Let w be the predecessor of v in the path, so that w is a descendant of u (w and u may in fact be the same vertex) and, by Corollary 23.7, â[w] [image] â[u]. Note that v must be discovered after u is discovered, but before w is finished. Therefore, d[u] < d[v] < â[w] <â[u]. Theorem 23.6 then implies that the interval [d[v], â[v]] is contained entirely within the interval [d[u],â[u]]. By Corollary 23.7, v must after all be a descendant of u.

Classification of edges

Another interesting property of depth-first search is that the search can be used to classify the edges of the input graph G = (V, E). This edge classification can be used to glean important information about a graph. For example, in the next section, we shall see that a directed graph is acyclic if and only if a depth-first search yields no "back" edges (Lemma 23.10).

We can define four edge types in terms of the depth-first forest G [image] produced by a depth-first search on G.

1. Tree edges are edges in the depth-first forest G [image] . Edge (u, v) is a tree edge if v was first discovered by exploring edge (u, v).

2. Back edges are those edges (u, v) connecting a vertex u to an ancestor v in a depth-first tree. Self-loops are considered to be back edges.

3. Forward edges are those nontree edges (u, v) connecting a vertex u to a descendant v in a depth-first tree.

4. Cross edges are all other edges. They can go between vertices in the same depth-first tree, as long as one vertex is not an ancestor of the other, or they can go between vertices in different depth-first trees.

In Figures 23.4 and 23.5, edges are labeled to indicate their type. Figure 23.5(c) also shows how the graph of Figure 23.5(a) can be redrawn so that all tree and forward edges head downward in a depth-first tree and all back edges go up. Any graph can be redrawn in this fashion.

The DFS algorithm can be modified to classify edges as it encounters them. The key idea is that each edge (u, v) can be classified by the color of the vertex v that is reached when the edge is first explored (except that forward and cross edges are not distinguished):

1. WHITE indicates a tree edge,

2. GRAY indicates a back edge, and

3. BLACK indicates a forward or cross edge.

The first case is immediate from the specification of the algorithm. For the second case, observe that the gray vertices always form a linear chain of descendants corresponding to the stack of active DFS-VISIT invocations; the number of gray vertices is one more than the depth in the depth-first forest of the vertex most recently discovered. Exploration always proceeds from the deepest gray vertex, so an edge that reaches another gray vertex reaches an ancestor. The third case handles the remaining possibility; it can be shown that such an edge (u, v) is a forward edge if d[u] < d[v] and a cross edge if d[u] > d[v]. (See Exercise 23.3-4.)

In an undirected graph, there may be some ambiguity in the type classification, since (u, v) and (v, u) are really the same edge. In such a case, the edge is classified as the first type in the classification list that applies. Equivalently (see Exercise 23.3-5), the edge is classified according to whichever of (u, v) or (v, u) is encountered first during the execution of the algorithm.

We now show that forward and cross edges never occur in a depth-first search of an undirected graph.

Theorem 23.9

In a depth-first search of an undirected graph G, every edge of G is either a tree edge or a back edge.

Proof Let (u, v) be an arbitrary edge of G, and suppose without loss of generality that d[u] < d[v]. Then, v must be discovered and finished before we finish u, since v is on u's adjacency list. If the edge (u, v) is explored first in the direction from u to v, then (u, v) becomes a tree edge. If (u, v) is explored first in the direction from v to u, then (u, v) is a back edge, since u is still gray at the time the edge is first explored.

We shall see many applications of these theorems in the following sections.

Exercises

23.3-1

Make a 3-by-3 chart with row and column labels WHITE, GRAY, and BLACK. In each cell (i, j), indicate whether, at any point during a depth-first search of a directed graph, there can be an edge from a vertex of color i to a vertex of color j. For each possible edge, indicate what edge types it can be. Make a second such chart for depth-first search of an undirected graph.

Figure 23.6 A directed graph for use in Exercises 23.3-2 and 23.5-2.

23.3-2

Show how depth-first search works on the graph of Figure 23.6. Assume that the for loop of lines 5-7 of the DFS procedure considers the vertices in alphabetical order, and assume that each adjacency list is ordered alphabetically. Show the discovery and finishing times for each vertex, and show the classification of each edge.

23.3-3

Show the parenthesis structure of the depth-first search shown in Figure 23.4.

23.3-4

Show that edge (u,v) is

a. a tree edge or forward edge if and only if d[u] < d[v] < f[v] < f[u],

b. a back edge if and only if d[v] < d[u] < f[u] < f[v], and

c. a cross edge if and only if d[v] < f[v] < d[u] < f[u].

23.3-5

Show that in an undirected graph, classifying an edge (u, v) as a tree edge or a back edge according to whether (u,v) or (v,u) is encountered first during the depth-first search is equivalent to classifying it according to the priority of types in the classification scheme.

23.3-6

Give a counterexample to the conjecture that if there is a path from u to v in a directed graph G, and if d[u] < d[v] in a depth-first search of G, then v is a descendant of u in the depth-first forest produced.

23.3-7

Modify the pseudocode for depth-first search so that it prints out every edge in the directed graph G, together with its type. Show what modifications, if any, must be made if G is undirected.

23.3-8

Explain how a vertex u of a directed graph can end up in a depth-first tree containing only u, even though u has both incoming and outgoing edges in G.

23.3-9

Show that a depth-first search of an undirected graph G can be used to identify the connected components of G, and that the depth-first forest contains as many trees as G has connected components. More precisely, show how to modify depth-first search so that each vertex v is assigned an integer label cc[v] between 1 and k, where k is the number of connected components of G, such that cc[u] = cc[v] if and only if u and v are in the same connected component.

23.3-10

A directed graph G = (V, E) is singly connected if [image] implies that there is at most one simple path from u to v for all vertices u, v [image] V. Give an efficient algorithm to determine whether or not a directed graph is singly connected.

23.4 Topological sort

This section shows how depth-first search can be used to perform topological sorts of directed acyclic graphs, or "dags" as they are sometimes called. A topological sort of a dag G = (V, E) is a linear ordering of all its vertices such that if G contains an edge (u, v), then u appears before v in the ordering. (If the graph is not acyclic, then no linear ordering is possible.) A topological sort of a graph can be viewed as an ordering of its vertices along a horizontal line so that all directed edges go from left to right. Topological sorting is thus different from the usual kind of "sorting" studied in Part II.

Directed acyclic graphs are used in many applications to indicate precedences among events. Figure 23.7 gives an example that arises when Professor Bumstead gets dressed in the morning. The professor must don certain garments before others (e.g., socks before shoes). Other items may be put on in any order (e.g., socks and pants). A directed edge (u,v) in the dag of Figure 23.7(a) indicates that garment u must be donned before garment v. A topological sort of this dag therefore gives an order for getting dressed. Figure 23.7(b) shows the topologically sorted dag as an ordering of vertices along a horizontal line such that all directed edges go from left to right.

Figure 23.7 (a) Professor Bumstead topologically sorts his clothing when getting dressed. Each directed edge (u,v) means that garment u must be put on before garment v. The discovery and finishing times from a depth-first search are shown next to each vertex. (b) The same graph shown topologically sorted. Its vertices are arranged from left to right in order of decreasing finishing time. Note that all directed edges go from left to right.

The following simple algorithm topologically sorts a dag.

TOPOLOGICAL-SORT(G)

1  call DFS(G) to compute finishing times f[v] for each vertex v

2  as each vertex is finished, insert it onto the front of a linked list

3  return the linked list of vertices

Figure 23.7(b) shows how the topologically sorted vertices appear in reverse order of their finishing times.

We can perform a topological sort in time [image] (V + E), since depth-first search takes [image] (V + E) time and it takes 0(1) time to insert each of the |V| vertices onto the front of the linked list.

We prove the correctness of this algorithm using the following key lemma characterizing directed acyclic graphs.

Lemma 23.10

A directed graph G is acyclic if and only if a depth-first search of G yields no back edges.

Proof [image]: Suppose that there is a back edge (u, v). Then, vertex v is an ancestor of vertex u in the depth-first forest. There is thus a path from v to u in G, and the back edge (u, v) completes a cycle.

Figure 23.8 A dag for topological sorting.

: Suppose that G contains a cycle c. We show that a depth-first search of G yields a back edge. Let v be the first vertex to be discovered in c, and let (u, v) be the preceding edge in c. At time d[v], there is a path of white vertices from v to u. By the white-path theorem, vertex u becomes a descendant of v in the depth-first forest. Therefore, (u, v) is a back edge.

Theorem 23.11

TOPOLOGICAL-SORT(G) produces a topological sort of a directed acyclic graph G.

Proof Suppose that DFS is run on a given dag G = (V, E) to determine finishing times for its vertices. It suffices to show that for any pair of distinct vertices u,v [image] V, if there is an edge in G from u to v, then f[v] < f[u]. Consider any edge (u,v) explored by DFS(G). When this edge is explored, v cannot be gray, since then v would be an ancestor of u and (u,v) would be a back edge, contradicting Lemma 23.10. Therefore, v must be either white or black. If v is white, it becomes a descendant of u, and so f[v] < f[u]. If v is black, then f[v] < f[u] as well. Thus, for any edge (u,v) in the dag, we have f[v] < f[u], proving the theorem.

Exercises

23.4-1

Show the ordering of vertices produced by TOPOLOGICAL-SORT when it is run on the dag of Figure 23.8.

23.4-2

There are many different orderings of the vertices of a directed graph G that are topological sorts of G. TOPOLOGICAL-SORT produces the ordering that is the reverse of the depth-first finishing times. Show that not all topological sorts can be produced in this way: there exists a graph G such that one of the topological sorts of G cannot be produced by TOPOLOGICAL-SORT, no matter what adjacency-list structure is given for G. Show also that there exists a graph for which two distinct adjacency-list representations yield the same topological sort.

23.4-3

Give an algorithm that determines whether or not a given undirected graph G = (V, E) contains a cycle. Your algorithm should run in O(V) time, independent of |E|.

23.4-4

Prove or disprove: If a directed graph G contains cycles, then TOPOLOGICAL-SORT(G) produces a vertex ordering that minimizes the number of "bad" edges that are inconsistent with the ordering produced.

23.4-5

Another way to perform topological sorting on a directed acyclic graph G = (V, E) is to repeatedly find a vertex of in-degree 0, output it, and remove it and all of its outgoing edges from the graph. Explain how to implement this idea so that it runs in time O(V + E). What happens to this algorithm if G has cycles?

23.5 Strongly connected components

We now consider a classic application of depth-first search: decomposing a directed graph into its strongly connected components. This section shows how to do this decomposition using two depth-first searches. Many algorithms that work with directed graphs begin with such a decomposition; this approach often allows the original problem to be divided into subproblems, one for each strongly connected component. Combining the solutions to the subproblems follows the structure of connections between strongly connected components; this structure can be represented by a graph known as the "component" graph, defined in Exercise 23.5-4.

Recall from Chapter 5 that a strongly connected component of a directed graph G = (V, E) is a maximal set of vertices U [image] V such that for every pair of vertices u and v in U, we have both [image] that is, vertices u and v are reachable from each other. Figure 23.9 shows an example.

Our algorithm for finding strongly connected components of a graph G = (V, E) uses the transpose of G, which is defined in Exercise 23.1-3 to be the graph GT = (V, ET), where ET = {(u, v): (v, u) [image] E}. That is, ET consists of the edges of G with their directions reversed. Given an adjacency-list representation of G, the time to create GT is O(V + E). It is interesting to observe that G and GT have exactly the same strongly connected components: u and v are reachable from each other in G if and only if they are reachable from each other in GT. Figure 23.9(b) shows the transpose of the graph in Figure 23.9(a), with the strongly connected components shaded.

Figure 23.9 (a) A directed graph G. The strongly connected components of G are shown as shaded regions. Each vertex is labeled with its discovery and finishing times. Tree edges are shaded. (b) The graph GT, the transpose of G. The depth-first tree computed in line 3 of STRONGLY-CONNECTED-COMPONENTS is shown, with tree edges shaded. Each strongly connected component corresponds to one depth-first tree. Vertices b, c, g, and h, which are heavily shaded, are forefathers of every vertex in their strongly connected component; these vertices are also the roots of the depth-first trees produced by the depth-first search of GT. (c) The acyclic component graph Gscc obtained by shrinking each strongly connected component of G to a single vertex.

The following linear-time (i.e., [image] (V + E)-time) algorithm computes the strongly connected components of a directed graph G = (V, E) using two depth-first searches, one on G and one on GT.

STRONGLY-CONNECTED-COMPONENTS(G)

1  call DFS(G) to compute finishing times f[u] for each vertex u

2  compute GT

3  call DFS(GT), but in the main loop of DFS, consider the vertices

in order of decreasing f[u] (as computed in line 1)

4  output the vertices of each tree in the depth-first forest of step 3 as a

separate strongly connected component

This simple-looking algorithm seems to have nothing to do with strongly connected components. In the remainder of this section, we unravel the mystery of its design and prove its correctness. We begin with two useful observations.

Lemma 23.12

If two vertices are in the same strongly connected component, then no path between them ever leaves the strongly connected component.

Proof Let u and v be two vertices in the same strongly connected component. By the definition of strongly connected component, there are paths from u to v and from v to u. Let vertex w be on some path [image], so that w is reachable from u. Moreover, since there is a path [image], we know that u is reachable from w by the path [image]. Therefore, u and w are in the same strongly connected component. Since w was chosen arbitrarily, the theorem is proved.

Theorem 23.13

In any depth-first search, all vertices in the same strongly connected component are placed in the same depth-first tree.

Proof Of the vertices in the strongly connected component, let r be the first discovered. Because r is first, the other vertices in the strongly connected component are white at the time it is discovered. There are paths from r to every other vertex in the strongly connected component; because these paths never leave the strongly connected component (by Lemma 23.12), all vertices on them are white. Thus, by the white-path theorem, every vertex in the strongly connected component becomes a descendant of r in the depth-first tree.

In the rest of this section, the notations d[u] and f[u] refer to the discovery and finishing times as computed by the first depth-first search in line 1 of STRONGLY-CONNECTED-COMPONENTS. Similarly, the notation [image] refers to the existence of a path in G, not in GT.

To prove STRONGLY-CONNECTED-COMPONENTS correct, we introduce the notion of the forefather [image] (u) of a vertex u, which is the vertex w reachable from u that finished last in the depth-first search of line l. In other words,

(u) = that vertex w such that [image] and f[w] is maximized .

Note that [image] (u) = u is possible because u is reachable from itself, and hence

f[u]    f[  (u)] .

(23.2)

We can also show that [image] ( [image] (u)) = [image] (u), by the following reasoning. For any vertices u, v [image] V,

(23.3)

since [image] and the forefather has the maximum finishing time of all reachable vertices. Since [image] (u) is reachable from u, formula (23.3) implies that f[ [image] ( [image] (u))] [image] f[ [image] (u)]. We also have f[ [image] (u)] [image] f[ [image] ( [image] (u))], by inequality (23.2). Thus, f[ [image] ( [image] (u))] = f[ [image] (u)], and so we have [image] ( [image] (u)) = [image] (u), since two vertices that finish at the same time are in fact the same vertex.

As we shall see, every strongly connected component has one vertex that is the forefather of every vertex in the strongly connected component; this forefather is a "representative vertex" for the strongly connected component. In the depth-first search of G, it is the first vertex of the strongly connected component to be discovered, and it is the last vertex of the strongly connected component to be finished. In the depth-first search of GT, it is the root of a depth-first tree. We now prove these properties.

The first theorem justifies calling ø(u) a "forefather" of u.

Theorem 23.14

In a directed graph G = (V, E), the forefather [image] (u) of any vertex u [image] V in any depth-first search of G is an ancestor of u.

Proof If [image] (u) = u, the theorem is trivially true. If [image] (u) [image] u, consider the colors of the vertices at time d[u]. If [image] (u) is black, then f[ [image] (u)] < f[u], contradicting inequality (23.2). If [image] (u) is gray, then it is an ancestor of u, and the theorem is proved.

It thus remains to prove that [image] (u) is not white. There are two cases, according to the colors of the intermediate vertices, if any, on the path from u to [image] (u).

1. If every intermediate vertex is white, then [image] (u) becomes a descendant of u, by the white-path theorem. But then f[ [image] (u)] < f[u], contradicting inequality (23.2).

2. If some intermediate vertex is nonwhite, let t be the last nonwhite vertex on the path from u to [image] (u). Then, t must be gray, since there is never an edge from a black vertex to a white vertex, and t's successor is white. But then there is a path of white vertices from t to [image] (u), and so [image] (u) is a descendant of t by the white-path theorem. This implies that f[t] > f[ [image] (u)], contradicting our choice of [image] (u), since there is a path from u to t.

Corollary 23.15

In any depth-first search of a directed graph G = (V, E), vertices u and ø(u), for all u [image] V, lie in the same strongly connected component.

Proof We have [image], by the definition of forefather, and [image] since ø(u) is an ancestor of u.

The following theorem gives a stronger result relating forefathers to strongly connected components.

Theorem 23.16

In a directed graph G = (V, E), two vertices u, v [image] V lie in the same strongly connected component if and only if they have the same forefather in a depth-first search of G.

Proof [image] : Assume that u and v are in the same strongly connected component. Every vertex reachable from u is reachable from v and vice versa, since there are paths in both directions between u and v. By the definition of forefather, then, we conclude that [image] (u) = [image] (v).

: Assume that [image] (u) = [image] (v). By Corollary 23.15, u is in the same strongly connected component as [image] (u), and v is in the same strongly connected component as [image] (v). Therefore, u and v are in the same strongly connected component.

With Theorem 23.16 in hand, the structure of the algorithm STRONGLY-CONNECTED-COMPONENTS can be more readily understood. The strongly connected components are sets of vertices with the same forefather. Moreover, by Theorem 23.14 and the parenthesis theorem (Theorem 23.6), during the depth-first search in line l of STRONGLY-CONNECTED-COMPONENTS a forefather is both the first vertex discovered and the last vertex finished in its strongly connected component.

To understand why we run the depth-first search in line 3 of STRONGLY-CONNECTED-COMPONENTS on GT, consider the vertex r with the largest finishing time computed by the depth-first search in line 1. By the definition of forefather, vertex r must be a forefather, since it is its own forefather: it can reach itself, and no vertex in the graph has a higher finishing time. What are the other vertices in r's strongly connected component? They are those vertices that have r as a forefather--those that can reach r but cannot reach any vertex with a finishing time greater than f[r]. But r's finishing time is the maximum of any vertex in G; thus, r's strongly connected component consists simply of those vertices that can reach r. Equivalently, r's strongly connected component consists of those vertices that r can reach in GT. Thus, the depth-first search in line 3 identifies all the vertices in r's strongly connected component and blackens them. (A breadth-first search, or any search for reachable vertices, could identify this set just as easily.)

After the depth-first search in line 3 is done identifying r's strongly connected component, it begins at the vertex r' with the largest finishing time of any vertex not in r's strongly connected component. Vertex r' must be its own forefather, since it can't reach anything with a higher finishing time (otherwise, it would have been included in r's strongly connected component). By similar reasoning, any vertex that can reach r' 'that is not already black must be in r''s strongly connected component. Thus, as the depth-first search in line 3 continues, it identifies and blackens every vertex in r''s strongly connected component by searching from r' in GT.

Thus, the depth-first search in line 3 "peels off" strongly connected components one by one. Each component is identified in line 7 of DFS by a call to DFS-VISIT with the forefather of the component as an argument. Recursive calls within DFS-VISIT ultimately blacken each vertex within the component. When DFS-VISIT returns to DFS, the entire component has been blackened and "peeled off." Then, DFS finds the vertex with maximum finishing time among those that have not been blackened; this vertex is the forefather of another component, and the process continues.

The following theorem formalizes this argument.

Theorem 23.17

STRONGLY-CONNECTED-COMPONENTS(G) correctly computes the strongly connected components of a directed graph G.

Proof We argue by induction on the number of depth-first trees found in the depth-first search of GT that the vertices of each tree form a strongly connected component. Each step of the inductive argument proves that a tree formed in the depth-first search of GT is a strongly connected component, assuming that all previous trees produced are strongly connected components. The basis for the induction is trivial, since for the first tree produced there are no previous trees, and hence this assumption is trivially true.

Consider a depth-first tree T with root r produced in the depth-first search of GT. Let C(r) denote the set of vertices with forefather r:

C(r) = {v    V:   (v) = r}.

We now prove that a vertex u is placed in T if and only if u [image] C(r).

: Theorem 23.13 implies that every vertex in C(r) ends up in the same depth-first tree. Since r [image] C(r) and r is the root of T, every element of C(r) ends up in T.

: We show that any vertex w such that â[ [image] (w)] > â[r] or â[ (w)] < â[r] is not placed in T, by considering these two cases separately. By induction on the number of trees found, any vertex w such that â[ (w)] > â[r] is not placed in tree T, since at the time r is selected w will have already been placed in the tree with root (w). Any vertex w such that â[ (w)] < â[r] cannot be placed in T, since such a placement would imply ; thus, by formula (23.3) and the property that r = (r), we obtain â[ (w)] â[ (r)] = â[r], which contradicts â[ (w)] < â[r].

Therefore, T contains just those vertices u for which (u) = r. That is, T is exactly equal to the strongly connected component C(r), which completes the inductive proof.

Exercises

23.5-1

How can the number of strongly connected components of a graph change if a new edge is added?

23.5-2

Show how the procedure STRONGLY-CONNECTED-COMPONENTS works on the graph of Figure 23.6. Specifically, show the finishing times computed in line 1 and the forest produced in line 3. Assume that the loop of lines 5-7 of DFS considers vertices in alphabetical order and that the adjacency lists are in alphabetical order.

23.5-3

Professor Deaver claims that the algorithm for strongly connected components can be simplified by using the original (instead of the transpose) graph in the second depth-first search and scanning the vertices in order of increasing finishing times. Is the professor correct?

23.5-4

We denote the component graph of G = (V, E) by GSCC = (VSCC, ESCC), where VSCC contains one vertex for each strongly connected component of G and ESCC contains the edge (u, v) if there is a directed edge from a vertex in the strongly connected component of G corresponding to u to a vertex in the strongly connected component of G corresponding to v. Figure 23.9(c) shows an example. Prove that GSCC is a dag.

23.5-5

Give an O(V + E)-time algorithm to compute the component graph of a directed graph G = (V,E). Make sure that there is at most one edge between two vertices in the component graph your algorithm produces.

23.5-6

Given a directed graph G = (V, E), explain how to create another graph G' = (V, E') such that (a) G' has the same strongly connected components as G, (b) G' has the same component graph as G, and (c) E' is as small as possible. Describe a fast algorithm to compute G'.

23.5-7

A directed graph G = (V, E) is said to be semiconnected if, for any two vertices u, v V, we have . Give an efficient algorithm to determine whether or not G is semiconnected. Prove that your algorithm is correct, and analyze its running time.

Problems

23-1 Classifying edges by breadth-first search

A depth-first forest classifies the edges of a graph into tree, back, forward, and cross edges. A breadth-first tree can also be used to classify the edges reachable from the source of the search into the same four categories.

a. Prove that in a breadth-first search of an undirected graph, the following properties hold:

1. There are no back edges and no forward edges.

2. For each tree edge (u,v), we have d[v] = d[u] + 1.

3. For each cross edge (u,v), we have d[v] = d[u] or d[v] = d[u] + 1.

b. Prove that in a breadth-first search of a directed graph, the following properties hold:

1. There are no forward edges.

2. For each tree edge (u,v), we have d[v] = d[u] + 1.

3. For each cross edge (u,v), we have d[v] d[u] + 1.

4. For each back edge (u,v), we have 0 d[v] < d[u].

23-2 Articulation points, bridges, and biconnected components

Let G = (V, E) be a connected, undirected graph. An articulation point of G is a vertex whose removal disconnects G. A bridge of G is an edge whose removal disconnects G. A biconnected component of G is a maximal set of edges such that any two edges in the set lie on a common simple cycle. Figure 23.10 illustrates these definitions. We can determine articulation points, bridges, and biconnected components using depth-first search. Let G = (V,E ) be a depth-first tree of G.

Figure 23.10 The articulation points, bridges, and biconnected components of a connected, undirected graph for use in Problem 23-2. The articulation points are the heavily shaded vertices, the bridges are the heavily shaded edges, and the biconnected components are the edges in the shaded regions, with a bcc numbering shown.

a. Prove that the root of G is an articulation point of G if and only if it has at least two children in G .

b. Let v be a nonroot vertex in G . Prove that v is an articulation point of G if and only if there is no back edge (u, w) such that in G , u is a descendant of and w is a proper ancestor of v.

c. Let

Show how to compute low[ ] for all vertices V in O(E) time.

d. Show how to compute all articulation points in O(E) time.

e. Prove that an edge of G is a bridge if and only if it does not lie on any simple cycle of G.

f. Show how to compute all the bridges of G in O(E) time.

g. Prove that the biconnected components of G partition the nonbridge edges of G.

h. Give an O(E)-time algorithm to label each edge e of G with a positive integer bcc[e] such that bcc[e] = bcc[e'] if and only if e and e' are in the same biconnected component.

23-3 Euler tour

An Euler tour of a connected, directed graph G = (V, E) is a cycle that traverses each edge of G exactly once, although it may visit a vertex more than once.

a. Show that G has an Euler tour if and only if

in-degree(v) = out-degree(v)

for each vertex v V.

b. Describe an O(E)-time algorithm to find an Euler tour of G if one exists. (Hint: Merge edge-disjoint cycles.)

Chapter notes

Even [65] and Tarjan [188] are excellent references for graph algorithms.

Breadth-first search was discovered by Moore [150] in the context of finding paths through mazes. Lee [134] independently discovered the same algorithm in the context of routing wires on circuit boards.

Hopcroft and Tarjan [102] advocated the use of the adjacency-list representation over the adjacency-matrix representation for sparse graphs and were the first to recognize the algorithmic importance of depth-first search. Depth-first search has been widely used since the late 1950's, especially in artificial intelligence programs.

Tarjan [185] gave a linear-time algorithm for finding strongly connected components. The algorithm for strongly connected components in Section 23.5 is adapted from Aho, Hopcroft, and Ullman [5], who credit it to S. R. Kosaraju and M. Sharir. Knuth [121] was the first to give a linear-time algorithm for topological sorting.

Go to Chapter 24 Back to Table of Contents


Intro to Algorithms: CHAPTER 24: MINIMUM SPANNING TREES

 Return to Table of Contents

CHAPTER 24: MINIMUM SPANNING TREES

In the design of electronic circuitry, it is often necessary to make the pins of several components electrically equivalent by wiring them together. To interconnect a set of n pins, we can use an arrangement of n - 1 wires, each connecting two pins. Of all such arrangements, the one that uses the least amount of wire is usually the most desirable.

We can model this wiring problem with a connected, undirected graph G = (V, E), where V is the set of pins, E is the set of possible interconnections between pairs of pins, and for each edge (u, v) [image] E, we have a weight w(u, v) specifying the cost (amount of wire needed) to connect u and v. We then wish to find an acyclic subset T [image] E that connects all of the vertices and whose total weight

is minimized. Since T is acyclic and connects all of the vertices, it must form a tree, which we call a spanning tree since it"spans" the graph G. We call the problem of determining the tree T the minimum-spanning-tree problem.1 Figure 24.1 shows an example of a connected graph and its minimum spanning tree.

1The phrase "minimum spanning tree" is a shortened form of the phrase "minimum-weight spanning tree." We are not, for example, minimizing the number of edges in T, since all spanning trees have exactly |V| - 1 edges by Theorem 5.2.

In this chapter, we shall examine two algorithms for solving the minimum-spanning-tree problem: Kruskal's algorithm and Prim's algorithm. Each can easily be made to run in time O(E lg V) using ordinary binary heaps. By using Fibonacci heaps, Prim's algorithm can be sped up to run in time O(E + V lg V), which is an improvement if |V| is much smaller than |E| .

The two algorithms also illustrate a heuristic for optimization called the "greedy" strategy. At each step of an algorithm, one of several possible choices must be made. The greedy strategy advocates making the choice that is the best at the moment. Such a strategy is not generally guaranteed to find globally optimal solutions to problems. For the minimum-spanning-tree problem, however, we can prove that certain greedy strategies do yield a spanning tree with minimum weight. Greedy strategies are discussed at length in Chapter 17. Although the present chapter can be read independently of Chapter 17, the greedy methods presented here are a classic application of the theoretical notions introduced there.

Figure 24.1 A minimum spanning tree for a connected graph. The weights on edges are shown, and the edges in a minimum spanning tree are shaded. The total weight of the tree shown is 37. The tree is not unique: removing the edge (b, c) and replacing it with the edge (a, h) yields another spanning tree with weight 37.

Section 24.1 introduces a "generic" minimum-spanning-tree algorithm that grows a spanning tree by adding one edge at a time. Section 24.2 gives two ways to implement the generic algorithm. The first algorithm, due to Kruskal, is similar to the connected-components algorithm from Section 22.1. The second, due to Prim, is similar to Dijkstra's shortest-paths algorithm (Section 25.2).

24.1 Growing a minimum spanning tree

Assume that we have a connected, undirected graph G = (V, E) with a weight function w : E [image] R and wish to find a minimum spanning tree for G. The two algorithms we consider in this chapter use a greedy approach to the problem, although they differ in how they apply this approach.

This greedy strategy is captured by the following "generic" algorithm, which grows the minimum spanning tree one edge at a time. The algorithm manages a set A that is always a subset of some minimum spanning tree. At each step, an edge (u, v) is determined that can be added to A without violating this invariant, in the sense that A [image] {(u, v)} is also a subset of a minimum spanning tree. We call such an edge a safe edge for A, since it can be safely added to A without destroying the invariant.

Figure 24.2 Two ways of viewing a cut (S, V - S) of the graph from Figure 24.1. (a) The vertices in the set S are shown in black, and those in V - S are shown in white. The edges crossing the cut are those connecting white vertices with black vertices. The edge (d, c) is the unique light edge crossing the cut. A subset A of the edges is shaded; note that the cut (S, V - S) respects A, since no edge of A crosses the cut. (b) The same graph with the vertices in the set S on the left and the vertices in the set V - S on the right. An edge crosses the cut if it connects a vertex on the left with a vertex on the right.

GENERIC-MST(G, w)

1  A

2  while A does not form a spanning tree

3      do find an edge (u, v) that is safe for A

4         A    A    {(u, v)}

5  return A

Note that after line 1, the set A trivially satisfies the invariant that it is a subset of a minimum spanning tree. The loop in lines 2-4 maintains the invariant. When the set A is returned in line 5, therefore, it must be a minimum spanning tree. The tricky part is, of course, finding a safe edge in line 3. One must exist, since when line 3 is executed, the invariant dictates that there is a spanning tree T such that A [image] T, and if there is an edge (u, v) [image] T such that (u, v) [image] A, then (u, v) is safe for A.

In the remainder of this section, we provide a rule (Theorem 24.1 ) for recognizing safe edges. The next section describes two algorithms that use this rule to find safe edges efficiently.

We first need some definitions. A cut (S, V - S) of an undirected graph G = (V, E) is a partition of V. Figure 24.2 illustrates this notion. We say that an edge (u, v) [image] E crosses the cut (S, V - S) if one of its endpoints is in S and the other is in V - S. We say that a cut respects the set A of edges if no edge in A crosses the cut. An edge is a light edge crossing a cut if its weight is the minimum of any edge crossing the cut. Note that there can be more than one light edge crossing a cut in the case of ties. More generally, we say that an edge is a light edge satisfying a given property if its weight is the minimum of any edge satisfying the property.

Figure 24.3 The proof of Theorem 24.1. The vertices in S are black, and the vertices in V - S are white. The edges in the minimum spanning tree T are shown, but the edges in the graph G are not. The edges in A are shaded, and (u, v) is a light edge crossing the cut (S, V - S). The edge (x, y) is an edge on the unique path p from u to v in T. A minimum spanning tree T' that contains (u, v) is formed by removing the edge (x, y) from T and adding the edge (u, v).

Our rule for recognizing safe edges is given by the following theorem.

Theorem 24.1

Let G = (V, E) be a connected, undirected graph with a real-valued weight function w defined on E. Let A be a subset of E that is included in some minimum spanning tree for G, let (S, V - S) be any cut of G that respects A, and let (u, v) be a light edge crossing (S, V - S). Then, edge (u, v) is safe for A.

Proof Let T be a minimum spanning tree that includes A, and assume that T does not contain the light edge (u, v), since if it does, we are done. We shall construct another minimum spanning tree T' that includes A [image] {(u, v)} by using a cut-and-paste technique, thereby showing that (u, v) is a safe edge for A.

The edge (u, v) forms a cycle with the edges on the path p from u to v in T, as illustrated in Figure 24.3. Since u and v are on opposite sides of the cut (S, V - S), there is at least one edge in T on the path p that also crosses the cut. Let (x, y) be any such edge. The edge (x, y) is not in A, because the cut respects A. Since (x, y) is on the unique path from u to v in T, removing (x, y) breaks T into two components. Adding (u, v) reconnects them to form a new spanning tree T' = T - {(x, y)} [image] {(u, v)}.

We next show that T' is a minimum spanning tree. Since (u, v) is a light edge crossing (S, V - S) and (x, y) also crosses this cut, w(u, v) [image] w(x, y). Therefore,

w(T')  = w  (T) - w(x, y) + w(u, v)

   w  (T) .

But T is a minimum spanning tree, so that w(T) [image] w(T'); thus, T' must be a minimum spanning tree also.

It remains to show that (u, v) is actually a safe edge for A. We have A [image] T', since A [image] T and (x, y) [image] A; thus, A [image] {(u, v)} [image] T'. Consequently, since T' is a minimum spanning tree, (u, v) is safe for A.

Theorem 24.1 gives us a better understanding of the workings of the GENERIC-MST algorithm on a connected graph G = (V, E). As the algorithm proceeds, the set A is always acyclic; otherwise, a minimum spanning tree including A would contain a cycle, which is a contradiction. At any point in the execution of the algorithm, the graph GA = (V, A) is a forest, and each of the connected components of GA is a tree. (Some of the trees may contain just one vertex, as is the case, for example, when the algorithm begins: A is empty and the forest contains |V| trees, one for each vertex.) Moreover, any safe edge (u, v) for A connects distinct components of GA, since A [image] {(u, v)} must be acyclic.

The loop in lines 2-4 of GENERIC-MST is executed |V| - 1 times as each of the |V| - 1 edges of a minimum spanning tree is successively determined. Initially, when [image], there are |V| trees in GA and each iteration reduces that number by 1. When the forest contains only a single tree, the algorithm terminates.

The two algorithms in Section 24.2 use the following corollary to Theorem 24.1.

Corollary 24.2

Let G = (V, E) be a connected, undirected graph with a real-valued weight function w defined on E. Let A be a subset of E that is included in some minimum spanning tree for G, and let C be a connected component (tree) in the forest GA = (V, A). If (u, v) is a light edge connecting C to some other component in GA, then (u, v) is safe for A.

Proof The cut (C, V - C) respects A, and (u, v) is therefore a light edge for this cut.

Exercises

24.1-1

Let (u, v) be a minimum-weight edge in a graph G. Show that (u, v) belongs to some minimum spanning tree of G.

24.1-2

Professor Sabatier conjectures the following converse of Theorem 24.1. Let G = (V, E) be a connected, undirected graph with a real-valued weight function w defined on E. Let A be a subset of E that is included in some minimum spanning tree for G, let (S, V - S) be any cut of G that respects A, and let (u, v) be a safe edge for A crossing (S, V - S). Then, (u, v) is a light edge for the cut. Show that the professor's conjecture is incorrect by giving a counterexample.

24.1-3

Show that if an edge (u, v) is contained in some minimum spanning tree, then it is a light edge crossing some cut of the graph.

24.1-4

Give a simple example of a graph such that the set of all edges that are light edges crossing some cut in the graph does not form a minimum spanning tree.

24.1-5

Let e be a maximum-weight edge on some cycle of G = (V, E). Prove that there is a minimum spanning tree of G' = (V, E - {e}) that is also a minimum spanning tree of G.

24.1-6

Show that a graph has a unique minimum spanning tree if, for every cut of the graph, there is a unique light edge crossing the cut. Show that the converse is not true by giving a counterexample.

24.1-7

Argue that if all of the edge weights of a graph are positive, then any subset of edges that connects all of the vertices and has minimum total weight must be a tree. Give an example to show that the same conclusion does not follow if we allow some weights to be nonpositive.

24.1-8

Let T be a minimum spanning tree of a graph G, and let L be the sorted list of the edge weights of T. Show that for any other minimum spanning tree T' of G, the list L is also the sorted list of edge weights of T'.

24.1-9

Let T be a minimum spanning tree of a graph G = (V, E), and let V' be a subset of V. Let T' be the subgraph of T induced by V', and let G' be the subgraph of G induced by V'. Show that if T' is connected, the T' is a minimum spanning tree of G'

24.2 The algorithms of Kruskal and Prim

The two minimum-spanning-tree algorithms described in this section are elaborations of the generic algorithm. They each use a specific rule to determine a safe edge in line 3 of GENERIC-MST. In Kruskal's algorithm, the set A is a forest. The safe edge added to A is always a least-weight edge in the graph that connects two distinct components. In Prim's algorithm, the set A forms a single tree. The safe edge added to A is always a least-weighted edge connecting the tree to a vertex not in the tree.

Kruskal's Al gorithm

Kruskal's algorithm is based directly on the generic minimum-spanning-tree algorithm given in Section 24.1. It finds a safe edge to add to the growing forest by finding, of all the edges that connect any two trees in the forest, an edge (u, v) of least weight. Let C1 and C2 denote the two trees that are connected by (u, v). Since (u,v) must be a light edge connecting C1 to some other tree, Corollary 24.2 implies that (u, v) is a safe edge for C1. Kruskal's algorithm is a greedy algorithm, because at each step it adds to the forest an edge of least possible weight.

Our implementation of Kruskal's algorithm is like the algorithm to compute connected components from Section 22.1. It uses a disjoint-set data structure to maintain several disjoint sets of elements. Each set contains the vertices in a tree of the current forest. The operation FIND-SET(u) returns a representative element from the set that contains u. Thus, we can determine whether two vertices u and v belong to the same tree by testing whether FIND-SET(u) equals FIND-SET(v). The combining of trees is accomplished by the UNION procedure.

MST-KRUSKAL(G, w)

1  A

2  for each vertex v    V[G]

3      do MAKE-SET (v)

4  sort the edges of E by nondecreasing weight w

5  for each edge (u, v)    E, in order by nondecreasing weight

6      do if FIND-SET(u)    FIND-SET(v)

7            then A    A    {(u, v)}

8                 UNION (u, v)

9  return A

Kruskal's algorithm works as shown in Figure 24.4. Lines 1-3 initialize the set A to the empty set and create |V| trees, one containing each vertex. The edges in E are sorted into order by nondecreasing weight in line 4. The for loop in lines 5-8 checks, for each edge (u, v), whether the endpoints u and v belong to the same tree. If they do, then the edge (u, v) cannot be added to the forest without creating a cycle, and the edge is discarded. Otherwise, the two vertices belong to different trees, and the edge (u, v) is added to A in line 7, and the vertices in the two trees are merged in line 8.

The running time of Kruskal's algorithm for a graph G = (V, E) depends on the implementation of the disjoint-set data structure. We shall assume the disjoint-set-forest implementation of Section 22.3 with the union-by-rank and path-compression heuristics, since it is the asymptotically fastest implementation known. Initialization takes time O(V), and the time to sort the edges in line 4 is O(E lg E). There are O(E) operations on the disjoint-set forest, which in total take O(E [image] (E, V)) time, where [image] is the functional inverse of Ackermann's function defined in Section 22.4. Since [image] (E, V) = O(lg E), the total running time of Kruskal's algorithm is O(E lg E).

Prim's algorithm

Like Kruskal's algorithm, Prim's algorithm is a special case of the generic minimum-spanning-tree algorithm from Section 24.1. Prim's algorithm operates much like Dijkstra's algorithm for finding shortest paths in a graph. (See Section 25.2.) Prim's algorithm has the property that the edges in the set A always form a single tree. As is illustrated in Figure 24.5, the tree starts from an arbitrary root vertex r and grows until the tree spans all the vertices in V. At each step, a light edge connecting a vertex in A to a vertex in V - A is added to the tree. By Corollary 24.2, this rule adds only edges that are safe for A; therefore, when the algorithm terminates, the edges in A form a minimum spanning tree. This strategy is "greedy" since the tree is augmented at each step with an edge that contributes the minimum amount possible to the tree's weight.

Figure 24.4 The execution of Kruskal's algorithm on the graph from Figure 24.1. Shaded edges belong to the forest A being grown. The edges are considered by the algorithm in sorted order by weight. An arrow points to the edge under consideration at each step of the algorithm. If the edge joins two distinct trees in the forest, it is added to the forest, thereby merging the two trees.

The key to implementing Prim's algorithm efficiently is to make it easy to select a new edge to be added to the tree formed by the edges in A. In the pseudocode below, the connected graph G and the root r of the minimum spanning tree to be grown are inputs to the algorithm. During execution of the algorithm, all vertices that are not in the tree reside in a priority queue Q based on a key field. For each vertex v, key[v] is the minimum weight of any edge connecting v to a vertex in the tree; by convention, key[v] = [image] if there is no such edge. The field [image] [v] names the "parent" of v in the tree. During the algorithm, the set A from GENERIC-MST is kept implicitly as

A = {(v,   [v]) : v    V - {r} - Q} .

When the algorithm terminates, the priority queue Q is empty; the minimum spanning tree A for G is thus

A = {(v,   [v]) : v    V - {r}} .

Figure 24.5 The execution of Prim's algorithm on the graph from Figure 24.1. The root vertex is a. Shaded edges are in the tree being grown, and the vertices in the tree are shown in black. At each step of the algorithm, the vertices in the tree determine a cut of the graph, and a light edge crossing the cut is added to the tree. In the second step, for example, the algorithm has a choice of adding either edge (b, c) or edge (a, h) to the tree since both are light edges crossing the cut.

MST-PRIM(G, w, r)

1  Q    V[G]

2  for each u    Q

3      do key[u]

4  key [r]    0

5    [r]    NIL

6  while Q

7      do u    EXTRACT-MIN(Q)

8         for each v    Adj[u]

9             do if v    Q and w (u, v) < key[v]

10                  then   [v]    u

11                       key[v]    w(u, v)

Prim's algorithm works as shown in Figure 24.5. Lines 1-4 initialize the priority queue Q to contain all the vertices and set the key of each vertex to [image] , except for the root r, whose key is set to 0. Line 5 initializes [image] [r] to NIL, since the root r has no parent. Throughout the algorithm, the set V - Q contains the vertices in the tree being grown. Line 7 identifies a vertex u [image] Q incident on a light edge crossing the cut (V - Q, Q) (with the exception of the first iteration, in which u = r due to line 4). Removing u from the set Q adds it to the set V - Q of vertices in the tree. Lines 8-11 update the key and [image] fields of every vertex v adjacent to u but not in the tree. The updating maintains the invariants that key[v] = w(v, [image] [v]) and that (v, [image] [v]) is a light edge connecting v to some vertex in the tree.

The performance of Prim's algorithm depends on how we implement the priority queue Q. If Q is implemented as a binary heap (see Chapter 7), we can use the BUILD-HEAP procedure to perform the initialization in lines 1-4 in O(V) time. The loop is executed |V| times, and since each EXTRACT-MIN operation takes O(lg V) time, the total time for all calls to EXTRACT-MIN is O(V 1g V). The for loop in lines 8-11 is executed O(E) times altogether, since the sum of the lengths of all adjacency lists is 2 |E|. Within the for loop, the test for membership in Q in line 9 can be implemented in constant time by keeping a bit for each vertex that tells whether or not it is in Q, and updating the bit when the vertex is removed from Q. The assignment in line 11 involves an implicit DECREASE-KEY operation on the heap, which can be implemented in a binary heap in O(lg V) time. Thus, the total time for Prim's algorithm is O(V 1g V + E 1g V) = O(E lg V), which is asymptotically the same as for our implementation of Kruskal's algorithm.

The asymptotic running time of Prim's algorithm can be improved, however, by using Fibonacci heaps. Chapter 21 shows that if |V| elements are organized into a Fibonacci heap, we can perform an EXTRACT-MIN operation in O(lg V) amortized time and a DECREASE-KEY operation (to implement line 11) in O(1) amortized time. Therefore, if we use a Fibonacci heap to implement the priority queue Q, the running time of Prim's algorithm improves to O(E + V 1g V).

Exercises

24.2-1

Kruskal's algorithm can return different spanning trees for the same input graph G, depending on how ties are broken when the edges are sorted into order. Show that for each minimum spanning tree T of G, there is a way to sort the edges of G in Kruskal's algorithm so that the algorithm returns T.

24.2-2

Suppose that the graph G = (V, E) is represented as an adjacency matrix. Give a simple implementation of Prim's algorithm for this case that runs in O(V2) time.

24.2-3

Is the Fibonacci-heap implementation of Prim's algorithm asymptotically faster than the binary-heap implementation for a sparse graph G = (V, E), where |E| = [image] (V)? What about for a dense graph, where |E| = [image] (V2)? How must |E| and |V| be related for the Fibonacci-heap implementation to be asymptotically faster than the binary-heap implementation?

24.2-4

Suppose that all edge weights in a graph are integers in the range from 1 to |V|. How fast can you make Kruskal's algorithm run? What if the edge weights are integers in the range from 1 to W for some constant W?

24.2-5

Suppose that all edge weights in a graph are integers in the range from 1 to |V|. How fast can you make Prim's algorithm run? What if the edge weights are integers in the range from 1 to W for some constant W?

24.2-6

Describe an efficient algorithm that, given an undirected graph G, determines a spanning tree of G whose largest edge weight is minimum over all spanning trees of G.

24.2-7

Suppose that the edge weights in a graph are uniformly distributed over the half-open interval [0, 1). Which algorithm, Kruskal's or Prim's, can you make run faster?

24.2-8

Suppose that a graph G has a minimum spanning tree already computed. How quickly can the minimum spanning tree be updated if a new vertex and incident edges are added to G?

Problems

24-1 Second-best minimum spanning tree

Let G = (V, E) be an undirected, connected graph with weight function w : E [image] R, and suppose that |E| [image] |V| .

a. Let T be a minimum spanning tree of G. Prove that there exist edges (u, v) [image] T and (x, y) [image] T such that T -{(u, v)} [image] {(x, y)} is a second-best minimum spanning tree of G.

b. Let T be a spanning tree of G and, for any two vertices u, v [image] V, let max[u, v] be an edge of maximum weight on the unique path between u and v in T. Describe an O(V2)-time algorithm that, given T, computes max[u, v] for all u, v [image] V.

c. Give an efficient algorithm to compute the second-best minimum spanning tree of G.

24-2 Minimum spanning tree in sparse graphs

For a very sparse connected graph G = (V, E), we can improve upon the O(E + V lg V) running time of Prim's algorithm with Fibonacci heaps by "preprocessing" G to decrease the number of vertices before running Prim's algorithm. The following procedure takes as input a weighted graph G and returns a "contracted" version of G, having added some edges to the minimum spanning tree T under construction. Initially, for each edge (u, v) [image] E, we assume that orig[u, v] = (u, v) and that w[u, v] is the weight of the edge.

MST-REDUCE(G, T)

1  for each v    V[G]

2      do mark[v]    FALSE

3         MAKE-SET(v)

4  for each u    V[G]

5      do if mark[u] = FALSE

6            then choose v    Adj[u] such that w[u, v] is minimized

7                 UNION(u,v)

8                 T    T    {orig[u, v]}

9                 mark[u]    mark[v]    TRUE

10  V[G']    {FIND-SET(v) : v    V[G]}

11  E[G']

12  for each (x, y)    E[G]

13      do u    FIND-SET(x)

14         v    FIND-SET(y)

15         if (u, v)    E[G']

16            then E[G']    E[G']    {(u, v)}

17                 orig[u, v]    orig[x, y]

18                 w[u, v]    w[x, y]

19            else if w[x, y] < w[u, v]

20                    then orig[u, v]   orig[x, y]

21                         w[u, v]    w[x, y]

22  construct adjacency lists Adj for G'

23  return G' and T

a. Let T be the set of edges returned by MST-REDUCE, and let T' be a minimum spanning tree of the graph G' returned by the procedure. Prove that T [image] {orig[x, y] : (x, y) [image] T'} is a minimum spanning tree of G.

b. Argue that |V[G']| [image] |V| /2.

c. Show how to implement MST-REDUCE so that it runs in O(E) time. (Hint: Use simple data structures.)

d. Suppose that we run k phases of MST-REDUCE, using the graph produced by one phase as input to the next and accumulating edges in T. Argue that the overall running time of the k phases is O(kE).

e. Suppose that after running k phases of MST-REDUCE, we run Prim's algorithm on the graph returned by the last phase. Show how to pick k so that the overall running time is O(E lg lg V). Argue that your choice of k minimizes the overall asymptotic running time.

f. For what values of |E| (in terms of |V|) does Prim's algorithm with preprocessing asymptotically beat Prim's algorithm without preprocessing?

Chapter notes

Tarjan [188] surveys the minimum-spanning-tree problem and provides excellent advanced material. A history of the minimum-spanning-tree problem has been written by Graham and Hell [92].

Tarjan attributes the first minimum-spanning-tree algorithm to a 1926 paper by O. Boruvka. Kruskal's algorithm was reported by Kruskal [131] in 1956. The algorithm commonly known as Prim's algorithm was indeed invented by Prim [163], but it was also invented earlier by V. Jarník in 1930.

The reason why greedy algorithms are effective at finding minimum spanning trees is that the set of forests of a graph forms a graphic matroid. (See Section 17.4.)

The fastest minimum-spanning-tree algorithm to date for the case in which |E| = [image] (V lg V) is Prim's algorithm implemented with Fibonacci heaps. For sparser graphs, Fredman and Tarjan [75] give an algorithm that runs in O(E [image] (|E|, |V|)) time, where [image] (|E|, |V|) = min{i: lg(i) |V| [image] |E| / |V|}. The fact that |E| [image] |V| implies that their algorithm runs in time O(E lg* V).

Go to Chapter 25 Back to Table of Contents


Intro to Algorithms: CHAPTER 25: SINGLE-SOURCE SHORTEST PATHS

 Return to Table of Contents

CHAPTER 25: SINGLE-SOURCE SHORTEST PATHS

A motorist wishes to find the shortest possible route from Chicago to Boston. Given a road map of the United States on which the distance between each pair of adjacent intersections is marked, how can we determine this shortest route?

One possible way is to enumerate all the routes from Chicago to Boston, add up the distances on each route, and select the shortest. It is easy to see, however, that even if we disallow routes that contain cycles, there are millions of possibilities, most of which are simply not worth considering. For example, a route from Chicago to Houston to Boston is obviously a poor choice, because Houston is about a thousand miles out of the way.

In this chapter and in Chapter 26, we show how to solve such problems efficiently. In a shortest-paths problem, we are given a weighted, directed graph G = (V, E), with weight function w : E [image] R mapping edges to real-valued weights. The weight of path p = [image] v0, v1, . . . , [image] k [image] is the sum of the weights of its constituent edges:

We define the shortest-path weight from u to v by

if there is a path from u to v, otherwise.

A shortest path from vertex u to vertex v is then defined as any path p with weight w (p) = [image] (u, v).

In the Chicago-to-Boston example, we can model the road map as a graph: vertices represent intersections, edges represent road segments between intersections, and edge weights represent road distances. Our goal is to find a shortest path from a given intersection in Chicago (say, Clark St. and Addison Ave.) to a given intersection in Boston (say, Brookline Ave. and Yawkey Way).

Edge weights can be interpreted as metrics other than distances. They are often used to represent time, cost, penalties, lossage, or any other quantity that accumulates linearly along a path and that one wishes to minimize.

The breadth-first-search algorithm from Section 23.2 is a shortest-paths algorithm that works on unweighted graphs, that is, graphs in which each edge can be considered to have unit weight. Because many of the concepts from breadth-first search arise in the study of shortest paths in weighted graphs, the reader is encouraged to review Section 23.2 before proceeding.

Variants

In this chapter, we shall focus on the single-source shortest-paths problem: given a graph G = (V, E), we want to find a shortest path from a given source vertex s [image] V to every vertex v [image] V. Many other problems can be solved by the algorithm for the single-source problem, including the following variants.

Single-destination shortest-paths problem: Find a shortest path to a given destination vertex t from every vertex v. By reversing the direction of each edge in the graph, we can reduce this problem to a single-source problem.

Single-pair shortest-path problem: Find a shortest path from u to v for given vertices u and v. If we solve the single-source problem with source vertex u, we solve this problem also. Moreover, no algorithms for this problem are known that run asymptotically faster than the best single-source algorithms in the worst case.

All-pairs shortest-paths problem: Find a shortest path from u to v for every pair of vertices u and v. This problem can be solved by running a single-source algorithm once from each vertex; but it can usually be solved faster, and its structure is of interest in its own right. Chapter 26 addresses the all-pairs problem in detail.

Negative-weight edges

In some instances of the single-source shortest-paths problem, there may be edges whose weights are negative. If the graph G = (V, E) contains no negative-weight cycles reachable from the source s, then for all v [image] V, the shortest-path weight [image] (s, v) remains well defined, even if it has a negative value. If there is a negative-weight cycle reachable from s, however, shortest-path weights are not well defined. No path from s to a vertex on the cycle can be a shortest path--a lesser-weight path can always be found that follows the proposed "shortest" path and then traverses the negative-weight cycle. If there is a negative-weight cycle on some path from s to v, we define [image] (s,v) = - [image].

Figure 25.1 illustrates the effect of negative weights on shortest-path weights. Because there is only one path from s to a (the path [image] s, a [image]), [image] (s, a = w(s, a) = 3. Similarly, there is only one path from s to b, and so [image] (s, b) = w(s, a) + w(a, b) = 3 + (-4) = - 1. There are infinitely many paths from s to c: s, c [image], [image] s, c, d, c [image], [image] s, c, d, c, d, c [image], and so on. Because the cycle [image] c, d, c [image] has weight 6 + (-3) = 3 > 0, the shortest path from s to c is [image] s, c [image], with weight [image] (s, c) = 5. Similarly, the shortest path from s to d is [image] s, c, d [image], with weight [image] (s, d) = w(s, c) + w(c, d) = 11. Analogously, there are infinitely many paths from s to e: s, e [image], [image] s, e, f, e [image], [image] s, e, f, e, f, e [image], and so on. Since the cycle [image] e, f, e [image] has weight 3 + (-6) = -3 < 0, however, there is no shortest path from s to e. By traversing the negative-weight cycle [image] e, f, e [image] arbitrarily many times, we can find paths from s to e with arbitrarily large negative weights, and so [image] (s, e) = - [image]. Similarly, [image] (s, f) - [image]. Because g is reachable from f, we can also find paths with arbitrarily large negative weights from s to g, and [image] (s, g) = - [image]. Vertices h, i, and j also form a negative-weight cycle. They are not reachable from s, however, and so [image] (s, h) = [image] (s, i) = [image] (s, j) = [image].

Figure 25.1 Negative edge weights in a directed graph. Shown within each vertex is its shortest-path weight from source s. Because vertices e and f form a negative-weight cycle reachable from s, they have shortest-path weights of - . Because vertex g is reachable from a vertex whose shortest-path weight is - , it, too, has a shortest-path weight of - . Vertices such as h, i, and j are not reachable from s, and so their shortest-path weights are , even though they lie on a negative-weight cycle.

Some shortest-paths algorithms, such as Dijkstra's algorithm, assume that all edge weights in the input graph are nonnegative, as in the road-map example. Others, such as the Bellman-Ford algorithm, allow negative-weight edges in the input graph and produce a correct answer as long as no negative-weight cycles are reachable from the source. Typically, if there is such a negative-weight cycle, the algorithm can detect and report its existence.

Representing shortest paths

We often wish to compute not only shortest-path weights, but the vertices on the shortest paths as well. The representation we use for shortest paths is similar to the one we used for breadth-first trees in Section 23.2. Given a graph G = (V, E), we maintain for each vertex v [image] V a predecessor [image] [v] that is either another vertex or NIL. The shortest-paths algorithms in this chapter set the [image] attributes so that the chain of predecessors originating at a vertex v runs backwards along a shortest path from s to v. Thus, given a vertex v for which [image] [v] [image] NIL, the procedure PRINT-PATH (G, s, v) from Section 23.2 can be used to print a shortest path from s to v.

During the execution of a shortest-paths algorithm, however, the [image] values need not indicate shortest paths. As in breadth-first search, we shall be interested in the predecessor subgraph G [image] = (V [image] , E [image] ) induced by the [image] values. Here again, we define the vertex set V [image] , to be the set of vertices of G with non-NIL predecessors, plus the source s:

V   = {v    V :   [v]    NIL}    {s} .

The directed edge set E [image] is the set of edges induced by the [image] values for vertices in V [image] :

E   = {(  [v], v)    E : v    V   - {  }}.

We shall prove that the [image] values produced by the algorithms in this chapter have the property that at termination G [image] is a "shortest-paths tree"--informally, a rooted tree containing a shortest path from a source s to every vertex that is reachable from s. A shortest-paths tree is like the breadth-first tree from Section 23.2, but it contains shortest paths from the source defined in terms of edge weights instead of numbers of edges. To be precise, let G = (V, E) be a weighted, directed graph with weight function w : E [image] R, and assume that G contains no negative-weight cycles reachable from the source vertex s [image] V, so that shortest paths are well defined. A shortest-paths tree rooted at s is a directed subgraph G' = ( V', E'), where V' [image] V and E' [image] E, such that

1. V' is the set of vertices reachable from s in G,

2. G' forms a rooted tree with root s, and

3. for all v [image] V', the unique simple path from s to v in G' is a shortest path from s to v in G.

Shortest paths are not necessarily unique, and neither are shortest-paths trees. For example, Figure 25.2 shows a weighted, directed graph and two shortest-paths trees with the same root.

Chapter outline

The single-source shortest-paths algorithms in this chapter are all based on a technique known as relaxation. Section 25.1 begins by proving some important properties of shortest paths in general and then proves some important facts about relaxation-based algorithms. Dijkstra's algorithm, which solves the single-source shortest-paths problem when all edges have nonnegative weight, is given in Section 25.2. Section 25.3 presents the Bellman-Ford algorithm, which is used in the more general case in which edges can have negative weight. If the graph contains a negative-weight cycle reachable from the source, the Bellman-Ford algorithm detects its presence. Section 25.4 gives a linear-time algorithm for computing shortest paths from a single source in directed acyclic graphs. Finally, Section 25.5 shows how the Bellman-Ford algorithm can be used to solve a special case of "linear programming."

Figure 25.2 (a) A weighted, directed graph with shortest-path weights from source s. (b) The shaded edges form a shortest-paths tree rooted at the source s. (c) Another shortest-paths tree with the same root.

Our analysis will require some conventions for doing arithmetic with infinities. We shall assume that for any real number a [image] - [image], we have a + [image] = [image] + a = [image]. Also, to make our proofs hold in the presence of negative-weight cycles, we shall assume that for any real number a [image] [image], we have a + (- [image]) = (- [image]) + a = - [image].

25.1 Shortest paths and relaxation

To understand single-source shortest-paths algorithms, it is helpful to understand the techniques that they use and the properties of shortest paths that they exploit. The main technique used by the algorithms in this chapter is relaxation, a method that repeatedly decreases an upper bound on the actual shortest-path weight of each vertex until the upper bound equals the shortest-path weight. In this section, we shall see how relaxation works and formally prove several properties it maintains.

On a first reading of this section, you may wish to omit proofs of theorems--reading only their statements--and then proceed immediately to the algorithms in Sections 25.2 and 25.3. Pay particular attention, however, to Lemma 25.7, which is a key to understanding the shortest-paths algorithms in this chapter. On a first reading, you may also wish to ignore completely the lemmas concerning predecessor subgraphs and shortest-paths trees (Lemmas 25.8 and 25.9), concentrating instead on the earlier lemmas, which pertain to shortest-path weights.

Optimal substructure of a shortest path

Shortest-paths algorithms typically exploit the property that a shortest path between two vertices contains other shortest paths within it. This optimal-substructure property is a hallmark of the applicability of both dynamic programming (Chapter 16) and the greedy method (Chapter 17). In fact, Dijkstra's algorithm is a greedy algorithm, and the Floyd-Warshall algorithm, which finds shortest paths between all pairs of vertices (see Chapter 26), is a dynamic-programming algorithm. The following lemma and its corollary state the optimal-substructure property of shortest paths more precisely.

Lemma 25.1

Given a weighted, directed graph G = (V, E) with weight function w: E [image] R, let p = [image] v1, v2, . . . , vk [image] be a shortest path from vertex v1, to vertex vk and, for any i and j such that 1 < i < j < k, let pij = [image] vi, vi+1, . . . , vi [image] be the subpath of p from vertex vi, to vertex vj. Then, pij is a shortest path from vi to vj.

Proof If we decompose path p into [image] then w(p) = w(p1i) + w(pij) + w(pjk). Now, assume that there is a path p'ij from vi to vj with weight w(p'ij) < w(pij). Then, [image] is a path from v1 to vk whose weight w(p1i) + w(p'ij) + w(pjk) is less than w(p), which contradicts the premise that p is a shortest path from v1 to vk..

In studying breadth-first search (Section 23.2), we proved as Lemma 23.1 a simple property of shortest distances in unweighted graphs. The following corollary to Lemma 25.1 generalizes the property to weighted graphs.

Corollary 25.2

Let G = (V, E) be a weighted, directed graph with weight function w : E [image] R. Suppose that a shortest path p from a source s to a vertex v can be decomposed into [image] for some vertex u and path p'. Then, the weight of a shortest path from s to v is [image] (s, v) = [image] (s, u) + w(u, v).

Proof By Lemma 25.1, subpath p' is a shortest path from source s to vertex u. Thus,

  (s, v)   =  w(p)

=  w(p') + w(u, v)

=    (s, u) + w(u, v).

The next lemma gives a simple but useful property of shortest-path weights.

Lemma 25.3

Let G = (V, E) be a weighted, directed graph G = (V, E) with weight function w: E [image] R and source vertex s. 0Then, for all edges (u, v) [image] E, we have [image] (s, v) < [image] (s, u) + w(u, v).

Proof A shortest path p from source s to vertex v has no more weight than any other path from s to v. Specifically, path p has no more weight than the particular path that takes a shortest path from source s to vertex u and then takes edge (u, v).

Relaxation

The algorithms in this chapter use the technique of relaxation. For each vertex v [image] V, we maintain an attribute d[v], which is an upper bound on the weight of a shortest path from source s to v. We call d[v] a shortest-path estimate. We initialize the shortest-path estimates and predecessors by the following procedure.

INITIALIZE-SINGLE-SOURCE(G,s)

1  for each vertex v    V[G]

2       do d[v]

3            [v]    NIL

4  d[s]    0

After initialization, [image] [v] = NIL for all v [image] V, d[v] = 0 for v = s, and d[v] = [image] for v [image] V - {s}.

The process of relaxing1 an edge (u, v) consists of testing whether we can improve the shortest path to v found so far by going through u and, if so, updating d[v] and [image] [v]. A relaxation step may decrease the value of the shortest-path estimate d[v] and update v's predecessor field [image] [v]. The following code performs a relaxation step on edge (u, v).

1It may seem strange that the term "relaxation" is used for an operation that tightens an upper bound. The use of the term is historical. The outcome of a relaxation step can be viewed as a relaxation of the constraint d[v] [image] d[u] + w(u, v), which, by Lemma 25.3, must be satisfied if d[u] = [image] (s, u) and d[v] = [image] (s, v). That is, if d[v] [image] d[u] + w(u, v), there is no "pressure" to satisfy this constraint, so the constraint is "relaxed."

RELAX(u, v, w)

1 if d[v] > d[u] + w(u,v)

2    then d[v]    d[u] + w(u,v)

3           [v]    u

Figure 25.3 shows two examples of relaxing an edge, one in which a shortest-path estimate decreases and one in which no estimate changes.

Figure 25.3 Relaxation of an edge (u, v). The shortest-path estimate of each vertex is shown within the vertex. (a) Because d[v] > d[u] + w(u, v) prior to relaxation, the value of d[v] decreases. (b) Here, d[v] d[u] + w(u, v) before the relaxation step, so d[v] is unchanged by relaxation.

Each algorithm in this chapter calls INITIALIZE-SINGLE-SOURCE and then repeatedly relaxes edges. Moreover, relaxation is the only means by which shortest-path estimates and predecessors change. The algorithms in this chapter differ in how many times they relax each edge and the order in which they relax edges. In Dijkstra's algorithm and the shortest-paths algorithm for directed acyclic graphs, each edge is relaxed exactly once. In the Bellman-Ford algorithm, each edge is relaxed several times.

Properties of relaxation

The correctness of the algorithms in this chapter depends on important properties of relaxation that are summarized in the next few lemmas. Most of the lemmas describe the outcome of executing a sequence of relaxation steps on the edges of a weighted, directed graph that has been initialized by INITIALIZE-SINGLE-SOURCE. Except for Lemma 25.9, these lemmas apply to any sequence of relaxation steps, not just those that produce shortest-path values.

Lemma 25.4

Let G = (V, E) be a weighted, directed graph with weight function w : E [image] R, and let (u, v) [image] E. Then, immediately after relaxing edge (u, v) by executing RELAX(u, v, w), we have d[v] [image] d[u] + w(u, v).

Proof If, just prior to relaxing edge (u, v), we have d[v] > d[u] + w(u, v), then d[v] = d[u] + w(u, v) afterward. If, instead, d[v] [image] d[u] + w(u, v) just before the relaxation, the neither d[u] nor d[v] changes, and so d[v] [image] d[u] + w(u, v) afterward.

Lemma 25.5

Let G = (V, E) be a weighted, directed graph with weight function w : E [image] R. Let s [image] V be the source vertex, and let the graph be initialized by INITIALIZE-SINGLE-SOURCE(G, s). Then, d[v] [image] [image] (s, v) for all v [image] V, and this invariant is maintained over any sequence of relaxation steps on the edges of G. Moreover, once d[v] achieves its lower bound [image] (s, v), it never changes.

Proof The invariant d[v] [image] [image] (s, v) is certainly true after initialization, since d[s] = 0 [image] [image] (s, s) (note that [image] (s, s) is - [image] if s is on a negative-weight cycle and 0 otherwise) and d[v] = [image] implies d[v] [image] [image] (s, v) for all v [image] V - {s}. We shall use proof by contradiction to show that the invariant is maintained over any sequence of relaxation steps. Let v be the first vertex for which a relaxation step of an edge (u, v) causes d[v] < [image] (s, v). Then, just after relaxing edge (u, v), we have

d[u] + w(u, v)  =  d[v]

<    (s,v)

      (s,u) + w(u, v)  (by Lemma 25.3),

which implies that d[u] < [image] (s, u). But because relaxing edge (u, v) does not change d[u], this inequality must have been true just before we relaxed the edge, which contradicts the choice of v as the first vertex for which d[v] < [image] (s, v). We conclude that the invariant d[v] [image] [image] (s, v) is maintained for all v [image] V.

To see that the value of d[v] never changes once d[v] = [image] (s, v), note that having achieved its lower bound, d[v] cannot decrease because we have just shown that d[v] [image] [image] (s, v), and it cannot increase because relaxation steps do not increase d values.

Corollary 25.6

Suppose that in a weighted, directed graph G = (V, E) with weight function w : E [image] R, no path connects a source vertex s [image] V to a given vertex v [image] V. Then, after the graph is initialized by INITIALIZE-SINGLE-SOURCE(G, s), we have d[v] = [image] (s, v), and this equality is maintained as an invariant over any sequence of relaxation steps on the edges of G.

Proof By Lemma 25.5, we always have [image] = [image] (s, v) [image] d[v]; thus, so d[v] = [image] = [image] (s, v).

The following lemma is crucial to proving the correctness of the shortest-paths algorithms that appear later in this chapter. It gives sufficient conditions for relaxation to cause a shortest-path estimate to converge to a shortest-parth weight.

Lemma 25.7

Let G = (V, E) be a weighted, directed graph with weight function w : E [image] R, let s [image] V be a source vertex, and let [image] be a shortest path in G for some vertices u, v [image] V. Suppose that G is initialized by INITIALIZE-SINGLE-SOURCE(G, s) and then a sequence of relaxation steps that includes the call RELAX(u, v, w) is executed on the edges of G. If d[u] = [image] (s, u) at any time prior to the call, then d[v] = [image] (s, v) at all times after the call.

Proof By Lemma 25.5, if d[u] = [image] (s, u) at some point prior to relaxing edge (u, v), then this equality holds thereafter. In particular, after relaxing edge (u, v) we have

d[v]      d[u] + w(u,v)    (by Lemma 25.4)

=    (s,u) + w(u,v)

=    (s,v)           (by Corollary 25.2).

By Lemma 25.5, [image] (s, v) bounds d[v] from below, from which we conclude that d[v] = [image] (s, v), and this equality is maintained thereafter.

Shortest-paths trees

So far, we have shown that relaxation causes the shortest-path estimates to descend monotonically toward the actual shortest-path weights. We would also like to show that once a sequence of relaxations has computed the actual shortest-path weights, the predecessor subgraph G [image] induced by the resulting [image] values is a shortest-paths ree for G . e start with the following lemma, which shows that the predecessor subgraph always forms a rooted tree whose root is the source.

Lemma 25.8

Let G = (V, E) be a weighted, directed graph with weight function w : E [image] R and source vertex s [image] V, and assume that G contains no negative-weight cycles that are reachable from s. Then, after the graph is initialized by INITIALIZE-SINGLE-SOURCE(G, s), the predecessor subgraph G [image] forms a rooted tree with root s, and any sequence of relaxation steps on edges of G maintains this property as an invariant.

Proof Initially, the only vertex in G [image] is the source vertex, and the lemma is trivially true. Consider a predecessor subgraph G [image] that arises after a sequence of relaxation steps. We shall first prove that G [image] is acyclic. Suppose for the sake of contradiction that some relaxation step creates a cycle in the graph G [image] . Let the cycle be c = [image] v0, v1, . . . , vk [image], where vk = v0. Then, [image] [vi] = vi - 1 for i = 1, 2, . . ., k and, without loss of generality, we can assume that it was the relaxation of edge (vk - 1, vk) that created the cycle in G [image].

We claim that all vertices on cycle c are reachable from the source s. Why? Each vertex on c has a non-NIL predecessor, and so each vertex on c was assigned a finite shortest-path estimate when it was assigned its non-NIL [image] value. By Lemma 25.5, each vertex on cycle c has a finite shortest-path weight, which implies that it is reachable from s.

We shall examine the shortest-path estimates on c just prior to the call RELAX (vk - 1, vk, w) and show that c is a negative-weight cycle, thereby contradicting the assumption that G contains no negative-weight cycles that are reachable from the source. Just before the call, we have [image] [vi] = vi-1 for i = 1, 2, . . . , k - 1. Thus, for i = 1, 2, . . . , k - 1, the last update to d[vi] was by the assignment d[vi] [image] d[vi - 1] + w(vi,vi - 1). If d[vi - 1] changed since then, it decreased. Therefore, just before the call RELAX (vk - 1, vk, w), we have

d[vi]    d[vi - 1]+w(vi - 1,vi)  for all i = 1, 2,...,k - 1 .

(25.1)

Because [image] [vk] is changed by the call, immediately beforehand we also have the strict inequality

d[vk] > d[vk - 1] + w(vk - 1, vk) .

Summing this strict inequality with the k - 1 inequalities (25.1), we obtain the sum of the shortest-path estimates around cycle c:

since each vertex in the cycle c appears exactly once in each summation. This implies

Thus, the sum of weights around the cycle c is negative, thereby providing the desired contradiction.

We have now proved that G [image] is a directed, acyclic graph. To show that it forms a rooted tree with root s, it sufiices (see Exercise 5.5-3) to prove that for each vertex v [image] V [image] , there is a unique path from s to v in G [image] .

We first must show that a path from s exists for each vertex in V [image]. The vertices in V [image] are those with non-NIL [image] values, plus s. The idea here is to prove by induction that a path exists from s to all vertices in V [image]. The details are left as Exercise 25.1-6.

To complete the proof of the lemma, we must now show that for any vertex v [image] V [image] , there is at most one path from s to v in the graph G [image] . Suppose otherwise. That is, suppose that there are two simple paths from s to some vertex v: p1, which can be decomposed into [image] and p2, which can be decomposed into [image], where x [image] y. (See Figure 25.4.) But then, [image] [z] = x and [image] [z] = y, which implies the contradiction that x = y. We conclude that there exists a unique simple path in G [image] from s to v, and thus G [image] forms a rooted tree with root s.

Figure 25.4 Showing that a path in G from source s to vertex v is unique. If there are two paths and , where x y, then [z] = x and [z] = y, a contradiction.

We can now show that if, after we have performed a sequence of relaxation steps, all vertices have been assigned their true shortest-path weights, then the predecessor subgraph G [image] is a shortest-paths tree.

Lemma 25.9

Let G = (V, E) be a weighted, directed graph with weight function w : E [image] R and source vertex s [image] V, and assume that G contains no negative-weight cycles that are reachable from s. Let us call INITIALIZE-SINGLE-SOURCE(G, s) and then execute any sequence of relaxation steps on edges of G that produces d[v] = [image] (s, v) for all v [image] V. Then, the predecessor subgraph G [image] is a shortest-paths tree rooted at s.

Proof We must prove that the three properties of shortest-paths trees hold for G [image] . To show the first property, we must show that V [image] is the set of vertices reachable from s. By definition, a shortest-path weight [image] (s, v) is finite if and only if v is reachable from s, and thus the vertices that are reachable from s are exactly those with finite d values. But a vertex v [image] V - {s} has been assigned a finite value for d[v] if and only if [image] [v] [image] NIL. Thus, the vertices in V [image] are exactly those reachable from s.

The second property follows directly from Lemma 25.8.

It remains, therefore, to prove the last property of shortest-paths trees: for all v V , the unique simple path in G is a shortest path from s to v in G. Let p = v0, v1, . . . , vk , where v0 = s and vk = v. For i = l, 2, . . . , k, we have both d[vi] = (s, vi) and d[vi] d[vi - 1] + w(vi - 1, vi), from which we conclude w(vi-1, vi) (s, vi) - (s, vi - 1). Summing the weights along path p yields

=    (s, vk) -   (s, v0)

=    (s, vk).

The third line comes from the telescoping sum on the second line, and the fourth line follows from (s, v0) = (s, s) = 0. Thus, w(p) (s, vk). Since (s, vk) is a lower bound on the weight of any path from s to vk, we conclude that w(p) = (s, vk), and thus p is a shortest path from s to v = vk.

Exercises

25.1-1

Give two shortest-paths trees for the directed graph of Figure 25.2 other than the two shown.

25.1-2

Give an example of a weighted, directed graph G = (V, E) with weight function w: E R and source s such that G satisfies the following property: For every edge (u, v) E, there is a shortest-paths tree rooted at s that contains (u, v) and another shortest-paths tree rooted at s that does not contain (u, v).

25.1-3

Embellish the proof of Lemma 25.3 to handle cases in which shortest-path weights are or - .

25.1-4

Let G = (V, E) be a weighted, directed graph with source vertex s, and let G be initialized by INITIALIZE-SINGLE-SOURCE(G, s). Prove that if a sequence of relaxation steps sets [s] to a non-NIL value, then G contains a negative-weight cycle.

25.1-5

Let G = (V, E) be a weighted, directed graph with no negative-weight edges. Let s V be the source vertex, and let us define [v] as usual: [v] is the predecessor of v on some shortest path to v from source s if v V - {s} is reachable from s, and NIL otherwise. Give an example of such a graph G and an assignment of values that produces a cycle in G . (By Lemma 25.8, such an assignment cannot be produced by a sequence of relaxation steps.)

25.1-6

Let G = (V, E) be a weighted, directed graph with weight function w : E R and no negative-weight cycles. Let s V be the source vertex, and let G be initialized by INITIALIZE-SINGLE-SOURCE(G, s). Prove that for every vertex v V , there exists a path from s to v in G and that this property is maintained as an invariant over any sequence of relaxations.

25.1-7

Let G = (V, E) be a weighted, directed graph that contains no negative-weight cycles. Let s V be the source vertex, and let G be initialized by INITIALIZE-SINGLE-SOURCE(G, s). Prove that there is a sequence of V - 1 relaxation steps that produces d[v] = (s ,v) for all v V.

25.1-8

Let G be an arbitrary weighted, directed graph with a negative-weight cycle reachable from the source vertex s. Show that an infinite sequence of relaxations of the edges of G can always be constructed such that every relaxation causes a shortest-path estimate to change.

25.2 Dijkstra's algorithm

Dijkstra's algorithm solves the single-source shortest-paths problem on a weighted, directed graph G = (V, E) for the case in which all edge weights are nonnegative. In this section, therefore, we assume that w(u, v) 0 for each edge (u, v) E.

Dijkstra's algorithm maintains a set S of vertices whose final shortest-path weights from the source s have already been determined. That is, for all vertices v S, we have d[v] = (s, v). The algorithm repeatedly selects the vertex u V - S with the minimum shortest-path estimate, inserts u into S, and relaxes all edges leaving u. In the following implementation, we maintain a priority queue Q that contains all the vertices in V - S, keyed by their d values. The implementation assumes that graph G is represented by adjacency lists.

DIJKSTRA(G,w,s)

1  INITIALIZE-SINGLE-SOURCE (G,s)

2

3  Q    V[G]

4  while

5       do u    EXTRACT-MIN(Q)

6          S    S    {u}

7          for each vertex v    Adj[u]

8              do RELAX (u,v,w)

Figure 25.5 The execution of Dijkstra's algorithm. The source is the leftmost vertex. The shortest-path estimates are shown within the vertices, and shaded edges indicate predecessor values: if edge (u,v) is shaded, then [v] = u. Black vertices are in the set S, and white vertices are in the priority queue Q = V - S. (a) The situation just before the first iteration of the while loop of lines 4-8. The shaded vertex has the minimum d value and is chosen as vertex u in line 5. (b)-(f) The situation after each successive iteration of the while loop. The shaded vertex in each part is chosen as vertex u in line 5 of the next iteration. The d and values shown in part (f) are the final values.

Dijkstra's algorithm relaxes edges as shown in Figure 25.5. Line 1 performs the usual initialization of d and values, and line 2 initializes the set S to the empty set. Line 3 then initializes the priority queue Q to contain all the vertices in . Each time through the while loop of lines 4-8, a vertex u is extracted from Q = V - S and inserted into set S. (The first time through this loop, u = s.) Vertex u, therefore, has the smallest shortest-path estimate of any vertex in V - S. Then, lines 7-8 relax each edge (u, v) leaving u, thus updating the estimate d[v] and the predecessor [v] if the shortest path to v can be improved by going through u. Observe that vertices are never inserted into Q after line 3 and that each vertex is extracted from Q and inserted into S exactly once, so that the while loop of lines 4-8 iterates exactly V times.

Because Dijkstra's algorithm always chooses the "lightest" or "closest" vertex in V - S to insert into set S, we say that it uses a greedy strategy. Greedy strategies are presented in detail in Chapter 17, but you need not have read that chapter to understand Dijkstra's algorithm. Greedy strategies do not always yield optimal results in general, but as the following theorem and its corollary show, Dijkstra's algorithm does indeed compute shortest paths. The key is to show that each time a vertex u is inserted into set S, we have d[u] = (s, u).

Figure 25.6 The proof of Theorem 25.10. Set S is nonempty just before vertex u is inserted into it. A shortest path p from source s to vertex u can be decomposed into where y is the first vertex on the path that is not in V - S and x S immediately precedes y. Vertices x and y are distinct, but we may have s = x or y = u. Path p2 may or may not reenter set S.

Theorem 25.10

If we run Dijkstra's algorithm on a weighted, directed graph G = (V, E) with nonnegative weight function w and source s, then at termination, d[u] = (s, u) for all vertices u V.

Proof We shall show that for each vertex u V, we have d[u] = (s, u) at the time when u is inserted into set S and that this equality is maintained thereafter.

For the purpose of contradiction, let u be the first vertex for which d[u] (s, u) when it is inserted into set S. We shall focus our attention on the situation at the beginning of the iteration of the while loop in which u is inserted into S and derive the contradiction that d[u] = (s, u) at that time by examining a shortest path from s to u. We must have u s because s is the first vertex inserted into set S and d[s] = (s, s) = 0 at that time. Because u s, we also have that just before u is inserted into S. There must be some path from s to u, for otherwise d[u] = (s, u) = by Corollary 25.6, which would violate our assumption that d[u] (s, u). Because there is at least one path, there is a shortest path p from s to u. Path p connects a vertex in S, namely s, to a vertex in V - S, namely u. Let us consider the first vertex y along p such that y V - S, and let x V be y's predecessor. Thus, as shown in Figure 25.6, path p can be decomposed as

We claim that d[y] = (s, y) when u is inserted into S. To prove this claim, observe that x S. Then, because u is chosen as the first vertex for which d[u] (s, u) when it is inserted into S, we had d[x] = (s, x) when x was inserted into S. Edge (x, y) was relaxed at that time, so the claim follows from Lemma 25.7.

We can now obtain a contradiction to prove the theorem. Because y occurs before u on a shortest path from s to u and all edge weights are nonnegative (notably those on path p2), we have (s, y) (s, u), and thus

d[y]  =    (s, y)

      (s, u)

    d[u]     (by Lemma 25.5).

(25.2)

But because both vertices u and y were in V - S when u was chosen in line 5, we have d[u] d[y]. Thus, the two inequalities in (25.2) are in fact equalities, giving

d[y] =   (s, y) =   (s, u) = d[u] .

Consequently, d[u] = (s, u), which contradicts our choice of u. We conclude that at the time each vertex u V is inserted into set S, we have d[u] = (s, u), and by Lemma 25.5, this equality holds thereafter.

Corollary 25.11

If we run Dijkstra's algorithm on a weighted, directed graph G = (V, E) with nonnegative weight function w and source s, then at termination, the predecessor subgraph G is a shortest-paths tree rooted at s.

Proof Immediate from Theorem 25.10 and Lemma 25.9.

Analysis

How fast is Dijkstra's algorithm? Consider first the case in which we maintain the priority queue Q = V - S as a linear array. For such an implementation, each EXTRACT-MIN operation takes time O(V), and there are |V| such operations, for a total EXTRACT-MIN time of O(V2). Each vertex v V is inserted into set S exactly once, so each edge in the adjacency list Adj[v] is examined in the for loop of lines 4-8 exactly once during the course of the algorithm. Since the total number of edges in all the adjacency lists is |E|, there are a total of |E| iterations of this for loop, with each iteration taking O(1) time. The running time of the entire algorithm is thus O(V2 + E) = O(V2).

If the graph is sparse, however, it is practical to implement the priority queue Q with a binary heap. The resulting algorithm is sometimes called the modified Dijkstra algorithm. Each EXTRACT-MIN operation then takes time O(lg V). As before, there are |V| such operations. The time to build the binary heap is O(V). The assignment d[v] d[u] + w(u, v) in RELAX is accomplished by the call DECREASE-KEY(Q, v, d[u] + w(u, v)), which takes time O(lg V) (see Exercise 7.5-4), and there are still at most |E| such operations. The total running time is therefore O((V + E) lg V), which is O(E lg V) if all vertices are reachable from the source.

We can in fact achieve a running time of O(V lg V + E) by implementing the priority queue Q with a Fibonacci heap (see Chapter 21). The amortized cost of each of the |V| EXTRACT-MIN operations is O(lg V), and each of the |E| DECREASE-KEY calls takes only O(1) amortized time. Historically, the development of Fibonacci heaps was motivated by the observation that in the modified Dijkstra algorithm there are potentially many more DECREASE-KEY calls than EXTRACT-MIN calls, so any method of reducing the amortized time of each DECREASE-KEY operation to o(lg V) without increasing the amortized time of EXTRACT-MIN would yield an asymptotically faster implementation.

Dijkstra's algorithm bears some similarity to both breadth-first search (see Section 23.2) and Prim's algorithm for computing minimum spanning trees (see Section 24.2). It is like breadth-first search in that set S corresponds to the set of black vertices in a breadth-first search; just as vertices in S have their final shortest-path weights, so black vertices in a breadth-first search have their correct breadth-first distances. Dijkstra's algorithm is like Prim's algorithm in that both algorithms use a priority queue to find the "lightest" vertex outside a given set (the set S in Dijkstra's algorithm and the tree being grown in Prim's algorithm), insert this vertex into the set, and adjust the weights of the remaining vertices outside the set accordingly.

Exercises

25.2-1

Run Dijkstra's algorithm on the directed graph of Figure 25.2, first using vertex s as the source and then using vertex y as the source. In the style of Figure 25.5, show the d and values and the vertices in set S after each iteration of the while loop.

25.2-2

Give a simple example of a directed graph with negative-weight edges for which Dijkstra's algorithm produces incorrect answers. Why doesn't the proof of Theorem 25.10 go through when negative-weight edges are allowed?

25.2-3

Suppose we change line 4 of Dijkstra's algorithm to the following.

4 while |Q| > 1

This change causes the while loop to execute |V| - 1 times instead of |V| times. Is this proposed algorithm correct?

25.2-4

We are given a directed graph G = (V, E) on which each edge (u, v) E has an associated value r(u, v), which is a real number in the range 0 r(u, v) 1 that represents the reliability of a communication channel from vertex u to vertex v. We interpret r(u, v) as the probability that the channel from u to v will not fail, and we assume that these probabilities are independent. Give an efficient algorithm to find the most reliable path between two given vertices.

25.2-5

Let G = (V, E) be a weighted, directed graph with weight function w : E {0, 1, . . . , W - 1} for some nonnegative integer W. Modify Dijkstra's algorithm to compute the shortest paths from a given source vertex s in O(WV + E) time.

25.2-6

Modify your algorithm from Exercise 25.2-5 to run in O((V + E) lg W) time. (Hint: How many distinct shortest-path estimates can there be in V - S at any point in time?)

25.3 The Bellman-Ford algorithm

The Bellman-Ford algorithm solves the single-source shortest-paths problem in the more general case in which edge weights can be negative. Given a weighted, directed graph G = (V, E) with source s and weight function w : E R , the Bellman-Ford algorithm returns a boolean value indicating whether or not there is a negative-weight cycle that is reachable from the source. If there is such a cycle, the algorithm indicates that no solution exists. If there is no such cycle, the algorithm produces the shortest paths and their weights.

Like Dijkstra's algorithm, the Bellman-Ford algorithm uses the technique of relaxation, progressively decreasing an estimate d[v] on the weight of a shortest path from the source s to each vertex v V until it achieves the actual shortest-path weight (s, v). The algorithm returns TRUE if and only if the graph contains no negative-weight cycles that are reachable from the source.

BELLMAN-FORD(G, w, s)

1 INITIALIZE-SINGLE-SOURCE(G, s)

2 for i    1 to |V[G]| - 1

3      do for each edge (u, v)    E[G]

4              do RELAX(u, v, w)

5 for each edge (u, v)    E[G]

6      do if d[v] > d[u] + w(u, v)

7            then return FALSE

8 return TRUE

Figure 25.7 shows how the execution of the Bellman-Ford algorithm works on a graph with 5 vertices. After performing the usual initialization, the algorithm makes |V| - 1 passes over the edges of the graph. Each pass is one iteration of the for loop of lines 2-4 and consists of relaxing each edge of the graph once. Figures 25.7(b)-(e) show the state of the algorithm after each of the four passes over the edges. After making |V|- 1 passes, lines 5-8 check for a negative-weight cycle and return the appropriate boolean value. (We shall see a little later why this check works.)

Figure 25.7 The execution of the Bellman-Ford algorithm. The source is vertex z. The d values are shown within the vertices, and shaded edges indicate the values. In this particular example, each pass relaxes the edges in lexicographic order: (u, v),(u, x),(u, y),(v, u),(x, v),(x, y),(y, v),(y, z),(z, u),(z, x). (a) The situation just before the first pass over the edges. (b)-(e) The situation after each successive pass over the edges. The d and values in part (e) are the final values. The Bellman-Ford algorithm returns TRUE in this example.

The Bellman-Ford algorithm runs in time O(V E), since the initialization in line 1 takes (V) time, each of the |V| - 1 passes over the edges in lines 2-4 takes O(E) time, and the for loop of lines 5-7 takes O(E) time.

To prove the correctness of the Bellman-Ford algorithm, we start by showing that if there are no negative-weight cycles, the algorithm computes correct shortest-path weights for all vertices reachable from the source. The proof of this lemma contains the intuition behind the algorithm.

Lemma 25.12

Let G = (V, E) be a weighted, directed graph with source s and weight function w : E R, and assume that G contains no negative-weight cycles that are reachable from s. Then, at the termination of BELLMAN-FORD, we have d[v] = (s, v) for all vertices v that are reachable from s.

Proof Let v be a vertex reachable from s, and let p = v0, v1, . . . , vk be a shortest path from s to v, where v0 = s and vk = v. The path p is simple, and so k |V| - 1. We want to prove by induction that for i = 0, 1, . . . , k, we have d[vi] = (s, vi) after the ith pass over the edges of G and that this equality is maintained thereafter. Because there are |V| - 1 passes, this claim suffices to prove the lemma.

For the basis, we have d[v0] = (s, v0) = 0 after initialization, and by Lemma 25.5, this equality is maintained thereafter.

For the inductive step, we assume that d[vi - 1] = (s, vi - 1) after the (i - 1)st pass. Edge (vi - 1, vi) is relaxed in the ith pass, so by Lemma 25.7, we conclude that d[vi] = (s, vi) after the ith pass and at all subsequent times, thus completing the proof.

Corollary 25.13

Let G = (V, E) be a weighted, directed graph with source vertex s and weight function w : E R. Then for each vertex v V, there is a path from s to v if and only if BELLMAN-FORD terminates with d[v] < when it is run on G.

Proof The proof is similar to that of Lemma 25.12 and is left as Exercise 25.3-2.

Theorem 25.14

Let BELLMAN-FORD be run on a weighted, directed graph G = (V, E) with source s and weight function w : E R. If G contains no negative-weight cycles that are reachable from s, then the algorithm returns TRUE, we have d[v] = (s,v) for all vertices v V, and the predecessor subgraph G is a shortest-paths tree rooted at s. If G does contain a negative-weight cycle reachable from S, then the algorithm returns FALSE.

Proof Suppose that graph G contains no negative-weight cycles that are reachable from the source s. We first prove the claim that at termination, d[v] = (s, v) for all vertices v V. If vertex v is reachable from s, then Lemma 25.12 proves this claim. If v is not reachable from s, then the claim follows from Corollary 25.6. Thus, the claim is proven. Lemma 25.9, along with the claim, implies that G is a shortest-paths tree. Now we use the claim to show that BELLMAN-FORD returns TRUE. At termination, we have for all edges (u, v) E,

d[v]  =    (s,v)

      (s,u) + w(u,v)  (by Lemma 25.3)

=  d[u] + w(u,v),

and so none of the tests in line 6 causes BELLMAN-FORD to return FALSE. It therefore returns TRUE.

Conversely, suppose that graph G contains a negative-weight cycle c = v0, v1, . . . , vk , where v0 = vk, that is reachable from the source s. Then,

(25.3)

Assume for the purpose of contradiction that the Bellman-Ford algorithm returns TRUE. Thus, d[vi] d[vi-1]+w(vi-l, vi) for i = 1, 2, . . . , k. Summing the inequalities around cycle c gives us

As in the proof of Lemma 25.8, each vertex in c appears exactly once in each of the first two summations. Thus,

Moreover, by Corollary 25.13, d [vi] is finite for i = 1, 2, . . . , k. Thus,

which contradicts inequality (25.3). We conclude that the Bellman-Ford algorithm returns TRUE if graph G contains no negative-weight cycles reachable from the source, and FALSE otherwise.

Exercises

25.3-1

Run the Bellman-Ford algorithm on the directed graph of Figure 25.7, using vertex y as the source. Relax edges in lexicographic order in each pass, and show the d and values after each pass. Now, change the weight of edge (y, v) to 4 and run the algorithm again, using z as the source.

25.3-2

Prove Corollary 25.13.

25.3-3

Given a weighted, directed graph G = (V, E) with no negative-weight cycles, let m be the maximum over all pairs of vertices u, v V of the minimum number of edges in a shortest path from u to v. (Here, the shortest path is by weight, not the number of edges.) Suggest a simple change to the Bellman-Ford algorithm that allows it to terminate in m + 1 passes.

25.3-4

Modify the Bellman-Ford algorithm so that it sets d[v] to - for all vertices v for which there is a negative-weight cycle on some path from the source to v.

25.3-5

Let G = (V, E) be a weighted, directed graph with weight function w : E R. Give an O(V E)-time algorithm to find, for each vertex v V, the value * (v) = minu V { (u, v)}.

25.3-6

Suppose that a weighted, directed graph G = (V, E) has a negative-weight cycle. Give an efficient algorithm to list the vertices of one such cycle. Prove that your algorithm is correct.

25.4 Single-source shortest paths in directed acyclic graphs

By relaxing the edges of a weighted dag (directed acyclic graph) G = (V, E) according to a topological sort of its vertices, we can compute shortest paths from a single source in (V + E) time. Shortest paths are always well defined in a dag, since even if there are negative-weight edges, no negative-weight cycles can exist.

The algorithm starts by topologically sorting the dag (see Section 23.4) to impose a linear ordering on the vertices. If there is a path from vertex u to vertex v, then u precedes v in the topological sort. We make just one pass over the vertices in the topologically sorted order. As each vertex is processed, all the edges that leave the vertex are relaxed.

DAG-SHORTEST-PATHS(G,w,s)

1 topologically sort the vertices of G

2 INITIALIZE-SINGLE-SOURCE(G,s)

3 for each vertex u taken in topologically sorted order

4      do for each vertex v   Adj[u]

5             do RELAX(u,v,w)

An example of the execution of this algorithm is shown in Figure 25.8.

The running time of this algorithm is determined by line 1 and by the for loop of lines 3-5. As shown in Section 23.4, the topological sort can be performed in (V + E) time. In the for loop of lines 3-5, as in Dijkstra's algorithm, there is one iteration per vertex. For each vertex, the edges that leave the vertex are each examined exactly once. Unlike Dijkstra's algorithm, however, we use only O( 1 ) time per edge. The running time is thus (V + E), which is linear in the size of an adjacency-list representation of the graph.

The following theorem shows that the DAG-SHORTEST-PATHS procedure correctly computes the shortest paths.

Figure 25.8 The execution of the algorithm for shortest paths in a directed acyclic graph. The vertices are topologically sorted from left to right. The source vertex is s. The d values are shown within the vertices, and shaded edges indicate the values.(a) The situation before the first iteration of the for loop of lines 3-5. (b)-(g) The situation after each iteration of the for loop of lines 3-5. The newly blackened vertex in each iteration was used as v in that iteration. The values shown in part (g) are the final values.

Theorem 25.15

If a weighted, directed graph G = (V, E) has source vertex s and no cycles, then at the termination of the DAG-SHORTEST-PATHS procedure, d[v] = (s, v) for all vertices v V, and the predecessor subgraph G is a shortest-paths tree.

Proof We first show that d[v] = (s, v) for all vertices v V at termination. If v is not reachable from s, then d[v] = (s, v) = by Corollary 25.6. Now, suppose that is reachable from s, so that there is a shortest path p = v 0, v 1, . . . , v k , where v 0 = s and vk = v. Because we process the vertices in topologically sorted order, the edges on p are relaxed in the order (v 0, v 1), (v 1, v 2), . . . , (v k-1, vk). A simple induction using Lemma 25.7 (as in the proof of Lemma 25.12) shows that d[vi] = (s, vi) at termination for i = 0, 1, . . . , k. Finally, by Lemma 25.9, G is a shortest-paths tree.

An interesting application of this algorithm arises in determining critical paths in PERT chart2 analysis. Edges represent jobs to be performed, and edge weights represent the times required to perform particular jobs. If edge (u, v) enters vertex v and edge (v, x) leaves v, then job (u, v) must be performed prior to job (v, x). A path through this dag represents a sequence of jobs that must be performed in a particular order. A critical path is a longest path through the dag, corresponding to the longest time to perform an ordered sequence of jobs. The weight of a critical path is a lower bound on the total time to perform all the jobs. We can find a critical path by either

negating the edge weights and running DAG-SHORTEST-PATHS, or

running DAG-SHORTEST-PATHS, replacing " " by "- " in line 2 of INITIALIZE-SINGLE-SOURCE and ">" by "< " in the RELAX procedure.

2"PERT" is an acronym for "program evaluation and review technique."

Exercises

25.4-1

Run DAG-SHORTEST-PATHS on the directed graph of Figure 25.8, using vertex r as the source.

25.4-2

Suppose we change line 3 of DAG-SHORTEST-PATHS to read

3 for the first |V| - 1 vertices, taken in topologically sorted order

Show that the procedure would remain correct.

25.4-3

The PERT chart formulation given above is somewhat unnatural. It would be more natural for vertices to represent jobs and edges to represent sequencing constraints; that is, edge (u, v) would indicate that job u must be performed before job v. Weights would then be assigned to vertices, not edges. Modify the DAG-SHORTEST-PATHS procedure so that it finds a longest path in a directed acyclic graph with weighted vertices in linear time.

25.4-4

Give an efficient algorithm to count the total number of paths in a directed acyclic graph. Analyze your algorithm and comment on its practicality.

25.5 Difference constraints and shortest paths

In the general linear-programming problem, we wish to optimize a linear function subject to a set of linear inequalities. In this section, we investigate a special case of linear programming that can be reduced to finding shortest paths from a single source. The single-source shortest-paths problem that results can then be solved using the Bellman-Ford algorithm, thereby also solving the linear-programming problem.

Linear programming

In the general linear-programming problem, we are given an m n matrix A, an m-vector b, and an n-vector c. We wish to find a vector x of n elements that maximizes the objective function subject to the m constraints given by Ax b.

Many problems can be expressed as linear programs, and for this reason much work has gone into algorithms for linear programming. The simplex algorithm3 solves general linear programs very quickly in practice. With some carefully contrived inputs, however, the simplex method can require exponential time. General linear programs can be solved in polynomial time by either the ellipsoid algorithm, which runs slowly in practice, or Karmarkar's algorithm, which in practice is often competitive with the simplex method.

3 The simplex algorithm finds an optimal solution to a linear programming problem by examining a sequence of points in the feasible region--the region in n-space that satisfies Ax b. The algorithm is based on the fact that a solution that maximizes the objective function over the feasible region occurs at some "extreme point," or "corner," of the feasible region. The simplex algorithm proceeds from corner to corner of the feasible region until no further improvement of the objective function is possible. A "simplex" is the convex hull (see Section 35.3) of d + 1 points in d-dimensional space (such as a triangle in the plane, or a tetrahedron in 3-space). According to Dantzig [53], it is possible to view the operation of moving from one corner to another as an operation on a simplex derived from a "dual" interpretation of the linear programming problem--hence the name "simplex method."

Due to the mathematical investment needed to understand and analyze them, this text does not cover general linear-programming algorithms. For several reasons, though, it is important to understand the setup of linear-programming problems. First, knowing that a given problem can be cast as a polynomial-sized linear-programming problem immediately means that there is a polynomial-time algorithm for the problem. Second, there are many special cases of linear programming for which faster algorithms exist. For example, as shown in this section, the single-source shortest-paths problem is a special case of linear programming. Other problems that can be cast as linear programming include the single-pair shortest-path problem (Exercise 25.5-4) and the maximum-flow problem (Exercise 27.1-8).

Sometimes we don't really care about the objective function; we just wish to find any feasible solution, that is, any vector x that satisfies Ax b, or to determine that no feasible solution exists. We shall focus on one such feasibility problem.

Systems of difference constraints

In a system of difference constraints, each row of the linear-programming matrix A contains one 1 and one - 1, and all other entries of A are 0. Thus, the constraints given by Ax b are a set of m difference constraints involving n unknowns, in which each constraint is a simple linear inequality of the form

xj - xi    bk ,

where 1 i, j n and 1 k m.

For example, consider problem of finding the 5-vector x = (xi) that satisfies

This problem is equivalent to finding the unknowns xi, for i = 1, 2, . . . , 5, such that the following 8 difference constraints are satisfied:

x1 - x2       0,

x1 - x5      -1,

x2 - x5       1,

x3 - x1       5,

x4 - x1       4,

x4 - x3      -1,

x5 - x3      -3,

x5 - x4      -3

(25.4)

One solution to this problem is x = (-5, -3, 0, -1, -4), as can be verified directly by checking each inequality. In fact, there is more than one solution to this problem. Another is x' = (0, 2, 5, 4, 1). These two solutions are related: each component of x' is 5 larger than the corresponding component of x. This fact is not mere coincidence.

Lemma 25.16

Let x = (x1, x2, . . . , xn) be a solution to a system Ax b of difference constraints, and let d be any constant. Then x +d = (x1 + d, x2 + d, . . . , xn + d) is a solution to Ax b as well.

Proof For each xi and xj, we have (xj + d) - (xi + d) = xj - xi. Thus, if x satisfies Ax b, so does x + d

Systems of difference constraints occur in many different applications. For example, the unknowns xi may be times at which events are to occur. Each constraint can be viewed as stating that one event cannot occur too much later than another event. Perhaps the events are jobs to be performed during the construction of a house. If the digging of the foundation begins at time xl and takes 3 days and the pouring of the concrete for the foundation begins at time x2, we may well desire that x2 x1 + 3 or, equivalently, that x1 - x2 -3. Thus, the relative timing constraint can be expressed as a difference constraint.

Constraint graphs

It is beneficial to interpret systems of difference constraints from a graph-theoretic point of view. The idea is that in a system Ax b of difference constraints, the n m linear-programming matrix A can be viewed as an incidence matrix (see Exercise 23.1-7) for a graph with n vertices and m edges. Each vertex vi in the graph, for i = 1, 2, . . . , n, corresponds to one of the n unknown variables xi. Each directed edge in the graph corresponds to one of the m inequalities involving two unknowns.

More formally, given a system Ax b of difference constraints, the corresponding constraint graph is a weighted, directed graph G= (V, E), where

V = {v0,v1, . . . , vn}

and

E = {(vi,vj) : xj - xi    bk is a constraint}

  {(v0,v1 ),(v0,v2),(v0,v3),...,(v0,vn)} .

The additional vertex v0 is incorporated, as we shall see shortly, to guarantee that every other vertex is reachable from it. Thus, the vertex set V consists of a vertex vi for each unknown xi, plus an additional vertex v0. The edge set E contains an edge for each difference constraint, plus an edge (v0, vi) for each unknown xi. If xj - xi bk is a difference constraint, then the weight of edge (vi, vj) is w(vi, vj) = bk. The weight of each edge leaving v0 is 0. Figure 25.9 shows the constraint graph for the system (25.4) of difference constraints.

Figure 25.9 The constraint graph corresponding to the system (25.4) of difference constraints. The value of (v0, vj) is shown in each vertex vi. A feasible solution to the system is x = (-5, -3, 0, -1, -4).

The following theorem shows that a solution to a system of difference constraints can be obtained by finding shortest-path weights in the corresponding constraint graph.

Theorem 25.17

Given a system Ax b of difference constraints, let G = (V, E) be the corresponding constraint graph. If G contains no negative-weight cycles, then

x = (  (v0,v1),   (v0,v2),   (v0,v3),...,   (v0,vn))

(25.5)

is a feasible solution for the system. If G contains a negative-weight cycle, then there is no feasible solution for the system.

Proof We first show that if the constraint graph contains no negative-weight cycles, then equation (25.5) gives a feasible solution. Consider any edge (vi, vj) E. By Lemma 25.3, (v0, vj) (v0, vi) +w(vi, vj) or, equivalently, (v0, vj) - (v0, vj) w(vi, vj). Thus, letting xi = (v0, vi) and xj = (v0, vj) satisfies the difference constraint xj - xi w(vi, vj) that corresponds to edge (vi, vj).

Now we show that if the constraint graph contains a negative-weight cycle, then the system of difference constraints has no feasible solution. Without loss of generality, let the negative-weight cycle be c = v1, v2, . . . , vk , where v1 = vk. (The vertex v0 cannot be on cycle c, because it has no entering edges.) Cycle c corresponds to the following difference constraints:

x2 - x1    w(v1, v2),

x3 - x2    w(v2, v3),

xk - xk_1    w(vk-1, vk),

x1 - xk    w(vk, v1) .

Since any solution for x must satisfy each of these k inequalities, any solution must also satisfy the inequality that results when we sum them together. If we sum the left-hand sides, each unknown xi is added in once and subtracted out once, so that the left-hand side of the sum is 0. The right-hand side sums to w(c), and thus we obtain 0 w(c). But since c is a negative-weight cycle, w(c) 0, and hence any solution for the x must satisfy 0 w(c) 0, which is impossible.

Solving systems of difference constraints

Theorem 25.17 tells us that we can use the Bellman-Ford algorithm to solve a system of difference constraints. Because there are edges from the source vertex v0 to all other vertices in the constraint graph, any negative-weight cycle in the constraint graph is reachable from v0. If the Bellman-Ford algorithm returns TRUE, then the shortest-path weights give a feasible solution to the system. In Figure 25.9, for example, the shortest-path weights provide the feasible solution x = (-5, -3, 0, -1, -4), and by Lemma 25.16, x = (d - 5, d - 3, d, d - 1, d - 4) is also a feasible solution for any constant d. If the Bellman-Ford algorithm returns FALSE, there is no feasible solution to the system of difference constraints.

A system of difference constraints with m constraints on n unknowns produces a graph with n + 1 vertices and n + m edges. Thus, using the Bellman-Ford algorithm, we can solve the system in O((n + 1 )(n + m)) = O(n2+nm) time. Exercise 25.5-5 asks you to show that the algorithm actually runs in O(nm) time, even if m is much less than n.

Exercises

25.5-1

Find a feasible solution or determine that no feasible solution exists for the following system of difference constraints:

x1 - x2       1,

x1 - x4      -4,

x2 - x3       2,

x2 - x5       7,

x2 - x6       5,

x3 - x6      10,

x4 - x2       2,

x5 - x1      -1,

x5 - x4       3,

x6 - x3      -8.

25.5-2

Find a feasible solution or determine that no feasible solution exists for the following system of difference constraints:

x1 - x2       4,

x1 - x5       5,

x2 - x4      -6,

x3 - x2       1,

x4 - x1       3,

x4 - x3       5,

x4 - x5      10,

x5 - x3      -4,

x5 - x4      -8.

25.5-3

Can any shortest-path weight from the new vertex v0 in a constraint graph be positive? Explain.

25.5-4

Express the single-pair shortest-path problem as a linear program.

25.5-5

Show how to modify the Bellman-Ford algorithm slightly so that when it is used to solve a system of difference constraints with m inequalities on n unknowns, the running time is O(nm).

25.5-6

Show how a system of difference constraints can be solved by a Bellman-Ford-like algorithm that runs on a constraint graph without the extra vertex v0.

25.5-7

Let Ax b be a system of m difference constraints in n unknowns. Show that the Bellman-Ford algorithm, when run on the corresponding constraint graph, maximizes subject to Ax b and xi 0 for all xi.

25.5-8

Show that the Bellman-Ford algorithm, when run on the constraint graph for a system Ax b of difference constraints, minimizes the quantity (max {xi} - min {xi}) subject to Ax b. Explain how this fact might come in handy if the algorithm is used to schedule construction jobs.

25.5-9

Suppose that every row in the matrix A of a linear program Ax b corresponds to a difference constraint, a single-variable constraint of the form xi bk, or a single-variable constraint of the form -xi bk. Show how the Bellman-Ford algorithm can be adapted to solve this variety of constraint system.

25.5-10

Suppose that in addition to a system of difference constraints, we want to handle equality constraints of the form xi = xj + bk. Show how the Bellman-Ford algorithm can be adapted to solve this variety of constraint system.

25.5-11

Give an efficient algorithm to solve a system Ax b of difference constraints when all of the elements of b are real-valued and all of the unknowns xi must be integers.

25.5-12

Give an efficient algorithm to solve a system Ax b of difference constraints when all of the elements of b are real-valued and some, but not necessarily all, of the unknowns xi must be integers.

Problems

25-1 Yen's improvement to Bellman-Ford

Suppose that we order the edge relaxations in each pass of the Bellman-Ford algorithm as follows. Before the first pass, we assign an arbitrary linear order v1, v2, . . . , v|V|, to the vertices of the input graph G = (V, E). Then, we partition the edge set E into E Eb, where E = {(vi, vj) E: i < j} and Eb = {(vi, vj) E: i > j}. Define G = (V, E ) and Gb = (V, Eb).

a. Prove that G is acyclic with topological sort v1, v2, . . . , v|V| and that Gb is acyclic with topological sort (vV|, v|V|-1, . . . v1).

Suppose that we implement each pass of the Bellman-Ford algorithm in the following way. We visit each vertex in the order v1, v2, . . . , v|V| relaxing edges of E that leave the vertex. We then visit each vertex in the order v|V|, v|V|-1, . . . v1, relaxing edges of Eb that leave the vertex.

b. Prove that with this scheme, if G contains no negative-weight cycles that are reachable from the source vertex s, then after only |V| /2 passes over the edges, d[v] = (s, v) for all vertices v V.

c. How does this scheme affect the running time of the Bellman-Ford algorithm?

25-2 Nesting boxes

A d-dimensional box with dimensions (x1, x2, . . . , xd) nests within another box with dimensions (y1, y2, . . . , yd) if there exists a permutation on{1, 2, . . . , d} such that x, (l) < y1, x (d) < yd.

a. Argue that the nesting relation is transitive.

b. Describe an efficient method to determine whether or not one d-dimensional box nests inside another.

c. Suppose that you are given a set of n d-dimensional boxes {B1, B2, . . . , Bn}. Describe an efficient algorithm to determine the longest sequence Bi1, Bi2, . . . , Bik of boxes such that Bij nests within Bij+1 for j = 1, 2, . . . , k - 1. Express the running time of your algorithm in terms of n and d.

25-3 Arbitrage

Arbitrage is the use of discrepancies in currency exchange rates to transform one unit of a currency into more than one unit of the same currency. For example, suppose that 1 U.S. dollar buys 0.7 British pound, 1 British pound buys 9.5 French francs, and 1 French franc buys 0.16 U.S. dollar. Then, by converting currencies, a trader can start with 1 U.S. dollar and buy 0.7 9.5 0.16 = 1.064 U.S. dollars, thus turning a profit of 6.4 percent.

Suppose that we are given n currencies cl, c2, . . . , cn and an n n table R of exchange rates, such that one unit of currency ci buys R[i, j] units of currency cj.

a. Give an efficient algorithm to determine whether or not there exists a sequence of currencies ci1, ci2, . . . , cik such that

R[i1, i2]    R[i2, i3]          R[ik-1, ik]    R[ik, i1] > 1.

Analyze the running time of your algorithm.

b. Give an efficient algorithm to print out such a sequence if one exists. Analyze the running time of your algorithm.

25-4 Gabow's scaling algorithm for single-source shortest paths

A scaling algorithm solves a problem by initially considering only the highest-order bit of each relevant input value (such as an edge weight). It then refines the initial solution by looking at the two highest-order bits. It progressively looks at more and more high-order bits, refining the solution each time, until all bits have been considered and the correct solution has been computed.

In this problem, we examine an algorithm for computing the shortest paths from a single source by scaling edge weights. We are given a directed graph G = (V, E) with nonnegative integer edge weights w. Let W = max(u,v) E {w(u, v)}. Our goal is to develop an algorithm that runs in O(E lg W) time.

The algorithm uncovers the bits in the binary representation of the edge weights one at a time, from the most significant bit to the least significant bit. Specifically, let k = lg (W + 1) be the number of bits in the binary representation of W, and for i = 1, 2, . . . , k, let wi(u, v) = w (u, v)/2k - i . That is, wi(u, v) is the "scaled-down" version of w(u, v) given by the i most significant bits of w(u, v). (Thus, wk(u, v) = w(u, v) for all (u, v) E.) For example, if k = 5 and w(u, v) = 25, which has the binary representation 11001 , then w3(u, v) = 110 = 6. As another example with k = 5, if w(u, v) = 00100 = 4, then w3(u, v) = 001 = 1. Let us define i(u, v) as the shortest-path weight from vertex u to vertex v using weight function wi. Thus, k(u, v) = (u, v) for all u, v V. For a given source vertex s, the scaling algorithm first computes the shortest-path weights 1(s, v) for all v V, then computes 2(s, v) for all v V, and so on, until it computes k(s, v) for all v V. We assume throughout that |E | |V | - 1, and we shall see that computing i from i - 1takes O(E) time, so that the entire algorithm takes O(kE) = O(E lg W) time.

a. uppose that for all vertices v V, we have (s, v) < |E|. Show that we can compute (s, v) for all v V in O(E) time.

b. Show that we can compute 1(s, v) for all v V in O(E) time.

Let us now concentrate on computing i from i - 1.

c. Prove that for i = 2, 3, . . . , k, either wi(u, v) = 2wi - 1(u, v) or wi(u, v) = 2wi - 1 (u, v) + 1. Then, prove that

2  i - 1(s,v) <   i(s,v) < 2  i - 1(s,v) + |V| - 1

for all v V.

d. Define for i = 2, 3, . . . , k and all (u, v) E,

Prove that for i = 2, 3, . . . , k and all u, v V, the "reweighted" value of edge (u, v) is a nonnegative integer.

e. Now, define as the shortest-path weight from s to v using the weight function . Prove that for i = 2, 3, . . . , k and all v V,

f. Show how to compute i(s, v) from i - 1 (s, v) for all v V in O(E) time, and conclude that (s, v) can be computed for all v V in O(E lg W) time.

25-5 Karp's minimum mean-weight cycle algorithm

Let G = (V, E) be a directed graph with weight function w: E R, and let n = |V|. We define the mean weight of a cycle c = e1, e2, . . . , ek of edges in E to be

Let * = minc (c), where c ranges over all directed cycles in G. A cycle c for which (c) = * is called a minimum mean-weight cycle. This problem investigates an efficient algorithm for computing *.

Assume without loss of generality that every vertex v V is reachable from a source vertex s V. Let (s, v) be the weight of a shortest path from s to v, and let k(s, v) be the weight of a shortest path from s to v consisting of exactly k edges. If there is no path from s to v with exactly k edges, then k(s, v) = .

a. Show that if * = 0, then G contains no negative-weight cycles and (s, v) = min0 k n - 1 k(s, v) for all vertices v V.

b. Show that if * = 0, then

for all vertices v V. (Hint: Use both properties from part (a).)

c. Let c be a 0-weight cycle, and let u and v be any two vertices on c. Suppose that the weight of the path from u to v along the cycle is x. Prove that (s, v) = (s, u) + x. (Hint: The weight of the path from v to u along the cycle is -x.)

d. Show that if * = 0, then there exists a vertex v on the minimum mean-weight cycle such that

(Hint: Show that a shortest path to any vertex on the minimum mean-weight cycle can be extended along the cycle to make a shortest path to the next vertex on the cycle.)

e. Show that if * = 0, then

f. Show that if we add a constant t to the weight of each edge of G, then * is increased by t. Use this to show that

g. Give an O(V E)-time algorithm to compute *.

Chapter notes

Dijkstra's algorithm [55] appeared in 1959, but it contained no mention of a priority queue. The Bellman-Ford algorithm is based on separate algorithms by Bellman [22] and Ford [71]. Bellman describes the relation of shortest paths to difference constraints. Lawler [132] describes the linear-time algorithm for shortest paths in a dag, which he considers part of the folklore.

When edge weights are relatively small integers, more efficient algorithms can be used to solve the single-source shortest-paths problem. Ahuja, Mehlhorn, Orlin, and Tarjan [6] give an algorithm that runs in time on graphs with nonnegative edge weights, where W is the largest weight of any edge in the graph. They also give an easily programmed algorithm that runs in O(E + V lg W) time. For graphs with negative edge weights, the algorithm due to Gabow and Tarjan [77] runs in time, where the magnitude of the largest-magnitude weight of any edge in the graph.

Papadimitriou and Steiglitz [154] have a good discussion of the simplex method and the ellipsoid algorithm as well as other algorithms related to linear programming. The simplex algorithm for linear programming was invented by G. Danzig in 1947. Variants of simplex remain the most popular method for solving linear-programming problems. The ellipsoid algorithm is due to L. G. Khachian in 1979, based on earlier work by N. Z. Shor, D. B. Judin, and A. S. Nemirovskii. Karmarkar describes his algorithm in [115].

Go to Chapter 26 Back to Table of Contents


Intro to Algorithms: CHAPTER 26: ALL-PAIRS SHORTEST PATHS

 Return to Table of Contents

CHAPTER 26: ALL-PAIRS SHORTEST PATHS

In this chapter, we consider the problem of finding shortest paths between all pairs of vertices in a graph. This problem might arise in making a table of distances between all pairs of cities for a road atlas. As in Chapter 25, we are given a weighted, directed graph G = (V, E) with a weight function w: E [image] R that maps edges to real-valued weights. We wish to find, for every pair of vertices u, v [image] V, a shortest (least-weight) path from u to v, where the weight of a path is the sum of the weights of its constituent edges. We typically want the output in tabular form: the entry in u's row and v's column should be the weight of a shortest path from u to v.

We can solve an all-pairs shortest-paths problem by running a single-source shortest-paths algorithm [image] V [image] times, once for each vertex as the source. If all edge weights are nonnegative, we can use Dijkstra's algorithm. If we use the linear-array implementation of the priority queue, the running time is O(V3 + VE) = O(V3). The binary-heap implementation of the priority queue yields a running time of O(V E lg V), which is an improvement if the graph is sparse. Alternatively, we can implement the priority queue with a Fibonacci heap, yielding a running time of O(V2 lg V + VE).

If negative-weight edges are allowed, Dijkstra's algorithm can no longer be used. Instead, we must run the slower Bellman-Ford algorithm once from each vertex. The resulting running time is O(V2 E), which on a dense graph is O(V4). In this chapter we shall see how to do better. We shall also investigate the relation of the all-pairs shortest-paths problem to matrix multiplication and study its algebraic structure.

Unlike the single-source algorithms, which assume an adjacency-list representation of the graph, most of the algorithms in this chapter use an adjacency-matrix representation. (Johnson's algorithm for sparse graphs uses adjacency lists.) The input is an n [image] n matrix W representing the edge weights of an n-vertex directed graph G = (V, E). That is, W = (wij), where

(26.1)

Negative-weight edges are allowed, but we assume for the time being that the input graph contains no negative-weight cycles.

The tabular output of the all-pairs shortest-paths algorithms presented in this chapter is an n [image] n matrix D = (dij), where entry dij contains the weight of a shortest path from vertex i to vertex j. That is, if we let [image] (i,j) denote the shortest-path weight from vertex i to vertex j (as in Chapter 25), then dij = [image] (i,j) at termination.

To solve the all-pairs shortest-paths problem on an input adjacency matrix, we need to compute not only the shortest-path weights but also a predecessor matrix [image] = ( [image] ij ), where [image] ij is NIL if either i = j or there is no path from i to j, and otherwise [image] ij is some predecessor of j on a shortest path from i. Just as the predecessor subgraph G [image] from Chapter 25 is a shortest-paths tree for a given source vertex, the subgraph induced by the ith row of the [image] matrix should be a shortest-paths tree with root i. For each vertex i [image] V, we define the predecessor subgraph of G for i as G [image] ,i = (V [image] ,i, E [image] ,i), where

V  ,i = {j    V :   ij    NIL}    {i}

and

E  ,i = {(  ij, j): j    V  ,i and   ij    NIL} .

If G [image] ,i is a shortest-paths tree, then the following procedure, which is a modified version of the PRINT-PATH procedure from Chapter 23, prints a shortest path from vertex i to vertex j.

PRINT-ALL-PAIRS-SHORTEST-PATH(  ,i,j)

1  if i = j

2      then print i

3      else if   ij = NIL

4              then print "no path from" i "to" j "exists"

5              else PRINT-ALL-PAIRS-SHORTEST-PATH(  ,i,  ij)

6                   print j

In order to highlight the essential features of the all-pairs algorithms in this chapter, we won't cover the creation and properties of predecessor matrices as extensively as we dealt with predecessor subgraphs in Chapter 25. The basics are covered by some of the exercises.

Chapter outline

Section 26.1 presents a dynamic-programming algorithm based on matrix multiplication to solve the all-pairs shortest-paths problem. Using the technique of "repeated squaring," this algorithm can be made to run in [image] (V3 1g V) time. Another dynamic-programming algorithm, the Floyd-Warshall algorithm, is given in Section 26.2. The Floyd-Warshall algorithm runs in time [image] (V3). Section 26.2 also covers the problem of finding the transitive closure of a directed graph, which is related to the all-pairs shortest-paths problem. Johnson's algorithm is presented in Section 26.3. Unlike the other algorithms in this chapter, Johnson's algorithm uses the adjacency-list representation of a graph. It solves the all-pairs shortest-paths problem in O(V2 lg V + V E) time, which makes it a good algorithm for large, sparse graphs. Finally, in Section 26.4, we examine an algebraic structure called a "closed semiring," which allows many shortest-paths algorithms to be applied to a host of other all-pairs problems not involving shortest paths.

Before proceeding, we need to establish some conventions for adjacency-matrix representations. First, we shall generally assume that the input graph G = (V, E) has n vertices, so that n = [image] V [image]. Second, we shall use the convention of denoting matrices by uppercase letters, such as W or D, and their individual elements by subscripted lowercase letters, such as wij or dij. Some matrices will have parenthesized superscripts, as in D(m) = [image], to indicate iterates. Finally, for a given n [image] n matrix A, we shall assume that the value of n is stored in the attribute rows[A].

26.1 Shortest paths and matrix multiplication

This section presents a dynamic-programming algorithm for the all-pairs shortest-paths problem on a directed graph G = (V, E). Each major loop of the dynamic program will invoke an operation that is very similar to matrix multiplication, so that the algorithm will look like repeated matrix multiplication. We shall start by developing a [image] (V4)-time algorithm for the all-pairs shortest-paths problem and then improve its running time to [image] (V3 lg V).

Before proceeding, let us briefly recap the steps given in Chapter 16 for developing a dynamic-programming algorithm.

1. Characterize the structure of an optimal solution.

2. Recursively define the value of an optimal solution.

3. Compute the value of an optimal solution in a bottom-up fashion.

(The fourth step, constructing an optimal solution from computed information, is dealt with in the exercises.)

The structure of a shortest path

We start by characterizing the structure of an optimal solution. For the all-pairs shortest-paths problem on a graph G = (V, E), we have proved (Lemma 25.1 ) that all subpaths of a shortest path are shortest paths.Supppose that the graph is represented by an adjacency matrix W = (wij). Consider a shortest path p from vertex i to vertex j, and suppose that p containsat most m edges. Assuming that there are no negative-weight cycles, m is finite. If i = j, then p has weight 0 and no edges. If vertices i and j are distinct, then we decompose path p into [image], where path p' now contains at most m - 1 edges. Moreover, by Lemma 25.1, p' is a shortest path from i to k. Thus, by Corollary 25.2, we have [image] (i, j) = [image] (i, k) + wkj.

A recursive solution to the all-pairs shortest-paths problem

Now, let [image] be the minimum weight of any path from vertex i to vertex j that contains at most m edges. When m = 0, there is a shortest path from i to j with no edges if and only if i = j. Thus,

For m [image] 1, we compute [image] as the minimum of [image] (the weight of the shortest path from i to j consisting of at most m - 1 edges) and the minimum weight of any path from i to j consisting of at most m edges, obtained by looking at all possible predecessors k of j. Thus, we recursively define

(26.2)

The latter equality follows since wjj = 0 for all j.

What are the actual shortest-path weights [image] (i, j)? If the graph contains no negative-weight cycles, then all shortest paths are simple and thus contain at most n - 1 edges. A path from vertex i to vertex j with more than n - 1 edges cannot have less weight than a shortest path from i to j. The actual shortest-path weights are therefore given by

(26.3)

Computing the shortest-path weights bottom up

Taking as our in put the matrix W = (wij), we now compute a series of matrices D(1), D(2), . . . , D(n-1), where for m = 1, 2, . . . n - 1,, we have [image]. The final matrix D(n-1) contains the actual shortest-path weights. Observe that since [image] for all vertices i, j [image] V, we have D(1) = W.

The heart of the algorithm is the following procedure, which, given matrices D(m-1) and W, returns the matrix D(m). That is, it extends the shortest paths computed so far by one more edge.

EXTEND-SHORTEST-PATHS(D,W)

1 n    rows[D]

2 let D' = (d'ij) be an n    n matrix

3 for i    1 to n

4      do for j    1 to n

5             do d'ij

6                for k    1 to n

7                    do d'ij    min(d'ij, dik + wkj)

8 return D'

The procedure computes a matrix D' = (d'ij), which it returns at the end. It does so by computing equation (26.2) for all i and j, using D for D(m-1)and D' for D(m). (It is written without the superscripts to make its input and output matrices independent of m.) Its running time is [image] (n3) due to the three nested for loops.

We can now see the relation to matrix multiplication. Suppose we wish to compute the matrix product C = A [image] B of two n [image] n matrices A and B. Then, for i, j = 1, 2, . . . , n, we compute

(26.4)

Observe that if we make the substitutions

d(m-1)      a ,

w      b ,

d(m)     c ,

min      + ,

+

in equation (26.2), we obtain equation (26.4). Thus, if we make these changes to EXTEND-SHORTEST-PATHS and also replace [image] (the identity for min) by 0 (the identity for +), we obtain the straightforward [image] (n3)-time procedure for matrix multiplication.

MATRIX-MULTIPLY(A, B)

1 n    rows[A]

2 let C be an n    n matrix

3 for i    1 to n

4      do for j    1 to n

5             do cij    0

6                for k    1 to n

7                    do cij    cij + aik    bkj

8 return C

Returning to the all-pairs shortest-paths problem, we compute the shortest-path weights by extending shortest paths edge by edge. Letting A [image] B denote the matrix "product" returned by EXTEND-SHORTEST-PATHS(A, B), we compute the sequence of n - 1 matrices

D(1)  =   D(0)    W  =  W,

D(2)  =   D(1)    W  =  W2,

D(3)  =   D(2)    W  =  W3,

D(n-1)  =  D(n-2)    W  =  Wn-1 .

As we argued above, the matrix D(n-1) = Wn-1 contains the shortest-path weights. The following procedure computes this sequence in [image] (n4) time.

SLOW-ALL-PAIRS-SHORTEST-PATHS(W)

1  n    rows[W]

2  D(1)    W

3  for m    2 to n - 1

4       do D(m)    EXTEND-SHORTEST-PATHS(D(m-1),W)

5  return D(n-1)

Figure 26.1 shows a graph and the matrices D(m) computed by the procedure SLOW-ALL-PAIRS-SHORTEST-PATHS.

Improving the running time

Our goal, however, is not to compute all the D(m) matrices: we are interested only in matrix D(n-1). Recall that in the absence of negative-weight cycles, equation (26.3) implies D(m) = D(n-1) for all integers m [image] n - 1. We can compute D(n-1) with only [image] lg(n - 1) [image] matrix products by computing the sequence

D(1)  =   W ,

D(2)  =   W2   =  W   W,

D(4)  =   W4   =  W2   W2

D(8)  =   W8   =  W4   W4,

D(2  lg(n-1)  )  = W2  lg(n-1)     =  W2  lg(n-1)  -1    W2  lg(n-1)  -1.

Since 2 [image] lg(n-1) [image] [image] n - 1, the final product D(2 [image] 1g(n-1) [image] ) is equal to D(n-1).

The following procedure computes the above sequence of matrices by using this technique of repeated squaring.

Figure 26.1 A directed graph and the sequence of matrices D(m) computed by SLOW-ALL-PAIRS-SHORTEST-PATHS. The reader may verify that D(5) = D(4) . W is equal to D(4), and thus D(m) = D(4) for all m 4.

FASTER-ALL-PAIRS-SHORTEST-PATHS(W)

1  n    rows[W]

2  D(1)    W

3  m    1

4  while n - 1 > m

5      do D(2m)    EXTEND-SHORTEST-PATHS(D(m),D(m))

6         m    2m

7  return D(m)

In each iteration of the while loop of lines 4-6, we compute D(2m) = (D(m))2, starting with m = 1. At the end of each iteration, we double the value of m. The final iteration computes D(n-1) by actually computing D(2m) for some n - 1 [image] 2m < 2n - 2. By equation (26.3), D(2m) = D(n-1). The next time the test in line 4 is performed, m has been doubled, so now n - 1 [image] m, the test fails, and the procedure returns the last matrix it computed.

The running time of FASTER-ALL-PAIRS-SHORTEST-PATHS is [image] (n3lg n) since each of the [image] lg(n - 1) [image] matrix products takes [image] (n3) time. Observe that the code is tight, containing no elaborate data structures, and the constant hidden in the [image] -notation is therefore small.

Figure 26.2 A weighted, directed graph for use in Exercises 26.1-1, 26.2-1, and 26.3-1.

Exercises

26.1-1

Run SLOW-ALL-PAIRS-SHORTEST-PATHS on the weighted, directed graph of Figure 26.2, showing the matrices that result for each iteration of the respective loops. Then do the same for FASTER-ALL-PAIRS-SHORTEST-PATHS.

26.1-2

Why do we require that wii = 0 for all 1 [image] i [image] n?

26.1-3

What does the matrix

used in the shortest-paths algorithms correspond to in regular matrix multiplication?

26.1-4

Show how to express the single-source shortest-paths problem as a product of matrices and a vector. Describe how evaluating this product corresponds to a Bellman-Ford-like algorithm (see Section 25.3).

26.1-5

Suppose we also wish to compute the vertices on shortest paths in the algorithms of this section. Show how to compute the predecessor matrix [image] from the completed matrix D of shortest-path weights in O(n3) time.

26.1-6

The vertices on shortest paths can also be computed at the same time as the shortest-path weights. Let us define [image] to be the predecessor of vertex j on any minimum-weight path from i to j that contains at most m edges. Modify EXTEND-SHORTEST-PATHS and SLOW-ALL-PAIRS-SHORTEST-PATHS to compute the matrices [image] (1), [image] (2), . . . , [image] (n-1) as the matrices D(1), D(2), . . . , D(n-1) are computed.

26.1-7

The FASTER-ALL-PAIRS-SHORTEST-PATHS procedure, as written, requires us to store [image] lg(n - 1) [image] matrices, each with n2 elements, for a total space requirement of [image] (n2 lg n). Modify the procedure to require only [image] (n2) space by using only two n [image] n matrices.

26.1-8

Modify FASTER-ALL-PAIRS-SHORTEST-PATHS to detect the presence of a negative-weight cycle.

26.1-9

Give an efficient algorithm to find the length (number of edges) of a minimum-length negative-weight cycle in a graph.

26.2 The Floyd-Warshall algorithm

In this section, we shall use a different dynamic-programming formulation to solve the all-pairs shortest-paths problem on a directed graph G = (V, E). The resulting algorithm, known as the Floyd-Warshall algorithm, runs in [image] (V3) time. As before, negative-weight edges may be present, but we shall assume that there are no negative-weight cycles. As in Section 26.1, we shall follow the dynamic-programming process to develop the algorithm. After studying the resulting algorithm, we shall present a similar method for finding the transitive closure of a directed graph.

The structure of a shortest path

In the Floyd-Warshall algorithm, we use a different characterization of the structure of a shortest path than we used in the matrix-multiplication-based all-pairs algorithms. The algorithm considers the "intermediate" vertices of a shortest path, where an intermediate vertex of a simple path p = [image] v1, v2, . . . , vl [image] is any vertex of p other than v1 or vl, that is, any vertex in the set {v2,v3, . . . , vl-1}.

The Floyd-Warshall algorithm is based on the following observation. Let the vertices of G be V = {1, 2, . . . , n}, and consider a subset {1, 2, . . . , k} of vertices for some k. For any pair of vertices i, j [image] V, consider all paths from i to j whose intermediate vertices are all drawn from {1, 2, . . . , k}, and let p be a minimum-weight path from among them. (Path p is simple, since we assume that G contains no negative-weight cycles.) The Floyd- Warshall algorithm exploits a relationship between path p and shortest paths from i to j with all intermediate vertices in the set {1, 2, . . . , k - 1}. The relationship depends on whether or not k is an intermediate vertex of path p.

Figure 26.3 Path p is a shortest path from vertex i to vertex j, and k is the highest-numbered intermediate vertex of p. Path p1, the portion of path p from vertex i to vertex k, has all intermediate vertices in the set {1, 2, . . . , k - 1}. The same holds for path p2 from vertex k to vertex j.

If k is not an intermediate vertex of path p, then all intermediate vertices of path p are in the set {1, 2, . . . , k - 1}. Thus, a shortest path from vertex i to vertex j with all intermediate vertices in the set {1, 2, . . . , k - 1} is also a shortest path from i to j with all intermediate vertices in the set {1, 2, . . . , k}.

If k is an intermediate vertex of path p, then we break p down into [image] as shown in Figure 26.3. By Lemma 25.1, p1 is a shortest path from i to k with all intermediate vertices in the set {1, 2, . . . , k}. In fact, vertex k is not an intermediate vertex of path p1, and so p1 is a shortest path from i to k with all intermediate vertices in the set {1, 2, . . . , k - 1}. Similarly, p2 is a shortest path from vertex k to vertex j with all intermediate vertices in the set {1, 2, . . . , k - 1}.

A recursive solution to the all-pairs shortest-paths problem

Based on the above observations, we define a different recursive formulation of shortest-path estimates than we did in Section 26.1. Let [image] be the weight of a shortest path from vertex i to vertex j with all intermediate vertices in the set {1, 2, . . . , k}. When k = 0, a path from vertex i to vertex j with no intermediate vertex numbered higher than 0 has no intermediate vertices at all. It thus has at most one edge, and hence [image]. A recursive definition is given by

(26.5)

The matrix [image] gives the final answer-- [image] for all i, j [image] V--because all intermediate vertices are in the set {1, 2, . . . , n}.

Computing the shortest-path weights bottom up

Based on recurrence (26.5), the following bottom-up procedure can be used to compute the values [image] in order of increasing values of k. Its input is an n [image] n matrix W defined as in equation (26.1). The procedure returns the matrix D(n) of shortest-path weights.

Figure 26.4 shows a directed graph and the matrices D(k) computed by the Floyd-Warshall algorithm.

The running time of the Floyd-Warshall algorithm is determined by the triply nested for loops of lines 3-6. Each execution of line 6 takes O(1) time. The algorithm thus runs in time [image] (n3). As in the final algorithm in Section 26.1, the code is tight, with no elaborate data structures, and so the constant hidden in the [image] -notation is small. Thus, the Floyd-Warshall algorithm is quite practical for even moderate-sized input graphs.

Constructing a shortest path

There are a variety of different methods for constructing shortest paths in the Floyd-Warshall algorithm. One way is to compute the matrix D of shortest-path weights and then construct the predecessor matrix [image] from the D matrix. This method can be implemented to run in O(n3) time (Exercise 26.1-5). Given the predecessor matrix [image], the PRINT-ALL-PAIRS-SHORTEST-PATH procedure can be used to print the vertices on a given shortest path.

We can compute the predecessor matrix [image] "on-line" just as the Floyd-Warshall algorithm computes the matrices D(k). Specifically, we compute a sequence of matrices [image] (0), [image] (1), . . . , [image] (n), where [image] = [image] (n) and [image] is defined to be the predecessor of vertex j on a shortest path from vertex i with all intermediate vertices in the set {1, 2, . . . , k}.

We can give a recursive formulation of [image]. When k = 0, a shortest path from i to j has no intermediate vertices at all. Thus,

Figure 26.4 The sequence of matrices D(k) and (k) computed by the Floyd-Warshall algorithm for the graph in Figure 26.1.

(26.6)

For k [image] 1, if we take the path [image], then the predecessor of j we choose is the same as the predecessor of j we chose on a shortest path from k with all intermediate vertices in the set {1, 2, . . . , k - 1}. Otherwise, we choose the same predecessor of j that we chose on a shortest path from i with all intermediate vertices in the set {1, 2, . . . , k - 1}. Formally, for k [image] 1,

(26.7)

We leave the incorporation of the [image] (k) matrix computations into the FLOYD-WARSHALL procedure as Exercise 26.2-3. Figure 26.4 shows the sequence of [image] (k) matrices that the resulting algorithm computes for the graph of Figure 26.1. The exercise also asks for the more difficult task of proving that the predecessor subgraph G [image] ,i is a shortest-paths tree with root i. Yet another way to reconstruct shortest paths is given as Exercise 26.2-6.

Transitive closure of a directed graph

Given a directed graph G = (V, E) with vertex set V = {1, 2, . . . , n}, we may wish to find out whether there is a path in G from i to j for all vertex pairs i, j [image] V. The transitive closure of G is defined as the graph G* = (V, E*), where

E* = {(i, j) : there is a path from vertex i to vertex j in G} .

One way to compute the transitive closure of a graph in [image] (n3) time is to assign a weight to 1 to each edge of E and run the Floyd-Warshall algorithm. If there is a path from vertex i to vertex j, we get dij < n. Otherwise, we get dij = [image] .

There is another, similar way to compute the transitive closure of G in [image] (n3) time that can save time and space in practice. This method involves substitutions of the logical operations [image] and [image] for the arithmetic operations min and + in the Floyd-Warshall algorithm. For i, j, k = 1, 2, . . . , n, we define [image] to be 1 if there exists a path in graph G from vertex i to j with all intermediate vertices in the set {1, 2, . . . , k}, and 0 otherwise. We construct the transitive closure G* = (V, E*) by putting edge (i, j) into E* if and only if [image] = 1. A recursive definition of [image], analogous to recurrence (26.5), is

and for k [image] 1,

(26.8)

As in the Floyd-Warshall algorithm, we compute the matrices [image] in order of increasing k.

TRANSITIVE-CLOSURE(G)

Figure 26.5 shows the matrices T(k) computed by the TRANSITIVE-CLOSURE procedure on a sample graph. Like the Floyd-Warshall algorithm, the running time of the TRANSITIVE-CLOSURE procedure is [image] (n3). On some computers, though, logical operations on single-bit values execute faster than arithmetic operations on integer words of data. Moreover, because the direct transitive-closure algorithm uses only boolean values rather than integer values, its space requirement is less than the Floyd-Warshall algorithm's by a factor corresponding to the size of a word of computer storage.

In Section 26.4, we shall see that the correspondence between FLOYD-WARSHALL and TRANSITIVE-CLOSURE is more than coincidence. Both algorithms are based on a type of algebraic structure called a "closed semiring."

Exercises

26.2-1

Run the Floyd-Warshall algorithm on the weighted, directed graph of Figure 26.2. Show the matrix D(k) that results for each iteration of the outer loop.

26.2-2

As it appears above, the Floyd-Warshall algorithm requires [image] (n3) space, since we compute [image] for i, j, k = 1, 2, . . . , n. Show that the following procedure, which simply drops all the superscripts, is correct, and thus only [image] (n2) space is required.

Figure 26.5 A directed graph and the matrices T(k) computed by the transitive-closure algorithm.

FLOYD-WARSHALL'(W)

1  n    rows[W]

2  D    W

3  for k    1 to n

4       do for i    1 to n

5              do for j    1 to n

6                 dij    min(dij, dik + dkj)

7  return D

26.2-3

Modify the FLOYD-WARSHALL procedure to include computation of the [image] (k) matrices according to equations (26.6) and (26.7). Prove rigorously that for all i [image] V, the predecessor subgraph G [image] , i is a shortest-paths tree with root i. (Hint: To show that G [image] ,i is acyclic, first show that [image] implies [image]. Then, adapt the proof of Lemma 25.8.)

26.2-4

Suppose that we modify the way in which equality is handled in equation (26.7):

Is this alternative definition of the predecessor matrix [image] correct?

26.2-5

How can the output of the Floyd-Warshall algorithm be used to detect the presence of a negative-weight cycle?

26.2-6

Another way to reconstruct shortest paths in the Floyd-Warshall algorithm uses values [image] for i, j, k = 1, 2, . . . , n, where [image] is the highest-numbered intermediate vertex of a shortest path from i to j. Give a recursive formulation for [image], modify the FLOYD-WARSHALL procedure to compute the [image] values, and rewrite the PRINT-ALL-PAIRS-SHORTEST-PATH procedure to take the matrix [image] as an input. How is the matrix [image] like the s table in the matrix-chain multiplication problem of Section 16.1?

26.2-7

Give an O(V E)-time algorithm for computing the transitive closure of a directed graph G = (V, E).

26.2-8

Suppose that the transitive closure of a directed acyclic graph can be computed in [image] (V, E) time, where [image] (V, E) = [image] (V + E) and [image] is monotonically increasing. Show that the time to compute the transitive closure of a general directed graph is O( [image] (V, E)).

26.3 Johnson's algorithm for sparse graphs

Johnson's algorithm finds shortest paths between all pairs in O(V2 lg V + V E) time; it is thus asymptotically better than either repeated squaring of matrices or the Floyd-Warshall algorithm for sparse graphs. The algorithm either returns a matrix of shortest-path weights for all pairs or reports that the input graph contains a negative-weight cycle. Johnson's algorithm uses as subroutines both Dijkstra's algorithm and the Bellman-Ford algorithm, which are described in Chapter 25.

Johnson's algorithm uses the technique of reweighting, which works as follows. If all edge weights w in a graph G = (V, E) are nonnegative, we can find shortest paths between all pairs of vertices by running Dijkstra's algorithm once from each vertex; with the Fibonacci-heap priority queue, the running time of this all-pairs algorithm is O(V2 lg V + V E). If G has negative-weight edges, we simply compute a new set of nonnegative edge weights that allows us to use the same method. The new set of edge weights [image] must satisfy two important properties.

1. For all pairs of vertices u, v [image] V, a shortest path from u to v using weight function w is also a shortest path from u to v using weight function [image].

2. For all edges (u, v), the new weight [image] is nonnegative.

As we shall see in a moment, the preprocessing of G to determine the new weight function [image] can be performed in O(V E) time.

Preserving shortest paths by reweighting

As the following lemma shows, it is easy to come up with a reweighting of the edges that satisfies the first property above. We use [image] to denote shortest-path weights derived from weight function w and [image] to denote shortest-path weights derived from weight function [image].

Lemma 26.1

Given a weighted, directed graph G = (V, E) with weight function w: E [image] R, let h: V [image] R be any function mapping vertices to real numbers. For each edge (u, v) [image] E, define

(26.9)

Let p = [image] v0, vl, . . . , vk) be a path from vertex [image] 0 to vertex vk. Then, w(p) = [image] (v0, vk) if and only if [image]. Also, G has a negative-weight cycle using weight function w if and only if G has a negative-weight cycle using weight function [image].

Proof We start by showing that

(26.10)

We have

The third equality follows from the telescoping sum on the second line.

We now show by contradiction that w(p) = [image] (v0, vk) implies [image]. Suppose that there is a shorter path p' from v0 to vk using weight function [image]. Then, [image]. By equation (26.10),

which implies that w(p') < w(p). But this contradicts our assumption that p is a shortest path from u to v using w. The proof of the converse is similar.

Finally, we show that G has a negative-weight cycle using weight function w if and only if G has a negative-weight cycle using weight function [image]. Consider any cycle c = <v0, v1, . . . , vk>, where v0= vk. By equation (26.10),

and thus c has negative weight using w if and only if it has negative weight using

Producing nonnegative weights by reweighting

Our next goal is to ensure that the second property holds: we want [image] (u, v) to be nonnegative for all edges (u,v) [image] E. Given a weighted, directed graph G = (V, E) with weight function [image] : E [image] R, we make a new graph G' = (V', E'), where V' = V [image] {s} for some new vertex s [image] V and E'= E [image] {(s, [image]): v [image] V}. We extend the weight function w so that [image] (s,v) = 0 for all v [image] V. Note that because s has no edges that enter it, no shortest paths in G', other than those with source s, contain s. Moreover, G' has no negative-weight cycles if and only if G has no negative-weight cycles. Figure 26.6(a) shows the graph G' corresponding to the graph G of Figure 26.1.

Now suppose that G and G' have no negative-weight cycles. Let us define h(v) = [image] (s,v) for all v [image] V'. By Lemma 25.3, we have h(v) [image] h(u) + [image] (u, v) for all edges (u,v) [image] E'. Thus, if we define the new weights [image] according to equation (26.9), we have [image] and the second property is satisfied. Figure 26.6(b) shows the graph G' from Figure 26.6(a) with reweighted edges.

Computing all-pairs shortest paths

Johnson's algorithm to compute all-pairs shortest paths uses the Bellman-Ford algorithm (Section 25.3) and Dijkstra's algorithm (Section 25.2) as subroutines. It assumes that the edges are stored in adjacency lists. The algorithm returns the usual [image] V [image] X [image] V [image] matrix D = dij, where dij = [image] (i, j), or it reports that the input graph contains a negative-weight cycle. (In order for the indices into the D matrix to make any sense, we assume that the vertices are numbered from 1 to [image] V [image].)

Figure 26.6 Johnson's all-pairs shortest-paths algorithm run on the graph of Figure 26.1. (a) The graph G' with the original weight function w. The new vertex s is black. Within each vertex v is h(v) = (s, v). (b) Each edge (u, v) is reweighted with weight function . (c)-(g) The result of running Dijkstra's algorithm on each vertex of G using weight function . In each part, the source vertex u is black. Within each vertex v are the values and (u, v), separated by a slash. The value duv = (u, v) is equal to .

This code simply performs the actions we specified earlier. Line 1 produces G'. Line 2 runs the Bellman-Ford algorithm on G' with weight function w. If G' , and hence G, contains a negative-weight cycle, line 3 reports the problem. Lines 4-11 assume that G' contains no negative-weight cycles. Lines 4-5 set h(v) to the shortest-path weight [image] (s, v) computed by the Bellman-Ford algorithm for all v [image] V'. Lines 6-7 compute the new weights [image]. For each pair of vertices u, v [image] V, the for loop of lines 8-11 computes the shortest-path weight [image] by calling Dijkstra's algorithm once from each vertex in V. Line 11 stores in matrix entry duv the correct shortest-path weight [image] (u, v), calculated using equation (26.10). Finally, line 12 returns the completed D matrix. Figure 26.6 shows the execution of Johnson's algorithm.

The running time of Johnson's algorithm is easily seen to be O(V2 lg V + VE) if the priority queue in Dijkstra's algorithm is implemented by a Fibonacci heap. The simpler binary-heap implementation yields a running time of O(V E 1g V), which is still asymptotically faster than the Floyd-Warshall algorithm if the graph is sparse.

Exercises

26.3-1

Use Johnson's algorithm to find the shortest paths between all pairs of vertices in the graph of Figure 26.2. Show the values of h and [image] computed by the algorithm.

26.3-2

What is the purpose of adding the new vertex s to V, yielding V'?

26.3-3

Suppose that w(u, v) [image] 0 for all edges (u, v) [image] E. What is the relationship between the weight functions w and [image] ?

* 26.4 A general framework for solving path problems in directed graphs

In this section, we examine "closed semirings," an algebraic structure that yields a general framework for solving path problems in directed graphs. We start by defining closed semirings and discussing how they relate to a calculus of directed paths. We then show some examples of closed semirings and a "generic" algorithm for computing all-pairs path information. Both the Floyd-Warshall algorithm and the transitive-closure algorithm from Section 26.2 are instantiations of this generic algorithm.

Definition of closed semirings

A closed semiring is a system [image] where S is a set of elements, [image] (the summary operator) and [image] (the extension operator) are binary operations on S, and [image] are elements of S, satisfying the following eight properties:

1. is a monoid:

S is closed under [image]: a [image] b [image] S for all a, b [image] S.

is associative: a [image] (b c) = (a b) c for all a,b,c S.

Likewise, is a monoid.

3. is commutative: a b = b a for all a, b S.

4. is idempotent: a a = a for all a S.

6. If a1, a2, a3, . . . is a countable sequence of elements of S, then a1 a2 a3 . . . is well defined and in S.

7. Associativity, commutativity, and idempotence apply to infinite summaries. (Thus, any infinite summary can be rewritten as an infinite summary in which each term of the summary is included just once and the order of evaluation is arbitrary.)

A calculus of paths in directed graphs

Although the closed-semiring properties may seem abstract, they can be related to a calculus of paths in directed graphs. Suppose we are given a directed graph G = (V, E) and a labeling function : V X V S mapping all ordered pairs of vertices into some codomain S.The label of edge (u, v) E is denoted , (u, v). Since is defined over the domain V X V, the label (u, v) is usually taken as if (u, v) is not an edge of G (we shall see why in a moment).

We use the associative extension operator to extend the notion of labels to paths. The label of path p = v1, v2, . . . ,vk is

The identity serves as the label of the empty path.

As a runing example of an application of closed semirings, we shall use shortest paths with nonnegative edge weights. The codomain S is R 0 { }, where R 0 is the set of nonnegative reals, and (i, j) = wij for all i, j V. The extension operator corresponds to the arithmetic operator +, and the label of path p = v1,v2, . . . ,vk is therefore

Not surprisingly, the role of , the identity for , is taken by 0, the identity for +. We denote the empty path by , and its label is .

Because the extension operator is associative, we can define the label of the concatenation of two paths in a natural way. Given paths p1 = v1,v2, . . . ,vk and p2 = vk,vk+, . . . ,,v1 , their concatenation is

p1    p2 =   v1, v2, . . . ,vk,vk+1, . . . ,vl  ,

and the label of their concatenation is

The summary operator , which is both commutative and associative, is used to summarize path labels. That is, the value (p1) (p2) gives a summary, the semantics of which are specific to the application, of the labels of paths p1 and p2.

Our goal will be to compute, for all pairs of vertices i, j V, the summary of all path labels from i to j:

(26.11)

We require commutativity and associativity of because the order in which paths are summarized should not matter. Because we use the annihilator as the label of an ordered pair (u, v) that is not an edge in the graph, any path that attempts to take an absent edge has label .

For shortest paths, we use min as the summary operator . The identity for min is , and is indeed an annihilator for + : a + = + a = for all a R 0 { }. Absent edges have weight , and if any edge of a path has weight , so does the path.

We want the summary operator to be idempotent, because from equation (26.11), we see that should summarize the labels of a set of paths. If p is a path, then {p} {p} = {p}; if we summarize path p with itself, the resulting label should be the label of p: (p) (p) = (p).

Because we consider paths that may not be simple, there may be a countably infinite number of paths in a graph. (Each path, simple or not, has a finite number of edges.) The operator should therefore be applicable to a countably infinite number of path labels. That is, if a1, a2, a3, . . . is a countable sequence of elements in codomain S, then the label a1 a2 a3 . . . should be well defined and in S. It should not matter in which order we summarize path labels, and thus associativity and commutativity should hold for infinite summaries. Furthermore, if we summarize the same path label a a countably infinite number of times, we should get a as the result, and thus idempotence should hold for infinite summaries.

Returning to the shortest-paths example, we ask if min is applicable to an infinite sequence of values in R 0 { }. For example, is the value of min well defined? It is, if we think of the min operator as actually returning the greatest lower bound (infimum) of its arguments, in which case we get min .

To compute labels of diverging paths, we need distributivity of the extension operator over the summary operator . As shown in Figure 26.7, suppose that we have paths By distributivity, we can summarize the labels of paths p1 2 and p1 3 by computing either .

Because there may be a countably infinite number of paths in a graph, should distribute over infinite summaries as well as finite ones. Figure 26.8, for example, contains paths along with the cycle . We must be able to summarize the paths p1 p2, p1 c p2, p1 c c p2, . . . . Distributivity of over countably infinite summaries gives us

Figure 26.7 Using distributivity of . To summarize the labels of paths p1 p2 and p1 p3, we may compute either or .

Figure 26.8 Distributivity of over countably infinite summaries of . Because of cycle c, there are a countably infinite number of paths from vertex v to vertex x. We must be able to summarize the paths p1 p2, p1 c p2, p1 c c p2, . . ..

We use a special notation to denote the label of a cycle that may be traversed any number of times. Suppose that we have a cycle c with label (c) = a We may traverse c zero times for a label of , once for a label of (c) = a, twice for a label of , and so on. The label we get by summarizing this infinite number of traversals of cycle c is the closure of a, defined by

Thus, in Figure 26.8, we want to compute .

For the shortest-paths example, for any nonnegative real a R 0 { },

The interpretation of this property is that since all cycles have nonnegative weight, no shortest path ever needs to traverse an entire cycle.

Examples of closed semirings

We have already seen one example of a closed semiring, namely S1 = (R 0 { }, min, +, ,0), which we used for shortest paths with nonnegative edge weights. (As previously noted, the min operator actually returns the greatest lower bound of its arguments.) We have also shown that a* = 0 for all a R 0 { }.

We claimed, however, that even if there are negative-weight edges, the Floyd-Warshall algorithm computes shortest-path weights as long as no negative-weight cycles are present. By adding the appropriate closure operator and extending the codomain of labels to R {- , + }, we can find a closed semiring to handle negative-weight cycles. Using min for and + for , the reader may verify that the closure of a R {- , + } is

The second case (a < 0) models the situation in which we can traverse a negative-weight cycle an infinite number of times to obtain a weight of - on any path containing the cycle. Thus, the closed semiring to use for the Floyd-Warshall algorithm with negative edge weights is S2 = (R {- , + }, min, +, + , 0). (See Exercise 26.4-3.)

For transitive closure, we use the closed semiring S3 = ({0, 1}, V, , 0, 1), where (i, j) = 1 if (i, j) E, and (i, j)= 0 otherwise. Here we have 0* = 1* = 1.

A dynamic-programming algorithm for directed-path labels

Suppose we are given a directed graph G = (V, E) with labeling function : V V S. The vertices are numbered 1 through n. For each pair of vertices i, j V, we want to compute equation (26.11):

which is the result of summarizing all paths from i to j using the summary operator . For shortest paths, for example, we wish to compute

There is a dynamic-programming algorithm to solve this problem, and its form is very similar to the Floyd-Warshall algorithm and the transitive- closure algorithm. Let be the set of paths from vertex i to vertex j with all intermediate vertices in the set { 1, 2, . . . , k}. We define

Note the analogy to the definitions of in the Floyd-Warshall algorithm and in the transitive-closure algorithm. We can define recursively by

(26.12)

Recurrence (26.12) is reminiscent of recurrences (26.5) and (26.8), but with an additional factor of included. This factor represents the summary of all cycles that pass through vertex k and have all other vertices in the set {1, 2, . . . , k - 1}. (When we assume no negative-weight cycles in the Floyd-Warshall algorithm, is 0, corresponding to , the weight of an empty cycle. In the transitive-closure algorithm, the empty path from k to k gives us . Thus, for both of these algorithms, we can ignore the factor of , since it is just the identity for .) The basis of the recursive definition is

which we can see as follows. The label of the one-edge path <i, j> is simply (i, j) (which is equal to if (i, j) is not an edge in E). If, in addition, i = j, then is the label of the empty path from i to i.

The dynamic-programming algorithm computes the values in order of increasing k. It returns the matrix L(n) = .

The running time of this algorithm depends on the time to compute , , and *. If we let , and T* represent these times, then the running time of COMPUTE-SUMMARIES is , which is (n3) if each of the three operations takes O(1) time.

Exercises

26.4-1

Verify that S1 = (R 0 { }, min, +, , 0) and S3 = ({0, 1}, V, , 0, 1) are closed semirings.

26.4-2

Verify that S2 = (R {- , + }, min, +, + , 0) is a closed semiring. What is the value of a + (- ) for a R? What about (- ) + (+ )?

26.4-3

Rewrite the COMPUTE-SUMMARIES procedure to use closed semiring S2, so that it implements the Floyd-Warshall algorithm. What should be the value of - + ?

26.4-4

Is the system S4 = (R, +, , 0, 1) a closed semiring?

26.4-5

Can we use an arbitrary closed semiring for Dijkstra's algorithm? What about for the Bellman-Ford algorithm? What about for the FASTER-ALL-PAIRS-SHORTEST-PATHS procedure?

26.4-6

A trucking firm wishes to send a truck from Castroville to Boston laden as heavily as possible with artichokes, but each road in the United States has a maximum weight limit on trucks that use the road. Model this problem with a directed graph G = (V, E) and an appropriate closed semiring, and give an efficient algorithm to solve it.

Problems

26-1 Transitive closure of a dynamic graph

Suppose that we wish to maintain the transitive closure of a directed graph G = (V, E) as we insert edges into E. That is, after each edge has been inserted, we want to update the transitive closure of the edges inserted so far. Assume that the graph G has no edges initially and that the transitive closure is to be represented as a boolean matrix.

a. Show how the transitive closure G* = (V, E*) of a graph G = (V, E) can be updated in 0(V2) time when a new edge is added to G.

b. Give an example of a graph G and an edge e such that (V2) time is required to update the transitive closure after the insertion of e into G.

c. Describe an efficient algorithm for updating the transitive closure as edges are inserted into the graph. For any sequence of n insertions, your algorithm should run in total time , where ti is the time to update the transitive closure when the ith edge is inserted. Prove that your algorithm attains this time bound.

26-2 Shortest paths in -dense graphs

A graph G = (V, E) is -dense if E = (V1 + ) for some constant in the range 0 < 1. By using d-ary heaps (see Problem 7-2) in shortest-paths algorithms on -dense graphs, we can match the running times of Fibonacci-heap-based algorithms without using as complicated a data structure.

a. What are the asymptotic running times for INSERT, EXTRACT-MIN, and DECREASE-KEY, as a function of d and the number n of elements in a d-ary heap? What are these running times if we choose d = (n ) for some constant 0 < 1? Compare these running times to the amortized costs of these operations for a Fibonacci heap.

b. Show how to compute shortest paths from a single source on an -dense directed graph G = (V, E) with no negative-weight edges in O(E) time. (Hint: Pick d as a function of .)

c. Show how to solve the all-pairs shortest-paths problem on an -dense directed graph G = (V, E) with no negative-weight edges in O(V E) time.

d. Show how to solve the all-pairs shortest-paths problem in O(V E) time on an -dense directed graph G = (V, E) that may have negative-weight edges but has no negative-weight cycles.

26-3 Minimum spanning tree as a closed semiring

Let G = (V, E) be a connected, undirected graph with weight function w : E R. Let the vertex set be V = {1, 2, . . . , n}, where n = V , and assume that all edge weights w (i, j) are unique. Let T be the unique (see Exercise 24.1-6) minimum spanning tree of G. In this problem, we shall determine T by using a closed semiring, as suggested by B. M. Maggs and S. A. Plotkin. We first determine, for each pair of vertices i, j V, the minimax weight

a. Briefly justify the assertion that S = (R {- , }, min, max, , - ) is a closed semiring.

Since S is a closed semiring, we can use the COMPUTE-SUMMARIES procedure to determine the minimax weights mij in graph G. Let be the minimax weight over all paths from vertex i to vertex j with all intermediate vertices in the set {1, 2, . . . , k}.

b. Give a recurrence for , where k 0.

c. Let Tm = {(i, j) E: w(i, j) = mij}. Prove that the edges in Tm form a spanning tree of G.

d. Show that Tm = T. (Hint: Consider the effect of adding edge (i, j) to T and removing an edge on another path from i to j. Consider also the effect of removing edge (i, j) from T and replacing it with another edge.)

Chapter notes

Lawler [132] has a good discussion of the all-pairs shortest-paths problem, although he does not analyze solutions for sparse graphs. He attributes the matrix-multiplication algorithm to the folklore. The Floyd-Warshall algorithm is due to Floyd [68], who based it on a theorem of Warshall [198] that describes how to compute the transitive closure of boolean matrices. The closed-semiring algebraic structure appears in Aho, Hopcroft, and Ullman [4]. Johnson's algorithm is taken from [114].

Go to Chapter 27 Back to Table of Contents


Intro to Algorithms: CHAPTER 27: MAXIMUM FLOW

 Return to Table of Contents

CHAPTER 27: MAXIMUM FLOW

Just as we can model a road map as a directed graph in order to find the shortest path from one point to another, we can also interpret a directed graph as a "flow network" and use it to answer questions about material flows. Imagine a material coursing through a system from a source, where the material is produced, to a sink, where it is consumed. The source produces the material at some steady rate, and the sink consumes the material at the same rate. The "flow" of the material at any point in the system is intuitively the rate at which the material moves. Flow networks can be used to model liquids flowing through pipes, parts through assembly lines, current through electrical networks, information through communication networks, and so forth.

Each directed edge in a flow network can be thought of as a conduit for the material. Each conduit has a stated capacity, given as a maximum rate at which the material can flow through the conduit, such as 200 gallons of liquid per hour through a pipe or 20 amperes of electrical current through a wire. Vertices are conduit junctions, and other than the source and sink, material flows through the vertices without collecting in them. In other words, the rate at which material enters a vertex must equal the rate at which it leaves the vertex. We call this property "flow conservation," and it is the same as Kirchhoff's Current Law when the material is electrical current.

The maximum-flow problem is the simplest problem concerning flow networks. It asks, What is the greatest rate at which material can be shipped from the source to the sink without violating any capacity constraints? As we shall see in this chapter, this problem can be solved by efficient algorithms. Moreover, the basic techniques used by these algorithms can be adapted to solve other network-flow problems.

This chapter presents two general methods for solving the maximum-flow problem. Section 27.1 formalizes the notions of flow networks and flows, formally defining the maximum-flow problem. Section 27.2 describes the classical method of Ford and Fulkerson for finding maximum flows. An application of this method, finding a maximum matching in an undirected bipartite graph, is given in Section 27.3. Section 27.4 presents the preflow-push method, which underlies many of the fastest algorithms for network-flow problems. Section 27.5 covers the "lift-to-front" algorithm, a particular implementation of the preflow-push method that runs in time O(V3). Although this algorithm is not the fastest algorithm known, it illustrates some of the techniques used in the asymptotically fastest algorithms, and it is reasonably efficient in practice.

27.1 Flow networks

In this section, we give a graph-theoretic definition of flow networks, discuss their properties, and define the maximum-flow problem precisely. We also introduce some helpful notation.

Flow networks and flows

A flow network G = (V, E) is a directed graph in which each edge (u,v) [image] E has a nonnegative capacity c(u, v) [image] 0. If (u, v) [image] E, we assume that c(u, v) = 0. We distinguish two vertices in a flow network: a source s and a sink t. For convenience, we assume that every vertex lies on some path from the source to the sink. That is, for every vertex v [image] V, there is a path [image]. The graph is therefore connected, and |E| [image] |V| - 1. Figure 27.1 shows an example of a flow network.

We are now ready to define flows more formally. Let G = (V,E) be a flow network (with an implied capacity function c). Let s be the source of the network, and let t be the sink. A flow in G is a real-valued function f: V x V [image] R that satisfies the following three properties:

Capacity constraint: For all u, v [image] V, we require â (u, v) [image] c(u, v).

Skew symmetry: For all u, v [image] V, we require â (u, v) = -â (v, u).

Flow conservation: For all u [image] V - {s, t}, we require

The quantity â (u, v), which can be positive or negative, is called the net flow from vertex u to vertex v. The value of a flow f is defined as

(27.1)

that is, the total net flow out of the source. (Here, the | [image] | notation denotes flow value, not absolute value or cardinality.) In the maximum-flow problem, we are given a flow network G with source s and sink t, and we wish to find a flow of maximum value from s to t.

Before seeing an example of a network-flow problem, let us briefly explore the three flow properties. The capacity constraint simply says that the net flow from one vertex to another must not exceed the given capacity. Skew symmetry says that the net flow from a vertex u to a vertex v is the negative of the net flow in the reverse direction. Thus, the net flow from a vertex to itself is 0, since for all u [image] V, we have â (u, u) = - f (u,u), which implies that â (u, u) = 0. The flow-conservation property says that the total net flow out of a vertex other than the source or sink is 0. By skew symmetry, we can rewrite the flow-conservation property as

for all v [image] V - {s, t}. That is, the total net flow into a vertex is 0.

Figure 27.1 (a) A flow network G = (V, E) for the Lucky Puck Company's trucking problem. The Vancouver factory is the source s, and the Winnipeg warehouse is the sink t. Pucks are shipped through intermediate cities, but only c (u, v) crates per day can go from city u to city v. Each edge is labeled with its capacity. (b) A flow â in G with value |â| = 19. Only positive net flows are shown. If â (u, v) > 0, edge (u, v) is labeled by â (u, v)/c(u, v). (The slash notation is used merely to separate the flow and capacity; it does not indicate division.) If â (u,v) 0, edge (u, v) is labeled only by its capacity.

Observe also that there can be no net flow between u and v if there is no edge between them. If neither (u, v) [image] E nor (v, u) E, then c(u, v) = c(v, u) = 0. Hence, by the capacity constraint, â(u, v) [image] 0 and â (v, u) [image] 0. But since â (u, v) = - â (v, u), by skew symmetry, we have â(u, v) = â(v, u) = 0. Thus, nonzero net flow from vertex u to vertex v implies that (u, v) [image] E or (v, u) [image] E (or both).

Our last observation concerning the flow properties deals with net flows that are positive. The positive net flow entering a vertex v is defined by

(27.2)

The positive net flow leaving a vertex is defined symmetrically. One interpretation of the flow-conservation property is that the positive net flow entering a vertex other than the source or sink must equal the positive net flow leaving the vertex.

An example of network flow

A flow network can model the trucking problem shown in Figure 27.1. The Lucky Puck Company has a factory (source s) in Vancouver that manufactures hockey pucks, and it has a warehouse (sink t) in Winnipeg that stocks them. Lucky Puck leases space on trucks from another firm to ship the pucks from the factory to the warehouse. Because the trucks travel over specified routes between cities and have a limited capacity, Lucky Puck can ship at most c(u, v) crates per day between each pair of cities u and v in Figure 27.1 (a). Lucky Puck has no control over these routes and capacities and so cannot alter the flow network shown in Figure 27.1(a). Their goal is to determine the largest number p of crates per day that can be shipped and then to produce this amount, since there is no point in producing more pucks than they can ship to their warehouse.

The rate at which pucks are shipped along any truck route is a flow. The pucks emanate from the factory at the rate of p crates per day, and p crates must arrive at the warehouse each day. Lucky Puck is not concerned with how long it takes for a given puck to get from the factory to the warehouse; they care only that p crates per day leave the factory and p crates per day arrive at the warehouse. The capacity constraints are given by the restriction that the flow â(u, v) from city u to city v to be at most c(u, v) crates per day. In a steady state, the rate at which pucks enter an intermediate city in the shipping network must equal the rate at which they leave; otherwise, they would pile up. Flow conservation is therefore obeyed. Thus, a maximum flow in the network determines the maximum number p of crates per day that can be shipped.

Figure 27.1(b) shows a possible flow in the network that is represented in a way that naturally corresponds to shipments. For any two vertices u and v in the network, the net flow â(u, v) corresponds to a shipment of â(u, v) crates per day from u to v. If â(u, v) is 0 or negative, then there is no shipment from u to v. Thus, in Figure 27.1(b), only edges with positive net flow are shown, followed by a slash and the capacity of the edge.

We can understand the relationship between net flows and shipments somewhat better by focusing on the shipments between two vertices. Figure 27.2(a) shows the subgraph induced by vertices v1 and v2 in the flow network of Figure 27.1. If Lucky Puck ships 8 crates per day from v1 to v2, the result is shown in Figure 27.2(b): the net flow from v1 to v2 is 8 crates per day. By skew symmetry, we also say that the net flow in the reverse direction, from v2 to v1, is -8 crates per day, even though we do not ship any pucks from v2 to v1. In general, the net flow from v1 to v2 is the number of crates per day shipped from v1 to v2 minus the number per day shipped from v2 to v1. Our convention for representing net flows is to show only positive net flows, since they indicate the actual shipments; thus, only an 8 appears in the figure, without the corresponding -8.

Figure 27.2 Cancellation. (a) Vertices v1, and v2, with c(v1, v2) = 10 and c(v2, v1 ) = 4. (b) How we indicate the net flow when 8 crates per day are shipped from v1 to v2. (c) An additional shipment of 3 crates per day is made from v2 to v1. (d) By cancelling flow going in opposite directions, we can represent the situation in (c) with positive net flow in one direction only. (e) Another 7 crates per day is shipped from v2 to v1.

Now let's add another shipment, this time of 3 crates per day from v2 to v1. One natural representation of the result is shown in Figure 27.2(c). We now have a situation in which there are shipments in both directions between v1 and v2. We ship 8 crates per day from v1 to v2 and 3 crates per day from v2 to v1. What are the net flows between the two vertices? The net flow from v1 to v2 is 8 - 3 = 5 crates per day, and the net flow from v2 to v1 is 3 - 8 = -5 crates per day.

The situation is equivalent in its result to the situation shown in Figure 27.2(d), in which 5 crates per day are shipped from v1 to v2 and no shipments are made from v2 to v1. In effect, the 3 crates per day from v2 to v1 are cancelled by 3 of the 8 crates per day from v1 to v2. In both situations, the net flow from v1 to v2 is 5 crates per day, but in (d), actual shipments are made in one direction only.

In general, cancellation allows us to represent the shipments between two cities by a positive net flow along at most one of the two edges between the corresponding vertices. If there is zero or negative net flow from one vertex to another, no shipments need be made in that direction. That is, any situation in which pucks are shipped in both directions between two cities can be transformed using cancellation into an equivalent situation in which pucks are shipped in one direction only: the direction of positive net flow. Capacity constraints are not violated by this transformation, since we reduce the shipments in both directions, and conservation constraints are not violated, since the net flow between the two vertices is the same.

Continuing with our example, let us determine the effect of shipping another 7 crates per day from v2 to v1. Figure 27.2(e) shows the result using the convention of representing only positive net flows. The net flow from v1 to v2 becomes 5 - 7 = -2, and the net flow from v2 to v1 becomes 7 - 5 = 2. Since the net flow from v2 to v1 is positive, it represents a shipment of 2 crates per day from v2 to v1. The net flow from v1 to v2 is -2 crates per day, and since the net flow is not positive, no pucks are shipped in this direction. Alternatively, of the 7 additional crates per day from v2 to v1, we can view 5 of them as cancelling the shipment of 5 per day from v1 to v2, which leaves 2 crates as the actual shipment per day from v2 to v1.

Figure 27.3 Converting a multiple-source, multiple-sink maximum-flow problem into a problem with a single source and a single sink. (a) A flow network with five sources S = {s1, s2, s3, s4, s5} and three sinks T={t1, t2, t3}. (b) An equivalent single-source, single-sink flow network. We add a supersource s' and an edge with infinite capacity from s' to each of the multiple sources. We also add a supersink t' and an edge with infinite capacity from each of the multiple sinks to t'.

Networks with multiple sources and sinks

A maximum-flow problem may have several sources and sinks, rather than just one of each. The Lucky Puck Company, for example, might actually have a set of m factories {s1, s2, . . . ,sm} and a set of n warehouses {t1, t2, . . . , tn}, as shown in Figure 27.3(a). Fortunately, this problem is no harder than ordinary maximum flow.

We can reduce the problem of determining a maximum flow in a network with multiple sources and multiple sinks to an ordinary maximum-flow problem. Figure 27.3(b) shows how the network from (a) can be converted to an ordinary flow network with only a single source and a single sink. We add a supersource s and add a directed edge (s, si) with capacity c(s, si) = [image] for each i = 1, 2, . . . , m. We also create a new supersink t and add a directed edge (tj, t) with capacity c(tj, t) = [image] for each i = 1, 2, . . . , n. Intuitively, any flow in the network in (a) corresponds to a flow in the network in (b), and vice versa. The single source s simply provides as much flow as desired for the multiple sources si, and the single sink t likewise consumes as much flow as desired for the multiple sinks ti. Exercise 27.1-3 asks you to prove formally that the two problems are equivalent.

Working with flows

We shall be dealing with several functions (like f) that take as arguments two vertices in a flow network. In this chapter, we shall use an implicit summation notation in which either argument, or both, may be a set of vertices, with the interpretation that the value denoted is the sum of all possible ways of replacing the arguments with their members. For example, if X and Y are sets of vertices, then

As another example, the flow-conservation constraint can be expressed as the condition that f(u, V) = 0 for all u [image] V -{s, t}. Also, for convenience, we shall typically omit set braces when they would otherwise be used in the implicit summation notation. For example, in the equation f(s, V - s) = f(s, V), the term V - s means the set V - {s}.

The implicit set notation often simplifies equations involving flows. The following lemma, whose proof is left as Exercise 27.1-4, captures several of the most commonly occurring identities that involve flows and the implicit set notation.

Lemma 27.1

Let G = (V, E) be a flow network, and let f be a flow in G. Then, for X [image] V ,

f(X, X) = 0 .

For X, Y [image] V ,

f(X, Y) = -f(Y, X) .

For X, Y, Z [image] V with [image],

f(X    Y, Z) = f(X, Z) + f(Y, Z)

and

f(Z, X    Y) = f(Z, X) + f(Z, Y) .

As an example of working with the implicit summation notation, we can prove that the value of a flow is the total net flow into the sink; that is,

|f| = f(V, t) .

(27.3)

This is intuitively true, since all vertices other than the source and sink have a net flow of 0 by flow conservation, and thus the sink is the only other vertex that can have a nonzero net flow to match the source's nonzero net flow. Our formal proof goes as follows:

|f|  =  f(s, V)                    (by definition)

=  f(V, V) - f(V - s, V)      (by Lemma 27.1)

=  f(V, V - s)                (by Lemma 27.1)

=  f(V, t) + f(V, V - s - t)  (by Lemma 27.1 )

=  f(V, t)                    (by flow conservation) .

Later in this chapter, we shall generalize this result (Lemma 27.5).

Exercises

27.1-1

Given vertices u and v in a flow network, where c(u, v) = 5 and c(v, u) = 8, suppose that 3 units of flow are shipped from u to v and 4 units are shipped from v to u. What is the net flow from u to v? Draw the situation in the style of Figure 27.2.

27.1-2

Verify each of the three flow properties for the flow f shown in Figure 27.1(b).

27.1-3

Extend the flow properties and definitions to the multiple-source, multiple-sink problem. Show that any flow in a multiple-source, multiple-sink flow network corresponds to a flow of identical value in the single-source, single-sink network obtained by adding a supersource and a supersink, and vice versa.

27.1-4

Prove Lemma 27.1.

27.1-5

For the flow network G = (V, E) and flow f shown in Figure 27.1(b), find a pair of subsets X, Y [image] V for which f(X, Y) = -f(V - X, Y). Then, find a pair of subsets X, Y [image] V for which f(X, Y) [image] - f(V - X, Y).

27.1-6

Given a flow network G = (V, E), let f1 and f2 be functions from V X V to R. The flow sum f1 + f2 is the function from V x V to R defined by

(f1 + f2)(u, v) = f1(u, v) + f2(u, v)

(27.4)

for all u, v [image] V. If f1 and f2 are flows in G, which of the three flow properties must the flow sum f1 + f2 satisfy, and which might it violate?

27.1-7

Let f be a flow in a network, and let [image] be a real number. The scalar flow product [image] f is a function from V X V to R defined by

(  f)(u, v) =       f(u, v).

Prove that the flows in a network form a convex set by showing that if f1 and f2 are flows, then so is [image] f1 + (1 - [image]) f2 for all [image] in the range 0 [image] [image] [image] 1.

27.1-8

State the maximum-flow problem as a linear-programming problem.

27.1-9

The flow-network model introduced in this section supports the flow of one commodity; a multicommodity flow network supports the flow of p commodities between a set of p source vertices S = {s1, s2, . . .,sp} and a set of p sink vertices T = {t1, t2, . . .,tp}. The net flow of the ith commodity from u to v is denoted fi(u, v). For the ith commodity, the only source is si and the only sink is ti. There is flow conservation independently for each commodity: the net flow of each commodity out of each vertex is zero unless the vertex is the source or sink for the commodity. The sum of the net flows of all commodities from u to v must not exceed c(u, v), and in this way the commodity flows interact. The value of the flow of each commodity is the net flow out of the source for that commodity. The total flow value is the sum of the values of all p commodity flows. Prove that there is a polynomial-time algorithm that solves the problem of finding the maximum total flow value of a multicommodity flow network by formulating the problem as a linear program.

27.2 The Ford-Fulkerson method

This section presents the Ford-Fulkerson method for solving the maximum-flow problem. We call it a "method" rather than an "algorithm" because it encompasses several implementations with differing running times. The Ford-Fulkerson method depends on three important ideas that transcend the method and are relevant to many flow algorithms and problems: residual networks, augmenting paths, and cuts. These ideas are essential to the important max-flow min-cut theorem (Theorem 27.7), which characterizes the value of a maximum flow in terms of cuts of the flow network. We end this section by presenting one specific implementation of the Ford-Fulkerson method and analyzing its running time.

The Ford-Fulkerson method is iterative. We start with f(u, v) = 0 for all u,v [image] V, giving an initial flow of value 0. At each iteration, we increase the flow value by finding an "augmenting path," which we can think of simply as a path from the source s to the sink t along which we can push more flow, and then augmenting the flow along this path. We repeat this process until no augmenting path can be found. The max-flow min-cut theorem will show that upon termination, this process yields a maximum flow.

FORD-FULKERSON-METHOD(G, s, t)

1  initialize flow f to 0

2  while there exists an augmenting path p

3      do augment flow f along p

4  return f

Residual networks

Intuitively, given a flow network and a flow, the residual network consists of edges that can admit more net flow. More formally, suppose that we have a flow network G = (V, E) with source s and sink t. Let f be a flow in G, and consider a pair of vertices u, v [image] V. The amount of additional net flow we can push from u to v before exceeding the capacity c(u, v) is the residual capacity of (u, v), given by

cf(u, v) = c(u, v) - f(u, v) .

(27.5)

For example, if c(u, v) = 16 and f(u, v) = 11, then we can ship cf(u, v) = 5 more units of flow before we exceed the capacity constraint on edge(u, v). When the net flow f(u, v) is negative, the residual capacity cf(u, v) is greater than the capacity c(u, v). For example, if c(u, v) = 16 and f(u, v) = -4, then the residual capacity cf(u, v) is 20. We can interpret this as follows. There is a net flow of 4 units from v to u, which we can cancel by pushing a net flow of 4 units from u to v. We can then push another 16 units from u to v before violating the capacity constraint on edge (u, v). We have thus pushed an additional 20 units of flow, starting with a net flow f(u, v) = -4, before reaching the capacity constraint.

Given a flow network G = (V, E) and a flow f, the residual network of G induced by f is Gf = (V, Ef), where

Ef = {(u, v)    V X V : cf(u, v) > 0} .

That is, as promised above, each edge of the residual network, or residual edge, can admit a strictly positive net flow. Figure 27.4(a) repeats the flow network G and flow f of Figure 27.1(b), and Figure 27.4(b) shows the corresponding residual network Gf.

Notice that (u, v) may be a residual edge in Ef even if it was not an edge in E. In other words, it may very well be the case that [image]. The residual network in Figure 27.4(b) includes several such edges not in the original flow network, such as (v1, s) and (v2, v3). Such an edge (u, v) appears in Gf only if (v, u) [image] E and there is positive net flow from v to u. Because the net flow f(u, v) from u to v is negative, cf(u, v) = c(u, v) - f(u, v) is positive and (u, v) [image] Ef. Because an edge (u, v) can appear in a residual network only if at least one of (u, v) and (v, u) appears in the original network, we have the bound

|Ef|   2 |E| .

Figure 27.4 (a) The flow network G and flow f of Figure 27.1(b). (b) The residual network Gf with augmenting path p shaded; its residual capacity is cf(p) = c(v2, v3) = 4. (c) The flow in G that results from augmenting along path p by its residual capacity 4. (d) The residual network induced by the flow in (c).

Observe that the residual network Gf is itself a flow network with capacities given by cf. The following lemma shows how a flow in a residual network relates to a flow in the original flow network.

Lemma 27.2

Let G = (V, E) be a flow network with source s and sink t, and let f be a flow in G. Let Gf be the residual network of G induced by f, and let f' be a flow in Gf. Then, the flow sum f + f' defined by equation (27.4) is a flow in G with value |f + f'| = |f| + |f'|.

Proof We must verify that skew symmetry, the capacity constraints, and flow conservation are obeyed. For skew symmetry, note that for all u, v [image] V, we have

(f + f')(u, v)  =  f(u, v) + f'(u, v)

=  -f(v, u) - f'(v, u)

=  -(f(v, u) + f'(v, u))

=  -(f + f')(v, u).

For the capacity constraints, note that f'(u, v) [image] cf(u, v) for all u, v [image] V. By equation (27.5), therefore,

(f + f')(u, v)  =  f(u, v) + f'(u, v)

    f(u, v) + (c(u, v) - f(u, v))

=  c(u, v) .

For flow conservation, note that for all u [image] V - {s, t}, we have

Finally, we have

Augmenting paths

Given a flow network G = (V, E) and a flow f, an augmenting path p is a simple path from s to t in the residual network Gf. By the definition of the residual network, each edge (u, v) on an augmenting path admits some additional positive net flow from u to v without violating the capacity constraint on the edge.

The shaded path in Figure 27.4(b) is an augmenting path. Treating the residual network Gf in the figure as a flow network, we can ship up to 4 units of additional net flow through each edge of this path without violating a capacity constraint, since the smallest residual capacity on this path is cf(v2, v3) = 4. We call the maximum amount of net flow that we can ship along the edges of an augmenting path p the residual capacity of p, given by

cf(p) = min{cf(u, v): (u, v) is on p}.

The following lemma, whose proof is left as Exercise 27.2-7, makes the above argument more precise.

Lemma 27.3

Let G = (V, E) be a flow network, let â be a flow in G, and let p be an augmenting path in Gâ. Define a function âp : V x V [image] R by

(27.6)

Then, âp is a flow in Gâ with value |âp| = câ(p) > 0.

The following corollary shows that if we add âp to â, we get another flow in G whose value is closer to the maximum. Figure 27.4(c) shows the result of adding âp in Figure 27.4(b) to â from Figure 27.4(a).

Corollary 27.4

Let G = (V, E) be a flow network, let â be a flow in G, and let p be an augmenting path in Gâ. Let âp be defined as in equation (27.6). Define a function â' : V x V [image] R by â' = â + âp. Then, â' is a flow in G with value | â' | = | â | + |âp| > | â |.

Proof Immediate from Lemmas 27.2 and 27.3.

Cuts of flow networks

The Ford-Fulkerson method repeatedly augments the flow along augmenting paths until a maximum flow has been found. The max-flow min-cut theorem, which we shall prove shortly, tells us that a flow is maximum if and only if its residual network contains no augmenting path. To prove this theorem, though, we must first explore the notion of a cut of a flow network.

A cut (S, T) of flow network G = (V, E) is a partition of V into S and T = V - S such that s [image] S and t [image] T. (This definition is like the definition of "cut" that we used for minimum spanning trees in Chapter 24, except that here we are cutting a directed graph rather than an undirected graph, and we insist that s [image] S and t [image] T.) If f is a flow, then the net flow across the cut (S, T) is defined to be â(S, T). The capacity of the cut (S, T) is c(S, T).

Figure 27.5 shows the cut ({s, vl, v2}, {v3, v4, t}) in the flow network of Figure 27.1(b). The net flow across this cut is

â(v1, v3) + â(v2, v3) + â(v2, v4)  =  12 + (-4) + 11

=  19,

and its capacity is

c(vl, v3) + c(v2, v4)  =  12 + 14

=  26.

Figure 27.5 A cut (S, T) in the flow network of Figure 27.1(b), where S = {s, v1, v2 } and T = {v3, v4, t}. The vertices in S are black, and the vertices in T are white. The net flow across (S, T) is â(S, T) = 19, and the capacity is c(S, T) = 26.

Observe that the net flow across a cut can include negative net flows between vertices, but that the capacity of a cut is composed entirely of non-negative values.

The following lemma shows that the value of a flow in a network is the net flow across any cut of the network.

Lemma 27.5

Let â be a flow in a flow network G with source s and sink t, and let (S, T) be a cut of G. Then, the net flow across (S, T) is â(S, T) = | â |.

Proof Using Lemma 27.1 extensively, we have

â(S, T)  =  â(S, V) - â(S, S)

=  â(S, V)

=  â(s, V) + â(S - s, V)

=  â(s, V)

=  | â | .

An immediate corollary to Lemma 27.5 is the result we proved earlier--equation (27.3)--that the value of a flow is the net flow into the sink.

Another corollary to Lemma 27.5 shows how cut capacities can be used to bound the value of a flow.

Corollary 27.6

The value of any flow â in a flow network G is bounded from above by the capacity of any cut of G.

Proof Let (S, T) be any cut of G and let â be any flow. By Lemma 27.5 and the capacity constraints,

|â|  =  â(S, T)

We are now ready to prove the important max-flow min-cut theorem.

Theorem 27.7

If â is a flow in a flow network G = (V, E) with source s and sink t, then the following conditions are equivalent:

1. â is a maximum flow in G.

2. The residual network Gâ contains no augmenting paths.

3. | â | = c(S, T) for some cut (S, T) of G.

Proof (1) [image] (2): Suppose for the sake of contradiction that â is a maximum flow in G but that Gâ has an augmenting path p. Then, by Corollary 27.4, the flow sum â + âp, where âp is given by equation (27.6), is a flow in G with value strictly greater than | â |, contradicting the assumption that â is a maximum flow.

(2) [image] (3): Suppose that Gâ has no augmenting path, that is, that Gâ contains no path from s to t. Define

S = {v [image] V : there exists a path from s to [image] in Gâ}

and T = V - S. The partition (S, T) is a cut: we have s [image] S trivially and t [image] S because there is no path from s to T in Gâ. For each pair of vertices u and v such that u [image] S and v [image] T, we have â(u, v) = c(u, v), since otherwise (u, v) [image] Eâ and v is in set S. By Lemma 27.5, therefore, | â | = â(S, T) = c(S, T).

(3) [image] (1): By Corollary 27.6, | â | [image] c(S, T) for all cuts (S, T). The condition | â | = c(S, T) thus implies that â is a maximum flow.

The basic Ford-Fulkerson algorithm

In each iteration of the Ford-Fulkerson method, we find any augmenting path p and augment flow â along p by the residual capacity câ(p). The following implementation of the method computes the maximum flow in a graph G = (V, E) by updating the net flow â[u, v] between each pair u, v of vertices that are connected by an edge.1 If u and v are not connected by an edge in either direction, we assume implicitly that â[u, v] = 0. The code assumes that the capacity from u to v is provided by a constant-time function c(u, v), with c(u, v) = 0 if (u, v) [image] E. (In a typical implementation, c(u, v) might be derived from fields stored within vertices and their adjacency lists.) The residual capacity câ(u, v) is computed in accordance with the formula (27.5). The expression câ(p) in the code is actually just a temporary variable that stores the residual capacity of the path p.

1We use square brackets when we treat an identifier--such as â--as a mutable field, and we use parentheses when we treat it as a function.

FORD-FULKERSON(G, s, t)

1  for each edge (u, v)    E[G]

2       do â[u, v]    0

3          â[v, u]    0

4  while there exists a path p from s to t in the residual network Gâ

5      do câ(p)    min {câ(u, v) : (u, v) is in p}

6         for each edge (u, v) in p

7             do â[u, v]    â[u, v]+câ(p)

8                â[v, u]    - â[u, v]

The FORD-FULKERSON algorithm simply expands on the FORD-FULKER-SON-METHOD pseudocode given earlier. Figure 27.6 shows the result of each iteration in a sample run. Lines 1-3 initialize the flow â to 0. The while loop of lines 4-8 repeatedly finds an augmenting path p in Gâ and augments flow â along p by the residual capacity câ(p). When no augmenting paths exist, the flow â is a maximum flow.

Analysis of Ford-Fulkerson

The running time of FORD-FULKERSON depends on how the augmenting path p in line 4 is determined. If it is chosen poorly, the algorithm might not even terminate: the value of the flow will increase with successive augmentations, but it need not even converge to the maximum flow value. If the augmenting path is chosen by using a breadth-first search (Section 23.2), however, the algorithm runs in polynomial time. Before proving this, however, we obtain a simple bound for the case in which the augmenting path is chosen arbitrarily and all capacities are integers.

Most often in practice, the maximum-flow problem arises with integral capacities. If the capacities are rational numbers, an appropriate scaling transformation can be used to make them all integral. Under this assumption, a straightforward implementation of FORD-FULKERSON runs in time O(E | â* |), where â* is the maximum flow found by the algorithm. The analysis is as follows. Lines 1-3 take time [image] (E). The while loop of lines 4-8 is executed at most | â* | times, since the flow value increases by at least one unit in each iteration.

The work done within the while loop can be made efficient if we efficiently manage the data structure used to implement the network G = (V, E). Let us assume that we keep a data structure corresponding to a directed graph G' = (V, E'), where E' = {(u, v) : (u, v) [image] E or (v, u) [image] E}. Edges in the network G are also edges in G', and it is therefore a simple matter to maintain capacities and flows in this data structure. Given a flow â on G, the edges in the residual network Gâ consist of all edges (u, v) of G' such that c(u, v) - â[u, v] [image] 0. The time to find a path in a residual network is therefore O(E') = O(E) if we use either depth-first search or breadth-first search. Each iteration of the while loop thus takes O(E) time, making the total running time of FORD-FULKERSON O(E | â* |).

Figure 27.6 The execution of the basic Ford-Fulkerson algorithm. (a)-(d) Successive iterations of the while loop. The left side of each part shows the residual network Gâ from line 4 with a shaded augmenting path p. The right side of each part shows the new flow â that results from adding âp to â. The residual network in (a) is the input network G. (e) The residual network at the last while loop test. It has no augmenting paths, and the flow â shown in (d) is therefore a maximum flow.

Figure 27.7 (a) A flow network for which FORD-FULKERSON can take (E |â*|) time, where â* is a maximum flow, shown here with | â* | = 2,000,000. An augmenting path with residual capacity 1 is shown. (b) The resulting residual network. Another augmenting path with residual capacity 1 is shown. (c) The resulting residual network.

When the capacities are integral and the optimal flow value | â* | is small, the running time of the Ford-Fulkerson algorithm is good. Figure 27.7(a) shows an example of what can happen on a simple flow network for which | â* | is large. A maximum flow in this network has value 2,000,000: 1,000,000 units of flow traverse the path s [image] u [image] t, and another 1,000,000 units traverse the path s [image] v [image] t. If the first augmenting path found by FORD-FULKERSON is s [image] u [image] v [image] t, shown in Figure 27.7(a), the flow has value 1 after the first iteration. The resulting residual network is shown in Figure 27.7(b). If the second iteration finds the augmenting path s [image] v [image] u [image] t, as shown in Figure 27.7(b), the flow then has value 2. Figure 27.7(c) shows the resulting residual network. We can continue, choosing the augmenting path s [image] u [image] v [image] t in the odd-numbered iterations and the augmenting path s [image] v [image] u [image] t in the even-numbered iterations. We would perform a total of 2,000,000 augmentations, increasing the flow value by only 1 unit in each.

The bound on FORD-FULKERSON can be improved if we implement the computation of the augmenting path p in line 4 with a breadth-first search, that is, if the augmenting path is a shortest path from s to t in the residual network, where each edge has unit distance (weight). We call the Ford-Fulkerson method so implemented the Edmonds-Karp algorithm. We now prove that the Edmonds-Karp algorithm runs in O(V E2) time.

The analysis depends on the distances to vertices in the residual network Gf. The following lemma uses the notation [image] [image] (u, v) for the shortest-path distance from u to v in Gf , where each edge has unit distance.

Lemma 27.8

If the Edmonds-Karp algorithm is run on a flow network G = (V, E) with source s and sink t, then for all vertices v [image] V - {s, t}, the shortest-path distance [image] [image] (s, v) in the residual network Gf increases monotonically with each flow augmentation.

Proof Suppose for the purpose of contradiction that for some vertex v [image] V - {s, t}, there is a flow augmentation that causes [image] [image] (s, v) to decrease. Let f be the flow just before the augmentation, and let [image] ' be the flow just afterward. Then,

    '(s, v) <     (s, v).

We can assume without loss of generality that [image] [image] '(s, v) [image] [image] [image] '(s, u) for all vertices u [image] V - {s, t} such that [image] [image] '(s, u) < [image] [image] (s, u). Equivalently, we can assume that for all vertices u [image] V - {s, t},

    '(s, u) <     '(s, v) implies     (s, u)        '(s, u).

(27.7)

We now take a shortest path p' in G [image] ' of the form [image] and consider the vertex u that precedes v on this path. We must have [image] [image] '(s, u) = [image] [image] '(s, v) - 1 by Corollary 25.2, since (u, v) is an edge on p', which is a shortest path from s to v. By our assumption (27.7), therefore,

    (s, u)        '(s, u) .

With vertices v and u thus established, we can consider the net flow â from u to v before the augmentation of flow in G [image]. If â[u, v] < c(u, v), then we have

    (s, v)          (s, u) + 1    (by Lemma 25.3)

         '(s, u) + 1

=       '(s, v) ,

which contradicts our assumption that the flow augmentation decreases the distance from s to v.

Thus, we must have [image] [u, v] = c(u, v), which means (u, v) [image] E [image] . Now, the augmenting path p that was chosen in G [image] to produce G [image] ' must contain the edge (v, u) in the direction from v to u, since (u, v) [image] E [image] ', (by supposition) and (u, v) [image] E [image] as we have just shown. That is, augmenting flow along the path p pushes flow back along (u, v), and v appears before u on p. Since p is a shortest path from s to t, its subpaths are shortest paths (Lemma 25.1), and thus we have [image] [image] (s, u) = [image] [image] (s, v) + 1. Consequently,

    (s, v)  =      (s, u) - 1

        '(s, u) - 1

=      '(s, v) - 2

<      '(s, v) ,

which contradicts our initial assumption.

The next theorem bounds the number of iterations of the Edmonds-Karp algorithm.

Theorem 27.9

If the Edmonds-Karp algorithm is run on a flow network G = (V, E) with source s and sink t, then the total number of flow augmentations performed by the algorithm is at most O(V E).

Proof We say that an edge (u, v) in a residual network G [image] is critical on an augmenting path p if the residual capacity of p is the residual capacity of (u, v), that is, if c [image] (p) = c [image] (u, v). After we have augmented flow along an augmenting path, any critical edge on the path disappears from the residual network. Moreover, at least one edge on any augmenting path must be critical.

Let u and v be vertices in V that are connected by an edge in E. How many times can (u, v) be a critical edge during the execution of the Edmonds-Karp algorithm? Since augmenting paths are shortest paths, when (u, v) is critical for the first time, we have

    (s, v) =     (s, u) + 1 .

Once the flow is augmented, the edge (u, v) disappears from the residual network. It cannot reappear later on another augmenting path until after the net flow from u to v is decreased, and this only happens if (v, u) appears on an augmenting path. If [image] ' is the flow in G when this event occurs, then we have

    '(s, u) =     '(s, v) + 1.

Since [image] [image] (s, v) [image] [image] [image] '(s, v) by Lemma 27.8, we have

    '(s, u)  =      '(s, v) + 1

        (s, v) + 1

=      (s, u) + 2.

Consequently, from the time (u, v) becomes critical to the time when it next becomes critical, the distance of u from the source increases by at least 2. The distance of u from the source is initially at least 1, and until it becomes unreachable from the source, if ever, its distance is at most [image] V [image] - 2. Thus, (u, v) can become critical at most O(V) times. Since there are O(E) pairs of vertices that can have an edge between them in a residual graph, the total number of critical edges during the entire execution of the Edmonds-Karp algorithm is O(V E). Each augmenting path has at least one critical edge, and hence the theorem follows.

Since each iteration of FORD-FULKERSON can be implemented in O(E) time when the augmenting path is found by breadth-first search, the total running time of the Edmonds-Karp algorithm is O(V E2). The algorithm of Section 27.4 gives a method for achieving an O(V2 E) running time, which forms the basis for the O(V3)-time algorithm of Section 27.5.

Exercises

27.2-1

In Figure 27.1(b), what is the flow across the cut ({s, v2, v4} , {v1, v3, t})? What is the capacity of this cut?

27.2-2

Show the execution of the Edmonds-Karp algorithm on the flow network of Figure 27.1(a).

27.2-3

In the example of Figure 27.6, what is the minimum cut corresponding to the maximum flow shown? Of the augmenting paths appearing in the example, which two cancel flow that was previously shipped?

27.2-4

Prove that for any pair of vertices u and v and any capacity and flow functions c and f, we have c [image] (u, v) + c [image] (v, u) = c(u, v) + c(v, u).

27.2-5

Recall that the construction in Section 27.1 that converts a multisource, multisink flow network into a single-source, single-sink network adds edges with infinite capacity. Prove that any flow in the resulting network has a finite value if the edges of the original multisource, multisink network have finite capacity.

27.2-6

Suppose that each source si in a multisource, multisink problem produces exactly pi units of flow, so that [image] (si,V) = pi. Suppose also that each sink tj consumes exactly qj units, so that [image] (V, tj) = qj, where [image] ipi = [image] jqj. Show how to convert the problem of finding a flow f that obeys these additional constraints into the problem of finding a maximum flow in a single-source, single-sink flow network.

27.2-7

Prove Lemma 27.3.

27.2-8

Show that a maximum flow in a network G = (V, E) can always be found by a sequence of at most |E| augmenting paths. (Hint: Determine the paths after finding the maximum flow.)

27.2-9

The edge connectivity of an undirected graph is the minimum number k of edge that must be removed to disconnect the graph. For example, the edge connectivity of a tree is 1, and the edge connectivity of a cyclic chain of vertices is 2. Show how the edge connectivity of an undirected graph G = (V, E) can be determined by running a maximum-flow algorithm on at most |V| flow network, each having O(V) vertices and O(E) edges.

27.2-10

Show that the Edmonds-Karp algorithm terminates after at most |V| |E| /4 iterations. (Hint: For any edge (u, v), consider how both [image] (s, u) and [image] (u, t) change between times at which (u, v) is critical.)

27.3 Maximum bipartite matching

Some combinatorial problems can easily be cast as maximum-flow problems. The multiple-source, multiple-sink maximum-flow problem from Section 27.1 gave us one example. There are other combinatorial problems that seem on the surface to have little to do with flow networks, but can in fact be reduced to a maximum-flow problem. This section presents one such problem: finding a maximum matching in a bipartite graph (see Section 5.4). In order to solve this problem, we shall take advantage of an integrality property provided by the Ford-Fulkerson method. We shall also see that the Ford-Fulkerson method can be made to solve the maximum-bipartite-matching problem on a graph G = (V, E) in O(V E) time.

The maximum-bipartite-matching problem

Given an undirected graph G = (V, E), a matching is a subset of edges M [image] E such that for all vertices v [image] V, at most one edge of M is incident on v. We say that a vertex v [image] V is matched by matching M if some edge in M is incident on v; otherwise, v is unmatched. A maximum matching is a matching of maximum cardinality, that is, a matching M such that for any matching M', we have |M| [image] |M'|. In this section, we shall restrict our attention to finding maximum matchings in bipartite graphs. We assume that the vertex set can be partitioned into V = L [image] R, where L and R are disjoint and all edges in E go between L and R. Figure 27.8 illustrates the notion of a matching.

Figure 27.8 A bipartite graph G = (V, E) with vertex partition V = L R. (a) A matching with cardinality 2. (b) A maximum matching with cardinality 3.

The problem of finding a maximum matching in a bipartite graph has many practical applications. As an example, we might consider matching a set L of machines with a set R of tasks to be performed simultaneously. We take the presence of edge (u, v) in E to mean that a particular machine u [image] L is capable of performing a particular task v [image] R. A maximum matching provides work for as many machines as possible.

Finding a maximum bipartite matching

We can use the Ford-Fulkerson method to find a maximum matching in an undirected bipartite graph G = (V, E) in time polynomial in |V| and |E|. The trick is to construct a flow network in which flows correspond to matchings, as shown in Figure 27.9. We define the corresponding flow network G' = (V', E') for the bipartite graph G as follows. We let the sources s and sink t be new vertices not in V, and we let V'= V [image] {s, t}. If the vertex partition of G is V = L [image] R, the directed edges of G' are given by

E'  =  {(s, u): u    L}

   {(u, v): u    L, v    R, and (u, v)    E}

   {(v, t): v    R} .

To complete the construction, we assign unit capacity to each edge in E'.

The following theorem shows that a maching in G corresponds directly to a flow in G's corresponding flow network G'. We say that a flow [image] on a flow network G = (V, E) is integer-valued if [image] (u, v) is an integer for all (u, v) [image] V [image] V.

Figure 27.9 The flow network corresponding to a bipartite graph. (a) The bipartite graph G = (V, E) with vertex partition V = L R from Figure 27.8. A maximum matching is shown by shaded edges. (b) The corresponding flow network G' with a maximum flow shown. Each edge has unit capcity. Shaded edges have a flow of 1, and all other edges carry no flow. The shaded edges from L to R correspond to those in a maximum matching of the bipartite graph.

Lemma 27.10

Let G = (V, E) be a bipartite graph with vertex partition V = L [image] R, and let G'= (V', E') be its corresponding flow network. If M is a matching in G, then there is an integer-valued flow [image] in G' with value |â| = |M|. Conversely, if f is an integer-valued flow in G', then there is a matching M in G with cardinality |M| = |â|.

Proof We first show that a matching M in G corresponds to an integer-valued flow in G'. Define f as follows. If (u, v) [image] M, then f(s, u) = f(u, v) = f(v, t) = 1 and f(u, s) = f(v, u) = f(t, v) = -1. For all other edges (u, v) [image] E', we define f(u, v) = 0.

Intuitively, each edge (u, v) [image] M corresponds to 1 unit of flow in G' that traverses the path s [image] u [image] v [image] [image] t. Moreover, the paths induced by edges in m are vertex-disjoint, except for s and t. To verify that f indeed satisfies skew symmetry, the capacity constraints, and flow conservation, we need only observe that f can be obtained by flow augmentation along each such path. The new flow across cut (L [image] {s}, R [image] {t}) is equal to |M|; thus, by Lemma 27.5, the value of the flow is |â| = |M|.

To prove the converse, let f be an integer-valued flow in G' and let

M = {(u, v): u    L, v    R, and f(u, v) > 0} .

Each vertex u [image] L has only one entering edge, namely (s, u), and its capacity is 1. Thus, each u [image] L has at most one unit of positive net flow entering it. Since f is integer-valued, for each u [image] L, 1 unit of positive net flow enters u if and only if there is exactly one vertex v [image] R such that f(u, v) = 1. Thus, at most one edge leaving each u [image] L carries positive net flow. A symmetric argument can be made for each v [image] R. The set M defined in the statement of the theorem is therefore a matching.

To see that |M| = |f|, observe that for every matched vertex u [image] L, we have f(s, u) = 1, and for every edge (u, v) [image] E - M, we have f(u,v) = 0. Consequently, using Lemma 27.1, skew symmetry, and there being no edges from L to t, we obtain

|M|  =  â(L,R)

=  â(L, V') - â(L, L) - â(L, s) - â(L, t)

=  0 - 0 + â(s, L) - 0

=  â(s, V')

=  |â| .

It is intuitive that a maximum matching in a bipartite graph G corresponds to a maximum flow in its corresponding flow network G'. Thus, we can compute a maximum matching in G by running a maximum-flow algorithm on G'. The only hitch in this reasoning is that the maximum-flow algorithm might return a flow in G' that consists of nonintegral amounts. The following theorem shows that if we use the Ford-Fulkerson method, this difficulty cannot arise.

Theorem 27.11

If the capacity function c takes on only integral values, then the maximum flow â produced by the Ford-Fulkerson method has the property that |â| is integer-valued. Moreover, for all vertices u and v, the value of â(u, v) is an integer.

Proof The proof is by induction on the number of iterations. We leave it as Exercise 27.3-2.

We can now prove the following corollary to Lemma 27.10.

Corollary 27.12

The cardinality of a maximum matching in a bipartite graph G is the value of a maximum flow in its corresponding flow network G'.

Proof We use the nomenclature from Lemma 27.10. Suppose that M is a maximum matching in G and that the corresponding flow f in G' is not maximum. Then there is a maximum flow f' in G' such that |f'| > |f|. Since the capacities in G' are integer-valued, by Theorem 27.11, so is f'. Thus, f' corresponds to a matching M' in G with cardinality |M'| = |f'| > |f| = |M|, contradicting the maximality of M. In a similar manner, we can show that if f is a maximum flow in G', its corresponding matching is a maximum matching on G.

Thus, given a bipartite undirected graph G, we can find a maximum matching by creating the flow network G', running the Ford-Fulkerson method, and directly obtaining a maximum matching M from the integer-valued maximum flow f found. Since any matching in a bipartite graph has cardinality at most min(L, R) = O(V), the value of the maximum flow in G' is O(V). We can therefore find a maximum matching in a bipartite graph in time O(V E).

Exercises

27.3-1

Run the Ford-Fulkerson algorithm on the flow network in Figure 27.9(b) and show the residual network after each flow augmentation. Number the vertices in L top to bottom from 1 to 5 and in R top to bottom from 6 to 9. For each iteration, pick the augmenting path that is lexicographically smallest.

27.3-2

Prove Theorem 27.11.

27.3-3

Let G = (V, E) be a bipartite graph with vertex partition V = L [image] R, and let G' be its corresponding flow network. Give a good upper bound on the length of any augmenting path found in G' during the execution of FORD-FULKERSON.

27.3-4

A perfect matching is a matching in which every vertex is matched. Let G = (V, E) be an undirected bipartite graph with vertex partition V = L [image] R, where |L| = |R|. For any X [image] V, define the neighborhood of X as

N(X) = {y   V : (x, y)    E for some x    X},

that is, the set of vertices adjacent to some member of X. Prove Hall's theorem: there exists a perfect matching in G if and only if |A| [image] |N(A)| for every subset A [image] L.

27.3-5

A bipartite graph G = (V, E), where V = L [image] R, is d-regular if every vertex v [image] V has degree exactly d. Every d-regular bipartite graph has |L| = |R|. Prove that every d-regular bipartite graph has a matching of cardinality |L| by arguing that a minimum cut of the corresponding flow network has capacity |L|.

* 27.4 Preflow-push algorithms

In this section, we present the "preflow-push" approach to computing maximum flows. The fastest maximum-flow algorithms to date are preflow-push algorithms, and other flow problems, such as the minimum-cost flow problem, can be solved efficiently by preflow-push methods. This section introduces Goldberg's "generic" maximum-flow algorithm, which has a simple implementation that runs in O(V2 E) time, thereby improving upon the O(V E2) bound of the Edmonds-Karp algorithm. Section 27.5 refines the generic algorithm to obtain another preflow-push algorithm that runs in O(V3) time.

Preflow-push algorithms work in a more localized manner than the Ford-Fulkerson method. Rather than examine the entire residual network G = (V, E) to find an augmenting path, preflow-push algorithms work on one vertex at a time, looking only at the vertex's neighbors in the residual network. Furthermore, unlike the Ford-Fulkerson method, preflow-push algorithms do not maintain the flow-conservation property throughout their execution. They do, however, maintain a preflow, which is a function f : V X V [image] R that satisfies skew symmetry, capacity constraints, and the following relaxation of flow conservation: f(V, u) [image] 0 for all vertices u [image] V - {s}. That is, the net flow into each vertex other than the source is nonnegative. We call the net flow into a vertex u the excess flow into u, given by

e(u) = f(V, u) .

(27.8)

We say that a vertex u [image] V - {s, t} is overflowing if e(u) > 0.

We shall start this section by describing the intuition behind the preflow-push method. We shall then investigate the two operations employed by the method: "pushing" preflow and "lifting" a vertex. Finally, we shall present a generic preflow-push algorithm and analyze its correctness and running time.

Intuition

The intuition behind the preflow-push method is probably best understood in terms of fluid flows: we consider a flow network G = (V, E) to be a system of interconnected pipes of given capacities. Applying this analogy to the Ford-Fulkerson method, we might say that each augmenting path in the network gives rise to an additional stream of fluid, with no branch points, flowing from the source to the sink. The Ford-Fulkerson method iteratively adds more streams of flow until no more can be added.

The generic preflow-push algorithm has a somewhat different intuition. As before, directed edges correspond to pipes. Vertices, which are pipe junctions, have two interesting properties. First, to accommodate excess flow, each vertex has an outflow pipe leading to an arbitrarily large reservoir that can accumulate fluid. Second, each vertex, its reservoir, and all its pipe connections are on a platform whose height increases as the algorithm progresses.

Vertex heights determine how flow is pushed: we only push flow downhill, that is, from a higher vertex to a lower vertex. There may be positive net flow from a lower vertex to a higher vertex, but operations that push flow always push it downhill. The height of the source is fixed at |V|, and the height of the sink is fixed at 0. All other vertex heights start at 0 and increase with time. The algorithm first sends as much flow as possible downhill from the source toward the sink. The amount it sends is exactly enough to fill each outgoing pipe from the source to capacity; that is, it sends the capacity of the cut (s, V - s). When flow first enters an intermediate vertex, it collects in the vertex's reservoir. From there, it is eventually pushed downhill.

It may eventually happen that the only pipes that leave a vertex u and are not already saturated with flow connect to vertices that are on the same level as u or are uphill from u. In this case, to rid an overflowing vertex u of its excess flow, we must increase its height--an operation called "lifting" vertex u. Its height is increased to one unit more than the height of the lowest of its neighbors to which it has an unsaturated pipe. After a vertex is lifted, therefore, there is at least one outgoing pipe through which more flow can be pushed.

Eventually, all the flow that can possibly get through to the sink has arrived there. No more can arrive, because the pipes obey the capacity constraints; the amount of flow across any cut is still limited by the capacity of the cut. To make the preflow a "legal" flow, the algorithm then sends the excess collected in the reservoirs of overflowing vertices back to the source by continuing to lift vertices to above the fixed height |V| of the source. (Shipping the excess back to the source is actually accomplished by canceling the flows that cause the excess.) As we shall see, once all the reservoirs have been emptied, the preflow is not only a "legal" flow, it is also a maximum flow.

The basic operations

From the preceding discussion, we see that there are two basic operations performed by a preflow-push algorithm: pushing flow excess from a vertex to one of its neighbors and lifting a vertex. The applicability of these operations depends on the heights of vertices, which we now define precisely.

Let G = (V, E) be a flow network with source s and sink t, and let f be a preflow in G. A function h: V [image] N is a height function if h(s) = |V|,

h(t) = 0, and

h(u)    h(v) + 1

for every residual edge (u, v) [image] Ef . We immediately obtain the following lemma.

Lemma 27.13

Let G = (V, E) be a flow network, let f be a preflow in G, and let h be a height function on V. For any two vertices u, v [image] V, if h(u) > h(v) + 1, then (u, v) is not an edge in the residual graph.

The basic operation PUSH(u, v) can be applied if u is an overflowing vertex, cf (u, v) > 0, and h(u) = h(v) + 1. The pseudocode below updates the preflow f in an implied network G = (V, E). It assumes that the capacities are given by a constant-time function c and that residual capacities can also be computed in constant time given c and f . The excess flow stored at a vertex u is maintained as e[u], and the height of u is maintained as h[u]. The expression df(u, v) is a temporary variable that stores the amount of flow that can be pushed from u to v.

PUSH(u, v)

1     Applies when: u is overflowing, cf [u, v] > 0, and h[u] = h[v] + 1.

2     Action: Push df (u, v) = min(e[u], cf(u, v)) units of flow

from u to v.

3  df(u, v)    min(e[u], cf(u, v))

4  f[u, v]    f[u, v] +df(u, v)

5  f[v, u]    - f[u, v]

6  e[u]    e[u] -     (u, v)

7  e[v]    e[v] + df(u, v)

The code for PUSH operates as follows. Vertex u is assumed to have a positive excess e[u], and the residual capacity of (u, v) is positive. Thus, we can ship up to df(u, v) = min (e[u], cf(u, v)) units of flow from u to v without causing e[u] to become negative or the capacity c(u, v) to be exceeded. This value is computed in line 3. We move the flow from u to v by updating f in lines 4-5 and e in lines 6-7. Thus, if â is a preflow before PUSH is called, it remains a preflow afterward.

Observe that nothing in the code for PUSH depends on the heights of u and v, yet we prohibit it from being invoked unless h[u] = h[v] + 1. Thus, excess flow is only pushed downhill by a height differential of 1. By Lemma 27.13, no residual edges exist between two vertices whose heights differ by more than 1, and thus there is nothing to be gained by allowing flow to be pushed downhill by a height differential of more than 1.

We call the operation PUSH(u, v) a push from u to v. If a push operation applies to some edge (u, v) leaving a vertex u, we also say that the push operation applies to u. It is a saturating push if edge (u, v) becomes saturated (cf (u, v) = 0 afterward); otherwise, it is a nonsaturating push. If an edge is saturated, it does not appear in the residual network.

The basic operation LIFT(u) applies if u is overflowing and if cf (u, v) > 0 implies h[u] [image] h[v] for all vertices v. In other words, we can lift an overflowing vertex u if for every vertex v for which there is residual capacity from u to v, flow cannot be pushed from u to v because v is not downhill from u. (Recall that by definition, neither the source s nor the sink t can be overflowing, so neither s nor t can be lifted.)

LIFT(u)

1     Applies when: u is overflowing and for all v    V,

(u, v)    Ef implies h[u]    h[v].

2     Action: Increase the height of u.

3  h[u]   1 + min {h[v]: (u, v)   Ef}

When we call the operation LIFT(u), we say that vertex u is lifted. It is important to note that when u is lifted, Ef must contain at least one edge that leaves u, so that the minimization in the code is over a nonempty set. This fact follows from the assumption that u is overflowing. Since e[u] > 0, we have e[u] = f [V, u] > 0, and hence there must be at least one vertex v such that â[v, u] > 0. But then,

câ (u, v)  =  c(u, v) - f[u, v]

=  c(u, v) + f [v, u]

>  0,

which implies that (u, v) [image] Eâ. The operation LIFT(u) thus gives u the greatest height allowed by the constraints on height functions.

The generic algorithm

The generic preflow-push algorithm uses the following subroutine to create an initial preflow in the flow network.

INITIALIZE-PREFLOW(G, s)

1  for each vertex u    V[G]

2       do h[u]    0

3          e[u]    0

4  for each edge (u, v)    E[G]

5       do â[u, v]    0

6          â[v, u]    0

7  h[s]    |V[G]|

8  for each vertex u    Adj[s]

9       do â[s, u]    c(s, u)

10          â[u, s]    -c(s, u)

11          e[u]    c(s, u)

INITIALIZE-PREFLOW creates an initial preflow â defined by

(27.9)

That is, each edge leaving the source is filled to capacity, and all other edges carry no flow. For each vertex v adjacent to the source, we initially have e[v] = c(s, v). The generic algorithm also begins with an initial height function h, given by

This is a height function because the only edges (u, v) for which h[u] > h[v] + 1 are those for which u = s, and those edges are saturated, which means that they are not in the residual network.

The following algorithm typifies the preflow-push method.

GENERIC-PREFLOW-PUSH(G)

1  INITIALIZE-PREFLOW(G, s)

2  while there exists an applicable push or lift operation

3      do select an applicable push or lift operation and perform it

After initializing the flow, the generic algorithm repeatedly applies, in any order, the basic operations wherever they are applicable. The following lemma tells us that as long as an overflowing vertex exists, at least one of the two operations applies.

Lemma 27.14

Let G = (V, E) be a flow network with source s and sink t, let [image] be a preflow, and let h be any height function for [image]. If u is any overflowing vertex, then either a push or lift operation applies to it.

Proof For any residual edge (u, v), we have h(u) [image] h(v) + 1 because h is a height function. If a push operation does not apply to u, then for all residual edges (u, v), we must have h(u) < h(v) + 1, which implies h(u) [image] h(v) . Thus, a lift operation can be applied to u.

Correctness of the preflow-push method

To show that the generic preflow-push algorithm solves the maximum-flow problem, we shall first prove that if it terminates, the preflow [image] is a maximum flow. We shall later prove that it terminates. We start with some observations about the height function h.

Lemma 27.15

During the execution of GENERIC-PREFLOW-PUSH on a flow network G = (V, E), for each vertex u [image] V, the height h[u] never decreases. Moreover, whenever a lift operation is applied to a vertex u, its height h[u] increases by at least 1.

Proof Because vertex heights change only during lift operations, it suffices to prove the second statement of the lemma. If vertex u is lifted, then for all vertices v such that (u, v) [image] E [image], we have h[u] h[v]; this implies that h[u] < 1 + min {h[v]: (u, v) E }, and so the operation must increase h[u].

Lemma 27.16

Let G = (V, E) be a flow network with source s and sink t. During the execution of GENERIC-PREFLOW-PUSH on G, the attribute h is maintained as a height function.

Proof The proof is by induction on the number of basic operations performed. Initially, h is a height function, as we have already observed.

We claim that if h is a height function, then an operation LIFT(u) leaves h a height function. If we look at a residual edge (u, v) E that leaves u, then the operation LIFT(u) ensures that h[u] h[v] + 1 afterward. Now consider a residual edge (w, u) that enters u. By Lemma 27.15, h[w] h[u] + 1 before the operation LIFT(u) implies h[w] < h[u] + 1 afterward. Thus, the operation LIFT(u) leaves h a height function.

Now, consider an operation PUSH(u, v). This operation may add the edge (v, u) to E , and it may remove (u, v) from E . In the former case, we have h[v] = h[u] - 1, and so h remains a height function. In the latter case, the removal of (u, v) from the residual network removes the corresponding constraint, and h again remains a height function.

The following lemma gives an important property of height functions.

Lemma 27.17

Let G = (V, E) be a flow network with source s and sink t, let f be a preflow in G, and let h be a height function on V. Then, there is no path from the source s to the sink t in the residual network G .

Proof Assume for the sake of contradiction that there is a path p = v0, v1, . . . , vk from s to t in G , where v0 = s and vk = t. Without loss of generality, p is a simple path, and so k < |V|. For i = 0, 1, . . . , k - 1, edge (vi, vi + 1) E Because h is a height function, h(vi) h(vi+1) + 1 for i = 0, 1, . . . , k - 1 . Combining these inequalities over path p yields h(s) h(t) + k. But because h(t) = 0, we have h(s) k < |V| , which contradicts the requirement that h(s) = |V| in a height function.

We are now ready to show that if the generic preflow-push algorithm terminates, the preflow it computes is a maximum flow.

Theorem 27.18

If the algorithm GENERIC-PREFLOW-PUSH terminates when run on a flow network G = (V, E) with source s and sink t, then the preflow f it computes is a maximum flow for G.

Proof If the generic algorithm terminates, then each vertex in V - {s, t} must have an excess of 0, because by Lemmas 27.14 and 27.16 and the invariant that is always a preflow, there are no overflowing vertices. Therefore, is a flow. Because h is a height function, by Lemma 27.17 there is no path from s to t in the residual network G . By the max-flow min-cut theorem, therefore, is a maximum flow.

Analysis of the preflow-push method

To show that the generic preflow-push algorithm indeed terminates, we shall bound the number of operations it performs. Each of the three types of operations--lifts, saturating pushes, and nonsaturating pushes--is bounded separately. With knowledge of these bounds, it is a straightforward problem to construct an algorithm that runs in O(V2 E) time. Before beginning the analysis, however, we prove an important lemma.

Lemma 27.19

Let G = (V, E) be a flow network with source s and sink t, and let be a preflow in G. Then, for any overflowing vertex u, there is a simple path from u to s in the residual network G .

Proof Let U = {v: there exists a simple path from u to v in G }, and suppose for the sake of contradiction that s U. Let

We claim for each pair of vertices v U and that (w, v) 0. Why? If (w, v) > 0, then (v, w) < 0, which implies that c (v, w) = c(v, w) - (v, w) > 0. Hence, there exists an edge (v, w) E , and therefore a simple path of the form in G , contradicting our choice of w.

Thus, we must have since every term in this implicit summation is nonpositive. Thus, from equation (27.8) and Lemma 27.1, we can conclude that

Excesses are nonnegative for all vertices in V - {s}; because we have assumed that U V - {s}, we must therefore have e(v) = 0 for all vertices v U. In particular, e(u) = 0, which contradicts the assumption that u is overflowing.

The next lemma bounds the heights of vertices, and its corollary bounds the number of lift operations that are performed in total.

Lemma 27.20

Let G = (V, E) be a flow network with source s and sink t. At any time during the execution of GENERIC-PREFLOW-PUSH on G, we have h[u] 2 |V| - 1 for all vertices u V.

Proof The heights of the source s and the sink t never change because these vertices are by definition not overflowing. Thus, we always have h[s] = |V| and h[t] = 0.

Because a vertex is lifted only when it is overflowing, we can consider any overflowing vertex u V - {s, t}. Lemma 27.19 tells us that there is a simple path p from u to s in G . Let p = v0, v1, . . . , vk , where v0 = u, vk = s, and k |V| - 1 because p is simple. For i = 0, 1, . . . , k - 1, we have (vi, vi + 1) E , and therefore, by Lemma 27.16, h[vi] h[vi+1] + 1. Expanding these inequalities over path p yields h[u] = h[v0] h[vk] + k h[s] + (|V| - 1) = 2 |V| - 1.

Corollary 27.21

Let G = (V, E) be a flow network with source s and sink t. Then, during the execution of GENERIC-PREFLOW-PUSH on G, the number of lift operations is at most 2 |V| - 1 per vertex and at most (2 |V| - 1)(|V| - 2) < 2 |V |2 overall.

Proof Only vertices in V - {s, t}, which number |V| - 2, may be lifted. Let u V - {s, t}. The operation LIFT(u) increases h[u]. The value of h[u] is initially 0 and by Lemma 27.20 grows to at most 2 |V| - 1. Thus, each vertex u V - {s, t} is lifted at most 2 |V| - 1 times, and the total number of lift operations performed is at most (2 |V| - 1)(|V| - 2) < 2 |V|2.

Lemma 27.20 also helps us to bound the number of saturating pushes.

Lemma 27.22

During the execution of GENERIC-PREFLOW-PUSH on any flow network G = (V, E), the number of saturating pushes is at most 2 |V| |E|.

Proof For any pair of vertices u, v V, consider the saturating pushes from u to v and from v to u. If there are any such pushes, at least one of (u, v) and (v, u) is actually an edge in E. Now, suppose that a saturating push from u to v has occurred. In order for another push from u to v to occur later, the algorithm must first push flow from v to u, which cannot happen until h[v] increases by at least 2. Likewise, h[u] must increase by at least 2 between saturating pushes from v to u.

Consider the sequence A of integers given by h[u] + h[v] for each saturating push that occurs between vertices u and v. We wish to bound the length of this sequence. When the first push in either direction between u and v occurs, we must have h[u] + h[v] 1; thus, the first integer in A is at least 1. When the last such push occurs, we have h[u] + h[v] (2 |V| - 1) + (2 |V| - 2) = 4 |V| - 3 by Lemma 27.20; the last integer in A is thus at most 4 |V| - 3. By the argument from the previous paragraph, at most every other integer can occur in A. Thus, the number of integers in A is at most ((4 |V| - 3) - 1)/2 + 1 = 2 |V| - 1. (We add 1 to make sure that both ends of the sequence are counted.) The total number of saturating pushes between vertices u and v is therefore at most 2 |V| - 1. Multiplying by the number of edges gives a total number of saturating pushes of at most (2 |V| - 1) |E| < 2 |V| |E|.

The following lemma bounds the number of nonsaturating pushes in the generic preflow-push algorithm.

Lemma 27.23

During the execution of GENERIC-PREFLOW-PUSH on any flow network G = (V, E), the number of nonsaturating pushes is at most 4 V 2 ( V + E ).

Proof Define a potential function = v X h[v], where X V is the set of overflowing vertices. Initially, = 0. Observe that lifting a vertex u increases by at most 2 V , since the set over which the sum is taken is the same and u cannot be lifted by more than its maximum possible height, which, by Lemma 27.20, is at most 2 V . Also, a saturating push from a vertex u to a vertex v increases by at most 2 V , since no heights change and only vertex v, whose height is at most 2 V , can possibly become overflowing. Finally, observe that a nonsaturating push from u to v decreases by at least 1, since u is no longer overflowing after the push, v is overflowing afterward even if it wasn't beforehand, and h[v] - h[u] = - 1.

Thus, during the course of the algorithm, the total amount of increase in is constrained by Corollary 27.21 and Lemma 27.22 to be at most (2 V )(2 V 2) + (2 V )(2 V E ) = 4 V 2 ( V + E ). Since 0, the total amount of decrease, and therefore the total number of nonsaturating pushes, is at most 4 V 2 ( V + E ).

We have now set the stage for the following analysis of the GENERIC-PREFLOW-PUSH procedure, and hence of any algorithm based on the preflow-push method.

Theorem 27.24

During the execution of GENERIC-PREFLOW-PUSH on any flow network G = (V, E), the number of basic operations is O(V2 E).

Proof Immediate from Corollary 27.21 and Lemmas 27.22 and 27.23.

Corollary 27.25

There is an implementation of the generic preflow-push algorithm that runs in O(V2 E) time on any flow network G = (V, E).

Proof Exercise 27.4-1 asks you to show how to implement the generic algorithm with an overhead of O(V) per lift operation and O(1) per push. The corollary then follows.

Exercises

27.4-1

Show how to implement the generic preflow-push algorithm using O(V) time per lift operation and O(1) time per push, for a total time of O(V2 E).

27.4-2

Prove that the generic preflow-push algorithm spends a total of only O(V E) time in performing all the O(V2) lift operations.

27.4-3

Suppose that a maximum flow has been found in a flow network G = (V, E) using a preflow-push algorithm. Give a fast algorithm to find a minimum cut in G.

27.4-4

Give an efficient preflow-push algorithm to find a maximum matching in a bipartite graph. Analyze your algorithm.

27.4-5

Suppose that all edge capacities in a flow network G = (V, E) are in the set {1, 2, . . . , k}. Analyze the running time of the generic preflow-push algorithm in terms of V , E , and k. (Hint: How many times can each edge support a nonsaturating push before it becomes saturated?)

27.4-6

Show that line 7 of INITIALIZE-PREFLOW can be changed to

h[s]      V[G]   - 2

without affecting the correctness or asymptotic performance of the generic preflow-push algorithm.

27.4-7

Let (u, v) be the distance (number of edges) from u to v in the residual network G . Show that GENERIC-PREFLOW-PUSH maintains the properties that h[u] < V implies h[u] (u, t) and that h[u] V implies h[u] - V (u, s).

27.4-8

As in the previous exercise, let (u, v) be the distance from u to v in the residual network G . Show how the generic preflow-push algorithm can be modified to maintain the property that h[u] < V implies h[u] = (u, t) and that h[u] V implies h[u] - V = (u, s). The total time that your implementation dedicates to maintaining this property should be O(V E).

27.4-9

Show that the number of nonsaturating pushes executed by GENERIC-PREFLOW-PUSH on a flow network G = (V, E) is at most 4 V 2 E for V 4.

* 27.5 The lift-to-front algorithm

The preflow-push method allows us to apply the basic operations in any order at all. By choosing the order carefully and managing the network data structure efficiently, however, we can solve the maximum-flow problem faster than the O(V2E) bound given by Corollary 27.25. We shall now examine the lift-to-front algorithm, a preflow-push algorithm whose running time is O(V3), which is asymptotically at least as good as O(V2E).

The lift-to-front algorithm maintains a list of the vertices in the network. Beginning at the front, the algorithm scans the list, repeatedly selecting an overflowing vertex u and then "discharging" it, that is, performing push and lift operations until u no longer has a positive excess. Whenever a vertex is lifted, it is moved to the front of the list (hence the name "lift-to-front") and the algorithm begins its scan anew.

The correctness and analysis of the lift-to-front algorithm depend on the notion of "admissible" edges: those edges in the residual network through which flow can be pushed. After proving some properties about the network of admissible edges, we shall investigate the discharge operation and then present and analyze the lift-to-front algorithm itself.

Admissible edges and networks

If G = (V, E) is a flow network with source s and sink t, is a preflow in G, and h is a height function, then we say that (u,v) is an admissible edge if c (u, v) > 0 and h(u) = h(v) + 1. Otherwise, (u, v) is inadmissible. The admissible network is G ,h = (V,E ,h), where E ,h is the set of admissible edges.

The admissible network consists of those edges through which flow can be pushed. The following lemma shows that this network is a directed acyclic graph (dag).

Lemma 27.26

If G = (V,E) is a flow network, is a preflow in G, and h is a height function on G, then the admissible network G ,h = (V,E ,h) is acyclic.

Proof The proof is by contradiction. Suppose that G ,h contains a cycle p = (v0 v1, . . ., vk), where v0 = vk and k > 0. Since each edge in p is admissible, we have h(vi-1) = h(vi) + 1 for i = 1,2, . . ., k. Summing around the cycle gives

Because each vertex in cycle p appears once in each of the summations, we derive the contradiction that 0 = k.

The next two lemmas show how push and lift operations change the admissible network.

Lemma 27.27

Let G = (V,E) be a flow network, let be a preflow in G, and let h be a height function. If a vertex u is overflowing and (u, v) is an admissible edge, then PUSH(u, v) applies. The operation does not create any new admissible edges, but it may cause (u, v) to become inadmissible.

Proof By the definition of an admissible edge, flow can be pushed from u to v. Since u is overflowing, the operation PUSH(u, v) applies. The only new residual edge that can be created by pushing flow from u to v is the edge (v, u). Since h(v) = h(u) - 1, edge (v, u) cannot become admissible. If the operation is a saturating push, then c (u, v) = 0 afterward and (u, v) becomes inadmissible.

Lemma 27.28

Let G = (V,E) be a flow network, let be a preflow in G, and let h be a height function. If a vertex u is overflowing and there are no admissible edges leaving u, then LIFT(u) applies. After the lift operation, there is at least one admissible edge leaving u, but there are no admissible edges entering u.

Proof If u is overflowing, then by Lemma 27.14, either a push or a lift operation applies to it. If there are no admissible edges leaving u, no flow can be pushed from u and LIFT(u) applies. After the lift operation, h[u] = 1 + min {h[v]: (u,v) E }. Thus, if is a vertex that realizes the minimum in this set, the edge (u,v) becomes admissible. Hence, after the lift, there is at least one admissible edge leaving u.

To show that no admissible edges enter u after a lift operation, suppose that there is a vertex v such that (v,u) is admissible. Then, h[ ] = h[u] + 1 after the lift, and therefore h[ ] > h[u] + 1 just before the lift. But by Lemma 27.13, no residual edges exist between vertices whose heights differ by more than 1. Moreover, lifting a vertex does not change the residual network. Thus, ( ,u) is not in the residual network, and hence it cannot be in the admissible network.

Neighbor lists

Edges in the lift-to-front algorithm are organized into "neighbor lists." Given a flow network G = (V,E), the neighbor list N[u] for a vertex u V is a singly linked list of the neighbors of u in G. Thus, vertex appears in the list N[u] if (u, ) E or ( ,u) E. The neighbor list N[u] contains exactly those vertices for which there may be a residual edge (u, ). The first vertex in N[u] is pointed to by head [N[u]]. The vertex following in a neighbor list is pointed to by next-neighbor[ ]; this pointer is NIL if is the last vertex in the neighbor list.

The lift-to-front algorithm cycles through each neighbor list in an arbitrary order that is fixed throughout the execution of the algorithm. For each vertex u, the field current[u] points to the vertex currently under consideration in N[u]. Initially, current[u] is set to head[N[u]].

Discharging an overflowing vertex

An overflowing vertex u is discharged by pushing all of its excess flow through admissible edges to neighboring vertices, lifting u as necessary to cause edges leaving u to become admissible. The pseudocode goes as follows.

DISCHARGE(u)

1  while e[u] > 0

2      do       current[u]

3         if    = NIL

4            then LIFT(u)

5                 current[u]    head[N[u]]

6         elseif c   (u,  ) > 0 and h[u] = h[  ] + 1

7           then PUSH(u,  )

8         else current[u]    next-neighbor[  ]

Figure 27.10 steps through several iterations of the while loop of lines 1-8, which executes as long as vertex u has positive excess. Each iteration performs exactly one of three actions, depending on the current vertex in the neighbor list N[u].

Figure 27.10 Discharging a vertex. It takes 15 iterations of the while loop of DISCHARGE to push all the excess flow from vertex y. Only the neighbors of y and edges entering or leaving y are shown. In each part, the number inside each vertex is its excess at the beginning of the first iteration shown in the part, and each vertex is shown at its height throughout the part. To the right is shown the neighbor list N[y] at the beginning of each iteration, with the iteration number on top. The shaded neighbor is current[y]. (a) Initially, there are 19 units of excess to push from y, and current[y] = s. Iterations 1, 2, and 3 just advance current[y], since there are no admissible edges leaving y. In iteration 4, current[y] = NIL (shown by the shading being below the neighbor list), and so y is lifted and current[y] is reset to the head of the neighbor list. (b) After lifting, vertex y has height 1. In iterations 5 and 6, edges (y, s) and (y, x) are found to be inadmissible, but 8 units of excess flow are pushed from y to z in iteration 7. Because of the push, current[y] is not advanced in this iteration. (c) Because the push in iteration 7 saturated edge (y, z), it is found inadmissible in iteration 8. In iteration 9, current[y] = NIL, and so vertex y is again lifted and current[y] is reset. (d) In iteration 10, (y, s) is inadmissible, but 5 units of excess flow are pushed from y to x in iteration 11. (e) Because current[y] was not advanced in iteration 11, iteration 12 finds (y, x) to be inadmissible. Iteration 13 finds (y, z) inadmissible, and iteration 14 lifts vertex y and resets current[y]. (f) Iteration 15 pushes 6 units of excess flow from y to s. (g) Vertex y now has no excess flow, and DISCHARGE terminates. In this example, DISCHARGE both starts and finishes with the current pointer at the head of the neighbor list, but in general this need not be the case.

1. If v is NIL, then we have run off the end of N[u]. Line 4 lifts vertex u, and then line 5 resets the current neighbor of u to be the first one in N[u]. (Lemma 27.29 below states that the lift operation applies in this situation.)

2. If v is non-NIL and (u, v) is an admissible edge (determined by the test in line 6), then line 7 pushes some (or possibly all) of u's excess to vertex .

3. If v is non-NIL but (u, v) is inadmissible, then line 8 advances current[u] one position further in the neighbor list N[u].

Observe that if DISCHARGE is called on an overflowing vertex u, then the last action performed by DISCHARGE must be a push from u. Why? The procedure terminates only when e[u] becomes zero, and neither the lift operation nor the advancing of the pointer current[u] affects the value of e[u].

We must be sure that when PUSH or LIFT is called by DISCHARGE, the operation applies. The next lemma proves this fact.

Lemma 27.29

If DISCHARGE calls PUSH(u, v) in line 7, then a push operation applies to (u, v). If DISCHARGE calls LIFT(u) in line 4, then a lift operation applies to u.

Proof The tests in lines 1 and 6 ensure that a push operation occurs only if the operation applies, which proves the first statement in the lemma.

To prove the second statement, according to the test in line 1 and Lemma 27.28, we need only show that all edges leaving u are inadmissible. Observe that as DISCHARGE(u) is repeatedly called, the pointer current[u] moves down the list N[u]. Each "pass" begins at the head of N[u] and finishes with current[u] = NIL, at which point u is lifted and a new pass begins. For the current[u] pointer to advance past a vertex N[u] during a pass, the edge (u, v) must be deemed inadmissible by the test in line 6. Thus, by the time the pass completes, every edge leaving u has been determined to be inadmissible at some time during the pass. The key observation is that at the end of the pass, every edge leaving u is still inadmissible. Why? By Lemma 27.27, pushes cannot create any admissible edges, let alone one leaving u. Thus, any admissible edge must be created by a lift operation. But the vertex u is not lifted during the pass, and by Lemma 27.28, any other vertex that is lifted during the pass has no entering admissible edges. Thus, at the end of the pass, all edges leaving u remain inadmissible, and the lemma is proved.

The lift-to-front algorithm

In the lift-to-front algorithm, we maintain a linked list L consisting of all vertices in V - {s, t}. A key property is that the vertices in L are topologically sorted according to the admissible network. (Recall from Lemma 27.26 that the admissible network is a dag.)

The pseudocode for the lift-to-front algorithm assumes that the neighbor lists N[u] have already been created for each vertex u. It also assumes that next[u] points to the vertex that follows u in list L and that, as usual, next[u] = NIL if u is the last vertex in the list.

LIFT-TO-FRONT(G, s, t)

1  INITIALIZE-PREFLOW(G, s)

2  L    V[G] - {s, t}, in any order

3  for each vertex u    V[G] - {s, t}

4       do current[u]    head[N[u]]

5  u    head[L]

6  while u    NIL

7      do old-height    h[u]

8         DISCHARGE(u)

9         if h[u] > old-height

10            then move u to the front of list L

11         u    next[u]

The lift-to-front algorithm works as follows. Line 1 initializes the preflow and heights to the same values as in the generic preflow-push algorithm. Line 2 initializes the list L to contain all potentially overflowing vertices, in any order. Lines 3-4 initialize the current pointer of each vertex u to the first vertex in u's neighbor list.

As shown in Figure 27.11, the while loop of lines 6-11 runs through the list L, discharging vertices. Line 5 makes it start with the first vertex in the list. Each time through the loop, a vertex u is discharged in line 8. If u was lifted by the DISCHARGE procedure, line 10 moves it to the front of list L. This determination is made by saving u's height in the variable old-height before the discharge operation (line 7) and comparing this saved height to u's height afterward (line 9). Line 11 makes the next iteration of the while loop use the vertex following u in list L. If u was moved to the front of the list, the vertex used in the next iteration is the one following u in its new position in the list.

To show that LIFT-TO-FRONT computes a maximum flow, we shall show that it is an implementation of the generic preflow-push algorithm. First, observe that it only performs push and lift operation when they apply, since Lemma 27.29 guarantees that DISCHARGE only performs them when they apply. It remains to show that when LIFT-TO-FRONT terminates, no basic operations apply. Observe that if u reaches the end of L, every vertex in L must have been discharged without causing a lift. Lemma 27.30, which we shall prove in a moment, states that the list L is maintained as a topological sort of the admissible network. Thus, a push operation causes excess flow to move to vertices further down the list (or to s or t). If the pointer u reaches the end of the list, therefore, the excess of every vertex is 0, and no basic operations apply.

Figure 27.11 The action of LIFT-TO-FRONT. (a) A flow network just before the first iteration of the while loop. Initially, 26 units of flow leave source s. On the right is shown the initial list L = x, y, z , where initially u = x. Under each vertex in list L is its neighbor list, with the current neighbor shaded. Vertex x is discharged. It is lifted to height 1, 5 units of excess flow are pushed to y, and the 7 remaining units of excess are pushed to the sink t. Because x is lifted, it is moved to the head of L, which in this case does not change the structure of L. (b) After x, the next vertex in L that is discharged is y. Figure 27.10 shows the detailed action of discharging y in this situation. Because y is lifted, it is moved to the head of L. (c) Vertex x now follows y in L, and so it is again discharged, pushing all 5 units of excess flow to t. Because vertex x is not lifted in this discharge operation, it remains in place in list L. (d) Since vertex z follows vertex x in L, it is discharged. It is lifted to height 1 and all 8 units of excess flow are pushed to t. Because z is lifted, it is moved to the front of L. (e) Vertex y now follows vertex z in L and is therefore discharged. But because y has no excess, DISCHARGE immediately returns, and y remains in place in L. Vertex x is then discharged. Because it, too, has no excess, DISCHARGE again returns, and x remains in place in L. LIFT-TO-FRONT has reached the end of list L and terminates. There are no overflowing vertices, and the preflow is a maximum flow.

Lemma 27.30

If we run LIFT-TO-FRONT on a flow network G = (V, E) with source s and sink t, then each iteration of the while loop in lines 6-11 maintains the invariant that list L is a topological sort of the vertices in the admissible network G ,h = (V, E ,h).

Proof Immediately after INITIALIZE-PREFLOW has been run, h[s] = V and h[v] = 0 for all v V - {s}. Since |V | 2 (because it contains at least s and t), no edge can be admissible. Thus, E ,h = , and any ordering of V - {s, t} is a topological sort of G ,h.

We now show that the invariant is maintained by each iteration of the while loop. The admissible network is changed only by push and lift operations. By Lemma 27.27, push operations only make edges inadmissible. Thus, admissible edges can be created only by lift operations. After a vertex is lifted, however, Lemma 27.28 states that there are no admissible edges entering u but there may be admissible edges leaving u. Thus, by moving u to the front of L, the algorithm ensures that any admissible edges leaving u satisfy the topological sort ordering.

Analysis

We shall now show that LIFT-TO-FRONT runs in O(V3) time on any flow network G = (V, E). Since the algorithm is an implementation of the generic preflow-push algorithm, we shall take advantage of Corollary 27.21, which provides an O(V) bound on the number of lift operations executed per vertex and an O(V2) bound on the total number of lifts overall. In addition, Exercise 27.4-2 provides an O(V E) bound on the total time spent performing lift operations, and Lemma 27.22 provides an O(V E) bound on the total number of saturating push operations.

Theorem 27.31

The running time of LIFT-TO-FRONT on any flow network G = (V, E) is O(V3).

Proof Let us consider a "phase" of the lift-to-front algorithm to be the time between two consecutive lift operations. There are O(V2) phases, since there are O(V2) lift operations. Each phase consists of at most |V | calls to DISCHARGE, which can be seen as follows. If DISCHARGE does not perform a lift operation, the next call to DISCHARGE is further down the list L, and the length of L is less than |V |. If DISCHARGE does perform a lift, the next call to DISCHARGE belongs to a different phase. Since each phase contains at most |V | calls to DISCHARGE and there are O(V2) phases, the number of times DISCHARGE is called in line 8 of LIFT-TO-FRONT is O(V3). Thus, the total work performed by the while loop in LIFT-TO-FRONT, excluding the work performed within DISCHARGE, is at most O(V3).

We must now bound the work performed within DISCHARGE during the execution of the algorithm. Each iteration of the while loop within DISCHARGE performs one of three actions. We shall analyze the total amount of work involved in performing each of these actions.

We start with lift operations (lines 4-5). Exercise 27.4-2 provides an O(V E) time bound on all the O(V2) lifts that are performed.

Now, suppose that the action updates the current[u] pointer in line 8. This action occurs O(degree(u)) times each time a vertex u is lifted, and O(V degree(u)) times overall for the vertex. For all vertices, therefore, the total amount of work done in advancing pointers in neighbor lists is O(V E) by the handshaking lemma (Exercise 5.4-1).

The third type of action performed by DISCHARGE is a push operation (line 7). We already know that the total number of saturating push operations is O(V E). Observe that if a nonsaturating push is executed, DISCHARGE immediately returns, since the push reduces the excess to 0. Thus, there can be at most one nonsaturating push per call to DISCHARGE. As we have observed, DISCHARGE is called O(V3) times, and thus the total time spent performing nonsaturating pushes is O(V3).

The running time of LIFT-TO-FRONT is therefore O(V3 + V E), which is O(V3).

Exercises

27.5-1

Illustrate the execution of LIFT-TO-FRONT in the manner of Figure 27.11 for the flow network in Figure 27.1 (a). Assume that the initial ordering of vertices in L is v1, v2, v3, v4 and that the neighbor lists are

N[v1]  =    s, v2, v3   ,

N[v2]  =    s, v1, v3, v4   ,

N[v3]  =    v1, v2, v4, t   ,

N[v4]  =    v2, v3, t   .

27.5-2

We would like to implement a preflow-push algorithm in which we maintain a first-in, first-out queue of overflowing vertices. The algorithm repeatedly discharges the vertex at the head of the queue, and any vertices that were not overflowing before the discharge but are overflowing afterward are placed at the end of the queue. After the vertex at the head of the queue is discharged, it is removed. When the queue is empty, the algorithm terminates. Show that this algorithm can be implemented to compute a maximum flow in O(V3) time.

27.5-3

Show that the generic algorithm still works if LIFT updates h[u] by simply computing h[u] h[u] + 1. How does this change affect the analysis of LIFT-To-FRONT?

27.5-4

Show that if we always discharge a highest overflowing vertex, the preflow-push method can be made to run in O(V3) time.

Problems

27-1 Escape problem

An n x n grid is an undirected graph consisting of n rows and n columns of vertices, as shown in Figure 27.12. We denote the vertex in the ith row and the jth column by (i, j). All vertices in a grid have exactly four neighbors, except for the boundary vertices, which are the points (i, j) for which i = 1, i = n, j = 1, or j = n.

Figure 27.12 Grids for the escape problem. Starting points are black, and other grid vertices are white. (a) A grid with an escape, shown by shaded paths. (b) A grid with no escape.

Given m n2 starting points (x1, y1), (x2, y2), . . . , (xm, ym) in the grid, the escape problem is to determine whether or not there are m vertex-disjoint paths from the starting points to any m different points on the boundary. For example, the grid in Figure 27.12(a) has an escape, but the grid in Figure 27.12(b) does not.

a. Consider a flow network in which vertices, as well as edges, have capacities. That is, the positive net flow entering any given vertex is subject to a capacity constraint. Show that determining the maximum flow in a network with edge and vertex capacities can be reduced to an ordinary maximum-flow problem on a flow network of comparable size.

b. Describe an efficient algorithm to solve the escape problem, and analyze its running time.

27-2 Minimum path cover

A path cover of a directed graph G = (V, E) is a set P of vertex-disjoint paths such that every vertex in V is included in exactly one path in P. Paths may start and end anywhere, and they may be of any length, including 0. A minimum path cover of G is a path cover containing the fewest possible paths.

a. Give an efficient algorithm to find a minimum path cover of a directed acyclic graph G = (V, E). (Hint: Assuming that V = {1, 2, . . . , n}, construct the graph G' = (V', E'), where

V'  =  {x0, x1, . . . , xn}    {y0, y1, . . . , yn} ,

E'  =  {(x0, xi) : i    V}    {(yi, y0) : i    V}    {(xi, yj) : (i, j)    E} ,

and run a maximum-flow algorithm.)

b. Does your algorithm work for directed graphs that contain cycles? Explain.

27-3 Space shuttle experiments

Professor Spock is consulting for NASA, which is planning a series of space shuttle flights and must decide which commercial experiments to perform and which instruments to have on board each flight. For each flight, NASA considers a set E = {E1, E2, . . . , Em} of experiments, and the commercial sponsor of experiment Ej has agreed to pay NASA pj dollars for the results of the experiment. The experiments use a set I = {I1, I2, . . . , In} of instruments; each experiment Ej requires all the instruments in a subset Rj I. The cost of carrying instrument Ik is ck dollars. The professor's job is to find an efficient algorithm to determine which experiments to perform and which instruments to carry for a given flight in order to maximize the net revenue, which is the total income from experiments performed minus the total cost of all instruments carried.

Consider the following network G. The network contains a source vertex s, vertices I1, I2, . . . , In, vertices E1, E2, . . . , Em, and a sink vertex t. For k = 1, 2 . . . , n, there is an edge (s, Ik) of capacity ck, and for j = 1, 2, . . . , m, there is an edge (Ej, t) of capacity pj. For k = 1, 2, . . . , n and j = 1, 2, . . . , m, if Ik Rj, then there is an edge (Ik, Ej) of infinite capacity.

a. Show that if Ej T for a finite-capacity cut (S, T) of G, then Ik T for each Ik Rj.

b. Show how to determine the maximum net revenue from the capacity of the minimum cut of G and the given pj values.

c. Give an efficient algorithm to determine which experiments to perform and which instruments to carry. Analyze the running time of your algorithm in terms of m, n, and .

27-4 Updating maximum flow

Let G = (V, E) be a flow network with source s, sink t, and integer capacities. Suppose that we are given a maximum flow in G.

a. Suppose that the capacity of a single edge (u, v) E is increased by 1. Give an O(V + E)-time algorithm to update the maximum flow.

b. Suppose that the capacity of a single edge (u, v) E is decreased by 1. Give an O(V + E)-time algorithm to update the maximum flow.

27-5 Maximum flow by scaling

Let G = (V, E) be a flow network with source s, sink t, and an integer capacity c(u, v) on each edge (u, v) E. Let C = max(u, v) E c(u, v).

a. Argue that a minimum cut of G has capacity at most C|E|.

b. For a given number K, show that an augmenting path of capacity at least K can be found in O(E) time, if such a path exists.

The following modification of FORD-FULKERSON-METHOD can be used to compute a maximum flow in G.

MAX-FLOW-BY-SCALING(G, s, t)

1  C    max(u,v)   E c(u, v)

2  initialize flow f to 0

3  K    2  1g C

4  while K    1

5      do while there exists an augmenting path p of capacity at least K

6            do augment flow f along p

7         K    K/2

8  return f

c. Argue that MAX-FLOW-BY-SCALING returns a maximum flow.

d. Show that the residual capacity of a minimum cut of G is at most 2K |E| each time line 4 is executed.

e. Argue that the inner while loop of lines 5-6 is executed O(E) times for each value of K.

f. Conclude that MAX-FLOW-BY-SCALING can be implemented to run in O(E2 1g C) time.

27-6 Maximum flow with upper and lower capacity bounds

Suppose that each edge (u, v) in a flow network G = (V, E) has not only an upper bound c(u, v) on the net flow from u to v, but also a lower bound b(u, v). That is, any flow f on the network must satisfy b(u, v) f(u, v) c(u, v). It may be the case for such a network that no feasible flow exists.

a. Prove that if f is a flow on G, then |f| c(S, T) - b(T, S) for any cut (S, T) of G.

b. Prove that the value of a maximum flow in the network, if it exists, is the minimum value of c(S, T) - b(T, S) over all cuts (S, T) of the network.

Let G = (V, E) be a flow network with upper and lower bound functions c and b, and let s and t be the source and sink of G. Construct the ordinary flow network G' = (V', E') with upper bound function c', source s', and sink t' as follows:

V'  =  V    {s', t'} ,

E'  =  E    {(s', v) : v    V}    {(u, t') : u    V}    {(s, t),(t, s)} .

We assign capacities to edges as follows. For each edge (u, v) E, we set c'(u, v) = c(u, v) - b(u, v). For each vertex u V, we set c'(s', u) = b(V, u) and c'(u, t') = b(u, V). We also set c'(s, t) = c'(t, s) = .

c. Prove that there exists a feasible flow in G if and only if there exists a maximum flow in G' such that all edges into the sink t' are saturated.

d. Give an algorithm that finds a maximum flow in a network with upper and lower bounds or determines that no feasible flow exists. Analyze the running time of your algorithm.

Chapter notes

Even [65], Lawler [132], Papadimitriou and Steiglitz [154], and Tarjan [188] are good references for network flow and related algorithms. Goldberg, Tardos, and Tarjan [83] provide a nice survey of algorithms for network-flow problems.

The Ford-Fulkerson method is due to Ford and Fulkerson [71], who originated many of the problems in the area of network flow, including the maximum-flow and bipartite-matching problems. Many early implementations of the Ford-Fulkerson method found augmenting paths using breadth-first search; Edmonds and Karp [63] proved that this strategy yields a polynomial-time algorithm. Karzanov [119] developed the idea of preflows. The preflow-push method is due to Goldberg [82]. The fastest preflow-push algorithm to date is due to Goldberg and Tarjan [85], who achieve a running time of O(V E lg(V2/E)). The best algorithm to date for maximum bipartite matching, discovered by Hopcroft and Karp [101], runs in time.

Go to Part VII Back to Table of Contents


Intro to Algorithms: CHAPTER 28: SORTING NETWORKS

 Return to Table of Contents

CHAPTER 28: SORTING NETWORKS

In Part II, we examined sorting algorithms for serial computers (random-access machines, or RAM's) that allow only one operation to be executed at a time. In this chapter, we investigate sorting algorithms based on a comparison network model of computation in which many comparison operations can be performed simultaneously.

Comparison networks differ from RAM's in two important respects. First, they can only perform comparisons. Thus, an algorithm such as counting sort (see Section 9.2) cannot be implemented on a comparison network. Second, unlike the RAM model, in which operations occur serially--that is, one after another--operations in a comparison network may occur at the same time, or "in parallel." As we shall see, this characteristic allows the construction of comparison networks that sort n values in sublinear time.

We begin in Section 28.1 by defining comparison networks and sorting networks. We also give a natural definition for the "running time" of a comparison network in terms of the depth of the network. Section 28.2 proves the "zero-one principle," which greatly eases the task of analyzing the correctness of sorting networks.

The efficient sorting network that we shall design is essentially a parallel version of the merge-sort algorithm from Section 1.3.1. Our construction will have three steps. Section 28.3 presents the design of a "bitonic" sorter that will be our basic building block. We modify the bitonic sorter slightly in Section 28.4 to produce a merging network that can merge two sorted sequences into one sorted sequence. Finally, in Section 28.5, we assemble these merging networks into a sorting network that can sort n values in O(lg2 n) time.

28.1 Comparison networks

Sorting networks are comparison networks that always sort their inputs, so it makes sense to begin our discussion with comparison networks and their characteristics. A comparison network is comprised solely of wires and comparators. A comparator, shown in Figure 28.1 (a), is a device with two inputs, x and y, and two outputs, x' and y', that performs the following function:

Figure 28.1 (a) A comparator with inputs x and y and outputs x' and y'. (b) The same comparator, drawn as a single vertical line. Inputs x = 7, y = 3 and outputs x' = 3, y' = 7 are shown.

x'  =  min(x, y) ,

y'  =  max(x, y) .

Because the pictorial representation of a comparator in Figure 28.1 (a) is too bulky for our purposes, we shall adopt the convention of drawing comparators as single vertical lines, as shown in Figure 28.1(b). Inputs appear on the left and outputs on the right, with the smaller input value appearing on the top output and the larger input value appearing on the bottom output. We can thus think of a comparator as sorting its two inputs.

We shall assume that each comparator operates in O(1) time. In other words, we assume that the time between the appearance of the input values x and y and the production of the output values x' and y' is a constant.

A wire transmits a value from place to place. Wires can connect the output of one comparator to the input of another, but otherwise they are either network input wires or network output wires. Throughout this chapter, we shall assume that a comparison network contains n input wires a1, a2, . . . , an, through which the values to be sorted enter the network, and n output wires b1, b2, . . . , bn, which produce the results computed by the network. Also, we shall speak of the input sequence [image] a1, a2, . . . , an [image] and the output sequence [image] b1,b2, . . . ,bn [image], referring to the values on the input and output wires. That is, we use the same name for both a wire and the value it carries. Our intention will always be clear from the context.

Figure 28.2 shows a comparison network, which is a set of comparators interconnected by wires. We draw a comparison network on n inputs as a collection of n horizontal lines with comparators stretched vertically. Note that a line does not represent a single wire, but rather a sequence of distinct wires connecting various comparators. The top line in Figure 28.2, for example, represents three wires: input wire a1, which connects to an input of comparator A; a wire connecting the top output of comparator A to an input of comparator C; and output wire b1, which comes from the top output of comparator C. Each comparator input is connected to a wire that is either one of the network's n input wires a1, a2, . . . , an or is connected to the output of another comparator. Similarly, each comparator output is connected to a wire that is either one of the network's n output wires b1, b2, . . . , bn or is connected to the input of another comparator. The main requirement for interconnecting comparators is that the graph of interconnections must be acyclic: if we trace a path from the output of a given comparator to the input of another to output to input, etc., the path we trace must never cycle back on itself and go through the same comparator twice. Thus, as in Figure 28.2, we can draw a comparison network with network inputs on the left and network outputs on the right; data move through the network from left to right.

Figure 28.2 (a) A 4-input, 4-output comparison network, which is in fact a sorting network. At time 0, the input values shown appear on the four input wires. (b) At time 1, the values shown appear on the outputs of comparators A and B, which are at depth l. (c) At time 2, the values shown appear on the outputs of comparators C and D, at depth 2. Output wires b1 and b4 now have their final values, but output wires b2 and b3 do not. (d) At time 3, the values shown appear on the outputs of comparator E, at depth 3. Output wires b2 and b3 now have their final values.

Each comparator produces its output values only when both of its input values are available to it. In Figure 28.2(a), for example, suppose that the sequence [image] 9, 5, 2, 6 [image] appears on the input wires at time 0. At time 0, then, only comparators A and B have all their input values available. Assuming that each comparator requires one time unit to compute its output values, comparators A and B produce their outputs at time 1; the resulting values are shown in Figure 28.2(b). Note that comparators A and B produce their values at the same time, or "in parallel." Now, at time 1, comparators C and D, but not E, have all their input values available. One time unit later, at time 2, they produce their outputs, as shown in Figure 28.2(c). Comparators C and D operate in parallel as well. The top output of comparator C and the bottom output of comparator D connect to output wires b1 and b4, respectively, of the comparison network, and these network output wires therefore carry their final values at time 2. Meanwhile, at time 2, comparator E has its inputs available, and Figure 28.2(d) shows that it produces its output values at time 3. These values are carried on network output wires b2 and b3, and the output sequence [image] 2, 5, 6, 9 [image] is now complete.

Under the assumption that each comparator takes unit time, we can define the "running time" of a comparison network, that is, the time it takes for all the output wires to receive their values once the input wires receive theirs. Informally, this time is the largest number of comparators that any input element can pass through as it travels from an input wire to an output wire. More formally, we define the depth of a wire as follows. An input wire of a comparison network has depth 0. Now, if a comparator has two input wires with depths dx and dy, then its output wires have depth max(dx, dy) + 1. Because there are no cycles of comparators in a comparison network, the depth of a wire is well defined, and we define the depth of a comparator to be the depth of its output wires. Figure 28.2 shows comparator depths. The depth of a comparison network is the maximum depth of an output wire or, equivalently, the maximum depth of a comparator. The comparison network of Figure 28.2, for example, has depth 3 because comparator E has depth 3. If each comparator takes one time unit to produce its output value, and if network inputs appear at time 0, a comparator at depth d produces its outputs at time d; the depth of the network therefore equals the time for the network to produce values at all of its output wires.

A sorting network is a comparison network for which the output sequence is monotonically increasing (that is, b1 [image] b2 [image] . . . bn) for every input sequence. Of course, not every comparison network is a sorting network, but the network of Figure 28.2 is. To see why, observe that after time 1, the minimum of the four input values has been produced by either the top output of comparator A or the top output of comparator B. After time 2, therefore, it must be on the top output of comparator C. A symmetrical argument shows that after time 2, the maximum of the four input values has been produced by the bottom output of comparator D. All that remains is for comparator E to ensure that the middle two values occupy their correct output positions, which happens at time 3.

A comparison network is like a procedure in that it specifies how comparisons are to occur, but it is unlike a procedure in that its physical size depends on the number of inputs and outputs. Therefore, we shall actually be describing "families" of comparison networks. For example, the goal of this chapter is to develop a family SORTER of efficient sorting networks. We specify a given network within a family by the family name and the number of inputs (which equals the number of outputs). For example, the n-input, n-output sorting network in the family SORTER is named SORTER[n].

Figure 28.3 A sorting network based on insertion sort for use in Exercise 28.1-6.

Exercises

28.1-1

Show the values that appear on all the wires of the network of Figure 28.2 when it is given the input sequence [image] 9, 6, 5, 2 [image].

28.1-2

Let n be an exact power of 2. Show how to construct an n-input, n-output comparison network of depth lg n in which the top output wire always carries the minimum input value and the bottom output wire always carries the maximum input value.

28.1-3

Professor Nielsen claims that if we add a comparator anywhere in a sorting network, the resulting network also sorts. Show that the professor is mistaken by adding a comparator to the network of Figure 28.2 in such a way that the resulting network does not sort every input permutation.

28.1-4

Prove that any sorting network on n inputs has depth at least lg n.

28.1-5

Prove that the number of comparators in any sorting network is at least [image] (n lg n).

28.1-6

Consider the comparison network shown in Figure 28.3. Prove that it is in fact a sorting network, and describe how its structure is related to that of insertion sort (Section 1.1).

28.1-7

We can represent an n-input comparison network with c comparators as a list of c pairs of integers in the range from 1 to n. If two pairs contain an integer in common, the order of the corresponding comparators in the network is determined by the order of the pairs in the list. Given this representation, describe an O(n+c)-time (serial) algorithm for determining the depth of a comparison network.

28.1-8

Suppose that in addition to the standard kind of comparator, we introduce an "upside-down" comparator that produces its minimum output on the bottom wire and its maximum output on the top wire. Show how to convert any sorting network that uses a total of c standard and upside-down comparators to one that uses c standard ones. Prove that your conversion method is correct.

28.2 The zero-one principle

The zero-one principle says that if a sorting network works correctly when each input is drawn from the set {0,1}, then it works correctly on arbitrary input numbers. (The numbers can be integers, reals, or, in general, any set of values from any linearly ordered set.) As we construct sorting networks and other comparison networks, the zero-one principle will allow us to focus on their operation for input sequences consisting solely of 0's and 1's. Once we have constructed a sorting network and proved that it can sort all zero-one sequences, we shall appeal to the zero-one principle to show that it properly sorts sequences of arbitrary values.

The proof of the zero-one principle relies on the notion of a monotonically increasing function (Section 2.2).

Lemma 28.1

If a comparison network transforms the input sequence a = [image] a1, a2, . . . , an [image] into the output sequence b = [image] b1, b2, . . . , bn [image], then for any monotonically increasing function f, the network transforms the input sequence f(a) = [image] f(a1), f(a2), . . . , f(an) [image] into the output sequence f(b) = [image] f(b1), f(b2), . . . , f(bn) [image].

Proof We shall first prove the claim that if f is a monotonically increasing function, then a single comparator with inputs f(x) and f(y) produces outputs f(min(x,y)) and f(max(x,y)). We shall then use induction to prove the lemma.

To prove the claim, consider a comparator whose input values are x and y. The upper output of the comparator is min(x,y) and the lower output is max(x,y). Suppose we now apply f(x) and f(y) to the inputs of the comparator, as is shown in Figure 28.4. The operation of the comparator yields the value min(f(x), f(y)) on the upper output and the value max(f(x), f(y)) on the lower output. Since f is monotonically increasing, x [image] y implies f(x) [image] f(y). Consequently, we have the identities

min(f(x), f(y))  =  f(min(x,y)) ,

max(f(x), f(y))  =  f(max(x,y)) .

Thus, the comparator produces the values f (min(x,y)) and f (max(x,y)) when f (x) and f (y) are its inputs, which completes the proof of the claim.

Figure 28.4 The operation of the comparator in the proof of Lemma 28.1. The function f is monotonically increasing.

We can use induction on the depth of each wire in a general comparison network to prove a stronger result than the statement of the lemma: if a wire assumes the value ai when the input sequence a is applied to the network, then it assumes the value f(ai) when the input sequence f(a) is applied. Because the output wires are included in this statement, proving it will prove the lemma.

For the basis, consider a wire at depth 0, that is, an input wire ai. The result follows trivially: when f(a) is applied to the network, the input wire carries f(ai). For the inductive step, consider a wire at depth d, where d [image] 1. The wire is the output of a comparator at depth d, and the input wires to this comparator are at a depth strictly less than d. By the inductive hypothesis, therefore, if the input wires to the comparator carry values ai and aj when the input sequence a is applied, then they carry f(ai) and f(aj) when the input sequence f(a) is applied. By our earlier claim, the output wires of this comparator then carry f(min(ai, aj)) and f(max(ai, aj)). Since they carry min(ai, aj) and max(ai, aj) when the input sequence is a, the lemma is proved.

As an example of the application of Lemma 28.1, Figure 28.5 shows the sorting network from Figure 28.2 with the monotonically increasing function f(x) = [image] x/2 [image] applied to the inputs. The value on every wire is f applied to the value on the same wire in Figure 28.2.

When a comparison network is a sorting network, Lemma 28.1 allows us to prove the following remarkable result.

Theorem 28.2

If a comparison network with n inputs sorts all 2n possible sequences of 0's and 1's correctly, then it sorts all sequences of arbitrary numbers correctly.

Figure 28.5 (a) The sorting network from Figure 28.2 with input sequence 9, 5, 2, 6 . (b) The same sorting network with the monotonically increasing function f(x) = f( x/2 ) applied to the inputs. Each wire in this network has the value of f applied to the value on the corresponding wire in (a).

Proof Suppose for the purpose of contradiction that the network sorts all zero-one sequences, but there exists a sequence of arbitrary numbers that the network does not correctly sort. That is, there exists an input sequence [image] a1, a2, . . . , an [image] containing elements ai and aj such that ai < aj, but the network places aj before ai in the output sequence. We define a monotonically increasing function f as

Since the network places aj before ai in the output sequence when [image] a1, a2, . . . , an [image] is input, it follows from Lemma 28.1 that it places f(aj) before f(ai) in the output sequence when [image] f(a1), f(a2), . . . , f(an) [image] is input. But since f(aj) = 1 and f(ai) = 0, we obtain the contradiction that the network fails to sort the zero-one sequence [image] f(a1), f(a2), . . . , f(an) [image] correctly.

Exercises

28.2-1

Prove that applying a monotonically increasing function to a sorted sequence produces a sorted sequence.

28.2-2

Prove that a comparison network with n inputs correctly sorts the input sequence [image] n, n - 1, . . . , 1 [image] if and only if it correctly sorts the n - 1 zero-one sequences [image] 1, 0, 0, . . . , 0, 0 [image], [image] 1, 1, 0, . . . , 0, 0 [image], . . . , [image] 1, 1, 1, . . . , 1, 0 [image].

28.2-3

Use the zero-one principle to prove that the comparison network shown in Figure 28.6 is a sorting network.

28.2-4

State and prove an analog of the zero-one principle for a decision-tree model. (Hint: Be sure to handle equality properly.)

Figure 28.6 A sorting network for sorting 4 numbers.

28.2-5

Prove that an n-input sorting network must contain at least one comparator between the ith and (i + 1 )st lines for all i = 1, 2, . . . , n - 1.

28.3 A bitonic sorting network

The first step in our construction of an efficient sorting network is to construct a comparison network that can sort any bitonic sequence: a sequence that either monotonically increases and then monotonically decreases, or else monotonically decreases and then monotonically increases. For example, the sequences [image] 1, 4, 6, 8, 3, 2 [image] and [image] 9, 8, 3, 2, 4, 6 [image] are both bitonic. The zero-one sequences that are bitonic have a simple structure. They have the form 0i1j0k or the form 1i0j1k, for some i, j, k [image] 0. Note that a sequence that is either monotonically increasing or monotonically decreasing is also bitonic.

The bitonic sorter that we shall construct is a comparison network that sorts bitonic sequences of 0's and 1's. Exercise 28.3-6 asks you to show that the bitonic sorter can sort bitonic sequences of arbitrary numbers.

The half-cleaner

A bitonic sorter is comprised of several stages, each of which is called a half-cleaner. Each half-cleaner is a comparison network of depth 1 in which input line i is compared with line i + n/2 for i = 1, 2, . . . , n/2. (We assume that n is even.) Figure 28.7 shows HALF-CLEANER[8], the half-cleaner with 8 inputs and 8 outputs.

When a bitonic sequence of 0's and 1's is applied as input to a half-cleaner, the half-cleaner produces an output sequence in which smaller values are in the top half, larger values are in the bottom half, and both halves are bitonic. In fact, at least one of the halves is clean--consisting of either all 0's or all 1's--and it is from this property that we derive the name "half-cleaner." (Note that all clean sequences are bitonic.) The next lemma proves these properties of half-cleaners.

Figure 28.7 The comparison network HALF-CLEANER[8]. Two different sample zero-one input and output values are shown. The input is assumed to be bitonic. A half-cleaner ensures that every output element of the top half is at least as small as every output element of the bottom half. Moreover, both halves are bitonic, and at least one half is clean.

Lemma 28.3

If the input to a half-cleaner is a bitonic sequence of 0's and 1's, then the output satisfies the following properties: both the top half and the bottom half are bitonic, every element in the top half is at least as small as every element of the bottom half, and at least one half is clean.

Proof The comparison network HALF-CLEANER[n] compares inputs i and i + n/2 for i = 1, 2, . . . , n/2. Without loss of generality, suppose that the input is of the form 00 . . . 011 . . . 100 . . . 0. (The situation in which the input is of the form 11 . . . 100 . . . 011 . . . 1 is symmetric.) There are three possible cases depending upon the block of consecutive 0's or 1s in which the midpoint n/2 falls, and one of these cases (the one in which the midpoint occurs in the block of 1's) is further split into two cases. The four cases are shown in Figure 28.8. In each case shown, the lemma holds.

The bitonic sorter

By recursively combining half-cleaners, as shown in Figure 28.9, we can build a bitonic sorter, which is a network that sorts bitonic sequences. The first stage of BITONIC-SORTER[n] consists of HALF-CLEANER[n], which, by Lemma 28.3, produces two bitonic sequences of half the size such that every element in the top half is at least as small as every element in the bottom half. Thus, we can complete the sort by using two copies of BITONIC-SORTER[n/2] to sort the two halves recursively. In Figure 28.9(a), the recursion has been shown explicitly, and in Figure 28.9(b), the recursion has been unrolled to show the progressively smaller half-cleaners that make up the remainder of the bitonic sorter. The depth D(n) of BITONIC-SORTER[n] is given by the recurrence whose solution is D(n) = lg n.

Figure 28.8 The possible comparisons in HALF-CLEANER[n]. The input sequence is assumed to be a bitonic sequence of 0's and 1's, and without loss of generality, we assume that it is of the form 00 . . . 011 . . . 100 . . . 0. Subsequences of 0's are white, and subsequences of 1's are gray. We can think of the n inputs as being divided into two halves such that for i = 1, 2, . . . , n/2, inputs i and i + n/2 are compared. (a)-(b) Cases in which the division occurs in the middle subsequence of 1's. (c)-(d) Cases in which the division occurs in a subsequence of 0's. For all cases, every element in the top half is at least as small as every element in the bottom half, both halves are bitonic, and at least one half is clean.

Figure 28.9 The comparison network BITONIC-SORTER[n], shown here for n = 8. (a) The recursive construction: HALF-CLEANER[n] followed by two copies of BITONIC-SORTER[n/2] that operate in parallel. (b) The network after unrolling the recursion. Each half-cleaner is shaded. Sample zero-one values are shown on the wires.

Thus, a zero-one bitonic sequence can be sorted by BITONIC-SORTER, which has a depth of lg n. It follows by the analog of the zero-one principle given as Exercise 28.3-6 that any bitonic sequence of arbitrary numbers can be sorted by this network.

Exercises

28.3-1

How many bitonic sequences of 0's and 1's are there?

28.3-2

Show that BITONIC-SORTER[n], where n is an exact power of 2, contains [image] (n lg n) comparators.

28.3-3

Describe how an O(lg n)-depth bitonic sorter can be constructed when the number n of inputs is not an exact power of 2.

28.3-4

If the input to a half-cleaner is a bitonic sequence of arbitrary numbers, prove that the output satisfies the following properties: both the top half and the bottom half are bitonic, and every element in the top half is at least as small as every element in the bottom half.

28.3-5

Consider two sequences of 0's and 1's. Prove that if every element in one sequence is at least as small as every element in the other sequence, then one of the two sequences is clean.

28.3-6

Prove the following analog of the zero-one principle for bitonic sorting networks: a comparison network that can sort any bitonic sequence of 0's and 1's can sort any bitonic sequence of arbitrary numbers.

28.4 A merging network

Our sorting network will be constructed from merging networks, which are networks that can merge two sorted input sequences into one sorted output sequence. We modify BITONIC-SORTER[n] to create the merging network MERGER[n]. As with the bitonic sorter, we shall prove the correctness of the merging network only for inputs that are zero-one sequences. Exercise 28.4-1 asks you to show how the proof can be extended to arbitrary input values.

The merging network is based on the following intuition. Given two sorted sequences, if we reverse the order of the second sequence and then concatenate the two sequences, the resulting sequence is bitonic. For example, given the sorted zero-one sequences X = 00000111 and Y = 00001111, we reverse Y to get YR = 11110000. Concatenating X and YR yields 0000011111110000, which is bitonic. Thus, to merge the two input sequences X and Y, it suffices to perform a bitonic sort on X concatenated with YR.

We can construct MERGER[n] by modifying the first half-cleaner of BITONIC-SORTER[n]. The key is to perform the reversal of the the second half of the inputs implicitly. Given two sorted sequences [image] a1, a2, . . . , an/2 [image] and [image] an/2+1, an/2+2, . . . , an [image] to be merged, we want the effect of bitonically sorting the sequence [image] a1, a2, . . . , an/2, an, an-1, . . . , an/2+1 [image]. Since the half-cleaner of BITONIC-SORTER[n] compares inputs i and n/2 + i, for i = 1, 2, . . . , n/2, we make the first stage of the merging network compare inputs i and n - i + 1. Figure 28.10 shows the correspondence. The only subtlety is that the order of the outputs from the bottom of the first stage of MERGER[n] are reversed compared with the order of outputs from an ordinary half-cleaner. Since the reversal of a bitonic sequence is bitonic, however, the top and bottom outputs of the first stage of the merging network satisfy the properties in Lemma 28.3, and thus the top and bottom can be bitonically sorted in parallel to produce the sorted output of the merging network.

Figure 28.10 Comparing the first stage of MERGER[n] with HALF-CLEANER[n], for n = 8. (a) The first stage of MERGER[n] transforms the two monotonic input sequences a1, a2, . . . , an/2 and an/2+1, an/2+2, . . . , an into two bitonic sequences b1, b2, . . . , bn/2 and bn/2+1, bn/2+2, . . . , bn . (b) The equivalent operation for HALF-CLEANER[n]. The bitonic input sequence a1, a2, . . . , an/2-1, an/2, an, an-1, . . . , an/2+2, an/2+1 is transformed into the two bitonic sequences b1, b2, . . . , bn/2 and bn, bn-1, . . . , bn/2+1 .

Figure 28.11 A network that merges two sorted input sequences into one sorted output sequence. The network MERGER[n] can be viewed as BITONIC-SORTER[n] with the first half-cleaner altered to compare inputs i and n - i + 1 for i = 1, 2, . . . , n/2. Here, n = 8. (a) The network decomposed into the first stage followed by two parallel copies of BITONIC-SORTER[n/2]. (b) The same network with the recursion unrolled. Sample zero-one values are shown on the wires, and the stages are shaded.

The resulting merging network is shown in Figure 28.11. Only the first stage of MERGER[n] is different from BITONIC-SORTER[n]. Consequently, the depth of MERGER[n] is lg n, the same as that of BITONIC-SORTER[n].

Exercises

28.4-1

Prove an analog of the zero-one principle for merging networks. Specifically, show that a comparison network that can merge any two monotonically increasing sequences of 0's and 1's can merge any two monotonically increasing sequences of arbitrary numbers.

28.4-2

How many different zero-one input sequences must be applied to the input of a comparison network to verify that it is a merging network?

28.4-3

Show that any network that can merge 1 item with n - 1 items to produce a sorted sequence of length n must have depth at least lg n.

28.4-4

Consider a merging network with inputs a1, a2, . . . , an, for n an exact power of 2, in which the two monotonic sequences to be merged are [image] a1, a3, . . . , an-1 [image] and [image] a2, a4, . . . , an [image]. Prove that the number of comparators in this kind of merging network is [image] (n lg n). Why is this an interesting lower bound? (Hint: Partition the comparators into three sets.)

28.4-5

Prove that any merging network, regardless of the order of inputs, requires [image] (n lg n) comparators.

28.5 A sorting network

We now have all the necessary tools to construct a network that can sort any input sequence. The sorting network SORTER[n] uses the merging network to implement a parallel version of merge sort from Section 1.3.1. The construction and operation of the sorting network are illustrated in Figure 28.12.

Figure 28.12(a) shows the recursive construction of SORTER[n]. The n input elements are sorted by using two copies of SORTER[n/2] recursively to sort (in parallel) two subsequences of length n/2 each. The two resulting sequences are then merged by MERGER[n]. The boundary case for the recursion is when n = 1, in which case we can use a wire to sort the 1-element sequence, since a 1-element sequence is already sorted. Figure 28.12(b) shows the result of unrolling the recursion, and Figure 28.12(c) shows the actual network obtained by replacing the MERGER boxes in Figure 28.12(b) with the actual merging networks.

Figure 28.12 The sorting network SORTER[n] constructed by recursively combining merging networks. (a) The recursive construction. (b) Unrolling the recursion. (c) Replacing the MERGER boxes with the actual merging networks. The depth of each comparator is indicated, and sample zero-one values are shown on the wires.

Data pass through lg n stages in the network SORTER[n]. Each of the individual inputs to the network is already a sorted 1-element sequence. The first stage of SORTER[n] consists of n/2 copies of MERGER[2] that work in parallel to merge pairs of 1-element sequences to produce sorted sequences of length 2. The second stage consists of n/4 copies of MERGER[4] that merge pairs of these 2-element sorted sequences to produce sorted sequences of length 4. In general, for k = 1, 2, . . . , lg n, stage k consists of n/2k copies of MERGER[2k] that merge pairs of the 2k-1-element sorted sequences to produce sorted sequences of length 2k. At the final stage, one sorted sequence consisting of all the input values is produced. This sorting network can be shown by induction to sort zero-one sequences, and consequently, by the zero-one principle (Theorem 28.2), it can sort arbitrary values.

We can analyze the depth of the sorting network recursively. The depth D(n) of SORTER[n] is the depth D(n/2) of SORTER[n/2] (there are two copies of SORTER[n/2], but they operate in parallel) plus the depth lg n of MERGER[n]. Consequently, the depth of SORTER[n] is given by the recurrence

whose solution is D(n) = [image] (lg2 n). Thus, we can sort n numbers in parallel in O(lg2 n) time.

Exercises

28.5-1

How many comparators are there in SORTER[n]?

28.5-2

Show that the depth of SORTER[n] is exactly (lg n)(lgn + 1)/2.

28.5-3

Suppose we modify a comparator to take two sorted lists of length k as inputs, merge them, and output the largest k to its "max" output and the smallest k to its "min" output. Show that any sorting network on n inputs with comparators modified in this fashion can sort nk numbers, assuming that each input to the network is a sorted list of length k.

28.5-4

Suppose that we have 2n elements [image] a1, a2, . . . , a2n [image] and wish to partition them into the n smallest and the n largest. Prove that we can do this in constant additional depth after separately sorting [image] a1,a2, . . . , an [image] and [image] an+1, an+2, . . . , a2n [image].

28.5-5

Let S(k) be the depth of a sorting network with k inputs, and let M(k) be the depth of a merging network with 2k inputs. Suppose that we have a sequence of n numbers to be sorted and know that every number is within k positions of its correct position in the sorted order. Show that we can sort the n numbers in depth S(k) + 2M(k).

28.5-6

We can sort the elements of an m X m matrix by repeating the following procedure k times:

1. Sort each odd-numbered row into monotonically increasing order.

2. Sort each even-numbered row into monotonically decreasing order.

3. Sort each column into monotonically increasing order.

How many iterations k are required for this procedure to sort, and what is the pattern of the sorted output?

Problems

28-1 Transposition sorting networks

A comparison network is a transposition network if each comparator connects adjacent lines, as in the network in Figure 28.3.

a. Show that any transposition network with n inputs that sorts has [image] (n2) comparators.

b. Prove that a transposition network with n inputs is a sorting network if and only if it sorts the sequence [image] n, n- 1, . . . , 1 [image]. (Hint: Use an induction argument analogous to the one in the proof of Lemma 28.1.)

An odd-even sorting network on n inputs [image] a1, a2, . . . , an [image] has n levels of comparators. Figure 28.13 shows an odd-even transposition network on 8 inputs. As can be seen in the figure, for i = 2, 3, . . . , n - 1 and d = 1, 2, . . . , n, line i is connected by a depth-d comparator to line j = i + (-1)i+d if 1 [image] j [image] n.

c. Prove that the family of odd-even sorting networks is indeed a family of sorting networks.

28-2 Batcher's odd-even merging network

In Section 28.4, we saw how to construct a merging network based on bitonic sorting. In this problem, we shall construct an odd-even merging network. We assume that n is an exact power of 2, and we wish to merge the sorted sequence of elements on lines [image] a1, a2, . . . , an [image] with those on lines [image] an+1, an+2, . . . , a2n [image]. We recursively construct two odd-even merging networks that merge sorted subsequences in parallel. The first merges the sequence on lines [image] a1, a3, . . . , an-1 [image] with the sequence on lines [image] an+1, an+3, . . . , a2n-1 [image] (the odd elements). The second merges [image] a2, a4, . . . , an [image] with [image] an+2, an+4, . . . , a2n [image] (the even elements). To combine the two sorted subsequences, we put a comparator between a2i-1 and a2i for i = 1, 2, . . . , n.

a. Draw a 2n-input merging network for n = 4.

Figure 28.13 An odd-even sorting network on 8 inputs.

Figure 28.14 Permutation networks. (a) The permutation network P2, which consists of a single switch that can be set in either of the two ways shown. (b) The recursive construction of P8 from 8 switches and two P4's. The switches and P4's are set to realize the permutation = 4, 7, 3, 5, 1, 6, 8, 2 .

b. Use the zero-one principle to prove that any 2n-input odd-even merging network is indeed a merging network.

c. What is the depth of a 2n-input odd-even merging network? What is its size?

28-3 Permutation networks

A permutation network on n inputs and n outputs has switches that allow it to connect its inputs to its outputs according to any of the n! possible permutations. Figure 28.14(a) shows the 2-input, 2-output permutation network P2, which consists of a single switch that can be set either to feed its inputs straight through to its outputs or to cross them.

a. Argue that if we replace each comparator in a sorting network with the switch of Figure 28.14(a), the resulting network is a permutation network. That is, for any permutation [image], there is a way to set the switches in the network so that input i is connected to output [image] (i).

Figure 28.14(b) shows the recursive construction of an 8-input, 8-output permutation network P8 that uses two copies of P4 and 8 switches. The switches have been set to realize the permutation [image] = [image] 4, 7, 3, 5, 1, 6, 8, 2 [image], which requires (recursively) that the top P4 realize [image] 4, 2, 3, 1 [image] and the bottom P4 realize [image] 2, 3, 1, 4 [image].

b. Show how to realize the permutation [image] 5, 3, 4, 6, 1, 8, 2, 7 [image] on P8 by drawing the switch settings and the permutations performed by the two P4's.

Let n be an exact power of 2. Define Pn recursively in terms of two Pn/2's in a manner similar to the way we defined P8.

c. Describe an O(n)-time (ordinary random-access machine) algorithm that sets the n switches connected to the inputs and outputs of Pn and specifies the permutations that must be realized by each Pn/2 in order to accomplish any given n-element permutation. Prove that your algorithm is correct.

d. What are the depth and size of Pn? How long does it take on an ordinary random-access machine to compute all switch settings, including those within the Pn/2's?

e. Argue that for n > 2, any permutation network--not just Pn--must realize some permutation by two distinct combinations of switch settings.

Chapter notes

Knuth [123] contains a discussion of sorting networks and charts their history. They apparently were first explored in 1954 by P. N. Armstrong, R. J. Nelson, and D. J. O'Connor. In the early 1960's, K. E. Batcher discovered the first network capable of merging two sequences of n numbers in O(lg n) time. He used odd-even merging (see Problem 28-2), and he also showed how this technique could be used to sort n numbers in O(lg2 n) time. Shortly afterwards, he discovered an O(lg n)-depth bitonic sorter similar to the one presented in Section 28.3. Knuth attributes the zero-one principle to W. G. Bouricius (1954), who proved it in the context of decision trees.

For a long time, the question remained open as to whether a sorting network with depth O(lg n) exists. In 1983, the answer was shown to be a somewhat unsatisfying yes. The AKS sorting network (named after its developers, Ajtai, Komlós, and Szemerédi [8]) can sort n numbers in depth O(lg n) using O(n lg n) comparators. Unfortunately, the constants hidden by the O-notation are quite large (many, many thousands), and thus it cannot be considered practical.

Go to Chapter 29 Back to Table of Contents


Intro to Algorithms: CHAPTER 29: ARITHMETIC CIRCUITS

 Return to Table of Contents

CHAPTER 29: ARITHMETIC CIRCUITS

The model of computation provided by an ordinary computer assumes that the basic arithmetic operations--addition, subtraction, multiplication, and division--can be performed in constant time. This abstraction is reasonable, since most basic operations on a random-access machine have similar costs. When it comes to designing the circuitry that implements these operations, however, we soon discover that performance depends on the magnitudes of the numbers being operated on. For example, we all learned in grade school how to add two natural numbers, expressed as n-digit decimal numbers, in [image] (n) steps (although teachers usually do not emphasize the number of steps required).

This chapter introduces circuits that perform arithmetic functions. With serial processes, [image] (n) is the best asymptotic time bound we can hope to achieve for adding two n-digit numbers. With circuits that operate in parallel, however, we can do better. In this chapter, we shall design circuits that can quickly perform addition and multiplication. (Subtraction is essentially the same as addition, and division is deferred to Problem 29-1.) We shall assume that all inputs are n-bit natural numbers, expressed in binary.

We start in Section 29.1 by presenting combinational circuits. We shall see how the depth of a circuit corresponds to its "running time." The full adder, which is a building block of most of the circuits in this chapter, serves as our first example of a combinational circuit. Section 29.2 presents two combinational circuits for addition: the ripple-carry adder, which works in [image] (n) time, and the carry-lookahead adder, which takes only O(1g n) time. It also presents the carry-save adder, which can reduce the problem of summing three numbers to the problem of summing two numbers in [image] (1) time. Section 29.3 introduces two combinational multipliers: the array multiplier, which takes [image] (n) time, and the Wallace-tree multiplier, which requires only [image] (1g n) time. Finally, Section 29.4 presents circuits with clocked storage elements (registers) and shows how hardware can be saved by reusing combinational circuitry.

29.1 Combinational circuits

Like the comparison networks of Chapter 28, combinational circuits operate in parallel: many elements can compute values simultaneously as a single step. In this section, we define combinational circuits and investigate how larger combinational circuits can be built up from elementary gates.

Combinational elements

Arithmetic circuits in real computers are built from combinational elements that are interconnected by wires. A combinational element is any circuit element that has a constant number of inputs and outputs and that performs a well-defined function. Some of the elements we shall deal with in this chapter are boolean combinational elements--their inputs and outputs are all drawn from the set {0,1}, where 0 represents FALSE and 1 represents TRUE.

A boolean combinational element that computes a simple boolean function is called a logic gate. Figure 29.1 shows the four basic logic gates that will serve as combinational elements in this chapter: the NOT gate (or inverter), the AND gate, the OR gate, and the XOR gate. (It also shows two other logic gates--the NAND gate and the NOR gate--that are required by some of the exercises.) The NOT gate takes a single binary input x, whose value is either 0 or 1, and produces a binary output z whose value is opposite that of the input value. Each of the other three gates takes two binary inputs x and y and produces a single binary output z.

The operation of each gate, and of any boolean combinational element, can be described by a truth table, shown under each gate in Figure 29.1. A truth table gives the outputs of the combinational element for each possible setting of the inputs. For example, the truth table for the XOR gate tells us that when the inputs are x = 0 and y = 1, the output value is z = 1; it computes the "exclusive OR" of its two inputs. We use the symbols [image] to denote the NOT function, [image] to denote the AND function, [image] to denote the XOR function, and [image] to denote the XOR function. Thus, for example, 0 [image] 1 = 1.

Combinational elements in real circuits do not operate instantaneously. Once the input values entering a combinational element settle, or become stable--that is, hold steady for a long enough time--the element's output value is guaranteed to become both stable and correct a fixed amount of time later. We call this time differential the propagation delay of the element. We assume in this chapter that all combinational elements have constant propagation delay.

Figure 29.1 Six basic logic gates, with binary inputs and outputs. Under each gate is the truth table that describes the gate's operation. (a) The NOT gate. (b) The AND gate. (c) The OR gate. (d) The XOR (exclusive-OR) gate. (e) The NAND (NOT-AND) gate. (f) The NOR (NOT-OR) gate.

Combinational circuits

A combinational circuit consists of one or more combinational elements interconnected in an acyclic fashion. The interconnections are called wires. A wire can connect the output of one element to the input of another, thereby providing the output value of the first element as an input value of the second. Although a single wire may have no more than one combinational-element output connected to it, it can feed several element inputs. The number of element inputs fed by a wire is called the fan-out of the wire. If no element output is connected to a wire, the wire is a circuit input, accepting input values from an external source. If no element input is connected to a wire, the wire is a circuit output, providing the results of the circuit's computation to the outside world. (An internal wire can also fan out to a circuit output.) Combinational circuits contain no cycles and have no memory elements (such as the registers described in Section 29.4).

Full adders

As an example, Figure 29.2 shows a combinational circuit, called a full adder, that takes as input three bits x, y, and z. It outputs two bits, s and c, according to the following truth table:

x  y  z  c  s

-------------

0  0  0  0  0

0  0  1  0  1

0  1  0  0  1

0  1  1  1  0

1  0  0  0  1

1  0  1  1  0

1  1  0  1  0

1  1  1  1  1

Output s is the parity of the input bits,

s = parity(x,y,z) = x    y    z,

(29.1)

and output c is the majority of the input bits,

c = majority(x,y,z) = (x    y)    (y    z)    (x    z).

(29.2)

(In general, the parity and majority functions can take any number of input bits. The parity is 1 if and only if an odd number of the inputs are 1's. The majority is 1 if and only if more than half the inputs are 1's.) Note that the c and s bits, taken together, give the sum of x, y, and z. For example, if x = 1, y = 0, and z = 1, then [image] c, s [image] = [image] 10 [image],1 which is the binary representation of 2, the sum of x, y, and z.

1For clarity, we omit the commas between sequence elements when they are bits.

Each of the inputs x, y, and z to the full adder has a fan-out of 3. When the operation performed by a combinational element is commutative and associative with respect to its inputs (such as the functions AND, OR, and XOR), we call the number of inputs the fan-in of the element. Although the fan-in of each gate in Figure 29.2 is 2, we could redraw the full adder to replace XOR gates A and E by a single 3-input XOR gate and OR gates F and G by a single 3-input OR gate.

To examine how the full adder operates, assume that each gate operates in unit time. Figure 29.2(a) shows a set of inputs that becomes stable at time 0. Gates A-D, and no other gates, have all their input values stable at that time and therefore produce the values shown in Figure 29.2(b) at time 1. Note that gates A-D operate in parallel. Gates E and F, but not gate G, have stable inputs at time 1 and produce the values shown in Figure 29.2(c) at time 2. The output of gate E is bit s, and so the s output from the full adder is ready at time 2. The c output is not yet ready, however. Gate G finally has stable inputs at time 2, and it produces the c output shown in Figure 29.2(d) at time 3.

Figure 29.2 A full-adder circuit. (a) At time 0, the input bits shown appear on the three input wires. (b) At time 1, the values shown appear on the outputs of gates A-D, which are at depth 1. (c) At time 2, the values shown appear on the outputs of gates E and F, at depth 2. (d) At time 3, gate G produces its output, which is also the circuit output.

Circuit depth

As in the case of the comparison networks discussed in Chapter 28, we measure the propagation delay of a combinational circuit in terms of the largest number of combinational elements on any path from the inputs to the outputs. Specifically, we define the depth of a circuit, which corresponds to its worst-case "running time," inductively in terms of the depths of its constituent wires. The depth of an input wire is 0. If a combinational element has inputs x1, x2, . . . ,xn at depths d1, d2, . . . ,dn respectively, then its outputs have depth max{d1, d2, . . . ,dn} + 1. The depth of a combinational element is the depth of its outputs. The depth of a combinational circuit is the maximum depth of any combinational element. Since we prohibit combinational circuits from containing cycles, the various notions of depth are well defined.

If each combinational element takes constant time to compute its output values, then the worst-case propagation delay through a combinational circuit is proportional to its depth. Figure 29.2 shows the depth of each gate in the full adder. Since the gate with the largest depth is gate G, the full adder itself has depth 3, which is proportional to the worst-case time it takes for the circuit to perform its function.

A combinational circuit can sometimes compute faster than its depth. Suppose that a large subcircuit feeds into one input of a 2-input AND gate but that the other input of the AND gate has value 0. The output of the gate will then be 0, independent of the input from the large subcircuit. In general, however, we cannot count on specific inputs being applied to the circuit, and the abstraction of depth as the "running time" of the circuit is therefore quite reasonable.

Circuit size

Besides circuit depth, there is another resource that we typically wish to minimize when designing circuits. The size of a combinational circuit is the number of combinational elements it contains. Intuitively, circuit size corresponds to the memory space used by an algorithm. The full adder of Figure 29.2 has size 7, for example, since it uses 7 gates.

This definition of circuit size is not particularly useful for small circuits. After all, since a full adder has a constant number of inputs and outputs and computes a well-defined function, it satisfies the definition of a combinational element. A full adder built from a single full-adder combinational element therefore has size 1. In fact, according to this definition, any combinational element has size 1.

The definition of circuit size is intended to apply to families of circuits that compute similar functions. For example, we shall soon see an addition circuit that takes two n-bit inputs. We are really not talking about a single circuit here, but rather a family of circuits--one for each size of input. In this context, the definition of circuit size makes good sense. It allows us to define convenient circuit elements without affecting the size of any implementation of the circuit by more than a constant factor. Of course, in practice, measurements of size are much more complicated, involving not only the choice of combinational elements, but also concerns such as the area the circuit requires when integrated on a silicon chip.

Exercises

29.1-1

In Figure 29.2, change input y to a 1. Show the resulting value carried on each wire.

29.1-2

Show how to construct an n-input parity circuit with n - 1 XOR gates and depth [image] lg n [image].

29.1-3

Show that any boolean combinational element can be constructed from a constant number of AND, OR, and NOT gates. (Hint: Implement the truth table for the element.)

29.1-4

Show that any boolean function can be constructed entirely out of NAND gates.

29.1-5

Construct a combinational circuit that performs the exclusive-or function using only four 2-input NAND gates.

29.1-6

Let C be an n-input, n-output combinational circuit of depth d. If two copies of C are connected, with the outputs of one feeding directly into the inputs of the other, what is the maximum possible depth of this tandem circuit? What is the minimum possible depth?

29.2 Addition circuits

We now investigate the problem of adding numbers represented in binary. We present three combinational circuits for this problem. First, we look at ripple-carry addition, which can add two n-bit numbers in [image] (n) time using a circuit with [image] (n) size. This time bound can be improved to O(lg n) using a carry-lookahead adder, which also has [image] (n) size. Finally, we present carry-save addition, which in O(1) time can reduce the sum of 3 n-bit numbers to the sum of an n-bit number and an (n + 1)-bit number. The circuit has [image] (n) size.

8  7  6  5  4  3  2  1  0     i

1  1  0  1  1  1  0  0  0  =  c

   0  1  0  1  1  1  1  0  =  a

   1  1  0  1  0  1  0  1  =  b

-------------------------------

1  0  0  1  1  0  0  1  1  =  s

Figure 29.3 Adding two 8-bit numbers a = 01011110 and b = 11010101 to produce a 9-bit sum s = 100110011 . Each bit ci is a carry bit. Each column of bits represents, from top to bottom, ci, ai, bi, and si for some i. Carry-in c0 is always 0.

29.2.1 Ripple-carry addition

We start with the ordinary method of summing binary numbers. We assume that a nonnegative integer a is represented in binary by a sequence of n bits [image] an-1, an-2, . . . , a0 [image], where n [image] [image] lg(a + 1) [image] and

Given two n-bit numbers a = [image] an-1, an-2, . . . , a0 [image] and b = [image] bn-1, bn-2, . . . , b0 [image], we wish to produce an (n + 1)-bit sum s = [image] sn, sn-1, . . . , s0 [image] . Figure 29.3 shows an example of adding two 8-bit numbers. We sum columns right to left, propagating any carry from column i to column i + 1, for i = 0, 1, . . . , n - 1. In the ith bit position, we take as inputs bits ai and bi and a carry-in bit ci, and we produce a sum bit si and a carry-out bit ci+1. The carry-out bit ci+1 from the ith position is the carry-in bit into the (i + 1)st position. Since there is no carry-in for position 0, we assume that c0 = 0. The carry-out cn is bit sn of the sum.

Observe that each sum bit si is the parity of bits ai, bi, and ci (see equation (29.1)). Moreover, the carry-out bit ci+1 is the majority of ai, bi, and ci (see equation (29.2)). Thus, each stage of the addition can be performed by a full adder.

An n-bit ripple-carry adder is formed by cascading n full adders FA0, FA1, . . . , FAn-1, feeding the carry-out ci+1 of FAi directly into the carry-in input of FAi+1. Figure 29.4 shows an 8-bit ripple-carry adder. The carry bits "ripple" from right to left. The carry-in c0 to full adder FA1 is hardwired to 0, that is, it is 0 no matter what values the other inputs take on. The output is the (n+1)-bit number s = [image] sn, sn-1, . . . , s0 [image], where sn equals cn, the carry-out bit from full adder FAn.

Because the carry bits ripple through all n full adders, the time required by an n-bit ripple-carry adder is [image] (n). More precisely, full adder FAi is at depth i + 1 in the circuit. Because FAn-1 is at the largest depth of any full adder in the circuit, the depth of the ripple-carry adder is n. The size of the circuit is [image] (n) because it contains n combinational elements.

Figure 29.4 An 8-bit ripple-carry adder performing the addition of Figure 29.3. Carry bit c0 is hardwired to 0, indicated by the diamond, and carry bits ripple from right to left.

29.2.2 Carry-lookahead addition

Ripple-carry addition requires [image] (n) time because of the rippling of carry bits through the circuit. Carry-lookahead addition avoids this [image] (n)-time delay by accelerating the computation of carries using a treelike circuit. A carry-lookahead adder can sum two n-bit numbers in O(lg n) time.

The key observation is that in ripple-carry addition, for i [image] 1, full adder FAi has two of its input values, namely ai and bi, ready long before the carry-in ci is ready. The idea behind the carry-lookahead adder is to exploit this partial information.

As an example, let ai-1 = bi-1. Since the carry-out ci is the majority function, we have ci = ai-1 = bi-1 regardless of the carry-in ci-1. If ai-1 = bi-1 = 0, we can kill the carry-out ci by forcing it to 0 without waiting for the value of ci-1 to be computed. Likewise, if ai-1 = bi-1 = 1, we can generate the carry-out ci = 1, irrespective of the value of ci-1.

If ai-1 [image] bi-1, however, then ci depends on ci-1. Specifically, ci = ci-1, because the carry-in ci-1 casts the deciding "vote" in the majority election that determines ci. In this case, we propagate the carry, since the carry-out is the carry-in.

Figure 29.5 summarizes these relationships in terms of carry statuses, where k is "carry kill," g is "carry generate," and p is "carry propagate."

Consider two consecutive full adders FAi-1 and FAi together as a combined unit. The carry-in to the unit is ci-1, and the carry-out is ci+1. We can view the combined unit as killing, generating, or propagating carries, much as for a single full adder. The combined unit kills its carry if FAi kills its carry or if FAi-1 kills its carry and FAi propagates it. Similarly, the combined unit generates a carry if FAi generates a carry or if FAi-1 generates a carry and FAi propagates it. The combined unit propagates the carry, setting ci+1 = ci-1, if both full adders propagate carries. The table in Figure 29.6 summarizes how carry statuses are combined when full adders are juxtaposed. We can view this table as the definition of the carry-status operator [image] over the domain {k, p, g}. An important property of this operator is that it is associative, as Exercise 29.2-2 asks you to verify.

ai-1  bi-1   ci   carry status

-----------------------------

  0    0     0        k

  0    1    ci-1      p

  1    0    ci-1      p

  1    1     1        g

Figure 29.5 The carry-out bit ci and carry status corresponding to inputs ai-1, bi-1, and ci-1 of full adder FAi-1 in ripple-carry addition.

              FAi

           k   p   g

--------------------

       k  k   k   g

FAi-1  p  k   p   g

       g  k   g   g

Figure 29.6 The carry status of the combination of full adders FAi-1 and FAi in terms of their individual carry statuses, given by the carry-status operator over the domain {k, p, g}.

We can use the carry-status operator to express each carry bit ci in terms of the inputs. We start by defining x0 = k and

(29.3)

for i = 1, 2, . . . , n. Thus, for i = 1, 2, . . . , n, the value of xi is the carry status given by Figure 29.5.

The carry-out ci of a given full adder FAi-1 can depend on the carry status of every full adder FAj for j = 0, 1, . . . , i - 1. Let us define y0 = x0 = k and

yi  = yi-1    xi

= x0    x1              xi

(29.4)

for i = 1, 2, . . . , n. We can think of yi as a "prefix" of x0 [image] x1 [image] [image] [image] [image] [image] xn; we call the process of computing the values y0, y1, . . . , yn a prefix computation. (Chapter 30 discusses prefix computations in a more general parallel context.) Figure 29.7 shows the values of xi and yi corresponding to the binary addition shown in Figure 29.3. The following lemma gives the significance of the yi values for carry-lookahead addition.

Figure 29.7 The values of xi and yi for i = 0, 1, . . . , 8 that correspond to the values of ai, bi, and ci in the binary-addition problem of Figure 29.3. Each value of xi is shaded with the values of ai-1 and bi-1 that it depends on.

Lemma 29.1

Define x0, x1, . . . , xn and y0, y1, . . . , yn by equations (29.3) and (29.4). For i = 0, 1, . . . ,n, the following conditions hold:

1. yi = k implies ci = 0,

2. yi = g implies ci = 1, and

3. yi = p does not occur.

Proof The proof is by induction on i. For the basis, i = 0. We have y0 = x0 = k by definition, and also c0 = 0. For the inductive step, assume that the lemma holds for i - 1. There are three cases depending on the value of yi.

1. If yi = k, then since yi = yi-1 [image] xi, the definition of the carry-status operator [image] from Figure 29.6 implies either that xi = k or that xi = p and yi-1 = k. If xi = k, then equation (29.3) implies that ai-1 = bi-1 = 0, and thus ci = majority (ai-1, bi-1, ci-1) = 0. If xi = p and yi-1 = k, then ai-1 [image] bi-1 and, by induction, ci-1 = 0. Thus, majority (ai-1, bi-1, ci-1) = 0, and thus ci = 0.

2. If yi = g, then either we have xi = g or we have xi = p and yi-1 = g. If xi = g, then ai-1 = bi-1= 1, which implies ci = 1. If xi = p and yi-1 = g, then ai-1 [image] bi-1 and, by induction, ci-1 = 1, which implies ci= 1.

3. If yi = p, then Figure 29.6 implies that yi-1 = p, which contradicts the inductive hypothesis.

Lemma 29.1 implies that we can compute each carry bit ci by computing each carry status yi. Once we have all the carry bits, we can compute the entire sum in [image] (1) time by computing in parallel the sum bits si = parity (ai, bi, ci) for i = 0, 1, . . . , n (taking an = bn = 0). Thus, the problem of quickly adding two numbers reduces to the prefix computation of the carry statuses y0, y1, . . . ,yn.

Computing carry statuses with a parallel prefix circuit

By using a prefix circuit that operates in parallel, as opposed to a ripple-carry circuit that produces its outputs one by one, we can compute all n carry statuses y0, y1, . . . , yn more quickly. Specifically, we shall design a parallel prefix circuit with O(lg n) depth. The circuit has [image] (n) size--asymptotically the same amount of hardware as a ripple-carry adder.

Before constructing the parallel prefix circuit, we introduce a notation that will aid our understanding of how the circuit operates. For integers i and j in the range 0 [image] i [image] j [image] n, we define

[i,j] = xi    xi+1              xj.

Thus, for i = 0, 1, . . . , n, we have [i, i] = xi, since the composition of just one carry status xi is itself. For i, j, and k satisfying 0 [image] i < j [image] k [image] n, we also have the identity

[i,k] = [i,j - 1]    [j,k],

(29.5)

since the carry-status operator is associative. The goal of a prefix computation, in terms of this notation, is to compute yi = [0, i] for i= 0, 1, . . . , n.

The only combinational element used in the parallel prefix circuit is a circuit that computes the [image] operator. Figure 29.8 shows how pairs of [image] elements are organized to form the internal nodes of a complete binary tree, and Figure 29.9 illustrates the parallel prefix circuit for n = 8. Note that the wires in the circuit follow the structure of a tree, but the circuit itself is not a tree, although it is purely combinational. The inputs x1, x2, . . . , xn are supplied at the leaves, and the input x0 is provided at the root. The outputs y0, y1, . . . , yn - 1 are produced at leaves, and the output yn is produced at the root. (For ease in understanding the prefix computation, variable indices increase from left to right in Figures 29.8 and 29.9, rather than from right to left as in other figures of this section.)

The two [image] elements in each node typically operate at different times and have different depths in the circuit. As shown in Figure 29.8, if the subtree rooted at a given node spans some range xi, xi+1, . . . , xk of inputs, its left subtree spans the range xi, xi+1, . . . , xj - 1, and its right subtree spans the range xj, xj+1, . . . , xk, then the node must produce for its parent the product [i, k] of all inputs spanned by its subtree. Since we can assume inductively that the node's left and right children produce the products [i, j - 1] and [j, k], the node simply uses one of its two elements to compute [i, k] [image] [i, j - 1] [image] [j, k].

Some time after this upward phase of computation, the node receives from its parent the product [0, i - 1] of all inputs that come before the leftmost input xi that it spans. The node now likewise computes values for its children. The leftmost input spanned by the node's left child is also xi, and so it passes the value [0, i - 1] to the left child unchanged. The leftmost input spanned by its right child is xj, and so it must produce [0, j - 1]. Since the node receives the value [0, i - 1] from its parent and the value [i, j - 1] from its left child, it simply computes [0, j - 1] [image] [0, i - 1] [image] [i, k] and sends this value to the right child.

Figure 29.8 The organization of a parallel prefix circuit. The node shown is the root of a subtree whose leaves input the values xi to xk. The node's left subtree spans inputs xi to xj - 1, and its right subtree spans inputs xj to xk. The node consists of two elements, which operate at different times during the operation of the circuit. One element computes [i, k] [i, j - 1] [j, k], and the other element computes [0, j - 1] [0, i - 1] [i, j - 1]. The values computed are shown on the wires.

Figure 29.9 shows the resulting circuit, including the boundary case that arises at the root. The value x0 = [0, 0] is provided as input at the root, and one more [image] element is used to compute (in general) the value yn = [0, n] = [0, 0] [image] [1, n].

If n is an exact power of 2, then the parallel prefix circuit uses 2n - 1 [image] elements. It takes only O(lg n) time to compute all n + 1 prefixes, since the computation proceeds up the tree and then back down. Exercise 29.2-5 studies the depth of the circuit in more detail.

Completing the carry-lookahead adder

Now that we have a parallel prefix circuit, we can complete the description of the carry-lookahead adder. Figure 29.10 shows the construction. An n-bit carry-lookahead adder consists of n + 1 KPG boxes, each of [image] (1) size, and a parallel prefix circuit with inputs x0, x1, . . . , xn (x0 is hardwired to k) and outputs y0, y1, . . . , yn. KPG box KPGi takes external inputs ai and bi and produces sum bit si. (Input bits an and bn are hardwired to 0.) Given ai-1 and bi-1, box KPGi-1 computes xi [image] {k,p,g} according to equation (29.3) and sends this value as the external input xi of the parallel prefix circuit. (The value of xn+1 is ignored.) Computing all the xi takes [image] (1) time. After a delay of O(lg n), the parallel prefix circuit produces y0, y1, . . . , yn. By Lemma 29.1, yi is either k or g; it cannot be p. Each value yi indicates the carry-in to full adder FAi in the ripple-carry adder: yi = k implies ci = 0, and yi = g implies ci = 1. Thus, the value of yi is fed into KPGi to indicate the carry-in ci, and the sum bit si = parity(ai, bi, ci) is produced in constant time. Thus, the carry-lookahead adder operates in O(lg n) time and has [image] (n) size.

Figure 29.9 A parallel prefix circuit for n = 8. (a) The overall structure of the circuit, and the values carried on each wire. (b) The same circuit with values corresponding to Figures 29.3 and 29.7.

Figure 29.10 The construction of an n-bit carry-lookahead adder, shown here for n = 8. It consists of n + 1 KPG boxes KPGi for i = 0, 1, . . . , n. Each box KPGi takes external inputs ai and bi (where an and bn are hardwired to 0, as indicated by the diamond) and computes carry status xi+1. These values are fed into the parallel prefix circuit, which returns the results yi of the prefix computation. Each box KPGi now takes yi as input, interprets it as the carry-in bit ci, and then outputs the sum bit si = parity (ai, bi, ci). Sample values corresponding to those shown in Figures 29.3 and 29.9 are shown.

29.2.3 Carry-save addition

A carry-lookahead adder can add two n-bit numbers in O(lg n) time. Perhaps surprisingly, adding three n-bit numbers takes only a constant additional amount of time. The trick is to reduce the problem of adding three numbers to the problem of adding just two numbers.

Given three n-bit numbers x = [image] xn-1, xn-2, . . . . , x0 [image], y = [image] yn-1, yn-2, . . . , y0 [image], and z = [image] zn-1,zn-2, . . . , z0 [image], an n-bit carry-save adder produces an n-bit number u = [image] un-1, un-2, . . . , u0 [image] and an (n + 1)-bit number v = [image] vn, vn-1, . . . , v0 [image] such that

u + v = x + y + z.

As shown in Figure 29.11 (a), it does this by computing

ui = parity (xi, yi, zi),

vi + 1 = majority (xi, yi, zi),

for i = 0, 1, . . . , n - 1. Bit v0 always equals 0.

The n-bit carry-save adder shown in Figure 29.11(b) consists of n full adders FA0, FA1, . . . , FAn - 1. For i = 0, 1, . . . , n - 1, full adder FAi takes inputs xi, yi, and zi. The sum-bit output of FAi is taken as ui, and the carry-out of FAi is taken as vi+1. Bit v0 is hardwired to 0.

Since the computations of all 2n + 1 output bits are independent, they can be performed in parallel. Thus, a carry-save adder operates in [image] (1) time and has [image] (n) size. To sum three n-bit numbers, therefore, we need only perform a carry-save addition, taking [image] (1) time, and then perform a carry-lookahead addition, taking O(lg n) time. Although this method is not asymptotically better than the method of using two carry-lookahead additions, it is much faster in practice. Moreover, we shall see in Section 29.3 that carry-save addition is central to fast algorithms for multiplication.

Figure 29.11 (a) Carry-save addition. Given three n-bit numbers x, y, and z, we produce an n-bit number u and an (n + 1)-bit number v such that x + y + z = u + v. The ith pair of shaded bits are a function of xi, yi, and zi. (b) An 8-bit carry-save adder. Each full adder FAi takes inputs xi, yi, and zi and produces sum bit ui and carry-out bit vi + 1. Bit v0 is hardwired to 0.

Exercises

29.2-1

Let a = [image] 01111111 [image], b = [image] 00000001 [image], and n = 8. Show the sum and carry bits output by full adders when ripple-carry addition is performed on these two sequences. Show the carry statuses x0, x1, . . . , x8 corresponding to a and b, label each wire of the parallel prefix circuit of Figure 29.9 with the value it has given these xi inputs, and show the resulting outputs y0, y1, . . . , y8.

29.2-2

Prove that the carry-status operator [image] given by Figure 29.5 is associative.

Figure 29.12 A parallel prefix circuit for use in Exercise 29.2-6.

29.2-3

Show by example how to construct an O(lg n)-time parallel prefix circuit for values of n that are not exact powers of 2 by drawing a parallel prefix circuit for n = 11. Characterize the performance of parallel prefix circuits built in the shape of arbitrary binary trees.

29.2-4

Show the gate-level construction of the box KPGi. Assume that each output xi is represented by [image] 00 [image] if xi = k, by [image] 11 [image] if xi = g, and by [image] 01 [image] or [image] 10 [image] if xi = p. Assume also that each input yi is represented by 0 if yi = k and by 1 if yi = g.

29.2-5

Label each wire in the parallel prefix circuit of Figure 29.9(a) with its depth. A critical path in a circuit is a path with the largest number of combinational elements on any path from inputs to outputs. Identify the critical path in Figure 29.9(a), and show that its length is O(lg n). Show that some node has [image] elements that operate [image] (lg n) time apart. Is there a node whose [image] elements operate simultaneously?

29.2-6

Give a recursive block diagram of the circuit in Figure 29.12 for any number n of inputs that is an exact power of 2. Argue on the basis of your block diagram that the circuit indeed performs a prefix computation. Show that the depth of the circuit is [image] (lg n) and that it has [image] (n lg n) size.

29.2-7

What is the maximum fan-out of any wire in the carry-lookahead adder? Show that addition can still be performed in O(lg n) time by a [image] (n)-size circuit even if we restrict gates to have O(1) fan-out.

29.2-8

A tally circuit has n binary inputs and m = [image] lg(n + 1) [image] outputs. Interpreted as a binary number, the outputs give the number of 1's in the inputs. For example, if the input is [image] 10011110 [image], the output is [image] 101 [image], indicating that there are five 1's in the input. Describe an O(lg n)-depth tally circuit having [image] (n) size.

29.2-9

Show that n-bit addition can be accomplished with a combinational circuit of depth 4 and size polynomial in n if AND and OR gates are allowed arbitrarily high fan-in. (Optional: Achieve depth 3.)

29.2-10

Suppose that two random n-bit numbers are added with a ripple-carry adder, where each bit is independently 0 or 1 with equal probability. Show that with probability at least 1 - 1/n, no carry propagates farther than O(1g n) consecutive stages. In other words, although the depth of the ripple-carry adder is [image] (n), for two random numbers, the outputs almost always settle within O(lg n) time.

29.3 Multiplication circuits

The "grade-school" multiplication algorithm in Figure 29.13 can compute the 2n-bit product p = [image] p2n-1, p2n-2, . . . . , p0 [image] of two n-bit numbers a = [image] an-1, an-2, . . . , a0 [image] and b = [image] bn-1, bn-2, . . . , b0 [image]. We examine the bits of b, from b0 up to bn-1. For each bit bi with a value of 1, we add a into the product, but shifted left by i positions. For each bit bi with a value of 0, we add in 0. Thus, letting m(i) = a . bi . 2i, we compute

Each term m(i) is called a partial product. There are n partial products to sum, with bits in positions 0 to 2n - 2. The carry-out from the highest bit yields the final bit in position 2n - 1.

In this section, we examine two circuits for multiplying two n-bit numbers. Array multipliers operate in [image] (n) time and have [image] (n2) size. Wallace-tree multipliers also have [image] (n2) size, but they operate in [image] (lg n) time. Both circuits are based on the grade-school algorithm.

            1  1  1  0  =  a

            1  1  0  1  =  b

-------------------------------

            1  1  1  0  =  m(0)

         0  0  0  0     =  m(1)

      1  1  1  0        =  m(2)

   1  1  1  0           =  m(3)

-------------------------------

1  0  1  1  0  1  1  0  =  p

Figure 29.13 The "grade-school" multiplication method, shown here multiplying a = 1110 by b = 1101 to obtain the product p = 10110110 . We add , where m(i) = a bi 2i. Here, n = 8. Each term m(i) is formed by shifting either a (if bi = 1 ) or 0 (if bi = 0) i positions to the left. Bits that are not shown are 0 regardless of the values of a and b.

29.3.1 Array multipliers

An array multiplier consists conceptually of three parts. The first part forms the partial products. The second sums the partial products using carry-save adders. Finally, the third sums the two numbers resulting from the carry-save additions using either a ripple-carry or carry-lookahead adder.

Figure 29.14 shows an array multiplier for two input numbers a = [image] an-1, an-2, . . . , a0 [image] and b = [image] bn-1, bn-2, . . . , b0 [image]. The aj values run vertically, and the bi values run horizontally. Each input bit fans out to n AND gates to form partial products. Full adders, which are organized as carry-save adders, sum partial products. The lower-order bits of the final product are output on the right. The higher-order bits are formed by adding the two numbers output by the last carry-save adder.

Let us examine the construction of the array multiplier more closely. Given the two input numbers a = [image] an-1, an-2, . . . , a0 [image] and b = [image] bn-1, bn-2, . . . , b0 [image] , the bits of the partial products are easy to compute. Specifically, for i, j = 0, 1, . . . , n - 1, we have

Since the product of 1-bit values can be computed directly with an AND gate, all the bits of the partial products (except those known to be 0, which need not be explicitly computed) can be produced in one step using n2 AND gates.

Figure 29.15 illustrates how the array multiplier performs the carry-save additions when summing the partial products in Figure 29.13. It starts by carry-save adding m(0), m(1), and 0, yielding an (n + 1)-bit number u(1) and an (n + 1)-bit number v(1). (The number v(1) has only n + 1 bits, not n + 2, because the (n + 1)st bits of both 0 and m(0) are 0.) Thus, m(0) + m(1) = u(1) + v(1). It then carry-save adds u(1), v(1), and m(2), yielding an (n + 2)-bit number u(2) and an (n + 2)-bit number v(2). (Again, v(2) has only n + 2 bits because both [image] and [image] are 0.) We then have m(0) + m(1) + m(2) = u(2) + v(2). The multiplier continues on, carry-save adding u(i-1), v(i-1), and m(i) for i = 2, 3, . . . , n - 1. The result is a (2n - 1)-bit number u(n - l) and a (2n - 1)-bit number v(n - 1), where

Figure 29.14 An array multiplier that computes the product p = p2n-1,p2n-2, . . . , p0 of two n-bit numbers a = an-1,an-2, . . . ,a0 and b = bn-1,bn-2, . . . , b0 , shown here for n = 4. Each AND gate computes partial-product bit . Each row of full adders constitutes a carry-save adder. The lower n bits of the product are and the u bits coming out from the rightmost column of full adders. The upper n product bits are formed by adding the u and v bits coming out from the bottom row of full adders. Shown are bit values for inputs a = 1110 and b = 1101 and product p = 10110110 , corresponding to Figures 29.13 and 29.15.

            0  0  0  0  =  0

            1  1  1  0  =  m(0)

         0  0  0  0     =  m(1)

-------------------------------

         0  1  1  1  0  =  u(1)

         0  0  0        =  v(1)

      1  1  1  0        =  m(2)

-------------------------------

      1  1  0  1  1  0  =  u(2)

      0  1  0           =  v(2)

   1  1  1  0           =  m(3)

-------------------------------

   1  0  1  0  1  1  0  =  u(3)

   1  1  0              =  v(3)

-------------------------------

1  0  1  1  0  1  1  0  =  p

Figure 29.15 Evaluating the sum of the partial products by repeated carry-save addition. For this example, a = 1110 and b = 1101 . Bits that are blank are 0 regardless of the values of a and b. We first evaluate m(0) + m(1) + 0 = u(1) + v(1) then u(1) + v(1)+ m(2) = u(2) + v(2), then u(2) + v(2) + m(3) = u(3) + v(3), and finally p = m(0) + m(1) + m(2) + m(3) = u(3) + v(3) Note that and for i = 1, 2, . . . , n - 1.

In fact, the carry-save additions in Figure 29.15 operate on more bits than strictly necessary. Observe that for i = 1, 2, . . . , n - 1 and j = 0, 1, . . . , i - 1, we have [image] because of how we shift the partial products. Observe also that [image] for i = 1, 2, . . . , n - 1 and j = 0, 1, . . . , i, i + n, i + n + 1, . . . , 2n - 1. (See Exercise 29.3-1.) Each carry-save addition, therefore, needs to operate on only n - 1 bits.

Let us now examine the correspondence between the array multiplier and the repeated carry-save addition scheme. Each AND gate is labeled by [image] for some i and j in the ranges 0 [image] i [image] n - 1 and 0 [image] j [image] 2n - 2. Gate [image] produces [image], the jth bit of the ith partial product. For i = 0, 1, . . . , n - 1, the ith row of AND gates computes the n significant bits of the partial product m(i), that is, [image].

Except for the full adders in the top row (that is, for i = 2, 3, . . . , n - 1), each full adder [image] takes three input [image] , and [image] --and produces two output [image] and [image]. (Note that in the leftmost column of full adders, [image].) Each full adder [image] in the top row takes inputs [image], and 0 and produces bits [image].

Finally, let us examine the output of the array multiplier. As we observed above, [image] for j = 0, 1, . . . , n - 1. Thus, [image] for j = 0, 1, . . . , n - 1. Moreover, since [image], we have [image], and since the lowest-order i bits of each m(i) and v(i - 1) are 0, we have [image] for i = 2, 3, . . . , n- 1 and j = 0, 1, . . . , i - 1. Thus, [image] and, by induction, [image] for i = 1, 2, . . . , n - 1. Product bits [image] p2n-1, p2n-2, . . . , pn [image] are produced by an n-bit adder that adds the outputs from the last row of full adders:

Analysis

Data ripple through an array multiplier from upper left to lower right. It takes [image] (n) time for the lower-order product bits [image] pn-1, pn-2, . . . , p0 [image] to be produced, and it takes [image] (n) time for the inputs to the adder to be ready. If the adder is a ripple-carry adder, it takes another [image] (n) time for the higher-order product bits [image] p2n-1, p2n-2, . . . , pn [image] to emerge. If the adder is a carry-lookahead adder, only [image] (lg n) time is needed, but the total time remains [image] (n).

There are n2 AND gates and n2 - n full adders in the array multiplier. The adder for the high-order output bits contributes only another [image] (n) gates. Thus, the array multiplier has [image] (n2) size.

29.3.2 Wallace-tree multipliers

A Wallace tree is a circuit that reduces the problem of summing n n-bit numbers to the problem of summing two [image] (n)-bit numbers. It does this by using [image] n/3 [image] carry-save adders in parallel to convert the sum of n numbers to the sum of [image] 2n/3 [image] numbers. It then recursively constructs a Wallace tree on the [image] 2n/3 [image] resulting numbers. In this way, the set of numbers is progressively reduced until there are only two numbers left. By performing many carry-save additions in parallel, Wallace trees allow two n-bit numbers to be multiplied in [image] (1g n) time using a circuit with [image] (n2) size.

Figure 29.16 shows a Wallace tree2 that adds 8 partial products m(0), m(1), . . . , m(7). Partial product m(l) consists of n + i bits. Each line represents an entire number, not just a single bit; next to each line is the number of bits the line represents (see Exercise 29.3-3). The carry-lookahead adder at the bottom adds a (2n - 1)-bit number to a 2n-bit number to give the 2n-bit product.

2 As you can see from the figure, a Wallace tree is not truly a tree, but rather a directed acyclic graph. The name is historical.

Figure 29.16 A Wallace tree that adds n = 8 partial products m(0), m(1), . . . , m(7). Each line represents a number with the number of bits indicated. The left output of each carry-save adder represents the sum bits, and the right output represents the carry bits.

Analysis

The time required by an n-input Wallace tree depends on the depth of the carry-save adders. At each level of the tree, each group of 3 numbers is reduced to 2 numbers, with at most 2 numbers left over (as in the case of m(6) and m(7) at the top level). Thus, the maximum depth D(n) of a carry-save adder in an n-input Wallace tree is given by the recurrence

which has the solution D(n) = [image] (1g n) by case 2 of the master theorem (Theorem 4.1). Each carry-save adder takes [image] (1) time. All n partial products can be formed in [image] (1) time in parallel. (The lowest-order i - 1 bits of m(i), for i = 1, 2, . . . , n - 1, are hardwired to 0.) The carry-lookahead adder takes O(lg n) time. Thus, the entire multiplication of two n-bit numbers takes [image] (lg n) time.

A Wallace-tree multiplier for two n-bit numbers has [image] (n2) size, which we can see as follows. We first bound the circuit size of the carry-save adders. A lower bound of [image] (n2) is easy to obtain, since there are [image] 2n/3 [image] carry-save adders at depth 1, and each one consists of at least n full adders. To get the upper bound of O(n2), observe that since the final product has 2n bits, each carry-save adder in the Wallace tree contains at most 2n full adders. We need to show that there are O(n) carry-save adders altogether. Let C(n) be the total number of carry-save adders in a Wallace tree with n input numbers. We have the recurrence

which has the solution C(n) = [image] (n) by case 3 of the master theorem. We thus obtain an asymptotically tight bound of [image] (n2) size for the carry-save adders of a Wallace-tree multiplier. The circuitry to set up the n partial products has [image] (n2) size, and the carry-lookahead adder at the end has [image] (n) size. Thus, the size of the entire multiplier is [image] (n2).

Although the Wallace-tree-based multiplier is asymptotically faster than the array multiplier and has the same asymptotic size, its layout when it is implemented is not as regular as the array multiplier's, nor is it as "dense" (in the sense of having little wasted space between circuit elements). In practice, a compromise between the two designs is often used. The idea is to use two arrays in parallel, one adding up half of the partial products and one adding up the other half. The propagation delay is only half of that incurred by a single array adding up all n partial products. Two more carry-save additions reduce the 4 numbers output by the arrays to 2 numbers, and a carry-lookahead adder then adds the 2 numbers to yield the product. The total propagation delay is a little more than half that of a full array multiplier, plus an additional O(lg n) term.

Exercises

29.3-1

Prove that in an array multiplier, [image] = 0 for i = 1, 2, . . . , n - 1 and j = 0, 1,. . . , i, i + n, i + n + 1, . . . , 2n - 1.

29.3-2

Show that in the array multiplier of Figure 29.14, all but one of the full adders in the top row are unnecessary. You will need to do some rewiring.

29.3-3

Suppose that a carry-save adder takes inputs x, y, and z and produces outputs s and c, with nx, ny, nz, ns, and nc bits respectively. Suppose also, without loss of generality, that nx [image] ny [image] nz. Show that ns= nz and that

29.3-4

Show that multiplication can still be performed in O(lg n) time with O(n2) size even if we restrict gates to have O(1) fan-out.

29.3-5

Describe an efficient circuit to compute the quotient when a binary number x is divided by 3. (Hint: Note that in binary, .010101 . . . = .01 X 1.01 X 1.0001 X [image] [image] [image].

29.3-6

A cyclic shifter, or barrel shifter, is a circuit that has two inputs x = xn-1, xn-2,. . . ,x0 and s = sm-1, sm-2,. . . , s0 , where m = lg n . Its output y = yn-1, yn-2,. . . , y0 is specified by yi = xi + smodn, for i = 0,1, . . ., n - 1. That is, the shifter rotates the bits of x by the amount specified by s. Describe an efficient cyclic shifter. In terms of modular multiplication, what function does a cyclic shifter implement?

29.4 Clocked circuits

The elements of a combinational circuit are used only once during a computation. By introducing clocked memory elements into the circuit, we can reuse combinational elements. Because they can use hardware more than once, clocked circuits can often be much smaller than combinational circuits for the same function.

This section investigates clocked circuits for performing addition and multiplication. We begin with a (1)-size clocked circuit, called a bit-serial adder, that can add two n-bit numbers in (n) time. We then investigate linear-array multipliers. We present a linear-array multiplier with (n) size that can multiply two n-bit numbers in (n) time.

29.4.1 Bit-serial addition

We introduce the notion of a clocked circuit by returning to the problem of adding two n-bit numbers. Figure 29.17 shows how we can use a single full adder to produce the (n + 1)-bit sum s = sn, sn - 1, . . . , s0 of two n-bit numbers a = an - 1, an - 2, . . . , a0 and b = bn - 1, bn - 2, . . . , b0 . The external world presents the input bits one pair at a time: first a0 and b0, then a1 and b1, and so forth. Although we want the carry-out from one bit position to be the carry-in to the next bit position, we cannot just feed the full adder's c output directly into an input. There is a timing issue: the carry-in ci entering the full adder must correspond to the appropriate inputs ai and bi. Unless these input bits arrive at exactly the same moment as the fed-back carry, the output may be incorrect.

Figure 29.17 The operation of a bit-serial adder. During the ith clock period, for i = 0, 1, . . . , n, the full adder FA takes input bits ai and bi from the outside world and a carry bit ci from the register. The full adder then outputs sum bit si, which is provided externally, and carry bit ci+1, which is stored back in the register to be used during the next clock period. The register is initialized with c0 = 0. (a)-(e) The state of the circuit in each of the five clock periods during the addition of a = 1011 and b = 1001 to produce s = 10100 .

As Figure 29.17 shows, the solution is to use a clocked circuit, or sequential circuit, consisting of combinational circuitry and one or more registers (clocked memory elements). The combinational circuitry has inputs from the external world or from the output of registers. It provides outputs to the external world and to the input of registers. As in combinational circuits, we prohibit the combinational circuitry in a clocked circuit from containing cycles.

Each register in a clocked circuit is controlled by a periodic signal, or clock. Whenever the clock pulses, or ticks, the register loads in and stores the value at its input. The time between successive clock ticks is a clock period. In a globally clocked circuit, every register works off the same clock.

Let us examine the operation of a register in a little more detail. We treat each clock tick as a momentary pulse. At a given tick, a register reads the input value presented to it at that moment and stores it. This stored value then appears at the register's output, where it can be used to compute values that are moved into other registers at the next clock tick. In other words, the value at a register's input during one clock period appears on the register's output during the next clock period.

Now let us examine the circuit in Figure 29.17, which we call a bit-serial adder. In order for the full adder's outputs to be correct, we require that the clock period be at least as long as the propagation delay of the full adder, so that the combinational circuitry has an opportunity to settle between clock ticks. During clock period 0, shown in Figure 29.17(a), the external world applies input bits a0 and b0 to two of the full adder's inputs. We assume that the register is initialized to store a 0; the initial carry-in bit, which is the register output, is thus c0 = 0. Later in this clock period, sum bit s0 and carry-out c1 emerge from the full adder. The sum bit goes to the external world, where presumably it will be saved as part of the entire sum s. The wire from the carry-out of the full adder feeds into the register, so that c1 is read into the register upon the next clock tick. At the beginning of clock period 1, therefore, the register contains c1. During clock period 1, shown in Figure 29.17(b), the outside world applies a1 and b1 to the full adder, which, reading c1 from the register, produces outputs s1 and c2. The sum bit s1 goes out to the outside world, and c2 goes to the register. This cycle continues until clock period n, shown in Figure 29.17(e), in which the register contains cn. The external world then applies an = bn = 0, so that we get sn = cn.

Analysis

To determine the total time t taken by a globally clocked circuit, we need to know the number p of clock periods and the duration d of each clock period: t = pd. The clock period d must be long enough for all combinational circuitry to settle between ticks. Although for some inputs it may settle earlier, if the circuit is to work correctly for all inputs, d must be at least proportional to the depth of the combinational circuitry.

Let us see how long it takes to add two n-bit numbers bit-serially. Each clock period takes (1) time because the depth of the full adder is (1). Since n + 1 clock ticks are required to produce all the outputs, the total time to perform bit-serial addition is (n + 1) (1) = (n).

The size of the bit-serial adder (number of combinational elements plus number of registers) is (1).

Ripple-carry addition versus bit-serial addition

Observe that a ripple-carry adder is like a replicated bit-serial adder with the registers replaced by direct connections between combinational elements. That is, the ripple-carry adder corresponds to a spatial "unrolling" of the computation of the bit-serial adder. The ith full adder in the ripple-carry adder implements the ith clock period of the bit-serial adder.

In general, we can replace any clocked circuit by an equivalent combinational circuit having the same asymptotic time delay if we know in advance how many clock periods the clocked circuit runs for. There is, of course, a trade-off involved. The clocked circuit uses fewer circuit elements (a factor of (n) less for the bit-serial adder versus the ripple-carry adder), but the combinational circuit has the advantage of less control circuitry--we need no clock or synchronized external circuit to present input bits and store sum bits. Moreover, although the circuits have the same asymptotic time delay, the combinational circuit typically runs slightly faster in practice. The extra speed is possible because the combinational circuit doesn't have to wait for values to stabilize during each clock period. If all the inputs stabilize at once, values just ripple through the circuit at the maximum possible speed, without waiting for the clock.

Figure 29.18 Multiplying 19 by 29 with the Russian peasant's algorithm. The a- column entry in each row is half of the previous row's entry with fractions ignored, and the b-column entries double from row to row. We add the b-column entries in all rows with odd a-column entries, which are shaded. This sum is the desired product. (a) The numbers expressed in decimal. (b) The same numbers in binary.

29.4.2 Linear-array multipliers

The combinational multipliers of Section 29.3 need (n2) size to multiply two n-bit numbers. We now present two multipliers that are linear, rather than two-dimensional, arrays of circuit elements. Like the array multiplier, the faster of these two linear-array multipliers runs in (n) time.

The linear-array multipliers implement the Russian peasant's algorithm (so called because Westerners visiting Russia in the nineteenth century found the algorithm widely used there), illustrated in Figure 29.18(a). Given two input numbers a and b, we make two columns of numbers, headed by a and b. In each row, the a-column entry is half of the previous row's a-column entry, with fractions discarded. The b-column entry is twice the previous row's b-column entry. The last row is the one with an a-column entry of 1. We look at all the a-column entries that contain odd values and sum the corresponding b-column entries. This sum is the product a b.

Although the Russian peasant's algorithm may seem remarkable at first, Figure 29.18(b) shows that it is really just a binary-number-system implementation of the grade-school multiplication method, but with numbers expressed in decimal rather than binary. Rows in which the a-column entry is odd contribute to the product a term of b multiplied by the appropriate power of 2.

A slow linear-array implementation

Figure 29.19(a) shows one way to implement the Russian peasant's algorithm for two n-bit numbers. We use a clocked circuit consisting of a linear array of 2n cells. Each cell contains three registers. One register holds a bit from an a entry, one holds a bit from a b entry, and one holds a bit of the product p. We use superscripts to denote cell values before each step of the algorithm. For example, the value of bit ai before the jth step is , and we define .

The algorithm executes a sequence of n steps, numbered 0,1, . . . , n - 1, where each step takes one clock period. The algorithm maintains the invariant that before the jth step,

a(j)    b(j) + p(j) = a    b

(29.6)

(see Exercise 29.4-2). Initially, a(0) = a, b(0) = b, and p(0) = 0. The jth step consists of the following computations.

1. If a(j) is odd (that is, ), then add b into p: p(j+1) b(j) + p(j). (The addition is performed by a ripple-carry adder that runs the length of the array; carry bits ripple from right to left.) If a(j) is even, then carry p through to the next step: p(j+1) p(j).

2. Shift a right by one bit position:

3. Shift b left by one bit position:

After running n steps, we have shifted out all the bits of a; thus, a(n) = 0. Invariant (29.6) then implies that p(n) = a b.

We now analyze the algorithm. There are n steps, assuming that the control information is broadcast to each cell simultaneously. Each step takes (n) time in the worst case, because the depth of the ripple-carry adder is (n), and thus the duration of the clock period must be at least (n). Each shift takes only (1) time. Overall, therefore, the algorithm takes (n2) time. Because each cell has constant size, the entire linear array has (n) size.

A fast linear-array implementation

By using carry-save addition instead of ripple-carry addition, we can decrease the time for each step to (1), thus improving the overall time to (n). As Figure 29.19(b) shows, once again each cell contains a bit of an a entry and a bit of a b entry. Each cell also contains two more bits, from u and v, which are the outputs from carry-save addition. Using a carry-save representation to accumulate the product, we maintain the invariant that before the jth step,

a(j)    b(j) + u(j) + v(j) = a    b

(29.7)

(again, see Exercise 29.4-2). Each step shifts a and b in the same way as the slow implementation, so that we can combine equations (29.6) and (29.7) to yield u(j) + v(j) = p(j). Thus, the u and v bits contain the same information as the p bits in the slow implementation.

Figure 29.19 Two linear-array implementations of the Russian peasant's algorithm, showing the multiplication of a = 19 = 10011 by b = 29 = 11101 , with n = 5. The situation at the beginning of each step j is shown, with the remaining significant bits of a(j) and b(j) shaded. (a) A slow implementation that runs in (n2) time. Because a(5) = 0, we have p(5) = a b. There are n steps, and each step uses a ripple-carry addition. The clock period is therefore proportional to the length of the array, or (n), leading to (n2) time overall. (b) A fast implementation that runs in (n) time because each step uses carry-save addition rather than ripple-carry addition, thus taking only (1) time. There are a total of 2n - 1 = 9 steps; after the last step shown, repeated carry-save addition of u and v yields u(9) = a b.

The jth step of the fast implementation performs carry-save addition on u and v, where the operands depend on whether a is odd or even. If , we compute

and

Otherwise, , and we compute

and

After updating u and v, the jth step shifts a to the right and b to the left in the same manner as the slow implementation.

The fast implementation performs a total of 2n - 1 steps. For j n, we have a(j) = 0, and invariant (29.7) therefore implies that u(j) + v(j) = a b. Once a(j) = 0, all further steps serve only to carry-save add u and v. Exercise 29.4-3 asks you to show that v(2n-1) = 0, so that u(2n-1) = a b.

The total time in the worst case is (n), since each of the 2n - 1 steps takes (1) time. Because each cell still has constant size, the total size remains (n).

Exercises

29.4-1

Let a = 101101 , b = 011110 , and n = 6. Show how the Russian peasant's algorithm operates, in both decimal and binary, for inputs a and b.

29.4-2

Prove the invariants (29.6) and (29.7) for the linear-array multipliers.

29.4-3

Prove that in the fast linear-array multiplier, v(2n-1) = 0.

29.4-4

Describe how the array multiplier from Section 29.3.1 represents an "un-rolling" of the computation of the fast linear-array multiplier.

29.4-5

Consider a data stream x1, x2, . . . that arrives at a clocked circuit at the rate of 1 value per clock tick. For a fixed value n, the circuit must compute the value

for t = n,n + 1, . . .. That is, yt is the maximum of the most recent n values received by the circuit. Give an O(n)-size circuit that on each clock tick inputs the value xt and computes the output value yt in O(1) time. The circuit can use registers and combinational elements that compute the maximum of two inputs.

29.4-6

Redo Exercise 29.4-5 using only O(1g n) "maximum" elements.

Problems

29-1 Division circuits

We can construct a division circuit from subtraction and multiplication circuits with a technique called Newton iteration. We shall focus on the related problem of computing a reciprocal, since we can obtain a division circuit by making one additional multiplication.

The idea is to compute a sequence y0, ,y1, y2, . . . of approximations to the reciprocal of a number x by using the formula

Assume that x is given as an n-bit binary fraction in the range 1/2 x 1. Since the reciprocal can be an infinite repeating fraction, we shall concentrate on computing an n-bit approximation accurate up to its least significant bit.

a. Suppose that yi - 1/x for some constant > 0. Prove that yi+1 - 1/x 2.

b. Give an initial approximation y0 such that yk satisfies yk - 1/x 2-2k for all k 0. How large must k be for the approximation yk to be accurate up to its least significant bit?

c. Describe a combinational circuit that, given an n-bit input x, computes an n-bit approximation to 1/x in O(lg2 n) time. What is the size of your circuit? (Hint: With a little cleverness, you can beat the size bound of (n2 lgn).)

29-2 Boolean formulas for symmetric functions

A n-input function (x1, x2, . . . , xn) is symmetric if

f(x1, x2, . . . , xn) =   (x  (1), x  (2), . . . , x  (n) )

for any permutation of {1,2, . . . , n}. In this problem, we shall show that there is a boolean formula representing whose size is polynomial in n. (For our purposes, a boolean formula is a string comprised of the variables x1, x2, . . . , xn, parentheses, and the boolean operators V, , and .) Our approach will be to convert a logarithmic-depth boolean circuit to an equivalent polynomial-size boolean formula. We shall assume that all circuits are constructed from 2-input AND, 2-input OR, and NOT gates.

a. We start by considering a simple symmetric function. The generalized majority function on n boolean inputs is defined by

Describe an O(lg n)-depth combinational circuit for majorityn. (Hint: Build a tree of adders.)

b. Suppose that is an arbitrary boolean function of the n boolean variables x1, x2, . . . , xn. Suppose further that there is a circuit C of depth d that computes . Show how to construct from C a boolean formula for of length O(2d). Conclude that there is polynomial-size formula for majorityn.

c. Argue that any symmetric boolean function f(x1, x2, . . . , xn ) can be expressed as a function of

d. Argue that any symmetric function on n boolean inputs can be computed by an O(lg n)-depth combinational circuit.

e. Argue that any symmetric boolean function on n boolean variables can be represented by a boolean formula whose length is polynomial in n.

Chapter notes

Most books on computer arithmetic focus more on practical implementations of circuitry than on algorithmic theory. Savage [173] is one of the few that investigates algorithmic aspects of the subject. The more hardware-oriented books on computer arithmetic by Cavanagh [39] and Hwang [108] are especially readable. Good books on combinational and sequential logic design include Hill and Peterson [96] and, with a twist toward formal language theory, Kohavi [126].

Aiken and Hopper [7] trace the early history of arithmetic algorithms. Ripple-carry addition is as at least as old as the abacus, which has been around for over 5000 years. The first mechanical calculator employing ripple-carry addition was devised by B. Pascal in 1642. A calculating machine adapted to repeated addition for multiplication was conceived by S. Morland in 1666 and independently by G. W. Leibnitz in 1671. The Russian peasant's algorithm for multiplication is apparently much older than its use in Russia in the nineteenth century. According to Knuth [122], it was used by Egyptian mathematicians as long ago as 1800 B.C.

The kill, generate, and propagate statuses of a carry chain were exploited in a relay calculator built at Harvard during the mid-1940's [180]. One of the first implementations of carry-lookahead addition was described by Weinberger and Smith [199], but their lookahead method requires large gates. Ofman [152] proved that n-bit numbers could be added in O(lg n) time using carry-lookahead addition with constant-size gates.

The idea of using carry-save addition to speed up multiplication is due to Estrin, Gilchrist, and Pomerene [64]. Atrubin [13] describes a linear-array multiplier of infinite length that can be used to multiply binary numbers of arbitrary length. The multiplier produces the nth bit of the product immediately upon receiving the nth bits of the inputs. The Wallace-tree multiplier is attributed to Wallace [197], but the idea was also independently discovered by Ofman [152].

Division algorithms date back to I. Newton, who around 1665 invented what has become known as Newton iteration. Problem 29-1 uses Newton iteration to construct a division circuit with (lg2 n) depth. This method was improved by Beame, Cook, and Hoover [19], who showed that n-bit division can in fact be performed in (lg n) depth.

Go to Chapter 30 Back to Table of Contents


Intro to Algorithms: CHAPTER 30: ALGORITHMS FOR PARALLEL COMPUTERS

 Return to Table of Contents

CHAPTER 30: ALGORITHMS FOR PARALLEL COMPUTERS

As parallel-processing computers have proliferated, interest has increased in parallel algorithms: algorithms that perform more than one operation at a time. The study of parallel algorithms has now developed into a research area in its own right. Indeed, parallel algorithms have been developed for many of the problems we have solved in this text using ordinary serial algorithms. In this chapter, we shall describe a few simple parallel algorithms that illustrate fundamental issues and techniques.

In order to study parallel algorithms, we must choose an appropriate model for parallel computing. The random-access machine, or RAM, which we have used throughout most of this book, is, of course, serial rather than parallel. The parallel models we have studied--sorting networks (Chapter 28) and circuits (Chapter 29)--are too restrictive for investigating, for example, algorithms on data structures.

The parallel algorithms in this chapter are presented in terms of one popular theoretical model: the parallel random-access machine, or PRAM (pronounced "PEE-ram"). Many parallel algorithms for arrays, lists, trees, and graphs can be easily described in the PRAM model. Although the PRAM ignores many important aspects of real parallel machines, the essential attributes of parallel algorithms tend to transcend the models for which they are designed. If one PRAM algorithm outperforms another PRAM algorithm, the relative performance is not likely to change substantially when both algorithms are adapted to run on a real parallel computer.

The PRAM model

Figure 30.1 shows the basic architecture of the parallel random-access machine (PRAM). There are p ordinary (serial) processors P0, P1, . . . , Pp-1 that have as storage a shared, global memory. All processors can read from or write to the global memory "in parallel" (at the same time). The processors can also perform various arithmetic and logical operations in parallel.

The key assumption regarding algorithmic performance in the PRAM model is that running time can be measured as the number of parallel memory accesses an algorithm performs. This assumption is a straight- forward generalization of the ordinary RAM model, in which the number of memory accesses is asymptotically as good as any other measure of running time. This simple assumption will serve us well in our survey of parallel algorithms, even though real parallel computers cannot perform parallel accesses to global memory in unit time: the time for a memory access grows with the number of processors in the parallel computer.

Figure 30.1 The basic architecture of the PRAM. There are p processors P0, P1, . . ., Pp - 1 connected to a shared memory. Each processor can access an arbitrary word of shared memory in unit time.

Nevertheless, for parallel algorithms that access data in an arbitrary fashion, the assumption of unit-time memory operations can be justified. Real parallel machines typically have a communication network that can support the abstraction of a global memory. Accessing data through the network is a relatively slow operation in comparison with arithmetic and other operations. Thus, counting the number of parallel memory accesses executed by two parallel algorithms does, in fact, yield a fairly accurate estimate of their relative performances. The principal way in which real machines violate the unit-time abstraction of the PRAM is that some memory-access patterns are faster than others. As a first approximation, however, the unit-time assumption in the PRAM model is quite reasonable.

The running time of a parallel algorithm depends on the number of processors executing the algorithm as well as the size of the problem input. Generally, therefore, we must discuss both time and processor count when analyzing PRAM algorithms; this contrasts with serial algorithms, in whose analysis we have focused mainly on time. Typically, there is a trade-off between the number of processors used by an algorithm and its running time. Section 30.3 discusses these trade-offs.

Concurrent versus exclusive memory accesses

A concurrent-read algorithm is a PRAM algorithm during whose execution multiple processors can read from the same location of shared memory at the same time. An exclusive-read algorithm is a PRAM algorithm in which no two processors ever read the same memory location at the same time. We make a similar distinction with respect to whether or not multiple processors can write into the same memory location at the same time, dividing PRAM algorithms into concurrent-write and exclusive-write algorithms. Commonly used abbreviations for the types of algorithms we encounter are

EREW: exclusive read and exclusive write,

CREW: concurrent read and exclusive write,

ERCW: exclusive read and concurrent write, and

CRCW: concurrent read and concurrent write.

(These abbreviations are usually pronounced not as words but rather as strings of letters.)

Of these types of algorithms, the extremes--EREW and CRCW--are the most popular. A PRAM that supports only EREW algorithms is called an EREW PRAM, and one that supports CRCW algorithms is called a CRCW PRAM. A CRCW PRAM can, of course, execute EREW algorithms, but an EREW PRAM cannot directly support the concurrent memory accesses required in CRCW algorithms. The underlying hardware of an EREW PRAM is relatively simple, and therefore fast, because it needn't handle conflicting memory reads and writes. A CRCW PRAM requires more hardware support if the unit-time assumption is to provide a reasonably accurate measure of algorithmic performance, but it provides a programming model that is arguably more straightforward than that of an EREW PRAM.

Of the remaining two algorithmic types--CREW and ERCW--more attention has been paid in the literature to the CREW. From a practical point of view, however, supporting concurrency for writes is no harder than supporting concurrency for reads. In this chapter, we shall generally treat an algorithm as being CRCW if it contains either concurrent reads or concurrent writes, without making further distinctions. We discuss the finer points of this distinction in Section 30.2.

When multiple processors write to the same location in a CRCW algorithm, the effect of the parallel write is not well defined without additional elaboration. In this chapter, we shall use the common-CRCW model: when several processors write into the same memory location, they must all write a common (the same) value. There are several alternative types of PRAM's in the literature that handle this problem with a different assumption. Other choices include

arbitrary: an arbitrary value from among those written is actually stored,

priority: the value written by the lowest-indexed processor is stored, and

combining: the value stored is some specified combination of the values written.

In the last case, the specified combination is typically some associative and commutative function such as addition (store the sum of all the values written) or maximum (store only the maximum value written).

Synchronization and control

PRAM algorithms must be highly synchronized to work correctly. How is this synchronization achieved? Also, the processors in PRAM algorithms must often detect termination of loop conditions that depend on the state of all processors. How is this control function implemented?

We won't discuss these issues extensively. Many real parallel computers employ a control network connecting the processors that helps with synchronization and termination conditions. Typically, the control network can implement these functions as fast as a routing network can implement global memory references.

For our purposes, it suffices to assume that the processors are inherently tightly synchronized. All processors execute the same statements at the same time. No processor races ahead while others are further back in the code. As we go through our first parallel algorithm, we shall point out where we assume that processors are synchronized.

For detecting the termination of a parallel loop that depends on the state of all processors, we shall assume that a parallel termination condition can be tested through the control network in O(1) time. Some EREW PRAM models in the literature do not make this assumption, and the (logarithmic) time for testing the loop condition must be included in the overall running time (see Exercise 30.1-8). As we shall see in Section 30.2, CRCW PRAM's do not need a control network to test termination: they can detect termination of a parallel loop in O(1) time through the use of concurrent writes.

Chapter outline

Section 30.1 introduces the technique of pointer jumping, which provides a fast way to manipulate lists in parallel. We show how pointer jumping can be used to perform prefix computations on lists and how fast algorithms on lists can be adapted for use on trees. Section 30.2 discusses the relative power of CRCW and EREW algorithms and shows that concurrent memory accessing provides increased power.

Section 30.3 presents Brent's theorem, which shows how combinational circuits can be efficiently simulated by PRAM's. The section also discusses the important issue of work efficiency and gives conditions under which a p-processor PRAM algorithm can be efficiently translated into a p'-processor PRAM algorithm for any p' < p. Section 30.4 reprises the problem of performing a prefix computation on a linked list and shows how a randomized algorithm can perform the computation in a work-efficient fashion. Finally, Section 30.5 shows how symmetry can be broken in parallel in much less than logarithmic time using a deterministic algorithm.

The parallel algorithms in this chapter have been drawn principally from the area of graph theory. They represent only a scant selection of the present array of parallel algorithms. The techniques introduced in this chapter, however, are quite representative of the techniques used for parallel algorithms in other areas of computer science.

30.1 Pointer jumping

Among the more interesting PRAM algorithms are those that involve pointers. In this section, we investigate a powerful technique called pointer jumping, which yields fast algorithms for operating on lists. Specifically, we introduce an O(lg n)-time algorithm that computes the distance to the end of the list for each object in an n-object list. We then modify this algorithm to perform a "parallel prefix" computation on an n-object list in O(lg n) time. Finally, we investigate a technique that allows many problems on trees to be converted to list problems, which can then be solved by pointer jumping. All of the algorithms in this section are EREW algorithms: no concurrent accesses to global memory are required.

30.1.1 List ranking

Our first parallel algorithm operates on lists. We can store a list in a PRAM much as we store lists in an ordinary RAM. To operate on list objects in parallel, however, it is convenient to assign a "responsible" processor to each object. We shall assume that there are as many processors as list objects, and that the ith processor is responsible for the ith object. Figure 30.2(a), for example, shows a linked list consisting of the sequence of objects [image] 3,4,6,1,0,5 [image]. Since there is one processor per list object, every object in the list can be operated on by its responsible processor in O(1) time.

Suppose that we are given a singly linked list L with n objects and wish to compute, for each object in L, its distance from the end of the list. More formally, if next is the pointer field, we wish to compute a value d[i] for each object i in the list such that

We call the problem of computing the d values the list-ranking problem.

One solution to the list-ranking problem is simply to propagate distances back from the end of the list. This method takes [image] (n) time, since the kth object from the end must wait for the k- 1 objects following it to determine their distances from the end before it can determine its own. This solution is essentially a serial algorithm.

Figure 30.2 Finding the distance from each object in an n-object list to the end of the list in O(lg n) time using pointer jumping. (a) A linked list represented in a PRAM with d values initialized. At the end of the algorithm, each d value holds the distance of its object from the end of the list. Each object's responsible processor appears above the object. (b)-(d) The pointers and d values after each iteration of the while loop in the algorithm LIST-RANK.

An efficient parallel solution, requiring only O(lg n) time, is given by the following parallel pseudocode.

LIST-RANK(L)

1  for each processor i, in parallel

2       do if next[i] = NIL

3             then d[i]    0

4             else d[i]    1

5  while there exists an object i such that next[i]    NIL

6      do for each processor i, in parallel

7             do if next[i]    NIL

8                   then d[i]    d[i] + d[next[i]]

9                        next[i]    next[next[i]]

Figure 30.2 shows how the algorithm computes the distances. Each part of the figure shows the state of the list before an iteration of the while loop of lines 5-9. Part (a) shows the list just after initialization. In the first iteration, the first 5 list objects have non-nil pointers, so that lines 8-9 are executed by their responsible processors. The result appears in part (b) of the figure. In the second iteration, only the first 4 objects have non-NIL pointers; the result of this iteration is shown in part (c). In the third iteration, only the first 2 objects are operated on, and the final result, in which all objects have NIL pointers, appears in part (d).

The idea implemented by line 9, in which we set next[i] [image] next[next[i]] for all non-nil pointers next[i], is called pointer jumping. Note that th e pointer fields are changed by pointer jumping, thus destroying the structure of the list. If the list structure must be preserved, then we make copies of the next pointers and use the copies to compute the distances.

Correctness

LIST-RANK maintains the invariant that at the beginning of each iteration of the while loop of lines 5-9, for each object i, if we add the d values in the sublist headed by i, we obtain the correct distance from i to the end of the original list L. In Figure 30.2(b), for example, the sublist headed by object 3 is the sequence [image] 3,6,0 [image] whose d values 2, 2, and 1 sum to 5, its distance from the end of the original list. The reason the invariant is maintained is that when each object "splices out" its successor in the list, it adds its successor's d value to its own.

Observe that for this pointer-jumping algorithm to work correctly, the parallel memory accesses must be synchronized. Each execution of line 9 can update several next pointers. We rely on all the memory reads on the right-hand side of the assignment (reading next[next[i]]) occurring before any of the memory writes (writing next[i]) on the left-hand side.

Now let us see why LIST-RANK is an EREW algorithm. Because each processor is responsible for at most one object, every read and write in lines 2-7 is exclusive, as are the writes in lines 8-9. Observe that pointer jumping maintains the invariant that for any two distinct objects i and j, either next[i] [image] next[j] or next[i] = next[j] = NIL. This invariant is certainly true for the initial list, and it is maintained by line 9. Because all non-NIL next values are distinct, all reads in line 9 are exclusive.

We do need to assume that some synchronization is performed in line 8 if all reads are to be exclusive. In particular, we require that all processors i read d[i] and then d[next[i]]. With this synchronization, if an object i has next[i] [image] NIL and there is another object j pointing to i (that is, next[j] = i), then the first read fetches d[i] for processor i and the second read fetches d[i] for processor j. Thus, List-Rank is an EREW algorithm.

From here on, we ignore such details of synchronization and assume that the PRAM and its pseudocode programming environment act in a consistent, synchronized manner, with all processors executing reads and writes at the same time.

Analysis

We now show that if there are n objects in list L, then List-Rank takes O(lg n) time. Since the initialization takes O(1) time and each iteration of the while loop takes O(1) time, it suffices to show that there are exactly [image] lg n [image] iterations. The key observation is that each step of pointer jumping transforms each list into two interleaved lists: one consisting of the objects in even positions and the other consisting of objects in odd positions. Thus, each pointer-jumping step doubles the number of lists and halves their lengths. By the end of [image] lg n [image] iterations, therefore, all lists contain only one object.

We are assuming that the termination test in line 5 takes O(1) time, presumably due to a control network in the EREW PRAM. Exercise 30.1-8 asks you to describe an O(1g n)-time EREW implementation of List-Rank that performs the termination test explicitly in the pseudocode.

Besides parallel running time, there is another interesting performance measure for parallel algorithms. We define the work performed by a parallel algorithm as the product of its running time and the number of processors it requires. Intuitively, the work is the amount of computing that a serial RAM performs when it simulates the parallel algorithm.

The procedure List-Rank performs [image] (n lg n) work, since it requires n processors and runs in [image] (lg n) time. The straightforward serial algorithm for the list-ranking problem runs in [image] (n) time, indicating that more work is performed by List-Rank than is absolutely necessary, but only by a logarithmic factor.

We define a PRAM algorithm A to be work-efficient with respect to another (serial or parallel) algorithm B for the same problem if the work performed by A is within a constant factor of the work performed by B. We also say more simply that a PRAM algorithm A is work-efficient if it is work-efficient with respect to the best possible algorithm on a serial RAM. Since the best possible serial algorithm for list ranking runs in [image] (n) time on a serial RAM, LIST-RANK is not work-efficient. We shall present a work-efficient parallel algorithm for list ranking in Section 30.4.

30.1.2 Parallel prefix on a list

The technique of pointer jumping extends well beyond the application of list ranking. Section 29.2.2 shows how, in the context of arithmetic circuits, a "prefix" computation can be used to perform binary addition quickly. We now investigate how pointer jumping can be used to perform prefix computations. Our EREW algorithm for the prefix problem runs in O(lg n) time on n-object lists.

A prefix computation is defined in terms of a binary, associative operator [image]. The computation takes as input a sequence [image] x1, x2, . . . , xn [image] and produces as output a sequence [image] y1, y, . . . , yn [image] such that y1 = x1 and

yk  =  yk-1   xk

=  x1    x2    . . .    xk

for k = 2,3, . . ., n. In other words, each yk is obtained by "multiplying" together the first k elements of the sequence of xk--hence, the term "prefix." (The definition in Chapter 29 indexes the sequences from 0, whereas this definitionindexes from 1--an inessential difference.)

As an example of a prefix computation, suppose that every element of an n-object list contains the value 1, and let [image] be ordinary addition. Since the kth element of the list contains the value xk = 1 for k = 1,2, . . ., n, a prefix computation produces yk = k, the index of the kth element. Thus, another way to perform list ranking is to reverse the list (which can be done in O(1) time), perform this prefix computation, and subtract 1 from each value computed.

We now show how an EREW algorithm can compute parallel prefixes in O(lg n) time on n-object lists. For convenience, we define the notation

[i, j] = xi    xi+1             xj

for integers i and j in the range 1 [image] i [image] j [image] n. Then, [k,k] = xk for

k = 1, 2,..., n, and

[i,k] = [i, j]    [j+1, k]

for 0 [image] i [image] j < k [image] n. In terms of this notation, the goal of a prefix computation is to compute yk = [1, k] for k = 1, 2, . . . , n.

When we perform a prefix computation on a list, we wish the order of the input sequence [image] x1, x2, . . . , xn [image] to be determined by how the objects are linked together in the list, and not by the index of the object in the array of memory that stores objects. (Exercise 30.1-2 asks for a prefix algorithm for arrays.) The following EREW algorithm starts with a value x[i] in each object i in a list L. If object i is the kth object from the beginning of the list, then x[i] = xk is the kth element of the input sequence. Thus, the parallel prefix computation produces y[i] = yk = [1, k].

LIST-PREFIX(L)

1  for each processor i, in parallel

2      do y[i]    x[i]

3  while there exists an object i such that next[i]    NIL

4      do for each processor i, in parallel

5             do if next[i]    NIL

6                   then y[next[i]]    y[i]    y[next[i]]

7                        next[i]    next[next[i]]

The pseudocode and Figure 30.3 indicate the similarity between this algorithm and List-Rank. The only differences are the initialization and the updating of d or y values. In List-Rank, processor i updates d[i]--its own d value--whereas in LIST-PREFIX, processor i updates y[next[i]]--another processor's y value. Note that LIST-PREFIX is EREW for the same reason as List-Rank: pointer jumping maintains the invariant that for distinct objects i and j, either next[i] [image] next[j] or next[i] = next[j] = NIL.

Figure 30.3 shows the state of the list before each iteration of the while loop. The procedure maintains the invariant that at the end of the tth execution of the while loop, the kth processor stores [max(1, k - 2t + 1), k], for k = 1, 2, . . . , n. In the first iteration, the kth list object points initially to the (k + 1)st object, except that the last object has a NIL pointer. Line 6 causes the kth object, for k = 1, 2, . . . , n - 1, to fetch the value [k + 1, k + 1] from its successor. It then performs the operation [k, k] [image] [k + 1, k + 1], yielding [k, k + 1], which it stores back into its successor. The next pointers are then jumped as in LIST-RANK, and the result of the first iteration appears in Figure 30.3(b). We can view the second iteration similarly. For k = 1, 2, . . . , n - 2, the kth object fetches the value [k + 1, k + 2] from its successor (as defined by the new value in its field next), and then it stores [k - 1, k] [image] [k + 1, k + 2] = [k - 1, k + 2] into its successor. The result is shown in Figure 30.3(c). In the third and final iteration, only the first two list objects have non-NIL pointers, and they fetch values from their successors in their respective lists. The final result appears in Figure 30.3(d). The key observation that makes LIST-PREFIX work is that at each step, if we perform a prefix computation on each of the several existing lists, each object obtains its correct value.

Figure 30.3 The parallel prefix algorithm LIST-PREFIX on a linked list. (a) The initial y value of the kth object in the list is [k, k]. The next pointer of the kth object points to the (k + 1)st object, or NIL for the last object. (b)-(d) The y and next values before each test in line 3. The final answer is in part (d), in which the y value for the kth object is [1, k] for all k.

Since the two algorithms use the same pointer-jumping mechanism, LIST-PREFIX has the same analysis as LIST-RANK: the running time is O(lg n) on an EREW PRAM, and the total work performed is [image] (n 1g n).

30.1.3 The Euler-tour technique

In this section, we shall introduce the Euler-tour technique and show how it can be applied to the problem of computing the depth of each node in an n-node binary tree. A key step in this O(lg n)-time EREW algorithm is a parallel prefix computation.

To store binary trees in a PRAM, we use a simple binary-tree representation of the sort presented in Section 11.4. Each node i has fields parent[i], left[i], and right[i], which point to node i's parent, left child, and right child, respectively. Let us assume that each node is identified by a non-negative integer. For reasons that will soon become apparent, we associate not one but three processors with each node; we call these the node's A, B, and C processors. We should be able to map between a node and its three processors easily; for example, node i might be associated with processors 3i, 3i + 1, and 3i + 2.

Computing the depth of each node in an n-node tree takes O(n) time on a serial RAM. A simple parallel algorithm to compute depths propagates a "wave" downward from the root of the tree. The wave reaches all nodes at the same depth simultaneously, and thus by incrementing a counter carried along with the wave, we can compute the depth of each node. This parallel algorithm works well on a complete binary tree, since it runs in time proportional to the tree's height. The height of the tree could be as large as n - 1, however, in which case the algorithm would run in [image] (n) time--no better than the serial algorithm. Using the Euler-tour technique, however, we can compute node depths in O(lg n) time on an EREW PRAM, whatever the height of the tree.

An Euler tour of a graph is a cycle that traverses each edge exactly once, although it may visit a vertex more than once. By Problem 23-3, a connected, directed graph has an Euler tour if and only if for all vertices v, the in-degree of v equals the out-degree of v. Since each undirected edge (u, v) in an undirected graph maps to two directed edges (u, v) and (v, u) in the directed version, the directed version of any connected, undirected graph--and therefore of any undirected tree--has an Euler tour.

To compute the depths of nodes in a binary tree T, we first form an Euler tour of the directed version of T (viewed as an undirected graph). The tour corresponds to a walk of the tree and is represented in Figure 30.4(a) by a linked list running through the nodes of the tree. Its structure is as follows:

A node's A processor points to the A processor of its left child, if it exists, and otherwise to its own B processor.

A node's B processor points to the A processor of its right child, if it exists, and otherwise to its own C processor.

A node's C processor points to the B processor of its parent if it is a left child and to the C processor of its parent if it is a right child. The root's C processor points to NIL.

Thus, the head of the linked list formed by the Euler tour is the root's A processor, and the tail is the root's C processor. Given the pointers composing the original tree, an Euler tour can be constructed in O(1) time.

Once we have the linked list representing the Euler tour of T, we place a 1 in each A processor, a 0 in each B processor, and a - 1 in each C processor, as shown in Figure 30.4(a). We then perform a parallel prefix computation using ordinary addition as the associative operation, as we did in Section 30.1.2. Figure 30.4(b) shows the result of the parallel prefix computation.

Figure 30.4 Using the Euler-tour technique to compute the depth of each node in a binary tree. (a) The Euler tour is a list corresponding to a walk of the tree. Each processor contains a number used by a parallel prefix computation to compute node depths. (b) The result of the parallel prefix computation on the linked list from (a). The C processor of each node (blackened) contains the node's depth. (You can verify the result of this prefix computation by computing it serially.)

We claim that after performing the parallel prefix computation, the depth of each node resides in the node's C processor. Why? The numbers are placed into the A, B, and C processors in such a way that the net effect of visiting a subtree is to add 0 to the running sum. The A processor of each node i contributes 1 to the running sum in i's left subtree, reflecting the depth of i's left child being one greater than the depth of i. The B processor contributes 0 because the depth of node i's left child equals the depth of node i's right child. The C processor contributes - 1, so that from the perspective of node i's parent, the entire visit to the subtree rooted at node i has no effect on the running sum.

The list representing the Euler tour can be computed in O(1) time. It has 3n objects, and thus the the parallel prefix computation takes only O(lg n) time. Thus, the total amount of time to compute all node depths is O(lg n). Because no concurrent memory accesses are needed, the algorithm is an EREW algorithm.

Exercises

30.1-1

Give an O(lg n)-time EREW algorithm that determines for each object in an n-object list whether it is the middle ( [image] n/2 [image] th) object.

30.1-2

Give an O(lg n)-time EREW algorithm to perform the prefix computation on an array x[1. . n]. Do not use pointers, but perform index computations directly.

30.1-3

Suppose that each object in an n-object list L is colored either red or blue. Give an efficient EREW algorithm to form two lists from the objects in L: one consisting of the blue objects and one consisting of the red objects.

30.1-4

An EREW PRAM has n objects distributed among several disjoint circular lists. Give an efficient algorithm that determines an arbitrary representative object for each list and acquaints each object in the list with the identity of the representative. Assume that each processor knows its own unique index.

30.1-5

Give an O(lg n)-time EREW algorithm to compute the size of the subtree rooted at each node of an n-node binary tree. (Hint: Take the difference of two values in a running sum along an Euler tour.)

30.1-6

Give an efficient EREW algorithm to compute preorder, inorder, and post-order numberings for an arbitrary binary tree.

30.1-7

Extend the Euler-tour technique from binary trees to ordered trees with arbitrary node degrees. Specifically, describe a representation for ordered trees that allows the Euler-tour technique to be applied. Give an EREW algorithm to compute the node depths of an n-node ordered tree in O(lg n) time.

30.1-8

Describe an O(lg n)-time EREW implementation of LIST-RANK that performs the loop-termination test explicitly. (Hint: Interleave the test with the loop body.)

30.2 CRCW algorithms versus EREW algorithms

The debate about whether or not concurrent memory accesses should be provided by the hardware of a parallel computer is a messy one. Some argue that hardware mechanisms to support CRCW algorithms are too expensive and used too infrequenly to be justified. Others complain that EREW PRAM's provide too restrictive a programming model. The answer to this debate probably lies somewhere in the middle, and various compromise models have been proposed. Nevertheless, it is instructive to examine what algorithmic advantage is provided by concurrent accesses to memory.

In this section, we shall show that there are problems on which a CRCW algorithm outperforms the best possible EREW algorithm. For the problem of finding the identities of the roots of trees in a forest, concurrent reads allow for a faster algorithm. For the problem of finding the maximum element in an array, concurrent writes permit a faster algorithm.

A problem in which concurrent reads help

Suppose we are given a forest of binary trees in which each node i has a pointer parent[i] to its parent, and we wish each node to find the identity of the root of its tree. Associating processor i with each node i in a forest F, the following pointer-jumping algorithm stores the identity of the root of each node i's tree in root[i].

FIND-ROOTS(F)

1.  for each processor i, in parallel

2.      do if parent[i] = NIL

3.            then root[i]   i

4.  while there exists a node i such that parent[i]    NIL

5.      do for each processor i, in parallel

6.             do if parent[i]    NIL

7.                   then root[i]    root[parent[i]]

8.                        parent[i]    parent[parent[i]]

Figure 30.5 illustrates the operation of this algorithm. After the initialization performed by lines 1-3, shown in Figure 30.5(a), the only nodes that know the identities of their roots are the roots themselves. The while loop of lines 4-8 performs the pointer jumping and fills in the root fields. Figures 30.5(b)-(d) show the state of the forest after the first, second, and third iterations of the loop. As you can see, the algorithm maintains the invariant that if parent[i] = NIL, then root[i] has been assigned the identity of the node's root.

We claim that FIND-ROOTS is a CREW algorithm that runs in O(lg d) time, where d is the depth of the maximum-depth tree in the forest. The only writes occur on lines 3, 7, and 8, and these are all exclusive because in each one, processor i writes into only node i. The reads in lines 7-8 are concurrent, however, because several nodes may have pointers to the same node. In Figure 30.5(b), for example, we see that during the second iteration of the while loop, root[4] and parent[4] are read by processors 18, 2, and 7.

The running time of FIND-ROOTS O(lg d) for essentially the same reason as for LIST-RANK: the length of each path is halved in each iteration. Figure 30.5 shows this characteristic plainly.

How fast can n nodes in a forest determine the roots of their binary trees using only exclusive reads? A simple argument shows that [image] (lg n) time is required. The key observation is that when reads are exclusive, each step of the PRAM allows a given piece of information to be copied to at most one other memory location; thus the number of locations that can contain a given piece of information at most doubles with each step. Looking at a single tree, we have initially that at most 1 memory location stores the identity of the root. After 1 step, at most 2 locations can contain the identity of the root; after k steps, at most 2k-1 locations can contain the identity of the root. If the size of the tree is [image] (n), we need [image] (n) locations to contain the root's identity when the algorithm terminates; thus, [image] (lg n) steps are required in all.

Whenever the depth d of the maximum-depth tree in the forest is 2o(lg n), the CREW algorithm FIND-ROOTS asymptotically outperforms any EREW algorithm. Specifically, for any n-node forest whose maximum-depth tree is a balanced binary tree with [image] (n) nodes, d = O(lg n), in which case FIND-ROOTS runs in O(lg lg n) time. Any EREW algorithm for this problem must run in [image] (lg n) time, which is asymptotically slower. Thus, concurrent reads help for this problem. Exercise 30.2-1 gives a simpler scenario in which concurrent reads help.

Figure 30.5 Finding the roots in a forest of binary trees on a CREW PRAM. Node numbers are next to the nodes, and stored root fields appear within nodes. The links represent parent pointers. (a)-(d) The state of the trees in the forest each time line 4 of FIND-ROOTS is executed. Note that path lengths are halved in each iteration.

A problem in which concurrent writes help

To demonstrate that concurrent writes offer a performance advantage over exclusive writes, we examine the problem of finding the the maximum element in an array of real numbers. We shall see that any EREW algorithm for this problem takes [image] (lg n) time and that no CREW algorithm does any better. The problem can be solved in O(1) time using a common-CRCW algorithm, in which when several processors write to the same location, they all write the same value.

The CRCW algorithm that finds the maximum of n array elements assumes that the input array is A[0 . . n-1]. The algorithm uses n2 processors, with each processor comparing A[i] and A[j] for some i and j in the range 0 [image] i, j [image] n - 1. In effect, the algorithm performs a matrix of comparisons, and so we can view each of the n2 processors as having not only a one-dimensional index in the PRAM, but also a two-dimensional index (i, j).

FAST-MAX(A)

1 n    length[A]

2 for i    0 to n - 1, in parallel

3       do m[i]    TRUE

4 for i    0 to n -1 and j    0 to n - 1, in parallel

5       do if A[i] < A[j]

6             then m[i]    FALSE

7 for i    0 to n - 1, in parallel

8       do if m[i] = TRUE

9             then max    A[i]

10 return max

Line 1 simply determines the length of the array A; it only needs to be executed on one processor, say processor 0. We use an array m[0 . . n -1], where processor i is responsible for m[i]. We want m[i] = TRUE if and only if A[i] is the maximum value in array A. We start (lines 2-3) by believing that each array element is possibly the maximum, and we rely on comparisons in line 5 to determine which array elements are not the maximum.

Figure 30.6 illustrates the remainder of the algorithm. In the loop of lines 4-6, we check each ordered pair of elements of array A. For each pair A[i] and A[j], line 5 checks whether A[i] < A[j]. If this comparison is TRUE, we know that A[i] cannot be the maximum, and line 6 sets m[i] [image] FALSE to record this fact. Several (i, j) pairs may be writing to m[i] simultaneously, but th ey all write the same value: FALSE.

Figure 30.6 Finding the maximum of n values in O(1) time by the CRCW algorithm FAST-MAX for each ordered pair of the elements in the input array A = 5, 6, 9, 2, 9 , the result of the comparison A[i] < A[j] is shown in the matrix, abbreviated T for TRUE and F for FALSE. For any row that contains a TRUE value, the corresponding element of m, shown at the right, is set to FALSE. Elements of m that contain TRUE correspond to the maximum-valued elements of A. In this case, the value 9 is written into the variable max.

After line 6 is executed, therefore, m[i] = TRUE for exactly the indices i such that A[i] achieves the maximum. Lines 7-9 then put the maximum value into the variable max, which is returned in line 10. Several processors may write into the variable max, but if they do, they all write the same value, as is consistent with the common-CRCW PRAM model.

Since all three "loops" in the algorithm are executed in parallel, FAST-MAX runs in O(1) time. Of course, it is not work-efficient, since it requires n2 processors, and the problem of finding the maximum number in an array can be solved by a [image] (n)-time serial algorithm. We can come closer to a work-efficient algorithm, however, as Exercise 30.2-6 asks you to show.

In a sense, the key to FAST-MAX is that a CRCW PRAM is capable of performing a boolean AND of n variables in O(1) time with n processors. (Since this capability holds in the common-CRCW model, it holds in the more powerful CRCW PRAM models as well.) The code actually performs several AND's at once, computing for i = 0, 1, . . . , n - 1,

which can be derived from DeMorgan's laws (5.2). This powerful AND capability can be used in other ways. For example, the capability of a CRCW PRAM to perform an AND in O(1) time obviates the need for a separate control network to test whether all processors are finished iterating a loop, such as we have assumed for EREW algorithms. The decision to finish the loop is simply the AND of all processors' desires to finish the loop.

The EREW model does not have this powerful AND facility. Any EREW algorithm that computes the maximum of n elements takes [image] (lg n) time. The proof is conceptually similar to the lower-bound argument for finding the root of a binary tree. In that proof, we looked at how many nodes can "know" the identity of the root and showed that it at most doubles for each step. For the problem of computing the maximum of n elements, we consider how many elements "think" that they might possibly be the maximum. Intuitively, with each step of an EREW PRAM, this number can at most halve, which leads to the [image] (lg n) lower bound.

Remarkably, the [image] (lg n) lower bound for computing the maximum holds even if we permit concurrent reading; that is, it holds for CREW algorithms. Cook, Dwork, and Reischuk [50] show, in fact, that any CREW algorithm for finding the maximum of n elements must run in [image] (lg n) time, even with an unlimited number of processors and unlimited memory. Their lower bound also holds for the problem of computing the AND of n boolean values.

Simulating a CRCW algorithm with an EREW algorithm

We now know that CRCW algorithms can solve some problems more quickly than can EREW algorithms. Moreover, any EREW algorithm can be executed on a CRCW PRAM. Thus, the CRCW model is strictly more powerful than the EREW model. But how much more powerful is it? In Section 30.3, we shall show that a p-processor EREW PRAM can sort p numbers in O(lg p) time. We now use this result to provide a theoretical upper bound on the power of a CRCW PRAM over an EREW PRAM.

Theorem 30.1

A p-processor CRCW algorithm can be no more than O(lg p) times faster than the best p-processor EREW algorithm for the same problem.

Proof The proof is a simulation argument. We simulate each step of the CRCW algorithm with an O(lg p)-time EREW computation. Because the processing power of both machines is the same, we need only focus on memory accessing. We only present the proof for simulating concurrent writes here. Implementation of concurrent reading is left as Exercise 30.2-8.

The p processors in the EREW PRAM simulate a concurrent write of the CRCW algorithm using an auxiliary array A of length p. Figure 30.7 illustrates the idea. When CRCW processor Pi, for i = 0, 1, . . . , p - 1, desires to write a datum xi to a location li, each corresponding EREW processor Pi instead writes the ordered pair (li, xi) to location A[i]. These writes are exclusive, since each processor writes to a distinct memory location. Then, the array A is sorted by the first coordinate of the ordered pairs in O(lg p) time, which causes all data written to the same location to be brought together in the output.

Figure 30.7 Simulating a concurrent write on an EREW PRAM. (a) A step of a common-CRCW algorithm in which 6 processors write concurrently to global memory. (b) Simulating the step on an EREW PRAM. First, ordered pairs containing location and data are written to an array A. The array is then sorted. By comparing adjacent elements in the array, we ensure that only the first of each group of identical writes into global memory is implemented. In this case, processors P0, P2, and P5 perform the write.

Each EREW processor Pi, for i = 1, 2, . . . , p - 1, now inspects A[i] = (lj, xj) and A[i - 1] = (lk, xk), where j and k are values in the range 0 [image] j, k [image] p - 1. If lj [image] lk or i = 0, then processor Pi, for i = 0, 1, . . . , p - 1, writes the datum xj to location lj in global memory. Otherwise, the processor does nothing. Since the array A is sorted by first coordinate, only one of the processors writing to any given location actually succeeds, and thus the write is exclusive. This process thus implements each step of concurrent writing in the common-CRCW model in O(lg p) time.

Other models for concurrent writing can be simulated as well. (See Exercise 30.2-9.)

The issue arises, therefore, of which model is preferable--CRCW or EREW--and if CRCW, which CRCW model. Advocates of the CRCW models point out that they are easier to program than the EREW model and that their algorithms run faster. Critics contend that hardware to implement concurrent memory operations is slower than hardware to implement exc lusive memory operations, and thus the faster running time of CRCW algorithms is fictitious. In reality, they say, one cannot find the maximum of n values in O(1) time.

Others say that the PRAM--either EREW or CRCW--is the wrong model entirely. Processors must be interconnected by a communication network, and the communication network should be part of the model. Processors should only be able to communicate with their neighbors in the network.

It is quite clear that the issue of the "right" parallel model is not going to be easily settled in favor of any one model. The important thing to realize, however, is that these models are just that: models. For a given real-world situation, the various models apply to differing extents. The degree to which the model matches the engineering situation is the degree to which algorithmic analyses in the model will predict real-world phenomena. It is important to study the various parallel models and algorithms, therefore, so that as the field of parallel computing grows, an enlightened consensus on which paradigms of parallel computing are best suited for implementation can emerge.

Exercises

30.2-1

Suppose we know that a forest of binary trees consists of only a single tree with n nodes. Show that with this assumption, a CREW implementation of FIND-ROOTS can be made to run in O(1) time, independent of the depth of the tree. Argue that any EREW algorithm takes [image] (lg n) time.

30.2-2

Give an EREW algorithm for FIND-ROOTS that runs in O(lg n) time on a forest of n nodes.

30.2-3

Give an n-processor CRCW algorithm that can compute the OR of n boolean values in O(1) time.

30.2-4

Describe an efficient CRCW algorithm to multiply two n [image] n boolean matrices using n3 processors.

30.2-5

Describe an O(lg n)-time EREW algorithm to multiply two n [image] n matrices of real numbers using n3 processors. Is there a faster common-CRCW algorithm? Is there a faster algorithm in one of the stronger CRCW models?

30.2-6

Prove that for any constant [image] > 0, there is an O(1)-time CRCW algorithm using O(n1+ [image]) processors to find the maximum element of an n-element array.

30.2-7

Show how to merge two sorted arrays, each with n numbers, in O(1) time using a priority-CRCW algorithm. Describe how to use this algorithm to sort in O(lg n) time. Is your sorting algorithm work-efficient?

30.2-8

Complete the proof of Theorem 30.1 by describing how a concurrent read on a p-processor CRCW PRAM is implemented in O(lg p) time on a p-processor EREW PRAM.

30.2-9

Show how a p-processor EREW PRAM can implement a p-processor combining-CRCW PRAM with only O(lg p) performance loss. (Hint:Use a parallel prefix computation.)

30.3 Brent's theorem and work efficiency

Brent's theorem shows how we can efficiently simulate a combinational circuit by a PRAM. Using this theorem, we can adapt many of the results for sorting networks from Chapter 28 and many of the results for arithmetic circuits from Chapter 29 to the PRAM model. Readers unfamiliar with combinational circuits may wish to review Section 29.1.

A combinational circuit is an acyclic network of combinational elements. Each combinational element has one or more inputs, and in this section, we shall assume that each element has exactly one output. (Combinational elements with k > 1 outputs can be considered to be k separate elements.) The number of inputs is the fan-in of the element, and the number of places to which its output feeds is its fan-out. We generally assume in this section that every combinational element in the circuit has bounded (O(1)) fan-in. It may, however, have unbounded fan-out.

The size of a combinational circuit is the number of combinational elements that it contains. The number of combinational elements on a longest path from an input of the circuit to an output of a combinational element is the element's depth. The depth of the entire circuit is the maximum depth of any of its elements.

Theorem 30.2

Any depth-d, size-n combinational circuit with bounded fan-in can be stimulated by a p-processor CREW algorithm in O(n/p + d) time.

Proof We store the inputs to the combinational circuit in the PRAM's global memory, and we reserve a location for each combinational element in the circuit to store its output value when it is computed. A given combinational element can then be simulated by a single PRAM processor in O(1) time as follows. The processor simply reads the input values for the element from the values in memory corresponding to circuit inputs or element outputs that feed it, thereby simulating the wires in the circuit. It then computes the function of the combinational element and writes the result in the appropriate position in memory. Since the fan-in of each circuit element is bounded, each function can be computed in O(1) time.

Our job, therefore, is to find a schedule of the p processors of the PRAM such that all combinational elements are simulated in O(n/p+d) time. The main constraint is that an element cannot be simulated until the outputs from any elements that feed it have been computed. Concurrent reads are employed whenever several combinational elements being simulated in parallel require the same value.

Since all elements at depth 1 depend only on circuit inputs, they are the only ones that can be simulated initially. Once they have been simulated, all elements at depth 2 can be simulated, and so forth, until we finish with all elements at depth d. The key idea is that if all elements from depths 1 to i have been simulated, we can simulate any subset of elements at depth i + 1 in parallel, since their computations are independent of one another.

Our scheduling strategy, therefore, is quite naive. We simulate all the elements at depth i before proceeding to simulate those at depth i + 1. Within a given depth i, we simulate the elements p at a time. Figure 30.8 illustrates such a strategy for p = 2.

Let us analyze this simulation strategy. For i = 1, 2, . . . , d, let ni be the number of elements at depth i in the circuit. Thus,

Figure 30.8 Brent's theorem. The combinational circuit of size 15 and depth 5 is simulated by a 2-processor CREW PRAM in 9 15/2 + 5 steps. The simulation proceeds from top to bottom through the circuit. The shaded groups of circuit elements indicate which elements are simulated at the same time, and each group is labeled with a number corresponding to the time step when its elements are simulated.

Consider the ni combinational elements at depth i. By grouping them into [image] ni/p [image] groups, where the first [image] ni/p [image] groups have p elements each and the leftover elements, if any, are in the last group, the PRAM can simulate the computations performed by these combinational elements in O( [image] ni/p [image] ) time. The total simulation time is therefore on the order of

Brent's theorem can be extended to EREW simulations when a combinational circuit has O(1) fan-out for each combinational element.

Corollary 30.3

Any depth-d, size-n combinational circuit with bounded fan-in and fan-out can be simulated on a p-processor EREW PRAM in O(n/p + d) time.

Proof We perform a simulation similar to that in the proof of Brent's theorem. The only difference is in the simulation of wires, which is where Theorem 30.2 requires concurrent reading. For the EREW simulation, after the output of a combinational element is computed, it is not directly read by processors requiring its value. Instead, the output value is copied by the processor simulating the element to the O(1) inputs that require it. The processors that need the value can then read it without interfering with each other.

This EREW simulation strategy does not work for elements with unbounded fan-out, since the copying can take more than constant time at each step. Thus, for circuits having elements with unbounded fan-out, we need the power of concurrent reads. (The case of unbounded fan-in can sometimes be handled by a CRCW simulation if the combinational elements are simple enough. See Exercise 30.3-1.)

Corollary 30.3 provides us with a fast EREW sorting algorithm. As explained in the chapter notes of Chapter 28, the AKS sorting network can sort n numbers in O(lg n) depth using O(n lg n) comparators. Since comparators have bounded fan-in, there is an EREW algorithm to sort n numbers in O(lg n) time using n processors. (We used this result in Theorem 30.1 to show that an EREW PRAM can simulate a CRCW PRAM with at most logarithmic slowdown.) Unfortunately, the constants hidden by the O-notation are so large that this sorting algorithm has solely theoretical interest. More practical EREW sorting algorithms have been discovered, however, notably the parallel merge-sorting algorithm due to Cole [46].

Now suppose that we have a PRAM algorithm that uses at most p processors, but we have a PRAM with only p' < p processors. We would like to be able to run the p-processor algorithm on the smaller p'-processor PRAM in a work-efficient fashion. By using the idea in the proof of Brent's theorem, we can give a condition for when this is possible.

Theorem 30.4

If a p-processor PRAM algorithm A runs in time t, then for any p' < p, there is an p'-processor PRAM algorithm A' for the same problem that runs in time O(pt/p').

Proof Let the time steps of algorithm A be numbered 1, 2, . . . , t. Algorithm A' simulates the execution of each time step i = 1, 2, . . . , t in time O( [image] p/p' [image] ). There are t steps, and so the entire simulation takes time O( [image] p/p' [image] t) = O(pt/p'), since p' < p.

The work performed by algorithm A is pt, and the work performed by algorithm A' is (pt/p')p' = pt; the simulation is therefore work-efficient. Consequently, if algorithm A is itself work-efficient, so is algorithm A'.

When developing work-efficient algorithms for a problem, therefore, one needn't necessarily create a different algorithm for each different number of processors. For example, suppose that we can prove a tight lower bound of t on the running time of any parallel algorithm, no matter how many processors, for solving a given problem, and suppose further that the be st serial algorithm for the problem does work w. Then, we need only develop a work-efficient algorithm for the problem that uses p = [image] (w/t) processors in order to obtain work-efficient algorithms for all numbers of processors for which a work-efficient algorithm is possible. For p' = o(p), Theorem 30.4 guarantees that there is a work-efficient algorithm. For p' = w (p), no work-efficient algorithms exist, since if t is a lower bound on the time for any parallel algorithm, p't = w(pt) = w(w).

Exercises

30.3-1

Prove a result analogous to Brent's theorem for a CRCW simulation of boolean combinational circuits having AND and OR gates with unbounded fan-in. (Hint: Let the "size" be the total number of inputs to gates in the circuit.)

30.3-2

Show that a parallel prefix computation on n values stored in an array of memory can be implemented in O(lg n) time on an EREW PRAM using O(n/lg n) processors. Why does this result not extend immediately to a list of n values?

30.3-3

Show how to multiply an n [image] n matrix A by an n-vector b in O(lg n) time with a work-efficient EREW algorithm. (Hint: Construct a combinational circuit for the problem.)

30.3-4

Give a CRCW algorithm using n2 processors to multiply two n [image] n matrices. The algorithm should be work-efficient with respect to the normal [image] (n3)-time serial algorithm for multiplying matrices. Can you make the algorithm EREW?

30.3-5

Some parallel models allow processors to become inactive, so that the number of processors executing at any step varies. Define the work in this model as the total number of steps executed during an algorithm by active processors. Show that any CRCW algorithm that performs w work and runs in t time can be run on a p-processor EREW PRAM in O((w/p + t) lg p) time. (Hint: The hard part is scheduling the active processors while the computation is proceeding.)

* 30.4 Work-efficient parallel prefix computation

In Section 30.1.2, we examined an O(lg n)-time EREW algorithm LIST- RANK that can perform a prefix computation on an n-object linked list. The algorithm uses n processors and performs [image] (n lg n) work. Since we can easily perform a prefix computation in [image] (n) time on a serial machine, LIST-RANK is not work-efficient.

This section presents a randomized EREW parallel prefix algorithm that is work-efficient. The algorithm uses [image] (n/lg n) processors, and it runs in O(lg n) time with high probability. Thus, it is work-efficient with high probability. Moreover, by Theorem 30.4, this algorithm immediately yields work-efficient algorithms for any number p = O(n/lg n) of processors.

Recursive parallel prefix computation

The randomized parallel prefix algorithm RANDOMIZED-LIST-PREFIX operates on a linked list of n objects using p = [image] (n/ lg n) processors. During the algorithm, each processor is responsible for n/p = [image] (lg n) of the objects in the original list. The objects are assigned to processors arbitrarily (not necessarily contiguously) before the recursion begins, and "ownership" of objects never changes. For convenience, we assume that the list is doubly linked, since doubly linking a single list takes O(1) time.

The idea of RANDOMIZED-LIST-PREFIX is to eliminate some of the objects in the list, perform a recursive prefix computation on the resulting list, and then expand it by splicing in the eliminated objects to yield a prefix computation on the original list. Figure 30.9 illustrates the recursive process, and Figure 30.10 shows how the recursion unfolds. We shall show a little later that each stage of the recursion obeys two properties:

1. At most one object of those belonging to a given processor is selected for elimination.

2. No two adjacent objects are selected for elimination.

Before we show how to select objects that satisfy these properties, let us examine in more detail how the prefix computation is performed. Suppose that at the first step of the recursion, the kth object in the list is selected for elimination. This object contains the value [k, k], which is fetched by the (k + 1)st object in the list. (Boundary situations, such as the one here when k is at the end of the list, can be handled straightforwardly and are not described.) The (k + 1)st object, which holds the value [k + 1, k + 1], then computes and stores [k, k + 1] = [k, k] [image] [k + 1, k + 1]. The kth object is then eliminated from the list by splicing it out.

Figure 30.9 The work-efficient, randomized, recursive, parallel algorithm RANDOMIZED-LIST-PREFIX for performing prefix computations on a linked list of n = 9 objects. (a)-(b) A set of nonadjacent objects (blackened) are selected for elimination. The value in each black object is used to update the value in the next object in the list, and then the black object is spliced out. The algorithm is called recursively to compute a parallel prefix on the contracted list. (c)-(d) The resulting values are the correct final values for objects in the contracted list. The eliminated objects are then spliced back in, and each uses the value of the previous object to compute its final value.

The procedure RANDOMIZED-LIST-PREFIX then calls itself recursively to perform a prefix computation on the "contracted" list. (The recursion bottoms out when the entire list is empty.) The key observation is that after returning from the recursive call, each object in the contracted list has the correct final value it needs for the parallel prefix computation on the original list. It remains only to splice back in the previously eliminated objects, such as the kth object, and update their values.

After the kth object is spliced in, its final prefix value can be computed using the value in the (k - 1)st object. After the recursion, the (k - 1)st object contains [1, k - 1], and thus the kth object--which still has the value [k, k]--needs only to fetch the value [ 1, k - 1] and compute [1,k] = [1, k - 1] [image] [k, k].

Because of property 1, each selected object has a distinct processor to perform the work needed to splice it out or in. Property 2 ensures that no confusion between processors arises when splicing objects out and in (see Exercise 30.4-1). The two properties together ensure that each step of the recursion can be implemented in O(1) time in an EREW fashion.

Figure 30.10 The recursive stages of RANDOMIZED-LIST-PREFIX, shown for n = 9 original objects. In each stage, the blackened objects are eliminated. The procedure recurses until the list is empty, and then the eliminated objects are spliced back in.

Selecting objects for elimination

How does RANDOMIZED-LIST-PREFIX select objects for elimination? It must obey the two properties above, and in addition, we want the time to select objects to be short (and preferably constant). Moreover, we would like as many objects as possible to be selected.

The following method for randomized selection satisfies these conditions. Objects are selected by having each processor execute the following steps:

1. The processor picks an object i that has not previously been selected from among those it owns.

2. It then "flips a coin," choosing the values HEAD and TAIL with equal probability.

3. If it chooses HEAD, it marks object i as selected, unless next[i] has been picked by another processor whose coin is also HEAD.

This randomized method takes only O(1) time to select objects for elimination, and it does not require concurrent memory accesses.

We must show that this procedure obeys the two properties above. That property 1 holds can be seen easily, since only one object is chosen by a processor for possible selection. To see that property 2 holds, suppose to the contrary that two consecutive objects i and next[i] are selected. This occurs only if both were picked by their processors, and both processors flipped HEAD. But object i is not selected if the processor responsible for next[i] flipped HEAD, which is a contradiction.

Analysis

Since each recursive step of RANDOMIZED-LIST-PREFIX runs in O(1) time, to analyze the algorithm we need only determine how many steps it takes to eliminate all the objects in the original list. At each step, a processor has at least probability 1/4 of eliminating the object i it picks. Why? It flips HEAD with probability 1/2, and the probability that it either does not pick next[i] or picks it and flips TAIL is at least 1/2. Since the two coin flips are independent events, we can multiply their probabilities, yielding the probability of at least 1/4 of a processor eliminating the object it picks. Since each processor owns [image] (lg n) objects, the expected time for a processor to eliminate all its objects is [image] (lg n).

Unfortunately, this simple analysis does not show that the expected running time of RANDOMIZED-LIST-PREFIX is [image] (1g n). For example, if most of the processors eliminate all their objects quickly and a few processors take much, much longer, the average time for a processor to eliminate all its objects might still be [image] (lg n), but the running time of the algorithm could be large.

The expected running time of the procedure RANDOMIZED-LIST-PREFIX is indeed [image] (lg n), even though the simple analysis does not show it. We shall use a high-probability argument to prove that with probability at least 1 - 1/n, all objects are eliminated within c lg n stages of the recursion, for some constant c. Exercises 30.4-4 and 30.4-5 ask you to generalize this argument to prove the [image] (lg n) bound on the expected running time.

Our high-probability argument is based on observing that the experiment of a given processor eliminating the objects it picks can be viewed as a sequence of Bernoulli trials (see Chapter 6). The experiment is a success if the object is selected for elimination, and it is a failure otherwise. Since we are interested in showing that the probability is small that very few successes are obtained, we can assume that successes occur with probability exactly 1/4, rather than with probability at least 1/4. (See Exercises 6.4-8 and 6.4-9 for a formal justification of similar assumptions.)

To further simplify the analysis, we assume that there are exactly n/ 1g n processors, each with lg n list objects. We are conducting c lg n trials, for some constant c that we shall determine, and we are interested in the event that fewer than lg n successes occur. Let X be the random variable denoting the total number of successes. By Corollary 6.3, the probability that a processor eliminates fewer than lg n objects in the c lg n trials is at most

as long as c [image] 20. (The second line follows from inequality (6.9).) Thus, the probability that all objects belonging to a given processor have not been eliminated after c lg n steps is at most 1/n2.

We now wish to bound the probability that all objects belonging to all processors have not been eliminated after c lg n steps. By Boole's inequality (6.22), this probability is at most the sum of the probabilities that each of processors has not eliminated its objects. Since there are n/ lg n processors, and each has probability at most 1/n2 of not eliminating all its objects, the probability that any processor has not finished all its objects is at most

We have thus proven that with probability at least 1 - 1/n, every object is spliced out after O(lg n) recursive calls. Since each recursive call takes O(1) time, RANDOMIZED-LIST-PREFIX takes O(lg n) time with high probability.

The constant c [image] 20 in the c 1g n running time may seem a bit large for practicality. In fact, this constant is more an artifact of the analysis than a reflection of the algorithm's performance. In practice, the algorithm tends to be fast. The constant factors in the analysis are large because the event that one processor finishes eliminating all its list objects is dependent on the event that another processor finishes all its work. Because of these dependencies, we used Boole's inequality, which does not require independence but results in a weaker constant than would generally be experienced in practice.

Exercises

30.4-1

Draw figures to illustrate what can go wrong in RANDOMIZED-LIST-PREFIX if two adjacent list objects are selected for elimination.

30.4-2

Suggest a simple change to make RANDOMIZED-LIST-PREFIX run in O(n) worst-case time on a list of n objects. Use the definition of expectation to prove that with this modification, the algorithm runs in O(lg n) expected time.

30.4-3

Show how to implement RANDOMIZED-LIST-PREFIX so that it uses at most O(n/p) space per processor in the worst case, independent of how deep the recursion goes.

30.4-4

Show that for any constant k [image] 1, RANDOMIZED-LIST-PREFIX runs in O(lg n) time with probability at least 1 - 1/nk. Show how the constant in the running-time bound is influenced by k.

30.4-5

Using the result of Exercise 30.4-4, show that the expected running time of RANDOMIZED-LIST-PREFIX is O(lg n).

30.5 Deterministic symmetry breaking

Consider a situation in which two processors wish to acquire mutually exclusive access to an object. How can the processors determine which should acquire access first? We wish to avoid the scenario in which both are granted access, as well as the scenario in which neither is granted access. The problem of choosing one of the processors is an example of symmetry breaking. We have all seen the momentary confusion and diplomatic impasses that arise when two people attempt to go through a door simultaneously. Similar symmetry-breaking problems are pervasive in the design of parallel algorithms, and efficient solutions are extremely useful.

One method for breaking symmetry is to flip coins. On a computer, coin flipping can be implemented by means of a random-number generator. For the two-processor example, both processors can flip coins. If one obtains HEAD and the other TAIL, the one obtaining HEAD proceeds. If both flip the same value, they try again. With this strategy, symmetry is broken in constant expected time (see Exercise 30.5-1).

We saw the effectiveness of a randomized strategy in Section 30.4. In RANDOMIZED-LIST-PREFIX, adjacent list objects must not be selected for elimination, but as many picked objects as possible should be selected. In the midst of a list of picked objects, however, all objects look pretty much the same. As we saw, randomization provides a simple and effective way to break the symmetry between adjacent list objects while guaranteeing that, with high probability, many objects are selected.

In this section, we investigate a deterministic method for breaking symmetry. The key to the algorithm is to employ processor indices or memory addresses rather than random coin flips. For instance, in the two-processor example, we can break the symmetry by allowing the processor with smaller processor index to go first--clearly a constant-time process.

We shall use the same idea, but in a much more clever fashion, in an algorithm to break symmetry in an n-object linked list. The goal is to choose a constant fraction of the objects in the list but to avoid picking two adjacent objects. This algorithm can be performed with n processors in O(lg* n) time by a deterministic EREW algorithm. Since lg* n [image] 5 for all n [image] 265536, the value lg* n can be viewed as a small constant for all practical purposes (see page 36).

Our deterministic algorithm has two parts. The first part computes a "6-coloring" of the linked list in O(lg*n) time. The second part converts the 6-coloring to a "maximal independent set" of the list in O(1) time. The maximal independent set will contain a constant fraction of the n objects of the list, and no two objects in the set will be adjacent.

Colorings and maximal independent sets

A coloring of an undirected graph G = (V, E) is a function C : V [image] N such that for all u, v [image] V, if C(u) = C(v), then (u, v) [image] E; that is, no adjacent vertices have the same color. In a 6-coloring of a linked list, all colors are in the range {0, 1, 2, 3, 4, 5} and no two consecutive vertices have the same color. In fact, any linked list has a 2-coloring, since we can color objects whose ranks are odd with color 0 and objects whose ranks are even with color 1. We can compute such a coloring in O(lg n) time using a parallel prefix computation, but for many applications, it suffices to compute only an O(1)-coloring. We shall show that a 6-coloring can be computed in O(lg* n) time without using randomization.

An independent set of a graph G = (V, E) is a subset V' [image] V of vertices such that each edge in E is incident on at most one vertex in V'. A maximal independent set, or MIS, is an independent set V' such that for all vertices v [image] V - V', the set V' [image] {v} is not independent--every vertex not in V' is adjacent to some vertex in V'. Do not confuse the problem of computing a maximal independent set--an easy problem--with the problem of computing a maximum independent set--a hard problem. The problem of finding an independent set of maximum size in a general graph is NP-complete. (See Chapter 36 for a discussion of NP-completeness. Problem 36-1 concerns maximum independent sets.)

For n-object lists, a maximum (and hence maximal) independent set can be determined in O(lg n) time by using a parallel prefix computation, as in the 2-coloring just mentioned, to identify the odd-ranked objects. This method selects [image] n/2 [image] objects. Observe, however, that any maximal independent set of a linked list contains at least n/3 objects, since for any 3 consecutive objects, at least one must be in the set. We shall show, however, that a maximal independent set of a list can be determined in O(1) time given an O(1)-coloring of the list.

Computing a 6-coloring

The algorithm SIX-COLOR computes a 6-coloring of a list. We won't give pseudocode for the algorithm, but we shall describe it in some detail. We assume that initially each object x in the linked list is associated with a distinct processor P(x) [image] {0, 1, . . . , n - 1}.

The idea of SIX-COLOR is to compute a sequence C0[x], C1[x], . . . , Cm[x] of colors for each object x in the list. The initial coloring C0 is a n-coloring. Each iteration of the algorithm defines a new coloring Ck + 1 based on the previous coloring Ck, for k = 0, 1, . . . , m - 1. The final coloring Cm is a 6-coloring, and we shall prove that m = O(lg* n).

The initial coloring is the trivial n-coloring in which C0[x] = P(x). Since no two list objects have the same color, no two adjacent list objects have the same color, and so the coloring is legal. Note that each of the initial colors can be described with [image] lg n [image] bits, which means that it can be stored in an ordinary computer word.

The subsequent colorings are obtained as follows. The kth iteration, for k = 0, 1, . . . , m - 1, starts with a coloring Ck and ends with a coloring Ck + 1 using fewer bits per object, as the first part of Figure 30.11 shows. Suppose that at the start of an iteration, each object's color Ck takes r bits. We determine the new color of an object x by looking forward in the list at the color of next[x].

To be more precise, suppose that for each object x, we have Ck[x] = a and Ck[next[x]] = b, where a = [image] ar-1, ar-2, . . . , a0 [image] and b = [image] br-1, br-2, . . . , b0 [image] are r-bit colors. Since Ck[x] [image] Ck[next[x]], there is some least index i at which the bits of the two colors differ: ai [image] bi. Because 0 [image] i [image] r - 1, we can write i with only [lg r] bits: i = [image] i [image] lg r [image] -1, i [image] lg r [image] -2, . . . , i0 [image] . We recolor x with the value of i concatenated with the bit ai. That is, we assign

Ck+1[x]  =    i, ai

=    i  lg r   - 1, i   lg r  -2, . . . , i0, ai  .

The tail of the list gets the new color (0, a0). The number of bits in each new color is therefore at most [image] lg r [image] + 1.

We must show that if each iteration of SIX-COLOR starts with a coloring, the new "coloring" it produces is indeed a legal coloring. To do this, we prove that Ck[x] [image] Ck next[x]] implies Ck+1[x] [image] Ck+1[next[x]]. Suppose that Ck[x] = a and Ck[next[x]] = b, and that Ck+1[x] = [image] i, ai [image] and Ck+1[next[x]] = [image] j, bj [image]. There are two cases to consider. If i [image] j, then [image] i, ai [image] [image] [image] j, bj [image], and so the new colors are different. If i = j, however, then ai [image] bi = bj by our recoloring method, and thus the new colors are once again different. (The situation at the tail of the list can be handled similarly.)

The recoloring method used by SIX-COLOR takes an r-bit color and replaces it with a ( [image] lg r [image] + 1)-bit color, which means that the number of bits is strictly reduced as long as r [image] 4. When r = 3, two colors can differ in bit position 0, 1, or 2. Each new color, therefore, is [image] 00 [image], [image] 01 [image], or [image] 10 [image] concatenated with either 0 or 1, thus leaving a 3-bit number once again. Only 6 of the 8 possible values for 3-bit numbers are used, however, so that SIX-COLOR indeed terminates with a 6-coloring.

Assuming that each processor can determine the appropriate index i in O(1) time and perform a shift-left operation in O(1) time--operations commonly supported on many actual machines--each iteration takes O(1) time. The SIX-COLOR procedure is an EREW algorithm: for each object x, its processor accesses only x and next[x].

Finally, let us see why only O(lg* n) iterations are required to bring the initial n-coloring down to a 6-coloring. We have defined lg* n as the number of times the algorithm function lg needs to be applied to n to reduce it to at most 1 or, letting lg(i) n denote i successive applications of the lg function,

Figure 30.11 The algorthms SIX-COLOR and LIST-MIS that break symmetry in a list. Together, the algorithms find a large set of nonadjacent objects in O(lg* n) time using n processors. The initial list of n = 20 objects is shown on the left, running vertically. Each object has an initial, distinct 5-bit color. For these parameters, the algorithm SIX-COLOR needs only the two iterations shown to recolor each object with a color in the range {0, 1, 2, 3, 4, 5}. White objects are placed into the MIS by LIST-MIS as the colors are processed, and black objects are killed.

lg* n = min {i    0 : lg(i) n    1} .

Let ri be the number of bits in the coloring just before the ith iteration. We shall prove by induction that if [image] lg(i) n [image] [image] 2, then ri [image] [image] lg(i) n [image] + 2. Initially, we have r1 [image] [image] lg n [image] . The ith iteration brings the number of bits in the coloring down to ri+1 = [image] lg ri [image] + 1. Assuming that the inductive hypothesis holds for ri - 1, we obtain

ri  =      lg ri-1   + 1

         lg(  lg(i-1)n   + 2)   + 1

         lg(lg(i-1)n + 3)   + 1

         lg(2 lg(i-1)n)   + 1

=       lg(lg(i-1)n) + 1   + 1

=       lg (i)n   + 2.

The fourth line follows from the assumption that [image] lg(i) n [image] [image] 2, which means that [image] lg(i-1)n [image] [image] 3. Therefore, after m = lg* n steps, the number of bits in the coloring is rm [image] [image] lg(m) n [image] + 2 = 3, since lg (m) n [image] 1 by defination of the lg* function. Thus, at most one more iteration suffices to produce a 6-coloring. The total time of SIX-COLOR is therefore O(lg* n).

Computing an MIS from a 6-coloring

Coloring is the hard part of symmetry breaking. The EREW algorithm LIST-MIS uses n processors to find a maximal independent set in O(c) time given a c-coloring of an n-object list. Thus, once we have computed a 6-coloring of a list, we can find a maximal independent set of the linked list in O(1) time.

The latter part of Figure 30.11 illustrates the idea behind LIST-MIS. We are given a c-coloring C. With each object x, we keep a bit alive[x], which tells us whether x is still a candidate for inclusion in the MIS. Initially, alive[x] = TRUE for all objects x.

The algorithm then iterates through each of the c colors. In the iteration for color i, each processor responsible for an object x checks whether C[x] = i and alive[x] = TRUE. If both conditions hold, then the processor marks x as belonging to the MIS being constructed. All objects adjacent to those added in the MIS--those immediately preceding or following--have their alive bits set to FALSE; they cannot be in the MIS because they are adjacent to an object in the MIS. after all c iterations, each object has either been "killed"--its alive bit has been set to FALSE--or placed into the MIS.

We must show that the resulting set is independent and maximal. To see that it is independent, suppose that two adjacent objects x and next[x] are placed into the set. Since they are adjacent, C[x] [image] C[next[x]], because C is a coloring. Without loss of generality, we assume that C[x] < C[next[x]], so that x is placed into the set before next[x] is. But then alive[next[x]] has been set to FALSE by the time objects of color C[next[x]] are considered, and next[x] could not have been place into the set.

To see that the set is maximal, suppose that none of three consecutive objects x, y, and z has been placed into the set. The only way that y could have avoided being placed into the set, though, is if it had been killed when an adjacent object was placed into the set. Since, by our supposition, neither x nor z was placed into the set, the object y must have been still alive at the time when objects of color C[y] were processed. It must have been placed into the MIS.

Each iteration of LIST-MIS takes O(1) time on a PRAM. The algorithm is EREW since each object accesses only itself, its predecessor, and its successor in the list. Combining LIST-MIS with SIX-COLOR, we can break symmetry in a linked list in O(lg* n) time deterministically.

Exercises

30.5-1

For the 2-processor symmetry-breaking example at the beginning of this section, show that symmetry is broken in constant expected time.

30.5-2

Given a 6-coloring of an n-object list, show how to 3-color the list in O(1) time using n processors in an EREW PRAM.

30.5-3

Suppose that every nonroot node in an n-node tree has a pointer to its parent. Give a CREW algorithm to O(1)-color the tree in O(lg* n) time.

30.5-4

Give an efficient PRAM algorithm to O(1)-color a degre-3 graph. Analyze your algorithm.

30.5-5

A k-ruling set of a linked list is a set of objects (rulers) in the list such that no rulers are adjacent and at most k nonrulers (subjects) separate rulers. Thus, an MIS is a 2-ruling set. Show how an O(lg n)-ruling set of an n-object list can be computed in O(1) time using n processors. Show how an O(lg lg n) ruling set can be computed in O(1) time under the same assumptions.

30.5-6

Show how to find a 6-coloring of an n-object linked list in O(lg(lg* n)) time. Assume that each processor can store a precomputed table of size O(lg n). (Hint: In SIX-COLOR, upon how many values does the final color of an object depend?)

Problems

30-1 Segmented parallel prefix

Like an ordinary prefix computation, a segmented prefix computation is defined in terms of a binary, associative operator [image]. It takes an input sequence x = [image] x1, x2, . . . , xn [image] whose elements are drawn from a domains and a segment sequence b = [image] b1, b2, . . . , bn [image] whose elements are drawn from the domain {0,1}, with b1 = 1. It produces an output sequence y = [image] y1, y2, . . . , yn [image] over the domain S. The bits of b determine a partitioning of x and y into segments; a new segment begins wherever bi = 1, and the current one continues if bi = 0. The segmented prefix computation performs an independent prefix computation within each segment of x to produce the corresponding segment of y. Figure 30.12 illustrates a segmented prefix computation using ordinary addition.

a. Define the operator [image] on ordered pairs (a, z), (a', z') [image] {0, 1} X S as follows:

Prove that [image] is associative.

b. Show how to implement any segmented prefix computation on an n-element list in O(lg n) time on an EREW PRAM.

c. Describe an O(k lg n)-time EREW algorithm to sort a list of n k-bit numbers.

30-2 Processor-efficient maximum algorithm

We wish to find the maximum of n numbers on a CRCW PRAM with p = n processors.

a. Show that the problem of finding the maximum of m [image] p/2 numbers can be reduced to the problem of finding the maximum of at most m2/p numbers in O(1) time on a p-processor CRCW PRAM.

b. If we start with m = [image] p/2 [image] numbers, how many numbers remain after k iterations of the algorithm in part (a)?

Figure 30.12 A segmented prefix computation with segment sequence b, input sequence x, and output sequence y. There are 5 segments.

c. Show that the problem of finding the maximum of n numbers can be solved in O(lg lg n) time on a CRCW PRAM with p = n processors.

30-3 Connected components

In this problem, we investigate an arbitrary-CRCW algorithm for computing the connected components of an undirected graph G = (V, E) that uses |V + E| processors. The data structure used is a disjoint-set forest (see Section 22.3). Each vertex v [image] V maintains a pointer p[v] to a parent. Initially, p[v] = v: the vertex points to itself. At the end of the algorithm, for any two vertices u, v [image] V, we have p[u] = p[v] if and only if [image] in G. During the algorithm, the p pointers form a forest of rooted pointer trees. A star is a pointer tree in which p[u] = p[v] for all vertices u and v in the tree.

The connected-components algorithm assumes that each edge (u, v) [image] E appears twice: once as (u,v) and once as (v,u). The algorithm uses two basic operations, HOOK and JUMP, and a subroutine STAR that sets star[v] = TRUE if v belongs to a star.

HOOK(G)

1 STAR (G)

2 for each edge (u, v)    E[G], in parallel

3      do if star[u] and p[u] > p[v]

4            then p[p[u]]    p[v]

5 STAR(G)

6 for each edge (u, v)    E[G], in parallel

7      do if star[u] and p[u]    p[v]

8            then p[p[u]]    p[v]

JUMP(G)

1  for each v    V[G], in parallel

2      do p[v]    p[p[v]]

The connected-components algorithm performs an initial HOOK, and then it repeatedly performs HOOK, JUMP, HOOK, JUMP, and so on, until no pointer is changed by a JUMP operation. (Note that two HOOK'S are performed before the first JUMP.)

a.     Give pseudocode for STAR(G).

b.     Show that the p pointers indeed form rooted trees, with the root of a tree pointing to itself. Show that if u and v are in the same pointer tree, then   [image]  in G.

c.     Show that the algorithm is correct: it terminates, and when it terminates, p[u] = p[v] if and only if   [image]  in G.

To analyze the connected-components algorithm, let us examine a single connected component C, which we assume has at least two vertices. Suppose that at some point during the algorithm, C is made up of a set {Ti} of pointer trees. Define the potential of C as

The goal of our analysis is to prove that each iteration of hooking and jumping decreases   [image] (C) by a constant factor.

d.     Prove that after the initial HOOK, there are no pointer trees of height 0 and   [image] (C)   [image]  |V|.

e.     Argue that after the initial HOOK, subsequent HOOK operations never increase   [image] (C).

f.     Show that after every noninitial HOOK operation, no pointer tree is a star unless the pointer tree contains all vertices in C.

g.     Argue that if C has not been collapsed into a single star, then after a JUMP operation,   [image] (C) is at most 2/3 its previous value. Illustrate the worst case.

h.     Conclude that the algorithm determines all the connected components of G in O(lg V) time.

30-4.     Transposing a raster image

A raster-graphics frame buffer can be viewed as a p X p matrix M of bits. The raster-graphics display hardware makes the n X n upper left submatrix of M visible on the user's screen. A BITBLT operation (BLock Transfer of BITs) is used to move a rectangle of bits from one position to another. Specifically, BITBLT(r1, c1, r2, c2, nr, nc, *) sets

M[r2 + i, c2 + j]    M[r2 + i, c2 + j] * M[r1 + i, c1 + j]

for i = 0, 1, . . . , nr - 1 and j = 0, 1, . . . , nc - 1, where * is any of the 16 boolean functions on two inputs.

We are interested in transposing the image (M[i, j]   [image]  M[j, i]) in the visible portion of the frame buffer. We assume that the cost of copying the bits is less than that of calling the BITBLT primitive, and hence we are interested in using as few BITBLT operations as possible.

Show that any image on the screen can be transposed with O(lg n) BITBLT operations. Assume that p is sufficiently larger than n so that the nonvisible portion of the frame buffer provides enough working storage. How much additional storage do you need? (Hint: Use a parallel divide-and-conquer approach in which some of the BITBLT's are performed with boolean AND's.)

Chapter notes

Akl [9], Karp and Ramachandran [118], and Leighton [135] survey parallel algorithms for combinatorial problems. Various parallel machine architectures are described by Hwang and Briggs [109] and Hwang and DeGroot [110].

The theory of parallel computing began in the late 1940's when J. Von Neumann [38] introduced a restricted model of parallel computing called a cellular automaton, which is essentially a two-dimensional array of finite-state processors interconnected in meshlike fashion. The PRAM model was formalized in 1978 by Fortune and Wyllie [73], although many other authors had previously discussed essentially similar models.

Pointer jumping was introduced by Wyllie [204]. The study of parallel prefix computations arose from the work of Ofman [152] in the context of carry-lookahead addition. The Euler-tour technique is due to Tarjan and Vishkin [191].

Processor-time trade-offs for computing the maximum of a set of n numbers were provided by Valiant [193], who also showed that an O(1)-time work-efficient algorithm does not exist. Cook, Dwork, and Reischuk [50] proved that the problem of computing the maximum requires   [image] (lg n) time on a CREW PRAM. The simulation of a CRCW algorithm with an EREW algorithm is due to Vishkin [195].

Theorem 30.2 is due to Brent [34]. The randomized algorithm for work-efficient list ranking was discovered by Anderson and Miller [11]. They also have a deterministic, work-efficient algorithm for the same problem [10]. The algorithm for deterministic symmetry breaking is due to Goldberg and Plotkin [84]. It is based on a similar algorithm with the same running time due to Cole and Vishkin [47].

Go to Chapter 31     Back to Table of Contents


Intro to Algorithms: CHAPTER 31: MATRIX OPERATIONS

 Return to Table of Contents

CHAPTER 31: MATRIX OPERATIONS

Operations on matrices are at the heart of scientific computing. Efficient algorithms for working with matrices are therefore of considerable practical interest. This chapter provides a brief introduction to matrix theory and matrix operations, emphasizing the problems of multiplying matrices and solving sets of simultaneous linear equations.

After Section 31.1 introduces basic matrix concepts and notations, Section 31.2 presents Strassen's surprising algorithm for multiplying two n [image] n matrices in [image] (nlg 7) = O(n2.81) time. Section 31.3 defines quasirings, rings, and fields, clarifying the assumptions required to make Strassen's algorithm work. It also contains an asymptotically fast algorithm for multiplying boolean matrices. Section 31.4 shows how to solve a set of linear equations using LUP decompositions. Then, Section 31.5 explores the close relationship between the problem of multiplying matrices and the problem of inverting a matrix. Finally, Section 31.6 discusses the important class of symmetric positive-definite matrices and shows how they can be used to find a least-squares solution to an overdetermined set of linear equations.

31.1 Properties of matrices

In this section, we review some basic concepts of matrix theory and some fundamental properties of matrices, focusing on those that will be needed in later sections.

Matrices and vectors

A matrix is a rectangular array of numbers. For example,

(31.1)

is a 2 [image] 3 matrix A = (aij), where for i = 1, 2 and j = 1, 2, 3, the element of the matrix in row i and column j is aij. We use uppercase letters to denote matrices and corresponding subscripted lowercase letters to denote their elements. The set of all m [image] n matrices with real-valued entries is denoted R m [image] n. In general, the set of m [image] n matrices with entries drawn from a set S is denoted Sm [image] n.

The transpose of a matrix A is the matrix AT obtained by exchanging the rows and columns of A. For the matrix A of equation (31.1),

A vector is a one-dimensional array of numbers. For example,

(31.2)

is a vector of size 3. We use lowercase letters to denote vectors, and we denote the ith element of a size-n vector x by xi, for i = 1, 2, . . . , n. We take the standard form of a vector to be as a column vector equivalent to an n [image] 1 matrix; the corresponding row vector is obtained by taking the transpose:

xT = ( 2 3 5 ) .

The unit vector ei is the vector whose ith element is 1 and all of whose other elements are 0. Usually, the size of a unit vector is clear from the context.

A zero matrix is a matrix whose every entry is 0. Such a matrix is often denoted 0, since the ambiguity between the number 0 and a matrix of 0's is usually easily resolved from context. If a matrix of 0's is intended, then the size of the matrix also needs to be derived from the context.

Square n [image] n matrices arise frequently. Several special cases of square matrices are of particular interest:

1. A diagonal matrix has aij = 0 whenever i [image] j. Because all of the off-diagonal elements are zero, the matrix can be specified by listing the elements along the diagonal:

2 The n [image] n identity matrix In is a diagonal matrix with 1's along the diagonal:

In = diag(1,1,...,1)

When I appears without a subscript, its size can be derived from context. The ith column of an identity matrix is the unit vector ei.

3. A tridiagonal matrix T is one for which tij = 0 if |i - j| > 1. Nonzero entries appear only on the main diagonal, immediately above the main diagonal (ti,i+1 for i = 1, 2, . . . , n - 1), or immediately below the main diagonal (ti+1,i for i = 1, 2, . . . , n - 1):

4. An upper-triangular matrix U is one for which uij = 0 if i > j. All entries below the diagonal are zero:

An upper-triangular matrix is unit upper-triangular if it has all 1's along the diagonal.

5. A lower-triangular matrix L is one for which lij = 0 if i < j. All entries above the diagonal are zero:

A lower-triangular matrix is unit lower-triangular if it has all 1's along the diagonal.

6. A permutation matrix P has exactly one 1 in each row or column, and 0's elsewhere. An example of a permutation matrix is

Such a matrix is called a permutation matrix because multiplying a vector x by a permutation matrix has the effect of permuting (rearranging) the elements of x.

7. A symmetric matrix A satisfies the condition A = AT. For example,

is a symmetric matrix.

Operations on matrices

The elements of a matrix or vector are numbers from a number system, such as the real numbers, the complex numbers, or integers modulo a prime. The number system defines how to add and multiply numbers. We can extend these definitions to encompass addition and multiplication of matrices.

We define matrix addition as follows. If A = (aij) and B = (bij) are m [image] n matrices, then their matrix sum C = (cij) = A + B is the m [image] n matrix defined by

cij = aij + bij

for i = 1, 2, . . . , m and j = 1, 2, . . . , n. That is, matrix addition is performed componentwise. A zero matrix is the identity for matrix addition:

A + 0 = A

= 0 + A .

If [image] is a number and A = (aij) is a matrix, then [image] A = ( [image] aij) is the scalar multiple of A obtained by multiplying each of its elements by [image]. As a special case, we define the negative of a matrix A = (aij) to be -1 [image] A = - A, so that the ijth entry of - A is -aij. Thus,

A + (-A) = 0

= (-A) + A .

Given this definition, we can define matrix subtraction as the addition of the negative of a matrix: A - B = A + (-B).

We define matrix multiplication as follows. We start with two matrices A and B that are compatible in the sense that the number of columns of A equals the number of rows of B. (In general, an expression containing a matrix product AB is always assumed to imply that matrices A and B are compatible.) If A = (aij) is an m [image] n matrix and B = (bjk) is an n [image] p matrix, then their matrix product C = AB is the m [image] p matrix C = (cik), where

(31.3)

for i = 1, 2, . . . , m and k = 1, 2, . . . , p. The procedure MATRIX-MULTIPLY in Section 26.1 implements matrix multiplication in the straightforward manner based on equation (31.3), assuming that the matrices are square: m = n = p. To multiply n [image] n matrices, MATRIX-MULTIPLY performs n3 multiplications and n2(n - 1) additions, and its running time is [image] (n3).

Matrices have many (but not all) of the algebraic properties typical of numbers. Identity matrices are identities for matrix multiplication:

ImA = AIn = A

for any m [image] n matrix A. Multiplying by a zero matrix gives a zero matrix:

A 0 = 0 .

Matrix multiplication is associative:

A(BC) = (AB)C

(31.4)

for compatible matrices A, B, and C. Matrix multiplication distributes over addition:

A(B + C) = AB + AC ,

(B + C)D = BD + CD .

(31.5)

Multiplication of n [image] n matrices is not commutative, however, unless n = 1. For example, if [image] and [image], then

and

Matrix-vector products or vector-vector products are defined as if the vector were the equivalent n [image] 1 matrix (or a 1 [image] n matrix, in the case of a row vector). Thus, if A is an m [image] n matrix and x is a vector of size n, then Ax is a vector of size m. If x and y are vectors of size n, then

is a number (actually a 1 [image] 1 matrix) called the inner product of x and y. The matrix xyT is an n [image] n matrix Z called the outer product of x and y, with zij = xiyj. The (euclidean) norm ||x|| of a vector x of size n is defined by

Thus, the norm of x is its length in n-dimensional euclidean space.

Matrix inverses, ranks, and determinants

We define the inverse of an n [image] n matrix A to be the n [image] n matrix, denoted A-1 (if it exists), such that AA-1 = In = A-1 A. For example,

Many nonzero n [image] n matrices do not have inverses. A matrix without an inverse is is called noninvertible, or singular. An example of a nonzero singular matrix is

If a matrix has an inverse, it is called invertible, or nonsingular. Matrix inverses, when they exist, are unique. (See Exercise 31.1-4.) If A and B are nonsingular n [image] n matrices, then

(BA)-1 = A-1B-1.

(31.6)

The inverse operation commutes with the transpose operation:

(A-1)T = (AT)-1 .

The vectors x1, x2, . . . , xn are linearly dependent if there exist coefficients c1, c2, . . . , cn, not all of which are zero, such that c1x1 + c2x2 + . . . + cnxn = 0. For example, the vectors x1 = ( 1 2 3 )T, x2 = ( 2 6 4 )T, and x3 = ( 4 11 9 )T are linearly dependent, since 2x1 + 3x2 - 2x3 = 0. If vectors are not linearly dependent, they are linearly independent. For example, the columns of an identity matrix are linearly independent.

The column rank of a nonzero m [image] n matrix A is the size of the largest set of linearly independent columns of A. Similarly, the row rank of A is the size of the largest set of linearly independent rows of A. A fundamental property of any matrix A is that its row rank always equals its column rank, so that we can simply refer to the rank of A. The rank of an m [image] n matrix is an integer between 0 and min(m, n), inclusive. (The rank of a zero matrix is 0, and the rank of an n [image] n identity matrix is n.) An alternate, but equivalent and often more useful, definition is that the rank of a nonzero m [image] n matrix A is the smallest number r such that there exist matrices B and C of respective sizes m [image] r and r [image] n such that

A = BC .

A square n [image] n matrix has full rank if its rank is n. A fundamental property of ranks is given by the following theorem.

Theorem 31.1

A square matrix has full rank if and only if it is nonsingular.

An m [image] n matrix has full column rank if its rank is n.

A null vector for a matrix A is a nonzero vector x such that Ax = 0. The following theorem, whose proof is left as Exercise 31.1-8, and its corollary relate the notions of column rank and singularity to null vectors.

Theorem 31.2

A matrix A has full column rank if and only if it does not have a null vector.

Corollary 31.3

A square matrix A is singular if and only if it has a null vector.

The ijth minor of an n [image] n matrix A, for n > 1, is the (n - 1) [image] (n - 1) matrix A[ij] obtained by deleting the ith row and jth column of A. The determinant of an n [image] n matrix A can be defined recursively in terms of its minors by

(31.7)

The term (-1)i+ j det(A[ij]) is known as the cofactor of the element aij.

The following theorems, whose proofs are omitted here, express fundamental properties of the determinant.

Theorem 31.4

The determinant of a square matrix A has the following properties:

If any row or any column of A is zero, then det (A) = 0.

The determinant of A is multiplied by [image] if the entries of any one row (or any one column) of A are all multiplied by [image].

The determinant of A is unchanged if the entries in one row (respectively, column) are added to those in another row (respectively, column).

The determinant of A equals the determinant of AT.

The determinant of A is multiplied by - 1 if any two rows (respectively, columns) are exchanged.

Also, for any square matrices A and B, we have det (AB) = det(A) det(B).

Theorem 31.5

An n [image] n matrix A is singular if and only if det(A) = 0.

Positive-definite matrices

Positive-definite matrices play an important role in many applications. An n [image] n matrix A is positive-definite if xT Ax > 0 for all size-n vectors x [image] 0. For example, the identity matrix is positive-definite, since for any nonzero vector x = ( x1 x2 . . . xn)T,

As we shall see, matrices that arise in applications are often positive-definite due to the following theorem.

Theorem 31.6

For any matrix A with full column rank, the matrix ATA is positive-definite.

Proof We must show that xT (ATA)x > 0 for any nonzero vector x. For any vector x,

xT(AT A)x = (Ax)T(Ax)   (by Exercise 31.1-3)

= ||Ax||2

   0 .

(31.8)

Note that ||Ax||2 is just the sum of the squares of the elements of the vector Ax. Therefore, if ||Ax||2 = 0, every element of Ax is 0, which is to say Ax = 0. Since A has full column rank, Ax = 0 implies x = 0, by Theorem 31.2. Hence, AT A is positive-definite.

Other properties of positive-definite matrices will be explored in Section 31.6.

Exercises

31.1-1

Prove that the product of two lower-triangular matrices is lower-triangular. Prove that the determinant of a (lower- or upper-) triangular matrix is equal to the product of its diagonal elements. Prove that the inverse of a lower-triangular matrix, if it exists, is lower-triangular.

31.1-2

Prove that if P is an n [image] n permutation matrix and A is an n [image] n matrix, then PA can be obtained from A by permuting its rows, and AP can be obtained from A by permuting its columns. Prove that the product of two permutation matrices is a permutation matrix. Prove that if P is a permutation matrix, then P is invertible, its inverse is PT, and PT is a permutation matrix.

31.1-3

Prove that (AB)T = BT AT and that AT A is always a symmetric matrix.

31.1-4

Prove that if B and C are inverses of A, then B = C.

31.1-5

Let A and B be n [image] n matrices such that AB = I. Prove that if A' is obtained from A by adding row j into row i, then the inverse B' of A' can be obtained by subtracting column i from column j of B.

31.1-6

Let A be a nonsingular n [image] n matrix with complex entries. Show that every entry of A-1 is real if and only if every entry of A is real.

31.1-7

Show that if A is a nonsingular symmetric matrix, then A-1 is symmetric. Show that if B is an arbitrary (compatible) matrix, then B ABT is symmetric.

31.1-8

Show that a matrix A has full column rank if and only if Ax = 0 implies x = 0. (Hint: Express the linear dependence of one column on the others as a matrix-vector equation.)

31.1-9

Prove that for any two compatible matrices A and B,

rank(AB) [image] min(rank(A), rank(B)) ,

where equality holds if either A or B is a nonsingular square matrix. (Hint: Use the alternate definition of the rank of a matrix.)

31.1-10

Given numbers x0, x1, . . . , xn-1, prove that the determinant of the Vandermonde matrix

is

(Hint: Multiply column i by -x0 and add it to column i + 1 for i = n - 1, n - 2, . . . , 1, and then use induction.)

31.2 Strassen's algorithm for matrix multiplication

This section presents Strassen's remarkable recursive algorithm for multiplying n [image] n matrices that runs in [image] (nlg 7) = O(n2.81) time. For sufficiently large n, therefore, it outperforms the naive [image] (n3) matrix-multiplication algorithm MATRIX-MULTIPLY from Section 26.1.

An overview of the algorithm

Strassen's algorithm can be viewed as an application of a familiar design technique: divide and conquer. Suppose we wish to compute the product C = AB, where each of A, B, and C are n [image] n matrices. Assuming that n is an exact power of 2, we divide each of A, B, and C into four n/2 [image] n/2 matrices, rewriting the equation C = AB as follows:

(31.9)

(Exercise 31.2-2 deals with the situation in which n is not an exact power of 2.) For convenience, the submatrices of A are labeled alphabetically from left to right, whereas those of B are labeled from top to bottom, in agreement with the way matrix multiplication is performed. Equation (31.9) corresponds to the four equations

r = ae + bf ,

(31.10)

s = ag + bh ,

(31.11)

t = ce + df ,

(31.12)

u = cg + dh .

(31.13)

Each of these four equations specifies two multiplications of n/2 [image] n/2 matrices and the addition of their n/2 [image] n/2 products. Using these equations to define a straightforward divide-and-conquer strategy, we derive the following recurrence for the time T(n) to multiply two n [image] n matrices:

T(n) = 8T(n/2) +   (n2).

(31.14)

Unfortunately, recurrence (31.14) has the solution T(n) = [image] (n3), and thus this method is no faster than the ordinary one.

Strassen discovered a different recursive approach that requires only 7 recursive multiplications of n/2 [image] n/2 matrices and [image] (n2) scalar additions and subtractions, yielding the recurrence

T(n) = 7T(n/2) +   (n2)

=   (nlg 7)

= O(n2.81) .

(31.15)

Strassen's method has four steps:

1. Divide the input matrices A and B into n/2 [image] n/2 submatrices, as in equation (31.9).

2. Using [image] (n2) scalar additions and subtractions, compute 14 n/2 [image] n/2 matrices A1, B1, A2, B2, . . . , A7, B7.

3. Recursively compute the seven matrix products Pi = AiBi for i = 1, 2, . . . , 7.

4. Compute the desired submatrices r, s, t, u of the result matrix C by adding and/or subtracting various combinations of the Pi matrices, using only [image] (n2) scalar additions and subtractions.

Such a procedure satisfies the recurrence (31.15). All that we have to do now is fill in the missing details.

Determining the submatrix products

It is not clear exactly how Strassen discovered the submatrix products that are the key to making his algorithm work. Here, we reconstruct one plausible discovery method.

Let us guess that each matrix product Pi can be written in the form

Pi  =  AiBi

=  (  i1a +   i2b +   i3c +   i4d)    (  i1e +   i2f +   i3g +   i4h) ,

(31.16)

where the coefficients [image] ij, [image] ij are all drawn from the set {-1, 0, 1}. That is, we guess that each product is computed by adding or subtracting some of the submatrices of A, adding or subtracting some of the submatrices of B, and then multiplying the two results together. While more general strategies are possible, this simple one turns out to work.

If we form all of our products in this manner, then we can use this method recursively without assuming commutativity of multiplication, since each product has all of the A submatrices on the left and all of the B submatrices on the right. This property is essential for the recursive application of this method, since matrix multiplication is not commutative.

For convenience, we shall use 4 [image] 4 matrices to represent linear combinations of products of submatrices, where each product combines one submatrix of A with one submatrix of B as in equation (31.16). For example, we can rewrite equation (31.10) as

The last expression uses an abbreviated notation in which "+" represents +1, " [image] represents 0, and "-" represents -1. (From here on, we omit the row and column labels.) Using this notation, we have the following equations for the other submatrices of the result matrix C:

We begin our search for a faster matrix-multiplication algorithm by observing that the submatrix s can be computed as s = P1 + P2, where P1 and P2 are computed using one matrix multiplication each:

The matrix t can be computed in a similar manner as t = P3 + P4, where

Let us define an essential term to be one of the eight terms appearing on the right-hand side of one of the equations (31.10)--(31.13). We have now used 4 products to compute the two submatrices s and t whose essential terms are ag, bh, ce, and df . Note that P1 computes the essential term ag, P2 computes the essential term bh, P3 computes the essential term ce, and P4 computes the essential term df. Thus, it remains for us to compute the remaining two submatrices r and u, whose essential terms are the diagonal terms ae, bf, cg, and dh, without using more than 3 additional products. We now try the innovation P5 in order to compute two essential terms at once:

In addition to computing both of the essential terms ae and dh, P5 computes the inessential terms ah and de, which need to be cancelled somehow. We can use P4 and P2 to cancel them, but two other inessential terms then appear:

By adding an additional product

however, we obtain

We can obtain u in a similar manner from P5 by using P1 and P3 to move the inessential terms of P5 in a different direction:

By subtracting an additional product

we now obtain

The 7 submatrix products P1, P2, . . . , P7 can thus be used to compute the product C = AB, which completes the description of Strassen's method.

Discussion

The large constant hidden in the running time of Strassen's algorithm makes it impractical unless the matrices are large (n at least 45 or so) and dense (few zero entries). For small matrices, the straightforward algorithm is preferable, and for large, sparse matrices, there are special sparsematrix algorithms that beat Strassen's in practice. Thus, Strassen's method is largely of theoretical interest.

By using advanced techniques beyond the scope of this text, one can in fact multiply n [image] n matrices in better than [image] (n1g 7) time. The current best upper bound is approximately O(n2.376). The best lower bound known is just the obvious [image] (n2) bound (obvious because we have to fill in n2 elements of the product matrix). Thus, we currently do not know how hard matrix multiplication really is.

Strassen's algorithm does not require that the matrix entries be real numbers. All that matters is that the number system form an algebraic ring. If the matrix entries do not form a ring, however, sometimes other techniques can be brought to bear to allow his method to apply. These issues are discussed more fully in the next section.

Exercises

31.2-1

Use Strassen's algorithm to compute the matrix product

Show your work.

31.2-2

How would you modify Strassen's algorithm to multiply n [image] n matrices in which n is not an exact power of 2? Show that the resulting algorithm runs in time [image] (n1g 7).

31.2-3

What is the largest k such that if you can multiply 3 [image] 3 matrices using k multiplications (not assuming commutativity of multiplication), then you can multiply n [image] n matrices in time o(n1g 7)? What would the running time of this algorithm be?

31.2-4

V. Pan has discovered a way of multiplying 68 [image] 68 matrices using 132,464 multiplications, a way of multiplying 70 [image] 70 matrices using 143,640 multiplications, and a way of multiplying 72 [image] 72 matrices using 155,424 multiplications. Which method yields the best asymptotic running time when used in a divide-and-conquer matrix-multiplication algorithm? Compare it with the running time for Strassen's algorithm.

131.2-5

How quickly can you multiply a kn [image] n matrix by an n [image] kn matrix, using Strassen's algorithm as a subroutine? Answer the same question with the order of the input matrices reversed.

31.2-6

Show how to multiply the complex numbers a + bi and c + di using only three real multiplications. The algorithm should take a, b, c, and d as input and produce the real component ac - bd and the imaginary component ad + bc separately.

* 31.3 Algebraic number systems and boolean matrix multiplication

The properties of matrix addition and multiplication depend on the properties of the underlying number system. In this section, we define three different kinds of underlying number systems: quasirings, rings, and fields. We can define matrix multiplication over quasirings, and Strassen's matrix-multiplication algorithm works over rings. We then present a simple trick for reducing boolean matrix multiplication, which is defined over a quasiring that is not a ring, to multiplication over a ring. Finally, we discuss why the properties of a field cannot naturally be exploited to provide better algorithms for matrix multiplication.

Quasirings

Let [image] denote a number system, where S is a set of elements, [image] and [image] are binary operations on S (the addition and multiplication operations, respectively), and [image] are distinct distinguished elements of S. This system is a quasiring if it satisfies the following properties:

1. is a monoid:

S is closed under [image]; that is, a [image] b [image] S for all a, b [image] S.

is associative; that is, a [image] (b [image] c) = (a [image] b) [image] c for all a, b, c [image] S.

is an identity for [image]; that is, [image] for all a [image] S.

Likewise, [image] is a monoid.

2. is an annihilator, that is, [image] for all a [image] S.

3. The operator [image] is commutative; that is, a [image] b = b [image] a for all a, b [image] S.

4. The operator [image] distributes over [image]; that is, [image] and [image] for all a,b,c [image] S.

Examples of quasirings include the boolean quasiring ({0,1}, [image] , [image] , 0, 1), where [image] denotes logical OR and [image] denotes logical AND, and the natural number system (N, +, [image], 0, 1), where + and [image] denote ordinary addition and multiplication. Any closed semiring (see Section 26.4) is also a quasiring; closed semirings obey additional idempotence and infinite-sum properties.

We can extend [image] and [image] to matrices as we did for + and [image] in Section 31.1. Denoting the n [image] n identity matrix composed of [image] we find that matrix multiplication is well defined and the matrix system is itself a quasiring, as the following theorem states.

Theorem 31.7

If [image] is a quasiring and n [image] 1, then [image] is a quasiring.

Proof The proof is left as Exercise 31.3-3.

Rings

Subtraction is not defined for quasirings, but it is for a ring, which is a quasiring [image] that satisfies the following additional property:

5. Every element in S has an additive inverse; that is, for all a [image] S, there exists an element b [image] S such that [image]. Such a b is also called the negative of a and is denoted (-a).

Given that the negative of any element is defined, we can define subtraction by a - b = a + (-b).

There are many examples of rings. The integers (Z, +, [image], 0, 1) under the usual operations of addition and multiplication form a ring. The integers modulo n for any integer n > 1--that is, (Z n, +, [image], 0, 1), where + is addition modulo n and [image] is multiplication modulo n--form a ring. Another example is the set R[x] of finite-degree polynomials in x with real coefficients under the usual operations--that is, (R[x], +, [image], 0, 1), where + is polynomial addition and [image] is polynomial multiplication.

The following corollary shows that Theorem 31.7 generalizes naturally to rings.

Corollary 31.8

If [image] is a ring and n [image] 1, then [image] is a ring.

Proof The proof is left as Exercise 31.3-3.

Using this corollary, we can prove the following theorem.

Theorem 31.9

Strassen's matrix-multiplication algorithm works properly over any ring of matrix elements.

Proof Strassen's algorithm depends on the correctness of the algorithm for 2 [image] 2 matrices, which requires only that the matrix elements belong to a ring. Since the matrix elements do belong to a ring, Corollary 31.8 implies the matrices themselves form a ring. Thus, by induction, Strassen's algorithm works correctly at each level of recursion.

Strassen's algorithm for matrix multiplication, in fact, depends critically on the existence of additive inverses. Out of the seven products P1, P2, . . . ,P7, four involve differences of submatrices. Thus, Strassen's algorithm does not work in general for quasirings.

Boolean matrix multiplication

Strassen's algorithm cannot be used directly to multiply boolean matrices, since the boolean quasiring ({0,1}, [image] , [image] , 0, 1) is not a ring. There are instances in which a quasiring is contained in a larger system that is a ring. For example, the natural numbers (a quasiring) are a subset of the integers (a ring), and Strassen's algorithm can therefore be used to multiply matrices of natural numbers if we consider the underlying number system to be the integers. Unfortunately, the boolean quasiring cannot be extended in a similar way to a ring. (See Exercise 31.3-4.)

The following theorem presents a simple trick for reducing boolean matrix multiplication to multiplication over a ring. Problem 31-1 presents another efficient approach.

Theorem 31.10

If M(n) denotes the number of arithmetic operations required to multiply two n [image] n matrices over the integers, then two n [image] n boolean matrices can be multiplied using O(M(n)) arithmetic operations.

Proof Let the two matrices be A and B, and let C = AB in the boolean quasiring, that is,

Instead of computing over the boolean quasiring, we compute the product C' over the ring of integers with the given matrix-multiplication algorithm that uses M(n) arithmetic operations. We thus have

Each term aikbkj of this sum is 0 if and only if aik [image] bkj = 0, and 1 if and only if aik [image] bkj = 1. Thus, the integer sum [image] is 0 if and only if every term is 0 or, equivalently, if and only if the boolean OR of the terms, which is cij, is 0. Therefore, the boolean matrix C can be reconstructed with [image] (n2) arithmetic operations from the integer matrix C' by simply comparing each [image] with 0. The number of arithmetic operations for the entire procedure is then O(M(n)) + [image] (n2) = O(M(n)), since M(n) = [image] (n2).

Thus, using Strassen's algorithm, we can perform boolean matrix multiplication in O(n lg 7) time.

The normal method of multiplying boolean matrices uses only boolean variables. If we use this adaptation of Strassen's algorithm, however, the final product matrix can have entries as large as n, thus requiring a computer word to store them rather than a single bit. More worrisome is that the intermediate results, which are integers, may grow even larger. One method for keeping intermediate results from growing too large is to perform all computations modulo n + 1. Exercise 31.3-5 asks you to show that working modulo n + 1 does not affect the correctness of the algorithm.

Fields

A ring [image] is a field if it satisfies the following two additional properties:

6. The operator [image] is commutative; that is, [image] for all a, b [image] S.

7. Every nonzero element in S has a multiplicative inverse; that is, for all [image], there exists an element b [image] S such that [image].

Such an element b is often called the inverse of a and is denoted a-1.

Examples of fields include the real numbers (R, +, [image], 0, 1), the complex numbers (C, +, [image], 0, 1), and the integers modulo a prime p: (Z p, +, [image], 0, 1).

Because fields offer multiplicative inverses of elements, division is possible. They also offer commutativity. By generalizing from quasirings to rings, Strassen was able to improve the running time of matrix multiplication. Since the underlying elements of matrices are often from a field--the real numbers, for instance--one might hope that by using fields instead of rings in a Strassen-like recursive algorithm, the running time might be further improved.

This approach seems unlikely to be fruitful. For a recursive divide-and-conquer algorithm based on fields to work, the matrices at each step of the recursion must form a field. Unfortunately, the natural extension of Theorem 31.7 and Corollary 31.8 to fields fails badly. For n > 1, the set of n [image] n matrices never forms a field, even if the underlying number system is a field. Multiplication of n [image] n matrices is not commutative, and many n [image] n matrices do not have inverses. Better algorithms for matrix multiplication are therefore more likely to be based on ring theory than on field theory.

Exercises

31.3-1

Does Strassen's algorithm work over the number system (Z[x], +, [image], 0, 1), where Z[x] is the set of all polynomials with integer coefficients in the variable x and + and [image] are ordinary polynomial addition and multiplication?

31.3-2

Explain why Strassen's algorithm doesn't work over closed semirings (see Section 26.4) or over the boolean quasiring ({0, 1}, [image] , [image] , 0, 1).

31.3-3

Prove Theorem 31.7 and Corollary 31.8.

31.3-4

Show that the boolean quasiring ({0, 1}, [image] , [image] , 0, 1) cannot be embedded in a ring. That is, show that it is impossible to add a "-1" to the quasiring so that the resulting algebraic structure is a ring.

31.3-5

Argue that if all computations in the algorithm of Theorem 31.10 are performed modulo n + 1, the algorithm still works correctly.

31.3-6

Show how to determine efficiently if a given undirected input graph contains a triangle (a set of three mutually adjacent vertices).

31.3-7

Show that computing the product of two n [image] n boolean matrices over the boolean quasiring is reducible to computing the transitive closure of a given directed 3n-vertex input graph.

31.3-8

Show how to compute the transitive closure of a given directed n-vertex input graph in time O(nlg 7 lg n). Compare this result with the performance of the TRANSITIVE-CLOSURE procedure in Section 26.2.

31.4 Solving systems of linear equations

Solving a set of simultaneous linear equations is a fundamental problem that occurs in diverse applications. A linear system can be expressed as a matrix equation in which each matrix or vector element belongs to a field, typically the real numbers R. This section discusses how to solve a system of linear equations using a method called LUP decomposition.

We start with a set of linear equations in n unknowns x1, x2, . . . , xn:

a11x1 + a12x2 +          + a1nxn = b1,

a21x1 + a22x2 +          + a2nxn = b2,

(31.17)

an1x1 + an2x2 +          + annxn = bn.

A set of values for x1, x2, . . . , xn that satisfy all of the equations (31.17) simultaneously is said to be a solution to these equations. In this section, we only treat the case in which there are exactly n equations in n unknowns.

We can conveniently rewrite equations (31.17) as the matrix-vector equation

or, equivalently, letting A = (aij), x = (xj), and b = (bi), as

Ax = b .

(31.18)

If A is nonsingular, it possesses an inverse A-1, and

x = A-1 b

(31.19)

is the solution vector. We can prove that x is the unique solution to equation (31.18) as follows. If there are two solutions, x and x', then Ax = Ax' = b and

x  =  (A-1 A)x

=  A-1(Ax)

=  A-1(Ax')

=  (A-1 A)x'

=  x'.

In this section, we shall be concerned predominantly with the case in which A is nonsingular or, equivalently (by Theorem 31.1), the rank of A is equal to the number n of unknowns. There are other possibilities, however, which merit a brief discussion. If the number of equations is less than the number n of unknowns--or, more generally, if the rank of A is less than n--then the system is underdetermined. An underdetermined system typically has infinitely many solutions (see Exercise 31.4-9), although it may have no solutions at all if the equations are inconsistent. If the number of equations exceeds the number n of unknowns, the system is overdetermined, and there may not exist any solutions. Finding good approximate solutions to overdetermined systems of linear equations is an important problem that is addressed in Section 31.6.

Let us return to our problem of solving the system Ax = b of n equations in n unknowns. One approach is to compute A-1 and then multiply both sides by A-1, yielding A-1Ax = A-1b, or x = A-1b. This approach suffers in practice from numerical instability: round-off errors tend to accumulate unduly when floating-point number representations are used instead of ideal real numbers. There is, fortunately, another approach--LUP decomposition--that is numerically stable and has the further advantage of being about a factor of 3 faster.

Overview of LUP decomposition

The idea behind LUP decomposition is to find three n [image] n matrices L, U, and P such that

PA = LU ,

(31.20)

where

L is a unit lower-triangular matrix,

U is an upper-triangular matrix, and

P is a permutation matrix.

We call matrices L, U, and P satisfying equation (31.20) an LUP decomposition of the matrix A. We shall show that every nonsingular matrix A possesses such a decomposition.

The advantage of computing an LUP decomposition for the matrix A is that linear systems can be solved more readily when they are triangular, as is the case for both matrices L and U. Having found an LUP decomposition for A, we can solve the equation (31.18) Ax = b by solving only triangular linear systems, as follows. Multiplying both sides of Ax = b by P yields the equivalent equation P Ax = Pb, which by Exercise 31.1-2 amounts to permuting the equations (31.17). Using our decomposition (31.20), we obtain

LUx = Pb .

We can now solve this equation by solving two triangular linear systems. Let us define y = Ux, where x is the desired solution vector. First, we solve the lower-triangular system

Ly = Pb

(31.21)

for the unknown vector y by a method called "forward substitution." Having solved for y, we then solve the upper-triangular system

Ux = y

(31.22)

for the unknown x by a method called "back substitution." The vector x is our solution to Ax = b, since the permutation matrix P is invertible (Exercise 31.1 -2):

Ax  =  P-1 LUx

=  P-1 Ly

=  P-1 Pb

=  b .

Our next step is to show how forward and back substitution work and then attack the problem of computing the LUP decomposition itself.

Forward and back substitution

Forward substitution can solve the lower-triangular system (31.21) in [image] (n2) time, given L, P, and b. For convenience, we represent the permutation P compactly by an array [image] [1 . . n]. For i = 1, 2, . . . , n, the entry [image] [i] indicates that Pi, [image] [i] = 1 and Pij = 0 for j [image] [image] [i]. Thus, PA has a [image] [i],j in row i and column j, and Pb has b [image] [i] as its ith element. Since L is unit lower-triangular, equation (31.21) can be rewritten as

y1                             =  b  [1],

l21y1 + y2                        =  b  [2],

l31y1 + l32y2 + y3                =  b  [3],

ln1y1 + ln2y2 + ln3y3 +          + yn  =  b  [n].

Quite apparently, we can solve for y1 directly, since the first equation tells us that y1 = b [image] [1]. Having solved for y1, we can substitute it into the second equation, yielding

y2 = b  [2] - l21y1.

Now, we can substitute both y1 and y2 into the third equation, obtaining

y3 = b  [3] - (l31y1 + l32y2).

In general, we substitute y1, y2, . . . ,yi-1 "forward" into the ith equation to solve for yi:

Back substitution is similar to forward substitution. Given U and y, we solve the nth equation first and work backward to to the first equation. Like forward substitution, this process runs in [image] (n2) time. Since U is upper-triangular, we can rewrite the system (31.22) as

u11x1 + u12x2 +          +   u1,n-2xn-2 +   u1,n-1xn-1  +    u1nxn = y1,

u22x2 +          +   u2,n-2xn-2 +   u2,n-1xn-1  +     u2nxn = y2,

un-2,n-2xn-2 + un-2,n-1xn-1 + un-2,nxn = yn-2,

un-1,n-1xn-1 + un-1,nxn = yn-1,

     un,nxn = yn .

Thus, we can solve for xn, xn-1, . . . , x1 successively as follows:

xn = yn/unn ,

xn-1 = (yn-1 - un-1,nxn)/un-1,n-1 ,

xn-2 = (yn-2 - (un-2,n-1xn-1 + un-2,nxn))/un-2,n-2 ,

or, in general,

Given P, L, U, and b, the procedure LUP-SOLVE solves for x by combining forward and back substitution. The pseudocode assumes that the dimension n appears in the attribute rows[L] and that the permutation matrix P is represented by the array [image].

Procedure LUP-SOLVE solves for y using forward substitution in lines 2-3, and then it solves for x using backward substitution in lines 4-5. Since there is an implicit loop in the summations within each of the for loops, the running time is [image] (n2).

As an example of these methods, consider the system of linear equations defined by

where

and we wish to solve for the unknown x. The LUP decomposition is

(The reader can verify that PA = LU.) Using forward substitution, we solve Ly = Pb for y:

obtaining

by computing first y1, then y2, and finally y3. Using back substitution, we solve Ux = y for x:

thereby obtaining the desired answer

by computing first x3, then x2, and finally x1.

Computing an LU decomposition

We have now shown that if an LUP decomposition can be computed for a nonsingular matrix A, forward and back substitution can be used to solve the system Ax = b of linear equations. It remains to show how an LUP decomposition for A can be found efficiently. We start with the case in which A is an n [image] n nonsingular matrix and P is absent (or, equivalently, P = In). In this case, we must find a factorization A = LU. We call the two matrices L and U an LU decomposition of A.

The process by which we perform LU decomposition is called Gaussian elimination. We start by subtracting multiples of the first equation from the other equations so that the first variable is removed from those equations. Then, we subtract multiples of the second equation from the third and subsequent equations so that now the first and second variables are removed from them. We continue this process until the system that is left has an upper-triangular form--in fact, it is the matrix U. The matrix L is made up of the row multipliers that cause variables to be eliminated.

Our algorithm to implement this strategy is recursive. We wish to construct an LU decomposition for an n [image] n nonsingular matrix A. If n = 1, then we're done, since we can choose L = I1 and U = A. For n > 1, we break A into four parts:

where v is a size-(n - 1) column vector, wT is a size-(n - 1) row vector, and A' is an (n - 1) [image] (n - 1) matrix. Then, using matrix algebra (verify the equations by simply multiplying through), we can factor A as

The 0's in the first and second matrices of the factorization are row and column vectors, respectively, of size n - 1. The term vwT/a11, formed by taking the outer product of v and w and dividing each element of the result by a11, is an (n - 1) [image] (n - 1) matrix, which conforms in size to the matrix A' from which it is subtracted. The resulting (n - 1) [image] (n - 1) matrix

A' - vwT/a11

(31.23)

is called the Schur complement of A with respect to a11.

We now recursively find an LU decomposition of the Schur complement. Let us say that

A' - vwT/a11 = L'U' ,

where L' is unit lower-triangular and U' is upper-triangular. Then, using matrix algebra, we have

thereby providing our LU decomposition. (Note that because L' is unit lower-triangular, so is L, and because U'is upper-triangular, so is U.)

Of course, if a11 = 0, this method doesn't work, because it divides by 0. It also doesn't work if the upper leftmost entry of the Schur complement A' - vwT/a11 is 0, since we divide by it in the next step of the recursion. The elements by which we divide during LU decomposition are called pivots, and they occupy the diagonal elements of the matrix U. The reason we include a permutation matrix P during LUP decomposition is that it allows us to avoid dividing by zero elements. Using permutations to avoid division by 0 (or by small numbers) is called pivoting.

An important class of matrices for which LU decomposition always works correctly is the class of symmetric positive-definite matrices. Such matrices require no pivoting, and thus the recursive strategy outlined above can be employed without fear of dividing by 0. We shall prove this result, as well as several others, in Section 31.6.

Our code for LU decomposition of a matrix A follows the recursive strategy, except that an iteration loop replaces the recursion. (This transformation is a standard optimization for a "tail-recursive" procedure--one whose last operation is a recursive call to itself.) It assumes that the dimension of A is kept in the attribute rows[A]. Since we know that the output matrix U has 0's below the diagonal, and since LU-SOLVE does not look at these entries, the code does not bother to fill them in. Likewise, because the output matrix L has 1's on its diagonal and 0's above the diagonal, these entries are not filled in either. Thus, the code computes only the "significant" entries of L and U.

The outer for loop beginning in line 2 iterates once for each recursive step. Within this loop, the pivot is determined to be ukk = akk in line 3. Within the for loop in lines 4-6 (which does not execute when k = n), the v and wT vectors are used to update L and U. The elements of the v vector are determined in line 5, where vi is stored in lik, and the elements of the wT vector are determined in line 6, where wiT is stored in uki. Finally, the elements of the Schur complement are computed in lines 7-9 and stored back in the matrix A. Because line 9 is triply nested, LU-DECOMPOSITION runs in time [image] (n3).

Figure 31.1 The operation of LU-DECOMPOSITION. (a) The matrix A. (b) The element a11 = 2 in black is the pivot, the shaded column is v/a11, and the shaded row is wT. The elements of U computed thus far are above the horizontal line, and the elements of L are to the left of the vertical line. The Schur complement matrix A' - vwT/a11 occupies the lower right. (c) We now operate on the Schur complement matrix produced from part (b). The element a22 = 4 in black is the pivot, and the shaded column and row are v/a22 and wT (in the partitioning of the Schur complement), respectively. Lines divide the matrix into the elements of U computed so far (above), the elements of L computed so far (left), and the new Schur complement (lower right). (d) The next step completes the factorization. (The element 3 in the new Schur complement becomes part of U when the recursion terminates.) (e) The factorization A = LU.

Figure 31.1 illustrates the operation of LU-DECOMPOSITION. It shows a standard optimization of the procedure in which the significant elements of L and U are stored "in place" in the matrix A. That is, we can set up a correspondence between each element aij and either lij (if i > j) or uij (if i [image] j) and update the matrix A so that it holds both L and U when the procedure terminates. The pseudocode for this optimization is obtained from the above pseudocode merely by replacing each reference to l or u by a; it is not difficult to verify that this transformation preserves correctness.

Computing an LUP decomposition

Generally, in solving a system of linear equations Ax = b, we must pivot on off-diagonal elements of A to avoid dividing by 0. Not only is division by 0 undesirable, so is division by any small value, even if A is nonsingular, because numerical instabilities can result in the computation. We therefore try to pivot on a large value.

The mathematics behind LUP decomposition is similar to that of LU decomposition. Recall that we are given an n [image] n nonsingular matrix A and wish to find a permutation matrix P, a unit lower-triangular matrix L, and an upper-triangular matrix U such that PA = LU. Before we partition the matrix A, as we did for LU decomposition, we move a nonzero element, say ak1, from the first column to the (1,1) position of the matrix. (If the first column contains only 0's, then A is singular, because its determinant is 0, by Theorems 31.4 and 31.5.) In order to preserve the set of equations, we exchange row 1 with row k, which is equivalent to multiplying A by a permutation matrix Q on the left (Exercise 31.1-2). Thus, we can write QA as

where v = (a21, a31, . . . , an1)T, except that a11 replaces ak1; wT = (ak2, ak3, . . . , akn); and A' is an (n - 1) [image] (n - 1) matrix. Since ak1 [image] 0, we can now perform much the same linear algebra as for LU decomposition, but now guaranteeing that we do not divide by 0:

The Schur complement A' - vwT/ak1 is nonsingular, because otherwise the second matrix in the last equation has determinant 0, and thus the determinant of matrix A is 0; but this means that A is singular, which contradicts our assumption that A is nonsingular. Consequently, we can inductively find an LUP decomposition for the Schur complement, with unit lower-triangular matrix L', upper-triangular matrix U', and permutation matrix P', such that

P'(A' - vwT/ak1) = L'U' .

Define

which is a permutation matrix, since it is the product of two permutation matrices (Exercise 31.1-2). We now have

yielding the LUP decomposition. Because L' is unit lower-triangular, so is L, and because U' is upper-triangular, so is U.

Notice that in this derivation, unlike the one for LU decomposition, both the column vector v/ak1 and the Schur complement A' - vwT/ak1 must be multiplied by the permutation matrix P'.

Like LU-DECOMPOSITION, our pseudocode for LUP decomposition replaces the recursion with an iteration loop. As an improvement over a direct implementation of the recursion, we dynamically maintain the permutation matrix P as an array [image], where [image] [i] = j means that the ith row of P contains a 1 in column j. We also implement the code to compute L and U "in place" in the matrix A. Thus, when the procedure terminates,

LUP-DECOMPOSITION(A)

1  n    rows[A]

2  for i    1 to n

3       do   [i]    i

4  for k    1 to n - 1

5       do p    0

6          for i    k to n

7              do if |aik| > p

8                    then p    |aik|

9                         k'    i

10          if p = 0

11             then error "singular matrix"

12          exchange   [k]      [k']

13          for i    1 to n

14               do exchange aki    ak'i

15          for i    k + 1 to n

16               do aik    aik/akk

17                  for j    k + 1 to n

18                       do aij    aij - aikakj

Figure 31.2 illustrates how LUP-DECOMPOSITION factors a matrix. The array [image] is initialized by lines 2-3 to represent the identity permutation. The outer for loop beginning in line 4 implements the recursion. Each time through the outer loop, lines 5-9 determine the element ak'k with largest absolute value of those in the current first column (column k) of the (n - k + l ) (n - k + 1) matrix whose LU decomposition must be found.

Figure 31.2 The operation of LUP-DECOMPOSITION. (a) The input matrix A with the identity permutation of the rows on the left. The first step of the algorithm determines that the element 5 in black in the third row is the pivot for the first column. (b) Rows 1 and 3 are swapped and the permutation is updated. The shaded column and row represent v and wT. (c) The vector v is replaced by v/5, and the the lower right of the matrix is updated with the Schur complement. Lines divide the matrix into three regions: elements of U (above), elements of L (left), and elements of the Schur complement (lower right).(d)-(f) The second step.(g)-(i) The third step finishes the algorithm. (j) The LUP decomposition PA = LU.

If all elements in the current first column are zero, lines 10-11 report that the matrix is singular. To pivot, we exchange [k'] with [k] in line 12 and exchange the kth and k'th rows of A in lines 13-14, thereby making the pivot element akk. (The entire rows are swapped because in the derivation of the method above, not only is A' - vwT/ak1 multiplied by P', but so is v/ak1.) Finally, the Schur complement is computed by lines 15-18 in much the same way as it is computed by lines 4-9 of LU-DECOMPOSITION, except that here the operation is written to work "in place."

Because of its triply nested loop structure, the running time of LUP-DECOMPOSITION is (n3), the same as that of LU-DECOMPOSITION. Thus, pivoting costs us at most a constant factor in time.

Exercises

31.4-1

Solve the equation

by using forward substitution.

31.4-2

Find an LU decomposition of the matrix

31.4-3

Why does the for loop in line 4 of LUP-DECOMPOSITION run only up to n - 1, whereas the corresponding for loop in line 2 of LU-DECOMPOSITION runs all the way to n?

31.4-4

Solve the equation

by using an LUP decomposition.

31.4-5

Describe the LUP decomposition of a diagonal matrix.

31.4-6

Describe the LUP decomposition of a permutation matrix A, and prove that it is unique.

31.4-7

Show that for all n 1, there exist singular n n matrices that have LU decompositions.

31.4-8

Show how we can efficiently solve a set of equations of the form Ax = b over the boolean quasiring ({0, 1}, , , 0, 1).

31.4-9

Suppose that A is an m n real matrix of rank m, where m < n. Show how to find a size-n vector x0 and an m (n - m) matrix B of rank n - m such that every vector of the form x0 + By, for y R n-m, is a solution to the underdetermined equation Ax = b.

31.5 Inverting matrices

Although in practice we do not generally use matrix inverses to solve systems of linear equations, preferring instead to use more numerically stable techniques such as LUP decomposition, it is sometimes necessary to compute a matrix inverse. In this section, we show how LUP decomposition can be used to compute a matrix inverse. We also discuss the theoretically interesting question of whether the computation of a matrix inverse can be sped up using techniques such as Strassen's algorithm for matrix multiplication. Indeed, Strassen's original paper was motivated by the problem of showing that a set of a linear equations could be solved more quickly than by the usual method.

Computing a matrix inverse from an LUP decomposition

Suppose that we have an LUP decomposition of a matrix A in the form of three matrices L, U, and P such that PA = LU. Using LU-SOLVE, we can solve an equation of the form Ax = b in time (n2). Since the LUP decomposition depends on A but not b, we can solve a second set of equations of the form Ax = b' in additional time (n2). In general, once we have the LUP decomposition of A, we can solve, in time (kn2), k versions of the equation Ax = b that differ only in b.

The equation

AX = In

(31.24)

can be viewed as a set of n distinct equations of the form Ax = b. These equations define the matrix X as the inverse of A. To be precise, let Xi denote the ith column of X, and recall that the unit vector ei is the ith column of In. Equation (31.24) can then be solved for X by using the LUP decomposition for A to solve each equation

AXi = ei

separately for Xi. Each of the n Xi can be found in time (n2), and so the computation of X from the LUP decomposition of A takes time (n3). Since the LUP decomposition of A can be computed in time (n3), the inverse A-1 of a matrix A can be determined in time (n3).

Matrix multiplication and matrix inversion

We now show that the theoretical speedups obtained for matrix multiplication translate to speedups for matrix inversion. In fact, we prove something stronger: matrix inversion is equivalent to matrix multiplication, in the following sense. If M(n) denotes the time to multiply two n n matrices and I(n) denotes the time to invert a nonsingular n n matrix, then I(n) = (M(n)). We prove this result in two parts. First, we show that M(n) = O(I(n)), which is relatively easy, and then we prove that I(n) = O(M(n)).

Theorem 31.11

If we can invert an n n matrix in time I(n), where I(n) = (n2) and I(n) satisfies the regularity condition I(3n) = O(I(n)), then we can multiply two n n matrices in time O(I(n)).

Proof Let A and B be n n matrices whose matrix product C we wish to compute. We define the 3n 3n matrix D by

The inverse of D is

and thus we can compute the product AB by taking the upper right n n submatrix of D-1.

We can construct matrix D in (n2) = O(I(n)) time, and we can invert D in O(I(3n)) = O(I(n)) time, by the regularity condition on I(n). We thus have

M(n) = O(I(n)).

Note that I(n) satisfies the regularity condition whenever I(n) does not have large jumps in value. For example, if I(n) = (nc lgd n) for any constants c > 0, d 0, then I(n) satisfies the regularity condition.

Reducing matrix inversion to matrix multiplication

The proof that matrix inversion is no harder than matrix multiplication relies on some properties of symmetric positive-definite matrices that will be proved in Section 31.6.

Theorem 31.12

If we can multiply two n n real matrices in time M(n), where M(n) = (n2) and M(n) = O(M(n + k)) for 0 k n, then we can compute the inverse of any real nonsingular n n matrix in time O(M(n)).

Proof We can assume that n is an exact power of 2, since we have

for any k > 0. Thus, by choosing k such that n + k is a power of 2, we enlarge the matrix to a size that is the next power of 2 and obtain the desired answer A-1 from the answer to the enlarged problem. The regularity condition on M(n) ensures that this enlargement does not cause the running time to increase by more than a constant factor.

For the moment, let us assume that the n n matrix A is symmetric and positive-definite. We partition A into four n/2 n/2 submatrices:

(31.25)

Then, if we let

S = D - CB-1 CT

(31.26)

be the Schur complement of A with respect to B, we have

(31.27)

since AA-1 = In, as can be verified by performing the matrix multiplication. The matrices B-1 and S-1 exist if A is symmetric and positive-definite, by Lemmas 31.13, 31.14, and 31.15 in Section 31.6, because both B and S are symmetric and positive-definite. By Exercise 31.1-3, B-1CT = (CB-1)T and B-1CTS-1 = (S-1 CB-1)T. Equations (31.26) and (31.27) can therefore be used to specify a recursive algorithm involving 4 multiplications of n/2 n/2 matrices:

C    B-1,

(CB-1)    CT,

S-1    (CB-1),

(CB-1)T    (S-1 CB-1).

Since we can multiply n/2 n/2 matrices using an algorithm for n n matrices, matrix inversion of symmetric positive-definite matrices can be performed in time

I(n)    2I(n/2) + 4M(n) + O(n2)

= 2I(n/2) + O(M(n))

= O(M(n)).

It remains to prove that the asymptotic running time of matrix multiplication can be obtained for matrix inversion when A is invertible but not symmetric and positive-definite. The basic idea is that for any nonsingular matrix A, the matrix ATA is symmetric (by Exercise 31.1-3) and positive-definite (by Theorem 31.6). The trick, then, is to reduce the problem of inverting A to the problem of inverting ATA.

The reduction is based on the observation that when A is an n n nonsingular matrix, we have

A-1 = (ATA)-1AT,

since ((ATA)-1 AT)A = (ATA)-1(ATA) = In and a matrix inverse is unique. Therefore, we can compute A-1 by first multiplying AT by A to obtain ATA, then inverting the symmetric positive-definite matrix ATA using the above divide-and-conquer algorithm, and finally multiplying the result by AT. Each of these three steps takes O(M(n)) time, and thus any nonsingular matrix with real entries can be inverted in O(M(n)) time.

The proof of Theorem 31.12 suggests a means of solving the equation Ax = b without pivoting, so long as A is nonsingular. We multiply both sides of the equation by AT, yielding (ATA)x = ATb. This transformation doesn't affect the solution x, since AT is invertible, so we can factor the symmetric positive-definite matrix ATA by computing an LU decomposition. We then use forward and back substitution to solve for x with the right-hand side ATb. Although this method is theoretically correct, in practice the procedure LUP-DECOMPOSITION works much better. LUP decomposition requires fewer arithmetic operations by a constant factor, and it has somewhat better numerical properties.

Exercises

31.5-1

Let M(n) be the time to multiply n n matrices, and let S(n) denote the time required to square an n n matrix. Show that multiplying and squaring matrices have essentially the same difficulty: S(n) = (M(n)).

31.5-2

Let M(n) be the time to multiply n n matrices, and let L(n) be the time to compute the LUP decomposition of an n n matrix. Show that multiplying matrices and computing LUP decompositions of matrices have essentially the same difficulty: L(n) = (M(n)).

31.5-3

Let M(n) be the time to multiply n n matrices, and let D(n) denote the time required to find the determinant of an n n matrix. Show that finding the determinant is no harder than multiplying matrices: D(n) = O(M(n)).

31.5-4

Let M(n) be the time to multiply n n boolean matrices, and let T(n) be the time to find the transitive closure of n n boolean matrices. Show that M(n) = O(T(n)) and T(n) = O(M(n)lg n).

31.5-5

Does the matrix-inversion algorithm based on Theorem 31.12 work when matrix elements are drawn from the field of integers modulo 2? Explain.

31.5-6

Generalize the matrix-inversion algorithm of Theorem 31.12 to handle matrices of complex numbers, and prove that your generalization works correctly. (Hint: Instead of the transpose of A, use the conjugate transpose A*, which is obtained from the transpose of A by replacing every entry with its complex conjugate. Instead of symmetric matrices, consider Hermitian matrices, which are matrices A such that A = A*.)

31.6 Symmetric positive-definite matrices and least-squares approximation

Symmetric positive-definite matrices have many interesting and desirable properties. For example, they are nonsingular, and LU decomposition can be performed on them without our having to worry about dividing by 0. In this section, we shall prove several other important properties of symmetric positive-definite matrices and show an interesting application to curve fitting by a least-squares approximation.

The first property we prove is perhaps the most basic.

Lemma 31.13

Any symmetric positive-definite matrix is nonsingular.

Proof Suppose that a matrix A is singular. Then by Corollary 31.3, there exists a nonzero vector x such that Ax = 0. Hence, xTAx = 0, and A cannot be positive-definite.

The proof that we can perform LU decomposition on a symmetric positive-definite matrix A without dividing by 0 is more involved. We begin by proving properties about certain submatrices of A. Define the kth leading submatrix of A to be the matrix Ak consisting of the intersection of the first k rows and first k columns of A.

Lemma 31.14

If A is a symmetric positive-definite matrix, then every leading submatrix of A is symmetric and positive-definite.

Proof That each leading submatrix Ak is symmetric is obvious. To prove that Ak is positive-definite, let x be a nonzero column vector of size k, and let us partition A as follows:

Then, we have

since A is positive-definite, and hence Ak is also positive-definite.

We now turn to some essential properties of the Schur complement. Let A be a symmetric positive-definite matrix, and let Ak be a leading k k submatrix of A. Partition A as

(31.28)

Then, the Schur complement of A with respect to Ak is defined to be

(31.29)

(By Lemma 31.14, Ak is symmetric and positive-definite; therefore, exists by Lemma 31.13, and S is well defined.) Note that our earlier definition (31.23) of the Schur complement is consistent with definition (31.29), by letting k = 1.

The next lemma shows that the Schur-complement matrices of symmetric positive-definite matrices are themselves symmetric and positive-definite. This result was used in Theorem 31.12, and its corollary is needed to prove the correctness of LU decomposition for symmetric positive-definite matrices.

Lemma 31.15

If A is a symmetric positive-definite matrix and Ak is a leading data k k submatrix of A, then the Schur complement of A with respect to Ak is symmetric and positive-definite.

Proof That S is symmetric follows from Exercise 31.1-7. It remains to show that S is positive-definite. Consider the partition of A given in equation (31.28).

For any nonzero vector x, we have xTAx > 0 by assumption. Let us break x into two subvectors y and z compatible with Ak and C, respectively. Because Ak-1 exists, we have

(31.30)

by matrix magic. (Verify by multiplying through.) This last equation amounts to "completing the square" of the quadractic form. (See Exercise 31.6-2.)

Since xTAx > 0 holds for any nonzero x, let us pick any nonzero z and then choose , which causes the first term in equation (31.30) to vanish, leaving

as the value of the expression. For any z 0, we therefore have zTSz = xTAx > 0, and thus S is positive-definite.

Corollary 31.16

LU decomposition of a symmetric positive-definite matrix never causes a division by 0.

Proof Let A be a symmetric positive-definite matrix. We shall prove something stronger than the statement of the corollary: every pivot is strictly positive. The first pivot is a11. Let e1 be the first unit vector, from which we obtain . Since the first step of LU decomposition produces the Schur complement of A with respect to A1 = (a11), Lemma 31.15 implies that all pivots are positive by induction.

Least-squares approximation

Fitting curves to given sets of data points is an important application of symmetric positive-definite matrices. Suppose that we are given a set of m data points

(x1,y1),(x2,y2),...,(xm,ym),

where the yi are known to be subject to measurement errors. We would like to determine a function F (x) such that

yi = F(xi) +   i,

(31.31)

for i = 1, 2, . . . , m, where the approximation errors i are small. The form of the function F depends on the problem at hand. Here, we assume that it has the form of a linearly weighted sum,

where the number of summands n and the specific basis functions fj are chosen based on knowledge of the problem at hand. A common choice is fj(x) = xj-1, which means that

F(x) = c1 + c2x + c3x2 +          + cnxn-1

is a polynomial of degree n - 1 in x.

By choosing n = m, we calculate each yi exactly in equation (31.31). Such a high-degree F "fits the noise" as well as the data, however, and generally gives poor results when used to predict y for previously unseen values of x. It is usually better to choose n significantly smaller than m and hope that by choosing the coefficients cj well, we can obtain a function F that finds the significantl patterns in the data points without paying undue attention to the noise. Some theoretical principles exist for choosing n, but they are beyond the scope of this text. In any case, once n is chosen, we end up with an overdetermined set of equations that we wish to solve as well as we can. We now show how this can be done.

Let

denote the matrix of values of the basis functions at the given points; that is, aij = fj(xi). Let c = (ck) denote the desired size-n vector of coefficients. Then,

is the size-m vector of "predicted values" for y. Thus,

   = Ac - y

is the size-m vector of approximation errors.

To minimize approximation errors, we choose to minimize the norm of the error vector , which gives us a least-squares solution, since

Since

we can minimize || || by differentiating || ||2 with respect to each ck and then setting the result to 0:

(31.32)

The n equations (31.32) for k = 1, 2, . . . , n are equivalent to the single matrix equation

(Ac -y)TA = 0

or, equivalently (using Exercise 31.1-3), to

AT(Ac - y) = 0 ,

which implies

ATAc = ATy .

(31.33)

In statistics, this is called the normal equation. The matrix AT A is symmetric by Exercise 31.1-3, and if A has full column rank, then ATA is positive-definite as well. Hence, (ATA)-1 exists, and the solution to equation (31.33) is

c = ((ATA)-1AT)y

= A+y ,

(31.34)

where the matrix A+ = ((ATA)-1AT) is called the pseudoinverse of the matrix A. The pseudoinverese is a natural generalization of the notion of a matrix inverse to the case in which A is nonsquare. (Compare equation (31.34) as the approximate solution to Ac = y with the solution A-1b as the exact solution to Ax = b.)

As an example of producing a least-squares fit, suppose that we have 5 data points

(-1,2),(1,1),(2,1),(3,0),(5,3),

shown as black dots in Figure 31.3. We wish to fit these points with a quadratic polynomial

F(x) = c1 + c2x + c3x2.

We start with the matrix of basis-function values

Figure 31.3 The least-squares fit of a quadratic polynomial to the set of data points {(-1, 2), (1, 1), (2, 1),(3, 0), (5, 3)}. The black dots are the data points, and the white dots are their estimated values predicted by the polynomial F(x) = 1.2 - 0.757x + 0.214x2, the quadratic polynomial that minimizes the sum of the squared errors. The error for each data point is shown as a shaded line.

whose pseudoinverse is

Multiplying y by A+, we obtain the coefficient vector

which corresponds to the quadratic polynomial

F(x) = 1.200 - 0.757x + 0.214x2

as the closest-fitting quadratic to the given data, in a least-squares sense.

As a practical matter, we solve the normal equation (31.33) by multiplying y by AT and then finding an LU decomposition of AT A. If A has full rank, the matrix AT A is guaranteed to be nonsingular, because it is symmetric and positive-definite. (See Exercise 31.1-3 and Theorem 31.6.)

Exercises

31.6-1

Prove that every diagonal element of a symmetric positive-definite matrix is positive.

31.6-2

Let be a 2 2 symmetric positive-definite matrix. Prove that its determinant ac - b2 is positive by "completing the square" in a manner similar to that used in the proof of Lemma 31.15.

31.6-3

Prove that the maximum element in a symmetric positive-definite matrix lies on the diagonal.

31.6-4

Prove that the determinant of each leading submatrix of a symmetric positive-definite matrix is positive.

31.6-5

Let Ak denote the kth leading submatrix of a symmetric positive-definite matrix A. Prove that det(Ak)/det(Ak-1) is the kth pivot during LU decomposition, where by convention det(A0) = 1.

31.6-6

Find the function of the form

F(x) = c1 + c2x lg x + c3ex

that is the best least-squares fit to the data points

(1,1),(2,1),(3,3),(4,8) .

31.6-7

Show that the pseudoinverse A+ satisfies the following four equations:

AA+ A =  A ,

A+ AA+  =  A+ ,

(AA+)T  =  AA+ ,

(A+A)T  =  A+A .

Problems

31-1 Shamir's boolean matrix multiplication algorithm

In Section 31.3, we observed that Strassen's matrix-multiplication algorithm cannot be applied directly to boolean matrix multiplication because the boolean quasiring Q = ({0, 1}, , , 0, 1) is not a ring. Theorem 31.10 showed that if we used arithmetic operations on words of O(lg n) bits, we could nevertheless apply Strassen's method to multiply n n boolean matrices in O(nlg 7) time. In this problem, we investigate a probabilistic method that uses only bit operations to achieve nearly as good a bound but with some small chance of error.

a. Show that R = ({0, 1}, , , 0, 1), where is the XOR (exclusive-or) function, is a ring.

Let A = (aij) and B = (bij) be n n boolean matrices, and let C = (cij) = AB in the quasiring Q. Generate A' = (a'ij) from A using the following randomized procedure:

If aij = 0, then let a'ij = 0.

If Aij = 1, then let a'ij = 1 with probability 1/2 and let a'ij = 0 with probability 1/2. The random choices for each entry are independent.

b. Let C' = (c'ij) = A'B in the ring R. Show that cij = 0 implies c'ij = 0. Show that cij = 1 implies c'ij = 1 with probability 1/2.

c. Show that for any > 0, the probability is at most /n2 that a given c'ij never takes on the value cij for lg(n2/ ) independent choices of the matrix A'. Show that the probability is at least 1 - that all c'ij take on their correct values at least once.

d. Give an O(nlg 7 lg n)-time randomized algorithm that computes the product in the boolean quasiring Q of two n n matrices with probability at least 1 - 1/nk for any constant k > 0. The only operations permitted on matrix elements are , , and .

31-2 Tridiagonal systems of linear equations

Consider the tridiagonal matrix

a. Find an LU decomposition of A.

b. Solve the equation Ax = (1 1 1 1 1)T by using forward and back substitution.

c. Find the inverse of A.

d. Show that for any n n symmetric, positive-definite, tridiagonal matrix A and any n-vector b, the equation Ax = b can be solved in O(n) time by performing an LU decomposition. Argue that any method based on forming A-1 is asymptotically more expensive in the worst case.

e. Show that for any n n nonsingular, tridiagonal matrix A and any n-vector b, the equation Ax = b can be solved in O(n) time by performing an LUP decomposition.

31-3 Splines

A practical method for interpolating a set of points with a curve is to use cubic splines. We are given a set {(xi, yi): i = 0,1, . . . , n} of n + 1 point-value pairs, where x0 < x1 < < xn. We wish to fit a piecewise-cubic curve (spline) f(x) to the points. That is, the curve f(x) is made up of n cubic polynomials fi(x) = ai + bix + cix2 + dix3 for i = 0,1, . . . , n - 1, where if x falls in the range xi x xi + 1, then the value of the curve is given by f(x) = fi(x - xi). The points xi at which the cubic polynomials are "pasted" together are called knots. For simplicity, we shall assume that xi = i for i = 0,1, . . . , n.

To ensure continuity of f(x), we require that

f(xi)   =  fi(0)  =  yi ,

f(xi+1) =  fi(1)  =  yi+1

for i = 0, 1, . . . , n - 1. To ensure that f(x) is sufficiently smooth, we also insist that there be continuity of the first derivative at each knot:

for i = 0, 1, . . . , n - 1.

a. Suppose that for i = 0, 1, . . . , n, we are given not only the point-value pairs {(xi, yi)} but also the first derivatives Di = f'(xi) at each knot. Express each coefficient ai, bi, ci, and di in terms of the values yi, yi+1, Di, and Di+1. (Remember that xi = i.) How quickly can the 4n coefficients be computed from the point-value pairs and first derivatives?

The question remains of how to choose the first derivatives of f(x) at the knots. One method is to require the second derivatives to be continuous at the knots:

for i = 0, 1, . . . ,n - 1. At the first and last knots, we assume that ; these assumptions make f(x) a natural cubic spline.

b. Use the continuity constraints on the second derivative to show that for i = 1, 2, . . . , n - 1,

Di - 1 + 4Di + Di + 1 = 3(yi + 1 - yi - 1) .

(31.35)

c. Show that

2D0 + D1 = 3(y1 - y0) ,

(31.36)

Dn - 1 + 2Dn = 3(yn - yn - 1) .

(31.37)

d. Rewrite equations (31.35)--(31.37) as a matrix equation involving the vector D = D0, D1, . . . , Dn of unknowns. What attributes does the matrix in your equation have?

e. Argue that a set of n + 1 point-value pairs can be interpolated with a natural cubic spline in O(n) time (see Problem 31-2).

f. Show how to determine a natural cubic spline that interpolates a set of n + 1 points (xi, yi) satisfying x0 < x1 < < xn, even when xi is not necessarily equal to i. What matrix equation must be solved, and how quickly does your algorithm run?

Chapter notes

There are many excellent texts available that describe numerical and scientific computation in much greater detail than we have room for here. The following are especially readable: George and Liu [81], Golub and Van Loan [89], Press, Flannery, Teukolsky, and Vetterling[161, 162], and Strang[181, 182].

The publication of Strassen's algorithm in 1969 [183] caused much excitement. Before then, it was hard to imagine that the naive algorithm could be improved upon. The asymptotic upper bound on the difficulty of matrix multiplication has since been considerably improved. The most asymptotically efficient algorithm for multiplying n n matrices to date, due to Coppersmith and Winograd [52], has a running time of O(n2.376). The graphical presentation of Strassen's algorithm is due to Paterson [155]. Fischer and Meyer [67] adapted Strassen's algorithm to boolean matrices (Theorem 31.10) .

Gaussian elimination, upon which the LU and LUP decompositions are based, was the first systematic method for solving linear systems of equations. It was also one of the earliest numerical algorithms. Although it was known earlier, its discovery is commonly attributed to C. F. Gauss (1777-1855). In his famous paper [183], Strassen also showed that an n n matrix can be inverted in O(nlg 7) time. Winograd [203] originally proved that matrix multiplication is no harder than matrix inversion, and the converse is due to Aho, Hopcroft, and Ullman [4].

Strang [182] has an excellent presentation of symmetric positive-definite matrices and on linear algebra in general. He makes the following remark on page 334: "My class often asks about unsymmetric positive definite matrices. I never use that term."

Go to Chapter 32 Back to Table of Contents


Intro to Algorithms: CHAPTER 32: POLYNOMIALS AND THE FFT

 Return to Table of Contents

CHAPTER 32: POLYNOMIALS AND THE FFT

The straightforward method of adding two polynomials of degree n takes [image] (n) time, but the straightforward method of multiplying them takes [image] (n2) time. In this chapter, we shall show how the Fast Fourier Transform, or FFT, can reduce the time to multiply polynomials to [image] (nl n).

Polynomials

A polynomial in the variable x over an algebraic field F is a function A(x) that can be represented as follows:

We call n the degree-bound of the polynomial, and we call the values a0, a1, . . ., an - 1 the coefficients of the polynomial. The coefficients are drawn from the field F, typically the set C of complex numbers. A polynomial A(x) is said to have degree k if its highest nonzero coefficient is ak. The degree of a polynomial of degree-bound n can be any integer between 0 and n - 1, inclusive. Conversely, a polynomial of degree k is a polynomial of degree-bound n for any n > k.

There are a variety of operations we might wish to define for polynomials. For polynomial addition, if A (x) and B (x) are polynomials of degree-bound n, we say that their sum is a polynomial C (x), also of degree-bound n, such that C (x) = A (x) + B (x) for all x in the underlying field. That is, if

and

then

where cj = aj + bj for j = 0,1, . . .,n - 1. For example, if A(x) = 6x3 + 7x2 - 10x + 9 and B(x) = -2x3+ 4x - 5, then C(x) = 4x3 + 7x2 - 6x + 4.

For polynomial multiplication, if A(x) and B(x) are polynomials of degree-bound n, we say that their product C(x) is a polynomial of degree-bound 2n - 1 such that C(x) = A(x)B(x) for all x in the underlying field. You have probably multiplied polynomials before, by multiplying each term in A(x) by each term in B(x) and combining terms with equal powers. For example, we can multiply A(x) = 6x3 + 7x2 - 10x + 9 and B(x) = -2x3 + 4x - 5 as follows:

                        6x3 + 7x2 - 10x  +  9

                     -  2x3       +  4x  -  5

                    -------------------------

                     - 30x3 - 35x2 + 50x - 45

                24x4 + 28x3 - 40x2 + 36x

- 12x6 - 14x5 + 20x4 - 18x3

---------------------------------------------

- 12x6 - 14x5 + 44x4 - 20x3 - 75x2 +  86x - 45

Another way to express the product C(x) is

(32.1)

where

(32.2)

Note that degree(C) = degree(A) + degree(B), implying

degree-bound(C) = degree-bound(A) + degree-bound(B) - 1

   degree-bound(A) + degree-bound(B).

We shall nevertheless speak of the degree-bound of C as being the sum of the degree-bounds of A and B, since if a polynomial has degree-bound k it also has degree-bound k + 1.

Chapter outline

Section 32.1 presents two ways to represent polynomials: the coefficient representation and the point-value representation. The straightforward methods for multiplying polynomials--equations (32.1) and (32.2)--take [image] (n2) time when the polynomials are represented in coefficient form, but only [image] (n) time when they are represented in point-value form. We can, however, multiply polynomials using the coefficient representation in only [image] (n lg n) time by converting between the two representations. To see why this works, we must first study complex roots of unity, which we do in Section 32.2. Then, we use the FFT and its inverse, also described in Section 32.2, to perform the conversions. Section 32.3 shows how to implement the FFT quickly in both serial and parallel models.

This chapter uses complex numbers extensively, and the symbol i will be used exclusively to denote [image].

32.1 Representation of polynomials

The coefficient and point-value representations of polynomials are in a sense equivalent; that is, a polynomial in point-value form has a unique counterpart in coefficient form. In this section, we introduce the two representations and show how they can be combined to allow multiplication of two degree-bound n polynomials in [image] (n lg n) time.

Coefficient representation

A coefficient representation of a polynomial [image] of degree-bound n is a vector of coefficients a = (a0, a1, . . ., an-1). In matrix equations in this chapter, we shall generally treat vectors as column vectors.

The coefficient representation is convenient for certain operations on polynomials. For example, the operation of evaluating the polynomial A(x) at a given point x0 consists of computing the value of A(x0). Evaluation takes time [image] (n) using Horner's rule:

A(x0) = a0 + x0(a1+x0(a2+ ... +x0(an-2+x0(an-1))...)).

Similarly, adding two polynomials represented by the coefficient vectors a = (a0, a1, . . ., an-1) and b = (b0, b1, . . ., bn-1) takes [image] (n) time: we just output the coefficient vector c = (c0, c1, . . ., cn-1), where cj = aj + bj for j = 0,1, . . . ,n - 1.

Now, consider the multiplication of two degree-bound n polynomials A(x) and B(x) represented in coefficient form. If we use the method described by equations (32.1) and (32.2), polynomial multiplication takes time [image] (n2), since each coefficient in the vector a must be multiplied by each coefficient in the vector b. The operation of multiplying polynomials in coefficient form seems to be considerably more difficult than that of evaluating a polynomial or adding two polynomials. The resulting coefficient vector c, given by equation (32.2), is also called the convolution of the input vectors a and b, denoted c = a [image] b. Since multiplying polynomials and computing convolutions are fundamental computational problems of considerable practical importance, this chapter concentrates on efficient algorithms for them.

Point-value representation

A point-value representation of a polynomial A(x) of degree-bound n is a set of n point-value pairs

{(x0, y0), (x1, y1), . . ., (xn-1, yn-1)}

such that all of the xk are distinct and

yk = A(xk)

(32.3)

for k = 0, 1, . . ., n - 1. A polynomial has many different point-value representations, since any set of n distinct points x0, x1, . . ., xn-1 can be used as a basis for the representation.

Computing a point-value representation for a polynomial given in coefficient form is in principle straightforward, since all we have to do is select n distinct points x0, x1, . . ., xn-1 and then evaluate A(xk) for k = 0, 1, . . ., n - 1. With Horner's method, this n-point evaluation takes time [image] (n2). We shall see later that if we choose the xk cleverly, this computation can be accelerated to run in time [image] (n lg n).

The inverse of evaluation--determining the coefficient form of a polynomial from a point-value representation--is called interpolation. The following theorem shows that interpolation is well defined, assuming that the degree-bound of the interpolating polynomial equals the number of given point-value pairs.

Theorem 32.1

For any set {(x0, y0), (x1, y1), . . ., (xn-1, yn-1)} of n point-value pairs, there is a unique polynomial A(x) of degree-bound n such that yk = A(xk) for k = 0, 1, . . ., n - 1.

Proof The proof is based on the existence of the inverse of a certain matrix. Equation (32.3) is equivalent to the matrix equation

(32.4)

The matrix on the left is denoted V(x0, x1, . . ., xn-1) and is known as a Vandermonde matrix. By Exercise 31.1-10, this matrix has determinant

and therefore, by Theorem 31.5, it is invertible (that is, nonsingular) if the xk are distinct. Thus, the coefficients aj can be solved for uniquely given the point-value representation:

a = V(x0, x1, . . ., xn-1)-1y.

The proof of Theorem 32.1 describes an algorithm for interpolation based on solving the set (32.4) of linear equations. Using the LU decomposition algorithms of Chapter 31, we can solve these equations in time O(n3).

A faster algorithm for n-point interpolation is based on Lagrange's formula:

(32.5)

You may wish to verify that the right-hand side of equation (32.5) is a polynomial of degree-bound n that satisfies A(xk) = yk for all k. Exercise 32.1-4 asks you how to compute the coefficients of A using Lagrange's formula in time [image] (n2).

Thus, n-point evaluation and interpolation are well-defined inverse operations that transform between the coefficient representation of a polynomial and a point-value representation.1 The algorithms described above for these problems take time [image] (n2).

1Interpolation is a notoriously tricky problem from the point of view of numerical stability. Although the approaches described here are mathematically correct, small differences in the inputs or round-off errors during computation can cause large differences in the result.

The point-value representation is quite convenient for many operations on polynomials. For addition, if C(x) = A(x) + B(x), then C(xk) = A(xk) + B(xk) for any point xk. More precisely, if we have a point-value representation for A,

{(x0, y0), (x1, y1), . . ., (xn-1, yn-1)} ,

and for B,

{(x0, y'0), (x1, y'1), . . ., (xn-1, y'n-1)}

(note that A and B are evaluated at the same n points), then a point-value representation for C is

{(x0, y0 + y'0), (x1, y1+y'1), . . ., (xn-1, yn-1+y'n-1)}.

The time to add two polynomials of degree-bound n in point-value form is thus [image] (n).

Similarly, the point-value representation is convenient for multiplying polynomials. If C(x) = A(x)B(x), then C(xk) = A(xk)B(xk) for any point xk, and we can pointwise multiply a point-value representation for A by a point-value representation for B to obtain a point-value representation for C. We must face the problem, however, that the degree-bound of C is the sum of the degree-bounds for A and B. A standard point-value representation for A and B consists of n point-value pairs for each polynomial. Multiplying these together gives us n point-value pairs for C, but since the degree-bound of C is 2n, Theorem 32.1 implies that we need 2n point-value pairs for a point-value representation of C. We must therefore begin with "extended" point-value representations for A and for B consisting of 2n point-value pairs each. Given an extended point-value representation for A,

{(x0,y0),(x1, y1),...,(x2n-1,y2n-1)},

and a corresponding extended point-value representation for B,

{(x0,y'0),(x1,y'1),...,(x2n-1,y'2n-1)},

then a point-value representation for C is

{(x0,y0y'0),(x1,y1y'1),...,(x2n-1,y2n-1y'2n-1)}.

Given two input polynomials in extended point-value form, we see that the time to multiply them to obtain the point-value form of the result is [image] (n), much less than the time required to multiply polynomials in coefficient form.

Finally, we consider how to evaluate a polynomial given in point-value form at a new point. For this problem, there is apparently no approach that is simpler than converting the polynomial to coefficient form first, and then evaluating it at the new point.

Fast multiplication of polynomials in coefficient form

Can we use the linear-time multiplication method for polynomials in point-value form to expedite polynomial multiplication in coefficient form? The answer hinges on our ability to convert a polynomial quickly from coefficient form to point-value form (evaluate) and vice-versa (interpolate).

We can use any points we want as evaluation points, but by choosing the evaluation points carefully, we can convert between representations in only [image] (n lg n) time. As we shall see in Section 32.2, if we choose "complex roots of unity" as the evaluation points, we can produce a point-value representation by taking the Discrete Fourier Transform (or DFT) of a coefficient vector. The inverse operation, interpolation, can be performed by taking the "inverse DFT" of point-value pairs, yielding a coefficient vector. Section 32.2 will show how the FFT performs the DFT and inverse DFT operations in [image] (n lg n) time.

Figure 32.1 shows this strategy graphically. One minor detail concerns degree-bounds. The product of two polynomials of degree-bound n is a polynomial of degree-bound 2n. Before evaluating the input polynomials A and B, therefore, we first double their degree-bounds to 2n by adding n high-order coefficients of 0. Because the vectors have 2n elements, we use "complex (2n)th roots of unity," which are denoted by the w2n terms in Figure 32.1.

Given the FFT, we have the following [image] (n lg n)-time procedure for multiplying two polynomials A(x) and B(x) of degree-bound n, where the input and output representations are in coefficient form. We assume that n is a power of 2; this requirement can always be met by adding high-order zero coefficients.

Figure 32.1 A graphical outline of an efficient polynomial-multiplication process. Representations on the top are in coefficient form, while those on the bottom are in point-value form. The arrows from left to right correspond to the multiplication operation. The w2n terms are complex (2n)th roots of unity.

1. Double degree-bound: Create coefficient representations of A(x) and B(x) as degree-bound 2n polynomials by adding n high-order zero coefficients to each.

2. Evaluate: Compute point-value representations of A(x) and B(x) of length 2n through two applications of the FFT of order 2n. These representations contain the values of the two polynomials at the (2n)th roots of unity.

3. Pointwise multiply: Compute a point-value representation for the polynomial C(x) = A(x)B(x) by multiplying these values together pointwise. This representation contains the value of C(x) at each (2n)th root of unity.

4. Interpolate: Create the coefficient representation of the polynomial C(x) through a single application of an FFT on 2n point-value pairs to compute the inverse DFT.

Steps (1) and (3) take time [image] (n), and steps (2) and (4) take time [image] (n lg n). Thus, once we show how to use the FFT, we will have proven the following.

Theorem 32.2

The product of two polynomials of degree-bound n can be computed in time [image] (n lg n), with both the input and output representations in coefficient form.

Exercises

32.1-1

Multiply the polynomials A(x) = 7x3 - x2 + x - 10 and B(x) = 8x3 - 6x + 3 using equations (32.1) and (32.2).

32.1-2

Evaluating a polynomial A(x) of degree-bound n at a given point x0 can also be done by dividing A(x) by the polynomial (x - x0) to obtain a quotient polynomial q(x) of degree-bound n - 1 and a remainder r, such that

A(x) = q(x)(x - x0) + r .

Clearly, A(x0) = r. Show how to compute the remainder r and the coefficients of q(x) in time [image] (n) from x0 and the coefficients of A.

32.1-3

Derive a point-value representation for [image] from a point-value representation for [image], assuming that none of the points is 0.

32.1-4

Show how to use equation (32.5) to interpolate in time [image] (n2). (Hint: First compute [image] j(x - xk) and [image] j(xj - xk) and then divide by (x - xk) and (xj - xk) as necessary for each term. See Exercise 32.1-2.)

32.1-5

Explain what is wrong with the "obvious" approach to polynomial division using a point-value representation. Discuss separately the case in which the division comes out exactly and the case in which it doesn't.

32.1-6

Consider two sets A and B, each having n integers in the range from 0 to 10n. We wish to compute the Cartesian sum of A and B, defined by

C = {x + y: x    A and y    B}.

Note that the integers in C are in the range from 0 to 20n. We want to find the elements of C and the number of times each element of C is realized as a sum of elements in A and B. Show that the problem can be solved in O(n lg n) time. (Hint: Represent A and B as polynomials of degree 10n.)

32.2 The DFT and FFT

In Section 32.1, we claimed that if we use complex roots of unity, we can evaluate and interpolate in [image] (n lg n) time. In this section, we define complex roots of unity and study their properties, define the DFT, and then show how the FFT computes the DFT and its inverse in just [image] (n lg n) time.

Figure 32.2 The values of in the complex plane, where w8 = e2 i/8 is the principal 8th root of unity.

Complex roots of unity

A complex nth root of unity is a complex number w such that

wn = 1 .

There are exactly n complex nth roots of unity; these are e2 [image] ik/n for k = 0, 1, . . . , n - 1. To interpret this formula, we use the definition of the exponential of a complex number:

eiu = cos(u) + i sin(u).

Figure 32.2 shows that the n complex roots of unity are equally spaced around the circle of unit radius centered at the origin of the complex plane. The value

wn = e2  i/n

(32.6)

is called the principal nth root of unity; all of the other complex nth roots of unity are powers of wn.

The n complex nth roots of unity,

form a group under multiplication (see Section 33.3). This group has the same structure as the additive group (Z n, +) modulo n, since [image] implies that [image]. Similarly, [image]. Essential properties of the complex nth roots of unity are given in the following lemmas.

Lemma 32.3

For any integers n [image] 0, k [image] 0, and d > 0,

(32.7)

Proof The lemma follows directly from equation (32.6), since

Corollary 32.4

For any even integer n > 0,

Proof The proof is left as Exercise 32.2-1.

Lemma 32.5

If n > 0 is even, then the squares of the n complex nth roots of unity are the n/2 complex (n/2)th roots of unity.

Proof By the cancellation lemma, we have [image], for any non-negative integer k. Note that if we square all of the complex nth roots of unity, then each (n/2)th root of unity is obtained exactly twice, since

Thus, [image] have the same square. This property can also be proved using Corollary 32.4, since [image] implies [image], and thus [image].

As we shall see, the halving lemma is essential to our divide-and-conquer approach for converting between coefficient and point-value representations of polynomials, since it guarantees that the recursive subproblems are only half as large.

Lemma 32.6

For any integer n [image] 1 and nonnegative integer k not divisible by n,

Proof Because equation (3.3) applies to complex values,

Requiring that k not be divisible by n ensures that the denominator is not 0, since [image] only when k is divisible by n.

The DFT

Recall that we wish to evaluate a polynomial

of degree-bound n at [image] (that is, at the n complex nth roots of unity).2 Without loss of generality, we assume that n is a power of 2, since a given degree-bound can always be raised--we can always add new high-order zero coefficients as necessary. We assume that A is given in coefficient form: a = (a0, a1, . . . , an-1). Let us define the results yk, for k = 0, 1, . . . , n - 1, by

(32.8)

The vector y = (y0, y1, . . . , yn-1) is the Discrete Fourier Transform (DFT) of the coefficient vector a = (a0, a1, . . . , an-1). We also write y = DFTn(a).

2The length n is actually what we referred to as 2n in Section 32.1, since we double the degree-bound of the given polynomials prior to evaluation. In the context of polynomial multiplication, therefore, we are actually working with complex (2n)th roots of unity.+

The FFT

By using a method known as the Fast Fourier Transform (FFT), which takes advantage of the special properties of the complex roots of unity, we can compute DFTn(a) in time [image] (n lg n), as opposed to the [image] (n2) time of the straightforward method.

The FFT method employs a divide-and-conquer strategy, using the even-index and odd-index coefficients of A(x) separately to define the two new degree-bound n/2 polynomials A[0](x) and A[1](x):

A[0](x)  =  a0 + a2x + a4x2 +          + an-2xn/2-1,

A[1](x)  =  a1 + a3x + a5x2 +          + an-1xn/2-1.

Note that A[0] contains all the even-index coefficients of A (the binary representation of the index ends in 0) and A[1] contains all the odd-index coefficients (the binary representation of the index ends in 1). It follows that

A(x) = A[0](x2) + xA[1](x2),

(32.9)

so that the problem of evaluating A(x) at [image] reduces to

1. evaluating the degree-bound n/2 polynomials A[0](x) and A[1](x) at the points

(32.10)

and then

2. combining the results according to equation (32.9).

By the halving lemma, the list of values (32.10) consists not of n distinct values but only of the n/2 complex (n/2)th roots of unity, with each root occurring exactly twice. Therefore, the polynomials A[0] and A[1] of degree-bound n/2 are recursively evaluated at the n/2 complex (n/2)th roots of unity. These subproblems have exactly the same form as the original problem, but are half the size. We have now successfully divided an n-element DFTn computation into two n/2-element DFTn/2 computations. This decomposition is the basis for the following recursive FFT algorithm, which computes the DFT of an n-element vector a = (a0, a1, . . . , an - 1), where n is a power of 2.

The RECURSIVE-FFT procedure works as follows. Lines 2-3 represent the basis of the recursion; the DFT of one element is the element itself, since in this case

Lines 6-7 define the coefficient vectors for the polynomials A[0] and A[1]. Lines 4, 5, and 13 guarantee that w is updated properly so that whenever lines 11-12 are executed, [image]. (Keeping a running value of w from iteration to iteration saves time over computing [image] from scratch each time through the for loop.) Lines 8-9 perform the recursive DFTn/2 computations, setting, for k = 0, 1, . . . , n/2 - 1,

or, since [image] by the cancellation lemma,

Lines 11-12 combine the results of the recursive DFTn/2 calculations. For y0, y1, . . . , yn/2 - 1, line 11 yields

where the last line of this argument follows from equation (32.9). For yn/2,yn/2+1, . . . , yn - 1, letting k = 0, 1, . . . , n/2 - 1, line 12 yields

The second line follows from the first since [image]. The fourth line follows from the third because [image] implies [image]. The last line follows from equation (32.9). Thus, the vector y returned by RECURSIVE-FFT is indeed the DFT of the input vector a.

To determine the running time of procedure RECURSIVE-FFT, we note that exclusive of the recursive calls, each invocation takes time [image] (n), where n is the length of the input vector. The recurrence for the running time is therefore

T(n) = 2T(n/2) +   (n)

=   (n lg n) .

Thus, we can evaluate a polynomial of degree-bound n at the complex nth roots of unity in time [image] (n lg n) using the Fast Fourier Transform.

Interpolation at the complex roots of unity

We now complete the polynomial multiplication scheme by showing how to interpolate the complex roots of unity by a polynomial, which enables us to convert from point-value form back to coefficient form. We interpolate by writing the DFT as a matrix equation and then looking at the form of the matrix inverse.

From equation (32.4), we can write the DFT as the matrix product y = Vna, where Vn is a Vandermonde matrix containing the appropriate powers of wn:

The (k, j) entry of [image], for j,k = 0, 1, . . . , n - 1, and the exponents of the entries of Vn form a multiplication table.

For the inverse operation, which we write as [image], we proceed by multiplying y by the matrix [image], the inverse of Vn.

Theorem 32.7

For j, k = 0, 1, . . . , n - 1, the (j, k) entry of [image].

Proof We show that [image], the n X n identity matrix. Consider the (j, j') entry of [image]:

This summation equals 1 if j' = j, and it is 0 otherwise by the summation lemma (Lemma 32.6). Note that we rely on -(n - 1) < j' - j < n - l, so that j' - j is not divisible by n, in order for the summation lemma to apply.

Given the inverse matrix [image], we have that [image] is given by

(32.11)

for j = 0,1, . . . , n - 1. By comparing equations (32.8) and (32.11), we see that by modifying the FFT algorithm to switch the roles of a and y, replace wn by [image], and divide each element of the result by n, we compute the inverse DFT (see Exercise 32.2-4). Thus, [image] can be computed in [image] (n lg n) time as well.

Thus, by using the FFT and the inverse FFT, we can transform a polynomial of degree-bound n back and forth between its coefficient representation and a point-value representation in time [image] (n lg n). In the context of polynomial multiplication, we have shown the following.

Theorem 32.8

For any two vectors a and b of length n, where n is a power of 2,

where the vectors a and b are padded with 0's to length 2n and [image] denotes the componentwise product of two 2n-element vectors.

Exercises

32.2-1

Prove Corollary 32.4.

32.2-2

Compute the DFT of the vector (0, 1, 2, 3).

32.2-3

Do Exercise 32.1-1 by using the [image] (n lg n)-time scheme.

32.2-4

Write pseudocode to compute [image] in [image] (n lg n) time.

32.2-5

Describe the generalization of the FFT procedure to the case in which n is a power of 3. Give a recurrence for the running time, and solve the recurrence.

32.2-6

Suppose that instead of performing an n-element FFT over the field of complex numbers (where n is even), we use the ring Zm of integers modulo m, where m = 2tn/2 + 1 and t is an arbitrary positive integer. Use w = 2t instead of wn as a principal nth root of unity, modulo m. Prove that the DFT and the inverse DFT are well defined in this system.

32.2-7

Given a list of values z0, z1, . . . , zn-1 (possibly with repetitions), show how to find the coefficients of the polynomial P(x) of degree-bound n that has zeros only at z0, z1, . . . , zn - 1 (possibly with repetitions). Your procedure should run in time O(n lg2 n). (Hint: The polynomial P(x) has a zero at zj if and only if P(x) is a multiple of (x - zj).)

32.2-8

The chirp transform of a vector a = (a0, a1, . . . , an - 1) is the vector y = (y0, y1, . . . , yn - 1), where [image] and z is any complex number. The DFT is therefore a special case of the chirp transform, obtained by taking z = wn. Prove that the chirp transform can be evaluated in time O(n lg n) for any complex number z. (Hint: Use the equation

to view the chirp transform as a convolution.)

32.3 Efficient FFT implementations

Since the practical applications of the DFT, such as signal processing, demand the utmost speed, this section examines two efficient FFT implementations. First, we shall examine an iterative version of the FFT algorithm that runs in [image] (n lg n) time but has a lower constant hidden in the [image] -notation than the recursive implementation in Section 32.2. Then, we shall use the insights that led us to the iterative implementation to design an efficient parallel FFT circuit.

Figure 32.3 A butterfly operation. The two input values enter from the left, is multiplied by , and the sum and difference are output on the right. The figure can be interpreted as a combinational circuit.

Figure 32.4 The tree of input vectors to the recursive calls of the RECURSIVE-FFT procedure. The initial invocation is for n = 8.

An iterative FFT implementation

We first note that the for loop of lines 10-13 of RECURSIVE-FFT involves computing the value [image] twice. In compiler terminology, this value is known as a common subexpression. We can change the loop to compute it only once, storing it in a temporary variable t.

The operation in this loop, multiplying w (which is equal to [image], storing the product into t, and adding and subtracting t from [image], is known as a butterfly operation and is shown schematically in Figure 32.3.

We now show how to make the FFT algorithm iterative rather than recursive in structure. In Figure 32.4, we have arranged the input vectors to the recursive calls in an invocation of RECURSIVE-FFT in a tree structure, where the initial call is for n = 8. The tree has one node for each call of the procedure, labeled by the corresponding input vector. Each RECURSIVE-FFT invocation makes two recursive calls, unless it has received a 1-element vector. We make the first call the left child and the second call the right child.

Looking at the tree, we observe that if we could arrange the elements of the initial vector a into the order in which they appear in the leaves, we could mimic the execution of the RECURSIVE-FFT procedure as follows. First, we take the elements in pairs, compute the DFT of each pair using one butterfly operation, and replace the pair with its DFT. The vector then holds n/2 2-element DFT's. Next, we take these n/2 DFT's in pairs and compute the DFT of the four vector elements they come from by executing two butterfly operations, replacing two 2-element DFT's with one 4-element DFT. The vector then holds n/4 4-element DFT's. We continue in this manner until the vector holds two (n/2)-element DFT's, which we can combine using n/2 butterfly operations into the final n-element DFT.

To turn this observation into code, we use an array A[0 . . n - 1] that initially holds the elements of the input vector a in the order in which they appear in the leaves of the tree of Figure 32.4. (We shall show later how to determine this order.) Because the combining has to be done on each level of the tree, we introduce a variable s to count the levels, ranging from 1 (at the bottom, when we are combining pairs to form 2-element DFT's) to lg n (at the top, when we are combining two (n/2)-element DFT's to produce the final result). The algorithm therefore has the following structure:

1  for s    1 to lg n

2       do for k    0 to n - 1 by 2s

3              do combine the two 2s-1 -element DFT's in

A[k . . k + 2s-1 - 1] and A[k + 2s-1 . . k + 2s - 1]

into one 2s-element DFT in A[k . . k + 2s - 1]

We can express the body of the loop (line 3) as more precise pseudocode. We copy the for loop from the RECURSIVE-FFT procedure, identifying y[0] with A[k . . k + 2s-1 _ 1] and y[1] with A[k + 2s-1. . k + 2s _ 1]. The value of w used in each butterfly operation depends on the value of s; we use wm, where m = 2s. (We introduce the variable m solely for the sake of readability.) We introduce another temporary variable u that allows us to perform the butterfly operation in place. When we replace line 3 of the overall structure by the loop body, we get the following pseudocode, which forms the basis of our final iterative FFT algorithm as well as the parallel implementation we shall present later.

FFT-BASE(a)

1  n    length[a]     n is a power of 2.

2  for s    1 to lg n

3       do m    2s

4          wm    e2  i/m

5          for k    0 to n - 1 by m

6              do w    1

7                 for j    0 to m/2 - 1

8                     do t    w A[k + j + m/2]

9                        u    A[ k + j]

10                        A[k + j]    u + t

11                        A[k + j + m /2]    u - t

12                        w    wwm

We now present the final version of our iterative FFT code, which inverts the two inner loops to eliminate some index computation and uses the auxiliary procedure BIT-REVERSE-COPY(a, A) to copy vector a into array A in the initial order in which we need the values.

ITERATIVE-FFT(a)

1  BIT-REVERSE-COPY (a, A)

2  n    length[a]         n is a power of 2.

3  for s    1 to lg n

4        do m    2s

5           wm    e2  i/m

6           w    1

7           for j    0 to m/2 - 1

8               do for k    j to n - 1 by m

9                      do t    w A[k + m/2]

10                         u    A[k]

11                         A[k]    u + t

12                         A[k + m/2]    u - t

13                  w    wwm

14  return A

How does BIT-REVERSE-COPY get the elements of the input vector a into the desired order in the array A? The order in which the leaves appear in Figure 32.4 is "bit-reverse binary." That is, if we let rev(k) be the lg n-bit integer formed by reversing the bits of the binary representation of k, then we want to place vector element ak in array position A[rev(k)]. In Figure 32.4, for example, the leaves appear in the order 0, 4, 2, 6, 1, 5, 3, 7; this sequence in binary is 000, 100, 010, 110, 001, 101, 011, 111, and in bit- reverse binary we get the sequence 000, 001, 010, 011, 100, 101, 110, 111. To see that we want bit-reverse binary order in general, we note that at the top level of the tree, indices whose low-order bit is 0 are placed in the left subtree and indices whose low-order bit is 1 are placed in the right subtree. Stripping off the low-order bit at each level, we continue this process down the tree, until we get the bit-reverse binary order at the leaves.

Since the function rev (k) is easily computed, the BIT-REVERSE-COPY procedure can be written as follows.

BIT-REVERSE-COPY(a, A)

1  n    length[a]

2  for k    0 to n - 1

3       do A[rev(k)]    ak

The iterative FFT implementation runs in time [image] (n lg n). The call to BIT-REVERSE-COPY(a,A) certainly runs in O(n lg n) time, since we iterate n times and can reverse an integer between 0 and n - 1, with lg n bits, in O(lg n) time. (In practice, we usually know the initial value of n in advance, so we would probably code a table mapping k to rev(k), making BIT-REVERSE-COPY run in [image] (n) time with a low hidden constant. Alternatively, we could use the clever amortized reverse binary counter scheme described in Problem 18-1.) To complete the proof that ITERATIVE-FFT runs in time [image] (n lg n), we show that L(n), the number of times the body of the innermost loop (lines 9-12) is executed, is [image] (n lg n). We have

A parallel FFT circuit

We can exploit many of the properties that allowed us to implement an efficient iterative FFT algorithm to produce an efficient parallel algorithm for the FFT. (See Chapter 29 for a description of the combinational-circuit model.) The combinational circuit PARALLEL-FFT that computes the FFT on n inputs is shown in Figure 32.5 for n = 8. The circuit begins with a bi-reverse permutation of the inputs, followed by lg n stages, each stage consisting of n/2 butterflies executed in parallel. The depth of the circuit is therefore [image] (lg n).

The leftmost part of the circuit PARALLEL-FFT performs the bit-reverse permutation, and the remainder mimics the iterative FFT-BASE procedure. We take advantage of the fact that each iteration of the outermost for loop performs n/2 independent butterfly operations that can be performed in parallel. The value of s in each iteration within FFT-BASE corresponds to a stage of butterflies shown in Figure 32.5. Within stage s, for s = 1, 2, . . . , lg n, there are n/2s groups of butterflies (corresponding to each value of k in FFT-BASE), with 2s-1 butterflies per group (corresponding to each value of j in FFT-BASE). The butterflies shown in Figure 32.5 correspond to the butterfly operations of the innermost loop (lines 8-11 of FFT-BASE). Note also that the values of w used in the butterflies correspond to those used in FFT-BASE: in stage s, we use [image], where m = 2s.

Figure 32.5 A combinational circuit PARALLEL-FFT that computes the FFT, here shown on n = 8 inputs. The stages of butterflies are labeled to correspond to iterations of the outermost loop of the FFT-BASE procedure. An FFT on n inputs can be computed in (1g n) depth with (n lg n) combinational elements.

Exercises

32.3-1

Show how ITERATIVE-FFT computes the DFT of the input vector (0, 2, 3, -1, 4, 5, 7, 9).

32.3-2

Show how to implement an FFT algorithm with the bit-reversal permutation occurring at the end, rather than at the beginning, of the computation. (Hint: Consider the inverse DFT.)

32.3-3

To compute DFTn, how many addition, subtraction, and multiplication elements, and how many wires, are needed in the PARALLEL-FFT circuit described in this section? (Assume that only one wire is needed to carry a number from one place to another.)

32.3-4

Suppose that the adders in the FFT circuit sometimes fail in such a manner that they always produce a zero output, independent of their inputs. Suppose that exactly one adder has failed, but that you don't know which one. Describe how you can identify the failed adder by supplying inputs to the overall FFT circuit and observing the outputs. Try to make your proeedure efficient.

Problems

32-1 Divide-and-conquer multiplication

a. Show how to multiply two linear polynomials ax + b and cx + d using only three multiplications. (Hint: One of the multiplications is (a + b) [image] (c + d).)

b. Give two divide-and-conquer algorithms for multiplying two polynomials of degree-bound n that run in time [image] (n1g 3). The first algorithm should divide the input polynomial coefficients into a high half and a low half, and the second algorithm should divide them according to whether their index is odd or even.

c. Show that two n-bit integers can be multiplied in O(n1g 3) steps, where each step operates on at most a constant number of 1-bit values.

32-2 Toeplitz matrices

A Toeplitz matrix is an n X n matrix A = (aij) such that aij = ai-1, j-1 for i = 2, 3, . . . , n and j = 2, 3, . . . , n.

a. Is the sum of two Toeplitz matrices necessarily Toeplitz? What about the product?

b. Describe how to represent a Toeplitz matrix so that two n X n Toeplitz matrices ean be added in O(n) time.

c. Give an O(n lg n)-time algorithm for multiplying an n X n Toeplitz matrix by a vector of length n. Use your representation from part (b).

d. Give an efficient algorithm for multiplying two n X n Toeplitz matrices. Analyze its running time.

32-3 Evaluating all derivatives of a polynomial at a point

Given a polynomial A(x) of degree-bound n, its tth derivative is defined by

From the coefficient representation (a0, a1, . . . , an-1) of A(x) and a given point x0, we wish to determine A(t) (x0) for t = 0, 1, . . . , n - 1.

a. Given coefficients b0, b1, . . . , bn-1 such that

show how to compute A(t) (x0), for t = 0, 1, . . . , n - 1, in O(n) time.

b. Explain how to find b0, b1, . . . , bn-1 in O(n lg n) time, given [image] for k = 0, 1, . . . , n - 1.

c. Prove that

where f(j) = aj [image] j! and

d. Explain how to evaluate [image] for k = 0, 1, . . . , n - 1 in O(n lg n) time. Conclude that all nontrivial derivatives of A(x) can be evaluated at x0 in O(n lg n) time.

32-4 Polynomial evaluation at multiple points

We have observed that the problem of evaluating a polynomial of degree-bound n - 1 at a single point can be solved in O(n) time using Horner's rule. We have also discovered that such a polynomial can be evaluated at all n complex roots of unity in O(n lg n) time using the FFT. We shall now show how to evaluate a polynomial of degree-bound n at n arbitrary points in O(n lg2 n) time.

To do so, we shall use the fact that we can compute the polynomial remainder when one such polynomial is divided by another in O(n lg n) time, a result that we assume without proof. For example, the remainder of 3x3 + x2 - 3x + 1 when divided by x2 + x + 2 is

(3x3 + x2 - 3x + 1) mod (x2 + x + 2) = 5x - 3.

Given the coefficient representation of a polynomial [image] and n points x0, x1, . . . , xn-1, we wish to compute the n values A(x0), A(x1), . . . , A(xn-1). For 0 [image] i [image] j [image] n - 1, define the polynomials [image] and Qij(x) = A(x) mod pij(x). Note that Qij(x) has degree-bound at most j - i.

a. Prove that A(x) mod (x - z) = A(z) for any point z.

b. Prove that Qkk(x) = A(xk) and that Q0,n-1(x) = A(x).

c. Prove that for i [image] k [image] j, we have Qik(x) = Qij(x) mod Pik(x) and Qkj(x) = Qij(x) mod Pkj (x).

d. Give an O(n lg2 n)-time algorithm to evaluate A(x0), A(x1), . . . , A(xn-1).

32-5 FFT using modular arithmetic

As defined, the Discrete Fourier Transform requires the use of complex numbers, which can result in a loss of precision due to round-off errors. For some problems, the answer is known to contain only integers, and it is desirable to utilize a variant of the FFT based on modular arithmetic in order to guarantee that the answer is calculated exactly. An example of such a problem is that of multiplying two polynomials with integer coefficients. Exercise 32.2-6 gives one approach, using a modulus of length [image] (n) bits to handle a DFT on n points. This problem gives another approach that uses a modulus of the more reasonable length O(lg n); it requires that you understand the material of Chapter 33. Let n be a power of 2.

a. Suppose that we search for the smallest k such that p = kn + 1 is prime. Give a simple heuristic argument why we might expect k to be approximately lg n. (The value of k might be much larger or smaller, but we can reasonably expect to examine O(lg n) candidate values of k on average.) How does the expected length of p compare to the length of n?

Let g be a generator of [image], and let w = gk mod p.

b. Argue that the DFT and the inverse DFT are well-defined inverse operations modulo p, where w is used as a principal nth root of unity.

c. Argue that the FFT and its inverse can be made to work modulo p in time O(n lg n), where operations on words of O(lg n) bits take unit time. Assume that the algorithm is given p and w.

d. Compute the DFT modulo p = 17 of the vector (0, 5, 3, 7, 7, 2, 1, 6). Note that g = 3 is a generator of [image].

Chapter notes

Press, Flannery, Teukolsky, and Vetterling[161, 162] have a good description of the Fast Fourier Transform and its applications. For an excellent introduction to signal processing, a popular FFT application area, see the text by Oppenheim and Willsky [153].

Cooley and Tukey [51] are widely credited with devising the FFT in the 1960's. The FFT had in fact been discovered many times previously, but its importance was not fully realized before the advent of modern digital computers. Press, Flannery, Teukolsky, and Vetterling attribute the origins of the method to Runge and König ( 1924).

Go to Chapter 33 Back to Table of Contents


Intro to Algorithms: CHAPTER 33: NUMBER-THEORETIC ALGORITHMS

 Return to Table of Contents

CHAPTER 33: NUMBER-THEORETIC ALGORITHMS

Number theory was once viewed as a beautiful but largely useless subject in pure mathematics. Today number-theoretic algorithms are used widely, due in part to the invention of cryptographic schemes based on large prime numbers. The feasibility of these schemes rests on our ability to find large primes easily, while their security rests on our inability to factor the product of large primes. This chapter presents some of the number theory and associated algorithms that underlie such applications.

Section 33.1 introduces basic concepts of number theory, such as divisibility, modular equivalence, and unique factorization. Section 33.2 studies one of the world's oldest algorithms: Euclid's algorithm for computing the greatest common divisor of two integers. Section 33.3 reviews concepts of modular arithmetic. Section 33.4 then studies the set of multiples of a given number a, modulo n, and shows how to find all solutions to the equation ax [image] b (mod n) by using Euclid's algorithm. The Chinese remainder theorem is presented in Section 33.5. Section 33.6 considers powers of a given number a, modulo n, and presents a repeated-squaring algorithm for efficiently computing ab mod n, given a, b, and n. This operation is at the heart of efficient primality testing. Section 33.7 then describes the RSA public-key cryptosystem. Section 33.8 describes a randomized primality test that can be used to find large primes efficiently, an essential task in creating keys for the RSA cryptosystem. Finally, Section 33.9 reviews a simple but effective heuristic for factoring small integers. It is a curious fact that factoring is one problem people may wish to be intractable, since the security of RSA depends on the difficulty of factoring large integers.

Size of inputs and cost of arithmetic computations

Because we shall be working with large integers, we need to adjust how we think about the size of an input and about the cost of elementary arithmetic operations.

In this chapter, a "large input" typically means an input containing "large integers" rather than an input containing "many integers" (as for sorting). Thus, we shall measure the size of an input in terms of the number of bits required to represent that input, not just the number of integers in the input. An algorithm with integer inputs a1, a2, . . . , ak is a polynomial-time algorithm if it runs in time polynomial in lg a1, lg a2, . . . , lg ak, that is, polynomial in the lengths of its binary-encoded inputs.

In most of this book, we have found it convenient to think of the elementary arithmetic operations (multiplications, divisions, or computing remainders) as primitive operations that take one unit of time. By counting the number of such arithmetic operations an algorithm performs, we have a basis for making a reasonable estimate of the algorithm's actual running time on a computer. Elementary operations can be time-consuming, however, when their inputs are large. It thus becomes convenient to measure how many bit operations a number-theoretic algorithm requires. In this model, a multiplication of two [image] -bit integers by the ordinary method uses [image] ( [image] 2) bit operations. Similarly, the operation of dividing a [image] -bit integer by a shorter integer, or the operation of taking the remainder of a [image] -bit integer when divided by a shorter integer, can be performed in time [image] ( [image] 2) by simple algorithms. (See Exercise 33.1-11.) Faster methods are known. For example, a simple divide-and-conquer method for multiplying two [image] -bit integers has a running time of [image] ( [image] lg2 3), and the fastest known method has a running time of [image] ( [image] lg [image] lg lg [image] ). For practical purposes, however, the [image] ( [image] 2) algorithm is often best, and we shall use this bound as a basis for our analyses.

In this chapter, algorithms are generally analyzed in terms of both the number of arithmetic operations and the number of bit operations they require.

33.1 Elementary number-theoretic notions

This section provides a brief review of notions from elementary number theory concerning the set Z = {. . . , -2, -1, 0, 1, 2, . . .} of integers and the set N = {0, 1, 2, . . .} of natural numbers.

Divisibility and divisors

The notion of one integer being divisible by another is a central one in the theory of numbers. The notation d | a (read "d divides a") means that a = kd for some integer k. Every integer divides 0. If a > 0 and d | a, then |d| [image] |a|. If d | a, then we also say that a is a multiple of d. If d does not divide a, we write [image].

If d | a and d [image] 0, we say that d is a divisor of a. Note that d | a if and only if -d | a, so that no generality is lost by defining the divisors to be nonnegative, with the understanding that the negative of any divisor of a also divides a. A divisor of an integer a is at least 1 but not greater than |a|. For example, the divisors of 24 are 1, 2, 3, 4, 6, 8, 12, and 24.

Every integer a is divisible by the trivial divisors 1 and a. Nontrivial divisors of a are also called factors of a. For example, the factors of 20 are 2, 4, 5, and 10.

Prime and composite numbers

An integer a > 1 whose only divisors are the trivial divisors 1 and a is said to be a prime number (or, more simply, a prime). Primes have many special properties and play a critical role in number theory. The small primes, in order, are

2, 3, 5, 7, 11, 13, 17, 19, 23, 29, 31, 37, 41, 43, 47, 53, 59,... .

Exercise 33.1-1 asks you to prove that there are infinitely many primes. An integer a > 1 that is not prime is said to be a composite number (or, more simply, a composite). For example, 39 is composite because 3 | 39. The integer 1 is said to be a unit and is neither prime nor composite. Similarly, the integer 0 and all negative integers are neither prime nor composite.

The division theorem, remainders, and modular equivalence

Given an integer n, the integers can be partitioned into those that are multiples of n and those that are not multiples of n. Much number theory is based upon a refinement of this partition obtained by classifying the nonmultiples of n according to their remainders when divided by n. The following theorem is the basis for this refinement. The proof of this theorem will not be given here (see, for example, Niven and Zuckerman [151]).

Theorem 33.1

For any integer a and any positive integer n, there are unique integers q and r such that 0 [image] r < n and a = qn + r.

The value q = [image] a/n [image] is the quotient of the division. The value r = a mod n is the remainder (or residue) of the division. We have that n | a if and only if a mod n = 0. It follows that

a =   a/n   n + (a mod n)

(33.1)

or

a mod n = a -   a/n   n .

(33.2)

Given a well-defined notion of the remainder of one integer when divided by another, it is convenient to provide special notation to indicate equality of remainders. If (a mod n) = (b mod n), we write a [image] b (mod n) and say that a is equivalent to b, modulo n. In other words, a [image] b (mod n) if a and b have the same remainder when divided by n. Equivalently, a [image] b (mod n) if and only if n | (b - a). We write a [image] b (mod n) if a is not equivalent to b, modulo n. For example, 61 [image] 6 (mod 11). Also, -13 [image] 22 [image] 2 (mod 5).

The integers can be divided into n equivalence classes according to their remainders modulo n. The equivalence class modulo n containing an integer a is

[a]n = {a + kn : k    Z} .

For example, [3]7 = {. . . , -11, -4, 3, 10, 17, . . .}; other denotations for this set are [-4]7 and [10]7. Writing a [image] [b]n is the same as writing a [image] b (mod n). The set of all such equivalence classes is

Zn = {[a]n : 0    a    n - 1}.

(33.3)

One often sees the definition

Zn = {0, 1,..., n - 1},

(33.4)

which should be read as equivalent to equation (33.3) with the understanding that 0 represents [0]n, 1 represents [1]n, and so on; each class is represented by its least nonnegative element. The underlying equivalence classes must be kept in mind, however. For example, a reference to - 1 as a member of Zn is a reference to [n - 1]n, since - 1 [image] n - 1 (mod n).

Common divisors and greatest common divisors

If d is a divisor of a and also a divisor of b, then d is a common divisor of a and b. For example, the divisors of 30 are 1, 2, 3, 5, 6, 10, 15, and 30, and so the common divisors of 24 and 30 are 1, 2, 3, and 6. Note that 1 is a common divisor of any two integers.

An important property of common divisors is that

d | a and d | b implies d | (a + b) and d | (a - b).

(33.5)

More generally, we have that

d | a and d | b implies d | (ax + by)

(33.6)

for any integers x and y. Also, if a | b, then either |a| [image] |b| or b = 0, which implies that

a | b and b | a implies a =   b.

(33.7)

The greatest common divisor of two integers a and b, not both zero, is the largest of the common divisors of a and b; it is denoted gcd(a, b). For example, gcd(24, 30) = 6, gcd(5, 7) = 1, and gcd(0, 9) = 9. If a and b are not both 0, then gcd(a, b) is an integer between 1 and min(|a|, |b|). We define gcd(0, 0) to be 0; this definition is necessary to make standard properties of the gcd function (such as equation (33.11) below) universally valid.

The following are elementary properties of the gcd function:

gcd(a,b) = gcd(b,a),

(33.8)

gcd(a,b) = gcd(-a,b),

(33.9)

gcd(a,b) = gcd(|a| , |b|),

(33.10)

gcd(a,0) = |a|,

(33.11)

gcd(a,ka) = |a|       for any k    Z.

(33.12)

Theorem 33.2

If a and b are any integers, not both zero, then gcd(a, b) is the smallest positive element of the set {ax + by : x, y [image] Z}of linear combinations of a and b.

Proof Let s be the smallest positive such linear combination of a and b, and let s = ax + by for some x, y [image] Z. Let q = [image] a/s [image] . Equation (33.2) then implies

a mod s  =  a - qs

=  a - q(ax + by)

=  a(1 - qx) + b(-qy),

and thus a mod s is a linear combination of a and b as well. But, since a mod s < s, we have that a mod s = 0, because s is the smallest positive such linear combination. Therefore, s | a and, by analogous reasoning, s | b. Thus, s is a common divisor of a and b, and so gcd(a, b) [image] s. Equation (33.6) implies that gcd(a, b) | s, since gcd(a, b) divides both a and b and s is a linear combination of a and b. But gcd(a, b) | s and s > 0 imply that gcd(a, b) [image] s. Combining gcd(a, b) [image] s and gcd(a, b) [image] s yields gcd(a, b) = s; we conclude that s is the greatest common divisor of a and b.

Corollary 33.3

For any integers a and b, if d | a and d | b then d | gcd(a, b) .

Proof This corollary follows from equation (33.6), because gcd(a, b) is a linear combination of a and b by Theorem 33.2.

Corollary 33.4

For all integers a and b and any nonnegative integer n,

gcd(an, bn) = n gcd(a, b).

Proof If n = 0, the corollary is trivial. If n > 0, then gcd(an, bn) is the smallest positive element of the set {anx + bny}, which is n times the smallest positive element of the set {ax + by}.

Corollary 33.5

For all positive integers n, a, and b, if n | ab and gcd(a, n) = 1, then n | b.

Proof The proof is left as Exercise 33.1-4.

Relatively prime integers

Two integers a, b are said to be relatively prime if their only common divisor is 1, that is, if gcd(a, b) = 1. For example, 8 and 15 are relatively prime, since the divisors of 8 are 1, 2, 4, and 8, while the divisors of 15 are 1, 3, 5, and 15. The following theorem states that if two integers are each relatively prime to an integer p, then their product is relatively prime to p.

Theorem 33.6

For any integers a, b, and p, if gcd(a, p) = 1 and gcd(b, p) = 1, then gcd(ab, p) = 1.

Proof It follows from Theorem 33.2 that there exist integers x, y, x', and y' such that

ax + py  =  1,

bx' + py'  =  1.

Multiplying these equations and rearranging, we have

ab(xx') + p(ybx' + y'ax + pyy') = 1.

Since 1 is thus a positive linear combination of ab and p, an appeal to Theorem 33.2 completes the proof.

We say that integers n1, n2, . . ., nk are pairwise relatively prime if, whenever i [image] j, we have gcd(ni, nj) = 1.

Unique factorization

An elementary but important fact about divisibility by primes is the following.

Theorem 33.7

For all primes p and all integers a, b, if p | ab, then p | a or p | b.

Proof Assume for the purpose of contradiction that p | ab but that [image]. Thus, gcd(a, p) = 1 and gcd(b, p) = 1, since the only divisors of p are 1 and p, and by assumption p divides neither a nor b. Theorem 33.6 then implies that gcd(ab, p) = 1, contradicting our assumption that p | ab, since p | ab implies gcd(ab, p) = p. This contradiction completes the proof.

A consequence of Theorem 33.7 is that an integer has a unique factorization into primes.

Theorem 33.8

A composite integer a can be written in exactly one way as a product of the form

where the pi are prime, p1 < p2 < . . . < pr, and the ei are positive integers.

Proof The proof is left Exercise 33.1-10.

As an example, the number 6000 can be uniquely factored as 24.3.53.

Exercises

33.1-1

Prove that there are infinitely many primes. (Hint: Show that none of the primes p1, p2, . . . , pk divide (p1p2 . . . pk) + 1.)

33.1-2

Prove that if a | b and b | c, then a | c.

33.1-3

Prove that if p is prime and 0 < k < p, then gcd (k, p) = 1.

33.1-4

Prove Corollary 33.5.

33.1-5

Prove that if p is prime and 0 < k < p, then [image]. Conclude that for all integers a, b, and primes p,

(a + b)p    ap + bp (mod p).

33.1-6

Prove that if a and b are any integers such that a | b and b > 0, then

(x mod b) mod a = x mod a

for any x. Prove, under the same assumptions, that

x    y (mod b) implies x    y (mod a)

for any integers x and y.

33.1-7

For any integer k > 0, we say that an integer n is a kth power if there exists an integer a such that ak = n. We say that n > 1 is a nontrivial power if it is a kth power for some integer k > 1. Show how to determine if a given [image] -bit integer n is a nontrivial power in time polynomial in [image] .

33.1-8

Prove equations (33.8)--(33.12).

33.1-9

Show that the gcd operator is associative. That is, prove that for all integers a, b, and c,

gcd(a, gcd(b, c)) = gcd(gcd(a, b), c) .

33.1-10

Prove Theorem 33.8.

33.1-11

Give efficient algorithms for the operations of dividing a [image] -bit integer by a shorter integer and of taking the remainder of a [image] -bit integer when divided by a shorter integer. Your algorithms should run in time O( [image] 2).

33.1-12

Give an efficient algorithm to convert a given [image] -bit (binary) integer to a decimal representation. Argue that if multiplication or division of integers whose length is at most [image] takes time M( [image] ), then binary-to-decimal conversion can be performed in time [image] (M( [image] ) lg [image] ). (Hint: Use a divide-and-conquer approach, obtaining the top and bottom halves of the result with separate recursions.)

33.2 Greatest common divisor

In this section, we use Euclid's algorithm to compute the greatest common divisor of two integers efficiently. The analysis of running time brings up a surprising connection with the Fibonacci numbers, which yield a worst-case input for Euclid's algorithm.

We restrict ourselves in this section to nonnegative integers. This restriction is justified by equation (33.10), which states that gcd(a,b) = gcd(|a|, |b|).

In principle, we can compute gcd(a, b) for positive integers a and b from the prime factorizations of a and b. Indeed, if

(33.13)

(33.14)

with zero exponents being used to make the set of primes p1, p2, . . . , pr the same for both a and b, then

(33.15)

As we shall show in Section 33.9, the best algorithms to date for factoring do not run in polynomial time. Thus, this approach to computing greatest common divisors seems unlikely to yield an efficient algorithm.

Euclid's algorithm for computing greatest common divisors is based on the following theorem.

Theorem 33.9

For any nonnegative integer a and any positive integer b,

gcd(a,b) = gcd(b,a mod b) .

Proof We shall show that gcd (a, b) and gcd (b, a mod b) divide each other, so that by equation (33.7) they must be equal (since they are both nonnegative).

We first show that gcd (a, b) | gcd(b, a mod b). If we let d = gcd (a, b), then d | a and d | b. By equation (33.2), (a mod b) = a - qb, where q = [image] a/b [image] . Since (a mod b) is thus a linear combination of a and b, equation (33.6) implies that d | (a mod b). Therefore, since d | b and d | (a mod b), Corollary 33.3 implies that d | gcd(b, a mod b) or, equivalently, that

gcd(a, b) | gcd(b, a mod b).

(33.16)

Showing that gcd (b, a mod b) | gcd(a, b) is almost the same. If we now let d = gcd(b, a mod b), then d | b and d | (a mod b). Since a = qb + (a mod b), where q = [image] a/b [image] , we have that a is a linear combination of b and (a mod b). By equation (33.6), we conclude that d | a. Since d | b and d | a, we have that d | gcd(a, b) by Corollary 33.3 or, equivalently, that

gcd (b,a mod b) | gcd(a,b).

(33.17)

Using equation (33.7) to combine equations (33.16) and (33.17) completes the proof.

Euclid's algorithm

The following gcd algorithm is described in the Elements of Euclid (circa 300 B.C.), although it may be of even earlier origin. It is written as a recursive program based directly on Theorem 33.9. The inputs a and b are arbitrary nonnegative integers.

EUCLID (a, b)

1  if b = 0

2      then return a

3      else return EUCLID(b,a mod b)

As an example of the running of EUCLID, consider the computation of gcd (30,21):

EUCLID(30, 21)  =  EUCLID(21, 9)

=  EUCLID(9, 3)

=  EUCLID(3, 0)

=  3 .

In this computation, there are three recursive invocations of EUCLID.

The correctness of EUCLID follows from Theorem 33.9 and the fact that if the algorithm returns a in line 2, then b = 0, so equation (33.11) implies that gcd(a, b) = gcd(a, 0) = a. The algorithm cannot recurse indefinitely, since the second argument strictly decreases in each recursive call. Therefore, EUCLID always terminates with the correct answer.

The running time of Euclid's algorithm

We analyze the worst-case running time of EUCLID as a function of the size of a and b. We assume with little loss of generality that a > b [image] 0. This assumption can be justified by the observation that if b > a [image] 0, then Euclid (a, b) immediately makes the recursive call EUCLID (b, a). That is, if the first argument is less than the second argument, EUCLID spends one recursive call swapping its arguments and then proceeds. Similarly, if b = a > 0, the procedure terminates after one recursive call, since a mod b = 0.

The overall running time of EUCLID is proportional to the number of recursive calls it makes. Our analysis makes use of the Fibonacci numbers Fk, defined by the recurrence (2.13).

Lemma 33.10

If a > b [image] 0 and the invocation Euclid(a, b) performs k [image] 1 recursive calls, then a [image] Fk+2 and b [image] Fk+l.

Proof The proof is by induction on k. For the basis of the induction, let k = 1. Then, b [image] 1 = F2, and since a > b, we must have a [image] 2 = F3. Since b > (a mod b), in each recursive call the first argument is strictly larger than the second; the assumption that a > b therefore holds for each recursive call.

Assume inductively that the lemma is true if k - 1 recursive calls are made; we shall then prove that it is true for k recursive calls. Since k > 0, we have b > 0, and EUCLID (a, b) calls EUCLID (b, a mod b) recursively, which in turn makes k - 1 recursive calls. The inductive hypothesis then implies that b [image] Fk+ 1 (thus proving part of the lemma), and (a mod b) [image] Fk. We have

b + (a mod b)  =  b + (a -   a/b   b)

    a,

since a > b > 0 implies [image] a/b [image] [image] 1. Thus,

a      b + (a mod b)

    Fk + 1 + Fk

=  Fk+2 .

The following theorem is an immediate corollary of this lemma.

Theorem 33.11

For any integer k [image] 1, if a > b [image] 0 and b < Fk+1, then the invocation EUCLID (a, b) makes fewer than k recursive calls.

We can show that the upper bound of Theorem 33.11 is the best possible. Consecutive Fibonacci numbers are a worst-case input for EUCLID. Since Euclid (F3, F2) makes exactly one recursive call, and since for k [image] 2 we have Fk+1 mod Fk = Fk-1, we also have

gcd(Fk+1,Fk)  =  gcd(Fk, (Fk+1 mod Fk))

=  gcd(Fk, Fk-1) .

Thus, Euclid(Fk+1,Fk) recurses exactly k - 1 times, meeting the upper bound of Theorem 33.11.

Since Fk is approximately [image], where [image] is the golden ratio [image] defined by equation (2.14), the number of recursive calls in EUCLID is O(lg b). (See Exercise 33.2-5 for a tighter bound.) It follows that if EUCLID is applied to two [image] -bit numbers, then it will perform O( [image] ) arithmetic operations and O( [image] 3) bit operations (assuming that multiplication and division of [image] -bit numbers take O( [image] 2) bit operations). Problem 33-2 asks you to show an O( [image] 2) bound on the number of bit operations.

The extended form of Euclid's algorithm

We now rewrite Euclid's algorithm to compute additional useful information. Specifically, we extend the algorithm to compute the integer coefficients x and y such that

d = gcd(a,b) = ax + by .

(33.18)

Note that x and y may be zero or negative. We shall find these coefficients useful later for the computation of modular multiplicative inverses. The

a   b    a/b    d   x     y

-------------------------

99  78   1    3  -11   14

78  21   3    3   3   -11

21  15   1    3   -2   3

15   6   2    3   1    -2

 6   3   2    3   0    1

 3   0   --    3   1    0

Figure 33.1 An example of the operation of EXTENDED-EUCLID on the inputs 99 and 78. Each line shows for one level of the recursion: the values of the inputs a and b, the computed value a/b , and the values d, x, and y returned. The triple (d,x,y) returned becomes the triple (d',x',y') used in the computation at the next higher level of recursion. The call EXTENDED-EUCLID(99, 78) returns (3, -11, 14), so gcd(99, 78) = 3 and gcd(99, 78) = 3 = 99 . (-11) + 78 14.

procedure EXTENDED-EUCLID takes as input an arbitrary pair of integers and returns a triple of the form (d, x, y) that satisfies equation (33.18).

EXTENDED-EUCLID(a, b)

1  f b = 0

2     then return (a, 1, 0)

3  (d',x',y')    EXTENDED-EUCLID(b, a mod b)

4  (d,x,y)    (d',y',x' -   a/b   y')

5  return (d,x,y)

Figure 33.1 illustrates the execution of EXTENDED-EUCLID with the computation of gcd(99,78).

The EXTENDED-EUCLID procedure is a variation of the EUCLID procedure. Line 1 is equivalent to the test "b = 0" in line 1 of EUCLID. If b = 0, then EXTENDED-EUCLID returns not only d = a in line 2, but also the coefficients x = 1 and y = 0, so that a = ax + by. If b [image] 0, EXTENDED-EUCLID first computes (d', x', y') such that d' = gcd(b, a mod b) and

d' = bx' + (a mod b)y' .

(33.19)

As for EUCLID, we have in this case d = gcd(a, b) = d' = gcd(b, a mod b). To obtain x and y such that d = ax + by, we start by rewriting equation (33.19) using the equation d = d' and equation (33.2):

d  =  bx' + (a -   a/b   b)y'

= ay' + b(x' -   a/b   y') .

Thus, choosing x = y' and y = x' - [image] a/b [image] y' satisfies the equation d = ax + by, proving the correctness of EXTENDED-EUCLID.

Since the number of recursive calls made in EUCLID is equal to the number of recursive calls made in EXTENDED-EUCLID, the running times of EUCLID and EXTENDED-EUCLID are the same, to within a constant factor. That is, for a > b > 0, the number of recursive calls is O(lg b).

Exercises

33.2-1

Prove that equations (33.13)--(33.14) imply equation (33.15).

33.2-2

Compute the values (d, x, y) that are output by the invocation EXTENDED-EUCLID(899, 493).

33.2-3

Prove that for all integers a, k, and n,

gcd(a, n) = gcd(a + kn, n).

33.2-4

Rewrite EUCLID in an iterative form that uses only a constant amount of memory (that is, stores only a constant number of integer values).

33.2-5

If a > b [image] 0, show that the invocation EUCLID(a, b) makes at most 1 + log [image] b recursive calls. Improve this bound to 1 +log [image] (b/gcd(a, b)).

33.2-6

What does EXTENDED-EUCLID(Fk + 1, Fk) return? Prove your answer correct.

33.2-7

Verify the output (d, x, y) of EXTENDED-EUCLID(a, b) by showing that if d | a, d | b, and d = ax + by, then d = gcd(a, b).

33.2-8

Define the gcd function for more than two arguments by the recursive equation gcd(a0, al, . . . , n) = gcd(a0, gcd(a1, . . . , an)). Show that gcd returns the same answer independent of the order in which its arguments are specified. Show how to find x0, x1, . . . . , xn such that gcd(a0, a1, . . . , an) = a0x0 + a1x1 + . . . + anxn. Show that the number of divisions performed by your algorithm is O(n + 1g(maxi, ai)).

33.2-9

Define lcm(a1, a2, . . . , an) to be the least common multiple of the integers a1, a2, . . . , an, that is, the least nonnegative integer that is a multiple of each ai. Show how to compute lcm(a1, a2, . . . , an) efficiently using the (two-argument) gcd operation as a subroutine.

33.2-10

Prove that n1, n2, n3, and n4 are pairwise relatively prime if and only if gcd(n1n2, n3n4) = gcd(n1n3, n2n4) = 1. Show more generally that n1, n2, . . . , nk are pairwise relatively prime if and only if a set of [image] lg k [image] pairs of numbers derived from the ni are relatively prime.

33.3 Modular arithmetic

Informally, we can think of modular arithmetic as arithmetic as usual over the integers, except that if we are working modulo n, then every result x is replaced by the element of {0, 1, . . . , n - 1} that is equivalent to x, modulo n (that is, x is replaced by x mod n). This informal model is sufficient if we stick to the operations of addition, subtraction, and multiplication. A more formal model for modular arithmetic, which we now give, is best described within the framework of group theory.

Finite groups

A group (S, [image]) is a set S together with a binary operation [image] defined on S for which the following properties hold.

1. Closure: For all a, b [image] S, we have a [image] b [image] S.

2. Identity: There is an element e [image] S such that e [image] a = a [image] e = a for all a [image] S.

3. Associativity: For all a, b, c [image] S, we have (a [image] b) [image] c = a [image] (b [image] c).

4. Inverses: For each a [image] S, there exists a unique element b [image] S such that a [image] b = b [image] a = e.

As an example, consider the familiar group (Z, +) of the integers Z under the operation of addition: 0 is the identity, and the inverse of a is -a. If a group (S, [image]) satisfies the commutative law a [image] b = b [image] a for all a, b [image] S, then it is an abelian group. If a group (S, [image]) satisfies |S| < [image], then it is a finite group.

The groups defined by modular addition and multiplication

We can form two finite abelian groups by using addition and multiplication modulo n, where n is a positive integer. These groups are based on the equivalence classes of the integers modulo n, defined in Section 33.1.

To define a group on Z n, we need to have suitable binary operations, which we obtain by redefining the ordinary operations of addition and multiplication. It is easy to define addition and multiplication operations for Z n, because the equivalence class of two integers uniquely determines the equivalence class of their sum or product. That is, if a [image] a' (mod n) and b [image] b' (mod n), then

a + b      a' + b'  (mod n) ,

ab      a'b'    (mod n) .

+6 0  1  2  3  4  5      15   1   2   4   7   8   11   13   14

--------------------   -------------------------------------

0  0  1  2  3  4  5     1   1   2   4   7   8   11   13   14

1  0  2  3  4  5  0     2   2   4   8  14   1    7   11   13

2  0  3  4  5  0  1     4   4   8   1  13   2   14    7   11

3  0  4  5  0  1  2     7   7  14  13   4  11    2    1    8

4  0  5  0  1  2  3     8   8   1   2  11   4   13   14    7

5  5  0  1  2  3  4    11  11   7  14   2  13    1    8    4

                       13  13  11   7   1  14    8    4    2

                       14  14  13  11   8   7    4    2    1

        (a)                            (b)

Figure 33.2 Two finite groups. Equivalence classes are denoted by their representative elements. (a) The group (Z6, +6). (b) The group

Thus, we define addition and multiplication modulo n, denoted +n and .n, as follows:

[a]n + n [b]n  =  [a + b]n ,

[a]n    n[b]n  =  [ab]n .

(Subtraction can be similarly defined on Z n by [a]n - n [b]n = [a - b]n, but division is more complicated, as we shall see.) These facts justify the common and convenient practice of using the least nonnegative element of each equivalence class as its representative when performing computations in Z n. Addition, subtraction, and multiplication are performed as usual on the representatives, but each result x is replaced by the representative of its class (that is, by x mod n).

Using this definition of addition modulo n, we define the additive group modulo n as (Z n, + n). This size of the additive group modulo n is |Zn| = n. Figure 33.2(a) gives the operation table for the group (Z6, + 6).

Theorem 33.12

The system (Z n, + n) is a finite abelian group.

Proof Associativity and commutativity of +n follow from the associativity and commutativity of +:

([a]n + n[b]n) +n[c]n  =  [(a + b) + c]n

=  [a + (b + c)]n

=  [a]n  + n([b]n + n[c]n),

[a]n + n[b]n  =  [a + b]n

=  [b + a]n

=  [b]n + n[a]n .

The identity element of (Z n,+n) is 0 (that is, [0]n). The (additive) inverse of an element a (that is, [a]n) is the element -a (that is, [-a]n or [n - a]n), since [a]n +n [-a]n=[a - a]n=[0]n.

Using the definition of multiplication modulo n, we define the multiplicative group modulo n as   [image]. The elements of this group are the set   [image]  of elements in Z n that are relatively prime to n:

To see that   [image]  is well defined, note that for 0   [image]  a < n, we have a   [image]  (a+kn) (mod n) for all integers k. By Exercise 33.2-3, therefore, gcd(a, n) = 1 implies gcd(a+kn, n) = 1 for all integers k. Since [a]n = {a + kn : k   [image]  Z}, the set   [image]  is well defined. An example of such a group is

where the group operation is multiplication modulo 15. (Here we denote an element [a]15 as a.) Figure 33.2(b) shows the group   [image]. For example, 8   [image]  11   [image]  13 (mod 15), working in   [image]. The identity for this group is 1.

Theorem 33.13

The system   [image]  is a finite abelian group.

Proof     Theorem 33.6 implies that   [image]  is closed. Associativity and commutativity can be proved for .n as they were for +n in the proof of Theorem 33.12. The identity element is [1]n. To show the existence of inverses, let a be an element of   [image]  and let (d, x, y) be the output of EXTENDED-EUCLID(a, n). Then d = 1, since a   [image]   [image], and

ax + ny = 1

or, equivalently,

ax    1 (mod n).

Thus, [x]n is a multiplicative inverse of [a]n, modulo n. The proof that inverses are uniquely defined is deferred until Corollary 33.26.

When working with the groups (Z n,+n) and (Z n,   [image] n) in the remainder of this chapter, we follow the convenient practice of denoting equivalence classes by their representative elements and denoting the operations +n and .n by the usual arithmetic notations + and   [image]  (or juxtaposition) respectively. Also, equivalences modulo n may also be interpreted as equations in Z n. For example, the following two statements are equivalent:

ax      b (mod n) ,

[a]n  n [x]n  =  [b]n .

As a further convenience, we sometimes refer to a group (S,   [image]) merely as S when the operation is understood from context. We may thus refer to the groups (Z n, + n) and   [image]  as Zn and   [image]  respectively.

The (multiplicative) inverse of an element a is denoted (a-1 mod n). Division in   [image]  is defined by the equation a/b   [image]  ab-1 (mod n). For example, in   [image]  we have that 7-1   [image]  13 (mod 15), since 7 . 13   [image]  91   [image]  1 (mod 15), so that 4/7   [image]  4 . 13   [image]  7 (mod 15).

The size of   [image]  is denoted   [image]  (n). This function, known as Euler's phi function, satisfies the equation

(33.20)

where p runs over all the primes dividing n (including n itself, if n is prime). We shall not prove this formula here. Intuitively, we begin with a list of the n remainders {0, 1, . . . , n - 1 } and then, for each prime p that divides n, cross out every multiple of p in the list. For example, since the prime divisors of 45 are 3 and 5,

If p is prime, then   [image], and

  (p) = p - 1.

(33.21)

If n is composite, then   [image] (n) < n - 1.

Subgroups

If (S,   [image]) is a group, S'   [image]  S, and (S',   [image]) is also a group, then (S',   [image]) is said to be a subgroup of (S,   [image]). For example, the even integers form a subgroup of the integers under the operation of addition. The following theorem provides a useful tool for recognizing subgroups.

Theorem 33.14

If (S,   [image]) is a finite group and S' is any subset of S such that a   [image]  b   [image]  S' for all a, b   [image]  S', then (S',  [image]) is a subgroup of (S,   [image]).

Proof     The proof is left as Exercise 33.3-2.

For example, the set {0, 2, 4, 6} forms a subgroup of Z8,since it is closed under the operation + (that is, it is closed under +8).

The following theorem provides an extremely useful constraint on the size of a subgroup.

Theorem 33.15

If (S,   [image]) is a finite group and (S,   [image]) is a subgroup of (S,   [image]), then |S'| is a divisor of |S|.

A subgroup S' of a group S is said to be a proper subgroup if S'   [image]  S. The following corollary will be used in our analysis of the Miller-Rabin primality test procedure in Section 33.8.

Corollary 33.16

If S' is a proper subgroup of a finite group S, then |S'|   [image]  |S|/2.

Subgroups generated by an element

Theorem 33.14 provides an interesting way to produce a subgroup of a finite group (S,   [image]): choose an element a and take all elements that can be generated from a using the group operation. Specifically, define a(k) for k   [image]  1 by

For example, if we take a = 2 in the group Z6, the sequence a(1), a(2), . . . is

2, 4, 0, 2, 4, 0, 2, 4, 0,... .

In the group Z n, we have a(k) = ka mod n, and in the group   [image], we have a(k) = ak mod n. The subgroup generated by a, denoted   [image] a  [image]  or (  [image] a  [image] ,   [image]), is defined by

  a   = {a(k): k    1}.

We say that a generates the subgroup   [image] a  [image]  or that a is a generator of   [image] a  [image]. Since S is finite,   [image] a  [image]  is a finite subset of S, possibly including all of S. Since the associativity of   [image]  implies

a(i)    a(j) = a(i+j),

  a  [image]  is closed and therefore, by Theorem 33.14,   [image] a  [image]  is a subgroup of S. For example, in Z6,we have

  0    =  {0} ,

  1    =  {0, 1, 2, 3, 4, 5} ,

  2    =  {0, 2, 4} .

Similarly, in   [image], we have

  1    =  {1} ,

  2    =  {1, 2, 4} ,

  3    =  {1, 2, 3, 4, 5, 6} .

The order of a (in the group S), denoted ord(a), is defined as the least t > 0 such that a(t) = e.

Theorem 33.17

For any finite group (S,   [image]) and any a   [image]  S, the order of an element is equal to the size of the subgroup it generates, or ord(a) = |  [image] a  [image] |.

Proof     Let t = ord(a). Since a(t) = e and a(t+k) = a(t)   [image]  a(k) = a(k) for k   [image]  1, if i > t, then a(i) = a(j) for some j < i. Thus, no new elements are seen after a(t), and   [image] a  [image]  = {a(1),a(2),...,a(t)}. To show that   [image]   [image] a  [image]   [image]  = t, suppose for the purpose of contradiction that a(i) = a(j) for some i, j satisfying 1   [image]  i < j    t. Then, a(i+k) = a(j+k) for k    0. But this implies that a(i+(t-j)) = a(j+(t-j)) = e, a contradiction, since i + (t - j) < t but t is the least positive value such that a(t) = e. Therefore, each element of the sequence a(1),a(2), . . . , a(t) is distinct, and     a     = t.

Corollary 33.18

The sequence a(1), a(2), . . . is periodic with period t = ord(a); that is, a(i) = a(j) if and only if i    j (mod t).

It is consistent with the above corollary to define a(0) as e and a(i) as a(i mod t) for all integers i.

Corollary 33.19

If (S,    ) is a finite group with identity e, then for all a    S,

a(|S|) = e.

Proof     Lagrange's theorem implies that ord(a) | |S|, and so |S|    0 (mod t), where t = ord(a).

Exercises

33.3-1

Draw the group operation tables for the groups (Z4, +4) and   . Show that these groups are isomorphic by exhibiting a one-to-one correspondence    between their elements such that a + b    c (mod 4) if and only if   (a).   (b)      (c) (mod 5).

33.3-2

Prove Theorem 33.14.

33.3-3

Show that if p is prime and e is a positive integer, then

  (pe) = pe-1(p - 1) .

33.3-4

Show that for any n > 1 and for any   , the function fa:    defined by fa(x) = ax mod n is a permutation of   .

33.3-5

List all of the subgroups of Z9 and of   .

33.4 Solving modular linear equations

We now consider the problem of finding solutions to the equation

ax    b (mod n),

(33.22)

where n > 0, an important practical problem. We assume that a, b, and n are given, and we are to find the x's, modulo n, that satisfy equation (33.22). There may be zero, one, or more than one such solution.

Let   a   denote the subgroup of Z n generated by a. Since   a   = {a(x): x > 0} = {ax mod n: x > 0}, equation (33.22) has a solution if and only if b      a  . Lagrange's theorem (Theorem 33.15) tells us that |  a  | must be a divisor of n. The following theorem gives us a precise characterization of   a  .

Theorem 33.20

For any positive integers a and n, if d = gcd(a, n), then

  a   =   d   = {0, d, 2d,..., ((n/d)- l)d},

(33.23)

and thus

|  a  | = n/d .

Proof     We begin by showing that d      a  . Observe that EXTENDED EUCLID(a, n) produces integers x' and y' such that ax' + ny' = d. Thus, ax'    d (mod n), so that d      a  .

Since d      a  , it follows that every multiple of d belongs to   a  , because a multiple of a multiple of a is a multiple of a. Thus,   a   contains every element in {0, d, 2d, . . . ,((n/d ) - l )d}. That is,   d        a   .

We now show that   a        d  . If m      a  , then m = ax mod n for some integer x, and so m = ax + ny for some integer y. However, d | a and d | n, and so d | m by equation (33.6). Therefore, m      d  .

Combining these results, we have that   a   =   d  . To see that |  (a  | = n/d, observe that there are exactly n/d multiples of d between 0 and n - 1, inclusive.

Corollary 33.21

The equation ax    b (mod n) is solvable for the unknown x if and only if gcd(a, n) | b.

Corollary 33.22

The equation ax    b (mod n) either has d distinct solutions modulo n, where d = gcd(a, n), or it has no solutions.

Proof     If ax    b (mod n) has a solution, then b      a  . The sequence ai mod n, for i = 0,1, . . . , is periodic with period |  a  | = n/d, by Corollary 33.18. If b      a  , then b appears exactly d times in the sequence ai mod n, for i = 0,1, . . . , n - 1, since the length-(n/d) block of values ( a) is repeated exactly d times as i increases from 0 to n - 1. The indices x of these d positions are the solutions of the equation ax    b (mod n).

Theorem 33.23

Let d = gcd(a, n), and suppose that d = ax' + ny' for some integers x' and y' (for example, as computed by EXTENDED EUCLID). If d | b, then the equation ax    b (mod n) has as one of its solutions the value x0, where

x0 = x'(b/d) mod n.

Proof     Since ax'    d (mod n), we have

ax0      ax'(b/d)  (mod n)

    d(b/d)    (mod n)

    b         (mod n) ,

and thus x0 is a solution to ax    b (mod n).

Theorem 33.24

Suppose that the equation ax    b (mod n) is solvable (that is, d | b, where d = gcd(a, n)) and that x0 is any solution to this equation. Then, this equation has exactly d distinct solutions, modulo n, given by xi = x0 + i(n/d) for i = 1, 2, . . . , d - 1.

Proof     Since n/d > 0 and 0    i( n/d) < n for i = 0, 1, . . . , d - 1, the values x 0,x 1, . . . , xd-1 are all distinct, modulo n. By the periodicity of the sequence ai mod n (Corollary 33.18), if x 0 is a solution of ax    b (mod n), then every xi is a solution. By Corollary 33.22, there are exactly d solutions, so that x 0,x l, . . . , xd-1 must be all of them.

We have now developed the mathematics needed to solve the equation ax    b (mod n); the following algorithm prints all solutions to this equation. The inputs a and b are arbitrary integers, and n is an arbitrary positive integer.

MODULAR-LINEAR-EQUATION-SOLVER(a,b,n)

1  (d,x',y')    EXTENDED-EUCLID(a,n)

2  if d | b

3      then x0    x'(b/d) mod n

4           for i    0 to d - 1

5                do print (x0 + i(n/d)) mod n

6      else print "no solutions"

As an example of the operation of this procedure, consider the equation 14x    30 (mod 100) (here, a = 14, b = 30, and n = 100). Calling EXTENDED-EUCLID in line 1, we obtain (d, x, y) = (2,-7,1). Since 2    30, lines 3-5 are executed. In line 3, we compute x0 = (-7)(15) mod 100 = 95. The loop on lines 4-5 prints the two solutions: 95 and 45.

The procedure MODULAR-LINEAR-EQUATION-SOLVER works as follows. Line 1 computes d = gcd(a, n) as well as two values x'' and y'' such that d = ax' + ny', demonstrating that x'' is a solution to the equation ax'    d (mod n). If d does not divide b, then the equation ax    b (mod n) has no solution, by Corollary 33.21. Line 2 checks if d    b; if not, line 6 reports that there are no solutions. Otherwise, line 3 computes a solution x0 to equation (33.22), in accordance with Theorem 33.23. Given one solution, Theorem 33.24 states that the other d - 1 solutions can be obtained by adding multiples of (n/d), modulo n. The for loop of lines 4-5 prints out all d solutions, beginning with x0 and spaced (n/d) apart, modulo n.

The running time of MODULAR-LINEAR-EQUATION-SOLVER is O(lg n + gcd(a,n)) arithmetic operations, since EXTENDED-EUCLID takes O(lg n) arithmetic operations, and each iteration of the for loop of lines 4-5 takes a constant number of arithmetic operations.

The following corollaries of Theorem 33.24 give specializations of particular interest.

Corollary 33.25

For any n > 1, if gcd(a,n) = 1, then the equation ax    b (mod n) has a unique solution modulo n.

If b = 1, a common case of considerable interest, the x we are looking for is a multiplicative inverse of a, modulo n.

Corollary 33.26

For any n > 1, if gcd(a, n) = 1, then the equation

ax    1 (mod n)

(33.24)

has a unique solution, modulo n. Otherwise, it has no solution.

Corollary 33.26 allows us to use the notation (a-1 mod n) to refer to the multiplicative inverse of a, modulo n, when a and n are relatively prime. If gcd(a, n) = 1, then one solution to the equation ax    1 (mod n) is the integer x returned by EXTENDED-EUCLID, since the equation

gcd(a, n) = 1 = ax + ny

implies ax    1 (mod n). Thus, (a-1 mod n) can be computed efficiently using EXTENDED-EUCLID.

Exercises

33.4-1

Find all solutions to the equation 35x    10 (mod 50).

33.4-2

Prove that the equation ax    ay (mod n) implies x    y (mod n) whenever gcd(a, n) = 1. Show that the condition gcd(a, n) = 1 is necessary by supplying a counterexample with gcd(a, n) > 1.

33.4-3

Consider the following change to line 3 of MODULAR-LINEAR-EQUATiON-SOLVER:

3  then x0    x'(b/d) mod (n/d)

Will this work? Explain why or why not.

33.4-4

Let f(x)    f0 + f1x + + ftx + . . . + ftxt (mod p) be a polynomial of degree t, with coefficients fi drawn from Z p, where p is prime. We say that a    Z p is a zero of f if f(a)    0 (mod p). Prove that if a is a zero of f, then f (x)    (x - a)g(x) (mod p) for some polynomial g(x) of degree t - 1. Prove by induction on t that a polynomial f (x) of degree t can have at most t distinct zeros modulo a prime p.

33.5 The Chinese remainder theorem

Around A.D. 100, the Chinese mathematician    solved the problem of finding those integers x that leave remainders 2, 3, and 2 when divided by 3, 5, and 7 respectively. One such solution is x = 23; all solutions are of the form 23 + 105k for arbitrary integers k. The "Chinese remainder theorem" provides a correspondence between a system of equations modulo a set of pairwise relatively prime moduli (for example, 3, 5, and 7) and an equation modulo their product (for example, 105).

The Chinese remainder theorem has two major uses. Let the integer n = nl n2 . . . nk, where the factors ni are pairwise relatively prime. First, the Chinese remainder theorem is a descriptive "structure theorem" that describes the structure of Z n as identical to that of the Cartesian product Z n1 XZ n2 X . . . X Z nk with componentwise addition and multiplication modulo ni in the ith component. Second, this description can often be used to yield efficient algorithms, since working in each of the systems Z ni can be more efficient (in terms of bit operations) than working modulo n.

Theorem 33.27

Let n = n1 n2 . . . nk, where the ni are pairwise relatively prime. Consider the correspondence

a    (al, a2, . . . , ak),

(33.25)

where a    Z n, ai    Z ni, and

ai = a mod ni

for i = 1, 2, . . . , k. Then, mapping (33.25) is a one-to-one correspondence (bijection) between Z n and the Cartesian product Z niXZ n2 X . . . X Z nk. Operations performed on the elements of Z n can be equivalently performed on the corresponding k-tuples by performing the operations independently in each coordinate position in the appropriate system. That is, if

a      (al, a2, . . . , ak) ,

b      (b1, b2, . . . , bk) ,

then

(a + b) mod n    ((al + bl) mod n1, . . . , (ak + bk) mod nk) ,

(33.26)

(a - b) mod n    ((al - bl) mod nl, . . . , (ak - bk) mod nk) ,

(33.27)

(ab) mod n    (a1b1 mod nl, . . . , akbk mod nk) .

(33.28)

Proof     Transforming between the two representations is quite straight-forward. Going from a to (al, a2, . . . , ak) requires only k divisions. Computing a from inputs (al, a2, . . . , ak) is almost as easy, using the following formula. Let mi = n/ni for i = 1,2, . . . , k. Note that mi = nln2 . . .  ni-lni+l . . .  nk, so that mi    0 (mod nj) for all j    i. Then, letting

ci = mi(mj-l mod ni)

(33.29)

for i = 1, 2, . . . , k, we have

a    (alc1 + a2c2 + . . . + akck) (mod n) .

(33.30)

Equation 33.29 is well defined, since mi and ni are relatively prime (by Theorem 33.6), and so Corollary 33.26 implies that (mi-l mod ni) is defined. To verify equation (33.30), note that cj    mj    0 (mod ni) if j    i, and that ci    1 (mod ni). Thus, we have the correspondence

ci    (0, 0, . . . , 0, 1, 0, . . . , 0) ,

a vector that has 0's everywhere except in the ith coordinate, where it has a 1. The ci thus form a "basis" for the representation, in a certain sense. For each i, therefore, we have

Since we can transform in both directions, the correspondence is one-to-one. Equations (33.26)--(33.28) follow directly from Exercise 33.1-6, since x mod ni = (x mod n) mod ni for any x and i = 1, 2, . . . , k.

The following corollaries will be used later in this chapter.

Corollary 33.28

If n1, n2, . . . , nk are pairwise relatively prime and n = n1n2...nk, then for any integers a1, a2, . . . , ak, the set of simultaneous equations

x    ai   (mod ni),

for i = 1, 2, . . . , k, has a unique solution modulo n for the unknown x.

Corollary 33.29

If n1, n2, . . . , nk are pairwise relatively prime and n = n1 n2 . . . nk, then for all integers x and a,

x    a  (mod ni)

for i = 1, 2, . . . , k if and only if

x    a  (mod n) .

As an example of the Chinese remainder theorem, suppose we are given the two equations

a      2  (mod 5) ,

a      3  (mod 13) ,

so that a1 = 2, n1 = m2 = 5, a2 = 3, and n2 = m1 = 13, and we wish to compute a mod 65, since n = 65. Because 13-1    2 (mod 5) and 5-1    8 (mod 13), we have

c1  =  13(2 mod 5)  =  26 ,

c2  =  5(8 mod 13)  =  40 ,

and

a       2 . 26 + 3    40  (mod 65)

    52 + 120         (mod 65)

    42               (mod 65) .

    0   1   2   3   4   5   6   7   8   9  10  11  12

------------------------------- ---------------------

0   0  40  15  55  30   5  45  20  60  35  10  50  25

1  26   1  41  16  56  31   6  46  21  61  36  11  51

2  52  27   2  42  17  57  32   7  47  22  62  37  12

3  13  53  28   3  43  18  58  33   8  48  23  63  38

4  39  14  54  29   4  44  19  59  34   9  49  24  64

Figure 33.3 An illustration of the Chinese remainder theorem for n1 = 5 and n2 = 13. For this example, c1 = 26 and c2 = 40. In row i, column j is shown the value of a, modulo 65, such that (a mod 5) = i and (a mod 13) = j. Note that row 0, column 0 contains a 0. Similarly, row 4, column 12 contains a 64 (equivalent to - 1). Since c1 = 26, moving down a row increases a by 26. Similarly, c2 = 40 means that moving right a column increases a by 40. Increasing a by 1 corresponds to moving diagonally downward and to the right, wrapping around from the bottom to the top and from the right to the left.

See Figure 33.3 for an illustration of the Chinese remainder theorem, modulo 65.

Thus, we can work modulo n by working modulo n directly or by working in the transformed representation using separate modulo ni computations, as convenient. The computations are entirely equivalent.

Exercises

33.5-1

Find all solutions to the equations x    4 (mod 5) and x    5  (mod 11).

33.5-2

Find all integers x that leave remainders 1, 2, 3, 4, 5 when divided by 2, 3, 4, 5, 6, respectively.

33.5-3

Argue that, under the definitions of Theorem 33.27, if gcd(a, n) = 1, then

33.5-4

Under the definitions of Theorem 33.27, prove that the number of roots of the equation f(x)    0 (mod n) is equal to the product of the number of roots of each the equations f(x)    0 (mod n1), f(x)    0 (mod n2), . . . , f(x)    0 (mod nk).

33.6 Powers of an element

Just as it is natural to consider the multiples of a given element a, modulo n, it is often natural to consider the sequence of powers of a, modulo n, where   :

a0,a1,a2,a3,... ,

(33.31)

modulo n. Indexing from 0, the 0th value in this sequence is a0 mod n = 1, and the ith value is ai mod n. For example, the powers of 3 modulo 7 are

    i     0  1  2  3  4  5  6  7  8  9  10  11  ...

---------------------------------------------------

3i mod 7  1  3  2  6  4  5  1  3  2  6   4   5  ...

whereas the powers of 2 modulo 7 are

    i     0  1  2  3  4  5  6  7  8  9  10  11  ...

---------------------------------------------------

2i mod 7  1  2  4  1  2  4  1  2  4  1   2   4  ...

In this section, let   a   denote the subgroup of    generated by a, and let ordn (a) (the "order of a, modulo n") denote the order of a in   . For example,   2   = {1, 2, 4} in   , and ord7 (2) = 3. Using the definition of the Euler phi function   (n) as the size of    (see Section 33.3), we now translate Corollary 33.19 into the notation of    to obtain Euler's theorem and specialize it to   , where p is prime, to obtain Fermat's theorem.

Theorem 33.30

For any integer n > 1,

(33.32)

Theorem 33.31

If p is prime, then

(33.33)

Proof     By equation (33.21),   (p) = p - 1 if p is prime.

This corollary applies to every element in Z p except 0, since   . For all a    Z p, however, we have ap    a (mod p) if p is prime.

If   , then every element in    is a power of g, modulo n, and we say that g is a primitive root or a generator of   . For example, 3 is a primitive root, modulo 7. If    possesses a primitive root, we say that the group    is cyclic. We omit the proof of the following theorem, which is proven by Niven and Zuckerman [151].

Theorem 33.32

The values of n > 1 for which    is cyclic are 2, 4, pe, and 2pe, for all odd primes p and all positive integers e.

If g is a primitive root of    and a is any element of   , then there exists a z such that gz    a (mod n). This z is called the discrete logarithm or index of a, modulo n, to the base g; we denote this value as indn,g(a).

Theorem 33.33

If g is a primitive root of   , then the equation gx    gy (mod n) holds if and only if the equation x    y (mod   (n)) holds.

Proof     Suppose first that x    y (mod   (n)). Then, x = y + k  (n) for some integer k. Therefore,

gx      gy+k  (n)        (mod n)

    gy    (g  (n))k  (mod n)

    gy    lk        (mod n)

    gy              (mod n).

Conversely, suppose that gx    gy (mod n). Because the sequence of powers of g generates every element of   g   and |  g  | =   (n), Corollary 33.18 implies that the sequence of powers of g is periodic with period   (n). Therefore, if gx    gy (mod n), then we must have x    y (mod   (n)).

Taking discrete logarithms can sometimes simplify reasoning about a modular equation, as illustrated in the proof of the following theorem.

Theorem 33.34

If p is an odd prime and e    1, then the equation

x2    1 (mod pe)

(33.34)

has only two solutions, namely x = 1 and x = - 1.

Proof     Let n = pe. Theorem 33.32 implies that    has a primitive root g. Equation (33.34) can be written

(gindn,g(x))2    gindn,g(l)  (mod n) .

(33.35)

After noting that indn,g(1) = 0, we observe that Theorem 33.33 implies that equation (33.35) is equivalent to

2    indn,g (x)    0 (mod   (n)) .

(33.36)

To solve this equation for the unknown indn,g(x), we apply the methods of Section 33.4. Letting d = gcd(2,   (n)) = gcd(2, (p - 1)pe-1) = 2, and noting that d | 0, we find from Theorem 33.24 that equation (33.36) has exactly d = 2 solutions. Therefore, equation (33.34) has exactly 2 solutions, which are x = 1 and x = - 1 by inspection.

A number x is a nontrivial square root of 1, modulo n, if it satisfies the equation x2    1 (mod n) but x is equivalent to neither of the two "trivial" square roots: 1 or -1, modulo n. For example, 6 is a nontrivial square root of 1, modulo 35. The following corollary to Theorem 33.34 will be used in the correctness proof for the Miller-Rabin primality-testing procedure in Section 33.8.

Corollary 33.35

If there exists a nontrivial square root of 1, modulo n, then n is composite.

Proof     This corollary is just the contrapositive to Theorem 33.34. If there exists a nontrivial square root of 1, modulo n, then n can't be a prime or a power of a prime.

Raising to powers with repeated squaring

A frequently occurring operation in number-theoretic computations is raising one number to a power modulo another number, also known as modular exponentiation. More precisely, we would like an efficient way to compute ab mod n, where a and b are nonnegative integers and n is a positive integer. Modular exponentiation is also an essential operation in many primality-testing routines and in the RSA public-key cryptosystem. The method of repeated squaring solves this problem efficiently using the binary representation of b.

Let   bk, bk-1, . . . , b1,b0   be the binary representation of b. (That is, the binary representation is k + 1 bits long, bk is the most significant bit, and b0 is the least significant bit.) The following procedure computes ac mod n as c is increased by doublings and incrementations from 0 to b.

MODULAR-EXPONENTIATION(a,b,n)

1  c    0

2  d    1

3  let   bk, bk-1, . . . , b0   be the binary representation of b

4  for i    k downto 0

5       do c    2c

6          d    (d    d) mod n

7          if bi = 1

8             then c    c + 1

9                  d    (d    a) mod n

10 return d

Each exponent computed in a sequence is either twice the previous exponent or one more than the previous exponent; the binary representation

i   9   8   7    6    5    4    3    2    1    0

-------------------------------------------------

bi  1   0   0    0    1    1    0    0    0    0

c   1   2   4    8    17   35   70  140  280  560

d   7  49  157  526  160  241  298  166   67   1

Figure 33.4 The results of MODULAR-EXPONENTIATION when computing ab (mod n), where a = 7, b = 560 =   1000110000  , and n = 561. The values are shown after each execution of the for loop. The final result is 1.

of b is read from right to left to control which operations are performed. Each iteration of the loop uses one of the identities

a2c mod n  =  (ac)2 mod n ,

a2c+l mod n  =  a . (ac)2 mod n ,

depending on whether bi = 0 or 1, respectively. The essential use of squaring in each iteration explains the name "repeated squaring." Just after bit bi is read and processed, the value of c is the same as the prefix   bk, bk-1, . . . , bi   of the binary representation of b. As an example, for a = 7, b = 560, and n = 561, the algorithm computes the sequence of values modulo 561 shown in Figure 33.4; the sequence of exponents used is shown in row c of the table.

The variable c is not really needed by the algorithm but is included for explanatory purposes: the algorithm preserves the invariant that d = ac mod n as it increases c by doublings and incrementations until c = b. If the inputs a, b, and n are   -bit numbers, then the total number of arithmetic operations required is O(  ) and the total number of bit operations required is O(  3).

Exercises

33.6-1

Draw a table showing the order of every element in   . Pick the smallest primitive root g and compute a table giving ind11,g (x) for all x      .

33.6-2

Give a modular exponentiation algorithm that examines the bits of b from right to left instead of left to right.

33.6-3

Explain how to compute a -1 mod n for any    using the procedure MODULAR-EXPONENTIATION, assuming that you know   (n).

33.7 The RSA public-key cryptosystem

A public-key cryptosystem can be used to encrypt messages sent between two communicating parties so that an eavesdropper who overhears the encrypted messages will not be able to decode them. A public-key cryptosystem also enables a party to append an unforgeable "digital signature" to the end of an electronic message. Such a signature is the electronic version of a handwritten signature on a paper document. It can be easily checked by anyone, forged by no one, yet loses its validity if any bit of the message is altered. It therefore provides authentication of both the identity of the signer and the contents of the signed message. It is the perfect tool for electronically signed business contracts, electronic checks, electronic purchase orders, and other electronic communications that must be authenticated.

The RSA public-key cryptosystem is based on the dramatic difference between the ease of finding large prime numbers and the difficulty of factoring the product of two large prime numbers. Section 33.8 describes an efficient procedure for finding large prime numbers, and Section 33.9 discusses the problem of factoring large integers.

Public-key cryptosystems

In a public-key cryptosystem, each participant has both a public key and a secret key. Each key is a piece of information. For example, in the RSA cryptosystem, each key consists of a pair of integers. The participants "Alice" and "Bob" are traditionally used in cryptography examples; we denote their public and secret keys as PA, SA for Alice and PB, SB for Bob.

Each participant creates his own public and secret keys. Each keeps his secret key secret, but he can reveal his public key to anyone or even publish it. In fact, it is often convenient to assume that everyone's public key is available in a public directory, so that any participant can easily obtain the public key of any other participant.

The public and secret keys specify functions that can be applied to any message. Let    denote the set of permissible messages. For example,    might be the set of all finite-length bit sequences. We require that the public and secret keys specify one-to-one functions from    to itself. The function corresponding to Alice's public key PA is denoted PA( ), and the function corresponding to her secret key SA is denoted SA( ). The functions PA( ) and SA( ) are thus permutations of   . We assume that the functions PA( ) and SA( ) are efficiently computable given the corresponding key PA or SA.

The public and secret keys for any participant are a "matched pair" in that they specify functions that are inverses of each other. That is,

M  =  SA(PA(M)) ,

(33.37)

M  =  PA(SA(M))

(33.38)

for any message   . Transforming M with the two keys PA and SA successively, in either order, yields the message M back.

In a public-key cryptosystem, it is essential that no one but Alice be able to compute the function SA( ) in any practical amount of time. The privacy of mail that is encrypted and sent to Alice and the authenticity of Alice's digital signatures rely on the assumption that only Alice is able to compute SA( ). This requirement is why Alice keeps SA secret; if she does not, she loses her uniqueness and the cryptosystem cannot provide her with unique capabilities. The assumption that only Alice can compute SA( ) must hold even though everyone knows PA and can compute PA( ), the inverse function to SA( ), efficiently. The major difficulty in designing a workable public-key cryptosystem is in figuring out how to create a system in which we can reveal a transformation PA( ) without thereby revealing how to compute the corresponding inverse transformation SA( ).

In a public-key cryptosystem, encryption works as follows. Suppose Bob wishes to send Alice a message M encrypted so that it will look like unintelligible gibberish to an eavesdropper. The scenario for sending the message goes as follows.

       Bob obtains Alice's public key PA (from a public directory or directly from Alice).

       Bob computes the ciphertext C = PA(M) corresponding to the message M and sends C to Alice.

       When Alice receives the ciphertext C, she applies her secret key SA to retrieve the original message: M = SA(C).

Figure 33.5 illustrates this process. Because SA( ) and PA( ) are inverse functions, Alice can compute M from C. Because only Alice is able to compute SA( ), only Alice can compute M from C. The encryption of M using PA( ) has protected M from disclosure to anyone except Alice.

Digital signatures are similarly easy to implement in a public-key cryptosystem. Suppose now that Alice wishes to send Bob a digitally signed response M'. The digital-signature scenario proceeds as follows.

       Alice computes her digital signature    for the message M' using her secret key SA and the equation    = SA(M').

       Alice sends the message/signature pair (M',  ) to Bob.

       When Bob receives (M',  ), he can verify that it originated from Alice using Alice's public key by verifying the equation M' = PA(  ). (Presumably, M' contains Alice's name, so Bob knows whose public key to use.) If the equation holds, then Bob concludes that the message M' was actually signed by Alice. If the equation doesn't hold, Bob concludes either that the message M' or the digital signature    was corrupted by transmission errors or that the pair (M',   ) is an attempted forgery.

Figure 33.5 Encryption in a public key system. Bob encrypts the message M using Alice's public key PA and transmits the resulting ciphertext C = PA(M) to Alice. An eavesdropper who captures the transmitted ciphertext gains no information about M. Alice receives C and decrypts it using her secret key to obtain the original message M = SA(C).

Figure 33.6 Digital signatures in a public-key system. Alice signs the message M' by appending her digital signature    = SA(M') to it. She transmits the message/signature pair (M',   ) to Bob, who verifies it by checking the equation M' = PA(  ). If the equation holds, he accepts (M',  ) as a message that has been signed by Alice.

Figure 33.6 illustrates this process. Because a digital signature provides both authentication of the signer's identity and authentication of the contents of the signed message, it is analogous to a handwritten signature at the end of a written document.

An important property of a digital signature is that it is verifiable by anyone who has access to the signer's public key. A signed message can be verified by one party and then passed on to other parties who can also verify the signature. For example, the message might be an electronic check from Alice to Bob. After Bob verifies Alice's signature on the check, he can give the check to his bank, who can then also verify the signature and effect the appropriate funds transfer.

We note that a signed message is not encrypted; the message is "in the clear" and is not protected from disclosure. By composing the above protocols for encryption and for signatures, we can create messages that are both signed and encrypted. The signer first appends his digital signature to the message and then encrypts the resulting message/signature pair with the public key of the intended recipient. The recipient decrypts the received message with his secret key to obtain both the original message and its digital signature. He can then verify the signature using the public key of the signer. The corresponding combined process using paper-based systems is to sign the paper document and then seal the document inside a paper envelope that is opened only by the intended recipient.

The RSA cryptosystem

In the RSA public-key cryptosystem, a participant creates his public and secret keys with the following procedure.

1.     Select at random two large prime numbers p and q. The primes p and q might be, say, 100 decimal digits each.

2.     Compute n by the equation n = pq.

3.     Select a small odd integer e that is relatively prime to   (n), which, by equation (33.20), equals (p-1)(q - 1).

4.     Compute d as the multiplicative inverse of e, modulo   (n). (Corollary 33.26 guarantees that d exists and is uniquely defined.)

5.     Publish the pair P = (e, n,) as his RSA public key.

6.     Keep secret the pair S = (d, n) as his RSA secret key.

For this scheme, the domain    is the set Z n. The transformation of a message M associated with a public key P = (e, n) is

P(M) = Me (mod n) .

(33.39)

The transformation of a ciphertext C associated with a secret key S = (d, n) is

S(C) = Cd (mod n) .

(33.40)

These equations apply to both encryption and signatures. To create a signature, the signer applies his secret key to the message to be signed, rather than to a ciphertext. To verify a signature, the public key of the signer is applied to it, rather than to a message to be encrypted.

The public-key and secret-key operations can be implemented using the procedure MODULAR-EXPONENTIATION described in Section 33.6. To analyze the running time of these operations, assume that the public key (e,n) and secret key (d,n) satisfy |e| = O(1), |d| = |n| =   . Then, applying a public key requires O(1) modular multiplications and uses O(  2) bit operations. Applying a secret key requires O(  ) modular multiplications, using O(  3) bit operations.

Theorem 33.36

The RSA equations (33.39) and (33.40) define inverse transformations of Z n satisfying equations (33.37) and (33.38).

Proof     From equations (33.39) and (33.40), we have that for any M    Zn,

P(S(M)) = S(P(M)) = Med (mod n).

Since e and d are multiplicative inverses modulo   (n) = (p - 1)(q - 1),

ed = 1 + k(p - 1)(q - 1)

for some integer k. But then, if    (mod p), we have (using Theorem 33.31)

Med      M(Mp - 1)k(q - 1)  (mod p)

    M(1)k(q - 1)       (mod p)

    M                (mod p).

Also, Med    M (mod p) if M    0 (mod p). Thus,

Med    M (mod p)

for all M. Similarly,

Med    M (mod q)

for all M. Thus, by Corollary 33.29 to the Chinese remainder theorem,

Med    M (mod n)

for all M.

The security of the RSA cryptosystem rests in large part on the difficulty of factoring large integers. If an adversary can factor the modulus n in a public key, then he can derive the secret key from the public key, using the knowledge of the factors p and q in the same way that the creator of the public key used them. So if factoring large integers is easy, then breaking the RSA cryptosystem is easy. The converse statement, that if factoring large integers is hard, then breaking RSA is hard, is unproven. After a decade of research, however, no easier method has been found to break the RSA public-key cryptosystem than to factor the modulus n. And as we shall see in Section 33.9, the factoring of large integers is surprisingly difficult. By randomly selecting and multiplying together two 100-digit primes, one can create a public key that cannot be "broken" in any feasible amount of time with current technology. In the absence of a fundamental breakthrough in the design of number-theoretic algorithms, the RSA cryptosystem is capable of providing a high degree of security in applications.

In order to achieve security with the RSA cryptosystem, however, it is necessary to work with integers that are 100-200 digits in length, since factoring smaller integers is not impractical. In particular, we must be able to find large primes efficiently, in order to create keys of the necessary length. This problem is addressed in Section 33.8.

For efficiency, RSA is often used in a "hybrid" or "key-management" mode with fast non-public-key cryptosystems. With such a system, the encryption and decryption keys are identical. If Alice wishes to send a long message M to Bob privately, she selects a random key K for the fast non-public-key cryptosystem and encrypts M using K, obtaining ciphertext C. Here, C is as long as M, but K is quite short. Then, she encrypts K using Bob's public RSA key. Since K is short, computing PB(K) is fast (much faster than computing PB(M)). She then transmits (C, PB(K)) to Bob, who decrypts PB(K) to obtain K and then uses K to decrypt C, obtaining M.

A similar hybrid approach is often used to make digital signatures efficiently. In this approach, RSA is combined with a public one-way hash function h--a function that is easy to compute but for which it is computationally infeasible to find two messages M and M' such that h(M) = h(M'). The value h(M) is a short (say, 128-bit) "fingerprint" of the message M. If Alice wishes to sign a message M, she first applies h to M to obtain the fingerprint h(M), which she then signs with her secret key. She sends (M, SA(h(M))) to Bob as her signed version of M. Bob can verify the signature by computing h(M) and verifying that PA applied to SA(h(M)) as received equals h(M). Because no one can create two messages with the same fingerprint, it is impossible to alter a signed message and preserve the validity of the signature.

Finally, we note that the use of certificates makes distributing public keys much easier. For example, assume there is a "trusted authority" T whose public key is known by everyone. Alice can obtain from T a signed message (her certificate) stating that "Alice's public key is PA". This certificate is "self-authenticating" since everyone knows PT. Alice can include her certificate with her signed messages, so that the recipient has Alice's public key immediately available in order to verify her signature. Because her key was signed by T, the recipient knows that Alice's key is really Alice's.

Exercises

33.7-1

Consider an RSA key set with p = 11, q = 29, n = 319, and e = 3. What value of d should be used in the secret key? What is the encryption of the message M = 100?

33.7-2

Prove that if Alice's public exponent e is 3 and an adversary obtains Alice's secret exponent d, then the adversary can factor Alice's modulus n in time polynomial in the number of bits in n. (Although you are not asked to prove it, you may be interested to know that this result remains true even if the condition e = 3 is removed. See Miller [147].)

33.7-3

Prove that RSA is multiplicative in the sense that

PA(M1)PA(M2)    PA(M1M2) (mod n) .

Use this fact to prove that if an adversary had a procedure that could efficiently decrypt 1 percent of messages randomly chosen from Zn and encrypted with PA, then he could employ a probabilistic algorithm to decrypt every message encrypted with PA with high probability.

* 33.8 Primality testing

In this section, we consider the problem of finding large primes. We begin with a discussion of the density of primes, proceed to examine a plausible (but incomplete) approach to primality testing, and then present an effective randomized primality test due to Miller and Rabin.

The density of prime numbers

For many applications (such as cryptography), we need to find large "random" primes. Fortunately, large primes are not too rare, so that it is not too time-consuming to test random integers of the appropriate size until a prime is found. The prime distribution function   (n) specifies the number of primes that are less than or equal to n. For example,   (10) = 4, since there are 4 prime numbers less than or equal to 10, namely, 2, 3, 5, and 7. The prime number theorem gives a useful approximation to   (n).

Theorem 33.37

The approximation n/ ln n gives reasonably accurate estimates of   (n) even for small n. For example, it is off by less than 6% at n = 109, where   (n) = 50,847,478 and n/ ln n = 48,254,942. (To a number theorist, 109 is a small number.)

We can use the prime number theorem to estimate the probability that a randomly chosen integer n will turn out to be prime as 1/ ln n. Thus, we would need to examine approximately ln n integers chosen randomly near n in order to find a prime that is of the same length as n. For example, to find a 100-digit prime might require testing approximately ln 10100    230 randomly chosen 100-digit numbers for primality. (This figure can be cut in half by choosing only odd integers.)

In the remainder of this section, we consider the problem of determining whether or not a large odd integer n is prime. For notational convenience, we assume that n has the prime factorization

(33.41)

where r    1 and p1, p2,. . . ., pr are the prime factors of n. Of course, n is prime if and only if r = 1 and e1 = 1.

One simple approach to the problem of testing for primality is trial division. We try dividing n by each integer   . (Again, even integers greater than 2 may be skipped.) It is easy to see that n is prime if and only if none of the trial divisors divides n. Assuming that each trial division takes constant time, the worst-case running time is   , which is exponential in the length of n. (Recall that if n is encoded in binary using    bits, then    =   lg(n + 1)  , and so   .) Thus, trial division works well only if n is very small or happens to have a small prime factor. When it works, trial division has the advantage that it not only determines whether n is prime or composite but actually determines the prime factorization if n is composite.

In this section, we are interested only in finding out whether a given number n is prime; if n is composite, we are not concerned with finding its prime factorization. As we shall see in Section 33.9, computing the prime factorization of a number is computationally expensive. It is perhaps surprising that it is much easier to tell whether or not a given number is prime than it is to determine the prime factorization of the number if it is not prime.

Pseudoprimality testing

We now consider a method for primality testing that "almost works" and in fact is good enough for many practical applications. A refinement of this method that removes the small defect will be presented later. Let    denote the nonzero elements of Z n:

If n is prime, then   .

We say that n is a base-a pseudoprime if n is composite and

an - 1    1 (mod n).

(33.42)

Fermat's theorem (Theorem 33.31) implies that if n is prime, then n satisfies equation (33.42) for every a in   . Thus, if we can find any    such that n does not satisfy equation (33.42), then n is certainly composite. Surprisingly, the converse almost holds, so that this criterion forms an almost perfect test for primality. We test to see if n satisfies equation (33.42) for a = 2. If not, we declare n to be composite. Otherwise, we output a guess that n is prime (when, in fact, all we know is that n is either prime or a base-2 pseudoprime).

The following procedure pretends in this manner to be checking the primality of n. It uses the procedure MODULAR-EXPONENTIATION from Section 33.6. The input n is assumed to be an integer larger than 2.

This procedure can make errors, but only of one type. That is, if it says that n is composite, then it is always correct. If it says that n is prime, however, then it makes an error only if n is a base-2 pseudoprime.

How often does this procedure err? Surprisingly rarely. There are only 22 values of n less than 10,000 for which it errs; the first four such values are 341, 561, 645, and 1105. It can be shown that the probability that this program makes an error on a randomly chosen   -bit number goes to zero as         . Using more precise estimates due to Pomerance [157] of the number of base-2 pseudoprimes of a given size, we may estimate that a randomly chosen 50-digit number that is called prime by the above procedure has less than one chance in a million of being a base-2 pseudoprime, and a randomly chosen 100-digit number that is called prime has less than one chance in 1013 of being a base-2 pseudoprime.

Unfortunately, we cannot eliminate all the errors by simply checking equation (33.42) for a second base number, say a = 3, because there are composite integers n that satisfy equation (33.42) for all   . These integers are known as Carmichael numbers. The first three Carmichael numbers are 561, 1105, and 1729. Carmichael numbers are extremely rare; there are, for example, only 255 of them less than 100,000,000. Exercise 33.8-2 helps explain why they are so rare.

We next show how to improve our primality test so that it won't be fooled by Carmichael numbers.

The Miller-Rabin randomized primality test

The Miller-Rabin primality test overcomes the problems of the simple test PSEUDOPRIME with two modifications:

       It tries several randomly chosen base values a instead of just one base value.

       While computing each modular exponentiation, it notices if a nontrivial square root of 1, modulo n, is ever discovered. If so, it stops and outputs COMPOSITE. Corollary 33.35 justifies detecting composites in this manner.

The pseudocode for the Miller-Rabin primality test follows. The input n > 2 is the odd number to be tested for primality, and s is the number of randomly chosen base values from    to be tried. The code uses the random-number generator RANDOM from Section 8.3: RANDOM(1, n - 1) returns a randomly chosen integer a satisfying 1    a    n - 1. The code uses an auxiliary procedure WITNESS such that WITNESS(a, n) is TRUE if and only if a is a "witness" to the compositeness of n--that is, if it is possible using a to prove (in a manner that we shall see) that n is composite. The test WITNESS(a, n) is similar to, but more effective than, the test

that formed the basis (using a = 2) for PSEUDOPRIME. We first present and justify the construction of WITNESS, and then show how it is used in the Miller-Rabin primality test.

WITNESS(a,n)

1   let   bk, bk-1,..., b0   be the binary representation of n - 1

2   d    1

3   for i    k downto 0

4         do x    d

5            d    (d    d) mod n

6            if d = 1 and x    1 and x    n - 1

7               then return TRUE

8            if bi = 1

9               then d    (d    a) mod n

10  if d    1

11      then return TRUE

12  return FALSE

This pseudocode for WITNESS is based on the pseudocode of the procedure MODULAR-EXPONENTIATION. Line 1 determines the binary representation of n - 1, which will be used in raising a to the (n - 1)st power. Lines 3-9 compute d as an-1 mod n. The method used is identical to that employed by MODULAR-EXPONENTIATION. Whenever a squaring step is performed on line 5, however, lines 6-7 check to see if a nontrivial square root of 1 has just been discovered. If so, the algorithm stops and returns TRUE. Lines 10-11 return TRUE if the value computed for an - 1 mod n is not equal to 1, just as the PSEUDOPRIME procedure returns COMPOSITE in this case.

We now argue that if WITNESS(a, n) returns TRUE, then a proof that n is composite can be constructed using a.

If WITNESS returns TRUE from line 11, then it has discovered that d = an - 1 mod n    1. If n is prime, however, we have by Fermat's theorem (Theorem 33.31) that an-1    1 (mod n) for all   . Therefore, n cannot be prime, and the equation an - 1 mod 1    1 is a proof of this fact.

If WITNESS returns TRUE from line 7, then it has discovered that x is a nontrivial square root of 1, modulo n, since we have that    (mod n) yet x2    1 (mod n). Corollary 33.35 states that only if n is composite can there be a nontrivial square root of 1 modulo n, so that a demonstration that x is a nontrivial square root of 1 modulo n is a proof that n is composite.

This completes our proof of the correctness of WITNESS. If the invocation WITNESS(a, n) outputs TRUE, then n is surely composite, and a proof that n is composite can be easily determined from a and n. We now examine the Miller-Rabin primality test based on the use of WITNESS.

The procedure MILLER-RABIN is a probabilistic search for a proof that n is composite. The main loop (beginning on line 1) picks s random values of a from    (line 2). If one of the a's picked is a witness to the compositeness of n, then MILLER-RABIN outputs COMPOSITE on line 4. Such an output is always correct, by the correctness of WITNESS. If no witness can be found in s trials, MILLER-RABIN assumes that this is because there are no witnesses to be found, and n is therefore prime. We shall see that this output is likely to be correct if s is large enough, but that there is a small chance that the procedure may be unlucky in its choice of a's and that witnesses do exist even though none has been found.

To illustrate the operation of MILLER-RABIN, let n be the Carmichael number 561. Supposing that a = 7 is chosen as a base, Figure 33.4 shows that WITNESS discovers a nontrivial square root of 1 in the last squaring step, since a280    67 (mod n) and a560    1 (mod n). Therefore, a = 7 is a witness to the compositeness of n, WITNESS(7, n) returns TRUE, and MILLER-RABIN returns COMPOSITE.

If n is a   -bit number, MILLER-RABIN requires O(s  ) arithmetic operations and O(s  3) bit operations, since it requires asymptotically no more work than s modular exponentiations.

Error rate of the Miller-Rabin primality test

If MILLER-RABIN outputs PRIME, then there is a small chance that it has made an error. Unlike PSEUDOPRIME, however, the chance of error does not depend on n; there are no bad inputs for this procedure. Rather, it depends on the size of s and the "luck of the draw" in choosing base values a. Also, since each test is more stringent than a simple check of equation (33.42), we can expect on general principles that the error rate should be small for randomly chosen integers n. The following theorem presents a more precise argument.

Theorem 33.38

If n is an odd composite number, then the number of witnesses to the compositeness of n is at least (n - 1)/2.

Proof     The proof shows that the number of nonwitnesses is no more than (n - 1)/2, which implies the theorem.

We first observe that any nonwitness must be a member of   , since every nonwitness a satisfies an - 1    1 (mod n), yet if gcd(a, n) = d > 1, then there are no solutions x to the equation ax    1 (mod n), by Corollary 33.21. (In particular, x = an - 2 is not a solution.) Thus every member of    is a witness to the compositeness of n.

To complete the proof, we show that the nonwitnesses are all contained in a proper subgroup B of   . By Corollary 33.16, we then have   , we obtain |B|    (n - 1)/2. Therefore, the number of nonwitnesses is at most (n - 1)/2, so that the number of witnesses must be at least (n - 1)/2.

We now show how to find a proper subgroup B of    containing all of the nonwitnesses. We break the proof into two cases.

Case 1: There exists an    such that

(33.43)

Let   . Since B is closed under multiplication modulo n, we have that B is a subgroup of    by Theorem 33.14. Note that every nonwitness belongs to B, since a nonwitness a satisfies an - 1    1 (mod n). Since   , we have that B is a proper subgroup of   .

Case 2: For all   ,

xn - 1    1 (mod n).

(33.44)

In this case, n cannot be a prime power. To see why, let n = pe, where p is an odd prime and e > 1. Theorem 33.32 implies that    contains an element g such that   . But then equation (33.44) and the discrete logarithm theorem (Theorem 33.33, taking y = 0) imply that n - 1    0 (mod   (n)), or

(p - 1) pe - 1 | pe - 1.

This condition fails for e > 1, since the left-hand side is then divisible by p but the right-hand side is not. Thus, n is not a prime power.

Since n is not a prime power, we decompose it into a product n1n2, where n1 and n2 are greater than 1 and relatively prime to each other. (There may be several ways to do this, and it doesn't matter which one we choose. For example, if   , then we can choose   .)

Define t and u so that n - 1 = 2tu, where t    1 and u is odd. For any   , consider the sequence

â =   au, a2u, a22u, . . . , a2tu   ,

(33.45)

where all elements are computed modulo n. Since 2t | n - 1, the binary representation of n - 1 ends in t zeros, and the elements of â are the last t + 1 values of d computed by WITNESS during a computation of an-1 mod n; the last t operations are squarings.

Now find a j    {0, 1, . . . , t} such that there exists a    such that v2ju    -1 (mod n); j should be as large as possible. Such a j certainly exists since u is odd: we can choose v = - 1 and j = 0. Fix    to satisfy the given condition. Let

Since B is closed under multiplication modulo n, it is a subgroup of   . Therefore, |B| divides   . Every nonwitness must be a member of B, since the sequence (33.45) produced by a nonwitness must either be all 1's or else contain a - 1 no later than the jth position, by the maximality of j.

We now use the existence of v to demonstrate that there exists a   . Since v2ju    -1 (mod n), we have v2ju    -1 (mod n1) by Corollary 33.29. By Corollary 33.28, there is a w simultaneously satisfying the equations

w      v (mod n1) ,

w      1 (mod n2) .

Therefore,

w2ju      -1 (mod n1) ,

w2ju      1 (mod n2) .

Together with Corollary 33.29, these equations imply that

(33.46)

and so w    B. Since   , we have that   . Thus,   . We conclude that B is a proper subgroup of   .

In either case, we see that the number of witnesses to the compositeness of n is at least (n - 1)/2.

Theorem 33.39

For any odd integer n > 2 and positive integer s, the probability that MILLER-RABIN(n, s) errs is at most 2-s.

Proof     Using Theorem 33.38, we see that if n is composite, then each execution of the loop of lines 1-4 has a probability of at least 1/2 of discovering a witness x to the compositeness of n. MILLER-RABIN only makes an error if it is so unlucky as to miss discovering a witness to the compositeness of n on each of the s iterations of the main loop. The probability of such a string of misses is at most 2-s.

Thus, choosing s = 50 should suffice for almost any imaginable application. If we are trying to find large primes by applying MILLER-RABIN to randomly chosen large integers, then it can be argued (although we won't do so here) that choosing a small value of s (say 3) is very unlikely to lead to erroneous results. That is, for a randomly chosen odd composite integer n, the expected number of nonwitnesses to the compositeness of n is likely to be much smaller than (n - 1)/2. If the integer n is not chosen randomly, however, the best that can be proven is that the number of nonwitnesses is at most (n - 1)/4, using an improved version of Theorem 33.39. Furthermore, there do exist integers n for which the number of nonwitnesses is (n - 1)/4.

Exercises

33.8-1

Prove that if an integer n > 1 is not a prime or a prime power, then there exists a nontrivial square root of 1 modulo n.

33.8-2

It is possible to strengthen Euler's theorem slightly to the form

where   (n) is defined by

(33.47)

Prove that   (n) |   (n). A composite number n is a Carmichael number if   (n) |n - 1. The smallest Carmichael number is 561 = 3   11   17; here,   (n) = 1cm(2, 10, 16) = 80, which divides 560. Prove that Carmichael numbers must be both "square-free" (not divisible by the square of any prime) and the product of at least three primes. For this reason, they are not very common.

33.8-3

Prove that if x is a nontrivial square root of 1, modulo n, then gcd(x - 1, n) and gcd(x + 1, n) are both nontrivial divisors of n.

* 33.9 Integer factorization

Suppose we have an integer n that we wish to factor, that is, to decompose into a product of primes. The primality test of the preceding section would tell us that n is composite, but it usually doesn't tell us the prime factors of n. Factoring a large integer n seems to be much more difficult than simply determining whether n is prime or composite. It is infeasible with today's supercomputers and the best algorithms to date to factor an arbitrary 200-decimal-digit number.

Pollard's rho heuristic

Trial division by all integers up to B is guaranteed to factor completely any number up to B2. For the same amount of work, the following procedure will factor any number up to B4 (unless we're unlucky). Since the procedure is only a heuristic, neither its running time nor its success is guaranteed, although the procedure is very effective in practice.

POLLARD-RHO(n)

1   i    1

2   x1    RANDOM(0, n - 1)

3   y    x1

4   k    2

5   while TRUE

6       do i    i + l

8          d    gcd(y - xi, n)

9          if d    1 and d    n

10             then print d

11          if i = k

12             then y    xi

13                  k    2k

The procedure works as follows. Lines 1-2 initialize i to 1 and x1 to a randomly chosen value in Z n. The while loop beginning on line 5 iterates forever, searching for factors of n. During each iteration of the while loop, the recurrence

(33.48)

is used on line 7 to produce the next value of xi in the infinite sequence

x1, x2, x3, x4,... ;

(33.49)

the value of i is correspondingly incremented on line 6. The code is written using subscripted variables xi for clarity, but the program works the same if all of the subscripts are dropped, since only the most recent value of xi need be maintained.

Every so often, the program saves the most recently generated xi value in the variable y. Specifically, the values that are saved are the ones whose subscripts are powers of 2:

x1, x2, x4, x8, x16,... .

Line 3 saves the value x1, and line 12 saves xk whenever i is equal to k. The variable k is initialized to 2 in line 4, and k is doubled in line 13 whenever y is updated. Therefore, k follows the sequence 1, 2, 4, 8, . . . and always gives the subscript of the next value xk to be saved in y.

Lines 8-10 try to find a factor of n, using the saved value of y and the current value of xi. Specifically, line 8 computes the greatest common divisor d = gcd(y - xi, n). If d is a nontrivial divisor of n (checked in line 9), then line 10 prints d.

This procedure for finding a factor may seem somewhat mysterious at first. Note, however, that POLLARD-RHO never prints an incorrect answer; any number it prints is a nontrivial divisor of n. POLLARD-RHO may not print anything at all, though; there is no guarantee that it will produce any results. We shall see, however, that there is good reason to expect POLLARD-RHO to print a factor p of n after approximately    iterations of the while loop. Thus, if n is composite, we can expect this procedure to discover enough divisors to factor n completely after approximately n1/4 updates, since every prime factor p of n except possibly the largest one is less than   .

We analyze the behavior of this procedure by studying how long it takes a random sequence modulo n to repeat a value. Since Z n is finite, and since each value in the sequence (33.49) depends only on the previous value, the sequence (33.49) eventually repeats itself. Once we reach an xi such that xi = xj for some j < i, we are in a cycle, since xi+1 = xj+l, xi+2 = xj+2, and so on. The reason for the name "rho heuristic" is that, as Figure 33.7 shows, the sequence x1, x2, . . . , xj-1 can be drawn as the "tail" of the rho, and the cycle xj, xj+l, . . . , xi as the "body" of the rho.

Let us consider the question of how long it takes for the sequence of xi to repeat. This is not exactly what we need, but we shall then see how to modify the argument.

For the purpose of this estimation, let us assume that the function (x2 - 1) mod n behaves like a "random" function. Of course, it is not really random, but this assumption yields results consistent with the observed behavior of POLLARD-RHO. We can then consider each xi to have been independently drawn from Z n according to a uniform distribution on Z n. By the birthday-paradox analysis of Section 6.6.1, the expected number of steps taken before the sequence cycles is   .

Now for the required modification. Let p be a nontrivial factor of n such that gcd(p, n/p) = 1. For example, if n has the factorization   , then we may take p to be   . (If e1 = 1, then p is just the smallest prime factor of n, a good example to keep in mind.) The sequence   xi   induces a corresponding sequence    modulo p, where

for all i. Furthermore, it follows from the Chinese remainder theorem that

(33.50)

since

(x mod n) mod p = x mod p ,

by Exercise 33.1-6.

Figure 33.7 Pollard's rho heuristic. (a) The values produced by the recurrence    mod 1387, starting with x1 = 2. The prime factorization of 1387 is 19    73. The heavy arrows indicate the iteration steps that are executed before the factor 19 is discovered. The light arrows point to unreached values in the iteration, to illustrate the "rho" shape. The shaded values are the y values stored by POLLARD-RHO. The factor 19 is discovered after x7 = 177 is reached, when gcd(63 - 177,1387) = 19 is computed. The first x value that would be repeated is 1186, but the factor 19 is discovered before this value is reached. (b) The values produced by the same recurrence, modulo 19. Every value xi given in part (a) is equivalent, modulo 19, to the value    shown here. For example, both x4 = 63 and x7 = 177 are equivalent to 6, modulo 19. (c) The values produced by the same recurrence, modulo 73. Every value xi given in part (a) is equivalent, modulo 73, to the value    shown here. By the Chinese remainder theorem, each node in part (a) corresponds to a pair of nodes, one from part (b) and one from part (c).

Reasoning as before, we find that the expected number of steps before the sequence    repeats is   . If p is small compared to n, the sequence    may repeat much more quickly than the sequence   xi  . Indeed, the    sequence repeats as soon as two elements of the sequence   xi   are merely equivalent modulo p, rather than equivalent modulo n. See Figure 33.7, parts (b) and (c), for an illustration.

Let t denote the index of the first repeated value in the    sequence, and let u > 0 denote the length of the cycle that has been thereby produced. That is, t and u > 0 are the smallest values such that    for all i    0. By the above arguments, the expected values of t and u are both   . Note that if   , then p | (xt+u+i - xt+i). Thus, gcd(xt+u+i - xt+i,n) > 1.

Therefore, once POLLARD-RHO has saved as y any value xk such that k    t, then y mod p is always on the cycle modulo p. (If a new value is saved as y, that value is also on the cycle modulo p.) Eventually, k is set to a value that is greater than u, and the procedure then makes an entire loop around the cycle modulo p without changing the value of y. A factor of n is then discovered when xi "runs into" the previously stored value of y, modulo p, that is, when xi    y (mod p).

Presumably, the factor found is the factor p, although it may occasionally happen that a multiple of p is discovered. Since the expected values of both t and u are   , the expected number of steps required to produce the factor p is   .

There are two reasons why this algorithm may not perform quite as expected. First, the heuristic analysis of the running time is not rigorous, and it is possible that the cycle of values, modulo p, could be much larger than   . In this case, the algorithm performs correctly but much more slowly than desired. In practice, this seems not to be an issue. Second, the divisors of n produced by this algorithm might always be one of the trivial factors 1 or n. For example, suppose that n = pq, where p and q are prime. It can happen that the values of t and u for p are identical with the values of t and u for q, and thus the factor p is always revealed in the same gcd operation that reveals the factor q. Since both factors are revealed at the same time, the trivial factor pq = n is revealed, which is useless. Again, this seems not to be a real problem in practice. If necessary, the heuristic can be restarted with a different recurrence of the form    mod n. (The values c = 0 and c = 2 should be avoided for reasons we won't go into here, but other values are fine.)

Of course, this analysis is heuristic and not rigorous, since the recurrence is not really "random." Nonetheless, the procedure performs well in practice, and it seems to be as efficient as this heuristic analysis indicates. It is the method of choice for finding small prime factors of a large number. To factor a   -bit composite number n completely, we only need to find all prime factors less than   n1/2  , and so we expect POLLARD-RHO to require at most n1/4 = 2  /4 arithmetic operations and at most n1/4,   3 = 2  /4  3 bit operations. POLLARD-RHO's ability to find a small factor p of n with an expected number    of arithmetic operations is often its most appealing feature.

Exercises

33.9-1

Referring to the execution history shown in Figure 33.7(a), when does POLLARD-RHO print the factor 73 of 1387?

33.9-2

Suppose that we are given a function f : Zn    Zn and an initial value x0    Zn. Define xi = f(xi-1) for i = 1, 2, . . . . Let t and u > 0 be the smallest values such that xt+i = xt+u+i for i = 0, 1, . . . . In the terminology of Pollard's rho algorithm, t is the length of the tail and u is the length of the cycle of the rho. Give an efficient algorithm to determine t and u exactly, and analyze its running time.

33.9-3

How many steps would you expect POLLARD-RHO to require to discover a factor of the form pe, where p is prime and e > 1?

33.9-4

One disadvantage of POLLARD-RHO as written is that it requires one gcd computation for each step of the recurrence. It has been suggested that we might batch the gcd computations by accumulating the product of several xi in a row and then taking the gcd of this product with the saved y. Describe carefully how you would implement this idea, why it works, and what batch size you would pick as the most effective when working on a   -bit number n.

Problems

33-1 Binary gcd algorithm

On most computers, the operations of subtraction, testing the parity (odd or even) of a binary integer, and halving can be performed more quickly than computing remainders. This problem investigates the binary gcd algorithm, which avoids the remainder computations used in Euclid's algorithm.

a.     Prove that if a and b are both even, then gcd(a, b) = 2 gcd(a/2, b/2).

b.     Prove that if a is odd and b is even, then gcd(a, b) = gcd(a, b/2).

c.     Prove that if a and b are both odd, then gcd(a, b) = gcd((a - b)/2, b).

d.     Design an efficient binary gcd algorithm for input integers a and b, where a    b, that runs in O(lg(max(a, b))) time. Assume that each subtraction, parity test, and halving can be performed in unit time.

33-2 Analysis of bit operations in Euclid's algorithm

a.     Show that using the ordinary "paper and pencil" algorithm for long division--dividing a by b, yielding a quotient q and remainder r--requires O((1 + lg q) lg b) bit operations.

b.     Define   (a, b) = (1 + lg a)(1 + lg b). Show that the number of bit operations performed by EUCLID in reducing the problem of computing gcd(a, b) to that of computing gcd(b, a mod b) is at most c(  (a, b) -   (b, a mod b)) for some sufficiently large constant c > 0.

c.     Show that EUCLID(a, b) requires O(  (a, b)) bit operations in general and O(  2) bit operations when applied to two   -bit inputs.

33-3 Three algorithms for Fibonacci numbers

This problem compares the efficiency of three methods for computing the nth Fibonacci number Fn, given n. Assume that the cost of adding, subtracting, or multiplying two numbers is O(1), independent of the size of the numbers.

a.     Show that the running time of the straightforward recursive method for computing Fn based on recurrence (2.13) is exponential in n.

b.     Show how to compute Fn in O(n) time using memoization.

c.     Show how to compute Fn in O(lg n) time using only integer addition and multiplication. (Hint: Consider the matrix

and its powers.)

d.     Assume now that adding two   -bit numbers takes   (  ) time and that multiplying two   -bit numbers takes   (  2) time. What is the running time of these three methods under this more reasonable cost measure for the elementary arithmetic operations?

33-4 Quadratic residues

Let p be an odd prime. A number    is a quadratic residue if the equation x2 = a (mod p) has a solution for the unknown x.

a.     Show that there are exactly (p - 1)/2 quadratic residues, modulo p.

b.     If p is prime, we define the Legendre symbol   , to be 1 if a is a quadratic residue modulo p and -1 otherwise. Prove that if   , then

Give an efficient algorithm for determining whether or not a given number a is a quadratic residue modulo p. Analyze the efficiency of your algorithm.

c.     Prove that if p is a prime of the form 4k + 3 and a is a quadratic residue in   , then ak+l mod p is a square root of a, modulo p. How much time is required to find the square root of a quadratic residue a modulo p?

d.     Describe an efficient randomized algorithm for finding a nonquadratic residue, modulo an arbitrary prime p. How many arithmetic operations does your algorithm require on average?

Chapter notes

Niven and Zuckerman [151] provide an excellent introduction to elementary number theory. Knuth [122] contains a good discussion of algorithms for finding the greatest common divisor, as well as other basic number-theoretic algorithms. Riesel [168] and Bach [16] provide more recent surveys of computational number theory. Dixon [56] gives an overview of factorization and primality testing. The conference proceedings edited by Pomerance [159] contains several nice survey articles.

Knuth [122] discusses the origin of Euclid's algorithm. It appears in Book 7, Propositions 1 and 2, of the Greek mathematician Euclid's Elements, which was written around 300 B.C. Euclid's description may have been derived from an algorithm due to Eudoxus around 375 B.C. Euclid's algorithm may hold the honor of being the oldest nontrivial algorithm; it is rivaled only by the Russian peasant's algorithm for multiplication (see Chapter 29), which was known to the ancient Egyptians.

Knuth attributes a special case of the Chinese remainder theorem (Theorem 33.27) to the Chinese mathematician   , who lived sometime between 200 B.C. and A.D. 200--the date is quite uncertain. The same special case was given by the Greek mathematician Nichomachus around A.D. 100. It was generalized by Chhin Chiu-Shao in 1247. The Chinese remainder theorem was finally stated and proved in its full generality by L. Euler in 1734.

The randomized primality-testing algorithm presented here is due to Miller [147] and Rabin [166]; it is the fastest randomized primality-testing algorithm known, to within constant factors. The proof of Theorem 33.39 is a slight adaptation of one suggested by Bach [15]. A proof of a stronger result for MILLER-RABIN was given by Monier[148, 149]. Randomization appears to be necessary to obtain a polynomial-time primality-testing algorithm. The fastest deterministic primality-testing algorithm known is the Cohen-Lenstra version [45] of the primality test by Adleman, Pomerance, and Rumely [3]. When testing a number n of length   lg(n + 1)   for primality, it runs in (lg n)O(1g 1g 1g n) time, which is just slightly superpolynomial.

The problem of finding large "random" primes is nicely discussed in an article by Beauchemin, Brassard, Crépeau, Goutier, and Pomerance [20].

The concept of a public-key cryptosystem is due to Diffie and Hellman [54]. The RSA cryptosystem was proposed in 1977 by Rivest, Shamir, and Adleman [169]. Since then, the field of cryptography has blossomed. In particular, many new techniques have been developed for proving cryptosystems to be secure. For example, Goldwasser and Micali [86] show that randomization can be an effective tool in the design of secure public-key encryption schemes. For signature schemes, Goldwasser, Micali, and Rivest [88] present a digital-signature scheme for which every conceivable type of forgery is provably as difficult as factoring. Recently, Goldwasser, Micali, and Rackoff [87] introduced a class of "zero-knowledge" encryption schemes for which it can be proven (under certain reasonable assumptions) that no party learns more than he is supposed to learn from a communication.

The rho heuristic for integer factoring was invented by Pollard [156]. The version presented here is a variant proposed by Brent [35].

The best algorithms for factoring large numbers have a running time that grows roughly exponentially with the square root of the length of the number n to be factored. The quadratic-sieve factoring algorithm, due to Pomerance [158], is perhaps the most efficient such algorithm in general for large inputs. Although it is difficult to give a rigorous analysis of this algorithm, under reasonable assumptions we can derive a running-time estimate of L(n)1+o(1), where   . The elliptic-curve method due to Lenstra [137] may be more effective for some inputs than the quadratic-sieve method, since, like Pollard's rho method, it can find a small prime factor p quite quickly. With this method, the time to find p is estimated to be   .

Go to Chapter 34     Back to Table of Contents


Intro to Algorithms: CHAPTER 34: STRING MATCHING

 Return to Table of Contents

CHAPTER 34: STRING MATCHING

Finding all occurrences of a pattern in a text is a problem that arises frequently in text-editing programs. Typically, the text is a document being edited, and the pattern searched for is a particular word supplied by the user. Efficient algorithms for this problem can greatly aid the responsiveness of the text-editing program. String-matching algorithms are also used, for example, to search for particular patterns in DNA sequences.

We formalize the string-matching problem as follows. We assume that the text is an array T[1 . . n] of length n and that the pattern is an array P[1 . . m] of length m. We further assume that the elements of P and T are characters drawn from a finite alphabet [image]. For example, we may have [image] = {0, 1} or [image] = {a, b, . . . , z}. The character arrays P and T are often called strings of characters.

We say that pattern P occurs with shift s in text T (or, equivalently, that pattern P occurs beginning at position s + 1 in text T) if 0 [image] s [image] n - m and T[s + 1 . . s + m] = P[1 . . m] (that is, if T[s + j] = P[j], for 1 [image] j [image] m). If P occurs with shift s in T, then we call s a valid shift; otherwise, we call s an invalid shift. The string-matching problem is the problem of finding all valid shifts with which a given pattern P occurs in a given text T. Figure 34.1 illustrates these definitions.

This chapter is organized as follows. In Section 34.1 we review the naive brute-force algorithm for the string-matching problem, which has worst-case running time O((n - m + 1)m). Section 34.2 presents an interesting string-matching algorithm, due to Rabin and Karp. This algorithm also has worst-case running time O((n - m + 1)m), but it works much better on average and in practice. It also generalizes nicely to other pattern-matching problems. Section 34.3 then describes a string-matching algorithm that begins by constructing a finite automaton specifically designed to search for occurrences of the given pattern P in a text. This algorithm runs in time O(n + m [image] [image] [image]). The similar but much cleverer Knuth-Morris-Pratt (or KMP) algorithm is presented in Section 34.4; the KMP algorithm runs in time O(n + m). Finally, Section 34.5 describes an algorithm due to Boyer and Moore that is often the best practical choice, although its worst-case running time (like that of the Rabin-Karp algorithm) is no better than that of the naive string-matching algorithm.

Figure 34.1 The string-matching problem. The goal is to find all occurrences of the pattern P = abaa in the text T = abcabaabcabac. The pattern occurs only once in the text, at shift s = 3. The shift s = 3 is said to be a valid shift. Each character of the pattern is connected by a vertical line to the matching character in the text, and all matched characters are shown shaded.

Notation and terminology

We shall let [image] * (read "sigma-star") denote the set of all finite-length strings formed using characters from the alphabet [image]. In this chapter, we consider only finite-length strings. The zero-length empty string, denoted [image] , also belongs to [image] *. The length of a string x is denoted |x|. The concatenation of two strings x and y, denoted xy, has length |x| + |y| and consists of the characters from x followed by the characters from y.

We say that a string w is a prefix of a string x, denoted [image] for some string y [image] [image] *. Note that if [image], then |w| [image] |x|. Similarly, we say that a string w is a suffix of a string x, denoted [image] for some y [image] [image] *. It follows from [image] that |w| [image] |x|. The empty string [image] is both a suffix and a prefix of every string. For example, we have ab [image] abcca and cca [image] abcca. It is useful to note that for any strings x and y and any character a, we have [image] if and only if [image]. Also note that [image] are transitive relations. The following lemma will be useful later.

Lemma 34.1

Suppose that x, y, and z are strings such that [image]. If |x| [image] |y|, then [image]. If |x| [image] |y|, then [image]. If |x| = |y|, then x = y.

Proof See Figure 34.2 for a graphical proof.

For brevity of notation, we shall denote the k-character prefix P[1 . . k] of the pattern P[1 . . m] by Pk. Thus, P0 = [image] and Pm = P = P[1 . . m]. Similarly, we denote the k-character prefix of the text T as Tk. Using this notation, we can state the string-matching problem as that of finding all shifts s in the range 0 [image] s [image] n - m such that [image].

In our pseudocode, we allow two equal-length strings to be compared for equality as a primitive operation. If the strings are compared from left to right and the comparison stops when a mismatch is discovered, we assume that the time taken by such a test is a linear function of the number of matching characters discovered. To be precise, the test "x = y" is assumed to take time [image] (t + 1), where t is the length of the longest string z such that [image].

Figure 34.2 A graphical proof of Lemma 34.1. We suppose that . The three parts of the figure illustrate the three cases of the lemma. Vertical lines connect matching regions (shown shaded) of the strings. (a) If |x| |y|, then . (b) If |x| |y|, then . (c) If |x| = |y|, then x = y.

34.1 The naive string-matching algorithm

The naive algorithm finds all valid shifts using a loop that checks the condition P[1 . . m] = T[s + 1 . . s + m] for each of the n - m + 1 possible values of s.

NAIVE-STRING-MATCHER(T, P)

1  n    length[T]

2  m    length[P]

3  for s    0 to n - m

4        do if P[1 . . m] = T[s + 1 . . s + m]

5              then print "Pattern occurs with shift" s

The naive string-matching procedure can be interpreted graphically as sliding a "template" containing the pattern over the text, noting for which shifts all of the characters on the template equal the corresponding characters in the text, as illustrated in Figure 34.3. The for loop beginning on line 3 considers each possible shift explicitly. The test on line 4 determines whether the current shift is valid or not; this test involves an implicit loop to check corresponding character positions until all positions match successfully or a mismatch is found. Line 5 prints out each valid shift s.

Figure 34.3 The operation of the naive string matcher for the pattern P = aab and the text T = acaabc. We can imagine the pattern P as a "template" that we slide next to the text. Parts (a)-(d) show the four successive alignments tried by the naive string matcher. In each part, vertical lines connect corresponding regions found to match (shown shaded), and a jagged line connects the first mismatched character found, if any. One occurrence of the pattern is found, at shift s = 2, shown in part (c).

Procedure NAIVE-STRING-MATCHER takes time [image] ((n - m + 1)m) in the worst case. For example, consider the text string an (a string of n a's) and the pattern am. For each of the n - m + 1 possible values of the shift s, the implicit loop on line 4 to compare corresponding characters must execute m times to validate the shift. The worst-case running time is thus [image] ((n - m + 1)m), which is [image] (n2) if m = [image] n/2 [image] .

As we shall see, NAIVE-STRING-MATCHER is not an optimal procedure for this problem. Indeed, in this chapter we shall show an algorithm with a worst-case running time of O(n + m). The naive string-matcher is inefficient because information gained about the text for one value of s is totally ignored in considering other values of s. Such information can be very valuable, however. For example, if P = aaab and we find that s = 0 is valid, then none of the shifts 1, 2, or 3 are valid, since T[4] = b. In the following sections, we examine several ways to make effective use of this sort of information.

Exercises

34.1-1

Show the comparisons the naive string matcher makes for the pattern P = 0001 in the text T = 000010001010001.

34.1-2

Show that the worst-case time for the naive string matcher to find the first occurrence of a pattern in a text is [image] ((n - m + 1)(m - 1)).

34.1-3

Suppose that all characters in the pattern P are different. Show how to accelerate NAIVE-STRING-MATCHER to run in time O(n) on an n-character text T.

34.1-4

Suppose that pattern P and text T are randomly chosen strings of length m and n, respectively, from the d-ary alphabet [image] d = {0, 1, . . . , d - 1}, where d [image] 2. Show that the expected number of character-to-character comparisons made by the implicit loop in line 4 of the naive algorithm is

(Assume that the naive algorithm stops comparing characters for a given shift once a mismatch is found or the entire pattern is matched.) Thus, for randomly chosen strings, the naive algorithm is quite efficient.

34.1-5

Suppose we allow the pattern P to contain occurrences of a gap character [image] that can match an arbitrary string of characters (even one of zero length). For example, the pattern [image] occurs in the text cabccbacbacab as

Note that the gap character may occur an arbitrary number of times in the pattern but is assumed not to occur at all in the text. Give a polynomial-time algorithm to determine if such a pattern P occurs in a given text T, and analyze the running time of your algorithm.

34.2 The Rabin-Karp algorithm

Rabin and Karp have proposed a string-matching algorithm that performs well in practice and that also generalizes to other algorithms for related problems, such as two-dimensional pattern matching. The worst-case running time of the Rabin-Karp algorithm is O((n - m + 1)m), but it has a good average-case running time.

This algorithm makes use of elementary number-theoretic notions such as the equivalence of two numbers modulo a third number. You may want to refer to Section 33.1 for the relevant definitions.

For expository purposes, let us assume that [image] = {0, 1, 2, . . . , 9}, so that each character is a decimal digit. (In the general case, we can assume that each character is a digit in radix-d notation, where d = [image] [image] [image].) We can then view a string of k consecutive characters as representing a length-k decimal number. The character string 31415 thus corresponds to the decimal number 31,415. Given the dual interpretation of the input characters as both graphical symbols and digits, we find it convenient in this section to denote them as we would digits, in our standard text font.

Given a pattern P[1 . . m], we let p denote its corresponding decimal value. In a similar manner, given a text T[1 . . n], we let ts denote the decimal value of the length-m substring T[s + 1 . . s + m], for s = 0, 1, . . . , n - m. Certainly, ts = p if and only if T[s + 1 . . s + m] = P[1 . . m]; thus, s is a valid shift if and only if ts = p. If we could compute p in time O(m) and all of the ti values in a total of O(n) time, then we could determine all valid shifts s in time O(n) by comparing p with each of the ts's. (For the moment, let's not worry about the possibility that p and the ts's might be very large numbers.)

We can compute p in time O(m) using Horner's rule (see Section 32.1):

p = P[m] + 10 (P[m - 1] + 10(P[m - 2] + . . . + 10(P[2] + 10P[1]) . . . )).

The value t0 can be similarly computed from T[1 . . m] in time O(m).

To compute the remaining values t1, t2, . . . , tn-m in time O(n - m), it suffices to observe that ts + 1 can be computed from ts in constant time, since

ts + 1   =   10(ts - 10m - 1T[s + 1]) + T[s + m + 1].

(34.1)

For example, if m= 5 and ts = 31415, then we wish to remove the high-order digit T[s + 1] = 3 and bring in the new low-order digit (suppose it is T[s + 5 + 1] = 2) to obtain

ts+1 = 10(31415 - 10000.3) + 2

= 14152 .

Subtracting 10m-1 T[s+1] removes the high-order digit from ts, multiplying the result by 10 shifts the number left one position, and adding T[s + m + 1] brings in the appropriate low-order digit. If the constant 10m-1 is precomputed (which can be done in time O(1g m) using the techniques of Section 33.6, although for this application a straightforward O(m) method is quite adequate), then each execution of equation (34.1) takes a constant number of arithmetic operations. Thus, p and t0, t1, . . . , tn-m can all be computed in time O(n + m), and we can find all occurrences of the pattern P[1 . . m] in the text T[1 . . n] in time O(n + m).

The only difficulty with this procedure is that p and ts may be too large to work with conveniently. If P contains m characters, then assuming that each arithmetic operation on p (which is m digits long) takes "constant time" is unreasonable. Fortunately, there is a simple cure for this problem, as shown in Figure 34.4 : compute p and the ts's modulo a suitable modulus q. Since the computation of p, t0, and the recurrence (34.1) can all be performed modulo q, we see that p and all the ts's can be computed modulo q in time O(n + m). The modulus q is typically chosen as a prime such that 10q just fits within one computer word, which allows all of the necessary computations to be performed with single-precision arithmetic.

Figure 34.4 The Rabin-Karp algorithm. Each character is a decimal digit, and we compute values modulo 13. (a) A text string. A window of length 5 is shown shaded. The numerical value of the shaded number is computed modulo 13, yielding the value 7. (b) The same text string with values computed modulo 13 for each possible position of a length-5 window. Assuming the pattern P = 31415, we look for windows whose value modulo 13 is 7, since 31415 7 (mod 13). Two such windows are found, shown shaded in the figure. The first, beginning at text position 7, is indeed an occurrence of the pattern, while the second, beginning at text position 13, is a spurious hit. (c) Computing the value for a window in constant time, given the value for the previous window. The first window has value 31415. Dropping the high-order digit 3, shifting left (multiplying by 10), and then adding in the low-order digit 2 gives us the new value 14152. All computations are performed modulo 13, however, so the value for the first window is 7, and the value computed for the new window is 8.

In general, with a d-ary alphabet {0, 1, . . . ,d - 1}, we choose q so that d q fits within a computer word and adjust the recurrence equation (34.1) to work modulo q, so that it becomes

ts+1 = (d(ts - T[s + 1]h) + T[s + m + 1]) mod q ,

(34.2)

where h [image] dm-1 (mod q) is the value of the digit "1" in the high-order position of an m-digit text window.

The ointment of working modulo q now contains a fly, however, since ts [image] p (mod q) does not imply that ts = p. On the other hand, if [image] (mod q), then we definitely have that ts [image] p, so that shift s is invalid. We can thus use the test ts [image] p (mod q) as a fast heuristic test to rule out invalid shifts s. Any shift s for which ts [image] p (mod q) must be tested further to see if s is really valid or we just have a spurious hit. This testing can be done by explicitly checking the condition P[1 . . m] = T[s + 1 . . s + m]. If q is large enough, then we can hope that spurious hits occur infrequently enough that the cost of the extra checking is low.

The following procedure makes these ideas precise. The inputs to the procedure are the text T, the pattern P, the radix d to use (which is typically taken to be | [image] |), and the prime q to use.

RABIN-KARP-MATCHER(T, P, d, q)

1 n    length[T]

2 m   length[P]

3 h    dm-1 mod q

4 p    0

5 t0    0

6 for i    1 to m

7       do p    (dp + P[i]) mod q

8          t0    (dt0 + T[i]) mod q

9 for s    0 to n - m

10      do if p = ts

11            then if P[1 . . m] = T[s + 1 . . s + m]

12                    then "Pattern occurs with shift" s

13         if s < n - m

14           then ts+1    (d(ts - T[s + 1]h) + T[s + m + 1]) mod q

The procedure RABIN-KARP-MATCHER works as follows. All characters are interpreted as radix-d digits. The subscripts on t are provided only for clarity; the program works correctly if all the subscripts are dropped. Line 3 initializes h to the value of the high-order digit position of an m-digit window. Lines 4-8 compute p as the value of P[1 . . m] mod q and t0 as the value of T[1 . . m] mod q. The for loop beginning on line 9 iterates through all possible shifts s. The loop has the following invariant: whenever line 10 is executed, ts = T[s + 1 . . s + m] mod q. If p = ts in line 10 (a "hit"), then we check to see if P[1 . . m] = T[s + 1 . . s + m] in line 11 to rule out the possibility of a spurious hit. Any valid shifts found are printed out on line 12. If s < n - m (checked in line 13), then the for loop is to be executed at least one more time, and so line 14 is first executed to ensure that the loop invariant holds when line 10 is again reached. Line 14 computes the value of ts+1 mod q from the value of ts mod q in constant time using equation (34.2) directly.

The running time of RABIN-KARP-MATCHER is [image] ((n - m + 1)m) in the worst case, since (like the naive string-matching algorithm) the Rabin-Karp algorithm explicitly verifies every valid shift. If P = am and T = an, then the verifications take time [image] ((n - m + 1)m), since each of the n - m + 1 possible shifts is valid. (Note also that the computation of dm-1 mod q on line 3 and the loop on lines 6-8 take time O(m) = O((n - m + 1 )m).)

In many applications, we expect few valid shifts (perhaps O(1) of them), and so the expected running time of the algorithm is O(n + m) plus the time required to process spurious hits. We can base a heuristic analysis on the assumption that reducing values modulo q acts like a random mapping from [image] * to Z q. (See the discussion on the use of division for hashing in Section 12.3.1. It is difficult to formalize and prove such an assumption, although one viable approach is to assume that q is chosen randomly from integers of the appropriate size. We shall not pursue this formalization here.) We can then expect that the number of spurious hits is O(n/q), since the chance that an arbitrary ts will be equivalent to p, modulo q, can be estimated as 1/q. The expected amount of time taken by the Rabin-Karp algorithm is then

O(n) + O(m(v + n/q)),

where v is the number of valid shifts. This running time is O(n) if we choose q [image] m. That is, if the expected number of valid shifts is small (O(1)) and the prime q is chosen to be larger than the length of the pattern, then we can expect the Rabin-Karp procedure to run in time O(n + m).

Exercises

34.2-1

Working modulo q = 11, how many spurious hits does the Rabin-Karp matcher encounter in the text T = 3141592653589793 when looking for the pattern P = 26?

34.2-2

How would you extend the Rabin-Karp method to the problem of searching a text string for an occurrence of any one of a given set of k patterns?

34.2-3

Show how to extend the Rabin-Karp method to handle the problem of looking for a given m X m pattern in an n X n array of characters. (The pattern may be shifted vertically and horizontally, but it may not be rotated.)

34.2-4

Alice has a copy of a long n-bit file A = [image] an-1, an-2, . . . , a0 [image], and Bob similarly has an n-bit file B = [image] bn-1, bn-2, . . . ,b0 [image]. Alice and Bob wish to know if their files are identical. To avoid transmitting all of A or B, they use the following fast probabilistic check. Together, they select a prime q > 1000n and randomly select an integer x from {0, 1, . . . , n - 1}. Then, Alice evaluates

and Bob similarly evaluates B(x). Prove that if A [image] B, there is at most one chance in 1000 that A(x) = B(x), whereas if the two files are the same, A(x) is necessarily the same as B(x). (Hint: See Exercise 33.4-4.)

34.3 String matching with finite automata

Many string-matching algorithms build a finite automaton that scans the text string T for all occurrences of the pattern P. This section presents a method for building such an automaton. These string-matching automata are very efficient: they examine each text character exactly once, taking constant time per text character. The time used--after the automaton is built--is therefore [image] (n). The time to build the automaton, however, can be large if [image] is large. Section 34.4 describes a clever way around this problem.

We begin this section with the definition of a finite automaton. We then examine a special string-matching automaton and show how it can be used to find occurrences of a pattern in a text. This discussion includes details on how to simulate the behavior of a string-matching automaton on a given text. Finally, we shall show how to construct the string-matching automaton for a given input pattern.

Finite automata

A finite automaton M is a 5-tuple (Q, q0, A, [image], [image]), where

Q is a finite set of states,

q0 [image] Q is the start state,

A [image] Q is a distinguished set of accepting states,

is a finite input alphabet,

is a function from Q X [image] into Q, called the transition function of M.

Figure 34.5 A simple two-state finite automaton with state set Q = {0, 1}, start state q0 = 0, and input alphabet = {a, b}. (a) A tabular representation of the transition function . (b) An equivalent state-transition diagram. State 1 is the only accepting state (shown blackened). Directed edges represent transitions. For example, the edge from state 1 to state 0 labeled b indicates (1, b) = 0. This automaton accepts those strings that end in an odd number of a's. More precisely, a string x is accepted if and only if x = yz, where y = or y ends with a b, and z = ak, where k is odd. For example, the sequence of states this automaton enters for input abaaa (including the start state) is 0, 1, 0, 1, 0, 1 , and so it accepts this input. For input abbaa, the sequence of states is 0, 1, 0, 0, 1, 0 , and so it rejects this input.

The finite automaton begins in state q0 and reads the characters of its input string one at a time. If the automaton is in state q and reads input character a, it moves ("makes a transition") from state q to state [image] (q, a). Whenever its current state q is a member of A, the machine M is said to have accepted the string read so far. An input that is not accepted is said to be rejected. Figure 34.5 illustrates these definitions with a simple two-state automaton.

A finite automaton M induces a function ø, called the final-state function, from [image] * to Q such that ø(w) is the state M ends up in after scanning the string w. Thus, M accepts a string w if and only if ø(w) [image] A. The function ø is defined by the recursive relation

ø(  )  = q   ,

ø(wa)  =   (ø(w), a)  for w      *, a      .

String-matching automata

There is a string-matching automaton for every pattern P; this automaton must be constructed from the pattern in a preprocessing step before it can be used to search the text string. Figure 34.6 illustrates this construction for the pattern P = ababaca. From now on, we shall assume that P is a given fixed pattern string; for brevity, we shall not indicate the dependence upon P in our notation.

Figure 34.6 (a) A state-transition diagram for the string-matching automaton that accepts all strings ending in the string ababaca. State 0 is the start state, and state 7 (shown blackened) is the only accepting state. A directed edge from state i to state j labeled a represents (i,a) = j. The right-going edges forming the "spine" of the automaton, shown heavy in the figure, correspond to successful matches between pattern and input characters. The left-going edges correspond to failing matches. Some edges corresponding to failing matches are not shown; by convention, if a state i has no outgoing edge labeled a for some a , then (i,a) = 0. (b) The corresponding transition function , and the pattern string P = ababaca. The entries corresponding to successful matches between pattern and input characters are shown shaded. (c) The operation of the automaton on the text T = abababacaba. Under each text character T[i] is given the state ø(Ti) the automaton is in after processing the prefix Ti. One occurrence of the pattern is found, ending in position 9.

In order to specify the string-matching automaton corresponding to a given pattern P[1 . . m], we first define an auxiliary function [image] , called the suffix function corresponding to P. The function [image] is a mapping from [image] * to {0, 1, . . . , m} such that [image] (x) is the length of the longest prefix of P that is a suffix of x:

The suffix function [image] is well defined since the empty string P0 = [image] is a suffix of every string. As examples, for the pattern P = ab, we have [image] ( [image] ) = 0, [image] (ccaca) = 1, and [image] (ccab) = 2. For a pattern P of length m, we have [image] (x) = m if and only if [image]. It follows from the definition of the suffix function that if [image], then [image] (x) [image] [image] (y).

We define the string-matching automaton corresponding to a given pattern P[1 . . m] as follows.

The state set Q is {0, 1, . . . , m}. The start state q0 is state 0, and state m is the only accepting state.

The transition function [image] is defined by the following equation, for any state q and character a:

   (q, a) =   (Pqa) .

(34.3)

Here is an intuitive rationale for defining [image] (q, a) = [image] (Pq a). The machine maintains as an invariant of its operation that

  (Ti) =   (Ti) ;

(34.4)

this result is proved as Theorem 34.4 below. In words, this means that after scanning the first i characters of the text string T, the machine is in state [image] (Ti) = q, where q = [image] (Ti) is the length of the longest suffix of Ti that is also a prefix of the pattern P. If the next character scanned is T[i + 1] = a, then the machine should make a transition to state [image] (Ti + 1) = [image] (Tia). The proof of the theorem shows that [image] (Tia) = [image] (Pqa). That is, to compute the length of the longest suffix of Tia that is a prefix of P, we can compute the longest suffix of Pqa that is a prefix of P. At each state, the machine only needs to know the length of the longest prefix of P that is a suffix of what has been read so far. Therefore, setting [image] (q, a) = [image] (Pqa) maintains the desired invariant (34.4). This informal argument will be made rigorous shortly.

In the string-matching automaton of Figure 34.6, for example, we have [image] (5, b) = 4. This follows from the fact that if the automaton reads a b in state q = 5, then Pq b = ababab, and the longest prefix of P that is also a suffix of ababab is P4 = abab.

To clarify the operation of a string-matching automaton, we now give a simple, efficient program for simulating the behavior of such an automaton (represented by its transition function [image] ) in finding occurrences of a pattern P of length m in an input text T[1 . . n]. As for any string-matching automaton for a pattern of length m, the state set Q is {0,1, . . . , m}, the start state is 0, and the only accepting state is state m.

Figure 34.7 An illustration for the proof of Lemma 34.2. The figure shows that r (x) + 1, where r = (xa).

FINITE-AUTOMATON-MATCHER(T,   , m)

1  n    length[T]

2  q    0

3  for i    1 to n

4        do q       (q, T[i])

5           if q = m

6              then s    i - m

7                   print "Pattern occurs with shift" s

The simple loop structure of FINITE-AUTOMATON-MATCHER implies that its running time on a text string of length n is O(n). This running time, however, does not include the time required to compute the transition function [image] . We address this problem later, after proving that the procedure FINITE-AUTOMATON-MATCHER operates correctly.

Consider the operation of the automaton on an input text T[1 . . n]. We shall prove that the automaton is in state [image] (Tj) after scanning character T[i]. Since [image] (Ti) = m if and only if [image], the machine is in the accepting state m if and only if the pattern P has just been scanned. To prove this result, we make use of the following two lemmas about the suffix function [image] .

Lemma 34.2

For any string x and character a, we have [image] (xa) [image] [image] (x) + 1.

Proof Referring to Figure 34.7, let r = [image] (xa). If r = 0, then the conclusion r [image] [image] (x) + 1 is trivially satisfied, by the nonnegativity of [image] (x). So assume that r > 0. Now, [image], by the definition of [image] . Thus, [image], by dropping the a from the end of Pr and from the end of xa. Therefore, r - 1 [image] [image] (x), since [image] (x) is largest k such that [image].

Lemma 34.3

For any string x and character a, if q = [image] (x), then [image] (xa) = [image] (Pqa).

Proof From the definition of [image], we have [image]. As Figure 34.8 shows, [image]. If we let r = [image] (xa), then r [image] q + 1 by Lemma 34.2. Since [image], and [image] Pr [image] [image] [image] Pqa [image], Lemma 34.1 implies that [image]. Therefore, r [image] [image] (Pqa), that is, [image] (xa) [image] [image] (Pqa). But we also have [image] (Pqa) [image] [image] (xa), since [image]. Thus, [image] (xa) = [image] (Pqa).

Figure 34.8 An illustration for the proof of Lemma 34.3. The figure shows that r = (Pqa), where q = (x) and r = (xa).

We are now ready to prove our main theorem characterizing the behavior of a string-matching automaton on a given input text. As noted above, this theorem shows that the automaton is merely keeping track, at each step, of the longest prefix of the pattern that is a suffix of what has been read so far.

Theorem 34.4

If [image] is the final-state function of a string-matching automaton for a given pattern P and T[1 . . n] is an input text for the automaton, then

Proof The proof is by induction on i. For i = 0, the theorem is trivially true, since T0 = [image] . Thus, [image].

Now, we assume that [image] and prove that [image]. Let q denote [image], and let a denote T[i + 1]. Then,

By induction, the theorem is proved.

By Theorem 34.4, if the machine enters state q on line 4, then q is the largest value such that [image]. Thus, we have q = m on line 5 if and only if an occurrence of the pattern P has just been scanned. We conclude that FINITE-AUTOMATON-MATCHER operates correctly.

Computing the transition function

The following procedure computes the transition function [image] from a given pattern P[1 . . m].

COMPUTE-TRANSITION-FUNCTION(P,   )

1 m    length[P]

2 for q    0 to m

3     do for each character a    *

4            do k    min (m + 1, q + 2)

5               repeat k    k - 1

7               (q, a)    k

8 return

This procedure computes [image] (q, a) in a straightforward manner according to its definition. The nested loops beginning on lines 2 and 3 consider all states q and characters a, and lines 4-7 set [image] (q, a) to be the largest k such that [image]. The code starts with the largest conceivable value of k, which is min(m, q + 1), and decreases k until [image].

The running time of COMPUTE-TRANSITION-FUNCTION is O(m3 [image] [image] [image]), because the outer loops contribute a factor of m [image] [image] [image] , the inner repeat loop can run at most m + 1 times, and the test [image] on line 6 can require comparing up to m characters. Much faster procedures exist; the time required to compute [image] from P can be improved to O(m [image] [image] [image]) by utilizing some cleverly computed information about the pattern P (see Exercise 34.4-6). With this improved procedure for computing [image], the total running time to find all occurrences of a length-m pattern in a length-n text over an alphabet [image] is O(n + m [image] [image] [image]).

Exercises

34.3-1

Construct the string-matching automaton for the pattern P = aabab and illustrate its operation on the text string T = aaababaabaababaab.

34.3-2

Draw a state-transition diagram for a string-matching automaton for the pattern ababbabbababbababbabb over the alphabet [image] = {a, b}.

34.3-3

We call a pattern P nonoverlappable if [image] implies k = 0 or k = q. Describe the state-transition diagram of the string-matching automaton for a nonoverlappable pattern.

34.3-4

Given two patterns P and P', describe how to construct a finite automaton that determines all occurrences of either pattern. Try to minimize the number of states in your automaton.

34.3-5

Given a pattern P containing gap characters (see Exercise 34.1-5), show how to build a finite automaton that can find an occurrence of P in a text T in O(n) time, where n = [image] T [image].

34.4 The Knuth-Morris-Pratt algorithm

We now present a linear-time string-matching algorithm due to Knuth, Morris, and Pratt. Their algorithm achieves a [image] (n + m) running time by avoiding the computation of the transition function [image] altogether, and it does the pattern matching using just an auxiliary function [image] [1 . . m] precomputed from the pattern in time O(m). The array [image] allows the transition function [image] to be computed efficiently (in an amortized sense) "on the fly" as needed. Roughly speaking, for any state q = 0, 1, . . . , m,and any character a [image] [image], the value [image] [q] contains the information that is independent of a and is needed to compute [image] (q, a). (This remark will be clarified shortly.) Since the array [image] has only m entries, whereas [image] has O(m [image] [image] ) entries, we save a factor of in the preprocessing by computing rather than .

The prefix function for a pattern

The prefix function for a pattern encapsulates knowledge about how the pattern matches against shifts of itself. This information can be used to avoid testing useless shifts in the naive pattern-matching algorithm or to avoid the precomputation of for a string-matching automaton.

Consider the operation of the naive string matcher. Figure 34.9(a) shows a particular shift s of a template containing the pattern P = ababaca against a text T. For this example, q = 5 of the characters have matched successfully, but the 6th pattern character fails to match the corresponding text character. The information that q characters have matched successfully determines the corresponding text characters. Knowing these q text characters allows us to determine immediately that certain shifts are invalid. In the example of the figure, the shift s + 1 is necessarily invalid, since the first pattern character, an a, would be aligned with a text character that is known to match with the second pattern character, a b. The shift s + 2 shown in part (b) of the figure, however, aligns the first three pattern characters with three text characters that must necessarily match. In general, it is useful to know the answer to the following question:

Given that pattern characters P[1 . . q] match text characters T[s + 1 . . s + q], what is the least shift s' > s such that

P[1 . . k] = T[s' 1 . . s' k],

(34.5)

where s' + k = s + q?

Such a shift s' is the first shift greater than s that is not necessarily invalid due to our knowledge of T[s + 1 . . s + q]. In the best case, we have that s' = s + q, and shifts s + 1, s + 2, . . . , s + q - 1 are all immediately ruled out. In any case, at the new shift s' we don't need to compare the first k characters of P with the corresponding characters of T, since we are guaranteed that they match by equation (34.5).

Figure 34.9 The prefix function . (a) The pattern P = ababaca is aligned with a text T so that the first q = 5 characters match. Matching characters, shown shaded, are connected by vertical lines. (b) Using only our knowledge of the 5 matched characters, we can deduce that a shift of s + 1 is invalid, but that a shift of s' = s + 2 is consistent with everything we know about the text and therefore is potentially valid. (c) The useful information for such deductions can be precomputed by comparing the pattern with itself. Here, we see that the longest prefix of P that is also a suffix of P5 is P3. This infomation is precomputed and represented in the array , so that [5] = 3. Given that q characters have matched successfully at shift s, the next potentially valid shift is at s' = s + (q - [q]).

The necessary information can be precomputed by comparing the pattern against itself, as illustrated in Figure 34.9(c). Since T[s' + 1 . . s' + k] is part of the known portion of the text, it is a suffix of the string Pq. Equation (34.5) can therefore be interpreted as asking for the largest k < q such that .Then, s' = s + (q - k) is the next potentially valid shift. It turns out to be convenient to store the number k of matching characters at the new shift s', rather than storing, say, s' - s. This information can be used to speed up both the naive string-matching algorithm and the finite-automaton matcher.

We formalize the precomputation required as follows. Given a pattern P[1 . . m], the prefix function for the pattern P is the function : {1, 2, . . . , m} {0, 1, . . . , m - 1 } such that

That is, [q] is the length of the longest prefix of P that is a proper suffix of Pq. As another example, Figure 34.10(a) gives the complete prefix function for the pattern ababababca.

The Knuth-Morris-Pratt matching algorithm is given in pseudocode below as the procedure KMP-MATCHER. It is mostly modeled after FINITE-AUTOMATON-MATCHER, as we shall see. KMP-MATCHER calls the auxiliary procedure COMPUTE-PREFIX-FUNCTION to compute .

KMP-MATCHER(T, P)

1 n    length[T]

2 m    length[P]

3       COMPUTE-PREFIX-FUNCTION(P)

4 q    0

5 for i    1 to n

6      do while q > 0 and P[q + 1]    T[i]

7             do q      [q]

8        if P[q + 1] = T[i]

9           then q    q + 1

10        if q = m

11           then print "Pattern occurs with shift" i - m

12                q      [q]

COMPUTE-PREFIX-FUNCTION(P)

1 m    length[P]

2    [1]    0

3 k    0

4 for q    2 to m

5      do while k > 0 and P[k + 1    P[q]

6            do k      [k]

7        if P[k + 1] = P[q]

8           then k    k + 1

9          [q]    k

10 return

We begin with an analysis of the running times of these procedures. Proving these procedures correct will be more complicated.

Running-time analysis

The running time of COMPUTE-PREFIX-FUNCTION is O(m), using an amortized analysis (see Chapter 18). We associate a potential of k with the current state k of the algorithm. This potential has an initial value of 0, by line 3. Line 6 decreases k whenever it is executed, since   [k] < k. Since   [k]    0 for all k, however, k can never become negative. The only other line that affects k is line 8, which increases k by at most one during each execution of the for loop body. Since k < q upon entering the for loop, and since q is incremented in each iteration of the for loop body, k < q always holds. (This justifies the claim that   [q] < q as well, by line 9.) We can pay for each execution of the while loop body on line 6 with the corresponding decrease in the potential function, since   [k] < k. Line 8 increases the potential function by at most one, so that the amortized cost of the loop body on lines 5-9 is O(1). Since the number of outer-loop iterations is O(m), and since the final potential function is at least as great as the initial potential function, the total actual worst-case running time of COMPUTE-PREFIX-FUNCTION is O(m).

The Knuth-Morris-Pratt algorithm runs in time O(m + n). The call of COMPUTE-PREFIX-FUNCTION takes O(m) time as we have just seen, and a similar amortized analysis, using the value of q as the potential function, shows that the remainder of KMP-MATCHER takes O(n) time.

Compared to FlNITE-AUTOMATON-MATCHER, by using    rather than   , we have reduced the time for preprocessing the pattern from O(m |  |) to O(m), while keeping the actual matching time bounded by O(m + n).

Correctness of the prefix-function computation

We start with an essential lemma showing that by iterating the prefix function   , we can enumerate all the prefixes Pk that are suffixes of a given prefix Pq. Let

  *[q] = {q,   [q],   2[q],   3[q], . . . ,   t[q]} ,

where   i[q] is defined in terms of functional composition, so that   0[q] = q and   i+1[q] =   [  i[q]] for i > 1, and where it is understood that the sequence in   *[q] stops when   t[q] = 0 is reached.

Lemma 34.5

Let P be a pattern of length m with prefix function   . Then, for q = 1, 2, . . . , m, we have   .

Proof     We first prove that

i      *[q] implies    .

(34.6)

If i      *[q], then i =   u[q] for some u. We prove equation (34.6) by induction on u. For u = 0, we have i = q, and the claim follows since   . Using the relation    and the transitivity of    establishes the claim for all i in   *[q]. Therefore,   .

Figure 34.10 An illustration of Lemma 34.5 for the pattern P = ababababca and q = 8. (a) The    function for the given pattern. Since   [8] = 6,   [6] = 4,   [4] = 2, and   [2] = 0, by iterating    we obtain   *[8] = {8, 6, 4, 2, 0}. (b) We slide the template containing the pattern P to the right and note when some prefix Pk of P matches up with some proper suffix of P8; this happens for k = 6, 4, 2, and 0. In the figure, the first row gives P, and the dotted vertical line is drawn just after P8  Successive rows show all the shifts of P that cause some prefix Pk of P to match some suffix of P8. Successfully matched characters are shown shaded. Vertical lines connect aligned matching characters. Thus,   . The lemma claims that    for all q.

We prove that    by contradiction. Suppose to the contrary that there is an integer in the set   , and let j be the largest such value. Because q is in   , we have j < q, and so we let j' denote the smallest integer in   *[q] that is greater than j. (We can choose j' = q if there is no other number in   *[q] that is greater than j.) We have    because    because j'      *[q]; thus,    by Lemma 34.1. Moreover, j is the largest such value with this property. Therefore, we must have   [j'] = j and thus j      *[q]. This contradiction proves the lemma.

Figure 34.10 illustrates this lemma.

The algorithm COMPUTE-PREFIX-FUNCTION computes   [q] in order for q = 1, 2, . . . , m. The computation of   [1] = 0 in line 2 of COMPUTE-PREFIX-FUNCTION is certainly correct, since   [q] < q for all q. The following lemma and its corollary will be used to prove that COMPUTE-PREFIX-FUNCTION computes   [q] correctly for q > 1.

Lemma 34.6

Let P be a pattern of length m, and let    be the prefix function for P. For q = 1, 2, . . . , m, if   [q] > 0, then   [q] - 1      *[q - 1].

Proof     If k =   [q] > 0, then    (by dropping the last character from Pk and Pq). By Lemma 34.5, therefore, k - l      *[q - 1].

For q = 2, 3, . . . , m, define the subset Eq-1      *[q - 1] by

Eq-1 = {k : k      *[q - 1] and P[k + 1] = P[q]} .

The set Eq-1 consists of the values k for which    (by Lemma 34.5); because P[k + 1] = P[q], it is also the case that for these values of k,   . Intuitively, Eq - 1 consists of those values k      *[q - 1] such that we can extend Pk to Pk+1 and get a suffix of Pq.

Corollary 34.7

Let P be a pattern of length m, and let    be the prefix function for P. For q = 2, 3, . . . , m,

Proof     If r =   [q], then   , and so r    1 implies P[r] = P[q]. By Lemma 34.6, therefore, if r    1, then

r = 1 + max {k      *[q - 1]: P[k + 1] = P[q]} .

But the set maximized over is just Eq-1, so that r = 1 + max {k    Eq-1} and Eq-1 is nonempty. If r = 0, there is no k      *[q - 1] for which we can extend Pk to Pk+1 and get a suffix of Pq, since then we would have   [q] > 0. Thus,   .

We now finish the proof that COMPUTE-PREFIX-FUNCTION computes    correctly. In the procedure COMPUTE-PREFIX-FUNCTION, at the start of each iteration of the for loop of lines 4-9, we have that k =   [q - 1]. This condition is enforced by lines 2 and 3 when the loop is first entered, and it remains true in each successive iteration because of line 9. Lines 5-8 adjust k so that it now becomes the correct value of   [q]. The loop on lines 5-6 searches through all values k      *[q - 1] until one is found for which P[k + 1] = P[q]; at that point, we have that k is the largest value in the set Eq-1, so that, by Corollary 34.7, we can set   [q] to k + 1. If no such k is found, k = 0 in lines 7-9, and   [q] is set to 0. This completes our proof of the correctness of COMPUTE-PREFIX-FUNCTION.

Correctness of the KMP algorithm

The procedure KMP-MATCHER can be viewed as a reimplementation of the procedure FINlTE-AUTOMATON-MATCHER. Specifically, we shall prove that the code on lines 6-9 of KMP-MATCHER is equivalent to line 4 of FINITE-AUTOMATON-MATCHER, which sets q to   (q,T[i]). Instead of using a stored value of   (q, T[i]), however, this value is recomputed as necessary from   . Once we have argued that KMP-MATCHER simulates the behavior of FINITE-AUTOMATON-MATCHER, the correctness of KMP-MATCHER follows from the correctness of FINITE-AUTOMATON-MATCHER (though we shall see in a moment why line 12 in KMP-MATCHER is necessary).

The correctness of KMP-MATCHER follows from the claim that either   (q, T[i]) = 0 or else   (q, T[i]) - 1      *[q]. To check this claim, let k =   (q, T[i]). Then,    by the definitions of    and   . Therefore, either k = 0 or else k    1 and    by dropping the last character from both Pk and PqT[i] (in which case k - 1      *[q]). Therefore, either k = 0 or k - 1      *[q], proving the claim.

The claim is used as follows. Let q' denote the value of q when line 6 is entered. We use the equivalence    to justify the iteration q      [q] that enumerates the elements of   . Lines 6-9 determine   (q', T[i]) by examining the elements of   *[q'] in decreasing order. The code uses the claim to begin with q =   (Ti - 1) =   (Ti - 1) and perform the iteration q      [q] until a q is found such that q = 0 or P[q + 1] = T[i]. In the former case,   (q', T[i]) = 0; in the latter case, q is the maximum element in Eq', so that    (q', T[i]) = q + 1 by Corollary 34.7.

Line 12 is necessary in KMP-MATCHER to avoid a possible reference to P[m + 1] on line 6 after an occurrence of P has been found. (The argument that q =   (Ti - 1) upon the next execution of line 6 remains valid by the hint given in Exercise 34.4-6:   (m, a) =   (  [m], a) or, equivalently,   (Pa) =   (P  [m]a) for any a      .) The remaining argument for the correctness of the Knuth-Morris-Pratt algorithm follows from the correctness of FINITE-AUTOMATON-MATCHER, since we now see that KMP-MATCHER simulates the behavior of FINITE-AUTOMATON-MATCHER.

Exercises

34.4-1

Compute the prefix function    for the pattern ababbabbababbababbabb when the alphabet is    = {a, b}.

34.4-2

Give an upper bound on the size of   *[q] as a function of q. Give an example to show that your bound is tight.

34.4-3

Explain how to determine the occurrences of pattern P in the text T by examining the    function for the string PT (the string of length m + n that is the concatenation of P and T).

34.4-4

Show how to improve KMP-MATCHER by replacing the occurrence of    in line 7 (but not line 12) by   ', where   ' is defined recursively for q = 1, 2, . . . , m by the equation

Explain why the modified algorithm is correct, and explain in what sense this modification constitutes an improvement.

34.4-5

Give a linear-time algorithm to determine if a text T is a cyclic rotation of another string T'. For example, arc and car are cyclic rotations of each other.

34.4-6

Give an efficient algorithm for computing the transition function    for the string-matching automaton corresponding to a given pattern P. Your algorithm should run in time O(m|  |). (Hint: Prove that   (q, a) =   (  [q], a ) if q = m or P[q + 1]    a.)

* 34.5 The Boyer-Moore algorithm

If the pattern P is relatively long and the alphabet    is reasonably large, then an algorithm due to Robert S. Boyer and J. Strother Moore is likely to be the most efficient string-matching algorithm.

BOYER-MOORE-MATCHER(T, P,   )

1 n    length[T]

2 m    length[P]

3       COMPUTE-LAST-OCCURRENCE-FUNCTION(P, m,   )

4       COMPUTE-GOOD-SUFFIX-FUNCTION(P, m)

5 s    0

6 while s    n - m

7      do j    m

8         while j > 0 and P[j] = T[s + j]

9             do j    j - 1

10        if j = 0

11           then print "Pattern occurs at shift" s

12                s    s +   [0]

13           else s    s + max(  [j], j -   [T[s + j]])

Aside from the mysterious-looking   's and   's, this program looks remarkably like the naive string-matching algorithm. Indeed, suppose we comment out lines 3-4 and replace the updating of s on lines 12-13 with simple incrementations as follows:

12                s    s + 1

13           else s    s + 1

The modified program now acts exactly like the naive string matcher: the while loop beginning on line 6 considers each of the n - m + 1 possible shifts s in turn, and the while loop beginning on line 8 tests the condition P[1 . . m] = T[s + 1 . . s + m] by comparing P[j] with T[s + j] for j = m, m - 1, . . . , 1. If the loop terminates with j = 0, a valid shift s has been found, and line 11 prints out the value of s. At this level, the only remarkable features of the Boyer-Moore algorithm are that it compares the pattern against the text from right to left and that it increases the shift s on lines 12-13 by a value that is not necessarily 1.

The Boyer-Moore algorithm incorporates two heuristics that allow it to avoid much of the work that our previous string-matching algorithms performed. These heuristics are so effective that they often allow the algorithm to skip altogether the examination of many text characters. These heuristics, known as the "bad-character heuristic" and the "good-suffix heuristic," are illustrated in Figure 34.11. They can be viewed as operating independently in parallel. When a mismatch occurs, each heuristic proposes an amount by which s can safely be increased without missing a valid shift. The Boyer-Moore algorithm chooses the larger amount and increases s by that amount: when line 13 is reached after a mismatch, the bad-character heuristic proposes increasing s by j -   [T[s + j]], and the good-suffix heuristic proposes increasing s by   [j].

Figure 34.11 An illustration of the Boyer-Moore heuristics. (a) Matching the pattern reminiscence against a text by comparing characters in a right-to-left manner. The shift s is invalid; although a "good suffix" ce of the pattern matched correctly against the corresponding characters in the text (matching characters are shown shaded), the "bad character" i, which didn't match the corresponding character n in the pattern, was discovered in the text. (b) The bad-character heuristic proposes moving the pattern to the right, if possible, by the amount that guarantees that the bad text character will match the rightmost occurrence of the bad character in the pattern. In this example, moving the pattern 4 positions to the right causes the bad text character i in the text to match the rightmost i in the pattern, at position 6. If the bad character doesn't occur in the pattern, then the pattern may be moved completely past the bad character in the text. If the rightmost occurrence of the bad character in the pattern is to the right of the current bad character position, then this heuristic makes no proposal. (c) With the good-suffix heuristic, the pattern is moved to the right by the least amount that guarantees that any pattern characters that align with the good suffix ce previously found in the text will match those suffix characters. In this example, moving the pattern 3 positions to the right satisfies this condition. Since the good-suffix heuristic proposes a movement of 3 positions, which is smaller than the 4-position proposal of the bad-character heuristic, the Boyer-Moore algorithm increases the shift by 4.

The bad-character heuristic

When a mismatch occurs, the bad-character heuristic uses information about where the bad text character T[s + j] occurs in the pattern (if it occurs at all) to propose a new shift. In the best case, the mismatch occurs on the first comparison (P[m]    T[s + m]) and the bad character T[s + m] does not occur in the pattern at all. (Imagine searching for am in the text string bn.) In this case, we can increase the shift s by m, since any shift smaller than s + m will align some pattern character against the bad character, causing a mismatch. If the best case occurs repeatedly, the Boyer-Moore algorithm examines only a fraction 1/m of the text characters, since each text character examined yields a mismatch, thus causing s to increase by m. This best-case behavior illustrates the power of matching right-to-left instead of left-to-right.

In general, the bad-character heuristic works as follows. Suppose we have just found a mismatch: P[j]    T[s + j] for some j, where 1    j    m. We then let k be the largest index in the range 1    k    m such that T[s + j] = P[k], if any such k exists. Otherwise, we let k = 0. We claim that we may safely increase s by j - k. We must consider three cases to prove this claim, as illustrated by Figure 34.12.

       k = 0: As shown in Figure 34.12(a), the bad character T[s + j] didn't occur in the pattern at all, and so we can safely increase s by j without missing any valid shifts.

       k < j: As shown in Figure 34.12(b), the rightmost occurrence of the bad character is in the pattern to the left of position j, so that j - k > 0 and the pattern must be moved j - k characters to the right before the bad text character matches any pattern character. Therefore, we can safely increase s by j - k without missing any valid shifts.

       k > j: As shown in Figure 34.12(c), j - k < 0, and so the bad-character heuristic is essentially proposing to decrease s. This recommendation will be ignored by the Boyer-Moore algorithm, since the good-suffix heuristic will propose a shift to the right in all cases.

The following simple program defines   [a] to be the index of the rightmost position in the pattern at which character a occurs, for each a      . If a does not occur in the pattern, then   [a] is set to 0. We call    the last-occurrence function for the pattern. With this definition, the expression j -   [T[s + j]] on line 13 of BOYER-MOORE MATCHER implements the bad-character heuristic. (Since j -   [T[s + j]] is negative if the rightmost occurrence of the bad character T[s + j] in the pattern is to the right of position j, we rely on the positivity of   [j], proposed by the good-suffix heuristic, to ensure that the algorithm makes progress at each step.)

Figure 34.12 The cases of the bad-character heuristic. (a) The bad character h occurs nowhere in the pattern, and so the pattern can be advanced j = 11 characters until it has passed over the bad character. (b) The rightmost occurrence of the bad character in the pattern is at position k < j, and so the pattern can be advanced j - k characters. Since j = 10 and k = 6 for the bad character i, the pattern can be advanced 4 positions until the i's line up. (c) The rightmost occurrence of the bad character in the pattern is at position k > j. In this example, j = 10 and k = 12 for the bad character e. The bad-character heuristic proposes a negative shift, which is ignored.

COMPUTE-LAST-OCCURRENCE-FUNCTION(P, m,   )

1 for each character a

2      do   [a] = 0

3 for j   1 to m

4      do   [P[j]]    j

5 return

The running time of procedure COMPUTE-LAST-OCCURRENCE-FUNCTION is O(|  | + m).

The good-suffix heuristic

Let us define the relation Q ~ R (read "Q is similar to R") for strings Q and R to mean that   . If two strings are similar, then we can align them with their rightmost characters matched, and no pair of aligned characters will disagree. The relation "~" is symmetric: Q ~ R if and only if R ~ Q. We also have, as a consequence of Lemma 34.1, that

(34.7)

If we find that P[j]    T[s + j], where j < m, then the good-suffix heuristic says that we can safely advance s by

   [j] = m - max {k: 0    k < m and P[j + 1 . . m] ~ Pk} .

That is,   [j] is the least amount we can advance s and not cause any characters in the "good suffix" T[s + j + 1 . . s + m] to be mismatched against the new alignment of the pattern. The function    is well defined for all j, since P[j + 1 . . m] ~ P0 for all j: the empty string is similar to all strings. We call    the good-suffix function for the pattern P.

We now show how to compute the good-suffix function   . We first observe that   [j]    m -   [m] for all j, as follows. If w =   [m], then    by the definition of   . Furthermore, since    for any j, we have Pw ~ P[j + 1 . . m], by equation (34.7). Therefore,   [j]    m -   [m] for all j.

We can now rewrite our definition of    as

  [j] = m - max {k :   [m]    k < m and P[j + 1 . . m] ~ Pk} .

The condition that P[j + 1 . . m] ~ Pk holds if either   . But the latter possibility implies that    and thus  that k      [m], by the definition of   . This latter possibility cannot reduce the value of   [j] below m -   [m]. We can therefore rewrite our definition of    still further as follows:

(The second set may be empty.) It is worth observing that the definition implies that   [j] > 0 for all j = 1, 2, . . . , m, which ensures that the Boyer-Moore algorithm makes progress.

To simplify the expression for    further, we define P' as the reverse of the pattern P and   ' as the corresponding prefix function. That is P'[i] = P[m - i + 1] for i = 1, 2, . . . , m, and   '[t] is the largest u such that u < t and   .

If k is the largest possible value such that   , then we claim that

  '[l] = m - j ,

(34.8)

where l = (m - k) + (m - j). To see that this claim is well defined, note that    implies that m - j    k, and thus l    m. Also, j < m and k    m, so that l    1. We prove this claim as follows. Since   . Therefore,   '[l]    m - j. Suppose now that p > m - j, where p =   '[l]. Then, by the definition of   ', we have    or, equivalently, P'[1 . . p] = P'[l - p + 1 . . l]. Rewriting this equation in terms of P rather than P', we have P[m - p + 1 . . m] = P[m - l + 1 . . m - l + p]. Substituting for l = 2m - k - j, we obtain P[m - p + 1 . . m] = P[k - m + j + 1 . . k - m + j +p], which implies   . Since p > m - j, we have j + 1 > m-p+1, and so   , implying that    by the transitivity of   . Finally, since p > m - j, we have k' > k, where k' = k - m + j + p, contradicting our choice of k as the largest possible value such that   . This contradiction means that we can't have p > m - j, and hus  = m - j, which proves the claim (34.8).

Using equation (34.8), and noting that   '[l] = m - j implies that j = m -   '[l] and k = m - l +   '[l], we can rewrite our definition of    still further:

  [j]  =  m - max({  [m]}

   {m - l +   '[l]: 1    l    m and j = m -   '[l]})

=  min({m -   [m]}

   {l -   '[l]: 1    l    m and j = m -   '[l]}) .

(34.9)

Again, the second set may be empty.

We are now ready to examine the procedure for computing   .

COMPUTE-GOOD-SUFFIX-FUNCTION(P, m)

1       COMPUTE-PREFIX-FUNCTION(P)

2 P'    reverse(P)

3   '    COMPUTE-PREFIX-FUNCTION(P')

4 for j    0 to m

5      do   [j]    m -   [m]

6 for l    1 to m

7      do j    m -   '[l]

8         if   [j] > l -   '[l]

9            then   [j]    l -   '[l]

10 return

The procedure COMPUTE-GOOD-SUFFIX-FUNCTION is a straightforward implementation of equation (34.9). Its running time is O(m).

The worst-case running time of the Boyer-Moore algorithm is clearly O((n - m +1)m + |  |), since COMPUTE-LAST-OCCURRENCE-FUNCTION takes time O(m + |  |), COMPUTE-GOOD-SUFFIX-FUNCTION takes time O(m), and the Boyer-Moore algorithm (like the Rabin-Karp algorithm) spends O(m) time validating each valid shift s. In practice, however, the Boyer-Moore algorithm is often the algorithm of choice.

Exercises

34.5-1

Compute the    and    functions for the pattern P = 0101101201 and the alphabet    = {0, 1, 2}.

34.5-2

Give examples to show that by combining the bad-character and good-suffix heuristics, the Boyer-Moore algorithm can perform much better than if it used just the good-suffix heuristic.

34.5-3

An improvement to the basic Boyer-Moore procedure that is often used in practice is to replace the    function by   ', defined by

  '[j] = m - max{k : 0    k < m and P [j + 1 . . m] ~ Pk and

(k - m + j > 0 implies P[j]    P[k - m + j])} .

In addition to ensuring that the characters in the good suffix will be mis-matched at the new shift, the   ' function also guarantees that the same pattern character will not be matched up against the bad text character. Show how to compute the   ' function efficiently.

Problems

34-1     String matching based on repetition factors

Let yi denote the concatenation of string y with itself i times. For example, (ab)3 = ababab. We say that a string x      * has repetition factor r if x = yr for some string y      * and some r > 0. Let p(x) denote the largest r such that x has repetition factor r.

a.     Give an efficient algorithm that takes as input a pattern P[1 . . m] and computes   (Pi) for i = 1, 2, . . . , m. What is the running time of your algorithm?

b.     For any pattern P[1 . . m], let p*(P) be defined as max1  i  m   (Pi). Prove that if the pattern P is chosen randomly from the set of all binary strings of length m, then the expected value of   *(P) is O(1).

c.     Argue that the following string-matching algorithm correctly finds all occurrences of pattern P in a text T[1 . . n] in time O(  *(P)n + m).

REPETITION-MATCHER(P,T)

1 m    length[P]

2 n    length[T]

3 k    1 + p*(P)

4 q    0

5 s    0

6 while s    n - m

7      do if T[s + q + 1] = P[q + 1]

8            then q    q + 1

9                 if q = m

10                     then print "Pattern occurs with shift" s

11         if q = m or T[s + q + 1]    P[q + 1]

12            then s    s + max(1,   q/k  )

13                 q    0

This algorithm is due to Galil and Seiferas. By extending these ideas greatly, they obtain a linear-time string-matching algorithm that uses only O(1) storage beyond what is required for P and T.

34-2     Parallel string matching

Consider the problem of string matching on a parallel computer. Assume that for a given pattern, we have a string-matching automaton M with state set Q. Let    be the final-state function for M. Suppose that our input ext is T[1 . . n]. We wish to compute   (Ti) for i = 1, 2, . . . , n; that is, we wish to compute the final state for each refix. Our strategy is to use the parallel prefix computation described in Section 30.1.2.

For any input string x, define the function   x : Q    Q such that if M starts in state q and reads input x, then M ends in state   x(q).

a.     Prove that    denotes functional composition:

b.     Argue that    is an associative operation.

c.     Argue that   xy can be computed from tabular representations of   x and   y in O(1) time on a CREW PRAM. Analyze how many processors are needed in terms of |Q|."

d.     Prove that    (Ti) =   Ti(q0), where q0 is the start state for M.

e.     Show how to find all occurrences of a pattern in a text of length n in O(lg n ) time on a CREW PRAM. Assume that the pattern is supplied in the form of the corresponding string-matching automaton.

Chapter notes

The relation of string matching to the theory of finite automata is discussed by Aho, Hopcroft, and Ullman [4]. The Knuth-Morris-Pratt algorithm [125] was invented independently by Knuth and Pratt and by Morris; they published their work jointly. The Rabin-Karp algorithm was proposed by Rabin and Karp [117], and the Boyer-Moore algorithm is due to Boyer and Moore [32]. Galil and Seiferas [78] give an interesting deterministic linear-time string-matching algorithm that uses only O(1) space beyond that required to store the pattern and text.

Go to Chapter 35     Back to Table of Contents


Intro to Algorithms: CHAPTER 35: COMPUTATIONAL GEOMETRY

 Return to Table of Contents

CHAPTER 35: COMPUTATIONAL GEOMETRY

Computational geometry is the branch of computer science that studies algorithms for solving geometric problems. In modern engineering and mathematics, computational geometry has applications in, among other fields, computer graphics, robotics, VLSI design, computer-aided design, and statistics. The input to a computational-geometry problem is typically a description of a set of geometric objects, such as a set of points, a set of line segments, or the vertices of a polygon in counterclockwise order. The output is often a response to a query about the objects, such as whether any of the lines intersect, or perhaps a new geometric object, such as the convex hull (smallest enclosing convex polygon) of the set of points.

In this chapter, we look at a few computational-geometry algorithms in two dimensions, that is, in the plane. Each input object is represented as a set of points {pi}, where each pi = (xi, yi) and xi, yi [image] R. For example, an n-vertex polygon P is represented by a sequence [image] p0, p1, p2, . . . , pn-1 [image] of its vertices in order of their appearance on the boundary of P. Computational geometry can also be performed in three dimensions, and even in higher- dimensional spaces, but such problems and their solutions can be very difficult to visualize. Even in two dimensions, however, we can see a good sample of computational-geometry techniques.

Section 35.1 shows how to answer simple questions about line segments efficiently and accurately: whether one segment is clockwise or counterclockwise from another that shares an endpoint, which way we turn when traversing two adjoining line segments, and whether two line segments intersect. Section 35.2 presents a technique called "sweeping" that we use to develop an O(n lg n)-time algorithm for determining whether there are any intersections among a set of n line segments. Section 35.3 gives two "rotational-sweep" algorithms that compute the convex hull (smallest enclosing convex polygon) of a set of n points: Graham's scan, which runs in time O(n 1g n), and Jarvis's march, which takes O(nh) time, where h is the number of vertices of the convex hull. Finally, Section 35.4 gives an O(n 1g n)-time divide-and-conquer algorithm for finding the closest pair of points in a set of n points in the plane.

35.1 Line-segment properties

Several of the computational-geometry algorithms in this chapter will require answers to questions about the properties of line segments. A convex combination of two distinct points p1 = (x1, y1) and p2 = (x2, y2) is any point p3 = (x3, y3) such that for some [image] in the range 0 [image] [image] [image] 1, we have x3 = ax1 + (1 - [image])x2 and y3 = [image] y1 + (1 - [image])y2. We also write that p3 = [image] p1 + (1 - [image])p2. Intuitively, p3 is any point that is on the line passing through p1 and p2 and is on or between p1 and p2 on the line. Given two distinct points p1 and p2, the line segment [image] is the set of convex combinations of p1 and p2. We call p1 and p2 the endpoints of segment [image]. Sometimes the ordering of p1 and p2 matters, and we speak of the directed segment [image]. If p1 is the origin (0, 0), then we can treat the directed segment [image] as the vector p2.

In this section, we shall explore the following questions:

There are no restrictions on the given points.

We can answer each question in O(1) time, which should come as no surprise since the input size of each question is O(1). Moreover, our methods will use only additions, subtractions, multiplications, and comparisons. We need neither division nor trigonometric functions, both of which can be computationally expensive and prone to problems with round-off error. For example, the "straightforward" method of determining whether two segments intersect--compute the line equation of the form y = mx + b for each segment (m is the slope and b is the y-intercept), find the point of intersection of the lines, and check whether this point is on both segments--uses division to find the point of intersection. When the segments are nearly parallel, this method is very sensitive to the precision of the division operation on real computers. The method in this section, which avoids division, is much more accurate.

Cross products

Computing cross products is at the heart of our line-segment methods. Consider vectors p1 and p2, shown in Figure 35.1(a). The cross product p1 x p2 can be interpreted as the signed area of the parallelogram formed by the points (0, 0), p1, p2, and p1 + p2 = (x1 + x2, y1 + y2). An equivalent, but more useful, definition gives the cross product as the determinant of

Figure 35.1 (a) The cross product of vectors p1 and p2 is the signed area of the parallelogram. (b) The lightly shaded region contains vectors that are clockwise from p. The darkly shaded region contains vectors that are counterclockwise from p.

a matrix:1

1 Actually, the cross product is a three-dimensional concept. It is a vector that is perpendicular to both p1 and p2 according to the "right-hand rule" and whose magnitude is |x1y2 - x2y1|. In this chapter, however, it will prove convenient to treat the cross product simply as the value x1y2 - x2y1.

If p1 X p2 is positive, then p1 is clockwise from p2 with respect to the origin (0, 0); if this cross product is negative, then p1 is counterclockwise from p2. Figure 35.1(b) shows the clockwise and counterclockwise regions relative to a vector p. A boundary condition arises if the cross product is zero; in this case, the vectors are collinear, pointing in either the same or opposite directions.

To determine whether a directed segment [image] is clockwise from a directed segment [image] with respect to their common endpoint p0, we simply translate to use p0 as the origin. That is, we let p1 - p0 denote the vector p'1 = (x'1, y'1), where x'1 = x1 - x0 and y'1 = y1 - y0, and we define p2 - p0 similarly. We then compute the cross product

(p1 - p0) x (p2 - p0) = (x1 - x0) (y2 - y0) - (x2 - x0) (y1 - y0).

If this cross product is positive, then [image] is clockwise from [image]; if negative, it is counterclockwise.

Determining whether consecutive segments turn left or right

Our next question is whether two consecutive line segments [image] turn left or right at point pl. Equivalently, we want a method to determine which way a given angle [image] p0p1p2 turns. Cross products allow us to answer this question without computing the angle. As shown in Figure 35.2, we simply check whether directed segment [image] is clockwise or counterclockwise relative to directed segment [image]. To do this, we compute the cross product (p2 - p0) X (p1 - p0). If the sign of this cross product is negative, then [image] is counterclockwise with respect to [image], and thus we make a left turn at P1. A positive cross product indicates a clockwise orientation and a right turn. A cross product of 0 means that points p0, p1, and p2 are collinear.

Figure 35.2 Using the cross product to determine how consecutive line segments turn at point p1. We check whether the directed segment is clockwise or counterclockwise relative to the directed segment . (a) If counterclockwise, the points make a left turn. (b) If clockwise, they make a right turn.

Determining whether two line segments intersect

We use a two-stage process to determine whether two line segments intersect. The first stage is quick rejection: the line segments cannot intersect if their bounding boxes do not intersect. The bounding box of a geometric figure is the smallest rectangle that contains the figure and whose segments are parallel to the x-axis and y-axis. The bounding box of line segment [image] is represented by the rectangle [image] with lower left point [image] and upper right point [image], where [image]. Two rectangles, represented by lower left and upper right points [image], intersect if and only if the conjunction

is true. The rectangles must intersect in both dimensions. The first two comparisons above determine whether the rectangles intersect in x; the second two comparisons determine whether the rectangles intersect in y.

The second stage in determining whether two line segments intersect decides whether each segment "straddles" the line containing the other. A segment [image] straddles a line if point p1 lies on one side of the line and point p2 lies on the other side. If p1 or p2 lies on the line, then we say that the segment straddles the line. Two line segments intersect if and only if they pass the quick rejection test and each segment straddles the line containing the other.

Figure 35.3 Determining whether line segment straddles the line containing segment . (a) If it does straddle, then the signs of the cross products (p3 - p1) x (p2 - p1) and (p4 - p1) x (p2 - p1) differ. (b) If it does not straddle, then the signs of the cross products are the same. (c)-(d) Boundary cases in which at least one of the cross products is zero and the segment straddles. (e) A boundary case in which the segments are collinear but do not intersect. Both cross products are zero, but they would not be computed by our algorithm because the segments fail the quick rejection test--their bounding boxes do not intersect.

We can use the cross-product method to determine whether line segment [image] straddles the line containing points p1 and p2. The idea, as shown in Figures 35.3(a) and (b), is to determine whether directed segments [image] and [image] have opposite orientations relative to [image]. If so, then the segment straddles the line. Recalling that we can determine relative orientations with cross products, we just check whether the signs of the cross products (p3 - p1) X (p2 - p1) and (p4 - p1) X (p2 - p1) are different. A boundary condition occurs if either cross product is zero. In this case, either p3 or p4 lies on the line containing segment [image]. Because the two segments have already passed the quick rejection test, one of the points p3 and p4 must in fact lie on segment [image]. Two such situations are shown in Figures 35.3(c) and (d). The case in which the two segments are collinear but do not intersect, shown in Figure 35.3(e), is eliminated by the quick rejection test. A final boundary condition occurs if one or both of the segments has zero length, that is, if its endpoints are coincident. If both segments have zero length, then the quick rejection test suffices. If just one segment, say [image], has zero length, then the segments intersect if and only if the cross product (p3 - p1) x (p2 - p1) is zero.

Other applications of cross products

Later sections of this chapter will introduce additional uses for cross products. In Section 35.3, we shall need to sort a set of points according to their polar angles with respect to a given origin. As Exercise 35.1-2 asks you to show, cross products can be used to perform the comparisons in the sorting procedure. In Section 35.2, we shall use red-black trees to maintain the vertical ordering of a set of nonintersecting line segments. Rather than keeping explicit key values, we shall replace each key comparison in the red-black tree code by a cross-product calculation to determine which of two segments that intersect a given vertical line is above the other.

Exercises

35.1-1

Prove that if p1 X p2 is positive, then vector p1 is clockwise from vector p2 with respect to the origin (0, 0) and that if this cross product is negative, then p1 is counterclockwise from p2.

35.1-2

Write pseudocode to sort a sequence [image] p 1, p2, . . . . ,pn) of n points according to their polar angles with respect to a given origin point p0. Your procedure should take 0(n lg n) time and use cross products to compare angles.

35.1-3

Show how to determine in 0(n2 1g n) time whether any three points in a set of n points are collinear.

35.1-4

Professor Amundsen proposes the following method to determine whether a sequence [image] p0, p1, . . . . , pn-1 [image] of n points forms the consecutive vertices of a convex polygon. (See Section 16.4 for definitions pertaining to polygons.) Output "yes" if the set { [image] pipi + 1pi + 2 : i = 0, 1, . . . ,n - 1}, where subscript addition is performed modulo n, does not contain both left turns and right turns; otherwise, output "no." Show that although this method runs in linear time, it does not always produce the correct answer. Modify the professor's method so that it always produces the correct answer in linear time.

35.1-5

Given a point p0 = (x0, y0), the right horizontal ray from p0 is the set of points { pi = (xi, yi) : xi [image] x0 and yi = y0}, that is, it is the set of points due right of p0 along with p0 itself. Show how to determine whether a given right horizontal ray from p0 intersects a line segment [image] in O(1) time by reducing the problem to that of determining whether two line segments intersect.

35.1-6

One way to determine whether a point p0 is in the interior of a simple, but not necessarily convex, polygon P is to look at any ray from p0 and check that the ray intersects the boundary of P an odd number of times but that p0 itself is not on the boundary of P. Show how to compute in [image] (n) time whether a point p0 is in the interior of an n-vertex polygon P. (Hint: Use Exercise 35.1-5. Make sure your algorithm is correct when the ray intersects the polygon boundary at a vertex and when the ray overlaps a side of the polygon.)

35.1-7

Show how to compute the area of an n-vertex simple, but not necessarily convex, polygon in [image] (n) time.

35.2 Determining whether any pair of segments intersects

This section presents an algorithm for determining whether any two line segments in a set of segments intersect. The algorithm uses a technique known as "sweeping," which is common to many computational-geometry algorithms. Moreover, as the exercises at the end of this section show, this algorithm, or simple variations of it, can be used to solve other computational-geometry problems.

The algorithm runs in O(n lg n) time, where n is the number of segments we are given. It determines only whether or not any intersection exists; it does not print all the intersections. (By Exercise 35.2-1, it takes [image] (n2) time in the worst case to find all the intersections in a set of n line segments.)

In sweeping, an imaginary vertical sweep line passes through the given set of geometric objects, usually from left to right. The spatial dimension that the sweep line moves across, in this case the x-dimension, is treated as a dimension of time. Sweeping provides a method for ordering geometric objects, usually by placing them into a dynamic data structure, and for taking advantage of relationships among them. The line-segment-intersection algorithm in this section considers all the line-segment endpoints in left-to-right order and checks for an intersection each time it encounters an endpoint.

Our algorithm for determining whether any two of n line segments intersect makes two simplifying assumptions. First, we assume that no input segment is vertical. Second, we assume that no three input segments intersect at a single point. (Exercise 35.2-8 asks you to describe an implementation that works even if these assumptions fail to hold.) Indeed, removing such simplifying assumptions and dealing with boundary conditions is often the most difficult part of programming computational-geometry algorithms and proving their correctness.

Figure 35.4 The ordering among line segments at various vertical sweep lines. (a) We have a >r c, a >t b, b >t c, a >t c, and b >u c. Segment d is comparable with no other segment shown. (b) When segments e and f intersect, their orders are reversed: we have e >v f but f >w e. Any sweep line (such as z) that passes through the shaded region has e and f consecutive in its total order.

Ordering segments

Since we assume that there are no vertical segments, any input segment that intersects a given vertical sweep line intersects it at a single point. We can thus order the segments that intersect a vertical sweep line according to the y-coordinates of the points of intersection.

To be more precise, consider two nonintersecting segments s1 and s2 We say that these segments are comparable at x if the vertical sweep line with x-coordinate x intersects both of them. We say that sl is above s2 at x, written s1 > X s2, if s1 and s2 are comparable at x and the intersection of s1 with the sweep line at x is higher than the intersection of s2 with the same sweep line. In Figure 35.4(a), for example, we have the relationships a >r c, a >t b, b >t c, a >t c, and b >u c. Segment d is not comparable with any other segment.

For any given x, the relation ">x" is a total order (see Section 5.2) on segments that intersect the sweep line at X. The order may differ for differing values of x, however, as segments enter and leave the ordering. A segment enters the ordering when its left endpoint is encountered by the sweep, and it leaves the ordering when its right endpoint is encountered.

What happens when the sweep line passes through the intersection of two segments? As Figure 35.4(b) shows, their positions in the total order are reversed. Sweep lines v and w are to the left and right, respectively, of the point of intersection of segments e and f, and we have e >v f and f >w e. Note that because we assume that no three segments intersect at the same point, there must be some vertical sweep line x for which intersecting segments e and f are consecutive in the total order >x. Any sweep line that passes through the shaded region of Figure 35.4(b), such as z, has e and f consecutive in its total order.

Moving the sweep line

Sweeping algorithms typically manage two sets of data:

1. The sweep-line status gives the relationships among the objects intersected by the sweep line.

2. The event-point schedule is a sequence of x-coordinates, ordered from left to right, that defines the halting positions of the sweep line. We call each such halting position an event point. Changes to the sweep-line status occur only at event points.

For some algorithms (the algorithm asked for in Exercise 35.2-7, for example), the event-point schedule is determined dynamically as the algorithm progresses. The algorithm at hand, however, determines the event points statically, based solely on simple properties of the input data. In particular, each segment endpoint is an event point. We sort the segment endpoints by increasing x-coordinate and proceed from left to right. We insert a segment into the sweep-line status when its left endpoint is encountered, and we delete it from the sweep-line status when its right endpoint is encountered. Whenever two segments first become consecutive in the total order, we check whether they intersect.

The sweep-line status is a total order T, for which we require the following operations:

INSERT(T, s): insert segment s into T.

DELETE(T, s): delete segment s from T.

ABOVE(T, s): return the segment immediately above segment s in T.

BELOW(T, s): return the segment immediately below segment s in T.

If there are n segments in the input, we can perform each of the above operations in O(lg n) time using red-black trees. Recall that the red-black-tree operations in Chapter 14 involve comparing keys. We can replace the key comparisons by cross-product comparisons that determine the relative ordering of two segments (see Exercise 35.2-2).

Segment-intersection pseudocode

The following algorithm takes as input a set S of n line segments, returning the boolean value TRUE if any pair of segments in S intersects, and FALSE otherwise. The total order T is implemented by a red-black tree.

Figure 35.5 The execution of ANY-SEGMENTS-INTERSECT. Each dashed line is the sweep line at an event point, and the ordering of segment names below each sweep line is the total order T at the end of the for loop in which the corresponding event point is processed. The intersection of segments d and b is found when segment c is deleted.

ANY-SEGMENTS-INTERSECT(S)

2  sort the endpoints of the segments in S from left to right,

breaking ties by putting points with lower y-coordinates first

3  for each point p in the sorted list of endpoints

4      do if p is the left endpoint of a segment s

5            then INSERT(T, s)

6                 if (ABOVE(T, s) exists and intersects s)

or (BELOW(T, s) exists and intersects s)

7                    then return TRUE

8                 if p is the right endpoint of a segment s

9                    then if both ABOVE(T, s) and BELOW(T, s) exist

and ABOVE(T, s) intersects BELOW(T, s)

10                            then return TRUE

11                         DELETE(T, s)

12  return FALSE

Figure 35.5 illustrates the execution of the algorithm. Line 1 initializes the total order to be empty. Line 2 determines the event-point schedule by sorting the 2n segment endpoints from left to right, breaking ties by putting points with lower y-coordinates first. Note that line 2 can be performed by lexicographically sorting the endpoints on (x, y).

Each iteration of the for loop of lines 3-11 processes one event point p. If p is the left endpoint of a segment s, line 5 adds s to the total order, and lines 6-7 return TRUE if s intersects either of the segments it is consecutive with in the total order defined by the sweep line passing through p. (A boundary condition occurs if p lies on another segment s'. In this case, we only require that s and s' be placed consecutively into T.) If p is the right endpoint of a segment s, then s is to be deleted from the total order. Lines 9-10 return TRUE if there is an intersection between the segments surrounding s in the total order defined by the sweep line passing through p; these segments will become consecutive in the total order when s is deleted. If these segments do not intersect, line 11 deletes segment s from the total order. Finally, if no intersections are found in processing all the 2n event points, line 12 returns FALSE.

Correctness

The following theorem shows that ANY-SEGMENTS-INTERSECT is correct.

Theorem 35.1

The call ANY-SEGMENTS-INTERSECT(S) returns TRUE if and only if there is an intersection among the segments in S.

Proof The procedure can be incorrect only by returning TRUE when no intersection exists or by returning FALSE when there is at least one intersection. The former case cannot occur, because ANY-SEGMENTS-INTERSECT returns TRUE only if it finds an intersection between two of the input segments.

To show that the latter case cannot occur, let us suppose for the sake of contradiction that there is at least one intersection, yet ANY-SEGMENTS-INTERSECT returns FALSE. Let p be the leftmost intersection point, breaking ties by choosing the one with the lowest y-coordinate, and let a and b be the segments that intersect at p. Since no intersections occur to the left of p, the order given by T is correct at all points to the left of p. Because no three segments intersect at the same point, there exists a sweep line z at which a and b become consecutive in the total order.2 Moreover, z is to the left of p or goes through p. There exists a segment endpoint q on sweep line z that is the event point at which a and b become consecutive in the total order. If p is on sweep line z, then q = p. If p is not on sweep line z, then q is to the left of p. In either case, the order given by T is correct just before q is processed. (Here we rely on p being the lowest of the leftmost intersection points. Because of the lexicographical order in which event points are processed, even if p is on sweep line z and there is another intersection point p' on z, event point q = p is processed before the other intersection p' can interfere with the total order T.) There are only two possibilities for the action taken at event point q:

2If we allow three segments to intersect at the same point, there may be an intervening segment c that intersects both a and b at point p. That is, we may have a <w c and c <w b for all sweep lines w to the left of p for which a <w b.

1. Either a or b is inserted into T, and the other segment is above or below it in the total order. Lines 4-7 detect this case.

2. Segments a and b are already in T, and a segment between them in the total order is deleted, making a and b become consecutive. Lines 8-11 detect this case.

In either case, the intersection p is found, contradicting the assumption that the procedure returns FALSE.

Running time

If there are n segments in set S, then ANY-SEGMENTS-INTERSECT runs in time O(n lg n). Line 1 takes O(1) time. Line 2 takes O(n lg n) time, using merge sort or heapsort. Since there are 2n event points, the for loop of lines 3-11 iterates at most 2n times. Each iteration takes 0(lg n) time, since each red-black-tree operation takes O(lg n) time and, using the method of Section 35.1, each intersection test takes O(1) time. The total time is thus O(n lg n).

Exercises

35.2-1

Show that there may be [image] (n2) intersections in a set of n line segments.

35.2-2

Given two nonintersecting segments a and b that are comparable at x, show how to use cross products to determine in O(1) time which of a >x b or b >x a holds.

35.2-3

Professor Maginot suggests that we modify ANY-SEGMENTS-INTERSECT so that instead of returning upon finding an intersection, it prints the segments that intersect and continues on to the next iteration of the for loop. The professor calls the resulting procedure PRINT-INTERSECTING-SEGMENTs and claims that it prints all intersections, left to right, as they occur in the set of line segments. Show that the professor is wrong on two counts by giving a set of segments for which the first intersection found by PRINT-INTERSECTING-SEGMENTS is not the leftmost one and a set of segments for which PRINT-INTERSECTING-SEGMENTS fails to find all the intersections.

35.2-4

Give an O(n lg n)-time algorithm to determine whether an n-vertex polygon is simple.

35.2-5

Give an O(n lg n)-time algorithm to determine whether two simple polygons with a total of n vertices intersect.

35.2-6

A disk consists of a circle plus its interior and is represented by its center point and radius. Two disks intersect if they have any point in common. Give an O(n lg n)-time algorithm to determine whether any two disks in a set of n intersect.

35.2-7

Given a set of n line segments containing a total of k intersections, show how to output all k intersections in O((n + k) lg n) time.

35.2-8

Show how to implement the red-black-tree procedures so that ANY-SEGMENTS-INTERSECT works correctly even if some segments are vertical or more than three segments intersect at the same point. Prove that your implementation is correct.

35.3 Finding the convex hull

The convex hull of a set Q of points is the smallest convex polygon P for which each point in Q is either on the boundary of P or in its interior. We denote the convex hull of Q by CH(Q). Intuitively, we can think of each point in Q as being a nail sticking out from a board. The convex hull is then the shape formed by a tight rubber band that surrounds all the nails. Figure 35.6 shows a set of points and its convex hull.

In this section, we shall present two algorithms that compute the convex hull of a set of n points. Both algorithms output the vertices of the convex hull in counterclockwise order. The first, known as Graham's scan, runs in O(n lg n) time. The second, called Jarvis's march, runs in O(nh) time, where h is the number of vertices of the convex hull. As can be seen from Figure 35.6, every vertex of CH(Q) is a point in Q. Both algorithms exploit this property, deciding which vertices in Q to keep as vertices of the convex hull and which vertices in Q to throw out.

Figure 35.6 A set of points Q with its convex hull CH(Q) in gray.

There are, in fact, several methods that compute convex hulls in O(n lg n) time. Both Graham's scan and Jarvis's march use a technique called "rotational sweep," processing vertices in the order of the polar angles they form with a reference vertex. Other methods include the following.

In the incremental method, the points are sorted from left to right, yielding a sequence <p1,p2, . . . , pn>. At the ith stage, the convex hull CH({p1, p2, . . . , pi-1}) of the i - 1 leftmost points is updated according to the ith point from the left, thus forming CH({p1, p2, . . . , pi}). As Exercise 35.3-6 asks you to show, this method can be implemented to take a total of O(n lg n) time.

In the divide-and-conquer method, in [image] (n) time the set of n points is divided into two subsets, one of the leftmost [image] n/2 [image] points and one of the rightmost [image] n/2 [image] points, the convex hulls of the subsets are computed recursively, and then a clever method is used to combine the hulls in O(n) time.

The prune-and-search method is similar to the worst-case linear-time median algorithm of Section 10.3. It finds the upper portion (or "upper chain") of the convex hull by repeatedly throwing out a constant fraction of the remaining points until only the upper chain of the convex hull remains. It then does the same for the lower chain. This method is asymptotically the fastest: if the convex hull contains h vertices, it runs in only O(n lg h) time.

Computing the convex hull of a set of points is an interesting problem in its own right. Moreover, algorithms for some other computational-geometry problems start by computing a convex hull. Consider, for example, the two-dimensional farthest-pair problem: we are given a set of n points in the plane and wish to find the two points whose distance from each other is maximum. As Exercise 35.3-3 asks you to prove, these two points must be vertices of the convex hull. Although we won't prove it here, the farthest pair of vertices of an n-vertex convex polygon can be found in O(n) time. Thus, by computing the convex hull of the n input points in O(n lg n) time and then finding the farthest pair of the resulting convex-polygon vertices, we can find the farthest pair of points in any set of n points in O(n lg n) time.

Graham's scan

Graham's scan solves the convex-hull problem by maintaining a stack S of candidate points. Each point of the input set Q is pushed once onto the stack, and the points that are not vertices of CH(Q) are eventually popped from the stack. When the algorithm terminates, stack S contains exactly the vertices of CH(Q), in counterclockwise order of their appearance on the boundary.

Figure 35.7 The execution of GRAHAM-SCAN on the set Q of Figure 35.6. The current convex hull contained in stack S is shown in gray at each step. (a) The ordered polar angles of p1,p2, . . . , p12 relative to p0 and the initial stack S containing p0, p1, and p2. (b)-(k) Stack S after each iteration of the for loop of lines 7-10. Dashed lines show nonleft turns, which cause points to be popped from the stack. In part (h), for example, the right turn at angle p7p8p9 causes p8 to be popped, and then the right turn at angle p6p7p9 causes p7 to be popped. (l) The convex hull returned by the procedure, which matches that of Figure 35.6.

The procedure GRAHAM-SCAN takes as input a set Q of points, where |Q| [image] 3. It calls the functions TOP(S), which returns the point on top of stack S without changing S, and NEXT-TO-TOP(S), which returns the point one entry below the top of stack S without changing S. As we shall prove in a moment, the stack S returned by GRAHAM-SCAN contains, from bottom to top, exactly the vertices of CH(Q) in counterclockwise order.

GRAHAM-SCAN(Q)

1  let p0 be the point in Q with the minimum y-coordinate,

or the leftmost such point in case of a tie

2  let   p1, p2, . . . , pm   be the remaining points in Q,

sorted by polar angle in counterclockwise order around p0

(if more than point has the same angle, remove all but

the one that is farthest from p0)

3  top[S]   0

4  PUSH(p0,S)

5  PUSH(p1,S)

6  PUSH(p2,S)

7  for i    3 to m

8        do while the angle formed by points NEXT-TO-TOP(S),

TOP(S), and pi makes a nonleft turn

9               do POP(S)

10           PUSH(S,pi)

11  return S

Figure 35.7 illustrates the progress of GRAHAM-SCAN. Line 1 chooses point p0 as the point with the lowest y-coordinate, picking the leftmost such point in case of a tie. Since there is no point in Q that is below p0 and any other points with the same y-coordinate are to its right, p0 is a vertex of CH(Q). Line 2 sorts the remaining points of Q by polar angle relative to p0, using the same method--comparing cross products--as in Exercise 35.1-2. If two or more points have the same polar angle relative to p0, all but the farthest such point are convex combinations of p0 and the farthest point, and so we remove them entirely from consideration. We let m denote the number of points other than p0 that remain. The polar angle, measured in radians, of each point in Q relative to p0 is in the half-open interval [0, [image] /2). Since polar angles increase in a counterclockwise fashion, the points are sorted in counterclockwise order relative to p0. We designate this sorted sequence of points by [image] p1, p2, . . . , pm [image] . Note that points p1 and pm are vertices of CH(Q) (see Exercise 35.3-1). Figure 35.7(a) shows the points of Figure 35.6, with the ordered polar angles of [image] p1, p2, . . . , p12 [image] relative to p0.

The remainder of the procedure uses the stack S Lines 3-6 initialize the stack to contain, from bottom to top, the first three points p0, p1, and p2. Figure 35.7(a) shows the initial stack S. The for loop of lines 7-10 iterates once for each point in the subsequent [image] p3, p4, . . . ,pm) The intent is that after processing point pi, stack S contains, from bottom to top, the vertices of CH({p0,p1, . . . ,pi}) in counterclockwise order. The while loop of lines 8-9 removes points from the stack if they are found not to be vertices of the convex hull. When we traverse the convex hull counterclockwise, we should make a left turn at each vertex. Thus, each time the while loop finds a vertex at which we make a nonleft turn, the vertex is popped from the stack. (By checking for a nonleft turn, rather than just a right turn, this test precludes the possibility of a straight angle at a vertex of the resulting convex hull. This is just what we want, since every vertex of a convex polygon must not be a convex combination of other vertices of the polygon.) After we pop all the vertices that have nonleft turns when heading toward point pi, we push pi onto the stack. Figures 35.7(b)-(k) show the state of the stack S after each iteration of the for loop. Finally, GRAHAM-SCAN returns the stack S in line 11. Figure 35.7(1) shows the corresponding convex hull.

The following theorem formally proves the correctness of GRAHAM-SCAN.

Theorem 35.2

If GRAHAM-SCAN is run on a set Q of points, where |Q| [image] 3, then a point of Q is on the stack S at termination if and only if it is a vertex of CH(Q).

Proof As noted above, a vertex that is a convex combination of p0 and some other vertex in Q is not a vertex of CH(Q). Such a vertex is not included in the sequence [image] pl, p2, . . . , p m [image], and so it can never appear on stack S.

The crux of the proof lies in the two situations shown in Figure 35.8. Part (a) deals with nonleft turns, and part (b) deals with left turns.

We first show that each point popped from stack S is not a vertex of CH(Q). Suppose that point pj is popped from the stack because angle [image] pkpjpi makes a nonleft turn, as shown in Figure 35.8(a). Because we scan the points in order of increasing polar angle relative to point p0, there is a triangle [image] with point pj either in the interior of the triangle or on line segment [image]. In either case, point pj cannot be a vertex of CH(Q).

We now show that each point on stack S is a vertex of CH(Q) at termination. We start by proving the following claim: GRAHAM-SCAN maintains the invariant that the points on stack S always form the vertices of a convex polygon in counterclockwise order.

The claim holds immediately after the execution of line 6, since points p0, p1, and p2 form a convex polygon. Now we examine how stack S changes during the course of GRAHAM-SCAN. Points are either popped or pushed. In the former case, we rely on a simply geometrical property: if a vertex is removed from a convex polygon, the resulting polygon is convex. Thus, popping a point from stack S preserves the invariant.

Figure 35.8 The two basic situations in the proof of correctness of GRAHAM- SCAN. (a) Showing that a point popped from the stack in GRAHAM-SCAN is not a vertex of CH(Q). If point pj is popped from the stack because angle pkpjpi makes a nonleft turn, then the shaded triangle contains point pj. Point pj is therefore not a vertex of CH(Q). (b) If point pi is pushed onto the stack, then there must be a left turn at angle pkpjpi. Because pi follows pj in the polar-angle ordering of points and because of how p0 was chosen, pi must be in the shaded region. If the points on the stack form a convex polygon before the push, then they must form a convex polygon afterward.

Before we consider the case in which a point is pushed onto the stack, let us examine another geometrical property, illustrated in Figures 35.9(a) and (b). Let P be a convex polygon, and choose any side [image] of P. Consider the region bounded by [image] and the extensions of the two adjacent sides. (Depending on the relative angles of the adjacent sides, the region may be either bounded, like the shaded region in part (a), or unbounded, like the shaded region in part (b).) If we add any point ps in this region to P as a new vertex, with the sides [image] replacing side [image], the resulting polygon is convex.

Now consider a point pi that is pushed onto S. Referring back to Figure 35.8(b), let pj be the vertex on the top of S just prior to pushing pi, and let pk be the predecessor of pj on S. We claim that pi must fall within the shaded region of Figure 35.8(b), which corresponds directly to the shaded regions of Figure 35.9. Because the angle [image] pkpjpi makes a left turn, pi must be on the shaded side of the extension of [image]. Because pi follows pj in the polar-angle ordering, it must be on the shaded side of [image]. Moreover, because of how we chose p0, point pi must be on the shaded side of the extension of [image]. Thus, pi is in the shaded region, and therefore after pi has been pushed onto stack S, the points on S form a convex polygon. This completes the proof of the claim.

At the end of GRAHAM-SCAN, therefore, the points of Q that are on stack S form the vertices of a convex polygon. We have shown that all points not on S are not vertices of CH(Q) or, equivalently, that all vertices of CH(Q) are on S. Since S contains only vertices from Q and its points form a convex polygon, they must form CH(Q).

Figure 35.9 Adding a point in the shaded region to a convex polygon P yields another convex polygon. The shaded region is bounded by a side of and the extensions of the two adjacent sides. (a) The shaded region is bounded. (b) The shaded region is unbounded.

We now show that the running time of GRAHAM-SCAN is O(n lg n), where n = |Q|. Line 1 takes [image] (n) time. Line 2 takes O(n lg n) time, using merge sort or heapsort to sort the polar angles and the cross-product method of Section 35.1 to compare angles. (Removing all but the farthest point with the same polar angle can be done in a total of O(n) time.) Lines 3-6 take O(1) time. Because m [image] n - 1, the for loop of lines 7-10 is executed at most n - 3 times. Since PUSH takes O(1) time, each iteration takes O(1) time exclusive of the time spent in the while loop of lines 8-9, and thus overall the for loop takes O(n) time exclusive of the nested while loop.

We use the aggregate method of amortized analysis to show that the while loop takes O(n) time overall. For i = O, 1, . . . , m, each point pi is pushed onto stack S exactly once. As in the analysis of the MULTIPOP procedure of Section 18.1, we observe that there is at most one POP operation for each PUSH operation. At least three points--p0, p1, and pm--are never popped from the stack, so that in fact at most m - 2 POP operations are performed in total. Each iteration of the while loop performs one POP, and so there are at most m - 2 iterations of the while loop altogether. Since the test in line 8 takes O(1) time, each call of POP takes O(1) time, and m [image] n - 1, the total time taken by the while loop is O(n). Thus, the running time of GRAHAM-SCAN is O(n lg n).

Jarvis's march

Jarvis's march computes the convex hull of a set Q of points by a technique known as package wrapping (or gift wrapping). The algorithm runs in time O(nh), where h is the number of vertices of CH(Q). When h is o(lg n), Jarvis's march is asymptotically faster than Graham's scan.

Figure 35.10 The operation of Jarvis's march. The first vertex chosen is the lowest point p0. The next vertex, p1, has the least polar angle of any point with respect to p0. Then, p2 has the least polar angle with respect to p1. The right chain goes as high as the highest point p3. Then, the left chain is constructed by finding least polar angles with respect to the negative x-axis.

Intuitively, Jarvis's march simulates wrapping a taut piece of paper around the set Q. We start by taping the end of the paper to the lowest point in the set, that is, to the same point p0 with which we start Graham's scan. This point is a vertex of the convex hull. We pull the paper to the right to make it taut, and then we pull it higher until it touches a point. This point must also be a vertex of the convex hull. Keeping the paper taut, we continue in this way around the set of vertices until we come back to our original point p0.

More formally, Jarvis's march builds a sequence H = [image] p0, p1, . . . , ph-1 [image] of the vertices of CH(Q). We start with p0. As Figure 35.10 shows, the next convex hull vertex p1 has the least polar angle with respect to p0. (In case of ties, we choose the point farthest from p0.) Similarly, p2 has the least polar angle with respect to pl, and so on. When we reach the highest vertex, say pk (breaking ties by choosing the farthest such vertex), we have constructed, as Figure 35.10 shows, the right chain of CH(Q). To construct the left chain, we start at pk and choose pk+l as the point with the least polar angle with respect to pk, but from the negative x-axis. We continue on, forming the left chain by taking polar angles from the negative x-axis, until we come back to our original vertex p0.

We could implement Jarvis's march in one conceptual sweep around the convex hull, that is, without separately constructing the right and left chains. Such implementations typically keep track of the angle of the last convex-hull side chosen and require the sequence of angles of hull sides to be strictly increasing (in the range of 0 to 2 [image] radians). The advantage of constructing separate chains is that we need not explicitly compute angles; the techniques of Section 35.1 suffice to compare angles.

If implemented properly, Jarvis's march has a running time of O(nh). For each of the h vertices of CH(Q), we find the vertex with the minimum polar angle. Each comparison between polar angles takes O(1) time, using the techniques of Section 35.1. As Section 10.1 shows, we can compute the minimum of n values in O(n) time if each comparison takes O(1) time. Thus, Jarvis's march takes O(nh) time.

Exercises

35.3-1

Prove that in the procedure GRAHAM-SCAN, points p1 and pm must be vertices of CH(Q).

35.3-2

Consider a model of computation that supports addition, comparison, and multiplication and for which there is a lower bound of [image] (n lg n) to sort n numbers. Prove that [image] (n lg n) is a lower bound for computing, in order, the vertices of the convex hull of a set of n points in such a model.

35.3-3

Given a set of points Q, prove that the pair of points farthest from each other must be vertices of CH(Q).

35.3-4

For a given polygon P and a point q on its boundary, the shadow of q is the set of points r such that the segment [image] is entirely on the boundary or in the interior of P. A polygon P is star-shaped if there exists a point p in the interior of P that is in the shadow of every point on the boundary of P. The set of all such points p is called the kernel of P. (See Figure 35.11.) Given an n-vertex, star-shaped polygon P specified by its vertices in counterclockwise order, show how to compute CH(P) in O(n) time.

35.3-5

In the on-line convex-hull problem, we are given the set Q of n points one point at a time. After receiving each point, we are to compute the convex hull of the points seen so far. Obviously, we could run Graham's scan once for each point, with a total running time of O(n2 l g n). Show how to solve the on-line convex-hull problem in a total of O(n2) time.

Figure 35.11 The definition of a star-shaped polygon, for use in Exercise 35.3-4. (a) A star-shaped polygon. The segment from point p to any point q on the boundary intersects the boundary only at q. (b) A non-star-shaped polygon. The shaded region on the left is the shadow of q, and the shaded region on the right is the shadow of q'. Since these regions are disjoint, the kernel is empty.

35.3-6

Show how to implement the incremental method for computing the convex hull of n points so that it runs in O(n lg n) time.

35.4 Finding the closest pair of points

We now consider the problem of finding the closest pair of points in a set Q of n [image] 2 points. "Closest" refers to the usual euclidean distance: the distance between points p1 = (x1, y1) and p2 = (x2, y2) is [image]. Two points in set Q may be coincident, in which case the distance between them is zero. This problem has applications in, for example, traffic-control systems. A system for controlling air or sea traffic might need to know which are the two closest vehicles in order to detect potential collisions.

A brute-force closest-pair algorithm simply looks at all the [image] pairs of points. In this section, we shall describe a divide-and-conquer algorithm for this problem whose running time is described by the familiar recurrence T(n) = 2T(n/2) + O(n). Thus, this algorithm uses only O(n lg n) time.

The divide-and-conquer algorithm

Each recursive invocation of the algorithm takes as input a subset P [image] Q and arrays X and Y, each of which contains all the points of the input subset P. The points in array X are sorted so that their x-coordinates are monotonically increasing. Similarly, array Y is sorted by monotonically increasing y-coordinate. Note that in order to attain the O(n lg n) time bound, we cannot afford to sort in each recursive call; if we did, the recurrence for the running time would be T(n) = 2T(n/2) + O(n lg n), whose solution is T(n) = O(n 1g2 n). We shall see a little later how to use "presorting" to maintain this sorted property without actually sorting in each recursive call.

A given recursive invocation with inputs P, X, and Y first checks whether |P| [image] 3. If so, the invocation simply performs the brute-force method described above: try all [image] pairs of points and return the closest pair. If |P| > 3, the recursive invocation carries out the divide-and-conquer paradigm as follows.

Divide: It finds a vertical line l that bisects the point set P into two sets PL and PR such that |PL| = [image] |P|/2 [image] , |PR| = [image] |P| /2 [image] , all points in PL are on or to the left of line l, and all points in PR are on or to the right of l. The array X is divided into arrays XL and XR, which contain the points of PL and PR respectively, sorted by monotonically increasing x-coordinate. Similarly, the array Y is divided into arrays YL and YR, which contain the points of PL and PR respectively, sorted by monotonically increasing y-coordinate.

Conquer: Having divided P into PL and PR, it makes two recursive calls, one to find the closest pair of points in PL and the other to find the closest pair of points in PR. The inputs to the first call are the subset PL and arrays XL and YL; the second call receives the inputs PR, XR, and YR. Let the closest-pair distances returned for PL and PR be [image] L and [image] R, respectively, and let [image] = min( [image] L, [image] R).

Combine: The closest pair is either the pair with distance [image] found by oneof the recursive calls, or it is a pair of points with one point in PL and the other in PR. The algorithm determines if there is such a pair whose distance is less than [image]. Observe that if there is a pair of points with distance less than [image], both points of the pair must be within [image] units of line l. Thus, as Figure 35.12(a) shows, they both must reside in the 2 [image] -wide vertical strip centered at line l. To find such a pair, if one exists, the algorithm does the following.

1. It creates an array Y', which is the array Y with all points not in the 2 [image] -wide vertical strip removed. The array Y' is sorted by y-coordinate, just as Y is.

2. For each point p in the array Y', the algorithm tries to find points in Y' that are within [image] units of p. As we shall see shortly, only the 7 points in Y' that follow p need be considered. The algorithm computes the distance from p to each of these 7 points and keeps track of the closest-pair distance [image] ' found over all pairs of points in Y'.

3. If [image] ' < [image], then the vertical strip does indeed contain a closer pair than was found by the recursive calls. This pair and its distance [image] ' are returned. Otherwise, the closest pair and its distance [image] found by the recursive calls are returned.

Figure 35.12 Key concepts in the proof that the closest-pair algorithm needs to check only 7 points following each point in the array Y'. (a) If PL PL and PR PR are less than units apart, they must reside within a x 2 rectangle centered at line l. (b) How 4 points that are pairwise at least units apart can all reside within a x square. On the left are 4 points in PL, and on the right are 4 points in PR There can be 8 points in the x 2 rectangle if the points shown on line l are actually pairs of coincident points with one point in PL and one in PR.

The above description omits some implementation details that are necessary to achieve the O(n 1g n) running time. After proving the correctness of the algorithm, we shall show how to implement the algorithm to achieve the desired time bound.

Correctness

The correctness of this closest-pair algorithm is obvious, except for two aspects. First, by bottoming out the recursion when |P| [image] 3, we ensure that we never try to divide a set of points with only one point. The second aspect is that we need only check the 7 points following each point p in array Y'; we shall now prove this property.

Suppose that at some level of the recursion, the closest pair of points is pL [image] PL and PR [image] PR. Thus, the distance [image] ' between PL and PR is strictly less than [image]. Point pL must be on or to the left of line l and less than [image] units away. Similarly, pR is on or to the right of l and less than [image] units away. Moreover, pL and pR are within [image] units of each other vertically. Thus, as Figure 35.12(a) shows, pL and pR are within a [image] X 2 [image] rectangle centered at line l. (There may be other points within this rectangle as well.)

We next show that at most 8 points of P can reside within this [image] X 2 [image] rectangle. Consider the [image] X [image] square forming the left half of this rectangle. Since all points within PL are at least [image] units apart, at most 4 points can reside within this square; Figure 35.12(b) shows how. Similarly, at most 4 points in PR can reside within the [image] X [image] square forming the right half of the rectangle. Thus, at most 8 points of P can reside within the [image] X 2 [image] rectangle. (Note that since points on line l may be in either PL or PR, there may be up to 4 points on l. This limit is achieved if there are two pairs of coincident points, each pair consisting of one point from PL and one point from PR, one pair is at the intersection of l and the top of the rectangle, and the other pair is where l intersects the bottom of the rectangle.)

Having shown that at most 8 points of P can reside within the rectangle, it is easy to see that we need only check the 7 points following each point in the array Y'. Still assuming that the closest pair is PL and PR, let us assume without loss of generality that pL precedes pR in array Y'. Then, even if pL occurs as early as possible in Y' and pR occurs as late as possible, pR is in one of the 7 positions following pL. Thus, we have shown the correctness of the closest-pair algorithm.

Implementation and running time

As we have noted, our goal is to have the recurrence for the running time be T(n) = 2T(n/2) + O(n), where T(n) is, of course, the running time for a set of n points. The main difficulty is in ensuring that the arrays XL, XR, YL, and YR, which are passed to recursive calls, are sorted by the proper coordinate and also that the array Y' is sorted by y-coordinate. (Note that if the array X that is received by a recursive call is already sorted, then the division of set P into PL and PR is easily accomplished in linear time.)

The key observation is that in each call, we wish to form a sorted subset of a sorted array. For example, a particular invocation is given the subset P and the array Y, sorted by y-coordinate. Having partitioned P into PL and PR, it needs to form the arrays YL and YR, which are sorted by y-coordinate. Moreover, these arrays must be formed in linear time. The method can be viewed as the opposite of the MERGE procedure from merge sort in Section 1.3.1: we are splitting a sorted array into two sorted arrays. The following pseudocode gives the idea.

1  length[YL]    length[YR]    0

2  for i    1 to length[Y]

3        do if Y[i]    PL

4              then length[YL]    length[YL] + 1

5                   Y[length[YL]]    Y[i]

6              else length[YR]    length[YR] + 1

7                   Y[length[YR]]    Y[i]

We simply examine the points in array Y in order. If a point Y[i] is in PL, we append it to the end of array YL; otherwise, we append it to the end of array YR. Similar pseudocode works for forming arrays XL, XR, and Y'.

The only remaining question is how to get the points sorted in the first place. We do this by simply presorting them; that is, we sort them once and for all before the first recursive call. These sorted arrays are passed into the first recursive call, and from there they are whittled down through the recursive calls as necessary. The presorting adds an additional O(n lg n) to the running time, but now each step of the recursion takes linear time exclusive of the recursive calls. Thus, if we let T(n) be the running time of each recursive step and T'(n) be the running time of the entire algorithm, we get T'(n) = T(n) + O(n lg n) and

Thus, T(n) = O(n lg n) and T'(n) = O(n lg n).

Exercises

35.4-1

Professor Smothers comes up with a scheme that allows the closest-pair algorithm to check only 5 points following each point in array Y'. The idea is always to place points on line l into set PL. Then, there cannot be pairs of coincident points on line l with one point in PL and one in PR. Thus, at most 6 points can reside in the [image] x 2 [image] rectangle. What is the flaw in the professor's scheme?

35.4-2

Without increasing the asymptotic running time of the algorithm, show how to ensure that the set of points passed to the very first recursive call contains no coincident points. Prove that it then suffices to check the points in the 6 (not 7) array positions following each point in the array Y'. Why doesn't it suffice to check only the 5 array positions following each point?

35.4-3

The distance between two points can be defined in ways other than euclidean. In the plane, the Lm-distance between points p1 and p2 is given by ((x1 - x2]m + (y1 - y2)m)1/m. Euclidean distance, therefore, is L2-distance. Modify the closest-pair algorithm to use the L1-distance, which is also known as the Manhattan distance.

35.4-4

Given two points p1 and p2 in the plane, the L [image] -distance between them is max(|x1 - x2|, |y1 - y2|). Modify the closest-pair algorithm to use the L [image] -distance.

Problems

35-1 Convex layers

Given a set Q of points in the plane, we define the convex layers of Q inductively. The first convex layer of Q consists of those points in Q that are vertices of CH(Q). For i > 1, define Qi to consist of the points of Q with all points in convex layers 1, 2, . . . , i - 1 removed. Then, the ith convex layer of Q is CH(Qi) if [image] and is undefined otherwise.

a. Give an O(n2)-time algorithm to find the convex layers of a set on n points.

b. Prove that [image] (n lg n) time is required to compute the convex layers of a set of n points on any model of computation that requires [image] (n lg n) time to sort n real numbers.

35-2 Maximal layers

Let Q be a set of n points in the plane. We say that point (x, y) dominates point (x', y') if x [image] x' and y [image] y'. A point in Q that is dominated by no other points in Q is said to be maximal. Note that Q may contain many maximal points, which can be organized into maximal layers as follows. The first maximal layer L1 is the set of maximal points of Q. For i > 1, the ith maximal layer Li is the set of maximal points in [image].

Suppose that Q has k nonempty maximal layers, and let yi be the y-coordinate of the leftmost point in Li for i = 1, 2, . . . ,k. For now, assume that no two points in Q have the same x - or y-coordinate.

a. Show that y1 > y2 > . . . > yk.

Consider a point (x, y) that is to the left of any point in Q and for which y is distinct from the y-coordinate of any point in Q. Let Q' = Q [image] {(x, y)}.

b. Let j be the minimum index such that yj < y, unless y < yk, in which case we let j = k + 1. Show that the maximal layers of Q' are as follows.

If j [image] k, then the maximal layers of Q' are the same as the maximal layers of Q, except that Lj also includes (x, y) as its new leftmost point.

If j = k + 1, then the first k maximal layers of Q' are the same as for Q, but in addition, Q' has a nonempty (k + 1)st maximal layer: Lk+1 = {(x, y)}.

c. Describe an O(n lg n)-time algorithm to compute the maximal layers of a set Q of n points. (Hint: Move a sweep line from right to left.)

d. Do any difficulties arise if we now allow input points to have the same x- or y-coordinate? Suggest a way to resolve such problems.

35-3 Ghostbusters and ghosts

A group of n Ghostbusters is battling n ghosts. Each Ghostbuster is armed with a proton pack, which shoots a stream at a ghost, e radicating it. A stream goes in a straight line and terminates when it hits the ghost. The Ghostbusters decide upon the following strategy. They will pair off with the ghosts, forming n Ghostbuster-ghost pairs, and then simultaneously each Ghostbuster will shoot a stream at his or her chosen ghost. As we all know, it is very dangerous to let streams cross, and so the Ghostbusters must choose pairings for which no streams will cross.

Assume that the position of each Ghostbuster and each ghost is a fixed point in the plane and that no three positions are collinear.

a. Argue that there exists a line passing through one Ghostbuster and one ghost such the number of Ghostbusters on one side of the line equals the number of ghosts on the same side. Describe how to find such a line in O(n lg n) time.

b. Give an O(n2 lg n)-time algorithm to pair Ghostbusters with ghosts in such a way that no streams cross.

35-4 Sparse-hulled distributions

Consider the problem of computing the convex hull of a set of points in the plane that have been drawn according to some known random distribution. Sometimes, the convex hull of n points drawn from such a distribution has O(n1- [image]) expected size for some constant [image] > 0. We call such a distribution sparse-hulled. Sparse-hulled distributions include the following:

Points drawn uniformly from a unit-radius disk. The convex hull has [image] (n1/3) expected size.

Points drawn uniformly from the interior of a convex polygon with k sides, for any constant k. The convex hull has [image] (1g n) expected size.

Points drawn according to a two-dimensional normal distribution. The convex hull has [image] expected size.

a. Given two convex polygons with n1 and n2 vertices respectively, show how to compute the convex hull of all n1 + n2 points in O(n1 + n2) time. (The polygons may overlap.)

b. Show that the convex hull of a set of n points drawn independently according to a sparse-hulled distribution can be computed in O(n) expected time. (Hint: Recursively find the convex hulls of the first n/2 points and the second n/2 points, and then combine the results.)

Chapter notes

This chapter barely scratches the surface of computational-geometry algorithms and techniques. Books on computational geometry include those by Preparata and Shamos [160] and Edelsbrunner [60].

Although geometry has been studied since antiquity, the development of algorithms for geometric problems is relatively new. Preparata and Shamos note that the earliest notion of the complexity of a problem was given by E. Lemoine in 1902. He was studying euclidean constructions--those using a ruler and a straightedge--and devised a set of five primitives: placing one leg of the compass on a given point, placing one leg of the compass on a given line, drawing a circle, passing the ruler's edge through a given point, and drawing a line. Lemoine was interested in the number of primitives needed to effect a given construction; he called this amount the "simplicity" of the construction.

The algorithm of Section 35.2, which determines whether any segments intersect, is due to Shamos and Hoey [176].

The original version of Graham's scan is given by Graham [91]. The package-wrapping algorithm is due to Jarvis [112]. Using a decision-tree model of computation, Yao [205] proved a lower bound of [image] (n lg n) for the running time of any convex-hull algorithm. When the number of vertices h of the convex hull is taken into account, the prune-and-search algorithm of Kirkpatrick and Seidel [120], which takes O(n lg h) time, is asymptotically optimal.

The O(n lg n)-time divide-and-conquer algorithm for finding the closest pair of points is by Shamos and appears in Preparata and Shamos [160]. Preparata and Shamos also show that the algorithm is asymptotically optimal in a decision-tree model.

Go to Chapter 36 Back to Table of Contents


Intro to Algorithms: CHAPTER 36: NPCOMPLETENESS

 Return to Table of Contents

CHAPTER 36: NP-COMPLETENESS

All of the algorithms we have studied thus far have been polynomial-time algorithms: on inputs of size n, their worst-case running time is O(nk) for some constant k. It is natural to wonder whether all problems can be solved in polynomial time. The answer is no. For example, there are problems, such as Turing's famous "Halting Problem," that cannot be solved by any computer, no matter how much time is provided. There are also problems that can be solved, but not in time O(nk) for any constant k. Generally, we think of problems that are solvable by polynomial-time algorithms as being tractable, and problems that require superpolynomial time as being intractable.

The subject of this chapter, however, is an interesting class of problems, called the "NP-complete" problems, whose status is unknown. No polynomial-time algorithm has yet been discovered for an NP-complete problem, nor has anyone yet been able to prove a superpolynomial-time lower bound for any of them. This so-called P [image] NP question has been one of the deepest, most perplexing open research problems in theoretical computer science since it was posed in 1971.

Most theoretical computer scientists believe that the NP-complete problems are intractable. The reason is that if any single NP-complete problem can be solved in polynomial time, then every NP-complete problem has a polynomial-time algorithm. Given the wide range of NP-complete problems that have been studied to date, without any progress toward a polynomial-time solution, it would be truly astounding if all of them could be solved in polynomial time.

To become a good algorithm designer, you must understand the rudiments of the theory of NP-completeness. If you can establish a problem as NP-complete, you provide good evidence for its intractability. As an engineer, you would then do better spending your time developing an approximation algorithm (see Chapter 37) rather than searching for a fast algorithm that solves the problem exactly. Moreover, many natural and interesting problems that on the surface seem no harder than sorting, graph searching, or network flow are in fact NP-complete. Thus, it is important to become familiar with this remarkable class of problems.

This chapter studies the aspects of NP-completeness that bear most directly on the analysis of algorithms. In Section 36.1, we formalize our notion of "problem" and define the complexity class P of polynomial-time solvable decision problems. We also see how these notions fit into the framework of formal-language theory. Section 36.2 defines the class NP of decision problems whose solutions can be verified in polynomial time. It also formally poses the P [image] NP question.

Section 36.3 shows how relationships between problems can be studied via polynomial-time "reductions." It defines NP-completeness and sketches a proof that one problem, called "circuit satisfiability," is NP-complete. Having found one NP-complete problem, we show in Section 36.4 how other problems can be proven to be NP-complete much more simply by the methodology of reductions. The methodology is illustrated by showing that two formula-satisfiability problems are NP-complete. A variety of other problems are shown to be NP-complete in Section 36.5.

36.1 Polynomial time

We begin our study of NP-completeness by formalizing our notion of polynomial-time solvable problems. These problems are generally regarded as tractable. The reason why is a philosophical, not a mathematical, issue. We can offer three supporting arguments.

First, although it is reasonable to regard a problem that requires time [image] (n100) as intractable, there are very few practical problems that require time on the order of such a high-degree polynomial. The polynomial-time computable problems encountered in practice typically require much less time.

Second, for many reasonable models of computation, a problem that can be solved in polynomial time in one model can be solved in polynomial time in another. For example, the class of problems solvable in polynomial time by the serial random-access machine used throughout most of this book is the same as the class of problems solvable in polynomial time on abstract Turing machines.1 It is also the same as the class of problems solvable in polynomial time on a parallel computer, even if the number of processors grows polynomially with the input size.

1See Hopcroft and Ullman [104] or Lewis and Papadimitriou [139] for a thorough treatment of the Turing-machine model. of calls to polynomial-time subroutines, the running time of the composite algorithm is polynomial.

Third, the class of polynomial-time solvable problems has nice closure properties, since polynomials are closed under addition, multiplication, and composition. For example, if the output of one polynomial-time algorithm is fed into the input of another, the composite algorithm is polynomial. If an otherwise polynomial-time algorithm makes a constant number of calls to polynomial-time subroutines, the running time of the composite algorithm is polynomial.

Abstract problems

To understand the class of polynomial-time solvable problems, we must first have a formal notion of what a "problem" is. We define an abstract problem Q to be a binary relation on a set I of problem instances and a set S of problem solutions. For example, consider the problem SHORTEST-PATH of finding a shortest path between two given vertices in an unweighted, undirected graph G = (V, E). An instance for SHORTEST-PATH is a triple consisting of a graph and two vertices. A solution is a sequence of vertices in the graph, with perhaps the empty sequence denoting that no path exists. The problem SHORTEST-PATH itself is the relation that associates each instance of a graph and two vertices with a shortest path in the graph that connects the two vertices. Since shortest paths are not necessarily unique, a given problem instance may have more than one solution.

This formulation of an abstract problem is more general than is required for our purposes. For simplicity, the theory of NP-completeness restricts attention to decision problems: those having a yes/no solution. In this case, we can view an abstract decision problem as a function that maps the instance set I to the solution set {0, 1}. For example, a decision problem PATH related to the shortest-path problem is, "Given a graph G = (V, E), two vertices u, v [image] V, and a nonnegative integer k, does a path exist in G between u and v whose length is at most k?" If i = <G, u, v, k> is an instance of this shortest-path problem, then PATH(i) = 1 (yes) if a shortest path from u to v has length at most k, and PATH(i) = 0 (no) otherwise.

Many abstract problems are not decision problems, but rather optimization problems, in which some value must be minimized or maximized. In order to apply the theory of NP-completeness to optimization problems, we must recast them as decision problems. Typically, an optimization problem can be recast by imposing a bound on the value to be optimized. As an example, in recasting the shortest-path problem as the decision problem PATH, we added a bound k to the problem instance.

Although the theory of NP-completeness compels us to recast optimization problems as decision problems, this requirement does not diminish the impact of the theory. In general, if we can solve an optimization problem quickly, we can also solve its related decision problem quickly. We simply compare the value obtained from the solution of the optimization problem with the bound provided as input to the decision problem. If an optimization problem is easy, therefore, its related decision problem is easy as well. Stated in a way that has more relevance to NP-completeness, if we can provide evidence that a decision problem is hard, we also provide evidence that its related optimization problem is hard. Thus, even though it restricts attention to decision problems, the theory of NP-completeness applies much more widely.

Encodings

If a computer program is to solve an abstract problem, problem instances must must be represented in a way that the program understands. An encoding of a set S of abstract objects is a mapping e from S to the set of binary strings.2 For example, we are all familiar with encoding the natural numbers N = {0, 1, 2, 3, 4, . . .} as the strings {0, 1, 10, 11, 100, . . .}. Using this encoding, e(l7) = 10001. Anyone who has looked at computer representations of keyboard characters is familiar with either the ASCII or EBCDIC codes. In the ASCII code, e(A) = 1000001. Even a compound object can be encoded as a binary string by combining the representations of its constituent parts. Polygons, graphs, functions, ordered pairs, programs--all can be encoded as binary strings.

2 The codomain of e need not be binary strings; any set of strings over a finite alphabet having at least 2 symbols will do.

Thus, a computer algorithm that "solves" some abstract decision problem actually takes an encoding of a problem instance as input. We call a problem whose instance set is the set of binary strings a concrete problem. We say that an algorithm solves a concrete problem in time O(T(n)) if, when it is provided a problem instance i of length n = |i|, the algorithm can produce the solution in at most O(T(n)) time. A concrete problem is polynomial-time solvable, therefore, if there exists an algorithm to solve it in time O(nk) for some constant k.

We can now formally define the complexity class P as the set of concrete decision problems that are solvable in polynomial time.

We can use encodings to map abstract problems to concrete problems. Given an abstract decision problem Q mapping an instance set I to {0, 1}, an encoding e : I [image] {0, 1}* can be used to induce a related concrete decision problem, which we denote by e(Q). If the solution to an abstract-problem instance i [image] I is Q(i) [image] {0, 1}, then the solution to the concrete-problem instance e(i) [image] {0, 1}* is also Q(i). As a technicality, there may be some binary strings that represent no meaningful abstract-problem instance. For convenience, we shall assume that any such string is mapped arbitrarily to 0. Thus, the concrete problem produces the same solutions as the abstract problem on binary-string instances that represent the encodings of abstract-problem instances.

We would like to extend the definition of polynomial-time solvability from concrete problems to abstract problems using encodings as the bridge, but we would like the definition to be independent of any particular encoding. That is, the efficiency of solving a problem should not depend on how the problem is encoded. Unfortunately, it depends quite heavily. For example, suppose that an integer k is to be provided as the sole input to an algorithm, and suppose that the running time of the algorithm is [image] (k). If the integer k is provided in unary--a string of k 1's--then the running time of the algorithm is O(n) on length-n inputs, which is polynomial time. If we use the more natural binary representation of the integer k, however, then the input length is n = [image] lg k [image] . In this case, the running time of the algorithm is [image] (k) = [image] (2n), which is exponential in the size of the input. Thus, depending on the encoding, the algorithm runs in either polynomial or superpolynomial time.

The encoding of an abstract problem is therefore quite important to our understanding of polynomial time. We cannot really talk about solving an abstract problem without first specifying an encoding. Nevertheless, in practice, if we rule out "expensive" encodings such as unary ones, the actual encoding of a problem makes little difference to whether the problem can be solved in polynomial time. For example, representing integers in base 3 instead of binary has no effect on whether a problem is solvable in polynomial time, since an integer represented in base 3 can be converted to an integer represented in base 2 in polynomial time.

We say that a function f : {0, 1}* [image] {0, 1}* is polynomial-time computable if there exists a polynomial-time algorithm A that, given any input x [image] {0, 1}*, produces as output f(x). For some set I of problem instances, we say that two encodings e and e are polynomially related if there exist two polynomial-time computable functions f12 and f21 such that for any i [image] I, we have f12(e1(i)) = e2(i) and f21(e2(i)) = e1(i). That is, the encoding e2(i) can be computed from the encoding e1(i) by a polynomial-time algorithm, and vice versa. If two encodings e1 and e2 of an abstract problem are polynomially related, which we use makes no difference to whether the problem is polynomial-time solvable or not, as the following lemma shows.

Lemma 36.1

Let Q be an abstract decision problem on an instance set I, and let e1 and e2 be polynomially related encodings on I. Then, e1(Q) [image] P if and only if e2(Q) [image] P.

Proof We need only prove the forward direction, since the backward direction is symmetric. Suppose, therefore, that e1(Q) can be solved in time O(nk) for some constant k. Further, suppose that for any problem instance i, the encoding e1(i) can be computed from the encoding e2(i) in time O(nc) for some constant c, where n = |e1(i)|. To solve problem e2(Q), on input e2(i), we first compute e1(i) and then run the algorithm for e1(Q) on e1(i). How long does this take? The conversion of encodings takes time O(nc), and therefore |e1(i)| = O(nc), since the output of a serial computer cannot be longer than its running time. Solving the problem on e1 (i) takes time O(|e1(i)|k) = O(nck), which is polynomial since both c and k are constants.

Thus, whether an abstract problem has its instances encoded in binary or base 3 does not affect its "complexity," that is, whether it is polynomial-time solvable or not, but if instances are encoded in unary, its complexity may change. In order to be able to converse in an encoding-independent fashion, we shall generally assume that problem instances are encoded in any reasonable, concise fashion, unless we specifically say otherwise. To be precise, we shall assume that the encoding of an integer is polynomially related to its binary representation, and that the encoding of a finite set is polynomially related to its encoding as a list of its elements, enclosed in braces and separated by commas. (ASCII is one such encoding scheme.) With such a "standard" encoding in hand, we can derive reasonable encodings of other mathematical objects, such as tuples, graphs, and formulas. To denote the standard encoding of an object, we shall enclose the object in angle braces. Thus, [image] G [image] denotes the standard encoding of a graph G.

As long as we implicitly use an encoding that is polynomially related to this standard encoding, we can talk directly about abstract problems without reference to any particular encoding, knowing that the choice of encoding has no effect on whether the abstract problem is polynomial-time solvable. Henceforth, we shall generally assume that all problem instances are binary strings encoded using the standard encoding, unless we explicitly specify the contrary. We shall also typically neglect the distinction between abstract and concrete problems. The reader should watch out for problems that arise in practice, however, in which a standard encoding is not obvious and the encoding does make a difference.

A formal-language framework

One of the convenient aspects of focusing on decision problems is that they make it easy to use the machinery of formal-language theory. It is worthwhile at this point to review some definitions from that theory. An alphabet [image] is a finite set of symbols. A language L over [image] is any set of strings made up of symbols from [image]. For example, if [image] = {0, 1}, the set L = {10, 11, 101, 111, 1011, 1101, 10001, . . .} is the language of binary representations of prime numbers. We denote the empty string by [image], and the empty language by [image]. The language of all strings over [image] is denoted [image] *. For example, if [image] = {0, 1}, then [image] * = { [image], 0, 1, 00, 01, 10, 11, 000, . . .} is the set of all binary strings. Every language L over [image] is a subset of [image] *.

There are a variety of operations on languages. Set-theoretic operations, such as union and intersection, follow directly from the set-theoretic definitions. We define the complement of L by [image]. The concatenation of two languages L1 and L2 is the language

L = {x1x2 : x1    L1 and x2    L2} .

The closure or Kleene star of a language L is the language

L* = {  }    L    L2   L3     . .  ,

where Lk is the language obtained by concatenating L to itself k times.

From the point of view of language theory, the set of instances for any decision problem Q is simply the set [image] *, where [image] = {0, 1}. Since Q is entirely characterized by those problem instances that produce a 1 (yes) answer, we can view Q as a language L over [image] = {0, 1}, where

L = {x      * : Q(x) = 1} .

For example, the decision problem PATH has the corresponding language

PATH = {  G,u,v,k   : G = (V,E) is an undirected graph,

u,v    V,

k    0 is an integer, and

there exists a path from u to v in G

whose length is at most k}.

(Where convenient, we shall sometimes use the same name--PATH in this case--to refer to both a decision problem and its corresponding language.)

The formal-language framework allows us to express the relation between decision problems and algorithms that solve them concisely. We say that an algorithm A accepts a string x [image] {0, 1}* if, given input x, the algorithm outputs A(x) = 1. The language accepted by an algorithm A is the set L = {x [image] {0, 1}* : A(x) = 1}, that is, the set of strings that the algorithm accepts. An algorithm A rejects a string x if A(x) = 0.

Even if language L is accepted by an algorithm A, the algorithm will not necessarily reject a string x [image] L provided as input to it. For example, the algorithm may loop forever. A language L is decided by an algorithm A if every binary string is either accepted or rejected by the algorithm. A language L is accepted in polynomial time by an algorithm A if for any length-n string x [image] L, the algorithm accepts x in time O(nk) for some constant k. A language L is decided in polynomial time by an algorithm A if for any length-n string x [image] {0, 1 }*, the algorithm decides x in time O(nk) for some constant k. Thus, to accept a language, an algorithm need only worry about strings in L, but to decide a language, it must accept or reject every string in {0, 1}*.

As an example, the language PATH can be accepted in polynomial time. One polynomial-time accepting algorithm computes the shortest path from u to v in G, using breadth-first search, and then compares the distance obtained with k. If the distance is at most k, the algorithm outputs 1 and halts. Otherwise, the algorithm runs forever. This algorithm does not decide PATH, however, since it does not explicitly output 0 for instances in which the shortest path has length greater than k. A decision algorithm for PATH must explicitly reject binary strings that do not belong to PATH. For a decision problem such as PATH, such a decision algorithm is easy to design. For other problems, such as Turing's Halting Problem, there exists an accepting algorithm, but no decision algorithm exists.

We can informally define a complexity class as a set of languages, membership in which is determined by a complexity measure, such as running time, on an algorithm that determines whether a given string x belongs to language L. The actual definition of a complexity class is somewhat more technical--the interested reader is referred to the seminal paper by Hartmanis and Stearns [95].

Using this language-theoretic framework, wee can provide an alternative definition of the complexity class P:

P = {L    {0,1}* : there exists an algorithm A

that decides L in polynomial time} .

In fact, P is also the class of languages that can be accepted in polynomial time.

Theorem 36.2

P = {L : L is accepted by a polynomial-time algorithm} .

Proof Since the class of languages decided by polynomial-time algorithms is a subset of the class of languages accepted by polynomial-time algorithms, we need only show that if L is accepted by a polynomial-time algorithm, it is decided by a polynomial-time algorithm. Let L be the language accepted by some polynomial-time algorithm A. We shall use a classic "simulation" argument to construct another polynomial-time algorithm A' that decides L. Because A accepts L in time O(nk) for some constant k, there also exists a constant c such that A accepts L in at most T = cnk steps. For any input string x, the algorithm A' simulates the action of A for time T. At the end of time T, algorithm A' inspects the behavior of A. If A has accepted x, then A' accepts x by outputting a 1. If A has not accepted x, then A' rejects x by outputting a 0. The overhead of A' simulating A does not increase the running time by more than a polynomial factor, and thus A' is a polynomial-time algorithm that decides L.

Note that the proof of Theorem 36.2 is nonconstructive. For a given language L [image] P, we may not actually know a bound on the running time for the algorithm A that accepts L. Nevertheless, we know that such a bound exists, and therefore, that an algorithm A' exists that can check the bound, even though we may not be able to find the algorithm A' easily.

Exercises

36.1-1

Define the optimization problem LONGEST-PATH-LENGTH as the relation that associates each instance of a undirected graph and two vertices with the length of the longest simple path between the two vertices. Define the decision problem LONGEST-PATH = { [image] G, u, v, k [image] : G = (V, E) is an undirected graph, u, v [image] V, k [image] 0 is an integer, and there exists a simple path from u to v in G whose length is at least k}. Show that the optimization problem LONGEST-PATH-LENGTH can be solved in polynomial time if and only if LONGEST-PATH [image] P.

36.1-2

Give a formal definition for the problem of finding the longest simple cycle in an undirected graph. Give a related decision problem. Give the language corresponding to the decision problem.

36.1-3

Give a formal encoding of directed graphs as binary strings using an adjacency-matrix representation. Do the same using an adjacency-list representation. Argue that the two representations are polynomially related.

36.1-4

Is the dynamic-programming algorithm for the 0-1 knapsack problem that is asked for in Exercise 17.2-2 a polynomial-time algorithm? Explain your answer.

36.1-5

Suppose that a language L can accept any string x [image] L in polynomial time, but that the algorithm that does this runs in superpolynomial time if x [image] L. Argue that L can be decided in polynomial time.

36.1-6

Show that an algorithm that makes at most a constant number of calls to polynomial-time subroutines runs in polynomial time, but that a polynomial number of calls to polynomial-time subroutines may result in an exponential-time algorithm.

36.1-7

Show that the class P, viewed as a set of languages, is closed under union, intersection, concatenation, complement, and Kleene star. That is, if Ll, L2 [image] P, then Ll [image] L2 [image] P, etc.

36.2 Polynomial-time verification

We now look at algorithms that "verify" membership in languages. For example, suppose that for a given instance [image] G, u, v, k [image] of the decision problem PATH, we are also given a path p from u to v. We can easily check whether the length of p is at most k, and if so, we can view p as a "certificate" that the instance indeed belongs to PATH. For the decision problem PATH, this certificate doesn't seem to buy us much. After all, PATH belongs to P--in fact, PATH can be solved in linear time--and so verifying membership from a given certificate takes as long as solving the problem from scratch. We shall now examine a problem for which we know of no polynomial-time decision algorithm yet, given a certificate, verification is easy.

Figure 36.1 (a) A graph representing the vertices, edges, and faces of a dodecahedron, with a hamiltonian cycle shown by shaded edges. (b) A bipartite graph with an odd number of vertices. Any such graph is nonhamiltonian.

Hamiltonian cycles

The problem of finding a hamiltonian cycle in an undirected graph has been studied for over a hundred years. Formally, a hamiltonian cycle of an undirected graph G = (V, E) is a simple cycle that contains each vertex in V. A graph that contains a hamiltonian cycle is said to be hamiltonian; otherwise, it is nonhamiltonian. Bondy and Murty [31] cite a letter by W. R. Hamilton describing a mathematical game on the dodecahedron (Figure 36.1(a)) in which one player sticks five pins in any five consecutive vertices and the other player must complete the path to form a cycle containing all the vertices. The dodecahedron is hamiltonian, and Figure 36.1 (a) shows one hamiltonian cycle. Not all graphs are hamiltonian, however. For example, Figure 36.1 (b) shows a bipartite graph with an odd number of vertices. (Exercise 36.2-2 asks you to show that all such graphs are nonhamiltonian.)

We can define the hamiltonian-cycle problem, "Does a graph G have a hamiltonian cycle?" as a formal language:

HAM-CYCLE = {  G   : G is a hamiltonian graph} .

How might an algorithm decide the language HAM-CYCLE? Given a problem instance [image] G [image], one possible decision algorithm lists all permutations of the vertices of G and then checks each permutation to see if it is a hamiltonian path. What is the running time of this algorithm? If we use the "reasonable" encoding of a graph as its adjacency matrix, the number m of vertices in the graph is [image] , where n = | [image] G [image] | is the length of the encoding of G. There are m! possible permutations of the vertices, and therefore the running time is [image], which is not O(nk) for any constant k. Thus, this naive algorithm does not run in polynomial time, and in fact, the hamiltonian-cycle problem is NP-complete, as we shall prove in Section 36.5.

Verification algorithms

Consider a slightly easier problem, however. Suppose that a friend tells you that a given graph G is hamiltonian, and then offers to prove it by giving you the vertices in order along the hamiltonian cycle. It would certainly be easy enough to verify the proof: simply verify that the provided cycle is hamiltonian by checking whether it is a permutation of the vertices of V and whether each of the consecutive edges along the cycle actually exists in the graph. This verification algorithm can certainly be implemented to run in O(n2) time, where n is the length of the encoding of G . Thus, a proof that a hamiltonian cycle exists in a graph can be verified in polynomial time.

We define a verification algorithm as being a two-argument algorithm A, where one argument is an ordinary input string x and the other is a binary string y called a certificate. A two-argument algorithm A verifies an input string x if there exists a certificate y such that A(x, y) = 1. The language verified by a verification algorithm A is

L = {x    {0, 1}* : there exists y   {0, 1}* such that A(x, y) = 1} .

Intuitively, an algorithm A verifies a language L if for any string x [image] L, there is a certificate y that A can use to prove that x [image] L. Moreover, for any string x [image] L there must be no certificate proving that x [image] L. For example, in the hamiltonian-cycle problem, the certificate is the list of vertices in the hamiltonian cycle. If a graph is hamiltonian, the hamiltonian cycle itself offers enough information to verify this fact. Conversely, if a graph is not hamiltonian, there is no list of vertices that can fool the verification algorithm into believing that the graph is hamiltonian, since the verification algorithm carefully checks the proposed "cycle" to be sure.

The complexity class NP

The complexity class NP is the class of languages that can be verified by a polynomial-time algorithm.3 More precisely, a language L belongs to NP if and only if there exists a two-input polynomial-time algorithm A and constant c such that

L = {x    {0,1}* : there exists a certificate y with |y| = 0(|x|c)

such that A(x,y) = 1} .

We say that algorithm A verifies language L in polynomial time.

3The name "NP" stands for "nondeterministic polynomial time." The class NP was originally studied in the context of nondeterminism, but this book uses the somewhat simpler yet equivalent notion of verification. Hopcroft and Ullman [104] give a good presentation of NP-completeness in terms of nondeterministic models of computation.

From our earlier discussion on the hamiltonian-cycle problem, it follows that HAM-CYCLE [image] NP. (It is always nice to know that an important set is nonempty.) Moreover, if L [image] P, then L [image] NP, since if there is a polynomial-time algorithm to decide L, the algorithm can be easily converted to a two-argument verification algorithm that simply ignores any certificate and accepts exactly those input strings it determines to be in L. Thus, P [image] NP.

It is unknown whether P = NP, but most researchers believe that P and NP are not the same class. Intuitively, the class P consists of problems that can be solved quickly. The class NP consists of problems for which a solution can be verified quickly. You may have learned from experience that it is often more difficult to solve a problem from scratch than to verify a clearly presented solution, especially when working under time constraints. Theoretical computer scientists generally believe that this analogy extends to the classes P and NP, and thus that NP includes languages that are not in P.

There is more compelling evidence that P [image] NP--the existence of "NP-complete" languages. We shall study this class in Section 36.3.

Many other fundamental questions beyond the P [image] NP question remain unresolved. Despite much work by many researchers, no one even knows if the class NP is closed under complement. That is, does L [image] NP imply [image]? We can define the complexity class co-NP as the set of languages L such that [image]. The question of whether NP is closed under complement can be rephrased as whether NP = co-NP. Since P is closed under complement (Exercise 36.1-7), it follows that P [image] NP [image] co-NP. Once again, however, it is not known whether P = NP [image] co-NP or whether there is some language in NP [image] co-NP - P. Figure 36.2 shows the four possible scenarios.

Thus, our understanding of the precise relationship between P and NP is woefully incomplete. Nevertheless, by exploring the theory of NP-completeness, we shall find that our disadvantage in proving problems to be intractable is, from a practical point of view, not nearly so great as we might suppose.

Figure 36.2 Four possibilities for relationships among complexity classes. In each diagram, one region enclosing another indicates a proper-subset relation. (a) P = NP = co-NP. Most researchers regard this possibility as the most unlikely. (b) If NP is closed under complement, then NP = co-NP, but it need not be the case that P = NP. (c) P = NP co-NP, but NP is not closed under complement. (d) NP co-NP and P NP co-NP. Most researchers regard this possibility as the most likely.

Exercises

36.2-1

Consider the language GRAPH-ISOMORPHISM = { [image] G1, G2 [image] : G1 and G2 are isomorphic graphs}. Prove that GRAPH-ISOMORPHISM [image] NP by describing a polynomial-time algorithm to verify the language.

36.2-2

Prove that if G is an undirected bipartite graph with an odd number of vertices, then G is nonhamiltonian.

36.2-3

Show that if HAM-CYCLE [image] P, then the problem of listing the vertices of a hamiltonian cycle, in order, is polynomial-time solvable.

36.2-4

Prove that the class NP of languages is closed under union, intersection, concatenation, and Kleene star. Discuss the closure of NP under complement.

36.2-5

Show that any language in NP can be decided by an algorithm running in time 2O(nk) for some constant k.

36.2-6

A hamiltonian path in a graph is a simple path that visits every vertex exactly once. Show that the language HAM-PATH = { [image] G, u, v [image] : there is a hamiltonian path from u to v in graph G} belongs to NP.

36.2-7

Show that the hamiltonian-path problem can be solved in polynomial time on directed acyclic graphs. Give an efficient algorithm for the problem.

36.2-8

Let [image] be a boolean formula constructed from the boolean input variables x1, x2, . . . , xk, negations ( [image]), AND's ( [image]), OR's ( [image]), and parentheses. The formula [image] is a tautology if it evaluates to 1 for every assignment of 1 and 0 to the input variables. Define TAUTOLOGY as the language of boolean formulas that are tautologies. Show that TAUTOLOGY [image] co-NP.

36.2-9

Prove that P [image] co-NP.

36.2-10

Prove that if NP [image] co-NP, then P [image] NP.

36.2-11

Let G be a connected, undirected graph with at least 3 vertices, and let G3 be the graph obtained by connecting all pairs of vertices that are connected by a path in G of length at most 3. Prove that G3 is hamiltonian. (Hint: Construct a spanning tree for G, and use an inductive argument.)

36.3 NP-completeness and reducibility

Perhaps the most compelling reason why theoretical computer scientists believe that P [image] NP is the existence of the class of "NP-complete" problems. This class has the surprising property that if any one NP-complete problem can be solved in polynomial time, then every problem in NP has a polynomial-time solution, that is, P = NP. Despite years of study, though, no polynomial-time algorithm has ever been discovered for any NP-complete problem.

The language HAM-CYCLE is one NP-complete problem. If we could decide HAM-CYCLE in polynomial time, then we could solve every problem in NP in polynomial time. In fact, if NP - P should turn out to be nonempty, we could say with certainty that HAM-CYCLE [image] NP - P.

Figure 36.3 An illustration of a polynomial-time reduction from a language L1 to a language L2 via a reduction function f. For any input x {0, 1}*, the question of whether x L1 has the same answer as the question of whether f(x) L2.

The NP-complete languages are, in a sense, the "hardest" languages in NP. In this section, we shall show how to compare the relative "hardness" of languages using a precise notion called "polynomial-time reducibility." First, we formally define the NP-complete languages, and then we sketch a proof that one such language, called CIRCUIT-SAT, is NP-complete. In Section 36.5, shall use the notion of reducibility to show that many other problems are NP-complete.

Reducibility

Intuitively, a problem Q can be reduced to another problem Q' if any instance of Q can be "easily rephrased" as an instance of Q', the solution to which provides a solution to the instance of Q. For example, the problem of solving linear equations in an indeterminate x reduces to the problem of solving quadratic equations. Given an instance ax + b = 0, we transform it to 0x2 + ax + b = 0, whose solution provides a solution to ax + b = 0. Thus, if a problem Q reduces to another problem Q', then Q is, in a sense, "no harder to solve" than Q'.

Returning to our formal-language framework for decision problems, we say that a language L1 is polynomial-time reducible to a language L2, written L1 [image] P L2, if there exists a polynomial-time computable function f : {0, 1}* [image] {0, 1}* such that for all x [image] {0, 1}*,

x    L1 if and only if f(x)    L2 .

(36.1)

We call the function f the reduction function, and a polynomial-time algorithm F that computes f is called a reduction algorithm.

Figure 36.3 illustrates the idea of a polynomial-time reduction from a language L1 to another language L2. Each language is a subset of {0, 1}*. The reduction function f provides a polynomial-time mapping such that if x [image] L1, then f(x) [image] L2. Moreover, if x [image] L1, then f(x) [image] L2. Thus, the reduction function maps any instance x of the decision problem represented by the language L1 to an instance f(x) of the problem represented by L2. Providing an answer to whether f(x) [image] L2 directly provides the answer to whether x [image] L1.

Figure 36.4 The proof of Lemma 36.3. The algorithm F is a reduction algorithm that computes the reduction function f from L1 to L2 in polynomial time, and A2 is a polynomial-time algorithm that decides L2. Illustrated is an algorithm A1 that decides whether x L1 by using F to transform any input x into f(x) and then using A2 to decide whether f(x) L2.

Polynomial-time reductions give us a powerful tool for proving that various languages belong to P.

Lemma 36.3

If L1, L2 [image] {0, 1}* are languages such that L1 [image] L2, then L2 [image] P implies L1 [image] P.

Proof Let A2 be a polynomial-time algorithm that decides L2, and let F be a polynomial-time reduction algorithm that computes the reduction function f. We shall construct a polynomial-time algorithm A1 that decides L1.

Figure 36.4 illustrates the construction of A1. For a given input x [image] {0, 1}*, the algorithm A1 uses F to transform x into f(x), and then it uses A2 to test whether f(x) [image] L2. The output of A2 is the value provided as the output from A1.

The correctness of A1 follows from condition (36.1). The algorithm runs in polynomial time, since both F and A2 run in polynomial time (see Exercise 36.1-6).

NP-completeness

Polynomial-time reductions provide a formal means for showing that one problem is at least as hard as another, to within a polynomial-time factor. That is, if L1 [image] p L2, then L1 is not more than a polynomial factor harder than L2, which is why the "less than or equal to" notation for reduction is mnemonic. We can now define the set of NP-complete languages, which are the hardest problems in NP.

A language L [image] {0, 1}* is NP-complete if

1. L [image] NP, and

2. L' [image] p L for every L' [image] NP.

Figure 36.5 How most theoretical computer scientists view the relationships among P, NP, and NPC. Both P and NPC are wholly contained within NP, and .

If a language L satisfies property 2, but not necessarily property 1, we say that L is NP-hard. We also define NPC to be the class of NP-complete languages.

As the following theorem shows, NP-completeness is at the crux of deciding whether P is in fact equal to NP.

Theorem 36.4

If any NP-complete problem is polynomial-time solvable, then P = NP. If any problem in NP is not polynomial-time solvable, then all NP-complete problems are not polynomial-time solvable.

Proof Suppose that L [image] P and also that L [image] NPC. For any L' [image] NP, we have L' [image] p L by property 2 of the definition of NP-completeness. Thus, by Lemma 36.3, we also have that L' [image] P, which proves the first statement of the lemma.

To prove the second statement, suppose that there exists an L [image] NP such that L [image] P. Let L' [image] NPC be any NP-complete language, and for the purpose of contradiction, assume that L' [image] P. But then, by Lemma 36.3, we have L [image] p L', and thus L [image] P.

It is for this reason that research into the P [image] NP question centers around the NP-complete problems. Most theoretical computer scientists believe that P [image] NP, which leads to the relationships among P, NP, and NPC shown in Figure 36.5. But for all we know, someone may come up with a polynomial-time algorithm for an NP-complete problem, thus proving that P = NP. Nevertheless, since no polynomial-time algorithm for any NP-complete problem has yet been discovered, a proof that a problem is NP-compete provides excellent evidence for its intractability.

Circuit satisfiability

We have defined the notion of an NP-complete problem, but up to this point, we have not actually proved that any problem is NP-complete. Once we prove that at least one problem is NP-complete, we can use polynomial-time reducibility as a tool to prove the NP-completeness of other problems. Thus, we now focus on demonstrating the existence of an NP-complete problem: the circuit-satisfiability problem.

Figure 36.6 Two instances of the circuit-satisfiability problem. (a) The assignment x1 = 1, x2 = 1, x3 = 0 to the inputs of this circuit causes the output of the circuit to be 1. The circuit is therefore satisfiable. (b) No assignment to the inputs of this circuit can cause the output of the circuit to be 1. The circuit is therefore unsatisfiable.

Unfortunately, the formal proof that the circuit-satisfiability problem is NP-complete requires technical detail beyond the scope of this text. Instead, we shall informally describe a proof that relies on a basic understanding of boolean combinational circuits. This material is reviewed at the beginning of Chapter 29.

Figure 36.6 shows two boolean combinational circuits, each with three inputs and a single output. A truth assignment for a boolean combinational circuit is a set of boolean input values. We say that a one-output boolean combinational circuit is satisfiable if it has a satisfying assignment: a truth assignment that causes the output of the circuit to be 1. For example, the circuit in Figure 36.6(a) has the satisfying assignment [image] x1 = 1, x2 = 1, x3 = 0 [image], and so it is satisfiable. No assignment of values to x1, x2, and x3 causes the circuit in Figure 36.6(b) to produce a 1 output; it always produces 0, and so it is unsatisfiable.

The circuit-satisfiability problem is, "Given a boolean combinational circuit composed of AND, OR, and NOT gates, is it satisfiable?" In order to pose this question formally, however, we must agree on a standard encoding for circuits. One can devise a graphlike encoding that maps any given circuit C into a binary string [image] C [image] whose length is not much larger than the size of the circuit itself. As a formal language, we can therefore define

CIRCUIT-SAT =

{  C   : C is a satisfiable boolean combinational circuit} .

The circuit-satisfiability problem has great importance in the area of computer-aided hardware optimization. If a circuit always produces 0, it can be replaced by a simpler circuit that omits all logic gates and provides the constant 0 value as its output. A polynomial-time algorithm for the problem would have considerable practical application.

Given a circuit C, we might attempt to determine whether it is satisfiable by simply checking all possible assignments to the inputs. Unfortunately, if there are k inputs, there are 2k possible assignments. When the size of C is polynomial in k, checking each one leads to a superpolynomial-time algorithm. In fact, as has been claimed, there is strong evidence that no polynomial-time algorithm exists that solves the circuit-satisfiability problem because circuit satisfiability is NP-complete. We break the proof of this fact into two parts, based on the two parts of the definition of NP-completeness.

Lemma 36.5

The circuit-satisfiability problem belongs to the class NP.

Proof We shall provide a two-input, polynomial-time algorithm A that can verify CIRCUIT-SAT. One of the inputs to A is (a standard encoding of) a boolean combinational circuit C. The other input is a certificate corresponding to an assignment of boolean values to the wires in C.

The algorithm A is constructed as follows. For each logic gate in the circuit, it checks that the value provided by the certificate on the output wire is correctly computed as a function of the values on the input wires. Then, if the output of the entire circuit is 1, the algorithm outputs 1, since the values assigned to the inputs of C provide a satisfying assignment. Otherwise, A outputs 0.

Whenever a satisfiable circuit C is input to algorithm A, there is a certificate whose length is polynomial in the size of C and that causes A to output a 1. Whenever an unsatisfiable circuit is input, no certificate can fool A into believing that the circuit is satisfiable. Algorithm A runs in polynomial time: with a good implementation, linear time suffices. Thus, CIRCUIT-SAT can be verified in polynomial time, and CIRCUIT-SAT [image] NP.

The second part of proving that CIRCUIT-SAT is NP-complete is to show that the language is NP-hard. That is, we must show that every language in NP is polynomial-time reducible to CIRCUIT-SAT. The actual proof of this fact is full of technical intricacies, and so we shall settle for a sketch of the proof based on some understanding of the workings of computer hardware.

A computer program is stored in the computer memory as a sequence of instructions. A typical instruction encodes an operation to be performed, addresses of operands in memory, and an address where the result is to be stored. A special memory location, called the program counter, keeps track of which instruction is to be executed next. The program counter is automatically incremented whenever an instruction is fetched, thereby causing the computer to execute instructions sequentially. The execution of an instruction can cause a value to be written to the program counter, however, and then the normal sequential execution can be altered, allowing the computer to loop and perform conditional branches.

At any point during the execution of a program, the entire state of the computation is represented in the computer's memory. (We take the memory to include the program itself, the program counter, working storage, and any of the various bits of state that a computer maintains for bookkeeping.) We call any particular state of computer memory a configuration. The execution of an instruction can be viewed as mapping one configuration to another. Importantly, the computer hardware that accomplishes this mapping can be implemented as a boolean combinational circuit, which we denote by M in the proof of the following lemma.

Lemma 36.6

The circuit-satisfiability problem is NP-hard.

Proof Let L be any language in NP. We shall describe a polynomial-time algorithm F computing a reduction function â that maps every binary string x to a circuit C = â(x) such that x [image] L if and only if C [image] CIRCUIT-SAT.

Since L [image] NP, there must exist an algorithm A that verifies L in polynomial-time. The algorithm F that we shall construct will use the two-input algorithm A to compute the reduction function â.

Let T(n) denote the worst-case running time of algorithm A on length-n input strings, and let k [image] 1 be a constant such that T(n) = O(nk) and the length of the certificate is O(nk). (The running time of A is actually a polynomial in the total input size, which includes both an input string and a certificate, but since the length of the certificate is polynomial in the length n of the input string, the running time is polynomial in n.)

The basic idea of the proof is to represent the computation of A as a sequence of configurations. As shown in Figure 36.7, each configuration can be broken into parts consisting of the program for A, the program counter and auxiliary machine state, the input x, the certificate y, and working storage. Starting with an initial configuration c0, each configuration ci is mapped to a subsequent configuration ci+1 by the combinational circuit M implementing the computer hardware. The output of the algorithm A--0 or 1--is written to some designated location in the working storage when A finishes executing, and if we assume that thereafter A halts, the value never changes. Thus, if the algorithm runs for at most T(n) steps, the output appears as one of the bits in cT(n).

Figure 36.7 The sequence of configurations produced by an algorithm A running on an input x and certificate y. Each configuration represents the state of the computer for one step of the computation and, besides A, x, and y, includes the program counter (PC), auxiliary machine state, and working storage. Except for the certificate y, the initial configuration c0 is constant. Each configuration is mapped to the next configuration by a boolean combinational circuit M. The output is a distinguished bit in the working storage.

The reduction algorithm F constructs a single combinational circuit that computes all configurations produced by a given initial configuration. The idea is to paste together T(n) copies of the circuit M. The output of the ith circuit, which produces configuration ci, is fed directly into the input of the (i + 1)st circuit. Thus, the configurations, rather than ending up in a state register, simply reside as values on the wires connecting copies of M.

Recall what the polynomial-time reduction algorithm F must do. Given an input x, it must compute a circuit C = â(x) that is satisfiable if and only if there exists a certificate y such that A(x, y) = 1. When F obtains an input x, it first computes n = |x| and constructs a combinational circuit C' consisting of T(n) copies of M. The input to C' is an initial configuration corresponding to a computation on A(x, y), and the output is the configuration cT(n).

The circuit C = â(x) that F computes is obtained by modifying C' slightly. First, the inputs to C' corresponding to the program for A, the initial program counter, the input x, and the initial state of memory are wired directly to these known values. Thus, the only remaining inputs to the circuit correspond to the certificate y. Second, all outputs to the circuit are ignored, except the one bit of cT(n) corresponding to the output of A. This circuit C, so constructed, computes C(y) = A(x, y) for any input y of length O(nk). The reduction algorithm F, when provided an input string x, computes such a circuit C and outputs it.

Two properties remain to be proved. First, we must show that F correctly computes a reduction function â. That is, we must show that C is satisfiable if and only if there exists a certificate y such that A(x, y) = 1. Second, we must show that F runs in polynomial time.

To show that F correctly computes a reduction function, let us suppose that there exists a certificate y of length O(nk) such that A(x, y) = 1. Then, if we apply the bits of y to the inputs of C, the output of C is C(y) = A(x, y) = 1. Thus, if a certificate exists, then C is satisfiable. For the other direction, suppose that C is satisfiable. Hence, there exists an input y to C such that C(y) = 1, from which we conclude that A(x, y) = 1. Thus, F correctly computes a reduction function.

To complete the proof, we need only show that F runs in time polynomial in n = |x|. The first observation we make is that the number of bits required to represent a configuration is polynomial in n. The program for A itself has constant size, independent of the length of its input x. The length of the input x is n, and the length of the certificate y is O(nk). Since the algorithm runs for at most O(nk) steps, the amount of working storage required by A is polynomial in n as well. (We assume that this memory is contiguous; Exercise 36.3-4 asks you to extend the argument to the situation in which the locations accessed by A are scattered across a much larger region of memory and the particular pattern of scattering can differ for each input x.)

The combinational circuit M implementing the computer hardware has size polynomial in the length of a configuration, which is polynomial in O(nk) and hence is polynomial in n. (Most of this circuitry implements the logic of the memory system.) The circuit C consists of at most t = O(nk) copies of M, and hence it has size polynomial in n. The construction of C from x can be accomplished in polynomial time by the reduction algorithm F, since each step of the construction takes polynomial time.

The language CIRCUIT-SAT is therefore at least as hard as any language in NP, and since it belongs to NP, it is NP-complete.

Theorem 36.7

The circuit-satisfiability problem is NP-complete.

Proof Immediate from Lemmas 36.5 and 36.6 and the definition of NP-completeness.

Exercises

36.3-1

Show that the [image] p relation is a transitive relation on languages. That is, show that if L1 [image] p L2 and L2 [image] p L3, then L1 [image] p L3.

36.3-2

Prove that [image] if and only if [image].

36.3-3

Show that a satisfying assignment can be used as a certificate in an alternative proof of Lemma 36.5. Which certificate makes for an easier proof?

36.3-4

The proof of Lemma 36.6 assumes that the working storage for algorithm A occupies a contiguous region of polynomial size. Where in the proof is this assumption exploited? Argue that this assumption does not involve any loss of generality.

36.3-5

A language L is complete for a language class C with respect to polynomial-time reductions if L [image] C and L' [image] p L for all L' [image] C. Show that [image] and {0, 1}* are the only languages in P that are not complete for P with respect to polynomial-time reductions.

36.3-6

Show that L is complete for NP if and only if [image] is complete for co-NP.

36.3-7

The reduction algorithm F in the proof of Lemma 36.6 constructs the circuit C = f(x) based on knowledge of x, A, and k. Professor Sartre observes that the string x is input to F, but only the existence of A and k is known to F (since the language L belongs to NP), not their actual values. Thus, the professor concludes that F can't possibly construct the circuit C and that the language CIRCUIT-SAT is not necessarily NP-hard. Explain the flaw in the professor's reasoning.

36.4 NP-completeness proofs

The NP-completeness of the circuit-satisfiability problem relies on a direct proof that L [image] P CIRCUIT-SAT for every language L [image] NP. In this section, we shall show how to prove that languages are NP-complete without directly reducing every language in NP to the given language. We shall illustrate this methodology by proving that various formula-satisfiability problems are NP-complete. Section 36.5 provides many more examples of the methodology.

The following lemma is the basis of our method for showing that a language is NP-complete.

Lemma 36.8

If L is a language such that L' [image] p L for some L' [image] NPC, then L is NP-hard. Moreover, if L [image] NP, then L [image] NPC.

Proof Since L' is NP-complete, for all L\" [image] NP, we have L\" [image] p L'. By supposition, L' [image] p L, and thus by transitivity (Exercise 36.3-1), we have L\" [image] p L, which shows that L is NP-hard. If L [image] NP, we also have L [image] NPC.

In other words, by reducing a known NP-complete language L' to L, we implicitly reduce every language in NP to L. Thus, Lemma 36.8 gives us a method for proving that a language L is NP-complete:

1. Prove L [image] NP.

2. Select a known NP-complete language L'.

3. Describe an algorithm that computes a function f mapping every instance of L' to an instance of L.

4. Prove that the function f satisfies x [image] L' if and only if f(x) [image] L for all x [image] {0, 1 }*.

5. Prove that the algorithm computing f runs in polynomial time.

This methodology of reducing from a single known NP-complete language is far simpler than the more complicated process of providing reductions from every language in NP. Proving CIRCUIT-SAT [image] NPC has given us a "foot in the door." Knowing that the circuit-satisfiability problem is NP-complete now allows us to prove much more easily that other problems are NP-complete. Moreover, as we develop a catalog of known NP-complete problems, applying the methodology will become that much easier.

Formula satisfiability

We illustrate the reduction methodology by giving an NP-completeness proof for the problem of determining whether a boolean formula, not a circuit, is satisfiable. This problem has the historical honor of being the first problem ever shown to be NP-complete.

We formulate the (formula) satisfiability problem in terms of the language SAT as follows. An instance of SAT is a boolean formula [image] composed of

1. boolean variables: x1, x2, . . . . ;

2. boolean connectives: any boolean function with one or two inputs and one output, such as [image] (AND), [image] (OR), [image] (NOT), [image] (implication), [image] (if and only if); and

3. parentheses.

As in boolean combinational circuits, a truth assignment for a boolean formula [image] is a set of values for the variables of [image] , and a satisfying assignment is a truth assignment that causes it to evaluate to 1. A formula with a satisfying assignment is a satisfiable formula. The satisfiability problem asks whether a given boolean formula is satisfiable; in formal-language terms,

SAT = {      :    is a satisfiable boolean formula} .

As an example, the formula

   = ((x1    x2)      ((  x1    x3)    x4))      x2

has the satisfying assignment [image] x1 = 0, x2 = 0, x3 = 1, x4 = 1 [image], since

   = ((0    0)      ((  0    1)    1))      0

= (1      (1    1))    1

= (1    0)    1

= 1 ,

(36.2)

and thus this formula [image] belongs to SAT.

The naive algorithm to determine whether an arbitrary boolean formula is satisfiable does not run in polynomial time. There are 2n possible assignments in a formula [image] with n variables. If the length of [image] [image] [image] is polynomial in n, then checking every assignment requires superpolynomial time. As the following theorem shows, a polynomial-time algorithm is unlikely to exist.

Figure 36.8 Reducing circuit satisfiability to formula satisfiability. The formula produced by the reduction algorithm has a variable for each wire in the circuit.

Theorem 36.9

Satisfiability of boolean formulas is NP-complete.

Proof We shall argue first that SAT [image] NP. Then, we shall show that CIRCUIT-SAT [image] p SAT; by Lemma 36.8, this will prove the theorem.

To show that SAT belongs to NP, we show that a certificate consisting of a satisfying assignment for an input formula [image] can be verified in polynomial time. The verifying algorithm simply replaces each variable in the formula with its corresponding value and then evaluates the expression, much as we did in equation (36.2) above. This task is easily doable in polynomial time. If the expression evaluates to 1, the formula is satisfiable. Thus, the first condition of Lemma 36.8 for NP-completeness holds.

To prove that SAT is NP-hard, we show that CIRCUIT-SAT [image] p SAT. In other words, any instance of circuit satisfiability can be reduced in polynomial time to an instance of formula satisfiability. Induction can be used to express any boolean combinational circuit as a boolean formula. We simply look at the gate that produces the circuit output and inductively express each of the gate's inputs as formulas. The formula for the circuit is then obtained by writing an expression that applies the gate's function to its inputs' formulas.

Unfortunately, this straightforward method does not constitute a polynomial-time reduction. Shared subformulas can cause the size of the generated formula to grow exponentially (see Exercise 36.4-1). Thus, the reduction algorithm must be somewhat more clever.

Figure 36.8 illustrates the basic idea of the reduction from CIRCUIT-SAT to SAT on the circuit from Figure 36.6(a). For each wire xi in the circuit C, the formula [image] has a variable xi. The proper operation of a gate can now be expressed as a formula involving the variables of its incident wires. For example, the operation of the output AND gate is x10 [image] (x7 [image] x8 [image] x9).

The formula [image] produced by the reduction algorithm is the AND of the circuit-output variable with the conjunction of clauses describing the operation of each gate. For the circuit in the figure, the formula is

    =  x10    (x4      x3)

   (x5    (x1    x2))

   (x6      x4)

   (x7    (x1    x2    x4))

   (x8    (x5    x6))

   (x9    (x6    x7))

   (x10    (x7    x8    x9)) .

Given a circuit C, it is straightforward to produce such a formula [image] in polynomial time.

Why is the circuit [image] satisfiable exactly when the formula [image] is satisfiable? If C has a satisfying assignment, each wire of the circuit has a well-defined value, and the output of the circuit is 1. Therefore, the assignment of wire values to variables in [image] makes each clause of [image] evaluate to 1, and thus the conjunction of all evaluates to 1. Conversely, if there is an assignment that causes [image] to evaluate to 1, the circuit C is satisfiable by an analogous argument. Thus, we have shown that CIRCUIT-SAT [image] p SAT, which completes the proof.

3-CNF satisfiability

Many problems can be proved NP-complete by reduction from formula satisfiability. The reduction algorithm must handle any input formula, though, and this can lead to a huge number of cases that must be considered. It is often desirable, therefore, to reduce from a restricted language of boolean formulas, so that fewer cases need be considered. Of course, we must not restrict the language so much that it becomes polynomial-time solvable. One convenient language is 3-CNF satisfiability, or 3-CNF-SAT.

We define 3-CNF satisfiability using the following terms. A literal in a boolean formula is an occurrence of a variable or its negation. A boolean formula is in conjunctive normal form, or CNF, if it is expressed as an AND of clauses, each of which is the OR of one or more literals. A boolean formula is in 3-conjunctive normal form, or 3-CNF, if each clause has exactly three distinct literals.

For example, the boolean formula

(x1      x1      x2)    (x3    x2    x4)    (  x1      x3      x4)

is in 3-CNF. The first of its three clauses is (x1 [image] [image] x1 [image] [image] x2 ), which contains the three literals x1, x1, and x2.

In 3-CNF-SAT, we are asked whether a given boolean formula in 3-CNF is satisfiable. The following theorem shows that a polynomial-time algorithm that can determine the satisfiability of boolean formulas is unlikely to exist, even when they are expressed in this simple normal form.

   = ((x1    x2)      ((  x1    x3)    x4))      x2 .

Figure 36.9 The tree corresponding to the formula

Theorem 36.10

Satisfiability of boolean formulas in 3-conjunctive normal form is NP-complete.

Proof The argument we used in the proof of Theorem 36.9 to show that SAT NP applies equally well here to show that 3-CNF-SAT NP. Thus, we need only show that 3-CNF-SAT is NP-hard. We prove this by showing that SAT p 3-CNF-SAT, from which the proof will follow by Lemma 36.8.

The reduction algorithm can be broken into three basic steps. Each step progressively transforms the input formula closer to the desired 3-conjunctive normal form.

The first step is similar to the one used to prove CIRCUIT-SAT p SAT in Theorem 36.9. First, we construct a binary "parse" tree for the input formula , with literals as leaves and connectives as internal nodes. Figure 36.9 shows such a parse tree for the formula

   = ((x1    x2)      ((  x1    x3)    x4))      x2 .

(36.3)

Should the input formula contain a clause such as the OR of several literals, associativity can be used to parenthesize the expression fully so that every internal node in the resulting tree has 1 or 2 children. The binary parse tree can now be viewed as a circuit for computing the function.

Mimicking the reduction in the proof of Theorem 36.9, we introduce a variable yi for the output of each internal node. Then, we rewrite the original formula as the AND of the root variable and a conjunction of clauses describing the operation of each node. For the formula (36.3), the resulting expression is

  ' = y1    (y1    (y2      x2))

   (y2    (y3    y4))

   (y3    (x1    x2))

   (y4      y5)

   (y5    (y6    y4))

   (y6    (  x1    x3)).

Observe that the formula ' thus obtained is a conjunction of clauses , each of which has at most 3 literals. The only additional requirement is that each clause be an OR of literals.

y1  y2  x2  (y1    (y2      x2))

------------------------------

1   1   1           0

1   1   0           1

1   0   1           0

1   0   0           0

0   1   1           1

0   1   0           0

0   0   1           1

0   0   0           1

Figure 36.10 The truth table for the clause (y1 (y2 x2)).

The second step of the reduction converts each clause into conjunctive normal form. We construct a truth table for by evaluating all possible assignments to its variables. Each row of the truth table consists of a possible assignment of the variables of the clause, together with the value of the clause under that assignment. Using the truth-table entries that evaluate to 0, we build a formula in disjunctive normal form (or DNF)--an OR of AND's--that is equivalent to . We then convert this formula into a CNF formula by using DeMorgan's laws (5.2) to complement all literals and change OR's into AND's and AND's into OR's.

In our example, we convert the clause into CNF as follows. The truth table for 'i is given in Figure 36.10. The DNF formula equivalent to

(y1    y2    x2)    (y1      y2    x2)    (y1      y2      x2)    (  y1    y2      x2) .

Applying DeMorgan's laws, we get the CNF formula

which is equivalent to the original clause .

Each clause of the formula ' has now been converted into a CNF formula , and thus ' is equivalent to the CNF formula \" consisting of the conjunction of the . Moreover, each clause of \" has at most 3 literals.

The third and final step of the reduction further transforms the formula so that each clause has exactly 3 distinct literals. The final 3-CNF formula ''' is constructed from the clauses of the CNF formula \". It also uses two auxiliary variables that we shall call p and q. For each clause Ci of \", we include the following clauses in ''':

If Ci has 3 distinct literals, then simply include Ci as a clause of '''.

If Ci has 2 distinct literals, that is, if Ci = (l1 l2), where l1 and l2 are literals, then include (l1 l2 p) (l1 l2 p) as clauses of f( ). The literals p and p merely fulfill the syntactic requirement that there be exactly 3 distinct literals per clause: (l1 l2 p) (l1 l2 p) is equivalent to (l1 l2) whether p = 0 or p = 1.

If Ci has just 1 distinct literal l, then include (l p q) (l p q) (l p q) (l p q) as clauses of '''. Note that every setting of p and q causes the conjunction of these four clauses to evaluate to l.

We can see that the 3-CNF formula ''' is satisfiable if and only if is satisfiable by inspecting each of the three steps. Like the reduction from CIRCUIT-SAT to SAT, the construction of ' from in the first step preserves satisfiability. The second step produces a CNF formula \" that is algebraically equivalent to '. The third step produces a 3-CNF formula ''' that is effectively equivalent to \", since any assignment to the variables p and q produces a formula that is algebraically equivalent to \".

We must also show that the reduction can be computed in polynomial time. Constructing ' from introduces at most 1 variable and 1 clause per connective in . Constructing \" from ' can introduce at most 8 clauses into \" for each clause from ', since each clause of ' has at most 3 variables, and the truth table for each clause has at most 23 = 8 rows. The construction of ''' from \" introduces at most 4 clauses into ''' for each clause of \". Thus, the size of the resulting formula ''' is polynomial in the length of the original formula. Each of the constructions can easily be accomplished in polynomial time.

Exercises

36.4-1

Consider the straightforward (nonpolynomial-time) reduction in the proof of Theorem 36.9. Describe a circuit of size n that, when converted to a formula by this method, yields a formula whose size is exponential in n.

36.4-2

Show the 3-CNF formula that results when we use the method of Theorem 36.10 on the formula (36.3).

36.4-3

Professor Jagger proposes to show that SAT P 3-CNF-SAT by using only the truth-table technique in the proof of Theorem 36.10, and not the other steps. That is, the professor proposes to take the boolean formula , form a truth table for its variables, derive from the truth table a formula in 3-DNF that is equivalent to , and then negate and apply DeMorgan's laws to produce a 3-CNF formula equivalent to . Show that this strategy does not yield a polynomial-time reduction.

36.4-4

Show that the problem of determining whether a boolean formula is a tautology is complete for co-NP. (Hint: See Exercise 36.3-6.)

36.4-5

Show that the problem of determining the satisfiability of boolean formulas in disjunctive normal form is polynomial-time solvable.

36.4-6

Suppose that someone gives you a polynomial-time algorithm to decide formula satisfiability. Describe how to use this algorithm to find satisfying assignments in polynomial time.

36.4-7

Let 2-CNF-SAT be the set of satisfiable boolean formulas in CNF with exactly 2 literals per clause. Show that 2-CNF-SAT P. Make your algorithm as efficient as possible. (Hint: Observe that x y is equivalent to x y. Reduce 2-CNF-SAT to a problem on a directed graph that is efficiently solvable.)

36.5 NP-complete problems

NP-complete problems arise in diverse domains: boolean logic, graphs, arithmetic, network design, sets and partitions, storage and retrieval, sequencing and scheduling, mathematical programming, algebra and number theory, games and puzzles, automata and language theory, program optimization, and more. In this section, we shall use the reduction methodology to provide NP-completeness proofs for a variety of problems drawn from graph theory and set partitioning.

Figure 36.11 outlines the structure of the NP-completeness proofs in this section and Section 36.4. Each language in the figure is proved NP-complete by reduction from the language that points to it. At the root is CIRCUIT-SAT, which we proved NP-complete in Theorem 36.7.

Figure 36.11 The structure of NP-completeness proofs in Sections 36.4 and 36.5. All proofs ultimately follow by reduction from the NP-completeness of CIRCUIT-SAT.

36.5.1 The clique problem

A clique in an undirected graph G = (V, E) is a subset V' V of vertices, each pair of which is connected by an edge in E. In other words, a clique is a complete subgraph of G. The size of a clique is the number of vertices it contains. The clique problem is the optimization problem of finding a clique of maximum size in a graph. As a decision problem, we ask simply whether a clique of a given size k exists in the graph. The formal definition is

CLIQUE = {  G,k  : G is a graph with a clique of size k}.

A naive algorithm for determining whether a graph G = (V, E) with |V| vertices has a clique of size k is to list all k-subsets of V, and check each one to see whether it forms a clique. The running time of this algorithm is , which is polynomial if k is a constant. In general, however, k could be proportional to |V|, in which case the algorithm runs in superpolynomial time. As one might suspect, an efficient algorithm for the clique problem is unlikely to exist.

Theorem 36.11

The clique problem is NP-complete.

Proof To show that CLIQUE NP, for a given graph G = (V, E), we use the set V' V of vertices in the clique as a certificate for G. Checking whether V' is a clique can be accomplished in polynomial time by checking whether, for every pair u, v V', the edge (u, v) belongs to E.

We next show that the clique problem is NP-hard by proving that 3-CNF-SAT p CLIQUE. That we should be able to prove this result is somewhat surprising, since on the surface logical formulas seem to have little to do with graphs.

Figure 36.12 The graph G derived from the 3-CNF formula = C1 C2 C3, where C1 = (x1 x2 x3), C2 =( x1 x2 x3), and C3 = (x1 x2 x3), in reducing 3-CNF-SAT to CLIQUE. A satisfying assignment of the formula is x1 = 0, x2 = 0, x3 = 1 . This satisfying assignment satisfies C1 with x2, and it satisfies C2 and C3 with x3, corresponding to the clique with lightly shaded vertices.

The reduction algorithm begins with an instance of 3-CNF-SAT. Let = C1 C2 . . . Ck be a boolean formula in 3-CNF with k clauses. For r = 1, 2, . . . , k, each clause Cr has exactly three distinct literals . We shall construct a graph G such that is satisfiable if and only if G has a clique of size k.

The graph G = (V, E) is constructed as follows. For each clause in , we place a triple of vertices in V. We put an edge between two vertices if both of the following hold:

are in different triples, that is, r s, and

their corresponding literals are consistent, that is, is not the negation of .

This graph can easily be computed from in polynomial time. As an example of this construction, if we have

   = (x1      x2      x3)    (  x1    x2    x3)    (x1    x2    x3) ,

then G is the graph shown in Figure 36.12.

We must show that this transformation of into G is a reduction. First, suppose that has a satisfying assignment. Then, each clause Cr contains at least one literal that is assigned 1, and each such literal corresponds to a vertex . Picking one such "true" literal from each clause yields a set of V' of k vertices. We claim that V' is a clique. For any two vertices , where r s, both corresponding literals are mapped to 1 by the given satisfying assignment, and thus the literals cannot be complements. Thus, by the construction of G, the edge belongs to E.

Figure 36.13 Reducing CLIQUE to VERTEX-COVER. (a) An undirected graph G = (V,E) with clique V' = {u,v,x,y}. (b) The graph produced by the reduction algorithm that has vertex cover V - V' = {w,z}.

Conversely, suppose that G has a clique V' of size k. No edges in G connect vertices in the same triple, and so V' contains exactly one vertex per triple. We can assign 1 to each literal such that without fear of assigning 1 to both a literal and its complement, since G contains no edges between inconsistent literals. Each clause is satisfied, and so satisfied. (Any variables that correspond to no vertex in the clique may be set arbitrarily.)

In the example of Figure 36.12, a satisfying assignment of is x1 = 0, x2 = 0, x3 = 1 . A corresponding clique of size k = 3 consists of the vertices corresponding to x2 from the first clause, x3 from the second clause, and x3 from the third clause.

36.5.2 The vertex-cover problem

A vertex cover of an undirected graph G = (V, E) is a subset V' Vsuch that if (u, v) E, then u V' or v V' (or both). That is, each vertex "covers" its incident edges, and a vertex cover for G is a set of vertices that covers all the edges in E. The size of a vertex cover is the number of vertices in it. For example, the graph in Figure 36.13(b) has a vertex cover {w, z} of size 2.

The vertex-cover problem is to find a vertex cover of minimum size in a given graph. Restating this optimization problem as a decision problem, we wish to determine whether a graph has a vertex cover of a given size k. As a language, we define

VERTEX-COVER = {  G, k  : graph G has vertex cover of size k} .

The following theorem shows that this problem is NP-complete.

Theorem 36.12

The vertex-cover problem is NP-complete.

Proof We first show that VERTEX-COVER NP. Suppose we are given a graph G = (V, E) and an integer k. The certificate we choose is the vertex cover V' V itself. The verification algorithm affirms that |V'| = k, and then it checks, for each edge (u, v) E, whether u V' or v V'. This verification can be performed straightforwardly in polynomial time.

We prove that the vertex-cover problem is NP-hard by showing that CLIQUE P VERTEX-COVER. This reduction is based on the notion of the "complement" of a graph. Given an undirected graph G = (V, E), we define the complement of G as . In other words, is the graph containing exactly those edges that are not in G. Figure 36.13 shows a graph and its complement and illustrates the reduction from CLIQUE to VERTEX-COVER.

The reduction algorithm takes as input an instance G, k of the clique problem. It computes the complement , which is easily doable in polynomial time. The output of the reduction algorithm is the instance , of the vertex-cover problem. To complete the proof, we show that this transformation is indeed a reduction: the graph G has a clique of size k if and only if the graph has a vertex cover of size |V| - k.

Suppose that G has a clique V' V with |V'| = k. We claim that V - V' is a vertex cover in . Let (u, v) be any edge in . Then, (u, v) E, which implies that at least one of u or v does not belong to V', since every pair of vertices in V' is connected by an edge of E. Equivalently, at least one of u or v is in V - V', which means that edge (u,v) is covered by V - V'. Since (u, v) was chosen arbitrarily from , every edge of is covered by a vertex in V - V'. Hence, the set V - V', which has size |V| - k, forms a vertex cover for .

Conversely, suppose that has a vertex cover V' V , where |V'| = |V| - k. Then, for all u, v V, if (u, v) , if , then u V' or v V' or both. The contrapositive of this implication is that for all u, v V, if u V' and v V', then (u, v) E. In other words, V - V' is a clique, and it has size |V| - |V'| = k.

Since VERTEX-COVER is NP-complete, we don't expect to find a polynomial-time algorithm for finding a minimum-size vertex cover. Section 37.1 presents a polynomial-time "approximation algorithm," however, which produces "approximate" solutions for the vertex-cover problem. The size of a vertex cover produced by the algorithm is at most twice the minimum size of a vertex cover.

Thus, we shouldn't give up hope just because a problem is NP-complete. There may be a polynomial-time approximation algorithm that obtains near-optimal solutions, even though finding an optimal solution is NP-complete. Chapter 37 gives several approximation algorithms for NP-complete problems.

36.5.3 The subset-sum problem

The next NP-complete problem we consider is arithmetic. In the subset-sum problem, we are given a finite set S N and a target t N. We ask whether there is a subset S' S whose elements sum to t. For example, if S = {1, 4, 16, 64, 256, 1040, 1041, 1093, 1284, 1344} and t = 3754, then the subset S' = {1, 16, 64, 256, 1040, 1093, 1284} is a solution.

As usual, we define the problem as a language:

SUBSET-SUM =

{  S, t  : there exists a subset S'    S such that t =   s  S'S} .

As with any arithmetic problem, it is important to recall that our standard encoding assumes that the input integers are coded in binary. With this assumption in mind, we can show that the subset-sum problem is unlikely to have a fast algorithm.

Theorem 36.13

The subset-sum problem is NP-complete.

Proof To show that SUBSET-SUM is in NP, for an instance S, t of the problem, we let the subset S' be the certificate. Checking whether t = s S's can be accomplished by a verification algorithm in polynomial time.

We now show that VERTEX-COVER P SUBSET-SUM. Given an instance G, k of the vertex-cover problem, the reduction algorithm constructs an instance S, t of the subset-sum problem such that G has a vertex cover of size k if and only if there is a subset of S whose sum is exactly t.

At the heart of the reduction is an incidence-matrix representation of G. Let G = (V, E) be an undirected graph and let V = {v0, v1, . . . ,v|V|-1} and E = {e0, e1, . . . , e|E|-1}. The incidence matrix of G is a |V| |E| matrix B = (bij) such that

For example, Figure 36.14(b) shows the incidence matrix for the undirected graph of Figure 36.14(a). The incidence matrix is shown with lower-indexed edges on the right, rather than on the left as is conventional, in order to simplify the formulas for the numbers in S.

Given a graph G and an integer k, the reduction algorithm computes a set S of numbers and an integer t. To understand how the reduction algorithm works, let us represent numbers in a "modified base-4" fashion. The |E| low-order digits of a number will be in base-4 but the high-order digit can be as large as k. The set of numbers is constructed in such a way that no carries can be propagated from lower digits to higher digits.

Figure 36.14 The reduction of the vertex-cover problem to the subset-sum problem. (a) An undirected graph G. A vertex cover {v1, v3, v4} of size 3 is lightly shaded. (b) The corresponding incidence matrix. Shading of the rows corresponds to the vertex cover of part (a). Each edge ej has a 1 in at least one lightly shaded row. (c) The corresponding subset-sum instance. The portion within the box is the incidence matrix. Here, the vertex cover {v1, v3, v4} of size k = 3 corresponds to the lightly shaded subset {1, 16, 64, 256, 1040, 1093, 1284}, which adds up to 3754.

The set S consists of two types of numbers, corresponding to vertices and edges respectively. For each vertex vi V, we create a positive integer xi whose modified base-4 representation consists of a leading 1 followed by |E| digits. The digits correspond to vi's row of the incidence matrix B = (bij) for G, as illustrated in Figure 36.14(c). Formally, for i = 0, 1, . . . , |V| - 1,

For each edge ej E, we create a positive integer yj that is just a row of the "identity" incidence matrix. (The identity incidence matrix is the |E| |E| matrix with 1's only in the diagonal positions.) Formally, for j = 0, 1, . . . , |E| - 1,

yj = 4j .

The first digit of the target sum t is k, and all |E| lower-order digits are 2's. Formally,

All of these numbers have polynomial size when we represent them in binary. The reduction can be performed in polynomial time by manipulating the bits of the incidence matrix.

We must now show that graph G has a vertex cover of size k if and only if there is a subset S' S whose sum is t. First, suppose that G has a vertex cover V' V of size k. Let V' = {vi1, vi2, . . . , vik}, and define S' by

S' = {xi1, xi2, . . . , xik}

{yj : ej is incident on precisely one vertex in V'} .

To see that s S'S = t, observe that summing the k leading 1's of the xim S' gives the leading digit k of the modified base-4 representation of t. To get the low-order digits of t, each of which is a 2, consider the digit positions in turn, each of which corresponds to an edge ej. Because V' is a vertex cover, ej is incident on at least one vertex in V'. Thus, for each edge ej, there is at least one xim S' with a 1 in the jth position. If ej is incident on two vertices in V', then both contribute a 1 to the sum in the jth position. The jth digit of yj contributes nothing, since ej is incident on two vertices, which implies that yj S'. Thus, in this case, the sum of S' produces a 2 in the jth position of t. For the other case--when ej is incident on exactly one vertex in V'--we have yj S', and the incident vertex and yj each contribute 1 to the sum of the jth digit of t, thereby also producing a 2. Thus, S' is a solution to the subset-sum instance S.

Now, suppose that there is a subset S' S that sums to t. Let S = {xi1, xi2, . . . , xim} {yj1, yj2, . . . , yjp}. We claim that m = k and that V' = {vi1, vi2, . . . , vim} is a vertex cover for G. To prove this claim, we start by observing that for each edge ej E, there are three 1's in set S in the ej position: one from each of the two vertices incident on ej, and one from yj. Because we are working with a modified base-4 representation, there are no carries from position ej to position ej+1. Thus, for each of the |E| low-order positions of t, at least one and at most two xi must contribute to the sum. Since at least one xi contributes to the sum for each edge, we see that V' is a vertex cover. To see that m = k, and thus that V' is a vertex cover of size k, observe that the only way the leading k in target t can be achieved is by including exactly k of the xi in the sum.

In Figure 36.14, the vertex cover V' = {v1, v3, v4} corresponds to the subset S' = {x1, x3, x4, y0, y2, y3, y4}. All of the yj are included in S', with the exception of y1, which is incident on two vertices in V'.

36.5.4 The hamiltonian-cycle problem

We now return to the hamiltonian-cycle problem defined in Section 36.2.

Figure 36.15 (a) Widget A, used in the reduction from 3-CNF-SAT to HAM-CYCLE. (b)-(c) If A is a subgraph of some graph G that contains a hamiltonian cycle and the only connections from A to the rest of G are through the vertices a, a', b, and b', then the shaded edges represent the only two possible ways in which the hamiltonian cycle may traverse the edges of subgraph A. (d) A compact representation of the A widget.

Theorem 36.14

The hamiltonian cycle problem is NP-complete.

Proof We first show that HAM-CYCLE belongs to NP. Given a graph G = (V, E), our certificate is the sequence of |V| vertices that make up the hamiltonian cycle. The verification algorithm checks that this sequence contains each vertex in V exactly once and that with the first vertex repeated at the end, it forms a cycle in G. This verification can be performed in polynomial time.

We now prove that HAM-CYCLE is NP-complete by showing that 3-CNF-SAT P HAM-CYCLE. Given a 3-CNF boolean formula over variables x1, x2, . . . , xn with clauses C1, C2, . . . , Ck, each containing exactly 3 distinct literals, we construct a graph G = (V, E) in polynomial time such that G has a hamiltonian cycle if and only if is satisfiable. Our construction is based on widgets, which are pieces of graphs that enforce certain properties.

Our first widget is the subgraph A shown in Figure 36.15(a). Suppose that A is a subgraph of some graph G and that the only connections between A and the remainder of G are through the vertices a, a', b, and b'. Furthermore, suppose that graph G has a hamiltonian cycle. Since any hamiltonian cycle of G must pass through vertices z1, z2, z3, and z4 in one of the ways shown in Figures 36.15(b) and (c), we may treat subgraph A as if it were simply a pair of edges (a, a') and (b, b') with the restriction that any hamiltonian cycle of G must include exactly one of these edges. We shall represent widget A as shown in Figure 36.15(d).

The subgraph B in Figure 36.16 is our second widget. Suppose that B is a subgraph of some graph G and that the only connections from B to the remainder of G are through vertices b1, b2, b3, and b4. A hamiltonian cycle of graph G cannot traverse all of the edges (b1, b2), (b2, b3), and (b3, b4), since then all vertices in the widget other than b1, b2, b3, and b4 would be missed. A hamiltonian cycle of G may, however, traverse any proper subset of these edges. Figures 36.16(a)-(e) show five such subsets; the remaining two subsets can be obtained by performing a top-to-bottom flip of parts (b) and (e). We represent this widget as in Figure 36.16(f), the idea being that at least one of the paths pointed to by the arrows must be taken by a hamiltonian cycle.

The graph G that we shall construct consists mostly of copies of these two widgets. The construction is illustrated in Figure 36.17. For each of the k clauses in , we include a copy of widget B, and we join these widgets together in series as follows. Letting bij be the copy of vertex bj in the ith copy of widget B, we connect bi,4 to bi+1,1 for i = 1, 2, . . . , k - 1.

Then, for each variable xm in , we include two vertices . We connect these two vertices by means of two copies of the edge , which we denote by em and to distinguish them. The idea is that if the hamiltonian cycle takes edge em, it corresponds to assigning variable xm the value 1. If the hamiltonian cycle takes edge , the variable is assigned the value 0. Each pair of these edges forms a two-edge loop; we connect these small loops in series by adding edges for m = 1, 2, . . . , n - 1. We connect the left (clause) side of the graph to the right (variable) side by means of two edges , which are the topmost and bottommost edges in Figure 36.17.

We are not yet finished with the construction of graph G, since we have yet to relate the variables to the clauses. If the jth literal of clause Ci is xm, then we use an A widget to connect edge (bij, bi,j+1) with edge em. If the jth literal of clause Ci is xm, then we instead put an A widget between edge (bij, bi,j+1) and edge . In Figure 36.17, for example, because clause C2 is (x1 x2 x3), we place three A widgets as follows:

between (b2,1, b2,2) and e1,

between (b2,2, b2,3) and , and

between (b2,3, b2,4) and e3.

Note that connecting two edges by means of A widgets actually entails replacing each edge by the five edges in the top or bottom of Figure 36.15(a) and, of course, adding the connections that pass through the z vertices as well. A given literal lm may appear in several clauses ( xm in Figure 36.17, for example), and thus an edge em or may be be influenced by several A widgets (edge , for example). In this case, we connect the A widgets in series, as shown in Figure 36.18, effectively replacing edge em or by a series of edges.

Figure 36.16 Widget B, used in the reduction from 3-CNF-SAT to HAM-CYCLE. No path from vertex b1 to vertex b4 containing all the vertices in the widget may use all three edges (b1, b2), (b2, b3), and (b3, b4). Any proper subset of these edges may be used, however. (a)-(e) Five such subsets. (f) A representation of this widget in which at least one of the paths pointed to by the arrows must be taken by a hamiltonian cycle.

Figure 36.17 The graph G constructed from the formula = ( x1 V x2 V x3) (x1 V x2 V x3) (x1 V x2 V x3). A satisfying assignment s to the variables of is s(x1) = 0, s(x2) = 1, and s(x3) = 1, which corresponds to the hamiltonian cycle shown. Note that if s(xm) = 1, then edge em is in the hamiltonian cycle, and if s(xm) = 0, then edge is in the hamiltonian cycle.

Figure 36.18 The actual construction used when an edge em or is influenced by multiple A widgets. (a) A portion of Figure 36.17. (b) The actual subgraph constructed.

We claim that formula is satisfiable if and only if graph G contains a hamiltonian cycle. We first suppose that G has a hamiltonian cycle h and show that is satisfiable. Cycle h must take a particular form:

First, it traverses edge to go from the top left to the top right.

It then follows all of the vertices from top to bottom, choosing either edge em or edge , but not both.

It next traverses edge to get back to the left side.

Finally, it traverses the B widgets from bottom to top on the left.

(It actually traverses edges within the A widgets as well, but we use these subgraphs to enforce the either/or nature of the edges it connects.)

Given the hamiltonian cycle h, we define a truth assignment for as follows. If edge em belongs to h, then we set xm = 1. Otherwise, edge belongs to h, and we set xm = 0.

We claim that this assignment satisfies . Consider a clause Ci and the corresponding B widget in G. Each edge (bi,jbi,j+l) is connected by an A widget to either edge em or edge , depending on whether xm or xm is the jth literal in the clause. The edge (bi,j,bi,j+1) is traversed by h if and only if the corresponding literal is 0. Since each of the three edges (bi,1,bi,2),(bi,2,bi,3),(bi,3,bi,4) in clause Ci is also in a B widget, all three cannot be traversed by the hamiltonian cycle h. One of the three edges, therefore, must have a corresponding literal whose assigned value is 1, and clause Ci is satisfied. This property holds for each clause Ci, i = 1, 2, . . . , k, and thus formula ø is satisfied.

Figure 36.19 An instance of the traveling-salesman problem. Shaded edges represent a minimum-cost tour, with cost 7.

Conversely, let us suppose that formula ø is satisfied by some truth assignment. By following the rules from above, we can construct a hamiltonian cycle for graph G: traverse edge em if xm = 1, traverse edge if xm = 0, and traverse edge (bi,j,bi,j+1) if and only if the jth literal of clause Ci is 0 under the assignment. These rules can indeed be followed, since we assume that s is a satisfying assignment for formula ø.

Finally, we note that graph G can be constructed in polynomial time. It contains one B widget for each of the k clauses in ø. There is one A widget for each instance of each literal in ø, and so there are 3k A widgets. Since the A and B widgets are of fixed size, the graph G has O(k) vertices and edges and is easily constructed in polynomial time. Thus, we have provided a polynomial-time reduction from 3-CNF-SAT to HAM-CYCLE.

36.5.5 The traveling-salesman problem

In the traveling-salesman problem, which is closely related to the hamiltonian-cycle problem, a salesman must visit n cities. Modeling the problem as a complete graph with n vertices, we can say that the salesman wishes to make a tour, or hamiltonian cycle, visiting each city exactly once and finishing at the city he starts from. There is an integer cost c(i, j) to travel from city i to city j, and the salesman wishes to make the tour whose total cost is minimum, where the total cost is the sum of the individual costs along the edges of the tour. For example, in Figure 36.19, a minimum-cost tour is u, w, v, x, u , with cost 7. The formal language for the traveling-salesman problem is

TSP = {  G,c,k   : G = (V,E) is a complete graph,

c is a function from V    V    Z,

k    Z, and

G has a traveling-salesman tour with cost at most k} .

The following theorem shows that a fast algorithm for the traveling-salesman problem is unlikely to exist.

Theorem 36.15

The traveling-salesman problem is NP-complete.

Proof We first show that TSP belongs to NP. Given an instance of the problem, we use as a certificate the sequence of n vertices in the tour. The verification algorithm checks that this sequence contains each vertex exactly once, sums up the edge costs, and checks whether the sum is at most k. This process can certainly be done in polynomial time.

To prove that TSP is NP-hard, we show that HAM-CYCLE p TSP. Let G = (V, E) be an instance of HAM-CYCLE. We construct an instance of TSP as follows. We form the complete graph G'= (V,E'), where E' = {(i, j): i, j V}, and we define the cost function c by

The instance of TSP is then (G', c, 0), which is easily formed in polynomial time.

We now show that graph G has a hamiltonian cycle if and only if graph G' has a tour of cost at most 0. Suppose that graph G has a hamiltonian cycle h. Each edge in h belongs to E and thus has cost 0 in G'. Thus, h is a tour in G' with cost 0. Conversely, suppose that graph G' has a tour h' of cost at most 0. Since the costs of the edges in E' are 0 and 1, the cost of tour h' is exactly 0. Therefore, h' contains only edges in E. We conclude that h is a hamiltonian cycle in graph G.

Exercises

36.5-1

The subgraph-isomorphism problem takes two graphs G1 and G2 and asks whether G1 is a subgraph of G2. Show that the subgraph-isomorphism problem is NP-complete.

36.5-2

Given an integer m-by-n matrix A and an integer m-vector b, the 0-1 integer-programming problem asks whether there is an integer n-vector x with elements in the set {0,1} such that Ax b. Prove that 0-1 integer programming is NP-complete. (Hint: Reduce from 3-CNF-SAT.)

36.5-3

Show that the subset-sum problem is solvable in polynomial time if the target value t is expressed in unary.

36.5-4

The set-partition problem takes as input a set S of numbers. The question is whether the numbers can be partitioned into two sets A and such that . Show that the set-partition problem is NP-complete.

36.5-5

Show that the hamiltonian-path problem is NP-complete.

36.5-6

The longest-simple-cycle problem is the problem of determining a simple cycle (no repeated vertices) of maximum length in a graph. Show that this problem is NP-complete.

36.5-7

Professor Marconi proclaims that the subgraph used as widget A in the proof of Theorem 36.14 is more complicated than necessary: vertices z3 and z4 of Figure 36.15(a) and the vertices above and below them are not needed. Is the professor correct? That is, does the reduction work with this smaller version of the widget, or does the "either/or" property of the widget disappear?

Problems

36-1 Independent set

An independent set of a graph G = (V,E) is a subset V' V of vertices such that each edge in E is incident on at most one vertex in V'. The independent-set problem is to find a maximum-size independent set in G.

a. Formulate a related decision problem for the independent-set problem, and prove that it is NP-complete. (Hint: Reduce from the clique problem.)

b. Suppose that you are given a subroutine to solve the decision problem you defined in part (a). Give an algorithm to find an independent set of maximum size. The running time of your algorithm should be polynomial in |V| and |E|, where queries to the black box are counted as a single step.

Although the independent-set decision problem is NP-complete, certain special cases are polynomial-time solvable.

c. Give an efficient algorithm to solve the independent-set problem when each vertex in G has degree 2. Analyze the running time, and prove that your algorithm works correctly.

d. Give an efficient algorithm to solve the independent-set problem when G is bipartite. Analyze the running time, and prove that your algorithm works correctly. (Hint: Use the results of Section 27.3.)

Figure 36.20 The widget corresponding to a clause (x V y V z), used in Problem 36-2.

36-2 Graph coloring

A k-coloring of an undirected graph G = (V,E) is a function c : V {1, 2, . . . , k} such that c(u) c(v) for every edge (u,v) E. In other words, the numbers 1, 2, . . . , k represent the k colors, and adjacent vertices must have different colors. The graph-coloring problem is to determine the minimum number of colors needed to color a given graph.

a. Give an efficient algorithm to determine a 2-coloring of a graph if one exists.

b. Cast the graph-coloring problem as a decision problem. Show that your decision problem is solvable in polynomial time if and only if the graph-coloring problem is solvable in polynomial time.

c. Let the language 3-COLOR be the set of graphs that can be 3-colored. Show that if 3-COLOR is NP-complete, then your decision problem from part (b) is NP-complete.

To prove that 3-COLOR is NP-complete, we use a reduction from 3-CNF-SAT. Given a formula ø of m clauses on n variables x1, x2, . . . , xn, we construct a graph G = (V,E) as follows. The set V consists of a vertex for each variable, a vertex for the negation of each variable, 5 vertices for each clause, and 3 special vertices: TRUE, FALSE, and RED. The edges of the graph are of two types: "literal" edges that are independent of the clauses and "clause" edges that depend on the clauses. The literal edges form a triangle on the special vertices and also form a triangle on xi, xi, and RED for i = 1, 2, . . . , n.

d. Argue that in any 3-coloring c of a graph containing the literal edges, exactly one of a variable and its negation is colored c(TRUE) and the other is colored c(FALSE). Argue that for any truth assignment for ø, there is a 3-coloring of the graph containing just the literal edges.

The widget shown in Figure 36.20 is used to enforce the condition corresponding to a clause (x V y V z). Each clause requires a unique copy of the 5 vertices that are heavily shaded in the figure; they connect as shown to the literals of the clause and the special vertex TRUE.

e. Argue that if each of x, y, and z is colored c(TRUE) or c(FALSE), then the widget is 3-colorable if and only if at least one of x, y, or z is colored c(TRUE).

f. Complete the proof that 3-COLOR is NP-complete.

Chapter notes

Garey and Johnson [79] provide a wonderful guide to NP-completeness, discussing the theory at length and providing a catalogue of many problems that were known to be NP-complete in 1979. (The list of NP-complete problem domains at the beginning of Section 36.5 is drawn from their table of contents.) Hopcroft and Ullman [104] and Lewis and Papadimitriou [139] have good treatments of NP-completeness in the context of complexity theory. Aho, Hopcroft, and Ullman [4] also cover NP-completeness and give several reductions, including a reduction for the vertex-cover problem from the hamiltonian-cycle problem.

The class P was introduced in 1964 by Cobham [44] and, independently, in 1965 by Edmonds [61], who also introduced the class NP and conjectured that P NP. The notion of NP-completeness was proposed in 1971 by Cook [49], who gave the first NP-completeness proofs for formula satisfiability and 3-CNF satisfiability. Levin [138] independently discovered the notion, giving an NP-completeness proof for a tiling problem. Karp [116] introduced the methodology of reductions in 1972 and demonstrated the rich variety of NP-complete problems. Karp's paper included the original NP-completeness proofs of the clique, vertex-cover, and hamiltonian-cycle problems. Since then, hundreds of problems have been proven to be NP-complete by many researchers.

The proof of Theorem 36.14 was adapted from Papadimitriou and Steiglitz [154].

Go to Chapter 37 Back to Table of Contents


Intro to Algorithms: CHAPTER 37: APPROXIMATION ALGORITHMS

 Return to Table of Contents

CHAPTER 37: APPROXIMATION ALGORITHMS

Many problems of practical significance are NP-complete but are too important to abandon merely because obtaining an optimal solution is intractable. If a problem is NP-complete, we are unlikely to find a polynomial-time algorithm for solving it exactly, but this does not imply that all hope is lost. There are two approaches to getting around NP-completeness. First, if the actual inputs are small, an algorithm with exponential running time may be perfectly satisfactory. Second, it may still be possible to find near-optimal solutions in polynomial time (either in the worst case or on the average). In practice, near-optimality is often good enough. An algorithm that returns near-optimal solutions is called an approximation algorithm. This chapter presents polynomial-time approximation algorithms for several NP-complete problems.

Performance bounds for approximation algorithms

Assume that we are working on an optimization problem in which each potential solution has a positive cost, and that we wish to find a near-optimal solution. Depending on the problem, an optimal solution may be defined as one with maximum possible cost or one with minimum possible cost; the problem may be a maximization or a minimization problem.

We say that an approximation algorithm for the problem has a ratio bound of [image] (n) if for any input of size n, the cost C of the solution produced by the approximation algorithm is within a factor of [image] (n) of the cost C* of an optimal solution:

(37.1)

This definition applies for both minimization and maximization problems. For a maximization problem, 0 < C [image] C*, and the ratio C*/C gives the factor by which the cost of an optimal solution is larger than the cost of the approximate solution. Similarly, for a minimization problem, 0 < C* [image] C, and the ratio C/C* gives the factor by which the cost of the approximate solution is larger than the cost of an optimal solution. Since all solutions are assumed to have positive cost, these ratios are always well defined. The ratio bound of an approximation algorithm is never less than 1, since C/C* < 1 implies C*/C > 1. An optimal algorithm has ratio bound 1, and an approximation algorithm with a large ratio bound may return a solution that is very much worse than optimal.

Sometimes, it is more convenient to work with a measure of relative error. For any input, the relative error of the approximation algorithm is defined to be

where, as before, C* is the cost of an optimal solution and C is the cost of the solution produced by the approximation algorithm. The relative error is always nonnegative. An approximation algorithm has a relative error bound of [image] (n) if

(37.2)

It follows from the definitions that the relative error bound can be bounded as a function of the ratio bound:

   (n)      (n) - 1.

(37.3)

(For a minimization problem, this is an equality, whereas for a maximization problem, we have [image] (n) = ( [image] (n) - 1) / [image] (n), which satisfies inequality (37.3) since [image] (n) [image] 1.)

For many problems, approximation algorithms have been developed that have a fixed ratio bound, independent of n. For such problems, we simply use the notation [image] or [image] , indicating no dependence on n.

For other problems, computer scientists have been unable to devise any polynomial-time approximation algorithm having a fixed ratio bound. For such problems, the best that can be done is to let the ratio bound grow as a function of the input size n. An example of such a problem is the set-cover problem presented in Section 37.3.

Some NP-complete problems allow approximation algorithms that can achieve increasingly smaller ratio bounds (or, equivalently, increasingly smaller relative error bounds) by using more and more computation time. That is, there is a trade-off between computation time and the quality of the approximation. An example is the subset-sum problem studied in Section 37.4. This situation is important enough to deserve a name of its own.

An approximation scheme for an optimization problem is an approximation algorithm that takes as input not only an instance of the problem, but also a value [image] > 0 such that for any fixed [image] , the scheme is an approximation algorithm with relative error bound [image] . We say that an approximation scheme is a polynomial-time approximation scheme if for any fixed [image] > 0, the scheme runs in time polynomial in the size n of its input instance.

The running time of a polynomial-time approximation scheme should not increase too rapidly as [image] decreases. Ideally, if [image] decreases by a constant factor, the running time to achieve the desired approximation should not increase by more than a constant factor. In other words, we would like the running time to be polynomial in 1/ [image] as well as in n.

We say that an approximation scheme is a fully polynomial-time approximation scheme if its running time is polynomial both in 1/ [image] and in the size n of the input instance, where [image] is the relative error bound for the scheme. For example, the scheme might have a running time of (1/ [image] )2n3. With such a scheme, any constant-factor decrease in [image] can be achieved with a corresponding constant-factor increase in the running time.

Chapter outline

The first three sections of this chapter present some examples of polynomial-time approximation algorithms for NP-complete problems, and the last section presents a fully polynomial-time approximation scheme. Section 37.1 begins with a study of the vertex-cover problem, an NP-complete minimization problem that has an approximation algorithm with a ratio bound of 2. Section 37.2 presents an approximation algorithm with ratio bound 2 for the case of the traveling-salesman problem in which the cost function satisfies the triangle inequality. It also shows that without triangle inequality, an [image] -approximation algorithm cannot exist unless P = NP. In Section 37.3, we show how a greedy method can be used as an effective approximation algorithm for the set-covering problem, obtaining a covering whose cost is at worst a logarithmic factor larger than the optimal cost. Finally, Section 37.4 presents a fully polynomial-time approximation scheme for the subset-sum problem.

37.1 The vertex-cover problem

The vertex-cover problem was defined and proved NP-complete in Section 36.5.2. A vertex cover of an undirected graph G = (V,E) is a subset V' [image] V such that if (u,v) is an edge of G, then either u [image] V' or v [image] V' (or both). The size of a vertex cover is the number of vertices in it.

The vertex-cover problem is to find a vertex cover of minimum size in a given undirected graph. We call such a vertex cover an optimal vertex cover. This problem is NP-hard, since the related decision problem is NP-complete, by Theorem 36.12.

Even though it may be difficult to find an optimal vertex cover in a graph G, however, it is not too hard to find a vertex cover that is near-optimal. The following approximation algorithm takes as input an undirected graph G and returns a vertex cover whose size is guaranteed to be no more than twice the size of an optimal vertex cover.

Figure 37.1 The operation of APPROX-VERTEX-COVER. (a) The input graph G, which has 7 vertices and 8 edges. (b) The edge (b,c), shown heavy, is the first edge chosen by APPROX-VERTEX-COVER. Vertices b and c, shown lightly shaded, are added to the set A containing the vertex cover being created. Edges (a, b), (c, e), and (c,d), shown dashed, are removed since they are now covered by some vertex in A. (c) Edge (e, f) is added to A. (d) Edge (d,g) is added to A. (e) The set A, which is the vertex cover produced by APPROX-VERTEX-COVER, contains the six vertices b, c, d, e, f, g. (f) The optimal vertex cover for this problem contains only three vertices: b, d, and e.

Figure 37.1 illustrates the operation of APPROX-VERTEX-COVER. The variable C contains the vertex cover being constructed. Line 1 initializes C to the empty set. Line 2 sets E' to be a copy of the edge set E[G] of the graph. The loop on lines 3-6 repeatedly picks an edge (u, v) from E', adds its endpoints u and v to C, and deletes all edges in E' that are covered by either u or v. The running time of this algorithm is O(E), using an appropriate data structure for representing E'.

Theorem 37.1

APPROX-VERTEX-COVER has a ratio bound of 2.

Proof The set C of vertices that is returned by APPROX-VERTEX-COVER is a vertex cover, since the algorithm loops until every edge in E[G] has been covered by some vertex in C.

To see that APPROX-VERTEX-COVER returns a vertex cover that is at most twice the size of an optimal cover, let A denote the set of edges that were picked in line 4 of APPROX-VERTEX-COVER. No two edges in A share an endpoint, since once an edge is picked in line 4, all other edges that are incident on its endpoints are deleted from E' in line 6. Therefore, each execution of line 5 adds two new vertices to C, and |C| = 2 |A|. In order to cover the edges in A, however, any vertex cover--in particular, an optimal cover C*--must include at least one endpoint of each edge in A. Since no two edges in A share an endpoint, no vertex in the cover is incident on more than one edge in A. Therefore, |A| [image] |C*|, and |C| [image] 2 |C*|, proving the theorem.

Exercises

37.1-1

Given an example of a graph for which APPROX-VERTEX-COVER always yields a suboptimal solution.

37.1-2

Professor Nixon proposes the following heuristic to solve the vertex-cover problem. Repeatedly select a vertex of highest degree, and remove all of its incident edges. Give an example to show that the professor's heuristic does not have a ratio bound of 2.

37.1-3

Give an efficient greedy algorithm that finds an optimal vertex cover for a tree in linear time.

37.1-4

From the proof of Theorem 36.12, we know that the vertex-cover problem and the NP-complete clique problem are complementary in the sense that an optimal vertex cover is the complement of a maximum-size clique in the complement graph. Does this relationship imply that there is an approximation algorithm with constant ratio bound for the clique problem? Justify your answer.

37.2 The traveling-salesman problem

In the traveling-salesman problem introduced in Section 36.5.5, we are given a complete undirected graph G = (V, E) that has a nonnegative integer cost c(u, v) associated with each edge (u, v) [image] E, and we must find a hamiltonian cycle (a tour) of G with minimum cost. As an extension of our notation, let c(A) denote the total cost of the edges in the subset A [image] E:

In many practical situations, it is always cheapest to go directly from a place u to a place w; going by way of any intermediate stop v can't be less expensive. Putting it another way, cutting out an intermediate stop never increases the cost. We formalize this notion by saying that the cost function c satisfies the triangle inequality if for all vertices u, v, w [image] V,

c(u, w)    c(u, v) + c(v, w).

The triangle inequality is a natural one, and in many applications it is automatically satisfied. For example, if the vertices of the graph are points in the plane and the cost of traveling between two vertices is the ordinary euclidean distance between them, then the triangle inequality is satisfied.

As Exercise 37.2-1 shows, restricting the cost function to satisfy the triangle inequality does not alter the NP-completeness of the traveling- salesman problem. Thus, it is unlikely that we can find a polynomial-time algorithm for solving this problem exactly. We therefore look instead for good approximation algorithms.

In Section 37.2.1, we examine an approximation algorithm for the traveling-salesman problem with triangle inequality that has a ratio bound of 2. In Section 37.2.2, we show that without triangle inequality, an approximation algorithm with constant ratio bound does not exist unless P = NP.

37.2.1 The traveling-salesman problem with triangle inequality

The following algorithm computes a near-optimal tour of an undirected graph G, using the minimum-spanning-tree algorithm MST-PRIM from Section 24.2. We shall see that when the cost function satisfies the triangle inequality, the tour that this algorithm returns is no worse than twice as long as an optimal tour.

APPROX-TSP-TOUR(G, c)

1 select a vertex r    V[G] to be a "root" vertex

2 grow a minimum spanning tree T for G from root r

using MST-PRIM(G, c, r)

3 let L be the list of vertices visited in a preorder tree walk of T

4 return the hamiltonian cycle H that visits the vertices in the order L

Recall from Section 13.1 that a preorder tree walk recursively visits every vertex in the tree, listing a vertex when it is first encountered, before any of its children are visited.

Figure 37.2 illustrates the operation of APPROX-TSP-TOUR. Part (a) of the figure shows the given set of vertices, and part (b) shows the minimum spanning tree T grown from root vertex a by MST-PRIM. Part (c) shows how the vertices are visited by a preorder walk of T, and part (d) displays the corresponding tour, which is the tour returned by APPROX-TSP-TOUR. Part (e) displays an optimal tour, which is about 23% shorter.

The running time of APPROX-TSP-TOUR is [image] (E) = [image] (V2), since the input is a complete graph (see Exercise 24.2-2). We shall now show that if the cost function for an instance of the traveling-salesman problem satisfies the triangle inequality, then APPROX-TSP-TOUR returns a tour whose cost is not more than twice the cost of an optimal tour.

Theorem 37.2

APPROX-TSP-TOUR is an approximation algorithm with a ratio bound of 2 for the traveling-salesman problem with triangle inequality.

Proof Let H* denote an optimal tour for the given set of vertices. An equivalent statement of the theorem is that c(H) [image] 2c(H*), where H is the tour returned by APPROX-TSP-TOUR. Since we obtain a spanning tree by deleting any edge from a tour, if T is a minimum spanning tree for the given set of vertices, then

c(T)    c(H*) .

(37.4)

A full walk of T lists the vertices when they are first visited and also whenever they are returned to after a visit to a subtree. Let us call this walk W. The full walk of our example gives the order

a, b, c, b, h, b, a, d, e, f, e, g, e, d, a.

Since the full walk traverses every edge of T exactly twice, we have

c(W) = 2c(T) .

(37.5)

Equations (37.4) and (37.5) imply that

c(W)    2c(H*),

(37.6)

and so the cost of W is within a factor of 2 of the cost of an optimal tour.

Figure 37.2 The operation of APPROX-TSP-TOUR. (a) The given set of points, which lie on vertices of an integer grid. For example, f is one unit to the right and two units up from h. The ordinary euclidean distance is used as the cost function between two points. (b) A minimum spanning tree T of these points, as computed by MST-PRIM. Vertex a is the root vertex. The vertices happen to be labeled in such a way that they are added to the main tree by MST-PRIM in alphabetical order. (c) A walk of T, starting at a. A full walk of the tree visits the vertices in the order a, b, c, b, h, b, a, d, e, f, e, g, e, d, a. A preorder walk of T lists a vertex just when it is first encountered, yielding the ordering a, b, c, h, d, e, f, g. (d) A tour of the vertices obtained by visiting the vertices in the order given by the preorder walk. This is the tour H returned by APPROX-TSP-TOUR. Its total cost is approximately 19.074. (e) An optimal tour H* for the given set of vertices. Its total cost is approximately 14.715.

Unfortunately, W is generally not a tour, since it visits some vertices more than once. By the triangle inequality, however, we can delete a visit to any vertex from W and the cost does not increase. (If a vertex v is deleted from W between visits to u and w, the resulting ordering specifies going directly from u to w.) By repeatedly applying this operation, we can remove from W all but the first visit to each vertex. In our example, this leaves the ordering

a, b, c, h, d, e, f, g .

This ordering is the same as that obtained by a preorder walk of the tree T. Let H be the cycle corresponding to this preorder walk. It is a hamiltonian cycle, since every vertex is visited exactly once, and in fact it is the cycle computed by APPROX-TSP-TOUR. Since H is obtained by deleting vertices from the full walk W, we have

c(H)    c(W).

(37.7)

Combining inequalities (37.6) and (37.7) completes the proof.

In spite of the nice ratio bound provided by Theorem 37.2, APPROX-TSP-TOUR is usually not the best practical choice for this problem. There are other approximation algorithms that typically perform much better in practice (see the references at the end of this chapter).

37.2.2 The general traveling-salesman problem

If we drop the assumption that the cost function c satisfies the triangle inequality, good approximate tours cannot be found in polynomial time unless P = NP.

Theorem 37.3

If P [image] NP and [image] [image] 1, there is no polynomial-time approximation algorithm with ratio bound [image] for the general traveling-salesman problem.

Proof The proof is by contradiction. Suppose to the contrary that for some number [image] [image] 1, there is a polynomial-time approximation algorithm A with ratio bound [image]. Without loss of generality, we assume that [image] is an integer, by rounding it up if necessary. We shall then show how to use A to solve instances of the hamiltonian-cycle problem (defined in Section 36.5.5) in polynomial time. Since the hamiltonian-cycle problem is NP-complete, by Theorem 36.14, solving it in polynomial time implies that P = NP, by Theorem 36.4.

Let G = (V, E) be an instance of the hamiltonian-cycle problem. We wish to determine efficiently whether G contains a hamiltonian cycle by making use of the hypothesized approximation algorithm A. We turn G into an instance of the traveling-salesman problem as follows. Let G' = (V, E') be the complete graph on V; that is,

E' = {(u, v): u, v    V and u    v}.

Assign an integer cost to each edge in E' as follows:

Representations of G' and c can be created from a representation of G in time polynomial in |V| and |E|.

Now, consider the traveling-salesman problem (G', c). If the originalgraph G has a hamiltonian cycle H, then the cost function c assigns to each edge of H a cost of 1, and so (G', c) contains a tour of cost |V| On the other hand, if G does not contain a hamiltonian cycle, then any tour of G' must use some edge not in E. But any tour that uses an edge not in E has a cost of at least

(   |V| + 1) + (|V| - 1) >    |V| .

Because edges not in G are so costly, there is a large gap between the cost of a tour that is a hamiltonian cycle in G (cost |V|) and the cost of any other tour (cost greater than [image] |V|).

What happens if we apply the approximation algorithm A to the traveling-salesman problem (G', c)? Because A is guaranteed to return a tour of cost no more than [image] times the cost of an optimal tour, if G contains a hamiltonian cycle, then A must return it. If G has no hamiltonian cycle, then A returns a tour of cost more than [image] |V|. Therefore, we can use A to solve the hamiltonian-cycle problem in polynomial time.

Exercises

37.2-1

Show how in polynomial time we can transform one instance of the traveling-salesman problem into another instance whose cost function satisfies the triangle inequality. The two instances must have the same set of optimal tours. Explain why such a polynomial-time transformation does not contradict Theorem 37.3, assuming that P [image] NP.

37.2-2

Consider the following closest-point heuristic for building an approximate traveling-salesman tour. Begin with a trivial cycle consisting of a single arbitrarily chosen vertex. At each step, identify the vertex u that is not on the cycle but whose distance to any vertex on the cycle is minimum. Suppose that the vertex on the cycle that is nearest u is vertex v. Extend the cycle to include u by inserting u just after v. Repeat until all vertices are on the cycle. Prove that this heuristic returns a tour whose total cost is not more than twice the cost of an optimal tour.

37.2-3

The bottleneck traveling-salesman problem is the problem of finding the hamiltonian cycle such that the length of the longest edge in the cycle is minimized. Assuming that the cost function satisfies the triangle inequality, show that there exists a polynomial-time approximation algorithm with ratio bound 3 for this problem. (Hint: Show recursively that we can visit all the nodes in a minimum spanning tree exactly once by taking a full walk of the tree and skipping nodes, but without skipping more than 2 consecutive intermediate nodes.)

37.2-4

Suppose that the vertices for an instance of the traveling-salesman problem are points in the plane and that the cost c(u, v) is the euclidean distance between points u and v. Show that an optimal tour never crosses itself.

37.3 The set-covering problem

The set-covering problem is an optimization problem that models many resource-selection problems. It generalizes the NP-complete vertex-cover problem and is therefore also NP-hard. The approximation algorithm developed to handle the vertex-cover problem doesn't apply here, however, and so we need to try other approaches. We shall examine a simple greedy heuristic with a logarithmic ratio bound. That is, as the size of the instance gets larger, the size of the approximate solution may grow, relative to the size of an optimal solution. Because the logarithm function grows rather slowly, however, this approximation algorithm may nonetheless give useful results.

An instance [image] of the set-covering problem consists of a finite set X and a family [image] of subsets of X, such that every element of X belongs to at least one subset in [image]:

We say that a subset [image] covers its elements. The problem is to find a minimum-size subset [image] whose members cover all of X:

(37.8)

We say that any [image] satisfying equation (37.8) covers X. Figure 37.3 illustrates the problem.

The set-covering problem is an abstraction of many commonly arising combinatorial problems. As a simple example, suppose that X represents a set of skills that are needed to solve a problem and that we have a given set of people available to work on the problem. We wish to form a committee, containing as few people as possible, such that for every requisite skill in X, there is a member of the committee having that skill. In the decision version of the set-covering problem, we ask whether or not a covering exists with size at most k, where k is an additional parameter specified in the problem instance. The decision version of the problem is NP-complete, as Exercise 37.3-2 asks you to show.

Figure 37.3 An instance of the set-covering problem, where X consists of the 12 black points and . A minimum-size set cover is . The greedy algorithm produces a cover of size 4 by selecting the sets S1,S4,S5, and S3 in order.

A greedy approximation algorithm

The greedy method works by picking, at each stage, the set S that covers the most remaining uncovered elements.

In the example of Figure 37.3, GREEDY-SET-COVER adds to [image] the sets Sl, S4, S5, S3 in order.

The algorithm works as follows. The set U contains, at each stage, the set of remaining uncovered elements. The set [image] contains the cover being constructed. Line 4 is the greedy decision-making step. A subset S is chosen that covers as many uncovered elements as possible (with ties broken arbitrarily). After S is selected, its elements are removed from U, and S is placed in [image]. When the algorithm terminates, the set [image] contains a subfamily of [image] that covers X.

The algorithm GREEDY-SET-COVER can easily be implemented to run in time polynomial in [image]. Since the number of iterations of the loop on lines 3-6 is at most min [image], and the loop body can be implemented to run in time [image], there is an implementation that runs in time [image]. Exercise 37.3-3 asks for a linear- time algorithm.

Analysis

We now show that the greedy algorithm returns a set cover that is not too much larger than an optimal set cover. For convenience, in this chapter we denote the dth harmonic number [image] (see Section 3.1) by H(d) .

Theorem 37.4

GREEDY-SET-COVER has a ratio bound

Proof The proof proceeds by assigning a cost to each set selected by the algorithm, distributing this cost over the elements covered for the first time, and then using these costs to derive the desired relationship between the size of an optimal set cover [image] and the size of the set cover [image] returned by the algorithm. Let Si denote the ith subset selected by GREEDY-SET-COVER; the algorithm incurs a cost of 1 when it adds Si to [image]. We spread this cost of selecting Si evenly among the elements covered for the first time by Si. Let cx denote the cost allocated to element x, for each x [image] X. Each element is assigned a cost only once, when it is covered for the first time. If x is covered for the first time by Si, then

The algorithm finds a solution [image] of total cost [image], and this cost has been spread out over the elements of X. Therefore, since the optimal cover [image] also covers X, we have

(37.9)

The remainder of the proof rests on the following key inequality, which we shall prove shortly. For any set S belonging to the family [image],

(37.10)

From inequalities (37.9) and (37.10), it follows that

proving the theorem. It thus remains only to prove inequality (37.10). For any set [image], let

be the number of elements in S remaining uncovered after S1, S2, . . . , Si have been selected by the algorithm. We define u0 = |S| to be the number of elements of S, which are all initially uncovered. Let k be the least index such that uk = 0, so that every element in S is covered by at least one of the sets S1, S2, . . . , Sk. Then, ui-1 [image] ui, and ui-1 - ui elements of S are covered for the first time by Si, for i = 1, 2, . . . , k. Thus,

Observe that

because the greedy choice of Si guarantees that S cannot cover more new elements than Si does (otherwise, S would have been chosen instead of Si). Consequently, we obtain

For integers a and b, where a < b, we have

Using this inequality, we obtain the telescoping sum

since H(0) = 0. This completes the proof of inequality (37.10).

Corollary 37.5

GREEDY-SET-COVER has a ratio bound of (ln |X| + 1).

Proof Use inequality (3.12) and Theorem 37.4.

In some applications, max [image] is a small constant, and so the solution returned by GREEDY-SET-COVER is at most a small constant times larger than optimal. One such application occurs when this heuristic is used to obtain an approximate vertex cover for a graph whose vertices have degree at most 3. In this case, the solution found by GREEDY-SET-COVER is not more than H(3) = 11/6 times as large as an optimal solution, a performance guarantee that is slightly better than that of APPROX-VERTEX-COVER.

Exercises

37.3-1

Consider each of the following words as a set of letters: {arid, dash, drain, heard, lost, nose, shun, slate, snare, thread}. Show which set cover GREEDY-SET-COVER produces when ties are broken in favor of the word that appears first in the dictionary.

37.3-2

Show that the decision version of the set-covering problem is NP-complete by reduction from the vertex-cover problem.

37.3-3

Show how to implement GREEDY-SET-COVER in such a way that it runs in time [image].

37.3-4

Show that the following weaker form of Theorem 37.4 is trivially true:

37.3-5

Create a family of set-cover instances demonstrating that GREEDY-SET-COVER can return a number of different solutions that is exponential in the size of the instance. (Different solutions result from ties being broken differently in the choice of S in line 4.)

37.4 The subset-sum problem

An instance of the subset-sum problem is a pair (S, t), where S is a set {x1, x2, . . . , xn} of positive integers and t is a positive integer. This decision problem asks whether there exists a subset of S that adds up exactly to the target value t. This problem is NP-complete (see Section 36.5.3).

The optimization problem associated with this decision problem arises in practical applications. In the optimization problem, we wish to find a subset of {x1, x2, . . . , xn} whose sum is as large as possible but not larger than t. For example, we may have a truck that can carry no more than t pounds, and n different boxes to ship, the ith of which weighs xi pounds. We wish to fill the truck as full as possible without exceeding the given weight limit.

In this section, we present an exponential-time algorithm for this optimization problem and then show how to modify the algorithm so that it becomes a fully polynomial-time approximation scheme. (Recall that a fully polynomial-time approximation scheme has a running time that is polynomial in 1/ [image] as well as in n.)

An exponential-time algorithm

If L is a list of positive integers and x is another positive integer, then we let L + x denote the list of integers derived from L by increasing each element of L by x. For example, if L = [image] 1, 2, 3, 5, 9 [image], then L + 2 = [image] 3, 4, 5, 7, 11 [image]. We also use this notation for sets, so that

S + x = {s + x : s    S} .

We use an auxiliary procedure MERGE-LISTS(L, L') that returns the sorted list that is the merge of its two sorted input lists L and L'. Like the MERGE procedure we used in merge sort (Section 1.3.1), MERGE-LISTS runs in time O(|L| + |L'|). (We omit giving pseudocode for MERGE-LISTS.) The procedure EXACT-SUBSET-SUM takes an input set S = {x1, x2, . . . , xn} and a target value t.

EXACT-SUBSET-SUM(S, t)

1  n    |S|

2  L0      0

3  for i    1 to n

4  do Li    MERGE-LISTS(Li-1, Li-1 + xi)

5  remove from Li every element that is greater than t

6  return the largest element in Ln

Let Pi denote the set of all values that can be obtained by selecting a (possibly empty) subset of {x1, x2, . . . , xi} and summing its members. For example, if S = {1, 4, 5}, then

P1 = {0, 1} ,

P2 = {0, 1, 4, 5} ,

P3 = {0, 1, 4, 5, 6, 9, 10} .

Given the identity

(37.11)

we can prove by induction on i (see Exercise 37.4-1) that the list Li is a sorted list containing every element of Pi whose value is not more than t. Since the length of Li can be as much as 2i, EXACT-SUBSET-SUM is an exponential-time algorithm in general, although it is a polynomial-time algorithm in the special cases in which t is polynomial in |S| or all of the numbers in S are bounded by a polynomial in |S|.

A fully polynomial-time approximation scheme

We can derive a fully polynomial-time approximation scheme for the subset-sum problem by "trimming" each list Li after it is created. We use a trimming parameter [image] such that 0 < [image] < 1. To trim a list L by [image] means to remove as many elements from L as possible, in such a way that if L'is the result of trimming L, then for every element y that was removed from L, there is an element z [image] y still in L' such that

or, equivalently,

(1 -   )y    z    y .

We can think of such a z as "representing" y in the new list L'. Each y is represented by a z such that the relative error of z, with respect to y, is at most [image]. For example, if [image] = 0.1 and

L =   10, 11, 12, 15, 20, 21, 22, 23, 24, 29  ,

then we can trim L to obtain

L' =   10, 12, 15, 20, 23, 29  ,

where the deleted value 11 is represented by 10, the deleted values 21 and 22 are represented by 20, and the deleted value 24 is represented by 23. It is important to remember that every element of the trimmed version of the list is also an element of the original version of the list. Trimming a list can dramatically decrease the number of elements in the list while keeping a close (and slightly smaller) representative value in the list for each element deleted from the list.

The following procedure trims an input list L = [image] y1, y2, . . . , ym [image] in time [image] (m), assuming that L is sorted into nondecreasing order. The output of the procedure is a trimmed, sorted list.

TRIM(L,   )

1  m      |L|

2  L'    <y1>

3  last    <y1>

4  for i    2 to m

5       do if last < (1 -   )yi

6             then append yi onto the end of L'

7                  last    yi

8  return L'

The elements of L are scanned in increasing order, and a number is put into the returned list L' only if it is the first element of L or if it cannot be represented by the most recent number placed into L'.

Given the procedure TRIM, we can construct our approximation scheme as follows. This procedure takes as input a set S = {x1, x2, . . . , xn} of n integers (in arbitrary order), a target integer t, and an "approximation parameter" [image] , where 0 < [image] < 1.

APPROX-SUBSET-SUM(S, t,   )

1  n    |S|

2  L0      0

3  for i    1 to n

4       do Li    MERGE-LISTS(Li-1, Li-1 + xi)

5          Li    TRIM(Li,    / n)

6          remove from Li every element that is greater than t

7  let z be the largest value in Ln

8  return z

Line 2 initializes the list L0 to be the list containing just the element 0. The loop in lines 3-6 has the effect of computing Li as a sorted list containing a suitably trimmed version of the set Pi, with all elements larger than t removed. Since Li is created from Li-1, we must ensure that the repeated trimming doesn't introduce too much inaccuracy. In a moment, we shall see that APPROX-SUBSET-SUM returns a correct approximation if one exists.

As an example, suppose we have the instance

L =   104, 102, 201, 101

with t = 308 and [image] = 0.20. The trimming parameter [image] is [image] / 4 = 0.05. APPROX-SUBSET-SUM computes the following values on the indicated lines:

line 2:  L0  =    0   ,

line 4:  L1  =    0, 104   ,

line 5:  L1  =    0, 104   ,

line 6:  L1  =    0, 104   ,

line 4:  L2  =    0, 102, 104, 206   ,

line 5:  L2  =    0, 102, 206   ,

line 6:  L2  =    0, 102, 206   ,

line 4:  L3  =    0,102, 201, 206, 303, 407   ,

line 5:  L3  =    0,102, 201, 303, 407   ,

line 6:  L3  =    0, 102, 201, 303   ,

line 4:  L4  =    0, 101, 102, 201, 203, 302, 303, 404   ,

line 5:  L4  =    0,101, 201, 302, 404   ,

line 6:  L4  =    0,101, 201, 302   .

The algorithm returns z = 302 as its answer, which is well within [image] = 20% of the optimal answer 307 = 104 + 102 + 101; in fact, it is within 2%.

Theorem 37.6

APPROX-SUBSET-SUM is a fully polynomial-time approximation scheme for the subset-sum problem.

Proof The operations of trimming Li in line 5 and removing from Li every element that is greater than t maintain the property that every element of Li is also a member of Pi. Therefore, the value z returned in line 8 is indeed the sum of some subset of S. It remains to show that it is not smaller than 1 - [image] times an optimal solution. (Note that because the subset-sum problem is a maximization problem, equation (37.2) is equivalent to C*(1- [image] ) [image] C.) We must also show that the algorithm runs in polynomial time.

To show that the relative error of the returned answer is small, note that when list Li is trimmed, we introduce a relative error of at most [image] /n between the representative values remaining and the values before trimming. By induction on i, it can be shown that for every element y in Pi that is at most t, there is a z [image] Li such that

(1 -   /n)iy    z    y.

(37.12)

If y* [image] Pn denotes an optimal solution to the subset-sum problem, then there is a z [image] Ln such that

(1 -   /n)ny*    z    y*;

(37.13)

the largest such z is the value returned by APPROX-SUBSET-SUM. Since it can be shown that

the function (1 - [image] /n)n increases with n, so that n > 1 implies

1 -    <(1 -   /n)n,

and thus,

(1 -   )y*    z.

Therefore, the value z returned by APPROX-SUBSET-SUM is not smaller than 1 - [image] times the optimal solution y*.

To show that this is a fully polynomial-time approximation scheme, we derive a bound on the length of Li. After trimming, successive elements z and z' of Li must have the relationship z/z' > 1/(1 - [image] /n). That is, they must differ by a factor of at least 1/(1 - [image] /n). Therefore, the number of elements in each Li is at most

using equation (2.10). This bound is polynomial in the number n of input values given, in the number of bits 1g t needed to represent t, and in 1/ [image] . Since the running time of APPROX-SUBSET-SUM is polynomial in the length of the Li, APPROX-SUBSET-SUM is a fully polynomial-time approximation scheme.

Exercises

37.4-1

Prove equation (37.11).

37.4-2

Prove equations (37.12) and (37.13).

37.4-3

How would you modify the approximation scheme presented in this section to find a good approximation to the smallest value not less than t that is a sum of some subset of the given input list?

Problems

37-1 Bin packing

Suppose that we are given a set of n objects, where the the size si of the ith object satisfies 0 < si < 1. We wish to pack all the objects into the minimum number of unit-size bins. Each bin can hold any subset of the objects whose total size does not exceed 1.

a. Prove that the problem of determining the minimum number of bins required is NP-hard. (Hint: Reduce from the subset-sum problem.)

The first-fit heuristic takes each object in turn and places it into the first bin that can accommodate it. Let

.

b. Argue that the optimal number of bins required is at least [image] S [image] .

c. Argue that the first-fit heuristic leaves at most one bin less than half full.

d. Prove that the number of bins used by the first-fit heuristic is never more than [image] 2S [image] .

e. Prove a ratio bound of 2 for the first-fit heuristic.

f. Give an efficient implementation of the first-fit heuristic, and analyze its running time.

37-2 Approximating the size of a maximum clique

Let G = (V, E) be an undirected graph. For any k [image] 1, define G(k) to be the undirected graph (V(k), E(k), where V(k) is the set of all ordered k-tuples of vertices from V and E(k) is defined so that (v1,v2, . . . , vk) is adjacent to (w1, w2, . . . , wk) if and only if for some i, vertex vi is adjacent to wi in G.

a. Prove that the size of the maximum clique in G(k) is equal to the kth power of the size of the maximum clique in G.

b. Argue that if there is an approximation algorithm that has a constant ratio bound for finding a maximum-size clique, then there is a fully polynomial-time approximation scheme for the problem.

37-3 Weighted set-covering problem

Suppose that we generalize the set-covering problem so that each set Si in the family [image] has an associated weight wi and the weight of a cover [image] is [image] Si [image] cwi. We wish to determine a minimum-weight cover. (Section 37.3 handles the case in which wi = 1 for all i.)

Show that the greedy set-covering heuristic can be generalized in a natural manner to provide an approximate solution for any instance of the weighted set-covering problem. Show that your heuristic has a ratio bound of H(d), where d is the maximum size of any set Si.

Chapter notes

There is a wealth of literature on approximation algorithms. A good place to start is Garey and Johnson [79]. Papadimitriou and Steiglitz [154] also have an excellent presentation of approximation algorithms. Lawler, Lenstra, Rinnooy Kan, and Shmoys [133] provide an extensive treatment of the traveling-salesman problem.

Papadimitriou and Steiglitz attribute the algorithm APPROX-VERTEX-COVER to F. Gavril and M. Yannakakis. The algorithm APPROX-TSP-TOUR appears in an excellent paper by Rosenkrantz, Stearns, and Lewis [170]. Theorem 37.3 is due to Sahni and Gonzalez [172]. The analysis of the greedy heuristic for the set-covering problem is modeled after the proof published by Chvátal [42] of a more general result; this basic result as presented here is due to Johnson [113] and Lovász [141]. The algorithm APPROX-SUBSET-SUM and its analysis are loosely modeled after related approximation algorithms for the knapsack and subset-sum problem by Ibarra and Kim [111].

Go to Bibliography Back to Table of Contents


Intro to Algorithms: PART I: Mathematical Foundations

 Return to Table of Contents

PART I: Mathematical Foundations

Introduction

The analysis of algorithms often requires us to draw upon a body of mathematical tools. Some of these tools are as simple as high-school algebra, but others, such as solving recurrences, may be new to you. This part of the book is a compendium of the methods and tools we shall use to analyze algorithms. It is organized primarily for reference, with a tutorial flavor to some of the topics.

We suggest that you not try to digest all of this mathematics at once. Skim the chapters in this part to see what material they contain. You can then proceed directly to the chapters that focus on algorithms. As you read those chapters, though, refer back to this part whenever you need a better understanding of the tools used in the mathematical analyses. At some point, however, you will want to study each of these chapters in its entirety, so that you have a firm foundation in the mathematical techniques.

Chapter 2 precisely defines several asymptotic notations, an example of which is the [image] -notation that you met in Chapter 1. The rest of Chapter 2 is primarily a presentation of mathematical notation. Its purpose is more to ensure that your use of notation matches that in this book than to teach you new mathematical concepts.

Chapter 3 offers methods for evaluating and bounding summations, which occur frequently in the analysis of algorithms. Many of the formulas in this chapter can be found in any calculus text, but you will find it convenient to have these methods compiled in one place.

Methods for solving recurrences, which we used to analyze merge sort in Chapter 1 and which we shall see many times again, are given in Chapter 4. One powerful technique is the "master method," which can be used to solve recurrences that arise from divide-and-conquer algorithms. Much of Chapter 4 is devoted to proving the correctness of the master method, though this proof may be skipped without harm.

Chapter 5 contains basic definitions and notations for sets, relations, functions, graphs, and trees. This chapter also gives some basic properties of these mathematical objects. This material is essential for an understanding of this text but may safely be skipped if you have already had a discrete mathematics course.

Chapter 6 begins with elementary principles of counting: permutations, combinations, and the like. The remainder of the chapter contains definitions and properties of basic probability. Most of the algorithms in this book require no probability for their analysis, and thus you can easily omit the latter sections of the chapter on a first reading, even without skimming them. Later, when you encounter a probabilistic analysis that you want to understand better, you will find Chapter 6 well organized for reference purposes.

Go to Chapter 2 Back to Table of Contents


Intro to Algorithms: PART II: Sorting and Order Statistics

 Return to Table of Contents

PART II: Sorting and Order Statistics

Introduction

This part presents several algorithms that solve the following sorting problem:

Input: A sequence of n numbers <a1, a2, . . . , an>.

Output: A permutation (reordering) <a'1, a'2, . . . , a'n> of the input sequence such that a'1 [image] a'2 [image] [image] [image] [image] [image] a'n.

The input sequence is usually an n-element array, although it may be represented in some other fashion, such as a linked list.

The structure of the data

In practice, the numbers to be sorted are rarely isolated values. Each is usually part of a collection of data called a record. Each record contains a key, which is the value to be sorted, and the remainder of the record consists of satellite data, which are usually carried around with the key. In practice, when a sorting algorithm permutes the keys, it must permute the satellite data as well. If each record includes a large amount of satellite data, we often permute an array of pointers to the records rather than the records themselves in order to minimize data movement.

In a sense, it is these implementation details that distinguish an algorithm from a full-blown program. Whether we sort individual numbers or large records that contain numbers is irrelevant to the method by which a sorting procedure determines the sorted order. Thus, when focusing on the problem of sorting, we typically assume that the input consists only of numbers. The translation of an algorithm for sorting numbers into a program for sorting records is conceptually straightforward, although in a given engineering situation there may be other subtleties that make the actual programming task a challenge.

Sorting algorithms

We introduced two algorithms that sort n real numbers in Chapter 1. Insertion sort takes [image] (n2) time in the worst case. Because its inner loops are tight, however, it is a fast in-place sorting algorithm for small input sizes. (Recall that a sorting algorithm sorts in place if only a constant number of elements of the input array are ever stored outside the array.) Merge sort has a better asymptotic running time, [image] (n lg n), but the MERGE procedure it uses does not operate in place.

In this part, we shall introduce two more algorithms that sort arbitrary real numbers. Heapsort, presented in Chapter 7, sorts n numbers in place in O(n lg n) time. It uses an important data structure, called a heap, to implement a priority queue.

Quicksort, in Chapter 8, also sorts n numbers in place, but its worst-case running time is [image] (n2). Its average-case running time is [image] (n lg n), though, and it generally outperforms heapsort in practice. Like insertion sort, quicksort has tight code, so the hidden constant factor in its running time is small. It is a popular algorithm for sorting large input arrays.

Insertion sort, merge sort, heapsort, and quicksort are all comparison sorts: they determine the sorted order of an input array by comparing elements. Chapter 9 begins by introducing the decision-tree model in order to study the performance limitations of comparison sorts. Using this model, we prove a lower bound of [image] (n lg n) on the worst-case running time of any comparison sort on n inputs, thus showing that heapsort and merge sort are asymptotically optimal comparison sorts.

Chapter 9 then goes on to show that we can beat this lower bound of [image] (n lg n) if we can gather information about the sorted order of the input by means other than comparing elements. The counting sort algorithm, for example, assumes that the input numbers are in the set {l, 2, . . . , k}. By using array indexing as a tool for determining relative order, counting sort can sort n numbers in O(k + n) time. Thus, when k = O(n), counting sort runs in time that is linear in the size of the input array. A related algorithm, radix sort, can be used to extend the range of counting sort. If there are n integers to sort, each integer has d digits, and each digit is in the set {1,2, . . . , k}, radix sort can sort the numbers in O(d (n + k)) time. When d is a constant and k is O(n), radix sort runs in linear time. A third algorithm, bucket sort, requires knowledge of the probabilistic distribution of numbers in the input array. It can sort n real numbers uniformly distributed in the half-open interval [0, 1) in average-case O(n) time.

Order statistics

The ith order statistic of a set of n numbers is the ith smallest number in the set. One can, of course, select the ith order statistic by sorting the input and indexing the ith element of the output. With no assumptions about the input distribution, this method runs in [image] (n lg n) time, as the lower bound proved in Chapter 9 shows.

In Chapter 10, we show that we can find the ith smallest element in O(n) time, even when the elements are arbitrary real numbers. We present an algorithm with tight pseudocode that runs in O(n2) time in the worst case, but linear time on average. We also give a more complicated algorithm that runs in O(n) worst-case time.

Background

Although most of this part does not rely on difficult mathematics, some sections do require mathematical sophistication. In particular, the average-case analyses of quicksort, bucket sort, and the order-statistic algorithm use probability, which is reviewed in Chapter 6. The analysis of the worst-case linear-time algorithm for the order statistic involves somewhat more sophisticated mathematics than the other worst-case analyses in this part.

Go to Chapter 7 Back to Table of Contents


Intro to Algorithms: PART III: Data Structures

 Return to Table of Contents

PART III: Data Structures

Introduction

Sets are as fundamental to computer science as they are to mathematics. Whereas mathematical sets are unchanging, the sets manipulated by algorithms can grow, shrink, or otherwise change over time. We call such sets dynamic. The next five chapters present some basic techniques for representing finite dynamic sets and manipulating them on a computer.

Algorithms may require several different types of operations to be performed on sets. For example, many algorithms need only the ability to insert elements into, delete elements from, and test membership in a set. A dynamic set that supports these operations is called a dictionary. Other algorithms require more complicated operations. For example, priority queues, which were introduced in Chapter 7 in the context of the heap data structure, support the operations of inserting an element into and extracting the smallest element from a set. Not surprisingly, the best way to implement a dynamic set depends upon the operations that must be supported.

Elements of a dynamic set

In a typical implementation of a dynamic set, each element is represented by an object whose fields can be examined and manipulated if we have a pointer to the object. (Chapter 11 discusses the implementation of objects and pointers in programming environments that do not contain them as basic data types.) Some kinds of dynamic sets assume that one of the object's fields is an identifying key field. If the keys are all different, we can think of the dynamic set as being a set of key values. The object may contain satellite data , which are carried around in other object fields but are otherwise unused by the set implementation. It may also have fields that are manipulated by the set operations; these fields may contain data or pointers to other objects in the set.

Some dynamic sets presuppose that the keys are drawn from a totally ordered set, such as the real numbers, or the set of all words under the usual alphabetic ordering. (A totally ordered set satisfies the trichotomy property, defined on page 31.) A total ordering allows us to define the minimum element of the set, for example, or speak of the next element larger than a given element in a set.

Operations on dynamic sets

Operations on a dynamic set can be grouped into two categories: queries, which simply return information about the set, and modifying operations , which change the set. Here is a list of typical operations. Any specific application will usually require only a few of these to be implemented.

SEARCH(S, k) A query that, given a set S and a key value k, returns a pointer x to an element in S such that key[x] = k, or NIL if no such element belongs to S.

INSERT(S, x) A modifying operation that augments the set S with the element pointed to by x. We usually assume that any fields in element x needed by the set implementation have already been initialized.

DELETE(S, x) A modifying operation that, given a pointer x to an element in the set S, removes x from S. (Note that this operation uses a pointer to an element x, not a key value.)

MINIMUM(S) A query on a totally ordered set S that returns the element of S with the smallest key.

MAXIMUM(S) A query on a totally ordered set S that returns the element of S with the largest key.

SUCCESSOR(S, x) A query that, given an element x whose key is from a totally ordered set S, returns the next larger element in S, or NIL if x is the maximum element.

PREDECESSOR(S, x) A query that, given an element x whose key is from a totally ordered set S, returns the next smaller element in S, or NIL if x is the minimum element.

The queries SUCCESSOR and PREDECESSOR are often extended to sets with nondistinct keys. For a set on n keys, the normal presumption is that a call to MINIMUM followed by n - 1 calls to SUCCESSOR enumerates the elements in the set in sorted order.

The time taken to execute a set operation is usually measured in terms of the size of the set given as one of its arguments. For example, Chapter 14 describes a data structure that can support any of the operations listed above on a set of size n in time O(1g n).

Overview of Part III

Chapters 11--15 describe several data structures that can be used to implement dynamic sets; many of these will be used later to construct efficient algorithms for a variety of problems. Another important data structure--the heap--has already been introduced in Chapter 7.

Chapter 11 presents the essentials of working with simple data structures such as stacks, queues, linked lists, and rooted trees. It also shows how objects and pointers can be implemented in programming environments that do not support them as primitives. Much of this material should be familiar to anyone who has taken an introductory programming course.

Chapter 12 introduces hash tables, which support the dictionary operations INSERT, DELETE, and SEARCH. In the worst case, hashing requires [image] (n) time to perform a SEARCH operation, but the expected time for hash-table operations is O(1). The analysis of hashing relies on probability, but most of the chapter requires no background in the subject.

Binary search trees, which are covered in Chapter 13, support all the dynamic-set operations listed above. In the worst case, each operation takes [image] (n) time on a tree with n elements, but on a randomly built binary search tree, the expected time for each operation is O(1g n). Binary search trees serve as the basis for many other data structures.

Red-black trees, a variant of binary search trees, are introduced in Chapter 14. Unlike ordinary binary search trees, red-black trees are guaranteed to perform well: operations take O(1g n) time in the worst case. A red-black tree is a balanced search tree; Chapter 19 presents another kind of balanced search tree, called a B-tree. Although the mechanics of red-black trees are somewhat intricate, you can glean most of their properties from the chapter without studying the mechanics in detail. Nevertheless, walking through the code can be quite instructive.

In Chapter 15, we show how to augment red-black trees to support operations other than the basic ones listed above. First, we augment them so that we can dynamically maintain order statistics for a set of keys. Then, we augment them in a different way to maintain intervals of real numbers.

Go to Chapter 11 Back to Table of Contents


Intro to Algorithms: PART IV: Advanced Design and Analysis Techniques

 Return to Table of Contents

PART IV: Advanced Design and Analysis Techniques

Introduction

This part covers three important techniques for the design and analysis of efficient algorithms: dynamic programming (Chapter 16), greedy algorithms (Chapter 17), and amortized analysis (Chapter 18). Earlier parts have presented other widely applicable techniques, such as divide-and-conquer, randomization, and the solution of recurrences. The new techniques are somewhat more sophisticated, but they are essential for effectively attacking many computational problems. The themes introduced in this part will recur later in the book.

Dynamic programming typically applies to optimization problems in which a set of choices must be made in order to arrive at an optimal solution. As choices are made, subproblems of the same form often arise. Dynamic programming is effective when a given subproblem may arise from more than one partial set of choices; the key technique is to store, or "memoize," the solution to each such subproblem in case it should reappear. Chapter 16 shows how this simple idea can easily transform exponential-time algorithms into polynomial-time algorithms.

Like dynamic-programming algorithms, greedy algorithms typically apply to optimization problems in which a set of choices must be made in order to arrive at an optimal solution. The idea of a greedy algorithm is to make each choice in a locally optimal manner. A simple example is coin-changing: to minimize the number of U.S. coins needed to make change for a given amount, it suffices to select repeatedly the largest-denomination coin that is not larger than the amount still owed. There are many such problems for which a greedy approach provides an optimal solution much more quickly than would a dynamic-programming approach. It is not always easy to tell whether a greedy approach will be effective, however. Chapter 17 reviews matroid theory, which can often be helpful in making such a determination.

Amortized analysis is a tool for analyzing algorithms that perform a sequence of similar operations. Instead of bounding the cost of the sequence of operations by bounding the actual cost of each operation separately, an amortized analysis can be used to provide a bound on the actual cost of the entire sequence. One reason this idea can be effective is that it may be impossible in a sequence of operations for all of the individual operations to run in their known worst-case time bounds. While some operations are expensive, many others might be cheap. Amortized analysis is not just an analysis tool, however; it is also a way of thinking about the design of algorithms, since the design of an algorithm and the analysis of its running time are often closely intertwined. Chapter 18 introduces three equivalent ways to perform an amortized analysis of an algorithm.

Go to Chapter 16 Back to Table of Contents


Intro to Algorithms: PART V: Advanced Data Structures

 Return to Table of Contents

PART V: Advanced Data Structures

Introduction

This part returns to the examination of data structures that support operations on dynamic sets but at a more advanced level than Part III. Two of the chapters, for example, make extensive use of the amortized analysis techniques we saw in Chapter 18.

Chapter 19 presents B-trees, which are balanced search trees designed to be stored on magnetic disks. Because magnetic disks operate much more slowly than random-access memory, we measure the performance of B-trees not only by how much computing time the dynamic-set operations consume but also by how many disk accesses are performed. For each B-tree operation, the number of disk accesses increases with the height of the B-tree, which is kept low by the B-tree operations.

Chapters 20 and 21 give implementations of mergeable heaps, which support the operations INSERT, MINIMUM, EXTRACT-MIN, and UNION. The UNION operation unites, or merges, two heaps. The data structures in these chapters also support the operations DELETE and DECREASE-KEY.

Binomial heaps, which appear in Chapter 20, support each of these operations in O(lg n) worst-case time, where n is the total number of elements in the input heap (or in the two input heaps together in the case of UNION). It is when the UNION operation must be supported that binomial heaps are superior to the binary heaps introduced in Chapter 7, because it takes [image] (n) time to unite two binary heaps in the worst case.

Fibonacci heaps, in Chapter 21, improve upon binomial heaps, at least in a theoretical sense. We use amortized time bounds to measure the performance of Fibonacci heaps. The operations INSERT, MINIMUM, and UNION take only O(1) actual and amortized time on Fibonacci heaps, and the operations EXTRACT-MIN and DELETE take O(lg n) amortized time. The most significant advantage of Fibonacci heaps, however, is that DECREASE-KEY takes onlyO(1) amortized time. The low amortized time of the DECREASE-KEY operation is why Fibonacci heaps are at the heart of some of the asymptotically fastest algorithms to date for graph problems.

Finally, Chapter 22 presents data structures for disjoint sets. We have a universe of n elements that are grouped into dynamic sets. Initially, each element belongs to its own singleton set. The operation UNION unites two sets, and the query FIND-SET identifies the set that a given element is in at the moment. By representing each set by a simple rooted tree, we obtain surprisingly fast operations: a sequence of m operations runs in O(m [image] (m, n)) time, where [image] (m, n) is an incredibly slowly growing function--as long as n is no more than the estimated number of atoms in the entire known universe, [image] (m, n) is at most 4. The amortized analysis that proves this time bound is as complex as the data structure is simple. Chapter 22 proves an interesting but somewhat simpler bound on the running time.

The topics covered in this part are by no means the only examples of "advanced" data structures. Other advanced data structures include the following.

A data structure invented by van Emde Boas [194] supports the operations MINIMUM, MAXIMUM, INSERT, DELETE, SEARCH, EXTRACT-MIN, EXTRACT-MAX, PREDECESSOR, and SUCCESSOR in worst-case time O(lg lg n), subject to the restriction that the universe of keys is the set { 1, 2, . . . , n}.

Dynamic trees, introduced by Sleator and Tarjan [177] and discussed by Tarjan [188], maintain a forest of disjoint rooted trees. Each edge in each tree has a real-valued cost. Dynamic trees support queries to find parents, roots, edge costs, and the minimum edge cost on a path from a node up to a root. Trees may be manipulated by cutting edges, updating all edge costs on a path from a node up to a root, linking a root into another tree, and making a node the root of the tree it appears in. One implementation of dynamic trees gives an O(lg n) amortized time bound for each operation; a more complicated implementation yields O(lg n) worst-case time bounds.

Splay trees, developed by Sleator and Tarjan [178] and discussed by Tarjan [188], are a form of binary search tree on which the standard search-tree operations run in O(lg n) amortized time. One application of splay trees simplifies dynamic trees.

Persistent data structures allow queries, and sometimes updates as well, on past versions of a data structure. Driscoll, Sarnak, Sleator, and Tarjan [59] present techniques for making linked data structures persistent with only a small time and space cost. Problem 14-1 gives a simple example of a persistent dynamic set.

Go to Chapter 19 Back to Table of Contents


Intro to Algorithms: PART VI: Graph Algorithms

 Return to Table of Contents

PART VI: Graph Algorithms

Introduction

Graphs are a pervasive data structure in computer science, and algorithms for working with them are fundamental to the field. There are hundreds of interesting computational problems defined in terms of graphs. In this part, we touch on a few of the more significant ones.

Chapter 23 shows how we can represent a graph on a computer and then discusses algorithms based on searching a graph using either breadth-first search or depth-first search. Two applications of depth-first search are given: topologically sorting a directed acyclic graph and decomposing a directed graph into its strongly connected components.

Chapter 24 describes how to compute a minimum-weight spanning tree of a graph. Such a tree is defined as the least-weight way of connecting all of the vertices together when each edge has an associated weight. The algorithms for computing minimum spanning trees are good examples of greedy algorithms (see Chapter 17).

Chapters 25 and 26 consider the problem of computing shortest paths between vertices when each edge has an associated length or "weight." Chapter 25 considers the computation of shortest paths from a given source vertex to all other vertices, and Chapter 26 considers the computation of shortest paths between every pair of vertices.

Finally, Chapter 27 shows how to compute a maximum flow of material in a network (directed graph) having a specified source of material, a specified sink, and specified capacities for the amount of material that can traverse each directed edge. This general problem arises in many forms, and a good algorithm for computing maximum flows can be used to solve a variety of related problems efficiently.

In describing the running time of a graph algorithm on a given graph G = (V, E), we usually measure the size of the input in terms of the number of vertices |V| and the number of edges |E| of the graph. That is, there are two relevant parameters describing the size of the input, not just one. We adopt a common notational convention for these parameters. Inside asymptotic notation (such as O-notation or [image] -notation), and only inside such notation, the symbol V denotes |V| and the symbol E denotes |E|. For example, we might say, "the algorithm runs in time O(V E)," meaning that the algorithm runs in time O(|V||E|). This convention makes the running-time formulas easier to read, without risk of ambiguity.

Another convention we adopt appears in pseudocode. We denote the vertex set of a graph G by V[G] and its edge set by E[G]. That is, the pseudocode views vertex and edge sets as attributes of a graph.

Go to Chapter 23 Back to Table of Contents


Intro to Algorithms: PART VII: Selected Topics

 Return to Table of Contents

PART VII: Selected Topics

Introduction

This part contains a selection of algorithmic topics that extend and complement earlier material in this book. Some chapters introduce new models of computation such as combinational circuits or parallel computers. Others cover specialized domains such as computational geometry or number theory. The last two chapters discuss some of the known limitations to the design of efficient algorithms and introduce techniques for coping with those limitations.

Chapter 28 presents our first parallel model of computation: comparison networks. Roughly speaking, a comparison network is an algorithm that allows many comparisons to be made simultaneously. This chapter shows how to build a comparison network that can sort n numbers in O(lg2 n) time.

Chapter 29 introduces another parallel model of computation: combinational circuits. This chapter shows that two n-bit numbers can be added in O(lg n) time using a combinational circuit called a carry-lookahead adder. It also shows how to multiply two n-bit numbers in O(lg n) time.

Chapter 30 introduces a general model of parallel computation called the PRAM. The chapter presents basic parallel techniques, including pointer jumping, prefix computations, and the Euler-tour technique. Most of the techniques are illustrated on simple data structures, including lists and trees. The chapter also discusses general issues in parallel computation, including work efficiency and concurrent access to shared memory. It proves Brent's theorem, which shows how a parallel computer can efficiently simulate a combinational circuit. The chapter concludes with a work-efficient, randomized algorithm for list ranking and a remarkably efficient deterministic algorithm for symmetry breaking in a list.

Chapter 31 studies efficient algorithms for operating on matrices. It begins with Strassen's algorithm, which can multiply two n x n matrices in O(n2.81) time. It then presents two general methods--LU decomposition and LUP decomposition--for solving linear equations by Gaussian elimination in O(n3) time. It also shows that Strassen's algorithm can be used to solve linear systems faster and that, asymptotically, matrix inversion and matrix multiplication can be performed equally fast. The chapter concludes by showing how a least-squares approximate solution can be computed when a set of linear equations has no exact solution.

Chapter 32 studies operations on polynomials and shows that a well-known signal-processing technique--the Fast Fourier Transform (FFT)--can be used to multiply two degree-n polynomials in O(n lg n) time. It also investigates efficient implementations of the FFT, including a parallel circuit.

Chapter 33 presents number-theoretic algorithms. After a review of elementary number theory, it presents Euclid's algorithm for computing greatest common divisors. Algorithms for solving modular linear equations and for raising one number to a power modulo another number are presented next. An interesting application of number-theoretic algorithms is presented next: the RSA public-key cryptosystem. This cryptosystem not only can be used to encrypt messages so that an adversary cannot read them, it also can be used to provide digital signatures. The chapter then presents the Miller-Rabin randomized primality test, which can be used to find large primes efficiently--an essential requirement for the RSA system. Finally, the chapter covers Pollard's "rho" heuristic for factoring integers and discusses the state of the art of integer factorization.

Chapter 34 studies the problem of finding all occurrences of a given pattern string in a given text string, a problem that arises frequently in text-editing programs. An elegant approach due to Rabin and Karp is considered first. Then, after examining an efficient solution based on finite automata, the chapter presents the Knuth-Morris-Pratt algorithm, which achieves efficiency by cleverly preprocessing the pattern. The chapter closes with a presentation of a string-matching heuristic due to Boyer and Moore.

Computational geometry is the topic of Chapter 35. After discussing basic primitives of computational geometry, the chapter shows how a "sweeping" method can efficiently determine whether or not a set of line segments contains any intersections. Two clever algorithms for finding the convex hull of a set of points--Graham's scan and Jarvis's march--also illustrate the power of sweeping methods. The chapter closes with an efficient algorithm for finding the closest pair from among a given set of points in the plane.

Chapter 36 concerns NP-complete problems. Many interesting computational problems are NP-complete, but no polynomial-time algorithm is known for solving any of them. This chapter presents techniques for determining when a problem is NP-complete. Several classic problems are proved to be NP-complete: determining if a graph has a hamiltonian cycle, determining if a boolean formula is satisfiable, and determining if a given set of numbers has a subset that adds up to a given target value. The chapter also proves that the famous traveling-salesman problem is NP-complete.

Chapter 37 shows how approximation algorithms can be used to find approximate solutions to NP-complete problems efficiently. For some NP-complete problems, approximate solutions that are near optimal are quite easy to produce, but for others even the best approximation algorithms known work progressively more poorly as the problem size increases. Then, there are some problems for which one can invest increasing amounts of computation time in return for increasingly better approximate solutions. This chapter illustrates these possibilities with the vertex-cover problem, the traveling-salesman problem, the set-covering problem, and the subset-sum problem.

Go to Chapter 28 Back to Table of Contents


Intro to Algorithms: PREFACE

 Return to Table of Contents

PREFACE

This book provides a comprehensive introduction to the modern study of computer algorithms. It presents many algorithms and covers them in considerable depth, yet makes their design and analysis accessible to all levels of readers. We have tried to keep explanations elementary without sacrificing depth of coverage or mathematical rigor.

Each chapter presents an algorithm, a design technique, an application area, or a related topic. Algorithms are described in English and in a "pseudocode" designed to be readable by anyone who has done a little programming. The book contains over 260 figures illustrating how the algorithms work. Since we emphasize efficiency as a design criterion, we include careful analyses of the running times of all our algorithms.

The text is intended primarily for use in undergraduate or graduate courses in algorithms or data structures. Because it discusses engineering issues in algorithm design, as well as mathematical aspects, it is equally well suited for self-study by technical professionals.

To the teacher

This book is designed to be both versatile and complete. You will find it useful for a variety of courses, from an undergraduate course in data structures up through a graduate course in algorithms. Because we have provided considerably more material than can fit in a typical one-term course, you should think of the book as a "buffet" or "smorgasbord" from which you can pick and choose the material that best supports the course you wish to teach.

You should find it easy to organize your course around just the chapters you need. We have made chapters relatively self-contained, so that you need not worry about an unexpected and unnecessary dependence of one chapter on another. Each chapter presents the easier material first and the more difficult material later, with section boundaries marking natural stopping points. In an undergraduate course, you might use only the earlier sections from a chapter; in a graduate course, you might cover the entire chapter.

We have included over 900 exercises and over 120 problems. Each section ends with exercises, and each chapter ends with problems. The exercises are generally short questions that test basic mastery of the material. Some are simple self-check thought exercises, whereas others are suitable as assigned homework. The problems are more elaborate case studies that often introduce new material; they typically consist of several questions that lead the student through the steps required to arrive at a solution.

We have starred (*) the sections and exercises that are more suitable for graduate students than for undergraduates. A starred section is not necessarily more difficult than an unstarred one, but it may require an understanding of more advanced mathematics. Likewise, starred exercises may require an advanced background or more than average creativity.

To the student

We hope that this textbook provides you with an enjoyable introduction to the field of algorithms. We have attempted to make every algorithm accessible and interesting. To help you when you encounter unfamiliar or difficult algorithms, we describe each one in a step-by-step manner. We also provide careful explanations of the mathematics needed to understand the analysis of the algorithms. If you already have some familiarity with a topic, you will find the chapters organized so that you can skim introductory sections and proceed quickly to the more advanced material.

This is a large book, and your class will probably cover only a portion of its material. We have tried, however, to make this a book that will be useful to you now as a course textbook and also later in your career as a mathematical desk reference or an engineering handbook.

What are the prerequisites for reading this book?

You should have some programming experience. In particular, you should understand recursive procedures and simple data structures such as arrays and linked lists.

You should have some facility with proofs by mathematical induction. A few portions of the book rely on some knowledge of elementary calculus. Beyond that, Part I of this book teaches you all the mathematical techniques you will need.

To the professional

The wide range of topics in this book makes it an excellent handbook on algorithms. Because each chapter is relatively self-contained, you can focus in on the topics that most interest you.

Most of the algorithms we discuss have great practical utility. We therefore address implementation concerns and other engineering issues. We generally provide practical alternatives to the few algorithms that are primarily of theoretical interest.

If you wish to implement any of the algorithms, you will find the translation of our pseudocode into your favorite programming language a fairly straightforward task. The pseudocode is designed to present each algorithm clearly and succinctly. Consequently, we do not address error-handling and other software-engineering issues that require specific assumptions about your programming environment. We attempt to present each algorithm simply and directly without allowing the idiosyncrasies of a particular programming language to obscure its essence.

Errors

A book of this length is certain to contain errors and omissions. If you find any errors or have other constructive suggestions, we would appreciate hearing from you. We particularly welcome suggestions for new exercises and problems, but please include solutions. You can mail your comments to

Introduction to Algorithms

MIT Laboratory for Computer Science

545 Technology Square

Cambridge, Massachusetts 02139

Alternatively, you can use Internet electronic mail to submit bug reports, request a list of known errors, or make constructive suggestions. To receive instructions, send electronic mail to algorithms@theory. lcs.mit.edu with "Subject: help" in the message header. We regret that we cannot personally respond to all mail.

Acknowledgments

Many friends and colleagues have contributed greatly to the quality of this book. We thank all of you for your help and constructive criticisms.

MIT's Laboratory for Computer Science has provided an ideal working environment. Our colleagues in the laboratory's Theory of Computation Group have been particularly supportive and tolerant of our incessant requests for critical appraisal of chapters. We specifically thank Baruch Awerbuch, Shafi Goldwasser, Leo Guibas, Tom Leighton, Albert Meyer, David Shmoys, and Eva Tardos. Thanks to William Ang, Sally Bemus, Ray Hirschfeld, and Mark Reinhold for keeping our machines (DEC Microvaxes, Apple Macintoshes, and Sun Sparcstations) running and for recompiling TEX whenever we exceeded a compile-time limit. Thinking Machines Corporation provided partial support for Charles Leiserson to work on this book during a leave of absence from MIT.

Many colleagues have used drafts of this text in courses at other schools. They have suggested numerous corrections and revisions. We particularly wish to thank Richard Beigel (Yale), Andrew Goldberg (Stanford), Joan Lucas (Rutgers), Mark Overmars (Utrecht), Alan Sherman (Tufts and Maryland), and Diane Souvaine (Rutgers).

Many teaching assistants in our courses have made significant contributions to the development of this material. We especially thank Alan Baratz, Bonnie Berger, Aditi Dhagat, Burt Kaliski, Arthur Lent, Andrew Moulton, Marios Papaefthymiou, Cindy Phillips, Mark Reinhold, Phil Rogaway, Flavio Rose, Arie Rudich, Alan Sherman, Cliff Stein, Susmita Sur, Gregory Troxel, and Margaret Tuttle.

Additional valuable technical assistance was provided by many individuals. Denise Sergent spent many hours in the MIT libraries researching bibliographic references. Maria Sensale, the librarian of our reading room, was always cheerful and helpful. Access to Albert Meyer's personal library saved many hours of library time in preparing the chapter notes. Shlomo Kipnis, Bill Niehaus, and David Wilson proofread old exercises, developed new ones, and wrote notes on their solutions. Marios Papaefthymiou and Gregory Troxel contributed to the indexing. Over the years, our secretaries Inna Radzihovsky, Denise Sergent, Gayle Sherman, and especially Be Hubbard provided endless support in this project, for which we thank them.

Many errors in the early drafts were reported by students. We particularly thank Bobby Blumofe, Bonnie Eisenberg, Raymond Johnson, John Keen, Richard Lethin, Mark Lillibridge, John Pezaris, Steve Ponzio, and Margaret Tuttle for their careful readings.

Colleagues have also provided critical reviews of specific chapters, or information on specific algorithms, for which we are grateful. We especially thank Bill Aiello, Alok Aggarwal, Eric Bach, Vaek Chvtal, Richard Cole, Johan Hastad, Alex Ishii, David Johnson, Joe Kilian, Dina Kravets, Bruce Maggs, Jim Orlin, James Park, Thane Plambeck, Herschel Safer, Jeff Shallit, Cliff Stein, Gil Strang, Bob Tarjan, and Paul Wang. Several of our colleagues also graciously supplied us with problems; we particularly thank Andrew Goldberg, Danny Sleator, and Umesh Vazirani.

This book was typeset with LATEX, a macro package for TEX. The figures were drawn on an Apple Macintosh using MacDraw II; thanks to Joanna Terry of Claris Corporation and Michael Mahoney of Advanced Computer Graphics for timely customer support. The index was compiled using Windex, a C program written by the authors. The bibliography was prepared using BIBTEX. The book was typeset at the American Mathematical Society with an Autologic phototypesetter; thanks to Ralph Youngen of AMS for his help. The cover for the book was designed by Jeannet Leendertse. The book design was created by Rebecca Daw, and Amy Hendrickson implemented the design in LATEX.

It has been a pleasure working with The MIT Press and McGraw-Hill in the development of this text. We especially thank Frank Satlow, Terry Ehling, Larry Cohen, and Lorrie Lejeune of The MIT Press and David Shapiro of McGraw-Hill for their encouragement, support, and patience. We are particularly grateful to Larry Cohen for his outstanding copyediting.

Finally, we thank our wives--Nicole Cormen, Linda Lue Leiserson, and Gail Rivest--and our children--Ricky, William, and Debby Leiserson and Alex and Christopher Rivest--for their love and support during the writing of this book. (Alex Rivest also helped with the "Martian birthday paradox.") The love, patience, and encouragement of our families made this project possible. We affectionately dedicate this book to them.

Cambridge, Massachusetts

March 1990

THOMAS H. CORMEN

CHARLES E. LEISERSON

RONALD L. RIVEST

Go to Chapter 1 Back to Table of Contents


Intro to Algorithms: Table of Contents

Introduction to Algorithms

by Thomas H. Cormen, Charles E. Leiserson, and Ronald L. Rivest

PREFACE

CHAPTER 1: INTRODUCTION

PART I: Mathematical Foundations

CHAPTER 2: GROWTH OF FUNCTIONS

CHAPTER 3: SUMMATIONS

CHAPTER 4: RECURRENCES

CHAPTER 5: SETS, ETC.

CHAPTER 6: COUNTING AND PROBABILITY

PART II: Sorting and Order Statistics

CHAPTER 7: HEAPSORT

CHAPTER 8: QUICKSORT

CHAPTER 9: SORTING IN LINEAR TIME

CHAPTER 10: MEDIANS AND ORDER STATISTICS

PART III: Data Structures

CHAPTER 11: ELEMENTARY DATA STRUCTURES

CHAPTER 12: HASH TABLES

CHAPTER 13: BINARY SEARCH TREES

CHAPTER 14: RED-BLACK TREES

CHAPTER 15: AUGMENTING DATA STRUCTURES

PART IV: Advanced Design and Analysis Techniques

CHAPTER 16: DYNAMIC PROGRAMMING

CHAPTER 17: GREEDY ALGORITHMS

CHAPTER 18: AMORTIZED ANALYSIS

PART V: Advanced Data Structures

CHAPTER 19: B-TREES

CHAPTER 20: BINOMIAL HEAPS

CHAPTER 21: FIBONACCI HEAPS

CHAPTER 22: DATA STRUCTURES FOR DISJOINT SETS

PART VI: Graph Algorithms

CHAPTER 23: ELEMENTARY GRAPH ALGORITHMS

CHAPTER 24: MINIMUM SPANNING TREES

CHAPTER 25: SINGLE-SOURCE SHORTEST PATHS

CHAPTER 26: ALL-PAIRS SHORTEST PATHS

CHAPTER 27: MAXIMUM FLOW

PART VII: Selected Topics

CHAPTER 28: SORTING NETWORKS

CHAPTER 29: ARITHMETIC CIRCUITS

CHAPTER 30: ALGORITHMS FOR PARALLEL COMPUTERS

CHAPTER 31: MATRIX OPERATIONS

CHAPTER 32: POLYNOMIALS AND THE FFT

CHAPTER 33: NUMBER-THEORETIC ALGORITHMS

CHAPTER 34: STRING MATCHING

CHAPTER 35: COMPUTATIONAL GEOMETRY

CHAPTER 36: NPCOMPLETENESS

CHAPTER 37: APPROXIMATION ALGORITHMS

BIBLIOGRAPHY


Practical Data Structures: CHAPTER 1: BASIC CONCEPTS

Next Chapter Return to Table of Contents Previous Chapter CHAPTER 1: BASIC CONCEPTS

Techniques for storing and processing data are at the heart of all programs. The term data structure is used to describe the way data is stored, and the term algorithm is used to describe the way data is processed. As you might expect, data structures and algorithms are interrelated. Choosing a data structure affects the kind of algorithm you might use, and choosing an algorithm affects the data structures you use. In this chapter, we'll explore the basic, underlying concepts of data structures, and you'll learn how to judge the effectiveness and practicality of one data structure over another. For a similar discussion of algorithms, see the companion volume, Practical Algorithms in C++ [Flamig 93].

DATA STRUCTURES: ORGANIZATIONS OF DATA

The collections of data you work with in a program have some kind of structure or organization. No matter how complex your data structures are, they can be broken down into two fundamental types: contiguous and non-contiguous.

In a contiguous structure, items of data are kept together in memory (either in RAM or in a file). An array is an example of a contiguous structure, since each element in the array is located next to one or two other elements. In contrast, items in a non-contiguous structure are scattered in memory, but are linked to each other in some way. A linked list is an example of a non-contiguous data structure. Here, the nodes of the list are linked together using pointers stored in each node. Figure 1.1 illustrates the difference between contiguous and non-contiguous structures.

Figure 1.1 Contiguous and non-contiguous structures compared.

Contiguous Structures

Contiguous structures can be broken down further into two kinds: those that contain data items of all the same size, and those where the sizes may differ. Figure 1.2 shows examples of each kind. The first kind we'll call arrays. Figure 1.2(a) shows an example of an array of numbers. In an array, each element is of the same type, and thus has the same size.

The second kind of contiguous structure we'll call structs, because they can be defined using the C++ struct keyword. Figure 1.2(b) shows a simple struct consisting of a person's name and age. In a struct, elements may be of different types, and thus may have different sizes. For instance, a person's age can be represented with a simple integer that occupies perhaps two bytes of memory. But his or her name, represented as a string of characters, may require many bytes, and may even be of varying length.

C++ allows you to represent arrays and structs using built-in language features. For example, here's how the structures in Figure 1.2 could be coded in C++:

int array[3];

struct personnel_data {

int age;

char name[80];

};

Figure 1.2 Examples on contaguous structures.

int arr[3] = {1, 2, 3};  struct customer_data {

        int age;

        char name[80];

      } ;

      customer_data billy = {21, "Billy the Kid"} ;

(a) Array

(b) Struct

Coupled with the atomic types (that is, the single data-item built-in types such as integers, floats, and pointers), arrays and structs provide all the "mortar" you need to build more exotic forms of data structures, including the non-contiguous forms you are about to see.

Note     The contiguous structures we call structs can be defined with the C++ class and union keywords, as well as with the struct keyword.

Non-Contiguous Structures

Non-contiguous structures are implemented as a collection of data items, called nodes, where each node can point to one or more other nodes in the collection. The simplest kind of non-contiguous structure is a linked list, as shown in Figure 1.3(a). The nodes of a linked list might be defined in this way:

struct Node  { // A node storing an integer

int data;   // The node data

Node *prev; // Pointer to previous node

Node *next; // Pointer to next node

};

Since this node has two pointers, it's called a doubly-linked list node. If you had just one pointer per node serving as, for example, next pointers, the node would be a singly-linked list node.

A linked-list represents a linear, one-dimensional type of non-contiguous structure, where there is only the notion of backwards and forwards. A tree, such as the one shown in Figure 1.3(b), is an example of a two-dimensional non-contiguous structure. Here, there is the notion of up and down, and left and right. In a tree, each node has only one link that leads into the node, and links can only go down the tree. The most general type of non-contiguous structure, called a graph, has no such restrictions. Figure 1.3(c) is an example of a graph.

Hybrid Structures

It's not uncommon for the two basic types of structures to be mixed into a hybrid form--part contiguous and part non-contiguous. For example, Figure 1.4 shows how to implement a doubly-linked list using three parallel arrays,  possibly stored apart from each other in memory. The array D contains the data for the list, whereas the arrays P and N hold the previous and next "pointers." The pointers are actually nothing more than indexes into the D array. For instance, D[i] holds the data for node i, and P[i] holds the index to the node previous to i, which may or may not reside at position i - 1. Likewise, N[i] holds the index to the next node in the list.

Figure 1.3 Examples of non-contiguous structures.

(a) Linked list

(b) Tree

(c) Graph

TREATING DATA STRUCTURES IN THE ABSTRACT

The design of a data structure involves more than just its organization. You also need to plan for the way the data will be accessed and processed--that is, how the data will be interpreted. The previous section showed how three parallel arrays can be interpreted as implementing a linked list. Actually, all of the  non-contiguous structures that we introduced--including lists, trees, and graphs--can be implemented either contiguously or non-contiguously. Likewise, the structures that are normally treated as contiguous--arrays and structs--can also be implemented non-contiguously.

Figure 1.4 A doubly-linked list via a hybrid data structure.

Why this seeming paradox? The problem is that the notion of a data structure in the abstract needs to be treated differently from whatever is used to implement the structure. The abstract notion of a data structure is defined in terms of the operations we plan to perform on the data.

For example, the tree structure we provided in Figure 1.3(b) is called a binary tree, since each node has two children. We could define a binary tree node as follows:

struct Node {  // A binary tree node holding integer data

int data;    // Node data

Node *left;  // Left child of node

Node *right; // Right child of node

};

Compare this definition with the one we provided earlier for a doubly-linked list node. You'll notice that the two types of nodes have the same memory layout. However, we've given two completely different interpretations of this layout. The difference lies in how we expect to be processing the data. With doubly-linked list nodes, we expect to be doing list-type operations, such as Next( ) and Prev( ). With binary tree nodes, we expect to use operations such as left( ) and Right( ).

Taken to its logical conclusion, considering both the organization of data and the expected operations on the data, leads to the notion of an abstract data type. An abstract data type is a theoretical construct that consists of data as well as the operations to be performed on the data.

A stack is a typical example of an abstract data type. Items stored in a stack can only be added and removed in a certain order--the last item added is the first item removed. We call these operations pushing and popping, which are illustrated in Figure 1.5. In this definition, we haven't specified how items are stored on the stack, or how the items are pushed and popped. We've only specified the valid operations that can be performed.

Figure 1.5 A push-down stack.

Of course, to be useful, an abstract data type (such as a stack) has to be implemented, and this is where data structures come into play. For instance, we might choose the simple data structure of an array to represent the stack, and then define the appropriate indexing operations to perform pushing and popping. C++ allows you to create user-defined types to implement abstract data types, as you'll see next.

CIasses: User-Defined Data Types

CIasses: User-Defined Data Types

C++ provides an elegant way to both define and implement abstract data types (although the term user-defined types may be more appropriate), through the use of the class, struct, and union keywords. (The structs we declared earlier were simple forms of classes.) For example, here's how we might define a stack of integers in C++:

class Stack {

private:

int data[10]; // Our stack can hold 10 items

int top;      // Current "top" of stack

public:

Stack( )    { top = 0; }

Push(int a) { data[top++] = a; }

int Pop( )  { return data[-top]; }

};

This code defines a Stack type and specifies a particular implementation for it. The function headers in the public portion of the class define the abstract operations that can be performed on the stack. The function headers provide the interface to the data and indicate what operations are expected to be performed on the data. The function bodies, along with the private data, provide the implementation. Of course, this implementation is only one of many possibilities. We could, for instance, use a linked-list to store the stack elements. The "guts" of the class would change accordingly, but the interface would remain the same.

The ability to merge the structure of a data type with the specifications that describe how to operate on that data, through classes, marks the departure of C++ programming from C programming. The class construct allows us to easily create more than one instance of the data type. These instances are called objects. For example, if we declare a variable of type Stack, we've created a Stack object:

Stack Mystack; // Mystack is a Stack object

A whole new style of programming has evolved around using objects, and in fact is called object-oriented programming (OOP). For a discussion of OOP principles, see the references. We'll be indirectly using OOP constructs throughout this book. But whether or not we use OOP to implement data structures is a side issue. The focus here is on the data structures themselves.

 SELECTING A DATA STRUCTURE TO MATCH THE OPERATION

Probably the most important process in designing a program involves choosing which data structures to use. The choice depends greatly on the type of operations you wish to perform. We can explore the fundamental issues for this decision-making process at a high level by examining the tradeoffs between contiguous and non-contiguous structures.

Suppose you have an application that uses a sequence of objects, where one of the main operations is to delete an object from the middle of the sequence. Here's how code might be written to perform deletion for an array of integers:

void Delete(int *seq, int &n, int posn)

// Delete the item at posn from an array of n elements

{

if (n) {

int i = posn;

n-;

while(i<n) seq[i] = seq[i+1];

}

return;

}

This function shifts toward the front all elements that follow the element at position posn. This shifting involves data movement that, for integer elements, isn't too costly. However, suppose the array stores large objects, and lots of them. In this case, the overhead for moving data becomes high. The problem is that, in a contiguous structure, such as an array, the logical ordering (the ordering that we wish to interpret our elements to have) is the same as the physical ordering (the ordering that the elements actually have in memory).

If we choose a non-contiguous representation, however, we can separate the logical ordering from the physical ordering and thus change one without affecting the other. For example, if we store our collection of elements using a doubly-linked list (with previous and next pointers), we can do the deletion without moving the elements. Instead, we just modify the pointers in each node.

Using the doubly-linked list Node structure we gave earlier, here's how we can rewrite the Delete( ) function:

void Delete(Node *seq, int posn)

// Delete the item at posn from a list of elements

{

int i = posn;

Node *q = seq;

while(i && q) {

i-; q = q->next;

}

if (q) {

// Not at end of list, so detach p by making previous

// and next nodes point to each other

Node *p = q->prev;

Node *n = q->next;

if (p) p->next = n;

if (n) n->prev = p;

}

return;

}

The process of deleting a node from a list is independent of the type of data stored in the node, and can be accomplished with some simple "pointer surgery" as illustrated in Figure 1.6. Since very little data is moved during this process, the deletion using linked lists will often be faster than when arrays are used.

It may seem that linked lists are superior to arrays. But is that always true? There are tradeoffs. Our linked lists yield faster deletions, but they take up more room because they require space for two extra pointers per element.

Complexity Functions

One way to evaluate the effectiveness of one data structure over another is to consider their space complexity functions, which measure their respective space

Figure 1.6 Detaching a node from a list.

overhead. The complexity function for many data structures is linear, and can be written as:

s(n)=an+b.

Here, n represents the number of objects being stored, a is the number of memory cells used per object, and b is the number of memory cells of overhead that's independent of how many objects are stored. For example, suppose you have an array of n elements, along with a number that indicates the length of the array. Suppose also that the memory cells are expressed in units of words, with each array element requiring two words and the array length requiring one word. The complexity function could then be written as:

s(n)=2n+1.

Compare this complexity with that of using a linked list. For a list of n objects, you not only have the space required by the objects themselves, but also space for the prev and next pointers. Assuming that each pointer requires one word, and that we again use a length indicator of one word, the following complexity function is obtained:

s(n)=4n+1.

These functions illustrate mathematically what we concluded in the previous section: that linked lists have more space overhead than arrays.

Complexity functions can also be used on algorithms as well. Here, the functions are usually expressed in time units, with n representing the number of items to process. Consider an array-printing algorithm, shown in the following code:

int i;

for(i = 0; i<n; i++) cout << a[i] << ' ';

cout << '\n';

For this algorithm, a step could be defined as the printing of one element. If n elements are in the array, there are n steps, and the time complexity of the algorithm can be written as the linear function:

t(n)=an+b.

Here, a is some constant multiplier that represents the time required to execute each step, perhaps in units of seconds, or in machine cycles. The constant b represents the setup time required to start the algorithm and to stop it.

While the time complexity function just given is linear, not all time complexities are linear. For example, the binary search trees of Chapter 10 can be searched with a logarithmic time complexity of:

t(n)=alog2n+b.

For large n, log2n is much smaller than n. If n is one million, log2n is approximately 20. This result is significant. It means we can search through one million nodes in 20 steps, rather than the half-million steps needed for an array or linked list (assuming randomly ordered elements). This points out in a rather strong way how important it is to choose a data structure that matches the operations to be performed.

Since a linear function is so radically different from a logarithmic one, the constant terms and multipliers become less important when comparing the functions. In fact, time complexities are often expressed with the least significant terms dropped. For example, in a linear function, we say that the function is proportional to n. This proportionality is often denoted using what is called the big-Oh notation. For instance:

t(n)=O(n).

This statement is read "t(n) is order n." If the time complexity were a logarithmic function, we would write:

t(n)=O(logn)

and say that the function is order logn. If the time complexity were quadratic, as is the case in a simple nested for loop, we would write:

t(n)=O(n2)

and say that the function t(n) is order n2.

Using the big-Oh notation for space complexities is less meaningful, because most data structures are order n in space complexity. Thus, the constant terms and multipliers become very important in comparisons.

Note     For a complete discussion of the big-Oh notation, see the companion volume [Flamig 93].

 PRACTICAL DESIGN ISSUES

Choosing an efficient algorithm or data structure is just one part of the design process. Next, we'll look at some design issues that are broader in scope. If you think that many of these issues are nothing more than common sense, you're right. However, it's easy to get caught up in the technical gadgetry of a sophisticated program, and sometimes our common-sense reasoning gets lost.

Three Design Goals

There are three basis design goals that you should strive for in a program:

1.     Try to save time.

2.     Try to save space.

3.     Try to save face.

In almost all situations, a program that runs faster is a better program, so saving time is an obvious goal. Likewise, a program that saves space over a competing program is considered desirable. However, even the fastest, most memory efficient program is a bad one if it doesn't work. You want to "save face" by preventing your program from locking up or generating reams of garbled data.

These three design goals are often mutually exclusive. The classic tradeoff involves speed versus space. Implementing a faster algorithm generally involves using more memory. For example, consider the tradeoffs involved in maintaining a sorted collection using a linked list versus an array. You saw where a linked list approach can be generally faster, but requires more memory to store extra pointers in each element. The extra pointers are used to save time in rearranging the elements.

The time/space tradeoff works in reverse, as well. Consider a database program that uses a paging scheme to keep memory requirements fixed, even for large databases. Instead of loading all records into memory, a small cache of, let's say, 50 records is used, and records are paged in and out of this cache as the database is accessed. While this strategy does keep memory requirements at a minimum, it slows down access to the records, due to the page shuffling that takes place.

You can liken the time-space tradeoff to the situation of a wood sculptor forced to work in a small, crowded room. As the sculptor chips away at his or her masterpiece, the chips pile up and soon begin to interfere with the work in progress. The sculptor must frequently stop and clear the chips away. If the room were larger, less time would be spent clearing the chips away. Likewise, programs that have an adequate amount of memory to work with are likely to be faster; less time is spent rearranging the contents of memory in order to keep working. In our database example, a lot of time is spent shuffling records in and out of memory. This activity could be eliminated if enough memory were available to store all the records in memory.

Tradeoffs also exist between saving time or space and maintaining the resulting code. In many cases, the most efficient algorithms are also the most complicated, since smarter code is needed to make the algorithms perform at their peak. Unfortunately, the more complicated the code, the more difficult it is to fix, modify, and port to other environments. You might actually be better off taking a performance hit and using simpler code instead.

Although it doesn't happen often, you can sometimes get lucky, and make your code faster, tighter, and easier to maintain--all at the same time. An example of this is the use of inline functions in C++. Used properly, inline functions can save you time and space by eliminating function call overhead. And inline functions can make your code easier to maintain by encapsulating the details of the functions (even if those details are rather brief).

Balancing the three main design goals makes programming more of an engineering art than an exact science because a large number of factors--many difficult to quantify--come into play. The next section outlines some design principles to help you perform this balancing act.

Some Design Principles

Unless your job is to be theorist, you want to be practical in your designs. As a programmer, you are being paid to produce, a fact that you should never lose sight of. Here are some practical design principles to keep you on the right track. Many of these principles are related to each other, and some are simply paraphrases of others.

       Design Principle 1. Maintain the viewpoint of a practitioner. You have some programming task to accomplish. Your job is, first and foremost, to get the program working. Use whatever techniques are necessary to get the program working, no matter how inelegant those techniques may seem--that is, of course, unless an elegant solution is readily apparent and easily implemented, or if the alternatives are clearly inadequate. Try incorporating more elegant techniques only after the program is working, and only if it's warranted. For example, are you going to be using this program for years to come, or only once or twice?

       Design Principle 2. "There is no such thing as a complex solution." This quote from a colleague (see [Mohle 93]) sums up what is wrong with many programs today. A design that's too complex is no solution at all, since it may not be capable of being implemented (perhaps it takes too much memory), and may not be capable of being maintained, (and thus riddled with bugs). Chances are that a simpler, more elegant design is waiting out there in the ether for you to discover--if you just spend more time in the design phase.

       Design Principle 3. Don't over-generalize. Many books on object-oriented programming show how to build general-purpose collection classes. While there is nothing wrong with this per se, it's easy to overdo it. General- purpose code tends to be less efficient than code written specifically for the task at hand, since more cases must be handled. Ironically, the more flexible your code is, the more difficult it can be to maintain--the opposite of what you might expect!

Use simple, direct designs whenever possible. For instance, if you need to store a small list of numbers, don't get carried away by using some massive set of general purpose collection classes. A simple array will do just fine.

This doesn't mean you shouldn't strive to make your designs reusable and robust. But you should temper your eagerness to write, for example, the ultimate collection class, by asking yourself: "Is this really warranted? Will I (or someone else), be able to make changes to this code later on? What's the purpose of my activities? Is it my goal to write a general-purpose set of routines, or am I trying to solve just this one problem?"

       Design Principle 4. General-purpose code is not always the most reusable. This rule may seem at first like a paradox. But consider this: code that is very general purpose tends to be bloated and inefficient, due to the need to handle many cases. If the code is too bloated, you may not want to include it in your application. So it's not very reusable, is it? A surprising number of commercial toolkits suffer from this problem. One test that you can use on the objects you design is to ask yourself the following question: "Would I want to incorporate ten of these objects into another object?" This question will force you to think about how streamlined you've made your objects.

       Design Principle 5. Follow the 80-20 rule. This rule comes in many variations, including this one: in many programs, approximately 80 percent of the execution time is spent in 20 percent of the code. Few designs work well in all situations, so make your designs work well for the portion of code that's used most often. For the remaining code, the performance may suffer, but as long as your design doesn't totally stop working or produce the wrong results, don't worry about it too much. Of course, if you know a design that works well 100 percent of the time, and that design is cost-effective, by all means use it!

       Design Principle 6. Design first, then optimize. As programmers, we've all made the mistake of trying to write and optimize code before we've even considered the design choices. Before optimizing a data structure, for instance, ask yourself if there is perhaps a better choice. Choosing a good data structure can make much more difference than merely optimizing the one you already have. For instance, no matter how clever you are in optimizing a linked list for searching a large number of records, a balanced binary tree is likely to perform much better, even without any optimization.

       Design Principle 7. Try to get the most leverage out of any overhead you must add. Writing robust code often entails writing smarter code. And smarter code usually has more overhead associated with it. The issue is where this overhead is placed. For instance, consider the linear space complexity function s(n) = a n + b. The terms a and b represent overhead. The overhead represented by a is multiplied n times (where n is the number of objects stored), so you obviously want to avoid adding overhead there if you can. The overhead represented by b is much more benign since it's just a constant factor. Thus, strive to add type b overhead rather than type a overhead.

An example of this principle is discussed in Chapter 10. There, we discuss the possibility of using a threaded tree over an ordinary tree, where each node has an extra pointer back to its parent. This parent pointer represents overhead that's incurred in every node (so it's type a overhead). The reason for using parent pointers is to simplify the code. But it might be better to have slightly more complex code, eliminating the parent pointers, and thus be able to use type b overhead instead of type a overhead.

       Design Principle 8. You shouldn't have to pay for something you aren't going to use. Consider using a general-purpose collection class when all you really want is a simple array. Why should you pay for the extra overhead? Yet many poorly designed libraries want you to do just that.

It's interesting to note that Design Principle 8 permeates the design of the C++ language itself. While C++ adds object-oriented programming capabilities to C, you never pay for these capabilities when coding straight C without objects. Contrast this to the design of Smalltalk, where you are forced to pay for the overhead of using objects, regardless of whether you actually need to use objects to accomplish the task at hand.

 SUMMARY

In this chapter, we've reviewed the basic concepts behind data structures, and have given a set of common-sense principles that apply to all designs. In Chapters 2 and 3 we'll begin to focus on more specific principles as they apply to designing code in C++, with data structures particularly in mind.

Go to Chapter 2            Return to Table of Contents


Practical Data Structures: BINARY TREES

 Next Chapter Return to Table of Contents Previous Chapter

CHAPTER 10: BINARY TREES

Linked-lists are examples of linear non-contiguous data structures, where movement takes place in a single dimension (forward and backward). In this chapter, we'll discuss a form of non-linear, non-contiguous data structures known as trees, which add a second up/down dimension. In particular, we'll explain special types of trees known as binary trees, which are useful in search operations.

WHAT ARE TREES?

A tree is a hierarchical collection of nodes. One of the nodes, known as the root, is at the top of the hierarchy. Each node can have at most one link coming into it. The node where the link originates is called the parent node. The root node has no parent. The links leaving a node (any number of links are allowed) point to child nodes. Trees are recursive structures. Each child node is itself the root of a subtree. At the bottom of the tree are leaf nodes, which have no children.

There is a directionality to the links of a tree. In the strictest definition of a tree, the links point down the tree, but never up. In addition, a tree cannot contain cycles. That is, no path originating from a node can return to that node. Figure 10.1(a) shows a set of nodes that have the properties of trees. Figure 10.1(b) shows a set of nodes that do not constitute a tree. That's because node d has two parents, and two cycles are present (a-b-d and a-c-d).

Trees represent a special case of more general structures known as graphs. In a graph, there are no restrictions on the number of links that can enter or leave a node, and cycles may be present in the graph. See the companion volume [Flamig 93] for a discussion of graphs.

Figure 10.1 A tree and a non-tree.

(a) A tree

(b) Not a tree

Every tree has a height, which is defined as the maximum number of nodes that must be visited on any path from the root to a leaf. For instance, the tree in Figure 10.1(a) has a height of 3. In turn, the depth of a node is the number of nodes along the path from the root to that node. For instance, node c in Figure 10.1(a) has a depth of 1.

Trees provide a natural way to represent the hierarchical relationship of data. For example, Figure 10.2 shows a tree representing the hierarchical organization of a company. Figure 10.3 shows an example of a tree representing the expression a*b+c. Here, the operators * and + have an order of precedence, where + binds looser than *.

Note In many of the figures in this chapter--including Figures 10.2 and 10.3--the direction of the arrows is implied. Unless otherwise shown, the direction is always from the root downward.

Figure 10.2 An organization tree for a typical company.

Figure 10.3 An expression tree.

Binary Trees

In general, tree nodes can have any number of children. In a binary tree, the nodes have no more than two children. Figures 10.1(a) and 10.3 are examples of binary trees. Figure 10.2 isn't a binary tree because several of its nodes have three children. A node in a binary tree isn't required to have two children. For example, node b in Figure 10.1(a) has only one child. Of course, the leaves of a binary tree have no children.

Binary trees are easy to implement because they have a small, fixed number of child links. Because of this characteristic, binary trees are the most common type of trees and form the basis of many important data structures.

Binary Tree Node Classes

As was true for the linked-list nodes described in Chapter 8, it's useful to define an abstract binary tree node class that doesn't store data, and then derive, using templates, a family of node classes holding different types of data. The following Bnodeb and Bnode classes use this technique:

struct Bnodeb  {  // An abstract binary tree node

Bnodeb *left;  // Pointer to left child

Bnodeb *right; // Pointer to right child

};

template<class TYPE>

struct Bnode : public Bnodeb { // Binary tree node with data

TYPE info;

Bnode();

Bnode(const TYPE &x, Bnode<TYPE> *l=0, Bnode<TYPE> *r=0);

Bnode<TYPE> *&Left();

Bnode<TYPE> *&Right();

} ;

template<class TYPE>

Bnode<TYPE>::Bnode()

{

left = 0; right = 0;

}

template<class TYPE>

Bnode<TYPE>::Bnode(const TYPE &x, Bnode<TYPE> *l, Bnode<TYPE> *r)

: info(x)

{

left = l; right = r;

}

template<class TYPE>

Bnode<TYPE> *&Bnode<TYPE>::Left()

{

return (Bnode<TYPE> *)left;

}

template<class TYPE>

Bnode<TYPE> *&Bnode<TYPE>::Right()

{

return (Bnode<TYPE> *)right;

}

The Bnodeb and Bnode class definitions are given on disk in the files bnodeb.h and bnode.h.

Many of the binary tree operations are data independent. That's the reason for the Bnodeb class. We can write functions that take Bnodeb pointers as parameters and share these functions across all binary trees, regardless of the type of data actually stored in the nodes. This tames the source code explosion that might otherwise result from the use of templates.

We do pay a price for allowing data independence. When we need to use nodes with data, typecasting problems occur. For instance, the following code fragment will not compile without the typecast shown:

Bnode<int> *t;

...

t = (Bnode<int> *)(t->left); // Base to derived typecast

Presumably, all nodes in the tree t store integer data. However, left is typed as a pointer to Bnodeb object, so we need to typecast the expression t->left to be a Bnode<int> pointer. You've seen a similar problem with the linked list nodes described in Chapter 8. The solution is the same: provide convenient functions that hide the typecasting. That's the purpose of Left( ) and Right( ). We can rewrite our code fragment as:

Bnode<int> *t;

...

t = t->Left();

Note carefully that Left( ) and Right( ) actually return references to pointers. This approach allows us to use these functions on the left side of assignments. For instance:

Bnode<int> *t, *p;

...

t->Left() = p;

Actually, we don't need to use the Left( ) function in this case, due to the one-way assignment compatibility between derived and base types. For instance, we could write

Bnode<int> *t, *p;

...

t->left = p; // Implicit derived to base typecast

In the code that follows, we use the pointers left and right directly whenever we can, since the syntax is simpler. We use Left( ) and Right( ) only when necessary. All this typecasting is a bother, but we can't entirely eliminate it and still allow for the use of data-independent code.

The Bnode<TYPE> class has two constructors that are convenient for building trees. The constructors set the left and right pointers to null, unless you specify otherwise. In the following example, we'll build the tree shown in Figure 10.3 :

Bnode<char> a('a'), b('b'), c('c');

Bnode<char> times('*', &a, &b);

Bnode<char> plus('+', &times, &c);

You'll notice that Bnodeb and Bnode are declared to have all public members. With the linked list nodes described in Chapter 8, we used encapsulation to the fullest, being careful to make all members private when possible. For binary tree nodes, the logistics involved in using fully encapsulated objects get very tedious and messy. For instance, we'll write many standalone functions that operate on Bnodeb pointers--to provide maximum code sharing. While we could make all of these functions friends, and thus make the members of Bnodeb private, it's arguable that little is gained by doing this.

  TREE TRAVERSALS

Navigating through trees is at the heart of all tree operations, so it's useful to examine the common issues that arise in tree navigation. The methods we'll show for binary trees can be modified for more general trees.

A tree traversal is a method of visiting every node in the tree. By "visit," we mean that some type of operation is performed. For example, you may wish to print the contents of the nodes. Four basic traversals are commonly used:

1.     In a preorder traversal, first the root is visited, then the left subtree, and then the right subtree, recursively.

2.     In an inorder traversal, first the left subtree is visited, then the root, and then the right subtree, recursively.

3.     In a postorder traversal, first the left subtree is visited, then the right subtree, and then the root, recursively.

4.     In a level-order traversal, the nodes are visited level by level, starting from the root, and going from left to right.

Figure 10.4 shows the expression tree presented in Figure 10.3, and the results of traversing the tree using the four types of traversals (where the "visit" function prints node contents). In the context of expression trees, preorder traversal yields what is known as the polish notation for the expression, inorder traversal yields the infix notation (with parenthesis removed, though), and postorder traversal yields the reverse polish notation.

The traversals all have symmetrical cases. For instance, we've defined the traversals to go from left to right. They could alternatively be defined to go right to left, although that is less common in practice.

We'll now explore the first three traversal algorithms, since they can be coded similarly. Level-order traversal will be discussed later.

Recursive Traversals

The first three traversal algorithms are recursive in nature. They must walk down the tree and then back up. Recall, however, that the links of a tree go downward, not upward. Thus, to move back up the tree, each visited parent must be saved for later use. A stack is the natural data structure to use here. As you walk down the tree, parent node pointers can be pushed on the stack. When you go up the tree, these pointers can be popped from the stack. Later, we'll show how to use an explicit stack for this purpose. But first, we'll use the implicit processor stack by writing recursive functions for the traversal algorithms. Here are three such functions, along with a defined type, VisitFunc, which points to a function to be used for the visit action:

Figure 10.4 The four tree traversals.

typedef void (*VisitFunc) (Bnodeb *n);

void PreOrder(Bnodeb *t, VisitFunc Visit)

{

if (t) {

Visit(t);

PreOrder(t->left, Visit);

PreOrder(t->right, Visit);

}

}

void InOrder(Bnodeb *t, VisitFunc Visit)

{

if (t) {

InOrder(t->left, Visit);

Visit(t);

InOrder(t->right, Visit);

}

}

void PostOrder(Bnodeb *t, VisitFunc Visit)

{

if (t) {

PostOrder(t->left, Visit);

PostOrder(t->right, Visit);

Visit(t);

}

}

The only difference between these functions is the placement of the Visit( ) calls. The placements intuitively suggest the type of traversal that takes place. You'll notice that these functions use generic Bnodeb pointers, and thus they will work for all binary trees. The trick lies in defining the Visit( ) function to work with the type of nodes actually being used. For example, here's how we might set up traversals of the tree given earlier in Figure 10.3 :

void PrtChNode(Bnodeb *b)

{

cout << ((Bnode<char> *)b)->info << ' ';

}

Bnode<char> a('a'), b('b'), c('c');

Bnode<char> times('*', &a, &b);

Bnode<char> plus('+', &times, &c);

PreOrder(&plus, PrtChNode);  cout << '\n';

InOrder(&plus, PrtChNode);   cout << '\n';

PostOrder(&plus, PrtChNode); cout << '\n';

Note the typecast in PrtChNode( ). The function must conform to the function pointer type VisitFunc, and thus its parameter must be typed as a Bnodeb pointer. The function assumes, however, that the node is really a Bnode<char> node. All caveats apply for such typecasts.

Eliminating Tail Recursion

Since recursion uses processor stack space, it's desirable to eliminate or reduce the recursion as much as possible. In our tree-traversal algorithms, the amount of stack space consumed depends on the height of the tree. That is, for a tree of height n, n parent nodes are placed on the stack. So, for our recursive functions, the recursive calls are up to n levels deep.

Look carefully at the PreOrder( ) and InOrder( ) functions. In these functions, nothing follows the last recursive calls. This type of recursion is known as tail recursion. In a tail-recursive call, the local variables are saved on the processor stack, but they don't really need to be since no action follows the recursive call that needs those variables. Thus, tail recursion can be optimized by eliminating the recursive call and turning it into iteration. For example, here is the PreOrder( ) function written with tail recursion transformed into a while loop. The InOrder( ) function can be coded similarly:

void PreOrder(Bnodeb *t, VisitFunc Visit)

// Tail recursion eliminated

{

while(t) {

Visit(t);

PreOrder(t->left, Visit);

t = t->right;

}

}

Instead of a call to PreOrder(t->right), t is simply set to t->right and the while loop cycles until a null pointer is found. Note that the recursive call for t->left cannot be eliminated since this call is not tail recursive. Also, no call in PostOrder( ) is tail recursive, so we can't optimize that function any further.

Flattening out Recursion

In the recursive calls, the parent node pointer, the Visit function pointer, and the return address are all saved on the processor stack. However, we actually only need to save the parent node pointer. The recursion can thus be optimized by using an explicit stack that saves only node pointers. In doing so, we can also flatten out the recursion and turn it completely into iteration.

The following PreOrderFlat( ) function, modified from the PreOrder( ) function given earlier with tail recursion eliminated, shows the use of an explicit stack:

void PreOrderFlat(Bnodeb *t, VisitFunc Visit)

{

ListStk<Bnodeb *> path;

while(1) {

if (t) {

Visit(t);

// Simulate call PreOrderFlat(t->left, Visit);

path.Push(t);

t = t->left;

}

else {

// Simulate function return

if (path.Pop(t) == 0) break;

// Optimize the tail recursion

t = t->right;

}

}

}

Here, we've used the ListStk class from Chapter 9, holding pointers to Bnodeb nodes. This stack represents the current path of parents through the tree. Note how the call to PreOrder(t->left) is replaced by a Push( ), followed by the assignment t = t->left. In the original recursive function, the function returns when t is null. This is simulated by doing a Pop( ) from the stack. An InOrderFlat( ) function can be similarly defined. Simply move the Visit( ) call so that it immediately precedes the statement t = t->right.

It's not as easy to turn postorder traversal into iteration because we have no tail recursion to start with. In the PostOrder( ) function, two recursive calls are placed back to back. To use an explicit stack, we must keep track of which recursive call we are in so that we know how to simulate the function return. After the first recursive call, we must pop the parent pointer from the stack and then do another recursive call. After the second recursive call, we must pop the parent from the stack and then visit the node.

Figure 10.5 shows the three different movements that take place during a post order traversal:

1.     Down to the left

2.     Up from the left and down to the right

3.     Up from the right and then on up the tree.

We can keep track of these movements with a binary state variable. In one state, we're going down the tree. In the other state, we're going up the tree. However, we must clarify whether we're coming up from the left or right. Note that the stack only stores the parent pointer, not which side of the parent we are on. We could store that information as well, but there's a better way. After popping the parent from the stack, we can compare its left or right child pointer to the node we're currently on to determine which side we are coming from. The following PostOrderFlat( ) function utilizes this method:

void PostOrderFlat(Bnodeb *t, VisitFunc Visit)

{

ListStk<Bnodeb *> path;

int state = 0;

while(1) {

if (state == 0) {

// Ready to go down to the left

if (t) {

path.Push(t);

t = t->left;

}

else state = 1;

}

else { // State 1: Ready to come up from either direction

Bnodeb *c = t;

if (path.IsEmpty ()) break;

t = *path.Top();

if (c == t->left && t->right) {

// Coming back up the tree from the left, and

// since there is a right child, go right.

// Note that t is still on top of stack.

t = t->right;

state = 0;

}

else {

// Coming back up the tree from the right,

// or there was no right child, so visit

// the node and continue on up. (State

// stays at 1.)

Visit(t);

path.Pop( );

}

}

}

}

Figure 10.5 The three movements in a traversal.

This function has some further optimizations in it. Notice in state 1 that we inspect the top of the stack (which holds the parent), rather than popping it. With this approach, we don't have to turn around and push the parent back on the stack, in case we need to go down to the right.

The binary tree traversal functions can be found on disk in the file bnodeb.cpp.

To Recurse or Not to Recurse

At first glance, it appears you would always want to use the flat traversal functions since they use less stack space. But the flat versions aren't necessarily better. For instance, some overhead is associated with the use of an explicit stack, which may negate the savings we gain from storing only node pointers. Use of the implicit function call stack may actually be faster due to special machine instructions that can be used.

The flat traversal functions are useful when the node processing you need to do doesn't depend on where you are in the traversal. Printing the nodes in sequence is one example of this. However, for some tree operations, you must manage more variables than just the nodes themselves. The states of these variables must also be saved on some type of stack. For these operations, using an explicit stack can become more cumbersome than using straightforward recursion. For example, here is a postorder recursive function to find the height of a tree:

int Height(Bnodeb *t)

{

int h = 0;

if (t != 0) {

int lh = Height(t- >left);

int rh = Height(t- >right);

h = ((lh > rh) ? lh : rh) + 1;

}

return h;

}

The height of a tree is the maximum of the height of its two subtrees. In this case, it's easier to write the function recursively. The flat version is awkward and error prone.

Level-Order Traversal

The level-order traversal of a tree (where you go from top to bottom, left to right, level by level) is different from the other traversals because it is not recursive. A queue is needed instead of a stack. The following LvlOrder( ) function shows how the traversal can be accomplished:

void LvlOrder(Bnodeb *t, VisitFunc Visit)

}

ListQ<Bnodeb *> nodes_left;

nodes_left.Insert(t);

while(1) {

Bnodeb *p;

if (nodes_left.Extract(p) == 0) break;

Visit(p);

if (p->left != 0) nodes_left.Insert(p->left);

if (p->right != 0) nodes_left.Insert(p->right);

}

}

In this function, we maintain a queue of nodes to be processed. Nodes are pulled off the queue one at a time, visited, and then the children of the nodes are inserted onto the end of the queue. The net effect is a level order traversal, as shown earlier in Figure 10.4.

  TREE WALKING ITERATORS

One interesting way of performing tree traversals is to use iterator objects. With iterators, it's possible to sequence through a collection of objects at a high level. An iterator is first initialized to point to a particular collection of objects (such as a linked list, array, or tree) and then, using a function such as Next( ), we can ask the iterator to give us the next object in sequence. The iterator object keeps track of the state of the iteration between calls to Next( ). When used on trees, an iterator allows us to "flatten" out a tree and abstract what we mean by "next object."

The following TreeWalkerb class shows how to define an iterator for traversing a binary tree, using any of the four traversal orderings given earlier. This class is written generically using Bnodeb pointers and can thus be shared by all types of binary trees:

enum WalkOrder { pre_order, post_order, in_order, lvl_order };

class TreeWalkerb { // Tree-walking base class

protected:

ListSQ<Bnodeb *> path;

Bnodeb *root, *curr;

WalkOrder worder;

int state;

Bnodeb *NextPreOrder();

Bnodeb *NextInOrder();

Bnodeb *NextPostOrder();

Bnodeb *NextLvlOrder();

Bnodeb *(TreeWalkerb::* NextFptr)();

public:

TreeWalkerb(Bnodeb *r, WalkOrder w);

virtual ~TreeWalkerb();

void Reset();

void Reset(Bnodeb *r, WalkOrder w);

Bnodeb *Next ();

};

Complete code for the TreeWalkerb class is given on disk in the files twalk.h and twalk.cpp. In addition, a derived template class, TreeWalker, is provided to work with specific Bnode<TYPE> trees.

A TreeWalkerb iterator is set up in the constructor by specifying the root of the tree you would like to walk and the traversal order, using the enumerated type WalkOrder. A Reset( ) function does all the work:

TreeWalkerb: :TreeWalkerb(Bnodeb *r, WalkOrder w)

{

Reset( r, w);

}

void TreeWalkerb::Reset(Bnodeb *r, WalkOrder w)

{

root = r;

worder = w;

if (worder == pre_order) NextFptr = &TreeWalkerb: :NextPreOrder;

else if (worder == in_order) NextFptr = &TreeWalkerb: :NextInOrder;

else if (worder == post_order) NextFptr = &TreeWalkerb: :NextPostOrder;

else NextFptr = &TreeWalkerb: :NextLvlOrder;

state = 0; curr = root;

path.Clear();

if (root && worder == lvl_order) path.Insert(root);

}

An iterator object keeps track of the start of the tree being iterated, in the variable root, plus the node currently being traversed, in the variable curr, plus which traversal function to use in the function pointer variable NextFptr. Note that NextFptr is actually a pointer to a member function, which must be handled differently than ordinary function pointers. See [Stroustrup 91] for more details on member function pointers.

For an iterator object to work, it must keep the state of the iteration between calls to Next( ). Since our traversals our recursive, a stack is needed. For postorder traversals, we also need the binary state variable that indicates whether we're going up or down the tree. In addition, we need a queue for level-order traversals. Rather than use both stack and queue objects, we use a combination staque object instead, which we call path.

The Next( ) function is used to return a pointer to the next node in sequence. Next( ) makes a call to the traversal function pointed to by NextFptr. (Notice the pointer-to-member function call, using the ->* operator.)

Bnodeb *TreeWalkerb: :Next()

{

return (this->*NextFptr)();

}

Four functions are provided for each type of traversal. These functions are adaptations of the flat-traversal functions given earlier. The difference is that they are member functions and thus have access to the stack stored in the iterator object. Also, instead of making a call to Visit( ), the functions return a pointer to the node to be visited. Before doing the return, though, they must perform actions to ensure that the iterator can pick up where it left off in later calls to Next( ). As an example, here is the NextPreOrder( ) function. You'll notice that it's very similar to the PreOrderFlat( ) function given earlier in the chapter. The other traversal functions can be defined in the same way:

Bnodeb *TreeWalkerb::NextPreOrder()

{

while(1) {

if (curr) {

path.Push(curr);

Bnodeb *p = curr;

curr = curr->left;  // Set up for next visit

return p;           // So the node can be "visited"

}

else  {

if (path.Pop(curr) == 0) {

return curr = 0;  // Signal no more nodes

}

else curr = curr->right;

}

}

}

Here is an example of a function that will print out the nodes of a tree using a TreeWalkerb object with all four traversal orders:

void PrintNode(Bnodeb *t)

// Assumes node is actually a Bnode<char> node

{

cout << ((Bnode<char> *)t)->info << '\n';

}

void PrintTreeOrderings(Bnode<char> *root)

{

Bnodeb *nxt;

if (root == 0) { cout << "Tree is empty\n"; return; }

cout << "PreOrder:    ";

TreeWalkerb tw(root, pre_order);

while((nxt = tw.Next()) != 0) PrintNode(nxt);

cout << '\n';

cout << "InOrder:     ";

tw.Reset(root, in_order);

while((nxt = tw.Next()) != 0) PrintNode(nxt);

cout << '\n';

cout << "PostOrder:   ";

tw.Reset(root, post_order);

while((nxt = tw.Next()) != 0) PrintNode(nxt);

cout << '\n';

cout << "Level Order:     ";

tw.Reset(root, lvl_order);

while((nxt = tw.Next()) != 0) PrintNode(nxt);

cout << '\n';

}

A TreeWalkerb iterator has the same problems as the flat-traversal functions. It works great when the nodes can be processed out of context--that is, without regard to where the nodes are in the tree--but it isn't so great when that context must be used in the processing. For example, it's difficult to write the Height( ) function we gave earlier using an iterator. In contrast, it's easy (if somewhat inefficient) to write a function that counts the number of nodes in the tree, as shown in the following function:

int NumNodes(Bnodeb *t)

// Determine number of nodes in the tree t

{

Bnodeb *nxt;

int cnt = 0;

TreeWalkerb tw(t, in_order); // Any ordering will do

while((nxt = tw.Next()) != 0) cnt++;

return cnt;

}

  PRINTING TREES

When you debug code that works with binary trees, it's useful to print the trees, with the structure of the tree clearly shown. In this section, we'll provide two types of tree printing functions that help in this regard. These functions use the traversal functions given in the previous sections, and they assume very little about the type of screen or printer output available. All that's needed is the C++ iostream object cout to print the nodes line by line.

Printing Sideways

Printing Upright

Printing Sideways

The following PrintSideWays( ) function will print a tree sideways, with the root of the tree on the left and the rightmost node on the first line. Figure 10.6 shows an example.

Figure 10.6 A tree printed sideways.

typedef void (*PrtFunc)(Bnodeb *n);

void PrintSideWays(Bnodeb *t, PrtFunc PrtNode, int inc, int space)

{

while(t) {

PrintSideWays(t->right, PrtNode, inc, space+inc);

for(int i = 0; i<space; i++) cout << ' ';

PrtNode(t); cout << '\n';

t = t->left;

space += inc;

}

}

This function works by using a recursive inorder traversal of the tree. Note, however, that the traversal takes place from right to left, rather than the usual left to right strategy. This means the first node printed is the rightmost node. At each stage in the recursion, we keep track of the number of spaces to print before the node itself is printed. The space variable keeps track of the total (which should start at zero), and inc specifies the spacing to allow between nodes. Since the function is written generically using Bnodeb pointers, the function pointer PrtNode determines the actual function to call in doing the printing. This function is supposed to know the type of data actually stored in the nodes. Here's how PrintSideWays( ) can be called to print the tree given in Figure 10.6 :

void PrtChNode(Bnodeb *t) { cout << ((Bnode<char> *)t)->info; }

Bnode<char> *t;

... // Set up t to be the tree in Figure 10.6

PrintSideWays(t, PrtChNode, 3, 0);

Printing Upright

While printing sideways is straightforward, the output isn't as easy to use as output that's printed upright. However, upright printing is much harder to do, unless your output device supports random-access positioning. Here, we'll develop a routine that prints a tree upright, line by line.

The key to easy upright printing is to make two passes. The first pass determines the position of each node in the tree. Once the positions are determined, a second pass uses a level-order traversal of the tree to print the nodes line by line.

To keep things simple, we'll assume that each node in the tree is given its own column and that nodes at the same level (that is, having the same depth) are printed on the same line. Thus, the line coordinate (which we'll call y) is equal to the depth of the node. The column, or x coordinate, can be found by the following observation: If you examine the tree shown in Figure 10.7, you'll notice that the node having an x position of one will be the first node visited in an inorder traversal, the node having an x position of two will be the second node visited, and so on.

The following GetNodePosn( ) function uses an inorder traversal to determine the (x, y) positions of each node in a tree. Here, NodeWidth is a pointer to a function that's supposed to return the printing width of a given node:

typedef int (*NodeWidthFunc)(Bnodeb *n);

BnodeXY *GetNodePosn(Bnodeb *t, int &x, int y, NodeWidthFunc NodeWidth)

// Using an inorder traversal, figure out x and y of each node

// in the tree t, assuming that the top left corner of the

// box bounding t is at (x, y). Builds a parallel shadow tree

// of nodes having x and y in them.

{

if (t == 0) return 0;

BnodeXY *l = GetNodePosn(t->left, x, y+1, NodeWidth);

BnodeXY *xyt = new BnodeXY;

xyt->left = l;

xyt->x = x;

x += NodeWidth(t);

xyt->y = y;

y++;

xyt->right = GetNodePosn(t->right, x, y, NodeWidth);

return xyt;

}

Figure 10.7. A row- and column-aligned tree.

One major problem lies in finding a place to store the node position information. You could always include (x, y) fields in the nodes themselves. However, that would limit the generality of the function. The GetNodePosn( ) function uses a different technique. It builds an auxilary shadow tree that has the same structure of the tree being traversed. The nodes of this shadow tree keep track of node positions. The node type is given by the following BnodeXY structure, derived from Bnodeb:

struct BnodeXY : public Bnodeb {

int x, y;

BnodeXY *Left( )  { return (BnodeXY *) left; }

BnodeXY *Right( ) { return (BnodeXY *) right; }

} ;

After the node positions have been determined, the tree nodes can be printed line by line, using a level-order traversal, with the x position of the nodes determining the spacing between the nodes. The following PrintUp( ) function gives the details:

void PrintUp(Bnodeb *t, PrtFunc PrtNode, NodeWidthFunc NodeWidth)

{

int spacing, last_sp, x, oldy;

Bnodeb *nxt;

BnodeXY *xynxt;

if (t == 0) { cout << "Empty tree\n"; return; }

x = 1;

BnodeXY *xyt = GetNodePosn(t, x, 0, NodeWidth);

TreeWalkerb tw(t, lvl_order);

TreeWalker<BnodeXY> xytw(xyt, lvl_order);

oldy = 0;

last_sp = 0;

while(1) {

xynxt = xytw.Next();

if (xynxt == 0) break;

if (xynxt->y != oldy) {

oldy = xynxt->y;

cout << '\n';

last_sp = 0;

}

spacing = xynxt->x - last_sp - 1;

if (spacing < 0) spacing = 0;

while(spacing-) cout << ' ';

nxt = tw.Next();

PrtNode(nxt);

last_sp = xynxt->x + NodeWidth(nxt) - 1;

}

cout << "\n\n";

// Need to delete shadow tree. Easy to do with postorder traversal

xytw.Reset(xyt, post_order);

while(1) {

xynxt = xytw.Next();

if (xynxt == 0) break;

delete xynxt;

}

}

Note that two level-order iterators are used in PrintUp( ) (the latter uses the TreeWalker template class) to traverse both the main tree and the shadow tree in parallel. You may wonder why we need to store the y position of each node. During a level-order traversal using an iterator, there's no way to tell when we've gone to a new level. Remember that using an iterator loses the shape information of the tree. (In fact, none of the iterative traversals, by themselves, provide enough shape information to reconstruct the tree. See [Lings 86] for more discussion of tree reconstruction.) We detect when a new level has been reached by comparing the y coordinate of the current node with that of the previous node.

Figure 10.8 shows the printed result for the tree shown in Figure 10.7. We used the following code to print the tree:

void PrtChNode(Bnodeb *t)

{

cout << ((Bnode<char> *)t)->info;

}

int ChNodeWidth(Bnodeb *t)

{

return 1; // This is the printing width

}

Bnode<char> *t;

// Build tree in Figure 10.7

...

PrintUp(t, PrtChNode, ChNodeWidth);

Complete code for printing a tree sideways and upright is given in the files treeprt.h, treeprt.mth, and treeprt.cpp.

  BINARY SEARCH TREES

Next, we look at a special class of trees known as binary search trees. A binary search tree has binary nodes and the following additional property: Given a node t, each node to the left is "smaller" than t, and each node to the right is "larger." This definition applies recursively down the left and right subtrees. Figure 10.9 shows a binary search tree where characters are stored in the nodes.

Figure 10.9 also shows what happens if you do an inorder traversal of a binary search tree: you'll get a list of the node contents in sorted order. In fact, that's probably how the name inorder originated.

You may wonder what we mean by "smaller" and "larger" in our definition. This depends on the type of data being stored at the nodes. If, for instance, the data is integers, then the meaning of smaller and larger is obvious. More typically, though, the node data is some kind of structure, with one of the fields (called the key) used to make comparisons. Next, we'll explain how to search a binary tree and how to set up the comparisons for any type of node.

Binary Tree Searching

An algorithm for searching a binary search tree follows immediately from the binary search tree property. Although the search algorithm can be cast in a recursive fashion, it's just as easy (and more efficient) to write it iteratively. Here is a Search( ) function that shows how this is done:

Figure 10.8 A tree printed upright.

Figure 10.9 A binary search tree.

template<class TYPE>

Bnode<TYPE> *Search(Bnode<TYPE> *t, const TYPE &x)

// Returns pointer to a node matching the data x if

// such a node is in the tree t; otherwise, a 0 is returned

{

while (t) {

if (x == t->info) break;

t = (x < t->info) ? t->Left() : t->Right();

}

return t;

}

Figure 10.10 shows the path taken when searching for a node in a sample tree. The Search( ) function assumes that TYPE has the comparison operators '==' and '<' defined. In the case where TYPE is a built--in type--such as integer or character--this is easy because the operators are already defined. But suppose TYPE is a structure that holds a customer name and a unique integer id. We could use either field as the search key. Here's how you might define the customer structure using the name as the key:

struct Customer {

char name[80];

int id;

Customer(char *n, int i = 0; // Notice default argument

};

Customer::Customer(char *n, int i)

{

strncpy(name, n, 80);

name[79] = 0; // Ensure null termination

Id = i;

}

int operator==(Customer &a, Customer &b)

{

return !strcmp(a.name, b.name);

}

int operator<(Customer &a, Customer &b)

{

return strcmp(a.name, b.name) < 0;

}

... // Other operators similarly defined

Bnode<Customer> *custdb;

...//  Build the tree, then search it

if (Search(custdb, "Ford Prefect")) cout << "Customer found\n";

Figure 10.10 Searching a binary tree.

Note that the search string gets converted by the constructor into a Customer object (the Id is given a default value). Obviously, more direct and efficient arrangements are possible. Sometimes, only the key is stored in the nodes, along with a pointer to the actual data. That is also easy to set up. The details are highly application dependent.

Why Use Binary Search Trees?

Binary search trees provide an efficient way to search through an ordered collection of objects. Consider the alternative of searching an ordered list. The search must proceed sequentially from one end of the list to the other. On average, n/2 nodes must be compared for an ordered list that contains n nodes. In the worst case, all n nodes might need to be compared. For a large collection of objects, this can get very expensive.

The inefficiency is due to the one-dimensionality of a linked list. We would like to have a way to jump into the middle of the list, in order to speed up the search process. In essence, that's what a binary search tree does. It adds a second dimension that allows us to manuever quickly through the nodes. In fact, the longest path we'll ever have to search is equal to the height of the tree. The efficiency of a binary search tree thus depends on the height of the tree. We would like the height to be as small as possible. For a tree holding n nodes, the smallest possible height is logn, so that's how many comparisons are needed on average to search the tree.

To obtain the smallest height, a tree must be balanced, where both the left and right subtrees have approximately the same number of nodes. Also, each node should have as many children as possible, with all levels being full except possibly the last. Searching a well-constructed tree can result in significant savings. For example, it takes an average of 20 comparisons to search through a million nodes! Figure 10.11 shows an example of a well-constructed tree.

Unfortunately, trees can become so unbalanced that they're no better for searching than linked lists. Such trees are called degenerate trees. Figure 10.12 shows an example. For a degenerate tree, an average of n/2 comparisons are needed, with a worst case of n comparisons--the same cs for a linked list.

When nodes are being added and deleted in a binary search tree, it's difficult to maintain the balance of the tree. We'll investigate methods of balancing trees in the next chapter. In this chapter, we'll use simple tree-construction methods. These methods may result in very unbalanced trees, depending on what order we insert nodes in the tree.

Figure 10.11 A well-constructed binary search tree.

Figure 10.12 A degenerate binary search tree.

Inserting Nodes into a Binary Search Tree

When adding nodes to a binary search tree, we must be careful to maintain the binary search tree property. This can be done by first searching the tree to see whether the key we're about to add is already in the tree. If the key can't be found, a new node is allocated and added at the same location where it would go if the search had been successful. The following Add( ) function shows this technique:

template<class TYPE>

Bnode<TYPE> *Add(Bnode<TYPE> *&root, const TYPE &x, int &exists)

// Returns matching node, new or old. May modify the root pointer.

{

Bnode<TYPE> *p;

int side;

// Look for a place to insert the node

Bnode<TYPE> *t = SearchP(root, x, p, side);

if (t == 0) { // No matching node found

t = new Bnode<TYPE>(x); // left and right should be set to 0

if (p) {

if (side) p->right = t; else p->left = t;

}

else root = t; // No parent, so we have a new root

existed = 0;

}

else existed = 1;

return t;

}

Note that Add( ) calls a function, SearchP( ), to determine where the node should be added. The SearchP( ) function is a modification of Search( ) that returns the parent of the matching node, as well as the matching node itself:

template<class TYPE>

Bnode<TYPE> *

SearchP(Bnode<TYPE> *t, const TYPE &x, Bnode<TYPE> *&p, int &side)

// Returns matching node in tree t, along with parent. If

// matching node is t itself, a 0 is passed back for the parent.

// Returns 0 if no match found.

{

p = 0;

while (t) {

if (x == t->info) break;

p = t;

if (x < t->info) {

side = 0;

t = t->Left();

}

else {

side = 1;

t = t->Right();

}

}

return t;

}

The SearchP( ) function keeps track of which side of the parent the matching node occurs on--in the variable called side. This isn't strictly necessary because we could do an extra compare in Add( ) to determine the side. The following code fragment, taken from Add( ) and modified, shows how:

if (p) {

// Used to be: if (side) p->right = t; else p->left = t;

if (p->info < x) p->right = t; else p->left = t;

}

If the comparison is an expensive operation, it's best to use the side variable instead. Figure 10.13 shows what happens when we use the Add( ) function to add some nodes to a tree.

Figure 10.13 Inserting nodes into a binary search tree.

Deleting Nodes from a Binary Search Tree

Deletions from a binary search tree are more involved than insertions. Given a node to delete, we need to consider these three cases:

1.     The node is a leaf.

2.     The node has only one child.

3.     The node has two children.

Case 1 is easy to handle because the node can simply be deleted and the corresponding child pointer of its parent set to null. Figure 10.14 shows an example. Case 2 is almost as easy to manage. Here, the single child can be promoted up the tree to take the place of the deleted node, as shown in Figure 10.15.

Case 3, where the node to be deleted has two children, is more difficult. We must find some node to take the place of the one deleted and still maintain the binary search tree property. There are two obvious candidates: the inorder predecessor and the successor of the node. We can detach one of these nodes from the tree and insert it where the deleted node used to be. A moment's reflection should convince you that using either of these nodes as the replacement will still maintain the binary search tree property.

Figure 10.14 Deleting a leaf node.

Figure 10.15 Deleting a node that has one child.

The predecessor of a node can be found by going down to the left once, and then all the way to the right as far as possible. To find the successor, an opposite traversal is used: first to the right, and then down to the left as far as possible. Figure 10.16 shows the path taken to find both the predecessor and successor of a node in a sample tree.

Both the predecessor and successor nodes are guaranteed to have no more than one child, and they may have none. (Think about how the traversals take place: they always end when the corresponding left or right child is null.) Because of this, detaching the predecessor or successor reduces to either Case 1 or 2, both of which are easy to handle. In Figure 10.17, we delete a node from the tree and use its successor as the replacement.

For the replacement, many texts suggest that you copy the data of the replacement node into the node that would have been detached, and then detach the replacement. However, if the node data is large, this involves a lot of data movement. In fact, it defeats one of the reasons for using a non-contiguous data structure! We'll perform the replacement by leaving the node data where it is, and use pointer surgery instead. That's the approach taken by the following DetachNode( ) function. This function is written generically using Bnodeb pointers, reinforcing the fact that no data is actually moved:

Bnodeb *DetachNode(Bnodeb *&root, Bnodeb *t, Bnodeb *p, int side)

// Detaches node t with parent p from the tree. Node t is

// the left child if side = 0; otherwise, it's the right child.

// If p is 0, then it is the root, and that is handled

// accordingly. Redundantly returns the pointer t. May

// have to update root pointer.

{

Bnodeb *psucc, *replacement;

if (t) {

if (t->left == 0 || t->right == 0) {

// At least one child is null, so use the other

// as the replacement. (It may be null, too.)

replacement = (t->left) ? t->left : t->right;

}

else { // Neither child is null

psucc = ParentOfSuccessor(t); // Guaranteed not null

if (psucc == t) { // Immediate successor

replacement = psucc->right;

}

else {

// Detach replacement from its current location

// and relocate it to where t used to be

replacement = psucc->left;

psucc->left =  psucc->left->right;

replacement->right = t->right;

}

// Finish relocating replacement to go where t used to be

replacement->left = t->left;

}

if (p) { // Fix parent of t to point to replacement

if (side)

p->right = replacement; else p->left = replacement;

}

else root = replacement; // No parent, so t was the root

}

return t;

}

Figure 10.16 Finding predecessor and successor nodes.

(a) Predecessor of 'e'

(b) Successor of 'e'

Figure 10.17 Deleting a node that has two children.

(a) Before deleting 'e'

(b) After deleting 'e'

The DetachNode( ) function uses the following ParentOfSuccessor( ) function to find the parent of the successor node. We want the parent so that we can easily detach the successor from its original location:

Bnodeb *ParentOfSuccessor(Bnodeb *t)

// If no successor, a 0 is returned

{

Bnodeb *p = 0;

// Go right, then all the way left

Bnodeb *q = t->right;

if (q) {

p = t;

while(q->left) {

p = q;

q = q->left;

}

}

return p;

}

Both DetachNode( ) and ParentOfSuccessor( ) can be written generically with Bnodeb pointers. We don't need to do any key comparisons since the structure of the tree lets us find the nodes we're looking for.

The DetachNode( ) function is called by higher-level functions that determine which node to detach. For example, here's a Detach( ) function that first searches for a node having matching data to x, and then proceeds to detach that node:

template<class TYPE>

Bnode<TYPE> *Detach(Bnode<TYPE> *&root, const TYPE &x)

// Detaches node matching x from the tree.

// May have to update the root pointer.

{

int side;

Bnode<TYPE> *p, *t;

t = SearchP(root, x, p, side);

return (Bnode<TYPE> *)DetachNode((Bnodeb *)root, t, p, side);

}

Finally, to actually deallocate the node, we use the following Delete( ) function:

template<class TYPE>

int Delete(Bnode<TYPE> *&root, const TYPE &x)

// Returns 1 if node actually found; otherwise, returns 0

// May have to update the root pointer

{

Bnode<TYPE> *n = Detach(root, x);

delete n;

return n != 0;

}

Clearing and Copying Binary Trees

Deleting all the nodes from a tree can be done recursively, using a postorder traversal. The following Clear( ) function clears a tree by calling a recursive DelTree( ) function:

template<class TYPE>

void Clear(Bnode<TYPE> *&root)

{

DelTree(root);

root = 0;

}

template<class TYPE>

void DelTree(Bnode<TYPE> *t)

{

if (t == 0) return;

DelTree(t->Left());

DelTree(t->Right());

delete t;

}

By using postorder traversal, the nodes are deleted from the bottom up. Note that it would be difficult to use any other type of traversal here because we would have to delete nodes before we're through with them. The DelTree( ) function can alternatively be written non-recursively, using a postorder iterator object:

template<class TYPE>

void Bstree<TYPE>::DelTree(Bnode<TYPE> *t)

{

TreeWalker< Bnode<TYPE> > tw(t, postorder);

Bnode<TYPE> *nxt;

while((nxt =  tw.Next()) != 0) delete nxt;

}

To copy a tree, you must first clear the destination tree and then duplicate the nodes of the source tree. The following Copy( ) function calls the Clear( ) function and then calls a recursive DupTree( ) function to do the node duplication. The latter function uses postorder traversal to build the duplicate tree from the bottom up:

template<class TYPE>

int Copy(Bnode<TYPE> *&root, const Bstree<TYPE> &t)

{

Clear(root);

Bnode<TYPE> *r = DupTree(t.root);

if (r) {

root = r;

return 1;   // Success

}

else return 0; // Failure

}

template<class TYPE>

Bnode<TYPE> *DupTree(Bnode<TYPE> *t)

{

if (t == 0) return 0;

Bnode<TYPE> *l = DupTree(t->Left());

Bnode<TYPE> *r = DupTree(t->Right());

return new Bnode<TYPE>(t->info, 1, r);

}

Here's an interesting twist to the technique for copying trees: Use an iterator object on the source tree, traverse the tree in some order, and insert duplicated nodes into the destination tree by using the Add( ) function. The following alternative Copy( ) function does this:

template<class TYPE>

int Copy(Bnode<TYPE> *&root, const Bstree<TYPE> &t, WalkOrder w)

{

Clear(root);

TreeWalker< Bnode<TYPE> > tw(t.root, w);

Bnode<TYPE> *nxt;

int dmy;

while((nxt = tw.Next()) != 0) Add(root, nxt->info, dmy);

return 1;

}

Figure 10.18 shows the results when we use the four different types of traversals on the source tree during copying. Notice that both preorder and level-order traversals produce a destination tree that has the same structure as the source tree. A postorder traversal yields a different structure, but it's still a valid binary search tree.

As you can see, the order that you insert nodes into a binary search tree does make a difference. In fact, if you insert nodes into a tree in sorted order, as exemplified by the inorder traversal copying in Figure 10.18, the result is a degenerate tree. This fact suggests you should avoid inserting nodes into a binary search tree in sorted order. In the next chapter, we'll look at ways of circumventing this problem.

Figure 10.18 Copying via source tree traversals.

  THREADED BINARY TREES

You'll notice that operations like Add( ) and Delete( ) are awkward due to the unidirectional capability of the links. We must write the operations in terms of finding the parent of a desired node, rather than find the node directly. This problem is similar to those we had with singly-linked lists in Chapter 8. The clumsiness can be alleviated by adding an extra pointer in each node, which points to the parent of the node. The result is known as a threaded binary tree. Figure 10.19 shows an example. A threaded tree is to a normal tree what a doubly-linked list is to a singly-linked list--it allows easy traversal in both directions. Note that, in a strict sense, a threaded binary tree isn't a true tree because it has cycles.

By using a threaded binary tree, it's possible to write the tree-traversal routines without recursion and without a stack. Since a stack isn't needed, tree operations can be faster, due to reduced overhead. However, before you try to reimplement all the algorithms you've seen in this chapter, keep in mind that the price you pay is space overhead in the tree nodes. (This is the same speed vs. space tradeoff you've seen repeatedly.) You pay for the overhead of parent pointers with every node, regardless of whether you need those pointers. When you use a stack, you pay for the overhead of the stack only when needed, and that overhead grows with the height of the tree, not with the number of nodes in the tree.

Figure 10.19 A threaded binary tree.

We're not recommending that you avoid using parent pointers, only that you be sure to consider the tradeoffs. Code without parent pointers is only slightly more complicated than code with them. After the operations are coded, the fact that they may be a little clumsly becomes a moot point. For maximum speed, however, you may want to use parent pointers.

As a final note, we should tell you that you can implement other forms of threaded trees. For example, one particular type uses extra pointers that allow you to directly find the next and previous nodes assuming an inorder traversal. See [Horosahni 90] for details.

  BINARY TREES AS CONCRETE DATA TYPES

We've used standalone functions to implement the basic binary tree operations. However, it might be useful to encapsulate these into a class and to make binary trees look like concrete data types. The following Bstree class shows an example:

template<class TYPE>

class Bstree { // A binary tree of nodes holding TYPE data

protected:

Bnode<TYPE> *root;

Bnode<TYPE> *SearchP(const TYPE &x, Bnode<TYPE> *&p, int &side);

virtual Bnode<TYPE> *AllocNode

(const TYPE &x, Bnode<TYPE> *1=0, Bnode<TYPE> *r=0);

Bnode<TYPE> *DupTree(Bnode<TYPE> *t);

virtual void FreeNode(Bnode<TYPE> *n);

void DelTree(Bnode<TYPE> *t);

public:

Bstree();

Bstree(const Bstree<TYPE> &t);

virtual ~Bstree();

void operator=(const Bstree<TYPE> &t);

int Copy(const Bstree<TYPE> &t);

int Copy(const Bstree<TYPE> &t, WalkOrder w);

void Clear();

Bnode<TYPE> *Member(const TYPE &x);

const Bnode<TYPE> *Member(const TYPE &x) const;

Bnode<TYPE> *Add(const TYPE &x, int &existed);

Bnode<TYPE> *Add(const TYPE &x);

Bnode<TYPE> *Detach(const TYPE &x);

Bnode<TYPE> *DetachMin();

Bnode<TYPE> *DetachMax();

int Delete(const TYPE &x);

void DeleteMin();

void DeleteMax();

Bnode<TYPE> *Root();

Bnode<TYPE> *Min();

Bnode<TYPE> *Max();

int IsEmpty() const;

};

Complete code for the Bstree class is given on disk in the files bstree.h, bstree.mth, and bstreeb.cpp. A test program is provided in tstbs.cpr. This program uses the tree-walking and tree printing functions given earlier in the chapter.

A Bstree object serves as a handle to a binary tree. Stored in each Bstree object is a pointer, root, to the top node of the tree. Many of the functions we gave earlier passed the root of the tree as the first argument. These functions can be adapted for the Bstree class by removing the first argument, and relying instead on the implicit this pointer that's passed. For example, here's the new Add( ) function:

template<class TYPE>

Bnode<TYPE> *Bstree<TYPE>::Add(const TYPE &x, int &exists)

{

Bnode<TYPE> *p;

int side;

Bnode<TYPE> *t = SearchP(x, p, side);

if (t == 0) {

t = AllocNode(x);

if (p) {

if (side) p->right = t; else p->left = t;

}

else root = t;

existed = 0;

}

else existed = 1;

return t;

}

  SUMMARY

The Bstree class is set up like most the concrete data type classes in this book--with ample constructors, destructors, overloaded assignment operators, and so on. Whether or not you need all these functions is, of course, up to you. We've added some functions that retrieve and delete the minimum and maximum nodes from the tree. The minimum node can be found by going left as far as possible. The maximum node can be found by going right as far as possible.

Similar to the linked-list classes defined in Chapter 8, we've provided the virtual functions AllocNode( ) and FreeNode( ), which you can override to change how the tree nodes are allocated. For example, a binary tree can be stored contiguously using an array of nodes, as well as on the heap with non-contiguous nodes.

A few hints for building trees with arrays: You'll want to maintain a singly-linked free list of deleted tree nodes. This can be done by reusing the left (or, alternatively, right) child pointers to serve as the free list's next pointers. Also, nothing prevents you from dispensing with pointers altogether and instead using two parallel arrays of indexes that correspond to the left and right child links.

If you don't plan on overriding AllocNode( ) and FreeNode( ), you can declare them as static, along with the recursive functions DupTree( ) and DelTree( ). This will result in more efficient code, since in the recursive function calls, the implicit this pointer won't be passed.

Go to Chapter 11            Return to Table of Contents


Practical Data Structures: BALANCED TREES

 Next Chapter Return to Table of Contents Previous Chapter

CHAPTER 11: BALANCED TREES

In Chapter 10, we explained that, for maximum efficiency, a binary search tree should be balanced. However, the tree-construction algorithms we gave can lead to decidedly unbalanced trees.

In this chapter, we'll explore some of the techniques you can use to keep trees balanced. In particular, we'll look at 2-3-4 trees and red-black trees.

DEGENERATE TREES

Very unbalanced trees are referred to as degenerate trees, so called because their searching performance degenerates to that of linked lists. Figure 11.1 shows two examples of degenerate trees. The first tree was built by inserting the keys a through i in sorted order into a binary search tree. The second tree was built using the insertion sequence a-i-b-h-c-g-d-f-e. This pathological sequence is often used to test the balancing capability of a tree. Figure 11.2 shows what the tree in Figure 11.1(b) would look like if it were balanced.

INTERNAL AND EXTERNAL NODES

At this point, we need to define two terms: internal node and external node. An internal node is a tree node having at least one key, and possibly some children. So far, all we've shown are internal nodes. It's sometimes convenient to have another type of node, called an external node, and pretend that all null child links point to such a node. An external node doesn't exist, but serves as a conceptual placeholder for nodes to be inserted.

Figure 11.1 Degenerate trees.

In this chapter, we'll draw internal nodes using circles, with letters as labels. External nodes are denoted by drawing links leading nowhere, and perhaps labelled using numbers, or they may not be drawn at all. Figure 11.3 shows a sample tree illustrating both internal and external nodes.

Sometimes we'll draw only a portion of a tree, with the top node of this portion having an implied parent (or no parent at all if it's the root). Subtrees are indicated using the same notation as external nodes, by drawing links leading nowhere and having numbers for labels. In fact, you can consider an external node to be an empty subtree.

There is a reason for treating external nodes and subtrees the same. In the tree transformations to come, we'll strive to maintain certain properties of the trees. In the drawings we'll show, those properties will be maintained regardless of whether labelled links are external nodes or subtrees, assuming the properties held before the transformations, of course.

Figure 11.2 A balanced tree.

Figure 11.3 Internal and external nodes.

BALANCING ACTS

There are two basic ways used to keep trees balanced:

1. Use tree rotations.

2. Allow nodes to have more than two children.

Tree Rotations

Multi-Way Trees

Tree Rotations

Certain types of tree-restructurings, known as rotations, can aid in balancing trees. Figure 11.4 illustrates the two types of single rotations.

The following sample code for single rotations uses generic Bnodeb pointers (as defined in Chapter 10), and assumes that all relevant pointers aren't null:

Bnodeb *RotateRight(Bnodeb *p, Bnodeb *t)

// Rotate right around p, assuming t is p's left child

// Returns t, which will be the new child of p's parent

{

p->left = t->right;

t->right = p;

return t;

}

Bnodeb *RotateLeft(Bnodeb *p, Bnodeb *t)

// Symmetrical case

{

p->right = t->left;

t->left = p;

return t;

}

Figure 11.4 Single rotations.

(a) Right rotation

(b) Left rotation

Another type of rotation is known as a double rotation. Figure 11.5 shows the two symmetrical cases. Here's the sample code:

Bnodeb *RotateLeftRight(Bnodeb *g, Bnodeb *p, Bnodeb *t)

// Rotate first around p, then around g. Assumes p is left

// child of g, and t is the right child of p. Returns new

// child of g's parent.

{

p->right = t->left;

t->left = p;

g->left = t->right;

t->right = g;

return t;

}

Bnodeb *RotateRightLeft(Bnodeb *g, Bnodeb *p, Bnodeb *t)

// Symmetrical case

{

p->left = t->right;

t->right = p;

g->right = t->left;

t->left = g;

return t;

}

Figure 11.5 Double rotations.

(a) Left-right rotation

(b) Right-left rotation

Both single and double rotations have one important feature: inorder traversals of the trees before and after the rotations are preserved. Thus, rotations help balance trees, but still maintain the binary search tree property.

Rotations don't guarantee a balanced tree, however. For example, Figure 11.6 shows a right rotation that makes a tree more unbalanced. The trick lies in determining when to do rotations and what kind of rotations to do. The red-black trees discussed later have certain rules used to determine when and how to rotate.

Multi-Way Trees

We can also help keep trees balanced by allowing nodes to have more than two children. Instead of using binary trees, we can have multi-way trees. A multiway tree node with n children is known as an n-node. An n-node has n-1 keys that are used for comparisons. For example, Figure 11.7 shows a 4-node with three keys. Nodes with keys less than a are inserted down subtree 1. Nodes greater than a but less than b are inserted down subtree 2. Nodes greater than b but less than c are inserted down subtree 3. And finally, nodes with keys greater than c are inserted down subtree 4.

Figure 11.6 An unbalancing rotation.

For a given set of keys, a balanced multi-way tree will have a smaller height than a balanced binary tree, because more keys can be stored per node. In our 4-node example, we can collapse what would be two or three levels of a binary tree into one level. This ability to collapse levels makes multi-way trees easier to balance than binary trees.

One special type of multi-way tree is known as a B-tree, which we'll study in depth in Chapter 14. A B-tree of order m has nodes that can have up to m children. In this chapter, we'll look at B-trees of order four. Such trees are often called 2-3-4 trees. A 2-3-4 tree has three types of nodes: 2-nodes (that is, binary nodes), 3-nodes, and 4-nodes. Figure 11.8 shows a sample 2-3-4 tree.

Insertion into a 2-3-4 tree works as follows: First, search the tree looking for the node to insert the new key into. If the candidate is a 2-node, then insert the key, making the node a 3-node. If the candidate is a 3-node, then insert the key, making the node a 4-node. If the candidate is a 4-node, it is full and must be split. The node is split into two 2-nodes, with the left key going to the left 2-node, the middle key moving up to the parent node, and the right key going to the right 2-node. The key to be inserted is then added to the appropriate 2-node.

Figure 11.9 shows how to split a 4-node, where there is no parent. The middle key is pushed up to a new root. If there is a parent, the parent itself may have to be split. The splitting process then propagates up the tree. Thus, insertion consists of a top-down searching pass, followed by a bottom-up splitting pass. Figure 11.10 shows an example of building a 2-3-4 tree using the same pathological sequence a-i-b-h-c-g-d-f-e that we've used before.

One consequence of 4-node splitting is that the tree grows upward, rather than downward as in a binary tree. As a result, the external nodes of a 2-3-4 tree are always on the same level, and the tree is quite balanced. (These same properties hold in general for all B-trees.)

Figure 11.7 A 4-node.

Figure 11.8 A 2-3-4 tree.

We won't actually implement 2-3-4 trees using the three different types of nodes. You can actually represent 2-3-4 trees using binary nodes, as we'll discuss next.

  RED-BLACK TREES

A red-black tree is a binary representation of a 2-3-4 tree. (Red-black trees can be used to represent a wide variety of trees. See [GuibSedg 78].) In a red-black tree, the links are given the colors red and black. These colors are used in mapping the 2-node, 3-node, and 4-nodes of a 2-3-4 tree into binary nodes. Figure 11.11 shows the mappings. In this figure and others to follow, red links are shown with thick lines, whereas black links are shown with lines of normal thickness. A node having a black parent link is referred to as a black node. A node with a red parent link is a red node. In the diagrams, we chose to color the links using thin/thick lines (rather than actually coloring the nodes).

The red-black mappings work as follows: a 2-node is represented using a single binary node with two black children. A 3-node is represented as a pair of binary nodes--one red, one black. Hence there is a red link between the nodes. Note that 3-nodes can be represented with two different orientations. The orientation used depends on the balancing required, as you'll see later. A 4-node is represented as a set of three binary nodes, with the parent node black and the child nodes red. Hence, two red links are used.

Figure 11.9 Splitting a 4-node.

(a) Original

(b) Split in two

(c) 'h' added

Figure 11.10 Building a 2-3-4 tree.

Three basic properties emerge from the way the red-black mappings are defined:

1.     There's no way to have two red links in a row on any path in the tree. That's because there's no way to map this arrangement back to a 2-3-4 tree. (What kind of node would have both a red parent and a red child?)

2.     The external nodes of a red-black tree have the same black-depth. The black-depth of a node is the number of black nodes found on the path from the root to that node. Remember that red links correspond to 3-nodes and 4-nodes, and thus don't contribute to the height of the equivalent 2-3-4 tree. This fact, coupled with the fact that external nodes of the 2-3-4 tree are at the same height, means that external nodes of a red-black tree have the same black-depth.

3.     The root node is always black. Since the root doesn't have a parent, there's no way for it to have a red parent link.

Figure 11.12 shows the red-black tree equivalent of the tree given in Figure 11.10. Note that the red-black tree is structurally the same as the binary tree given earlier in Figure 11.2. The only difference is the coloring of the links. The link colors serve as guideposts in determining what kind of rotations to do when balancing the tree. Note that the external nodes all have a black-depth of 2. This corresponds to the height of the equivalent 2-3-4 tree.

Figure 11.11 Red-black mappings for 2-3-4 nodes.

The tree shown in Figure 11.12 is fairly balanced, but isn't as balanced as its 2-3-4 tree counterpart, and is twice as high. Why the difference? The 3-nodes and 4-nodes of the 2-3-4 tree must be simulated using multiple binary nodes, each set consuming two levels of the tree. However, it can be shown that no path in a red-black tree is longer than twice as long as the shortest path in the tree. Thus, although the red-black trees aren't perfectly balanced, their worst case isn't all that bad.

Figure 11.13 illustrates an illegal red-black tree, with two consecutive red links in a row (d-h and h-i). Also, the external nodes have different black-depths as indicated in the figure. The resulting tree is still a valid binary search tree, but because it doesn't have the strict red-black tree properties, updates to the tree (assuming they could even be done) would tend to produce an unbalanced tree.

Figure 11.12 A red-black tree.

Red-Black Tree Classes

We'll represent red-black tree nodes with the following RBnodeb and RBnode classes:

#define RED 0

#define BLACK 1

struct RBnodeb : Bnodeb {

char color;

RBnodeb *&Left( );

RBnodeb *&Right( );

};

template<class TYPE>

struct RBnode : Rbnodeb {

TYPE info;

RBnode( );

RBnode(const TYPE &x, char c, RBnode<TYPE> *1=0, RBnode<TYPE> *r=0);

RBnode<TYPE> *&Left( );

RBnode<TYPE> *&Right( );

};

As with all designs, it's worthwhile to strive for data independence whenever possible. That's the purpose of the RBnodeb class. This class is derived from the Bnodeb class of Chapter 10, and adds a color field, but no other data. The color field represents the color of the link coming into the node--0 for red, 1 for black. We've represented color as a character, which represents additional overhead in every red-black node. Actually, only one bit is required. In your own classes, you may be able to arrange this bit to be stored in some unused location in the node's data field. An alternative representation would have colors associated with the child links, rather than the nodes themselves. However, this requires two bits per node, rather than one.

Figure 11.13 An illegal red-black tree.

Since RBnodeb is derived from Bnodeb, it can be used in many generic binary tree functions--such as the tree-traversal functions or the tree-walking iterators. The RBnode class is a template derived from RBnodeb that adds data to the node. Unfortunately, an RBnode object cannot be substituted for a Bnode object (a binary node with data), since the info field of these two types of nodes is at a different offset in the node structure. As usual, convenience functions Left( ) and Right( ) are supplied to help alleviate type casting problems. The details are given on disk.

The RBtree class implements red-black trees as concrete data types, with the appropriate constructors, destructors, assignment functions, and other supporting functions. This class is very similar in design to the Bstree class developed in Chapter 10. We'll look only at the main operations of searching, inserting, and deleting. The remaining details are given on disk.

template<class TYPE>

class RBtree {

protected:

RBnode<TYPE> *root;

// Other protected functions--such as AllocNode( ), DupTree( ),

// FreeNode( ), and DelTree( )...

public:

RBnode<TYPE> *Member(const TYPE &x);

RBnode<TYPE> *Add(const TYPE &x, int &existed);

RBnode<TYPE> *Detach(const TYPE &x);

int Delete(const TYPE &x);

// Constructor functions, plus other functions for copying,

// clearing, and deleting trees, and functions for handling

// min/max elements, and so forth...

};

Complete code for red-black trees is given on disk in the files RBnode.h, RBtree.h, RBtreeb.cpp, and RBtree.mth. A test program is given in the files tstrb.cpp and tsttree.cpp. Many of the files from Chapter 10 are used as support files.

Searching a Red-Black Tree

A red-black tree can be searched in the same way as an ordinary binary search tree, as the following Member( ) function shows. Note that the colors of the nodes are not used in the search. The fact that the tree is balanced in no way impinges on the searching code (other than to make the searches faster due to shorter trees, of course).

template<class TYPE>

RBnode<TYPE> *RBtree<TYPE>::Member(const TYPE &x)

// Returns pointer to node containing data x if

// such a node is in the tree t; otherwise, a 0 is returned

// ASSUMES TYPE has '<' and '==' defined.

{

RBnode<TYPE> *t = root;

while (t) {

if (x == t->info) break;

t = (x < t->info) ? t->Left( ) : t->Right( );

}

return t;

}

Inserting into a Red-Black Tree

We described an insertion process earlier for 2-3-4 trees that involved two passes: a top-down pass to determine where the new key should go, and a bottom-up pass to handle any full 4-nodes that need to be split. It's possible to do the insertion in a single top-down pass, and that's the method we'll use for red-black trees.

Single-pass top-down insertions can be done by ensuring that the new key won't be added to a full leaf node. We can do this by splitting any 4-node we see on the path to the leaf into two 2-nodes. Splitting a 4-node, as represented with red-black nodes, is particularly easy. As Figure 11.14 illustrates, the process involves three color flips, and that's all. Note how the two child links are turned to black, but the parent link is turned to red. This is done to preserve the black-depths of the external nodes 1, 2, 3, and 4 (which may be subtrees). In all the red-black tree transformations to follow, we always ensure that the relative black-depths of the external nodes or subtrees are preserved. That way, we can guarantee that the new tree will still be a valid red-black tree. A side effect is that the tree will be kept as balanced as possible.

Since the top node of the split 4-node becomes red during a split, a problem occurs if its parent is also red, because we then end up with two reds in a row. This must be fixed by using appropriate rotations. We'll look at all the possible cases now.

Figure 11.15 shows the two orientations of a 4-node connected to a 2-node. In both orientations, we end up with a 3-node having two 2-nodes as children when the 4-node is split. The results are still legal red-black trees, so nothing else has to be done.

Figure 11.14 Splitting a 4-node.

Figure 11.16 illustrates the three cases that occur when a 4-node is connected to a 3-node. Here, the 4-node can be the left child, middle child, or right child of the 3-node. Recall that, in a red-black representation, a 3-node has two orientations. Thus, there are six cases to consider in the red-black tree representations. In Figure 11.17, we show three of these cases. Figures 11.17(a) and 11.17(b) show the two cases that arise when the 4-node is the right child of the 3-node, with the two possible 3-node orientations. Not shown are the two symmetrical cases that occur when the 4-node is the left child. Figure 11.17(c) shows one of the cases that occurs when the 4-node is the middle child. Again, there is a symmetrical case not shown.

In Figure 11.17(a), splitting the 4-node leads to a valid red-black tree, so nothing more needs to be done other than the color flips. However, in Figures 11.17(b) and 11.17(c), splitting the 4-node leads to trees having two reds in a row. Thus, after the color flips, rotations must be performed to restore the redblack property. For the case in Figure 11.17(b), a single rotation is needed. For the case in Figure 11.17(c), a double rotation is required. Note that the colors "follow" the rotations. In all three figures, you should verify for yourself that the black-depths of the external nodes are preserved.

Figure 11.15 Splitting a 4-node connected to a 2-node.

(a) Connected to the left

(b) Connected to the right

Figure 11.16 Splitting 4-nodes connected to 3-nodes.

We don't need to be concerned about any 4-node to 4-node cases. That's because the parent 4-node will be split before we get to the child 4-node.

The insertion code is given in the routines Add( ) and InsBalance( ). The Add( ) function walks down the tree looking for a place to insert the new key, splitting any 4-nodes along the way. After a split, there may be two red nodes in a row. If so, a call is made to InsBalance( ), which performs the proper rotations. Add( ) keeps track of the current node t, its parent p, grandparent g, and great-grandparent gg, all of which are needed to do the rotations. To make the call to InsBalance( ) efficient, the pointers are packaged up into the structure PtrPak, and a single parameter is then passed for these pointers. A pointer to the root node is also passed, since the root may change due to a rotation.

Figure 11.17 Splitting 4-nodes connected to 3-nodes, red-black style.

When the bottom of the tree is reached and the key to be inserted isn't found, a new node is added, colored red, and if needed, an adjusting balance is performed. (It's useful to pretend the new node is a black 4-node that is split, with its colors adjusted like any split 4-node.) If the key already exists, a flag is set to indicate that. It's important to realize that Add( ) may cause the tree to be restructured even if the key already exists.

If the root is a 4-node before the insertion, it will be "split" and the root becomes red. The Add( ) function corrects this situation by always setting the root black. During this color change, the black-depths of the external nodes increase by one, but the tree remains valid, since the black-depths are still all identical.

struct PtrPack {

RBnodeb *t, *p, *g, *gg, *s, *rs, *ls, *m, *pm;

} ;

template<class TYPE>

RBnode<TYPE> *RBtree<TYPE>::Add(const TYPE &x, int &exists)

// Add the key x to the tree if it doesn't already exist.

// Returns a pointer to the node containing the key, and

// sets the exists flag accordingly.

{

PtrPack pp;

int side;

exists = 1;

pp.t = root; pp.p = 0; pp.g = 0; pp.gg = 0;

while(pp.t && x ! = ((RBnode<TYPE> *)(pp.t))->info) {

// If on a 4-node, we must split it into two 2-nodes

if ((pp.t->left && pp.t->Left( )->color == RED) &&

(pp.t->right && pp.t->Right( )->color == RED)) {

pp.t->color = RED;

pp.t->Left( )->color = BLACK;

pp.t->Right( )->color = BLACK;

// Check for two reds in a row, and adjust if so

if (pp.p && pp.p->color == RED)

root = (RBnode<TYPE> *)InsBalance(root, pp);

// else Figure 11.15 or Figure 11.17(a)

}

pp.gg = pp.g; pp.g = pp.p; pp.p = pp.t;

if (x < ((RBnode<TYPE> *)(pp.t))->info) {

pp.t = pp.t->Left( ); side = 0;

}

else {

pp.t = pp.t->Right( ); side = 1;

}

}

// Reached the bottom, with no matching node

if (pp.t == 0) {

exists = 0;

pp.t = AllocNode(x, RED);

if (pp.t == 0) return 0; // Couldn't add

if (pp.p) {

if (side) pp.p->right = pp.t; else pp.p->left = pp.t;

}

else root = (RBnode<TYPE> *)(pp.t);

// Check for two reds in a row, and adjust if so

if (pp.p && pp.p->color == RED)

root = (RBnode<TYPE> *)InsBalance(root, pp);

}

root->color = BLACK; // Root always made black

return (RBnode<TYPE> *)(pp.t);

}

// Define some macros for convenience

#define T  (pp.t)

#define P  (pp.p)

#define G  (pp.g)

#define GG (pp.gg)

#define S  (pp.s)

#define RS (pp.rs)

#define LS (pp.ls)

#define M  (pp.m)

#define PM (pp.pm)

RBnodeb *InsBalance(RBnodeb *root, PtrPack &pp)

// Balance adjusting for top-down insertions. Eliminates

// both p and t from being red by doing rotations and

// color changes. g, p, t are assumed not null coming in.

// gg may be null. At the end of this routine, only t

// and p will be valid with respect to each other. g and gg will

// not reflect the proper ordering.

RBnodeb *cofgg; // New child of great-grandparent

int side;

side = GG && GG->right == G;

if (G->left == P) {

if (P->right == T) { // Do double rotate. (Figure 11.17(c))

G->left = T->right;

T->right = G;

P->right = T->left;

T->left = P;

P = GG;

cofgg = T;

}

else { // Do single rotate right. (Figure 11.17(b) reversed)

G->left = P->right;

P->right = G;

cofgg = P;

}

}

else { // G->right == p

if (P->left == T) { // Do double rotate. (Figure 11.17(c) reversed)

G->right = T->left;

T->left = G;

P->left = T->right;

T->right = P;

P = GG:

cofgg = T;

}

else { // Do single rotate left. (Figure 11.17(b))

G->right = P->left;

P->left = G:

cofgg = P;

}

}

cofgg->color = BLACK;

G->color = RED;

if (GG) {

if (side) GG->right = cofgg; else GG->left = cofgg;

}

else root = cofgg;

return root;

}

Figure 11.18 shows an example of building a red-black tree using the pathological insertion sequence a-i-b-h-c-g-d-f-e. The resulting tree is equivalent to the 2-3-4 tree in Figure 11.10, except that the 4-node root has been split into two 2-nodes. Note that the tree is quite balanced.

Deleting from a Red-Black Tree

Recall from Chapter 10 how deletions work in a regular binary search tree: The basic idea is to swap the key of the node to be deleted with the key of the node's successor. (We used pointer surgery rather than actually swapping keys. Also, the predecessor could be used instead of the successor.) Then, the successor is deleted, which reduces to the case of removing a node with at least one null child. As we showed in Chapter 10, this case is easy to handle.

Figure 11.18 Building a red-black tree.

Deleting from a red-black tree works very much the same way. There's a catch, though: If the successor is black, and we remove it from the tree, then all the nodes under the successor (including any external nodes) will have their black-depths reduced by one, resulting in an illegal red-black tree. When the successor is red, this problem doesn't occur.

Thus, we must somehow handle the case where the successor is black. One approach is to ensure that the successor is always red. This can be done by a single top-down pass, in the following way: an "extra" red can be pushed down the tree as we work our way towards the successor. (We must also keep track of the node to be deleted when it is encountered.) Thus, when the successor is reached, it's guaranteed to be red. The trick is to keep the tree a valid red-black tree. We accomplish this by doing appropriate rotations and color flips, keeping the black-depths of all external nodes identical. Recall that this approach keeps the leaves of the equivalent 2-3-4 tree at the same level, and thus the tree remains fairly balanced.

From the viewpoint of the equivalent 2-3-4 tree, pushing an extra red down the tree involves three types of operations:

1.     Merging two 2-nodes into a single 4-node.

2.     Flipping the red-black orientations of 3-nodes.

3.     Rotating 3-nodes and 4-nodes from one side of the tree to the other. The rotations always take place toward the side containing the current node being traversed.

Figure 11.19 shows examples of the first and third cases. The second case is not shown since it doesn't structurally change the equivalent 2-3-4 tree; it just changes the orientation of the red-black interpretation. Please note that these cases have other orientations not shown in the figure. It's interesting that the act of merging 2-nodes into 4-nodes is exactly the opposite of what we do during insertions, where 4-nodes are split in two. Thus, deletions are the opposite of insertions, with the added step of pushing an extra red down the tree.

Figures 11.20, 11.21, and 11.22 show the orientations and transformations for the three balancing operations, in terms of red-black trees. In all three figures, the node labelled t is the current node being traversed, p is its parent, g is its grandparent, s is its sibling, and ls and rs are the left and right children of s. Vertical lines shown for grandparents mean that the parent node can either be on the left or right. Dashed lines indicate links that can be either red or black. You should verify for yourself that the transformations shown preserve the relative black-depths of all external nodes.

The algorithm for deleting a node from a red-black tree is split into four procedures: Delete( ), Detach( ), DelBalance( ), and DoReplacement( ). The Delete( ) function (not given) calls Detach( ) to detach the node from the tree, and then frees the node. Detach( ) works by walking the tree from the top down, searching for the node to delete, recording its location, and then continuing to walk the tree pretending that the node wasn't found. By doing this, when a null node is reached, the current parent will be the successor node. All the while, rotations and color flips are done by calling DelBalance( ). At the end, the DoReplacement( ) function is called to "swap" the key of the node to delete with the key of the successor. (Actually, the replacement is done by pointer surgery, rather than actually swapping the keys.)

Figure 11.19 Sample top-down deletion transformations.

(a) Merging nodes

(b) 2-3-4 rotation

Figure 11.20 Case 1: Merging 2-nodes into a 4-node.

To save function-calling overhead, the necessary pointers are packaged into a PtrPak structure (defined earlier), which is then passed via another pointer. The root is also passed in case it needs to be updated.

Here are the Detach( ), DelBalance( ), and DoReplacement( ) functions. Note that DelBalance( ) and DoReplacement( ) are written using generic RBnodeb pointers and are thus shared by all red-black trees, regardless of the type of data they hold.

template<class TYPE>

RBnode<TYPE> *RBtree<TYPE>::Detach(const TYPE &x)

// Detaches (but does not free) the node having key x from the tree

{

struct PtrPack pp;

if (root == 0) return 0;

pp.t = root; pp.p = 0; pp.g = 0; pp.m = 0; pp.pm = 0;

while (pp.t) {

// Go down the tree one level, searching for node to delete

if ( ((RBnode<TYPE> *)(pp.t))->info == x ) {

pp.m = pp.t;  // Record matching node

pp.pm = pp.p; // And its parent

}

// Update ancestor pointers

pp.g = pp.p; pp.p = pp.t;

if (x < ((RBnode<TYPE> *)(pp.t))->info) {

pp.t = pp.p->Left( ); pp.s = pp.p->Right( );

}

else {

pp.t = pp.p->Right( ); pp.s = pp.p->Left( );

}

if (pp.s) {

pp.ls = pp.s->Left( ); pp.rs = pp.s->Right( );

}

else {

pp.ls = 0; pp.rs = 0;

}

root = (RBnode<TYPE> *)DelBalance(root, pp);

} // end of while loop

root = (RBnode<TYPE> *)DoReplacement(root, pp);

if (root) root->color = BLACK;

return (RBnode<TYPE> *)pp.m; // Node to delete

}

RBnodeb *DelBalance(RBnodeb *root, PtrPack &pp)

// Balancing code during top-down deletion

{

if ((T == 0 | | T->color == BLACK) && S && S->color == RED) {

// Case 2: t black, s red. t might be null,

// but s and p must not be. Flip the orientation

// of the 3-node that s is a part of.

// Fix g child to be s. Also s may become p of m

if (G) {

if (G->left == P) G->left = S; else G->right - S;

}

else root = S;

if (P == M) PM = S;

// Finish rotating

if (P->left == T) { // Figure 11.21(a)

// RotateLeftt(p);

P->right = LS;

S->left = P;

G = S;

S = LS;

}

else { // Figure 11.21(b)

// RotateRight(p);

P->left = RS;

S->right = P;

G = S;

S = RS;

}

// Fix children of s

if (S) { LS = S->Left( ); RS = S->Right( ); }

// Fixup colors

P->color = RED; G->color = BLACK;

}

if (T && T->color == BLACK &&

(T->left == 0 || T->Left( )->color == BLACK) &&

(T->right == 0 || T->Right( )->color == BLACK)) {

if (S && S->color == BLACK &&

(S->left == 0 || S->Left( )->color == BLACK) &&

(S->right == 0 || S->Right( )->color == BLACK)) {

// Case 1: t and s are black 2-nodes. Merge them

// into a 4-node. (Figure 11.20.)

P->color = BLACK; T->color = RED; S->color = RED;

}

else if (LS && LS->color == RED) {

// Case 3: t is a black 2-node, s is 3- or 4-node,

// LS is red (RS might be). Rotate 3- or 4-node to

// the side that t is on.

if (P- >left == T) { // Figure 11.22(b)

// Fix colors

LS->color = P->color; P->color = BLACK; T->color = RED;

// Fix g child to be LS. Also LS may become p of m.

if (G) {

if (G->left == P) G->left = LS; else G->right = LS;

}

else root = LS;

if (P == M) PM = LS;

// Finish: DoubleRotateLeft(s, p)

P->right = LS->left;

LS->left = P;

S->left = LS->right;

LS->right = S;

G = LS;

// We won't fix s and children since they get re-assigned

// at the top of next loop

}

else { // Figure 11.22(b)

// Fix colors

S->color = P->color; LS->color = BLACK;

P->color = BLACK; T->color = RED;

// Fix g child to be s. Also s may become p of m.

if (G) {

if (G->left == P) G->left = S; else G->right = S;

}

else root = S;

if (P == M) PM = S;

// Finish: RotateRight(p);

P->left = RS;

S->right = P;

G = S;

// We won't fix s and children since they get re-assigned

// at the top of next loop

}

}

else if (RS && RS->color == RED) {

// Case 3: t is a 2-node, s is a 3-node, LS black, RS red

// Rotate 3-node to the side of t

if (P->left == T) { // Figure 11.22(b)

// Fix colors

RS->color = BLACK; S->color = P->color;

P->color = BLACK; T->color = RED;

// Fix g child to be s. Also, s may become p of m.

if (G) {

if (G->left == P) G->left = S; else G->right = S;

}

else root = S;

if (P == M) PM = S;

// Finish: RotateLeft(p);

P->right = LS;

S->left = P;

G = S;

// We won't fix s and children, since they get reassigned

// at the top of next loop

}

else { // Figure 11.22(d)

// Fix colors

RS->color = P->color; P->color = BLACK; T->color = RED;

// Fix g child to become RS. Also, RS may become p of m.

if (G) {

if (G->left == P) G->left = RS; else G->right = RS;

}

else root = RS;

if (P == M) PM = RS;

// Finish: DoubleRotateRight(s, p);

P->left = RS->right;

RS->right = P;

S->right = RS->left;

RS->left = S;

G = RS;

// We won't fix s and children, since they get reassigned

// at the top of next loop

}

}

}

return root;

}

RBnodeb *DoReplacement(RBnodeb *root, PtrPack &pp)

{

// At this point, m is the node to delete, pm is its parent,

// p is the replacement node, g is its parent. If m has no

// successor, p will = m, and replacement is the non-null

// child of m.

if (M) { // Matching node was found

if (P == M || M->left == O || M->right == O) {

// No successor, and/or at least one child null

// Get non-null child, if any; else p will be null

P = (M->left) ? M->Left( ) : M->Right( );

}

else { // m has a successor to use as a replacement

if (G != M) {

// Successor isn't immediate child of m, so detach

// from where it is, attach to where m is

G->left = P->right;

P->right = M->right;

}

// Finish attaching where m is

P->left = M->left;

}

// p should have same color as m since it's going where m was

if (P) P->color = M->color;

}

// Fix pm child link to be p

if (M) {

if (PM) {

if (PM->left == M) PM->left = P; else PM->right = P;

}

else root = P; // New root, possibly null

}

return root;

}

Figure 11.21 Case 2: Reorienting a 3-node.

Figure 11.22 Case 3: Rotating 2-3-4 trees in red-black form.

Figure 11.23 shows the tree given in Figure 11.18 as it is destroyed by removing nodes in the sequence e-f-d-g-c-h-b-i-a.

There's one curious fact about the deletion algorithm. Even if the node to be deleted from the tree isn't found, the tree structure may still be modified, as node merging and tree rotations are performed in anticipation of a node deletion. If the Detach( ) routine is called repeatedly this way, the tree structuring will eventually converge onto a "stable" configuration.

  BOUNDARY CONDITIONS

The insertion and deletion routines given in this chapter are complicated by the need to test for boundary conditions, such as non-existent parents, grandparents, children, or siblings. You'll notice explicit tests for null pointers in the code to handle these conditions. The routines can be simplified somewhat by creating a special sentinel node to be used for all otherwise non-existent nodes. This node can be colored black, and all null child links can be modified to point to this sentinel. For more details on using artificial nodes like this, see [GuibSedg 78].

Figure 11.23 Removing nodes from a red-black tree.

  SUMMARY

If you need to ensure that a binary search tree never degenerates to worst-case scenarios, then a 2-3-4 tree, as implemented using a binary red-black tree, is the data structure of choice. Recall that 2-3-4 trees are B-trees of order 4. It's possible to generalize the technique behind red-black trees to work for B-trees of any order, by having links with more than two color choices. However, it's not really beneficial to do this; it's better instead to implement B-trees directly with multi-way nodes. This is due to the I/O paging characteristics of most disk devices. For example, there will be some page size that corresponds to a B-tree node of a certain order that will be optimal for any given disk. We'll explore B- trees on disk in Chapter 14. Just remember that red-black trees are efficient implementations of B-trees of order 4 that happen to perform well in memory.

Go to Chapter 12            Return to Table of Contents


Practical Data Structures: SPLAY TREES

 Next Chapter Return to Table of Contents Previous Chapter

CHAPTER 12: SPLAY TREES

The balanced trees presented in Chapter 11 are optimal when the keys in the trees are accessed in a uniform way. However, if a few keys are accessed more frequently than others, a balanced tree isn't necessarily the best choice. It's better to have the most frequently used keys close to the top of the tree, so that the searching costs, amortized over the life of the tree, are optimal. We'll now take a look at data structures called splay trees that have this goal in mind. Splay trees were invented by D. Sleator and R. Tarjan. See [Sleatar 85] for the definitive paper on splay trees. These intriguing structures have many uses, but we'll examine them in the context of binary search trees and priority queues.

SELF-ORGANIZING TREES

A splay tree is an example of a self-organizing structure that adapts itself to changing conditions. The principle behind self organization is simple: When a node is accessed, it is moved to the top of the tree. Then, if the node is accessed again, it is available with only minimal searching. As the tree is used, the most frequently accessed nodes tend to float to the top. When the access patterns change, other keys float to the top.

Rotation is the best way to move a node to the top, since this approach preserves the binary search tree property. Various strategies are possible for choosing the rotations to perform. In one strategy, a single rotation is used to move the desired node one level closer to the root. Another strategy involves moving the node all the way to the top using a series of single rotations. Unfortunately, neither of these strategies performs well in practice. However, a third approach does a yield good results: splaying.

Bottom-Up Splaying

The splaying algorithm is similar to the previous strategy given in that it moves the desired node all the way to the root, but the movement is performed in a rigorously defined way. Assume that you have a pointer to the node you would like to move to the top. Three rules are involved, which are applied repeatedly from the bottom up until the node is at the root. (These rules assume you have some way to find the parent of a node.)

1. Zig.When the desired node is one level below the root, a single rotation is performed (called a zig in splaying parlance) to move the node to the root. Figure 12.1(a) shows an example. (There is also a symmetrical orientation, which is not shown.)

2. Zig-zig. When the desired node is more than one level deep, and the parent and grandparent are aligned the same way, two single rotations in the same direction (called a zig-zig) are performed to move the node closer to the root. Figure 12.1(b) shows an example. (Again, there is a symmetrical orientation not shown.)

3. Zig-zag. When the desired node is more than one level deep, and its parent and grandparent are aligned in opposite directions, then a double rotation (called a zig-zag) is performed. Figure 12.1(c) shows an example (with the symmetrical case omitted once again).

Figure 12.1 The three splaying cases.

(a) Zig (single rotation)

(b) Zig-zig (two single rotations)

(c) Zig-zag (double rotation)

Figure 12.2 shows a complete example of splaying, with all three rules being applied. In this example, node c is the desired node to be moved to the root. Note that the tree undergoes a large amount of restructuring. This is typical of splaying. Don't be misled by the fact that the final tree is more balanced than the original. Splaying can greatly unbalance the tree. For example, Figure 12.3 shows how accessing the nodes of a tree in sorted order can lead to a degenerate tree.

The purpose of splaying isn't to balance the tree, but rather to move the most frequently used nodes to the top. Splay trees have been found to have the following behavior: For a tree of n nodes, as many as n steps may be required to find any given node. This happens when searching for a node at the bottom of a degenerate tree. However, if a sufficiently long series of searches is performed, the searching averages out to just log n steps, due to the way splaying restructures the tree. In other words, splay trees have an amortized efficiency of logn, the same as for balanced trees. Unfortunately, splay trees have the same worst case that linked lists have.

Figure 12.2 Bottom-up splaying.

(a) Zig-zag

(b) Zig-zag

(c) Zig

Figure 12.3 Degenerate splaying.

Top-Down Splaying

So far, we've described splaying in terms of bottom-up operations. If splaying is used in conjunction with searching, this implies two passes: a top-down pass to find the node, and then the bottom-up splaying pass. The bottom-up pass requires that you either have saved the parent pointers on a stack on the way down or that each node stores a parent pointer.

It's possible to combine splaying and searching into a single top-down pass, eliminating the problem of finding the parents. In top-down splaying, the strategy is to split the tree into middle, left, and right subtrees. The middle subtree contains that portion of the splayed tree not yet visited. The left subtree contains those nodes known to be smaller than any node in the middle subtree, and the right subtree contains those nodes known to be greater than any node in the middle subtree. As we pass down the tree searching for the desired node, we perform the required splaying rotations and then split off chunks of the tree, attaching them to either the left or right subtrees. When the splaying is finished, the trees are combined again to produce the splayed tree.

In the figures to follow, the left subtree is labelled L and the right subtree is labelled R. The root nodes are actually dummy header nodes holding no data. In the left subtree, nodes are always attached on the right side. In fact, the attachment point, drawn as a black square, is always the right child of the maximum node. The left child link of L, which would otherwise be null, is used to point directly to the maximum node. When L is empty, we make the left child link point back to L. The right subtree has a symmetrical orientation, with the attachment point always the left child of the minimum node, and with the right child of R pointing to the minimum node.

As is true with bottom-up splaying, top-down splaying involves three configurations. The zig operation, which corresponds to a single rotation, is accomplished by cutting off part of the tree and attaching it to the appropriate subtree, depending on the direction of the rotation. Figure 12.4 illustrates the process. Note that the splayed node becomes the root of the middle subtree. The following LinkRight( ) and LinkLeft( ) routines show how the attachments can be coded. Here, p is the root of the middle subtree, and l and r point to the roots of the left and right subtrees, respectively. The routines use generic Bnodeb pointers, and thus are independent of the data stored in the nodes.

Bnodeb *LinkRight(Bnodeb *p, Bnodeb *r)

// p->left becomes the new parent of the tree that p

// used to belong to, so we return p->left

{

Bnodeb *newp = p->left;

p->left = 0;

r->right->left = p; // Set left child of minimum node

r->right = p;       // Point directly to minimum node

return newp;

}

Bnodeb *LinkLeft(Bnodeb *p, Bnodeb *l )

// p->right becomes the new parent of the tree that p

// used to belong to, so we return p->right

{

Bnodeb *newp = p->right;

p->right = 0;

r->left->right = p; // Set right child of maximum node

r->left = p;        // Point directly to maximum node

return newp;

}

Figure 12.4 Top-down zigging.

(a) Link right

(b) Link left

Figure 12.5 illustrates the zig-zig operation, in both the left and right orientations. Again, the splayed node becomes the root of the middle subtree. Recall that a zig-zig corresponds to two single rotations in the same direction. In top-down splaying, the last rotation is accomplished by either LinkRight( ) or LinkLeft( ). The first rotation is handled by one of the following rotation routines: RotateRight( ) or RotateLeft( ).

Bnodeb *RotateRight(Bnodeb *p)

// Rotates right about p. Returns a pointer

// to the node that takes the place of p.

{

Bnodeb *t = p->left;

p->left = t->right;

t->right = p;

return t;

}

Bnodeb *RotateLeft(Bnodeb *p)

// Rotates left about p. Returns a pointer

// to the node that takes the place of p.

{

Bnodeb *t = p->right;

p->right = t->left;

t->left = p;

return t;

}

Figure 12.5 Top-down zig-zigging.

(a) Rotate right, link right

(b) Rotate left, link left

Figure 12.6 illustrates the zig-zag configuration, in which a double rotation is performed. Using LinkRight( ) and LinkLeft( ) in a series has the same effect as doing the two opposite rotations that make up a double rotation. As always, the splayed node becomes the root of the middle subtree.

The operations for zig, zig-zig, and zig-zag are applied repeatedly until the desired node (the one having the matching key) becomes the root of the middle subtree. At this point, the three subtrees are assembled back into a single tree, as illustrated in Figure 12.7. The splayed node becomes the root of the reassembled tree. The following Assemble( ) routine gives the necessary pointer manipulations.

void Assemble(Bnodeb *p, Bnodeb *l, Bnodeb *r)

// Assembles p, l, and r into one tree, with p as the root,

// l & r as the left and right subtrees of p, and the old

// left and right subtrees of p as right and left subtrees

// of l & r, respectively

{

if (1->right) {

1->left->right = p->left;

p->left = l->right;

}

if (r->left) {

r->right->left = p->right;

p->right = r->left;

}

}

Figure 12.6. Top-down zig-zagging.

(a) Link right, link left

(b) Link left, link right

Figure 12.7 Reassembling a splayed tree.

Figure 12.8 shows a complete example of top-down splaying. First a zig-zig is performed, followed by a zig-zag, and finally a zig. We've used the same tree as the one shown earlier in Figure 12.2 (illustrating bottom-up splaying). Note that the resulting splayed trees are different, although the splayed node c is at the root in both. The difference in the two cases exists because the rotations are performed in different orders.

Figure 12.8 Top-down splaying.

We are now in a position to show how a search routine for a splay tree might be coded. But first, we'll define some classes to be used for splay trees.

 SPLAY TREE CLASSES

We'll now implement splay trees using a simple two-class hierarchy. The following SplayTreeb class defines a data-independent base class. This class is unusual in that it has all static member functions, which are declared protected. This is done to hide the names of these otherwise ordinary functions, and to make them accessible only to the SplayTree class that you are about to see. The functions included in SplayTreeb are the LinkRight( ), LinkLeft( ), RotateRight( ), and RotateLeft( ) functions shown earlier. Also included are some functions for manipulating the minimum and maximum nodes in the tree, which we'll discuss later.

class SplayTreeb {

protected:

static Bnodeb *LinkRight(Bnodeb *p, Bnodeb *r);

static Bnodeb *LinkLeft(Bnodeb *p, Bnodeb *l);

static void Assemble(Bnodeb *p, Bnodeb *l, Bnodeb *r);

static Bnodeb *SplayMin(Bnodeb *t);

static Bnodeb *SplayMax(Bnodeb *t);

static Bnodeb *DetachMin(Bnodeb *&root);

static Bnodeb *DetachMax(Bnodeb *&root);

};

The following SplayTree class is derived from the SplayTreeb class:

template<class TYPE>

class SplayTree : public SplayTreeb {

protected:

Bnodeb *root;

void SearchAndSplay(const TYPE &x);

Bnode<TYPE> *SearchP(const TYPE &x, Bnode<TYPE> *&p, int &side);

// Allocation functions. . .

public:

// Constructors, destructors, copy functions,

// and other supporting functions. . .

Bnode<TYPE> *Member(const TYPE &x);

int Add(const TYPE &x, int &existed);

Bnode<TYPE> *AddFixed(const TYPE &x, int &existed);

Bnode<TYPE> *Detach(const TYPE &x);

Bnode<TYPE> *DetachMin();

Bnode<TYPE> *DetachMax();

int Delete(const TYPE &x);

void DeleteMin();

void DeleteMax();

Bnode<TYPE> *Min();

Bnode<TYPE> *Max();

int IsEmpty();

Bnode<TYPE> *Root() const;

};

Complete code for splay trees is given on disk in the files splay.h, splay.cpp, and splay.mth. A test program is given in tstsplay.cpp. In addition, many supporting files from Chapter 10 are used.

As usual, the SplayTree class is implemented as a concrete data type, with various constructors, destructors, overloaded assignment operators, allocation functions, and so on. These functions are similar to those used in the Bstree and RBtree classes of earlier chapters, and won't be discussed further. Instead, we'll focus on routines for searching, inserting, and deleting nodes.

Searching in a Splay Tree

Inserting into a Splay Tree

Deleting from a Splay Tree

Searching in a Splay Tree

The following SearchAndSplay( ) routine shows how to search for a node in a splay tree. The basic idea is to do a standard binary search and perform top-down splaying at the same time. The appropriate splaying operations to perform are determined by comparing the current node t with the desired key, as well as testing the children of t.

template<class TYPE>

void SplayTree<TYPE>::SearchAndSplay(const TYPE &x)

// Search for node containing the key x, moving that

// node to the root using top-down splaying

{

Bnodeb l, r; // Temp subtrees holding no data

if (root == 0) return;

l.left = &l; l.right = 0;

r.left = 0;  r.right = &r;

while (x != Root()->info) { // While root doesn't match

if (x < Root()->info) {   // Search left down the tree

if (root->left) {

if (x == Root()->Left()->info) {

// zig

root = LinkRight(root, &r);

}

else if (x < Root()->Left ()->info) {

// zig-zig

root = RotateRight(root);

if (root->left) root = LinkRight(root, &r);

}

else if (x > Root()->Left()->info) {

// zig-zag

root = LinkRight(root, &r);

if (root->right) root = LinkLeft(root, &l);

}

} else break;    // Node isn't in the tree

}

else {               // x > root->info, so search right down the tree

if (root-> right) {

if (x == Root()->Right()->info) {

// zig

root = LinkLeft(root, &l);

}

else if (x > Root()->Right()->info) {

// zig-zig

root = RotateLeft(root);

if (root->right) root = LinkLeft(root, &l);

}

else if (x < Root()->Right()->info) {

// zig-zag

root = LinkLeft(root, &l);

if (root->left) root = LinkRight(root, &r);

}

} else break; // Node isn't in the tree

}

}

Assemble(root, &l, &r);

}

If the matching node is in the tree, it will become the root of the tree after the search. However, if the node isn't in the tree, either the predecessor or successor of the node will become the root. Thus, the root must be tested to see if it is indeed the matching node, as the following Member( ) function illustrates:

template<class TYPE>

Bnode<TYPE> *SplayTree<TYPE>::Member(const TYPE &x)

{

SearchAndSplay(x);

return (Root() && Root()->info == x) ? Root() : 0;

}

Inserting into a Splay Tree

Unlike a normal binary search tree, nodes are added to a splay tree at the root. To make this work, the SplayAndSearch( ) routine is first called. If the node already exists, the insertion is skipped. If the node doesn't exist, a new node is allocated to become the new root. The splaying that takes place during the unsuccessful search restructures the tree so that splicing in the new root is easy. The old root will be the predecessor or successor of the new root. We make the old root the left or right child of the new root. In turn, one of the children of the old root will become the other child of the new root. Figure 12.9 shows an example. The following Add( ) function gives the details.

template<class TYPE>

int SplayTree<TYPE>::Add(const TYPE &x, int &existed)

// Search for a node containing key x in the tree. If found,

// set existed to 1 and return. If not found, add a new

// node to the tree containing x. This node will become

// the new root.

// Returns 1 if successful; returns 0 if allocation failed

{

Bnode<TYPE> *p;

SearchAndSplay(x); // Moves closest match to the root

existed = (Root() && Root()->info == x) ? 1 : 0;

if (!existed) {

p = AllocNode(x);

if (p == 0) return 0;

// p is about to become the new root

if (Root()) {

//  Determine which side of p the old root should go on

if (x < Root()->info) {

p->right = root;

p->left = root->left;

root->left = 0;

}

else { // x > root->info at this point

p->left = root;

p->right = root->right;

root->right = 0;

}

}

root = p;

}

return 1;

}

Figure 12.9 Insertion into a splay tree.

(a) Original tree

(b) Unsuccessful search for g

(c) Adding g as the root

Deleting from a Splay Tree

In Chapter 10, we detached a node from a binary search tree by swapping the node's key with that of its successor, and then detaching the successor. We can use a similar process with a splay tree, except that we want to perform splaying at the same time, so that the amortized searching cost will remain at logn.

The deletion can be performed by first calling SearchAndSplay( ) to move the node to delete to the root. Then, we splay the successor node to the top of the right subtree by calling a special SplayMin( ) function. (The successor of the root is always the minimum node of the right subtree.) We'll describe the SplayMin( ) function in the next section. Note that the successor node is guaranteed to have a null left child (Can you figure out why?). Thus, we can easily make the successor the new root and make, as its left child, the left child of the old root. Figure 12.10 illustrates this process. If the root doesn't have a successor, the left child of the root can take its place. If there is no left child, the tree is now empty.

We could also use an alternate, symmetrical operation for the detachment: The predecessor node could be used as the replacement by splaying the maximum node of the left subtree to the top, making it the new root, and using the right child of the old root as the right child of the new root.

Figure 12.10. Detaching a node from a splay tree.

(a) Original tree

(b) Splay b to top

(c) Splay c to top of subtree

(d) Make c the new root

At this point, the node targeted for deletion is detached from the tree. The following Detach( ) function gives the details. The node can then be de-allocated if desired. A Delete( ) routine (not shown) handles this chore. Note that, if the node to be detached isn't found in the tree, the tree may be restructured anyway by SearchAndSplay( )--in anticipation of the possible detachment.

template<class TYPE>

Bnode<TYPE> *SplayTree<TYPE>::Detach(const TYPE &x)

// Searches for a node matching x and detaches it from

// the tree. Returns a pointer to the node detached;

// returns 0 if not found.

{

if (root == 0) return 0;

SearchAndSplay(x);

if (Root()->info == x) { // Matching node found

Bnode<TYPE> *oldroot = Root();

if (root->right) {

root = SplayMin(root->right);

root->left = oldroot->left;

}

else root = root->left;

return oldroot;

}

return 0;

}

 SPLAY TREES AS PRIORITY QUEUES

In Chapter 9, we briefly mentioned priority queues, in which items are inserted in any order, but extracted in sorted order, based on a priority determined by a key within each item. Because splay trees are highly adaptable, they can be used quite effectively as priority queues. The technique involves inserting items into the splay tree in binary search tree order, and then extracting either the minimum (or maximum) node. The following discussion assumes that we wish to extract the minimum node, but the techniques described can easily be modified to extract the maximum node.

A regular binary search tree can be used as a priority queue, but a splay tree is more effective. Recall that the minimum node of a tree can be found by continuously walking left until a null left child is found. The minimum node may be near the bottom of the tree, so the first extraction may not be very efficient. However, in a splay tree, extracting the minimum node tends to cause nodes close the minimum to migrate to the top. Thus, subsequent extractions can be quite efficient.

However, if intervening insertions take place, the minimum node may fall toward the bottom again. This can be circumvented by modifying the behavior for insertions: Don't do splaying when a node is inserted. Instead, use an ordinary binary search tree insertion. Recall that such insertions don't restructure the tree. Instead, the new node is added somewhere close to the bottom. The minimum node will thus remain close to the top for a subsequent extraction. If the new node is the minimum, it will be added below the old minimum (in fact, on the left), and thus will be quite close to the top.

Note     The technique just described is believed to have been invented by the author.

Figure 12.11 illustrates a splay tree used as a priority queue. Note that, once the splaying begins, the minimum node stays close to the top. If the extraction process begins on a degenerate tree with n nodes, with the minimum node at the bottom, the worst-case cost is n steps. However, the amortized cost will probably be closer to logn steps, and may be much better than this. Because the insertions don't do splaying, it helps if the insertions are intermixed with extractions (which is likely to be the case). If there is a long series of insertions without extractions, an expensive extraction might be forthcoming, but subsequent extractions are likely to be cheap due to the splaying that finally takes place. It is difficult to precisely analyze the costs of the insertions and extractions for the strategy given here. The costs when using normal splaying operations for priority queue insertion have been empirically analyzed [Jones 86].

Figure 12.11 Priority queue splaying.

(a) Original tree

(b) Extract a

(c) Add h without splaying

(d) Extract b

(e) Add e without splaying

(f) Extract c

(g) Extract d

The following SplayMin( ) function shows how the minimum node can be splayed to the top. This function is an adaptation of the SearchAndSplay( ) function. But here, we don't need to do any searching, since the minimum node is always to our left. Note that the splaying will be in the form of zig-zagging operations, except for perhaps the final operation, which may be a single zig. A SplayMax( ) function can be symmetrically defined. Here, the maximum is always on the right.

Bnodeb *SplayTreeb::SplayMin(Bnodeb *t)

// Move the minimum of t up the tree, replacing t

// as the root. The new root is returned.

{

Bnodeb l, r;

if (t == 0) return 0;

l.left = &l; l.right = 0;

r.left = 0;  r.right = &r;

while(t->left) {

// Go left as far as possible, splaying along the way

// We have either a zig-zig, or a zig orientation

if (t->left->left) t = RotateRight(t);

t = LinkRight(t, &r); // t->left guaranteed not null

}

Assemble(t, &l, &r);

return t;

}

After the minimum is splayed to the top, the node can be detached in exactly the same way we detached nodes with the Detach( ) function. The only difference is that the left child of the splayed node is always null. So, if the right subtree is empty, we know the tree is empty. Here is the corresponding DetachMin( ) function. A DetachMax( ) function could be symmetrically defined:

Bnodeb *SplayTreeb::DetachMin(Bnodeb *&root)

// Detach the minimum node of the tree, returning

// a pointer to it, or 0 if tree is empty

{

if (root == 0) return 0;

root = SplayMin(root);

// Replace the minimum node, now at the root,

// with its successor, found by splaying the

// minimum node of the right subtree to the top.

// If there is no right subtree, then we know the

// tree is empty, since the left child is null.

Bnodeb *oldroot = root;

if (root->right) {

root = SplayMin(root->right);

root->left = oldroot->left;

}

else root = 0;

return oldroot;

}

To insert a node into a splay tree-based priority queue without splaying, we can use functions identical to the SearchP( ) and Add( ) functions of the Bstree class described in Chapter 10. (We won't repeat those functions here.) For the code given on disk, we've renamed Add( ) to AddFixed( ) to differentiate it from the Add( ) function that uses splaying.

While splay trees make good non-contiguous priority queues, other data structures, called heaps (not to be confused with the heaps used for dynamic memory allocation), work well for contiguous priority queues. Heaps are advantageous because they can be implemented without storing left and right pointers in the nodes. Only the data needs to be stored. For details on heaps, see the companion volume [Flamig 93].

 SUMMARY

Splay trees are intriguing structures because they are so adaptable. For example, we can use a splay tree for normal binary searching operations, and then decide to switch gears and use it like a priority queue. In fact, we can switch between extracting the minimum nodes and extracting the maximum nodes, and the splay tree will automatically adjust itself. (However, alternating between minimum and maximum extractions will lead to bad performance. There are better structures for this type of activity, called min-max priority queues. See [Atk 86].)

Unfortunately, splay trees do not necessarily make good general-purpose search trees. There are two fundamental problems:

1.     Splay trees have the same worst-case searching performance as linked list. For a large number of nodes, this may be unacceptable. If you want to guarantee logn performance for any single operation, a balanced binary search tree is better.

2.     For the most efficient operation, a splay tree must always restructure the tree, even during what would otherwise be read-only searching, and even when the searching is unsuccessful. Thus, you couldn't use a splay tree to search a database on CD-ROM, for instance. You could implement the searching to forego the splaying, but then the amortized efficiency would suffer. Alternatively, you could "train" the splay tree by using a long series of typical accesses, using splaying, and then "freeze" the structure. From then on, you could search without the splaying. However, this would only be effective if no new nodes were added, and if the access patterns didn't change.

Splay trees are advantageous when the access patterns favor a few nodes over the rest. A priority queue provides a good example because the nodes being favored are the ones with the smallest keys. These nodes tend to percolate to the top during the minimum node extraction process.

Go to Chapter 13            Return to Table of Contents


Practical Data Structures: FILE-BASED OBJECTS

 Next Chapter Return to Table of Contents Previous Chapter

CHAPTER 13: FILE-BASED OBJECTS

Up to this point in the book, we've implemented data structures that reside in memory. However, it can be just as useful to have data structures that reside in files on disk. In this chapter, you'll see a system of classes that allow you to have such file-based objects.

One data structure that's often used in conjunction with files, called a cache, is a set of memory buffers that hold copies of file-based objects. Since reading from and writing to files is a relatively slow process, caches are used to enhance performance. Rather than fetching objects from the file every time they are needed, the copies kept in memory are accessed instead.

In this chapter, you'll see a unique cache design invented by the author. By using smart pointers called cached-object pointers (or Coptrs, pronounced as cop-ters), you can work with file-based objects with a pointer syntax that's similar to what you normally use for objects in memory. As you use Coptrs, the objects being referenced are stored and fetched to and from a file by way of a cache--almost transparently behind the scenes.

Persistent objects, which can save their state between program invocations, are currently a popular OOP topic. The file-based objects presented here are a weak form of persistent objects. Our file-based objects don't have the full capabilities of normal objects, lacking the ability to have pointers, destructors, and virtual functions. These file-based objects can, however, have constructors. In fact, you can construct file-based objects using an overloaded new operator with a syntax very close to that used for normal objects.

Even though our file-based objects aren't full objects, they are still quite useful for database applications. For example, you'll see how to use file-based objects to implement binary search trees that reside on disk. In fact, the material in this chapter lays the groundwork for Chapter 14, where we implement file-based B-trees.

A FILE-BASED ALLOCATOR CLASS

At the heart of our file-based object design is the following Fmgr class (and associated data structures) that manages objects stored in files:

struct FmgrHeader {  // Goes at the beginning of every fmgr file

long fs;           // Address to first block of "heap" free space

long fe;           // Address of byte after end of file

long hs;           // Address of the start of the "heap"

char sg[4];        // Signature used for every fmgr-based file

};

struct FBlkHeader {  // For free blocks

int check_word;      // For file integrity checks (optional)

unsigned len;        // Length of block not including header

long next;           // Pointer to next free block

};

const long CURRADDR = -1; // Indicates current location

class Fmgr : public CountedObj {

public:

enum Io_op       { fetch, store, seek };

enum AccessMode  { read_write, read_only };

enum CheckWord   { BLKINUSE = OxFDFD, FREEBLK = OxFEFE };

protected:

char name[128];     // Name of the file

static char sg[4];  // Signature for fmgr style files

FmgrHeader h;       // File header as stored in memory

FILE *fp;           // Stream file handle

Io_op lastop;       // Last I/O operation

cjar status;        // Status bits

// Various internal functions. . .

public:

Fmgr();

virtual ~Fmgr();

virtual int Create(char *fname, long static_sz = 0);

virtual int Open(char *fname, AccessMode mode = read_write);

virtual void Close(int flush=1);

virtual void Flush();

long Alloc(unsigned nbytes);

void Free(unsigned nbytes, long p);

void Fetch(void *d, unsigned n, long p = CURRADDR);

void Store(const void *d, unsigned n, long p = CURRADDR);

void Seek(long ofs, int seek_mode = SEEK_SET);

int IsOpen() const;

int ReadOnly() const;

int ReadyForWriting() const;

void ClearErr():

void Throw(StatusCode c, char *src=0, int lno=0, char *msg=0):

int OK() const;

int operator!() const;

operator const int () const;

} ;

typedef SmartPtr<Fmgr> FmgrPtr;

Unfortunately, we don't have space to explain all the design details of the Fmgr class. Instead, we'll provide a high-level description of how the class works. As always, the class is fully documented on disk. Keep in mind that the Fmgr class was painstakingly designed to be as versatile and efficient as possible.

Complete code for the Fmgr class is given on disk in the files fmgr.h and fmgr.cpp. The sample program about to be shown is given in the file fmgrtst.cpp.

Features of the Fmgr Class

Layout of an Fmgr File

The Fmgr Heap

Exception Handling

Features of the Fmgr Class

The Fmgr class is the file-based equivalent of a heap manager. It has two functions, Alloc( ) and Free( ), which are analogous to the C functions malloc( ) and free( ). In addition, the Fmgr class has Fetch( ) and Store( ) functions that let you read and write objects. This example creates an Fmgr file and stores a Part object in it:

// fmgrtst.cpp: Sample program using an Fmgr object

#include <iostream.h>

#include "fmgr.h"

struct Part {

int id;

float price;

Part(int i=0, float p=0) { id = i; price = p; }

} ;

main()

{

Part part(17, 42.0);       // Memory buffer of part to be stored

FmgrPtr f(new Fmgr);       // Should always create dynamically

f - >Create("test.dat");  // Create and open file

long addr = f - >Alloc(sizeof(Part)); // Allocate room for part

f - >Store(&part, sizeof(Part), addr);

// Close and reopen file (for testing)

f - >Open("test.dat"); // Open() closes first

// See if we can get the part back

f - >Fetch(&part, sizeof(Part), addr);

cout << "Part: <" << part.id << ", " << part.price << ">\n";

// Now delete the part

f - >Free(sizeof(Part), addr);

return 0; // (File automatically closed by destructor)

}

Fmgr files are reference counted to help detect synchronization problems when closing files. For instance, you may have several caches connected to the same file. By using smart pointers, we can disconnect the caches in any order. The file can only be closed when the last cache is disconnected. In the class definition, you'll notice that Fmgr is derived from the CountedObj class we gave in Chapter 5. You access Fmgr files through FmgrPtr objects, which are smart pointers defined as SmartPtr<Fmgr> types. As with all SmartPtrs, the reference counted object (in this case an Fmgr object) must be allocated dynamically.

The Fmgr class is fairly sophisticated, but it's still a low-level design. For example, although Fmgr will allocate objects for you, it doesn't keep track of their locations. Note how we kept the address of the Part in the addr variable. This is akin to allocating an object in memory and then recording its address in a pointer.

Further evidence of the low-level nature of Fmgr can be found in functions like Alloc( ), Fetch( ), Store( ), and Free( ), which must be told the size of the objects being used. Also, Fetch( ) and Store( ) use void pointers to pass the object data. We could derive template-based classes from Fmgr to serve as high-level type-casting interfaces. In fact, we showed an example of this in the RecFile and FileOf classes in Chapter 3. In this chapter, we'll show an alternative technique. The type-casting will be encapsulated in pointers, with a different pointer type for each type of object. Before we discuss these pointers, we'll review the layout of Fmgr-based files.

Layout of an Fmgr File

All Fmgr files have the same layout, being composed of three parts, (as illustrated in Figure 13.1): a header, a static data area, and a dynamic data area. The header is composed of four fields, as shown in Figure 13.2. The first three are used by the allocation routines (which we'll describe shortly). The fourth field is a four-byte signature of FMGR.. This signature is used by the Open( ) routine to determine whether a file is indeed an Fmgr-based file. Note that, while the file is open, the file header is also stored in memory using the FmgrHeader object h.

Figure 13.1 Fmgr file layout.

Following the header is a static data area, whose size is specified by the creator of the file. This area is analogous to the static data segment of your program, and can be used to store objects whose locations should be fixed. The static area is a good place to store small indices. For example, we'll store addresses to the roots of trees at known locations in the static area, so that we can easily find the roots.

The Fmgr Heap

The dynamic data area is analogous to a heap residing in memory. It starts out empty and grows when objects are allocated. The end of the heap is always the end of the file. Thus, the file grows as the heap grows.

The header fields hs, fe, and fs are used to maintain the heap. The hs field stores the address of the start of the heap. This field is used when the first object is allocated. Initially, hs points to the end of the static area, whose location is also stored in fe. As the heap grows, fe is adjusted to point to the end of the file, and represents the location where the file can be extended again, if necessary. Figure 13.3 shows how an object is added to an empty heap.

Figure 13.2 Fmgr header layout.

Figure 13.3 Allocating a new object.

(a)Empty heap

(b) After adding new object

The fe field isn't strictly necessary because the end of the file could always be found by asking the operating system for the file size. However, keeping track of the end of the file ourselves is more efficient and can be used to make file integrity checks. For instance, if fe is tested and found not to be the size of the file, we know the file has been corrupted.

Objects are de-allocated by placing them on the free space list. This list is not unlike the free lists we used with linked lists in Chapter 8. The start of the free space is stored in the fs field. This field is initially zero. When an object is freed, its block of data is placed on the free list. The first portion of the block is overwritten with a free block header, as illustrated in Figure 13.4. This header holds a check word (given an arbitrary value of 0xfefe), the length of the block (not including the header itself), and a long integer giving the address of the next free block. This latter field is null when the block's at the end of the list.

Figure 13.5 shows the layout of a file with two of its four objects (object 1 and object 3) freed and placed on the free list. In this figure, object 1 was freed first, and object 3 freed last. Note that freed objects are always added to the front of the list and that the list is null terminated.

When a new object is to be allocated and the free list isn't empty (that is, fs isn't zero), then the list is walked, looking for a block of an appropriate size. The block must either be equal to the size of the object or greater than the size of the object plus the size of the free block header. The block is split in two--one part for the object and the other part to be placed on the free list again. The address of the first part is returned to become the address of the object. Note that no data is written to the block at this time. As far as the file is concerned, the object stays uninitialized until we write to its location.

Figure 13.4 Free block header layout.

Figure 13.5 Objects on the free list.

The strategy we use for finding an appropriate free block is called the first-fit strategy because we scan down the free list until we find the first block the object will fit into. We may have to scan down the entire list, which can be quite long. And even then, there's no guarantee we'll find a good block. If we don't, then the file is extended to accomodate the new object and the fe field is adjusted accordingly.

We could use other, more sophisticated strategies. For instance, we could keep the list sorted in some way to shorten the time spent searching for a good block to use. We use the first-fit strategy because it's the simplest. Fortunately, when all the objects tend to be the same size, the first-fit strategy works quite well. This will be the case in later examples where we'll store a single homogenous tree in a file in which the nodes are all the same size. Keep in mind that nothing stops you from storing all types of data in a single file. It's just that the first-fit allocator may not yield optimum performance for these situations. For a discussion about heap allocation strategies, see [Knuth 73].

In the allocation strategy presented here, almost no overhead is required for objects that are allocated. The free block header is used only when objects are deleted, and then the header overwrites a portion of the object. For this strategy to function properly, however, the object must be large enough to hold a free block header. The size of the header as we've defined it is eight bytes, and that's the minimum size of an allocated object. If you try to allocate a smaller object, eight bytes will be allocated anyway and some of the space will be wasted.

Recall that the header contains an integer check word. This field is used for file integrity checks. For instance, if you follow the next field of a free block header and land on a block that doesn't start with this check word, you know the file has been corrupted. The check word is optional, and if you remove it, the minimum allocated size will be reduced to six bytes. However, we strongly recommend keeping the check word because it's a great debugging aid.

Exception Handling

A proposed addition to C++, exception handling, gives you the ability to elegantly handle errors, or exceptions, that arise during program execution. When an invalid condition is detected, an exception is thrown, and is caught by code that can hopefully make a sane recovery. For the caching design you are about to see, exception handling would be invaluable. Unfortunately, at the time of this writing, exception handling isn't yet available. So, we've implemented the following ExceptionHandler class to crudely simulate exception handling:

class ExceptionHandler {

protected:

virtual void Report(char *src, int lno, char *msg, char *dev);

virtual void TakeAction();

public:

ExceptionCode status;

ExceptionHandler();

void Clear();

virtual void Throw(ExceptionCode c,

char *src=0, int lno=0, char *msg=0, char *dev=0);

};

#define THROW(c) Throw(c, _ FILE_, _ LINE_ )

Complete code for the ExceptionHandler class is given on disk in the files exchdlr.h and exchdlr.cpp.

Exceptions are thrown by calling the Throw( ) function. An exception code is passed, along with parameters indicating the line number and source file where the Throw( ) function was called. Other auxiliary messages defined by the user can be passed as well. In most cases, Throw( ) prints the messages and then terminates the program. Unlike true exception handling, no attempt is made to recover from an error. The THROW( ) macro, which calls Throw( ), is provided for convenience. It sets the line number and source file parameters, and defaults the others to zero.

The exception codes are defined in the following enumeration:

enum ExceptionCode {

DANGLINGPTR,

NULLPTR,

CACHEFULL,

STKFULL,

CLOSERR,

// Many other codes . . .

} ;

The following Close( ) function of Fmgr shows how to throw exceptions. A check is made to ensure that no dangling smart pointer references will be made to the file about to be closed. An exception is also thrown if an error unexpectedly occurs when closing the file:

void Fmgr::Close(int flush)

// Closes the file if not already closed, flushing the

// basic header if flush = 1. Does nothing if in the

// error state. Checks for dangling references to

// this file.

{

if (OK()) {

if (refcnt > 1) THROW(DANGLINGPTR);

if (flush && !ReadOnly()) WriteHdr();

if (::fclose(fp) != 0) THROW(CLOSERR);

strcpy(name, closed_file_name);

}

status &= Oxfd; // Reset open bit

}

The Fmgr class actually has its own Throw( ) function, which traps certain exception codes and modifies them based on the context we're in:

void Fmgr::Throw(ExceptionCode c, char *src, int lno, char *msg)

{

if (c == NOTWRITEABLE && OK( )) c = READONLYERR;

f (c == NOTREADY && (status & 1)) c = NOTOPENERR;

excp - >Throw(c, src, lno, msg, name);

status &= Oxfe; // Reset good bit

}

After fixing the exception code, this function calls the basic Throw( ) function via the global exception handler pointer excp, which is declared in the exchdlr.cpp file. You can customize the exception handling by deriving a new class from ExceptionHandler and then pointing excp to a derived handler.

Here is the message that's printed when Close( ) throws a DANGLINGPTR exception:

EXCEPTION on line 132 of fmgr.cpp

Dangling 'smart pointer'

Error occured with file tree.dat

As we've mentioned before, the excepvion handling described here is a very weak form of what is proposed for C++. For a description of the proposed design, see [Stroustrup 91].

  TOWARD A CACHING SYSTEM

In this section, we look at some issues involved in safely using the Fmgr class, and how handling those issues leads us logically to the caching design you are about to see.

To use a file-based object, we need two components (besides the file): a buffer for a copy of the object stored in memory and the address of the object in the file. In trivial programs (such as the fmgrtst.cpp program given earlier), working with these two components separately isn't a problem. In larger, non-trivial programs, however, four basic issues emerge:

1.     When using more than one file-based object, it's easy to associate a file address with the wrong memory buffer. We need a way to keep the addresses and memory buffers paired together. We can do this by defining a structure with both components as members. We'll call such structures Buckets. Here's how a Bucket structure could be defined:

template<class TYPE> struct Bucket { TYPE data; long addr; }

2.     Nothing stops you from having two addresses that point to the same object in the file, even though each address has its own associated memory buffer. This is a particularly insidious form of aliasing. Which memory buffer has the "official" data that should be written to the file? We would like to guarantee a unique object address for each memory buffer. If we're using buckets, that means no two buckets should have identical addr members.

3.     Even though each bucket should have a unique file address, we don't necessarily want to make only one reference to any given bucket. For example, assume that buckets store tree nodes and that we use pointers to buckets in implementing tree operations, such as searching and inserting. You've seen from earlier chapters that it is quite common to copy pointers during tree operations, which results in a lot of aliasing. This aliasing is bad enough for objects stored in memory, but it's even worse for file-based objects. We must ensure that bucket pointers work in concert with each other. Using smart pointers to buckets with reference counts is one way to do this.

4.     In applications with a large number of file-based objects, such as a file-based tree with many tree nodes, it's not possible or desirable to keep all the objects in memory. Thus, we end up shuffling objects in and out of a fixed number of buckets and onto disk. We must keep track of which buckets can be reused at any given time--a difficult and error-prone task. It would be nice to automate this process.

We can solve all of these problems by using a collection of buckets known as a cache. The purpose of a cache is to be the intermediary mechanism between code using a file-based object and the file itself. You make requests to the cache for a particular object (keyed by its file address), and the cache sees to it that the object is fetched from disk and stored in a bucket. If the object is already in a bucket, no fetching need be done. (This, by the way, guarantees a unique file address for each bucket.) When the appropriate bucket is reserved, a pointer to the bucket is returned to be used by the code wishing to access the file-based object.

As requests are made to the cache, more buckets are reserved. Eventually, the cache fills up. When another bucket is needed, the data in one of the occupied buckets must be flushed back to disk so the bucket can be reused for the new request. A special flag can be kept with each bucket, known as the dirty bit, which indicates whether the bucket data has been modified since it was last loaded. If the dirty bit is set, the object data in the bucket is written to disk. If the dirty bit isn't set, the bucket can be reused without further action.

Of course, the problem with this scheme lies in determining which buckets can be reused. By adding reference counts to the buckets, this becomes easy: A bucket can be reused if there are no references to it. Since the buckets are reference counted, it makes sense to access them with smart pointers. The Coptrs to be presented later allow you to do just that. A Coptr is a smart pointer to a cache bucket. When Coptrs are assigned to each other, the reference counts of the corresponding buckets are updated. The reference count of the original bucket of the target Coptr gets decremented, whereas the reference count of the source bucket gets incremented. In this way, Coptrs can share buckets without any confusion.

In using a cache to solve aliasing problems, a fortuitous side effect occurs. Code that uses caching generally runs faster than code without caching. Thus, we get both safer and faster code.

Figure 13.6 illustrates the organization of a caching system that incorporates all of the design features just discussed. Next, we'll look at the design of this system in detail.

  CACHE BUCKETS

In this section, we examine the design of cache buckets. As we've done throughout the book, we'll separate the design into two parts. We'll first define buckets independently of the data stored in those buckets, then add the data. This provides the greatest amount of code sharing. Here is the Bucketb class, which defines buckets independently of the bucket data:

class Bucketb {

protected:

long addr;      // Location of bucket's data in the file

int refcnt;     // How many references to this bucket

char dirty;     // True when data doesn't match what's in file

Bucketb *prev;  // Pointers to adjacent buckets in the cache

Bucketb *next;  // in most-recently reserved order.

void MakeNull() { addr = 0; refcnt = 0; dirty = 0; }

public:

Bucketb()            {} // Other classes do the initialization

void Lock()          { ++refcnt; }

void Unlock ()       { -refcnt; }

int IsLocked()       { return refcnt; }

void SetDirty()      { dirty = 1; }

int IsNull()         { return addr == 0; }

void Flush(Fmgr &f)  { if (addr && dirty) Store(f); }

virtual void Fetch(Fmgr &f) = 0; // Depends on bucket data

virtual void Store(Fmgr &f) = 0; // Depends on bucket data

friend class Cacheb;

friend class Coptrb;

friend void Report(Cacheb &c, int full_report=0);

};

The addr, refcnt, and dirty members implement the features of buckets described in the previous section. Two other members, prev and next, are pointers used in organizing the buckets in a doubly-linked list. We'll discuss these pointers when the Cacheb class is examined.

The most striking feature about the Bucketb class is that it can be completely inlined. This means that very little, if any, overhead is induced by using the Bucketb class over more low-level and hard-coded techniques. Only two of the member functions, Fetch( ) and Store( ), depend on the type of data stored, so we've declared these functions as pure virtual. Three of the functions--Lock( ), Unlock( ), and IsLocked( )--are used to access and manipulate the reference count stored in the bucket. The rationale behind the names of these functions will become apparent when we dis cuss the Cacheb class.

Figure 13.6 Organization of a caching system

From the Bucketb class, we can derive a template class called Bucket that defines a family of classes, each specific to a particular type of data we wish to store. There are several ways to derive the Bucket class. The first way is the most obvious; include a TYPE object inside the Bucket, as in:

template<class TYPE>

class Bucket : public Bucketb {

public:

TYPE info;

// Other definitions ...

} ;

To access the bucket data, we need to go through the info field. For example, suppose we define file-based tree nodes holding integers, and we wish to store these tree nodes in buckets. Here's how we might write code to use these buckets:

struct Node {

char data;         // The node data

long left, right;  // We use long integers, not pointers

Node( );

Node(const char &x, long left=0. long right=0);

} ;

. . .

Bucket<Node> *root;

. . .

root - >info - >left = 42L; // Set the left child of root

Going through the info field is clumsy. We would prefer to write expressions like root->left. To do this, the buckets must somehow act like tree nodes, yet we still want them to behave like buckets, too. Multiple inheritance is the perfect tool to accomplish this. Here's a new definition for Bucket that uses both Bucketb and TYPE as base classes:

template<class TYPE>

class Bucket : public Bucketb, public TYPE {

public:

Bucket();

virtual void Fetch(Fmgr &f);

virtual void Store(Fmgr &f);

} ;

Note     Some compilers don't allow you to use a template parameter as a base class (even though that's perfectly legal in C++). If this is the case with your compiler, you'll have to remove the template syntax, and code this class directly for the types you have in mind. Fortunately, this class is small enough that it can be replicated by hand easily. But you'll also have to hand-code the upcoming Cache and Coptr classes.

By using multiple inheritance for the Bucket class, we can write code like the following:

Bucket<Node> *root;

. . .

root - >left = 42L;   // Accessing one of TYPE's members

root - >SetDirty( );  // Accessing one of Bucketb's members

The Bucket class is one of those rare classes that is simultaneously simple, elegant, and powerful. Just three functions need to be added to the Bucketb class to derive a data-dependent class: a constructor and the virtual functions Fetch( ) and Store( ).

template<class TYPE>

Bucket<TYPE>::Bucket( )

// Calls default Bucketbt() and TYPE() constructors

{

}

template<class TYPE>

void Bucket<TYPE>::Fetch(Fmgr &f)

// Fetch TYPE object data from the file f

{

f.Fetch((TYPE *)this, sizeof(TYPE), addr);

dirty = 0;

}

template<class TYPE>

void Bucket<TYPE>::Store(Fmgr &f)

// Store TYPE object data onto the file f

{

f.Store((TYPE *)this, sizeof(TYPE), addr);

dirty = 0;

}

The Fetch( ) and Store( ) routines employ a subtle but powerful trick. When it's time to fetch or store the data in the bucket, we only want to use the TYPE portion of the bucket. Members like addr, refcnt, and dirty shouldn't be stored on disk. But how do we accesss only the TYPE portion of the bucket? We can employ a little-known feature of multiple inheritance: Given a pointer D to a multiply derived object with base classes A and B, you can get a pointer to the A portion of the object by type casting D to an A pointer. Similarly, to get the B portion of D, a B pointer type cast can be used. For example:

class D : public A, public B { ... };

D *Dptr; A *Aptr; B *Bptr;

Aptr = (A *)Dptr;  // Grab A portion of D

Bptr = (B *)Dptr;  // Grab B portion of D

When the compiler sees the last two statements, it calculates the pointers Aptr and Bptr by adding the appropriate offset to the this pointer of the object pointed to by Dptr. Figure 13.7 illustrates this process. Note that the precise layout of multiply-derived objects is implementation dependent. Figure 13.7 shows one possible layout. Also, the offsets to add to the this pointer are constant and can be calculated at compile time. Thus, multiply derived-to-base-class pointer conversions can be quite fast.

For a Bucket<TYPE> pointer, a TYPE pointer type cast will give you a pointer to the TYPE portion of the bucket. Thus, the Fetch( ) and Store( ) routines type cast the this pointer of the Bucket to calculate a pointer to the TYPE data. This pointer is then passed when calling the Fetch( ) and Store( ) routines of Fmgr.

In using the TYPE parameter, you need to observe these restrictions:

       TYPE should not have any embedded pointers. The values of pointers are transitory, changing from one program invocation to the next. Thus, it's of little use (and probably erroneous) to store pointers on disk. However, you can store file offsets (usually in long integers). These offsets can be used to reference other file-based objects, and can take the place of pointers. The tree node example we gave earlier illustrates this.

       TYPE cannot have any virtual functions. If TYPE has virtual functions, the virtual function table pointer will be stored on disk with the object. Of course, the value of this pointer may not be valid from one program invocation to the next. You could arrange to have the virtual function table pointer updated to the correct value after the object has been loaded, but that is highly implementation dependent. (This approach would be dependent on the way objects are organized, which isn't standardized.)

       TYPE should not have a destructor. Although the compiler-generated destructor of Bucket will implicitly call the TYPE destructor, it is not clear that the TYPE destructor can do anything that makes sense. The cache classes to be presented are not set up to call the TYPE destructor when a bucket is overwritten, thus TYPE shouldn't have a destructor (other than an empty one).

       TYPE must have a default constructor. The Bucket class constructor requires a default TYPE constructor. It's also useful for TYPE to have other constructors, as you'll see later.

       TYPE cannot be a built-in type. That's because TYPE is used as a base class of Bucket, and built-in types can't be used as base classes.

Figure 13.7 Multiple inheritance and type casting

Complete code for the Bucketb and Bucket classes is given on disk in the files bucketb.h, bucket.h, and bucket.mth.

  CACHING STRATEGIES

The main task of a cache is to handle requests for file-based objects. The cache must determine whether an object is already loaded and, if not, the cache must reserve a bucket for the object and load it in memory. However, what should the cache do when all the buckets are filled? We must provide a way for the cache to choose which bucket to overwrite with the new object. Caches are differentiated by the strategies they use to determine which buckets to reuse. For optimal performance, those buckets with data not likely to be used again soon should be chosen as candidates. In general, this is difficult to determine.

Fortunately, there is a strategy that works reasonably well. This strategy is based on the observation that, in most programs, the object most recently requested is likely to be requested again very soon. This principle is often called locality of reference. The objects currently in use tend to be localized to a small working set. As program execution proceeds, this working set slowly changes. You can take advantage of locality of reference by designing your caches to overwrite the least-recently used bucket, since that bucket is the least likely to be needed again any time soon.

Caches that overwrite least-recently used buckets are often called LRU caches. In an LRU cache, the buckets are kept in order based on how "old" the buckets are--that is, how long ago they were last used. There are various ways to "age" buckets. The simplest way is to keep the buckets on a self-organizing linked list. Each time a bucket is used, it is moved to the front of the list. Thus, the most recently used buckets tend to be at the front. Those buckets that haven't been used recently will fall to the back of the list. When a new bucket is needed, it can be chosen from the back of the list. Figure 13.8 illustrates the LRU strategy.

Our cache design uses a slight modification of the LRU strategy. We will use Coptrs that reference cache buckets directly, with each Coptr being initialized after a bucket is reserved. If we were to use a strict LRU strategy, the buckets would have to be reorganized every time a Coptr was used. This would be clearly inefficient. Instead, we reorganize the buckets only at the time a bucket is reserved. In effect, we use a least-recently reserved (LRR) strategy. Although it's difficult to prove this, the LRR strategy is likely to perform as well as, or better than, the LRU strategy.

The Cache Classes

The Cache Classes

We use two classes to implement caches. The Cacheb class incorporates those functions that are independent of the type of buckets, whereas Cache is the type-specific template version:

class Cacheb {

protected:

long hits, misses;  // For performance statistics

int nbuckets:       // Size of cache

FmgrPtr fptr;       // File cache is connected to

Bucketb *head;      // Most recently reserved bucket

Bucketb *AcquireBkt();

Bucketb *FindBkt(long addr);

void MoveToFront(Bucketb *b);

Cacheb(Bucketb *b, int n, unsigned bkt_size);

virtual ~Cacheb();

public:

friend class Coptrb;

void Connect(FmgrPtr &fp);

void Disconnect();

void Flush(int empty_bkts = 0);

void Clear();

virtual Bucketb *ReserveBkt(long addr, int ensure_loaded = 1);

friend void Report(Cacheb &c, int full_report = 0);

};

template<class TYPE>

class Cache : public Cacheb {

protected:

Bucket<TYPE> *buckets;

public:

Cache(int n);

virtual ~Cache();

};

Figure 13.8 A self-organizing LRU Cache

(a) Requesting an object already in the cache

(b) Requesting an object not in the cache, with an empty bucket

(c) Requesting an object not in the cache, with no empty buckets

Complete code for the Cacheb and Cache classes is given on disk in the files cacheb.h, cacheb.cpp, and cache.h.

The type-specific Cache class adds just two functions, which handle the allocation and de-allocation of the cache buckets. The buckets are stored in an array of size n, as shown in the following Cache constructor and destructor:

template<class TYPE>

Cache<TYPE>::Cache(int n)

// Allocate buckets. Send them to Cacheb() for further setup.

: Cacheb(buckets = new Bucket<TYPE>[n], n, sizeof(Bucket<TYPE>))

{

}

template<class TYPE>

Cache<TYPE::~Cache()

{

if (fptr) Flush();  // Process all pending bucket writes

Disconnect();       // Disconnect from the file manager

delete[] buckets;

}

Most of the code resides in the Cacheb class, which is shared among all caches. This means you can have many different caches with little source code explosion.

Although the buckets are stored in an array, the Cacheb class organizes them as a circular doubly-linked list, as shown in the following Cacheb( ) constructor. Thus, the buckets are actually organized as a hybrid structure. Unlike the circular lists we defined in Chapter 8, we don't use a dummy header node in the bucket list.

Cacheb::Cacheb(Bucketb *b, int n, unsigned bkt_size)

// Assumes n buckets of size bkt_size have already been

// allocated. Sets up their prev and next pointers to form

// a circular list.

: fptr(0) // Start with a null file manager

{

Bucketb *p = b;

for (int i = 0; i<n; i++) {

p->MakeNull();

p->prev = (Bucketb *)((char *)p - bkt_size);

p->next = (Bucketb *)((char *)p + bkt_size);

p = p->next;

}

// Make the list circular

Bucketb *tail = (Bucketb *)((char *)b + (n-1)*bkt_size);

tail->next = b;

b->prev = tail;

// Set up other cache variables

nbuckets = n; head = b; hits = 0; misses = 0;

}

The buckets are organized in a doubly-linked list to facilitate the LRR strategy of the cache. When a bucket is reserved, it is moved to the front of the list, using the following MoveToFront( ) function. This in effect makes the bucket the most-recently reserved (MRR) bucket:

void Cacheb::MoveToFront(Bucketb *b)

// Logically move bucket to front of list so that it is

// treated as the most recently reserved bucket.

// Shortcuts are used when possible.

{

if (b == head - >prev) { // b is tail

head = head - >prev;

}

else if (b != head) {

// Unhook bucket from where it is

b - >prev->next = b - >next;

b - >next->prev = b - >prev;

// Put it before head

b - >next = head;

b - >prev = head - >prev;

b - >prev - >next = b;

head - >prev = b;

head = b;

}

}

Often, MoveToFront( ) will need to move the LRR bucket at the back of the list to the front--to make it the MRR bucket. Thus, MoveToFront( ) traps for this case and uses an optimized movement. Because of the circularity of the list, only the head pointer needs to be updated.

When an object is requested from the cache, the ReserveBkt( ) function is called:

Bucketb *Cacheb::ReserveBkt(long addr, int ensure_loaded)

{

Bucketb *b;

if (addr == 0) return 0; // Don't reserve bkt for null object

b = FindBkt(addr);

if (b == 0) { // Bkt data not loaded, so acquire a bucket

misses++;

b = AcquireBkt();

if (b) {

b->addr = addr;

if (ensure_loaded && addr) b->Fetch(*fptr);

if (!fptr->OK()) b = 0; else b->Lock();

}

}

else {

hits++;

MoveToFront(b);

b->Lock();

}

return b;

}

The ReserveBkt( ) function uses the file address of the object requested as a key and searches through the cache for the object by calling FindBkt( ). This latter function does a sequential search starting from the MRR bucket because, most likely, the object being requested was requested just recently.

Bucketb *Cacheb::FindBkt(long addr)

{

Bucketb *b = head; // Start with the MRR bucket

do {

if (b - >addr == addr) return b;

b = b - >next;

} while(b != head);

return 0; // No bucket found

}

If the requested object is not found, a bucket is acquired by calling the AcquireBkt( ) function:

Bucketb *Cacheb::AcquireBkt()

{

Bucketb *b = head->prev;        // LRR bucket

while(1) {

if (!b->IsLocked()) break;

b = b- >prev;

if (b == head->prev) {

excp->THROW(CACHEFULL);   // Most likely will exit program

return 0;                 // Make compiler happy

}

}

b->Flush(*fptr);

MoveToFront(b);

return b;

}

Because buckets are going to be used directly via Coptrs, we can't simply have AcquireBkt( ) return the LRR bucket. That bucket may be referenced by one or more Coptrs. A bucket that has a non-zero reference count is called a locked bucket. So AcquireBkt( ) starts with the LRR bucket and searches toward the MRR bucket, stopping at the first unlocked bucket it finds. When an unlocked bucket is found, we prepare for it to be overwritten by flushing any dirty data it might contain. The bucket is then moved to the front of the list to become the MRR bucket.

It's possible that all buckets in the cache will be locked. When this occurs, the cache is full, and an exception is thrown. Note that we can't simply grab a bucket and use it. That would leave us with one or more dangling Coptrs.

After acquiring a bucket, the ReserveBkt( ) function fetches the object data from the file so that the data is loaded and ready to go. However, if the object being requested has yet to be constructed, there's no data in the file to load. The ensure_loaded flag is used to indicate this situation so that the bucket data isn't erroneously loaded.

Once a bucket has been either found or acquired, the reference count of the bucket is incremented by calling Lock( ). The bucket is then moved to become the MRR bucket. A pointer to the bucket is returned, presumably to be assigned to a Coptr.

You may want to have more than one cache connected to the same file. Recall that the Fmgr class is reference counted to handle this situation. The following Connect( ) and Disconnect( ) functions are used to connect or disconnect a cache from a file:

void Cacheb::Connect(FmgrPtr &f)

// Connects cache to file f, assumed already opened

{

Clear( );  // Flush any pending data for the old file

fptr = f;  // Connect to a new one

}

void Cacheb::Disconnect()

// Disconnects cache from the file it was connected to

{

Clear();   // Flush any pending data

fptr = 0;  // Connect to null object

}

Recall that Fmgr Ptris a smart pointer. The assignment âptr = âin Connect( ) decrements the reference count of the old Fmgr in use and increments the reference count for â. A cache can be disconnected from a file by connecting it to the null Fmgr object, using the statement âptr = 0. Both Connect( ) and Disconnect( ) clear the cache before changing the connection. The following Clear( ) and Flush( ) functions are used:

void Cacheb::Clear()

// Flush all buckets in the cache, and then null them out

{

if (fptr) Flush(1);

hits = 0; misses = 0;

}

void Cacheb::Flush(int empty_bkts)

// Flushes all buckets in the cache. Makes them empty if

// empty_bkts is true. Checks for dangling Coptrs.

{

// Flush, starting at the front

Bucketb *b = head;

do {

if (empty_bkts && b->IsLocked()) excp->THROW(DANGLINGPTR);

if (fptr->ReadyForWriting()) b->Flush(*fptr);

if (empty_bkts) b->MakeNull();

b = b->next;

} while(b != head);

}

When Clear( ) callsFlush( ), it sets the empty_bkts flag to true. This causes not only the bucket data to be flushed, but also makes the buckets "null" by setting their addr, refcnt, and dirty variables to zero. A bucket to be emptied should not have any pointers referencing it. That condition is checked and an exception is thrown if true. Thus, before calling Clear( ), all Coptrs must be unbound from the buckets they are referencing.

  CACHED-OBJECT POINTERS (COPTRS)

Since the cache buckets are reference counted, we need smart pointers to work in conjunction with them. We use cached-object pointers (Coptrs) for that purpose. Coptrs are implemented with two classes: a data independent Coptrb class and the type-specific Coptr class template:

class Coptrb { // Data independent base class for Coptrs

protected:

long addr;      // File address of bucket data

Bucketb *bkt;   // Pointer to bucket in the cache

Cacheb *cache;  // Pointer to the cache being used

char bound;     // True if pointer is bound to a bucket

void Copy(const Coptrb &c);

Bucketb *Alloc(unsigned n, long p=0);

void Delete(unsigned n);

void Bind();

Coptrb(Cacheb &c, long p);

Coptrb(const Coptrb &c);

void operator=(const Coptrb &c) {} // Disallowed

public:

~Coptrb();

void Grab();

void Release();

void NotifyUseOf();

operator long() const;

};

template<class TYPE>

class Coptr : public Coptrb {

public:

friend void *operator new(size_t n, Coptr<TYPE> &cp, long p);

void Delete();

Coptr(Cache<TYPE> &c, long p = 0);

Coptr(const Coptr<TYPE> &c);

void operator=(const Coptr<TYPE> &c);

void operator=(long p);

Bucket<TYPE> &operator*();

Bucket<TYPE> *operator - >();

} ;

Complete code for the Coptrb and Coptr classes is given on disk in the files coptrb.h, coptrb.cpp, and coptr.h.

Each Coptr stores the file address addr of the object being pointed to, a pointer bkt to the bucket containing the memory buffer of the object, and a pointer cache to the Cache the Coptr is connected to. Another important variable is the bound flag. This flag indicates whether the Coptr is currently bound to a bucket.

Before a Coptr can be used, it must be connected to a cache. The Coptr( ) constructor, which calls the following base class Coptrb( ) constructor, is used for this purpose:

Coptrb::Coptrb(Cacheb &c, long p)

// General Coptr constructor. Note that a cache

// is required. The Coptr stays unbound until needed.

{

cache = &c; addr = p; bound = 0; bkt = 0;

}

Coptrs can optionally be given the address p of the object being referenced. Note that p defaults to zero. A Coptr with an address of zero is treated as a null pointer. Thus, by default, Coptrs are initialized as null pointers.

The main function of a Coptr is to take a file address and make it appear to be a pointer to a TYPE object. This magic is accomplished by the clever combination of smart-pointer-style reference counting and multiply-derived buckets. The two most important functions of a Coptr are the following - > and * operator functions:

template<class TYPE>

Bucket<TYPE> &Coptr<TYPE>::operator*()

{

if (!bound) Bind();

return *(Bucket<TYPE> *)bkt;

}

template<class TYPE>

Bucket<TYPE> &Coptr<TYPE>::operator->()

{

if (!bound) Bind();

return (Bucket<TYPE> *)bkt;

}

Auto-Loading of Cached Objects

Detecting Modifications to Cached Objects

Binding Objects

Assigning Coptrs

Releasing Coptrs

Allocating Objects Referenced by Coptrs

Auto-Loading of Cached Objects

An important Coptr feature is the automatic loading of object data into the cache. Whenever a Coptr is accessed through the overloaded operators - > and *, the bound flag is checked to see whether the Coptr is currently bound to a bucket. If not, the Bind( ) function (described later) is called to reserve a bucket for the Coptr. In the operator*( ) function, a reference to this bucket is returned. The operator->( ) function returns a pointer to the bucket. (The type casts are needed because bkt is typed as a generic Bucketb pointer.)

Due to multiple inheritance, we can use the return values of our operator functions to access both the members of Bucketb and the members of TYPE. The following example constructs a Coptr to a tree node (which we defined earlier) and uses the Coptr to access the SetDirty( ) member of Bucketb and the left member of Node:

#include "coptr.h"

main()

{

FmgrPtr f(new Fmgr);   // Set up a file

f->Open("test.dat");  // Open it

Cache<Node> c(12);     // Set up a cache of 12 buckets

c.Connect(f);          // Connect it to the file

// Construct a Coptr pointing to a Node at file address 42

Coptr<Node> p(c, 42L);

p->left = 17L;   // 'left' member of Node accessed

p->SetDirty();   // Call Bucketb's SetDirty() function

p.Release();      // Must release Coptr before clearing cache

c.Disconnect();   // Disconnect the cache; all buckets flushed

f->Close( );     // Close the file; all buffers flushed

}

Coptrs are designed to follow one principle: A bucket isn't loaded until needed. When Coptr p is initialized to point to a Node at the file address 42 (presumably the Node has already been allocated), that Node isn't loaded into a bucket until the left member of Node is accessed.

Detecting Modifications to Cached Objects

Even though Coptrs cause file-based objects to be fetched automatically, we must tell a Coptr specifically when the objects need to be written back to disk. After the statement p->left = 17 is executed, the in-memory copy of the Node is different from what's stored on disk. We inform the cache of this fact by calling SetDirty( ). If the corresponding bucket is to be overwritten later (or the cache cleared), the cache knows to flush the bucket data to disk.

If the delay in writing the bucket data to disk is unacceptable, you can call Store( ) instead of SetDirty( ). This causes the bucket data to be written immediately. In fact, you could add a flag to the Bucketb class, which, when true, causes SetDirty( ) to store the object data to disk, rather than simply set the dirty bit. By doing this, you have in effect created what is known as a write-through cache. A write-through cache is safer than a regular cache. There's less danger of losing data that should be on disk but is still in memory when a hardware failure (such as power loss) occurs. We chose not to implement our caches this way. It's better to choose by hand whether to call SetDirty( ) or Store( ). That way, you have more flexibility in trading speed over safety.

You may wonder why we even need SetDirty( ). Couldn't the caching system automatically detect when the bucket data has been modified? The answer is yes in principle, but no in practice. You can detect when an object is modified by overloading its assignment operator, and setting the bucket's dirty bit when an assignment is detected. There are two problems, however.

First, you have to overload the assignment operator for each member of the object, and this must be done recursively if the members themselves are nested objects. The second problem is worse. Even if each member can detect assignments, how do those members set the bucket's dirty bit? Remember, nested objects don't have access to the objects that enclose them. A pointer to the dirty bit could be stored in each member, but this would waste a lot of space, and that space would be taken up in the file as well. Also, just adding the dirty bit pointers to each member would be difficult. How would you go about using predefined objects?

There are alternatives, but they are all less than satisfactory. One alternative is to keep two copies of each object in memory. The first copy represents the state of the object as it was loaded from disk. The second copy is used during all in-memory accesses. When it's time to flush the bucket, these two copies are compared to see if the second copy, containing any changes, should be written to disk. This technique would be simple to implement, but it wastes time in making the extra copies, and it also wastes storage.

You might wonder why you couldn't just set the dirty bit each time the overloaded - > and * operators are called for a Coptr. The problem is that the dirty bit would be set even if we're just reading the values of the members, rather than modifying them. The result is that the bucket data might be written back to disk during a flushing operation when it didn't need to be. You might as well forget about the dirty bits and always write cached objects back to disk, since that would have the same effect.

Binding Objects

Coptrs are bound to buckets by calling the following Bind( ) function. This binding is usually done for you automatically, but you can do the binding yourself by calling Grab( ):

void Coptrb::Bind( )

// Binds the Coptr to a bucket containing data stored at

// file location addr. Note that we may already be pointing

// to the correct bucket, so check for that.

// Assumes the Coptr not already bound. For internal use only.

{

if (bkt && bkt->addr == addr) { // Fast binding

if (addr) {

bkt->Lock();

bound = 1;

}

else excp->THROW(NULLPTR);

}

else {

bkt = cache->ReserveBkt(addr, 1);

if (bkt) bound = 1; else excp->THROW(NULLPTR);

}

}

void Coptrb::Grab()

// Like Bind(), but makes no assumptions about the Coptr

// being unbound coming in. For public use.

{

if (!bound) Bind();

}

The Bind( ) function has both safety and speed considerations built in. Bind( ) assumes you are trying to bind a non-null pointer (with addr being non-zero). If that's not the case, a NULLPTR exception is thrown. This is how Coptrs check for null pointers. Note that this test only occurs when binding to a bucket, and not on every access to a Coptr. This dramatically speeds up the use of Coptrs, yet it doesn't sacrifice safety. You can pass around null Coptrs all you want, and no error occurs until you try to access the associated non-existent objects.

Bind( ) is also built for speed in another way. If the bkt pointer isn't null, a test is made to see if bkt is already pointing to the right bucket. If it is, the bucket is simply locked. This saves the searching that would otherwise be required. If bkt is null or doesn't point to the right bucket, a relatively expensive call to ReserveBkt( ) is made, which searches the cache looking for a bucket. Note that ReserveBkt( ) handles the locking of the bucket in this case. For certain applications, fast binding can take place more often than you might think. You'll see this later when we discuss recursive traversals of trees on disk.

Assigning Coptrs

Coptrs can be assigned by using the overloaded assignment operator, which in turn calls the Copy( ) function:

template<class TYPE>

void Coptr<TYPE>::operator=(const Coptr<TYPE> &c) {copy(e); }

void Coptrb::Copy(const Coptrb &c)

{

Release();

addr = c.addr; bkt = c.bkt; bound = 0; cache = c.cache;

}

The Copy( ) function releases any bucket the destination Coptr might be referencing by calling Release( ) (described later), and then assigns the members of the source Coptr. The Coptr( ) copy constructor is similar to Copy( ) except the call to Release( ) isn't needed.

template<class TYPE>

Coptr<TYPE>::Coptr<TYPE>(const Coptrb &c)

{

addr = c.addr; bkt = c.bkt; cache = c.cache; bound = 0;

}

As is true with ordinary pointers, when Coptrs are copied, the data being referenced isn't copied. In fact, for Coptrs, the destination is left unbound, even though the source Coptr might be bound. This practice is in keeping with the principle that Coptrs aren't bound until needed. Using this principle has important performance ramifications. Consider operations on file-based trees. During the tree operations, many pointers are assigned, often with very transitory values. By deferring the pointer binding until it's absolutely necessary, we can ensure data isn't loaded during otherwise transitory pointer assignments.

Even though the destination is unbound, it has the source Coptr's bkt pointer. When it's time to bind the destination, it can probably be done quickly by using fast binding. The only time this wouldn't occur is if, in the interim, the referenced bucket became unlocked and overwritten.

Coptrs can also be used as though they are long file addresses. The following two operator functions help facilitate this:

Coptrb: :operator long() const

// We let a Coptr look like a file address, since in a way

// it's just a smart version of a file address

{

return addr;

}

template<class TYPE>

void Coptr<TYPE>::operator=(long p)

// Redirects the Coptr to a new address. Releases the old

// bucket (if any) and does not bind to a new one.

{

Release();

addr = p;

}

These two functions allow us to work easily with file-based tree nodes, which have long integers instead of pointers for links to child nodes. For example, we can write functions such as the following GetMin( ) function:

// File-based example

struct Node {

char info;

long left, right;

Node();

Node (const char &x, long 1=0, long r=0);

}

Coptr<Node> GetMin(Coptr<Node> t)

// Get the minimum node of tree t

{

while(t) t = t->left;

return t;

}

We'll now work through this small example to see what actually occurs behind the scenes. First, the parameter t is passed by value, which causes the Coptr( ) copy constructor to be called. After the copying, t has an address, a cache pointer, and even a bucket pointer (which may point to the right bucket if we're lucky). However, t is unbound at this time.

When the while loop is entered, t is tested for null. This is accomplished by converting t to a long integer whose value is the address of the object that t points to. If that value is non-zero, we know we're pointing to a real object. Then, t is assigned to the address of its left child. The assignment operator taking a long integer is called here, since t->left is a long integer. Note that the expression t->left causes t to be bound (with the Node loaded from disk if necessary) so that the value of left is defined. However, t is immediately unbound by the assignment statement, since it takes on a new address. Upon return from the function, we make and return an unbound copy of the final version of t.

The striking observation here is that GetMin( ) looks almost identical to the function we would write if the tree resided in memory. To make a memory-based version of GetMin( ), all we need to change is how the pointers are typed.

// Memory-based example

struct Node {

char info;

Node *left, *right;

Node();

Node(const char &x, node * 1=0, node * r=0);

}

Node *GetMin(Node *t)

// Get the minimum node of tree t

{

while(t) t = t->left;

return t;

}

The ability to write code for file-based objects so that it appears almost identical to code written for memory-based objects has important ramifications. If you've ever tried to write code for file-based trees, you know what we mean. For example, try writing code for red-black trees using low-level functions such as Fetch( ) and Store( ). It's a virtual nightmare due to the aliasing that occurs when pointers are being frequently assigned and modified. It's difficult to tell when something needs to be written to disk and which memory buffer has the right values. It's easy to be wrong.

By using Coptrs, it's relatively painless to write code for file-based trees. As a very fortunate side effect, the use of caching makes the code run much faster as well. In fact, using a cache for a file-based tree can make a huge difference in performance. You'll get at least double the performance, and sometimes much better than that.

Unfortunately, using Coptrs isn't always quite as transparent as we would like. In the next section, you'll see that, in some cases, we must coax the Coptrs along by manually intervening with the binding process. Also, don't forget that we must manually set dirty bits for any objects we modify. Failure to do this can wreak havoc on the data structures being stored to disk.

Releasing Coptrs

Coptrs are unbound from the buckets they reference by using a call to the Release( ) function:

void Coptrb::Release()

{

if (bound) {

bkt->Unlock();

bound = 0;

}

}

Unlike binding, releasing Coptrs is only done semi-automatically. It's not possible to tell, in general, when a Coptr needs to be released. There are some obvious situations when we can tell, however. For instance, you saw automatic releasing during Coptr assignments. Another situation occurs when the Coptr gets destroyed. Here is the Coptr destructor:

Coptrb::~Coptrb() {Release(); }

When we use Coptrs in code involving straight iteration, the two automatic release points of assignment and destruction work just fine. However, consider the following recursive inorder traversal of a file-based tree:

void InOrder(Coptr<Node> t)

{

while(t) {

InOrder(Coptr<Node>(t, t->left));

Visit(t);

t = t->right; // Automatic releasing takes place here

}

}

Consider what happens during this function. The copy constructor for t is called during the pass-by-value parameter passing. At this point, t is unbound. In the recursive call, a temporary Coptr is created by using the copy constructor with a second argument, which indicates the address for the Coptr. The address given is to t's left child, and the expression t->left causes t to become bound. The temporary Coptr is unbound. As a side effect of the copy, the temporary Coptr gets t's cache pointer, which is important because all Coptrs are required to reference a cache. We must use the same cache that t has, or things would quickly go haywire.

When the recursive function call is made, another unbound copy of the temporary Coptr is made. This copy becomes bound during the next recursive call, and so on. When we return from a recursive call, the temporary copy gets released and destroyed automatically.

At each level of recursion, a new version of t is bound. Recall that the buckets for these bound Coptrs are locked and can't be reused. Essentially, the cache must have as many buckets as the depth of the tree, plus a few extra buckets for other Coptrs that might be in use. Otherwise, the cache would overflow. If you don't want to calculate ahead of time how big the cache needs to be (which would tend be difficult anyway), the alternative is to ensure that t is unbound during the recursion. This can be done by manually releasing t before the recursive call. We may also want to release the temporary Coptr before calling Visit( ). For example:

void InOrder(Coptr<Node> t)

{

while(t) {

Coptr<Node> temp(t, t->left);

t.Release();      // Must do this or cache becomes full

InOrder(temp);

temp.Release( );  // May want to do this

Visit(t);         // t probably gets rebound during this call

t = t->right;     // t will be rebound here if not already bound

}

}

Note that t probably gets rebound during the Visit( ) function. If we're lucky, the rebinding will be fast, since t's bucket pointer may still be pointing to the right bucket. Remember that releasing a Coptr doesn't cause the bucket to go away. The bucket is merely made available for reuse, and then only if there are no other Coptrs referencing the bucket. If you're lucky, the bucket will still be intact when you need it again.

The conclusion from this discussion is that, as long as you're using iteration, Coptrs can be manipulated virtually the same as ordinary pointers. (You still must set dirty bits, though.) However, with recursion, the fact that Coptrs are really more than ordinary pointers becomes apparent, and we must carefully consider the bind and release points.

Allocating Objects Referenced by Coptrs

Since Coptrs look a lot like ordinary pointers, it would be nice if we could allocate and construct file-based objects with them using the new operator. In fact, we can, as you'll now see. The workhorse function is the following Alloc( ) function:

Bucketb *Coptrb::Alloc(size_t n, long p)

{

Release( ); // Let go of any currently bound data

if (p == 0) p = cache->fptr->Alloc(n);

bkt = cache->ReserveBkt(p, 0);

if (bkt) {

bkt->SetDirty(); // Data to be modified by the constructor

addr = p;

bound = 1;

}

else {

addr = 0; // bound will = 0 here as well

}

return bkt;

}

This function allocates n bytes (presumably sizeof(TYPE))at address p in the cache's file, and sets up a cache bucket to support it. If p isn't zero coming in, it means the TYPE object has already been allocated. (This can be used when creating a new object in the static area of the file.) Note that ReserveBkt( ) sets up the file address, cache pointer, and reference count, but it doesn't load any data. A pointer to the bucket is returned.

The Alloc( ) function is used by the following overloaded new operator:

template<class TYPE>

void *operator new(size_t n, Coptr<TYPE> &cp, long p)

{

return (TYPE *)((Bucket<TYPE> *)(cp.Alloc(n, p)));

}

This function has only one statement, but there's a lot of subtlety and power packed into that statement. First, Alloc( ) is called using size n and address p (which defaults to zero), and a pointer to a reserved bucket is returned. Since the pointer is typed as a generic Bucketb pointer, it is type cast to a more specific Bucket<TYPE> pointer. Then, some magic happens. We type cast the pointer to be a TYPE pointer. Recall that Bucket<TYPE> is multiply derived. Thus, the type cast to a TYPE pointer causes a pointer to the TYPE portion of the bucket to be computed. This TYPE pointer is then returned from new, and is passed to the constructor as the this pointer of the TYPE object to be constructed. (We had to pass the pointer back as a void pointer to satisfy the syntax for the new operator.)

Here is an example of the overloaded new operator in action:

Coptr<Node> t(cache);    // Set up a null Coptr

new(t) Node('a', 0, 0);  // Allocate and construct a Node

The single call to new causes a Node to be allocated in the file used by the cache, a bucket to be reserved in the cache, and the Node portion of that bucket to be constructed. Note that, in effect, we're partially constructing a multiply-derived object (think about that for a while). This is a nice trick to have in your arsenal.

If you want to construct an object at a pre-allocated address (perhaps in the static area of the file), you can pass an extra parameter to new:

new(t, 42L) Node('a', 0, 0); // Construct object at address 42

This form of new is the file-based analogue to in-place construction.

You can also free a file-based object by way of a Coptr. The following Delete( ) function is used. It calls a lower-level Delete( ) function of the Coptrb class.

template<class TYPE>

void Coptr<TYPE>::Delete()

// Frees the data associated with the Coptr c. The Coptr is

// then set to null.

{

Coptrb::Delete(sizeof(TYPE));

}

void Coptrb::Delete(unsigned n)

// De-allocates the file object pointed to by this Coptr, supposedly

// of size n. De-allocation only takes place if this Coptr is

// pointing to a bucket bound by nothing else. The Coptr is unbound

// afterward and both the bucket and the Coptr are made null.

{

if (bound && bkt->IsLocked( ) == 1

!bound && (!bkt      !bkt->IsLocked())) {

cache->fptr->Free(n, addr);

Release();

bkt->MakeNull();

addr = 0;

}

else excp->THROW(DANGLINGPTR);

}

Note that you can only call Delete( ) for Coptrs that have unique buckets or aren't bound. Otherwise, you have dangling pointers and an exception is thrown. Here is an example that uses Delete( ):

Coptr<Node> t(cache);    // Set up a null coptr

new(t) Node('a', 0, 0);  // Allocate and construct a Node

. . . .

t.Delete( );             // Free the Node from the file

You might wonder why we can't use the delete operator. Here's one attempt at defining an overloaded version of delete:

template<class TYPE>

void Coptr<TYPE>::operator delete(void *p)

{

((Coptrb *)p)->Delete(sizeof(TYPE));

}

Due to the rules of C++, we must pass a void pointer to delete. While it's easy to do a type cast to a Coptrb pointer in calling the low-level Delete( ) function, the call to delete becomes awkward since we must take the address of the Coptr during the call. For instance:

Coptr<Node> t(cache);    // Set up a null coptr

new(t) Node('a', 0, 0);  // Allocate and construct a Node

. . . .

delete &t;               // Must take the address of t

This was deemed to be too confusing and error prone, so we decided to use a Delete( ) function call instead.

  CREATING FILE-BASED BINARY SEARCH TREES

On disk, you'll find complete code for building file-based binary search trees. This code uses all the techniques we've described in this chapter, and is a direct "port" of the memory-based binary search tree code given in Chapter 10. When you examine the file-based code, notice how similar the code is to the memory- based version.

Complete code for file-based binary search trees is given on disk in ftree.cpp and ftree2.cpp. The latter illustrates how easy it is to have more than one tree stored in the same file, each tree having its own cache.

  PERFORMANCE TUNING

You may wonder just how fast Coptrs are. Since they use caching implicitly, they will be reasonably fast. However, even disregarding slow file access, Coptrs won't be as fast as memory-based objects. That's because on every access to a Coptr, a test is made to see if it's bound. This test, while simple, still takes time. You can fine-tune the performance of your file-based objects by carefully analyzing the bind and release points, and handling the binding and unbinding yourself.

For example, you can use a Coptr initially to get access to a bucket, and then use an ordinary pointer to the bucket from that point on. Here's some code that illustrates this:

Coptr<Node> t(cache);

new(t) Node('a', 0, 0);

. . .

Bucket<Node> *p = &(*t);  // Get direct bucket pointer

. . .

p->left = 42;             // Access bucket directly

p->SetDirty();

. . .

t.Release();              // p no longer valid

Of course, using code like this leads to a lot of unsafe aliasing, so you must be extremely careful.

You can also optimize performance by manually intervening in the LRR caching strategy. Recall that, when using the caches designed here, a bucket moves to the front to become the MRR bucket only when that bucket has just been reserved for a Coptr. During the course of using the Coptr, the bucket may actually fall to the back of the list if it hasn't been reserved by other Coptrs in the interim. You can notify the cache that the bucket is being used frequently by calling the NotifyUseOf( ) function:

void Coptrb::NotifyUseOf()

// Moves the referenced bucket to the front of the

// cache queue. For cache performance tuning.

{

if (bound) cache->MoveToFront(bkt);

}

For example:

Coptr<Node> t;

. . .

t.NotifyUseOf();

You don't have to use NotifyUseOf( ). Even if a bucket is at the back of the list, it won't be reused as long as there are references to it.

  SUMMARY

In this chapter, you've seen a versatile and powerful way to manage objects stored on files. While the code is template-based to make it type-safe, there's a minimal amount of source code explosion. The classes were carefully designed so that most of the work is done with data-independent code that is shared by all types of objects stored.

To make using Coptrs simple and elegant, we've employed several multiple inheritance and type-casting tracks. Unfortunately, some current compilers can't handle the statements that use these tricks, even though legal C++ is used. However, the code as given compiles and runs fine with Borland C++ 3.1, currently the only widely used native-code compiler that supports templates.

Go to Chapter 14            Return to Table of Contents


Practical Data Structures: B-TREES

 Next Chapter Return to Table of Contents Previous Chapter

CHAPTER 14: B-TREES

In this, the final chapter of the book, we'll discuss B-trees, which are balanced multi-way trees. B-trees have become commonplace in database applications because they offer fast disk-based searching. B-trees were invented by Bayer and McCreight (see [BayeMc 72]) and are a natural evolution of earlier database designs. The basic premise behind these designs is that, to access a disk efficiently, it's best to load chunks of data, called pages, rather than small individual items.

In a B-tree, one tree node can be made to correspond to a page. Because binary nodes aren't usually large enough to take up one page, a B-tree node stores multiple keys and branches. A beneficial side effect of such multi-way nodes is that the height of the tree can be smaller than that of a binary tree. Using multi-way nodes also makes it easier to balance the tree. Both of these properties lead to even faster searching than can be obtained by using disk paging alone.

This chapter first shows the data structures used to implement B-trees, followed by the operations on B-trees in order of complexity--from searching to inserting and deleting. Since B-trees were designed with files in mind, we'll make our B-trees file-based, using the classes developed in Chapter 13 for this purpose.

PROPERTIES OF B-TREES

B-trees are classified by their order, which refers to the maximum number of branches in a node. For example, in a B-tree of order 4, each node can have up to 4 branches. Corresponding to these branches are 3 keys that help determine which branch to take during a se arch. In general, a B-tree of order m has nodes with up to m branches and m-1 keys. Note that the order specifies the maximum number of branches. A node may have fewer branches than the maximum. Figure 14.1 shows a B-tree of order 4, with all nodes full. You've seen B-trees of order 4 before. They are the 2-3-4 trees we introduced in Chapter 11. (We've drawn them differently in this chapter, however.)

Not all multi-way trees are B-trees. A multi-way tree is considered to be a B-tree only if it is balanced. In the classical definition of a B-tree of order m, the balance is achieved by maintaining the following properties:

Except for the root node, all nodes must have at least (m-1)/2 keys and (m-1)/2 + 1 branches. This means all nodes except the root are roughly half full. For example, in an order 5 B-tree, all non-root nodes must have at least two keys. For an order 4 B-tree, the nodes must have at least one key.

The leaves of the tree are always on the same level.

Note Some texts define the order of a B-tree differently than we have here. In the alternative definition, a B-tree of order d has nodes with a maximum of 2d keys and 2d+1 branches. Implied in this definition is that the maximum number of keys is always even, and the minimum number of keys in a non-root node is d. In our definition, the maximum number of keys can be odd.

As was the case with 2-3-4 trees, the key to efficient B-trees lies in maintaining the two properties just mentioned. Doing so will yield balanced trees that are reasonably compact.

MULTI-WAY NODES

At the heart of B-trees are the multi-way nodes that make up the trees. Multi-way nodes are essentially generalizations of binary nodes. Rather than show generic template-based nodes in this chapter, we'll show a direct design that's typical of what might be used in practice. (You're probably tired of seeing templates, anyway.)

For keys, we'll use a character string of some maximum size. Stored with each key is a long integer that represents the associated data. While in some applications you might be able to fit all of your data in the long integer, the intent is for this variable to be the file address of the data--as stored elsewhere. For example, you might store the B-tree in one file that serves as an index to another file holding the actual data.

Figure 14.1 A B-tree of order 4.

Node Entries

The Mnode Class

THE B-TREE CLASS

The following B-tree class, and its associated data structures, can be used to set up a B-tree:

// Function return status codes

const int NODE_OVERFLOW =  2;

const int SUCCESS       =  1;

const int FAIL          =  0;

const int DUPLKEY       = -1;

const int ALLOCERR      = -2;

struct BtreeHeader  {  // Stored with every tree

long root_addr;

unsigned order;

unsigned num_entries;

unsigned num_nodes;

int height;

};

class Btree  {         // Btree file class

protected:

FmgrPtr f;          // File the Btree is connected to

long bh_addr;       // Address of the Btree header

BtreeHeader bh;     // Btree header

Cache<Mnode> cache; // Node cache

Coptr<Mnode> root;  // Pointer to the root node

void ReadHdr( );

void WriteHdr( );

int Insert(Entry &e, Coptr<Mnode> t);

int Delete(Entry &e, Coptr<Mnode> t);

void RestoreBalance(Coptr<Mnode> p, int posn);

void RotateRight(Coptr<Mnode> p, int posn);

void RotateLeft(Coptr<Mnode> p, int posn);

void Merge(Coptr<Mnode> p, int posn);

static void PrintNode(Coptr<Mnode> &n);

static void PrintTree(Coptr<Mnode> t, int sp);

public:

Btree(int cs = 8);

~Btree( );

// File-management routines

int Connect(FmgrPtr &fptr, int create,

long bh_addr-sizeof(FmgrHeader));

void Disconnect( );

int Create(char *fname, long bh_addr=sizeof(FmgrHeader));

int Open(char *fname, Fmgr::AccessMode mode,

long bh_addr=sizeof(FmgrHeader));

void Flush(int clear=0);

void Close( );

int Search(Entry &e);

int FullSearch(const Entry &e):

int Add(char *k, long d=0);

int Remove(char *k, long d = 0);

int IsEmpty( ) const;

int IsOpen( ) const;

int OK( ) const;

int operator!( ) const;

operator int( ) const;

void ClearErr( );

void Statistics(int full);

void PrintTree( );

};

Complete code for the B-tree class is given on disk in the files btree.h and btree.cpp test program is provided in the file btreet1.cpp.

Since the B-tree class is file based, it has numerous file-management functions, such as Connect( ), Disconnect( ), Create( ), Open( ), Close( ), and Flush( ), among others. These functions allow multiple B-trees to be stored in a single file. Each B-tree has a header, defined in B-treeHeader, which points to the root node of the tree, and stores some other pertinent data used mostly for testing. The headers are meant to be stored in the static data area of the file, although their exact locations are user defined. Unfortunately, we don't have sufficient space to describe the file-management functions in detail. Refer to the disk for more information.

Each B-tree has its own cache of buckets that store Mnodes. Also, a Coptr to the root of the tree is kept for easy access. The constructor is used to initialize these members, as well as to set the file manager pointer to null at the outset:

Btree::Btree(int cs)

// Create a Btree with a null file manager, and with a

// cache size of cs

: f(0), cache(cs), root(cache)

{

}

  SEARCHING A B-TREE

The following Search( ) routine can be used to search a B-tree. The searching algorithm is very similar to that used for a binary tree, except now there are more keys and branches to choose from. The Mnode Search( ) function is at the heart of the search. Since the tree is file based, note how Coptrs are used instead of Mnode pointers:

int Btree::Search(Entry &e)

// Search the tree for the first node having a matching

// entry (i.e., keys must match). If found, the data

// field of e is filled in. Returns SUCCESS or FAIL.

{

Coptr<Mnode> t(root);

int rv, posn;

while(t)  {

rv = t->Search(e, posn);  // Search node t, get back posn

if (rv == 0) {            // Found a match

e = t - >entry[posn]; // Causes the data to be filled in

return SUCCESS;

}

t = t -> Branch(posn);    // No match, keep walking

}

return FAIL;

}

The Search( ) function assumes you've set up an entry with the key field defined. When a match is found, the data field is filled in with the appropriate value. This example calls Search( ):

Entry e("look for me");

if (mytree.Search(e) == SUCCESS)  {

cout << "Entry found, with data: " << e.data << '\n';

}

A companion function, FullSearch( ), compares both the key and data fields of the entry. This function is similar to search, except the call to Mnode's Search( ) function is replaced by a call to Mnode's FullSearch( ) function.

  INSERTION INTO A B-TREE

In Chapter 11, we described how to insert nodes into a 2-3-4 tree. The method involves inserting a new entry into a leaf and, if the appropriate leaf is full, it is split in two, with node splits propagating up the tree as necessary. Not surprisingly, a similar method is used for general B-trees. The only drawback is that it isn't possible to define an efficient, single-pass, top-down algorithm for general B-trees. (At least, nobody has come up with such a method.) Instead, we'll use a two-pass recursive algorithm.

First, we walk down the tree until we either find a matching key and data pair (which means we have a duplicate entry--not allowed) or we hit a leaf node. At this point, we attempt to insert the new entry into the leaf. If the leaf isn't already full, we can simply do the insertion. Otherwise, the node is split in half. For this technique to work, we must send one of the entries up the tree to be inserted into the parent. We always use the median entry (as defined in sorted order) for this purpose. Note that the median may actually be the entry we tried to insert in the first place.

If the parent is full and can't handle the median entry, the parent must be split as well, with the parent's median entry (which may be different from the original median entry) sent further up the tree. Eventually, the root node might have to be split in half, and a new root created to contain the median entry. Due to the bottom-up node splitting, B-trees always grow at the root. This keeps the leaves all at the same level, and results in a well-balanced tree.

Figure 14.3 shows some insertions when the corresponding leaf nodes aren't full. Figure 14.4 shows a node being split to accommodate a new key. Figure 14.5 shows an example of two splits occurring, one which involves growing a new root.

Figure 14.3 Inserting into a B-tree without splits.

(a) Original tree

(b) After adding b

(c) After adding j

(d) After adding o

Figure 14.4 Inserting into a B-tree with a split.

(a) Original tree

(b) Inserting f

(c) Node split in half

(d) Median inserted into parent

The insertion algorithm is partitioned into two functions. The Add( ) function is the top-level "driver" function. It constructs an entry to be searched, and then calls the recursive Insert( ) function. This latter function walks down to a leaf node, inserts the entry there, and then propagates any splits up the tree as the recursion unwinds. Function return status codes are used to control the recursion.

When Insert( ) returns a NODE_OVERFLOW code, this means the node below was split, and the median entry is passed back for further processing.

Figure 14.5 Propagating splits and growing a new root.

(a) Before inserting f and causing splits

(b) After two splits, including the root

Eventually, the top of the tree is reached, and the last call to Insert( ) is unwound. At this point, Add( ) checks to see if the node that was split was the root. If so, a new root is created, with the median entry passed back as the root's sole entry.

int Btree::Add(char *k, long d)

// Creates a new entry with key k and data d, and attempts to

// add the entry to the tree. Returns SUCCESS, DUPLKEY, or

// ALLOCERR.

{

Entry e(k, d);

int rv = Insert(e, root);

if (rv == NODE_OVERFLOW) { // Need to grow a new root

Coptr<Mnode> old_root(root);

new(root) Mnode;

if (root == 0) return ALLOCERR;

bh.num_nodes++: bh.height++;

bh.root_addr = root;

root->left = old_root;

root->InsEntry(e, 0);

rv = SUCCESS;

}

if (rv == SUCCESS) bh.num_entries++;

return rv;

}

int Btree::Insert(Entry &e, Coptr<Mnode> t)

// Recursive function that tries to insert entry e into subtree t.

// Returns SUCCESS, DUPLKEY, ALLOCERR, or NODE_OVERFLOW.

// If NODE_OVERFLOW, then e becomes the median_entry to pass

// back to t's parent.

{

int rv, posn;

if (t == 0) {

// We went beyond the leaves. Pretend that we've overflowed.

e.right = 0;

return NODE_OVERFLOW;

}

// Search the node t. If entry found, send back duplicate key error.

rv = t->FullSearch(e, posn); // Use both key and data

if (rv == 0) return DUPLKEY;

// Entry not found; go down to the appropriate leaf to store entry

Coptr<Mnode> child(t);

child = t->Branch(posn);

t.Release( );            // Prevents cache from filling up during recursion

rv = Insert(e, child);   // Recursive call

// Back from recursion;  see if we have an overflow

child.Release( );        // Prevents cache from filling up

if (rv != NODE_OVERFLOW) return rv;

// We have overflowed the node below us. See if we can

// insert the entry e (now the median) into node t.

posn++;  // Move to the appropriate entry for insertion

if (!t->IsFull( )) {     // Simply add entry to node t

t->InsEntry(e, posn);

t->SetDirty( );

return SUCCESS;

}

else {  // Need to split node t.

// Create new node to be the right node. t will be the left node.

Coptr<Mnode> r(t);

new(r) Mnode;

if (r == 0) return ALLOCERR;

bh.num_nodes++;

// Move roughly half the nodes to the right node. If inserting

// entry into left node, we need to move the split position back

// one, so that when the median is taken, the split node will have

// enough entries.

int split_posn = NBR/2;

if (posn < split_posn) split_posn-;

t->Split(*r, split_posn);

t->SetDirty( );

// Determine where entry e should go

if (posn > split_posn) {  // e goes to the right node

// First entry of right node is the median

Entry median_entry = r->entry[0];

r->DelEntry(0);

// Insert e into its correct spot

r->InsEntry(e, posn-split_posn-1);

e = median_entry;     // For passing up the tree

}

else if (posn < split_posn) { // e goes to the left node

t->InsEntry(e, posn);  t->SetDirty( );

e = r->entry[0]; // Get median from the right

r->DelEntry(0);

}

// At this point, e is median entry to add to parent.

// Record what e's right pointer is going to be.

r->left = e.right;

e.right = r;

return NODE_OVERFLOW;

}

}

As you examine the Add( ) and Insert( ) routines, note how Coptrs allow us to write the routines almost as though we were working with trees in memory. The process isn't quite automatic, as we've explained in Chapter 13. For example, before a recursive call, we must release any Coptrs that might be bound--to prevent the cache from becoming full.

We must also remember to set the dirty bits for any nodes that might be modified. We've made it a convention to set dirty bits each time we modify a node, even though the dirty bits might already have been set due to prior modifications. With careful analysis, it's possible to optimize the setting of the dirty bits. But when you first write the code, the redundancy is well worth it. If you forget to set a dirty bit, the code will eventually fail, but the problem may remain undetected for a long time and may be difficult to locate.

  DELETING ENTRIES FROM A B-TREE

As you might expect, deleting from a B-tree is much more involved than insertion. We use the same technique that's used for all search trees. When the entry to be deleted isn't in a leaf node, we replace the entry with its successor, and then delete the successor entry from its original home, which will always be a leaf node. Recall that, with red-black trees (and their 2-3-4 tree equivalents), even deleting from a leaf isn't always so simple. We must maintain the balance of the tree. The same applies to B-trees.

B-tree Rotations

For B-trees, maintaining balance translates into ensuring that each non-root node is at least half full. (Notice that the splitting process just described for insertions does just that.) After deleting an entry from a leaf, we check to see whether the node still has the minimum required number of entries. If not, we must borrow an entry from some other node. The appropriate nodes to borrow entries from are the left and right siblings. We can't just simply move an entry from a sibling, however. We must maintain the order of the entries so that the tree can be searched properly. We can do this by first moving the sibling entry into the parent node, then moving the original parent entry into the node needing the entry.

This borrowing process is interesting because, in effect, we end up performing tree rotations that are the generalized versions of what we used for binary trees. Figure 14.6 illustrates both left and right rotations. When doing a left rotation, we always borrow the leftmost entry of the right sibling. In a right rotation, the rightmost entry of the left sibling is used.

When both the left and right siblings have entries to spare, it's arbitrary which sibling you borrow from. In the code we'll present later, we always choose the left sibling. In some cases, neither sibling has any extra entries. When this happens, we can combine the given node with one of the siblings. (Again, the choice is arbitrary.) This merging process involves an entry from the parent node as well.

Merging Nodes

Figure 14.7 illustrates two examples that merge a node with its left and right siblings. In this figure, we've assumed the nodes shown are in the middle of a B-tree. As we did with the red-black trees defined in Chapter 11, we've represented external nodes (which are subtrees that may be null) using links that go nowhere and that are labeled with numbers. These numbers help you to easily see how the links are re-arranged. Note that, in the case of a left merge, the original parent entry has its right link set to the left link of the right node. Thus, we don't lose track of any nodes below ones involved in the merge. Right-hand merges have a similar linking process.

Figure 14.6 B-tree rotations.

(a) Left rotation (borrowing from the right)

(b) Right rotation (borrowing from the left)

Figure 14.7 Merging B-tree nodes.

(a) Before merging with the left sibling

(b) After merge

(c) Before merging with the right sibling

(d) After merge

When nodes are merged, we end up with one less entry in the parent node. It's possible for the parent to end up with too few entries, so it must borrow entries or merge with one of its siblings. This borrowing and merging may thus propagate up the tree. Note that it's legal for the root to have fewer than the minimum number of entries. But what if the root becomes empty due to a merge? In this case, the tree shrinks one level, with the new root being the node to the left of the old empty root. Figure 14.8 illustrates this process.

The Deletion Functions

It's relatively easy to describe how deletions work in a B-tree, but it's another matter entirely to write code for deletions. We've split the task into six functions: Remove( ), Delete( ), RestoreBalance( ), RotateLeft( ), RotateRight( ), and Merge( ). The Remove( ) function is the "driver," which sets up the entry to delete and calls the recursive Delete( ) function. This latter function searches the tree for the entry to delete, using recursive calls to itself. When the entry is found, the tree is further traversed in search of the successor entry to use as the replacement. When the entry has been replaced, the function is recursively called again to delete the successor entry in its original leaf node.

Figure 14.8 Shrinking a B-tree.

(a) Before merge

(b) After merge

(c) Final tree

The recursion bottoms out when an entry is to be deleted from a leaf node. At this point, the entry is removed from the node. Then, RestoreBalance( ) is called to do whatever rotation or merging is necessary to ensure that the node with the deleted entry has at least the minimum number of entries. As the recursion in Delete( ) unwinds, the merging is propagated up the tree as needed. Eventually, the program execution returns to the Remove( ) function. A test is made to see if the root node has become empty due to a merge that took place below it. If so, the root is removed from the tree and the node to the root's left becomes the new root. If the old empty root is the only node in the tree, it is left empty. Thus, we also have at least one node in the tree, even if it's empty.

Here are the six functions used for deletions:

int Btree::Remove(char *k, long d)

// Deletes entry having key k and data d from the tree.

// Returns SUCCESS, or FAIL if we couldn't find the entry.

{

Entry e(k, d);

int rv = Delete(e, root);

if (rv == SUCCESS && root->IsEmpty( ) && bh.height > 1) {

// We need to shrink the tree

Coptr<Mnode> p = root;

root = root->Branch(-1);

bh.root_addr = root;

bh.num_nodes-; bh.height-;

p.Delete( );

}

return rv;

}

int Btree::Delete(Entry &e, Coptr<Mnode> p)

// Recursive function that deletes entry e from the subtree

// with root p. Returns SUCCESS, or FAIL if we couldn't find

// the entry.

{

Coptr<Mnode> t(p), s(p);

int rv, posn, sr;

sr = p->FullSearch(e, posn); // Search node p

t = p->Branch(posn);         // Get ready to walk down the tree

if (sr == 0) {

// p has entry to delete. Replace the entry with its

// successor, if there is one.

if (t) {

s = t;

while(s->Branch(-1)) s = s->Branch(-1);

p->entry[posn] = s->entry[0];

p->Branch(posn) = t;      // Remember to restore the branch

p->SetDirty( );      // Don't forget

// Now, using recursion, delete the successor entry.

// Release the coptrs in use, to keep cache from.

// filling up during the recursion, and to prevent

// dangling pointer exceptions

p.Release( );

Entry &se = s->entry[0];

s.Release( );

if (Delete(se, t) != SUCCESS)

excp->THROW(ASSERTERR); // We should have found the entry

}

else { // At a leaf, so the deletion is easy

p->DelEntry(posn);

p->SetDirty( ); // Don't forget

bh.num_entries-;

}

rv = SUCCESS;

}

else {

if (t) {

// Walk down the tree, looking for entry

p.Release( );   // Keep cache from filling up on recursive call

rv = Delete(e, t);

}

else return FAIL;  // Couldn't find said entry

}

if (rv == SUCCESS && t && t->IsPoor( )) {

// The node below p doesn't have enough entries. We need to

// move some entries to it and thereby restore the balance.

t.Release( ); // To prevent dangling ptrs

RestoreBalance(p, posn);

}

return rv;

}

void Btree::RestoreBalance(Coptr<Mnode> p, int posn)

// Node down branch at position posn in node p has one too

// few entries. Give it an entry from either its left or right

// sibling, or perhaps just merge the node with a sibling.

{

Coptr<Mnode> t(p);

if (posn == -1) {

// We have no left sibling, so try the right sibling

t = p->Branch(0);

if (t->IsPlentiful( ))  { // Can we borrow from the right?

RotateLeft(p, 0);     // Do so

}

else {          // Right sibling has no spare entries

t.Release( ); // To prevent dangling ptr errors

Merge(p, 0);  // Merge with right node and parent entry

}

}

else if (posn == p->LastPosn( )) {

// We have no right sibling, so try the left sibling

t = p->Branch(posn-1);

if (t->IsPlentiful( )) {  // Can we borrow from the left?

RotateRight(p, posn); // Do so

}

else { // Left sibling has no spare entries.

t.Release( );   // To prevent dangling ptr errors

Merge(p, posn); // Merge with left node and parent entry

}

}

else {

// We have both left and right siblings

t = p->Branch(posn-1);

if (t->IsPlentiful( )) {  // Can we borrow from the left?

RotateRight(p, posn); // Do so

}

else {

t = p->Branch(posn+1);

if (t->IsPlentiful( ))  {  // Can we borrow from the right?

RotateLeft(p, posn+1); // Do so

}

else {

// Neither the left or right sibling has spare entries.

// Merge arbitrarily with the left node.

t.Release( );   // To prevent dangling ptr errors

Merge(p, posn); // Merging with the left

}

}

}

}

void Btree::RotateRight(Coptr<Mnode> p, int posn)

// Performs a "right rotation" using the entry at node p,

// position posn as the pivot point. Assumes p is not

// a leaf and that a left and right child exist.

// Also assumes right child isn't full, and that p and

// left child aren't empty.

{

Coptr<Mnode> l(p), r(p);

r = p->Branch(posn);   // Right child of p

l = p->Branch(posn-1); // Left child of p

// Move entry from parent p into rightmost entry of right node.

// This entry will have the old left branch of right node. (The

// left branch will be updated later.)

p->Branch(posn) = r->left;       p->SetDirty( );

r->InsEntry(p->entry[posn], 0);  r->SetDirty( );

// Now, move rightmost entry of the left node into p. This

// entry's branch becomes the right node's left pointer.

// Be sure to point entry to right node.

int last_posn = l->LastPosn( );

r->left = l->Branch(last_posn);        r->SetDirty( );

l->Branch(last_posn) = r;              l->SetDirty( );

p->entry[posn] = l->entry[last_posn];  p->SetDirty( );

l->DelEntry(last_posn);                l->SetDirty( );

}

void Btree::RotateLeft(Coptr<Mnode> p, int posn)

// Does a "left rotation" using the entry at node p,

// position posn as the pivot point. Assumes p is not

// a leaf and that a left and right child exist.

// Also assumes left child isn't full, and that p and

// right child aren't empty

{

Coptr<Mnode> l(p) , r(p) ;

r = p->Branch(posn) ;    // Right child of p

l = p->Branch(posn-1) ;  // Left child of p

// Move entry from parent p into leftmost entry of left node.

// This entry gets the left pointer of the right node.

p->Branch(posn) = r->left ;       p->SetDirty( ) ;

l->Concatenate(p->entry[posn]) ;  l->SetDirty( ) ;

// Now, move rightmost entry of the right node into p. Make

// sure this entry points to the right node. Also, we have

// a new left branch of the right node.

r->left = r->Branch(0) ;

r->Branch(0) = r ;              r->SetDirty( ) ;

p->entry[posn] = r->entry[0] ;  p->SetDirty( ) ;

r->DelEntry(0) ;                r->SetDirty( ) ;

}

void Btree: :Merge(Coptr<Mnode> p, int posn)

// Merges the node on the branch left of the entry at position

// posn of node p with the entry of p, and the node on the

// branch to the right of the entry of p. Assumes posn in range.

{

Coptr<Mnode> l(p) , r(p) ; // Ptrs to left and right children

r = p->Branch(posn) ;     // Right child of p

l = p->Branch(posn-1) ;   // Left child of p

// (1) Fix entry of p to point to left branch of r

// (2) Insert this into leftmost entry of l

// (3) Add all of r's entries

// (4) Delete entry from p, then delete r

p->Branch(posn) = r->left ;       p->SetDirty( ) ;

l->Concatenate(p->entry[posn]) ;  l->SetDirty( ) ;

l->Concatenate(*r) ;              l->SetDirty( ) ;

p->DelEntry(posn) ;               p->SetDirty( ) ;

r.Delete( ) ;

bh.num_nodes- ;

}

  BEYOND BASIC B-TREES

The design we've just presented for B-trees is simple and basic. We "hard coded" the type of keys stored in the B-tree nodes as character strings. We also used global constants such as MAXKEYSIZE and NBR to size the nodes. These could obviously be made into template parameters, and it is possible to write a generic B-tree class that allows you to share the code for B-trees across all types of nodes.

For instance, you could build an abstract node class that doesn't know the order of the node, or the key types stored within it. Then, more type-specific template classes could be derived. However, it gets a little tricky to use the abstract nodes, since Coptrs are being used. Because the node types are parameters to Coptrs, the fact that one node type is derived from another does not mean that the corresponding Coptrs have any special base-to-derived relationship. The different Coptr classes remain siblings of the Coptrb class, and are otherwise not related. Thus, where you can use implicit derived-to-base type equivalence with ordinary node pointers, with Coptrs, you must do explicit type casting.

One problem with the B-tree design presented here: Fixed-length keys are used, which aren't desirable when the keys are character strings. A lot of wasted space results in the nodes when the keys aren't the maximum size. It's possible to design B-tree nodes that allow for variable-length keys, where each key occupies only as much room as needed in a node. As many keys as possible can be packed into each node. This means the nodes can have a variable number of keys and branches. The result is a variable order B-tree. In such a tree, we use half the size of the node as the criteria for minimum node size, rather than using half the number of keys.

Searching and inserting into a variable-order B-tree is about as simple as it is for trees with fixed orders. However, deletion is more complicated (as if it weren't complicated enough). With fixed-length keys, the rotation and merge operations are fairly straightforward. We can always guarantee that one key can take the place of another, because they all have the same size. With variable-length keys, such replacements can't be guaranteed. For example, if we attempt to do a rotation when the sibling key is larger than the parent key, theres no guarantee that the sibling key will fit in the parent node.

We designed the insertion and deletion routines using recursive functions. It's possible to use an explicit stack instead, and thus convert the recursion into iteration. Each stack element must have both a node pointer and an index representing which entry in the node we are on. If you use Coptrs for the node pointers, they must be released before putting them on the stack. Because the bucket pointers for released Coptrs will still be intact, there's a good chance that when you pop the Coptrs off the stack, you can quickly rebind them to their respective buckets.

There's an important advantage that results from using an explicit stack. You can have a built-in iterator, which allows you to move back and forth through the entries in the tree in sorted order. This feature is particularly handy in such applications as a customer contact list. For example, you can search for a customer record using the customer name, and then quickly scan back and forth through other customer records in sorted order.

For information about obtaining source code for a variable-order B-tree class with a built-in iterator, see the README file on disk.

In our B-tree design, we included "data" with each key. (Usually, this data is simply the file address of the real data record.) In some variations of B-trees, only the leaf nodes have data. The internal nodes only contain the keys to be used in the search process. These types of trees are sometimes called B+-trees. You can go one step further and, instead of storing the full keys in each node, you store only prefixes of each key, enough to guide the search process to the correct leaf node. Trees with these types of keys are often called prefix B-trees. For a discussion about some of these B-tree variants, see [Aoe 91].

  SUMMARY

B-trees, or variations of them, have become the data structures of choice for database applications. B-trees allow fast database searching, due to the ability to optimally size the nodes to the paging requirements of a file system, and due to the relative flatness that results by using multi-way nodes. By using a cache in conjunction with the B-tree nodes themselves, you can speed up the searching even more.

While B-trees were designed with files in mind, it's possible to use memory-based B-trees. However, you're probably better off using red-black trees for trees that totally reside in memory. That's because the larger nodes of a B-tree don't provide an advantage for memory-based designs. You must search sequentially through the entries of a node, looking for the appropriate branch to take. For nodes with many entries, this can become time-consuming. (With file-based trees, searching within a node is often masked by the relatively slower file access needed to retrieve the node.) Of course, you could use a binary search to find the appropriate location in the array of entries. However, you're not much better off than if you had simply used a binary tree.

Go to References             Return to Table of Contents


Practical Data Structures: USING C++ EFFECTIVELY

 Next Chapter Return to Table of Contents Previous Chapter

CHAPTER 2: USING C++ EFFECTIVELY

In the previous chapter, we discussed general design principles without regard to any specific programming language. In this chapter, we will look at design principles that are specific to C++. We'll also discuss our general philosophy on how to use C++ effectively. To explain these general principles and their underlying philosophy, we'll provide examples showing the attributes that make up a well-designed class, and we'll show how to use those attributes in derived classes. This chapter is by no means a tutorial on C++, nor does it pretend to even remotely cover all aspects of the language. It's assumed you already know the basics, but might want a brief refresher course on creating well-designed classes.

C++: AS HIGH LEVEL AS DESIRED, BUT NO HIGHER

One of the great attractions of C++ is the breadth and scope of programs that can be expressed in the language. Since C++ inherits virtually all the characteristics of C, you have the ability to code at a fairly low level, almost at the level of assembly language. At the same time, C++ allows you to express your designs at a fairly high level by using classes, constructors, destructors, and operator overloading. You can select the desired level of abstraction you would like to work with and, in fact, mix both high-level and low-level code together. Few other languages allow these combined capabilities.

C, Pascal, FORTRAN, and assembly are, for the most part, low-level languages because they can't be used to directly express high-level concepts. However, these languages have advantages, since they can produce code that's quite efficient. On the other hand, languages such as Lisp, Smalltalk, and Prolog are very high level, and it can be a joy working with the abstractions these languages allow. Unfortunately, these languages often produce very inefficient code. If you want to use their high-level constructs, you pay a high price. There needs to be some middle ground.

C++ provides that middle ground. When you want efficiency, you can have it. When you want high-level design, you can have that, too. Figure 2.1 illustrates this point. Of course, you can't always have both of these attributes at the same time. There is almost always a tradeoff between efficient code and high-level code. The important point is that C++ allows you to control this tradeoff.

Consider one of the main philosophies that went into creating C++: You should never have to pay for a feature you aren't using. This can be summed up another way:

C++: As high level as desired, but no higher.

This means you can pick the level of abstraction you would like to use, but not pay for any higher abstractions. Great care went into the design of C++ to allow this. The designers were very concerned about efficiency issues, but also wanted the language to be much more expressive than C in specifying higher-level concepts. Inline functions are a good example. Used properly, inline functions aid in the encapsulation of the inner workings of a data type, without causing the overhead normally associated with functions.

Figure 2.1 C++: Low level and high level.

It is highly recommended that you follow C++'s philosophy when designing your data structures and algorithms. Use as high a level of abstraction as you need, but don't get carried away and try to create the ultimate in general-purpose code. Don't try to create Smalltalk-like data structures unless that's your goal. As we pointed out in Chapter 1, simpler designs are often easier to debug and maintain than general designs, and will most likely be more efficient.

WELL-DESIGNED CLASSES

The most important feature of C++, especially in terms of data structure design, is the class construct. It's the class construct that allows you to encapsulate your data structures, hiding the inner workings. It's also the class construct that allows you to use inheritance, which is a powerful way of abstracting and reusing your code. Since classes are such a central feature, it's important to know what constitutes a good class design. Almost every non-trivial, well-designed class has the following attributes:

1. Encapsulated, or hidden, members

2. A default constructor

3. A copy constructor

4. An overloaded assignment operator

5. A virtual destructor

These attributes are illustrated in Figure 2.2. To demonstrate these five desired class attributes, let's take a look at a sample Image class. This class is used to create objects that store and display bitmaps. The bitmaps are allocated dynamically. The class definition is:

class Image {

protected:

int w, h;     // Dimensions of image

char *bitmap; // Bitmap of image

int Copy(int iw, int ih, char *bm);

public:

Image( );                          // Default constructor

Image(const Image &im);            // Copy constructor

Image(int iw, int ih, char *bm);

virtual ~Image( );                 // Virtual destructor

Image &operator=(const Image &im); // Overloaded assignment

virtual void Show(int xs, int ys);

};

The code for this Image class can be found on disk in the file ch2_1.cpp.

Figure 2.2 Components of a well-designed class.

You can see that the Image class has all five of the attributes we've just mentioned. Please note that you are not required to use all of these attributes in your classes. In fact, don't include them if they don't make sense. Rather, when you see a class that doesn't have all of these attributes, it should signal a warning to you, and you should ask yourself why the class doesn't have them.

Encapsulation

There are two access sections in the sample Image class: a protected section and a public section. The protected section contains the inner workings of an Image object that are meant to be hidden from users of the Image class. The public section defines the interface to the class--that is, those members that are meant to be used directly. Quite often, the private and protected members of a class are predominately data members, whereas the public members are predominately functions.

Recall from your C++ studies that protected members are private to all outside users of a class, except for any classes derived from the class. We could have made the protected members of Image private, and disallowed even derived classes from accessing them. Some people argue that protected access should rarely be used, if at all. They even go so far as to say that the very way derived classes are implemented violates the principles of encapsulation. We won't get embroiled in such controversies here. Instead, we'll use the following convention:

Whenever a class is designed to be a base class, with inheritance in mind, the hidden members of the class will be made protected. If the class isn't to be a base class, the hidden members will be made private.

It is up to you to determine whether you wish to follow this convention for your own classes. Don't be afraid to use your own judgment.

We've chosen to place all the data members of the Image class in the protected section, along with the following function, which is used to copy one bitmap into another:

int Image::Copy(int iw, int ih, char *bm)

// Private helper function that allocates a bitmap

// of size iw*ih, and copies the bits stored in

// bm into it.

// ASSUMES bitmap isn't already allocated.

// Returns 1 if allocation successful; 0 otherwise.

{

w = iw; h = ih;

bitmap = new char[w*];

if (bitmap) {  // Allocation worked

memcpy(bitmap, bm, w*);

return 1;

}

else { // Allocation failed. Set image to null.

w = 0; h = 0;

return 0; // Allocation failed

}

}

The Copy( ) function is used by several other functions in the class, and thus helps reuse some common functionality in the class. Functions like this are typical, and are sometimes called helper functions. Note that Copy( ) assumes that the bitmap isn't already allocated. Thus, it wouldn't be safe for this function to be used arbitrarily. For that reason, we've made it hidden. Only other member functions of the class have access to Copy( ), and they presumably will call it in a safe manner.

Default Constructors

The Image class has the following default constructor:

Image::Image()

//  Default constructor to create an empty image

{

w = 0; h = 0; bitmap = 0;

}

Default constructors are so-named because they are used whenever a constructor must be called implicitly, such as when declaring an array of objects. A default constructor doesn't require any arguments when called, meaning that it either has no arguments or that all of its arguments have default values.

What should a default constructor do? Most people design their default constructors to create a "null" object, whatever that means. (It depends on what the object is.) Quite often, it means setting all data members to zero. Here, we've chosen to create an image that has an empty bitmap.

If you don't declare a default constructor, one will be generated automatically by the compiler, but the data members will not be initialized by this generated constructor. Thus, it behooves you to define your own default constructor.

Copy Constructors

Copy constructors are used whenever an object is to be created and initialized as a copy of some other existing object of the same type. For example, copy constructors are called whenever an object is a parameter passed by value to a function, or when an object is the return value of a function. This copy construction approach is illustrated in Figure 2.3.

In these cases, the constructor is called implicitly to make a copy of the parameter being passed in or the object being returned. Here is the copy constructor for the Image class. Note how the helper function Copy( ) is called:

Image::Image(const Image &im)

// Copy constructor. Allocates room for bitmap

// of same size and copies im's image into it.

{

Copy(im.w, im.h, im.bitmap);

}

To be a copy constructor, a constructor must be callable by passing a single argument. That means it must either have only one argument or all arguments except the first must have default values. The first argument must be of the same class to which the constructor belongs, and must be passed by reference.

Figure 2.3 Copy constructing during function calling.

Note If you don't pass the first argument of a copy constructor by reference, you'll end up with recursive calls to the constructor that will go on indefinitely, locking up your program.

The Image( ) copy constructor meets these criteria. Note that we pass not just a reference to an Image object to the constructor, but a reference to a constant object:

Image::Image(const Image &im);

We could have also used:

Image::Image(Image &im);

In a later section, we'll discuss why it's preferrable to use the const modifier in cases like this.

If you don't declare a copy constructor, the compiler will generate one automatically for you. This generated constructor will do a member-wise copy. That is, it will copy each data member, one by one. If a member is a pointer, then only the value of the pointer is copied, not what the pointer points to. This can lead to serious memory management problems, because two objects can end up with members pointing to the same dynamic storage locations, a situation called aliasing. This becomes a problem when it's time to free up the dynamic storage, since both objects will attempt to free the same memory locations, wreaking havoc on the heap.

Aliasing can be avoided at copy-construction time by defining your own copy constructor and having that constructor make copies of any dynamic data being pointed to, rather than just copying the pointers. Our Image( ) copy constructor does just that by calling the Copy( ) helper function.

Overloaded Assignments

We've overloaded the assignment operator for the Image class as follows:

Image &Image::operator=(const Image &im)

// Assignment operator. Deallocates existing

// bitmap; allocates a new one of same size.

// as im, and copies im's image into it.

{

delete[] bitmap;

Copy(im.w, im.h, im.bitmap);

return *this;

}

The overloaded assignment operator, operator=( ), performs much the same function as the copy constructor by allowing one object to be copied into another. One crucial difference is that, unlike the copy constructor, operator=( ) assumes that the target object is already built, and already has a bitmap allocated. Thus, operator=( ) must first delete the allocated bitmap, so that Copy( ) can allocate a new one of the proper size.

The previous paragraph points out the critical difference between initializing an object and assigning an object. Copy constructors are used whenever an object is created and initialized to be a copy, whereas assignment operators are used when assignment takes place. While it's usually fairly obvious which is occurring, it isn't always so. Consider the following code:

Image snapshot(3, 5, "some bitmap");

Image picture = snapshot; // Assignment or initialization?

picture = snapshot;       // Assignment or initialization?

The last two statements look almost identical, but they do very different things. The first creates and initializes picture to be a copy of snapshot; the second assigns a copy of the snapshot image to the existing picture object. The first statement uses the copy constructor; the second statement uses the overloaded assignment operator. The tipoff is that the first contains a type declaration, using the type Image; the second doesn't. Thus, we know that the first statement is actually a call to a constructor.

Like the copy constructor, the operator=( ) function takes a single argument that's a reference to a constant Image object. The const modifier is used for the same reason as it is in the copy constructor (discussed later). The object is passed by reference for efficiency's sake, and for notational convenience. Contrast the following two operator=( ) functions. One uses a reference parameter, while the other uses a pointer parameter:

X &X::operator=(const X &v);

Y &Y::operator=(const Y *v);

X a, b;

Y c, d;

a = b;  // Clean syntax

c = &d; // Awkward syntax

You are not required to use pass by reference for overloaded assignment operators like you are for copy constructors, but as you can see, it's a good idea.

We show the assignment operator as returning a reference to the object that was the target of the assignment. You are not required to do this, but it's often done to allow assignments to be chained together, as in the following:

Image ia, ib, ic;

ia = ib = ic;

This would actually result in the following calls:

ia.operator=(ib.operator=(ic));

If you don't overload the assignment operator for a class, the compiler will subsititute member-wise copying whenever assignments occur between two objects of that class. As is the case with copy constructing, this situation can result in aliasing. Thus, it's a good idea to include an overloaded assignment operator, especially when dynamic storage is being pointed to by the object.

Virtual Destructors

To properly de-allocate the bitmap data of an image, the Image destructor is used:

Image::~Image()

// Destructor that de-allocates bitmap. Note that bitmap

// might be null coming in, but delete() is supposed

// to handle that.

{

delete[] bitmap;

}

Notice in the class definition that the destructor is virtual. This is done for safety reasons. For example, we'll be deriving a HideableImage class later on, which dynamically allocates some additional storage for a "backing store." This additional storage is to be used in saving what's underneath an image about to be drawn. When the destructor for HideableImage is called, we want the backing store to be freed as well as the bitmap data. It's critical that the correct destructor is called when the object is destroyed. By making the destructors virtual, code like the following will work just fine:

Image *p; // Can point to Image as well as HideableImage objects

Image *StaticPicture = new Image(3, 5, "Picture bits here");

HideableImage *NowYouSeeMe = new HideableImage(8, 10, "Bits go here");

p = StaticPicture;

delete p; // Image::~Image() called

p = NowYouSeeMe;

delete p; //, HideableImage::~HideableImage() called

It's not always necessary or desirable to make a destructor virtual. For instance, if you don't plan on deriving from a class, there is no need to make any functions virtual. If you go ahead and make the destructor for the class virtual, you will cause an extra pointer to be stored in every object created from the class, thus wasting space. (This extra pointer points to a virtual function table.) The following rule of thumb will help you decide whether to use a virtual destructor:

If your class uses virtual functions, you should always include a virtual destructor, even if that destructor doesn't do anything. (The derived class destructors might need to do something.) If your class needs no virtual functions, making the destructor virtual is optional. You might want to keep it non-virtual to save space.

Note     Even though destructors can and most likely should be virtual, constructors can never be virtual.

Constants

One often overlooked feature of C++ is the use of constants. Constants in C++ are more robust than they are in C. In C++, constants have local scope. That is, a constant can be safely declared (defined and allocated) in a C++ header file, without worrying about having duplicate definitions when modules are linked. This allows C++ constants to be used where macros once were in C. For example, you might have a header file like the following:

// trigcons.h

const double pi = 3.141592654;

const double degtorad = 0.017453292;

This header file can be included in more than one source file. Each source file gets its own private copies of the constants. Since these copies don't have global, external names, the linker won't complain when linking the source files.

Note     For simple constants, such as integers and floating point numbers, a C++ compiler may not actually create any storage for the constants, but instead will use the values directly when needed.

In C, it would not be a good idea to include the complete declarations of constants in a header file like the one above. In C, constants have global scope, and the storage for the constants might be duplicated. You would probably use constants as in the following files:

/* trigcons.h: Declare constants in header */

const double pi;         /* extern by default */

const double degtorad;   /* extern by default */

/* trigcons.c: Allocate and define constants in source file */

const double pi = 3.141592654;

const double degtorad = 0.017453292;

Or, you might be tempted to use macros, saving yourself the bother of having another source file to link in:

#define pi 3.141592654

#define degtorad 0.017453292

However, the macro approach is less descriptive, since it doesn't tell you the intended type of the constants.

Constants are often used in conjunction with parameters passed by reference. When you use constants in this way, you can get safe, elegant, and efficient code. Contrast the following two functions. One passes a reference to an X object (which is presumed to be some large structure), while the other passes a pointer to an X object:

void DoSomething(const X &x);

void DoSomethingElse(X *x);

X myobj;

DoSomething(myobj);      // Elegant, efficient, and safe

DoSomethingElse(&myobj); // Efficient, but not as elegant or safe

Passing by reference argument is cleaner than passing by pointer, since you don't need to take the address of the object via the & operator. However, some people object to using references this way, since it isn't obvious that the object might be modified inside the function. With the pointer method, the & operator sends a clear warning flag. Using the const modifier solves that problem for the reference argument case, since the function won't be able to modify the object. (Actually, the function can modify the object by using some clever type casting, but the programmer must consciously choose to do that.)

You will see the const X &x syntax used quite often in this book. In particular, it's considered good style to use this syntax for overloaded assignment operators and copy constructors.

Another use of the const modifier is to declare constant member functions. Ordinarily, if you declare an object to be constant, then you can't call any of the object's functions. The functions might change the internal state of the object, violating the concept of a constant object. However, if a function truly doesn't change any part of the object, you can declare the function to be constant, and the compiler will let that function be called. Here's an example:

class TwoNum {

private:

int a, b;

public:

TwoNum(int aa, int bb);

int First() { return a; }       // Not a constant function

int Second() const { return b;} // A constant function

};

const TwoNum x(17, 42);

int y = x.First();  // Not legal

int z = x.Second(); // Legal

The First( ) function can't be called for the constant object x, since it's an ordinary member function. The Second( ) function can be called, since it was a declared constant. Note that the const keyword is placed after the parameter list but before the function body.

It's a good idea to declare member functions as being constant whenever possible. Here's why:

1.     Someone might decide to declare an object of the class as being constant. By handling this possibility ahead of time, you've made the class more usable.

2.     The const declaration allows the compiler to perform more optimizations.

Declaring an object constant happens more than you might think, especially if you are following good coding style. For instance, suppose we added a copy constructor for the TwoNum class, and chose to use the functions First( ) and Second( ) to aid in the copying:

TwoNum::TwoNum(const TwoNum &tn)

{

a = tn.First();  // Not legal

b = tn.Second(); // Legal

}

We could have avoided the problem here by using the members tn.a and tn.b directly. But you might not always be so fortunate in your classes.

Don't overlook using the const modifier as an aid to the compiler for optimization. Consider the following loop:

TwoNum tn(17, 42);

for (int i = 0; i<10; i++) {

int y = tn.Second( );

...

}

Since Second( ) is declared as a constant member function, the compiler knows that tn is not modified by the statement y = tn.Second( ). Thus, if the data members of tn are for some reason already loaded in registers, they won't need to be reloaded after the statement, making the loop more efficient. Of course, the ability of your compiler to perform such optimizations is implementation-dependent.

 USING INHERITANCE

Inheritance is a powerful part of C++ programming. Through inheritance, which is achieved by class derivation, you can express type-subtype relationships, as well as elegantly share and reuse code common to a system of classes. We'll now continue our Image class example to derive a new type of Image class, one that allows the image to be hidden and then later restored:

class HideableImage : public Image {

protected:

int showing;         // Flag indicating that image is showing

int x, y;            // Location where image is showing

char *backing_store; // Bitmap of what's underneath

public:

HideableImage();

HideableImage(const HideableImage &hi);

HideableImage(int iw, int ih, char *bm);

virtual ~HideableImage();

HideableImage &operator=(const Image &im);

virtual void Show(int xs, int ys);

void Hide();

};

The code for this class can be found on disk in the file ch2_1.cpp.

The HideableImage class adds data members to keep track of whether the image is currently showing and where. It also adds a "backing store" buffer, which keeps a copy of what was on the screen underneath the image before the image was shown. The Show( ) function is overidden to first copy to the backing store before displaying the image, and a Hide( ) function is added to allow the image to be temporarily hidden.

Like the Image class, the five attributes we've already listed are desirable for the HideableImage class. HideableImage maintains a protected section, which hides the additional variables, and it has a default constructor, copy constructor, overloaded assignment, and a virtual destructor. We'll now look at how each of these attributes is affected by the inheritance process.

Derived Class Constructors and Destructors

Derived Class Overloaded Assignments

Derived Class Constructors and Destructors

The HideableImage class has the following constructors defined:

HideableImage::HideableImage()

// Default constructor sets everything to zero.

// Note that base class constructor called implicitly

// before the body of this function executes.

{

x = 0; y = 0; showing = 0; backing_store = 0;

}

HideableImage::HideableImage(const HideableImage &hi)

// Copy constructor. Note that the backing_store

// and associated flags are NOT copied. Thus, the

// base class constructor can do most of the work.

: Image(hi) // Call base class constructor explicitly

{

// Initialize to "not showing," then allocate backing_store

x = 0; y = 0; showing = 0;

backing_store = new char[w*h];

}

HideableImage::HideableImage(int iw, int ih, char *bm)

// General constructor. Base class constructor does

// a lot of the work.

: Image(iw, ih, bm) // Call base class constructor explicitly

{

// Initialize to "not showing," then allocate backing_store

x = 0; y = 0; showing = 0;

backing_store = new char[w*h];

}

These constructors all have one thing in common: they call the base class constructor. The derived class default constructor calls the default base class constructor implicitly. The other derived class constructors call the respective base class constructors explicitly. In each case, the base class constructor is called before the derived class constructor executes.

The way to remember the order of constructor calls is to look at how the base class constructors are called explicitly. Their calls are placed between the constructor parameter list and the constructor body, in the member initialization list:

HideableImage::HideableImage(const HideableImage &hi)

: Image(hi) // Call to base class constructor

{

x = 0; y = 0; showing = 0;

backing_store = new char[w*h];

}

The placement of the member initialization list correctly suggests when the base class constructors are called.

The member initialization list can also be used to initialize members of the class (hence its name). For instance, we could modify the constructor to initialize x and y to zero, as follows:

HideableImage::HideableImage(const HideableImage &hi)

: x(0), y(0), Image(hi)

{

showing = 0;

backing_store = new char[w*h];

}

Here, the syntax x(0), y(0), means to set x = 0, and y = 0. If x and y were objects of classes with constructors, the appropriate constructor calls would take place instead. Like the base class constructor calls, these initializations take place before the constructor body executes, so x and y get initialized before backing_store and showing. However, x and y get initialized after the call to the base class Image( ) constructor, even though they appear first in the list. That's because base class constructor calls always take place before the members of a derived class are initialized.

Except for the order in which the initializations take place, this modified constructor works the same as the original one. So why initialize members in the member initialization list? The most common reason is to initialize members that are constants, references, or objects that require constructors to be called explicitly in order to initialize them. (That is the only way these types of members can be initialized.) For instance:

class FunnyNums {

private:

const int a;

const int &b;

CompoundNum c;

public:

FunnyNums(int u, const CompoundNum &v);

};

FunnyNums::FunnyNums(int u, const CompoundNum &v)

// Initialize const a to have value u, and for reference

// b to point to a. Also, copy-construct c.

: a(u), b(a), c(v)

{

// Nothing else to do

}

In the FunnyNums constructor, a gets initialized before b (and before c), not because a is first in the initialization list, but because it was declared first in the class definition. Actually, you shouldn't rely on the order in which members get initialized, as we've done here; it's not considered good style.

Because base class constructors must be called for derived class objects, it's very rare for a derived class not to have any constructors. This would happen only if the base class has only a default constructor or no constructor at all. In this case, you don't need to declare a derived class constructor. A default one will be generated, which will call the base class constructor.

Sometimes, the derived class constructors do nothing more than turn around and call the base class constructors. It can be tedious to declare derived class constructors for cases like these. But at least the constructors can be declared inline, to hopefully avoid function call overhead.

Derived class destructors work just the opposite of derived class constructors. First, the members of the derived class are destroyed (with their destructors being called if they have any); then, the derived class constructor body is executed; and finally, the base class destructor is called, implicitly. For example, here is the HideableImage destructor:

HideableImage::~HideableImage()

// Destructor de-allocates both the backing store

// and the shown bitmap. Note that the base class

// destructor, which de-allocates the shown bitmap,

// is called implicitly AFTER this body executes.

{

Hide(); // Hide image first, if not already hidden

delete[] backing_store;

}

Note that this destructor was declared virtual in the class definition, per our rules on well-designed classes.

Derived Class Overloaded Assignments

Derived classes can have overloaded assignment operators, just as base classes do. Assignments between two derived class objects work as follows: if the derived class has an overloaded assignment, it is called. Otherwise, a member-wise copy is performed on each additional member of the derived class (that is, on those members not in the base class). If the base class has an overloaded assignment operator, it is called for the base class members; otherwise, the base class members are copied in a member-wise fashion.

The consequences of these rules are twofold:

1.      If the derived class object has members pointing to dynamic storage, you'll want to overload the assignment operator for the derived class to take care of potential aliasing problems. Of course, if the dynamic data is pointed to by base class members, then presumably an overloaded base class assignment operator is already handling the problem.

2.      If you do overload the assignment operator in the derived class, you must be sure that any base class overloaded assignment operator function is called, too. Or at least ensure that the actions that might have been performed by the base class assignment are performed in the derived class assignment operator function.

These two rules beg for an example. Here is the HideableImage( ) overloaded assignment operator, which embodies both rules:

HideableImage &HideableImage::operator=(const Image &im)

// Overloaded assignment operator. Copy just the

// image to be shown, and not the backing store.

// A new backing store will be created, though,

// with the same size.

{

// We must re-allocate backing_store to new

// size. But first we must hide the image

// if it was showing.

int ss = showing; // Keep track of show status

Hide();

delete[] backing_store;

// Let base class assignment do its thing.

Image::operator=(im);

// Now, re-allocate the backing store to correct size

backing_store = new char[w*h];

if (ss) Show(x, y); // Redisplay image if necessary

return *this;

}

Figure 2.4 illustrates the workings of the HideableImage::operator=( ) function, which must take care of two types of dynamic memory: the backing store and the bitmap stored in the base class. The storage for the backing store must be resized in case the source image has different dimensions. The bitmap storage is handled via an explicit call to the Image::operator=( ) function:

Image::operator=(im);

You might have noticed the parameter type used in the derived class assignment operator function:

HideableImage &HideableImage::operator=(const Image &im);

Figure 2.4 Assigning hideable images.

Instead of typing the parameter as a HideableImage, we've typed it as an Image. Why have we done this? We can assign an Image object to a HideableImage object just as easily as we can assign a HideableImage object to another HideableImage object. We could have provided two overloaded operators for this, but since the code would be the same, we chose to use just one function. Because a HideableImage object can be used anywhere an Image object is used, we chose to type the parameter as an Image object. Note that, had we done the reverse,

HideableImage &HideableImage::operator=(const HideableImage &im);

we would not be able to assign an Image object to a HideableImage object.

Although it's a little unusual, we'll use this technique in several places--particularly with the Array classes used in Chapter 4.

 COOPERATING CLASSES

Class derivation, as you've seen in the previous sections, is just one example of two classes working together to create a data structure. It's also possible for two or more cooperating classes, not related by inheritance, to work in tandem. In this section, we'll show as an example a singly-linked list implementation, which has node and list classes. In this manner, not only will you learn about setting up cooperating classes, but you'll also be introduced to the inner workings of linked lists. We won't cover all aspects of linked lists here. See Chapter 8 for a more complete discussion of them.

Your First Singly-Linked List

An Improved List Design

Nested Classes

Your First Singly-Linked List

A singly-linked list consists of nodes linked in one direction. Figure 2.5 shows an example of a singly-linked list. Each node contains two parts: the data held in the node and a pointer to the next node in the list (if any).

Here's how you might define a node structure holding a single character as its data:

struct ChNode {  // Node holding a character

char data;    // Data for the node

ChNode *next; // Next node on the list

};

You'll notice we made this node a simple class, using the struct keyword instead of the class keyword. We've defined no constructors, destructors, or assignment operators. This isn't necessarily bad. For a simple structure such as this, there might not be a reason to hide any details, and the behaviors of the generated default constructor, copy constructor, and assignment functions may be satisfactory. You don't need to use all the C++ features just because they are there. We'll start with the simple ChNode definition and then dress it up later (after we've shown why you might want to do so).

The following code shows how we might construct the linked list in Figure 2.5 with ChNode objects:

main()

{

ChNode a, b, c;

a.data = 'a'; a.next = &b;

b.data = 'b'; b.next = &c;

c.data = 'c'; c.next = 0;

return 0;

}

In this code, we set up the links between each node by hand, using the address of the next node. The last node gets a null pointer to end the list.

Constructing lists like this can be tedious and error prone. It's also easy to lose track of which node heads the list. (In our list, node a is the head.) To solve these problems, a new class, ChList, can be introduced to handle the overall management of a list. This new class holds a pointer to the head of the list and adds nodes to the list. In essence, the ChNode and ChList classes work

Figure 2.5 A singly-linked list.

together to form the actual list structure. Such classes are called cooperating classes.

Here's a simple example of a list class that allows nodes to be added to the front of the list:

class ChList {

private:

ChNode *head;

public:

ChList();

void Add(char c);

};

ChList::ChList()

// Default constructor creates an empty list

{

head = 0;

}

void ChList::Add(char c)

// Add new node to the front, holding c as its data

{

ChNode *p = new ChNode;

p->data = c;

p->next = head;

head = p; // New node now head of list

}

Here's how we can rewrite our list construction example. Figure 2.6 illustrates the new way of representing the list.

main()

{

ChList mylist:

mylist.Add('c');

mylist.Add('b');

mylist.Add('a');

return 0;

}

The file ch2_2.cpp on disk shows a complete running program of the simple node and list classes.

Note how we build the list backwards because we can only add nodes to the front of the list. However, we could certainly define a function for adding nodes to the tail of the list (although it's harder). In fact, our list class is quite incomplete, because we have no way of deleting from the list, no way to walk through the contents of the list, and so forth. But before we fill out the list class, let's first see how we can improve the node class.

Figure 2.6 Alternate implementation of a singly-linked list.

An Improved List Design

Here's a new ChNode class that embodies all five attributes of a well-designed class:

// More complete node class

class ChNode {

friend ChList; // Let ChList access hidden parts

private:

ChNode *next;  // Hide this "detail"

public:

char data; // Keep this available to user

ChNode();

ChNode(const char &d, ChNode *n);

ChNode(const ChNode &cn);

ChNode &operator=(const ChNode &cn);

};

ChNode::ChNode()

// Default constructor

{

data = 0; next = 0;

}

ChNode::ChNode(const char &d, ChNode *n)

// General constructor

{

data = a; next = n;

}

ChNode::ChNode(const ChNode &cn)

// Copy constructor. Note that we don't copy the

// next pointer, but rather set it to zero.

{

data = cn.data; next = 0;

}

ChNode &ChNode::operator=(const ChNode &cn)

// Assignment. Only data is copied; the next

// pointer is set to zero.

{

data = cn.data; next = 0;

}

In our new ChNode class, we've hidden the next pointer, reasoning that the less we directly use pointers like this, the better. However, something must be able to access the next pointer, so we've made all of the ChList class a friend of ChNode, since ChList will be managing the nodes. This is a typical arrangement for cooperating classes. At least one of the classes is a friend of the other. Sometimes you may have a mutual friendship.

In contrast to the next pointer, we left data public, thinking that it might be useful for those who use ChNode to access the data directly. Using this variable directly is relatively harmless. We could, however, make it private and force it to be accessed through special access functions. Here's an example:

class ChNode {

private:

char data;

...

public:

void SetData(char c);

char GetData();

...

};

Would this be worthwhile? Probably not. Whenever you start coding functions like SetData( ) and GetData( ), you should stop and consider what you are doing. Using access functions in this manner isn't much different than using the data directly; and adding the functions just makes the class more complicated than it needs to be.

This isn't to say that all access functions are bad, but that perhaps you should think about how you want your design to be used at a high level. You need to ask yourself how the node objects are to be used. Here, since they are to be handled by the ChList class, it might be better for that class to handle the node data. For instance, we might design ChList to allow a user to add data to the list, without the user needing to know that list nodes even exist. Instead, we just pass ChList the data to be added, and let the list pass back any data that needs to be retrieved. This abstracts the workings of the data structure in a beneficial way. The best bet, then, is to make the data private, but leave out the access functions. Let ChList's friendly access directly manipulate the data.

The copy constructor and overloaded assignment operator functions point out another interesting design issue. What does it mean to copy a list node? If we hadn't methodically included the copy and assignment functions in the node class, we might not have ever asked that question. This proves it's always a good idea to include these two functions in your classes, because you are then forced to think more about your design.

It seems likely that we would want the node data copied, but what about the next pointer? If we allow this pointer to be copied, aliasing problems could occur. It seems best to just set the next pointer to null in the node copy. That's what our copy constructor and overloaded assignment operator do.

Actually, we might want to go further than that. We might even want to disallow nodes from being copied. We can do this by placing both the copy constructor and overloaded assignment operator in the private section of the class, as we've done here:

class ChNode {

friend ChList; // Let ChList access hidden parts

private:

ChNode *next;  // Hide this "detail"

char data;

ChNode(const ChNode &cn); // Disallow copying

ChNode &operator=(const ChNode &cn); // Disallow assignment

public:

ChNode();

ChNode(const char &d, ChNode *n);

};

Statements like the following would now be illegal:

ChNode a, b;

ChNode c(a); // Illegal

b = a; // Illegal

We've now put almost all of our members in the private section of the ChNode class. Why not go the full mile and make the entire class private? There are actually some valid reasons for doing so. Recall that the user of a list doesn't really need to know that list nodes exist. We can reinforce this by making the ChNode class completely private. That way, only the ChList class can use it.

The ChNode class is given on disk in the files chlist.h and chlist.cpp.

Passing Parameters Using Constant References

You'll see in the ChNode( ) general constructor that we pass in characters using a constant reference, as in

ChNode::ChNode(const char &d, ChNode *n);

This isn't the most efficient way to pass a character, since a character is generally smaller than a reference (the latter being implemented usually as a pointer). However, the use of constant reference syntax has benefits, because it keeps the code consistent with more general cases where we might be passing much larger structures than a character. Passing parameters using constant references is a good habit to develop, even when it may not be completely warranted from an efficiency standpoint. You'll notice that we use the syntax const X &x quite often.

It's also a good idea to use a constant reference rather than a plain reference. Besides preventing the object being passed in from being accidently changed, the use of a constant reference makes it easier to pass literals. For instance, we can call the ChNode constructor as follows:

ChNode mynode('x', 0);

Had we declared the constructor without the const modifier,

ChNode::ChNode(char &d, ChNode *n);

then the preceding constructor call would have caused a temporary copy of 'x' to be made. That prevents the literal 'x' from accidentally being changed to something else inside the function. Although making a copy of the character isn't really a problem, some compilers print warnings about it, which can be quite a nuisance. (It can also be a blessing in other cases.) Using const char &d solves the whole problem.

We're now ready to complete the design of ChList, making it more robust. Here is the new definition:

class ChList     {

protected:

int out_of_memory;

ChNode *head;

void Copy(const ChList &c);

public:

ChList();

ChList(const ChList &c);

~ChList();

ChList &operator=(const ChList &c);

int AddtoFront(const char &c);

int RmvHead(char &c);

void Clear();

int IsEmpty();

int IsFull();

};

We've added all the proper constructors, an overloaded assignment operator, and a destructor. We've also added a function to copy one list into another, as well as functions to clear the list by removing and freeing all nodes in the list. Also included are functions to determine whether the list is empty or whether it's full. For the latter, we say a list is full if the last AddtoFront( ) operation failed because we couldn't allocate enough memory for a list node. The out_of_memory flag keeps track of this. Here are some other functions of the ChList class:

void ChList::Copy(const ChList &s)

// Copies the list s into this list.

// Clears the contents of this list first.

// ASSUMES the default constructor sets the

// next pointer to zero. May set out_of_memory

// flag if couldn't allocate a node.

{

Clear();

ChNode *sn = s.head;

if (sn)  { // Anything to copy?

ChNode *tn = new ChNode;

head = tn;

while(sn && tn) {

tn->data = sn->data;

sn = sn->next;

if (sn) {

tn->next = new ChNode;

tn = tn->next;

}

}

if (!tn) out_of_memory = 1;

}

}

ChList::ChList(const ChList &c)

// Copy constructor

{

head = 0; out_of_memory = 0;

Copy(c);

}

ChList::~ChList()

// Destructor removes all nodes from the list

{

Clear();

}

ChList &ChList::operator=(const ChList &c)

// Assignment operator. Clears contents of existing

// list, and makes a copy of list c.

{

Copy(c);

return *this;

}

int ChList::AddtoFront(const char &c)

// Adds a node to the front of the list,

// making a new head. Returns 1 if successful;

// otherwise, sets out_of_memory flag, and returns 0.

{

ChNode *t = new ChNode(c, head);

if (t) {

head = t;

return 1;

}

else {

out_of_memory = 1;

return 0;

}

}

int ChList::RmvHead(char &c)

// Removes the head from the list, and set

// next node as the head. Returns 1 if there

// was a head to remove (i.e. list not empty),

// and clears memory full flag as well.

// Otherwise, a 0 is returned.

{

if (head) {

c = head->data;

ChNode *p = head;

head = head->next;

delete p;

out_of_memory = 0;

return 1;

}

else return 0;

}

void ChList::Clear()

// Removes all nodes from the list

{

char dmy;

int r;

do {

r = RmvHead(dmy);

} while(r);

}

The ChList class is given on disk in the files chlist.h and chlist.cpp.

Nested Classes

Another way to "hide" the ChNode class, besides making all of its members private, is to simply include its class definition inside the ChList class. For example:

class ChList {

protected:

class ChNode { // Hide the class by nesting it

private:

ChNode(const ChNode &cn);

ChNode &operator=(const ChNode &cn);

public:

ChNode *next;

char data;

ChNode();

ChNode(const char &d, ChNode *n);

};

int out_of_memory;

ChNode *head;

void Copy(const ChList &c);

public:

...

};

Nested classes work as follows: the nested class definition is hidden and the only way to access a nested class name is by qualifying it with the name of the enclosing class. For instance, here's how the default ChNode destructor is defined outside the class definitions:

ChList::ChNode::ChNode()

// Default constructor

{

data = 0; next = 0;

}

The member function name ChNode( ) has two qualifications: ChList and ChNode, signifying that the function belongs to class ChNode, which belongs to class ChList. Here's another example using the overloaded assignment operator.

ChList::ChNode & ChList::ChNode::operator=(const ChNode & cn)

{

data = cn.data; next = 0;

}

Note that the third use of ChNode (inside the parameter list) isn't qualified. We don't need to make this qualification because the context tells the compiler that we're referring to a ChList::ChNode object.

The nested class definition is affected by the private, protected, and public keywords in the enclosing class. We've made ChNode protected. That means only ChList member functions, or member functions of classes derived from ChList, have access to the name ChNode. Thus, only these functions can create ChNode objects. Had we placed the ChNode class definition in the public section of ChList, we could create ChNode objects on the outside, like this:

ChList::ChNode mynode; // Assumes ChNode has public access

Remember that this only works if ChNode is declared in a public section. If ChNode is declared in a protected or private section, you can't access the name ChNode, even using the ChList qualification.

Nested classes have no special access to the members of the enclosing class nor do members of the enclosing class have any special access to nested classes. It's as though the two classes were defined separately. The distinction is this: the classes are nested, but not objects of the classes. The member functions of ChList can only access the public members of ChNode, and only by way of a ChNode object. There is no direct access, as you might think. Likewise, members of ChNode can only access the public members of ChList, and only through ChList objects.

Note that we made most of the ChNode members public so that ChList could easily access them through an object. However, we kept the copy constructor and overloaded assignment operator private. This way, even ChList member functions cannot accidently copy ChNode objects.

While this alternative method for declaring cooperating classes has its merits, we'll be primarily using the first method (declaring the classes separately) because it is often less confusing.

The alternative design for the ChNode and ChList classes is given on disk in the files chlist2.h and chlist2.cpp.

Go to Chapter 3            Return to Table of Contents


Practical Data Structures: CHAPTER 3: DATA STRUCTURE DESIGN IN C++

 Next Chapter Return to Table of Contents Previous Chapter

CHAPTER 3: DATA STRUCTURE DESIGN IN C++

Chapter 2 provided some overall principles for working with classes in C++. In this chapter, we'll get more specific by discussing some fundamental design issues involved in defining data structures. In particular, we'll discuss concrete data types, abstract data types, and then parameterized types (templates). Finally, we'll provide a discussion on using containers and single-rooted hierarchies.

This chapter should help explain why the data structures you'll see in later chapters are designed the way they are. We advocate a direct approach to data structure design, with concrete data types and templates playing a major role.

CONCRETE DATA TYPES

As you implement different data structures, you'll probably begin to build your own library containing the most effective and elegant designs. There are two fundamental ways you can go about this:

Design the tools using concrete data types

Design the tools using abstract data types

Concrete data types represent direct implementations of a concept, usually designed for specific purposes. For instance, the list and node classes used in Chapter 2 are concrete, since they give a specific implementation for linked lists and only work with character nodes. In contrast, abstract data types provide a general-purpose design; the idea is to allow the user to program at a fairly high level, with all the low-level details kept hidden. Abstract data types have concrete data types somewhere in their implementations. Typically, an abstract data type might provide two or more concrete data types as alternative implementations that can be interchanged freely, without disturbing too much of an existing application.

To see an example of a concrete data type, take a look at this very simple, and very hard-coded, character stack:

class ChStack {

private:

char data[32];  // Room for 32 characters

int top;        // Index to top of stack

public:

ChStack( )                { top = 0; }

void Push(const char &c)  { data[top++] = c; }

void Pop(char &c)         { c = data[-top]; }

};

In many ways, classes like ChStack embody the best aspects of C++. The ChStack class is very efficient, almost and possibly as efficient as a more low-level design would be, yet encapsulation is used effectively. You can only access the contents of the stack through the conventional push and pop stack operations, and you can easily create multiple stacks without worrying about the stack variables interacting with each other. If you use this class in an application, and then later decide to use a more sophisticated implementation (perhaps one that does error checking), it would be simple to make the replacement. The code for the main part of the application might not even have to be changed.

Rational and Compound Numbers

Rational and Compound Numbers

The built-in types--such as int, float, and double--are concrete data types. It's also possible to make user-defined concrete data types that look like built-in types. This can be accomplished by judicious use of constructors, destructors, and overloaded operators. Many C++ implementations provide a complex number class along these lines. You can use the arithmetic and comparison operators on the complex numbers just as you would for any int or float. In this section, we'll take a similar approach and sketch out the implementation for rational and compound numbers.

A rational number is a representation for fractions. Each rational number has a numerator and a denominator. You can extend the notion of rational numbers to include a whole number part as well. A compound number contains both a whole number part and a fractional part (that is, a rational number). The following code highlights the main features of a Rational class and a Compound class used to implement rational and compound numbers:

class Rational{

private:

long n, d; // Numerator and denominator

public:

// Constructors and assignment

Rational( );

Rational(long u);

Rational(long u, long v);

Rational(const Rational &r);

Rational &operator=(const Rational &r);

// Stream I/0 operators

friend istream &operator>>(istream &s, Rational &r);

friend ostream &operator<<(ostream &s, const Rational &r);

// Arithmetic operators that modify their operand

Rational operator++(int);   // Postfix

Rational operator-(int);    // Postfix

Rational &operator++( );    // Prefix

Rational &operator-( );     // Prefix

Rational &operator+=(const Rational &r);

Rational &operator-=(const Rational &r);

Rational &operator*=(const Rational &r);

Rational &operator/=(const Rational &r);

// Arithmetic operators resulting in new object

Rational operator-( ) const;

Rational operator+( ) const;

friend Rational operator*(const Rational &a, const Rational &b);

friend Rational operator/(const Rational &a, const Rational &b);

friend Rational operator+(const Rational &a, const Rational &b);

friend Rational operator-(const Rational &a, const Rational &b);

// Comparison operators

friend int operator==(const Rational &a, const Rational &b);

friend int operator!=(const Rational &a, const Rational &b);

friend int operator<(const Rational &a, const Rational &b);

friend int operator<=(const Rational &a, const Rational &b);

friend int operator>(const Rational &a, const Rational &b);

friend int operator>=(const Rational &a, const Rational &b);

// Other helper functions

...

};

class Compound {

private:

long w;     // Whole number part

Rational f; // Fractional part

public:

// Member functions, similar to those for the Rational class

...

};

Besides having the usual assortment of constructors, a destructor, and an overloaded assignment operator, the Rational and Compound classes implement most of the arithmetic operators, as well as the comparison operators. The idea is to make rational and compound numbers work just like ordinary integers or floats work for standard arithmetic and comparison operations.

The details of the Rational and Compound classes are beyond the scope of this book. See the code note below for more information. Our main goal here is to show that classes like Rational and Compound are good examples of concrete data types.

The code for the Rational class, along with a test program, can be found on disk in the files rational.h, rational. cpp, and rattst.cpp. The code for the Compound class, along with a test program, can be found in the files cmpnd.h, cmpnd.cpp, and cmpndtst.cpp.

 ABSTRACT DATA TYPES

Abstract data types are the opposite of concrete data types, but are just as useful. The idea is to avoid being committed to a specific implementation. For instance, a stack can be implemented using an array or by using a linked list. At a high level, it doesn't matter which of these structures you use. What matters is that the operations push and pop are defined.

Abstract data types can be realized in C++ by using a class derivation hierarchy. The base class of the hierarchy spells out the operations to be performed on the abstract data type, providing as few details as possible. The derived classes then provide more concrete implementations. It's often advantageous to make the base class an abstract class, which has one or more pure virtual functions. A pure virtual function is a virtual function that does not have a body. Instead, the body is set to null. For example, here is an abstract character stack class:

class AbsChStk { // An abstract character stack class

private:

// Prevent copy constructing and assignment

AbsChStk(const AbsChStk &) { }

void operator=(const AbsChStk &) { }

public:

AbsChStk( ) { }

virtual ~AbsChStk( ) { }

virtual int Push(const chart &c) = 0;

virtual int Pop(char &c) =0;

virtual int IsEmpty( ) = 0;

virtual int IsFull( ) = 0;

};

The functions Push( ), Pop( ), IsEmpty( ), and IsFull( ) are all pure virtual functions. Since they have no bodies, it's not possible to actually call these functions (without getting runtime errors). Instead, the idea is to derive one or more classes that override these functions. The consequence is that you can't directly declare an object of an abstract class. You must only work with derived class objects. However, you can use abstract class pointers (or references), which point to derived class objects. For example:

class ChArrStk : public AbsChStk { }   // Derive array version

class ChListStk : public AbsChStk { }  // Derive list version

void Doit(AbsChStk *pstk)

{

pstk->Push('x');                     // Calls virtual function

}

ChArrStk my_arr_stk;

ChListStk my_list_stk;

...

// Choose stack implementation at runtime:

Doit(&my_arr_stk);                     // Use array version

...

Doit(&my_list_stk);                    // Use list version

Code that uses abstract class pointers in this manner can be quite versatile.

Even though you can't directly declare abstract class objects, it's possible to cause their copy constructors or overloaded assignment operators to be invoked. Consider the following code:

AbsChStk *p, *q;

p = q;   // AbsChStk::operator=( ) called

Normally you wouldn't do this, but it may happen by accident. The way to prevent accidental assignments like this is to define an overloaded assignment operator and make it private, as we did earlier for the ChNode class. We use this technique on the AbsChStk class for both the assignment operator and the copy constructor.

Once you have an abstract class defined, you can derive concrete implemenations in two ways: using single inheritance or using multiple inheritance. With single inheritance, the derived class has only one direct parent. With multiple inheritance, the derived class may have two or more parents. Figure 3.1 illustrates these techniques within the context of creating stack classes.

Figure 3.1 Stack classes created with single versus multiple inheritance.

(a) Single Inheritance

(b) Multiple Inheritance

Using Single Inheritance

Let's look first at using the single inheritance technique for deriving classes. Here's how we might derive an array-based stack from the abstract stack class:

class ChArrStk : public AbsChStk {

private:

int sz, top;  // Dimensioned size and top of stack

char *data;   // Stack data

void Copy(const ChArrStk &s);

public:

ChArrStk( );

ChArrStk(const ChArrStk &s);

ChArrStk(int n);

virtual ~ChArrStk( );

ChArrStk &operator=(const ChArrStk &s);

virtual int Push(const char &c);

virtual int Pop(char &c);

virtual int IsEmpty( );

virtual int IsFull( );

};

The storage for the stack is allocated dynamically to hold up to n elements using a constructor and de-allocated using the destructor:

ChArrStk: ChArrStk(int n)

{

data = new char[sz = n] ;

top = 0;

}

ChArrStk::~ChArrStk( )

{

delete[] data ;

}

Again, we provide a full complement of constructors and an overloaded assignment, even though we may never need to use them. They provide us with an element of safety. Both the copy constructor and assignment operator (not shown), work by copying the stack data, rather than by copying just the pointer to the data.

The interesting part is how the pure virtual functions are overloaded in ChArrStk to implement array-based stack operations:

int ChArrStk :: Push(const char &c)

// Pushes c onto the stack, if it can. Returns

// 1 if successful. Returns 0 if stack was

// full coming in.

{

if (!IsFull( )) {

data[top++] = c ;

return 1 ;

}

else return 0 ;

}

int ChArrStk::Pop(char &c)

// Pops element from the stack, and returns

// its value in c. Returns 1 if stack wasn't

// empty before pop; otherwise, returns 0.

{

if (!IsEmpty()) {

c = data[-top ;

return 1;

}

else return 0;

}

int ChArrStk::IsEmpty( ) { return top == 0; }

int ChArrStk: IsFull( )  { return top == sz; }

Both Push( ) and Pop( ) check for the stack being full and empty, respectively, before allowing the operations to occur. The IsEmpty( ) and IsFull( ) functions check the current top of the stack to see if it is in bounds, given the size of the array holding the stack data.

How efficient is the derived stack compared to the concrete stack we gave earlier? The derived stack class will be slower, due to the overhead caused by the indirect virtual function calls. In addition to this performance hit, the use of virtual functions also reduces the chances that the functions can be made inline. In our concrete stack class, Push( ) and Pop( ) could be inlined, resulting in significant performance gains. Whether this matters depends on how often Push( ) and Pop( ) are called and how tight a loop they might be in. For many applications, the virtual function overhead may not be noticeable.

Next, let's derive a list-based stack class:

class ChListStk : public AbsChStk {

protected:

int out_of_memory;

ChNode *head ;

void Copy(const ChListStk &s);

public:

ChListStk( );

ChListStk(const ChListStk &s);

virtual ~ChListStk( ) ;

ChListStk &operator=(const ChListStk &s);

void Clear( );

virtual int Push(const char &c);

virtual int Pop(char &c);

virtual int IsEmpty( );

virtual int IsFull( );

};

This class is very similar to the ChList class we provided in Chapter 2. It even uses the same cooperating ChNode class to define the nodes of the stack. The main difference is that the list operations are defined in terms of stack operations. The Push( ) and Pop( ) routines are the same as the AddToFront( ) and RmvFront( ) ChList routines. For example, here's the Push( ) routine:

int ChListStk::Push(const char &c)

// Pushes c onto the stack, if it can. Returns

// 1 if successful; returns 0 if not.

{

ChNode *t = new ChNode(c, head);

if (t) {

head = t;

return 1;

}

else {

out_of_memory = 1;

return 0;

}

}

The code for the array and list-based stacks, using single inheritance, is given on disk in the files stk1.cpp and stk2.cpp, respectively. The abstract stack class is given in the file abschstk.h.

Using Multiple Inheritance

In the previous section, we derived a list-based stack class from scratch. But you might have wondered why we didn't use the existing ChList class. In fact, we can do this by using multiple inheritance. This technique is one of the main uses of multiple inheritance. The basic idea is to merge one class with another, creating a new type of class. It's like doing genetic cross breeding. For instance, we can merge AbsChStk with ChList to create a new ChListStk class:

class ChListStk : public AbsChStk, private ChList {

private:

void Copy(const ChListStk &s);

public:

ChListStk( );

ChListStk(const ChListStk &s);

virtual ~ChListStk( );

ChListStk &operator=(const ChListStk &s);

virtual int Push(const char &c);

virtual int Pop(char &c);

virtual int IsEmpty( );

virtual int IsFull( );

};

The trick is to override the virtual functions of the abstract base class and make calls to the list class to perform the operations. For example, here's how Push( ), Pop( ), IsEmpty( ), and IsFull( ) are defined:

int ChListStk::Push(const char &c)

{

return AddtoFront(c);    // Calls ChList::AddToFront()

}

int ChListStk::Pop (char &c)

{

return RmvHead(c);       // Calls ChList::RmvHead()

}

int ChListStk::IsEmpty( ) { return ChList::IsEmpty( ); }

int ChListStk::IsFull( )  { return ChList::IsFull( ); }

Notice how we had to qualify the IsEmpty( )and IsFull( ) function calls to avoid ambiguities. That's because both AbsChStk and ChList have IsEmpty( ) and IsFull( )functions. Resolving ambiguities like this is one reason why multiple inheritance can be tricky to use.

One subtle feature of our derived class is that we used private inheritance for the ChList base class. By doing so, we disallow the user of the stack class from directly calling the list-oriented functions (such as AddToFront( )). We also disallow stack objects from being used as lists. We want to use the stack in terms of stack operations, not list operations.

How well does the multiple inheritance approach perform? Due to the multiple base classes, there is more overhead in virtual function calling. As a result, the runtime performance may be slightly slower than it would be for single inheritance. The class will probably also consume more space. However, these costs may be small prices to pay for the ability to elegantly piece together new classes from existing ones.

A complete running program using a list-based, multiply-derived stack class is given on disk in the file stk4.cpp. Also, a corresponding array-based version is given in stk3.cpp, and in the files charr.h and charr.cpp.

 PARAMETERIZED TYPES (TEMPLATES)

The AbsChStk class that we created in the previous section is fairly general, but not as general as it could be. AbsChStk only works for stacks that store characters. But what if we want to store integers, or floats, or window objects? Ideally, we want the type of element that gets stored to be a parameter to the basic stack type itself. We can do this through the use of templates, which are C++'s way to implement parameterized types. Concrete data types and abstract data types can both be made into templates.

A Parameterized Stack Class

Let's return to our first concrete stack class and parameterize it so that we can define it for different types of elements. Here is the original definition:

class ChStack {

private:

char data[32];

int top;

public:

ChStack( )                { top = 0; }

void Push(const char &c)  { data[top++] = c; }

void Pop(char &c)         { c = data[-top]; }

}:

Note how we explicitly define the stack data to be of type char. We can make this definition more general by using a typedef to define the stack data type. While we're at it, we'll use a constant to define the size of the stack, and change the class name to Stack:

typedef char TYPE;

const int SIZE = 32;

class Stack {

private:

TYPE data [SIZE];

int top;

public:

Stack( )                  { top = 0; }

void Push(const TYPE &c)  { data[top++] = c; }

void Pop(TYPE &c)         { c = data[-top]; }

};

Now, by changing the definition of TYPE and SIZE, we can easily define the stack to work with any type, and be any size we desire. Unfortunately, we can only define TYPE and SIZE once, which means we can't easily define more than one stack class. For example, we can't define a stack of windows as well as a stack of characters. Nor can we have a stack of 10 characters in conjunction with a stack of 100 characters.

The solution is to use C++ templates. Here is the stack class rewritten using templates:

template<class TYPE, int SIZE>

class Stack {

private:

TYPE data[SIZE];

int top;

public:

Stack( )                  { top = 0; }

void Push(const TYPE &c)  { data [top++] = c; }

void Pop(&TYPE &c)        { c = data [-top]; }

};

The template<> syntax is placed in front of the class definition. Between the < > brackets are the parameters that you want for the template. You can have two kinds of parameters: those that specify type names and those that specify constant expressions. Type names are identified by using the class keyword. (This use of the class keyword has nothing to do with its other use--to define classes.) You can use any valid type name as a parameter, such as char, float, or ChNode. You can even use another parameterized type! For constant expression parameters, you must specify the type of the expression, otherwise it defaults to int.

With our stack template in hand, we can now define different types of stacks, then declare objects from them. For example:

Stack<char, 32> mystack;

Stack<int, 100> yourstack;

Stack<ChList, 42> exoticstack;

Each of these declarations defines a new stack type and then creates an object of that type. The third declaration defines a stack of linked lists (for some exotic purpose, presumably). Syntax such as Stack<char, 32> is known as a template declaration. You can use template declarations anywhere a type name can be used. For example, a template declaration can be used in a typedef statement:

typedef Stack<ChList, 42> StackOfLists;

StackOfLists exoticstack;

Template declarations actually cause classes to be generated internally by the compiler. Each generated class is called a template instantiation. For example, the compiler might internally generate the following class for the declaration Stack<ChList, 42>. (We've given the generated class an arbitrary but presumably unique name.)

class Stack_ChList_42 {

private:

ChList data[42];

int top;

public:

Stack_ChList_42( )          { top = 0; }

void Push(const ChList &c)  { data[top++] = c; }

void Pop(ChList &c)         { c = data[-top]; }

};

For every template declaration with different parameters, the compiler creates a new instantiation of the template. With templates, we're reusing source code rather than compiled code. This is both the boon and the bane of templates. There is no runtime overhead associated with templates. A class instantiated from a template will be just as efficient as one defined directly. Unfortunately, a classic speed versus space tradeoff occurs, because each new template instantiation increases the program's code space. All of the methods of the class must be replicated (and then modified accordingly). Taken to the extreme, this is known as source code explosion.

With our stack template, the space requirements can become severe. Not only is a new class generated each time we use a new stack element type, but each time we declare a stack of a different size. For instance, the following declarations cause two stack classes to be generated:

typedef Stack<char, 4> QuadStack;

typedef Stack<char, 6> HexStack;

If, however, both parameters were to match in the declarations, only one stack class would be generated (as long as the compiler is smart enough not to generate the class twice). This is true even if we use parameters that appear different but are really the same. For instance, the following code creates two objects but only one stack class:

Stack<char, 4> mystack;

Stack<char, 2+2> yourstack;

One way to alleviate the code explosion in our stack template is to take the size parameter out of the template and place it in the stack constructor instead. This technique changes the allocation of the stack data from static to dynamic, as in the following code:

template<class TYPE>

class Stack {

private:

TYPE *data;

int top;

public:

Stack(int sz) { top = 0; data = new TYPE[sz]; }

~Stack( ) { delete[] data; }

void Push(const TYPE &c)  { data[top++] = c; }

void Pop(TYPE &c)         { c = data[-top]; }

};

The constructor takes care of the allocation, and the destructor de-allocates the data. Compare the way we create objects from this new stack template with the old way:

Stack<char, 42> mystk;       // Using static allocation

Stack<char> yourstk(42);     // Using dynamic allocation

void f(int sz)

{

Stack<char, sz> mystk;     // Can't do; sz isn't constant!

Stack<char> yourstk(sz);   // OK

}

Is it better to statically allocate or dynamically allocate the stack data? The dynamic method is certainly more flexible and leads to less source code explosion. However, it requires the use of a heap. With static allocation, we don't need a heap. This means we could more easily use a statically allocated stack in environments with scarce resources, such as ROM-based process control software.

A sample program using a stack template similar to the one just show is given on disk in the file stktpl.cpp.

Function Templates

You can also define function templates. Actually, we've already done this with the stack class, since the stack methods are all templates. You may not have realized this because the methods were declared inline. To show the template syntax, we'll declare a few of the stack methods outside the class:

template<class TYPE>

Stack<TYPE>::Stack(int sz)

{

top = 0; data = new TYPE[sz];

}

template<class TYPE>

void Stack<TYPE>::Push(const TYPE &c)

{

data[top++] = c;

}

The template< > parameter list is placed in front of function templates, as is the case with class templates. Recall that class methods are qualified with the class type, using the :: scoping operator. For class templates, the class type is parameterized. That's why we use the following syntax to qualify the functions:

Stack<TYPE>::Stack(int sz)

void Stack<TYPE>::Push(const TYPE &c)

Note that we don't use the syntax Stack<TYPE>::Stack<TYPE>(int sz) for the constructor. The function name Stack isn't parameterized, just as the function name Push isn't parameterized. Only the type name is parameterized.

Class methods aren't the only types of functions that can be made into templates. For example, we can parameterize a sort function, such as the following InsertionSort( ) routine. (See the companion volume for an explanation of insertion sorting [Flamig 93].)

template<class TYPE>

void InsertionSort(TYPE *a, int n)

{

int i, j;

TYPE item_to_sort;

for(i = 1; i < n; i++) {

item_to_sort = a[i];

j = i;

while(item_to_sort > a[j-1]) {

a[j] = a[j-1];

j -= i;

if (j == 0) break;

}

a[j] = item_to_sort;

}

}

This sorting template will work for arrays of most types of data. For instance, we could sort an array of compound numbers as follows:

Compound arr[10];

...

InsertionSort(arr, 10);

When the compiler sees this function call, it looks at how the array is defined and determines that the TYPE parameter is Compound. It then generates a sort function specific to compound number arrays. Here's a sketch of what's generated:

void InsertionSort_Compound(Compound *a, int n)

{

...

Compound item_to_sort;

for(...) {

...

while(item_to_sort < a[j-1]) {

...

}

...

}

}

Although the sort template will work for most array types, it does make one assumption. The operator < must be defined for TYPE. Thus, the template would work fine for integer arrays, float arrays, and so on, but you must be careful with arrays of user-defined objects. In our example, we defined < for compound numbers, so we're safe. If < isn't defined for TYPE, a compiler error will be generated.

In general, the types you use as parameters to templates should be fairly robust. At the very least, these types should have well-defined default constructors, copy constructors, and assignment operators. Also, if your function plates involve any arithmetic or comparison operations, you must ensure that the template parameter types have these operations defined as well. In fact, the closer the types are to looking like built-in types (as is the case with compound numbers), the more likely it is that you won't have problems.

Overriding Function Templates

A typical problem occurs when you try to sort character strings with a sort function template. For instance:

char names[10][80]; // An array of 10, 80 byte character strings

InsertionSort(names, 10);

Will this work as expected? No. The problem lies in the compare statement within the sort function:

while(item_to_sort < a[j-1]) {

Since the items being sorted are actually character pointers, we're sorting the values of the pointers instead of the text being pointed to. Thus, we would get back the array sorted by addresses. One solution is to define a class to handle the comparisons. For example:

class String {

private:

char *text;

...

public:

String(char *s);

friend int operator<String &a, String &b);

...

};

String::String(char *s)

{

text = strdup(s);

}

...

int operator<(String &a, String &b)

{

return strcmp(a.text, b.text) < 0;

}

...

String strings[3] = { // Declare array of strings

String("frosty"), String("morning"), String("cold")

};

InsertSort(strings, 3); // Sort the strings

Another method is to override the sort function template itself and provide a specific version just for character strings. You can override a function template for a specific set of template parameters by providing a function that takes the correct parameters. For example, here is a character string insertion sort function:

void InsertionSort(char **a, int n)

{

int i, j;

char *item_to_sort;

for(i = 1; i < n; i++) {

item_to_sort = a[i];

j = i;

while(strcmp(item_to_sort, a[j-1]) > 0) {

a[j] = a[j-1];

j -= 1;

if (j == 0) break;

}

a[j] = item_to_sort;

}

}

Here, we've replaced TYPE with char *, and we've modified the while loop to call the strcmp( ) function. If the compiler sees this function when trying to instantiate the InsertionSort( ) template for char *, it will use the function instead of the template. We can make the compiler use the overriding function by placing its function prototype in the same header file where the template occurs. The exact placement is implementation dependent. (We'll discuss the implementation-dependent details of templates later in this chapter.)

The InsertionSort( ) template is small enough to be easily overriden for different array element types. In general, though, the first method we described (defining the > operator properly through a String class) is probably a better solution. However, overriding template functions can be quite handy, as you'll see in later chapters. Also, there are other ways to handle the comparison operator problems for sort function templates. The book [Stroustrup 91] discusses some additional techniques that you can use.

Restrictions on Template Functions

Although most functions can be turned into templates, there is a restriction: Each parameter of the template must somehow be used in the parameters of the function. Why is this necessary? The compiler can only determine which template instantiation to use for a function call by looking at the function parameter types. For example, in the InsertionSort( ) function, the array parameter's type determines the template parameter TYPE. Here are some examples of illegal function templates:

template<class TYPE> void f(int v);            // Illegal: TYPE not used

template<class TYPE, int SZ> void g(TYPE *a);  // Illegal: SZ not used

This restriction isn't as noticeable for member functions. You don't need to explicitly use the template parameters here. Why? Each member function has a hidden this pointer passed as an argument, and this argument will have all of the template parameters as part of its type specification. For instance, the stack constructor is actually represented internally as something like:

Stack_TYPE(Stack<TYPE> *this, int sz);

Combining Templates and Inheritance

Since templates can cause source code explosion, you may wonder if they are really all that useful. If you were to use the InsertionSort( ) template for 20 different array types, then 20 different sort routines would be generated. A more general sort routine, built to use abstract class pointers and virtual functions, might be better in this case. But how often would you actually use a sort template for 20 different types, in any given application? Probably not as often as you might think. If the number of instantiations is small, and the templates in question are small, the total code space for the templates might actually be smaller than the space required for a general-purpose, inheritance-oriented version. In any case, the templates will most likely be faster.

You can sometimes get the best of both worlds by combining templates with inheritance. The idea is to create a base class that does most of the work, and then derive class templates that work for specific types. For example, consider the following RecFile class, which is designed to make it easy to do record-oriented I/O with file stream objects:

class RecFile {

protected:

enum io_dir { f_read, f_write };

fstream fs;     // Embedded fstream object

int recsize;    // Record size in bytes

long curr_rec;  // Current record (count by zero)

char name[80];  // Name of the file

int IO(io_dir dir, void *data, int nrecs, long recno=-1);

public:

RecFile(int rsize = 1); // Default to character I/O

~RecFile( );

virtual int Open(char *fname, int omode);

virtual int Close(void);

int IsOpen( );

int State( );

long Seek(long recno, io_dir which, ios::seek_dir mode=ios::beg);

};

With this class, the record size is stored in recsize, which is then used by the Seek( ) and IO( ) routines to compute the actual byte offset of a record. The trouble is that to read and write records, IO( ) uses a void pointer to pass the record data. Obviously, this isn't very type-safe, so we've made IO( ) a protected member. To actually use the RecFile class, you must derive classes that are type specific. The following FileOf derived class template shows how:

template<class TYPE>

class FileOf : public RecFile {

public:

FileOf( ) : RecFile(sizeof(TYPE)) { }

int Read(TYPE &data, long recno = -1) {

return IO(f_read, &data, 1, recno);

}

int Write(TYPE &data, long recno = -1) {

return IO(f_write, &data, 1, recno);

}

};

Note how sizeof(TYPE) is passed to the base class RecFile( ) constructor to determine the record size, and how Read( ) and Write( ) cause the necessary type conversions to be performed when calling IO( ). Here's a sketch of how you might use FileOf:

FileOf<Compound> myfile; // Declare file of compound number records

myfile.Open("test.fil", ios::in | ios::out | ios::trunc);

Compound x;

. . .

myfile.Write(x, 17);  // Write x to record 17

myfile.Read(x, 1);    // Read record 1 into x

By using FileOf, we can implement an entire family of record-oriented file classes, one for each type of record, as shown in Figure 3.2. FileOf is especially useful because it is type-safe, but because it uses inline functions exclusively it may not cause any extra code to be generated. Classes such as FileOf are often called interface classes, and they show once again that C++ can be both efficient and high level.

Figure 3.2 A template-based family of classes.

A complete running program using the RecFile and FileOf classes can be found on disk in the files recfile.h, recfile.cpp, and rftst.cpp.

Compiler-Dependent Template Issues

As of this writing, the implementation of templates has not been completely standardized for the C++ language. The most significant problem lies in trying to keep all the template instantiations straight. There are three main difficulties:

       Like inline functions, the template definitions must be available when a template declaration is seen, so that the compiler can expand the template and generate the specific code.

       It's easy to produce multiple template instantiations, which can cause linker errors.

       Some compilers aren't able to automatically instantiate the correct templates in complex cases (such as when the templates are part of a class derivation hierarchy.)

To solve the first two problems, the ideal solution is to have the compiler automatically determine when a template is first being instantiated for a given set of parameters, and to prevent further instantiations. Currently, there is no standard way of doing this. However, two basic approaches have emerged:

1.     Include the complete template definition in a header file--methods and all. That way, wherever the template is used, the compiler has all the information about the template at its disposal. Since header files are often included in more than one source file, this can lead to multiple definition errors. Some compilers have smart linkers that detect these cases and throw out all but one of the instantiations.

2.     Include only the class template definitions (but not the non-inline functions) in the header files. For non-inline function templates, include only the function prototypes. In one or more special source files, define the methods (and function bodies) for the templates, and then instantiate them for all sets of template parameters to be used by the application. Then link these source files into the application.

In this book, we will use the first approach (including all the template definitions in the header file). We do this not because it's better than the second approach, but because it's easier. Keep this in mind, and don't be afraid to change the code on the disk to suit your needs. To make it easier to switch between the two approaches, we will use the following strategy: only the class template definitions and function template prototypes will be given directly in the header files. The class template methods will be placed in special files with .mth extensions. (Here, mth stands for methods.) These .mth files will then be included in the header files via #include statements. Figure 3.3 illustrates this idea. You can easily take the #include statements out of the header files and rearrange the template code any way you wish.

The other problem we mentioned--(the inability of the compilers to automatically instantiate the correct template) may occur in some C++ compilers with early implementations of templates. (Borland C++ 3.0 and 3.1 are examples.) In some cases, you'll get linker errors saying that a function cannot be found, even though there is a template for it. In cases like this, the best strategy is to go ahead and declare a template function prototype with the correct parameters. Here is an example that uses the Array templates from Chapter 4 :

void PrtArray(DVarray<int> &a); // Declare template function prototype

This is often enough to help out the compilers. See the code on disk for Chapter 4 for numerous examples of this.

Figure 3.3 File organization for templates used in this book.

Getting Along  without Templates

In case your compiler doesn't support templates, we'll now show you how to turn the template code in this book into non-template code. The technique involves using a good editor that has search and replace capabilities. We'll use the constructor for the stack template as an example and turn it into a constructor that's specific for compound numbers. Similar techniques can be used for normal function templates, as well as complete template classes. Here is the original definition of the stack template constructor:

template<class TYPE>

Stack<TYPE>::Stack(int sz)

{

top = 0; data = new TYPE[sz];

}

Follow these basic conversion steps:

1.     Make a copy of the template to be modified. Then, remove all occurrences of the template prefix (for instance, template<Parm1, Parm2, . . .>) from this copy. The code provided in this book always keeps the prefix on a separate line, making this easy to do: Just delete the line that the prefix is on.

2.     Remove all occurrences of <Parm1, Parm2, . . .> by using search and replace. (That is, replace the string with nothing.) For example:

Stack<TYPE>::Stack ===> Stack::Stack

3.     Replace each occurrence of the template parameter names with the specific names you would like to use. For instance, we'll replace every occurrence of TYPE with Compound. Now our stack template looks like the following:

Stack::Stack(int sz)

{

top = 0; data = new Compound[sz];

}

4.     Replace all occurrences of the class name with a unique name. You can do this by using the template parameters in the name. For example, you could add the suffix _Compound:

Stack_Compound::Stack_Compound(int sz)

{

top = 0; data = new Compound[sz];

}

Note that this search and replace process is fairly easy for simple templates, but when you start working with complicated template schemes (such as those that have multiple parameters or that mix in inheritance), it can get tricky.

It's possible to define templates using macros. In fact, many early C++ implementations had a generic.h header that provided a standard way to define templates with macros. However, this approach involves using multiple-line macros, a tedious and extremely error-prone technique. Your best bet is to find a compiler that directly supports templates.

 CONTAINERS

Data structures such as arrays and linked lists are often called containers, because they are used to store, or contain, other objects. Stacks and trees are also examples of containers. Because containers are so widely used, it's worthwhile to look at some issues that are common to all containers. There are two fundamental issues:

       Can you have different types of data in the container?

       Who owns the data in the container?

Homogenous versus Heterogenous Containers

A container can hold objects that are all the same type, or it can hold objects of different types. The former is called a homogenous container, the latter is called a heterogenous container.

An array is an example of a homogenous container, since by definition it is a sequence of objects of the same type. Strictly speaking, all containers in C++--such as linked lists, trees, and so forth--are homogenous. That's because C++ is a statically typed language. For instance, list nodes can only store one type of data, as in the following character list node:

struct ChNode {

char data; // Node data is a character

ChNode *next;

};

If this is true, how can we get heterogenous containers? The answer for C++ is that we can't--at least not exactly. However, we can emulate hetorogenous containers by storing pointers to objects in the containers, rather than the objects themselves. We could make our list node general purpose by changing the char type to a void pointer, as in:

struct GNode {

void *data; // Points to data allocated elsewhere

ChNode *next;

};

The same idea can be used with arrays:

void *general_array[42];

By using void pointers, we can use any type of data with our containers. While this is certainly the ultimate in flexibility, the downside is that we must be very careful that an object taken out of the container is given the same type it had when it went into the container. Safe type casting becomes a major concern. For example:

class Container { ... }; // Some type of container class

int i;

Compound c;

Container w;

w.PutAt(1, &i); // Store objects in container

w.PutAt(2, &c);

int j = *((int *)w.GetAt(2)); // Wrong type casts!

Compound d = *((Compound *)w.GetAt(1);

One way to tame unruly containers that store object pointers is to allow only certain types of pointers to be stored. For example, we can use base class pointers rather than void pointers. Because of the special relationship between base and derived classes, the base class pointers can point not only to base class objects but also to objects of any classes derived from the base. It can't, however, point to other types of objects (without unscrupulous typecasts, that is). It's actually quite useful and intuitive to restrict the type of object pointers being stored to those that belong to some hierarchy.

A perfect example of this approach is a stack of window objects. We might have different types of windows--such as dialog boxes, menus, and browsing windows--all derived from the same window root class. Figure 3.4 illustrates this setup, where we've used the class names Window, Dialog, Menu, and Browser. A window stack is used to define the overlapping order of the windows as they appear on screen. The stack might be defined and used as follows:

class WindowStack {

private:

Window *contents[42]; // Allows 42 window object pointers

...

public:

...

int Push(const Window *w);

int Pop(const Window *&w);

};

WindowStack wstk;

Window *top;

Dialog d;

Browser b;

wstk.Push(&d);

wstk.Push(&b);

wstk.Pop(top);

top->Remove( ); // Call virtual Remove( ) function

wstk.Pop(top);

((Browser *)top)->LoadFile( ); // OOPS! Wrong cast

Figure 3.4 A window class hierarchy.

The last statement shows that, even with this setup, we don't eliminate type casting problems. Because generic Windows pointers are used, when we pop the current window off the stack, we won't know what type of window we just obtained. If we guess wrong, the results could be catastrophic. We'll have more to say about this later in the chapter.

A Question of Ownership

A problem immediately surfaces whenever we store an object in a container indirectly, through a pointer: Who owns the object? Who is responsible for making sure the object gets destroyed when it's no longer needed? Should it be the container, or the entity that initially created the object? Aliasing problems abound because we're using pointers to the object. It's safer to simply store a copy of the object. However, by storing a copy, we may lose the flexibility of storing different types of data. And if the objects are large, we may waste a lot of space. Here, the tradeoff is one of space versus flexibility.

Containers that share objects with other entities are said to have share semantics. Containers that store copies of objects (even if they're the only copies) are said to have copy semantics. Figure 3.5 illustrates this difference.

Containers with share semantics always use pointers or references to objects. Containers with copy semantics typically do not. But it's interesting to note that containers with copy semantics can use pointers. Consider the following code:

// An array of pointers with copy semantics

int *a[2];

a[0] = new int(42);  // Only a[0] has access to new object

delete a[0];         // Only a[0] destroys the object

a[1] = new int(17);  // Only a[1] has access to new object

delete a[1];         // Only a[1] destroys the object

As long as each element of the array is the only entity using the object being pointed to, no sharing occurs. Thus, the array a can certainly be said to have copy semantics. You won't normally see pointer arrays with copy semantics used directly like this, but it can occur quite often in classes. Consider the following WindowManager class, which handles the creation and deletion of the objects it stores.

class WindowMgr {

private:

Stack<Window *> wstk;

public:

...

CreateDialog(int w, int h) { wstk.Push(new Dialog(w, h)); }

CreateBrowser(int w, int h) { wstk.Push(new Browser(w, h)); }

RmvWindow(int id) { delete wstk[id]; FixupStack(); }

};

Figure 3.5 Copy and share semantics.

Sometimes, you'll want a container class to manage the objects for you. Sometimes you won't. To build in this flexibility, you might store a flag in the container. The flag indicates who is to own the objects: the container or something outside the container. This "ownership" flag can be set at runtime when the container is constructed.

To provide even more flexibility, you might let each object decide who owns it. For instance, an ownership flag could be stored in each object. When an object is passed to a container, the ownership flag can be set to indicate whether the object should be deleted when the container no longer needs it.

Generalizing further, you can store a counter in each object. The counter gets incremented and decremented when containers hold and let go of the objects. Counters such as this are called reference counts. When an object's reference count goes to zero, it can safely be destroyed, since no other object points to it. We'll be using reference count schemes throughout the book.

Collection Classes and Single-Rooted Hierarchies

It's possible to abstract the notion of container classes into what are called collection classes. While a container class directly uses a specific storage mechanism--such as an array, linked list, or tree--with a collection class, that underlying storage mechanism is kept hidden. Collection classes were made popular by the Smalltalk language, and many C++ libraries have attempted to emulate Smalltalk-like collection classes.

The following Bag class is an example of a collection class:

class Bag {

protected:

// Some way of containing objects defined here

...

public:

// Functions to add and remove objects from the bag

int Add(Object *x);

int Find(char *objkey, Object *x);

int Remove(char *objkey);

...

};

A Bag is probably the ultimate collection class. You can "throw" objects into a bag and then retrieve them using a key. Ordering and sequencing are not concerns. Of course, you can have collections that are ordered or sequenced. For instance, you might have a SortableCollection class, derived from a SequencedCollection class.

Most collection classes are implemented so that they have share semantics and are by nature heterogenous. Indeed, that's where they get their power. To mitigate the aliasing and type casting problems caused by using share semantics, many collection classes only allow you to store objects derived from a single root class. In such a single-rooted hierarchy, the root class defines common features that all objects are to have. For example, each object might have the attributes shown in Table 3.1.

The following Object class represents a simple base class of a single-rooted hierarchy. It implements all of the features of Table 3.1, except reference counting:

class Object {

protected:

virtual int IsEqualTo(const Object &fo) const = 0;

public:

virtual ~Object( ) { }

virtual const char *Name( ) const = 0;

virtual int ClassId( ) const = 0;

virtual ostream &Print(ostream &s) const = 0;

int IsSameTypeAs(const Object &x) const;

friend int operator==(const Object &a, const Object &b);

friend int operator!=(const Object &a, const Object &b);

friend ostream &operator<<(ostream &s, const Object &x);

};

Note that this class is abstract, since it uses pure virtual functions. Some of these functions perform important runtime type checking operations. For instance, the ClassId( ) function is supposed to return a unique integer code representing the actual type of the object. The IsSameTypeAs( ) function uses this:

int Object::IsSameTypeAs(const Object &x) const

// Same type of object?

{

return ClassId( ) == x.ClassId( )};

}

Table 3.1 Typical attributes of an object from a single-rooted hierarchy.

Attribute             Purpose

Class name            Debugging aid

Class Id              Runtime type checking

Comparison functions  General sorting purposes

Reference count       Memory management

Printing function     Debugging aid, reporting

Note that the Object class does not specify where the Id is stored or how it is derived. That way, no space needs to be taken in the objects for the Id. However, you don't get something for nothing; to avoid storing the Id, we must use a virtual function, and that means each object must have a virtual function table pointer. (The objects probably will have virtual functions anyway.)

 Each Object is required to provide a comparison function. For example, the IsEqualTo( ) virtual function is used by the overloaded operator==( ) function, as follows:

int operator==(const Object &a, const Object &b)

{

return a.IsSameTypeAs(b) && a.IsEqualTo(b);

}

Note that, for two objects to compare equal, they must be of the same type, which is checked at runtime. This is one example of how a single-rooted hierarchy can alleviate type casting problems.

Let's look at the type casting issue further. Suppose we want to store a collection of bank accounts. For simplicity, we'll assume two possible types of accounts: checking and savings. Furthermore, we'll store these accounts using an array. The following code sketch shows how we might go about this:

//

// Checking account class

//

class CheckAcct : public Object {

protected:

float balance;

virtual int IsEqualTo (const Object &x) const;

public:

CheckAcct( );

CheckAcct(float starting_balance);

CheckAcct(const CheckAcct &ca);

CheckAcct &operator=(const CheckAcct &ca);

virtual float Deposit(float amt);

virtual float Withdraw(float amt);

float Balance( ) const;

// Object management functions

virtual const char *Name( ) const;

virtual int ClassId( ) const;

virtual ostream &Print(ostream &s) const;

};

//

// Savings account with interest

//

class SavingsAcct : public CheckAcct {

protected:

float rate;            // Interest rate

float accum_interest;  // Interest last paid out

virtual int IsEqualTo(const Object &x) const;

public:

SavingsAcct(float starting_balance=0.0, float r = 0.05);

SavingsAcct(const SavingsAcct &ca);

SavingsAcct &operator=(const SavingsAcct &ca);

void PayInterest( );

float InterestPaid( );

// Object management functions

virtual const char *Name( ) const;

virtual int ClassId( ) const;

};

These classes are both derived from Object and, in particular, override the type management functions as follows:

int CheckAcct::IsEqualTo(const Object &x) const

// ASSUMES x is a checking account object. (That's why

// this function is protected.)

{

return ((CheckAcct &)x).balance == balance;

}

const char *CheckAcct::Name() const

{

return "Checking Account";

}

int CheckAcct::ClassId() const

{

return CheckAcctType;

}

int SavingsAcct::IsEqualTo(const Object &x) const

// ASSUMES x is a savings account object. (That's

// why this function is protected.)

{

return ((SavingsAcct &)x).balance == balance;

}

const char *SavingsAcct::Name( ) const

{

return "Savings Account";

}

int SavingsAcct::ClassId( ) const

{

return SavingsAcctType;

}

The ClassId( ) functions are the most interesting, since they return a constant storing a unique code for each class. These codes must be defined somewhere:

// Define class Id's for all types used

const int CheckAcctType = 1;

const int SavingsAcctType = 2;

Where should these constant definitions go? How do we ensure that two classes don't get the same Id? Unfortunately, C++, being a statically typed language, isn't really set up for runtime type checking. (Future versions may provide this feature, though.) We'll lay aside these problems for now, and show how the runtime type checking can come in handy. The following code sketch shows how an array of accounts can be built. Then, for each savings account, we pay the interest due on the account. We use runtime type checking to determine whether an account is a savings account.

const int num_accts = 2;

CheckAcct *MyAccts[num_accts];

MyAccts[0] = new SavingsAcct(1000.0);

MyAccts[1] = new CheckAcct(500.0);

// Pay interest on all accounts

for (i = 0; i<num_accts; i++) {

if (MyAccts[ i ]->ClassId( ) == SavingsAcctType) {

// Only now is type cast safe

SavingsAcct *acct = ((SavingsAcct *)MyAccts[ i ]);

acct->PayInterest( );

}

}

A complete running program for the checking account objects is given on disk in the files object.h, object.cpp, and objtst.cpp.

 SUMMARY

In this chapter, you've seen some strikingly different ways to design data structures in C++. Which way is best? There really isn't one. Each technique can be very useful, given the right application. However, one approach will be used more frequently in this book than the others. Specifically, we'll use concrete data types, and we'll define these types using templates. The reason for doing this is threefold:

       Concrete data types are ideal for this book, since they allows us to show direct designs of a data structure

       Concrete data types are usually more efficient than the other approaches, and in many ways provide a "better match" for C++'s capabilities and philosophy

       By using templates, we can show direct designs that are still general enough to be used for many types of data

The second point is an important one. Concrete data types provide a good match for one of the main design philosophies behind C++, which is to allow code to be written in an encapsulated, object-oriented style--without sacrificing efficiency.

In contrast, the single-rooted hierachy, collection class approach isn't as well suited to C++. Many Smalltalk-like libraries have been built for C++, without considering whether these libraries make any sense in C++. Using these libraries often results in slow and bloated code. Also, the otherwise worthy notion of collection classes is sometimes abused by overzealous designers who don't stop to think about what they are creating. A case in point: some class libraries allow collections to be indexed as though the collection were an array, when in fact a linked-list or tree is being used! It's best not to use a data structure in unnatural ways such as this. Use a data structure for the purpose it was designed.

On the other hand, we don't want to imply that the single-rooted hierarchy approach is bad. Quite the contrary. Single-rooted hierarchies are ideal for applications like graphical user interfaces. Indeed, such interfaces were a major influence on the design of Smalltalk. But, aside from applications like these, you may want to consider whether a single-rooted hierarchy really makes sense.

Go to Chapter 4            Return to Table of Contents


Practical Data Structures: CHAPTER 4: BASIC ARRAY DESIGN

 Next Chapter Return to Table of Contents Previous Chapter

CHAPTER 4: BASIC ARRAY DESIGN

In Chapters 4 through 7, we'll take an in-depth look at arrays. You might find it surprising that, even though an array is basically a simple data structure, there are many issues involved in it's design. C++ has built-in support for arrays, but this support is fairly rudimentary and is often more low-level than you might like. It can be advantageous to create user-defined arrays, so that's what we'll be doing in this set of chapters. Chapter 4 focuses on basic one-dimensional, homogenous arrays, with some discussion of multi-dimensional arrays. Other kinds of arrays will be covered in the next three chapters.

We begin by providing a brief review showing how arrays can be implemented using the built-in facilities of C++. Along the way, we'll explain some of the terminology associated wth arrays. Although some of this review may seem elementary, it provides an important foundation for the the design of the array classes that we'll show later in this and other chapters.

AN ARRAY CLASS HIERARCHY

The design for arrays given in this chapter centers around a hierarchy of classes, as shown in Figure 4.2. All of these classes are templates so that we can use them for any type of array element. The Array class is at he top of the hierarchy. This class serves as the common interface to all the array types, which are as follows:

Fixed-length arrays, including those that are statically and dynamically sized.

Variable-length arrays, where internal indexes keep track of current versus dimensioned lengths. Both statically and dynamically sized arrays are supported.

Dynamic, resizable arrays, whose dimensions can grow and shrink on demand.

BASE ARRAY CLASS

The Array class has the following basic features:

Storage for the array elements is allocated elsewhere.

All copying is done via element-by-element assignment.

The logical length and dimensioned length are differentiated, although only the logical length is actually stored.

Subscript checking is controllable at compile time.

To make functions that use arrays the most general, Array-typed pointers or references can be used for the parameters. This allows any type of array in the hierarchy to be passed to the functions.

Figure 4.2 The Array Class Hierarchy.

Here is the Array class definition:

template<class TYPE>

class Array {

protected:

TYPE *data;    //  Pointer to actual data

unsigned len;  //  Logical length

public:

Array(TYPE *m, unsigned n);

virtual ~Array();

virtual void CopyN(const TYPE *s, unsigned n);

void Copy(const Array<TYPE> &s);

Array<TYPE> & operator=(const Array<TYPE> &s):

#ifndef NO_RANGE_CHECK

unsigned CheckIndx(unsigned i) const;

#endif

TYPE &operator[] (unsigned i);

const TYPE &operator[] (unsigned i) const;

unsigned Length() const;

virtual unsigned DimLength() const;

TYPE *Data();

const TYPE *Data() const;

}:

The code for the Array class is given on disk in the files array.h and array.mth.

Aliased Data

An Array object doesn't actually hold array elements; it stores a pointer to them. We assume this data is allocated elsewhere. For example, we might wrap an Array object around a low-level built-in array so that we can provide subscript checking for the elements. Here's how to declare such a "wrapper array":

int low_level_array[3] = {10, 2, 4};

Array<int> wrap_array(low_level_array, 3);

The Array( ) constructor is used to initialize the wrapper:

template<class TYPE>

Array<TYPE>::Array(TYPE *m, unsigned n)

: data(m), len(n)

{

// Nothing more to do

}

Note that we don't provide a default constructor for Array. We chose not to because you shouldn't really declare Array objects directly (unless you are trying to create a wrapper array). Instead, you should be declaring objects from classes derived from Array. True to form, however, we do provide a virtual destructor, even though it doesn't do anything. Some of the derived array classes will use dynamic allocation for the array data, and we'll need to override the destructor to de-allocate this memory.

Copying Arrays

One important member function is CopyN( ), which allows us to copy data from a low-level array into an Array object:

template<class TYPE>

void Array<TYPE>::CopyN(const TYPE *s, unsigned slen)

{

unsigned clen = DimLength ( );

if (clen > slen) clen = slen;

for (unsigned i = 0; i<clen; i++) data[i] = s[i];

}

In CopyN( ), we only copy as many elements as the source has, and never more than the target array is dimensioned to handle. The length of the array never changes, as illustrated in Figure 4.3. Also, the copying is carefully done, element-by-element (rather than by using something like memcpy( )). This approach allows any overloaded assignment operator that TYPE might have to be called correctly. For example, the statement data[i]=s[i] is actually translated into:

data[i].operator=(s[i]);

Figure 4.3 Copying fixed-length arrays.

It is extremely important to strictly adhere to element-by-element assignment during copying, because it allows the appropriate assignment operator to be called, which may be necessary to avoid aliasing.

The CopyN( ) function is used by the Copy( ) function, which in turn is used by the overloaded assignment operator:

template<class TYPE>

void Array<TYPE>::Copy(const Array<TYPE> &s)

{

CopyN(s.data, s.Length());

}

template<class TYPE>

Array<TYPE> &Array<TYPE>::operator=(const Array<TYPE> &s)

{

if (this != &s) Copy(s);

return *this;

}

Note that the assignment operator traps the case where we try to assign an array to itself; thus, we prevent redundant copying.

Subscript Checking

Array objects, unlike low-level arrays, allow us to provide range checking on subscripts. This can be accomplished by overloading the subscript operator. For performance reasons, it's also useful to allow range checking to be turned on and off at will. How can we accomplish this? Three methods are possible:

1.     Allow range checking to be controlled at runtime, either for all arrays at once, or for individual arrays. The checking can be controlled by using internal flags inside the arrays or by using virtual functions.

2.     Allow range checking to be controlled only at compile time by using conditional macros.

3.     Provide direct access to the array data through an access function, allowing the overloaded subscript operator to be bypassed (and thus the range checking).

With runtime control of range checking, either a flag must be tested or a virtual function call must take place each time a subscript is used. The flag test or the function call allow us to determine whether the index should be checked. Here's one way this could be done:

TYPE &Array::operator[](unsigned i)

{

if (range_check) i = CheckIndx(i);

return data[i]:  //  Use low-level subscripting

}

This approach leads to runtime overhead--even when range_check is false--and the overhead can be significant for tight loops. By using a conditional macro at compile time instead, we can turn off range checking and thus have no overhead at runtime. In conjunction with the macro approach, we can also allow direct access to the array elements in order to bypass the overloaded subscript operator. This approach allows us to turn off range checking locally in code where speed is critical (and where the code has been thoroughly tested). The two Data( ) functions (one for non-const arrays, one for const arrays) are supplied for this purpose. Here are the Data( ) functions, followed by an example of their use:

template<class TYPE>

TYPE *Array<TYPE>::Data()

{

return data;

}

template<class TYPE>

const TYPE *Array<TYPE>::Data() const

{

return data;

}

void f(Array<int> &arr)

// Some function that's been thoroughly tested for

// proper bounds on array indexes

{

int *a = arr.Data( );  //  Get to the low-level array

...

a[17] = 42;

...

}

void g(const Array<int> &arr)

{

int *a = arr.Data( );  //  Const form of Data() is used

int x = a[17];         //  Legal access

a[0] = 1;              //  Illegal access

...

}

In this chapter, we'll use the conditional macro approach combined with the raw data access functions. This combination allows us to use range checking if we need it, but there will be no residual overhead if we don't need range checking. Although this approach isn't as flexible as allowing runtime control of range checking, it meets one of our main design goals: we shouldn't have to pay for something that we're not going to use.

We'll now show how to implement the conditional macro approach. The overloaded subscript operator looks like the following:

template<class TYPE>

TYPE &Array<TYPE>::operator[ ](unsigned i)

{

return data [CHECK (i)];

}

The operator function checks the incoming index by calling the macro CHECK( ), which is conditionally defined as follows:

#ifdef NO_RANGE_CHECK

#define CHECK(i) i

#else

#define CHECK(i) CheckIndx(i)

#endif

The macro symbol NO_RANGE_CHECK is used to control range checking. If this symbol is defined, then Check( ) becomes a no-op. Otherwise, a call to the function CheckIndx( ) is made, which actually does the range checking:

template<class TYPE>

unsigned Array<TYPE>::CheckIndx(unsigned i) const

{

if (i >= len) i = HandleRangeErr(i, len);

return i;

}

HandleRangeErr is a pointer to a function that's designed to handle out-of- range subscripts. We've provided a default error handler that prints an error message and exits the program:

unsigned (*HandleRangeErr)(unsigned i, unsigned sz)

= DefaultRangeErrHandler;

unsigned DefaultRangeErrHandler(unsigned i, unsigned sz)

{

cout << "Subscript " << i << " out of range (0. "

<< (sz-1) << ")\n";

exit(1);

return 0; //  Here to prevent compiler warnings

}

We could also define alternate handlers. All we would need to do is point to them via the HandleRangeErr function pointer.

Note      Future versions of C++ promise more elegant error handling facilities that could be used in place of the function pointer technique.

There is also an alternate subscript operator function, which is used to provide read-only subscripting of const arrays:

template<class TYPE>

const TYPE &Array<TYPE>::operator[](unsigned i) const

// Subscripting operator for constant arrays.

{

return data[CHECK(i)];

}

Note carefully the two uses of the const keyword. These keywords allow us to read the value of an element of a constant array, but they don't allow us to write to that element. For example:

int low_level_array[3] = {10, 2, 4};

const Array<int> myarray(low_level_array, 3);

int x = myarray[2]; //  OK

myarray[2] = 17;    //  Illegal

Without the const version of the subscript operator, even reading from a constant array element would be disallowed.

Logical versus Dimensioned Length

The functions Length( ) and Dimlength( ), respectively, return the logical and dimensioned length of an array:

template<class TYPE>

unsigned Array<TYPE>::Length() const

{

return len;

}

template<class TYPE>

unsigned Array<TYPE>::DimLength( ) const

{

return len;

}

Notice that the dimensioned length by default is the same as the logical length, but a derived class can override the virtual DimLength( ) function to change this. To save space, we avoid actually storing the dimensioned length in the Array object--in case both the logical and dimensioned length are the same (as it will be for fixed-length arrays). Again, our arrays have no more overhead than is absolutely necessary.

 FIXED-LENGTH ARRAYS

With the base Array class defined, it's easy to derive fixed-length array classes. These are arrays that have the same logical length and dimensioned length. We'll define the two classes Darray and Sarray, which support dynamically and statically sized arrays, respectively. Here is the Darray class:

template<class TYPE>

class Darray : public Array<TYPE> {

public:

Darray(unsigned n);

Darray(const Darray<TYPE> &s);

Darray(const Array<TYPE> &s);

Darray(const TYPE *s, unsigned n);

virtual ~Darray();

};

The constructors of Darray work by allocating the array elements using new, and by passing a pointer to these allocated elements to the base class constructor. Here are two constructors that illustrate this:

template<class TYPE>

Darray<TYPE>::Darray(unsigned n)

//  Constructs an array of n dynamically allocated elements

: Array<TYPE>(new TYPE[n], n)

{

//  Nothing else to do

}

template<class TYPE>

Darray<TYPE>::Darray(const Darray<TYPE> &s)

// Copy constructor. Allocates enough room, and

// copies data from the source.

: Array<TYPE>(new TYPE[s.Length()], s.Length())

{

Copy(s);

}

The destructor de-allocates the array elements:

template<class TYPE>

Darray<TYPE>::~Darray()

// De-allocates dynamically allocated array data

{

delete[] data;

}

In contrast, the Sarray class uses a low-level array internally, which is sized and allocated at compile time:

template<class TYPE, unsigned N>

class Sarray : public Array<TYPE>  {

protected:

TYPE storage[N]; //  Statically sized array data

public:

Sarray( );

Sarray(const Sarray<TYPE, N> &s);

Sarray(const Array<TYPE> &s);

Sarray(const TYPE *s, unsigned n);

Sarray<TYPE, N> &operator=(const Array<TYPE> &s);

};

In addition to the usual TYPE parameter, Sarray has the parameter N, which determines the size of the array. This parameter is used to declare an embedded low-level array of N elements that we've called storage. At construction time, the data pointer in the base Array class is linked to storage, as shown in the Sarray default constructor:

template<class TYPE, unsigned N>

Sarray<TYPE, N>::Sarray( )

// Default constructor creates an array of length N

: Array<TYPE>(storage, N) // Link data to storage

{

//  Nothing more to do

}

The other Sarray constructors are defined similarly. Note that no destructor is defined for the Sarray class, since we don't need to delete the statically allocated elements. (A default destructor is created by the compiler, though, which calls the TYPE::~TYPE( ) destructor for each element in the array.)

The code for the Darray and Sarray classes is given on disk in the files darray.h, darray.mth, sarray.h, and sarray.mth. Two test programs are supplied: tstarr.cpp and tstarr2.cpp.

Fixed-Length Array Assignments

Static versus Dynamic Arrays

Fixed-Length Array Assignments

The Sarray class has an overloaded assignment operator function:

template<class TYPE>

Sarray<TYPE.> &Sarray<TYPE>::operator=(const Array<TYPE> &s)

{

if (this != &s) Copy(s);

return *this;

}

We need to make two important points about this operator function. First, by passing a base class Array reference for the source of the assignment, we allow any object from the Array class hierarchy to be assigned to a Sarray object, such as a Darray object. Second, notice that the body of this function is identical to the Array class version. So why did we need to redefine it?

Recall how assignments in derived classes work: without an overloaded assignment function, the additional derived class members are copied member by member. Then, the base class members are copied either memberwise or with an overloaded assignment in the base class. Figure 4.4 illustrates this scenario. If we had no assignment function in Sarray, we would create a problem for the the storage member of Sarray. We want storage to be copied only once. Without the Sarray assignment function, storage would be copied as part of the default derived class assignment behavior, and then copied again via the Copy( ) function inside the Array base class assignment function. Defining an assignment function in Sarray prevents this double copying.

Figure 4.4 A derived class assignment scenario.

class Base   {                     class Derived : public Base  {

protected :                        protected :

char *p, *q;                       char *r;

public :                           public :

Base() :                          Derived();

~Base();                          ~Derived();

Base &operator=(const Base &x);  };

};

You may have noticed that the Darray class doesn't have an assignment operator function. Why not? The Darray class has no additional data members defined, so the base Array class assignment function will handle everything properly. This is a subtle point. So, for consistency, you may want to define a Darray assignment function. (It would look identical to the Sarray version.) We haven't done so here because we want to minimize the amount of code generated by the Darray class. Also, to save space, Darray was not given a default constructor. It's not clear what the default constructor should do anyway. Should it make an array with no elements, or an array with a default length of, say, 10 elements?

Static versus Dynamic Arrays

Dynamic arrays are versatile because their dimensions can be determined at runtime. This same versatility makes dynamic arrays less efficient than those sized statically, since time is spent at runtime allocating and de-allocating the array data. This level of efficiency may be critical when arrays are used inside functions that are called frequently. For instance:

void f()

// Some function called frequently

{

Darray<int>  myarr(10);  //  Allocated on heap

...

// Destructor called automatically to de-allocate array

}

Contrast this with the static allocation of the array:

void g()

// Some function called frequently

{

Sarray<int, 10> myarr;  //  Allocated on stack

...

// Array vanishes as stack is unwound

}

In the static array version, the array is allocated on the stack, which involves a simple manipulation of the stack pointer. The dynamic array version involves a much more involved heap management process. In addition, space overhead is needed for heap management. And don't forget the simple fact that a heap is required. If you want the fastest, most efficient arrays, and you aren't using a heap in your program for other purposes, you might want to use statically sized arrays.

Of course, a tradeoff is involved here. Since the size of the static array is a template parameter, a new class is generated every time you declare a static array of a different size. For instance, the following code causes three classes to be generated (one base class for character arrays in general, and two derived classes for different-sized arrays):

typedef  Sarray<char, 10> tenchar;

typedef  Sarray<char, 20> twentychar;

For this reason, you should limit the number of different-sized static arrays that you declare in your application (and limit the number of arrays of different element types). Note that dynamic arrays do not have this problem, since the array length is passed as a parameter to the constructor.

It's important to understand the difference between sizing an array and allocating it. For instance, statically sized arrays don't have to be statically allocated. You can allocate them dynamically, as in:

Sarray<int, 10>  *p = new Sarray<int, 10>;

We might do this, for instance, to free up memory in the static data segments of our program, without having to implement both the static and dynamic array classes.

 VARIABLE-LENGTH ARRAYS

A variable-length array is one where the logical length of the array is different from the dimensioned length. With a variable-length array, we can add elements to the array (thereby increasing the logical length) and remove elements (thereby decreasing the logical length).

As is true with fixed-length arrays, variable-length arrays can be either dynamic or static. Both of these types of arrays have much in common--in particular, the code that handles adding and removing elements. So it makes sense to make a general variable-length array class, and then derive specific dynamic and static versions. For instance, we've created the Varray class (which is derived from the Array class) and two classes derived from Varray--DVarray and SVarray. The Varray class is defined as follows:

template<class TYPE>

class Varray : public Array<TYPE>  {

protected :

unsigned dimlen;

public :

Varray(TYPE *m, unsigned dim_sz);

void CleanUpElements(unsigned s, unsigned f);

virtual void CopyN(const TYPE *s, unsigned slen);

Varray<TYPE> &operator=(const Array<TYPE> &s);

virtual unsigned Dimlength() const;

virtual unsigned InsertNAt(unsigned p, const TYPE *s, unsigned n);

unsigned InsertAt (unsigned p, const TYPE &s);

unsigned InsertAt (unsigned p, const Array<TYPE> &s);

unsigned Concatenate (const TYPE *s, unsigned n=1);

unsigned Concatenate (const Array<TYPE> &a);

unsigned Add (const TYPE &s);

unsigned DeleteAt (unsigned p, unsigned n=1);

unsigned Shrink (unsigned n);

};

The code for the Varray class is given on disk in the files varray.h and varray.mth. Two test programs are supplied: tstvarr.cpp and tstvarr2.cpp.

As is the case with the Array class, the Varray class doesn't dictate how the array elements are allocated. Instead, it aliases the elements through the following constructor:

template<class TYPE>

Varray<TYPE>: :Varray(TYPE *m, unsigned dim_sz)

: Array<TYPE>(m, 0)

{

dimlen = dim_sz;

}

The base Array constructor is called to do the aliasing and, surpisingly, a length of zero is passed. This represents the logical length of the newly constructed array. The dimensioned length is recorded in the additional data member dimlen. To differentiate between the logical length and dimensioned length, the DimLength( ) virtual function is overridden to return dimlen:

template<class TYPE>

unsigned Varray<TYPE>: :DimLength() const

{

return dimlen;

}

It may seem backwards to have the base class keep track of the logical length and the derived class keep track of the dimensioned length. But this arrangement gives the system of array classes its flexibility and economy.

With the Varray class, you can add single or multiple elements either to the end of the array (that is, after the current logical length) or somewhere in the middle. The InsertNAt( ), InsertAt( ), Concatenate( ), and Add( ) functions handle these chores. The DeleteAt( ) and Shrink( ) functions allow you to delete one or more elements from the middle or end of the array, respectively. All of these functions manipulate the logical length of the array. We'll take a look at these functions later. First we need to discuss an important consideration: user-controlled allocation.

In-Place Construction

Some subtle issues are involved in adding and removing elements from a variable-length array. These issues revolve around two questions: when and where does an array element get constructed? When and where does it get destructed? To have safe, efficient, robust code, we must ensure that the constructors and destructors get called the proper number of times, in the right order, and only when necessary.

With fixed-length arrays, the construction and destruction of elements is quite simple: when the array elements are allocated--either by using new or through a static declaration--the default constructor of the element type is called for each element. When the array is to be destroyed, the destructor is called for each element. For example:

void f( )

{

MyType *darr = new MyType[17];  //  Mytype constructor called 17 times

MyType sarr[42];  //  MyType constructor called 42 times

. . .

delete[] darr;  //  MyType destructor called 17 times

//  At end of function, MyType destructor called 42

//  times for the elements of sarr

}

If you inspect the Darray and Sarray classes, you'll see that their array elements are constructed and destructed in this way.

For variable-length arrays, things get trickier. When a variable-length array is first constructed, it has a logical length of zero. Although the space for the array has been allocated, no constructed elements are stored there. That is, the array storage consists initially of unconstructed elements. As elements are added to the array, the constructor must be called for those elements. Figure 4.5 shows a variable-length array with four constructed elements. When elements are removed, the destructor must be called for each removed element. When it is time to destroy the array itself, if there are remaining elements in the array, the destructor should be called for each one.

Thus, the problem boils down to your ability to construct and destruct elements at will. You must be able to do the construction and destruction at the place where the element occurs in the array. This is the tricky part. Normally, when you construct an object, either the compiler and linker control the address of the object (for statically allocated objects), or the heap manager controls the address (via the new operator). Fortunately, C++ provides the tools to let you control the address of the object yourself. For instance, you can override the new operator and pass in the address of the object to be constructed:

Figure 4.5 A variable-length array.

void *operator new(size_t, void *p)

{

return p;

}

Actually, the new operator itself won't cause the constructor for the object to be called, but the compiler will insert the constructor call when it sees the new operator. The address of the object to be passed to the constructor (the this pointer) is returned by the new operator. Now you can see why the simple overloaded new operator works: it merely has to return the same address passed in. Note that the first parameter of new (the size of the object, which is required by the C++ syntax) isn't needed by the function. Normally, the size is used to tell the heap manager how many bytes to allocate. But, in this case, the space is assumed to be allocated already.

Note     An overloaded new operator function--such as the one we've just given--is sometimes called a placement new operator; it places a new object at a user-defined address. Using a placement new operator to construct an object at a specified address is called in-place construction.

Figure 4.6 illustrates how we could construct the fourth element of an array of Compound numbers (see Chapter 3), and initialize it to the value <9 0/1> using the placement new operator. The address of the second element is passed to new, and then the default constructor is called, as in the following code:

// Declare array of 6 compound numbers, initially unconstructed:

char myarr[6*sizeof(Compound)];

...

new(&myarr[3]) Compound(9);  //  Construct fourth element

Figure 4.6 In-Place construction of an array element.

Note how the array elements are initially allocated as characters, rather than Compound numbers. That's done to make the array have unconstructed bits initially. If we had used

Compound myarr[6];

then the Compound default constructor would have been called for all six elements, which isn't what we want.

You can also call the default constructor (or any other type of constructor, for that matter) with the placement new operator. For example:

new(&myarr[3]) Compound;  //  Default constructor called

With these ideas in mind, the following helper function templates are used by the Varray class to construct the elements of a variable-length array:

template<class TYPE>

void DefConstruct(TYPE *p)

// This function constructs an element at place p.

// Calls the default constructor.

{

new(p) TYPE;

}

template<class TYPE>

void DefConstructElements(TYPE *t, unsigned len)

// Default constructs the elements of low- level array t,

// assumed to have been properly allocated

{

for (unsigned i = 0: i<len; i++) DefConstruct(t++):

}

template<class TYPE>

void CopyConstruct(TYPE *p, const TYPE &x)

// Copy constructs object at address p, using

// object x for the copy

{

new(p) TYPE(x):

}

template<class TYPE>

void CopyConstructElements(TYPE *t, const TYPE *s, unsigned len)

// Here, we assume raw space for t has been properly allocated,

// and we're ready to copy construct the elements of s into them

{

for (unsigned i = 0; i<len; i++) CopyConstruct(t++, *s++):

}

The CopyConstruct( ) function template assumes TYPE has a copy constructor. If that's not the case, you can override the function template for a specific type. The overriding function can call the default constructor, and then do an assignmenv. For example:

void CopyConstruct(MyType *p, const MyType &x)

{

new(p) MyType;  //  Call default constructor

*p = x;         //  Do assignment

}

In cases where TYPE doesn't have a constructor at all, such as for built-in types, you can simply do an assignment, as shown in the following overridden CopyConstruct( ) function for integers:

void CopyConstruct(int *p, const int &x)

{

*p = x;  //  Do assignment

}

Explicit Destructor Calls

Calls to destructors are normally handled automatically by the compiler. You can control when a destructor is called if you use the delete operator. However, this assumes that the object to be destroyed resides on the heap. But what do you do if the object resides as an element of an array? The solution is to make an explicit call to the destructor. Here's an example:

Compound *p = (Compound *) (&myarr[1]);  //  Point to element

...

p->Compound::~Compound( );  //  Explicit destructor call

The explicit destructor call requires that you know the address of the object being destructed, and you can only call the destructor through a pointer. You should never call the destructor for the same object twice, and never for an object that resides on the heap (since it won't be freed properly unless delete is called). The Varray class uses an explicit destructor call in the function CleanUpElements( ):

template<class TYPE>

void Varray<TYPE>::CleanUpElements(unsigned s, unsigned f)

// Cleans up elements from start s to finish f-1, by calling

// destructors explicitly

{

TYPE *p = data+s;

for (unsigned i=s; i<f; i++, p++) p->TYPE::~TYPE( );

}

This template function does have a problem, though. What if TYPE has no destructor? In theory, the compiler can simply ignore the explicit destructor call, since nothing needs to be done. In practice, if TYPE is a built-in type--such as int or float--rather than a user-defined class, many compilers will give you an error. For example:

int *p;

p->int::~int( );  //  Some earlier compilers can't handle this

The ability to use  "user-defined class syntax" for built-in types--such as in explicit calls to destructors--was added to C++ 3.0. Many C++ 2.1 compilers will have troubles in this area. To solve this problem, a special Destruct( ) function template can be defined, as follows:

template<class TYPE>

void Destruct(TYPE *p)

// Calls the destructor for the object pointed to by p.

// ASSUMES the object was constructed using the placement

// new operator.

{

p->TYPE::~TYPE( );

}

Then, if the array you are using is for a built-in type, you can override the function template to do nothing. For example, here's how you can override Destruct( ) for ints:

void Destruct(int *p)

{

// Do nothing

}

By using appropriate versions of Destruct( ), we can rewrite CleanUpElements( ) to work for any type, as follows:

template<class TYPE>

void Varray<TYPE>::CleanUpElements(unsigned s, unsigned f)

{

TYPE *p = data+s;

for (unsigned i=s; i<f; i++) ::Destruct(p++);

}

Note     The code for the placement new operator (and for associated special constructor and destructor helper functions) is given on the disk in the files placenew.h and placenew.mth.

Inserting Elements into an Array

Elements are inserted into a variable-length array by first shifting array elements down to make room for the new elements (unless the elements are added to the end), and then copy-constructing the new elements in place. Figure 4.7 illustrates the process. The workhorse function is InsertNAt( ):

template<class TYPE>

unsigned Varray<TYPE>::InsertNAt(unsigned p, const TYPE *s, unsigned n)

// Inserts the data pointed to by s into the array. Up to n

// elements are inserted (truncating if necesary), starting at

// position p (counted by zero). If p >= len, then the data

// is concatenated to the end.

// Returns number of elements inserted.

{

unsigned i;

if (n > dimlen - len)  { // Keep things in range

n = dimlen - len;

}

if (p >= len) { // We're concatenating

p = len;

}

else {

// Make room for inserted data somewhere in the middle

memmove(data+p+n, data+p, (len-p)*sizeof(TYPE));

}

// Copy in source

len += n;

for(i = 0; i<n; i++) {

// Copy-construct the elements in place (they're presumed

// to be unconstructed at this time.)

::CopyConstruct(data+p+i, s[i]);

}

return n;

}

Figure 4.7 Inserting elements into a variable-length array.

The important feature of this function is that the newly inserted elements are created element-by-element, using a copy-construction process. The other insert functions use InsertNAt( ) to do most of the work:

template<class TYPE>

unsigned Varray<TYPE>::Add(const TYPE &s)

// Add an element onto the end of the array

{

return InsertNAt(len, &s, 1);

}

template<class TYPE>

unsigned Varray<TYPE>::InsertAt(unsigned p, const Array<TYPE> &a)

// Insert an array of elements into position p of the array

{

return InsertNAt(p, a.Data( ), a.Length( ));

}

template<class TYPE>

unsigned Varray<TYPE>::Concatenate(const Array<TYPE> &a)

// Add array a onto the end of this array

{

return InsertAt(len, a);

}

template<class TYPE>

unsigned Varray <TYPE>::Concatenate(const TYPE *s, unsigned n)

// Concatenate an array of n elements, (from low-level memory),

// onto the end of the array.

{

return InsertNAt(len, s, n);

}

Deleting Elements from an Array

Deleting elements from an array is the opposite of inserting elements, as illustrated in Figure 4.8. First, the destructor is called for each element to be deleted, and the remaining elements are then shifted up to take the p place of the deleted elements. The DeleteAt( ) function does all the work:

template<class TYPE>

unsigned Varray<TYPE>::DeleteAt(unsigned p, unsigned n)

// Deletes up to n elements from array at position p. If p

// is out of range nothing happens. If n == 0, nothing happens.

// Returns number of elements deleted.

{

long pel;  //  long is used to prevent overflows during adds

unsigned pe;

if (p < len && n != 0) {

// We may be chopping off the end, so keep in range

pel = p + n;

if (pel > len) pel = len;

pe = unsigned(pel); // Guaranteed to be in range

n = pe - p;

// Now, clean up each element deleted

CleanUpElements(p, pe);

// Now, move elements up to take their place

memmove(data+p, data+pe, (len-pe)*sizeof(TYPE));

len -= n;

} else n = 0;

return n;

}

Figure 4.8 Deleting elements from a variable-length array.

A special purpose function to delete elements from the end of the array, called Shrink( ), uses DeleteAt( ):

template<class TYPE>

unsigned Varray<TYPE>::Shrink(unsigned n)

// Shrink the array by n elements.

// Return number of elements we've shrunk.

{

if (n > len) n = len;

return DeleteAt(len - n, n);

}

Copying Variable-Length Arrays

As is true with all the array classes, an overloaded assignment operator is provided for the variable-length arrays. This assignment operator can use all the different types of arrays as the source:

Varray<TYPE> &Varray<TYPE>::operator=(const Array<TYPE> &s)

// Note that source of assignment can be any array type.

{

if (this != &s) Copy(s);

return *this;

}

The assignment operator calls the Copy( ) function, which in turn calls the virtual CopyN( ) function to do the actual copying. The details for copying into a variable-length array are different from those for copying into a fixed-length array. For this reason, the Varray class overrides CopyN( ).

When you copy data into a variable-length array, you might not only be adding elements to the array, but you might also be overwriting existing elements. A copy-construction process is used for adding new elements, whereas an assignment operation takes place for existing elements. Also, if the source array is smaller than the target array, elements must be deleted from the target array. Figure 4.9 illustrates these cases. The following CopyN( ) function for Varray handles the details:

template<class TYPE>

void Varray<TYPE>::Copy N(const TYPE *s, unsigned slen)

// Copies as much data as possible from s into this

// array, truncating if need be. Copying is done via

// element-to-element assignment.

{

unsigned i, tlen;

// First, copy into the constructed portion

tlen = len;

if (tlen > slen) {

Shrink(tlen-slen); // We might be getting smaller

tlen = slen;

}

for (i = 0; i<tlen; i++) data[i] = s[i];

len = tlen;

// Now, copy into the unconstructed portion via concatenation

if (tlen < slen) Concatenate(s+i, slen-tlen);

}

The Concatenate( ) function is cleverly called to handle the elements that must be copy constructed.

Dynamic Variable-Length Arrays

Now that we've created the Varray class, we can derive more specific dynamic and static versions. Here is the DVarray class for dynamic, variable-length arrays:

template<class TYPE>

class DVarray : public Varray<TYPE> {

public:

DVarray (unsigned n);

DVarray (const DVarray<TYPE> &s);

DVarray (const Array<TYPE> &s);

DVarray (const TYPE *s, unsigned n);

virtual ~DVarray();

DVarray<TYPE> &operator=(const Array<TYPE> &s);

};

Figure 4.9 Copying variable-length arrays.

(a) Source longer than target

(b) Source shorter than target

The DVarray class allocates the array elements from the heap, but keeps those elements unconstructed until needed. This is done by creating an array of characters, which is then typecast into an array of TYPE. For the copy constructor, the array elements from the source array are constructed and copied, again by clever use of the concatenate function. (Remember that the array initially has a logical length of zero.)

template<class TYPE>

DVarray<TYPE>::DVarray(const DVarray<TYPE> &s)

//  Copy constructor. Note that we first allocate an

//  array of unconstructed bits. Using concatenate

//  here effectively does the element-by-element copy

//  construction for us.

: Varray<TYPE> ((TYPE *)new char[s.DimLength( )*sizeof(TYPE)],

s.DimLength( ))

{

Concatenate(s);

}

The other constructors work in a similar manner. The destructor performs the reverse process. First, any constructed elements remaining in the array are destroyed, and then the storage is freed from the heap:

template<class TYPE>

DVarray<TYPE>::~ DVarray()

{

//  Clean up after all constructed members

CleanUpElements(0, len);

//  Delete the storage

delete[] (char *)data;

}

Note the typecast to char* in the delete call. Had we not used this typecast, the destructor for the array elements would be erroneously called--not only for unconstructed elements, but also for elements that had just been destroyed.

The code for the DVarray class is given on disk in the files dvarray.h and dvarray.mth.

Static Variable-Length Arrays

The SVarray class, used for static variable-length arrays, is similar to the DVarray class:

template<class TYPE, unsigned N>

class SVarray : public Varray<TYPE>  {

protected:

char storage[N*sizeof(TYPE)];

public:

SVarray( );

SVarray(const SVarray<TYPE, N> &s);

SVarray(const Array<TYPE> &s);

SVarray(const TYPE *s, unsigned n);

virtual ~SVarray( );

SVarray<TYPE, N> &operator=(const Array<TYPE> &s);

};

As was true with the DVarray class, the storage is allocated as an array of characters and then typecast to an array of TYPE, as the following copy constructor shows:

template<class TYPE, unsigned N>

SVarray<TYPE, N>::SVarray(const SVarray<TYPE, N> &s)

//  Copy constructor. Note that storage is presumed

//  to be unconstructed bits coming in. Using concatenate

//  here effectively does the element-by-element copy

//  construction for us.

: Varray<TYPE>((TYPE *)storage, N)  //  Link to storage

{

Concatenate(s);

}

The destructor must clean up any remaining constructed elements, but of course no call to delete is needed:

template<class TYPE, unsigned N>

SVarray<TYPE, N>::~SVarray( )

{

//  Clean up all constructed members

CleanUpElements(0, len);

}

The code for the SVarray class is given on disk in the files svarray.h and svarray.mth.

 RESIZABLE ARRAYS

A resizable array is a more sophisticated type of array. Not only is a resizable array variable length (in terms of logical length), but its allocated, dimensioned length can also vary. By definition, resizable arrays are dynamically sized. In fact, we can derive a resizable array class, RVarray, from DVarray:

template<class TYPE.>

class RVarray : public DVarray<TYPE> {

protected:

int grow_by;

public:

RVarray(unsigned n, int gb);

RVarray(const RVarray<TYPE> &s);

RVarray(const Array<TYPE> &s);

RVarray(const TYPE *s, unsigned n);

virtual void CopyN(const TYPE *s, unsigned n);

RVarray<TYPE> &operator=(const Array<TYPE> &s);

virtual unsigned InsertNAt(unsigned p, const TYPE *s, unsigned n=1);

int Realloc(unsigned new_dimlen, int keep=1);

int GrowBy(unsigned amt);

int FreeExtra( );

void ChgGrowByInc(int gb);

};

Resizable arrays work as follows: if the array runs out of room while elements are being added, the array grows to a larger size. This is accomplished by allocating a new, larger space, for the array, copying all of the existing elements into the new space, and then deleting the old space. To avoid frequent resizing, the increment for the new size is always above a certain threshold, determined by the grow_by variable, which can be controlled by the user at construction time and through the ChgGrowByInc( ) function. If grow_by is zero, then the array can't be resized.

Realloc( ) is the workhorse function of the RVarray class:

template<class TYPE>

int RVarray<TYPE>::Realloc(unsigned new_dimlen, int keep)

//  NOTE: If an error occurs, the data is left intact, and a

//  0 is returned. Otherwise it returns a 1 to indicate success.

{

if (grow_by == 0) return 0;

char *new_data = new char[new_dimlen * sizeof(TYPE)];

if (new_data == 0) return 0;

dimlen = new_dimlen;

if (keep) {

if (new_dimlen < len) {

//  Array is getting shorter. We must call the

//  destructor on all orphaned elements.

CleanUpElements(new_dimlen, len);

len = new_dimlen;

}

//  Copy old data into new space

memmove(new_data, data, len*sizeof(TYPE));

}

else {

//  We're not keeping any of the data, so clean it up

CleanUpElements(0, len);

len = 0;

}

delete[] (char *)data;  //  Delete old storage place

data = (TYPE *)new_data;

return 1;

}

Realloc( ) redimensions the array to the new length of new_dimlen. The keep option flag determines whether the existing elements are to be kept (when the array grows because we're adding elements) or whether the elements can be destroyed (when we're about to replace the contents of the array with data from another array). If the new size is to be smaller, the elements to be chopped off must be properly destroyed as well. Elements to be destroyed are handled through the CleanUpElements( ) function, which makes sure the appropriate destructor is called for each element.

The RVarray class overrides both the CopyN( ) and InsertNAt( ) functions of the Varray class, in order to allow resizing. Unlike all the other types of arrays, copying to a resizable array may change that array's dimensioned length if the source array is longer. Figure 4.10 illustrates the process. After the potential resizing has taken place, the Varray version of CopyN( ) is called, as shown in the following function:

template<class TYPE>

void RVarray<TYPE>::CopyN(const TYPE *s, unsigned n)

{

if (n > dimlen) Realloc(n, 0);

Varray<TYPE>::CopyN(s, n);

}

The new InsertNAt( ) function works in a similar fashion, first by handling any necessary resizing, and then calling the Varray version of InsertNAt( ):

template<class TYPE>

unsigned RVarray<TYPE>::InsertNAt(unsigned p,const TYPE *s,unsigned n)

//  Inserts the data pointed to by s into the array. Up to n

//  elements are inserted (truncating if necesary), starting at

//  position p (counted by 0). The array will grow in size

//  if needed and if grow_by != 0. The size it will grow to

//  will be the larger of grow_by and the room_needed.

//  If p >= len, then the data is concatenated onto the end.

//  Returns number of elements inserted, or 0 if error.

{

unsigned room, needed;

room = dimlen - len;

if (n > room && grow_by != 0) {

needed = n - room;

if (needed < grow_by) needed = grow_by;

//  Grow the array by needed amount. Note that the

//  growth may fail, in which case the array stays

//  the same size, with data intact.

GrowBy(needed);

}

return Varray<TYPE>::InsertNAt(p, s, n);

}

Figure 4.10 Copying resizable arrays.

Although the resizing of RVarrays takes place automatically, you can manually resize the array through the GrowBy( ) and FreeExtra( ) functions.

template<class TYPE>

int RVarray<TYPE>::GrowBy(unsigned amt)

//  Grows the dimensions of the array by the specifed amt.

{

return Realloc(dimlen+amt);

}

template<class TYPE>

int RVarray<TYPE>::FreeExtra( )

//  Frees all unused space in the array.

{

return Realloc(len);

}

The code for the RVarray class is given on disk in the files rvarray.h and rvarray.mth. Two test programs are supplied: tstrva.cpp and tstrva2.cpp.

 MULTI-DIMENSIONAL ARRAYS

The Array classes allow you to use any type for the elements. So what will happen if you use an array itself as the element type? The result is an array of arrays, also called a two-dimensional array. Continuing the process further, you can use a two-dimensional array as the element type for a three-dimensional array, and so on. We'll now take a detailed look at the ways we can accomplish this. Note that the techniques shown in this chapter represent just one way to create user-defined, multi-dimensional arrays. In Chapter 7, we'll look at alternative methods--in the context of two-dimensional matrices.

Note      In this book, we'll denote a multi-dimensional array of n dimensions as an n-d array.

Static Multi-Dimensional Arrays

Dynamic Multi-Dimensional Arrays

Other Approaches to Multi-Dimensional Arrays

Static Multi-Dimensional Arrays

It is particularly easy to declare user-defined n-d arrays if we fix the sizes statically. This approach involves the use of nested template instantiations of the Sarray class. (Realize that we are not nesting the template definitions themselves, only instantiations of them.) In this example, we define an array type that holds three rows by five columns of integers:

typedef Sarray<int, 5> IntArr5;

typedef Sarray<IntArr5, 3> IntArr3  5;

Note how the IntArr5 type, used for each row of the array, is used as a parameter to the IntArr3x5 type. The layout of the resulting 2-d array is given in Figure 4.11. We've shown all of the memory used, including the virtual function table pointer for Sarray objects. Note that all of the memory cells shown are actually laid out in memory linearly, contrary to what's shown.

Figure 4.11 A 2-d array implemented with sarray objects.

In this example, we'll declare and use such an array:

IntArr3x5 myarr;

myarr[2][4] = 17;  //  Set the lower-right corner element

Take a look at the double subscripting. The first subscript a[2] causes element 2 of the array myarr to be returned. This element happens to be an array itself, and the second subscript [4] causes element 4 of this secondary array to be returned. The double-subscripting results in the following code, where we call appropriate subscript operator functions from the IntArr5 and IntArr3x5 classes:

//  First call IntArr3x5: :operator[] , then IntArr5: :operator[ ]

(a.operator[] (2) ).operator[] (4) = 17;

With this arrangement, we are able to use the same double-subscripting syntax that we would use with a built-in two-dimensional array. Here, though, we can have range checking done on the subscripts of our user-defined array--by virtue of the overloaded subscript operators.

We can take this approach one step further and define 3-d arrays from 2-d arrays. Here's a 2x3x5 integer array type and an example of using it:

typedef Sarray<lntArr3x5, 2> IntArr2x3x5;

IntArr2x3x5 b;

b [1][2][4] = 17;  //  Set lower-right corner element

Defining even higher-dimensional arrays works in a similar fashion.

You can get more sophisticated and create 2-d static arrays whose logical dimensions vary at runtime by using SVarray instead of Sarray, as in the following code:

typedef SVarray<int, 5> IntArr5;

typedef SVarray<IntArr5, 3> IntArr3x5;

IntArr3x5 a;  //  Logical dimensions start out 0x0

a.Concatenate( IntArr5 () );  //  Add a row

a[0].Concatenate (42);        //  Add a column to row 0

a[0][0] = 17;  //  Change top-left corner element

a[0][1] = 25;  //  Error : second subscript out of bounds

a[1][0] = 17;  //  Error : first subscript out of bounds

This code illustrates the difference between the allocated dimensionality and the logical dimensionality of a 2-d array. Each IntArr3x5 array has 3x5 elements allocated, but starts out with a logical dimensionality of 0x0. In effect, the array has no rows (and no columns for that matter). To use the array, we must add rows, and to each row we must add columns. Vhe total number of rows that can be added is 3, and the total number of columns per row is 5. Each row can actually have a different number of logical columns (but only up to 5), as shown in Figure 4.12. (Again, the actual linearity of the data isn't shown.) In any case, each array actually has 3 rows of 5 columns allocated for its use.

Although there is a considerable amount of power here, the allocated dimensions of the array must be determined at compile time. But what if we want to determine the dimensions at runtime? For instance, suppose we sometimes need 5x100 arrays, but at other times only 3x5 arrays? In cases like this, we may want to defer the sizing until runtime, and only allocate space for 5x100 arrays when necessary.

Dynamic Multi-Dimensional Arrays

By using a technique similar to the one used in the previous section, we can define n-d arrays that have their allocated sizes determined in part at runtime. It's quite easy to size the first dimension at runtime, but unfortunately the other sizes must be set at compile time. For example, with a 2-d array, we can determine the number of rows to be allocated at runtime, but we must fix the number of columns. This method involves using a Darray object instead of a Sarray object for the final dimension. For each row, we can use either Sarray or SVarray objects:

Figure 4.12 A jagged 2-d array implemented with SVarray objects.

typedef SVarray<int, 5> IntArr5;

typedef Darray<IntArr5> IntArrNx5;  //  Note use of Darray

IntArrNx5 a(10);  // Make a 10x5 array

IntArrNx5 b(3);  //  Make a 3x5 array

We can get even more exotic by allowing the number of logical rows to vary after the array has been constructed. For example, we could use DVarray instead of Darray:

typedef SVarray<int, 5>  IntArr5;

typedef DVarray<IntArr5> IntArrNx5;  //  Note use of DVarray

IntArrNx5 a(10);  //  Logical dimensions start out 0x0

a.Concatenate(IntArr5( ));  //  Add a row

a[0].Concatenate(42);       //  Add a column

a[1][4] = 17;  //  Subscript error!

Getting even fancier, we can use an RVarray instead of a DVarray so that we can change the allocated number of rows at runtime:

typedef Sarray<int, 5> IntArr5;

typedef RVarray<IntArr5> IntArrNx5;  //  Note use of RVarray

//  Make an array that's initially allocated 3x5, but that

//  can grow by 2 row increments.

IntArrNx5 a(3, 2);         //  Logical size is 0x0

a.Concatenate(IntArr5());  //  Add three rows

a.Concatenate(IntArr5());

a.Concatenate(IntArr5());  //  Logical size is now 3x0

a.Grow();  //  Allocate room for two more rows

Figure 4.13 shows what the memory layout of such an array might look like after adding some elements.

One thing we can't do, even using the RVarray class, is determine the number of columns to be allocated at runtime. That is, we can't implement the rows themselves with RVarray. (We could use SVarray, though.) Why not? Let's try it:

typedef RVarray<int> IntArrM;

typedef RVarray<IntArrM> IntArrNxM;

//  Starting with 10 rows, but how many columns?

IntArrNxM a(10);  //  Compiler error!

Figure 4.13 A resizable 2-d array declared with the RVarray class.

We could never declare objects of type IntArrNxM. RVarray doesn't have a default constructor, but one would be needed to set up each row. This default constructor would have to specify a default number of columns. But what should the default be? Even if we do supply a default, we're in essence fixing one dimension of the array, which puts us right back where we started.

It's interesting to note that built-in 2-d arrays have a similar problem. For example, it's quite common to pass a 2-d array to a function, where the number of rows is variable, but the number of columns is fixed:

void f(int (*a)[10], int nr)

//  Pass in an array of nr rows and 10 columns

{

for (int i = 0; i<nr; i++) a[i][0] = 1;  / /  Set first col to 1's

}

Unfortunately, it's not possible to pass a 2-d array to a function where the columns dimension is also specified by a parameter. The reason involves the way built-in 2-d arrays are implemented using pointers, a topic we'll discuss in Chapter 7.

Other Approaches to Multi-Dimensional Arrays

The inability to specify the number of columns dynamically is a major limitation to the nested template approach to declaring 2-d arrays. And even when we allow the number of rows to vary, using the RVarray class, a lot of pointers and length variables are redundant and add overhead. Fortunately, there are ways around these problems. In Chapter 7, you'll see an alternative design to 2-d arrays that uses a more compact representation, which also allows both dimensions to be determined at runtime.

 USER-DEFINED ARRAYS VERSUS BUILT-IN ARRAYS

Let's step back and see how efficient the Array class hierarchy is compared to using built-in arrays. At a minimum, each array object requires space for a pointer to the array data, the array data itself, the length of the array, and a pointer to a virtual function table. Thus, the minimum space overhead, per array, is two pointers plus an integer. If you want variable-length arrays, an extra integer is required to store the dimensioned length. For resizable arrays, add another integer to store the grow_by increment. In any case, only when you need more sophistication do you pay for it.

Since each class in the Array hierarchy is actually a template, class methods will be replicated for each type of array element used. A specific Array class will be generated for any given array element type, plus one or more derived classes. With resizable arrays, four classes are generated per element type: Array<TYPE>, Varray<TYPE>, DVarray<TYPE>, and RVarray<TYPE>. The potential for code explosion does exist here, but it is mitigated by three facts:

       You may not have very many different types of objects being stored in arrays for a single application.

       Many of the class methods are short enough to be inlined, and often do nothing more than refer to other methods or return a data member value.

       You only pay for classes you need. For instance, no RVarray<TYPE> class will be generated if you're not using resizable arrays.

Subscripting will be slower with range checking, and in tight loops the reduction in speed may be significant. For instance, sorting an array with range checking turned on is typically twice as slow as it would be with range checking off. Even with range checking off, subscripting will be slightly slower than it would be if we used low-level arrays directly. The reason: we must access the array elements indirectly through the this pointer of the Array object. A clever compiler might preload a pointer to the elements into a register before a loop is performed. Thus, the subscripting in the loop would be as efficient as possible. Unfortunately, many compilers aren't that clever.

Even though subscripting with range checking is slower, we've made it as efficient as possible by carefully designing the arrays so that the overloaded subscript operator functions are not virtual. Also, the subscript operator functions need only be specified once, in the Array class. The result is minimum speed and space overhead.

 ALTERNATIVE ARRAY DESIGNS

We chose to make the base Array class a template. That means whenever we instantiate a derived class array template, at least two classes will be generated: the base class and the derived class. We could eliminate generating a base class every time by using a non-template Array class. This can be done by using a void pointer to point to the array elements, rather than a TYPE pointer. For example:

class Array  {  //  Note, not a template

protected:

void *data;    //  Point to any type of array data

unsigned len;  //  Length of array

int size;      //  Size of an array element

. . .

};

The trouble with this approach lies in array subscripting. Because the type of data isn't known, we must store the size of the array element and use that size during subscripting operations. Worse, we must use a typecast to get the actual type of the array element. This leads both to performance and maintenance problems. We might decide not to define subscripting in the Array class, instead leaving that up to the derived classes (which are templates with specified element types). But then we couldn't pass generic Array pointers or references, rendering functions that use arrays less versatile.

Go to Chapter 5            Return to Table of Contents


Practical Data Structures: CHAPTER 5: HETEROGENOUS ARRAYS

 Next Chapter Return to Table of Contents Previous Chapter

CHAPTER 5: HETEROGENOUS ARRAYS

In Chapter 3, we introduced the use of heterogenous arrays to allow a container to store different types of objects. In that chapter, we mentioned ways that arrays could be used as heterogenous containers. This chapter will investigate some of these ways more thoroughly. In particular, we'll look at ways to implement reference counting, a technique used to handle the problems of pointer aliasing. But first, we'll review some of the ideas we presented in Chapter 3.

ARRAYS OF POINTERS

By definition, an array can only easily store one type of object, since each element must be the same size. So how do we create heterogenous arrays, where the elements are different object types--perhaps of different sizes? The answer is that we can't. We can, however, store pointers to the objects. That's because pointers are usually the same size, regardless of the type of object they point to. Figure 5.1 shows an example of a heterogenous array that points to shape objects.

Arrays Pointing to Any Type of Object

The Array classes discussed in Chapter 4 can be readily used to create heterogenous arrays. In the most general case, the element type should be void *, which allows any type of object to be pointed to. However, this leads to all sorts of problems, since we can use the objects only if the pointers are typecast, an error-prone activity. For example:

Figure 5.1 A heterogenous array.

Darray<void *> vpa(2)  // Create an array of void pointers

vpa[0] = new Toaster; // Store a Toaster object

vpa[1] = new Ufo;     // Store a Ufo object

((Ufo *)vpa[1])->Fly();            // Okay

((Ufo *)vpa[0])->Fly();            // Likely program crash!

Arrays Pointing to Objects from a Hierarchy

One way to reduce typecasting problems like those shown in the previous section is to restrict the objects to be only those from a particular class hierarchy. Then, good use of virtual functions will reduce (but not eliminate) the need for typecasting.

For example, the following code shows a class hierarchy that consists of a base Shape class and a derived Circle class. A virtual Area( ) function is defined in each class. For Shape objects, Area( ) returns 0. For Circle objects, the standard r2 formula is used. We can store both types of objects in an array consisting of Shape pointers:

class Shape {

protected:

float x, y;

public:

Shape()                    { x = 0; y = 0; }

Shape(float xx, float yy)  { x = xx; y = yy; }

virtual float Area()       { return 0; }

};

class Circle : public Shape  {

protected:

float r;

public:

Circle() { r = 0; }

Circle(float xx, float yy, float rr) : Shape(xx,yy) { r = rr; }

virtual float Area() { return 3.14*r*r; }

};

Darray<Shape *> spa(2);  // Create array of shape pointers

spa[0] = new Shape(1,2);

spa[1] = new Circle(3, 4, 5);

for(int i = 0; i<2; i++) cout << spa[i]->Area() << '\n';

Pointer Aliasing

On the surface, the use of arrays of pointers to implement heterogenous arrays seems quite easy. However, one major problem lurks beneath the surface: pointer aliasing. Both examples that we've shown neglected to do one important thing: delete the objects that were created on the heap. This isn't terribly difficult to do in simple cases. For instance, we could add the following two lines to our previous example:

delete spa[0]; // Delete object being pointed to

delete spa[1];

However, what happens if we were to have two arrays in our example, and we decide to copy one array into the other?

Darray<Shape *> spb(2);

spb = spa; // What happens here?

Due to the design of the Array classes, only the pointers to the Shape objects would be copied, not the objects themselves. The result is that both arrays would point to the same objects. Now the question is: which array should be responsible for deleting the actual objects? Or should we make copies of the objects instead?

We can use four basic methods--ranging from simple to complex--to handle this aliasing problem:

1.     Use the convention that heterogenous arrays are never responsible for cleaning up the objects they point to. Instead, whatever created the objects is responsible. This is the default behavior when using one of the Array classes with pointer elements.

2.     A flag could be stored in each array to indicate whether the array is responsible for cleaning up the objects or whether some other array is responsible. Only the first array that references a particular set of objects will have this flag set to one. For all other array copies, the flag is set to zero.

3.     Store a flag in each object, indicating whether the object itself or the array it's associated with is responsible for cleanup of the object. In this scenario, a flag is still needed in each array, (as in method 2), in case the object's flag says it's some array's responsibility, and in case multiple array copies exist. We have to know which array is responsible for the objects.

4.     Extend the idea of method 3 and, instead of using a boolean flag, use a counter to count how many references are made to the object. When this count goes to zero, the object should be destroyed. With this method, no flags are needed in the arrays themselves.

Method 4 is the reference counting scheme mentioned in Chapter 3. In that chapter, we didn't actually implement reference counting, so we'll do this next.

 REFERENCE COUNTING

Reference counting works by storing a counter with each object. Each time the object is referenced by another entity (such as an array or perhaps some other object), the counter is incremented. When an entity is finished with the object, the counter is decremented. When the counter goes to zero, it means no entities are referencing the object, so it can be deleted. Thus, one way to safely implement heterogenous arrays is to point only to objects that are reference counted.

You can implement refererence counting in two basic ways:

1.     Embed a reference count directly into each object. This works if you haven't defined your classes yet or don't mind modifying existing classes.

2.     Indirectly associate each object with a reference count. This method is for those cases where you don't want to (or can't) modify your existing classes.

We'll investigate both of these methods next.

 REFERENCE-COUNTED OBJECTS

The simplest and most efficient way to implement reference counting is to directly embed a reference count into each object. This suggests a single-rooted hierarchy, where every object type is derived from a base class that sets up the reference counting. For example, we can define a CountedObj class for this purpose:

class CountedObj {

protected:

unsigned refcnt;

public:

CountedObj()           { refcnt = 1; }

virtual ~CountedObj()  { }

void IncRef()          { refcnt++; }

void DecRef()          { refcnt-; }

int UnReferenced()     { return refcnt == 0; }

};

From this class, we derive all the other types of objects we wish to have reference counted. In the case of the existing Shape and Circle classes given earlier, one simple modification will suffice--merely derive Shape from CountedObj:

class Shape : public CountedObj {

...

};

Now Shape and Circle objects have embedded reference counts. However, we have yet to show how these reference counts get manipulated. That's the responsibility of whatever uses the objects. In the case of heterogenous arrays, where pointers to objects are used, the responsibility falls on the pointers.

 SMART POINTERS

To handle pointers to reference-counted objects, we'll define a new type of pointer called smart pointers. These work much like normal pointers, except they keep track of the number of references to the objects they point to. The following SmartPtr class implements smart pointers as a concrete data type.

template<class TYPE>

class SmartPtr {

protected:

TYPE *objptr; // Actual pointer to object

void Bind(const SmartPtr<TYPE> &p);

void Unbind();

void NewBinding(const SmartPtr<TYPE> &p);

public:

static TYPE null_obj;

SmartPtr(TYPE *p=0);

SmartPtr<const SmartPtr<TYPE> &p);

SmartPtr<TYPE> &operator=(const SmartPtr<TYPE> &p);

~SmartPtr();

TYPE &operator*() const;

TYPE *operator->() const;

};

Complete code for the SmartPtr class is given in the file smartptr.h. (The class is completely inlined.)

Each smart pointer has one data member, objptr, that points to the object being referenced. Any type of object can be used as long as it is reference-counted and has the member functions IncRef( ), DecRef( ), and UnReferenced( ). The reference counter should be set to one by the type's constructor. As you might have suspected, any object type derived from the CountedObj class fits these requirements.

Constructing Smart Pointers

Binding Smart Pointers

Using Smart Pointers Like Ordinary Pointers

Arrays of Smart Pointers

Constructing Smart Pointers

To set up a smart pointer to an object, you must first allocate and initialize the object on the heap (with the reference count set to one), and then pass a pointer to the object in a call to the SmartPtr constructor. For example:

SmartPtr<Shape> ps(new Shape(1,1));

SmartPtr<Shape> pc(new Circle(2,3,4));

In the second constructor call above, pc can point to a Circle object--even though pc is typed to point to a Shape object--since Circle was derived from Shape. This important feature lends itself quite nicely to the creation of heterogenous arrays. For example, we can define a heterogenous array of SmartPtr<Shape> objects that point to both Shape and Circle objects. For simplicity, we use a built-in array:

SmartPtr<Shape> arr[2] = { new Shape(1, 2), new Circle(3, 4, 5) };

Figure 5.2 shows the memory layout of the array we've just constructed. Note that, when building the array, two implicit calls are made to the SmartPtr constructor. We'll look at that constructor now:

template<class TYPE>

SmartPtr<TYPE>::SmartPtr(TYPE *p)

{

if (p) {        // Do we have an object to point to?

objptr = p;   // Point to the object

}

else {          // No. Then point to the null object.

objptr = &null_obj;

objptr->IncRef();  // Must account for reference

}

}

The constructor works as follows: if the parameter p isn't a null pointer, we initialize objptr to reference the object pointed to by p. However, if the allocation of the object fails, p will be null. Rather than have objptr also be null, we point to the special static object null_obj, set aside for just this purpose. By using null_obj, we can guarantee that objptr will never be null. This greatly simplifies the rest of the SmartPtr code. If we do reference null_obj, we must increment its reference count, as shown in the constructor. If p isn't null, it's assumed that the reference count for the object pointed to by p is already set, and that this SmartPtr is currently the only reference to it.

The constructor can be used as a default constructor because p defaults to zero. By using the default constructor, you can have a smart pointer automatically set up to point to null_obj. This is useful in declaring an array without explicitly constructing each element. For example:

// Default SmartPtr constructor called 10 times below

Darray< SmartPtr<Shape> > myarr(10);

Notice how we're using a nested template declaration here, declaring a Darray object of SmartPtr<Shape> elements. In this example, each element of myarr will initially point to null_obj.

The static null_obj object must be allocated once somewhere in our program. Also note that we must have one of these objects for each type of smart pointer. For example, if we're going to use smart pointers to point to Shape objects and, let's say, Toaster objects, we would write:

// Allocate null objects for each type of smart pointer

Shape SmartPtr<Shape>::null_obj;

Toaster SmartPtr<Toaster>::null_obj;

Figure 5.2 A heterogenous array of smart Shape pointers.

Binding Smart Pointers

A smart pointer can be bound to an object by pointing objptr to it and then adjusting the reference count for the object. As you've seen, the constructors do this for us initially. But afterwards, we can bind to another object by using the following three routines:

template<class TYPE>

void SmartPtr<TYPE>::Bind(const SmartPtr<TYPE> &s)

// Assumes the object is not already bound

{

objptr = s.objptr;

objptr->IncRef();

}

template<class TYPE>

void SmartPtr<TYPE>::Unbind()

{

objptr->DecRef();

if (objptr->UnReferenced()) delete obj;

}

template<class TYPE>

void SmartPtr<TYPE>::NewBinding(const SmartPtr<TYPE> &s)

{

if (objptr != s.objptr) { // Prevents accidental deletion

Unbind() ;

Bind(s);

}

}

The Bind( ) routine sets objptr to point to the same object that some other smart pointer is using. Aliasing occurs at this point, which is recorded by incrementing the reference count for the object. The Unbind( ) routine does the reverse by decrementing the reference count. If the reference count goes to zero, the object is de-allocated by calling delete. The NewBinding( ) routine combines the actions of Bind( ) and Unbind( ) by first unbinding from the current object and then binding to a new one. Figure 5.3 shows an example of this process.

Figure 5.3 Binding to a new object.

The copy constructor, overloaded assignment, and destructor use the binding routines:

template<class TYPE>

SmartPtr<TYPE>::SmartPtr(const SmartPtr<TYPE> &p)

{

Bind(p);

}

template<class TYPE>

SmartPtr<TYPE> &SmartPtr<TYPE>::operator=(const SmartPtr<TYPE> &p)

{

NewBinding(p);

return *this;

}

template<class TYPE>

SmartPtr<TYPE>::~SmartPtr()

{

Unbind();

}

As is true with any normal pointer, copying one smart pointer into another causes only the pointer objptr to be copied, not the object being pointed to. However, smart pointers, unlike normal pointers, also keep track of the number of references made to the object. In the case of the copy constructor, a single call to Bind( ) does the trick. But for assignments, we must first unbind from the object currently being pointed to, and then bind to the new one. The NewBinding( ) routine does the work here.

When a smart pointer is destroyed, a call to Unbind( ) is made to decrement the reference count for the object being referenced. The object itself won't be deleted unless the reference count goes to zero.

Using Smart Pointers Like Ordinary Pointers

In Chapter 3, we espoused the virtues of using concrete data types. The idea is to be able to use a user-defined type as though it were built into the language. It would be convenient if we could use smart pointers like any other pointer. For example, we should be able to de-reference them as though they were ordinary pointers. We can do this by overloading the two operators '*' and '->':

template<class TYPE>

const TYPE &SmartPtr<TYPE>::operator*() const

{

return *objptr;

}

template<class TYPE>

TYPE *SmartPtr<TYPE>::operator->() const

{

return objptr;

}

When combined, these operators make a SmartPtr<TYPE> object appear to be a TYPE * object. For example, we can make a smart Shape pointer look like a Shape pointer, as in:

SmartPtr<Shape> sp = new Shape(1, 2);

// Two calls to operator'->'

sp->x = 42;

cout << "Area: " << sp->Area() << '\n';

Shape s = *sp; // Call to operator'*'

*sp = s;       // Illegal!

The last statement tries to do a wholesale assignment to a de-referenced Shape object via the operator*( ) function. This shouldn't be allowed because the embedded reference count will get set as well--greatly messing up the scheme of things. To disallow such dangerous statements, operator*( ) returns a reference to a constant TYPE, not just a TYPE. This means the object can only be read, not written to.

We didn't use the same const modifier on the operator->( ) function, however. There ought to be some way to change the underlying object's data! Unfortunately, this creates a loophole because, even though refcnt is protected, the functions IncRef( ) and DecRef( ) aren't. This means we can write corrupting code like:

sp->DecRef(); // Not cool!

Surprisingly, it can be difficult to fix this loophole. We would like to make the functions DecRef( ) and IncRef( ) private to only the classes that need access to them, such as the appropriate SmartPtr class. We could do this by declaring refcnt protected or private in the CountedObj class, and then declare SmartPtr as a friend. The problem is that SmartPtr is a template, and we can't specify all instantiations of a template as friends of a particular class. We must spell out the instantiations one at a time. For example:

class Shape;    // Forward class declarations

class Widget;

template<class TYPE> class SmartPtr; // Forward template declaration

class CountedObj {

private:

friend class SmartPtr<Shape>;

friend class SmartPtr<Widget>;

... // All other friends go here

unsigned refcnt;

public:

...

};

We've left the loophole in so that we don't have to go to all this trouble. So be careful!

To make smart pointers look even more like ordinary pointers, you might also want to overload the comparison operators '==' and '!='. We didn't do that in the SmartPtr class, but here's how operator==( ) and operator!=( ) might be defined as friend functions:

template<class TYPE>

int operator==(const SmartPtr<TYPE> &a, const SmartPtr<TYPE> &b)

{

return a.objptr == b.objptr;

}

template<class TYPE>

int operator!=(const SmartPtr<TYPE> &a, const SmartPtr<TYPE> &b)

{

return a.objptr != b.objptr;

}

Finally, you might also want to conveniently test whether a SmartPtr is "null" by overloading the '!' operator and providing an int type conversion operator. Here's how:

template<class TYPE>

int SmartPtr<TYPE>::operator!()

// Returns 1 if pointer references null_obj; otherwise, returns 0

{

return objptr == &null_obj;

}

template<class TYPE>

SmartPtr<TYPE>::operator int()

// Returns 1 if pointer doesn't reference null_obj; otherwise, returns 0

{

return rep != &null_obj;

}

...

SmartPtr<Shape> p(new Shape(1, 2));

if (!p) cout << "Allocation failed\n";

if (p) cout << "Allocation succeeded\n";

Arrays of Smart Pointers

We'll now take a closer look at smart pointers by using them with the user-defined Array classes created in Chapter 4. Here is a dynamic array of smart shape pointers:

typedef SmartPtr<Shape> SmartShapePtr;

// Declare a dynamic array of smart shape pointers

Darray<SmartShapePtr> arr(2);

With such a declaration, the two smart shape pointers stored in the array will reference the null shape object. We could assign them to new objects, as follows:

arr[0] = new Shape(42, 25);

arr[1] = new Circle(55, 17, 3);

Note that, when the assignments take place, the array elements are unbound from the null shape object, and then bound to the new objects just created.

The array elements can be used just as though they were pointers to Shape objects, as in:

for (int i = 0; i<2; i++) cout << arr[0]->Area() << '\n';

Since Area( ) is virtual, and we are using base class Shape pointers, the appropriate area of each shape is printed.

When you use an array of smart pointers, rest assured that the constructors and destructors for all referenced objects are called the correct number of times, even if the array is a variable-length, resizable array. This is true because both the Array and SmartPtr classes are designed to carefully handle construction and destruction of array element objects.

For example, suppose we define a static variable-length array of Shape pointers, and then add some elements. As a twist, we'll make the second and third elements point to the same Shape object. To understand what's going on, remember that all elements of a variable-length array start out unconstructed.

SVarray<SmartShapePtr, 5> shapes; // 5 unconstructed elements

// Construct the first two elements

shapes.Add(new Shape(1,1));

shapes.Add(new Circle(0, 0, 4));

// Create a third element bound to null shape object

shapes.Add(0);

// Now, bind it to same shape that the second element references

shapes[2] =  shapes[1];

Figure 5.4 shows the memory layout of the resulting array. As you can see from the figure, when one element is assigned to another, only the pointers are assigned--not the objects being pointed to. Since the overloaded SmartPtr<Shape> assignment operator is in use, the proper binding and unbinding takes place. For the same reason, copying between smart pointer arrays also works properly. This copying is very efficient, especially when the referenced objects are large.

Figure 5.4 Memory layout of a smart pointer array.

What is the cost of using smart pointers rather than ordinary pointers in an array? In both cases, heap space will be allocated for the target object of each array element. When smart pointers are used, each target object will also have one extra integer to store the reference count. Hence, that is the net overhead for using smart pointers over ordinary pointers. Normally, this overhead won't be noticeable, but it is there.

At this point, you might be concerned about the amount of code space required for the smart pointer class templates themselves. Note that the member functions are all very simple and can easily be inlined. The amount of code added by using smart pointers, while not zero, won't be too great for most applications.

Two test programs tstsp.cpp and tstsp2.cpp are provided on the disk to illustrate how smart pointers are used in both fixed-length and resizable heterogenous arrays of shapes.

 INDIRECT SMART POINTERS

The smart pointer technique that we've just shown works great--as long as you can directly embed a reference count into each object. But suppose you have an existing class that you want reference counted, and don't want to modify the class? One way to add the reference counting is by using indirect smart pointers.

Indirect smart pointers are like ordinary smart pointers except that one extra level of indirection is required to get to the object. Between the pointer and the target object is another object, which stores a reference count and a pointer to the target object. This intermediate object serves to make the target object look like a reference-counted object. Figure 5.5 illustrates the proposed setup.

Figure 5.5 Sample memory layout for indirect smart pointers.

Here is a class that can serve as the intermediate object type:

template<class TYPE> class iSmartPtr; // Forward template declaration

template<class TYPE>

class ObjRep {

// Assumes the object pointed to is stored on heap,

// except for null_obj

private:

friend class iSmartPtr<TYPE>;

unsigned refcnt;

TYPE *objptr;

ObjRep();

void IncRef();

void DecRef();

ObjRep(TYPE *p);

static TYPE null_obj;

};

template<class TYPE>

ObjRep<TYPE>::ObjRep()

// To be used only once to create a null ObjRep object

{

objptr = &null_obj;

refcnt = 1;

}

template<class TYPE>

ObjRep<TYPE>::ObjRep(TYPE *p)

{

objptr = p;

refcnt = 1;

}

template<class TYPE>

void ObjRep<TYPE>::IncRef()

{

refcnt++;

}

template<class TYPE>

void ObjRep<TYPE>::DecRef()

{

refcnt-;

if (refcnt == 0) delete objptr;

}

The name ObjRep stands for object representation. The class represents an ordinary object as though it were a counted object. You'll notice that ObjRep is very similar to the CountedObj class. The main difference is that the target object is not stored in the ObjRep object, but rather is pointed to by objptr. The parameterized constructor sets up objptr and also initializes the reference count. The DecRef( ) function is responsible for deleting the target object when the reference count goes to zero. Note that the CountedObj class does not delete the object, since it is the object! In this case, the deletion is left as a task for the iSmartPtr class.

Notice that we've declared a static member, null_obj. This is the target object to point to when we want a null ObjRep object. We do this for the same reason it was done in the SmartPtr class--to guarantee that we never have a real null pointer.

Having defined the ObjRep class, we can now show the iSmartPtr class, which is used to implement indirect smart pointers. Note the lowercase i in the prefix:

template<class TYPE>

class iSmartPtr {  // Indirect smart pointers

protected:

static ObjRep<TYPE> null_rep;

ObjRep<TYPE> *rep;

void Bind(const iSmartPtr<TYPE> &p);

void Unbind();

void NewBinding(const iSmartPtr<TYPE> &p);

public:

iSmartPtr(TYPE *p=0);

iSmartPtr<TYPE> &operator=(const iSmartPtr<TYPE> &p);

~iSmartPtr();

TYPE &operator*() const;

// Will only compile if TYPE is a structure

TYPE *operator->() const;

};

Complete code for the ObjRep and iSmartPtr classes is given in the file ismartptr.h. (The classes are completely inlined.)

The two classes ObjRep and iSmartPtr provide an example of cooperating classes. The ObjRep class gives the low-level data representation for reference-counted objects, while the iSmartPtr class provides the high-level interface. This technique of arranging two classes, one being a high-level wrapper around a low-level representation, is sometimes referred to as the letter-envelope idiom. See [Coplien 92]. Here, the iSmartPtr class is the "envelope" that wraps around the "letter" class ObjRep.

The iSmartPtr class is very similar to the SmartPtr class. The main difference is that it points to an ObjRep<TYPE> object rather than directly to a TYPE object. A static ObjRep object, null_rep, is declared (to be used in lieu of a null pointer). This is in addition to the null static TYPE object declared in the ObjRep class, which is in fact referenced by null_rep. Both of these objects must be allocated once somewhere in your program--one pair per type of indirect smart pointer. The following statements show how to do this for indirect smart Shape pointers:

// Allocate a null Shape object to be referenced by

// a null ObjRep object, also allocated

Shape ObjRep<Shape>::null_obj;

ObjRep<Shape> iSmartPtr<Shape>::null_rep;

The iSmartPtr functions are virtually the same as their SmartPtr counterparts, and, as such, many will not be shown here. The functions not shown can be defined by taking the corresponding SmartPtr functions and replacing objptr with rep. Here are the functions that require changes other than this:

template<class TYPE>

void iSmartPtr<TYPE>::Unbind()

{

rep->DecRef(); // ObjRep handles any object deletions

}

template<class TYPE>

iSmartPtr<TYPE>::iSmartPtr(TYPE *p)

}

rep = (p) ? new ObjRep<TYPE>(p) : 0;

if (rep == 0) {

rep = &null_rep;

rep->IncRef();

}

}

template<class TYPE>

TYPE &iSmartPtr<TYPE>::operator*() const

{

return *(rep->objptr);

}

template<class TYPE>

TYPE *iSmartPtr<TYPE>::operator->() const

// Will only compile if TYPE is a structure

{

return rep->objptr;

}

Note how the functions operator*( ) and operator->( ) return a pointer not to the object representation, but rather directly to the target object. Since the target object does not store the reference count, we don't have any unsafe loopholes with these functions. For that reason, operator*( ) returns a TYPE reference, instead of a const TYPE reference as was the case in the SmartPtr class. This allows assignments like the following to take place:

iSmartPtr<Shape> sp(new Shape(1,1));

Shape s(42, 17);

*sp = s; // Update the referenced shape object with a new shape

One potential problem with the operator->( ) function: it will only compile if TYPE is a structure. If TYPE isn't a structure, you'll have to comment the function out. (Too bad there's no way to tell at compile time whether a type is a structure!) Note that the operator->( ) function for SmartPtr doesn't have this problem, since in this case TYPE will always be a structure.

Arrays of Indirect Smart Pointers

Arrays of Indirect Smart Pointers

We'll now show what an array of indirect smart pointers looks like. The following array is the same as one we gave earlier for direct smart pointers, except now we use indirect smart pointers:

typedef iSmartPtr<Shape> SmartShapePtr;

SVarray<SmartShapePtr, 5> shapes; // 5 unconstructed elements

// Construct the first two elements

shapes.Add(new Shape(1, 1));

shapes.Add(new Circle(0, 0, 4));

// Create a third element bound to the null ObjRep<Shape> object

shapes.Add(0);

// Now, bind it to same shape that the second element references

Shapes[2] = shapes[1];

Figure 5.6 shows the resulting memory layout of the indirect smart pointer array. Compared to the use of ordinary pointers (or even to the use of direct smart pointers), indirect smart pointers cost more overhead. An extra heap allocation is needed for the intermediate ObjRep object, which stores the reference count and target object pointer. Note that, like the SmartPtr class, the iSmartPtr class and its associated ObjRep class can be completely inlined, so very little overhead occurs in creating the class itself.

Figure 5.6 Memory layout of a variable-length indirect smart pointer array.

Whether the overhead that results from using indirect pointers becomes a problem depends on how big the target objects are initially, and how many of them you try to store. In the case of large arrays of reference-counted integers, the overhead would be significant. If you need aliasing protection, try to use direct smart pointers if you can, since their overhead is much smaller.

Test programs for indirect smart pointers are given on disk in the files tstisp.cpp and tstisp2.cpp. These programs test both fixed-length and resizable arrays of indirect smart pointers.

 SMART POINTER SAFETY ISSUES

The SmartPtr class has a loophole where the reference count can be manipulated directly by using the operator->( ) function. Unfortunately, there is also another loophole, (which exists in the iSmartPtr class as well). When either a direct or indirect smart pointer object is constructed, a pointer to the target object to be referenced is passed. Although the target object is supposed to be allocated on the heap, nothing prevents you from passing the address of an object allocated statically. For instance, you might accidentally try:

Shape s(1, 1):

Shape> sp(&s); // Not a good idea!

Things will go haywire when the reference count for sp goes to zero because an attempt will be made to free the statically allocated object from the heap. A program crash will probably result. Note that both the SmartPtr and iSmartPtr classes violate this rule with their static null target objects. However, these objects are defined private to the classes, and can't otherwise be manipulated. Their reference counts should never go to zero--as long as everything is working correctly.

Another problem is the possibility of passing the address of the same target object to two different smart pointers, creating multiple reference counts for the object. Again, confusion will result. Here's an example of something you should never do:

Shape *p = new Shape(1, 1);

SmartPtr<Shape> sp1(p); // Okay

SmartPtr<Shape> sp2(p); // Not!

Keep in mind that the iSmartPtr class has these same problems.

 DERIVED SMART POINTERS

You can help prevent duplicate reference counts from being accidentally created for an object. The idea is to define a smart pointer class that handles the construction of the referenced object itself. For example, we could create a SmartShapePtr class, derived from SmartPtr, to work directly with Shape objects, as follows:

class SmartShapePtr : public SmartPtr<Shape> {

public:

SmartShapePtr(int x, int y);

};

SmartShapePtr::SmartShapePtr(int x, int y)

: SmartPtr<Shape>(new Shape(x, y))

{

// Nothing else to do

}

...

SmartShapePtr(0, 1); // Much higher level than before

Now there's no possibility for accidental duplicate reference counts. A SmartCirclePtr class could be coded similarly:

class SmartCirclePtr : public SmartPtr<Shape> {

public:

SmartCirclePtr(int x, int y);

} ;

SmartCirclePtr::SmartCirclePtr(int x, int y, int r)

// Note that we use the SmartPtr<Shape> class below, rather

// than SmartPtr<Circle> to save some template generation

: SmartPtr<Shape>(new Circle(x, y, r))

{

// Nothing else to do

}

Note that this same technique can also be used for the indirect smart pointer class iSmartPtr.

With this technique, we must construct, by hand, a smart pointer class for every type of object we wish to point to. This can quickly get tedious. We need constructors with different parameters for each object that we want to point to, but templates are not flexible enough to allow us to specify the various types of constructors. So, unless we restrict the types being used to always have constructors with similar arguments, we're stuck.

The result is this: if you want absolute safety, you must code separate smart pointer classes by hand. If you don't need absolute safety, or don't want to pay for it, you can use the SmartPtr and iSmartPtr classes by themselves.

Go to Chapter 6            Return to Table of Contents


Practical Data Structures: STRINGS

 Next Chapter Return to Table of Contents Previous Chapter

CHAPTER 6: STRINGS

Strings are perhaps one of the more useful type of arrays. Strings are variable-length sequences of characters that normally have defined operations such as concatenation (adding one string to the end of another), pattern matching, and the ability to form substrings from larger strings. They are useful mainly for representing data in human-readable form, and for this reason it's hard to imagine a language without strings. In this chapter, we'll look at a robust, efficient design for strings that's user-definable.

C++ does not have a string type per se, but instead provides library support for null-terminated character arrays, where the last character is a '\0' to signal the end of the array. Compared to strings in languages such as BASIC, null-terminated strings in C++ are quite primitive:

You must deal with arrays that hopefully have enough memory allocated for them, and you must ensure the last character has the value '\0'. This is, to say the least, quite error prone.

It's not possible to store arbitrary binary data in a null-terminated string, since '\0', a valid binary number, would end the string prematurely. A better way would be to store the length of the string in a string header. Figure 6.1 contrasts these two approaches.

There's no high-level way to assign one string to another, nor to do efficient substring and concatenation operations. For example, to add one string to another, you must scan completely down the first string to find the point of concatenation.

Figure 6.1 Two methods for representing strings.

A null-terminated string

String with stored logical and dimensioned lengths

A USER-DEFINED STRING DESIGN

As you might expect, you can get around the problems of null-terminated strings by defining your own strings. As always, you want a design that's efficient enough to be practical, yet at the same time high-level enough to be easily used. Toward this end, the String class given in this chapter has the following features:

Strings are implemented as a concrete data type, with the appropriate constructors, destructors, and overloaded operators defined.

Resizable, variable-length strings of arbitrary binary data are supported.

Concatenation, substring, pattern matching and filling operations are supported.

The strings use lazy copying. When doing string assignments or creating substrings, string text is shared. No copying takes place until one of the strings is modified.

The last feature is perhaps the most critical in the design of the String class. With lazy copying, it's possible to define substring and concatenation operations, safely and efficiently.

REFERENCE COUNTING REVISITED

Lazy copying implies that aliasing is occurring, which of course is something to be handled carefully. As we did with the smart pointers in Chapter 5, we can handle this aliasing by using reference counting. But instead of reference counting each array element, the elements of a string will be reference counted as a unit.

Strings can be defined with the same letter-envelope technique that we used for smart pointers, with two classes involved. The first class, StrRep, holds the string text and its associated reference count. The second class, String, provides the high-level interface and keeps track of which portion of the string text is used by a particular string. This layout of reference-counted strings is shown in Figure 6.2.

THE STRREP CLASS

The StrRep class is defined as follows:

class StrRep {

private:

friend class String;

unsigned refcnt; // Reference count for the text of the string

char data[1];    // Start of the string text

StrRep( );

void *operator new(size_t n, unsigned d);

void operator delete(void *p);

static StrRep null_rep; // For null strings

};

The peculiar thing about this class is that data, which is the start of the string text, appears to have room for only one character! However, the StrRep structure is actually meant to be allocated dynamically and, during that time, enough additional memory will be reserved to handle the desired maximum string length. This additional memory follows immediately after data. Figure 6.3 illustrates this layout.

We achieve this kind of dynamic allocation by overloading the new and delete operators for the StrRep class. Here is the new operator:

void *StrRep::operator new(size_t n, unsigned d)

// Here, n is sizeof(StrRep) and d is the desired

Figure 6.2 The layout of reference-counted strings.

// number of allocated characters in the string.

// If allocation fails, we reference null_rep.

{

// Note: n already accounts for one of the characters

void *p = new char[n+d-1];

if (p == 0) {

p = &null_rep;

((StrRep *)p)->refcnt++;

}

return p;

}

Normally, new takes just one argument: the size of the object to be allocated, in this case sizeof(StrRep). We've added a second argument, d, that indicates how long the text should be. The overloaded new operator then uses d to determine how much memory to actually allocate. The memory is allocated as though it were a simple array of characters. Keep in mind that the first portion of this memory will be occupied by the reference count. Here's how a call to new is made for StrRep objects:

// Allocate enough room for 10 characters

StrRep *p = new(10) StrRep;

Note the call to the constructor for StrRep after the new operator. The lexical ordering in this statement suggests what actually happens: first, new is called to allocate space for the object, then the object is constructed. The StrRep constructor is quite simple--it initializes the reference count to one. The text is left uninitialized, however:

StrRep::StrRep( )

{

refcnt = 1;

}

Figure 6.3 Dynamic allocation of a StrRep object.

Because of their special memory allocation scheme, StrRep objects must always be allocated dynamically. Otherwise, how would you allocate a text size greater than one? The exception to this rule is for the unique StrRep object, null_rep, that we've incorporated into our design. This object is a static member of the StrRep class and is thus statically allocated. As such, null_rep can hold only one byte of text. The purpose of null_rep is the same as used for the smart pointers in Chapter 5: to guarantee that a string will have somewhere to point to. Null strings always point to null_rep. You'll see later that this allows the String functions to quite elegantly handle null strings without doing any explicit testing.

The delete operator is responsible for deleting a StrRep object, which is pointed to by argument p in the delete( ) function.

void StrRep::operator delete(void *p)

{

delete[] (char *)p;

}

Note that we delete the object as though it were a character array, which is exactly how it was allocated in the first place by new. Also, note that delete( ) won't ever be called for null_rep because, supposedly, its reference count will never go to zero. Because all StrRep objects besides null_rep are dynamically allocated, we don't need to explicitly declare a destructor for StrRep. The overloaded delete operator handles everything.

The StrRep class doesn't actually handle many of the reference counting chores. As was true with smart pointers, the envelope class (in this case String) handles the binding and unbinding.

THE STRING CLASS

With the StrRep class defined, we can show the definition for the String class:

class String {

private:

StrRep *rep;

char *text;      // Pointer to start of string text

unsigned dimlen; // Maximum length of text without reallocation

unsigned len;    // Logical length of text

int grow_by;     // Minimum resizing increment

enum { defsize = 16, definc = 16 };

int Alloc(unsigned n, int gb);

void Bind(const String &s);

void Unbind( );

void NewBinding(const String &s);

public:

String(unsigned n = String::defsize, int gb = String::definc);

String(const String &s);

String(const String &s,

unsigned ofs, unsigned n, int gb = String::definc);

String(const char *s, int gb = String::definc);

String(const char *s, unsigned n, int gb = String::definc);

~String();

int IsNull() const;

int IsUnique() const;

String Clone() const;

int EnsureUnique();

String &operator=(const String &s);

void Copy(const char *s);

void Copy(const String &s);

String &operator=(const char *s);

#ifndef NO_RANGE_CHECK

unsigned CheckIndx(unsigned i) const;

#endif

char &operator[](unsigned i);

const char &operator[](unsigned i) const;

unsigned DimLength() const;

unsigned Length() const;

int GrowBy(unsigned amt);

int Grow();

int FreeExtra();

void ChgGrowByInc(int gb);

void SetLength(unsigned n);

int Realloc(unsigned new_dimlen, int keep=1);

void CopyN(const char *s, unsigned n);

unsigned InsReplAt

(unsigned p, const char *s, unsigned n, int ins=1);

unsigned DeleteAt(unsigned p, unsigned n);

char *Text();

const char *Text() const;

unsigned InsertAt(unsigned p, const char *s, unsigned n);

unsigned InsertAt(unsigned p, const char *s);

unsigned InsertAt(unsigned p, const String &s);

unsigned ReplaceAt(unsigned p, const char *s, unsigned n);

unsigned ReplaceAt(unsigned p, const char *s);

unsigned ReplaceAt(unsigned p, const String &s);

void Fill(const char c, unsigned ofs = 0, unsigned m = 0);

void Fill(const char *p, unsigned n, unsigned ofs=0, unsigned m=0);

void Fill(const char *s, unsigned ofs = 0, unsigned m = 0);

void Fill(const String &s, unsigned ofs = 0, unsigned m = 0);

const char *String::NullTermStr();

String Mid(unsigned p, unsigned n) const;

String Left(unsigned n) const;

String Right(unsigned n) const;

unsigned Find(char *s, unsigned ofs=0) const;

unsigned Find(unsigned n, char *s, unsigned ofs=0) const;

unsigned Find(const String &s, unsigned ofs=0) const;

void Concatenate(const char *s, unsigned n);

void operator+=(const String &s);

void operator+=(const char *s);

void operator+=(const char c);

friend String operator+(const String &a, const String &b);

friend int Compare(const String &a, const String &b);

friend int operator==(const String &a, const String &b);

friend int operator!=(const String &a, const String &b);

friend int operator>(const String &a, const String &b);

friend int operator>=(const String &a, const String &b);

friend int operator<(const String &a, const String &b);

friend int operator<=(const String &a, const String &b);

friend ostream &operator<<(ostream &strm, const String &s);

friend istream &operator>>(istream &strm, String &s);

};

The complete code for the String class--and it's associated StrRep class--is given on disk in the files strobj.h and strobj.cpp. A test program is given in the file strtst.cpp.

The String class is quite large, having more than 60 members. This is typical for a concrete data type. Ordinarily, you should try to avoid large classes, but many of the member functions here are used to make strings appear built-in to the language. Not all of the functions are strictly necessary, being mainly "convenience functions" that make it easier to work with strings. As a saving grace, many of these functions are simple enough to be inlined.

The String class has five data members: rep, text, len, dimlen, and grow_by. The most important is rep, which points to the associated StrRep object. Note that StrRep only keeps track of the actual text and the reference count for that text. It doesn't store the logical and dimensioned lengths, which is the reponsibility of the String class. Since the strings are to be resizable, we have members such as len, dimlen, and grow_by, which are used exactly as they were in the RVarray class of Chapter 4.

This brings up an important point. Since strings are to be resizable, why not implement them using an RVarray<char> class? In fact, we could. For instance, StrRep could be defined as:

class StrRep {  //  Alternative representation

private:

unsigned refcnt;

RVarray<char>;

...

};

We chose to use a more direct representation for several reasons: First, if we were to use an RVarray object, the memory layout for strings would be more complicated than necessary. It would be composed of the String object, the StrRep object, and the RVarray object inside of the Strep object, which has a pointer to the actual text. That's three separate pieces to be allocated on the heap, rather than just two with the more direct implementation. Also, the RVarray class is more general than we need, since it calls constructors and destructors for array elements, which in this case are characters that don't have constructors or destructors. (We can circumvent the latter problem by overriding the appropriate placement constructor and destructor templates to do nothing.)

Thus, even though the RVarray class could be used, it's probably better to implement a more direct design for strings. The code will be smaller and easier to understand, and the resulting String objects will be more compact. Having made these points, we should also tell you that the String class was implemented originally using an RVarray<char> class, and then this code was optimized specifically for character arrays. You'll notice that some of the member functions of String are similar, if not identical, to their RVarray counterparts. These functions include Realloc( ), InsReplAt( ), DeleteAt( ), Grow( ), GrowBy( ), FreeExtra( ), Length( ), and DimLength( ). Most of these functions will not be repeated here.

The last data member of the String class, text, is a character pointer to the start of the actual data for the string. Normally, text is set to point to the beginning of the data member of the corresponding StrRep object. However, for substrings, text will actually point to some offset from data. It wouldn't be strictly necessary to store a pointer here. We could use an integer representing the offset. However, pointing directly using text makes operations such as subscripting simpler and faster. Contrast these two logically equivalent functions:

const char &operator[](unsigned i) const

// One way to do subscripting

{

return rep->data[i+ofs]; // Ofs = where substring starts

}

const char &operator[](unsigned i) const

// Another, simpler way using a pre-nitialized text

// pointer to the start of the substring

{

return text[i];

}

The idea is to initialize text once when the string is built, with a statement such as:

text = rep->data[i+ofs];

Then, subsequent string operations don't have to keep adding the offset to find the start of the text. Figure 6.4 shows how to use the text pointer for a substring.

The String class has functions such as Bind( ), Unbind( ), and NewBinding( ) to handle the reference counting needed when binding and unbinding text to a string. These functions are similar to those used with the smart pointers of Chapter 5, and are called by the higher-level constructor, destructor, and assignment functions.

 CONSTRUCTING STRINGS

A full complement of constructors is defined for the String class. You can create strings with a specified starting size and growth increment, create strings that copy data from null-terminated strings, and create strings from raw memory bytes. Here are a few of the constructors:

String::String(unsigned n, int gb)

// Constructor to create a string of allocated

// length n, and a grow_by increment of gb.

{

Alloc(n, gb) ;

}

String::String(const char *s, int gb)

// Creates a string by copying bytes from a null-

// terminated array, with a grow_by increment of gb.

{

unsigned slen = strlen(s);

unsigned dlen = slen;

if (gb) { // Compute next highest multiple of gb

unsigned addval = (dlen % gb) ? gb : 0;

dlen = (dlen / gb) * gb + addval ;

}

Figure 6.4 The string text pointer.

if (Alloc(dlen, gb)) CopyN(s, slen);

}

String::String(const char *s, unsigned n, int gb)

// Create a string of allocated length n, grow_by

// increment of gb, copying bytes from s

{

if (Alloc(n, gb)) CopyN(s, n);

}

The first constructor can actually be used as a default constructor, since both its arguments have defaults. An enumerated type, defined in String, holds a default initial size and growth increment:

enum { defsize = 16, definc = 16 } ;

You'll notice that this enumerated type has no name (that is, it's an anonymous enumerated type). You can use anonymous enumerated types like this to conveniently define and initialize integer-based constants in a class.

All three of the constructors shown call an auxiliary function, Alloc( ), which handles the initial allocation of the string object. Note the call to the overloaded StrRep new operator:

int String::Alloc(unsigned n, int gb)

// Allocate a new string of allocated size n, and

// a grow by increment of gb

{

rep = new(n) StrRep; // Allocate string rep

text = rep->data;    // Point to starting text element

len = 0;

if (IsNull()) { // Wasn't able to allocate space

dimlen = 1; // null_rep has one byte of text

grow_by = 0;

return 0; // Indicate failure

}

else {

dimlen = n;

grow_by = gb;

return 1; // Indicate success

}

}

If Alloc( ) isn't able to allocate space for the string, a null string is constructed by first referencing the null_rep object (handled by the call to the overloaded new operator), and then setting the grow-by increment to zero. This prevents the null string from growing, and also allows other string operations (such as concatenation and subscripting) to behave properly without any explicit checks for a null string.

Note the logical length is always set to zero--regardless of whether the memory was allocated or not. Thus, to use a string, you must explicitly add text to it. Two of the constructors do this by calling the CopyN( ) function, (given later).

The destructor is responsible for unbinding the string from its possibly shared text, and deleting the text if necessary. The Unbind( ) function does all the work:

String::~String()

// Destructor unbinds from the shared text

{

Unbind();

}

Here are some examples of constructing strings:

// Empty string with initial dimensioned length of 10,

// and a default growth increment

String a(10);

// Empty string with intial dimensioned length of 10

// that can't grow

String b(10,0);

// String containing a copy of a null-terminated string,

// with default growth increment

String s("This is a test");

// String containing a copy of some raw bytes, with a

// default growth increment

char bytes[7] = {34, 56, 7, 42, 17, 25, 55];

String t(bytes, 7);

// String that's a copy of another string

String u(t):

 STRING COPYING

Unlike the other constructors, the following copy constructor doesn't actually do any copying of string data. Instead, the new string is bound to the same data as the old string by calling Bind( ). In other words, the copy constructor uses share semantics:

String::String(const String &s)

//  Copy constructor, which uses share semantics

{

Bind(s);

len = s.len;

dimlen = s.dimlen;

grow_by = s.grow_by;

text = s.text;

}

What isn't shared between two strings are the bookkeeping variables len, dimlen, grow_by, and text. Separate copies are maintained for them. This keeps the strings independent in terms of the way they interpret the length of the text, the text's starting location, and how the strings may be resized. This independence is useful in creating substrings, as you are about to see.

The assignment operator is defined with the same share semantics as the copy constructor:

String &String::operator=(const String &s)

// Assigns one string to another. Checks for assignment

// to self. Uses share semantics.

{

if (this != &s) {

NewBinding(s);

len = s.len;

dimlen = s.dimlen;

grow_by = s.grow_by;

text = s.text;

}

return *this;

}

 CREATING SUBSTRINGS

To illustrate how two or more strings can have different interpretations of the same shared text, we'll look at how substrings are defined. The following constructor is at the heart of substring creation. Like the copy constructor, it uses share semantics. However, the substring's variables len, dimlen, grow_by, and text all have different values than the source string:

String::String(const String &s, unsigned ofs, unsigned n, int gb)

// A constructor to create a substring of string s.

// The substring shares its data with s, starting at

// offset ofs, and of length n. It has an allocated

// length of n, and a grow_by increment of gb.

{

Bind(s);

// Keep ofs and length in range

if (ofs > s.len-1) ofs = s.len-1;

if (n + ofs > s.len) n = s.len-ofs;

// Set max and logical bounds of the substring

dimlen = n;

len = n;

grow_by = gb;

text = s.text + ofs; // Compute starting text element

}

The only differences between a string and a substring is that the text pointer may start at some offset from data, and that dimlen doesn't have quite the same meaning for substrings as it does for normal strings. In normal strings, dimlen is the actual number of characters allocated in the corresponding StrRep object. For substrings, dimlen gives the maximum extent that the substring is allowed to cover over the underlying shared text.

Rather than call the substring constructor directly, the following convenience functions are defined to return left, mid, and right substrings of another string:

String String::Left(unsigned n) const

// Returns the left substring of length n

{

return String(*this, 0, n);

}

String String::Mid(unsigned p, unsigned n) const

// Returns a substring of length n, starting at

// index p

{

return String(*this, p, n);

}

String String::Right(unsigned n) const

// Returns a right substring of length n

{

if (n > len) n = len; // Trap possible overflow

return String(*this, len-n, n);

}

The substring functions call the substring constructor to do most of the work. Note that the gb parameter, which defines the grow_by increment, has a default value of String::definc for the substring constructor. The default is used because the substring functions don't specify this parameter.

In each substring function, the newly created substring is returned by value, which causes the object's copy constructor to be called. In the case here, this means two identical substrings are created, the second to be the one actually returned by the function. However, since the String copy constructor uses share semantics, very little is actually copied. The initially constructed substring is destroyed upon the function return, but because of the reference counting, the shared text lives on. Both the original source string and the returned substring have references to it.

These examples illustrate how to use the substring functions:

String s("Supercalifragilisticexpialadocius");

String Super = s.Left(5);

String fragil = s.Mid(9,6);

String docius = s.Right(6);

You might wonder what happens if you modify a substring. Do both the substring and original string get modified? Also, what happens if you make substrings overlap, as shown in Figure 6.5? Won't things get confused? The answer to all of these questions is "No," as we'll explain in the next section.

 IMPLEMENTING LAZY COPYING

Lazy copying allows strings that have been copied to share their text, until one of the strings gets modified. Only then does the text actually get copied. We'll use the term unique string for a string that has data with a reference count of one. Thus, we want to ensure a string is unique before modifying it. If it's not unique, a copy of its data is made. Figure 6.6 illustrates this process.

Figure 6.5 Overlapping substrings.

Figure 6.6 Updating shared string text.

The following functions allow us to take a string and make it unique:

int String::IsUnique() const

// Returns 1 if the string's text has refcnt == 1

{

return rep->refcnt == 1;

}

String String::Clone() const

// Return a copy of this string. The copy will

// be unique (with refcnt = 1).

// Note: If allocation of copy fails, a null

// string is returned.

{

String temp(dimlen, grow_by);

temp.CopyN(text, len);

return temp;

}

int String::EnsureUnique()

// Ensures that this string uniquely owns its string

// rep (ie. the reference count is 1).

// Might have to copy the string text to ensure this.

// Returns 1 if it can make the string unique, O if it can't (allocation

// for copy failed).

{

if (!IsUnique( )) {            //  Need to copy to make unique

String &c = Clone( );      //  Attempt to copy

if (c.IsNull( )) return 0; //  Couldn't copy

NewBinding(c);  // Bind to copy

text = c.text;  // Reset starting text pointer

}

return 1;

}

The EnsureUnique( ) function returns one if it was able to guarantee a unique string. Otherwise, it returns zero, which means it was unable to allocate enough space to make a copy of the string. The following InsReplAt( ) function shows how the EnsureUnique( ) function can be used:

unsigned String::

InsReplAt(unsigned p, const char *s, unsigned n, int ins)

// Inserts/replaces (depending on ins flag) the data pointed to

// by s into the string. Up to n elements are inserted/replaced

// (truncating if necesary), starting at position p (counted

// by zero). The string will grow in size if inserting and needed

// and grow_by != 0. The size it will grow to will be the next highest

// multiple of grow_by >= the size needed. If p >= len, then the

// data is concatenated onto the end. Returns number of elements

// inserted/replaced, or 0 if error occurs.

{

unsigned room, needed, addval;

if (ins) {

room = dimlen - len;

if (n > room && grow_by != 0) {

needed = (n -  room);

addval = (needed % grow_by) ? grow_by : 0;

needed = (needed / grow_by) * grow_by + addval;

// Grow the string by needed amount. Note that the

// growth may fail, in which case the string stays

// the same size, with data intact.

GrowBy(needed);

}

}

// Note: At this point, GrowBy( ) may have already made

// our string unique

if  (!EnsureUnique( )) return 0; // Can't update

// Don't allow gaps

if (p >= len) p = len; // We'll be concatenating

if (ins) {

// Keep things in range. May have to truncate.

if (n > dimlen - len) n = dimlen - len;

if (p < len) {

// Make room for inserted data somewhere in the middle

memmove(text+p+n, text+p, len-p);

}

}

else {

// Keep things in range. May have to truncate.

if (n > dimlen - p) n = dimlen - p;

}

// Copy in source; compute final length

memmove(text+p, s, n);

if (ins) {

len += n;

}

else {

if ((p+n) > len) len = p+n;

}

return n;

}

The InsReplAt( ) function is essentially the same as the InsertNAt( ) function used in the RVarray class, except this one also allows text replacement as well as insertions. Note how a check is made to ensure that the string is unique before modifying it. If the string can't be made unique, the function exits and returns a failure code of zero. The DeleteAt( ) function (not shown) uses a similar technique.

By placing calls to EnsureUnique( ) judiciously, it's possible to make the check for uniqueness in only a few places. For instance, InsReplAt( ) is a workhorse function called by many of the other string functions. To illustrate, the following CopyN( ) function copies raw bytes of data into a string. The CopyN( ) function in turn is called by two Copy( ) functions, and another form of overloaded assignment:

void String::CopyN(const char *s, unsigned n)

// Copies the data from s, resizing the string if n is

// larger than the dimensioned length, and grow_by != 0.

// The string is guaranteed unique as well.

// If the string can't be resized, it keeps its old size,

// and not all of the data will be copied.

{

len = 0;            // Cleverly set length to zero

InsReplAt(0, s, n); // and let InsReplAt() do all the work

}

void String::Copy(const char *s)

// Copies data from a null-terminated string into

// this object. The string might be resized if

// necessary and allowable.

{

CopyN(s, strlen(s));

}

void String::Copy(const String &s)

// Copies data from String s into this string.

// The string might be resized if necessary

// and allowable.

{

CopyN(s.text, s.len);

}

String &String::operator=(const char *s)

// Copy a null-terminated string into this object. The

// string might be resized if necessary and allowable.

{

Copy(s);

return *this;

}

Unlike the standard assignment operator (which assigns one String to another), this operator assigns a null-terminated string to a String object, and uses copy semantics rather than share semantics. (The latter would make no sense in this case). Here is an example:

String s(80);

s = "The answer is 42"; // Copied into s

 SUBTLE SUBSCRIPTING ISSUES

When using subscripts, we must be careful when we modify a string. We've defined an overloaded subscripting operator that allows us to treat a String like any other subscripted array. Since the returned element may be an lvalue(on the left-side of an assignment), the subscripting operation may modify the string. Thus, we must be careful to make the String unique before returning the element:

char &String::operator[](unsigned i)

// Subscripting that may update a character in the string,

// so we must make the string unique

{

if (EnsureUnique( ))

return text[CHECK(i)];

else return StrRep::null_rep.data[0];

}

What do we do if we can't make the string unique? The way operator[] is defined, we must return a reference to something. That something might be modified, so we reference the lone data byte in null_rep and allow it to be modified. This may not be the best approach. Instead, we might want to print an error message and exit the program. Note the call to the CHECK( ) macro. This is the same macro used by the Array class to invoke range checking on subscripts.

Here's an example showing the use of the subscript operator:

String s("I can't speel");

s[11] = 'l'; // Correct the spelling

When the last statement is executed, EnsureUnique( ) is called to ensure that the string data is unique before we modify the eleventh element. Note that EnsureUnique( ) is called even when the content of the string element is only being read, as in:

char c = s[11]; // s still is made unique

This situation is unfortunate, because it means an unnecessary copy of the string is made. Unfortunately, there's no way to detect it. About the only thing we can do is declare the string as a constant, and then provide a second operator[]( ) function meant for constant strings. This operator doesn't have to ensure that the string is unique. For instance:

const char &String::operator[](unsigned i) const

// Read only version of subscript operator.

// Note the two uses of the const keyword!

{

return text[CHECK(i)];

}

const String s("I can't speel"); // A constant string

// Invoke constant subscript operator

char c = s[11]; // Fine, just reading

s[11] = 'l';    // Can't do. Compiler error.

The critical part of this design is that the operator[ ]( ) function passes back a constant reference to the element, not just a reference. This is what disallows the updating.

 STRING CONCATENATION

String concatenation is another form of string updating, where you add a copy of one string to the end of another. The '+=' operator is overloaded to implement string concatenation. Three versions are supplied to allow you to add one String object to another, add a null-terminated string, or add a single character. The string grows in size if necessary:

void String::operator+=(const String &s)

// Adds String s to the end of this string

{

Concatenate(s.text, s.len);

}

void String::operator+=(const char *s)

// Adds a null-terminated string s onto the end of

// this string. The null byte isn't added.

{

Concatenate(s, strlen(s));

}

void String::operator+=(const char c)

// Adds a single character to the end of the string

{

Concatenate(&c, 1);

}

All of these functions call the following Concatenate( ) function, which in turn calls the InsReplAt( ) function to do all the work (including ensuring uniqueness):

void String::Concatenate(const char *s, unsigned n)

// Adds n bytes from s onto the end of this string

{

InsReplAt(len, s, n);

}

The '+' operator is also overloaded to add two strings together, resulting in a third string. The following overloaded '+' operator cleverly lets the '+=' operator do most of the work:

String operator+(const String &a, const String &b)

// Adds two strings together and returns the result

{

String r(a.text->Length( ) + b.text->Length( ));

r += a;

r += b;

return r;

}

Note how a string is constructed to be big enough to hold the result, and how this result is returned from the function by value. (That's the only way we can do it.) Again, because of lazy copying, only a minimum amount of data is actually moved during the function return process.

Here are some examples of string concatenation and string addition:

String a("This is "), b("a test);

String c(80);

c = a + b;

c += " of the emergency broadcast system";

 STRING COMPARISONS

To make strings look as much as possible like built-in types, all of the comparison operators are overloaded. Here's how two of them are defined:

int operator==(const String &a, const String &b)

{

return Compare(a, b) == 0;

}

int operator>=(const String &a, const String &b)

{

return Compare(a, b) >= 0;

}

The other comparison operators are defined similarly. They all call the following Compare( ) function, which does the actual work:

int Compare(const String &a, const String &b)

// Uses unsigned decimal value of characters for collating order.

// Returns -1 if a < b, 0 if a == b, and 1 if a > b.

{

unsigned an = a.text->Length( );

unsigned bn = b.text->Length( );

unsigned sn = (an < bn) ? an : bn;

unsigned char *ap = (unsigned char *)a.text->Data( );

unsigned char *bp = (unsigned char *)b.text->Data( );

for (unsigned i = 0; i<sn; i++) {

if (*ap < *bp) return -1;     // a < b

if (*ap++ > *bp++) return 1;  // a > b

}

// Equal unless lengths are different

if (an == bn) return 0;

if (an < bn) return -1;

return 1;

}

These examples show the use of string comparisons:

String a("abc"), x("xyz"), s("ab");

if (a < x) cout << "Yes"; else cout << "No";  // Returns "Yes"

if (a == s) cout << "Yes"; else cout << "No"; // Returns "No"

 PATTERN MATCHING AND FILLING

Three Find( ) functions are defined to search for the starting index of a pattern in a string. The pattern can be another String object, a null-terminated string, or a sequence of raw bytes of a specified length. Here's how Find( ) is defined for null-terminated pattern strings. The others are defined similarly:

unsigned String::Find(char *s, unsigned ofs) const

// Returns index of first occurrence of pattern s

// (assumed to be null-terminated) beginning at

// offset ofs. Returns Oxffff if not found, or

// offset not in bounds.

{

char *q = text + ofs; // Start of string data

char *p = q;          // Next string element ptr

char *t = s;          // Next pattern element ptr

unsigned i = ofs;     // Next string element index

while(i<len && *t) {

if (*p == *t) {

t++;

if (*t == 0) return i; // We found it!

p++ ;

}

else {

i++;

q++;

p = q;

t = s;

}

}

return Oxffff;

}

Note that a value of 0xffff is returned if no match is found. (We can't use a flag such as -1 since the indexes are unsigned.)

An optional ofs parameter (which defaults to zero) can be defined to specify where to start the search in the string. The algorithm used to search for a pattern in a string is implemented in a brute force way. Other, more sophisticated methods are available to search for patterns in a string, as given in the companion volume [Flamig 93]. You could incorporate those algorithms into the Find( ) functions.

Here's an example that shows how to use Find( ) to locate all occurrences of the pattern "the" in a string:

String test("The cow jumped over the moon");

char pat[] = "the";

unsigned ofs = 0;

while((ofs = test.Find(pat, ofs)) != Oxffff) {

cout << "Pattern found at offset " << ofs << '\n';

ofs += strlen(pat); // Start at next possible spot in string

}

Also defined are four Fill( ) functions, which can fill a string--up to the dimensioned length--with a pattern that can be either another string, a null-terminated string, a single characer, or a pattern of raw bytes of a specified length. The latter function is used as the workhorse function for the others:

void String::

Fill(const char *p, unsigned n, unsigned ofs, unsigned m)

// Fills the string with pattern p of length n, up to length m

// or the dimensioned length of the string, whichever is smaller,

// starting at offset ofs. If m is 0, it means use the dimensioned

// length of the string. Pattern repeats if necessary. Does not

// cause the string to grow, but it may have to make a copy to

// ensure the string is unique.

{

if (dimlen == 0 || n == 0) return; // Nothing to do

if (!EnsureUnique( )) return;      // Can't fill

// Keep parms in range

if (m == 0) m = dimlen;

if (ofs >= dimlen) ofs = dimlen-1; // dimlen must be > 0!

if (m+ofs > dimlen) m = dimlen - ofs;

len = m+ofs;

char *t = text+ofs;

if (n == 1) {

// Use fast method for single character fills

memset(t, *p, m);

}

else {

// Multi-character pattern fills

unsigned np = n;

const char *s = p;

while(m) {

*t++ = *s++;

if (np == 1) {

np = n;

s = p;

} else np-;

m-;

}

}

}

As is true for all functions that modify a string, we must ensure the uniqueness of the string before filling it. Here's an example where we fill a string with a pattern up to the maximum length of the string:

String s(15);

s.Fill("1234567890"); // s = "123456789012345"

 DESIGN CRITIQUE

It's difficult to imagine how the String class could be implemented more robustly or efficiently. It successfuly defines strings as concrete data types and provides a lot more versatility and flexibility than null-terminated strings. At the same time, String objects are quite safe, due to the use of reference counting and the incorporation of a special null_rep object. (Although, as with anything in C++, it's possible to compromise that safety through typecasting trickery.)

At what cost does this versatility come? Analyzing the data structures, we see that each string has one StrRep pointer, a character pointer, and three integers to keep track of string lengths and resizability. In addition, each string has an associated StrRep object, which contains one integer and then the data itself.

In contrast, a null-terminated string has just the text and an extra null byte at the end. We see a net difference, then, of two pointers--and three integers minus a character. Don't forget that a StrRep object consumes heap space whether we want it to or not, while a null-terminated string does not require heap space. Partially offsetting this disadvantage is the fact that multiple String objects can share the same StrRep heap space. This makes heap usage less of a factor when a lot of substring operations take place.

The extra space requirement is the price paid for speed: String objects can be copied, concatenated, and made into substrings much more efficiently than null-terminated strings. In addition, finding the length of a String object involves a simple member reference, not a scan down the array as is the case for null-terminated strings. Once again, we see the almost inevitable speed versus space tradeoff. Here, the tradeoff is more than worth it. The String class is certainly efficient and robust enough for routine use.

Go to Chapter 7            Return to Table of Contents


Practical Data Structures: VECTORS AND MATRICES

 Next Chapter Return to Table of Contents Previous Chapter

CHAPTER 7: VECTORS AND MATRICES

Vectors and matrices are important components in mathematical computations. These are fixed-length 1-d and 2-d arrays, respectively, usually composed of numbers, with defined operations such as matrix multiplication, transposition, determinants, and so on. Vectors and matrices figure prominently in applications like coordinate transformations and in solving simultaneous equations. In this chapter, we'll look at how to create user-defined versions of these useful data structures.

Vectors and matrices have a special relationship. It's common to take 1-d slices of a matrix and call them vectors. For example, a row of a matrix is called a row vector. A column is known as a column vector. If the matrix is square (with the number of rows equal to the number of columns), you can think in terms of diagonal vectors, which span the diagonal of a matrix. Figure 7.1 shows a matrix with an example of each type of vector.

Figure 7.1 Vector types.

Since matrices are arrays, you might be tempted to implement them using the array-of-arrays technique described in Chapter 4. If you do, however, the result will be less than satisfactory--for several reasons. First, it's not possible to allow both dimensions to be determined at runtime. Second, even if you use the most compact statically-sized form, you will still have some unnecessary overhead. For example, each row of a 2-d array implemented with Sarray objects has its own length variable, virtual function table pointer, and pointer to its array elements, as illustrated in Figure 7.2. Most of these variables are redundant. You could keep just one of each for the entire matrix.

A worse problem lies in figuring out how to represent vectors associated with a matrix. Row vectors are easy, since each row is already an array. But what about column vectors, or diagonal vectors? There is no easy way to treat the elements of a column or diagonal as an array in its own right, since each element is stored in a separate row array. To further explore this fundam ental problem, we'll step back and take a look at how two-dimensional arrays are organized using the built-in C++ array types.

ROW-MAJOR AND COLUMN-MAJOR MATRICES

Consider the following two-dimensional 3x5 matrix:

float m[3][5];

In C++, this matrix is stored, row by row, as a 1-d array. Matrices stored in this way are called row-major matrices. Scanning down the 1-d representation, the column index varies the fastest, while the row index varies the slowest. Some languages, such as Fortran, use column-major matrices, where the matrices are laid out column by column. Here, the row index varies fastest, while the column index varies the slowest. Figure 7.3 shows the difference between these two types of organization.

Figure 7.2 A matrix implemented as an array of arrays.

The way a matrix is stored affects the way subscripting works. To find the ith row and jth column of a row-major matrix, the following subscript calculation is used:

m[i][j] <==> *(m + i*ncols + j); // For row-major matrices

For a column-major matrix, the calculation is:

m[i][j] <==> *(m + i + j*nrows); // For column-major matrices

In both cases, we need the size of only one dimension to compute the subscript. For row-major matrices, we need the number of columns, and for column-major matrices, we must know the number of rows.

The memory layout of a matrix directly affects what type of vectors can be easily obtained from the matrix. For a row-major matrix, it's easy to define row vectors, since the elements of each row are contiguous. Column vectors are more difficult to define, since the elements are non-contiguous. For column-major matrices, the opposite is true, with row-vectors being difficult to define and column vectors easy to define.

Actually, it's not all that difficult to define non-contiguous vectors. The secret lies in usng vector strides, our next topic.

ROW-MAJOR AND COLUMN-MAJOR MATRICES

Consider the following two-dimensional 3x5 matrix:

float m[3][5];

In C++, this matrix is stored, row by row, as a 1-d array. Matrices stored in this way are called row-major matrices. Scanning down the 1-d representation, the column index varies the fastest, while the row index varies the slowest. Some languages, such as Fortran, use column-major matrices, where the matrices are laid out column by column. Here, the row index varies fastest, while the column index varies the slowest. Figure 7.3 shows the difference between these two types of organization.

Figure 7.2 A matrix implemented as an array of arrays.

The way a matrix is stored affects the way subscripting works. To find the ith row and jth column of a row-major matrix, the following subscript calculation is used:

m[i][j] <==> *(m + i*ncols + j); // For row-major matrices

For a column-major matrix, the calculation is:

m[i][j] <==> *(m + i + j*nrows); // For column-major matrices

In both cases, we need the size of only one dimension to compute the subscript. For row-major matrices, we need the number of columns, and for column-major matrices, we must know the number of rows.

The memory layout of a matrix directly affects what type of vectors can be easily obtained from the matrix. For a row-major matrix, it's easy to define row vectors, since the elements of each row are contiguous. Column vectors are more difficult to define, since the elements are non-contiguous. For column-major matrices, the opposite is true, with row-vectors being difficult to define and column vectors easy to define.

Actually, it's not all that difficult to define non-contiguous vectors. The secret lies in usng vector strides, our next topic.

VECTOR POINTERS

It's possible to incorporate the ideas in the previous section into a new type of pointer, which we'll call vector pointers. These pointers are generalizations of ordinary pointers. An ordinary pointer always has a stride of one. That is, it's assumed the array elements are contiguous. Vectors may be non-contiguous (with evenly spaced elements, though), so a vector pointer can have a stride different than one. The following VecPtr class encapsulates these notions:

template<class TYPE>

class VecPtr {

private:

TYPE *p;               // Ordinary pointer

const unsigned stride; // Pointer stride

public:

VecPtr(TYPE *t, unsigned s=1)       : p(t), stride(s) { }

VecPtr(const VecPtr<TYPE> &s)       : p(s.p), stride(s.stride) { }

void operator=(const VecPtr<TYPE> &s) { p = s.p; }

void operator=(TYPE *t)               { p = t; }

void operator++()                     { p += stride; }

void operator++(int)                  { p += stride; }

void operator+=(unsigned n)           { p += stride * n; }

TYPE &operator[](unsigned i) const    { return p[stride*i]; }

operator TYPE *() const               { return p; }

TYPE &operator *() const              { return *p; }

#ifdef TYPE_IS_STRUCTURE

TYPE *operator->() const              { return p; }

#endif

};

This class exemplifies everything that's right about C++. It turns vector pointers into concrete data types that look virtually like ordinary pointers. The class is flexible in that, being a template, it can be used for any vector element type. Yet because the class is completely inlined, there is very little overhead (if any). This class can be as efficient as any hand-coded, low-level pointer implementation.

The basic purpose of the VecPtr class is to hide the stride calculations. For example, the overloaded '++' operator turns vector pointer incrementing into stride additions. This allows us to write code like:

for(int i = 0; i<n; i++) { *v = 0; v++; }

rather than

for(int i = 0; i<n; i++) { *v = 0; v += stride; }

The pointer de-referencing operators `*' and '->' are overloaded to make vector pointers look even more like ordinary pointers. For instance, the expression *v returns the vector element that is currently being referenced. Likewise, v->m returns the member m of the element being referenced. For this to work, TYPE must be a structure; otherwise, the class won't compile. We've commented out the operator->( ) function using ifdef statements in the class declaration.

The subscript operator is overloaded to allow vector pointers to be used like the name of an array. For example:

for(int i = 0; i<n; i++) v[i] = 0;

Of course, a loop like this won't be as efficient as one that uses pointer incrementing, since it takes an extra multiplication for the subscript operation.

Optimizing Vector Pointers

Optimizing Vector Pointers

Vector pointers can be further optimized. Note that, when a value is added to a pointer, more is going on than just the addition. In fact, a multiplication takes place! This is because the compiler must turn pointers into machine addresses, which work in byte units. Thus, given some pointer p, which points to an object of type T, we have the following equivalence:

p + i <==> ((char *)p) + i * sizeof(T)

Vector pointers can be optimized by pre-multiplying the vector stride by sizeof(T). The stride then becomes a byte stride. In pointer arithmetic, the pointer can be cast as a char *, which saves the extra multiplication. For example, the following functions of the VecPtr class can be optimized this way:

template<class TYPE>

class VecPtr {

...

VecPtr (TYPE *t, unsigned s=1) : p(t), stride(s*sizeof(TYPE)) { }

void operator++()                   { (char *)p += stride; }

void operator++(int)                { (char *)p += stride; }

void operator+=(unsigned n)         { (char *)p += stride * n; }

TYPE &operator[](unsigned i) const  {

return *((TYPE *)((char *)p + stride*i));

}

...

};

This change may speed up vector pointers by as much as 20 percent. Note that the optimization is completely encapsulated in the VecPtr class. Outside the class, you continue to work with pointer strides in their normal vector element units.

The following example shows how to use vector pointers with matrices. A matrix is created using built-in arrays and is then initialized to be a unit matrix (with all elements zero except on the diagonal, where ones are stored.)

float m[3][3]; // Declare a 3x3 row-major matrix

// Set all values to zero by pointing to matrix with

// a vector that can access all elements. A stride of

// 1 will do the trick.

VecPtr<float> va(m, 1);

for (int i = 0; i<9; i++) va[i] = 0;

// Set diagonal elements to 1 with another vector pointer

VecPtr<float> vd(m, 4); // Stride is ncols + 1

for (int j = 0; j<3; j++) vd[j] = 1;

The resulting matrix would be:

100

010

001

Pointer Increment Problems

One unfortunate limitation with vector pointers is the inability to use expressions like *p++. Why can't you use this? Consider what the expression translates into in terms of operation function calls:

*p++ <==> (v.operator++()).operator*()

First, the post-increment operator is called, and the pointer--as it existed before the increment--is returned. This pointer is then de-referenced to return the element. However, there is no way to return the original pointer before it has been incremented. Only a copy can be returned. On the surface, the following operator++( ) function seems to work:

VecPtr<TYPE> operator++(int)

{

VecPtr before (*this);  // Make copy of object

p += stride;            // Do increment

return before;          // Return copy

}

However, what happen in the following (admittedly concocted) expression?

(v++)++; // Undefined result

Such constructs are actually undefined. That's because the result of a post-increment operation is not an 1-Value (it can be assigned to). So it's really illegal to try and increment the result again. Note that pre-increment operation are 1-Values and don't have this problem.

We've forced both forms of operator++( ) to return void. That way, expressions like (v++)++ and *p++ won't compile, so there's no danger of receiving erroneous results. Due to the void returns, both the prefix and suffix forms of operator++( ) can be implemented using the same prefix semantics. There's no way to use the operators in illegal ways.

 DYNAMIC MATRICES

When you use the built-in array types, it's not possible to directly define matrices that have both their dimensions determined at runtime. Now that you know about pointers and strides, we can show you why this is true. In Chapter 4, we mentioned that, although the number of columns must be fixed, there is a way to make the number of rows variable. Here is one way to do it:

typedef float ArrNx5[5]; // Declare array-of-5-columns type

void f(unsigned n)

{

// Dynamically allocate nx5 matrix, referenced by p

ArrNx5 *p = new ArrNx5[n];

// Set lower-right element to 99

p[n - 1][4] = 99;

}

The variable p represents a pointer to the first element in the first row of the respective matrix. Even though this variable points to a floating point element, its type isn't float * or even float ** as you first might think. Instead, p is a pointer to floating point array of five columns. This can easily be seen by p's declaration.

A stride is implicit with a pointer to an array. In this case, p has the constant 5 as its stride. With this in mind, we have the following equivalence:

P[i] <==> *(p + i*5)

An ordinary pointer gets returned by subscript of a pointer to array. This pointer points to the first element of the ith row. The pointer can then be used in a subscript calculation to find the jth column of the row. Hence, double subscripts such as p[i][j] can be used.

Interestingly enough, the subscripting equation just shown is the same one used for vector pointer strides! The crucial difference is that, here, the stride is constant. Because of this, the compiler is able to easily plug in the constant stride in subscript and pointer increment calculations.

Contrast this with variable strides (such as in a dynamic matrix with the number of columns determined at runtime). Here, the compiler could not simply plug in values for the stride, because it doesn't know what the stride should be! The compiler would be forced to create a special data structure that carries the stride around with the pointer. This approach ought to sound familiar to you, since that's exactly what vector pointers do!

The conclusion is this: if you want to have dynamic matrices with both dimensions determined at runtime, you need to have vector pointers. Since C++ doesn't have vector pointers built in, it isn't able to handle dynamic matrices directly. However, nothing prevents you from creating your own dynamic matrices, as you'll now see.

A Simple Dynamic Matrix Class

From the preceding discussions, two basic techniques emerge for defining dynamically sized matrices:

1.     Represent the matrix as a one-dimensional array, using row-major or column-major order (row-major ordering is easiest). Allocate this array dynamically.

2.     Use vector pointers to conveniently access rows, columns, and diagonals.

The following SimpleMatrix class uses these techniques:

template<class TYPE>

class SimpleMatrix {

private:

TYPE *data;

unsigned nrows, ncols;

public:

SimpleMatrix(unsigned nr, unsigned nc);

SimpleMatrix(const SimpleMatrix<TYPE> &m);

~SimpleMatrix();

void Copy(const SimpleMatrix<TYPE> &m);

SimpleMatrix<TYPE> &operator=(const SimpleMatrix<TYPE> &m);

TYPE *operator[](unsigned r);

VecPtr<TYPE> RowPtr(unsigned r=0);

VecPtr<TYPE> ColPtr(unsigned c=0);

VecPtr<TYPE> DiagPtr();

VecPtr<TYPE> PtrToAll();

const TYPE *operator[](unsigned r) const;

VecPtr<const TYPE> RowPtr(unsigned r=0) const;

VecPtr<const TYPE> ColPtr(unsigned c=0) const;

VecPtr<const TYPE> DiagPtr() const;

VecPtr<const TYPE> PtrToAll() const;

unsigned NRows() const;

unsigned NCols() const;

unsigned NElems() const;

};

Note     The complete code for the SimpleMatrix class is given on disk in the files simpmat.h and simpmat.mth. Test programs for this class are given in the files tstsmat.cpp and tstsmat2.cpp.

This class isn't a complete implementation of matrices. Its purpose is to show you the basic design and to point out some performance and safety problems. Later in this chapter, we'll create complete Vector and Matrix classes that are more robust.

The SimpleMatrix class stores the matrix data in a row-major 1-d array, pointed to by the member data, as shown in the following constructor:

template<class TYPE>

SimpleMatrix<TYPE>::SimpleMatrix(unsigned nr, unsigned nc)

// General constructor

{

nrows = nr; ncols = nc;

data = new TYPE[nrows*ncols];

}

The size of the matrix is stored in the variables nrows and ncols, which are also used in calculating vector strides. The functions RowPtr( ), ColPtr( ), and DiagPtr( ) all return vector pointers of their respective types:

template<class TYPE>

VecPtr<TYPE> SimpleMatrix<TYPE>::RowPtr(unsigned r)

// Return a row vector pointer to row r

{

return VecPtr<TYPE>(data + r*ncols, 1);

}

template<class TYPE>

VecPtr<TYPE> SimpleMatrix<TYPE>::ColPtr(unsigned c)

// Return a column vector pointer to column c

{

return VecPtr<TYPE>(data+c, nrows);

}

template<class TYPE>

VecPtr<TYPE> SimpleMatrix<TYPE>::DiagPtr()

// Return a diagonal vector pointer. In case matrix

// isn't square, the smallest dimension is used to

// determine the diagonal stride.

{

unsigned dstride = ((nrows > ncols) ? ncols : nrows) + 1;

return VecPtr<TYPE>(data, dstride);

}

Also, the PtrToAll( ) function returns a vector pointer, with a stride of one, which allows you to access the matrix as though it were a one-dimensional array. This is useful for operations that apply to all elements, such as zeroing the matrix, scaling the matrix, and so on.

template<class TYPE>

VecPtr<TYPE> SimpleMatrix<TYPE>::PtrToAll()

// Return pointer to all elements of the matrix

{

return VecPtr<TYPE>(data, 1);

}

Here is an example that uses these functions:

SimpleMatrix<int> m(3, 3);

int i;

// Set all elements to '4'

VecPtr<int> va = m.PtrToAll();

for(i = 0; i<m.NElems(); i++) va[i] = 4;

// Set row 1 elements to '2'

VecPtr<int> vr = m.RowPtr(1);

for(i = 0; i<m.NCols(); i++) vr[i] = 2;

// Set col 1 elements to '3'

VecPtr<int> vc = m.ColPtr(1);

for(i = 0; i<m.NRows(); i++) vc[i] = 3;

// Set diagonal elements to '1'

VecPtr<int> vd = m.DiagPtr();

for(i = 0; i<m.NRows(); i++) vd[i] = 1;

The resulting matrix would be:

134

212

431

Two-Dimensional Subscripting

The SimpleMatrix class overloads the subscript operator [ ] to return an ordinary pointer to the first element in row r.

template<class TYPE>

TYPE *SimpleMatrix<TYPE>::operator[](unsigned r)

{

return data + r*ncols;

}

By using a subscript on the returned pointer as well, two-dimensional subscripting can be achieved using a double subscript, as illustrated in Figure 7.5, and in the following code:

SimpleMatrix<float> m(3, 4); // Declare 3x4 matrix

m[1][2] = 99;

Figure 7.5 Two-dimensional subscripting.

You might wonder why the operator[ ]( ) function doesn't return a row vector pointer. For instance, we could have written:

// Alternative definition for []

template<class TYPE>

VecPtr<TYPE> SimpleMatrix<TYPE>::operator[](unsigned r)

{

return VecPtr(data + r*ncols, 1);

}

This function is identical to RowPtr( ) (except for its name). Since row vectors themselves have subscripting defined, we can use the same double subscripting as we used earlier:

m[2][4] = 99; // Set lower-right element to 99

However, the result won't be as efficient as using the original form of the subscript function. The statement translates into:

// First call SimpleMatrix::operator[], then VecPtr::operator[]

(m.operator[](2)).operator[](4) = 99;

By substituting the bodies of the two different operator functions, we can transform the statement into the following equivalent pointer arithmetic:

*((data + 2*ncols) + 1*4) = 99;

Two multiplications take place: one to compute the start of row 2 and one to find column 4 of that row. Note that the multiplication 1*4 has to take place, even though just using the number 4 would suffice. That's because the number 1 represents the value stored in the stride variable for the row vector. Because a general vector pointer type is being used, we can't optimize for some vectors having a stride of 1. In the case where an ordinary pointer is returned, this optimization automatically takes place. A more efficient calculation results:

*((data + 2*ncols) + 4) = 99; // Optimized version

Tests show that returning an ordinary pointer from the matrix subscript function, rather than returning a vector pointer, yields speeds almost twice as fast for operations like matrix multiplication. Next, we'll show you how to define matrix multiplication.

Matrix Multiplication

Matrix multiplication is perhaps the most common operation done on matrices, and can be defined as follows: given two matrices a and b, one of size mxn and the other of size nxp, the result of multiplying the two matrices is a new matrix, c, of size mxp, where the elements of c are computed by the equation:

In other words, each row of a is paired with each column of b, and the corresponding elements of these vectors are multiplied and then summed. The sum becomes one element of the result matrix. Figure 7.6 illustrates this process.

The following function implements matrix multiplication:

void MatMult(SimpleMatrix<Number> &c,

const SimpleMatrix<Number> &a, const SimpleMatrix<Number> &b)

{

unsigned i, j, k;

for (i = 0; i < a.NRows(); i++) {

for (j = 0; j < b.NCols(); j++) {

Number sum = 0;

for (k = 0; k < b.NRows(); k++) sum += a[i][k] * b[k][j];

c[i][j] = sum;

}

}

}

We can also overload the '*' operator for matrix multiplication, as follows:

template<class TYPE>

SimpleMatrix<TYPE>

operator*(const SimpleMatrix<TYPE> &a, const SimpleMatrix<TYPE> &b)

{

SimpleMatrix<TYPE> r(a.NRows(), b.NCols());

MatMult(r, a, b);

return r; // Copy constructor called

}

Figure 7.6 Matrix multiplication.

This example multiplies two matrices:

SimpleMatrix<float> a(3, 2), b(2, 4), c(3, 4);

...

c = a * b;

Unfortunately, the operator "*" function leads to a lot of copying. First, the result matrix is created, the matrix multiply is performed by MatMult( ), and a copy of the result is returned. (We can't safely return a reference to the result.) Then, the assignment to c takes place, where yet another copy takes place. The Matrix class given later in this chapter avoids this problem.

As shown in MatMult( ), double subscripting is the obvious way to implement matrix multiplication. However, this implies a multiplication for each subscripting operation. Since we are accessing array elements in sequence, it might be faster to use pointer incrementing. Here's an alternative matrix multiplication function that uses vector pointers--and is roughly 25 percent faster than the double-subscripting method--for double-precision floating-point matrices:

template<class TYPE>

void MatMult1(SimpleMatrix<TYPE> &c,

const SimpleMatrix<TYPE> &a, const SimpleMatrix<TYPE> &b)

{

unsigned i, j, k;

VecPtr<const TYPE> ar = a.RowPtr();

VecPtr<const TYPE> ac = a.ColPtr();

VecPtr<const TYPE>br = b.RowPtr();

VecPtr<const TYPE>bc = b.ColPtr();

VecPtr<TYPE> cr = c.RowPtr();

VecPtr<TYPE> cc = c.ColPtr();

TYPE const *bstart = br;

for (i = 0; i < a.NRows(); i++) {

cr = cc; // Point to row i. Note: stride not copied

br = bstart;

for (j = 0; j < b.NCols(); j++) {

TYPE sum = 0;

ar = ac; // Point row i. Note: stride not copied.

bc = br; // Point column j. Note: stride not copied.

for (k = 0; k < b.NRows(); k++) {

sum += *ar * *bc;

ar++; // Next column

bc++; // Next row

}

br++; // Next column

*cr = sum;

cr++; // Next column

}

ac++; // Next row

cc++; // Next row

}

}

This function uses six vector pointers, two for each matrix. By incrementing a column vector pointer, we can advance to the next row of a matrix. Incrementing a row vector pointer advances us to the next column of the row. To understand MatMult1( ), it's important to realize that when vector pointer assignments take place only the underlying ordinary pointers are copied--not the strides. (You can verify this by looking at the definition for VecPtr given earlier.) Thus, we can safely assign a column vector pointer to a row vector pointer without confusing the strides.

There's a way to be even more efficient. Note that the row vectors always have a stride of one, so we can replace them by ordinary pointers. You may see speed improvements of 5 to 10 percent by doing this. The changes involved are:

Original Statement                   New Statement

VecPtr<const TYPE> ar = a.RowPtr();  const TYPE *ar = a.RowPtr();

VecPtr<const TYPE> br = b.RowPtr();  const TYPE *br = b.RowPtr();

VecPtr<TYPE> cr = c.RowPtr();        TYPE *cr = c.RowPtr();

You may wonder about the use of the const keywords in these statements. We'll look at these keywords next.

Constant Vectors and Matrices

True to the design principles explained in Chapters 2 and 3, we've made the VecPtr and SimpleMatrix classes work for constant objects. You'll notice that we supply two versions of the[ ]operator for SimpleMatrix:

template<class TYPE>

TYPE *SimpleMatrix<TYPE>::operator [](unsigned r)

// Called for non-const matrices

{

return data + r*ncols;

template<class TYPE>

const TYPE *SimpleMatrix<TYPE>::operator[](unsigned i) const

// Called for const matrices

{

return data + r*ncols;

}

The latter is a constant function, and is the one that's called for matrices declared to be constant. In this case, we shouldn't be able to modify any elements in the row that's returned, so a const TYPE * is returned (meaning a pointer to a constant), instead of just a TYPE*. We also use this strategy when returning vector pointers. For example, two ColPtr( ) functions are defined:

template<class TYPE>

VecPtr<TYPE> SimpleMatrix<TYPE>::ColPtr(unsigned c)

// Called for non-const matrices

{

return VecPtr<TYPE>(data+c, nrows);

}

template<class TYPE>

VecPtr<const TYPE> SimpleMatrix<TYPE>::ColPtr(unsigned c) const

// Called for const matrices

{

return VecPtr<const TYPE>(data+c, nrows);

}

When returning a column vector from a constant matrix, we must somehow prevent the elements of the vector from being modified. To do this, we type the vector pointer to reference const TYPE elements, rather than TYPE elements. Note this is not the same as making the vector pointer itself constant. For instance, had we written const VecPtr<TYPE> instead of VecPtr<const TYPE>, we would have created a pointer that couldn't be modified. Thus, we couldn't do things like pointer incrementing, so the pointer would be somewhat useless.

Matrix Transposition

Another common matrix operation is to take the transpose of a matrix. The B transposition is accomplished by interchanging the rows and the columns. Figure 7.7 shows an example.

The following Transpose( ) function shows the obvious implementation of a matrix transposition.

template<class TYPE>

SimpleMatrix<TYPE> Transpose(const SimpleMatrix<TYPE> &m)

{

SimpleMatrix<TYPE> t(m.NCols(), m.NRows());

for (unsigned i = 0; i<m.NRows(); i++)

for (unsigned j = 0; j<m.NCols(); j++) t[j][i] = m[i][j];

return t; // Copy constructor called.

}

Figure 7.7 Transpose of a matrix.

With this implementation, though, an extra copy takes place when the transposed version is returned. We could circumvent this problem by modifying Transpose( ) to assume that the result matrix has already been built, and to pass it as a parameter:

template<class TYPE>

void SimpleMatrix<TYPE>

Transpose(SimpleMatrix<TYPE> &t, const SimpleMatrix<TYPE> &m)

{

for (unsigned i = 0; i<m.NRows(); i++)

for (unsigned j = 0; j<m.NCols(); j++) t[j][i] = m[i][j];

}

Of course, this design assumes that you have created the result matrix with the proper size, an error-prone activity.

When it's acceptable to modify the matrix in place, there is another way to accomplish matrix transposition. Transposing a matrix is like changing it from row-major to column-major ordering. Instead of actually moving elements around, you can simply view the matrix as though it were column major. Here are the steps needed to make this work:

1.     Generalize the vector operations--such as RowPtr( ), ColPtr( ), and operator[ ]( )--to use row and column strides, rather than the number of rows and columns directly. Maintain the row and column strides in separate variables.

2.     Interchange the number of rows and columns, and interchange the row and column strides.

The basic idea here is to change what used to be row vectors into column vectors, and vice versa. In a row-major matrix, the row stride is one and the column stride is the number of columns in the matrix. If the matrix is transposed in place, the row stride becomes equal to the number of rows in the new matrix (which is the same as the number of columns in the original matrix), and the column stride becomes one. Assuming the variables rowstride and colstride were added to the SimpleMatrix class, here is a new in-place Transpose( ) function, along with modified RowPtr( ), ColPtr( ), and operator[ ]( ) functions:

template<class TYPE>

void SimpleMatrix<TYPE>::Transpose()

// Transpose a matrix in place

{

// Interchange nrows and ncols

unsigned temp = nrows;

nrows = ncols;

ncols = temp;

// Interchange row stride and column stride

temp = rowstride;

rowstride = colstride;

colstride = temp;

}

template<class TYPE>

VecPtr<TYPE> SimpleMatrix<TYPE>::RowPtr(unsigned r)

{

return VecPtr<TYPE>(data + r*colstride, rowstride);

}

template<class TYPE>

VecPtr<TYPE> SimpleMatrix<TYPE>::ColPtr(unsigned r)

{

return VecPtr<TYPE>(data + c*rowstride, colstride);

}

template<class TYPE>

VecPtr<TYPE> SimpleMatrix<TYPE>::operator[](unsigned r)

{

return VecPtr<TYPE>(data + r*colstride, rowstride);

}

Notice the generalized calculations used to determine the row and column vectors. Remember that, in these functions, either rowstride or colstride is equal to one. The other variable is equal to the number of columns in the original, non-transposed matrix. Another important point is that the operator[ ]( ) function can no longer return an ordinary pointer because there's no guarantee that a row vector will have a stride of one. The matrix might be transposed.

This technique eliminates the need to actually move the matrix elements around. Of course, this only works if you want to transpose a matrix in place. If you need to leave the original matrix alone, you must make a copy. In this case, doing the actual interchange between rows and columns is just as efficient. However, as we'll discuss next, a more sophisticated matrix class can be defined that eliminates the data movement, even when the original matrix is to be left untouched.

 MORE ROBUST VECTORS AND MATRICES

The SimpleMatrix class introduced you to techniques for defining dynamic matrices. However, some aspects of SimpleMatrix are not entirely safe, and can lead to a lot of redundant copying.

The SimpleMatrix class uses vector pointers as a fundamental part of its design. While vector pointers are fast and efficient, there is no way to do range checking on subscripts, since vector pointers do not keep track of the bounds of the vectors. This isn't necessarily a problem. Since range checking can easily reduce the performance of your code by 50 percent, you don't always want to use it. However, the capability should still be provided.

A more insidious problem is that, by their very nature, vector pointers induce aliasing. In the SmartPtr and String classes, aliasing was made safe by using reference counting. The same can be done for vectors and matrices. By allowing safe sharing, we can also eliminate a lot of the redundant copying that takes place.

Along these lines, we'll next present two classes, Vector and Matrix, that are extensions of the VecPtr and SimpleMatrix classes and that incorporate the techniques of range checking and reference counting. Fundamental to the design of these two classes is that, strangely enough, each matrix object is actually composed of a 1-d vector object! The purpose of the Matrix class is to interpret the vector data as though it were two-dimensional.

Because vector objects will use a shared representation, it follows that matrices will, as well. This shared representation allows us to easily support efficient submatrix and matrix transpose operations. As you study the following design, note how much overhead is caused by the shared representation, and how much that overhead is compensated for by more efficient matrix operations.

 THE VECTOR CLASS

The Vector class is similar in many ways to the String class described in Chapter 6. (You should review that chapter now if you haven't already read it.) The Vector class uses the same letter-envelope reference counting scheme, having an underlying vector representation class VecRep:

template<class TYPE> class Vector; // Forward declaration

template<class TYPE>

class VecRep {

private:

friend class Vector<TYPE>;

unsigned alloclen;

unsigned refcnt;

TYPE data[1];

VecRep(unsigned d);

void *operator new(size_t n, unsigned d);

void operator delete(void *p);

public: // So we can set up null_rep

static VecRep<TYPE> null_rep;

VecRep();

~VecRep();

};

The complete code for the VecRep class is given on disk in the files vector.h and vector.mth.

The VecRep class dynamically allocates the elements of the vector in the same way that the StrRep class allocates the text for strings: by overloading the new and delete operators (not shown here). The difference is that vector elements can be of any type, not just characters, and these types might have constructors and destructors. As is true in StrRep, the vector elements are initially allocated as characters so that we can easily keep the reference count next to the elements in memory. Thus, we must construct each vector element by hand from this array of characters. This task is handled by the VecRep constructor, using the same in-place construction technique covered in Chapter 4:

template<class TYPE>

VecRep<TYPE>::VecRep(unsigned d)

// Constructor to initialize a shared vector rep

{

if (this != &null_rep) {

// We only want to do the following if we're

// not referencing null_rep

alloclen = d;

refcnt = 1;

// We must call the default constructor for

// all but the first element

TYPE *q = data + 1;

for(unsigned i = 1; i<d; i++, q++) new(q) TYPE;

}

}

Note that the first element of the vector already has its constructor called implicitly by the VecRep constructor, because VecRep is defined with one element allocated. The VecRep destructor works exactly in reverse of the constructor:

template<class TYPE>

void VecRep<TYPE>::~VecRep()

// Destructor to explicitly destroy all elements

// of the vector, except the first

{

TYPE *q = data + 1;

for(unsigned i = 1; i<alloclen; i++, q++) q->TYPE::~TYPE();

}

With VecRep objects, we must track the number of elements in the vector so that we know how many times to call the constructor and destructor. To do this, the alloclen member is stored with each VecRep object.

As is true with strings, we use a null-vector representation called null_rep, which allows us to elegantly handle empty vectors--particularly those caused by memory allocation problems. Unlike all other VecRep objects that must be allocated dynamically, null_rep must be allocated statically. You must allocate one null_rep object per vector element type somewhere in your program. A macro is supplied to make this easier:

#define INITNULLVEC(TYPE) VecRep<TYPE> VecRep<TYPE>::null_rep;

With the VecRep class in hand, the Vector class can be defined:

template<class TYPE> class Matrix; // Forward declaration

template<class TYPE>

class Vector {

protected:

friend class Matrix<TYPE>;

VecRep<TYPE> *rep; // Pointer to shared vector data

TYPE *start;       // Pointer to logical start of data

unsigned len;      // Number of logical elements

unsigned stride;   // Stride (offset to next logical element)

int Alloc(unsigned n);

void Bind(const Vector<TYPE> &v);

void Unbind();

void NewBinding(const Vector<TYPE> &v);

public:

Vector(unsigned n = 0, const TYPE *s = 0);

Vector(const Vector<TYPE> &v);

Vector(const Vector<TYPE> &v, SliceType styp,

unsigned n=0, unsigned str=1, unsigned ofs=0);

~Vector();

#ifndef NO_RANGE_CHECK

unsigned CheckIndx(unsigned i) const;

#endif

TYPE &operator[](unsigned i);

const TYPE &operator[](unsigned i) const;

void CopyN(const TYPE *src, unsigned n);

void Copy(const Vector<TYPE> &v);

void Share(const Vector<TYPE> &v);

void SetElements(const TYPE &x);

Vector<TYPE> &operator=(const Vector<TYPE> &v);

Vector<TYPE> &operator=(const TYPE &x);

int IsNull() const;

int IsUnique() const;

Vector<TYPE> Clone() const;

int EnsureUnique();

unsigned Length() const;

unsigned Stride() const;

// Low-level hooks

VecPtr<TYPE> All();

VecPtr<const TYPE> All() const;

};

The complete code for the Vector class is given on disk in the files vector.h and vector.mth. Test programs for vectors are supplied in tstvec.cpp, tstvec2.cpp, and tstvec3.cpp.

The Vector class has four data members: rep, a pointer to the vector representation object; start, a pointer to the logical start of the vector elements; len, the logical length of the vector; and stride, the vector's logical stride. The start member is analogous to the text member of the String class. Normally, start points to rep->data, but in case the vector is a subvector, it may point at some offset from rep->data.

The Vector class has functions virtually identical to those in the String class to support lazy copying--such as Bind( ), Unbind( ), NewBinding( ), Share( ), EnsureUnique( ), Clone( ), and IsUnique( ). The main difference is that lazy copying isn't automatically performed in the Vector class. With the String class, we want to ensure that, when modifications are made to a string, the string is unique in order not to affect any other strings sharing the same text. With vectors, the design is just the opposite. Because vectors are going to be used to access shared slices of matrices, we want vector modifications to affect the underlying shared matrix data. Functions like EnsureUnique( ) are not used automatically, but are provided in case you do want to ensure a unique copy before modifying data.

The Vector class has three constructors. The first allocates a new vector representation of a specified length, and can optionally copy data from a low-level array into it:

template<class TYPE>

Vector<TYPE>::Vector(unsigned n, const TYPE *s)

// Constructor to allocate space for an n element vector,

// and, if s != 0, to copy low-level array into it.

// If n == 0 or allocation fails, we bind to null rep,

// and do no copying.

{

if (Alloc(n) && s) CopyN(s, n);

}

This constructor is also used as the default, with n and s defaulting to 0. In this mode, the vector created will reference the null vector representation null_rep, as shown in the following Alloc( ) function called by the constructor:

template<class TYPE>

int Vector<TYPE>::Alloc(unsigned n)

// Allocates a block of n elements for this vector.

// If n == 0 or allocation fails, we bind to the null rep.

// ASSUMES TYPE has a default constructor.

// Returns 0 if we've bound to null_rep otherwise, returns 1.

{

if (n) {

rep = new(n) VecRep<TYPE>(n);

}

else {

rep = &VecRep<TYPE>::null_rep;

rep->refcnt++;

}

start = rep->data;

if (rep != &VecRep<TYPE>::null_rep) {

len = n;

stride = 1;

return 1;

}

else {

len = 0;

stride = 1;

return 0;

}

}

The Alloc( ) function is similar to the one used in the String class, with the overloaded new operator for VecRep handling the dynamic allocation chores. Also, to handle any allocation problems, we use a zero-length vector that references null_rep. Note that new vectors initially have a stride of one, and that the vectors are fixed length (with the logical length set to the allocated dimensioned length). Here are some ways to create vectors:

Vector<float> v(10); // Create a new vector of 10 elements

Vector<int> w;       // Create a null vector

Like the String copy constructor, the Vector copy constructor actually uses share semantics:

template<class TYPE>

Vector<TYPE>::Vector(const Vector<TYPE> &v)

// Copy constructor that shares all of the vector v

{

Bind(v);

len = v.len;

stride = v.stride;

start = v.start;

}

The copy constructor's use of share semantics is all-important, since this allows us to return vectors from functions by value, without having to copy all of the vector elements.

The assignment operator also uses share semantics:

template<class TYPE>

Vector<TYPE> &Vector<TYPE>::operator=(const Vector<TYPE> &v)

// Share semantics used for assignment

{

if (this != &v) Share(v); // Note trap for assignment to self

return *this;

}

The Share( ) function is very similar to the copy constructor, except it calls NewBinding( ) instead of Bind( ) because the vector is presumably already bound to a vector representation:

template<class TYPE>

void Vector<TYPE>::Share(const Vector<TYPE> &v)

// Used to shares data with vector v

{

NewBinding(v);

len = v.len;

stride = v.stride;

start = v.start;

}

Subvectors

Copying Vectors

Accessing Vector Elements

 THE MATRIX CLASS

The extended Matrix class adds safe, efficient, matrix transposition and submatrices to the basic SimpleMatrix design, and is defined as follows:

template<class TYPE>

class Matrix {

protected:

Vector<TYPE> data;

unsigned nrows, ncols, rowstride, colstride;

public:

Matrix(unsigned nr=0, unsigned nc=0, const TYPE *s = 0);

Matrix(const Matrix<TYPE> &m);

Matrix(const Matrix<TYPE> &m, SliceType styp,

unsigned sr=0, unsigned sc=0,

unsigned nr=0, unsigned nc=0);

void Copy(const Matrix<TYPE> &m);

void Share(const Matrix<TYPE> &m);

Matrix<TYPE> Transpose();

Matrix<TYPE> &operator=(const Matrix<TYPE> &m);

Matrix<TYPE> &operator=(const TYPE &x);

#ifndef NO_RANGE_CHECK

unsigned CheckRow(unsigned i) const;

unsigned CheckCol(unsigned i) const;

#endif

Vector<TYPE> operator[](unsigned r);

const Vector<TYPE> operator[](unsigned r) const;

TYPE &operator()(unsigned r, unsigned c);

const TYPE &operator()(unsigned r, unsigned c) const;

Vector<TYPE> Row(unsigned r, SliceType styp=SHARED);

Vector<TYPE> Col(unsigned c, SliceType styp=SHARED);

Vector<TYPE> Diag(SliceType styp=SHARED);

Vector<TYPE> All(SliceType styp=SHARED);

const Vector<TYPE> Row(unsigned r, SliceType styp=SHARED) const;

const Vector<TYPE> Col(unsigned c, SliceType styp=SHARED) const;

const Vector<TYPE> Diag(SliceType styp=SHARED) const;

const Vector<TYPE> All(SliceType styp=SHARED) const;

// Mid-level hooks. Use these at your own risk:

VecPtr<TYPE> RowPtr(unsigned r=0);

VecPtr<TYPE> ColPtr(unsigned c=0);

VecPtr<TYPE> DiagPtr();

VecPtr<TYPE> PtrToAll();

VecPtr<const TYPE> RowPtr(unsigned r=0) const;

VecPtr<const TYPE> ColPtr(unsigned c=0) const;

VecPtr<const TYPE> DiagPtr() const;

VecPtr<const TYPE> PtrToAll() const;

int IsNull() const;

int IsUnique() const;

Matrix<TYPE> Clone() const;

int EnsureUnique();

unsigned NCols() const;

unsigned NRows() const;

unsigned RowStride() const;

unsigned ColStride() const;

int IsRowMajor() const;

int IsColMajor() const;

int IsSquare() const;

};

The complete code for the Matrix class is given on disk in the files matrix.h and matrix.mth. Test programs are provided in the files tstmat.cpp, tstmat2.cpp, tstmat3.cpp, and tstmat4.cpp.

The Matrix class has five data members. The elements of a matrix are stored in the 1-d Vector object data, which of course means the matrix data can be shared. This data is interpreted as a 2-d array, with nrows number of rows and ncols number of columns. The two other members, rowstride and colstride, keep track of the current row and column stride of the matrix. These members are useful when the matrix is transposed and also for submatrices, as you'll see. Figure 7.9 shows the layout of a typical matrix.

Figure 7.9 Layout of a typical matrix.

Constructing and Assigning Matrices

Creating Submatrices

Transposing Matrix Object

Extracting Vectors from a Matrix

Subscripting with the Matrix Class

Non-Contiguous Submatrices

Constructing and Assigning Matrices

The first Matrix constructor allocates and constructs a brand new matrix by calling the Vector constructor for data and then setting up the matrix size and strides:

template<class TYPE>

Matrix<TYPE>::Matrix(unsigned nr, unsigned nc, const TYPE *s)

: data(nr * nc, s)

{

if (data.IsNull()) {

nrows = 0; ncols = 0; colstride = 1; rowstride = 1;

}

else {

nrows = nr; ncols = nc; colstride = nc; rowstride = 1;

}

}

Note how the possibility of null matrices (due either to passing zero sizes or memory allocation problems) is handled properly.

The copy constructor and overloaded assignment operators use share semantics:

template<class TYPE>

Matrix<TYPE>::Matrix(const Matrix<TYPE> &m)

: data(m.data)

{

if (data.IsNull()) {

nrows = 0; ncols = 0; colstride = 1; rowstride = 1;

}

else {

nrows = m.nrows; ncols = m.ncols;

colstride = m.colstride; rowstride = m.rowstride;

}

}

template<class TYPE>

Matrix<TYPE> &Matrix<TYPE>::operator=(const Matrix<TYPE> &m)

// Share semantics used for assignment.

{

if (this != &m) Share(m); // Note trap for assignment to self

return *this;

}

template<class TYPE>

void Matrix<TYPE>::Share(const Matrix<TYPE> &m)

{

data.Share(m.data);

nrows = m.nrows; ncols = m.ncols;

colstride = m.colstride; rowstride = m.rowstride;

}

Here's an example of these functions:

float one_d_arr[4] = {0, 1, 2, 3};

Matrix<float> m(3, 5);            // Elements left uninitialized

Matrix<float> w(2, 2, one_d_arr); // Elements set to {{0, 1},{2, 3}}

Matrix<float> z(w);               // Copy constructor called

m = z;                            // m now reassigned to z's data

Creating Submatrices

The Matrix class can support submatrices, which SimpleMatrix couldn't do. These submatrices can either overlay an existing matrix (with the data being shared) or be separate copies. Figure 7.10 shows an example of overlaying a 3x5 submatrix onto a 6x7 matrix.

One of the Matrix constructors is used to construct submatrices. This constructor is in a sense the two-dimensional analog of the subvector constructor of the Vector class. The submatrix can either be shared or copied, depending on the value of styp:

template<class TYPE>

Matrix<TYPE>::Matrix(const Matrix<TYPE> &m, SliceType styp,

unsigned sr, unsigned sc,

unsigned nr, unsigned nc)

// Constructor that constructs a submatrix of matrix m

// If styp==SHARED, it means to share the data with m

// NOTE: If sharing, we initially create a null vector, and

// then we immediately rebind to shared vector. If copying,

// a row-major submatrix is created, and if nr == 0,

// then m.nrows is used. If nc == 0, then m.ncols is used.

: data((styp == SHARED) ?

0 : ((nr ? nr : m.nrows) * (nc ? nc : m.ncols)))

{

unsigned start_ofs, n;

if (nr) nrows = nr; else { nrows = m.nrows; sr = 0; }

if (nc) ncols = nc; else { ncols = m.ncols; sc = 0; }

if (styp == SHARED) { // Sharing

// When sharing, strides are always the same as parent strides

colstride = m.colstride;

rowstride = m.rowstride;

// Calculate length of underlying vector and offset.

// Remember: either colstride or rowstride == 1.

if (rowstride == 1) {

n = colstride * (nrows-1) + ncols;

}

else {

n = rowstride * (ncols-1) + nrows;

}

start_ofs = sr * colstride + rowstride * sc;

data.Share(Vector<TYPE>(m.data, SHARED, n, 1, start_ofs));

}

else {

if (data.IsNull()) {

nrows = 0; ncols = 0; colstride = 1; rowstride = 1;

}

else {

// Set up as a row-major matrix

colstride = ncols; rowstride = 1;

// Copy data from shared submatrix into allocated space.

// Note that the constructor is called recursively here

// to create a shared submatrix used for the copy.

Copy(Matrix<TYPE>(m, SHARED, sr, sc, nrows, ncols));

}

}

}

Figure 7.10 A submatrix of a matrix.

When making a submatrix copy, we use the same type of recursive call to the constructor that we used in the constructor for subvectors in the Vector class.

Here's an example of a 3x5 submatrix that shares its data with a 6x7 matrix, starting at row and column (2, 2), as illustrated earlier in Figure 7.10 :

Matrix<float> m(6, 7);

...

Matrix<float> subm(m, SHARED, 2, 2, 3, 5);

The last four parameters of the submatrix constructor, which define the origin and size of the submatrix, default to zero. In this case, the submatrix encompasses all of the original matrix. This approach can be used to create an actual copy of the matrix--which the copy constructor can't do--since it uses share semantics exclusively. For example:

Matrix<float> copy_of_m(m, COPIED);

The two variables rowstride and colstride provide the key to allowing a submatrix to share data with its parent. For normal row-major matrices, the column stride is always equal to the number of columns in the matrix. However, that isn't necessarily true for submatrices. Consider, for instance, the 3x5 submatrix shown earlier in Figure 7.10. You can see that its column stride should be the number of columns of the parent 6x7 matrix, and thus should be 7 instead of 5.

If one of the matrices is transposed or yet another submatrix is taken of a submatrix, the matter is further complicated. However, by judicious use of rowstride and colstride, the strides are never confused. Here's the basic rule: the row and column strides of a submatrix are always the same as their parents' strides and must always reflect the physical layout of the data. The real data lies somewhere up the chain of parents and is organized in row-major order. Of course, if one parent is transposed, the row and column strides will be interchanged.

Transposing Matrix Object

Data sharing allows us to define an efficient matrix transposition function for Matrix objects by merely interchanging nrows and ncols and interchanging rowstride and colstride:

template<class TYPE>

Matrix<TYPE> Matrix<TYPE>::Transpose()

// Returns a transposition of this matrix. Does not have to move

// the elements of the matrix around, just merely changes

// interpretation between row-major and column-major ordering.

{

Matrix<TYPE> t(*this); // Remember, copy constructor shares

// Interchange number of rows and cols

unsigned temp = t.nrows;

t.nrows = t.ncols;

t.ncols = temp;

// Interchange row stride and column stride

temp = t.colstride;

t.colstride = t.rowstride;

t.rowstride = temp;

return t;

}

Unlike the similar optimized Transpose( ) we showed for the SimpleMatrix class, this function does not need to assume that the matrix is to be transposed in place. Thus, the original matrix is untouched. Here's an example of this function:

Matrix<float> m(3, 5); // Create a 3x5 matrix

...

Matrix<float> t;       // Create a null matrix

t = m.Transpose();

Because both assignment and copy construction use share semantics, we can write statements like the last one without worrying about undue copying taking place. If the matrix being transposed is large, we can indeed avoid a lot of data movement.

Of course, you don't get something for nothing. Clever use of rowstride and colstride for matrix transposition does have its price. The vector extraction functions, to be given next, aren't as efficient as they could be; they can't assume anything about the vector strides. For instance, a row vector may or may not have a stride of one, so we can't use an ordinary pointer to it.

Extracting Vectors from a Matrix

As is true with the SimpleMatrix class, the functions RowPtr( ), ColPtr( ), DiagPtr( ), and PtrToAll( ) all are defined for the Matrix class. We'll refer to these as vector extraction functions. These functions use the modifications we suggested earlier--to handle the potential transposition of the matrix--by using rowstride and colstride rather than nrows and ncols directly. By using rowstride and colstride, the extraction functions also work for submatrices.

Here are the RowPtr( ) and ColPtr( ) functions for the Matrix class. The other extraction functions are defined similarly. Also note that const versions can be defined for constant matrices (not shown):

template<class TYPE>

VecPtr<TYPE> Matrix<TYPE>::RowPtr(unsigned r)

// Returns vector pointer to row r

{

return VecPtr<TYPE>(data.start+CHECKROW(r)*colstride, rowstride);

}

template<class TYPE>

VecPtr<TYPE> Matrix<TYPE>::ColPtr(unsigned c)

// Returns vector pointer to column c

{

return VecPtr<TYPE>(data.start+CHECKCOL(c)*rowstride, colstride);

}

In these functions, range checking is handled by calls to the macros CHECKROW( ) and CHECKCOL( ). These macros are the two-dimensional counterparts to the CHECK( ) macro used in the Array classes (described in Chapter 4), and are defined in the range.h header file. The macros do nothing if NO_RANGE_CHECK is undefined.

The range checking is done when the vector pointers are created. But remember that, when you use the vector pointers, nothing prevents you from going out of bounds. For this reason, another set of vector extraction functions are defined. These functions, which return fully range-checked Vector objects, are Row( ), Col( ), Diag( ), and All( ). Here are examples of Row( )and Col( ):

template<class TYPE>

Vector<TYPE> Matrix<TYPE>::Row(unsigned r, SliceType styp)

// Return a row slice of a matrix

{

return Vector<TYPE>(data, styp,

ncols, rowstride, CHECKROW(r)*colstride);

}

template<class TYPE>

Vector<TYPE> Matrix<TYPE>::Col(unsigned c, SliceType styp)

// Return a column slice of a matrix

{

return Vector<TYPE>(data, styp,

nrows, colstride, CHECKCOL(c)*rowstride);

}

Notice that, by using the styp parameter, the vectors returned can either share their data with the original matrix or can be separate copies. The default action is to share the data.

Subscripting with the Matrix Class

To return a row vector, the subscript operator is overloaded for the Matrix class. As such, operator[ ]( ) is the same as Row( ), except the returned vector always shares its data with the matrix:

template<class TYPE>

Vector<TYPE> Matrix<TYPE>::operator[](unsigned r)

// Row subscripting operator for non-const matrices.

// This routine does the same thing as Row(r, SHARED).

{

return Vector<TYPE>(data, SHARED,

ncols, rowstride, CHECKROW(r)*colstride);

}

Since vectors have their own subscript operator function defined, you can write code like the following:

Matrix<float> m(3, 5);

m[2][4] = 99; // Set lower-right element to 99

Thus, you can use the same double-subscripting syntax that's used for built-in arrays. There's only one problem. The SimpleMatrix class returned an ordinary pointer from the first subscript. Here, though, a full vector object is returned, and this object must be constructed. The construction process, while fairly efficient due to share semantics, still takes time. In fact, a matrix multiply routine that uses Matrix objects with double-subscripting is more than six times slower than its SimpleMatrix counterpart! And you'll only get this performance level when range checking is turned off. If you turn on range checking, the performance could be more than ten times slower.

You can circumvent this problem by defining operator[ ]( ) to return a vector pointer rather than a full vector. This will yield a more efficient subscript operation (perhaps only twice as slow as the fastest version), but you lose the ability to do range checking on the row vector. Of course, you can eliminate the subscripting altogether and instead use vector pointer incrementing. Doing so will yield speeds as fast as those for the SimpleMatrix class. This is an important benefit because you can obtain improved the performance even though you now have generalized matrices (such as transposed submatrices).

Another alternative is to define a different style of subscript operator. For example, you can overload the ( ) operator (normally used for function call syntax) to do two-dimensional subscripting:

template<class TYPE>

TYPE &Matrix<TYPE>::operator()(unsigned r, unsigned c)

{

return data.start[CHECKROW(r)*colstride + CHECKCOL(c)*rowstride];

}

...

Matrix<float> m(3, 5); // Construct matrix

...

m(2, 4) = 99;          // Set lower-right element to 99

The equation used in the operator( )( ) function is a generalization of the two-dimensional subscript equation used for row-major matrices. Normally, you multiply the row index by the number of columns and then add the column index, as in:

m(r, c) = *(m + r*ncols + c)

However, in the general case, the matrix might be transposed and might be a submatrix. Thus, we have to multiply both the row index and the column index by the appropriate row and column strides. Note that one of the strides will be one, so in effect we'll have either

m(r, c) = *(m + r*colstride + c) // If row major

or

m(r, c) = *(m + r + c*rowstride) // If column major

Thus, one of the multiplications in operator( )( ) is redundant. That's the price we pay for generality. Even so, expressions like m(i,j) have been timed to be two to three times faster than their m[i][j] counterparts.

The expression m(i,j) is actually the conventional notation used in mathematics for subscripting. Because of this, some people prefer to use the two-dimensional operator over its double-subscripting m[i][j] counterpart. Others prefer the more conventional C++ syntax for subscripting. Regardless of your personal preference, the mathematical notation will be significantly faster unless you define the [ ] operator for matrices to return a vector pointer rather than a full vector. In that case, the results are fairly even.

Non-Contiguous Submatrices

A matrix is said to be contiguous if you can access all of the elements using a single 1-d vector. Normal matrices created by the Matrix class will always be contiguous, regardless of whether they are transposed. However, its possible for submatrices that are shared with a parent to be non-contiguous. Figure 7.11 shows some examples of contiguous and non-contiguous submatrices, assuming a row-major organization.

From Figure 7.11, the following rule emerges for determining whether a submatrix is contiguous: either the number of columns must equal the column stride or the number of rows must be one. For column-major matrices (not shown in Figure 7.11), the number of rows must equal the row stride or the number of columns must be one. The function All( ), which attempts to return a 1-d vector to all the elements in a matrix, makes these tests for contiguousness. If the matrix isn't contiguous, All( ) returns a null vector because it isn't possible to access all the elements with a single vector:

Figure 7.11 Some non-contigous submatrices.

template<class TYPE>

Vector<TYPE> Matrix<TYPE>::All(SliceType styp)

// Returns a 1-d vector with all of the data for the matrix.

// If it isn't possible for the 1-d data to be contiguous

// (as will be the case for some submatrices), then a

// null vector is returned.

// ASSUMES either colstride or rowstride == 1.

{

if ((rowstride == 1 && nrows > 1 && ncols != colstride) ||

(colstride == 1 && ncols > 1 && nrows != rowstride))

return Vector<TYPE>(); // Return a null vector

return Vector<TYPE>(data, styp, nrows * ncols, 1, 0);

}

As an example of handling non-contiguous submatrics, here is an overloaded assignment operator that sets all the elements of the matrix to a specified value. This is done by doing the assignments row by row, using vector pointers:

template<class TYPE>

Matrix<TYPE> &Matrix<TYPE>::operator=(const TYPE &x)

// Set all elements in the matrix to the value x.

// Assumes a worst case non-contiguous submatrix.

{

// tc = column vector pointer to rows of t

// tr = row vector pointer to columns of t

VecPtr<TYPE> tc(data.start, colstride);

VecPtr<TYPE> tr(data.start, rowstride);

for (unsigned i = 0; i<nrows; i++) {

tr = tc; // Remember, strides aren't assigned!

for (unsigned j = 0; j<ncols; j++) {

*tr = x;

tr++;  // Next column

}

tc++;       // Next row

}

return *this;

}

 DESIGN CRITIQUE

In this chapter, you've seen in detail how to define generalized dynamic vectors and matrices. In our design, we were careful to remove all potential inefficiencies, without sacrificing generality. However, any time you make something general, performance is bound to suffer. This is manifested in basically one form: vector pointers must be used rather than ordinary pointers. Because of their generality, vector pointers with strides of one may be less efficient than ordinary pointers (depending on how well your compiler optimizes), even though the two are logically equivalent.

However, by using such tricks as vector pointers and reference counting, we can define certain operations to be much more efficient than the obvious ways would suggest. Consider, for instance, matrix transposition, which can be done by merely swapping some variables. Returning matrices from functions also becomes very efficient, making the code less clumsy. Never does an unnecessary copy of matrix elements take place.

There are some overriding questions: What is the cost of generality? Do the potential speed improvements offset the potential slowdowns? The answers, of course, depend on the particular application (and your compiler). However, in most non-trivial applications, the Vector and Matrix classes will probably be just as as efficient as simple, static built-in matrices.

An interesting test is to compare matrix multiply times for built-in statically sized matrices, SimpleMatrix objects, and Matrix objects. Table 7.1 shows the fastest times possible for all three types (using all the tricks described in this chapter). The results were obtained by multiplying two 30x30 double precision floating-point matrices together, 100 times, and taking the average. The tests were performed using 32-bit pointers on a 486 33-MHz machine.

As you might expect, statically sized matrices yielded the fastest results. Using a generalized matrix resulted in a 32 percent drop in speed, which isn't bad considering the flexibility that we gain. Interestingly enough, the general Matrix class performed slightly better than the SimpleMatrix class (probably due to some compiler optimization quirk). Their performance is close enough to suggest that the main performance penalty is due mostly to the use of dynamically sized matrices and not the fact that, in the case of the Matrix class, submatrices and efficient transposition are supported.

What is the space cost of generalized matrices? Figure 7.9 given earlier shows the memory layout of a Matrix object. Note that, in addition to the matrix elements themselves, a Matrix object has four data members plus a Vector object with four data members--one of which points to a VecRep object using two data members. Thus, an independent, unique matrix has an overhead of 10 data members (probably in the range of 20 to 40 bytes, depending on the implementation). This may be significant if you have a lot of small 3x3 or 4x4 matrices. However, for larger matrices, the overhead becomes less important, especially in light of the flexibility gained. And, of course, if the matrix data is shared, the overhead is reduced by two members plus the elements shared.

Table 7.1 Relative performance of different matrix types.

Type                      Time     Performance Relative to Fastest

of Matrix              in Seconds       (Smaller Is Better)

Statically sized

 built-in 2-d arrays      0.0470               1.0

SimpleMatrix objects      0.0660               1.40

Matrix objects            0.0621               1.32

Go to Chapter 8            Return to Table of Contents


Practical Data Structures: LINKED LISTS

 Next Chapter Return to Table of Contents Previous Chapter

CHAPTER 8: LINKED LISTS

Linked lists are the (usually) non-contiguous counterparts to arrays. Like arrays, linked lists are a fundamental type of data structure used to represent sequences of objects. In the first three chapters, you were given various examples of linked-lists. Here, we will take a detailed look at linked-lists, from basic design to some of the possible variations. The chapter culminates with a contiguous array-based linked list hybrid design.

WHY USE LINKED LISTS?

Before you begin studying the design of linked lists, you might be wondering what they are good for. Like arrays, linked lists are good for representing sequences of objects, but they have a different set of advantages, as shown in Table 8.1. Linked lists excel in allowing you to rearrange the sequences, and in allowing those sequences to grow and shrink. The tradeoff is that linked lists take more space per element than arrays.

Table 8.1 Tradeoffs between linked lists and arrays.

Feature               Arrays       Linked Lists

Sequential access     Efficient    Efficient

Random access         Efficient    Inefficient

Resizing              Inefficient  Efficient

Element rearranging   Inefficient  Efficient

Overhead per element  None         1 or 2 links

LINKED LIST BASICS

A linked list is composed of nodes, or elements, linked together. Conceptually, each node holds data, plus one or two links to other nodes. A list containing nodes with only one link is called a singly-linked list. Figure 8.1(a) shows an example. In a singly-linked list, each node points to the next node in the sequence. Because only a single pointer is used, singly-linked lists can only be efficiently processed in one direction.

A doubly-linked list node contains two links: one pointing to the next node and one to the previous node. Figure 8.1(b) shows an example. Doubly-linked lists can be efficiently processed both forwards and backwards. However, they require an extra link in each node.

A linked-list node can be represented as a simple structure holding the node's data and pointers for links to other nodes. Within this structure, it makes sense to also incorporate functions that aid in building lists. Examples are functions to insert and remove nodes. Next, we'll show you how to define both singly-linked and doubly-linked node classes, and how to construct lists from them.

Singly-Linked List Nodes

Doubly-Linked List Nodes

Singly-Linked List Nodes

Here is a typical definition of a singly-linked node class, written as a template holding information of type TYPE:

template<class TYPE>

struct Snode {

TYPE info;          // Holds the data for the node

Snode<TYPE> *next;  // Pointer to next node in the list

Snode(const TYPE &x);

void InsertAfter(Snode<TYPE> *n);

Snode<TYPE> *RmvNext();

};

The constructor initializes the info field by copying the parameter x and then sets the next pointer to zero. It's assumed that TYPE has a copy constructor or is a built-in type:

Figure 8.1 Singly-linked versus doubly-linked lists.

(a) Singly-linked list

(b) Doubly-linked list

template<class TYPE>

Snode<TYPE>::Snode(const TYPE &x)

: info(x)

{

next = 0;

}

Setting next to zero is optional, but it aids in constructing null-terminated lists--that is, lists whose last nodes have null next pointers.

Inserting a node into a linked list is illustrated in Figure 8.2, and is implemented with the following InsertAfter( ) function:

template<class TYPE>

void Snode<TYPE>::InsertAfter(Snode<TYPE> *n)

// Insert node n after this node

{

n->next = next;  // See Fig 8.2a

next = n;        // See Fig 8.2b

}

Insertion works by copying the current code's next pointer into the next pointer of the new node, and then having the current node point to the new node. Here is an example that creates the list shown previously in Figure 8.1(a):

Snode<char> mylist('a');

mylist.InsertAfter(new Snode<char>('c'));

mylist.InsertAfter(new Snode<char>('b'));

This code takes advantage of the fact that the constructor for Snode sets the next pointer to zero. Thus, the linked list is guaranteed to be null terminated. You'll notice that we built part of the list backwards because it's more efficient to add to the front of the list than to the back. (Later, we'll show you some ways to get around this.) Also, the first node was allocated statically, but the other nodes were allocated dynamically. It doesn't really matter how linked-list nodes are allocated. Later, we'll exploit that fact, implementing linked-lists through various memory allocation schemes.

Figure 8.2 Inserting a node into a singly-linked list.

Suppose you want to insert a node before another node. With singly-linked lists, this is awkward because you must scan the list from the beginning to get to the previous node. Assume that p starts out pointing to the front of a null-terminated list, and n is the node you wish to insert before the node q. Here are the necessary steps:

Snode<char> *p = &mylist;

while(p && p->next != q) p = p->next; // Scan down the list

if (p) p->InsertAfter(n); // Equivalent to inserting before q

You may have noticed that it isn't possible to insert before the first node in the list. (Think about the case where q equals the intial value of p.) To solve this problem, you can incorporate a dummy header node at the front of the list. This node's only purpose is to provide a "handle" for the list. The info field is ignored. Here, we show another way to build the list shown previously in Figure 8.1a--this time, using a dummy header node (the header node isn't shown in Figure 8.1a):

Snode<char> mylist(0); // Header node, with an arbitrary info field

mylist.InsertAfter(new Snode<char>('c')); // List built backwards

mylist.InsertAfter(new Snode<char>('b'));

mylist.InsertAfter(new Snode<char>('a'));

Each InsertAfter( ) operation inserts before the previous first node of the list, starting with no first node.

The complement to the InsertAfter( ) function is the following RmvNext( ) function, which, given a node, allows you to remove the node that follows it. Figure 8.3 illustrates the steps involved in removing a node from a singly-linked list.

template<class TYPE>

Snode<TYPE> *Snode<TYPE>::RmvNext()

// Removes the node following this node.

// Tests for next being null.

// Returns a pointer to the node removed.

{

Snode *p = next;

if (p) next = p->next; // See Fig 8.3

return p;

}

Figure 8.3 Removing a node from a singly-linked list.

Here's an example that shows how to remove and delete the first node of the list given previously:

Snode<char> *p;

p = mylist.RmvNext();  // Detach the node from the list

delete p;              // Then, de-allocate the node

Note that RmvNext( ) does not de-allocate the node, but rather just detaches the node from the list. The de-allocation is handled separately. Removing a node from a singly-linked list has the same awkwardness as inserting a node. Suppose you have a node q, and you wish to remove that node rather than the one that follows it. Here's how this can be done (assuming a null-terminated list with a header):

Snode<char> *p = &mylist;  // Point to head of list

while(p && p->next != q) p = p->next;

if (p) p->RmvNext();

The removal involves m steps, where m is the position of q in the list. The awkardness of singly-linked lists can be alleviated by adding another link to each node, pointing to the previous node. We'll examine such doubly-linked nodes next.

Sample code for implementing a singly-linked list is given on disk in the file ch8_1 .cpp.

Doubly-Linked List Nodes

Here is a typical definition of a doubly-linked list node, written as a template holding information of type TYPE:

template<class TYPE>

struct Dnode {

TYPE info;

Dnode<TYPE>*prev;

Dnode<TYPE>*next;

Dnode(const TYPE &x);

void InsertBefore(Dnode<TYPE> *n);

void InsertAfter(Dnode<TYPE> *n);

Dnode<TYPE> *Rmv();

} ;

template<class TYPE>

Dnode<TYPE>::Dnode(const TYPE &x)

: info(x)

{

prev = 0; next = 0;

}

template<class TYPE>

void Dnode<TYPE>::InsertBefore(Dnode<TYPE> *n)

// Inserts node n before this node.

// Tests for prev being null.

{

n->prev = prev;

n->next = this;

if (prev) prev->next = n;

prev = n;

}

template<class TYPE>

void Dnode<TYPE>::InsertAfter(Dnode<TYPE> *n)

// Attaches node n after this node.

// Tests for next being null.

{

n - > next = next;         // See Fig 8.4a

n - > prev = this;         // See Fig 8.4b

if (next) next->prev = n;  // See Fig 8.4c

next = n;                  // See Fig 8.4d

}

template<class TYPE>

Dnode<TYPE> *Dnode<TYPE>::Rmv()

// Removes this node from the list it is on.

// Tests for prev and next being null.

{

if (prev) prev->next = next;  // See Fig 8.5a

if (next) next->prev = prev;  // See Fig 8.5b

return this;

}

Each Dnode has both a prev and next pointer. With the prev pointer, it's easy to write the InsertBefore( ) function. Also the RmvNext( ) function can be replaced with a Rmv( ) function that directly removes a node--without having to find the node that precedes it.

The steps for inserting a node into a doubly-linked list are given in Figure 8.4; the steps for removing a node are shown in Figure 8.5. Note that the only difference between InsertBefore( ) and InsertAfter( ) is the interchanging of prev and next.

This next example creates the list shown previously in Figure 8.1(b)--this time using both the InsertBefore( ) and InsertAfter( ) functions. Again, a null-terminated list is built (in both directions) due to the constructor setting the prev and next pointers to null. We've created a dummy header node as we did before:

Dnode<char> mylist(0); // Header with arbitrary info field

mylist.InsertAfter(new Dnode<char>('c'));

Dnode<char> *p = new Dnode<char>('b');

mylist.InsertAfter(p);

p->InsertBefore(new Dnode<char>('a'));

Sample code for implementing a doubly-linked list is given on disk in the file ch8_2.cpp.

Figure 8.4 Inserting a node into a doubly-linked list.

Figure 8.5 Removing a node from a doubly-linked list.

HETEROGENOUS LISTS

The lists constructed in the previous sections are homogenous. That is, their nodes hold only one kind of data. In contrast, the nodes in a heterogenous list have different kinds of data. As is true with heterogenous arrays, two issues are involved with heterogenous lists: we have to deal with different sized data and typecasting problems.

With linked lists, handling different sized data is easy, since the nodes don't have to be the same size (as do array elements). This fact is due to the non-contiguous nature of linked lists. A heterogenous list node can store its data directly instead of using an indirect pointer scheme. In a sense, the prev and next pointers of a list node can serve the same purpose as the indirect pointers of a heterogenous array: allowing data of different types and sizes.

To avoid clumsiness, however, the prev and next pointers of the different nodes must be type compatible. How can this be accomplished? The following observation will give you a clue: if you look at the insert and remove functions of the Snode and Dnode classes, you'll notice that they do not involve the node data! This fact suggests you can define base classes that deal only with the list aspects of a node, without regard to what is stored on the list. Then, classes could be derived that deal with the different types of node data. The prev and next pointers, typed as base class pointers, can point to any node derived from the base class. Thus, lists with different types of nodes can be handled.

Abstract List Nodes

Here is an abstract class, Snodeb, designed for singly-linked list nodes, that uses the concepts of the previous section:

struct Snodeb { // An abstract singly-linked list node base class

Snodeb *next; // Pointer to next node in the list

Snodeb();

void InsertAfter(Snodeb *n);

Snodeb *RmvNext();

};

The functions of Snodeb are just like those of Snode, except the links are typed as Snodeb pointers. Also, keep in mind that a Dnodeb class can be similarly defined for doubly-linked nodes.

To show how to create a heterogenous list using Snodeb, we'll use the Shape and Circle class hierarchy defined in Chapter 5 (where heterogenous arrays were discussed). Given those two classes, we can derive node classes specifically for storing Shape and Circle objects:

struct ShapeNode : public Snodeb {

Shape info;

ShapeNode(float x=0, float y=0);

};

ShapeNode::ShapeNode(float x, float y)

: info(x, y)

{

// Nothing else to do

}

struct CircleNode : public Snodeb {

Circle info;

CircleNode(float x = 0, float y = 0, float r = 0);

};

CircleNode::CircleNode(float x, float y, float r)

: info(x, y, r)

{

// Nothing else to do

}

The following example creates and uses a list with two nodes (plus a dummy header) that stores a shape and circle object:

Snodeb mylist; // Create the header

mylist.InsertAfter(new ShapeNode(1, 2));

mylist.InsertAfter(new CircleNode(3, 4, 5));

Figure 8.6 illustrates the list that is created. Note in particular that the header, typed as an Snodeb node, doesn't contain any data. We've conveniently eliminated the problem of wasting space in the header for data that isn't used. You'll see this technique used again later in this chapter.

Figure 8.6 A heterogenous list.

Polymorphic List Nodes

While it's easy to create a heterogenous list, it's another matter to actually use the list. Take a look at what happens if we walk the list created in the previous section and print the areas of the shapes in each node:

Snodeb *p = &mylist.next;

cout << "Area is: " << ((CircleNode *)p)->info.Area() << '\n';

p = p->next;

cout << "Area is: " << ((ShapeNode *)p)->info.Area() << '\n';

A nasty typecasting problem surfaces. We must remember what the node types really are so that the proper Area( ) function is called. This is the same typecasting problem that occurs with all heterogenous containers.

One way to tame the typecasting is to use the same technique we did for arrays: store pointers to the data in the nodes, rather than storing the data directly. Again, it's especially beneficial to use base class pointers. For example, we could define the ShapeNode class as follows, which will work for both Shapes and Circle objects:

struct ShapeNode : public Snodeb {

Shape *info; // Point to shape object

ShapeNode(const Shape *p);

};

ShapeNode::Shape(const Shape *p)

{

info = p;

}

Now we can create a heterogenous list, as follows:

Snodeb mylist; // Create the header.

mylist.InsertAfter(new ShapeNode(new Shape(1, 2)));

mylist.InsertAfter(new ShapeNode(new Circle(3, 4, 5)));

The list created is shown in Figure 8.7. Now take a look at what happens when we walk the list:

Snodeb *p = &mylist.next;

cout << "Area is: " << ((ShapeNode *)p)->info->Area() << '\n';

p = p->next;

cout << "Area is: " << ((ShapeNode *)p)->info->Area() << '\n';

Because p is defined as an Snodeb pointer, we haven't eliminated the typecasting, but we've at least eliminated the problem of knowing whether a node points to a Shape or Circle object. That is handled by the fact that Area( ) is a virtual function.

We can reduce the typecasting further by providing another function, Next( ), whose sole purpose is to typecast the next pointer as a pointer to a ShapeNode object:

ShapeNode *ShapeNode::Next()

{

return (ShapeNode *)next;

}

. . .

ShapeNode *p = ((ShapeNode *)(&mylist))->Next();

cout << "Area is: " << p->info->Area() << '\n';

p = p->Next( );

cout << "Area is: " << p->info->Area() << '\n';

We still must do a typecast for the header node, but at least when walking the list no other typecasts are necessary--as long as we use Next( ) instead of next. We could eliminate the typecast on the header node by defining it as a ShapeNode, rather than an Snodeb object. However, the info field of the header would then be wasted.

Figure 8.7 A heterogenous list using indirect pointers.

List Nodes as Roots of Class Hierarchies

We can also create polymorphic list nodes using a method that allows us to store data directly in the nodes. The idea is to define the types of data to be stored on lists as bona fide nodes themselves. We can do this in two basic ways:

1.     Derive the data types to be stored on lists from abstract node classes. For example, we could derive Shape from Snodeb:

class Shape : public Snodeb {

//  . . . Shape class as in Chapter 5

Shape *Next();

};

. . .

Shape *Shape::Next()

{

return (Shape *)next;

}

2.     Define the data type to directly have list pointers. For example, you could define Shape to have a next pointer and the corresponding InsertAfter( ) and RmvNext( ) functions:

class Shape {

//  . . . Shape class code as in Chapter 5

Shape *next;

void InsertAfter(Shape *n);

Shape *RmvNext();

};

With both of these techniques, every class derived from Shape (such as Circle) will inherit list pointers and thus will be list nodes themselves. This is better than using indirect data pointers. The resulting list structures will be less complicated and take up less room. However, both techniques require list pointers for all data types to be stored on lists, which may be undesirable. For example, you may want to use the data types for applications other than lists.

Here is an example of the first technique. Note that, instead of writing p->info->Area( ), we write p->Area( ). That is, the info field no longer exists. The node data is a direct part of the node itself:

Snodeb mylist; // Create the header

mylist.InsertAfter(new Shape(1, 2)));

mylist.InsertAfter(new Circle(3, 4, 5)));

Shape *p = ((Shape *)(&mylist))->Next();

cout << "Area is: " << p->Area() << '\n';

p = p->Next();

cout << "Area is: " << p->Area( ) << '\n';

Contrast this with the second technique:

Shape mylist; // Create a header with shape data not used

mylist.lnsertAfter(new Shape(1, 2)));

mylist.InsertAfter(new Circle(3, 4, 5)));

Shape *p = mylist.next;

cout << "Area is: " << p->Area() << '\n';

p = p->next;

cout << "Area is: " << p->Area() << '\n';

The second technique is cleaner because we don't need a Next( ) function. The next pointer is already typed as a Shape pointer. Unfortunately, as the code shows, header nodes must store shape data that isn't used. If we make the header an Snodeb node, then typecasting difficulties are created.

The second technique also forces every type of list (such as a shape list, a window list, a bank account list, and so on) to be from a different hierarchy. There is no common Snodeb class. As a result, functions like InsertAfter( ) and RmvNext( ) have to be duplicated for each type of list. Even with these problems, the second technique is often used because of its directness. In fact, you'll see us use it in Chapter 13 when we develop a caching data structure that implements a doubly-linked list of "buckets."

LINKED LISTS AS CONCRETE DATA TYPES

Classes like Snodeb and Snode define the nuts and bolts of a linked list, but they don't allow us to treat linked lists at a high level. For example, it would be convenient to use linked lists the way we use arrays (without the subscripting, of course). Toward this end, it would be helpful to have functions to support such operations as copying, assignment, and concatenation. In this section, we'll show you how to define linked lists as concrete data types. Ten classes are implemented--Snodeb, Slistb, Snode, Slist, TempSlist, Dnodeb, Dlistb, Dnode, Dlist, and TempDlist--to provide all the necessary tools for implementing singly-linked and doubly-linked lists. (Don't worry, many of these classes are nothing but typecasting interfaces--with many functions that can be inlined.)

Before we explain the design details, we should note that, traditionally, many programmers never use linked-lists as separate entities and are prone to directly embedding list pointers (and associated functions) into their objects. It's not clear whether this is due to the historical lack of good development tools (for example, defining and using generic lists in C is somewhat clumsy) or whether the directness outweighs the advantages of reusable linked-list code. It is possible to define reasonably reusable, safe, and efficient linked-list classes using C++, as you are about to see. Only time will tell whether using linked lists as objects in their own right becomes a widespread technique.

Linked List Design Features

Singly-Linked List Abstract Classes

Singly-Linked List Templates

Doubly-Linked List Classes

Linked List Design Features

The linked lists that we'll present have the following design features:

       Header nodes are used to allow convenient insertion into the front of a list. Also, extra back pointers are added to allow convenient insertions to the back of lists.

       Circular lists are used rather than null-terminated lists (you'll see this arrangement shortly) because they lead to more elegant insertion and removal operations.

       The linked list code is abstracted so that it can be reused for lists storing different types of data.

       The list classes are designed with robustness and efficiency in mind--with a full complement of constructors, destructors, and overloaded operators.

       The lists are homogenous by nature, although heterogenous lists can be created by using pointers as the node data types.

       The allocation of nodes for a linked list is made flexible so that many different memory allocation schemes can be supported.

       We implement a clever way to efficiently return linked lists by value. This technique, in combination with overloading the '+' operator, allows us to use expressions like a = b + c without redundant copying.

We'll now explain the second feature (circular lists) in detail. Rather than terminate a list by giving the last node a null next pointer, you can instead have the last node point to the first node. Such a list is called a circular list. Figure 8.8 shows an example of a singly-linked circular list with a header node. The figure also shows an empty circular list consisting of a header node that points to itself. With circular lists, node insertion and removal can be done more elegantly. You never have to worry about a prev or next pointer being null, eliminating many special cases, as you'll see later.

Figure 8.8 A circular singly-linked list.

Singly-Linked List Abstract Classes

We'll now focus on the two classes Snodeb and Slistb, which support singly-linked lists in an abstract sense. (The node data isn't part of their definitions.) The Snodeb class is almost identical to the one given earlier, except that it uses encapsulation to the fullest (all members are protected). It is also set up to support circular lists, through the addition of a SelfRef( ) function:

class Snodeb {

protected:

Snodeb *next;

void InsertAfter(Snodeb *n);

Snodeb *RmvNext();

void SelfRef();       //  New function

friend class Slistb;  //  For convenient access

};

void Snodeb::SelfRef()

// Make this node point to itself

{

next = this;

}

Because we'll use circular lists, the functions InsertAfter( ) and RmvNext( ) can be coded quite elegantly--with no special cases needed when the list is empty (or about to become empty).

void Snodeb::InsertAfter(Snodeb *n)

{

n->next = next;

next = n;

}

Snodeb *Snodeb::RmvNext()

{

Snodeb *p = next;

next = p->next;

return p;

}

You might try these functions on paper--using various situations that involve circular lists--to convince yourself that they will always work properly (as long as you don't try to remove a list header node).

Complete code for the Snodeb class, as well as the Slistb class (presented next), is given on disk in the files slistb.h and slistb.cpp.

With the Snodeb class available, we can define the Slistb class:

class Slistb : private Snodeb {

protected:

virtual void MakeEmpty();

public:

Snodeb *back;

Slistb();

virtual ~Slistb();

Snodeb *Front() const;

Snodeb *Back() const;

virtual Snodeb *DupNode(const Snodeb *n) = 0;

virtual void FreeNode(Snodeb *n) = 0;

virtual void Clear();

int Copy(const Slistb &sl);

int Concatenate(const Slistb &sl);

void InsertAfter(Snodeb *a, Snodeb *b);

Snodeb *RmvNext(Snodeb *n);

Snodeb *RmvFront();

void AttachToFront(Snodeb *n);

void AttachToBack(Snodeb *n);

void Absorb(Slistb &sl);

void SplitAfter(Snodeb *n, Slistb &sl);

int IsEmpty() const;

int IsHeader(const Snodeb *n) const;

};

This class is truly abstract, having the pure virtual functions DupNode( ) and FreeNode( ). These are made virtual to support different types of memory allocation. Also, because Slistb doesn't know the type of data to be stored in the nodes, it can't be responsible for constructing them. Note that the code for Slistb will be shared for many types of singly-linked lists.

One of the most unusual features of Slistb is that it is derived from Snodeb. An Slistb object represents the header of a list. Since the list is circular, the last node will point to this header. By making Slistb type-compatible with Snodeb, we alleviate some typecasting difficulties otherwise encountered.

Since Slistb is derived from Snodeb, it inherits a next pointer, which is used to point to the first node of the list or back to itself if the list is empty. Another pointer, back, is defined to point to the last node in the list. This allows the efficient addition of nodes at the end of the list. (Otherwise, you would have to scan the list from start to end.) Interestingly, the presence of the back pointer doesn't mean you can efficiently remove nodes from the end of the list. There's no efficient way to do this with singly-linked lists.

Many design alternatives are possible for circular lists. Figure 8.9 shows some of these alternatives. You can choose whether to have a header, and whether to include the header in the cycle--with or without a back pointer. Figure 8.9(d) shows the alternative we've chosen for the Slistb class. By making the header node part of the cycle, it can serve as a flag to test when the end of the list is reached during a list walk. The IsHeader( ) function is provided for this purpose:

int Slistb::IsHeader(const Snodeb *n) const

// Sees if node n is the list head

{

return n == this;

}

This example uses IsHeader( ) to walk a list:

Snodeb *p = &mylist;

while(1) {

p = p->next;

if (mylist.IsHeader(p)) break;

// Process node p

}

When deriving the Slistb class, Snodeb was declared as a private base class. This disallows direct access to the next pointer and to the functions InsertAfter( ) and RmvNext( ). Instead, we must use access functions, such as the following Front( ) function:

Figure 8.9 Some alternative circular list designs.

(a) No header

(b) Header not in cycle

(c) Header in cycle

(d) Header in cycle w/back pointer

Snodeb *Slistb::Front() const

{

return next; // Header is returned if list is empty

}

The Slistb class defines its own InsertAfter( ) and RmvNext( ) functions, which is done to support the back pointer. Whenever a node is added to the end of the list, or if the last node of the list is removed, the back pointer must be updated. The RmvNext( ) function includes a test to ensure that we don't try to remove the list header itself. Both functions assume that the appropriate node actually resides on the list.

void Slistb::InsertAfter(Snodeb *a, Snodeb *b)

{

a->InsertAfter(b);

if (a == back) back = b; // Assumes a resides on this list

}

Snodeb *Slistb::RmvNext(Snodeb *n)

// ASSUMES n resides on this list.

{

if (n->next == this) return 0; // Can't detach list header!

Snodeb *p = n->RmvNext();

if (p == back) back = n;

return p;

}

Three more higher-level functions that use InsertAfter( ) and RmvNext( ) are provided to allow convenient insertion to the front or back of the list, and to remove the front of the list:

void Slistb::AttachToFront(Snodeb *n)

// Inserts node n after the header

{

InsertAfter(this, n);

}

Snodeb *Slistb::RmvFront()

// Removes the first node from the list. Note that

// RmvNext() will return 0 if the list is empty.

{

return RmvNext(this);

}

void Slistb::AttachToBack(Snodeb *n)

// Inserts the node n after the back node.

{

InsertAfter(back, n);

}

A default constructor is provided to create an empty list:

Slistb::Slistb()

{

MakeEmpty();

}

The constructor calls the MakeEmpty( ) function, which makes the list point to itself and also sets the back pointer to point to the list header:

void Slistb::MakeEmpty()

{

SelfRef();

back = this;

}

A companion function can be used to test if a list is empty:

int Slistb::IsEmpty() const

{

// Typecast needed because of const keyword

return next == (Snodeb *)this;

}

You can clear a list by calling the Clear( ) function:

void Slistb::Clear( )

{

Snodeb *n = next;

while(!IsHeader(n)) {

Snodeb *nn = n->next;

FreeNode(n);

n = nn;

}

MakeEmpty();

}

The Clear( ) function works by freeing each node in the list and then setting the list empty. Note that FreeNode( ) is a pure virtual function. Since Slistb doesn't know what kind of data is stored in a node, it can't properly free the node. Classes derived from Slistb (such as Slist, to be presented later in this chaper) are responsible for providing the appropriate definition of FreeNode( ).

The Slistb class has a virtual destructor. Contrary to what you might think, this destructor doesn't do anything, not even call Clear( ) to empty the list! The reason is that Clear( ) calls FreeNode( ) and, since that function is virtual, it isn't safe to call it from the base class destructor. Classes derived from Slistb should override their destructors to call Clear( ).

Two functions are provided to help in concatenating one list to the end of another, and in doing list copying. There's only one difference between list concatenation and copying. When a copy is made, the target list is cleared first:

int Slistb::Concatenate(const Slistb &sl)

{

if (this == &sl) return 0;  // Can't concatenate to self

const Snodeb *p = sl.next;

while(!sl.IsHeader(p)) {    // For all nodes in sl

Snodeb *q = DupNode(p);   // Make duplicate of the node

if (q == 0) return 0;     // Incomplete copy made

AttachToBack(q);

p = p->next;

}

return 1; // Successful concatenation

}

int Slistb::Copy(const Slistb &sl)

{

if (this == &sl) return 1; // Nothing to do

Clear();

return Concatenate(sl);

}

Note that Concatenate( ) calls the virtual function DupNode( ), which is supposed to allocate a new node and duplicate the data from the source node into it. Again, derived classes must provide their own versions of DupNode( ) to handle the type of data they are going to use.

An Absorb( ) function is provided, similar to Concatenate( ), which can be used to "smash" two lists together. The nodes of the list to be appended are not copied, but rather are absorbed through pointer manipulation. The source list then becomes empty. The steps in the absorption process are illustrated in Figure 8.10.

void Slistb::Absorb(Slistb &sl)

{

if (sl.IsEmpty() | | this == &sl) return; //  Already absorbed

back->next = sl.next;  // See Figure 8.10a

sl.back->next = this;       // See Figure 8.10b

back = sl.back;         // See Figure 8.10c

sl.MakeEmpty();         // See Figure 8.10d

}

The counterpart to Absorb( ) is the SplitAfter( ) function, which splits a source list after some specified node n. The tail of the split list is absorbed onto the end of a target list. If the target list comes in empty, the effect is to split the source list into two parts. The steps involved are similar to Absorb( ) in that most of the work concentrates on adjusting the back pointers. Figure 8.11 shows the result of splitting a list after the first node.

Figure 8.10 Absorbing one list into another.

Figure 8.11 Splitting a list.

void Slistb::SplitAfter(Snodeb *n, Slistb &tl)

// Assumes n is actually on this list. It's allowed to be

// the header (in which case, the list will go empty).

{

if (this == &tl) return;  // Can't split to yourself

if (n == back) return;    // Nothing to split

tl.back->next = n->next;

tl.back = back;

back->next = &tl;

n->next = this;

back = n;

}

Singly-Linked List Templates

Now that we've defined the Snodeb and Slistb base classes, we can derive specific classes tailored to nodes of a given type. To provide generality, the Snode and Slist classes are defined as templates. We start with the Snode class:

template<class TYPE>

class Snode : public Snodeb {

public:

TYPE info;

Snode();

Snode(const TYPE &x);

Snode<TYPE> *Next();

};

This class allows data of type TYPE to be stored on a node. Of particular importance is the Next( ) function, which typecasts the next pointer to be of the proper type, for reasons we gave earlier.

template<class TYPE>

Snode<TYPE> *Snode<TYPE>::Next()

// This function is for typecasting convenience

{

return (Snode<TYPE> *)next;

}

The Slist class, used to create lists storing TYPE data, is defined as follows:

template<class TYPE>

class Slist : private Slistb {

protected:

virtual Snode<TYPE> *AllocNode(const TYPE &x);

virtual Snodeb *DupNode(const Snodeb *sn);

virtual void FreeNode(Snodeb *dn);

Slistb::MakeEmpty; // Keep this protected

public:

friend class TempSlist<TYPE>;

friend class ArraySlist<TYPE>;

Slist();

Slist(const Slist<TYPE> &s);

Slist(TempSlist<TYPE> &s);

virtual ~Slist();

void operator=(const Slist<TYPE> &s);

void operator=(TempSlist<TYPE> &s);

Snode<TYPE> *Header() const;

Snode<TYPE> *Front() const;

Snode<TYPE> *Back() const;

Slistb::Clear;    //  Make this base function public

Slistb::IsEmpty;  //  Make this base function public

int Copy(const Slist<TYPE> &sl);

void InsertAfter(Snode<TYPE> *a, Snode<TYPE> *b);

void AttachToFront(Snode<TYPE> *n);

void AttachToBack(Snode<TYPE> *n);

Snode<TYPE> *RmvNext(Snode<TYPE> *n);

Snode<TYPE> *RmvFront();

int DelNext(Snode<TYPE> *n);

int DelFront();

void Absorb(Slist<TYPE> &sl);

void SplitAfter(Snode<TYPE> *n, Slist<TYPE> &sl);

int IsHeader(const Snode<TYPE> *n) const;

int Concatenate(const Slist<TYPE> &sl);

int operator+=(const Slist<TYPE> &sl);

Snode<TYPE> *operator+=(const TYPE &x);

friend TempSlist<TYPE>

operator+(const Slist<TYPE> &a, const Slist<TYPE> &b);

Snode<TYPE> *

NodeBeforeMatch(const TYPE &x, Snode<TYPE> *p=0) const;

Snode<TYPE> *AddToFront(const TYPE &x);

Snode<TYPE> *AddToBack(const TYPE &x);

Snode<TYPE> *AddAfter(const TYPE &x, Snode<TYPE> *n);

};

Complete code for the Snode and Slist classes is given on disk in the files slist.h and slist.mth. A test program is given in the file sltst.cpp.

The main purpose of Slist is to serve as a type-specific interface to the Slistb class. Many of the functions of Slist are nothing more than typecasting interfaces that can be easily inlined, such as the following Front( ) and Back( ) functions:

template<class TYPE>

Snode<TYPE> *Slist<TYPE>::Front() const

{

return (Snode<TYPE> *)Slistb::Front();

}

template<class TYPE>

Snode<TYPE> *Slist<TYPE>::Back() const

{

return (Snode<TYPE> *)Slistb::Back( );

}

The typecasts in these functions are somewhat unsafe. In the case of an empty list, both Front( ) and Back( ) will return a pointer to the header node, which isn't an Snode<TYPE> node even though it's returned as one. Before you try to redesign this, remember that the design we've given is really no more unsafe than returning a null pointer in case of an empty list. In either case, you must check for a valid pointer. The IsHeader( ) function can be used for that purpose (not shown here). The Header( ) function, which returns a pointer to the header node, is the counterpart to IsHeader( ). The Header( ) function has a similar problem to Front( ) and Back( ) in that it returns a pointer to the header node as though it were an Snode<TYPE> node:

template<class TYPE>

Snode<TYPE> *Slist<TYPE>::Header() const

{

return (Snode<TYPE> *)this;

}

The Header( ) function was designed for typecasting convenience when iterating through a list, and should be used only for that purpose. The followingNodeBeforeMatch( ) function shows an example of using Header( ):

template<class TYPE>

Snode<TYPE> *Slist<TYPE>::

NodeBeforeMatch(const TYPE &x, Snode<TYPE> *p) const

{

// Note that Next() is used, rather than 'next'

// directly. This is for typecasting convenience.

if (p == 0) p = Header(); // p == 0 is the default

while(!IsHeader(p->Next())) { // Scan until end of list

if (p->Next()->info == x) return p; // Match found

p = p->Next();

}

return 0; // No match

}

Given some data of type TYPE, this function scans the list looking for the node that contains matching data. Note that the node before the matching node is returned, in case you want to remove the matching node using RmvNext( ). BothHeader( ) and Next( ) return Snode<TYPE> pointers, which alleviate the need to use direct typecasts between Snodeb pointers and Snode<TYPE> pointers. The typecasting problem isn't eliminated; it's just encapsulated as much as possible.

The following InsertAfter( ) and RmvNext( ) functions also serve as type-casting interfaces:

template<class TYPE>

void Slist<TYPE>::InsertAfter(Snode<TYPE> *a, Snode<TYPE> *b)

{

Slistb::InsertAfter(a, b);

}

template<class TYPE>

Snode<TYPE> *Slist<TYPE>::RmvNext(Snode<TYPE> *n)

{

return (Snode<TYPE> *)(Slistb::RmvNext(n));

}

Other functions are defined similarly, such as AttachToFront( ), RmvFront( ), and AttachToBack( ). As is true with the Slistb class, these functions are only responsible for attaching or detaching nodes to a list; they don't allocate or deallocate the nodes. Since Slist knows the type of node data to be used, functions can be added to automatically allocate and de-allocate the nodes, such as the following AddAfter( ) and DelNext( ) functions:

template<class TYPE>

Snode<TYPE> *Slist<TYPE>::AddAfter(const TYPE &x, Snode<TYPE> *n)

{

Snode<TYPE> *p = AllocNode(x);

if (p) InsertAfter(n, p);

return p;

}

template<class TYPE>

int Slist<TYPE>::DelNext(Snode<TYPE> *n)

{

Snode<TYPE> *p = RmvNext(n);

if (p) {

FreeNode(p);

return 1;

}

return 0;

}

The AllocNode( ) and FreeNode( ) functions are virtual. Here are the specific versions for the Slist class:

template<class TYPE>

Snode<TYPE> *Slist<TYPE>::AllocNode(const TYPE &x)

// A function that allocates a new Snode<TYPE> node,

// holding a copy of x as its data

{

return new Snode<TYPE>(x);

}

template<class TYPE>

void Slist<TYPE>::FreeNode(Snodeb *n)

// Deletes node n. Assumed to be a Snode<TYPE> node.

{

delete((Snode<TYPE> *)n);

}

The typecast used in FreeNode( ) seems rather unsafe, since it assumes that n is actually an Snode<TYPE> node. Consequently, you'll notice this function(as well as AllocNode( )) is protected. Note that n is guaranteed to be an Snode<TYPE> node for all uses of FreeNode( ) in the Slist class.

You may wonder why the unsafe typecast is used at all. Why not instead make the destructor for Snodeb virtual? Then, the delete operator would work properly without the typecast. However, adding a virtual function to Snodeb would add overhead in the form of one extra pointer (to the virtual function table) in every node. The typecast, then, can save a lot of space in the lists. Here, we're trading elegance for space efficiency. As always, you should pay attention to details like this, particularly if your designs are to become part of a general-purpose library.

The following DupNode( ) function, used to create a copy (except for the next pointer) of a node, is related to AllocNode( ):

template<class TYPE>

Snodeb *Slist<TYPE>::DupNode(const Snodeb *n)

{

return AllocNode(((Snode<TYPE> *)n)->info);

}

Recall that SListb::Copy( ) uses DupNode( ) when making copies of each node in a list. The Slistb::Copy( ) function is used by Slist::Copy( ), which in turn is used by the operator=( )function:

template<class TYPE>

int Slist<TYPE>::Copy(const Slist<TYPE> &sl)

{

return Slistb::Copy(sl);

}

template<class TYPE>

void Slist<TYPE>::operator=(const Slist<TYPE> &sl)

{

Copy(sl); // Traps copies to self

}

In addition to overloading the '=' operator, we've overloaded the '+=' operator to provide a convenient notation for concatenating both single nodes and lists onto other lists:

template<class TYPE>

int Slist<TYPE>::operator+=(const Slist<TYPE> &sl)

{

return Concatenate(sl);

}

template<class TYPE>

Snode<TYPE> *Slist<TYPE>::operator+=(const TYPE &x)

{

return AddToBack(x);

}

The Slist class provides another way to add one list to another by using the '+' operator:

template<class TYPE>

TempSlist<TYPE> operator+(const Slist<TYPE> &a, const Slist<TYPE> &b)

{

Slist<TYPE>copy_of_a(a);

copy_of_a.Concatenate(b);

return copy_of_a; // copy_of_a absorbed into temp result

}

The operator+( ) function makes a copy of both source operands and returns the result as a new list. This allows statements like the following to be used, where a, b, and c are lists:

a = b + c; // Add lists a and b together; store in c

You might think a lot of copying is taking place in such statements: one copy made in returning the result and another in assigning the result to the destination list. Instead, a clever trick is used to avoid the copying. The temporary list that's constructed to hold the result can be absorbed (using Absorb( )) into the list that's returned. The returned list can then be absorbed into the destination list, which is cleared beforehand. (Recall the Absorb( ) function smashes two lists together, rather than copying the nodes.)

In effect, we must find a way for the copy constructor (used in the function return) and the overloaded assignment operator (used in the assignment) to use absorption rather than copying. How can we do this? One way is to provide special versions of the copy constructor and assignment operator that work only for temporary lists. You'll notice that operator+( ) returns a TempSlist object, instead of an Slist object. The TempSlist class is defined as follows:

template<class TYPE>

class TempSlist : private Slist {

private:

friend class Slistb<TYPE>;

virtual Snodeb *DupNode(const Snodeb *);

virtual void FreeNode(Snodeb *);

public:

TempSlist(Slist<TYPE>  &sl);

} ;

Complete code for the TempSlist class is given on disk in the file Slist.h.

The sole purpose of TempList is to provide a way to trap cases when the list is meant to be temporary. A TempSlist object is just like an Slist object: all that's changed is the type name (and the implicit restriction of its use only for temporary lists). The class has a singlecopy constructor that takes a normal list and uses absorption to turn it into a temporary list:

template<class TYPE>

TempSlist<TYPE>::TempSlist(Slist<TYPE> &sl)

{

Slistb::Absorb(sl); // Note: sl goes empty

}

The TempSlist copy constructor is used in returning the result of a list addition. Then, if the temporary list is to be used in an assignment (such as c = a + b), the following overloaded assignment operator for Slist objects is used:

template<class TYPE>

void Slist<TYPE>::operator=(TempSlist<TYPE> &sl)

{

Clear();

Slistb::Absorb(sl; // sl goes empty in the process

}

You should walk through the steps involved in statements like c = a + b to assure yourself that indeed no redundant copying takes place. The trick of defining a new type, such as TempSlist, for the sole purpose of trapping special cases in overloaded function calls is useful to remember.

There's one last function to consider for Slist--the destructor:

template<class TYPE>

Slist<TYPE>::~Slist()

{

Clear();

}

This destructor causes any remaining nodes in the list to be freed at list-destruction time. As we mentioned earlier, it's very important that the call to Clear( ) be placed here, not in the base Slistb class destructor, due to the virtual function calls to FreeNode( ) inside Clear( ).

In the following example, we've used the Slist class to construct and manipulate singly-linked lists:

void f()

{

Slist<char> x, y, z;  // Construct empty lists

x.AddToBack('a');

x += 'b';             // Same as x = AddToBack('b');

y += 'c';

y += 'd';

x += y;              // x now holds [a b c d], y holds [c d]

y.Clear();           // Remove all nodes from y

y.AddToFront('f');

y.AddToFront('e');

z = x + y;           // z now holds [a b c d e f], x [a b c d], y [e f]

Snode<char. *p = z.NodeBeforeMatch('c); // Look for 'c'

if (p) z.RmvNext(p); // Remove node holding 'c'

// All lists destroyed implicitly here

}

Doubly-Linked List Classes

The five classes that support doubly-linked lists--Dnodeb, Dlistb, Dnode, Dlist, and TempDlist--are very similar to their singly-linked counterparts and will not be shown here. (They are given on disk, however.) Figure 8.12 shows examples of circular doubly-linked lists that can be created using the five classes.

Complete code for the Dnodeb, Dlistb, Dnode, Dlist, and TempDlist classes is given on disk in the files dlistb.h, dlistb.cpp, dlist.h, and dlist.mth. A test program is given in the file dltst.cpp.

VARIATIONS ON A THEME

With AllocNode( ) and FreeNode( ) available as virtual functions, we can create many variations in allocating the nodes of linked lists. In this section, we'll look at some of the possibilities. We'll culminate with a hybrid array-based linked-list design.

VARIATIONS ON A THEME

With AllocNode( ) and FreeNode( ) available as virtual functions, we can create many variations in allocating the nodes of linked lists. In this section, we'll look at some of the possibilities. We'll culminate with a hybrid array-based linked-list design.

Cached List Nodes

Array-Based Linked Lists

Cursor-Based Singly-Linked Lists

Cursor-Based Doubly-Linked Lists

Array-Based Linked Lists

While the Cslist class caches freed nodes, it still uses the heap to allocate nodes initially. Also, each node is allocated separately, which can lead to heap fragmentation. Another alternative is to allocate the nodes in blocks. In fact, if you know the maximum size of a list ahead of time, you can pre-allocate the nodes in a single array. The result is a hybrid structure--an array-based linked list. Figure 8.14 shows an example of a null-terminated singly-linked list where all the nodes are allocated contiguously in an array.

It's actually quite easy to derive array-based linked lists from the Slist and Dlist classes and, in the process, reuse a lot of existing code. The secret is to include a constructor that allocates an array of nodes, and then to override AllocNode( ) to grab nodes from this array. Also, a free list can be maintained simultaneously with the main list, threading together all the nodes in the array that are not in use. Unlike the Cslists used in the previous section, though, each array-based list will have its own free list, rather than sharing a global one. The ArraySlist class, sketched below, illustrates how an array-based singly-linked list can be defined:

template<class TYPE>

class ArraySlist : private Slist<TYPE> {

protected:

virtual Snode<TYPE> *AllocNode(const TYPE &d);

virtual void FreeNode(Snodeb *n);

int nodes_used, max_nodes;

Slist<TYPE> free_list;

Snode<TYPE> *nodes;

virtual void MakeEmpty();

void Setup(int mn);

. . . // Other private functions

public:

ArraySlist(int mn);

virtual ~ArraySlist();

virtual void Clear();

. . . // Other public functions

};

Figure 8.14 An array-based linked list.

Complete code for the ArraySlist class is given on disk in the files aslist.h and aslist.mth. A test program is provided in the file asltst.cpp. Also, a doubly-linked version, ArrayDlist, is provided in the files adlist.h, adlist.mth, and adltst.cpp.

The ArraySlist constructor sets up the array-based linked list by calling a Setup( ) function, which allocates an array of nodes of size mn.

template<class TYPE>

ArraySlist<TYPE>::ArraySlist(int mn)

{

Setup(mn);

}

template<class TYPE>

void ArraySlist<TYPE>::Setup(int mn)

{

nodes_used = 0;

nodes = (Snode<TYPE> *)(new char[mn * sizeof(Snode<TYPE>)]);

max_nodes = (nodes == 0) ? 0 : mn;

}

The node array that gets created is actually a variable-length array, where num_nodes represents the logical length and max_nodes represents the dimensioned length. Initially, no nodes are in use. Note that the array's logical length can only grow, not shrink. When a node is to be freed, it is placed on the free list. Eventually, all the nodes will either be on the main list or the free list. As we did for the variable-length arrays defined in Chapter 4, the node array is actually constructed as an array of characters. This prevents the nodes from being constructed until they are actually used.

When a node is to be used, the placement new operator is called to handle the construction of the nodes. This can be seen in the following AllocNode( ) function (which overrides Slist's version):

template<class TYPE>

Snode<TYPE> *ArraySlist<TYPE>::AllocNode(const TYPE &x)

{

Snode<TYPE> *p = 0; // Default to failure

if (!free_list.IsEmpty()) {

p = free_list.RmvFront();

}

else if (nodes_used < max_nodes) {

p = nodes + nodes_used++;

}

if_(p)_new(p) Snode<TYPE>(x); // Use in-place construction

return p;

}

The AllocNode( ) function works as follows: if the free list is empty (which it will be at the start), then a node is grabbed from the array and constructed in place. It's possible for all the nodes in the array to be in use, in which case the list is full, so a null pointer is returned. If the free list isn't empty, one of its nodes is used. It's important to realize that the nodes themselves are never moved physically. All that changes is their placement either on the main list or the free list.

The FreeNode( ) function works by attaching a node onto the free list. A critical part of this process is the call to the node's destructor--using an explicit destructor call. This must be done because AllocNode( ) assumes that the nodes it grabs aren't constructed.

template<class TYPE>

void ArraySlist<TYPE>::FreeNode(Snodeb *n)

{

typedef Snode<TYPE> typenode;  // This typing helps some compilers

typenode *p = (typenode *)n;

p->typenode::~typenode();      //  Explicitly destroy node

free_list.AttachToFront(p);    //  Attach to the free list

}

Figure 8.15 A sample ArraySlist object.

We explicitly destroy the node to ensure that the destructor for the info field is called. Note, however, that the next pointer will be set to point to the next node in the free list. Thus, nodes in the free list are actually partially constructed. Figure 8.15 shows a sample list created with the ArraySlist class, with some nodes on the main list (fully constructed), some on the free list (partially constructed), and some nodes not yet constructed. Note that the main list and free list headers are stored separately from the node array.

Even though a node might be destroyed, it isn't actually freed until the list itself is destroyed, as the following ArraySlist destructor shows.

template<class TYPE>

ArraySlist<TYPE>::~ArraySlist()

{

Clear();

delete[] (char *)nodes;

}

The ArraySlist destructor is somewhat tricky because the destructors for the nodes must be handled properly. It's quite possible for some of the nodes to be in use (and thus constructed) but others to be unused (and thus be unconstructed). Somehow, the ArraySlist destructor must cause the node destructor to be called only for those nodes currently in use. This is accomplished by calling the following virtual Clear( ) function:

template<class TYPE>

void ArraySlist<TYPE>::Clear()

{

typedef Snode<TYPE> typenode;

typenode *q, *p;

p = Front( );

while(!IsHeader(p)) {          // For each node in use

q = p->Next();

p->typenode::~typenode();  // Explicitly destroy node

p = q;

}

// The list is now made to appear empty, so that when the

// base class destructor calls Clear(), it works properly.

// Note that both the main and free lists are made empty.

MakeEmpty();

}

Once the nodes in use are destroyed, the memory for the nodes is de-allocated by calling delete in the ArraySlist destructor. The nodes are de-allocated as though they were an array of characters, which is how they were allocated initially. The typecast to char * is important; otherwise, the node destructors would be erroneously called again.

Array-based linked lists provide compactness and speed. Because they are contiguous, array-based linked lists don't waste space that might result from heap fragmentation. Also, adding and removing nodes from the list is very fast since no heap manipulation is used. Of course, these improvements come at a price: you must allocate the maximum size of the linked list ahead of time, negating one of the advantages of linked lists. However, all other advantages remain, the most important being the ability to rearrange the elements of the list with minimal data movement.

Cursor-Based Singly-Linked Lists

If you wish to use array-based linked lists, you should be aware that yet another variation is possible. Rather than use pointers for the node links, you can use array indexes. For example, a singly-linked node might be defined as follows:

template<class TYPE>

class Snode {

TYPE info;

unsigned next; // Index, or cursor, to next node

};

The next variable is now an index that provides the offset in the array to the next node in the list. This index is sometimes called a cursor. In fact, array-based linked lists that use indexes rather than pointers are often called cursor-based linked lists. Figure 8.16 shows an example of a null-terminated, cursor-based singly-linked list. In the figure, it's assumed that the elements are indexed starting at 1, and that 0 represents the end of the list.

Figure 8.16 A sample cursor-based singly-linked list.

You gain the following advantages by using cursors instead of pointers for links:

       Smaller. Depending on the implementation, a cursor may occupy less room than a pointer. For instance, a cursor might be an 8- or 16-bit integer, whereas a pointer will probably be either 16 or 32 bits. Thus, cursor-based linked lists can be more compact than those using pointers.

       Relocatable. Since cursors are indexes relative to the beginning of the array, the array can be relocated elsewhere in memory--without having to update the cursors. If pointers are used instead for links, they would all have to be modified to reflect the new locations of the nodes.

There is one drawback, however, to the use of cursors rather than pointers: cursors are slower. To find the next node using a cursor, subscript arithmetic must be used (involving an addition and multiplication). In contrast, to find the next node using a pointer, a simple pointer de-reference is all that's required.

There's another variation sometimes used with cursors. Represent the linked list as two separate parallel arrays--one holding the node data and one holding the cursors--rather than implementing a single node array that holds both. In Figure 8.17, the singly-linked list portrayed in Figure 8.16 is implemented using parallel arrays.

The following CursorSlist class uses this parallel array approach to implement a singly-linked list as a concrete data type:

template<class TYPE>

class CursorSlist {

protected:

TYPE *info;         // The node data array

unsigned *next;     // Next cursors (next[0] == front)

unsigned num_nodes; // Number of nodes allocated

unsigned back;      // Cursor to back of list

unsigned avail;     // Cursor to front of avail list

void InsertAfter(unsigned p, unsigned x);

unsigned RmvNext(unsigned p);

int Alloc(int n);

void MakeEmpty();

public:

CursorSlist(int n);

unsigned Next(unsigned p) const;

unsigned AddAfter(const TYPE &x, unsigned p);

int DelNext(unsigned p);

TYPE &operator[](unsigned p);

// ... Other functions

};

Figure 8.17 Using parallel arrays to represent a singly-linked list.

Complete code for the CursorSlist class is given on disk in the files curslist.h and curslist.mth. A test program is given in the file cursltst.cpp.

The first element of the next array is reserved for the list header, and the list is circular. Thus, an empty list has next[0] = 0. Note that the reserved header element forces all non-header list elements to have cursors > 0. To avoid wasting node data for the header, the info array has one less element than next. Given a positive cursor p, the elements info[p-1] and next[p] make up, conceptually, one full list node.

In addition to the two arrays, we need to provide some other variables. The back variable is a cursor to the last node on the list. If the list is empty, back = 0. The num_node variable keeps track of the dimensioned size of the arrays. As is the case with the ArraySlist class, CursorSlist incorporates a built-in free list. The avail cursor points to the front of the free list. Initially, all nodes except for the header are placed on the free list. (In contrast, the ArraySlist class started the free list as an empty list). Thus, avail is initialized to one. Note that the free list does not use a next element for its header and is null-terminated, with the cursor of the last node being zero. (Recall that zero can't index a non-header node.)

The following Alloc( ) and MakeEmpty( ) functions, called by the constructor, show how a CursorSlist object is set up to represent an empty list. Figure 8.18 illustrates the resulting layout.

template<class TYPE>

int CursorSlist<TYPE>::Alloc(int n)

// Allocates room for n list nodes

{

// Create unconstructed array for the node data

info = (TYPE *) new char[n * sizeof(TYPE)];

if (!info) return 0; // Out of memory

// Create the next cursor array. Leave room for the

// header element.

next = new unsigned[n+1];

if (!next) {

delete[] (char *)info;

return 0; // Out of memory

}

// Set up all nodes on the free list

// Boundaries handled after loop

for(unsigned i = 1; i<n+1; i++) next[i] = i+1;

avail = 1;    // Starting cursor of the free list

next[n] = 0;  // Null-terminate the free list

MakeEmpty();  // Start the main list to be empty

num_nodes = n;

return 1;     // Successful

}

template<class TYPE>

void CursorSlist<TYPE>::MakeEmpty()

// Makes the list empty by pointing header to

// itself and pointing back to the header, too

{

next[0] = 0;

back = 0;

}

Figure 8.18 An empty CursorSlist object.

The Alloc( ) function allocates the info array as unconstructed bits. The corresponding info element will be constructed (using an in-place construction process) only when a node is placed on the main list. The following InsertAfter( ) and AddAfter( ) functions show how nodes are obtained from the free list and added to the main list.

template<class TYPE>

void CursorSlist<TYPE>::InsertAfter(unsigned p, unsigned x)

// Inserts a node indexed by x after node indexed by p.

{

next[x] = next[p];

next[p] = x;

if (p == back) back = x;

}

template<class TYPE>

unsigned CursorSlist<TYPE>::AddAfter(const TYPE &x, unsigned p)

// Grabs a node from the free list, copies data x into it,

// and adds the node after node indexed by p

{

unsigned n = avail;         // Get cursor to next avail node

if (n == 0) return 0;       // No room

avail = next[avail];        // One less node available

InsertAfter(p, n);          // Insert node into main list

new(info + n - 1) TYPE(x);  // In-place copy construction

return n;                   // Return cursor the node

}

It's interesting to compare node insertions using cursors with those using pointers, as shown in Table 8.2.

Removing a node from a list works in the opposite fashion. The node is detached from the list and placed at the front of the free list. The node data is then explicitly destructed. The following RmvNext( ) and DelNext( ) functions show the details:

template<class TYPE>

unsigned CursorSlist<TYPE>::RmvNext(unsigned p)

// Removes from the main list the node

// following the node indexed by p

{

unsigned n = next[p];  // Like n = p->next

next[p] = next[n];     // Like p->next = n->next

if (n == back) back = p;

return n;

}

template<class TYPE>

int CursorSlist<TYPE>::DelNext(unsigned p)

// Removes the node following the node indexed by p, placing

// the node on the free list. The node data is destroyed.

{

if (next[p] == 0) return 0;    // Can't remove header

unsigned n = RmvNext(p);       // Detach from list

(info + n - 1)->TYPE::~TYPE(); // Destroy node data

next[n] = avail;               // Add to front of free list

avail = n;                     // New free list header

return 1;                      // Indicate success

}

Table 8.2 Different techniques for node insertions.

Cursor-Based  Pointer-Based

next[x] = p;   x->next = p;

next[p] = x;   p->next = x;

Figure 8.19 illustrates the layout of a cursor-based list with nodes both on the main list and free list.

The CursorSlist class has functions similar to the ArraySlist class, such as Front( ), Back( ), IsHeader( ), AddToFront( ), AddToBack( ), and so on. One unusual function is the following overloaded [ ] operator function (a const version is also supplied in the class):

Figure 8.19 A non-empty CursorSlist object.

template<class TYPE>

TYPE &CursorSlist<TYPE>::operator[](unsigned p)

{

return info[p-1];

}

This allows you to obtain the information stored at a node by using a normal array-subscripting operation. The index p must point to a valid non-header node, but that isn't checked. (Doing so would be very difficult.) Here's an example, of walking through a list and printing out the node information:

unsigned p = mylist.Front();

While(!mylist.IsHeader(p)) {

cout << mylist[p] << ' - ';

p = mylist.Next(p);

}

The last statement calls the Next( ) function, which allows you to conveniently get the next node in sequence:

template<class TYPE>

unsigned CursorSlist<TYPE>::Next(unsigned p) const

{

return next[p];

}

Figure 8.20 A non-empty CursorDlist object.

Cursor-Based Doubly-Linked Lists

A doubly-linked cursor-based list can be defined by adding a third array, representing prev cursors, as illustrated in Figure 8.20. Note that the free list is implemented as a singly-linked null-terminated list, even though the main list is a circular doubly-linked list. The CursorDlist class, a doubly-linked analog to CursorSlist, can be defined as:

template<class TYPE>

class CursorDlist {

protected:

TYPE *info;         // The node data array

unsigned *prev;     // Prev cursors

unsigned *next;     // Next cursors

unsigned avail;     // Cursor to front of avail list

unsigned num_nodes; // Number of nodes allocated

void InsertAfter(unsigned p, unsigned x);

void InsertBefore(unsigned p, unsigned x);

unsigned Rmv(unsigned p);

...

};

Complete code for the CursorDlist class is given on disk in the files curdlist.h and curdlist.mth. A test program is given in the file curdltst.cpp.

The CursorDlist class is very similar to the other list classes we've defined, so we won't show it in detail here. However, it's interesting to note the symmetry of the InsertAfter( ), InsertBefore( ), and Rmv( ) functions. (The comments illustrate equivalent pointer-based manipulations.)

template<class TYPE>

void CursorDlist<TYPE>::InsertBefore(unsigned p, unsigned x)

// Inserts a node indexed by x, assumed to be already

// detached from the free list, before node indexed by p

{

prev[x] = prev[p]; // Like x->prev = p->prev

next[prev[p]] = x; // Like p->prev->next = x

next[x] = p;       // Like x->next = p

prev[p] = x;       // Like p->prev = x

}

template<class TYPE>

void CursorDlist<TYPE>::InsertAfter(unsigned p, unsigned x)

// Inserts a node indexed by x, assumed to be already

// detached from the free list, after node indexed by p

{

next[x] = next[p]; // Like x->next = p->next

prev[next[p]] = x; // Like p->next->prev = x

prev[x] = p;       // Like x->prev = p

next[p] = x;       // Like p->next = x

}

template<class TYPE>

unsigned CursorDlist<TYPE>::Rmv(unsigned p)

// Removes the node indexed by p from the main list

{

next[prev[p]] = next[p]; // Like p->prev->next = p->next

prev[next[p]] = prev[p]; // Like p->next->prev = p->prev

return p;

}

SUMMARY

In this chapter, you've seen some of the variations possible for linked lists. These data structures are quite versatile and, along with arrays, form the basis for many of the more complicated data structures. We've attempted to turn linked lists into concrete data types with classes like Slist, ArraySlist, CursorSlist, and so on. However, keep in mind that list-type links are sometimes directly embedded into other objects, as you will see in later chapters. As such, the most important functions to keep in mind are ones like InsertAfter( ), InsertBefore( ), Rmv( ), and RmvNext( ), which work on a node-by-node basis.

The linked lists presented here are designed to work in memory. However, it's quite possible, and useful, to have linked lists that are stored on files. This can be done by using file offsets for links (equivalent to cursors), rather than pointers. In later chapters, we'll show you how to do this.

One final comment: as is true with any data structure, linked lists are designed for certain types of applications and may work very poorly for others. Linked lists excel in cases where you need only sequential access and where the objects to be stored are rearranged frequently. Linked lists aren't as useful with random access or when the objects being stored need to be searched. Other data structures--such as arrays or search trees--work better in this regard.

Go to Chapter 9            Return to Table of Contents


Practical Data Structures: STACKS AND QUEUES

 Next Chapter Return to Table of Contents Previous Chapter

CHAPTER 9: STACKS AND QUEUES

In this chapter, we take a look at two data structures, stacks and queues, that can be created from the fundamental array and linked-list structures. In Chapters 1 and 3, you saw some examples of stacks. Here we will design stacks as concrete data types, with an eye on efficiency. Queues will be implemented with the same design philosophy. The idea is to define stack and queue classes that are readily usable in your own applications--with minimum overhead.

Stacks and queues are related data structures. Both are sequences of elements with restricted access. A stack is known as a last-in, first-out (LIFO) structure, where access to the elements takes place at one end of the sequence (called the top), using the operations push and pop. Figure 9.1 shows an example of a stack. A queue is a sequence where elements are inserted at one end (the tail), and extracted from the other end (the head). Queues work in a first-in, first out (FIFO) manner, as illustrated in Figure 9.2. Lines of cars at a traffic light, lines at the post office, and computer jobs waiting to be printed are all conceptual examples of queues.

Figure 9.1 A sample stack.

Figure 9.2 A sample queue.

It's possible to combine a stack and queue to arrive at another data structure, which we'll call a staque. (This terminology was invented by the author.) In a staque, the stack operations push and pop are defined, as well as the queue operations insert (into the tail) and extract (from the front.) Of course, the extract operation is identical to the pop operation.

By taking this approach one step further, we could also allow extraction from the tail. The result is known as a double-ended queue, or deque. Figure 9.3 shows an example of a deque.

Figure 9.3 A Double-ended queue.

A priority queue is another variation of a queue. Here, the elements are ordered by priorities that have been assigned to them, rather than by the order in which the elements are inserted. Figure 9.4 shows an example of a prioriity queue, with numbers representing the priority order. Notice that the sequence of elements are in descending order. You'll see priority queues implemented in Chapters 12. See also the companion volume [Flamig 93].

We'll now take a detailed look at stacks and queues, starting with fixed-size stacks.

FIXED-SIZE STACKS

The most common type of stack is a fixed-size stack, implemented as an array of elements. When you define array-based stacks, you need to consider these three main design issues:

Should the stack be statically or dynamically allocated?

How should you keep track of the top of the stack and the bounds of the stack?

When should the stack elements be constructed and destructed?

The first issue can be handled quite easily. As we did for the arrays defined in Chapter 4, a base class can be defined that purposely avoids specifying how the stack elements are allocated. Then, we can derive specific classes for both statically and dynamically allocated stacks.

You can take two basic approaches to resolve the second issue (keeping track of the top and bounds of the stack). One approach is to use an index to indicate which element is at the top of the stack. When doing pushing and popping, this index can be checked with the bounds of the stack. This is the approach used in most books to implement array-based stacks. (Indeed, we used this approach in Chapters 3.) However, an index is not the best way to push and pop stack elements because subscripting involves multiplication and addition. A more efficient approach is to use pointers, both to reference the top of the stack and to mark the bounds of the stack.

Figure 9.4 A priority queue.

You can also take two approaches to resolve the third issue (determining when to construct and deconstruct stack elements). The easiest approach is to construct the elements using default constructor calls when the stack is first built, and to destroy all the elements when the stack itself is destroyed. Indeed, that's the conventional way that C++ handles arrays. The second approach is to construct the stack elements individually during push operations, and to destroy elements one at a time during pop operations. This can be done using in-place copy constructor calls and explicit destructor calls--the same techniques we used for the variable-length arrays in Chapter 4.

Why bother with the in-place construction method? It's intuitively appealing and, in some cases, more correct to construct elements only when they are used. For example, constructed elements might use other resources (such as open file handles), allocate memory buffers, and so on. By deferring construction until the element is actually used, and destroying the element after it's no longer needed, you can ensure that the resources used by the element are acquired and freed in a timely fashion.

The types of constructors available can help you determine whether to use in-place construction. It may not make sense for the element type to have a default constructor (perhaps there is no sensible notion of a "null" object of that type). In this case, in-place construction is the best choice. Of course, for in-place construction to be used, the type must have a copy constructor, something that may not always be desirable either.

A Base Class for Array-Based Stacks

Using the preceding design principles, we'll now define a base class, ArrStk, which handles the basic operations of a fixed-sized array-based stack. The class assumes the stack elements are allocated elsewhere:

template<class TYPE>

class ArrStk {

protected:

TYPE *start, *finish, *nxt;

unsigned dimlen;

void Setup(TYPE *m, unsigned n);

ArrStk(const ArrStk<TYPE> &);

public:

ArrStk(TYPE *m, unsigned n);

virtual ~ArrStk( );

void operator=(const ArrStk<TYPE> &s);

void Copy(const ArrStk<TYPE> &s);

int Push(const TYPE &x);

int Pop(TYPE &x);

int Pop( );

void FastPush(const TYPE &x);

void FastPop(TYPE &x);

TYPE *Top( );

const TYPE *Top( ) const;

TYPE *History(unsigned n);

const TYPE *History(unsigned n) const;

void Rewind(unsigned n=0);

int IsFull( ) const;

int IsEmpty( ) const;

unsigned Size( ) const;

};

Complete code for the ArrStk class is given on disk in the files arrstk.h, arrstk.mth, and placenew.h. Through the use of defined macros, the code on disk can handle both the in-place copy-construction method and the conventional default-construction method. In this book, we show only the in-place construction method.

The ArrStk constructor takes a pointer to the stack elements (assumed to be already allocated, but not constructed) and the number of elements allocated. The Setup( ) function does all the work:

template<class TYPE>

ArrStk<TYPE>::ArrStk(TYPE *m, unsigned n)

// Sets up a stack with n already-allocated elements

// pointed to by m

{

Setup(m, n);

}

template<class TYPE>

void ArrStk<TYPE>::Setup(TYPE *m, unsigned n)

{

start = m;

nxt = start;

if (start) { // Was memory actually allocated?

finish = start + n;

dimlen = n;

}

else {

finish = start;

dimlen = 0;

}

}

The Setup( ) function initializes the three pointers start, nxt, and finish, and the maximum stack size dimlen. The start and finish pointers mark the beginning and end of the stack, where finish actually points one element past the end. The nxt pointer references the next available stack element. The top of the stack--that is, the previous element pushed--can be found at nxt-1. When nxt equals start, the stack is empty. When nxt equals finish, the stack is full. Figure 9.5 shows the arrangement of an ArrStk object at various stages.

An element is pushed onto the stack by copy-constructing the data in place, at the location referenced by nxt, and then incrementing the nxt pointer, as given in the following Push( ) function. Figure 9.6(a) illustrates the pushing process.

template<class TYPE>

int ArrStk<TYPE>::Push(const TYPE &x)

// Pushes a copy of x onto the stack unless

// stack is full. Returns 1 if not full;returns 0 otherwise.

{

if (!IsFull( )) {

new(nxt++) TYPE(x); // In-place copy construction

return 1;

}

else return 0;

}

The Pop( ) function works in reverse, first decrementing the nxt pointer and then invoking an explicit destructor call. Figure 9.6(b) illustrates the popping process.

template<class TYPE>

int ArrStk<TYPE>::Pop(TYPE &x)

// Pops the top element from the stack, returning

// a copy of it in x, unless stack is empty. Returns 1

// if stack wasn't empty before the pop; otherwise, returns 0.

{

if (!IsEmpty( )) {

x = *(--nxt);         // Make a copy

nxt->TYPE::~TYPE( );  // Explicit destruction

return 1;

}

else return 0;

}

Figure 9.5 An ArrStk object at various stages.

(a) Empty

(b) With two elements

(c) Full

Figure 9.6 Using an ArrStk object.

(a) Pushing

(b) Popping

Both Push( ) and Pop( ) check to make sure the bounds of the stack aren't violated. To optimize speed, you may want to forgo these checks--but only if this is warranted! The following FastPush( ) and FastPop( ) functions are provided to bypass the stack checks. These functions can be easily inlined:

template<class TYPE>

void ArrStk<TYPE>::FastPush(const TYPE &x)

// For fastest, but unsafest pushes.

{

new(nxt++) TYPE(x); // In-place copy construction

}

template<class TYPE>

void ArrStk<TYPE>::FastPop(TYPE &x)

// For fastest, but unsafest pops.

{

x = *(--nxt);         // Make a copy

nxt->TYPE::~TYPE( );  // Explicit destruction

}

It's interesting to compare in-place construction and explicit destruction with the conventional method, where all elements remain constructed during the stack's life. Here's how FastPush( ) and FastPop( ) would be coded for the latter:

template<class TYPE>

void ArrStk<TYPE>::FastPush(const TYPE &x)

// If in-place construction isn't being used

// (assumes element already constructed)

{

x = *nxt++;

}

template<class TYPE>

void ArrStk<TYPE>::FastPop(TYPE &x)

// If in-place construction isn't being used

// (assumes element to remain constructed)

{

x = *(--nxt);

}

You can see that the in-place construction used during a push is replaced by an assignment and that the code for a pop is the same as before, except no destructor call is made. For many element types, an assignment and a copy construction involve roughly the same amount of work. Thus, the push operation will take basically the same amount of time regardless of the method used. For pop operations, more work is being done in the extra destructor call for the in-place construction method. The data type being stored determines whether this is a problem or whether the reverse situation (not doing a destructor call) is a problem.

The in-place construction method provides the most rigorous design and is probably worth using, even if there is a slight performance penalty. Of course, for built-in types, this is a moot point since copy construction reduces to an assignment, and the destructor becomes a no-op.

The Rewind( ) function is related to Pop( ), and lets you unpop n elements off the stack:

template<class TYPE>

void ArrStk<TYPE>::Rewind(unsigned n)

{

if (n == 0) n = dimlen; // Ensures all elements popped

// Explicit destructor called for each element popped

while(n-- && nxt != start) (--nxt)->TYPE::~TYPE();

}

Unlike Pop( ), the Rewind( ) function doesn't copy and pass back the data being popped from the stack. The data is simply discarded and destroyed. The Rewind( ) function can be used to clear the stack by unpopping all the elements. By convention, passing a number 0 to Rewind( ) (the default) results in the stack being cleared.

History( ) is another related function, and lets you inspect the element n positions back into the stack:

template<class TYPE>

TYPE *ArrStk<TYPE>::History(unsigned n)

{

if (IsEmpty( )) return 0;

TYPE *p = nxt-l;

while(1) {

if (n-- == 0) return p;     // Found the element

if (p-- == start) return 0; // Out of range

}

}

In the strictest sense, History( ) violates the rules of a stack. However, this function can be quite useful, when stacks are used in conjunction with tree-traversal algorithms. In the case of n = 0, the top element of the stack is inspected. The Top( ) function provides an efficient form for this special case:

template<class TYPE>

TYPE *ArrStk<TYPE>::Top()

{

return IsEmpty() ? 0 : (nxt-1);

}

To make ArrStk a concrete data type, we also provide an overloaded assignment operator to allow stacks to be copied. This operator function calls Copy( ) to do most of the work:

template<class TYPE>

void ArrStk<TYPE>::operator=(const ArrStk<TYPE> &s)

// Traps assignment to self

{

if (this != &s) Copy(s);

}

template<class TYPE>

void ArrStk<TYPE>::Copy(const ArrStk<TYPE> &s)

// Clears the stack, and then copies the elements

// of s onto the stack. Note that if s.dimlen >

// dimlen, not all elements will be copied.

{

Rewind();

TYPE *t = s.start;

unsigned n = dimlen;

// Elements copied via in-place copy construction

while (n- && t != s.nxt) new(nxt++) TYPE(*t++);

}

The Copy( ) function doesn't change the dimensioned length of the stack and, for that reason, a truncation may result during the copy.

You'll notice that a copy constructor is defined for ArrStk, but it is placed in the private section of the class. The constructor body (not shown) is actually  empty. This approach prevents the user from trying to create a stack via copy construction. As defined, the ArrStk class is not responsible for allocating the stack elements, so it can't allocate a new stack to hold a copy of another stack. Likewise, a virtual destructor is defined in ArrStk, but it does nothing. It's up to the derived classes to handle the de-allocation of the stack elements.

The remaining ArrStk functions are those that determine the maximum size of the stack and whether the stack is empty or full:

template<class TYPE>

unsigned ArrStk<TYPE>::Size( ) const

{

return dimlen;

}

template<class TYPE>

int ArrStk<TYPE>::IsEmpty( ) const

{

return nxt == start;

}

template<class TYPE>

int ArrStk<TYPE> : : IsFull ( ) const

{

return nxt == finish;

}

Statically Allocated Stacks

With the ArrStk class now defined, it's easy to derive a new class, SarrStk, that allocates the stack elements statically:

template<class TYPE, unsigned SIZE>

class SarrStk : public ArrStk<TYPE> {

protected:

char elem[sizeof (TYPE)*SIZE]; // Notice the char type!

public:

SarrStk();

SarrStk (const ArrStk<TYPE> &s);

SarrStk (const SarrStk<TYPE, SIZE> &s);

virtual ~SarrStk ();

void operator=(const ArrStk<TYPE> &s);

} ;

The SarrStk class uses the same strategy that was used for the statically allocated, variable-length arrays in Chapter 4. The stack elements are allocated and de-allocated as an array of characters. This approach defers the construction of the elements until they are actually used, and prevents them from being destroyed twice when the stack itself is destroyed. Here is one of the constructors, and the destructor, to illustrate this technique. In the destructor, the stack must be rewound so that we can call the destructors for all elements in use:

template<class TYPE, unsigned SIZE>

SarrStk<TYPE, SIZE>: :SarrStk()

: ArrStk<TYPE>((TYPE *) elem, SIZE)

{

// Nothing else to do

}

template<class TYPE, unsigned SIZE>

SarrStk<TYPE, SIZE>: :~SarrStk()

{

Rewind();

}

Complete code for the SarrStk class is given on disk in the file sarrstk.h. A test program tstsastk.cpp is also provided.

Dynamically Allocated Stacks

The DarrStk class defines stacks that are dynamically allocated. As was done with the SarrStk class, we use a typecasting trick so that the stack elements can be allocated without being constructed at the same time. This next class definition, followed by one of the constructors and the destructor, illustrates the technique. This time, the destructor must not only rewind the stack, but also must delete the elements:

template<class TYPE>

class DarrStk : public ArrStk<TYPE> {

public:

DarrStk(unsigned n);

DarrStk(const ArrStk<TYPE> &s);

DarrStk(const DarrStk<TYPE> &s);

virtual ~DarrStk();

void operator=(const ArrStk<TYPE> &s);

};

template<class TYPE>

DarrStk<TYPE>: :DarrStk(unsigned n)

// Elements are allocated as an array of characters

// to defer their construction until needed

: ArrStk<TYPE>( (TYPE *)new char[sizeof(TYPE)*n], n)

{

// Nothing else to do

}

template<class TYPE>

DarrStk<TYPE>::~DarrStk()

{

Rewind();

// Elements are deleted as an array of characters,

// since that's how they were allocated

delete[] (char *)start;

}

Complete code for the DarrStk class is given on disk in the file darrstk.h. The test program tstdastk.cpp is also provided.

 RESIZABLE STACKS

Fixed-size, array-based stacks provide the most efficient form of stack; however, it can be inconvenient to fix the size of the stack. A linked list is the obvious candidate to implement a resizable stack. We provided examples of this in Chapter 3. Unfortunately, when we use a linked list, a new node must be allocated from the heap with every push, and a node must be de-allocated with every pop. For some heap-implementations, this can be a time-consuming process. In Chapter 8, we demonstrated one way around this problem: implement an auxiliary list to cache nodes that would otherwise be freed.

A resizable array, as implemented in Chapter 4, provides another alternative. However, this approach presents a problem: each time the array grows beyond its allocated bounds, it has to be copied into a new location that has more room.

Stacks of Stacks

The RarrStk Class

Stacks of Stacks

There is an array-based solution that's better than all of the alternatives just presented. The idea is based on working with multiple stacks. As a stack becomes full, a switch is made to a new stack. As a stack becomes empty, a switch is made back to a previously used stack. This type of resizable stack can be implemented using a stack-of-stacks. Initially, the stack-of-stacks is empty. When the first element is to be added, a new substack is allocated and a pointer to it is pushed onto the stack-of-stacks. When the first substack becomes full, another substack is allocated and its pointer is pushed onto the stack-of-stacks. As elements are removed and a substack becomes empty, the substack is popped from the stack-of-stacks.

You can implement a stack-of-stacks using the DarrStk template, with the element type being the DarrStk type itself. In the following example, we'll declare such a stack, with room to hold mxstks substacks:

DarrStk< DarrStk<TYPE> > stkostk (mxstks);

When stkostk is first allocated, the individual substacks remain unconstructed. As a result, the only space initially occupied by stkostk is the space for the instance variables of DarrStk (start, finish, nxt, and dimlen) multiplied mxstks times. Substacks are constructed only when elements are pushed onto the stack. Here's how an element x of type TYPE can be pushed onto and popped from the stack:

// Push function:

if (!stkostk.IsFull()) {

if (stkostk.IsEmpty() || stkostk.Top()->IsFull()) {

stkostk.Push(DarrStk<TYPE>(blksize));

}

return stkostk.Top() ->Push(x);

}

else return 0 ;

// Pop function:

if (!stkostk.IsEmpty()) {

if (stkostk.Top()->IsEmpty()) {

stkostk.Rewind(1);

if (stkostk.IsEmpty()) return 0;

}

return stkostk.Top( )->Pop(x);

}

else return 0;

A working example of a stack-of-stacks is given on disk in the file stkostk.cpp.

The RarrStk Class

The nested template, stack-of-stacks approach works fine, but some stack operations--such as Rewind( ) and History( )--are somewhat difficult to implement under this approach. For this reason, we'll implement a stack-of-stacks more directly, using the following RarrStk class:

template<class TYPE>

class RarrStk {

protected:

TYPE **stkostk;             // Stack-of-stacks pointers

TYPE *nxt, *start, *finish; // Current stack variables

unsigned maxnelems, stksize, nxtstk, maxstks;

int Setup(unsigned nb);

int PushStk();

void PopStk(unsigned b);

void Dealloc();

public:

RarrStk(unsigned stksz, unsigned mxstks=1);

RarrStk(const RarrStk<TYPE> &s);

~RarrStk();

void operator=(const RarrStk<TYPE> &s);

int Copy(const RarrStk<TYPE> &s);

int Push(const TYPE &x);

int Pop(TYPE &x);

int Pop();

TYPE *Top();

const TYPE *Top() const;

TYPE *History (unsigned n);

const TYPE *History(unsigned n) const;

void Rewind(unsigned n = 0);

int IsFull() const;

int IsEmpty() const;

unsigned Size() const;

};

Complete code for the RarrStk class is given in the files rarrstk.h and rarrstk.mth. A test program is provided in the file tstrastk.cpp.

A RarrStk object is made up of an array of stack pointers called stkostk and one or more substacks. Each substack is allocated and de-allocated separately as needed, and stkostk keeps track of their locations. As is true with the ArrStk class, RarrStk uses the pointers nxt, start, and finish, but this time they point to elements in the current substack. When a switch is made to another substack, these pointers are modified accordingly. Figure 9.7 shows an example of a RarrStk object with two substacks allocated (but with room for four stacks).

The RarrStk constructor, which calls Setup( ), creates an empty stack with no substacks allocated, and the stkostk array is initialized to hold null pointers.

template<class TYPE>

RarrStk<TYPE>::RarrStk(unsigned stksz, unsigned mxstks)

{

stksize = stksz;

Setup(mxstks);

}

template<class TYPE>

int RarrStk<TYPE>::Setup(unsigned ns)

{

nxtstk = 0; nxt = 0; start = 0; finish = 0;

stkostk = new TYPE *[ns]; // Up to ns sub-stacks

if (stkostk) {

maxstks = ns;

while(ns) stkostk[-ns] = 0;

}

else maxstks = 0;

maxnelems = stksize * maxstks;

return maxstks > 0;

}

Figure 9.7 A partially filled RarrStk object.

When calling the constructor, you specify the size of each substack and the number of substacks that you wish to allow. Note that a RarrStk object does have a maximum size (equal to stksixe * maxstks), but unlike an ArrStk object, this size doesn't affect the initial amount of memory reserved. Initially, RarrStk only requires enough room to store the stkostk array of pointers.

Elements are pushed on the stack in the same way that we used previously, with one exception: if the current substack is full, a new substack is started. Figure 9.8 illustrates this process. The following Push( ) and PushStk( ) routines show how the pushing is done. You'll notice that in-place construction is used for the new element:

template<class TYPE>

int RarrStk<TYPE>::Push(const TYPE &x)

}

if (nxt == finish) {           // Need another stack

if (!PushStk()) return 0;  // Couldn't allocate

}

new(nxt++) TYPE(x);  // In-place copy construction

return 1;            // Success

}

template<class TYPE>

int RarrStk<TYPE>::PushStk( )

// Ensures room has been allocated for the next highest

// stack (ie: stk # nxtstk.) Returns 1 if successful;

// returns 0 if unable to allocate.

{

if (nxtstk == maxstks) return 0: // Table full

TYPE *p = stkostk[nxtstk];

if (p == 0) { // Block not already allocated

p = (TYPE *)new char[stksize*sizeof(TYPE)];

stkostk[nxtstk] = p; // Record stack pointer

}

// else stack still allocated from before

if (p) { // We have stack allocated

nxt = start = p;

finish = p + stksize;

nxtstk++;

return 1;

}

else { // Allocation error, so leave everything alone

return 0;

}

}

The PushStk( ) routine--responsible for ensuring that a new substack gets allocated--is somewhat tricky. First, the substack may already be allocated (more on this later), which we determine by checking for a non-null pointer in stkostk. Second, since we're using in-place construction, the new substack is allocated as an array of characters to prevent the element constructors from being called prematurely. Note how the variable nxtstk keeps track of the next available location in stkostk, and how it's possible for stkostk to become completely full.

Popping an element from the stack uses the reverse approach. If the current substack becomes empty when an element is popped off the stack, the sub-stack can be de-allocated. However, the substack shouldn't be de-allocated immediately. You should postpone the de-allocation to avoid the "thrashing" that can occur at the boundary of a substack. Consider what would happen if the first element of a substack is popped and a new element is then immediately pushed. If the substack were de-allocated as soon as it became empty, it would have to be immediately re-allocated when a new element is added. To avoid this, the following Pop( ) function de-allocates a substack only when the one precedes it becomes empty. When a substack is ready to be freed, the PopStk( ) function is called. Figure 9.9 illustrates the process.

Figure 9.8 Pushing onto a RarrStk object.

template<class TYPE>

int RarrStk<TYPE>::Pop(TYPE &x)

// Pops the top element from the stack, returning

// a copy of it in x. Returns 1 if stack wasn't empty

// before push; otherwise, returns 0. If stack was empty, the

// data in x will be untouched.

{

if (nxt == start) {

// Pop next higher stack if this has not already been done

PopStk(nxtstk);

// Return if stack is empty, but note that there may

// still be a stack allocated (when nxtstk = 1)

if (nxtstk <= 1) return 0;

// Otherwise, we must move to end of lower stack

-nxtstk; // nxtstk guaranteed >= 1 after decrement

start = stkostk[nxtstk-1];

finish = start + stksize;

nxt = finish - 1;

}

else -nxt;

x = *nxt;

nxt->TYPE::~TYPE( );

return 1;

}

template<class TYPE>

void RarrStk<TYPE>::PopStk(unsigned b)

// De-allocates stack b if not already de-allocated,

// destructing elements along the way if INPLACECONST

// isn't true. Sets pointer in stack table

// to null. Does nothing if b is out of range.

{

if (b >= maxstks) return; //  b is out of range

if (stkostk[b] == 0) return;

delete[] (char *)(stkostk[b]);

stkostk[b] = 0;

}

The RarrStk class contains a full complement of constructors, a destructor, and an overloaded assignment operator. As is the case with the ArrStk class, RarrStk includes functions like IsEmpty( ), IsFull( ), History( ), Top( ), Rewind( ), and so on. The details of these functions are given on disk.

Stacks based on the RarrStk class are quite efficient, but they are slightly slower than those based on ArrStk--due to the need to allocate and de-allocate sub-stacks. This problem can be alleviated by increasing the size of the blocks, and thus reducing the frequency of substack allocations. Note that RarrStk objects are significantly faster than a linked list approach. The RarrStk objects also occupy less space, since each element doesn't need an extra link pointer. The only disadvantage is a slight increase in the code required (which could be magnified if many classes are instantiated from the template).

Figure 9.9 Delayed de-allocation of a substack.

To allow you to compare RarrStk objects with linked-list based stacks, we've provided the following additional files on disk: liststk.h, liststk.mth, tstlstk.cpp, and stktimer.cpp. The latter program tests the runtime efficiency of the two approaches.

 LIST-BASED QUEUES

We'll now focus on the implementations of queues. We'll begin by discussing list-based queues, since they are the easiest to implement. Next, we'll look at compact array-based implementations. Recall that, with a queue, elements are inserted at the rear of the queue and extracted from the front. A singly-linked list with a back pointer can handle these operations quite nicely. There are different methods we could use:

1.     Implement a linked-list queue class from scratch. This is the most direct method. However, with this approach we must duplicate a lot of list functionality that could be reused for other lists.

2.     Use the Slist class of Chapter 8 as is, without creating a special queue version. We can use functions like AddToBack( ) and DelFront( ) to support the queue operations. However, all the other list operations are available as well, such as DelNext( ), which violate restricted access to the ends of the queue. Whether this is a problem depends on how strictly you wish to follow the definition for a queue.

3.     A queue-specific class template can be derived from the Slist class. We can use private derivation, and thus control which list functions are available. By using inline functions, we can change the names of AddToBack( ) and DelFront( ) to more high-level names such as Insert( ) and Extract( ). This approach gets messy because an extra template class, Slist, must be instantiated, plus we run into all sorts of protected-access problems. Of course, if you need the Slist class for some other part of your program, this approach may be best since you'll be reusing the Slist class.

4.     A queue-specific class template can be derived from the Slistb base class. This represents a compromise of the other three approaches. By deriving from Slistb, we allow much of the code to be reused. Yet, because the derivation is direct, we'll have a simpler implementation than if we had used the Slist class.

For this book, we've chosen the fourth method, since it is perhaps the best engineering tradeoff. However, the method that works best for you might depend on a lot of factors; don't hesitate to implement queues using your own methods.

We can derive a ListQ class from Slistb, whose purpose is to implement listbased queues:

template<class TYPE>

class ListQ : public Slistb {

protected:

virtual Snode<TYPE> *AllocNode(const TYPE &x);

virtual Snodeb *DupNode(const Snodeb *n);

virtual void FreeNode(Snodeb *n);

public:

ListQ();

ListQ(const ListQ<TYPE> &s);

~ListQ();

void operator=(const ListQ<TYPE> &s);

int Extract(TYPE &x);

int Extract();

int Insert(const TYPE &x);

TYPE *Head();

const TYPE *Head() const;

int IsFull() const;

};

Complete code for the ListQ class is given on disk in the files listq.h, listq.mth, and snode.h. A test program is given in tstlq.cpp.

The ListQ class is very similar to the Slist class, but has fewer functions. For instance, we chose not to support queue concatenations, even though this might be useful in some cases. As was true with the Slist class, ListQ allocates nodes on the heap. We could, of course, use the free-list technique for node caching or use array-based allocation.

The most important functions of ListQ are Insert( ) and Extract( ), which implement the basic queue operations:

template<class TYPE>

int ListQ<TYPE>::Insert(const TYPE &x)

// Creates a new node having a copy of x for its data

// and puts in on the back of the queue. Returns 1

// if allocation successful; otherwise, returns 0.

{

Snode<TYPE> *p = AllocNode(x);

if (p == 0) return 0;

AttachToBack(p);

return 1;

}

template<class TYPE>

int ListQ<TYPE>::Extract(TYPE &x)

// Gets the data from the front node of the queue and

// copies it into x. If queue is empty, x is left

// untouched and a 0 is returned; otherwise, returns a 1.

{

Snode<TYPE> *p = (Snode<TYPE> *)RmvFront( );

if (p == 0) return 0;

x = p->info;

FreeNode(p);

return 1;

}

The Insert( ) and Extract( ) functions use the AttachToBack( ) and RmvFront( ) functions of Slistb, as well as the virtual functions FreeNode( ) and AllocNode( ).

Sometimes it's useful to peek at the front element in the queue without actually removing it. The Head( ) function is used for that purpose:

template<class TYPE>

TYPE *ListQ<TYPE>::Head( )

// Returns the data contained in the head node of the queue,

// or a null pointer if the queue is empty

{

return IsEmpty() ? 0 : &(((Snode<TYPE> *)Front())->info);

}

Figure 9.10 illustrates the process of adding and removing elements from a ListQ object.

 LIST-BASED STAQUES

You can add stack-like operations to the ListQ class and derive a combination stack and queue (staque) class. Alternatively, you can derive the staque class from a stack class or, by using multiple inheritance, combine a stack and queue class. In the first case, the Pop( ) function would be identical to Exctract( ) and the Push( ) operation could be implemented by making a call to the AddToFront( ) function of Slistb. The details for this are given on disk.

Complete code for a list-based staque class, ListSQ, is given on disk in the files listsq.h and listsq.mth. This class is derived from a list-based stack class, ListStk, which is given in the files liststk.h and liststk.mth. A test program is provided in tstlsq.cpp.

 LIST-BASED DEQUES

Here, a singly-linked list with a back pointer isn't quite up to the job. The problem is that we can't efficiently remove the last node from the list, because we must scan the list from beginning to end to find the node immediately preceding the last one.

Figure 9.10 Using a ListQ object.

You can either choose to accept this inefficiency or use a doubly-linked list. Of course, a doubly-linked list has its own inefficiency, costing an extra pointer per node. As always, you are faced with the classic speed versus space tradeoff. Here, sacrificing space for speed is probably the better choice. The space required by a queue is rarely an issue. However, the speed to remove an element might be. (If space is an issue, an array-based queue is a better approach, as you'll see later on.)

By using the doubly-linked list approach, we can implement a deque with the following ListDQ class, which is derived from Dlistb:

template<class TYPE>

class ListDQ : public Dlistb {

protected:

virtual Dnode<TYPE> *AllocNode (const TYPE &x);

virtual Dnodeb *DupNode(const Dnodeb *n);

virtual void FreeNode(Dnodeb *n);

public:

ListDQ();

ListDQ(const ListDQ<TYPE> &s);

~ListDQ();

void operator=(const ListDQ<TYPE> &s);

int ExtractHead(TYPE &x);

int ExtractTail(TYPE &x);

int InsertHead(const TYPE &x);

int InsertTail(const TYPE &x);

TYPE *Head();

const TYPE *Head() const;

TYPE *Tail();

const TYPE *Tail() const;

int IsFull() const;

};

Complete code for the ListDQ class is given in the files listdq.h, listdq.mth, and dnode.h. A test program is provided in the file tstldq.cpp.

The design of the ListDQ class is similar to ListQ, but splits the Insert( ) and Exctract( ) functions into four functions-InsertHead( ), InsertTail( ), ExtractHead( ), and ExtractTail( ):

template<class TYPE>

int ListDQ<TYPE>::InsertHead(const TYPE &x)

{

Dnode<TYPE> *p = AllocNode(x);

if (p == 0) return 0;

AttachToFront(p);

return 1;

}

template<class TYPE>

int ListDQ<TYPE>::InsertTail(const TYPE &x)

{

Dnode<TYPE> *p = AllocNode(x);

if (p == 0) return 0;

AttachToBack(p);

return 1;

}

template<class TYPE>

int ListDQ<TYPE>::ExtractHead(TYPE &x)

{

Dnode<TYPE> *p = (Dnode<TYPE> *)RmvFront( );

if (p == 0) return 0;

x = p->info;

FreeNode(p);

return 1;

}

template<class TYPE>

int ListDQ<TYPE>::ExtractTail(TYPE &x)

{

Dnode<TYPE> *p = (Dnode<TYPE> *)RmvBack( );

if (p == 0) return 0;

x = p->info;

FreeNode(p);

return 1;

}

 ARRAY-BASED QUEUES

List-based queues are especially useful because you don't have to pre-allocate storage for the maximum size of the queue. However, if the queues you are using tend to be of some fixed size, then arrays offer more compact representations. Such array-based queues will be discussed next.

The simplest way to implement an array-based queue is to have the first element of the array be the head of the queue, and utilize a pointer to the tail of the queue. It's easy to insert an element into this type of queue. We simply increment the tail pointer to the next available slot. However, we run into a problem in extracting from the front of the queue, because doing so means all remaining elements of the array must be shifted up--a time-consuming process. Figure 9.11 illustrates this inefficiency.

Figure 9.11 An inefficient array-based queue.

An alternative is to use a chasing pointer technique. The idea is to add another pointer to the head of the queue. The head pointer is initialized to the start of the array. As elements are inserted, the tail pointer is incremented. As elements are extracted, the head pointer is incremented, chasing the tail pointer. With this strategy, both insertion and extraction are efficient, as illustrated in Figure 9.12.

The chasing pointer technique does present some problems. As elements are extracted, the storage space at the beginning of the array is discarded and never reused. Also, the head and tail pointers are always incremented (but never decremented). Thus, even if the number of extractions equals the number of insertions, the array eventually runs out of space.

To solve these problems, we can use a circular array, also called a ring buffer. In such an array, the end wraps around to the beginning. Circular arrays don't physically exist. Instead, they are simulated by using normal linear arrays, where all pointers or indexes scanning the array are reset to the starting element when the ending element has been passed. Figure 9.13 illustrates this wraparound effect.

The ArrQ Class

Next, we'll develop an array-based queue class, ArrQ, that uses the circular array technique. The design for this class is very similar to the ArrStk class, in the following ways:

       The ArrQ class isn't responsible for allocating memory for the queue. The derived classes SarrQ and DarrQ determine whether the memory is allocated statically or dynamically. These classes are similar to SarrStk and DarrStk.

       Pointers are used to represent the head and tail of the queue, and also for the actual bounds of the underlying array. This is more efficient than using indexes, since we can avoid the multiplication needed by subscripting.

       In-place construction is used when elements are added and explicit destruction is used when elements are removed. This approach leads to a more rigorous and robust design.

       The class is a concrete data type implemented as a template, with the appropriate constructors, destructor, and overloaded assignment operator.

Figure 9.12 An array-based queue with chasing pointers.

Figure 9.13 A circular array with wrapround pointers.

Here is the ArrQ class definition:

template<class TYPE>

class ArrQ {

protected:

TYPE *st,   // Pointer to start element

*fn,   // Pointer to finish element

*hd,   // Pointer to head element in the queue

*nx;   // Pointer to next available element

int isfull; //  Flag to indicate a full queue

unsigned dimlen;  // Maximum number of elements

void Setup(TYPE *m, unsigned n);

ArrQ(const ArrQ<TYPE> &); // To disallow copy constructing

TYPE *Inc(TYPE *x) const;

TYPE *Dec(TYPE *x) const;

public:

ArrQ(TYPE *m, unsigned n);

virtual ~ArrQ();

void operator=(const ArrQ<TYPE> &s);

int Copy(const ArrQ<TYPE> &s);

void Clear();

int Extract(TYPE &x);

int Extract();

int Insert(const TYPE &x);

TYPE *Head();

const TYPE *Head() const;

int IsEmpty() const;

int IsFull() const;

unsigned Size() const;

};

Complete code for the ArrQ class is given on disk in the files arrq.h and arrq.mth. Static and dynamic derived classes are given in the files sarrq.h and darrq.h. Test programs can be found in the files tstsaq.cpp and tstdaq.cpp.

Four pointers are used to represent the queue. The variable st points to the starting element in the array,  fn points to the last element in the array, hd points to the logical start of the queue, and nx points to the next element available for insertions. The tail of the queue can be found at location nx-1. The ArrQ constructor sets up these pointers by calling Setup( ).

template<class TYPE>

ArrQ<TYPE>::ArrQ(TYPE *m, unsigned n)

// Constructs a new queue with a maximum of n elements

// stored at location m, assumed to be already allocated

{

Setup(m, n);

}

template<class TYPE>

void ArrQ<TYPE>::Setup(TYPE *m, unsigned n)

{

st = m;

if (st && n) { // Array isn't null

fn = st+n-1;

dimlen = n;

isfull = 0;

}

else { // A null array, which is always 'full'

fn = st;

dimlen = 0;

isfull = 1;

}

hd = st;

nx = st;

}

Figure 9.14(a) shows the configuration of an empty queue as created by Setup( ). Note how the nx and hd pointers are equal at this stage. The condition nx==hd is one way to test for an empty queue. However, this condition also occurs when the queue is full, as illustrated by Figure 9.14(b).

We can solve this ambiguity in two ways. Under one approach, we never let the array fill up entirely, holding the last remaining element in reserve. (The location of this element rotates through the array as the queue is accessed.) In this scheme, when nx ==hd-1, the queue is said to be full. Unfortunately, the reserved element represents wasted space, which can be costly if the array elements are large. Under the second approach, we maintain a flag (which we call isfull) that is set when the queue becomes full during an insertion and reset as elements are extracted. The following IsEmpty( ) and IsFull( ) functions show how the isfull flag resolves the empty/full ambiguity:

Figure 9.14 Ambiguous empty and full conditions.

(A) Empty

(B) Full

template<class TYPE>

int ArrQ<TYPE>::IsEmpty( ) const

{

return hd == nx && !isfull;

}

template<class TYPE>

int ArrQ<TYPE>::IsFull( ) const

{

return isfull;

}

By maintaining the isfull flag, we avoid wasting one of the array elements, but we do create some complications. Here are the Insert( ) and Extract( ) functions, which illustrate how the hd and nx pointers are updated, and how the isfull flag is maintained:

template<class TYPE>

int ArrQ<TYPE>::Insert(const TYPE &x)

// Copies x into the next available element at the end

// of the queue. If queue is full, nothing happens.

{

if (IsFull( )) return 0;

new(nx) TYPE(x);

nx = Inc(nx);

if (nx == hd) isfull = 1; // We just went full

return 1;

}

template<class TYPE>

int ArrQ<TYPE>::Extract(TYPE &x)

// Gets the data from the head element and stores a copy of

// it in x. Removes and possibly destroys the head element.

// If queue is empty, x is left untouched and a

// 0 is returned; otherwise, a 1 is returned.

{

if (IsEmpty()) return 0;

x = *hd;

hd->TYPE::~TYPE();

hd = Inc(hd); // We have a new head

isfull = 0;   // And we certainly can't be full anymore

return 1;

}

The functions Inc( ) and Dec( ) are used to conveniently wrap the pointers around when we reach the ends of the array:

template<class TYPE>

TYPE *ArrQ<TYPE>::Inc(TYPE *x) const

// Increments a pointer and wraps around to

// the start if necessary

{

return (x == fn) ? st : x+1;

}

template<class TYPE>

TYPE *ArrQ<TYPE>::Dec(TYPE *x) const

// Decrements a pointer, wrapping around to

// the end if necessary

{

return (x == st) ? fn : x-1;

}

Array-Based Staques

We can now derive a new class that adds stack-like operations to the ArrQ class, similar to the way used for list-based stacks. Such a class, ArrSQ, is given on disk.

Complete code for an array-based staque class is given on disk in the files arrsq.h and arrsq.mth. Static and dynamic versions, SarrSQ and DarrSQ, are provided in the files sarrsq.h and darrsq.h. Two test programs are provided in tstsasq.cpp and tstdasq.cpp.

The ArrDQ Class

A double-ended queue class is easy to derive from ArrQ, as the following ArrDQ class shows:

template<class TYPE>

class ArrDQ : public ArrQ<TYPE> {

protected:

ArrDQ(const ArrDQ<TYPE> &s);

public:

ArrDQ(TYPE *m, unsigned n);

void operator=(const ArrQ<TYPE> &s);

int ExtractHead(TYPE &x);

int ExtractTail(TYPE &x);

int InsertHead(const TYPE &x);

int InsertTail(const TYPE &x);

TYPE *Tail( );

const TYPE *Tail( ) const;

};

Complete code for the ArrDQ class is given on disk in the files arrdq.h and arrdq.mth. Static and dynamic classes, SarrDQ and DarrDQ, are given in the files sarrdq.h and darrdq.h. Test programs are provided in the files tstsadq.cpp and tstdadq.cpp.

The Insert( ) and Extract( ) functions of the ArrQ class are replaced by InsertHead( ), InsertTail( ), ExtractHead( ), and ExtractTail( ).

template<class TYPE>

int ArrDQ<TYPE>::InsertHead(const TYPE &x)

// Puts a new element at the front of the queue, holding

// a copy of x

{

if (IsFull( )) return 0;

hd = Dec(hd);

new(hd) TYPE(x);

if (nx == hd) isfull = 1;

return 1;

}

template<class TYPE>

int ArrDQ<TYPE>::InsertTail(const TYPE &x)

// Puts a new element at the end of the queue, holding a

// copy of x

{

return ArraQ<TYPE>::Insert(x);

}

template<class TYPE>

int ArrDQ<TYPE>::ExtractHead(TYPE &x)

// Copies the data of the front element into x and

// removes the element from the queue

{

return ArrQ<TYPE>::Extract(x);

}

template<class TYPE>

int ArrDQ<TYPE>::ExtractTail(TYPE &x)

// Copies the data of the rear element into x and

// removes the element from the queue

{

if (IsEmpty( )) return 0;

nx = Dec(nx);

x = *nx;

nx->TYPE::~TYPE( );

isfull = 0;

return 1;

}

Note that we could have derived the ArrDQ class from the ArrSQ class, since we would only need to add the function to remove elements from the end of the queue. We chose not to do this because the chain of template-based derivations (ArrQ to ArrSQ to ArrDQ to SarrDQ and DarrDQ) starts to get a little unwieldy. The derivation of ArrDQ directly from ArrQ eliminates one class from this chain.

 HETEROGENOUS STACKS AND QUEUES

The discussion so far has implicitly assumed that the stacks and queues store only one kind of object, and are thus homogenous. However, you will probably have situations where it is useful to store different types of objects. Any of the techniques given earlier in Chapters 5 and 8 for creating heterogenous arrays and linked-lists could also be used for stacks and queues. As always, you can accomplish this by storing indirect pointers to objects, perhaps using reference-counted smart pointers to provide the safest designs.

 SUMMARY

In this chapter, you've seen how to efficiently implement stacks and queues by using two opposing data structures: arrays and linked lists. Both forms offer advantages. Linked lists allow more flexibility for variable stack and queue lengths, while arrays offer more compact representations when a fixed size is acceptable. The RarrStk class provides a very efficient implementation of resizable stacks by using a stack-of-stacks. Note that resizable array-based queues could be implemented in a way similar to the RarrStk class, using a queue of queues. The details are somewhat messy, though, because the deferred allocation and deallocation of the subqueues must be handled in both directions.

Perhaps the most important design feature presented in this chapter is the use of in-place construction and explicit destruction of elements when they are added and removed from the stacks and queues. Most texts on C++ do not mention this technique, yet in-place construction does lead to a more robust design.

Go to Chapter 10            Return to Table of Contents


Practical Data Structures: PREFACE

 Return to Table of Contents

PREFACE

This book is about designing practical data structures. The practical orientation of this book is due, in no small part, to the fact that I was first trained as a mechanical engineer, and so my tendency is to design things from an engineering point of view, always balancing the ideals of theory with the reality of practice. The engineer in me has always been eager to understand the trade-offs involved in using one approach over another.

This book is the first part of a planned two-volume set. A subsequent volume, Practical Algorithms in C++, will pick up where this book leaves off, using the data structures designed in this book as tools to be used in many different algorithms. This second volume will be written from the same practical perspective.

In writing these two books, I hope to impart to you some of the interest and knowledge I have gained over the years on the subjects of data structures and algorithms, and to show you how elegantly and efficiently these constructs can be implemented in C++.

HOW THIS BOOK IS DIFFERENT

A quick trip down to your local bookstore will show you that surprisingly few books cover the topic of data structures, let alone show you how to code them in C++. This is surprising because data structures are such a fundamental part of the programming process. In the books that do exist on data structures, the subject matter is covered more from a theoretical viewpoint, with sparse and incomplete coding examples. Many of these books leave a lot of the more interesting and complicated techniques as exercises for the reader.

This book, in contrast, was designed to cover data structures from a practical point of view, with as many complete examples as possible. There are no exercises left for the reader. You won't have to spend days trying to figure out how to accomplish something described in the book. If you're like me, you want to put the techniques to use immediately, to see if they are promising for your own applications. Only then do you want to dissect the code to see what makes it tick.

The data structures shown in this book are the most efficient, elegant, and robust designs that I know of. I didn't just code up the data structures using the first approach that came to mind. Instead, many different designs were implemented and studied to find those that seemed to work the best.

The data structures were crafted specifically for C++. This isn't just a rewrite of a C book with a few programs changed to use classes. With each data structure, I spent much time (literally months) trying to determine how C++ could best be put to use.

Since it's not feasible to show all of the code in the manuscript itself (due to space limitations), only the salient features of each technique are given. The disk included with this book has everything typed in, ready to run. Many of the programs on disk are designed to thoroughly test the data structures presented in this book.

WHO SHOULD READ THIS BOOK

This book is written primarily for intermediate-level C++ programmers. However, the book has been designed to be useful to both those programmers with little C++ experience as well as those at an advanced level. This is accomplished by approaching each subject at a basic level, providing simple techniques, and then progressing to more exotic and advanced techniques. Thus, if you are at the starting point of the C++ learning curve, you can work with the early parts of each chapter. As you become more proficient, you can tackle the more advanced sections. If you're a more experienced programmer, you can use the first parts of each chapter as refresher material, and then quickly move on to the advanced material.

I assume that you know the basics of C++. The focus is on learning how to code data structures, not on learning C++. One exception is the use of templates, a feature relatively new to C++ that allows you to defne parameterized types. I devote the better part of a chapter to templates, since they are used quite heavily throughout the book. I use templates because they allow me to show data structures in a generic way, and they make it easy for you to "plug-in" the specific types of objects you wish to store in the data structures.

WHY C++?

In my early years of programming, I was somewhat frustrated by the lack of a programming language that was both expressive and efficient enough for day-to-day use. The languages at hand were either high level but unfortunately too slow, or fast and regrettably too low level. Then came C++, the language I had been looking for. C++ is both expressive enough to code data structures in an elegant way, yet low level enough to allow efficient programs to be constructed.

Traditionally, most books covering data structures have used Pascal or some other Algol-like language. Pascal has been heavily favored, due to its reputation as a good language for teaching. The problem is that many professional programmers do not use Pascal, but rather C and C++. That's because standard Pascal lacks many features needed for writing real-world applications. Many dialects of Pascal have been invented to make up for these deficiencies, but these dialects are not standard, and you are tied to specific compiler vendors for their implementations.

C, on the other hand, is powerful enough to be used for a wide range of applications, without needing wildly divergent dialects. But because of C's relative terseness and low-level characteristics, many authors have rejected using C as a language for teaching data structures. However, C++ overcomes many of these difficulties, due to its superior type checking and encapsulation facilities. It does this without losing the efficiency of C. Thus, I feel C++ makes an excellent choice for showcasing the most practical data structures in current use.

HOW THIS BOOK IS ORGANIZED

This book includes fourteen chapters, progressing from the basics to more advanced material. The chapters are:

1. Basic Concepts looks at the design of data structures from a very general perspective, showing what all data structures have in common. Also, you learn the ways that you can judge the effectiveness and practicality of one data structure over another.

2. Using C++ Effectively provides a review of the features of C++ as they apply to the design of data structures. The emphasis is on creating safe, robust, and efficient classes.

3. Data Structure Design in C++ discusses the four main approaches to data structures in C++. The difference between concrete data types and abstract data types is explained, along with a thorough discussion of templates, as well as the alternative approach of using single- rooted hierarchies.

4. Basic Array Design takes you step by step through the design of efficient, yet safe arrays. Covered are fixed-length, variable-length, resizable, statically allocated, and dynamically allocated arrays. Techniques for creating multidimensional arrays are also given.

5. Heterogenous Arrays discusses the issues involved in designing arrays to hold different types of data. Integral to this discussion is the use of "smart pointers" to reference-counted objects.

6. Strings combines the techniques explained in Chapters 4 and 5 in designing a safe and efficient string class.

7. Vectors and Matrices shows a relatively new way to design vector and matrix data structures using the notion of vector strides and shared representations of the underlying matrix data. You'll see objects invented by the author, called vector pointers, which allow unique and efficient implementations of sub-matrices and matrix transpositions.

8. Linked Lists takes you step by step through the design of linked lists as high-level objects in their"own right. Contiguous (array-based) and non-contiguous (pointer-based) implementations are given for both singly-linked and doubly-linked lists. You'll see ways to improve the efficiency of allocating and de-allocating nodes used for linked lists.

9. Stacks and Queues takes a detailed look at the design of stacks and queues, two related data structures, with efficiency and flexibility in mind. Both array-based and list-based designs are given.

10. Binary Trees introduces another fundamental data structure, trees, often used in database searching applications. This chapter focuses on binary trees and, in particular, binary search trees. You'll learn how binary trees can be searched and how to efficiently traverse and print trees.

11. Balanced Trees is a continuation of Chapter 10 and describes how to balance search trees to improve their performance. Covered are 2-3-4 trees and their binary analogues, red-black trees.

12. Splay Trees shows an intriguing type of self-organizing binary search tree. These trees have amortized costs that are equivalent to ordinary binary trees, yet splay trees can adapt to changing situations to provide very efficient searching. For instance, you'll see how splay trees can be used to implement priority queues.

13. File-Based Objects shows a unique and never before published technique of caching objects to disk. A new type of pointer, cached-object pointers (Coptrs), is implemented. With Coptrs, you can use cached objects almost like objects residing totally in memory. A file-based object heap allocator is also given, which works in conjunction with the caching.

14. B-trees is the culmination of the book and discusses perhaps the most widely used data structure for database applications, B-trees. An implementation is given for this important data structure that works from disk and uses the file-based caching techniques explained in Chapter 13.

WHAT YOU NEED

Every attempt was made to use generic C++ code in this book, with little that relies on any particular machine or compiler vendor. The code was written to be compliant with the AT&T C++ 3.0 specification. You'll need a compiler that's at least up to the AT&T C++ 2.1 specification and that also supports templates and the user-defined syntax of built-in types. If your compiler does not support templates, it's still possible to use the code in this book, though you'll have to make modifications to remove the template syntax. Chapter 3 discusses how to make such modifications.

The disk that's included with this book is formatted for MS-DOS, and project files are supplied so that you can easily compile the examples using Borland C++ 3.1, currently the only MS-DOS compiler that supports templates. (Many other compiler vendors will no doubt have templates implemented by the time you read this.) The code was extensively tested using Borland C++ 3.1, and was also tested (but not extensively) on Unix using the latest version of Cfront possible.

ACKNOWLEDGMENTS

I would like to thank Holly Mathews for her support and understanding in seeing me through the massive undertaking that was this book. Thanks go as well to my editor Paul Farrell at Wiley for the seemingly infinite amount of patience required to see this book come to fruition. I would also like to thank the production people at the Coriolis Group for doing a fine job on the layout. Special thanks go to technical reviewer Michael Mohle for providing a "sanity check" of the techniques presented, and for his efforts in ensuring that the code runs under Unix. Finally, my apologies to Dog Zilla, for all the walks he missed as I was writing this book.

CONTACTING THE AUTHOR

If you have questions about the contents of this book, you may contact me by mail at the address: Azarona Software, Practical Data Structures in C++, P.O. Box 768, Conifer, CO 80433. I can also be contacted on CompuServe[73057,3172], and on Bix [bflamig].

Bryan Flamig

Conifer, CO

February, 1993

Go to Chapter 1 Return to Table of Contents


Practical Data Structures: REFERENCES

 Return to Table of Contents

REFERENCES

BALANCED TREES

[Aoe 91] Computer Algorithms: Key Search Strategies, (IEEE Computer Society Press), Edited by J. Aoe, 1991. ISBN 0-8186-2123-0.

[BayeMc 72] "Organization and maintenance of large ordered indexes," R. Bayer and C. McCreight, Acta Informatica, Vol. 1, No. 3, 1972, pp. 173-189.

[GuibaSedge 78] "A Dichromatic Framework for Balanced Trees," L. Guibas and R. Sedgewick, Proceedings of the 19th Annual Symposium on Foundations of Computer Science, IEEE, 1978.

[SleatorTar 85] Self-Adjusting Binary Search Trees, D. Sleator and R. Tarjan, Journal of the ACM, Vol. 32, No. 3, July, 1985, pp. 652-686.

C++ PROGRAMMING

[Coplien 92] Advanced C++ Programming Styles and Idioms, (Addison Wesley), J. Coplien, 1992. ISBN 0-201-54855-0.

[Stroustrup 91] The C++ Programming Language, 2nd Edition, (Addison Wesley), B. Stroustrup, 1991. ISBN 0-201-53992-6.

GENERAL ALGORITHMS AND DATA STRUCTURES

[Flamig 93] Practical Algorithms in C++, (John Wiley & Sons), B. Flamig, Fall, 1993.

[HoroSahni 90] Fundamentals of Data Structures in Pascal, 3rd Edition, (Computer Science Press), E. Horowitz and Sartaj Sahni, 1990. ISBN 0-7167-8217-0.

[Knuth 73] The Art of Computer Programming: Fundamental Algorithms, Vol. 1, (Addison Wesley), D. Knuth, 1973.

[Knuth 73b] The Art of Computer Programming: Sorting and Searching, Vol. 3, (Addison Wesley), D. Knuth, 1973. ISBN 0-201-03803-X.

[Lings 86] Information Structures: A uniform approach using Pascal, (Chapman and Hall Computing), B. Lings, 1986. ISBN 0-412-26500-1.

[Mohle 92] Private communications with M. Mohle about general design priciples for data structures and algorithms, 1992.

OBJECT-ORIENTED DESIGN

[Booch 91] Object Oriented Design with Applications, (The Benjamin/Cummings Publishing Company), G. Booch, 1991.

[RumbBla 91] Object-Oriented Modeling and Design, (Prentice Hall), J. Rumbaugh, M. Blaha, W. Premerlani, F. Eddy, and W. Lorenson, 1991.

PRIORITY QUEUES

[Atk 86] "Min-Max Heaps and Generalized Priority Queues," M. Atkinson, J. Sack, N. Santoro, and T. Strothotte, Communications of the ACM, Vol. 29, No. 10, October, 1986, pp. 996-1000.

[Jones 86] "An Empirical Comparison Of Priority-Queue and Event-Set Implementations," D. Jones, Communications of the ACM, Vol. 29, No. 4, April 1986, pp. 300-311.


Practical Data Structures: Table of Contents

Practical Data Structures in C++

by Bryan Flamig

PREFACE

CHAPTER 1: BASIC CONCEPTS

CHAPTER 2: USING C++ EFFECTIVELY

CHAPTER 3: DATA STRUCTURE DESIGN IN C++

CHAPTER 4: BASIC ARRAY DESIGN

CHAPTER 5: HETEROGENOUS ARRAYS

CHAPTER 6: STRINGS

CHAPTER 7: VECTORS AND MATRICES

CHAPTER 8: LINKED LISTS

CHAPTER 9: STACKS AND QUEUES

CHAPTER 10: BINARY TREES

CHAPTER 11: BALANCED TREES

CHAPTER 12: SPLAY TREES

CHAPTER 13: FILE-BASED OBJECTS

CHAPTER 14: B-TREES

REFERENCES


Reliable Data Structures: [1-1] List of Reliability Rules

 Return to Table of Contents

[1-1] List of Reliability Rules

The reliability rules from the text are summarized in this list, with section-number references to the discussion of the rule.

Rule 1-1 (Section 1.1): Any macro definition containing operators needs parentheses around the entire definition. Each appearance of a macro argument in the definition also needs to be parenthesized if an embedded operator in the argument could cause a precedence problem.

Rule 1-2 (Section 1.1): Reliable modification of defined constants requires an environmental capability: there must be a means for ensuring that all files comprising a program have been compiled using the same set of headers. The UNIX make command is one such capability.)

Rule 1-3 (Section 1.1): If there are limitations on the modifiability of a defined constant, indicate the limitations with a comment:

#define EOF (-1)  /* DO NOT M0DIFY: ctype.h expects -1 value */

Rule 1-4 (Section 1.1): If one definition affects another, embody the relationship in the definition; do not give two independent definitions.

Rule 1-5 (Section 1.1): If a value is given for a #defined name, do not defeat its modifiability by assuming its value in expressions.

Rule 1-6 (Section 1.1): Use limits.h for environment-dependent values.

Rule 1-7 (Section 1.2): Use a consistent set of project-wide defined types.

Rule 1-8 (Section 1.3): Be sure that all functions are declared before use; headers are the most reliable way.

Rule 1-9 (Section 1.3): Create a project-wide "local" header for standard definitions and inclusions.

Rule 1-10 (Section 1.4): Use UPPERCASE names for unsafe macro functions, to emphasize the restrictions on their usage.

Rule 1-11 (Section 1.4): Never invoke an unsafe macro with arguments containing assignment, increment/decrement, or function call.

Rule 1-12 (Section 1.4): Whenever possible, use safe macro functions.

Rule 1-13 (Section 1.6): Use #if 0 if there is a need to comment-out sections of code.

Rule 1-14 (Section 1.6): Enclose each header in an "inclusion sandwich."

Rule 2-1 (Section 2.1): When exactness counts in converting floating-point to integer, be sure the value being converted is non-negative.

Rule 2-2 (Section 2.1): Test errno before using results from the math functions.

Rule 2-3 (Section 2.3): Use the <ctype.h> facilities for character tests and upper-lower conversions.

Rule 2-4 (Section 2.4): Make sure that Boolean variables are assigned the values zero and one. This means that the type tbool is always adequate, and if this rule is part of local standards, the types bool and tbool could be made synonymous.

Rule 2-5 (Section 2.4): Make sure that each test condition is Boolean, involving only Boolean types or relational and logical operators.

Rule 2-6 (Section 2.5): An enumeration's constants should all be initialized, or else none of them should be initialized.

Rule 2-7 (Section 2.5): Write programs as if enumeration variables could receive no values other than the associated enumeration constants. Treat the enumeration types as if they were unique types, not for any arithmetic int usages. Convert between enumeration variables and integer values only by use of an explicit cast.

Rule 2-8 (Section 2.6): Include "one-too-far" values in the ranges for variables, if they are needed for loop terminations or other testing purposes.

Rule 2-9 (Section 2.6): Function parameters accepting the size of an arbitrarily large object should be declared with size_t type.

Rule 2-10 (Section 2.7): When two unsigned int's are subtracted, convert the result using either (unsigned) or UI_TO_I.

Rule 2-11 (Section 2.7): Use IMOD (or some similar mechanism) to ensure that a non-negative modulo result is produced.

Rule 2-12 (Section 2.8): In (signed) integer arithmetic, assume that

overflow is illegal, may be detected (hence should never be programmed), and cannot be trapped or ignored.

Rule 2-13 (Section 2.9): Document the defining properties of declared names in a comment on the declaration, using a convention such as "colon followed by property."

Rule 3-1 (Section 3.1): Storage class (if any) should precede the type specifier.

Rule 3-2 (Section 3.1): If a variable has an initialization, its declaration should have a source line to itself.

Rule 3-3 (Section 3.1): Document the defining property of a data object with a comment on its declaration. Ensure that this defining property remains invariant (unchanging) as much as possible throughout the computation, and document any exceptions.

Rule 3-4 (Section 3.1): A program is easier to write correctly and to understand if all arrays are made complete before the array is used.

Rule 3-5 (Section 3.1): If an array's defining property can be true even if not all elements are defined, indicate the property on the array's declaration. For example,

char s[10];    /*: string */

Rule 3-6 (Section 3.2): Use executable assertions whenever they are simpler than the code being protected, and when the time to execute the assertions is not much greater than the time required to execute the code.

Rule 4-l (Section 4.1): In each pointer assignment, the right-hand-side value must have exactly the same ("converted") pointer type as the left-hand-side.

Rule 4-2 (Section 4.2): The default requirement for pointer parameters is that they must point to storage that is entirely defined. Whenever a pointer parameter can accept something else, this should be explicitly stated on that parameter's declaration comment.

Rule 4-3 (Section 4.3): A function in which the address of an automatic variable is assigned to a non-automatic pointer must contain a comment to that effect. In any function with such a comment, each return from the function is an event requiring verification that no dangling pointers are left.

Rule 6-1 (Section 6.1): In portable programming, do not hard-code the numeric values of structure offsets. The values may be different in each environment. Refer to members by their symbolic member names only.

Rule 6-2 (Section 6.2): Names with leading underscore should only appear in code that is privy to the internal details of the associated data structure, not in "user-level" portable code.

Rule 6-3 (Section 6.2): Use the "leading underscore" name format for tag and member names if the internal details of the structure are not to be inspected by functions outside of the package. Conversely, avoid leading underscores if the details of the structure are available for inspection by functions that use the structure.

Rule 6-4 (Section 6.3): If a structure is not well-defined when initialized to zero, document this fact in a comment. (The program will in general be simpler if the members are defined such that the zero-initialized structure is well-defined.)

Rule 6-5 (Section 6.5): In portable code, do not depend upon the allocation order of bit-fields within a word.

Rule 6-6 (Section 6.7): Regarding parameters which are pointers to structures, an "out" pointer parameter is assumed to be non-NULL, pointing to the storage for a structure of the specified type. "In" and "in-out" pointer parameters are assumed to point to a well-defined structure of the specified type. Any exceptions should be noted in a comment on the parameter declaration.

Rule 7-1 (Section 7.1): When a pointer p is passed to the free function, the programmer must determine how many pointers are pointing into the freed storage. (This number is known as the "reference count" of the storage.) Steps must be taken (such as assigning NULL) to ensure that none of these pointers are subsequently used to access the freed storage.

Rule 7-2 (Section 7.1): For every instance in which a program allocates storage, there should be an easily identifiable instance in which that storage is later freed.

Rule 8-1 (Section 8.2): Always test the returned value from fopen to be sure that the open succeeded.

[1-2] Declaring Function Parameters

The headers shown in this book show function parameters in comments associated with each function declaration, in this style:

double atof();  /* PARMS(char *s) */

In this form, the comments serve purely for documentation purposes. However, the ANSI C draft envisions a style of function declaration called a "prototype," in which the function parameters are actually stated in the declaration, like this:

double atof(char *s);

When the compiler has seen a prototype declaration, it can then perform type-checking of the arguments, much as the lint checker does. More than this, prototypes can also cause the conversion of function arguments. If, for example, the header <math.h> declares sqrt as

double sqrt(double x);

a function call such as

int i = 10;

double y;

y = sqrt(i);

will cause the conversion of the integer i to the expected double parameter type.

All of this makes it more important than ever that library functions should be properly declared, and the standard headers are the most reliable way to do this.

Some recent C compilers which are incorporating concepts from the ANSI draft have already introduced prototypes; you may be seeing them in your next compiler version. Thus, for some period of time you may be concerned to write programs in such a way that they can be compiled both by the previous generation of compilers and by the new-style compilers as well. To accomplish this, you will want to do two things:

1.     During this transition, you must make sure never to depend upon the argument conversions being done by the newer compilers. Your programs must continue to follow the argument-passing restrictions of the older compilers.

2.     You could, if you like, avoid declaring any argument information in your function declarations, thus keeping to the old style. You could, alternatively, package the argument declarations in a fashion which can be targeted either to old or new compilers. To do this, you could add a PARMS macro to your local header and use it in your function declarations like this:

void reverse PARMS((char *s));

The definition of PARMS for older compilers would simply discard the argument (which must be enclosed in a second set of parentheses, to allow for an arbitrary list of arguments):

#define PARMS(x) ()      /* empty parens */

The declaration of reverse as given above would produce

void reverse ();

The definition of PARMS for newer compilers would simply produce the argument:

#define PARMS(x) x

Thus the declaration of reverse would produce

void reverse (char *s);

If you adopt this transitional approach, you could take the PARMS information out of the comments, add an extra set of parentheses, and put the information onto the actual function declaration itself. I thought that it was somewhat premature to present all my examples this way in this book, but the translation from my notation should be rather easy if you want to adopt the approach described here.

[1-3] Side-Effects and Unsafe Macros

A recent compiler innovation opens the way to reliably diagnosing errors from side-effects on the invocation of unsafe macros. The White-smiths 3.0 compiler provides, as an extension, the pseudo-operator $noside(expr), which causes a compile-time error if the expr contains any side-effects. In order for the source code to remain portable, one would add a definition to a header like portdefs.h for an identifier like NOSIDE:

#define NOSIDE(expr)  $noside(expr)

Then, the definition of unsafe macros could be given in this form:

#define ABS(x)  (NOSIDE(x) < 0 ? -(x) : (x))

The compiler would then give a diagnostic if x contained side-effects.

In an environment that did not provide this extra checking, the invocation NOSIDE(x) could simply produce (x):

#define NOSIDE(x) (x)

I have not followed this style in the book, only because the release of the Whitesmiths 3.0 compiler is so recent.

[2-1] Round-off Errors in Math Functions

The output file shown as sqrtx.out was produced by the White-smiths PDP-11/23 compiler (version 2.3). Similar small round-off errors are produced by most other math libraries. Here is a similar output from the Lattice 8088 compiler (version 2.15), which uses the 8087 auxiliary FPP chip:

sqrtx86.out:

2.0    2.00000000000000044  -4.44E-16

3.0    2.99999999999999955   4.44E-16

5.0    5.00000000000000088  -8.88E-16

6.0    5.99999999999999911   8.88E-16

7.0    7.00000000000000088  -8.88E-16

8.0    8.00000000000000177  -1.78E-15

10.0   10.00000000000000176  -1.78E-15

12.0   11.99999999999999821   1.78E-15

13.0   12.99999999999999822   1.78E-15

15.0   15.00000000000000176  -1.78E-15

18.0   17.99999999999999642   3.55E-15

19.0   19.00000000000000355  -3.55E-15

20.0   20.00000000000000353  -3.55E-15

23.0   22.99999999999999642   3.55E-15

24.0   23.99999999999999642   3.55E-15

[2-2] Floating-to-Integer Truncation

As of June 1985, draft ANSI C requires that floating-to-integer truncation must be toward zero, whether the floating value is positive or negative. Since earlier C compilers may differ in the treatment of the rounding of negative floating values, the conservative course for portability is still to convert only positive values.

[2-3] Error Behavior of Math Functions

Some libraries (e.g., UNIX System V) provide more control over the treatment of errors in the math library. The user can provide a function named matherr which will be invoked if errors occur in a math function. This function could print diagnostics, terminate the execution, or specify the desired return-value.

The matherr function has not been adopted by ANSI C, so its use is not generally portable.

[2-4] Math Functions in ANSI C

These are the math functions that are declared in <math.h> in recent C libraries:

double acos(x)       - arc cosine of x

double asin(x)       - arc sine of x

double atan(x)       - arc tangent of x

double atan2(y, x)   - arc tangent of y/x

double ceil(x)       - smallest integer not less than x

double cos(x)        - cosine of x

double cosh(x)       - hyperbolic cosine of x

double exp(x)        - exponential function of x

double fabs(x)       - (floating) absolute value of x

double floor(x)      - largest integer not greater than x

double fmod(x, y)    - (floating) modulo function

double frexp(x, pi)  - normalized fraction

double ldexp(x, n)   - computes x times 2**n

double log(x)        - natural log of x

double logl0(x)      - base-10 log of x

double modf(x, pd)   - compute integer and fractional part of x

double pow(x, y)     - raise x to the power y

double sin(x)        - sine of x

double sinh(x)       - hyperbolic tangent of x

double sqrt(x)       - square root of x

double tan(x)        - tangent of x

double tanh(x)       - hyperbolic tangent of x

In this tabulation, these are the types of the parameters:

double x, y;

double *pd;

int n;

int *pi;

[2-5] Character-Test Functions in ANSI C

These are the math functions that are declared in <ctype.h> in recent C libraries:

int isalnum(c)  - is c a letter or a digit?

int isalpha(c)  - is c a letter?

int iscntrl(c)  - is c a non-printing character (other than space)?

int isdigit(c)  - is c a digit?

int isgraph(c)  - is c a printing character (other than space)?

int islower(c)  - is c a lower-case letter?

int isprint(c)  - is c a printing character (including space)?

int ispunct(c)  - is c a punctuation character?

int isspace(c)  - is c a whitespace character?

int isupper(c)  - is c an upper-case letter?

int isxdigit(c)  - is c a hexadecimal digit?

int tolower(c)  - convert upper-case c to lower case

int toupper(c)  - convert lower-case c to upper case

[4-1] Question

Just to be sure that you understand the "sandwich rule," jot down the variable name and type for the following declarations.

DECLARATION     VARIABLE NAME   TYPE ("DECLARED")

short **ap[5];       ap            short ** [5]

long (*pf)[2];       pf            long (*)[2]

double **pps;        pps           double **

Go to  Bibliography     Return to Table of Contents


Reliable Data Structures: BIBLIOGRAPHY

 Return to Table of Contents

BIBLIOGRAPHY

ANSI Technical Committee X3J11 [1985] C Information Bulletin (Preliminary Draft Proposed ANS). X3 Secretariat, 311 First Street NW, Suite 500, Washington DC 20001, 1985.

Bentley, Jon Louis [1982] Writing Efficient Programs. Prentice-Hall, 1982.

Bentley, Jon Louis [1984]. "How to Sort." Communications of the ACM. Vol 27, No 4, April 1984, Pp 287-291.

Chirlian, Paul. [1984] Introduction To C. Matrix Publishers, 1984.

Chris DeVoney (Editor) [1985+] /c: The Journal for C Users. Que Corporation, 7999 Knue Road, Indianapolis IN 46250, 1985+.

Feuer, Alan, and Narain Gehani [1984] Comparing And Assessing Programming Languages: Ada, C, Pascal. Prentice Hall, 1984.

Hancock, Les, and Morris Krieger [1982] The C Primer. McGraw Hill, 1982.

Harbison, Samuel, and Guy Steele [1984] A C Reference Manual. Prentice Hall, 1984.

Hunter, Bruce [1984] Understanding C. Sybex Computer Books, 1984.

Jaeschke, Rex [1985+] The C Journal. InfoPro Systems, POB 849, Denville NJ 07834, 1985+.

Kelley, Al, and Ira Pohl [1984] A Book On C. Benjamin Cummmings, 1984.

Kernighan, Brian, and Dennis Ritchie [1978] The C Programming Language. Prentice Hall, 1978.

Kernighan, Brian, and Rob Pike [1984] The UNIX Programming Environment. Prentice Hall, 1984.

Kochan, Stephen [1983] Programming in C. Hayden Book Company, 1983.

Knuth, Donald E. [1973] The Art of Computer Programming: Volume 1: Fundamental Algorithms. Addison-Wesley, 1973.

Parker, Robert V [1985]. A Catskill Eagle. Delacorte, 1985.

Plum, Thomas [1983] Learning To Program In C. Plum Hall, 1983.

Plum, Thomas [1984] C Programming Guidelines. Plum Hall, 1984.

Plum, Thomas, and Jim Brodie [1985]. Efficient C. Plum Hall, 1985.

Pugh, Kenneth [1985] C Language for Programmers. Scott Foresman Foresman Company, 1985.

Purdum, Jack J., Timothy C Leslie, and Alan L. Stegemoller [1984] C Programmer's Library. Que, 1984.

Tondo, Clovis, and Scott Gimpel [1985] The C Answer Book. Prentice Hall, 1985.

Traister, Robert [1984] Programming In C For The Microcomputer User. Prentice Hall, 1984.

Waite, Mitchell, Stephen Prata, and Donald Martin [1984] C Primer Plus. Howard W Sams & Company, 1984.

Ward, Terry [1985] Applied Programming Techniques In C. Scott Foresman & Company, 1985.

Wortman, Leon, and Thomas Sidebottom [1984] The C Programming Tutor. Robert J Brady Company, 1984.

________________[1975] A Course in Miracles. Foundation for Inner Peace, Tiburon, CA, 1975.


Reliable Data Structures: CHAPTER 1: PREPROCESSING

 Return to Table of Contents

CHAPTER 1: PREPROCESSING

The "definition" capabilities of C provide techniques for writing programs that are more portable, more readable and easier to modify reliably. For a simple example,

#define SCR_CLR_ALL "\33[2J"

defines a string which will clear the entire screen (on "ANSI" terminals). A program that contained numerous instances of printf(SRC_CLR_ALL) would be considerably more readable than one containing lots of printf("\33[2J") uses.

In addition, definitions can aid in modifiability. Using the SCR_CLR_ALL example, the program that uses the defined name can be more easily modified than the one which contains instances of the actual string sprinkled throughout the program.

In this chapter, we will see a variety of uses of definitions and header files which are helpful in achieving reliable programs.

1.1 Defined Constants

Topics assumed from previous book: #include [2.10], #define [2.10], ctype.h [3.5], stdio.h [3.21], defined constant [3.21].

A file of definitions to be #include'd is known (in ANSI C terminology) as a header. (In other Plum Hall books, it is also referred to as an include-file.) In the standard header stdio.h are found some definitions such as

#define EOF (-1)

This definition causes any subsequent appearance of the name EOF to be replaced by the characters (-1). Since the definition contains an operator (the "minus"), we enclose the definition in parentheses. If we left the parentheses off the definition, as in

#define EOF -1

it is possible that the name could generate an expression that would be misunderstood by the compiler. For example, this erroneous code

if (c EOF) WRONG - should be (c != EOF)

would be interpreted as

if (c - 1) syntactically correct, but unintended

which would not generate any compile errors, even though it is certainly not what the programmer intended. This suggests a rule for reliable programming (the first of many that will be presented):

Rule 1-1: Any macro definition containing operators needs parentheses around the entire definition. Each appearance of a macro argument in the definition also needs to be parenthesized if an embedded operator in the argument could cause a precedence problem.

All the reliability rules are summarized in the appendix [1-1].

There are two main reasons for creating defined constants. The first is readability: the defined constant clarifies the meaning of the constant. The name EOF tells the reader that it is a special indication for "end-of-file." For another example from <stdio.h>, consider

#define BUFSIZ 512

Again, the name clarifies the meaning: BUFSIZ is the "buffer size" used in efficient I/O transfers.

The second reason for defined constants is modifiability: the defined constant shows how the program can be modified for a different value. Thus, on systems where 1024 is a better value for the size of disk I/O transfers, the standard header stdio.h could specify the value 1024 for BUFSIZ.

Rule 1-2: Reliable modification of defined constants requires an environmental capability: there must be a means for ensuring that all files comprising a program have been compiled using the same set of headers. (The UNIX make command is one such capability.)

EOF, however, exemplifies a defined constant which cannot be modified arbitrarily: many implementations of the character-type tests (ctype.h) assume that its value is -1. The UNIX System V manual page for ctype(3C), for example, says explicitly that EOF equals -1. Thus the following usage rule:

Rule 1-3: If there are limitations on the modifiability of a defined constant, indicate the limitations with a comment:

#define EOF (-1)  /* DO NOT MODIFY: ctype.h expects -1 value */

Another application of this rule is the explicit indication of minimum and maximum values:

#define NBUFS 5   /* min 2, max 30 */

(to give a hypothetical example).

Rule 1-4: If one definition affects another, embody the relationship in the definition; do not give two independent definitions.

This pair of definitions

#define XX 5

#define XX2 (XX + 2)

follow the rule by showing the relationship, whereas

#define XX 5

#define XX2 7  misleading, no indication of relationship

does not. In the former case, we could reliably modify the program by changing the definition of XX; in the latter case, we would have to make a guess that XX2 should probably be changed to equal XX plus two.

Just giving the constant a name is not enough to ensure modifiability; you must be careful always to use the name, and remember that the value could change. One project had difficulties changing the value of BUFSIZ because some programmers had written

nblocks = nbytes >> 9; hard to modify, uses "magic number"

in a number of places where

nblocks = nbytes / BUFSIZ;

was needed. The programmers figured that "everyone knows that BUFSIZ equals 512," and right-shifting nine bits is the same (for positive numbers) as dividing by 512. But when BUFSIZ changed to 1024 on some systems, modifications were difficult. Hence, this rule:

Rule 1-5: If a value is given for a #defined name, do not defeat its modifiability by assuming its value in expressions.

In most compilers, it is also possible to define a constant on the invocation of the compiler, rather than putting a #define in the code itself. Suppose, for example, that we wish to have the constant NBUFS specified at compilation time, rather than putting the value into the program. Using the UNIX compiler cc, we accomplish the definition like this:

cc -DNBUFS=10 pgm.c

This technique allows a program to be targeted to different environments without changing the source code for the program.

The UNIX function library has recently been standardized by /usr/group, the UNIX industry association. This standard specifies a new header named <limits.h>, wherein are defined environment- dependent limits. (ANSI C will probably standardize a similar capability.)

A sample version of <limits.h> for C on an IBM PC-type machine might contain definitions like these:

#define CHAR_BIT            8    /* number of bits in a char */

#define CHAR_MAX          255    /* largest char value */

#define CHAR_MIN            0    /* smallest char value */

#define INT_MAX         32767    /* largest int value */

#define INT_MIN        -32768    /* smallest int value */

#define LONG_MAX   2147483647    /* largest long value */

#define LONG_MIN  -2147483648    /* smallest long value */

#define SHRT_MAX        32767    /* largest short value */

#define SHRT_MIN       -32768    /* smallest short value */

#define UCHAR_MAX         255    /* largest unsigned char value */

#define UINT_MAX        65535    /* largest unsigned int value */

#define ULONG_MAX  4294967295    /* largest unsigned long value */

#define USHRT_MAX       65535    /* largest unsigned short value */

Note that no extra definitions are needed for numbers like "number of bits in an int;" we can use sizeof(int) * CHAR_BIT, since every data type occupies an integral number of bytes.

Rule 1-6: Use limits.h for environment-dependent values.

1.2 Defined Types

Topics assumed from previous book: defined type [3.21], void [3.21], typedef [3.21].

It will be very important for purposes of reliability to make more precise definitions of data types than the basic types supplied by C. In this book, we will describe special semantic rules for various of these defined types. Here is a list of some useful defined types:

bits        /* an unsigned short integer used for bitwise operations */

ushort      /* an unsigned short integer used for arithmetic */

tbool       /* a char (one byte) to be tested for zero or non-zero */

metachar    /* a short integer holding a char value or EOF */

bool        /* an integer to be tested for zero or non-zero */

void        /* the "return type" for a function that returns no value */

These types can be defined using #define or typedef; here is a typedef version:

typedef unsigned short ushort, bits;

typedef char tbool;

typedef short metachar;

typedef int bool;

typedef int void;   /* delete if compiler supports void */

The important difference between #define and typedef is that #define replaces the name with its definition during preprocessing, whereas typedef is handled by the C syntax analysis. Thus, a defined type created via typedef cannot be "undefined" or re-defined, so its usage is more reliable. Also, if the name is mistakenly used as a variable name, the diagnostics are more intelligible.

This particular set of defined types are consistently used in the various Plum Hall books on C. In "publication" code, where one desires the fewest possible augmentations to standard C, an alternative approach is to use comments:

short status;   /* interface status: bits */

char is_open;   /* is device open?: bool */

int input_c;    /* most recent input character: metachar */

Even in publication code, however, the symbols ushort and void are important for portability reasons. Not all compilers currently accept the unsigned short type, but most that do not accept it are targeted for small machines where ushort can simply be translated into unsigned int.

Regarding void, any function that does not return a value should be indicated as being a void function. The following scr_beep function outputs the "bell" character, but does not return any value to the calling function, so we give it the void type:

void scr_beep( )

{

putchar('\7');  /* "bell" - ASCII version */

}

Recent C compilers implement void as a keyword. If your compiler supports void, you should not try to #define or typedef a definition for it; in this case, you should remove any such definitions from your headers. If your compiler does not support void, you should define it as int.

Rule 1-7: Use a consistent set of project-wide defined types.

1.3 Standard Headers

Topics assumed from previous book: #include [2.10], stdio.h [3,4]. math.h [3.17], ctype.h [3.5].

Headers are used for several purposes: creating defined constants, creating defined types, and declaring the type of various functions.

The modern treatment of libraries is to create a header for each related group of functions, along with any special symbols that are useful with those functions. Thus the math functions (and also the symbol HUGE_VAL) are declared in <math.h>:

acos      cos       fmod     modf     tan

asin      cosh      frexp    pow      tanh

atan      exp       ldexp    sin

atan2     fabs      log      sinh

ceil      floor     log10    sqrt

The character-test functions are declared in <ctype.h>:

isalnum   isdigit   isprint  isupper  toupper

isalpha   isgraph   ispunct  isxdigit

iscntrl   islower   isspace  tolower

And the standard I/O functions (plus some useful symbols) are declared in <stdio.h>:

clearerr  fopen     fseek    printf   scanf    ungetc

fclose    fprintf   ftell    putc     setbuf

feof      fputc     fwrite   putchar  setvbuf

ferror    fputs     getc     puts     sprintf

fflush    fread     getchar  remove   sscanf

fgetc     freopen   gets     rename   tmpfile

fgets     fscanf    perror   rewind   tmpnam

BUFSIZ    EOF       FILE     NULL

stdin     stdout    stderr

Another standard header is <assert.h>, which provides an assert macro for debugging. This header will be covered in Section 1.6.

The logical extension of this approach is to partition each library entirely into related groups of functions, with each header declaring all functions in its group. ANSI C envisions two more headers, <string.h> and <stdlib.h>. (<string.h> is already found in UNIX libraries, beginning with System V. System V has another header named <memory.h>, which ANSI C will probably merge into <string.h>.)

In <string.h>, there should be declarations for these functions:

memchr    memset    strcoll  strncat  strrchr

memcmp    strcat    strcpy   strncmp  strspn

memcpy    strchr    strcspn  strncpy  strtok

memmove   strcmp    strlen   strpbrk

In <stdlib.h> there should be declarations for several miscellaneous functions:

abort     atof     bsearch   exit     ldiv    rand     system

abs       atoi    calloc     free     malloc  realloc

atexit    atol    div        getenv   qsort   srand

Most of these names will be described in this book. Each header will be described when we need its functions; the names are given here just for completeness.

The variety of libraries available with existing compilers does create a portability problem for your code. Until the time that all your compilers conform to an ANSI standard, the following approach will help shelter your programs from library differences: The headers <math.h>, <ctype.h>, and <stdio.h> are well-nigh universal, and can simply be #included. For each of the other headers limits.h, string.h, and stdlib.h, create a header of your own which can be included without using the "angle-bracket" <xxx.h> form. In that header, provide declarations for all the functions that are supported on the environment that you are using. You might, of course, provide any missing functions yourself, and include their declarations in the header.

One other ANSI header needs to be mentioned. It is called <stddef.h> ("standard definitions") and it includes a few type definitions and named constants. The only one that we will need in this book is called size_t, and it is defined to be an integer that is big enough to hold the size of any object (unsigned int or unsigned long int).

The minimal versions of these "pre-ANSI" headers required for the programs in this book are as follows:

stddef.h:

/* stddef.h - standard definitions (partial listing) */

/* ENVIRONMENT-DEPENDENT - ADJUST TO LOCAL SYSTEM */

#ifndef STDDEF_H

#define STDDEF_H

typedef unsigned size_t;   /* use unsigned long in large-object model */

#ifndef NULL

#define NULL 0             /* use 0L if int-size < long-size == ptr-size */

#endif

extern int errno;

#endif

limits.h:

/* limits.h - environment limits (partial listing) */

#ifndef LIMITS_H

#define LIMITS_H

#define CHAR_BIT    8

#endif

string.h:

/* string.h - string functions (partial listing) */

#ifndef STRING_H

#define STRING_H

#include "stddef.h"

data_ptr memcpy( );  /* PARMS(data_ptr s1, data_ptr s2, size_t n) */

char *strcat( );     /* PARMS(char *s1, char *s2) */

char *strchr( );     /* PARMS(char *s1, int c) */

int strcmp( );       /* PARMS(char *s1, char *s2) */

char *strcpy( );     /* PARMS(char *s1, char *s2) */

size_t strlen( );    /* PARMS(char *s1) */

char *strncat( );    /* PARMS(char *s1, char *s2, size_t n) */

int strncmp( );      /* PARMS(char *s1, char *s2, size_t n) */

char *strncpy( );    /* PARMS(char *s1, char *s2, size_t n) */

#endif

stdlib.h:

/* stdlib.h - miscellaneous library functions (partial listing) */

#ifndef STDLIB_H

#define STDLIB_H

#include "stddef.h"

double atof( );      /* PARMS(char *s) */

int atoi( );         /* PARMS(char *s) */

long atol( );        /* PARMS(char *s) */

data_ptr calloc( );  /* PARMS(unsigned int n, size_t size) */

void exit( );        /* PARMS(int status) */

void free( );        /* PARMS(data_ptr ptr) */

data_ptr malloc( );  /* PARMS(size_t size) */

int rand( );         /* PARMS(void) */

void srand( );       /* PARMS(unsigned int seed) */

#endif

Each header declares the returned type of various functions. The types of the function parameters are also specified in comments [1-2].

Assuming that you have all these headers, or have created your own versions, when and why should you use them? First of all, notice that several problems can arise from not declaring the returned type of a library function. For example, most of the functions in the math library return double values. If you use one of these functions, such as

x = sqrt(y);

without including the math.h header (and without declaring the function in your own program), the compiler assumes that sqrt returns an int value, which will give a garbage value to x in this example.

A more subtle problem arises in using the string functions. A number of these functions return a value of type char * ("pointer to character"), and can be embedded into expressions:

strcat(strcat(s, ".1"), ".a");

will end up catenating ".1.a" onto the end of s. In order for this to work reliably, the compiler must know that strcat returns a char * value. Otherwise, the compiler assumes that it returns int, and the program will not work in some environments where int and char * are different sizes. (The lint checker will complain in any case, if you have lint.)

For these reasons, it is important to declare the library functions. Should you use the headers, or declare each function yourself? In general, the headers are the more reliable way to be sure that you have all the correct declarations, but there is one (temporary) problem. Some compilers will link the object code for every library function that you declare, whether or not your program actually calls it. The ANSI C standard will probably say that functions should not be linked unless they are used, though it may be a while before all compilers behave this way. I recommend that you use the standard headers for the library, unless your compiler links everything.

If you are going to use the standard headers, I also recommend that you create your own local header file that brings in all the headers needed for your project. I have used the generic name local.h for this header; your own local version will presumably have its own name. The simplest course is to have your local.h include all the standard library headers, as the safest way to be sure that they are all included.

Rule 1-8: Be sure that all functions are declared before use; headers are the most reliable way.

Rule 1-9: Create a project-wide "local" header for standard definitions and inclusions.

One of the headers that we will use is called portdefs.h ("portability definitions header"). Into it are collected various defined types, defined constants, and macros which will be important in producing portable code. By including it in our local.h we ensure that all programs compile with these "portability definitions." The following listings of local.h and portdefs.h show the definitions that are needed for the Plum Hall books:

local.h:

/* local.h - Definitions for use with Reliable Data Structures in C */

#ifndef LOCAL_H

#define LOCAL_H

#include <stdio.h>

#include <ctype.h>

#include <math.h>

#define FALSE           0          /* Boolean value */

#define FOREVER         for(;;)    /* endless loop */

#define NO              0          /* Boolean value */

#define TRUE            1          /* Boolean value */

#define YES             1          /* Boolean value */

#define getln(s, n)     ((fgets(s, n, stdin)==NULL) ? EOF : strlen(s))

#define ABS(x)          (((x) < 0) ? -(x) : (x))

#define MAX(x, y)       (((x) < (y)) ? (y) : (x))

#define MIN(x, y)       (((x) < (y)) ? (x) : (y))

#define DIM(a)          (sizeof(a) / sizeof(a[O]))

#define IN_RANGE(n, lo, hi) ((lo) <= (n) && (n) <= (hi))

#ifndef NDEBUG

#define asserts(cond, str) \

{if (!(cond)) fprintf(stderr, "Assertion '%s' failed\n", str);}

#else

#define asserts(cond, str)

#endif

#define SWAP(a, b, t)   ((t) = (a), (a) = (b), (b) = (t))

#define LOOPDN(r, n)    for ((r) = (n)+1; --(r) > 0; )

#define STREQ(s, t)     (strcmp(s, t) == 0)

#define STRLT(s, t)     (strcmp(s, t) < 0)

#define STRGT(s, t)     (strcmp(s, t) > 0)

#include "portdefs.h"   /* portability definitions */

#include "stddef.h"     /* (ANSI) standard definitions */

#include "limits.h"     /* (ANSI) machine parameters */

#include "string.h"     /* (ANSI) string functions */

#include "stdlib.h"     /* (ANSI) miscellaneous standard functions */

#include "rdslib.h"     /* functions from Reliable Data Structures in C */

#endif

portdefs.h:

/* portdefs.h - definitions for portability */

/* ENVIRONMENT-DEPENDENT - ADJUST TO LOCAL SYSTEM */

#ifndef PORTDEFS_H

#define PORTDEFS_H

/* adjust these names to local machine/compiler environment */

typedef unsigned short ushort;  /* or "unsigned" if short-size == int-size */

typedef unsigned char utiny;    /* to get unsigned byte */

typedef int void;               /* delete if compiler supports void */

typedef unsigned index_t;       /* may be chosen ad-lib locally */

typedef char *data_ptr;         /* use ANSI "generic ptr" if available */

/* next 5 names require no local changes, will work anywhere */

typedef char tbits;             /* one byte, for bitwise uses */

typedef char tbool;             /* one byte: {0:1} */

typedef ushort bits;            /* 16 bits (or more), for bitwise uses */

typedef int bool;               /* for function returns: {0:1} */

typedef short metachar;         /* return from getchar: {EOF,0:UCHAR_MAX} */

/* modulo function giving non-negative result */

#define IMOD(i, j) (((i) % (j)) < O ? ((i) % (j)) + (j) : ((i) % (j)))

/* if i % j is never negative, replace with the following line: */

/* #define IMOD(i, j) ((i) % (j)) */

/* portably convert unsigned number to signed */

#define UI_TO_I(ui) (int)(ui)   /* more complicated on ones complement */

/* structure offsets and bounds; adjust to local system */

#define STRICT_ALIGN int        /* adjust to local alignment requirement */

#define OFFSET(st, m) \

((char *)&((st *)&struct_addr)->m - (char *)&struct_addr)

#define BOUNDOF(t) \

((char *)(struct {char byte0; t byten; } *)&struct_addr)->byten - \

(char *)&struct_addr)

static STRICT_ALIGN struct_addr = 0;

#define STRUCTASST(a, b) memcpy(&(a), &(b), sizeof(a))

/* defined constants */

#define FAIL            1          /* failure exit */

#define SUCCEED         0          /* normal exit */

#define STDIN           0          /* standard input */

#define STDOUT          1          /* standard output */

#define STDERR          2          /* standard error output */

#define SEEK_SET        0          /* seek relative to start of file */

#define SEEK_CUR        1          /* seek relative to current position */

#define SEEK_END        2          /* seek relative to end */

#endif

To be consistent with the approach of giving headers for all function declarations, we will also provide a header for all the utility functions that will be presented in this book. We will call it rdslib.h ("Reliable Data Structures library header") and it looks like this:

rdslib.h:

#ifndef RDSLIB_H

#define RDSLIB_H

bool itoa( );      /* PARMS(int n, char *str, int ndigits) */

int fgetsnn( );    /* PARMS(char *str, int size, FILE *fp) */

int getsnn( );     /* PARMS(char *str, int size) */

int getreply( );   /* PARMS(char *prompt, char *reply, int size) */

bool getpstr( );   /* PARMS(char *p, char *s, size_t n) */

bool getplin( );   /* PARMS(char *p, char *s, size_t n) */

void plot_trk( );  /* PARMS(int n, char c) */

void reverse( );   /* PARMS(char *s) */

bool strfit( );    /* PARMS(char *s1, char *s2, size_t n) */

#endif

On any sizeable project that you work on, a significant part of the early work will be involved with choosing the project-wide set of headers to be used and putting them into the appropriate places in the system. Once the groundwork is done, however, the shared set of definitions makes everything else much easier.

1.4 Macro Functions

Topics assumed from previous book: macros with parameters [5.17].

A macro with parameters will be referred to as a macro function, for brevity. Of course, it is not really a function, but it is easier to talk about this way.

Our local.h specifies three macro functions:

#define ABS(x)     (((x) < 0) ? -(x) : (x))

#define MAX(x, y)  (((x) < (y)) ? (y) : (x))

#define MIN(x, y)  (((x) < (y)) ? (x) : (y))

Since each argument can contain operators, each parameter is parenthesized in the definition to avoid precedence conflicts. And since the entire result is an expression usable with other operators, the entire definition is also parenthesized. These are two reliability techniques totally under control of the programmer who writes the macro.

Unfortunately, one reliability problem is beyond the macro writer's control: side-effects on macro arguments. A typical example is

ABS(++n)  bug!

which increments n twice. To introduce some terminology, an unsafe macro function is one which evaluates a parameter more than once in the code expansion. Stated positively, a safe macro function evaluates each parameter only once in the code expansion. By this definition, all three macro functions (ABS, MAX, and MIN) are unsafe. As things stand now, the documentation for such macros must warn about putting side-effects on the invocation, and the responsibility is upon the programmer using the macro. (Some new compilers provide assistance in locating such bugs; see the appendix [1-3].)

Rule 1-10: Use UPPERCASE names for unsafe macro functions, to emphasize the restrictions on their usage.

For safe macros, there are some advantages to using lowercase names. Each safe macro could be replaced by an actual function call, and at different times during project development, one might want the macro version or the function version. For an example from the library, the character-type tests in ctype.h are usually implemented as safe macros; they could therefore be replaced with actual function versions.

Rule 1-11: Never invoke an unsafe macro with arguments containing assignment, increment/decrement, or function call.

Rule 1-12: Whenever possible, use safe macro functions.

To be more precise about our usage, all the macros described so far are what we will call expression macros: after replacement of arguments, they produce a valid C expression value. Thus, an expression macro (such as MAX(i, j)) can be used any place that C allows an expression.

An expression macro can be used purely for its side-effects, just like calling a void function. One such macro is useful for swapping two data objects using a third temporary:

#define SWAP(a, b, t) ((t) = (a), (a) = (b), (b) = (t))

Since the macro produces a single expression (three assignments connected with the comma operator), an invocation of the macro can be made into a statement just by adding a semicolon:

/* reverse the characters in a string */

for (i = 0, j = strlen(s) - 1; i < j; ++i, --j)

SWAP(s[i], s[j], t);

Another form of macro is the statement macro. For example, the "swap" operation can also be written as a statement macro, like this:

#define SWAP_SHORT(a, b) {short _t;_t = (a); (a) = (b); (b) =_t;}

Because the definition constitutes a block (compound statement), it can have local variables, such as _t. This eliminates the need for an explicit parameter, but it limits the type of data to which the macro can be applied.

A statement macro like SWAP_SHORT, which contains code enclosed in braces, has one small syntactic restriction on its usage. We would like to put a semicolon onto each invocation of the macro, just because this looks more familiar:

if (a[i] < a[j])

SWAP_SHORT(a[i], a[j]);

However, the macro replacement itself already is a statement, so the semicolon becomes an extra null statement tacked on. Usually, there is no harm, except for this one situation:

if (a[i] < a[j])

SWAP_SHORT(a[i], a[j]);  syntax error

else

/* ... */

Now we have a syntax error, because there are two statements between the if and the else.  The unfortunate conclusion is that we need to be sure that the statement body following the if is enclosed in braces:

if (a[i] < a[j])

{

SWAP_SHORT(a[i], a[j]);

}

else

/* ... */

To keep things in perspective, statement macros comprise a small minority of the macros that you will typically use. The troublesome context is fairly uncommon, and the compiler will give a clear indication of any problem. Therefore, no special rules are necessary.

It would be nice to have a form for a statement macro which becomes a valid C statement when a semi-colon is appended. One interesting way to construct it is to terminate the macro with a trailing else:

#define SWAP_SHORT(a, b) if (1) {short_t; _t = a; a = b; b = _t} else

This does ensure that the macro will be syntactically valid when embedded into other C statements like this:

if (a[i] < a[j])

SWAP_SHORT(a [i], a[j]);

else

/* ... */

Unfortunately, the error of forgetting a semicolon is fairly common, and it will silently produce a very unreliable result in a common context like this:

SWAP_SHORT(i, j)        well-hidden bug

++n;

This generates

if (1) {short_t; _t = a; a = b; b = _t} else

++n;

Re-arranged for readability, it looks like this:

if (1)

{ short _t;  _t = a; a = b; b =  _t }

else

++n;

The increment statement following the macro has become the body of the (never executed) else, producing a mysterious result with no chance of syntax diagnosis (unless the compiler complains about "unreachable code").

The "trailing else" form for statement macros is therefore not recommended.

A control macro is one which resembles a control structure. One simple example is the FOREVER macro:

#define FOREVER  for (;;)

which is sometimes used in implementing an "N+1/2-time loop":

FOREVER

{

perform some action

if (time to quit)

break;

perform further actions

}

A more involved example is useful in constructing highly optimized inner loops; our local.h header includes a macro named LOOPDN, which counts a variable downwards toward zero:

#define LOOPDN(r, n) for ((r) = (n)+1; --(r) > 0; )

In most environments, using L00PDN (with a register variable r) is the most time-efficient way of specifying n iterations. (The expression is compared versus zero, and the decrement is part of an expression evaluation; see Plum and Brodie [1985].) Using the macro is preferable to writing the actual in-line code, because the code itself is rather obscure and ugly -- the macro hides the messy details. If some environment allows a more efficient loop than this one, using the macro allows us to change the definition in just one place.

1.5 Undefining

Once a symbol has been #define'd, its definition can be deleted via #undef. For a dangerous example, consider

#define NBUFS 5

char b1[NBUFS] [BUFSIZ];

/* ... one page of code here ... */

#undef NBUFS

/* ... another page of code here ... */

#define NBUFS 7

char b2[NBUFS] [BUFSIZ];

This example is deliberately chosen to illustrate the reliability problems of #undef. The declarations for b1 and b2 appear to the reader to be of equal size, but the definition of NBUFS has changed underfoot. Thus, as a general rule, the re-definition of symbols is unreliable.

To put it in a positive way, we want each symbol to have an invariant meaning, so that each instance of the symbol denotes the same thing.

One usage of #undef may become more common, as a certain discipline of library construction becomes standard. Namely: each name in a library is declared in its associated header (as we saw in previous section). The header can #define a function name as a macro function (provided that it is a safe macro). Then, if the programmer wishes to be sure that a given name is a true function, it will be adequate to #undef that name.

ANSI C will probably specify that it is always allowable to #undef a name, even if it has not been #define'd. Some existing compilers, however, will complain about #undef'ing a non-existent name. For the time being, the safest course is to use this pattern:

#ifdef NAME

#undef NAME

#endif

1.6 Conditional Compilation

The preprocessor provides for conditional compilation, whereby some lines of code may be selectively excluded from compilation, depending upon the outcome of some test. This is the general pattern:

#if-line

source lines

#else

other source lines

#endif

The #else part can be omitted:

#if-line

source lines

#endif

The #if-line can have any of these forms:

#if constant-expression

True if constant-expression is non-zero.

#ifdef identifier

True if identifier has been #define'd.

#ifndef identifier

True if identifier has not been #define'd.

Recent C compilers provide two more capabilities:

#if defined(identifier)

or

#if defined identifier

means the same as

#ifdef identifier

And "else-if" logic is provided:

#if defined(A)

n = A_LIMIT;

#elif defined(B)

n = B_LIMIT;

#endif

means the same as

#ifdef A

n = A_LIMIT;

#else

#ifdef B

n = B_LIMIT;

#endif

#endif

Now we will look at some examples of the application of conditional compilation to practical programming situations.

Creating portable defined types

Tuning defined types for size

Commenting-out sections of code

Inclusion Sandwich

Test drivers for library functions

Conditional debugging

Creating portable defined types

It is often useful to have a defined type to represent an unsigned short integer. Many compilers support this type directly, but some others (mostly 16-bit environments) do not. Suppose that our compile procedure ("shell script," "batch file," etc.) defines a constant named USHORT if the compiler supports the unsigned short data type. Then, we can specify the defined type ushort like this:

#ifdef USHORT

typedef unsigned short ushort;

#else

typedef unsigned ushort;   /* assumes 16-bit machine */

#endif

Using this definition in a local standard header, we can write our programs using ushort in declarations. Provided that our compilation procedure defines USHORT appropriately for the compilation environment, we will get the appropriate definition for each environment.

Tuning defined types for size

Suppose we need a defined type (to be called a TOKEN) which can represent any of a specified number of alternatives. (For simplicity, assume that the number of distinct "tokens" is never more than 64K.) We could define TOKEN like this:

#if MAXTOKEN <= 127

typedef char TOKEN;

#else

typedef ushort TOKEN;

#endif

Thus, for compilations in which the number of "tokens" is small, we can store each of them in a single char object (which can reliably hold a number between 0 and 127, in any environment).

Commenting-out sections of code

Sometimes one needs to "comment-out" several lines of code. Most compilers (and ANSI C) do not nest comments, so ordinary comments cannot be used. But the construct

#if 0

/* ... code to be commented-out ... */

#endif

will reliably delete the enclosed lines, even if they contain other #if constructs.

Rule 1-13: Use #if 0 if there is a need to comment-out sections of code.

Inclusion Sandwich

Until the early 1980's, large projects had a continual problem with the inclusion of headers. One group might have produced a graphics.h, for example, which started by including <stdio.h>. Another group might have produced keyboard.h, which also included <stdio.h>. And if <stdio.h> could not safely be included several times, arguments would break out about which header should include it. Sometimes an agreement was reached that each header should include no other headers, and therefore some application programs started with dozens of #include lines -- and sometimes they got the ordering wrong, or forgot a header that was needed.

All these complications disappeared with the discovery of a simple technique: each header should #define a symbol which means "I have already been included." Then the entire header should be enclosed in a "sandwich":

#ifndef HEADER_H

#define HEADER_H

/* ... contents of the header ... */

#endif

Thus, the first time that header.h is #include'd, all of its contents will be included. If it should subsequently be #include'd again, its contents will be by-passed.

Rule 1-14: Enclose each header in an "inclusion sandwich."

Test drivers for library functions

When writing a library function -- a general-purpose callable function, which has a source file to itself -- it can be useful to provide a simple test harness within the same source file. The following pattern can be used:

code for the function

#ifdef TRYMAIN

main()

{

code for the test harness

}

#endif

Now, when we want to compile the function with the test harness, we define the symbol TRYMAIN in the compilation procedure. When we want just the object code for the function, we do not define TRYMAIN, and compile for object-code only. Keeping the harness in the same source file as the function makes later modification easier to test, and provides useful documentation of how the function is supposed to be used.

Conditional debugging

One very important application of conditional compilation is the use of conditional debugging code. The macro name NDEBUG is conventionally used to turn off any conditional debugging code:

cc -DNDEBUG pgm.c

turns off debugging, while an ordinary compilation with

cc pgm.c

will enable debugging. For a specific example, the local.h header contains the following definition:

#ifndef NDEBUG

#define asserts(cond, str) \

{ if (!(cond)) fprintf(stderr, "Assertion '%s' failed\n", str); }

#else

#define asserts(cond, str)

#endif

If the program being compiled contains the line

asserts(x >= 0, "x is non-negative");

an ordinary compilation will generate a test for the non-negativity of x, while compilation with NDEBUG defined will turn off the checking.

Recent C compilers provide a more streamlined version of this facility, in the header <assert.h>. The assert macro takes only one argument, the condition being tested:

assert(x >= 0);

If the assertion fails during execution, a message of this form will be printed:

Assertion failed: "x >= 0", file pgm.c, line 17

Furthermore, the macro calls the function abort, which terminates program execution. (Under UNIX, a "core-image" file is produced, for further assistance in debugging.)

The assert macro is an excellent tool for putting executable assertions into a program, but its streamlined implementation requires several preprocessor features which are not yet universal. The preprocessor must provide the symbols _ _FILE_ _ and _ _LINE_ _ (which identify the source file name and current line number), the preprocessor must be able to make a string constant out of the macro argument, and the library must support the abort function. The asserts macro is intended to provide a more universally-portable substitute until these features are provided by all compilers. In the following chapters, there will be many uses for the asserts macro.

Go to Chapter 2    Return to Table of Contents


Reliable Data Structures: CHAPTER 2: SCALARS

 Return to Table of Contents

CHAPTER 2: SCALARS

A data object (or just object) is a sequence of bytes to be manipulated as a unit. For now, "object" is just a technical name for a variable. When we discuss pointers we will see that there are other objects besides variables.

We start our consideration of reliable data structures with the low-level objects, the individual scalars. In this book, a scalar object is one having floating, integer, or pointer type.

Throughout this chapter, we will address the reliable use of scalars. As always, our desire is that the program actually does what it appears to do. Our motto for reliable programming continues to be "No surprises!"

2.1 The Math Library, with <math.h>

Our search for non-surprising programs begins with the math library and the floating-point numbers that it supports. These are some of the more common math functions:

double ceil(x)    - smallest integer not less than x

double cos(x)     - cosine of x

double exp(x)     - exponential function of x

double floor(x)   - largest integer not greater than x

double log(x)     - natural log of x

double log10(x)   - base-0 log of x

double pow(x, y)  - raise x to the power y

double sin(x)     - sine of x

double sqrt(x)    - square root of x

double tan(x)     - tangent of x

All of these functions deal with floating-point numbers (of the size specified by double), which are only binary approximations of a "true" real number. Also, each library function that produces a fractional result may introduce some small error in the result. For example, in "real" numbers, multiplying the square root of x by itself should produce exactly the value of x. The actual results typically show a small error in the low-order bits, something like this [2-1]:

sqrtx.c:

/* sqrttst - test accuracy of sqrt

*/

#include "local.h"

main()

{

double x, y;

for (x = 1.; x <= 25.; x = x + 1.)

{

y = sqrt(x) * sqrt(x);

if (y != x)

printf("%5.1f %21.17f %10.2e\n", x, y, x-y);

}

}

sqrtx.out:

2.0   2.00000000000000010  -5.55e-17

3.0   3.00000000000000010  -5.55e-17

7.0   6.99999999999999990   2.22e-16

8.0   8.00000000000000020  -2.22e-16

10.0  10.00000000000000000  -2.22e-16

12.0  12.00000000000000000  -2.22e-16

15.0  15.00000000000000000  -2.22e-16

17.0  17.00000000000000100  -4.44e-16

18.0  18.00000000000000000   4.44e-16

21.0  21.00000000000000000   4.44e-16

22.0  22.00000000000000000   4.44e-16

23.0  22.99999999999999900   8.88e-16

Another aspect of floating representation concerns conversion from floating-point to integer. When a positive floating-point value is converted to an integer value, the fractional digits are lost, i.e., truncated [2-2]. When a negative floating-point value is converted, the truncation may be toward or away from zero, depending on implementation.

Rule 2-1: When exactness counts in converting floating-point to integer, be sure the value being converted is non-negative.

Another type of rounding behavior is needed when using floating-point numbers to represent decimal fractions such as currency. It is possible to use floating representations for small amounts of currency, such as six figures with two decimal places; errors in the low-order bits will not produce a noticeable error in addition or subtraction. However, when arbitrary fractions are computed, they must be rounded to the proper number of decimal places. For example, consider the computation of monthly payment on a mortgage [Plum, 1983, 3-45]:

pmt = bal * (intmo / (1. - pow(1. + intmo, (double)-npmts)));

With balance (bal) equal to $82,500.00, monthly interest (intmo) equal to 16.25% / 12, and number of payments (npmts) equal to 25 times 12 (i.e. 300), the C computation produces a result (pmt) equal to 1137.29654996 (showing just eight places). Since this result is about 35/100 different from an exact number of pennies, it will not take many additions or subtractions before the intermediate results start to differ noticeably from true figures.

Thus, we are able to use floating representations only if we make sure to round the intermediate fractional results (from multiplies, divides, and function calls) back to even pennies, with a function such as this one:

round:

/* round - adjust floating dollars to even pennies */

#include "local.h"

double round(dollars)

double dollars;

{

double pennies;

pennies = floor(dollars * 100 + .5);

return (pennies / 100.);

}

which will give a "fair" or "symmetric" rounding. Thus, round(1137.2965) produces the result 1137.30, but round(1137.2925) would produce 1137.29.

The floor function produces a double result equal to the largest integer that is not larger than the argument; i.e., it "truncates" the argument's fraction. If we wanted a function that would round all fractional pennies upward (in favor of the bank), we could use the ceil function, which produces a double result equal to the smallest integer that is not smaller than the argument.

roundup:

/* roundup - adjust floating dollars to even pennies

* round all fractional pennies upward

*/

#include "local.h"

double roundup(dollars)

double dollars;

{

double pennies;

pennies = ceil(dollars * 100.);

return (pennies / 100.);

}

Using roundup(1137.2925) produces 1137.30.

These observations on representation and rounding are a few of the simpler things that can be said about avoiding surprises with floating-point data. If you have uses for floating-point arithmetic that are more complex than these simple examples, consult a text on numerical analysis such as Hamming [1973].

We turn now from errors of representation to errors of computation.  Most of the math functions can fail, if they are given arguments which do not produce a proper result. For example, the square root of a negative number, such as sqrt(-1.0), has no meaning in "real" arithmetic.  This is known as a domain error; the argument is outside the domain over which the function is defined. For another example, ten raised to the one-millionth power, pow(10., 1e6), cannot be represented in any (current) floating representation. This is known as a range error; the result cannot be represented as a proper double number. In all such error cases, the function will return some value, but the value returned will not be the "correct" value [2-3]. The sqrt function returns zero, when given a negative argument. The pow function returns a very large number (appropriately positive or negative) on an "overflow" range error. This "very large number" is defined in <math.h> as HUGE_VAL (or in some versions, HUGE).

I would advise you against testing for errors by comparing the returned value against HUGE_VAL (or -HUGE_VAL, or zero), for several reasons. First of all, these are in general valid (albeit unlikely) data values. Secondly, making such tests requires detailed knowledge of the various error returns for each math function - knowledge which is otherwise useless. Also, there are three different possibilities - -HUGE_VAL, O, and HUGE_VAL - and you must know which are possible in each case. Finally, different versions of the library have differed in their error-return behavior.

A more reliable way to test for errors is by using the global variable errno. This variable is set to a non-zero value by any of the math functions that encounters an error. Thus, setting errno to zero prior to a computation, and testing it afterwards, will reveal whether any library errors were reported during the computation:

errno = 0;

perform computation

if (errno != 0)

handle the error situation: report, compute differently, etc.

else

computation was error-free

Rule 2-2: Test errno before using results from the math functions.

(See the appendix for a list of math functions in the library of recent C compilers [2-4].)

2.2 Error-handling

One important issue in "no surprises" programming is how to handle errors encountered during computation.  To be general about it, we define "error" as any indication that the program cannot produce the expected result. Associated with each possible error is a "restart point," the place at which some action is taken so that the process can continue. The method of handling errors often depends upon whether the program is intended for use in a "stand-alone" environment, where execution must continue no matter what, or for use under an operating system, where a user is available to re-execute the command.

Throughout the rest of the book, we will refer to these categories of error-handling:

Diagnose and terminate: This is one of the simplest approaches, suitable to interactive use only. The restart point is re-execution of the entire program.

Diagnose and re-prompt: In this approach, computation does not begin until the program determines that input values will not cause errors in processing. If inputs are invalid, the program can prompt for replacement values. The restart point is the prompting for re-entry.

Re-try the operation: If the error is of a transient nature (e.g., a read error from disk), the program can re-try the operation, up to some pre-set maximum number of re-tries. The restart point is the re-try.

Terminate one transaction and proceed to the next: In this approach, the input is a series of independent transactions, such that one transaction can be diagnosed and terminated. This approach is common on batch systems in which the user is not assumed to be interactively monitoring the execution, but it may also be appropriate in some standalone systems.  The restart point is the beginning of the next transaction.

Re-execute the computation with different algorithm: Sometimes one has a fast algorithm which sometimes fails, and a slower algorithm which always works. In this case, the restart point is the execution of the slower, more robust algorithm.

Ignore errors completely: This approach is only suitable when the error will be obvious to the user from the output of the program. Since no action is taken, there is no restart point. This is usually not suitable for reliable programming.

2.3 Character Tests, with <ctype.h>

Topics assumed from previous book: <ctype.h> [3.5].

In Chapter 1, a safe macro function is defined to be one which evaluates each argument exactly once, and observed that safe macro functions can reliably be used as replacements for function calls. This approach to macro-versus-function tradeoff is most simply illustrated by the "character-tests" in <ctype.h>. These have usually been implemented as safe macros. ANSI C requires that they also have true function versions in the library. As mentioned in Chapter 1, the true function version of, for example, isdigit can be obtained by using

#undef isdigit

(See the appendix for a list of the character-test functions from the library of recent C compilers[2-5].)

These functions are the most portable way of testing and converting characters, so their use is encouraged.

Rule 2-3: Use the <ctype.h> facilities for character tests and upper-lower conversions.

2.4 Boolean Data

Topics assumed from previous book: Semi-Boolean tests [3.2], bool [3.21], tbool [3.21].

As described in the introductory book, the logic of test conditions (if, while, for, do) is "semi-Boolean," in that zero means false and non-zero means true. In other words, any test expression is implicitly compared against zero, and many existing C programs employ this implicit comparison in shorthand uses. One often sees

if (c)  instead of  if (c != '\0')  and

if (p)  instead of  if (p != NULL)

There is no problem with portability here; nul character and NULL pointer are always guaranteed to be equal to zero in comparisons. However, the C syntax does have a potential for one common bug:

if (i = j)  either wrong or misleading

is syntactically valid; it means the same as

if ((i = j) != 0)

but the mistaken use of the "single-equal" assignment instead of the "double-equal" comparison is a common mistake. A new generation of C interpreters has the potential for eliminating this type of mistake, but the only simple way to achieve it is to require that all test conditions must clearly be "Boolean" expressions. Any expression whose top-level operators are relational and logical is obviously a "Boolean" expression, but what about Boolean variables? From the interpreter's point of view, any value other than zero or one might be assumed to be non-Boolean.

From the human reader's point of view, the program will be easier to understand if Boolean variables are clearly indicated as such. In the type scheme of C Programming Guidelines [Plum, 1984], two defined-types are provided for Boolean variables:

bool   designates an int-sized Boolean variable, and

tbool  designates a char-sized Boolean variable.

With an eye toward future automated assistance, we propose the following reliability rules:

Rule 2-4: Make sure that Boolean variables are assigned the values zero and one. This means that the type tbool is always adequate, and if this rule is part of local standards, the types bool and tbool could be made synonymous.

Rule 2-5: Make sure that each test condition is Boolean, involving only Boolean types or relational and logical operators.

In most modern compilers, there is no efficiency penalty for saying

while (*s1++ != '\0')

instead of

while (*s1++)       not strictly Boolean

2.5 Enumeration Types

Recent C compilers provide enumeration types , which allow declarations like this:

enum stoplight {red, yellow, green};

This establishes enum stoplight as a type which can be used in further declarations, such as

enum stoplight main_st, front_st;

And the enumeration constants (red, yellow, and green) become symbolic names for the integer constants 0, 1, and 2. Each of the enumeration constants can be explicitly initialized, as in

enum stoplight {red = 'r', yellow = 'y', green = 'g'};

If an uninitialized enumeration constant follows one which is initialized, its value is one greater than the previous constant. To avoid surprises, we suggest this rule:

Rule 2-6: An enumeration's constants should all be initialized, or else none of them should be initialized.

There are some reliability problems with the semantics of enum. In a strict usage of enum data, the variables main_st and front_st would be assigned no values other than the named enumeration constants, red, yellow, and green, like this:

main_st = red;

That is not, however, what C requires. Instead, an enumeration variable is treated just like an int variable, except that its actual storage size might be optimized according to the range of the enumeration constants. The declaration of an enumeration type is therefore more or less equivalent to a listing of #define'd names, with a few minor advantages: The scope of the names is limited (just like ordinary identifiers), and it is marginally easier to add new constants to the list without having to manually change their values.

Future implementations could invoke stricter checking if programs were written according to stricter rules:

Rule 2-7: Write programs as if enumeration variables could receive no values other than the associated enumeration constants. Treat the enumeration types as if they were unique types, not for any arithmetic int usages. Convert between enumeration variables and integer values only by use of an explicit cast.

This allows, at least, easier inspection of the code by reviewers, and creates the possibility of stricter lint-like checking.

2.6 Range-Checking

Failure to attend to proper ranges of variables can lead to interesting reliability problems. One example from the field of computerized aircraft navigation: a heading variable was specified to be measured in degrees. But it was not discovered until airborne system test that one part of the system assumed that heading ranged from 0 to 360, while another part assumed that the range was -180 to 180.

For a concise representation of ranges in comments, we can use {low:high}. (Using a "two-dot" notation like "low..high" might seem more natural, but it becomes confusing when the range values are floating-point constants with decimal points.) The specification of a range comment could look like this:

double heading;  /* degrees: {0:360} */

If we follow the notational convention that the colon is taken to mean "must be," then the comment can be understood to mean "heading is measured in degrees, and must be between 0 and 360, inclusive."

Executable assertions can be useful for putting teeth into assumptions about ranges of data. In this case, an appropriate assertion on entry to a function using heading would look like this:

asserts(0<= heading && heading <= 360, "heading range ok");

Tests of this form are common enough to deserve a macro:

#define IN_RANGE(n, lo, hi)  ((lo) <= (n) && (n) <= (hi))

which would allow the assertion to be given in this form:

asserts(IN_RANGE(heading, 0, 360), "heading range ok");

In C, it is often appropriate to consider the "one-too-far" value as part of the range of a variable. Consider, for example, a variable month_no which is a subscript into an array of monthly information. We might consider that its range would be from 0 through 11, but such variables are often used as loop indexes in contexts like this:

for (month_no = 0; month_no < 12; ++month_no)

process array[month_no];

The for loop always involves a "one-too-far" value at the end of the loop; month_no must be incremented to 12 in order for the loop test (month_no < 12) to terminate the loop.

Rule 2-8: Include "one-too-far" values in the ranges for variables, if they are needed for loop terminations or other testing purposes.

The discussion of ranges would not be complete without considering the sizes of objects and the ranges of subscripts. C is moving toward the ability to handle objects whose size cannot be represented in an unsigned int; ANSI C envisions that some environments will need to use unsigned long to represent the sizeof anything. Several of the standard headers will define a type named size_t, which will be either unsigned int or unsigned long, according to the environment. Hence this rule:

Rule 2-9: Function parameters accepting the size of an arbitrarily large object should be declared with size_t type.

For an example of the use of size_t, consider the reverse function, which reverses the characters in a string. To be applicable to any string in any environment, the function needs to use a subscript of type size_t:

reverse:

/* reverse - reverse the order of a string */

#include "local.h"

void reverse(s)

char s[ ];

{

char t;

size_t i, j;

if ((j = strlen(s)) == 0)

return;

for (i = 0, j = j - 1; i < j; ++i, -- j)

SWAP(s[i], s[j], t);

}

A related question concerns how a subscript variable should be declared. The lowest subscript of an array is always zero in C, but regarding the highest subscript there are new complications with recent versions of C. Some implementations allow declaration of arrays with more elements than the largest (signed) int value, whereas in K&R [Kernighan and Ritchie, 1978], subscripts were always representable as int values. Recent compilers for C allow for larger subscripts. For one example, the "large memory model" of the 8086 family of machines allows for objects larger than 32K elements, but the int is still 16 bits. Thus, in general, portable code (especially library functions callable in any context) must assume that a subscript could require an unsigned representation. Fortunately for portability, there is always a valid "one-too-far" value greater than the largest possible subscript, because in zero-origin indexing, the largest subscript is always one less than the largest dimension value for the array. However, a variable capable of holding the largest subscript must be an unsigned variety of integer, which eliminates the possibility of a "one-too-far" value at the low end.

One could always use size_t for subscripts, but this may be over-kill in many situations where the allowable range of subscripts does not need to span the memory space. We propose instead to use a type named index_t. What is the distinction between size_t and index_t? size_t is determined by the implementation, whereas index_t is chosen by the project. size_t is required for all general library functions, because there is no restriction upon the calling function. index_t could, by project-wide choice, be made smaller than size_t, if a maximum size for arrays is mutually agreed upon within the project. In the rest of this book, we will use index_t variables for subscripts. In our local.h header, index_t is defined as unsigned int. You can make the definition int, unsigned int, long, or unsigned long, according to the needs of your project. In stddef.h, size_t is also defined as unsigned int; you should change size_t to unsigned long if your implementation requires it.

2.7 Signed and Unsigned Arithmetic

Topics assumed from previous book: ones complement [2.1], twos complement [2.1], overflow [3.20].

In this section, all our examples will be easier to describe if we assume that int has 16 bits, but the principles are the same for any size of int. Most C machines use twos complement arithmetic, in which the smallest int has the value -32768, and the largest int has the value 32767. (These values are named INT_MIN and INT_MAX, respectively, in <limits.h>, as described in Section 1.1.) On a hypothetical 16-bit ones complement machine, INT_MIN would be -32767 and INT_MAX would be 32767. Notice that the very smallest twos-complement integer has no counterpart in ones complement, nor does it have a corresponding positive integer in either system.

While twos complement is the most popular usage in the C world, programmers aiming for the widest portability must consider various problems regarding ones complement, even if their current machine does not use it. More precisely, these are reliability problems for portable code, in that they can cause nasty surprises if not attended to. We shall see several such problems in this section.

Signed and unsigned arithmetic have significantly different properties in C. Ordinary signed integers correspond to positive and negative numbers along the number line, like a thermometer:

+- ... +-+-+-+-+ ... -+

INT_MIN      0 1 2    INT_MIX

Subtraction of two ordinary int values produces a (signed) int result, as in the subtraction of two temperatures:

int temp0, temp1, delta_temp;

delta_temp = temp1 - temp0;

This gives an algebraically correct result as long as the true difference lies between INT_MIN and INT_MAX, inclusive. But if temp1 and temp0 are too far apart, an overflow takes place in computing delta_temp, and the result is undefined.

Unsigned arithmetic is based on a "modulus" arithmetic, like an odometer or a timer counter. Consider two unsigned integers timer0 and timer1 which represent successive samples from a timer counter:

unsigned int delta_time, timer0, timer1;

delta_time = timer1 - timer0;

Assuming that our integers are 16-bit values, the largest unsigned int value (named UINT_MAX in <limits.h>) has the value 65535. If the value of timer0 is 12, say, and the value of timer1 (sampled some time later) is 10, then delta_time will have the value 65534; i.e., timer1 was sampled 65534 ticks later than timer0. By the definition of unsigned arithmetic, no overflow is possible in the computation. The "wrap-around" that can occur is not considered an error, whereas overflow is an error. From the formal point of view, any result of the subtraction is a well-defined result. From the practical point of view, the result corresponds to the actual delta_time only if the true value is less than or equal to UINT_MAX (65535, in this environment). This "timer" model corresponds closely to the underlying semantics of C, in that the difference between a later time and an earlier time is always a positive number.

Notice that it would be an error to assign the unsigned int value of delta_time to an int variable; all the values greater than INT_MAX would exceed the range of the int. Such an assignment is not a syntax error, however; the compiler would map a very large value into some negative integer value. The details of the mapping will be discussed shortly.

Programmers often use unsigned arithmetic as if it modeled the "positive number line," like the measurement of weight:

unsigned int weight0, weight1;

int delta_weight;

delta_weight = weight1 - weight0;

In this sort of usage, the difference between the two unsigned numbers is conceptually a signed number, according to whether one weight is greater or lesser than the other. In twos complement arithmetic, this computation is quite well-behaved and unobjectionable. The unsigned difference between weight1 and weight0 when converted into a signed number produces just the right result, as long as the "real" result is between INT_MIN and INT_MAX, inclusive.

In ones complement, however, there is an uncertainty about the conversion. Assigning the unsigned int value to the (signed) int variable can be done in any way that the compiler writer has chosen. The choice may have been to model the twos-complement behavior, in which case one possible result (-32768) is unrepresentable. (We will call this the "twos-complement mimic" treatment.)

Alternatively, the compiler writer may have chosen to simply assign the same bit representation, in which case all negative results will be "one greater" than expected. (We call this the "raw-bits" treatment.)

In either case, there are surprises, but the "twos-complement mimic" behavior is slightly less surprising than the "raw-bits" behavior. If a ones-complement environment uses the "raw-bits" treatment of conversion from unsigned to signed integers, we must do some work in order to reverse the "modulus" behavior of signed-to-unsigned conversion.

Here is an example. If the (signed) integer -1 is converted to an unsigned integer, the unsigned result consists of all l-bits. When these "raw bits" are converted back to signed integer, the result is zero. Here, then, is a 36-bit ones-complement version of an "unsigned int to int" conversion macro, called UI_TO_I, which can be found in the header portdefs.h:

#define UI_TO_I(n) (((n) & 0x800000000) == 0 ? (n) : -(n) -1)

Be aware that this is a non-unique mapping. Both 0 and 0x800000000 (2 to the 35-th power) are mapped into 0. Since 2 to the 35-th power could not occur in the typical 32-bit twos-complement environment, this is not likely to cause any portability problem.

In a twos-complement environment, all that UI_TO_I has to do is to cast the result to int:

#define UI_TO_I(n)  (int)(n)

The problems arising from subtraction of unsigned int values are not restricted to obscure portability problems regarding ones-complement systems. If they were, most programmers could safely ignore them. But as these examples show, the difference between unsigned values has a built-in duality of interpretation. The situation is as if C had a special type available for unsigned difference; let us call it "questionably signed," or qsigned, to invent a C-like notation. A qsigned number is a value on a number line with two different, coexisting value labelings:

UINT_MAX

V 0 1 2     INT_MAX  [unsigned interpretation]

+- ... +-+-+-+-+ ... -+

INT_MIN?   -1 0 1 2     INT_MAX  [signed (2s-comp) interpretation]

Any computations involving qsigned values are performed in the same (twos-complement) arithmetic that is used for unsigned values. (If the machine's signed arithmetic is ones-complement, the INT_MIN limit is actually equal to the smallest qsigned value plus one; this corresponds to the conversions that should take place when going from unsigned values to signed values in such environments.) When a qsigned int value is assigned to an unsigned int variable, the unsigned interpretation is followed. When a qsigned int value is converted to a (signed) int value (using the UI_TO_I macro), the signed (twos-complement) interpretation is followed.

So far, this is just an explanation of the underlying meaning of well-behaved computations. However, if a qsigned int value is assigned to a long int variable (in an environment where int is smaller than long), and no explicit conversion is used, there is an intrinsic confusion. In the examples above, one could be misled into thinking that

long delta_timer;

delta_timer = timer1 - timer0;

produces a non-negative value, and that

long delta_weight;

delta_weight = weight1 - weight0;

produces a signed value. Of course, both cannot be true; the difference between them is only in the mind of the programmer, not in the semantics of C. (The second example is the incorrect usage -- the unsigned difference becomes a positive long integer.) Accordingly, we suggest that the conversion of a qsigned int result to a larger long int type should (for maximum reliability) be considered an undefined operation. In other words, before assigning or casting a qsigned value to a larger int type, one should convert the value explicitly either with UI_TO_I or with (unsigned). Thus, we should write

delta_timer = (unsigned)(timer1 - timer0);

and

delta_weight = UI_TO_I(weight1 - weight0);

This can all be summarized by a reliability rule:

Rule 2-10: When two unsigned int's are subtracted, convert the result using either (unsigned) or UI_TO_I.

One final observation about integer signs: the sign of the remainder (%) operator is implementation-defined, when the operands are of different sign. This sometimes constitutes a portability problem, when the programmer has assumed that i % j is always positive. To provide a true (never negative) modulo operation, we can use an IMOD ("integer modulo") macro from portdefs.h:

/* modulo macro giving non-negative result */

#define IMOD(i, j) (((i) % (j)) < 0 ? ((i) % (j)) + (j) : ((i) % (j)))

/* if i % j is never negative, replace with the following line: */

/* #define IMOD(i, j) ((i) % (j)) */

Rule 2-11: Use IMOD (or some similar mechanism) to ensure that a nonnegative modulo result is produced.

2.8 Overflow

Strictly speaking, there are two different types of overflow. When an object is assigned a value that is too large for the object, we have a "range" overflow. And when an operation produces a result that is too large for the intermediate result, we have an "intermediate" overflow. The consequences are the same in both cases: the end result is incorrect. Range overflows have been discussed earlier; now we will attend to intermediate overflows.

Many C programmers have assumed that (on a twos-complement machine) overflows can be ignored in most cases. Consider the following (simplified) version of the atoi function:

atopi(#1):

/* atopi - convert string to positive integer */

int atopi(s)

char s[ ];

{

int n = 0;

int i;

for (i = 0; isdigit(s[i]); ++i)

n = 10 * n + s[i] - '0';

return (n);

}

We will refer to a "fussy" overflow as one which would not prevent the computation of the correct answer if overflow-checking were not performed. For example, during the computation of atopi("32767"),  a "fussy" overflow takes place when the final '7' is added:

3276 * 10  =  32760

  + '7'      +   55

             ------

              32815  momentarily greater than INT_MAX

  - '0'      -   48

             ------

              32767  now within proper range

A "true" overflow takes place when the correct answer is not capable of being produced, as in atopi ("999999999999").

On a machine without overflow checking, "fussy" overflows are often ignored by the programmer. The overflow causes a "wraparound" in the temporary value, but the subsequent subtraction brings the result back into range. The programmer is often unaware of the overflow until the program is ported to a machine which checks for overflow in the machine hardware. One simple way to prevent fussy overflows is to perform the operation in unsigned arithmetic, since overflow checking is explicitly disabled for unsigned operations. A more precise way, in the case of atopi, would be to parenthesize the problematic expression like this:

n = 10 * n + (s[i] - '0');

If the operator inside the parentheses were a "plus," the technique would not work, because the compiler is free to re-arrange commutative-associative operators, even across parentheses. Since, however, it is a "minus," the subtraction is guaranteed to be performed before the addition (unless the compiler has a bug in this delicate area, that is).

Existing conversion functions in C libraries have tended to ignore overflows. In recent C libraries, there are conversion functions with a more constrained behavior. On overflow, they return a maximum value, and use errno to indicate that an error has taken place. In the case of atopi, we could more simply return a special value, such as -1, to indicate the overflow error. No overflows will take place during the computation, and an excessively-large argument value is diagnosed. Applying this philosophy to our atopi function would produce something like this:

atopi(#2):

/* atopi - convert string to positive integer {0:INT_MAX} */

#include <limits.h>

int atopi(s)

char s[ ];

{

unsigned n = 0; /* the converted number: {0:INT_MAX} */

int i;

for (i = 0; isdigit(s[i]) && n <= INT_MAX/10; ++i)

{

if (n > INT_MAX / 10)

return (-1);

n = 10 * n + (s[i] - '0');

}

if (n > INT_MAX      isdigit(s[i]))

return (-1);

else

return (n);

}

Rule 2-12: In (signed) integer arithmetic, assume that overflow is illegal, may be detected (hence should never be programmed), and cannot be trapped or ignored. earlier, using unsigned can at least prevent "fussy" overflows.

2.9 Data Properties

Throughout the book, we will refer to three aspects of reliable use of data: type, properties, and allowable operations.

Type

Properties

Allowable operations

Type

By "type," we refer both to the way the data is represented in memory, and to the syntactic type on the declaration. Early C compilers were rather loose about the agreement of data types; code like this was allowable:

char *p;

int i;

i = p + 2;   loose and dangerous

Recent C compilers will flag this free interchange between pointers and integers as a syntax error. One reason for the tightening-up is that programmers have been found to make mistakes in mixing up pointers and integers, which leads to obscure bugs. Another reason is that, in some environments, an ordinary int is not large enough to hold a pointer. In the so-called "large model" of the Intel 8086 and 8088, a pointer takes 32 bits, while the int is only 16 bits.

C performs a number of conversions upon expression values, so there is an important distinction to make between the "declared" type and the "converted" type. For example, in

int i, j;

char c;

i = j + c;

the declared type of c is char, but when c appears in an expression like this, its converted type is int.

The main point is that C programmers must know the exact type of each variable and each expression in which the variable is used.

Properties

The most basic distinction when talking about data properties is the distinction between an undefined value (e.g., "garbage," or uninitialized), and a defined value. One common example of a "undefined" error comes from forgetting to initialize a counter:

long sum;

int i;

for ( i = 0; i < N; ++i )

sum += a[i];      error - sum not initialized

In C, automatic variables are undefined until they are initialized; static variables are initialized by default, so they are initially defined. A variable can acquire an undefined value as the result of an invalid operation;

n = m / 0;

causes n to become undefined, possibly terminating execution as well.

Similarly, when an overflow takes place in a computation, the result of the expression is undefined. And when a value is assigned to an object that is too small to hold the value, the object's value becomes undefined.

Besides this fundamental distinction between defined and undefined values, a variable may have other properties ascribed to it by the programmer. In Section 2.6, we discussed programmer-defined ranges for variables, documented like this:

double heading;   /* degrees: {0:360} */

By this comment, the programmer indicates that the property of being between 0 and 360 is an important property for this variable. If heading receives the value -1, the variable is "defined" from the point of view of C, but it is not properly defined according to the programmer's intention.

In previous sections, we saw other properties that the programmer can ascribe to a variable, by using an informative defined-type: metachar (a valid char-sized number or EOF), bool and tbool (zero or one), and bits (an unsigned number for bit manipulation).

We will use the term well-defined to refer to a value that conforms to the programmer's stated defining property. If, for example, a metachar variable were to receive the value 5000, the variable would not be "well-defined," because it exceeds the range of proper char-sized numbers.

Besides stating desired properties on the declaration of variables, the programmer can also document properties of function returned values. A function declared like this

bool is_empty();   /* is node empty? */

or like this

int is_empty();    /* is node empty?: bool */

is indicated to return only Boolean values. This documentation is important for determining that the function is producing a value that is appropriate for the context in which it is used.

We will be following a convention that a comment with a colon followed by a property name means "must have this property."

Rule 2-13: Document the defining properties of declared names in a comment on the declaration, using a convention such as "colon followed by property."

Allowable operations

The type and properties of a expression determine the operations that are allowable on that expression. For example, a bool variable should not be the operand of arithmetic operations like multiply or divide; the operations just do not make sense for the intended properties of the variable. Sometimes the type itself restricts the operations. For example, bitwise operations are not allowed upon any floating-point values.

In later chapters, we will see more complicated objects such as arrays, structures, stacks, and trees, each of which has its own repertoire of allowable operations.

Go to Chapter 3    Return to Table of Contents


Reliable Data Structures: CHAPTER 3: ARRAYS

 Return to Table of Contents

CHAPTER 3: ARRAYS

An array is an aggregate object, which is composed of other objects, its elements. The dimension of an array is the number of objects that it contains. Its size is given by multiplying the dimension by the size of each element.

In C, arrays are used for storing character strings as well as for their more universal uses. As we shall see, this makes a reliable treatment of "properties of an array" somewhat more complex than in languages in which strings are objects in their own right. As always, our concern is for rules of programming which will ensure the reliable use of arrays.

3.1 Array Data

Topics assumed from previous book: arrays and subscripting [3.14], array arguments [5.4].

Just as we did for scalars, we will consider three aspects of array data: type, properties, and allowable operations.

Type

Properties

Allowable operations

Type

In C, the declaration of one variable, such as line in

static char line[10] = "msg";

consists of five components:

static    storage class (optional)

char      type specifier

line[10]  declarator

= "msg"   initialization (optional)

;         semicolon (of course)

Strictly speaking, the storage class and the type specifier can appear in any order, but declarations are clearer in a conventional order:

Rule 3-1: Storage class (if any) should precede the type specifier.

The examples will henceforth assume this rule.

For another variation, several variables can be declared in one declaration:

static char line_a[10], line_b[10];

declares two variables, line_a and line_b. This is in every way equivalent to two separate declarations:

static char line_a[10];

static char line_b[10];

It is a common error to assume that the following initialization applies to both line_a and line_b:

static char line_a[10], line_b[10] = "msg"; misleading

Rule 3-2: If a variable has an initialization, its declaration should have a source line to itself.

Setting aside the above complications, we will consider a declaration to consist of an optional storage class, a type specifier, a declarator, and an optional initialization. Now let us restrict our consideration to the non-optional parts, the type specifier and the declarator. For the rest of this section, when we refer to a "declaration," we will be referring only to the mandatory parts.

So declarations can be sliced into type specifier and declarator. But these syntactic categories are not of much practical importance to programmer; they are merely the syntactic terms. The important way to slice a declaration is into a variable name and a type. In a very simple declaration such as

short i;

the declarator consists only of the variable name (i), and the type specifier consists only of the type (short). But in a more complex declaration such as

short m[2];

the declarator m[2] contains the variable name m plus a part of the type. The type consists of what is left when the variable name is "scratched out" of the declaration, namely short [2]. In this sense, we can say that a declaration is a "sandwich" composed of a variable name and a type. The type is the "bread" wrapped around the variable name "meat" in the middle:

short m[2]  is the declaration

m     is the name

short  [2]  is the type

In C, the representation of an array consists of a sequence of contiguous objects (the elements of the array), running from low addresses to high addresses.

The meaning of an array declaration depends upon context. A parameter (typically declared with no stated dimension) is understood by C to be a pointer rather than an array (see Section 3.4). And a non-defining declaration of an external array, such as

extern short em[];

is an allowable form which declares an array of unknown size. (The size is determined by the defining instance, but a separately compiled function has no built-in way of determining that size.)

For an array whose size is known from the declaration (externally defined, or declared to be auto or static), we can define a useful macro for the dimension:

#define DIM(a) (sizeof(a)/sizeof(a[0]))

In other words, the dimension is the size of the array divided by the size of one element. Since both sizes are known at compile time, DIM(a) is effectively a constant, and can be used as such. One caution: the macro DIM should not be used with a pointer argument. The expression sizeof(p) always produces the size of one pointer, not the size of an array.

Properties

Reliability is often enhanced by maintaining a clear intuitive meaning for each data object. A data object, after all, reflects something about the world outside the program: a person's address, an amount of money, the pressure on an automobile brake pad, the temperature of a blast furnace, etc. Such interpretations are clearer if each object has a defining property which is maintained throughout a computation.

Rule 3-3: Document the defining property of a data object with a comment on its declaration. Ensure that this defining property remains invariant (unchanging) as much as possible throughout the computation, and document any exceptions.

This book's system of properties uses different names for aggregate properties than for scalar properties; this will allow us to say some simpler things about pointers later. An array is either complete (all scalar elements contained within the array are defined) or incomplete (one or more elements are not defined).

An array can become complete either by assignment to all its elements, or by an initializer on its declaration. (Any initializer causes the initialization of the entire array.)

Rule 3-4: A program is easier to write correctly and to understand if all arrays are made complete before the array is used.

Some arrays have other properties that are more important than "complete" or "incomplete." A "string," for example, requires defined characters only up to a nul (i.e., '\0') terminator. The characters after the nul terminator can be total garbage, and the array still has its defining property satisfied, namely being a string.

Rule 3-5: If an array's defining property can be true even if not all elements are defined, indicate the property on the array's declaration. For example,

char s[10];   /* : string */

Lacking such mention, it is assumed that the array must be complete (all elements defined) before its value is used.

For any array, there are boundary values to be considered at the lowest and highest subscripts. These will always be important in determining valid access ("avoiding out-of-bounds reference"). Every array algorithm requires verification of the proper handling of the subscript boundaries.

A second issue concerns the "one-too-far" value required to terminate an ordinary for loop that walks over an array. As discussed in Section 2.7, if subscripts can be unsigned integers, there is always a valid "one-too-far" subscript value at the high end, but none at the low end.

In many cases, the defining property for an array is more restrictive than mere completeness. Let us now consider an array as an object to be sorted. In doing so, we will need a notation for a subarray, i.e. a contiguous sequence of elements of an array. The notation a[low:high] will be used to mean the sequence of elements from a[low] to a[hi]. And the notation a[*] will mean "the entire array a." There are, of course, no such notations in C itself, but the notations will help us to speak more clearly about array algorithms.

To focus for now upon the general problem of sorting the elements of an array, the basic invariant property of such an array is that it contains individual-element values, the set of which remains unchanged from the start to the finish of the sorting process. In other words, the set of values remains invariant, although the positions of those values will be altered as the sort progresses. This is analogous to the sorting of a deck of cards; the values written on the cards do not change, but the ordering does change. To be precise, the array is a "multiset," which is like a set but individual values can occur more than one time. A deck of bridge cards is a proper set; each card appears only once (considering suits as distinct, as of course they are in bridge). A deck of Canasta cards is a multiset; each card appears several times in the deck.

We need to define one more property of an array. An array a is said to be sorted in ascending order if, for each i from 1 to DIM(a)-1, a[i] is greater than (or equal to) each element of a[0:i-1]. (An array containing only one element will be considered to be sorted, trivially.) More generally, a[low:high] is sorted (in ascending order) if, for each i from low+1 to high, a[i] is greater than (or equal to) each element of a[low:i-1].

The property of being sorted is not invariant during the operation of sorting an array. Being sorted is the goal property, not the invariant.

Allowable operations

C provides no built-in operators that act upon all elements of an array, but operations can be created from a succession of operations upon the individual elements. As we examine different data structures in this book, one common theme will be the creation of C functions and macros which perform specific operations upon a data structure. Designing the interface for a function is a subject in itself, and we will discuss the issue throughout the book. For now, our packaging criterion is minimal: we will employ the simplest C function that will encapsulate the data structure and algorithm that are our focus of attention.

The only allowable operations upon multiset arrays are those that perform a re-ordering without altering the actual values in the array. We have seen one such operation, in the form of the SWAP macro. Now we will examine the operation of sorting an array. For simplicity, we will restrict ourselves to an array of short integers. The "sorting" operation will take the form of a function with two arguments: an array parameter a designating the array to be sorted, and an integer n designating the number of elements in the array to be sorted. The function is supposed to arrange the n elements of a such that a[0:n-1] is sorted.

3.2 Sorting an Array

When we design the operations to be performed upon data, there are two more aspects of importance: loop invariants and assertions.

Loop invariants

Assertions and run-time validation

Loop invariants

Assertions and run-time validation

Loop invariants

In designing an algorithm to accomplish an operation such as sorting, the key design principle is to formulate a loop invariant which is true at each step of the computation, and which guarantees that the loop has attained its goal when the loop terminates.

The first sorting algorithm we will consider is called insertion sort. Using i as the loop subscript, the invariant of the insertion sort is "the subarray a[0:i-1] is sorted." The loop terminates when i reaches the "one-too-far" value, n. When it does, the invariant becomes "the subarray a[0:n-1] is sorted," which is the goal of the loop.

We will use the notation "name => property" to mean "name now has this property." Describing the loop invariants in program comments is often very useful documentation; we could, for example, write

for (i = 1; i < n; ++i)

{

/* a[0:i-1] => sorted */

insert a[i] into its correct place

}

/* a[0:n-1] => sorted */

To insert a[i] into the correct place, we copy it into a temporary (t), thus freeing up the location a[i]. Then, for each element of a[0:i-1] which is greater than t, we move it one position to the right, then insert t into its ordered position. (This is the insertion sort algorithm from Bentley [1984a].) Implemented as a function, this is what our algorithm looks like:

isort:

/* isort - sort array a (dimension n) using insertion sort

*/

#include "local.h"

void isort(a, n)

short a[];  /* array to be sorted; a[0:n-1]: multiset */

index_t n;  /* dimension of a */

{

index_t i;

index_t j;

short t;

for (i = 1; i < n; ++i)

{

/* a[0:i-1] => sorted */

t = a[i];

for (j = i; j > 0 && a[j-1] > t; --j)

a[j] = a[j-1];  /* a[0:n-1] => !multiset */

a[j] = t;  /* a[0:n-1] => multiset */

}

/* a[0:n-1] => sorted */

}

During the inner loop, the array momentarily loses its "multiset" property, because the value a[i] has been removed into the temporary t. This is indicated by the comment in the inner loop. As soon as t is assigned into its proper place, the array once again becomes a multiset.

Assertions and run-time validation

We have placed comments in our code that indicate the intended invariant conditions, but unfortunately in real life, errors in creating and maintaining the program may cause such comments to be false and misleading. A more reliable way of stating such properties is to insert executable assertions in the code.

There are trade-offs to consider: The complexity of the assertion tests can exceed the complexity of the code being protected by the assertions, in which case the assertions can become a source of more, not fewer, errors. And the time to execute the assertions can swamp the execution time of the code itself. We do not mind if the assertions are more time-consuming than the program specification allows for the final production program; the assertions are all disabled by compilation with the NDEBUG flag.

Rule 3-6: Use executable assertions whenever they are simpler than the code being protected, and when the time to execute the assertions is not much greater than the time required to execute the code.

In the case of a sorting function, we could provide an executable method of asserting that a given array or subarray is sorted, something like this:

tst_sort:

/* tst_sort - returns YES if array a is sorted

* specialized "short" version

*/

#include "local.h"

bool tst_sort(a, n)

short a[];

index_t n;

{

index_t i;

if (n <= 0)

return (NO);

for (i = 1; i < n; ++i)

{

/* a[0:i-1] => sorted */

if (a[i] < a[i-1])      /* compare adjacent elements */

return (NO);

}

/* a[0:n-1] => sorted */

return (YES);

}

According to our guidelines, we would not test this assertion inside the loop of isort. The time to test the assertions would become many times greater than the time to execute the code itself. But we could replace the final comment in isort.c

/* a[0:n-1] => sorted */

with this executable assertion:

asserts(tst_sort(a, n-1), "a is sorted");

The assertion is not required in the production version of the function; it checks the logic of the function itself, not any data provided from outside the function. Compiling with NDEBUG would turn off code generation for the assertion. We are using the executable assertions for documentation and for assistance during development, not for run-time checking in the final product.

Now we will consider the same underlying data structure, with a new algorithm, "quicksort."

For consistency, we will create a function with a calling sequence similar to that above:

void qksort(a, n)

short a[];

index_t n;

Loop invariants

Given an array or subarray, we pick the middle element and call it the pivot. Then the array is rearranged: those elements less than the pivot are moved to the left of the pivot, and those greater than the pivot are moved to its right. For convenience, we start by swapping the pivot with the initial element. The lowest subscript is lo, the highest subscript is hi, the subscript of the rightmost "less-than-pivot" element is lastlo, and the subscript of the element currently being tested is i. (This quicksort is somewhat simpler than most; it is adapted from Bentley [1984a].) During the loop of the sort, the invariant looks like this:

When the loop terminates, this is the picture:

Swapping a[lo] with a[lastlo] gives

Thus we end up with two smaller subarrays, a[lo:lastlo-1] (all of whose elements are strictly less than a[lastlo]) and a[lastlo+1:hi] (all of whose elements are greater than or equal to a[lastlo]). If the array is trivial (only one element), then both subarrays are empty; there is no work to do, and the function returns immediately. If there is more than one element, at least one of the subarrays is non-empty. The function calls itself recursively for both subarrays. When it returns, the array a[lo:hi] is sorted.

Assertions and run-time validation

There is little to add here except to assert tst_sort after completion of the function. The function is so short that assertions during the algorithm would be needlessly complex in comparison.

Here is the qksort function:

qksort:

/* qksort - sort array a (dimension n) using quicksort */

#include "local.h"

/* iqksort - internal function to partition a[lo:hi] */

static void iqksort(a, lo, hi)

short a[];   /* array to be sorted; a[lo:hi]: multiset */

index_t lo;  /* lowest subscript */

index_t hi;  /* highest subscript */

{

index_t mid = lo + (hi-lo)/2;    /* : {lo:hi} */

index_t i;                       /* : {lo+1:hi+1} */

index_t lastlo;                  /* : {lo:hi-1} */

short tmp;

SWAP(a[lo], a[mid], tmp);

lastlo = lo;

for (i = lo + 1; i <= hi; ++i)

{

if (a[lo] > a[i])

{

++lastlo;

SWAP(a[lastlo], a[i], tmp);

}

}

SWAP(a[lo], a[lastlo], tmp);

if (lastlo != 0 && lo < lastlo - 1)

iqksort(a, lo, lastlo - 1);

if (lastlo + 1 < hi)

iqksort(a, lastlo + 1, hi);

}

/* qksort - external entry point */

void qksort(a, n)

short a[]; /* array to be sorted; a[0:n-1]: multiset */

index_t n; /* number of elements */

{

if (n > 1)

iqksort(a, 0, n - 1);

asserts(tst_sort(a, n), "a is sorted");

}

The computation of mid is chosen to avoid a possible overflow. In the simpler expression

(lo + hi) / 2

the sum is susceptible to overflow for large values.

In Plum and Brodie [1985], optimization of this function is discussed at length. It is important to note that this recursive version of quicksort is vulnerable to some array values that cause the depth of the recursion stack to almost reach the number of elements in the array.

3.3 Strings

Topics assumed from previous book: string [2.4].

In C, a string is an array of char objects, containing a nul terminator '\0' that marks the end of the current string value. A string constant is written with double-quotes, like

"hello, world\n"

C does not require that every array of char'S must be a string; indeed, the only built-in knowledge of "string" semantics is that string constants receive a nul terminator. Nonetheless, a char array is so frequently used to hold a string that we will identify a special property to signify a nul terminated string: an array has the string property if all the characters up to the first nul terminator are defined.

If a char array has an initializer, the initializer will determine whether the array has the string property initially.

char a1[2] = {'a', 'b'};

is not a string, because there is no nul terminator within a1. On the other hand,

char a2[2] = "a";  and

char a3[2] = {'a'};

are both strings, according to the properties of string constants and of static array initializers.

It is often a useful style to declare an array of char'S with an explicit +1 in its bound, to emphasize that space is reserved for the nul terminator. All other things being equal, algorithms involving char arrays are simpler if all the arrays involved remain proper nul-terminated strings during the execution.

3.4 Pointers as Array Parameters

Topics assumed from first book: pointers as function parameters [7.3], pointers and arrays [7.4].

In this book, we will follow the usage of declaring pointer parameters to scalars with the "asterisk" notation and declaring pointer parameters to arrays with the "empty brackets" notation. Thus the function definition

fn(a, b)

char *a;

char b[];

indicates that a points to a scalar char, and that b points to the initial element of an array of charS.

No strong semantics can be attached to the distinction, because the distinction is unavailable for non-parameter pointers. The distinction still seems worth keeping, because parameter pointers are so often used. This style leads to a more graceful notation for two-dimensional parameters:

fn(table)

char table[] [20];

looks more natural than

fn(table)

char (*table)[20];

Relying upon the foregoing context, we will henceforth refer to array parameters when talking about pointer parameters which point to the initial elements of arrays. Similarly, string parameters are pointer parameters which point to the initial char in a string.

Further generalizing, we will use the term array pointer for any pointer used to point to array elements. (The term "pointer into array" might be more precise, but the brevity of "array pointer" is worthwhile because the term will be used so often in discussing array algorithms.) Remember, it would be inaccurate to call such a pointer a "pointer to array" -- an array pointer points to single elements, not to an entire array.

Array pointers (including array parameters) are different from array objects. The pointer itself can be undefined, or defined. It can be NULL, or it can point to valid storage. The array object being accessed through the pointer has its own properties: It could be complete or incomplete. It could be a multiset array (either sorted or unsorted). It could be a string. Or it could have any other property that we have defined for arrays.

In order to talk sensibly about the object being referenced we will need some new terminology. Because of the way C references arrays through pointers, the array being accessed is not simply the "indirect object" of the pointer (*p), because *p is just the individual array element pointed to by the pointer. We define the target of an array pointer to be the array object being accessed through the pointer. We will use the notation p[*] to mean "the entire array to which the pointer provides access."

In defining a meaning for p[*], we have reached the heart of an important reliability issue in the use of C: given an array pointer, exactly what object (i.e., what range of addresses) can we reliably access using that pointer? In the freest and least reliable usages, a pointer can be used to give access to the entire data space of the program:

int i;

int *p = &i;

printf("%d\n", p[12000]);   wild and dangerous usage

shows an access to whatever object is 12,000 elements further into the memory. But certainly this is not what we have in mind in using a pointer to access an array; the target of the pointer is meant to be only that array into which it is pointing.

In other words, *p ("indirect p") is only one array element; p[*] is the entire array accessed through p. In this example

short a[10];

short *pa = a;

the indirect object of p (*p) is the element a[0]. The target of (p[*]) is the entire array a.

For single-dimension arrays, there is no ambiguity; in the example

short a[10];

short *pa = a;

it is clear that a is intended to be the target of pa. But what of this example:

short b[10] [10];

short *pb2 = b[2];

Now we must make a choice and be consistent about it. Did we mean that pb2 is meant only for providing access to b[2], the second row of b, or that it is meant to provide access to the entire array b? Our choice is that when an array pointer is given a value, by assignment or passing an argument, the actual right-hand-side expression being assigned will be taken to indicate the target of the array. Thus, in the second example, the target of pb2 is the array b[2] only.

One concession to common usage of C: The assignment

p = &a[n];

is taken to mean the same thing as

p = a + n;

which means that the entire array a is the target of the pointer.

One further notational convenience: The properties of the target can be attributed to the pointer itself. For example, "pointer p is a string" means that the target of p is an array with the "string" property. This allows more conventional and convenient documentation:

/* p: string */

means the same thing as

/* p[*]: string */

What are the practical implications of all these distinctions? There are two: determining the properties of the target of the pointer, and providing the possibility of range-checking the use of the pointer (i.e., avoiding "run-away pointers").

"Properties of the target" is the simpler of the two considerations. All that is meant here is that, for example, when an array of char'S is passed as the second argument to strcpy, that array should be a string. If we wish to state this constraint explicitly in documenting a function such as strcpy, we can use a notation like this:

char *strcpy(s1, s2)

char s1[];      /* s1:!NULL, sizeof(s1[*]) > strlen(s2) */

char s2[];      /* s2:string */

which means that the pointer s1 is not NULL, that the array to which s1 provides access is large enough to hold the result, and that the array to which s2 provides access is a string.

Regarding range-checking in reliable programming, we desire not to access any objects that are outside the extent of the pointer's target. (This is why the extent of the target must be unambiguous.) We will therefore adopt the convention that every pointer value has a range associated with it, which may actually be monitored by an interpreter or may just be a conceptualization guiding the programmer towards reliable code.

Section 2.6 discussed how scalar data objects could have a range associated with their declaration; the range lasts throughout the lifetime of the variable. For pointers, the situation is different; a pointer can be used to point into several different arrays as the computation progresses. Therefore, we will consider that the range of a pointer variable is dynamically set by each operation upon the pointer.

When a pointer is assigned an expression value involving an array target, the address range of the target determines the range of the pointer. When a pointer is assigned an expression value involving a pointer, the left-hand-side pointer acquires the range of the right-hand-side pointer. (By the rules of C, the right-hand-side can have only one array or pointer operand, so the source of range information is unambiguous.)

Before we say exactly what the range of the pointer is, we have to consider the possible "one-too-far" value of an array pointer. In a loop like

for (p = a; p < a + DIM(a); ++p)

process *p;

we are counting upon p being eventually incremented to the address which is one byte greater than the last byte of the array a. The validity of the "one-too-far" value was agreed upon by the ANSI C committee in March 1985; it is vital to many existing C programs, which would be needlessly complicated otherwise. We will therefore always include the "one-too-far" value in the range of a pointer, with the understanding that this value is only good for arithmetic and comparisons, not for indirection.

If a pointer receives a value outside its range, we will consider that pointer to be undefined.

Now it is time for some examples. The comments will indicate the hypothetical addresses of the array elements:

static char m[10][10] = {0};  /* hypothetical address range: 2000:2099 */

char *p;

p = m[0];       /* p == 2000; range {2000:2010} */

/* p can access all of m[0], plus "one-too-far" */

p = (char *)m;  /* p == 2000; range {2000:2100} */

/* p can access all of m, plus "one-too-far" */

p = m[2];       /* p == 2020; range {2020:2030} */

/* p can access all of m[2], plus "one-too-far" */

p = p + 1;      /* p == 2021; range {2020:2030} */

/* value of p changes, but not its range */

++p;            /* p == 2022; range {2020:2030} */

/* again, value changes, but not range */

p += 8;         /* p == 2030; range {2020:2030} */

/* p is now "one-too-far"; valid value, */

/* but cannot be used for indirection */

++p;            /* p is now out-of-range; "undefined," strictly */

These rules for the range of pointers are, at the least, useful guidelines for the writing of reliable programs. They also can be enforced by compilers and interpreters which do dynamic bounds-checking [3-1]. They reflect the actual practice of working C programmers who write reliable programs that do not make out-of-bounds references in their use of pointers. The existence of precise rules such as these means that C is a language which can have both the expressive freedom of pointers plus the reliability of bounds-checking.

These concerns for the reliable uses of array parameters motivate the creation of a new general-purpose string-copying function. We will describe here a new function named strfit, which ensures that the target string is both bounds-checked and nul-terminated. If insufficient space was provided, the target is truncated but still nul-terminated. The function returns a bool value: a return of YES indicates that there was sufficient space for a complete copy.

strfit:

/* strfit - copy s2 to s1, chopping to n chars

* return YES if sufficient space, no if not

*/

#include "local.h"

bool strfit(s1, s2, n)

char s1[];

char s2[];

size_t n;

{

size_t i;

for (i = 0; i = n-1 && s2[i] != '\0'; ++i)

s1[i] = s2[i];

s1[i] = '\0';

return (s2[i] == '\0');

}

3.5 Arrays of Strings

An array of strings can be a handy format for the storage of simple tables. Consider

static char meas_codes[][6] =

{"each", "box", "lb");

This defines an array meas_codes with three rows of six char'S each, which looks something like this:

If we have a string to look up in this table, we can use a simple loop like this:

for (i = 0; i = DIM(meas_codes); ++i)

if (strcmp(s, meas_codes[i]) == 0)

found a match

We are passing the expression meas_code[i] to the strcmp function. According to the rules above, this means that we expect strcmp to be able to examine all the characters of the i-th row of meas_codes, but not to walk around in any other storage.

The strings are all self-contained in the array meas_codes; the table does not itself contain any pointers. For that variety of table, we must cover pointers in more detail. The details of pointers are the subject of the next chapter.

Go to Chapter 4    Return to Table of Contents


Reliable Data Structures: CHAPTER 4: POINTERS

 Return to Table of Contents

CHAPTER 4: POINTERS

Pointers are both powerful and dangerous in C. Programs can perform arbitrary manipulations of their data space in C. This is essential for many of the system-level uses of C. Because C has this expressive power, a style of programming oriented toward portability and reliability must place restrictions on the uses of pointers.

4.1 Declaration of Pointers

Topics assumed from first book: Pointers [7].

The first rule for reliable use of pointers requires that the programmer fully understand the declaration syntax of C:

Rule 4-1: In each pointer assignment, the right-hand-side value must have exactly the same ("converted") pointer type as the left-hand-side.

Recall from Section 2.9 that the "declared" type of the right-hand-side may differ from the "converted" type. In the assignment

char s[10], *p;

p = s;

the "declared" type of s is char [10] and the "converted" type is char *, so the assignment is proper. Here, by contrast, is an improper assignment:

char s[10];

int *pi;

pi = s;  improper mixed-type pointer assignment

On some systems, pointers to char have a different representation than pointers to int. On many systems, the address of s could be an odd number which would cause a hardware trap if we attempted to reference an int at that location. These are some of the reasons why compilers enforce the agreement of pointer types.

We will, therefore, take some time now to attend to the exact form of pointer declarations in C.

As we saw in Section 3.1, a declaration in C is a "sandwich" of variable name and type; the declaration

short m[2];

says that the variable m has type short [2] (array of 2 short's).

Turning now to pointers, the asterisk (*) when used as a declarator-symbol means "pointer-to." Thus,

char *pc;

declares pc to be a pointer to objects of type char, or as it is more commonly said, a pointer to char. (The phrase "pointer to type" really means "pointer to an object of type type.") According to the "sandwich rule," the declaration is composed of the variable name (pc) and the type (char *). For another example,

char **ppc;

declares the name ppc to have the type char ** (pointer to pointer to char):

char **pc  declaration

pc  name

char **    type

Another way of stating the "sandwich rule" is to list the possibilities:

if x() has the type T,  then x has the type "function returning T";

if x[] has the type T,  then x has the type "array of T";

if *x  has the type T,  then x has the type "pointer to T";

One important aspect of the declaration rules of C is that each declarator-symbol (* [] ()) has a "precedence" that is strictly analogous to the precedence of the same symbol used as an operator. In other words, the parentheses and square brackets "bind tighter" in declarations than the asterisk does. As these examples get more complicated, we expand our interpretation of the "sandwich rule" to consider the type of expressions, not just names, like this:

char *a[N];  declaration

a[i]   expression

char *       type

Since the square-brackets binds more tightly than the asterisk, the sandwich rule shows that a[i] is a char *. Or, in other words, a is an array of char *'s. Here is the contrasting case:

char (*pa)[N];  declaration

(*pa)       expression

char      [N]   type

Here the parentheses cause the declarator *pa to be taken as a unit; the declaration says that *pa is an array of N char's. That is, pa is a pointer to an array of N char's.

Question [4-1] Just to be sure that you understand the "sandwich rule," jot down the variable name and type for the following declarations.

      DECLARATION     VARIABLE NAME  TYPE ("DECLARED")

      short **ap[5];  _____________  __________________

      long (*pf)[2];  _____________  __________________

      double **pps;   _____________  __________________

Having discussed the declaration of pointers, let us turn to some of the operators associated with them. Taking first the "address-of" operator (unary &), we note that it can be applied only to things that have an address, namely lvalues that are not declared with register storage class. (Bit-fields, which we have not yet discussed, are also not addressable.) If c is a char variable, then the expression &c evaluates to the address of c. The type of the expression &c is char * (pointer to char). To be more general, if an lvalue x has type T, then the expression &x has the type T *.

Thus, &c is a pointer expression, producing a value that is suitable for assignment to a pointer of type char *, such as pc.

Let us turn back to initializations for a moment. In any initialized declaration, the variable name designates the object that is being initialized. Thus in the declaration

char *pc = &c;

what gets initialized is the variable pc, not the declarator *pc. We could emphasize the point by writing the declaration as

char* pc = &c;

which is fine as far as the compiler is concerned, but rather nonstandard for human readers. The types are correct for the initialization; the name pc and the expression &c have the same type, char *. It would, however, be a type mismatch (and a serious bug) to write

char *pc = c;

The type of c is just char, and a char expression should not be assigned to a char * variable.

For a more complicated example, consider

char c = '1';   /* ASCII value is 49 decimal */

char *pc = &c;

char **ppc = &pc;

Here the pointer pc is being initialized to point to the character named c. And ppc is being initialized to point to the pointer named pc. A picture may help; hypothetical address values are being used for clarity:

What is the type of the expression &pc? The type is char ** (pointer to pointer to char), by the rules above. And this is an appropriate type to assign to ppc, because its type also is char **.

Now we will consider the indirection operator (unary *). This is, of course, the same character (asterisk) which means "pointer to" in a declaration. It has a different, but very compatible, meaning when used as "indirection" in an expression. Using the same example above, the expression *pc means "the thing that pc points to," or "indirect pc," for brevity. Assuming the initialization given above, *pc is an lvalue which could be used interchangeably with the variable c.

If an expression x has the type T *, then the expression *x has the type T. (Adding an asterisk to an expression removes an asterisk from its type.) Therefore, the type of *pc is simply char; *pc is a char lvalue. We can properly assign a char value to *pc, just as we could assign *pc to a char variable.

Similarly, *ppc has the type char *, and (with the initialization above) could be used interchangeably with the variable pc.

One reason why agreement of pointer types is so important is that in several implementations of C, pointers to different types have different internal representations, sometimes even different sizes. In these environments, it is especially important to be sure that all uses of pointer types follow the agreement rules strictly.

Sometimes we have a need to print the numeric value of a pointer itself, e.g. for debugging purposes. ANSI C envisions a new printf format for printing pointer values [4-2]. In the meantime, we suggest a macro PR_PTR(p) which will print pointer p in a fashion which will work in almost all current implementations.

#define PR_PTR(p) (printf("X%10ld", (long)(p))

If we were to use this macro to print the two pointers declared above, we could do it like this:

PR_PTR(pc);

PR_PTR (ppc);

printf("\n");

And the output would look like this:

104      112

4.2 Pointers to Scalars

Consider once again

char c = '1';

char *pc = &c;

char **ppc = &pc;

Assuming the same addresses as before, the memory looks like this after the initialization:

We emphasized earlier that it is vital to be clear about the type of pointer variables and expressions. It is also important to be clear about their values.

In high-level languages like C, determination of the value of an expression depends upon whether the expression is an lvalue (something with storage). The value of an lvalue is found by going to the storage and loading the value that is found there. Thus in the example above, the value of c (assuming ASCII) is 49, the numeric value of the constant '1'. Similarly, the value of pc is the value that is found in the storage for pc, namely 104. And the value of ppc is currently l12.

On the other hand, an rvalue (an expression that is not an lvalue) simply is a value: the value of '1' is 49, assuming ASCII. This concept of value is regular and straightforward. It is the the concept that is followed in most high-level languages, including Basic, Fortran, and Cobol. It is, however, different from the behavior of assemblers, in which the value of a symbol is its address.

Consider the expression &x, where x is some lvalue expression. The expression &x is an rvalue: it has a value, but no storage for itself. Thus the value of &pc is (in this example) l 12, the address of pc. The value of &c is 104.

If x is an expression of type "pointer to something," the expression *x is an lvalue, and its value is found in the storage that x is pointing to. Thus, *pc has the value 49 (assuming ASCII).

Question [4-3] Not all of the following "expressions" are valid. For each valid expression, write its value. Write "INVALID" otherwise.

&c __________    c __________    *c __________

&pc __________   pc __________   *pc __________

&ppc __________  ppc __________  *ppc __________

Question [4-4] The function call scanf( "%c", &c) will read one character into c. What is the value being passed as the second argument to scanf?

______

Which of these expressions would accomplish the same result?

_____ scanf("%c", c)

_____ scanf( "%c", pc)

_____ scanf("%c", &pc)

_____ scanf("%c", *pc)

_____ scanf("%c ", *ppc)

In the Question above, the scanf function was told where to store something by being passed a pointer value (i.e. an expression of type "pointer to char"). We now look at short_order, a function which also expects pointer values for arguments:

short_order:

/* short_order - put two short integers into order

*/

# include "local.h"

void short_order(pi, pj)

short *pi, *pj;

{

short t;

if (*pj < *pi)

t = *pi, *pi = *pj, *pj = t;

}

If the main program contained these declarations

short i1 = 1111;

short i2 = 22;

short *pi1 = &i1;

short *pi2 = &i2;

we could call short_order in several ways:

short_order(&i1, &i2);       short_order(pi1, &i2);

short_order(&i1, pi2);       short_order(pi1, pi2);

All of these would accomplish the same result.

But short_order(i1, i2) is incorrect, because the values passed to short_order must be pointer values (specifically addresses of short integers).

C is indeed a "call by value" language, but the arguments may be pointer values which tell the function where something is located.

Having discussed the types and values of simple pointers, we now turn to their semantic properties. A NULL pointer is one which contains the unique NULL value. A pointer becomes NULL when the integer zero is assigned to it. A NULL pointer always compares equal to integer zero. An undefined pointer is one that is not NULL, and yet is not pointing to an object of a compatible type (e.g., "garbage," uninitialized). Thus, a defined pointer is either pointing to valid storage, or else is NULL. And when we say that a pointer is non-NULL, we imply that it is defined. So a non-NULL pointer is one that points to valid storage. We can illustrate the importance of these concepts by considering various errors in calling short_order.

Undefined pointers: Suppose we invoked short_order like this:

short *ps1, *ps2;

short_order(ps1, ps2);

Then, without initializations or assignments, the values of ps1 and ps2 are undefined (uninitialized, "garbage"). We have, in effect, asked short_order to rearrange the contents of two random memory locations!

 pointers: Suppose instead that we invoked short_order like this:

short *ps1 = NULL;

short *ps2 = NULL;

short_order(ps1, ps2);

In the short_order function, the local variables pi and pj will receive the value NULL from the calling function. We then encounter a problem evaluating *pi and *pj. The problem is that NULL is not a proper value for indirection, because a NULL pointer does not point to any valid object. A strict checking environment could complain; most will attempt to access memory location zero, producing unpredictable results. So NULL pointers are also no good for the arguments to short_order.

The pointer parameters of short_order are used both to access already-existing values and also to modify the storage that is pointed to. We can refer to such pointer parameters as in-out-pointer parameters. (An in-pointer would be used only to access existing values, but not to modify anything. An out-pointer would be used only to modify something, but not to examine any existing value.) Let us define an incomplete pointer as a non-NULL pointer whose target contains one or more undefined scalars. In other words, an incomplete pointer is being used to access an object which is not totally defined. A complete pointer, then, is one whose target is totally defined. It is all right for an outpointer to be incomplete, since the called function does not care what is already in the pointer's target. The safest default assumption regarding in-pointers and in-out-pointers is that they must be complete, since they must access the values that are in their target. In the short_order example, both pointers must be complete in order for the function call to be reliable.

It is an important part of the definition and documentation of a function to indicate what it expects of its pointer arguments.

Rule 4-2: The default requirement for pointer parameters is that they must point to storage that is entirely defined. Whenever a pointer parameter can accept something else, this should be explicitly stated on that parameter's declaration comment.

4.3 Dangling Pointers

There are further requirements for reliable programming with pointers. A pointer can become undefined if the object that it is pointing to should "disappear" during the lifetime of the pointer. Here is a simple example.

dangling.c:

/* dangling - example of dangling pointer

*/

# include "local.h"

static short *pi = NULL;

main( )

{

void f1( );

/* pi => NULL initially */

f1( );       /* pi => undefined suddenly! */

}

void f1( )

{

short i;

pi = &i;     /* pi => complete, momentarily */

}

The event which causes the pointer pi to become "undefined" is the return from function f1. The specific term for a pointer in this undefined state is a dangling pointer; the pointer points to storage no longer allocated. It is the programmer's responsibility to be sure that the program contains no instances of dangling pointers. (There are, however, techniques by which a strict-checking environment can detect dangling pointers [4-5].)

Rule 4-3: A function in which the address of an automatic variable is assigned to a non-automatic pointer must contain a comment to that effect. In any function with such a comment, each return from the function is an event requiring verification that no dangling pointers are left.

Further consideration of dangling pointers with respect to the free function will be found in Section 7.1.

4.4 Array Pointers and Strings

Remember that an "array pointer" (i.e., "pointer into array") is one used to access an array, not a "pointer to array." When declared as a parameter, an array pointer can be declared either with an asterisk or with empty brackets: char *s and char s[] are equivalent from the compiler's point of view. To make distinctions for the human reader, however, we can use char s[] to indicate an array pointer as opposed to a pointer to scalar. (However, a non-parameter pointer must be declared with the asterisk.)

Several languages provide string variables, and the compiler automatically keeps track of their current length. C does not. Loosely speaking, a string in C is a nul-terminated array of chars. More strictly, string is a particular property for a character array or character array pointer, namely the property of being nul-terminated. And a string constant, of course, always has the property string. Henceforth, when we use the noun "string," we are referring to a character array pointer, a character array, or a string constant, each with the property string.

There are advantages and disadvantages from this scheme in C. The main disadvantage is that more responsibility lies with the programmer for correct handling of strings. One advantage is that the compiler and the generated code are both simpler than in languages with built-in strings. Also, any appropriate region of memory can be treated as a string (which is useful in low-level programming).

Let us look at the string functions in the standard library, with an eye to the reliability requirements of each function.

size_t strlen(s)

char s[];              /* : string */

The parameter s must be a string. If it is not, most implementations of strlen will wander through the memory looking for a nul terminator. The "length" of a string, as always in C, is the number of characters prior to the nul terminator; thus,

strlen("hello")

returns the value 5.

int strcmp(s1, s2)

char s1[];             /* : string */

char s2[];             /* : string */

The parameters s1 and s2 must both be strings. The result is less than zero, zero, or greater than zero according to whether string s1 compares less than, equal to, or greater than string s2. Thus

strcmp("ab", "aa")

returns a positive number.

The result cannot be guaranteed to be portable if either string contains a character that is not in the machine's native character set, since the comparison may be a signed comparison on some machines. For example,

strcmp("\377", "a")

will yield a positive result if characters are signed, and a negative result if characters are not signed. Even for characters in the native character set, the collating sequence may be different on different machines.

char *strcpy(s1, s2)   /* at return, s1 => string */

char s1[];             /* : !NULL */

char s2[];             /* : string */

The s2 parameter must be a string. The s1 parameter must specify some storage into which the string can be copied, so it must be non-NULL. In general, strcpy should be used only when the program makes certain that the string length of s2 is less than the size of the array being referenced by s1. Otherwise, an out-of-bounds assignment will result.

The function strcpy returns the (original) value of s1, the address where the characters have been stored.

char *strcat(s1, s2)    /* at return, s1 => string */

char s1[];              /* : string */

char s2[];              /* : string */

The parameter s2 must be a string. The parameter s1 must be a string and its target (the storage that it accesses) must be larger than the sum of the string lengths of both parameters.

The parameter s1 is returned (the address of the new catenated string).

char *strchr(s, c)  /* returned value => string|NULL */

char s[];           /* : string */

int c;              /* : {CHAR_MIN:CHAR_MAX} */

Invoking

strchr("abc", 'b')

will return a pointer to the second character in the string. If there is no occurrence of character c in string s, strchr returns a NULL pointer. Thus, the returned value is either a string (pointer) or a NULL pointer.

char *strrchr(s, c) /* returned value => string|NULL */

char s[];           /* : string */

int c;              /* : {CHAR_MIN:CHAR_MAX} */

The behavior is the same as for strchr, except that the a pointer to the last occurrence (or ) is returned. Thus,

strrchr("aba", 'a')

will return a pointer to the third character of the string.

char *strpbrk(s1, s2)  /* returned value => string|NULL */

char s1[];             /* : string */

char s2[];             /* : string */

Parameters s1 and s2 must both be strings. The function returns a pointer to the first occurrence in s1 of any character in s2. Thus

strpbrk("abc123", "0123456789")

will return a pointer to the fourth character of the first string. If no characters from s2 are in s1, strpbrk returns a NULL pointer.

size_t strspn(s1, s2)

char s1[];              /* : string */

char s2[];              /* : string */

Both parameters must be strings. The function returns the length of the initial segment of string s1 which consists entirely of characters from string s2. Thus

strspn("\t\tabc", " \t\n")

will return the number 2.

size_t strcspn(s1, s2)

char s1[];              /* : string */

char s2[];              /* : string */

Both parameters must be strings. The function returns the length of the initial segment of string s1 which consists entirely of characters not from s2. Thus

strcspn("abc\t", " \t\n")

will return the number 3.

char *strtok(s1, s2)   /* returned value => string|NULL */

char s1[];             /* : string */

char s2[];             /* : string */

This function splits string s1 into "tokens," more specifically, strings of characters delimited by occurrences of the characters in string s2. Calling strtok with a non-NULL argument will produce a pointer to the first token. Calling strtok with a NULL first argument will cause it to point to the next token from s1. Each token receives a nul terminator. Here is an example:

static char buf[] = " abc 123\t\txyz\n";

static char seps[] = " \t\n";

char *p;

printf("first token=<%s>\n", strtok(buf, seps));

while ((p = strtok(NULL, seps)) != NULL)

printf("next token=<%s>\n", p);

Three output lines will be printed:

first token=<abc>

next token=<123>

next token=<xyz>

The returned value is a string pointer, or else NULL.

One uses string comparisons so often in C that macros are sometimes useful. The local.h header defines three:

#define STREQ(s, t) (strcmp(s, t) == 0)

#define STRLT(s, t) (strcmp(s, t) < 0)

#define STRGT(s, t) (strcmp(s, t) > 0)

Besides being easier to type, the macros allow the possibility of various optimizations of the comparison. For example, in many classes of problems, strings usually differ in their first character. Each macro could be revised, to optimize speed at the expense of code space, something like this:

#define STREQ(s, t) ((*s) == (*t) && strcmp(s, t) == 0)

An array can be treated with semantics other than "string," if that is more convenient. For example, an alternative property for an array of char's is a property known as nul-padded: either the array is completely filled with (defined) non-nul characters, or else all characters after the first nul are also nul.

Similarly, one can maintain a blank-padded array: the array is completely filled with (defined) non-nul characters, of which the final sequence of blanks is considered "padding." Blank-padded arrays are often used in data-processing contexts where C is interfaced with other business-oriented software.

The string library contains the functions strncat, strncmp, and strncpy, which involve yet another array property. This one lacks a conventional name, but we can call it the strn property: An array with this property is either complete up to a nul-terminator, or complete up to the end of the array. In other words, if there is space for a nul-terminator, then one is present. Otherwise, the entire array is filled with (defined) non-nul characters. Here are the descriptions of the strn functions:

char *strncat(s1, s2, n)   /* at return, s1 => string */

char s1[];                 /* : string */

char s2[];                 /* s2[0:n-1] : strn */

size_t n;                  /* : {0:DIM(s2[*])} */

strncat copies at most n characters of the array s2 onto the end of string s1. Fewer than n characters are copied if a nul terminator is encountered first in s2. The resulting contents of s1 are always nul terminated, and the address of the resulting string is returned.

int strncmp(s1, s2, n)

char s1[];                 /* s1[0:n-1] : strn */

char s2[];                 /* s2[0:n-1] : strn */

size_t n;                  /* : {0:DIM(s2[*])} */

The arrays s1 and s2 are compared. The comparison stops when corresponding positions differ, or a nul terminator is reached, or all n characters of both arrays are equal. The returned value is less than zero, zero, or greater than zero according to the sense of the comparison.

char *strncpy(s1, s2, n)   /* at return, s1 => nul-padded */

char s1[];                 /*: !NULL, DIM(s1[*]) >= n */

char s2[];                 /*: strn */

size_t n;                  /*: {0:DIM(s2[*])} */

The contents of array s2 are copied into array s1. At most n bytes are copied. If the first n bytes of s2 include a nul terminator, the remainder of s1[0:n-1] is nul-padded. Otherwise, s1[0:n-1] is complete but not a string (no nul terminator).

Recent versions of C also include some new functions called the "memory" functions. These operate upon arrays of bytes with no regard for any special meaning of nul. In some environments this allows a more efficient implementation, since some machines have "block move" and "block compare" instructions.

These functions make use of a new concept in recent C, the generic pointer. This is a pointer which is much like a char * in that it can reference any particular byte, but its type allows it to be a "pointer to any data." Thus, you can assign any kind of data pointer to a generic pointer, or assign a generic pointer to any kind of data pointer. No pointer operations are allowed upon generic pointers, however. They are mostly useful for passing pointers to functions that can appropriately accept "pointer to any data." Inside the function, it is necessary to copy the generic pointer into some other pointer before doing pointer operations. In (draft) ANSI C, the generic pointer is a void *, but there are good reasons to use a defined type name instead. We will use the name data_ptr for the generic pointer.

One important reason to use a defined name such as data_ptr is that non-ANSI compilers will not know what a void * is. In these environments, we will need to define data_ptr as char *.

For maximum portability in non-ANSI environments, we should also cast each assignment (or argument passing) when going to or from generic pointers. This, of course, is unfortunate, since the original reason for generic pointers was to avoid so many nuisance casts. (If your compiler does not have lint-like checking of function arguments versus parameters, and it provides the same representation form for all varieties of pointers, you could safely eliminate the casts.) As long as we are still putting casts on function arguments, they might as well be (char *) casts, since this will work fine in both old and new environments.

Here, then, is the description of the memcpy function, using generic pointer notation. (In order to refer to the contents of the target of a generic pointer, we will refer to it as if it pointed to an array of char'S.)

data_ptr memcpy(s1, s2, n)

data_ptr s1;            /* : !NULL */

data_ptr s2;            /* : !NULL */

size_t n;               /* : {0:DIM(s1[*])-1 */

The array s2[0:n-1] is copied into s1[0:n-1]. The pointer s1 is returned.

All the "memory" functions are easy to implement in portable C, if your implementation does not already have them in the library. Here, for example, is a version of the memcpy function:

memcpy:

/* memcpy - copy n bytes from b to a */

#include "local.h"

data_ptr memcpy(s1, s2, n)

data_ptr s1;

data_ptr s2;

register size_t n;

{

register char *a = s1;

register char *b = s2;

for (; n > 0; --n, ++a, ++b)

*a = *b;

return (s1);

}

For an example using memcpy, suppose that your compiler does not have structure assignments. You can efficiently copy one structure into another like this:

struct xxx a;

struct xxx b;

memcpy((char *)&a, (char *)&b, sizeof(a));

An even handier way to handle structure assignments is to define a macro:

#define STRUCTASST(a, b) memcpy((char *)&(a), (char *)&(b), sizeof(a))

This way, if your compiler does support structure assignments, you can change the definition like this:

#define STRUCTASST(a, b) ((a) = (b))

The macro STRUCTASST is defined in the local.h file described in Chapter 1.

Here are the definitions of the other "memory" functions:

data_ptr memchr(s, c, n)

data_ptr s;             /* : ! NULL */

int c;                  /* : {CHAR_MIN:CHAR_MAX} */

size_t n;               /* : {0:DIM(s[*])-1} */

The array s[0:n-1] is searched for a match to the byte specified by c. If one is found, a pointer to the matching byte is returned; otherwise, NULL is returned.

int memcmp(s1, s2, n)

data_ptr s1;            /* : ! NULL */

data_ptr s2;            /* : ! NULL */

size_t n;               /* : {0:DIM(s1 [*] )-1 */

The arrays s1[0:n-1] and s2[0:n-1] are compared byte-by-byte. If they are identical, zero is returned. Otherwise, a number less than zero or greater than zero is returned, according to the sense of the comparison.

data_ptr memset(s, c, n)

data_ptr s;             /* : ! NULL */

int c;                  /* : {CHAR_MIN:CHAR_MAX} */

size_t n;               /* : {0:DIM(s[*])-1 */

All the bytes of s[0:n-1] are filled with the byte value specified by c. The pointer s is returned.

4.5 Changing Subscripts into Pointers

One often has the problem of converting an algorithm that uses subscripts into one that uses pointers (usually for greater efficiency). This works best when each subscript is used sequentially (with increments or decrements). Each subscripted reference like a[i] becomes a pointer reference like *p_a. Each increment or decrement of a subscript becomes an increment or decrement of a pointer instead. Instead of comparing subscripts, we will compare pointers.

If, for example, we converted the qksort function of Section 3.2 to use pointer references instead of subscript references, it would look something like this:

qksortp:

/* qksort - sort array a (dimension n) using quicksort */

#include "local.h"

/* iqksort - internal function to partition the array [*lo:*hi] */

static void iqksort(p_lo, p_hi)

short *p_lo, *p_hi;

{

short *p_mid = p_lo + (p_hi - p_lo) / 2;    /* : {p_lo:p_hi} */

short *p_i;                                 /* : {p_lo+1:p_hi+1} */

short *p_lastlo;                            /* : {p_lo:p_hi-1} */

short tmp;

SWAP(*p_lo, *p_mid, tmp);

p_lastlo = p_lo;

for (p_i = p_lo+1; p_i <= p_hi; ++p_i)

{

if (*p_lo > *p_i)

{

++p lastlo;

SWAP(*p_lastlo, *p_i, tmp);

}

}

SWAP(*p_lo, *p_lastlo, tmp);

if (p_lo < p_lastlo && p_lo < p_lastlo - 1)

iqksort(p_to, p_lastlo - 1);

if (p_lastlo + 1 < p_hi)

iqksort(p_lastlo + 1, p_hi);

}

/* qksort - external entry point */

void qksort(a, n)

short a[];  /* array to be sorted; a[0:n-1]: multiset */

size_t n;   /* number of elements */

{

if (n > 1)

iqksort(a, &a[n-1]);

}

Nothing has changed in the algorithm. All the invariants are the same as before. We have just replaced the subscripts with pointers.

4.6 Multi-dimensional Arrays

In C, a multi-dimensional array is really an array of arrays: if scores is declared via

scores consists of two elements, each of which is an array of three short integers. If, for instance, we were to typedef the name trio to stand for an array of three shorts, then scores could equivalently be declared to be an array of two trios.

typedef short trio[3];

trio scores[2];

As we saw earlier for pointers, there is a complementary relationship between array operations and array declarations. The "sandwich" rule tells us that the type of scores is short [2][3]. The same rule says that the type of the array scores[0] is short [3].

This becomes of particular importance when passing multidimensional arrays to functions, because what actually gets passed to the function is the address of the initial subarray. Consider a function weight which accepts n-by-three arrays of scores, and weights each column by one, two, and three.

long weight(a, n)

short a[][3];

int n;

{

index_t i, j;

long sum = 0;

for (i = 0; i < n; ++i)

for (j = 0; j < DIM(a[0]); ++j)

sum += (j + 1) * a[i][j];

return (sum);

}

The weight function should be passed the address of an array of three-element arrays. By C pointer parameter rules, the parameter declaration

short a[][3];

is understood as

short (*a)[3];

In other words, a is actually a pointer to three-element arrays. (The first form indicates the usage more clearly, but the second form reveals more clearly the nature of the parameter a.) Using the array scores above, we could call the function like this:

weight(scores, 2)

or like this:

weight(&scores[0], 2)

The two forms are equivalent, and the address of the array scores[0] is what gets passed. As discussed in Section 3.4, the range of pointer values that is associated with the argument is {&scores[0][0]:&scores[2][0]}, that is, all the elements of scores, plus the "one-too-far" address.

Note that in the called function, the compiler must be told the bounds of all dimensions except the major dimension. If the function needs to know the major dimension, it must be passed explicitly, as in weight above. Just as with single-dimensional arrays, there is no way to induce C to reveal the major dimension to the called function. It is certainly a mistake to use sizeof(a) to infer the dimension; sizeof(a) is only the number of bytes in a pointer, and has no relation to the size of the actual array.

4.7 Arrays of Pointers

An array of pointers is a compact way to store a table of strings. In Section 3.5 we saw a table that looked like this when stored in a two-dimensional array:

static char meas_codes[][6] =

{"each", "box", "lb"};

We could store the same information using an array of string pointers:

static char *meas_codes[] =

{"each", "box", "lb"};

The ("declared") type of meas_codes[i] is char *, i.e., pointer to char. Each pointer meas_codes[i] is thus a proper argument to strcmp in the same loop that we used before:

for (i = 0; i < DlM(meas_codes); ++i)

if (strcmp(s, meas_codes[i]) == 0) or if (STREQ(s, meas_codes[i]))

found a match

A very simple lookup operation like this one sometimes has its practical uses. Recent C compilers provide a function called getenv ("get environment") which searches in a system-specific environment for a given name and returns a pointer to the definition associated with that name. One of the most common uses is as a way for a program to determine what type of terminal is being used:

char *termtype;

termtype = getenv("TERM");

will give us the answer, if the information is available. And if we were porting a program to a machine with no operating system, and hence no "environment," the getenv function would be one of the easiest ones to write, especially if all the "environment" definitions were constant:

getenv(dummy):

/* getenv - get "environment" definition (dummy standal one version) */

# include "local.h"

char *getenv(s)

char s[];

{

extern char *env_names[];  /* : array of string ptrs, last one NULL */

extern char *env_defs[];   /* : array of string ptrs */

int i;

for (i = 0; env_names[i] != NULL; ++i)

if (STREQ(s, env_names[i]))

return (env_defs[i]);

return (NULL);

}

The function assumes that the array env_names will contain a NULL pointer as its last entry. This is a very common technique, since it eliminates the need for a separate variable to communicate the dimension of the array.

4.8 Command Line Arguments

Topics assumed from previous book: command line arguments [7.8].

The arguments to the main function are passed as an array of string pointers, along with a count which tells the number of non-NULL pointers in the array. Both the count and the array pointer are parameters of main, so their names can be whatever you like. The count is conventionally named argc or ac, and the array pointer is conventionally argv or av.

As with any array, the array accessed through av can be traversed either with a subscript or with a pointer. However, most programmers seem to make fewer mistakes when they use a subscript, as in the following function:

args.c:

/* args - show command-line arguments, one line each

*/

# include "local.h"

main(ac, av)

int ac;

char *av[];

{

int i;

for (i = 0; i < ac; ++i)

printf("av[%d]=<%s>\n", i, av[i]);

}

If we were to invoke args from the command line like this

args  one  two  "quoted string for three"

most systems would pass the quoted string as one argument, giving this printout:

av[0]=<args>

av[1]=<one>

av[2]=<two>

av[3]=<quoted string for three>

4.9 Pointers to Functions

In most implementations, each function has an address, just as data objects do. Thus a pointer-to-function (or "function pointer") is typically an address-sized variable which can contain the address of a function. Declaring that pfn points to a function is equivalent to declaring that (*pfn) (the thing that pfn points to) is a function:

double (*pfn)();

says that (*pfn) is a function returning double, or that pfn is a pointer to a function returning double. The parentheses around *pfn are needed; the declaration

double *fnpd();

says that fnpd returns a pointer to double.

No operators are defined for function pointers, except for assignment, comparison, and (of course) indirection. And the only things that can be assigned to function pointers are the addresses of functions having the appropriate type:

extern int strcmp();

int (*cmpfn)() = &strcmp;

The address-of operator can be omitted; indeed, some current compilers prohibit it. We can write

int (*cmpfn)() = strcmp;

This is the more familiar style, once you understand that it is the address of strcmp that is being assigned to cmpfn.

The only useful thing to do with a function pointer is to call the function that it is pointing to. To call (*cmpfn) (the function that cmpfn is pointing to), with arguments s1 and s2, we write

(*cmpfn)(s1, s2)

Function pointers are useful for dispatch tables, i.e. tables of different functions to call. Instead of implementing a dispatch table like this

switch (code)   /* layout to highlight structure */

{

case 0: initdb();   break;

case 1: opendb();   break;

case 2: readdb();   break;

...

case 7: closdb();   break;

}

we could put the addresses of all the functions into an array:

typedef void (*pvoidfn)(); /* pvoidfn is ptr to fn ret void */

static pvoidfn dispatch[] = {initdb, opendb, readdb, ..., closdb};

...

if (0 <= code && code < DlM(dispatch))

(*dispatch[code])();

else

process invalid code

In other words, dispatch[code] is a function pointer (of the defined-type pvoidfn), so *dispatch[code] is the function that it points to.

The typedef for pvoidfn ("pointer to function returning void'') makes the code more readable and less error-prone.

Another common use for function pointers is to provide a function as a parameter of another function. For example, the short-integer version of qksort could be made into a general-purpose sort function by passing a pointer to the comparison function, rather than programming the comparison operation into the function as we did in qksort. Thus, we will add a cmpfn parameter. The comparison function will be called with the addresses of two array elements, and it must return a value less than, equal to, or greater than zero, according to whether the first array element sorts before, equal to, or later than the second array element. (If the array elements are strings, the strcmp function would behave exactly in the right way.)

We will also need a new parameter to tell the size in bytes of the elements in the array, because the general-purpose version will not know the specific data type of the array elements. Accordingly, the interface will now look like this:

void qsort(a, n, size, cmpfn)

data_ptr a;

size_t n;

size_t size;

int (*cmpfn)();

Where earlier we had to re-compile a new sort function for each different array type, now one function will work for all array types. To sort an array of strings, call

qsort((char *)a, DIM(a), sizeof(a[0]), strcmp);

That is, we pass the starting address a, the number of array elements DIM(a), the size of each element sizeof(a[0]), and the function to use for comparisons strcmp. Again, what gets passed to qsort is the address of the strcmp function.

Or to go back to the sorting of an array of shorts, we would need a little function to accomplish the short comparison.

short shortcmp(pi, pj)

short *pi;

short *pj;

{

if (*pi > *pj)

return (1);

else if (*pi == *pj)

return (0);

else

return (-1);

}

It is tempting to use a subtraction

return (*pi - *pj)

as a quick way to produce the three-way (positive, zero, negative) result, but this would be prone to overflow, as we discussed in Section 2.3. Then to sort an array of shorts, we would invoke

qsort((char *)a, DIM(a), sizeof(a[0]), shortcmp);

Because of this ability to parameterize the sort algorithm into a single function, it can become part of a general-purpose function library. (The specification, in fact, is taken directly from the UNIX library manuals.)

Thus, in almost all respects, we would consider the callable general-purpose library function (suitably parameterized) to be the more reliable implementation of the algorithm -- there are fewer degrees of freedom for making mistakes.

Here is the C source for the qsort function:

qsort:

/* qsort - sort array a (dimension n) using quicksort

* based on Bentley, CACM April 84

* Comments use notation A[i], a (fictitious) array of things

* that are of size elt_size.

*/

#include "local.h"

/* swapfn - swap elt_size bytes a <--> b (internal routine)

*/

static void swapfn(a, b, elt_size)

register char *a;   /* pointer to one element of A */

register char *b;   /* pointer to another element of A */

size_t elt_size;    /* size of each element */

{

register size_t i;

char tmp;

for (i = 0 i < elt_size; ++i, ++a, ++b)

SWAP(*a, *b, tmp);

}

/* iqsort - internal quicksort routine */

static void iqsort(p_lo, p_hi, elt_size, cmpfn)

char *p_lo;         /* ptr to low element of (sub)array */

char *p_hi;         /* ptr to high element of (sub)array */

size_t elt_size;    /* size of each element */

int (*cmpfn)();     /* comparison function ptr */

{

char *p_mid = p_lo + ((((p_hi - p_lo) / elt_size) / 2) * elt_size);

register char *p_i;         /* pointer to A[i] */

register char *p_lastlo;    /* pointer to A[lastlo] */

swapfn(p_lo, p_mid, elt_size);  /* pick the middle element as pivot */

p_lastlo = p_lo;

for (p_i = p_lo + elt_size; p_i <= p_hi; p_i += elt_size)

{

if ((*cmpfn)(p_lo, p_i) > 0)

{

p_lastlo += elt_size;

swapfn(p_lastlo, p_i, elt_size);

}

}

swapfn(p_lo, p_lastlo, elt_size);

if (p_lo < p_lastlo && p_lo < p lastlo - elt_size)

iqsort(p_lo, p-lastlo - elt_size, elt_size, cmpfn); /* lower */

if (p_lastlo + elt_size < p_hi)

iqsort(p_lastlo + elt_size, p_hi, elt_size, cmpfn); /* upper */

}

/* qsort - the callable entry point */

void qsort(a, n, size, pf)

data_ptr a;     /* address of array A to be sorted */

size_t n;       /* number of elements in A */

size_t size;    /* size of each element */

int (*pf)();    /* comparison function ptr */

{

if (n > 1)

iqsort((char *)a, (char *)a + (n-1) * size, size, pf);

}

We started in Section 3.2 with the qksort function, which accepted an array of short integers and sorted it with subscripts. Then in Section 4.4 we transformed it into qksortp, which still worked upon short integers but used pointers. Finally, we transformed qksortp into qsort, shown just above. The next section will describe these transformations.

4.10 Pointer Transformations

The transformations that took qksort into qsort are typical of the messiest aspects of programming with pointers. It will be helpful to consider the transformations step-by-step.

First transformation: an ordinary array reference (such as a[lastlo])is to be performed using a pointer. A pointer p_lastlo is created which will point to a[lastlo]. Then each reference to a[lastlo] becomes a reference to *p_lastlo. Finally, if any storage was dynamically allocated, it must be freed.

Second transformation: Pointers to known type are generalized into pointers to varying-size storage. Here, the rules are much trickier. The type (and therefore the size) that a pointer points to is fixed at compiletime; there is no direct way to obtain a pointer to varying-size storage. The best we can do in C is to use a "byte" pointer (char *) and perform any scaling directly. (In the qsort example, the pointers are all scaled by elt_size, the size of each element in the argument array.) Thus the declaration (from qksortp)

short *p_lastlo;

becomes (in qsort)

char *p_lastlo;

And ++p_lastlo becomes p_lastlo += elt_size.

The difference between two pointers must be divided by the pointed-to size: (p_hi - p_lo) becomes (p_hi - p_lo) / elt_size. The difference between two pointers is an integer result, so multiplying and dividing both take place in transforming

short *p_mid;

p_mid = p_lo + (p_hi - p_lo) / 2;

into

char *p_mid;

p_mid = p_lo + ((p_hi - p_lo) / elt_size) / 2) * elt_size;

It is tempting to cut corners on the apparently redundant multiply-and-divide, but what gets added to p_lo must be an multiple of elt_size.

Third transformation: No operations can take place directly upon the pointed-to values; functions must be called for all operations. Thus

if (*p_lo > *p_i)

must be rewritten as

if ((*cmpfn)(p_lo, p_i) > 0)

Swapping two pointed-to values becomes too complicated for our simple SWAP macro; a new function (swapfn) is now used.

To summarize these transformation rules in a table:

Original                   Transformed

(p_hi - p_lo)             (p_hi - p_lo) / elt_size

p_lo + expr                p_lo + expr * elt_size

++p_lastlo                 p_lastlo += elt_size

SWAP(*p_lo, *p_mid, tmp)   swapfn(p_lo, p_mid, elt_size)

Such transformations must be done carefully, but it is testimony to the expressive power of C that they are possible at all. Certainly, in many languages it is difficult, if not impossible, to write a function which will accept an array of any dimension, composed of elements of any size, and then sort the elements according to an arbitrary ordering function. The C implementation of qsort does all this, in strictly portable code.

Go to Chapter 5    Return to Table of Contents


Reliable Data Structures: CHAPTER 5: STANDARD FILE AND TERMINAL I/O

 Return to Table of Contents

CHAPTER 5: STANDARD FILE AND TERMINAL I/O

In this chapter we examine the facilities for I/O on the "standard files": standard input, standard output, and the standard error file. Among these facilities are also the functions sprintf and sscanf for formatted operations to and from strings. Finally, we will look at methods for performing I/O with terminal screens and keyboards.

5.1 Formatted Output

5.2 Standard Files

5.3 Reading Lines of Input

5.4 Formatted Input

5.5 Direct Terminal I/O

5.1 Formatted Output

Topics assumed from previous book: Basic printf formats [2.8], file redirection [3.22].

You are presumably familiar with the basic formats of printf: %c (character), %s (string), %d (decimal integer), %u (decimal unsigned integer), %o (octal integer), %x (hexadecimal integer), %f ("fixed-point" representation of floating-point number), and %e ("E-notation" floating-point number). The modifier l (for "long") can precede d, u, o, or x, to indicate a long int argument.

Format specifiers are actually somewhat more complicated than the simple examples above; the full syntax looks like this:

% flag width .precision conversion

In the simpler forms of printf (such as in K&R), the flag may consist of a minus sign, indicating left-adjustment of the output. The width is a string of digits indicating the minimum output width. (In K&R and in UNIX Version 7, a width beginning with a zero digit indicated that extra output places were to be filled with zeros, not blanks. This feature disappeared in later UNIX manuals, but not always from the library code itself!) The precision (for floating-point formats) is the number of decimal places; for strings, it is the maximum number of characters to print. And the conversion characters were covered in the previous paragraph.

One other format completes the capabilities of basic printf: the g format. When g is specified, printf tries the output both in e and f for mats and prints whichever takes less space. This can be useful for outputs of widely differing magnitude.

Over the years, other features have been found useful and incorporated in newer libraries. New flags are provided: A plus sign (+) produces either a plus (+) or a minus (-) sign on the printed output. A blank produces either a blank or a minus sign. And a # flag produces an "alternate form." For example, %#o causes the (octal) printed result to have a high-order zero, and %#x causes the (hex) printed result to begin with 0x.

One innovation most useful for reliability is that the width and/or precision can be replaced by an asterisk, in which case the appropriate number will be taken from the argument list. For example,

printf("%*.*f\n", fld_width, dec_places, x);

will print the (floating) number x with dec_places number of decimal places and fld_width total field width. Previously, the hard-coded numbers appearing in format strings were likely to be missed when modifying programs with new output parameters.

Some cosmetic changes were also provided: the conversion characters X, E, and G cause the appropriate alphabetic characters to appear in upper case, to the delight of former FORTRAN programmers who never adapted to 1.86272E5 being printed as 1.86272e5. (For a detailed comparison of the K&R, /usr/group, and ANSI C libraries, see the appendix [5-1].)

All of the printf formatting is available for formatted output into a string, using the sprintf function. For example, the invocation

sprintf(s, %3s %3s %2d %2d:%02d:%02d %4d",

weekday, mo, day, hh, mm, ss, yr);

could leave s looking like this:

"Fri Nov 11 15:47:04 1982"

Exercise 5-1. Write a function

char *fround(x, s, nsignif, width, precis)

double x;

char s[];

int nsignif;

int width;

int precis;

which will round x to nsignif significant digits and then "print" the result into the string s in %f format with width and precision specified by width and precis. The function should return the (address) value of s. For example,

printf("%s\n", fround(x, buf, 3, 8, 3));

would produce, when given successive x values of 12.345, .6666, and 4853.4836, the output

12.3

.667

4850.

There are some reliability limitations to the printf family of functions. Since numeric output will occupy as many characters as are needed, it is dangerous to use sprintf to write into strings which are not large enough for the largest possible output, nor should we use printf to write into files whose contents are strictly formatted in size. The only way to be certain of the size of numeric output is to bounds-check each number before printing or to allow the absolute maximum output size for each number.

Oftentimes it is more convenient to use a more specialized, but more reliable, function for converting numbers into strings. The itoa function below is one example. This is much less powerful than the printf functions, but it is suitable for converting integers into small strings. It assumes that the stated string size is inviolate, so a string of 9's is the best response to an out-of-bounds argument, with a Boolean return of NO if the full result would not fit. Here is the itoa function:

itoa:

/* itoa - convert integer n to string (ndigits max) */

#include "local.h"

bool itoa(n, str, ndigits)

int n;

char str[];

int ndigits;

{

int sign;

short i;

bool success;

unsigned un = n;    /* need room for most-negative n */

success = YES;

if (ndigits < 1 | | n < 0 && ndigits < 2)

return (NO);         /* can't even get started */

else if (n == 0)

{

strcpy(str, "0");    /* dispose of a special case */

return (YES);

}

else if (n < 0)

{

sign = -1;

un = -n;

--ndigits;           /* sign consumes one print digit */

}

else

sign = 1;

for (i = 0; i < ndigits && un != 0; ++i)

{

str[i] = `0' + un % 10;

un /= 10;

}

if (un != 0)/            /* still more digits to print, and no room */

{

for (i = 0; i < ndigits; ++i)

str[i] = `9';

success = NO;

}

if (sign < 0)

str[i++] = '-';

str[i] = `\0';

reverse(str);

return (success);

}

5.2 Standard Files

Before the main function is invoked, three standard files are opened, known as the standard input, the standard output, and the standard error files. The standard input is used by getchar and scanf; the standard output is used by printf and putchar. All the information needed to keep track of each file is kept in a struct declared in <stdio.h> with the defined-type FILE [5-2]. Also in <stdio.h> are definitions of three symbols -- stdin, stdout, and stderr -- each of which is defined to be the address of a FILE structure. Thus, stdin has the type FILE *, or "pointer to FILE," as do stdout and stderr.

Most hosted environments for C provide a method for redirecting the standard input and the standard output on the command line:

pgm <filel >file2

will associate stdin with file1, and stdout with file2. This redirection is accomplished by the command language interpreter (e.g., "shell" or COMMAND.COM) on UNIX systems, MS-DOS (since Version 2.0), and many other systems. Other environments provide special startup code in the program itself. In either event, the redirection is accomplished before the main function is called.

Files that are accessed through the FILE mechanism provided by <stdio.h> are known as stream files, or (less officially) as buffered files - "buffered," because most implementations provide a char array (whose size is named BUFSIZ in <stdio.h>) to provide in-memory temporary storage of characters being read or written.

The main advantage of buffering is execution speed. On many systems, input and output are vastly more efficient if done in large chunks; disk transfers in particular are often more efficient in blocks of 256, 512, or 1024 characters. Taking putchar as an example, its usual implementation involves storing the output character into the buffer, updating a pointer to the next buffer location, and decrementing a count of the space left in the buffer. When the buffer is full, a physical I/O transfer then takes place to write out all the characters in the buffer. This scheme is known as "fully buffered" I/O.

Each of the default-file I/O functions has an equivalent version that accepts a FILE * argument:

getchar()         is equivalent to   getc(stdin)

putchar()         is equivalent to   putc(stdout)

scanf(fmt, ...)   is equivalent to   fscanf(stdin, fmt, ...)

printf(fmt, ...)  is equivalent to   fprintf(stdout, fmt, ...)

Until we discuss how to open files by name, the most useful of these functions is fprintf for the printing of error messages onto stderr:

if (x < x_low_lim)

fprintf(stderr, "x=%.6e is below x_low_lim=%.6e\n", x, x_low_lim);

It is often advantageous not to buffer the standard error file, since the last error message printed by a program in deep trouble might get lost in the buffer if the program's demise is untidy. Many environments provide an unbuffered stderr by default, but the safe course is to use the setbuf function:

setbuf(stderr, NULL);

will, if called before the first I/O operations upon stderr, put stderr into the "unbuffered" mode. Some environments will also put stdin and/or stdout into unbuffered mode if they are associated with an interactive terminal.

One final advantage of the "buffered" scheme of stream input is that "pushing-back" a character is easily provided for. If c contains a character read from stdin, calling ungetc(c, stdin) will cause the next input operation to read that character again. This is often useful in scanning an input file in which some punctuation character marks the end of each word or item. The punctuator can be "pushed back" onto the stream for another part of the program to handle.

5.3 Reading Lines of Input

For reading lines of input, two functions are provided in the standard library: gets and fgets. The gets function has the simpler calling sequence:

gets(buf)

reads one line from stdin, stores it in buf (with a nul-terminator but without a newline), and returns NULL at end-of-file. Since it provides no means of specifying the size of the receiving string, it can seldom be used in reliable programs. Furthermore, it gives no convenient way to tell whether a newline was present in the input. The fgets function is more reliable, but oftentimes awkward to use. Calling

fgets(buf, n, stdin)

will store at most n bytes into buf, reading from the file specified by the FILE * third argument (stdin, in this example). Of these bytes, the last byte stored is always a nul-terminator. If a newline is encountered in the input, it will precede the nul-terminator. Like gets, fgets returns  at end-of-file.

In many applications, the newline in the string is a nuisance, and the more useful returned value would be the length of the stored string. I have many times found more use for a slightly different function, named getsnn ("get string with no newline"). This function deletes the newline, and returns the length of the stored string, or EOF at end-of-file. If the length is less than its maximum (i.e., the specified size minus one), a newline was present in the input; if the length reaches the maximum, no newline was present. The function looks like this:

getsnn;

/* getsnn - get one line, omitting the newline (if any)

* Returns the number of characters stored,

* which equals size-1 only if no newline was found to that point.

* Returns EOF at end of file.

*/

#include "local.h"

int getsnn(s, size)

char s[];

int size;

{

int i;

metachar c;

for (i = 0; i < size-1 && (c = getchar()) != EOF && c != '\n'; ++i)

s[i] = c;

s[i] = '\0';

if (c == EOF)

return (EOF);

return (i);

}

Question [5-3] Modify getsnn into fgetsnn, a function with identical specification except for a third (FILE *) argument:

int fgetsnn(s, size, fp)

char s[];

int size;

FILE *fp;

As an example of the use of getsnn, consider the function getreply, designed for the common task of printing a prompting message and receiving a reply. If the user types more characters than are wanted, we are not interested in them. (In the next chapter we will see more sophisticated means of receiving replies in which the user sees immediately that no more characters are allowed on input. The getreply function will adequately handle the simple cases such as getting a 'y' or 'n' reply.) The function prints the prompt, gets a reply using getsnn, consumes the rest of the line if getsnn did not consume the newline, and returns the length of the reply string, or EOF if end-of-file.

getreply:

/* getreply - print a prompt and get one-line reply

* Returns length of reply string, or EOF if end-of-file

* Consumes an entire input line, even if over-long.

*/

#include "local.h"

int getreply(prompt, reply, size)

char prompt[];

char reply[];

int size;

{

metachar last_char;      /* the last char read */

int get_ret;             /* getsnn return: length of reply or EOF */

printf("%s", prompt);

get_ret = getsnn(reply, size);

if (get_ret == EOF)

return (EOF);

else if (get_ret == size-1) /* no newline was read */

while ((last_char = getchar()) != EOF && last_char != '\n')

;

if (last_char == EOF)

return (EOF);

return (get_ret);

}

5.4 Formatted Input

Topics assumed from first book: Basic scanf formats [3.12].

The combinations of basic conversion characters for scanf can be illustrated by the following table:

data size

short  int  long      conversion

%hd  %d    %ld      decimal integer

%ho  %o    %lo      octal integer

%hx  %x    %lx      hexadecimal integer

%hu  %u    %lu      unsigned integer

float      double

%e or %f   %le or %lf  floating-point number

A "one-word" string (delimited by whitespace) can be read with the %s format:

char next_word[LARGE_ENOUGH];

if (scanf("%s", next_word) != 1)

/* must be EOF if scanf failed */

The value of LARGE_ENOUGH must be adequate for the largest possible input word.

For all the input formats discussed so far, scanf will skip over any input whitespace (including newline!) to find the next input item. If a width specification precedes the conversion character, leading whitespace is still skipped, and then only the specified number of input characters are considered in the conversion. Thus,

char next_char[2];

scanf("%1s", next_char)

will read into next_char the next non-whitespace input character, followed by a nul-terminator. If the percent-sign is followed by an asterisk (the "assignment-suppression flag"), the corresponding input item will be matched but not assigned. This means that

scanf("%*1s%1s", next_char)

will read into next_char the second non-whitespace character from the input.

The %c input format behaves somewhat differently: it produces the next input character, whitespace or not. For example, an input file consisting of 80-character "card-images" could be read by

scanf("%80c", card_image)

Note that no nul-terminator will be placed after the characters.

For discussion of recent additions to scanf, as well as a comparison of different versions, see the appendix [5-4].

Now we will consider some of the reliability problems of the scanf functions. Numeric input is not checked for overflow. This means that numerical input using scanf can be trusted only if the inputs are somehow known to be in-bounds (perhaps previously produced by a reliable program). As alluded to earlier, strings read by the %s format have no guaranteed upper bound on their length. And character input, such as %80c, works reliably only when the input file is known to have the proper number of characters available. A format like this would be a disaster if applied to input lines of varying length; the newlines would just be swallowed up into the 80-character inputs.

Another problem with scanf is that its returned value is a very crude indicator of the success of the scan. The format argument can specify literal text characters that are supposed to be matched in the input, but scanf tells only how many arguments were successfully assigned. In particular, if a format string ended with literal text instead of a format specifier ("percent-sign something"), there would be no way to tell whether the text was successfully matched or not. For example, if a format string ends with a whitespace character, this requests scanf to skip over any trailing whitespace after the last input item -- but there is no way to tell if there was any such trailing whitespace after scanf returns.

Finally, there is scanf's "push-back" behavior: unless the final format stopped because of a specified width bound, there is one input character which scanf pushed back onto the input stream, using ungetc, or its equivalent. This makes for problems when scanf input conversion is combined with other input functions. For example, unless the format ends with whitespace, scanf will not consume the newline at the end of a line that it has read, so calling a line-oriented function must be done with full awareness that the input is still on the same line that scanf partially consumed.

Somewhat finer control is available by reading an entire input line with a line-oriented function such as getsnn, and then splitting it up with sscanf. This works for reasonably error-free inputs where "diagnose-and-terminate" (or "diagnose and re-prompt") is adequate for errors.

The scanf functions provide no control over the size of strings read via %s formats. When the input can be expected not to exceed some generous maximum line length, an intermediate buffer can solve the problem. For example, we can make a getpstr function which looks for a specified "prompt" sequence in the input and passes back the whitespace-delimited "word" which follows:

getpstr:

/* getpstr - get a string from input

* string must be preceded by specified "prompt" text

* return YES if text matched, NO otherwise

*/

#include "local.h"

#define FMTSIZE 84  /* arbitrary limit on size of "prompt" */

bool getpstr(p, s, n)

char p[];                /* : !NULL */

char s[];                /* : string */

size_t n;                /* : {1:sizeof(p[*])} */

{

char buf[BUFSIZ];        /* : string */

char fmt[FMTSlZE];       /* : string */

metachar c;

if (!strfit(fmt, p, FMTSIZE - 4))    /* copy "prompt" into fmt */

return (NO);         /* prompt text is too long */

strcat(fmt, "%s");       /* make fmt into a scanf format */

while (isspace(c = getchar()))

;                    /* skip leading whitespace */

ungetc(c, stdin);        /* put the non-whitespace back */

if (scanf(fmt, buf) != 1)

return (NO);         /* scanf match failed */

if (!strfit(s, buf, n))

fprintf(stderr, "<%s> chopped to <%s>\n", buf, s);

return (YES);

}

Here is a typical usage: the input is expected to contain the characters "Name: " (preceded and followed by any amount of whitespace) and then one "word" (whitespace-delimited string) which is to be read into a string (name_str, say). The corresponding call to getpstr would look like this:

if (!getpstr("Name:", name_str, sizeof(name_str)))

error("Expected   Name: ...", "");

The getpstr function consumes only one word after the expected "prompt" prefix. If we want a similar function which will consume the entire remainder of the line, it is simpler to read the rest of the line with a line-oriented function such as getsnn. Here is the function getplin ("get prompted line") which does so:

getplin:

/* getplin - get a line from input

* line must start with specified "prompt" text

* return YES if text matched, NO otherwise

*/

#include "local.h"

#define FMTSIZE 84

bool getplin(p, s, n)

char p[];

char s[];

size_t n;

{

char buf[BUFSIZ];

char fmt[FMTSIZE];

metachar c;

if (!strfit(fmt, p, FMTSIZE - 4))

return (NO);    /* prompt text is too long */

strcat(fmt, "%c");

while (isspace(c = getchar()))

;

ungetc(c, stdin);

if (scanf(fmt, buf) != 1)

return (NO);

if (getsnn(buf, BUFSIZ) == EOF)

return (NO);

if (!strfit(s, buf, n))

fprintf(stderr, "<%s> chopped to \n<%s>\n", buf, s);

return (YES);

}

5.5 Direct Terminal I/O

Most of the reliability problems discussed above are eliminated by using direct terminal I/O. In this mode, the program reads the user's inputs character-by-character, accepting only good data as it is typed. If the user types too many characters, or the wrong kind of characters (e.g., non-numeric in a numeric field), the program can "beep" and refuse to echo the character. This mode also has the advantage of controlling the physical appearance of the screen directly.

To accomplish this, we need control over the cursor position and control over the echoing of characters. There is, unfortunately, no portable means to enter this mode in standard C; the best that we can do is to define a minimal set of functions and write an implementation for them in each different environment. (The libraries known as curses, termcap, and terminfo in the UNIX world provide all of this and more, but these libraries are not available on most non-UNIX C implementations.)

Our functions will address the screen by line and column numbers. Line zero, column zero is the "home" position, in the upper left corner. Global variables scr_lins and scr_cols give the number of lines and columns on the terminal:

These are the functions that we will need:

scr_open() will do whatever is necessary in order that each key be received by the program as it is typed.

scr_close() will restore the ordinary terminal state.

scr_clear() erases the screen and moves the cursor to the home position.

scr_cel() clears to the end of the current line.

scr_curs(line, col) causes the cursor to move to the specified location.

scr_putc(c) prints the character c at the current screen position and moves the cursor accordingly.

scr_print(fmt, ...) performs a printf at the current screen position.

scr_refresh() forces any buffered screen changes to be written to the physical screen.

To achieve the needed control over the terminal screen, some environments have to turn off the usual mapping of newline ('\n') into carriage-return+linefeed. (The newline character when unmapped produces a linefeed only, dropping down one line without return to the margin.) If we need to accomplish a carriage-return+newline, we will have to transmit "\r\n" explicitly. If newline mapping is still turned on, nothing has been lost in doing so.

The scr_refresh macro forces the flushing of the standard-output buffer by calling the library function fflush.

Regarding the input, there is a function scr_getkey which performs some useful mappings. Many terminals now have cursor-control keys (typically left-arrow, right-arrow, up-arrow, down-arrow, and home) but the actual code sequences which they generate are not standard. The scr_getkey function interprets these special sequences and returns a single scalar value, which is either an ordinary character or a special code for each of the mapped keys.

The screen functions work together to ensure that the terminal screen has an invariant property that we might call the screen-sync property: The program knows what the screen looks like. There are no characters on the screen that were not known to the program.

To be very strict about the screen-sync property, we should take steps to ensure that no other programs or other users can write to the screen during execution of the program. The details will vary, of course, from system to system.

The code examples below are simple implementations of the above operations. One implementation (screenU.c) works on a UNIX system with an ANSI terminal (DEC VT-100, Zenith Z-19 and Z-29, etc.), the other one works on an IBM PC-compatible system with the "ANSI.SYS" driver.

screen.h:

/* screen.h - header for terminal package

* Assumes ANSI terminal

*/

# ifndef SCREEN_H

#define SCREEN_H

typedef short SCR_CMDCHAR;

#define SCR_RETURN '\r'

#define SCR_EXIT   0x101

#define SCR_UP     0x102

#define SCR_DOWN   0x103

#define SCR_LEFT   0x104

#define SCR_RIGHT  0x105

#define SCR_HOME   0x106

#define SCR_CLEAR  "\33[2J"    /* clear entire screen, go HOME */

#define SCR_CEL    "\33[K"     /*clear to end of line */

#define SCR_TTY 1  /* screen is in ordinary (non-direct) mode */

#define SCR_RAW 2  /* screen is in "total-control" mode */

#define SCR_CMD(c) \

(c == SCR_RETURN || c >= 0x101 && c <= 0x106)

extern short scr_mode;     /* {SCR_TTY, SCR_RAW} */

extern short scr_lins;     /* number of lines on screen */

extern short scr_cols;     /* number of columns on screen */

void scr_close();          /* PARMS(void) */

SCR_CMDCHAR scr_getkey();  /* PARMS(void) */

void scr_open();           /* PARMS(void) */

#define scr_beep()     putchar('\7')

#define scr_clear()    printf(SCR_CLEAR)

#define scr_cel()      printf(SCR_CEL)

#define scr_curs(r, c) printf("\33[%d;%dH", (r)+1, (c)+1)

#define scr_print      printf

#define scr_putc(c)    putchar(c)

#define scr_refresh()  fflush(stdout)

#endif

screenU.c:

/* screen - environment-specific terminal functions

* UNIX version for ANSI terminal

*/

#include "local.h"

#include "screen.h"

#define get_7bit_char() (getchar() & 0177)  /* "raw" getchar never EOF */

short scr_mode = SCR_TTY;   /* screen mode - TTY or RAW */

short scr_lins = 24;    /* screen lines (default) */

short scr_cols = 80;    /* screen columns (default) */

/* scr_getkey - get a (coded) keyboard char */

SCR_CMDCHAR scr_getkey()

{

char c1, c2;

scr_refresh();

if ((c1 = get_7bit_char()) != '\33')

return (c1);

else if ((c2 = get_7bit_char()) == 'O')

{

if (get_7bit_char() =='S')      /* F1 function key */

return ( SCR_EXIT);

scr_beep();

return (scr_getkey());

}

else if (c2 == '[')

{

switch (get_7bit_char())

{

case 'A':   return (SCR_UP);    /* no "break" needed - all returns */

case 'B':   return (SCR_DOWN);

case 'C':   return ( SCR_RIGHT ) ;

case 'D':   return (SCR_LEFT);

case 'H':   return (SCR_HOME);

default:    scr_beep();

return (scr_getkey());

}

}

else

{

scr_beep();

return (scr_getkey());

}

}

/* remark - print error message, appropriate for scr_mode */

void remark(sl, s2)

char s1[], s2[];    /* strings to be printed */

{

if (scr_mode == SCR_TTY)

fprintf(stderr, "%s %s\n", s1, s2);

else

{

scr_curs(scr_lins-1, 0);

scr_print("%s %s; hit any key to continue", s1, s2);

scr_getkey();

scr_curs(scr_lins-1, 0);

scr_cel();

}

}

/* scr_open - initialize the terminal */

void scr_open()

{

system("stty raw -echo");   /* slow but universal */

printf("\33[>6h");          /* keypad-shifted; not universal ANSI */

scr_mode = SCR_RAW;

}

/* scr_close - re-establish normal terminal environment */

void scr_close()

{

printf("\33[>6l");          /* exit keypad-shifted mode */

system("stty - raw echo");  /* slow but universal */

scr_mode = SCR_TTY;

}

screen86.c:

/* screen - environment-specific terminal functions

* PC Version - uses bdos function

*/

#inclue "local.h"

#include "screen.h"

short scr_mote = SCR_TTY;   /* screen mode - TTY or RAW */

short scr_lins = 24;    /* screen lines (default) */

short scr_cols = 80;    /* screen columns (default) */

/* scr_getkey - get a (coded) keyboard char */

SCR_CMDCHAR scr_getkey()

{

char c1;

scr_refresh();

if ((c1 = bdos(8)) != '\0')

return (c1 & 0x7F);

switch (c1 = bdos(8))

{

/* no "break" needed - all returns */

case 'H':   return (SCR_UP);

case 'P':   return (SCR_DOWN);

case 'M':   return (SCR_RIGHT);

case 'K':   return (SCR_LEFT);

case 'G':   return (SCR_HOME);

case ';':   return (SCR_EXIT);  /* F1 function key */

default:         scr_beep();

return (scr_getkey());

}

}

/* remark - print error message, appropriate for scr_mode */

void remark(s1, s2)

char s1[], s2[];    /* strings to be printed */

{

if (scr_mode == SCR_TTY)

fprintf(stderr, "%s %s\n", s1, s2);

else

{

scr_curs(scr_lins-1, 0);

scr_print("%s %s; hit any key to continue", s1, s2);

scr_getkey();

scr_curs(scr_l ins-1, 0);

scr_cel();

}

}

/* scr_open - enter "raw" screen mode */

void scr_open()

{

scr_mode = SCR_RAW; /* otherwise no work; bdos(8) is unbuffered */

}

/* scr_close - restore normal tty state */

void scr_close()

{

scr_mode = SCR_TTY;

}

This simple implementation of scr_putc outputs each character with putchar, which may be totally unbuffered in some environments. A significantly more efficient implementation would store up characters internally until the "screen-refresh" function (scr_refresh) was called. Even more sophisticated is the scheme of the curses library. Output to the screen is saved in a screen-image buffer until a refresh function is called. Then, the desired new screen image is compared with the existing image, and only those screen positions which have changed are output, with appropriate cursor motions, character-inserts, character-deletes, etc. Both approaches to screen refreshing are easy to add onto the basic scheme, if you want to.

As a simple demonstration of cursor addressing, the function plot_trk (listed below) simulates a pseudo-oval track around the screen. (There are 100 points on the track, chosen so that the physical distance between points is roughly equal.)

plot_trk:

/* plot_trk - plot position around a pseudo-oval track */

#include "local.h"

#include "screen.h"

#define LIN_ORIGIN 11

#define COL_ORIGIN 40

#define NPOINTS 26

#define IMAX (NPOINTS - 1)

#define POS(n, signlin, signcol) \

scr_curs(LIN_ORIGIN + (signlin) * coords[n].lin, \

COL_ORIGIN + (signcol) * coords[n].col)

static struct coords

{

short lin, col;

} coords[INPOINTS] =

{       /* points for one quadrant (quarter-screen) */

{11,  0},  {11,  2},  {11,  4},  {11,  6},  {11,  8},

{11, 10},  {11, 12},  {11, 14},  {11, 16},  {11, 18},

{11, 20},  {11, 22},  {11, 24},  {11, 26},  {11, 28},

{10, 29},  { 9, 30},  { 8, 31},  { 7, 32},  { 6, 33},

{ 5, 34},  { 4, 35},  { 3, 36},  { 2, 36},  { 1, 36},

{ 0, 36},

};

/* plot_trk - plot a point */

void plot_trk(n, c)

int n;

char c;

{

asserts(n >= 0,"plot_trk: n is non-negative");

n %= 4 * IMAX;

if (n < IMAX)

POS(n, 1, 1);               /* 1st quadrant - lower right */

else if (n < 2 * IMAX)

POS(2 * IMAX - n, -1, 1);   /* 2nd quadrant - upper right */

else if (n < 3 * IMAX)

POS(n - 2 * IMAX, -1, -1);  /* 3rd quadrant - upper left */

else

POS(4 * IMAX - n, 1, -1);   /* 4th quadrant - lower left */

scr_putc(c);

}

A simple "test-drive" function runs four times around the track, then stops.

plot_m.c:

/* plot_m - demonstrate plot_trk function */

#include "local.h"

#include "screen.h"

main()

{

short i;

scr_open();

scr_clear();

for (i = 0; i < 400; ++i)  /* four times around the track */

plot_trk(i, 'X');

scr_curs(scr_lins-1, 0);

scr_close();

}

Those of you programming race-track simulations will immediately see the applicability. In the next chapter, we will have more substantive uses for all these new functions.

Go to Chapter 6    Return to Table of Contents


Reliable Data Structures: CHAPTER 6: STRUCTURES

 Return to Table of Contents

CHAPTER 6: STRUCTURES

Structures are useful for grouping a collection of objects into a single aggregate object. Structures can even be subdivided down to the level of machine bits, using bit-fields. Unions are syntactically very similar to structures, allowing the same memory to be used in different ways.

In this chapter, we will examine various uses for structures and unions, as well as relevant reliability rules. The chapter concludes with a complex data structure known as a "menu," which provides a convenient generalized user interface.

6.1 Records

Topics assumed from previous book: structures [8.1], members [8.2].

One of the oldest uses of structures is the record, a collection of information about a single entity. For example, a structure can be used to collect information about a specific part in an assembly:

struct part

{

char part_no[14];   /* part number: string */

short lead_time;    /* lead time (in weeks): {0:99} */

char unit_meas[6];  /* unit of measure: string {"each", "lb", "box"} */

float unit_cost;    /* cost for one unit_meas: {0.00:9999.99} */

short cost_qty;     /* quantity required for price: {0:9999} */

};

In a typical environment where short takes two bytes and float takes four bytes, the memory layout of a struct part might look like this:

Each data type has its own alignment requirement, a requirement (imposed by the CPU hardware and/or the compiler) that the address of this type of data must be evenly divisible by some number. In the example above, we have assumed that each alignment requirement is no more restrictive than "even address" (divisible by two). In such implementations, a struct part occupies 28 bytes.

Suppose, however, that we are compiling for a machine on which float data is required to have an divisible-by-four address. Such an alignment requirement would force the compiler to leave a hole in the structure, an unused portion of the storage which exists solely to preserve alignment. Furthermore, a structure has its own alignment requirement, which is the most restrictive of its members' alignment requirements. Thus, in this implementation, a struct part would be required to have a divisible-by-four address. Another hole would be required at the end, to cause the size of a struct part to be evenly divisible by four. (If a structure's size were not an even multiple of its alignment requirement, an array of structures would come out mis-aligned.) In this more restrictive implementation, the memory layout would look like this:

And the size of a struct part would be 32 bytes, in this more restrictive implementation.

Rule 6-1: In portable programming, do not hard-code the numeric values of structure offsets. The values may be different in each environment. Refer to members by their symbolic member names only.

The advantage of a structure over an array is that the members can have different data types. In this example, the information recorded for each part consists of two char arrays for strings, two short integers, and one float.

The declaration given above does not reserve any storage for any variable; it merely describes a "shape" or template for a part record. The template is identified by a tag name (part, in this example). We can declare one or more variables to have this template by using the type struct part in a declaration:

struct part part1;

This creates storage for the variable part1. Thus, the entire structure part1 and also its individual members (part1.part_no, part1.lead_time, partl. unit_meas, ...) are all lvalues.

6.2 Structures for Information-Hiding

One of the important uses for C structures is to encapsulate interface information. Consider, for example, the FILE structure provided by the standard I/O library. On many systems, the header stdio.h defines the name FILE something like this:

#define FILE  struct_file

struct_file

{

char  *_cur_ptr;  /* where to get/put the next character: !NULL */

int   _size;      /* how big is the buffer: {0:INT_MAX} */

char  *_buffer;   /* where is the buffer: char[_size] | NULL */

int   _rcount;    /* how many more gets left: {0:_size} */

int   _wcount;    /* how many more puts left: {0:_size} */

char  _status;    /* file state: bits */

char  _fd;        /* file descriptor: {0:_NFILES-1} */

};

Thus the name FILE becomes a synonym for the type struct_file. The tag name (_file) and each of the member names (_cur_ptr, _size, etc.) are given names that start with a leading underscore. There is an associated reliability rule:

Rule 6-2: Names with leading underscore should only appear in code that is privy to the internal details of the associated data structure, not in "user-level" portable code.

In particular, you can see that if a supposedly portable program were to use one of these "private names," the program would probably not even compile correctly on another system, since the actual names themselves will very likely be different. If the user uses only the name FILE, the program will portably conform to each target environment.

In C, the "information-hiding" of private names is purely advisory, enforced by style conventions rather than compilers. The advantage for simplicity in programming is that the header provided with a functional package (such as the stdio functions) can easily define macros that allow efficient use of the data structure, just by using the private member names needed.

The main advantage of this defined-type approach is that the internal details of the representation are somewhat more "hidden" from the functions that use the type. This allows the internal details to be tailored to each particular target system, or to be changed to incorporate more efficient algorithms when needed. Furthermore, those functions that do not examine the internal representation of the object need no source-code changes if the object is changed from a structure to a scalar (or to a union - see Section 6.4).

Creating defined-types for structures can also be done with typedef. For example, the FILE type could be declared like this:

typedef struct_file

{

char  *_cur_ptr;  /* where to get/put the next character */

int   _size;      /* how big is the buffer: {0:INT_MAX} */

char  *_buffer;   /* where is the buffer: char[_size] | NULL */

int   _rcount;    /* how many more gets left: {0:_size} */

int  _wcount;     /* how many more puts left: {0:_size} */

char  _status;    /* file state: bits */

char  _fd;        /* file descriptor: {0:_NFILES-1} */

} FILE;

The typedef approach has the minor advantage of preventing any subsequent re-definition of the name, but either approach will serve adequately.

Using either approach, we will assume that for each generally-useable structure, both the defined (upper-case) name and the tag (lower-case) name will be available. Here is another useful convention: Structures used in several different source files should be declared in a header to be included wherever needed. The "part-record" structure might, for example, be declared like this:

part.h(#1):

/* part.h - header for PART structure */

#ifndef PART_H

#define PART_H

typedef struct part

{

char part_no[14];   /* part number: string */

short lead_time;    /* lead time (in weeks): {0:99} */

char unit_meas[6];  /* unit of measure: string {"each", "lb", "box"} */

float unit_cost;    /* cost for one unit_meas: {0.00:9999.99} */

short cost_qty;     /* quantity required for price: {0:9999} */

} PART;

#endif

Rule 6-3: Use the "leading underscore" name format for tag and member names if the internal details of the structure are not to be inspected by functions outside of the package. Conversely, avoid leading underscores if the details of the structure are available for inspection by functions that use the structure.

6.3 Properties of Structures

Recall from Section 2.9 that an object is well-defined if it conforms to the programmer's stated defining property. Since structures are an aggregate of individual members, it will be useful to provide a default meaning for "well-defined" as applied to structures. In the absence of a specific defining property, a structure will be said to be well-defined if all its scalar constituents are well-defined. Referring back to the definition of the PART structure, in order for a PART to be well-defined, all of the following must be true:

part_no    must be a (nul-terminated) string

lead_time  must be within the range {0:99}

unit_meas  must contain the string "each", "lb", or "box"

unit_cost  must be within the range {0.00:9999.99}

cost_qty   must be within the range {0:9999}

Why is it important to be clear about whether a structure is well-defined? We desire that we be able to talk about the behavior of a program in simple terms without a lot of qualifying phrases. For example, in talking about a particular program, we want to be able to say "part1 is a well-defined PART structure" rather than having to verbalize a whole list of qualifications such as "the part_no member is a nul-terminated string, lead_time is within its allowable range, ... ."

One of the common abbreviations allowed by C is the initialization of a structure to "all zeros":

PART partl = {0};

This convention of zero-initialization is sufficiently common that it deserves a reliability rule:

Rule 6-4: If a structure is not well-defined when initialized to zero, document this fact in a comment. (The program will in general be simpler if the members are defined such that the zero-initialized structure is well-defined.)

Examine for a moment the declaration of a PART structure. Most of its members conform to their defining property when initialized to zero, all except for unit_meas, which is required to be one of three specific strings. Since the result of initializing a string to all zeros is a null string, the rule above suggests that we consider re-defining the properties of the unit_meas member like this:

char unit_meas[6];  /* unit of measure; string {" ", "each", "lb", "box"} */

Alternatively, we would have to add a comment to the declaration of PART to the effect that it is not well-defined upon zero-initialization.

Oftentimes a structure has other properties that go beyond "well-definedness" as described above. Regarding the FILE structure, for example, we might wish to document two properties of a FILE structure called "is-open" and "is-closed." Hypothetically, let us say that the cur_ptr member is NULL if the file is closed and non-NULL if the file is open. We might then add these lines to the declaration:

/* cur_ptr != NULL ==> FILE is open */

/* cur_ptr == NULL ==> FILE is closed */

Consideration of the "zero-initialization" rule would suggest one more comment:

/* A zero-initialized FILE is closed */

6.4 Unions

A union is basically a struct in which all the offsets are zero. That is, all the members of a union start at byte zero of the object's storage, and hence occupy the same space.

The most portable use for a union is to overlay two or more data objects in the same space. For example,

struct rec

{

char name[20];

char type;

union

{

TYPE_A_INFO a;

TYPE_B_INFO b;

} info;

};

contains a member named info which contains either "type-a" information or "type-b" information. Presumably, the type member tells which of the alternatives is used in each particular instance. In C, there is no automatic way for the compiler to tell which member of the union is being used; it is entirely the programmer's responsibility to keep track of the state of the union. For portable programming, the state of a union object includes (at least in concept) the name of the member last assigned into. Any use of the value of a union member should be used through the last member assigned into.

Another use for union is alignment-forcing:

union buffer

{

int align;

char buf[BUFSIZ];

};

union buffer dsk_block;

will provide an array of characters (named dsk_block) which is aligned on an int boundary. The program will probably never refer to the align member except in the declaration; the only purpose of the member is to force alignment. The actual array of characters is accessed by the expression dsk_block.buf (the buf member of dsk_block). One of the more common reasons for alignment-forcing is interfacing with specific hardware or operating system interfaces.

6.5 Bit Fields

C provides a storage-compaction capability for structure members, in which each member occupies only a specified number of bits. Such a member is known as a bit-field. Bit-fields can be useful for reducing the storage needed for a large array of structures. They are also useful for defining various hardware interfaces which specify the individual bits within a machine word.

Consider the representation of time-of-day in hours, minutes, seconds and milliseconds. Bit-fields provide one way to represent such times:

time_day.h(#1):

/* time day.h - bit-field structure for hh:mm:ss.fff */

#ifndef TIME_DAY_H

#define TIME_DAY_H

typedef struct time_day

{

unsigned h1 : 2;    /* tens digit of hours      {0:2} */

unsigned h2 : 4;    /* units digit of hours     {0:9} */

unsigned m1 : 3;    /* tens digit of minutes    {0:5} */

unsigned m2 : 4;    /* units digit of minutes   {0:9} */

unsigned s1 : 3;    /* tens digit of seconds    {0:5} */

unsigned s2 : 4;    /* units digit of seconds   {0:9} */

unsigned f1 : 4;    /* first digit of fraction  {0:9} */

unsigned f2 : 4;    /* second digit of fraction {0:9} */

unsigned f3 : 4;    /* third digit of fraction  {0:9} */

} TIME_DAY;     /* 32 bits total */

#endif

The last millisecond of the day is

23:59:59.999 (hh:mm:ss.fff)

Each member (bit-field) is declared to be unsigned (int); this is the only bit-field type that is guaranteed to be portable to all current compilers. Each member is declared to have only as many bits as are necessary to represent the possible digits at its position in the time representation. Representing h1 (first digit of hours) takes only two bits to represent the possible values (0, 1, and 2). And the largest members need only four bits to represent ten digits, 0 through 9. The total number of bits is 32.

Consecutive bit-field members are allocated by the compiler to the same int-sized word, as long as they fit completely. Thus, on a 32-bit machine, a TIME_DAY object will occupy exactly one int-sized word. On a 16-bit machine, the first five members (totalling 16 bits) will fit into one int-sized word, and the last four members will fit into an immediately following word. Such an exact fit is rare, however. Add another member such as "day-of-year" to the structure, and the nice size-fitting property disappears. Thus, bit-fields are useful for storage-saving only if they occupy most or all of the space of an int, and if the storage-saving property is to be reasonably portable, they must occupy most of the space in a 32-bit integer [6-1].

The order of allocation within a word is different in different implementations. Some implementations are "right-to-left": the first member occupies the low-order position of the word. Most PDP-11 and VAX compilers allocate right-to-left. Following the convention that the low-order bit of a word is on the right, the right-to-left allocation would look like this:

Most other implementations are "left-to-right":

This variability of allocation order leads to a portability rule:

Rule 6-5: In portable code, do not depend upon the allocation order of bit-fields within a word.

Of course, in machine-specific non-portable code one knows exactly how the bit-fields are laid out, and the internal details can be inspected with bitwise operations. A union provides a convenient way to say what is going on:

typedef union time_overlay      /* MACHINE DEPENDENT */

{

struct time_day time_as_fields;

long time_as_long;

} TIME_OVERLAY;

TIME_OVERLAY time_port;

This allows bitwise operations like time_port.time_as_long & OxFOO as well as providing access via bit-field names like time_port.time_as_fields.h1.

Specifying a field size of zero causes any subsequent allocations to begin on a new word boundary. Un-named bit-fields are allowed; they occupy space but are inaccessible, which is useful for "padding" within a structure.

Since most C machines do not support bit-addressing, the "address-of" (&) operator is not allowed upon bit-field members.

Aside from these complications, bit-fields can be treated just like any other structure member. The following declaration

#include "time_day.h"

struct time_day last_msec = {2, 3, 5, 9, 5, 9, 9, 9, 9};

initializes last_msec to the last millisecond of the day.

struct time_day now;

/* ... */

if (now.h1 == 0 | | (now.h1 == 1 && now.h2 < 2))

tests whether now is less than "noon."

If we wish to use the TIME_DAY structure for an "information-hiding" purpose, so that it could be changed without affecting the programs that use it, we should employ the "leading-underscore" convention mentioned earlier in Section 6.2:

time_day.h(#2):

/* time_day.h - bit-field structure for hh:mm:ss.fff */

#ifndef TIME_DAY_H

#define TIME_DAY_H

typedef struct_time_day

{

unsigned _h1 : 2;   /* tens digit of hours      {0:2} */

unsigned _h2 : 4;   /* units digit of hours     {0:9} */

unsigned _m1 : 3;   /* tens digit of minutes    {0:5} */

unsigned _m2 : 4;   /* units digit of minutes   {0:9} */

unsigned _s1 : 3;   /* tens digit of seconds    {0:5} */

unsigned _s2 : 4;   /* units digit of seconds   {0:9} */

unsigned _f1 : 4;   /* first digit of fraction  {0:9} */

unsigned _f2 : 4;   /* second digit of fraction {0:9} */

unsigned _f3 : 4;   /* third digit of fraction  {0:9} */

} TIME _DAY;    /* 32 bits total */

#endif

We chose the TIME_DAY example because it illustrates the use of bitfields nicely, but there are numerous other ways to represent time-of-day. One obvious method is simply to count milliseconds since midnight. Adding and subtracting times is easier this way, but printing or displaying times is easier with the bit-field structure. The choice of representation ultimately depends upon the intended uses for the data.

6.6 Structures of Strings

One of the simplest and most useful structures is one containing nothing but (nul-terminated) strings. For ordinary string data (such as names and addresses), this approach requires only that the programmer be careful to allow space for each nul-terminator.

Numeric data, if present, is stored in readable string form. This may necessitate conversion from numeric representation to string representation, which takes a significant amount of CPU time.

Coded data (a choice among a small number of alternatives) can usually be represented by a single char. But in order to preserve the consistent "string" property of all the structure members, an array of two char's is needed, to allow space for the nul terminator.

The advantage is that data stored in this fashion can be processed in a uniform manner; each member of the structure is a nul-terminated string. We will shortly describe a "menu" program which deals with data stored uniformly in strings. Its version of the part structure will look like this:

part.h(#2):

/* part.h - header for parts structure

*/

#ifndef PART_H

#define PART_H

typedef struct part

{

char part_no[14];   /* part number: string */

char lead_time[3];  /* lead time (in weeks): num string {0:99} */

char unit_meas[2];  /* unit of measure: coded string */

/* {"0"=each, "1"=lb, "2"=box, other=each} */

char unit_cost[8];  /* cost: num string {0.00 : 9999.99} */

char cost_qty[5];   /* qty required for price: num string {0:9999} */

} PART;

extern PART part1;

#endif

6.7 Pointers to Structures

Structures and unions may be used as operands for address-of (&), member (.), and sizeof operators. If part1 has the type struct part, then the expression &part1 has the type struct part * (pointer to struct part), and produces the address of part1. The member operator, as in part1.part_no, we have seen before. And sizeof(part1) gives the size (in bytes) of part1.

Recent versions of C also allow assignment, argument passing, and function return for structures and unions. One could, for example, write a function which would accept a structure argument, update the structure, and return it to the calling function, where the returned value could be assigned to a structure variable:

#include "part.h"

main()

{

struct part part1;

struct part chg_part( );

/* ... */

part1 = chg_part(part1);

/* ... */

}

struct part chg_part(arg_part)

struct part arg_part;

/* get new values for arg_part.lead_time, ... */

return (arg_part);

}

However, passing and returning structures can cost considerable CPU time; the entire structure is copied each time. Often it is more efficient to pass a pointer to a structure. The declaration

struct part *part_ptr;

declares that part_ptr points to struct part's. To access the members of the structure that part_ptr points to, we use the "arrow" (->) operator, as in

part_ptr->lead_time

Changing the definition of the function chg_part for use with a pointer to struct part's would produce something like this:

#include "part.h"

main()

{

struct part part1;

void chg_part();

/* ... */

chg_part(&part1);

/* ... */

}

void chg_part(ppart)

struct part *ppart;

{

/* Get new values for ppart -> lead_time, ... */

/* The changes take place directly in the pointed-to structure. */

}

The "dot" and "arrow" member operators can be written in terms of each other:

part_ptr->lead_time  is equivalent to (*part_ptr).lead_time

part1.lead_time      is equivalent to (&part1)->lead_time

The rules for when to use each form are simple: If ps is a pointer to a structure, the members of the pointed-to structure are accessed via the "arrow"; if s is a structure, its members are accessed via the "dot."

Now let us consider the properties of pointers to structures. As discussed in Section 4.2, a pointer value can be either undefined (not valid in any pointer contexts), or defined (either NULL or pointing to a valid object of the proper type). In order to speak simply about programs, if a pointer points to a structure, we will say that the pointer is well-defined if the structure that it points to is well-defined.

Since one of the common uses of pointers to structures is as function parameters, we should consider some rules for their reliable usage. Taking the simplest case first, an "out" pointer (used only to modify the pointed-to object) does not care what properties the object has when its address is passed to the function. It is usually an error to pass NULL to an "out" parameter, since it does not point to any object. (A NULL parameter could be used to mean"do not store anything this time," but an explicit comment should be given.) An "in" pointer (used only to read values from the pointed-to object) should, for reliability, be passed the address of a "well-defined" structure. If, however, the structure is supposed to have some other defined property, the parameter declaration should so indicate in a comment. The same conditions apply to an "in-out" pointer (used both to read values from, and change values in, the pointed-to structure).

Rule 6-6: Regarding parameters which are pointers to structures, an "out" pointer parameter is assumed to be non-NULL, pointing to the storage for a structure of the specified type. "In" and "in-out" pointer parameters are assumed to point to a well-defined structure of the specified type. Any exceptions should be noted in a comment on the parameter declaration.

6.8 Example: Menu Processor

Topics assumed from previous book: array of structures [8.5].

We can now apply these observations about structures to a practical problem, the execution of screen-oriented menus.

In screen-oriented programming, the keyboard is handled in a "raw no-echo" fashion. The typed key does not automatically echo on the screen. The program determines whether a character will be echoed to the screen, or whether it will instead cause some action.

As described in Section 5.5, we can make use of scr_getkey, which returns the user's latest input character with no intermediate buffering and no echoing of the input. The scr_getkey function returns each ordinary character entered, but also returns special coded values for the keys EXIT, UP, DOWN, RIGHT, LEFT, and HOME.

The particular menu processor that we will present is somewhat generalized. We will call each component of a menu a field. In the most conventional usage, the fields of the screen are actions to be performed:

MAIN PART MENU

Add a part record

Change a part record

Delete a part record

In this example, there are three fields, each one a possible action.

The location of the screen cursor indicates the "current field." (On our diagrams, the cursor is indicated by an underscore.) The cursor is moved among the fields by cursor-movement keys. The DOWN key moves to the next field downward; the UP key moves upward; the HOME key moves to the first field. The UP and DOWN movements "wrap around" from first to last, or vice-versa. When the current field is an action field, hitting RETURN causes that action to be invoked, and typing the first letter of a field description (A, C, or D, here) moves the cursor to that field.

Thus, in our sample MAIN PART MENU, if the user were to type C then RETURN, the action field entitled "Change a part record" would be invoked. An action can lead to the presentation of another menu, and a menu can contain data fields as well as, or instead of, action fields:

CHANGE A PART RECORD

Part number: 39849-02_

Lead time (weeks): 12

Unit of measure: each

Unit cost: 17.25

Cost qty: 100

All fields of the CHANGE A PART RECORD menu are data fields. Cursor keys UP, DOWN, and HOME work as before. Hitting RETURN moves the cursor to the next field. The cursor always appears at the righthand end of the data value. Hitting BACKSPACE moves the cursor leftward, erasing the rightmost character of the data value. Hitting the EXIT key returns control to the place where the current menu was invoked, back to the main menu, in this example. Typing an ordinary text character appends that character to the current field, until the maximum field length is reached. (Special handling of a "coded choice" field will be described later in this section.)

Before we turn to the overall design of the menu processor, it will be helpful to show the details involved in updating one data field's value. A string data value is described by a STRING_VAL structure:

typedef struct string_val

{

char edit_type; /* 'n' = numeric, 'a' = anything */

char *string;   /* ptr to the string value */

short maxsize;  /* maximum size of string; {2:SHORT_MAX} */

short *pnum;    /* ptr to short; NULL=store as string */

} STRING_VAL;

The function which updates string values is mu_sval ("menu string value"). It is called when the cursor is sitting just to the right of the displayed contents of the current field's value, like this:

Part number: 39849-02_

...

The associated STRING_VAL structure would look something like this, when mu_sval is called:

mu_sval reads the next input character. If it is a command character, work on this string is complete for now, and mu_sval returns. If it is a backspace, the prior character is erased (if there is one). If the character is invalid for entry in a numeric field, mu_sval calls the scr_beep function to indicate the problem. If there is space to store the new character, it is added to the string and echoed. Otherwise, a scr_beep indicates the out-of-bounds situation.

Before showing the mu_sval function, one refinement needs discussion. The STRING_VAL structure allows for the possibility that the field is actually stored as a short int instead of as a string. All of the editing and user interaction are the same as is provided for numeric strings, using an auxiliary string of the appropriate size. Upon leaving the field, this numeric string is converted into an integer value in *pnun. During the modification of the string, the function preserves a property that we can call "string-sync": the displayed image on the screen agrees with the memory contents of the string. Furthermore, the string contents are valid, according to the specification of the field: the length is within the stated bound, and numeric strings consist only of digits with a possible leading minus sign.

Here, then, is the code for the mu_sval function.

mu_sval:

/* mu_sval - get the string contents for a STRING_VAL field

* Assumes cursor is just to right of string

*/

#include "menu.h"

SCR_CMDCHAR mu_sval (psv)

STRING_VAL *psv;      /* ptr to the STRING_VAL structure : str_sync */

/* str_sync = screen agrees with contents of */

/*            psv->string */

{

char *s;             /* ptr to start of string */

short i;             /* index of current nul-terminator */

SCR_CMDCHAR c;       /* latest SCR_CMDCHAR returned from scr_getkey */

s = psv->string;

i = strlen(s);

FOREVER

{

c = scr_getkey();

if (SCR_CMD(c))             /* if c is a command character, */

return (c);             /* return it */

else if (c == '\b')

{

if (i > 0)

{

scr_print("\b \b"); /* erase previous character */

s[--i] = '\0';      /* shorten the string */

}

}

else if (psv->edit_type == 'n' && !isdigit(c) &&

!(i == 0 && c == '-'))

{

scr_beep( );              /* non-digit invalid */

}

else if (i < psv->maxsize - 1)

{

scr_putc(s[i++] = c);     /* append the char, and echo it */

s[i] = '\0';

}

else

scr_beep( );              /* no room left in string */

}

}

Notice that the parameter, psv, is a pointer to STRING_VALS. The edit_type member of the object pointed to by psv is accessed by writing psv->edit_type. Similarly, we see psv->string and psv->maxsize.

A data field may be a "choice field" instead of a string field. A choice field has a fixed set of possible values. In the "part record" example, the "unit-of-measure" field (unit_meas) might allow the values "each", "lb", and "box", which are coded into the strings "0", "1", and "2".

The user cycles among the possible values by typing LEFT or RIGHT; like the other special keys, these are interpreted by scr_getkey. The structure which controls a choice field is called a CHOICE_VAL:

typedef struct choice_val

{

short nchoices;     /* number of possible choices: {2:10} */

char **choices;     /* ptr to start of array of choice strings */

char *pc;           /* ptr to the one-char coded choice value */

} CHOICE_VAL;

A CHOICE_VAL structure for the unit_meas component of the part record might look like this:

This diagram corresponds to the following initialized declarations:

PART part1 = {0};

char *Tunit_meas[] = ("each", "lb", "box"}; /* table of code meanings */

CHOICE_VAL Cunit_meas =

{DIM(Tunit_meas), Tunit_meas, part1.unit_meas};

Having seen how structures like STRING_VAL and CHOICE_VAL are used to control the access to an individual data value, let us look now at the overall design. Each field is specified by a C structure, FIELD:

typedef struct field

{

short line, col;    /* co-ordinates of screen display position */

char *desc;         /* description (title) of this field: string */

char val_type;      /* what type of field is this? */

STRING_VAL *psv;    /* non-NULL iff val_type == 's' or 'S' */

CHOICE_VAL *pcv;    /* non-NULL iff val_type == 'c' */

struct menu *pmenu; /* non-NULL iff val_type == 'm' */

bool (*pfn)( );     /* function to do after string or choice field */

} FIELD;

A more sophisticated way of declaring val_type would have involved defining an enumeration,

enum val_types {STRING_TYPE, CHOICE_TYPE, MENU_TYPE, FN_TYPE};

and declaring val_type as

enum val_types val_type;

We chose the more primitive method, a simple char, mainly because enum is still not universally supported in existing compilers, and because the enumeration does not add any noticeable clarity.

If a FIELD structure's val_type equals 's' or 'S', then this field is a string field. The function mu_str accepts a pointer to a string FIELD, positions the cursor just past its currently-displayed string value, and calls mu_sval to update the string value. The relevant property ("field-sync") is that the screen display of the field agrees with the field's stored contents.

mu_str:

/* mu_str - get a string for a STRING_VAL field */

#include "menu.h"

SCR_CMDCHAR mu_str(pf)

FIELD *pf;    /* : fld_sync */

{             /* fld sync = field in data rec agrees with screen */

SCR_CMDCHAR ret;

STRING_VAL *psv;

psv = pf->psv;     /*get the string-value pointer for this field */

/* position scr_curs just to right of string value */

scr_curs(pf->line, pf->col + strlen(pf->desc) + 2 + strlen(psv->string));

ret = mu_sval(psv);                     /* get string value */

if (psv->pnum != NULL)                 /* pf => !fld_sync */

*psv->pnum = atoi(psv->string);   /* pf => fld_sync */

return (ret);

}

If a FIELD structure's val_type equals 'c', then this field is a choice field. The function mu_chv ("get choice value") allows the user to change the current value of a choice field:

mu_chv:

/* mu_chv - get a value for a CHOICE_VAL field

* Return the SCR_CMDCHAR that terminated the input of this field.

* fld_sync = field in data rec agrees with screen

*/

#include "menu.h"

SCR_CMDCHAR mu_chv(pf)

FIELD *pf;          /* ptr to current FIELD: fld_sync */

{

short nchoice;      /* current choice value: {0:9} */

CHOICE_VAL *pcv;    /* ptr to this field's CHOICE_VAL */

SCR_CMDCHAR c;      /* most recent user input char */

short prevlen;      /* length of previous choice display string */

short curlen;       /* length of current choice display string */

short i;

prevlen = 0;

pcv = pf->pcv;

if ('0' <= *pcv->pc && *pcv->pc <= '9')

nchoice = *pcv->pc - '0';

else

nchoice = 0;

FOREVER

{

/* scr_curs to start of choice display, display current choice */

scr_curs(pf->line, pf->col + strlen(pf->desc) + 2);

scr_print("%s", pcv->choices[nchoice]);       /* pf => fld_sync */

/* erase any leftover from previous choice */

curlen = strlen(pcv->choices[nchoice]);

for (i = 0; i < prevlen - curlen; ++i)

scr_putc( ' ' );

for (i = 0; i < prevlen - curlen; ++i)

scr_putc('\b');

prevlen = curlen;

/* get user input and process it */

c = scr_getkey();

if (c != SCR_LEFT && c != SCR_RIGHT)

return (c);

else if (c == SCR_RIGHT)

nchoice = IMOD(nchoice + 1, pcv->nchoices);

else /* c == SCR_LEFT */

nchoice = IMOD(nchoice - 1, pcv->nchoices);

*pcv->pc = '0' + nchoice;                     /* pf => !fld_sync */

}

}

Each time the keys LEFT or RIGHT are hit, nchoice is decremented or incremented, modulo pcv->nchoices, the number of choices for this field. (The IMOD macro from Section 2.7 is used to ensure that the new nchoice is positive.)

For both string and choice fields, the user can terminate the entry of the current field by typing the cursor keys UP, DOWN, or HOME. Additionally, the user can leave a choice field by typing a letter which matches the initial letter of a field description. In either case, if a non-NULL pointer to function (the "post-function") is provided for this field, that function is called (with a pointer to the current menu). If the post-function returns NO, the cursor returns to the current field. Thus the post-function could perform arbitrarily complicated validation of the current field.

A menu is specified by a structure, MENU:

typedef struct menu

{

char *title;          /* alphabetic title for menu: string */

FIELD *fields;        /* ptr to beginning of array of FIELDs */

short nfields;        /* how many fields in this menu */

short cur_field;      /* index of the current field */

} MENU;

In English translation, these structures are saying that a menu has a title, a specified number of fields, a current field, and an array of individual fields. Each field has screen co-ordinates for its display, a description, a value-type (string, choice, menu, or function), and four pointers: a pointer to a STRING_VAL structure, to a CHOICE_VAL structure, to another menu, and to a C function. (If the field is a function field, the menu processor simply calls the function that is pointed to. For simplicity of design, the processor ignores any return value from the function.) Here is a schematic picture of the data structures:

There is a function mu_pr for the displaying of menus. It makes use of two environment-dependent functions: scr_clear (to erase the screen) and scr_curs (to move the cursor to a specified (line, col) location). Its job is to create a certain property for the menu, namely that the screen display agrees with the contents of all the field descriptions and field contents ("menu-sync"). It accomplishes this by clearing the screen and then looping over all the fields, moving the cursor to the proper place on the screen for each field, and then displaying the field description and field contents.

mu_pr:

/* mu_pr - display a menu on the screen */

#include "menu.h"

void mu_pr(pm)      /* at return, pm => menu-sync */

MENU *pm;       /* ptr to the MENU to be displayed */

{

short i;        /* index for loop over fields */

FIELD *pf;      /* pointer to each FIELD */

short i_choice; /* numeric value of choice field */

scr_clear( );

/* print the menu title, centered on top line */

scr_curs(0, (scr_cols - strlen(pm->title))/2 - 3);

scr_print("%s", pm->title);

for (i = 0; i < pm->nfields; ++i)         /* for each field */

{

pf = &pm->fields[i];

/* print the field description at proper x,y location */

scr_curs(pf->line, pf->col);

scr_print("%s ", pf->desc);

/* for string and choice fields, print current value */

if (tolower(pf->val_type) == 's')

{

if (pf->psv->pnum != NULL)  /* field has numeric form */

itoa(*pf->psv->pnum, pf->psv->string, pf->psv->maxsize -1);

scr_print("%s", pf->psv->string);

}

else if (pf->val_type == 'c')

{

if ('0' <= *pf->pcv->pc && *pf->pcv->pc <= '9')

i_choice = *pf->pcv->pc - '0';

else

i_choice = 0;

scr_print("%s", pf->pcv->choices[i_choice]);

}

}

scr_refresh( );

}

Now we are ready for the function which processes the user responses to a menu. The mu_ask function allows the user to move around among the fields of a menu, entering or altering the values of data fields, until the user selects a MENU field, selects a function field, or hits EXIT. It is the job of mu_ask to call any associated post-function for each field. If the post-function returns failure (NO), mu_ask prevents the user from moving to another field. (EXIT is allowed, to prevent locking the user into a "no-exit" situation.) mu_ask embodies one special case: a string field with capitalized value-type 's' is "mandatory": mu_ask will not allow movement to another field until a non-null string has been entered in this field.

mu_ask:

/* mu_ask - get responses to menu items

* Keeps looping over string and choice fields;

* terminates when user selects an action field, or hits SCR_EXIT.

*/

#include "menu.h"

FIELD *mu_ask(pm)

MENU *pm;        /* ptr to current MENU */

{

FIELD *pf;       /* ptr to current FIELD */

FIELD *pfj;      /* ptr to j-th FIELD of this MENU */

SCR_CMDCHAR c;   /* most recent user key input */

char vtype;      /* value type of current FIELD */

bool legal;      /* is c valid input in this state? */

short j;         /* index over FlELDs of this MENU */

pm->cur_field = 0;     /* start with first field */

FOREVER

{

pf = &pm->fields [pm->cur_field];

legal = YES;

vtype = pf->val_type;

/* accept user input, get user's SCR_CMDCHAR */

if (vtype == 'c' || tolower(vtype) == 's')

{

if (vtype == 'c')

c = mu_chv(pf);

else            /* vtype == 's' OR 'S' */

c = mu_str(pf);

if (pf->pfn != NULL)

legal = (*pf->pfn)(pm);

if (c == SCR_EXIT)

return (NULL);

}

else

{

scr_curs(pf->line, pf->col);

c = scr_getkey();

}

/* at this point, c is the user's response, and */

/* legal records validity of 'c' or 's' field */

/* (more) */

/* determine next action, based on c */

if (c == SCR_RETURN && (vtype == 'm' || vtype == 'f'))

return (pf);

else if (c == SCR_EXIT)

return (NULL);

else if (!legal)

;       /* no actions allowed if validation failed */

else if (vtype == 'S' && pf->psv->string[0] == '\0')

legal = NO;

else if (!SCR_CMD(c) && tolower(vtype) != 's')

{

legal = NO; /* tentatively */

for (j = 0; j < pm->nfields; ++j)

{

pfj = &pm->fields[j];

if (toupper(pfj->desc[0]) == toupper(c))

{

pm->cur_field = j;

legal = YES;

break;

}

}

}

else if (SCR_CMD(c))

{

switch (c)

{

case SCR_UP:

pm->cur_field = IMOD(pm->cur_field - 1, pm->nfields);

break;

case SCR_DOWN:

case SCR_RETURN:

pm->cur_field = IMOD(pm->cur_field + 1, pm->nfields);

break;

case SCR_HOME:

pm->cur_field = 0;

break;

default:

legal = NO;

break;

}

}

if (!legal)

scr_beep();

} /* end FOREVER */

}

The top-level function of the package is mu_do. In a FOREVER loop, mu_do prints a menu and calls mu_ask. As long as the user remains within string fields and choice fields, mu_ask handles the dialog. If mu_ask returns NULL, the user requested to exit, and mu_do returns. Otherwise, the return specifies either a menu field or a function field. If a menu field, mu_do is called recursively to process the sub-menu; if a function field, the function is called.

mu_do:

#include "menu.h"

/* mu_do - present a menu, process actions until user SCR_EXIT

*/

void mu_do(pm)

MENU *pm;

{

FIELD *pf;

static short level = 0;

if (level == 0)

scr_open( ); /* initialize the terminal state */

++level;

FOREVER

{

mu_pr(pm);                        /* print the menu */

pf = mu_ask(pm);                  /* get a new field ptr */

if (pf == NULL)

break;

else if (pf->val_type == 'm')    /* new field is a menu */

mu_do(pf->pmenu);

else /* pf->val_type == 'f' -- new field is a C fn */

{

asserts(pf->val_type == 'f', "val_type is 'f' ");

(void)(*pf->pfn)(pm);

}

}

--level;

if (level == 0)

{

scr_curs(scr_lins-1, 0);

scr_close();    /* restore terminal state */

}

}

One utility function is provided for use by the user program. It is called mu_reply, and it behaves just like the getreply function shown in Section 5.3, except that in SCR_RAW mode the dialog takes place at the bottom of the screen.

mu_reply:

/* mu_reply -- get a reply in response to a prompt

*              on the last screen line

*/

#include "local.h"

#include "menu.h"

mu_reply(prompt, reply, size)

char prompt[];      /* : string */

char reply[];       /* : !NULL */

int size;           /* max number of chars allowed in reply */

{

STRING_VAL rsv;     /* reply string value structure */

if (scr_mode == SCR_TTY)

return (getreply(prompt, reply, size));

else

{

scr_curs(scr_lins - 1, 0);

scr_print("%s", prompt);

reply[0] = '\0';        /* reply string initially empty */

rsv.edit_type = 'a';    /* initialize the STRING_VAL rsv */

rsv.string = reply;

rsv.maxsize = size;

rsv.pnum = NULL;        /* rsv => well-defined */

mu_sval(&rsv);          /* let mu_sval handle the dialog */

return (strlen(reply));

}

}

Finally, here is the header menu.h:

menu.h:

/* menu.h - header for menu pgm */

#ifndef MENU_H

#define MENU_H

#include "local.h"

#include "screen.h"

typedef struct string_val

{

char edit_type;      /* 'n' = numeric, 'd' = short int, 'a' = anything */

char *string;        /* ptr to the string value */

short maxsize;       /* maximum size of string: {2:SHORT_MAX} */

short *pnum;         /* if non-NULL, ptr to numeric storage */

} STRING_VAL;

typedef struct choice_val

{

short nchoices;      /* number of possible choices: {2:10} */

char **choices;      /* ptr to start of array of choice strings */

char *pc;            /* ptr to the one-char coded choice value */

} CHOICE_VAL;

/* (more)

*/

typedef struct field

{

short line, col;      /* co-ordinates of screen display position */

char *desc;           /* description (title) of this field: string */

char val_type;        /* what type of field is this? */

STRING_VAL *psv;      /* non-NULL iff val_type == 's' or 'S' */

CHOICE_VAL *pcv;      /* non-NULL iff val_type == 'c' */

struct menu *pmenu;   /* non-NULL iff val_type == 'm' */

bool (*pfn)();        /* function to do after str or choice field */

} FIELD;

typedef struct menu

{

char *title;          /* alphabetic title for menu: string */

FIELD *fields;        /* ptr to beginning of array of FlELDs */

short nfields;        /* how many fields in this menu */

short cur_field;      /* index of the current field */

} MENU;

FIELD *mu_ask( );        /* PARMS(MENU *pm) */

SCR_CMDCHAR mu_chv( );   /* PARMS(FIELD *pf) */

void mu_do( );           /* PARMS(MENU *pm) */

void mu_pr( );           /* PARMS(MENU *pm) */

int mu_reply( );         /* PARMS(char *prompt, char *reply, int size) */

SCR_CMDCHAR mu_str( );   /* PARMS(FIELD *pf) */

SCR_CMDCHAR mu_sval( );  /* PARMS(STRING_VAL *psv) */

#endif

This completes the source listing for the menu processor. It is a rather concise program, considering the capabilities that it provides. This is largely due to the encoding of so much information into the structures that drive it.

Much more flexibility and many more features are available in commercially available menu and screen-generator software. Here are a few simple extensions that could be added to this menu processor, left as exercises:

Exercise 6-1. An extra member (e.g., horizontal_ok) could be added to the MENU structure; if its value is YES, this menu allows the LEFT key as synonym for UP and RIGHT for DOWN. (This is an intuitive behavior for horizontal menus.)

Exercise 6-2. The cursor can be turned off except when entering strings. The current field can instead be highlighted (underscore, reverse video, etc.).

Exercise 6-3. The distinction between string fields and other types of field can be emphasized by emboldening the first character of other-field names.

Exercise 6-4. All data fields are currently stored as nul-terminated strings. Revise the program so that fields are blank-padded to their full size. (This creates a more traditional data processing record.)

For a concrete example of the use of the menu processor, let us consider the "change part record" menu. The first field ("part number") is mandatory, as indicated by the capitalized 's' in its field entry. Its description string "Part number " is to be displayed at row 2 and column 0. The field specifies the address of a STRING VAL structure named Spart_no. This field has a C function, is_part, associated with it; when the user attempts to leave this field, the is_part function will be called. Movement to another field succeeds only if a YES return is received. (More about is_part shortly.) The "unit of measure" field is a choice field, and all the others are string fields. Here is the C source file that defines the "change part record" menu:

chg_menu.c:

/* chg_menu - CHANGE A PART RECORD */

#include "local.h"

#include "menu.h"

#include "part.h"

extern bool is_part();

static STRING_VAL Spart_no =

{'a', (part1).part_no, sizeof((part 1).part_no), NULL};

static STRING_VAL Slead_time =

{'n', (part1).lead_time, sizeof((part1).lead_time), NULL};

static char *Tunit_meas[]=

{

"each", "lb", "box",

};

static CHOICE_VAL Cunit_meas =

{DIM(Tunit_meas), Tunit_meas, (part1).unit_meas};

static STRING_VAL Sunit_cost =

{'a', (part1).unit_cost, sizeof((part1).unit_cost), NULL};

static STRING_VAL Scost_qty =

{'n', (part1).cost_qty, sizeof((part1).cost_qty), NULL};

static FIELD Fchg_menu[ ] =

{

{2, 0, "Part number", 'S', &Spart_no, NULL, NULL, is_part},

{4, 0, "Lead time (in weeks)", 's', &Slead_time, NULL, NULL, 0},

{6, 0, "Unit of measure", 'c', NULL, &Cunit_meas, NULL, 0},

{8, 0, "Cost per unit", 's', &Sunit_cost, NULL, NULL, 0},

{10, 0, "Qty required for price", 's', &Scost_qty, NULL, NULL, 0},

};

MENU chg_menu = {"CHANGE A PART RECORD", Fchg_menu, DIM(Fchg_menu), 0};

The other submenus (add_menu, acd_menu, and del_menu) are similar. The source file for the main program, part_menu, also contains, for convenience, the functions called by each of the menus.

part_menu.c:

# include "menu.h"

# include "part.h"

/* main menu */

/* menu-sync = screen agrees with menu contents */

PART part1 = {0};

PART null_part = {0};

extern MENU add_menu, chg_menu, del_menu, acd_menu;

/* isn_part - verify that no record exists for this key */

bool isn_part(pm)

MENU *pm;         /* : menu-sync */

{

if (db_find_part(&part1))

{

/* pm => !menu-sync, new part contents */

remark("This part is entered already", "");

STRUCTASST(part1, null_part);   /* clear the part struct */

mu_pr(pm);                      /* pm => menu-sync */

return (NO);

}

return (YES);

}

/* is_part - verify that record exists and read it into part1 */

bool is_part(pm)

MENU *pm;

{

if (!db_find_part(&part1))

{

remark("Unknown part number", "");

STRUCTASST(part1, null_part);       /* clear the part struct */

mu_pr(pm);      /* display the cleared record, pm => menu-sync */

return (NO);

}

/* pm => !menu-sync, new part contents */

mu_pr(pm);           /* display the found record, pm => menu-sync */

return (YES);

}

/* addpart - add a part record */

bool addpart(pm)

MENU *pm;

{

char reply[2];

STRUCTASST(part1, null_part);

mu_do(&add_menu);

/* if no valid transaction occured, exit without change to db */

if (part1.part_no[0] == '\0')

return (NO);

mu_reply("Add this part [y/n]?", reply, 2);

if (reply[0] == 'y')

{

return (db_add_part(&part1));

}

else

return (NO);

}

/* chgpart - change a part record */

bool chgpart(pm)

MENU *pm;

{

char reply[2];

STRUCTASST(part1, null_part);

mu_do(&chg_menu);

/* if no valid transaction occured, exit without change to db */

if (part1.part_no[0] == '\0')

return (NO);

mu_reply("Change this part [y/n]? ", reply, 2);

if (reply[0] == 'y')

return (db_chg_part(&part1));

else

return (NO);

}

/* delpart - delete a part record */

bool delpart(pm)

MENU *pm;

{

char reply[2];

STRUCTASST(part1, null_part);

mu_do(&del_menu);

/* if no valid transaction occured, exit without change to db */

if (part1.part_no[0] == '\0')

return (NO);

mu_reply("Delete this part [y/n]? ", reply, 2);

if (reply[0] == 'y')

return (db_del_part(&part1));

else

return (NO);

}

/* part_menu (main) - execute the parts menu */

main()

{

db_open_part();

mu_do(&acd_menu);

db_close_part();

}

The simulated database of parts records is provided by part_db.c, containing several functions which manipulate a small array of PART records.

part_db.c:

/* part_db.c - simulated database access to parts */

# include "local.h"

# include "part.h"

static PART parts[3] = {0};

static short n_part = 0;

/* db_locate_part -- see if a part is in the database */

static int db_locate_part(partp)

PART *partp;

{

short i;

for (i = 0; i < n_part; ++i)

if (STREQ(partp-> part_no, parts[i].part_no))

return (i);       /* this is the part, return its index */

return (-1); /* no part from 0 to n_part matches this part_no */

}

/* db_add_part - add a part record to database */

bool db_add_part(partp)

PART *partp;

{

if (n_part >= DIM(parts))

{

remark("Part storage is full", "");

return (NO);

}

STRUCTASST(parts[n_part], *partp);

++n_part;

return (YES);

}

/* db_find_part - find a part record in database */

bool db_find_part(partp)

PART *partp;

{

short i;

i = db_locate_part(partp);

if (i == -1)

return(NO);

STRUCTASST(*partp, parts[i]);

return(YES);

}

/* db_del_part - delete a part record in database */

bool db_del_part(partp)

PART *partp;

{

short i;

for (i = 0; i < n_part; ++i)

{

if (STREQ(partp->part_no, parts[i].part_no))

{

--n_part;

STRUCTASST(parts[i], parts[n_part]);

return (YES);

}

}

return (NO);

}

/* db_chg_part - change a part record in database */

bool db_chg_part(partp)

PART *partp;

{

short i;

for (i = 0; i < n_part; ++i)

{

if (STREQ(partp->part_no, parts[i].part_no))

{

STRUCTASST(parts[i], *partp);

return (YES);

}

}

return (NO);

}

/* db_open_part - open the parts database

* Dummy - no action needed

*/

void db_open_part()

{

}

/* db_close_part - close the parts database

* Dummy - no action needed

*/

void db_close_part()

{

}

Thus, the entire program consists of the main program file (part_menu), the part_db package, and the three source files for the submenus ( add_menu, chg_menu, and del_menu.)

Lest you think that I am teaching you the wrong lessons, let me add a few words about the overall functionality. This example is chosen primarily for its recognizability and its simplicity. In a real application, the separation of "add," "change," and "delete" would be considered rather awkward; the three operations could be combined into one somewhat more complicated menu. Also, the user must EXIT from each menu in order to affect a change to the database, which is distracting.

All these shortcomings could be overcome with a different approach to the problem, which is left as an exercise to the reader:

Exercise 6-5. Program part_menu with all operations combined into one menu that would look something like this:

UPDATE PART RECORDS

Part number:

Action [a, c, d, p]:

Lead time (weeks):

Unit of measure:

Unit cost:

Cost qty:

In your design, you may find it useful to add a new function (or macro) to the menu package: mu_set_fld(n) will set the current field number to n.

6.9 A Translator for Menus

Preparing the C source code for a menu is highly repetitIve work with the potential for introduction of errors. We will look at a simple program which will allow menus to be written in a more human-readable form. The user is assumed to be a C programmer who is familiar with the conventions of the menu package; the translator will just make the programmer's job a bit easier.

The human-readable form for a menu looks something like this:

chg_menu.mu:

Menu name: chg_menu     Menu title: CHANGE A PART RECORD

Header: part.h        C-struct: part1

Field name: part_no     Desc: Part number

Line: 2 Col: 0    Type: S     Edit: a     Post-fn: is_part

Field name: lead_time     Desc: Lead time (in weeks)

Line: 4 Col: 0    Type: s      Edit: n     Post-fn: 0

Field name: unit_meas     Desc: Unit of measure

Line: 6 Col: 0   Type: c              Post-fn: 0

{

"each", "lb", "box",

};

Field name: unit_cost    Desc: Cost per unit

Line: 8 Col: 0   Type: s     Edit: a      Post-fn: 0

Field name: cost_qty    Desc: Qty required for price

Line: 10 Col: 0  Type: s     Edit: n     Post-fn: 0

Compare this with the corresponding C source given at the end of the previous section.

The menu generator itself is a straightforward application of the functions gestpstr and getpl in shown in the previous chapter. Whenever the data required is a single "word," gestpstr will read it for us. Whenever an input string may contain more than one word, we use getpl in. Thus, the input format is extremely free-form, and the menu inputs may be formatted ad-lib. The only inputs that are not read with either of these functions are the list of alternatives for a choice field. Here were simple accept a comma-separated list of C strings. The end of the list is marked by a line consisting only of initial whitespace followed by };.

Purely for purposes of simplification, the Edit: input is coded like this:

a       String of any characters

n       String of digits only (with possible leading '-')

n1      short integer, 1 digit

n2      short integer, 2 digits (with possible leading '-')

... etc. ...

n6      short integer, 6 digits (with possible leading '-')

The full program is quite large but unsophisticated. The only deviation from a straight translation of input to output is the buffering of FIELD lines so that they can be printed at the end to initialize the array of FIELDS.

gen_menu.c:

/* gen_menu - generate C source for menu from description file */

#include "local.h"

#define MAXBFLDS 8000

#define C_ID_LEN 31

#define FILE_NM_LEN 64   /* max length of file name */

#define C_EXPR_LEN 80    /* arbitrarly limit on length of struct ref */

#define TITLE_LEN 80     /* max length of displayed title */

#define SCR_POS_LEN 3    /* max digits in screen position */

#define FTYPE_LEN 1      /* field type is only one char */

#define EDIT_LEN 2       /* edit type is 1 char plus 1 optional digit */

#define MAXFLINE (TITLE_LEN+FTYPE_LEN+EDIT_LEN+SCR_POS_LEN+SCR_POS_LEN+50)

#define GETPSTR(p, s) \

{  if (!getpstr(p, s, sizeof(s))) error("Cannot match" , p); }

#define OPTPSTR(p, s) \

getpstr(p, s, sizeof(s))

#define GETPLIN(p, s) \

{  if (!getplin(p, s, sizeof(s))) error("Cannot match" , p); }

/* internal variables */

static char buf[BUFSIZ] = {""};

static char buf_flds[MAXBFLDS] = {""};

static short iflds = 0;

static char mname[C_ID_LEN+1] = {""};

static char mstruct[C_EXPR_LEN+1] = {""};

static char mtitle[TlTLE_LEN+1] = {""};

static char mheader[FILE_NM_LEN+1] = {""};

static char fname[C_ID_LEN+1] = {""};

static char fline[SCR_POS_LEN+1] = {""};

static char fcol[SCR_POS_LEN+1] = {""};

static char ffn[C_ID_LEN+1] = {"");

static char ftype[FTYPE_LEN+1] = {""};

static char fedit[EDIT_LEN+1] = {""};

static char fdesc[TITLE_LEN+1] = {""};

void do_1_menu(), do_1_fld(), pr_s_fld(), pr_c_fld(), pr_m_fld(), pr_f_fld();

/* gen_menu (main) */

void main()

{

do_1_menu();

while (OPTPSTR("Field name:", fname))

do_1_fld(); /* process one field */

printf("static FIELD F%s[] =\n", mname);

printf("\t{\n");

for (i_flds = 0; buf_flds[i_flds] != '\0'; ++i_flds)

putchar(buf_flds[i_flds]);

printf("\t);\n");

printf("MENU %s = {\"%s\", F%s, DIM(F%s), 0};\n",

mname, mtitle, mname, mname);

}

/* do_1_menu - process the menu information */

void do_1_menu()

{

GETPSTR("Menu name:", mname);

GETPLIN("Menu title:", mtitle);

GETPSTR("Header:", mheader);

GETPSTR("C-struct:", mstruct);

printf("/* %s - %s */\n", mname, mtitle);

printf("#include \"local.h\"\n");

printf("#include \"menu.h\"\n");

printf("#include \"%s\ "\n", mheader);

}

/* do_1_fld - process the info for one field */

void do_1_fld()

{

asserts(IN_RANGE(i_flds, 0, MAXBFLDS - MAXFLINE), "buffer space ok");

GETPLIN("Desc:", fdesc);

GETPSTR("Line:", fline);

GETPSTR("Col:", fcol);

GETPSTR("Type:", ftype);

if (tolower(ftype[0]) == 's')

GETPSTR("Edit:", fedit);

if (ftype[0] != 'm')

GETPSTR("Post-fn:", ffn);

if (!STREQ(ffn, "0") && !STREQ(ffn, "NULL"))

printf("extern bool %s();\n", ffn);

switch (ftype[0])

{

case 's':

case 'S':

pr_s_fld();

break;

case 'c':

pr_c_fld();

break;

case 'm':

pr_m_fld();

break;

case 'f':

pr_f_fld();

break;

default:

error("Unknown field type", ftype);

}

i_flds += strlen(&buf_flds[i_flds]);

if (i_flds > MAXBFLDS - MAXFLINE)

error("out of buffer space", "");

}

/* pr_s_fld - print the info for a string field */

void pr_s_fld()

{

sprintf(&buf_flds[i_flds],

"\t{%s, %s, \"%s\", '%c', &S%s, NULL, NULL, %s},\n",

fline, fcol, fdesc, ftype[0], fname, ffn);

if (fedit[1] == '\0')

{

printf("static STRING_VAL S%s =\n", fname);

printf("\t{'%c', (%s).%s, sizeof((%s).%s), NULL};\n",

fedit[0], mstruct, fname, mstruct, fname);

}

else

{

printf("static char N%s[%c+1] = {0};\n",

fname, fedit[1]);

printf("static STRING_VAL S%s =\n", fname);

printf("\t{'%c', N%s, sizeof(N%s), &(%s).%s};\n",

fedit[0], fname, fname, mstruct, fname);

}

}

/* pr_c_fld - print the info for a choice field */

void pr_c_fld()

{

int ret;

sprintf(&buf_flds[i_flds,

"\t{%s, %s, \"%s\", '%c', NULL, &C%s, NULL, %s},\n",

fline, fcol, fdesc, ftype[0], fname, ffn);

printf("static char *T%s[]=\n", fname);

do {

ret = getsnn(buf, BUFSIZ);

if (ret != EOF && ret > 0) /* skip blank lines */

printf("%s\n", buf);

} while (ret != EOF && strchr(buf, ';') == NULL);

printf("static CHOICE_VAL C%s =\n", fname);

printf("\t{DIM(T%s), T%s, (%s).%s};\n",

fname, fname, mstruct, fname);

}

/* pr_m_fld - print the info for a menu field */

void pr_m_fld()

{

printf("extern MENU %s;\n", fname);

sprintf(&buf_flds[i_flds],

"\t{%s, %s, \"%s\", '%c', NULL, NULL, &%s, %s},\n",

fline, fcol, fdesc, ftype[0], fname, ffn);

}

/* pr_f_fld - print the info for a function field */

void pr_f_fld()

{

sprintf(&buf_flds[i_flds],

"\t{%s, %s, \"%s\", '%c', NULL, NULL, NULL, %s},\n",

fline, fcol, fdesc, ftype[0], ffn);

}

Exercise 6-6. Write a program (using menus) which will write ".mu"menu files.

Go to Chapter 7    Return to Table of Contents


Reliable Data Structures: CHAPTER 7: DYNAMIC DATA STRUCTURES

 Return to Table of Contents

CHAPTER 7: DYNAMIC DATA STRUCTURES

In this chapter, we will look at data structures which dynamically grow and shrink as the computation progresses. By contrast, the structures that we saw in the previous chapter were totally determined at program-initialization time.

7.1 Dynamic Storage Allocation

The function malloc provides dynamically-allocated storage;

char *p;

p = malloc(N);

will set aside a contiguous "chunk" of N bytes of data storage and return the address of this chunk to be stored in the pointer p. This memory is guaranteed not to overlap the storage of any declared variable or any other memory obtained by the allocation functions.

Besides the malloc function, there is also a calloc function, which behaves like malloc with two extra features: the allocated memory is filled with 0-bits, and the number of bytes requested is given by the product of the two arguments.

p = calloc(N, M);

will cause p to point to a new chunk of N * M bytes filled with zeroes.

The conventional term for the location of allocated storage is the heap. In many environments, the heap starts at the low addresses of the dynamic data segment and grows upwards to high addresses, while the stack starts at high addresses and grows downwards. If a function call needs more stack space than the dynamic segment has available for it ("stack-heap collision"), this is a fatal error. Nothing reliable can happen afterwards. Most environments will terminate the program with a severe error.

If a request for allocated memory asks for more bytes than are available in the heap, the malloc (or calloc) function returns a NULL pointer. Thus, the returned pointer should always be tested to be sure that it is non-NULL. Moreover, early in the program design, if the program uses dynamic memory allocation, an error-handling strategy must be determined for the "out-of-heap" condition. (On many non-UNIX systems, it is necessary to specify a maximum size for the program's dynamic memory; the details depend upon the environment [7-1].)

When allocation is successful, the memory provided will have the most restrictive alignment of any data type on the target machine. In other words, it is safe to store any kind of data in the allocated memory, because the alignment will always be adequate.

One of the most common uses of dynamic allocation is storing a structure. In this case, the pointer by which we will access the memory will be a pointer to some type of structure, and the returned pointer from malloc (or calloc) should be cast to the type of the pointer:

struct x *px;

px = (struct x *)malloc(sizeof(struct x));

When allocated storage has been used and is no longer needed, it can be returned to the heap by the free function:

free(px);

will give back to the heap the storage which px is pointing to. Note carefully that each time storage is allocated, the allocation functions record internally the size that was given to this allocation. When free is called, this internally-recorded size is used to determine the amount of storage that is being given back. It is very important that the pointer value passed to the free function must be one which was originally delivered by malloc (or calloc).

One of the important reliability aspects of dynamic allocation is the avoidance of dangling pointers. After calling free(px), the pointer px should be considered to have an undefined value. To be sure, the pointer does still contain a valid machine address, but the storage that it points to may subsequently be allocated to some other use. One useful style convention is to immediately assign NULL to a pointer after passing it to the free function:

free(px);

px = NULL;

In some environments, this will ensure that any further access using px will generate an execution error [7-2]. And in any environment, it will warn the reader that px should not be used to access data.

This convention is not a complete preventative measure by itself, because more than one pointer may be pointing to the freed storage. Here is a more general rule:

Rule 7-1: When a pointer p is passed to the free function, the programmer must determine how many pointers are pointing into the freed storage. (This number is known as the "reference count" of the storage.) Steps must be taken (such as assigning NULL) to ensure that none of these pointers are subsequently used to access the freed storage.

We have seen that a "dangling pointer" results from continued use of a pointer to freed storage. There is a complementary problem known as dead storage which results from failure to free a chunk of storage when it is no longer needed. Here is an unusually blatant case of dead storage:

p = malloc(N);

p = NULL;

Access to the allocated storage is immediately lost, and there is no way to let the allocation functions know that the storage is available for other uses. A few isolated cases of dead storage are unlikely to be noticed; the storage available is simply decreased. But as the dead storage begins to accumulate, the odds increase that the program will run out of allocatable storage. The preventive measures are fairly simple:

Rule 7-2: For every instance in which a program allocates storage, there should be an easily identifiable instance in which that storage is later freed.

7.2 Singly-linked Lists

A structure can contain a pointer to the same type of structure, which is useful for building data structures which can grow and shrink during execution. The simplest such data structure is a singly-linked list, or chain.

Each box represents a node on the stack. Each node contains a pointer to another node; we will conventionally name this member of the node structure the next pointer. The bottom node is distinguished by having a null next pointer. In the other nodes, the next pointer points one node further down the chain.

One common use of a chain (singly-linked list) is to implement a stack, a data structure where insertions and deletions take place at one end, called the "top" of the stack. Thus, one pointer (pointing to the top node) is all that is needed to access the stack.

Note that we have here a situation very similar to the use of pointers to access an array: the indirect object of the head pointer (the thing that it points to, syntactically) is one single object, but it is being used to provide access to several such objects. When the data structure being accessed through head is a stack, we will often use the verbal shorthand of saying that head is a stack.

For the time being, we will use an array of four char'S as the data portion of each node in the stack.

Throughout this chapter, each structure template will contain one or more pointers to other structures of the same type. Declarations of such structure templates look more uniform if the #define method is used, rather than the typedef method, because the typedef method does not allow us to use the structure's uppercase name inside its own declaration:

#define ST_NODE struct st_node

ST_NODEtypedef struct st_node

{                            {

ST_NODE *next;               struct st_node *next;

char data[4];                char data[4];

};                           } ST_NODE;

With the typedef method, the name ST_NODE is not available for use inside the structure itself, because the name is not seen by the compiler until the end of the declaration. The comparison is aesthetic, not substantive; use whichever form you like. The declaration of the stack node template will go into a stack.h header, along with the declaration of some functions to appear shortly.

A new (empty) stack can be created simply by assigning (or initializing) NULL to a "head" pointer:

ST_NODE *head;

head = NULL;

The treatment of different data structures will be made easier, however, if we define a macro ST_OPEN which accepts a pointer to the head pointer (a ST_NODE ** parameter), and assign NULL to the head pointer:

#define ST_OPEN(headp)  (*(headp) = NULL)

Then the "opening" of a stack is accomplished via

ST_OPEN(&head);

To avoid special cases in the processing of dynamic structures, we will make sure that each node in a structure is well-defined, i.e., all members are well-defined. Thus, no "garbage" values will be found within the structure. The invariant properties of a singly-linked stack can therefore be expressed like this:

0.     If head == NULL, the stack is empty, i.e., contains no nodes.

N.     Otherwise, head points to a well-defined ST_NODE; this is the "first" or "top" node in the stack. Some node, node-N, has a NULL next- pointer; this is the "last" or "bottom" node in the stack. The stack accessed through head consists of the N nodes reachable via the chain of next-pointers. Every ST_NODE in the stack is well-defined.

The basic operations upon a stack are "pushing" a node onto the stack, and "popping" a node off the stack. The simplest implementation is obtained by requiring the user's calling function to allocate the space for a new node before calling the "push" function. And the "pop" function will detach a node from the stack and hand it to the calling function. It is thus up to the calling function to free the node's storage when it is needed no further.

The head pointer will also belong to the calling function. This allows the stack functions to be invoked upon several different stacks, but requires that each function must accept the value or the address of the head pointer as a parameter of each function. In this example, both the st_push and st_pop functions must modify the head pointer, so they must be passed its address. This is the st_push function:

st_push:

/* st_push - install ST_ NODE at head of stack */

#include "stack.h"

void st_push(headp, p)

ST_NODE **headp;

ST_NODE *p;

{

p->next = *headp;  /* p->next now points to previous head (or is NULL) */

*headp = p;        /* *headp now points to p */

}

And this is the st_pop function:

st_pop:

/* st_pop - detach top node of stack and return pointer to it */

#include "stack.h"

ST_NODE *st_pop(headp)

ST_NODE **headp;

{

ST_NODE *p;

if (*headp == NULL)

return (NULL);

p = *headp;        /* p points to top of stack */

*headp = p->next;  /* headp now points to next node (or is NULL) */

p->next = NULL;    /* prevent dangling ptr */

return (p);

}

Let us pause to examine the types of the pointers and expressions. The parameter headp is declared to have the type ST_NODE **, i.e., pointer to pointer to ST_NODE. The expression *headp thus has the type ST_NODE *, or pointer to ST_NODE. And p->next is the next member of the node that p points to. Its type is also ST_NODE *.

Also notice in the st_pop function that we assign NULL to the next member of the popped node, before turning the node over to the calling function. This will ensure that the popped node is not used to provide further access to the stack, whose structure may subsequently be altered by further pushes and pops.

To close a singly-linked stack, we must free any nodes which remain in the stack. We must then assign NULL to its head pointer.

st_close:

/* st_close - close the stack accessed via *headp */

# include "stack.h"

void st_close(headp)

ST_NODE **headp;

{

ST_NODE *p;

ST_NODE *pnext;

for (p = *headp; p != NULL; p = pnext)

{

pnext = p->next;

free(p);    /* p is (momentarily) undefined */

}

*headp = NULL;      /* prevent dangling ptr */

}

A simple for loop can step a pointer over each of the nodes on the stack:

for (p = head; p != NULL; p = p->next)

process *p

This operation -- looping a pointer over each node in a data structure -- will be useful for each data structure, so we will define a macro

#define EACH_ST(head, p) for ((p) = (head); (p) != NULL; (p) = (p)->next)

which we can use like this:

EACH_ST(head, p)

process *p

Three other macros will be useful: ST_IS_EMPTY(head) produces YES if the stack is empty, NO otherwise; ST_FIRST(head) produces a pointer to the first node on the stack; and ST_NEXT(p) produces a pointer to the ("successor") node that follows the node pointed to by p:

#define ST_IS_EMPTY(head)  ((head) == NULL)

#define ST_FIRST(head)     (head)

#define ST_NEXT(p)         ((p)->next)

This completes the list of operations needed to implement a stack as a singly-linked list. As configured, however, the stack works for only one specific type of node, one in which the data member is an array of four charS. We would prefer for our data structure operations to be more general than this, even at the expense of the simplicity of the implementation. After all, we will create the data structure headers and functions only once but we will use them many different times in different contexts.

The packaging approach that we will show in this chapter will allow the data structure functions and macros to be used upon any structure whose initial members contain the proper pointers. We will, for example, generalize the "stack" package to allow it to be used upon any structure whose first member is a pointer to other such structures, and the name of the first member must be next. The stack.h header will contain a declaration of an ST_NODE that looks like this:

#define ST_NODE struct st_node

ST_NODE

{

ST_NODE *next;

/* ... */

};

Suppose, then, that we wish to use the "stack" package with nodes that look like this:

#define NAME_NODE struct name_node

NAME_NODE

{

NAME_NODE *next;

char data[4];

};

NAME_NODE *name_head = NULL;

NAME_NODE *p = NULL;

A macro like ST_NEXT will generate the proper code when applied to NAME_NODES: ST_NEXT(name_head) becomes ((name_head)->next) as desired. When calling the functions, such as st_push, there is a type-matching problem to solve. A type-checking environment such as lint will notice that we are passing a pointer to NAME_NODE'S to a function that expects pointers to ST_NODES, and will properly complain. Since we know that the two structures are in fact compatible with each other for the uses that are being made within the function, we will add macros to the stack.h which call the function with pointer arguments that have had the proper casts applied to them.

It will also be useful to provide in each data structure header any appropriate "non-NULL" tests for each pointer argument. If the tests are conditional upon NDEBUG they will not slow down the execution version of the system. Since we will make use of the non-NULL tests and the pointer casts in several different data-structure headers, we will make a header, pointer.h, for these useful pointer operations:

pointer.h:

/* pointer.h - concise macros for pointer checking and casting */

/* PNN(p)       - test that p is not NULL, return p */

/* PNNC(p, t)   - test that p is not NULL, return p cast to type t */

#ifndef POINTER_H

#define POINTER_H

#ifndef NDEBUG

#define PNN(p)          ((p) != NULL ? (p) : PNN_BAD(p))

#define PNNC(p, t)      ((p) != NULL ? (t)(p) : (t)PNN_BAD(p))

#define PNN_BAD(p)      (fprintf(stderr, "NULL\n"), p)

#else

#define PNN(p)          (p)

#define PNNC(p,t)       ((t)(p))

#endif

#endif

We can now show the contents of the stack.h header:

stack.h:

/* stack.h - header for stack package */

#ifndef STACK_H

#define STACK_H

#include "local.h"

#include "pointer.h"

#define ST_NODE struct st_node

ST_NODE

{

ST_NODE *next;

/* ... */

};

void st_close();           /* PARMS(ST_NODE **headp) */

ST_NODE *st_pop();         /* PARMS(ST_NODE **headp) */

void st_push();            /* PARMS(ST_NODE **headp, ST_NODE *p) */

#define ST_P(p)            PNNC(p, ST_NODE *)

#define ST_PP(p)           PNNC(p, ST_NODE **)

#define ST_CLOSE(h)        (st_close(ST_PP(h)))

#define ST_POP(h)          (st_pop(ST_PP(h)))

#define ST_PUSH(h, p)      (st_push(ST_PP(h), ST_P(p)))

#define EACH_ST(head, p)   for ((p) = (head); (p) != NULL; (p) = (p)->next)

#define ST_FIRST(head)     (head)

#define ST_IS_EMPTY(head)  ((head) == NULL)

#define ST_OPEN(headp)     (*(headp) = NULL)

#define ST_NEXT(p)         ((p)->next)

#endif

Consider the ST_CLOSE macro. It makes use of an auxiliary macro named ST_PP which ensures that its argument is non-NULL and casts it to the type ST_NODE ** (pointer to pointer to ST_NODE). The macros ST_POP and ST_PUSH are similarly implemented.

The end result of all this packaging is that we can use the "stack" package with any structure that has a next pointer as its first member. All the operations are provided as macros that have uniform uppercase names beginning with "ST_."

In order to give a simple demonstration of the stack functions, we will create a small interactive program, st_main.c:

st_main.c:

/* st_main - test routine for stack package

*/

#include "local.h"

#include "stack.h"

#define NAME_NODE struct name_node

NAME_NODE

{

NAME_NODE *next;

char data[4];

};

NAME_NODE *head = NULL;     /* head node of stack */

NAME_NODE *p = NULL;        /* current node */

void show_cmds(), push_name(), pop_name(), dump_names();

/* st_main (main) */

main()

{

char buf[2];            /* buffer for input */

ST_OPEN(&head);         /* open the stack */

show_cmds();

while (getreply("?: ", buf, 2) != EOF)

{

switch (buf[0])

{

case '+' :

push_name();

break;

case '-':

pop_name();

break;

case '=':

dump_names();

break;

case '0':

ST_CLOSE(&head);

ST_OPEN(&head);    /* open the stack again */

break;

case '?':

show_cmds();

break;

default:

printf("unknown command: %c\n", buf[0]);

show_cmds();

break;

}

}

exit(SUCCEED);

}

/* show_cmds -- show legal commands */

void show_cmds()

{

printf("Type + to push, - to pop, = to print, 0 to reset:\n");

}

/* push_name - push new name on stack */

void push_name()

{

p = (NAME_NODE *)malloc(sizeof(NAME_NODE));

if (p == NULL)

error("out of space", "");

if (getreply("name: ", p->data, 4) == EOF)

error("unexpected EOF", "");

ST_PUSH(&head, p);

}

/* pop_name - pop a name off stack */

void pop_name()

{

p = (NAME_NODE *)ST_POP(&head);

if (p == NULL)

printf("EMPTY STACK\n");

else

{

printf("name= %s\n", p->data);

free(p);

}

}

/* dump_names - print the current stack of names */

void dump_names()

{

if (ST_IS_EMPTY(head))

printf("EMPTY STACK\n");

else

EACH_ST(head, p)

printf("name= %s\n", p->data);

}

We are handling the "out-of-heap" condition by simple "diagnose-and-terminate" logic. This is adequate for simple interactive uses.

7.3 Queues

With a small modification to the stack structure, we obtain a data structure that allows insertion at the one end and deletion at the other -- a queue, that is.

The properties of a queue are similar to those of a stack:

0.     If either front or rear is NULL, both are NULL, and the queue is empty, i.e., contains no nodes.

N.     Otherwise, front points to a well-defined node; this is the "first" or "front" node in the queue. Some node, node-N, has a NULL next-pointer; this is the "last" or "rear" node in the queue, and rear points to this node. The queue accessed through front and rear contains the N nodes reachable via the chain of next-pointers. Every Q_NODE in the queue is well-defined, i.e., all its members are well-defined.

The data structure itself is the same singly-linked list that we used for the stack; we have just added a second access pointer at the "rear." The operation of "opening" a queue assigns NULL to both the front and the rear pointers:

#define Q_OPEN(frontp, rearp) (*(frontp) = *(rearp) = NULL)

As with the stack, the "first" and "next" operations are simple macros:

#define Q_FIRST   (front) (front)

define Q_NEXT(p)  ((p)->next)

A queue can be "popped" just like a stack: simply detach the first node. (When the single remaining node is popped, the rear pointer must be set to NULL.)

q_pop:

/* q_pop - detach front node of queue and return pointer to it */

#include "queue.h"

Q_NODE *q_pop(frontp, rearp)

Q_NODE **frontp, **rearp;

{

Q_NODE *p;

if (*frontp == NULL)

return (NULL);

p = *frontp;            /* p points to front of queue */

*frontp = Q_NEXT(p);    /* front now points to next node (or NULL) */

if (*frontp == NULL)    /* if queue is now empty, */

*rearp = NULL;      /* then make both ptrs NULL */

Q_NEXT(p) = NULL;       /* prevent dangling ptr */

return (p);

}

However, we can define a somewhat more general operation, "detach to the right." Define the link-in of a node on a queue as that pointer which points to it. Specifically, the link-in to the first node is the front pointer; the link-in of each other node is the next pointer of the node to its left. Given a pointer to a link-in, we can detach the node to the right of the link-in:

q_r_detach:

/* q_r detach - detach queue node that *pp links-in and return pointer to it */

#include "queue.h"

Q_NODE *q_r_detach(frontp, rearp, pp)

Q_NODE **frontp, **rearp;

Q_NODE **pp;

{

Q_NODE *p2;             /* ptr to detached node */

if (*pp == NULL)

return (NULL);

p2 = *pp;               /* p2 points to node to detach */

*pp = Q_NEXT(p2);       /* *pp now points to next node (or NULL) */

if (*frontp == NULL)    /* if queue is now empty, */

*rearp = NULL;      /* then make both ptrs NULL */

Q_NEXT(p2) = NULL;      /* prevent dangling ptr */

return (p2);

}

A node can be "pushed" onto a queue (impolite, but useful sometimes), just as with a stack, except that the rear pointer must be set when the new node is the first arrival onto an empty queue.

q_push:

/* q_push - install Q_NODE at front of queue */

#include "queue.h"

void q_push(frontp, rearp, p)

Q_NODE **frontp, **rearp;

Q_NODE *p;

{

Q_NEXT(p) = *frontp;    /* p points to previous front (or NULL) */

*frontp = p;            /* *frontp now points to p */

if (*rearp == NULL)     /* if queue was empty, */

*rearp = p;         /* *rearp also points to p */

}

As with "pop," the "push" is a special case of inserting a node to the right of a pointer which becomes the node's link-in:

q_r_insert:

/* q_r_insert - install Q_NODE after *pp */

#include "queue.h"

void q_r_insert(frontp, rearp, p, pp)

Q_NODE **frontp, **rearp;

Q_NODE *p;

Q_NODE **pp;

{

Q_NEXT(p) = *pp;        /* p points to node following *pp (or NULL) */

*pp = p;                /* *pp now points to p */

if (*rearp == NULL)     /* if queue was empty, */

*rearp = p;         /* *rearp also points to p */

}

The queue allows an "append" operation, an insertion onto the end of the queue:

q_append:

/* q_append - install Q_NODE at rear of queue */

#include "queue.h"

void q_append(frontp, rearp, p)

Q_NODE **frontp, **rearp;

Q_NODE *p;

{

Q_NEXT(p) = NULL;              /* new node will be rear of queue */

if (*frontp == NULL)           /* if queue was empty, */

*frontp = *rearp = p;      /* p becomes front and rear */

else

{

Q_NEXT(*rearp) = p;        /* splice p into queue */

*rearp = p;                /* p becomes rear of queue */

}

}

The "close" operation must free all remaining nodes and assign NULL to both front and rear pointers:

q_close:

/* q_close - close the queue accessed via *frontp and *rearp */

#include "queue.h"

void q_close(frontp, rearp)

Q_NODE **frontp, **rearp;

{

Q_NODE *p;

Q_NODE *pnext;

for (p = *frontp; p != NULL; p = pnext)

{

pnext = Q_NEXT(p);

free(p);        /* p is (momentarily) undefined */

}

*frontp = *rearp = NULL;        /* prevent dangling ptr */

}

The macro EACH_Q behaves the same as the EACH_ST macro did:

#define EACH_Q(front, p) for ((p) = front; (p) != NULL; (p) = (p)->next)

A queue is empty if its front pointer is NULL:

#define Q_IS_EMPTY(front) ((front) == NULL)

A queue can conveniently be used to store an ordered list of nodes; we will soon see a queue of tasks stored in order of the time that they are supposed to be performed. In order for an "ordered insert" function to be general-purpose, it must accept a pointer to a comparison function, just as we saw in the "quicksort" examples:

q_insert:

/* q_insert - install Q_NODE at proper place in queue */

#include "queue.h"

void q_insert(frontp, rearp, p, cmpfn)

Q_NODE **frontp, **rearp;

Q_NODE *p;

int (*cmpfn)();             /* function for comparisons */

{

Q_NODE *p2;

if (*frontp == NULL                /* if queue was empty, */

|| (*cmpfn)(p, *frontp) < 0)   /* or if p comes first, */

Q_PUSH(frontp, rearp, p);      /* push p onto front */

else

{

for (p2 = *frontp; ; p2 = Q_NEXT(p2))

{

if (Q_NEXT(p2) == NULL)

{                            /* if end of queue, */

Q_APPEND(frontp, rearp, p);  /* append p at rear of queue */

break;

}

else if ((*cmpfn)(p, Q_NEXT(p2)) < 0)

{                       /* if p sorts before Q_NEXT(p2) */

Q_R_INSERT(frontp, rearp, p, p2);   /* insert after p2 */

break;

}

}

}

}

A "find" function is also easy, given a comparison function pointer. It will, however, be more useful for the "find" operation to produce a pointer to the link-in of the found node, since the link-in is necessary for insertions or deletions inside the queue:

q_lfind:

/* q_lfind - search queue for equal match to p, return its link-in

* Return NULL if no match

*/

#include "queue.h"

Q_NODE **q_lfind(frontp, rearp, p, cmpfn)

Q_NODE **frontp, **rearp;

Q_NODE *p;

int (*cmpfn)();

{

Q_NODE **pp;

for (pp = frontp; *pp != NULL; pp = &Q_NEXT(*pp))

if ((*cmpfn)(*pp, p) == 0)

return (pp);

return (NULL);

}

Access to all these operations is provided by the header for the queue package:

queue.h:

/* queue.h - header for queue package */

#ifndef QUEUE_H

#define QUEUE_H

#include "local.h"

#include "pointer.h"

#define Q_NODE struct q_node

Q_NODE

{

Q_NODE *next;

/* ... */

};

void q_append();    /* PARMS(Q_NODE **frontp, Q_NODE **rearp, Q_NODE *p) */

void q_close();     /* PARMS(Q_NODE **frontp, Q_NODE **rearp) */

void q_insert();

/* PARMS(Q_NODE **frontp, Q_NODE **rearp, Q_NODE *p, int (*cmpfn)()) */

Q_NODE **q_lfind();

/* PARMS(Q_NODE **frontp, Q_NODE **rearp, Q_NODE *p, int (*cmpfn)()) */

Q_NODE *q_pop();    /* PARMS(Q_NODE **frontp, Q_NODE **rearp) */

void q_push();      /* PARMS(Q_NODE **frontp, Q_NODE **rearp, Q_NODE *p) */

Q_NODE *q_r_detach();

/* PARMS(Q_NODE **frontp, Q_NODE **rearp, Q_NODE **pp) */

void q_r_insert();

/* PARMS(Q_NODE **frontp, Q_NODE **rearp, Q_NODE *p, Q_NODE **pp) */

/* (more)

*/

#define Q_PP(p)     PNNC(p, Q_NODE **)

#define Q_P(p)      PNNC(p, Q_NODE *)

#define Q_APPEND(fp, rp, p)     q_append(Q_PP(fp), Q_PP(rp), Q_P(p))

#define Q_CLOSE(fp, rp)         q_close(Q_PP(fp), Q_PP(rp))

#define Q_INSERT(fp, rp, p, fn) q_insert(Q_PP(fp), Q_PP(rp), Q_P(p), fn)

#define Q_LFIND(fp, rp, p, fn)  q_lfind(Q_PP(fp), Q_PP(rp), Q_P(p), fn)

#define Q_POP(fp, rp)           q_pop(Q_PP(fp), Q_PP(rp))

#define Q_PUSH(fp, rp, p)       q_push(Q_PP(fp), Q_PP(rp), Q_P(p))

#define Q_R_DETACH(fp, rp, pp)  q_r_detach(Q_PP(fp), Q_PP(rp), Q_PP(pp))

#define Q_R_INSERT(fp, rp, p, pp) q_r_insert(Q_PP(fp), Q_PP(rp), Q_P(p), Q_PP(pp

#define Q_IS EMPTY(front)       ((front) == NULL)

#define Q_OPEN(frontp, rearp)   (*(frontp) = *(rearp) = NULL)

#define Q_NEXT(p)               ((p)->next)

#define EACH_Q(front, p)        for ((p) = front; (p) != NULL; p = (p)->next)

#endif

A queue is a convenient data structure to use for a list of tasks to be performed. Each task has a specified "start time," and can be inserted into the proper chronological position in the queue.

task.h:

/* task.h - header for task package */

#ifndef TASK_H

#define TASK_H

#include "local.h"

#include "queue.h"

typedef short TIME;

#define TASK struct task

TASK

{

TASK *next;

TIME start;

char desc[40];

};

#endif

Here is a program which will accept the following transactions: insert a task into the queue, print the task with the earliest start time and remove it from the queue, print all tasks on the queue.

t_main.c:

/* t_main - manage a queue of tasks */

#include "local.h"

#include "queue.h"

#include "task.h"

TASK *front = NULL;     /* front node of queue */

TASK *rear = NULL;      /* rear node of queue */

TASK *p = NULL;         /* current node */

void insert_task(), pop_task(), dump_tasks(), show_cmds();

main()

{

char buf[2];                /* buffer for input */

show_cmds();

Q_OPEN(&front, &rear);      /* open the queue */

while (getreply("?: ", buf, 2) != EOF)

{

switch (buf[0])

{

case '>':

insert_task();

break;

case '-':

pop_task();

break;

case '=':

dump_tasks();

break;

case '0':

Q_CLOSE(&front, &rear);

Q_OPEN(&front, &rear);        /* open the queue */

break;

default:

printf("unknown command: %c\n", buf[0]);

show_cmds();

break;

}

}

exit(SUCCEED);

}

/* show_cmds -- show legal commands */

void show_cmds()

{

printf("Type > to insert, - to pop, = to print, 0 to reset:\n");

}

/* cmptime - compare the start members of two TASKS */

int cmptime(t1, t2)

TASK *t1, *t2;

{

if (t1->start < t2->start)

return (-1);

else if (t1->start == t2->start)

return (0);

else

return (1);

}

/* insert_task - insert new name on queue */

void insert_task()

{

char sstart[5];

p = (TASK *)malloc(sizeof(TASK));

if (p == NULL)

error("out of space", "");

if (getreply("start: ", sstart, sizeof(sstart)) == EOF)

error("unexpected EOF", "");

p->start = atoi(sstart);

if (getreply("desc: ", p->desc, sizeof(p->desc)) == EOF)

error("unexpected EOF", "");

Q_INSERT(&front, &rear, p, cmptime);

}

/* pop task - pop a name off queue */

void pop_task()

{

p = (TASK *)Q_POP(&front, &rear);

if (p == NULL)

printf("EMPTY QUEUE\n");

else

{

printf("start=%5d desc=%s\n", p->start, p->desc);

free(p);

}

}

/* dump_tasks - print the current queue of names */

void dump_tasks()

{

if (front == NULL)

printf("EMPTY QUEUE\n");

else

EACH_Q(front, p)

printf("start=%5d desc=%s\n", p->start, p->desc);

}

7.4 Deques: Double-Ended Queues

A "double-ended queue" or deque is a data structure in which each node has two pointers, one forward and one backward. Thus, it is also known as a "doubly-linked list."

A deque is often implemented with an extra node which contains no data but serves to make the operations more uniform; we will call this node a master node. A picture may help:

Given a pointer to any node, it is possible to reach all the other nodes by traversing pointers.

A node can be inserted to the right or to the left of any other node; all that is required is a pointer to the new node and a pointer to the insertion point. The following picture shows the insertion of the node d (i.e., the node pointed to by pointer d), to the right of node p, or to the left of node q. (Either p or q is adequate; either can be determined from the other.)

It is possible to remove a node from a deque given nothing more than a pointer to the node itself. Referring to the "after insertion" diagram above, given the pointer d, the nodes to the left and right can be located using information in the node *d. Thus, a deletion is just the exact reversal of an insertion.

An empty deque consists of one node, the master node, whose left and right pointers both point to itself:

Thus, even in the empty deque, the master node has a left and right node (itself). There are no special cases in the insertion logic. To implement the deletion logic, the only special case is that the master node should not be freed; the manipulation of left and right pointers is the same for the deletion of any node.

Having a master node makes the calling sequence of the deque operations much more uniform. In the header deque.h will appear a type definition for the name DQ_NODE, defined like this:

#define DQ_NODE struct dq_node

DQ_NODE

{

DQ_NODE *left;

DQ_NODE *right;

/* ... */

};

All the nodes in the deque, including the master, conform to this template. Since the master node is not a data node, we can require that the calling program must refer to the master node as a DQ_NODE, rather than as a data node. Thus, any program using the "deque" package must provide a "head" pointer which points to DQ_NODES.

These are the properties of a deque:

0.     If the left or right pointer of the master node contains the address of the master node itself, then both do, and the deque is empty, i.e., contains no data nodes.

N.     Otherwise, the right pointer of the master node points to a well-defined node; this is the "first" node in the deque. Some node, node-N, has a right pointer which points to the master node. The deque consists of the N nodes reachable via the chain of right-pointers, from the first node to node-N. For each node a in the deque, the node to the right of node a has a left-pointer that points back to a. Every node in the deque is well-defined, i.e., all its members are well-defined.

The operations dq_open and dq_close will alter the deque pointer and must be given its address, a "pointer to pointer to DQ_NODE" value. Most of the other operations will take a "pointer to DQ_NODE" first argument. Here are the dq_open and dq_close functions:

dq_open:

/* dq_open - open a deque */

#include "deque.h"

void dq_open(pdq)

DQ_NODE **pdq;

{

*pdq = (DQ_NODE *)malloc(sizeof(DQ_NODE));

if (*pdq == NULL)

return;

(*pdq)->left = (*pdq)->right = (*pdq);

}

dq_close:

/* dq_close - close a deque */

#include "deque.h"

void dq_close(pdq)

DQ_NODE **pdq; /* ptr to ptr to master DQ_NODE */

{

DQ_NODE *p;

DQ_NODE *pnext;

for (p = (*pdq)->right; p != (*pdq); p = pnext)

{

pnext = p->right;

free(p);

}

free(*pdq);

*pdq = NULL;

}

The dq_close function frees any nodes remaining in the deque, including the master node.

Deques are often used to represent "circular lists," where the first node is considered to be the successor of the last node. The master node is not a data node, so is not to be considered part of the circular list. (As in the other headers, we make use of the macro PNN -- "pointer non-NULL" -- to provide debugging assistance in preventing NULL pointer bugs.)

#define DQ_PRED(dq, d) \

(DQ_P(d)->left == PNN(dq) ? (dq->left) : (DQ_P(d)->left))

#define DQ_SUCC(dq, d) \

(DQ_P(d)->right == PNN(dq) ? (dq->right) : (DQ_P(d)->right))

Detaching a node requires only a pointer to the node itself. The function does not allow the master node to be detached; a NULL pointer is returned if an attempt is made to detach the master node.

dq_detach:

/* dq_detach - detach node d from deque */

#include "deque.h"

DQ_NODE *dq_detach(dq, d)

DQ_NODE *dq;

DQ_NODE *d;

{

DQ_NODE *p;

DQ_NOOE *q;

if (d == dq)

return (NULL);

q = d->right;

p = d->left;

p->right = q;

q->left = p;

return (d);

}

The operation "insert to the right of" was shown in the diagrams earlier. Because of the master node, there are no special cases in the insertion logic.

dq_r_insert:

/* dq_r_insert - insert node d to the right of node p */

#include "deque.h"

void dq_r_insert(d, p)

DQ_NODE *d;

DQ_NODE *p;

{

DQ_NODE *q;

q = p->right;

d->left = p;

d->right = q;

p->right = d;

q->left = d;

}

Consistent with our adopted packaging, a macro DQ_R_INSERT is provided which verifies that pointers are non-NULL and casts them to the appropriate types:

#define DQ_P(p)     PNNC(p, DQ_NODE *)  /* non-NULL, cast to DQ_NODE * */

#define DQ_R_INSERT(d, p)   dq_r_insert(DQ_P(d), DQ_P(p))

Insertion to the left of node p is most simply implemented by inserting to the right of node p->left:

#define DQ_L_INSERT(d, p)   DQ_R_INSERT((d), (p)->left)

The operation DQ_PUSH is trivial; it simply uses DQ_R_INSERT. In other words, to "push" node d onto deque dq, simply insert d after dq:

#define DQ_PUSH(dq, d)   DQ_R_INSERT((d), (dq))

The dq_pop function returns NULL if the deque is empty (no nodes except the master node). Otherwise, it detaches the node to the right of the master node.

dq_pop:

/* dq_pop - remove leftmost node and return pointer to it

* Treats deque as a stack to the right of dq master node.

*/

#include "deque.h"

DQ_NODE *dq_pop(dq)

DQ_NODE *dq;

{

DQ_NODE *d;

d = dq->right;

if (d == dq)

return (NULL);

return (DQ_DETACH(dq, d));

}

The work of DQ_APPEND is accomplished by inserting node d to the right of the last node in the deque:

#define DQ_APPEND(dq, d)  DQ_R_INSERT(d, (dq)->left)

The test for "emptiness" of a deque is whether the node to the right of the master node is the master node itself:

#define DQ_IS_EMPTY(dq)  (PNN(dq)->right == PNN(dq))

The operations dq_insert and dq_find (which require a cmpfn pointer to provide the ordering function) are similar to the corresponding queue functions, except that the end of the list is marked by reaching the master node instead of reaching a NULL pointer. Another difference is that the dq_find function can simply return a pointer to the found node itself, since that pointer is all that is needed for insertion, deletion, or access to the rest of the deque.

dq_insert:

/* dq_insert - install DQ_NODE at proper place in deque */

#include "deque.h"

void dq_insert(dq, p, cmpfn)

DQ_NODE *dq;

DQ_NODE *p;

int (*cmpfn)( )     /* function for comparisons: neg, zero, or pos */

{

DQ_NODE *p2;

if (DQ_IS_EMPTY(dq)             /* if deque was empty, */

|| (*cmpfn)(p, DQ_FIRST(dq)) < 0) /* or if p comes first, */

DQ_PUSH(dq, p);             /* push p onto front */

else                            /* else, p must sort after p2 */

{

EACH_DQ(dq, p2, DQ_NODE)    /* find where p belongs */

{

if (p2->right == dq)

{                    /* if end of deque, */

DQ_APPEND(dq, p);    /* append p at rear of deque */

break;

}

else if ((*cmpfn)(p, p2->right) < 0)

{                    /* if p sorts before p2->right */

DQ_R_INSERT(p, p2);  /* insert after p2 */

break;

}

}

}

}

dq_find:

/* dq_find - search deque for equal match to d, return ptr to match

* Return NULL if no match

*/

#include "deque.h"

DQ_NODE *dq_find(dq, d, cmpfn)

DQ_NODE *dq;

DQ_NODE *d;

int (*cmpfn)( );                    /*function for comparisons */

{

DQ_NODE *d2;

EACH_DQ(dq, d2, DQ_NODE)

if ((*cmpfn)(d2, d) == 0)

return (d2);

return (NULL);

}

dq_first:

/* dq_first - produce ptr to first deque node, otherwise NULL */

#include "deque.h"

DQ_NODE *dq_first(dq)

DQ_NODE *dq;

{

if (dq->right == dq)

return (NULL);

return (dq->right);

}

The loop macro, EACH_DQ, 1oops until the pointer reaches the master node. A third argument is necessary to specify the type of each node:

#define EACH_DQ(dq, d, t) \

for (d = (t *)(dq)->right; (DQ_NODE *)(d) != dq; d = (d)->right)

Here is the full deque.h header:

deque.h:

/* deque.h - header for deque package */

#ifndef DEQUE_H

#define DEQUE_H

#include "local.h"

#include "pointer.h"

#define DQ_NODE struct dq_node

DQ_NODE

{

DQ_NODE *left;

DQ_NODE *right;

/* ... */

};

#define DQ_P(p)          PNNC(p, DQ_NODE *)

#define DQ_PP(p)         PNNC(p, DQ_NODE **)

void dq_close( );        /* PARMS(DQ_NODE **pdq) */

DQ_NODE *dq_detach( );   /* PARMS(DQ_NODE *dq, DQ_NODE *d) */

DQ_NODE *dq_find( );     /* PARMS(DQ_NODE *dq, DQ_NODE *d, int (*cmpfn)( )) */

void dq_insert( );       /* PARMS(DQ_NODE *dq, DQ_NODE *d, int (*cmpfn)( )) */

void dq_open( );         /* PARMS(DQ_NODE **pdq) */

DQ_NODE *dq_pop( );      /* PARMS(DQ_NODE *dq) */

void dq_r_insert( );     /* PARMS(DQ_NODE *d, DQ_NODE *p) */

/* (more) */

/* pure macros */

#define DQ_APPEND(dq, d)    DQ_R_INSERT(d, (dq)->left)

#define DQ_IS_EMPTY(dq)     (PNN(dq)->right == PNN(dq))

#define DQ_L_DETACH(dq, d)  DQ_DETACH(dq, (d)->left)

#define DQ_L_INSERT(d, p)   DQ_R_INSERT((d), (p)->left)

#define DQ_PRED(dq, d) \

(DQ_P(d)->left == PNN(dq) ? (dq->left) : (DQ_P(d)->left))

#define DQ_PUSH(dq, d)      DQ_R_INSERT((d), (dq))

#define DQ_R_DETACH(dq, d)  DQ_DETACH(dq, (d)->right)

#define DQ_SUCC(dq, d) \

(DQ_P(d)->right == PNN(dq) ? (dq->right) : (DQ_P(d)->right))

#define EACH_DQ(dq, d, t) \

for (d = (t *)(dq)->right; (DQ_NODE *)(d) != dq; d = (d)->right)

/* function-interface macros */

#define DQ_CLOSE(pdq)       dq_close(PNN(pdq))

#define DQ_DETACH(dq, d)    dq_detach(DQ_P(dq), DQ_P(d))

#define DQ_FIND(dq, d, fn)  dq_find(PNN(dq), DQ_P(d), fn)

#define DQ_FIRST(dq)        dq_first(PNN(dq))

#define DQ_INSERT(dq, d, fn)  dq_insert(PNN(dq), DQ_P(d), fn)

#define DQ_OPEN(pdq)        dq_open(PNN(pdq))

#define DQ_POP(dq)          dq_pop(PNN(dq))

#define DQ_R_INSERT(d, p)   dq_r_insert(DQ_P(d), DQ_P(p))

#endif

A natural example of a circular list is provided by train cars on a circular track. To be specific, consider an airport with four terminals (A, B, C, and D, naturally). An automated system of train cars circulates among the four terminals. Each car remains at each terminal for a small random period of time, and then proceeds on its way. When braking, each car's velocity decreases by one meter per second; when accelerating, velocity increases by one meter per second. A car's stopping distance, then, is a simple function of its velocity:

stop_dist(0)== 0

stop_dist(v) == v + stop_dist(v-1)

(In other words, stop_dist is the sum from 0 to v of v.) A car begins braking whenever the distance to the next station or to the car ahead (whichever is closer) is less than or equal to a conservative stopping distance, based on current velocity plus one. When not at rest in the station, each car is either accelerating or braking.

The station is four cars long, to accommodate this vastly oversimple algorithm; the patrons do not seem to mind that cars tend to jerk viciously, both in the station and on the track. And in any event, the track is 1000 meters long, but the display resolution is only 100 positions, so the car appears to stand still in the station. The track layout of the airport is, by a remarkable coincidence, the exact layout that was provided by the plot_trk function in Section 5.5:

The question of interest is, what is the number of cars which provides the greatest number of station stops per hour?

The program models the cars as nodes on a deque. (The "car" structure is a CAR.) The car just ahead of car p is located via

psucc = (CAR *)DQ_SUCC(p);

This is the program:

run_cars.c:

/* run_cars - simulate airport terminal train cars */

#include "deque.h"

#include "screen.h"

#define TRKSIZE 1000

#define CAR struct car

CAR

{

CAR *left;      /* pointer to previous deque node; !NULL */

CAR *right;     /* pointer to next deque node; !NULL */

short pos;      /* position on track; {0:TRKSIZE-1} */

short vel;      /* velocity; {0:SHORT_MAX} */

char ident[2];  /* identifier for this car; {"a":"z":} */

};

void init();            /* initialize the simulation */

void run();             /* run one time step */

DQ_NODE *cars = NULL;   /* pointer to master deque node */

CAR *p = NULL;          /* pointer to a CAR */

short ncars = 0;        /* number of cars on track; {2:26} */

char idents[] = "abcdefghijklmnopqrstuvwxyz";

main(ac, av)

int ac;

char *av[];

{

short t;            /* loop counter */

SCR_CMDCHAR c;      /* input from keyboard */

if (ac < 2)

error("usage: run_cars #-of-cars", "");

ncars = atoi(av[1]);

if (ncars < 2 || ncars > 26)

error("#-of-cars must be between 2 and 26", "");

scr_open();

scr_clear();

scr_curs(21, 40); scr_putc('A');

scr_curs(11, 75); scr_putc('B');

scr_curs( 1, 40); scr_putc('C');

scr_curs(11,  5); scr_putc('D');

init();

do {

for (t = 0; t < 200; ++t)

run();

scr_curs(scr_lins - 1, 0);

scr_print("More? ");

scr_refresh();

c = scr_getkey();

scr_putc(c);

} while (c == 'y');

scr_close();

exit(SUCCEED);

}

/* init - initialize the simulation */

void init()

{

short i;           /* loop counter; (0:ncars-1} */

DQ_OPEN(&cars);

for (i = 0; i < ncars; ++i)

{

p = (CAR *)malloc(sizeof(CAR));

if (p == NULL)

error("out of space", "");

p->pos = i * (TRKSIZE / ncars);

p->vel = 0;

p->ident[0] = idents[i];

p->ident[1] = '\0';

DQ_APPEND(cars, p);

}

}

/* run - run the simulation for one time step */

void run()

{

short to_station;   /* distance to next station */

short to_car;       /* distance to next car */

short to_stop;      /* safe distance required to stop this car */

short i;            /* loop counter */

CAR *psucc;         /* ptr to successor of car p */

EACH_DQ(cars, p, CAR)

{

plot_trk(p->pos / (TRKSIZE/100), ' ');

p->pos = IMOD(p->pos + p->vel, TRKSIZE);

plot_trk(p->pos / (TRKSIZE/100), p->ident[0]);

to_station = IMOD(p->pos, TRKSIZE / 4);

psucc = (CAR *)DQ_SUCC(cars, p);

to_car = IMOD(psucc->pos - p->pos, TRKSIZE);

for (i = 1, to_stop = 0; i <= p->vel+1; ++i)

to_stop += i;

if (to_car < 10)

p->vel = 0;             /* screeching halt */

else if (to_station <= 5)

p->vel = rand() & 0x1;  /* random jerk in station */

else if (MIN(to_station, to_car) < to_stop && p->vel > 0)

--p->vel;               /* slow down */

else

++p->vel;                  /* speed up */

}

scr_refresh();

}

7.5 Trees

A binary tree is a data structure consisting of branch nodes which contain pointers to sub-trees, and leaf nodes which do not point to other nodes. In the diagram below, there are two branch nodes (b and d), and three leaf nodes (a c, and e).

The top node is known as the root node. A node immediately below a branch node is known as its child; the branch immediately above a node is its parent.

Referring to the small diagram above, if the node names can be taken as indicative of some ordering of the data, then this tree illustrates the use of a tree for storing data in ordered sequence. Every node in a left subtree compares low to the branch above it, and every node in a right subtree compares high.

One common implementation of trees in C uses two pointers in each node to access the left and right subtrees. In the leaf nodes, the pointers are null. The illustration above might look like this in C structures:

According to the packaging conventions of this chapter, we define a TR_NODE ("tree node") like this:

#define TR_NODE struct tr_node

TR_NODE

{

TR_NODE *right;

TR_NODE *left;

/* ... */

};

In the simplest representation, each tree is accessed through a single root pointer. In this simple form, "opening a tree" consists of assigning NULL to its root pointer:

#define TR_OPEN(ppt) (*(ppt) = NULL)

Now we can define the properties of the (binary) tree pointed to by pt:

0.     If pt is NULL, the tree is empty, i.e., has no nodes.

N.     Otherwise, the node *pt is the root node of the tree. The pointers pt->right and pt->left point to trees which are called the subtrees of the node *pt. None of the right and left pointers within either subtree of the root node point back to the root node (i.e., there are no "cycles").

Notice that, unlike the other data structures of this chapter, the tree has a recursive definition. The tree consists of its root node and the two (possibly empty) subtrees. And what does a (non-empty) subtree consist of? Its root node and more subtrees of its own.

As with our other data structures, we will assume that each node being installed in a tree has been allocated unique storage by the calling function, and that we will simply splice its existing storage into the structure of the tree. The simplest (and often the only) place to install a node is as a leaf, either as the left subtree of an existing leaf, or as the right subtree of an existing leaf, or as the first node to be installed in the tree. In all three cases, some pointer which is initially null will be assigned the address of the node being installed; this initially-null pointer is the link-in of the new node. Thus, to accomplish the insertion we need a pointer to a pointer to tree nodes ("pointer to link-in of node"):

TR_NODE **pln;

To get ready for the insertion, pln must be caused to point to the pointer which should receive the address of the new node. Here are the three cases:

Once pln points to the proper insertion point, and the node to be inserted is pointed to by some pointer d, say, the actual insertion is accomplished by

*pln = d;

after which the node pointed to by d has become a leaf of the tree.

After the insertion, the new node is part of the tree, and there is some link-in pointer -- some node's left pointer, some right pointer, or the root pointer itself -- which points to the new node. The link-in is that unique pointer by which the node is made part of the tree, and it will play an important part in the algorithms to follow.

Trees are often processed by recursive algorithms, because their structure naturally lends itself to recursion. For example, given a pointer cmpfn to an ordering function, the tr_insert function involves a simple recursion:

tr_insert:

/* tr_insert - insert a single node in tree (recursive version) */

#include "tree.h"

void tr_insert(plt, pn, cmpfn)

TR_NODE **plt;      /* ptr to link-in of tree or subtree */

TR_NODE *pn;        /* ptr to node to be inserted */

int (*cmpfn)( );    /* ptr to comparison function */

{

TR_NODE *pt;        /* ptr to tree (or subtree) */

int cmp;            /* result of key comparison; neg, zero, or pos */

pt = *plt;          /* pt now pts to current tree (if any) */

if (pt == NULL)     /* if plt pointed to a null pointer, */

{

pn->left = pn->right = NULL;.   /* has no sub trees yet */

*plt = pn;      /* then this is the place to install pn; do so */

}

else if ((cmp = (*cmpfn)(pn, pt)) == 0) /* if already in tree, */

return;                             /* then don't install */

else if (cmp < 0)               /* if node's key compares low */

TR_INSERT(&pt->left, pn, cmpfn);    /* then insert in left tree */

else                            /* otherwise, */

TR_INSERT(&pt->right, pn, cmpfn);   /* insert in right tree */

}

Searching a tree for a match to a specified key is equally simple, in this recursive form.

tr_lfind(#1):

/* tr_lfind - find node matching a specified key

* recursive version

* never returns NULL

*/

#include "tree.h"

TR_NODE **tr_lfind(plt, pn, cmpfn)

TR_NODE **plt;            /* ptr to link-in of tree or subtree */

TR_NODE *pn;              /* ptr to structure containing key to match */

int (*cmpfn)( );          /* ptr to comparison function */

{

TR_NODE *pt;              /* ptr to current tree */

int cmp;                  /* comparison result: neg, zero, pos */

pt = *plt;                /* pt now points to current tree */

if (pt == NULL)           /* if plt points to a null pointer, */

return (plt);         /* the data isn't in the tree */

else if ((cmp = (*cmpfn)(pn, pt)) == 0) /* if key already in tree, */

return (plt);                   /* return its node's link-in */

else if (cmp < 0)                   /* if key compares low, */

return (TR_LFIND(&pt->left, pn, cmpfn));         /* search in left tree */

else                                 /* otherwise, /*

return (TR_LFIND(&pt->right, pn, cmpfn));    /* search in right tree /*

}

The recursion can take a lot of stack space and CPU time, so we will look at an equivalent iterative version:

tr_lfind(#2):

/* tr_lfind - find node matching a specified key

* iterative version

* never returns NULL

*/

#include "tree.h"

TR_NODE **tr_lfind(plt, pn, cmpfn)

TR NODE **plt;          /* ptr to link-in of tree or subtree */

TR_NODE *pn;            /* ptr to structure containing key to match */

int (*cmpfn)( );        /* ptr to comparison function */

{

TR_NODE *pt;            /* ptr to current tree */

int cmp;                /* comparison result: neg, zero, pos */

FOREVER

{

pt = *plt;                  /* pt now points to current tree */

if (pt == NULL)             /* if plt points to a null pointer, */

return (plt);            /* the data isn't in the tree */

else if ((cmp = (*cmpfn)(pn, pt)) == 0) /* if key already in tree, */

return (plt);            /* return its node's link-in */

else if (cmp < 0)           /* if key compares low, */

plt = &pt->left;         /* then search in left tree */

else                        /* otherwise, */

plt = &pt->right;        /* search in right tree */

}

}

A similar function can locate the link-in to the parent of a specified node in the tree:

tr_lpfind:

/* tr_lpfind - find parent of node matching a specified key

* iterative version

* never returns NULL; may return addr of null ptr

*/

#include "tree.h"

static TR_NODE *nullp = NULL;   /* used to return addr of null ptr */

TR NODE **tr_lpfind(plt, pn, cmpfn)

TR NODE **plt;          /* ptr to link-in of tree or subtree */

TR_NODE *pn;            /* ptr to structure containing key to match */

int (*cmpfn)();         /* ptr to comparison function */

{

TR_NODE *pt;            /* ptr to current tree */

TR_NODE **plp;          /* ptr to link-in of parent of tree */

int cmp;                /* comparison result: neg, zero, pos */

plp = &nullp;                   /* root has no parent */

FOREVER

{

pt = *plt;                  /* pt now points to current tree */

if (pt == NULL)             /* if plt points to a null pointer, */

return (&nullp);         /* the data isn't in the tree */

else if ((cmp = (*cmpfn)(pn, pt)) == 0) /* if key already in tree, */

return (plp);            /* return its parent's link-in */

plp = plt;              /* before starting subtree, save plt */

if (cmp < 0)                /* if key compares low, */

plt = &pt->left;         /* then search in left tree */

else                        /* otherwise, */

plt = &pt->right;   /* search in right tree */

}

}

Notice that tr_lfind and tr_lpfind return the link-in, not a pointer to the found node itself. The link-in is more generally useful; for example, it will allow us to detach a node without tracing down from the root.

tr_detach:

/* tr_detach - detach node (or subtree) from tree, given link-in

*/

#include "tree.h"

TR_NODE *tr_detach(pln)

TR_NODE **pln;      /* ptr to link-in of node to be detached */

{

TR_NODE *p;

p = *pln;           /* hold the address of the node */

*pln = NULL;        /* detach the node */

return (p);         /* return its address */

}

The tr_detach function has entirely detached a single node or a whole subtree and handed its address back to the calling function. It is no longer part of the tree, and when the calling function is finished with it, its storage must be freed in some fashion. One easy way is to use a function:

tr_close:

/* tr_close - free a tree or subtree */

#include "tree.h"

void tr_close(plt)

TR_NODE **plt;              /* ptr to link-in of tree */

{

TR_NODE *pt;                /* ptr to root of tree */

pt = *plt;                  /* pt now points to root of tree */

if (pt == NULL)             /* if link-in is NULL, */

return;                 /* nothing to do, so return */

TR_CLOSE(&pt->left);        /* free left subtree */

TR_CLOSE(&pt->right);       /* free right subtree */

free(pt);                   /* free the node itself */

}

Assuming that the tree has been built in some ordered sequence, the "first" node in the tree (the one that compares lowest in the ordering sequence) is the "left-most" node in the tree, i.e., the node that is found by following the left subtree pointers:

tr_lfirst:

/* tr_lfirst - find first node in tree

* Returned value is ptr to link-in of first node

* never returns NULL, and never points to null ptr

*/

#include "tree.h"

TR_NODE **tr_lfirst(plt)

TR_NODE **plt;          /* ptr to (non-null) link-in of tree */

{

TR_NODE **pln;          /* ptr to link-in of node */

for (pln = plt; (*pln)->left != NULL; pln = &(*pln)->left)

;

return (pln);

}

One of the trickiest operations on a sorted tree is to find the successor of a given node, i.e., the node which follows it in sorted sequence. There are several cases. If the node has a right subtree, the successor is the "first" node in the right subtree. The successor of a is b, in this diagram:

If the node has no right subtree, and the node is the left subnode of its parent, then its successor is its parent node. (In the diagram, the successor of c is d.)

Finally, if neither of the above cases applies, the node has no successor; it is the last node in the tree.

tr_lnext:

/* tr_lnext - find the successor of node pn

* Returned value is ptr to link-in of successor node

* Never returns NULL; may return addr of a null ptr

*/

#include "tree.h"

static TR_NODE *nullp = NULL;   /* used to return addr of null ptr */

TR_NODE **tr_lnext(plt, pn, cmpfn)

TR_NODE **plt;          /* ptr to link-in of tree */

TR_NODE *pn;            /* ptr to specific node of tree */

int (*cmpfn)( );        /* ptr to comparison function */

{

TR_NODE **pls;          /* ptr to link-in of successor */

TR_NODE **pln;          /* ptr to link-in of a TR_NODE */

if (pn->right != NULL)  /* if node has a right subtree, */

{ /* return ptr to link-in of "leftmost" node in right subtree */

return (TR_LFIRST(&pn->right));

}

else                       /* if node has no right subtree, */

{       /* return ptr to link-in of parent of pn */

pln = TR_LPFIND(plt, pn, cmpfn);

if (*pln != NULL && (*pln)->left == pn)

return (pln);       /* node is left subtree of parent */

else

return (&nullp);    /* node is rightmost in tree */

}

}

We can use TR_LFIRST and TR_LNEXT to construct an EACH_TR macro:

#define EACH_TR(plt, p, t, fn) \

for (p = (t *)*TR_LFIRST(plt); (p) != NULL; p = (t *)*TR_LNEXT(plt, p, fn))

(The tr_lnext function is rather slow for this purpose. See the exercises below for suggestions for faster methods.)

The final operation that we consider is the deletion of a node in an ordered tree. Simply detaching the node is insufficient; we must be sure that the remaining tree is still ordered. In all cases, we assume that we have a pointer, pln, to the link-in of the node to be deleted, possibly provided by tr_lfind. Three cases must be considered. (1) If the node being deleted is a leaf, we simply detach it which sets its link-in to NULL. The detached node, having no children, is then freed. The deletion of b fits this case:

(2) If the node has only one child, we will set the node's link-in to point to the child node, and then free the node itself, as shown by the deletion of c:

(3) If the node has two children, we need to locate its successor. As we saw earlier, since the node has a right subtree, its successor is the "left most" node in that right subtree. Therefore, the successor itself cannot have a left subtree, so the successor's place can be taken by its right subnode, and the successor itself moved into the place of the deleted node. After all this, the node can be freed. This is how a would be deleted:

Expressed in C, here is what it looks like:

tr_delete:

/* tr_delete - delete the node **pln from tree *plt

*/

#include "tree.h"

void tr_delete(plt, pln, cmpfn)

TR_NODE **plt;          /* ptr to link-in of tree */

TR_NODE **pln;          /* ptr to link-in of node */

int (*cmpfn)();         /* ptr to comparison function */

{

TR_NODE *pn;            /* ptr to specific node of tree */

TR_NODE *ps;            /* ptr to node's successor */

TR_NODE **pls;          /* ptr to link-in of successor */

pn = *pln;                  /* pn pts to the node to delete */

if (pn->right == NULL)      /* if node has no right subtree, */

{

if (pn->left == NULL)   /* if node has no children, */

*pln = NULL;         /* replacement for pn is NULL */

else                    /* if node has L subtree, but not R, */

*pln = pn->left;     /* replacement is pn's L subtree */

}

else if (pn->left == NULL)       /* if node has R subtree, but not L */

*pln = pn->right;        /* replacement is pn's R subtree */

else                        /* if node has R and L subtrees *

{

pls = TR_LNEXT(plt, pn, cmpfn); /* get ptr to link-in of succ */

ps = *pls;              /* ps now points to successor */

*pls = ps->right;       /* succ's R subtree takes succ's place */

ps->right = pn->right;      /* succ acquires node's R ... */

ps->left = pn->left;    /* ... and L subtree */

*pln = ps;              /* replacement is successor */

}

free(pn);

}

The tree.h header summarizes the tree functions from this section:

tree.h:

/* tree.h - header for tree functions

*/

#ifndef TREE_H

#define TREE_H

#include "local.h"

#include "pointer.h"

#define TR_NODE struct tr_node

TR_NODE

{

TR_NODE *right;

TR_NODE *left;

/* ... */

};

void tr_close();        /* PARMS(TR_NODE **plt) */

void tr_delete();       /* PARMS(TR_NODE **plt, TR_NODE **pln, int (*cmpfn)()) */

TR_NODE *tr_detach();   /* PARMS(TR_NODE **pln) */

void tr_insert();       /* PARMS(TR_NODE **plt, TR_NODE *pn, int (*cmpfn)()) */

TR_NODE **tr_lfind();   /* PARMS(TR_NODE **plt, TR_NODE *pn, int (*cmpfn)()) */

TR_NODE **tr_lfirst();  /* PARMS(TR_NODE **plt) */

TR_NODE **tr_lpfind();  /* PARMS(TR_NODE **plt, TR_NODE *pn, int (*cmpfn)()) */

TR_NODE **tr_lnext();   /* PARMS(TR_NODE **plt, TR_NODE *pn, int (*cmpfn)()) */

#define TR_RIGHT(p)     ((p)->right)

#define TR_LEFT(p)      ((p)->left)

#define TR_P(p)         PNNC(p, TR_NODE *)

#define TR_PP(p)        PNNC(p, TR_NODE **)

#define TR_CLOSE(plt)           tr_close(TR_PP(plt))

#define TR_DELETE(plt, pln, fn) tr_delete(TR_PP(plt), TR_PP(pln), fn)

#define TR_DETACH(pln)          tr_detach(TR_PP(pln))

#define TR_INSERT(plt, pn, fn)  tr_insert(TR_PP(plt), TR_P(pn), fn)

#define TR_LFIND(plt, pn, fn)   tr_lfind(TR_PP(plt), TR_P(pn), fn)

#define TR_LFIRST(plt)          tr_lfirst(TR_PP(plt))

#define TR_LPFIND(plt, pn, fn)  tr_lpfind(TR_PP(plt), TR_P(pn), fn)

#define TR_LNEXT(plt, pn, fn)   tr_lnext(TR_PP(plt), TR_P(pn), fn)

/* true macros */

#define TR_OPEN(plt)            (*(plt) = NULL)

#define TR_FIRST(plt)           (*TR_LFIRST(plt))

#define TR_FIND(plt, pn, fn)    (*TR_LFIND(plt, pn, fn))

#define TR_NEXT(plt, pn, fn)    (*TR_LNEXT(plt, pn, fn))

#define EACH_TR(plt, p, t, fn) \

for (p = (t *)*TR_LFIRST(plt); (p) != NULL; p = (t *)*TR_LNEXT(plt, p, fn))

#endif

Exercise 7-1. If a revised version of tr_lfirst and tr_lnext cooperated to create a traversal stack, the time to perform EACH_TR could be drastically reduced. Define a "visit state" for each node: 1 means its left subtree is being visited, 2 means that the node itself is being visited, 3 means that the right subtree is being visited. Here are the five states of the traversal stack while traversing the following tree:

We now have all the functions that we would need to maintain an in-memory "database" of PARTS records, suitable for use with the "parts menu" program from the previous chapter.

part_db_tr.c:

/* part_db_tr.c - simulated database access to parts

* uses binary tree for in-memory storage

*/

#include "tree.h"

*include "part.h"

#define TPART struct tr_part

TPART

{

TPART *right;

TPART *left;

PART part;

};

static TPART tpart = {0};

static TPART *parts = NULL;

/* db_cmp_part - comparison function for database access */

static int db_cmp_part(p1, p2)

TPART *p1, *p2;

{

return (strcmp(p1->part.part_no, p2->part.part_no));

}

/* (more) */

/* db_add_part - add a part record to database */

bool db_add_part(partp)

PART *partp;

{

TPART *p;

p = (TPART *)malloc(sizeof(TPART));

if (p == NULL)

{

remark("Part storage is full", "");

return (NO);

}

STRUCTASST(p->part, *partp);

TR_INSERT(&parts, p, db_cmp_part);

return (YES);

}

/* db_find_part - find a part record in database */

bool db_find_part(partp)

PART *partp;

{

TPART *pn;      /* ptr to part record in tree */

STRUCTASST(tpart.part, *partp);

pn = (TPART *)TR_FIND(&parts, &tpart, db_cmp_part);

if (pn == NULL)

return (NO);

STRUCTASST(*partp, pn->part);

return (YES);

}

/* db_del_part - delete a part record in database */

bool db_del_part(partp)

PART *partp;

{

TPART **pln;        /* ptr to link-in of part record in tree */

STRUCTASST(tpart.part, *partp);

pln = (TPART **)TR_LFIND(&parts, &tpart, db_cmp_part);

if (*pln == NULL)

{

remark("No record for this part", "");

return (NO);

}

TR_DELETE(&parts, pln, db_cmp_part);

return (YES);

}

/* db_chg_part - change a part record in database */

bool db_chg_part(partp)

PART *partp;

{

TPART *pn; /* ptr to part record in tree */

STRUCTASST(tpart.part, *partp);

pn = (TPART *)TR_FIND(&parts, &tpart, db_cmp_part);

if (pn == NULL)     /* validation done in menu should prevent */

return (NO);    /* non-existent part from reaching here */

STRUCTASST(pn->part, *partp);

return (YES);

}

/* (more) */

/* db_open_part - open a part record database */

void db_open_part()

{

TR_OPEN(&parts);

}

/* db_close_part - close the part record database */

void db_close_part()

{

TR_CLOSE(&parts);

}

Go to Chapter 8    Return to Table of Contents


Reliable Data Structures: CHAPTER 8: FILES

 Return to Table of Contents

CHAPTER 8: FILES

8.1 Standard (Default) Files

One of the virtues that C inherited from UNIX systems is a uniform concept of data transfer through files. A file, to a C program, is a transmission stream of bytes to or from the main memory. The ultimate source or sink for the bytes may be of many forms: disk file, tape, printer, communication channel to other programs, etc. With a few exceptions, the C program can be written independent of the ultimate source or sink

One important distinction about files is present in most operating systems except for UNIX: the distinction between binary files and text files.

A binary file is a one-for-one transmission stream of bytes. If a program writes a binary file onto an external medium, that file will read byte-for-byte the same when read back in as a binary file (provided that the same compiler and machine are used both times). However, some implementations may append extra nul characters to conform to binary block-size conventions.

A text file is a mapping between internal machine bytes and external conventions for representation of text. For example, some implementations map the newline ('\n') into a two-character sequence (carriage-return, newline). Some append a ctrl-Z character to mark the end of file. Some systems delete some nonprinting characters from text files.

On UNIX systems, of course, there are no such distinctions. On other systems, binary files might seem simpler to use and understand, but on such systems most of the utility programs that deal with text will expect to see files that conform to the text-file conventions. Therefore, on non-UNIX systems, the default type for files will generally be the text file.

The most portable scheme for accessing files originated with UNIX 7th Edition. Each file is represented by a FILE structure, whose contents are specified by the header <stdio.h>. (A FILE object is, strictly speaking, not required to be a C structure, but most implementations do define it that way.) Such files have generally been implemented with buffers associated with the structure, so they have sometimes been referred to as "buffered" files. Since each operating system may have its own treatment of I/O buffers, we will follow ANSI C in adopting the more neutral term stream file to designate the type of file represented by a FILE structure.

8.2 Processing Named Files

To perform I/O on files opened by name, we use the function

fopen:

FILE *fp;

fp = fopen("input.dat", "r");

will open the file input.dat for reading ("r"). The variable fp was declared as a pointer to FILE. After the fopen call, fp will point to a FILE properly initialized for reading. If the open fails, fp will contain a NULL pointer; hence this rule:

Rule 8-1: Always test the returned value from fopen to be sure that the open succeeded.

The reasons why an open might fail are implementation-defined; some possible reasons are that no file exists by the specified name, or the file may exist but be unreadable to the program, or the program has already opened the maximum allowable number of files.

The file-name argument must specify a name in a format acceptable to the host system, and systems differ in their naming rules for files. In general, a name in the form xxxxxx.xxx, where x is a letter or a digit, will be acceptable to most C environments. Name formats such as /tmp/t11111, a:infile, or [70,40]datfil will work only on some systems.

Note that the environment may be allowed some liberty in mapping the name string into a system's format for file names, for example by mapping lower case into upper case, or by mapping the "'dot" (.) into some other character.

Note also that the file name may specify a device (terminal, printer, tape drive, etc.) if the host system provides the appropriate name mapping. However, there are no standard names for any devices, so such usage is potentially non-portable.

A file thus opened for reading can be used with functions analogous to the default-input functions shown earlier:

Named-file function  Default-file function

getc(fp)              getchar()

fscanf(fp, fmt, ...)  scanf(fmt, ...)

fgets(s, n, fp)       gets(s)

Note, however, that fgets (unlike getc and fscanf) is not functionally equivalent to its default-file version: fgets takes an argument specifying the size of the available storage in s, and fgets does not delete the newline from the input characters (whereas gets does delete the newline).

A file can be opened for output by specifying "w" as its usage. If the file existed previously, it will be truncated to zero length. If it did not exist, it is created with zero length. Output functions analogous to the default-file versions are available:

Named-file function   Default-file function

putc(c, fp)            putchar(c)

fprintf(fp, fmt, ...)  printf(fmt, ...)

fputs(s, fp)           puts(s)

As with gets, there is a functional difference between fputs and puts: fputs does not tack a newline onto its output, whereas puts does.

A third option available from fopen is "a" ("append") mode. Theprevious contents of the file (if any) are left undisturbed; each write operation will take place at the end of the file.

Besides the functions above which have default-file analogs, there are a few functions available only in their named-file versions, notably fread, and fwrite. The call

n = fread(s, elt_size, num_elts, fp);

will read a designated number of elements (each of a designated size) into the object pointed to by s, using the file specified by fp. The returned value tells how many elements were successfully read. A return of zero is ambiguous; it might indicate end-of-file, or it might reflect the occurrence of an I/O error before any elements were read.

A call upon feof(fp) will return YES if the file fp is at end-of-file, and NO otherwise. Similarly, ferror(fp) will tell whether any I/O errors have taken place on the file. A program can call clearerr(fp) to clear both the end-of-file and the I/O error indicators for the file.

When fread is called upon to read an array of char's, with elt_size specified as 1, the usage is portable onto any system. If the elements are multi-byte data, the byte-ordering is environment-dependent. Similarly for writing files with fwrite:

n = fwrite(s, elt_size, num_elts, fp);

will write a designated number of elements (each of a designated size) onto the file specified by fp.

When the program is finished with a file, the file should be closed via fclose(fp).

As mentioned in Section 5.2, the buffering behavior of a file can be changed by calling the setbuf function after opening the file but prior to any file accesses. Calling setbuf(fp, NULL) turns off the buffering; calling setbuf(fp, mybuf) causes the array mybuf (which should be of size BUFSIZ) to be used as the buffer for file fp. One use for the latter option might be an environment in which dynamic "heap" memory is limited; a statically-declared buffer can be used instead of one that would be dynamically allocated by the I/O functions.

8.3 Direct Access and Fseek

If a stream file is accessing a "seekable" medium (such as a disk or diskette), a certain measure of direct access is provided by the functions fseek and ftell. For now, we will continue to discuss "text" files, which are the type of file that is specified by the "r", "w", and "a" modes of fopen. The number of char's actually read by a C program, or the number of char's written by the program, may be different from the number of bytes recorded in the file (in a non-UNIX environment). For example, some systems record two bytes (carriage-return, newline) at the end of each text line, but the C program will only see the '\n' (newline) character.

During the reading or writing of a file, the program can call

position = ftell(fp);

which sets position to a long integer that represents the current position of the file. At some later time, the program could call

success = fseek(fp, position, SEEK_SET);

which would re-position the file to the same place where the original ftell call was performed. A success return of zero indicates no errors; anything else is an error. (The name SEEK_SET is defined in various places in recent C libraries: <unistd.h> for systems conforming to the /usr/group standard, <stdio.h> and also <stddef.h> according to draft ANSI C. If neither definition is available, put one in your stddef.h header. The conventional definition for SEEK_SET is zero.)

It is possible to test whether a file is on a "seekable" device by using

seekable = (fseek(fp, 0L, SEEK_CUR) == 0);

where SEEK_CUR is conventionally equal to l. This call of fseek specifies a displacement of zero bytes relative to the current position. In other words, it has no effect except to test the returned value from fseek. (If fp turns out not to specify a seekable file, its error indicator will be set as a result of the call to fseek. Invoking clearerr(fp) will clear the error indicator.)

A text file can be positioned to the end of the file, by calling

success = fseek(fp, 0L, SEEK_END);

where SEEK_END is conventionally equal to 2.

And a file can be re-positioned to the very beginning by calling

success = fseek(fp, 0L, SEEK_SET);

Because of the text-mapping problem, these four usages of fseek are the only ones that can be portably used on text files.

It is possible to fopen a file for both reading and writing, but because of the text-mapping problem, these read/write modes are difficult to use on text files. It is time to discuss binary I/O.

8.4 Binary l/O

In the library proposed for ANSI C, a trailing b can be appended to the fopen mode string to indicate a binary file. For example,

fopen("payroll.dat", " rb" )

would open the binary file payroll.dat for reading. Here, then, is the complete list of mode options for fopen:

Text  Binary  Action

"r"   "rb"   Open for reading

"w"   "wb"   Truncate or create for writing

"a"   "ab"   Open or create for writing at end

"r+"  "r+b"  Open for reading and writing

"w+"  "w+b"  Truncate or create for reading and writing

"a+"  "a+b"  Open or create for reading and writing at end

The "reading and writing" modes are most generally useful in binary mode, since over-writing a text file can give unpredictable results; the new text may take more or less space than the previous file contents. In binary mode, however, the number of bytes written into the file is exactly the number of characters specified by the program. (Some environments may provide binary treatment for r+, w+, and a+ modes even without the trailing b on the mode.)

We could, for example, treat a binary file as consisting of a number of fixed-size records, each of which can be read, modified in some way, and then written back into its previous space:

FILE *fp;

fp = fopen("records.dat", "r+b");

if (fp == NULL)

error("can' t open", "records.dat");

/* ... */

if (fseek(fp, (long)recno * recsize, SEEK_SET) != 0) /* seek the record */

can't find specified record

if (fread(rec, recsize, 1, fp) != 1)  /* read the record */

can't read the record

modify the record contents

if (fseek(fp, (long)recno * recsize, SEEK_SET) != 0) /* seek record again */

can't find record

if (fwrite(rec, recsize, 1, fp) != 1)  /* re-write the record */

can't rewrite record

To allow reasonably efficient treatment of buffered files, it is required that an fseek must be performed when switching from reading to writing, or vice versa.

There are some problems involved with this form of binary I/O. For one, many versions of fread and fwrite do not perform very efficiently when used in this manner. And many existing compilers will complain about the b in a mode string. For these reasons, the next section will present a different approach to binary I/O.

8.5 Record I/O

Most existing implementations of C provide another set of functions for performing record-oriented I/O, namely the functions open, close, read, write, and lseek. (These functions are sometimes known as the "file-descriptor library," because they refer to files by numbers known as file descriptors, rather than by the "pointer to FILE" mechanism provided for stream files.) In many implementations they are the most efficient functions for the purpose. They are, however, implemented with a variety of system-dependent options and behaviors, and they will probably not be part of the ANSI C standard. If we restrict ourselves to the problem of doing record-oriented I/O on seekable devices like disks, we can still make effective use of this type of functions.

Our approach will be to provide a header to give access to a set of functions to be known as bin_open, bin_close, bin_read, bin_write, and bin_lseek. On systems with direct equivalents in the underlying library, these names can simply be mapped into the corresponding library name. On other systems, some new C functions must be provided, possibly making use of the (ANSI-style) stream library. Thus, the actual implementation of bin_open will be system-dependent, but the program that calls bin_open will itself be portable [8-1].

We must define a new header, bin_io.h, which will provide the necessary mappings. This version of the bin_io.h header will work on MS-DOS (with Lattice), Idris, and UNIX systems:

bin_io.h:

/* bin_io.h - header for binary file I/O functions

* SYSTEM DEPENDENT - MUST BE CONFIGURED FOR EACH TARGET SYSTEM

*/

#ifndef BIN_IO_H

#define BIN_IO_H

#include "local.h"

#include "fcntl.h"    /* provide your own if not standard on system */

typedef int bin_fd;   /* "binary file descriptor" {0:BIN_NFILE-1} */

#define BIN_NFILE 20  /* adjust to local system */

#define O_RWMODE (O_RDONLY|O_WRONLY|O_RDWR) /* uses symbols from fcntl.h */

#ifndef IDRIS

#define bin_open(s, m)      open(s, m)

#endif

#define bin_close(f)        close(f)

#define bin_lseek(f, o, w)  lseek(f, o, w)

#define bin_read(f, b, n)   read(f, b, n)

#define bin_write(f, b, n)  write(f, b, n)

#endif

The header bin_io.h includes another header via

#include "fcntl.h"

Many systems already have this header in one of the standard places for headers. If yours does not, you will need to create one that is compatible with the open options on your system, using this model:

fcntl.h:

/* fcntl.h - definitions for binary open

* Compatible with UNIX Sys V, ...

*/

#ifndef FCNTL_H

#define FCNTL_H

#define O_RDONLY  0      /* delete or change to conform to local */

#define O_WRONLY  1      /* delete or change to conform to local */

#define O_RDWR    2      /* delete or change to conform to local */

#define O_NDELAY  4      /* NOT USED BY bin_io FUNCTIONS */

#define O_APPEND  8      /* delete or change to conform to local */

#define O_CREAT   0x100  /* delete or change to conform to local */

#define O_TRUNC   0x200  /* delete or change to conform to local */

#define O_EXCL    0x400  /* delete or change to conform to local */

#endif

Each of the binary I/O functions sets the global variable errno to a non- zero value if an error is encountered, and then returns an error indication in the returned value. Here are the descriptions of the binary I/O functions:

int bin_open(fname, mode)

char fname[ ];  /* : string */

int mode;       /* as described in fcntl.h */

opens the file specified by the string fname, according to the specified mode (which is OR'ed together from "fcntl.h " options). The returned value is an int file descriptor if successful; otherwise, -l is returned. For the uses of this chapter, the file should be a seekable disk file.

int bin_close(fd)

int fd;      /* : {0:BIN_NFILE-1} */

closes the file associated with file descriptor fd. The returned value is zero if successful, otherwise -l.

int bin_read(fd, s, n)

int fd;      /* : {0:BIN_NFILE-1} */

dats_ptr s;  /* : !NULL */

int n;       /* : {0:INT_MAX} */

reads n bytes into the char array s from the file specified by fd. The returned value is the number of bytes read, or -l for an I/O error, or 0 for end-of-file.

int bin_write(fd, s, n)

int fd;      /* : {0:BIN_NFILE-1} */

data_ptr s;  /* : !NULL */

int n;       /* : {0:INT_MAX} */

writes n bytes from the char array s onto the file specified by fd. The returned value is the number of bytes written; anything less than n indicates an error.

long bin_lseek(fd, n, whence)

int fd;      /* : {0:BIN_NFILE-01} */

long n;

int whence;  /* : {SEEK_SET, SEEK_CUR, SEEK_END} */

seeks n bytes relative to beginning-of-file, current position, or end-of-file, according to this scheme:

SEEK_SET  relative to beginning-of-file  (usually 0)

SEEK_CUR  relative to current position   (usually 1)

SEEK_END  relative to end-of-file        (usually 2)

The returned value is the new file position, relative to beginning-of-file, or -1 if any error occurred. Seeking past end-of-file is not portable, and gives different results on different systems. Note that, unlike the stream file functions, the binary I/O functions allow alternation between reading and writing with no intervening seeks.

Now we need to take note of an important problem about binary I/O on some systems. Sometimes, binary files are padded with nul characters at the end of the file, to make the file conform to a local fixed block-size. If we want to have record files that can grow at the end, the only portable approach is to record the file size at the start of the file. To keep this detail out of the application program, and to make access to fixed-size records easier, we can make up a second level of functions, the "record I/O" functions.

The header rec_io.h defines a rec_fd as an int. (In early printings of this book, it was a short integer, but int gives more consistent calling sequences for these functions.) The following functions are provided for record I/O:

rec_fd rec_open(fname, mode, recsize)

char fname[ ];  /* : string */

int mode;       /* as specified in fcntl.h */

int recsize;    /* : {0:SHORT_MAX} */

opens the file named by fname for mode (as in fcntl.h), with the record size specified by recsize. Note that the record size is limited to SHORT_MAX, for maximum portability of the functions.

int rec_close(fd)

rec_fd fd;      /* : {0:REC_NFILE-1} */

closes the file specified by fd. The return values are 0 for success and -l for errors.

bool rec_get(fd, rec, recno)

rec_fd fd;     /* : {0:REC_NFILE-1} */

data_ptr rec;  /* : !NULL */

long recno;    /* : {REC_NEXT,0:rec_nrecs(fd)-1} */

reads from fd into rec the record whose record number is recno. If recno is specified to be REC_NEXT, the next sequential record is read. The return is YES for success and NO for failure. More detailed (but system-dependent) error information is available in errno.

bool rec_put(fd, rec, recno)

rec_fd fd;     /* : {0:REC_NFILE-1} */

data_ptr rec;  /* : !NULL */

long recno;    /* : {REC_W_END,REC_NEXT,0:rec_nrecs(fd)} */

writes onto file fd the contents of rec in the position specified by recno. If recno is specified as REC_NEXT, the record is written in the current file position; if it is specified as REC_W_END, the record is appended at the end of the file, thus extending the file by one record. The return is YES if success, NO if failure.

long rec_nrecs(fd)

rec_fd fd;  /* : {0:REC_NFILE-1} */

returns the current number of records in file fd.

int rec_recsize(fd)

rec_fd fd;  /* : {0:REC_NFILE-1} */

returns the record size associated with file fd.

Here are the listings of the header and the functions in the rec_io package.

rec_io.h:

/* rec_io.h - header for record file I/O functions */

#ifndef REC_IO_H

#define REC_IO_H

#include "local.h"

#include "bin_io.h"

#define REC_NFILE      BIN_NFILE  /*  adjust to local system */

#define REC_NEXT       (-1L)

#define REC_W_END      (-2L)

#define REC_NOT_FOUND  (-3L)  /* codes for hashed file access */

#define REC_FULL       (-4L)  /* codes for hashed file access */

#define REC_ERROR      (-5L)  /* codes for hashed file access */

#define REC_AVAIL      'a'    /* codes for hashed file access */

#define REC_OCCUP      'o'    /* codes for hashed file access */

#define REC_DELET      'd'    /* codes for hashed file access */

#define REC_FILE struct rec_file

REC_FILE

{

short _byte0;    /* offset of the first data byte in file */

short _recsize;  /* size (in bytes) of each record */

long_nrecs;      /* size (in records) of current file */

bits_status;     /* open-modes of file */

};

#define REC_BYTE0 BUFSIZ

#define rec_nrecs(fd)    (rec_files[fd]._nrecs + 0)

#define rec_recsize(fd)  (rec_files[fd]._recsize + 0)

typedef int rec_fd;

extern int errno;

extern REC_FILE rec_files[REC_NFILE];

int rec_close();     /* PARMS(rec_fd fd) */

rec_fd rec_open();   /* PARMS(char fname[], int type, int recsize) */

bool rec_get();      /* PARMS(rec_fd fd, data_ptr buf, long i) */

bool rec_put();      /* PARMS(rec_fd fd, data_ptr buf, long i) */

long rec_hfind();

/* PARMS(rec fd fd, data_ptr keybuf, data_ptr buf, int (*cmp)(), long (*hash)()) */

long rec_havail();

/* PARMS(rec_fd fd, data_ptr keybuf, data_ptr buf, long (*hash)()) */

#endif

The "record file descriptor" (rec_fd) is used as an index into the array rec_files. The rec_open function gets a file descriptor from bin_open. If the file existed previously, its REC_FILE information is read. Otherwise, the REC_FILE information is written out at the head of the new file.

rec_open:

/* rec_open- open a record-binary file */

#include "rec_io.h"

REC_FILE rec_files[REC_NFILE] = {0};

rec_fd rec_open(fname, type, recsize)

char fname[ ];

int type;

int recsize;

{

rec_fd fd;

REC_FILE *rfp;          /* pointer to a REC_FILE entry */

int old_errno = errno;  /* save system error indicator */

short i;                /* counter to clear first record */

errno = 0;              /* clear error indicator */

fd = bin_open(fname, type);

if (fd < 0 || fd >= REC_NFILE)  /* validate fd */

return (-1);

rfp = &rec_files[fd];

bin_lseek(fd, (long) 0, SEEK_SET);  /*seek to initial byte of file */

if ((type & O_RWMODE) == O_WRONLY || /* new file: opened write-only, or */

bin_read(fd, rfp, sizeof(*rfp)) != sizeof(*rfp))  /* can't be read */

{

bin_lseek(fd, (long)0, SEEK_SET);

rfp->_byte0 = REC_BYTE0;

rfp->_recsize = recsize;

rfp->_nrecs = 0;

bin_write(fd, rfp, sizeof(*rfp));

for (i = 1; i -< REC_BYTE0 - sizeof(*rfp); ++i)

bin_write(fd, "\0", 1);

}

else if (recsize != rfp->_recsize) /* file exists, but bad recsize */

{

bin_close(fd);

return (-1);

}

if (errno == 0)

{

errno = old_errno;    /* restore saved value */

bin_lseek(fd, (long)rfp->_byte0, 0);

rfp->_status = type;  /* save the open-mode of file */

return (fd);

}

else /* error was returned from some bin_io function */

{

bin_close(fd);

return (-1);

}

}

rec_close:

/* rec_close - close the REC_FILE fd */

#include "rec_io.h"

int rec_close(fd)

rec_fd fd;

{

REC_FILE *rfp;

int old_errno = errno;

errno = 0;

if (fd < 0 || fd >= REC_NFILE)    /* validate fd */

return (-1);

rfp = &rec_files[fd];

if ((rfp->_status & O_RWMODE) == O_RDONLY)

return (bin_close(fd));        /* if read-only, all done */

bin_lseek(fd, (long)0, SEEK_SET);

bin_write(fd, rfp, sizeof(*rfp));  /* write new REC_FILE info */

bin_close(fd);

if (errno == 0)

{

errno = old_errno;

return (0);

}

else

{

return (-1);

}

}

If the file was opened read-only, rec_close needs only to close the file. Otherwise, the rec_close function has to write the REC_FILE information back to the disk. If any errors took place in the low-level I/O functions, rec_close returns -1.

rec_get:

/* rec_get - read one record from record-binary file */

#include "rec_io.h"

bool rec_get(fd, buf, recno)

rec_fd fd;     /* which file */

data_ptr buf;  /* where to put the data */

long recno;    /* {REC_NEXT,0:rec_nrecs(fd)-1} */

{

REC_FILE *rfp;

short n;  /* size of each record */

if (fd < 0 || fd >= REC_NFILE)  /* validate fd */

return (NO);

rfp = &rec_files[fd];

n = rfp->_recsize;

if (recno != REC_NEXT && recno < 0 || recno >= rfp->_nrecs)

return (NO);

else if (recno == REC_NEXT)

;  /* no seek, ready for next record */

else if (bin_lseek(fd, rfp->_byte0 + recno * n, SEEK_SET) < 0)

return (NO);

return (bin_read(fd, buf, n) == n);

}

Aside from the necessary validation of fd and recno, all that rec_get needs to do is seek to the proper byte and read the record. If either the seek or the read fails, rec_get returns NO.

rec_put:

/* rec_put - write one record to record-binary file

*/

#include "rec_io.h"

bool rec_put(fd, buf, recno)

rec_fd fd;

data_ptr buf;  /* where to get the data */

long recno;  /* {REC_W_END,REC_NEXT,0:rec_nrecs(fd)} */

{

REC_FILE *rfp;

short n;  /* size of each record */

if (fd < 0 || fd >= REC_NFILE)

return (NO);

rfp = &rec_files[fd];

n = rfp->_recsize;

if (recno != REC_NEXT && recno != REC_W_END && recno < 0 ||

recno > rfp->_nrecs)

return (NO);

else if (recno == REC_W_END || recno == rfp->_nrecs)

{

recno = rfp->_nrecs;  /* block will be added at end */

++rfp->_nrecs;  /* extend the file size */

}

if (bin_lseek(fd, rfp->_byte0 + recno * n, SEEK_SET) < 0)

return (NO);

return (bin_write(fd, buf, n) == n);

}

The parameter recno can be a proper record number, or the coded value REC_W_END (requesting to write onto the end of the file), or the coded value REC_NEXT (requesting to write the next sequential record). After validating fd and recno, rec_put seeks to the appropriate record and writes the new record. The return is YES if success, NO if failure.

The following demonstration program (rec_main) writes a record file consisting of ten records, each containing one long integer. Then it reads the entire file sequentially and prints each number to the standard output. Next it opens the file for reading and writing, and writes new data into the records, using direct access by record numbers. Finally it reads the file with direct access by record numbers and prints each number to the standard output.

rec_main.c:

/* rec_main - simple test harness for rec_io */

#include "local.h"

#include "rec_io.h"

main( )

{

long lnum;

long rec_no;

rec_fd rfd;

rfd = rec_open("lnum.dat", O_WRONLY | O_CREAT | O_TRUNC, sizeof(lnum));

if (rfd < 0)

error("can't open (output)", "lnum.dat");

for (lnum = 0; lnum <= 9; ++lnum)

if (!rec_get(rfd, (data_ptr)&lnum, REC_W_END))

error("rec_put (END) error", "");

rec_close(rfd);

rfd = rec_open("lnum.dat", O_RDONLY, sizeof(lnum));

if (rfd < 0)

error("can't open (input)", "lnum.dat");

while (rec_get(rfd, (data_ptr)&lnum, REC_NEXT))

printf("%4ld\n", lnum);

rec_close(rfd);

rfd = rec_open("lnum.dat", O_RDWR, sizeof(lnum)):

if (rfd < 0)

error("can't open (update-output)", "lnum.dat");

for (lnum = 109; lnum >= 100; --lnum)

if (!rec_put(rfd, (data_ptr)&lnum, lnum - 100))

error("rec_put (direct) error", "");

rec_close(rfd);

rfd = rec_open("lnum.dat", O_RDWR, sizeof(lnum));

if (rfd < 0)

error("can't open (update-input)", "lnum.dat");

for (rec_no = 0; rec_no <= 9; ++rec_no)

{

if (!rec_get(rfd, (data_ptr)&lnum, rec_no))

error("rec_get (direct) error", "");

else if (lnum != rec_no + 100)

error("bad data on re-read", "");

else

printf("%4ld %4ld\n", rec_no, lnum);

}

rec_close(rfd);

exit(SUCCEED);

}

8.6 A Record File with Hashed Access

The simple scheme of integer-numbered records is convenient only when there is an easy mapping from the data to be stored and an integer record number. Very few databases are keyed with such simple integer keys. One way to use record-numbered files with arbitrary keys is to "hash" each key into an integer:

for (p = part.part_no; *p != '\0'; ++p)

sum += *p;

rec_ no = IMOD(sum, rec_nrecs(fd));

Here the bytes of a "part-number" string are simply being added together to form a "hash-sum." Taking the remainder upon division by the number of records in the file produces a record-number specifying the place to start looking for this particular record.

Each record on the file contains an extra byte to indicate its status. The possible statuses are "available," "occupied," or "deleted."

When creating a hashed file for the first time, all the records must be marked as "available." The program crhash ("create hash file") will create a file of a specified number of records of specified size.

crhash.c:

/* crhash -- create and initialize an empty hash file

* used to hold part database records

*/

#include "local.h"

#include "part.h"

#include "rec_io.h"

main(ac, av)

int ac;

char *av[];

{

rec_fd fd;   /* record file descriptor */

short size;  /* size of one record in created file */

long nrecs;  /* maximum numbers of records in created file */

char *buf;   /* data to init. each record */

long i;      /* counter to write records in created file */

if (ac != 4)

error("usage: crhash filename nrecs recsize", "");

nrecs = atol(av[2]);

size = atoi(av[3]);

fd = rec_open(av[1], O_RDONLY, size);

if (fd > 0)

{

rec_close(fd);

error("file already exists:", av[1]);

}

fd = rec_open(av[1], O_RDWR|O_CREAT, size);

if (fd < 0)

error("can't create file", av[1]);

buf = (char *)calloc(size, 1);

if (buf ==NULL)

error("Need more heap space to allow size =", av[3]);

*buf = REC_AVAIL;

/* file was created ok, initialize file */

for (i = 0; i < nrecs; ++i)

{

if (!rec_put(fd, buf, REC_W_END))

error("I/O error writing new file", "");

}

rec_close(fd);

exit (SUCCEED );

}

The function rec_hfind starts reading at a specified record-number. If the record contains the desired key, the search is finished. Otherwise, successive records are read (wrapping from end-of-file to beginning-of-file if necessary) until the desired record is found or an unoccupied record space is found. rec_hfind needs two pointer-to-function arguments: one to specify the key-comparison function, and one to specify the hashing function. The keybuf argument points to a record that contains the key being searched for. The buf argument points to the storage into which the record will be read. Here is the rec_hfind function:

rec_hfind:

/* rec_hfind.c -- find a record in a hashed file */

#include "rec_io.h"

long rec_hfind(fd, keybuf, buf, cmpfn, hashfn)

rec_fd fd;         /* hashed file */

data_ptr keybuf;   /* record key to find */

data_ptr buf;      /* buffer for found record */

int (*cmpfn)();    /* record key comparison function */

long (*hashfn)();  /* hash lookup function */

{

long nrecs;        /* number of records in file */

long ntry;

long i;

nrecs = rec_nrecs(fd);

i = (*hashfn)(keybuf);

for (ntry = 0; ntry < nrecs; ++ntry)

{

if (!rec_get(fd, buf, i))

return (REC_ERROR);          /* i/o error during rec_get */

if (*(char *)buf == REC AVAIL)  /* examine first byte of buf */

return (REC_NOT_FOUND);

if (*(char *)buf == REC_OCCUP)

{

if ((*cmpfn)(keybuf, buf) == 0)

return (i);

}

i = IMOD(i + 1, nrecs);

}

return (REC_FULL);

}

When a new record is to be added to the file, a different function (rec_havail) must be used to search for an available space to store the record. This function requires that the new record's key not be present in the file already, so rec_hfind must have been called first. The rec_hfind function skips over deleted records, because the record it is searching for might still be found further down in the file. On the other hand, rec_havail can stop when it sees the deleted record (or an "available" record), because a new record could be added here.

rec_havail:

/* rec_havail.c -- find where a record belongs in a hashed file */

#include "rec_io.h"

long rec_havail(fd, keybuf, buf, hashfn)

rec_fd fd;         /* hashed file */

data_ptr keybuf;   /* record key to find */

data_ptr buf;      /* buffer for found record */

long (*hashfn)();  /* hash lookup function */

{

long nrecs;        /* number of records in file */

long ntry;

long i;

nrecs = rec_nrecs(fd);

i = (*hashfn)(keybuf);

for (ntry = 0; ntry < nrecs; ++ntry)

{

if (!rec_get(fd, buf, i))

return (REC_ERROR);

if (*(char *)buf == REC_AVAIL || *(char *)buf == REC_DELET)

return (i);

i = IMOD(i + 1, nrecs);

}

return (REC_FULL);

}

Now we have all we need to implement a simple database such as the "parts" file. Each of the "part database" functions now makes use of the "hash file" functions to find, add, change, and delete records.

part_hash.c:

/* part_hash.c -- hash file database access to parts */

#include "rec_io.h"

#include "part.h"

#define HPART struct hash_part

HPART

{

char hcode;

PART part;

};

static HPART hpart = {0};

static HPART kpart = {0};

static rec_fd part_fd = 0;

/* db_cmp_part - compare function for database access */

static int db_cmp_part(p1, p2)

HPART *pl, *p2;

{

return (strcmp(p1->part.part_no, p2->part.part_no));

}

/* db_hash_part - hash the part_no field */

static long db_hash_part(p1)

HPART *p1;

{

char *p;

long sum = 0;

for (p = p1->part.part_no; *p != '\0'; ++p)

sum += *p;

return (IMOD(sum, rec_nrecs(part_fd)));

}

/* db_open_part - open the parts database */

void db_open_part()

{

part_fd = rec_open("part.dat", O_RDWR, sizeof(HPART));

if (part_fd < 0)

error("can't open ", "part.dat");

}

/* db_close_part - close the parts database */

void db_close_part()

{

if (rec_close(part_fd) < 0)

error("can't close ", "part.dat");

}

/* db_add_part - add a part to the database */

bool db_add_part(partp)

PART *partp;

{

long i;

STRUCTASST(hpart.part, *partp);

hpart.hcode = REC_OCCUP;

i = rec_havail(part_fd, &hpart, &kpart, db_hash_part);

if (i == REC_FULL)

{

remark("Part storage is full", "");

return (NO);

}

else if (i == REC_ERROR || !rec_put(part_fd, &hpart, i))

{

remark("I/O error", "");

return (NO);

}

return (YES);

}

/* db_chg_part - change a part record on database */

bool db_chg part(partp)

PART *partp;

{

long this_rec;

STRUCTASST(kpart.part, *partp);

this_rec = rec_hfind(part_fd, &kpart, &hpart, db_cmp_part, db_hash_part);

kpart.hcode = REC_OCCUP;

return (rec_put(part_fd, &kpart, this_rec));

}

/* db_del_part - delete a part record on database */

bool db_del_part(partp)

PART *partp;

{

long this_rec;

STRUCTASST(kpart.part, *partp);

this_rec = rec_hfind(part_fd, &kpart, &hpart, db_cmp_part, db_hash_part);

kpart.hcode = REC_DELET;

return (rec_put(part_fd, &kpart, this_rec));

}

/* db_find_part - find a part on the database */

bool db_find_part(partp)

PART *partp;

{

long i;

STRUCTASST(kpart.part, *partp);

this_rec = rec_hfind(fd, &kpart, &hpart, db_cmp_part, db_hash_part);

if (this_rec >= 0)

{

STRUCTASST(*partp, hpart.part);

return (YES);

}

else if (this_rec == REC_ERROR)

{

remark("I/O error", "");

return (NO);

}

else

return (NO);

}

This simple database concludes our discussion of file handling. The "hash file" is adequate for very simple applications, with only one key field and a generous amount of disk available for the record storage. Its handling of deleted records is rudimentary. It lacks the ability to find records sequentially. For more sophisticated requirements, there are now numerous database and file management packages available for use with C.

My main purpose throughout the book has been to introduce you to the use of C for programming reliable data structures. Wherever possible, I have tried to show examples which may also be useful in their own right, but ultimately the techniques are the most important thing. You now have all the tools of C language at your disposal, and the necessary concepts to use them reliably.

Go to  Appendix    Return to Table of Contents


Reliable Data Structures: INTRODUCTION

 Return to Table of Contents

INTRODUCTION

A programmer needs to know several things, in order to write reliable programs using C data structures (simple and complex):

1. The exact meaning of "data types of variables and expressions": Type mismatches are usually accompanied by a bug; the code is likely to be either erroneous, or hard-to-modify, or non-portable.

2. What is available in the C library: Libary knowledge is required for correct use of the functions and avoidance of potentially buggy "re-inventing the wheel."

3. The meaning of "properties of data": Starting with the simple distinction between "defined" and "undefined," a grasp of data properties is essential to producing correct programs.

4. Relevant algorithms, including static structures, dynamic structures, and files.

5. How to create re-useable libraries of functions and macros.

Here is what a programmer needs to do to create reliable programs:

1. Follow a design discipline that ensures that data objects and program control structures have clear invariants.

2. Wherever possible, make use of re-useable code (headers and libraries) to avoid "re-inventing."

3. Write in a style that would allow all possible automated checking assistance. Whether or not automated assistance is used, the discipline means fewer reliability problems.

4. As much as possible, preserve the defining properties of data objects, and give warning in the transitional regions of code.

5. Use executable assertions wherever practical.

There is an important relationship between reliability and portability. A successful C program is very likely to be ported to new environments. If it was not written with due regard for portability, reliability problems pop up when the code is ported. Thus, in most cases it is not adequate for the program to work in only one environment.

There is, of course, an important role for code which is intentionally targeted to just one environment. C can be used to replace assembler code for low-level applications where one controls the special resources of a particular machine. What reliability demands in these cases is that the environment-dependent nature of the code be clearly documented, preferably with explicit mention of the environment-dependent features.

To give you now a brief overview of the book's contents, Chapter 1 describes the definition capabilities of C, describing both the preprocessor and the use of typedef. Header files are presented both for the standard library and for the functions that will be presented in the book.

Chapter 2 covers the reliable use of the scalar data types of C: floating-point numbers, integers, and characters. The relevant libaries are discussed, along with suggestions for the handling of errors.

Chapter 3 describes arrays. The use of loop invariants and assertions is described in the context of two algorithms for sorting an array. Strings, and other arrays of characters, are covered, along with the library functions that use them.

Chapter 4 deals with pointers, their types and their properties.

Chapter 5 discusses the standard I/O facilities of the C library, as well as methods for direct control of screen-oriented I/O.

Chapter 6 covers structures, unions, bit-fields, and the use of structures to create a handler for "menu" interactions.

Chapter 7 describes dynamic data structures, the malloc and free functions, and the macros and functions needed to create stacks, queues, double-ended queues ("deques"), and trees.

Chapter 8 covers file I/O of both the "stream" and "record" varieties. Binary I/O, direct access, and "hashed" files are used to implement a simple database.

An appendix summarizes the reliability rules presented in the text, along with notes on the state of the ANSI C draft.

One consistent focus of this book is precision of speaking and of documentation. To this end, I have made use of several abbreviations borrowed from various sources. Each will be explained in its proper context, but it will be useful to provide a brief preview here:

nul            means the char value '\0'

{lo:hi}        means the range of values from lo to hi

a[i:j]         means the subarray a[i] through a[j]

a[*]           means the entire array a (for emphasis)

p[*]           means the array to which pointer p provides access

a[*] => sorted means that a[*] is now sorted

a[*] : string  means that a[*] must be a string

None of these notations are part of C syntax. They just provide a concise way to describe the behavior of a program. You could replace each of them with a verbal phrase, if you prefer.

In this book, the use of the word "reliable" as applied to programs is meant to have very practical connotations: Reliable programs produce no surprises. The actual behavior of the code is readily determined by a reading of the program. The program does what it appears to do. This is easy to say, and not quite so easy to do.

Our quest for reliable C programs begins with the definition capabilities of C, which is the subject of the first chapter.

Go to Chapter 1 Return to Table of Contents


Reliable Data Structures: PREFACE

 Return to Table of Contents

PREFACE

This book is meant to follow an introductory book on C. It assumes that you have successfully written some programs in C and that you are familiar with the basics of variables, operators, functions, pointers, and structures. It does not assume that you know the more advanced features such as pointers to functions, pointers to structures, dynamic allocation, enumerations, typedef, bit-fields, union's, FILE I/O, or dynamic data structures; these will be covered herein.

Also covered are rules for reliable programming, programming with no surprises.

The book also introduces the important topics of data structures: arrays, strings, pointers, records, stacks, queues, double-ended queues ("deques"), trees, record files, and hashed files. Besides discussing the formal aspects of each data structure, we also present working general-purpose C functions that implement the technique.

From its conception in the early 1970's, the design of C allowed for small compilers that could run on small hardware. Extra checking was available from tools like lint, the "extended syntax checker" distributed with UNIX systems. Now, some of the checking that lint does has been incorporated into recent C compilers.

A new level of reliability checking has been introduced by run-time checking environments such as Safe C from Catalytix and bcc from Delft, such as enforcing strict bounds for arrays and pointers, catching overflow errors, use of uninitialized variables, etc. The trend toward greater automated assistance in programming has my support, and I hope the book is useful in this area.

In the future we expect to see greater use of the computer's reasoning ability to assist in the production of software for new devices. The existing techniques go by the names "artificial intelligence" and "expert systems." And just as a growing number of existing language translators produce C language as their intermediate output, I expect that we will see a growing interest in C as the output from "expert program writer" systems. In order to achieve the reliability that is needed for this level of automation, the properties needed in the generated code must be strictly verifiable. One of my intentions in this book is to prepare the way for new levels of expressive power in programming.

One of the central tenets of my approach to programming instruction is that there is an important difference between project code and publication code. Project code is written to achieve a programming purpose. It tends to make extensive use of software resources developed by the project team, especially headers and libraries. Appropriate use is made of defined types and defined constants for the usages that are typical in the project.

Publication code is written for journals, articles, and some textbooks. It uses only the standard headers and libraries.

I have chosen to present C in a "project code" style, because I think that this style comes closer to the realistic programming environment. Indeed, many of you will be working in an environment that has adopted our C Programming Guidelines as a local standard. But even if your local project standard is different, it will probably resemble the "project" style more than the "publication" style.

Working in "project" style means that more effort is needed at the start to create an environment of headers, but the effort pays off in readability and portability.

At the beginning of some sections you will find the notation "Topics assumed from previous book." Then a number of topics are listed, with page references to Learning to Program in C [Plum, 1983]. These assumed topics are covered in most of the introductory books on C. If you find that you are not familiar with one of the assumed topics, consult one of the introductory books first for further details. I have tried to make this book reasonably self-contained without recapitulating C language from the ground up.

This book's approach to reliable programming attempts to be a codifying of successful practice, building upon a growing number of successful applications in C. Rules for reliable use of C are presented which are based upon the experience of numerous working programmers.

One of the most frequent questions regarding a readable style for C is "where should code be commented and what should be in the comments." My approach to "properties of data" is meant to provide an important part of the answer to this question: the comment appearing on a variable's declaration should describe the defining property of the data, and anywhere in the code that this property is not true, there should be a succinct comment that documents this warning.

The C language and its library have been the subject of several standardization efforts during the middle 1980's. One effort was organized by /usr/group, an organization of commercially-oriented users of UNIX operating systems and similar other systems. In 1984, /usr/group published a "standard for computer operating systems that are functionally compatible with UNIX Operating Systems developed by AT&T Bell Laboratories." In particular, the function library associated with C was standardized in this document. Those who served as officers of the /usr/group Standards Committee include Heinz Lycklama, David Buck, Donald Kretsch, Michael Tilson, John Bass, Thomas Hoffman, and Eric Peterson.

In 1983, a committee was organized under the auspices of ANSI (the American National Standards Institute) to begin the standardization of C. This committee is under the administrative control of ANSI X3, the group responsible for computer systems, and is known as X3J11. Those who have served as officers of X3J11 and its subcommittees include Jim Brodie, P. J. Plauger, Ralph Phraner, Larry Ros1er, Ralph Ryan, and myself.

One of the early decisions of X3J11 was to adopt the /usr/group library standard as the starting point ("base document") for the standardization of the C library. Careful attention was paid to eliminating the system-specific aspects of this library; the eventual library specification for ANSI C is meant to be portable to all operating systems which provide support for C compilation and execution. As of the writing of this book (mid-1985), the current draft of X3J11 has been forwarded to ANSI X3 for publication as an Information Bulletin, one step short of a formal Draft Proposed Standard. Current expectations are for an ANSI standard for C to be adopted during the year 1986. Many vendors are already incorporating many of the features of this evolving standard, so you are likely to encounter these features with the next releases of your compilers.

This book attempts to provide a style of writing C which will allow compilation of programs both with current compilers and with new compilers that adopt features of the evolving standard. I have attempted throughout the book to inform you of the likely future direction of C. Still, there will be changes to the X3J11 draft before it becomes an official standard, so it must be understood throughout that none of this book is an official statement of the X3J11 committee. I especially ask reviewers not to cite my observations about the ANSI draft as representations of the final decisions of X3J11.

This book describes a style of programming that is compatible both with the (June 1985) draft of ANSI C and with a broad spectrum of existing compilers. The approach has been specifically verified against the compiler and library of UNIX System V (Release 2), Whitesmiths 2.3 and 3.0, Lattice 2.15, and Catalytix Safe C, as well as the library standardized by /usr/group. This level of C will be referred to as "recent C" in the book. But where it is specifically necessary to refer to the (draft) ANSI C standard, I have done so.

The official ANSI standard for C is expected during 1986, but the reliability advantages of recent C compilers are available now. The rules described in this book represent techniques that can be used now, with existing compilers, for maximum reliability in both the current and the future (ANSI C) environments.

There are a number of people whom I would like to acknowledge for their important contributions to this book. Yoram Bar-Am and Steve Bennett wrote the original version of the menu processor which I adapted for Chapter 6. Jim Brodie, Jim Ginn, Jim D'Ottavi, David Graham, Steve Hersee, Andrew Johnson, Bob Racko, Roland Racko, Christopher Skelly, and my mother, Ruby Schunior, read several drafts and made many helpful suggestions. Dave Brown, Dennis Cook, Debbie Fullam, Jim Griffin, Tom Hoffman, Paul Kleppner, Bill Koenig, Neil Ludlam, Mike Meissner, Ed Rathje, and Larry Rosler provided numerous helpful ideas during various stages of the book. Many thanks go to Joe Lochel for his clams and his friendship. Tim Cwik provided tireless assistance with the programming examples and the final production work. I am especially grateful to Roland Racko for writing the ev editor, which provides an identical screen-editing interface on the four different machine environments in our office. My thanks also to Whitesmiths, Ltd., for the reliability of the Idris systems, on which I did my development work up to the typesetting stage. Alan Feuer and Jonathan Lettvin advanced the state-of-the-art in C reliability with the Safe C environments and made numerous helpful suggestions, especially regarding the ranges of pointers. Brent Byers, the author of the Tplus formatter, provided valuable assistance with the typesetting. I am indebted to P. J. Plauger for many contributions: for pioneering the development of C compilers for a wide range of popular operating systems, for innumerable discussions of the subject matter across several continents, and for his careful reading of a preliminary draft of the book. My thanks should in no way imply that each of these people agrees with every one of my conclusions.

I also wish to thank Dennis Ritchie for designing a language of unique elegance, power, and utility.

There are some other people who made important contributions to the work which has culminated in this book. They include John Colligan, Paul Davis, Kee Dewdney, Dennis Geller, John Holland, G. M. Weinberg, Steve Robertshaw, J. A. Robinson, Gil Roeder, Steve Schustack, Barbara Smyly, Glenn Smyly, and Ed Yourdon. My sincere thanks to each of them.

My heartfelt thanks to all the staff of Plum Hall Inc -- Linda Deutsch, Suzanne Battista, Cathy Bertino, and Anne Hall -- for everything that they did to bring this book into existence. I am profoundly grateful to my late father, George E. Schunior, for his love and direction. And to my family, the Halls, McEvoys, Pritchetts, Richters, Schuniors, Schanzes, and Whynmans, goes my deepest gratitude for their steadfast support. To them, and especially to Joan Hall, my thanks for everything.

Go to Introduction Return to Table of Contents


Reliable Data Structures: Table of Contents

Reliable Data Structures in C

by Thomas Plum

PREFACE

INTRODUCTION

CHAPTER 1: PREPROCESSING

CHAPTER 2: SCALARS

CHAPTER 3: ARRAYS

CHAPTER 4: POINTERS

CHAPTER 5: STANDARD FILE AND TERMINAL I/O

CHAPTER 6: STRUCTURES

CHAPTER 7: DYNAMIC DATA STRUCTURES

CHAPTER 8: FILES

APPENDIX

BIBLIOGRAPHY


Data Structures and Algorithms: CHAPTER 1: Design and Analysis of Algorithms

 Return to Table of Contents

Design and Analysis of Algorithms

There are many steps involved in writing a computer program to solve a given problem. The steps go from problem formulation and specification, to design of the solution, to implementation, testing and documentation, and finally to evaluation of the solution. This chapter outlines our approach to these steps. Subsequent chapters discuss the algorithms and data structures that are the building blocks of most computer programs.

1.1 From Problems to Programs

Half the battle is knowing what problem to solve. When initially approached, most problems have no simple, precise specification. In fact, certain problems, such as creating a "gourmet" recipe or preserving world peace, may be impossible to formulate in terms that admit of a computer solution. Even if we suspect our problem can be solved on a computer, there is usually considerable latitude in several problem parameters. Often it is only by experimentation that reasonable values for these parameters can be found.

If certain aspects of a problem can be expressed in terms of a formal model, it is usually beneficial to do so, for once a problem is formalized, we can look for solutions in terms of a precise model and determine whether a program already exists to solve that problem. Even if there is no existing program, at least we can discover what is known about this model and use the properties of the model to help construct a good solution.

Almost any branch of mathematics or science can be called into service to help model some problem domain. Problems essentially numerical in nature can be modeled by such common mathematical concepts as simultaneous linear equations (e.g., finding currents in electrical circuits, or finding stresses in frames made of connected beams) or differential equations (e.g., predicting population growth or the rate at which chemicals will react). Symbol and text processing problems can be modeled by character strings and formal grammars. Problems of this nature include compilation (the translation of programs written in a programming language into machine language) and information retrieval tasks such as recognizing particular words in lists of titles owned by a library.

Algorithms

Once we have a suitable mathematical model for our problem, we can attempt to find a solution in terms of that model. Our initial goal is to find a solution in the form of an algorithm, which is a finite sequence of instructions, each of which has a clear meaning and can be performed with a finite amount of effort in a finite length of time. An integer assignment statement such as x := y + z is an example of an instruction that can be executed in a finite amount of effort. In an algorithm instructions can be executed any number of times, provided the instructions themselves indicate the repetition. However, we require that, no matter what the input values may be, an algorithm terminate after executing a finite number of instructions. Thus, a program is an algorithm as long as it never enters an infinite loop on any input.

There is one aspect of this definition of an algorithm that needs some clarification. We said each instruction of an algorithm must have a "clear meaning" and must be executable with a "finite amount of effort." Now what is clear to one person may not be clear to another, and it is often difficult to prove rigorously that an instruction can be carried out in a finite amount of time. It is often difficult as well to prove that on any input, a sequence of instructions terminates, even if we understand clearly what each instruction means. By argument and counterargument, however, agreement can usually be reached as to whether a sequence of instructions constitutes an algorithm. The burden of proof lies with the person claiming to have an algorithm. In Section 1.5 we discuss how to estimate the running time of common programming language constructs that can be shown to require a finite amount of time for their execution.

In addition to using Pascal programs as algorithms, we shall often present algorithms using a pseudo-language that is a combination of the constructs of a programming language together with informal English statements. We shall use Pascal as the programming language, but almost any common programming language could be used in place of Pascal for the algorithms we shall discuss. The following example illustrates many of the steps in our approach to writing a computer program.

Example 1.1. A mathematical model can be used to help design a traffic light for a complicated intersection of roads. To construct the pattern of lights, we shall create a program that takes as input a set of permitted turns at an intersection (continuing straight on a road is a "turn") and partitions this set into as few groups as possible such that all turns in a group are simultaneously permissible without collisions. We shall then associate a phase of the traffic light with each group in the partition. By finding a partition with the smallest number of groups, we can construct a traffic light with the smallest number of phases.

For example, the intersection shown in Fig. 1.1 occurs by a watering hole called JoJo's near Princeton University, and it has been known to cause some navigational difficulty, especially on the return trip. Roads C and E are oneway, the others two way. There are 13 turns one might make at this intersection. Some pairs of turns, like AB (from A to B) and EC, can be carried out simultaneously, while others, like AD and EB, cause lines of traffic to cross and therefore cannot be carried out simultaneously. The light at the intersection must permit turns in such an order that AD and EB are never permitted at the same time, while the light might permit AB and EC to be made simultaneously.

Fig. 1.1. An intersection.

We can model this problem with a mathematical structure known as a graph. A graph consists of a set of points called vertices, and lines connecting the points, called edges. For the traffic intersection problem we can draw a graph whose vertices represent turns and whose edges connect pairs of vertices whose turns cannot be performed simultaneously. For the intersection of Fig. 1.1, this graph is shown in Fig. 1.2, and in Fig. 1.3 we see another representation of this graph as a table with a 1 in row i and column j whenever there is an edge between vertices i and j.

The graph can aid us in solving the traffic light design problem. A coloring of a graph is an assignment of a color to each vertex of the graph so that no two vertices connected by an edge have the same color. It is not hard to see that our problem is one of coloring the graph of incompatible turns using as few colors as possible.

The problem of coloring graphs has been studied for many decades, and the theory of algorithms tells us a lot about this problem. Unfortunately, coloring an arbitrary graph with as few colors as possible is one of a large class of problems called "NP-complete problems," for which all known solutions are essentially of the type "try all possibilities." In the case of the coloring problem, "try all possibilities" means to try all assignments of colors to vertices using at first one color, then two colors, then three, and so on, until a legal coloring is found. With care, we can be a little speedier than this, but it is generally believed that no algorithm to solve this problem can be substantially more efficient than this most obvious approach.

We are now confronted with the possibility that finding an optimal solution for the problem at hand is computationally very expensive. We can adopt

Fig. 1.2. Graph showing incompatible turns.

Fig. 1.3. Table of incompatible turns.

one of three approaches. If the graph is small, we might attempt to find an optimal solution exhaustively, trying all possibilities. This approach, however, becomes prohibitively expensive for large graphs, no matter how efficient we try to make the program. A second approach would be to look for additional information about the problem at hand. It may turn out that the graph has some special properties, which make it unnecessary to try all possibilities in finding an optimal solution. The third approach is to change the problem a little and look for a good but not necessarily optimal solution. We might be happy with a solution that gets close to the minimum number of colors on small graphs, and works quickly, since most intersections are not even as complex as Fig. 1.1. An algorithm that quickly produces good but not necessarily optimal solutions is called a heuristic.

One reasonable heuristic for graph coloring is the following "greedy" algorithm. Initially we try to color as many vertices as possible with the first color, then as many as possible of the uncolored vertices with the second color, and so on. To color vertices with a new color, we perform the following steps.

  1. Select some uncolored vertex and color it with the new color.

  2. Scan the list of uncolored vertices. For each uncolored vertex, determine whether it has an edge to any vertex already colored with the new color. If there is no such edge, color the present vertex with the new color.

This approach is called "greedy" because it colors a vertex whenever it can, without considering the potential drawbacks inherent in making such a move. There are situations where we could color more vertices with one color if we were less "greedy" and skipped some vertex we could legally color. For example, consider the graph of Fig. 1.4, where having colored vertex 1 red, we can color vertices 3 and 4 red also, provided we do not color 2 first. The greedy algorithm would tell us to color 1 and 2 red, assuming we considered vertices in numerical order.

Fig. 1.4. A graph.

As an example of the greedy approach applied to Fig. 1.2, suppose we start by coloring AB blue. We can color AC, AD, and BA blue, because none of these four vertices has an edge in common. We cannot color BC blue because there is an edge between AB and BC. Similarly, we cannot color BD, DA, or DB blue because each of these vertices is connected by an edge to one or more vertices already colored blue. However, we can color DC blue. Then EA, EB, and EC cannot be colored blue, but ED can.

Now we start a second color, say by coloring BC red. BD can be colored red, but DA cannot, because of the edge between BD and DA. Similarly, DB cannot be colored red, and DC is already blue, but EA can be colored red. Each other uncolored vertex has an edge to a red vertex, so no other vertex can be colored red.

The remaining uncolored vertices are DA, DB, EB, and EC. If we color DA green, then DB can be colored green, but EB and EC cannot. These two may be colored with a fourth color, say yellow. The colors are summarized in Fig. 1.5. The "extra" turns are determined by the greedy approach to be compatible with the turns already given that color, as well as with each other. When the traffic light allows turns of one color, it can also allow the extra turns safely.

Fig. 1.5. A coloring of the graph of Fig. 1.2.

The greedy approach does not always use the minimum possible number of colors. We can use the theory of algorithms again to evaluate the goodness of the solution produced. In graph theory, a k-clique is a set of k vertices, every pair of which is connected by an edge. Obviously, k colors are needed to color a k-clique, since no two vertices in a clique may be given the same color.

In the graph of Fig. 1.2 the set of four vertices AC, DA, BD, EB is a 4-clique. Therefore, no coloring with three or fewer colors exists, and the solution of Fig. 1.5 is optimal in the sense that it uses the fewest colors possible. In terms of our original problem, no traffic light for the intersection of Fig. 1.1 can have fewer than four phases.

Therefore, consider a traffic light controller based on Fig. 1.5, where each phase of the controller corresponds to a color. At each phase the turns indicated by the row of the table corresponding to that color are permitted, and the other turns are forbidden. This pattern uses as few phases as possible.

Pseudo-Language and Stepwise Refinement

Once we have an appropriate mathematical model for a problem, we can formulate an algorithm in terms of that model. The initial versions of the algorithm are often couched in general statements that will have to be refined subsequently into smaller, more definite instructions. For example, we described the greedy graph coloring algorithm in terms such as "select some uncolored vertex." These instructions are, we hope, sufficiently clear that the reader grasps our intent. To convert such an informal algorithm to a program, however, we must go through several stages of formalization (called stepwise refinement) until we arrive at a program the meaning of whose steps are formally defined by a language manual.

Example 1.2. Let us take the greedy algorithm for graph coloring part of the way towards a Pascal program. In what follows, we assume there is a graph G, some of whose vertices may be colored. The following program greedy determines a set of vertices called newclr, all of which can be colored with a new color. The program is called repeatedly, until all vertices are colored. At a coarse level, we might specify greedy in pseudo-language as in Fig. 1.6.

   procedure greedy ( var G: GRAPH; var newclr: SET );
       { greedy assigns to newclr a set of vertices of G that may be
             given the same color }
    begin
(1)          newclr := Ø; †
(2)          for each uncolored vertex v of G do
(3)               if v is not adjacent to any vertex in newclr then begin
(4)                    mark v colored;
(5)                    add v to newclr
         end
    end; { greedy }

Fig. 1.6. First refinement of greedy algorithm.

We notice from Fig. 1.6 certain salient features of our pseudo-language. First, we use boldface lower case keywords corresponding to Pascal reserved words, with the same meaning as in standard Pascal. Upper case types such as GRAPH and SET‡ are the names of "abstract data types." They will be defined by Pascal type definitions and the operations associated with these abstract data types will be defined by Pascal procedures when we create the final program. We shall discuss abstract data types in more detail in the next two sections.

The flow-of-control constructs of Pascal, like if, for, and while, are available for pseudo-language statements, but conditionals, as in line (3), may be informal statements rather than Pascal conditional expressions. Note that the assignment at line (1) uses an informal expression on the right. Also, the for-loop at line (2) iterates over a set.

To be executed, the pseudo-language program of Fig. 1.6 must be refined into a conventional Pascal program. We shall not proceed all the way to such a program in this example, but let us give one example of refinement, transforming the if-statement in line (3) of Fig. 1.6 into more conventional code.

To test whether vertex v is adjacent to some vertex in newclr, we consider each member w of newclr and examine the graph G to see whether there is an edge between v and w. An organized way to make this test is to use found, a boolean variable to indicate whether an edge has been found. We can replace lines (3)-(5) of Fig. 1.6 by the code in Fig. 1.7.

    procedure greedy ( var G: GRAPH; var newclr: SET );
        begin
(1)             newclr : = Ø;
(2)             for each uncolored vertex v of G do begin
(3.1)                found := false;
(3.2)                for each vertex w in newclr do
(3.3)                    if there is an edge between v and w in G then
(3.4)                         found := true;
(3.5)                if found = false then begin
                  { v is adjacent to no vertex in newclr }
(4)                       mark v colored;
(5)                       add v to newclr
             end
            end
        end; { greedy }

Fig. 1.7. Refinement of part of Fig. 1.6.

We have now reduced our algorithm to a collection of operations on two sets of vertices. The outer loop, lines (2)-(5), iterates over the set of uncolored vertices of G. The inner loop, lines (3.2)-(3.4), iterates over the vertices currently in the set newclr. Line (5) adds newly colored vertices to newclr.

There are a variety of ways to represent sets in a programming language like Pascal. In Chapters 4 and 5 we shall study several such representations. In this example we can simply represent each set of vertices by another abstract data type LIST, which here can be implemented by a list of integers terminated by a special value null (for which we might use the value 0). These integers might, for example, be stored in an array, but there are many other ways to represent LIST's, as we shall see in Chapter 2.

We can now replace the for-statement of line (3.2) in Fig. 1.7 by a loop, where w is initialized to be the first member of newclr and changed to be the next member, each time around the loop. We can also perform the same refinement for the for-loop of line (2) in Fig. 1.6. The revised procedure greedy is shown in Fig. 1.8. There is still more refinement to be done after Fig. 1.8, but we shall stop here to take stock of what we have done.

    procedure greedy ( var G: GRAPH; var newclr: LIST );
       { greedy assigns to newclr those vertices that may be
        given the same color }
       var
        found: boolean;
        v, w: integer;
       begin
        newclr := Ø;
        v := first uncolored vertex in G;
        while v < > null do begin
           found := false;
           w := first vertex in newclr;
           while w < > null do begin
            if there is an edge between v and w in G then
                found := true;
            w := next vertex in newclr
           end;
           if found = false do begin
            mark v colored;
            add v to newclr
           end;
           v := next uncolored vertex in G
        end
       end; { greedy }

Fig. 1.8. Refined greedy procedure.

Summary

In Fig. 1.9 we see the programming process as it will be treated in this book. The first stage is modeling using an appropriate mathematical model such as a graph. At this stage, the solution to the problem is an algorithm expressed very informally.

At the next stage, the algorithm is written in pseudo-language, that is, a mixture of Pascal constructs and less formal English statements. To reach that stage, the informal English is replaced by progressively more detailed sequences of statements, in the process known as stepwise refinement. At some point the pseudo-language program is sufficiently detailed that the

Fig. 1.9. The problem solving process.

operations to be performed on the various types of data become fixed. We then create abstract data types for each type of data (except for the elementary types such as integers, reals and character strings) by giving a procedure name for each operation and replacing uses of each operation by an invocation of the corresponding procedure.

In the third stage we choose an implementation for each abstract data type and write the procedures for the various operations on that type. We also replace any remaining informal statements in the pseudo-language algorithm by Pascal code. The result is a running program. After debugging it will be a working program, and we hope that by using the stepwise development approach outlined in Fig. 1.9, little debugging will be necessary.

1.2 Abstract Data Types

Most of the concepts introduced in the previous section should be familiar ideas from a beginning course in programming. The one possibly new notion is that of an abstract data type, and before proceeding it would be useful to discuss the role of abstract data types in the overall program design process. To begin, it is useful to compare an abstract data type with the more familiar notion of a procedure.

Procedures, an essential tool in programming, generalize the notion of an operator. Instead of being limited to the built-in operators of a programming language (addition, subtraction, etc.), by using procedures a programmer is free to define his own operators and apply them to operands that need not be basic types. An example of a procedure used in this way is a matrix multiplication routine.

Another advantage of procedures is that they can be used to encapsulate parts of an algorithm by localizing in one section of a program all the statements relevant to a certain aspect of a program. An example of encapsulation is the use of one procedure to read all input and to check for its validity. The advantage of encapsulation is that we know where to go to make changes to the encapsulated aspect of the problem. For example, if we decide to check that inputs are nonnegative, we need to change only a few lines of code, and we know just where those lines are.

Definition of Abstract Data Type

We can think of an abstract data type (ADT) as a mathematical model with a collection of operations defined on that model. Sets of integers, together with the operations of union, intersection, and set difference, form a simple example of an ADT. In an ADT, the operations can take as operands not only instances of the ADT being defined but other types of operands, e.g., integers or instances of another ADT, and the result of an operation can be other than an instance of that ADT. However, we assume that at least one operand, or the result, of any operation is of the ADT in question.

The two properties of procedures mentioned above -- generalization and encapsulation -- apply equally well to abstract data types. ADT's are generalizations of primitive data types (integer, real, and so on), just as procedures are generalizations of primitive operations (+, -, and so on). The ADT encapsulates a data type in the sense that the definition of the type and all operations on that type can be localized to one section of the program. If we wish to change the implementation of an ADT, we know where to look, and by revising one small section we can be sure that there is no subtlety elsewhere in the program that will cause errors concerning this data type. Moreover, outside the section in which the ADT's operations are defined, we can treat the ADT as a primitive type; we have no concern with the underlying implementation. One pitfall is that certain operations may involve more than one ADT, and references to these operations must appear in the sections for both ADT's.

To illustrate the basic ideas, consider the procedure greedy of the previous section which, in Fig. 1.8, was implemented using primitive operations on an abstract data type LIST (of integers). The operations performed on the LIST newclr were:

  1. make a list empty,

  2. get the first member of the list and return null if the list is empty,

  3. get the next member of the list and return null if there is no next member, and

  4. insert an integer into the list.

There are many data structures that can be used to implement such lists efficiently, and we shall consider the subject in depth in Chapter 2. In Fig. 1.8, if we replace these operations by the statements

  1. MAKENULL(newclr);

  2. w := FIRST(newclr);

  3. w := NEXT(newclr);

  4. INSERT(v, newclr);

then we see an important aspect of abstract data types. We can implement a type any way we like, and the programs, such as Fig. 1.8, that use objects of that type do not change; only the procedures implementing the operations on the type need to change.

Turning to the abstract data type GRAPH we see need for the following operations:

  1. get the first uncolored vertex,

  2. test whether there is an edge between two vertices,

  3. mark a vertex colored, and

  4. get the next uncolored vertex.

There are clearly other operations needed outside the procedure greedy, such as inserting vertices and edges into the graph and making all vertices uncolored. There are many data structures that can be used to support graphs with these operations, and we shall study the subject of graphs in Chapters 6 and 7.

It should be emphasized that there is no limit to the number of operations that can be applied to instances of a given mathematical model. Each set of operations defines a distinct ADT. Some examples of operations that might be defined on an abstract data type SET are:

  1. MAKENULL(A). This procedure makes the null set be the value for set A.

  2. UNION(A, B, C). This procedure takes two set-valued arguments A and B, and assigns the union of A and B to be the value of set C.

  3. SIZE(A). This function takes a set-valued argument A and returns an object of type integer whose value is the number of elements in the set A.

An implementation of an ADT is a translation, into statements of a programming language, of the declaration that defines a variable to be of that abstract data type, plus a procedure in that language for each operation of the ADT. An implementation chooses a data structure to represent the ADT; each data structure is built up from the basic data types of the underlying programming language using the available data structuring facilities. Arrays and record structures are two important data structuring facilities that are available in Pascal. For example, one possible implementation for variable S of type SET would be an array that contained the members of S.

One important reason for defining two ADT's to be different if they have the same underlying model but different operations is that the appropriateness of an implementation depends very much on the operations to be performed. Much of this book is devoted to examining some basic mathematical models such as sets and graphs, and developing the preferred implementations for various collections of operations.

Ideally, we would like to write our programs in languages whose primitive data types and operations are much closer to the models and operations of our ADT's. In many ways Pascal is not well suited to the implementation of various common ADT's but none of the programming languages in which ADT's can be declared more directly is as well known. See the bibliographic notes for information about some of these languages.

1.3 Data Types, Data Structures and Abstract Data Types

Although the terms "data type" (or just "type"), "data structure" and "abstract data type" sound alike, they have different meanings. In a programming language, the data type of a variable is the set of values that the variable may assume. For example, a variable of type boolean can assume either the value true or the value false, but no other value. The basic data types vary from language to language; in Pascal they are integer, real, boolean, and character. The rules for constructing composite data types out of basic ones also vary from language to language; we shall mention how Pascal builds such types momentarily.

An abstract data type is a mathematical model, together with various operations defined on the model. As we have indicated, we shall design algorithms in terms of ADT's, but to implement an algorithm in a given programming language we must find some way of representing the ADT's in terms of the data types and operators supported by the programming language itself. To represent the mathematical model underlying an ADT we use data structures, which are collections of variables, possibly of several different data types, connected in various ways.

The cell is the basic building block of data structures. We can picture a cell as a box that is capable of holding a value drawn from some basic or composite data type. Data structures are created by giving names to aggregates of cells and (optionally) interpreting the values of some cells as representing connections (e.g., pointers) among cells.

The simplest aggregating mechanism in Pascal and most other programming languages is the (one-dimensional) array, which is a sequence of cells of a given type, which we shall often refer to as the celltype. We can think of an array as a mapping from an index set (such as the integers 1, 2, . . . , n) into the celltype. A cell within an array can be referenced by giving the array name together with a value from the index set of the array. In Pascal the index set may be an enumerated type, such as (north, east, south, west), or a subrange type, such as 1..10. The values in the cells of an array can be of any one type. Thus, the declaration

name: array[indextype] of celltype;

declares name to be a sequence of cells, one for each value of type indextype; the contents of the cells can be any member of type celltype.

Incidentally, Pascal is somewhat unusual in its richness of index types. Many languages allow only subrange types (finite sets of consecutive integers) as index types. For example, to index an array by letters in Fortran, one must simulate the effect by using integer indices, such as by using index 1 to stand for 'A', 2 to stand for 'B', and so on.

Another common mechanism for grouping cells in programming languages is the record structure. A record is a cell that is made up of a collection of cells, called fields, of possibly dissimilar types. Records are often grouped into arrays; the type defined by the aggregation of the fields of a record becomes the "celltype" of the array. For example, the Pascal declaration

var
reclist: array[l..4] of record
data: real;
next: integer
end

declares reclist to be a four-element array, whose cells are records with two fields, data and next.

A third grouping method found in Pascal and some other languages is the file. The file, like the one-dimensional array, is a sequence of values of some particular type. However, a file has no index type; elements can be accessed only in the order of their appearance in the file. In contrast, both the array and the record are "random-access" structures, meaning that the time needed to access a component of an array or record is independent of the value of the array index or field selector. The compensating benefit of grouping by file, rather than by array, is that the number of elements in a file can be time-varying and unlimited.

Pointers and Cursors

In addition to the cell-grouping features of a programming language, we can represent relationships between cells using pointers and cursors. A pointer is a cell whose value indicates another cell. When we draw pictures of data structures, we indicate the fact that cell A is a pointer to cell B by drawing an arrow from A to B.

In Pascal, we can create a pointer variable ptr that will point to cells of a given type, say celltype, by the declaration

var
ptr: � celltype

A postfix up-arrow is used in Pascal as the dereferencing operator, so the expression ptr� denotes the value (of type celltype) in the cell pointed to by ptr.

A cursor is an integer-valued cell, used as a pointer to an array. As a method of connection, the cursor is essentially the same as a pointer, but a cursor can be used in languages like Fortran that do not have explicit pointer types as Pascal does. By treating a cell of type integer as an index value for some array, we effectively make that cell point to one cell of the array. This technique, unfortunately, works only when cells of arrays are pointed to; there is no reasonable way to interpret an integer as a "pointer" to a cell that is not part of an array.

We shall draw an arrow from a cursor cell to the cell it "points to." Sometimes, we shall also show the integer in the cursor cell, to remind us that it is not a true pointer. The reader should observe that the Pascal pointer mechanism is such that cells in arrays can only be "pointed to" by cursors, never by true pointers. Other languages, like PL/I or C, allow components of arrays to be pointed to by either cursors or true pointers, while in Fortran or Algol, there being no pointer type, only cursors can be used.

Example 1.3. In Fig. 1.10 we see a two-part data structure that consists of a chain of cells containing cursors to the array reclist defined above. The purpose of the field next in reclist is to point to another record in the array. For example, reclist[4].next is 1, so record 4 is followed by record 1. Assuming record 4 is first, the next field of reclist orders the records 4, 1, 3, 2. Note that the next field is 0 in record 2, indicating that there is no following record. It is a useful convention, one we shall adopt in this book, to use 0 as a "NIL pointer," when cursors are being used. This idea is sound only if we also make the convention that arrays to which cursors "point" must be indexed starting at 1, never at 0.

Fig. 1.10. Example of a data structure.

The cells in the chain of records in Fig. 1.10 are of the type

type
recordtype = record
cursor: integer;
ptr: � recordtype
end

The chain is pointed to by a variable named header, which is of type � record-type; header points to an anonymous record of type recordtype.† That record has a value 4 in its cursor field; we regard this 4 as an index into the array reclist. The record has a true pointer in field ptr to another anonymous record. The record pointed to has an index in its cursor field indicating position 2 of reclist; it also has a nil pointer in its ptr field.

1.4 The Running Time of a Program

When solving a problem we are faced frequently with a choice among algorithms. On what basis should we choose? There are two often contradictory goals.

  1. We would like an algorithm that is easy to understand, code, and debug.

  2. We would like an algorithm that makes efficient use of the computer's resources, especially, one that runs as fast as possible.

When we are writing a program to be used once or a few times, goal (1) is most important. The cost of the programmer's time will most likely exceed by far the cost of running the program, so the cost to optimize is the cost of writing the program. When presented with a problem whose solution is to be used many times, the cost of running the program may far exceed the cost of writing it, especially, if many of the program runs are given large amounts of input. Then it is financially sound to implement a fairly complicated algorithm, provided that the resulting program will run significantly faster than a more obvious program. Even in these situations it may be wise first to implement a simple algorithm, to determine the actual benefit to be had by writing a more complicated program. In building a complex system it is often desirable to implement a simple prototype on which measurements and simulations can be performed, before committing oneself to the final design. It follows that programmers must not only be aware of ways of making programs run fast, but must know when to apply these techniques and when not to bother.

Measuring the Running Time of a Program

The running time of a program depends on factors such as:

  1. the input to the program,

  2. the quality of code generated by the compiler used to create the object program,

  3. the nature and speed of the instructions on the machine used to execute the program, and

  4. the time complexity of the algorithm underlying the program.

The fact that running time depends on the input tells us that the running time of a program should be defined as a function of the input. Often, the running time depends not on the exact input but only on the "size" of the input. A good example is the process known as sorting, which we shall discuss in Chapter 8. In a sorting problem, we are given as input a list of items to be sorted, and we are to produce as output the same items, but smallest (or largest) first. For example, given 2, 1, 3, 1, 5, 8 as input we might wish to produce 1, 1, 2, 3, 5, 8 as output. The latter list is said to be sorted smallest first. The natural size measure for inputs to a sorting program is the number of items to be sorted, or in other words, the length of the input list. In general, the length of the input is an appropriate size measure, and we shall assume that measure of size unless we specifically state otherwise.

It is customary, then, to talk of T(n), the running time of a program on inputs of size n. For example, some program may have a running time T(n) = cn2, where c is a constant. The units of T(n) will be left unspecified, but we can think of T(n) as being the number of instructions executed on an idealized computer.

For many programs, the running time is really a function of the particular input, and not just of the input size. In that case we define T(n) to be the worst case running time, that is, the maximum, over all inputs of size n, of the running time on that input. We also consider Tavg(n), the average, over all inputs of size n, of the running time on that input. While Tavg(n) appears a fairer measure, it is often fallacious to assume that all inputs are equally likely. In practice, the average running time is often much harder to determine than the worst-case running time, both because the analysis becomes mathematically intractable and because the notion of "average" input frequently has no obvious meaning. Thus, we shall use worst-case running time as the principal measure of time complexity, although we shall mention average-case complexity wherever we can do so meaningfully.

Now let us consider remarks (2) and (3) above: that the running time of a program depends on the compiler used to compile the program and the machine used to execute it. These facts imply that we cannot express the running time T(n) in standard time units such as seconds. Rather, we can only make remarks like "the running time of such-and-such an algorithm is proportional to n2." The constant of proportionality will remain unspecified since it depends so heavily on the compiler, the machine, and other factors.

Big-Oh and Big-Omega Notation

To talk about growth rates of functions we use what is known as "big-oh" notation. For example, when we say the running time T(n) of some program is O(n2), read "big oh of n squared" or just "oh of n squared," we mean that there are positive constants c and n0 such that for n equal to or greater than n0, we have T(n) � cn2.

Example 1.4. Suppose T(0) = 1, T(1) = 4, and in general T(n) = (n+l)2. Then we see that T(n) is O(n2), as we may let n0 = 1 and c = 4. That is, for n � 1, we have (n + 1)2 � 4n2, as the reader may prove easily. Note that we cannot let n0 = 0, because T(0) = 1 is not less than c02 = 0 for any constant c.

In what follows, we assume all running-time functions are defined on the nonnegative integers, and their values are always nonnegative, although not necessarily integers. We say that T(n) is O(f(n)) if there are constants c and n0 such that T(n) � cf(n) whenever n � n0. A program whose running time is O(f (n)) is said to have growth rate f(n).

Example 1.5. The function T(n)= 3n3 + 2n2 is O(n3). To see this, let n0 = 0 and c = 5. Then, the reader may show that for n � 0, 3n3 + 2n2 � 5n3. We could also say that this T(n) is O(n4), but this would be a weaker statement than saying it is O(n3).

As another example, let us prove that the function 3n is not O (2n). Suppose that there were constants n0 and c such that for all n � n0, we had 3n � c2n. Then c � (3/2)n for any n � n0. But (3/2)n gets arbitrarily large as n gets large, so no constant c can exceed (3/2)n for all n.

When we say T(n) is O(f(n)), we know that f(n) is an upper bound on the growth rate of T(n). To specify a lower bound on the growth rate of T(n) we can use the notation T(n) is W(g(n)), read "big omega of g(n)" or just "omega of g(n)," to mean that there exists a positive constant c such that T(n) � cg(n) infinitely often (for an infinite number of values of n).†

Example 1.6. To verify that the function T(n)= n3 + 2n2 is W(n3), let c = 1. Then T(n) � cn3 for n = 0, 1, . . ..

For another example, let T(n) = n for odd n � 1 and T(n) = n2/100 for even n � 0. To verify that T(n) is W (n2), let c = 1/100 and consider the infinite set of n's: n = 0, 2, 4, 6, . . ..

The Tyranny of Growth Rate

We shall assume that programs can be evaluated by comparing their running-time functions, with constants of proportionality neglected. Under this assumption a program with running time O(n2) is better than one with running time O(n3), for example. Besides constant factors due to the compiler and machine, however, there is a constant factor due to the nature of the program itself. It is possible, for example, that with a particular compiler-machine combination, the first program takes 100n2 milliseconds, while the second takes 5n3 milliseconds. Might not the 5n3 program be better than the 100n2 program?

The answer to this question depends on the sizes of inputs the programs are expected to process. For inputs of size n < 20, the program with running time 5n3 will be faster than the one with running time 100n2. Therefore, if the program is to be run mainly on inputs of small size, we would indeed prefer the program whose running time was O(n3). However, as n gets large, the ratio of the running times, which is 5n3/100n2 = n/20, gets arbitrarily large. Thus, as the size of the input increases, the O(n3) program will take significantly more time than the O(n2) program. If there are even a few large inputs in the mix of problems these two programs are designed to solve, we can be much better off with the program whose running time has the lower growth rate.

Another reason for at least considering programs whose growth rates are as low as possible is that the growth rate ultimately determines how big a problem we can solve on a computer. Put another way, as computers get faster, our desire to solve larger problems on them continues to increase. However, unless a program has a low growth rate such as O(n) or O(nlogn), a modest increase in computer speed makes very little difference in the size of the largest problem we can solve in a fixed amount of time.

Example 1.7. In Fig. 1.11 we see the running times of four programs with different time complexities, measured in seconds, for a particular compiler-machine combination. Suppose we can afford 1000 seconds, or about 17 minutes, to solve a given problem. How large a problem can we solve? In 103 seconds, each of the four algorithms can solve roughly the same size problem, as shown in the second column of Fig. 1.12.

Fig. 1.11. Running times of four programs.

Suppose that we now buy a machine that runs ten times faster at no additional cost. Then for the same cost we can spend 104 seconds on a problem where we spent 103 seconds before. The maximum size problem we can now solve using each of the four programs is shown in the third column of Fig. 1.12, and the ratio of the third and second columns is shown in the fourth column. We observe that a 1000% improvement in computer speed yields only a 30% increase in the size of problem we can solve if we use the O(2n) program. Additional factors of ten speedup in the computer yield an even smaller percentage increase in problem size. In effect, the O(2n) program can solve only small problems no matter how fast the underlying computer.

Fig. 1.12. Effect of a ten-fold speedup in computation time.

In the third column of Fig. 1.12 we see the clear superiority of the O(n) program; it returns a 1000% increase in problem size for a 1000% increase in computer speed. We see that the O(n3) and O(n2) programs return, respectively, 230% and 320% increases in problem size for 1000% increases in speed. These ratios will be maintained for additional increases in speed.

As long as the need for solving progressively larger problems exists, we are led to an almost paradoxical conclusion. As computation becomes cheaper and machines become faster, as will most surely continue to happen, our desire to solve larger and more complex problems will continue to increase. Thus, the discovery and use of efficient algorithms, those whose growth rates are low, becomes more rather than less important.

A Few Grains of Salt

We wish to re-emphasize that the growth rate of the worst case running time is not the sole, or necessarily even the most important, criterion for evaluating an algorithm or program. Let us review some conditions under which the running time of a program can be overlooked in favor of other issues.

  1. If a program is to be used only a few times, then the cost of writing and debugging dominate the overall cost, so the actual running time rarely affects the total cost. In this case, choose the algorithm that is easiest to implement correctly.

  2. If a program is to be run only on "small" inputs, the growth rate of the running time may be less important than the constant factor in the formula for running time. What is a "small" input depends on the exact running times of the competing algorithms. There are some algorithms, such as the integer multiplication algorithm due to Schonhage and Strassen [1971], that are asymptotically the most efficient known for their problem, but have never been used in practice even on the largest problems, because the constant of proportionality is so large in comparison to other simpler, less "efficient" algorithms.

  3. A complicated but efficient algorithm may not be desirable because a person other than the writer may have to maintain the program later. It is hoped that by making the principal techniques of efficient algorithm design widely known, more complex algorithms may be used freely, but we must consider the possibility of an entire program becoming useless because no one can understand its subtle but efficient algorithms.

  4. There are a few examples where efficient algorithms use too much space to be implemented without using slow secondary storage, which may more than negate the efficiency.

  5. In numerical algorithms, accuracy and stability are just as important as efficiency.

1.5 Calculating the Running Time of a Program

Determining, even to within a constant factor, the running time of an arbitrary program can be a complex mathematical problem. In practice, however, determining the running time of a program to within a constant factor is usually not that difficult; a few basic principles suffice. Before presenting these principles, it is important that we learn how to add and multiply in "big oh" notation.

Suppose that T1(n) and T2(n) are the running times of two program fragments P1 and P2, and that T1(n) is O(f(n)) and T2(n) is O(g(n)). Then T1(n)+T2(n), the running time of P1 followed by P2, is O(max(f(n),g(n))). To see why, observe that for some constants c1, c2, n1, and n2, if n � n1 then T1(n) � c1f(n), and if n � n2 then T2(n) � c2g(n). Let n0 = max(n1, n2). If n � n0, then T1(n) + T2(n) � c1f(n) + c2g(n). From this we conclude that if n � n0, then T1(n) + T2(n) � (c1 + c2)max(f(n), g(n)). Therefore, the combined running time T1(n) + T2(n) is O (max(f (n), g (n))).

Example 1.8. The rule for sums given above can be used to calculate the running time of a sequence of program steps, where each step may be an arbitrary program fragment with loops and branches. Suppose that we have three steps whose running times are, respectively, O(n2), O(n3) and O(n log n). Then the running time of the first two steps executed sequentially is O(max(n2, n3)) which is O(n3). The running time of all three together is O(max(n3, n log n)) which is O(n3).

In general, the running time of a fixed sequence of steps is, to within a constant factor, the running time of the step with the largest running time. In rare circumstances there will be two or more steps whose running times are incommensurate (neither is larger than the other, nor are they equal). For example, we could have steps of running times O(f (n)) and O(g (n)), where

In such cases the sum rule must be applied directly; the running time is O(max(f(n), g(n))), that is, n4 if n is even and n3 if n is odd.

Another useful observation about the sum rule is that if g(n) � f(n) for all n above some constant n0, then O(f(n) + g(n)) is the same as O(f(n)). For example, O(n2+n) is the same as O(n2).

The rule for products is the following. If T1(n) and T2(n) are O(f(n)) and O(g(n)), respectively, then T1(n)T2(n) is O(f(n)g(n)). The reader should prove this fact using the same ideas as in the proof of the sum rule. It follows from the product rule that O(cf(n)) means the same thing as O(f(n)) if c is any positive constant. For example, O(n2/2) is the same as O(n2).

Before proceeding to the general rules for analyzing the running times of programs, let us take a simple example to get an overview of the process.

Example 1.9. Consider the sorting program bubble of Fig. 1.13, which sorts an array of integers into increasing order. The net effect of each pass of the inner loop of statements (3)-(6) is to "bubble" the smallest element toward the front of the array.

    procedure bubble ( var A: array [1..n] of integer );
       { bubble sorts array A into increasing order }
    var
       i, j, temp: integer;
       begin
(1)           for i := 1 to n-1 do
(2)               for j := n downto i+1 do
(3)                    if A[j-1] > A[j] then begin
               { swap A[j - 1] and A[j] }
(4)                        temp := A[j-1];
(5)                        A[j-1] := A[j];
(6)                        AI> [j] := temp
               end
       end; { bubble }

Fig. 1.13. Bubble sort.

The number n of elements to be sorted is the appropriate measure of input size. The first observation we make is that each assignment statement takes some constant amount of time, independent of the input size. That is to say, statements (4), (5) and (6) each take O(1) time. Note that O(1) is "big oh" notation for "some constant amount." By the sum rule, the combined running time of this group of statements is O(max(1, 1, 1)) = O(1).

Now we must take into account the conditional and looping statements. The if- and for-statements are nested within one another, so we may work from the inside out to get the running time of the conditional group and each loop. For the if-statement, testing the condition requires O(1) time. We don't know whether the body of the if-statement (lines (4)-(6)) will be executed. Since we are looking for the worst-case running time, we assume the worst and suppose that it will. Thus, the if-group of statements (3)-(6) takes O(1) time.

Proceeding outward, we come to the for-loop of lines (2)-(6). The general rule for a loop is that the running time is the sum, over each iteration of the loop, of the time spent executing the loop body for that iteration. We must, however, charge at least O(1) for each iteration to account for incrementing the index, for testing to see whether the limit has been reached, and for jumping back to the beginning of the loop. For lines (2)-(6) the loop body takes O(1) time for each iteration. The number of iterations of the loop is n-i, so by the product rule, the time spent in the loop of lines (2)-(6) is O((n-i) X 1) which is O(n-i).

Now let us progress to the outer loop, which contains all the executable statements of the program. Statement (1) is executed n - 1 times, so the total running time of the program is bounded above by some constant times

which is O(n2). The program of Fig. 1.13, therefore, takes time proportional to the square of the number of items to be sorted. In Chapter 8, we shall give sorting programs whose running time is O(nlogn), which is considerably smaller, since for large n, logn † is very much smaller than n.

Before proceeding to some general analysis rules, let us remember that determining a precise upper bound on the running time of programs is sometimes simple, but at other times it can be a deep intellectual challenge. There are no complete sets of rules for analyzing programs. We can only give the reader some hints and illustrate some of the subtler points by examples throughout this book.

Now let us enumerate some general rules for the analysis of programs. In general, the running time of a statement or group of statements may be parameterized by the input size and/or by one or more variables. The only permissible parameter for the running time of the whole program is n, the input size.

  1. The running time of each assignment, read, and write statement can usually be taken to be O(1). There are a few exceptions, such as in PL/I, where assignments can involve arbitrarily large arrays, and in any language that allows function calls in assignment statements.

  2. The running time of a sequence of statements is determined by the sum rule. That is, the running time of the sequence is, to within a constant factor, the largest running time of any statement in the sequence.

  3. The running time of an if-statement is the cost of the conditionally executed statements, plus the time for evaluating the condition. The time to evaluate the condition is normally O(1). The time for an if-then-else construct is the time to evaluate the condition plus the larger of the time needed for the statements executed when the condition is true and the time for the statements executed when the condition is false.

  4. The time to execute a loop is the sum, over all times around the loop, of the time to execute the body and the time to evaluate the condition for termination (usually the latter is O(1)). Often this time is, neglecting constant factors, the product of the number of times around the loop and the largest possible time for one execution of the body, but we must consider each loop separately to make sure. The number of iterations around a loop is usually clear, but there are times when the number of iterations cannot be computed precisely. It could even be that the program is not an algorithm, and there is no limit to the number of times we go around certain loops.

Procedure Calls

If we have a program with procedures, none of which is recursive, then we can compute the running time of the various procedures one at a time, starting with those procedures that make no calls on other procedures. (Remember to count a function invocation as a "call.") There must be at least one such procedure, else at least one procedure is recursive. We can then evaluate the running time of procedures that call only procedures that make no calls, using the already-evaluated running times of the called procedures. We continue this process, evaluating the running time of each procedure after the running times of all procedures it calls have been evaluated.

If there are recursive procedures, then we cannot find an ordering of all the procedures so that each calls only previously evaluated procedures. What we must now do is associate with each recursive procedure an unknown time function T(n), where n measures the size of the arguments to the procedure. We can then get a recurrence for T(n), that is, an equation for T(n) in terms of T(k) for various values of k.

Techniques for solving many different kinds of recurrences exist; we shall present some of these in Chapter 9. Here we shall show how to analyze a simple recursive program.

Example 1.10. Figure 1.14 gives a recursive program to compute n!, the product of all the integers from 1 to n inclusive.

An appropriate size measure for this function is the value of n. Let T(n) be the running time for fact(n). The running time for lines (1) and (2) is O(1), and for line (3) it is O(1) + T(n-1). Thus, for some constants c and d,

    function fact ( n: integer ): integer;
       { fact(n) computes n! }
       begin
(1)           if n <= 1 then
(2)                fact := 1
          else
(3)                fact := n * fact(n-1)
       end; { fact }

Fig. 1.14. Receursive program to compute factorials.

Assuming n > 2, we can expand T(n-1) in (1.1) to obtain

T(n) = 2c + T(n-2) if n > 2

That is, T(n-1) = c + T(n-2), as can be seen by substituting n-1 for n in (1.1). Thus, we may substitute c + T(n-2) for T(n-1) in the equation T(n) = c + T(n-1). We can then use (1.1) to expand T(n-2) to obtain

T(n) = 3c + T(n-3) if n > 3

and so on. In general,

T(n) = ic + T(n-i) if n > i

Finally, when i = n-1 we get

T(n) = c(n-1) + T(1) = c(n-1) + d (1.2)

From (1.2) we can conclude that T(n) is O(n). We should note that in this analysis we have assumed that the multiplication of two integers is an O(1) operation. In practice, however, we cannot use the program in Fig. 1.14 to compute n! for large values of n, because the size of the integers being computed will exceed the word length of the underlying machine.

The general method for solving recurrence equations, as typified by Example 1.10, is repeatedly to replace terms T(k) on the right side of the equation by the entire right side with k substituted for n, until we obtain a formula in which T does not appear on the right as in (1.2). Often we must then sum a series or, if we cannot sum it exactly, get a close upper bound on the sum to obtain an upper bound on T(n).

Programs with GOTO's

In analyzing the running time of a program we have tacitly assumed that all flow of control within a procedure was determined by branching and 1ooping constructs. We relied on this fact as we determined the running time of progressively larger groups of statements by assuming that we needed only the sum rule to group sequences of statements together. Goto statments, however, make the logical grouping of statements more complex. For this reason, goto statements should be avoided, but Pascal lacks break- and continue-statements to jump out of loops. The goto-statement is often used as a substitute for statements of this nature in Pascal.

We suggest the following approach to handling goto's that jump from a loop to code that is guaranteed to follow the loop, which is generally the only kind of goto that is justified. As the goto is presumably executed conditionally within the loop, we may pretend that it is never taken. Because the goto takes us to a statement that will be executed after the loop completes, this assumption is conservative; we can never underestimate the worst case running time of the program if we assume the loop runs to completion. However, it is a rare program in which ignoring the goto is so conservative that it causes us to overestimate the growth rate of the worst case running time for the program. Notice that if we were faced with a goto that jumped back to previously executed code we could not ignore it safely, since that goto may create a loop that accounts for the bulk of the running time.

We should not leave the impression that the use of backwards goto's by themselves make running times unanalyzable. As long as the loops of a program have a reasonable structure, that is, each pair of loops are either disjoint or nested one within the other, then the approach to running time analysis described in this section will work. (However, it becomes the responsibility of the analyzer to ascertain what the loop structure is.) Thus, we should not hesitate to apply these methods of program analysis to a language like Fortran, where goto's are essential, but where programs written in the language tend to have a reasonable loop structure.

Analyzing a Pseudo-Program

If we know the growth rate of the time needed to execute informal English statements, we can analyze pseudo-programs just as we do real ones. Often, however, we do not know the time to be spent on not-fully-implemented parts of a pseudo-program. For example, if we have a pseudo-program in which the only unimplemented parts are operations on ADT's, one of several implementations for an ADT may be chosen, and the overall running time may depend heavily on the implementation. Indeed, one of the reasons for writing programs in terms of ADT's is so we can consider the trade-offs among the running times of the various operations that we obtain by different implementations.

To analyze pseudo-programs consisting of programming language statements and calls to unimplemented procedures, such as operations on ADT's, we compute the running time as a function of unspecified running times for each procedure. The running time for a procedure will be parameterized by the "size" of the argument or arguments for that procedure. Just as for "input size," the appropriate measure of size for an argument is a matter for the analyzer to decide. If the procedure is an operation on an ADT, then the underlying mathematical model for the ADT often indicates the logical notion of size. For example, if the ADT is based on sets, the number of elements in a set is often the right notion of size. In the remaining chapters we shall see many examples of analyzing the running time of pseudo-programs.

1.6 Good Programming Practice

There are a substantial number of ideas we should bear in mind when designing an algorithm and implementing it as a program. These ideas often appear platitudinous, because by-and-large they can be appreciated only through their successful use in real problems, rather than by development of a theory. They are sufficiently important, however, that they are worth repeating here. The reader should watch for the application of these ideas in the programs designed in this book, as well as looking for opportunities to put them into practice in his own programs.

  1. Plan the design of a program. We mentioned in Section 1.1 how a program can be designed by first sketching the algorithm informally, then as a pseudo-program, and gradually refining the pseudo-program until it becomes executable code. This strategy of sketch-then-detail tends to produce a more organized final program that is easier to debug and maintain.

  2. Encapsulate. Use procedures and ADT's to place the code for each principal operation and type of data in one place in the program listing. Then, if changes become necessary, the section of code requiring change will be localized.

  3. Use or modify an existing program. One of the chief inefficiencies in the programming process is that usually a project is tackled as if it were the first program ever written. One should first look for an existing program that does all or a part of the task. Conversely, when writing a program, one should consider making it available to others for possibly unanticipated uses.

  4. Be a toolsmith. In programming parlance, a tool is a program with a variety of uses. When writing a program, consider whether it could be written in a somewhat more general way with little extra effort. For example, suppose one is assigned the task of writing a program to schedule final examinations. Instead, write a tool that takes an arbitrary graph and colors the vertices with as few colors as possible, so that no two vertices connected by an edge have the same color. In the context of examination scheduling, the vertices are classes, the colors are examination periods, and an edge between two classes means that the classes have a student in common. The coloring program, together with routines that translate class lists into graphs and colors into specific times and days, is the examination scheduler. However, the coloring program can be used for problems totally unrelated to examination scheduling, such as the traffic light problem of Section 1.1.

  5. Program at the command level. Often we cannot find in a library the one program needed to do a job, nor can we adapt one tool to do the job. A well-designed operating system will allow us to connect a network of available programs together without writing any programs at all, except for one list of operating system commands. To make commands composable, it is generally necessary that each behave as a filter, a program with one input file and one output file. Notice that any number of filters can be composed, and if the command language of the operating system is intelligently designed, merely listing the commands in the order in which they are to be performed will suffice as a program.

Example 1.11. As an example, let us consider the program spell, as it was originally written by S.C. Johnson from UNIX† commands. This program takes as input a file f1consisting of English text and produces as output all those words in f1that are not found in a small dictionary.‡ spell tends to list proper names as misspellings and may also list real words not in its dictionary, but the typical output of spell is short enough that it can be scanned by eye, and human intelligence can be used to determine whether a word in the output of spell is a misspelling. (This book was checked using spell.)

The first filter used by spell is a command called translate that, given appropriate parameters, replaces capital letters by lower case letters and blanks by newlines, leaving other characters unchanged. The output of translate consists of a file f2 that has the words of f1, uncapitalized, one to a line. Next comes a command sort that sorts the lines of its input file into lexicographic (alphabetical) order. The output of sort is a file f3 that has all the words of f2 in alphabetical order, with repetitions. Then a command unique removes duplicate lines from its input file, producing an output file f4 that has the words of the original file, without capitalization or duplicates, in alphabetical order. Finally, a command diff, with a parameter indicating a second file f5 that holds the alphabetized list of words in the dictionary, one to a line, is applied to f4. The result is all words in f4 (and hence f1) but not in f5, i.e., all words in the original input that are not in the dictionary. The complete program spell is just the following sequence of commands.

spell : translate [A-Z] � [a-z], blank � newline
sort
unique
diff dictionary

Command level programming requires discipline from a community of programmers; they must write programs as filters wherever possible, and they must write tools instead of special purpose programs wherever possible. Yet the reward, in terms of the overall ratio of work to results, is substantial.

1.7 Super Pascal

Most of the programs written in this book are in Pascal. To make programs more readable, however, we occasionally use three constructs not found in standard Pascal, each of which can be mechanically translated into pure Pascal. One such construct is the nonnumeric label. The few times we need labels, we shall use nonnumeric labels since they make programs easier to understand. For example, "goto output" is invariably more meaningful than "goto 561." To convert a program containing nonnumeric labels into pure Pascal, we must replace each nonnumeric label by a distinct numeric label and we must then declare those labels with a label declaration at the beginning of the program. This process can be clone mechanically.

The second nonstandard construct is the return statement, which we use because it allows us to write more understandable programs without using goto statements to interrupt the flow of control. The return statement we use has the form

return (expression)

where the (expression) is optional. We can convert a procedure containing return statements into a standard Pascal program quite simply. First, we declare a new label, say 999, and let it label the last end statement of the procedure. If the statement return (x) appears in a function zap, say, we replace this statement with the block

    begin
        zap := x;
        goto 999
    end

In a procedure, the statement return, which can have no argument, is simply replaced by goto 999.

Example 1.12. Figure 1.15 shows the factorial program written using return statements. Figure 1.16 shows the resulting Pascal program if we apply this transformation systematically to Fig. 1.15.

    function fact ( n: integer ): integer;
         begin
             if n <= l then
           return (1)
        else
           return ( n * fact(n- 1))
         end; { fact }

Fig. 1.15. Factorial program with return statements.

The third extension is that we use expressions as names of types

    function fact ( n: integer ) :integer;
       label
           999;
       begin
           if n <= 1 then
           begin
               fact := 1;
               goto 999
           end
           else
           begin
               fact := n * fact(n - 1);
               goto 999
           end
      999:
      end; { fact }

Fig. 1.16. Resulting Pascal program.

uniformly throughout a program. For example, an expression like � celltype, while permissible everywhere else, is not permitted as the type of a parameter of a procedure or the type of the value returned by a function. Technically, Pascal requires that we invent a name for the type expression, say ptrtocell. In this book, we shall allow such expressions, expecting that the reader could invent such a type name and mechanically replace type expressions by the type name. Thus, we shall write statements like

function zap ( A: array[1..10] of integer ) : � celltype

to stand for

function zap (A: arrayoftenints ) : ptrtocell

Finally, a note on our typesetting conventions for programs. Pascal reserved words are in boldface, types are in roman, and procedure, function, and variable names are in italic. We distinguish between upper and lower case letters.

Exercises

1.1 | There are six teams in the football league: the Vultures, the Lions, the Eagles, the Beavers, the Tigers, and the Skunks. The Vultures have already played the Lions and the Eagles; the Lions have also played the Beavers and Skunks. The Tigers have played the Eagles and Skunks. Each team plays one game per week. Find a schedule so that all teams will have played each other in the fewest number of weeks. Hint. Create a graph whose vertices are the pairs of teams that have not yet played each other. What should the edges be so that in a legal coloring of the graph, each color can represent the games played in one week?
*1.2 | Consider a robot arm that is fixed at one end. The arm contains two elbows at each of which it is possible to rotate the arm 90 degrees up and down in a vertical plane. How would you mathematically model the possible movements of the end of the arm? Describe an algorithm to move the end of the robot arm from one permissible position to another.
*1.3 | Suppose we wish to multiply four matrices of real numbers M1 × M2 × M3 × M4 where M1 is 10 by 20, M2 is 20 by 50, M3 is 50 by 1, and M4 is 1 by 100. Assume that the multiplication of a p × q matrix by a q × r matrix requires pqr scalar operations, as it does in the usual matrix multiplication algorithm. Find the optimal order in which to multiply the matrices so as to minimize the total number of scalar operations. How would you find this optimal ordering if there are an arbitrary number of matrices?
**1.4 | Suppose we wish to partition the square roots of the integers from 1 to 100 into two piles of fifty numbers each, such that the sum of the numbers in the first pile is as close as possible to the sum of the numbers in the second pile. If we could use two minutes of computer time to help answer this question, what computations would you perform in those two minutes?
1.5 | Describe a greedy algorithm for playing chess. Would you expect it to perform very well?
1.6 | In Section 1.2 we considered an ADT SET, with operations MAKE-NULL, UNION, and SIZE. Suppose for convenience that we assume all sets are subsets of {0, 1, . . . , 31} and let the ADT SET be interpreted as the Pascal data type set of 0..31. Write Pascal procedures for these operations using this implementation of SET.
1.7 The greatest common divisor of two integers p and q is the largest integer d that divides both p and q evenly. We wish to develop a program for computing the greatest common divisor of two integers p and q using the following algorithm. Let r be the remainder of p divided by q. If r is O, then q is the greatest common divisor. Otherwise, set p equal to q, then q equal to r, and repeat the process.
  1. Show that this process does find the correct greatest common divisor.

  2. Refine this algorithm into a pseudo-language program.

  3. Convert your pseudo-language program into a Pascal program.
1.8 We want to develop a program for a text formatter that will place words on lines that are both left and right justified. The program will have a word buffer and a line buffer. Initially both are empty. A word is read into the word buffer. If there is sufficient room in the line buffer, the word is transferred to the line buffer. Otherwise, additional spaces are inserted between words in the line buffer to fill out the line, and then the line buffer is emptied by printing the line.
  1. Refine this algorithm into a pseudo-language program.

  2. Convert your pseudo-language program to a Pascal program.
1.9 | Consider a set of n cities and a table of distances between pairs of cities. Write a pseudo-language program for finding a short path that goes through each city exactly once and returns to the city from which it started. There is no known method for obtaining the shortest such tour except by exhaustive searching. Thus try to find an efficient algorithm for this problem using some reasonable heuristic.
1.10 Consider the following functions of n:

Indicate for each distinct pair i and j whether fi(n) is O(fj(n)) and whether fi(n) is W(fj(n)).
1.11 Consider the following functions of n:

Indicate for each distinct pair i and j whether gi(n) is O(gj(n)) and whether gi(n) is W(gj(n)).
1.12 Give, using "big oh" notation, the worst case running times of the following procedures as a function of n.

  1. 	procedure matmpy ( n: integer);
       var
        i, j, k: integer;
       begin
            for i := 1 to n do
             for j := 1 to n do begin
                C[i, j] := O;
                for k := 1 to n do
                C[i, j] := C[i, j,] + A[i, k] * B[k, j]
             end
        end

    2.

  2. 	procedure mystery ( n: integer);
       var
        i, j, k: integer;
       begin
            for i:= 1 to n-1 do
            for j:= i + 1 to n do
                for k := 1 to j do
                 { some statement requiring O(1) time }
       end

    3.

  3. 	procedure veryodd ( n: integer );
       var
        i, j, x, y: integer;
       begin
            for i := 1 to n do
          if odd(i) then begin
              for j := i to n do
            x := x + 1;
              for j := 1 to i do
            y := y + l
         end
       end

    4.

  4. 	function recursive (n: integer ) : integer;
         begin
        if n <= 1 then
             return (l)
        else
             return (recursive(n-1) + recursive(n-1))
         end

    5.
1.13 Show that the following statements are true.
  1. 17 is O(1).

  2. n(n-1)/2 is O(n2).

  3. max(n3, 10n2) is O(n3).

e) If p(x) is any kth degree polynomial with a positive leading coefficient, then p(n) is O(nk) and W(nk).
*1.14 Suppose T1(n) is W(f(n)) and T2(n) is W(g(n)). Which of the following statements are true?
  1. T1(n) + T2(n) is W(max(f(n), g(n))).

  2. T1(n)T2(n) is W(f(n)g(n)).
*1.15 Some authors define big omega by saying f(n) is W(g(n)) if there is some n0 and c > 0 such that for all n � n0 we have f(n) � cg(n).
  1. Is it true for this definition that f(n) is W(g(n)) if and only if g(n) is O(f(n))?

  2. Is (a) true for the definition of big omega in Section 1.4?

  3. Does Exercise 1.14(a) or (b) hold for this definition of big omega?
1.16 | Order the following functions by growth rate: (a) n, (b) ��n, (c) logn, (d) loglogn, (e) log2n, (f) n/logn, (g) ��nlog2n, (h) (1/3)n, (i) (3/2)n, (j) 17.
1.17 Assume the parameter n in the procedure below is a positive power of 2, i.e., n = 2, 4, 8, 16 , . . .. Give the formula that expresses the value of the variable count in terms of the value of n when the procedure terminates.

procedure mystery ( n: integer );
var
x, count: integer;
begin
count := 0;
x := 2;
while x < n do begin
x := 2 * x;
count := count + 1
end;
writeln(count)
end
1.18 Here is a function max(i, n) that returns the largest element in positions i through i+n-1 of an integer array A. You may assume for convenience that n is a power of 2.

               function max ( i, n: integer ): integer;

                    var

                         m1, m2: integer;

                    begin

                         if n = 1 then

                              return (A[i])

                         else begin

                              m1 := max(i, n div 2);

                              m2 := max(i+n div 2,
n div 2);

                              if m1 < m2 then

                                   return (m2)

                              else

                                   return (m1)

                         end

                    end
  1. Let T(n) be the worst-case time taken by max with second argument n. That is, n is the number of elements of which the largest is found. Write an equation expressing T(n) in terms of T(j) for one or more values of j less than n and a constant or constants that represent the times taken by individual statements of the max program.

  2. Give a tight big oh upper bound on T(n). Your answer should be equal to the big omega lower bound, and be as simple as possible.

Bibliographic Notes

The concept of an abstract data type can be traced to the class type in the language SIMULA 67 (Birtwistle et al. [1973]). Since that time, a variety of other languages that support abstract data types have been developed including Alphard (Shaw, Wulf, and London [1977]), C with classes (Stroustrup [1982]), CLU (Liskov, et al. [1977]), MESA (Geschke, Morris, and Satterthwaite [1977]), and Russell (Demers and Donahue [1979]). The ADT concept is further discussed in works such as Gotlieb and Gotlieb [1978] and Wulf et al. [1981].

Knuth [1968] was the first major work to advocate the systematic study of the running time of programs. Aho, Hopcroft, and Ullman [1974] relate the time and space complexity of algorithms to various models of computation, such as Turing machines and random-access machines. See also the bibliographic notes to Chapter 9 for more references to the subject of analysis of algorithms and programs.

For additional material on structured programming see Hoare, Dahl, and Dijkstra [1972], Wirth [1973], Kernighan and Plauger [1974], and Yourdon and Constantine [1975]. Organizational and psychological problems arising in the development of large software projects are discussed in Brooks [1974] and Weinberg [1971]. Kernighan and Plauger [1981] show how to build useful software tools for a programming environment.

† The symbol Ø stands for the empty set.

‡ We distinguish the abstract data type SET from the built-in set type of Pascal.

† The record has no known name because it was created by a call new(header), which made header point to this newly-created record. Internal to the machine, however, there is a memory address that can be used to locate the cell.

† Note the asymmetry between big-oh and big-omega notation. The reason such asymmetry is often useful is that there are many times when an algorithm is fast on many but not all inputs. For example, there are algorithms to test whether their input is of prime length that run very fast whenever that length is even, so we could not get a good lower bound on running time that held for all n � n0.

† Unless otherwise specified all logarithms are to the base 2. Note that O(logn) does not depend on the base of the logarithm since logan = clogbn, where c = logab.

† UNIX is a Trademark of Bell Laboratories.

‡ We could use an unabridged dictionary, but many misspellings are real words one has never heard of.

Table of Contents Go to Chapter 2


Data Structures and Algorithms: CHAPTER 2: Basic Abstract DataTypes

 Next Chapter Return to Table of Contents Previous Chapter

Basic Abstract DataTypes

In this chapter we shall study some of the most fundamental abstract data types. We consider lists, which are sequences of elements, and two special cases of lists: stacks, where elements are inserted and deleted at one end only, and queues, where elements are inserted at one end and deleted at the other. We then briefly study the mapping or associative store, an ADT that behaves as a function. For each of these ADT's we consider several implementations and compare their relative merits.

2.1 The Abstract Data Type "List"

Lists are a particularly flexible structure because they can grow and shrink on demand, and elements can be accessed, inserted, or deleted at any position within a list. Lists can also be concatenated together or split into sublists. Lists arise routinely in applications such as information retrieval, programming language translation, and simulation. Storage management techniques of the kind we discuss in Chapter 12 use list-processing techniques extensively. In this section we shall introduce a number of basic list operations, and in the remainder of this chapter present data structures for lists that support various subsets of these operations efficiently.

Mathematically, a list is a sequence of zero or more elements of a given type (which we generally call the elementtype). We often represent such a list by a comma-separated sequence of elements

al, a2, . . . ,an

where n � 0, and each ai is of type elementtype. The number n of elements is said to be the length of the list. Assuming n � 1, we say that a1 is the first element and an is the last element. If n = 0, we have an empty list, one which has no elements.

An important property of a list is that its elements can be linearly ordered according to their position on the list. We say ai precedes ai+1 for i = 1, 2, . . . , n-1, and ai follows ai-1 for i = 2, 3, . . . ,n. We say that the element ai is at position i. It is also convenient to postulate the existence of a position following the last element on a list. The function END(L) will return the position following position n in an n-element list L. Note that position END(L) has a distance from the beginning of the list that varies as the list grows or shrinks, while all other positions have a fixed distance from the beginning of the list.

To form an abstract data type from the mathematical notion of a list we must define a set of operations on objects of type LIST.† As with many other ADT's we discuss in this book, no one set of operations is suitable for all applications. Here, we shall give one representative set of operations. In the next section we shall offer several data structures to represent lists and we shall write procedures for the typical list operations in terms of these data structures.

To illustrate some common operations on lists, let us consider a typical application in which we have a mailing list from which we wish to purge duplicate entries. Conceptually, this problem can be solved quite simply: for each item on the list, remove all equivalent following items. To present this algorithm, however, we need to define operations that find the first element on a list, step through all successive elements, and retrieve and delete elements.

We shall now present a representative set of list operations. In what follows, L is a list of objects of type elementtype, x is an object of that type, and p is of type position. Note that "position" is another data type whose implementation will vary for different list implementations. Even though we informally think of positions as integers, in practice, they may have another representation.

  1. INSERT(x, p, L). Insert x at position p in list L, moving elements at p and following positions to the next higher position. That is, if L is al, a2, . . . ,an, then L becomes a1, a2,. . . ,ap- 1, x, ap, . . . ,an. If p is END(L), then L becomes a1, a2, . . . , an, x. If list L has no position p, the result is undefined.
  2. LOCATE(x, L). This function returns the position of x on list L. If x appears more than once, then the position of the first occurrence is returned. If x does not appear at all, then END(L) is returned.
  3. RETRIEVE(p, L). This function returns the element at position p on list L. The result is undefined if p = END(L) or if L has no position p. Note that the elements must be of a type that can be returned by a function if RETRIEVE is used. In practice, however, we can always modify RETRIEVE to return a pointer to an object of type elementtype.
  4. DELETE(p, L). Delete the element at position p of list L. If L is a1, a2, . . . ,an, then L becomes a1, a2, . . . ,ap- 1, ap+1, . . . ,an. The result is undefined if L has no position p or if p = END(L).
  5. NEXT(p, L) and PREVIOUS(p, L) return the positions following and preceding position p on list L. If p is the last position on L, then NEXT(p, L) = END(L). NEXT is undefined if p is END(L). PREVIOUS is undefined if p is 1. Both functions are undefined if L has no position p.
  6. MAKENULL(L). This function causes L to become an empty list and returns position END(L).
  7. FIRST(L). This function returns the first position on list L. If L is empty, the position returned is END(L).
  8. PRINTLIST(L). Print the elements of L in the order of occurrence.
Example 2.1. Let us write, using these operators, a procedure PURGE that takes a list as argument and eliminates duplicates from the list. The elements of the list are of type elementtype, and a list of such elements has type LIST, a convention we shall follow throughout this chapter. There is a function same(x,y), where x and y are of elementtype, that is true if x and y are "the same" and false if not. The notion of sameness is purposely left vague. If elementtype is real, for example, we might want same(x,y) true if and only if x = y. However, if elementtype is a record containing the account number, name, and address of a subscriber as in

   type
       elementtype = record
         acctno: integer;
         name: packed array [1..20] of char;
         address: packed array [1..50] of char
       end
then we might want same(x, y) to be true whenever x.acctno=y.acctno.†

Figure 2.1 shows the code for PURGE. The variables p and q are used to represent two positions in the list. As the program proceeds, duplicate copies of any elements to the left of position p have been deleted from the list. In one iteration of the loop (2)-(8), q is used to scan the list following position p to delete any duplicates of the element at position p. Then p is moved to the next position and the process is repeated.

In the next section we shall provide appropriate declarations for LIST and position, and implementations for the operations so that PURGE becomes a working program. As written, the program is independent of the manner in which lists are represented so we are free to experiment with various list implementations.

     procedure PURGE ( var L: LIST );
         { PURGE removes duplicate elements from list L }
         var
           p, q: position; { p will be the "current" position
              in L, and q will move ahead to find equal elements }
         begin
(1)        p := FIRST(L);
(2)        while p <> END(L) do begin
(3)            q := NEXT(p, L);
(4)            while q <> END(L) do
(5)                if same(RETRIEVE(p, L), RETRIEVE(q,
L)) then
(6)                        DELETE(q, L)
                   else
(7)                    q := NEXT(q, L);
(8)            p := NEXT(p, L)
           end
         end; { PURGE }

Fig. 2.1. Program to remove duplicates.

A point worth observing concerns the body of the inner loop, lines (4)-(7) of Fig. 2.1. When we delete the element at position q at line (6), the elements that were at positions q+1, q+2, . . . , and so on, move up one position in the list. In particular, should q happen to be the last position on L, the value of q would become END(L). If we then executed line (7), NEXT(END(L), L) would produce an undefined result. Thus, it is essential that either (6) or (7), but never both, is executed between the tests for q = END(L) at line (4).

2.2 Implementation of Lists

In this section we shall describe some data structures that can be used to represent lists. We shall consider array, pointer, and cursor implementations of lists. Each of these implementations permits certain list operations to be done more efficiently than others.

Array Implementation of Lists

In an array implementation of a list, the elements are stored in contiguous cells of an array. With this representation a list is easily traversed and new elements can be appended readily to the tail of the list. Inserting an element into the middle of the list, however, requires shifting all following elements one place over in the array to make room for the new element. Similarly, deleting any element except the last also requires shifting elements to close up the gap.

Fig. 2.2. Array implementation of a list.

In the array implementation we define the type LIST to be a record having two fields. The first field is an array of elements whose length is sufficient to hold the maximum size list that will be encountered. The second field is an integer last indicating the position of the last list element in the array. The i th element of the list is in the ith cell of the array, for 1 � i � last, as shown in Fig. 2.2. Positions in the list are represented by integers, the ith position by the integer i. The function END(L) has only to return last + 1. The important declarations are:

   const
      maxlength = 100 { some suitable constant };
   type
      LIST = record
         elements: array[1..maxlength] of elementtype;
         last: integer
      end;
      position = integer;
   function END ( var L: LIST ): position;†
      begin
         return (L.last + 1)
      end; { END }

Figure 2.3 shows how we might implement the operations INSERT, DELETE, and LOCATE using this array-based implementation. INSERT moves the elements at locations p,p+1, . . . , last into locations p+1, p+2, . . . ,last+1 and then inserts the new element at location p. If there is no room in the array for an additional element, the routine error is invoked, causing its argument to be printed, followed by termination of execution of the program. DELETE removes the element at position p by moving the elements at positions p + 1, p + 2, . . . , last into positions p, p+ 1, . . . , last-1. LOCATE sequentially scans the array to look for a given element. If the element is not found, LOCATE returns last+ 1.

It should be clear how to encode the other list operations using this implementation of lists. For example, FIRST always returns 1; NEXT returns one more than its argument and PREVIOUS returns one less, each first checking that the result is in range; MAKENULL(L) sets L.last to 0.

If procedure PURGE of Fig. 2.1 is preceded by

  1. the definitions of elementtype and the function same,
  2. the definitions of LIST, position and END(L) as above,
  3. the definition of DELETE from Fig. 2.3, and
  4. suitable definitions for the trivial procedures FIRST, NEXT, and RETRIEVE,
then a working procedure PURGE results.

At first, it may seem tedious writing procedures to govern all accesses to the underlying structures. However, if we discipline ourselves to writing programs in terms of the operations for manipulating abstract data types rather than making use of particular implementation details, then we can modify programs more readily by reimplementing the operations rather than searching all programs for places where we have made accesses to the underlying data structures. This flexibility can be particularly important in large software efforts, and the reader should not judge the concept by the necessarily tiny examples found in this book.

Pointer Implementation of Lists

Our second implementation of lists, singly-linked cells, uses pointers to link successive list elements. This implementation frees us from using contiguous memory for storing a list and hence from shifting elements to make room for new elements or to close up gaps created by deleted elements. However, one price we pay is extra space for pointers.

In this representation, a list is made up of cells, each cell consisting of an element of the list and a pointer to the next cell on the list. If the list is a1, a2, . . . , an, the cell holding ai has a pointer to the cell holding ai+1, for

     procedure INSERT ( x: elementtype; p: position; var L:
LIST );
        { INSERT places x at position p in list L }
    var
        q: position;
       begin
            if L.last > = maxlength then
             error('list is full')
            else if (p > L.last + 1) or (p < 1) then
         error('position does not exist')
            else begin
         for q := L.last downto p do
             { shift elements at p, p + 1,  . .  . down one position }
                     L.elements[q + 1 ]: = L.elements[q];
                 L.last := L.last + 1;
                 L.elements[p] := x
            end
        end; { INSERT }

     procedure DELETE ( p: position; var L: LIST );
        { DELETE removes the element at position p of list L }
        var
           q: position;
        begin
           if (p > L.last) or (p < 1) then
               error('position does not exist')
           else begin
                L.last := L.last - 1;
                for q := p to L.last do
                    { shift elements at p + 1, p + 2,... up one position }
                    L.elements[q] := L.elements[q + 1]
           end
        end; { DELETE }

     function LOCATE ( x: elementtype; L: LIST ): position;
         { LOCATE returns the position of x on list L }
         var
             q: position;
         begin
             for q := 1 to L.last do
                 if L.elements[q] = x then
                      return (q);
             return (L.last + 1) { if not found }
         end; { LOCATE }

Fig. 2.3. Array-based implementation of some list operations.

i = 1, 2 , . . . , n-1. The cell holding an has a nil pointer. There is also a header cell that points to the cell holding a1; the header holds no element.† In the case of an empty list, the header's pointer is nil, and there are no other cells. Figure 2.4 shows a linked list of this form.

Fig. 2.4. A linked list.

For singly-linked lists, it is convenient to use a definition of position that is somewhat different than the definition of position in an array implementation. Here, position i will be a pointer to the cell holding the pointer to ai for i = 2, 3 , . . . , n. Position 1 is a pointer to the header, and position END(L) is a pointer to the last cell of L.

The type of a list happens to be the same as that of a position -- it is a pointer to a cell, the header in particular. We can formally define the essential parts of a linked list data structure as follows.

   type
       celltype = record
            element: elementtype;
            next: � celltype
       end;
       LIST = � celltype;
       position = � celltype;

The function END(L) is shown in Fig. 2.5. It works by moving pointer q down the list from the header, until it reaches the end, which is detected by the fact that q points to a cell with a nil pointer. Note that this implementation of END is inefficient, as it requires us to scan the entire list every time we need to compute END(L). If we need to do so frequently, as in the PURGE program of Fig. 2.1, we could either

  1. Use a representation of lists that includes a pointer to the last cell, or
  2. Replace uses of END(L) where possible. For example, the condition p <> END(L) in line (2) of Fig. 2.1 could be replaced by p �.next <> nil, at a cost of making that program dependent on one particular implementation of lists.

    function END ( L: LIST ): position;
            { END returns a pointer to the last cell of L }
            var
               q: position;
            begin
(1)            q := L;
(2)            while q�.next <> nil
do
(3)                q := q�.next;
(4)            return (q)
            end; { END }

Fig. 2.5. The function END.

Figure 2.6 contains routines for the four operations INSERT, DELETE, LOCATE, and MAKENULL using this pointer implementation of lists. The other operations can be implemented as one-step routines, with the exception of PREVIOUS, which requires a scan of the list from the beginning. We leave these other routines as exercises. Note that many of the commands do not use parameter L, the list, and we omit it from those that do not.

The mechanics of the pointer manipulations of the INSERT procedure in Fig. 2.6 are shown in Fig. 2.7. Figure 2.7(a) shows the situation before executing INSERT. We wish to insert a new element in front of the cell containing b, so p is a pointer to the list cell that contains the pointer to b. At line (1), temp is set to point to the cell containing b. At line (2) a new list cell is created and the next field of the cell containing a is made to point to this cell. At line (3) the element field of the newly-created cell is made to hold x, and at line (4) the next field is given the value of temp, thus making it point to the cell containing b. Figure 2.7(b) shows the result of executing INSERT. The new pointers are shown dashed, and marked with the step at which they were created.

The DELETE procedure is simpler. Figure 2.8 shows the pointer manipulations of the DELETE procedure in Fig. 2.6. Old pointers are solid and the new pointers dashed.

We should note that a position in a linked-list implementation of a list behaves differently from a position in an array implementation. Suppose we have a list with three elements a, b, c and a variable p, of type position, which currently has position 3 as its value; i.e., it points to the cell holding b, and thus represents the position of c. If we execute a command to insert x at position 2, the list becomes a, x, b, c, and element b now occupies position 3. If we use the array implementation of lists described earlier, b and c would be moved down the array, so b would indeed occupy the third position.

    procedure INSERT ( x: elementtype; p: position);
           var
               temp : position;
           begin
(1)            temp := p �.next;
(2)            new(p �.next );
(3)            p �.next �.element := x;
(4)            p �.next �.next := temp
           end; { INSERT }

    procedure DELETE ( p: position );
           begin
               p �.next := p �.next �.next
           end; { DELETE }

    function LOCATE ( x: elementtype; L: LIST ): position;
           var
               p: position;
           begin
               p := L;
               while p �.next <> nil
do
                    if p �.next �.element = x then
                         return (p)
                    else
                         p := p �.next;
               return (p) { if not found }
           end; { LOCATE }

    function MAKENULL ( var L: LIST ) : position;
           begin
                new(L);
                L �.next := nil;
                return (L )
           end; { MAKENULL }

Fig. 2.6. Some operations using the linked-list implementation.

However, if the linked-list implementation is used, the value of p, which is a pointer to the cell containing b, does not change because of the insertion, so afterward, the value of p is "position 4," not 3. This position variable must be updated, if it is to be used subsequently as the position of b.†

Fig. 2.7. Diagram of INSERT.

Fig. 2.8. Diagram of DELETE.

Comparison of Methods

We might wonder whether it is best to use a pointer-based or array-based implementation of lists in a given circumstance. Often the answer depends on which operations we intend to perform, or on which are performed most frequently. Other times, the decision rests on how long the list is likely to get. The principal issues to consider are the following.

  1. The array implementation requires us to specify the maximum size of a list at compile time. If we cannot put a bound on the length to which the list will grow, we should probably choose a pointer-based implementation.
  2. Certain operations take longer in one implementation than the other. For example, INSERT and DELETE take a constant number of steps for a linked list, but require time proportional to the number of following elements when the array implementation is used. Conversely, executing PREVIOUS and END require constant time with the array implementation, but time proportional to the length of the list if pointers are used.
  3. If a program calls for insertions or deletions that affect the element at the position denoted by some position variable, and the value of that variable will be used later on, then the pointer representation cannot be used as we have described it here. As a general principle, pointers should be used with great care and restraint.
  4. The array implementation may waste space, since it uses the maximum amount of space independent of the number of elements actually on the list at any time. The pointer implementation uses only as much space as is needed for the elements currently on the list, but requires space for the pointer in each cell. Thus, either method could wind up using more space than the other in differing circumstances.

Cursor-Based Implementation of Lists

Some languages, such as Fortran and Algol, do not have pointers. If we are working with such a language, we can simulate pointers with cursors, that is, with integers that indicate positions in arrays. For all the lists of elements whose type is elementtype, we create one array of records; each record consists of an element and an integer that is used as a cursor. That is, we define

   var
       SPACE: array [1..maxlength] of record
           element: elementtype;
           next: integer
       end

If L is a list of elements, we declare an integer variable say Lhead, as a header for L. We can treat Lhead as a cursor to a header cell in SPACE with an empty element field. The list operations can then be implemented as in the pointer-based implementation just described.

Here, we shall describe an alternative implementation that avoids the use of header cells by making special cases of insertions and deletions at position 1. This same technique can also be used with pointer-based linked-lists to avoid the use of header cells. For a list L, the value of SPACE[Lhead].element is the first element of L. The value of SPACE[Lhead].next is the index of the cell containing the second element, and so on. A value of 0 in either Lhead or the field next indicates a "nil pointer"; that is, there is no element following.

A list will have type integer, since the header is an integer variable that represents the list as a whole. Positions will also be of type integer. We adopt the convention that position i of list L is the index of the cell of SPACE holding element i-1 of L, since the next field of that cell will hold the cursor to element i. As a special case, position 1 of any list is represented by 0. Since the name of the list is always a parameter of operations that use positions, we can distinguish among the first positions of different lists. The position END(L) is the index of the last element on L.

Figure 2.9 shows two lists, L = a, b, c and M = d, e, sharing the array SPACE, with maxlength = 10. Notice that all the cells of the array that are not on either list are linked on another list called available. This list is necessary so we can obtain an empty cell when we want to insert into some list, and so we can have a place to put deleted cells for later reuse.

Fig. 2.9. A cursor implementation of linked lists.

To insert an element x into a list L we take the first cell in the available list and place it in the correct position of list L. Element x is then placed in the element field of this cell. To delete an element x from list L we remove the cell containing x from L and return it to the beginning of the available list. These two actions can be viewed as special cases of the act of taking a cell C pointed to by one cursor p and causing another cursor q to point to C, while making p point where C had pointed and making C point where q had pointed. Effectively, C is inserted between q and whatever q pointed to. For example, if we delete b from list L in Fig. 2.9, C is row 8 of SPACE, p is SPACE[5].next, and q is available. The cursors before (solid) and after (dashed) this action are shown in Fig. 2.10, and the code is embodied in the function move of Fig. 2.11, which performs the move if C exists and returns false if C does not exist.

Figure 2.12 shows the procedures INSERT and DELETE, and a procedure initialize that links the cells of the array SPACE into an available space list. These procedures omit checks for errors; the reader may insert them as an exercise. Other operations are left as exercises and are similar to those for pointer-based linked lists.

Fig. 2.10. Moving a cell C from one list to another.

    function move ( var p, q: integer ): boolean;
        { move puts cell pointed to by p ahead of cell pointed to by q }
            var
            temp: integer;
            begin
                if p = 0 then begin { cell nonexistent }
                     writeln('cell does not exist');
                     return (false)
                end
                else begin
                     temp := q;
                     q := p;
                     p := SPACE[q ].next;
                     SPACE[q ].next := temp;
                     return (true)
                end
            end; { move }

Fig. 2.11. Code to move a cell.

Doubly-Linked Lists

In a number of applications we may wish to traverse a list both forwards and backwards efficiently. Or, given an element, we may wish to determine the preceding and following elements quickly. In such situations we might wish

    procedure INSERT ( x: elementtype; p: position; var L:
LIST );
        begin
        if p = 0 then begin
             { insert at first position }
             if move(available, L) then
             SPACE[L].element := x
        end
        else { insert at position other than first }
             if move(available, SPACE[p].next)
then
             { cell for x now pointed to by SPACE[p].next }
             SPACE[SPACE[p].next].element :=
x
        end; { INSERT }

    procedure DELETE ( p: position; var L: LIST );
        begin
        if p = 0 then
             move(L, available)
        else
             move(SPACE[p].next, available)
        end; { DELETE }

    procedure initialize;
        { initialize links SPACE into one available list }
        var
        i: integer;
        begin
        for i := maxsize - 1 downto 1 do
            SPACE[i].next := i + 1;
        available := 1;
        SPACE[maxsize].next := 0 { mark end of available list }
        end; { initialize }

Fig. 2.12. Some procedures for cursor-based linked lists.

to give each cell on a list a pointer to both the next and previous cells on the list, as suggested by the doubly-linked list in Fig. 2.13. Chapter 12 mentions some specific situations where doubly-linked lists are essential for efficiency.

Fig. 2.13. A doubly linked list.

Another important advantage of doubly linked lists is that we can use a pointer to the cell containing the ith element of a list to represent position i, rather than using the less natural pointer to the previous cell. The only price we pay for these features is the presence of an additional pointer in each cell, and somewhat lengthier procedures for some of the basic list operations. If we use pointers (rather than cursors), we may declare cells consisting of an element and two pointers by

   type
       celltype = record
         element: elementtype;
         next, previous: � celltype
       end;
       position = � celltype;

A procedure for deleting an element at position p in a doubly-linked list is given in Fig. 2.14. Figure 2.15 shows the changes in pointers caused by Fig. 2.14, with old pointers drawn solid and new pointers drawn dashed, on the assumption that the deleted cell is neither first nor last.† We first locate the preceding cell using the previous field. We make the next field of this cell point to the cell following the one in position p. Then we make the previous field of this following cell point to the cell preceding the one in position p. The cell pointed to by p becomes useless and should be reused automatically by the Pascal run-time system if needed.

    procedure DELETE (var p: position );
          begin
              if p�.previous <> nil
then
                     { deleted cell not the first }
                     p�.previous�.next: = p�.next;
              if p�.next <> nil
then
                     { deleted cell not the last }
                     p�.next�.previous : = p�.previous
          end; { DELETE }

Fig. 2.14. Deletion from a doubly linked list.

Fig. 2.15. Pointer changes for implementation of a deletion.

2.3 Stacks

A stack is a special kind of list in which all insertions and deletions take place at one end, called the top. Other names for a stack are "pushdown list," and "LIFO" or "last-in-first-out" list. The intuitive model of a stack is a pile of poker chips on a table, books on a floor, or dishes on a shelf, where it is only convenient to remove the top object on the pile or add a new one above the top. An abstract data type in the STACK family often includes the following five operations.

  1. MAKENULL(S). Make stack S be an empty stack. This operation is exactly the same as for general lists.
  2. TOP(S). Return the element at the top of stack S. If, as is normal, we identify the top of a stack with position 1, then TOP(S) can be written in terms of list operations as RETRIEVE(FIRST(S), S).
  3. POP(S). Delete the top element of the stack, that is, DELETE(FIRST(S), S). Sometimes it is convenient to implement POP as a function that returns the element it has just popped, although we shall not do so here.
  4. PUSH(x, S). Insert the element x at the top of stack S. The old top element becomes next-to-top, and so on. In terms of list primitives this operation is INSERT(x, FIRST(S), S).
  5. EMPTY(S). Return true if S is an empty stack; return false otherwise.
Example 2.2. Text editors always allow some character (for example, "backspace") to serve as an erase character, which has the effect of canceling the previous uncanceled character. For example, if '#' is the erase character, then the string abc#d##e is really the string ae. The first '#' cancels c, the second d, and the third b.

Text editors also have a kill character, whose effect is to cancel all previous characters on the current line. For the purposes of this example, we shall use '@' as the kill character.

A text editor can process a line of text using a stack. The editor reads one character at a time. If the character read is neither the kill nor the erase character, it pushes the character onto the stack. If the character read is the erase character, the editor pops the stack, and if it is the kill character, the editor makes the stack empty. A program that executes these actions is shown in Fig. 2.16.

     procedure EDIT;
          var
               S: STACK;
               c: char;
          begin
               MAKENULL(S);
               while not eoln do begin
                      read(c);
                      if c = '#' then
                          POP(S)
                      else if c = '@' then
                          MAKENULL(S)
                      else { c is an ordinary character }
                          PUSH(c, S)
               end;
               print S in reverse order
          end; { EDIT }

Fig. 2.16. Program to carry out effects of erase and kill characters.

In this program, the type of STACK must be declared as a list of characters. The process of writing the stack in reverse order in the last line of the program is a bit tricky. Popping the stack one character at a time gets the line in reverse order. Some stack implementations, such as the array-based implementation to be discussed next, allow us to write a simple procedure to print the stack from the bottom. In general, however, to reverse a stack, we must pop each element and push it onto another stack; the characters can then be popped from the second stack and printed in the order they are popped.

An Array Implementation of Stacks

Every implementation of lists we have described works for stacks, since a stack with its operations is a special case of a list with its operations. The linked-list representation of a stack is easy, as PUSH and POP operate only on the header cell and the first cell on the list. In fact, headers can be pointers or cursors rather than complete cells, since there is no notion of "position" for stacks, and thus no need to represent position 1 in a way analogous to other positions.

However, the array-based implementation of lists we gave in Section 2.2 is not a particularly good one for stacks, as every PUSH or POP requires moving the entire list up or down, thus taking time proportional to the number of elements on the stack. A better arrangement for using an array takes account of the fact that insertions and deletions occur only at the top. We can anchor the bottom of the stack at the bottom (high-indexed end) of the array, and let the stack grow towards the top (low-indexed end) of the array. A cursor called top indicates the current position of the first stack element. This idea is shown in Fig. 2.17.

Fig. 2.17. An array implementation of a stack.

For this array-based implementation of stacks we define the abstract data type STACK by

   type
       STACK = record
         top: integer;
         elements: array[1..maxlength] of elementtype
       end;
An instance of the stack consists of the sequence elements[top], elements[top+1] , . . . , elements[maxlength]. Note that if top = maxlength + 1, then the stack is empty.

The five typical stack operations are implemented in Fig. 2.18. Note that for TOP to return an elementtype, that type must be legal as the result of a function. If not, TOP must be a procedure that modifies its second argument by assigning it the value TOP(S), or a function that returns a pointer to elementtype.

     procedure MAKENULL ( var S: STACK );
          begin
               S.top := maxlength + 1
          end; { MAKENULL }

     function EMPTY ( S: STACK ): boolean;
          begin
               if S.top > maxlength then
                    return (true)
               else
                    return (false)
          end; { EMPTY }

     function TOP ( var S: STACK ) : elementtype;
          begin
               if EMPTY(S) then
                    error('stack is empty')
               else
                     return(S.elements[S.top])
          end; { TOP }

     procedure POP ( var S: STACK );
          begin
               if EMPTY(S) then
                     error('stack is empty')
               else
                     S.top := S.top + 1
          end; { POP }

     procedure PUSH ( x: elementtype; var S: STACK );
          begin
               if S.top = 1 then
                     error('stack is full')
               else begin
                     S.top := S.top - 1;
                     S.elements[S.top]: = x
               end
          end; { PUSH }

Fig. 2.18. Operations on stacks

2.4 Queues

A queue is another special kind of list, where items are inserted at one end (the rear) and deleted at the other end (the front). Another name for a queue is a "FIFO" or "first-in first-out" list. The operations for a queue are analogous to those for a stack, the substantial differences being that insertions go at the end of the list, rather than the beginning, and that traditional terminology for stacks and queues is different. We shall use the following operations on queues.

  1. MAKENULL(Q) makes queue Q an empty list.
  2. FRONT(Q) is a function that returns the first element on queue Q. FRONT(Q) can be written in terms of list operations as RETRIEVE(FIRST(Q), Q).
  3. ENQUEUE(x, Q) inserts element x at the end of queue Q. In terms of list operations, ENQUEUE(x, Q) is INSERT(x, END(Q), Q).
  4. DEQUEUE(Q) deletes the first element of Q; that is, DEQUEUE(Q) is DELETE(FIRST(Q), Q).
  5. EMPTY(Q) returns true if and only if Q is an empty queue.

A Pointer Implementation of Queues

As for stacks, any list implementation is legal for queues. However, we can take advantage of the fact that insertions are only done at the rear to make ENQUEUE efficient. Instead of running down the list from beginning to end each time we wish to enqueue something, we can keep a pointer (or cursor) to the last element. As for all kinds of lists, we also keep a pointer to the front of the list; for queues that pointer is useful for executing FRONT and DEQUEUE commands. In Pascal, we shall use a dummy cell as a header and have the front pointer point to it. This convention allows us to handle an empty queue conveniently.

Here we shall develop an implementation for queues that is based on Pascal pointers. The reader may develop a cursor-based implementation that is analogous, but we have available, in the case of queues, a better array-oriented representation than would be achieved by mimicking pointers with cursors directly. We shall discuss this so-called "circular array" implementation at the end of this section. To proceed with the pointer-based implementation, let us define cells as before:

   type
       celltype = record
         element: elementtype;
         next: � celltype
       end;
Then we can define a list to consist of pointers to the front and rear of the queue. The first cell on a queue is a header cell in which the element field is ignored. This convention, as mentioned above, allows a simple representation for an empty queue. We define:

   type
       QUEUE = record
           front, rear: � celltype
       end;
Figure 2.19 shows programs for the five queue operations. In MAKENULL the first statement new(Q.front) allocates a variable of type celltype and assigns its address to Q.front. The second statement puts nil into the next field of that cell. The third statement makes the header both the first and last cell of the queue.

The procedure DEQUEUE(Q) deletes the first element of Q by disconnecting the old header from the queue. The first element on the list becomes the new dummy header cell.

Figure 2.20 shows the results created by the sequence of commands MAKENULL(Q), ENQUEUE(x, Q), ENQUEUE(y, Q), DEQUEUE(Q). Note that after dequeueing, the element x being in the element field of the header cell, is no longer considered part of the queue.

A Circular Array Implementation of Queues

The array representation of lists discussed in Section 2.2 can be used for queues, but it is not very efficient. True, with a pointer to the last element, we can execute ENQUEUE in a fixed number of steps, but DEQUEUE, which removes the first element, requires that the entire queue be moved up one position in the array. Thus DEQUEUE takes W(n) time if the queue has length n.

To avoid this expense, we must take a different viewpoint. Think of an array as a circle, where the first position follows the last, as suggested in Fig. 2.21. The queue is found somewhere around the circle in consecutive positions,† with the rear of the queue somewhere clockwise from the front. To enqueue an element, we move the Q.rear pointer one position clockwise and write the element in that position. To dequeue, we simply move Q.front one position clockwise. Thus, the queue migrates in a clockwise direction as we enqueue and dequeue elements. Observe that the procedures ENQUEUE and DEQUEUE can be written to take some constant number of steps if the circular array model is used.

There is one subtlety that comes up in the representation of Fig 2.21 and in any minor variation of that strategy (e.g., if Q.rear points one position clockwise of the last element, rather than to that element itself). The problem is that there is no way to tell an empty queue from one that occupies the entire circle, short of maintaining a bit that is true if and only if the queue is empty. If we are not willing to maintain this bit, we must prevent the queue from ever filling the entire array.

To see why, suppose the queue of Fig. 2.21 had maxlength elements. Then Q.rear would point one position counterclockwise of Q.front. What if the queue were empty? To see how an empty queue is represented, let us first consider a queue of one element. Then Q.front and Q.rear point to the same position. If we dequeue the one element, Q.front moves one position

     procedure MAKENULL ( var Q: QUEUE );
          begin
              new(Q.front); { create header cell }
              Q.front�.next := nil;
              Q.rear := Q.front { header is both first and last cell }
          end; { MAKENULL }

     function EMPTY ( Q: QUEUE): boolean;
          begin
              if Q.front = Q.rear then
                  return (true)
              else
                  return (false)
          end; { EMPTY }

     function FRONT ( Q: QUEUE ): elementtype;
          begin
              if EMPTY(Q) then
                  error('queue is empty')
              else
                  return (Q.front�.next�.element)
          end; { FRONT }

     procedure ENQUEUE ( x: elementtype; var Q: QUEUE );
          begin
              new(Q.rear�.next); { add new cell to
rear of queue }
              Q.rear: = Q.rear�.next;
              Q.rear�.element := x;
              Q.rear�.next := nil
          end; { ENQUEUE }

     procedure DEQUEUE ( var Q: QUEUE );
          begin
              if EMPTY(Q) then
                  error('queue is empty')
              else
                  Q.front := Q.front�.next
          end; { DEQUEUE }

Fig. 2.19. Implementation of queue commands.

clockwise; forming an empty queue. Thus an empty queue has Q.rear one position counterclockwise of Q.front, which is exactly the same relative position as when the queue had maxlength elements. We thus see that even though the array has maxlength places, we cannot let the queue grow longer than maxlength-1, unless we introduce another mechanism to distinguish empty queues.

Fig. 2.20. A sequence of queue operations.

Let us now write the five queue commands using this representation for a queue. Formally, queues are defined by:

   type
       QUEUE = record
         element: array[1..maxlength] of elementtype;
         front, rear: integer
       end;

The commands appear in Fig. 2.22. The function addone(i) adds one to position i in the circular sense.

Fig. 2.21. A circular implementation of queues.

2.5 Mappings

A mapping or associative store is a function from elements of one type, called the domain type to elements of another (possibly the same) type, called the range type. We express the fact that the mapping M associates element r of range type rangetype with element d of domain type domaintype by M(d) = r.

Certain mappings such as square(i) = i2 can be implemented easily as a Pascal function by giving an arithmetic expression or other simple means for calculating M(d) from d. However, for many mappings there is no apparent way to describe M(d) other than to store for each d the value of M(d). For example, to implement a payroll function that associates with each employee a weekly salary seems to require that we store the current salary for each employee. In the remainder of this section we describe a method of implementing functions such as the "payroll" function.

Let us consider what operations we might wish to perform on a mapping M. Given an element d of some domain type, we may wish to obtain M(d) or know whether M(d) is defined (i.e., whether d is currently in the domain of M). Or we may wish to enter new elements into the current domain of M and state their associated range values. Alternatively, we might wish to change the value of M(d). We also need a way to initialize a mapping to the null mapping, the mapping whose domain is empty. These operations are summarized by the following three commands.

  1. MAKENULL(M). Make M be the null mapping.
  2. ASSIGN(M, d, r). Define M(d) to be r, whether or not M(d) was defined previously.
    3.
     function addone ( i: integer ): integer;
          begin
              return (( i mod maxlength ) + 1)
          end; { addone }

     procedure MAKENULL ( var Q: QUEUE );
          begin
              Q.front := 1;
              Q.rear := maxlength
          end; { MAKENULL }

     function EMPTY ( var Q: QUEUE ): boolean;
          begin
              if addone(Q.rear) = Q.front then
                 return (true)
              else
                 return (false)
          end; { EMPTY }

     function FRONT ( var Q: QUEUE ): elementtype;
         begin
              if EMPTY(Q) then
                 error('queue is empty')
              else
                 return (Q.elements[Q.front])
         end; { FRONT }

     procedure ENQUEUE ( x: elementtype; var Q: QUEUE );
         begin
              if addone (addone(Q.rear)) = Q.front then
                 error('queue is full')
              else begin
                 Q.rear := addone(Q.rear);
                 Q.elements[Q.rear] := x
              end
         end; { ENQUEUE )

     procedure DEQUEUE ( var Q: QUEUE );
         begin
              if EMPTY(Q) then
                 error('queue is empty')
         else
                 Q.front := addone(Q.front)
         end; { DEQUEUE }

    Fig. 2.22. Circular queque implementation.

  3. COMPUTE(M, d, r). Return true and set variable r to M(d) if the latter is defined; return false otherwise.

Array Implementation of Mappings

Many times, the domain type of a mapping will be an elementary type that can be used as an index type of an array. In Pascal, the index types include all the finite subranges of integers, like 1..100 or 17..23, the type char and subranges of char like 'A'..'Z', and enumerated types like (north, east, south, west). For example, a cipher-breaking program might keep a mapping crypt, with 'A'..'Z' as both its domain type and its range type, such that crypt (plaintext) is the letter currently guessed to stand for the letter plaintext.

Such mappings can be implemented simply by arrays, provided there is some range type value that can stand for "undefined." For example, the above mapping crypt might be defined to have range type char, rather than 'A'..'Z', and '?' could be used to denote "undefined."

Suppose the domain and range types are domaintype and rangetype, and domaintype is a basic Pascal type. Then we can define the type MAPPING (strictly speaking, mapping from domaintype to rangetype) by the declaration

   type
       MAPPING = array[domaintype] of rangetype;
On the assumption that "undefined" is a constant of rangetype, and that firstvalue and lastvalue are the first and last values in domaintype,† we can implement the three commands on mappings as in Fig. 2.23.

List Implementations of Mappings

There are many possible implementations of mappings with finite domains. For example, hash tables are an excellent choice in many situations, but one whose discussion we shall defer to Chapter 4. Any mapping with a finite domain can be represented by the list of pairs (d1, r1), (d2, r2), . . . , (dk, rk), where d1, d2, . . . , dk are all the current members of the domain, and ri is the value that the mapping associates with di, for i = 1, 2 , . . . ,k. We can then use any implementation of lists we choose for this list of pairs.

To be precise, the abstract data type MAPPING can be implemented by lists of elementtype, if we define

   type
       elementtype = record
         domain: domaintype;
         range: rangetype
       end;
and then define MAPPING as we would define type LIST (of elementtype) in

      procedure MAKENULL ( var M: MAPPING );
          var
               i: domaintype;
          begin
               for i := firstvalue to lastvalue do
                    M[i] := undefined
          end; { MAKENULL }

      procedure ASSIGN ( var M: MAPPING;
               d: domaintype; r: rangetype );
          begin
               M[d] := r
          end; { ASSIGN }

      function COMPUTE ( var M: MAPPING;
               d: domaintype; var r: rangetype ): boolean;
          begin
               if M[d] = undefined then
                    return (false)
               else begin
                    r := M[d];
                    return (true)
               end
          end; { COMPUTE }

Fig. 2.23. Array implementation of mappings.

whatever implementation of lists we choose. The three mapping commands are defined in terms of commands on type LIST in Fig. 2.24.

2.6 Stacks and Recursive Procedures

One important application of stacks is in the implementation of recursive procedures in programming languages. The run-time organization for a programming language is the set of data structures used to represent the values of the program variables during program execution. Every language that, like Pascal, allows recursive procedures, uses a stack of activation records to record the values for all the variables belonging to each active procedure of a program. When a procedure P is called, a new activation record for P is placed on the stack, regardless of whether there is already another activation record for P on the stack. When P returns, its activation record must be on top of the stack, since P cannot return until all procedures it has called have returned to P. Thus, we may pop the activation record for this call of P to cause control to return to the point at which P was called (that point, known as the return address, was placed in P's activation record when the call to P

   procedure MAKENULL ( var M: MAPPING );
       { same as for list }

   procedure ASSIGN ( var M: MAPPING;
          d: domaintype; r: rangetype );
       var
          x: elementtype; { the pair (d, r) }
          p: position; { used to go from first to last position on list M }
       begin
          x.domain := d;
          x.range := r;
          p := FIRST(M);
          while p <> END(M) do
              if RETRIEVE(p, M).domain = d then
                  DELETE(p, M) { remove element with domain d }
              else
                  p := NEXT(p, M);
          INSERT(x, FIRST(M), M) { put (d, r) at front of list }
       end; { ASSIGN }

   function COMPUTE ( var M: MAPPING;
          d: domaintype; var r: rangetype ): boolean;
       var
          p: position;
       begin
          p := FIRST(M);
          while p <> END(M) do begin
              if RETRIEVE(p, M).domain = d then begin
                  r := RETRIEVE(p, M).range;
                  return (true)
              end;
              p := NEXT(p, M)
          end;
          return (false) { if d is not in domain }
       end; { COMPUTE }

Fig. 2.24. Mapping implementation in terms of lists.

was made).

Recursion simplifies the structure of many programs. In some languages, however, procedure calls are much more costly than assignment statements, so a program may run faster by a large constant factor if we eliminate recursive procedure calls from it. We do not advocate that recursion or other procedure calls be eliminated habitually; most often the structural simplicity is well worth the running time. However, in the most frequently executed portions of programs, we may wish to eliminate recursion, and it is the purpose of this discussion to illustrate how recursive procedures can be converted to nonrecursive ones by the introduction of a user-defined stack.

Example 2.3. Let us consider recursive and nonrecursive solutions to a simplified version of the classic knapsack problem in which we are given target t and a collection of positive integer weights w1, w2 , . . . , wn. We are asked to determine whether there is some selection from among the weights that totals exactly t. For example, if t = 10, and the weights are 7, 5, 4, 4, and 1, we could select the second, third, and fifth weights, since 5+4+ 1 = 10.

The image that justifies the name "knapsack problem" is that we wish to carry on our back no more than t pounds, and we have a choice of items with given weights to carry. We presumably find the items' utility to be proportional to their weight,† so we wish to pack our knapsack as closely to the target weight as we can.

In Fig. 2.25 we see a function knapsack that operates on an array

weights : array [l..n] of integer.

A call to knapsack(s, i) determines whether there is a collection of the elements in weight[i] through weight[n] that sums to exactly s, and prints these weights if so. The first thing knapsack does is determine if it can respond immediately. Specifically, if s = 0, then the empty set of weights is a solution. If s < 0, there can be no solution, and if s > 0 and i > n, then we are out of weights to consider and therefore cannot find a sum equal to s.

If none of these cases applies, then we simply call knapsack(s-wi, i + 1) to see if there is a solution that includes wi. If there is such a solution, then the total problem is solved, and the solution includes wi, so we print it. If there is no solution, then we call knapsack(s, i + 1) to see if there is a solution that does not use wi.

Elimination of Tail Recursion

Often, we can eliminate mechanically the last call a procedure makes to itself. If a procedure P(x) has, as its last step, a call to P(y), then we can replace the call to P(y) by an assignment x := y, followed by a jump to the beginning of the code for P. Here, y could be an expression, but x must be a parameter passed by value, so its value is stored in a location private to this call to P ‡ P could have more than one parameter, of course, and if so, they are each treated exactly as x and y above.

This change works because rerunning P with the new value of x has exactly the same effect as calling P(y) and then returning from that call.

      function knapsack ( target: integer; candidate: integer ): boolean;
          begin
(1)          if target = 0 then
(2)              return (true)
(3)          else if (target < 0) or (candidate > n) then
(4)              return (false)
             else { consider solutions with and without candidate }
(5)          if knapsack(target - weights[candidate], candidate + 1)
then
                 begin
(6)                  writeln(weights[candidate]);
(7)                  return (true)
                 end
             else { only possible solution is without candidate }
(8)              return (knapsack(target, candidate + 1))
          end; { knapsack }

Fig. 2.25. Recursive knapsack solution.

Notice that the fact that some of P's local variables have values the second time around is of no consequence. P could not use any of those values, or had we called P(y) as originally intended, the value used would not have been defined.

Another variant of tail recursion is illustrated by Fig. 2.25, where the last step of the function knapsack just returns the result of calling itself with other parameters. In such a situation, again provided the parameters are passed by value (or by reference if the same parameter is passed to the call), we can replace the call by assignments to the parameters and a jump to the beginning of the function. In the case of Fig. 2.25, we can replace line (8) by

        candidate:= candidate + 1;
        goto beginning
where beginning stands for a label to be assigned to statement (1). Note no change to target is needed, since it is passed intact as the first parameter. In fact, we could observe that since target has not changed, the tests of statements (1) and (3) involving target are destined to fail, and we could instead skip statements (1) and (2), test only for candidate > n at line (3), and proceed directly to line (5).

Complete Recursion Elimination

The tail recursion elimination procedure removes recursion completely only when the recursive call is at the end of the procedure, and the call has the correct form. There is a more general approach that converts any recursive procedure (or function) into a nonrecursive one, but this approach introduces a user-defined stack. In general, a cell of this stack will hold:

  1. The current values of the parameters of the procedure;
  2. The current values of any local variables of the procedure; and
  3. An indication of the return address, that is, the place to which control returns when the current invocation of the procedure is done.

In the case of the function knapsack, we can do something simpler. First, observe that whenever we make a call (push a record onto the stack), candidate increases by 1. Thus, we can keep candidate as a global variable, incrementing it by one every time we push the stack and decreasing it by one when we pop.

A second simplification we can make is to keep a modified "return address" on the stack. Strictly speaking, the return address for this function is either a place in some other procedure that calls knapsack, or the call at line (5), or the call at line (8). We can represent these three conditions by a "status," which has one of three values:

  1. none, indicating that the call is from outside the function knapsack,
  2. included, indicating the call at line (5), which includes weights[candidate] in the solution, or
  3. excluded, indicating the call at line (8), which excludes weights[candidate].

If we store this status symbol as the return address, then we can treat target as a global variable. When changing from status none to included, we subtract weights[candidate] from target, and we add it back in when changing from status included to excluded. To help represent the effect of the knapsack's return indicating whether a solution has been found, we use a global winflag. Once set to true, winflag remains true and causes the stack to be popped and those weights with status included to be printed. With these modifications, we can declare our stack to be a list of statuses, by

        type
        statuses = (none, included, excluded);
        STACK = { suitable declaration for stack of statuses }
Figure 2.26 shows the resulting nonrecursive procedure knapsack operating on an array weights as before. Although this procedure may be faster than the original function knapsack, it is clearly longer and more difficult to understand. For this reason, recursion elimination should be used only when speed is very important.

procedure knapsack ( target: integer );
    var
        candidate: integer;
        winflag: boolean;
        S: STACK;
    begin
        candidate := 1;
        winflag := false;
        MAKENULL(S);
        PUSH(none, S); { initialize stack to consider weights[1] }
        repeat
            if winflag then begin
                { pop stack, printing weights included in solution }
                if TOP(S) = included then
                     writeln(weights[candidate]);
                candidate := candidate - 1;
                POP(S)
            end
            else if target = 0 then begin { solution found }
                winflag := true;
                candidate := candidate - 1;
                POP(S)
            end
            else if (((target < 0) and (TOP(S) = none))
                   or (candidate > n)) then begin
                { no solution possible with choices made }
                candidate := candidate - 1;
                POP(S)
            end
            else { no resolution yet; consider status of current candidate }
            if TOP(S) = none then begin { first try including candidate }
                target := target - weights[candidate];
                candidate := candidate + 1;
                POP(S); PUSH(included, S); PUSH(none, S)
            end
            else if TOP(S) = included then begin { try excluding candidate }
                target := target + weights[candidate];
                candidate := candidate + 1;
                 POP(S); PUSH(excluded, S); PUSH(none, S)
            end
            else begin { TOP(S) = excluded; give up on current choice }
                 POP(S);
                 candidate := candidate - 1
            end
        until EMPTY(S)
    end; { knapsack }

Fig. 2.26. Nonrecursive knapsack procedure.

Exercises

2.1 | Write a program to print the elements on a list. Throughout these exercises use list operations to implement your programs.
2.2 Write programs to insert, delete, and locate an element on a sorted list using
  1. array,
  2. pointer, and
  3. cursor implementations of lists.
What is the running time of each of your programs?
2.3 Write a program to merge
  1. two sorted lists,
  2. n sorted lists.
2.4 | Write a program to concatenate a list of lists.
2.5 | Suppose we wish to manipulate polynomials of the form p(x) = c1xe1 + c2xe2 + . . . + cnxen, where e1 > e2 > . . . > en � 0. Such a polynomial can be represented by a linked list in which each cell has three fields: one for the coefficient ci, one for the exponent ei, and one for the pointer to the next cell. Write a program to differentiate polynomials represented in this manner.
2.6 | Write programs to add and multiply polynomials of the form in Exercise 2.5. What is the running time of your programs as a function of the number of terms?
*2.7 Suppose we declare cells by

   type
       celltype = record
         bit: 0..1;
         next: � celltype
       end;
A binary number b1b2 . . . bn, where each bi is 0 or 1, has numerical value [image]. This number can be represented by the list b1, b2 , . . . , bn. That list, in turn, can be represented as a linked list of cells of type celltype. Write a procedure increment(bnumber) that adds one to the binary number pointed to by bnumber. Hint: Make increment recursive.
2.8 | Write a procedure to interchange the elements at positions p and NEXT(p) in a singly linked list.
*2.9 The following procedure was intended to remove all occurrences of element x from list L. Explain why it doesn't always work and suggest a way to repair the procedure so it performs its intended task.

   procedure delete ( x: elementtype; var L: LIST );
       var
          p: position;
       begin
          p := FIRST(L);
          while p <> END(L) do begin
              if RETRIEVE(p, L) = x then
                  DELETE(p, L);
              p := NEXT(p, L)
           end
       end; { delete }
2.10 We wish to store a list in an array A whose cells consist of two fields, data to store an element and position to give the (integer) position of the element. An integer last indicates that A[1] through A[last] are used to hold the list. The type LIST can be defined by

   type
       LIST = record
          last: integer;
          elements: array[1..maxlength] of record
                 data: elementtype;
                 position: integer
          end
       end;
Write a procedure DELETE(p, L) to remove the element at position p. Include all necessary error checks.
2.11 Suppose L is a LIST and p, q, and r are positions. As a function of n, the length of list L, determine how many times the functions FIRST, END, and NEXT are executed by the following program.

   p := FIRST(L);
   while p <> END(L) do begin
       q := p;
       while q <> END(L) do begin
           q := NEXT(q, L);
           r := FIRST(L);
           while r <> q do
               r := NEXT(r, L)
       end;
       p := NEXT(p, L)
   end;
2.12 | Rewrite the code for the LIST operations assuming a linked list representation, but without a header cell. Assume true pointers are used and position 1 is represented by nil.
2.13 | Add the necessary error checks in the procedure of Fig. 2.12.
2.14 | Another array representation of lists is to insert as in Section 2.2, but when deleting, simply replace the deleted element by a special value "deleted," which we assume does not appear on lists otherwise. Rewrite the list operations to implement this strategy. What are the advantages and disadvantages of the approach compared with our original array representation of lists?
2.15 | Suppose we wish to use an extra bit in queue records to indicate whether a queue is empty. Modify the declarations and operations for a circular queue to accommodate this feature. Would you expect the change to be worthwhile?
2.16 | A dequeue (double-ended queue) is a list from which elements can be inserted or deleted at either end. Develop array, pointer, and cursor implementations for a dequeue.
2.17 | Define an ADT to support the operations ENQUEUE, DEQUEUE, and ONQUEUE. ONQUEUE(x) is a function returning true or false depending on whether x is or is not on the queue.
2.18 | How would one implement a queue if the elements that are to be placed on the queue are arbitrary length strings? How long does it take to enqueue a string?
2.19 | Another possible linked-list implementation of queues is to use no header cell, and let front point directly to the first cell. If the queue is empty, let front = rear = nil. Implement the queue operations for this representation. How does this implementation compare with the list implementation given for queues in Section 2.4 in terms of speed, space utilization, and conciseness of the code?
2.20 A variant of the circular queue records the position of the front element and the length of the queue.
  1. Is it necessary in this implementation to limit the length of a queue to maxlength - 1?
  2. Write the five queue operations for this implementation.
  3. Compare this implementation with the circular queue implementation of Section 2.4.
2.21 | It is possible to keep two stacks in a single array, if one grows from position 1 of the array, and the other grows from the last position. Write a procedure PUSH(x, S) that pushes element x onto stack S, where S is one or the other of these two stacks. Include all necessary error checks in your procedure.
2.22 We can store k stacks in a single array if we use the data structure suggested in Fig. 2.27, for the case k = 3. We push and pop from each stack as suggested in connection with Fig. 2.17 in Section 2.3. However, if pushing onto stack i causes TOP(i) to equal BOTTOM(i-1), we first move all the stacks so that there is an appropriate size gap between each adjacent pair of stacks. For example, we might make the gaps above all stacks equal, or we might make the gap above stack i proportional to the current size of stack i (on the theory that larger stacks are likely to grow sooner, and we want to postpone as long as possible the next reorganization).
  1. On the assumption that there is a procedure reorganize to call when stacks collide, write code for the five stack operations.
  2. On the assumption that there is a procedure makenewtops that computes newtop[i], the "appropriate" position for the top of stack i, for 1 � i � k, write the procedure reorganize. Hint. Note that stack i could move up or down, and it is necessary to move stack i before stack j if the new position of stack j overlaps the old position of stack i. Consider stacks 1, 2 , . . . , k in order, but keep a stack of "goals," each goal being to move a particular stack. If on considering stack i, we can move it safely, do so, and then reconsider the stack whose number is on top of the goal stack. If we cannot safely move stack i, push i onto the goal stack.
  3. What is an appropriate implementation for the goal stack in (b)? Do we really need to keep it as a list of integers, or will a more succinct representation do?
  4. Implement makenewtops in such a way that space above each stack is proportional to the current size of that stack.
  5. What modifications of Fig. 2.27 are needed to make the implementation work for queues? For general lists?
2.23 | Modify the implementations of POP and ENQUEUE in Sections 2.3 and 2.4 to return the element removed from the stack or queue. What modifications must be made if the element type is not a type that can be returned by a function?
2.24 Use a stack to eliminate recursion from the following procedures.

a. function comb ( n, m: integer ): integer;
       { computes (
  ) assuming 0 � m � n and n
� 1 }
       begin
          if (n = 1) or (m = 0) or (m = n)
then
              return (1)
          else
              return (comb(n-1, m) + comb(n-1, m-1))
       end; { comb }

Fig. 2.27. Many stacks in one array.

b. procedure reverse ( var L: LIST );
       { reverse list L }
       var
          x: elementtype;
       begin
          if not EMPTY(L) then begin
              x := RETRIEVE(FIRST(L), L);
              DELETE(FIRST(L), L);
              reverse(L);
              INSERT(x, END(L), L)
          end
        end; { reverse }
*2.25 | Can we eliminate the tail recursion from the programs in Exercise 2.24? If so, do it.

Bibliographic Notes

Knuth [1968] contains additional information on the implementation of lists, stacks, and queues. A number of programming languages, such as LISP and SNOBOL, support lists and strings in a convenient manner. See Sammet [1969], Nicholls [1975], Pratt [1975], or Wexelblat [1981] for a history and description of many of these languages.

† Strictly speaking, the type is "LIST of elementtype." However, the implementations of lists we propose do not depend on what elementtype is; indeed, it is that independence that justifies the importance we place on the list concept. We shall use "LIST" rather than "LIST of elementtype," and similarly treat other ADT's that depend on the types of elements.

† In this case, if we eliminate records that are "the same" we might wish to check that the names and addresses are also equal; if the account numbers are equal but the other fields are not, two people may have inadvertently gotten the same account number. More likely, however, is that the same subscriber appears on the list more than once with distinct account numbers and slightly different names and/or addresses. In such cases, it is difficult to eliminate all duplicates.

† Even though L is not modified, we pass L by reference because frequently it will be a big structure and we don't want to waste time copying it.

† Making the header a complete cell simplifies the implementation of list operation in Pascal. We can use pointers for headers if we are willing to implement our operations so they do insertions and deletions at the beginning of a list in a special way. See the discussion under cursor-based implementation of lists in this section.

† Of course, there are many situations where we would like p to continue to represent the position of c.

† Incidentally, it is common practice to make the header of a doubly linked list be a cell that effectively "completes the circle." That is, the header's previous field points to the last cell and next points to the first cell. In this manner, we need not check for nil pointers in Fig. 2.14.

† Note that "consecutive" must be taken in circular sense. That is, a queue of length four could occupy the last two and first two positions of the array, for example.

† For example, firstvalue = 'A' and lastvalue = 'Z' if domaintype is 'A'..'Z'.

† In the "real" knapsack problem, we are given utility values as well as weights and are asked to maximize the utility of the items carried, subject to a weight constraint.

‡ Alternatively, x could be passed by reference if y is x.

Table of Contents Go to Chapter 3


Data Structures and Algorithms: CHAPTER 3: Trees

 Return to Table of Contents

Trees

A tree imposes a hierarchical structure on a collection of items. Familiar examples of trees are genealogies and organization charts. Trees are used to help analyze electrical circuits and to represent the structure of mathematical formulas. Trees also arise naturally in many different areas of computer science. For example, trees are used to organize information in database systems and to represent the syntactic structure of source programs in compilers. Chapter 5 describes applications of trees in the representation of data. Throughout this book, we shall use many different variants of trees. In this chapter we introduce the basic definitions and present some of the more common tree operations. We then describe some of the more frequently used data structures for trees that can be used to support these operations efficiently.

3.1 Basic Terminology

A tree is a collection of elements called nodes, one of which is distinguished as a root, along with a relation ("parenthood") that places a hierarchical structure on the nodes. A node, like an element of a list, can be of whatever type we wish. We often depict a node as a letter, a string, or a number with a circle around it. Formally, a tree can be defined recursively in the following manner.

  1. A single node by itself is a tree. This node is also the root of the tree.
  2. Suppose n is a node and T1, T2, . . . , Tk are trees with roots n1, n2, . . . , nk, respectively. We can construct a new tree by making n be the parent of nodes n1, n2, . . . , nk. In this tree n is the root and T1, T2, . . . , Tk are the subtrees of the root. Nodes n1, n2, . . . , nk are called the children of node n.

Sometimes, it is convenient to include among trees the null tree, a "tree" with no nodes, which we shall represent by L.

Example 3.1. Consider the table of contents of a book, as suggested by Fig. 3.1(a). This table of contents is a tree. We can redraw it in the manner shown in Fig. 3.1(b). The parent-child relationship is depicted by a line. Trees are normally drawn top-down as in Fig. 3.1(b), with the parent above the child.

The root, the node called "Book," has three subtrees with roots corresponding to the chapters C1, C2, and C3. This relationship is represented by the lines downward from Book to C1, C2, and C3. Book is the parent of C1, C2, and C3, and these three nodes are the children of Book.

Fig. 3.1. A table of contents and its tree representation.

The third subtree, with root C3, is a tree of a single node, while the other two subtrees have a nontrivial structure. For example, the subtree with root C2 has three subtrees, corresponding to the sections s2.1, s2.2, and s2.3; the last two are one-node trees, while the first has two subtrees corresponding to the subsections s2.1.1 and s2.1.2.

Example 3.1 is typical of one kind of data that is best represented as a tree. In this example, the parenthood relationship stands for containment; a parent node is comprised of its children, as Book is comprised of C1, C2, and C3. Throughout this book we shall encounter a variety of other relationships that can be represented by parenthood in trees.

If n1, n2, . . . , nk is a sequence of nodes in a tree such that ni is the parent of ni+1 for 1 � i < k, then this sequence is called a path from node n1 to node nk. The length of a path is one less than the number of nodes in the path. Thus there is a path of length zero from every node to itself. For example, in Fig. 3.1 there is a path of length two, namely (C2, s2.1, s2.1.2) from C2 to s2.1.2.

If there is a path from node a to node b, then a is an ancestor of b, and b is a descendant of a. For example, in Fig. 3.1, the ancestors of s2.1, are itself, C2, and Book, while its descendants are itself, s2.1.1, and s2.1.2. Notice that any node is both an ancestor and a descendant of itself.

An ancestor or descendant of a node, other than the node itself, is called a proper ancestor or proper descendant, respectively. In a tree, the root is the only node with no proper ancestors. A node with no proper descendants is called a leaf. A subtree of a tree is a node, together with all its descendants.

The height of a node in a tree is the length of a longest path from the node to a leaf. In Fig. 3.1 node C1 has height 1, node C2 height 2, and node C3 height 0. The height of a tree is the height of the root. The depth of a node is the length of the unique path from the root to that node.

The Order of Nodes

The children of a node are usually ordered from left-to-right. Thus the two trees of Fig. 3.2 are different because the two children of node a appear in a different order in the two trees. If we wish explicitly to ignore the order of children, we shall refer to a tree as an unordered tree.

Fig. 3.2. Two distinct (ordered) trees.

The "left-to-right" ordering of siblings (children of the same node) can be extended to compare any two nodes that are not related by the ancestor-descendant relationship. The relevant rule is that if a and b are siblings, and a is to the left of b, then all the descendants of a are to the left of all the descendants of b.

Example 3.2. Consider the tree in Fig. 3.3. Node 8 is to the right of node 2, to the left of nodes 9, 6, 10, 4, and 7, and neither left nor right of its ancestors 1, 3, and 5.

Fig. 3.3. A tree.

A simple rule, given a node n, for finding those nodes to its left and those to its right, is to draw the path from the root to n. All nodes branching off to the left of this path, and all descendants of such nodes, are to the left of n. All nodes and descendants of nodes branching off to the right are to the right of n.

Preorder, Postorder, and Inorder

There are several useful ways in which we can systematically order all nodes of a tree. The three most important orderings are called preorder, inorder and postorder; these orderings are defined recursively as follows.

  • If a tree T is null, then the empty list is the preorder, inorder and postorder listing of T.
  • If T consists a single node, then that node by itself is the preorder, inorder, and postorder listing of T.

Otherwise, let T be a tree with root n and subtrees T1, T2, . . . , Tk, as suggested in Fig. 3.4.

Fig. 3.4. Tree T.

  1. The preorder listing (or preorder traversal) of the nodes of T is the root n of T followed by the nodes of T1 in preorder, then the nodes of T2 in preorder, and so on, up to the nodes of Tk in preorder.
  2. The inorder listing of the nodes of T is the nodes of T1 in inorder, followed by node n, followed by the nodes of T2, . . . , Tk, each group of nodes in inorder.
  3. The postorder listing of the nodes of T is the nodes of T1 in postorder, then the nodes of T2 in postorder, and so on, up to Tk, all followed by node n.

Figure 3.5(a) shows a sketch of a procedure to list the nodes of a tree in preorder. To make it a postorder procedure, we simply reverse the order of steps (1) and (2). Figure 3.5(b) is a sketch of an inorder procedure. In each case, we produce the desired ordering of the tree by calling the appropriate procedure on the root of the tree.

Example 3.3. Let us list the tree of Fig. 3.3 in preorder. We first list 1 and then call PREORDER on the first subtree of 1, the subtree with root 2. This subtree is a single node, so we simply list it. Then we proceed to the second subtree of 1, the tree rooted at 3. We list 3, and then call PREORDER on the first subtree of 3. That call results in listing 5, 8, and 9, in that order.

    procedure PREORDER ( n: node );
       begin
(1)	       list n;
(2)	       for each child c of n, if any, in order from the left do
               PREORDER(c)
       end; { PREORDER }

                 (a) PREORDER procedure.

    procedure INORDER ( n: node );
       begin
           if n is a leaf then
               list n;
           else begin
               INORDER(leftmost child of n);
               list n;
               for each child c of n, except for the leftmost,
                   in order from the left do
                       INORDER(c)
           end
       end; { INORDER }

                  (b) INORDER procedure.

Fig. 3.5. Recursive ordering procedures.

Continuing in this manner, we obtain the complete preorder traversal of Fig. 3.3: 1, 2, 3, 5, 8, 9, 6, 10, 4, 7.

Similarly, by simulating Fig. 3.5(a) with the steps reversed, we can discover that the postorder of Fig. 3.3 is 2, 8, 9, 5, 10, 6, 3, 7, 4, 1. By simulating Fig. 3.5(b), we find that the inorder listing of Fig. 3.3 is 2, 1, 8, 5, 9, 3, 10, 6, 7, 4.

A useful trick for producing the three node orderings is the following. Imagine we walk around the outside of the tree, starting at the root, moving counterclockwise, and staying as close to the tree as possible; the path we have in mind for Fig. 3.3 is shown in Fig. 3.6.

For preorder, we list a node the first time we pass it. For postorder, we list a node the last time we pass it, as we move up to its parent. For inorder, we list a leaf the first time we pass it, but list an interior node the second time we pass it. For example, node 1 in Fig. 3.6 is passed the first time at the beginning, and the second time while passing through the "bay" between nodes 2 and 3. Note that the order of the leaves in the three orderings is

Fig. 3.6. Traversal of a tree.

always the same left-to-right ordering of the leaves. It is only the ordering of the interior nodes and their relationship to the leaves that vary among the three.

Labeled Trees and Expression Trees

Often it is useful to associate a label, or value, with each node of a tree, in the same spirit with which we associated a value with a list element in the previous chapter. That is, the label of a node is not the name of the node, but a value that is "stored" at the node. In some applications we shall even change the label of a node, while the name of a node remains the same. A useful analogy is tree:list = label:element = node:position.

Example 3.4. Figure 3.7 shows a labeled tree representing the arithmetic expression (a+b) * (a+c), where n1, . . . , n7 are the names of the nodes, and the labels, by convention, are shown next to the nodes. The rules whereby a labeled tree represents an expression are as follows:

  1. Every leaf is labeled by an operand and consists of that operand alone. For example, node n4 represents the expression a.
  2. Every interior node n is labeled by an operator. Suppose n is labeled by a binary operator q, such as + or *, and that the left child represents expression E1 and the right child E2. Then n represents expression (E1) q (E2). We may remove the parentheses if they are not necessary.

For example, node n2 has operator +, and its left and right children represent the expressions a and b, respectively. Therefore, n2 represents (a)+(b), or just a+b. Node n1 represents (a+b)*(a+c), since * is the label at n1, and a+b and a+c are the expressions represented by n2 and n3, respectively.

Fig. 3.7. Expression tree with labels.

Often, when we produce the preorder, inorder, or postorder listing of a tree, we prefer to list not the node names, but rather the labels. In the case of an expression tree, the preorder listing of the labels gives us what is known as the prefix form of an expression, where the operator precedes its left operand and its right operand. To be precise, the prefix expression for a single operand a is a itself. The prefix expression for (E1) q (E2), with q a binary operator, is q P1P2, where P1 and P2 are the prefix expressions for E1 and E2. Note that no parentheses are necessary in the prefix expression, since we can scan the prefix expression q P1P2 and uniquely identify P1 as the shortest (and only) prefix of P1P2 that is a legal prefix expression.

For example, the preorder listing of the labels of Fig. 3.7 is *+ab+ac. The prefix expression for n2, which is +ab, is the shortest legal prefix of +ab+ac.

Similarly, a postorder listing of the labels of an expression tree gives us what is known as the postfix (or Polish) representation of an expression. The expression (E1) q (E2) is represented by the postfix expression P1P2q, where P1 and P2 are the postfix representations of E1 and E2, respectively. Again, no parentheses are necessary in the postfix representation, as we can deduce what P2 is by looking for the shortest suffix of P1P2 that is a legal postfix expression. For example, the postfix expression for Fig. 3.7 is ab+ac+*. If we write this expression as P1P2*, then P2 is ac+, the shortest suffix of ab+ac+ that is a legal postfix expression.

The inorder traversal of an expression tree gives the infix expression itself, but with no parentheses. For example, the inorder listing of the labels of Fig. 3.7 is a+b * a+c. The reader is invited to provide an algorithm for traversing an expression tree and producing an infix expression with all needed pairs of parentheses.

Computing Ancestral Information

The preorder and postorder traversals of a tree are useful in obtaining ancestral information. Suppose postorder(n) is the position of node n in a post-order listing of the nodes of a tree. Suppose desc(n) is the number of proper descendants of node n. For example, in the tree of Fig. 3.7 the postorder numbers of nodes n2, n4, and n5 are 3, 1, and 2, respectively.

The postorder numbers assigned to the nodes have the useful property that the nodes in the subtree with root n are numbered consecutively from postorder(n) - desc(n) to postorder(n). To test if a vertex x is a descendant of vertex y, all we need do is determine whether

postorder(y) - desc(y) � postorder(x) � postorder(y).

A similar property holds for preorder.

3.2 The ADT TREE

In Chapter 2, lists, stacks, queues, and mappings were treated as abstract data types (ADT's). In this chapter trees will be treated both as ADT's and as data structures. One of our most important uses of trees occurs in the design of implementations for the various ADT's we study. For example, in Section 5.1, we shall see how a "binary search tree" can be used to implement abstract data types based on the mathematical model of a set, together with operations such as INSERT, DELETE, and MEMBER (to test whether an element is in a set). The next two chapters present a number of other tree implementations of various ADT's.

In this section, we shall present several useful operations on trees and show how tree algorithms can be designed in terms of these operations. As with lists, there are a great variety of operations that can be performed on trees. Here, we shall consider the following operations:

  1. PARENT(n, T). This function returns the parent of node n in tree T. If n is the root, which has no parent, L is returned. In this context, L is a "null node," which is used as a signal that we have navigated off the tree.
  2. LEFTMOST_CHILD(n, T) returns the leftmost child of node n in tree T, and it returns L if n is a leaf, which therefore has no children.
  3. RIGHT_SIBLING(n, T) returns the right sibling of node n in tree T, defined to be that node m with the same parent p as n such that m lies immediately to the right of n in the ordering of the children of p. For example, for the tree in Fig. 3.7, LEFTMOST_CHILD(n2) = n4; RIGHT_SIBLING(n4) = n5, and RIGHT_SIBLING (n5) = L.
  4. LABEL(n, T) returns the label of node n in tree T. We do not, however, require labels to be defined for every tree.
  5. CREATEi(v, T1, T2, . . . , Ti) is one of an infinite family of functions, one for each value of i = 0, 1, 2, . . .. CREATEi makes a new node r with label v and gives it i children, which are the roots of trees T1, T2, . . . , Ti, in order from the left. The tree with root r is returned. Note that if i = 0, then r is both a leaf and the root.
  6. ROOT(T) returns the node that is the root of tree T, or L if T is the null tree.
  7. MAKENULL(T) makes T be the null tree.

Example 3.5. Let us write both recursive and nonrecursive procedures to take a tree and list the labels of its nodes in preorder. We assume that there are data types node and TREE already defined for us, and that the data type TREE is for trees with labels of the type labeltype. Figure 3.8 shows a recursive procedure that, given node n, lists the labels of the subtree rooted at n in preorder. We call PREORDER(ROOT(T)) to get a preorder listing of tree T.

    procedure PREORDER ( n: node );
        { list the labels of the descendants of n in preorder }
        var
              c: node;
        begin
              print(LABEL(n, T));
              c := LEFTMOST_CHILD(n, T);
              while c <> L do
begin
                  PREORDER(c);
                  c := RIGHT_SIBLING(c, T)
              end
        end;   { PREORDER }

Fig. 3.8. A recursive preorder listing procedure.

We shall also develop a nonrecursive procedure to print a tree in preorder. To find our way around the tree, we shall use a stack S, whose type STACK is really "stack of nodes." The basic idea underlying our algorithm is that when we are at a node n, the stack will hold the path from the root to n, with the root at the bottom of the stack and node n at the top.†

One way to perform a nonrecursive preorder traversal of a tree is given by the program NPREORDER shown in Fig. 3.9. This program has two modes of operation. In the first mode it descends down the leftmost unexplored path in the tree, printing and stacking the nodes along the path, until it reaches a leaf.

The program then enters the second mode of operation in which it retreats back up the stacked path, popping the nodes of the path off the stack, until it encounters a node on the path with a right sibling. The program then reverts back to the first mode of operation, starting the descent from that unexplored right sibling.

The program begins in mode one at the root and terminates when the stack becomes empty. The complete program is shown in Fig. 3.9.

3.3 Implementations of Trees

In this section we shall present several basic implementations for trees and discuss their capabilities for supporting the various tree operations introduced in Section 3.2.

An Array Representation of Trees

Let T be a tree in which the nodes are named 1, 2, . . . , n. Perhaps the simplest representation of T that supports the PARENT operation is a linear array A in which entry A[i] is a pointer or a cursor to the parent of node i. The root of T can be distinguished by giving it a null pointer or a pointer to itself as parent. In Pascal, pointers to array elements are not feasible, so we shall have to use a cursor scheme where A[i] = j if node j is the parent of node i, and A[i] = 0 if node i is the root.

This representation uses the property of trees that each node has a unique parent. With this representation the parent of a node can be found in constant time. A path going up the tree, that is, from node to parent to parent, and so on, can be traversed in time proportional to the number of nodes on the path. We can also support the LABEL operator by adding another array L, such that L[i] is the label of node i, or by making the elements of array A be records consisting of an integer (cursor) and a label.

Example 3.6. The tree of Fig. 3.10(a) has the parent representation given by the array A shown in Fig. 3.10(b).

The parent pointer representation does not facilitate operations that require child-of information. Given a node n, it is expensive to determine the children of n, or the height of n. In addition, the parent pointer representation does not specify the order of the children of a node. Thus, operations like LEFTMOST_CHILD and RIGHT_SIBLING are not well defined. We could impose an artificial order, for example, by numbering the children of each node after numbering the parent, and numbering the children in

    procedure NPREORDER ( T: TREE );
        { nonrecursive preorder traversal of tree T }

        var
        m: node; { a temporary }
        S: STACK; { stack of nodes holding path from the root
            to the parent TOP(S) of the "current" node m }

        begin
        { initialize }
        MAKENULL(S);
        m := ROOT(T);

        while true do
            if m < > L then begin
            print(LABEL(m, T));
            PUSH(m, S);
            { explore leftmost child of m }
            m := LEFTMOST_CHILD(m, T)
        end
        else begin
             { exploration of path on stack
             is now complete }
             if EMPTY(S) then
             return;
             { explore right sibling of node
             on top of stack }
             m := RIGHT_SIBLING(TOP(S), T);
             POP(S)
         end
           end; { NPREORDER }

Fig. 3.9. A nonrecursive preorder procedure.

increasing order from left to right. On that assumption, we have written the function RIGHT_SIBLING in Fig. 3.11, for types node and TREE that are defined as follows:

    type
        node = integer;
        TREE = array [1..maxnodes] of node;

For this implementation we assume the null node L is represented by 0.

Fig. 3.10. A tree and its parent pointer representation.

    function RIGHT_SIBLING ( n: node; T: TREE ): node;
        { return the right sibling of node n in tree T }
        var
           i, parent: node;
        begin
        parent: = T[n];
        for i := n + 1 to maxnodes do
             { search for node after n with same parent }
             if T[i] = parent then
                  return (i);
             return (0) { null node will be returned
             if no right sibling is ever found }
        end; { RIGHT_SIBLING }

Fig. 3.11. Right sibling operation using array representation.

Representation of Trees by Lists of Children

An important and useful way of representing trees is to form for each node a list of its children. The lists can be represented by any of the methods suggested in Chapter 2, but because the number of children each node may have can be variable, the linked-list representations are often more appropriate.

Figure 3.12 suggests how the tree of Fig. 3.10(a) might be represented. There is an array of header cells, indexed by nodes, which we assume to be numbered 1, 2, . . . , 10. Each header points to a linked list of "elements," which are nodes. The elements on the list headed by header[i] are the children of node i; for example, 9 and 10 are the children of 3.

Fig. 3.12. A linked-list representation of a tree.

Let us first develop the data structures we need in terms of an abstract data type LIST (of nodes), and then give a particular implementation of lists and see how the abstractions fit together. Later, we shall see some of the simplifications we can make. We begin with the following type declarations:

     type
         node = integer;
         LIST =  { appropriate definition for list of nodes };
         position = { appropriate definition for positions in lists };
         TREE = record
           header: array [1..maxnodes] of LIST;
           labels: array [1..maxnodes] of labeltype;
           root: node
         end;

We assume that the root of each tree is stored explicitly in the root field. Also, 0 is used to represent the null node.

Figure 3.13 shows the code for the LEFTMOST_CHILD operation. The reader should write the code for the other operations as exercises.

    function LEFTMOST_CHILD ( n: node; T: TREE ): node;
        { returns the leftmost child of node n of tree T }
        var
         L: LIST; { shorthand for the list of n's children }
        begin
         L := T.header[n];
           if EMPTY(L) then { n is a leaf }
             return (0)
         else
             return (RETRIEVE(FIRST(L), L))
        end;  { LEFTMOST_CHILD }

Fig. 3.13. Function to find leftmost child.

Now let us choose a particular implementation of lists, in which both LIST and position are integers, used as cursors into an array cellspace of records:

    var
        cellspace : array [1..maxnodes] of record
             node: integer;
             next: integer
        end;

To simplify, we shall not insist that lists of children have header cells. Rather, we shall let T.header [n] point directly to the first cell of the list, as is suggested by Fig. 3.12. Figure 3.14(a) shows the function LEFTMOST_CHILD of Fig. 3.13 rewritten for this specific implementation. Figure 3.14(b) shows the operator PARENT, which is more difficult to write using this representation of lists, since a search of all lists is required to determine on which list a given node appears.

The Leftmost-Child, Right-Sibling Representation

The data structure described above has, among other shortcomings, the inability to create large trees from smaller ones, using the CREATEi operators. The reason is that, while all trees share cellspace for linked lists of children, each has its own array of headers for its nodes. For example, to implement CREATE2(v, T1, T2) we would have to copy T1 and T2 into a third tree and add a new node with label v and two children -- the roots of T1 and T2.

If we wish to build trees from smaller ones, it is best that the representation of nodes from all trees share one area. The logical extension of Fig. 3.12 is to replace the header array by an array nodespace consisting of records with

    function LEFTMOST_CHILD ( n: node; T: TREE ): node;
        { returns the leftmost child of node n on tree T }
        var
         L: integer; { a cursor to the beginning of the list of n's children }
        begin
        L := T.header[n];
        if L = 0 then { n is a leaf }
            return (0)
        else
            return (cellspace[L].node)
        end; { LEFTMOST_CHILD }

          (a) The function LEFTMOST_CHILD.

    function PARENT ( n: node; T: TREE ): node;
        { returns the parent of node n in tree T }
        var
         p: node; { runs through possible parents of n }
         i: position; { runs down list of p's children }
        begin
         for p := 1 to maxnodes do begin
            i := T.header[p];
            while i <> 0 do { see if n is among children of
p }
            if cellspace[i].node = n then
                 return (p)
            else
                 i := cellspace[i].next
        end;
        return (0) { return null node if parent not found }
        end; { PARENT }

              (b) The function PARENT.

Fig. 3.14. Two functions using linked-list representation of trees.

two fields label and header. This array will hold headers for all nodes of all trees. Thus, we declare

    var
        nodespace : array [1..maxnodes] of record
            label: labeltype;
            header: integer; { cursor to cellspace }
        end;

Then, since nodes are no longer named 1, 2, . . . , n, but are represented by arbitrary indices in nodespace, it is no longer feasible for the field node of cellspace to represent the "number" of a node; rather, node is now a cursor into nodespace, indicating the position of that node. The type TREE is simply a cursor into nodespace, indicating the position of the root.

Example 3.7. Figure 3.15(a) shows a tree, and Fig. 3.15(b) shows the data structure where we have placed the nodes labeled A, B, C, and D arbitrarily in positions 10, 5, 11, and 2 of nodespace. We have also made arbitrary choices for the cells of cellspace used for lists of children.

Fig. 3.15. Another linked-list structure for trees.

The structure of Fig. 3.15(b) is adequate to merge trees by the CREATEi operations. This data structure can be significantly simplified, however, First, observe that the chains of next pointers in cellspace are really right-sibling pointers.

Using these pointers, we can obtain leftmost children as follows. Suppose cellspace[i].node = n. (Recall that the "name" of a node, as opposed to its label, is in effect its index in nodespace, which is what cellspace[i].node gives us.) Then nodespace[n].header indicates the cell for the leftmost child of n in cellspace, in the sense that the node field of that cell is the name of that node in nodespace.

We can simplify matters if we identify a node not with its index in nodespace, but with the index of the cell in cellspace that represents it as a child. Then, the next pointers of cellspace truly point to right siblings, and the information contained in the nodespace array can be held by introducing a field leftmost_child in cellspace. The datatype TREE becomes an integer used as a cursor to cellspace indicating the root of the tree. We declare cellspace to have the following structure.

    var
        cellspace : array [1..maxnodes] of record
             label: labeltype;
             leftmost_child: integer;
             right_sibling: integer;
        end;

Example 3.8. The tree of Fig. 3.15(a) is represented in our new data structure in Fig. 3.16. The same arbitrary indices as in Fig. 3.15(b) have been used for the nodes.

Fig. 3.16. Leftmost-child, right-sibling representation of a tree.

All operations but PARENT are straightforward to implement in the leftmost-child, right-sibling representation. PARENT requires searching the entire cellspace. If we need to perform the PARENT operation efficiently, we can add a fourth field to cellspace to indicate the parent of a node directly.

As an example of a tree operation written to use the leftmost- child, right-sibling structure as in Fig. 3.16, we give the function CREATE2 in Fig. 3.17. We assume that unused cells are linked in an available space list, headed by avail, and that available cells are linked by their right-sibling fields. Figure 3.18 shows the old (solid) and the new (dashed) pointers.

    function CREATE2 ( v: labeltype; T1, T2: integer ): integer;
        { returns new tree with root v, having T1 and T2 as subtrees }
        var
        temp: integer; { holds index of first available cell
            for root of new tree }
        begin
        temp := avail;
        avail := cellspace [avail].right_sibling;
        cellspace[temp].leftmost_child := T1;
        cellspace[temp].label := v;
        cellspace[temp].right_sibling := 0;
        cellspace[T1].right_sibling := T2;
        cellspace[T2].right_sibling := 0; { not necessary;
             that field should be 0 as the cell was formerly a root }
        return (temp)
        end; { CREATE2 }

Fig. 3.17. The function CREATE2.

Fig. 3.18. Pointer changes produced by CREATE2.

Alternatively, we can use less space but more time if we put in the right-sibling field of the rightmost child a pointer to the parent, in place of the null pointer that would otherwise be there. To avoid confusion, we need a bit in every cell indicating whether the right-sibling field holds a pointer to the right sibling or to the parent.

Given a node, we find its parent by following right-sibling pointers until we find one that is a parent pointer. Since all siblings have the same parent, we thereby find our way to the parent of the node we started from. The time required to find a node's parent in this representation depends on the number of siblings a node has.

3.4 Binary Trees

The tree we defined in Section 3.1 is sometimes called an ordered, oriented tree because the children of each node are ordered from left-to-right, and because there is an oriented path (path in a particular direction) from every node to its descendants. Another useful, and quite different, notion of "tree" is the binary tree, which is either an empty tree, or a tree in which every node has either no children, a left child, a right child, or both a left and a right child. The fact that each child in a binary tree is designated as a left child or as a right child makes a binary tree different from the ordered, oriented tree of Section 3.1.

Example 3.9. If we adopt the convention that left children are drawn extending to the left, and right children to the right, then Fig. 3.19 (a) and (b) represent two different binary trees, even though both "look like" the ordinary (ordered, oriented) tree of Fig. 3.20. However, let us emphasize that Fig. 3.19(a) and (b) are not the same binary tree, nor are either in any sense equal to Fig. 3.20, for the simple reason that binary trees are not directly comparable with ordinary trees. For example, in Fig. 3.19(a), 2 is the left child of 1, and 1 has no right child, while in Fig. 3.19(b), 1 has no left child but has 2 as a right child. In either binary tree, 3 is the left child of 2, and 4 is 2's right child.

The preorder and postorder listings of a binary tree are similar to those of an ordinary tree given on p. 78. The inorder listing of the nodes of a binary tree with root n, left subtree T1 and right subtree T2 is the inorder listing of T1 followed by n followed by the inorder listing of T2. For example, 35241 is the inorder listing of the nodes of Fig. 3.19(a).

Representing Binary Trees

A convenient data structure for representing a binary tree is to name the nodes 1, 2, . . . , n, and to use an array of records declared

    var
        cellspace : array [1..maxnodes] of record
             leftchild: integer;
             rightchild: integer;
        end;

Fig. 3.19. Two binary trees.

Fig. 3.20. An "ordinary" tree.

The intention is that cellspace[i].leftchild is the left child of node i, and rightchild is analogous. A value of 0 in either field indicates the absence of a child.

Example 3.10. The binary tree of Fig. 3.19(a) can be represented as shown in Fig. 3.21.

An Example: Huffman Codes

Let us give an example of how binary trees can be used as a data structure. The particular problem we shall consider is the construction of "Huffman codes." Suppose we have messages consisting of sequences of characters. In each message, the characters are independent and appear with a known

Fig. 3.21. Representation of a binary tree.

probability in any given position; the probabilities are the same for all positions. As an example, suppose we have a message made from the five characters a, b, c, d, e, which appear with probabilities .12, .4, .15, .08, .25, respectively.

We wish to encode each character into a sequence of 0's and 1's so that no code for a character is the prefix of the code for any other character. This prefix property allows us to decode a string of 0's and 1's by repeatedly deleting prefixes of the string that are codes for characters.

Example 3.11. Figure 3.22 shows two possible codes for our five symbol alphabet. Clearly Code 1 has the prefix property, since no sequence of three bits can be the prefix of another sequence of three bits. The decoding algorithm for Code 1 is simple. Just "grab" three bits at a time and translate each group of three into a character. Of course, sequences 101, 110, and 111 are impossible, if the string of bits really codes characters according to Code 1. For example, if we receive 001010011 we know the original message was bcd.

Fig. 3.22. Two binary codes.

It is easy to check that Code 2 also has the prefix property. We can decode a string of bits by repeatedly "grabbing" prefixes that are codes for characters and removing them, just as we did for Code 1. The only difference is that here, we cannot slice up the entire sequence of bits at once, because whether we take two or three bits for a character depends on the bits. For example, if a string begins 1101001, we can again be sure that the characters coded were bcd. The first two bits, 11, must have come from b, so we can remove them and worry about 01001. We then deduce that the bits 01 came from c, and so on.

The problem we face is: given a set of characters and their probabilities, find a code with the prefix property such that the average length of a code for a character is a minimum. The reason we want to minimize the average code length is to compress the length of an average message. The shorter the average code for a character is, the shorter the length of the encoded message. For example, Code 1 has an average code length of 3. This is obtained by multiplying the length of the code for each symbol by the probability of occurrence of that symbol. Code 2 has an average length of 2.2, since symbols a and d, which together appear 20% of the time, have codes of length three, and the other symbols have codes of length two.

Can we do better than Code 2? A complete answer to this question is to exhibit a code with the prefix property having an average length of 2.15. This is the best possible code for these probabilities of symbol occurrences. One technique for finding optimal prefix codes is called Huffman's algorithm. It works by selecting two characters a and b having the lowest probabilities and replacing them with a single (imaginary) character, say x, whose probability of occurrence is the sum of the probabilities for a and b. We then find an optimal prefix code for this smaller set of characters, using this procedure recursively. The code for the original character set is obtained by using the code for x with a 0 appended as the code for a and with a 1 appended as a code for b.

We can think of prefix codes as paths in binary trees. Think of following a path from a node to its left child as appending a 0 to a code, and proceeding from a node to its right child as appending a 1. If we label the leaves of a binary tree by the characters represented, we can represent any prefix code as a binary tree. The prefix property guarantees no character can have a code that is an interior node, and conversely, labeling the leaves of any binary tree with characters gives us a code with the prefix property for these characters.

Example 3.12. The binary trees for Code 1 and Code 2 of Fig. 3.22 are shown in Fig. 3.23(a) and (b), respectively.

We shall implement Huffman's algorithm using a forest (collection of trees), each of which has its leaves labeled by characters whose codes we desire to select and whose roots are labeled by the sum of the probabilities of all the leaf labels. We call this sum the weight of the tree. Initially, each character is in a one-node tree by itself, and when the algorithm ends, there will be only one tree, with all the characters at its leaves. In this tree, the path from the root to any leaf represents the code for the label of that leaf, according to the left = 0, right = 1 scheme of Fig. 3.23.

The essential step of the algorithm is to select the two trees in the forest that have the smallest weights (break ties arbitrarily). Combine these two trees into one, whose weight is the sum of the weights of the two trees. To combine the trees we create a new node, which becomes the root and has the

Fig. 3.23. Binary trees representing codes with the prefix property.

roots of the two given trees as left and right children (which is which doesn't matter). This process continues until only one tree remains. That tree represents a code that, for the probabilities given, has the minimum possible average code length.

Example 3.13. The sequence of steps taken for the characters and probabilities in our running example is shown in Fig. 3.24. From Fig. 3.24(e) we see the code words for a, b, c, d, and e are 1111, 0, 110, 1110, and 10. In this example, there is only one nontrivial tree, but in general, there can be many. For example, if the probabilities of b and e were .33 and .32, then after Fig. 3.24(c) we would combine b and e, rather than attaching e to the large tree as we did in Fig. 3.24(d).

Let us now describe the needed data structures. First, we shall use an array TREE of records of the type

    record
        leftchild: integer;
        rightchild: integer;
        parent: integer;
    end

to represent binary trees. Parent pointers facilitate finding paths from leaves to roots, so we can discover the code for a character. Second, we use an array ALPHABET of records of type

    record
        symbol: char;
        probability: real;
        leaf: integer { cursor into tree }
    end

(e) Final tree

Fig. 3.24. Steps in the construction of a Huffman tree.

to associate, with each symbol of the alphabet being encoded, its corresponding leaf. This array also records the probability of each character. Third, we need an array FOREST of records that represent the trees themselves. The type of these records is

    record
        weight: real;
        root: integer { cursor into tree }
    end

The initial values of all these arrays, assuming the data of Fig. 3.24(a), are shown in Fig. 3.25. A sketch of the program to build the Huffman tree is shown in Fig. 3.26.

Fig. 3.25. Initial contents of arrays.

(1)       while there is more then one tree in the forest do
          begin
(2)               i := index of the tree in FOREST with smallest weight;
(3)               j := index of the tree in FOREST with second smallest weight;
(4)               create a new node with left child FOREST[i].root and
                      right child FOREST[j].root;
(5)               replace tree i in FOREST by a tree whose root
              is the new node and whose weight is
              FOREST[i].weight +
FOREST[j].weight;
(6)               delete tree j from FOREST
          end;

Fig. 3.26. Sketch of Huffman tree construction.

To implement line (4) of Fig. 3.26, which increases the number of cells of the TREE array used, and lines (5) and (6), which decrease the number of utilized cells of FOREST, we shall introduce cursors lasttree and lastnode, pointing to FOREST and TREE, respectively. We assume that cells 1 to lasttree of FOREST and 1 to lastnode of TREE are occupied.† We assume that arrays of Fig. 3.25 have some declared lengths, but in what follows we omit comparisons between these limits and cursor values.

    procedure lightones ( var least, second: integer );
        { sets least and second to the indices in FOREST of
         the trees of smallest weight. We assume lasttree �2. }
        var
        i: integer;
        begin { initialize least and second, considering first two trees }
        if FOREST[1].weight < = FOREST[2].weight
then
            begin least := 1; second := 2 end
        else
            begin least := 2; second := 1 end;
        { Now let i run from 3 to lasttree. At each iteration
            least is the tree of smallest weight among the first i trees
            in FOREST, and second is the next smallest of these }
        for i := 3 to lasttree do
            if FOREST[i].weight <
FOREST[least].weight then
            begin second := least; least := i
end
            else if FOREST[i].weight <
FOREST[second].weight then
            second: = i
        end; { lightones }

    function create ( lefttree, righttree: integer ): integer;
        { returns new node whose left and right children are
        FOREST[lefttree].root and
FOREST[righttree].root }
        begin
        lastnode := lastnode + 1;
             { cell for new node is TREE[lastnode] }
        TREE[lastnode].leftchild : =
FOREST[lefttree].root;
        TREE[lastnode].rightchild : =
FOREST[righttree].root;
        { now enter parent pointers for new node and its children }
        TREE[lastnode].parent := 0;
        TREE[FOREST[lefttree].root].parent :=
lastnode;
        TREE[FOREST[righttree].root].parent :=
lastnode;
        return(lastnode)
        end; { create }

Fig. 3.27. Two procedures.

Figure 3.27 shows two useful procedures. The first implements lines (2) and (3) of Fig. 3.26 to select indices of the two trees of smallest weight. The second is the command create(n1, n2) that creates a new node and makes n1 and n2 its left and right children.

Now the steps of Fig. 3.26 can be described in greater detail. A procedure Huffman, which has no input or output, but works on the global structures of Fig. 3.25, is shown in Fig. 3.28.

    procedure Huffman;
       var
           i, j: integer; { the two trees of least weight in FOREST }
           newroot: integer;
       begin
           while lasttree > 1 do begin
           lightones(i, j);
           newroot := create(i, j);
           { Now replace tree i by the tree whose root is newroot }
               FOREST[i].weight := FOREST[i].weight +
FOREST[j].weight;
           FOREST[i].root := newroot;
           { next, replace tree j, which is no longer needed, by lasttree,
               and shrink FOREST by one }
           FOREST[j] := FOREST[lasttree];
           lasttree := lasttree - 1
           end
       end; { Huffman }

Fig. 3.28. Huffman's algorithm.

Figure 3.29 shows the data structure of Fig. 3.25 after lasttree has been reduced to 3, that is, when the forest looks like Fig. 3.24(c).

Fig. 3.29. Tree data structure after two iterations.

After completing execution of the algorithm, the code for each symbol can be determined as follows. Find the symbol in the symbol field of the ALPHABET array. Follow the leaf field of the same record to get to a record of the TREE array; this record corresponds to the leaf for that symbol. Repeatedly follow the parent pointer from the "current" record, say for node n, to the record of the TREE array for its parent p. Remember node n, so it is possible to examine the leftchild and rightchild pointers for node p and see which is n. In the former case, print 0, in the latter print 1. The sequence of bits printed is the code for the symbol, in reverse. If we wish the bits printed in the correct order, we could push each onto a stack as we go up the tree, and then repeatedly pop the stack, printing symbols as we pop them.

Pointer-Based Implementations of Binary Trees

Instead of using cursors to point to left and right children (and parents if we wish), we can use true Pascal pointers. For example, we might declare

    type
        node = record
           leftchild: � node;
           rightchild: � node;
           parent: � node;
        end

For example, if we used this type for nodes of a binary tree, the function create of Fig. 3.27 could be written as in Fig. 3.30.

    function create ( lefttree, righttree: � node ): � node;
        var
        root: � node;
        begin
        new(root);
        root �.leftchild := lefttree;
        root �.rightchild :=
righttree;
        root �.parent := 0;
        lefttree �.parent := root;
        righttree �.parent := root;
        return (root)
        end; { create }

Fig. 3.30. Pointer-based implementation of binary trees.

Exercises

3.1 Answer the following questions about the tree of Fig. 3.31.
  1. Which nodes are leaves?
  2. Which node is the root?
  3. What is the parent of node C?
  4. Which nodes are children of C?
  5. Which nodes are ancestors of E?
  6. Which nodes are descendants of E?
  7. What are the right siblings of D and E?
  8. Which nodes are to the left and to the right of G?
  9. What is the depth of node C?
  10. What is the height of node C?

Fig. 3.31. A tree.
3.2 | In the tree of Fig. 3.31 how many different paths of length three are there?
3.3 | Write programs to compute the height of a tree using each of the three tree representations of Section 3.3.
3.4 List the nodes of Fig. 3.31 in
  1. preorder,
  2. postorder, and
  3. inorder.
3.5 If m and n are two different nodes in the same tree, show that exactly one of the following statements is true:
  1. m is to the left of n
  2. m is to the right of n
  3. m is a proper ancestor of n
  4. m is a proper descendant of n.
3.6 Place a check in row i and column j if the two conditions represented by row i and column j can occur simultaneously.

For example, put a check in row 3 and column 2 if you believe that n can be a proper ancestor of m and at the same time n can precede m in inorder.
3.7 | Suppose we have arrays PREORDER[n], INORDER[n], and POSTORDER[n] that give the preorder, inorder, and postorder positions, respectively, of each node n of a tree. Describe an algorithm that tells whether node i is an ancestor of node j, for any pair of nodes i and j. Explain why your algorithm works.
*3.8 | We can test whether a node m is a proper ancestor of a node n by testing whether m precedes n in X-order but follows n in Y-order, where X and Y are chosen from {pre, post, in}. Determine all those pairs X and Y for which this statement holds.
3.9 Write programs to traverse a binary tree in
  1. preorder,
  2. postorder,
  3. inorder.
3.10 | The level-order listing of the nodes of a tree first lists the root, then all nodes of depth 1, then all nodes of depth 2, and so on. Nodes at the same depth are listed in left-to-right order. Write a program to list the nodes of a tree in level-order.
3.11 Convert the expression ((a + b) + c * (d + e) + f) * (g + h) to a
  1. prefix expression
  2. postfix expression.
3.12 Draw tree representations for the prefix expressions
  1. *a + b*c + de
  2. *a + *b + cde
3.13 Let T be a tree in which every nonleaf node has two children. Write a program to convert
  1. a preorder listing of T into a postorder listing,
  2. a postorder listing of T into a preorder listing,
  3. a preorder listing of T into an inorder listing.
3.14 Write a program to evaluate
  1. preorder
  2. postorder
arithmetic expressions.
3.15 | We can define a binary tree as an ADT with the binary tree structure as a mathematical model and with operations such as LEFTCHILD(n), RIGHTCHILD(n), PARENT(n), and NULL(n). The first three operations return the left child, the right child, and the parent of node n (L if there is none) and the last returns true if and only if n is L. Implement these procedures using the binary tree representation of Fig. 3.21.
3.16 Implement the seven tree operations of Section 3.2 using the following tree implementations:
  1. parent pointers
  2. lists of children
  3. leftmost-child, right-sibling pointers.
3.17 | The degree of a node is the number of children it has. Show that in any binary tree the number of leaves is one more than the number of nodes of degree two.
3.18 | Show that the maximum number of nodes in a binary tree of height h is 2h+1 - 1. A binary tree of height h with the maximum number of nodes is called a full binary tree.
*3.19 | Suppose trees are implemented by leftmost-child, right-sibling and parent pointers. Give nonrecursive preorder, postorder, and inorder traversal algorithms that do not use "states" or a stack, as Fig. 3.9 does.
3.20 | Suppose characters a, b, c, d, e, f have probabilities .07, .09, .12, .22, .23, .27, respectively. Find an optimal Huffman code and draw the Huffman tree. What is the average code length?
*3.21 | Suppose T is a Huffman tree, and that the leaf for symbol a has greater depth than the leaf for symbol b. Prove that the probability of symbol b is no less than that of a.
*3.22 | Prove that Huffman's algorithm works, i.e., it produces an optimal code for the given probabilities. Hint: Use Exercise 3.21.

Bibliographic Notes

Berge [1958] and Harary [1969] discuss the mathematical properties of trees. Knuth [1973] and Nievergelt [1974] contain additional information on binary search trees. Many of the works on graphs and applications referenced in Chapter 6 also cover material on trees.

The algorithm given in Section 3.4 for finding a tree with a minimal weighted path length is from Huffman [1952]. Parker [1980] gives some more recent explorations into that algorithm.

† Recall our discussion of recursion in Section 2.6 in which we illustrated how the implementation of a recursive procedure involves a stack of activation records. If we examine Fig. 3.8, we can observe that when PREORDER(n) is called, the active procedure calls, and therefore the stack of activation records, correspond to the calls of PREORDER for all the ancestors of n. Thus our nonrecursive preorder procedure, like the example in Section 2.6, models closely the way the recursive procedure is implemented.

† For the data reading phase, which we omit, we also need a cursor for the array ALPHABET as it fills with symbols and their probabilities.

Table of Contents Go to Chapter 4


Data Structures and Algorithms: CHAPTER 4: Basic Operations on Sets

 Next Chapter Return to Table of Contents Previous Chapter

Basic Operations on Sets

The set is the basic structure underlying all of mathematics. In algorithm design, sets are used as the basis of many important abstract data types, and many techniques have been developed for implementing set-based abstract data types. In this chapter we review the basic operations on sets and introduce some simple implementations for sets. We present the "dictionary" and "priority queue," two abstract data types based on the set model. Implementations for these abstract data types are covered in this and the next chapter.

4.1 Introduction to Sets

A set is a collection of members (or elements); each member of a set either is itself a set or is a primitive element called an atom. All members of a set are different, which means no set can contain two copies of the same element.

When used as tools in algorithm and data structure design, atoms usually are integers, characters, or strings, and all elements in any one set are usually of the same type. We shall often assume that atoms are linearly ordered by a relation, usually denoted "<" and read "less than" or "precedes." A linear order < on a set S satisfies two properties:

  1. For any a and b in S, exactly one of a < b, a = b, or b < a is true.

  2. For all a, b, and c in S, if a < b and b < c, then a < c (transitivity).

Integers, reals, characters, and character strings have a natural linear ordering for which < is used in Pascal. A linear ordering can be defined on objects that consist of sets of ordered objects. We leave as an exercise how one develops such an ordering. For example, one question to be answered in constructing a linear order for a set of integers is whether the set consisting of integers 1 and 4 should be regarded as being less than or greater than the set consisting of 2 and 3.

Set Notation

A set of atoms is generally exhibited by putting curly brackets around its members, so {1, 4} denotes the set whose only members are 1 and 4. We should bear in mind that a set is not a list, even though we represent sets in this manner as if they were lists. The order in which the elements of a set are listed is irrelevant, and we could just as well have written {4, 1} in place of {1, 4}. Note also that in a set each element appears exactly once, so {1, 4, 1} is not a set.†

Sometimes we represent sets by set formers, which are expressions of the form

{ x | statement about x }

where the statement about x is a predicate that tells us exactly what is needed for an arbitrary object x to be in the set. For example, {x | x is a positive integer and x � 1000} is another way of representing {1, 2, . . . , 1000}, and {x | for some integer y, x = y2} denotes the set of perfect squares. Note that the set of perfect squares is infinite and cannot be represented by listing its members.

The fundamental relationship of set theory is membership, which is denoted by the symbol �. That is, x � A means that element x is a member of set A; the element x could be an atom or another set, but A cannot be an atom. We use x � A for "x is not a member of A." There is a special set, denoted Ø and called the null set or empty set, that has no members. Note that Ø is a set, not an atom, even though the set Ø does not have any members. The distinction is that x � Ø is false for every x, whereas if y is an atom, then x � y doesn't even make sense; it is syntactically meaningless rather than false.

We say set A is included (or contained) in set B, written A � B, or B � A, if every member of A is also a member of B. We also say A is a subset of B and B is a superset of A, if A � B. For example, {1, 2} � {1, 2, 3}, but {1, 2, 3} is not a subset of {1, 2} since 3 is a member of the former but not the latter. Every set is included in itself, and the empty set is included in every set. Two sets are equal if each is included in the other, that is, if their members are the same. Set A is a proper subset or proper superset of set B if A � B, and A � B or A � B, respectively.

The most basic operations on sets are union, intersection, and difference. If A and B are sets, then A � B, the union of A and B, is the set of elements that are members of A or B or both. The intersection of A and B, written A � B, is the set of elements in both A and B, and the difference, A - B, is the set of elements in A that are not in B. For example, if A = {a, b, c} and B = {b, d}, then A � B = {a, b, c, d}, A � B = {b}, and A - B = {a, c}.

Abstract Data Types Based on Sets

We shall consider ADT's that incorporate a variety of set operations. Some collections of these operations have been given special names and have special implementations of high efficiency. Some of the more common set operations are the following.

  1. -3. The three procedures UNION(A, B, C), INTERSECTION(A, B, C), and DIFFERENCE(A, B, C) take set-valued arguments A and B, and assign the result, A � B, A � B, or A - B, respectively, to the set variable C.
  4. We shall sometimes use an operation called merge, or disjoint set union, that is no different from union, but that assumes its operands are disjoint (have no members in common). The procedure MERGE(A, B, C) assigns to the set variable C the value A � B, but is not defined if A � B � Ø, i.e., if A and B are not disjoint.

  5. The function MEMBER(x, A) takes set A and object x, whose type is the type of elements of A, and returns a boolean value -- true if x � A and false if x � A.

  6. The procedure MAKENULL(A) makes the null set be the value for set variable A.

  7. The procedure INSERT(x, A), where A is a set-valued variable, and x is an element of the type of A's members, makes x a member of A. That is, the new value of A is A � {x}. Note that if x is already a member of A, then INSERT(x, A) does not change A.

  8. DELETE(x, A) removes x from A, i.e., A is replaced by A - {x}. If x is not in A originally, DELETE(x, A) does not change A.

  9. ASSIGN(A, B) sets the value of set variable A to be equal to the value of set variable B.

  10. The function MIN(A) returns the least element in set A. This operation may be applied only when the members of the parameter set are linearly ordered. For example, MIN({2, 3, 1}) = 1 and MIN({'a','b','c'}) = 'a'. We also use a function MAX with the obvious meaning.

  11. EQUAL(A, B) is a function whose value is true if and only if sets A and B consist of the same elements.

  12. The function FIND(x) operates in an environment where there is a collection of disjoint sets. FIND(x) returns the name of the (unique) set of which x is a member.

4.2 An ADT with Union, Intersection, and Difference

We begin by defining an ADT for the mathematical model "set" with the three basic set-theoretic operations, union, intersection, and difference. First we give an example where such an ADT is useful and then we discuss several simple implementations of this ADT.

Example 4.1. Let us write a program to do a simple form of "data-flow analysis" on flowcharts that represent procedures. The program will use variables of an abstract data type SET, whose operations are UNION, INTERSECTION, DIFFERENCE, EQUAL, ASSIGN, and MAKENULL, as defined in the previous section.

In Fig. 4.1 we see a flowchart whose boxes have been named B1, . . . , B8, and for which the data definitions (read and assignment statements) have been numbered 1, 2, . . . , 9. This flowchart happens to implement the Euclidean algorithm, to compute the greatest common divisor of inputs p and q, but the details of the algorithm are not relevant to the example.

In general, data-flow analysis refers to that part of a compiler that examines a flowchart-like representation of a source program, such as Fig. 4.1, and collects information about what can be true as control reaches each box of the flowchart. The boxes are often called blocks or basic blocks, and they represent collections of statements through which the flow-of-control proceeds sequentially. The information collected during data-flow analysis is used to help improve the code generated by the compiler. For example, if data-flow analysis told us that each time control reached block B, variable x had the value 27, then we could substitute 27 for all uses of x in block B, unless x were assigned a new value within block B. If constants can be accessed more quickly than variables, this change could speed up the code produced by the compiler.

In our example, we want to determine where a variable could last have been given a new value. That is, we want to compute for each block Bi the set DEFIN[i] of data definitions d such that there is a path from B1 to Bi in which d appears, but is not followed by any other definition of the same variable as d defines. DEFIN[i] is called the set of reaching definitions for Bi.

To see how such information could be useful, consider Fig. 4.1. The first block B1 is a "dummy" block of three data definitions, making the three variables t, p, and q have "undefined" values. If we discover, for example, that DEFIN[7] includes definition 3, which gives q an undefined value, then the program might contain a bug, as apparently it could print q without first assigning a valid value to q. Fortunately, we shall discover that it is impossible to reach block B7 without assigning to q; that is, 3 is not in DEFIN[7].

The computation of the DEFIN[i]'s is aided by several rules. First, we precompute for each block i two sets GEN[i] and KILL[i]. GEN[i] is the set of data definitions in block i, with the exception that if Bi contains two or more definitions of variable x, then only the last is in GEN[i]. Thus, GEN[i] is the set of definitions in Bi that are "generated" by Bi; they reach the end of Bi without having their variables redefined.

The set KILL[i] is the set of definitions d not in Bi such that Bi has a definition of the same variable as d. For example, in Fig. 4.1, GEN[4] = {6}, since definition 6 (of variable t) is in B4 and there are no subsequent definitions of t in B4. KILL[4] = {l, 9}, since these are the definitions of variable t that are not in B4.

Fig. 4.1. A flowchart of the Euclidean algorithm.

In addition to the DEFIN[i]'s, we compute the set DEFOUT[i] for each block Bi. Just as DEFIN[i] is the set of definitions that reach the beginning of Bi, DEFOUT[i] is the set of definitions reaching the end of Bi. There is a simple formula relating DEFIN and DEFOUT, namely

That is, definition d reaches the end of Bi if and only if it either reaches the beginning of Bi and is not killed by Bi, or it is generated in Bi. The second rule relating DEFIN and DEFOUT is that DEFIN[i] is the union, over all predecessors p of Bi, of DEFOUT[p], that is:

Rule (4.2) says that a data definition enters Bi if and only if it reaches the end of one of Bi's predecessors. As a special case, if Bi has no predecessors, as B1 in Fig. 4.1, then DEFIN[i] = �.

Because we have introduced a variety of new concepts in this example, we shall not try to complicate matters by writing a general algorithm for computing the reaching definitions of an arbitrary flowgraph. Rather, we shall write a part of a program that assumes GEN[i] and KILL[i] are available for i = 1, . . . , 8 and computes DEFIN[i] and DEFOUT[i] for 1, . . . , 8, assuming the particular flowgraph of Fig. 4.1. This program fragment assumes the existence of an ADT SET with operations UNION, INTERSECTION, DIFFERENCE, EQUAL, ASSIGN, and MAKENULL; we shall give alternative implementations of this ADT later.

The procedure propagate( GEN, KILL, DEFIN, DEFOUT) applies rule (4.1) to compute DEFOUT for a block, given DEFIN. If a program were loop-free, then the calculation of DEFOUT would be straightforward. The presence of a loop in the program fragment of Fig. 4.2 necessitates an iterative procedure. We approximate DEFIN[i] by starting with DEFIN[i] = Ø and DEFOUT[i] = GEN[i] for all i, and then repeatedly apply (4.1) and (4.2) until no more changes to DEFIN's and DEFOUT's occur. Since each new value assigned to DEFIN[i] or DEFOUT[i] can be shown to be a superset (not necessarily proper) of its former value, and there are only a finite number of data definitions in any program, the process must converge eventually to a solution to (4.1) and (4.2).

The successive values of DEFIN[i] after each iteration of the repeat-loop are shown in Fig. 4.3. Note that none of the dummy assignments 1, 2, and 3 reaches a block where their variable is used, so there are no undefined variable uses in the program of Fig. 4.1. Also note that by deferring the application of (4.2) for Bi until just before we apply (4.1) for Bi would make the process of Fig. 4.2 converge in fewer iterations in general.

4.3 A Bit-Vector Implementation of Sets

The best implementation of a SET ADT depends on the operations to be performed and on the size of the set. When all sets in our domain of discourse are subsets of a small "universal set" whose elements are the integers 1, . . . , N for some fixed N, then we can use a bit-vector (boolean array) implementation. A set is represented by a bit vector in which the ith bit is true if i is an element of the set. The major advantage of this representation

    var
         GEN, KILL, DEFIN, DEFOUT: array[l..8] of SET;
             { we assume GEN and KILL are computed externally }
         i: integer;
         changed: boolean;

    procedure propagate ( G, K, I: SET; var
O: SET );
        { apply (4.1) and set changed to true if
            a change in DEFOUT is detected }
        var
            TEMP: SET;
        begin
            DIFFERENCE(I, K, TEMP);
            UNION(TEMP, G, TEMP);
            if not EQUAL(TEMP, O) do begin
                ASSIGN(O, TEMP);
                changed := true
            end
         end; { propagate }

    begin
             for i:= 1 to 8 do
             ASSIGN(DEFOUT[i], GEN[i]);
         repeat
             changed := false;
             {the next 8 statements apply (4.2) for the
                 graph of Fig. 4.1 only}
             MAKENULL(DEFIN[1]);
             ASSIGN(DEFIN[2], DEFOUT[1]);
             ASSIGN(DEFIN[3], DEFOUT[2]);
             ASSIGN(DEFIN[4], DEFOUT[3]);
             UNION(DEFOUT[4], DEFOUT[8], DEFIN[5]);
             UNION(DEFOUT[3], DEFOUT[5], DEFIN[6]);
             ASSIGN(DEFIN[7], DEFOUT[6]);
             ASSIGN(DEFIN[8], DEFOUT[6]);
             for i:= 1 to 8 do
                 propagate(GEN[i], KILL[i],
DEFIN[i], DEFOUT[i]);
         until
             not changed
    end.

Fig. 4.2. Program to compute reaching definitions.

Fig. 4.3. DEFIN[i] after each iteration.

is that MEMBER, INSERT, and DELETE operations can be performed in constant time by directly addressing the appropriate bit. UNION, INTERSECTION, and DIFFERENCE can be performed in time proportional to the size of the universal set.

If the universal set is sufficiently small so that a bit vector fits in one computer word, then UNION, INTERSECTION, and DIFFERENCE can be performed by single logical operations in the language of the underlying machine. Certain small sets can be represented directly in Pascal using the set construct. The maximum size of such sets depends on the particular compiler used and, unfortunately, it is often too small to be used in typical set problems. However, in writing our own programs, we need not be constrained by any limit on our set size, as long as we can treat our sets as subsets of some universal set {1, . . . , N}. We intend that if A is a set represented as a boolean array, then A[i] is true if and only if element i is a member of A. Thus, we can define an ADT SET by the Pascal declaration

    const
         N = { whatever value is appropriate };
    type
         SET = packed array[1..N] of boolean;

We can then implement the procedure UNION as shown in Fig. 4.4. To implement INTERSECTION and DIFFERENCE, we replace "or" in Fig. 4.4 by "and" and "and not," respectively. The reader can implement the other operations mentioned in Section 4.1 (except MERGE and FIND, which make little sense in this context) as easy exercises.

It is possible to use the bit-vector implementation of sets when the universal set is a finite set other than a set of consecutive integers. Normally, we would then need a way to translate between members of the universal set and the integers 1, . . . , N. Thus, in Example 4.1 we assumed that the data definitions were assigned numbers from 1 to 9. In general, the translations in

    procedure UNION ( A, B: SET; var C: SET );
        var
           i: integer;
        begin
           for i := 1 to N do
           C[i] := A[i] or B[i]
        end

Fig. 4.4. Implementation of UNION.

both directions could be performed by the MAPPING ADT described in Chapter 2. However, the inverse translation from integers to elements of the universal set can be accomplished better using an array A, where A[i] is the element corresponding to integer i.

4.4 A Linked-List Implementation of Sets

It should also be evident that sets can be represented by linked lists, where the items in the list are the elements of the set. Unlike the bit-vector representation, the list representation uses space proportional to the size of the set represented, not the size of the universal set. Moreover, the list representation is somewhat more general since it can handle sets that need not be subsets of some finite universal set.

When we have operations like INTERSECTION on sets represented by linked lists, we have several options. If the universal set is linearly ordered, then we can represent a set by a sorted list. That is, we assume all set members are comparable by a relation "<" and the members of a set appear on a list in the order e1, e2, . . . , en, where e1 < e2 < e3 < . . . < en. The advantage of a sorted list is that we do not need to search the entire list to determine whether an element is on the list.

An element is in the intersection of lists L1 and L2 if and only if it is on both lists. With unsorted lists we must match each element on L1 with each element on L2, a process that takes O(n2) steps on lists of length n. The reason that sorting the lists makes intersection and some other operations easy is that if we wish to match an element e on one list L1 with the elements of another list L2, we have only to look down L2 until we either find e, or find an element greater than e; in the first case we have found the match, while in the second case, we know none exists. More importantly, if d is the element on L1 that immediately precedes e, and we have found onL2 the first element, say f, such that d � f, then to search L2 for an occurrence of e we can begin with f. The conclusion from this reasoning is that we can find matches for all the elements of L1, if they exist, by scanning L1 and L2 once, provided we advance the position markers for the two lists in the proper order, always advancing the one with the smaller element. The routine to implement INTERSECTION is shown in Fig. 4.5. There, sets are represented by linked lists of "cells" whose type is defined

    type
         celltype = record
              element: elementtype;
              next: � celltype
         end

Figure 4.5 assumes elementtype is a type, such as integer, that can be compared by <. If not, we have to write a function that determines which of two elements precedes the other.

The linked lists in Fig. 4.5 are headed by empty cells that serve as entry points to the lists. The reader may, as an exercise, write this program in a more general abstract form using list primitives. The program in Fig. 4.5, however, may be more efficient than the more abstract program. For example, Fig. 4.5 uses pointers to particular cells rather than "position" variables that point to previous cells. We can do so because we only append to list C, and A and B are only scanned, with no insertions or deletions done on those lists.

The operations of UNION and DIFFERENCE can be written to look surprisingly like the INTERSECTION procedure of Fig. 4.5. For UNION, we must attach all elements from either the A or B list to the C list, in their proper, sorted order, so when the elements are unequal (lines 12-14), we add the smaller to the C list just as we do when the elements are equal. We also append to list C all elements on the list not exhausted when the test of line (5) fails. For DIFFERENCE we do not add an element to the C list when equal elements are found. We only add the current A list element to the C list when it is smaller than the current B list element; for then we know the former cannot be found on the B list. Also, we add to C those elements on A when and if the test of line (5) fails because B is exhausted.

The operator ASSIGN(A, B) copies list A into list B. Note that, this operator cannot be implemented simply by making the header cell of A point to the same place as the header cell of B, because in that case, subsequent changes to B would cause unexpected changes to A. The MIN operator is especially easy; just return the first element on the list. DELETE and FIND can be implemented by finding the target item as discussed for general lists and in the case of a DELETE, disposing of its cell.

Lastly, insertion is not difficult to implement, but we must arrange to insert the new element into the proper position. Figure 4.6 shows a procedure INSERT that takes as parameters an element and a pointer to the header cell of a list, and inserts the element into the list. Figure 4.7 shows the crucial cells and pointers just before (solid) and after (dashed) insertion.

      procedure INTERSECTION ( ahead, bhead: � celltype;
                     var pc: � celltype );
        { computes the intersection of sorted lists A and B with
             header cells ahead and bhead, leaving the result
         as a sorted list whose header is pointed to by pc }
      var
        acurrent, bcurrent, ccurrent: � celltype;
        {the current cells of lists A and B, and the last cell added list C }
      begin
(1)        new(pc); { create header for list C }
(2)        acurrent := ahead�.next;
(3)        bcurrent := bhead �.next;
(4)        ccurrent := pc;
(5)        while (acurrent <> nil) and (bcurrent <> nil)
do begin
            { compare current elements on lists A and B }
(6)             if acurrent �.element =
bcurrent �.element then begin
           { add to intersection }
(7)                new( ccurrent �.next );
(8)                ccurrent := ccurrent �.next;
(9)                ccurrent �.element := acurrent
�.element;
(10                acurrent := acurrent�.next;
(11                bcurrent := bcurrent�.next
        end
        else { elements unequal }
(12)                 if acurrent�.element <
bcurrent�.element then
(13)                        acurrent := acurrent�.next
             else
(14)                        bcurrent := bcurrent�.next
        end;
(15)    ccurrent�.next := nil
     end; { INTERSECTION }

Fig. 4.5. Intersection procedure using sorted lists.

4.5 The Dictionary

When we use a set in the design of an algorithm, we may not need powerful operations like union and intersection. Often, we only need to keep a set of "current" objects, with periodic insertions and deletions from the set. Also, from time to time we may need to know whether a particular element is in the set. A set ADT with the operations INSERT, DELETE, and MEMBER has been given the name dictionary. We shall also include MAKENULL as a dictionary operation to initialize whatever data structure is used in the implementation. Let us consider an example application of the dictionary, and then discuss implementations that are well suited for representing dictionaries.

    procedure INSERT ( x: elementtype; p: � celltype );
         { inserts x onto list whose header is pointed to by p }
         var
        current, newcell: � celltype;
         begin
        current := p;
        while current�.next <>
nil do begin
             if current�.next�.element = x then
            return; { if x is already on the list, return }
             if current�.next�.element > x then
            goto add; { break }
             current: = current�.next
        end;
         add: { current is now the cell after which x is to be inserted }
        new(newcell);
        newcell�.element := x;
        newcell�.next :=
current�.next;
        current�.next := newcell
         end; { INSERT }

Fig. 4.6. Insertion procedure.

Example 4.2. The Society for the Prevention of Injustice to Tuna (SPIT) keeps a database recording the most recent votes of legislators on issues of importance to tuna lovers. This database is, conceptually, two sets of legislators' names, called goodguys and badguys. The society is very forgiving of past mistakes, but also tends to forget former friends easily. For example, if a vote to declare Lake Erie off limits to tuna fishermen is taken, all legislators voting in favor will be inserted into goodguys and deleted from badguys, while the opposite will happen to those voting against. Legislators not voting remain in whatever set they were in, if any.

In operation, the database system accepts three commands, each represented by a single character, followed by a ten character string denoting the name of a legislator. Each command is on a separate line. The commands are:

  1. F (a legislator voting favorably follows)

  2. U (a legislator voting unfavorably follows)

  3. ? (determine the status of the legislator that follows).

We also allow the character 'E' on the input line to signal the end of processing. Figure 4.8 shows the sketch of the program, written in terms of the as-yet-undefined ADT DICTIONARY, which in this case is intended to be a set of strings of length 10.

Fig. 4.7. The insertion picture.

4.6 Simple Dictionary Implementations

A dictionary can be implemented by a sorted or unsorted linked list. Another possible implementation of a dictionary is by a bit vector, provided the elements of the underlying set are restricted to the integers 1, . . . , N for some N, or are restricted to a set that can be put in correspondence with such a set of integers.

A third possible implementation of a dictionary is to use a fixed- length array with a pointer to the last entry of the array in use. This implementation is only feasible if we can assume our sets never get larger than the length of the array. It has the advantage of simplicity over the linked-list representation, while its disadvantages are that (1) sets cannot grow arbitrarily, (2 deletion is slower, and (3) space is not utilized efficiently if sets are of varying sizes.

It is for the last of these reasons that we did not discuss the array implementation in connection with sets whose unions and intersections were taken frequently. Since arrays as well as lists can be sorted, however, the reader could consider the array implementation we now describe for dictionaries as a possible implementation for sets in general. Figure 4.9 shows the declarations and procedures necessary to supplement Fig. 4.8 to make it a working program.

    program tuna ( input, output );
           { legislative database }
           type
              nametype = array[l..10] of char;
           var
              command: char;
              legislator: nametype;
              goodguys, badguys: DICTIONARY;

           procedure favor (friend: nametype );
              begin
                  INSERT(friend, goodguys) ;
                  DELETE(friend, badguys)
              end; { favor }

           procedure unfavor ( foe: nametype );
              begin
                  INSERT(foe, badguys ) ;
                  DELETE(foe, goodguys )
              end; { unfavor }
           procedure report ( subject: nametype );
              begin
                  if MEMBER(subject, goodguys) then
                      writeln(subject, 'is a friend')
                  else if MEMBER(subject, badguys) then
                      writeln(subject, 'is a foe')
                  else
                      writeln('we have no information about ', subject)
                  end; { report }

    begin { main program }
            MAKENULL(goodguys);
            MAKENULL(badguys);
            read(command);
            while command < > 'E' do begin
                readln (legislator);
                if command = 'F' then
                     favor(legislator)
                else if command = 'U' then
                     unfavor(legislator)
                else if command = '?' then
                     report(legislator)
                else
                     error('unknown command');
                read(command)
            end
    end. { tuna }

Fig. 4.8. Outline of the SPIT database program.

    const
       maxsize = { some suitable number };
    type
       DICTIONARY = record
          last: integer;
          data: array[l..maxsize] of nametype
       end;
    procedure MAKENULL ( var A: DICTIONARY );
       begin
          A.last := 0
       end; { MAKENULL }
    function MEMBER ( x: nametype; var A: DICTIONARY ):
boolean;
       var
          i: integer;
       begin
          for i := 1 to A.last do
             if A.data [i] = x then return (true);
          return (false) { if x is not found }
       end; { MEMBER }
    procedure INSERT ( x: nametype; var A: DICTIONARY );
       begin
          if not MEMBER(x, A ) then
             If A.last < maxsize then begin
                A.last := A.last + 1;
                A.data[A.last ] := x
             end
          else error ('database is full')
       end; { INSERT }
    procedure DELETE ( x: nametype; var A: DICTIONARY );
       var
          i: integer;
       begin
          if A.last > 0 then begin
             i := 1;
             while (A.data[i] <> x) and (i <
A.last) do
                i := i + 1;
             { when we reach here, either x has been found, or
                we are at the last element in set A, or both }
             if A.data[i] = x then begin
                A.data[i] := A.data[A.last];
                       { move the last element into the place of x;
                          Note that if i = A.last, this step does
                          nothing, but the next step will delete x }
                    A.last := A.last - 1
                 end
              end
           end; { DELETE }

Fig. 4.9. Type and procedure declarations for array dictionary.

4.7 The Hash Table Data Structure

The array implementation of dictionaries requires, on the average, O(N) steps to execute a single INSERT, DELETE, or MEMBER instruction on a dictionary of N elements; we get a similar speed if a list implementation is used. The bit-vector implementation takes constant time to do any of these three operations, but we are limited to sets of integers in some small range for that implementation to be feasible.

There is another important and widely useful technique for implementing dictionaries called "hashing." Hashing requires constant time per operation, on the average, and there is no requirement that sets be subsets of any particular finite universal set. In the worst case, this method requires time proportional to the size of the set for each operation, just as the array and list implementations do. By careful design, however, we can make the probability of hashing requiring more than a constant time per operation be arbitrarily small.

We shall consider two somewhat different forms of hashing. One, called open or external hashing, allows the set to be stored in a potentially unlimited space, and therefore places no limit on the size of the set. The second, called closed or internal hashing, uses a fixed space for storage and thus limits the size of sets.†

Open Hashing

In Fig. 4.10 we see the basic data structure for open hashing. The essential idea is that the (possibly infinite) set of potential set members is partitioned into a finite number of classes. If we wish to have B classes, numbered 0, 1, . . . , B-1, then we use a hash function h such that for each object x of the data type for members of the set being represented, h(x) is one of the integers 0 through B-1. The value of h(x), naturally, is the class to which x belongs. We often call x the key and h(x) the hash value of x. The "classes" we shall refer to as buckets, and we say that x belongs to bucket h(x).

In an array called the bucket table, indexed by the bucket numbers 0, 1, . . . , B-1, we have headers for B lists. The elements on the ith list are the members of the set being represented that belong to class i, that is, the elements x in the set such that h(x) = i.

It is our hope that the buckets will be roughly equal in size, so the list for each bucket will be short. If there are N elements in the set, then the average bucket will have N/B members. If we can estimate N and choose B to be roughly as large, then the average bucket will have only one or two members, and the dictionary operations take, on the average, some small constant number of steps, independent of what N (or equivalently B) is.

Fig. 4.10. The open hashing data organization.

It is not always clear that we can select h so that a typical set will have its members distributed fairly evenly among the buckets. We shall later have more to say about selecting h so that it truly "hashes" its argument, that is, so that h(x) is a "random" value that is not dependent on x in any trivial way.

    function h ( x: nametype ) : O..B - 1;
       var
        i, sum: integer
       begin
        sum: = 0;
        for i: 1 to 10 do
           sum: = sum + ord(x[i]);
        h : = sum + ord(x[i]);
       end { h }

Fig. 4.11. A simple hash function h.

Let us here, for specificity, introduce one hash function on character strings that is fairly good, although not perfect. The idea is to treat the characters as integers, using the character code of the machine to define the correspondence. Pascal provides the ord built-in function, where ord(c) is the integer code for character c. Thus, if x is a key and the type of keys is array[l..10] of char (what we called nametype in Example 4.2), we might declare the hash function h as in Fig. 4.11. In that function we sum the integers for each character and divide the result by B, taking the remainder, which is an integer from 0 to B - 1.

In Fig. 4.12 we see the declarations of the data structures for an open hash table and the procedures implementing the operations for a dictionary. The member type for the dictionary is assumed to be nametype (a character array), so these declarations can be used in Example 4.2 directly. We should note that in Fig. 4.12 we have made the headers of the bucket lists be pointers to cells, rather than complete cells. We did so because the bucket table, where the headers are, could take as much space as the lists themselves if we made the bucket be an array of cells rather than pointers. Notice, however, that we pay a price for saving this space. The code for the DELETE procedure now must distinguish the first cell from the remaining ones.

    const
       B = { suitable constant };
    type
       celltype = record
        element: nametype;
        next: � celltype
       end;
       DICTIONARY = array[0..B - l] of � celltype;

    procedure MAKENULL ( var A: DICTIONARY );
       var
        i: integer;
       begin
        for i:= 0 to B-1 do
            A [i]:= nil
       end; { MAKENULL }

    function MEMBER ( x: nametype; var A: DICTIONARY ) : boolean;
       var
        current: � celltype;
       begin
        current := A [h (x)];
        { initially current is the header of x's bucket }
        while current < > nil do
           if current�.element =
x then
            return (true)
           else
            current := current�.next;
        return (false) { if x is not found }
       end; { MEMBER }

    procedure INSERT ( x: nametype; var A: DICTIONARY );
       var
        bucket: integer;
        oldheader: � celltype;
       begin

        if not MEMBER(x, A) then begin
           bucket := h(x);
           oldheader := A [bucket ];
           new (A [bucket]);
           A [bucket ]�.element :=
x;
           A [bucket ]�.next: =
oldheader
        end
       end; { INSERT }

    procedure DELETE ( x: nametype; var A: DICTIONARY );
       var
          current: � celltype;
          bucket: integer;
    begin
      bucket := h(x);
      if A [bucket ] < > nil then begin
          if A [bucket]�.element
= x then { x is in first cell }
                     A [bucket ] := A [bucket ]�.next { remove x from list }
      else begin { x, if present at all, is not in first cell of bucket }
          current := A [bucket ]; { current points to the previous
cell }
          while current�.next <> nil
do
                   if current�.next�.element = x then begin
               current�.next :=
current�.next�.next;
                   { remove x from list }
              return { break }
              end
              else {x not yet found}
                 current := current�.next
        end
    end
   end; { DELETE }

Fig. 4.12. Dictionary implementation by an open hash table.

Closed Hashing

A closed hash table keeps the members of the dictionary in the bucket table itself, rather than using that table to store list headers. As a consequence, it appears that we can put only one element in any bucket. However, associated with closed hashing is a rehash strategy. If we try to place x in bucket h(x) and find it already holds an element, a situation called a collision occurs, and the rehash strategy chooses a sequence of alternative locations, h1(x), h2(x), . . . , within the bucket table in which we could place x. We try each of these locations, in order, until we find an empty one. If none is empty, then the table is full, and we cannot insert x.

Example 4.3. Suppose B = 8 and keys a, b, c, and d have hash values h(a) = 3, h(b) = 0, h(c) = 4, h(d) = 3. We shall use the simplest rehash strategy, called linear hashing, where hi(x) = (h(x) + i) mod B. Thus, for example, if we wished to insert a and found bucket 3 already filled, we would try buckets 4, 5, 6, 7, 0, 1, and 2, in that order.

Initially, we assume the table is empty, that is, each bucket holds a special value empty, which is not equal to any value we might try to insert.† If we insert a, b, c, and d, in that order, into an initially empty table, we find a goes into bucket 3, b into 0 and c into 4. When we try to insert d, we first try h(d) = 3 and find it filled. Then we try h1(d) = 4 and find it filled as well. Finally, when we try h2(d) = 5, we find an empty space and put d there. The resulting positions are shown in Fig. 4.13.

Fig. 4.13. Partially filled hash table.

The membership test for an element x requires that we examine h(x), hi(x), h2(x), . . . , until we either find x or find an empty bucket. To see why we can stop looking if we reach an empty bucket, suppose first that there are no deletions permitted. If, say, h3(x) is the first empty bucket found in the series, it is not possible that x is in bucket h4(x), h5(x), or further along the sequence, because x could not have been placed there unless h3(x) were filled at the time we inserted x.

Note, however, that if we allow deletions, then we can never be sure, if we reach an empty bucket without finding x, that x is not somewhere else, and the now empty bucket was occupied when x was inserted. When we must do deletions, the most effective approach is to place a constant called deleted into a bucket that holds an element that we wish to delete. It is important that we can distinguish between deleted and empty, the constant found in all buckets that have never been filled. In this way it is possible to permit deletions without having to search the entire table during a MEMBER test. When inserting we can treat deleted as an available space, so with luck the space of a deleted element will eventually be reused. However, as the space of a deleted element is not immediately reclaimable as it is with open hashing, we may prefer the open to the closed scheme.

Example 4.4. Suppose we wish to test if e is in the set represented by Fig. 4.13. If h(e) = 4, we try buckets 4, 5, and then 6. Bucket 6 is empty and since we have not found e, we conclude e is not in the set.

If we delete c, we must put the constant deleted in bucket 4. In that way, when we look for d, and we begin at h(d) = 3, we shall scan 4 and 5 to find d, and not stop at 4 as we would if we put empty in that bucket.

In Fig 4.14 we see the type declarations and operations for the DICTIONARY ADT with set members of type nametype and the closed hash table as the underlying implementation. We use an arbitrary hash function h, of which Fig. 4.11 is one possibility, and we use the linear hashing strategy to rehash in case of collisions. For convenience, we have identified empty with a string of ten blanks and deleted with a string of 10 *'s, assuming that neither of these strings could be real keys. (In the SPIT database these strings are unlikely to be names of legislators.) The procedure INSERT(x, A ) first uses locate to determine if x is already in A, and, if not, it uses a special routine locate 1 to find a location in which to insert x. Note that locate 1 searches for deleted as well as empty locations.

    const
                empty = '   '; { 10 blanks }
          deleted = '**********'{ 10 *'s }
    type
                 DICTIONARY = array[0..B -1] of nametype;
    procedure MAKENULL ( var A: DICTIONARY );
                var
         i: integer;
          begin
         for i := 0 to B - 1 do
            A [i] := empty
          end; {MAKENULL}
    function locate ( x: nametype; A: DICTIONARY ) : integer;
                { locate scans DICTIONARY from the bucket for h(x) until
        either x is found, or an empty bucket is found, or it has
        scanned completely around the table, thereby determining
        that the table does not contain x. locate returns the index
        of the bucket at which it stops for any of these reasons }
          var
         initial, i: integer;
          { initial holds h(x); i counts the number of
            buckets thus far scanned when looking for x }
          begin
         initial := h(x);
         i := 0;
         while (i < B) and (A [(initial +
i) mod B ] <> x) and
            (A [(initial + i) mod B] <>
empty) do
                i := i + 1;
         return ((initial + i) mod B)
          end; { locate }
    function locate 1 ( x: nametype; A: DICTIONARY ) : integer;
          { like locate, but it will also stop at and return a deleted entry }
    function MEMBER ( x: nametype; var A: DICTIONARY ) :
boolean;
          begin
              if A [locate(x)] = x then
                return (true)
              else
                return (false)
           end; { MEMBER }
    procedure INSERT (x: nametype; var A: DICTIONARY );
           var
          bucket: integer;
    begin
           if A[locate (x)] = x then return; { x
is already in A }
           bucket := locate 1(x);
           if (A [bucket] = empty) or (A
[bucket] = deleted) then
            A [bucket] := x
           else
            error('INSERT failed: table is full')
    end; { INSERT }
    procedure DELETE ( x: nametype; var A: DICTIONARY );
           var
            bucket: integer;
           begin
            bucket := locate(x);
                   if A [bucket] = x then
                A [bucket] := deleted
    end; { DELETE }

Fig. 4.14. Dictionary implementation by a closed hash table.

4.8 Estimating the Efficiency of Hash Functions

As we have mentioned, hashing is an efficient way to represent dictionaries and some other abstract data types based on sets. In this section we shall examine the average time per dictionary operation in an open hash table. If there are B buckets and N elements stored in the hash table, then the average bucket has N/B members, and we expect that an average INSERT, DELETE, or MEMBER operation will take O(1 + N/B) time. The constant 1 represents the time to find the bucket, and N/B the time to search the bucket. If we can choose B to be about N, this time becomes a constant per operation. Thus the average time to insert, delete, or test an element for membership, assuming the element is equally likely to be hashed to any bucket, is a constant independent of N.

Suppose we are given a program written in some language like Pascal, and we wish to insert all identifiers appearing in the program into a hash table. Whenever a declaration of a new identifier is encountered, the identifier is inserted into the hash table after checking that it is not already there. During this phase it is reasonable to assume that an identifier is equally likely to be hashed into any given bucket. Thus we can construct a hash table with N elements in time O(N(1 + N/B)). By choosing B equal to N this becomes O(N).

In the next phase identifiers are encountered in the body of the program. We must locate the identifiers in the hash table to retrieve information associated with them. But what is the expected time to locate an identifier? If the element searched for is equally likely to be any element in the table, then the expected time to search for it is just the average time spent inserting an element. To see this, observe that in searching once for each element in the table, the time spent is exactly the same as the time spent inserting each element, assuming that elements are always appended to the end of the list for the appropriate bucket. Thus the expected time for a search is also 0(1 + N/B).

The above analysis assumes that a hash function distributes elements uniformly over the buckets. Do such functions exist? We can regard a function such as that of Fig. 4.11 (convert characters to integers, sum and take the remainder modulo B) as a typical hash function. The following example examines the performance of this hash function.

Example 4.5. Suppose we use the hash function of Fig. 4.11 to hash the 100 keys consisting of the character strings A0, A1, . . . , A99 into a table of 100 buckets. On the assumption that ord(O), ord(1), . . . , ord(9) form an arithmetic progression, as they do in the common character codes ASCII and EBCDIC, it is easy to check that the keys hash into at most 29 of the 100 buckets,† and the largest bucket contains A18, A27, A36, . . . , A90, or 9 of the 100 elements. If we compute the average number of steps for insertion, using the fact that the insertion of the ith element into a bucket takes i + 1 steps, we get 395 steps for these 100 keys. In comparison, our estimate N(1 + N/B) would suggest 200 steps.

The simple hashing function of Fig. 4.11 may treat certain sets of inputs, such as the consecutive strings in Example 4.5, in a nonrandom manner. "More random" hash functions are available. As an example, we can use the idea of squaring and taking middle digits. Thus, if we have a 5-digit number n as a key and square it, we get a 9 or 10 digit number. The "middle digits," such as the 4th through 7th places from the right, have values that depend on almost all the digits of n. For example, digit 4 depends on all but the leftmost digit of n, and digit 5 depends on all digits of n. Thus, if B = 100, we might choose to take digits 6 and 5 to form the bucket number.

This idea can be generalized to situations where B is not a power of 10. Suppose keys are integers in the range 0, 1, . . . , K. If we pick an integer C such that BC2 is about equal to K2, then the function

h(n) = [n2/C] mod B

effectively extracts a base-B digit from the middle of n2.

Example 4.6. If K = 1000 and B = 8, we might choose C = 354. Then

To use the "square and take the middle" strategy when keys are character strings, first group the characters in the string from the right into blocks of a fixed number of characters, say 4, padding the left with blanks, if necessary. Treat each block as a single integer, formed by concatenating the binary codes for the characters. For example, ASCII uses a 7-bit character code, so characters may be viewed as "digits" in base 27, or 128. Thus we can regard the character string abcd as the integer (128)3a + (128)2b + (128)c + d. After converting all blocks to integers, add them† and proceed as we have suggested previously for integers.

Analysis of Closed Hashing

In a closed hashing scheme the speed of insertion and other operations depends not only on how randomly the hash function distributes the elements into buckets, but also on how well the rehashing strategy avoids additional collisions when a bucket is already filled. For example, the linear strategy for resolving collisions is not as good as possible. While the analysis is beyond the scope of this book, we can observe the following. As soon as a few consecutive buckets are filled, any key that hashes to one of them will be sent, by the rehash strategy, to the end of the group, thereby increasing the length of that consecutive group. We are thus likely to find more long runs of consecutive buckets that are filled than if elements filled buckets at random. Moreover, runs of filled blocks cause some long sequences of tries before a newly inserted element finds an empty bucket, so having unusually large runs of filled buckets slows down insertion and other operations.

We might wish to know how many tries (or probes) are necessary on the average to insert an element when N out of B buckets are filled, assuming all combinations of N out of B buckets are equally likely to be filled. It is generally assumed, although not proved, that no closed hashing strategy can give a better average time performance than this for the dictionary operations. We shall then derive a formula for the cost of insertion if the alternative locations used by the rehashing strategy are chosen at random. Finally, we shall consider some strategies for rehashing that approximate random behavior.

The probability of a collision on our initial probe is N/B. Assuming a collision, our first rehash will try one of B - 1 buckets, of which N - 1 are filled, so the probability of at least two collisions is [image] Similarly, the probability of at least i collisions is

If B and N are large, this probability approximates (N/B)i. The average number of probes is one (for the successful insertion) plus the sum over all i � 1 of the probability of at least i collisions, that is, approximately [image]. It can be shown that the exact value of the summation when formula (4.3) is substituted for (N/B)i is [image], so our approximation is a good one except when N is very close to B.

Observe that [image] grows very slowly as N begins to grow from 0 to B - 1, the largest value of N for which another insertion is possible. For example, if N is half B, then about two probes are needed for the next insertion, on the average. The average insertion cost per bucket to fill M of the B buckets is [image], which is approximately [image], or [image] Thus, to fill the table completely (M = B) requires an average of logeB per bucket, or B logeB probes in total. However, to fill the table to 90% of capacity (M = .9B) only requires B((10/9) logel0) or approximately 2.56B probes.

The average cost of the membership test for a nonexistent element is the same as the cost of inserting the next element, but the cost of the membership test for an element in the set is the average cost of all insertions made so far, which is substantially less if the table is fairly full. Deletions have the same average cost as membership testing. But unlike open hashing, deletions from a closed hash table do not help speed up subsequent insertions or membership tests. It should be emphasized that if we never fill closed hash tables to more than any fixed fraction less than one, the average cost of operations is a constant; the constant grows as the permitted fraction of capacity grows. Figure 4.15 graphs the cost of insertions, deletions and membership tests, as a function of the percentage of the table that is full at the time the operation is performed.

Fig. 4.15. Average operation cost.

"Random" Strategies for Collision Resolution

We have observed that the linear rehashing strategy tends to group full buckets into large consecutive blocks. Perhaps we could get more "random" behavior if we probed at a constant interval greater than one. That is, let hi(x) = (h(x)+ci) mod B for some c > 1. For example, if B = 8, c = 3, and h(x) = 4, we would probe buckets 4, 7, 2, 5, 0, 3, 6, and 1, in that order. Of course, if c and B have a common factor greater than one, this strategy doesn't even allow us to search all buckets; try B = 8 and c = 2, for example. But more significantly, even if c and B are relatively prime (have no common factors), we have the same "bunching up" problem as with linear hashing, although here it is sequences of full buckets separated by difference c that tend to occur. This phenomenon slows down operations as for linear hashing, since an attempted insertion into a full bucket will tend to travel down a chain of full buckets separated by distance c, and the length of this chain will increase by one.

In fact, any rehashing strategy where the target of a probe depends only on the target of the previous probe (as opposed to depending on the number of unsuccessful probes so far, the original bucket h(x), or the value of the key x itself) will exhibit the bunching property of linear hashing. Perhaps the simplest strategy in which the problem does not occur is to let hi(x) = (h(x)+di) mod B where d1, d2, . . . , dB-1 is a "random" permutation of the integers 1, 2, . . . , B-1. Of course, the same sequence d1, . . . , dB-1 is used for all insertions, deletions and membership tests; the "random" shuffle of the integers is decided upon once, when we design the rehash algorithm.

The generation of "random" numbers is a complicated subject, but fortunately, many common methods do produce a sequence of "random" numbers that is actually a permutation of integers from 1 up to some limit. These random number generators, if reset to their initial values for each operation on the hash table, serve to generate the desired sequence d1, . . . , dB-1.

One effective approach uses "shift register sequences." Let B be a power of 2 and k a constant between 1 and B-1. Start with some number d1 in the range 1 to B - 1, and generate successive numbers in the sequence by taking the previous value, doubling it, and if the result exceeds B, subtracting B and taking the bitwise modulo 2 sum of the result and the selected constant k. The bitwise modulo 2 sum of x and y, written x � y, is computed by writing x and y in binary, with leading 0's if necessary so both are of the same length, and forming the numbers whose binary representation has a 1 in those positions where one, but not both, of x and y have a 1.

Example 4.7. 25 � 13 is computed by taking

25 = 11001

13 = 01101

______

25 � 13 = 10100

Note that this "addition" can be thought of as ordinary binary addition with carries from place to place ignored.

Not every value of k will produce a permutation of 1, 2, . . . , B-1; sometimes a number repeats before all are generated. However, for given B, there is a small but finite chance that any particular k will work, and we need only find one k for each B.

Example 4.8. Let B = 8. If we pick k = 3, we succeed in generating all of 1, 2, . . . , 7. For example, if we start with d1 = 5, then we compute d2 by first doubling d1 to get 10. Since 10 > 8, we subtract 8 to get 2, and then compute d2 = 2 � 3 = 1. Note that x � 3 can be computed by complementing the last two bits of x.

It is instructive to see the 3-bit binary representations of d1, d2, . . . , d7. These are shown in Fig. 4.16, along with the method of their calculation. Note that multiplication by 2 corresponds to a shift left in binary. Thus we have a hint of the origin of the term "shift register sequence."

Fig. 4.16. Calculating a shift register sequence.

The reader should check that we also generate a permutation of 1, 2, . . . ,7 if we choose k = 5, but we fail for other values of k.

Restructuring Hash Tables

If we use an open hash table, the average time for operations increases as N/B, a quantity that grows rapidly as the number of elements exceeds the number of buckets. Similarly, for a closed hash table, we saw from Fig. 4.15 that efficiency goes down as N approaches B, and it is not possible that N exceeds B.

To retain the constant time per operation that is theoretically possible with hash tables, we suggest that if N gets too large, for example N � .9B for a closed table or N � 2B for an open one, we simply create a new hash table with twice as many buckets. The insertion of the current members of the set into the new table will, on the average, take less time than it took to insert them into the smaller table, and we more than make up this cost when doing subsequent dictionary operations.

4.9 Implementation of the Mapping ADT

Recall our discussion of the MAPPING ADT from Chapter 2 in which we defined a mapping as a function from domain elements to range elements. The operations for this ADT are:

  1. MAKENULL(A) initializes the mapping A by making each domain element have no assigned range value.

  2. ASSIGN(A, d, r) defines A(d) to be r.

  3. COMPUTE(A, d, r) returns true and sets r to A(d) if A(d) is defined; false is returned otherwise.

The hash table is an effective way to implement a mapping. The operations ASSIGN and COMPUTE are implemented much like INSERT and MEMBER operations for a dictionary. Let us consider an open hash table first. We assume the hash function h(d) hashes domain elements to bucket numbers. While for the dictionary, buckets consisted of a linked list of elements, for the mapping we need a list of domain elements paired with their range values. That is, we replace the cell definition in Fig. 4.12 by

type
celltype = record
domainelement: domaintype;
range: rangetype;
next: � celltype
end

where domaintype and rangetype are whatever types domain and range
elements have in this mapping. The declaration of a MAPPING is

type
MAPPING = array[0..B-1] of � celltype

This array is the bucket array for a hash table. The procedure ASSIGN is written in Fig. 4.17. The code for MAKENULL and COMPUTE are left as an exercise.

Similarly, we can use a closed hash table as a mapping. Define cells to consist of domain element and range fields and declare a MAPPING to be an array of cells. As for open hash tables, let the hash function apply to domain elements, not range elements. We leave the implementation of the mapping operations for a closed hash table as an exercise.

4.10 Priority Queues

The priority queue is an ADT based on the set model with the operations INSERT and DELETEMIN, as well as the usual MAKENULL for initialization of the data structure. To define the new operation, DELETEMIN, we first assume that elements of the set have a "priority" function defined on

    procedure ASSIGN ( var A: MAPPING; d: domaintype;
r: rangetype );
       var
        bucket: integer;
        current: � celltype;
       begin
        bucket := h(d);
        current := A[bucket];
        while current <> nil do
           if current�.domainelement = d then begin
            current�.range := r;
{ replace old value for d }
            return
           end
           else
            current := current�.next;
        { at this point, d was not found on the list }
        current := A[bucket]; { use current to remember first cell
}
        new(A[bucket]);
        A [bucket]�.domainelement
:= d;
        A [bucket]�.range: =
r;
        A [bucket]�.next :=
current
    end; { ASSIGN }

Fig. 4.17. The procedure ASSIGN for an open hash table.

them; for each element a, p(a), the priority of a, is a real number or, more generally, a member of some linearly ordered set. The operation INSERT has the usual meaning, while DELETEMIN is a function that returns some element of smallest priority and, as a side effect, deletes it from the set. Thus, as its name implies, DELETEMIN is a combination of the operations DELETE and MIN discussed earlier in the chapter.

Example 4.9. The term "priority queue" comes from the following sort of use for this ADT. The word "queue" suggests that people or entities are waiting for some service, and the word "priority" suggests that the service is given not on a "first-come-first-served" basis as for the QUEUE ADT, but rather that each person has a priority based on the urgency of need. An example is a hospital waiting room, where patients having potentially fatal problems will be taken before any others, no matter how long the respective waits are.

As a more mundane example of the use of priority queues, a time- shared computing system needs to maintain a set of processes waiting for service. Usually, the system designers want to make short processes appear to be instantaneous (in practice, response within a second or two appears instantaneous), so these are given priority over processes that have already consumed substantial time. A process that requires several seconds of computing time cannot be made to appear instantaneous, so it is sensible strategy to defer these until all processes that have a chance to appear instantaneous have been done. However, if we are not careful, processes that have taken substantially more time than average may never get another time slice and will wait forever.

One possible way to favor short processes, yet not lock out long ones is to give process P a priority 100tused(P) - tinit(P). The parameter tused gives the amount of time consumed by the process so far, and tinit gives the time at which the process initiated, measured from some "time zero." Note that priorities will generally be large negative integers, unless we choose to measure tinit from a time in the future. Also note that 100 in the above formula is a "magic number"; it is selected to be somewhat larger than the largest number of processes we expect to be active at once. The reader may observe that if we always pick the process with the smallest priority number, and there are not too many short processes in the mix, then in the long run, a process that does not finish quickly will receive 1% of the processor's time. If that is too much or too little, another constant can replace 100 in the priority formula.

We shall represent processes by records consisting of a process identifier and a priority number. That is, we define

type
processtype = record
id: integer;
priority: integer
end;

The priority of a process is the value of the priority field, which here we have defined to be an integer. We can define the priority function as follows.

function p ( a: processtype ): integer;
begin
return (a.priority)
end;

In selecting processes to receive a time slice, the system maintains a priority queue WAITING of processtype elements and uses two procedures, initial and select, to manipulate the priority queue by the operations INSERT and DELETEMIN. Whenever a process is initiated, procedure initial is called to place a record for that process in WAITING. Procedure select is called when the system has a time slice to award to some process. The record for the selected process is deleted from WAITING, but retained by select for reentry into the queue with a new priority; the priority is increased by 100 times the amount of time used.

We make use of function currenttime, which returns the current time, in whatever time units are used by the system, say microseconds, and we use procedure execute(P) to cause the process with identifier P to execute for one time slice. Figure 4.18 shows the procedures initial and select.

    procedure initial ( P: integer );
       { initial places process with id P on the queue }
       var
        process: processtype;
       begin
        process.id := P;
        process.priority := - currenttime;
        INSERT (process, WAITING)
       end; { initial }

    procedure select;
       { select allocates a time slice to process with highest priority }
       var
        begintime, endtime: integer;
        process: processtype;
       begin
        process := � DELETEMIN(WAITING);
            { DELETEMIN returns a pointer to the deleted element }
        begintime := currenttime;
        execute (process.id);
        endtime := currenttime;
        process.priority := process.priority + 100*(endtime -
begintime);
            { adjust priority to incorporate amount of time used }
        INSERT (process, WAITING)
            { put selected process back on queue with new priority }
       end; { select }

Fig. 4.18. Allocating time to processes.

4.11 Implementations of Priority Queues

With the exception of the hash table, the set implementations we have studied so far are also appropriate for priority queues. The reason the hash table is inappropriate is that there is no convenient way to find the minimum element, so hashing merely adds complications, and does not improve performance over, say, a linked list.

If we use a linked list, we have a choice of sorting it or leaving it unsorted. If we sort the list, finding a minimum is easy -- just take the first element on the list. However, insertion requires scanning half the list on the average to maintain the sorted list. On the other hand, we could leave the list unsorted, which makes insertion easy and selection of a minimum more difficult.

Example 4.10. We shall implement DELETEMIN for an unsorted list of elements of type processtype, as defined in Example 4.9. The list is headed by an empty cell. The implementations of INSERT and MAKENULL are straightforward, and we leave the implementation using sorted lists as an exercise. Figure 4.19 gives the declaration for cells, for the type PRIORITYQUEUE, and for the procedure DELETEMIN.

Partially Ordered Tree Implementation of Priority Queues

Whether we choose sorted or unsorted lists to represent priority queues, we must spend time proportional to n to implement one or the other of INSERT or DELETEMIN on sets of size n. There is another implementation in which DELETEMIN and INSERT each require O(logn) steps, a substantial improvement for large n (say n � 100). The basic idea is to organize the elements of the priority queue in a binary tree that is as balanced as possible; an example is in Fig. 4.20. At the lowest level, where some leaves may be missing, we require that all missing leaves are to the right of all leaves that are present on the lowest level.

Most importantly, the tree is partially ordered, meaning that the priority of node v is no greater than the priority of the children of v, where the priority of a node is the priority number of the element stored at the node. Note from Fig. 4.20 that small priority numbers need not appear at higher levels than larger priority numbers. For example, level three has 6 and 8, which are less than the priority number 9 appearing on level 2. However, the parent of 6 and 8 has priority 5, which is, and must be, at least as small as the priorities of its children.

To execute DELETEMIN, we return the minimum-priority element, which, it is easily seen, must be at the root. However, if we simply remove the root, we no longer have a tree. To maintain the property of being a partially ordered tree, as balanced as possible, with leaves at the lowest level as far left as can be, we take the rightmost leaf at the lowest level and temporarily put it at the root. Figure 4.21(a) shows this change from Fig. 4.20. Then we push this element as far down the tree as it will go, by exchanging it with the one of its children having smaller priority, until the element is either at a leaf or at a position where it has priority no larger than either of its children.

In Fig. 4.21(a) we must exchange the root with its smaller-priority child, which has priority 5. The result is shown in Fig. 4.21(b). The element being pushed down is still larger than its children, with priorities 6 and 8. We exchange it with the smaller of these, 6, to reach the tree of Fig. 4.21(c). This tree has the partially ordered property.

In this percolation, if a node v has an element with priority a, if its children have elements with priorities b and c, and if at least one of b and c is smaller than a, then exchanging a with the smaller of b and c results in v having an element whose priority is smaller than either of its children. To prove this, suppose b � c. After the exchange, node v acquires b, and its

    type
       celltype = record
        element: processtype;
        next: � celltype
       end;
       PRIORITYQUEUE = � celltype;
        { cell pointed to is a list header }

    function DELETEMIN ( var A: PRIORITYQUEUE ): � celltype;
       var
        current: � celltype; { cell before one being
"scanned" }
        lowpriority: integer; { smallest priority found so far }
        prewinner: � celltype; { cell before one with
element
           of smallest priority }
       begin
        if A �.next = nil
then
            error('cannot find minimum of empty list')
        else begin
           lowpriority := p(A�.next�.element);
            { p returns priority of the first element. Note A points
                 to a header cell that does not contain an element }
           prewinner := A;
           current := A�.next;
           while current�.next <>
nil do begin
              { compare priorities of current winner
                and next element }
                      if p( current�.next�.element) <
lowpriority then begin
                prewinner := current;
                lowpriority : = p (current�.next�. element)
              end;
              current := current�.next
           end;
           DELETEMIN := prewinner�.next;
            { return pointer to the winner }
           prewinner�.next :=
prewinner�.next�.next
            { remove winner from list }
           end
        end; { DELETEMIN }

Fig. 4.19. Linked list priority queue implementation.

children hold a and c. We assumed b � c, and we were told that a is larger than at least one of b and c. Therefore b � a surely holds. Thus the insertion procedure outlined above percolates an element down the tree until there

Fig. 4.20. A partially ordered tree.

are no more violations of the partially ordered property.

Let us also observe that DELETEMIN applied to a set of n elements takes O(logn) time. This follows since no path in the tree has more than 1 + logn nodes, and the process of forcing an element down the tree takes a constant time per node. Observe that for any constant c, the quantity c(1+logn) is at most 2clogn for n � 2. Thus c(1+logn) is O(logn).

Now let us consider how INSERT should work. First, we place the new element as far left as possible on the lowest level, starting a new level if the current lowest level is all filled. Fig. 4.22(a) shows the result of placing an element with priority 4 into Fig. 4.20. If the new element has priority lower than its parent, exchange it with its parent. The new element is now at a position where it is of lower priority than either of its children, but it may also be of lower priority than its parent, in which case we must exchange it with its parent, and repeat the process, until the new element is either at the root or has larger priority than its parent. Figures 4.22(b) and (c) show how 4 moves up by this process.

We could prove that the above steps result in a partially ordered tree. While we shall not attempt a rigorous proof, let us observe that an element with priority a can become the parent of an element with priority b in three ways. (In what follows, we identify an element with its priority.)

  1. a is the new element and moves up the tree replacing the old parent of b. Let the old parent of b have priority c. Then a < c, else the exchange would not take place. But c � b, since the original tree was partially ordered. Thus a < b. For example in Fig. 4.22(c) 4 became the parent of 6, replacing a parent of larger priority, 5.

  2. a was pushed down the tree due to an exchange with the new element. In this case a must have been an ancestor of b in the original partially ordered tree. Thus a � b. For example, in Fig. 4.22(c), 5 becomes the

    Fig. 4.21. Pushing an element down the tree.

    Fig. 4.22. Inserting an element.

    parent of elements with priority 8 and 9. Originally, 5 was the parent of the former and the "grandparent" of the latter.

  3. b might be the new element, and it moves up to become a child of a. If a > b, then a and b will be exchanged at the next step, so this violation of the partially ordered property will be removed.

The time to perform an insertion is proportional to the distance up the tree that the new element travels. As for DELETEMIN, we observe that this distance can be no greater than 1 + logn, so both INSERT and DELETEMIN take O(log n) steps.

An Array Implementation of Partially Ordered Trees

The fact that the trees we have been considering are binary, balanced as much as possible, and have leaves at the lowest level pushed to the left means that we can use a rather unusual representation for these trees, called a heap. If there are n nodes, we use the first n positions of an array A. A [1] holds the root. The left child of the node in A[i], if it exists, is at A [2i], and the right child, if it exists, is at A [2i + 1]. An equivalent viewpoint is that the parent of A [i] is A [i div 2], for i > 1. Still another equivalent observation is that the nodes of a tree fill up A[1], A[2], . . . , A [n] level-by-level, from the top, and within a level, from the left. For example, Fig. 4.20 corresponds to an array containing the elements 3, 5, 9, 6, 8, 9, 10, 10, 18, 9.

We can declare a priority queue of elements of some type, say processtype as in Example 4.9, to consist of an array of processtype and an integer last, indicating the currently last element of the array that is in use. If we assume maxsize is the desired size of arrays for a priority queue, we can declare:

    type
       PRIORITYQUEUE = record
        contents: array[1..maxsize] of processtype;
        last: integer
       end;

    The priority queue operations are implemented below in Fig. 4.23.

    procedure MAKENULL ( var A: PRIORITYQUEUE );
       begin
        A .last := 0
       end; { MAKENULL }

    procedure INSERT ( x: processtype; var A:
PRIORITYQUEUE );
       var
        i: integer;
        temp : processtype;
       begin
        if A .last >= maxsize then
        error ('priority queue is full')

    else begin
       A .last := A .last + 1;
       A .contents [A .last] := x;
       i := A .last; { i is index of current position of x }
       while (i > 1) and†
(p(A .contents[i]) < p(A .contents[i
div 2])) do
        begin { push x up the tree by exchanging it with its parent
           of larger priority. Recall p computes the priority of
           a processtype element }
        temp := A .contents [i];
        A .contents [i] := A .contents [i
div 2];
        A .contents [i div 2] := temp;
        i := i div 2
       end
        end
     end; { INSERT }

     function DELETEMIN ( var A: PRIORITYQUEUE ): � processtype;
       var
        i, j: integer;
        temp : processtype;
        minimum: � processtype;
       begin
        if A .last = 0 then
            error('priority queue is empty')
       else begin
        new (minimum);
        minimum � := A .contents
[1];
           { we shall return a pointer to a copy of the root of A }
        A .contents [1] := A .contents [A .last];
        A .last := A .last - 1;
           { move the last element to the beginning }
        i := 1; { i is the current position of the old last element }
        while i < = A .last div 2 do begin
           { push old last element down tree }
           if (p (A .contents [2*i]) < p
(A .contents [2*i + 1]))
            or (2*i = A.last) then
                j := 2 * i
           else
            j := 2 * i + 1;
           { j will be the child of i having the smaller priority
            or if 2*i = A.last, then j is the only child of i }
           if p (A .contents [i]) > p (A
.contents [j]) then begin
            { exchange old last element with smaller priority child }

            temp := A.contents[i];
            A.contents[i] := A.contents[j];
            A.contents[j] := temp;
            i := j
           end
           else
            return (minimum) { cannot push further }
        end;
        return (minimum) { pushed all the way to a leaf }
       end
    end; { DELETEMIN }

Fig. 4.23. Array implementation of priority queue.

4.12 Some Complex Set Structures

In this section we consider two more complex uses of sets to represent data. The first problem is that of representing many-many relationships, as might occur in a database system. A second case study exhibits how a pair of data structures representing the same object (a mapping in our example) can be a more efficient representation than either acting alone.

Many-Many Relationships and the Multilist Structure

An example of a many-many relationship between students and courses is suggested in Fig. 4.24. It is called a "many-many" relationship because there can be many students taking each course and many courses taken by each student.

Fig. 4.24. An example relationship between students and courses.

From time to time the registrar may wish to insert or delete students from courses, to determine which students are taking a given course, or to know which courses a given student is taking. The simplest data structure with which these questions can be answered is obvious; just use the 2-dimensional array suggested by Fig. 4.24, where the value 1 (or true) replaces the X's and 0 (or false) replaces the blanks.

For example, to insert a student into a course, we need a mapping, MS, perhaps implemented as a hash table, that translates student names into array indices, and another, MC, that translates course names into array indices. Then, to insert student s into course c, we simply set

Enrollment[MS(s ), MC(c)] := 1.

Deletions are performed by setting this element to 0. To find the courses taken by the student with name s, run across the row MS(s), and similarly run down the column MC(c) find the students in course c.

Why might we wish to look further for a more appropriate data structure? Consider a large university with perhaps 1000 courses and 20,000 students, taking an average of three courses each. The array suggested by Fig. 4.24 would have 20,000,000 elements, of which 60,000, or 0.3%, would be 1. † Such an array, which is called sparse to indicate that almost all its elements are zero, can be represented in far less space if we simply list the nonzero entries. Moreover, we can spend a great deal of time scanning a column of 20,000 entries searching for, on the average, 60 that are nonzero; row scans take considerable time as well.

One way to do better is to begin by phrasing the problem as one of maintaining a collection of sets. Two of these sets are S and C, the sets of all students and all courses. Each element of S is actually a record of a type such as

type
studenttype = record
id: integer;
name: array[l..30] of char;
end

and we would invent a similar record type for courses. To implement the structure we have in mind, we need a third set of elements, E, standing for enrollments. The elements of E each represent one of the boxes in the array of Fig. 4.24 in which there is an X. The E elements are records of a fixed type. As of now, we don't know any fields that belong in these records‡, although we shall soon learn of them. For the moment, let us simply postulate that there is an enrollment record for each X-entry in the array and that enrollment records are distinguishable from one another somehow.

We also need sets that represent the answers to the crucial questions: given a student or course, what are the related courses or students, respectively. It would be nice if for each student s there were a set Cs of all the courses s was taking and conversely, a set Sc of the students taking course c.

Such sets would be hard to implement because there would be no limit on the number of sets any element could be in, forcing us to complicate student and course records. We could instead let Sc and Cs be sets of pointers to course and student records, rather than the records themselves, but there is a method that saves a significant fraction of the space and allows equally fast answers to questions about students and courses.

Let us make each set Cs be the set of enrollment records corresponding to student s and some course c. That is, if we think of an enrollment as a pair (s, c), then

Cs = {(s, c) | s is taking course c}

We can similarly define

Sc = {(s, c) | s is taking course c}

Note the only difference in the meaning of the two set formers above is that in the first case s is constant and in the second c is. For example, based on Fig. 4.24, CAlex = {(Alex, CS101), (Alex, CS202)} and SCS101 = {(Alex, CS101), (Amy, CS101), (Ann, CS101)}.

Multilist Structures

In general, a multilist structure is any collection of cells some of which have more than one pointer and can therefore be on more than one list simultaneously. For each type of cell in a multilist structure, it is important to distinguish among pointer fields so we can follow one particular list and not get confused about which of several pointers in a particular cell we are following.

As a case in point, we can put one pointer field in each student and course record, pointing to the first enrollment record in the set Cs or Sc, respectively. Each enrollment record needs two pointer fields, one, which we shall call cnext, for the next enrollment on the list for the Cs set to which it belongs and the other, snext, for the Sc set to which it belongs.

It turns out that an enrollment record indicates explicitly neither the student nor the course that it represents. The information is implicit in the lists on which the enrollment record appears. Call the student and course records heading these lists the owners of the enrollment record. Thus, to tell what courses student s is taking, we must scan the enrollment records in Cs and find for each one the owner course record. We could do that by placing a pointer in each enrollment record to the owning course record, and we would also need a pointer to the owning student record.

While we might wish to use these pointers and thereby answer questions in the minimum possible time, we can save substantial space† at the cost of slowing down some computations, if we eliminate these pointers and instead place at the end of each Sc list a pointer to the owning course and at the end of each Cs list a pointer to the owning student. Thus, each student and course record becomes part of a ring that includes all the enrollment records it owns. These rings are depicted in Fig. 4.25 for the data of Fig. 4.24. Note that each enrollment record has its cnext pointer first and its snext pointer second.

Fig. 4.25. Multilist representation of Fig. 4.24.

Example 4.11. To answer a question like "which students are taking CS101," we find the course record for CS101. How we find such a record depends on how the set of courses is maintained. For example, there might be a hash table containing all such records, and we obtain the desired record by applying some hash function to "CS101".

We follow the pointer in the CS101 record to the first enrollment record in the ring for CS101. This is the second enrollment record from the left. We must then find the student owner of this enrollment record, which we do by following cnext pointers (the first pointer in enrollment records) until we reach a student record.† In this case, we reach the third enrollment record, then the student record for Alex; we now know that Alex is taking CS101.

Now we must find the next student in CS101. We do so by following the snext pointer (second pointer) in the second enrollment record, which leads to the fifth enrollment record. The cnext pointer in that record leads us directly to the owner, Amy, so Amy is in CS101. Finally, we follow the snext pointer in the fifth enrollment record to the eighth enrollment record. The ring of cnext pointers from that record leads to the ninth enrollment record, then to the student record for Ann, so Ann is in CS101. The snext pointer in the eighth enrollment record leads back to CS101, so we are done.

Abstractly, we can express the operation in Example 4.11 above as

for each enrollment record in the set for CS101 do begin
s := the student owner of the enrollment record;
print(s)
end

The above assignment to s can be elaborated as

f := e;
repeat
f := f�.cnext
until
f is a pointer to a student record;
s := studentname field of record pointed to by f;

where e is a pointer to the first enrollment record in the set for CS101.

In order to implement a structure like Fig. 4.25 in Pascal, we need to have only one record type, with variants to cover the cases of student, course, and enrollment records. Pascal forces this arrangement on us since the fields cnext and snext each have the ability to point to different record types. This arrangement, however, does solve one of our other problems. It is now easy to tell what type of record we have reached as we travel around a ring. Figure 4.26 shows a possible declaration of the records and a procedure that prints the students taking a particular class.

Dual Data Structures for Efficiency

Often, a seemingly simple representation problem for a set or mapping presents a difficult problem of data structure choice. Picking one data structure for the set makes certain operations easy, but others take too much time, and it seems that there is no one data structure that makes all the operations easy. In that case, the solution often turns out to be the use of two or more different structures for the same set or mapping.

Suppose we wish to maintain a "tennis ladder," in which each player is on a unique "rung." New players are added to the bottom, that is, the highest- numbered rung. A player can challenge the player on the rung above, and if the player below wins the match, they trade rungs. We can represent this situation as an abstract data type, where the underlying model is a mapping from names (character strings) to rungs (integers 1, 2, . . . ). The three operations we perform are

  1. ADD(name) adds the named person to the ladder at the highest-numbered rung.

  2. CHALLENGE(name) is a function that returns the name of the person on rung i- 1 if the named person is on rung i, i > 1.

  3. EXCHANGE(i) swaps the names of the players on rungs i and i-l, i > l.

    type
       stype = array[l..20] of char;
       ctype = array[1..5] of char;
       recordkinds = (student, course, enrollment);
       recordtype = record
        case kind: recordkinds of
            student: (studentname: stype;
               firstcourse: � recordtype);
            course: (coursename: ctype;
               firststudent: � recordtype);
            enrollment: (cnext, snext: � recordtype)
        end;

    procedure printstudents ( cname: ctype );
       var
        c, e, f: � recordtype;
       begin
        c := pointer to course record with c�.coursename = cname;
        { above depends on how course set is implemented }
        e := c �.firststudent;
        { e runs around the ring of enrollments pointed to by c }
        while e �.kind =
enrollment do begin
           f := e;
           repeat
        f := f�.cnext
           until
                    f�.kind = student;
           { f now points to student owner of enrollment e � }
        writeln (f�.studentname ) ;
        e := e�.snext
       end
    end

Fig. 4.26. Implementation of search through a multilist.

Notice that we have chosen to pass only the higher rung number to EXCHANGE, while the other two operations take a name as argument.

We might, for example, choose an array LADDER, where LADDER[i] is the name of the person on rung i. If we also keep a count of the number of players, adding a player to the first unoccupied rung can be done in some small constant number of steps.

EXCHANGE is also easy, as we simply swap two elements of the array. However, CHALLENGE(name) requires that we examine the entire array for the name, which takes O(n) time, if n is the number of players on the ladder.

On the other hand, we might consider a hash table to represent the mapping from names to rungs. On the assumption that we can keep the number of buckets roughly proportional to the number of players, ADD takes O(1) time, on the average. Challenging takes O(1) time on the average to look up the given name, but O(n) time to find the name on the next lower-numbered rung, since the entire hash table may have to be searched. Exchanging requires O(n) time to find the players on rungs i and i - 1.

Suppose, however, that we combine the two structures. The cells of the hash table will contain pairs consisting of a name and a rung, while the array will have in LADDER[i] a pointer to the cell for the player on rung i as suggested in Fig. 4.27.

Fig. 4.27. Combined structure for high performance.

We can add a name by inserting into the hash table in O(1) time on the average, and also placing a pointer to the newly created cell into the array LADDER at a position marked by the cursor nextrung in Fig. 4.27. To challenge, we look up the name in the hash table, taking O(1) time on the average, get the rung i for the given player, and follow the pointer in LADDER[i-1] to the cell of the hash table for the player to be challenged. Consulting LADDER[i - 1] takes constant time in the worst case, and the lookup in the hash table takes O(1) time on the average, so CHALLENGE is O(1) in the average case.

EXCHANGE(i) takes O(1) time to find the cells for the players on rungs i and i-l, swap the rung numbers in those cells, and swap the pointers to the two cells in LADDER. Thus EXCHANGE requires constant time in even the worst case.

Exercises

4.1 If A = {1, 2, 3} and B = {3, 4, 5}, what are the results of
  1. UNION(A,B, C)

  2. INTERSECTION(A, B, C)

  3. DIFFERENCE(A, B, C)

  4. MEMBER(1, A)

  5. INSERT(1, A)

  6. DELETE(1, A)

  7. MIN(A)?
*4.2 | Write a procedure in terms of the basic set operations to print all the elements of a (finite) set. You may assume that a procedure to print an object of the element type is available. You must not destroy the set you are printing. What data structures would be most suitable for implementing sets in this case?
4.3 The bit-vector implementation of sets can be used whenever the "universal set" can be translated into the integers 1 through N. Describe how this translation would be done if the universal set were
  1. the integers 0, 1, . . . , 99

  2. the integers n through m for any n � m

  3. the integers n, n +2, n +4, . . . , n + 2k for any n and k

  4. the characters 'a', 'b', . . . , 'z'

  5. arrays of two characters, each chosen from 'a' through 'z'.
4.4 | Write MAKENULL, UNION, INTERSECTION, MEMBER, MIN, INSERT, and DELETE procedures for sets represented by lists using the abstract operations for the sorted list ADT. Note that Fig. 4.5 is a procedure for INTERSECTION using a specific implementation of the list ADT.
4.5 Repeat Exercise 4.4 for the following set implementations:
  1. open hash table (use abstract list operations within buckets).

  2. closed hash table with linear resolution of collisions.

  3. unsorted list (use abstract list operations).

  4. fixed length array and pointer to the last position used.
4.6 | For each of the operations and each of the implementations in Exercises 4.4 and 4.5, give the order of magnitude for the running time of the operations on sets of size n.
4.7 Suppose we are hashing integers with a 7-bucket hash table using the hash function h(i) = i mod 7.
  1. Show the resulting open hash table if the perfect cubes 1, 8, 27, 64, 125, 216, 343 are inserted.

  2. Repeat part (a) using a closed hash table with linear resolution of collisions.
4.8 | Suppose we are using a closed hash table with 5 buckets and the hashing function h(i) = i mod 5. Show the hash table that results with linear resolution of collisions if the sequence 23, 48, 35, 4, 10 is inserted into an initially empty table.
4.9 | Implement the mapping ADT operations using open and closed hash tables.
4.10 | To improve the speed of operations, we may wish to replace an open hash table with B1 buckets holding many more than B1 elements by another hash table with B2 buckets. Write a procedure to construct the new hash table from the old, using the list ADT operations to process each bucket.
4.11 In Section 4.8 we discussed "random" hash functions, where hi(x), the bucket to be tried after i collisions, is (h (x) + di) mod B for some sequence d1, d2, . . . ,dB-1. We also suggested that one way to compute a suitable sequence d 1, d2, . . . , dB-1 was to pick a constant k, an arbitrary d1 > 0, and let

where i > 1, B is a power of 2, and � stands for the bitwise modulo 2 sum. If B = 16, find those values of k for which the sequence d1, d2, . . . , d15 includes all the integers 1, 2, . . . , 15.
4.12
  1. Show the partially ordered tree that results if the integers 5, 6, 4, 9, 3, 1, 7 are inserted into an empty tree.

  2. What is the result of three successive DELETEMIN operations on the tree from (a)?
4.13 Suppose we represent the set of courses by
  1. a linked list

  2. a hash table

  3. a binary search tree

Modify the declarations in Fig. 4.26 for each of these structures.
4.14 | Modify the data structure of Fig. 4.26 to give each enrollment record a direct pointer to its student and course owners. Rewrite the procedure printstudents of Fig. 4.26 to take advantage of this structure.
4.15 Assuming 20,000 students, 1,000 courses, and each student in an average of three courses, compare the data structure of Fig. 4.26 with its modification suggested in Exercise 4.14 as to
  1. the amount of space required

  2. the average time to execute printstudents

  3. the average time to execute the analogous procedure that prints the courses taken by a given student.
4.16 | Assuming the data structure of Fig. 4.26, given course record c and student record s, write procedures to insert and to delete the fact that s is taking c.
4.17 | What, if anything, is the difference between the data structure of Exercise 4.14 and the structure in which sets Cs and Sc are represented by lists of pointers to course and student records, respectively?
4.18 | Employees of a certain company are represented in the company data base by their name (assumed unique), employee number, and social security number. Suggest a data structure that lets us, given one representation of an employee, find the other two representations of the same individual. How fast, on the average, can you make each such operation?

Bibliographic Notes

Knuth [1973] is a good source for additional information on hashing. Hashing was developed in the mid-to-late 1950's, and Peterson [1957] is a fundamental early paper on the subject. Morris [1968] and Maurer and Lewis [1975] are good surveys of hashing.

The multilist is a central data structure of the network-based database systems proposed in DBTG [1971]. Ullman [1982] provides additional information on database applications of structures such as these.

The heap implementation of partially ordered trees is based on an idea in Williams [1964]. Priority queues are discussed further in Knuth [1973].

Reingold [1972] discusses the computational complexity of basic set operations. Set-based data flow analysis techniques are treated in detail in Cocke and Allen [1976] and Aho and Ullman [1977].

†Sometimes the term multiset is used for a "set with repetitions." That is, the multiset {1, 4, 1} has 1 (twice) and 4 (once) as members. A multiset is not a list any more than a set is, and thus this multiset could also have been written {4, 1, 1} or {1, 1, 4}.

†Changes to the data structure can be made that will speed up open hashing and allow closed hashing to handle larger sets. We shall describe these techniques after giving the basic methods.

†If the type of members of the dictionary does not suggest an appropriate value for empty, let each bucket have an extra one-bit field to tell whether or not it is empty.

†Note that A2 and A20 do not necessarily hash to the same bucket, but A23 and A41, for example, must do so.

†If strings can be extremely long, this addition will have to be performed modulo some constant c. For example, c could be one more than the largest integer obtainable from a single block.

†If this were a real database system, the array would be kept in secondary storage. However, this data structure would waste a great deal of space.

†Note that there are probably many more enrollment records than course or student records, so shrinking enrollment records shrinks the total space requirement almost as much.

‡It would in practice be useful to place fields like grade or status (credit or audit) in enrollment records, but our original problem statement does not require this.

†We must have some way of identifying the type of records; we shall discuss a way to do this momentarily.

†We assume this and is a conditional and.

Table of Contents Go to Chapter 5


Data Structures and Algorithms: CHAPTER 5: Advanced Set Representation Methods

 Next Chapter Return to Table of Contents Previous Chapter

Advanced Set Representation Methods

This chapter introduces data structures for sets that permit more efficient implementation of common collections of set operations than those of the previous chapter. These structures, however, are more complex and are often only appropriate for large sets. All are based on various kinds of trees, such as binary search trees, tries, and balanced trees.

5.1 Binary Search Trees

We shall begin with binary search trees, a basic data structure for representing sets whose elements are ordered by some linear order. We shall, as usual, denote that order by <. This structure is useful when we have a set of elements from a universe so large that it is impractical to use the elements of the set themselves as indices into arrays. An example of such a universe would be the set of possible identifiers in a Pascal program. A binary search tree can support the set operations INSERT, DELETE, MEMBER, and MIN, taking O(logn) steps per operation on the average for a set of n elements.

A binary search tree is a binary tree in which the nodes are labeled with elements of a set. The important property of a binary search tree is that all elements stored in the left subtree of any node x are all less than the element stored at x, and all elements stored in the right subtree of x are greater than the element stored at x. This condition, called the binary search tree property, holds for every node of a binary search tree, including the root.

Figure 5.1 shows two binary search trees representing the same set of integers. Note the interesting property that if we list the nodes of a binary search tree in inorder, then the elements stored at those nodes are listed in sorted order.

Suppose a binary search tree is used to represent a set. The binary search tree property makes testing for membership in the set simple. To determine whether x is a member of the set, first compare x with the element r at the root of the tree. If x = r we are done and the answer to the membership query is "true." If x < r, then by the binary search tree property, x can only

Fig. 5.1. Two binary search trees.

be a descendant of the left child of the root, if x is present at all.† Similarly, if x > r, then x could only be at a descendant of the right child of the root.

We shall write a simple recursive function MEMBER(x, A) to implement this membership test. We assume the elements of the set are of an unspecified type that will be called elementtype. For convenience, we assume elementtype is a type for which < and = are defined. If not, we must define functions LT(a, b) and EQ(a, b), where a and b are of type elementtype, such that LT(a, b) is true if and only if a is "less than" b, and EQ(a, b) is true if and only if a and b are the same.

The type for nodes consists of an element and two pointers to other nodes:

    type
        nodetype = record
           element: elementtype;
           leftchild, rightchild: � nodetype
        end;

Then we can define the type SET as a pointer to a node, which we take to be the root of the binary search tree representing the set. That is:

    type
        SET = � nodetype;

Now we can specify fully the function MEMBER, in Fig. 5.2. Notice that since SET and "pointer to nodetype" are synonymous, MEMBER can call itself on subtrees as if those subtrees represented sets. In effect, the set can be subdivided into the subset of members less than x and the subset of members greater than x.

    function MEMBER ( x: elementtype; A: SET ): boolean,
          { returns true if x is in A, false otherwise }
          begin
         if A = nil then
             return (false) { x is never in Ø }
         else if x = A �.element then
             return (true)
         else if x < A �.element then
             return (MEMBER(x, A �.leftchild ) )
         else { x > A �.element }
             return (MEMBER(x, A �.rightchild ) )
    end; { MEMBER }

Fig. 5.2. Testing membership in a binary search tree.

The procedure INSERT(x, A), which adds element x to set A, is also easy to write. The first action INSERT must do is test whether A = nil, that is, whether the set is empty. If so, we create a new node to hold x and make A point to it. If the set is not empty, we search for x more or less as MEMBER does, but when we find a nil pointer during our search, we replace it by a pointer to a new node holding x. Then x will be in the right place, namely, the place where the function MEMBER will find it. The code for INSERT is shown in Fig. 5.3.

Deletion presents some problems. First, we must locate the element x to be deleted in the tree. If x is at a leaf, we can delete that leaf and be done. However, x may be at an interior node inode, and if we simply deleted inode, we would disconnect the tree.

If inode has only one child, as the node numbered 14 in Fig. 5.1(b), we can replace inode by that child, and we shall be left with the appropriate binary search tree. If inode has two children, as the node numbered 10 in Fig. 5.1(a), then we must find the lowest-valued element among the descendants of the right child.† For example, in the case the element 10 is deleted from Fig. 5.1(a), we must replace it by 12, the minimum-valued descendant of

    procedure INSERT ( x: elementtype; var A: SET );
       { add x to set A }
       begin
           if A = nil then begin
            new (A);
            A � .element: = x;
            A � .leftchild := nil;
            A � .rightchild: = nil
           end
           else if x < A �.element
then
            INSERT(x, A �.leftchild)
           else if x > A �.element
then
            INSERT (x, A �.rightchild)
           { if x = A �.element, we do
nothing; x is already in the set }
       end; { INSERT }

Fig. 5.3. Inserting an element into a binary search tree.

the right child of 10.

To write DELETE, it is useful to have a function DELETEMIN(A) that removes the smallest element from a nonempty tree and returns the value of the element removed. The code for DELETEMIN is shown in Fig. 5.4. The code for DELETE uses DELETEMIN and is shown in Fig. 5.5.

    function DELETEMIN ( var A: SET ): elementtype;
        { returns and removes the smallest element from set A }
    begin
        if A �.leftchild = nil then
begin
         { A points to the smallest element }
         DELETEMIN := A �.element;
         A := A �.rightchild
             { replace the node pointed to by A by its right child }
        end
        else { the node pointed to by A has a left child }
         DELETEMIN := DELETEMIN(A �.leftchild)
    end; { DELETEMIN }

Fig. 5.4. Deleting the smallest element.

Example 5.1. Suppose we try to delete 10 from Fig. 5.1(a). Then, in the last statement of DELETE we call DELETEMIN with argument a pointer to node 14. That pointer is the rightchild field in the root. That call results in another call to DELETEMIN. The argument is then a pointer to node 12; this pointer is found in the leftchild field of node 14. We find that 12 has no

    procedure DELETE ( x: elementtype; var A: SET );
        { remove x from set A }
    begin
        if A < > nil then
        if x < A �.element
then
             DELETE(x, A �.leftchild)
        else if x > A �.element then
             DELETE(x, A �.rightchild)
        { if we reach here, x is at the node pointed to by A }
        else if (A �.leftchild =
nil) and (A �.rightchild =
nil) then
             A := nil { delete the leaf holding x }
        else if A �.leftchild =
nil then
             A := A �.rightchild
        else if A �.rightchild =
nil then
             A := A �.leftchild
        else { both children are present }
             A �.element :=
DELETEMIN(A �.rightchild)
    end; { DELETE }

Fig. 5.5. Deletion from a binary search tree.

left child, so we return element 12 and make the left child for 14 be the right child of 12, which happens to be nil. Then DELETE takes the value 12 returned by DELETEMIN and replaces 10 by it. The resulting tree is shown in Fig. 5.6.

Fig. 5.6. Tree of Fig.5.1 after deleting 1-0.

5.2 Time Analysis of Binary Search Tree Operations

In this section we analyze the average behavior of various binary search tree operations. We show that if we insert n random elements into an initially empty binary search tree, then the average path length from the root to a leaf is O(logn). Testing for membership, therefore, takes O(logn) time.

It is easy to see that if a binary tree of n nodes is complete (all nodes, except those at the lowest level have two children), then no path has more than 1+logn nodes.† Thus, each of the procedures MEMBER, INSERT, DELETE, and DELETEMIN takes O(logn) time. To see this, observe that they all take a constant amount of time at a node, then may call themselves recursively on at most one child. Therefore, the sequence of nodes at which calls are made forms a path from the root. Since that path is O(logn) in length, the total time spent following the path is O(logn).

However, when we insert n elements in a "random" order, they do not necessarily arrange themselves into a complete binary tree. For example, if we happen to insert smallest first, in sorted order, then the resulting tree is a chain of n nodes, in which each node except the lowest has a right child but no left child. In this case, it is easy to show that, as it takes O(i) steps to insert the ith element, and [image], the whole process of n insertions takes O(n2) steps, or O(n) steps per operation.

We must determine whether the "average" binary search tree of n nodes is closer to the complete tree in structure than to the chain, that is, whether the average time per operation on a "random" tree takes O(logn) steps, O(n) steps, or something in between. As we cannot know the true frequency of insertions and deletions, or whether deleted elements have some special property (e.g., do we always delete the minimum?), we can only analyze the average path length of "random" trees if we make some assumptions. The particular assumptions we make are that trees are formed by insertions only, and all orders of the n inserted elements are equally likely.

Under these fairly natural assumptions, we can calculate P(n), the average number of nodes on the path from the root to some node (not necessarily a leaf). We assume the tree was formed by inserting n random elements into an initially empty tree. Clearly P(0) = 0 and P(1) = 1. Suppose we have a list of n�2 elements to insert into an empty tree. The first element on the list, call it a, is equally likely to be first, second, or nth in the sorted order. Suppose that i elements on the list are less than a, so n-i-1 are greater than a. When we build the tree, a will appear at the root, the i smaller elements will be descendants of the left child of the root, and the remaining n-i-1 will be descendants of the right child. This tree is sketched in Fig. 5.7.

As all orders for the i small elements and for the n-i-1 large elements are equally likely, we expect that the left and right subtrees of the root will

Fig. 5.7. Binary search tree.

have average path lengths P(i) and P(n-i-1), respectively. Since these elements are reached through the root of the complete tree, we must add 1 to the number of nodes on every path. Thus P(n) can be calculated by averaging, for all i between 0 and n-1, the sum

The first term is the average path length in the left subtree, weighted by its size. The second term is the analogous quantity for the right subtree, and the 1/n term represents the contribution of the root. By averaging the above sum for all i between 1 and n, we obtain the recurrence

The first part of the summation (5.1), [image], can be made identical to the second part [image] if we substitute i for n-i-1 in the second part. Also, the term for i=0 in the summation [image] is zero, so we can begin the summation at 1. Thus (5.1) can be written

We shall show by induction on n, starting at n=1, that P(n) � 1 + 4logn. Surely this statement is true for n = 1, since P(1) = 1. Suppose it is true for all i < n. Then by (5.2)

The last step is justified, since [image], and therefore, the last term of the second line is at most 1. We shall divide the terms in the summation of (5.3) into two parts, those for i � [n/2]-1, which do not exceed ilog(n/2), and those for i> [n/2]-1, which do not exceed ilogn. Thus (5.3) can be rewritten

Whether n is even or odd, one can show that the first sum of (5.4) does not exceed (n2/8)log(n/2), which is (n2/8)logn - (n2/8), and the second sum does not exceed (3n2/8)logn. Thus, we can rewrite (5.4) as

as we wished to prove. This step completes the induction and shows that the average time to follow a path from the root to a random node of a binary search tree constructed by random insertions is O(logn), that is to within a constant factor as good as if the tree were complete. A more careful analysis shows that the constant 4 above is really about 1.4.

We can conclude from the above that the time of membership testing for a random member of the set takes O(logn) time. A similar analysis shows that if we include in our average path length only those nodes that are missing both children, or only those missing left children, then the average path length still obeys an equation similar to (5.1), and is therefore O(logn). We may then conclude that testing membership of a random element not in the set, inserting a random new element, and deleting a random element also all take O(logn) time on the average.

Evaluation of Binary Search Tree Performance

Hash table implementations of dictionaries require constant time per operation on the average. Although this performance is better than that for a binary search tree, a hash table requires O(n) steps for the MIN operation, so if MIN is used frequently, the binary search tree will be the better choice; if MIN is not needed, we would probably prefer the hash table.

The binary search tree should also be compared with the partially ordered tree used for priority queues in Chapter 4. A partially ordered tree with n elements requires only O(logn) steps for each INSERT and DELETEMIN operation not only on the average, but also in the worst case. Moreover, the actual constant of proportionality in front of the logn factor will be smaller for a partially ordered tree than for a binary search tree. However, the binary search tree permits general DELETE and MIN operations, as well as the combination DELETEMIN, while the partially ordered tree permits only the latter. Moreover, MEMBER requires O(n) steps on a partially ordered tree but only O(logn) steps on a binary search tree. Thus, while the partially ordered tree is well suited to implementing priority queues, it cannot do as efficiently any of the additional operations that the binary search tree can do.

5.3 Tries

In this section we shall present a special structure for representing sets of character strings. The same method works for representing data types that are strings of objects of any type, such as strings of integers. This structure is known as the trie, derived from the middle letters of the word "retrieval." † By way of introduction, consider the following use of a set of character strings.

Example 5.2. As indicated in Chapter 1, one way to implement a spelling checker is to read a text file, break it into words (character strings separated by blanks and new lines), and find those words not in a standard dictionary of words in common use. Words in the text but not in the dictionary are printed out as possible misspellings. In Fig. 5.8 we see a sketch of one possible program spell. It makes use of a procedure getword(x, f) that sets x to be the next word in text file f; variable x is of a type called wordtype, which we shall define later. The variable A is of type SET; the SET operations we shall need are INSERT, DELETE, MAKENULL, and PRINT. The PRINT operator prints the members of the set.

The trie structure supports these set operations when the elements of the set are words, i.e., character strings. It is appropriate when many words begin with the same sequences of letters, that is, when the number of distinct prefixes among all the words in the set is much less than the total length of all the words.

In a trie, each path from the root to a leaf corresponds to one word in the represented set. This way, the nodes of the trie correspond to the prefixes of words in the set. To avoid confusion between words like THE and THEN, let us add a special endmarker symbol, $, to the ends of all words, so no prefix of a word can be a word itself.

Example 5.3. In Fig. 5.9 we see a trie representing the set of words {THE, THEN, THIN, THIS, TIN, SIN, SING}. That is, the root corresponds to the empty string, and its two children correspond to the prefixes T and S. The

    program spell ( input, output, dictionary );
       type
           wordtype = { to be defined };
           SET = { to be defined, using the trie structure };
    var
           A: SET; { holds input words not yet found in the dictionary }
           nextword: wordtype;
           dictionary: file of char;

    procedure getword ( var x: wordtype; f: file
of char );
        { a procedure to be defined that sets x
            to be the next word in file f}

    procedure INSERT ( x: wordtype; var S: SET );
        { to be defined }

    procedure DELETE (x: wordtype; var S: SET );
        { to be defined }

    procedure MAKENULL ( var S: SET );
        { to be defined }

    procedure PRINT ( var S: SET );
        { to be defined }

    begin
       MAKENULL(A);
       while not eof(input) do begin
           getword ( nextword, input);
           INSERT(nextword, A)
       end;
       while not eof (dictionary) do begin
           getword ( nextword, dictionary);
           DELETE(nextword, A)
       end;
       PRINT(A)
    end; { spell }

Fig. 5.8. Sketch of spelling checker.

leftmost leaf represents the word THE, the next leaf the word THEN, and so on.

We can make the following observations about the trie in Fig. 5.9.

  1. Each node has at most 27 children, one for each letter and $.

    2. Fig. 5.9 A Trie.

  2. Most nodes will have many fewer than 27 children.
  3. A leaf reached by an edge labeled $ cannot have any children, and may as well not be there.

Trie Nodes as ADT's

We can view a node of a trie as a mapping whose domain is {A, B, ... , Z, $} (or whatever alphabet we choose) and whose value set is the type "pointer to trie node." Moreover, the trie itself can be identified with its root, so the ADT's TRIE and TRIENODE have the same data type, although their operations are substantially different. On TRIENODE we need the following operations:

  1. procedure ASSIGN(node, c, p) that assigns value p (a pointer to a node) to character c in node node,
  2. function VALUEOF(node, c) that produces the value associated with character c in node,† and
  3. procedure GETNEW(node, c) to make the value of node for character c be a pointer to a new node.

Technically, we also need a procedure MAKENULL(node) to make node be the null mapping. One simple implementation of trie nodes is an array node of pointers to nodes, with the index set being {A, B, ..., Z, $}. That is, we define

    type
         chars = ('A', 'B', ... , 'Z', '$');
         TRIENODE = array[chars] of �;
TRIENODE;

If node is a trie node, node[c] is VALUEOF(node, c) for any c in the set chars. To avoid creating many leaves that are children corresponding to '$', we shall adopt the convention that node['$'] is either nil or a pointer to the node itself. In the former case, node has no child corresponding to '$', and in the latter case it is deemed to have such a child, although we never create it. Then we can write the procedures for trie nodes as in Fig. 5.10.

    procedure MAKENULL ( var node: TRIENODE );
        { makes node a leaf, i.e., a null mapping }
        var
        c: chars;
        begin
        for c: = 'A' to '$' do
             node [c] := nil
        end; { MAKENULL }

        procedure ASSIGN ( var node: TRIENODE; c: chars;
p: � TRIENODE );
        begin
            node [c] := p
        end; { ASSIGN }

        function VALUEOF ( var node: TRIENODE; c: chars ):
� TRIENODE;
        begin
            return (node [c])
        end; { VALUEOF }

        procedure GETNEW ( var node: TRIENODE; c: chars );
        begin
            new (node [c]);
            MAKENULL(node [c])
        end; { GETNEW }

Fig. 5.10 Operation in trie nodes.

Now let us define

    type
        TRIE = � TRIENODE;

We shall assume wordtype is an array of characters of some fixed length. The value of such an array will always be assumed to have at least one '$'; we take the end of the represented word to be the first '$', no matter what follows (presumably more '$' 's). On this assumption, we can write the procedure INSERT(x, words) to insert x into set words represented by a trie, as shown in Fig. 5.11. We leave the writing of MAKENULL, DELETE, and PRINT for tries represented as arrays for exercises.

    procedure INSERT ( x: wordtype; var words: TRIE );
        var
        i: integer; { counts positions in word x }
        t: TRIE; { used to point to trie nodes
            corresponding to prefixes of x }
        begin
        i : = 1;
        t := words;
        while x[i] <> '$' do begin
            if VALUEOF(t�,
x[i]) = nil then
            { if current node has no child for character x[i],
                 create one }
            GETNEW(t�, x[i]);
            t := VALUEOF(t�,
x[i]);
             { proceed to the child of t for character x[i],
                 whether or not that child was just created }
            i := i+1 { move along the word x }
        end;
        { now we have reached the first '$' in x }
        ASSIGN(t�, '$', t)
            { make loop for '$' to represent a leaf }
        end; {INSERT}

Fig. 5.11 The procedure INSERT.

A List Representation for Trie Nodes

The array representation of trie nodes takes a collection of words, having among them p different prefixes, and represents them with 27p bytes of storage. That amount of space could far exceed the total length of the words in the set. However, there is another implementation of tries that may save space. Recall that each trie node is a mapping, as discussed in Section 2.6. In principle, any implementation of mappings would do. Yet in practice, we want a representation suitable for mappings with a small domain and for mappings defined for comparatively few members of that domain. The linked list representation for mappings satisfies these requirements nicely. We may represent the mapping that is a trie node by a linked list of the characters for which the associated value is not the nil pointer. That is, a trie node is a linked list of cells of the type

    type
        celltype = record
        domain: chars;
        value: � celltype;
             { pointer to first cell on list for the child node }
        next: � celltype
             { pointer to next cell on the list }
        end;

We shall leave the procedures ASSIGN, VALUEOF, MAKENULL, and GETNEW for this implementation of trie nodes as exercises. After writing these procedures, the INSERT operations on tries, in Fig. 5.11, and the other operations on tries that we left as exercises, should work correctly.

Evaluation of the Trie Data Structure

Let us compare the time and space needed to represent n words with a total of p different prefixes and a total length of l using a hash table and a trie. In what follows, we shall assume that pointers require four bytes. Perhaps the most space-efficient way to store words and still support INSERT and DELETE efficiently is in a hash table. If the words are of varying length, the cells of the buckets should not contain the words themselves; rather, the cells consist of two pointers, one to link the cells of the bucket and the other pointing to the beginning of a word belonging in the bucket.

The words themselves are stored in a large character array, and the end of each word is indicated by an endmarker character such as '$'. For example, the words THE, THEN, and THIN could be stored as

THE$THEN$THIN$ . . .

The pointers for the three words are cursors to positions 1, 5, and 10 of the array. The amount of space used in the buckets and character array is

  1. 8n bytes for the cells of the buckets, there being one cell for each of the n words, and a cell has two pointers or 8 bytes,
  2. l + n bytes for the character array to store the n words of total length l and their endmarkers.

The total space is thus 9n + l bytes plus whatever amount is used for the bucket headers.

In comparison, a trie with nodes implemented by linked lists requires p + n cells, one cell for each prefix and one cell for the end of each word. Each trie cell has a character and two pointers, and needs nine bytes, for a total space of 9n + 9p. If l plus the space for the bucket headers exceeds 9p, the trie uses less space. However, for applications such as storing a dictionary where l/p is typically less than 3, the hash table would use less space.

In favor of the trie, however, let us note that we can travel down a trie, and thus perform operations INSERT, DELETE, and MEMBER in time proportional to the length of the word involved. A hash function to be truly "random" should involve each character of the word being hashed. It is fair, therefore, to state that computing the hash function takes roughly as much time as performing an operation like MEMBER on the trie. Of course the time spent computing the hash function does not include the time spent resolving collisions or performing the insertion, deletion, or membership test on the hash table, so we can expect tries to be considerably faster than hash tables for dictionaries whose elements are character strings.

Another advantage to the trie is that it supports the MIN operation efficiently, while hash tables do not. Further, in the hash table organization described above, we cannot easily reuse the space in the character array when a word is deleted (but see Chapter 12 for methods of handling such a problem).

5.4 Balanced Tree Implementations of Sets

In Sections 5.1 and 5.2 we saw how sets could be implemented by binary search trees, and we saw that operations like INSERT could be performed in time proportional to the average depth of nodes in that tree. Further, we discovered that this average depth is O(logn) for a "random" tree of n nodes. However, some sequences of insertions and deletions can produce binary search trees whose average depth is proportional to n. This suggests that we might try to rearrange the tree after each insertion and deletion so that it is always complete; then the time for INSERT and similar operations would always be O(logn).

In Fig. 5.12(a) we see a tree of six nodes that becomes the complete tree of seven nodes in Fig. 5.12(b), when element 1 is inserted. Every element in Fig. 5.12(a), however, has a different parent in Fig. 5.12(b), so we must take n steps to insert 1 into a tree like Fig. 5.12(a), if we wish to keep the tree as balanced as possible. It is thus unlikely that simply insisting that the binary search tree be complete will lead to an implementation of a dictionary, priority queue, or other ADT that includes INSERT among its operations, in O(logn) time.

There are several other approaches that do yield worst case O(logn) time per operation for dictionaries and priority queues, and we shall consider one of them, called a "2-3 tree," in detail. A 2-3 tree is a tree with the following two properties.

  1. Each interior node has two or three children.
  2. Each path from the root to a leaf has the same length.

We shall also consider a tree with zero nodes or one node as special cases of a 2-3 tree.

We represent sets of elements that are ordered by some linear order <, as follows. Elements are placed at the leaves; if element a is to the left of

Fig. 5.12. Complete trees.

element b, then a < b must hold. We shall assume that the "<" ordering of elements is based on one field of a record that forms the element type; this field is called the key. For example, elements might represent people and certain information about people, and in that case, the key field might be "social security number."

At each interior node we record the key of the smallest element that is a descendant of the second child and, if there is a third child, we record the key of the smallest element descending from that child as well.† Figure 5.13 is an example of a 2-3 tree. In that and subsequent examples, we shall identify an element with its key field, so the order of elements becomes obvious.

Fig. 5.13 A 2-3 tree.

Observe that a 2-3 tree of k levels has between 2k-1 and 3k-1 leaves. Put another way, a 2-3 tree representing a set of n elements requires at least 1 + log3n levels and no more than 1 + log2n levels. Thus, path lengths in the tree are O(logn).

We can test membership of a record with key x in a set represented by a 2-3 tree in O(logn) time by simply moving down the tree, using the values of the elements recorded at the interior nodes to guide our path. At a node node, compare x with the value y that represents the smallest element descending from the second child of node. (Recall we are treating elements as if they consisted solely of a key field.) If x < y, move to the first child of node. If x�y, and node has only two children, move to the second child of node. If node has three children and x�y, compare x with z, the second value recorded at node, the value that indicates the smallest descendant of the third child of node. If x<z, go to the second child, and if x�z, go to the third child. In this manner, we find ourselves at a leaf eventually, and x is in the represented set if and only if x is at the leaf. Evidently, if during this process we find x=y or x=z, we can stop immediately. However, we stated the algorithm as we did because in some cases we shall wish to find the leaf with x as well as to verify its existence.

Insertion into a 2-3 Tree

To insert a new element x into a 2-3 tree, we proceed at first as if we were testing membership of x in the set. However, at the level just above the leaves, we shall be at a node node whose children, we discover, do not include x. If node has only two children, we simply make x the third child of node, placing the children in the proper order. We then adjust the two numbers at node to reflect the new situation.

For example, if we insert 18 into Fig. 5.13, we wind up with node equal to the rightmost node at the middle level. We place 18 among the children of node, whose proper order is 16, 18, 19. The two values recorded at node become 18 and 19, the elements at the second and third children. The result is shown in Fig. 5.14.

Suppose, however, that x is the fourth, rather than the third child of node. We cannot have a node with four children in a 2-3 tree, so we split node into two nodes, which we call node and node'. The two smallest elements among the four children of node stay with node, while the two larger elements become children of node'. Now, we must insert node' among the children of p, the parent of node. This part of the insertion is analogous to the insertion of a leaf as a child of node. That is, if p had two children, we make node' the third and place it immediately to the right of node. If p had three children before node' was created, we split p into p and p', giving p the two leftmost children and p' the remaining two, and then we insert p' among the children of p's parent, recursively.

One special case occurs when we wind up splitting the root. In that case we create a new root, whose two children are the two nodes into which the

Fig. 5.14 2-3 tree with 18 inserted.

old root was split. This is how the number of levels in a 2-3 tree increases.

Example 5.4. Suppose we insert 10 into the tree of Fig. 5.14. The intended parent of 10 already has children 7, 8, and 12, so we split it into two nodes. The first of these has children 7 and 8; the second has 10 and 12. The result is shown in Fig. 5.15(a). We must now insert the new node with children 10 and 12 in its proper place as a child of the root of Fig. 5.15(a). Doing so gives the root four children, so we split it, and create a new root, as shown in Fig. 5.15(b). The details of how information regarding smallest elements of subtrees is carried up the tree will be given when we develop the program for the command INSERT.

Deletion in a 2-3 tree

When we delete a leaf, we may leave its parent node with only one child. If node is the root, delete node and let its lone child be the new root. Otherwise, let p be the parent of node. If p has another child, adjacent to node on either the right or the left, and that child of p has three children, we can transfer the proper one of those three to node. Then node has two children, and we are done.

If the children of p adjacent to node have only two children, transfer the lone child of node to an adjacent sibling of node, and delete node. Should p now have only one child, repeat all the above, recursively, with p in place of node.

Example 5.5. Let us begin with the tree of Fig. 5.15(b). If 10 is deleted, its parent has only one child. But the grandparent has another child that has three children, 16, 18, and 19. This node is to the right of the deficient node, so we pass the deficient node the smallest element, 16, leaving the 2-3 tree in Fig. 5.16(a).

Fig. 5.15 Insertion of 10 into the tree of Fig.5.14.

Next suppose we delete 7 from the tree of Fig. 5.16(a). Its parent now has only one child, 8, and the grandparent has no child with three children. We therefore make 8 be a sibling of 2 and 5, leaving the tree of Fig. 5.16(b). Now the node starred in Fig. 5.16(b) has only one child, and its parent has no other child with three children. Thus we delete the starred node, making its child be a child of the sibling of the starred node. Now the root has only one child, and we delete it, leaving the tree of Fig. 5.16(c).

Observe in the above examples, the frequent manipulation of the values at interior nodes. While we can always calculate these values by walking the tree, it can be done as we manipulate the tree itself, provided we remember the smallest value among the descendants of each node along the path from the root to the deleted leaf. This information can be computed by a recursive deletion algorithm, with the call at each node being passed, from above, the correct quantity (or the value "minus infinity" if we are along the leftmost path). The details require careful case analysis and will be sketched later when we consider the program for the DELETE operation.

Data Types for 2-3 Trees

Let us restrict ourselves to representing by 2-3 trees sets of elements whose keys are real numbers. The nature of other fields that go with the field key, to make up a record of type elementtype, we shall leave unspecified, as it has no bearing on what follows.

Fig. 5.16 Deletion in a 2-3 tree.

In Pascal, the parents of leaves must be records of consisting of two reals (the key of the smallest elements in the second and third subtrees) and of three pointers to elements. The parents of these nodes are records consisting of two reals and of three pointers to parents of leaves. This progression continues indefinitely; each level in a 2-3 tree is of a different type from all other levels. Such a situation would make programming 2-3 tree operations in Pascal impossible, but fortunately, Pascal provides a mechanism, the variant record structure, that enables us to regard all 2-3 tree nodes as having the same type, even though some are elements, and some are records with pointers and reals.† We can define nodes as in Fig. 5.17. Then we declare a set, represented by a 2-3 tree, to be a pointer to the root as in Fig. 5.17.

Implementation of INSERT

The details of operations on 2-3 trees are quite involved, although the principles are simple. We shall therefore describe only one operation, insertion, in detail; the others, deletion and membership testing, are similar in spirit, and finding the minimum is a trivial search down the leftmost path. We shall write the insertion routine as main procedure, INSERT, which we call at the root, and a procedure insert 1, which gets called recursively down the tree.

    type
        elementtype = record
        key: real;
        {other fields as warranted}
        end;
        nodetypes = (leaf, interior);
        twothreenode = record
        case kind: nodetypes of
             leaf: (element: elementtype);
             interior: (firstchild, secondchild, thirdchild: � twothreenode;
              lowofsecond, lowofthird: real)
        end;
        SET = � twothreenode;

Fig. 5.17. Definition of a node in a 2-3 tree.

For convenience, we assume that a 2-3 tree is not a single node or empty. These two cases require a straightforward sequence of steps which the reader is invited to provide as an exercise.

procedure insert1 ( node: � twothreenode;
    x: elementtype; { x is to be inserted into the subtree of node }
    var pnew: � twothreenode; { pointer to new node
created to right of node }
    var low: real ); { smallest element in the subtree pointed to by pnew}

    begin
    pnew := nil;
    if node is a leaf then begin
         if x is not the element at node then begin
          create new node pointed to by pnew;
          put x at the new node;
          low := x.key
         end
    end
    else begin { node is an interior node }
         let w be the child of node to whose subtree x belongs;
         insert1(w, x, pback, lowback);
         if pback < > nil then begin
         insert pointer pback among the children of node just
             to the right of w;
         if node has four children then begin
             create new node pointed to by pnew;
             give the new node the third and fourth children of node;
             adjust lowofsecond and lowofthird in node and the new node;
                      set low to be the lowest key among the
                       children of the new node
         end
         end
        end
end; { insert 1 }

Fig. 5.18. Sketch of 2-3 tree insertion program.

We would like insert 1 to return both a pointer to a new node, if it must create one, and the key of the smallest element descended from that new node. As the mechanism in Pascal for creating such a function is awkward, we shall instead declare insert 1 to be a procedure that assigns values to parameters pnew and low in the case that it must "return" a new node. We sketch insert 1 in Fig. 5.18. The complete procedure is shown in Fig. 5.19; some comments in Fig. 5.18 have been omitted from Fig. 5.19 to save space.

procedure insert 1 ( node: � twothreenode;
x: elementtype;
    var pnew: � twothreenode; var
low: real );
    var
    pback: � twothreenode;
    lowback : real;
    child: 1..3; { indicates which child of node is followed
        in recursive call (cf. w in Fig. 5.18) }
    w: � twothreenode; { pointer to the child }

    begin
    pnew := nil;
    if node �.kind = leaf
then begin
         if node �.element.key < >
x .key then begin
          { create new leaf holding x and "return" this node }
          new (pnew, leaf);
          if (node �.element.key < x.key ) then
               { place x in new node to right of current node }
               begin pnew �.element
:= x; low := x .key end
          else begin { x belongs to left of element at current node }
               pnew �.element := node
�.element;
               node �.element : = x;
                  low: =pnew �.element.key
          end
         end
    end
    else begin { node is an interior node }
         { select the child of node that we must follow }
         if x.key < node �.lowofsecond then
          begin child := 1; w := node �.firstchild end
    else if (node �.thirdchild
= nil) or (x.key < node �.lowofthird) then begin
         { x is in second subtree }
         child := 2;
         w := node �.secondchild
    end
    else begin { x is in third subtree }
         child := 3;
         w := node �.thirdchild
    end;
    insert 1 ( w , x, pback, lowback ) ;
    if pback < > nil then
         { a new child of node must be inserted }
         if node �.thirdchild =
nil then
         { node had only two children, so insert new node in proper place }
         if child = 2 then begin
              node �.thirdchild :=
pback ;
              node �.lowofthird :=
lowback
         end
         else begin { child = 1 }
              node �.thirdchild :=
node �.secondchild ;
              node �.lowofthird : =
node �.lowofsecond ;
              node �.secondchild :=
pback ;
              node �.lowofsecond :=
lowback
         end
         else begin { node already had three children }
         new (pnew , interior );
         if child = 3 then begin
              { pback and third child become children of new node }
              pnew �.firstchild :=
node �.thirdchild ;
              pnew �.secondchild : =
pback ;
              pnew �.thirdchild :=
nil;
              pnew �.lowofsecond :=
lowback;
             { lowofthird is undefined for pnew}
              low := node �.lowofthird ;
              node �.thirdchild := nil
         end
         else begin { child � 2; move third
child of node to pnew}
              pnew �.secondchild :=
node �.thirdchild ;
              pnew �.lowofsecond :=
node �.lowofthird ;
              pnew �.thirdchild : =
nil;
              node �.thirdchild := nil
         end;
         if child = 2 then begin
              { pback becomes first child of pnew}
              pnew �.firstchild :=
pback ;
              low := lowback
         end;
         if child = 1 then begin
              { second child of node is moved to pnew;
                   pback becomes second child of node }
              pnew �.firstchild :=
node �.secondchild ;
              low := node �.lowofsecond ;
              node �.secondchild : =
pback ;
              node �.lowofsecond :=
lowback
         end
        end
    end
end; { insert 1 }

Fig. 5.19. The procedure insert 1.

Now we can write the procedure INSERT, which calls insert 1. If insert 1 "returns" a new node, then INSERT must create a new root. The code is shown in Fig. 5.20 on the assumption that the type SET is � twothreenode, i.e., a pointer to the root of a 2-3 tree whose leaves contain the members of the set.

procedure INSERT ( x: elementtype; var S: SET );
    var
    pback: � twothreenode; { pointer to new node
returned by insert 1 }
     lowback: real; { low value in subtree of pback }
    saveS: SET; { place to store a temporary copy of the pointer S }
    begin
    { checks for S being empty or a single node should occur here,
           and an appropriate insertion procedure should be included }
    insert 1(S, x, pback , lowback ) ;
    if pback < > nil then begin
         { create new root; its children are now pointed to by S and pback }
         saveS := S;
         new(S);
         S �.firstchild := saveS;
         S �.secondchild := pback;
         S �.lowofsecond := lowback;
         S �.thirdchild := nil;
    end
end; { INSERT }

Fig. 5.20. INSERT for sets represented by 2-3 trees.

Implementation of DELETE

We shall sketch a function delete 1 that takes a pointer to a node node and an element x, and deletes a leaf descended from node having value x, if there is one.† Function delete 1 returns true if after deletion node has only one child, and it returns false if node still has two or three children. A sketch of the code for delete 1 is shown in Fig. 5.21.

We leave the detailed code for function delete 1 for the reader. Another exercise is to write a procedure DELETE(S, x) that checks for the special cases that the set S consists only of a single leaf or is empty, and otherwise calls delete 1(S, x); if delete 1 returns true, the procedure removes the root (the node pointed to by S) and makes S point to its lone child.

function delete 1 ( node : � twothreenode;
x: elementtype ) : boolean;
      var
     onlyone: boolean; { to hold the value returned by a call to delete 1 }
      begin
     delete 1 := false;
     if the children of node are leaves then begin
     if x is among those leaves then begin
          remove x;
          shift children of node to the right of x one position left;
          if node now has one child then
        delete 1 := true
      end
    end
        else begin { node is at level two or higher }
          determine which child of node could have x as a descendant;
          onlyone := delete 1(w, x); ( w stands for node
�.firstchild,
             node �.secondchild, or node
�.thirdchild, as appropriate }
          if onlyone then begin ( fix children of node )
             if w is the first child of node then
               if y, the second child of node, has three children then
              make the first child of y be the second child of w
               else begin { y has two children }
                    make the child of w be the first child of y;
              remove w from among the children of node;
              if node now has one child then
                   delete 1 := true
               end;
             if w is the second child of node then
               if y, the first child of node, has three children then
              make the third child of y be the first child of w
           else { y has two children }
                if z, the third child of node, exists
              and has three children then
                    make first child of z be the second child of w
                  else begin { no other child of node has three children }
                  make the child of w be the third child of y;
                  remove w from among the children of node;
                  if node now has one child then
                     delete 1 := true
                  end;
               if w is the third child of node then
                 if y, the second child of node, has three children then
                make the third child of y be the first child of w
                 else begin { y has two children }
                make the child of w be the third child of y;
                remove w from among the children of node
                 end { note node surely has two children left in this case }
          end
          end
end; { delete 1 }

Fig. 5.21. Recursive deletion procedure.

5.5 Sets with the MERGE and FIND Operations

In certain problems we start with a collection of objects, each in a set by itself; we then combine sets in some order, and from time to time ask which set a particular object is in. These problems can be solved using the operations MERGE and FIND. The operation MERGE(A, B, C) makes C equal to the union of sets A and B, provided A and B are disjoint (have no member in common); MERGE is undefined if A and B are not disjoint. FIND(x) is a function that returns the set of which x is a member; in case x is in two or more sets, or in no set, FIND is not defined.

Example 5.6. An equivalence relation is a reflexive, symmetric, and transitive relation. That is, if � is an equivalence relation on set S, then for any (not necessarily distinct) members a, b, and c in S, the following properties hold:

  1. a � a (reflexivity)
  2. If a � b, then b � a (symmetry).
  3. If a � b and b � c, then a � c (transitivity).

The relation "is equal to" (=) is the paradigm equivalence relation on any set S. For a, b, and c in S, we have (1) a = a, (2) if a = b, then b = a, and (3) if a = b and b = c, then a = c. There are many other equivalence relations, however, and we shall shortly see several additional examples.

In general, whenever we partition a collection of objects into disjoint groups, the relation a � b if and only if a and b are in the same group is an equivalence relation. "Is equal to" is the special case where every element is in a group by itself.

More formally, if a set S has an equivalence relation defined on it, then the set S can be partitioned into disjoint subsets S1, S2, ..., called equivalence classes, whose union is S. Each subset Si consists of equivalent members of S. That is, a � b for all a and b in Si, and a ?? b if a and b are in different subsets. For example, the relation congruence modulo n † is an equivalence relation on the set of integers. To check that this is so, note that a-a = 0, which is a multiple of n (reflexivity); if a-b = dn, then b-a = (-d)n (symmetry); and if a-b = dn and b-c = en, then a-c = (d+e)n (transitivity). In the case of congruence modulo n there are n equivalence classes, which are the set of integers congruent to 0, the set of integers congruent to 1, ... , the set of integers congruent to n - 1.

The equivalence problem can be formulated in the following manner. We are given a set S and a sequence of statements of the form "a is equivalent to b." We are to process the statements in order in such a way that at any time we are able to determine in which equivalence class a given element belongs. For example, suppose S = {1, 2, ... , 7} and we are given the sequence of statements

1�2 5�6 3�4 1�4

to process. The following sequence of equivalence classes needs to be constructed, assuming that initially each element of S is in an equivalence class by itself.

1�2 {1,2} {3} {4} {5} {6} {7}

5�6 {1,2} {3} {4} {5,6} {7}

3�4 {1,2} {3,4} {5,6} {7}

1�4 {1,2,3,4} {5,6} {7}

We can "solve" the equivalence problem by starting with each element in a named set. When we process statement a�b, we FIND the equivalence classes of a and b and then MERGE them. We can at any time use FIND to tell us the current equivalence class of any element.

The equivalence problem arises in several areas of computer science. For example, one form occurs when a Fortran compiler has to process "equivalence declarations" such as

EQUIVALENCE (A(1),B(1,2),C(3)), (A(2),D,E), (F,G)

Another example, presented in Chapter 7, uses solutions to the equivalence problem to help find minimum-cost spanning trees.

A Simple Implementation of MFSET

Let us begin with a simplified version of the MERGE-FIND ADT. We shall define an ADT, called MFSET, consisting of a set of subsets, which we shall call components, together with the following operations:

  1. MERGE(A, B) takes the union of the components A and B and calls the result either A or B, arbitrarily.
  2. FIND(x) is a function that returns the name of the component of which x is a member.
  3. INITIAL(A, x) creates a component named A that contains only the element x.

To make a reasonable implementation of MFSET, we must restrict our underlying types, or alternatively, we should recognize that MFSET really has two other types as "parameters" - the type of set names and the type of members of these sets. In many applications we can use integers as set names. If we take n to be the number of elements, we may also use integers in the range [1..n] for the members of components. For the implementation we have in mind, it is important that the type of set members be a subrange type, because we want to index into an array defined over that subrange. The type of set names is not important, as this type is the type of array elements, not their indices. Observe, however, that if we wanted the member type to be other than a subrange type, we could create a mapping, with a hash table, for example, that assigned these to unique integers in a subrange. We only need to know the total number of elements in advance.

The implementation we have in mind is to declare

    const
         n = { number of elements };
    type
        MFSET = array[1..n] of integer;

as a special case of the more general type

        array[subrange of members] of (type of set names);

Suppose we declare components to be of type MFSET with the intention that components[x] holds the name of the set currently containing x. Then the three MFSET operations are easy to write. For example, the operation MERGE is shown in Fig. 5.22. INITIAL(A, x) simply sets components[x] to A, and FIND(x) returns components [x].

The time performance of this implementation of MFSET is easy to

    procedure MERGE ( A, B: integer; var C: MFSET );
          var
        x: 1..n;
          begin
        for x:= 1 to n do
             if C[x] = B then
            C[x] := A
          end; { MERGE }

Fig. 5.22. The procedure MERGE.

analyze. Each execution of the procedure MERGE takes O(n) time. On the other hand, the obvious implementations of INITIAL(A, x) and FIND(x) have constant running times.

A Faster Implementation of MFSET

Using the algorithm in Fig. 5.22, a sequence of n-1 MERGE instructions will take O(n2) time.† One way to speed up the MERGE operation is to link together all members of a component in a list. Then, instead of scanning all members when we merge component B into A, we need only run down the list of members of B. This arrangement saves time on the average. However, it could happen that the ith merge is of the form MERGE(A, B) where A is a component of size 1 and B is a component of size i, and that the result is named A. This merge operation would require O(i) steps, and a sequence of n-1 such merge instructions would take on the order of [image] time.

One way to avoid this worst case situation is to keep track of the size of each component and always merge the smaller into the larger.‡ Thus, every time a member is merged into a bigger component, it finds itself in a component at least twice as big. Thus, if there are initially n components, each with one member, none of the n members can have its component changed more than 1+log n times. As the time spent by this new version of MERGE is proportional to the number of members whose component names are changed, and the total number of such changes is at most n(1+log n), we see that O(n log n) work suffices for all merges.

Now let us consider the data structure needed for this implementation. First, we need a mapping from set names to records consisting of

  1. a count giving the number of members in the set and
  2. the index in the array of the first element of that set.

We also need another array of records, indexed by members, to indicate

  1. the set of which each element is a member and
  2. the next array element on the list for that set.

We use 0 to serve as NIL, the end-of-list marker. In a language that lent itself to such constructs, we would prefer to use pointers in this array, but Pascal does not permit pointers into arrays.

In the special case where set names, as well as members, are chosen from the subrange 1..n, we can use an array for the mapping described above. That is, we define

    type
        nametype: 1..n;
        elementtype = 1..n;
        MFSET = record
           setheaders: array[ 1..n] of record
            { headers for set lists }
            count: 0..n;
            firstelement: 0..n
           end;
           names; array[1..n] of record
            { table giving set containing each member }
            setname: nametype;
            nextelement: 0..n
          end
    end;

The procedures INITIAL, MERGE, and FIND are shown in Fig. 5.23.

Figure 5.24 shows an example of the data structure used in Fig. 5.23, where set 1 is {1, 3, 4}, set 2 is {2}, and set 5 is {5, 6}.

A Tree Implementation of MFSET's

Another, completely different, approach to the implementation of MFSET's uses trees with pointers to parents. We shall describe this approach informally. The basic idea is that nodes of trees correspond to set members, with an array or other implementation of a mapping leading from set members to their nodes. Each node, except the root of each tree, has a pointer to its parent. The roots hold the name of the set, as well as an element. A mapping from set names to roots allows access to any given set, when merges are done.

Figure 5.25 shows the sets A = {1, 2, 3, 4}, B = {5, 6}, and C = {7} represented in this form. The rectangles are assumed to be part of the root node, not separate nodes.

procedure INITIAL ( A: nametype; x: elementtype; var C:
MFSET );
    { initialize A to a set containing x only }
    begin
       C.names [x].setname : = A;
       C.names [x].nextelement := 0;
          { null pointer at end of list of members of A }
       C.setheaders [A].count := 1;
       C.setheaders [A].firstelement := x
    end; { INITIAL }

procedure MERGE ( A, B: nametype; var C: MFSET );
    { merge A and B, calling the result A or B, arbitrarily }
    var
        i: 0..n; { used to find end of smaller list }
    begin
        if C.setheaders [A ].count > C.setheaders
[B].count then begin
        { A is the larger set; merge B into A }
        { find end of B, changing set names to A as we go }
        i := C.setheaders[B].firstelement;
        while C.names [i].nextelement <> 0 do
begin
           C.names [i].setname := A;
           i := C.names [i].nextelement
        end;
        { append list A to the end of B and call the result A }
        { now i is the index of the last member of B }
        C.names [i].setname := A;
        C.names [i].nextelement := C.setheaders [A
].firstelement;
        C.setheaders [A ].firstelement := C.setheaders [B
].firstelement;
        C.setheaders [A ].count := C.setheaders [A
].count +
            C.setheaders [B ].count;
        C.setheaders[B ].count := 0;
        C.setheaders[B ].firstelement := 0
             { above two steps not really necessary, as set B no longer exists }
       end
       else { B is at least as large as A }
        { code similar to case above, but with A and B interchanged }
      end; { MERGE }

function FIND ( x: 1..n; var C: MFSET );
    { return the name of the set of which x is a member }
    begin
         return ( C. names [x ].setname)
    end; { FIND }

Fig. 5.23. The operations of an MFSET.

To find the set containing an element x, we first consult a mapping (e.g., an array) not shown in Fig. 5.25, to obtain a pointer to the node for x. We then follow the path from that node to the root of its tree and read the name of the set there.

The basic merge operation is to make the root of one tree be a child of the root of the other. For example, we could merge A and B of Fig. 5.25 and call the result A, by making node 5 a child of node 1. The result is shown in Fig. 5.26. However, indiscriminate merging could result in a tree of n nodes that is a single chain. Then doing a FIND operation on each of those nodes would take O(n2) time. Observe that although a merge can be done in O(1) steps, the cost of a reasonable number of FIND's will dominate the total cost, and this approach is not necessarily better than the simplest one for executing n merges and n finds.

However, a simple improvement guarantees that if n is the number of elements, then no FIND will take more than O(log n) steps. We simply keep at each root a count of the number of elements in the set, and when called upon to merge two sets, we make the root of the smaller tree be a child of the root of the larger. Thus, every time a node is moved to a new tree, two things happen: the distance from the node to its root increases by one, and the node will be in a set with at least twice as many elements as before. Thus, if n is the total number of elements, no node can be moved more than logn times; hence, the distance to its root can never exceed logn. We conclude that each FIND requires at most O(log n) time.

Path Compression

Another idea that may speed up this implementation of MFSET's is path compression. During a FIND, when following a path from some node to the root, make each node encountered along the path be a child of the root. The easiest way to do this is in two passes. First, find the root, and then retraverse the same path, making each node a child of the root.

Example 5.7. Figure 5.27(a) shows a tree before executing a FIND operation on the node for element 7 and Fig. 5.27(b) shows the result after 5 and 7 are made children of the root. Nodes 1 and 2 on the path are not moved because 1 is the root, and 2 is already a child of the root.

Path compression does not affect the cost of MERGE's; each MERGE still takes a constant amount of time. There is, however, a subtle speedup in FIND's since path compression tends to shorten a large number of paths from various nodes to the root with relatively little effort.

Unfortunately, it is very difficult to analyze the average cost of FIND's when path compression is used. It turns out that if we do not require that smaller trees be merged into larger ones, we require no more than O(n log n) time to do n FIND's. Of course, the first FIND may take O(n) time by itself for a tree consisting of one chain. But path compression can change a tree very rapidly and no matter in what order we apply FIND to elements of any tree no more than O(n) time is spent on n FIND's. However, there are

Fig. 5.24. Example of the MFSET data structure.

Fig. 5.25. MFSET represented by a collection of trees.

sequences of MERGE and FIND instructions that require W(nlog n) time.

The algorithm that both uses path compression and merges the smaller tree into the larger is asymptotically the most efficient method known for implementing MFSET's. In particular, n FIND's require no more than O(n a(n)) time, where a(n) is a function that is not constant, yet grows much more slowly than logn. We shall define a(n) below, but the analysis that leads to this bound is beyond the scope of this book.

Fig. 5.26. Merging B into A.

Fig. 5.27. An example of path compression.

The Function a(n)

The function a(n) is closely related to a very rapidly growing function A(x, y), known as Ackermann's function. A(x, y) is defined recursively by:

A(0,y) = 1 for y � 0

A(1, 0) = 2

A(x, 0) = x+2 for x � 2

A(x,y) = A(A(x-1, y),y-1) for x, y � 1

Each value of y defines a function of one variable. For example, the third line above tells us that for y=0, this function is "add 2." For y = 1, we have A(x, 1) = A(A(x-1, 1),0) = A(x-1, 1) + 2, for x > 1, with A(1, 1) = A(A(0, 1),0) = A(1, 0) = 2. Thus A(x, 1) = 2x for all x � 1. In other words, A(x, 1) is "multiply by 2." Then, A(x, 2) = A(A(x-1, 2), 1) = 2A(x-1, 2) for x > 1. Also, A(1, 2) = A(A(0,2), 1) = A(1, 1) = 2. Thus A(x, 2) = 2x. Similarly, we can show that A(x, 3) = 22...2 (stack of x 2's), while A(x, 4) is so rapidly growing there is no accepted mathematical notation for such a function.

A single-variable Ackermann's function can be defined by letting A(x) = A(x, x). The function a(n) is a pseudo-inverse of this single variable function. That is, a(n) is the least x such that n � A(x). For example, A(1) = 2, so a(1) = a(2) = 1. A(2) = 4, so a(3) = a(4) = 2. A(3) = 8, so a(5) = . . . = a(8) = 3. So far, a(n) seems to be growing rather steadily.

However, A(4) is a stack of 65536 2's. Since log(A(4)) is a stack of 65535 2's, we cannot hope even to write A(4) explicitly, as it would take log(A(4)) bits to do so. Thus a(n) � 4 for all integers n one is ever likely to encounter. Nevertheless, a(n) eventually reaches 5, 6, 7, . . . on its unimaginably slow course toward infinity.

5.6 An ADT with MERGE and SPLIT

Let S be a set whose members are ordered by the relation <. The operation SPLIT(S, S1, S2, x) partitions S into two sets: S1={ a � a is in S and a < x} and S2 = {a � a is in S and a � x}. The value of S after the split is undefined, unless it is one of S1 or S2. There are several situations where the operation of splitting sets by comparing each member with a fixed value x is essential. We shall consider one such problem here.

The Longest Common Subsequence Problem

A subsequence of a sequence x is obtained by removing zero or more (not necessarily contiguous) elements from x. Given two sequences x and y, a longest common subsequence (LCS) is a longest sequence that is a subsequence of both x and y.

For example, an LCS of 1, 2, 3, 2, 4, 1, 2 and 2, 4, 3, 1, 2, 1 is the subsequence 2, 3, 2, 1, formed as shown in Fig. 5.28. There are other LCS's as well, such as 2, 4, 1, 2, but there are no common subsequences of length 5.

Fig. 5.28. A longest common subsequence.

There is a UNIX command called diff that compares files line-by-line, finding a longest common subsequence, where a line of a file is considered an element of the subsequence. That is, whole lines are analogous to the integers 1, 2, 3, and 4 in Fig. 5.28. The assumption behind the command diff is that the lines of each file that are not in this LCS are lines inserted, deleted or modified in going from one file to the other. For example, if the two files are versions of the same program made several days apart, diff will, with high probability, find the changes.

There are several general solutions to the LCS problem that work in O(n2) steps on sequences of length n. The command diff uses a different strategy that works well when the files do not have too many repetitions of any line. For example, programs will tend to have lines "begin" and "end" repeated many times, but other lines are not likely to repeat.

The algorithm used by diff for finding an LCS makes use of an efficient implementation of sets with operations MERGE and SPLIT, to work in time O(plogn), where n is the maximum number of lines in a file and p is the number of pairs of positions, one from each file, that have the same line. For example, p for the strings in Fig. 5.28 is 12. The two 1's in each string contribute four pairs, the 2's contribute six pairs, and 3 and 4 contribute one pair each. In the worst case, p,. could be n2, and this algorithm would take O(n2logn) time. However, in practice, p is usually closer to n, so we can expect an O(nlogn) time complexity.

To begin the description of the algorithm let A = a1a2 � � � an and B = b1b2 � � � bm be the two strings whose LCS we desire. The first step is to tabulate for each value a, the positions of the string A at which a appears. That is, we define PLACES(a)= { i � a = ai }. We can compute the sets PLACES(a) by constructing a mapping from symbols to headers of lists of positions. By using a hash table, we can create the sets PLACES(a) in O(n) "steps" on the average, where a "step" is the time it takes to operate on a symbol, say to hash it or compare it with another. This time could be a constant if symbols are characters or integers, say. However, if the symbols of A and B are really lines of text, then steps take an amount of time that depends on the average length of a line of text.

Having computed PLACES(a) for each symbol a that occurs in string A, we are ready to find an LCS. To simplify matters, we shall only show how to find the length of the LCS, leaving the actual construction of the LCS as an exercise. The algorithm considers each bj, for j = 1, 2, ... , m, in turn. After considering bj, we need to know, for each i between 0 and n, the length of the LCS of strings a1 � � � ai and b1 � � � bj.

We shall group values of i into sets Sk, for k = 0, 1, . . . , n, where Sk consists of all those integers i such that the LCS of a1 � � � ai and b1 � � � bj has length k. Note that Sk will always be a set of consecutive integers, and the integers in Sk+1 are larger than those in Sk, for all k.

Example 5.8. Consider Fig. 5.28, with j = 5. If we try to match zero symbols from the first string with the first five symbols of the second (24312), we naturally have an LCS of length 0, so 0 is in S0. If we use the first symbol from the first string, we can obtain an LCS of length 1, and if we use the first two symbols, 12, we can obtain an LCS of length 2. However, using 123, the first three symbols, still gives us an LCS of length 2 when matched against 24312. Proceeding in this manner, we discover S0 = {0}, S1 = {1}, S2 = {2, 3}, S3 = {4, 5, 6}, and S4 = {7}.

Suppose that we have computed the Sk's for position j-1 of the second string and we wish to modify them to apply to position j. We consider the set PLACES(bj). For each r in PLACES(bj), we consider whether we can improve some of the LCS's by adding the match between ar and bj to the LCS of a1 � � � ar- 1 and b1 � � � bj. That is, if both r-1 and r are in Sk, then all s � r in Sk really belong in Sk+1 when bj is considered. To see this we observe that we can obtain k matches between a1 � � � ar-1 and bl � � � bj-1, to which we add a match between ar and bj. We can modify Sk and Sk+1 by the following steps.

  1. FIND(r) to get Sk.
  2. If FIND(r-1) is not Sk, then no benefit can be had by matching bj with ar. Skip the remaining steps and do not modify Sk or Sk+1.
  3. If FIND(r-1) = Sk, apply SPLIT(Sk, Sk, S'k, r) to separate from Sk those members greater than or equal to r.
  4. MERGE(S'k, Sk+1, Sk+1) to move these elements into Sk+1.

It is important to consider the members of PLACES(bj) largest first. To see why, suppose for example that 7 and 9 are in PLACES(bj), and before bj is considered, S3={6, 7, 8, 9} and S4={10, 11}.

If we consider 7 before 9, we split S3 into S3 = {6} and S'3 = {7, 8, 9}, then make S4 = {7, 8, 9, 10, 11}. If we then consider 9, we split S4 into S4={7, 8} and S'4 = {9, 10, 11}, then merge 9, 10 and 11 into S5. We have thus moved 9 from S3 to S5 by considering only one more position in the second string, representing an impossibility. Intuitively, what has happened is that we have erroneously matched bj against both a7 and a9 in creating an imaginary LCS of length 5.

In Fig. 5.29, we see a sketch of the algorithm that maintains the sets Sk as we scan the second string. To determine the length of an LCS, we need only execute FIND(n) at the end.

    procedure LCS;
       begin
(1)      initialize S0 = {0, 1, ... , n} and Si = � for i = 1, 2, ... , n;
(2)      for j := 1 to n do { compute Sk's for position
j }
(3)            for r in PLACES(bj), largest first do begin
(4)                k := FIND(r);
(5)                if k = FIND(r-1) then begin { r is not
smallest in Sk }
(6)                     SPLIT(Sk, Sk, S'k,
r);
(7)                     MERGE(S'k, Sk+1,
Sk+1)
          end
          end
    end; { LCS }

Fig. 5.29. Sketch of longest common subsequence program.

Time Analysis of the LCS Algorithm

As we mentioned earlier, the algorithm of Fig. 5.29 is a useful approach only if there are not too many matches between symbols of the two strings. The measure of the number of matches is

where |PLACES(bj)| denotes the number of elements in set PLACES(bj). In other words, p is the sum over all bj of the number of positions in the first string that match bj. Recall that in our discussion of file comparison, we expect p to be of the same order as m and n, the lengths of the two strings (files).

It turns out that the 2-3 tree is a good structure for the sets Sk. We can initialize these sets, as in line (1) of Fig. 5.29, in O(n) steps. The FIND operation requires an array to serve as a mapping from positions r to the leaf for r and also requires pointers to parents in the 2-3 tree. The name of the set, i.e., k for Sk, can be kept at the root, so we can execute FIND in O(logn) steps by following parent pointers until we reach the root. Thus all executions of lines (4) and (5) together take O(plogn) time, since those lines are each executed exactly once for each match found.

The MERGE operation of line (5) has the special property that every member of S'k is lower than every member of Sk+1, and we can take advantage of this fact when using 2-3 trees for an implementation.† To begin the MERGE, place the 2-3 tree for S'k to the left of that for Sk+1. If both are of the same height, create a new root with the roots of the two trees as children. If S'k is shorter, insert the root of that tree as the leftmost child of the leftmost node of Sk+1 at the appropriate level. If this node now has four children, we modify the tree exactly as in the INSERT procedure of Fig. 5.20. An example is shown in Fig. 5.30. Similarly, if Sk+1 is shorter, make its root the rightmost child of the rightmost node of S'k at the appropriate level.

Fig. 5.30. Example of MERGE.

The SPLIT operation at r requires that we travel up the tree from leaf r, duplicating every interior node along the path and giving one copy to each of the two resulting trees. Nodes with no children are eliminated, and nodes with one child are removed and have that child inserted into the proper tree at the proper level.

Example 5.9. Suppose we split the tree of Fig. 5.30(b) at node 9. The two trees, with duplicated nodes, are shown in Fig. 5.31(a). On the left, the parent of 8 has only one child, so 8 becomes a child of the parent of 6 and 7. This parent now has three children, so all is as it should be; if it had four children, a new node would have been created and inserted into the tree. We need only eliminate nodes with zero children (the old parent of 8) and the chain of nodes with one child leading to the root. The parent of 6, 7, and 8 becomes the new root, as shown in Fig. 5.31(b). Similarly, in the right-hand tree, 9 becomes a sibling of 10 and 11, and unnecessary nodes are eliminated, as is also shown in Fig. 5.31(b).

Fig. 5.31. An example of SPLIT.

If we do the splitting and reorganization of the 2-3 tree bottom up, it can be shown by consideration of a large number of cases that O(logn) steps suffices. Thus, the total time spent in lines (6) and (7) of Fig. 5.29 is O(plogn), and hence the entire algorithm takes O(plogn) steps. We must add in the preprocessing time needed to compute and sort PLACES(a) for symbols a. As we mentioned, if the symbols a are "large" objects, this time can be much greater than any other part of the algorithm. As we shall see in Chapter 8, if the symbols can be manipulated and compared in single "steps," then O(nlogn) time suffices to sort the first string a1a2 � � � an (actually, to sort objects (i, ai) on the second field), whereupon PLACES(a) can be read off from this list in O(n) time. Thus, the length of the LCS can be computed in O(max(n, p) logn) time which, since p � n is normal, can be taken as O(plogn).

Exercises

5.1 | Draw all possible binary search trees containing the four elements 1, 2, 3, 4.
5.2 | Insert the integers 7, 2, 9, 0, 5, 6, 8, 1 into a binary search tree by repeated application of the procedure INSERT of Fig. 5.3.
5.3 | Show the result of deleting 7, then 2 from the final tree of Exercise 5.2.
*5.4 | When deleting two elements from a binary search tree using the procedure of Fig. 5.5, does the final tree ever depend on the order in which you delete them?
5.5 | We wish to keep track of all 5-character substrings that occur in a given string, using a trie. Show the trie that results when we insert the 14 substrings of length five of the string ABCDABACDEBACADEBA.
*5.6 | To implement Exercise 5.5, we could keep a pointer at each leaf, which, say, represents string abcde, to the interior node representing the suffix bcde. That way, if the next symbol, say f, is received, we don't have to insert all of bcdef, starting at the root. Furthermore, having seen abcde, we may as well create nodes for bcde, cde, de, and e, since we shall, unless the sequence ends abruptly, need those nodes eventually. Modify the trie data structure to maintain such pointers, and modify the trie insertion algorithm to take advantage of this data structure.
5.7 | Show the 2-3 tree that results if we insert into an empty set, represented as a 2-3 tree, the elements 5, 2, 7, 0, 3, 4, 6, 1, 8, 9.
5.8 | Show the result of deleting 3 from the 2-3 tree that results from Exercise 5.7.
5.9 | Show the successive values of the various Si's when implementing the LCS algorithm of Fig. 5.29 with first string abacabada, and second string bdbacbad.
5.10 Suppose we use 2-3 trees to implement the MERGE and SPLIT operations as in Section 5.6.
  1. Show the result of splitting the tree of Exercise 5.7 at 6.
  2. Merge the tree of Exercise 5.7 with the tree consisting of leaves for elements 10 and 11.
5.11 Some of the structures discussed in this chapter can be modified easily to support the MAPPING ADT. Write procedures MAKENULL, ASSIGN, and COMPUTE to operate on the following data structures.
  1. Binary search trees. The "<" ordering applies to domain elements.
  2. 2-3 trees. At interior nodes, place only the key field of domain elements.
5.12 | Show that in any subtree of a binary search tree, the minimum element is at a node without a left child.
5.13 | Use Exercise 5.12 to produce a nonrecursive version of DELETE-MIN.
5.14 | Write procedures ASSIGN, VALUEOF, MAKENULL and GETNEW for trie nodes represented as lists of cells.
*5.15 | How do the trie (list of cells implementation), the open hash table, and the binary search tree compare for speed and for space utilization when elements are strings of up to ten characters?
*5.16 | If elements of a set are ordered by a "<" relation, then we can keep one or two elements (not just their keys) at interior nodes of a 2-3 tree, and we then do not have to keep these elements at the leaves. Write INSERT and DELETE procedures for 2-3 trees of this type.
5.17 Another modification we could make to 2-3 trees is to keep only keys at interior nodes, but do not require that the keys k1 and k2 at a node truly be the minimum keys of the second and third subtrees, just that all keys k of the third subtree satisfy k � k2, all keys k of the second satisfy k1 � k < k2, and all keys k of the first satisfy k < k1.
  1. How does this convention simplify the DELETE operation?
  2. Which of the dictionary and mapping operations are made more complicated or less efficient?
*5.18 | Another data structure that supports dictionaries with the MIN operation is the AVL tree (named for the inventors' initials) or height-balanced tree. These trees are binary search trees in which the heights of two siblings are not permitted to differ by more than one. Write procedures to implement INSERT and DELETE, while maintaining the AVL-tree property.
5.19 | Write the Pascal program for procedure delete1 of Fig. 5.21.
*5.20 | A finite automaton consists of a set of states, which we shall take to be the integers 1..n and a table transitions[state, input] giving a next state for each state and each input character. For our purposes, we shall assume that the input is always either 0 or 1. Further, certain of the states are designated accepting states. For our purposes, we shall assume that all and only the even numbered states are accepting. Two states p and q are equivalent if either they are the same state, or (i) they are both accepting or both nonaccepting, (ii) on input 0 they transfer to equivalent states, and (iii) on input 1 they transfer to equivalent states. Intuitively, equivalent states behave the same on all sequences of inputs; either both or neither lead to accepting states. Write a program using the MFSET operations that computes the sets of equivalent states of a given finite automaton.
**5.21 In the tree implementation of MFSET:
  1. Show that W(n log n) time is needed for certain lists of n operations if path compression is used but larger trees are permitted to be merged into smaller ones.
  2. Show that O(n a(n)) is the worst case running time for n operations if path compression is used, and the smaller tree is always merged into the larger.
5.22 | Select a data structure and write a program to compute PLACES (defined in Section 5.6) in average time O(n) for strings of length n.
*5.23 | Modify the LCS procedure of Fig. 5.29 to compute the LCS, not just its length.
*5.24 | Write a detailed SPLIT procedure to work on 2-3 trees.
*5.25 | If elements of a set represented by a 2-3 tree consist only of a key field, an element whose key appears at an interior node need not appear at a leaf. Rewrite the dictionary operations to take advantage of this fact and avoid storing any element at two different nodes.

Bibliographic Notes

Tries were first proposed by Fredkin [1960]. Bayer and McCreight [1972] introduced B-trees, which, as we shall see in Chapter 11, are a generalization of 2-3 trees. The first uses of 2-3 trees were by J. E. Hopcroft in 1970 (unpublished) for insertion, deletion, concatenation, and splitting, and by Ullman [1974] for a code optimization problem.

The tree structure of Section 5.5, using path compression and merging smaller into larger, was first used by M. D. McIlroy and R. Morris to construct minimum-cost spanning trees. The performance of the tree implementation of MFSET's was analyzed by Fischer [1972] and by Hopcroft and Ullman [1973]. Exercise 5.21(b) is from Tarjan [1975].

The solution to the LCS problem of Section 5.6 is from Hunt and Szymanski [1977]. An efficient data structure for FIND, SPLIT, and the restricted MERGE (where all elements of one set are less than those of the other) is described in van Emde Boas, Kaas, and Zijlstra [1977].

Exercise 5.6 is based on an efficient algorithm for matching patterns developed by Weiner [1973]. The 2-3 tree variant of Exercise 5.16 is discussed in detail in Wirth [1976]. The AVL tree structure in Exercise 5.18 is from Adel'son-Vel'skii and Landis [1962].

† Recall the left child of the root is a descendant of itself, so we have not ruled out the possibility that x is at the left child of the root.

† The highest-valued node among the descendants of the left child would do as well.

† Recall that all logarithms are to the base 2 unless otherwise noted.

† Trie was originally intended to be a homonym of "tree" but to distinguish these two terms many people prefer to pronounce trie as though it rhymes with "pie."

† VALUEOF is a function version of COMPUTE in Section 2.5.

† There is another version of 2-3 trees that places whole records at interior nodes, as a binary search tree does.

† All nodes, however, take the largest amount of space needed for any variant types, so Pascal is not really the best language of implementing 2-3 trees in practice.M

† A useful variant would take only a key value and delete any element with that key.

† We say a is congruent to b modulo n if a and b have the same remainders when divided by n, or put another way, a-b is a multiple of n.

† Note that n- 1 is the largest number of merges that can be performed before all elements are in one set.

‡ Note that our ability to call the resulting component by the name of either of its constituents is important here, although in the simpler implementation, the name of the first argument was always picked.

† Strictly speaking we should use a different name for the MERGE operation, as the implementation we propose will not work to compute the arbitrary union of disjoint sets, while keeping the elements sorted so operations like SPLIT and FIND can be performed.

Table of Contents Go to Chapter 6


Data Structures and Algorithms: CHAPTER 6: Directed Graphs

 Next Chapter Return to Table of Contents Previous Chapter

Directed Graphs

In problems arising in computer science, mathematics, engineering, and many other disciplines we often need to represent arbitrary relationships among data objects. Directed and undirected graphs are natural models of such relationships. This chapter presents the basic data structures that can be used to represent directed graphs. Some basic algorithms for determining the connectivity of directed graphs and for finding shortest paths are also presented.

6.1 Basic Definitions

A directed graph (digraph for short) G consists of a set of vertices V and a set of arcs E. The vertices are also called nodes or points; the arcs could be called directed edges or directed lines. An arc is an ordered pair of vertices (v, w); v is called the tail and w the head of the arc. The arc (v, w) is often expressed by v � w and drawn as

Notice that the "arrowhead" is at the vertex called the "head" and the tail of the arrow is at the vertex called the "tail." We say that arc v � w is from v to w, and that w is adjacent to v.

Example 6.1. Figure 6.1 shows a digraph with four vertices and five arcs.

The vertices of a digraph can be used to represent objects, and the arcs relationships between the objects. For example, the vertices might represent cities and the arcs airplane flights from one city to another. As another example, which we introduced in Section 4.2, a digraph can be used to represent the flow of control in a computer program. The vertices represent basic blocks and the arcs possible transfers of flow of control.

A path in a digraph is a sequence of vertices v1, v2, . . . , vn, such that v1 � v2, v2 � v3, . . . , vn-1 � vn are arcs. This path is from vertex v1 to vertex vn, and passes through vertices v2, v3, . . . , vn-1, and ends at vertex vn. The length of a path is the number of arcs on the path, in this case, n-1. As a

Fig. 6.1. Directed graph.

special case, a single vertex v by itself denotes a path of length zero from v to v. In Fig. 6.1, the sequence 1, 2, 4 is a path of length 2 from vertex 1 to vertex 4.

A path is simple if all vertices on the path, except possibly the first and last, are distinct. A simple cycle is a simple path of length at least one that begins and ends at the same vertex. In Fig. 6.1, the path 3, 2, 4, 3 is a cycle of length three.

In many applications it is useful to attach information to the vertices and arcs of a digraph. For this purpose we can use a labeled digraph, a digraph in which each arc and/or each vertex can have an associated label. A label can be a name, a cost, or a value of any given data type.

Example 6.2. Figure 6.2 shows a labeled digraph in which each arc is labeled by a letter that causes a transition from one vertex to another. This labeled digraph has the interesting property that the arc labels on every cycle from vertex 1 back to vertex 1 spell out a string of a's and b's in which both the number of a's and b's is even.

In a labeled digraph a vertex can have both a name and a label. Quite frequently, we shall use the vertex label as the name of the vertex. Thus, the numbers in Fig. 6.2 could be interpreted as vertex names or vertex labels.

6.2 Representations for Directed Graphs

Several data structures can be used to represent a directed graph. The appropriate choice of data structure depends on the operations that will be applied to the vertices and arcs of the digraph. One common representation for a digraph G = (V,E) is the adjacency matrix. Suppose V = {1, 2 , . . . , n}. The adjacency matrix for G is an n x n matrix A of booleans, where A[i, j] is true if and only if there is an arc from vertex i to j. Often, we shall exhibit adjacency matrices with 1 for true and 0 for false; adjacency matrices may even be implemented that way. In the adjacency matrix representation the time required to access an element of an adjacency matrix is independent of the size of V and E. Thus the adjacency matrix

Fig. 6.2. Transition digraph.

representation is useful in those graph algorithms in which we frequently need to know whether a given arc is present.

Closely related is the labeled adjacency matrix representation of a digraph, where A[i, j] is the label on the arc going from vertex i to vertex j. If there is no arc from i to j, then a value that cannot be a legitimate label must be used as the entry for A[i, j].

Example 6.3. Figure 6.3 shows the labeled adjacency matrix for the digraph of Fig. 6.2. Here, the label type is char, and a blank represents the absence of an arc.

Fig. 6.3. Labeled adjacency matrix for digraph of Fig. 6.2.

The main disadvantage of using an adjacency matrix to represent a digraph is that the matrix requires W(n2) storage even if the digraph has many fewer than n2 arcs. Simply to read in or examine the matrix would require O(n2) time, which would preclude O(n) algorithms for manipulating digraphs with O(n) arcs.

To avoid this disadvantage we can use another common representation for a digraph G = (V,E) called the adjacency list representation. The adjacency list for a vertex i is a list, in some order, of all vertices adjacent to i. We can represent G by an array HEAD, where HEAD[i] is a pointer to the adjacency list for vertex i. The adjacency list representation of a digraph requires storage proportional to sum of the number of vertices plus the number of arcs; it is often used when the number of arcs is much less than n2. However, a potential disadvantage of the adjacency list representation is that it may take O(n) time to determine whether there is an arc from vertex i to vertex j, since there can be O(n) vertices on the adjacency list for vertex i.

Example 6.4. Figure 6.4 shows an adjacency list representation for the digraph of Fig. 6.1, where singly linked lists are used. If arcs had labels, these could be included in the cells of the linked list.

Fig. 6.4. Adjacency list representation for digraph of Fig. 6.1.

If we did insertions and deletions from the adjacency lists, we might prefer to have the HEAD array point to header cells that did not contain adjacent vertices.† Alternatively, if the graph were expected to remain fixed, with no (or very few) changes to be made to the adjacency lists, we might prefer HEAD[i] to be a cursor to an array ADJ, where ADJ[HEAD[i]], ADJ[HEAD[i]+ 1] , . . . , and so on, contained the vertices adjacent to vertex i, up to that point in ADJ where we first encounter a 0, which marks the end of the list of adjacent vertices for i. For example, Fig. 6.1 could be represented as in Fig. 6.5.

Directed Graph ADT's

We could define an ADT corresponding to the directed graph formally and study implementations of its operations. We shall not pursue this direction extensively, because there is little of a surprising nature, and the principal data structures for graphs have already been covered. The most common operations on directed graphs include operations to read the label of a vertex or arc, to insert or delete vertices and arcs, and to navigate by following arcs

Fig. 6.5. Another adjacency list representation of Fig. 6.1.

from tail to head.

The latter operations require a little thought. Most frequently, we shall encounter in informal programs statements like

   for each vertex w adjacent to vertex v do       (6.1)
                  { some action on w }

To implement such a step, we need the notion of an index type for the set of vertices adjacent to some one vertex v. For example, if adjacency lists are used to represent the graph, then an index is really a position on the adjacency list for v. If an adjacency matrix is used, an index is an integer representing an adjacent vertex. We need the following three operations on directed graphs.

  1. FIRST(v) returns the index for the first vertex adjacent to v. The index for the null vertex L is returned if there is no vertex adjacent to v.
  2. NEXT(v, i) returns the index after index i for the vertices adjacent to v. L is returned if i is the last index for vertices adjacent to v.
  3. VERTEX(v, i) returns the vertex with index i among the vertices adjacent to v.

Example 6.5. If the adjacency matrix representation is chosen, VERTEX(v, i) returns i. FIRST(v) and NEXT(v, i) can be written as in Fig. 6.6 to operate on an externally defined n x n boolean matrix A. We assume A is declared

   array [1..n, 1..n] of boolean
and that 0 is used for L. We can then implement the statement (6.1) as in Fig. 6.7. Note that FIRST(v) can also be implemented as NEXT(v, 0).

     function FIRST ( v: integer ): integer;
         var
            i: integer;
         begin
             for i := 1 to n do
                  if A[v, i] then
                       return (i);
             return (0) { if we reach here, v has no adjacent vertex }
         end; { FIRST }

     function NEXT ( v: integer; i: integer ): integer;
         var
            j: integer;
         begin
             for j := i + 1 to n do
                  if A[v, j] then
                       return (j);
             return (0)
         end; { NEXT }

Fig. 6.6. Operations to scan adjacent vertices.

     i := FIRST(v);
     while i <> L do begin
         w := VERTEX(v, i);
         { some action on w }
         i := NEXT(v, i)
     end

Fig. 6.7. Iteration over vertices adjacent to v.

6.3 The Single Source Shortest Paths Problem

In this section we consider a common path-finding problem on directed graphs. We are given a directed graph G = (V, E) in which each arc has a nonnegative label, and one vertex is specified as the source. Our problem is to determine the cost of the shortest path from the source to every other vertex in V, where the length of a path is just the sum of the costs of the arcs on the path. This problem is often called the single-source shortest paths problem.† Note that we shall talk of paths as having "length" even if costs represent something different, like time.

We might think of G as a map of airline flights, in which each vertex represents a city and each arc v � w an airline route from city v to city w. The label on arc v � w is the time to fly from v to w.‡ Solving the single-source shortest paths problem for this directed graph would determine the minimum travel time from a given city to every other city on the map.

To solve this problem we shall use a "greedy" technique, often known as Dijkstra's algorithm. The algorithm works by maintaining a set S of vertices whose shortest distance from the source is already known. Initially, S contains only the source vertex. At each step, we add to S a remaining vertex v whose distance from the source is as short as possible. Assuming all arcs have nonnegative costs, we can always find a shortest path from the source to v that passes only through vertices in S. Call such a path special. At each step of the algorithm, we use an array D to record the length of the shortest special path to each vertex. Once S includes all vertices, all paths are "special," so D will hold the shortest distance from the source to each vertex.

The algorithm itself is given in Fig. 6.8. It assumes that we are given a directed graph G = (V, E) where V = {1, 2, . . . , n} and vertex 1 is the source. C is a two-dimensional array of costs, where C[i, j] is the cost of going from vertex i to vertex j on arc i � j. If there is no arc i � j, then we assume C[i, j] is �, some value much larger than any actual cost. At each step D [i] contains the length of the current shortest special path to vertex i.

Example 6.6. Let us apply Dijkstra to the directed graph of Fig. 6.9. Initially, S = {1}, D[2] = 10, D[3] = �, D[4] = 30 and D[5] = 100. In the first iteration of the for-loop of lines (4)-(8), w = 2 is selected as the vertex with the minimum D value. Then we set D[3] = min(�, 10+50)= 60. D(4) and D(5) do not change, because reaching them from 1 directly is shorter than going through vertex 2. The sequence of D-values after each iteration of the for-loop is shown in Fig. 6.10.

If we wish to reconstruct the shortest path from the source to each vertex, then we can maintain another array P of vertices, such that P [v] contains the vertex immediately before vertex v in the shortest path. Initialize P[v] to 1 for all v � 1. The P-array can be updated right after line (8) of Dijkstra. If D[w]+C[w,v]<D[v] at line (8), then we set P[v]:= w. Upon termination

          procedure Dijkstra;
              { Dijkstra computes the cost of the shortest paths
               from vertex 1 to every vertex of a directed graph }
      begin
(1)           S := {1};
(2)           for i := 2 to n do
(3)                D[i] := C[1, i]; { initialize D }
(4)           for i := 1 to n-1 do begin
(5)                choose a vertex w in V-S such that
                       D[w] is a minimum;
(6)                add w to S;
(7)                for each vertex v in V-S do
(8)                     D[v] := min(D[v], D[w] +
C[w, v])
              end
          end; { Dijkstra }

Fig. 6.8. Dijkstra's algorithm.

Fig. 6.9. Digraph with labeled arcs.

of Dijkstra the path to each vertex can be found by tracing backward the predecessor vertices in the P-array.

Example 6.7. For the digraph in Example 6.6 the P-array would have the values P[2] = 1, P[3] = 4, P[4] = 1, and P[5] = 3. To find the shortest path from vertex 1 to vertex 5, for example, we would trace the predecessors in reverse order beginning at vertex 5. From the P-array we determine 3 is the predecessor of 5, 4 the predecessor of 3, and 1 the predecessor of 4. Thus the shortest path from vertex 1 to vertex 5 is 1, 4, 3, 5.

Fig. 6.10. Computation of Dijkstra on digraph of Fig. 6.9.

Why Dijkstra's Algorithm Works

Dijkstra's algorithm is an example where "greed" pays off, in the sense that what appears locally as the best thing to do turns out to be the best over all. In this case, the locally "best" thing to do is to find the distance to the vertex w that is outside S but has the shortest special path. To see why in this case there cannot be a shorter nonspecial path from the source to w, observe Fig. 6.11. There we show a hypothetical shorter path to w that first leaves S to go to vertex x, then (perhaps) wanders into and out of S several times before ultimately arriving at w.

But if this path is shorter than the shortest special path to w, then the initial segment of the path from the source to x is a special path to x shorter than the shortest special path to w. (Notice how important the fact that costs are nonnegative is here; without it our argument wouldn't work, and in fact Dijkstra's algorithm would not work correctly.) In that case, when we selected w at line (5) of Fig. 6.8, we should have selected x instead, because D [x] was less than D [w].

To complete a proof that Fig. 6.8 works, we should verify that at all times D [v] is truly the shortest distance of a special path to vertex v. The crux of this argument is in observing that when we add a new vertex w to S at line (6), lines (7) and (8) adjust D to take account of the possibility that there is now a shorter special path to v going through w. If that path goes through the old S to w and then immediately to v, its cost, D [w]+C [w, v], will be compared with D [v] at line (8), and D [v] will be reduced if the new special path is shorter. The only other possibility for a shorter special path is shown in Fig. 6.12, where the path travels to w, then back into the old S, to some member x of the old S, then to v.

But there really cannot be such a path. Since x was placed in S before w, the shortest of all paths from the source to x runs through the old S alone. Therefore, the path to x through w shown in Fig. 6.12 is no shorter than the path directly to x through S. As a result, the length of the path in Fig. 6.12 from the source to w, x, and v is no less from the old value of D [v], since D [v] was no greater than the length of the shortest path to x through S and then directly to v. Thus D [v] cannot be reduced at line (8) by a path through w and x as in Fig. 6.12, and we need not consider the length of such paths.

Fig. 6.11. Hypothetical shorter path to w.

Fig. 6.12. Impossible shortest special path.

Running Time of Dijkstra's Algorithm

Suppose Fig. 6.8 operates on a digraph with n vertices and e edges. If we use an adjacency matrix to represent the digraph, then the loop of lines (7) and (8) takes O(n) time, and it is executed n-1 times for a total time of O(n2). The rest of the algorithm is easily seen to require no more time than this.

If e is much less than n2, we might do better by using an adjacency list representation of the digraph and using a priority queue implemented as a partially ordered tree to organize the vertices in V-S. The loop of lines (7) and (8) can then be implemented by going down the adjacency list for w and updating the distances in the priority queue. A total of e updates will be made, each at a cost of O(logn) time, so the total time spent in lines (7) and (8) is now O(elogn), rather than O(n2).

Lines (1)-(3) clearly take O(n) time, as do lines (4) and (6). Using the priority queue to represent V-S, lines (5)-(6) implement exactly the DELETEMIN operation, and each of the n-1 iterations of these lines requires O(logn) time.

As a result, the total time spent on this version of Dijkstra's algorithm is bounded by O(elogn). This running time is considerably better than O(n2) if e is very small compared with n2.

6.4 The All-Pairs Shortest Paths Problem

Suppose we have a labeled digraph that gives the flying time on certain routes connecting cities, and we wish to construct a table that gives the shortest time required to fly from any one city to any other. We now have an instance of the all-pairs shortest paths (APSP) problem. To state the problem precisely, we are given a directed graph G = (V, E) in which each arc v � w has a non-negative cost C[v, w]. The APSP problem is to find for each ordered pair of vertices (v, w) the smallest length of any path from v to w.

We could solve this problem using Dijkstra's algorithm with each vertex in turn as the source. A more direct way of solving the problem is to use the following algorithm due to R. W. Floyd. For convenience, let us again assume the vertices in V are numbered 1, 2 , . . . , n. Floyd's algorithm uses an n x n matrix A in which to compute the lengths of the shortest paths. We initially set A[i, j] = C[i, j] for all i � j. If there is no arc from i to j, we assume C[i, j] = �. Each diagonal element is set to 0.

We then make n iterations over the A matrix. After the kth iteration, A[i, j] will have for its value the smallest length of any path from vertex i to vertex j that does not pass through a vertex numbered higher than k. That is to say, i and j, the end vertices on the path, may be any vertex, but any intermediate vertex on the path must be less than or equal to k.

In the kth iteration we use the following formula to compute A.

The subscript k denotes the value of the A matrix after the kth iteration, and it should not be assumed that there are n different matrices. We shall eliminate these subscripts shortly. This formula has the simple interpretation shown in Fig. 6.13.

To compute Ak[i, j] we compare Ak- 1[i, j], the cost of going from i to j without going through k or any higher-numbered vertex, with Ak-1[i, k] + Ak- 1[k, j], the cost of going first from i to k and then from k to j, without passing through a vertex numbered higher than k. If passing through vertex k produces a cheaper path than what we had for Ak- 1[i, j], then we choose that cheaper cost for Ak[i, j].

Fig. 6.13. Including k among the vertices to go from i to j.

Example 6.8. Consider the weighted digraph shown in Fig. 6.14. The values of the A matrix initially and after the three iterations are shown in Fig. 6.15.

Fig. 6.14. Weighted digraph.

Since Ak[i, k] = Ak-1[i, k] and Ak[k, j] = Ak-1[k, j], no entry with either subscript equal to k changes during the kth iteration. Therefore, we can perform the computation with only one copy of the A matrix. A program to perform this computation on n x n matrices is shown in Fig. 6.16.

The running time of this program is clearly O(n3), since the program is basically nothing more than a triply nested for-loop. To verify that this program works, it is easy to prove by induction on k that after k passes through the triple for-loop, A[i, j] holds the length of the shortest path from vertex i to vertex j that does not pass through a vertex numbered higher than k.

Fig. 6.15. Values of successive A matrices.

     procedure Floyd ( var A: array[1..n, 1..n]
of real;
            C: array[1..n, 1..n] of real );
        { Floyd computes shortest path matrix A given arc cost matrix C }
        var
            i, j, k: integer;
        begin
            for i := 1 to n do
                for j := 1 to n do
                     A[i, j] := C[i, j];
            for i:= 1 to n do
                A[i, i] := 0;
            for k:= 1 to n do
                for i := 1 to n do
                    for j:= 1 to n do
                        if A[i, k] + A[k, j] < A
[i, j] then
                            A[i, j] := A[i, k] + A[k,
j]
        end; { Floyd }

Fig. 6.16. Floyd's algorithm.

Comparison Between Floyd's and Dijkstra's Algorithms

Since the adjacency-matrix version of Dijkstra finds shortest paths from one vertex in O(n2) time, it, like Floyd's algorithm, can find all shortest paths in O(n3) time. The compiler, machine, and implementation details will determine the constants of proportionality. Experimentation and measurement are the easiest way to ascertain the best algorithm for the application at hand.

If e, the number of edges, is very much less than n2, then despite the relatively low constant factor in the O(n3) running time of Floyd, we would expect the adjacency list version of Dijkstra, taking O(ne logn) time to solve the APSP, to be superior, at least for large sparse graphs.

Recovering the Paths

In many situations we may want to print out the cheapest path from one vertex to another. One way to accomplish this is to use another matrix P, where P[i, j] holds that vertex k that led Floyd to find the smallest value of A[i, j]. If P[i, j]=0, then the shortest path from i to j is direct, following the arc from i to j. The modified version of Floyd in Fig. 6.17 stores the appropriate intermediate vertices into P.

     procedure shortest ( var A: array[1..n, 1..n]
of real;
         C: array[1..n, 1..n] of real; P:
array[1..n, 1..n] of integer );
         { shortest takes an n X n matrix C of arc costs and produces an
             n X n matrix A of lengths of shortest paths and an n X n
matrix
             P giving a point in the "middle" of each shortest path }
         var
             i, j, k: integer;
         begin
             for i:= 1 to n do
                  for j := 1 to n do begin
                       A[i, j] := C[i, j];
                       P[i, j] := 0
                  end;
             for i:= 1 to n do
                  A[i, i] := 0;
             for k := 1 to n do
                  for i:= 1 to n do
                       for j:= 1 to n do
                           if A[i, k] + A[k, j] <
A[i, j] then begin
                                A[i, j] := A[i, k] + A[k,
j];
                                P[i, j] := k
                    end
         end; { shortest }

Fig. 6.17. Shortest paths program.

To print out the intermediate vertices on the shortest path from vertex i to vertex j, we invoke the procedure path(i, j) where path is given in Fig. 6.18. While on an arbitrary matrix P, path could loop forever, if P comes from shortest, we could not, say, have k on the shortest path from i to j and also have j on the shortest path from i to k. Note how our assumption of nonnegative weights is again crucial.

     procedure path ( i, j: integer );
         var
             k: integer;
         begin
             k := P[i, j];
             if k = 0 then
                  return;
             path(i, k);
             writeln(k);
             path(k, j)
         end; { path }

Fig. 6.18. Procedure to print shortest path.

Example 6.9. Figure 6.19 shows the final P matrix for the digraph of Fig. 6.14.

Fig. 6.19. P matrix for digraph of Fig. 6.14.

Transitive Closure

In some problems we may be interested in determining only whether there exists a path of length one or more from vertex i to vertex j. Floyd's algorithm can be specialized readily to this problem; the resulting algorithm, which predates Floyd's, is called Warshall's algorithm.

Suppose our cost matrix C is just the adjacency matrix for the given digraph. That is, C[i, j] = 1 if there is an arc from i to j, and 0 otherwise. We wish to compute the matrix A such that A[i, j] = 1 if there is a path of length one or more from i to j, and 0 otherwise. A is often called the transitive closure of the adjacency matrix.

Example 6.10. Figure 6.20 shows the transitive closure for the adjacency matrix of the digraph of Fig. 6.14.

The transitive closure can be computed using a procedure similar to Floyd by applying the following formula in the kth pass over the boolean A matrix.

   Ak[i, j] = Ak-1[i,
j] or (Ak-1[i, k] and Ak-
1[k, j])

This formula states that there is a path from i to j not passing through a

Fig. 6.20. Transitive closure.

vertex numbered higher than k if

  1. there is already a path from i to j not passing through a vertex numbered higher than k- 1 or
  2. there is a path from i to k not passing through a vertex numbered higher than k - 1 and a path from k to j not passing through a vertex numbered higher than k-1.

As before Ak[i, k] = Ak-1[i, k] and Ak[k, j] = Ak-1[k, j] so we can perform the computation with only one copy of the A matrix. The resulting Pascal program, named Warshall after its discoverer, is shown in Fig. 6.21.

     procedure Warshall ( var A: array[1..n, 1..n
] of boolean;
            C: array[1..n, 1..n] of boolean );
         { Warshall makes A the transitive closure of C }
         var
             i, j, k: integer;
         begin
             for i := 1 to n do
                  for j := 1 to n do
                       A[i, j] := C[i, j];
             for k := 1 to n do
                 for i := 1 to n do
                      for j := 1 to n do
                          if A[i, j ] = false then
                               A[i, j] := A[i, k] and
A[k, j]
         end; { Warshall }

Fig. 6.21. Warshall's algorithm for transitive closure.

An Example: Finding the Center of a Digraph

Suppose we wish to determine the most central vertex in a digraph. This problem can be readily solved using Floyd's algorithm. First, let us make more precise the term "most central vertex." Let v be a vertex in a digraph G = (V, E). The eccentricity of v is

The center of G is a vertex of minimum eccentricity. Thus, the center of a digraph is a vertex that is closest to the vertex most distant from it.

Example 6.11. Consider the weighted digraph shown in Fig. 6.22.

Fig. 6.22. Weighted digraph.

The eccentricities of the vertices are

                  vertex     eccentricity
                    a             �
                    b             6
                    c             8
                    d             5
                    e             7
Thus the center is vertex d.

Finding the center of a digraph G is easy. Suppose C is the cost matrix for G.

  1. First apply the procedure Floyd of Fig. 6.16 to C to compute the all-pairs shortest paths matrix A.
  2. Find the maximum cost in each column i. This gives us the eccentricity of vertex i.
  3. Find a vertex with minimum eccentricity. This is the center of G.

This running time of this process is dominated by the first step, which takes O(n3) time. Step (2) takes O(n2) time and step (3) O(n) time.

Example 6.12. The APSP cost matrix for Fig. 6.22 is shown in Fig. 6.23. The maximum value in each column is shown below.

Fig. 6.23. APSP cost matrix.

6.5 Traversals of Directed Graphs

To solve many problems dealing with directed graphs efficiently we need to visit the vertices and arcs of a directed graph in a systematic fashion. Depth-first search, a generalization of the preorder traversal of a tree, is one important technique for doing so. Depth-first search can serve as a skeleton around which many other efficient graph algorithms can be built. The last two sections of this chapter contain several algorithms that use depth-first search as a foundation.

Suppose we have a directed graph G in which all vertices are initially marked unvisited. Depth-first search works by selecting one vertex v of G as a start vertex; v is marked visited. Then each unvisited vertex adjacent to v is searched in turn, using depth-first search recursively. Once all vertices that can be reached from v have been visited, the search of v is complete. If some vertices remain unvisited, we select an unvisited vertex as a new start vertex. We repeat this process until all vertices of G have been visited.

This technique is called depth-first search because it continues searching in the forward (deeper) direction as long as possible. For example, suppose x is the most recently visited vertex. Depth-first search selects some unexplored arc x � y emanating from x. If y has been visited, the procedure looks for another unexplored arc emanating from x. If y has not been visited, then the procedure marks y visited and initiates a new search at y. After completing the search through all paths beginning at y, the search returns to x, the vertex from which y was first visited. The process of selecting unexplored arcs emanating from x is then continued until all arcs from x have been explored.

An adjacency list L[v] can be used to represent the vertices adjacent to vertex v, and an array mark, whose elements are chosen from (visited, unvisited), can be used to determine whether a vertex has been previously visited. The recursive procedure dfs is outlined in Fig. 6.24. To use it on an n-vertex graph, we initialize mark to unvisited and then commence a depth-first search from each vertex that is still unvisited when its turn comes, by

   for v := 1 to n do
       mark[v] := unvisited;
   for v := 1 to n do
       if mark[v] = unvisited then
           dfs( v )
Note that Fig. 6.24 is a template to which we shall attach other actions later, as we apply depth-first search. The code in Fig. 6.24 doesn't do anything but set the mark array.

Analysis of Depth-First Search

All the calls to dfs in the depth-first search of a graph with e arcs and n � e vertices take O(e) time. To see why, observe that on no vertex is dfs called more than once, because as soon as we call dfs(v) we set mark[v] to visited at line (1), and we never call dfs on a vertex that previously had its mark set to visited. Thus, the total time spent at lines (2)-(3) going down the adjacency lists is proportional to the sum of the lengths of those lists, that is, O(e). Thus, assuming n�e, the total time spent on the depth- first search of an entire graph is O(e), which is, to within a constant factor, the time needed merely to "look at" each arc.

          procedure dfs ( v: vertex );
             var
                 w: vertex;
             begin
(1)              mark[v]: = visited;
(2)              for each vertex w on L[v] do
(3)                  if mark[w] = unvisited then
(4)                      dfs(w)
             end; { dfs }

Fig. 6.24. Depth-first search.

Example 6.13. Assume the procedure dfs(v) is applied to the directed graph of Fig. 6.25 with v = A. The algorithm marks A visited and selects vertex B from the adjacency list of vertex A. Since B is unvisited, the search continues by calling dfs(B). The algorithm now marks B visited and selects the first vertex from the adjacency list for vertex B. Depending on the order of the vertices on the adjacency list of B the search will go to C or D next.

Assuming that C appears ahead of D, dfs(C) is invoked. Vertex A is on the adjacency list of C. However, A is already visited at this point so the

Fig. 6.25. Directed graph.

search remains at C. Since all vertices on the adjacency list at C have now been exhausted, the search returns to B, from which the search proceeds to D. Vertices A and C on the adjacency list of D were already visited, so the search returns to B and then to A.

At this point the original call of dfs(A) is complete. However, the digraph has not been entirely searched; vertices E, F and G are still unvisited. To complete the search, we can call dfs(E).

The Depth-first Spanning Forest

During a depth-first traversal of a directed graph, certain arcs, when traversed, lead to unvisited vertices. The arcs leading to new vertices are called tree arcs and they form a depth-first spanning forest for the given digraph. The solid arcs in Fig. 6.26 form a depth-first spanning forest for the digraph of Fig. 6.25. Note that the tree arcs must indeed form a forest, since a vertex cannot be unvisited when two different arcs to it are traversed.

In addition to the tree arcs, there are three other types of arcs defined by a depth-first search of a directed graph. These are called back arcs, forward arcs, and cross arcs. An arc such as C � A is called a back arc, since it goes from a vertex to one of its ancestors in the spanning forest. Note that an arc from a vertex to itself is a back arc. A nontree arc that goes from a vertex to a proper descendant is called a forward arc. There are no forward arcs in Fig. 6.26.

Arcs such as D � C or G � D, which go from a vertex to another vertex that is neither an ancestor nor a descendant, are called cross arcs. Observe that all cross arcs in Fig. 6.26 go from right to left, on the assumption that we add children to the tree in the order they were visited, from left to right, and that we add new trees to the forest from left to right. This pattern is not accidental. Had the arc G � D been the arc D � G, then G would have been unvisited when the search at D was in progress, and thus on encountering arc D � G, vertex G would have been made a descendant of D, and D � G would have become a tree arc.

Fig. 6.26. Depth-first spanning forest for Fig. 6.25.

How do we distinguish among the four types of arcs? Clearly tree arcs are special since they lead to unvisited vertices during the depth-first search. Suppose we number the vertices of a directed graph in the order in which we first mark them visited during a depth-first search. That is, we may assign to an array

     dfnumber [v] := count;
    count : = count + 1;
after line (1) of Fig. 6.24. Let us call this the depth-first numbering of a directed graph; notice how depth-first numbering generalizes the preorder numbering introduced in Section 3.1.

All descendants of a vertex v are assigned depth-first search numbers greater than or equal to the number assigned v. In fact, w is a descendant of v if and only if dfnumber(v) � dfnumber(w) � dfnumber(v) + number of descendants of v. Thus, forward arcs go from low-numbered to high-numbered vertices and back arcs go from high-numbered to low-numbered vertices.

All cross arcs go from high-numbered vertices to low-numbered vertices. To see this, suppose that x � y is an arc and dfnumber(x) � dfnumber(y). Thus x is visited before y. Every vertex visited between the time dfs(x) is first invoked and the time dfs(x) is complete becomes a descendant of x in the depth-first spanning forest. If y is unvisited at the time arc x � y is explored, x � y becomes a tree arc. Otherwise, x � y is a forward arc. Thus there can be no cross arc x � y with dfnumber (x) � dfnumber (y).

In the next two sections we shall show how depth- first search can be used in solving various graph problems.

6.6 Directed Acyclic Graphs

A directed acyclic graph, or dag for short, is a directed graph with no cycles. Measured in terms of the relationships they can represent, dags are more general than trees but less general than arbitrary directed graphs. Figure 6.27 gives an example of a tree, a dag, and a directed graph with a cycle.

Fig. 6.27. Three directed graphs.

Among other things, dags are useful in representing the syntactic structure of arithmetic expressions with common subexpressions. For example, Fig. 6.28 shows a dag for the expression

((a+b)*c + ((a+b)+e) * (e+f)) * ((a+b)*c)

The terms a+b and (a+b)*c are shared common subexpressions that are represented by vertices with more than one incoming arc.

Dags are also useful in representing partial orders. A partial order R on a set S is a binary relation such that

  1. for all a in S, a R a is false (R is irreflexive)
  2. for all a, b, c in S, if a R b and b R c, then a R c (R is transitive)

Two natural examples of partial orders are the "less than" (<) relation on integers, and the relation of proper containment (�) on sets.

Example 6.14. Let S = {1, 2, 3} and let P(S) be the power set of S, that is, the set of all subsets of S. P(S) = {Ø, {1}, {2}, {3}, {1,2}, {1,3}, {2,3}, {1,2,3}}. � is a partial order on P(S). Certainly, A � A is false for any set A (irreflexivity), and if A � B and B � C, then A � C (transitivity).

Dags can be used to portray partial orders graphically. To begin, we can view a relation R as a set of pairs (arcs) such that (a, b) is in the set if and only if a R b is true. If R is a partial order on a set S, then the directed graph G = (S ,R) is a dag. Conversely, suppose G = (S ,R) is a dag and R+ is the relation defined by a R+ b if and only if there is a path of length one or

Fig. 6.28. Dag for arithmetic expression.

more from a to b. (R+ is the transitive closure of the relation R.) Then, R+ is a partial order on S.

Example 6.15. Figure 6.29 shows a dag (P(S), R), where S = {1, 2 ,3}. The relation R+ is proper containment on the power set P(S).

Fig. 6.29. Dag of proper containments.

Test for Acyclicity

Suppose we are given a directed graph G = (V, E), and we wish to determine whether G is acyclic, that is, whether G has no cycles. Depth-first search can be used to answer this question. If a back arc is encountered during a depth-first search of G, then clearly the graph has a cycle. Conversely, if a directed graph has a cycle, then a back arc will always be encountered in any depth-first search of the graph.

To see this fact, suppose G is cyclic. If we do a depth-first search of G, there will be one vertex v having the lowest depth-first search number of any vertex on a cycle. Consider an arc u � v on some cycle containing v. Since u is on the cycle, u must be a descendant of v in the depth-first spanning forest. Thus, u � v cannot be a cross arc. Since the depth- first number of u is greater than the depth-first number of v, u � v cannot be a tree arc or a forward arc. Thus, u � v must be a back arc, as illustrated in Fig. 6.30.

Fig. 6.30. Every cycle contains a back arc.

Topological Sort

A large project is often divided into a collection of smaller tasks, some of which have to be performed in certain specified orders so that we may complete the entire project. For example, a university curriculum may have courses that require other courses as prerequisites. Dags can be used to model such situations naturally. For example, we could have an arc from course C to course D if C is a prerequisite for D.

Example 6.16. Figure 6.31 shows a dag giving the prerequisite structure on five courses. Course C3, for example, requires courses C1 and C2 as prerequisites.

Fig. 6.31. Dag of prerequisites.

Topological sort is a process of assigning a linear ordering to the vertices of a dag so that if there is an arc from vertex i to vertex j, then i appears before j in the linear ordering. For example, C1, C2, C3, C4, C5 is a topological sort of the dag in Fig. 6.31. Taking the courses in this sequence would satisfy the prerequisite structure given in Fig. 6.31.

Topological sort can be easily accomplished by adding a print statement after line (4) to the depth-first search procedure in Fig. 6.24:

    procedure topsort ( v: vertex );

         { print vertices accessible from v in reverse topological order }

         var

           w: vertex;

         begin

           mark[v] := visited;

           for each vertex w on L[v] do

            if mark[w] = unvisited then

                topsort(w) ;

           writeln(v)

         end; { topsort }

When topsort finishes searching all vertices adjacent to a given vertex x, it prints x. The effect of calling topsort (v) is to print in a reverse topological order all vertices of a dag accessible from v by a path in the dag.

This technique works because there are no back arcs in a dag. Consider what happens when depth-first search leaves a vertex x for the last time. The only arcs emanating from v are tree, forward, and cross arcs. But all these arcs are directed towards vertices that have already been completely visited by the search and therefore precede x in the order being constructed.

6.7 Strong Components

A strongly connected component of a directed graph is a maximal set of vertices in which there is a path from any one vertex in the set to any other vertex in the set. Depth-first search can be used to determine strongly connected components of a directed graph efficiently.

Let G = (V, E) be a directed graph. We can partition V into equivalence classes Vi, 1�i�r, such that vertices v and w are equivalent if and only if there is a path from v to w and a path from w to v. Let Ei, 1�i�r, be the set of arcs with head and tail in Vi. The graphs Gi = (Vi, Ei) are called the strongly connected components (or just strong components) of G. A directed graph with only one strong component is said to be strongly connected.

Example 6.17. Figure 6.32 illustrates a directed graph with the two strong components shown in Fig. 6.33.

Fig. 6.32. Directed graph.

Fig. 6.33. The strong components of the digraph of Fig. 6.32.

Note that every vertex of a directed graph G is in some strong component, but that certain arcs may not be in any component. Such arcs, called cross-component arcs, go from a vertex in one component to a vertex in another. We can represent the interconnections among the components by constructing a reduced graph for G. The vertices of the reduced graph are the strongly connected components of G. There is an arc from vertex C to a different vertex C' of the reduced graph if there is an arc in G from some vertex in the component C to some vertex in component C'. The reduced graph is always a dag, because if there were a cycle, then all the components in the cycle would really be one strong component, meaning that we didn't compute strong components properly. Figure 6.34 shows the reduced graph for the digraph in Figure 6.32.

We shall now present an algorithm to find the strongly connected

Fig. 6.34. Reduced graph.

components of a given directed graph G.

  1. Perform a depth-first search of G and number the vertices in order of completion of the recursive calls; i.e., assign a number to vertex v after line (4) of Fig. 6.24.

  2. Construct a new directed graph Gr by reversing the direction of every arc in G.

  3. Perform a depth-first search on Gr starting the search from the highest-numbered vertex according to the numbering assigned at step (1). If the depth-first search does not reach all vertices, start the next depth-first search from the highest-numbered remaining vertex.

  4. Each tree in the resulting spanning forest is a strongly connected component of G.

Example 6.18. Let us apply this algorithm to the directed graph of Fig. 6.32, starting at a and progressing first to b. After step (1) we number the vertices as shown in Fig. 6.35. Reversing the direction of the arcs we obtain the directed graph Gr in Fig. 6.36.

Performing the depth-first search on Gr we obtain the depth-first spanning forest shown in Fig. 6.37. We begin with a as a root, because a has the highest number. From a we can only reach c and then b. The next tree has root d, since that is the highest numbered (and only) remaining vertex. Each tree in this forest forms a strongly connected component of the original directed graph.

We have claimed that the vertices of a strongly connected component correspond precisely to the vertices of a tree in the spanning forest of the second depth-first search. To see why, observe that if v and w are vertices in the same strongly connected component, then there are paths in G from v to w and from w to v. Thus there are also paths from v to w and from w to v in Gr.

Suppose that in the depth-first search of Gr, we begin a search at some root x and reach either v or w. Since v and w are reachable from each other,

Fig. 6.35. After step 1.

Fig. 6.36. Gr.

Fig. 6.37.Depth-first spanning forest for Gr.

both v and w will end up in the spanning tree with root x.

Now suppose v and w are in the same spanning tree of the depth-first spanning forest of Gr. We must show that v and w are in the same strongly connected component. Let x be the root of the spanning tree containing v and w. Since v is a descendant of x, there exists a path in Gr from x to v. Thus there exists a path in G from v to x.

In the construction of the depth-first spanning forest of Gr, vertex v was still unvisited when the depth-first search at x was initiated. Thus x has a higher number than v, so in the depth-first search of G, the recursive call at v terminated before the recursive call at x did. But in the depth-first search of G, the search at v could not have started before x, since the path in G from v to x would then imply that the search at x would start and end before the search at v ended.

We conclude that in the search of G, v is visited during the search of x and hence v is a descendant of x in the first depth-first spanning forest for G. Thus there exists a path from x to v in G. Therefore x and v are in the same strongly connected component. An identical argument shows that x and w are in the same strongly connected component and hence v and w are in the same strongly connected component, as shown by the path from v to x to w and the path from w to x to v.

Exercises

  1. adjacency matrices,

  2. linked adjacency lists, and

  3. adjacency lists represented as in Fig. 6.5.
  1. Use the algorithm Dijkstra to find the shortest paths from a to the other vertices.

  2. Use the algorithm Floyd to find the shortest distances between all pairs of points. Also construct the matrix P that lets us recover the shortest paths.
6.1 Represent the digraph of Fig. 6.38
  1. by an adjacency matrix giving arc costs.

  2. by a linked adjacency list with arc costs indicated.

Fig. 6.38. Directed graph with arc costs.
6.2 | Describe a mathematical model for the following scheduling problem. Given tasks T1, T2, . . . , Tn, which require times t1, t2, . . . , tn to complete, and a set of constraints, each of the form "Ti must be completed prior to the start of Tj," find the minimum time necessary to complete all tasks.
6.3 | Implement the operations FIRST, NEXT, and VERTEX for digraphs represented by
6.4 | In the digraph of Fig. 6.38
6.5 | Write a complete program for Dijkstra's algorithm using a partially ordered tree as a priority queue and linked adjacency lists.
*6.6 | Show that the program Dijkstra does not work correctly if arc costs can be negative.
**6.7 | Show that the program Floyd still works if some of the arcs have negative cost but no cycle has negative cost.
6.8 | Assuming the order of the vertices is a, b , . . . , f in Fig. 6.38, construct a depth-first spanning forest; indicate the tree, forward, back and cross arcs, and indicate the depth-first numbering of the vertices.
*6.9 | Suppose we are given a depth-first spanning forest, and we list in postorder each of the spanning trees (trees composed of spanning edges), from the leftmost to the rightmost. Show that this order is the same as the order in which the calls of dfs ended when the spanning forest was constructed.
6.10 | A root of a dag is a vertex r such that every vertex of the dag can be reached by a directed path from r. Write a program to determine whether a dag is rooted.
*6.11 | Consider a dag with e arcs and with two distinguished vertices s and t. Construct an O (e) algorithm to find a maximal set of vertex disjoint paths from s to t. By maximal we mean no additional path may be added, but it does not mean that it is the largest size such set.
6.12 | Construct an algorithm to convert an expression tree with operators + and * into a dag by sharing common subexpressions. What is the time complexity of your algorithm?
6.13 | Construct an algorithm for evaluating an arithmetic expression represented as a dag.
6.14 | Write a program to find the longest path in a dag. What is the time complexity of your program?
6.15 | Find the strong components of Fig. 6.38.
*6.16 | Prove that the reduced graph of the strong components of Section 6.7 must be a dag.
6.17 | Draw the first spanning forest, the reverse graph, and the second spanning forest developed by the strong components algorithm applied to the digraph of Fig. 6.38.
6.18 | Implement the strong components algorithm discussed in Section 6.7.
*6.19 | Show that the strong components algorithm takes time O (e) on a directed graph of e arcs and n vertices, assuming n � e.
*6.20 | Write a program that takes as input a digraph and two of its vertices. The program is to print all simple paths from one vertex to the other. What is the time complexity of your program?
*6.21 | A transitive reduction of a directed graph G = (V, E) is any graph G' with the same vertices but with as few arcs as possible, such that the transitive closure of G' is the same as the transitive closure of G. how that if G is a dag, then the transitive reduction of G is unique.
*6.22 | Write a program to compute the transitive reduction of a digraph. What is the time complexity of your program?
*6.23 | G' = (V, E') is called a minimal equivalent digraph for a digraph G = (V, E) if E' is a smallest subset of E and the transitive closure of both G and G' are the same. Show that if G is acyclic, then there is only one minimal equivalent digraph, namely, the transitive reduction.
*6.24 | Write a program to find a minimal equivalent digraph for a given digraph. What is the time complexity of your program?
*6.25 | Write a program to find the longest simple path from a given vertex of a digraph. What is the time complexity of your program?

Bibliographic Notes

Berge [1958] and Harary [1969] are two good sources for additional material on graph theory. Some books treating graph algorithms are Deo [1975], Even [1980], and Tarjan [1983].

The single-source shortest paths algorithm in Section 6.3 is from Dijkstra [1959]. The all-pairs shortest paths algorithm is from Floyd [1962] and the transitive closure algorithm is from Warshall [1962]. Johnson [1977] discusses efficient algorithms for finding shortest paths in sparse graphs. Knuth [1968] contains additional material on topological sort.

The strong components algorithm of Section 6.7 is similar to one suggested by R. Kosaraju in 1978 (unpublished), and to one published by Sharir [1981]. Tarjan [1972] contains another strong components algorithm that needs only one depth-first search traversal.

Coffman [1976] contains many examples of how digraphs can be used to model scheduling problems as in Exercise 6.2. Aho, Garey, and Ullman [1972] show that the transitive reduction of a dag is unique, and that computing the transitive reduction of a digraph is computationally equivalent to computing the transitive closure (Exercises 6.21 and 6.22). Finding the minimal equivalent digraph (Exercises 6.23 and 6.24), on the other hand, appears to be computationally much harder; this problem is NP-complete [Sahni (1974)].

† This is another manifestation of the old Pascal problem of doing insertion and deletion at arbitrary positions of singly linked lists.

† One might expect that a more natural problem is to find the shortest path from the source to one particular destination vertex. However, that problem appears just as hard in general as the single-source shortest paths problem (except that by luck we may find the path to the destination before some of the other vertices and thereby terminate the algorithm a bit earlier than if we wanted paths to all the vertices).

‡ We might assume that an undirected graph would be used here, since the label on arcs v � w and w � v would be the same. However, in fact, travel times are different in different directions because of prevailing winds. Anyway, assuming that labels on v � w and w � v are identical doesn't seem to help solve the problem.

Table of Contents Go to Chapter 7


Data Structures and Algorithms: CHAPTER 7: Undirected Graphs

 Return to Table of Contents

Undirected Graphs

An undirected graph G = (V, E) consists of a finite set of vertices V and a set of edges E. It differs from a directed graph in that each edge in E is an unordered pair of vertices.† If (v, w) is an undirected edge, then (v, w) = (w, v). Hereafter we refer to an undirected graph as simply a graph.

Graphs are used in many different disciplines to model a symmetric relationship between objects. The objects are represented by the vertices of the graph and two objects are connected by an edge if the objects are related. In this chapter we present several data structures that can be used to represent graphs. We then present algorithms for three typical problems involving undirected graphs: constructing minimal spanning trees, biconnected components, and maximal matchings.

7.1 Definitions

Much of the terminology for directed graphs is also applicable to undirected graphs. For example, vertices v and w are adjacent if (v, w) is an edge [or, equivalently, if (w, v) is an edge]. We say the edge (v, w) is incident upon vertices v and w.

A path is a sequence of vertices v1, v2, . . . , vn such that (vi, vi+1) is an edge for 1 � i < n. A path is simple if all vertices on the path are distinct, with the exception that v1 and vn may be the same. The length of the path is n-1, the number of edges along the path. We say the path v1, v2, . . . , vn connects v1 and vn. A graph is connected if every pair of its vertices is connected.

Let G = (V, E) be a graph with vertex set V and edge set E. A subgraph of G is a graph G' = (V', E') where

  1. V' is a subset of V.
  2. E' consists of edges (v, w) in E such that both v and w are in V'.

If E' consists of all edges (v, w) in E, such that both v and w are in V', then G' is called an induced subgraph of G.

Example 7.1. In Fig. 7.1(a) we see a graph G = (V, E) with V = {a, b, c, d} and E = {(a, b), (a, d), (b, c), (b, d), (c, d)}. In Fig. 7.1(b) we see one of its induced subgraphs, the one defined by the set of vertices {a, b, c} and all the edges in Fig. 7.1(a) that are not incident upon vertex d.

Fig. 7.1. A graph and one of its subgraphs.

A connected component of a graph G is a maximal connected induced subgraph, that is, a connected induced subgraph that is not itself a proper subgraph of any other connected subgraph of G.

Example 7.2 Figure 7.1 is a connected graph. It has only one connected component, namely itself. Figure 7.2 is a graph with two connected components.

Fig. 7.2. An unconnected graph.

A (simple) cycle in a graph is a (simple) path of length three or more that connects a vertex to itself. We do not consider paths of the form v (path of length 0), v, v (path of length 1), or v, w, v (path of length 2) to be cycles. A graph is cyclic if it contains at least one cycle. A connected, acyclic graph is sometimes called a free tree. Figure 7.2 shows a graph consisting of two connected components where each connected component is a free tree. A free tree can be made into an ordinary tree if we pick any vertex we wish as the root and orient each edge from the root.

Free trees have two important properties, which we shall use in the next section.

  1. Every free tree with n � 1 vertices contains exactly n-1 edges.
  2. If we add any edge to a free tree, we get a cycle.

We can prove (1) by induction on n, or what is equivalent, by an argument concerning the "smallest counterexample." Suppose G = (V, E) is a counterexample to (1) with the fewest vertices, say n vertices. Now n cannot be 1, because the only free tree on one vertex has zero edges, and (1) is satisfied. Therefore, n must be greater than 1.

We now claim that in the free tree there must be some vertex with exactly one incident edge. In proof, no vertex can have zero incident edges, or G would not be connected. Suppose every vertex has at least two edges incident. Then, start at some vertex v1, and follow any edge from v1. At each step, leave a vertex by a different edge from the one used to enter it, thereby forming a path v1, v2, v3, . . ..

Since there are only a finite number of vertices in V, all vertices on this path cannot be distinct; eventually, we find vi = vj for some i < j. We cannot have i = j-1 because there are no loops from a vertex to itself, and we cannot have i = j-2 or else we entered and left vertex vi+1 on the same edge. Thus, i � j-3, and we have a cycle vi, vi+1, . . . , vj = vi. Thus, we have contradicted the hypothesis that G had no vertex with only one edge incident, and therefore conclude that such a vertex v with edge (v, w) exists.

Now consider the graph G' formed by deleting vertex v and edge (v, w) from G. G' cannot contradict (1), because if it did, it would be a smaller counterexample than G. Therefore, G' has n-1 vertices and n-2 edges. But G has one more edge and one more vertex than G', so G has n-1 edges, proving that G does indeed satisfy (1). Since there is no smallest counterexample to (1), we conclude there can be no counterexample at all, so (1) is true.

Now we can easily prove statement (2), that adding an edge to a free tree forms a cycle. If not, the result of adding the edge to a free tree of n vertices would be a graph with n vertices and n edges. This graph would still be connected, and we supposed that adding the edge left the graph acyclic. Thus we would have a free tree whose vertex and edge count did not satisfy condition (1).

Methods of Representation

The methods of representing directed graphs can be used to represent undirected graphs. One simply represents an undirected edge between v and w by two directed edges, one from v to w and the other from w to v.

Example 7.3. The adjacency matrix and adjacency list representations for the graph of Fig. 7.1(a) are shown in Fig. 7.3.

Clearly, the adjacency matrix for a graph is symmetric. In the adjacency list representation if (i, j) is an edge, then vertex j is on the list for vertex i and vertex i is on the list for vertex j.

Fig. 7.3. Representations.

7.2 Minimum-Cost Spanning Trees

Suppose G = (V, E) is a connected graph in which each edge (u, v) in E has a cost c(u, v) attached to it. A spanning tree for G is a free tree that connects all the vertices in V. The cost of a spanning tree is the sum of the costs of the edges in the tree. In this section we shall show how to find a minimum-cost spanning tree for G.

Example 7.4. Figure 7.4 shows a weighted graph and its minimum-cost spanning tree.

A typical application for minimum-cost spanning trees occurs in the design of communications networks. The vertices of a graph represent cities and the edges possible communications links between the cities. The cost associated with an edge represents the cost of selecting that link for the network. A minimum-cost spanning tree represents a communications network that connects all the cities at minimal cost.

Fig. 7.4. A graph and spanning tree.

The MST Property

There are several different ways to construct a minimum-cost spanning tree. Many of these methods use the following property of minimum-cost spanning trees, which we call the MST property. Let G = (V, E) be a connected graph with a cost function defined on the edges. Let U be some proper subset of the set of vertices V. If (u, v) is an edge of lowest cost such that u � U and v � V-U, then there is a minimum-cost spanning tree that includes (u, v) as an edge.

The proof that every minimum-cost spanning tree satisfies the MST property is not hard. Suppose to the contrary that there is no minimum-cost spanning tree for G that includes (u, v). Let T be any minimum-cost spanning tree for G. Adding (u, v) to T must introduce a cycle, since T is a free tree and therefore satisfies property (2) for free trees. This cycle involves edge (u, v). Thus, there must be another edge (u', v') in T such that u' � U and v' � V-U, as illustrated in Fig. 7.5. If not, there would be no way for the cycle to get from u to v without following the edge (u, v) a second time.

Deleting the edge (u', v') breaks the cycle and yields a spanning tree T' whose cost is certainly no higher than the cost of T since by assumption c(u, v) � c(u', v'). Thus, T' contradicts our assumption that there is no minimum-cost spanning tree that includes (u, v).

Prim's Algorithm

There are two popular techniques that exploit the MST property to construct a minimum-cost spanning tree from a weighted graph G = (V, E). One such method is known as Prim's algorithm. Suppose V = {1, 2, . . . , n}. Prim's algorithm begins with a set U initialized to {1}. It then "grows" a spanning tree, one edge at a time. At each step, it finds a shortest edge (u, v) that connects U and V-U and then adds v, the vertex in V-U, to U. It repeats this

Fig. 7.5. Resulting cycle.

step until U = V. The algorithm is summarized in Fig. 7.6 and the sequence of edges added to T for the graph of Fig. 7.4(a) is shown in Fig. 7.7.

    procedure Prim ( G: graph; var T: set of edges );
           { Prim constructs a minimum-cost spanning tree T for G }
           var
                U: set of vertices;
                u, v: vertex;
       begin
               T:= Ø;
               U := {1};
               while U � V do begin
                     let (u, v) be a lowest cost edge such that
                          u is in U and v is in V-U;
                  T := T � {(u, v)};
                  U := U � {v}
               end
          end; { Prim }

Fig. 7.6. Sketch of Prim's algorithm.

One simple way to find the lowest-cost edge between U and V-U at each step is to maintain two arrays. One array CLOSEST[i] gives the vertex in U that is currently closest to vertex i in V-U. The other array LOWCOST[i] gives the cost of the edge (i, CLOSEST[i]).

At each step we can scan LOWCOST to find the vertex, say k, in V-U that is closest to U. We print the edge (k, CLOSEST[k]). We then update the LOWCOST and CLOSEST arrays, taking into account the fact that k has been

Fig. 7.7. Seqeunces of edges added by Prim's algorithm.

added to U. A Pascal version of this algorithm is given in Fig. 7.8. We assume C is an n x n array such that C[i, j] is the cost of edge (i, j). If edge (i, j) does not exist, we assume C[i, j] is some appropriate large value.

Whenever we find another vertex k for the spanning tree, we make LOWCOST[k] be infinity, a very large value, so this vertex will no longer be considered in subsequent passes for inclusion in U. The value infinity is greater than the cost of any edge or the cost associated with a missing edge.

The time complexity of Prim's algorithm is O(n2), since we make n-1 iterations of the loop of lines (4)-(16) and each iteration of the loop takes O(n) time, due to the inner loops of lines (7)-(10) and (13)-(16). As n gets large the performance of this algorithm may become unsatisfactory. We now give another algorithm due to Kruskal for finding minimum-cost spanning trees whose performance is at most O(eloge), where e is the number of edges in the given graph. If e is much less than n2, Kruskal's algorithm is superior, although if e is about n2, we would prefer Prim's algorithm.

       procedure Prim ( C: array[l..n, 1..n] of
real );
           { Prim prints the edges of a minimum-cost spanning tree for a graph
                with vertices {1, 2, . . . , n} and cost matrix C on edges }
           var
                LOWCOST: array[1..n] of real;
                CLOSEST: array[1..n] of integer;
                i, j, k, min; integer;
                { i and j are indices. During a scan of the LOWCOST array,
                     k is the index of the closest vertex found so far, and
                     min = LOWCOST[k] }
           begin
 (1)            for i := 2 to n do begin
                     { initialize with only vertex 1 in the set U }
 (2)           	     LOWCOST[i] := C[1, i];
 (3)                 CLOSEST[i] := 1
                end;
 (4)            for i := 2 to n do begin
                  { find the closest vertex k outside of U to
                  some vertex in U }
 (5)                 min := LOWCOST[2];
 (6)                 k := 2;
 (7)                 for j := 3 to n do
 (8)                      if LOWCOST[j] < min then begin
 (9)                          min := LOWCOST[j];
(10)                          k := j
                       end;
(11)                 writeln(k, CLOSEST[k]); { print edge }
(12)                 LOWCOST[k] := infinity; { k is added to U }
(13)                 for j := 2 to n do { adjust costs to U }
(14)                      if (C[k, j] < LOWCOST[j]) and
                                (LOWCOST[j] < infinity) then begin
(15)                          LOWCOST[j] := C[k, j];
(16)                          CLOSEST[j] := k
                      end
                end
           end; { Prim }

Fig. 7.8. Prim's algorithm.

Kruskal's Algorithm

Suppose again we are given a connected graph G = (V, E), with V = {1, 2, . . . , n} and a cost function c defined on the edges of E. Another way to construct a minimum-cost spanning tree for G is to start with a graph T = (V, Ø) consisting only of the n vertices of G and having no edges. Each vertex is therefore in a connected component by itself. As the algorithm proceeds, we shall always have a collection of connected components, and for each component we shall have selected edges that form a spanning tree.

To build progressively larger components, we examine edges from E, in order of increasing cost. If the edge connects two vertices in two different connected components, then we add the edge to T. If the edge connects two vertices in the same component, then we discard the edge, since it would cause a cycle if we added it to the spanning tree for that connected component. When all vertices are in one component, T is a minimum-cost spanning tree for G.

Example 7.5. Consider the weighted graph of Fig. 7.4(a). The sequence of edges added to T is shown in Fig. 7.9. The edges of cost 1, 2, 3, and 4 are considered first, and all are accepted, since none of them causes a cycle. The edges (1, 4) and (3, 4) of cost 5 cannot be accepted, because they connect vertices in the same component in Fig. 7.9(d), and therefore would complete a cycle. However, the remaining edge of cost 5, namely (2, 3), does not create a cycle. Once it is accepted, we are done.

We can implement this algorithm using sets and set operations discussed in Chapters 4 and 5. First, we need a set consisting of the edges in E. We then apply the DELETEMIN operator repeatedly to this set to select edges in order of increasing cost. The set of edges therefore forms a priority queue, and a partially ordered tree is an appropriate data structure to use here.

We also need to maintain a set of connected components C. The operations we apply to it are:

  1. MERGE(A, B, C) to merge the components A and B in C and to call the result either A or B arbitrarily.†
  2. FIND(v, C) to return the name of the component of C of which vertex v is a member. This operation will be used to determine whether the two vertices of an edge are in the same or different components.
  3. INITIAL(A, v, C) to make A the name of a component in C containing only vertex v initially.

These are the operations of the MERGE-FIND ADT called MFSET, which we encountered in Section 5.5. A sketch of a program called Kruskal to find a minimum-cost spanning tree using these operations is shown in Fig. 7.10.

We can use the techniques of Section 5.5 to implement the operations used in this program. The running time of this program is dependent on two factors. If there are e edges, it takes O(eloge) time to insert the edges into the priority queue.‡ In each iteration of the while-loop, finding the least cost

Fig. 7.9. Sequence of edges added by Kruskal's algorithm.

edge in edges takes O(log e) time. Thus, the priority queue operations take O(e log e) time in the worst case. The total time required to perform the MERGE and FIND operations depends on the method used to implement the MFSET. As shown in Section 5.5, there are O(e log e) and O(e a (e)) methods. In either case, Kruskal's algorithm can be implemented to run in O(e log e) time.

7.3 Traversals

In a number of graph problems, we need to visit the vertices of a graph systematically. Depth-first search and breadth-first search, the subjects of this section, are two important techniques for doing this. Both techniques can be used to determine efficiently all vertices that are connected to a given vertex.

    procedure Kruskal ( V: SET of vertex;
           E: SET of edges;
           var T: SET of edges );
       var
               ncomp: integer; { current number of components }
                  edges: PRIORITYQUEUE; { the set of edges }
               components: MFSET; { the set V grouped into
             a MERGE-FIND set of components }
           u, v: vertex;
           e: edge;
            nextcomp: integer; { name for new component }
           ucomp, vcomp: integer; { component names }

           begin
           MAKENULL(T);
           MAKENULL(edges);
           nextcomp := 0;
           ncomp := number of members of V;
           for v in V do begin { initialize a
             component to contain one vertex of V }
         nextcomp := nextcomp + 1;
         INITIAL( nextcomp, v, components)
           end;
           for e in E do { initialize priority queue of edges }
             INSERT(e, edges);
              while ncomp > 1 do begin { consider next edge }
             e := DELETEMIN(edges);
         let e = (u, v);
           ucomp := FIND(u, components);
         vcomp := FIND(v, components);
         if ucomp <> vcomp then begin
             { e connects two different components }
             MERGE (ucomp, vcomp, components);
             ncomp := ncomp - 1;
             INSERT(e, T)
         end
       end
        end; { Kruskal }

Fig. 7.10. Kruskal's algorithm.

Depth-First Search

Recall from Section 6.5 the algorithm dfs for searching a directed graph. The same algorithm can be used to search undirected graphs, since the undirected edge (v, w) may be thought of as the pair of directed edges v � w and w � v.

In fact, the depth-first spanning forests constructed for undirected graphs are even simpler than for digraphs. We should first note that each tree in the forest is one connected component of the graph, so if a graph is connected, it has only one tree in its depth-first spanning forest. Second, for digraphs we identified four kinds of arcs: tree, forward, back, and cross. For undirected graphs there are only two kinds: tree edges and back edges.

Because there is no distinction between forward edges and backward edges for undirected graphs, we elect to refer to them all as back edges. In an undirected graph, there can be no cross edges, that is, edges (v, w) where v is neither an ancestor nor descendant of w in the spanning tree. Suppose there were. Then let v be a vertex that is reached before w in the search. The call to dfs(v) cannot end until w has been searched, so w is entered into the tree as some descendant of v. Similarly, if dfs(w) is called before dfs(v), then v becomes a descendant of w.

As a result, during a depth-first search of an undirected graph G, all edges become either

  1. tree edges, those edges (v, w) such that dfs(v) directly calls dfs(w) or vice versa, or
  2. back edges, those edges (v, w) such that neither dfs(v) nor dfs(w) called the other directly, but one called the other indirectly (i.e., dfs(w) calls dfs (x), which calls dfs (v), so w is an ancestor of v).

Example 7.6. Consider the connected graph G in Fig. 7.11(a). A depth-first spanning tree T resulting from a depth-first search of G is shown in Fig. 7.11(b). We assume the search began at vertex a, and we have adopted the convention of showing tree edges solid and back edges dashed. The tree has been drawn with the root on the top and the children of each vertex have been drawn in the left-to-right order in which they were first visited in the procedure of dfs.

To follow a few steps of the search, the procedure dfs(a) calls dfs(b) and adds edge (a, b) to T since b is not yet visited. At b, dfs calls dfs(d) and adds edge (b, d) to T. At d, dfs calls dfs(e) and adds edge (d, e) to T. At e, vertices a, b, and d are marked visited so dfs(e) returns without adding any edges to T. At d, dfs now sees that vertices a and b are marked visited so dfs(d) returns without adding any more edges to T. At b, dfs now sees that the remaining adjacent vertices a and e are marked visited, so dfs(b) returns. The search then continues to c, f, and g.

Fig. 7.11. A graph and its depth-first search.

Breadth-First Search

Another systematic way of visiting the vertices is called breadth-first search. The approach is called "breadth-first" because from each vertex v that we visit we search as broadly as possible by next visiting all the vertices adjacent to v. We can also apply this strategy of search to directed graphs.

As for depth-first search, we can build a spanning forest when we perform a breadth-first search. In this case, we consider edge (x, y) a tree edge if vertex y is first visited from vertex x in the inner loop of the search procedure bfs of Fig. 7.12.

It turns out that for the breadth-first search of an undirected graph, every edge that is not a tree edge is a cross edge, that is, it connects two vertices neither of which is an ancestor of the other.

The breadth-first search algorithm given in Fig. 7.12 inserts the tree edges into a set T, which we assume is initially empty. Every entry in the array mark is assumed to be initialized to the value unvisited; Figure 7.12 works on one connected component. If the graph is not connected, bfs must be called on a vertex of each component. Note that in a breadth-first search we must mark a vertex visited before enqueuing it, to avoid placing it on the queue more than once.

Example 7.7. The breadth-first spanning tree for the graph G in Fig. 7.11(a) is shown in Fig. 7.13. We assume the search began at vertex a. As before, we have shown tree edges solid and other edges dashed. We have also drawn the tree with the root at the top and the children in the left-to-right order in which they were first visited.

The time complexity of breadth-first search is the same as that of depth-

   procedure bfs ( v );
       { bfs visits all vertices connected to v using breadth-first search }
       var
          Q: QUEUE of vertex;
          x, y: vertex;
       begin
          mark[v] := visited;
           ENQUEUE(v, Q);
           while not EMPTY(Q) do begin
              x := FRONT(Q);
              DEQUEUE(Q);
              for each vertex y adjacent to x do
                 if mark[y] = unvisited then begin
                     mark[y] := visited;
                     ENQUEUE(y, Q);
                    INSERT((x, y), T)
                 end
           end
       end; { bfs }

Fig. 7.12. Breadth-first search.

Fig. 7.13. Breadth-first search of G.

first search. Each vertex visited is placed in the queue once, so the body of the while loop is executed once for each vertex. Each edge (x, y) is examined twice, once from x and once from y. Thus, if a graph has n vertices and e edges, the running time of bfs is O(max(n, e)) if we use an adjacency list representation for the edges. Since e � n is typical, we shall usually refer to the running time of breadth-first search as O(e), just as we did for depth-first search.

Depth-first search and breadth-first search can be used as frameworks around which to design efficient graph algorithms. For example, either method can be used to find the connected components of a graph, since the connected components are the trees of either spanning forest.

We can test for cycles using breadth-first search in O(n) time, where n is the number of vertices, independent of the number of edges. As we discussed in Section 7.1, any graph with n vertices and n or more edges must have a cycle. However, a graph could have n-1 or fewer edges and still have a cycle, if it had two or more connected components. One sure way to find the cycles is to build a breadth-first spanning forest. Then, every cross edge (v, w) must complete a simple cycle with the tree edges leading to v and w from their closest common ancestor, as shown in Fig. 7.14.

Fig. 7.14 A cycle found by breadth-first search.

7.4 Articulation Points and Biconnected Components

An articulation point of a graph is a vertex v such that when we remove v and all edges incident upon v, we break a connected component of the graph into two or more pieces. For example, the articulation points of Fig. 7.11(a) are a and c. If we delete a, the graph, which is one connected component, is divided into two triangles: {b, d, e} and {c, f, g}. If we delete c, we divide the graph into {a, b, d, e} and {f, g}. However, if we delete any one of the other vertices from the graph of Fig. 7.11(a), we do not split the connected component. A connected graph with no articulation points is said to be biconnected. Depth-first search is particularly useful in finding the biconnected components of a graph.

The problem of finding articulation points is the simplest of many important problems concerning the connectivity of graphs. As an example of applications of connectivity algorithms, we may represent a communication network as a graph in which the vertices are sites to be kept in communication with one another. A graph has connectivity k if the deletion of any k-1 vertices fails to disconnect the graph. For example, a graph has connectivity two or more if and only if it has no articulation points, that is, if and only if it is biconnected. The higher the connectivity of a graph, the more likely the graph is to survive the failure of some of its vertices, whether by failure of the processing units at the vertices or external attack.

We shall here give a simple depth-first search algorithm to find all the articulation points of a connected graph, and thereby test by their absence whether the graph is biconnected.

  1. Perform a depth-first search of the graph, computing dfnumber [v] for each vertex v as discussed in Section 6.5. In essence, dfnumber orders the vertices as in a preorder traversal of the depth-first spanning tree.

  2. For each vertex v, compute low [v], which is the smallest dfnumber of v or of any vertex w reachable from v by following down zero or more tree edges to a descendant x of v (x may be v) and then following a back edge (x, w). We compute low [v] for all vertices v by visiting the vertices in a postorder traversal. When we process v, we have computed low [y] for every child y of v. We take low [v] to be the minimum of
    1. dfnumber [v],

    2. dfnumber [z] for any vertex z for which there is a back edge (v, z) and

    3. low [y] for any child y of v.
  • Now we find the articulation points as follows.
    1. The root is an articulation point if and only if it has two or more children. Since there are no cross edges, deletion of the root must disconnect the subtrees rooted at its children, as a disconnects {b, d, e} from {c, f, g} in Fig. 7.11(b).
    2. A vertex v other than the root is an articulation point if and only if there is some child w of v such that low [w] � dfnumber [v]. In this case, v disconnects w and its descendants from the rest of the graph. Conversely, if low [w] < dfnumber [v], then there must be a way to get from w down the tree and back to a proper ancestor of v (the vertex whose dfnumber is low [w]), and therefore deletion of v does not disconnect w or its descendants from the rest of the graph.

    Example 7.8. dfnumber and low are computed for the graph of Fig. 7.11(a) in Fig. 7.15. As an example of the calculation of low, our postorder traversal visits e first. At e, there are back edges (e, a) and (e, b), so low [e] is set to min(dfnumber [e], dfnumber [a], dfnumber [b]) = 1. Then d is visited, and low [d] is set to the minimum of dfnumber [d], low [e], and dfnumber [a ]. The second of these arises because e is a child of d and the third because of the back edge (d, a).

    Fig. 7.15. Depth-first and low numberings.

    After computing low we consider each vertex. The root, a, is an articulation point because it has two children. Vertex c is an articulation point because it has a child f with low [f] � dfnumber [c]. The other vertices are not articulation points.

    The time taken by the above algorithm on a graph of e edges and n � e vertices is O(e). The reader should check that the time spent in each of the three phases can be attributed either to the vertex visited or to an edge emanating from that vertex, there being only a constant amount of time attributable to any vertex or edge in any pass. Thus, the total time is O(n+e), which is O(e) under the assumption n � e.

    7.5 Graph Matching

    In this section we outline an algorithm to solve "matching problems" on graphs. A simple example of a matching problem occurs when we have a set of teachers to assign to a set of courses. Each teacher is qualified to teach certain courses but not others. We wish to assign a course to a qualified teacher so that no two teachers are assigned the same course. With certain distributions of teachers and courses it is impossible to assign every teacher a course; in those situations we wish to assign as many teachers as possible.

    We can represent this situation by a graph as in Fig. 7.16 where the vertices are divided into two sets V1 and V2, such that vertices in the set V1 represent teachers and vertices in the set V2 courses. That teacher v is qualified to teach course w is represented by an edge (v, w). A graph such as this whose vertices can be divided into two disjoint groups with each edge having one end in each group is called bipartite. Assigning a teacher a course is equivalent to selecting an edge between a teacher vertex and a course vertex.

    Fig. 7.16. A bipartite graph.

    The matching problem can be formulated in general terms as follows. Given a graph G =(V, E), a subset of the edges in E with no two edges incident upon the same vertex in V is called a matching. The task of selecting a maximum subset of such edges is called the maximal matching problem. The heavy edges in Fig. 7.16 are an example of one maximal matching in that graph. A complete matching is a matching in which every vertex is an endpoint of some edge in the matching. Clearly, every complete matching is a maximal matching.

    There is a straightforward way to find maximal matchings. We can systematically generate all matchings and then pick one that has the largest number of edges. The difficulty with this method is that it has a running time that is an exponential function of the number of edges.

    There are more efficient algorithms for finding maximal matchings. These algorithms generally use a technique known as "augmenting paths." Let M be a matching in a graph G. A vertex v is matched if it is the endpoint of an edge in M. A path connecting two unmatched vertices in which alternate edges in the path are in M is called an augmenting path relative to M. Observe that an augmenting path must be of odd length, and must begin and end with edges not in M. Also observe that given an augmenting path P we can always find a bigger matching by removing from M those edges that are in P, and then adding to M the edges of P that were initially not in M. This new matching is M � P where � denotes "exclusive or" on sets. That is, the new matching consists of those edges that are in M or P, but not in both.

    Example 7.9. Figure 7.17(a) shows a graph and a matching M consisting of the heavy edges (1, 6), (3, 7), and (4, 8). The path 2, 6, 1, 8, 4, 9 in Fig. 7.17(b) is an augmenting path relative to M. Figure 7.18 shows the matching (l, 8), (2, 6), (3, 7), (4, 9) obtained by removing from M those edges that are in the path, and then adding to M the other edges in the path.

    Fig. 7.17. A matching and an augmenting path.

    The key observation is that M is a maximal matching if and only if there is no augmenting path relative to M. This observation is the basis of our maximal matching algorithm.

    Suppose M and N are matchings with �M� < �N�. (�M� denotes the number of edges in M.) To see that M � N contains an augmenting path relative to M consider the graph G' = (V, M � N). Since M and N are both matchings, each vertex of V is an endpoint of at most one edge from M and an endpoint of at most one edge from N. Thus each connected component of G' forms a simple path (possibly a cycle) with edges alternating between M and N. Each path that is not a cycle is either an augmenting path relative to M or an augmenting path relative to N depending on whether it has more edges from N or

    Fig. 7.18. The larger matching.

    from M. Each cycle has an equal number of edges from M and N. Since �M� < �N�, M � N has more edges from N than M, and hence has at least one augmenting path relative to M.

    We can now outline our procedure to find a maximal matching M for a graph G = (V, E).

    1. Start with M = Ø.
    2. Find an augmenting path P relative to M and replace M by M � P.
    3. Repeat step (2) until no further augmenting paths exist, at which point M is a maximal matching.

    It remains only to show how to find an augmenting path relative to a matching M. We shall do this for the simpler case where G is a bipartite graph with vertices partitioned into sets V1 and V2. We shall build an augmenting path graph for G for levels i = 0, 1, 2, . . . using a process similar to breadth-first search. Level 0 consists of all unmatched vertices from V1. At odd level i, we add new vertices that are adjacent to a vertex at level i-1, by a non-matching edge, and we also add that edge. At even level i, we add new vertices that are adjacent to a vertex at level i-1 because of an edge in the matching M, together with that edge.

    We continue building the augmenting path graph level-by-level until an unmatched vertex is added at an odd level, or until no more vertices can be added. If there is an augmenting path relative to M, an unmatched vertex v will eventually be added at an odd level. The path from v to any vertex at level 0 is an augmenting path relative to M.

    Example 7.10. Figure 7.19 illustrates the augmenting path graph for the graph in Fig. 7.17(a) relative to the matching in Fig. 7.18, in which we have chosen vertex 5 as the unmatched vertex at level 0. At level 1 we add the non-matching edge (5, 6). At level 2 we add the matching edge (6, 2). At level 3 we can add either of the non-matching edges (2, 9) or (2, 10). Since both vertices 9 and 10 are currently unmatched, we can terminate the construction of the augmenting path graph after the addition of either one of these vertices. Both paths 9, 2, 6, 5 and 10, 2, 6, 5 are augmenting paths relative to the matching in Fig. 7.18.

    Fig. 7.19. Augmenting path graph.

    Suppose G has n vertices and e edges. Constructing the augmenting path graphs for a given matching takes O(e) time if we use an adjacency list representation for edges. Thus, to find each new augmenting path takes O(e) time. To find a maximal matching, we construct at most n/2 augmenting paths, since each enlarges the current matching by at least one edge. Therefore, a maximal matching may be found in O(ne) time for a bipartite graph G.

    Exercises

    7.1 | Describe an algorithm to insert and delete edges in the adjacency list representation for an undirected graph. Remember that an edge (i, j) appears on the adjacency list for both vertex i and vertex j.
    7.2 | Modify the adjacency list representation for an undirected graph so that the first edge on the adjacency list for a vertex can be deleted in constant time. Write an algorithm to delete the first edge at a vertex using your new representation. Hint. How do you arrange that the two cells representing edge (i, j) can be found quickly from one another?
    7.3 Consider the graph of Fig. 7.20.
    1. Find a minimum-cost spanning tree by Prim's algorithm.

    2. Find a minimum-cost spanning tree by Kruskal's algorithm.

    3. Find a depth-first spanning tree starting at a and at d.

    4. Find a breadth-first spanning tree starting at a and at d.

    Fig. 7.20. A graph.
    7.4 Let T be a depth-first spanning tree and B be the back edges for a connected undirected graph G = (V, E).
    1. Show that when each back edge from B is added to T a unique cycle results. Call such a cycle a basic cycle.

    2. The linear combination of cycles C1, C2, . . . , Cn is C1 � C2 � . . . � Cn. Prove that the linear combination of two distinct nondisjoint cycles is a cycle.

    3. Show that every cycle in G can be expressed as a linear combination of basic cycles.
    *7.5 | Let G = (V, E) be a graph. Let R be a relation on V such that u R v if and only if u and v lie on a common (not necessarily simple) cycle. Prove that R is an equivalence relation on V.
    7.6 | Implement both Prim's algorithm and Kruskal's algorithm. Compare the running times of your programs on a set of "random" graphs.
    7.7 | Write a program to find all the connected components of a graph.
    7.8 | Write an O(n) program to determine whether a graph of n vertices has a cycle.
    7.9 | Write a program to enumerate all simple cycles of a graph. How many such cycles can there be? What is the time complexity of your program?
    7.10 | Show that all edges in a breadth-first search are either tree edges or cross edges.
    7.11 | Implement the algorithm for finding articulation points discussed in Section 7.4.
    *7.12 | Let G = (V, E) be a complete graph, that is, a graph in which there is an edge between every pair of distinct vertices. Let G' = (V, E') be a directed graph in which E' is E with each edge given an arbitrary orientation. Show that G' has a directed path that includes every vertex exactly once.
    **7.13 | Show that an n-vertex complete graph has nn-2 spanning trees.
    7.14 | Find all maximal matchings for the graph of Fig. 7.16.
    7.15 | Write a program to find a maximal matching for a bipartite graph.
    7.16 Let M be a matching and let m be the number of edges in a maximal matching.
    1. Prove there exists an augmenting path relative to M of length at most 2(�M�/(m - �M�)) + 1.

    2. Prove that if P is a shortest augmenting path relative to M and if P' is an augmenting path relative to M � P, then �P'� � �P� + �P � P'�.
    *7.17 | Prove that a graph is bipartite if and only if it has no odd length cycles. Give an example of a non-bipartite graph for which the augmenting path graph technique of Section 7.5 no longer works.
    7.18 | Let M and N be matchings in a bipartite graph with �M� < �N�. Prove that M � N has at least �N� - �M� vertex disjoint augmenting paths relative to M.

    Bibliographic Notes

    Methods for constructing minimal spanning trees have been studied since at least Boruvka [1926]. The two algorithms given in this chapter are based on Kruskal [1956] and Prim [1957]. Johnson [1975] shows how k-ary partially ordered trees can be used to implement Prim's algorithm. Cheriton and Tarjan [1976] and Yao [1975] present O(e 1og log n) spanning tree algorithms. Tarjan [1981] provides a comprehensive survey and history of spanning tree algorithms.

    Hopcroft and Tarjan [1973] and Tarjan [1972] popularized the use of depth-first search in graph algorithms. The biconnected components algorithm is from there.

    Graph matching was studied by Hall [1948] and augmenting paths by Berge [1957] and by Edmonds [1965]. Hopcroft and Karp [1973] give an O(n2.5) algorithm for maximal matching in bipartite graphs, and Micali and Vazirani [1980] give an [image] algorithm for maximal matching in general graphs. Papadimitriou and Steiglitz [1982] contains a good discussion of general matching.

    † Unless otherwise stated, we shall assume an edge is always a pair of distinct vertices.

    † Note that MERGE and FIND are defined slightly differently from Section 5.5, since C is a parameter telling where A and B can be found.

    ‡We can initialize a partially ordered tree of e elements in O(e) time if we do it all at once. We discuss this technique in Section 8.4, and we should probably use it here, since if many fewer than e edges are examined before the minimum-cost spanning tree is found, we may save significant time.

    Table of Contents Go to Chapter 8


Data Structures and Algorithms: CHAPTER 8: Sorting

 Next Chapter Return to Table of Contents Previous Chapter

Sorting

The process of sorting, or ordering, a list of objects according to some linear order such as � for numbers is so fundamental, and is done so frequently, that a careful examination of the subject is warranted. We shall divide the subject into two parts -- internal and external sorting. Internal sorting takes place in the main memory of a computer, where we can use the random access capability of the main memory to advantage in various ways. External sorting is necessary when the number of objects to be sorted is too large to fit in main memory. There, the bottleneck is usually the movement of data between main and secondary storage, and we must move data in large blocks to be efficient. The fact that physically contiguous data is most conveniently moved in one block constrains the kinds of external sorting algorithms we can use. External sorting will be covered in Chapter 11.

8.1 The Internal Sorting Model

In this chapter, we shall present the principal internal sorting algorithms. The simplest algorithms usually take O(n2) time to sort n objects and are only useful for sorting short lists. One of the most popular sorting algorithms is quicksort, which takes O(nlogn) time on average. Quicksort works well for most common applications, although, in the worst case, it can take O(n2) time. There are other methods, such as heapsort and mergesort, that take O(nlogn) time in the worst case, although their average case behavior may not be quite as good as that of quicksort. Mergesort, however, is well-suited for external sorting. We shall consider several other algorithms called "bin" or "bucket" sorts. These algorithms work only on special kinds of data, such as integers chosen from a limited range, but when applicable, they are remarkably fast, taking only O(n) time in the worst case.

Throughout this chapter we assume that the objects to be sorted are records consisting of one or more fields. One of the fields, called the key, is of a type for which a linear-ordering relationship � is defined. Integers, reals, and arrays of characters are common examples of such types, although we may generally use any key type for which a "less than" or "less than or equal to" relation is defined.

The sorting problem is to arrange a sequence of records so that the values of their key fields form a nondecreasing sequence. That is, given records r1, r2, . . . ,rn, with key values k1, k2, . . . , kn, respectively, we must produce the same records in an order ri1, ri2, . . . , rin, such that ki1 � ki2 � � � � � kin. The records all need not have distinct values, nor do we require that records with the same key value appear in any particular order.

We shall use several criteria to evaluate the running time of an internal sorting algorithm. The first and most common measure is the number of algorithm steps required to sort n records. Another common measure is the number of comparisons between keys that must be made to sort n records. This measure is often useful when a comparison between a pair of keys is a relatively expensive operation, as when keys are long strings of characters. If the size of records is large, we may also want to count the number of times a record must be moved. The application at hand usually makes the appropriate cost measure evident.

8.2 Some Simple Sorting Schemes

Perhaps the simplest sorting method one can devise is an algorithm called "bubblesort." The basic idea behind bubblesort is to imagine that the records to be sorted are kept in an array held vertically. The records with low key values are "light" and bubble up to the top. We make repeated passes over the array, from bottom to top. As we go, if two adjacent elements are out of order, that is, if the "lighter" one is below, we reverse them. The effect of this operation is that on the first pass the "lightest" record, that is, the record with the lowest key value, rises all the way to the top. On the second pass, the second lowest key rises to the second position, and so on. We need not, on pass two, try to bubble up to position one, because we know the lowest key already resides there. In general, pass i need not try to bubble up past position i. We sketch the algorithm in Fig. 8.1, assuming A is an array[1..n] of recordtype, and n is the number of records. We assume here, and throughout the chapter, that one field called key holds the key value for each record.

(1)       for i := 1 to n-1 do
(2)            for j := n downto i+1 do
(3)                 if A[j].key < A[j-1].key
then
(4)                      swap(A[j], A[j-1])

Fig. 8.1. The bubblesort algorithm.

The procedure swap is used in many sorting algorithms and is defined in Fig. 8.2.

Example 8.1. Figure 8.3 shows a list of famous volcanoes, and a year in which each erupted.

For this example we shall use the following type definitions in our sorting programs:

     procedure swap ( var x, y: recordtype )
         { swap exchanges the values of x and y }
         var
             temp: recordtype;
         begin
             temp := x;
             x := y;
             y := temp
         end; { swap }

Fig. 8.2. The procedure swap.

Fig. 8.3. Famous volcanoes.

     type
        keytype = array[1..10] of char;
        recordtype = record
            key: keytype; { the volcano name }
            year: integer
        end;

The bubblesort algorithm of Fig. 8.1 applied to the list of Fig. 8.3 sorts the list in alphabetical order of names, if the relation � on objects of this key type is the ordinary lexicographic order. In Fig. 8.4, we see the five passes made by the algorithm when n = 6. The lines indicate the point above which the names are known to be the smallest in the list and in the correct order. However, after i = 5, when all but the last record is in its place, the last must also be in its correct place, and the algorithm stops.

At the start of the first pass, St. Helens bubbles past Vesuvius, but not past Agung. On the remainder of that pass, Agung bubbles all the way up to the top. In the second pass, Etna bubbles as far as it can go, to position 2. In the third pass, Krakatoa bubbles past Pelee, and the list is now in lexicographically sorted order although according to the algorithm of Fig. 8.1, two additional passes are made.

Fig. 8.4. The passes of bubblesort.

Insertion Sorting

The second sorting method we shall consider is called "insertion sort," because on the ith pass we "insert" the ith element A[i] into its rightful place among A[1], A[2], . . . , A[i-1], which were previously placed in sorted order. After doing this insertion, the records occupying A[1], . . . , A[i] are in sorted order. That is, we execute

     for i := 2 to n do
          move A[i] forward to the position j �
i such that
              A[i] < A[k] for j �
k < i, and
              either A[i] � A[j-1] or
j = 1

To make the process of moving A[i] easier, it helps to introduce an element A[0], whose key has a value smaller than that of any key among A[1], . . . ,A[n]. We shall postulate the existence of a constant -� of type keytype that is smaller than the key of any record that could appear in practice. If no constant -� can be used safely, we must, when deciding whether to push A[i] up before position j, check first whether j = 1, and if not, compare A[i] (which is now in position j) with A[j-1]. The complete program is shown in Fig. 8.5.

(1)      A[0].key := -�;
(2)      for i := 2 to n do begin
(3)           j := i;
(4)           while A[j] < A[j-1] do begin
(5)               swap(A[j], A[j-1]);
(6)               j := j-1
              end
         end

Fig. 8.5. Insertion sort.

Example 8.2. We show in Fig. 8.6 the initial list from Fig. 8.3, and the result of the passes of insertion sort for i = 2, 3, . . . , 6. After each pass, the elements above the line are guaranteed to be sorted relative to each other, although their order bears no relation to the records below the line, which will be inserted later.

Fig. 8.6. The passes of insertion sort.

Selection Sort

The idea behind "selection sorting" is also elementary. In the ith pass, we select the record with the lowest key, among A[i], . . . , A[n], and we swap it with A[i]. As a result, after i passes, the i lowest records will occupy A[1] , . . . , A[i], in sorted order. That is, selection sort can be described by

     for i := 1 to n-1 do
          select the smallest among A[i], . . . , A[n] and
              swap it with A[i];

A more complete program is shown in Fig. 8.7.

Example 8.3. The passes of selection sort on the list of Fig. 8.3 are shown in Fig. 8.8. For example, on pass 1, the ultimate value of lowindex is 4, the position of Agung, which is swapped with Pelee in A[1].

The lines in Fig. 8.8 indicate the point above which elements known to be smallest appear in sorted order. After n - 1 passes, record A[n], Vesuvius in Fig. 8.8, is also in its rightful place, since it is the element known not to be among the n - 1 smallest.

Time Complexity of the Methods

Bubblesort, insertion sort, and selection sort each take O(n2) time, and will take W(n2) time on some, in fact, on most input sequences of n elements. Consider bubblesort in Fig. 8.1. No matter what recordtype is, swap takes a constant time. Thus lines (3) and (4) of Fig. 8.1 take at most c1 time units for some constant c1. Hence for a fixed value of i, the loop of lines (2- 4)

     var
         lowkey: keytype; { the currently smallest key found
             on a pass through A[i], . . . , A[n] }
         lowindex : integer; { the position of lowkey }
     begin
(1)      for i := 1 to n-1 do begin
              { select the lowest among A[i], . . . , A[n] and swap it with
A[i] }
(2)           lowindex := i;
(3)           lowkey := A[i].key;
(4)           for j := i + 1 to n do
                   { compare each key with current lowkey }
(5)                if A[j].key < lowkey then begin
(6)                          lowkey := A[j].key;
(7)                          lowindex := j
                   end;
(8)          swap(A[i], A[lowindex])
         end
     end;

Fig. 8.7. Selection sort.

Fig. 8.8. Passes of selection sort.

takes at most c2(n-i) steps, for some constant c2; the latter constant is somewhat larger than c1 to account for decrementation and testing of j. Consequently, the entire program takes at most

steps, where the term c3n accounts for incrementating and testing i. As the latter formula does not exceed (c2/2+c3)n2, for n � 1, we see that the time complexity of bubblesort is O(n2). The algorithm requires W(n2) steps, since even if no swaps are ever needed (i.e., the input happens to be already sorted), the test of line (3) is executed n(n - 1)/2 times.

Next consider the insertion sort of Fig. 8.5. The while-loop of lines (4-6) of Fig. 8.5 cannot take more than O(i) steps, since j is initialized to i at line (3), and decreases each time around the loop. The loop terminates by the time j = 1, since A[0] is -�, which forces the test of line (4) to be false when j = 1. We may conclude that the for-loop of lines (2-6) takes at most [image] steps for some constant c. This sum is O(n2).

The reader may check that if the array is initially sorted in reverse order, then we actually go around the while-loop of lines (4-6) i-1 times, so line (4) is executed [image] times. Therefore, insertion sort requires W(n2) time in the worst case. It can be shown that this smaller bound holds in the average case as well.

Finally, consider selection sort in Fig. 8.7. We may check that the inner for-loop of lines (4-7) takes O(n-i) time, since j ranges from i+1 to n. Thus the total time taken by the algorithm is c [image], for some constant c. This sum, which is cn(n - 1)/2, is seen easily to be O(n2). Conversely, one can show that line (5), at least, is executed [image] times regardless of the initial array A, so selection sort takes W(n2) time in the worst case and the average case as well.

Counting Swaps

If the size of records is large, the procedure swap, which is the only place in the three algorithms where records are copied, will take far more time than any of the other steps, such as comparison of keys and calculations on array indices. Thus, while all three algorithms take time proportional to n2, we might be able to compare them in more detail if we count the uses of swap.

To begin, bubblesort executes the swap step of line (4) of Fig. 8.1 at most

times, or about n2/2 times. But since the execution of line (4) depends on the outcome of the test of line (3), we might expect that the actual number of swaps will be considerably less than n2/2.

In fact, bubblesort swaps exactly half the time on the average, so the expected number of swaps if all initial sequences are equally likely is about n2/4. To see this, consider two initial lists of keys that are the reverse of one another, say L1 = k1, k2 , . . . , kn and L2 = kn, kn - 1, . . . , k1. A swap is the only way ki and kj can cross each other if they are initially out of order. But ki and kj are out of order in exactly one of L1 and L2. Thus the total number of swaps executed when bubblesort is applied to L1 and L2 is equal to the number of pairs of elements, that is, ( [image]) or n(n - 1)/2. Therefore the average number of swaps for L1 and L2 is n(n - 1)/4 or about n2/4. Since all possible orderings can be paired with their reversals, as L1 and L2 were, we see that the average number of swaps over all orderings will likewise be about n2/4.

The number of swaps made by insertion sort on the average is exactly what it is for bubblesort. The same argument applies; each pair of elements gets swapped either in a list L or its reverse, but never in both.

However, in the case that swap is an expensive operation we can easily see that selection sort is superior to either bubblesort or insertion sort. Line (8) in Fig. 8.7 is outside the inner loop of the selection sort algorithm, so it is executed exactly n - 1 times on any array of length n. Since line (8) has the only call to swap in selection sort, we see that the rate of growth in the number of swaps by selection sort, which is O(n), is less than the growth rates of the number of swaps by the other two algorithms, which is O(n2). Intuitively, unlike bubblesort or insertion sort, selection sort allows elements to "leap" over large numbers of other elements without being swapped with each of them individually.

A useful strategy to use when records are long and swaps are expensive is to maintain an array of pointers to the records, using whatever sorting algorithm one chooses. One can then swap pointers rather than records. Once the pointers to the records have been arranged into the proper order, the records themselves can be arranged into the final sorted order in O(n) time.

Limitations of the Simple Algorithms

We should not forget that each of the algorithms mentioned in this section has an O(n2) running time, in both the worst case and average case. Thus, for large n, none of these algorithms compares favorably with the O(nlogn) algorithms to be discussed in the next sections. The value of n at which these more complex O(nlogn) algorithms become better than the simple O(n2) algorithms depends on a variety of factors such as the quality of the object code generated by the compiler, the machine on which the programs are run, and the size of the records we must swap. Experimentation with a time profiler is a good way to determine the cutover point. A reasonable rule of thumb is that unless n is at least around one hundred, it is probably a waste of time to implement an algorithm more complicated than the simple ones discussed in this section. Shellsort, a generalization of bubblesort, is a simple, easy-to-implement O(n1.5) sorting algorithm that is reasonably efficient for modest values of n. Shellsort is presented in Exercise 8.3.

8.3 Quicksort

The first O(nlogn) algorithm† we shall discuss, and probably the most efficient for internal sorting, has been given the name "quicksort." The essence of quicksort is to sort an array A[1], . . . , A[n] by picking some key value v in the array as a pivot element around which to rearrange the elements in the array. We hope the pivot is near the median key value in the array, so that it is preceded by about half the keys and followed by about half. We permute the elements in the array so that for some j, all the records with keys less than v appear in A[1] , . . . , A[ j], and all those with keys v or greater appear in A[j+ 1], . . . , A[n]. We then apply quicksort recursively to A[1], . . . , A[j] and to A[j + 1] , . . . , A[n] to sort both these groups of elements. Since all keys in the first group precede all keys in the second group, the entire array will thus be sorted.

Example 8.4. In Fig. 8.9 we display the recursive steps that quicksort might take to sort the sequence of integers 3, 1, 4, 1, 5, 9, 2, 6, 5, 3. In each case, we have chosen to take as our value v the larger of the two leftmost distinct values. The recursion stops when we discover that the portion of the array we have to sort consists of identical keys. We have shown each level as consisting of two steps, one before partitioning each subarray, and the second after. The rearrangement of records that takes place during partitioning will be explained shortly.

Let us now begin the design of a recursive procedure quicksort(i, j) that operates on an array A with elements A[1] , . . . , A[n], defined externally to the procedure. quicksort(i, j) sorts A[i] through A[j], in place. A preliminary sketch of the procedure is shown in Fig. 8.10. Note that if A[i], . . . , A[ j] all have the same key, the procedure does nothing to A.

We begin by developing a function findpivot that implements the test of line (1) of Fig. 8.10, determining whether the keys of A[i], . . . , A[j] are all the same. If findpivot never finds two different keys, it returns 0. Otherwise, it returns the index of the larger of the first two different keys. This larger key becomes the pivot element. The function findpivot is written in Fig. 8.11.

Next, we implement line (3) of Fig. 8.10, where we face the problem of permuting A[i], . . . , A[j], in place‡, so that all the keys smaller than the pivot value appear to the left of the others. To do this task, we introduce two cursors, l and r, initially at the left and right ends of the portion of A being sorted, respectively. At all times, the elements to the left of l, that is, A[i], . . . , A[l - 1] will have keys less than the pivot. Elements to the right of r, that is, A[r+1], . . . , A[j], will have keys equal to or greater than the pivot, and elements in the middle will be mixed, as suggested in Fig. 8.12.

Initially, i = l and j = r so the above statement holds since nothing is to the left of l or to the right of r. We repeatedly do the following steps, which move l right and r left, until they finally cross, whereupon A[i], . . . , A[l - 1] will contain all the keys less than the pivot and A[r+1], . . . , A[j] all the keys equal to or greater than the pivot.

  1. Scan. Move l right over any records with keys less than the pivot. Move r left over any keys greater than or equal to the pivot. Note that our

    Fig. 8.9. The operation of quicksort.

    selection of the pivot by findpivot guarantees that there is at least one key less than the pivot and at least one not less than the pivot, so l and r will surely come to rest before moving outside the range i to j.

  2. Test. If l > r (which in practice means l = r+1), then we have successfully partitioned A[i], . . . , A[j], and we are done.

  3. Switch. If l < r (note we cannot come to rest during the scan with l = r, because one or the other of these will move past any given key), then

(1)       if A[i] through A[j] contains at least two distinct keys
then begin
(2)           let v be the larger of the first two distinct keys found;
(3)            permute A[i], . . . ,A[j] so that for some k between
                  i + 1 and j, A[i], . . . ,A[k - 1 ] all have keys less
than
                  v and A[k], . . . ,A[j] all have keys � v;
(4)            quicksort(i, k - 1 );
(5)            quicksort(k, j)
          end

    Fig. 8.10. Sketch of quicksort.

function findpivot ( i, j: integer ) :integer;
   { returns 0 if A[i], . . . ,A[j] have identical keys, otherwise
   returns the index of the larger of the leftmost two different keys }
   var
      firstkey: keytype; { value of first key found, i.e., A[i].key }
      k: integer; { runs left to right looking for a different key }
   begin
      firstkey := A[i].key;
      for k := i + 1 to j do {scan for different key}
         if A[k].key > firstkey then { select larger key }
            return (k)
         else if A[k].key < firstkey then
        return (i);
         return (0) { different keys were never found }
   end; { findpivot }

    Fig. 8.11. The procedure findpivot.

    Fig. 8.12. Situation during the permutation process.

    swap A[l] with A[r]. After doing so, A[l] has a key less than the pivot and A[r] has a key at least equal to the pivot, so we know that in the next scan phase, l will move at least one position right, over the old A[r], and r will move at least one position left.

The above loop is awkward, since the test that terminates it is in the middle. To put it in the form of a repeat-loop, we move the switch phase to the beginning. The effect is that initially, when i = l and j = r, we shall swap A[i] with A[j]. This could be right or wrong; it doesn't matter, as we assume no particular order for the keys among A[i], . . . , A[j] initially. The reader, however, should be aware of this "trick" and not be puzzled by it. The function partition, which performs the above operations and returns l, the point at which the upper half of the partitioned array begins, is shown in Fig. 8.13.

    function partition ( i, j: integer; pivot: keytype ): integer;
     { partitions A[i], . . . ,A[j] so keys < pivot are at the left
     and keys � pivot are on the right. Returns the
beginning of the
     group on the right. }
       var
          l, r: integer; { cursors as described above }
       begin
(1)            l := i;
(2)            r := j;
           repeat
(3)               swap(A[l], A[r]);
               { now the scan phase begins }
(4)               while A[l].key < pivot do
(5)                  l := l + 1;
(6)               while A[r].key > = pivot do
(7)                  r := r - 1
               until
(8)               l > r;
(9)            return (l)
       end; { partition }

Fig. 8.13. The procedure partition.

We are now ready to elaborate the quicksort sketch of Fig. 8.10. The final program is shown in Fig. 8.14. To sort an array A of type array[1..n] of recordtype we simply call quicksort(1, n).

The Running Time of Quicksort

We shall show that quicksort takes O(nlogn) time on the average to sort n elements, and O(n2) time in the worst case. The first step in substantiating both claims is to prove that partition takes time proportional to the number of elements that it is called upon to separate, that is, O(j - i + 1) time.

     procedure quicksort ( i, j: integer );
        { sort elements A[i], . . . ,A[j] of external array A }
        var
           pivot: keytype; { the pivot value }
           pivotindex: integer; { the index of an element of A where
              key is the pivot }
           k: integer; { beginning index for group of elements �
pivot }
        begin
(1)        pivotindex := findpivot(i, j);
(2)        if pivotindex <> 0 then begin { do nothing if all keys are equal }
(3)                 pivot := A[pivotindex].key;
(4)                 k := partition(i, j, pivot);
(5)                 quicksort(i, k - 1);
(6)                 quicksort(k, j)
           end
        end; { quicksort }

Fig. 8.14. The procedure quicksort.

To see why that statement is true, we must use a trick that comes up frequently in analyzing algorithms; we must find certain "items" to which time may be "charged," and then show how to charge each step of the algorithm so no item is charged more than some constant. Then the total time spent is no greater than this constant times the number of "items."

In our case, the "items" are the elements from A[i] to A[j], and we charge to each element all the time spent by partition from the time l or r first points to that element to the time l or r leaves that element. First note that neither l nor r ever returns to an element. Because there is at least one element in the low group and one in the high group, and because partition stops as soon as l exceeds r, we know that each element will be charged at most once.

We move off an element in the loops of lines (4) and (6) of Fig. 8.13, either by increasing l or decreasing r. How long could it be between times when we execute l : = l + 1 or r := r - 1? The worst that can happen is at the beginning. Lines (1) and (2) initialize l and r. Then we might go around the loop without doing anything to l or r. On second and subsequent passes, the swap at line (3) guarantees that the while-loops of lines (4) and (6) will be successful at least once each, so the worst that can be charged to an execution of l := l + 1 or r := r - 1 is the cost of line (1), line (2), twice line (3), and the tests of lines (4), (6), (8), and (4) again. This is only a constant amount, independent of i or j, and subsequent executions of l := l + 1 or r := r - 1 are charged less -- at most one execution of lines (3) and (8) and once around the loops of lines (4) or (6).

There are also the final two unsuccessful tests at lines (4), (6), and (8), that may not be charged to any "item," but these represent only a constant amount and may be charged to any item. After all charges have been made, we still have some constant c so that no item has been charged more than c time units. Since there are j - i + 1 "items," that is, elements in the portion of the array to be sorted, the total time spent by partition(i, j, pivot) is O(j - i + 1).

Now let us turn to the running time spent by quicksort(i, j). We can easily check that the time spent by the call to findpivot at line (1) of Fig. 8.14 is O(j - i + 1), and in most cases much smaller. The test of line (2) takes a constant amount of time, as does step (3) if executed. We have just argued that line (4), the call to partition, will take O(j - i + 1) time. Thus, exclusive of recursive calls it makes to quicksort, each individual call of quicksort takes time at most proportional to the number of elements it is called upon to sort.

Put another way, the total time taken by quicksort is the sum over all elements of the number of times that element is part of the subarray on which a call to quicksort is made. Let us refer back to Fig. 8.9, where we see calls to quicksort organized into levels. Evidently, no element can be involved in two calls at the same level, so the time taken by quicksort can be expressed as the sum over all elements of the depth, or maximum level, at which that element is found. For example, the 1's in Fig. 8.9 are of depth 3 and the 6 of depth 5.

In the worst case, we could manage to select at each call to quicksort a worst possible pivot, say the largest of the key values in the subarray being sorted. Then we would divide the subarray into two smaller subarrays, one with a single element (the element with the pivot as key) and the other with everything else. That sequence of partitions leads to a tree like that of Fig. 8.15, where r1, r2 , . . . , rn is the sequence of records in order of increasing keys.

Fig. 8.15. A worst possible sequence of pivot selections.

The depth of ri is n - i + 1 for 2 � i � n, and the depth of r1 is n - 1. Thus the sum of the depths is

which is W(n2). Thus in the worst case, quicksort takes time proportional to n2 to sort n elements.

Average Case Analysis of Quicksort

Let us, as always, interpret "average case" for a sorting algorithm to mean the average over all possible initial orderings, equal probability being attributed to each possible ordering. For simplicity, we shall assume no two elements have equal keys. In general, equalities among elements make our sorting task easier, not harder, anyway.

A second assumption that makes our analysis of quicksort easier is that, when we call quicksort(i, j), then all orders for A [i], . . . , A [j] are equally likely. The justification is that prior to this call, no pivots with which A [i], . . . , A [j] were compared distinguished among them; that is, for each such pivot v, either all were less than v or all were equal to or greater than v. A careful examination of the quicksort program we developed shows that each pivot element is likely to wind up near the right end of the subarray of elements equal to or greater than this pivot, but for large subarrays, the fact that the minimum element (the previous pivot, that is) is likely to appear near the right end doesn't make a measurable difference.†

Now, let T(n) be the average time taken by quicksort to sort n elements. Clearly T(1) is some constant c1, since on one element, quicksort makes no recursive calls to itself. When n > 1, since we assume all elements have unequal keys, we know quicksort will pick a pivot and split the subarray, taking c2n time to do this, for some constant c2, then call quicksort on the two subarrays. It would be nice if we could claim that the pivot was equally likely to be any of the first, second, . . . , nth element in the sorted order for the subarray being sorted. However, to guarantee ourselves that quicksort would find at least one key less than each pivot and at least one equal to or greater than the pivot (so each piece would be smaller than the whole, and therefore infinite loops would not be possible), we always picked the larger of the first two elements found. It turns out that this selection doesn't affect the distribution of sizes of the subarrays, but it does tend to make the left groups (those less than the pivot) larger then the right groups.

Let us develop a formula for the probability that the left group has i of the n elements, on the assumption that all elements are unequal. For the left group to have i elements, the pivot must be the i + 1st among the n. The pivot, by our method of selection, could either have been in the first position, with one of the i smaller elements second, or it could have been second, with one of the i smaller ones first. The probability that any particular element, such as the i + 1st, appears first in a random order is 1/n. Given that it did appear first, the probability that the second element is one of the i smaller elements out of the n -1 remaining elements is i/(n - 1). Thus the probability that the pivot appears in the first position and is number i + 1 out of n in the proper order is i/n(n - 1). Similarly, the probability that the pivot appears in the second position and is number i + 1 out of n in the sorted order is i/n(n - 1), so the probability that the left group is of size i is 2i/n(n - 1), for 1 � i < n.

Now we can write a recurrence for T(n).

Equation (8.1) says that the average time taken by quicksort is the time, c2n, spent outside of the recursive calls, plus the average time spent in recursive calls. The latter time is expressed in (8.1) as the sum, over all possible i, of the probability that the left group is of size i (and therefore the right group is of size n - i) times the cost of the two recursive calls: T(i) and T(n - i), respectively.

Our first task is to transform (8.1) so that the sum is simplified, and in fact, so (8.1) takes on the form it would have had if we had picked a truly random pivot at each step. To make the transformation, observe that for any function f (i) whatsoever, by substituting i for n - i we may prove

By replacing half the left side of (8.2) by the right side, we see

Applying (8.3) to (8.1), with f(i) equal to the expression inside the summation of (8.1), we get

Next, we can apply (8.3) to (8.4), with f(i) = T(i). This transformation yields

Note that (8.4) is the recurrence we would get if all sizes between 1 and n - 1 for the left group were equally likely. Thus, picking the larger of two keys as the pivot really doesn't affect the size distribution. We shall study recurrences of this form in greater detail in Chapter 9. Here we shall solve recurrence (8.5) by guessing a solution and showing that it works. The solution we guess is that T(n) � cn log n for some constant c and all n � 2.

To prove that this guess is correct, we perform an induction on n. For n = 2, we have only to observe that for some constant c, T(2) � 2c log2 = 2c. To prove the induction, assume T(i) � ci log i for i < n, and substitute this formula for T(i) in the right side of (8.5) to show the resulting quantity is no greater than cn log n. Thus, (8.5) becomes

Let us split the summation of (8.6) into low terms, where i � n/2, and therefore log i is no greater than log(n/2), which is (logn) - 1, and high terms, where i > n/2, and log i may be as large as log n. Then (8.6) becomes

If we pick c � 4c2, then the sum of the second and fourth terms of (8.7) is no greater than zero. The third term of (8.7) makes a negative contribution, so from (8.7) we may claim that T(n) � cn log n, if we pick c = 4c2. This completes the proof that quicksort requires O (n log n) time in the average case.

Improvements to Quicksort

Not only is quicksort fast, its average running time is less than that of all other currently known O(n log n) sorting algorithms (by a constant factor, of course). We can improve the constant factor still further if we make some effort to pick pivots that divide each subarray into close-to-equal parts. For example, if we always divide subarrays equally, then each element will be of depth exactly logn, in the tree of partitions analogous to Fig. 8.9. In comparison, the average depth of an element for quicksort as constituted in Fig. 8.14, is about 1.4logn. Thus we could hope to speed up quicksort by choosing pivots carefully.

For example, we could choose three elements of a subarray at random, and pick the middle one as the pivot. We could pick k elements at random for any k, sort them either by a recursive call to quicksort or by one of the simpler sorts of Section 8.2, and pick the median element, that is, the [image] element, as pivot.† It is an interesting exercise to determine the best value of k, as a function of the number of elements in the subarray to be sorted. If we pick k too small, it costs time because on the average the pivot will divide the elements unequally. If we pick k too large, we spend too much time finding the median of the k elements.

Another improvement to quicksort concerns what happens when we get small subarrays. Recall from Section 8.2 that the simple O(n2) methods are better than the O(n log n) methods such as quicksort for small n. How small n is depends on many factors, such as the time spent making a recursive call, which is a property of the machine architecture and the strategy for implementing procedure calls used by the compiler of the language in which the sort is written. Knuth [1973] suggests 9 as the size subarray on which quicksort should call a simpler sorting algorithm.

There is another "speedup" to quicksort that is really a way to trade space for time. The same idea will work for almost any sorting algorithm. If we have the space available, create an array of pointers to the records in array A. We make the comparisons between the keys of records pointed to. But we don't move the records; rather, we move the pointers to records in the same way quicksort moves the records themselves. At the end, the pointers, read in left-to-right order, point to the records in the desired order, and it is then a relatively simple matter to rearrange the records of A into the correct order.

In this manner, we make only n swaps of records, instead of O(n log n) swaps, which makes a substantial difference if records are large. On the negative side, we need extra space for the array of pointers, and accessing keys for comparisons is slower than it was, since we have first to follow a pointer, then go into the record, to get the key field.

8.4 Heapsort

In this section we develop a sorting algorithm called heapsort whose worst case as well as average case running time is O(nlogn). This algorithm can be expressed abstractly using the four set operations INSERT, DELETE, EMPTY, and MIN introduced in Chapters 4 and 5. Suppose L is a list of the elements to be sorted and S a set of elements of type recordtype, which will be used to hold the elements as they are being sorted. The MIN operator applies to the key field of records; that is, MIN(S) returns the record in S with the smallest key value. Figure 8.16 presents the abstract sorting algorithm that we shall transform into heapsort.

(1)       for x on list L do
(2)           INSERT(x, S);
(3)       while not EMPTY(S) do begin
(4)           y := MIN(S);
(5)           writeln(y);
(6)           DELETE(y, S)
      end

Fig. 8.16. An abstract sorting algorithm.

In Chapters 4 and 5 we presented several data structures, such as 2-3 trees, capable of supporting each of these set operations in O(logn) time per operation, if sets never grow beyond n elements. If we assume list L is of length n, then the number of operations performed is n INSERT's, n MIN's, n DELETE's, and n + 1 EMPTY tests. The total time spent by the algorithm of Fig. 8.16 is thus O(nlogn), if a suitable data structure is used.

The partially ordered tree data structure introduced in Section 4.11 is well suited for the implementation of this algorithm. Recall that a partially ordered tree can be represented by a heap, an array A[1], . . . , A[n], whose elements have the partially ordered tree property: A[i].key � A[2*i].key and A[i].key � A[2*i + 1].key. If we think of the elements at 2i and 2i + 1 as the "children" of the element at i, then the array forms a balanced binary tree in which the key of the parent never exceeds the keys of the children.

We showed in Section 4.11 that the partially ordered tree could support the operations INSERT and DELETEMIN in O(logn) time per operation. While the partially ordered tree cannot support the general DELETE operation in O(logn) time (just finding an arbitrary element takes linear time in the worst case), we should notice that in Fig. 8.16 the only elements we delete are the ones found to be minimal. Thus lines (4) and (6) of Fig. 8.16 can be combined into a DELETEMIN function that returns the element y. We can thus implement Fig. 8.16 using the partially ordered tree data structure of Section 4.11.

We shall make one more modification to the algorithm of Fig. 8.16 to avoid having to print the elements as we delete them. The set S will always be stored as a heap in the top of array A, as A[1], . . . , A[i], if S has i elements. By the partially ordered tree property, the smallest element is always in A[1]. The elements already deleted from S can be stored in A [i + 1], . . . ,A [n], sorted in reverse order, that is, with A[i + 1] � A[i + 2]� � � � � A[n].† Since A[1] must be smallest among A[1], . . . , A[i], we may effect the DELETEMIN operation by simply swapping A[1] with A[i]. Since the new A[i] (the old A[1]) is no smaller than A[i + 1] (or else the former would have been deleted from S before the latter), we now have A[i], . . . , A[n] sorted in decreasing order. We may now regard S as occupying A[1], . . . ,A [i - 1].

Since the new A[1] (old A[i]) violates the partially ordered tree property, we must push it down the tree as in the procedure DELETEMIN of Fig. 4.23. Here, we use the procedure pushdown shown in Fig. 8.17 that operates on the externally defined array A. By a sequence of swaps, pushdown pushes element A[first] down through its descendants to its proper position in the tree. To restore the partially ordered tree property to the heap, we call pushdown with first = 1.

We now see how lines (4 - 6) of Fig. 8.16 are to be done. Selecting the minimum at line (4) is easy; it is always in A[l] because of the partially ordered tree property. Instead of printing at line (5), we swap A[1] with A[i], the last element in the current heap. Deleting the minimum element from the partially ordered tree is now easy; we just decrement i, the cursor that indicates the end of the current heap. We then invoke pushdown(1, i) to restore the partially ordered tree property to the heap A[1], . . . , A[i].

The test for emptiness of S at line (3) of Fig. 8.16 is done by testing the value of i, the cursor marking the end of the current heap. Now we have only to consider how to perform lines (1) and (2). We may assume L is originally present in A[1], . . . , A[n] in some order. To establish the partially ordered tree property initially, we call pushdown(j,n) for all j = n/2, n/2 - 1 , . . . , 1. We observe that after calling pushdown(j, n), no violations of the partially ordered tree property occur in A[j], . . . , A[n] because pushing a record down the tree does not introduce new violations, since we only swap a violating record with its smaller child. The complete procedure, called heapsort, is shown in Fig. 8.18.

Analysis of Heapsort

Let us examine the procedure pushdown to see how long it takes. An inspection of Fig. 8.17 confirms that the body of the while-loop takes constant time. Also, after each iteration, r has at least twice the value it had before. Thus, since r starts off equal to first, after i iterations we have r � first * 2i.

   procedure pushdown ( first, last: integer );
      { assumes A[first], . . . ,A[last] obeys partially ordered tree property
          except possibly for the children of A[first]. The procedure pushes
          A[first] down until the partially ordered tree property is restored }
      var
          r: integer; { indicates the current position of A[first] }
      begin
          r := first; { initialization }
          while r < = last div 2 do
             if last = 2*r then begin { r has one child at 2*r }
                   if A[r].key > A[2*r].key
then
                       swap(A[r], A[2*r]);
                       r := last { forces a break from the while-loop }
                   end
             else { r has two children, elements at 2*r and 2*r + 1 }
                   if A[r].key > A[2*r].key
and
                       A[2*r].key < = A[2*r + l].key then
begin
                           { swap r with left child }
                           swap(A[r], A[2*r]);
                           r := 2*r
                   end
                   else if A[r].key > A[2*r + l].key
and
                        A[2*r + l].key < A[2*r].key then
begin
                            { swap r with right child }
                            swap(A[r], A[2*r + 1]);
                            r := 2*r + l
                   end
                   else { r does not violate partially ordered tree property }
                        r := last { to break while-loop }
       end; { pushdown }

Fig. 8.17. The procedure pushdown.

Surely, r > last/2 if first * 2i > last/2, that is if

            i > log(last/first) - 1               (8.8)

Hence, the number of iterations of the while-loop of pushdown is at most log(last/first).

Since first � 1 and last � n in each call to pushdown by the heapsort algorithm of Fig. 8.18, (8.8) says that each call to pushdown, at line (2) or (5) of Fig. 8.18, takes O(logn) time. Clearly the loop of lines (1) and (2) iterates n/2 times, so the time spent in lines (1 - 2) is O(n logn).† Also, the loop of

    procedure heapsort;
       { sorts array A[1], . . . ,A[n] into decreasing order }
       var
           i: integer; { cursor into A }
       begin
               { establish the partially ordered tree property initially }
(1)            for i := n div 2 downto 1 do
(2)                 pushdown(i, n );
(3)            for i := n downto 2 do begin
(4)                swap(A[1], A[i]);
              { remove minimum from front of heap }
(5)                pushdown(1, i - 1)
                      { re-establish partially ordered tree property }
           end
    end; { heapsort }

Fig. 8.18. The procedure heapsort.

lines (3 - 5) iterates n - 1 times. Thus, a total of O(n) time is spent in all repetitions of swap at line (4), and O (n log n) time is spent during all repetitions of line (5). Hence, the total time spent in the loop of lines (3 - 5) is O (n log n ), and all of heapsort takes O (n log n) time.

Despite its O (n log n) worst case time, heapsort will, on the average take more time than quicksort, by a small constant factor. Heapsort is of intellectual interest, because it is the first O(n log n) worst-case sort we have covered. It is of practical utility in the situation where we want not all n elements sorted, but only the smallest k of them for some k much less than n. As we mentioned, lines (1 - 2) really take only O(n) time. If we make only k iterations of lines (3 - 5), the time spent in that loop is O(k log n). Thus, heapsort, modified to produce only the first k elements takes O(n + k log n) time. If k � n/log n, that is, we want at most (1/log n)th of the entire sorted list, then the required time is O(n).

8.5 Bin Sorting

One might wonder whether W (n log n) steps are necessary to sort n elements. In the next section, we shall show that to be the case for sorting algorithms that assume nothing about the data type of keys, except that they can be ordered by some function that tells whether one key value is "less than" another. We can often sort in less than O(n log n) time, however, when we know something special about the keys being sorted.

Example 8.5. Suppose keytype is integer, and the values of the key are known to be in the range l to n, with no duplicates, where n is the number of elements. Then if A and B are of type array[1..n] of recordtype, and the n elements to be sorted are initially in A, we can place them in the array B, in order of their keys by

     for i := 1 to n do
          B[A[i].key] := A[i];                     (8.9)

This code calculates where record A[i] belongs and places it there. The entire loop takes O(n) time. It works correctly only when there is exactly one record with key v for each value of v between 1 and n. A second record with key value v would also be placed in B[v], obliterating the previous record with key v.

There are other ways to sort an array A with keys 1,2, . . . , n in place in only O(n) time. Visit A[1], . . . , A[n] in turn. If the record in A[i] has key j � i, swap A[i] with A[j]. If, after swapping, the record with key k is now in A[i], and k � i, swap A[i] with A[k], and so on. Each swap places some record where it belongs, and once a record is where it belongs, it is never moved. Thus, the following algorithm sorts A in place, in O(n) time, provided there is one record with each of keys 1, 2 , . . . , n.

     for i := 1 to n do
          while A [i].key < > i do
             swap(A[i], A[A[i].key]);

The program (8.9) given in Example 8.5 is a simple example of a "binsort," a sorting process where we create a "bin" to hold all the records with a certain key value. We examine each record r to be sorted and place it in the bin for the key value of r. In the program (8.9) the bins are the array elements B[1], . . . , B[n], and B[i] is the bin for key value i. We can use array elements as bins in this simple case, because we know we never put more than one record in a bin. Moreover, we need not assemble the bins into a sorted list here, because B serves as such a list.

In the general case, however, we must be prepared both to store more than one record in a bin and to string the bins together (or concatenate) the bins in proper order. To be specific, suppose that, as always, A[1], . . . , A[n] is an array of type recordtype, and that the keys for records are of type keytype. Suppose for the purposes of this section only, that keytype is an enumerated type, such as l..m or char. Let listtype be a type that represents lists of elements of type recordtype; listtype could be any of the types for lists mentioned in Chapter 2, but a linked list is most effective, since we shall be growing lists of unpredictable size in each bin, yet the total lengths of the lists is fixed at n, and therefore an array of n cells can supply the lists for the various bins as needed.

Finally, let B be an array of type array[keytype] of listtype. Thus, B is an array of bins, which are lists (or, if the linked list representation is used, headers for lists). B is indexed by keytype, so there is one bin for each possible key value. In this manner we can effect the first generalization of (8.9) -- the bins have arbitrary capacity.

Now we must consider how the bins are to be concatenated. Abstractly, we must form from lists a1, a2, . . . ,ai and b1, b2, . . . , bj the concatenation of the lists, which is a1, a2, . . . ,ai. b1, b2, . . . ,bj. The implementation of this operation CONCATENATE(L1, L2), which replaces list L1 by the concatenation L1L2, can be done in any of the list representations we studied in Chapter 2.

For efficiency, however, it is useful to have, in addition to a header, a pointer to the last element on each list (or to the header if the list is empty). This modification facilitates finding the last element on list L1 without running down the entire list. Figure 8.19 shows, by dashed lines, the revised pointers necessary to concatenate L1 and L2 and to have the result be called L1. List L1 is assumed to "disappear" after concatenation, in the sense that the header and end pointer for L2 become null.

Fig. 8.19. Concatenation of linked lists.

We can now write a program to binsort arbitrary collections of records, where the key field is of an enumerated type. The program, shown in Fig. 8.20, is written in terms of list processing primitives. As we mentioned, a linked list is the preferred implementation, but options exist. Recall also that the setting for the procedure is that an array A of type array[1..n] of recordtype holds the elements to be sorted, and array B, of type array [keytype] of listtype represents the bins. We assume keytype is expressible as lowkey..highkey, as any enumerated type must be, for some quantities lowkey and highkey.

    procedure binsort;
       { binsort array A, leaving the sorted list in B[lowkey] }
       var
              i: integer;
              v: keytype;
           begin
               { place the records into bins }
(1)            for i := l to n do
                   { push A[i] onto the front of the bin for its key }
(2)                INSERT(A[i], FIRST(B[A[i].key]),
B[A[i].key]);
(3)            for v := succ(lowkey) to highkey do
                   { concatenate all the bins onto the end of B[lowkey] }
(4)                CONCATENATE (B[lowkey], B[v])
       end; { binsort }

Fig. 8.20. The abstract binsort program.

Analysis of Binsort

We claim that if there are n elements to be sorted, and there are m different key values (hence m different bins), then the program of Fig. 8.20 takes O(n + m) time, if the proper data structure for bins is used. In particular, if m � n, then binsort takes O(n) time. The data structure we have in mind is a linked list. Pointers to list ends, as indicated in Fig. 8.19, are useful but not required.

The loop of lines (1 - 2) of Fig. 8.20, which places the records in bins, takes O(n) time, since the INSERT operation of line (2) requires constant time, the insertion always occurring at the beginning of the list. For the loop of lines (3 - 4), concatenating the bins, temporarily assume that pointers to the ends of lists exist. Then step (4) takes constant time, so the loop takes O(m) time. Hence the entire binsort program takes O(n + m) time.

If pointers to the ends of lists do not exist, then in line (4) we must spend time running down to the end of B[v] before concatenating it to B[lowkey]. In this manner, the end of B[lowkey] will always be available for the next concatenation. The extra time spent running to the end of each bin once totals O(n), since the sum of the lengths of the bins is n. This extra time does not affect the order of magnitude of running time for the algorithm, since O(n) is no greater than O(n + m).

Sorting Large Key Sets

If m, the number of keys, is no greater than n, the number of elements, then the O(n + m) running time of Fig. 8.20 is really O(n). But what if m=n2, say. Evidently Fig. 8.20 will take O(n + n2), which is O(n2) time. But can we still take advantage of the fact that the key set is limited and do better? The surprising answer is that even if the set of possible key values is 1,2 , . . . , nk, for any fixed k, then there is a generalization of the binsorting technique that takes only O(n) time.

Example 8.6. Consider the specific problem of sorting n integers in the range 0 to n2 - 1. We sort the n integers in two phases. The first phase appears not to be of much help at all, but it is essential. We use n bins, one for each of the integers 0, 1 , . . . , n - 1. We place each integer i on the list to be sorted, into the bin numbered i mod n. However, unlike Fig. 8.20, it is important that we append each integer to the end of the list for the bin, not the beginning. If we are to append efficiently, we require the linked list representation of bins, with pointers to list ends.

For example, suppose n = 10, and the list to be sorted is the perfect squares from 02 to 92 in the random order 36, 9, 0, 25, 1, 49, 64, 16, 81, 4. In this case, where n = 10, the bin for integer i is just the rightmost digit of i written in decimal. Fig. 8.21(a) shows the placement of our list into bins. Notice that integers appear in bins in the same order that they appear in the original list; e.g., bin 6 has contents 36, 16, not 16, 36, since 36 precedes 16 on the original list.

Now, we concatenate the bins in order, producing the list

             0, 1, 81, 64, 4, 25, 36, 16, 9, 49             (8.10)
from Fig. 8.21(a). If we use the linked list data structure with pointers to list ends, then the placement of the n integers into bins and the concatenation of the bins can each be done in O(n) time.

The integers on the list created by concatenating the bins are redistributed into bins, but using a different bin selection strategy. Now place integer i into bin [i/n], that is, the greatest integer equal to or less than i/n. Again, append integers to the ends of the lists for the bins. When we now concatenate the bins in order, we find the list is sorted.

For our example, Fig. 8.21(b) shows the list (8.10) distributed into bins, with i going into bin [i/10].

To see why this algorithm works, we have only to notice that when several integers are placed in one bin, as 0, 1, 4, and 9 were placed in bin 0, they must be in increasing order, since the list (8.10) resulting from the first pass ordered them by rightmost digit. Hence, in any bin the rightmost digits must form an increasing sequence. Of course, any integer placed in bin i must precede an integer placed in a bin higher than i, so concatenating the bins in order produces the sorted list.

More generally, we can think of the integers between 0 and n2 - 1 as

Fig. 8.21. Two-pass binsorting.

two-digit numbers in base n and use the same argument to see that the sorting strategy works. Consider the integers i = an + b and j = cn + d, where a, b, c, and d are each in the range 0 to n - l; i.e., they are base- n digits. Suppose i<j. Then, a>c is not possible, so we may assume a�c. If a<c, then i appears in a lower bin than j after the second pass, so i will precede j in the final order. If a=c, then b must be less than d. Then after the first pass, i precedes j, since i was placed in bin b and j in bin d. Thus, while both i and j are placed in bin a (the same as bin c), i is inserted first, and j must follow it in the bin.

General Radix Sorting

Suppose keytype is a sequence of fields, as in

   type
         keytype = record
             day: 1..31;
             month: (jan , . . . ,  dec);            (8.11)
             year: 1900..1999
         end;

or an array of elements of the same type, as in

   type
         keytype = array[1..10] of char;             (8.12)

We shall assume from here that keytype consists of k components, f1,f2, . . . ,fk of types t1, t2, . . . , tk. For example, in (8.11) t1 = 1..31, t2 = (jan , . . . , dec), and t3 = 1900..1999. In (8.12), k = 10, and t1 = t2 = � � � = tk = char.

Let us also assume that we desire to sort records in the lexicographic order of their keys. That is, key value (a1, a2, . . . ,ak) is less than key value (b1, b2, . . . ,bk), where ai and bi are the values in field fi, for i = 1, 2, . . . ,k, if either

1. a1 < b1, or
2. a1 = b1 and a2 < b2, or

      .
      .
      .

k. a1 = b1, a2 = b2, . . . ,ak - 1=bk - 1, and ak < bk.

That is, for some j between 0 and k - 1, a1 = b1, . . . , aj = bj, † and aj + 1 <bj + 1.

We can view keys of the type defined above as if key values were integers expressed in some strange radix notation. For example, (8.12) where each field is a character, can be viewed as expressing integers in base 128, or however many characters there are in a character set for the machine at hand. Type definition (8.11) can be viewed as if the rightmost place were in base 100 (corresponding to the values between 1900 and 1999), the next place in base 12, and the third in base 31. Because of this view, generalized binsort has become known as radix sorting. In the extreme, we can even use it to sort integers up to any fixed limit by seeing them as arrays of digits in base 2, or another base.

The key idea behind radix sorting is to binsort all the records, first on fk, the "least significant digit," then concatenate the bins, lowest value first, binsort on fk - 1, and so on. As in Example 8.6, when inserting into bins, make sure that each record is appended to the end of the list, not the beginning. The radix sorting algorithm is sketched in Fig. 8.22; the reason it works was illustrated in Example 8.6. In general, after binsorting on fk, fk - 1, . . . ,fi, the records will appear in lexicographic order if the key consisted only of fields fi, . . . ,fk.

Analysis of Radix Sort

First, we must use the proper data structures to make radix sort efficient. Notice that we assume the list to be sorted is already in the form of a linked list, rather than an array. In practice, we need only to add one additional field, the link field, to the type recordtype, and we can link A[i] to A[i + 1] for i=1, 2 , . . . ,n - 1 and thus make a linked list out of the array A in O(n) time. Notice also that if we present the elements to be sorted in this way, we never copy a record. We just move records from one list to another.

    procedure radixsort;
       { sorts list A of n records with keys consisting of fields
f1, . . . , fk
         of types t1, . . . , tk, respectively. The
procedure uses
             arrays Bi of type array[ti]
of listtype for 1 � i � k,
             where listtype is a linked list of records. }
         begin
(1)            for i := k downto l do begin
(2)                for each value v of type ti do {
clear bins }
(3)                    make Bi[v] empty;
(4)                for each record r on list A do
(5)                    move r from A onto the end of bin
Bi[v],
                       where v is the value of field fi of the key of
r;
(6)                for each value v of type ti, from lowest to
highest do
(7)                    concatenate Bi[v] onto the end of A
               end
    end; { radixsort }

Fig. 8.22. Radix sorting.

As before, for concatenation to be done quickly, we need pointers to the end of each list. Then, the loop of lines (2 - 3) in Fig. 8.22 takes O(si) time, where si is the number of different values of type ti. The loop of lines (4 - 5) takes O(n) time, and the loop of lines (6 - 7) takes O(si) time. Thus, the total time taken by radix sort is [image], which is [image] if k is assumed a constant.

Example 8.7. If keys are integers in the range 0 to nk - 1, for some constant k, we can generalize Example 8.6 and view keys as base-n integers k digits long. Then ti is 0..(n - 1) for all i between 1 and k, so si = n. The expression [image] becomes O(n + kn) which, since k is a constant, is O(n).

As another example, if keys are character strings of length k, for constant k, then si = 128 (say) for all i, and [image] is a constant. Thus radix sort on fixed length character strings is also O(n). In fact, whenever k is a constant and the si's are constant, or even O(n), then radix sort takes O(n) time. Only if k grows with n can radix sort not be O(n). For example, if keys are regarded as binary strings of length logn, then k = logn, and si = 2 for l�i�k. Thus radix sort would be [image], which is O(n log n).†

8.6 A Lower Bound for Sorting by Comparisons

There is a "folk theorem" to the effect that sorting n elements "requires n log n time." We saw in the last section that this statement is not always true; if the key type is such that binsort or radix sort can be used to advantage, then O(n) time suffices. However, these sorting algorithms rely on keys being of a special type -- a type with a limited set of values. All the other general sorting algorithms we have studied rely only on the fact that we can test whether one key value is less than another.

The reader should notice that in all the sorting algorithms prior to Section 8.5, progress toward determining the proper order for the elements is made when we compare two keys, and then the flow for control in the algorithm goes one of only two ways. In contrast, an algorithm like that of Example 8.5 causes one of n different things to happen in only one step, by storing a record with an integer key in one of n bins depending on the value of that integer. All programs in Section 8.5 use a capability of programming languages and machines that is much more powerful than a simple comparison of values, namely the ability to find in one step a location in an array, given the index of that location. But this powerful type of operation is not available if keytype were, say, real. We cannot, in Pascal or most other languages, declare an array indexed by real numbers, and even if we could, we could not then concatenate in a reasonable amount of time all the bins corresponding to the computer-representable real numbers.

Decision Trees

Let us focus our attention on sorting algorithms whose only use of the elements to be sorted is in comparisons of two keys. We can draw a binary tree in which the nodes represent the "status" of the program after making some number of comparisons of keys. We can also see a node as representing those initial orderings of that data that will bring the program to this "status." Thus a program "status" is essentially knowledge about the initial ordering gathered so far by the program.

If any node represents two or more possible initial orderings, then the program cannot yet know the correct sorted order, and so must make another comparison of keys, such as "is k1<k2?" We may then create two children for the node. The left child represents those initial orderings that are consistent with the outcome that k1<k2; the right child represents the orderings that are consistent with the fact that k1>k2.† Thus each child represents a "status" consisting of the information known at the parent plus either the fact k1<k2 or the fact k1>k2, depending on whether the child is a left or right child, respectively.

Example 8.8. Consider the insertion sort algorithm with n = 3. Suppose that initially A[1], A[2], and A[3] have key values a, b, and c, respectively. Any of the six orderings of a, b, and c could be the correct one, so we begin construction of our decision tree with the node labeled (1) in Fig. 8.23, which represents all possible orderings. The insertion sort algorithm first compares A[2] with A[1], that is, b with a. If b turns out to be the smaller, then the correct ordering can only be bac, bca, or cba, the three orderings in which b precedes a. These three orderings are represented by node (2) in Fig. 8.23. The other three orderings are represented by node (3), the right child of (1). They are the orderings for which a precedes b.

Fig. 8.23. Decision tree for insertion sort with n = 3.

Now consider what happens if the initial data is such that we reach node (2). We have just swapped A[1] with A[2], and find A[2] can rise no higher since it is at the "top" already. The current ordering of the elements is bac. Insertion sort next begins inserting A[3] into its rightful place, by comparing A[3] with A[2]. Since A[2] now holds a and A[3] holds c, we are comparing c with a; node (4) represents the two orderings from node (2) in which c precedes a, while (5) represents the one ordering where it does not.

If the algorithm reaches the status of node (5), it will end, since it has already moved A[3] as high as it can go. On the other hand in the status of node (4), we found A[3]<A[2], and so swapped them, leaving b in A[1] and c in A[2]. Insertion sort next compares these two elements and swaps them if c<b. Nodes (8) and (9) represents the orderings consistent with c<b and its opposite, respectively, as well as the information gathered going from nodes (1) to (2) to (4), namely b<a and c<a. Insertion sort ends, whether or not it has swapped A[2] with A[1], making no more comparisons. Fortunately, each of the leaves (5), (8), and (9) represent a single ordering, so we have gathered enough information to determine the correct sorted order.

The description of the tree descending from node (3) is quite symmetric with what we have seen, and we omit it. Figure 8.23, since all its leaves are associated with a single ordering, is seen to determine the correct sorted order of three key values a, b, and c.

The Size of Decision Trees

Figure 8.23 has six leaves, corresponding to the six possible orderings of the initial list a, b, c. In general, if we sort a list of n elements, there are n! = n(n - 1)(n - 2) � � � (2)(1) possible outcomes, which are the correct sorted orders for the initial list a1, a2, . . . , an. That is, any of the n elements could be first, any of the remaining n - 1 second, any of the remaining n - 2 third, and so on. Thus any decision tree describing a correct sorting algorithm working on a list of n elements must have at least n! leaves, since each possible ordering must be alone at a leaf. In fact, if we delete nodes corresponding to unnecessary comparisons, and if we delete leaves that correspond to no ordering at all (since the leaves can only be reached by an inconsistent series of comparison results), there will be exactly n! leaves.

Binary trees that have many leaves must have long paths. The length of a path from the root to a leaf gives the number of comparisons made when the ordering represented by that leaf is the sorted order for a certain input list L. Thus, the length of the longest path from the root to a leaf is a lower bound on the number of steps performed by the algorithm in the worst case. Specifically, if L is the input list, the algorithm will make at least as many comparisons as the length of the path, in addition, probably, to other steps that are not comparisons of keys.

We should therefore ask how short can all paths be in a binary tree with k leaves. A binary tree in which all paths are of length p or less can have 1 root, 2 nodes at level 1, 4 nodes at level 2, and in general, 2i nodes at level i. Thus, the largest number of leaves of a tree with no nodes at levels higher than p is 2p. Put another way, a binary tree with k leaves must have a path of length at least logk. If we let k = n!, then we know that any sorting algorithm that only uses comparisons to determine the sorted order must take W(log(n!)) time in the worst case.

But how fast does log(n!) grow? A close approximation to n! is (n/e)n, where e = 2.7183 � � � is the base of natural logarithms. Since log((n/e)n) = nlogn-nloge, we see that log(n!) is on the order of nlogn. We can get a precise lower bound by noting that n(n - 1) � � � (2)(1) is the product of at least n/2 factors that are each at least n/2. Thus n! � (n/2)n/2. Hence log(n!) � (n/2)log(n/2) = (n/2)logn - n/2. Thus sorting by comparisons requires W(nlogn) time in the worst case.

The Average Case Analysis

One might wonder if there could be an algorithm that used only comparisons to sort, and took W(nlogn) time in the worst case, as all such algorithms must, but on the average took time that was O(n) or something less than O(nlogn). The answer is no, and we shall only mention how to prove the statement, leaving the details to the reader.

What we want to prove is that in any binary tree with k leaves, the average depth of a leaf is at least logk. Suppose that were not the case, and let tree T be the counterexample with the fewest nodes. T cannot be a single node, because our statement says of one-leaf trees only that they have average depth of at least 0. Now suppose T has k leaves. A binary tree with k � 2 leaves looks either like the tree in Fig. 8.24(a) or that in (b).

Fig. 8.24. Possible forms of binary tree T.

Figure 8.24(a) cannot be the smallest counterexample, because the tree rooted at n1 has as many leaves as T, but even smaller average depth. If Fig. 8.24(b) were T, then the trees rooted at n1 and n2, being smaller then T, would not violate our claim. That is, the average depth of leaves in T1 is at least log(k1), and the average depth in T2 is at least log(k2). Then the average depth in T is

Since k1 + k2 = k, we can express the average depth as

The reader can check that when k1 = k2 = k/2, (8.13) has value logk. The reader must show that (8.13) has a minimum when k1 = k2, given the constraint that k1 + k2 = k. We leave this proof as an exercise to the reader skilled in differential calculus. Granted that (8.13) has a minimum value of logk, we see that T was not a counterexample at all.

8.7 Order Statistics

The problem of computing order statistics is, given a list of n records and an integer k, to find the key of the record that is kth in the sorted order of the records. In general, we refer to this problem as "finding the kth out of n." Special cases occur when k = 1 (finding the minimum), k = n (finding the maximum), and the case where n is odd and k = (n + 1)/2, called finding the median.

Certain cases of the problem are quite easy to solve in linear time. For example, finding the minimum of n elements in O (n) time requires no special insight. As we mentioned in connection with heapsort, if k � n/logn then we can find the kth out of n by building a heap, which takes O (n) time, and then selecting the k smallest elements in O(n + k log n) = O(n) time. Symmetrically, we can find the kth of n in O(n) time when k � n - n/log n.

A Quicksort Variation

Probably the fastest way, on the average, to find the kth out of n is to use a recursive procedure similar to quicksort, call it select(i, j, k), that finds the kth element among A[i], . . . , A[j] within some larger array A [1], . . . , A [n]. The basic steps of select are:

  1. Pick a pivot element, say v.

  2. Use the procedure partition from Fig. 8.13 to divide A[i], . . . , A [j] into two groups: A[i], . . . , A[m - 1] with keys less than v, and A [m], . . . , A [j] with keys v or greater.

  3. If k � m - i, so the kth among A [i], . . . , A [j] is in the first group, then call select(i, m - 1, k) . If k > m - i, then call select(m, j, k - m + i).

Eventually we find that select(i, j, k) is called, where all of A[i], . . . , A [j] have the same key (usually because j=i). Then we know the desired key; it is the key of any of these records.

As with quicksort, the function select outlined above can take W(n2) time in the worst case. For example, suppose we are looking for the first element, but by bad luck, the pivot is always the highest available key. However, on the average, select is even faster than quicksort; it takes O(n) time. The principle is that while quicksort calls itself twice, select calls itself only once. We could analyze select as we did quicksort, but the mathematics is again complex, and a simple intuitive argument should be convincing. On the average, select calls itself on a subarray half as long as the subarray it was given. Suppose that to be conservative, we say that each call is on an array 9/10 as long as the previous call. Then if T(n) is the time spent by select on an array of length n, we know that for some constant c we have

Using techniques from the next chapter, we can show that the solution to (8.14) is T(n) = O(n).

A Worst-Case Linear Method for Finding Order Statistics

To guarantee that a function like select has worst case, rather than average case, complexity O(n), it suffices to show that in linear time we can find some pivot that is guaranteed to be some positive fraction of the way from either end. For example, the solution to (8.14) shows that if the pivot of n elements is never less than the [image] element, nor greater than the [image] element, so the recursive call to select is on at most nine-tenths of the array, then this variant of select will be O(n) in the worst case.

The trick to finding a good pivot is contained in the following two steps.

  1. Divide the n elements into groups of 5, leaving aside between 0 and 4 elements that cannot be placed in a group. Sort each group of 5 by any algorithm and take the middle element from each group, a total of �n/5� elements.

  2. Use select to find the median of these �n/5� elements, or if �n/5� is even, an element in a position as close to the middle as possible. Whether �n/5� is even or odd, the desired element is in position [image].

This pivot is far from the extremes, provided that not too many records have the pivot for key.† To simplify matters let us temporarily assume that all keys are different. Then we claim the chosen pivot, the [image] element of the [image] middle elements out of groups of 5 is greater than at least [image] of the n elements. For it exceeds [image] of the middle elements, and each of those exceeds two more, from the five of which it is the middle. If n � 75, then [image] is at least n/4. Similarly, we may check that the chosen pivot is less than or equal to at least [image] elements, so for n � 75, the pivot lies between the 1/4 and 3/4 point in the sorted order. Most importantly, when we use the pivot to partition the n elements, the kth element will be isolated to within a range of at most 3n/4 of the elements. A sketch of the complete algorithm is given in Fig. 8.25; as for the sorting algorithms it assumes an array A[1], . . . , A[n] of recordtype, and that recordtype has a field key of type keytype. The algorithm to find the kth element is just a call to select(1,n,k), of course.

    function select ( i, j, k: integer ): keytype;
        { returns the key of the kth element in sorted order
            among A[i], . . . ,A[j] }
        var
            m: integer; { used as index }
            begin
(1)             if j-i < 74 then begin { too few to use select recursively }
(2)                  sort A[i], . . . ,A[j] by some simple algorithm;
(3)                  return (A[i + k - 1 ].key)
                end
                else begin { apply select recursively }
(4)                  for m := 0 to (j - i - 4) div 5 do
                         { get the middle elements of groups of 5
                             into A[i], A[i + 1], . . . }
(5)                      find the third element among A[i + 5*m] through
                             A[i + 5*m + 4] and swap it with A[i +
m];
(6)                  pivot := select(i, i+(j-i-4) div 5, (j -
i - 4) div 10);
                         { find median of middle elements. Note j - i - 4
                             here is n - 5 in the informal description above }
(7)                  m := partition(i, j, pivot);
(8)                  if k <= m - i then
(9)                       return (select(i, m - 1, k))
                     else
(10)                      return (select(m, j, k-(m - i)))
                end
        end; { select }

Fig. 8.25. Worst-case linear algorithm for finding kth element.

To analyze the running time of select of Fig. 8.25, let n be j - i + 1. Lines (2) and (3) are only executed if n is 74 or less. Thus, even though step (2) may take O(n2) steps in general, there is some constant c1, however large, such that for n � 74, lines (1 - 3) take no more than c1 time.

Now consider lines (4 - 10). Line (7), the partition step, was shown in connection with quicksort to take O(n) time. The loop of lines (4 - 5) is iterated about n/5 times, and each execution of line (5), requiring the sorting of 5 elements, takes some constant time, so the loop takes O(n) time as a whole.

Let T(n) be the time taken by a call to select on n elements. Then line (6) takes at most T(n/5) time. Since n � 75 when line (10) is reached, and we have argued that if n � 75, at most 3n/4 elements are less than the pivot, and at most 3n/4 are equal to or greater than the pivot, we know that line (9) or (10) takes time at most T(3n/4). Hence, for some constants c1 and c2 we have

The term c2n in (8.15) represents lines (1), (4), (5), and (7); the term T(n/5) comes from line (6), and T(3n/4) represents lines (9) and (10).

We shall show that T(n) in (8.15) is O(n). Before we proceed, the reader should now appreciate that the "magic number" 5, the size of the groups in line (5), and our selection of n = 75 as the breakpoint below which we did not use select recursively, were designed so the arguments n/5 and 3n/4 of T in (8.15) would sum to something less than n. Other choices of these parameters could be made, but observe when we solve (8.15) how the fact that 1/5 + 3/4 < 1 is needed to prove linearity.

Equation (8.15) can be solved by guessing a solution and verifying by induction that it holds for all n. We shall chose a solution of the form cn, for some constant c. If we pick c � c1, we know T(n) � cn for all n between 1 and 74, so consider the case n � 75. Assume by induction that T(m) � cm for m<n. Then by (8.15),

T(n) � c2n + cn/5 + 3cn/4 � c2n + 19cn/20 (8.16)

If we pick c = max(c1, 20c2), then by (8.16), we have T(n) � cn/20 + cn/5 + 3cn/4 = cn, which we needed to show. Thus, T(n) is O(n).

The Case Where Some Equalities Among Keys Exist

Recall that we assumed no two keys were equal in Fig. 8.25. The reason this assumption was needed is that otherwise line (7) cannot be shown to partition A into blocks of size at most 3n/4. The modification we need to handle key equalities is to add, after step (7), another step like partition, that groups together all records with key equal to the pivot. Say there are p � 1 such keys. If m - i � k � m - i + p, then no recursion is necessary; simply return A[m].key. Otherwise, line (8) is unchanged, but line (10) calls select(m + p, j, k - (m - i) - p).

Exercises

  1. What modifications to the quicksort algorithm of Fig. 8.14 do we have to make to avoid infinite loops when there is a sequence of equal elements?
  2. Show that the modified quicksort has average-case running time O(n log n).
8.1 | Here are eight integers: 1, 7, 3, 2, 0, 5, 0, 8. Sort them using (a) bubblesort, (b) insertion sort, and (c) selection sort.
8.2 | Here are sixteen integers: 22, 36, 6, 79, 26, 45, 75, 13, 31, 62, 27, 76, 33, 16, 62, 47. Sort them using (a) quicksort, (b) insertion sort, (c) heapsort, and (d) bin sort, treating them as pairs of digits in the range 0-9.
8.3 The procedure Shellsort of Fig. 8.26, sometimes called diminishing-increment sort, sorts an array A[1..n] of integers by sorting n/2 pairs (A[i],A[n/2 + i]) for 1 � i � n/2 in the first pass, n/4 four-tuples (A[i], A[n/4 + i], A[n/2 + i], A[3n/4 + i]) for 1 � i � n/4 in the second pass, n/8 eight-tuples in the third pass, and so on. In each pass the sorting is done using insertion sort in which we stop sorting once we encounter two elements in the proper order.

     procedure Shellsort ( var A: array[l..n]
of integer );
    var
       i, j, incr: integer;
    begin
       incr := n div 2;
       while incr > 0 do begin
          for i := incr + l to n do begin
              j := i - incr;
              while j > 0 do
                 if A[j] > A[j + incr] then begin
                     swap(A[j], A[j + incr]);
                     j := j - incr
                 end
                 else
                     j := 0 { break }
          end;
          incr := incr div 2
       end
    end; { Shellsort }

Fig. 8.26. Shellsort.

  1. Sort the sequences of integers in Exercises 8.1 and 8.2 using Shellsort.
  2. Show that if A[i] and A[n/2k + i] became sorted in pass k (i.e., they were swapped), then these two elements remain sorted in pass k + 1.
  3. The distances between elements compared and swapped in a pass diminish as n/2, n/4, . . . , 2, 1 in Fig. 8.26. Show that Shellsort will work with any sequence of distances as long as the last distance is 1.
  4. Show that Shellsort works in O (n1.5) time.
*8.4 | Suppose you are to sort a list L consisting of a sorted list followed by a few "random" elements. Which of the sorting methods discussed in this chapter would be especially suitable for such a task?
*8.5 | A sorting algorithm is stable if it preserves the original order of records with equal keys. Which of the sorting methods in this chapter are stable?
*8.6 | Suppose we use a variant of quicksort where we always choose as the pivot the first element in the subarray being sorted.
8.7 Show that any sorting algorithm that moves elements only one position at a time must have time complexity at least W(n2).
8.8 In heapsort the procedure pushdown of Fig. 8.17 establishes the partially ordered tree property in time O(n). Instead of starting at the leaves and pushing elements down to form a heap, we could start at the root and push elements up. What is the time complexity of this method?
*8.9 Suppose we have a set of words, i.e., strings of the letters a - z, whose total length is n. Show how to sort these words in O(n) time. Note that if the maximum length of a word is constant, binsort will work. However, you must consider the case where some of the words are very long.
*8.10 Show that the average-case running time of insertion sort is W(n2).
*8.11 Consider the following algorithm randomsort to sort an array A[1..n] of integers: If the elements A[1], A[2], . . . , A[n] are in sorted order, stop; otherwise, choose a random number i between 1 and n, swap A[1] and A[i], and repeat. What is the expected running time of randomsort?
*8.12 We showed that sorting by comparisons takes W(n log n) comparisons in the worse case. Prove that this lower bound holds in the average case as well.
*8.13 Prove that the procedure select, described informally at the beginning of Section 8.7, has average case running time of O(n).
*8.14 Implement CONCATENATE for the data structure of Fig. 8.19.
*8.15 Write a program to find the k smallest elements in an array of length n. What is the time complexity of your program? For what value of k does it become advantageous to sort the array?
*8.16 Write a program to find the largest and smallest elements in an array. Can this be done in fewer than 2n - 3 comparisons?
*8.17 Write a program to find the mode (the most frequently occurring element) of a list of elements. What is the time complexity of your program?
*8.18 Show that any algorithm to purge duplicates from a list requires at least W(n log n) time under the decision tree model of computation of Section 8.6.
*8.19 Suppose we have k sets, S1, S2, . . . , Sk, each containing n real numbers. Write a program to list all sums of the form s1 + s2 + . . . + sk, where si is in Si, in sorted order. What is the time complexity of your program?
*8.20 Suppose we have a sorted array of strings s1, s2, . . . , sn. Write a program to determine whether a given string x is a member of this sequence. What is the time complexity of your program as a function of n and the length of x?

Bibliographic Notes

Knuth [1973] is a comprehensive reference on sorting methods. Quicksort is due to Hoare [1962] and subsequent improvements to it were published by Singleton [1969] and Frazer and McKellar [1970]. Heapsort was discovered by Williams [1964] and improved by Floyd [1964]. The decision tree complexity of sorting was studied by Ford and Johnson [1959]. The linear selection algorithm in Section 8.7 is from Blum, Floyd, Pratt, Rivest, and Tarjan [1972].

Shellsort is due to Shell [1959] and its performance has been analyzed by Pratt [1979]. See Aho, Hopcroft, and Ullman [1974] for one solution to Exercise 8.9.

† Technically, quicksort is only O(n log n) in the average case; it is O(n2) in the worst case.

‡We could copy A[i], . . . ,A[j] and arrange them as we do so, finally copying the result back into A[i], . . . ,A[j]. We choose not to do so because that approach would waste space and take longer than the in-place arrangement method we use.

† If there is reason to believe nonrandom orders of elements might make quicksort run slower than expected, the quicksort program should permute the elements of the array at random before sorting.

† Since we only want the median, not the entire sorted list of k elements, it may be better to use one of the fast median-finding algorithms of Section 8.7.

† We could at the end, reverse array A, but if we wish A to end up sorted lowest first, then simply apply a DELETEMAX operator in place of DELETEMIN, and partially order A in such a way that a parent has a key no smaller (rather than no larger) than its children.

† In fact, this time is O(n), by a more careful argument. For j in the range n/2 to n/4+1, (8.8) says only one iteration of pushdown's while-loop is needed. For j between n/4 and n/8+1, only two iterations, and so on. The total number of iterations as j ranges between n/2 and 1 is bounded by [image] Note that the improved bound for lines (1 - 2) does not imply an improved bound for heapsort as a whole; all the time is taken in lines (3 - 5).

† Note that a sequence ranging from 1 to 0 (or more generally, from x to y, where y < x) is deemed to be an empty sequence.

† But in this case, if log n-bit integers can fit in one word, we are better off treating keys as consisting of one field only, of type l..n, and using ordinary binsort.

† We may as well assume all keys are different, since if we can sort a collection of distinct keys we shall surely produce a correct order when some keys are the same.

† In the extreme case, when all keys are equal, the pivot provides no separation at all. Obviously the pivot is the kth element for any k, and another approach is needed.

Table of Contents Go to Chapter 9


Data Structures and Algorithms: CHAPTER 9: Algorithm Analysis Techniques

 Next Chapter Return to Table of Contents Previous Chapter

Algorithm Analysis Techniques

What is a good algorithm? There is no easy answer to this question. Many of the criteria for a good algorithm involve subjective issues such as simplicity, clarity, and appropriateness for the expected data. A more objective, but not necessarily more important, issue is run-time efficiency. Section 1.5 covered the basic techniques for establishing the running time of simple programs. However, in more complex cases such as where programs are recursive, some new techniques are needed. This short chapter presents some general techniques for solving recurrence equations that arise in the analysis of the running times of recursive algorithms.

9.1 Efficiency of Algorithms

One way to determine the run-time efficiency of an algorithm is to program it and measure the execution time of the particular implementation on a specific computer for a selected set of inputs. Although popular and useful, this approach has some inherent problems. The running times depend not only on the underlying algorithm, but also on the instruction set of the computer, the quality of the compiler, and the skill of the programmer. The implementation may also be tuned to work well on the particular set of test inputs chosen. These dependencies may become strikingly evident with a different computer, a different compiler, a different programmer, or a different set of test inputs. To overcome these objections, computer scientists have adopted asymptotic time complexity as a fundamental measure of the performance of an algorithm. The term efficiency will refer to this measure, and particularly to the worst-case (as opposed to average) time complexity.

The reader should recall from Chapter 1 the definitions of O(f(n)) and W(f(n)). The efficiency, i.e., worst-case complexity, of an algorithm is said to be O(f(n)), or just f(n) for short, if the function of n that gives the maximum, over all inputs of length n, of the number of steps taken by the algorithm on that input, is O(f(n)). Put another way, there is some constant c such that for sufficiently large n, cf(n) is an upper bound on the number of steps taken by the algorithm on any input of length n.

There is the implication in the assertion that "the efficiency of a given algorithm is f(n)" that the efficiency is also W(f(n)), so that f(n) is the slowest growing function of n that bounds the worst-case running time from above. However, this latter requirement is not part of the definition of O(f(n)), and sometimes it is not possible to be sure that we have the slowest growing upper bound.

Our definition of efficiency ignores constant factors in running time, and there are several pragmatic reasons for doing so. First, since most algorithms are written in a high level language, we must describe them in terms of "steps," which each take a constant amount of time when translated into the machine language of any computer. However, exactly how much time a step requires depends not only on the step itself, but on the translation process and the instruction repertoire of the machine. Thus to attempt to be more precise than to say that the running time of an algorithm is "on the order of f(n)", i.e., O(f(n)), would bog us down in the details of specific machines and would apply only to those machines.

A second important reason why we deal with asymptotic complexity and ignore constant factors is that the asymptotic complexity, far more than constant factors, determines for what size inputs the algorithm may be used to provide solutions on a computer. Chapter 1 discussed this viewpoint in detail. The reader should be alert, however, to the possibility that for some very important problems, like sorting, we may find it worthwhile to analyze algorithms in such detail that statements like "algorithm A should run twice as fast as algorithm B on a typical computer" are possible.

A second situation in which it may pay to deviate from our worst- case notion of efficiency occurs when we know the expected distribution of inputs to an algorithm. In such situations, average case analysis can be much more meaningful than worst case analysis. For example, in the previous chapter we analyzed the average running time of quicksort under the assumption that all permutations of the correct sorted order are equally likely to occur as inputs.

9.2 Analysis of Recursive Programs

In Chapter 1 we showed how to analyze the running time of a program that does not call itself recursively. The analysis for a recursive program is rather different, normally involving the solution of a difference equation. The techniques for solving difference equations are sometimes subtle, and bear considerable resemblance to the methods for solving differential equations, some of whose terminology we borrow.

Consider the sorting program sketched in Fig. 9.1. There the procedure mergesort takes a list of length n as input, and returns a sorted list as its output. The procedure merge(L1, L2) takes as input two sorted lists L1 and L2, scans them each, element by element, from the front. At each step, the larger of the two front elements is deleted from its list and emitted as output. The result is a single sorted list containing the elements of L1 and L2. The details of merge are not important here, although we discuss this sorting algorithm in detail in Chapter 11. What is important is that the time taken by merge on lists of length n/2 is O(n).

     function mergesort ( L: LIST; n: integer ): LIST;
         { L is a list of length n. A sorted version of L
           is returned. We assume n is a power of 2. }
         var
              L1, L2: LIST
         begin
              if n = 1 then
                  return (L);
              else begin
                  break L into two halves, L1 and L2,
each of length n/2;
                  return (merge (mergesort (L1,n/2),
mergesort(L2, n/2)));
              end
         end; { mergesort }

Fig. 9.1. Recursive procedure mergesort.

Let T(n) be the worst case running time of the procedure mergesort of Fig. 9.1. We can write a recurrence (or difference) equation that upper bounds T(n), as follows

The term c1 in (9.1) represents the constant number of steps taken when L has length 1. In the case that n > 1, the time taken by mergesort can be divided into two parts. The recursive calls to mergesort on lists of length n/2 each take time T(n/2), hence the term 2T(n/2). The second part consists of the test to discover that n ¹ 1, the breaking of list L into two equal parts and the procedure merge. These three operations take time that is either a constant, in the case of the test, or proportional to n for the split and the merge. Thus the constant c2 can be chosen so the term c2n is an upper bound on the time taken by mergesort to do everything except the recursive calls. We now have equation (9.1).

Observe that (9.1) applies only when n is even, and hence it will provide an upper bound in closed form (that is, as a formula for T(n) not involving any T(m) for m < n) only if n is a power of 2. However, even if we only know T(n) when n is a power of 2, we have a good idea of T(n) for all n. In particular, for essentially all algorithms, we may suppose that T(n) lies between T(2i) and T(2i+1) if n lies between 2i and 2i+1. Moreover, if we devote a little more effort to finding the solution, we could replace the term 2T(n/2) in (9.1) by T((n+1)/2) + T((n-1)/2) for odd n > 1. Then we could solve the revised difference equation to get a closed form solution for all n.

9.3 Solving Recurrence Equations

There are three different approaches we might take to solving a recurrence equation.

  1. Guess a solution f(n) and use the recurrence to show that T(n) � f(n). Sometimes we guess only the form of f(n), leaving some parameters unspecified (e.g., guess f(n) = an2 for some a) and deduce suitable values for the parameters as we try to prove T(n) � f(n) for all n.
  2. Use the recurrence itself to substitute for any T(m), m<n, on the right until all terms T(m) for m>1 have been replaced by formula involving only T(1). Since T(1) is always a constant, we have a formula for T(n) in terms of n> and constants. This formula is what we have referred to as a "closed form" for T(n).
  3. Use the general solution to certain recurrence equations of common types found in this section or elsewhere (see the bibliographic notes).

This section examines the first two methods.

Guessing a Solution

Example 9.1. Consider method (1) applied to Equation (9.1). Suppose we guess that for some a, T(n) = anlogn. Substituting n = 1, we see that this guess will not work, because anlogn has value 0, independent of the value of a. Thus, we might next try T(n) = anlogn + b. Now n = 1 requires that b � c1.

For the induction, we assume that

        T(k) � aklogk +
b			(9.2)

for all k < n and try to establish that

        T(n) � anlogn +
b

To begin our proof, assume n � 2. From (9.1), we have

        T(n) �
2T(n/2)+c2n

From (9.2), with k = n/2, we obtain

provided a � c2 + b.

We thus see that T(n) � anlogn + b holds provided two constraints are satisfied: b � c1 and a � c2 + b. Fortunately, there are values we can choose for a that satisfy these two constraints. For example, choose b = c1 and a = c1 + c2. Then, by induction on n, we conclude that for all n � 1

        T(n) � (c1 +
c2)nlogn + c1		(9.4)

In other words, T(n) is O(nlogn).

Two observations about Example 9.1 are in order. If we assume that T(n) is O(f(n)), and our attempt to prove T(n) � cf(n) by induction fails, it does not follow that T(n) is not O(f(n)). In fact, an inductive hypothesis of the form T(n) � cf(n) - 1 may succeed!

Secondly we have not yet determined the exact asymptotic growth rate for f(n), although we have shown that it is no worse than O(nlogn). If we guess a slower growing solution, like f(n) = an, or f(n) = anloglogn, we cannot prove the claim that T(n) � f(n). However, the matter can only be settled conclusively by examining mergesort and showing that it really does take W(nlogn) time; in fact, it takes time proportional to nlogn on all inputs, not just on the worst possible inputs. We leave this observation as an exercise.

Example 9.1 exposes a general technique for proving some function to be an upper bound on the running time of an algorithm. Suppose we are given the recurrence equation

    T(1) = c

    T(n) � g(T(n/2),
n), for  n > 1			(9.5)

Note that (9.5) generalizes (9.1), where g(x, y) is 2x + c2y. Also observe that we could imagine equations more general than (9.5). For example, the formula g might involve all of T(n-1), T(n-2), . . . , T(1), not just T(n/2). Also, we might be given values for T(1), T(2), . . . , T(k), and the recurrence would then apply only for n > k. The reader can, as an exercise, consider how to solve these more general recurrences by method (1) -- guessing a solution and verifying it.

Let us turn our attention to (9.5) rather than its generalizations. Suppose we guess a function f(a1, . . . , aj, n), where a1, . . . , aj are parameters, and attempt to prove by induction on n that T(n) � f(a1, . . . , aj, n). For example, our guess in Example 9.1 was f(a1, a2, n) = a1nlogn + a2, but we used a and b for a1 and a2. To check that for some values of a1, . . . , aj we have T(n) � f(a1, . . . , aj, n) for all n � 1, we must satisfy

   f(a1, . . . , aj, 1) � c

   f(a1, . . . , aj, n) � g(f(a1, . . . ,
aj, n/2), n)   (9.6)

That is, by the inductive hypothesis, we may substitute f for T on the right side of the recurrence (9.5) to get

    T(n) �
g(f(a1, . . . , aj, n/2),
n)		(9.7)

When the second line of (9.6) holds, we can combine it with (9.7) to prove that T(n) � f(a1, . . . , an, n), which is what we wanted to prove by induction on n.

For example, in Example 9.1, we have g(x, y) = 2x + c2y and f(a1, a2, n) = a1nlogn + a2. Here we must try to satisfy

As we discussed, a2 = c1 and a1 = c1 + c2 is a satisfactory choice.

Expanding Recurrences

If we cannot guess a solution, or we are not sure we have the best bound on T(n), we can use a method that in principle always succeeds in solving for T(n) exactly, although in practice we often have trouble summing series and must resort to computing an upper bound on the sum. The general idea is to take a recurrence like (9.1), which says T(n) � 2T(n/2) + c2n, and use it to get a bound on T(n/2) by substituting n/2 for n. That is

        T(n/2) � 2T(n/4) +
c2n/2		(9.8)

Substituting the right side of (9.8) for T(n/2) in (9.1) yields

   T(n) � 2(2T(n/4) +
c2n/2) + c2n = 4T(n/4) +
2c2n	(9.9)

Similarly, we could substitute n/4 for n in (9.1) and use it to get a bound on T(n/4). That bound, which is 2T(n/8) + c2n/4, can be substituted in the right of (9.9) to yield

        T(n) �
8T(n/8)+3c2n			(9.10)

Now the reader should perceive a pattern. By induction on i we can obtain the relationship

        T(n) �
2iT(n/2i)+ic2n			(9.11)

for any i. Assuming n is a power of 2, say 2k, this expansion process comes to a halt as soon as we get T(1) on the right of (9.11). That will occur when i = k, whereupon (9.11) becomes

        T(n) �
2kT(1) + kc2n
    (9.12)

Then, since 2k = n, we know k = logn. As T(1) � c1, (9.12) becomes

        T(n) �
c1n + c2nlogn
    (9.13)

Equation (9.13) is actually as tight a bound as we can put on T(n), and proves that T(n) is O(nlogn).

9.4 A General Solution for a Large Class of Recurrences

Consider the recurrence that one gets by dividing a problem of size n into a subproblems each of size n/b. For convenience, we assume that a problem of size 1 takes one time unit and that the time to piece together the solutions to the subproblems to make a solution for the size n problem is d(n), in the same time units. For our mergesort example, we have a = b = 2, and d(n) = c2n/c1, in units of c1. Then if T(n) is the time to solve a problem of size n, we have

       T(1) = 1

       T(n) = aT(n/b) + d(n)
(9.14)

Note that (9.14) only applies to n's that are an integer power of b, but if we assume T(n) is smooth, getting a tight upper bound on T(n) for those values of n tells us how T(n) grows in general.

Also note that we use equality in (9.14) while in (9.1) we had an inequality. The reason is that here d(n) can be arbitrary, and therefore exact, while in (9.1) the assumption that c2n was the worst case time to merge, for one constant c2 and all n, was only an upper bound; the actual worst case running time on inputs of size n may have been less than 2T(n/2) + c2n. Actually, whether we use = or � in the recurrence makes little difference, since we wind up with an upper bound on the worst case running time anyway.

To solve (9.14) we use the technique of repeatedly substituting for T on the right side, as we did for a specific example in our previous discussion of expanding recurrences. That is, substituting n/bi for n in the second line of (9.14) yields

Thus, starting with (9.14) and substituting (9.15) for i = 1, 2, . . . , we get

Now, if we assume n = bk, we can use the fact that T(n/bk) = T(1) = 1 to get from the above, with i = k, the formula

If we use the fact that k = logbn, the first term of (9.16) can be written as alogbn , or equivalently nlogba (take logarithms to the base b of both expressions to see that they are the same). This expression is n to a constant power. For example, in the case of mergesort, where a = b = 2, the first term is n. In general, the larger a is, i.e., the more subproblems we need to solve, the higher the exponent will be; the higher b is, i.e., the smaller each subproblem is, the lower will be the exponent.

Homogeneous and Particular Solutions

It is interesting to see the different roles played by the two terms in (9.16). The first, ak or nlogba, is called the homogeneous solution, in analogy with differential equation terminology. The homogeneous solution is the exact solution when d(n), called the driving function, is 0 for all n. In other words, the homogeneous solution represents the cost of solving all the subproblems, even if subproblems can be combined "for free."

On the other hand, the second term of (9.16) represents the cost of creating the subproblems and combining their results. We call this term the particular solution. The particular solution is affected by both the driving function and the number and size of subproblems. As a rule of thumb if the homogeneous solution is greater than the driving function, then the particular solution has the same growth rate as the homogeneous solution. If the driving function grows faster than the homogeneous solution by more than n� for some � > 0, then the particular solution has the same growth rate as the driving function. If the driving function has the same growth rate as the homogeneous solution, or grows faster by at most logkn for some k, then the particular solution grows as logn times the driving function.

It is important to recognize that when searching for improvements in an algorithm, we must be alert to whether the homogeneous solution is larger than the driving function. For example, if the homogeneous solution is larger, then finding a faster way to combine subproblems will have essentially no effect on the efficiency of the overall algorithm. What we must do in that case is find a way to divide a problem into fewer or smaller subproblems. That will affect the homogeneous solution and lower the overall running time.

If the driving function exceeds the homogeneous solution, then one must try to decrease the driving function. For example, in the mergesort case, where a = b = 2, and d(n) = cn, we shall see that the particular solution is O(nlogn). However, reducing d(n) to a slightly sublinear function, say n0.9, will, as we shall see, make the particular solution less than linear as well and reduce the overall running time to O(n), which is the homogeneous solution. †

Multiplicative Functions

The particular solution in (9.16) is hard to evaluate, even if we know what d(n) is. However, for certain common functions d(n), we can solve (9.16) exactly, and there are others for which we can get a good upper bound. We say a function f on integers is multiplicative if f(xy) = f(x)f(y) for all positive integers x and y.

Example 9.2. The multiplicative functions that are of greatest interest to us are of the form na for any positive a. To prove f(n) = na is multiplicative, we have only to observe that (xy)a = xaya.

Now if d(n) in (9.16) is multiplicative, then d(bk-j) = (d(b))k-j, and the particular solution of (9.16) is

There are three cases to consider, depending on whether a is greater than, less than, or equal to d(b).

  1. If a > d(b), then the formula (9.17) is O(ak), which we recall is nlogba, since k = logbn. In this case, the particular and homogeneous solutions are the same, and depend only on a and b, and not on the driving function d. Thus, improvements in the running time must come from decreasing a or increasing b; decreasing d(n) is of no help.
  2. If a < d(b), then (9.17) is O(d(b)k), or equivalently O(nlogbd(b)). In this case, the particular solution exceeds the homogeneous, and we may also look to the driving function d(n) as well as to a and b, for improvements. Note the important special case where d(n) = na. Then d(b) = ba, and logb(ba) = a. Thus the particular solution is O(na), or O(d(n)).
  3. If a = d(b), we must reconsider the calculation involved in (9.17), as our formula for the sum of a geometric series is now inappropriate. In this case we have

Since a = d(b), the particular solution given by (9.18) is logbn times the homogeneous solution, and again the particular solution exceeds the homogeneous. In the special case d(n) = na, (9.18) reduces to O(nalogn), by observations similar to those in Case (2).

Example 9.3. Consider the following recurrences, with T(1) = 1.

  1. T(n) = 4T(n/2) + n
  2. T(n) = 4T(n/2) + n2
  3. T(n) = 4T(n/2) + n3

In each case, a = 4, b = 2, and the homogeneous solution is n2. In Equation (1), with d(n) = n, we have d(b) = 2. Since a = 4 > d(b), the particular solution is also n2, and T(n) is O(n2) in (1).

In Equation (3), d(n) = n3 d(b) = 8, and a < d(b). Thus the particular solution is O(nlogbd(b)) = O(n3), and T(n) of Equation (3) is O(n3). We can deduce that the particular solution is of the same order as d(n) = n3, using the observations made above about d(n)'s of the form na in analyzing the case a < d(b) of (9.17). In Equation (2) we have d(b) = 4 = a, so (9.18) applies. As d(n) is of the form na, the particular solution, and therefore T(n) itself, is O(n2 log n).

Other Driving Functions

These are other functions that are not multiplicative, yet for which we can get solutions for (9.16) or even (9.17). We shall consider two examples. The first generalizes to any function that is the product of a multiplicative function and a constant greater than or equal to one. The second is typical of a case where we must examine (9.16) in detail and get a close upper bound on the particular solution.

Example 9.4. Consider

    T(1) = 1

    T(n) = 3T(n/2) + 2n1.5

Now 2n1.5 is not multiplicative, but n1.5 is. Let U(n) = T(n)/2 for all n. Then

    U(1) = 1/2

    U(n) = 3U(n/2) + n1.5

The homogeneous solution, if U(1) were 1, would be nlog 3 < n1.59; since U(1) = 1/2 we can easily show the homogeneous solution is less than n1.59/2; certainly it is O(n1.59). For the particular solution, we can ignore the fact that U(1) ¹ 1, since increasing U(1) will surely not lower the particular solution. Then since a = 3, b = 2, and b1.5 = 2.83 < a, the particular solution is also O(n1.59), and that is the growth rate of U(n). Since T(n) = 2U(n), T(n) is also O(n1.59) or O(nlog 3).

Example 9.5. Consider

    T(1) = 1

    T(n) = 2T(n/2) + n log n

The homogeneous solution is easily seen to be n, since a = b = 2. However, d(n) = n log n is not multiplicative, and we must sum the particular solution formula in (9.16) by ad hoc means. That is, we want to evaluate

Since k = log n we have the particular solution O (n log2n), and this solution, being greater than the homogeneous, is also the value of T(n) that we obtain.

Exercises

9.1 Write recurrence equations for the time and space complexity of the following algorithm, assuming n is a power of 2.

    function path ( s, t, n: integer ) : boolean;
        begin
        if n = 1 then
            if edge (s, t) then
             return (true)
            else
             return (false);
            { if we reach here, n > 1 }
        for i := 1 to n do
            if path(s, i, n div 2) and
path (i, t, n div 2) then
             return (true);
        return (false)
        end; { path }

The function edge(i, j) returns true if vertices i and j of an n-vertex graph are connected by an edge or if i = j; edge(i, j) returns false otherwise. What does the program do?
9.2 Solve the following recurrences, where T(1) = 1 and T(n) for n � 2 satisfies:
  1. T(n) = 3T(n/2) + n

  2. T(n) = 3T(n/2) + n2

  3. T(n) = 8T(n/2) + n3
9.3 Solve the following recurrences, where T(1) = 1 and T(n) for n � 2 satisfies:
  1. T(n) = 4T(n/3) + n

  2. T(n) = 4T(n/3) + n2

  3. T(n) = 9T(n/3) + n2
9.4 Give tight big-oh and big-omega bounds on T(n) defined by the following recurrences. Assume T(1) = 1.
  1. T(n) = T(n/2) + 1

  2. T(n) = 2T(n/2) + logn

  3. T(n) = 2T(n/2) + n

  4. T(n) = 2T(n/2) + n2
*9.5 Solve the following recurrences by guessing a solution and checking your answer.
  1. T(1) = 2
    T(n) = 2T(n-1) + 1 for n � 2

  2. T(1) = 1
    T(n) = 2T(n-1) + n for n � 2
9.6 | Check your answers to Exercise 9.5 by solving the recurrences by repeated substitution.
9.7 Generalize Exercise 9.6 by solving all recurrences of the form

    T(1) = 1
    T(n) = aT(n-1) + d(n)    for n � 1

in terms of a and d(n).
*9.8 | Suppose in Exercise 9.7 that d(n) = cn for some constant c � 1. How does the solution to T(n) depend on the relationship between a and c. What is T(n)?
**9.9 Solve for T(n):

    T(1) = 1
    T(n) = Ö`nT(Ö`n) + n    for n � 2
9.10 Find closed form expressions for the following sums.

*9.11 Show that the number of different orders in which to multiply a sequence of n matrices is given by the recurrence

Show that [image]. The T(n)'s are called Catalan numbers.
**9.12 Show that the number of comparisons required to sort n elements using mergesort is given by

   T(1) = 0
   T(n) = T([n/2]) + T([n/2]) + n - 1

where [x] denotes the integer part of x and [x] denotes the smallest integer � x. Show that the solution to this recurrence is

   T(n) = n[logn] - 2[logn] + 1
9.13 Show that the number of Boolean functions of n variables is given by the recurrence

   T(1) = 4
   T(n) = (T(n - 1))2

Solve for T(n).
**9.14 Show that the number of binary trees of height � n is given by the recurrence

   T(1) = 1
   T(n) = (T(n - 1))2 + 1

Show that T(n) = [k2n] for some constant k. What is the value of k?

Bibliographic Notes

Bentley, Haken, and Saxe [1978], Greene and Knuth [1983], Liu [1968], and Lueker [1980] contain additional material on the solution of recurrences. Aho and Sloane [1973] show that many nonlinear recurrences of the form T(n) = (T(n-1))2 + g(n) have a solution of the form T(n) = [k2n] where k is a constant, as in Exercise 9.14.

† But don't hold out much hope for discovering a way to merge two sorted lists of n/2 elements in less than linear time; we couldn't even look at all the elements on the list in that case.
Table of Contents Go to Chapter 10


Data Structures and Algorithms: CHAPTER 10: Algorithm Design Techniques

 Next Chapter Return to Table of Contents Previous Chapter

Algorithm Design Techniques

Over the years computer scientists have identified a number of general techniques that often yield effective algorithms in solving large classes of problems. This chapter presents some of the more important techniques, such as divide-and-conquer, dynamic programming, greedy techniques, backtracking, and local search. In trying to devise an algorithm to solve a given problem, it is often useful to ask a question such as "What kind of solution does divide-and-conquer, dynamic programming, a greedy approach, or some other standard technique yield?"

It should be emphasized, however, that there are problems, such as the NP-complete problems, for which these or any other known techniques will not produce efficient solutions. When such a problem is encountered, it is often useful to determine if the inputs to the problem have special characteristics that could be exploited in trying to devise a solution, or if an easily found approximate solution could be used in place of the difficult-to-compute exact solution.

10.1 Divide-and-Conquer Algorithms

Perhaps the most important, and most widely applicable, technique for designing efficient algorithms is a strategy called "divide-and-conquer." It consists of breaking a problem of size n into smaller problems in such a way that from solutions to the smaller problems we can easily construct a solution to the entire problem. We have already seen a number of applications of this technique, such as mergesort or binary search trees.

To illustrate the method consider the familiar "towers of Hanoi" puzzle. It consists of three pegs A, B, and C. Initially peg A has on it some number of disks, starting with the largest one on the bottom and successively smaller ones on top, as shown in Fig. 10.1. The object of the puzzle is to move the disks one at a time from peg to peg, never placing a larger disk on top of a smaller one, eventually ending with all disks on peg B.

One soon learns that the puzzle can be solved by the following simple algorithm. Imagine the pegs arranged in a triangle. On odd-numbered moves, move the smallest disk one peg clockwise. On even-numbered moves

Fig. 10.1. Initial position in towers of Hanoi puzzle.

make the only legal move not involving the smallest disk.

The above algorithm is concise, and correct, but it is hard to understand why it works, and hard to invent on the spur of the moment. Consider instead the following divide-and-conquer approach. The problem of moving the n smallest disks from A to B can be thought of as consisting of two subproblems of size n - 1. First move the n - 1 smallest disks from peg A to peg C, exposing the nth smallest disk on peg A. Move that disk from A to B. Then move the n - 1 smallest disks from C to B. Moving the n-1 smallest disks is accomplished by a recursive application of the method. As the n disks involved in the moves are smaller than any other disks, we need not concern ourselves with what is below them on pegs A, B, or C. Although the actual movement of individual disks is not obvious, and hand simulation is hard because of the stacking of recursive calls, the algorithm is conceptually simple to understand, to prove correct and, we would like to think, to invent in the first place. It is probably the ease of discovery of divide-and-conquer algorithms that makes the technique so important, although in many cases the algorithms are also more efficient than more conventional ones.†

The Problem of Multiplying Long Integers

Consider the problem of multiplying two n-bit integers X and Y. Recall that the algorithm for multiplication of n-bit (or n-digit) integers usually taught in elementary school involves computing n partial products of size n and thus is an O(n2) algorithm, if we count single bit or digit multiplications and additions as one step. One divide-and-conquer approach to integer multiplication would break each of X and Y into two integers of n/2 bits each as shown in Fig. 10.2. (For simplicity we assume n is a power of 2 here.)

Fig. 10.2. Breaking n-bit integers into [image] -bit pieces.

The product of X and Y can now be written

      XY = AC2n + (AD+BC) 2n/2 +
BD             (10.1)

If we evaluate XY in this straightforward way, we have to perform four multiplications of (n/2)-bit integers (AC, AD, BC, and BD), three additions of integers with at most 2n bits (corresponding to the three + signs in (10.1)), and two shifts (multiplication by 2n and 2n/2). As these additions and shifts take O(n) steps, we can write the following recurrence for T(n), the total number of bit operations needed to multiply n-bit integers according to (10.1).

        T(1) = 1
        T(n) = 4T(n/2) + cn                     (10.2)

Using reasoning like that in Example 9.4, we can take the constant c in (10.2) to be 1, so the driving function d(n) is just n, and then deduce that the homogeneous and particular solutions are both O(n2).

In the case that formula (10.1) is used to multiply integers, the asymptotic efficiency is thus no greater than for the elementary school method. But recall that for equations like (10.2) we get an asymptotic improvement if we decrease the number of subproblems. It may be a surprise that we can do so, but consider the following formula for multiplying X by Y.

      XY = AC2n + [(A - B)(D - C) +
AC + BD]2n/2 + BD      (10.3)

Although (10.3) looks more complicated than (10.1) it requires only three multiplications of (n/2)-bit integers, AC, BD, and (A - B)(D - C), six additions or subtractions, and two shifts. Since all but the multiplications take O(n) steps, the time T(n) to multiply n-bit integers by (10.3) is given by

       T(1) = 1
       T(n) = 3T(n/2) + cn

whose solution is T(n) = O(nlog23) = O(n1.59).

The complete algorithm, including the details implied by the fact that (10.3) requires multiplication of negative, as well as positive, (n/2)-bit integers, is given in Fig. 10.3. Note that lines (8)-(11) are performed by copying bits, and the multiplication by 2n and 2n/2 in line (16) by shifting. Also, the multiplication by s in line (16) simply introduces the proper sign into the result.

     function mult ( X, Y, n: integer ): integer;
        { X and Y are signed integers �
2n.
          n is a power of 2. The function returns XY }
    var
        s: integer; { holds the sign of XY }
        m1, m2, m3: integer; { hold the three products }
        A, B, C, D: integer; { hold left and right halves of X and Y }
    begin
(1)          s := sign(X) * sign(Y);
(2)          X := abs(X);
(3)          Y := abs(Y); { make X and Y positive }
(4)          if n = 1 then
(5)               if (X = 1) and (Y = 1) then
(6)                  return (s)
                  else
(7)		     return (0)
             else begin
(8)               A := left n/2 bits of X;
(9)               B := right n/2 bits of X;
(10)              C := left n/2 bits of Y;
(11)              D := right n/2 bits of Y;
(13)              m1 := mult(A, C, n/2);
(14)              m2 := mult(A-B, D-C, n/2);
(15)              m3 := mult(B, D, n/2);
(16)              return (s * (m1*2n + (m1 +
m2 + m3) * 2n/2 + m3))
             end
        end; { mult }

Fig. 10.3. Divide-and-conquer integer multiplication algorithm.

Observe that the divide-and-conquer algorithm of Fig. 10.3 is asymptotically faster than the method taught in elementary school, taking O(n1.59) steps against O(n2). We may thus raise the question: if this algorithm is so superior why don't we teach it in elementary school? There are two answers. First, while easy to implement on a computer, the description of the algorithm is sufficiently complex that if we attempted to teach it in elementary school students would not learn to multiply. Furthermore, we have ignored constants of proportionality. While procedure mult of Fig. 10.3 is asymptotically superior to the usual method, the constants are such that for small problems (actually up to about 500 bits) the elementary school method is superior, and we rarely ask elementary school children to multiply such numbers.

Constructing Tennis Tournaments

The technique of divide-and-conquer has widespread applicability, not only in algorithm design but in designing circuits, constructing mathematical proofs and in other walks of life. We give one example as an illustration. Consider the design of a round robin tennis tournament schedule, for n = 2k players. Each player must play every other player, and each player must play one match per day for n -1 days, the minimum number of days needed to complete the tournament.

The tournament schedule is thus an n row by n -1 column table whose entry in row i and column j is the player i must contend with on the jth day.

The divide-and-conquer approach constructs a schedule for one- half of the players. This schedule is designed by a recursive application of the algorithm by finding a schedule for one half of these players and so on. When we get down to two players, we have the base case and we simply pair them up.

Suppose there are eight players. The schedule for players 1 through 4 fills the upper left corner (4 rows by 3 columns) of the schedule being constructed. The lower left corner (4 rows by 3 columns) of the schedule must pit the high numbered players (5 through 8) against one another. This subschedule is obtained by adding 4 to each entry in the upper left.

We have now simplified the problem. All that remains is to have lower-numbered players play high-numbered players. This is easily accomplished by having players 1 through 4 play 5 through 8 respectively on day 4 and cyclically permuting 5 through 8 on subsequent days. The process is illustrated in Fig. 10.4. The reader should now be able to generalize the ideas of this figure to provide a schedule for 2k players for any k.

Balancing Subproblems

In designing algorithms one is always faced with various trade-offs. One rule that has emerged is that it is generally advantageous to balance competing costs wherever possible. For example in Chapter 5 we saw that the 2-3 tree balanced the costs of searching with those of inserting, while more straightforward methods take O(n) steps either for each lookup or for each insertion, even though the other operation can be done in a constant number of steps.

Similarly, for divide-and-conquer algorithms, we are generally better off if the subproblems are of approximately equal size. For example, insertion sort can be viewed as partitioning a problem into two subproblems, one of size 1 and one of size n - 1, with a maximum cost of n steps to merge. This gives a recurrence

           T(n) = T(1) + T(n - 1) + n
which has an O(n2) solution. Mergesort, on the other hand, partitions the problems into two subproblems each of size n/2 and has O(nlogn) performance. As a general principle, we often find that partitioning a problem into

Fig. 10.4. A round-robin tournament for eight players.

equal or nearly equal subproblems is a crucial factor in obtaining good performance.

10.2 Dynamic Programming

Often there is no way to divide a problem into a small number of subproblems whose solution can be combined to solve the original problem. In such cases we may attempt to divide the problem into as many subproblems as necessary, divide each subproblem into smaller subproblems and so on. If this is all we do, we shall likely wind up with an exponential-time algorithm.

Frequently, however, there are only a polynomial number of subproblems, and thus we must be solving some subproblem many times. If instead we keep track of the solution to each subproblem solved, and simply look up the answer when needed, we would obtain a polynomial-time algorithm.

It is sometimes simpler from an implementation point of view to create a table of the solutions to all the subproblems we might ever have to solve. We fill in the table without regard to whether or not a particular subproblem is actually needed in the overall solution. The filling-in of a table of subproblems to get a solution to a given problem has been termed dynamic programming, a name that comes from control theory.

The form of a dynamic programming algorithm may vary, but there is the common theme of a table to fill and an order in which the entries are to be filled. We shall illustrate the techniques by two examples, calculating odds on a match like the World Series, and the "triangulation problem."

World Series Odds

Suppose two teams, A and B, are playing a match to see who is the first to win n games for some particular n. The World Series is such a match, with n = 4. We may suppose that A and B are equally competent, so each has a 50% chance of winning any particular game. Let P(i, j) be the probability that if A needs i games to win, and B needs j games, that A will eventually win the match. For example, in the World Series, if the Dodgers have won two games and the Yankees one, then i = 2, j = 3, and P(2, 3), we shall discover, is 11/16.

To compute P(i, j), we can use a recurrence equation in two variables. First, if i = 0 and j > 0, then team A has won the match already, so P(0, j) = 1. Similarly, P(i, 0) = 0 for i > 0. If i and j are both greater than 0, at least one more game must be played, and the two teams each win half the time. Thus, P(i, j) must be the average of P(i - 1, j) and P(i, j - 1), the first of these being the probability A will win the match if it wins the next game and the second being the probability A wins the match even though it loses the next game. To summarize:

    P(i, j) = 1 if i = 0 and j > 0
            = 0 if i > 0 and j = 0
            = (P(i - 1, j) + P(i, j - 1))/2   if i > 0 and
j > 0     (10.4)

If we use (10.4) recursively as a function, we can show that P(i, j) takes no more than time O(2i+j). Let T(n) be the maximum time taken by a call to P(i, j), where i + j = n. Then from (10.4),

        T(1) = c
        T(n) = 2T(n - 1) + d
for some constants c and d. The reader may check by the means discussed in the previous chapter that T(n) � 2n- 1c + (2n-1-1)d, which is O(2n) or O(2i+j).

We have thus proven an exponential upper bound on the time taken by the recursive computation of P(i, j). However, to convince ourselves that the recursive formula for P(i, j) is a bad way to compute it, we need to get a big-omega lower bound. We leave it as an exercise to show that when we call P(i, j), the total number of calls to P that gets made is [image], the number of ways to choose i things out of i+j. If i = j, that number is [image], where n = i+j. Thus, T(n) is [image], and in fact, we can show it is [image] also. While [image] grows asymptotically more slowly than 2n, the difference is not great, and T(n) grows far too fast for the recursive calculation of P(i, j) to be practical.

The problem with the recursive calculation is that we wind up computing the same P(i, j) repeatedly. For example, if we want to compute P(2, 3), we compute, by (10.4), P(1, 3) and P(2, 2). P(1, 3) and P(2, 2) both require the computation of P(1, 2), so we compute that value twice.

A better way to compute P(i, j) is to fill in the table suggested by Fig. 10.5. The bottom row is all 0's and the rightmost column all 1's by the first two lines of (10.4). By the last line of (10.4), each other entry is the average of the entry below it and the entry to the right. Thus, an appropriate way to fill in the table is to proceed in diagonals beginning at the lower right corner, and proceeding up and to the left along diagonals representing entries with a constant value of i+j, as suggested in Fig. 10.6. This program is given in Fig. 10.7, assuming it works on a two-dimensional array P of suitable size.

Fig. 10.5. Table of odds.

Fig. 10.6. Pattern of computation.

The analysis of function odds is easy. The loop of lines (4)-(5) takes O(s) time, and that dominates the O(1) time for lines (2)-(3). Thus, the outer loop takes time [image], where i+j = n. Thus dynamic

    function odds ( i, j: integer ): real;
        var
                    s, k: integer;
             begin
(1)                 for s := 1 to i + j do begin
             { compute diagonal of entries whose indices sum to s }
(2)                      P[0, s] := 1.0;
(3)                      P[s, 0] := 0.0;
(4)                      for k := 1 to s-1 do
(5)                           P[k, s-k] := (P[k-1, s-k]
+ P[k, s-k-1])/2.0
            end;
(6)                 return (P[i, j])
              end; { odds }

Fig. 10.7. Odds calculation.

programming takes O(n2) time, compared with [image] for the straightforward approach. Since [image] grows wildly faster than n2, we would prefer dynamic programming to the recursive approach under essentially any circumstances.

The Triangulation Problem

As another example of dynamic programming, consider the problem of triangulating a polygon. We are given the vertices of a polygon and a distance measure between each pair of vertices. The distance may be the ordinary (Euclidean) distance in the plane, or it may be an arbitrary cost function given by a table. The problem is to select a set of chords (lines between nonadjacent vertices) such that no two chords cross each other, and the entire polygon is divided into triangles. The total length (distance between endpoints) of the chords selected must be a minimum. We call such a set of chords a minimal triangulation.

Example 10.1. Figure 10.8 shows a seven-sided polygon and the (x, y) coordinates of its vertices. The distance function is the ordinary Euclidean distance. A triangulation, which happens not to be minimal, is shown by dashed lines. Its cost is the sum of the lengths of the chords (v0, v2), (v0, v3), (v0, v5), and (v3, v5), or [image].

As well as being interesting in its own right, the triangulation problem has a number of useful applications. For example, Fuchs, Kedem, and Uselton [1977] used a generalization of the triangulation problem for the following purpose. Consider the problem of shading a two-dimensional picture of an object whose surface is defined by a collection of points in 3-space. The light source comes from a given direction, and the brightness of a point on the surface depends on the angles between the direction of light, the direction of the viewer's eye, and a perpendicular to the surface at that point. To estimate the direction of the surface at a point, we can compute a minimum triangulation

Fig. 10.8. A heptagon and a triangulation.

of the points defining the surface.

Each triangle defines a plane in a 3-space, and since a minimum triangulation was found, the triangles are expected to be very small. It is easy to find the direction of a perpendicular to a plane, so we can compute the light intensity for the points of each triangle, on the assumption that the surface can be treated as a triangular plane in a given region. If the triangles are not sufficiently small to make the light intensity look smooth, then local averaging can improve the picture.

Before proceeding with the dynamic programming solution to the triangulation problem, let us state two observations about triangulations that will help us design the algorithm. Throughout we assume we have a polygon with n vertices v0, v1, . . . , vn-1, in clockwise order.

Fact 1. In any triangulation of a polygon with more than three vertices, every pair of adjacent vertices is touched by at least one chord. To see this, suppose neither vi nor vi+1† were touched by a chord. Then the region that edge (vi, vi+1) bounds would have to include edges (vi-1, vi), (vi+1, Vi+2) and at least one additional edge. This region then would not be a triangle.

Fact 2. If (vi, vj) is a chord in a triangulation, then there must be some vk such that (vi, vk) and (vk, vj) are each either edges of the polygon or chords. Otherwise, (vi, vj) would bound a region that was not a triangle.

To begin searching for a minimum triangulation, we pick two adjacent vertices, say v0 and v1. By the two facts we know that in any triangulation, and therefore in the minimum triangulation, there must be a vertex vk such that (v1, vk) and (vk, v0) are chords or edges in the triangulation. We must therefore consider how good a triangulation we can find after selecting each possible value for k. If the polygon has n vertices, there are a total of (n-2) choices to make.

Each choice of k leads to at most two subproblems, which we define to be polygons formed by one chord and the edges in the original polygon from one end of the chord to the other. For example, Fig. 10.9 shows the two subproblems that result if we select the vertex v3.

Fig. 10.9. The two subproblems after selecting v3.

Next, we must find minimum triangulations for the polygons of Fig. 10.9(a) and (b). Our first instinct is that we must again consider all chords emanating from two adjacent vertices. For example, in solving Fig. 10.9(b), we might consider choosing chord (v3, v5), which leaves subproblem (v0, v3, v5, v6), a polygon two of whose sides, (v0, v3) and (v3, v5), are chords of the original polygon. This approach leads to an exponential-time algorithm.

However, by considering the triangle that involves the chord (v0, vk) we never have to consider polygons more than one of whose sides are chords of the original polygon. Fact 2 tells us that, in the minimal triangulation, thechord in the subproblem, such as (v0, v3) in Fig. 10.9(b), must make a triangle with one of the other vertices. For example, if we select v4, we get the triangle (v0, v3, v4) and the subproblem (v0, v4, v5, v6) which has only one chord of the original polygon. If we try v5, we get the subproblems (v3, v4, v5) and (v0, v5, v6), with chords (v3, v5) and (v0, v5) only.

In general, define the subproblem of size s beginning at vertex vi, denoted Sis, to be the minimal triangulation problem for the polygon formed by the s vertices beginning at vi and proceeding clockwise, that is, vi, vi+1, ..., vi+s-1. The chord in Sis is (vi, vi+s-1). For example, Fig. 10.9(a) is S04 and Fig. 10.9(b) is S35. To solve Sis we must consider the following three options.

  1. We may pick vertex vi+s-2 to make a triangle with the chords (vi, vi+s-1) and (vi, vi+s-2) and third side (vi+s-2, vi+s-l), and then solve the subproblem Si,s-1.
  2. We may pick vertex vi+1 to make a triangle with the chords (vi, vi+s-1) and (vi+1, vi+s-1) and third side (vi, vi+1), and then solve the subproblem Si + 1,s-1.
  3. For some k between 2 and s-3 we may pick vertex vi+k and form a triangle with sides (vi, vi+k), (vi+k, vi+s-1), and (vi, vi+s-1) and then solve subproblems Si,k+1 and Si+k,s-k.

If we remember that "solving" any subproblem of size three or less requires no action, we can summarize (1)-(3) by saying that we pick some k between 1 and s-2 and solve subproblems Si,k+1 and Si+k,s-k. Figure 10.10 illustrates this division into subproblems.

Fig. 10.10. Division of Sis into subproblems.

If we use the obvious recursive algorithm implied by the above rules to solve subproblems of size four or more, then it is possible to show that each call on a subproblem of size s gives rise to a total of 3s-4 recursive calls, if we "solve" subproblems of size three or less directly and count only calls on subproblems of size four or more. Thus the number of subproblems to be solved is exponential in s. Since our initial problem is of size n, where n is the number of vertices in the given polygon, the total number of steps performed by this recursive procedure is exponential in n.

Yet something is clearly wrong in this analysis, because we know that besides the original problem, there are only n(n-4) different subproblems that ever need to be solved. They are represented by Sis, where 0 � i < n and 4 � s < n. Evidently not all the subproblems solved by the recursive procedure are different. For example, if in Fig. 10.8 we choose chord (v0, v3), and then in the subproblem of Fig. 10.9(b) we pick v4, we have to solve subproblem S44. But we would also have to solve this problem if we first picked chord (v0, v4), or if we picked (v1, v4) and then, when solving subproblem S45, picked vertex v0 to complete a triangle with v1 and v4.

This suggests an efficient way to solve the triangulation problem. We make a table giving the cost Cis of triangulating Sis for all i and s. Since the solution to any given problem depends only on the solution to problems of smaller size, the logical order in which to fill in the table is in size order. That is, for sizes s = 4, 5, . . . ,n-1 we fill in the minimum cost for problems Sis, for all vertices i. It is convenient to include problems of size 0 � s < 4 as well, but remember that Sis has cost 0 if s < 4.

By rules (1)-(3) above for finding subproblems, the formula for computing Cis for s � 4 is:

where D(vp, vq) is the length of the chord between vertices vp and vq, if vp and vq are not adjacent points on the polygon; D(vp, vq) is 0 if vp and vq are adjacent.

Example 10.2. Figure 10.11 holds the table of costs for Si,s for 0 � i � 6 and 4 � s � 6, based on the polygon and distances of Fig. 10.8. The costs for the rows with s < 3 are all zero. We have filled in the entry C07, in column 0 and the row for s = 7. This entry, like all in that row, represents the triangulation of the entire polygon. To see that, just notice that we can, if we wish, consider the edge (v0, v6) to be a chord of a larger polygon and the polygon of Fig. 10.8 to be a subproblem of this polygon, which has a series of additional vertices extending clockwise from v6 to v0. Note that the entire row for s = 7 has the same value as C07, to within the accuracy of the computation.

Let us, as an example, show how the entry 38.09 in the column for i = 6 and row for s = 5 is filled in. According to (10.5) the value of this entry, C65, is the minimum of three sums, corresponding to k = 1, 2, or 3. These sums are:

             C62 + C04 + D
(v6, v0) + D(v0,
v3)
             C63 + C13 + D
(v6, v1) + D(v1,
v3)
             C64 + C22 + D
(v6, v2) + D(v2,
v3)

Fig. 10.11 Table of Cis's.

The distances we need are calculated from the coordinates of the vertices as:

           D(v2, v3) =
D(v6, v0) = 0
(since these are polygon edges, not chords, and are present "for free")

           D(v6, v2) = 26.08
           D(v1, v3) = 16.16
           D(v6, v1) = 22.36
           D(v0, v3) = 21.93

The three sums above are 38.09, 38.52, and 43.97, respectively. We may conclude that the minimum cost of the subproblem S65 is 38.09. Moreover, since the first sum was smallest, we know that to achieve this minimum we must utilize the subproblems S62 and S04, that is, select chord (v0, v3) and then solve S64 as best we can; chord (v1, v3) is the preferred choice for that subproblem.

There is a useful trick for filling out the table of Fig. 10.11 according to the formula (10.5). Each term of the min operation in (10.5) requires a pair of entries. The first pair, for k = 1, can be found in the table (a) at the "bottom" (the row for s = 2)† of the column of the element being computed, and (b) just below and to the right‡ of the element being computed. The second pair is (a) next to the bottom of the column, and (b) two positions down and to the right. Fig. 10.12 shows the two lines of entries we follow to get all the pairs of entries we need to consider simultaneously. The pattern -- up the column and down the diagonal -- is a common one in filling tables during dynamic programming.

Fig. 10.12. Pattern of table scan to compute one element.

Finding Solutions from the Table

While Fig. 10.11 gives us the cost of the minimum triangulation, it does not immediately give us the triangulation itself. What we need, for each entry, is the value of k that produced the minimum in (10.5). Then we can deduce that the solution consists of chords (vi, vi+k), and (vi+k, vi+s-1) (unless one of them is not a chord, because k = 1 or k = s-2), plus whatever chords are implied by the solutions to Si,k+1 and Si+k,s-k. It is useful, when we compute an element of the table, to include with it the value of k that gave the best solution.

Example 10.3. In Fig. 10.11, the entry C07, which represents the solution to the entire problem of Fig. 10.8, comes from the terms for k = 5 in (10.5). That is, the problem S07 is split into S06 and S52; the former is the problem with six vertices v0, v1, . . . ,v5, and the latter is a trivial "problem" of cost 0. Thus we introduce the chord (v0, v5) of cost 22.09 and must solve S06.

The minimum cost for C06 comes from the terms for k = 2 in (10.5), whereby the problem S06 is split into S03 and S24. The former is the triangle with vertices v0, v1, and v2, while the latter is the quadrilateral defined by v2, v3, v4, and v5. S03 need not be solved, but S24 must be, and we must include the costs of chords (v0, v2) and (v2, v5) which are 17.89 and 19.80, respectively. We find the minimum value for C24 is assumed when k = 1 in (10.5), giving us the subproblems C22 and C33, both of which have size less than or equal to three and therefore cost 0. The chord (v3, v5) is introduced, with a cost of 15.65.

Fig. 10.13. A minimal cost triangulation.

10.3 Greedy Algorithms

Consider the problem of making change. Assume coins of values 25¢ (quarter), 10¢ (dime), 5¢ (nickel) and 1¢ (penny), and suppose we want to return 63¢ in change. Almost without thinking we convert this amount to two quarters, one dime and three pennies. Not only were we able to determine quickly a list of coins with the correct value, but we produced the shortest list of coins with that value.

The algorithm the reader probably used was to select the largest coin whose value was not greater than 63¢ (a quarter), add it to the list and subtract its value from 63 getting 38¢. We then selected the largest coin whose value was not greater than 38¢ (another quarter) and added it to the list, and so on.

This method of making change is a greedy algorithm. At any individual stage a greedy algorithm selects that option which is "locally optimal" in some particular sense. Note that the greedy algorithm for making change produces an overall optimal solution only because of special properties of the coins. If the coins had values 1¢, 5¢, and 11¢ and we were to make change of 15¢, the greedy algorithm would first select an 11¢ coin and then four 1¢ coins, for a total of five coins. However, three 5¢ coins would suffice.

We have already seen several greedy algorithms, such as Dijkstra's shortest path algorithm and Kruskal's minimum cost spanning tree algorithm. Dijkstra's shortest path algorithm is "greedy" in the sense that it always chooses the closest vertex to the source among those whose shortest path is not yet known. Kruskal's algorithm is also "greedy"; it picks from the remaining edges the shortest among those that do not create a cycle.

We should emphasize that not every greedy approach succeeds in producing the best result overall. Just as in life, a greedy strategy may produce a good result for a while, yet the overall result may be poor. As an example, we might consider what happens when we allow negative-weight edges in Dijkstra's and Kruskal's algorithms. It turns out that Kruskal's spanning tree algorithm is not affected; it still produces the minimum cost tree. But Dijkstra's algorithm fails to produce shortest paths in some cases.

Example 10.4. We see in Fig. 10.14 a graph with a negative cost edge between b and c. If we apply Dijkstra's algorithm with source s, we correctly discover first that the minimum path to a has length 1. Now, considering only edges from s or a to b or c, we expect that b has the shortest path from s, namely s � a � b, of length 3. We then discover that c has a shortest path from s of length 1.

Fig. 10.14. Graph with negative weight edge.

However, our "greedy" selection of b before c was wrong from a global point of view. It turns out that the path s � a � c � b has length only 2, so our minimum distance of 3 for b was wrong.†

Greedy Algorithms as Heuristics

For some problems no known greedy algorithm produces an optimal solution, yet there are greedy algorithms that can be relied upon to produce "good" solutions with high probability. Frequently, a suboptimal solution with a cost a few percent above optimal is quite satisfactory. In these cases, a greedy algorithm often provides the fastest way to get a "good" solution. In fact, if the problem is such that the only way to get an optimal solution is to use an exhaustive search technique, then a greedy algorithm or other heuristic for getting a good, but not necessarily optimal, solution may be our only real choice.

Example 10.5. Let us introduce a famous problem where the only known algorithms that produce optimal solutions are of the "try-all-possibilities" variety and can have running times that are exponential in the size of the input. The problem, called the traveling salesman problem, or TSP, is to find, in an undirected graph with weights on the edges, a tour (a simple cycle that includes all the vertices) the sum of whose edge-weights is a minimum. A tour is often called a Hamilton (or Hamiltonian) cycle.

Figure 10.15(a) shows one instance of the traveling salesman problem, a graph with six vertices (often called "cities"). The coordinates of each vertex are given, and we take the weight of each edge to be its length. Note that, as is conventional with the TSP, we assume all edges exist, that is, the graph is complete. In more general instances, where the weight of edges is not based on Euclidean distance, we might find a weight of infinity on an edge that really was not present.

Figure 10.15(b)-(e) shows four tours of the six "cities" of Fig. 10.15(a). The reader might ponder which of these four, if any, is optimal. The lengths of these four tours are 50.00, 49.73, 48.39, and 49.78, respectively; (d) is the shortest of all possible tours.

Fig. 10.15. An instance of the traveling salesman problem.

The TSP has a number of practical applications. As its name implies, it can be used to route a person who must visit a number of points and return to his starting point. For example, the TSP has been used to route collectors of coins from pay phones. The vertices are the phones and the "home base." The cost of each edge is the travel time between the two points in question.

Another "application" of the TSP is in solving the knight's tour problem: find a sequence of moves whereby a knight can visit each square of the chessboard exactly once and return to its starting point. Let the vertices be the chessboard squares and let the edges between two squares that are a knight's move apart have weight 0; all other edges have weight infinity. An optimal tour has weight 0 and must be a knight's tour. Surprisingly, good heuristics for the TSP have no trouble finding knight's tours, although finding one "by hand" is a challenge.

The greedy algorithm for the TSP we have in mind is a variant of Kruskal's algorithm. Like that algorithm, we shall consider edges shortest first. In Kruskal's algorithm we accept an edge in its turn if it does not form a cycle with the edges already accepted, and we reject the edge otherwise. For the TSP, the acceptance criterion is that an edge under consideration, together with already selected edges,

  1. does not cause a vertex to have degree three or more, and
  2. does not form a cycle, unless the number of selected edges equals the number of vertices in the problem.

Collections of edges selected under these criteria will form a collection of unconnected paths, until the last step, when the single remaining path is closed to form a tour.

In Fig. 10.15(a), we would first pick edge (d, e), since it is the shortest, having length 3. Then we consider edges (b, c), (a, b), and (e, f), all of which have length 5. It doesn't matter in which order we consider them; all meet the conditions for selection, and we must select them if we are to follow the greedy approach. Next shortest edge is (a, c), with length 7.08. However, this edge would form a cycle with (a, b) and (b, c), so we reject it. Edge (d, f) is next rejected on similar grounds. Edge (b, e) is next to be considered, but it must be rejected because it would raise the degrees of b and e to three, and could then never form a tour with what we had selected. Similarly we reject (b, d). Next considered is (c, d), which is accepted. We now have one path, a � b � c � d � e � f, and eventually accept (a, f) to complete the tour. The resulting tour is Fig. 10.15(b), which is fourth best of all the tours, but less than 4% more costly than the optimal.

10.4 Backtracking

Sometimes we are faced with the task of finding an optimal solution to a problem, yet there appears to be no applicable theory to help us find the optimum, except by resorting to exhaustive search. We shall devote this section to a systematic, exhaustive searching technique called backtracking and a technique called alpha-beta pruning, which frequently reduces the search substantially.

Consider a game such as chess, checkers, or tic-tac-toe, where there are two players. The players alternate moves, and the state of the game can be represented by a board position. Let us assume that there are a finite number of board positions and that the game has some sort of stopping rule to ensure termination. With each such game, we can associate a tree called the game tree. Each node of the tree represents a board position. With the root we associate the starting position. If board position x is associated with node n, then the children of n correspond to the set of allowable moves from board position x, and with each child is associated the resulting board position. For example, Fig. 10.16 shows part of the tree for tic-tac-toe.

Fig. 10.16. Part of the tic-tac-toe game tree.

The leaves of the tree correspond to board positions where there is no move, either because one of the players has won or because all squares are filled and a draw resulted. We associate a value with each node of the tree.

First we assign values to the leaves. Say the game is tic-tac-toe. Then a leaf is assigned -l, 0 or 1 depending on whether the board position corresponds to a loss, draw or win for player 1 (playing X).

The values are propagated up the tree according to the following rule. If a node corresponds to a board position where it is player 1's move, then the value is the maximum of the values of the children of that node. That is, we assume player 1 will make the move most favorable to himself i.e., that which produces the highest-valued outcome. If the node corresponds to player 2's move, then the value is the minimum of the values of the children. That is, we assume player 2 will make his most favorable move, producing a loss for player 1 if possible, and a draw as next preference.

Example 10.6. The values of the boards have been marked in Fig. 10.16. The leaves that are wins for O get value -1, while those that are draws get 0, and wins for X get + 1. Then we proceed up the tree. On level 8, where only one empty square remains, and it is X's move, the values for the unresolved boards is the "maximum" of the one child at level 9.

On level 7, where it is O's move and there are two choices, we take as a value for an interior node the minimum of the values of its children. The leftmost board shown on level 7 is a leaf and has value 1, because it is a win for X. The second board on level 7 has value min(0, -1) = -1, while the third board has value min(0, l) = 0. The one board shown at level 6, it being X's move on that level, has value max(l, -1, 0) = 1, meaning that there is some choice X can make that will win; in this case the win is immediate.

Note that if the root has value 1, then player 1 has a winning strategy. Whenever it is his turn he is guaranteed that he can select a move that leads to a board position of value 1. Whenever it is player 2's move he has no real choice but to select a moving leading to a board position of value 1, a loss for him. The fact that a game is assumed to terminate guarantees an eventual win for the first player. If the root has value 0, as it does in tic-tac-toe, then neither player has a winning strategy but can only guarantee himself a draw by playing as well as possible. If the root has value -1, then player 2 has a winning strategy.

Payoff Functions

The idea of a game tree, where nodes have values -1, 0, and l, can be generalized to trees where leaves are given any number (called the payoff) as a value, and the same rules for evaluating interior nodes applies: take the maximum of the children on those levels where player 1 is to move, and the minimum of the children on levels where player 2 moves.

As an example where general payoffs are useful, consider a complex game, like chess, where the game tree, though finite, is so huge that evaluating it from the bottom up is not feasible. Chess programs work, in essence, by building for each board position from which it must move, the game tree with that board as root, extending downward for several levels; the exact number of levels depends on the speed with which the computer can work. As most of the leaves of the tree will be ambiguous, neither wins, losses, nor draws, each program uses a function of board positions that attempts to estimate the probability of the computer winning in that position. For example, the difference in material figures heavily into such an estimation, as do such factors as the defensive strength around the kings. By using this payoff function, the computer can estimate the probability of a win after making each of its possible next moves, on the assumption of subsequent best play by each side, and chose the move with the highest payoff.†

Implementing Backtrack Search

Suppose we are given the rules for a game,† that is, its legal moves and rules for termination. We wish to construct its game tree and evaluate the root. We could construct the tree in the obvious way, and then visit the nodes in postorder. The postorder traversal assures that we visit an interior node n after all its children, whereupon we can evaluate n by taking the min or max of the values of its children, as appropriate.

The space to store the tree can be prohibitively large, but by being careful we need never store more than one path, from the root to some node, at any one time. In Fig. 10.17 we see the sketch of a recursive program that represents the path in the tree by the sequence of active procedure calls at any time. That program assumes the following:

  1. Payoffs are real numbers in a limited range, for example - 1 to + 1.
  2. The constant � is larger than any positive payoff and its negation is smaller than any negative payoff.
  3. The type modetype is declared
    4.
              type
           modetype = (MIN, MAX)
  4. There is a type boardtype declared in some manner suitable for the representation of board positions.
  5. There is a function payoff that computes the payoff for any board that is a leaf (i.e., won, lost, or drawn position).

      function search ( B: boardtype; mode: modetype ): real;
          { evaluates the payoff for board B, assuming it is
            player 1's move if mode = MAX and player 2's move
             if mode = MIN. Returns the payoff }
          var
               C: boardtype; { a child of board B }
           value: real; { temporary minimum or maximum value }
           begin
(1)            if B is a leaf then
(2)                  return (payoff(B))
             else begin
               { initialize minimum or maximum value of children }
(3)                 if mode = MAX then
(4)                     value := -�
                  else
(5)                     value := �;
(6)                 for each child C of board B do
(7)                     if mode = MAX then
(8)                         value := max(value, search(C, MIN))
                   else
(9)                         value := min(value, search(C, MAX));
(10)                 return (value)
          end
      end; { search }

Fig. 10.17. Recursive backtrack search program.

Another implementation we might consider is to use a nonrecursive program that keeps a stack of boards corresponding to the sequence of active calls to search. The techniques discussed in Section 2.6 can be used to construct such a program.

Alpha-Beta Pruning

There is a simple observation that allows us to eliminate from consideration much of a typical game tree. In terms of Fig. 10.17, the for-loop of line (6) can skip over certain children, often many of the children. Suppose we have a node n, as in Fig. 10.18, and we have already determined that c1, the first of n's children, has a value of 20. As n is a max node, we know its value is at least 20. Now suppose that continuing with our search we find that d, a child of c2 has value 15. As c2, another child of n, is a min node, we know the value of c2 cannot exceed 15. Thus, whatever value c2 has, it cannot affect the value of n or any parent of n.

It is thus possible in the situation of Fig. 10.18, to skip consideration of the children of c2 that we have not already examined. The general rules for

Fig. 10.18. Pruning the children of a node.

skipping or "pruning" nodes involves the notion of final and tentative values for nodes. The final value is what we have simply been calling the "value." A tentative value is an upper bound on the value of a min node, or a lower bound on the value of a max node. The rules for computing final and tentative values are the following.

  1. If all the children of a node n have been considered or pruned, make the tentative value of n final.
  2. If a max node n has tentative value v1 and a child with final value v2, then set the tentative value of n to max(v1, v2). If n is a min node, set its tentative value to min(v1, v2).
  3. If p is a min node with parent q (a max node), and p and q have tentative values v1 and v2, respectively, with v1 � v2, then we may prune all the unconsidered children of p. We may also prune the unconsidered children of p if p is a max node (and therefore q is a min node) and v2 � v1.

Example 10.7. Consider the tree in Fig. 10.19. Assuming values for the leaves as shown, we wish to calculate the value for the root. We begin a postorder traversal. After reaching node D, by rule (2) we assign a tentative value of 2, which is the final value of D, to node C. We then search E and return to C and then to B. By rule (1), the final value of C is fixed at 2 and the value of B is tentatively set to 2. The search continues down to G and then back to F. The value F is tentatively set to 6. By rule (3), with p and q equal to F and B, respectively, we may prune H. That is, there is no need to search node H, since the tentative value of F can never decrease and it is already greater than the value of B, which can never increase.

Fig. 10.19. A game tree.

Continuing our example, A is assigned a tentative value of 2 and the search proceeds to K. J is assigned a tentative value of 8. L does not determine the value of max node J. I is assigned a tentative value of 8. The search goes down to N, and M is assigned a tentative value of 5. Node O must be searched, since 5, the tentative value of M, is less than the tentative value of I. The tentative values of I and A are revised, and the search goes down to R. Eventually R and S are searched, and P is assigned a tentative value of 4. We need not search T or below, since that can only lower P's value and it is already too low to affect the value of A.

Branch-and-Bound Search

Games are not the only sorts of "problems" that can be solved exhaustively by searching a complete tree of possibilities. A wide variety of problems where we are asked to find a minimum or maximum configuration of some sort are amenable to solution by backtracking search over a tree of all possibilities. The nodes of the tree can be thought of as sets of configurations, and the children of a node n each represent a subset of the configurations that n represents. Finally, the leaves each represent single configurations, or solutions to the problem, and we may evaluate each such configuration to see if it is the best solution found so far.

If we are reasonably clever in how we design the search, the children of a node will each represent far fewer configurations than the node itself, so we need not go to too great a depth before reaching leaves. Lest this notion of searching appear too vague, let us take a concrete example.

Example 10.8. Recall from the previous section our discussion of the traveling salesman problem. There we gave a "greedy algorithm" for finding a good but not necessarily optimum tour. Now let us consider how we might find the optimum tour by systematically considering all tours. One way is to consider all permutations of the nodes, and evaluate the tour that visits the nodes in that order, remembering the best found so far. The time for such an approach is O(n!) on an n node graph, since we must consider (n--1)! different permutations,† and each permutation takes O(n) time to evaluate.

We shall consider a somewhat different approach that is no better than the above in the worst case, but on the average, when coupled with a technique called "branch-and-bound" that we shall discuss shortly, produces the answer far more rapidly than the "try all permutations" method. Start constructing a tree, with a root that represents all tours. Tours are what we called "configurations" in the prefatory material. Each node has two children, and the tours that a node represents are divided by these children into two groups -- those that have a particular edge and those that do not. For example, Fig. 10.20 shows portions of the tree for the TSP instance from Fig. 10.15(a).

In Fig. 10.20 we have chosen to consider the edges in lexicographic order (a, b), (a, c), . . . ,(a, f), (b, c), . . . ,(b, f), (c, d), and so on. We could, of course pick any other order. Observe that not every node in the tree has two children. We can eliminate some children because the edges selected do not form part of a tour. Thus, there is no node for "tours containing (a, b), (a, c), and (a, d)," because a would have degree 3 and the result would not be a tour. Similarly, as we go down the tree we shall eliminate some nodes because some city would have degree less than 2. For example, we shall find no node for tours without (a, b), (a, c), (a, d), or (a, e).

Bounding Heuristics Needed for Branch-and-Bound

Using ideas similar to those in alpha-beta pruning, we can eliminate far more nodes of the search tree than would be suggested by Example 10.8. Suppose, to be specific, that our problem is to minimize some function, e.g., the cost of a tour in the TSP. Suppose also that we have a method for getting a lower bound on the cost of any solution among those in the set of solutions represented by some node n. If the best solution found so far costs less than the lower bound for node n, we need not explore any of the nodes below n.

Example 10.9. We shall discuss one way to get a lower bound on certain sets of solutions for the TSP, those sets represented by nodes in a tree of solutions as suggested in Fig. 10.20. First of all, suppose we wish a lower bound on all solutions to a given instance of the TSP. Observe that the cost of any tour can be expressed as one half the sum over all nodes n of the cost of the two tour edges adjacent to n. This remark leads to the following rule. The sum

Fig. 10.20. Beginning of a solution tree for a TSP instance.

of the two tour edges adjacent to node n is no less than the sum of the two edges of least cost adjacent to n. Thus, no tour can cost less than one half the sum over all nodes n of the two lowest cost edges incident upon n.

For example, consider the TSP instance in Fig. 10.21. Unlike the instance in Fig. 10.15, the distance measure for edges is not Euclidean; that is, it bears no relation to the distance in the plane between the cities it connects. Such a cost measure might be traveling time or fare, for example. In this instance, the least cost edges adjacent to node a are (a, d), and (a, b), with a total cost of 5. For node b, we have (a, b) and (b, e), with a total cost of 6. Similarly, the two lowest cost edges adjacent to c, d, and e, total 8, 7, and 9, respectively. Our lower bound on the cost of a tour is thus (5+6+8+7+9)/2 = 17.5.

Fig. 10.21. An instance of TSP.

Now suppose we want a lower bound on the cost of a subset of tours defined by some node in the search tree. If the search tree is constructed as in Example 10.8, each node represents tours defined by a set of edges that must be in the tour and a set of edges that may not be in the tour. These constraints alter our choices for the two lowest-cost edges at each node. Obviously an edge constrained to be in any tour must be included among the two edges selected, no matter whether they are or are not lowest or second lowest in cost.† Similarly, an edge constrained to be out cannot be selected, even if its cost is lowest.

Thus, if we are constrained to include edge (a, e), and exclude (b, c), the two edges for node a are (a, d) and (a e), with a total cost of 9. For b we select (a b) and (b, e), as before, with a total cost of 6. For c, we cannot select (b, c), and so select (a, c) and (c, d), with a total cost of 9. For d we select (a, d) and (c, d), as before, while for e we must select (a, e), and choose to select (b, e). The lower bound for these constraints is thus (9+6+9+7+10)/2 = 20.5.

Now let us construct the search tree along the lines suggested in Example 10.8. We consider the edges in lexicographic order, as in that example. Each time we branch, by considering the two children of a node, we try to infer additional decisions regarding which edges must be included or excluded from tours represented by those nodes. The rules we use for these inference are:

  1. If excluding an edge (x, y) would make it impossible for x or y to have as many as two adjacent edges in the tour, then (x, y) must be included.

  2. If including (x, y) would cause x or y to have more than two edges adjacent in the tour, or would complete a non-tour cycle with edges already included, then (x, y) must be excluded.

When we branch, after making what inferences we can, we compute lower bounds for both children. If the lower bound for a child is as high or higher than the lowest cost tour found so far, we can "prune" that child and need not construct or consider its descendants. Interestingly, there are situations where the lower bound for a node n is lower than the best solution so far, yet both children of n can be pruned because their lower bounds exceed the cost of the best solution so far.

If neither child can be pruned, we shall, as a heuristic, consider the child with the smaller lower bound first, in the hope of more rapidly reaching a solution that is cheaper than the one so far found best.† After considering one child, we must consider again whether its sibling can be pruned, since a new best solution may have been found. For the instance of Fig. 10.21, we get the search tree of Fig. 10.22. To interpret nodes of that tree, it helps to understand that the capital letters are names of the search tree nodes. The numbers are the lower bounds, and we list the constraints applying to that node but none of its ancestors by writing xy if edge (x, y) must be included and [image] if (x, y) must be excluded. Also note that the constraints introduced at a node apply to all its descendants. Thus to get all the constraints applying at a node we must follow the path from that node to the root.

Lastly, let us remark that as for backtrack search in general, we construct the tree one node at a time, retaining only one path, as in the recursive algorithm of Fig. 10.17, or its nonrecursive counterpart. The nonrecursive version is probably to be preferred, so that we can maintain the list of constraints conveniently on a stack.

Example 10.10. Figure 10.22 shows the search tree for the TSP instance of Fig. 10.21. To see how it is constructed, we begin at the root A of Fig. 10.22. The first edge in lexicographic order is (a, b), so we consider the two children B and C, corresponding to the constraints ab and [image], respectively. There is, as yet, no "best solution so far," so we shall consider both B and C eventually.‡ Forcing (a b) to be included does not raise the lower bound, but excluding it raises the bound to 18.5, since the two cheapest legal edges for nodes a and b now total 6 and 7, respectively, compared with 5 and 6 with no constraints. Following our heuristic, we shall consider the descendants of node B first.

The next edge in lexicographic order is (a, c). We thus introduce children D and E corresponding to tours where (a, c) is included and excluded, respectively. In node D, we can infer that neither (a, d) nor (a, e) can be in a tour, else a would have too many edges incident. Following our heuristic we consider E before D, and branch on edge (a, d). The children F and G are introduced with lower bounds 18 and 23, respectively. For each of these children we know about three of the edges incident upon a, and so can infer something about the remaining edge (a, e).

Consider the children of F first. The first remaining edge in lexicographic order is (b, c). If we include (b, c), then, as we have included (a, b), we cannot include (b, d) or (b, e). As we have eliminated (a, e) and (b, e), we must have (c, e) and (d, e). We cannot have (c, d) or c and d would have three incident edges. We are left with one tour (a, b, c, e, d, a), whose cost is 23. Similarly, node I, where (b, c) is excluded, can be proved to represent only the tour (a b, e, c, d, a), of cost 21. That tour has the lowest cost found so far.

We now backtrack to E and consider its second child, G. But G has a lower bound of 23, which exceeds the best cost so far, 21. Thus we prune G. We now backtrack to B and consider its other child, D. The lower bound on D is 20.5, but since costs are integers, we know no tour represented by D can have cost less than 21. Since we already have a tour that cheap, we need not explore the descendants of D, and therefore prune D. Now we backtrack to A and consider its second child, C.

At the level of node C, we have only considered edge (a, b). Nodes J and K are introduced as children of C. J corresponds to those tours that have (a, c) but not (a, b), and its lower bound in 18.5. K corresponds to tours having neither (a, b) nor (a, c), and we may infer that those tours have (a, d) and (a, e). The lower bound for k is 21, and we may immediately prune k , since we already know a tour that is low in cost.

We next consider the children of J, which are L and M, and we prune M because its lower bound exceeds the best tour cost so far. The children of L are N and P, corresponding to tours that have (b, c), and that exclude (b, c). By considering the degree of nodes b and c, and remembering that the selected edges cannot form a cycle of fewer than all five cities, we can infer that nodes N and P each represent single tours. One of these, (a, c, b, e, d, a), has the lowest cost of any tour, 19. We have explored or pruned the entire tree and therefore end.

Fig. 10.22. Search tree for TSP solution.

10.5 Local Search Algorithms

Sometimes the following strategy will produce an optimal solution to a problem.

  1. Start with a random solution.

  2. Apply to the current solution a transformation from some given set of transformations to improve the solution. The improvement becomes the new "current" solution.

  3. Repeat until no transformation in the set improves the current solution. The resulting solution may or may not be optimal. In principle, if the "given set of transformations" includes all the transformations that take one solution and replace it by any other, then we shall never stop until we reach an optimal solution. But then the time to apply (2) above is the same as the time needed to examine all solutions, and the whole approach is rather pointless.

    The method makes sense when we can restrict our set of transformations to a small set, so we can consider all transformations in a short time; perhaps O(n2) or O(n3) transformations should be allowed when the problem is of "size" n. If the transformation set is small, it is natural to view the solutions that can be transformed to one another in one step as "close." The transformations are called "local transformations," and the method is called local search.

Example 10.11. One problem we can solve exactly by local search is the minimal spanning tree problem. The local transformations are those in which we take some edge not in the current spanning tree, add it to the tree, which must produce a unique cycle, and then eliminate exactly one edge of the cycle (presumably that of highest cost) to form a new tree.

For example, consider the graph of Fig. 10.21. We might start with the tree shown in Fig. 10.23(a). One transformation we could perform is to add edge (d, e) and remove another edge in the cycle formed, which is (e, a, c, d, e). If we remove edge (a, e), we decrease the cost of the tree from 20 to 19. That transformation can be made, leaving the tree of Fig. 10.23(b), to which we again try to apply an improving transformation. One such is to insert edge (a, d) and delete edge (c, d) from the cycle formed. The result is shown in Fig. 10.23(c). Then we might introduce (a, b) and delete (b, c) as in Fig. 10.23(d), and subsequently introduce (b, e) in favor of (d, e). The resulting tree of Fig. 10.23(e) is minimal. We can check that every edge not in that tree has the highest cost of any edge in the cycle it forms. Thus no transformation is applicable to Fig. 10.23(e).

The time taken by the algorithm of Example 10.11 on a graph of n nodes and e edges depends on the number of times we need to improve the solution. Just testing that no transformation is applicable could take O(ne) time, since e edges must be tried, and each could form a cycle of length nearly n Thus the algorithm is not as good as Prim's or Kruskal's algorithms, but serves as an example where an optimal solution can be obtained by local search.

Local Search Approximation Algorithms

Local search algorithms have had their greatest effectiveness as heuristics for the solution to problems whose exact solutions require exponential time. A common method of search is to start with a number of random solutions, and apply the local transformations to each, until reaching a locally optimal solution, one that no transformation can improve. We shall frequently reach different locally optimal solutions, from most or all of the random starting solutions, as suggested in Fig. 10.24. If we are lucky, one of them will be

Fig. 10.23. Local search for a minimal spanning tree.

globally optimal, that is, as good as any solution.

In practice, we may not find a globally optimal solution as suggested in Fig. 10.24, since the number of locally optimal solutions may be enormous. However, we may at least choose that locally optimal solution that has the least cost among all those found. As the number of kinds of local transformations that have been used to solve various problems is great, we shall close the section with two examples -- the TSP, and a simple problem of "package placement."

The Traveling Salesman Problem

The TSP is one for which local search techniques have been remarkably successful. The simplest transformation that has been used is called "2-opting." It consists of taking any two edges, such as (A, B) and (C, D) in Fig. 10.25, removing them, and reconnecting their endpoints to form a new tour. In Fig. 10.25, the new tour runs from B, clockwise to C, then along the edge (C, A), counterclockwise from A to D, and finally along the edge (D, B). If the sum of the lengths of (A, C) and (B, D) is less than the sum of the lengths of (A B) and (C, D), then we have an improved tour.† Note that we cannot

Fig. 10.24. Local search in the space of solutions.

connect A to D and B to C, as the result would not be a tour, but two disjoint cycles.

To find a locally optimal tour, we start with a random tour, and consider all pairs of nonadjacent edges, such as (A, B) and (C, D) in Fig. 10.25. If the tour can be improved by replacing these edges with (A, C) and (B, D), do so, and continue considering pairs of edges that have not been considered before. Note that the introduced edges (A, C) and (B, D) must each be paired with all the other edges of the tour, as additional improvements could result.

Example 10.12. Reconsider Fig. 10.21, and suppose we start with the tour of Fig. 10.26(a). We might replace (a, e) and (c, d), with a total cost of 12, by (a, d) and (c, e), with a total cost of 10, as shown in Fig. 10.26(b). Then we might replace (a, b) and (c, e) by (a, c) and (b, e), giving the optimal tour shown in Fig. 10.26(c). One can check that no pair of edges can be removed from Fig. 10.26(c) and be profitably replaced by crossing edges with the same endpoints. As one case in point, (b, c) and (d, e) together have the relatively high cost of 10. But (c, e) and (b, d) are worse, costing 14 together.

Fig. 10.25. 2-opting.

Fig. 10.26. Optimizing a TSP instance by 2-opting.

We can generalize 2-opting to k-opting for any constant k , where we remove up to k edges and reconnect the remaining pieces in any order so that result is a tour. Note that we do not require the removed edges to be nonadjacent in general, although for the 2-opting case there was no point in considering the removal of two adjacent edges. Also note that for k >2, there is more than one way to connect the pieces. For example, Fig. 10.27 shows the general process of 3-opting using any of the following eight sets of edges.

Fig. 10.27. Pieces of a tour after removing three edges.

It is easy to check that, for fixed k , the number of different k-opting transformations we need to consider if there are n vertices is O(n k). For example, the exact number is n(n-3)/2 for k = 2. The time taken to obtain a locally optimal tour may be considerably higher than this, however, since we could make many local transformations before reaching a locally optimum tour, and each improving transformation introduces new edges that may participate in later transformations that improve the tour still further. Lin and Kernighan [1973] have found that variable-depth-opting is in practice a very powerful method and has a good chance of getting the optimum tour on 40-100 city problems.

Package Placement

The one-dimensional package placement problem can be stated as follows. We have an undirected graph, whose vertices we call "packages." The edges are labeled by "weights," and the weight w(a, b) of edge (a, b) is the number of "wires" between packages a and b. The problem is to order the vertices p1, p2,..., pn, such that the sum of [i-j] w(pi, pj) over all pairs i and j is minimized. That is, we want to minimize the sum of the lengths of the wires needed to connect all the packages with the required number of wires.

The package placement problem has had a number of applications. For example, the "packages" could be logic cards on a rack, and the weight of an interconnection between cards is the number of wires connecting them. A similar problem comes up when we try to design integrated circuits from arrays of standard modules and interconnections between them. A generalization of the one-dimensional package placement problem allows placement of "packages," which have height and width, in a two-dimensional region, while minimizing the sum of the lengths of the wires between packages. This problem also has application to the design of integrated circuits, among other areas.

There are a number of local transformations we could use to find local optima for instances of the one-dimensional package placement problem. Here are several.

  1. Interchange adjacent packages pi and pi+1 if the resulting order is less costly. Let L(j) be the sum of the weights of the edges extending to the left of pj, i.e., [image]. Similarly, let R(j) be [image]. Improvement results if L(i) - R(i) + R(i+ 1) - L(i+ 1) + 2w(pi, pi+1) is negative. The reader should verify this formula by computing the costs before and after the interchange and taking the difference.

  2. Take a package pi and insert it between pj and pj+1 for some i and j.
  3. Interchange any two packages pi and pj.

Example 10.13. Suppose we take the graph of Fig. 10.21 to represent a package placement instance. We shall restrict ourselves to the simple transformation set (1). An initial placement, a, b, c, d, e, is shown in Fig. 10.28(a); it has a cost of 97. Note that the cost function weights edges by their distance, so (a, e) contributes 4x7 = 28 to the cost of 97. Let us consider interchanging d with e. We have L(d) = 13, R(d) = 6, L(e) = 24, and R(e) = 0. Thus L(d)-R(d)+R(e)- L(e)+2w(d, e) = -5, and we can interchange d and e to improve the placement to (a, b, c, e, d), with a cost of 92 as shown in Fig. 10.28(b).

In Fig. 10.28(b), we can interchange c with e profitably, producing Fig. 10.28(c), whose placement (a, b, e, c, d) has a cost of 91. Fig. 10.28(c) is locally optimal for set of transformations (1). It is not globally optimal; (a, c, e, d, b) has a cost of 84.

As with the TSP, we cannot estimate closely the time it takes to reach a local optimum. We can only observe that for set of transformations (1), there are only n-1 transformations to consider. Further, if we compute L(i) and R(i) once, we only have to change them when pi is interchanged with pi-1 or pi+1. Moreover, the recalculation is easy. If pi and pi+1 are interchanged, for example, then the new L(i) and R(i) are, respectively, L(i+1) - w(pi, pi+1) and R(i+1) + w(pi, pi+1). Thus O(n) time suffices to test for an improving transformation and to recalculate the L(i)'s and R(i)'s. We also need only O(n) time to initialize the L(i)'s and R(i)'s, if we use the recurrence

Fig. 10.28. Local optimizations.

L(1) = 0

L(i) = L(i-1) + w(pi-1, pi)

and a similar recurrence for R.

In comparison, the sets of transformations (2) and (3) each have O(n2) members. It will therefore take O(n2) time just to confirm that we have a locally optimal solution. However, as for set (1), we cannot closely bound the total time taken when a sequence of improvements are made, since each improvement can create additional opportunities for improvement.

Exercises

10.1 | How many moves do the algorithms for moving n disks in the Towers of Hanoi problem take?
*10.2 | Prove that the recursive (divide and conquer) algorithm for the Tower of Hanoi and the simple nonrecursive algorithm described at the beginning of Section 10.1 perform the same steps.
10.3 | Show the actions of the divide-and-conquer integer multiplication algorithm of Fig. 10.3 when multiplying 1011 by 1101.
*10.4 | Generalize the tennis tournament construction of Section 10.1 to tournaments where the number of players is not a power of two. Hint. If the number of players n is odd, then one player must receive a bye (not play) on any given day, and it will take n days, rather than n-1 to complete the tournament. However, if there are two groups with an odd number of players, then the players receiving the bye from each group may actually play each other.
10.5 We can recursively define the number of combinations of m things out of n, denoted [image], for n �1 and 0 � m � n, by

  1. Give a recursive function to compute

  2. What is its worst-case running time as a function of n?

  3. Give a dynamic programming algorithm to compute [image].Hint. The algorithm constructs a table generally known as Pascal's triangle.

  4. What is the running time of your answer to (c) as a function of n.
10.6 One way to compute the number of combinations of m things out of n is to calculate (n)(n-1)(n-2) � � � (n-m+l)/(1)(2) � � � (m).

  1. What is the worst case running time of this algorithm as a function of n?

  2. Is it possible to compute the "World Series Odds" function P(i, j) from Section 10.2 in a similar manner? How fast can you perform this computation?
10.7
  1. Rewrite the odds calculation of Fig. 10.7 to take into account the fact that the first team has a probability p of winning any given game.

  2. If the Dodgers have won one game and the Yankees two, but the Dodgers have a .6 probability of winning any given game, who is more likely to win the World Series?
10.8 | The odds calculation of Fig. 10.7 requires O(n2) space. Rewrite the algorithm to use only O (n) space.
*10.9 | Prove that Equation (10.4) results in exactly [image] calls to P.
10.10 | Find a minimal triangulation for a regular octagon, assuming distances are Euclidean.
10.11 | The paragraphing problem, in a very simple form, can be stated as follows: We are given a sequence of words w1, w2, . . . ,wk of lengths l1, l2, . . . ,lk, which we wish to break into lines of length L. Words are separated by blanks whose ideal width is b, but blanks can stretch or shrink if necessary (but without overlapping words), so that a line wi wi+1 ... wj has length exactly L. However, the penalty for stretching or shrinking is the magnitude of the total amount by which blanks are stretched or shrunk. That is, the cost of setting line wi wi+1 ... wj for j>i is (j-i) �b'-b�, where b', the actual width of the blanks, is (L - li - li + 1- ...- lj)/(j-i). However, if j = k (we have the last line), the cost is zero unless b' < b, since we do not have to stretch the last line. Give a dynamic programming algorithm to find a least-cost separation of w1, w2, . . . ,wk into lines of length L. Hint. For i = k , k -1, . . ., 1, compute the least cost of setting wi , wi + 1, . . . ,wk.
10.12 | Suppose we are given n elements x1, x2,. . . ,xn related by a linear order x1 < x2 < · · · < xn, and that we wish to arrange these elements m in a binary search tree. Suppose that pi is the probability that a request to find an element will be for xi. Then for any given binary search tree, the average cost of a lookup is [image], where di is the depth of the node holding xi. Given the pi's, and assuming the xi's never change, we can find a binary search tree that minimizes the lookup cost. Find a dynamic programming algorithm to do so. What is the running time of your algorithm? Hint. Compute for all i and j the optimal lookup cost among all trees containing only xi, xi +1, . . . , xi +j-1, that is, the j elements beginning with xi.
**10.13 | For what values of coins does the greedy change-making algorithm of Section 10.3 produce an optimal solution?
10.14
  1. Write the recursive triangulation algorithm discussed in Section 10.2.
  2. Show that the recursive algorithm results in exactly 3s-4 calls on nontrivial problems when started on a problem of size s�4.
10.15 Describe a greedy algorithm for
  1. The one-dimensional package placement problem.

  2. The paragraphing problem (Exercise 10.11).

Give an example where your algorithm does not produce an optimal answer, or show that no such example exists.
10.16 | Give a nonrecursive version of the tree search algorithm of Fig. 10.17.
10.17 Consider a game tree in which there are six marbles, and players 1 and 2 take turns picking from one to three marbles. The player who takes the last marble loses the game.
  1. Draw the complete game tree for this game.

  2. If the game tree were searched using the alpha-beta pruning technique, and nodes representing configurations with the smallest number of marbles are searched first, which nodes are pruned?

  3. Who wins the game if both play their best?
*10.18 | Develop a branch and bound algorithm for the TSP based on the idea that we shall begin a tour at vertex 1, and at each level, branch based on what node comes next in the tour (rather than on whether a particular edge is chosen as in Fig. 10.22). What is an appropriate lower bound estimator for configurations, which are lists of vertices 1, v1, v2, . . . that begin a tour? How does your algorithm behave on Fig. 10.21, assuming a is vertex 1?
*10.19 | A possible local search algorithm for the paragraphing problem is to allow local transformations that move the first word of one line to the previous line or the last word of a line to the line following. Is this algorithm locally optimal, in the sense that every locally optimal solution is a globally optimal solution?
10.20 | If our local transformations consist of 2-opts only, are there any locally optimal tours in Fig. 10.21 that are not globally optimal?

Bibliographic Notes

There are many important examples of divide-and-conquer algorithms including the O(nlogn) Fast Fourier Transform of Cooley and Tukey [1965], the O(nlognloglogn) integer multiplication algorithm of Schonhage and Strassen [1971], and the O(n2.81) matrix multiplication algorithm of Strassen [1969]. The O(n1.59) integer multiplication algorithm is from Karatsuba and Ofman [1962]. Moenck and Borodin [1972] develop several efficient divide-and-conquer algorithms for modular arithmetic and polynomial interpolation and evaluation.

Dynamic programming was popularized by Bellman [1957]. The application of dynamic programming to triangulation is due to Fuchs, Kedem, and Uselton [1977]. Exercise 10.11 is from Knuth [1981]. Knuth [1971] contains a solution to the optimal binary search tree problem in Exercise 10.12.

Lin and Kernighan [1973] describe an effective heuristic for the traveling salesman problem.

See Aho, Hopcroft, and Ullman [1974] and Garey and Johnson [1979] for a discussion of NP-complete and other computationally difficult problems.

1. In the towers of Hanoi case, the divide-and-conquer algorithm is really the same as the one given initially.

†In what follows, we take all subscripts to be computed modulo n. Thus, in Fig. 10.8, vi and vi+1 could be v6 and v0, respectively, since n = 7.

†Remember that the table of Fig. 10.11 has rows of 0's below those shown.

‡By "to the right" we mean in the sense of a table that wraps around. Thus, if we are at the rightmost column, the column "to the right" is the leftmost column.

†In fact, we should be careful what we mean by "shortest path" when there are negative edges. If we allow negative cost cycles, then we could traverse such a cycle repeatedly to get arbitrarily large negative distances, so presumably we want to restrict ourselves to acyclic paths.

†Incidentally, some of the other things good chessplaying programs do are:

  1. Use heuristics to eliminate from consideration certain moves that are unlikely to be good. This helps expand the tree to more levels in a fixed time.

  2. Expand "capture chains", which are sequences of capturing moves beyond the last level to which the tree is normally expanded. This helps estimate the relative material strength of positions more accurately.

  3. Prune the tree search by alpha-beta pruning, as discussed later in this section.

‡We should not imply that only "games" can be solved in this manner. As we shall see in subsequent examples, the "game" could really represent the solution to a practical problem.

†Note that we need not consider all n! permutations, since the starting point of a tour is immaterial. We may therefore consider only those permutations that begin with 1.

†The rules for constructing the search tree will be seen to eliminate any set of constraints that cannot yield any tour, e.g., because three edges adjacent to one node are required to be in the tour.

†An alternative is to use a heuristic to obtain a good solution using the constraints required for each child. For example, the reader should be able to modify the greedy TSP algorithm to respect constraints.

‡We could start with some heuristically found solution, say the greedy one, although that would not affect this example. The greedy solution for Fig. 10.21 has cost 21.

†Do not be fooled by the picture of Fig. 10.25. True, if lengths of edges are distances in the plane, then the dashed edges in Fig. 10.25 must be longer than those they replace. In general, however, there is no reason to assume the distances in Fig. 10.25 are distances in the plane, or if they are, it could have been (A B) and (C, D) that crossed, not (a C) and (B, D).

Table of Contents Go to Chapter 11


Data Structures and Algorithms: CHAPTER 11: Data Structures and Algorithms for External Storage

 Next Chapter Return to Table of Contents Previous Chapter

Data Structures and Algorithms for External Storage

We begin this chapter by considering the differences in access characteristics between main memory and external storage devices such as disks. We then present several algorithms for sorting files of externally stored data. We conclude the chapter with a discussion of data structures and algorithms, such as indexed files and B-trees, that are well suited for storing and retrieving information on secondary storage devices.

11.1 A Model of External Computation

In the algorithms discussed so far, we have assumed that the amount of input data is sufficiently small to fit in main memory at the same time. But what if we want to sort all the employees of the government by length of service or store all the information in the nation's tax returns? In such problems the amount of data to be processed exceeds the capacity of the main memory. Most large computer systems have on-line external storage devices, such as disks or mass storage devices, on which vast quantities of data can be stored. These on-line storage devices, however, have access characteristics that are quite different from those of main memory. A number of data structures and algorithms have been developed to utilize these devices more effectively. This chapter discusses data structures and algorithms for sorting and retrieving information stored in secondary memory.

Pascal, and some other languages, provide the file data type, which is intended to represent data stored in secondary memory. Even if the language being used does not have a file data type, the operating system undoubtedly supports the notion of files in secondary memory. Whether we are talking about Pascal files or files maintained by the operating system directly, we are faced with limitations on how files may be accessed. The operating system divides secondary memory into equal-sized blocks. The size of blocks varies among operating systems, but 512 to 4096 bytes is typical.

We may regard a file as stored in a linked list of blocks, although more typically the operating system uses a tree-like arrangement, where the blocks holding the file are leaves of the tree, and interior nodes each hold pointers to many blocks of the file. If, for example, 4 bytes suffice to hold the address of a block, and blocks are 4096 bytes long, then a root block can hold pointers to up to 1024 blocks. Thus, files of up to 1024 blocks, i.e., about four million bytes, could be represented by a root block and blocks holding the file. Files of up to 220 blocks, or 232 bytes could be represented by a root block pointing to 1024 blocks at an intermediate level, each of which points to 1024 leaf blocks holding a part of the file, and so on.

The basic operation on files is to bring a single block to a buffer in main memory; a buffer is simply a reserved area of main memory whose size is the same as the size of a block. A typical operating system facilitates reading the blocks in the order in which they appear in the list of blocks that holds the file. That is, we initially read the first block into the buffer for that file, then replace it by the second block, which is written into the same buffer, and so on.

We can now see the rationale behind the rules for reading Pascal files. Each file is stored in a sequence of blocks, with a whole number of records in each block. (Space may be wasted, as we avoid having one record split across block boundaries.) The read-cursor always points to one of the records in the block that is currently in the buffer. When that cursor must move to a record not in the buffer, it is time to read the next block of the file.

Similarly, we can view the Pascal file-writing process as one of creating a file in a buffer. As records are "written" into the file, they are placed in the buffer for that file, in the position immediately following any previously placed records. When the buffer cannot hold another complete record, the buffer is copied into an available block of secondary storage and that block is appended to the end of the list of blocks for that file. We can now regard the buffer as empty and write more records into it.

The Cost Measure for Secondary Storage Operations

It is the nature of secondary storage devices such as disks that the time to find a block and read it into main memory is large compared with the time to process the data in that block in simple ways. For example, suppose we have a block of 1000 integers on a disk rotating at 1000 rpm. The time to position the head over the track holding this block (seek time) plus the time spent waiting for the block to come around to the head (latency time) might average 100 milliseconds. The process of writing a block into a particular place on secondary storage takes a similar amount of time. However, the machine could typically do 100,000 instructions in those 100 milliseconds. This is more than enough time to do simple processing to the thousand integers once they are in main memory, such as summing them or finding their maximum. It might even be sufficient time to quicksort the integers.

When evaluating the running time of algorithms that operate on data stored as files, we are therefore forced to consider as of primary importance the number of times we read a block into main memory or write a block onto secondary storage. We call such an operation a block access. We assume the size of blocks is fixed by the operating system, so we cannot appear to make an algorithm run faster by increasing the block size, thereby decreasing the number of block accesses. As a consequence, the figure of merit for algorithms dealing with external storage will be the number of block accesses. We begin our study of algorithms for external storage by looking at external sorting.

11.2 External Sorting

Sorting data organized as files, or more generally, sorting data stored in secondary memory, is called "external" sorting. Our study of external sorting begins with the assumption that the data are stored on a Pascal file. We show how a "merge sorting" algorithm can sort a file of n records with only O(log n) passes through the file; that figure is substantially better than the O(n) passes needed by the algorithms studied in Chapter 8. Then we consider how utilization of certain powers of the operating system to control the reading and writing of blocks at appropriate times can speed up sorting by reducing the time that the computer is idle, waiting for a block to be read into or written out of main memory.

Merge Sorting

The essential idea behind merge sort is that we organize a file into progressively larger runs, that is, sequences of records r1, . . . , rk, where the key of ri is no greater than the key of ri+1 for 1 � i < k. We say a file r1, . . . ,rm of records is organized into runs of length k if for all i � 0 such that ki � m, rk(i-l)+l, rk(i-1)+2, . . . ,rki is a run of length k, and furthermore if m is not divisible by k, and m = pk+q, where q<k, then the sequence of records rm-q+1, rm-q+2, . . . ,rm, called the tail, is a run of length q. For example, the sequence of integers shown in Fig. 11.1 is organized into runs of length 3 as shown. Note that the tail is of length less than 3, but consists of records in sorted order, namely 5, 12.

Fig. 11.1. File with runs of length three.

The basic step of a merge sort on files is to begin with two files, say f1 and f2, organized into runs of length k. Assume that

  1. the numbers of runs, including tails, on f1 and f2 differ by at most one,
  2. at most one of f1 and f2 has a tail, and
  3. the one with a tail has at least as many runs as the other.

Then it is a simple process to read one run from each of f1 and f2, merge the runs and append the resulting run of length 2k onto one of two files g1 and g2, which are being organized into runs of length 2k. By alternating between g1 and g2, we can arrange that these files are not only organized into runs of length 2k, but satisfy (l), (2), and (3) above. To see that (2) and (3) are satisfied it helps to observe that the tail among the runs of f1 and f2 gets merged into (or perhaps is) the last run created.

We begin by dividing all n of our records into two files f1 and f2, as evenly as possible. Any file can be regarded as organized into runs of length 1. Then we can merge the runs of length 1 and distribute them into files g1 and g2, organized into runs of length 2. We make f1 and f2 empty, and merge g1 and g2 into f1 and f2, which will then be organized into runs of length 4. Then we merge f1 and f2 to create g1 and g2 organized into runs of length 8, and so on.

After i passes of this nature, we have two files consisting of runs of length 2i. If 2i � n, then one of the two files will be empty and the other will contain a single run of length n, i.e., it will be sorted. As 2i � n when i � logn, we see that [logn] passes suffice. Each pass requires the reading of two files and the writing of two files, all of length about n/2. The total number of blocks read or written on a pass is thus about 2n/b, where b is the number of records that fit on one block. The number of block reads and writes for the entire sorting process is thus O((nlogn)/b), or put another way, the amount of reading and writing is about the same as that required by making O(logn) passes through the data stored on a single file. This figure is a large improvement over the O(n) passes required by many of the sorting algorithms discussed in Chapter 8.

Figure 11.2 shows the merge process in Pascal. We read two files organized into runs of length k and write two files organized into runs of length 2k. We leave to the reader the specification of an algorithm, following the ideas above, that uses the procedure merge of Fig. 11.2 logn times to sort a file of n records.

  procedure merge ( k: integer; { the input run length }
         f1, f2, g1, g2: file of recordtype );

  var
      outswitch: boolean; { tells if writing g1 (true) or g2 (false) }
      winner: integer; { selects file with smaller key in current record }
      used: array [1..2] of integer; { used[j] tells how many
         records have been read so far from the current run of file fj }
      fin: array [1..2] of boolean; { fin[j] is true if we have
         finished the run from fj - either we have read k records,
         or reached the end of the file of fj }
      current: array [1..2] of recordtype; { the current records
         from the two files }

   procedure getrecord ( i: integer ); { advance file
fi, but
          not beyond the end of the file or the end of the run.
          Set fin[i] if end of file or run found }
     begin
          used[i] := used[i] + 1;
          if (used[i] = k) or
               (i = 1) and eof(f 1) or
               (i = 2) and eof(f 2) then fin[i]:= true
          else if i = 1 then read(f 1, current[1])
          else read(f 2, current[2])
      end; { getrecord }

   begin { merge }
      outswitch := true; { first merged run goes to g 1 }
      rewrite(g 1); rewrite(g 2);
      reset(f 1); reset(f 2);
      while not eof(f 1) or not eof(f 2) do begin
{ merge two file }
          { initialize }
          used[1] := 0; used[2] := 0;
          fin[1] := false; fin[2] := false;
          getrecord(1); getrecord(2);
          while not fin[1] or not fin[2] do begin { merge two runs
}
              { select winner }
              if fin[1] then winner : = 2
                  { f 2 wins by "default" - run from f 1 exhausted }
              else if fin[2] then winner := 1
                  { f 1 wins by default }
              else { neither run exhausted }
                  if current[1].key < current[2].key then
winner := 1
                  else winner := 2;
              { write winning record }
              if outswitch then write(g 1,
current[winner])
              else write(g 2, current[winner]);
              { advance winning file }
              getrecord(winner)
          end;
          { we have finished merging two runs - switch output
              file and repeat }
          outswitch := not outswitch
      end
   end; { merge }

Fig. 11.2. The procedure merge.

Notice that the procedure merge, of Fig. 11.2, is not required ever to have a complete run in memory; it reads and writes a record at a time. It is our desire not to store whole runs in main memory that forces us to have two input files. Otherwise, we could read two runs at a time from one file.

Example 11.1. Suppose we have the list of 23 numbers shown divided into two files in Fig. 11.3(a). We begin by merging runs of length 1 to create the two files of Fig. 11.3(b). For example, the first runs of length 1 are 28 and 31; these are merged by picking 28 and then 31. The next two runs of length one, 3 and 5, are merged to form a run of length two, and that run is placed on the second file of Fig. ll.3(b). The runs are separated in Fig. 11.3(b) by vertical lines that are not part of the file. Notice that the second file in Fig. 11.3(b) has a tail of length one, the record 22, while the first file has no tail.

We go from Fig. 11.3(b) to (c) by merging runs of length two. For example, the two runs 28, 31 and 3, 5 are merged to form 3, 5, 28, 31 in Fig. 11.3(c). By the time we get to runs of length 16 in Fig. 11.3(e), one file has one complete run and the other has only a tail, of length 7. At the last stage, where the files are ostensibly organized as runs of length 32, we in fact have one file with a tail only, of length 23, and the second file is empty. The single run of length 23 is, of course, the desired sorted order.

Speeding Up Merge Sort

We have, for the sake of a simple example, shown the merge sort process as starting from runs of length one. We shall save considerable time if we begin with a pass that, for some appropriate k, reads groups of k records into main memory, sorts them, say by quicksort, and writes them out as a run of length k.

For example, if we have a million records, it would take 20 passes through the data to sort starting with runs of length one. If, however, we can hold 10,000 records at once in main memory, we can, in one pass, read 100 groups of 10,000 records, sort each group, and be left with 100 runs of length 10,000 distributed evenly between two files. Seven more merging passes results in a file organized as one run of length 10,000 x 27 = 1,280,000, which is more than a million and means that the data are sorted.

Minimizing Elapsed Time

In modern time-shared computer systems, one is generally not charged for the time one's program spends waiting for blocks of a file to be read in, as must happen during the merge sort process. However, the fact is that the elapsed time of such a sorting process is greater, often substantially so, than the time spent computing with data found in the main memory. If we are sorting really large files, where the whole operation takes hours, the elapsed time becomes important, even if we do not "pay" for the time, and we should consider how the merging process might be performed in minimum elapsed time.

As was mentioned, it is typical that the time to read data from a disk or tape is greater than the time spent doing simple computations with that data,

Fig. 11.3. Merge-sorting a list.

such as merging. We should therefore expect that if there is only one channel over which data can be transferred into or out of main memory, then this channel will form a bottleneck; the data channel will be busy all the time, and the total elapsed time equals the time spent moving data into and out of main memory. That is, all the computation will be done almost as soon as the data becomes available, while additional data are being read or written.

Even in this simple environment, we must exercise a bit of care to make sure that we are done in the minimum amount of time. To see what can go wrong, suppose we read the two input files f1 and f2 one block at a time, alternately. The files are organized into runs of some length much larger than the size of a block, so to merge two runs we must read many blocks from each file. However, suppose it happens that all the records in the run from file f1 happen to precede all the records from file f2. Then as we read blocks alternately, all the blocks from f2 have to remain in memory. There may not be space to hold all these blocks in main memory, and even if there is, we must, after reading all the blocks of the run, wait while we copy and write the whole run from f2.

To avoid these problems, we consider the keys of the last records in the last blocks read from f1 and f2, say keys k1 and k2, respectively. If either run is exhausted, we naturally read next from the other file. However, if a run is not exhausted, we next read a block from f1 if k1<k2, and we read from f2 otherwise. That is, we determine which of the two runs will first have all its records currently in main memory selected, and we replenish the supply of records for that run first. If selection of records proceeds faster than reading, we know that when we have read the last block of the two runs, there cannot be more than two "blockfuls" of records left to merge; perhaps the records will be distributed over three blocks, but never more.

Multiway Merge

If reading and writing between main and secondary memory is the bottleneck, perhaps we could save time if we had more than one data channel. Suppose, for example, that we have 2m disk units, each with its own channel. We could place m files, f1, f2, . . .,fm on m of the disk units, say organized as runs of length k. We can read m runs, one from each file, and merge them into one run of length mk. This run is placed on one of m output files g1, g2, . . . ,gm, each getting a run in turn.

The merging process in main memory can be carried out in O(logm) steps per record if we organize the m candidate records, that is, the currently smallest unselected records from each file, into a partially ordered tree or other data structure that supports the priority queue operations INSERT and DELETEMIN in logarithmic time. To select from the priority queue the record with the smallest key, we perform DELETEMIN, and then INSERT into the priority queue the next record from the file of the winner, as a replacement for the selected record.

If we have n records, and the length of runs is multiplied by m with each pass, then after i passes runs will be of length mi. If mi � n, that is, after i = logmn passes, the entire list will be sorted. As logmn = log2n/log2m, we save by a factor of log2m in the number of times we read each record. Moreover, if m is the number of disk units used for input files, and m are used for output, we can process data m times as fast as if we had only one disk unit for input and one for output, or 2m times as fast as if both input and output files were stored on one disk unit. Unfortunately, increasing m indefinitely does not speed processing by a factor of logm. The reason is that for large enough m, the time to do the merging in main memory, which is actually increasing as logm, will exceed the time to read or write the data. At that point, further increases in m actually increase the elapsed time, as computation in main memory becomes the bottleneck.

Polyphase Sorting

We can perform an m-way merge sort with only m + 1 files, as an alternative to the 2m-file strategy described above. A sequence of passes is made, merging runs from m of the files into longer runs on the remaining file. The insights needed are the following:

  1. In one pass, when runs from each of m files are merged into runs of the m + 1st file, we need not use all the runs on each of the m input files. Rather, each file, when it becomes the output file, is filled with runs of a certain length. It uses some of these runs to help fill each of the other m files when it is their turn to be the output file.
  2. Each pass produces files of a different length. Since each of the files loaded with runs on the previous m passes contributes to the runs of the current pass, the length on one pass is the sum of the lengths of the runs produced on the previous m passes. (If fewer than m passes have taken place, regard passes prior to the first as having produced runs of length 1.)

Such a merge-sorting process is called a polyphase sort. The exact calculation of the numbers of passes needed, as a function of m and n (the number of records), and calculation of the initial distribution of the runs into m files are left as exercises. However, we shall give one example here to suggest the general case.

Example 11.2. If m = 2, we start with two files f1 and f2, organized into runs of length 1. Records from f1 and f2 are merged to make runs of length 2 on the third file, f3. Just enough runs are merged to empty f1. We then merge the remaining runs of length 1 from f2 with an equal number of runs of length 2 from f3. These yield runs of length 3, and are placed on f1. Then we merge runs of length 2 on f3 with runs of length 3 on f1. These runs, of length 5, are placed on f2, which was emptied at the previous pass.

The run length sequence: 1, 1, 2, 3, 5, 8, 13, 21, . . . . . is the Fibonacci sequence. This sequence is generated by the recurrence relation F0 = F1 = 1, and Fi = Fi-1 + Fi- 2, for i � 2. Note that the ratio of consecutive Fibonacci numbers Fi+1/Fi approaches the "golden ratio" (�`5+1)/2 = 1.618 . . . as i gets large.

It turns out that in order to keep this process going, until the list is sorted, the initial numbers of records on f1 and f2 must be two consecutive Fibonacci numbers. For example, Fig. 11.4 shows what happens when we start with n = 34 records (34 is the Fibonacci number F8) distributed 13 on f1 and 21 on f2. (13 and 21 are F6 and F7, so the ratio F7/F6 is very close to 1.618. It is 1.615, in fact). The status of a file is represented in Fig. 11.4 as a(b), meaning it has a runs of length b.

Fig. 11.4. Example of polyphase sorting.

When Input/Output Speed is not a Bottleneck

When reading files is the bottleneck, the next block read must be carefully chosen. As we have mentioned, the situation to avoid is one where we have to store many blocks of one run because that run happened to have records with high keys, which get selected after most or all of the records of the other run. The trick to avoiding this problem is to determine quickly which run will first exhaust those of its records currently in main memory, by comparing the last such records in each file.

When the time taken to read data into main memory is comparable to, or less than, the time taken to process the data, it becomes even more critical that we select the input file from which to read a block carefully, since there is no hope of building a reserve of records in main memory in case the merge process suddenly starts taking more records from one run than the other. The "trick" mentioned above helps us in a variety of situations, as we shall see.

We consider the case where merging, rather than reading or writing, is a bottleneck for two reasons.

  1. As we have seen, if we have many disk or tape units available, we may speed input/output sufficiently that the time to do the merge exceeds input time or output time.
  2. Higher speed channels may yet become commercially available.

We therefore shall consider a simple model of the problem that might be encountered when merging becomes the bottleneck in a merge sort performed on data stored in secondary memory. Specifically, we assume that

  1. We are merging runs that are much larger than blocks.
  2. There are two input files and two output files. The input files are stored on one disk (or other device connected to main memory by a single channel) and the output files are on another similar unit with one channel.
  3. The times to
    1. read a block
    2. write a block, and
    3. select enough of the lowest-keyed records from among those of two runs presently in main memory to fill a block,
    are all the same.

Under these assumptions, we can consider a class of merging strategies where several input buffers (space to hold a block) are allocated in main memory. At all times some of these buffers will hold the unselected records from the two input runs, and one of them will be in the process of being read into from one of the input files. Two other buffers will hold output, that is, the selected records in their properly merged order. At all times, one of these buffers is in the process of being written into one of the output files and the other is being filled from records selected from the input buffers.

A stage consists of doing the following (possibly at the same time):

  1. reading an input block into an input buffer,
  2. filling one of the output buffers with selected records, that is, records with the smallest keys among those currently held in the input buffer, and
  3. writing the other output buffer into one of the two output files being formed.

By our assumptions, (1), (2), and (3) all take the same time. For maximum efficiency, we must do them in parallel. We can do so, unless (2), the selection of records with the smallest keys, includes some of the records currently being read in.† We must thus devise a strategy for selecting buffers to be read so that at the beginning of each stage the b unselected records with smallest keys are already in input buffers, where b is the number of records that fills a block or buffer.

The conditions under which merging can be done in parallel with reading are simple. Let k1 and k2 be the highest keys among unselected records in main memory from the first and second runs, respectively. Then there must be in main memory at least b unselected records whose keys do not exceed min(k1, k2). We shall first show how to do the merge with six buffers, three used for each file, then show that four buffers suffice if we share them between the two files.

A Six-Input Buffer Scheme

Our first scheme is represented by the picture in Fig. 11.5. The two output buffers are not shown. There are three buffers for each file; each buffer has capacity b records. The shaded area represents available records, and keys are in increasing order clockwise around the circles. At all times, the total number of unselected records is 4b (unless fewer than that number of records remain from the runs being merged). Initially, we read the first two blocks from each run into buffers.† As there are always 4b records available, and at most 3b can be from one file, we know there are b records at least from each file. If k1 and k2 are the largest available keys from the two runs, there must be b records with keys equal to or less than k1 and b with keys equal to or less than k2. Thus, there are b records with keys equal to or less than min(k1, k2).

Fig. 11.5. A six-input buffer merge.

The question of which file to read from next is trivial. Usually, since two buffers will be partially full, as in Fig. 11.5, there will be only one empty buffer, and this must be filled. If it happens, as in the beginning, that each run has two completely filled and one empty buffer, pick either empty buffer to fill. Note that our proof that we could not exhaust a run [b records with keys equal to or less than min(k1, k2) exist] depended only on the fact that 4b records were present.

As an aside, the arrows in Fig. 11.5 represent pointers to the first (lowest-keyed) available records from the two runs. In Pascal, we could represent such a pointer by two integers. The first, in the range 1..3, represents the buffer pointed to, and the second, in the range 1..b, represents the record within the buffer. Alternatively, we could let the buffers be the first, middle, and last thirds of one array and use one integer in the range 1..3b. In other languages, where pointers can point to elements of arrays, a pointer of type � recordtype would be preferred.

A Four-Buffer Scheme

Figure 11.6 suggests a four-buffer scheme. At the beginning of each stage, 2b records are available. Two of the input buffers are assigned to one of the files; B1 and B2 in Fig. 11.6 are assigned to file one. One of these buffers will be partially full (empty in the extreme case) and the other full. A third buffer is assigned to the other file, as B3 is assigned to file two in Fig. 11.6. It is partially full (completely full in the extreme case). The fourth buffer is uncommitted, and will be filled from one of the files during the stage.

Fig. 11.6. A four-input buffer merge.

We shall maintain, of course, the property that allows us to merge in parallel with reading; at least b of the records in Fig. 11.6 must have keys equal to or less than min(k1, k2), where k1 and k2 are the keys of the last available records from the two files, as indicated in Fig. 11.6. We call a configuration that obeys the property safe. Initially, we read one block from each run (this is the extreme case, where B1 is empty and B3 is full in Fig. 11.6), so that the initial configuration is safe. We must, on the assumption that Fig. 11.6 is safe, show that the configuration will be safe after the next stage is complete.

If k1<k2, we choose to fill B4 with the next block from file one, and otherwise, we fill it from file two. Suppose first that k1<k2. Since B1 and B3 in Fig. 11.6 have together exactly b records, we must, during the next stage, exhaust B1, else we would exhaust B3 and contradict the safety of Fig. 11.6. Thus after a stage the configuration looks like Fig. 11.7(a).

Fig. 11.7. Configuration after one stage.

To see that Fig. 11.7(a) is safe, consider two cases. First, if k3, the last key in the newly-read block B4, is less than k2, then as B4 is full, there are surely b records equal to or less than min(k3, k2), and the configuration is safe. If k2�k3, then since k1<k2 was assumed (else we would have filled B4 from file two), the b records in B2 and B3 have keys equal to or less than min(k2, k3)=k2.

Now let us consider the case where k1�k2 in Fig. 11.6. Then we choose to read the next block from file two. Figure 11.7(b) shows the resulting situation. As in the case k1<k2, we can argue that B1 must exhaust, which is why we show file one as having only buffer B2 in Fig. 11.7(b). The proof that Fig. 11.7(b) is safe is just like the proof for Fig. 11.7(a).

Note that, as in the six buffer scheme, we do not read a file past the end of a run. However, if there is no need to read in a block from one of the present runs, we can read a block from the next run on that file. In this way we have the opportunity to read one block from each of the next runs, and we are ready to begin merging runs as soon as we have selected the last records of the previous run.

11.3 Storing Information in Files

In this section we consider data structures and algorithms for storing and retrieving information in externally stored files. We shall view a file as a sequence of records, where each record consists of the same sequence of fields. Fields can be either fixed length, having a predetermined number of bytes, or variable length, having an arbitrary size. Files with fixed length records are commonly used in database management systems to store highly structured data. Files with variable length records are typically used to store textual information; they are not available in Pascal. In this section we shall assume fixed-length fields; the fixed-length techniques can be modified simply to work for variable-length records.

The operations on files we shall consider are the following.

  1. INSERT a particular record into a particular file.
  2. DELETE from a particular file all records having a designated value in each of a designated set of fields.
  3. MODIFY all records in a particular file by setting to designated values certain fields in those records that have a designated value in each of another set of fields.
  4. RETRIEVE all records having designated values in each of a designated set of fields.

Example 11.3. For example, suppose we have a file whose records consist of three fields: name, address, and phone. We might ask to retrieve all records with phone = "555-1234", to insert the record ("Fred Jones", "12 Apple St.", "555-1234"), or to delete all records with name = "Fred Jones" and address = "12 Apple St." As another example, we might wish to modify all records with name = "Fred Jones" by setting the phone field to "555- 1234".

To a great extent we can view operations on files as if the files were sets of records and the operations were those discussed in Chapters 4 and 5. There are two important differences, however. First, when we talk of files on external storage devices, we must use the cost measure discussed in Section l l.l when evaluating strategies for organizing the files. That is, we assume that files are stored in some number of physical blocks, and the cost of an operation is the number of blocks that we must read into main memory or write from main memory onto external storage.

The second difference is that records, being concrete data types of most programming languages, can be expected to have pointers to them, while the abstract elements of a set would not normally have "pointers" to them. In particular, database systems frequently make use of pointers to records when organizing data. The consequence of such pointers is that records frequently must be regarded as pinned; they cannot be moved around in storage because of the possibility that a pointer from some unknown place would fail to point to the record after it was moved.

A simple way to represent pointers to records is the following. Each block has a physical address, which is the location on the external storage device of the beginning of the block. It is the job of the file system to keep track of physical addresses. One way to represent record addresses is to use the physical address of the block holding the record together with an offset, giving the number of bytes in the block preceding the beginning of the record. These physical address--offset pairs can then be stored in fields of type "pointer to record."

A Simple Organization

The simplest, and also least efficient, way to implement the above file operations is to use the file reading and writing primitives such as found in Pascal. In this "organization" (which is really a "lack of organization"), records can be stored in any order. Retrieving a record with specified values in certain fields is achieved by scanning the file and looking at each record to see if it has the specified values. An insertion into a file can be performed by appending the record to the end of the file.

For modification of records, scan the file and check each record to see if it matches the designated fields, and if so, make the required changes to the record. A deletion operation works almost the same way, but when we find a record whose fields match the values required for the deletion to take place, we must find a way to delete the record. One possibility is to shift all subsequent records one position forward in their blocks, and move the first record of each subsequent block into the last position of the previous block of the file. However, this approach will not work if records are pinned, because a pointer to the ith record in the file would then point to the i + 1st record.

If records are pinned, we must use a somewhat different approach. We mark deleted records in some way, but we do not move records to fill their space, nor do we ever insert a new record into their space. Thus, the record becomes deleted logically from the file, but its space is still used for the file. This is necessary so that if we ever follow a pointer to a deleted record, we shall discover that the record pointed to was deleted and take some appropriate action, such as making the pointer NIL so it will not be followed again. Two ways to mark records as deleted are:

  1. Replace the record by some value that could never be the value of a "real" record, and when following a pointer, assume the record is deleted if it has that value.
  2. Let each record have a deletion bit, a single bit that is 1 in records that have been deleted and 0 otherwise.

Speeding Up File Operations

The obvious disadvantage of a sequential file is that file operations are slow. Each operation requires us to read the entire file, and some blocks may have to be rewritten as well. Fortunately, there are file organizations that allow us to access a record by reading into main memory only a small fraction of the entire file.

To make such organizations possible, we assume each record of a file has a key, a set of fields that uniquely identifies each record. For example, the name field of the name-address-phone file might be considered a key by itself. That is, we might assume that two records with the same name field value cannot exist simultaneously in the file. Retrieval of a record, given values for its key fields, is a common operation, and one that is made especially easy by many common file organizations.

Another element we need for fast file operations is the ability to access blocks directly, rather than running sequentially through the blocks holding a file. Many of the data structures we use for fast file operations will use pointers to the blocks themselves, which are the physical addresses of the blocks, as described above. Unfortunately, we cannot write in Pascal, or in many other languages, programs that deal with data on the level of physical blocks and their addresses; such operations are normally done by file system commands. However, we shall briefly give an informal description of how organizations that make use of direct block access work.

Hashed Files

Hashing is a common technique used to provide fast access to information stored on secondary files. The basic idea is similar to open hashing discussed in Section 4.7. We divide the records of a file among buckets, each consisting of a linked list of one or more blocks of external storage. The organization is similar to that portrayed in Fig. 4.10. There is a bucket table containing B pointers, one for each bucket. Each pointer in the bucket table is the physical address of the first block of the linked-list of blocks for that bucket.

The buckets are numbered 0, 1, . . . . ,B - 1. A hash function h maps each key value into one of the integers 0 through B - 1. If x is a key, h(x) is the number of the bucket that contains the record with key x, if such a record is present at all. The blocks making up each bucket are chained together in a linked list. Thus, the header of the ith block of a bucket contains a pointer to the physical address of the i + 1st block. The last block of a bucket contains a NIL pointer in its header.

This arrangement is illustrated in Fig. 11.8. The major difference between Figs. 11.8 and 4.10 is that here, elements stored in one block of a bucket do not need to be chained by pointers; only the blocks need to be chained.

Fig. 11.8. Hashing with buckets consisting of chained blocks.

If the size of the bucket table is small, it can be kept in main memory. Otherwise, it can be stored sequentially on as many blocks as necessary. To look for the record with key x, we compute h(x), and find the block of the bucket table containing the pointer to the first block of bucket h(x). We then read the blocks of bucket h(x) successively, until we find a block that contains the record with key x. If we exhaust all blocks in the linked list for bucket h(x), we conclude that x is not the key of any record.

This structure is quite efficient if the operation is one that specifies values for the fields in the key, such as retrieving the record with a specified key value or inserting a record (which, naturally, specifies the key value for that record). The average number of block accesses required for an operation that specifies the key of a record is roughly the average number of blocks in a bucket, which is n/bk if n is the number of records, a block holds b records, and k is the number of buckets. Thus, on the average, operations based on keys are k times faster with this organization than with the unorganized file. Unfortunately, operations not based on keys are not speeded up, as we must examine essentially all the buckets during these other operations. The only general way to speed up operations not based on keys seems to be the use of secondary indices, which are discussed at the end of this section.

To insert a record with key value x, we first check to see if there is already a record with key x. If there is, we report error, since we assume that the key uniquely identifies each record. If there is no record with key x, we insert the new record in the first block in the chain for bucket h(x) into which the record can fit. If the record cannot fit into any existing block in the chain for bucket h(x), we call upon the file system to find a new block into which the record is placed. This new block is then added to the end of the chain for bucket h(x).

To delete a record with key x, we first locate the record, and then set its deletion bit. Another possible deletion strategy (which cannot be used if the records are pinned) is to replace the deleted record with the last record in the chain for h(x). If the removal of the last record makes the last block in the chain for h(x) empty, we can then return the empty block to the file system for later re-use.

A well-designed hashed-access file organization requires only a few block accesses for each file operation. If our hash function is good, and the number of buckets is roughly equal to the number of records in the file divided by the number of records that can fit on one block, then the average bucket consists of one block. Excluding the number of block accesses to search the bucket table, a typical retrieval based on keys will then take one block access, and a typical insertion, deletion, or modification will take two block accesses. If the average number of records per bucket greatly exceeds the number that will fit on one block, we can periodically reorganize the hash table by doubling the number of buckets and splitting each bucket into two. The ideas were covered at the end of Section 4.8.

Indexed Files

Another common way to organize a file of records is to maintain the file sorted by key values. We can then search the file as we would a dictionary or telephone directory, scanning only the first name or word on each page. To facilitate the search we can create a second file, called a sparse index, which consists of pairs (x, b), where x is a key value and b is the physical address of the block in which the first record has key value x. This sparse index is maintained sorted by key values.

Example 11.4. In Fig. 11.9 we see a file and its sparse index file. We assume that three records of the main file, or three pairs of the index file, fit on one block. Only the key values, assumed to be single integers, are shown for records in the main file.

To retrieve a record with a given key x, we first search the index file for a pair (x, b). What we actually look for is the largest z such that z � x and there is a pair (z, b) in the index file. Then key x appears in block b if it is present in the main file at all.

There are several strategies that can be used to search the index file. The simplest is linear search. We read the index file from the beginning until we encounter a pair (x, b), or until we encounter the first pair (y, b) where y > x. In the latter case, the preceding pair (z, b�) must have z < x, and if the record with key x is anywhere, it is in block b�.

Linear search is only suitable for small index files. A faster method is

Fig. 11.9. A main file and its sparse index.

binary search. Assume the index file is stored on blocks b1, b2, . . . ,bn. To search for key value x, we take the middle block b[n/2] and compare x with the key value y in the first pair in that block. If x < y, we repeat the search on blocks b1, b2, . . .,b[n/2]-1. If x � y, but x is less than the key of block b[n/2]+1 (or if n=1, so there is no such block), we use linear search to see if x matches the first component of an index pair on block b[n/2]. Otherwise, we repeat the search on blocks b[n/2]+1, b[n/2]+2, . . . ,bn. With binary search we need examine only [log2(n + 1)] blocks of the index file.

To initialize an indexed file, we sort the records by their key values, and then distribute the records to blocks, in that order. We may choose to pack as many as will fit into each block. Alternatively, we may prefer to leave space for a few extra records that may be inserted later. The advantage is that then insertions are less likely to overflow the block into which the insertion takes place, with the resultant requirement that adjacent blocks be accessed. After partitioning the records into blocks in one of these ways, we create the index file by scanning each block in turn and finding the first key on each block. Like the main file, some room for growth may be left on the blocks holding the index file.

Suppose we have a sorted file of records that are stored on blocks B1, B2, . . . ,Bm. To insert a new record into this sorted file, we use the index file to determine which block Bi should contain the new record. If the new record will fit in Bi, we place it there, in the correct sorted order. We then adjust the index file, if the new record becomes the first record on Bi.

If the new record cannot fit in Bi, a variety of strategies are possible. Perhaps the simplest is to go to block Bi+1, which can be found through the index file, to see if the last record of Bi can be moved to the beginning of Bi+1. If so, this last record is moved to Bi+1, and the new record can then be inserted in the proper position in Bi. The index file entry for Bi+1, and possibly for Bi, must be adjusted appropriately.

If Bi+1 is also full, or if Bi is the last block (i = m), then a new block is obtained from the file system. The new record is inserted in this new block, and the new block is to follow block Bi in the order. We now use this same procedure to insert a record for the new block in the index file.

Unsorted Files with a Dense Index

Another way to organize a file of records is to maintain the file in random order and have another file, called a dense index, to help locate records. The dense index consists of pairs (x, p), where p is a pointer to the record with key x in the main file. These pairs are sorted by key value, so a structure like the sparse index mentioned above, or the B-tree mentioned in the next section, could be used to help find keys in the dense index.

With this organization we use the dense index to find the location in the main file of a record with a given key. To insert a new record, we keep track of the last block of the main file and insert the new record there, getting a new block from the file system if the last block is full. We also insert a pointer to that record in the dense index file. To delete a record, we simply set the deletion bit in the record and delete the corresponding entry in the dense index (perhaps by setting a deletion bit there also).

Secondary Indices

While the hashed and indexed structures speed up operations based on keys substantially, none of them help when the operation involves a search for records given values for fields other than the key fields. If we wish to find the records with designated values in some set of fields F1, . . . . ,Fk we need a secondary index on those fields. A secondary index is a file consisting of pairs (v, p), where v is a list of values, one for each of the fields F1, . . . ,Fk, and p is a pointer to a record. There may be more than one pair with a given v, and each associated pointer is intended to indicate a record of the main file that has v as the list of values for the fields F1, . . . ,Fk.

To retrieve records given values for the fields F1, . . . ,Fk, we look in the secondary index for a record or records with that list of values. The secondary index itself can be organized in any of the ways discussed for the organization of files by key value. That is, we pretend that v is a key for (v, p).

For example, a hashed organization does not really depend on keys being unique, although if there were very many records with the same "key" value, then records might distribute themselves into buckets quite nonuniformly, with the effect that hashing would not speed up access very much. In the extreme, say there were only two values for the fields of a secondary index. Then all but two buckets would be empty, and the hash table would only speed up operations by a factor of two at most, no matter how many buckets there were. Similarly, a sparse index does not require that keys be unique, but if they are not, then there may be two or more blocks of the main file that have the same lowest "key" value, and all such blocks will be searched when we are looking for records with that value.

With either the hashed or sparse index organization for our secondary index file, we may wish to save space by bunching all records with the same value. That is, the pairs (v, p1), (v, p2), . . . ,(v, pm) can be replaced by v followed by the list p1, p2, . . . ,pm.

One might naturally wonder whether the best response time to random operations would not be obtained if we created a secondary index for each field, or even for all subsets of the fields. Unfortunately, there is a penalty to be paid for each secondary index we choose to create. First, there is the space needed to store the secondary index, and that may or may not be a problem, depending on whether space is at a premium.

In addition, each secondary index we create slows down all insertions and all deletions. The reason is that when we insert a record, we must also insert an entry for that record into each secondary index, so the secondary indices will continue to represent the file accurately. Updating a secondary index takes at least two block accesses, since we must read and write one block. However, it may take considerably more than two block accesses, since we have to find that block, and any organization we use for the secondary index file will require a few extra accesses, on the average, to find any block. Similar remarks hold for each deletion. The conclusion we draw is that selection of secondary indices requires judgment, as we must determine which sets of fields will be specified in operations sufficiently frequently that the time to be saved by having that secondary index available more than balances the cost of updating that index on each insertion and deletion.

11.4 External Search Trees

The tree data structures presented in Chapter 5 to represent sets can also be used to represent external files. The B-tree, a generalization of the 2-3 tree discussed in Chapter 5, is particularly well suited for external storage, and it has become a standard organization for indices in database systems. This section presents the basic techniques for retrieving, inserting, and deleting information in B-trees.

Multiway Search Trees

An m-ary search tree is a generalization of a binary search tree in which each node has at most m children. Generalizing the binary search tree property, we require that if n1 and n2 are two children of some node, and n1 is to the left of n2, then the elements descending from n1 are all less than the elements descending from n2. The operations of MEMBER, INSERT, and DELETE for an m-ary search tree are implemented by a natural generalization of those operations for binary search trees, as discussed in Section 5.1.

However, we are interested here in the storage of records in files, where the files are stored in blocks of external storage. The correct adaptation of the multiway tree idea is to think of the nodes as physical blocks. An interior node holds pointers to its m children and also holds the m-1 key values that separate the descendants of the children. Leaf nodes are also blocks; these blocks hold the records of/P>

If we used a binary search tree of n nodes to represent an externally stored file, then it would require log2n block accesses to retrieve a record from the file, on the average. If instead, we use an m-ary search tree to represent the file, it would take an average of only logmn block accesses to retrieve a record. For n = 106, the binary search tree would require about 20 block accesses, while a 128-way search tree would take only 3 block accesses.

We cannot make m arbitrarily large, because the larger m is, the larger the block size must be. Moreover, it takes longer to read and process a larger block, so there is an optimum value for m to minimize the amount of time needed to search an external m-ary search tree. In practice a value close to the minimum is obtained for a broad range of m's. (See Exercise 11.18).

B-trees

A B-tree is a special kind of balanced m-ary tree that allows us to retrieve, insert, and delete records from an external file with a guaranteed worst-case performance. It is a generalization of the 2-3 tree discussed in Section 5.4. Formally, a B-tree of order m is an m-ary search tree with the following properties:

  1. The root is either a leaf or has at least two children.
  2. Each node, except for the root and the leaves, has between [m/2] and m children.
  3. Each path from the root to a leaf has the same length.

Note that every 2-3 tree is a B-tree of order 3. Figure 11.10 shows a B-tree of order 5, in which we assume that at most three records fit in a leaf block.

Fig. 11.10. B-tree of order 5.

We can view a B-tree as a hierarchical index in which each node occupies a block in external storage. The root of the B-tree is the first level index. Each non-leaf node in the B-tree is of the form

(p0, k1, p1, k2, p2, . . . ,kn, pn)

where pi is a pointer to the ith child of the node, 0 � i � n, and ki is a key, 1 � i � n. The keys within a node are in sorted order so k1 < k2 < � � � < kn. All keys in the subtree pointed to by po are less than k1. For 1 � i < n, all keys in the subtree pointed to by pi have values greater than or equal to ki and less than ki+1. All keys in the subtree pointed to by pn are greater than or equal to kn.

There are several ways to organize the leaves. Here we shall assume that the main file is stored only in the leaves. Each leaf is assumed to occupy one block.

Retrieval

To retrieve a record r with key value x, we trace the path from the root to the leaf that contains r, if it exists in the file. We trace this path by successively fetching interior nodes (p0, k1, p1, . . . ,kn, pn) from external storage into main memory and finding the position of x relative to the keys k1, k2, . . . ,kn. If ki � x < ki+1, we next fetch the node pointed to by pi and repeat the process. If x < k1, we use p0 to fetch the next node; if x � kn, we use pn. When this process brings us to a leaf, we search for the record with key value x. If the number of entries in a node is small, we can use linear search within the node; otherwise, it would pay to use binary search.

Insertion

Insertion into a B-tree is similar to insertion into a 2-3 tree. To insert a record r with key value x into a B-tree, we first apply the lookup procedure to locate the leaf L at which r should belong. If there is room for r in L, we insert r into L in the proper sorted order. In this case no modifications need to be made to the ancestors of L.

If there is no room for r in L, we ask the file system for a new block L� and move the last half of the records from L to L�, inserting r into its proper place in L or L�.† Let node P be the parent of node L. P is known, since the lookup procedure traced a path from the root to L through P. We now apply the insertion procedure recursively to place in P a key k� and a pointer l� to L�; k� and l� are inserted immediately after the key and pointer for L. The value of k� is the smallest key value in L�.

If P already has m pointers, insertion of k� and l� into P will cause P to be split and require an insertion of a key and pointer into the parent of P. The effects of this insertion can ripple up through the ancestors of node L back to the root, along the path that was traced by the original lookup procedure. It may even be necessary to split the root, in which case we create a new root with the two halves of the old root as its two children. This is the only situation in which a node may have fewer than m/2 children.

Deletion

To delete a record r with key value x, we first find the leaf L containing r. We then remove r from L, if it exists. If r is the first record in L, we then go to P, the parent of L, to set the key value in P's entry for L to be the new first key value of L. However, if L is the first child of P, the first key of L is not recorded in P, but rather will appear in one of the ancestors of P, specifically, the lowest ancestor A such that L is not the leftmost descendant of A. Therefore, we must propagate the change in the lowest key value of L backwards along the path from the root to L.

If L becomes empty after deletion, we give L back to the file system.† We now adjust the keys and pointers in P to reflect the removal of L. If the number of children of P is now less than m/2, we examine the node P� immediately to the left (or the right) of P at the same level in the tree. If P� has at least [m/2] + 1 children, we distribute the keys and pointers in P and P� evenly between P and P�, keeping the sorted order of course, so that both nodes will have at least [m/2] children. We then modify the key values for P and P� in the parent of P, and, if necessary, recursively ripple the effects of this change to as many ancestors of P as are affected.

If P� has exactly [m/2] children, we combine P and P� into a single node with 2[m/2] - 1 children (this is m children at most). We must then remove the key and pointer to P� from the parent for P�. This deletion can be done with a recursive application of the deletion procedure.

If the effects of the deletion ripple all the way back to the root, we may have to combine the only two children of the root. In this case the resulting combined node becomes the new root, and the old root can be returned to the file system. The height of the B-tree has now been reduced by one.

Example 11.5. Consider the B-tree of order 5 in Fig. 11.10. Inserting the record with key value 23 into this tree produces the B-tree in Fig. 11.11. To insert 23, we must split the block containing 22, 23, 24, and 26, since we assume that at most three records fit in one block. The two smaller stay in that block, and the larger two are placed in a new block. A pointer-key pair for the new node must be inserted into the parent, which then splits because it cannot hold six pointers. The root receives the pointer-key pair for the new node, but the root does not split because it has excess capacity.

Removing record 10 from the B-tree of Fig. 11.11 results in the B-tree of Fig. 11.12. Here, the block containing 10 is discarded. Its parent now has only two children, and the right sibling of the parent has the minimum number, three. Thus we combine the parent with its sibling, making one node with five children.

Fig. 11.11. B-tree after insertion.

Fig. 11.12. B-tree after deletion.

Time Analysis of B-tree Operations

Suppose we have a file with n records organized into a B-tree of order m. If each leaf contains b records on the average, then the tree has about [n/b] leaves. The longest possible paths in such a tree will occur if each interior node has the fewest children possible, that is, m/2 children. In this case there will be about 2[n/b]/m parents of leaves, 4[n/b]/m2 parents of parents of leaves, and so on.

If there are j nodes along the path from the root to a leaf, then 2j-1[n/b]/mj-1 � 1, or else there would be fewer than one node at the root's level. Therefore, [n/b] � (m/2)j-1, and j � 1 -- logm/2[n/b]. For example, if n = 106, b = 10, and m = 100, then j � 3.9. Note that b is not the maximum number of records we can put in a block, but an average or expected number. However, by redistributing records among neighboring blocks whenever one gets less than half full, we can ensure that b is at least half the maximum value. Also note that we have assumed in the above that each interior node has the minimum possible number of children. In practice, the average interior node will have more than the minimum, and the above analysis is therefore conservative.

For an insertion or deletion, j block accesses are needed to locate the appropriate leaf. The exact number of additional block accesses that are needed to accomplish the insertion or deletion, and to ripple its effects through the tree, is difficult to compute. Most of the time only one block, the leaf storing the record of interest, needs to be rewritten. Thus, 2 + logm/2[n/b] can be taken as the approximate number of block accesses for an insertion or deletion.

Comparison of Methods

We have discussed hashing, sparse indices, and B-trees as possible methods for organizing external files. It is interesting to compare, for each method, the number of block accesses involved in a file operation.

Hashing is frequently the fastest of the three methods, requiring two block accesses on average for each operation (excluding the block accesses required to search the bucket table), if the number of buckets is sufficiently large that the typical bucket uses only one block. With hashing, however, we cannot easily access the records in sorted order.

A sparse index on a file of n records allows the file operations to be done in about 2 + log(n/bb�) block accesses using binary search; here b is the number of records that fit on a block, and b� is the number of key-pointer pairs that fit on a block for the index file. B-trees allow file operations in about 2 + logm/2[n/b] block accesses, where m, the maximum degree of the interior nodes, is approximately b�. Both sparse indices and B-trees allow records to be accessed in sorted order.

All of these methods are remarkably good compared to the obvious sequential scan of a file. The timing differences among them, however, are small and difficult to determine analytically, especially considering that the relevant parameters such as the expected file length and the occupancy rates of blocks are hard to predict in advance.

It appears that the B-tree is becoming increasingly popular as a means of accessing files in database systems. Part of the reason lies in its ability to handle queries asking for records with keys in a certain range (which benefit from the fact that the records appear in sorted order in the main file). The sparse index also handles such queries efficiently, but is almost sure to be less efficient than the B-tree. Intuitively, the reason B-trees are superior to sparse indices is that we can view a B-tree as a sparse index on a sparse index on a sparse index, and so on. (Rarely, however, do we need more than three levels of indices.)

B-trees also perform relatively well when used as secondary indices, where "keys" do not really define a unique record. Even if the records with a given value for the designated fields of a secondary index extend over many blocks, we can read them all with a number of block accesses that is just equal to the number of blocks holding these records plus the number of their ancestors in the B-tree. In comparison, if these records plus another group of similar size happen to hash to the same bucket, then retrieval of either group from a hash table would require a number of block accesses about double the number of blocks on which either group would fit. There are possibly other reasons for favoring the B-tree, such as their performance when several processes are accessing the structure simultaneously, that are beyond the scope of this book.

Exercises

11.1 | Write a program concatenate that takes a sequence of file names as arguments and writes the contents of the files in turn onto the standard output, thereby concatenating the files.
11.2 | Write a program include that copies its input to its output except when it encounters a line of the form #include filename, in which case it is to replace this line with the contents of the named file. Note that included files may also contain #include statements.
11.3 | How does your program for Exercise 11.2 behave when a file includes itself?
11.4 | Write a program compare that will compare two files record-by-record to determine whether the two files are identical.
*11.5 | Rewrite the file comparison program of Exercise 11.4 using the LCS algorithm of Section 5.6 to find the longest common subsequence of records in both files.
11.6 | Write a program find that takes two arguments consisting of a pattern string and a file name, and prints all lines of the file containing the pattern string as a substring. For example, if the pattern string is "ufa" and the file is a word list, then find prints all words containing the trigram "ufa."
11.7 | Write a program that reads a file and writes on its standard output the records of the file in sorted order.
11.8 | What are the primitives Pascal provides for dealing with external files? How would you improve them?
*11.9 Suppose we use a three-file polyphase sort, where at the ith phase we create a file with ri runs of length li. At the nth phase we want one run on one of the files and none on the other two. Explain why each of the following must be true
  1. li = li-i + li- 2 for i � 1, where l0 and l-1 are taken to be the lengths of runs on the two initially occupied files.

  2. ri = ri-2 - ri- 1 (or equivalently, ri-2 = ri-1 + ri for i � 1), where r0 and r-1 are the number of runs on the two initial files.

  3. rn = rn-1 = 1, and therefore, rn, rn-1, . . . ,r1, forms a Fibonacci sequence.
*11.10 What additional condition must be added to those of Exercise 11.9 to make a polyphase sort possible
  1. with initial runs of length one (i.e., lo=l- 1=1)

  2. running for k phases, but with initial runs other than one allowed.
Hint. Consider a few examples, like ln = 50, ln-1 = 31, or ln = 50, ln-1 = 32.
**11.11 | Generalize Exercises 11.9 and 11.10 to polyphase sorts with more than three files.
**11.12 Show that:
  1. Any external sorting algorithm that uses only one tape as external storage must take W(n2) time to sort n records.

  2. O(n log n) time suffices if there are two tapes to use as external storage.
11.13 Suppose we have an external file of directed arcs x � y that form a directed acyclic graph. Assume that there is not enough space in internal memory to hold the entire set of vertices or edges at one time.
  1. Write an external topological sort program that prints out a linear ordering of the vertices such that if x � y is a directed arc, then vertex x appears before vertex y in the linear

  2. What is the time and space complexity of your program as a function of the number of block accesses?

  3. What does your program do if the directed graph is cyclic?

  4. What is the minimum number of block accesses needed to topologically sort an externally stored dag?
11.14 | Suppose we have a file of one million records, where each record takes 100 bytes. Blocks are 1000 bytes long, and a pointer to a block takes 4 bytes. Devise a hashed organization for this file. How many blocks are needed for the bucket table and the buckets?
11.15 | Devise a B-tree organization for the file of Exercise 11.14.
11.16 Write programs to implement the operations RETRIEVE, INSERT, DELETE, and MODIFY on
  1. hashed files,

  2. indexed files,

  3. B-tree files.
11.17 Write a program to find the kth largest element in
  1. a sparse-indexed file

  2. a B-tree file
11.18 Assume that it takes a + bm milliseconds to read a block containing a node of an m-ary search tree. Assume that it takes c + d log2m milliseconds to process each node in internal memory. If there are n nodes in the tree, we need to read about logmn nodes to locate a given record. Therefore, the total time taken to find a given record in the tree is

(logmn)(a + bm + c + d log2m) = (log2n)((a + c + bm)/log2m + d)

milliseconds. Make reasonable estimates for the values of a, b, c, and d and plot this quantity as a function of m. For what value of m is the minimum attained?
*11.19 | A B*-tree is a B-tree in which each interior node is at least 2/3 full (rather than just 1/2 full). Devise an insertion scheme for B*-trees that delays splitting interior nodes until two sibling nodes are full. The two full nodes can then be divided into three, each 2/3 full. What are the advantages and disadvantages of B*-trees compared with B-trees?
*11.20 | When the key of a record is a string of characters, we can save space by storing only a prefix of the key as the key separator in each interior node of the B-tree. For example, "cat" and "dog" could be separated by the prefix "d" or "do" of "dog." Devise a B-tree insertion algorithm that uses prefix key separators that at all times are as short as possible.
*11.21 | Suppose that the operations on a certain file are insertions and deletions fraction p of the time, and the remaining 1-p of the time are retrievals where exactly one field is specified. There are k fields in records, and a retrieval specifies the ith field with probability qi. Assume that a retrieval takes a milliseconds if there is no secondary index for the specified field, and b milliseconds if the field has a secondary index. Also assume that an insertion or deletion takes c + sd milliseconds, where s is the number of secondary indices. Determine, as a function of a, b, c, d, p, and the qi's, which secondary indices should be created for the file in order that the average time per operation be minimized.
*11.22 Suppose that keys are of a type that can be linearly ordered, such as real numbers, and that we know the probability distribution with which keys of given values will appear in the file. We could use this knowledge to outperform binary search when looking for a key in a sparse index. One scheme, called interpolation search, uses this statistical information to predict where in the range of index blocks Bi, . . . ,Bj to which the search has been limited, a key x is most likely to lie. Give
  1. an algorithm to take advantage of statistical knowledge in this way, and

  2. a proof that O(loglogn ) block accesses suffice, on the average, to find a key.
11.23 Suppose we have an external file of records, each consisting of an edge of a graph G and a cost associated with that edge.
  1. Write a program to construct a minimum-cost spanning tree for G, assuming that there is enough memory to store all the vertices of G in core but not all the edges.

  2. What is the time complexity of your program as a function of the number of vertices and edges?

Hint. One approach to this problem is to maintain a forest of currently connected components in core. Each edge is read and processed as follows: If the next edge has ends in two different components, add the edge and merge the components. If the edge creates a cycle in an existing component, add the edge and remove the highest cost edge from that cycle (which may be the current edge). This approach is similar to Kruskal's algorithm but does not require the edges to be sorted, an important consideration in this problem.
11.24 | Suppose we have a file containing a sequence of positive and negative numbers a1, a2, . . . ,an. Write an O(n) program to find a contiguous subsequence ai, ai+1, . . . ,aj that has the largest sum ai + ai+1 � � � + aj of any such subsequence.

Bibliographic Notes

For additional material on external sorting see Knuth [1973]. Further material on external data structures and their use in database systems can be found there and in Ullman [1982] and Wiederhold [1982]. Polyphase sorting in discussed by Shell [1971]. The six-buffer merging scheme in Section 11.2 is from Friend [1956] and the four-buffer scheme from Knuth [1973].

Secondary index selection, of which Exercise 11.21 is a simplification, is discussed by Lum and Ling [1970] and Schkolnick [1975]. B-trees originated with Bayer and McCreight [1972]. Comer [1979] surveys the many variations, and Gudes and Tsur [1980] discusses their performance in practice.

Information about Exercise 11.12, one- and two- tape sorting, can be found in Floyd and Smith [1973]. Exercise 11.22 on interpolation search is discussed in detail by Yao and Yao [1976] and Perl, Itai, and Avni [1978].

An elegant implementation of the approach suggested in Exercise 11.23 to the external minimum-cost spanning tree problem was devised by V. A. Vyssotsky around 1960 (unpublished). Exercise 11.24 is due to M. I. Shamos.

† It is tempting to assume that if (1) and (2) take the same time, then selection could never catch up with reading; if the whole block were not yet read, we would select from the first records of the block, those that had the lower keys, anyway. However, the nature of reading from disks is that a long period elapses before the block is found and anything at all is read. Thus our only safe assumption is that nothing of the block being read in a stage is available for selection during that stage.

† If these are not the first runs from each file, then this initialization can be done after the previous runs were read and the last 4b records from these runs are being merged.

† This strategy is the simplest of a number of responses that can be made to the situation where a block has to be split. Some other choices, providing higher average occupancy of blocks at the cost of extra work with each insertion, are mentioned in the exercises.

† We can use a variety of strategies to prevent leaf blocks from ever becoming completely empty. In particular, we describe below a scheme for preventing interior nodes from getting less than half full, and this technique can be applied to the leaves as well, with a value of m equal to the largest number of records that will fit in one block.

Table of Contents Go to Chapter 12


Data Structures and Algorithms: CHAPTER 12: Memory Management

 Next Chapter Return to Table of Contents Previous Chapter

Memory Management

This chapter discusses the basic strategies whereby memory space can be reused, or shared by different objects that grow and shrink in arbitrary ways. For example, we shall discuss methods that maintain linked lists of available space, and "garbage collection" techniques, where we figure out what is available only when it seems we have run out of available space.

12.1 Issues in Memory Management

There are numerous situations in computer system operation when a limited memory resource is managed, that is, shared among several "competitors." A programmer who does not engage in the implementation of systems programs (compilers, operating systems, and so on) may be unaware of such activities, since they frequently are carried out "behind the scenes." As a case in point, Pascal programmers are aware that the procedure new(p) will make pointer p point to a new object of the correct type. But where does space for that object come from? The procedure new has access to a large region of memory, called the "heap," that the program variables do not use. From that region, an unused block of consecutive bytes sufficient to hold an object of the type that p points to is selected, and p is made to hold the address of the first byte of that block. But how does the procedure new know which bytes of the memory are "unused'"? Section 12.4 suggests the answer.

Even more mysterious is what happens if the value of p is changed, either by an assignment or by another call to new(p). The block of memory p pointed to may now be inaccessible, in the sense that there is no way to get to it through the program's data structures, and we could reuse its space. On the other hand, before p was changed, the value of p may have been copied into some other variable. In that case, the memory block is still part of the program's data structures. How do we know whether a block in the memory region used by procedure new is no longer needed by the program?

Pascal's sort of memory management is only one of several different types. For example, in some situations, like Pascal, objects of different sizes share the same memory space. In others, all objects sharing the space are of the same size. This distinction regarding object sizes is one way we can classify the kinds of memory management problems we face. Some more examples follow.

  1. In the programming language Lisp, memory space is divided into cells, which are essentially records consisting of two fields; each field can hold either an atom (object of elementary type, e.g., an integer) or a pointer to a cell. Atoms and pointers are the same size, so all cells require the same number of bytes. All known data structures can be made out of these cells. For example, linked lists of atoms can use the first fields of cells to hold atoms and second fields to hold pointers to the next cells on the list. Binary trees can be represented by using the first field of each cell to point to the left child and the second field to point to the right child. As a Lisp program runs, the memory space used to hold a cell may find itself part of many different structures at different times, either because one cell is moved among structures, or because the cell becomes detached from all structures and its space is reused.
  2. A file system generally divides secondary storage devices, like disks, into fixed length blocks. For example, UNIX typically uses blocks of 512 bytes. Files are stored in a sequence of (not necessarily consecutive) blocks. As files are created and destroyed, blocks of secondary storage are made available to be reused.
  3. A typical multiprogramming operating system allows several programs to share main memory at one time. Each program has a required amount of memory, which is known to the operating system, that requirement being part of the request for service issued when it is desired to run the program. While in examples (1) and (2), the objects sharing memory (cells and blocks, respectively) were each of the same size, different programs require different amounts of memory. Thus, when a program using say 100K bytes terminates, it may be replaced by two programs using 50K each, or one using 20K and another 70K (with 10K left unused). Alternatively, the 100K bytes freed by the program terminating may be combined with an adjacent 50K that are unused and a program needing up to 150K may be run. Another possibility is that no new program can fit in the space made available, and that 100K bytes is left free temporarily.
  4. There are a large number of programming languages, like Snobol, APL, or SETL, that allocate space to objects of arbitrary size. These objects, which are values assigned to variables, are allocated a block of space from a larger block of memory, which is often called the heap. When a variable changes value, the new value is allocated space in the heap, and a pointer for the variable is set to point to the new value. Possibly the old value of the variable is now unused, and its space can be reused. However, languages like Snobol or SETL implement assignments like A = B by making the pointer for A point to the same object that B's pointer points to. If either A or B is reassigned, the previous object is not freed, and its space cannot be reclaimed.

Example 12.1. In Fig. 12.1(a) we see the heap that might be used by a Snobol program with three variables, A, B, and C. The value of any variable in Snobol is a character string, and in this case, the value of both A and B is 'OH HAPPY DAY' and the value of C is 'PASS THE SALT'.

We have chosen to represent character strings by pointers to blocks of memory in the heap. These blocks have their first 2 bytes (the number 2 is a typical value that could be changed) devoted to an integer giving the length of the string. For example, 'OH HAPPY DAY' has length 12, counting the blanks between words, so the value of A (and of B) occupies 14 bytes.

If the value of B were changed to 'OH HAPPY DAZE', we would find an empty block in the heap 15 bytes long to store the new value of B, including the 2 bytes for the length. The pointer for B is made to point to the new value, as shown in Fig. 12.1(b). The block holding integer 12 and 'OH HAPPY DAY' is still useful, since A points to it. If the value of A now changes, that block would become useless and could be reused. How one tells conveniently that there are no pointers to such a block is a major subject of this chapter.

Fig. 12.1. String variables in a heap.

In the four examples above, we can see differences along at least two orthogonal "dimensions." The first issue is whether objects sharing storage are or are not of equal length. In the first two examples, Lisp programs and file storage, the objects, which are Lisp cells in one case and blocks holding parts of files in the other, are of the same size. This fact allows certain simplifications of the memory management problem. For example, in Lisp implementation, a region of memory is divided into spaces, each of which can hold exactly one cell. The management problem is to find empty spaces to hold newly-created cells, and it is never necessary to store a cell in such a position that it overlaps two spaces. Similarly, in the second example, a disk is divided into equal sized blocks, and each block is assigned to hold part of one file; we never use a block to store parts of two or more files, even if a file ends in the middle of a block.

In contrast, the third and fourth examples, covering memory allocation by a multiprogramming system and heap management for those languages that deal with variables whose values are "big" objects, speak of allocating space in blocks of different sizes. This requirement presents certain problems that do not appear in the fixed-length case. For example, we fear fragmentation, a situation in which there is much unused space, but it is in such small pieces that space for one large object cannot be found. We shall say more about heap management in Sections 12.4 and 12.5.

The second major issue is whether garbage collection, a charming term for the recovery of unused space, is done explicitly or implicitly, that is, by program command or only in response to a request for space that cannot be satisfied otherwise. In the case of file management, when a file is deleted, the blocks used to hold it are known to the file system. For example, the file system could record the address of one or more "master blocks" for each file in existence; the master blocks list the addresses of all the blocks used for the file. Thus, when a file is deleted, the file system can explicitly make available for reuse all the blocks used for that file.

In contrast, Lisp cells, when they become detached from the data structures of the program, continue to occupy their memory space. Because of the possibility of multiple pointers to a cell, we cannot tell when a cell is completely detached; therefore we cannot explicitly collect cells as we do blocks of a deleted file. Eventually, all memory spaces will become allocated to useful or useless cells, and the next request for space for another cell implicitly will trigger a "garbage collection." At that time, the Lisp interpreter marks all the useful cells, by an algorithm such as the one we shall discuss in Section 12.3, and then links all the blocks holding useless cells into an available space list, so they can be reused.

Figure 12.2 illustrates the four kinds of memory management and gives an example of each. We have already discussed the fixed block size examples in Fig. 12.2. The management of main memory by a multiprogramming system is an example of explicit reclamation of variable length blocks. That is, when a program terminates, the operating system, knowing what area of memory was given to the program, and knowing no other program could be using that space, makes the space available immediately to another program.

The management of a heap in Snobol or many other languages is an example of variable length blocks with garbage collection. As for Lisp, a typical Snobol interpreter does not try to reclaim blocks of memory until it runs out of space. At that time the interpreter performs a garbage collection as the Lisp interpreter does, but with the additional possibility that strings will be moved around the heap to reduce fragmentation, and that adjacent free blocks will be combined to make larger blocks. Notice that the latter two steps are pointless in the Lisp environment.

Fig. 12.2. Examples of the four memory management strategies.

12.2 Managing Equal-Sized Blocks

Let us imagine we have a program that manipulates cells each consisting of a pair of fields; each field can be a pointer to a cell or can hold an "atom." The situation, of course, is like that of a program written in Lisp, but the program may be written in almost any programming language, even Pascal, if we define cells to be of a variant record type. Empty cells available for incorporation into a data structure are kept on an available space list, and each program variable is represented by a pointer to a cell. The cell pointed to may be one cell of a large data structure.

Example 12.2. In Fig. 12.3 we see a possible structure. A, B, and C are variables; lower case letters represent atoms. Notice some interesting phenomena. The cell holding atom a is pointed to by the variable A and by another cell. The cell holding atom c is pointed to by two different cells. The cells holding g and h are unusual in that although each points to the other, they cannot be reached from any of the variables A, B, or C, nor are they on the available space list.

Let us assume that as the program runs, new cells may on occasion be seized from the available space list. For example, we might wish to have the null pointer in the cell with atom c in Fig. 12.3 replaced by a pointer to a new cell that holds atom i and a null pointer. This cell will be removed from the top of the available space list. It is also possible that from time to time, pointers will change in such a way that cells become detached from the program variables, as the cells holding g and h in Fig. 12.3 have been. For example, the cell holding c may, at one time, have pointed to the cell holding g. As another example, the value of variable B may at some time change, which would, if nothing else has changed, detach the cell now pointed to by B in Fig. 12.3 and also detach the cell holding d and e (but not the cell holding c, since it would still be reached from A). We call cells not reachable from any variable and not on the available space list inaccessible.

Fig. 12.3. A network of cells.

When cells are detached, and therefore are no longer needed by the program, it would be nice if they found their way back onto the available space list, so they could be reused. If we don't reclaim such cells, we shall eventually reach the unacceptable situation where the program is not using all the cells, yet it wants a cell from available space, and the available space list is empty. It is then that a time-consuming garbage collection must be performed. This garbage collection step is "implicit," in the sense that it was not explicitly called for by the request for space.

Reference Counts

One seemingly attractive approach to detecting inaccessible cells is to include in each cell a reference count, that is, an integer-valued field whose value equals the number of pointers to the cell. It is easy to maintain reference counts. When making some pointer point to a cell, add one to the reference count for that cell. When a non-null pointer is reassigned, first decrease by one the reference count for the cell pointed to. If a reference count reaches zero, the cell is inaccessible, and it can be returned to the available list.

Unfortunately, reference counts don't always work. The cells with g and h in Fig. 12.3 are inaccessible cells linked in a cycle. Their reference counts are each 1, so we would not return them to the available list. One can attempt to detect cycles of inaccessible cells in a variety of ways, but it is probably not worth doing so. Reference counts are useful for structures that do not have pointer cycles. One example of a structure with no possibility of cycles is a collection of variables pointing to blocks holding data, as in Fig. 12. I. There, we can do explicit garbage collection simply by collecting a block when its reference count reaches zero. However, when data structures allow pointer cycles, the reference count strategy is usually inferior, both in terms of the space needed in cells and the time taken dealing with the issue of inaccessible cells, to another approach which we shall discuss in the next section.

12.3 Garbage Collection Algorithms for Equal-Sized Blocks

Let us now give an algorithm for finding which of a collection of cells of the types suggested in Fig. 12.3 are accessible from the program variables. We shall define the setting for the problem precisely by defining a cell type in Pascal that is a variant record type; the four variants, which we call PP, PA, AP, and AA, are determined by which of the two data fields are pointers and which are atoms. For example, PA means the left field is a pointer and the right field an atom. An additional boolean field in cells, called mark, indicates whether the cell has been found accessible. That is, by setting mark to true when garbage collecting, we "mark" the cell, indicating it is accessible. The important type definitions are shown in Fig. 12.4.

   type
       atomtype = { some appropriate type;
            preferably of the same size as pointers }
       patterns = (PP, PA, AP, AA);
       celltype = record
           mark: boolean;
           case pattern: patterns of
               PP: (left: � celltype; right:
� celltype);
               PA: (left: � celltype; right:
atomtype);
               AP: (left: atomtype; right: � celltype);
               AA: (left: atomtype; right: atomtype);
   end;

Fig. 12.4. Definition of the type for cells.

We assume there is an array of cells, taking up most of the memory, and some collection of variables, that are pointers to cells. For simplicity, we assume there is only one variable, called source, pointing to a cell, but the extension to many variables is not hard. † That is, we declare

    var
        source: � celltype;
        memory: array [1..memorysize ] of celltype;

To mark the cells accessible from source, we first "unmark" all cells, accessible or not, by running down the array memory and setting the mark field to false. Then we perform a depth-first search of the graph emanating from source, marking all cells visited. The cells visited are exactly those that are accessible. We then traverse the array memory and add to the available space list all unmarked cells. Figure 12.5 shows a procedure dfs to perform the depth-first search; dfs is called by the procedure collect that unmarks all cells, and then marks accessible cells by calling dfs. We do not show the code linking the available space list because of the peculiarities of Pascal. For example, while we could link available cells using either all left or all right cells, since pointers and atoms are assumed the same size, we are not permitted to replace atoms by pointers in cells of variant type AA.

(1)   procedure dfs ( currentcell: �
celltype );
           { If current cell was marked, do nothing. Otherwise, mark
           it and call dfs on any cells pointed to by the current cell }
         begin
(2)          with currentcell� do
(3)              if mark = false then begin
(4)                  mark := true;
(5)                  if (pattern = PP ) or (pattern = PA ) then
(6)                       if left <> nil then
(7)                           dfs (left);
(8)                  if (pattern = PP ) or (pattern = AP ) then
(9)                       if right <> nil then
(10)                           dfs (right)
             end
         end; { dfs }

(11)  procedure collect;
          var
              i: integer;
          begin
(12)          for i := 1 to memorysize do { "unmark"
all cells }
(13)               memory[i].mark := false;
(14)          dfs(source); { mark accessible cells }
(15)          { the code for the collection goes here }
          end; { collect }

Fig. 12.5. Algorithm for marking accessible cells.

Collection in Place

The algorithm of Fig. 12.5 has a subtle flaw. In a computing environment where memory is limited, we may not have the space available to store the stack required for recursive calls to dfs. As discussed in Section 2.6, each time dfs calls itself, Pascal (or any other language permitting recursion) creates on "activation record" for that particular call to dfs. In general, an activation record contains space for parameters and variables local to the procedure, of which each call needs its own copy. Also necessary in each activation record is a "return address," the place to which control must go when this recursive call to the procedure terminates.

In the particular case of dfs, we only need space for the parameter and the return address (i.e., was it called by line (14) in collect or by line (7) or line (10) of another invocation of dfs). However, this space requirement is enough that, should all of memory be linked in a single chain extending from source (and therefore the number of active calls to dfs would at some time equal the length of this chain), considerably more space would be required for the stack of activation records than was allotted for memory. Should such space not be available, it would be impossible to carry out the marking.

Fortunately, there is a clever algorithm, known as the Deutsch-Schorr-Waite Algorithm, for doing the marking "in place." The reader should convince himself that the sequence of cells upon which a call to dfs has been made but not yet terminated indeed forms a path from source to the cell upon which the current call to dfs was made. Thus we can use a nonrecursive version of dfs, and instead of a stack of activation records to record the path of cells from source to the cell currently being examined, we can use the pointer fields along that path to hold the path itself. That is, each cell on the path, except the last, holds in either the left or right field, a pointer to its predecessor, the cell immediately closer to source. We shall describe the Deutsch-Schorr-Waite algorithm with an extra, one-bit field, called back. Field back is of an enumerated type (L, R), and tells whether field left or field right points to the predecessor. Later, we shall discuss how the information in back can be held in the pattern field, so no extra space in cells is required.

The new procedure for nonrecursive depth-first search, which we call nrdfs, uses a pointer current to point to the current cell, and a pointer previous to the predecessor of the current cell. The variable source points to a cell source1, which has a pointer in its right field only.† Before marking, source1 is initialized to have back = R, and its right field pointing to itself. The cell ordinarily pointed to by source1, is pointed to by current, and source1 is pointed to by previous. We halt the marking operation when current = previous, which can only occur when they both point to source1, the entire structure having been searched.

Example 12.3. Figure 12.6(a) shows a possible structure emanating from source. If we depth-first search this structure, we visit (1), (2), (3) and (4), in that order. Figure 12.6(b) shows the pointer modifications made when cell (4) is current. The value of the back field is shown, although fields mark and pattern are not. The current path is from (4) to (3) to (2) to (1) back to sourcel; it is represented by dashed pointers. For example, cell (1) has back = L, since the left field of (1), which in Fig. 12.6(a) held a pointer to (2), is being used in Fig. 12.6(b) to hold part of the path. Now the left field in (1) points backwards, rather than forwards along the path; however, we shall restore that pointer when the depth-first search finally retreats from cell (2) back to (1). Similarly, in cell (2), back = R, and the right field of (2) points backwards along the path to (1), rather than ahead to (3), as it did in Fig. 12.6(a).

Fig. 12.6. Using pointers to represent the path back to source.

There are three basic steps to performing the depth-first search. They are:

  1. Advance. If we determine that the current cell has one or more nonnull pointers, advance to the first of these, that is, follow the pointer in left, or if there is none, the pointer in right. By "advance," we mean to make the cell pointed to become the current cell and the current cell become previous. To help find the way back, the pointer just followed is made to point to the old previous cell. These changes are shown in Fig. 12.7(a), on the assumption that the left pointer is followed. In that figure, the old pointers are solid lines and the new ones dashed.
  2. Switch. If we determine that the cells emanating from the current cell have been searched (for example, the current cell may have only atoms, may be marked, or we may just have "retreated" to the current cell from the cell pointed to by the right field of the current cell), we consult the field back of the previous cell. If that value is L, and the right field of the previous cell holds a nonnull pointer to some cell C, we make C become the current cell, while the identity of the previous cell is not changed. However, the value of back in the previous cell is set to R, and the left pointer in that cell receives its correct value; that is, it is made to point to the old current cell. To maintain the path back to source from the previous cell, the pointer to C in field right is made to point where left used to point. Figure 12.7(b) shows these changes.
  3. Retreat. If we determine, as in (2), that the cells emanating from the current cell have been searched, but field back of the previous cell is R, or it is L but field right holds an atom or null pointer, then we have now searched all cells emanating from the previous cell. We retreat, by making the previous cell be the current cell and the next cell along the path from the previous cell to source be the new previous cell. These changes are shown in Fig. 12.7(c), on the assumption that back = R in the previous cell.

One fortuitous coincidence is that each of the steps of Fig. 12.7 can be viewed as the simultaneous rotation of three pointers. For example, in Fig. 12.7(a), we simultaneously replace (previous, current, current�.left) by (current, current�.left, previous), respectively. The simultaneity must be emphasized; the location of current�.left does not change when we assign a new value to current. To perform these pointer modifications, a procedure rotate, shown in Fig. 12.8, is naturally useful. Note especially that the passing of parameters by reference assures that the locations of the pointers are established before any values change.

Now we turn to the design of the nonrecursive marking procedure nrdfs. This procedure is one of these odd processes that is most easily understood when written with labels and gotos. Particularly, there are two "states" of the procedure, "advancing," represented by label 1, and "retreating," represented by label 2. We enter the first state initially, and also whenever we have moved to a new cell, either by an advance step or a switch step. In this state, we attempt another advance step, and only retreat or switch if we are blocked. We can be blocked for two reasons: (l) The cell just reached is already marked, or (2) there are no nonnull pointers in the cell. When blocked, we change to the second, or "retreating" state.

Fig. 12.7. Three basic steps.

    procedure rotate ( var p1, p2, p3: � celltype );
        var
        temp: � celltype;
        begin
        temp := p1;
        p1 := p2;
        p2 := p3;
        p3 := temp
        end; { rotate }

Fig. 12.8. Pointer modification procedure.

The second state is entered whenever we retreat, or when we cannot stay in the advancing state because we are blocked. In the retreating state we check whether we have retreated back to the dummy cell source1. As discusssed before, we shall recognize this situation because previous = current, in which case we go to state 3. Otherwise, we decide whether to retreat and stay in the retreating state or switch and enter the advancing state. The code for nrdfs is shown in Fig. 12.9. It makes use of functions blockleft, blockright, and block, which test if the left or right fields of a cell, or both, have an atom or null pointer. block also checks for a marked cell.

   function blockleft ( cell: celltype ): boolean;
       { test if left field is atom or null pointer }
       begin
            with cell do
               if (pattern = PP) or (pattern = PA)
then
                  if left <> nil then return (false);
            return (true)
       end; { blockleft }
   function blockright ( cell : celltype ): boolean;
       { test if right field is atom or null pointer }
       begin
            with cell do
               if (pattern = PP) or (pattern = AP)
then
                  if right <> nil then
                       return (false);
            return (true)
       end; { blockright }

   function block ( cell: celltype ): boolean;
       { test if cell is marked or contains no nonnull pointers }
       begin
           if (cell.mark = true) or blockleft (cell) and
blockright (cell) then
               return (true)
           else
               return (false)
       end; { block }
   procedure nrdfs; { marks cells accessible from source }
       var
           current, previous: � celltype;
       begin { initialize }
             current := source 1�.right; { cell
pointed to by source1 }
             previous := source 1; { previous points to source1 }
             source 1�.back := R;
             source 1�.right := source 1; {
source1 points to itself }
             source 1�.mark := true;

   state 1: { try to advance }
       if block (current�) then begin
{ prepare to retreat }
          current�.mark := true;
          goto state2
       end
       else begin { mark and advance }
          current�.mark := true;
          if blockleft (current�) then
begin { follow right pointer }
             current�.back := R;
             rotate(previous, current, current�.right); { implements
                 changes of Fig. 12.7(a), but following right pointer }
             goto state1
          end
          else begin { follow left pointer }
             current�.back := L;
             rotate (previous, current, current�.left);
                  { implements changes of Fig. 12.7(a) }
             goto state1
          end
       end;

   state2: { finish, retreat or switch }
       if previous = current then { finish }
          goto state3
       else if (previous�.back = L )
and
             not blockright (previous�) then
begin { switch }
                  previous�.back := R;
                  rotate (previous�.left,
current, previous�.right);
                       { implements changes of Fig. 12.7(b) }
                  goto state1
             end
       else if previous�.back = R
then { retreat }
          rotate (previous, previous�.right, current)
             { implements changes of Fig. 12.7(c) }
       else { previous�.back = L }
          rotate (previous, previous�.left,
current);
             { implements changes of Fig. 12.7(c), but with
                  left field of previous cell involved in the path }
       goto state2
       end;

   state3: { put code to link unmarked cells on available list here }
   end; { nrdfs }

Fig. 12.9. Nonrecursive marking algorithm.

Deutsch-Schorr-Waite Algorithm Without an Extra Bit for the Field Back

It is possible, although unlikely, that the extra bit used in cells for the field back might cause cells to require an extra byte, or even an extra word. In such a case, it is comforting to know that we do not really need the extra bit, at least not if we are programming in a language that, unlike Pascal, allows the bits of the field pattern to be used for purposes other than those for which they were declared: designators of the variant record format. The "trick" is to observe that if we are using the field back at all, because its cell is on the path back to source1, then the possible values of field pattern are restricted. For example, if back = L, then we know the pattern must be PP or PA, since evidently the field left holds a pointer. A similar observation can be made when back = R. Thus, if we have two bits available to represent both pattern and (when needed) back, we can encode the necessary information as in Fig. 12.10, for example.

The reader should observe that in the program of Fig. 12.9, we always know whether back is in use, and thus can tell which of the interpretations in Fig. 12.10 is applicable. Simply, when current points to a record, the field back in that record is not in use; when previous points to it, it is. Of course, as these pointers move, we must adjust the codes. For example, if current points to a cell with bits 10, which we interpret according to Fig. 12.10 as pattern = AP, and we decide to advance, so previous will now point to this cell, we make back=R, as only the right field holds a pointer, and the appropriate bits are 11. Note that if the pattern were AA, which has no representation in the middle column of Fig. 12.10, we could not possibly want previous to point to the cell, as there are no pointers to follow in an advancing move.

Fig. 12.10. Interpreting two bits as pattern and back.

12.4 Storage Allocation for Objects with Mixed Sizes

Let us now consider the management of a heap, as typified by Fig. 12.1, where there is a collection of pointers to allocated blocks. The blocks hold data of some type. In Fig. 12.1, for example, the data are character strings. While the type of data stored in the heap need not be character strings, we assume the data contain no pointers to locations within the heap.

The problem of heap management has aspects that make it both easier and harder than the maintenance of list structures of equal-sized cells as discussed in the previous section. The principal factor making the problem easier is that marking used blocks is not a recursive process; one has only to follow the external pointers to the heap and mark the blocks pointed to. There is no need for a depth-first search of a linked structure or for anything like the Deutsch-Schorr-Waite algorithm.

On the other hand, managing the available space list is not as simple as in Section 12.3. We might imagine that the empty regions (there are three empty regions in Fig. 12.1(a), for example) are linked together as suggested in Fig. 12.11. There we see a heap of 3000 words divided into five blocks. Two blocks of 200 and 600 words, hold the values of X and Y. The remaining three blocks are empty, and are linked in a chain emanating from avail, the header for available space.

Fig. 12.11. A heap with available space list.

In order that empty blocks may be found when they are needed to hold new data, and blocks holding useless data may be made available, we shall throughout this section make the following assumptions about blocks.

  1. Each block is sufficiently large to hold
    1. A count indicating the size (in bytes or words, as appropriate for the computer system) of the block,
    2. A pointer (to link the block in available space), plus
    3. A bit to indicate whether the block is empty or not. This bit is referred to as the full/empty or used/unused bit.
  2. An empty block has, from the left (low address end), a count indicating its length, a full/empty bit with a value of 0, indicating emptiness of the block, a pointer to the next available block, and empty space.
  3. A block holding data has, from the left, a count, a full/empty bit with value 1 indicating the block is in use, and the data itself.†

One interesting consequence of the above assumptions is that blocks must be capable of storing data sometimes (when in use) and pointers at other times (when unused) in precisely the same place. It is thus impossible or very inconvenient to write programs that manipulate blocks of the kind we propose in Pascal or any other strongly typed language. Thus, this section must of necessity be discursive; only pseudo-Pascal programs can be written, never real Pascal programs. However, there is no problem writing programs to do the things we describe in assembly languages or in most systems programming languages such as C.

Fragmentation and Compaction of Empty Blocks

To see one of the special problems that heap management presents, let us suppose that the variable Y of Fig. 12.11 changes, so the block representing Y must be returned to available space. We can most easily insert the new block at the beginning of the available list, as suggested by Fig. 12.12. In that figure, we see an instance of fragmentation, the tendency for large empty areas to be represented on the available space list by "fragments," that is, several small blocks making up the whole. In the case in point, the last 2300 bytes of the heap in Fig. 12.12 are empty, yet the space is divided into three blocks of 1000, 600, and 700 bytes, and these blocks are not even in consecutive order on the available list. Without some form of garbage collection, it would be impossible to satisfy a request for, say, a block of 2000 bytes.

Fig. 12.12. After returning the block of Y.

Evidently, when returning a block to available space it would be desirable to look at the blocks to the immediate left and right of the block being made available. Finding the block to the right is easy. If the block being returned begins at position p and has count c, the block to the right begins at position p+c. If we know p (for example, the pointer Y in Fig. 12.11 holds the value p for the block made available in Fig. 12.12), we have only to read the bytes starting at position p, as many as are used for holding c, to obtain the value c. From byte p+c, we skip over the count field to find the bit that tells whether or not the block is empty. If empty, the blocks beginning at p and p + c can be combined.

Example 12.4. Let us assume the heap of Fig. 12.11 begins at position 0. Then the block for Y being returned begins in byte 1700, so p=1700 and c=600. The block beginning at p+c=2300 is also empty so we could combine them into one block beginning at 1700 with a count of 1300, the sum of the counts in the two blocks.

It is, however, not so easy to fix the available list after combining blocks. We can create the combined block by simply adding the count of the second block to c. However, the second block will still be linked in the available list and must be removed. To do so requires finding the pointer to that block from its predecessor on the available list. Several strategies present themselves; none is strongly recommended over the others.

  1. Run down the available list until we find a pointer with value p+c. This pointer must be in the predecessor block of the block we have combined with its neighbor. Replace the pointer found by the pointer in the block at p+c. This effectively eliminates the block beginning at position p+c from the available list. Its space is still available, of course; it is part of the block beginning at p. On the average, we shall have to scan half the available list, so the time taken is proportional to that length.
  2. Use a doubly-linked available space list. Then the predecessor block can be found quickly and the block at p+c eliminated from the list. This approach takes constant time, independent of the length of the available list, but it requires extra space for another pointer in each empty block, thus increasing the minimum size for a block to that required to hold a count, a full/empty bit, and two pointers.
  3. Keep the available space list sorted by position. Then we know the block at position p is the predecessor on the available list of the block at p+c, and the pointer manipulation necessary to eliminate the second block can be done in constant time. However, insertion of a new available block requires a scan of half the available list on the average, and is thus no more efficient than method (1).

Of the three methods, all but the second require time proportional to the length of the available list to return a block to available space and combine it with its right neighbor if that neighbor is empty. This time may or may not be prohibitive, depending on how long the list gets and what fraction of the total program time is spent manipulating the heap. The second method - doubly linking the available list - has only the penalty of an increased minimum size for blocks. Unfortunately, when we consider how to combine a returned block with its neighbor to the left, we see that double linking, like the other methods, is no help in finding left neighbors in less than the time it takes to scan the available list.

To find the block immediately to the left of the block in question is not so easy. The position p of a block and its count c, while they determine the position of the block to the right, give no clue as to where the block to the left begins. We need to find an empty block that begins in some position p1, and has a count c1, such that p1+c1=p. It ap>

  1. Scan the available list for a block at position p1 and count c1 where p1+c1=p. This operation takes time proportional to the length of the available list.
  2. Keep a pointer in each block (used or unused) indicating the position of the block to the left. This approach allows us to find the block to the left in constant time; we can check whether it is empty and if so merge it with the block in question. We can find the block at position p+c and make it point to the beginning of the new block, so these left-going pointers can be maintained.†
  3. Keep the available list sorted by position. Then the empty block to the left is found when we insert the newly emptied block onto the list, and we have only to check, using the position and count of the previous empty block, that no nonempty blocks intervene.

As with the merger of a newly empty block with the block to its right, the first and third approaches to finding and merging with the block on the left require time proportional to the length of the available list. Method (2) again requires constant time, but it has a disadvantage beyond the problems involved with doubly linking the available list (which we suggested in connection with finding right neighbor blocks). While doubly linking the empty blocks raises the minimum block size, the approach cannot be said to waste space, since it is only blocks not used for storing data that get linked. However, pointing to left neighbors requires a pointer in used blocks as well as unused ones, and can justifiably be accused of wasting space. If the average block size is hundreds of bytes, the extra space for a pointer may be negligible. On the other hand, the extra space may be prohibitive if the typical block is only l0 bytes long.

To summarize the implications of our explorations regarding the question of how we might merge newly empty blocks with empty neighbors, we see three approaches to handling fragmentation.

  1. Use one of several approaches, such as keeping the available list sorted, that requires time proportional to the length of the available list each time a block becomes unused, but enables us to find and merge empty neighbors.
  2. Use a doubly linked available space list each time a block becomes unused, and also use left-neighbor pointers in all blocks, whether available or not, to merge empty neighbors in constant time.
  3. Do nothing explicit to merge empty neighbors. When we cannot find a block large enough to hold a new data item, scan the blocks from left to right, merging empty neighbors and then creating a new available list. A sketched program to do this is shown in Fig. 12.13.

(1)  procedure merge;
     var
(2)   p, q: pointers to blocks;
      { p indicates left end of empty block being accumulated.
        q indicates a block to the right of p that we
        shall incorporate into block p if empty }
     begin
(3)   p:= leftmost block of heap;
(4)   make available list empty;
(5)   while p < right end of heap do
(6)     if p points to a full block with count c then
(7)         p := p + c { skip over full blocks }
(8)     else begin { p points to the beginning
            of a sequence of empty blocks; merge them }
(9)         q := p + c; { initialize q to the next block }
(10)        while q points to an empty block with some
                count, say d, and q < right end of heap do begin
(11)             add d to count of the block pointed to by p;
(12)             q := q + d
                end;
(13)        insert block pointed to by p onto the available list;
(14)        p := q
          end
    end; { merge }

Fig. 12.13. Merging adjacent empty blocks.

Example 12.5. As an example, consider the program of Fig. 12.13 applied to the heap of Fig. 12.12. Assume the leftmost byte of the heap is 0, so initially p=0. As c=500 for the first block, q is initialized to p + c=500. As the block beginning at 500 is full, the loop of lines (10)-(12) is not executed and the block consisting of bytes 0-499 is attached to the available list, by making avail point to byte 0 and putting a nil pointer in the designated place in that block (right after the count and full/empty bit). Then p is given the value 500 at line (14), and incremented to 700 at line (7). Pointer q is given value 1700 at line (9), then 2300 and 3000 at line (12), while at the same time, 600 and 700 are added to count 1000 in the block beginning at 700. As q exceeds the rightmost byte, 2999, the block beginning at 700, which now has count 2300, is inserted onto the available list. Then at line (14), p is set to 3000, and the outer loop ends at line (5).

As the total number of blocks and the number of available blocks are likely not to be too dissimilar, and the frequency with which no sufficiently large empty block can be found is likely to be low, we believe that method (3), doing the merger of adjacent empty blocks only when we run out of adequate space, is superior to (1) in any realistic situation. Method (2) is a possible competitor, but if we consider the extra space requirement and the fact that extra time is needed each time a block is inserted or deleted from the available list, we believe that (2) is preferable to (3) in extremely rare circumstances, and can probably be forgotten.

Selecting Available Blocks

We have discussed in detail what should happen when a block is no longer needed and can be returned to available space. There is also the inverse process of providing blocks to hold new data items. Evidently we must select some available block and use some or all of it to hold the new data. There are two issues to address. First, which empty block do we select? Second, if we must use only part of the selected block, which part do we use?

The second issue can be dispensed with easily. If we are to use a block with count c, and we need d<c bytes from that block, we choose the last d bytes. In this way, we need only to replace count c by c - d, and the remaining empty block can stay as it is in the available list.†

Example 12.6. Suppose we need 400 bytes for variable W in the situation represented by Fig. 12.12. We might choose to take the 400 bytes out of the 600 in the first block on the available list. The situation would then be as shown in Fig. 12.14.

Choosing a block in which to place the new data is not so easy, since there are conflicting goals for such strategies. We desire, on one hand, to be able to quickly pick an empty block in which to hold our data and, on the other hand, to make a selection of an empty block that will minimize the fragmentation. Two strategies that represent extremes in the spectrum are known as "first-fit" and "best-fit." They are described below.

  1. First-Fit. To follow the first-fit strategy, when we need a block of size d, scan the available list from the beginning until we come to a block of size

    Fig. 12.14. Memory configuration.

    c � d. Utilize the last d words of this block, as we have described above.
  2. Best-fit. To follow the best-fit strategy, when we need a block of size d, examine the entire available list to find that block of size at least d, whose size is as little greater than d as is possible. Seize the last d words of this block.

Some observations about these strategies can be made. Best-fit is considerably slower than first-fit, since with the latter we can expect to find a fitting block fairly quickly on the average, while with best-fit, we are obliged to scan the entire available list. Best-fit can be speeded up if we keep separate available lists for blocks in various size ranges. For example, we could keep a list of available blocks between 1 and 16 bytes in length,† from 17- 32, 33-64, and so on. This "improvement" does not speed up first-fit appreciably, and in fact may slow it down if the statistics of block sizes are bad. (Compare looking for the first block of size at least 32 on the full available list and on the list for blocks of size 17-32, e.g.) A last observation is that we can define a spectrum of strategies between first-fit and best-fit by looking for a best-fit among the first k available blocks for some fixed k.

The best-fit strategy seems to reduce fragmentation compared with first-fit, in the sense that best-fit tends to produce very small "fragments", i.e., left-over blocks. While the number of these fragments is about the same as for first-fit, they tend to take up a rather small area. However, best-fit will not tend to produce "medium size fragments." Rather, the available blocks will tend to be either very small fragments or will be blocks returned to available space. As a consequence, there are sequences of requests that first-fit can satisfy but not best-fit, as well as vice-versa.

Example 12.7. Suppose, as in Fig. 12.12, the available list consists of blocks of sizes 600, 500, 1000, and 700, in that order. If we are using the first-fit strategy, and a request for a block of size 400 is made, we shall carve it from the block of size 600, that being the first on the list in which a block of size 400 fits. The available list now has blocks of size 200, 500, 1000, 700. We are thus unable to satisfy immediately three requests for blocks of size 600 (although we might be able to do so after merging adjacent empty blocks and/or moving utilized blocks around in the heap).

However, if we were using the best-fit strategy with available list 600, 500, 1000, 700, and the request for 400 came in, we would place it where it fit best, that is, in the block of 500, leaving a list of available blocks 600, 100, 1000, 700. We would, in that event, be able to satisfy three requests for blocks of size 600 without any form of storage reorganization.

On the other hand, there are situations where, starting with the list 600, 500, 1000, 700 again, best-fit would fail, while first-fit would succeed without storage reorganization. Let the first request be for 400 bytes. Best-fit would, as before, leave the list 600, 100, 1000, 700, while first-fit leaves 200, 500, 1000, 700. Suppose the next two requests are for 1000 and 700, so either strategy would allocate the last two empty blocks completely, leaving 600, 100 in the case of best-fit, and 200, 500 in the case of first-fit. Now, first-fit can honor requests for blocks of size 200 and 500, while best-fit obviously cannot.

12.5 Buddy Systems

There is a family of strategies for maintaining a heap that partially avoids the problems of fragmentation and awkward distribution of empty block sizes. These strategies, called "buddy systems," in practice spend very little time merging adjacent empty blocks. The disadvantage of buddy systems is that blocks come in a limited assortment of sizes, so we may waste some space by placing a data item in a bigger block than necessary.

The central idea behind all buddy systems is that blocks come only in certain sizes; let us say that s1 < s2 < s3 < � � � < sk are all the sizes in which blocks can be found. Common choices for the sequence sl, s2, . . . are 1, 2, 4, 8, . . . (the exponential buddy system) and 1, 2, 3, 5, 8, 13, . . . (the Fibonacci buddy system, where si+1 = si+si-1). All the empty blocks of size si are linked in a list, and there is an array of available list headers, one for each size si allowed.† If we require a block of size d for a new datum, we choose an available block of that size si such that si � d, but si-1 < d, that is, the smallest permitted size in which the new datum fits.

Difficulties arise when no empty blocks of the desired size si exist. In that case, we find a block of size si+1 and split it into two, one of size si and the other of size si+1-si.† The buddy system constrains us that si+1 - si be some sj, for j � i. We now see the way in which the choices of values for the si's are constrained. If we let j = i - k, for some k � 0, then since si+1-si = si- k, it follows that

                   si+1 = si +
si-k                         (12.1)

Equation (12.1) applies when i > k, and together with values for s1, s2, . . . , sk, completely determines sk+1, sk+2, . . . . For example, if k = 0, (12.1) becomes

                   si+1 = 2si
(12.2)
Beginning with s1 = 1 in (12.2), we get the exponential sequence 1, 2, 4, 8, . . .. Of course no matter what value of s1 we start with, the s's grow exponentially in (12.2). As another example, if k=1, s1=1, and s2=2, ( 12. l ) becomes

                   si+1 = si +
si-1                         (12.3)
(12.3) defines the Fibonacci sequence: 1, 2, 3, 5, 8, 13, . . ..

Whatever value of k we choose in (12.1) we get a kth order buddy system. For any k, the sequence of permitted sizes grows exponentially; that is, the ratio si+1/si approximates some constant greater than one. For example, for k=0, si+1/si is exactly 2. For k = 1 the ratio approximates the "golden ratio" ((�`5+1)/2 = 1.618), and the ratio decreases as k increases, but never gets as low as 1.

Distribution of Blocks

In the kth order buddy system, each block of size si+1 may be viewed as consisting of a block of size si and one of size si- k. For specificity, let us suppose that the block of size si is to the left (in lower numbered positions) of the block of size si-k.‡ If we view the heap as a single block of size sn, for some large n, then the positions at which blocks of size si can begin are completely determined.

The positions in the exponential, or 0th order, buddy system are easily determined. Assuming positions in the heap are numbered starting at 0, a block of size si begins at any position beginning with a multiple of 2i, that is, 0, 2i, . . .. Moreover, each block of size 2i+1, beginning at say, j2i+1 is composed of two "buddies" of size 2i, beginning at positions (2j)2i, which is j2i+1, and (2j+1)2i. Thus it is easy to find the buddy of a block of size 2i. If it begins at some even multiple of 2i, say (2j)2i, its buddy is to the right, at position (2j+1)2i. If it begins at an odd multiple of 2i, say (2j+1)2i, its buddy is to the left, at (2j)2i.

Example 12.8. Matters are not so simple for buddy systems of order higher than 0. Figure 12.15 shows the Fibonacci buddy system used in a heap of size 55, with blocks of sizes s1, s2, . . ., s8 = 2, 3, 5, 8, 13, 21, 34, and 55. For example, the block of size 3 beginning at 26 is buddy to the block of size 5 beginning at 21; together they comprise the block of size 8 beginning at 21, which is buddy to the block of size 5 beginning at 29. Together, they comprise the block of size 13 starting at 21, and so on.

Fig. 12.15. Division of a heap according to the Fibonacci buddy system.

Allocating Blocks

If we require a block of size n, we choose any one from the available list of blocks of size si, where si � n and either i = 1 or si-1 < n; that is, we choose a best fitting block. In a kth order buddy system, if no blocks of size si exist, we may choose a block of size si+1 or si+k+1 to split, as one of the resulting blocks will be of size si in either case. If no blocks in either of these sizes exist, we may create one by applying this splitting strategy recursively for size si+1.

There is a small catch, however. In a kth order system, we may not split blocks of size s1, s2, . . ., sk, since these would result in a block whose size is smaller than s1. Rather we must use the block whole, if no smaller block is available. This problem does not come up if k=0, i.e., in the exponential buddy system. It could be alleviated in the Fibonacci buddy system if we start with s1 = 1, but that choice may not be acceptable since blocks of size 1 (byte or word, perhaps) could be too small to hold a pointer and a full/empty bit.

Returning Blocks to Available Storage

When a block becomes available for reuse, we can see one of the advantages of the buddy system. We can sometimes reduce fragmentation by combining the newly available block with its buddy, if the buddy is also available.† In fact, should that be the case, we can combine the resulting block with its buddy, if that buddy is empty, and so on. The combination of empty buddies takes only a constant amount of time, and thus is an attractive alternative to periodic mergers of adjacent empty blocks, suggested in the previous section, which takes time proportional to the number of empty blocks.

The exponential buddy system makes the locating of buddies especially easy. If we have just returned the block of size 2i beginning at p2i, its buddy is at (p+1)2i if p is even, and at (p- 1)2i if p is odd.

For a buddy system of order k � 1, finding buddies is not that simple. To make it easier, we shall store certain pieces of information in each block.

  1. A full/empty bit, as every block has.
  2. The size index, which is that integer i such that the block is of size si.
  3. The left buddy count, described below.

In each pair of buddies, one (the left buddy) is to the left of the other (the right buddy). Intuitively, the left buddy count of a block tells how many times consecutively it is all or part of a left buddy. Formally, the entire heap, treated as a block of size sn has a left buddy count of 0. When we divide any block of size si+1, with left buddy count b, into blocks of size si and si-k, which are the left and right buddies respectively, the left buddy gets a left buddy count of b+1, while the right gets a left buddy count of 0, independent of b. For example, in Fig. 12.15, the block of size 3 beginning at 0 has a left buddy count of 6, and the block of size 3 beginning at 13 has a left buddy count of 2.

In addition to the above information, empty blocks, but not utilized ones, have forward and backward pointers for the available list of the appropriate size. The bidirectional pointers make mergers of buddies, which requires deletion from available lists, easy.

The way we use this information is as follows. Suppose k is the order of the buddy system. Any block beginning at position p with a left buddy count of 0 is a right buddy. Thus, if it has size index j, its left buddy is of size sj+k and begins at position p - sj+k. If the left buddy count is greater than 0, then the block is left buddy to a block of size sj-k, which is located beginning at position p+sj.

If we combine a left buddy of size si, having a left buddy count of b, with a right buddy of size si-k, the resulting block has size index i+1, begins at the same position as the block of size si, and has a left buddy count b- 1. Thus, the necessary information can be maintained easily when we merge two empty buddies. The reader may check that the information can be maintained when we split an empty block of size si+1 into a used block of size si and an empty one of size si-k.

If we maintain all this information, and link the available lists in both directions, we spend only a constant amount of time on each split of a block into buddies or merger of buddies into a larger block. Since the number of mergers can never exceed the number of splits, the total amount of work is proportional to the number of splits. It is not hard to recognize that most requests for an allocated block require no splits at all, since a block of the correct size is already available. However, there are bizarre situations where each allocation requires many splits. The most extreme example is where we repeatedly request a block of the smallest size, then return it. If there are n different sizes, we require at least n/k splits in a kth order buddy system, which are then followed by n/k merges when the block is returned.

12.6 Storage Compaction

There are times when, even after merging all adjacent empty blocks, we cannot satisfy a request for a new block. It could be, of course, that there simply is not the space in the entire heap to provide a block of the desired size. But more typical is a situation like Fig. 12.11, where although there are 2200 bytes available, we cannot satisfy a request for a block of more than 1000. The problem is that the available space is divided among several noncontiguous blocks. There are two general approaches to this problem.

  1. Arrange that the available space for a datum can be composed of several empty blocks. If we do so, we may as well require that all blocks are the same size and consist of space for a pointer and space for data. In a used block, the pointer indicates the next block used for the datum and is null in the last block. For example, if we were storing data whose size frequently was small, we might choose blocks of sixteen bytes, with four bytes used for a pointer and twelve for data. If data items were usually long, we might choose blocks of several hundred bytes, again allocating four for a pointer and the balance for data.
  2. When merging adjacent empty blocks fails to provide a sufficiently large block, move the data around in the heap so all full blocks are at the left (low position) end, and there is one large available block at the right.

Method (1), using chains of blocks for a datum, tends to be wasteful of space. If we choose a small block size, we use a large fraction of space for "overhead," the pointers needed to maintain chains. If we use large blocks, we shall have little overhead, but many blocks will be almost wasted, storing a little datum. The only situation in which this sort of approach is to be preferred is when the typical data item is very large. For example, many file systems work this way, dividing the heap, which is typically a disk unit, into equal-sized blocks, of say 512 to 4096 bytes, depending on the system. As many files are much longer than these numbers, there is not too much wasted space, and pointers to the blocks composing a file take relatively little space. Allocation of space under this discipline is relatively straightforward, given what has been said in previous sections, and we shall not discuss the matter further here.

The Compaction Problem

The typical problem we face is to take a collection of blocks in use, as suggested by Fig. 12.16(a), each of which may be of a different size and pointed to by more than one pointer, and slide them left until all available space is at the right end of the heap, as shown in Fig. 12.16(b). The pointers must continue to point to the same data as before, naturally.

Fig. 12.16. The storage compaction process.

There are some simple solutions to this problem if we allocate a little extra space in each block, and we shall discuss another, more complicated method that is efficient, yet requires no extra space in utilized blocks beyond what is required for any of the storage management schemes we have discussed, namely a full/empty bit and a count indicating the size of the block.

A simple scheme for compacting is first to scan all blocks from the left, whether full or empty, and compute a "forwarding address" for each full block. The forwarding address of a block is its present position minus the sum of all the empty space to its left, that is, the position to which the block should be moved eventually. It is easy to calculate forwarding addresses. As we scan blocks from the left, accumulate the amount of empty space we see and subtract this amount from the position of each block we see. The algorithm is sketched in Fig. 12.17.

(1)   var
         p: integer; { the position of the current block }
         gap: integer; { the total amount of empty space seen so far }
      begin
(2)      p := left end of heap;
(4)      gap := 0;
(5)      while p � right end of heap do begin
                { let p point to block B }
(6)             if B is empty then
(7)                 gap := gap + count in block B
                else { B is full }
(8)                 forwarding address of B := p - gap;
(9)             p := p + count in block B
          end
      end;

Fig. 12.17. Computation of forwarding addresses.

Having computed forwarding addresses, we then look at all pointers to the heap.† We follow each pointer to some block B and replace the pointer by the forwarding address found in block B. Finally, we move all full blocks to their forwarding addresses. This process is similar to Fig. 12.17, with line (8) replaced by

      for i := p to p - 1 + count in B do
          heap[i - gap] := heap[i];
to move block B left by an amount gap. Note that the movement of full blocks, which takes time proportional to the amount of the heap in use, will likely dominate the other costs of the compaction.

Morris' Algorithm

F. L. Morris discovered a method for compacting a heap without using space in blocks for forwarding addresses. It does, however, require an endmarker bit associated with each pointer and with each block to indicate the end of a chain of pointers. The essential idea is to create a chain of pointers emanating from a fixed position in each full block and linking all the pointers to that block. For example, we see in Fig. 12.16(a) three pointers, A, D, and E, pointing to the leftmost full block. In Fig. 12.18, we see the desired chain of pointers. A chunk of the data of size equal to that of a pointer has been removed from the block and placed at the end of the chain, where pointer A used to be.

Fig. 12.18. Chaining pointers.

The method for creating such chains of pointers is as follows. We scan all the pointers in any convenient order. Suppose we consider a pointer p to block B. If the endmarker bit in block B is 0, then p is the first pointer found that points to B. We place in p the contents of those positions of B used for the pointer chain, and we make these positions of B point to p. Then we set the endmarker bit in B to 1, indicating it now has a pointer, and we set the endmarker bit in p to 0, indicating the end of the pointer chain and the presence of the displaced data.

Suppose now that when we first consider pointer p to block B the endmarker bit in B is 1. Then B already has the head of a chain of pointers. We copy the pointer in B into p, make B point to p, and set the endmarker bit in p to 1. Thus we effectively insert p at the head of the chain.

Once we have all the pointers to each block linked in a chain emanating from that block, we can move the full blocks far left as possible, just as in the simpler algorithm previously discussed. Lastly, we scan each block in its new position and run down its chain of pointers. Each pointer encountered is made to point to the block in its new position. When we encounter the end of the chain, we restore the data from B, held in the last pointer, to its rightful place in block B and set the endmarker bit in the block to 0.

Exercises

12.1 Consider the following heap of 1000 bytes, where blank blocks are in use, and the labeled blocks are linked on a free list in alphabetical order. The numbers indicate the first byte in each block.

Suppose the following requests are made:
  1. allocate a block of 120 bytes
  2. allocate a block of 70 bytes
  3. return to the front of the available list the block in bytes 700-849
  4. allocate a block of 130 bytes.
Give the free list, in order, after executing the above sequence of steps, assuming free blocks are selected by the strategy of
  1. first fit
  2. best fit.
12.2 Consider the following heap in which blank regions are in use and labeled regions are empty.

Give sequences of requests that can be satisfied if we use
  1. first fit but not best fit
  2. best fit but not first fit.

12.3 | Suppose we use an exponential buddy system with sizes 1, 2, 4, 8, and 16 on a heap of size 16. If we request a block of size n, for 1 � n � 16, we must allocate a block of size 2i, where 2i-1 < n � 2i. The unused portion of the block, if any, cannot be used to satisfy any other request. If we need a block of size 2i, i < 4, and no such free block exists, then we first find a block of size 2i+1 and split it into two equal parts. If no block of size 2i+1 exists, we first find and split a free block of size 2i+2, and so on. If we find ourselves looking for a free block of size 32, we fail and cannot satisfy the request. For the purposes of this question, we never combine adjacent free blocks in the heap.
 | There are sequences of requests a1, a2, . . . , an whose sum is less than 16, such that the last request cannot be satisfied. For example, consider the sequence 5, 5, 5. The first request causes the initial block of size 16 to be split into two blocks of size 8 and one of them is used to satisfy the request. The remaining free block of size 8 satisfies the second request, and there is no free space to satisfy the third request.
 | Find a sequence a1, a2, . . . , an of (not necessarily identical) integers between 1 and 16, whose sum is as small as possible, such that, treated as a sequence of requests for blocks of size a1, a2, . . . , an, the last request cannot be satisfied. Explain why your sequence of requests cannot be satisfied, but any sequence whose sum is smaller can be satisfied.
12.4 | Consider compacting memory while managing equal-sized blocks. Assume each block consists of a data field and a pointer field, and that we have marked every block currently in use. The blocks are currently located between memory locations a and b. We wish to relocate all active blocks so that they occupy contiguous memory starting at a. In relocating a block remember that the pointer field of any block pointing to the relocated block must be updated. Design an algorithm for compacting the blocks.
12.5 | Consider an array of size n. Design an algorithm to shift all items in the array k places cyclically counterclockwise with only constant additional memory independent of k and n. Hint. Consider what happens if we reverse the first k elements, the last n-k elements, and then finally the entire array.
12.6 | Design an algorithm to replace a substring y of a string xyz by another substring y' using as little additional memory as possible. What is the time and space complexity of your algorithm?
12.7 | Write a program to make a copy of a given list. What is the time and space complexity of your program?
12.8 | Write a program to determine whether two lists are identical. What is the time and space complexity of your program?
12.9 | Implement Morris' heap compaction algorithm of Section 12.6.
*12.10 | Design a storage allocation scheme for a situation in which memory is allocated and freed in blocks of lengths one and two. Give bounds on how well your algorithm works.

Bibliographic Notes

Efficient storage management is a central concern in many programming languages, including Snobol [Farber, Griswold, and Polonsky (1964)], Lisp [McCarthy (1965)], APL [Iverson (1962)], and SETL [Schwartz (1973)]. Nicholls [1975] and Pratt [1975] discuss storage management techniques in the context of programming language compilation.

The buddy system of storage allocation was first published by Knowlton [1965]. Fibonacci buddy systems were studied by Hirschberg [1973].

The elegant marking algorithm for use in garbage collection was discovered by Peter Deutsch (Deutsch and Bobrow [1966]) and by Schorr and Waite [1967]. The heap compaction scheme in Section 12.6 is from Morris [1978].

Robson [1971] and Robson [1974] analyzes the amount of memory needed for dynamic storage allocation algorithms. Robson [1977] presents a bounded workspace algorithm for copying cyclic structures. Fletcher and Silver [1966] contains another solution to Exercise 12.5 that uses little additional memory.

† Each programming language must provide for itself a method of representing the current set of variables, and any of the methods discussed in Chapters 4 and 5 is appropriate. For example, most implementations use a hash table to hold the variables.

† This awkwardness is made necessary by peculiarities of Pascal.

† Note that in Fig. 12.1, instead of a count indicating the length of the block, we used the length of the data.

† The reader should, as an exercise, discover how to maintain the pointers when a block is split into two; presumably one piece is used for a new data item, and the other remains empty.

† If c - d is so small that a count and pointer cannot fit, we must use the whole block and delete it from the available list.

† Actually, there is a minimum block size larger than 1, since blocks must hold a pointer, a count and a full/empty bit if they are to be chained to an available list.

† Since empty blocks must hold pointers (and, as we shall see, other information as well) we do not really start the sequence of permitted sizes at 1, but rather at some suitably larger number in the sequence, say 8 bytes.

† Of course, if there are no empty blocks of size si+1, we create one by splitting a block of size si+2, and so on. If no blocks of any larger size exist, we are effectively out of space and must reorganize the heap as in the next section.

‡ Incidentally, it is convenient to think of the blocks of sizes si and si - k making up a block of size si + 1 as "buddies," from whence comes the term "buddy system."

† As in the previous section, we must assume that one bit of each block is reserved to tell whether the block is in use or empty.

† In all that follows we assume the collection of such pointers is available. For example, a typical Snobol implementation stores pairs consisting of a variable name and a pointer to the value for that name in a hash table, with the hash function computed from the name. Scanning the whole hash table allows us to visit all pointers.

Table of Contents


Data Structures and Algorithms: Bibliography

 Return to Table of Contents

Bibliography

Adel'son-Velskii, G. M., and Y. M. Landis [1962]. "An algorithm for the organization of information," Dokl. Akad. Nauk SSSR 146, pp. 263-266 English translation in Soviet Math. Dokl. 3, pp. 1259-1262.

Aho, A. V., M. R. Garey, and J. D. Ullman [1972]. "The transitive reduction of a directed graph," SIAM J. Computing 1:2, pp. 131-137.

Aho, A. V., J. E. Hopcroft, and J. D. Ullman [1974]. The Design and Analysis of Computer Algorithms," Addison-Wesley, Reading, Mass.

Aho, A. V., and N. J. A. Sloane [1973]. "Some doubly exponential sequences," Fibonnaci Quarterly 11:4, pp. 429-437.

Aho, A. V., and J. D. Ullman [1977]. Principles of Compiler Design, Addison-Wesley, Reading, Mass.

Bayer, R., and E. M. McCreight [1972]. "Organization and maintenance of large ordered indices," Acta Informatica 1:3, pp. 173-189.

Bellman, R. E. [1957]. Dynamic Programming, Princeton University Press, Princeton, N. J.

Bentley, J. L. [1982]. Writing Efficient Programs, Prentice-Hall, Englewood Cliffs, N. J.

Bentley, J. L., D. Haken, and J. B. Saxe [1978]. "A general method for solving divide-and-conquer recurrences," CMU-CS-78-154, Dept. of CS, Carnegie-Mellon Univ., Pittsburg, Pa.

Berge, C. [1957]. "Two theorems in graph theory," Proc. National Academy of Sciences 43, pp. 842-844.

Berge, C. [1958]. The Theory of Graphs and its Applications, Wiley, N. Y.

Birtwistle, G. M., O.-J. Dahl, B. Myhrhaug, and K. Nygaard [1973]. SIMULA Begin, Auerbach Press, Philadelphia, Pa.

Blum, M., R. W. Floyd, V. R. Pratt, R. L. Rivest, and R. E. Tarjan [1972]. "Time bounds for selection," J. Computer and System Sciences 7:4, pp. 448- 461.

Boruvka, O. [1926]. "On a minimal problem," Práce Moraské Pridovedecké Spolecnosti 3:3, pp. 37-58.

Brooks, F. P. [1974]. The Mythical Man Month, Addison-Wesley, Reading, Mass.

Carter, J. L., and M. N. Wegman [1977]. "Universal classes of hash functions," Proc. Ninth Annual ACM Syrup. on Theory of Computing, pp. 106-112.

Cheriton, D., and R. E. Tarjan [1976]. "Finding minimum spanning trees," SIAM J. Computing 5:4, pp. 724-742.

Cocke, J., and F. E. Allen [1976]. "A program data flow analysis procedure," Comm. ACM 19:3, pp. 137-147.

Coffman, E. G. (ed.) [1976]. Computer and Job Shop Scheduling Theory, John Wiley and Sons, New York.

Comer, D. [1979]. "The ubiquitous B-tree," Computing Surveys 11, pp. 121-137.

Cooley, J. M., and J. W. Tukey [1965]. "An algorithm for the machine calculation of complex Fourier series," Math. Comp. 19, pp. 297-301.

DBTG [1971]. CODASYL Data Base Task Group April 1971 Report, ACM, New York.

Demers, A., and J. Donahue [1979]. "Revised report on RUSSELL," TR 79-389, Dept. of Computer Science, Cornell Univ., Ithaca, N. Y.

Deo, N. [1975]. Graph Theory with Applications to Engineering and Computer Science, Prentice-Hall, Englewood Cliffs, N. J.

Deutsch, L. P., and D. G. Bobrow [1966]. "An efficient incremental automatic garbage collector," Comm. ACM 9:9, pp. 522-526.

Dijkstra, E. W. [1959]. "A note on two problems in connexion with graphs," Numerische Mathematik 1, pp. 269-271.

Edmonds, J. [1965]. "Paths, trees, and flowers," Canadian J. Math 17, pp. 449-467.

Even, S. [1980]. Graph Algorithms, Computer Science Press, Rockville, Md.

Even, S., and O. Kariv [1975]. "An 0(n2.5) algorithm for maximum matching in general graphs," Proc. IEEE Sixteenth Annual Symp. on Foundations of Computer Science, pp. 100-112.

Farber, D., R. E. Griswold, and I. Polonsky [1964]. "SNOBOL, a string manipulation language," J. ACM 11:1, pp. 21-30.

Fischer, M. J. [1972]. "Efficiency of equivalence algorithms," in Complexity of Computer Computations (R. E. Miller and J. W. Thatcher, eds.) pp. 153-168.

Fletcher, W., and R. Silver [1966]. "Algorithm 284: interchange of two blocks of data," Comm. ACM 9:5, p. 326.

Floyd, R. W. [1962]. "Algorithm 97: shortest path," Comm. ACM 5:6, p. 345.

Floyd, R. W. [1964]. "Algorithm 245: treesort 3," Comm. ACM 7:12, p. 701.

Floyd, R. W., and A. Smith [1973]. "A linear time two-tape merge," Inf. Processing letters 2:5, pp. 123-126.

Ford, L. R., and S. M. Johnson [1959]. "A tournament problem," Amer. Math. Monthly 66, pp. 387-389.

Frazer, W. D., and A. C. McKellar [1970]. "Samplesort: a sampling approach to minimal tree storage sorting," J. ACM 17:3, pp. 496-507.

Fredkin, E. [1960]. "Trie memory," Comm. ACM 3:9, pp. 490-499.

Friend, E. H. [1956]. "Sorting on electronic computer systems," J. ACM 3:2, pp. 134-168.

Fuchs, H., Z. M. Kedem, and S. P. Uselton [1977]. "Optimal surface reconstruction using planar contours," Comm. ACM 20:10, pp. 693-702.

Garey, M. R., and D. S. Johnson [1979]. Computers and Intractability: a Guide to the Theory of NP-Completeness, Freeman, San Francisco.

Geschke, C. M., J. H. Morris, Jr., and E. H. Satterthwaite [1977]. "Early expreience with MESA," Comm. ACM 20:8, pp. 540-552.

Gotlieb, C. C., and L. R. Gotlieb [1978]. Data Types and Data Structures, Prentice-Hall, Englewood Cliffs, N. J.

Greene, D. H., and D. E. Knuth [1983]. Mathematics for the Analysis of Algorithms, Birkhauser, Boston, Mass.

Gudes, E., and S. Tsur [1980]. "Experiments with B-tree reorganization," ACM SIGMOD Symposium on Management of Data, pp. 200-206.

Hall, M. [1948]. "Distinct representatives of subsets," Bull. AMS 54, pp. 922-926.

Harary, F. [1969]. Graph Theory, Addison-Wesley, Reading, Mass.

Hirschberg, D. S. [1973]. "A class of dynamic memory allocation algorithms," Comm. ACM 16:10, pp. 615-618.

Hoare, C. A. R. [1962]. "Quicksort," Computer J. 5:1, pp. 10-15.

Hoare, C. A. R., O.-J. Dahl, and E. W. Dijkstra [1972]. Structured Programming, Academic Press, N. Y.

Hopcroft, J. E., and R. M. Karp [1973]. "An n5/2 algorithm for maximum matchings in bipartite graphs," SIAM J. Computing 2:4, pp. 225-231.

Hopcroft, J. E., and R. E. Tarjan [1973]. "Efficient algorithms for graph manipulation," Comm. ACM 16:6, pp. 372-378.

Hopcroft, J. E., and J. D. Ullman [1973]. "Set merging algorithms," SIAM J. Computing 2:4, pp. 294-303.

Huffman, D. A. [1952]. "A method for the construction of minimum-redundancy codes," Proc. IRE 40, pp. 1098-1101.

Hunt, J. W., and T. G. Szymanski [1977]. "A fast algorithm for computing longest common subsequences," Comm. ACM 20:5, pp. 350-353.

Iverson, K. [1962]. A Programming Language, John Wiley and Sons, New York.

Johnson, D. B. [1975]. "Priority queues with update and finding minimum spanning trees," Inf. Processing Letters 4:3, pp. 53-57.

Johnson, D. B. [1977]. "Efficient algorithms for shortest paths is sparse networks," J. ACM 24:1, pp. 1-13.

Karatsuba, A., and Y. Ofman [1962]. "Multiplication of multidigit numbers on automata," Dokl. Akad. Nauk SSSR 145, pp. 293-294.

Kernighan, B. W., and P. J. Plauger [1974]. The Elements of Programming Style, McGraw-Hill, N. Y.

Kernighan, B. W., and P. J. Plauger [1981]. Software Tools in Pascal, Addison-Wesley, Reading, Mass.

Knowlton, K. C. [1965]. "A fast storage allocator," Comm. ACM 8:10, pp. 623-625.

Knuth, D. E. [1968]. The Art of Computer Programming Vol. I: Fundamental Algorithms, Addison-Wesley, Reading, Mass.

Knuth, D. E. [1971]. "Optimum binary search trees," Acta Informatica 1:1, pp. 14-25.

Knuth, D. E. [1973]. The Art of Computer Programming Vol. III: Sorting and Searching, Addison-Wesley, Reading, Mass.

Knuth, D. E. [1981]. TEX and Metafont, New Directions in Typesetting, Digital Press, Bedford, Mass.

Kruskal, J. B. Jr. [1956]. "On the shortest spanning subtree of a graph and the traveling salesman problem," Proc. AMS 7:1, pp. 48-50.

Lin, S., and B. W. Kernighan [1973]. "A heuristic algorithm for the traveling salesman problem," Operations Research 21, pp. 498-516.

Liskov, B., A. Snyder, R. Atkinson, and C. Scaffert [1977]. "Abstraction mechanisms in CLU," Comm. ACM 20:8, pp. 564-576.

Liu, C. L. [1968]. Introduction to Combinatorial Mathematics, McGraw-Hill, N. Y.

Lueker, G. S. [1980]. "Some techniques for solving recurrences," Computing Surveys, 12:4, pp. 419-436.

Lum, V., and H. Ling [1970]. "Multi-attribute retrieval with combined indices," Comm. ACM 13:11, pp. 660-665.

Maurer, W. D., and T. G. Lewis [1975]. "Hash table methods," Computing Surveys 7:1, pp. 5-20.

McCarthy, J. et al. [1965]. LISP 1.5 Programmers Manual, MIT Press, Cambridge, Mass.

Micali, S., and V. V. Vazirani [1980]. "An [image] algorithm for finding maximum matching in general graphs," Proc. IEEE Twenty-first Annual Symp. on Foundations of Computer Science, pp. 17-27.

Moenck, R., and A. B. Borodin [1972]. "Fast modular transforms via division," Proc. IEEE Thirteenth Annual Symp. on Switching and Automata Theory, pp. 90-96.

Morris, F. L. [1978]. "A time- and space-efficient garbage compaction algorithm," Comm. ACM 21:8, pp. 662-665.

Morris, R. [1968]. "Scatter storage techniques," Comm. ACM 11:1, pp. 35-44.

Moyles, D. M., and G. L. Thompson [1969]. "Finding a minimum equivalent graph of a digraph," J. ACM 16:3, pp. 455-460.

Nicholls, J. E. [1975]. The Structure and Design of Programming Languages, Addison-Wesley, Reading, Mass.

Nievergelt, J. [1974]. "Binary search trees and file organization," Computer Surveys 6:3, pp. 195-207.

Papadimitriou, C. H., and K. Steiglitz [1982]. Combinatorial Optimization: Algorithms and Complexity, Prentice-Hall, Englewood Cliffs, N. J.

Parker, D. S. Jr. [1980]. "Conditions for optimality of the Huffman algorithm," SIAM J. Computing 9:3, pp. 470-489.

Perl, Y., A. Itai, and H. Avni [1978]. "Interpolation search -- a log log n search," Comm. ACM 21:7, pp. 550-553.

Peterson, W. W. [1957]. "Addressing for random access storage," IBM J. Res. and Devel. 1:2, pp. 130-146.

Pratt, T. W. [1975]. Programming Languages: Design and Implementation, Prentice-Hall, Englewood Cliffs, N. J.

Pratt, V. R. [1979]. Shellsort and Sorting Networks, Garland, New York.

Prim, R. C. [1957]. "Shortest connection networks and some generalizations," Bell System Technical J. 36, pp. 1389-1401.

Reingold, E. M. [1972]. "On the optimality of some set algorithms," J. ACM 19:4, pp. 649-659.

Robson, J. M. [1971]. "An estimate of the store size necessary for dynamic storage allocation," J. ACM 18:3, pp. 416-423.

Robson, J. M. [1974]. "Bounds for some functions concerning dynamic storage allocation," J. ACM 21:3, pp. 491-499.

Robson, J. M. [1977]. "A bounded storage algorithm for copying cyclic structures," Comm. ACM 20:6, pp. 431-433.

Sahni, S. [1974]. "Computationally related problems," SIAM J. Computing 3:3, pp. 262-279.

Sammet, J. E. [1968]. Programming Languages: History and Fundamentals, Prentice-Hall, Englewood Cliffs, N. J.

Schkolnick, M. [1975]. "The optimal selection of secondary indices for files," Information Systems 1, pp. 141-146.

Schoenhage, A., and V. Strassen [1971]. "Schnelle multiplikation grosser zahlen," Computing 7, pp. 281-292.

Schorr, H., and W. M. Waite [1967]. "An efficient machine-independent procedure for garbage collection in various list structures," Comm. ACM 10:8, pp. 501-506.

Schwartz, J. T. [1973]. On Programming: An Interrum Report on the SETL Project, Courant Inst., New York.

Sharir, M. [1981]. "A strong-connectivity algorithm and its application in data flow analysis," Computers and Mathematics with Applications 7:1, pp. 67-72.

Shaw, M., W. A. Wulf, and R. L. London [1977]. "Abstraction and verification in ALPHARD: defining and specifying iteration and generators," Comm. ACM 20:8, pp. 553-563.

Shell, D. L. [1959]. "A high-speed sorting procedure," Comm. ACM 2:7, pp. 30-32.

Shell, D. L. [1971]. "Optimizing the polyphase sort," Comm. ACM 14:11, pp. 713-719.

Singleton, R. C. [1969]. "Algorithm 347: an algorithm for sorting with minimal storage," Comm. ACM 12:3, pp. 185-187.

Strassen, V. [1969]. "Gaussian elimination is not optimal," Numerische Mathematik 13, pp. 354-356.

Stroustrup, B. [1982]. "Classes: an abstract data type facility for the C language," SIGPLAN Notices 17:1, pp. 354-356.

Suzuki, N. [1982]. "Analysis of pointer 'rotation'," Comm. ACM 25:5, pp. 330-335.

Tarjan, R. E. [1972]. "Depth first search and linear graph algorithms," SIAM J. Computing 1:2, pp. 146-160.

Tarjan, R. E. [1975]. "On the efficiency of a good but not linear set merging algorithm," J. ACM 22:2, pp. 215-225.

Tarjan, R. E. [1981]. "Minimum spanning trees," unpublished memorandum, Bell Laboratories, Murray Hill, N. J.

Tarjan, R. E. [1983]. Data Structures and Graph Algorithms, unpublished manuscript, Bell Laboratories, Murray Hill, N. J.

Ullman, J. D. [1974]. "Fast algorithms for the elimination of common subexpressions," Acta Informatica 2:3, pp. 191-213.

Ullman, J. D. [1982]. Principles of Database Systems, Computer Science Press, Rockville, Md.

van Emde Boas, P., R. Kaas, and E. Zijlstra [1977]. "Design and implementation of an efficient priority queue structure," Math Syst. Theory, 10, pp. 99-127.

Warshall, S. [1962]. "A theorem on Boolean matrices," J. ACM 9:1, pp. 11-12.

Weinberg, G. M. [1971]. The Psychology of Computer Programming, Van Nostrand, N. Y.

Weiner, P. [1973]. "Linear pattern matching algorithms," Proc. IEEE Fourteenth Annual Symp. on Switching and Automata Theory, pp. 1-11.

Wexelblat, R. L. (ed.) [1981]. History of Programming Languages, Academic Press, N. Y.

Wiederhold, G. [1982]. Database Design, McGraw-Hill, New York.

Williams, J. W. J. [1964]. "Algorithm 232: Heapsort," Comm. ACM 7:6, pp. 347-348.

Wirth, N. [1973]. Systematic Programming: An Introduction, Prentice-Hall, Englewood Cliffs, N. J.

Wirth, N. [1976]. Algorithms + Data Structures = Programs, Prentice-Hall, Englewood Cliffs, N. J.

Wulf, W. A., M. Shaw, P. Hilfinger, and L. Flon [1981]. Fundamental Structures of Computer Science, Addison-Wesley, Reading, Mass.

Yao, A. C. [1975]. "An O(|E| log log |V|) algorithm for finding minimum spanning trees," Inf. Processing Letters 4:1, pp. 21-23.

Yao, A. C., and F. F. Yao [1976]. "The complexity of searching a random ordered table," Proc. IEEE Seventeenth Annual Symp. on Foundations of Computer Science, pp. 173-177.

Yourdon, E., and L. L. Constantine [1975]. Structured Design, Yourdon, New York.

Table of Contents


Preface

This book presents the data structures and algorithms that underpin much of today's computer programming. The basis of this book is the material contained in the first six chapters of our earlier work, The Design and Analysis of Computer Algorithms. We have expanded that coverage and have added material on algorithms for external storage and memory management. As a consequence, this book should be suitable as a text for a first course on data structures and algorithms. The only prerequisite we assume is familiarity with some high-level programming language such as Pascal.

We have attempted to cover data structures and algorithms in the broader context of solving problems using computers. We use abstract data types informally in the description and implementation of algorithms. Although abstract data types are only starting to appear in widely available programming languages, we feel they are a useful tool in designing programs, no matter what the language.

We also introduce the ideas of step counting and time complexity as an integral part of the problem solving process. This decision reflects our longheld belief that programmers are going to continue to tackle problems of progressively larger size as machines get faster, and that consequently the time complexity of algorithms will become of even greater importance, rather than of less importance, as new generations of hardware become available.

The Presentation of Algorithms

We have used the conventions of Pascal to describe our algorithms and data structures primarily because Pascal is so widely known. Initially we present several of our algorithms both abstractly and as Pascal programs, because we feel it is important to run the gamut of the problem solving process from problem formulation to a running program. The algorithms we present, however, can be readily implemented in any high-level programming language.

Use of the Book

Chapter 1 contains introductory remarks, including an explanation of our view of the problem-to-program process and the role of abstract data types in that process. Also appearing is an introduction to step counting and "big-oh" and "big-omega" notation.

Chapter 2 introduces the traditional list, stack and queue structures, and the mapping, which is an abstract data type based on the mathematical notion of a function. The third chapter introduces trees and the basic data structures that can be used to support various operations on trees efficiently.

Chapters 4 and 5 introduce a number of important abstract data types that are based on the mathematical model of a set. Dictionaries and priority queues are covered in depth. Standard implementations for these concepts, including hash tables, binary search trees, partially ordered trees, tries, and 2-3 trees are covered, with the more advanced material clustered in Chapter 5.

Chapters 6 and 7 cover graphs, with directed graphs in Chapter 6 and undirected graphs in 7. These chapters begin a section of the book devoted more to issues of algorithms than data structures, although we do discuss the basics of data structures suitable for representing graphs. A number of important graph algorithms are presented, including depth-first search, finding minimal spanning trees, shortest paths, and maximal matchings.

Chapter 8 is devoted to the principal internal sorting algorithms: quicksort, heapsort, binsort, and the simpler, less efficient methods such as insertion sort. In this chapter we also cover the linear-time algorithms for finding medians and other order statistics.

Chapter 9 discusses the asymptotic analysis of recursive procedures, including, of course, recurrence relations and techniques for solving them.

Chapter 10 outlines the important techniques for designing algorithms, including divide-and-conquer, dynamic programming, local search algorithms, and various forms of organized tree searching.

The last two chapters are devoted to external storage organization and memory management. Chapter 11 covers external sorting and large-scale storage organization, including B-trees and index structures.

Chapter 12 contains material on memory management, divided into four subareas, depending on whether allocations involve fixed or varying sized blocks, and whether the freeing of blocks takes place by explicit program action or implicitly when garbage collection occurs.

Material from this book has been used by the authors in data structures and algorithms courses at Columbia, Cornell, and Stanford, at both undergraduate and graduate levels. For example, a preliminary version of this book was used at Stanford in a 10-week course on data structures, taught to a population consisting primarily of Juniors through first-year graduate students. The coverage was limited to Chapters 1-4, 9, 10, and 12, with parts of 5-7.

Exercises

A number of exercises of varying degrees of difficulty are found at the end of each chapter. Many of these are fairly straightforward tests of the mastery of the material of the chapter. Some exercises require more thought, and these have been singly starred. Doubly starred exercises are harder still, and are suitable for more advanced courses. The bibliographic notes at the end of each chapter provide references for additional reading.

Acknowledgments

We wish to acknowledge Bell Laboratories for the use of its excellent UNIX™-based text preparation and data communication facilities that significantly eased the preparation of a manuscript by geographically separated authors. Many of our colleagues have read various portions of the manuscript and have given us valuable comments and advice. In particular, we would like to thank Ed Beckham, Jon Bentley, Kenneth Chu, Janet Coursey, Hank Cox, Neil Immerman, Brian Kernighan, Steve Mahaney, Craig McMurray, Alberto Mendelzon, Alistair Moffat, Jeff Naughton, Kerry Nemovicher, Paul Niamkey, Yoshio Ohno, Rob Pike, Chris Rouen, Maurice Schlumberger, Stanley Selkow, Chengya Shih, Bob Tarjan, W. Van Snyder, Peter Weinberger, and Anthony Yeracaris for helpful suggestions. Finally, we would like to give our warmest thanks to Mrs. Claire Metzger for her expert assistance in helping prepare the manuscript for typesetting.

A.V.A.
J.E.H.
J.D.U.

Table of Contents Go to Chapter 1


Data Structures and Algorithms: Table of Contents

Data Structures and Algorithms

Alfred V. Aho, Bell Laboratories, Murray Hill, New Jersey
John E. Hopcroft, Cornell University, Ithaca, New York
Jeffrey D. Ullman, Stanford University, Stanford, California

PREFACE

Chapter 1 Design and Analysis of Algorithms

Chapter 2 Basic Data Types

Chapter 3 Trees

Chapter 4 Basic Operations on Sets

Chapter 5 Advanced Set Representation Methods

Chapter 6 Directed Graphs

Chapter 7 Undirected Graphs

Chapter 8 Sorting

Chapter 9 Algorithm Analysis Techniques

Chapter 10 Algorithm Design Techniques

Chapter 11 Data Structures and Algorithms for External Storage

Chapter 12 Memory Management

Bibliography


JObjects QuestAgent - Large size applet

Search

Click here for search tips.

Search Tips
  • Enter one or more keywords.
  • Searches are not case sensitive.
  • Do not use "quotes" around keywords.
  • You can enter multiple keywords, such as good food. Or you can enter a Boolean expression, using AND, OR and NOT.
  • Multiple keywords entered without a Boolean use the AND expression. Thus, entering good food is the same as entering good and food.
  • You can use an expression like pizza or pie to find pages containing either keyword.
  • You can also use parentheses in Boolean expressions. For example, you can enter (pizza and cheese) or pie.
  • You can use the asterisk (*) as a suffix wildcard character in keywords. For example, enter cook* to find pages with keywords such as cook or cookery or cooking.
  • After entering the query, click the Search button or press Enter.
  • If no matches are found, reword your query and try again.
  • If matches are listed, double click a matched entry; or, select the entry and click the Show button.
  • You can check the option Show in New Window to display the target HTML page in a new browser window. This option allows you to preserve the search page's current results while you check the content of the target HTML page. Use Alt+Tab to switch between open browser windows.
  • When the destination HTML page appears, you may want to enter Ctrl+F and use the browser's Find function to locate the keyword on the page.


Dr. Dobb's Journal CD-ROM Catalog

Don't forget to visit our web site for special offers on our products. US & Canadian customers can call us toll free at 1-800-992-0549. International orders can be taken from this number +1-785-841-1631.
Dr. Dobb's/CD Release 7 CD-ROM
Dr. Dobb's/CD Release 7 contains over 11 years of cutting-edge computer information and expertise from the most respected programmers in the industry. This means over 132 issues of Dr. Dobb's Journal plus all the issues of Dr. Dobb's Sourcebook on one CD-ROM! Every Programming Paradigms, every Algorithm Alley, every column, and every bit of source code from January 1988 to June 1999, all at your fingertips. Now in HTML, Dr. Dobb's/CD Release 7 provides you with a fast, accurate and extensive cross-platform search engine that will run on any operating system.
Essential Books on Algorithms and Data Structures - Release 2 CD-ROM
Complete text of nine essential books and over a dozen articles related to algorithms in a new HTML format. These books, selected by the editors of Dr. Dobb's Journal, are the most important books ever written on the subject and need to succeed. Plus, this CD-ROM features a full-text, cross-platform search engine, giving you instant access to the entire contents of all nine books. Learn from the experts. Set up your algorithms the right way the first time-before you ever write that first line of code!
Dr. Dobb's Python Resource CD-ROM
The Dr. Dobb's Python Resource CD-ROM is the best available source of Python information you'll find! It includes Python distributions for platforms ranging from Linux to Win32, and utilities and applications for programmers and end users alike. The CD-ROM also includes tutorials, FAQs, and Python articles from leading magazines such as Dr. Dobb's Journal, Web Techniques, and more!
Essential Books on Numerics and Numerical Programming CD-ROM
Dr. Dobb's Essential Books on Numerics and Numerical Programming CD-ROM is the answer for anyone who needs solutions for complex numerical programming problems. Selected for the value of their content by the editors of DDJ, these books on CD-ROM are a must have for all programmers. Compiled into easy-to-read PDF format, Essential Books on Numerics and Numerical Programming CD-ROM includes these special features: Adobe Acrobat 3.01, fast and accurate full-text search utility!
Dr. Dobb's Tcl/Tk Resource CD-ROM
Dr. Dobb's Journal & the Tcl/Tk Consortium deliver the most comprehensive resource of Tcl/Tk available on one CD-ROM! Known for its flexibility, Tcl/Tk can be implemented quickly and easily into new and legacy applications. Use Tcl/Tk to: Build web sites, perform hardware and software testing, act as an embedded command language, create network-aware applications and much more!
Dr. Dobb's Windows CE Programming CD-ROM
Dr. Dobb's Journal Windows CE Programming: For The Handheld PC CD-ROM teaches you how to write compact efficient programs in C++ using the Windows CE API. You get solid technical information with time proven instructional techniques- plus the comfort of selecting your learning environment. Quizzes, graphics and hands- on lab exercises insure that you become a skilled Windows CE programmer.
Essential Books on Cryptography and Security CD-ROM
This CD-ROM presents both the theory and practice of network security implemented in C, Basic, and other familiar programming languages. Special Features include: Full-Text Search Engine, Complete text of all books, Hyperlinks across all books, Technical bulletins, Security briefs, and Cryptographic FAQs from RSA Data Security. Now available for all platforms in Adobe PDF.
Dr. Dobb's Systems Internals CD-ROM
The Dr. Dobb's Systems Internals CD-ROM is a complete resource for people who need to gain a better understanding of how Windows operate. Network administrators, software developers, and users alike stand to benefit from this release. The NT Internals CD-ROM contains source code and executables for freeware and shareware utilities. Articles from Dr. Dobb's Journal, Microsoft Systems Journal, C/C++ Users Journal and Windows Developer's Journal all relating to Windows 3.x/95/98/NT/CE internals and systems programming.
Programmers at Work CD-ROM
Compiled by the editors of Dr. Dobb's Journal, the Programmers at Work CD-ROM contains full text of Susan Lammers' Programmers at Work - a timeless glimpse into the masterminds of the software industry and James Hague's Halcyon Days: Interviews with Classic Computer and Video Game Programmers. In addition, Programmers at Work includes never before published interviews by DDJ and audio & video clips from Dr. Dobb's Technetcast.com Internet Radio Program. Over 50 interviews with the industries leading programmers!
Essential Books on File-Formats CD-ROM
The Essential Books on File Formats CD-ROM contains the complete text from six books, which will provide you with the most comprehensive and detailed information on all the important and popular file formats in use today. Selected by the editors of Dr. Dobb's, this CD-ROM contains invaluable information on file formats used for graphics, multimedia, sound, databases, spreadsheets, Windows, the internet, and much more!
Essential Books on Graphics Programming CD-ROM
Get seven of the essential books in graphics programming, with full text, diagrams, graphics, and source code-all on one CD-ROM. From fundamental algorithms to the most complex techniques, this CD-ROM lets you find all the critical information you need for your graphics programming projects. Your CD-ROM includes these features: Texture mapping, color modeling, and morphing all optimized for speedy 2-D and 3-D graphics programming. Practical image acquisition and processing. High-quality 3-D photorealistic graphics. Digital halftoning Image synthesis and special effects.
Al Stevens Cram Course on C/C++ CD-ROM
Al Stevens, a world renowned programming expert and contributing editor for Dr. Dobb's Journal, has created the CD-ROM to answer all your C/C++ programming questions. This CD-ROM includes the complete text of three books written by Al Stevens, an interactive step-by step tutorial with precise explanations, video clips of Al discussing important topic, the GNU Compiler Suite directly connected to the exercises, lots of usable source code, plus a memory feature that bookmarks your place for quick returns to prior sessions. Designed for Win95 and NT (will work on 3.1).
Dr. Dobb's Small-C Resource CD-ROM
The Small-C Resource CD-ROM contains the definitive collection of compilers, information, and source code for the Small-C compiler. Packed with books, tools, utilities, and Small-C implementations, this CD-ROM includes James Hendrix's out-of-print book, A Small-C Compiler: Language, Usage, Theory, and Design and several relevant articles from Dr. Dobb's Journal written by C experts such as Allen Holub. Plus all content is hyperlinked in HTML format, which makes viewing easy and convenient. And for those who don't have an HTML browser, included is Netscape's Navigator browser supporting all major operating systems.
Alternative Programming Languages V.2 CD-ROM
The Alternative Programming languages CD-ROM is your definitive resource for cutting-edge programming environments, giving you access to the most innovative, solution-providing languages and tools available today. This CD-ROM will get you up to speed on distributed computing, embeddable languages, increased productivity, and includes languages such as Perl, Python, Dylan, Lout, Bob, Parasol, DSC, Glish, the GNU compiler suite, plus much more.
Windows 32-Bit API Programming Book w/ CD-ROM
Dr. Dobb's Journal is excited to introduce a fundamental book for all Win32-bit API programmers -- Windows 32-bit API Programming - The User Interface. This book/CD-ROM will help you learn Windows 32-bit API programming quickly and proficiently, using an intuitive tutorial that takes you through a series of interactive CD-ROM-based workshops. This hands-on workshop enables you to visualize concepts, as well as listen to audio explanations for each visual, with pop-up windows containing the voice-over text. Also included are comprehensive hands-on lab exercises and quizzes to test your comprehension.


Untitled Document

 Copyright Information for Dr. Dobb's Essential Books on Algorithms and Data Structures CD-ROM

Data Structures and Algorithm Analysis in C (C) 1993, The Benjamin/Cummings Publishing Co., Inc. All rights reserved.

Data Structures, Algorithms, and Program Style Using C (C) 1988, PWS Kent Publishing Co. All rights reserved.

Data Structures: From Arrays to Priority Queues (C)1985, Wadsworth, Inc. All rights reserved.

Fundamentals of Data Structures (C) 1976, 1983 Computer Science Press, Inc. All rights reserved.

Information Retrieval: Data Structures and Algorithms (C)1992, PTR Prentice-Hall, Inc. All rights reserved.

Introduction to Algorithms (C) 1990, The Massachusetts Institute of Technology. All rights reserved.

Practical Data Structures in C++ (C) 1993, John Wiley & Sons, Inc. All rights reserved.

Reliable Data Structures in C (C) 1985, Plum Hall Inc. All rights reserved.

Data Structures and Algorithms (C) 1983, Bell Laboratories, Inc. All rights reserved.

Articles from Dr. Dobb's Journal (C) 1998, Miller Freeman, Inc. All rights reserved.


Fundamentals of Data Structures

Data Structures, Algorithms and Program Style Using C

Data Structures and Algorithm Analysis in C

Data Structures: From Arrays to Priority Queues

Information Retrieval: Data Structures & Algorithms

Introduction to Algorithms

Practical Data Structures in C++

Reliable Data Structures in C

Data Structures and Algorithms

DDJ Algorithms and Data Structures Articles


Dr. Dobb's Algorithms Collection Search | Catalog | Home


Dr. Dobb's Algorithms Collection

 A frames capable browser is required to run this product.


_Algorithm Alley_
by John Boyer

Figure 1:

Input:  Graph G, vertices S (start), T (terminate)
Declare:  H (initially empty heap)
1: For all vertices v
2:    if v == S then v.cost := 0
3:    else v.cost := infinity
3:    Insert v into H
4: Repeat
5:    M := ExtractMin(H)
6:    For each vertex A attached to M
7:       w := cost of edge from M to A
8:       if (M.cost + w < A.cost)
9:          Decrease A's Key to (M.cost + w)
10:         A.backlink := M
11: Until M = T
12: Output T.cost
13: Output vertices on chain of backlinks from T to S

Listing One
//=====================================================================
// ExtractMin() - O(n) worst-case actual; O(lg n) amortized
//=====================================================================

FibHeapNode *FibHeap::ExtractMin()
{
FibHeapNode *Result;
FibHeap *ChildHeap = NULL;

// Remove minimum node and set MinRoot to next node
     if ((Result = Minimum()) == NULL)
          return NULL;
     MinRoot = Result->Right;
     Result->Right->Left = Result->Left;
     Result->Left->Right = Result->Right;
     Result->Left = Result->Right = NULL;

     NumNodes --;
     if (Result->Mark) {
     NumMarkedNodes --;
         Result->Mark = 0;
     }
     Result->Degree = 0;

// Attach child list of Minimum node to the root list of the heap
// If there is no child list, then do no work
     if (Result->Child == NULL) {
     if (MinRoot == Result)
         MinRoot = NULL;
     }
// If MinRoot==Result then there was only one root tree, so the root list is
// the child list of that node (NULL if this is the last node in the list)
     else if (MinRoot == Result)
         MinRoot = Result->Child;
// If MinRoot is different, then the child list is pushed into a new temporary
// heap, which is then merged by Union() onto the root list of this heap.
     else {
         ChildHeap = new FibHeap();
         ChildHeap->MinRoot = Result->Child;
     }
// Complete the disassociation of the Result node from the heap
     if (Result->Child != NULL)
     Result->Child->Parent = NULL;
     Result->Child = Result->Parent = NULL;
// If there was a child list, then we now merge it with rest of the root list
     if (ChildHeap)
         Union(ChildHeap);
// Consolidate heap to find new minimum and do reorganize work
     if (MinRoot != NULL)
         _Consolidate();
// Return the minimum node, which is now disassociated with the heap
// It has Left, Right, Parent, Child, Mark and Degree cleared.
     return Result;
}
//====================================================================
// Consolidate(). Internal function that reorganizes heap as part of an
// ExtractMin(). We must find new minimum in heap, which could be anywhere
// along the root list. The search could be O(n) since all nodes could be on
// the root list. So, we reorganize the tree into more of a binomial forest
// structure, and then find the new minimum on the consolidated O(lg n) sized
// root list, and in the process set each Parent pointer to NULL,
// and count the number of resulting subtrees.
// After a list of n inserts, there will be n nodes on the root list,
// so the first ExtractMin() will be O(n) regardless of whether or not we
// consolidate. However, the consolidation causes subsequent ExtractMin()
// operations to be O(lg n). Furthermore, the extra cost of the first
// ExtractMin() is covered by the higher amortized cost of Insert(), which is
// the real governing factor in how costly the first ExtractMin() will be.
//=====================================================================
void FibHeap::_Consolidate()
{
FibHeapNode *x, *y, *w;
FibHeapNode *A[1+8*sizeof(long)]; // 1+lg(n)
int  I=0, Dn = 1+8*sizeof(long);
short d;

// Initialize the consolidation detection array
     for (I=0; I < Dn; I++)
          A[I] = NULL;
// We need to loop through all elements on root list.  When a collision of
// degree is found, the two trees are consolidated in favor of the one with
// the lesser element key value. We first need to break the circle so that we
// can have a stopping condition (we can't go around until we reach the tree
// we started with because all root trees are subject to becoming a child
// during the consolidation).
     MinRoot->Left->Right = NULL;
     MinRoot->Left = NULL;
     w = MinRoot;

     do {
    x = w;
        d = x->Degree;
        w = w->Right;
// We need another loop here because the consolidated result
// may collide with another large tree on the root list.
        while (A[d] != NULL) {
             y = A[d];
         if (*y < *x)
         _Exchange(x, y);
             if (w == y) w = y->Right;
             _Link(y, x);
             A[d] = NULL;
             d++;
    }
    A[d] = x;
     } while (w != NULL);
// Now we rebuild the root list, find the new minimum, set all root list
// nodes' parent pointers to NULL and count the number of subtrees.
     MinRoot = NULL;
     NumTrees = 0;
     for (I = 0; I < Dn; I++)
          if (A[I] != NULL)
              _AddToRootList(A[I]);
}
//=====================================================================
void FibHeap::_AddToRootList(FibHeapNode *x)
{
     if (x->Mark) NumMarkedNodes --;
     x->Mark = 0;
     NumNodes--;
     Insert(x);
}
//==== Node y is moved from the root list to become a subtree of node x. ====
void FibHeap::_Link(FibHeapNode *y, FibHeapNode *x)
{
// Remove node y from root list
     if (y->Right != NULL)
     y->Right->Left = y->Left;
     if (y->Left != NULL)
         y->Left->Right = y->Right;
     NumTrees--;
// Make node y a singleton circular list with a parent of x
     y->Left = y->Right = y;
     y->Parent = x;
// If node x has no children, then list y is its new child list
     if (x->Child == NULL)
     x->Child = y;
// Otherwise, node y must be added to node x's child list
     else {
         y->Left = x->Child;
         y->Right = x->Child->Right;
         x->Child->Right = y;
         y->Right->Left = y;
     }
// Increase the degree of node x because it's now a bigger tree
     x->Degree ++;
// Node y has just been made a child, so clear its mark
     if (y->Mark) NumMarkedNodes--;
     y->Mark = 0;
}

Listing Two
//=====================================================================
// DecreaseKey() - O(lg n) actual; O(1) amortized
// The O(lg n) actual cost stems from the fact that the depth, and
// therefore the number of ancestor parents, is bounded by O(lg n).
//=====================================================================

int  FibHeap::DecreaseKey(FibHeapNode *theNode, FibHeapNode& NewKey)
{
FibHeapNode *theParent;

     if (theNode==NULL || *theNode < NewKey)
         return NOTOK;
     *theNode = NewKey;

     theParent = theNode->Parent;
     if (theParent != NULL && *theNode < *theParent) {
         _Cut(theNode, theParent);
         _CascadingCut(theParent);
     }
     if (*theNode < *MinRoot)
         MinRoot = theNode;
     return OK;
}
//==== Remove node x from the child list of its parent node y ====
void FibHeap::_Cut(FibHeapNode *x, FibHeapNode *y)
{
     if (y->Child == x)
         y->Child = x->Right;
     if (y->Child == x)
     y->Child = NULL;
     y->Degree --;

     x->Left->Right = x->Right;
     x->Right->Left = x->Left;

     _AddToRootList(x);
}
//=====================================================================
// Cuts each node in parent list, putting successive ancestor nodes on the root
// list until we either arrive at the root list or until we find an ancestor
// that is unmarked. When a mark is set (which only happens during a cascading
// cut), it means that one child subtree has been lost; if a second subtree is
// lost later during another cascading cut, then we move the node to the root
// list so that it can be re-balanced on the next consolidate.
//=====================================================================

void FibHeap::_CascadingCut(FibHeapNode *y)
{
FibHeapNode *z = y->Parent;

     while (z != NULL) {
         if (y->Mark == 0) {
             y->Mark = 1;
             NumMarkedNodes++;
             z = NULL;
         } else {
             _Cut(y, z);
             y = z;
         z = y->Parent;
         }
     }
}


Algorithm Alley
by Tim Kientzle

Listing One
short BitwiseCrc16(int bit, short crc) {
 long longcrc = crc;
 longcrc = (longcrc << 1) ^ bit;  /* next bit */
 if (longcrc & 0x10000)
   longcrc ^= 0x11021; /* reduce */
 return(longcrc & 0xFFFF);
}

Listing Two
short BitwiseCrc16(int bit, short crc) {
 if (crc & 0x8000)
   crc = (crc << 1) ^ bit ^ 0x1021 ;
 else crc = (crc << 1) ^ bit;
 return (crc & 0xffff);
}

Listing Three
static short Crc16Table[256];

void InitCrc16() {
  short i, j, crc;
  for(i=0; i < 256; i++) {
    crc = (i << 8); /* Put i into MSB */
    for(j=0; j < 8; j++) /* Do 8 reductions */
      crc = (crc << 1) ^((crc & 0x8000)? 0x1021:0);
    Crc16Table[i] = crc & 0xFFFF;
  }
}
short Crc16(int ch, short crc) {
    crc = Crc16Table[((crc >> 8) & 255)] ^ (crc << 8) ^ ch;
    return crc & 0xFFFF;
}

Listing Four
/* InitCrc16() is same as Listing Three */
short XYZModemCrc16(int ch, short crc) {
  crc = Crc16Table[((crc >> 8) ^ ch) & 255] ^ (crc << 8);
  return crc & 0xFFFF;
}

Listing Five
static short KermitCrc16Table[256];
void KermitInitCrc16() {
  int i, j, crc;
  for(i=0; i <256; i++) {
    crc = i;  /* Start with i in low-order byte */
    for(j=0; j < 8; j++)
      crc = (crc >> 1) ^ ((crc & 1) ? 0x8408 : 0);
    KermitCrc16Table[i] = crc & 0xFFFF;
  }
}
short KermitCrc16(int ch, short crc) {
  crc = KermitCrc16Table[(crc ^ ch) & 255)] ^ (crc >> 8);
  return crc & 0xFFFF;
}

Listing Six
static long Crc32Table[256];
InitCrc32() {
  int i, j;  long crc;
  for(i=0; i <256; i++) {
    crc = (i << 24); /* Put i into MSB */
    for(j=0; j < 8; j++) /* 8 reductions */
      crc = (crc << 1) ^ ((crc & 0x80000000L) ? 0x04c11db7L : 0);
    Crc32Table[i] = crc;
  }
}
long Crc32(int ch, long crc) {
  crc = Crc32Table[((crc >> 24) ^ ch) & 255] ^ (crc << 8);
  return crc & 0xFFFFFFFF;
}
2


The Block Cipher Encryption Algorithm
by Joan Daemen, Lars R. Knudsen, Vincent Rijmen

/* square.h */

#ifndef __SQUARE_H
#define __SQUARE_H

#define R 8	/* number of rounds */
#define SQUARE_BLOCKSIZE (4*sizeof(word32))

#ifndef USUAL_TYPES
#define USUAL_TYPES
typedef unsigned char	byte;	/*  8 bit */
typedef unsigned short	word16;	/* 16 bit */
typedef unsigned long	word32;	/* 32 bit */
#endif /* ?USUAL_TYPES */

void squareGenerateRoundkeys (word32 key[4],
    word32 roundkeys_e[R+1][4], word32 roundkeys_d[R+1][4]);
void squareEncrypt (word32 text[4], word32 roundkeys_e[R+1][4]);
void squareDecrypt (word32 text[4], word32 roundkeys_d[R+1][4]);

#endif /* __SQUARE_H */

/* square.c */
/*
 * The Square block cipher.
 *
 * Algorithm developed by Joan Daemen <Daemen.J@banksys.com> and
 * Vincent Rijmen <vincent.rijmen@esat.kuleuven.ac.be>
 *
 * This public domain implementation by Paulo S.L.M. Barreto
 * <pbarreto@uninet.com.br> based on software written by Vincent Rijmen.
 *
 * Caveat: this code assumes 32-bit words and probably will not work
 * otherwise.
 *
 * Version 2.0 (1997.02.11)
 *
 * THIS SOFTWARE IS PROVIDED BY THE AUTHORS ''AS IS'' AND ANY EXPRESS
 * OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE IMPLIED
 * WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE
 * ARE DISCLAIMED.  IN NO EVENT SHALL THE AUTHORS OR CONTRIBUTORS BE
 * LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL, SPECIAL, EXEMPLARY, OR
 * CONSEQUENTIAL DAMAGES (INCLUDING, BUT NOT LIMITED TO, PROCUREMENT OF
 * SUBSTITUTE GOODS OR SERVICES; LOSS OF USE, DATA, OR PROFITS; OR
 * BUSINESS INTERRUPTION) HOWEVER CAUSED AND ON ANY THEORY OF LIABILITY,
 * WHETHER IN CONTRACT, STRICT LIABILITY, OR TORT (INCLUDING NEGLIGENCE
 * OR OTHERWISE) ARISING IN ANY WAY OUT OF THE USE OF THIS SOFTWARE,
 * EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE.
 */

#include <stdio.h>
#include <stdlib.h>

#define R 8

#if R != 8
    #error "This implementation is optimized for (and assumes) exactly 8 rounds"
#endif

#ifndef USUAL_TYPES
#define USUAL_TYPES
typedef unsigned char	byte;	/*  8 bit */
typedef unsigned short	word16;	/* 16 bit */
typedef unsigned long	word32;	/* 32 bit */
#endif /* ?USUAL_TYPES */

#include "square.tab"			/* substitution boxes */

#ifdef HARDWARE_ROTATIONS
#define ROTL(x, s) (_lrotl ((x), (s)))
#else  /* !HARDWARE_ROTATIONS */
#define ROTL(x, s) (((x) << (s)) | ((x) >> (32 - (s))))
#endif /* ?HARDWARE_ROTATIONS */

#ifdef MASKED_BYTE_EXTRACTION
#define MSB(x) (((x) >> 24) & 0xffU)	/* most  significant byte */
#define SSB(x) (((x) >> 16) & 0xffU)	/* second in significance */
#define TSB(x) (((x) >>  8) & 0xffU)	/* third  in significance */
#define LSB(x) (((x)      ) & 0xffU)	/* least significant byte */
#else  /* !MASKED_BYTE_EXTRACTION */
#define MSB(x) ((byte)  ((x) >> 24))	/* most  significant byte */
#define SSB(x) ((byte)  ((x) >> 16))	/* second in significance */
#define TSB(x) ((byte)  ((x) >>  8))	/* third  in significance */
#define LSB(x) ((byte)  ((x)      ))	/* least significant byte */
#endif /* ?MASKED_BYTE_EXTRACTION */

#define mul(a, b) ((a && b) ? alogtab[logtab[a] + logtab[b]] : 0)

void squareTransform (word32 in[4], word32 out[4])
    /* apply theta to a roundkey */
{
    int i, j;
    byte A[4][4], B[4][4];

    for (i = 0; i < 4; i++) {
        A[i][0] = (byte) (in[i] >> 24);
        A[i][1] = (byte) (in[i] >> 16);
        A[i][2] = (byte) (in[i] >>  8);
        A[i][3] = (byte) (in[i]      );
    }

    /* B = A * G */
    for (i = 0; i < 4; i++) {
        for (j = 0; j < 4; j++) {
            B[i][j] =
                mul (A[i][0], G[0][j]) ^
                mul (A[i][1], G[1][j]) ^
                mul (A[i][2], G[2][j]) ^
                mul (A[i][3], G[3][j]);
        }
    }

    for (i = 0; i < 4; i++) {
        out[i] =
            (B[i][0] << 24) ^
            (B[i][1] << 16) ^
            (B[i][2] <<  8) ^
            (B[i][3]      );
    }
#ifdef DESTROY_TEMPORARIES
    memset (A, 0, sizeof (A));
    memset (B, 0, sizeof (B));
#endif /* ?DESTROY_TEMPORARIES */
} /* squareTransform */

void squareGenerateRoundkeys (word32 key[4],
    word32 roundkeys_e[R+1][4],
    word32 roundkeys_d[R+1][4])
{
    int t, u;
    word32 tempkeys[R+1][4];

    /* apply the key evolution function */

    for (u = 0; u < 4; u++) {
        tempkeys[0][u] = key[u];
    }

    for (t = 1; t < R+1; t++) {
        tempkeys[t][0] =
            tempkeys[t-1][0] ^ ROTL (tempkeys[t-1][3], 8) ^ offset[t-1];
        tempkeys[t][1] =
            tempkeys[t-1][1] ^ tempkeys[t][0];
        tempkeys[t][2] =
            tempkeys[t-1][2] ^ tempkeys[t][1];
        tempkeys[t][3] =
            tempkeys[t-1][3] ^ tempkeys[t][2];
    }

    /* produce the round keys */
    for (t = 0; t < R; t++) {
        squareTransform (tempkeys[t], roundkeys_e[t]);
    }
    for (u = 0; u < 4; u++) {
        roundkeys_e[R][u] = tempkeys[R][u];
    }
    for (t = 0; t < R; t++) {
        for (u = 0; u < 4; u++) {
            roundkeys_d[t][u] = tempkeys[R-t][u];
        }
    }
    for (u = 0; u < 4; u++) {
        roundkeys_d[R][u] = roundkeys_e[0][u];
    }
#ifdef DESTROY_TEMPORARIES
    memset (tempkeys, 0, sizeof (tempkeys));
#endif /* ?DESTROY_TEMPORARIES */
} /* squareGenerateRoundkeys */

#define squareRound(text, temp, T0, T1, T2, T3, roundkey) \
{ \
    temp[0] = T0[MSB (text[0])] \
            ^ T1[MSB (text[1])] \
            ^ T2[MSB (text[2])] \
            ^ T3[MSB (text[3])] \
            ^ roundkey[0]; \
    temp[1] = T0[SSB (text[0])] \
            ^ T1[SSB (text[1])] \
            ^ T2[SSB (text[2])] \
            ^ T3[SSB (text[3])] \
            ^ roundkey[1]; \
    temp[2] = T0[TSB (text[0])] \
            ^ T1[TSB (text[1])] \
            ^ T2[TSB (text[2])] \
            ^ T3[TSB (text[3])] \
            ^ roundkey[2]; \
    temp[3] = T0[LSB (text[0])] \
            ^ T1[LSB (text[1])] \
            ^ T2[LSB (text[2])] \
            ^ T3[LSB (text[3])] \
            ^ roundkey[3]; \
} /* squareRound */

#define squareFinal(text, temp, S, roundkey) \
{ \
    text[0] = ((word32) (S[MSB (temp[0])]) << 24) \
            ^ ((word32) (S[MSB (temp[1])]) << 16) \
            ^ ((word32) (S[MSB (temp[2])]) <<  8) \
            ^  (word32) (S[MSB (temp[3])]) \
            ^ roundkey[0]; \
    text[1] = ((word32) (S[SSB (temp[0])]) << 24) \
            ^ ((word32) (S[SSB (temp[1])]) << 16) \
            ^ ((word32) (S[SSB (temp[2])]) <<  8) \
            ^  (word32) (S[SSB (temp[3])]) \
            ^ roundkey[1]; \
    text[2] = ((word32) (S[TSB (temp[0])]) << 24) \
            ^ ((word32) (S[TSB (temp[1])]) << 16) \
            ^ ((word32) (S[TSB (temp[2])]) <<  8) \
            ^  (word32) (S[TSB (temp[3])]) \
            ^ roundkey[2]; \
    text[3] = ((word32) (S[LSB (temp[0])]) << 24) \
            ^ ((word32) (S[LSB (temp[1])]) << 16) \
            ^ ((word32) (S[LSB (temp[2])]) <<  8) \
            ^  (word32) (S[LSB (temp[3])]) \
            ^ roundkey[3]; \
} /* squareFinal */

void squareEncrypt(word32 text[4], word32 roundkeys[R+1][4])
{
    word32 temp[4];

    /* initial key addition */
    text[0] ^= roundkeys[0][0];
    text[1] ^= roundkeys[0][1];
    text[2] ^= roundkeys[0][2];
    text[3] ^= roundkeys[0][3];

    /* R - 1 full rounds */
    squareRound (text, temp, Te0, Te1, Te2, Te3, roundkeys[1]);
    squareRound (temp, text, Te0, Te1, Te2, Te3, roundkeys[2]);
    squareRound (text, temp, Te0, Te1, Te2, Te3, roundkeys[3]);
    squareRound (temp, text, Te0, Te1, Te2, Te3, roundkeys[4]);
    squareRound (text, temp, Te0, Te1, Te2, Te3, roundkeys[5]);
    squareRound (temp, text, Te0, Te1, Te2, Te3, roundkeys[6]);
    squareRound (text, temp, Te0, Te1, Te2, Te3, roundkeys[7]);

    /* last round (diffusion becomes only transposition) */
    squareFinal (text, temp, Se, roundkeys[R]);
#ifdef DESTROY_TEMPORARIES
    memset (temp, 0, sizeof (temp));
#endif /* ?DESTROY_TEMPORARIES */
} /* squareEncrypt */

void squareDecrypt(word32 text[4], word32 roundkeys[R+1][4])
{
    word32 temp[4];

    /* initial key addition */
    text[0] ^= roundkeys[0][0];
    text[1] ^= roundkeys[0][1];
    text[2] ^= roundkeys[0][2];
    text[3] ^= roundkeys[0][3];

    /* R - 1 full rounds */
    squareRound (text, temp, Td0, Td1, Td2, Td3, roundkeys[1]);
    squareRound (temp, text, Td0, Td1, Td2, Td3, roundkeys[2]);
    squareRound (text, temp, Td0, Td1, Td2, Td3, roundkeys[3]);
    squareRound (temp, text, Td0, Td1, Td2, Td3, roundkeys[4]);
    squareRound (text, temp, Td0, Td1, Td2, Td3, roundkeys[5]);
    squareRound (temp, text, Td0, Td1, Td2, Td3, roundkeys[6]);
    squareRound (text, temp, Td0, Td1, Td2, Td3, roundkeys[7]);

    /* last round (diffusion becomes only transposition) */
    squareFinal (text, temp, Sd, roundkeys[R]);
#ifdef DESTROY_TEMPORARIES
    memset (temp, 0, sizeof (temp));
#endif /* ?DESTROY_TEMPORARIES */
} /* squareDecrypt */

#ifdef TEST_SQUARE

#include <time.h>

#define TIMING_ITERATIONS 20000000L

int main (void)
{
    word32 key[4], text[4];
    word32 roundkeys_e[R+1][4], roundkeys_d[R+1][4];
    long n; clock_t elapsed; double sec;

    printf ("Square cipher (compiled on " __DATE__ " " __TIME__ ").\n");

    printf ("Checking correctness...\n");
    key[0] = 0x00010203UL;
    key[1] = 0x04050607UL;
    key[2] = 0x08090a0bUL;
    key[3] = 0x0c0d0e0fUL;
    printf ("%08lx %08lx %08lx %08lx  user key\n",
        key[0], key[1], key[2], key[3]);
    squareGenerateRoundkeys (key, roundkeys_e, roundkeys_d);
    text[0] = 0x00010203UL;
    text[1] = 0x04050607UL;
    text[2] = 0x08090a0bUL;
    text[3] = 0x0c0d0e0fUL;
    printf ("%08lx %08lx %08lx %08lx  in\n",
        text[0], text[1], text[2], text[3]);
    squareEncrypt (text, roundkeys_e);
    printf ("%08lx %08lx %08lx %08lx  encrypted\n",
        text[0], text[1], text[2], text[3]);
    printf ("7c3491d9 4994e70f 0ec2e7a5 ccb5a14f  expected\n");
    squareDecrypt (text, roundkeys_d);
    printf ("%08lx %08lx %08lx %08lx  out\n",
        text[0], text[1], text[2], text[3]);

    printf ("Measuring encryption speed...");
    squareGenerateRoundkeys(key, roundkeys_e, roundkeys_d);
    elapsed = -clock ();
    for (n = TIMING_ITERATIONS; n > 0; n--) {
        squareEncrypt(text, roundkeys_e);
    }
    elapsed += clock ();
    sec = elapsed ? (double) elapsed / CLOCKS_PER_SEC : 1.0;
    printf (" %.2f sec, %.1f K/sec.\n",
        sec, 16.0*TIMING_ITERATIONS/1024.0/sec);

    printf ("Measuring decryption speed...");
    elapsed = -clock ();
    for (n = TIMING_ITERATIONS; n > 0; n--) {
        squareDecrypt(text, roundkeys_d);
    }
    elapsed += clock ();
    sec = elapsed ? (double) elapsed / CLOCKS_PER_SEC : 1.0;
    printf (" %.2f sec, %.1f K/sec.\n",
        sec, 16.0*TIMING_ITERATIONS/1024.0/sec);

    return 0;
} /* main */

#endif /* TEST_SQUARE */

/* maketabs.c */

#include <stdio.h>
#include <stdlib.h>

#define R 8
#define ROOT 0x1f5U
#define ROTR(x, s) (((x) >> (s)) | ((x) << (32 - (s))))

typedef unsigned char byte;
typedef unsigned short word16;
typedef unsigned long word32;

byte exptab[256], logtab[256];
byte offset[R];

byte mul(byte a, byte b)
/* multiply two elements of GF(2^m)
 */
{
   if (a && b) return exptab[(logtab[a] + logtab[b])%255];
   else return 0;
}

void init()
/* produce logtab, exptab, and offset,
 * needed for multiplying in the field GF(2^m)
 * and/or in the key schedule
 */
{
   word16 i, j;
   exptab[0] = 1;
   for(i = 1; i < 256; i++) {
      j = exptab[i-1] << 1;
      if (j & 0x100U) j ^= ROOT;
      exptab[i] = (byte)j;
      }
   logtab[0] = 0;
   for(i = 1; i < 255; i++)
      logtab[exptab[i]] = (byte)i;
   /* generate the offset values
    */
   offset[0] = 1;
   for(i = 1; i < R; i++) offset[i] = mul(2,offset[i-1]);
}

static word32 T[256];

void main()
{
   FILE *out;
   byte ibox[256], g[9];
   byte in, u, t, pivot, tmp;
   byte box[256], G[4][4], iG[4][4], A[4][8];
   byte trans[9] = { 0xd6, 0x7b, 0x3d, 0x1f,
                     0x0f, 0x05, 0x03, 0x01,
                     0xb1};
   word16 i, j, k;

   init();
   /* the substitution box based on F^{-1}(x)
    * + affine transform of the output
    */
   box[0] = 0;
   box[1] = 1;
   for(i = 2; i < 256; i++)
      box[i] = exptab[255 - logtab[i]];

   for(i = 0; i < 256; i++) {
      in = box[i];
      box[i] = 0;
      for(t = 0; t < 8; t++) {
         u = in & trans[t];
         box[i] ^= ((1 & (u ^ (u >> 1) ^ (u >> 2) ^ (u >> 3)
                  ^ (u >> 4) ^ (u >> 5) ^ (u >> 6) ^ (u >> 7)))
                   << (7 - t));
         }
      box[i] ^= trans[8];
      }

   /* diffusion box G
    * created by make_g.c
    */
   g[3] = 3;
   g[2] = 1;
   g[1] = 1;
   g[0] = 2;

   for(i = 0; i < 4; i++)
      for(j = 0; j < 4; j++)
         G[i][j] = g[(4 + j - i) % 4];

   for(i = 0; i < 4; i++) {
      for(j = 0; j < 4; j++) A[i][j] = G[i][j];
      for(j = 4; j < 8; j++) A[i][j] = 0;
      A[i][i+4] = 1;
      }
   for(i = 0; i < 4; i++) {
      pivot = A[i][i];
      if (pivot == 0) {
         t = i + 1;
         while ((A[t][i] == 0) && (t < 4)) t++;
         if (t == 4) fprintf(stderr,"noninvertible matrix G\n");
         else {
            for(j = 0; j < 8; j++) {
               tmp = A[i][j];
               A[i][j] = A[t][j];
               A[t][j] = tmp;
               }
            pivot = A[i][i];
            }
         }
      for(j = 0; j < 8; j++)
         if (A[i][j])
            A[i][j] = exptab[(255 + logtab[A[i][j]] - logtab[pivot])%255];
      for(t = 0; t < 4; t++)
         if (i != t)
            {
            for(j = i+1; j < 8; j++)
               A[t][j] ^= mul(A[i][j],A[t][i]);
            A[t][i] = 0;
            }
      }
   for(i = 0; i < 4; i++)
      for(j = 0; j < 4; j++) iG[i][j] = A[i][j+4];

   /* output
    */
   out = fopen("square.tab","w");
   for(i = 0; i < 256; i++) ibox[box[i]] = (byte)i;

   fprintf(out,"static const byte Se[256] = {\n");
   for(i = 0; i < 16; i++) {
      for(j = 0; j < 16; j++) fprintf(out,"%3d, ",box[i*16+j]);
      fprintf(out,"\n");
      }
   fprintf(out,"};\n\n");
   fprintf(out,"static const byte Sd[256] = {\n");
   for(i = 0; i < 16; i++) {
      for(j = 0; j < 16; j++) fprintf(out,"%3d, ",ibox[i*16+j]);
      fprintf(out,"\n");
      }
   fprintf(out,"};\n\n");
   fprintf(out,"static const byte G[4][4] = {\n");
   for(i = 0; i < 4; i++) {
      for(k = 0; k < 4; k++) fprintf(out,"0x%02xU, ",G[i][k]);
      fprintf(out,"\n");
      }
   fprintf(out,"};\n\n");
   fprintf(out,"static const byte iG[4][4] = {\n");
   for(i = 0; i < 4; i++) {
      for(k = 0; k < 4; k++) fprintf(out,"0x%02xU, ",iG[i][k]);
      fprintf(out,"\n");
      }
   fprintf(out,"};\n\n");

   for(t = 0; t < 64; t++) {
      for(k = 0; k < 4; k++) {
         if (box[k])
            T[4*t+k] =
                ((word32) mul(box[4*t+k],G[0][0]) << 24) ^
                ((word32) mul(box[4*t+k],G[0][1]) << 16) ^
                ((word32) mul(box[4*t+k],G[0][2]) <<  8) ^
                ((word32) mul(box[4*t+k],G[0][3]));
         else
            T[4*t+k] = 0L;
         }
      }

   fprintf(out,"static const byte logtab[256] = {\n");
   for(i = 0; i < 16; i++) {
      for(j = 0; j < 16; j++) fprintf(out,"%3u, ",logtab[i*16+j]);
      fprintf(out,"\n");
      }
   fprintf(out,"};\n\n");
   fprintf(out,"static const byte alogtab[512] = {\n");
   for(i = 0; i < 32; i++) {
      for(j = 0; j < 16; j++) fprintf(out,"%3u, ",exptab[(i*16+j)%255]);
      fprintf(out,"\n");
      }
   fprintf(out,"};\n\n");
   fprintf(out,"static const word32 offset[R] = {\n");
   for(i = 0; i < R; i++) {
       fprintf(out,"0x%08lxUL,%s", (word32)(offset[i]) << 24, (i+1)%4 == 0? "\n" : " ");
      }
   fprintf(out,"};\n\n");

   fprintf(out,"static const word32 Te0[256] = {\n");
   for(t = 0; t < 64; t++) {
      for(k = 0; k < 4; k++) {
         fprintf(out,"0x%08lxUL, ", T[4*t+k]);
         }
      fprintf(out,"\n");
      }
   fprintf(out,"};\n\n");

   fprintf(out,"static const word32 Te1[256] = {\n");
   for(t = 0; t < 64; t++) {
      for(k = 0; k < 4; k++) {
         fprintf(out,"0x%08lxUL, ", ROTR(T[4*t+k],  8));
         }
      fprintf(out,"\n");
      }
   fprintf(out,"};\n\n");

   fprintf(out,"static const word32 Te2[256] = {\n");
   for(t = 0; t < 64; t++) {
      for(k = 0; k < 4; k++) {
         fprintf(out,"0x%08lxUL, ", ROTR(T[4*t+k], 16));
         }
      fprintf(out,"\n");
      }
   fprintf(out,"};\n\n");

   fprintf(out,"static const word32 Te3[256] = {\n");
   for(t = 0; t < 64; t++) {
      for(k = 0; k < 4; k++) {
         fprintf(out,"0x%08lxUL, ", ROTR(T[4*t+k], 24));
         }
      fprintf(out,"\n");
      }
   fprintf(out,"};\n\n");

   for(t = 0; t < 64; t++) {
      for(k = 0; k < 4; k++) {
         if (ibox[k])
            T[4*t+k] =
                ((word32) mul(ibox[4*t+k],iG[0][0]) << 24) ^
                ((word32) mul(ibox[4*t+k],iG[0][1]) << 16) ^
                ((word32) mul(ibox[4*t+k],iG[0][2]) <<  8) ^
                ((word32) mul(ibox[4*t+k],iG[0][3]));
         else
            T[4*t+k] = 0L;
         }
      }

   fprintf(out,"static const word32 Td0[256] = {\n");
   for(t = 0; t < 64; t++) {
      for(k = 0; k < 4; k++) {
         fprintf(out,"0x%08lxUL, ", T[4*t+k]);
         }
      fprintf(out,"\n");
      }
   fprintf(out,"};\n\n");

   fprintf(out,"static const word32 Td1[256] = {\n");
   for(t = 0; t < 64; t++) {
      for(k = 0; k < 4; k++) {
         fprintf(out,"0x%08lxUL, ", ROTR(T[4*t+k],  8));
         }
      fprintf(out,"\n");
      }
   fprintf(out,"};\n\n");

   fprintf(out,"static const word32 Td2[256] = {\n");
   for(t = 0; t < 64; t++) {
      for(k = 0; k < 4; k++) {
         fprintf(out,"0x%08lxUL, ", ROTR(T[4*t+k], 16));
         }
      fprintf(out,"\n");
      }
   fprintf(out,"};\n\n");

   fprintf(out,"static const word32 Td3[256] = {\n");
   for(t = 0; t < 64; t++) {
      for(k = 0; k < 4; k++) {
         fprintf(out,"0x%08lxUL, ", ROTR(T[4*t+k], 24));
         }
      fprintf(out,"\n");
      }
   fprintf(out,"};\n\n");

   fclose(out);
   }


Algorithm Alley
by Tim Kientzle

Example 1:

static const int _stepSizeTable[89] = {
   7, 8, 9, 10, 11, 12, 13, 14, 16, 17, 19, 21, 23, 25, 28, 31, 34,
   37, 41, 45, 50, 55, 60, 66, 73, 80, 88, 97, 107, 118, 130, 143,
   157, 173, 190, 209, 230, 253, 279, 307, 337, 371, 408, 449, 494,
   544, 598, 658, 724, 796, 876, 963, 1060, 1166, 1282, 1411, 1552,
   1707, 1878, 2066, 2272, 2499, 2749, 3024, 3327, 3660, 4026,
   4428, 4871, 5358, 5894, 6484, 7132, 7845, 8630, 9493, 10442,
   11487, 12635, 13899, 15289, 16818, 18500, 20350, 22385, 24623,
   27086, 29794, 32767
};

Example 2:
static const int indexAdjustTable[16] = {
   -1, -1, -1, -1,  // +0 - +3, decrease the step size
    2, 4, 6, 8,     // +4 - +7, increase the step size
   -1, -1, -1, -1,  // -0 - -3, decrease the step size
    2, 4, 6, 8,     // -4 - -7, increase the step size
};

Listing One
short ImaAdpcmDecode(unsigned char deltaCode, ImaState &state) {
   // Get the current step size
   int step = _stepSizeTable[state.index];
   // Construct the difference by scaling the current step size
   // This is approximately: difference = (deltaCode+.5)*step/4
   int difference = step>>3;
   if ( deltaCode & 1 ) difference += step>>2;
   if ( deltaCode & 2 ) difference += step>>1;
   if ( deltaCode & 4 ) difference += step;
   if ( deltaCode & 8 ) difference = -difference;
   // Build the new sample
   state.previousValue += difference;
   if (state.previousValue > 32767) state.previousValue = 32767;
   else if (state.previousValue < -32768) state.previousValue = -32768;

   // Update the step for the next sample
   state.index += indexAdjustTable[deltaCode];
   if (state.index < 0) state.index = 0;
   else if (state.index > 88) state.index = 88;

   return state.previousValue;
}

Listing Two
unsigned char ImaAdpcmEncode(short sample, ImaState &state) {
   int diff = sample - state.previousValue;
   int step = _stepSizeTable[state.index];
   int deltaCode = 0;

   // Set sign bit
   if (diff < 0) { deltaCode = 8; diff = -diff; }

   // This is essentially deltaCode = (diff<<2)/step,
   // except the roundoff is handled differently.
   if ( diff >= step ) {  deltaCode |= 4;  diff -= step;  }
   step >>= 1;
   if ( diff >= step ) {  deltaCode |= 2;  diff -= step;  }
   step >>= 1;
   if ( diff >= step ) {  deltaCode |= 1;  diff -= step;  }

   ImaAdpcmDecode(deltaCode,state);  // update state
   return deltaCode;
}

#ifndef IMAADPCM_H_INCLUDED
#define IMAADPCM_H_INCLUDED

struct ImaState {
   int index;    // Index into step size table
   int previousValue; // Most recent sample value
};
// Decode/Encode a single sample and update state
short ImaAdpcmDecode(unsigned char deltaCode, ImaState &);
unsigned char ImaAdpcmEncode(short sample, ImaState &);
// Microsoft-compatible decoder
class DecompressImaAdpcmMs {
private:
   int  _channels;
   short *_samples[2];  // Left and right sample buffers
   short *_samplePtr[2]; // Pointers to current samples
   size_t   _samplesRemaining; // Samples remaining in each channel
   size_t   _samplesPerPacket; // Total samples per packet
public:
   DecompressImaAdpcmMs(int packetLength, int channels);
   ~DecompressImaAdpcmMs();
   size_t GetSamples(short *outBuff, size_t len);
private:
   unsigned char *_packet;   // Temporary buffer for packets
   size_t   _bytesPerPacket; // Size of a packet
   size_t  NextPacket();
   size_t ReadBytes(void *buffer, size_t request);
};
// Apple-compatible decoder
class DecompressImaAdpcmApple {
private:
   int _channels;
   ImaState _state[2];
   short _samples[2][64];
   short *_samplePtr[2];
   size_t   _samplesRemaining;
   size_t  NextPacket(short *sampleBuffer, ImaState &state);
public:
   DecompressImaAdpcmApple(int channels);
   size_t GetSamples(short *outBuff, size_t len);
   size_t ReadBytes(void *buffer, size_t request);
};
#endif

Listing Four
size_t  DecompressImaAdpcmApple::NextPacket(short *sampleBuffer,
                                           ImaState &state) {
   unsigned char _packet[34];
   // Pull in the packet and check the header
   size_t bytesRead = ReadBytes(_packet,34);
   if (bytesRead < 34) return 0;

   // Check the decompressor state
   state.index = _packet[1] & 0x7f;
   if (state.index > 88) {
      cerr << "Synchronization error.\n";
      exit(1);
   }
   // Recover upper nine bits of last sample
   short lastSample =
      (static_cast<signed char>(_packet[0])*0x100 +
       static_cast<signed char>(_packet[1])) & 0xff80;
   // If ours doesn't match, reset to the value in the file
   if ((state.previousValue & 0xff80) != lastSample)
      state.previousValue = lastSample;
   // Decompress nybbles
   for(int i=0; i<32; i++) {
      *sampleBuffer++ = ImaAdpcmDecode(_packet[i+2] & 0xf, state);
      *sampleBuffer++ = ImaAdpcmDecode((_packet[i+2]>>4) & 0xf, state);
   }
   return 64;
};

1


Algorithm Alley
by John Boyer

Listing One
char *lex_growbuffer(char *Buffer, long *pBufferSize)
{
char    *TempP;
long    NewSize;
    if (*pBufferSize==0) NewSize = 16;
    else NewSize = *pBufferSize * 2;
    TempP = (char *) realloc(Buffer, NewSize+1);
    if (TempP != NULL) *pBufferSize = NewSize;
    return TempP;
}
  ... /* Snippet to add a character to the lex buffer */

   if (BufferLen == BufferSize)
        Buffer = lex_growbuffer(Buffer, &BufferSize);
    if (Buffer == NULL) return NULL;
    Buffer[BufferLen++] = NewChar;
    Buffer[BufferLen] = '\0';
    return Buffer;
  ...
/* After string is read, reduce to exact size */
    Buffer = (char *) realloc(Buffer, BufferLen+1);

Listing Two
typedef unsigned long ulong;
typedef unsigned char uchar;
ulong HashJB(uchar *KeyStr, ulong N)
{
ulong Hash=0, I=0;
uchar Ch;
    for (;Ch=*KeyStr++; I^=1)
        if (I)
            Hash *= Ch;
        else    Hash += Ch;
    Hash += ((Hash&0xFFFF0000)>>16);
    Hash += ((Hash&0x0000FF00)>>8);
    return Hash & (N-1);
}

1


Algorithm Alley
by Karl-Dietrich Neubert

Listing One
      SUBROUTINE FLASH1 (A. N. L. M)
C     SORTS ARRY A WITH N ELEMENTS BY USE OF INDEX VECTOR L
C     OF DIMENSION M WITH M ABOUT 0.1 N.
C     COPYRIGHT (C)  K. - D. NEUBERT  1997

      DIMENSION A(1),L(1)
C     ============================ CLASS FORMATION =====
      ANMIN=A(1)
      NMAX=1
      DO I=1,N
         IF( A(I).LT.ANMIN) ANMIN=A(I)
         IF( A(I).GT.A(NMAX)) NMAX=I
      END DO
      IF (ANMIN.EQ.A(NMAX)) RETURN
      C1=(M - 1) / (A(NMAX) - ANMIN))
      DO K=1,M
         L(K)=0
      END DO
      DO I=1,N
         K=1 + INT(C1 * (A(I) - ANMIN))
         L(K)=L(K) + 1
      END DO
      DO K=2,M
         L(K)=L(K) + L(K - 1)
      END DO
      HOLD=A(NMAX)
      A(NMAX)=A(1)
      A(1)=HOLD
C     =============================== PERMUTATION =====
      NMOVE=0
      J=1
      K=M
      DO WHILE (NMOVE.LT.N - 1)
         DO WHILE (J.GT.L(K))
            J=J + 1
            K=1 + INT(C1 * (A(J) - ANMIN))
         END DO
         FLASH=A(J)
         DO WHILE (.NOT.(J.EQ.L(K) + 1))
            K=1 + INT(C1 * (FLASH - ANMIN))
            HOLD=A(L(K))
            A(L(K))=FLASH
            FLASH=HOLD
            L(K)=L(K) - 1
            NMOVE=NMOVE + 1
         END DO
      END DO
C     ========================= STRAIGHT INSERTION =====
      DO I=N-2,1,-1
         IF  (A(I + 1).LT.A(I)) THEN
            HOLD=A(I)
            J=I
            DO WHILE  (A(J + 1).LT.HOLD)
               A(J)=A(J + 1)
               J=J + 1
            END DO
            A(J)=HOLD
         ENDIF
      END DO
C     =========================== RETURN,END FLASH1 =====
      RETURN
      END


Algorithm Alley

by Gary D. Knott

Listing One

FUNCTION F(T)=IF T<=6 THEN SIN(T)+2*COS(3*T)*EXP(-T) ELSE = J-J*EXP(-2*(T-6));

J = -.27614; MINA = .05; MAXA = .41; MAXD = .28; MIND = .05;

T0 = 0; T1 = 11; D = MIND; A = MINA; G = 1.6; C = 1.5;

TYPE "SET T0,T1,A,D, ETC. AS DESIRED."; DO CONSOLE;

FUNCTION ADM(I)= IF T+A<=T1 THEN (B:=((PV:=P(ME[I]:=OLDP(ADM)))<=F(MT[I+1]:=

(T:=T+A))))+0*(A:=NEWA(X1:=X2,X2:=B))+0*(D:=NEWD()) ELSE 1000-I;

FUNCTION OLDP(B)=PV+D*(2*B-1);

FUNCTION P(H1)=H1;

FUNCTION NEWA(X1,X2)=IF X1=X2 THEN MINA ELSE MIN(G*A,MAXA);

FUNCTION NEWD()=IF X1=X2 THEN MIN(D*C,MAXD) ELSE MAX(D/C,MIND);

X2 = 1; ADM =.5; T = T0; IF T1 <= T0 THEN TYPE ("null interval"); = PV = F(T0);

"PRE-ALLOCATE THE ARRAYS MT, ME.";

MT = 0^^360; ME[360] = 0; MT[1] = T0;MB = ADM ON 1:360;

N = MAXI(MB); IF N >=360 THEN TYPE "TOO MANY POINTS.";

ME(N) = OLDP(B); ME = ME ROW 1:N; MT = MT ROW 1:N; MB = MB ROW = 1:(N-1);

SME = MESH(MT,MT)&'ROTATE(MESH(ME,ME),1); DELETE SME ROW (1,N+N);

ME = MT&'ME; MF = POINTS(F,MT);

"MB IS THE DELTA-MODULATION BIT-VECTOR, MF IS THE SAMPLED SIGNAL POINTS, ME IS

THE ESTIMATED SIGNAL POINTS, AND SME IS THE STEP-FUNCTION ESTIMATE.";

Example 2:

(a)

ADMencode(s0, s1, d, a, f):

begin  real x1,x2,t,h,b;

real procedure P: [h <-=C8h+(2b-1)d; return(h)];

b,x2=C8.5; t=C8s0; h=C8f(s0);

send (t0, h,d, a) to ADMdecode; "This send starts the decoder."

while t <=3D  s1 do=20

      [t=C8t+a; x1=C8x2; if f(t) > P then b=C81 else b=C80;

       x2=C8b; output(b);

       if x1=3Dx2 then (a=C8amin; d=C8min(cd,dmax))

       else (d=C8max(d/c,dmin); a=C8min(ga,amax))

     ];

output(end-message); "This will stop the decoder."

end.

(b)

ADMdecode(t, h, d, a):

begin  real x1,x2,h,b;

real procedure P: [h=C8h+(2b-1)d; return(h)];

b,x2=C8.5;

repeat=20

     [output (t,P);

      "Initially, the output point (t,P) is the point (s0,(s0));

        then subsequent points on the -curve are output."

       receive a bit in b, on end-message, goto exit;

       x1=C8x2; x2=C8b; t=C8t+a;

       if x1=3Dx2 then (a=C8amin; d=C8min(cd,dmax))

       else (d=C8max(d/c,dmin); a=C8min(ga,amax))

      ];

exit: end.


Algorithm Alley

by Jon Bentley

Example 1:

(a)

int istrlen(char *s)

{   int n = 0;

    for ( ; *s; s++)

        n++;

    return n;

}

(b)

int rstrlen(char *s)

{   if (*s) return 1+rstrlen(s+1);

    else return 0;

}

Example 2:

(a)

int istrcmp(char *s, char *t)

{   for ( ; *s == *t; s++, t++)

    if (*s == 0)

        return 0;

    return *s - *t;

}

(b)

int rstrcmp(char *s, char *t)

{   if (*s == *t) {

        if (*s)

            return rstrcmp(s+1, t+1);

        else

            return 0;

    } else

        return (*s - *t);

}

(c)

int rcstrcmp(char *s, char *t)

{   return (*s == *t)

           ? (*s ? rcstrcmp(s+1, t+1) : 0)

           : *s - *t;

}

Example 3:

(a)

typedef struct tnode *Tptr;

typedef struct tnode {

    int val;

    Tptr lo, hi;

} Tnode;

(b)

int rbstsearch(Tptr p, int t)

{   if (!p)

        return -1;

    if (t == p->val)

        return 1;

    if (t < p->val)

        return rbstsearch(p->lo, t);

    return rbstsearch(p->hi, t);

}

(c)

int ibstsearch(int t)

{   Tptr p;

    for (p = root; p; ) {

        if (t == p->val)

            return 1;

        p = (t < p->val) ? p->lo : p->hi;

    }

    return -1;

}

(d)

Tptr rinsert(Tptr p, int t)

{   if (p) {

        if (t < p->val)

            p->lo = rinsert(p->lo, t);

        else

            p->hi = rinsert(p->hi, t);

    } else {

        p = (Tptr) malloc(sizeof(Tnode));

        p->lo = p->hi = 0;

        p->val = t;

    }

    return p;

}

Example 4:

void iinsert(int t)

{   Tptr p, q;

    Tptr next = (Tptr) malloc(sizeof(Tnode));

    if (!root)

        root = next;

    else {

        for (q = root; q; ) {

            p = q;

            q = (t < p->val) ? q->lo : q->hi;

        }

        if (t < p->val)

            p->lo = next;

        else

            p->hi = next;

    }

    next->lo = next->hi = 0;

    next->val = t;

}

Example 5:

(a)

void rcount(Tptr p)

{   if (!p) return;

    rcount(p->lo);

    count++;

    rcount(p->hi);

}

(b)

void rcount2(Tptr p)

{   if (p->lo) rcount2(p->lo);

    count++;

    if (p->hi) rcount2(p->hi);

}

Example 6:

(a)

void rcount3(Tptr p)

{   while (p) {

        if (p->lo) rcount3(p->lo);

        count++;

        p = p->hi;

    }

}

(b)

void icount(Tptr p)

{   Tptr s[MAXN];

    int sp = 0;

    for (;;) {

        while (p) {

            s[sp++] = p;

            p = p->lo;

        }

        if (sp == 0) break;

        p = s[--sp];

        count++;

        p = p->hi;

    }

}

Listing One

void iinsertpp(int t)

{   Tptr *p = &root;

    while (*p) {

        if (t < (*p)->val)

            p = &((*p)->lo);

        else

            p = &((*p)->hi);

    }

    *p = (Tptr) malloc(sizeof(Tnode));

    (*p)->lo = (*p)->hi = 0;

    (*p)->val = t;

}

Listing Two

void pmsearch(Tptr p, char *s)

{   if (!p) return;

    nodecnt++;

    if (*s == '.' || *s < p->splitchar)

        pmsearch(p->lokid, s);

    if (*s == '.' || *s == p->splitchar)

        if (p->splitchar && *s)

            pmsearch(p->eqkid, s+1);

    if (*s == 0 && p->splitchar == 0)

        srcharr[srchtop++] =

            (char *) p->eqkid;

    if (*s == '.' || *s > p->splitchar)

        pmsearch(p->hikid, s);

}

Listing Three

void pmsearch2(Tptr p, char *s)

{   while (p) {

        nodecnt++;

        if (*s == '.') {

            pmsearch2(p->lokid, s);

            pmsearch2(p->hikid, s);

            if (p->splitchar == 0) return;

            p = p->eqkid;

            ++s;

        } else {

            if (*s < p->splitchar)

                p = p->lokid;

            else if (*s > p->splitchar)

                p = p->hikid;

            else /* *s == p->splitchar */ {

                if (*s == 0) {

                    if (p->splitchar == 0)

                        srcharr[srchtop++] =

                            (char *) p->eqkid;

                    return;

                }

                p = p->eqkid;

                ++s;

            }

        }

    }

}